2024-09-15T23:04:20.638 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T23:04:20.665 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T23:04:20.756 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/fs/test_o_trunc}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906309' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: 8921c8ef947807b717db041d85f0761b7914d5fc subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt dry_run: true install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3393 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 294/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEfud6y97rX2glgu7dZo0nDNVUlI2rO/4bYetnoC5tKokUT1L+1ibmPrT6S8Dqa8/pHE6mRvqvY/23PpCFSZvDs= smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZyXzkuKPmb191FrDjavJqPbt8VURfosvojKdkywhb9KFeu1b7Jzn3duxlDTEO5DIIZqRR0Wlfin9pfuE/dWsM= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJt43uFMRlIUCMUxUwDIcXkx5aqKEOElGHMLHm9DP00YpNknptG95FezrRckJlJD3zSyHi/hMvyH/NetE2OfptA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - fs/test_o_trunc.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T23:04:20.757 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-15T23:04:20.758 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-15T23:04:20.758 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T23:04:20.759 INFO:teuthology.task.internal:Checking packages... 2024-09-15T23:04:20.784 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-15T23:04:20.784 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T23:04:20.784 INFO:teuthology.packaging:ref: None 2024-09-15T23:04:20.784 INFO:teuthology.packaging:tag: None 2024-09-15T23:04:20.784 INFO:teuthology.packaging:branch: squid 2024-09-15T23:04:20.785 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:04:20.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-15T23:04:21.017 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-15T23:04:21.018 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T23:04:21.026 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T23:04:21.026 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T23:04:21.035 INFO:teuthology.task.internal:Saving configuration 2024-09-15T23:04:21.058 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T23:04:21.066 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T23:04:21.092 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 22:59:22.082534', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEfud6y97rX2glgu7dZo0nDNVUlI2rO/4bYetnoC5tKokUT1L+1ibmPrT6S8Dqa8/pHE6mRvqvY/23PpCFSZvDs='} 2024-09-15T23:04:21.113 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 22:59:22.081600', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAZyXzkuKPmb191FrDjavJqPbt8VURfosvojKdkywhb9KFeu1b7Jzn3duxlDTEO5DIIZqRR0Wlfin9pfuE/dWsM='} 2024-09-15T23:04:21.134 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 22:59:22.080460', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJt43uFMRlIUCMUxUwDIcXkx5aqKEOElGHMLHm9DP00YpNknptG95FezrRckJlJD3zSyHi/hMvyH/NetE2OfptA='} 2024-09-15T23:04:21.134 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T23:04:21.142 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-15T23:04:21.142 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-15T23:04:21.142 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-15T23:04:21.142 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T23:04:21.228 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa38b2c7be0>, signals=[15]) 2024-09-15T23:04:21.228 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T23:04:21.235 INFO:teuthology.task.internal:Opening connections... 2024-09-15T23:04:21.235 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-09-15T23:04:21.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:04:21.318 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2024-09-15T23:04:21.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:04:21.398 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-09-15T23:04:21.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:04:21.474 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T23:04:21.483 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-09-15T23:04:21.503 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-09-15T23:04:21.503 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-09-15T23:04:21.561 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-09-15T23:04:21.561 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="9" 2024-09-15T23:04:21.561 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-09-15T23:04:21.561 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-09-15T23:04:21.561 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="9" 2024-09-15T23:04:21.561 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el9" 2024-09-15T23:04:21.562 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T23:04:21.562 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-09-15T23:04:21.562 INFO:teuthology.orchestra.run.smithi089.stdout:LOGO="fedora-logo-icon" 2024-09-15T23:04:21.562 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T23:04:21.562 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-09-15T23:04:21.562 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T23:04:21.562 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T23:04:21.562 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T23:04:21.564 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-09-15T23:04:21.583 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2024-09-15T23:04:21.602 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2024-09-15T23:04:21.602 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2024-09-15T23:04:21.660 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="9" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="9" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el9" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:LOGO="fedora-logo-icon" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T23:04:21.661 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T23:04:21.662 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T23:04:21.663 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2024-09-15T23:04:21.683 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-09-15T23:04:21.702 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-09-15T23:04:21.702 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-09-15T23:04:21.760 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-09-15T23:04:21.760 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T23:04:21.761 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T23:04:21.762 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-09-15T23:04:21.783 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T23:04:21.794 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T23:04:21.801 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T23:04:21.801 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-09-15T23:04:21.804 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2024-09-15T23:04:21.806 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-09-15T23:04:21.824 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T23:04:21.831 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T23:04:21.831 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-09-15T23:04:21.862 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2024-09-15T23:04:21.865 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-09-15T23:04:21.969 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T23:04:22.050 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T23:04:22.067 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-15T23:04:22.067 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-09-15T23:04:22.067 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-09-15T23:04:22.067 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T23:04:22.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:04:22.085 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-09-15T23:04:22.144 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-503.el9.x86_64 2024-09-15T23:04:22.145 INFO:teuthology.task.kernel:Running kernel on smithi089: 5.14.0-503.el9.x86_64 2024-09-15T23:04:22.145 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2024-09-15T23:04:24.516 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-15T23:04:26.425 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-15T23:04:30.668 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-15T23:04:32.051 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 104 kB/s | 19 kB 00:00 2024-09-15T23:04:33.156 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-15T23:04:38.637 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-15T23:04:40.651 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T23:04:40.651 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T23:04:40.746 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-15T23:04:40.754 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-15T23:04:40.755 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-15T23:04:40.756 INFO:teuthology.orchestra.run.smithi089.stdout:Install 4 Packages 2024-09-15T23:04:40.756 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:04:40.756 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 79 M 2024-09-15T23:04:40.756 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 126 M 2024-09-15T23:04:40.756 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-15T23:04:40.921 INFO:teuthology.orchestra.run.smithi089.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 341 kB/s | 23 kB 00:00 2024-09-15T23:04:44.913 INFO:teuthology.orchestra.run.smithi089.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 3.9 MB/s | 16 MB 00:04 2024-09-15T23:04:48.133 INFO:teuthology.orchestra.run.smithi089.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 4.0 MB/s | 29 MB 00:07 2024-09-15T23:04:49.921 INFO:teuthology.orchestra.run.smithi089.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 3.8 MB/s | 35 MB 00:09 2024-09-15T23:04:49.921 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-15T23:04:49.921 INFO:teuthology.orchestra.run.smithi089.stdout:Total 8.6 MB/s | 79 MB 00:09 2024-09-15T23:04:50.602 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-15T23:04:50.951 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-15T23:04:50.951 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-15T23:04:51.934 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-15T23:04:51.934 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-15T23:04:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-15T23:04:55.091 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T23:04:55.181 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T23:04:57.978 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T23:04:58.221 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T23:05:03.017 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T23:05:03.059 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:05:07.873 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:05:46.292 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:05:46.339 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:05:46.731 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:05:46.731 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T23:05:46.732 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T23:05:46.732 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T23:05:47.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T23:05:47.580 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:05:47.580 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-15T23:05:47.580 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T23:05:47.580 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T23:05:47.580 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T23:05:47.580 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T23:05:47.580 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:05:47.581 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-15T23:05:47.929 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T23:05:48.017 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T23:05:48.017 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T23:05:48.018 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T23:05:48.018 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T23:05:48.018 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T23:05:48.018 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-15T23:05:48.018 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi089.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T23:05:48.018 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T23:05:48.077 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T23:05:48.077 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T23:05:48.077 DEBUG:teuthology.orchestra.run.smithi089:> sudo rpm -qi grub2-tools 2024-09-15T23:05:48.119 INFO:teuthology.orchestra.run.smithi089.stdout:Name : grub2-tools 2024-09-15T23:05:48.119 INFO:teuthology.orchestra.run.smithi089.stdout:Epoch : 1 2024-09-15T23:05:48.119 INFO:teuthology.orchestra.run.smithi089.stdout:Version : 2.06 2024-09-15T23:05:48.119 INFO:teuthology.orchestra.run.smithi089.stdout:Release : 82.el9 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:Architecture: x86_64 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:Group : Unspecified 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:Size : 8274670 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:License : GPLv3+ 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:Packager : builder@centos.org 2024-09-15T23:05:48.120 INFO:teuthology.orchestra.run.smithi089.stdout:Vendor : CentOS 2024-09-15T23:05:48.121 INFO:teuthology.orchestra.run.smithi089.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T23:05:48.121 INFO:teuthology.orchestra.run.smithi089.stdout:Summary : Support tools for GRUB. 2024-09-15T23:05:48.121 INFO:teuthology.orchestra.run.smithi089.stdout:Description : 2024-09-15T23:05:48.121 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:05:48.121 INFO:teuthology.orchestra.run.smithi089.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T23:05:48.121 INFO:teuthology.orchestra.run.smithi089.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T23:05:48.121 INFO:teuthology.orchestra.run.smithi089.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T23:05:48.121 INFO:teuthology.orchestra.run.smithi089.stdout:hardware devices. 2024-09-15T23:05:48.121 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:05:48.121 INFO:teuthology.orchestra.run.smithi089.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T23:05:48.122 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T23:05:48.122 INFO:teuthology.task.kernel:Updating grub on smithi089 to boot 5.14.0-505.el9.x86_64 2024-09-15T23:05:48.122 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T23:05:48.735 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2024-09-15T23:05:50.011 INFO:teuthology.orchestra.run.smithi089.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T23:05:50.043 INFO:teuthology.orchestra.run.smithi089.stderr:done 2024-09-15T23:05:50.046 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-09-15T23:05:50.061 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.8fkOxizcin 2024-09-15T23:05:50.062 DEBUG:teuthology.orchestra.run.smithi089:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8fkOxizcin 2024-09-15T23:05:50.128 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.8fkOxizcin 2024-09-15T23:05:50.279 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.8fkOxizcin is 6KB 2024-09-15T23:05:50.327 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.8fkOxizcin 2024-09-15T23:05:50.341 DEBUG:teuthology.orchestra.run.smithi089:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T23:05:50.404 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T23:05:50.404 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T23:05:50.404 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T23:05:50.404 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T23:05:50.405 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T23:05:50.560 DEBUG:teuthology.orchestra.run.smithi089:> sudo shutdown -r now 2024-09-15T23:06:20.563 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-15T23:06:20.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-15T23:06:20.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:06:39.041 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-09-15T23:06:48.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-15T23:06:48.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:06:51.101 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-09-15T23:07:03.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-15T23:07:03.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:07:21.533 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-09-15T23:07:36.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-15T23:07:36.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:07:37.057 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-09-15T23:07:37.594 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-15T23:07:37.594 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-505.el9.x86_64"... 2024-09-15T23:07:37.594 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-09-15T23:07:37.644 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-505.el9.x86_64 2024-09-15T23:07:37.644 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T23:07:37.644 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T23:07:37.644 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T23:07:38.646 DEBUG:teuthology.parallel:result is None 2024-09-15T23:07:38.646 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T23:07:38.653 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T23:07:38.653 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T23:07:38.656 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T23:07:38.658 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T23:07:38.677 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T23:07:38.683 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T23:07:38.689 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T23:07:38.689 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T23:07:38.716 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T23:07:38.719 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T23:07:38.822 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T23:07:38.828 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T23:07:38.829 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T23:07:38.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:07:38.846 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T23:07:38.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:07:38.864 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T23:07:38.867 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T23:07:38.889 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T23:07:38.932 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T23:07:38.937 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T23:07:38.946 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T23:07:38.953 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T23:07:38.978 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T23:07:39.028 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T23:07:39.030 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T23:07:39.038 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T23:07:39.038 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T23:07:39.040 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T23:07:39.043 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T23:07:39.072 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T23:07:39.082 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T23:07:39.082 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T23:07:39.109 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T23:07:39.111 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T23:07:39.131 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T23:07:39.207 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T23:07:39.267 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T23:07:39.352 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T23:07:39.394 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:07:39.395 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T23:07:39.459 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T23:07:39.504 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T23:07:39.565 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T23:07:39.611 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T23:07:39.652 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:07:39.652 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T23:07:39.718 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T23:07:39.764 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T23:07:39.821 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T23:07:39.906 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T23:07:39.950 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:07:39.950 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T23:07:40.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-09-15T23:07:40.020 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2024-09-15T23:07:40.023 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-09-15T23:07:40.056 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T23:07:40.058 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T23:07:40.096 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T23:07:40.531 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T23:07:40.538 INFO:teuthology.task.internal:Starting timer... 2024-09-15T23:07:40.538 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T23:07:40.549 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T23:07:40.558 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-15T23:07:40.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-09-15T23:07:40.614 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-09-15T23:07:40.616 DEBUG:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2024-09-15T23:07:40.693 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: 2024-09-15T23:07:40.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-09-15T23:07:40.759 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-09-15T23:07:40.761 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T23:07:40.761 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-09-15T23:07:40.794 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-09-15T23:07:40.794 DEBUG:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2024-09-15T23:07:40.817 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2024-09-15T23:07:40.818 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-09-15T23:07:40.844 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-09-15T23:07:40.845 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi089.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi120.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive'} 2024-09-15T23:07:40.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T23:07:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:07:40.871 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T23:07:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:07:40.901 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T23:07:40.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:07:40.930 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T23:07:40.930 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-09-15T23:07:40.958 DEBUG:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2024-09-15T23:07:40.991 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-09-15T23:07:41.028 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T23:07:41.037 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-15T23:07:41.040 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-15T23:07:41.153 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T23:07:41.164 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T23:07:41.165 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi089.front.sepia.ceph.com,smithi120.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T23:11:40.403 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi120.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-09-15T23:11:40.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-15T23:11:40.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:11:40.481 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-09-15T23:11:40.561 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-15T23:11:40.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-09-15T23:11:40.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:11:40.635 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-09-15T23:11:40.712 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-09-15T23:11:40.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-15T23:11:40.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:11:40.789 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-15T23:11:40.887 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-15T23:11:40.888 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T23:11:40.898 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T23:11:40.899 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T23:11:40.899 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:11:40.902 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T23:11:40.902 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:11:40.905 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T23:11:40.905 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:11:40.933 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T23:11:40.934 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T23:11:40.948 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T23:11:40.950 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T23:11:40.960 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T23:11:40.973 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2024-09-15T23:11:40.976 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T23:11:40.977 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-09-15T23:11:40.984 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2024-09-15T23:11:40.988 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-09-15T23:11:40.998 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T23:11:41.002 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-09-15T23:11:41.003 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T23:11:41.012 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T23:11:41.014 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-09-15T23:11:41.017 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T23:11:41.029 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T23:11:41.043 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T23:11:41.068 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:11:41.072 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:11:41.072 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-09-15T23:11:41.072 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.072 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.072 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.072 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.073 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:11:41.076 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:11:41.077 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-09-15T23:11:41.077 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.077 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.077 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.077 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.109 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:11:41.113 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:11:41.113 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-09-15T23:11:41.113 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.113 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.113 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.113 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:11:41.114 INFO:teuthology.run_tasks:Running task install... 2024-09-15T23:11:41.123 DEBUG:teuthology.task.install:project ceph 2024-09-15T23:11:41.123 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-15T23:11:41.123 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-15T23:11:41.123 INFO:teuthology.task.install:Using flavor: default 2024-09-15T23:11:41.129 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T23:11:41.129 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-15T23:11:41.129 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-15T23:11:41.130 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:11:41.132 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-15T23:11:41.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:11:41.133 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-15T23:11:41.133 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:11:41.280 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-15T23:11:41.280 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-15T23:11:41.285 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-15T23:11:41.285 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-15T23:11:41.289 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-15T23:11:41.289 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-15T23:11:41.389 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T23:11:41.389 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:11:41.389 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T23:11:41.396 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T23:11:41.396 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:11:41.396 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T23:11:41.402 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T23:11:41.403 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:11:41.403 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T23:11:41.414 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-15T23:11:41.415 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T23:11:41.422 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-15T23:11:41.423 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T23:11:41.429 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-15T23:11:41.430 DEBUG:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T23:11:41.481 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T23:11:41.488 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T23:11:41.496 DEBUG:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T23:11:41.556 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T23:11:41.563 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T23:11:41.571 DEBUG:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T23:11:41.620 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-09-15T23:11:41.622 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-09-15T23:11:41.627 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-09-15T23:11:41.629 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-09-15T23:11:41.635 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2024-09-15T23:11:41.636 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum clean all 2024-09-15T23:11:41.848 INFO:teuthology.orchestra.run.smithi174.stdout:45 files removed 2024-09-15T23:11:41.859 INFO:teuthology.orchestra.run.smithi089.stdout:45 files removed 2024-09-15T23:11:41.859 INFO:teuthology.orchestra.run.smithi120.stdout:45 files removed 2024-09-15T23:11:41.874 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-15T23:11:41.886 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-15T23:11:41.887 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-15T23:11:42.421 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 312 kB/s | 83 kB 00:00 2024-09-15T23:11:42.441 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 310 kB/s | 83 kB 00:00 2024-09-15T23:11:42.454 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 285 kB/s | 83 kB 00:00 2024-09-15T23:11:42.705 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-15T23:11:42.725 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-15T23:11:42.730 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 50 kB/s | 12 kB 00:00 2024-09-15T23:11:42.980 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-15T23:11:42.997 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-15T23:11:43.010 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 7.6 kB/s | 1.9 kB 00:00 2024-09-15T23:11:43.514 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-15T23:11:44.543 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.2 MB 00:01 2024-09-15T23:11:44.814 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.2 MB 00:01 2024-09-15T23:11:45.373 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-15T23:11:46.995 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-09-15T23:11:47.692 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-15T23:11:49.641 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-15T23:11:51.350 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 19 kB 00:00 2024-09-15T23:11:52.000 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-15T23:11:52.471 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-15T23:11:52.705 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-15T23:11:53.659 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-15T23:11:54.038 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 93 kB/s | 19 kB 00:00 2024-09-15T23:11:54.989 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-15T23:11:55.044 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-15T23:11:58.294 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-09-15T23:12:00.424 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-15T23:12:00.545 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-15T23:12:00.556 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:12:00.557 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:12:00.570 INFO:teuthology.orchestra.run.smithi174.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-15T23:12:00.571 INFO:teuthology.orchestra.run.smithi174.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-15T23:12:00.572 INFO:teuthology.orchestra.run.smithi174.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-15T23:12:00.573 INFO:teuthology.orchestra.run.smithi174.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-15T23:12:00.575 INFO:teuthology.orchestra.run.smithi174.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-15T23:12:00.578 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-15T23:12:00.579 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-15T23:12:00.642 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-15T23:12:00.661 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-09-15T23:12:00.661 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-15T23:12:00.661 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-09-15T23:12:00.661 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-15T23:12:00.661 INFO:teuthology.orchestra.run.smithi174.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-15T23:12:00.661 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-15T23:12:00.661 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-15T23:12:00.662 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-15T23:12:00.663 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-15T23:12:00.664 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-15T23:12:00.665 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T23:12:00.666 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-15T23:12:00.667 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T23:12:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-15T23:12:00.669 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T23:12:00.670 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T23:12:00.671 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T23:12:00.672 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T23:12:00.673 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================== 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout:Install 148 Packages 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 8 Packages 2024-09-15T23:12:00.674 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:12:00.675 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 226 M 2024-09-15T23:12:00.675 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-15T23:12:01.353 INFO:teuthology.orchestra.run.smithi174.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 49 kB/s | 6.5 kB 00:00 2024-09-15T23:12:01.570 INFO:teuthology.orchestra.run.smithi174.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 4.0 MB/s | 896 kB 00:00 2024-09-15T23:12:01.637 INFO:teuthology.orchestra.run.smithi174.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 2.1 MB/s | 145 kB 00:00 2024-09-15T23:12:01.771 INFO:teuthology.orchestra.run.smithi174.stdout:(4/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 9.8 MB/s | 5.4 MB 00:00 2024-09-15T23:12:01.854 INFO:teuthology.orchestra.run.smithi174.stdout:(5/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 11 MB/s | 2.4 MB 00:00 2024-09-15T23:12:01.905 INFO:teuthology.orchestra.run.smithi174.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 11 MB/s | 1.5 MB 00:00 2024-09-15T23:12:02.246 INFO:teuthology.orchestra.run.smithi174.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 12 MB/s | 4.7 MB 00:00 2024-09-15T23:12:02.522 INFO:teuthology.orchestra.run.smithi174.stdout:(8/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 16 MB/s | 21 MB 00:01 2024-09-15T23:12:02.563 INFO:teuthology.orchestra.run.smithi174.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 609 kB/s | 25 kB 00:00 2024-09-15T23:12:02.803 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:12:02.804 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:12:02.817 INFO:teuthology.orchestra.run.smithi089.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-15T23:12:02.818 INFO:teuthology.orchestra.run.smithi089.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-15T23:12:02.819 INFO:teuthology.orchestra.run.smithi089.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-15T23:12:02.820 INFO:teuthology.orchestra.run.smithi089.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-15T23:12:02.822 INFO:teuthology.orchestra.run.smithi089.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-15T23:12:02.824 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-15T23:12:02.825 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-15T23:12:02.889 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-15T23:12:02.897 INFO:teuthology.orchestra.run.smithi174.stdout:(10/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 17 MB/s | 17 MB 00:00 2024-09-15T23:12:02.915 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-09-15T23:12:02.915 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-15T23:12:02.915 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-09-15T23:12:02.915 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-15T23:12:02.915 INFO:teuthology.orchestra.run.smithi089.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-15T23:12:02.915 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-15T23:12:02.915 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-15T23:12:02.916 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-15T23:12:02.917 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-15T23:12:02.918 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-15T23:12:02.919 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T23:12:02.920 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T23:12:02.921 INFO:teuthology.orchestra.run.smithi089.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T23:12:02.922 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T23:12:02.923 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T23:12:02.924 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T23:12:02.925 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-15T23:12:02.926 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout:Installing weak dependencies: 2024-09-15T23:12:02.927 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T23:12:02.928 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:02.928 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-15T23:12:02.928 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-09-15T23:12:02.928 INFO:teuthology.orchestra.run.smithi089.stdout:Install 148 Packages 2024-09-15T23:12:02.928 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 8 Packages 2024-09-15T23:12:02.928 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:02.928 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 226 M 2024-09-15T23:12:02.929 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-15T23:12:02.974 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:12:02.975 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:12:02.988 INFO:teuthology.orchestra.run.smithi120.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-15T23:12:02.989 INFO:teuthology.orchestra.run.smithi120.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-15T23:12:02.990 INFO:teuthology.orchestra.run.smithi120.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-15T23:12:02.991 INFO:teuthology.orchestra.run.smithi120.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-15T23:12:02.993 INFO:teuthology.orchestra.run.smithi120.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-15T23:12:02.996 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-15T23:12:02.997 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-15T23:12:03.014 INFO:teuthology.orchestra.run.smithi174.stdout:(11/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 14 MB/s | 11 MB 00:00 2024-09-15T23:12:03.039 INFO:teuthology.orchestra.run.smithi174.stdout:(12/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 1.6 MB/s | 228 kB 00:00 2024-09-15T23:12:03.060 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-09-15T23:12:03.065 INFO:teuthology.orchestra.run.smithi174.stdout:(13/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 539 kB/s | 27 kB 00:00 2024-09-15T23:12:03.074 INFO:teuthology.orchestra.run.smithi120.stdout:====================================================================================== 2024-09-15T23:12:03.074 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-09-15T23:12:03.074 INFO:teuthology.orchestra.run.smithi120.stdout:====================================================================================== 2024-09-15T23:12:03.074 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-09-15T23:12:03.074 INFO:teuthology.orchestra.run.smithi120.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-15T23:12:03.074 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-15T23:12:03.075 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-15T23:12:03.076 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-15T23:12:03.077 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-15T23:12:03.078 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T23:12:03.079 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T23:12:03.080 INFO:teuthology.orchestra.run.smithi120.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T23:12:03.081 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T23:12:03.082 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T23:12:03.083 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T23:12:03.084 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T23:12:03.085 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-15T23:12:03.086 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T23:12:03.087 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-15T23:12:03.087 INFO:teuthology.orchestra.run.smithi120.stdout:Installing weak dependencies: 2024-09-15T23:12:03.087 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T23:12:03.087 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:03.087 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-09-15T23:12:03.087 INFO:teuthology.orchestra.run.smithi120.stdout:====================================================================================== 2024-09-15T23:12:03.087 INFO:teuthology.orchestra.run.smithi120.stdout:Install 148 Packages 2024-09-15T23:12:03.087 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 8 Packages 2024-09-15T23:12:03.087 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:03.088 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 226 M 2024-09-15T23:12:03.088 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-09-15T23:12:03.106 INFO:teuthology.orchestra.run.smithi174.stdout:(14/156): libcephfs-devel-19.1.1-326.g8921c8ef. 475 kB/s | 32 kB 00:00 2024-09-15T23:12:03.140 INFO:teuthology.orchestra.run.smithi174.stdout:(15/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 9.8 MB/s | 751 kB 00:00 2024-09-15T23:12:03.165 INFO:teuthology.orchestra.run.smithi174.stdout:(16/156): libcephsqlite-19.1.1-326.g8921c8ef.el 2.7 MB/s | 163 kB 00:00 2024-09-15T23:12:03.191 INFO:teuthology.orchestra.run.smithi174.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 2.5 MB/s | 127 kB 00:00 2024-09-15T23:12:03.224 INFO:teuthology.orchestra.run.smithi174.stdout:(18/156): libradosstriper1-19.1.1-326.g8921c8ef 8.4 MB/s | 506 kB 00:00 2024-09-15T23:12:03.283 INFO:teuthology.orchestra.run.smithi174.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 776 kB/s | 45 kB 00:00 2024-09-15T23:12:03.375 INFO:teuthology.orchestra.run.smithi174.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 1.5 MB/s | 136 kB 00:00 2024-09-15T23:12:03.434 INFO:teuthology.orchestra.run.smithi089.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2024-09-15T23:12:03.450 INFO:teuthology.orchestra.run.smithi174.stdout:(21/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 21 MB/s | 5.3 MB 00:00 2024-09-15T23:12:03.475 INFO:teuthology.orchestra.run.smithi174.stdout:(22/156): python3-cephfs-19.1.1-326.g8921c8ef.e 1.6 MB/s | 161 kB 00:00 2024-09-15T23:12:03.501 INFO:teuthology.orchestra.run.smithi174.stdout:(23/156): python3-rados-19.1.1-326.g8921c8ef.el 6.2 MB/s | 320 kB 00:00 2024-09-15T23:12:03.535 INFO:teuthology.orchestra.run.smithi174.stdout:(24/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 5.0 MB/s | 300 kB 00:00 2024-09-15T23:12:03.560 INFO:teuthology.orchestra.run.smithi174.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.7 MB/s | 99 kB 00:00 2024-09-15T23:12:03.585 INFO:teuthology.orchestra.run.smithi174.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-15T23:12:03.677 INFO:teuthology.orchestra.run.smithi174.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 1.8 MB/s | 170 kB 00:00 2024-09-15T23:12:03.717 INFO:teuthology.orchestra.run.smithi089.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 3.1 MB/s | 896 kB 00:00 2024-09-15T23:12:03.760 INFO:teuthology.orchestra.run.smithi174.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 383 kB/s | 32 kB 00:00 2024-09-15T23:12:03.801 INFO:teuthology.orchestra.run.smithi089.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.7 MB/s | 145 kB 00:00 2024-09-15T23:12:03.826 INFO:teuthology.orchestra.run.smithi174.stdout:(29/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 12 MB/s | 3.1 MB 00:00 2024-09-15T23:12:03.859 INFO:teuthology.orchestra.run.smithi174.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 1.4 MB/s | 146 kB 00:00 2024-09-15T23:12:03.885 INFO:teuthology.orchestra.run.smithi089.stdout:(4/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 7.9 MB/s | 5.4 MB 00:00 2024-09-15T23:12:03.988 INFO:teuthology.orchestra.run.smithi120.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2024-09-15T23:12:04.335 INFO:teuthology.orchestra.run.smithi089.stdout:(5/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 4.5 MB/s | 2.4 MB 00:00 2024-09-15T23:12:04.360 INFO:teuthology.orchestra.run.smithi174.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 6.7 MB/s | 3.6 MB 00:00 2024-09-15T23:12:04.443 INFO:teuthology.orchestra.run.smithi174.stdout:(32/156): ceph-mgr-diskprediction-local-19.1.1- 13 MB/s | 7.4 MB 00:00 2024-09-15T23:12:04.469 INFO:teuthology.orchestra.run.smithi174.stdout:(33/156): ceph-mgr-modules-core-19.1.1-326.g892 2.2 MB/s | 249 kB 00:00 2024-09-15T23:12:04.502 INFO:teuthology.orchestra.run.smithi174.stdout:(34/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 842 kB/s | 49 kB 00:00 2024-09-15T23:12:04.527 INFO:teuthology.orchestra.run.smithi174.stdout:(35/156): ceph-prometheus-alerts-19.1.1-326.g89 281 kB/s | 16 kB 00:00 2024-09-15T23:12:04.554 INFO:teuthology.orchestra.run.smithi120.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 1.5 MB/s | 896 kB 00:00 2024-09-15T23:12:04.578 INFO:teuthology.orchestra.run.smithi174.stdout:(36/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 3.7 MB/s | 285 kB 00:00 2024-09-15T23:12:04.653 INFO:teuthology.orchestra.run.smithi174.stdout:(37/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 6.0 MB/s | 765 kB 00:00 2024-09-15T23:12:04.672 INFO:teuthology.orchestra.run.smithi120.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.2 MB/s | 145 kB 00:00 2024-09-15T23:12:04.678 INFO:teuthology.orchestra.run.smithi174.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 182 kB/s | 18 kB 00:00 2024-09-15T23:12:04.735 INFO:teuthology.orchestra.run.smithi089.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 1.8 MB/s | 1.5 MB 00:00 2024-09-15T23:12:04.820 INFO:teuthology.orchestra.run.smithi174.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 479 kB/s | 80 kB 00:00 2024-09-15T23:12:04.929 INFO:teuthology.orchestra.run.smithi174.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 1.7 MB/s | 191 kB 00:00 2024-09-15T23:12:05.013 INFO:teuthology.orchestra.run.smithi174.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-09-15T23:12:05.046 INFO:teuthology.orchestra.run.smithi174.stdout:(42/156): libgfortran-11.5.0-2.el9.x86_64.rpm 2.1 MB/s | 801 kB 00:00 2024-09-15T23:12:05.086 INFO:teuthology.orchestra.run.smithi089.stdout:(7/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 11 MB/s | 21 MB 00:01 2024-09-15T23:12:05.097 INFO:teuthology.orchestra.run.smithi120.stdout:(4/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 5.6 MB/s | 2.4 MB 00:00 2024-09-15T23:12:05.105 INFO:teuthology.orchestra.run.smithi174.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-15T23:12:05.155 INFO:teuthology.orchestra.run.smithi174.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-09-15T23:12:05.161 INFO:teuthology.orchestra.run.smithi089.stdout:(8/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 5.7 MB/s | 4.7 MB 00:00 2024-09-15T23:12:05.203 INFO:teuthology.orchestra.run.smithi089.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 604 kB/s | 25 kB 00:00 2024-09-15T23:12:05.256 INFO:teuthology.orchestra.run.smithi120.stdout:(5/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 9.6 MB/s | 1.5 MB 00:00 2024-09-15T23:12:05.572 INFO:teuthology.orchestra.run.smithi174.stdout:(45/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 16 MB/s | 49 MB 00:03 2024-09-15T23:12:05.598 INFO:teuthology.orchestra.run.smithi174.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 282 kB/s | 125 kB 00:00 2024-09-15T23:12:05.631 INFO:teuthology.orchestra.run.smithi174.stdout:(47/156): python3-cryptography-36.0.1-4.el9.x86 2.0 MB/s | 1.2 MB 00:00 2024-09-15T23:12:05.665 INFO:teuthology.orchestra.run.smithi120.stdout:(6/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 12 MB/s | 4.7 MB 00:00 2024-09-15T23:12:05.678 INFO:teuthology.orchestra.run.smithi089.stdout:(10/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 18 MB/s | 11 MB 00:00 2024-09-15T23:12:05.729 INFO:teuthology.orchestra.run.smithi089.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 4.5 MB/s | 228 kB 00:00 2024-09-15T23:12:05.770 INFO:teuthology.orchestra.run.smithi089.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 650 kB/s | 27 kB 00:00 2024-09-15T23:12:05.821 INFO:teuthology.orchestra.run.smithi089.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 634 kB/s | 32 kB 00:00 2024-09-15T23:12:05.823 INFO:teuthology.orchestra.run.smithi174.stdout:(48/156): python3-urllib3-1.26.5-6.el9.noarch.r 858 kB/s | 215 kB 00:00 2024-09-15T23:12:05.896 INFO:teuthology.orchestra.run.smithi089.stdout:(14/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 9.8 MB/s | 751 kB 00:00 2024-09-15T23:12:05.955 INFO:teuthology.orchestra.run.smithi089.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 2.7 MB/s | 163 kB 00:00 2024-09-15T23:12:06.013 INFO:teuthology.orchestra.run.smithi089.stdout:(16/156): librados-devel-19.1.1-326.g8921c8ef.e 2.1 MB/s | 127 kB 00:00 2024-09-15T23:12:06.089 INFO:teuthology.orchestra.run.smithi089.stdout:(17/156): libradosstriper1-19.1.1-326.g8921c8ef 6.6 MB/s | 506 kB 00:00 2024-09-15T23:12:06.448 INFO:teuthology.orchestra.run.smithi089.stdout:(18/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 15 MB/s | 5.3 MB 00:00 2024-09-15T23:12:06.473 INFO:teuthology.orchestra.run.smithi120.stdout:(7/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_ 21 MB/s | 17 MB 00:00 2024-09-15T23:12:06.490 INFO:teuthology.orchestra.run.smithi174.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 160 kB/s | 107 kB 00:00 2024-09-15T23:12:06.498 INFO:teuthology.orchestra.run.smithi089.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 905 kB/s | 45 kB 00:00 2024-09-15T23:12:06.515 INFO:teuthology.orchestra.run.smithi174.stdout:(50/156): automake-1.16.2-8.el9.noarch.rpm 751 kB/s | 689 kB 00:00 2024-09-15T23:12:06.549 INFO:teuthology.orchestra.run.smithi120.stdout:(8/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 1.9 MB/s | 5.4 MB 00:02 2024-09-15T23:12:06.565 INFO:teuthology.orchestra.run.smithi089.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 2.0 MB/s | 136 kB 00:00 2024-09-15T23:12:06.568 INFO:teuthology.orchestra.run.smithi174.stdout:(51/156): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-09-15T23:12:06.599 INFO:teuthology.orchestra.run.smithi120.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 505 kB/s | 25 kB 00:00 2024-09-15T23:12:06.632 INFO:teuthology.orchestra.run.smithi089.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 2.4 MB/s | 161 kB 00:00 2024-09-15T23:12:06.716 INFO:teuthology.orchestra.run.smithi089.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 3.8 MB/s | 320 kB 00:00 2024-09-15T23:12:06.800 INFO:teuthology.orchestra.run.smithi089.stdout:(23/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 3.5 MB/s | 300 kB 00:00 2024-09-15T23:12:06.850 INFO:teuthology.orchestra.run.smithi089.stdout:(24/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.9 MB/s | 99 kB 00:00 2024-09-15T23:12:06.900 INFO:teuthology.orchestra.run.smithi089.stdout:(25/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-15T23:12:06.907 INFO:teuthology.orchestra.run.smithi120.stdout:(10/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 24 MB/s | 11 MB 00:00 2024-09-15T23:12:06.974 INFO:teuthology.orchestra.run.smithi120.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 3.4 MB/s | 228 kB 00:00 2024-09-15T23:12:07.025 INFO:teuthology.orchestra.run.smithi120.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 539 kB/s | 27 kB 00:00 2024-09-15T23:12:07.032 INFO:teuthology.orchestra.run.smithi174.stdout:(52/156): elfutils-libelf-devel-0.191-4.el9.x86 40 kB/s | 22 kB 00:00 2024-09-15T23:12:07.066 INFO:teuthology.orchestra.run.smithi120.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 761 kB/s | 32 kB 00:00 2024-09-15T23:12:07.117 INFO:teuthology.orchestra.run.smithi120.stdout:(14/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 15 MB/s | 751 kB 00:00 2024-09-15T23:12:07.149 INFO:teuthology.orchestra.run.smithi174.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 56 kB/s | 33 kB 00:00 2024-09-15T23:12:07.150 INFO:teuthology.orchestra.run.smithi089.stdout:(26/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 7.0 MB/s | 17 MB 00:02 2024-09-15T23:12:07.167 INFO:teuthology.orchestra.run.smithi120.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 3.2 MB/s | 163 kB 00:00 2024-09-15T23:12:07.209 INFO:teuthology.orchestra.run.smithi089.stdout:(27/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 10 MB/s | 3.1 MB 00:00 2024-09-15T23:12:07.217 INFO:teuthology.orchestra.run.smithi120.stdout:(16/156): librados-devel-19.1.1-326.g8921c8ef.e 2.5 MB/s | 127 kB 00:00 2024-09-15T23:12:07.243 INFO:teuthology.orchestra.run.smithi089.stdout:(28/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 1.8 MB/s | 170 kB 00:00 2024-09-15T23:12:07.268 INFO:teuthology.orchestra.run.smithi120.stdout:(17/156): libradosstriper1-19.1.1-326.g8921c8ef 9.9 MB/s | 506 kB 00:00 2024-09-15T23:12:07.268 INFO:teuthology.orchestra.run.smithi089.stdout:(29/156): ceph-grafana-dashboards-19.1.1-326.g8 543 kB/s | 32 kB 00:00 2024-09-15T23:12:07.302 INFO:teuthology.orchestra.run.smithi089.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.4 MB/s | 146 kB 00:00 2024-09-15T23:12:07.349 INFO:teuthology.orchestra.run.smithi174.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 376 kB/s | 313 kB 00:00 2024-09-15T23:12:07.535 INFO:teuthology.orchestra.run.smithi120.stdout:(18/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 20 MB/s | 5.3 MB 00:00 2024-09-15T23:12:07.568 INFO:teuthology.orchestra.run.smithi120.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 1.3 MB/s | 45 kB 00:00 2024-09-15T23:12:07.577 INFO:teuthology.orchestra.run.smithi089.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 12 MB/s | 3.6 MB 00:00 2024-09-15T23:12:07.619 INFO:teuthology.orchestra.run.smithi120.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 2.7 MB/s | 136 kB 00:00 2024-09-15T23:12:07.627 INFO:teuthology.orchestra.run.smithi089.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 4.9 MB/s | 249 kB 00:00 2024-09-15T23:12:07.666 INFO:teuthology.orchestra.run.smithi174.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 35 kB/s | 18 kB 00:00 2024-09-15T23:12:07.669 INFO:teuthology.orchestra.run.smithi120.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 3.2 MB/s | 161 kB 00:00 2024-09-15T23:12:07.669 INFO:teuthology.orchestra.run.smithi089.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 1.2 MB/s | 49 kB 00:00 2024-09-15T23:12:07.720 INFO:teuthology.orchestra.run.smithi089.stdout:(34/156): ceph-prometheus-alerts-19.1.1-326.g89 329 kB/s | 16 kB 00:00 2024-09-15T23:12:07.736 INFO:teuthology.orchestra.run.smithi120.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 4.7 MB/s | 320 kB 00:00 2024-09-15T23:12:07.778 INFO:teuthology.orchestra.run.smithi089.stdout:(35/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 4.8 MB/s | 285 kB 00:00 2024-09-15T23:12:07.786 INFO:teuthology.orchestra.run.smithi120.stdout:(23/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 5.9 MB/s | 300 kB 00:00 2024-09-15T23:12:07.816 INFO:teuthology.orchestra.run.smithi174.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 24 kB/s | 11 kB 00:00 2024-09-15T23:12:07.845 INFO:teuthology.orchestra.run.smithi120.stdout:(24/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.7 MB/s | 99 kB 00:00 2024-09-15T23:12:07.854 INFO:teuthology.orchestra.run.smithi089.stdout:(36/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 10 MB/s | 765 kB 00:00 2024-09-15T23:12:07.896 INFO:teuthology.orchestra.run.smithi120.stdout:(25/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-15T23:12:07.904 INFO:teuthology.orchestra.run.smithi089.stdout:(37/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 366 kB/s | 18 kB 00:00 2024-09-15T23:12:08.037 INFO:teuthology.orchestra.run.smithi120.stdout:(26/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 22 MB/s | 3.1 MB 00:00 2024-09-15T23:12:08.054 INFO:teuthology.orchestra.run.smithi089.stdout:(38/156): ceph-mgr-diskprediction-local-19.1.1- 9.8 MB/s | 7.4 MB 00:00 2024-09-15T23:12:08.067 INFO:teuthology.orchestra.run.smithi174.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.9 MB/s | 3.0 MB 00:01 2024-09-15T23:12:08.088 INFO:teuthology.orchestra.run.smithi120.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 3.3 MB/s | 170 kB 00:00 2024-09-15T23:12:08.130 INFO:teuthology.orchestra.run.smithi120.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 762 kB/s | 32 kB 00:00 2024-09-15T23:12:08.163 INFO:teuthology.orchestra.run.smithi089.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 308 kB/s | 80 kB 00:00 2024-09-15T23:12:08.189 INFO:teuthology.orchestra.run.smithi120.stdout:(29/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.4 MB/s | 146 kB 00:00 2024-09-15T23:12:08.247 INFO:teuthology.orchestra.run.smithi089.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 191 kB 00:00 2024-09-15T23:12:08.314 INFO:teuthology.orchestra.run.smithi089.stdout:(41/156): libgfortran-11.5.0-2.el9.x86_64.rpm 3.0 MB/s | 801 kB 00:00 2024-09-15T23:12:08.339 INFO:teuthology.orchestra.run.smithi120.stdout:(30/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 24 MB/s | 3.6 MB 00:00 2024-09-15T23:12:08.339 INFO:teuthology.orchestra.run.smithi089.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-09-15T23:12:08.390 INFO:teuthology.orchestra.run.smithi089.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-15T23:12:08.417 INFO:teuthology.orchestra.run.smithi174.stdout:(58/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 213 kB/s | 160 kB 00:00 2024-09-15T23:12:08.431 INFO:teuthology.orchestra.run.smithi089.stdout:(44/156): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.2 MB 00:00 2024-09-15T23:12:08.442 INFO:teuthology.orchestra.run.smithi174.stdout:(59/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-09-15T23:12:08.457 INFO:teuthology.orchestra.run.smithi089.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-09-15T23:12:08.482 INFO:teuthology.orchestra.run.smithi089.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-09-15T23:12:08.516 INFO:teuthology.orchestra.run.smithi089.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-09-15T23:12:08.598 INFO:teuthology.orchestra.run.smithi120.stdout:(31/156): ceph-mgr-diskprediction-local-19.1.1- 29 MB/s | 7.4 MB 00:00 2024-09-15T23:12:08.656 INFO:teuthology.orchestra.run.smithi120.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 4.2 MB/s | 249 kB 00:00 2024-09-15T23:12:08.706 INFO:teuthology.orchestra.run.smithi120.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 990 kB/s | 49 kB 00:00 2024-09-15T23:12:08.754 INFO:teuthology.orchestra.run.smithi120.stdout:(34/156): ceph-prometheus-alerts-19.1.1-326.g89 347 kB/s | 16 kB 00:00 2024-09-15T23:12:08.804 INFO:teuthology.orchestra.run.smithi120.stdout:(35/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 5.6 MB/s | 285 kB 00:00 2024-09-15T23:12:08.863 INFO:teuthology.orchestra.run.smithi120.stdout:(36/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 13 MB/s | 765 kB 00:00 2024-09-15T23:12:08.914 INFO:teuthology.orchestra.run.smithi120.stdout:(37/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 366 kB/s | 18 kB 00:00 2024-09-15T23:12:08.951 INFO:teuthology.orchestra.run.smithi174.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 757 kB/s | 662 kB 00:00 2024-09-15T23:12:08.976 INFO:teuthology.orchestra.run.smithi174.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 46 kB/s | 24 kB 00:00 2024-09-15T23:12:09.039 INFO:teuthology.orchestra.run.smithi120.stdout:(38/156): fuse-2.9.9-16.el9.x86_64.rpm 640 kB/s | 80 kB 00:00 2024-09-15T23:12:09.139 INFO:teuthology.orchestra.run.smithi120.stdout:(39/156): libgfortran-11.5.0-2.el9.x86_64.rpm 7.8 MB/s | 801 kB 00:00 2024-09-15T23:12:09.181 INFO:teuthology.orchestra.run.smithi120.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-15T23:12:09.208 INFO:teuthology.orchestra.run.smithi089.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 950 kB/s | 689 kB 00:00 2024-09-15T23:12:09.223 INFO:teuthology.orchestra.run.smithi120.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.9 MB/s | 253 kB 00:00 2024-09-15T23:12:09.242 INFO:teuthology.orchestra.run.smithi089.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 1.3 MB/s | 943 kB 00:00 2024-09-15T23:12:09.293 INFO:teuthology.orchestra.run.smithi174.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 664 kB/s | 581 kB 00:00 2024-09-15T23:12:09.299 INFO:teuthology.orchestra.run.smithi120.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-09-15T23:12:09.341 INFO:teuthology.orchestra.run.smithi120.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-15T23:12:09.383 INFO:teuthology.orchestra.run.smithi120.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-09-15T23:12:09.425 INFO:teuthology.orchestra.run.smithi120.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-09-15T23:12:09.466 INFO:teuthology.orchestra.run.smithi120.stdout:(46/156): python3-urllib3-1.26.5-6.el9.noarch.r 5.1 MB/s | 215 kB 00:00 2024-09-15T23:12:09.526 INFO:teuthology.orchestra.run.smithi174.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 83 kB/s | 47 kB 00:00 2024-09-15T23:12:09.667 INFO:teuthology.orchestra.run.smithi120.stdout:(47/156): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 689 kB 00:00 2024-09-15T23:12:09.692 INFO:teuthology.orchestra.run.smithi089.stdout:(50/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 11 MB/s | 49 MB 00:04 2024-09-15T23:12:09.717 INFO:teuthology.orchestra.run.smithi089.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 46 kB/s | 22 kB 00:00 2024-09-15T23:12:09.742 INFO:teuthology.orchestra.run.smithi089.stdout:(52/156): boost-program-options-1.75.0-8.el9.x8 200 kB/s | 107 kB 00:00 2024-09-15T23:12:09.759 INFO:teuthology.orchestra.run.smithi120.stdout:(48/156): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 943 kB 00:00 2024-09-15T23:12:09.785 INFO:teuthology.orchestra.run.smithi174.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 362 kB/s | 292 kB 00:00 2024-09-15T23:12:09.843 INFO:teuthology.orchestra.run.smithi089.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-09-15T23:12:09.876 INFO:teuthology.orchestra.run.smithi120.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 918 kB/s | 107 kB 00:00 2024-09-15T23:12:09.951 INFO:teuthology.orchestra.run.smithi120.stdout:(50/156): elfutils-libelf-devel-0.191-4.el9.x86 289 kB/s | 22 kB 00:00 2024-09-15T23:12:09.977 INFO:teuthology.orchestra.run.smithi089.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 136 kB/s | 18 kB 00:00 2024-09-15T23:12:10.018 INFO:teuthology.orchestra.run.smithi120.stdout:(51/156): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-09-15T23:12:10.043 INFO:teuthology.orchestra.run.smithi174.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 251 kB/s | 188 kB 00:00 2024-09-15T23:12:10.094 INFO:teuthology.orchestra.run.smithi174.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 69 kB/s | 38 kB 00:00 2024-09-15T23:12:10.102 INFO:teuthology.orchestra.run.smithi089.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 89 kB/s | 11 kB 00:00 2024-09-15T23:12:10.141 INFO:teuthology.orchestra.run.smithi120.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 268 kB/s | 33 kB 00:00 2024-09-15T23:12:10.160 INFO:teuthology.orchestra.run.smithi089.stdout:(56/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.2 MB/s | 3.0 MB 00:00 2024-09-15T23:12:10.211 INFO:teuthology.orchestra.run.smithi089.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-09-15T23:12:10.261 INFO:teuthology.orchestra.run.smithi120.stdout:(53/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 25 MB/s | 3.0 MB 00:00 2024-09-15T23:12:10.269 INFO:teuthology.orchestra.run.smithi089.stdout:(58/156): flex-2.6.4-9.el9.x86_64.rpm 544 kB/s | 313 kB 00:00 2024-09-15T23:12:10.286 INFO:teuthology.orchestra.run.smithi089.stdout:(59/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 360 kB/s | 45 kB 00:00 2024-09-15T23:12:10.336 INFO:teuthology.orchestra.run.smithi120.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 243 kB/s | 18 kB 00:00 2024-09-15T23:12:10.395 INFO:teuthology.orchestra.run.smithi120.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-09-15T23:12:10.395 INFO:teuthology.orchestra.run.smithi089.stdout:(60/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 226 kB/s | 24 kB 00:00 2024-09-15T23:12:10.420 INFO:teuthology.orchestra.run.smithi089.stdout:(61/156): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-09-15T23:12:10.471 INFO:teuthology.orchestra.run.smithi089.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 2.8 MB/s | 581 kB 00:00 2024-09-15T23:12:10.495 INFO:teuthology.orchestra.run.smithi120.stdout:(56/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-09-15T23:12:10.496 INFO:teuthology.orchestra.run.smithi089.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 471 kB/s | 47 kB 00:00 2024-09-15T23:12:10.538 INFO:teuthology.orchestra.run.smithi089.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-15T23:12:10.562 INFO:teuthology.orchestra.run.smithi120.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 680 kB/s | 45 kB 00:00 2024-09-15T23:12:10.588 INFO:teuthology.orchestra.run.smithi089.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-09-15T23:12:10.614 INFO:teuthology.orchestra.run.smithi089.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 327 kB/s | 38 kB 00:00 2024-09-15T23:12:10.764 INFO:teuthology.orchestra.run.smithi089.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-15T23:12:10.795 INFO:teuthology.orchestra.run.smithi120.stdout:(58/156): ceph-common-19.1.1-326.g8921c8ef.el9. 3.0 MB/s | 21 MB 00:07 2024-09-15T23:12:10.837 INFO:teuthology.orchestra.run.smithi120.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-09-15T23:12:10.906 INFO:teuthology.orchestra.run.smithi089.stdout:(68/156): postgresql-contrib-13.16-1.el9.x86_64 6.3 MB/s | 910 kB 00:00 2024-09-15T23:12:10.913 INFO:teuthology.orchestra.run.smithi120.stdout:(60/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 326 kB/s | 24 kB 00:00 2024-09-15T23:12:10.936 INFO:teuthology.orchestra.run.smithi174.stdout:(67/156): openblas-openmp-0.3.26-2.el9.x86_64.r 4.3 MB/s | 4.9 MB 00:01 2024-09-15T23:12:10.996 INFO:teuthology.orchestra.run.smithi120.stdout:(61/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 571 kB/s | 47 kB 00:00 2024-09-15T23:12:11.015 INFO:teuthology.orchestra.run.smithi089.stdout:(69/156): openblas-openmp-0.3.26-2.el9.x86_64.r 10 MB/s | 4.9 MB 00:00 2024-09-15T23:12:11.030 INFO:teuthology.orchestra.run.smithi120.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 2.4 MB/s | 581 kB 00:00 2024-09-15T23:12:11.040 INFO:teuthology.orchestra.run.smithi089.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 1.0 MB/s | 137 kB 00:00 2024-09-15T23:12:11.064 INFO:teuthology.orchestra.run.smithi120.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-09-15T23:12:11.099 INFO:teuthology.orchestra.run.smithi089.stdout:(71/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-09-15T23:12:11.130 INFO:teuthology.orchestra.run.smithi120.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-09-15T23:12:11.164 INFO:teuthology.orchestra.run.smithi120.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 382 kB/s | 38 kB 00:00 2024-09-15T23:12:11.169 INFO:teuthology.orchestra.run.smithi174.stdout:(68/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-09-15T23:12:11.182 INFO:teuthology.orchestra.run.smithi089.stdout:(72/156): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-09-15T23:12:11.207 INFO:teuthology.orchestra.run.smithi089.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 804 kB/s | 87 kB 00:00 2024-09-15T23:12:11.219 INFO:teuthology.orchestra.run.smithi174.stdout:(69/156): postgresql-13.16-1.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-09-15T23:12:11.300 INFO:teuthology.orchestra.run.smithi089.stdout:(74/156): postgresql-server-13.16-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-09-15T23:12:11.323 INFO:teuthology.orchestra.run.smithi120.stdout:(66/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-09-15T23:12:11.333 INFO:teuthology.orchestra.run.smithi089.stdout:(75/156): python3-colorama-0.4.6-3.el9.noarch.r 488 kB/s | 61 kB 00:00 2024-09-15T23:12:11.399 INFO:teuthology.orchestra.run.smithi120.stdout:(67/156): openblas-openmp-0.3.26-2.el9.x86_64.r 18 MB/s | 4.9 MB 00:00 2024-09-15T23:12:11.425 INFO:teuthology.orchestra.run.smithi089.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-15T23:12:11.449 INFO:teuthology.orchestra.run.smithi120.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-15T23:12:11.459 INFO:teuthology.orchestra.run.smithi089.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-09-15T23:12:11.499 INFO:teuthology.orchestra.run.smithi120.stdout:(69/156): postgresql-contrib-13.16-1.el9.x86_64 8.9 MB/s | 910 kB 00:00 2024-09-15T23:12:11.525 INFO:teuthology.orchestra.run.smithi120.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 1.8 MB/s | 137 kB 00:00 2024-09-15T23:12:11.551 INFO:teuthology.orchestra.run.smithi089.stdout:(78/156): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-15T23:12:11.576 INFO:teuthology.orchestra.run.smithi089.stdout:(79/156): python3-jsonpatch-1.21-16.el9.noarch. 170 kB/s | 26 kB 00:00 2024-09-15T23:12:11.600 INFO:teuthology.orchestra.run.smithi120.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-09-15T23:12:11.601 INFO:teuthology.orchestra.run.smithi089.stdout:(80/156): python3-jsonpointer-2.0-4.el9.noarch. 135 kB/s | 19 kB 00:00 2024-09-15T23:12:11.660 INFO:teuthology.orchestra.run.smithi089.stdout:(81/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-15T23:12:11.710 INFO:teuthology.orchestra.run.smithi089.stdout:(82/156): python3-markupsafe-1.1.1-12.el9.x86_6 320 kB/s | 35 kB 00:00 2024-09-15T23:12:11.734 INFO:teuthology.orchestra.run.smithi120.stdout:(72/156): postgresql-server-13.16-1.el9.x86_64. 25 MB/s | 5.8 MB 00:00 2024-09-15T23:12:11.759 INFO:teuthology.orchestra.run.smithi120.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 552 kB/s | 87 kB 00:00 2024-09-15T23:12:11.777 INFO:teuthology.orchestra.run.smithi089.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-09-15T23:12:11.826 INFO:teuthology.orchestra.run.smithi120.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 918 kB/s | 61 kB 00:00 2024-09-15T23:12:11.836 INFO:teuthology.orchestra.run.smithi089.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-09-15T23:12:11.870 INFO:teuthology.orchestra.run.smithi174.stdout:(70/156): postgresql-contrib-13.16-1.el9.x86_64 975 kB/s | 910 kB 00:00 2024-09-15T23:12:11.895 INFO:teuthology.orchestra.run.smithi174.stdout:(71/156): postgresql-private-libs-13.16-1.el9.x 191 kB/s | 137 kB 00:00 2024-09-15T23:12:11.911 INFO:teuthology.orchestra.run.smithi089.stdout:(85/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-09-15T23:12:11.926 INFO:teuthology.orchestra.run.smithi120.stdout:(75/156): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 6.0 MB 00:00 2024-09-15T23:12:11.936 INFO:teuthology.orchestra.run.smithi089.stdout:(86/156): python3-packaging-20.9-5.el9.noarch.r 485 kB/s | 77 kB 00:00 2024-09-15T23:12:11.952 INFO:teuthology.orchestra.run.smithi120.stdout:(76/156): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-15T23:12:11.962 INFO:teuthology.orchestra.run.smithi089.stdout:(87/156): python3-prettytable-0.7.2-27.el9.noar 336 kB/s | 42 kB 00:00 2024-09-15T23:12:12.002 INFO:teuthology.orchestra.run.smithi120.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-09-15T23:12:12.019 INFO:teuthology.orchestra.run.smithi120.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 383 kB/s | 26 kB 00:00 2024-09-15T23:12:12.029 INFO:teuthology.orchestra.run.smithi089.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 269 kB 00:00 2024-09-15T23:12:12.054 INFO:teuthology.orchestra.run.smithi089.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-15T23:12:12.061 INFO:teuthology.orchestra.run.smithi120.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 327 kB/s | 19 kB 00:00 2024-09-15T23:12:12.079 INFO:teuthology.orchestra.run.smithi089.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-09-15T23:12:12.086 INFO:teuthology.orchestra.run.smithi120.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-15T23:12:12.128 INFO:teuthology.orchestra.run.smithi120.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 520 kB/s | 35 kB 00:00 2024-09-15T23:12:12.130 INFO:teuthology.orchestra.run.smithi089.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 506 kB/s | 51 kB 00:00 2024-09-15T23:12:12.155 INFO:teuthology.orchestra.run.smithi089.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 532 kB/s | 54 kB 00:00 2024-09-15T23:12:12.229 INFO:teuthology.orchestra.run.smithi120.stdout:(82/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-09-15T23:12:12.230 INFO:teuthology.orchestra.run.smithi089.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 413 kB/s | 42 kB 00:00 2024-09-15T23:12:12.254 INFO:teuthology.orchestra.run.smithi120.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 250 kB 00:00 2024-09-15T23:12:12.256 INFO:teuthology.orchestra.run.smithi089.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 440 kB/s | 44 kB 00:00 2024-09-15T23:12:12.304 INFO:teuthology.orchestra.run.smithi120.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-09-15T23:12:12.338 INFO:teuthology.orchestra.run.smithi120.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 923 kB/s | 77 kB 00:00 2024-09-15T23:12:12.348 INFO:teuthology.orchestra.run.smithi089.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-15T23:12:12.362 INFO:teuthology.orchestra.run.smithi174.stdout:(72/156): postgresql-server-13.16-1.el9.x86_64. 5.1 MB/s | 5.8 MB 00:01 2024-09-15T23:12:12.371 INFO:teuthology.orchestra.run.smithi120.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 632 kB/s | 42 kB 00:00 2024-09-15T23:12:12.373 INFO:teuthology.orchestra.run.smithi089.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 307 kB/s | 36 kB 00:00 2024-09-15T23:12:12.422 INFO:teuthology.orchestra.run.smithi120.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 269 kB 00:00 2024-09-15T23:12:12.447 INFO:teuthology.orchestra.run.smithi120.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-09-15T23:12:12.457 INFO:teuthology.orchestra.run.smithi089.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 436 kB/s | 47 kB 00:00 2024-09-15T23:12:12.497 INFO:teuthology.orchestra.run.smithi120.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-09-15T23:12:12.499 INFO:teuthology.orchestra.run.smithi089.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-15T23:12:12.531 INFO:teuthology.orchestra.run.smithi120.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 608 kB/s | 51 kB 00:00 2024-09-15T23:12:12.566 INFO:teuthology.orchestra.run.smithi089.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 531 kB/s | 58 kB 00:00 2024-09-15T23:12:12.573 INFO:teuthology.orchestra.run.smithi120.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 710 kB/s | 54 kB 00:00 2024-09-15T23:12:12.604 INFO:teuthology.orchestra.run.smithi174.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 123 kB/s | 87 kB 00:00 2024-09-15T23:12:12.616 INFO:teuthology.orchestra.run.smithi089.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 818 kB/s | 96 kB 00:00 2024-09-15T23:12:12.640 INFO:teuthology.orchestra.run.smithi120.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 626 kB/s | 42 kB 00:00 2024-09-15T23:12:12.716 INFO:teuthology.orchestra.run.smithi120.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 591 kB/s | 44 kB 00:00 2024-09-15T23:12:12.775 INFO:teuthology.orchestra.run.smithi089.stdout:(101/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-09-15T23:12:12.791 INFO:teuthology.orchestra.run.smithi120.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-09-15T23:12:12.800 INFO:teuthology.orchestra.run.smithi089.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 244 kB/s | 45 kB 00:00 2024-09-15T23:12:12.821 INFO:teuthology.orchestra.run.smithi174.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-09-15T23:12:12.825 INFO:teuthology.orchestra.run.smithi089.stdout:(103/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 245 kB/s | 64 kB 00:00 2024-09-15T23:12:12.892 INFO:teuthology.orchestra.run.smithi089.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 8.0 MB/s | 548 kB 00:00 2024-09-15T23:12:12.935 INFO:teuthology.orchestra.run.smithi089.stdout:(105/156): lua-devel-5.4.4-4.el9.x86_64.rpm 156 kB/s | 22 kB 00:00 2024-09-15T23:12:12.960 INFO:teuthology.orchestra.run.smithi089.stdout:(106/156): gperftools-libs-2.9.1-3.el9.x86_64.r 4.5 MB/s | 308 kB 00:00 2024-09-15T23:12:12.966 INFO:teuthology.orchestra.run.smithi120.stdout:(95/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-09-15T23:12:12.991 INFO:teuthology.orchestra.run.smithi120.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 179 kB/s | 36 kB 00:00 2024-09-15T23:12:13.002 INFO:teuthology.orchestra.run.smithi089.stdout:(107/156): protobuf-compiler-3.14.0-13.el9.x86_ 4.2 MB/s | 863 kB 00:00 2024-09-15T23:12:13.027 INFO:teuthology.orchestra.run.smithi089.stdout:(108/156): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-09-15T23:12:13.034 INFO:teuthology.orchestra.run.smithi120.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 710 kB/s | 47 kB 00:00 2024-09-15T23:12:13.061 INFO:teuthology.orchestra.run.smithi089.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.8 MB/s | 49 kB 00:00 2024-09-15T23:12:13.076 INFO:teuthology.orchestra.run.smithi120.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 3.5 MB/s | 304 kB 00:00 2024-09-15T23:12:13.094 INFO:teuthology.orchestra.run.smithi089.stdout:(110/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 282 kB/s | 26 kB 00:00 2024-09-15T23:12:13.101 INFO:teuthology.orchestra.run.smithi120.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 864 kB/s | 58 kB 00:00 2024-09-15T23:12:13.119 INFO:teuthology.orchestra.run.smithi089.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-09-15T23:12:13.143 INFO:teuthology.orchestra.run.smithi120.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.4 MB/s | 96 kB 00:00 2024-09-15T23:12:13.161 INFO:teuthology.orchestra.run.smithi089.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-15T23:12:13.168 INFO:teuthology.orchestra.run.smithi120.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 953 kB/s | 64 kB 00:00 2024-09-15T23:12:13.202 INFO:teuthology.orchestra.run.smithi120.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 767 kB/s | 45 kB 00:00 2024-09-15T23:12:13.212 INFO:teuthology.orchestra.run.smithi089.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-09-15T23:12:13.227 INFO:teuthology.orchestra.run.smithi120.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 380 kB/s | 22 kB 00:00 2024-09-15T23:12:13.237 INFO:teuthology.orchestra.run.smithi089.stdout:(114/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-15T23:12:13.262 INFO:teuthology.orchestra.run.smithi089.stdout:(115/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 860 kB/s | 43 kB 00:00 2024-09-15T23:12:13.271 INFO:teuthology.orchestra.run.smithi174.stdout:(75/156): python3-colorama-0.4.6-3.el9.noarch.r 92 kB/s | 61 kB 00:00 2024-09-15T23:12:13.286 INFO:teuthology.orchestra.run.smithi120.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 9.2 MB/s | 548 kB 00:00 2024-09-15T23:12:13.288 INFO:teuthology.orchestra.run.smithi089.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 639 kB/s | 32 kB 00:00 2024-09-15T23:12:13.313 INFO:teuthology.orchestra.run.smithi089.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 280 kB/s | 14 kB 00:00 2024-09-15T23:12:13.319 INFO:teuthology.orchestra.run.smithi120.stdout:(105/156): gperftools-libs-2.9.1-3.el9.x86_64.r 9.1 MB/s | 308 kB 00:00 2024-09-15T23:12:13.336 INFO:teuthology.orchestra.run.smithi120.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-15T23:12:13.339 INFO:teuthology.orchestra.run.smithi089.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-15T23:12:13.395 INFO:teuthology.orchestra.run.smithi120.stdout:(107/156): protobuf-compiler-3.14.0-13.el9.x86_ 4.4 MB/s | 863 kB 00:00 2024-09-15T23:12:13.414 INFO:teuthology.orchestra.run.smithi089.stdout:(119/156): libarrow-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-09-15T23:12:13.437 INFO:teuthology.orchestra.run.smithi120.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2024-09-15T23:12:13.439 INFO:teuthology.orchestra.run.smithi089.stdout:(120/156): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-09-15T23:12:13.454 INFO:teuthology.orchestra.run.smithi120.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 2.8 MB/s | 49 kB 00:00 2024-09-15T23:12:13.465 INFO:teuthology.orchestra.run.smithi089.stdout:(121/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-09-15T23:12:13.488 INFO:teuthology.orchestra.run.smithi120.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-15T23:12:13.490 INFO:teuthology.orchestra.run.smithi089.stdout:(122/156): python3-google-auth-2.34.0-1.el9.noa 2.8 MB/s | 220 kB 00:00 2024-09-15T23:12:13.515 INFO:teuthology.orchestra.run.smithi089.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-15T23:12:13.521 INFO:teuthology.orchestra.run.smithi120.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 4.5 MB/s | 151 kB 00:00 2024-09-15T23:12:13.538 INFO:teuthology.orchestra.run.smithi174.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 5.1 MB/s | 6.0 MB 00:01 2024-09-15T23:12:13.541 INFO:teuthology.orchestra.run.smithi089.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2024-09-15T23:12:13.558 INFO:teuthology.orchestra.run.smithi089.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 419 kB/s | 18 kB 00:00 2024-09-15T23:12:13.575 INFO:teuthology.orchestra.run.smithi089.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 691 kB/s | 23 kB 00:00 2024-09-15T23:12:13.580 INFO:teuthology.orchestra.run.smithi174.stdout:(77/156): python3-devel-3.9.19-8.el9.x86_64.rpm 323 kB/s | 245 kB 00:00 2024-09-15T23:12:13.588 INFO:teuthology.orchestra.run.smithi120.stdout:(112/156): libarrow-9.0.0-11.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-09-15T23:12:13.592 INFO:teuthology.orchestra.run.smithi089.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 580 kB/s | 19 kB 00:00 2024-09-15T23:12:13.609 INFO:teuthology.orchestra.run.smithi089.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 584 kB/s | 20 kB 00:00 2024-09-15T23:12:13.630 INFO:teuthology.orchestra.run.smithi120.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-09-15T23:12:13.634 INFO:teuthology.orchestra.run.smithi089.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-15T23:12:13.659 INFO:teuthology.orchestra.run.smithi089.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-15T23:12:13.664 INFO:teuthology.orchestra.run.smithi120.stdout:(114/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.8 MB/s | 839 kB 00:00 2024-09-15T23:12:13.681 INFO:teuthology.orchestra.run.smithi120.stdout:(115/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 863 kB/s | 43 kB 00:00 2024-09-15T23:12:13.685 INFO:teuthology.orchestra.run.smithi089.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-15T23:12:13.698 INFO:teuthology.orchestra.run.smithi120.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 949 kB/s | 32 kB 00:00 2024-09-15T23:12:13.715 INFO:teuthology.orchestra.run.smithi120.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 418 kB/s | 14 kB 00:00 2024-09-15T23:12:13.735 INFO:teuthology.orchestra.run.smithi089.stdout:(132/156): python3-grpcio-1.46.7-10.el9.x86_64. 6.9 MB/s | 2.0 MB 00:00 2024-09-15T23:12:13.741 INFO:teuthology.orchestra.run.smithi120.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-09-15T23:12:13.760 INFO:teuthology.orchestra.run.smithi089.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 657 kB/s | 16 kB 00:00 2024-09-15T23:12:13.766 INFO:teuthology.orchestra.run.smithi120.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-09-15T23:12:13.786 INFO:teuthology.orchestra.run.smithi089.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-15T23:12:13.791 INFO:teuthology.orchestra.run.smithi120.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-09-15T23:12:13.811 INFO:teuthology.orchestra.run.smithi089.stdout:(135/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-15T23:12:13.817 INFO:teuthology.orchestra.run.smithi120.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-15T23:12:13.836 INFO:teuthology.orchestra.run.smithi089.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-09-15T23:12:13.842 INFO:teuthology.orchestra.run.smithi120.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-09-15T23:12:13.868 INFO:teuthology.orchestra.run.smithi120.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 430 kB/s | 11 kB 00:00 2024-09-15T23:12:13.870 INFO:teuthology.orchestra.run.smithi089.stdout:(137/156): python3-kubernetes-26.1.0-2.el9.noar 3.7 MB/s | 1.0 MB 00:00 2024-09-15T23:12:13.893 INFO:teuthology.orchestra.run.smithi120.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 707 kB/s | 18 kB 00:00 2024-09-15T23:12:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 366 kB/s | 31 kB 00:00 2024-09-15T23:12:13.918 INFO:teuthology.orchestra.run.smithi120.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 928 kB/s | 23 kB 00:00 2024-09-15T23:12:13.920 INFO:teuthology.orchestra.run.smithi089.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-09-15T23:12:13.943 INFO:teuthology.orchestra.run.smithi120.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 781 kB/s | 19 kB 00:00 2024-09-15T23:12:13.945 INFO:teuthology.orchestra.run.smithi089.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-09-15T23:12:13.968 INFO:teuthology.orchestra.run.smithi120.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 787 kB/s | 20 kB 00:00 2024-09-15T23:12:13.971 INFO:teuthology.orchestra.run.smithi089.stdout:(141/156): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2024-09-15T23:12:13.996 INFO:teuthology.orchestra.run.smithi089.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 986 kB/s | 74 kB 00:00 2024-09-15T23:12:14.019 INFO:teuthology.orchestra.run.smithi120.stdout:(128/156): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-09-15T23:12:14.022 INFO:teuthology.orchestra.run.smithi089.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-15T23:12:14.044 INFO:teuthology.orchestra.run.smithi120.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-15T23:12:14.047 INFO:teuthology.orchestra.run.smithi089.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-15T23:12:14.070 INFO:teuthology.orchestra.run.smithi120.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-15T23:12:14.072 INFO:teuthology.orchestra.run.smithi174.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 39 kB/s | 19 kB 00:00 2024-09-15T23:12:14.072 INFO:teuthology.orchestra.run.smithi089.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 396 kB/s | 20 kB 00:00 2024-09-15T23:12:14.095 INFO:teuthology.orchestra.run.smithi120.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-15T23:12:14.098 INFO:teuthology.orchestra.run.smithi089.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-15T23:12:14.123 INFO:teuthology.orchestra.run.smithi089.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-09-15T23:12:14.137 INFO:teuthology.orchestra.run.smithi120.stdout:(132/156): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-09-15T23:12:14.138 INFO:teuthology.orchestra.run.smithi174.stdout:(79/156): python3-jsonpatch-1.21-16.el9.noarch. 43 kB/s | 26 kB 00:00 2024-09-15T23:12:14.163 INFO:teuthology.orchestra.run.smithi120.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 659 kB/s | 16 kB 00:00 2024-09-15T23:12:14.164 INFO:teuthology.orchestra.run.smithi174.stdout:(80/156): python3-jinja2-2.11.3-6.el9.noarch.rp 278 kB/s | 248 kB 00:00 2024-09-15T23:12:14.172 INFO:teuthology.orchestra.run.smithi089.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-15T23:12:14.188 INFO:teuthology.orchestra.run.smithi120.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-09-15T23:12:14.213 INFO:teuthology.orchestra.run.smithi120.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-09-15T23:12:14.238 INFO:teuthology.orchestra.run.smithi120.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-09-15T23:12:14.264 INFO:teuthology.orchestra.run.smithi120.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-09-15T23:12:14.289 INFO:teuthology.orchestra.run.smithi120.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-15T23:12:14.314 INFO:teuthology.orchestra.run.smithi120.stdout:(139/156): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-09-15T23:12:14.348 INFO:teuthology.orchestra.run.smithi120.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-09-15T23:12:14.373 INFO:teuthology.orchestra.run.smithi120.stdout:(141/156): python3-typing-extensions-4.12.2-2.e 1.2 MB/s | 74 kB 00:00 2024-09-15T23:12:14.399 INFO:teuthology.orchestra.run.smithi120.stdout:(142/156): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-15T23:12:14.406 INFO:teuthology.orchestra.run.smithi089.stdout:(149/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 11 MB/s | 3.2 MB 00:00 2024-09-15T23:12:14.424 INFO:teuthology.orchestra.run.smithi120.stdout:(143/156): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-15T23:12:14.449 INFO:teuthology.orchestra.run.smithi120.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 804 kB/s | 20 kB 00:00 2024-09-15T23:12:14.473 INFO:teuthology.orchestra.run.smithi089.stdout:(150/156): librados2-19.1.1-326.g8921c8ef.el9.x 9.1 MB/s | 3.4 MB 00:00 2024-09-15T23:12:14.491 INFO:teuthology.orchestra.run.smithi120.stdout:(145/156): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-09-15T23:12:14.517 INFO:teuthology.orchestra.run.smithi120.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-15T23:12:14.540 INFO:teuthology.orchestra.run.smithi089.stdout:(151/156): NetworkManager-1.49.90-1.el9.x86_64. 6.3 MB/s | 2.3 MB 00:00 2024-09-15T23:12:14.598 INFO:teuthology.orchestra.run.smithi089.stdout:(152/156): NetworkManager-tui-1.49.90-1.el9.x86 4.2 MB/s | 249 kB 00:00 2024-09-15T23:12:14.649 INFO:teuthology.orchestra.run.smithi089.stdout:(153/156): NetworkManager-team-1.49.90-1.el9.x8 232 kB/s | 41 kB 00:00 2024-09-15T23:12:14.689 INFO:teuthology.orchestra.run.smithi174.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 63 kB/s | 35 kB 00:00 2024-09-15T23:12:14.699 INFO:teuthology.orchestra.run.smithi089.stdout:(154/156): openssl-3.2.2-6.el9.x86_64.rpm 13 MB/s | 1.4 MB 00:00 2024-09-15T23:12:14.741 INFO:teuthology.orchestra.run.smithi089.stdout:(155/156): NetworkManager-libnm-1.49.90-1.el9.x 5.4 MB/s | 1.8 MB 00:00 2024-09-15T23:12:14.881 INFO:teuthology.orchestra.run.smithi174.stdout:(82/156): python3-mako-1.1.4-6.el9.noarch.rpm 213 kB/s | 172 kB 00:00 2024-09-15T23:12:14.908 INFO:teuthology.orchestra.run.smithi089.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 9.2 MB/s | 2.4 MB 00:00 2024-09-15T23:12:14.913 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-15T23:12:14.913 INFO:teuthology.orchestra.run.smithi089.stdout:Total 19 MB/s | 226 MB 00:11 2024-09-15T23:12:14.984 INFO:teuthology.orchestra.run.smithi120.stdout:(147/156): ceph-test-19.1.1-326.g8921c8ef.el9.x 5.8 MB/s | 49 MB 00:08 2024-09-15T23:12:15.034 INFO:teuthology.orchestra.run.smithi120.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00 2024-09-15T23:12:15.093 INFO:teuthology.orchestra.run.smithi120.stdout:(149/156): librados2-19.1.1-326.g8921c8ef.el9.x 5.9 MB/s | 3.4 MB 00:00 2024-09-15T23:12:15.268 INFO:teuthology.orchestra.run.smithi120.stdout:(150/156): NetworkManager-1.49.90-1.el9.x86_64. 9.9 MB/s | 2.3 MB 00:00 2024-09-15T23:12:15.273 INFO:teuthology.orchestra.run.smithi174.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.6 MB/s | 5.1 MB 00:01 2024-09-15T23:12:15.318 INFO:teuthology.orchestra.run.smithi120.stdout:(151/156): NetworkManager-libnm-1.49.90-1.el9.x 8.0 MB/s | 1.8 MB 00:00 2024-09-15T23:12:15.344 INFO:teuthology.orchestra.run.smithi120.stdout:(152/156): NetworkManager-team-1.49.90-1.el9.x8 543 kB/s | 41 kB 00:00 2024-09-15T23:12:15.369 INFO:teuthology.orchestra.run.smithi120.stdout:(153/156): NetworkManager-tui-1.49.90-1.el9.x86 4.8 MB/s | 249 kB 00:00 2024-09-15T23:12:15.419 INFO:teuthology.orchestra.run.smithi120.stdout:(154/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 7.3 MB/s | 3.2 MB 00:00 2024-09-15T23:12:15.470 INFO:teuthology.orchestra.run.smithi120.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 11 MB/s | 1.4 MB 00:00 2024-09-15T23:12:15.511 INFO:teuthology.orchestra.run.smithi120.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2024-09-15T23:12:15.517 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-09-15T23:12:15.517 INFO:teuthology.orchestra.run.smithi120.stdout:Total 18 MB/s | 226 MB 00:12 2024-09-15T23:12:15.540 INFO:teuthology.orchestra.run.smithi174.stdout:(84/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 295 kB/s | 250 kB 00:00 2024-09-15T23:12:15.665 INFO:teuthology.orchestra.run.smithi174.stdout:(85/156): python3-oauthlib-3.1.1-5.el9.noarch.r 283 kB/s | 222 kB 00:00 2024-09-15T23:12:15.957 INFO:teuthology.orchestra.run.smithi174.stdout:(86/156): python3-packaging-20.9-5.el9.noarch.r 113 kB/s | 77 kB 00:00 2024-09-15T23:12:16.103 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-15T23:12:16.115 INFO:teuthology.orchestra.run.smithi174.stdout:(87/156): python3-prettytable-0.7.2-27.el9.noar 73 kB/s | 42 kB 00:00 2024-09-15T23:12:16.255 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-15T23:12:16.255 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-15T23:12:16.499 INFO:teuthology.orchestra.run.smithi174.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 323 kB/s | 269 kB 00:00 2024-09-15T23:12:16.741 INFO:teuthology.orchestra.run.smithi174.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 203 kB/s | 159 kB 00:00 2024-09-15T23:12:16.742 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-09-15T23:12:16.893 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-09-15T23:12:16.893 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-09-15T23:12:16.950 INFO:teuthology.orchestra.run.smithi174.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 335 kB/s | 279 kB 00:00 2024-09-15T23:12:17.169 INFO:teuthology.orchestra.run.smithi174.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 76 kB/s | 51 kB 00:00 2024-09-15T23:12:17.342 INFO:teuthology.orchestra.run.smithi174.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 90 kB/s | 54 kB 00:00 2024-09-15T23:12:17.750 INFO:teuthology.orchestra.run.smithi174.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 72 kB/s | 42 kB 00:00 2024-09-15T23:12:17.917 INFO:teuthology.orchestra.run.smithi174.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 77 kB/s | 44 kB 00:00 2024-09-15T23:12:18.418 INFO:teuthology.orchestra.run.smithi174.stdout:(95/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-15T23:12:18.481 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-15T23:12:18.481 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-15T23:12:18.618 INFO:teuthology.orchestra.run.smithi174.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 51 kB/s | 36 kB 00:00 2024-09-15T23:12:18.694 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-09-15T23:12:18.694 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-09-15T23:12:18.718 INFO:teuthology.orchestra.run.smithi174.stdout:(97/156): qatlib-24.02.0-1.el9.x86_64.rpm 228 kB/s | 221 kB 00:00 2024-09-15T23:12:19.001 INFO:teuthology.orchestra.run.smithi174.stdout:(98/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 81 kB/s | 47 kB 00:00 2024-09-15T23:12:19.344 INFO:teuthology.orchestra.run.smithi174.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 93 kB/s | 58 kB 00:00 2024-09-15T23:12:19.452 INFO:teuthology.orchestra.run.smithi174.stdout:(100/156): socat-1.7.4.1-6.el9.x86_64.rpm 365 kB/s | 304 kB 00:00 2024-09-15T23:12:19.669 INFO:teuthology.orchestra.run.smithi174.stdout:(101/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 144 kB/s | 96 kB 00:00 2024-09-15T23:12:19.761 INFO:teuthology.orchestra.run.smithi174.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 269 kB/s | 22 kB 00:00 2024-09-15T23:12:19.903 INFO:teuthology.orchestra.run.smithi174.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 6.0 MB/s | 863 kB 00:00 2024-09-15T23:12:19.978 INFO:teuthology.orchestra.run.smithi174.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 7.2 MB/s | 548 kB 00:00 2024-09-15T23:12:20.012 INFO:teuthology.orchestra.run.smithi174.stdout:(105/156): gperftools-libs-2.9.1-3.el9.x86_64.r 9.1 MB/s | 308 kB 00:00 2024-09-15T23:12:20.037 INFO:teuthology.orchestra.run.smithi174.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-15T23:12:20.062 INFO:teuthology.orchestra.run.smithi174.stdout:(107/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 89 kB/s | 64 kB 00:00 2024-09-15T23:12:20.088 INFO:teuthology.orchestra.run.smithi174.stdout:(108/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 71 kB/s | 45 kB 00:00 2024-09-15T23:12:20.138 INFO:teuthology.orchestra.run.smithi174.stdout:(109/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 353 kB/s | 26 kB 00:00 2024-09-15T23:12:20.171 INFO:teuthology.orchestra.run.smithi174.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.1 MB/s | 67 kB 00:00 2024-09-15T23:12:20.197 INFO:teuthology.orchestra.run.smithi174.stdout:(111/156): liboath-2.6.7-2.el9.x86_64.rpm 481 kB/s | 49 kB 00:00 2024-09-15T23:12:20.230 INFO:teuthology.orchestra.run.smithi174.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-15T23:12:20.289 INFO:teuthology.orchestra.run.smithi174.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 9.2 MB/s | 548 kB 00:00 2024-09-15T23:12:20.314 INFO:teuthology.orchestra.run.smithi174.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-15T23:12:20.340 INFO:teuthology.orchestra.run.smithi174.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-15T23:12:20.390 INFO:teuthology.orchestra.run.smithi174.stdout:(116/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.3 MB/s | 839 kB 00:00 2024-09-15T23:12:20.415 INFO:teuthology.orchestra.run.smithi174.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 188 kB/s | 14 kB 00:00 2024-09-15T23:12:20.440 INFO:teuthology.orchestra.run.smithi174.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-15T23:12:20.482 INFO:teuthology.orchestra.run.smithi174.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-09-15T23:12:20.549 INFO:teuthology.orchestra.run.smithi174.stdout:(120/156): libarrow-9.0.0-11.el9.x86_64.rpm 8.6 MB/s | 4.4 MB 00:00 2024-09-15T23:12:20.574 INFO:teuthology.orchestra.run.smithi174.stdout:(121/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-09-15T23:12:20.600 INFO:teuthology.orchestra.run.smithi174.stdout:(122/156): python3-google-auth-2.34.0-1.el9.noa 1.8 MB/s | 220 kB 00:00 2024-09-15T23:12:20.625 INFO:teuthology.orchestra.run.smithi174.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-15T23:12:20.651 INFO:teuthology.orchestra.run.smithi174.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-09-15T23:12:20.676 INFO:teuthology.orchestra.run.smithi174.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-09-15T23:12:20.709 INFO:teuthology.orchestra.run.smithi174.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 397 kB/s | 23 kB 00:00 2024-09-15T23:12:20.734 INFO:teuthology.orchestra.run.smithi174.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 333 kB/s | 19 kB 00:00 2024-09-15T23:12:20.760 INFO:teuthology.orchestra.run.smithi174.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-09-15T23:12:20.785 INFO:teuthology.orchestra.run.smithi174.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-15T23:12:20.811 INFO:teuthology.orchestra.run.smithi174.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-15T23:12:20.836 INFO:teuthology.orchestra.run.smithi174.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-15T23:12:20.878 INFO:teuthology.orchestra.run.smithi174.stdout:(132/156): python3-kubernetes-26.1.0-2.el9.noar 7.1 MB/s | 1.0 MB 00:00 2024-09-15T23:12:20.903 INFO:teuthology.orchestra.run.smithi174.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-09-15T23:12:20.928 INFO:teuthology.orchestra.run.smithi174.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-09-15T23:12:20.954 INFO:teuthology.orchestra.run.smithi174.stdout:(135/156): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-09-15T23:12:20.979 INFO:teuthology.orchestra.run.smithi174.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-09-15T23:12:21.004 INFO:teuthology.orchestra.run.smithi174.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-09-15T23:12:21.029 INFO:teuthology.orchestra.run.smithi174.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-15T23:12:21.055 INFO:teuthology.orchestra.run.smithi174.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-09-15T23:12:21.080 INFO:teuthology.orchestra.run.smithi174.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 711 kB/s | 36 kB 00:00 2024-09-15T23:12:21.122 INFO:teuthology.orchestra.run.smithi174.stdout:(141/156): python3-grpcio-1.46.7-10.el9.x86_64. 3.6 MB/s | 2.0 MB 00:00 2024-09-15T23:12:21.147 INFO:teuthology.orchestra.run.smithi174.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 809 kB/s | 74 kB 00:00 2024-09-15T23:12:21.172 INFO:teuthology.orchestra.run.smithi174.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-15T23:12:21.206 INFO:teuthology.orchestra.run.smithi174.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-09-15T23:12:21.231 INFO:teuthology.orchestra.run.smithi174.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-09-15T23:12:21.257 INFO:teuthology.orchestra.run.smithi174.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-15T23:12:21.282 INFO:teuthology.orchestra.run.smithi174.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-15T23:12:21.332 INFO:teuthology.orchestra.run.smithi174.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-15T23:12:21.699 INFO:teuthology.orchestra.run.smithi174.stdout:(149/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 7.6 MB/s | 3.2 MB 00:00 2024-09-15T23:12:21.758 INFO:teuthology.orchestra.run.smithi174.stdout:(150/156): librados2-19.1.1-326.g8921c8ef.el9.x 6.8 MB/s | 3.4 MB 00:00 2024-09-15T23:12:21.875 INFO:teuthology.orchestra.run.smithi174.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 350 kB/s | 41 kB 00:00 2024-09-15T23:12:21.917 INFO:teuthology.orchestra.run.smithi174.stdout:(152/156): NetworkManager-1.49.90-1.el9.x86_64. 4.0 MB/s | 2.3 MB 00:00 2024-09-15T23:12:22.025 INFO:teuthology.orchestra.run.smithi174.stdout:(153/156): NetworkManager-tui-1.49.90-1.el9.x86 1.6 MB/s | 249 kB 00:00 2024-09-15T23:12:22.109 INFO:teuthology.orchestra.run.smithi174.stdout:(154/156): NetworkManager-libnm-1.49.90-1.el9.x 4.4 MB/s | 1.8 MB 00:00 2024-09-15T23:12:22.735 INFO:teuthology.orchestra.run.smithi174.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 1.7 MB/s | 1.4 MB 00:00 2024-09-15T23:12:22.934 INFO:teuthology.orchestra.run.smithi174.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 2.6 MB/s | 2.4 MB 00:00 2024-09-15T23:12:22.942 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-15T23:12:22.942 INFO:teuthology.orchestra.run.smithi174.stdout:Total 10 MB/s | 226 MB 00:22 2024-09-15T23:12:24.189 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-15T23:12:24.339 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-15T23:12:24.339 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-15T23:12:24.924 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-15T23:12:25.029 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-15T23:12:25.075 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-15T23:12:25.137 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-15T23:12:25.170 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-15T23:12:25.196 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-15T23:12:25.203 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-09-15T23:12:25.257 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-15T23:12:25.307 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-15T23:12:25.345 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-15T23:12:25.416 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-15T23:12:25.466 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-15T23:12:25.492 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-15T23:12:25.534 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-15T23:12:25.598 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-15T23:12:25.706 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:12:25.829 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:12:25.856 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-15T23:12:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:12:25.948 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:12:25.968 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:12:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:12:26.090 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:12:26.111 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:12:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-15T23:12:26.201 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:12:26.219 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-15T23:12:26.220 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-15T23:12:26.226 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-15T23:12:26.232 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:12:26.254 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:12:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:12:26.378 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-15T23:12:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:12:26.438 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-15T23:12:26.443 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:12:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:12:26.465 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:12:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:12:26.568 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:12:26.591 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:12:26.654 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:12:26.659 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:12:26.676 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:12:26.696 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:12:26.832 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:12:26.885 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:12:26.972 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-15T23:12:27.026 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:12:27.093 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:12:27.150 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-15T23:12:27.182 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:12:27.249 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:12:27.318 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:12:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:12:27.525 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-15T23:12:27.544 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:12:27.616 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-15T23:12:27.654 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-15T23:12:27.658 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-15T23:12:27.697 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-15T23:12:27.763 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-15T23:12:27.770 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-15T23:12:27.803 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-15T23:12:27.823 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-15T23:12:27.838 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-15T23:12:27.864 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-15T23:12:27.907 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-15T23:12:27.912 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-15T23:12:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-15T23:12:27.964 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-15T23:12:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-15T23:12:28.012 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-15T23:12:28.036 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-15T23:12:28.056 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-15T23:12:28.088 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-15T23:12:28.098 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-15T23:12:28.116 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-15T23:12:28.139 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-15T23:12:28.182 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-15T23:12:28.186 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-15T23:12:28.248 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-15T23:12:28.258 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-15T23:12:28.278 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-15T23:12:28.293 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-15T23:12:28.322 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-15T23:12:28.345 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-15T23:12:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-15T23:12:28.408 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-15T23:12:28.441 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-15T23:12:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-15T23:12:28.484 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-15T23:12:28.488 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-15T23:12:28.513 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-15T23:12:28.562 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-15T23:12:28.582 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-15T23:12:28.628 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-15T23:12:28.702 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-15T23:12:28.704 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-15T23:12:28.766 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-15T23:12:28.843 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-15T23:12:28.897 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-15T23:12:29.279 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-15T23:12:29.369 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-15T23:12:29.408 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-15T23:12:29.417 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-15T23:12:29.451 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-15T23:12:29.489 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-15T23:12:29.502 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-15T23:12:29.539 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-15T23:12:29.539 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-15T23:12:29.581 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-15T23:12:29.619 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-15T23:12:29.661 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-15T23:12:29.718 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-15T23:12:29.776 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-15T23:12:29.823 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-15T23:12:29.830 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-15T23:12:29.871 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-15T23:12:29.926 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-15T23:12:29.928 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-15T23:12:29.960 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-15T23:12:29.996 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-15T23:12:29.998 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-15T23:12:30.030 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-15T23:12:30.043 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-15T23:12:30.066 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-15T23:12:30.087 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-15T23:12:30.114 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-15T23:12:30.138 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-15T23:12:30.151 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-15T23:12:30.181 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-15T23:12:30.194 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-15T23:12:30.245 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-15T23:12:30.317 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-15T23:12:30.367 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-15T23:12:30.390 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-15T23:12:30.447 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-15T23:12:30.488 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-15T23:12:30.600 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-15T23:12:30.655 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-15T23:12:30.841 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-15T23:12:30.864 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-15T23:12:30.877 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-15T23:12:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-15T23:12:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-15T23:12:30.991 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-15T23:12:31.014 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-15T23:12:31.047 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-15T23:12:31.079 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-15T23:12:31.100 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-15T23:12:31.169 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-15T23:12:31.229 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-15T23:12:31.262 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-15T23:12:31.299 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-15T23:12:31.299 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-15T23:12:31.337 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-15T23:12:31.342 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-15T23:12:31.365 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-15T23:12:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-15T23:12:31.408 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-15T23:12:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-15T23:12:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-15T23:12:31.447 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-15T23:12:31.476 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-15T23:12:31.487 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-15T23:12:31.518 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-15T23:12:31.525 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-15T23:12:31.560 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-15T23:12:31.567 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-15T23:12:31.593 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-15T23:12:31.602 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-15T23:12:31.635 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-15T23:12:31.669 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-15T23:12:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-15T23:12:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-15T23:12:31.756 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-15T23:12:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-15T23:12:31.800 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-15T23:12:31.839 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-15T23:12:31.866 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-15T23:12:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-15T23:12:31.942 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-15T23:12:31.991 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-15T23:12:32.638 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-15T23:12:32.688 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-15T23:12:32.713 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-15T23:12:32.722 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-15T23:12:32.763 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-15T23:12:32.778 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-15T23:12:32.806 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-15T23:12:32.829 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-15T23:12:32.884 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-15T23:12:32.904 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-15T23:12:32.912 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-15T23:12:32.929 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-15T23:12:32.942 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-15T23:12:32.977 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-15T23:12:32.995 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-15T23:12:33.017 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-15T23:12:33.022 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-15T23:12:33.030 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-15T23:12:33.056 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-15T23:12:33.065 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-15T23:12:33.098 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-15T23:12:33.113 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-15T23:12:33.142 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-15T23:12:33.251 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-15T23:12:33.288 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-15T23:12:33.352 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-15T23:12:33.370 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-15T23:12:33.390 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-15T23:12:33.420 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-15T23:12:33.469 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-15T23:12:33.523 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:12:33.646 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:12:33.674 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:12:33.749 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:12:33.786 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:12:33.946 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-15T23:12:33.991 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-15T23:12:34.020 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:12:34.024 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-15T23:12:34.115 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-15T23:12:34.122 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:12:34.141 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-15T23:12:34.145 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:12:34.225 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:12:34.239 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-15T23:12:34.246 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:12:34.274 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:12:34.473 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:12:34.601 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:12:34.719 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-15T23:12:34.818 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:12:34.896 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:12:35.116 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:12:35.227 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-15T23:12:35.326 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-15T23:12:35.354 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-15T23:12:35.386 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-15T23:12:35.397 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-15T23:12:35.470 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-15T23:12:35.484 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-15T23:12:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-15T23:12:35.514 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-15T23:12:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-15T23:12:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-15T23:12:35.554 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-15T23:12:35.580 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-15T23:12:35.606 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-15T23:12:35.608 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-15T23:12:35.639 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-15T23:12:35.641 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-15T23:12:35.667 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-15T23:12:35.736 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-15T23:12:35.789 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-15T23:12:35.820 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-15T23:12:35.863 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-15T23:12:35.867 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-15T23:12:35.921 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:12:35.940 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-15T23:12:35.949 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-15T23:12:35.974 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-15T23:12:35.995 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:12:36.003 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:12:36.017 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-15T23:12:36.034 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-15T23:12:36.055 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-15T23:12:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-15T23:12:36.087 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:12:36.118 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-15T23:12:36.126 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-15T23:12:36.162 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-15T23:12:36.169 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-15T23:12:36.244 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-15T23:12:36.380 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-15T23:12:36.431 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-15T23:12:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-15T23:12:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:12:36.650 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-15T23:12:36.653 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:12:36.675 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-15T23:12:36.682 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:12:36.753 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:12:36.784 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-15T23:12:36.958 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-15T23:12:37.035 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-15T23:12:37.063 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-15T23:12:37.112 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-15T23:12:37.150 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-15T23:12:37.192 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-15T23:12:37.356 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-15T23:12:37.422 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-15T23:12:37.477 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-15T23:12:37.550 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-15T23:12:37.581 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-15T23:12:37.626 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-15T23:12:37.699 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-15T23:12:37.733 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-15T23:12:37.776 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-15T23:12:37.821 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-15T23:12:37.956 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-15T23:12:38.004 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-15T23:12:38.045 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-15T23:12:38.150 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-15T23:12:38.187 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-15T23:12:38.226 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-15T23:12:38.252 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-15T23:12:38.277 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-15T23:12:38.353 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-15T23:12:38.478 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-15T23:12:38.518 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-15T23:12:38.554 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-15T23:12:38.596 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-15T23:12:38.635 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-15T23:12:38.667 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-15T23:12:38.697 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-15T23:12:38.731 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-15T23:12:38.765 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-15T23:12:38.806 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-15T23:12:38.841 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-15T23:12:38.928 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-15T23:12:38.971 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-15T23:12:39.010 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-15T23:12:39.012 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:12:39.049 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:12:39.090 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:12:39.113 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-15T23:12:39.131 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:12:39.170 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-15T23:12:39.618 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:12:39.660 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:12:39.675 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:12:39.718 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:12:39.751 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:12:39.751 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T23:12:39.751 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T23:12:39.752 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:12:39.814 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:12:39.814 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T23:12:39.814 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T23:12:39.815 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:39.845 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:12:39.898 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-15T23:12:39.950 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-15T23:12:39.984 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-15T23:12:40.086 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-15T23:12:40.147 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-15T23:12:40.192 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-15T23:12:40.281 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-15T23:12:40.320 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-15T23:12:40.504 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-15T23:12:40.543 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-15T23:12:40.620 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-15T23:12:41.269 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-15T23:12:41.382 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-15T23:12:42.623 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-15T23:12:42.695 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-15T23:12:42.731 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-15T23:12:42.772 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-15T23:12:43.083 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-15T23:12:43.119 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:12:43.209 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:12:43.249 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-15T23:12:43.284 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-15T23:12:43.764 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-15T23:12:43.788 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:12:43.860 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:12:43.890 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-15T23:12:46.203 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:12:46.244 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:12:46.788 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:12:46.831 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:12:46.922 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:12:46.922 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T23:12:46.923 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T23:12:46.923 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:12:46.957 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:12:51.688 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:12:51.688 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-09-15T23:12:51.688 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-09-15T23:12:51.688 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-09-15T23:12:51.688 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-09-15T23:12:51.689 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-09-15T23:12:51.689 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /root 2024-09-15T23:12:51.689 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-09-15T23:12:51.689 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:51.749 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:12:51.749 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2024-09-15T23:12:51.749 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2024-09-15T23:12:51.749 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2024-09-15T23:12:51.749 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2024-09-15T23:12:51.749 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2024-09-15T23:12:51.749 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /root 2024-09-15T23:12:51.749 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2024-09-15T23:12:51.749 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:51.870 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:12:51.886 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:12:51.886 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:51.887 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T23:12:51.887 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:51.919 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:12:51.936 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:12:51.936 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:51.937 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T23:12:51.937 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:53.366 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:12:53.392 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:12:53.392 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:53.392 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T23:12:53.392 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:12:53.392 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:12:53.393 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:53.431 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:12:53.458 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:12:53.458 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:53.458 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T23:12:53.458 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:12:53.459 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:12:53.459 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:53.616 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:12:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:12:53.640 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:53.640 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T23:12:53.640 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:12:53.640 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:12:53.640 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:53.682 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:12:53.705 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:12:53.705 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:53.705 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T23:12:53.705 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:12:53.706 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:12:53.706 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:54.067 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:12:54.091 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:12:54.091 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:54.091 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T23:12:54.091 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:12:54.091 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:12:54.091 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:54.141 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:12:54.164 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:12:54.164 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:54.164 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T23:12:54.164 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:12:54.164 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:12:54.165 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-15T23:12:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-15T23:12:54.271 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-15T23:12:54.310 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-15T23:12:54.345 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-15T23:12:54.409 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-15T23:12:54.494 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-15T23:12:54.552 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-15T23:12:54.595 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-15T23:12:54.629 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-15T23:12:54.651 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-15T23:12:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-15T23:12:54.737 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-15T23:12:54.825 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-15T23:12:54.848 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-15T23:12:54.900 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-15T23:12:54.934 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-15T23:12:54.962 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-15T23:12:54.973 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-15T23:12:55.018 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-15T23:12:55.067 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-15T23:12:55.095 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-15T23:12:55.391 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:12:55.420 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:12:55.451 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-15T23:12:55.538 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-15T23:12:55.557 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:12:55.568 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:12:55.585 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:12:55.617 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-15T23:12:55.679 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-15T23:12:55.708 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:12:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:12:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:12:56.754 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:12:56.819 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:12:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:12:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:12:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:12:57.777 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:12:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:12:57.913 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:12:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-15T23:12:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:12:57.981 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:12:57.981 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:57.981 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T23:12:57.981 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:12:57.981 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:12:57.981 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:58.033 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:12:58.044 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:12:58.082 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-15T23:12:58.113 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:12:58.137 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:12:58.137 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:58.138 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T23:12:58.138 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:12:58.138 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:12:58.138 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:58.142 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-15T23:12:58.189 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:12:58.200 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-15T23:12:58.201 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:12:58.298 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-15T23:12:58.345 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-15T23:12:58.548 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:12:58.548 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-09-15T23:12:58.548 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-09-15T23:12:58.548 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-09-15T23:12:58.548 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-09-15T23:12:58.548 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-09-15T23:12:58.548 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-09-15T23:12:58.548 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-09-15T23:12:58.548 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:12:58.723 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:12:58.739 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:12:58.740 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:58.740 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T23:12:58.740 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:12:59.123 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-15T23:12:59.152 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:12:59.175 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:12:59.176 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:59.176 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T23:12:59.176 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:12:59.176 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:12:59.176 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:59.220 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:12:59.242 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:12:59.243 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:59.243 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T23:12:59.243 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:59.269 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-15T23:12:59.288 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:12:59.300 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:12:59.310 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:12:59.310 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:59.310 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-15T23:12:59.310 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:12:59.310 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:12:59.310 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:59.324 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:12:59.325 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:59.325 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T23:12:59.325 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:12:59.325 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:12:59.325 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:59.367 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:12:59.391 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:12:59.391 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:59.391 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T23:12:59.391 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:59.452 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:12:59.475 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:12:59.475 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:59.475 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-15T23:12:59.475 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:12:59.475 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:12:59.475 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:12:59.613 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:12:59.636 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:12:59.636 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:59.636 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T23:12:59.636 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:12:59.636 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:12:59.637 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:12:59.777 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:12:59.801 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:12:59.801 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:12:59.801 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T23:12:59.801 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:12:59.801 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:12:59.801 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:13:00.225 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:13:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:13:00.253 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:13:00.253 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T23:13:00.253 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:13:00.253 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:13:00.253 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:13:00.494 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:13:00.518 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:13:00.518 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:13:00.518 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T23:13:00.518 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:13:00.518 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:13:00.518 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:13:00.970 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:13:00.995 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:13:00.995 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:13:00.995 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T23:13:00.995 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:13:00.995 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:13:00.995 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:13:01.101 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-15T23:13:01.158 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-15T23:13:01.213 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-15T23:13:01.400 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-15T23:13:01.458 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-15T23:13:01.533 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-15T23:13:01.629 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-15T23:13:01.770 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-15T23:13:01.813 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-15T23:13:01.863 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-15T23:13:01.897 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-15T23:13:02.277 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:13:02.312 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:13:02.344 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-15T23:13:02.406 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-15T23:13:02.435 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:13:03.488 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:13:03.545 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:13:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-15T23:13:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-15T23:13:04.324 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-15T23:13:04.339 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-15T23:13:04.357 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-15T23:13:04.381 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-15T23:13:04.384 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-15T23:13:04.422 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-15T23:13:04.454 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-15T23:13:04.481 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-15T23:13:04.504 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:13:04.537 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:13:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-15T23:13:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-15T23:13:04.642 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:13:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-15T23:13:04.663 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-15T23:13:04.675 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-15T23:13:04.711 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-15T23:13:04.721 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-15T23:13:04.742 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-15T23:13:04.772 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-15T23:13:04.798 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-15T23:13:04.818 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-15T23:13:04.822 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:13:04.847 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:13:04.847 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:13:04.847 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T23:13:04.847 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:13:04.847 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:13:04.847 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:13:04.898 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:13:04.910 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:13:05.006 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-15T23:13:05.063 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-15T23:13:05.322 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:13:05.370 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:13:05.389 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:13:05.417 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:13:05.467 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:13:05.487 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:13:05.988 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-15T23:13:06.017 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:13:06.042 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:13:06.042 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:13:06.042 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T23:13:06.042 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:13:06.042 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:13:06.042 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:13:06.085 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:13:06.108 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:13:06.108 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:13:06.108 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T23:13:06.109 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:13:06.160 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:13:06.184 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:13:06.184 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:13:06.184 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-15T23:13:06.184 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:13:06.184 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:13:06.184 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:13:06.211 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-15T23:13:06.310 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-15T23:13:06.323 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-15T23:13:06.362 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-15T23:13:06.423 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-15T23:13:06.452 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-15T23:13:06.463 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-15T23:13:06.486 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:13:06.488 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-15T23:13:06.509 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:13:06.509 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:13:06.509 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T23:13:06.509 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:13:06.509 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:13:06.510 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:13:06.513 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-15T23:13:06.539 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-15T23:13:06.540 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:13:06.545 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-15T23:13:06.584 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-15T23:13:06.618 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-15T23:13:06.624 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:13:06.624 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:13:06.644 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-15T23:13:06.644 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:13:06.708 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:13:06.729 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:13:06.729 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:13:06.733 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-15T23:13:06.738 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:13:06.738 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:13:06.813 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:13:06.837 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-15T23:13:06.843 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:13:06.843 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:13:06.919 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:13:06.919 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:13:07.003 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:13:07.023 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:13:07.023 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:13:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:13:07.099 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:13:07.162 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:13:11.039 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-15T23:13:11.080 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-15T23:13:11.130 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-15T23:13:11.170 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-15T23:13:11.197 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-15T23:13:11.382 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-15T23:13:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-15T23:13:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-15T23:13:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-15T23:13:11.533 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-15T23:13:12.156 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:13:12.189 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:13:12.210 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:13:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:13:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-15T23:13:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-15T23:13:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-15T23:13:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-15T23:13:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-15T23:13:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-15T23:13:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-15T23:13:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-15T23:13:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-15T23:13:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-15T23:13:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-15T23:13:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-15T23:13:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-15T23:13:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-15T23:13:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-15T23:13:12.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-15T23:13:12.940 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-15T23:13:12.941 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-15T23:13:13.063 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-15T23:13:13.160 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-15T23:13:13.200 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-15T23:13:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-15T23:13:13.321 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-15T23:13:13.334 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:13:13.334 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-15T23:13:13.334 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-15T23:13:13.334 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-15T23:13:13.335 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-15T23:13:13.336 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-15T23:13:13.348 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-15T23:13:13.349 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-15T23:13:13.350 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-15T23:13:13.351 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-15T23:13:13.352 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-15T23:13:13.353 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-15T23:13:13.354 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-15T23:13:13.355 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-15T23:13:13.356 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-15T23:13:13.357 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-15T23:13:13.358 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-15T23:13:13.358 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-15T23:13:13.358 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-15T23:13:13.358 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-15T23:13:13.358 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-15T23:13:13.358 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-15T23:13:13.358 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-15T23:13:13.358 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-15T23:13:13.372 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-15T23:13:13.372 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:13:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:13:13.448 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:13:13.532 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:13:13.557 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-15T23:13:13.563 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:13:13.563 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:13:13.750 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:13:13.750 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:13:13.834 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:13:13.897 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:13:15.711 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-15T23:13:15.712 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: automake-1.16.2-8.el9.noarch 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: bison-3.7.4-5.el9.x86_64 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.713 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.714 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: flex-2.6.4-9.el9.x86_64 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T23:13:15.715 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.716 INFO:teuthology.orchestra.run.smithi089.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-13.16-1.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-15T23:13:15.717 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T23:13:15.718 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T23:13:15.719 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T23:13:15.720 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T23:13:15.721 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T23:13:15.722 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-15T23:13:15.723 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T23:13:15.724 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-15T23:13:15.724 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:13:15.724 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-15T23:13:16.012 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:13:16.012 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:13:16.012 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-09-15T23:13:16.012 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:16.012 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: automake-1.16.2-8.el9.noarch 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: bison-3.7.4-5.el9.x86_64 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T23:13:16.013 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:16.014 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: flex-2.6.4-9.el9.x86_64 2024-09-15T23:13:16.015 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.016 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T23:13:16.017 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-13.16-1.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-15T23:13:16.018 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-15T23:13:16.019 INFO:teuthology.orchestra.run.smithi120.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T23:13:16.020 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T23:13:16.021 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T23:13:16.022 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T23:13:16.023 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-15T23:13:16.024 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-15T23:13:16.025 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T23:13:16.025 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-15T23:13:16.025 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:13:16.025 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-09-15T23:13:16.169 DEBUG:teuthology.parallel:result is None 2024-09-15T23:13:16.466 DEBUG:teuthology.parallel:result is None 2024-09-15T23:13:19.462 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:13:19.462 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-15T23:13:19.462 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-15T23:13:19.463 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-15T23:13:19.464 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-15T23:13:19.465 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-15T23:13:19.465 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-15T23:13:19.476 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-15T23:13:19.476 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-15T23:13:19.476 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-15T23:13:19.476 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-15T23:13:19.476 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-15T23:13:19.476 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-15T23:13:19.476 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-15T23:13:19.476 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-15T23:13:19.476 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-15T23:13:19.476 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-15T23:13:19.477 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-15T23:13:19.478 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-15T23:13:19.479 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-15T23:13:19.480 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-15T23:13:19.481 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-15T23:13:19.482 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-15T23:13:19.483 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-15T23:13:19.484 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-15T23:13:19.485 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-15T23:13:19.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-15T23:13:19.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-15T23:13:19.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-15T23:13:19.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-15T23:13:19.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-15T23:13:19.486 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-15T23:13:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:13:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:13:22.088 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-09-15T23:13:22.088 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-15T23:13:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.089 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-15T23:13:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-15T23:13:22.090 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-15T23:13:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T23:13:22.090 INFO:teuthology.orchestra.run.smithi174.stdout: automake-1.16.2-8.el9.noarch 2024-09-15T23:13:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: bison-3.7.4-5.el9.x86_64 2024-09-15T23:13:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T23:13:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.091 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:22.092 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.093 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:22.094 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:13:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-15T23:13:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: flex-2.6.4-9.el9.x86_64 2024-09-15T23:13:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T23:13:22.095 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T23:13:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T23:13:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-15T23:13:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T23:13:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T23:13:22.096 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-15T23:13:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T23:13:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T23:13:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.097 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T23:13:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T23:13:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T23:13:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T23:13:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T23:13:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.098 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T23:13:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-15T23:13:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T23:13:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-15T23:13:22.099 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-15T23:13:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T23:13:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T23:13:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T23:13:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T23:13:22.100 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T23:13:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T23:13:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-15T23:13:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T23:13:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-13.16-1.el9.x86_64 2024-09-15T23:13:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-15T23:13:22.101 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-15T23:13:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-15T23:13:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T23:13:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T23:13:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T23:13:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-15T23:13:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T23:13:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T23:13:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T23:13:22.102 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T23:13:22.103 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T23:13:22.104 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T23:13:22.105 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T23:13:22.106 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.107 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:13:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T23:13:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T23:13:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T23:13:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-15T23:13:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-15T23:13:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T23:13:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-15T23:13:22.108 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:13:22.108 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-15T23:13:22.539 DEBUG:teuthology.parallel:result is None 2024-09-15T23:13:22.540 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:13:22.693 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T23:13:22.719 INFO:teuthology.orchestra.run.smithi089.stdout:19.1.1-326.g8921c8ef.el9 2024-09-15T23:13:22.720 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-15T23:13:22.720 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-15T23:13:22.722 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:13:22.867 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T23:13:22.893 INFO:teuthology.orchestra.run.smithi120.stdout:19.1.1-326.g8921c8ef.el9 2024-09-15T23:13:22.893 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-15T23:13:22.894 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-15T23:13:22.895 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:13:23.040 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T23:13:23.065 INFO:teuthology.orchestra.run.smithi174.stdout:19.1.1-326.g8921c8ef.el9 2024-09-15T23:13:23.065 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-15T23:13:23.065 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-15T23:13:23.067 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T23:13:23.067 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:13:23.067 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T23:13:23.101 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:13:23.101 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T23:13:23.133 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:13:23.133 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T23:13:23.162 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T23:13:23.163 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:13:23.163 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T23:13:23.190 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T23:13:23.257 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:13:23.258 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T23:13:23.287 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T23:13:23.356 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:13:23.356 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T23:13:23.383 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T23:13:23.450 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T23:13:23.450 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:13:23.450 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T23:13:23.478 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T23:13:23.545 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:13:23.545 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T23:13:23.574 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T23:13:23.642 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:13:23.642 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T23:13:23.669 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T23:13:23.735 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T23:13:23.735 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:13:23.735 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T23:13:23.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T23:13:23.828 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:13:23.828 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T23:13:23.856 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T23:13:23.923 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:13:23.923 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T23:13:23.949 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T23:13:24.014 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-15T23:13:24.106 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'subvols': {'create': 2}} 2024-09-15T23:13:24.107 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:13:24.107 INFO:tasks.cephadm:Cluster fsid is 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:13:24.107 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-15T23:13:24.107 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.89', 'mon.b': '172.21.15.120', 'mon.c': '172.21.15.174'} 2024-09-15T23:13:24.108 INFO:tasks.cephadm:First mon is mon.a on smithi089 2024-09-15T23:13:24.108 INFO:tasks.cephadm:First mgr is x 2024-09-15T23:13:24.108 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-15T23:13:24.108 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname $(hostname -s) 2024-09-15T23:13:24.136 DEBUG:teuthology.orchestra.run.smithi120:> sudo hostname $(hostname -s) 2024-09-15T23:13:24.165 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2024-09-15T23:13:24.191 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-15T23:13:24.192 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:13:24.334 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56081/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-326-g8921c8ef', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-326.g8921c8ef'}, 'url': 'https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-15 16:06:42.385367', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-15T23:13:24.473 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref squid, sha1 8921c8ef947807b717db041d85f0761b7914d5fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:13:24.475 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-15T23:13:24.475 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-15T23:13:24.475 DEBUG:teuthology.orchestra.run.smithi089:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-15T23:13:24.833 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 15 23:13 /home/ubuntu/cephtest/cephadm 2024-09-15T23:13:24.833 DEBUG:teuthology.orchestra.run.smithi120:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-15T23:13:25.201 INFO:teuthology.orchestra.run.smithi120.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 15 23:13 /home/ubuntu/cephtest/cephadm 2024-09-15T23:13:25.201 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-15T23:13:25.604 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 15 23:13 /home/ubuntu/cephtest/cephadm 2024-09-15T23:13:25.604 DEBUG:teuthology.orchestra.run.smithi089:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-15T23:13:25.625 DEBUG:teuthology.orchestra.run.smithi120:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-15T23:13:25.644 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-15T23:13:25.681 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc on all hosts... 2024-09-15T23:13:25.681 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-15T23:13:25.684 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-15T23:13:25.686 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-15T23:13:27.298 INFO:teuthology.orchestra.run.smithi120.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T23:13:27.315 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T23:13:27.344 INFO:teuthology.orchestra.run.smithi089.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T23:13:56.652 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-09-15T23:13:56.652 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T23:13:56.652 INFO:teuthology.orchestra.run.smithi120.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-15T23:13:56.652 INFO:teuthology.orchestra.run.smithi120.stdout: "repo_digests": [ 2024-09-15T23:13:56.652 INFO:teuthology.orchestra.run.smithi120.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-15T23:13:56.653 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-09-15T23:13:56.653 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-09-15T23:13:57.933 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-15T23:13:57.933 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T23:13:57.933 INFO:teuthology.orchestra.run.smithi174.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-15T23:13:57.933 INFO:teuthology.orchestra.run.smithi174.stdout: "repo_digests": [ 2024-09-15T23:13:57.933 INFO:teuthology.orchestra.run.smithi174.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-15T23:13:57.933 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-09-15T23:13:57.933 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-15T23:13:59.420 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:13:59.420 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T23:13:59.420 INFO:teuthology.orchestra.run.smithi089.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-15T23:13:59.420 INFO:teuthology.orchestra.run.smithi089.stdout: "repo_digests": [ 2024-09-15T23:13:59.420 INFO:teuthology.orchestra.run.smithi089.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-15T23:13:59.420 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-15T23:13:59.420 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:13:59.445 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph 2024-09-15T23:13:59.491 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph 2024-09-15T23:13:59.525 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2024-09-15T23:13:59.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /etc/ceph 2024-09-15T23:13:59.583 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /etc/ceph 2024-09-15T23:13:59.610 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2024-09-15T23:13:59.639 INFO:tasks.cephadm:Writing seed config... 2024-09-15T23:13:59.640 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-09-15T23:13:59.640 INFO:tasks.cephadm: override: [client] debug client = 20 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-09-15T23:13:59.641 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-15T23:13:59.642 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-15T23:13:59.643 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-15T23:13:59.644 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-15T23:13:59.644 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-09-15T23:13:59.644 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-15T23:13:59.644 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-09-15T23:13:59.644 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:13:59.645 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-15T23:13:59.660 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1b4e699c-73b8-11ef-bceb-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-09-15T23:13:59.660 DEBUG:teuthology.orchestra.run.smithi089:mon.a> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.a.service 2024-09-15T23:13:59.702 DEBUG:teuthology.orchestra.run.smithi089:mgr.x> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.x.service 2024-09-15T23:13:59.746 INFO:tasks.cephadm:Bootstrapping... 2024-09-15T23:13:59.746 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc -v bootstrap --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.89 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:13:59.947 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-15T23:13:59.947 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc', '-v', 'bootstrap', '--fsid', '1b4e699c-73b8-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.89', '--skip-admin-label'] 2024-09-15T23:13:59.947 INFO:teuthology.orchestra.run.smithi089.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-15T23:13:59.947 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying podman|docker is present... 2024-09-15T23:13:59.971 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 5.2.2 2024-09-15T23:13:59.971 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying lvm2 is present... 2024-09-15T23:13:59.971 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying time synchronization is in place... 2024-09-15T23:13:59.979 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-15T23:13:59.979 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-15T23:13:59.985 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-15T23:13:59.985 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-09-15T23:13:59.991 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout enabled 2024-09-15T23:13:59.997 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout active 2024-09-15T23:13:59.997 INFO:teuthology.orchestra.run.smithi089.stdout:Unit chronyd.service is enabled and running 2024-09-15T23:13:59.998 INFO:teuthology.orchestra.run.smithi089.stdout:Repeating the final host check... 2024-09-15T23:14:00.020 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout 5.2.2 2024-09-15T23:14:00.020 INFO:teuthology.orchestra.run.smithi089.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-15T23:14:00.020 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl is present 2024-09-15T23:14:00.020 INFO:teuthology.orchestra.run.smithi089.stdout:lvcreate is present 2024-09-15T23:14:00.027 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-15T23:14:00.027 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-15T23:14:00.033 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-15T23:14:00.033 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-09-15T23:14:00.039 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout enabled 2024-09-15T23:14:00.046 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout active 2024-09-15T23:14:00.046 INFO:teuthology.orchestra.run.smithi089.stdout:Unit chronyd.service is enabled and running 2024-09-15T23:14:00.046 INFO:teuthology.orchestra.run.smithi089.stdout:Host looks OK 2024-09-15T23:14:00.046 INFO:teuthology.orchestra.run.smithi089.stdout:Cluster fsid: 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:00.046 INFO:teuthology.orchestra.run.smithi089.stdout:Acquiring lock 139791859975168 on /run/cephadm/1b4e699c-73b8-11ef-bceb-c7b262605968.lock 2024-09-15T23:14:00.046 INFO:teuthology.orchestra.run.smithi089.stdout:Lock 139791859975168 acquired on /run/cephadm/1b4e699c-73b8-11ef-bceb-c7b262605968.lock 2024-09-15T23:14:00.047 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying IP 172.21.15.89 port 3300 ... 2024-09-15T23:14:00.047 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying IP 172.21.15.89 port 6789 ... 2024-09-15T23:14:00.047 INFO:teuthology.orchestra.run.smithi089.stdout:Base mon IP(s) is [172.21.15.89:3300, 172.21.15.89:6789], mon addrv is [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2024-09-15T23:14:00.051 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.89 metric 100 2024-09-15T23:14:00.051 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.89 metric 100 2024-09-15T23:14:00.053 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-15T23:14:00.053 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-15T23:14:00.053 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-15T23:14:00.055 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-15T23:14:00.055 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-15T23:14:00.055 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-15T23:14:00.055 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-15T23:14:00.056 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d501/64 scope link noprefixroute 2024-09-15T23:14:00.056 INFO:teuthology.orchestra.run.smithi089.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-15T23:14:00.056 INFO:teuthology.orchestra.run.smithi089.stdout:Mon IP `172.21.15.89` is in CIDR network `172.21.0.0/20` 2024-09-15T23:14:00.056 INFO:teuthology.orchestra.run.smithi089.stdout:Mon IP `172.21.15.89` is in CIDR network `172.21.0.0/20` 2024-09-15T23:14:00.056 INFO:teuthology.orchestra.run.smithi089.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-15T23:14:00.057 INFO:teuthology.orchestra.run.smithi089.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-15T23:14:00.057 INFO:teuthology.orchestra.run.smithi089.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T23:14:00.831 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stdout f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-15T23:14:00.831 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T23:14:00.831 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Getting image source signatures 2024-09-15T23:14:00.831 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying blob sha256:c0574da0c3174f3dc49daa274908f97d06456bd6a3cfab87e94022c1369a83a6 2024-09-15T23:14:00.832 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-15T23:14:00.832 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Copying config sha256:f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-15T23:14:00.832 INFO:teuthology.orchestra.run.smithi089.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-15T23:14:01.735 INFO:teuthology.orchestra.run.smithi089.stdout:ceph: stdout ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-15T23:14:01.735 INFO:teuthology.orchestra.run.smithi089.stdout:Ceph version: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-15T23:14:01.735 INFO:teuthology.orchestra.run.smithi089.stdout:Extracting ceph user uid/gid from container image... 2024-09-15T23:14:02.700 INFO:teuthology.orchestra.run.smithi089.stdout:stat: stdout 167 167 2024-09-15T23:14:02.700 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial keys... 2024-09-15T23:14:03.565 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQA6audmZ0l+OhAAR0IRTu6oedFhAEuM7rwAYw== 2024-09-15T23:14:04.561 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQA7audmt2h+NRAAOga5xFgxxkZbZSy8DmIN1g== 2024-09-15T23:14:05.550 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph-authtool: stdout AQA8audmUQJvMxAAKU6Ie8x4CBSbSM84NEe9sw== 2024-09-15T23:14:05.551 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial monmap... 2024-09-15T23:14:06.423 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-15T23:14:06.423 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-15T23:14:06.423 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:06.423 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-15T23:14:06.423 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool for a [v2:172.21.15.89:3300,v1:172.21.15.89:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-15T23:14:06.423 INFO:teuthology.orchestra.run.smithi089.stdout:setting min_mon_release = quincy 2024-09-15T23:14:06.423 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: set fsid to 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:06.423 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-15T23:14:06.424 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:14:06.424 INFO:teuthology.orchestra.run.smithi089.stdout:Creating mon... 2024-09-15T23:14:07.269 INFO:teuthology.orchestra.run.smithi089.stdout:create mon.a on 2024-09-15T23:14:07.473 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-15T23:14:07.619 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1b4e699c-73b8-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-1b4e699c-73b8-11ef-bceb-c7b262605968.target. 2024-09-15T23:14:07.788 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1b4e699c-73b8-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-1b4e699c-73b8-11ef-bceb-c7b262605968.target. 2024-09-15T23:14:08.001 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.a 2024-09-15T23:14:08.001 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-15T23:14:08.158 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1b4e699c-73b8-11ef-bceb-c7b262605968.target.wants/ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@.service. 2024-09-15T23:14:08.739 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T23:14:08.739 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-09-15T23:14:08.747 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T23:14:08.747 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-09-15T23:14:08.747 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-09-15T23:14:08.747 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to enable service . firewalld.service is not available 2024-09-15T23:14:08.748 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mon to start... 2024-09-15T23:14:08.748 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mon... 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout cluster: 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout id: 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout services: 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.50002s) 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout data: 2024-09-15T23:14:09.670 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout pgs: 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.135+0000 7f93e80e5640 1 Processor -- start 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.135+0000 7f93e80e5640 1 -- start start 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.135+0000 7f93e80e5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e010a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.135+0000 7f93e80e5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f93e007bed0 con 0x7f93e01080f0 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.136+0000 7f93e5e81640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e010a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.136+0000 7f93e5e81640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e010a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57888/0 (socket says 172.21.15.89:57888) 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.136+0000 7f93e5e81640 1 -- 172.21.15.89:0/3672829554 learned_addr learned my addr 172.21.15.89:0/3672829554 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.136+0000 7f93e5e81640 1 -- 172.21.15.89:0/3672829554 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e010aa20 con 0x7f93e01080f0 2024-09-15T23:14:09.671 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.138+0000 7f93e5e81640 1 --2- 172.21.15.89:0/3672829554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e010a4e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009820 tx=0x7f93dc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a816019ad9313fc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.138+0000 7f93e4e7f640 1 -- 172.21.15.89:0/3672829554 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93dc037070 con 0x7f93e01080f0 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.138+0000 7f93e4e7f640 1 -- 172.21.15.89:0/3672829554 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93dc032440 con 0x7f93e01080f0 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.138+0000 7f93e4e7f640 1 -- 172.21.15.89:0/3672829554 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93dc032740 con 0x7f93e01080f0 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.139+0000 7f93e80e5640 1 -- 172.21.15.89:0/3672829554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 msgr2=0x7f93e010a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.139+0000 7f93e80e5640 1 --2- 172.21.15.89:0/3672829554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e010a4e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009820 tx=0x7f93dc02a270 comp rx=0 tx=0).stop 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.139+0000 7f93e80e5640 1 -- 172.21.15.89:0/3672829554 shutdown_connections 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.139+0000 7f93e80e5640 1 --2- 172.21.15.89:0/3672829554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e010a4e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.139+0000 7f93e80e5640 1 -- 172.21.15.89:0/3672829554 >> 172.21.15.89:0/3672829554 conn(0x7f93e0101d10 msgr2=0x7f93e0104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.139+0000 7f93e80e5640 1 -- 172.21.15.89:0/3672829554 shutdown_connections 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.139+0000 7f93e80e5640 1 -- 172.21.15.89:0/3672829554 wait complete. 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.140+0000 7f93e80e5640 1 Processor -- start 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.140+0000 7f93e80e5640 1 -- start start 2024-09-15T23:14:09.672 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.140+0000 7f93e80e5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e0107970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.140+0000 7f93e80e5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f93e010af60 con 0x7f93e01080f0 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.140+0000 7f93e5e81640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e0107970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.140+0000 7f93e5e81640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e0107970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57898/0 (socket says 172.21.15.89:57898) 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.140+0000 7f93e5e81640 1 -- 172.21.15.89:0/3114595025 learned_addr learned my addr 172.21.15.89:0/3114595025 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.140+0000 7f93e5e81640 1 -- 172.21.15.89:0/3114595025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e0107eb0 con 0x7f93e01080f0 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.140+0000 7f93e5e81640 1 --2- 172.21.15.89:0/3114595025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e0107970 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0090b0 tx=0x7f93dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.141+0000 7f93d6ffd640 1 -- 172.21.15.89:0/3114595025 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93dc041070 con 0x7f93e01080f0 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.141+0000 7f93d6ffd640 1 -- 172.21.15.89:0/3114595025 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93dc03c7a0 con 0x7f93e01080f0 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.141+0000 7f93e80e5640 1 -- 172.21.15.89:0/3114595025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e01060a0 con 0x7f93e01080f0 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.141+0000 7f93d6ffd640 1 -- 172.21.15.89:0/3114595025 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93dc037040 con 0x7f93e01080f0 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.141+0000 7f93e80e5640 1 -- 172.21.15.89:0/3114595025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e01065e0 con 0x7f93e01080f0 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.141+0000 7f93d6ffd640 1 -- 172.21.15.89:0/3114595025 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f93dc044070 con 0x7f93e01080f0 2024-09-15T23:14:09.673 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.141+0000 7f93d6ffd640 1 -- 172.21.15.89:0/3114595025 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f93dc03d3f0 con 0x7f93e01080f0 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.142+0000 7f93e80e5640 1 -- 172.21.15.89:0/3114595025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e0108550 con 0x7f93e01080f0 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.144+0000 7f93d6ffd640 1 -- 172.21.15.89:0/3114595025 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f93dc046450 con 0x7f93e01080f0 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.204+0000 7f93e80e5640 1 -- 172.21.15.89:0/3114595025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f93e01947b0 con 0x7f93e01080f0 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.204+0000 7f93d6ffd640 1 -- 172.21.15.89:0/3114595025 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f93dc046630 con 0x7f93e01080f0 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.205+0000 7f93e80e5640 1 -- 172.21.15.89:0/3114595025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 msgr2=0x7f93e0107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.205+0000 7f93e80e5640 1 --2- 172.21.15.89:0/3114595025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e0107970 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0090b0 tx=0x7f93dc0046c0 comp rx=0 tx=0).stop 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.205+0000 7f93e80e5640 1 -- 172.21.15.89:0/3114595025 shutdown_connections 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.205+0000 7f93e80e5640 1 --2- 172.21.15.89:0/3114595025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93e01080f0 0x7f93e0107970 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.205+0000 7f93e80e5640 1 -- 172.21.15.89:0/3114595025 >> 172.21.15.89:0/3114595025 conn(0x7f93e0101d10 msgr2=0x7f93e0102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.205+0000 7f93e80e5640 1 -- 172.21.15.89:0/3114595025 shutdown_connections 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:09.205+0000 7f93e80e5640 1 -- 172.21.15.89:0/3114595025 wait complete. 2024-09-15T23:14:09.674 INFO:teuthology.orchestra.run.smithi089.stdout:mon is available 2024-09-15T23:14:09.675 INFO:teuthology.orchestra.run.smithi089.stdout:Assimilating anything we can from ceph.conf... 2024-09-15T23:14:10.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:10.683 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [global] 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout fsid = 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [osd] 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-15T23:14:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.081+0000 7f3012342640 1 Processor -- start 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.081+0000 7f3012342640 1 -- start start 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.081+0000 7f3012342640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c107b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.081+0000 7f3012342640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f300c1080f0 con 0x7f300c107720 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.081+0000 7f300bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c107b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.081+0000 7f300bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c107b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57902/0 (socket says 172.21.15.89:57902) 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.081+0000 7f300bfff640 1 -- 172.21.15.89:0/741470762 learned_addr learned my addr 172.21.15.89:0/741470762 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.082+0000 7f300bfff640 1 -- 172.21.15.89:0/741470762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300c108920 con 0x7f300c107720 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.082+0000 7f300bfff640 1 --2- 172.21.15.89:0/741470762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c107b20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc009820 tx=0x7f2ffc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f1e5bf98801b29 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:10.685 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.082+0000 7f300affd640 1 -- 172.21.15.89:0/741470762 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ffc037070 con 0x7f300c107720 2024-09-15T23:14:10.686 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.082+0000 7f300affd640 1 -- 172.21.15.89:0/741470762 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ffc032440 con 0x7f300c107720 2024-09-15T23:14:10.688 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.082+0000 7f300affd640 1 -- 172.21.15.89:0/741470762 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ffc032740 con 0x7f300c107720 2024-09-15T23:14:10.688 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.083+0000 7f3012342640 1 -- 172.21.15.89:0/741470762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 msgr2=0x7f300c107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.083+0000 7f3012342640 1 --2- 172.21.15.89:0/741470762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c107b20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc009820 tx=0x7f2ffc02a270 comp rx=0 tx=0).stop 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.083+0000 7f3012342640 1 -- 172.21.15.89:0/741470762 shutdown_connections 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.083+0000 7f3012342640 1 --2- 172.21.15.89:0/741470762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c107b20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.083+0000 7f3012342640 1 -- 172.21.15.89:0/741470762 >> 172.21.15.89:0/741470762 conn(0x7f300c102ed0 msgr2=0x7f300c1052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.083+0000 7f3012342640 1 -- 172.21.15.89:0/741470762 shutdown_connections 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.083+0000 7f3012342640 1 -- 172.21.15.89:0/741470762 wait complete. 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.084+0000 7f3012342640 1 Processor -- start 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.084+0000 7f3012342640 1 -- start start 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.084+0000 7f3012342640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c19ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.084+0000 7f3012342640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f300c108e50 con 0x7f300c107720 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.084+0000 7f300bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c19ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.084+0000 7f300bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c19ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57914/0 (socket says 172.21.15.89:57914) 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.084+0000 7f300bfff640 1 -- 172.21.15.89:0/1615747021 learned_addr learned my addr 172.21.15.89:0/1615747021 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:10.689 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.084+0000 7f300bfff640 1 -- 172.21.15.89:0/1615747021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300c19b3b0 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.084+0000 7f300bfff640 1 --2- 172.21.15.89:0/1615747021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c19ae70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0090b0 tx=0x7f2ffc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.085+0000 7f30097fa640 1 -- 172.21.15.89:0/1615747021 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ffc041070 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.085+0000 7f30097fa640 1 -- 172.21.15.89:0/1615747021 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ffc03c7a0 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.085+0000 7f3012342640 1 -- 172.21.15.89:0/1615747021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f300c19b640 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.085+0000 7f30097fa640 1 -- 172.21.15.89:0/1615747021 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ffc037040 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.085+0000 7f3012342640 1 -- 172.21.15.89:0/1615747021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f300c19e330 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.085+0000 7f30097fa640 1 -- 172.21.15.89:0/1615747021 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2ffc044070 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.085+0000 7f30097fa640 1 -- 172.21.15.89:0/1615747021 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2ffc03d730 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.086+0000 7f3012342640 1 -- 172.21.15.89:0/1615747021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f300c1094a0 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.088+0000 7f30097fa640 1 -- 172.21.15.89:0/1615747021 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f2ffc03cbe0 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.144+0000 7f3012342640 1 -- 172.21.15.89:0/1615747021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f300c194330 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.147+0000 7f30097fa640 1 -- 172.21.15.89:0/1615747021 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f2ffc03cdc0 con 0x7f300c107720 2024-09-15T23:14:10.690 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.148+0000 7f30097fa640 1 -- 172.21.15.89:0/1615747021 <== mon.0 v2:172.21.15.89:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2ffc056070 con 0x7f300c107720 2024-09-15T23:14:10.691 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.148+0000 7f3012342640 1 -- 172.21.15.89:0/1615747021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 msgr2=0x7f300c19ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:10.691 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.148+0000 7f3012342640 1 --2- 172.21.15.89:0/1615747021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c19ae70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0090b0 tx=0x7f2ffc0046c0 comp rx=0 tx=0).stop 2024-09-15T23:14:10.691 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.149+0000 7f3012342640 1 -- 172.21.15.89:0/1615747021 shutdown_connections 2024-09-15T23:14:10.691 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.149+0000 7f3012342640 1 --2- 172.21.15.89:0/1615747021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f300c107720 0x7f300c19ae70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:10.691 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.149+0000 7f3012342640 1 -- 172.21.15.89:0/1615747021 >> 172.21.15.89:0/1615747021 conn(0x7f300c102ed0 msgr2=0x7f300c18f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:10.691 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.149+0000 7f3012342640 1 -- 172.21.15.89:0/1615747021 shutdown_connections 2024-09-15T23:14:10.691 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:10.149+0000 7f3012342640 1 -- 172.21.15.89:0/1615747021 wait complete. 2024-09-15T23:14:10.691 INFO:teuthology.orchestra.run.smithi089.stdout:Generating new minimal ceph.conf... 2024-09-15T23:14:11.655 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.094+0000 7f77bb3d6640 1 Processor -- start 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.094+0000 7f77bb3d6640 1 -- start start 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.095+0000 7f77bb3d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b4107af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.095+0000 7f77bb3d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41080c0 con 0x7f77b41076f0 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.095+0000 7f77b9172640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b4107af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.095+0000 7f77b9172640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b4107af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57930/0 (socket says 172.21.15.89:57930) 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.095+0000 7f77b9172640 1 -- 172.21.15.89:0/1912508834 learned_addr learned my addr 172.21.15.89:0/1912508834 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.095+0000 7f77b9172640 1 -- 172.21.15.89:0/1912508834 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b41088f0 con 0x7f77b41076f0 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.096+0000 7f77b9172640 1 --2- 172.21.15.89:0/1912508834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b4107af0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f77b0009820 tx=0x7f77b002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1595e353f10d56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.096+0000 7f77abfff640 1 -- 172.21.15.89:0/1912508834 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77b0037070 con 0x7f77b41076f0 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.096+0000 7f77abfff640 1 -- 172.21.15.89:0/1912508834 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f77b0032900 con 0x7f77b41076f0 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.096+0000 7f77abfff640 1 -- 172.21.15.89:0/1912508834 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77b0030850 con 0x7f77b41076f0 2024-09-15T23:14:11.656 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.096+0000 7f77bb3d6640 1 -- 172.21.15.89:0/1912508834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 msgr2=0x7f77b4107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.096+0000 7f77bb3d6640 1 --2- 172.21.15.89:0/1912508834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b4107af0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f77b0009820 tx=0x7f77b002a270 comp rx=0 tx=0).stop 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.097+0000 7f77bb3d6640 1 -- 172.21.15.89:0/1912508834 shutdown_connections 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.097+0000 7f77bb3d6640 1 --2- 172.21.15.89:0/1912508834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b4107af0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.097+0000 7f77bb3d6640 1 -- 172.21.15.89:0/1912508834 >> 172.21.15.89:0/1912508834 conn(0x7f77b4102e80 msgr2=0x7f77b41052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.097+0000 7f77bb3d6640 1 -- 172.21.15.89:0/1912508834 shutdown_connections 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.097+0000 7f77bb3d6640 1 -- 172.21.15.89:0/1912508834 wait complete. 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.097+0000 7f77bb3d6640 1 Processor -- start 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.097+0000 7f77bb3d6640 1 -- start start 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.097+0000 7f77bb3d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b419b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.097+0000 7f77bb3d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f77b410bca0 con 0x7f77b41076f0 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77b9172640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b419b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77b9172640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b419b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57934/0 (socket says 172.21.15.89:57934) 2024-09-15T23:14:11.657 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77b9172640 1 -- 172.21.15.89:0/2586014149 learned_addr learned my addr 172.21.15.89:0/2586014149 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77b9172640 1 -- 172.21.15.89:0/2586014149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b419bc50 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77b9172640 1 --2- 172.21.15.89:0/2586014149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b419b710 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f77b00097f0 tx=0x7f77b0030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77aa7fc640 1 -- 172.21.15.89:0/2586014149 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77b0037040 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77aa7fc640 1 -- 172.21.15.89:0/2586014149 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f77b0040070 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77bb3d6640 1 -- 172.21.15.89:0/2586014149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b419bee0 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77aa7fc640 1 -- 172.21.15.89:0/2586014149 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77b003a390 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77bb3d6640 1 -- 172.21.15.89:0/2586014149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b419ebd0 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77aa7fc640 1 -- 172.21.15.89:0/2586014149 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f77b003a5e0 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.098+0000 7f77aa7fc640 1 -- 172.21.15.89:0/2586014149 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f77b0045950 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.099+0000 7f77bb3d6640 1 -- 172.21.15.89:0/2586014149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77b410be20 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.101+0000 7f77aa7fc640 1 -- 172.21.15.89:0/2586014149 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f77b0045460 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.158+0000 7f77bb3d6640 1 -- 172.21.15.89:0/2586014149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f77b419eda0 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.158+0000 7f77aa7fc640 1 -- 172.21.15.89:0/2586014149 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f77b003b430 con 0x7f77b41076f0 2024-09-15T23:14:11.658 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.159+0000 7f77bb3d6640 1 -- 172.21.15.89:0/2586014149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 msgr2=0x7f77b419b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:11.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.159+0000 7f77bb3d6640 1 --2- 172.21.15.89:0/2586014149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b419b710 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f77b00097f0 tx=0x7f77b0030d80 comp rx=0 tx=0).stop 2024-09-15T23:14:11.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.159+0000 7f77bb3d6640 1 -- 172.21.15.89:0/2586014149 shutdown_connections 2024-09-15T23:14:11.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.159+0000 7f77bb3d6640 1 --2- 172.21.15.89:0/2586014149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77b41076f0 0x7f77b419b710 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:11.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.159+0000 7f77bb3d6640 1 -- 172.21.15.89:0/2586014149 >> 172.21.15.89:0/2586014149 conn(0x7f77b4102e80 msgr2=0x7f77b4103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:11.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.159+0000 7f77bb3d6640 1 -- 172.21.15.89:0/2586014149 shutdown_connections 2024-09-15T23:14:11.659 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:11.159+0000 7f77bb3d6640 1 -- 172.21.15.89:0/2586014149 wait complete. 2024-09-15T23:14:11.659 INFO:teuthology.orchestra.run.smithi089.stdout:Restarting the monitor... 2024-09-15T23:14:11.712 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:11 smithi089 systemd[1]: Stopping Ceph mon.a for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:14:12.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:11 smithi089 ceph-mon[30560]: from='client.? 172.21.15.89:0/2586014149' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:14:12.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:11 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a[30549]: 2024-09-15T23:14:11.917+0000 7fc02db55640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:14:12.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:11 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a[30549]: 2024-09-15T23:14:11.917+0000 7fc02db55640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-15T23:14:12.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:12 smithi089 podman[30808]: 2024-09-15 23:14:12.143304969 +0000 UTC m=+0.327739165 container died 0bc77a708fb65a8fb28915fb2debc39c454a4a4b2bc054bfc7405ea87edb260b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-15T23:14:12.678 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:12 smithi089 podman[30808]: 2024-09-15 23:14:12.502161959 +0000 UTC m=+0.686596158 container cleanup 0bc77a708fb65a8fb28915fb2debc39c454a4a4b2bc054bfc7405ea87edb260b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-15T23:14:12.679 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:12 smithi089 bash[30808]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a 2024-09-15T23:14:13.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:12 smithi089 podman[30820]: 2024-09-15 23:14:12.678165576 +0000 UTC m=+0.532207459 container remove 0bc77a708fb65a8fb28915fb2debc39c454a4a4b2bc054bfc7405ea87edb260b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-15T23:14:13.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:12 smithi089 systemd[1]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-15T23:14:13.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:12 smithi089 systemd[1]: Stopped Ceph mon.a for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:14:13.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:12 smithi089 systemd[1]: Starting Ceph mon.a for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:14:13.375 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 podman[30914]: 2024-09-15 23:14:13.0258478 +0000 UTC m=+0.015867464 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:14:13.375 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 podman[30914]: 2024-09-15 23:14:13.137238859 +0000 UTC m=+0.127258510 container create 587b46d282067b8382327780cd0a638e50de500b8e42fe1c77433d5f6c940b41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-15T23:14:13.375 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 podman[30914]: 2024-09-15 23:14:13.347903554 +0000 UTC m=+0.337923204 container init 587b46d282067b8382327780cd0a638e50de500b8e42fe1c77433d5f6c940b41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-15T23:14:13.375 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 podman[30914]: 2024-09-15 23:14:13.351062673 +0000 UTC m=+0.341082324 container start 587b46d282067b8382327780cd0a638e50de500b8e42fe1c77433d5f6c940b41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-15T23:14:13.375 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: set uid:gid to 167:167 (ceph:ceph) 2024-09-15T23:14:13.375 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable), process ceph-mon, pid 2 2024-09-15T23:14:13.375 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: pidfile_write: ignore empty --pid-file 2024-09-15T23:14:13.375 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: load: jerasure load: lrc 2024-09-15T23:14:13.427 INFO:teuthology.orchestra.run.smithi089.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-15T23:14:13.649 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: RocksDB version: 7.9.2 2024-09-15T23:14:13.649 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Git sha 0 2024-09-15T23:14:13.649 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Compile date 2024-09-15 15:04:35 2024-09-15T23:14:13.649 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: DB SUMMARY 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: DB Session ID: RJMZLDBIQD4WCF9UDFVH 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: CURRENT file: CURRENT 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: IDENTITY file: IDENTITY 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105948 ; 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.error_if_exists: 0 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.create_if_missing: 0 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.paranoid_checks: 1 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-15T23:14:13.650 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.env: 0x55ccdf829940 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.fs: PosixFileSystem 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.info_log: 0x55cce0b3f6e0 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_file_opening_threads: 16 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.statistics: (nil) 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.use_fsync: 0 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_log_file_size: 0 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.keep_log_file_num: 1000 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.recycle_log_file_num: 0 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.allow_fallocate: 1 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.allow_mmap_reads: 0 2024-09-15T23:14:13.651 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.allow_mmap_writes: 0 2024-09-15T23:14:13.652 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.use_direct_reads: 0 2024-09-15T23:14:13.652 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-15T23:14:13.652 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.create_missing_column_families: 0 2024-09-15T23:14:13.652 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.db_log_dir: 2024-09-15T23:14:13.652 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.wal_dir: 2024-09-15T23:14:13.652 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-15T23:14:13.652 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-15T23:14:13.652 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-15T23:14:13.652 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-15T23:14:13.653 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.advise_random_on_open: 1 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.db_write_buffer_size: 0 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.write_buffer_manager: 0x55cce0b4edc0 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.rate_limiter: (nil) 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.wal_recovery_mode: 2 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.enable_thread_tracking: 0 2024-09-15T23:14:13.654 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.enable_pipelined_write: 0 2024-09-15T23:14:13.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.unordered_write: 0 2024-09-15T23:14:13.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-15T23:14:13.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-15T23:14:13.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-15T23:14:13.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-15T23:14:13.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.row_cache: None 2024-09-15T23:14:13.655 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.wal_filter: None 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.allow_ingest_behind: 0 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.two_write_queues: 0 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.manual_wal_flush: 0 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.wal_compression: 0 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.atomic_flush: 0 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.log_readahead_size: 0 2024-09-15T23:14:13.656 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.best_efforts_recovery: 0 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.allow_data_in_errors: 0 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.db_host_id: __hostname__ 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_background_jobs: 2 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_background_compactions: -1 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_subcompactions: 1 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-15T23:14:13.657 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_total_wal_size: 0 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_open_files: -1 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bytes_per_sync: 0 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_readahead_size: 0 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_background_flushes: -1 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Compression algorithms supported: 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: kZSTD supported: 0 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: kXpressCompression supported: 0 2024-09-15T23:14:13.658 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: kBZip2Compression supported: 0 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: kLZ4Compression supported: 1 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: kZlibCompression supported: 1 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: kLZ4HCCompression supported: 1 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: kSnappyCompression supported: 1 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.merge_operator: 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_filter: None 2024-09-15T23:14:13.659 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_filter_factory: None 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.sst_partitioner_factory: None 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cce0b3f320) 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout: cache_index_and_filter_blocks: 1 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout: pin_top_level_index_and_filter: 1 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout: index_type: 0 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout: data_block_index_type: 0 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout: index_shortening: 1 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout: checksum: 4 2024-09-15T23:14:13.660 INFO:journalctl@ceph.mon.a.smithi089.stdout: no_block_cache: 0 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache: 0x55cce0b331f0 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache_name: BinnedLRUCache 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache_options: 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: capacity : 536870912 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: num_shard_bits : 4 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: strict_capacity_limit : 0 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: high_pri_pool_ratio: 0.000 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_cache_compressed: (nil) 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: persistent_cache: (nil) 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_size: 4096 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_size_deviation: 10 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_restart_interval: 16 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: index_block_restart_interval: 1 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: metadata_block_size: 4096 2024-09-15T23:14:13.661 INFO:journalctl@ceph.mon.a.smithi089.stdout: partition_filters: 0 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: use_delta_encoding: 1 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: filter_policy: bloomfilter 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: whole_key_filtering: 1 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: verify_compression: 0 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: read_amp_bytes_per_bit: 0 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: format_version: 5 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: enable_index_compression: 1 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: block_align: 0 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: max_auto_readahead_size: 262144 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: prepopulate_block_cache: 0 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: initial_auto_readahead_size: 8192 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout: num_file_reads_for_auto_readahead: 2 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.write_buffer_size: 33554432 2024-09-15T23:14:13.662 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_write_buffer_number: 2 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compression: NoCompression 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bottommost_compression: Disabled 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.prefix_extractor: nullptr 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.num_levels: 7 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-15T23:14:13.663 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compression_opts.level: 32767 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compression_opts.strategy: 0 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compression_opts.enabled: false 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-15T23:14:13.664 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.target_file_size_base: 67108864 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-15T23:14:13.665 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.arena_block_size: 1048576 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.disable_auto_compactions: 0 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-15T23:14:13.666 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.inplace_update_support: 0 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.bloom_locality: 0 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.max_successive_merges: 0 2024-09-15T23:14:13.667 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.paranoid_file_checks: 0 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.force_consistency_checks: 1 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.report_bg_io_stats: 0 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.ttl: 2592000 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.enable_blob_files: false 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.min_blob_size: 0 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.blob_file_size: 268435456 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-15T23:14:13.668 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.blob_file_starting_level: 0 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 77a9dfe4-daed-45b6-a969-e134a9c8ea1a 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726442053376752, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726442053377789, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100963, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 330, "table_properties": {"data_size": 98848, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14453, "raw_average_key_size": 50, "raw_value_size": 91020, "raw_average_value_size": 314, "num_data_blocks": 12, "num_entries": 289, "num_filter_entries": 289, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726442053, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "77a9dfe4-daed-45b6-a969-e134a9c8ea1a", "db_session_id": "RJMZLDBIQD4WCF9UDFVH", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726442053377893, "job": 1, "event": "recovery_finished"} 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-15T23:14:13.669 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cce0b60e00 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: DB pointer 0x55cce0c5c000 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** DB Stats ** 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** Compaction Stats [default] ** 2024-09-15T23:14:13.670 INFO:journalctl@ceph.mon.a.smithi089.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: L0 2/0 100.46 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: Sum 2/0 100.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** Compaction Stats [default] ** 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 145.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-15T23:14:13.671 INFO:journalctl@ceph.mon.a.smithi089.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: Cumulative compaction: 0.00 GB write, 17.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: Interval compaction: 0.00 GB write, 17.42 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: Block cache BinnedLRUCache@0x55cce0b331f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: starting mon.a rank 0 at public addrs [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] at bind addrs [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:13.672 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: mon.a@-1(???) e1 preinit fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: mon.a@-1(???).mds e1 new map 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: mon.a@-1(???).mds e1 print_map 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout: e1 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout: btime 2024-09-15T23:14:08:706210+0000 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout: legacy client fscid: -1 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout: 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout: No filesystems configured 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T23:14:13.673 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 bash[30914]: 587b46d282067b8382327780cd0a638e50de500b8e42fe1c77433d5f6c940b41 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 systemd[1]: Started Ceph mon.a for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: monmap epoch 1 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: last_changed 2024-09-15T23:14:05.881012+0000 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: created 2024-09-15T23:14:05.881012+0000 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: min_mon_release 19 (squid) 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: election_strategy: 1 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: fsmap 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: osdmap e1: 0 total, 0 up, 0 in 2024-09-15T23:14:13.674 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:13 smithi089 ceph-mon[30933]: mgrmap e1: no daemons active 2024-09-15T23:14:14.402 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.839+0000 7f1a52a13640 1 Processor -- start 2024-09-15T23:14:14.402 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.840+0000 7f1a52a13640 1 -- start start 2024-09-15T23:14:14.403 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.840+0000 7f1a52a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:14.403 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.840+0000 7f1a52a13640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c106f50 con 0x7f1a4c106580 2024-09-15T23:14:14.403 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.840+0000 7f1a4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:14.403 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.840+0000 7f1a4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57950/0 (socket says 172.21.15.89:57950) 2024-09-15T23:14:14.408 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.840+0000 7f1a4bfff640 1 -- 172.21.15.89:0/2440622041 learned_addr learned my addr 172.21.15.89:0/2440622041 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:14.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.840+0000 7f1a4bfff640 1 -- 172.21.15.89:0/2440622041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4c107780 con 0x7f1a4c106580 2024-09-15T23:14:14.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.842+0000 7f1a4bfff640 1 --2- 172.21.15.89:0/2440622041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c106980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c009820 tx=0x7f1a3c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3878700ff42951d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:14.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.842+0000 7f1a4b7fe640 1 -- 172.21.15.89:0/2440622041 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a3c037070 con 0x7f1a4c106580 2024-09-15T23:14:14.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.843+0000 7f1a4b7fe640 1 -- 172.21.15.89:0/2440622041 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1a3c032900 con 0x7f1a4c106580 2024-09-15T23:14:14.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.843+0000 7f1a4b7fe640 1 -- 172.21.15.89:0/2440622041 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a3c030850 con 0x7f1a4c106580 2024-09-15T23:14:14.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.843+0000 7f1a52a13640 1 -- 172.21.15.89:0/2440622041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 msgr2=0x7f1a4c106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:14.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.843+0000 7f1a52a13640 1 --2- 172.21.15.89:0/2440622041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c106980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c009820 tx=0x7f1a3c02a270 comp rx=0 tx=0).stop 2024-09-15T23:14:14.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.843+0000 7f1a52a13640 1 -- 172.21.15.89:0/2440622041 shutdown_connections 2024-09-15T23:14:14.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.843+0000 7f1a52a13640 1 --2- 172.21.15.89:0/2440622041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c106980 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:14.409 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.843+0000 7f1a52a13640 1 -- 172.21.15.89:0/2440622041 >> 172.21.15.89:0/2440622041 conn(0x7f1a4c101d10 msgr2=0x7f1a4c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.843+0000 7f1a52a13640 1 -- 172.21.15.89:0/2440622041 shutdown_connections 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.843+0000 7f1a52a13640 1 -- 172.21.15.89:0/2440622041 wait complete. 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.844+0000 7f1a52a13640 1 Processor -- start 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.844+0000 7f1a52a13640 1 -- start start 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.844+0000 7f1a52a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c19b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.844+0000 7f1a52a13640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c10ab30 con 0x7f1a4c106580 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.844+0000 7f1a4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c19b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.844+0000 7f1a4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c19b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57964/0 (socket says 172.21.15.89:57964) 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.844+0000 7f1a4bfff640 1 -- 172.21.15.89:0/766366785 learned_addr learned my addr 172.21.15.89:0/766366785 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.844+0000 7f1a4bfff640 1 -- 172.21.15.89:0/766366785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4c19bc00 con 0x7f1a4c106580 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.845+0000 7f1a4bfff640 1 --2- 172.21.15.89:0/766366785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c19b6c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c009940 tx=0x7f1a3c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.845+0000 7f1a49ffb640 1 -- 172.21.15.89:0/766366785 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a3c042070 con 0x7f1a4c106580 2024-09-15T23:14:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.845+0000 7f1a49ffb640 1 -- 172.21.15.89:0/766366785 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1a3c0308c0 con 0x7f1a4c106580 2024-09-15T23:14:14.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.845+0000 7f1a52a13640 1 -- 172.21.15.89:0/766366785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a4c19be90 con 0x7f1a4c106580 2024-09-15T23:14:14.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.845+0000 7f1a49ffb640 1 -- 172.21.15.89:0/766366785 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a3c037040 con 0x7f1a4c106580 2024-09-15T23:14:14.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.845+0000 7f1a52a13640 1 -- 172.21.15.89:0/766366785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a4c19eb80 con 0x7f1a4c106580 2024-09-15T23:14:14.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.845+0000 7f1a49ffb640 1 -- 172.21.15.89:0/766366785 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1a3c045070 con 0x7f1a4c106580 2024-09-15T23:14:14.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.845+0000 7f1a49ffb640 1 -- 172.21.15.89:0/766366785 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1a3c047b70 con 0x7f1a4c106580 2024-09-15T23:14:14.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.846+0000 7f1a52a13640 1 -- 172.21.15.89:0/766366785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a4c10acb0 con 0x7f1a4c106580 2024-09-15T23:14:14.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.848+0000 7f1a49ffb640 1 -- 172.21.15.89:0/766366785 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f1a3c0327b0 con 0x7f1a4c106580 2024-09-15T23:14:14.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.904+0000 7f1a52a13640 1 -- 172.21.15.89:0/766366785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f1a4c02d9b0 con 0x7f1a4c106580 2024-09-15T23:14:14.411 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.909+0000 7f1a49ffb640 1 -- 172.21.15.89:0/766366785 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f1a3c03c760 con 0x7f1a4c106580 2024-09-15T23:14:14.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.909+0000 7f1a52a13640 1 -- 172.21.15.89:0/766366785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 msgr2=0x7f1a4c19b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:14.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.909+0000 7f1a52a13640 1 --2- 172.21.15.89:0/766366785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c19b6c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c009940 tx=0x7f1a3c032ce0 comp rx=0 tx=0).stop 2024-09-15T23:14:14.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.910+0000 7f1a52a13640 1 -- 172.21.15.89:0/766366785 shutdown_connections 2024-09-15T23:14:14.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.910+0000 7f1a52a13640 1 --2- 172.21.15.89:0/766366785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a4c106580 0x7f1a4c19b6c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:14.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.910+0000 7f1a52a13640 1 -- 172.21.15.89:0/766366785 >> 172.21.15.89:0/766366785 conn(0x7f1a4c101d10 msgr2=0x7f1a4c102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:14.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.910+0000 7f1a52a13640 1 -- 172.21.15.89:0/766366785 shutdown_connections 2024-09-15T23:14:14.412 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:13.910+0000 7f1a52a13640 1 -- 172.21.15.89:0/766366785 wait complete. 2024-09-15T23:14:14.412 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-15T23:14:14.421 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:14:14.421 INFO:teuthology.orchestra.run.smithi089.stdout:Creating mgr... 2024-09-15T23:14:14.421 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-15T23:14:14.422 INFO:teuthology.orchestra.run.smithi089.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-15T23:14:14.676 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.x 2024-09-15T23:14:14.677 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.x.service: Unit ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.x.service not loaded. 2024-09-15T23:14:14.831 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1b4e699c-73b8-11ef-bceb-c7b262605968.target.wants/ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@.service. 2024-09-15T23:14:15.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:14 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/766366785' entity='client.admin' 2024-09-15T23:14:15.376 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:15 smithi089 podman[31146]: 2024-09-15 23:14:15.277820673 +0000 UTC m=+0.329162272 container start 9504be6ed98639a60dec28ddaf742447e9faff729765a7346e7898e5effce887 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-15T23:14:15.376 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:15 smithi089 bash[31146]: 9504be6ed98639a60dec28ddaf742447e9faff729765a7346e7898e5effce887 2024-09-15T23:14:15.390 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T23:14:15.390 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-09-15T23:14:15.397 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T23:14:15.397 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-09-15T23:14:15.397 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-09-15T23:14:15.397 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to enable service . firewalld.service is not available 2024-09-15T23:14:15.405 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T23:14:15.406 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-09-15T23:14:15.413 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T23:14:15.413 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-09-15T23:14:15.413 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-09-15T23:14:15.413 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-15T23:14:15.413 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr to start... 2024-09-15T23:14:15.414 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr... 2024-09-15T23:14:15.627 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:15 smithi089 systemd[1]: Started Ceph mgr.x for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:14:15.627 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:15 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:15.396+0000 7fb19d857040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T23:14:15.628 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:15 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:15.463+0000 7fb19d857040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T23:14:15.628 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:15 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:15.615+0000 7fb19d857040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T23:14:15.910 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:15 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:15.684+0000 7fb19d857040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T23:14:15.910 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:15 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:15.839+0000 7fb19d857040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:14:16.199 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:15 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/579217367' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-15T23:14:16.199 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:15 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:15.997+0000 7fb19d857040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T23:14:16.444 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:16.444 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:14:16.444 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:14:16.444 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-09-15T23:14:16.444 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-15T23:14:16.445 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-15T23:14:16.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:16.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-15T23:14:16.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-15T23:14:16.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:16.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-15T23:14:16.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:16.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-15T23:14:16.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-15T23:14:16.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-15T23:14:16.446 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:16.447 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-15T23:14:16.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:16.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "btime": "2024-09-15T23:14:08:706210+0000", 2024-09-15T23:14:16.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-15T23:14:16.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-15T23:14:16.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:16.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-15T23:14:16.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-15T23:14:16.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-15T23:14:16.448 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-09-15T23:14:08.706909+0000", 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-15T23:14:16.449 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.789+0000 7f410fdc0640 1 Processor -- start 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.789+0000 7f410fdc0640 1 -- start start 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.790+0000 7f410fdc0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f4108106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.790+0000 7f410fdc0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4108106f50 con 0x7f4108106580 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.790+0000 7f410db5c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f4108106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.790+0000 7f410db5c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f4108106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46570/0 (socket says 172.21.15.89:46570) 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.790+0000 7f410db5c640 1 -- 172.21.15.89:0/926569000 learned_addr learned my addr 172.21.15.89:0/926569000 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.790+0000 7f410db5c640 1 -- 172.21.15.89:0/926569000 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4108107780 con 0x7f4108106580 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.791+0000 7f410db5c640 1 --2- 172.21.15.89:0/926569000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f4108106980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4104009820 tx=0x7f410402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9222607dd6a896 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.791+0000 7f410cb5a640 1 -- 172.21.15.89:0/926569000 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4104037070 con 0x7f4108106580 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.791+0000 7f410cb5a640 1 -- 172.21.15.89:0/926569000 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4104032900 con 0x7f4108106580 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.791+0000 7f410cb5a640 1 -- 172.21.15.89:0/926569000 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4104030850 con 0x7f4108106580 2024-09-15T23:14:16.450 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.792+0000 7f410fdc0640 1 -- 172.21.15.89:0/926569000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 msgr2=0x7f4108106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:16.451 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.792+0000 7f410fdc0640 1 --2- 172.21.15.89:0/926569000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f4108106980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4104009820 tx=0x7f410402a270 comp rx=0 tx=0).stop 2024-09-15T23:14:16.451 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.792+0000 7f410fdc0640 1 -- 172.21.15.89:0/926569000 shutdown_connections 2024-09-15T23:14:16.451 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.792+0000 7f410fdc0640 1 --2- 172.21.15.89:0/926569000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f4108106980 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:16.451 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.792+0000 7f410fdc0640 1 -- 172.21.15.89:0/926569000 >> 172.21.15.89:0/926569000 conn(0x7f4108101d10 msgr2=0x7f4108104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:16.451 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.792+0000 7f410fdc0640 1 -- 172.21.15.89:0/926569000 shutdown_connections 2024-09-15T23:14:16.451 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.792+0000 7f410fdc0640 1 -- 172.21.15.89:0/926569000 wait complete. 2024-09-15T23:14:16.451 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.792+0000 7f410fdc0640 1 Processor -- start 2024-09-15T23:14:16.451 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f410fdc0640 1 -- start start 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f410fdc0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f410819d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f410fdc0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f410810ab30 con 0x7f4108106580 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f410db5c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f410819d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f410db5c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f410819d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46574/0 (socket says 172.21.15.89:46574) 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f410db5c640 1 -- 172.21.15.89:0/579217367 learned_addr learned my addr 172.21.15.89:0/579217367 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f410db5c640 1 -- 172.21.15.89:0/579217367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f410819ddf0 con 0x7f4108106580 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f410db5c640 1 --2- 172.21.15.89:0/579217367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f410819d8b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f41040097f0 tx=0x7f4104030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f40feffd640 1 -- 172.21.15.89:0/579217367 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4104037040 con 0x7f4108106580 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f40feffd640 1 -- 172.21.15.89:0/579217367 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4104040070 con 0x7f4108106580 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f40feffd640 1 -- 172.21.15.89:0/579217367 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f410403a390 con 0x7f4108106580 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.793+0000 7f410fdc0640 1 -- 172.21.15.89:0/579217367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f410819e080 con 0x7f4108106580 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.794+0000 7f410fdc0640 1 -- 172.21.15.89:0/579217367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41081a0d70 con 0x7f4108106580 2024-09-15T23:14:16.452 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.794+0000 7f40feffd640 1 -- 172.21.15.89:0/579217367 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f410403a630 con 0x7f4108106580 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.794+0000 7f40feffd640 1 -- 172.21.15.89:0/579217367 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f41040459a0 con 0x7f4108106580 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.795+0000 7f410fdc0640 1 -- 172.21.15.89:0/579217367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4108106a00 con 0x7f4108106580 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.797+0000 7f40feffd640 1 -- 172.21.15.89:0/579217367 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f4104045460 con 0x7f4108106580 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.858+0000 7f410fdc0640 1 -- 172.21.15.89:0/579217367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4108197a50 con 0x7f4108106580 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.858+0000 7f40feffd640 1 -- 172.21.15.89:0/579217367 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f410403b430 con 0x7f4108106580 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.859+0000 7f410fdc0640 1 -- 172.21.15.89:0/579217367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 msgr2=0x7f410819d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.859+0000 7f410fdc0640 1 --2- 172.21.15.89:0/579217367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f410819d8b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f41040097f0 tx=0x7f4104030d80 comp rx=0 tx=0).stop 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.859+0000 7f410fdc0640 1 -- 172.21.15.89:0/579217367 shutdown_connections 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.859+0000 7f410fdc0640 1 --2- 172.21.15.89:0/579217367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4108106580 0x7f410819d8b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.859+0000 7f410fdc0640 1 -- 172.21.15.89:0/579217367 >> 172.21.15.89:0/579217367 conn(0x7f4108101d10 msgr2=0x7f4108102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.859+0000 7f410fdc0640 1 -- 172.21.15.89:0/579217367 shutdown_connections 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:15.859+0000 7f410fdc0640 1 -- 172.21.15.89:0/579217367 wait complete. 2024-09-15T23:14:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (1/15)... 2024-09-15T23:14:16.560 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:16 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:16.317+0000 7fb19d857040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T23:14:16.560 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:16 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:16.378+0000 7fb19d857040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T23:14:17.060 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:16 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:16.563+0000 7fb19d857040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T23:14:17.810 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:17 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:17.438+0000 7fb19d857040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T23:14:17.810 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:17 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:17.504+0000 7fb19d857040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T23:14:18.472 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:18.212+0000 7fb19d857040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T23:14:18.472 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:18.325+0000 7fb19d857040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T23:14:18.472 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:18.468+0000 7fb19d857040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:14:18.754 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:18.531+0000 7fb19d857040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T23:14:18.755 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:18.682+0000 7fb19d857040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T23:14:18.755 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:18.753+0000 7fb19d857040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T23:14:19.051 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:18 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3679475881' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-15T23:14:19.051 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:18.813+0000 7fb19d857040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T23:14:19.051 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:18.984+0000 7fb19d857040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T23:14:19.051 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:19 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:19.049+0000 7fb19d857040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T23:14:19.310 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:19 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:19.120+0000 7fb19d857040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T23:14:19.310 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:19 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:19.202+0000 7fb19d857040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T23:14:19.310 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:19 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:19.261+0000 7fb19d857040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T23:14:19.593 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:19 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:19.320+0000 7fb19d857040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T23:14:19.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:19.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:14:19.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:14:19.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-09-15T23:14:19.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-15T23:14:19.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-15T23:14:19.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-15T23:14:19.695 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-15T23:14:19.696 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-15T23:14:19.697 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "btime": "2024-09-15T23:14:08:706210+0000", 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:19.698 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-09-15T23:14:08.706909+0000", 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.878+0000 7efcd3ba7640 1 Processor -- start 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.878+0000 7efcd3ba7640 1 -- start start 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.878+0000 7efcd3ba7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc10a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.878+0000 7efcd3ba7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efccc07be90 con 0x7efccc108130 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.878+0000 7efcd1943640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc10a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.878+0000 7efcd1943640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc10a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46586/0 (socket says 172.21.15.89:46586) 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.878+0000 7efcd1943640 1 -- 172.21.15.89:0/3695782377 learned_addr learned my addr 172.21.15.89:0/3695782377 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.879+0000 7efcd1943640 1 -- 172.21.15.89:0/3695782377 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efccc10aa60 con 0x7efccc108130 2024-09-15T23:14:19.700 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.879+0000 7efcd1943640 1 --2- 172.21.15.89:0/3695782377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc10a520 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009820 tx=0x7efcc802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20722673caacdc77 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:19.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.880+0000 7efcd0941640 1 -- 172.21.15.89:0/3695782377 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc8037070 con 0x7efccc108130 2024-09-15T23:14:19.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.880+0000 7efcd0941640 1 -- 172.21.15.89:0/3695782377 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efcc8032900 con 0x7efccc108130 2024-09-15T23:14:19.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.880+0000 7efcd0941640 1 -- 172.21.15.89:0/3695782377 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc8030850 con 0x7efccc108130 2024-09-15T23:14:19.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.880+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3695782377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 msgr2=0x7efccc10a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:19.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.880+0000 7efcd3ba7640 1 --2- 172.21.15.89:0/3695782377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc10a520 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009820 tx=0x7efcc802a270 comp rx=0 tx=0).stop 2024-09-15T23:14:19.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.880+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3695782377 shutdown_connections 2024-09-15T23:14:19.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.880+0000 7efcd3ba7640 1 --2- 172.21.15.89:0/3695782377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc10a520 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.880+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3695782377 >> 172.21.15.89:0/3695782377 conn(0x7efccc101d10 msgr2=0x7efccc104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:19.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.880+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3695782377 shutdown_connections 2024-09-15T23:14:19.701 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.880+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3695782377 wait complete. 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.881+0000 7efcd3ba7640 1 Processor -- start 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.881+0000 7efcd3ba7640 1 -- start start 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.881+0000 7efcd3ba7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc19b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.881+0000 7efcd3ba7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efccc10baa0 con 0x7efccc108130 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.881+0000 7efcd1943640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc19b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.881+0000 7efcd1943640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc19b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46602/0 (socket says 172.21.15.89:46602) 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.881+0000 7efcd1943640 1 -- 172.21.15.89:0/3679475881 learned_addr learned my addr 172.21.15.89:0/3679475881 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.881+0000 7efcd1943640 1 -- 172.21.15.89:0/3679475881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efccc19bbc0 con 0x7efccc108130 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.882+0000 7efcd1943640 1 --2- 172.21.15.89:0/3679475881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc19b680 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efcc8032ce0 tx=0x7efcc8032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.882+0000 7efcc2ffd640 1 -- 172.21.15.89:0/3679475881 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc8042070 con 0x7efccc108130 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.882+0000 7efcc2ffd640 1 -- 172.21.15.89:0/3679475881 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efcc80308c0 con 0x7efccc108130 2024-09-15T23:14:19.702 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.882+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3679475881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efccc19be50 con 0x7efccc108130 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.882+0000 7efcc2ffd640 1 -- 172.21.15.89:0/3679475881 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efcc8037040 con 0x7efccc108130 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.882+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3679475881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efccc19eb40 con 0x7efccc108130 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.882+0000 7efcc2ffd640 1 -- 172.21.15.89:0/3679475881 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efcc8045070 con 0x7efccc108130 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.882+0000 7efcc2ffd640 1 -- 172.21.15.89:0/3679475881 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efcc8047bd0 con 0x7efccc108130 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.883+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3679475881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efccc108590 con 0x7efccc108130 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.885+0000 7efcc2ffd640 1 -- 172.21.15.89:0/3679475881 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7efcc803c830 con 0x7efccc108130 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.947+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3679475881 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7efccc19ed10 con 0x7efccc108130 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.948+0000 7efcc2ffd640 1 -- 172.21.15.89:0/3679475881 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7efcc803ca10 con 0x7efccc108130 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.948+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3679475881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 msgr2=0x7efccc19b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.948+0000 7efcd3ba7640 1 --2- 172.21.15.89:0/3679475881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc19b680 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efcc8032ce0 tx=0x7efcc8032d10 comp rx=0 tx=0).stop 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.949+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3679475881 shutdown_connections 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.949+0000 7efcd3ba7640 1 --2- 172.21.15.89:0/3679475881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efccc108130 0x7efccc19b680 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.949+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3679475881 >> 172.21.15.89:0/3679475881 conn(0x7efccc101d10 msgr2=0x7efccc102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:19.703 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.949+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3679475881 shutdown_connections 2024-09-15T23:14:19.704 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:18.949+0000 7efcd3ba7640 1 -- 172.21.15.89:0/3679475881 wait complete. 2024-09-15T23:14:19.704 INFO:teuthology.orchestra.run.smithi089.stdout:mgr not available, waiting (2/15)... 2024-09-15T23:14:20.560 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:20.117+0000 7fb19d857040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: Activating manager daemon x 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: mgrmap e2: x(active, starting, since 0.00295363s) 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: from='mgr.14100 172.21.15.89:0/1138700679' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: from='mgr.14100 172.21.15.89:0/1138700679' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: from='mgr.14100 172.21.15.89:0/1138700679' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: from='mgr.14100 172.21.15.89:0/1138700679' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: from='mgr.14100 172.21.15.89:0/1138700679' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: Manager daemon x is now available 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: from='mgr.14100 172.21.15.89:0/1138700679' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: from='mgr.14100 172.21.15.89:0/1138700679' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T23:14:20.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: from='mgr.14100 172.21.15.89:0/1138700679' entity='mgr.x' 2024-09-15T23:14:20.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: from='mgr.14100 172.21.15.89:0/1138700679' entity='mgr.x' 2024-09-15T23:14:20.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:20 smithi089 ceph-mon[30933]: from='mgr.14100 172.21.15.89:0/1138700679' entity='mgr.x' 2024-09-15T23:14:22.344 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:22 smithi089 ceph-mon[30933]: mgrmap e3: x(active, since 1.00737s) 2024-09-15T23:14:22.908 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:22.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:14:22.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:14:22.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "health": { 2024-09-15T23:14:22.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-15T23:14:22.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-15T23:14:22.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-15T23:14:22.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:22.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-15T23:14:22.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-15T23:14:22.909 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 0 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "a" 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-15T23:14:22.910 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-15T23:14:22.911 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-15T23:14:22.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-15T23:14:22.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-15T23:14:22.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-15T23:14:22.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:22.919 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "btime": "2024-09-15T23:14:08:706210+0000", 2024-09-15T23:14:22.920 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "restful" 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:22.921 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "modified": "2024-09-15T23:14:08.706909+0000", 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.098+0000 7fc7af91f640 1 Processor -- start 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.099+0000 7fc7af91f640 1 -- start start 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.099+0000 7fc7af91f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.099+0000 7fc7af91f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a8106f50 con 0x7fc7a8106580 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.099+0000 7fc7ad6bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.099+0000 7fc7ad6bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46678/0 (socket says 172.21.15.89:46678) 2024-09-15T23:14:22.922 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.099+0000 7fc7ad6bb640 1 -- 172.21.15.89:0/3044715063 learned_addr learned my addr 172.21.15.89:0/3044715063 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.099+0000 7fc7ad6bb640 1 -- 172.21.15.89:0/3044715063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a8107780 con 0x7fc7a8106580 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.100+0000 7fc7ad6bb640 1 --2- 172.21.15.89:0/3044715063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8106980 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400bac0 tx=0x7fc7a402c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9b3be8128ac090 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.100+0000 7fc79bfff640 1 -- 172.21.15.89:0/3044715063 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7a40326d0 con 0x7fc7a8106580 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.100+0000 7fc79bfff640 1 -- 172.21.15.89:0/3044715063 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc7a4038070 con 0x7fc7a8106580 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.100+0000 7fc79bfff640 1 -- 172.21.15.89:0/3044715063 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7a4033d70 con 0x7fc7a8106580 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.101+0000 7fc7af91f640 1 -- 172.21.15.89:0/3044715063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 msgr2=0x7fc7a8106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.101+0000 7fc7af91f640 1 --2- 172.21.15.89:0/3044715063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8106980 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400bac0 tx=0x7fc7a402c920 comp rx=0 tx=0).stop 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.101+0000 7fc7af91f640 1 -- 172.21.15.89:0/3044715063 shutdown_connections 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.101+0000 7fc7af91f640 1 --2- 172.21.15.89:0/3044715063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8106980 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.101+0000 7fc7af91f640 1 -- 172.21.15.89:0/3044715063 >> 172.21.15.89:0/3044715063 conn(0x7fc7a8101d10 msgr2=0x7fc7a8104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.101+0000 7fc7af91f640 1 -- 172.21.15.89:0/3044715063 shutdown_connections 2024-09-15T23:14:22.923 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.101+0000 7fc7af91f640 1 -- 172.21.15.89:0/3044715063 wait complete. 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.102+0000 7fc7af91f640 1 Processor -- start 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.102+0000 7fc7af91f640 1 -- start start 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.102+0000 7fc7af91f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.102+0000 7fc7af91f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a807b170 con 0x7fc7a8106580 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.102+0000 7fc7ad6bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.102+0000 7fc7ad6bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46686/0 (socket says 172.21.15.89:46686) 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.102+0000 7fc7ad6bb640 1 -- 172.21.15.89:0/4087455790 learned_addr learned my addr 172.21.15.89:0/4087455790 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.102+0000 7fc7ad6bb640 1 -- 172.21.15.89:0/4087455790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a8199960 con 0x7fc7a8106580 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.103+0000 7fc7ad6bb640 1 --2- 172.21.15.89:0/4087455790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8199420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400b2c0 tx=0x7fc7a40327d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.103+0000 7fc79a7fc640 1 -- 172.21.15.89:0/4087455790 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7a4042070 con 0x7fc7a8106580 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.103+0000 7fc79a7fc640 1 -- 172.21.15.89:0/4087455790 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc7a4032b70 con 0x7fc7a8106580 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.103+0000 7fc7af91f640 1 -- 172.21.15.89:0/4087455790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a8199bf0 con 0x7fc7a8106580 2024-09-15T23:14:22.924 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.103+0000 7fc79a7fc640 1 -- 172.21.15.89:0/4087455790 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc7a403c580 con 0x7fc7a8106580 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.103+0000 7fc7af91f640 1 -- 172.21.15.89:0/4087455790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a819c8e0 con 0x7fc7a8106580 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.104+0000 7fc79a7fc640 1 -- 172.21.15.89:0/4087455790 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 3) ==== 46344+0+0 (secure 0 0 0) 0x7fc7a403c840 con 0x7fc7a8106580 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.104+0000 7fc7af91f640 1 -- 172.21.15.89:0/4087455790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7a8106980 con 0x7fc7a8106580 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.104+0000 7fc79a7fc640 1 --2- 172.21.15.89:0/4087455790 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fc794039050 0x7fc79403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.104+0000 7fc79a7fc640 1 -- 172.21.15.89:0/4087455790 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc7a4030050 con 0x7fc7a8106580 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.104+0000 7fc7aceba640 1 --2- 172.21.15.89:0/4087455790 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fc794039050 0x7fc79403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.104+0000 7fc7aceba640 1 --2- 172.21.15.89:0/4087455790 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fc794039050 0x7fc79403b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0098c0 tx=0x7fc79c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.109+0000 7fc79a7fc640 1 -- 172.21.15.89:0/4087455790 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc7a4009250 con 0x7fc7a8106580 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.123+0000 7fc79a7fc640 1 -- 172.21.15.89:0/4087455790 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7fc7a404f3f0 con 0x7fc7a8106580 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.342+0000 7fc7af91f640 1 -- 172.21.15.89:0/4087455790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc7a8192890 con 0x7fc7a8106580 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.343+0000 7fc79a7fc640 1 -- 172.21.15.89:0/4087455790 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1290 (secure 0 0 0) 0x7fc7a403cb30 con 0x7fc7a8106580 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.345+0000 7fc7af91f640 1 -- 172.21.15.89:0/4087455790 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fc794039050 msgr2=0x7fc79403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:22.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.345+0000 7fc7af91f640 1 --2- 172.21.15.89:0/4087455790 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fc794039050 0x7fc79403b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0098c0 tx=0x7fc79c006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.345+0000 7fc7af91f640 1 -- 172.21.15.89:0/4087455790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 msgr2=0x7fc7a8199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:22.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.345+0000 7fc7af91f640 1 --2- 172.21.15.89:0/4087455790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8199420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc7a400b2c0 tx=0x7fc7a40327d0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.345+0000 7fc7af91f640 1 -- 172.21.15.89:0/4087455790 shutdown_connections 2024-09-15T23:14:22.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.345+0000 7fc7af91f640 1 --2- 172.21.15.89:0/4087455790 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fc794039050 0x7fc79403b510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.345+0000 7fc7af91f640 1 --2- 172.21.15.89:0/4087455790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7a8106580 0x7fc7a8199420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.345+0000 7fc7af91f640 1 -- 172.21.15.89:0/4087455790 >> 172.21.15.89:0/4087455790 conn(0x7fc7a8101d10 msgr2=0x7fc7a818fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:22.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.345+0000 7fc7af91f640 1 -- 172.21.15.89:0/4087455790 shutdown_connections 2024-09-15T23:14:22.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:22.345+0000 7fc7af91f640 1 -- 172.21.15.89:0/4087455790 wait complete. 2024-09-15T23:14:22.926 INFO:teuthology.orchestra.run.smithi089.stdout:mgr is available 2024-09-15T23:14:23.397 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:23 smithi089 ceph-mon[30933]: mgrmap e4: x(active, since 2s) 2024-09-15T23:14:23.397 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:23 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/4087455790' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-15T23:14:24.026 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:24.026 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [global] 2024-09-15T23:14:24.026 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout fsid = 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.89:3300,v1:172.21.15.89:6789] 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout [osd] 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-15T23:14:24.027 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.299+0000 7fed3e829640 1 Processor -- start 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.299+0000 7fed3e829640 1 -- start start 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.299+0000 7fed3e829640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed38107ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.299+0000 7fed3e829640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fed381080b0 con 0x7fed381076e0 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.299+0000 7fed37fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed38107ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.299+0000 7fed37fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed38107ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46700/0 (socket says 172.21.15.89:46700) 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.299+0000 7fed37fff640 1 -- 172.21.15.89:0/755218842 learned_addr learned my addr 172.21.15.89:0/755218842 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.299+0000 7fed37fff640 1 -- 172.21.15.89:0/755218842 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed381088e0 con 0x7fed381076e0 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.300+0000 7fed37fff640 1 --2- 172.21.15.89:0/755218842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed38107ae0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fed28009820 tx=0x7fed2802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b080fa9d05898a9f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.300+0000 7fed36ffd640 1 -- 172.21.15.89:0/755218842 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed28037070 con 0x7fed381076e0 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.300+0000 7fed36ffd640 1 -- 172.21.15.89:0/755218842 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fed28032900 con 0x7fed381076e0 2024-09-15T23:14:24.028 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.300+0000 7fed36ffd640 1 -- 172.21.15.89:0/755218842 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed28030850 con 0x7fed381076e0 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.301+0000 7fed3e829640 1 -- 172.21.15.89:0/755218842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 msgr2=0x7fed38107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.301+0000 7fed3e829640 1 --2- 172.21.15.89:0/755218842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed38107ae0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fed28009820 tx=0x7fed2802a270 comp rx=0 tx=0).stop 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.301+0000 7fed3e829640 1 -- 172.21.15.89:0/755218842 shutdown_connections 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.301+0000 7fed3e829640 1 --2- 172.21.15.89:0/755218842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed38107ae0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.301+0000 7fed3e829640 1 -- 172.21.15.89:0/755218842 >> 172.21.15.89:0/755218842 conn(0x7fed38102e90 msgr2=0x7fed381052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.301+0000 7fed3e829640 1 -- 172.21.15.89:0/755218842 shutdown_connections 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.301+0000 7fed3e829640 1 -- 172.21.15.89:0/755218842 wait complete. 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.301+0000 7fed3e829640 1 Processor -- start 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.301+0000 7fed3e829640 1 -- start start 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed3e829640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed3819b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed3e829640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fed3810bc90 con 0x7fed381076e0 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed37fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed3819b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed37fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed3819b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46712/0 (socket says 172.21.15.89:46712) 2024-09-15T23:14:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed37fff640 1 -- 172.21.15.89:0/1064412734 learned_addr learned my addr 172.21.15.89:0/1064412734 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed37fff640 1 -- 172.21.15.89:0/1064412734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3819bc70 con 0x7fed381076e0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed37fff640 1 --2- 172.21.15.89:0/1064412734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed3819b730 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fed28009940 tx=0x7fed280326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed357fa640 1 -- 172.21.15.89:0/1064412734 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed28042070 con 0x7fed381076e0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed3e829640 1 -- 172.21.15.89:0/1064412734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3819bf00 con 0x7fed381076e0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed357fa640 1 -- 172.21.15.89:0/1064412734 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fed28032be0 con 0x7fed381076e0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.302+0000 7fed357fa640 1 -- 172.21.15.89:0/1064412734 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed28037040 con 0x7fed381076e0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.303+0000 7fed3e829640 1 -- 172.21.15.89:0/1064412734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3819ebf0 con 0x7fed381076e0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.303+0000 7fed357fa640 1 -- 172.21.15.89:0/1064412734 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7fed28045070 con 0x7fed381076e0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.303+0000 7fed357fa640 1 --2- 172.21.15.89:0/1064412734 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fed240391b0 0x7fed2403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.304+0000 7fed357fa640 1 -- 172.21.15.89:0/1064412734 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fed2806c080 con 0x7fed381076e0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.303+0000 7fed3e829640 1 -- 172.21.15.89:0/1064412734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3810be10 con 0x7fed381076e0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.304+0000 7fed377fe640 1 --2- 172.21.15.89:0/1064412734 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fed240391b0 0x7fed2403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.304+0000 7fed377fe640 1 --2- 172.21.15.89:0/1064412734 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fed240391b0 0x7fed2403b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fed200098c0 tx=0x7fed20006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.308+0000 7fed357fa640 1 -- 172.21.15.89:0/1064412734 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fed28045360 con 0x7fed381076e0 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.477+0000 7fed3e829640 1 -- 172.21.15.89:0/1064412734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fed38107b60 con 0x7fed381076e0 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.478+0000 7fed357fa640 1 -- 172.21.15.89:0/1064412734 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+429 (secure 0 0 0) 0x7fed28032d90 con 0x7fed381076e0 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.480+0000 7fed3e829640 1 -- 172.21.15.89:0/1064412734 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fed240391b0 msgr2=0x7fed2403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.480+0000 7fed3e829640 1 --2- 172.21.15.89:0/1064412734 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fed240391b0 0x7fed2403b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fed200098c0 tx=0x7fed20006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.480+0000 7fed3e829640 1 -- 172.21.15.89:0/1064412734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 msgr2=0x7fed3819b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.480+0000 7fed3e829640 1 --2- 172.21.15.89:0/1064412734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed3819b730 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fed28009940 tx=0x7fed280326e0 comp rx=0 tx=0).stop 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.480+0000 7fed3e829640 1 -- 172.21.15.89:0/1064412734 shutdown_connections 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.480+0000 7fed3e829640 1 --2- 172.21.15.89:0/1064412734 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fed240391b0 0x7fed2403b670 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.480+0000 7fed3e829640 1 --2- 172.21.15.89:0/1064412734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed381076e0 0x7fed3819b730 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.480+0000 7fed3e829640 1 -- 172.21.15.89:0/1064412734 >> 172.21.15.89:0/1064412734 conn(0x7fed38102e90 msgr2=0x7fed38103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.480+0000 7fed3e829640 1 -- 172.21.15.89:0/1064412734 shutdown_connections 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:23.480+0000 7fed3e829640 1 -- 172.21.15.89:0/1064412734 wait complete. 2024-09-15T23:14:24.031 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling cephadm module... 2024-09-15T23:14:24.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:24 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1064412734' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-15T23:14:27.769 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:27 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3468385660' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-15T23:14:27.770 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:27 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: ignoring --setuser ceph since I am not root 2024-09-15T23:14:27.770 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:27 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: ignoring --setgroup ceph since I am not root 2024-09-15T23:14:27.770 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:27 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:27.709+0000 7f63405c2040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T23:14:27.770 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:27 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:27.767+0000 7f63405c2040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T23:14:28.060 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:27 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:27.918+0000 7f63405c2040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T23:14:28.060 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:27 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:27.989+0000 7f63405c2040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T23:14:28.560 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:28 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:28.138+0000 7f63405c2040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:14:28.560 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:28 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:28.279+0000 7f63405c2040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T23:14:28.836 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:28 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3468385660' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-15T23:14:28.837 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:28 smithi089 ceph-mon[30933]: mgrmap e5: x(active, since 7s) 2024-09-15T23:14:28.837 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:28 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:28.589+0000 7f63405c2040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T23:14:28.837 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:28 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:28.650+0000 7f63405c2040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T23:14:28.837 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:28 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:28.833+0000 7f63405c2040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T23:14:30.060 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:29 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:29.745+0000 7f63405c2040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T23:14:30.061 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:29 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:29.816+0000 7f63405c2040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T23:14:30.668 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:30.551+0000 7f63405c2040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T23:14:30.668 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:30.666+0000 7f63405c2040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T23:14:31.020 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:30.808+0000 7f63405c2040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:14:31.020 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:30.871+0000 7f63405c2040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T23:14:31.021 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:31.018+0000 7f63405c2040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T23:14:31.305 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:31.086+0000 7f63405c2040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T23:14:31.305 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:31.147+0000 7f63405c2040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T23:14:31.305 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:31.303+0000 7f63405c2040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T23:14:31.560 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:31.368+0000 7f63405c2040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T23:14:31.662 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:31.433+0000 7f63405c2040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T23:14:31.662 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:31.515+0000 7f63405c2040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T23:14:32.060 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:31.576+0000 7f63405c2040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T23:14:32.060 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:31.634+0000 7f63405c2040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T23:14:32.810 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:32.422+0000 7f63405c2040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T23:14:32.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: Active manager daemon x restarted 2024-09-15T23:14:32.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: Activating manager daemon x 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: osdmap e2: 0 total, 0 up, 0 in 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: mgrmap e6: x(active, starting, since 0.00370558s) 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: Manager daemon x is now available 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:14:32.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T23:14:32.812 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:32 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T23:14:33.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:33 smithi089 ceph-mon[30933]: Found migration_current of "None". Setting to last migration. 2024-09-15T23:14:33.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:33 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:33.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:33 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:33.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:33 smithi089 ceph-mon[30933]: mgrmap e7: x(active, since 1.00531s) 2024-09-15T23:14:33.967 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.294+0000 7fb8ab378640 1 Processor -- start 2024-09-15T23:14:33.972 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.294+0000 7fb8ab378640 1 -- start start 2024-09-15T23:14:33.972 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.294+0000 7fb8ab378640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a407c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:33.972 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.294+0000 7fb8ab378640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a407c910 con 0x7fb8a407dee0 2024-09-15T23:14:33.972 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.294+0000 7fb8a9114640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a407c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:33.973 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.295+0000 7fb8a9114640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a407c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51466/0 (socket says 172.21.15.89:51466) 2024-09-15T23:14:33.973 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.295+0000 7fb8a9114640 1 -- 172.21.15.89:0/393318232 learned_addr learned my addr 172.21.15.89:0/393318232 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:33.976 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.295+0000 7fb8a9114640 1 -- 172.21.15.89:0/393318232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a407ca90 con 0x7fb8a407dee0 2024-09-15T23:14:33.976 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.295+0000 7fb8a9114640 1 --2- 172.21.15.89:0/393318232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a407c340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009820 tx=0x7fb8a002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e58e14ba0fd1fe1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:33.976 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.296+0000 7fb89bfff640 1 -- 172.21.15.89:0/393318232 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8a0037070 con 0x7fb8a407dee0 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.296+0000 7fb89bfff640 1 -- 172.21.15.89:0/393318232 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb8a0032900 con 0x7fb8a407dee0 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.296+0000 7fb89bfff640 1 -- 172.21.15.89:0/393318232 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8a0030850 con 0x7fb8a407dee0 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.296+0000 7fb8ab378640 1 -- 172.21.15.89:0/393318232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 msgr2=0x7fb8a407c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.296+0000 7fb8ab378640 1 --2- 172.21.15.89:0/393318232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a407c340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009820 tx=0x7fb8a002a270 comp rx=0 tx=0).stop 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.296+0000 7fb8ab378640 1 -- 172.21.15.89:0/393318232 shutdown_connections 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.296+0000 7fb8ab378640 1 --2- 172.21.15.89:0/393318232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a407c340 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.296+0000 7fb8ab378640 1 -- 172.21.15.89:0/393318232 >> 172.21.15.89:0/393318232 conn(0x7fb8a4102ed0 msgr2=0x7fb8a41052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.296+0000 7fb8ab378640 1 -- 172.21.15.89:0/393318232 shutdown_connections 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.296+0000 7fb8ab378640 1 -- 172.21.15.89:0/393318232 wait complete. 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.297+0000 7fb8ab378640 1 Processor -- start 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.297+0000 7fb8ab378640 1 -- start start 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.297+0000 7fb8ab378640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a419d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:33.977 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.297+0000 7fb8ab378640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a410dd10 con 0x7fb8a407dee0 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.297+0000 7fb8a9114640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a419d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.297+0000 7fb8a9114640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a419d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51472/0 (socket says 172.21.15.89:51472) 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.297+0000 7fb8a9114640 1 -- 172.21.15.89:0/3468385660 learned_addr learned my addr 172.21.15.89:0/3468385660 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.298+0000 7fb8a9114640 1 -- 172.21.15.89:0/3468385660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a419de40 con 0x7fb8a407dee0 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.298+0000 7fb8a9114640 1 --2- 172.21.15.89:0/3468385660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a419d900 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00097f0 tx=0x7fb8a0030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.298+0000 7fb89a7fc640 1 -- 172.21.15.89:0/3468385660 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8a0042070 con 0x7fb8a407dee0 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.298+0000 7fb89a7fc640 1 -- 172.21.15.89:0/3468385660 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb8a0032ca0 con 0x7fb8a407dee0 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.298+0000 7fb8ab378640 1 -- 172.21.15.89:0/3468385660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a419e0d0 con 0x7fb8a407dee0 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.298+0000 7fb89a7fc640 1 -- 172.21.15.89:0/3468385660 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8a0037040 con 0x7fb8a407dee0 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.298+0000 7fb8ab378640 1 -- 172.21.15.89:0/3468385660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a41a0dc0 con 0x7fb8a407dee0 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.299+0000 7fb89a7fc640 1 -- 172.21.15.89:0/3468385660 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7fb8a0045070 con 0x7fb8a407dee0 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.299+0000 7fb89a7fc640 1 --2- 172.21.15.89:0/3468385660 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fb8900391b0 0x7fb89003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.299+0000 7fb8ab378640 1 -- 172.21.15.89:0/3468385660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8a407c3c0 con 0x7fb8a407dee0 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.299+0000 7fb89a7fc640 1 -- 172.21.15.89:0/3468385660 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb8a006c210 con 0x7fb8a407dee0 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.299+0000 7fb8a8913640 1 --2- 172.21.15.89:0/3468385660 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fb8900391b0 0x7fb89003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.300+0000 7fb8a8913640 1 --2- 172.21.15.89:0/3468385660 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fb8900391b0 0x7fb89003b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb8940098c0 tx=0x7fb894006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.304+0000 7fb89a7fc640 1 -- 172.21.15.89:0/3468385660 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb8a0045520 con 0x7fb8a407dee0 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.517+0000 7fb8ab378640 1 -- 172.21.15.89:0/3468385660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fb8a41a1440 con 0x7fb8a407dee0 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.568+0000 7fb89a7fc640 1 -- 172.21.15.89:0/3468385660 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fb8a0050090 con 0x7fb8a407dee0 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.570+0000 7fb89a7fc640 1 -- 172.21.15.89:0/3468385660 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7fb8a003c6a0 con 0x7fb8a407dee0 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.571+0000 7fb8ab378640 1 -- 172.21.15.89:0/3468385660 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fb8900391b0 msgr2=0x7fb89003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.571+0000 7fb8ab378640 1 --2- 172.21.15.89:0/3468385660 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fb8900391b0 0x7fb89003b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb8940098c0 tx=0x7fb894006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.571+0000 7fb8ab378640 1 -- 172.21.15.89:0/3468385660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 msgr2=0x7fb8a419d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.571+0000 7fb8ab378640 1 --2- 172.21.15.89:0/3468385660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a419d900 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00097f0 tx=0x7fb8a0030ec0 comp rx=0 tx=0).stop 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.571+0000 7fb8ab378640 1 -- 172.21.15.89:0/3468385660 shutdown_connections 2024-09-15T23:14:33.979 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.571+0000 7fb8ab378640 1 --2- 172.21.15.89:0/3468385660 >> [v2:172.21.15.89:6800/4057509998,v1:172.21.15.89:6801/4057509998] conn(0x7fb8900391b0 0x7fb89003b670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:33.980 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.571+0000 7fb8ab378640 1 --2- 172.21.15.89:0/3468385660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8a407dee0 0x7fb8a419d900 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:33.980 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.571+0000 7fb8ab378640 1 -- 172.21.15.89:0/3468385660 >> 172.21.15.89:0/3468385660 conn(0x7fb8a4102ed0 msgr2=0x7fb8a4103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:33.980 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.571+0000 7fb8ab378640 1 -- 172.21.15.89:0/3468385660 shutdown_connections 2024-09-15T23:14:33.980 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:27.571+0000 7fb8ab378640 1 -- 172.21.15.89:0/3468385660 wait complete. 2024-09-15T23:14:34.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:14:34.811 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:34.471+0000 7f62a791c640 -1 log_channel(cephadm) log [ERR] : [15/Sep/2024:23:14:34] ENGINE Error in HTTPServer.serve 2024-09-15T23:14:34.811 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: Traceback (most recent call last): 2024-09-15T23:14:34.811 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T23:14:34.811 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: self._connections.run(self.expiration_interval) 2024-09-15T23:14:34.811 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T23:14:34.811 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: self._run(expiration_interval) 2024-09-15T23:14:34.811 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T23:14:34.811 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T23:14:34.811 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T23:14:34.811 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T23:14:34.812 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T23:14:34.812 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: s = self.context.wrap_socket( 2024-09-15T23:14:34.812 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T23:14:34.812 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: return self.sslsocket_class._create( 2024-09-15T23:14:34.812 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T23:14:34.812 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: self.do_handshake() 2024-09-15T23:14:34.812 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T23:14:34.812 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: self._sslobj.do_handshake() 2024-09-15T23:14:34.812 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T23:14:34.812 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:35.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:35 smithi089 ceph-mon[30933]: [15/Sep/2024:23:14:34] ENGINE Bus STARTING 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:35 smithi089 ceph-mon[30933]: [15/Sep/2024:23:14:34] ENGINE Serving on http://172.21.15.89:8765 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:35 smithi089 ceph-mon[30933]: [15/Sep/2024:23:14:34] ENGINE Serving on https://172.21.15.89:7150 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:35 smithi089 ceph-mon[30933]: [15/Sep/2024:23:14:34] ENGINE Bus STARTED 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:35 smithi089 ceph-mon[30933]: [15/Sep/2024:23:14:34] ENGINE Error in HTTPServer.serve 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout: Traceback (most recent call last): 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout: self._connections.run(self.expiration_interval) 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout: self._run(expiration_interval) 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T23:14:35.811 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout: s = self.context.wrap_socket( 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout: return self.sslsocket_class._create( 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout: self.do_handshake() 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout: self._sslobj.do_handshake() 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T23:14:35.812 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:35 smithi089 ceph-mon[30933]: mgrmap e8: x(active, since 2s) 2024-09-15T23:14:37.759 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:37 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2240051248' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-15T23:14:38.084 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:14:38.084 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 8, 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.232+0000 7f4ce1675640 1 Processor -- start 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.232+0000 7f4ce1675640 1 -- start start 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.232+0000 7f4ce1675640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc1069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.232+0000 7f4ce1675640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc106fa0 con 0x7f4cdc1065d0 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.233+0000 7f4cdaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc1069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.233+0000 7f4cdaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc1069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47862/0 (socket says 172.21.15.89:47862) 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.233+0000 7f4cdaffd640 1 -- 172.21.15.89:0/698291500 learned_addr learned my addr 172.21.15.89:0/698291500 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.233+0000 7f4cdaffd640 1 -- 172.21.15.89:0/698291500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cdc1077d0 con 0x7f4cdc1065d0 2024-09-15T23:14:38.085 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.234+0000 7f4cdaffd640 1 --2- 172.21.15.89:0/698291500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc1069d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009820 tx=0x7f4ccc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8968be8f51cc9ad3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.234+0000 7f4cd9ffb640 1 -- 172.21.15.89:0/698291500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ccc037070 con 0x7f4cdc1065d0 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.234+0000 7f4cd9ffb640 1 -- 172.21.15.89:0/698291500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4ccc032900 con 0x7f4cdc1065d0 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.234+0000 7f4cd9ffb640 1 -- 172.21.15.89:0/698291500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ccc030850 con 0x7f4cdc1065d0 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.235+0000 7f4ce1675640 1 -- 172.21.15.89:0/698291500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 msgr2=0x7f4cdc1069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.235+0000 7f4ce1675640 1 --2- 172.21.15.89:0/698291500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc1069d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009820 tx=0x7f4ccc02a270 comp rx=0 tx=0).stop 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.235+0000 7f4ce1675640 1 -- 172.21.15.89:0/698291500 shutdown_connections 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.235+0000 7f4ce1675640 1 --2- 172.21.15.89:0/698291500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc1069d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.235+0000 7f4ce1675640 1 -- 172.21.15.89:0/698291500 >> 172.21.15.89:0/698291500 conn(0x7f4cdc101da0 msgr2=0x7f4cdc1041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.235+0000 7f4ce1675640 1 -- 172.21.15.89:0/698291500 shutdown_connections 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.235+0000 7f4ce1675640 1 -- 172.21.15.89:0/698291500 wait complete. 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.235+0000 7f4ce1675640 1 Processor -- start 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.236+0000 7f4ce1675640 1 -- start start 2024-09-15T23:14:38.086 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.236+0000 7f4ce1675640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc191800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.236+0000 7f4ce1675640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc10ab80 con 0x7f4cdc1065d0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.236+0000 7f4cdaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc191800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.236+0000 7f4cdaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc191800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47870/0 (socket says 172.21.15.89:47870) 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.236+0000 7f4cdaffd640 1 -- 172.21.15.89:0/2240051248 learned_addr learned my addr 172.21.15.89:0/2240051248 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.236+0000 7f4cdaffd640 1 -- 172.21.15.89:0/2240051248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cdc191d40 con 0x7f4cdc1065d0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.236+0000 7f4cdaffd640 1 --2- 172.21.15.89:0/2240051248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc191800 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0097f0 tx=0x7f4ccc030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.236+0000 7f4cc3fff640 1 -- 172.21.15.89:0/2240051248 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ccc037040 con 0x7f4cdc1065d0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.237+0000 7f4cc3fff640 1 -- 172.21.15.89:0/2240051248 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4ccc040070 con 0x7f4cdc1065d0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.237+0000 7f4ce1675640 1 -- 172.21.15.89:0/2240051248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cdc18ff30 con 0x7f4cdc1065d0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.237+0000 7f4cc3fff640 1 -- 172.21.15.89:0/2240051248 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ccc03a390 con 0x7f4cdc1065d0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.237+0000 7f4ce1675640 1 -- 172.21.15.89:0/2240051248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cdc190410 con 0x7f4cdc1065d0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.238+0000 7f4cc3fff640 1 -- 172.21.15.89:0/2240051248 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f4ccc03b430 con 0x7f4cdc1065d0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.238+0000 7f4ce1675640 1 -- 172.21.15.89:0/2240051248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cdc10ad00 con 0x7f4cdc1065d0 2024-09-15T23:14:38.087 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.238+0000 7f4cc3fff640 1 --2- 172.21.15.89:0/2240051248 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4cc40391b0 0x7f4cc403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.238+0000 7f4cc3fff640 1 -- 172.21.15.89:0/2240051248 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4ccc06c7a0 con 0x7f4cdc1065d0 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.238+0000 7f4cda7fc640 1 --2- 172.21.15.89:0/2240051248 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4cc40391b0 0x7f4cc403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.238+0000 7f4cda7fc640 1 --2- 172.21.15.89:0/2240051248 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4cc40391b0 0x7f4cc403b670 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00098c0 tx=0x7f4cd0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.243+0000 7f4cc3fff640 1 -- 172.21.15.89:0/2240051248 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4ccc032900 con 0x7f4cdc1065d0 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.452+0000 7f4ce1675640 1 -- 172.21.15.89:0/2240051248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f4cdc195600 con 0x7f4cdc1065d0 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.453+0000 7f4cc3fff640 1 -- 172.21.15.89:0/2240051248 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v8) ==== 56+0+88 (secure 0 0 0) 0x7f4ccc040ca0 con 0x7f4cdc1065d0 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.455+0000 7f4ce1675640 1 -- 172.21.15.89:0/2240051248 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4cc40391b0 msgr2=0x7f4cc403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.455+0000 7f4ce1675640 1 --2- 172.21.15.89:0/2240051248 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4cc40391b0 0x7f4cc403b670 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00098c0 tx=0x7f4cd0006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.455+0000 7f4ce1675640 1 -- 172.21.15.89:0/2240051248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 msgr2=0x7f4cdc191800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.455+0000 7f4ce1675640 1 --2- 172.21.15.89:0/2240051248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc191800 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc0097f0 tx=0x7f4ccc030d80 comp rx=0 tx=0).stop 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.455+0000 7f4ce1675640 1 -- 172.21.15.89:0/2240051248 shutdown_connections 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.455+0000 7f4ce1675640 1 --2- 172.21.15.89:0/2240051248 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4cc40391b0 0x7f4cc403b670 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.088 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.455+0000 7f4ce1675640 1 --2- 172.21.15.89:0/2240051248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cdc1065d0 0x7f4cdc191800 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.455+0000 7f4ce1675640 1 -- 172.21.15.89:0/2240051248 >> 172.21.15.89:0/2240051248 conn(0x7f4cdc101da0 msgr2=0x7f4cdc102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:38.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.455+0000 7f4ce1675640 1 -- 172.21.15.89:0/2240051248 shutdown_connections 2024-09-15T23:14:38.089 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:37.455+0000 7f4ce1675640 1 -- 172.21.15.89:0/2240051248 wait complete. 2024-09-15T23:14:38.089 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for the mgr to restart... 2024-09-15T23:14:38.089 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr epoch 8... 2024-09-15T23:14:39.024 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:38 smithi089 ceph-mon[30933]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-15T23:14:39.024 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:38 smithi089 ceph-mon[30933]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-15T23:14:39.032 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:14:39.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 8, 2024-09-15T23:14:39.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-15T23:14:39.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:14:39.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.471+0000 7f3026a95640 1 Processor -- start 2024-09-15T23:14:39.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.471+0000 7f3026a95640 1 -- start start 2024-09-15T23:14:39.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.471+0000 7f3026a95640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302007c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:39.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.471+0000 7f3026a95640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f302007c910 con 0x7f302007dee0 2024-09-15T23:14:39.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.472+0000 7f3024831640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302007c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:39.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.472+0000 7f3024831640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302007c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47880/0 (socket says 172.21.15.89:47880) 2024-09-15T23:14:39.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.472+0000 7f3024831640 1 -- 172.21.15.89:0/3497900128 learned_addr learned my addr 172.21.15.89:0/3497900128 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:39.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.472+0000 7f3024831640 1 -- 172.21.15.89:0/3497900128 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f302007ca90 con 0x7f302007dee0 2024-09-15T23:14:39.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.472+0000 7f3024831640 1 --2- 172.21.15.89:0/3497900128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302007c340 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3014009820 tx=0x7f301402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1bb93f05cf26ab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:39.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.473+0000 7f301f7fe640 1 -- 172.21.15.89:0/3497900128 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3014037070 con 0x7f302007dee0 2024-09-15T23:14:39.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.473+0000 7f301f7fe640 1 -- 172.21.15.89:0/3497900128 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3014032900 con 0x7f302007dee0 2024-09-15T23:14:39.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.473+0000 7f301f7fe640 1 -- 172.21.15.89:0/3497900128 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3014030850 con 0x7f302007dee0 2024-09-15T23:14:39.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.473+0000 7f3026a95640 1 -- 172.21.15.89:0/3497900128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 msgr2=0x7f302007c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:39.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.473+0000 7f3026a95640 1 --2- 172.21.15.89:0/3497900128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302007c340 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3014009820 tx=0x7f301402a270 comp rx=0 tx=0).stop 2024-09-15T23:14:39.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.473+0000 7f3026a95640 1 -- 172.21.15.89:0/3497900128 shutdown_connections 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.473+0000 7f3026a95640 1 --2- 172.21.15.89:0/3497900128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302007c340 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.473+0000 7f3026a95640 1 -- 172.21.15.89:0/3497900128 >> 172.21.15.89:0/3497900128 conn(0x7f3020102ed0 msgr2=0x7f30201052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.473+0000 7f3026a95640 1 -- 172.21.15.89:0/3497900128 shutdown_connections 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.473+0000 7f3026a95640 1 -- 172.21.15.89:0/3497900128 wait complete. 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.474+0000 7f3026a95640 1 Processor -- start 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.474+0000 7f3026a95640 1 -- start start 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.474+0000 7f3026a95640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.474+0000 7f3026a95640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f302010dd70 con 0x7f302007dee0 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.474+0000 7f3024831640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.474+0000 7f3024831640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302019d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47888/0 (socket says 172.21.15.89:47888) 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.474+0000 7f3024831640 1 -- 172.21.15.89:0/383638052 learned_addr learned my addr 172.21.15.89:0/383638052 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.474+0000 7f3024831640 1 -- 172.21.15.89:0/383638052 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f302019ded0 con 0x7f302007dee0 2024-09-15T23:14:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.475+0000 7f3024831640 1 --2- 172.21.15.89:0/383638052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302019d990 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f30140097f0 tx=0x7f3014030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.475+0000 7f301dffb640 1 -- 172.21.15.89:0/383638052 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3014042070 con 0x7f302007dee0 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.475+0000 7f301dffb640 1 -- 172.21.15.89:0/383638052 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3014032ca0 con 0x7f302007dee0 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.475+0000 7f3026a95640 1 -- 172.21.15.89:0/383638052 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f302019e160 con 0x7f302007dee0 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.475+0000 7f301dffb640 1 -- 172.21.15.89:0/383638052 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3014037040 con 0x7f302007dee0 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.475+0000 7f3026a95640 1 -- 172.21.15.89:0/383638052 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30201a0e50 con 0x7f302007dee0 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.476+0000 7f301dffb640 1 -- 172.21.15.89:0/383638052 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f3014045070 con 0x7f302007dee0 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.476+0000 7f301dffb640 1 --2- 172.21.15.89:0/383638052 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f3010039200 0x7f301003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.476+0000 7f3026a95640 1 -- 172.21.15.89:0/383638052 --> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f302007c3c0 con 0x7f3010039200 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.476+0000 7f301dffb640 1 -- 172.21.15.89:0/383638052 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f301406c170 con 0x7f302007dee0 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.476+0000 7f301ffff640 1 --2- 172.21.15.89:0/383638052 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f3010039200 0x7f301003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.476+0000 7f301ffff640 1 --2- 172.21.15.89:0/383638052 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f3010039200 0x7f301003b6c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f300c0098c0 tx=0x7f300c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.477+0000 7f301dffb640 1 -- 172.21.15.89:0/383638052 <== mgr.14118 v2:172.21.15.89:6800/429862840 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f302007c3c0 con 0x7f3010039200 2024-09-15T23:14:39.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.483+0000 7f3026a95640 1 -- 172.21.15.89:0/383638052 --> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f30201041c0 con 0x7f3010039200 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.483+0000 7f301dffb640 1 -- 172.21.15.89:0/383638052 <== mgr.14118 v2:172.21.15.89:6800/429862840 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f30201041c0 con 0x7f3010039200 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.484+0000 7f3026a95640 1 -- 172.21.15.89:0/383638052 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f3010039200 msgr2=0x7f301003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.484+0000 7f3026a95640 1 --2- 172.21.15.89:0/383638052 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f3010039200 0x7f301003b6c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f300c0098c0 tx=0x7f300c006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.484+0000 7f3026a95640 1 -- 172.21.15.89:0/383638052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 msgr2=0x7f302019d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.484+0000 7f3026a95640 1 --2- 172.21.15.89:0/383638052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302019d990 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f30140097f0 tx=0x7f3014030ec0 comp rx=0 tx=0).stop 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.484+0000 7f3026a95640 1 -- 172.21.15.89:0/383638052 shutdown_connections 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.484+0000 7f3026a95640 1 --2- 172.21.15.89:0/383638052 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f3010039200 0x7f301003b6c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.484+0000 7f3026a95640 1 --2- 172.21.15.89:0/383638052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f302007dee0 0x7f302019d990 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.484+0000 7f3026a95640 1 -- 172.21.15.89:0/383638052 >> 172.21.15.89:0/383638052 conn(0x7f3020102ed0 msgr2=0x7f3020103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.484+0000 7f3026a95640 1 -- 172.21.15.89:0/383638052 shutdown_connections 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:38.484+0000 7f3026a95640 1 -- 172.21.15.89:0/383638052 wait complete. 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:mgr epoch 8 is available 2024-09-15T23:14:39.037 INFO:teuthology.orchestra.run.smithi089.stdout:Setting orchestrator backend to cephadm... 2024-09-15T23:14:40.106 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.403+0000 7fa918d82640 1 Processor -- start 2024-09-15T23:14:40.106 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.404+0000 7fa918d82640 1 -- start start 2024-09-15T23:14:40.106 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.404+0000 7fa918d82640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa914106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.106 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.404+0000 7fa918d82640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa914106f50 con 0x7fa914106580 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.404+0000 7fa91259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa914106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.404+0000 7fa91259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa914106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47900/0 (socket says 172.21.15.89:47900) 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.404+0000 7fa91259c640 1 -- 172.21.15.89:0/4146843908 learned_addr learned my addr 172.21.15.89:0/4146843908 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.404+0000 7fa91259c640 1 -- 172.21.15.89:0/4146843908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa914107780 con 0x7fa914106580 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.405+0000 7fa91259c640 1 --2- 172.21.15.89:0/4146843908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa914106980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009820 tx=0x7fa90c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a0d0897934b215 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.405+0000 7fa91159a640 1 -- 172.21.15.89:0/4146843908 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa90c037070 con 0x7fa914106580 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.405+0000 7fa91159a640 1 -- 172.21.15.89:0/4146843908 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa90c032900 con 0x7fa914106580 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.405+0000 7fa91159a640 1 -- 172.21.15.89:0/4146843908 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa90c030850 con 0x7fa914106580 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.406+0000 7fa918d82640 1 -- 172.21.15.89:0/4146843908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 msgr2=0x7fa914106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.406+0000 7fa918d82640 1 --2- 172.21.15.89:0/4146843908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa914106980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009820 tx=0x7fa90c02a270 comp rx=0 tx=0).stop 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.406+0000 7fa918d82640 1 -- 172.21.15.89:0/4146843908 shutdown_connections 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.406+0000 7fa918d82640 1 --2- 172.21.15.89:0/4146843908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa914106980 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.107 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.406+0000 7fa918d82640 1 -- 172.21.15.89:0/4146843908 >> 172.21.15.89:0/4146843908 conn(0x7fa914101d10 msgr2=0x7fa914104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.406+0000 7fa918d82640 1 -- 172.21.15.89:0/4146843908 shutdown_connections 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.406+0000 7fa918d82640 1 -- 172.21.15.89:0/4146843908 wait complete. 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa918d82640 1 Processor -- start 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa918d82640 1 -- start start 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa918d82640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa9141917b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa918d82640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa91410ab30 con 0x7fa914106580 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa91259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa9141917b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa91259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa9141917b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47904/0 (socket says 172.21.15.89:47904) 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa91259c640 1 -- 172.21.15.89:0/1949085089 learned_addr learned my addr 172.21.15.89:0/1949085089 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa91259c640 1 -- 172.21.15.89:0/1949085089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa914191cf0 con 0x7fa914106580 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa91259c640 1 --2- 172.21.15.89:0/1949085089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa9141917b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009940 tx=0x7fa90c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa9037fe640 1 -- 172.21.15.89:0/1949085089 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa90c042070 con 0x7fa914106580 2024-09-15T23:14:40.108 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa9037fe640 1 -- 172.21.15.89:0/1949085089 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa90c0308c0 con 0x7fa914106580 2024-09-15T23:14:40.109 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa918d82640 1 -- 172.21.15.89:0/1949085089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91418fee0 con 0x7fa914106580 2024-09-15T23:14:40.109 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.407+0000 7fa9037fe640 1 -- 172.21.15.89:0/1949085089 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa90c037040 con 0x7fa914106580 2024-09-15T23:14:40.109 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.408+0000 7fa918d82640 1 -- 172.21.15.89:0/1949085089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa914190420 con 0x7fa914106580 2024-09-15T23:14:40.109 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.408+0000 7fa9037fe640 1 -- 172.21.15.89:0/1949085089 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fa90c045070 con 0x7fa914106580 2024-09-15T23:14:40.109 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.408+0000 7fa918d82640 1 -- 172.21.15.89:0/1949085089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa91410acb0 con 0x7fa914106580 2024-09-15T23:14:40.109 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.409+0000 7fa9037fe640 1 --2- 172.21.15.89:0/1949085089 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fa8fc0391b0 0x7fa8fc03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.109 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.409+0000 7fa9037fe640 1 -- 172.21.15.89:0/1949085089 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa90c06c330 con 0x7fa914106580 2024-09-15T23:14:40.109 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.409+0000 7fa911d9b640 1 --2- 172.21.15.89:0/1949085089 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fa8fc0391b0 0x7fa8fc03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.109 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.409+0000 7fa911d9b640 1 --2- 172.21.15.89:0/1949085089 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fa8fc0391b0 0x7fa8fc03b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa90800ad30 tx=0x7fa9080093f0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:40.109 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.413+0000 7fa9037fe640 1 -- 172.21.15.89:0/1949085089 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa90c02fd90 con 0x7fa914106580 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.597+0000 7fa918d82640 1 -- 172.21.15.89:0/1949085089 --> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fa9140008d0 con 0x7fa8fc0391b0 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.602+0000 7fa9037fe640 1 -- 172.21.15.89:0/1949085089 <== mgr.14118 v2:172.21.15.89:6800/429862840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa9140008d0 con 0x7fa8fc0391b0 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.604+0000 7fa918d82640 1 -- 172.21.15.89:0/1949085089 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fa8fc0391b0 msgr2=0x7fa8fc03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.604+0000 7fa918d82640 1 --2- 172.21.15.89:0/1949085089 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fa8fc0391b0 0x7fa8fc03b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa90800ad30 tx=0x7fa9080093f0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.604+0000 7fa918d82640 1 -- 172.21.15.89:0/1949085089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 msgr2=0x7fa9141917b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.604+0000 7fa918d82640 1 --2- 172.21.15.89:0/1949085089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa9141917b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009940 tx=0x7fa90c032ce0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.604+0000 7fa918d82640 1 -- 172.21.15.89:0/1949085089 shutdown_connections 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.604+0000 7fa918d82640 1 --2- 172.21.15.89:0/1949085089 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fa8fc0391b0 0x7fa8fc03b670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.604+0000 7fa918d82640 1 --2- 172.21.15.89:0/1949085089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa914106580 0x7fa9141917b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.604+0000 7fa918d82640 1 -- 172.21.15.89:0/1949085089 >> 172.21.15.89:0/1949085089 conn(0x7fa914101d10 msgr2=0x7fa914102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.604+0000 7fa918d82640 1 -- 172.21.15.89:0/1949085089 shutdown_connections 2024-09-15T23:14:40.110 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:39.604+0000 7fa918d82640 1 -- 172.21.15.89:0/1949085089 wait complete. 2024-09-15T23:14:40.699 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:40 smithi089 ceph-mon[30933]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:40.699 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:40 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:40.699 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:40 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:14:41.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-15T23:14:41.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.520+0000 7f15ce7f9640 1 Processor -- start 2024-09-15T23:14:41.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.520+0000 7f15ce7f9640 1 -- start start 2024-09-15T23:14:41.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.520+0000 7f15ce7f9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c8107b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:41.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.520+0000 7f15ce7f9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f15c81080f0 con 0x7f15c8107720 2024-09-15T23:14:41.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.520+0000 7f15c7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c8107b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:41.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.520+0000 7f15c7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c8107b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47914/0 (socket says 172.21.15.89:47914) 2024-09-15T23:14:41.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.520+0000 7f15c7fff640 1 -- 172.21.15.89:0/3145609650 learned_addr learned my addr 172.21.15.89:0/3145609650 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:41.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.520+0000 7f15c7fff640 1 -- 172.21.15.89:0/3145609650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c81088d0 con 0x7f15c8107720 2024-09-15T23:14:41.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.521+0000 7f15c7fff640 1 --2- 172.21.15.89:0/3145609650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c8107b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009820 tx=0x7f15b802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d4e73b6fe1c0cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:41.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.521+0000 7f15c6ffd640 1 -- 172.21.15.89:0/3145609650 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15b8037070 con 0x7f15c8107720 2024-09-15T23:14:41.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.521+0000 7f15c6ffd640 1 -- 172.21.15.89:0/3145609650 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f15b8032900 con 0x7f15c8107720 2024-09-15T23:14:41.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.521+0000 7f15c6ffd640 1 -- 172.21.15.89:0/3145609650 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15b8030850 con 0x7f15c8107720 2024-09-15T23:14:41.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.522+0000 7f15ce7f9640 1 -- 172.21.15.89:0/3145609650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 msgr2=0x7f15c8107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:41.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.522+0000 7f15ce7f9640 1 --2- 172.21.15.89:0/3145609650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c8107b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009820 tx=0x7f15b802a270 comp rx=0 tx=0).stop 2024-09-15T23:14:41.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.522+0000 7f15ce7f9640 1 -- 172.21.15.89:0/3145609650 shutdown_connections 2024-09-15T23:14:41.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.522+0000 7f15ce7f9640 1 --2- 172.21.15.89:0/3145609650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c8107b20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:41.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.522+0000 7f15ce7f9640 1 -- 172.21.15.89:0/3145609650 >> 172.21.15.89:0/3145609650 conn(0x7f15c8102e90 msgr2=0x7f15c81052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.522+0000 7f15ce7f9640 1 -- 172.21.15.89:0/3145609650 shutdown_connections 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.522+0000 7f15ce7f9640 1 -- 172.21.15.89:0/3145609650 wait complete. 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.522+0000 7f15ce7f9640 1 Processor -- start 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15ce7f9640 1 -- start start 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15ce7f9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c819b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15ce7f9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f15c810bc80 con 0x7f15c8107720 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15c7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c819b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15c7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c819b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47926/0 (socket says 172.21.15.89:47926) 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15c7fff640 1 -- 172.21.15.89:0/463968074 learned_addr learned my addr 172.21.15.89:0/463968074 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15c7fff640 1 -- 172.21.15.89:0/463968074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c819bc80 con 0x7f15c8107720 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15c7fff640 1 --2- 172.21.15.89:0/463968074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c819b740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009940 tx=0x7f15b80326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15c57fa640 1 -- 172.21.15.89:0/463968074 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15b8042070 con 0x7f15c8107720 2024-09-15T23:14:41.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15c57fa640 1 -- 172.21.15.89:0/463968074 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f15b8032be0 con 0x7f15c8107720 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.523+0000 7f15ce7f9640 1 -- 172.21.15.89:0/463968074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15c819bf10 con 0x7f15c8107720 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.524+0000 7f15c57fa640 1 -- 172.21.15.89:0/463968074 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15b8037040 con 0x7f15c8107720 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.524+0000 7f15ce7f9640 1 -- 172.21.15.89:0/463968074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15c819ec00 con 0x7f15c8107720 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.524+0000 7f15c57fa640 1 -- 172.21.15.89:0/463968074 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f15b8045070 con 0x7f15c8107720 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.524+0000 7f15ce7f9640 1 -- 172.21.15.89:0/463968074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15c810be00 con 0x7f15c8107720 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.525+0000 7f15c57fa640 1 --2- 172.21.15.89:0/463968074 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f15b00391b0 0x7f15b003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.525+0000 7f15c57fa640 1 -- 172.21.15.89:0/463968074 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f15b806c360 con 0x7f15c8107720 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.525+0000 7f15c77fe640 1 --2- 172.21.15.89:0/463968074 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f15b00391b0 0x7f15b003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.525+0000 7f15c77fe640 1 --2- 172.21.15.89:0/463968074 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f15b00391b0 0x7f15b003b670 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0098c0 tx=0x7f15bc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.529+0000 7f15c57fa640 1 -- 172.21.15.89:0/463968074 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f15b8032e00 con 0x7f15c8107720 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.698+0000 7f15ce7f9640 1 -- 172.21.15.89:0/463968074 --> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f15c81041d0 con 0x7f15b00391b0 2024-09-15T23:14:41.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.698+0000 7f15c57fa640 1 -- 172.21.15.89:0/463968074 <== mgr.14118 v2:172.21.15.89:6800/429862840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f15c81041d0 con 0x7f15b00391b0 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.701+0000 7f15ce7f9640 1 -- 172.21.15.89:0/463968074 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f15b00391b0 msgr2=0x7f15b003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.701+0000 7f15ce7f9640 1 --2- 172.21.15.89:0/463968074 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f15b00391b0 0x7f15b003b670 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0098c0 tx=0x7f15bc006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.701+0000 7f15ce7f9640 1 -- 172.21.15.89:0/463968074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 msgr2=0x7f15c819b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.701+0000 7f15ce7f9640 1 --2- 172.21.15.89:0/463968074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c819b740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009940 tx=0x7f15b80326e0 comp rx=0 tx=0).stop 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.701+0000 7f15ce7f9640 1 -- 172.21.15.89:0/463968074 shutdown_connections 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.701+0000 7f15ce7f9640 1 --2- 172.21.15.89:0/463968074 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f15b00391b0 0x7f15b003b670 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.701+0000 7f15ce7f9640 1 --2- 172.21.15.89:0/463968074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15c8107720 0x7f15c819b740 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.701+0000 7f15ce7f9640 1 -- 172.21.15.89:0/463968074 >> 172.21.15.89:0/463968074 conn(0x7f15c8102e90 msgr2=0x7f15c8103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.701+0000 7f15ce7f9640 1 -- 172.21.15.89:0/463968074 shutdown_connections 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:40.701+0000 7f15ce7f9640 1 -- 172.21.15.89:0/463968074 wait complete. 2024-09-15T23:14:41.275 INFO:teuthology.orchestra.run.smithi089.stdout:Generating ssh key... 2024-09-15T23:14:41.813 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:41 smithi089 ceph-mon[30933]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:42.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: Generating public/private rsa key pair. 2024-09-15T23:14:42.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: Your identification has been saved in /tmp/tmpc2oup_2v/key 2024-09-15T23:14:42.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: Your public key has been saved in /tmp/tmpc2oup_2v/key.pub 2024-09-15T23:14:42.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: The key fingerprint is: 2024-09-15T23:14:42.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: SHA256:eqDO+LMJ7aekl+SrBq+CojbuawyI4mVd0P5a7EtmOWk ceph-1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:42.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: The key's randomart image is: 2024-09-15T23:14:42.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: +---[RSA 3072]----+ 2024-09-15T23:14:42.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: | . | 2024-09-15T23:14:42.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: | . . | 2024-09-15T23:14:42.312 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: | o | 2024-09-15T23:14:42.312 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: | o | 2024-09-15T23:14:42.312 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: |o . o S | 2024-09-15T23:14:42.312 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: |* +.o o +o | 2024-09-15T23:14:42.312 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: |*o+o+.. =E | 2024-09-15T23:14:42.312 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: |+*oO=..o=.. | 2024-09-15T23:14:42.312 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: |@**+X* .. | 2024-09-15T23:14:42.312 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: +----[SHA256]-----+ 2024-09-15T23:14:42.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.684+0000 7fba707c1640 1 Processor -- start 2024-09-15T23:14:42.752 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.684+0000 7fba707c1640 1 -- start start 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.684+0000 7fba707c1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6810a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.684+0000 7fba707c1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fba6807be90 con 0x7fba68108180 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.684+0000 7fba6e55d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6810a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.684+0000 7fba6e55d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6810a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47940/0 (socket says 172.21.15.89:47940) 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.684+0000 7fba6e55d640 1 -- 172.21.15.89:0/3854947529 learned_addr learned my addr 172.21.15.89:0/3854947529 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.685+0000 7fba6e55d640 1 -- 172.21.15.89:0/3854947529 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6810aab0 con 0x7fba68108180 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.685+0000 7fba6e55d640 1 --2- 172.21.15.89:0/3854947529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6810a570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fba64009820 tx=0x7fba6402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902adec79c66db97 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.685+0000 7fba6d55b640 1 -- 172.21.15.89:0/3854947529 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba64037070 con 0x7fba68108180 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.685+0000 7fba6d55b640 1 -- 172.21.15.89:0/3854947529 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fba64032900 con 0x7fba68108180 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.685+0000 7fba6d55b640 1 -- 172.21.15.89:0/3854947529 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba64030850 con 0x7fba68108180 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.686+0000 7fba707c1640 1 -- 172.21.15.89:0/3854947529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 msgr2=0x7fba6810a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.686+0000 7fba707c1640 1 --2- 172.21.15.89:0/3854947529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6810a570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fba64009820 tx=0x7fba6402a270 comp rx=0 tx=0).stop 2024-09-15T23:14:42.753 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.686+0000 7fba707c1640 1 -- 172.21.15.89:0/3854947529 shutdown_connections 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.686+0000 7fba707c1640 1 --2- 172.21.15.89:0/3854947529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6810a570 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.686+0000 7fba707c1640 1 -- 172.21.15.89:0/3854947529 >> 172.21.15.89:0/3854947529 conn(0x7fba68101da0 msgr2=0x7fba681041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.686+0000 7fba707c1640 1 -- 172.21.15.89:0/3854947529 shutdown_connections 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.686+0000 7fba707c1640 1 -- 172.21.15.89:0/3854947529 wait complete. 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba707c1640 1 Processor -- start 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba707c1640 1 -- start start 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba707c1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6819b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba707c1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fba6810baf0 con 0x7fba68108180 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba6e55d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6819b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba6e55d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6819b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47950/0 (socket says 172.21.15.89:47950) 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba6e55d640 1 -- 172.21.15.89:0/3066985103 learned_addr learned my addr 172.21.15.89:0/3066985103 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba6e55d640 1 -- 172.21.15.89:0/3066985103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6819bbd0 con 0x7fba68108180 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba6e55d640 1 --2- 172.21.15.89:0/3066985103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6819b690 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba64032ce0 tx=0x7fba64032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba5f7fe640 1 -- 172.21.15.89:0/3066985103 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba64042070 con 0x7fba68108180 2024-09-15T23:14:42.754 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba707c1640 1 -- 172.21.15.89:0/3066985103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba6819be60 con 0x7fba68108180 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.687+0000 7fba5f7fe640 1 -- 172.21.15.89:0/3066985103 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fba640308c0 con 0x7fba68108180 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.688+0000 7fba5f7fe640 1 -- 172.21.15.89:0/3066985103 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba64037040 con 0x7fba68108180 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.688+0000 7fba707c1640 1 -- 172.21.15.89:0/3066985103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba6819eb50 con 0x7fba68108180 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.688+0000 7fba5f7fe640 1 -- 172.21.15.89:0/3066985103 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fba64045070 con 0x7fba68108180 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.689+0000 7fba5f7fe640 1 --2- 172.21.15.89:0/3066985103 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fba540391b0 0x7fba5403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.689+0000 7fba707c1640 1 -- 172.21.15.89:0/3066985103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba681085e0 con 0x7fba68108180 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.689+0000 7fba5f7fe640 1 -- 172.21.15.89:0/3066985103 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fba6406c560 con 0x7fba68108180 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.689+0000 7fba6dd5c640 1 --2- 172.21.15.89:0/3066985103 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fba540391b0 0x7fba5403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.689+0000 7fba6dd5c640 1 --2- 172.21.15.89:0/3066985103 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fba540391b0 0x7fba5403b670 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fba580098c0 tx=0x7fba58006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.693+0000 7fba5f7fe640 1 -- 172.21.15.89:0/3066985103 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fba6406beb0 con 0x7fba68108180 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:41.859+0000 7fba707c1640 1 -- 172.21.15.89:0/3066985103 --> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fba68103040 con 0x7fba540391b0 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.083+0000 7fba5f7fe640 1 -- 172.21.15.89:0/3066985103 <== mgr.14118 v2:172.21.15.89:6800/429862840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fba68103040 con 0x7fba540391b0 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.085+0000 7fba707c1640 1 -- 172.21.15.89:0/3066985103 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fba540391b0 msgr2=0x7fba5403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:42.755 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.085+0000 7fba707c1640 1 --2- 172.21.15.89:0/3066985103 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fba540391b0 0x7fba5403b670 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fba580098c0 tx=0x7fba58006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:42.756 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.085+0000 7fba707c1640 1 -- 172.21.15.89:0/3066985103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 msgr2=0x7fba6819b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:42.756 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.085+0000 7fba707c1640 1 --2- 172.21.15.89:0/3066985103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6819b690 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fba64032ce0 tx=0x7fba64032d10 comp rx=0 tx=0).stop 2024-09-15T23:14:42.756 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.085+0000 7fba707c1640 1 -- 172.21.15.89:0/3066985103 shutdown_connections 2024-09-15T23:14:42.756 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.085+0000 7fba707c1640 1 --2- 172.21.15.89:0/3066985103 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fba540391b0 0x7fba5403b670 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:42.756 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.085+0000 7fba707c1640 1 --2- 172.21.15.89:0/3066985103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba68108180 0x7fba6819b690 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:42.756 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.085+0000 7fba707c1640 1 -- 172.21.15.89:0/3066985103 >> 172.21.15.89:0/3066985103 conn(0x7fba68101da0 msgr2=0x7fba68102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:42.756 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.085+0000 7fba707c1640 1 -- 172.21.15.89:0/3066985103 shutdown_connections 2024-09-15T23:14:42.756 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:42.085+0000 7fba707c1640 1 -- 172.21.15.89:0/3066985103 wait complete. 2024-09-15T23:14:43.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:43 smithi089 ceph-mon[30933]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:43.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:43 smithi089 ceph-mon[30933]: Generating ssh key... 2024-09-15T23:14:43.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:43 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:43.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:43 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:43.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrKymXigMXdUhkKySbQi7v+c/DFHgKKnpxhw4JiOWN1Zm0jCJ0MB7Gt5QWEzJ4gMTVxcezDv7r1DZaZ5ySddyiKVVLkp6EZcCbxRd71CNlFI25qjpV8xElmzen9iUV/DOeFqcWyijT9ZMkVB7QTtrgMmOFTWNscCtw9Xc8XxKPhInkfoqa53r0OHLsmeYHz2pMSL4GwaW1FbwbtjftWTAdD/9X6m3zlHXn72x9Nrs8nPqaVxhxJ9PDPFRLN+ouLlSi90VdoiJTUwTNfp9hpCFMP639TEssXcvsNhxN3XPB+iSeroXc1WZ97nY4BHvSNMQLufkc7GDXKMEBkBef1lRnkKWbzEsklGEanD2JgOtNyTMAYbxtq/aviZ69Nu72G2v+RrHhmO3C7naEypEd7gRa3MFY9JLyYJlEpH+lTUfTRAcPoFix1mEqWm6TUNxEi6WC2BcPhS6voOVVamNR7ybyvQyRSWEdcM4XrUuXPYtR+RsN3pFL0AmSMPC4kDrBUhs= ceph-1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:14:43.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.164+0000 7f0983e0a640 1 Processor -- start 2024-09-15T23:14:43.925 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.164+0000 7f0983e0a640 1 -- start start 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.164+0000 7f0983e0a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c10a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.164+0000 7f0983e0a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f097c07bdf0 con 0x7f097c1081d0 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.165+0000 7f0981ba6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c10a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.165+0000 7f0981ba6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c10a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47952/0 (socket says 172.21.15.89:47952) 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.165+0000 7f0981ba6640 1 -- 172.21.15.89:0/1302385603 learned_addr learned my addr 172.21.15.89:0/1302385603 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.165+0000 7f0981ba6640 1 -- 172.21.15.89:0/1302385603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c10ab00 con 0x7f097c1081d0 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0981ba6640 1 --2- 172.21.15.89:0/1302385603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c10a5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0974009820 tx=0x7f097402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32e435077d46bc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0980ba4640 1 -- 172.21.15.89:0/1302385603 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0974037070 con 0x7f097c1081d0 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0980ba4640 1 -- 172.21.15.89:0/1302385603 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0974032900 con 0x7f097c1081d0 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0980ba4640 1 -- 172.21.15.89:0/1302385603 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0974030850 con 0x7f097c1081d0 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0983e0a640 1 -- 172.21.15.89:0/1302385603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 msgr2=0x7f097c10a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0983e0a640 1 --2- 172.21.15.89:0/1302385603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c10a5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0974009820 tx=0x7f097402a270 comp rx=0 tx=0).stop 2024-09-15T23:14:43.926 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0983e0a640 1 -- 172.21.15.89:0/1302385603 shutdown_connections 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0983e0a640 1 --2- 172.21.15.89:0/1302385603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c10a5c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0983e0a640 1 -- 172.21.15.89:0/1302385603 >> 172.21.15.89:0/1302385603 conn(0x7f097c101d10 msgr2=0x7f097c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0983e0a640 1 -- 172.21.15.89:0/1302385603 shutdown_connections 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.166+0000 7f0983e0a640 1 -- 172.21.15.89:0/1302385603 wait complete. 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.167+0000 7f0983e0a640 1 Processor -- start 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.167+0000 7f0983e0a640 1 -- start start 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.167+0000 7f0983e0a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c19b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.167+0000 7f0983e0a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f097c10ba80 con 0x7f097c1081d0 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.167+0000 7f0981ba6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c19b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.167+0000 7f0981ba6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c19b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47966/0 (socket says 172.21.15.89:47966) 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.167+0000 7f0981ba6640 1 -- 172.21.15.89:0/56525018 learned_addr learned my addr 172.21.15.89:0/56525018 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.168+0000 7f0981ba6640 1 -- 172.21.15.89:0/56525018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c19bb90 con 0x7f097c1081d0 2024-09-15T23:14:43.927 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.168+0000 7f0981ba6640 1 --2- 172.21.15.89:0/56525018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c19b650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0974032ce0 tx=0x7f0974032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.168+0000 7f0972ffd640 1 -- 172.21.15.89:0/56525018 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0974042070 con 0x7f097c1081d0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.168+0000 7f0972ffd640 1 -- 172.21.15.89:0/56525018 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f09740308c0 con 0x7f097c1081d0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.168+0000 7f0972ffd640 1 -- 172.21.15.89:0/56525018 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0974037040 con 0x7f097c1081d0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.168+0000 7f0983e0a640 1 -- 172.21.15.89:0/56525018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097c19be20 con 0x7f097c1081d0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.168+0000 7f0983e0a640 1 -- 172.21.15.89:0/56525018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f097c19eb10 con 0x7f097c1081d0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.169+0000 7f0972ffd640 1 -- 172.21.15.89:0/56525018 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f0974045070 con 0x7f097c1081d0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.169+0000 7f0983e0a640 1 -- 172.21.15.89:0/56525018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f097c108630 con 0x7f097c1081d0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.169+0000 7f0972ffd640 1 --2- 172.21.15.89:0/56525018 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f0968039200 0x7f096803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.169+0000 7f0972ffd640 1 -- 172.21.15.89:0/56525018 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f097406c2c0 con 0x7f097c1081d0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.169+0000 7f09813a5640 1 --2- 172.21.15.89:0/56525018 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f0968039200 0x7f096803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.170+0000 7f09813a5640 1 --2- 172.21.15.89:0/56525018 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f0968039200 0x7f096803b6c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f096c0098c0 tx=0x7f096c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.174+0000 7f0972ffd640 1 -- 172.21.15.89:0/56525018 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f097404b170 con 0x7f097c1081d0 2024-09-15T23:14:43.928 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.339+0000 7f0983e0a640 1 -- 172.21.15.89:0/56525018 --> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f097c102fb0 con 0x7f0968039200 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.340+0000 7f0972ffd640 1 -- 172.21.15.89:0/56525018 <== mgr.14118 v2:172.21.15.89:6800/429862840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f097c102fb0 con 0x7f0968039200 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.342+0000 7f0983e0a640 1 -- 172.21.15.89:0/56525018 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f0968039200 msgr2=0x7f096803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.342+0000 7f0983e0a640 1 --2- 172.21.15.89:0/56525018 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f0968039200 0x7f096803b6c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f096c0098c0 tx=0x7f096c006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.342+0000 7f0983e0a640 1 -- 172.21.15.89:0/56525018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 msgr2=0x7f097c19b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.342+0000 7f0983e0a640 1 --2- 172.21.15.89:0/56525018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c19b650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0974032ce0 tx=0x7f0974032d10 comp rx=0 tx=0).stop 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.342+0000 7f0983e0a640 1 -- 172.21.15.89:0/56525018 shutdown_connections 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.342+0000 7f0983e0a640 1 --2- 172.21.15.89:0/56525018 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f0968039200 0x7f096803b6c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.342+0000 7f0983e0a640 1 --2- 172.21.15.89:0/56525018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f097c1081d0 0x7f097c19b650 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.342+0000 7f0983e0a640 1 -- 172.21.15.89:0/56525018 >> 172.21.15.89:0/56525018 conn(0x7f097c101d10 msgr2=0x7f097c1028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.342+0000 7f0983e0a640 1 -- 172.21.15.89:0/56525018 shutdown_connections 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:43.342+0000 7f0983e0a640 1 -- 172.21.15.89:0/56525018 wait complete. 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-15T23:14:43.929 INFO:teuthology.orchestra.run.smithi089.stdout:Adding key to root@localhost authorized_keys... 2024-09-15T23:14:43.930 INFO:teuthology.orchestra.run.smithi089.stdout:Adding host smithi089... 2024-09-15T23:14:44.237 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:44 smithi089 ceph-mon[30933]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:45.274 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:45 smithi089 ceph-mon[30933]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi089", "addr": "172.21.15.89", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:46.192 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:46 smithi089 ceph-mon[30933]: Deploying cephadm binary to smithi089 2024-09-15T23:14:48.267 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Added host 'smithi089' with addr '172.21.15.89' 2024-09-15T23:14:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.314+0000 7fd1b1bf2640 1 Processor -- start 2024-09-15T23:14:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.314+0000 7fd1b1bf2640 1 -- start start 2024-09-15T23:14:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.315+0000 7fd1b1bf2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac10a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.315+0000 7fd1b1bf2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac07be90 con 0x7fd1ac108130 2024-09-15T23:14:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.315+0000 7fd1ab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac10a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.315+0000 7fd1ab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac10a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47976/0 (socket says 172.21.15.89:47976) 2024-09-15T23:14:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.315+0000 7fd1ab7fe640 1 -- 172.21.15.89:0/2028864594 learned_addr learned my addr 172.21.15.89:0/2028864594 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.315+0000 7fd1ab7fe640 1 -- 172.21.15.89:0/2028864594 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ac10aa60 con 0x7fd1ac108130 2024-09-15T23:14:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.316+0000 7fd1ab7fe640 1 --2- 172.21.15.89:0/2028864594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac10a520 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009820 tx=0x7fd19c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5401af66253f4240 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.316+0000 7fd1aaffd640 1 -- 172.21.15.89:0/2028864594 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd19c037070 con 0x7fd1ac108130 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.316+0000 7fd1aaffd640 1 -- 172.21.15.89:0/2028864594 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd19c032900 con 0x7fd1ac108130 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.316+0000 7fd1aaffd640 1 -- 172.21.15.89:0/2028864594 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd19c030850 con 0x7fd1ac108130 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.316+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/2028864594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 msgr2=0x7fd1ac10a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.316+0000 7fd1b1bf2640 1 --2- 172.21.15.89:0/2028864594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac10a520 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009820 tx=0x7fd19c02a270 comp rx=0 tx=0).stop 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/2028864594 shutdown_connections 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1b1bf2640 1 --2- 172.21.15.89:0/2028864594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac10a520 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/2028864594 >> 172.21.15.89:0/2028864594 conn(0x7fd1ac101d10 msgr2=0x7fd1ac104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/2028864594 shutdown_connections 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/2028864594 wait complete. 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1b1bf2640 1 Processor -- start 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1b1bf2640 1 -- start start 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1b1bf2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac19b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1b1bf2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac10baa0 con 0x7fd1ac108130 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1ab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac19b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1ab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac19b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47990/0 (socket says 172.21.15.89:47990) 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.317+0000 7fd1ab7fe640 1 -- 172.21.15.89:0/1357507947 learned_addr learned my addr 172.21.15.89:0/1357507947 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.318+0000 7fd1ab7fe640 1 -- 172.21.15.89:0/1357507947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ac19bbc0 con 0x7fd1ac108130 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.318+0000 7fd1ab7fe640 1 --2- 172.21.15.89:0/1357507947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac19b680 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd19c032ce0 tx=0x7fd19c032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.318+0000 7fd1a97fa640 1 -- 172.21.15.89:0/1357507947 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd19c042070 con 0x7fd1ac108130 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.318+0000 7fd1a97fa640 1 -- 172.21.15.89:0/1357507947 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd19c0308c0 con 0x7fd1ac108130 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.318+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/1357507947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1ac19be50 con 0x7fd1ac108130 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.318+0000 7fd1a97fa640 1 -- 172.21.15.89:0/1357507947 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd19c037040 con 0x7fd1ac108130 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.318+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/1357507947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1ac19eb40 con 0x7fd1ac108130 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.319+0000 7fd1a97fa640 1 -- 172.21.15.89:0/1357507947 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fd19c045070 con 0x7fd1ac108130 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.319+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/1357507947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1ac108590 con 0x7fd1ac108130 2024-09-15T23:14:48.270 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.319+0000 7fd1a97fa640 1 --2- 172.21.15.89:0/1357507947 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd198039200 0x7fd19803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.319+0000 7fd1a97fa640 1 -- 172.21.15.89:0/1357507947 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd19c06c330 con 0x7fd1ac108130 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.319+0000 7fd1a3fff640 1 --2- 172.21.15.89:0/1357507947 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd198039200 0x7fd19803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.320+0000 7fd1a3fff640 1 --2- 172.21.15.89:0/1357507947 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd198039200 0x7fd19803b6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd1940098c0 tx=0x7fd194006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.324+0000 7fd1a97fa640 1 -- 172.21.15.89:0/1357507947 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd19c04b170 con 0x7fd1ac108130 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:44.492+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/1357507947 --> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi089", "addr": "172.21.15.89", "target": ["mon-mgr", ""]}) -- 0x7fd1ac103020 con 0x7fd198039200 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.617+0000 7fd1a97fa640 1 -- 172.21.15.89:0/1357507947 <== mgr.14118 v2:172.21.15.89:6800/429862840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fd1ac103020 con 0x7fd198039200 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.620+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/1357507947 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd198039200 msgr2=0x7fd19803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.620+0000 7fd1b1bf2640 1 --2- 172.21.15.89:0/1357507947 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd198039200 0x7fd19803b6c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd1940098c0 tx=0x7fd194006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.620+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/1357507947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 msgr2=0x7fd1ac19b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.620+0000 7fd1b1bf2640 1 --2- 172.21.15.89:0/1357507947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac19b680 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd19c032ce0 tx=0x7fd19c032d10 comp rx=0 tx=0).stop 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.620+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/1357507947 shutdown_connections 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.620+0000 7fd1b1bf2640 1 --2- 172.21.15.89:0/1357507947 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd198039200 0x7fd19803b6c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:48.271 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.620+0000 7fd1b1bf2640 1 --2- 172.21.15.89:0/1357507947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1ac108130 0x7fd1ac19b680 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:48.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.620+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/1357507947 >> 172.21.15.89:0/1357507947 conn(0x7fd1ac101d10 msgr2=0x7fd1ac102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:48.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.620+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/1357507947 shutdown_connections 2024-09-15T23:14:48.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:47.620+0000 7fd1b1bf2640 1 -- 172.21.15.89:0/1357507947 wait complete. 2024-09-15T23:14:48.272 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying unmanaged mon service... 2024-09-15T23:14:48.891 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:48 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:48.891 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:48 smithi089 ceph-mon[30933]: Added host smithi089 2024-09-15T23:14:48.891 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:48 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:14:49.678 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-15T23:14:49.678 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.717+0000 7f8777713640 1 Processor -- start 2024-09-15T23:14:49.678 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.717+0000 7f8777713640 1 -- start start 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.717+0000 7f8777713640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f8770106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.717+0000 7f8777713640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8770106f50 con 0x7f8770106580 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.718+0000 7f87754af640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f8770106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.718+0000 7f87754af640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f8770106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51774/0 (socket says 172.21.15.89:51774) 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.718+0000 7f87754af640 1 -- 172.21.15.89:0/687087576 learned_addr learned my addr 172.21.15.89:0/687087576 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.718+0000 7f87754af640 1 -- 172.21.15.89:0/687087576 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8770107780 con 0x7f8770106580 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.718+0000 7f87754af640 1 --2- 172.21.15.89:0/687087576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f8770106980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f876c009820 tx=0x7f876c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74190460dc7aa65f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.719+0000 7f8767fff640 1 -- 172.21.15.89:0/687087576 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f876c037070 con 0x7f8770106580 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.719+0000 7f8767fff640 1 -- 172.21.15.89:0/687087576 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f876c032900 con 0x7f8770106580 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.719+0000 7f8767fff640 1 -- 172.21.15.89:0/687087576 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f876c030850 con 0x7f8770106580 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.719+0000 7f8777713640 1 -- 172.21.15.89:0/687087576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 msgr2=0x7f8770106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.719+0000 7f8777713640 1 --2- 172.21.15.89:0/687087576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f8770106980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f876c009820 tx=0x7f876c02a270 comp rx=0 tx=0).stop 2024-09-15T23:14:49.679 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.719+0000 7f8777713640 1 -- 172.21.15.89:0/687087576 shutdown_connections 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.719+0000 7f8777713640 1 --2- 172.21.15.89:0/687087576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f8770106980 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.719+0000 7f8777713640 1 -- 172.21.15.89:0/687087576 >> 172.21.15.89:0/687087576 conn(0x7f8770101d10 msgr2=0x7f8770104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.719+0000 7f8777713640 1 -- 172.21.15.89:0/687087576 shutdown_connections 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.719+0000 7f8777713640 1 -- 172.21.15.89:0/687087576 wait complete. 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.720+0000 7f8777713640 1 Processor -- start 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.720+0000 7f8777713640 1 -- start start 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.720+0000 7f8777713640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f877019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.720+0000 7f8777713640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f877010ab30 con 0x7f8770106580 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.720+0000 7f87754af640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f877019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.720+0000 7f87754af640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f877019a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51778/0 (socket says 172.21.15.89:51778) 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.720+0000 7f87754af640 1 -- 172.21.15.89:0/1992920308 learned_addr learned my addr 172.21.15.89:0/1992920308 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.720+0000 7f87754af640 1 -- 172.21.15.89:0/1992920308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877019aae0 con 0x7f8770106580 2024-09-15T23:14:49.680 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.721+0000 7f87754af640 1 --2- 172.21.15.89:0/1992920308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f877019a5a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f876c009940 tx=0x7f876c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.721+0000 7f87667fc640 1 -- 172.21.15.89:0/1992920308 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f876c042070 con 0x7f8770106580 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.721+0000 7f87667fc640 1 -- 172.21.15.89:0/1992920308 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f876c0308c0 con 0x7f8770106580 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.721+0000 7f8777713640 1 -- 172.21.15.89:0/1992920308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f877019ad70 con 0x7f8770106580 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.721+0000 7f87667fc640 1 -- 172.21.15.89:0/1992920308 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f876c037040 con 0x7f8770106580 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.721+0000 7f8777713640 1 -- 172.21.15.89:0/1992920308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f877019da60 con 0x7f8770106580 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.722+0000 7f87667fc640 1 -- 172.21.15.89:0/1992920308 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f876c045070 con 0x7f8770106580 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.722+0000 7f8777713640 1 -- 172.21.15.89:0/1992920308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f877010acb0 con 0x7f8770106580 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.722+0000 7f87667fc640 1 --2- 172.21.15.89:0/1992920308 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f875c039200 0x7f875c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.722+0000 7f87667fc640 1 -- 172.21.15.89:0/1992920308 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f876c06c2e0 con 0x7f8770106580 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.722+0000 7f8774cae640 1 --2- 172.21.15.89:0/1992920308 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f875c039200 0x7f875c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.722+0000 7f8774cae640 1 --2- 172.21.15.89:0/1992920308 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f875c039200 0x7f875c03b6c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f87600098c0 tx=0x7f8760006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.727+0000 7f87667fc640 1 -- 172.21.15.89:0/1992920308 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f876c04b170 con 0x7f8770106580 2024-09-15T23:14:49.681 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.912+0000 7f8777713640 1 -- 172.21.15.89:0/1992920308 --> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f8770102f80 con 0x7f875c039200 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.916+0000 7f87667fc640 1 -- 172.21.15.89:0/1992920308 <== mgr.14118 v2:172.21.15.89:6800/429862840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8770102f80 con 0x7f875c039200 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.919+0000 7f8777713640 1 -- 172.21.15.89:0/1992920308 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f875c039200 msgr2=0x7f875c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.919+0000 7f8777713640 1 --2- 172.21.15.89:0/1992920308 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f875c039200 0x7f875c03b6c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f87600098c0 tx=0x7f8760006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.919+0000 7f8777713640 1 -- 172.21.15.89:0/1992920308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 msgr2=0x7f877019a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.919+0000 7f8777713640 1 --2- 172.21.15.89:0/1992920308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f877019a5a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f876c009940 tx=0x7f876c032ce0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.919+0000 7f8777713640 1 -- 172.21.15.89:0/1992920308 shutdown_connections 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.919+0000 7f8777713640 1 --2- 172.21.15.89:0/1992920308 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f875c039200 0x7f875c03b6c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.919+0000 7f8777713640 1 --2- 172.21.15.89:0/1992920308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8770106580 0x7f877019a5a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.919+0000 7f8777713640 1 -- 172.21.15.89:0/1992920308 >> 172.21.15.89:0/1992920308 conn(0x7f8770101d10 msgr2=0x7f8770102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.919+0000 7f8777713640 1 -- 172.21.15.89:0/1992920308 shutdown_connections 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:48.919+0000 7f8777713640 1 -- 172.21.15.89:0/1992920308 wait complete. 2024-09-15T23:14:49.682 INFO:teuthology.orchestra.run.smithi089.stdout:Deploying unmanaged mgr service... 2024-09-15T23:14:49.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:49 smithi089 ceph-mon[30933]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:49.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:49 smithi089 ceph-mon[30933]: Saving service mon spec with placement count:5 2024-09-15T23:14:49.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:49 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:51 smithi089 ceph-mon[30933]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:51 smithi089 ceph-mon[30933]: Saving service mgr spec with placement count:2 2024-09-15T23:14:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:51 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:51.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:51 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:51.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:51 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:51.608 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-15T23:14:51.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.051+0000 7fd199d64640 1 Processor -- start 2024-09-15T23:14:51.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.051+0000 7fd199d64640 1 -- start start 2024-09-15T23:14:51.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.051+0000 7fd199d64640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194107fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:51.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.051+0000 7fd199d64640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd1941085a0 con 0x7fd194107bd0 2024-09-15T23:14:51.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.051+0000 7fd1937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194107fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:51.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.051+0000 7fd1937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194107fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51784/0 (socket says 172.21.15.89:51784) 2024-09-15T23:14:51.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.051+0000 7fd1937fe640 1 -- 172.21.15.89:0/3265930964 learned_addr learned my addr 172.21.15.89:0/3265930964 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:51.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.051+0000 7fd1937fe640 1 -- 172.21.15.89:0/3265930964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd194106830 con 0x7fd194107bd0 2024-09-15T23:14:51.609 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.052+0000 7fd1937fe640 1 --2- 172.21.15.89:0/3265930964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194107fd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd184009820 tx=0x7fd18402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4d46d523983786 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.052+0000 7fd1927fc640 1 -- 172.21.15.89:0/3265930964 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd184037070 con 0x7fd194107bd0 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.052+0000 7fd1927fc640 1 -- 172.21.15.89:0/3265930964 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd184032900 con 0x7fd194107bd0 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.052+0000 7fd1927fc640 1 -- 172.21.15.89:0/3265930964 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd184030850 con 0x7fd194107bd0 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.053+0000 7fd199d64640 1 -- 172.21.15.89:0/3265930964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 msgr2=0x7fd194107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.053+0000 7fd199d64640 1 --2- 172.21.15.89:0/3265930964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194107fd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd184009820 tx=0x7fd18402a270 comp rx=0 tx=0).stop 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.053+0000 7fd199d64640 1 -- 172.21.15.89:0/3265930964 shutdown_connections 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.053+0000 7fd199d64640 1 --2- 172.21.15.89:0/3265930964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194107fd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.053+0000 7fd199d64640 1 -- 172.21.15.89:0/3265930964 >> 172.21.15.89:0/3265930964 conn(0x7fd194101530 msgr2=0x7fd194103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.053+0000 7fd199d64640 1 -- 172.21.15.89:0/3265930964 shutdown_connections 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.053+0000 7fd199d64640 1 -- 172.21.15.89:0/3265930964 wait complete. 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.053+0000 7fd199d64640 1 Processor -- start 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd199d64640 1 -- start start 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd199d64640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194191800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:51.610 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd199d64640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd19410abf0 con 0x7fd194107bd0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd1937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194191800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd1937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194191800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51798/0 (socket says 172.21.15.89:51798) 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd1937fe640 1 -- 172.21.15.89:0/2213450504 learned_addr learned my addr 172.21.15.89:0/2213450504 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd1937fe640 1 -- 172.21.15.89:0/2213450504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd194191d40 con 0x7fd194107bd0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd1937fe640 1 --2- 172.21.15.89:0/2213450504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194191800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd184009940 tx=0x7fd184032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd190ff9640 1 -- 172.21.15.89:0/2213450504 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd184042070 con 0x7fd194107bd0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd190ff9640 1 -- 172.21.15.89:0/2213450504 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd1840308c0 con 0x7fd194107bd0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd199d64640 1 -- 172.21.15.89:0/2213450504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd19418ff30 con 0x7fd194107bd0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd190ff9640 1 -- 172.21.15.89:0/2213450504 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd184037040 con 0x7fd194107bd0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.054+0000 7fd199d64640 1 -- 172.21.15.89:0/2213450504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd194190470 con 0x7fd194107bd0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.055+0000 7fd190ff9640 1 -- 172.21.15.89:0/2213450504 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fd184045070 con 0x7fd194107bd0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.055+0000 7fd190ff9640 1 --2- 172.21.15.89:0/2213450504 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd17c0391b0 0x7fd17c03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.055+0000 7fd190ff9640 1 -- 172.21.15.89:0/2213450504 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd18406c610 con 0x7fd194107bd0 2024-09-15T23:14:51.611 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.055+0000 7fd192ffd640 1 --2- 172.21.15.89:0/2213450504 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd17c0391b0 0x7fd17c03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:51.612 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.056+0000 7fd199d64640 1 -- 172.21.15.89:0/2213450504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd194108050 con 0x7fd194107bd0 2024-09-15T23:14:51.612 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.056+0000 7fd192ffd640 1 --2- 172.21.15.89:0/2213450504 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd17c0391b0 0x7fd17c03b670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd1880098c0 tx=0x7fd188006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:51.612 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.060+0000 7fd190ff9640 1 -- 172.21.15.89:0/2213450504 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd18403cd90 con 0x7fd194107bd0 2024-09-15T23:14:51.612 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.238+0000 7fd199d64640 1 -- 172.21.15.89:0/2213450504 --> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fd194190be0 con 0x7fd17c0391b0 2024-09-15T23:14:51.612 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.243+0000 7fd190ff9640 1 -- 172.21.15.89:0/2213450504 <== mgr.14118 v2:172.21.15.89:6800/429862840 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd194190be0 con 0x7fd17c0391b0 2024-09-15T23:14:51.612 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.246+0000 7fd199d64640 1 -- 172.21.15.89:0/2213450504 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd17c0391b0 msgr2=0x7fd17c03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:51.612 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.246+0000 7fd199d64640 1 --2- 172.21.15.89:0/2213450504 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd17c0391b0 0x7fd17c03b670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd1880098c0 tx=0x7fd188006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:51.612 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.247+0000 7fd199d64640 1 -- 172.21.15.89:0/2213450504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 msgr2=0x7fd194191800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:51.612 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.247+0000 7fd199d64640 1 --2- 172.21.15.89:0/2213450504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194191800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd184009940 tx=0x7fd184032ce0 comp rx=0 tx=0).stop 2024-09-15T23:14:51.612 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.247+0000 7fd199d64640 1 -- 172.21.15.89:0/2213450504 shutdown_connections 2024-09-15T23:14:51.613 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.247+0000 7fd199d64640 1 --2- 172.21.15.89:0/2213450504 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fd17c0391b0 0x7fd17c03b670 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:51.613 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.247+0000 7fd199d64640 1 --2- 172.21.15.89:0/2213450504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd194107bd0 0x7fd194191800 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:51.613 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.247+0000 7fd199d64640 1 -- 172.21.15.89:0/2213450504 >> 172.21.15.89:0/2213450504 conn(0x7fd194101530 msgr2=0x7fd1941020c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:51.613 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.247+0000 7fd199d64640 1 -- 172.21.15.89:0/2213450504 shutdown_connections 2024-09-15T23:14:51.613 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:50.247+0000 7fd199d64640 1 -- 172.21.15.89:0/2213450504 wait complete. 2024-09-15T23:14:53.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:53 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2101153145' entity='client.admin' 2024-09-15T23:14:53.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:53 smithi089 ceph-mon[30933]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.006+0000 7f9d1868f640 1 Processor -- start 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.006+0000 7f9d1868f640 1 -- start start 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.006+0000 7f9d1868f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d10107b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.006+0000 7f9d1868f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9d101080f0 con 0x7f9d10107720 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.006+0000 7f9d1642b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d10107b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.006+0000 7f9d1642b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d10107b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51812/0 (socket says 172.21.15.89:51812) 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.006+0000 7f9d1642b640 1 -- 172.21.15.89:0/333584934 learned_addr learned my addr 172.21.15.89:0/333584934 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.006+0000 7f9d1642b640 1 -- 172.21.15.89:0/333584934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d10108920 con 0x7f9d10107720 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.007+0000 7f9d1642b640 1 --2- 172.21.15.89:0/333584934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d10107b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009820 tx=0x7f9d0c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e82a2baeb32864 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.007+0000 7f9d15429640 1 -- 172.21.15.89:0/333584934 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d0c037070 con 0x7f9d10107720 2024-09-15T23:14:53.333 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.007+0000 7f9d15429640 1 -- 172.21.15.89:0/333584934 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c032920 con 0x7f9d10107720 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.007+0000 7f9d15429640 1 -- 172.21.15.89:0/333584934 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d0c030810 con 0x7f9d10107720 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.008+0000 7f9d1868f640 1 -- 172.21.15.89:0/333584934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 msgr2=0x7f9d10107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.008+0000 7f9d1868f640 1 --2- 172.21.15.89:0/333584934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d10107b20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009820 tx=0x7f9d0c02a270 comp rx=0 tx=0).stop 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.008+0000 7f9d1868f640 1 -- 172.21.15.89:0/333584934 shutdown_connections 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.008+0000 7f9d1868f640 1 --2- 172.21.15.89:0/333584934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d10107b20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.008+0000 7f9d1868f640 1 -- 172.21.15.89:0/333584934 >> 172.21.15.89:0/333584934 conn(0x7f9d10102ed0 msgr2=0x7f9d101052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.008+0000 7f9d1868f640 1 -- 172.21.15.89:0/333584934 shutdown_connections 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.008+0000 7f9d1868f640 1 -- 172.21.15.89:0/333584934 wait complete. 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d1868f640 1 Processor -- start 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d1868f640 1 -- start start 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d1868f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d1019b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d1868f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1010bcd0 con 0x7f9d10107720 2024-09-15T23:14:53.334 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d1642b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d1019b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d1642b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d1019b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51826/0 (socket says 172.21.15.89:51826) 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d1642b640 1 -- 172.21.15.89:0/2101153145 learned_addr learned my addr 172.21.15.89:0/2101153145 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d1642b640 1 -- 172.21.15.89:0/2101153145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1019bc80 con 0x7f9d10107720 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d1642b640 1 --2- 172.21.15.89:0/2101153145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d1019b740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c030560 tx=0x7f9d0c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d077fe640 1 -- 172.21.15.89:0/2101153145 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d0c037040 con 0x7f9d10107720 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d077fe640 1 -- 172.21.15.89:0/2101153145 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c040070 con 0x7f9d10107720 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d1868f640 1 -- 172.21.15.89:0/2101153145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1019bf10 con 0x7f9d10107720 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.009+0000 7f9d077fe640 1 -- 172.21.15.89:0/2101153145 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d0c03a6d0 con 0x7f9d10107720 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.010+0000 7f9d1868f640 1 -- 172.21.15.89:0/2101153145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1019ec00 con 0x7f9d10107720 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.010+0000 7f9d077fe640 1 -- 172.21.15.89:0/2101153145 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f9d0c03b420 con 0x7f9d10107720 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.010+0000 7f9d077fe640 1 --2- 172.21.15.89:0/2101153145 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f9cfc0391b0 0x7f9cfc03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.011+0000 7f9d077fe640 1 -- 172.21.15.89:0/2101153145 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9d0c06c800 con 0x7f9d10107720 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.011+0000 7f9d1868f640 1 -- 172.21.15.89:0/2101153145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d10107ba0 con 0x7f9d10107720 2024-09-15T23:14:53.335 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.011+0000 7f9d15c2a640 1 --2- 172.21.15.89:0/2101153145 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f9cfc0391b0 0x7f9cfc03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.011+0000 7f9d15c2a640 1 --2- 172.21.15.89:0/2101153145 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f9cfc0391b0 0x7f9cfc03b670 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9d000098c0 tx=0x7f9d00006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.015+0000 7f9d077fe640 1 -- 172.21.15.89:0/2101153145 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9d0c04a200 con 0x7f9d10107720 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.202+0000 7f9d1868f640 1 -- 172.21.15.89:0/2101153145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f9d1019f120 con 0x7f9d10107720 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.204+0000 7f9d077fe640 1 -- 172.21.15.89:0/2101153145 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f9d0c031e80 con 0x7f9d10107720 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.207+0000 7f9d1868f640 1 -- 172.21.15.89:0/2101153145 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f9cfc0391b0 msgr2=0x7f9cfc03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.207+0000 7f9d1868f640 1 --2- 172.21.15.89:0/2101153145 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f9cfc0391b0 0x7f9cfc03b670 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9d000098c0 tx=0x7f9d00006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.207+0000 7f9d1868f640 1 -- 172.21.15.89:0/2101153145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 msgr2=0x7f9d1019b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.207+0000 7f9d1868f640 1 --2- 172.21.15.89:0/2101153145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d1019b740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c030560 tx=0x7f9d0c032670 comp rx=0 tx=0).stop 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.207+0000 7f9d1868f640 1 -- 172.21.15.89:0/2101153145 shutdown_connections 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.207+0000 7f9d1868f640 1 --2- 172.21.15.89:0/2101153145 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f9cfc0391b0 0x7f9cfc03b670 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.207+0000 7f9d1868f640 1 --2- 172.21.15.89:0/2101153145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d10107720 0x7f9d1019b740 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.207+0000 7f9d1868f640 1 -- 172.21.15.89:0/2101153145 >> 172.21.15.89:0/2101153145 conn(0x7f9d10102ed0 msgr2=0x7f9d10103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.207+0000 7f9d1868f640 1 -- 172.21.15.89:0/2101153145 shutdown_connections 2024-09-15T23:14:53.337 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:52.207+0000 7f9d1868f640 1 -- 172.21.15.89:0/2101153145 wait complete. 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.749+0000 7f27fd297640 1 Processor -- start 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.749+0000 7f27fd297640 1 -- start start 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.749+0000 7f27fd297640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f8107ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.749+0000 7f27fd297640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f27f81080b0 con 0x7f27f81076e0 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.749+0000 7f27f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f8107ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.749+0000 7f27f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f8107ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51840/0 (socket says 172.21.15.89:51840) 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.749+0000 7f27f6ffd640 1 -- 172.21.15.89:0/2591831762 learned_addr learned my addr 172.21.15.89:0/2591831762 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.749+0000 7f27f6ffd640 1 -- 172.21.15.89:0/2591831762 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f81088e0 con 0x7f27f81076e0 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.750+0000 7f27f6ffd640 1 --2- 172.21.15.89:0/2591831762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f8107ae0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f27e8009e80 tx=0x7f27e802a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12ef7368017c050 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.750+0000 7f27f5ffb640 1 -- 172.21.15.89:0/2591831762 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27e8036070 con 0x7f27f81076e0 2024-09-15T23:14:54.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.750+0000 7f27f5ffb640 1 -- 172.21.15.89:0/2591831762 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e80318e0 con 0x7f27f81076e0 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.750+0000 7f27f5ffb640 1 -- 172.21.15.89:0/2591831762 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27e8007a00 con 0x7f27f81076e0 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.751+0000 7f27fd297640 1 -- 172.21.15.89:0/2591831762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 msgr2=0x7f27f8107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.751+0000 7f27fd297640 1 --2- 172.21.15.89:0/2591831762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f8107ae0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f27e8009e80 tx=0x7f27e802a7a0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.751+0000 7f27fd297640 1 -- 172.21.15.89:0/2591831762 shutdown_connections 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.751+0000 7f27fd297640 1 --2- 172.21.15.89:0/2591831762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f8107ae0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.751+0000 7f27fd297640 1 -- 172.21.15.89:0/2591831762 >> 172.21.15.89:0/2591831762 conn(0x7f27f8102e90 msgr2=0x7f27f81052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.751+0000 7f27fd297640 1 -- 172.21.15.89:0/2591831762 shutdown_connections 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.751+0000 7f27fd297640 1 -- 172.21.15.89:0/2591831762 wait complete. 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.751+0000 7f27fd297640 1 Processor -- start 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27fd297640 1 -- start start 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27fd297640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f819b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27fd297640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f27f810bc90 con 0x7f27f81076e0 2024-09-15T23:14:54.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f819b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f819b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51852/0 (socket says 172.21.15.89:51852) 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27f6ffd640 1 -- 172.21.15.89:0/2341014950 learned_addr learned my addr 172.21.15.89:0/2341014950 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27f6ffd640 1 -- 172.21.15.89:0/2341014950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f819bca0 con 0x7f27f81076e0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27f6ffd640 1 --2- 172.21.15.89:0/2341014950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f819b760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f27e8031cc0 tx=0x7f27e8031cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27effff640 1 -- 172.21.15.89:0/2341014950 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27e8036050 con 0x7f27f81076e0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27effff640 1 -- 172.21.15.89:0/2341014950 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e8042070 con 0x7f27f81076e0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.752+0000 7f27fd297640 1 -- 172.21.15.89:0/2341014950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f819bf30 con 0x7f27f81076e0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.753+0000 7f27effff640 1 -- 172.21.15.89:0/2341014950 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27e803d9b0 con 0x7f27f81076e0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.753+0000 7f27fd297640 1 -- 172.21.15.89:0/2341014950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f819ec20 con 0x7f27f81076e0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.753+0000 7f27effff640 1 -- 172.21.15.89:0/2341014950 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f27e803dcb0 con 0x7f27f81076e0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.753+0000 7f27effff640 1 --2- 172.21.15.89:0/2341014950 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f27e40391b0 0x7f27e403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.753+0000 7f27fd297640 1 -- 172.21.15.89:0/2341014950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f810be10 con 0x7f27f81076e0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.754+0000 7f27effff640 1 -- 172.21.15.89:0/2341014950 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f27e806d6e0 con 0x7f27f81076e0 2024-09-15T23:14:54.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.754+0000 7f27f67fc640 1 --2- 172.21.15.89:0/2341014950 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f27e40391b0 0x7f27e403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.754+0000 7f27f67fc640 1 --2- 172.21.15.89:0/2341014950 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f27e40391b0 0x7f27e403b670 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f27e00098c0 tx=0x7f27e0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.758+0000 7f27effff640 1 -- 172.21.15.89:0/2341014950 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f27e8044310 con 0x7f27f81076e0 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.948+0000 7f27fd297640 1 -- 172.21.15.89:0/2341014950 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f27f8107b60 con 0x7f27f81076e0 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.951+0000 7f27effff640 1 -- 172.21.15.89:0/2341014950 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f27e8039300 con 0x7f27f81076e0 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.954+0000 7f27fd297640 1 -- 172.21.15.89:0/2341014950 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f27e40391b0 msgr2=0x7f27e403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.954+0000 7f27fd297640 1 --2- 172.21.15.89:0/2341014950 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f27e40391b0 0x7f27e403b670 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f27e00098c0 tx=0x7f27e0006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.954+0000 7f27fd297640 1 -- 172.21.15.89:0/2341014950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 msgr2=0x7f27f819b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.954+0000 7f27fd297640 1 --2- 172.21.15.89:0/2341014950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f819b760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f27e8031cc0 tx=0x7f27e8031cf0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.954+0000 7f27fd297640 1 -- 172.21.15.89:0/2341014950 shutdown_connections 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.954+0000 7f27fd297640 1 --2- 172.21.15.89:0/2341014950 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f27e40391b0 0x7f27e403b670 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.954+0000 7f27fd297640 1 --2- 172.21.15.89:0/2341014950 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f27f81076e0 0x7f27f819b760 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.954+0000 7f27fd297640 1 -- 172.21.15.89:0/2341014950 >> 172.21.15.89:0/2341014950 conn(0x7f27f8102e90 msgr2=0x7f27f8103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:54.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.954+0000 7f27fd297640 1 -- 172.21.15.89:0/2341014950 shutdown_connections 2024-09-15T23:14:54.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:53.954+0000 7f27fd297640 1 -- 172.21.15.89:0/2341014950 wait complete. 2024-09-15T23:14:54.577 INFO:teuthology.orchestra.run.smithi089.stdout:Enabling the dashboard module... 2024-09-15T23:14:55.156 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:54 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:55.156 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:54 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2341014950' entity='client.admin' 2024-09-15T23:14:55.156 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:54 smithi089 ceph-mon[30933]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:14:55.157 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:54 smithi089 ceph-mon[30933]: from='mgr.14118 172.21.15.89:0/953684906' entity='mgr.x' 2024-09-15T23:14:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:55 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1409883425' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-15T23:14:56.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: ignoring --setuser ceph since I am not root 2024-09-15T23:14:56.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: ignoring --setgroup ceph since I am not root 2024-09-15T23:14:56.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:56.102+0000 7f8b4fcd5040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T23:14:56.311 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:56.160+0000 7f8b4fcd5040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T23:14:56.637 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:56.311+0000 7f8b4fcd5040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T23:14:56.637 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:56.378+0000 7f8b4fcd5040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T23:14:56.637 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:56.526+0000 7f8b4fcd5040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:14:56.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.000+0000 7fcae15e8640 1 Processor -- start 2024-09-15T23:14:56.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.000+0000 7fcae15e8640 1 -- start start 2024-09-15T23:14:56.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.000+0000 7fcae15e8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc10a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:56.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.000+0000 7fcae15e8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc07be70 con 0x7fcadc108170 2024-09-15T23:14:56.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.000+0000 7fcadaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc10a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:56.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.000+0000 7fcadaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc10a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51862/0 (socket says 172.21.15.89:51862) 2024-09-15T23:14:56.644 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.000+0000 7fcadaffd640 1 -- 172.21.15.89:0/2127681706 learned_addr learned my addr 172.21.15.89:0/2127681706 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.001+0000 7fcadaffd640 1 -- 172.21.15.89:0/2127681706 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcadc10aaa0 con 0x7fcadc108170 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.001+0000 7fcadaffd640 1 --2- 172.21.15.89:0/2127681706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc10a560 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009820 tx=0x7fcacc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ae79eb42d524b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.002+0000 7fcada7fc640 1 -- 172.21.15.89:0/2127681706 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcacc037070 con 0x7fcadc108170 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.002+0000 7fcada7fc640 1 -- 172.21.15.89:0/2127681706 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcacc032920 con 0x7fcadc108170 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.002+0000 7fcada7fc640 1 -- 172.21.15.89:0/2127681706 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcacc030810 con 0x7fcadc108170 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.002+0000 7fcae15e8640 1 -- 172.21.15.89:0/2127681706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 msgr2=0x7fcadc10a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.002+0000 7fcae15e8640 1 --2- 172.21.15.89:0/2127681706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc10a560 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009820 tx=0x7fcacc02a270 comp rx=0 tx=0).stop 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.002+0000 7fcae15e8640 1 -- 172.21.15.89:0/2127681706 shutdown_connections 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.002+0000 7fcae15e8640 1 --2- 172.21.15.89:0/2127681706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc10a560 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.002+0000 7fcae15e8640 1 -- 172.21.15.89:0/2127681706 >> 172.21.15.89:0/2127681706 conn(0x7fcadc101d50 msgr2=0x7fcadc104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.002+0000 7fcae15e8640 1 -- 172.21.15.89:0/2127681706 shutdown_connections 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.002+0000 7fcae15e8640 1 -- 172.21.15.89:0/2127681706 wait complete. 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.003+0000 7fcae15e8640 1 Processor -- start 2024-09-15T23:14:56.645 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.003+0000 7fcae15e8640 1 -- start start 2024-09-15T23:14:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.003+0000 7fcae15e8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc19b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.003+0000 7fcae15e8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc10bb20 con 0x7fcadc108170 2024-09-15T23:14:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.003+0000 7fcadaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc19b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.004+0000 7fcadaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc19b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51872/0 (socket says 172.21.15.89:51872) 2024-09-15T23:14:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.004+0000 7fcadaffd640 1 -- 172.21.15.89:0/1409883425 learned_addr learned my addr 172.21.15.89:0/1409883425 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.004+0000 7fcadaffd640 1 -- 172.21.15.89:0/1409883425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcadc19bc60 con 0x7fcadc108170 2024-09-15T23:14:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.004+0000 7fcadaffd640 1 --2- 172.21.15.89:0/1409883425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc19b720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcacc030420 tx=0x7fcacc032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.004+0000 7fcad8ff9640 1 -- 172.21.15.89:0/1409883425 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcacc037040 con 0x7fcadc108170 2024-09-15T23:14:56.646 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.004+0000 7fcad8ff9640 1 -- 172.21.15.89:0/1409883425 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcacc040070 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.004+0000 7fcae15e8640 1 -- 172.21.15.89:0/1409883425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcadc19bef0 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.004+0000 7fcad8ff9640 1 -- 172.21.15.89:0/1409883425 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcacc03a6e0 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.004+0000 7fcae15e8640 1 -- 172.21.15.89:0/1409883425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcadc19ebe0 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.005+0000 7fcad8ff9640 1 -- 172.21.15.89:0/1409883425 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fcacc03a990 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.005+0000 7fcae15e8640 1 -- 172.21.15.89:0/1409883425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcadc1085d0 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.005+0000 7fcad8ff9640 1 --2- 172.21.15.89:0/1409883425 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcac8039200 0x7fcac803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.006+0000 7fcad8ff9640 1 -- 172.21.15.89:0/1409883425 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcacc06c960 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.006+0000 7fcad3fff640 1 --2- 172.21.15.89:0/1409883425 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcac8039200 0x7fcac803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.006+0000 7fcad3fff640 1 --2- 172.21.15.89:0/1409883425 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcac8039200 0x7fcac803b6c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcac40098c0 tx=0x7fcac4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.011+0000 7fcad8ff9640 1 -- 172.21.15.89:0/1409883425 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcacc044c70 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.235+0000 7fcae15e8640 1 -- 172.21.15.89:0/1409883425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fcadc19edb0 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.954+0000 7fcad8ff9640 1 -- 172.21.15.89:0/1409883425 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fcacc0328c0 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.955+0000 7fcad8ff9640 1 -- 172.21.15.89:0/1409883425 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7fcacc03bb20 con 0x7fcadc108170 2024-09-15T23:14:56.647 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.957+0000 7fcae15e8640 1 -- 172.21.15.89:0/1409883425 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcac8039200 msgr2=0x7fcac803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:56.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.957+0000 7fcae15e8640 1 --2- 172.21.15.89:0/1409883425 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcac8039200 0x7fcac803b6c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcac40098c0 tx=0x7fcac4006db0 comp rx=0 tx=0).stop 2024-09-15T23:14:56.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.957+0000 7fcae15e8640 1 -- 172.21.15.89:0/1409883425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 msgr2=0x7fcadc19b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:56.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.957+0000 7fcae15e8640 1 --2- 172.21.15.89:0/1409883425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc19b720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcacc030420 tx=0x7fcacc032670 comp rx=0 tx=0).stop 2024-09-15T23:14:56.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.957+0000 7fcae15e8640 1 -- 172.21.15.89:0/1409883425 shutdown_connections 2024-09-15T23:14:56.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.957+0000 7fcae15e8640 1 --2- 172.21.15.89:0/1409883425 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcac8039200 0x7fcac803b6c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:56.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.957+0000 7fcae15e8640 1 --2- 172.21.15.89:0/1409883425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcadc108170 0x7fcadc19b720 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:56.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.957+0000 7fcae15e8640 1 -- 172.21.15.89:0/1409883425 >> 172.21.15.89:0/1409883425 conn(0x7fcadc101d50 msgr2=0x7fcadc1029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:56.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.957+0000 7fcae15e8640 1 -- 172.21.15.89:0/1409883425 shutdown_connections 2024-09-15T23:14:56.648 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:55.957+0000 7fcae15e8640 1 -- 172.21.15.89:0/1409883425 wait complete. 2024-09-15T23:14:56.887 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:56.671+0000 7f8b4fcd5040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T23:14:57.252 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1409883425' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-15T23:14:57.252 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-mon[30933]: mgrmap e9: x(active, since 23s) 2024-09-15T23:14:57.252 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:56.995+0000 7f8b4fcd5040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T23:14:57.253 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:57 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:57.060+0000 7f8b4fcd5040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T23:14:57.253 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:57 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:57.251+0000 7f8b4fcd5040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T23:14:57.952 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:14:57.952 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-15T23:14:57.952 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-15T23:14:57.952 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-15T23:14:57.952 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.051+0000 7f4435360640 1 Processor -- start 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.051+0000 7f4435360640 1 -- start start 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.051+0000 7f4435360640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f4430107b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.051+0000 7f4435360640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f44301080f0 con 0x7f4430107720 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.051+0000 7f442effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f4430107b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.051+0000 7f442effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f4430107b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46736/0 (socket says 172.21.15.89:46736) 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.051+0000 7f442effd640 1 -- 172.21.15.89:0/400658197 learned_addr learned my addr 172.21.15.89:0/400658197 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.052+0000 7f442effd640 1 -- 172.21.15.89:0/400658197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4430108920 con 0x7f4430107720 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.052+0000 7f442effd640 1 --2- 172.21.15.89:0/400658197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f4430107b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4420009820 tx=0x7f442002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7064f18ea4cb7a39 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.052+0000 7f442dffb640 1 -- 172.21.15.89:0/400658197 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4420037070 con 0x7f4430107720 2024-09-15T23:14:57.953 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.052+0000 7f442dffb640 1 -- 172.21.15.89:0/400658197 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4420032920 con 0x7f4430107720 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.052+0000 7f442dffb640 1 -- 172.21.15.89:0/400658197 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4420030810 con 0x7f4430107720 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.053+0000 7f4435360640 1 -- 172.21.15.89:0/400658197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 msgr2=0x7f4430107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.053+0000 7f4435360640 1 --2- 172.21.15.89:0/400658197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f4430107b20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4420009820 tx=0x7f442002a270 comp rx=0 tx=0).stop 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.053+0000 7f4435360640 1 -- 172.21.15.89:0/400658197 shutdown_connections 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.053+0000 7f4435360640 1 --2- 172.21.15.89:0/400658197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f4430107b20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.053+0000 7f4435360640 1 -- 172.21.15.89:0/400658197 >> 172.21.15.89:0/400658197 conn(0x7f4430102ed0 msgr2=0x7f44301052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.053+0000 7f4435360640 1 -- 172.21.15.89:0/400658197 shutdown_connections 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.053+0000 7f4435360640 1 -- 172.21.15.89:0/400658197 wait complete. 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.054+0000 7f4435360640 1 Processor -- start 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.054+0000 7f4435360640 1 -- start start 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.054+0000 7f4435360640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f443007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.054+0000 7f4435360640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f443010bcd0 con 0x7f4430107720 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.054+0000 7f442effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f443007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:57.954 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.054+0000 7f442effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f443007db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46746/0 (socket says 172.21.15.89:46746) 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.054+0000 7f442effd640 1 -- 172.21.15.89:0/3651122211 learned_addr learned my addr 172.21.15.89:0/3651122211 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.054+0000 7f442effd640 1 -- 172.21.15.89:0/3651122211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443007e0d0 con 0x7f4430107720 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.055+0000 7f442effd640 1 --2- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f443007db90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f44200097f0 tx=0x7f44200327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.055+0000 7f4417fff640 1 -- 172.21.15.89:0/3651122211 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4420037040 con 0x7f4430107720 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.055+0000 7f4417fff640 1 -- 172.21.15.89:0/3651122211 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4420030c40 con 0x7f4430107720 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.055+0000 7f4435360640 1 -- 172.21.15.89:0/3651122211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443007c2c0 con 0x7f4430107720 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.055+0000 7f4417fff640 1 -- 172.21.15.89:0/3651122211 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f442003a5b0 con 0x7f4430107720 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.055+0000 7f4435360640 1 -- 172.21.15.89:0/3651122211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f443007c800 con 0x7f4430107720 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.056+0000 7f4417fff640 1 -- 172.21.15.89:0/3651122211 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f442003a7f0 con 0x7f4430107720 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.056+0000 7f4435360640 1 -- 172.21.15.89:0/3651122211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4430107ba0 con 0x7f4430107720 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.056+0000 7f4417fff640 1 --2- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4418039200 0x7f441803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.056+0000 7f4417fff640 1 -- 172.21.15.89:0/3651122211 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f442006c720 con 0x7f4430107720 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.056+0000 7f442e7fc640 1 -- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4418039200 msgr2=0x7f441803b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/429862840 2024-09-15T23:14:57.955 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.056+0000 7f442e7fc640 1 --2- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4418039200 0x7f441803b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.061+0000 7f4417fff640 1 -- 172.21.15.89:0/3651122211 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4420030df0 con 0x7f4430107720 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.257+0000 7f442e7fc640 1 -- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4418039200 msgr2=0x7f441803b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/429862840 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.257+0000 7f442e7fc640 1 --2- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4418039200 0x7f441803b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.283+0000 7f4435360640 1 -- 172.21.15.89:0/3651122211 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f443007ca80 con 0x7f4430107720 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.283+0000 7f4417fff640 1 -- 172.21.15.89:0/3651122211 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f4420031c00 con 0x7f4430107720 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.286+0000 7f4435360640 1 -- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4418039200 msgr2=0x7f441803b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.286+0000 7f4435360640 1 --2- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4418039200 0x7f441803b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.286+0000 7f4435360640 1 -- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 msgr2=0x7f443007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.286+0000 7f4435360640 1 --2- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f443007db90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f44200097f0 tx=0x7f44200327b0 comp rx=0 tx=0).stop 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.286+0000 7f4435360640 1 -- 172.21.15.89:0/3651122211 shutdown_connections 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.286+0000 7f4435360640 1 --2- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7f4418039200 0x7f441803b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.286+0000 7f4435360640 1 --2- 172.21.15.89:0/3651122211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430107720 0x7f443007db90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.286+0000 7f4435360640 1 -- 172.21.15.89:0/3651122211 >> 172.21.15.89:0/3651122211 conn(0x7f4430102ed0 msgr2=0x7f4430103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:57.956 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.286+0000 7f4435360640 1 -- 172.21.15.89:0/3651122211 shutdown_connections 2024-09-15T23:14:57.957 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:57.286+0000 7f4435360640 1 -- 172.21.15.89:0/3651122211 wait complete. 2024-09-15T23:14:57.957 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for the mgr to restart... 2024-09-15T23:14:57.957 INFO:teuthology.orchestra.run.smithi089.stdout:Waiting for mgr epoch 9... 2024-09-15T23:14:58.230 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:14:57 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3651122211' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-15T23:14:58.230 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:58 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:58.158+0000 7f8b4fcd5040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T23:14:58.230 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:58 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:58.228+0000 7f8b4fcd5040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T23:14:59.210 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:58 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:58.950+0000 7f8b4fcd5040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T23:14:59.210 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:59.066+0000 7f8b4fcd5040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T23:14:59.501 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:59.208+0000 7f8b4fcd5040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:14:59.501 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:59.272+0000 7f8b4fcd5040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T23:14:59.501 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:59.428+0000 7f8b4fcd5040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T23:14:59.501 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:59.499+0000 7f8b4fcd5040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T23:14:59.758 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:59.563+0000 7f8b4fcd5040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T23:14:59.759 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:59.730+0000 7f8b4fcd5040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T23:15:00.018 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:59.799+0000 7f8b4fcd5040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T23:15:00.018 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:59.868+0000 7f8b4fcd5040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T23:15:00.018 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:14:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:14:59.955+0000 7f8b4fcd5040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T23:15:00.018 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:15:00.016+0000 7f8b4fcd5040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T23:15:00.310 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:15:00.078+0000 7f8b4fcd5040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T23:15:01.310 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:15:00.882+0000 7f8b4fcd5040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T23:15:01.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: Active manager daemon x restarted 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: Activating manager daemon x 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: osdmap e3: 0 total, 0 up, 0 in 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: mgrmap e10: x(active, starting, since 0.00339566s) 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: Manager daemon x is now available 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:01.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T23:15:01.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T23:15:02.379 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:15:02.120+0000 7f8abafe9640 -1 log_channel(cephadm) log [ERR] : [15/Sep/2024:23:15:02] ENGINE Error in HTTPServer.serve 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: Traceback (most recent call last): 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: self._connections.run(self.expiration_interval) 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: self._run(expiration_interval) 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: s = self.context.wrap_socket( 2024-09-15T23:15:02.405 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T23:15:02.406 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: return self.sslsocket_class._create( 2024-09-15T23:15:02.406 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T23:15:02.406 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: self.do_handshake() 2024-09-15T23:15:02.406 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T23:15:02.406 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: self._sslobj.do_handshake() 2024-09-15T23:15:02.406 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T23:15:02.406 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:15:02.621 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:15:02.621 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-15T23:15:02.621 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-15T23:15:02.621 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:15:02.621 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.349+0000 7fcdbc654640 1 Processor -- start 2024-09-15T23:15:02.621 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.349+0000 7fcdbc654640 1 -- start start 2024-09-15T23:15:02.621 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.349+0000 7fcdbc654640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb407bd10 0x7fcdb410a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:02.621 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.349+0000 7fcdbc654640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb410a840 con 0x7fcdb407bd10 2024-09-15T23:15:02.621 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.350+0000 7fcdba3f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb407bd10 0x7fcdb410a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.350+0000 7fcdba3f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb407bd10 0x7fcdb410a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46750/0 (socket says 172.21.15.89:46750) 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.350+0000 7fcdba3f0640 1 -- 172.21.15.89:0/2338149013 learned_addr learned my addr 172.21.15.89:0/2338149013 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.350+0000 7fcdba3f0640 1 -- 172.21.15.89:0/2338149013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb410a9c0 con 0x7fcdb407bd10 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.351+0000 7fcdba3f0640 1 --2- 172.21.15.89:0/2338149013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb407bd10 0x7fcdb410a270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009820 tx=0x7fcdb002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd3f382a8e0cf2b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.351+0000 7fcdb93ee640 1 -- 172.21.15.89:0/2338149013 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdb0037070 con 0x7fcdb407bd10 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.351+0000 7fcdb93ee640 1 -- 172.21.15.89:0/2338149013 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdb0032920 con 0x7fcdb407bd10 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.351+0000 7fcdb93ee640 1 -- 172.21.15.89:0/2338149013 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdb0030810 con 0x7fcdb407bd10 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 -- 172.21.15.89:0/2338149013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb407bd10 msgr2=0x7fcdb410a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 --2- 172.21.15.89:0/2338149013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb407bd10 0x7fcdb410a270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009820 tx=0x7fcdb002a270 comp rx=0 tx=0).stop 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 -- 172.21.15.89:0/2338149013 shutdown_connections 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 --2- 172.21.15.89:0/2338149013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb407bd10 0x7fcdb410a270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 -- 172.21.15.89:0/2338149013 >> 172.21.15.89:0/2338149013 conn(0x7fcdb4101da0 msgr2=0x7fcdb41041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:02.622 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 -- 172.21.15.89:0/2338149013 shutdown_connections 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 -- 172.21.15.89:0/2338149013 wait complete. 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 Processor -- start 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 -- start start 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb419b6b0 0x7fcdb419bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.352+0000 7fcdbc654640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb410bb00 con 0x7fcdb419b6b0 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.353+0000 7fcdba3f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb419b6b0 0x7fcdb419bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.353+0000 7fcdba3f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb419b6b0 0x7fcdb419bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46766/0 (socket says 172.21.15.89:46766) 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.353+0000 7fcdba3f0640 1 -- 172.21.15.89:0/1566351869 learned_addr learned my addr 172.21.15.89:0/1566351869 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.353+0000 7fcdba3f0640 1 -- 172.21.15.89:0/1566351869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb419c010 con 0x7fcdb419b6b0 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.353+0000 7fcdba3f0640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb419b6b0 0x7fcdb419bad0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcdb00097f0 tx=0x7fcdb00327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.353+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdb0037040 con 0x7fcdb419b6b0 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.353+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdb0030c40 con 0x7fcdb419b6b0 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.353+0000 7fcdbc654640 1 -- 172.21.15.89:0/1566351869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb419c2a0 con 0x7fcdb419b6b0 2024-09-15T23:15:02.623 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.353+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdb003a5d0 con 0x7fcdb419b6b0 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.353+0000 7fcdbc654640 1 -- 172.21.15.89:0/1566351869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb419ed80 con 0x7fcdb419b6b0 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.354+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7fcdb003a7c0 con 0x7fcdb419b6b0 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.354+0000 7fcdab7fe640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 0x7fcda003b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.354+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 --> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcda003be30 con 0x7fcda0039260 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.354+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcdb006c8c0 con 0x7fcdb419b6b0 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.354+0000 7fcdb9bef640 1 -- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 msgr2=0x7fcda003b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/429862840 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.354+0000 7fcdb9bef640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 0x7fcda003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.555+0000 7fcdb9bef640 1 -- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 msgr2=0x7fcda003b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/429862840 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.555+0000 7fcdb9bef640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 0x7fcda003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.956+0000 7fcdb9bef640 1 -- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 msgr2=0x7fcda003b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/429862840 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:58.956+0000 7fcdb9bef640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 0x7fcda003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:59.757+0000 7fcdb9bef640 1 -- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 msgr2=0x7fcda003b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.89:6800/429862840 2024-09-15T23:15:02.624 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:14:59.757+0000 7fcdb9bef640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 0x7fcda003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:00.885+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mgrmap(e 10) ==== 46241+0+0 (secure 0 0 0) 0x7fcdb006b370 con 0x7fcdb419b6b0 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:00.885+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 msgr2=0x7fcda003b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:00.885+0000 7fcdab7fe640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 0x7fcda003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.887+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7fcdb0044920 con 0x7fcdb419b6b0 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.887+0000 7fcdab7fe640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcda003cd20 0x7fcda003f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.887+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcdb0043e50 con 0x7fcda003cd20 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.888+0000 7fcdb9bef640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcda003cd20 0x7fcda003f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.888+0000 7fcdb9bef640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcda003cd20 0x7fcda003f110 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcda4003980 tx=0x7fcda40092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.889+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fcdb0043e50 con 0x7fcda003cd20 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.896+0000 7fcdbc654640 1 -- 172.21.15.89:0/1566351869 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fcdb41082e0 con 0x7fcda003cd20 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdab7fe640 1 -- 172.21.15.89:0/1566351869 <== mgr.14150 v2:172.21.15.89:6800/3594493718 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fcdb41082e0 con 0x7fcda003cd20 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 -- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcda003cd20 msgr2=0x7fcda003f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcda003cd20 0x7fcda003f110 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcda4003980 tx=0x7fcda40092b0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.625 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 -- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb419b6b0 msgr2=0x7fcdb419bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb419b6b0 0x7fcdb419bad0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcdb00097f0 tx=0x7fcdb00327b0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 -- 172.21.15.89:0/1566351869 shutdown_connections 2024-09-15T23:15:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcda003cd20 0x7fcda003f110 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:6800/429862840,v1:172.21.15.89:6801/429862840] conn(0x7fcda0039260 0x7fcda003b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 --2- 172.21.15.89:0/1566351869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdb419b6b0 0x7fcdb419bad0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 -- 172.21.15.89:0/1566351869 >> 172.21.15.89:0/1566351869 conn(0x7fcdb4101da0 msgr2=0x7fcdb41029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 -- 172.21.15.89:0/1566351869 shutdown_connections 2024-09-15T23:15:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:01.897+0000 7fcdbc654640 1 -- 172.21.15.89:0/1566351869 wait complete. 2024-09-15T23:15:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:mgr epoch 9 is available 2024-09-15T23:15:02.626 INFO:teuthology.orchestra.run.smithi089.stdout:Generating a dashboard self-signed certificate... 2024-09-15T23:15:02.991 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-mon[30933]: mgrmap e11: x(active, since 1.00597s) 2024-09-15T23:15:02.991 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-mon[30933]: [15/Sep/2024:23:15:02] ENGINE Bus STARTING 2024-09-15T23:15:02.991 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:03.961 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:03 smithi089 ceph-mon[30933]: [15/Sep/2024:23:15:02] ENGINE Serving on https://172.21.15.89:7150 2024-09-15T23:15:03.961 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:03 smithi089 ceph-mon[30933]: [15/Sep/2024:23:15:02] ENGINE Error in HTTPServer.serve 2024-09-15T23:15:03.961 INFO:journalctl@ceph.mon.a.smithi089.stdout: Traceback (most recent call last): 2024-09-15T23:15:03.961 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T23:15:03.961 INFO:journalctl@ceph.mon.a.smithi089.stdout: self._connections.run(self.expiration_interval) 2024-09-15T23:15:03.961 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: self._run(expiration_interval) 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: s = self.context.wrap_socket( 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: return self.sslsocket_class._create( 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: self.do_handshake() 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: self._sslobj.do_handshake() 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T23:15:03.962 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:03 smithi089 ceph-mon[30933]: [15/Sep/2024:23:15:02] ENGINE Serving on http://172.21.15.89:8765 2024-09-15T23:15:03.963 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:03 smithi089 ceph-mon[30933]: [15/Sep/2024:23:15:02] ENGINE Bus STARTED 2024-09-15T23:15:03.963 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:03.963 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:03 smithi089 ceph-mon[30933]: mgrmap e12: x(active, since 2s) 2024-09-15T23:15:03.963 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:03.963 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:04.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-15T23:15:04.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.071+0000 7fdf52c61640 1 Processor -- start 2024-09-15T23:15:04.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.071+0000 7fdf52c61640 1 -- start start 2024-09-15T23:15:04.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.071+0000 7fdf52c61640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c10a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:04.573 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.071+0000 7fdf52c61640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c07bed0 con 0x7fdf4c1080f0 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.072+0000 7fdf509fd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c10a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.072+0000 7fdf509fd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c10a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46832/0 (socket says 172.21.15.89:46832) 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.072+0000 7fdf509fd640 1 -- 172.21.15.89:0/3616817140 learned_addr learned my addr 172.21.15.89:0/3616817140 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.072+0000 7fdf509fd640 1 -- 172.21.15.89:0/3616817140 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4c10aa20 con 0x7fdf4c1080f0 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.073+0000 7fdf509fd640 1 --2- 172.21.15.89:0/3616817140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c10a4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009820 tx=0x7fdf4002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e615795d8172d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.073+0000 7fdf4b7fe640 1 -- 172.21.15.89:0/3616817140 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf40037070 con 0x7fdf4c1080f0 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.073+0000 7fdf4b7fe640 1 -- 172.21.15.89:0/3616817140 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf40032920 con 0x7fdf4c1080f0 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.073+0000 7fdf4b7fe640 1 -- 172.21.15.89:0/3616817140 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf40030810 con 0x7fdf4c1080f0 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.073+0000 7fdf52c61640 1 -- 172.21.15.89:0/3616817140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 msgr2=0x7fdf4c10a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.073+0000 7fdf52c61640 1 --2- 172.21.15.89:0/3616817140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c10a4e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009820 tx=0x7fdf4002a270 comp rx=0 tx=0).stop 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.073+0000 7fdf52c61640 1 -- 172.21.15.89:0/3616817140 shutdown_connections 2024-09-15T23:15:04.574 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.073+0000 7fdf52c61640 1 --2- 172.21.15.89:0/3616817140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c10a4e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.073+0000 7fdf52c61640 1 -- 172.21.15.89:0/3616817140 >> 172.21.15.89:0/3616817140 conn(0x7fdf4c101d10 msgr2=0x7fdf4c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.074+0000 7fdf52c61640 1 -- 172.21.15.89:0/3616817140 shutdown_connections 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.074+0000 7fdf52c61640 1 -- 172.21.15.89:0/3616817140 wait complete. 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.074+0000 7fdf52c61640 1 Processor -- start 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.074+0000 7fdf52c61640 1 -- start start 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.074+0000 7fdf52c61640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c19b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.074+0000 7fdf52c61640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c10bb00 con 0x7fdf4c1080f0 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.074+0000 7fdf509fd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c19b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.074+0000 7fdf509fd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c19b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46838/0 (socket says 172.21.15.89:46838) 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.074+0000 7fdf509fd640 1 -- 172.21.15.89:0/120557092 learned_addr learned my addr 172.21.15.89:0/120557092 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.075+0000 7fdf509fd640 1 -- 172.21.15.89:0/120557092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4c19bbd0 con 0x7fdf4c1080f0 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.075+0000 7fdf509fd640 1 --2- 172.21.15.89:0/120557092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c19b690 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdf400097f0 tx=0x7fdf400327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.075+0000 7fdf49ffb640 1 -- 172.21.15.89:0/120557092 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf40037040 con 0x7fdf4c1080f0 2024-09-15T23:15:04.575 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.075+0000 7fdf52c61640 1 -- 172.21.15.89:0/120557092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4c19be60 con 0x7fdf4c1080f0 2024-09-15T23:15:04.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.075+0000 7fdf49ffb640 1 -- 172.21.15.89:0/120557092 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf40030cc0 con 0x7fdf4c1080f0 2024-09-15T23:15:04.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.075+0000 7fdf49ffb640 1 -- 172.21.15.89:0/120557092 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf4003a6b0 con 0x7fdf4c1080f0 2024-09-15T23:15:04.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.075+0000 7fdf52c61640 1 -- 172.21.15.89:0/120557092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf4c19eb50 con 0x7fdf4c1080f0 2024-09-15T23:15:04.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.076+0000 7fdf49ffb640 1 -- 172.21.15.89:0/120557092 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7fdf4003a8a0 con 0x7fdf4c1080f0 2024-09-15T23:15:04.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.076+0000 7fdf52c61640 1 -- 172.21.15.89:0/120557092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf4c108550 con 0x7fdf4c1080f0 2024-09-15T23:15:04.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.077+0000 7fdf49ffb640 1 --2- 172.21.15.89:0/120557092 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf3c0390f0 0x7fdf3c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:04.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.077+0000 7fdf49ffb640 1 -- 172.21.15.89:0/120557092 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdf4006c940 con 0x7fdf4c1080f0 2024-09-15T23:15:04.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.077+0000 7fdf4bfff640 1 --2- 172.21.15.89:0/120557092 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf3c0390f0 0x7fdf3c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:04.576 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.077+0000 7fdf4bfff640 1 --2- 172.21.15.89:0/120557092 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf3c0390f0 0x7fdf3c03b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf380098c0 tx=0x7fdf38006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:04.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.081+0000 7fdf49ffb640 1 -- 172.21.15.89:0/120557092 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdf4003ace0 con 0x7fdf4c1080f0 2024-09-15T23:15:04.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.168+0000 7fdf49ffb640 1 -- 172.21.15.89:0/120557092 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7fdf400448d0 con 0x7fdf4c1080f0 2024-09-15T23:15:04.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.259+0000 7fdf52c61640 1 -- 172.21.15.89:0/120557092 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fdf4c103000 con 0x7fdf3c0390f0 2024-09-15T23:15:04.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.317+0000 7fdf49ffb640 1 -- 172.21.15.89:0/120557092 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fdf4c103000 con 0x7fdf3c0390f0 2024-09-15T23:15:04.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.319+0000 7fdf52c61640 1 -- 172.21.15.89:0/120557092 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf3c0390f0 msgr2=0x7fdf3c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:04.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.319+0000 7fdf52c61640 1 --2- 172.21.15.89:0/120557092 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf3c0390f0 0x7fdf3c03b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf380098c0 tx=0x7fdf38006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:04.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.319+0000 7fdf52c61640 1 -- 172.21.15.89:0/120557092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 msgr2=0x7fdf4c19b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:04.577 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.319+0000 7fdf52c61640 1 --2- 172.21.15.89:0/120557092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c19b690 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdf400097f0 tx=0x7fdf400327b0 comp rx=0 tx=0).stop 2024-09-15T23:15:04.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.319+0000 7fdf52c61640 1 -- 172.21.15.89:0/120557092 shutdown_connections 2024-09-15T23:15:04.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.319+0000 7fdf52c61640 1 --2- 172.21.15.89:0/120557092 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf3c0390f0 0x7fdf3c03b5b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:04.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.319+0000 7fdf52c61640 1 --2- 172.21.15.89:0/120557092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf4c1080f0 0x7fdf4c19b690 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:04.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.319+0000 7fdf52c61640 1 -- 172.21.15.89:0/120557092 >> 172.21.15.89:0/120557092 conn(0x7fdf4c101d10 msgr2=0x7fdf4c1028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:04.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.319+0000 7fdf52c61640 1 -- 172.21.15.89:0/120557092 shutdown_connections 2024-09-15T23:15:04.578 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:03.320+0000 7fdf52c61640 1 -- 172.21.15.89:0/120557092 wait complete. 2024-09-15T23:15:04.578 INFO:teuthology.orchestra.run.smithi089.stdout:Creating initial admin user... 2024-09-15T23:15:05.173 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:04 smithi089 ceph-mon[30933]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:06.032 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$3ukGEVcGvhXaSs9rSv/N7.TcpZD4LFmoYTm34QlzwUGCrcADtbNxO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726442105, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.978+0000 7f3404edb640 1 Processor -- start 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.978+0000 7f3404edb640 1 -- start start 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.978+0000 7f3404edb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f3400107ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.978+0000 7f3404edb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f34001080b0 con 0x7f34001076e0 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.978+0000 7f33fe59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f3400107ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.978+0000 7f33fe59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f3400107ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46842/0 (socket says 172.21.15.89:46842) 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.978+0000 7f33fe59c640 1 -- 172.21.15.89:0/1700893673 learned_addr learned my addr 172.21.15.89:0/1700893673 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.979+0000 7f33fe59c640 1 -- 172.21.15.89:0/1700893673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34001088e0 con 0x7f34001076e0 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.979+0000 7f33fe59c640 1 --2- 172.21.15.89:0/1700893673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f3400107ae0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f33f0009820 tx=0x7f33f002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01de4dbb4c44262 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.979+0000 7f33fdd9b640 1 -- 172.21.15.89:0/1700893673 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33f0037070 con 0x7f34001076e0 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.980+0000 7f33fdd9b640 1 -- 172.21.15.89:0/1700893673 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33f0032920 con 0x7f34001076e0 2024-09-15T23:15:06.033 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.980+0000 7f33fdd9b640 1 -- 172.21.15.89:0/1700893673 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33f0030810 con 0x7f34001076e0 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.980+0000 7f3404edb640 1 -- 172.21.15.89:0/1700893673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 msgr2=0x7f3400107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.980+0000 7f3404edb640 1 --2- 172.21.15.89:0/1700893673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f3400107ae0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f33f0009820 tx=0x7f33f002a270 comp rx=0 tx=0).stop 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.980+0000 7f3404edb640 1 -- 172.21.15.89:0/1700893673 shutdown_connections 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.980+0000 7f3404edb640 1 --2- 172.21.15.89:0/1700893673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f3400107ae0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.980+0000 7f3404edb640 1 -- 172.21.15.89:0/1700893673 >> 172.21.15.89:0/1700893673 conn(0x7f3400102e90 msgr2=0x7f34001052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.980+0000 7f3404edb640 1 -- 172.21.15.89:0/1700893673 shutdown_connections 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.980+0000 7f3404edb640 1 -- 172.21.15.89:0/1700893673 wait complete. 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.981+0000 7f3404edb640 1 Processor -- start 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.981+0000 7f3404edb640 1 -- start start 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.981+0000 7f3404edb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f340019b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.981+0000 7f3404edb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f340010bc90 con 0x7f34001076e0 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.981+0000 7f33fe59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f340019b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:06.034 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.981+0000 7f33fe59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f340019b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46854/0 (socket says 172.21.15.89:46854) 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.981+0000 7f33fe59c640 1 -- 172.21.15.89:0/1014854937 learned_addr learned my addr 172.21.15.89:0/1014854937 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.981+0000 7f33fe59c640 1 -- 172.21.15.89:0/1014854937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340019bca0 con 0x7f34001076e0 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.982+0000 7f33fe59c640 1 --2- 172.21.15.89:0/1014854937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f340019b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f33f0030560 tx=0x7f33f0032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.982+0000 7f33f77fe640 1 -- 172.21.15.89:0/1014854937 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33f0037040 con 0x7f34001076e0 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.982+0000 7f33f77fe640 1 -- 172.21.15.89:0/1014854937 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33f0040070 con 0x7f34001076e0 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.982+0000 7f3404edb640 1 -- 172.21.15.89:0/1014854937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340019bf30 con 0x7f34001076e0 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.982+0000 7f33f77fe640 1 -- 172.21.15.89:0/1014854937 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f33f003a6d0 con 0x7f34001076e0 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.982+0000 7f3404edb640 1 -- 172.21.15.89:0/1014854937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f340019ec20 con 0x7f34001076e0 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.983+0000 7f33f77fe640 1 -- 172.21.15.89:0/1014854937 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f33f003a870 con 0x7f34001076e0 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.983+0000 7f3404edb640 1 -- 172.21.15.89:0/1014854937 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f340010be10 con 0x7f34001076e0 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.983+0000 7f33f77fe640 1 --2- 172.21.15.89:0/1014854937 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33ec039260 0x7f33ec03b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:06.035 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.983+0000 7f33f77fe640 1 -- 172.21.15.89:0/1014854937 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f33f006c970 con 0x7f34001076e0 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.983+0000 7f33f7fff640 1 --2- 172.21.15.89:0/1014854937 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33ec039260 0x7f33ec03b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.984+0000 7f33f7fff640 1 --2- 172.21.15.89:0/1014854937 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33ec039260 0x7f33ec03b720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f33e80098c0 tx=0x7f33e8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:04.988+0000 7f33f77fe640 1 -- 172.21.15.89:0/1014854937 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f33f004a200 con 0x7f34001076e0 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.171+0000 7f3404edb640 1 -- 172.21.15.89:0/1014854937 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f340019eeb0 con 0x7f33ec039260 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.393+0000 7f33f77fe640 1 -- 172.21.15.89:0/1014854937 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f340019eeb0 con 0x7f33ec039260 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.396+0000 7f3404edb640 1 -- 172.21.15.89:0/1014854937 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33ec039260 msgr2=0x7f33ec03b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.396+0000 7f3404edb640 1 --2- 172.21.15.89:0/1014854937 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33ec039260 0x7f33ec03b720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f33e80098c0 tx=0x7f33e8006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.396+0000 7f3404edb640 1 -- 172.21.15.89:0/1014854937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 msgr2=0x7f340019b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.396+0000 7f3404edb640 1 --2- 172.21.15.89:0/1014854937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f340019b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f33f0030560 tx=0x7f33f0032670 comp rx=0 tx=0).stop 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.396+0000 7f3404edb640 1 -- 172.21.15.89:0/1014854937 shutdown_connections 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.396+0000 7f3404edb640 1 --2- 172.21.15.89:0/1014854937 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33ec039260 0x7f33ec03b720 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.396+0000 7f3404edb640 1 --2- 172.21.15.89:0/1014854937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34001076e0 0x7f340019b760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:06.036 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.396+0000 7f3404edb640 1 -- 172.21.15.89:0/1014854937 >> 172.21.15.89:0/1014854937 conn(0x7f3400102e90 msgr2=0x7f3400103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:06.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.396+0000 7f3404edb640 1 -- 172.21.15.89:0/1014854937 shutdown_connections 2024-09-15T23:15:06.037 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:05.396+0000 7f3404edb640 1 -- 172.21.15.89:0/1014854937 wait complete. 2024-09-15T23:15:06.037 INFO:teuthology.orchestra.run.smithi089.stdout:Fetching dashboard port number... 2024-09-15T23:15:06.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:06 smithi089 ceph-mon[30933]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:06.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:07.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stdout 8443 2024-09-15T23:15:07.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.487+0000 7f79f926a640 1 Processor -- start 2024-09-15T23:15:07.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.487+0000 7f79f926a640 1 -- start start 2024-09-15T23:15:07.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.487+0000 7f79f926a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f4107b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:07.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.487+0000 7f79f926a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79f41080f0 con 0x7f79f4107720 2024-09-15T23:15:07.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.487+0000 7f79f2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f4107b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:07.272 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.488+0000 7f79f2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f4107b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34074/0 (socket says 172.21.15.89:34074) 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.488+0000 7f79f2ffd640 1 -- 172.21.15.89:0/1550724792 learned_addr learned my addr 172.21.15.89:0/1550724792 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.488+0000 7f79f2ffd640 1 -- 172.21.15.89:0/1550724792 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f4108920 con 0x7f79f4107720 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.488+0000 7f79f2ffd640 1 --2- 172.21.15.89:0/1550724792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f4107b20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009e80 tx=0x7f79e402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ca44ee77bf47a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.489+0000 7f79f1ffb640 1 -- 172.21.15.89:0/1550724792 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79e4036070 con 0x7f79f4107720 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.489+0000 7f79f1ffb640 1 -- 172.21.15.89:0/1550724792 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e40318e0 con 0x7f79f4107720 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.489+0000 7f79f1ffb640 1 -- 172.21.15.89:0/1550724792 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79e4007a00 con 0x7f79f4107720 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.489+0000 7f79f926a640 1 -- 172.21.15.89:0/1550724792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 msgr2=0x7f79f4107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.489+0000 7f79f926a640 1 --2- 172.21.15.89:0/1550724792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f4107b20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009e80 tx=0x7f79e402a7a0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.489+0000 7f79f926a640 1 -- 172.21.15.89:0/1550724792 shutdown_connections 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.489+0000 7f79f926a640 1 --2- 172.21.15.89:0/1550724792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f4107b20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.273 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.489+0000 7f79f926a640 1 -- 172.21.15.89:0/1550724792 >> 172.21.15.89:0/1550724792 conn(0x7f79f4102ed0 msgr2=0x7f79f41052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.489+0000 7f79f926a640 1 -- 172.21.15.89:0/1550724792 shutdown_connections 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.489+0000 7f79f926a640 1 -- 172.21.15.89:0/1550724792 wait complete. 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.490+0000 7f79f926a640 1 Processor -- start 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.490+0000 7f79f926a640 1 -- start start 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.490+0000 7f79f926a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f407db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.490+0000 7f79f926a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79f410bcd0 con 0x7f79f4107720 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.490+0000 7f79f2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f407db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.490+0000 7f79f2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f407db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34076/0 (socket says 172.21.15.89:34076) 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.490+0000 7f79f2ffd640 1 -- 172.21.15.89:0/584366160 learned_addr learned my addr 172.21.15.89:0/584366160 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.490+0000 7f79f2ffd640 1 -- 172.21.15.89:0/584366160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f407e0d0 con 0x7f79f4107720 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.491+0000 7f79f2ffd640 1 --2- 172.21.15.89:0/584366160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f407db90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79e4031cc0 tx=0x7f79e4031cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.491+0000 7f79ebfff640 1 -- 172.21.15.89:0/584366160 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79e4036050 con 0x7f79f4107720 2024-09-15T23:15:07.274 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.491+0000 7f79ebfff640 1 -- 172.21.15.89:0/584366160 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e4042070 con 0x7f79f4107720 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.491+0000 7f79f926a640 1 -- 172.21.15.89:0/584366160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f407c2c0 con 0x7f79f4107720 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.491+0000 7f79ebfff640 1 -- 172.21.15.89:0/584366160 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79e403d9b0 con 0x7f79f4107720 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.491+0000 7f79f926a640 1 -- 172.21.15.89:0/584366160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f407c800 con 0x7f79f4107720 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.491+0000 7f79ebfff640 1 -- 172.21.15.89:0/584366160 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f79e403dc60 con 0x7f79f4107720 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.492+0000 7f79f926a640 1 -- 172.21.15.89:0/584366160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f4107ba0 con 0x7f79f4107720 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.492+0000 7f79ebfff640 1 --2- 172.21.15.89:0/584366160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e0039260 0x7f79e003b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.492+0000 7f79ebfff640 1 -- 172.21.15.89:0/584366160 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f79e406d780 con 0x7f79f4107720 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.492+0000 7f79f27fc640 1 --2- 172.21.15.89:0/584366160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e0039260 0x7f79e003b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.492+0000 7f79f27fc640 1 --2- 172.21.15.89:0/584366160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e0039260 0x7f79e003b720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0098c0 tx=0x7f79dc006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.496+0000 7f79ebfff640 1 -- 172.21.15.89:0/584366160 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f79e4044310 con 0x7f79f4107720 2024-09-15T23:15:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.666+0000 7f79f926a640 1 -- 172.21.15.89:0/584366160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f79f407ca80 con 0x7f79f4107720 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.666+0000 7f79ebfff640 1 -- 172.21.15.89:0/584366160 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f79e4045520 con 0x7f79f4107720 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.668+0000 7f79f926a640 1 -- 172.21.15.89:0/584366160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e0039260 msgr2=0x7f79e003b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.668+0000 7f79f926a640 1 --2- 172.21.15.89:0/584366160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e0039260 0x7f79e003b720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0098c0 tx=0x7f79dc006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.668+0000 7f79f926a640 1 -- 172.21.15.89:0/584366160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 msgr2=0x7f79f407db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.668+0000 7f79f926a640 1 --2- 172.21.15.89:0/584366160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f407db90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79e4031cc0 tx=0x7f79e4031cf0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.668+0000 7f79f926a640 1 -- 172.21.15.89:0/584366160 shutdown_connections 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.668+0000 7f79f926a640 1 --2- 172.21.15.89:0/584366160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e0039260 0x7f79e003b720 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.668+0000 7f79f926a640 1 --2- 172.21.15.89:0/584366160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4107720 0x7f79f407db90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.668+0000 7f79f926a640 1 -- 172.21.15.89:0/584366160 >> 172.21.15.89:0/584366160 conn(0x7f79f4102ed0 msgr2=0x7f79f4103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.668+0000 7f79f926a640 1 -- 172.21.15.89:0/584366160 shutdown_connections 2024-09-15T23:15:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:06.668+0000 7f79f926a640 1 -- 172.21.15.89:0/584366160 wait complete. 2024-09-15T23:15:07.278 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T23:15:07.278 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout disabled 2024-09-15T23:15:07.285 INFO:teuthology.orchestra.run.smithi089.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T23:15:07.286 INFO:teuthology.orchestra.run.smithi089.stdout:systemctl: stdout inactive 2024-09-15T23:15:07.286 INFO:teuthology.orchestra.run.smithi089.stdout:firewalld.service is not enabled 2024-09-15T23:15:07.286 INFO:teuthology.orchestra.run.smithi089.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-15T23:15:07.287 INFO:teuthology.orchestra.run.smithi089.stdout:Ceph Dashboard is now available at: 2024-09-15T23:15:07.287 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:07.287 INFO:teuthology.orchestra.run.smithi089.stdout: URL: https://smithi089.front.sepia.ceph.com:8443/ 2024-09-15T23:15:07.287 INFO:teuthology.orchestra.run.smithi089.stdout: User: admin 2024-09-15T23:15:07.288 INFO:teuthology.orchestra.run.smithi089.stdout: Password: qasizj90t2 2024-09-15T23:15:07.288 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:07.288 INFO:teuthology.orchestra.run.smithi089.stdout:Saving cluster configuration to /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config directory 2024-09-15T23:15:07.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:07 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/584366160' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-15T23:15:08.512 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:08 smithi089 ceph-mon[30933]: mgrmap e13: x(active, since 6s) 2024-09-15T23:15:08.512 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:08 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/651796936' entity='client.admin' 2024-09-15T23:15:08.519 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.660+0000 7fd854bda640 1 Processor -- start 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.660+0000 7fd854bda640 1 -- start start 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.660+0000 7fd854bda640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85010a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.660+0000 7fd854bda640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd85007be90 con 0x7fd850108190 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.660+0000 7fd84e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85010a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.660+0000 7fd84e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85010a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34086/0 (socket says 172.21.15.89:34086) 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.660+0000 7fd84e59c640 1 -- 172.21.15.89:0/3717066856 learned_addr learned my addr 172.21.15.89:0/3717066856 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.660+0000 7fd84e59c640 1 -- 172.21.15.89:0/3717066856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85010aac0 con 0x7fd850108190 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.661+0000 7fd84e59c640 1 --2- 172.21.15.89:0/3717066856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85010a580 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd840009820 tx=0x7fd84002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f835184eeacd4492 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.661+0000 7fd84d59a640 1 -- 172.21.15.89:0/3717066856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd840037070 con 0x7fd850108190 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.661+0000 7fd84d59a640 1 -- 172.21.15.89:0/3717066856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd840032920 con 0x7fd850108190 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.661+0000 7fd84d59a640 1 -- 172.21.15.89:0/3717066856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd840030810 con 0x7fd850108190 2024-09-15T23:15:08.520 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.662+0000 7fd854bda640 1 -- 172.21.15.89:0/3717066856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 msgr2=0x7fd85010a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.662+0000 7fd854bda640 1 --2- 172.21.15.89:0/3717066856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85010a580 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd840009820 tx=0x7fd84002a270 comp rx=0 tx=0).stop 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.662+0000 7fd854bda640 1 -- 172.21.15.89:0/3717066856 shutdown_connections 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.662+0000 7fd854bda640 1 --2- 172.21.15.89:0/3717066856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85010a580 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.662+0000 7fd854bda640 1 -- 172.21.15.89:0/3717066856 >> 172.21.15.89:0/3717066856 conn(0x7fd850101db0 msgr2=0x7fd8501041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.662+0000 7fd854bda640 1 -- 172.21.15.89:0/3717066856 shutdown_connections 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.662+0000 7fd854bda640 1 -- 172.21.15.89:0/3717066856 wait complete. 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.663+0000 7fd854bda640 1 Processor -- start 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.663+0000 7fd854bda640 1 -- start start 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.663+0000 7fd854bda640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85019b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.663+0000 7fd854bda640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd85010bb40 con 0x7fd850108190 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.663+0000 7fd84e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85019b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.663+0000 7fd84e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85019b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34088/0 (socket says 172.21.15.89:34088) 2024-09-15T23:15:08.521 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.663+0000 7fd84e59c640 1 -- 172.21.15.89:0/651796936 learned_addr learned my addr 172.21.15.89:0/651796936 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.663+0000 7fd84e59c640 1 -- 172.21.15.89:0/651796936 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd85019bc00 con 0x7fd850108190 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.663+0000 7fd84e59c640 1 --2- 172.21.15.89:0/651796936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85019b6c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd840030380 tx=0x7fd840032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.664+0000 7fd83f7fe640 1 -- 172.21.15.89:0/651796936 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd840037040 con 0x7fd850108190 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.664+0000 7fd854bda640 1 -- 172.21.15.89:0/651796936 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd85019be90 con 0x7fd850108190 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.664+0000 7fd83f7fe640 1 -- 172.21.15.89:0/651796936 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd840040070 con 0x7fd850108190 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.664+0000 7fd83f7fe640 1 -- 172.21.15.89:0/651796936 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd84003a7b0 con 0x7fd850108190 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.664+0000 7fd854bda640 1 -- 172.21.15.89:0/651796936 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd85019eb80 con 0x7fd850108190 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.665+0000 7fd83f7fe640 1 -- 172.21.15.89:0/651796936 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fd84003bc00 con 0x7fd850108190 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.665+0000 7fd854bda640 1 -- 172.21.15.89:0/651796936 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8501085f0 con 0x7fd850108190 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.665+0000 7fd83f7fe640 1 --2- 172.21.15.89:0/651796936 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8380392a0 0x7fd83803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.665+0000 7fd83f7fe640 1 -- 172.21.15.89:0/651796936 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd84006cb70 con 0x7fd850108190 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.665+0000 7fd84dd9b640 1 --2- 172.21.15.89:0/651796936 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8380392a0 0x7fd83803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.665+0000 7fd84dd9b640 1 --2- 172.21.15.89:0/651796936 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8380392a0 0x7fd83803b760 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd8440098c0 tx=0x7fd844006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:08.522 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.669+0000 7fd83f7fe640 1 -- 172.21.15.89:0/651796936 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd840043360 con 0x7fd850108190 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.921+0000 7fd854bda640 1 -- 172.21.15.89:0/651796936 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fd85019ed50 con 0x7fd850108190 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.923+0000 7fd83f7fe640 1 -- 172.21.15.89:0/651796936 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fd840042da0 con 0x7fd850108190 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.925+0000 7fd854bda640 1 -- 172.21.15.89:0/651796936 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8380392a0 msgr2=0x7fd83803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.925+0000 7fd854bda640 1 --2- 172.21.15.89:0/651796936 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8380392a0 0x7fd83803b760 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd8440098c0 tx=0x7fd844006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.925+0000 7fd854bda640 1 -- 172.21.15.89:0/651796936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 msgr2=0x7fd85019b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.925+0000 7fd854bda640 1 --2- 172.21.15.89:0/651796936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85019b6c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd840030380 tx=0x7fd840032670 comp rx=0 tx=0).stop 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.925+0000 7fd854bda640 1 -- 172.21.15.89:0/651796936 shutdown_connections 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.925+0000 7fd854bda640 1 --2- 172.21.15.89:0/651796936 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8380392a0 0x7fd83803b760 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.925+0000 7fd854bda640 1 --2- 172.21.15.89:0/651796936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd850108190 0x7fd85019b6c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.925+0000 7fd854bda640 1 -- 172.21.15.89:0/651796936 >> 172.21.15.89:0/651796936 conn(0x7fd850101db0 msgr2=0x7fd850102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:08.523 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.925+0000 7fd854bda640 1 -- 172.21.15.89:0/651796936 shutdown_connections 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout:/usr/bin/ceph: stderr 2024-09-15T23:15:07.925+0000 7fd854bda640 1 -- 172.21.15.89:0/651796936 wait complete. 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout:Or, if you are only running a single cluster on this host: 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: ceph telemetry on 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:08.524 INFO:teuthology.orchestra.run.smithi089.stdout:For more information see: 2024-09-15T23:15:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-15T23:15:08.525 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:08.525 INFO:teuthology.orchestra.run.smithi089.stdout:Bootstrap complete. 2024-09-15T23:15:08.556 INFO:tasks.cephadm:Fetching config... 2024-09-15T23:15:08.556 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:15:08.556 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-15T23:15:08.572 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-15T23:15:08.572 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:15:08.572 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-15T23:15:08.628 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-15T23:15:08.628 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:15:08.628 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-15T23:15:08.693 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-15T23:15:08.693 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:15:08.693 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-15T23:15:08.749 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-15T23:15:08.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrKymXigMXdUhkKySbQi7v+c/DFHgKKnpxhw4JiOWN1Zm0jCJ0MB7Gt5QWEzJ4gMTVxcezDv7r1DZaZ5ySddyiKVVLkp6EZcCbxRd71CNlFI25qjpV8xElmzen9iUV/DOeFqcWyijT9ZMkVB7QTtrgMmOFTWNscCtw9Xc8XxKPhInkfoqa53r0OHLsmeYHz2pMSL4GwaW1FbwbtjftWTAdD/9X6m3zlHXn72x9Nrs8nPqaVxhxJ9PDPFRLN+ouLlSi90VdoiJTUwTNfp9hpCFMP639TEssXcvsNhxN3XPB+iSeroXc1WZ97nY4BHvSNMQLufkc7GDXKMEBkBef1lRnkKWbzEsklGEanD2JgOtNyTMAYbxtq/aviZ69Nu72G2v+RrHhmO3C7naEypEd7gRa3MFY9JLyYJlEpH+lTUfTRAcPoFix1mEqWm6TUNxEi6WC2BcPhS6voOVVamNR7ybyvQyRSWEdcM4XrUuXPYtR+RsN3pFL0AmSMPC4kDrBUhs= ceph-1b4e699c-73b8-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-15T23:15:08.822 INFO:teuthology.orchestra.run.smithi089.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrKymXigMXdUhkKySbQi7v+c/DFHgKKnpxhw4JiOWN1Zm0jCJ0MB7Gt5QWEzJ4gMTVxcezDv7r1DZaZ5ySddyiKVVLkp6EZcCbxRd71CNlFI25qjpV8xElmzen9iUV/DOeFqcWyijT9ZMkVB7QTtrgMmOFTWNscCtw9Xc8XxKPhInkfoqa53r0OHLsmeYHz2pMSL4GwaW1FbwbtjftWTAdD/9X6m3zlHXn72x9Nrs8nPqaVxhxJ9PDPFRLN+ouLlSi90VdoiJTUwTNfp9hpCFMP639TEssXcvsNhxN3XPB+iSeroXc1WZ97nY4BHvSNMQLufkc7GDXKMEBkBef1lRnkKWbzEsklGEanD2JgOtNyTMAYbxtq/aviZ69Nu72G2v+RrHhmO3C7naEypEd7gRa3MFY9JLyYJlEpH+lTUfTRAcPoFix1mEqWm6TUNxEi6WC2BcPhS6voOVVamNR7ybyvQyRSWEdcM4XrUuXPYtR+RsN3pFL0AmSMPC4kDrBUhs= ceph-1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:08.834 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrKymXigMXdUhkKySbQi7v+c/DFHgKKnpxhw4JiOWN1Zm0jCJ0MB7Gt5QWEzJ4gMTVxcezDv7r1DZaZ5ySddyiKVVLkp6EZcCbxRd71CNlFI25qjpV8xElmzen9iUV/DOeFqcWyijT9ZMkVB7QTtrgMmOFTWNscCtw9Xc8XxKPhInkfoqa53r0OHLsmeYHz2pMSL4GwaW1FbwbtjftWTAdD/9X6m3zlHXn72x9Nrs8nPqaVxhxJ9PDPFRLN+ouLlSi90VdoiJTUwTNfp9hpCFMP639TEssXcvsNhxN3XPB+iSeroXc1WZ97nY4BHvSNMQLufkc7GDXKMEBkBef1lRnkKWbzEsklGEanD2JgOtNyTMAYbxtq/aviZ69Nu72G2v+RrHhmO3C7naEypEd7gRa3MFY9JLyYJlEpH+lTUfTRAcPoFix1mEqWm6TUNxEi6WC2BcPhS6voOVVamNR7ybyvQyRSWEdcM4XrUuXPYtR+RsN3pFL0AmSMPC4kDrBUhs= ceph-1b4e699c-73b8-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-15T23:15:08.877 INFO:teuthology.orchestra.run.smithi120.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrKymXigMXdUhkKySbQi7v+c/DFHgKKnpxhw4JiOWN1Zm0jCJ0MB7Gt5QWEzJ4gMTVxcezDv7r1DZaZ5ySddyiKVVLkp6EZcCbxRd71CNlFI25qjpV8xElmzen9iUV/DOeFqcWyijT9ZMkVB7QTtrgMmOFTWNscCtw9Xc8XxKPhInkfoqa53r0OHLsmeYHz2pMSL4GwaW1FbwbtjftWTAdD/9X6m3zlHXn72x9Nrs8nPqaVxhxJ9PDPFRLN+ouLlSi90VdoiJTUwTNfp9hpCFMP639TEssXcvsNhxN3XPB+iSeroXc1WZ97nY4BHvSNMQLufkc7GDXKMEBkBef1lRnkKWbzEsklGEanD2JgOtNyTMAYbxtq/aviZ69Nu72G2v+RrHhmO3C7naEypEd7gRa3MFY9JLyYJlEpH+lTUfTRAcPoFix1mEqWm6TUNxEi6WC2BcPhS6voOVVamNR7ybyvQyRSWEdcM4XrUuXPYtR+RsN3pFL0AmSMPC4kDrBUhs= ceph-1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:08.889 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrKymXigMXdUhkKySbQi7v+c/DFHgKKnpxhw4JiOWN1Zm0jCJ0MB7Gt5QWEzJ4gMTVxcezDv7r1DZaZ5ySddyiKVVLkp6EZcCbxRd71CNlFI25qjpV8xElmzen9iUV/DOeFqcWyijT9ZMkVB7QTtrgMmOFTWNscCtw9Xc8XxKPhInkfoqa53r0OHLsmeYHz2pMSL4GwaW1FbwbtjftWTAdD/9X6m3zlHXn72x9Nrs8nPqaVxhxJ9PDPFRLN+ouLlSi90VdoiJTUwTNfp9hpCFMP639TEssXcvsNhxN3XPB+iSeroXc1WZ97nY4BHvSNMQLufkc7GDXKMEBkBef1lRnkKWbzEsklGEanD2JgOtNyTMAYbxtq/aviZ69Nu72G2v+RrHhmO3C7naEypEd7gRa3MFY9JLyYJlEpH+lTUfTRAcPoFix1mEqWm6TUNxEi6WC2BcPhS6voOVVamNR7ybyvQyRSWEdcM4XrUuXPYtR+RsN3pFL0AmSMPC4kDrBUhs= ceph-1b4e699c-73b8-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-15T23:15:08.931 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrKymXigMXdUhkKySbQi7v+c/DFHgKKnpxhw4JiOWN1Zm0jCJ0MB7Gt5QWEzJ4gMTVxcezDv7r1DZaZ5ySddyiKVVLkp6EZcCbxRd71CNlFI25qjpV8xElmzen9iUV/DOeFqcWyijT9ZMkVB7QTtrgMmOFTWNscCtw9Xc8XxKPhInkfoqa53r0OHLsmeYHz2pMSL4GwaW1FbwbtjftWTAdD/9X6m3zlHXn72x9Nrs8nPqaVxhxJ9PDPFRLN+ouLlSi90VdoiJTUwTNfp9hpCFMP639TEssXcvsNhxN3XPB+iSeroXc1WZ97nY4BHvSNMQLufkc7GDXKMEBkBef1lRnkKWbzEsklGEanD2JgOtNyTMAYbxtq/aviZ69Nu72G2v+RrHhmO3C7naEypEd7gRa3MFY9JLyYJlEpH+lTUfTRAcPoFix1mEqWm6TUNxEi6WC2BcPhS6voOVVamNR7ybyvQyRSWEdcM4XrUuXPYtR+RsN3pFL0AmSMPC4kDrBUhs= ceph-1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:08.944 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-15T23:15:09.191 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:09.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.670+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/386033587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe741020a0 msgr2=0x7fbe74104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:09.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.670+0000 7fbe7a3a2640 1 --2- 172.21.15.89:0/386033587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe741020a0 0x7fbe74104490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009880 tx=0x7fbe6402a5d0 comp rx=0 tx=0).stop 2024-09-15T23:15:09.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.670+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/386033587 shutdown_connections 2024-09-15T23:15:09.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.670+0000 7fbe7a3a2640 1 --2- 172.21.15.89:0/386033587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe741020a0 0x7fbe74104490 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:09.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.670+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/386033587 >> 172.21.15.89:0/386033587 conn(0x7fbe740fbc80 msgr2=0x7fbe740fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:09.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.670+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/386033587 shutdown_connections 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.670+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/386033587 wait complete. 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.671+0000 7fbe7a3a2640 1 Processor -- start 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.671+0000 7fbe7a3a2640 1 -- start start 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.671+0000 7fbe7a3a2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe741020a0 0x7fbe74196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.671+0000 7fbe7a3a2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbe741076b0 con 0x7fbe741020a0 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.671+0000 7fbe73fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe741020a0 0x7fbe74196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.671+0000 7fbe73fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe741020a0 0x7fbe74196d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34100/0 (socket says 172.21.15.89:34100) 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.671+0000 7fbe73fff640 1 -- 172.21.15.89:0/654477609 learned_addr learned my addr 172.21.15.89:0/654477609 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.671+0000 7fbe73fff640 1 -- 172.21.15.89:0/654477609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe74197240 con 0x7fbe741020a0 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.671+0000 7fbe73fff640 1 --2- 172.21.15.89:0/654477609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe741020a0 0x7fbe74196d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbe64004f60 tx=0x7fbe640048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.672+0000 7fbe71ffb640 1 -- 172.21.15.89:0/654477609 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe6403f070 con 0x7fbe741020a0 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.672+0000 7fbe71ffb640 1 -- 172.21.15.89:0/654477609 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe64003ad0 con 0x7fbe741020a0 2024-09-15T23:15:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.672+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/654477609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe74197530 con 0x7fbe741020a0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.672+0000 7fbe71ffb640 1 -- 172.21.15.89:0/654477609 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe64035030 con 0x7fbe741020a0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.672+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/654477609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7419a1b0 con 0x7fbe741020a0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.672+0000 7fbe71ffb640 1 -- 172.21.15.89:0/654477609 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fbe64042070 con 0x7fbe741020a0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.673+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/654477609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe74107e30 con 0x7fbe741020a0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.673+0000 7fbe71ffb640 1 --2- 172.21.15.89:0/654477609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbe60039300 0x7fbe6003b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.673+0000 7fbe71ffb640 1 -- 172.21.15.89:0/654477609 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbe6406d130 con 0x7fbe741020a0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.673+0000 7fbe6bfff640 1 --2- 172.21.15.89:0/654477609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbe60039300 0x7fbe6003b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.673+0000 7fbe6bfff640 1 --2- 172.21.15.89:0/654477609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbe60039300 0x7fbe6003b7c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0098c0 tx=0x7fbe5c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.677+0000 7fbe71ffb640 1 -- 172.21.15.89:0/654477609 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbe640303a0 con 0x7fbe741020a0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.846+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/654477609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fbe7402d940 con 0x7fbe741020a0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.848+0000 7fbe71ffb640 1 -- 172.21.15.89:0/654477609 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fbe6403b4d0 con 0x7fbe741020a0 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.851+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/654477609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbe60039300 msgr2=0x7fbe6003b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:09.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.851+0000 7fbe7a3a2640 1 --2- 172.21.15.89:0/654477609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbe60039300 0x7fbe6003b7c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c0098c0 tx=0x7fbe5c006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:09.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.851+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/654477609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe741020a0 msgr2=0x7fbe74196d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:09.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.851+0000 7fbe7a3a2640 1 --2- 172.21.15.89:0/654477609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe741020a0 0x7fbe74196d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbe64004f60 tx=0x7fbe640048f0 comp rx=0 tx=0).stop 2024-09-15T23:15:09.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.851+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/654477609 shutdown_connections 2024-09-15T23:15:09.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.851+0000 7fbe7a3a2640 1 --2- 172.21.15.89:0/654477609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbe60039300 0x7fbe6003b7c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:09.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.851+0000 7fbe7a3a2640 1 --2- 172.21.15.89:0/654477609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe741020a0 0x7fbe74196d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:09.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.851+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/654477609 >> 172.21.15.89:0/654477609 conn(0x7fbe740fbc80 msgr2=0x7fbe740fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:09.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.851+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/654477609 shutdown_connections 2024-09-15T23:15:09.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:09.851+0000 7fbe7a3a2640 1 -- 172.21.15.89:0/654477609 wait complete. 2024-09-15T23:15:10.465 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-15T23:15:10.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-15T23:15:10.954 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:11.171 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:10 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/654477609' entity='client.admin' 2024-09-15T23:15:11.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.847+0000 7fb3bae96640 1 -- 172.21.15.89:0/2628497360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b4100510 msgr2=0x7fb3b4100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:11.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.847+0000 7fb3bae96640 1 --2- 172.21.15.89:0/2628497360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b4100510 0x7fb3b4100910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009880 tx=0x7fb3a802a5d0 comp rx=0 tx=0).stop 2024-09-15T23:15:11.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.847+0000 7fb3bae96640 1 -- 172.21.15.89:0/2628497360 shutdown_connections 2024-09-15T23:15:11.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.847+0000 7fb3bae96640 1 --2- 172.21.15.89:0/2628497360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b4100510 0x7fb3b4100910 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:11.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.847+0000 7fb3bae96640 1 -- 172.21.15.89:0/2628497360 >> 172.21.15.89:0/2628497360 conn(0x7fb3b40fbc80 msgr2=0x7fb3b40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:11.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.847+0000 7fb3bae96640 1 -- 172.21.15.89:0/2628497360 shutdown_connections 2024-09-15T23:15:11.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.847+0000 7fb3bae96640 1 -- 172.21.15.89:0/2628497360 wait complete. 2024-09-15T23:15:11.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.848+0000 7fb3bae96640 1 Processor -- start 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.848+0000 7fb3bae96640 1 -- start start 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.848+0000 7fb3bae96640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b4100510 0x7fb3b4195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.848+0000 7fb3bae96640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b4106620 con 0x7fb3b4100510 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.848+0000 7fb3b8c32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b4100510 0x7fb3b4195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.848+0000 7fb3b8c32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b4100510 0x7fb3b4195cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34120/0 (socket says 172.21.15.89:34120) 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.848+0000 7fb3b8c32640 1 -- 172.21.15.89:0/1091145534 learned_addr learned my addr 172.21.15.89:0/1091145534 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.848+0000 7fb3b8c32640 1 -- 172.21.15.89:0/1091145534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b4196200 con 0x7fb3b4100510 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.848+0000 7fb3b8c32640 1 --2- 172.21.15.89:0/1091145534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b4100510 0x7fb3b4195cc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8004f60 tx=0x7fb3a80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.849+0000 7fb3b1ffb640 1 -- 172.21.15.89:0/1091145534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3a803f070 con 0x7fb3b4100510 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.849+0000 7fb3bae96640 1 -- 172.21.15.89:0/1091145534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3b4196490 con 0x7fb3b4100510 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.849+0000 7fb3b1ffb640 1 -- 172.21.15.89:0/1091145534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3a8004d40 con 0x7fb3b4100510 2024-09-15T23:15:11.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.849+0000 7fb3b1ffb640 1 -- 172.21.15.89:0/1091145534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3a8035030 con 0x7fb3b4100510 2024-09-15T23:15:11.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.849+0000 7fb3bae96640 1 -- 172.21.15.89:0/1091145534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3b4069ed0 con 0x7fb3b4100510 2024-09-15T23:15:11.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.850+0000 7fb3b1ffb640 1 -- 172.21.15.89:0/1091145534 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fb3a8042070 con 0x7fb3b4100510 2024-09-15T23:15:11.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.850+0000 7fb3b1ffb640 1 --2- 172.21.15.89:0/1091145534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3a40392a0 0x7fb3a403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:11.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.850+0000 7fb3bae96640 1 -- 172.21.15.89:0/1091145534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3b4106da0 con 0x7fb3b4100510 2024-09-15T23:15:11.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.850+0000 7fb3b1ffb640 1 -- 172.21.15.89:0/1091145534 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb3a806d0f0 con 0x7fb3b4100510 2024-09-15T23:15:11.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.850+0000 7fb3b3fff640 1 --2- 172.21.15.89:0/1091145534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3a40392a0 0x7fb3a403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:11.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.851+0000 7fb3b3fff640 1 --2- 172.21.15.89:0/1091145534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3a40392a0 0x7fb3a403b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00098c0 tx=0x7fb3a0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:11.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:11.855+0000 7fb3b1ffb640 1 -- 172.21.15.89:0/1091145534 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb3a80303a0 con 0x7fb3b4100510 2024-09-15T23:15:12.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.042+0000 7fb3bae96640 1 -- 172.21.15.89:0/1091145534 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fb3b406aae0 con 0x7fb3a40392a0 2024-09-15T23:15:12.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.046+0000 7fb3b1ffb640 1 -- 172.21.15.89:0/1091145534 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb3b406aae0 con 0x7fb3a40392a0 2024-09-15T23:15:12.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.048+0000 7fb3bae96640 1 -- 172.21.15.89:0/1091145534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3a40392a0 msgr2=0x7fb3a403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:12.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.048+0000 7fb3bae96640 1 --2- 172.21.15.89:0/1091145534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3a40392a0 0x7fb3a403b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00098c0 tx=0x7fb3a0006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.048+0000 7fb3bae96640 1 -- 172.21.15.89:0/1091145534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b4100510 msgr2=0x7fb3b4195cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:12.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.048+0000 7fb3bae96640 1 --2- 172.21.15.89:0/1091145534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b4100510 0x7fb3b4195cc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8004f60 tx=0x7fb3a80048f0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.048+0000 7fb3bae96640 1 -- 172.21.15.89:0/1091145534 shutdown_connections 2024-09-15T23:15:12.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.048+0000 7fb3bae96640 1 --2- 172.21.15.89:0/1091145534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3a40392a0 0x7fb3a403b760 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.048+0000 7fb3bae96640 1 --2- 172.21.15.89:0/1091145534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3b4100510 0x7fb3b4195cc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.048+0000 7fb3bae96640 1 -- 172.21.15.89:0/1091145534 >> 172.21.15.89:0/1091145534 conn(0x7fb3b40fbc80 msgr2=0x7fb3b40fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:12.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.048+0000 7fb3bae96640 1 -- 172.21.15.89:0/1091145534 shutdown_connections 2024-09-15T23:15:12.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:12.048+0000 7fb3bae96640 1 -- 172.21.15.89:0/1091145534 wait complete. 2024-09-15T23:15:12.698 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi120 2024-09-15T23:15:12.698 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:15:12.698 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/etc/ceph/ceph.conf 2024-09-15T23:15:12.723 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:15:12.723 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:15:12.802 INFO:tasks.cephadm:Adding host smithi120 to orchestrator... 2024-09-15T23:15:12.802 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch host add smithi120 2024-09-15T23:15:13.183 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:13.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:13 smithi089 ceph-mon[30933]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:13.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:13 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:13.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.863+0000 7fbf48c00640 1 -- 172.21.15.89:0/245860531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf44076220 msgr2=0x7fbf44074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:13.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.863+0000 7fbf48c00640 1 --2- 172.21.15.89:0/245860531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf44076220 0x7fbf44074680 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009880 tx=0x7fbf3402a5d0 comp rx=0 tx=0).stop 2024-09-15T23:15:13.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.863+0000 7fbf48c00640 1 -- 172.21.15.89:0/245860531 shutdown_connections 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.863+0000 7fbf48c00640 1 --2- 172.21.15.89:0/245860531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf44076220 0x7fbf44074680 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.863+0000 7fbf48c00640 1 -- 172.21.15.89:0/245860531 >> 172.21.15.89:0/245860531 conn(0x7fbf440fcdb0 msgr2=0x7fbf440ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.863+0000 7fbf48c00640 1 -- 172.21.15.89:0/245860531 shutdown_connections 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.863+0000 7fbf48c00640 1 -- 172.21.15.89:0/245860531 wait complete. 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.864+0000 7fbf48c00640 1 Processor -- start 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.864+0000 7fbf48c00640 1 -- start start 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf48c00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf44076220 0x7fbf44198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf48c00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbf44109920 con 0x7fbf44076220 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf4259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf44076220 0x7fbf44198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf4259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf44076220 0x7fbf44198fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34144/0 (socket says 172.21.15.89:34144) 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf4259c640 1 -- 172.21.15.89:0/1972308623 learned_addr learned my addr 172.21.15.89:0/1972308623 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:13.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf4259c640 1 -- 172.21.15.89:0/1972308623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf441994e0 con 0x7fbf44076220 2024-09-15T23:15:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf4259c640 1 --2- 172.21.15.89:0/1972308623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf44076220 0x7fbf44198fa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbf34004f60 tx=0x7fbf340048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf3b7fe640 1 -- 172.21.15.89:0/1972308623 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf3403f070 con 0x7fbf44076220 2024-09-15T23:15:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf48c00640 1 -- 172.21.15.89:0/1972308623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf44199770 con 0x7fbf44076220 2024-09-15T23:15:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf3b7fe640 1 -- 172.21.15.89:0/1972308623 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf34004d40 con 0x7fbf44076220 2024-09-15T23:15:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf3b7fe640 1 -- 172.21.15.89:0/1972308623 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf34035030 con 0x7fbf44076220 2024-09-15T23:15:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.865+0000 7fbf48c00640 1 -- 172.21.15.89:0/1972308623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf4419c460 con 0x7fbf44076220 2024-09-15T23:15:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.866+0000 7fbf3b7fe640 1 -- 172.21.15.89:0/1972308623 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fbf34042070 con 0x7fbf44076220 2024-09-15T23:15:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.867+0000 7fbf48c00640 1 -- 172.21.15.89:0/1972308623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf4410a0a0 con 0x7fbf44076220 2024-09-15T23:15:13.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.867+0000 7fbf3b7fe640 1 --2- 172.21.15.89:0/1972308623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbf300392a0 0x7fbf3003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.867+0000 7fbf3b7fe640 1 -- 172.21.15.89:0/1972308623 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbf3406d080 con 0x7fbf44076220 2024-09-15T23:15:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.867+0000 7fbf3bfff640 1 --2- 172.21.15.89:0/1972308623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbf300392a0 0x7fbf3003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.867+0000 7fbf3bfff640 1 --2- 172.21.15.89:0/1972308623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbf300392a0 0x7fbf3003b760 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0098c0 tx=0x7fbf2c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:13.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:13.872+0000 7fbf3b7fe640 1 -- 172.21.15.89:0/1972308623 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbf34040260 con 0x7fbf44076220 2024-09-15T23:15:14.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:14.053+0000 7fbf48c00640 1 -- 172.21.15.89:0/1972308623 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}) -- 0x7fbf4402d830 con 0x7fbf300392a0 2024-09-15T23:15:14.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:14.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:14.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:15:14.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: Updating smithi089:/etc/ceph/ceph.conf 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: Updating smithi089:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: Updating smithi089:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:14.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:15.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:15 smithi089 ceph-mon[30933]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:16.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:16 smithi089 ceph-mon[30933]: Deploying cephadm binary to smithi120 2024-09-15T23:15:17.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.278+0000 7fbf3b7fe640 1 -- 172.21.15.89:0/1972308623 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fbf4402d830 con 0x7fbf300392a0 2024-09-15T23:15:17.279 INFO:teuthology.orchestra.run.smithi089.stdout:Added host 'smithi120' with addr '172.21.15.120' 2024-09-15T23:15:17.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.280+0000 7fbf48c00640 1 -- 172.21.15.89:0/1972308623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbf300392a0 msgr2=0x7fbf3003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.280+0000 7fbf48c00640 1 --2- 172.21.15.89:0/1972308623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbf300392a0 0x7fbf3003b760 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0098c0 tx=0x7fbf2c006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.280+0000 7fbf48c00640 1 -- 172.21.15.89:0/1972308623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf44076220 msgr2=0x7fbf44198fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.280+0000 7fbf48c00640 1 --2- 172.21.15.89:0/1972308623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf44076220 0x7fbf44198fa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbf34004f60 tx=0x7fbf340048f0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.280+0000 7fbf48c00640 1 -- 172.21.15.89:0/1972308623 shutdown_connections 2024-09-15T23:15:17.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.280+0000 7fbf48c00640 1 --2- 172.21.15.89:0/1972308623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbf300392a0 0x7fbf3003b760 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.280+0000 7fbf48c00640 1 --2- 172.21.15.89:0/1972308623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf44076220 0x7fbf44198fa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.280+0000 7fbf48c00640 1 -- 172.21.15.89:0/1972308623 >> 172.21.15.89:0/1972308623 conn(0x7fbf440fcdb0 msgr2=0x7fbf440fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:17.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.280+0000 7fbf48c00640 1 -- 172.21.15.89:0/1972308623 shutdown_connections 2024-09-15T23:15:17.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:17.280+0000 7fbf48c00640 1 -- 172.21.15.89:0/1972308623 wait complete. 2024-09-15T23:15:17.903 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-15T23:15:18.141 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:18.519 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:18.520 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:18 smithi089 ceph-mon[30933]: Added host smithi120 2024-09-15T23:15:18.520 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:18.520 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:18.520 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:18.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 -- 172.21.15.89:0/3082882933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a301027f0 msgr2=0x7f5a30102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:18.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 --2- 172.21.15.89:0/3082882933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a301027f0 0x7f5a30102bd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009880 tx=0x7f5a2802a670 comp rx=0 tx=0).stop 2024-09-15T23:15:18.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 -- 172.21.15.89:0/3082882933 shutdown_connections 2024-09-15T23:15:18.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 --2- 172.21.15.89:0/3082882933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a301027f0 0x7f5a30102bd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 -- 172.21.15.89:0/3082882933 >> 172.21.15.89:0/3082882933 conn(0x7f5a300fc1b0 msgr2=0x7f5a300fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 -- 172.21.15.89:0/3082882933 shutdown_connections 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 -- 172.21.15.89:0/3082882933 wait complete. 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 Processor -- start 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 -- start start 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a301027f0 0x7f5a30199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.598+0000 7f5a38709640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3010bcc0 con 0x7f5a301027f0 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.599+0000 7f5a364a5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a301027f0 0x7f5a30199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.599+0000 7f5a364a5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a301027f0 0x7f5a30199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44798/0 (socket says 172.21.15.89:44798) 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.599+0000 7f5a364a5640 1 -- 172.21.15.89:0/3195513685 learned_addr learned my addr 172.21.15.89:0/3195513685 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.599+0000 7f5a364a5640 1 -- 172.21.15.89:0/3195513685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a30199970 con 0x7f5a301027f0 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.599+0000 7f5a364a5640 1 --2- 172.21.15.89:0/3195513685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a301027f0 0x7f5a30199430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009230 tx=0x7f5a28003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:18.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.599+0000 7f5a2f7fe640 1 -- 172.21.15.89:0/3195513685 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a2803f070 con 0x7f5a301027f0 2024-09-15T23:15:18.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.599+0000 7f5a2f7fe640 1 -- 172.21.15.89:0/3195513685 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a2803bc80 con 0x7f5a301027f0 2024-09-15T23:15:18.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.599+0000 7f5a38709640 1 -- 172.21.15.89:0/3195513685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a30199c00 con 0x7f5a301027f0 2024-09-15T23:15:18.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.599+0000 7f5a2f7fe640 1 -- 172.21.15.89:0/3195513685 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a28035030 con 0x7f5a301027f0 2024-09-15T23:15:18.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.599+0000 7f5a38709640 1 -- 172.21.15.89:0/3195513685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a3019a060 con 0x7f5a301027f0 2024-09-15T23:15:18.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.600+0000 7f5a2f7fe640 1 -- 172.21.15.89:0/3195513685 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f5a28042070 con 0x7f5a301027f0 2024-09-15T23:15:18.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.600+0000 7f5a2f7fe640 1 --2- 172.21.15.89:0/3195513685 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5a18039300 0x7f5a1803b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:18.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.600+0000 7f5a2f7fe640 1 -- 172.21.15.89:0/3195513685 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5a2806cf70 con 0x7f5a301027f0 2024-09-15T23:15:18.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.600+0000 7f5a38709640 1 -- 172.21.15.89:0/3195513685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a3006a710 con 0x7f5a301027f0 2024-09-15T23:15:18.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.600+0000 7f5a35ca4640 1 --2- 172.21.15.89:0/3195513685 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5a18039300 0x7f5a1803b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:18.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.601+0000 7f5a35ca4640 1 --2- 172.21.15.89:0/3195513685 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5a18039300 0x7f5a1803b7c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5a240098c0 tx=0x7f5a24006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:18.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.605+0000 7f5a2f7fe640 1 -- 172.21.15.89:0/3195513685 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5a280314c0 con 0x7f5a301027f0 2024-09-15T23:15:18.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.778+0000 7f5a38709640 1 -- 172.21.15.89:0/3195513685 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5a3019a790 con 0x7f5a18039300 2024-09-15T23:15:18.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.779+0000 7f5a2f7fe640 1 -- 172.21.15.89:0/3195513685 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f5a3019a790 con 0x7f5a18039300 2024-09-15T23:15:18.780 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:18.780 INFO:teuthology.orchestra.run.smithi089.stdout:[{"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}, {"addr": "172.21.15.120", "hostname": "smithi120", "labels": [], "status": ""}] 2024-09-15T23:15:18.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.781+0000 7f5a38709640 1 -- 172.21.15.89:0/3195513685 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5a18039300 msgr2=0x7f5a1803b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:18.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.781+0000 7f5a38709640 1 --2- 172.21.15.89:0/3195513685 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5a18039300 0x7f5a1803b7c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5a240098c0 tx=0x7f5a24006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.781+0000 7f5a38709640 1 -- 172.21.15.89:0/3195513685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a301027f0 msgr2=0x7f5a30199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:18.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.781+0000 7f5a38709640 1 --2- 172.21.15.89:0/3195513685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a301027f0 0x7f5a30199430 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009230 tx=0x7f5a28003d40 comp rx=0 tx=0).stop 2024-09-15T23:15:18.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.781+0000 7f5a38709640 1 -- 172.21.15.89:0/3195513685 shutdown_connections 2024-09-15T23:15:18.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.781+0000 7f5a38709640 1 --2- 172.21.15.89:0/3195513685 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5a18039300 0x7f5a1803b7c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.781+0000 7f5a38709640 1 --2- 172.21.15.89:0/3195513685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a301027f0 0x7f5a30199430 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.781+0000 7f5a38709640 1 -- 172.21.15.89:0/3195513685 >> 172.21.15.89:0/3195513685 conn(0x7f5a300fc1b0 msgr2=0x7f5a301079b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:18.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.781+0000 7f5a38709640 1 -- 172.21.15.89:0/3195513685 shutdown_connections 2024-09-15T23:15:18.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:18.781+0000 7f5a38709640 1 -- 172.21.15.89:0/3195513685 wait complete. 2024-09-15T23:15:19.494 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi174 2024-09-15T23:15:19.494 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:15:19.494 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.conf 2024-09-15T23:15:19.514 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:15:19.514 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:15:19.573 INFO:tasks.cephadm:Adding host smithi174 to orchestrator... 2024-09-15T23:15:19.573 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch host add smithi174 2024-09-15T23:15:19.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:19.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:19 smithi089 ceph-mon[30933]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:19.810 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.222+0000 7ff61b289640 1 -- 172.21.15.89:0/2989631744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff614106560 msgr2=0x7ff614106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.222+0000 7ff61b289640 1 --2- 172.21.15.89:0/2989631744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff614106560 0x7ff614106940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff610009880 tx=0x7ff61002a5d0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.222+0000 7ff61b289640 1 -- 172.21.15.89:0/2989631744 shutdown_connections 2024-09-15T23:15:20.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.222+0000 7ff61b289640 1 --2- 172.21.15.89:0/2989631744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff614106560 0x7ff614106940 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.222+0000 7ff61b289640 1 -- 172.21.15.89:0/2989631744 >> 172.21.15.89:0/2989631744 conn(0x7ff6140fc1b0 msgr2=0x7ff6140fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.222+0000 7ff61b289640 1 -- 172.21.15.89:0/2989631744 shutdown_connections 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.222+0000 7ff61b289640 1 -- 172.21.15.89:0/2989631744 wait complete. 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.223+0000 7ff61b289640 1 Processor -- start 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.223+0000 7ff61b289640 1 -- start start 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.223+0000 7ff61b289640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff614106560 0x7ff6141970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.223+0000 7ff61b289640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff61410bc90 con 0x7ff614106560 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.223+0000 7ff619025640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff614106560 0x7ff6141970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.223+0000 7ff619025640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff614106560 0x7ff6141970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44822/0 (socket says 172.21.15.89:44822) 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.223+0000 7ff619025640 1 -- 172.21.15.89:0/1269473342 learned_addr learned my addr 172.21.15.89:0/1269473342 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:20.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.223+0000 7ff619025640 1 -- 172.21.15.89:0/1269473342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff614197620 con 0x7ff614106560 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.223+0000 7ff619025640 1 --2- 172.21.15.89:0/1269473342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff614106560 0x7ff6141970e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff610009230 tx=0x7ff610003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.224+0000 7ff60a7fc640 1 -- 172.21.15.89:0/1269473342 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff61003f070 con 0x7ff614106560 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.224+0000 7ff61b289640 1 -- 172.21.15.89:0/1269473342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6141978b0 con 0x7ff614106560 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.224+0000 7ff60a7fc640 1 -- 172.21.15.89:0/1269473342 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff61003bc80 con 0x7ff614106560 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.224+0000 7ff60a7fc640 1 -- 172.21.15.89:0/1269473342 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff610035030 con 0x7ff614106560 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.224+0000 7ff61b289640 1 -- 172.21.15.89:0/1269473342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff61419b9b0 con 0x7ff614106560 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.224+0000 7ff60a7fc640 1 -- 172.21.15.89:0/1269473342 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7ff610042070 con 0x7ff614106560 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.225+0000 7ff61b289640 1 -- 172.21.15.89:0/1269473342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff614101bd0 con 0x7ff614106560 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.225+0000 7ff60a7fc640 1 --2- 172.21.15.89:0/1269473342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6000392a0 0x7ff60003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.225+0000 7ff60a7fc640 1 -- 172.21.15.89:0/1269473342 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff61006d080 con 0x7ff614106560 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.225+0000 7ff618824640 1 --2- 172.21.15.89:0/1269473342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6000392a0 0x7ff60003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:20.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.225+0000 7ff618824640 1 --2- 172.21.15.89:0/1269473342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6000392a0 0x7ff60003b760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff6040098c0 tx=0x7ff604006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:20.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.229+0000 7ff60a7fc640 1 -- 172.21.15.89:0/1269473342 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff61004c350 con 0x7ff614106560 2024-09-15T23:15:20.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:20.400+0000 7ff61b289640 1 -- 172.21.15.89:0/1269473342 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}) -- 0x7ff6141069c0 con 0x7ff6000392a0 2024-09-15T23:15:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:21 smithi089 ceph-mon[30933]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:21 smithi089 ceph-mon[30933]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:21 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:21 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:21 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:21 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:21 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi120", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:15:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:21 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:21 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:22 smithi089 ceph-mon[30933]: Deploying cephadm binary to smithi174 2024-09-15T23:15:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:22 smithi089 ceph-mon[30933]: Updating smithi120:/etc/ceph/ceph.conf 2024-09-15T23:15:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:22 smithi089 ceph-mon[30933]: Updating smithi120:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:22 smithi089 ceph-mon[30933]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:15:23.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.553+0000 7ff60a7fc640 1 -- 172.21.15.89:0/1269473342 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7ff6141069c0 con 0x7ff6000392a0 2024-09-15T23:15:23.555 INFO:teuthology.orchestra.run.smithi089.stdout:Added host 'smithi174' with addr '172.21.15.174' 2024-09-15T23:15:23.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.555+0000 7ff61b289640 1 -- 172.21.15.89:0/1269473342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6000392a0 msgr2=0x7ff60003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.555+0000 7ff61b289640 1 --2- 172.21.15.89:0/1269473342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6000392a0 0x7ff60003b760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff6040098c0 tx=0x7ff604006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.555+0000 7ff61b289640 1 -- 172.21.15.89:0/1269473342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff614106560 msgr2=0x7ff6141970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.555+0000 7ff61b289640 1 --2- 172.21.15.89:0/1269473342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff614106560 0x7ff6141970e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff610009230 tx=0x7ff610003d40 comp rx=0 tx=0).stop 2024-09-15T23:15:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.555+0000 7ff61b289640 1 -- 172.21.15.89:0/1269473342 shutdown_connections 2024-09-15T23:15:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.555+0000 7ff61b289640 1 --2- 172.21.15.89:0/1269473342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6000392a0 0x7ff60003b760 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.555+0000 7ff61b289640 1 --2- 172.21.15.89:0/1269473342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff614106560 0x7ff6141970e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.555+0000 7ff61b289640 1 -- 172.21.15.89:0/1269473342 >> 172.21.15.89:0/1269473342 conn(0x7ff6140fc1b0 msgr2=0x7ff6140fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.555+0000 7ff61b289640 1 -- 172.21.15.89:0/1269473342 shutdown_connections 2024-09-15T23:15:23.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:23.556+0000 7ff61b289640 1 -- 172.21.15.89:0/1269473342 wait complete. 2024-09-15T23:15:24.146 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:23 smithi089 ceph-mon[30933]: Updating smithi120:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T23:15:24.146 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:23 smithi089 ceph-mon[30933]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:24.146 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:24.147 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:24.147 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:24.147 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:24.147 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:24.147 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:24.174 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-15T23:15:24.413 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.862+0000 7fdfbef5e640 1 -- 172.21.15.89:0/2503291867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb8106540 msgr2=0x7fdfb8106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.862+0000 7fdfbef5e640 1 --2- 172.21.15.89:0/2503291867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb8106540 0x7fdfb8106920 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009c10 tx=0x7fdfa802aa10 comp rx=0 tx=0).stop 2024-09-15T23:15:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.862+0000 7fdfbef5e640 1 -- 172.21.15.89:0/2503291867 shutdown_connections 2024-09-15T23:15:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.862+0000 7fdfbef5e640 1 --2- 172.21.15.89:0/2503291867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb8106540 0x7fdfb8106920 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.862+0000 7fdfbef5e640 1 -- 172.21.15.89:0/2503291867 >> 172.21.15.89:0/2503291867 conn(0x7fdfb8077310 msgr2=0x7fdfb80fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.863+0000 7fdfbef5e640 1 -- 172.21.15.89:0/2503291867 shutdown_connections 2024-09-15T23:15:24.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.863+0000 7fdfbef5e640 1 -- 172.21.15.89:0/2503291867 wait complete. 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.863+0000 7fdfbef5e640 1 Processor -- start 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.863+0000 7fdfbef5e640 1 -- start start 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.863+0000 7fdfbef5e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb8106540 0x7fdfb8197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.863+0000 7fdfbef5e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb810ab70 con 0x7fdfb8106540 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.863+0000 7fdfbccfa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb8106540 0x7fdfb8197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.863+0000 7fdfbccfa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb8106540 0x7fdfb8197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44844/0 (socket says 172.21.15.89:44844) 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.863+0000 7fdfbccfa640 1 -- 172.21.15.89:0/288335666 learned_addr learned my addr 172.21.15.89:0/288335666 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.864+0000 7fdfbccfa640 1 -- 172.21.15.89:0/288335666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb81984b0 con 0x7fdfb8106540 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.864+0000 7fdfbccfa640 1 --2- 172.21.15.89:0/288335666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb8106540 0x7fdfb8197f70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8005dd0 tx=0x7fdfa8030c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.864+0000 7fdfb5ffb640 1 -- 172.21.15.89:0/288335666 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfa8034070 con 0x7fdfb8106540 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.864+0000 7fdfbef5e640 1 -- 172.21.15.89:0/288335666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb8198740 con 0x7fdfb8106540 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.864+0000 7fdfb5ffb640 1 -- 172.21.15.89:0/288335666 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfa802f8c0 con 0x7fdfb8106540 2024-09-15T23:15:24.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.864+0000 7fdfb5ffb640 1 -- 172.21.15.89:0/288335666 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfa8004520 con 0x7fdfb8106540 2024-09-15T23:15:24.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.864+0000 7fdfbef5e640 1 -- 172.21.15.89:0/288335666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb8198ae0 con 0x7fdfb8106540 2024-09-15T23:15:24.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.865+0000 7fdfb5ffb640 1 -- 172.21.15.89:0/288335666 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fdfa8037020 con 0x7fdfb8106540 2024-09-15T23:15:24.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.865+0000 7fdfbef5e640 1 -- 172.21.15.89:0/288335666 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb8101bb0 con 0x7fdfb8106540 2024-09-15T23:15:24.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.865+0000 7fdfb5ffb640 1 --2- 172.21.15.89:0/288335666 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdfa40392a0 0x7fdfa403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:24.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.865+0000 7fdfb5ffb640 1 -- 172.21.15.89:0/288335666 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdfa806d3e0 con 0x7fdfb8106540 2024-09-15T23:15:24.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.865+0000 7fdfb7fff640 1 --2- 172.21.15.89:0/288335666 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdfa40392a0 0x7fdfa403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:24.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.866+0000 7fdfb7fff640 1 --2- 172.21.15.89:0/288335666 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdfa40392a0 0x7fdfa403b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00098c0 tx=0x7fdfa0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:24.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:24.870+0000 7fdfb5ffb640 1 -- 172.21.15.89:0/288335666 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdfa803f070 con 0x7fdfb8106540 2024-09-15T23:15:25.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.046+0000 7fdfbef5e640 1 -- 172.21.15.89:0/288335666 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdfb81069a0 con 0x7fdfa40392a0 2024-09-15T23:15:25.048 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:24 smithi089 ceph-mon[30933]: Added host smithi174 2024-09-15T23:15:25.048 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:25.048 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:25.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.047+0000 7fdfb5ffb640 1 -- 172.21.15.89:0/288335666 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fdfb81069a0 con 0x7fdfa40392a0 2024-09-15T23:15:25.049 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:25.049 INFO:teuthology.orchestra.run.smithi089.stdout:[{"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}, {"addr": "172.21.15.120", "hostname": "smithi120", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}] 2024-09-15T23:15:25.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.050+0000 7fdfbef5e640 1 -- 172.21.15.89:0/288335666 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdfa40392a0 msgr2=0x7fdfa403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:25.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.050+0000 7fdfbef5e640 1 --2- 172.21.15.89:0/288335666 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdfa40392a0 0x7fdfa403b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00098c0 tx=0x7fdfa0006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.050+0000 7fdfbef5e640 1 -- 172.21.15.89:0/288335666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb8106540 msgr2=0x7fdfb8197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:25.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.050+0000 7fdfbef5e640 1 --2- 172.21.15.89:0/288335666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb8106540 0x7fdfb8197f70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8005dd0 tx=0x7fdfa8030c30 comp rx=0 tx=0).stop 2024-09-15T23:15:25.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.050+0000 7fdfbef5e640 1 -- 172.21.15.89:0/288335666 shutdown_connections 2024-09-15T23:15:25.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.050+0000 7fdfbef5e640 1 --2- 172.21.15.89:0/288335666 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdfa40392a0 0x7fdfa403b760 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.050+0000 7fdfbef5e640 1 --2- 172.21.15.89:0/288335666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdfb8106540 0x7fdfb8197f70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.050+0000 7fdfbef5e640 1 -- 172.21.15.89:0/288335666 >> 172.21.15.89:0/288335666 conn(0x7fdfb8077310 msgr2=0x7fdfb80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:25.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.050+0000 7fdfbef5e640 1 -- 172.21.15.89:0/288335666 shutdown_connections 2024-09-15T23:15:25.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:25.050+0000 7fdfbef5e640 1 -- 172.21.15.89:0/288335666 wait complete. 2024-09-15T23:15:25.656 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-15T23:15:25.656 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-15T23:15:25.896 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:26.265 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:25 smithi089 ceph-mon[30933]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.345+0000 7f7c79e17640 1 -- 172.21.15.89:0/468818830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c74106560 msgr2=0x7f7c74106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.345+0000 7f7c79e17640 1 --2- 172.21.15.89:0/468818830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c74106560 0x7f7c74106940 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009880 tx=0x7f7c6402a5d0 comp rx=0 tx=0).stop 2024-09-15T23:15:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.345+0000 7f7c79e17640 1 -- 172.21.15.89:0/468818830 shutdown_connections 2024-09-15T23:15:26.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.345+0000 7f7c79e17640 1 --2- 172.21.15.89:0/468818830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c74106560 0x7f7c74106940 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:26.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.345+0000 7f7c79e17640 1 -- 172.21.15.89:0/468818830 >> 172.21.15.89:0/468818830 conn(0x7f7c740fc1b0 msgr2=0x7f7c740fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:26.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.345+0000 7f7c79e17640 1 -- 172.21.15.89:0/468818830 shutdown_connections 2024-09-15T23:15:26.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.345+0000 7f7c79e17640 1 -- 172.21.15.89:0/468818830 wait complete. 2024-09-15T23:15:26.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.345+0000 7f7c79e17640 1 Processor -- start 2024-09-15T23:15:26.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c79e17640 1 -- start start 2024-09-15T23:15:26.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c79e17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c74106560 0x7f7c74197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:26.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c79e17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7410ab90 con 0x7f7c74106560 2024-09-15T23:15:26.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c74106560 0x7f7c74197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:26.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c74106560 0x7f7c74197100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60866/0 (socket says 172.21.15.89:60866) 2024-09-15T23:15:26.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c737fe640 1 -- 172.21.15.89:0/1774750376 learned_addr learned my addr 172.21.15.89:0/1774750376 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:26.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c737fe640 1 -- 172.21.15.89:0/1774750376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c74197640 con 0x7f7c74106560 2024-09-15T23:15:26.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c737fe640 1 --2- 172.21.15.89:0/1774750376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c74106560 0x7f7c74197100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009230 tx=0x7f7c64003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:26.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c70ff9640 1 -- 172.21.15.89:0/1774750376 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c6403f070 con 0x7f7c74106560 2024-09-15T23:15:26.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c70ff9640 1 -- 172.21.15.89:0/1774750376 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c6403bd40 con 0x7f7c74106560 2024-09-15T23:15:26.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c70ff9640 1 -- 172.21.15.89:0/1774750376 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c64035030 con 0x7f7c74106560 2024-09-15T23:15:26.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.346+0000 7f7c79e17640 1 -- 172.21.15.89:0/1774750376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c741978d0 con 0x7f7c74106560 2024-09-15T23:15:26.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.347+0000 7f7c79e17640 1 -- 172.21.15.89:0/1774750376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c7419b9d0 con 0x7f7c74106560 2024-09-15T23:15:26.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.347+0000 7f7c70ff9640 1 -- 172.21.15.89:0/1774750376 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f7c64042070 con 0x7f7c74106560 2024-09-15T23:15:26.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.347+0000 7f7c79e17640 1 -- 172.21.15.89:0/1774750376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c74101bd0 con 0x7f7c74106560 2024-09-15T23:15:26.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.348+0000 7f7c70ff9640 1 --2- 172.21.15.89:0/1774750376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7c5c0392a0 0x7f7c5c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:26.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.348+0000 7f7c70ff9640 1 -- 172.21.15.89:0/1774750376 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7c6406cf50 con 0x7f7c74106560 2024-09-15T23:15:26.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.348+0000 7f7c72ffd640 1 --2- 172.21.15.89:0/1774750376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7c5c0392a0 0x7f7c5c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:26.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.348+0000 7f7c72ffd640 1 --2- 172.21.15.89:0/1774750376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7c5c0392a0 0x7f7c5c03b760 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7c680098c0 tx=0x7f7c68006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:26.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.352+0000 7f7c70ff9640 1 -- 172.21.15.89:0/1774750376 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7c6403a890 con 0x7f7c74106560 2024-09-15T23:15:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.522+0000 7f7c79e17640 1 -- 172.21.15.89:0/1774750376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f7c74198460 con 0x7f7c74106560 2024-09-15T23:15:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.948+0000 7f7c70ff9640 1 -- 172.21.15.89:0/1774750376 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f7c6403a3c0 con 0x7f7c74106560 2024-09-15T23:15:26.950 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2024-09-15T23:15:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.951+0000 7f7c79e17640 1 -- 172.21.15.89:0/1774750376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7c5c0392a0 msgr2=0x7f7c5c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.951+0000 7f7c79e17640 1 --2- 172.21.15.89:0/1774750376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7c5c0392a0 0x7f7c5c03b760 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7c680098c0 tx=0x7f7c68006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.951+0000 7f7c79e17640 1 -- 172.21.15.89:0/1774750376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c74106560 msgr2=0x7f7c74197100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.951+0000 7f7c79e17640 1 --2- 172.21.15.89:0/1774750376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c74106560 0x7f7c74197100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009230 tx=0x7f7c64003d80 comp rx=0 tx=0).stop 2024-09-15T23:15:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.951+0000 7f7c79e17640 1 -- 172.21.15.89:0/1774750376 shutdown_connections 2024-09-15T23:15:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.951+0000 7f7c79e17640 1 --2- 172.21.15.89:0/1774750376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7c5c0392a0 0x7f7c5c03b760 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.951+0000 7f7c79e17640 1 --2- 172.21.15.89:0/1774750376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c74106560 0x7f7c74197100 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.951+0000 7f7c79e17640 1 -- 172.21.15.89:0/1774750376 >> 172.21.15.89:0/1774750376 conn(0x7f7c740fc1b0 msgr2=0x7f7c74069ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.951+0000 7f7c79e17640 1 -- 172.21.15.89:0/1774750376 shutdown_connections 2024-09-15T23:15:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:26.951+0000 7f7c79e17640 1 -- 172.21.15.89:0/1774750376 wait complete. 2024-09-15T23:15:27.251 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:26 smithi089 ceph-mon[30933]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:27.251 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:26 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1774750376' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-15T23:15:27.572 INFO:tasks.cephadm:Adding mon.a on smithi089 2024-09-15T23:15:27.572 INFO:tasks.cephadm:Adding mon.b on smithi120 2024-09-15T23:15:27.572 INFO:tasks.cephadm:Adding mon.c on smithi174 2024-09-15T23:15:27.572 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi089:172.21.15.89=a;smithi120:172.21.15.120=b;smithi174:172.21.15.174=c' 2024-09-15T23:15:27.823 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-15T23:15:27.869 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-15T23:15:28.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:27 smithi089 ceph-mon[30933]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:28.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:27 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1774750376' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-15T23:15:28.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:27 smithi089 ceph-mon[30933]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:28.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:28.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:28.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:28.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:28.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:15:28.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:28.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:28.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.360+0000 7f84857dc640 1 -- 172.21.15.174:0/3218145934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8480101620 msgr2=0x7f8480101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:28.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.360+0000 7f84857dc640 1 --2- 172.21.15.174:0/3218145934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8480101620 0x7f8480101a20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8470009880 tx=0x7f847002a5d0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.360+0000 7f84857dc640 1 -- 172.21.15.174:0/3218145934 shutdown_connections 2024-09-15T23:15:28.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.360+0000 7f84857dc640 1 --2- 172.21.15.174:0/3218145934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8480101620 0x7f8480101a20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.360+0000 7f84857dc640 1 -- 172.21.15.174:0/3218145934 >> 172.21.15.174:0/3218145934 conn(0x7f84800fcdb0 msgr2=0x7f84800ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:28.415 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.360+0000 7f84857dc640 1 -- 172.21.15.174:0/3218145934 shutdown_connections 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.360+0000 7f84857dc640 1 -- 172.21.15.174:0/3218145934 wait complete. 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.360+0000 7f84857dc640 1 Processor -- start 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.361+0000 7f84857dc640 1 -- start start 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.361+0000 7f84857dc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8480101620 0x7f8480196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.361+0000 7f84857dc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8480107730 con 0x7f8480101620 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.361+0000 7f847effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8480101620 0x7f8480196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.361+0000 7f847effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8480101620 0x7f8480196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.174:54746/0 (socket says 172.21.15.174:54746) 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.361+0000 7f847effd640 1 -- 172.21.15.174:0/3762888770 learned_addr learned my addr 172.21.15.174:0/3762888770 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.361+0000 7f847effd640 1 -- 172.21.15.174:0/3762888770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84801972e0 con 0x7f8480101620 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.362+0000 7f847effd640 1 --2- 172.21.15.174:0/3762888770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8480101620 0x7f8480196da0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8470004f60 tx=0x7f84700048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.362+0000 7f8467fff640 1 -- 172.21.15.174:0/3762888770 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f847003f070 con 0x7f8480101620 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.362+0000 7f8467fff640 1 -- 172.21.15.174:0/3762888770 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8470004cc0 con 0x7f8480101620 2024-09-15T23:15:28.416 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.362+0000 7f84857dc640 1 -- 172.21.15.174:0/3762888770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8480197570 con 0x7f8480101620 2024-09-15T23:15:28.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.362+0000 7f8467fff640 1 -- 172.21.15.174:0/3762888770 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8470035030 con 0x7f8480101620 2024-09-15T23:15:28.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.362+0000 7f84857dc640 1 -- 172.21.15.174:0/3762888770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f848019a220 con 0x7f8480101620 2024-09-15T23:15:28.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.363+0000 7f8467fff640 1 -- 172.21.15.174:0/3762888770 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f8470042070 con 0x7f8480101620 2024-09-15T23:15:28.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.363+0000 7f84857dc640 1 -- 172.21.15.174:0/3762888770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8480107eb0 con 0x7f8480101620 2024-09-15T23:15:28.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.363+0000 7f8467fff640 1 --2- 172.21.15.174:0/3762888770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84680392a0 0x7f846803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:28.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.363+0000 7f8467fff640 1 -- 172.21.15.174:0/3762888770 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f847006d2b0 con 0x7f8480101620 2024-09-15T23:15:28.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.363+0000 7f847e7fc640 1 --2- 172.21.15.174:0/3762888770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84680392a0 0x7f846803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:28.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.364+0000 7f847e7fc640 1 --2- 172.21.15.174:0/3762888770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84680392a0 0x7f846803b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f84740098c0 tx=0x7f8474006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:28.417 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.368+0000 7f8467fff640 1 -- 172.21.15.174:0/3762888770 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f84700433b0 con 0x7f8480101620 2024-09-15T23:15:28.566 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.565+0000 7f84857dc640 1 -- 172.21.15.174:0/3762888770 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi120:172.21.15.120=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}) -- 0x7f848002da00 con 0x7f84680392a0 2024-09-15T23:15:28.571 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.570+0000 7f8467fff640 1 -- 172.21.15.174:0/3762888770 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f848002da00 con 0x7f84680392a0 2024-09-15T23:15:28.571 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mon update... 2024-09-15T23:15:28.573 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.572+0000 7f84857dc640 1 -- 172.21.15.174:0/3762888770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84680392a0 msgr2=0x7f846803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:28.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.572+0000 7f84857dc640 1 --2- 172.21.15.174:0/3762888770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84680392a0 0x7f846803b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f84740098c0 tx=0x7f8474006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.572+0000 7f84857dc640 1 -- 172.21.15.174:0/3762888770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8480101620 msgr2=0x7f8480196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:28.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.572+0000 7f84857dc640 1 --2- 172.21.15.174:0/3762888770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8480101620 0x7f8480196da0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8470004f60 tx=0x7f84700048f0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.572+0000 7f84857dc640 1 -- 172.21.15.174:0/3762888770 shutdown_connections 2024-09-15T23:15:28.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.572+0000 7f84857dc640 1 --2- 172.21.15.174:0/3762888770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84680392a0 0x7f846803b760 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.572+0000 7f84857dc640 1 --2- 172.21.15.174:0/3762888770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8480101620 0x7f8480196da0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.572+0000 7f84857dc640 1 -- 172.21.15.174:0/3762888770 >> 172.21.15.174:0/3762888770 conn(0x7f84800fcdb0 msgr2=0x7f84800fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:28.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.572+0000 7f84857dc640 1 -- 172.21.15.174:0/3762888770 shutdown_connections 2024-09-15T23:15:28.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:28.572+0000 7f84857dc640 1 -- 172.21.15.174:0/3762888770 wait complete. 2024-09-15T23:15:29.208 DEBUG:teuthology.orchestra.run.smithi120:mon.b> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.b.service 2024-09-15T23:15:29.211 DEBUG:teuthology.orchestra.run.smithi174:mon.c> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.c.service 2024-09-15T23:15:29.214 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T23:15:29.214 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T23:15:29.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:28 smithi089 ceph-mon[30933]: Updating smithi174:/etc/ceph/ceph.conf 2024-09-15T23:15:29.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:28 smithi089 ceph-mon[30933]: Updating smithi174:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:29.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:28 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:29.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:28 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:29.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:28 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:29.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:28 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:29.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:28 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:29.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:28 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:29.925 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:30.297 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.295+0000 7fe76b4dc640 1 -- 172.21.15.174:0/843504102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe764102730 msgr2=0x7fe764102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:30.297 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.295+0000 7fe76b4dc640 1 --2- 172.21.15.174:0/843504102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe764102730 0x7fe764102b10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe75c009810 tx=0x7fe75c02a540 comp rx=0 tx=0).stop 2024-09-15T23:15:30.297 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.295+0000 7fe76b4dc640 1 -- 172.21.15.174:0/843504102 shutdown_connections 2024-09-15T23:15:30.297 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.295+0000 7fe76b4dc640 1 --2- 172.21.15.174:0/843504102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe764102730 0x7fe764102b10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.297 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.295+0000 7fe76b4dc640 1 -- 172.21.15.174:0/843504102 >> 172.21.15.174:0/843504102 conn(0x7fe764077310 msgr2=0x7fe7640fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:30.298 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.295+0000 7fe76b4dc640 1 -- 172.21.15.174:0/843504102 shutdown_connections 2024-09-15T23:15:30.298 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.295+0000 7fe76b4dc640 1 -- 172.21.15.174:0/843504102 wait complete. 2024-09-15T23:15:30.298 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.296+0000 7fe76b4dc640 1 Processor -- start 2024-09-15T23:15:30.298 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.296+0000 7fe76b4dc640 1 -- start start 2024-09-15T23:15:30.298 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.296+0000 7fe76b4dc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe764102730 0x7fe7641971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:30.298 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.296+0000 7fe76b4dc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe76410bc80 con 0x7fe764102730 2024-09-15T23:15:30.299 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.296+0000 7fe769278640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe764102730 0x7fe7641971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:30.299 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.296+0000 7fe769278640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe764102730 0x7fe7641971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.174:54770/0 (socket says 172.21.15.174:54770) 2024-09-15T23:15:30.299 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.296+0000 7fe769278640 1 -- 172.21.15.174:0/2595210264 learned_addr learned my addr 172.21.15.174:0/2595210264 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.297+0000 7fe769278640 1 -- 172.21.15.174:0/2595210264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7641976f0 con 0x7fe764102730 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.297+0000 7fe769278640 1 --2- 172.21.15.174:0/2595210264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe764102730 0x7fe7641971b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0045e0 tx=0x7fe75c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.297+0000 7fe7627fc640 1 -- 172.21.15.174:0/2595210264 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe75c030e70 con 0x7fe764102730 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.297+0000 7fe7627fc640 1 -- 172.21.15.174:0/2595210264 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe75c0318e0 con 0x7fe764102730 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.297+0000 7fe76b4dc640 1 -- 172.21.15.174:0/2595210264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe764197980 con 0x7fe764102730 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.297+0000 7fe76b4dc640 1 -- 172.21.15.174:0/2595210264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe764197de0 con 0x7fe764102730 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.297+0000 7fe7627fc640 1 -- 172.21.15.174:0/2595210264 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe75c0049c0 con 0x7fe764102730 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.298+0000 7fe7627fc640 1 -- 172.21.15.174:0/2595210264 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fe75c039040 con 0x7fe764102730 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.298+0000 7fe76b4dc640 1 -- 172.21.15.174:0/2595210264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe764069f90 con 0x7fe764102730 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.298+0000 7fe7627fc640 1 --2- 172.21.15.174:0/2595210264 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe74c0392a0 0x7fe74c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.298+0000 7fe7627fc640 1 -- 172.21.15.174:0/2595210264 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe75c06d960 con 0x7fe764102730 2024-09-15T23:15:30.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.298+0000 7fe768a77640 1 --2- 172.21.15.174:0/2595210264 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe74c0392a0 0x7fe74c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:30.301 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.299+0000 7fe768a77640 1 --2- 172.21.15.174:0/2595210264 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe74c0392a0 0x7fe74c03b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe7580098c0 tx=0x7fe758006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:30.304 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.303+0000 7fe7627fc640 1 -- 172.21.15.174:0/2595210264 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe75c040030 con 0x7fe764102730 2024-09-15T23:15:30.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:29 smithi089 ceph-mon[30933]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:15:30.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:29 smithi089 ceph-mon[30933]: Updating smithi174:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T23:15:30.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:29 smithi089 ceph-mon[30933]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi120:172.21.15.120=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:30.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:29 smithi089 ceph-mon[30933]: Saving service mon spec with placement smithi089:172.21.15.89=a;smithi120:172.21.15.120=b;smithi174:172.21.15.174=c;count:3 2024-09-15T23:15:30.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:29 smithi089 ceph-mon[30933]: Deploying daemon mon.c on smithi174 2024-09-15T23:15:30.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:29 smithi089 ceph-mon[30933]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:30.537 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.536+0000 7fe76b4dc640 1 -- 172.21.15.174:0/2595210264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe764198510 con 0x7fe764102730 2024-09-15T23:15:30.538 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.536+0000 7fe7627fc640 1 -- 172.21.15.174:0/2595210264 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fe75c037d70 con 0x7fe764102730 2024-09-15T23:15:30.538 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:15:30.538 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","modified":"2024-09-15T23:14:05.881012Z","created":"2024-09-15T23:14:05.881012Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-15T23:15:30.538 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-09-15T23:15:30.540 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.539+0000 7fe76b4dc640 1 -- 172.21.15.174:0/2595210264 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe74c0392a0 msgr2=0x7fe74c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:30.540 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.539+0000 7fe76b4dc640 1 --2- 172.21.15.174:0/2595210264 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe74c0392a0 0x7fe74c03b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe7580098c0 tx=0x7fe758006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.541 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.539+0000 7fe76b4dc640 1 -- 172.21.15.174:0/2595210264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe764102730 msgr2=0x7fe7641971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:30.541 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.539+0000 7fe76b4dc640 1 --2- 172.21.15.174:0/2595210264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe764102730 0x7fe7641971b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0045e0 tx=0x7fe75c004610 comp rx=0 tx=0).stop 2024-09-15T23:15:30.541 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.539+0000 7fe76b4dc640 1 -- 172.21.15.174:0/2595210264 shutdown_connections 2024-09-15T23:15:30.541 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.539+0000 7fe76b4dc640 1 --2- 172.21.15.174:0/2595210264 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe74c0392a0 0x7fe74c03b760 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.541 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.539+0000 7fe76b4dc640 1 --2- 172.21.15.174:0/2595210264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe764102730 0x7fe7641971b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.541 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.539+0000 7fe76b4dc640 1 -- 172.21.15.174:0/2595210264 >> 172.21.15.174:0/2595210264 conn(0x7fe764077310 msgr2=0x7fe7641054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:30.542 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.539+0000 7fe76b4dc640 1 -- 172.21.15.174:0/2595210264 shutdown_connections 2024-09-15T23:15:30.542 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:30.539+0000 7fe76b4dc640 1 -- 172.21.15.174:0/2595210264 wait complete. 2024-09-15T23:15:31.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:30 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/2595210264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:31 smithi089 ceph-mon[30933]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:32.680 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T23:15:32.680 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T23:15:32.930 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:15:33.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.562+0000 7f5790c97640 1 -- 172.21.15.174:0/2864554229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f578c107660 msgr2=0x7f578c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:33.793 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.562+0000 7f5790c97640 1 --2- 172.21.15.174:0/2864554229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f578c107660 0x7f578c107a40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5774009880 tx=0x7f577402a5d0 comp rx=0 tx=0).stop 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.562+0000 7f5790c97640 1 -- 172.21.15.174:0/2864554229 shutdown_connections 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.562+0000 7f5790c97640 1 --2- 172.21.15.174:0/2864554229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f578c107660 0x7f578c107a40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.562+0000 7f5790c97640 1 -- 172.21.15.174:0/2864554229 >> 172.21.15.174:0/2864554229 conn(0x7f578c0fd320 msgr2=0x7f578c0ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.562+0000 7f5790c97640 1 -- 172.21.15.174:0/2864554229 shutdown_connections 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.562+0000 7f5790c97640 1 -- 172.21.15.174:0/2864554229 wait complete. 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.563+0000 7f5790c97640 1 Processor -- start 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.563+0000 7f5790c97640 1 -- start start 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.563+0000 7f5790c97640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f578c107660 0x7f578c197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.563+0000 7f5790c97640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f578c10bd20 con 0x7f578c107660 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.563+0000 7f578a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f578c107660 0x7f578c197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:33.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.563+0000 7f578a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f578c107660 0x7f578c197290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.174:54792/0 (socket says 172.21.15.174:54792) 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.563+0000 7f578a59c640 1 -- 172.21.15.174:0/2418643628 learned_addr learned my addr 172.21.15.174:0/2418643628 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.564+0000 7f578a59c640 1 -- 172.21.15.174:0/2418643628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578c1977d0 con 0x7f578c107660 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.564+0000 7f578a59c640 1 --2- 172.21.15.174:0/2418643628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f578c107660 0x7f578c197290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5774009230 tx=0x7f5774003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.564+0000 7f57837fe640 1 -- 172.21.15.174:0/2418643628 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f577403f070 con 0x7f578c107660 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.564+0000 7f5790c97640 1 -- 172.21.15.174:0/2418643628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f578c197ac0 con 0x7f578c107660 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.564+0000 7f57837fe640 1 -- 172.21.15.174:0/2418643628 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f577403bc80 con 0x7f578c107660 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.564+0000 7f57837fe640 1 -- 172.21.15.174:0/2418643628 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5774035030 con 0x7f578c107660 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.564+0000 7f5790c97640 1 -- 172.21.15.174:0/2418643628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f578c19bb60 con 0x7f578c107660 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.565+0000 7f57837fe640 1 -- 172.21.15.174:0/2418643628 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f5774042070 con 0x7f578c107660 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.565+0000 7f5790c97640 1 -- 172.21.15.174:0/2418643628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f578c102da0 con 0x7f578c107660 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.565+0000 7f57837fe640 1 --2- 172.21.15.174:0/2418643628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57780392a0 0x7f577803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.565+0000 7f57837fe640 1 -- 172.21.15.174:0/2418643628 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f577406cfe0 con 0x7f578c107660 2024-09-15T23:15:33.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.565+0000 7f5789d9b640 1 --2- 172.21.15.174:0/2418643628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57780392a0 0x7f577803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:33.796 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.566+0000 7f5789d9b640 1 --2- 172.21.15.174:0/2418643628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57780392a0 0x7f577803b760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f577c0098c0 tx=0x7f577c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:33.796 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.570+0000 7f57837fe640 1 -- 172.21.15.174:0/2418643628 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5774031e20 con 0x7f578c107660 2024-09-15T23:15:33.815 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.814+0000 7f5790c97640 1 -- 172.21.15.174:0/2418643628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f578c1985f0 con 0x7f578c107660 2024-09-15T23:15:33.815 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.814+0000 7f57837fe640 1 -- 172.21.15.174:0/2418643628 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f57740372d0 con 0x7f578c107660 2024-09-15T23:15:33.816 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:15:33.816 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","modified":"2024-09-15T23:14:05.881012Z","created":"2024-09-15T23:14:05.881012Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-15T23:15:33.816 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-09-15T23:15:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.816+0000 7f5790c97640 1 -- 172.21.15.174:0/2418643628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57780392a0 msgr2=0x7f577803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.816+0000 7f5790c97640 1 --2- 172.21.15.174:0/2418643628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57780392a0 0x7f577803b760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f577c0098c0 tx=0x7f577c006db0 comp rx=0 tx=0).stop 2024-09-15T23:15:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.816+0000 7f5790c97640 1 -- 172.21.15.174:0/2418643628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f578c107660 msgr2=0x7f578c197290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.816+0000 7f5790c97640 1 --2- 172.21.15.174:0/2418643628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f578c107660 0x7f578c197290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5774009230 tx=0x7f5774003d40 comp rx=0 tx=0).stop 2024-09-15T23:15:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.816+0000 7f5790c97640 1 -- 172.21.15.174:0/2418643628 shutdown_connections 2024-09-15T23:15:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.816+0000 7f5790c97640 1 --2- 172.21.15.174:0/2418643628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57780392a0 0x7f577803b760 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.816+0000 7f5790c97640 1 --2- 172.21.15.174:0/2418643628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f578c107660 0x7f578c197290 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.816+0000 7f5790c97640 1 -- 172.21.15.174:0/2418643628 >> 172.21.15.174:0/2418643628 conn(0x7f578c0fd320 msgr2=0x7f578c105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.816+0000 7f5790c97640 1 -- 172.21.15.174:0/2418643628 shutdown_connections 2024-09-15T23:15:33.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:33.817+0000 7f5790c97640 1 -- 172.21.15.174:0/2418643628 wait complete. 2024-09-15T23:15:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:33 smithi089 ceph-mon[30933]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:33 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/2418643628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:34.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:34 smithi174 systemd[1]: Started Ceph mon.c for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:15:36.037 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T23:15:36.037 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:15:39.618 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:15:39.618 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.618 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: mon.a calling monitor election 2024-09-15T23:15:39.618 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:39.618 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.618 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.618 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: mon.c calling monitor election 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: monmap epoch 2 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: last_changed 2024-09-15T23:15:34.293185+0000 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: created 2024-09-15T23:14:05.881012+0000 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: min_mon_release 19 (squid) 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: election_strategy: 1 2024-09-15T23:15:39.619 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-15T23:15:39.620 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-09-15T23:15:39.620 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: fsmap 2024-09-15T23:15:39.620 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:39.620 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: mgrmap e13: x(active, since 38s) 2024-09-15T23:15:39.620 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: overall HEALTH_OK 2024-09-15T23:15:39.620 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:39.620 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:39.620 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:39.620 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:39.620 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:39.689 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:15:39.689 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: mon.a calling monitor election 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: mon.c calling monitor election 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: monmap epoch 2 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:39.690 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: last_changed 2024-09-15T23:15:34.293185+0000 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: created 2024-09-15T23:14:05.881012+0000 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: min_mon_release 19 (squid) 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: election_strategy: 1 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: fsmap 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: mgrmap e13: x(active, since 38s) 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: overall HEALTH_OK 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:39.691 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:39.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.715+0000 7f9a07823640 1 -- 172.21.15.174:0/3138535720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec003530 msgr2=0x7f99ec0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.715+0000 7f9a07823640 1 --2- 172.21.15.174:0/3138535720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec003530 0x7f99ec0059c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9a001016c0 tx=0x7f99f002bce0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.715+0000 7f9a07823640 1 -- 172.21.15.174:0/3138535720 shutdown_connections 2024-09-15T23:15:39.718 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.715+0000 7f9a07823640 1 --2- 172.21.15.174:0/3138535720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99ec003530 0x7f99ec0059c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.719 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.715+0000 7f9a07823640 1 -- 172.21.15.174:0/3138535720 >> 172.21.15.174:0/3138535720 conn(0x7f9a000fc280 msgr2=0x7f9a000fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:39.719 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.715+0000 7f9a07823640 1 -- 172.21.15.174:0/3138535720 shutdown_connections 2024-09-15T23:15:39.719 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.715+0000 7f9a07823640 1 -- 172.21.15.174:0/3138535720 wait complete. 2024-09-15T23:15:39.719 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.716+0000 7f9a07823640 1 Processor -- start 2024-09-15T23:15:39.719 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.716+0000 7f9a07823640 1 -- start start 2024-09-15T23:15:39.719 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.716+0000 7f9a07823640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec003530 0x7f9a00198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.719 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.716+0000 7f9a07823640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a00198690 0x7f9a00198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.719 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.716+0000 7f9a07823640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9a00108690 con 0x7f9a00198690 2024-09-15T23:15:39.719 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.716+0000 7f9a07823640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a00108510 con 0x7f99ec003530 2024-09-15T23:15:39.719 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.716+0000 7f9a055bf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec003530 0x7f9a00198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a055bf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec003530 0x7f9a00198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:60576/0 (socket says 172.21.15.174:60576) 2024-09-15T23:15:39.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a055bf640 1 -- 172.21.15.174:0/2905000095 learned_addr learned my addr 172.21.15.174:0/2905000095 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:15:39.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a055bf640 1 -- 172.21.15.174:0/2905000095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec003530 msgr2=0x7f9a00198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-15T23:15:39.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a055bf640 1 -- 172.21.15.174:0/2905000095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec003530 msgr2=0x7f9a00198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T23:15:39.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a055bf640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec003530 0x7f9a00198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T23:15:39.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a04dbe640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a00198690 0x7f9a00198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a055bf640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec003530 0x7f9a00198150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:15:39.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a04dbe640 1 -- 172.21.15.174:0/2905000095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec003530 msgr2=0x7f9a00198150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:15:39.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a04dbe640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec003530 0x7f9a00198150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a04dbe640 1 -- 172.21.15.174:0/2905000095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a001a11c0 con 0x7f9a00198690 2024-09-15T23:15:39.722 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a04dbe640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a00198690 0x7f9a00198af0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00b400 tx=0x7f99fc00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:39.722 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f99f67fc640 1 -- 172.21.15.174:0/2905000095 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f99fc011630 con 0x7f9a00198690 2024-09-15T23:15:39.722 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f99f67fc640 1 -- 172.21.15.174:0/2905000095 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99fc011cb0 con 0x7f9a00198690 2024-09-15T23:15:39.722 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.717+0000 7f9a07823640 1 -- 172.21.15.174:0/2905000095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a001a1390 con 0x7f9a00198690 2024-09-15T23:15:39.722 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.718+0000 7f99f67fc640 1 -- 172.21.15.174:0/2905000095 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f99fc010d80 con 0x7f9a00198690 2024-09-15T23:15:39.722 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.718+0000 7f9a07823640 1 -- 172.21.15.174:0/2905000095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a001a18f0 con 0x7f9a00198690 2024-09-15T23:15:39.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.718+0000 7f99f67fc640 1 -- 172.21.15.174:0/2905000095 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f99fc018020 con 0x7f9a00198690 2024-09-15T23:15:39.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.719+0000 7f99f67fc640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99e4039090 0x7f99e403b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.719+0000 7f9a07823640 1 -- 172.21.15.174:0/2905000095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a00101cc0 con 0x7f9a00198690 2024-09-15T23:15:39.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.719+0000 7f99f67fc640 1 -- 172.21.15.174:0/2905000095 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f99fc04bf10 con 0x7f9a00198690 2024-09-15T23:15:39.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.719+0000 7f9a055bf640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99e4039090 0x7f99e403b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.719+0000 7f9a055bf640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99e4039090 0x7f99e403b550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f99f0005f70 tx=0x7f99f002c6e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:39.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.724+0000 7f99f67fc640 1 -- 172.21.15.174:0/2905000095 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f99fc01f020 con 0x7f9a00198690 2024-09-15T23:15:39.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.957+0000 7f9a07823640 1 -- 172.21.15.174:0/2905000095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9a00109840 con 0x7f9a00198690 2024-09-15T23:15:39.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.958+0000 7f99f67fc640 1 -- 172.21.15.174:0/2905000095 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f99fc018310 con 0x7f9a00198690 2024-09-15T23:15:39.960 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:15:39.960 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":2,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","modified":"2024-09-15T23:15:34.293185Z","created":"2024-09-15T23:14:05.881012Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-15T23:15:39.960 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 2 2024-09-15T23:15:39.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 -- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99e4039090 msgr2=0x7f99e403b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99e4039090 0x7f99e403b550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f99f0005f70 tx=0x7f99f002c6e0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 -- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a00198690 msgr2=0x7f9a00198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a00198690 0x7f9a00198af0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00b400 tx=0x7f99fc00b8c0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 -- 172.21.15.174:0/2905000095 shutdown_connections 2024-09-15T23:15:39.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99e4039090 0x7f99e403b550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9a00198690 0x7f9a00198af0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 --2- 172.21.15.174:0/2905000095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99ec003530 0x7f9a00198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 -- 172.21.15.174:0/2905000095 >> 172.21.15.174:0/2905000095 conn(0x7f9a000fc280 msgr2=0x7f9a000fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:39.962 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 -- 172.21.15.174:0/2905000095 shutdown_connections 2024-09-15T23:15:39.963 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:39.961+0000 7f9a07823640 1 -- 172.21.15.174:0/2905000095 wait complete. 2024-09-15T23:15:40.546 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:40 smithi174 ceph-mon[35586]: Deploying daemon mon.b on smithi120 2024-09-15T23:15:40.547 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:40 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/2905000095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:40.547 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:40 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:40.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:40 smithi089 ceph-mon[30933]: Deploying daemon mon.b on smithi120 2024-09-15T23:15:40.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:40 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/2905000095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:40.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:40 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:41.576 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T23:15:41.576 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T23:15:41.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:41 smithi174 ceph-mon[35586]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:41.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:41 smithi089 ceph-mon[30933]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:41.830 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:15:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.219+0000 7f08fe178640 1 -- 172.21.15.174:0/3008110544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08f8106630 msgr2=0x7f08f8106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:42.222 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.219+0000 7f08fe178640 1 --2- 172.21.15.174:0/3008110544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08f8106630 0x7f08f8106a10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f08e8009880 tx=0x7f08e802a5d0 comp rx=0 tx=0).stop 2024-09-15T23:15:42.222 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.219+0000 7f08fe178640 1 -- 172.21.15.174:0/3008110544 shutdown_connections 2024-09-15T23:15:42.222 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.219+0000 7f08fe178640 1 --2- 172.21.15.174:0/3008110544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08f8106630 0x7f08f8106a10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:42.222 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.219+0000 7f08fe178640 1 -- 172.21.15.174:0/3008110544 >> 172.21.15.174:0/3008110544 conn(0x7f08f80fc220 msgr2=0x7f08f80fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:42.222 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.219+0000 7f08fe178640 1 -- 172.21.15.174:0/3008110544 shutdown_connections 2024-09-15T23:15:42.222 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.219+0000 7f08fe178640 1 -- 172.21.15.174:0/3008110544 wait complete. 2024-09-15T23:15:42.222 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.220+0000 7f08fe178640 1 Processor -- start 2024-09-15T23:15:42.222 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08fe178640 1 -- start start 2024-09-15T23:15:42.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08fe178640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08f8106630 0x7f08f81962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08fe178640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08f8196820 0x7f08f819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08fe178640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f08f810ac60 con 0x7f08f8196820 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08fe178640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f08f810aae0 con 0x7f08f8106630 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08f8196820 0x7f08f819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08f8106630 0x7f08f81962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08f8196820 0x7f08f819abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.174:47936/0 (socket says 172.21.15.174:47936) 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f6ffd640 1 -- 172.21.15.174:0/1091264047 learned_addr learned my addr 172.21.15.174:0/1091264047 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f77fe640 1 -- 172.21.15.174:0/1091264047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08f8106630 msgr2=0x7f08f81962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f77fe640 1 -- 172.21.15.174:0/1091264047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08f8106630 msgr2=0x7f08f81962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f77fe640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08f8106630 0x7f08f81962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f77fe640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08f8106630 0x7f08f81962e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:15:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f6ffd640 1 -- 172.21.15.174:0/1091264047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08f8106630 msgr2=0x7f08f81962e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:15:42.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f6ffd640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08f8106630 0x7f08f81962e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:42.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.221+0000 7f08f6ffd640 1 -- 172.21.15.174:0/1091264047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f8196e40 con 0x7f08f8196820 2024-09-15T23:15:42.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.222+0000 7f08f6ffd640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08f8196820 0x7f08f819abb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f08ec00b4a0 tx=0x7f08ec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:42.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.222+0000 7f08f4ff9640 1 -- 172.21.15.174:0/1091264047 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f08ec010610 con 0x7f08f8196820 2024-09-15T23:15:42.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.222+0000 7f08f4ff9640 1 -- 172.21.15.174:0/1091264047 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08ec010c90 con 0x7f08f8196820 2024-09-15T23:15:42.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.222+0000 7f08f4ff9640 1 -- 172.21.15.174:0/1091264047 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f08ec017e60 con 0x7f08f8196820 2024-09-15T23:15:42.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.222+0000 7f08fe178640 1 -- 172.21.15.174:0/1091264047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08f819b0f0 con 0x7f08f8196820 2024-09-15T23:15:42.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.222+0000 7f08fe178640 1 -- 172.21.15.174:0/1091264047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08f819b630 con 0x7f08f8196820 2024-09-15T23:15:42.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.223+0000 7f08f4ff9640 1 -- 172.21.15.174:0/1091264047 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f08ec017070 con 0x7f08f8196820 2024-09-15T23:15:42.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.223+0000 7f08fe178640 1 -- 172.21.15.174:0/1091264047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08f8101ca0 con 0x7f08f8196820 2024-09-15T23:15:42.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.223+0000 7f08f4ff9640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08e0039040 0x7f08e003b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:42.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.223+0000 7f08f4ff9640 1 -- 172.21.15.174:0/1091264047 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f08ec00d040 con 0x7f08f8196820 2024-09-15T23:15:42.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.223+0000 7f08f77fe640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08e0039040 0x7f08e003b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:42.226 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.224+0000 7f08f77fe640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08e0039040 0x7f08e003b500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f08e80314e0 tx=0x7f08e80058b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.228+0000 7f08f4ff9640 1 -- 172.21.15.174:0/1091264047 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f08ec01e030 con 0x7f08f8196820 2024-09-15T23:15:42.456 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.454+0000 7f08fe178640 1 -- 172.21.15.174:0/1091264047 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f08f819b8f0 con 0x7f08f8196820 2024-09-15T23:15:42.456 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.455+0000 7f08f4ff9640 1 -- 172.21.15.174:0/1091264047 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f08ec01f510 con 0x7f08f8196820 2024-09-15T23:15:42.456 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:15:42.457 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":2,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","modified":"2024-09-15T23:15:34.293185Z","created":"2024-09-15T23:14:05.881012Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-15T23:15:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 2 2024-09-15T23:15:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 -- 172.21.15.174:0/1091264047 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08e0039040 msgr2=0x7f08e003b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08e0039040 0x7f08e003b500 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f08e80314e0 tx=0x7f08e80058b0 comp rx=0 tx=0).stop 2024-09-15T23:15:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 -- 172.21.15.174:0/1091264047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08f8196820 msgr2=0x7f08f819abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08f8196820 0x7f08f819abb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f08ec00b4a0 tx=0x7f08ec00b960 comp rx=0 tx=0).stop 2024-09-15T23:15:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 -- 172.21.15.174:0/1091264047 shutdown_connections 2024-09-15T23:15:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08e0039040 0x7f08e003b500 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08f8196820 0x7f08f819abb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 --2- 172.21.15.174:0/1091264047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08f8106630 0x7f08f81962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 -- 172.21.15.174:0/1091264047 >> 172.21.15.174:0/1091264047 conn(0x7f08f80fc220 msgr2=0x7f08f80fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 -- 172.21.15.174:0/1091264047 shutdown_connections 2024-09-15T23:15:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:42.457+0000 7f08fe178640 1 -- 172.21.15.174:0/1091264047 wait complete. 2024-09-15T23:15:42.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:42 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/1091264047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:42.958 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:42 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/1091264047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:43.188 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 podman[35231]: 2024-09-15 23:15:43.165694036 +0000 UTC m=+0.353837471 container start fc28fc27ee928358192cebfc32c26606625735ec2812e95e754fcec43fcd2dcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-15T23:15:43.189 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: set uid:gid to 167:167 (ceph:ceph) 2024-09-15T23:15:43.189 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable), process ceph-mon, pid 2 2024-09-15T23:15:43.189 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pidfile_write: ignore empty --pid-file 2024-09-15T23:15:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: load: jerasure load: lrc 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: RocksDB version: 7.9.2 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Git sha 0 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Compile date 2024-09-15 15:04:35 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: DB SUMMARY 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: DB Session ID: VE7N6Y1FJ4MXE0Z7OM2S 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: CURRENT file: CURRENT 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: IDENTITY file: IDENTITY 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.error_if_exists: 0 2024-09-15T23:15:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.create_if_missing: 0 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.paranoid_checks: 1 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.env: 0x5596e3b11940 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.fs: PosixFileSystem 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.info_log: 0x5596e5de56e0 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_file_opening_threads: 16 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.statistics: (nil) 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.use_fsync: 0 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_log_file_size: 0 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-15T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.keep_log_file_num: 1000 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.recycle_log_file_num: 0 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.allow_fallocate: 1 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.allow_mmap_reads: 0 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.allow_mmap_writes: 0 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.use_direct_reads: 0 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.create_missing_column_families: 0 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.db_log_dir: 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.wal_dir: 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-15T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-15T23:15:43.477 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-15T23:15:43.477 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-15T23:15:43.478 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-15T23:15:43.478 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-15T23:15:43.478 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.advise_random_on_open: 1 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.db_write_buffer_size: 0 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.write_buffer_manager: 0x5596e5df4dc0 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.rate_limiter: (nil) 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.wal_recovery_mode: 2 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.enable_thread_tracking: 0 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.enable_pipelined_write: 0 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.unordered_write: 0 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-15T23:15:43.479 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.row_cache: None 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.wal_filter: None 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.allow_ingest_behind: 0 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.two_write_queues: 0 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.manual_wal_flush: 0 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.wal_compression: 0 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.atomic_flush: 0 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.log_readahead_size: 0 2024-09-15T23:15:43.480 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.best_efforts_recovery: 0 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.allow_data_in_errors: 0 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.db_host_id: __hostname__ 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_background_jobs: 2 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_background_compactions: -1 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_subcompactions: 1 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-15T23:15:43.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_total_wal_size: 0 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_open_files: -1 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bytes_per_sync: 0 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_readahead_size: 0 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_background_flushes: -1 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Compression algorithms supported: 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: kZSTD supported: 0 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: kXpressCompression supported: 0 2024-09-15T23:15:43.482 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: kBZip2Compression supported: 0 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: kLZ4Compression supported: 1 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: kZlibCompression supported: 1 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: kLZ4HCCompression supported: 1 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: kSnappyCompression supported: 1 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.merge_operator: 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_filter: None 2024-09-15T23:15:43.483 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_filter_factory: None 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.sst_partitioner_factory: None 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5596e5de5320) 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout: cache_index_and_filter_blocks: 1 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout: pin_top_level_index_and_filter: 1 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout: index_type: 0 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout: data_block_index_type: 0 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout: index_shortening: 1 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-15T23:15:43.484 INFO:journalctl@ceph.mon.b.smithi120.stdout: checksum: 4 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: no_block_cache: 0 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_cache: 0x5596e5dd91f0 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_cache_name: BinnedLRUCache 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_cache_options: 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: capacity : 536870912 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: num_shard_bits : 4 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: strict_capacity_limit : 0 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: high_pri_pool_ratio: 0.000 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_cache_compressed: (nil) 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: persistent_cache: (nil) 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_size: 4096 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_size_deviation: 10 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_restart_interval: 16 2024-09-15T23:15:43.485 INFO:journalctl@ceph.mon.b.smithi120.stdout: index_block_restart_interval: 1 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: metadata_block_size: 4096 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: partition_filters: 0 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: use_delta_encoding: 1 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: filter_policy: bloomfilter 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: whole_key_filtering: 1 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: verify_compression: 0 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: read_amp_bytes_per_bit: 0 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: format_version: 5 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: enable_index_compression: 1 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: block_align: 0 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: max_auto_readahead_size: 262144 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: prepopulate_block_cache: 0 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: initial_auto_readahead_size: 8192 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout: num_file_reads_for_auto_readahead: 2 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.write_buffer_size: 33554432 2024-09-15T23:15:43.486 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_write_buffer_number: 2 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compression: NoCompression 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bottommost_compression: Disabled 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.prefix_extractor: nullptr 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.num_levels: 7 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-15T23:15:43.487 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compression_opts.level: 32767 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compression_opts.strategy: 0 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compression_opts.enabled: false 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-15T23:15:43.488 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.target_file_size_base: 67108864 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-15T23:15:43.489 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.arena_block_size: 1048576 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.disable_auto_compactions: 0 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-15T23:15:43.490 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.inplace_update_support: 0 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.bloom_locality: 0 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.max_successive_merges: 0 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.paranoid_file_checks: 0 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.force_consistency_checks: 1 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.report_bg_io_stats: 0 2024-09-15T23:15:43.491 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.ttl: 2592000 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.enable_blob_files: false 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.min_blob_size: 0 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.blob_file_size: 268435456 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.blob_file_starting_level: 0 2024-09-15T23:15:43.492 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 120a63aa-8e9c-4041-8bec-23dae8917118 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726442143193284, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726442143193690, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726442143, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "120a63aa-8e9c-4041-8bec-23dae8917118", "db_session_id": "VE7N6Y1FJ4MXE0Z7OM2S", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726442143193791, "job": 1, "event": "recovery_finished"} 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5596e5e06e00 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: DB pointer 0x5596e5f02000 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-15T23:15:43.493 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: ** DB Stats ** 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: ** Compaction Stats [default] ** 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-15T23:15:43.494 INFO:journalctl@ceph.mon.b.smithi120.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: ** Compaction Stats [default] ** 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-15T23:15:43.495 INFO:journalctl@ceph.mon.b.smithi120.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: Block cache BinnedLRUCache@0x5596e5dd91f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: using public_addr v2:172.21.15.120:0/0 -> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-09-15T23:15:43.496 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: starting mon.b rank -1 at public addrs [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] at bind addrs [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(???) e0 preinit fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).mds e1 new map 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout: e1 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout: btime 2024-09-15T23:14:08:706210+0000 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout: legacy client fscid: -1 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout: 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout: No filesystems configured 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-15T23:15:43.497 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: Updating smithi120:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.498 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: Added host smithi174 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1774750376' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1774750376' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.499 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: Updating smithi174:/etc/ceph/ceph.conf 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: Updating smithi174:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:43.500 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: Updating smithi174:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi089:172.21.15.89=a;smithi120:172.21.15.120=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: Saving service mon spec with placement smithi089:172.21.15.89=a;smithi120:172.21.15.120=b;smithi174:172.21.15.174=c;count:3 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: Deploying daemon mon.c on smithi174 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/2595210264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/2418643628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.a calling monitor election 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:43.501 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.c calling monitor election 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: monmap epoch 2 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: last_changed 2024-09-15T23:15:34.293185+0000 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: created 2024-09-15T23:14:05.881012+0000 2024-09-15T23:15:43.502 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: min_mon_release 19 (squid) 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: election_strategy: 1 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: fsmap 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mgrmap e13: x(active, since 38s) 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: overall HEALTH_OK 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: Deploying daemon mon.b on smithi120 2024-09-15T23:15:43.503 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/2905000095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:43.504 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:43.504 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:43.504 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/1091264047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:43.504 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 ceph-mon[35250]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-15T23:15:43.504 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 bash[35231]: fc28fc27ee928358192cebfc32c26606625735ec2812e95e754fcec43fcd2dcc 2024-09-15T23:15:43.504 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:43 smithi120 systemd[1]: Started Ceph mon.b for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:15:44.091 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T23:15:44.092 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T23:15:44.344 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:15:48.364 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.362+0000 7f3e29d55640 1 -- 172.21.15.174:0/4194815996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e10003530 msgr2=0x7f3e100059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.362+0000 7f3e29d55640 1 --2- 172.21.15.174:0/4194815996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e10003530 0x7f3e100059c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3e2406a1a0 tx=0x7f3e1402c3f0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.362+0000 7f3e29d55640 1 -- 172.21.15.174:0/4194815996 shutdown_connections 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.362+0000 7f3e29d55640 1 --2- 172.21.15.174:0/4194815996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e10003530 0x7f3e100059c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.362+0000 7f3e29d55640 1 -- 172.21.15.174:0/4194815996 >> 172.21.15.174:0/4194815996 conn(0x7f3e240fc280 msgr2=0x7f3e240fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.362+0000 7f3e29d55640 1 -- 172.21.15.174:0/4194815996 shutdown_connections 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.362+0000 7f3e29d55640 1 -- 172.21.15.174:0/4194815996 wait complete. 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e29d55640 1 Processor -- start 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e29d55640 1 -- start start 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e29d55640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e10003530 0x7f3e24100540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e29d55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e24100a80 0x7f3e24100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e29d55640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e241a35a0 0x7f3e241a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.365 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e29d55640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241097a0 con 0x7f3e24100a80 2024-09-15T23:15:48.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e29d55640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24109620 con 0x7f3e241a35a0 2024-09-15T23:15:48.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e29d55640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24109920 con 0x7f3e10003530 2024-09-15T23:15:48.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e1bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e24100a80 0x7f3e24100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e237fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e10003530 0x7f3e24100540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e1bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e24100a80 0x7f3e24100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.174:49864/0 (socket says 172.21.15.174:49864) 2024-09-15T23:15:48.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e1bfff640 1 -- 172.21.15.174:0/2446403393 learned_addr learned my addr 172.21.15.174:0/2446403393 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:15:48.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.363+0000 7f3e23fff640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e241a35a0 0x7f3e241a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.367 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e23fff640 1 -- 172.21.15.174:0/2446403393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e241a35a0 msgr2=0x7f3e241a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e23fff640 1 -- 172.21.15.174:0/2446403393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e241a35a0 msgr2=0x7f3e241a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e237fe640 1 -- 172.21.15.174:0/2446403393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e10003530 msgr2=0x7f3e24100540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e23fff640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e241a35a0 0x7f3e241a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e237fe640 1 -- 172.21.15.174:0/2446403393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e10003530 msgr2=0x7f3e24100540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e237fe640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e10003530 0x7f3e24100540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e23fff640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e241a35a0 0x7f3e241a5990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e237fe640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e10003530 0x7f3e24100540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e1bfff640 1 -- 172.21.15.174:0/2446403393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e10003530 msgr2=0x7f3e24100540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e1bfff640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e10003530 0x7f3e24100540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e1bfff640 1 -- 172.21.15.174:0/2446403393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e241a35a0 msgr2=0x7f3e241a5990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e1bfff640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e241a35a0 0x7f3e241a5990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e1bfff640 1 -- 172.21.15.174:0/2446403393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e241a6060 con 0x7f3e24100a80 2024-09-15T23:15:48.368 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e1bfff640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e24100a80 0x7f3e24100ee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c00b520 tx=0x7f3e0c00b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e217fa640 1 -- 172.21.15.174:0/2446403393 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c004230 con 0x7f3e24100a80 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e217fa640 1 -- 172.21.15.174:0/2446403393 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e0c0043d0 con 0x7f3e24100a80 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e29d55640 1 -- 172.21.15.174:0/2446403393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e241a6350 con 0x7f3e24100a80 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e217fa640 1 -- 172.21.15.174:0/2446403393 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c003d60 con 0x7f3e24100a80 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.364+0000 7f3e29d55640 1 -- 172.21.15.174:0/2446403393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e241a68e0 con 0x7f3e24100a80 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.365+0000 7f3e217fa640 1 -- 172.21.15.174:0/2446403393 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f3e0c00e070 con 0x7f3e24100a80 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.366+0000 7f3e217fa640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e04039730 0x7f3e0403bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.366+0000 7f3e217fa640 1 -- 172.21.15.174:0/2446403393 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3e0c04bd40 con 0x7f3e24100a80 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.366+0000 7f3e29d55640 1 -- 172.21.15.174:0/2446403393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e2410a8d0 con 0x7f3e24100a80 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.366+0000 7f3e237fe640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e04039730 0x7f3e0403bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.369 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.366+0000 7f3e237fe640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e04039730 0x7f3e0403bbf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3e14014cc0 tx=0x7f3e14005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:48.373 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.372+0000 7f3e217fa640 1 -- 172.21.15.174:0/2446403393 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3e0c014020 con 0x7f3e24100a80 2024-09-15T23:15:48.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:15:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: mon.a calling monitor election 2024-09-15T23:15:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: mon.c calling monitor election 2024-09-15T23:15:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: monmap epoch 3 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: last_changed 2024-09-15T23:15:43.218163+0000 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: created 2024-09-15T23:14:05.881012+0000 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: min_mon_release 19 (squid) 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: election_strategy: 1 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: fsmap 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: mgrmap e13: x(active, since 47s) 2024-09-15T23:15:48.563 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: overall HEALTH_OK 2024-09-15T23:15:48.563 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:48.563 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:48.563 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:48.563 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:48.617 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.616+0000 7f3e29d55640 1 -- 172.21.15.174:0/2446403393 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3e241a6ba0 con 0x7f3e24100a80 2024-09-15T23:15:48.618 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.616+0000 7f3e217fa640 1 -- 172.21.15.174:0/2446403393 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f3e0c020350 con 0x7f3e24100a80 2024-09-15T23:15:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:15:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: mon.a calling monitor election 2024-09-15T23:15:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: mon.c calling monitor election 2024-09-15T23:15:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: monmap epoch 3 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: last_changed 2024-09-15T23:15:43.218163+0000 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: created 2024-09-15T23:14:05.881012+0000 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: min_mon_release 19 (squid) 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: election_strategy: 1 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-09-15T23:15:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-09-15T23:15:48.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: fsmap 2024-09-15T23:15:48.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:48.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: mgrmap e13: x(active, since 47s) 2024-09-15T23:15:48.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: overall HEALTH_OK 2024-09-15T23:15:48.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:48.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:48.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:48.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:48.620 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:15:48.620 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":3,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","modified":"2024-09-15T23:15:43.218163Z","created":"2024-09-15T23:14:05.881012Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:3300","nonce":0},{"type":"v1","addr":"172.21.15.120:6789","nonce":0}]},"addr":"172.21.15.120:6789/0","public_addr":"172.21.15.120:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-15T23:15:48.621 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 3 2024-09-15T23:15:48.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 -- 172.21.15.174:0/2446403393 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e04039730 msgr2=0x7f3e0403bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e04039730 0x7f3e0403bbf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3e14014cc0 tx=0x7f3e14005f60 comp rx=0 tx=0).stop 2024-09-15T23:15:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 -- 172.21.15.174:0/2446403393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e24100a80 msgr2=0x7f3e24100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e24100a80 0x7f3e24100ee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c00b520 tx=0x7f3e0c00b9e0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 -- 172.21.15.174:0/2446403393 shutdown_connections 2024-09-15T23:15:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e04039730 0x7f3e0403bbf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e241a35a0 0x7f3e241a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e24100a80 0x7f3e24100ee0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 --2- 172.21.15.174:0/2446403393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e10003530 0x7f3e24100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 -- 172.21.15.174:0/2446403393 >> 172.21.15.174:0/2446403393 conn(0x7f3e240fc280 msgr2=0x7f3e240fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 -- 172.21.15.174:0/2446403393 shutdown_connections 2024-09-15T23:15:48.625 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:48.620+0000 7f3e29d55640 1 -- 172.21.15.174:0/2446403393 wait complete. 2024-09-15T23:15:49.203 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-15T23:15:49.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-15T23:15:49.464 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:49 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/2446403393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:49 smithi174 ceph-mon[35586]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:49 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/2446403393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:49 smithi089 ceph-mon[30933]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:49.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.881+0000 7fcc541e3640 1 -- 172.21.15.89:0/976084949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc4c100540 msgr2=0x7fcc4c100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:49.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.881+0000 7fcc541e3640 1 --2- 172.21.15.89:0/976084949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc4c100540 0x7fcc4c100920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009880 tx=0x7fcc3c02a670 comp rx=0 tx=0).stop 2024-09-15T23:15:49.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.881+0000 7fcc541e3640 1 -- 172.21.15.89:0/976084949 shutdown_connections 2024-09-15T23:15:49.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.881+0000 7fcc541e3640 1 --2- 172.21.15.89:0/976084949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc4c100540 0x7fcc4c100920 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.881+0000 7fcc541e3640 1 -- 172.21.15.89:0/976084949 >> 172.21.15.89:0/976084949 conn(0x7fcc4c0fc1b0 msgr2=0x7fcc4c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.881+0000 7fcc541e3640 1 -- 172.21.15.89:0/976084949 shutdown_connections 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.881+0000 7fcc541e3640 1 -- 172.21.15.89:0/976084949 wait complete. 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.882+0000 7fcc541e3640 1 Processor -- start 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.882+0000 7fcc541e3640 1 -- start start 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.882+0000 7fcc541e3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100540 0x7fcc4c1a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.882+0000 7fcc541e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc4c1a3c60 0x7fcc4c19d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.882+0000 7fcc541e3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc4c19de20 0x7fcc4c19e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.882+0000 7fcc541e3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c112280 con 0x7fcc4c1a3c60 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.882+0000 7fcc541e3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c112100 con 0x7fcc4c19de20 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.882+0000 7fcc541e3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c112400 con 0x7fcc4c100540 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.882+0000 7fcc5177e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc4c1a3c60 0x7fcc4c19d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:49.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc5177e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc4c1a3c60 0x7fcc4c19d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56580/0 (socket says 172.21.15.89:56580) 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc5177e640 1 -- 172.21.15.89:0/3866314672 learned_addr learned my addr 172.21.15.89:0/3866314672 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc52780640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc4c19de20 0x7fcc4c19e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc5177e640 1 -- 172.21.15.89:0/3866314672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100540 msgr2=0x7fcc4c1a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc5177e640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100540 0x7fcc4c1a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc5177e640 1 -- 172.21.15.89:0/3866314672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc4c19de20 msgr2=0x7fcc4c19e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc5177e640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc4c19de20 0x7fcc4c19e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc5177e640 1 -- 172.21.15.89:0/3866314672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4c1aa5f0 con 0x7fcc4c1a3c60 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc52780640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc4c19de20 0x7fcc4c19e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc5177e640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc4c1a3c60 0x7fcc4c19d8e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800b520 tx=0x7fcc4800b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc42ffd640 1 -- 172.21.15.89:0/3866314672 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc48004230 con 0x7fcc4c1a3c60 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc42ffd640 1 -- 172.21.15.89:0/3866314672 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc480043d0 con 0x7fcc4c1a3c60 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc541e3640 1 -- 172.21.15.89:0/3866314672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4c1aa820 con 0x7fcc4c1a3c60 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc42ffd640 1 -- 172.21.15.89:0/3866314672 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc48003d60 con 0x7fcc4c1a3c60 2024-09-15T23:15:49.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.883+0000 7fcc541e3640 1 -- 172.21.15.89:0/3866314672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc4c1aadb0 con 0x7fcc4c1a3c60 2024-09-15T23:15:49.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.884+0000 7fcc42ffd640 1 -- 172.21.15.89:0/3866314672 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fcc4800e070 con 0x7fcc4c1a3c60 2024-09-15T23:15:49.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.884+0000 7fcc42ffd640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcc38039730 0x7fcc3803bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:49.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.884+0000 7fcc541e3640 1 -- 172.21.15.89:0/3866314672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc4c1009a0 con 0x7fcc4c1a3c60 2024-09-15T23:15:49.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.884+0000 7fcc42ffd640 1 -- 172.21.15.89:0/3866314672 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcc4804cc60 con 0x7fcc4c1a3c60 2024-09-15T23:15:49.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.884+0000 7fcc51f7f640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcc38039730 0x7fcc3803bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:49.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.886+0000 7fcc51f7f640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcc38039730 0x7fcc3803bbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c19ea20 tx=0x7fcc3c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:49.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:49.890+0000 7fcc42ffd640 1 -- 172.21.15.89:0/3866314672 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcc48014020 con 0x7fcc4c1a3c60 2024-09-15T23:15:50.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.061+0000 7fcc541e3640 1 -- 172.21.15.89:0/3866314672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fcc4c1ab100 con 0x7fcc4c1a3c60 2024-09-15T23:15:50.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.061+0000 7fcc42ffd640 1 -- 172.21.15.89:0/3866314672 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fcc48020350 con 0x7fcc4c1a3c60 2024-09-15T23:15:50.063 INFO:teuthology.orchestra.run.smithi089.stdout:# minimal ceph.conf for 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:50.063 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-09-15T23:15:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:50.063 INFO:teuthology.orchestra.run.smithi089.stdout: mon_host = [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.063+0000 7fcc541e3640 1 -- 172.21.15.89:0/3866314672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcc38039730 msgr2=0x7fcc3803bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.063+0000 7fcc541e3640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcc38039730 0x7fcc3803bbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c19ea20 tx=0x7fcc3c000bc0 comp rx=0 tx=0).stop 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.063+0000 7fcc541e3640 1 -- 172.21.15.89:0/3866314672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc4c1a3c60 msgr2=0x7fcc4c19d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.063+0000 7fcc541e3640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc4c1a3c60 0x7fcc4c19d8e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800b520 tx=0x7fcc4800b9e0 comp rx=0 tx=0).stop 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.063+0000 7fcc541e3640 1 -- 172.21.15.89:0/3866314672 shutdown_connections 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.063+0000 7fcc541e3640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcc38039730 0x7fcc3803bbf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.063+0000 7fcc541e3640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc4c19de20 0x7fcc4c19e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.063+0000 7fcc541e3640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcc4c1a3c60 0x7fcc4c19d8e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.063+0000 7fcc541e3640 1 --2- 172.21.15.89:0/3866314672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc4c100540 0x7fcc4c1a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.063+0000 7fcc541e3640 1 -- 172.21.15.89:0/3866314672 >> 172.21.15.89:0/3866314672 conn(0x7fcc4c0fc1b0 msgr2=0x7fcc4c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.064+0000 7fcc541e3640 1 -- 172.21.15.89:0/3866314672 shutdown_connections 2024-09-15T23:15:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:50.064+0000 7fcc541e3640 1 -- 172.21.15.89:0/3866314672 wait complete. 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: mon.a calling monitor election 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: mon.c calling monitor election 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:50.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: monmap epoch 3 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: last_changed 2024-09-15T23:15:43.218163+0000 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: created 2024-09-15T23:14:05.881012+0000 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: min_mon_release 19 (squid) 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: election_strategy: 1 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: fsmap 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:50.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: mgrmap e13: x(active, since 47s) 2024-09-15T23:15:50.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: overall HEALTH_OK 2024-09-15T23:15:50.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:50.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:50.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:50.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:50.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:50.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/2446403393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:15:50.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:50.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:50 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:50.782 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-15T23:15:50.783 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:15:50.783 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.conf 2024-09-15T23:15:50.810 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:15:50.810 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:15:50.879 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:15:50.879 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.conf 2024-09-15T23:15:50.906 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:15:50.906 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:15:50.971 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:15:50.972 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.conf 2024-09-15T23:15:50.998 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:15:50.998 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:15:51.064 INFO:tasks.cephadm:Adding mgr.x on smithi089 2024-09-15T23:15:51.064 INFO:tasks.cephadm:Adding mgr.y on smithi120 2024-09-15T23:15:51.064 INFO:tasks.cephadm:Adding mgr.z on smithi174 2024-09-15T23:15:51.065 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch apply mgr '3;smithi089=x;smithi120=y;smithi174=z' 2024-09-15T23:15:51.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: mon.b calling monitor election 2024-09-15T23:15:51.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: mon.b calling monitor election 2024-09-15T23:15:51.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: mon.c calling monitor election 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: mon.a calling monitor election 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: monmap epoch 3 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: last_changed 2024-09-15T23:15:43.218163+0000 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: created 2024-09-15T23:14:05.881012+0000 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: min_mon_release 19 (squid) 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: election_strategy: 1 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: fsmap 2024-09-15T23:15:51.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: mgrmap e13: x(active, since 49s) 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: overall HEALTH_OK 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:51.226 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:15:51.226 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:51.226 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.304 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: mon.b calling monitor election 2024-09-15T23:15:51.304 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: mon.b calling monitor election 2024-09-15T23:15:51.304 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: mon.c calling monitor election 2024-09-15T23:15:51.304 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: mon.a calling monitor election 2024-09-15T23:15:51.304 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: monmap epoch 3 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: last_changed 2024-09-15T23:15:43.218163+0000 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: created 2024-09-15T23:14:05.881012+0000 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: min_mon_release 19 (squid) 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: election_strategy: 1 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: fsmap 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: mgrmap e13: x(active, since 49s) 2024-09-15T23:15:51.305 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: overall HEALTH_OK 2024-09-15T23:15:51.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.307 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.307 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:51.307 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:15:51.307 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:51.307 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.339 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:15:51.460 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: mon.b calling monitor election 2024-09-15T23:15:51.460 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: mon.b calling monitor election 2024-09-15T23:15:51.461 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: mon.c calling monitor election 2024-09-15T23:15:51.461 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: mon.a calling monitor election 2024-09-15T23:15:51.461 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-15T23:15:51.461 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: monmap epoch 3 2024-09-15T23:15:51.461 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:15:51.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: last_changed 2024-09-15T23:15:43.218163+0000 2024-09-15T23:15:51.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: created 2024-09-15T23:14:05.881012+0000 2024-09-15T23:15:51.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: min_mon_release 19 (squid) 2024-09-15T23:15:51.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: election_strategy: 1 2024-09-15T23:15:51.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: 0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-15T23:15:51.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-09-15T23:15:51.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: 2: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.b 2024-09-15T23:15:51.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: fsmap 2024-09-15T23:15:51.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:15:51.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: mgrmap e13: x(active, since 49s) 2024-09-15T23:15:51.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: overall HEALTH_OK 2024-09-15T23:15:51.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:51.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:15:51.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:51.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:51.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.812+0000 7fba61499640 1 -- 172.21.15.174:0/1195195912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba5c1005f0 msgr2=0x7fba5c1009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:51.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.812+0000 7fba61499640 1 --2- 172.21.15.174:0/1195195912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba5c1005f0 0x7fba5c1009f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009880 tx=0x7fba4c02a5d0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.812+0000 7fba61499640 1 -- 172.21.15.174:0/1195195912 shutdown_connections 2024-09-15T23:15:51.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.812+0000 7fba61499640 1 --2- 172.21.15.174:0/1195195912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba5c1005f0 0x7fba5c1009f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.812+0000 7fba61499640 1 -- 172.21.15.174:0/1195195912 >> 172.21.15.174:0/1195195912 conn(0x7fba5c0fbda0 msgr2=0x7fba5c0fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:51.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.812+0000 7fba61499640 1 -- 172.21.15.174:0/1195195912 shutdown_connections 2024-09-15T23:15:51.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.812+0000 7fba61499640 1 -- 172.21.15.174:0/1195195912 wait complete. 2024-09-15T23:15:51.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba61499640 1 Processor -- start 2024-09-15T23:15:51.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba61499640 1 -- start start 2024-09-15T23:15:51.816 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba61499640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba5c1005f0 0x7fba5c196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba61499640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba5c197530 0x7fba5c19c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba61499640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba5c1979b0 0x7fba5c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba61499640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c1066e0 con 0x7fba5c1005f0 2024-09-15T23:15:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba61499640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c106560 con 0x7fba5c1979b0 2024-09-15T23:15:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba61499640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c106860 con 0x7fba5c197530 2024-09-15T23:15:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba5a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba5c197530 0x7fba5c19c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba5a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba5c197530 0x7fba5c19c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:53598/0 (socket says 172.21.15.174:53598) 2024-09-15T23:15:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba5a7fc640 1 -- 172.21.15.174:0/3472252051 learned_addr learned my addr 172.21.15.174:0/3472252051 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:15:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.813+0000 7fba5affd640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba5c1005f0 0x7fba5c196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:51.817 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba5b7fe640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba5c1979b0 0x7fba5c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:51.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba5a7fc640 1 -- 172.21.15.174:0/3472252051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba5c1979b0 msgr2=0x7fba5c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:51.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba5a7fc640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba5c1979b0 0x7fba5c19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba5a7fc640 1 -- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba5c1005f0 msgr2=0x7fba5c196ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:51.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba5a7fc640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba5c1005f0 0x7fba5c196ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba5a7fc640 1 -- 172.21.15.174:0/3472252051 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5c19f3f0 con 0x7fba5c197530 2024-09-15T23:15:51.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba5b7fe640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba5c1979b0 0x7fba5c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:51.818 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba5affd640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba5c1005f0 0x7fba5c196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:15:51.819 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba5a7fc640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba5c197530 0x7fba5c19c5a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba5000b520 tx=0x7fba5000b9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:51.819 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba43fff640 1 -- 172.21.15.174:0/3472252051 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50004230 con 0x7fba5c197530 2024-09-15T23:15:51.819 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba43fff640 1 -- 172.21.15.174:0/3472252051 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba500043d0 con 0x7fba5c197530 2024-09-15T23:15:51.819 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba61499640 1 -- 172.21.15.174:0/3472252051 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba5c19f6e0 con 0x7fba5c197530 2024-09-15T23:15:51.819 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba43fff640 1 -- 172.21.15.174:0/3472252051 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50003d60 con 0x7fba5c197530 2024-09-15T23:15:51.819 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.814+0000 7fba61499640 1 -- 172.21.15.174:0/3472252051 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba5c19fca0 con 0x7fba5c197530 2024-09-15T23:15:51.821 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.815+0000 7fba43fff640 1 -- 172.21.15.174:0/3472252051 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fba5000e070 con 0x7fba5c197530 2024-09-15T23:15:51.821 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.815+0000 7fba43fff640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba44039730 0x7fba4403bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:51.821 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.815+0000 7fba61499640 1 -- 172.21.15.174:0/3472252051 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba5c1904b0 con 0x7fba5c197530 2024-09-15T23:15:51.821 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.815+0000 7fba43fff640 1 -- 172.21.15.174:0/3472252051 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fba5004be30 con 0x7fba5c197530 2024-09-15T23:15:51.821 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.815+0000 7fba5affd640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba44039730 0x7fba4403bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:51.822 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.816+0000 7fba5affd640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba44039730 0x7fba4403bbf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009230 tx=0x7fba4c0315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:51.822 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.820+0000 7fba43fff640 1 -- 172.21.15.174:0/3472252051 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fba50014020 con 0x7fba5c197530 2024-09-15T23:15:52.000 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:51.998+0000 7fba61499640 1 -- 172.21.15.174:0/3472252051 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi089=x;smithi120=y;smithi174=z", "target": ["mon-mgr", ""]}) -- 0x7fba5c0626c0 con 0x7fba44039730 2024-09-15T23:15:52.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.005+0000 7fba43fff640 1 -- 172.21.15.174:0/3472252051 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fba5c0626c0 con 0x7fba44039730 2024-09-15T23:15:52.007 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mgr update... 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.007+0000 7fba61499640 1 -- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba44039730 msgr2=0x7fba4403bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.007+0000 7fba61499640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba44039730 0x7fba4403bbf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009230 tx=0x7fba4c0315d0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.007+0000 7fba61499640 1 -- 172.21.15.174:0/3472252051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba5c197530 msgr2=0x7fba5c19c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.007+0000 7fba61499640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba5c197530 0x7fba5c19c5a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fba5000b520 tx=0x7fba5000b9e0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.007+0000 7fba61499640 1 -- 172.21.15.174:0/3472252051 shutdown_connections 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.007+0000 7fba61499640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba44039730 0x7fba4403bbf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.007+0000 7fba61499640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba5c1979b0 0x7fba5c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.007+0000 7fba61499640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba5c197530 0x7fba5c19c5a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.007+0000 7fba61499640 1 --2- 172.21.15.174:0/3472252051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba5c1005f0 0x7fba5c196ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.007+0000 7fba61499640 1 -- 172.21.15.174:0/3472252051 >> 172.21.15.174:0/3472252051 conn(0x7fba5c0fbda0 msgr2=0x7fba5c06ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.008+0000 7fba61499640 1 -- 172.21.15.174:0/3472252051 shutdown_connections 2024-09-15T23:15:52.009 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:15:52.008+0000 7fba61499640 1 -- 172.21.15.174:0/3472252051 wait complete. 2024-09-15T23:15:52.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:52 smithi089 ceph-mon[30933]: Updating smithi089:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:52.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:52 smithi089 ceph-mon[30933]: Updating smithi174:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:52.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:52 smithi089 ceph-mon[30933]: Updating smithi120:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:52.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:52 smithi089 ceph-mon[30933]: Reconfiguring mon.a (unknown last config time)... 2024-09-15T23:15:52.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:52 smithi089 ceph-mon[30933]: Reconfiguring daemon mon.a on smithi089 2024-09-15T23:15:52.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:52 smithi089 ceph-mon[30933]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:52.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:52 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:52.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:52 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:52 smithi120 ceph-mon[35250]: Updating smithi089:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:52 smithi120 ceph-mon[35250]: Updating smithi174:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:52 smithi120 ceph-mon[35250]: Updating smithi120:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:52 smithi120 ceph-mon[35250]: Reconfiguring mon.a (unknown last config time)... 2024-09-15T23:15:52.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:52 smithi120 ceph-mon[35250]: Reconfiguring daemon mon.a on smithi089 2024-09-15T23:15:52.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:52 smithi120 ceph-mon[35250]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:52.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:52 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:52.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:52 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:52 smithi174 ceph-mon[35586]: Updating smithi089:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:52 smithi174 ceph-mon[35586]: Updating smithi174:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:52 smithi174 ceph-mon[35586]: Updating smithi120:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:15:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:52 smithi174 ceph-mon[35586]: Reconfiguring mon.a (unknown last config time)... 2024-09-15T23:15:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:52 smithi174 ceph-mon[35586]: Reconfiguring daemon mon.a on smithi089 2024-09-15T23:15:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:52 smithi174 ceph-mon[35586]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:52 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:15:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:52 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:52.630 DEBUG:teuthology.orchestra.run.smithi120:mgr.y> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.y.service 2024-09-15T23:15:52.633 DEBUG:teuthology.orchestra.run.smithi174:mgr.z> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.z.service 2024-09-15T23:15:52.635 INFO:tasks.cephadm:Deploying OSDs... 2024-09-15T23:15:52.635 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:15:52.635 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T23:15:52.651 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T23:15:52.651 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-09-15T23:15:52.707 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T23:15:52.707 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:52.707 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-15T23:15:52.707 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:52.707 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:52.707 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-15 23:15:11.681958078 +0000 2024-09-15T23:15:52.707 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-15 23:11:04.830235175 +0000 2024-09-15T23:15:52.707 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-15 23:11:04.830235175 +0000 2024-09-15T23:15:52.707 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-15 23:11:04.830235175 +0000 2024-09-15T23:15:52.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T23:15:52.771 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-15T23:15:52.772 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-15T23:15:52.772 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000210395 s, 2.4 MB/s 2024-09-15T23:15:52.773 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T23:15:52.829 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-09-15T23:15:52.885 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T23:15:52.885 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:52.885 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-15T23:15:52.885 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:52.885 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:52.885 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-15 23:15:11.707957494 +0000 2024-09-15T23:15:52.885 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-15 23:11:05.148227846 +0000 2024-09-15T23:15:52.885 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-15 23:11:05.148227846 +0000 2024-09-15T23:15:52.886 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-15 23:11:05.148227846 +0000 2024-09-15T23:15:52.886 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T23:15:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-15T23:15:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-15T23:15:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000179948 s, 2.8 MB/s 2024-09-15T23:15:52.950 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T23:15:53.007 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-09-15T23:15:53.063 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T23:15:53.063 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:53.063 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-15T23:15:53.063 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:53.064 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:53.064 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-15 23:15:11.733956910 +0000 2024-09-15T23:15:53.064 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-15 23:11:05.471220401 +0000 2024-09-15T23:15:53.064 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-15 23:11:05.471220401 +0000 2024-09-15T23:15:53.064 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-15 23:11:05.471220401 +0000 2024-09-15T23:15:53.065 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T23:15:53.126 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-15T23:15:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-15T23:15:53.127 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000207475 s, 2.5 MB/s 2024-09-15T23:15:53.128 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T23:15:53.184 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-09-15T23:15:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T23:15:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:53.240 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-15T23:15:53.241 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:53.241 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:53.241 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-15 23:15:11.766956168 +0000 2024-09-15T23:15:53.241 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-15 23:11:05.807212657 +0000 2024-09-15T23:15:53.241 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-15 23:11:05.807212657 +0000 2024-09-15T23:15:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-15 23:11:05.807212657 +0000 2024-09-15T23:15:53.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T23:15:53.301 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:53 smithi089 ceph-mon[30933]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi089=x;smithi120=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:53.302 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:53 smithi089 ceph-mon[30933]: Saving service mgr spec with placement smithi089=x;smithi120=y;smithi174=z;count:3 2024-09-15T23:15:53.302 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:53 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:53.302 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:53 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:53.302 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:53 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:15:53.302 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:53 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:15:53.302 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:53 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-15T23:15:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-15T23:15:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000211124 s, 2.4 MB/s 2024-09-15T23:15:53.305 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T23:15:53.361 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-09-15T23:15:53.362 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T23:15:53.379 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T23:15:53.379 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 2024-09-15T23:15:53.435 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T23:15:53.435 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:53.435 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-15T23:15:53.435 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:53.436 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:53.436 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-09-15 23:15:42.627813974 +0000 2024-09-15T23:15:53.436 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-09-15 23:11:18.477169403 +0000 2024-09-15T23:15:53.436 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-09-15 23:11:18.477169403 +0000 2024-09-15T23:15:53.436 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-09-15 23:11:18.477169403 +0000 2024-09-15T23:15:53.436 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T23:15:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:53 smithi120 ceph-mon[35250]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi089=x;smithi120=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:53.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:53 smithi120 ceph-mon[35250]: Saving service mgr spec with placement smithi089=x;smithi120=y;smithi174=z;count:3 2024-09-15T23:15:53.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:53 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:53.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:53 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:53.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:53 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:15:53.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:53 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:15:53.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:53 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:53 smithi174 ceph-mon[35586]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi089=x;smithi120=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:53 smithi174 ceph-mon[35586]: Saving service mgr spec with placement smithi089=x;smithi120=y;smithi174=z;count:3 2024-09-15T23:15:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:53 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:53 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:53 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:15:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:53 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:15:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:53 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:53.497 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-09-15T23:15:53.497 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-09-15T23:15:53.498 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000269504 s, 1.9 MB/s 2024-09-15T23:15:53.499 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T23:15:53.555 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 2024-09-15T23:15:53.612 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T23:15:53.612 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:53.612 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-15T23:15:53.612 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:53.612 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:53.612 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-09-15 23:15:42.627813974 +0000 2024-09-15T23:15:53.612 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-09-15 23:11:18.815161761 +0000 2024-09-15T23:15:53.613 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-09-15 23:11:18.815161761 +0000 2024-09-15T23:15:53.613 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-09-15 23:11:18.815161761 +0000 2024-09-15T23:15:53.613 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T23:15:53.677 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-09-15T23:15:53.678 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-09-15T23:15:53.678 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000184339 s, 2.8 MB/s 2024-09-15T23:15:53.679 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T23:15:53.736 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 2024-09-15T23:15:53.790 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T23:15:53.790 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:53.790 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-15T23:15:53.790 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:53.790 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:53.791 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-09-15 23:15:42.628813952 +0000 2024-09-15T23:15:53.791 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-09-15 23:11:19.126154730 +0000 2024-09-15T23:15:53.791 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-09-15 23:11:19.126154730 +0000 2024-09-15T23:15:53.791 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-09-15 23:11:19.126154730 +0000 2024-09-15T23:15:53.791 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T23:15:53.854 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-09-15T23:15:53.854 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-09-15T23:15:53.854 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000212167 s, 2.4 MB/s 2024-09-15T23:15:53.855 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T23:15:53.911 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 2024-09-15T23:15:53.967 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T23:15:53.968 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:53.968 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-15T23:15:53.968 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:53.968 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:53.968 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-09-15 23:15:42.628813952 +0000 2024-09-15T23:15:53.968 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-09-15 23:11:19.476146817 +0000 2024-09-15T23:15:53.968 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-09-15 23:11:19.476146817 +0000 2024-09-15T23:15:53.968 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-09-15 23:11:19.476146817 +0000 2024-09-15T23:15:53.969 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T23:15:54.032 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-09-15T23:15:54.032 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-09-15T23:15:54.032 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.00017327 s, 3.0 MB/s 2024-09-15T23:15:54.034 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T23:15:54.066 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:54 smithi120 ceph-mon[35250]: Reconfiguring mgr.x (unknown last config time)... 2024-09-15T23:15:54.066 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:54 smithi120 ceph-mon[35250]: Reconfiguring daemon mgr.x on smithi089 2024-09-15T23:15:54.066 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:54 smithi120 ceph-mon[35250]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:54.066 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:54.066 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:54.067 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:54.067 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:15:54.067 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:54.083 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-15T23:15:54.083 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T23:15:54.103 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T23:15:54.103 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-09-15T23:15:54.163 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T23:15:54.163 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:54.163 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-15T23:15:54.164 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:54.164 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:54.164 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-15 23:15:33.281407306 +0000 2024-09-15T23:15:54.164 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-15 23:11:20.041305902 +0000 2024-09-15T23:15:54.164 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-15 23:11:20.041305902 +0000 2024-09-15T23:15:54.164 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-15 23:11:20.041305902 +0000 2024-09-15T23:15:54.164 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T23:15:54.233 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-15T23:15:54.233 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-15T23:15:54.233 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000270553 s, 1.9 MB/s 2024-09-15T23:15:54.234 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T23:15:54.295 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-09-15T23:15:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:54 smithi089 ceph-mon[30933]: Reconfiguring mgr.x (unknown last config time)... 2024-09-15T23:15:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:54 smithi089 ceph-mon[30933]: Reconfiguring daemon mgr.x on smithi089 2024-09-15T23:15:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:54 smithi089 ceph-mon[30933]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:54.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:54.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:15:54.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:54.353 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T23:15:54.354 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:54.354 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-15T23:15:54.354 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:54.354 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:54.354 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-15 23:15:33.282407293 +0000 2024-09-15T23:15:54.354 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-15 23:11:20.393301404 +0000 2024-09-15T23:15:54.354 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-15 23:11:20.393301404 +0000 2024-09-15T23:15:54.355 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-15 23:11:20.393301404 +0000 2024-09-15T23:15:54.355 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T23:15:54.420 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:54 smithi174 ceph-mon[35586]: Reconfiguring mgr.x (unknown last config time)... 2024-09-15T23:15:54.420 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:54 smithi174 ceph-mon[35586]: Reconfiguring daemon mgr.x on smithi089 2024-09-15T23:15:54.420 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:54 smithi174 ceph-mon[35586]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:54.420 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:54.420 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:54.420 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:54.420 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:15:54.420 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:54.422 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-15T23:15:54.422 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-15T23:15:54.422 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000235889 s, 2.2 MB/s 2024-09-15T23:15:54.424 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T23:15:54.483 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-09-15T23:15:54.542 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T23:15:54.542 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:54.542 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-15T23:15:54.542 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:54.542 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:54.542 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-15 23:15:33.282407293 +0000 2024-09-15T23:15:54.542 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-15 23:11:20.734297047 +0000 2024-09-15T23:15:54.542 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-15 23:11:20.734297047 +0000 2024-09-15T23:15:54.543 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-15 23:11:20.734297047 +0000 2024-09-15T23:15:54.543 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T23:15:54.609 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-15T23:15:54.609 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-15T23:15:54.609 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000196644 s, 2.6 MB/s 2024-09-15T23:15:54.610 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T23:15:54.669 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-09-15T23:15:54.727 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T23:15:54.727 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:15:54.727 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-09-15T23:15:54.727 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:15:54.727 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:15:54.728 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-15 23:15:33.282407293 +0000 2024-09-15T23:15:54.728 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-15 23:11:21.108292269 +0000 2024-09-15T23:15:54.728 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-15 23:11:21.108292269 +0000 2024-09-15T23:15:54.728 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-15 23:11:21.108292269 +0000 2024-09-15T23:15:54.728 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T23:15:54.793 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-15T23:15:54.793 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-15T23:15:54.793 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000200053 s, 2.6 MB/s 2024-09-15T23:15:54.794 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T23:15:54.852 INFO:tasks.cephadm:Deploying osd.0 on smithi089 with /dev/vg_nvme/lv_4... 2024-09-15T23:15:54.852 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-15T23:15:55.082 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:55 smithi174 ceph-mon[35586]: Reconfiguring mon.b (monmap changed)... 2024-09-15T23:15:55.082 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:55 smithi174 ceph-mon[35586]: Reconfiguring daemon mon.b on smithi120 2024-09-15T23:15:55.082 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:55.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:55.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:55.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:15:55.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:55.102 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:55.206 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:55 smithi089 ceph-mon[30933]: Reconfiguring mon.b (monmap changed)... 2024-09-15T23:15:55.206 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:55 smithi089 ceph-mon[30933]: Reconfiguring daemon mon.b on smithi120 2024-09-15T23:15:55.207 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:55.207 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:55.207 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:55.207 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:15:55.207 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:55 smithi120 ceph-mon[35250]: Reconfiguring mon.b (monmap changed)... 2024-09-15T23:15:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:55 smithi120 ceph-mon[35250]: Reconfiguring daemon mon.b on smithi120 2024-09-15T23:15:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:15:55.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:15:55.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:56 smithi089 ceph-mon[30933]: Reconfiguring mon.c (monmap changed)... 2024-09-15T23:15:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:56 smithi089 ceph-mon[30933]: Reconfiguring daemon mon.c on smithi174 2024-09-15T23:15:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:56 smithi089 ceph-mon[30933]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:56 smithi120 ceph-mon[35250]: Reconfiguring mon.c (monmap changed)... 2024-09-15T23:15:56.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:56 smithi120 ceph-mon[35250]: Reconfiguring daemon mon.c on smithi174 2024-09-15T23:15:56.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:56 smithi120 ceph-mon[35250]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:56 smithi174 ceph-mon[35586]: Reconfiguring mon.c (monmap changed)... 2024-09-15T23:15:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:56 smithi174 ceph-mon[35586]: Reconfiguring daemon mon.c on smithi174 2024-09-15T23:15:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:56 smithi174 ceph-mon[35586]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:57.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: Deploying daemon mgr.z on smithi174 2024-09-15T23:15:57.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:57 smithi174 ceph-mon[35586]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:15:57.453 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:57.453 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:57.453 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:57.453 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:57.453 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:57.453 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:57.454 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:15:57.454 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:15:57.454 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:15:57.454 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:57.454 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: Deploying daemon mgr.z on smithi174 2024-09-15T23:15:57.454 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:57 smithi089 ceph-mon[30933]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:57.455 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi089:vg_nvme/lv_4 2024-09-15T23:15:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:15:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:15:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:15:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: Deploying daemon mgr.z on smithi174 2024-09-15T23:15:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:57 smithi120 ceph-mon[35250]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:58.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.095+0000 7f2239992640 1 -- 172.21.15.89:0/3901975032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22341037c0 msgr2=0x7f223410a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.095+0000 7f2239992640 1 --2- 172.21.15.89:0/3901975032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22341037c0 0x7f223410a050 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f222c009ca0 tx=0x7f222c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.095+0000 7f2239992640 1 -- 172.21.15.89:0/3901975032 shutdown_connections 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.095+0000 7f2239992640 1 --2- 172.21.15.89:0/3901975032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22341037c0 0x7f223410a050 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.095+0000 7f2239992640 1 --2- 172.21.15.89:0/3901975032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2234102e00 0x7f2234103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.095+0000 7f2239992640 1 --2- 172.21.15.89:0/3901975032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234101c00 0x7f2234102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.095+0000 7f2239992640 1 -- 172.21.15.89:0/3901975032 >> 172.21.15.89:0/3901975032 conn(0x7f22340fd390 msgr2=0x7f22340ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.095+0000 7f2239992640 1 -- 172.21.15.89:0/3901975032 shutdown_connections 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.095+0000 7f2239992640 1 -- 172.21.15.89:0/3901975032 wait complete. 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f2239992640 1 Processor -- start 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f2239992640 1 -- start start 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f2239992640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234101c00 0x7f2234198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:58.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f2239992640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2234102e00 0x7f2234198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f2239992640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22341037c0 0x7f223419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f2239992640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f223410d980 con 0x7f22341037c0 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f2239992640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f223410d800 con 0x7f2234102e00 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f2239992640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f223410db00 con 0x7f2234101c00 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f22337fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22341037c0 0x7f223419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f22337fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22341037c0 0x7f223419f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49896/0 (socket says 172.21.15.89:49896) 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f22337fe640 1 -- 172.21.15.89:0/3736888791 learned_addr learned my addr 172.21.15.89:0/3736888791 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f2232ffd640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234101c00 0x7f2234198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.096+0000 7f22327fc640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2234102e00 0x7f2234198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f22337fe640 1 -- 172.21.15.89:0/3736888791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234101c00 msgr2=0x7f2234198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f22337fe640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234101c00 0x7f2234198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f22337fe640 1 -- 172.21.15.89:0/3736888791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2234102e00 msgr2=0x7f2234198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f22337fe640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2234102e00 0x7f2234198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f22337fe640 1 -- 172.21.15.89:0/3736888791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223419fc60 con 0x7f22341037c0 2024-09-15T23:15:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f2232ffd640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234101c00 0x7f2234198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f22337fe640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22341037c0 0x7f223419f5f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f222c0098d0 tx=0x7f222c02f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f2238990640 1 -- 172.21.15.89:0/3736888791 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c02fc30 con 0x7f22341037c0 2024-09-15T23:15:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f2238990640 1 -- 172.21.15.89:0/3736888791 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f222c003e50 con 0x7f22341037c0 2024-09-15T23:15:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f2239992640 1 -- 172.21.15.89:0/3736888791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223419fef0 con 0x7f22341037c0 2024-09-15T23:15:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f2238990640 1 -- 172.21.15.89:0/3736888791 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222c004810 con 0x7f22341037c0 2024-09-15T23:15:58.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.097+0000 7f2239992640 1 -- 172.21.15.89:0/3736888791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22341a0420 con 0x7f22341037c0 2024-09-15T23:15:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.098+0000 7f2238990640 1 -- 172.21.15.89:0/3736888791 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f222c0305c0 con 0x7f22341037c0 2024-09-15T23:15:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.098+0000 7f2239992640 1 -- 172.21.15.89:0/3736888791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2234109050 con 0x7f22341037c0 2024-09-15T23:15:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.098+0000 7f2238990640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f221c039500 0x7f221c03b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.098+0000 7f2238990640 1 -- 172.21.15.89:0/3736888791 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f222c06db70 con 0x7f22341037c0 2024-09-15T23:15:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.098+0000 7f2232ffd640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f221c039500 0x7f221c03b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:58.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.099+0000 7f2232ffd640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f221c039500 0x7f221c03b9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2234102c60 tx=0x7f2224008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:58.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.103+0000 7f2238990640 1 -- 172.21.15.89:0/3736888791 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f222c039bf0 con 0x7f22341037c0 2024-09-15T23:15:58.234 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 systemd[1]: Starting Ceph mgr.z for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:15:58.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:15:58.277+0000 7f2239992640 1 -- 172.21.15.89:0/3736888791 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2234062630 con 0x7f221c039500 2024-09-15T23:15:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:15:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:15:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:58.486 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 podman[37389]: 2024-09-15 23:15:58.136036544 +0000 UTC m=+0.015892689 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:15:58.486 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 podman[37389]: 2024-09-15 23:15:58.2512648 +0000 UTC m=+0.131120931 container create 86570c43e9480ce5bff3a92fea76a40a2906d4fc8ad2e827b0ffa1c99b20eb57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-15T23:15:58.486 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 podman[37389]: 2024-09-15 23:15:58.453537302 +0000 UTC m=+0.333393446 container init 86570c43e9480ce5bff3a92fea76a40a2906d4fc8ad2e827b0ffa1c99b20eb57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:15:58.486 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 podman[37389]: 2024-09-15 23:15:58.456274243 +0000 UTC m=+0.336130381 container start 86570c43e9480ce5bff3a92fea76a40a2906d4fc8ad2e827b0ffa1c99b20eb57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-15T23:15:58.486 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 ceph-mgr[37408]: -- 172.21.15.174:0/3392862469 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a6a1ac30e0 con 0x55a6a1aa1000 2024-09-15T23:15:58.681 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:15:58.681 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:15:58.681 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:58.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:15:58.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:15:58.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:58.848 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 bash[37389]: 86570c43e9480ce5bff3a92fea76a40a2906d4fc8ad2e827b0ffa1c99b20eb57 2024-09-15T23:15:58.848 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 systemd[1]: Started Ceph mgr.z for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:15:58.848 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:15:58.576+0000 7f8a8e695040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T23:15:58.848 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:15:58.636+0000 7f8a8e695040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T23:15:58.848 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:15:58.781+0000 7f8a8e695040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T23:15:58.848 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:15:58.846+0000 7f8a8e695040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T23:15:59.130 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:15:58.992+0000 7f8a8e695040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:15:59.130 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:15:59.129+0000 7f8a8e695040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T23:15:59.736 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:15:59.441+0000 7f8a8e695040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T23:15:59.736 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:15:59.503+0000 7f8a8e695040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T23:15:59.736 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:15:59.686+0000 7f8a8e695040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T23:15:59.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-mon[35586]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:59.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:59.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:59.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:59.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:15:59.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:15:59.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:15:59.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:59.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-mon[35586]: Deploying daemon mgr.y on smithi120 2024-09-15T23:15:59.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:15:59 smithi174 ceph-mon[35586]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:59.857 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:59 smithi089 ceph-mon[30933]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:59.858 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:59.858 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:59.858 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:59.858 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:15:59.858 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:15:59.858 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:15:59.858 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:59.858 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:59 smithi089 ceph-mon[30933]: Deploying daemon mgr.y on smithi120 2024-09-15T23:15:59.858 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:15:59 smithi089 ceph-mon[30933]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:15:59.893 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:59 smithi120 ceph-mon[35250]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:59.893 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:59.893 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:59.894 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:15:59.894 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:15:59.894 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:15:59.894 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:15:59.894 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:59.894 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:59 smithi120 ceph-mon[35250]: Deploying daemon mgr.y on smithi120 2024-09-15T23:15:59.894 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:15:59 smithi120 ceph-mon[35250]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:00.473 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:00 smithi120 systemd[1]: Starting Ceph mgr.y for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:16:00.849 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:00 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2643588264' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e292c3de-4e37-4c40-9085-a85a262fcf6b"}]: dispatch 2024-09-15T23:16:00.850 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:00 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2643588264' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e292c3de-4e37-4c40-9085-a85a262fcf6b"}]': finished 2024-09-15T23:16:00.850 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:00 smithi120 ceph-mon[35250]: osdmap e5: 1 total, 0 up, 1 in 2024-09-15T23:16:00.850 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:00.850 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:00 smithi120 podman[36601]: 2024-09-15 23:16:00.491197975 +0000 UTC m=+0.015798623 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:16:00.850 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:00 smithi120 podman[36601]: 2024-09-15 23:16:00.653604942 +0000 UTC m=+0.178205625 container create 5dccf040190a221225ff8263e1c9ff7c6dc69abb922cacb5c579251f49cff117 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:16:00.850 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:00 smithi120 podman[36601]: 2024-09-15 23:16:00.822179106 +0000 UTC m=+0.346779752 container init 5dccf040190a221225ff8263e1c9ff7c6dc69abb922cacb5c579251f49cff117 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-15T23:16:00.850 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:00 smithi120 podman[36601]: 2024-09-15 23:16:00.824917365 +0000 UTC m=+0.349518005 container start 5dccf040190a221225ff8263e1c9ff7c6dc69abb922cacb5c579251f49cff117 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-09-15T23:16:00.850 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:00 smithi120 ceph-mgr[36620]: -- 172.21.15.120:0/1034900954 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5569975dd0e0 con 0x5569975ba800 2024-09-15T23:16:00.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:00 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2643588264' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e292c3de-4e37-4c40-9085-a85a262fcf6b"}]: dispatch 2024-09-15T23:16:00.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:00 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2643588264' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e292c3de-4e37-4c40-9085-a85a262fcf6b"}]': finished 2024-09-15T23:16:00.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:00 smithi174 ceph-mon[35586]: osdmap e5: 1 total, 0 up, 1 in 2024-09-15T23:16:00.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:00.986 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:00 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:00.558+0000 7f8a8e695040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T23:16:00.986 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:00 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:00.626+0000 7f8a8e695040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T23:16:01.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:00 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2643588264' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e292c3de-4e37-4c40-9085-a85a262fcf6b"}]: dispatch 2024-09-15T23:16:01.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:00 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2643588264' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e292c3de-4e37-4c40-9085-a85a262fcf6b"}]': finished 2024-09-15T23:16:01.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:00 smithi089 ceph-mon[30933]: osdmap e5: 1 total, 0 up, 1 in 2024-09-15T23:16:01.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:01.163 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:00 smithi120 bash[36601]: 5dccf040190a221225ff8263e1c9ff7c6dc69abb922cacb5c579251f49cff117 2024-09-15T23:16:01.163 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:00 smithi120 systemd[1]: Started Ceph mgr.y for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:16:01.163 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:00 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:00.944+0000 7eff51ebf040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T23:16:01.163 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:01.006+0000 7eff51ebf040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T23:16:01.163 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:01.161+0000 7eff51ebf040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T23:16:01.473 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:01.230+0000 7eff51ebf040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T23:16:01.473 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:01.393+0000 7eff51ebf040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:16:01.640 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:01.359+0000 7f8a8e695040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T23:16:01.641 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:01.493+0000 7f8a8e695040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T23:16:01.641 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:01.639+0000 7f8a8e695040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:16:01.850 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:01.546+0000 7eff51ebf040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T23:16:01.931 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:01.703+0000 7f8a8e695040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T23:16:01.931 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:01.859+0000 7f8a8e695040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T23:16:01.931 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:01.929+0000 7f8a8e695040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T23:16:02.131 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-mon[35250]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:02.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/630696217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:16:02.132 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:02.132 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:01.874+0000 7eff51ebf040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T23:16:02.132 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:01.936+0000 7eff51ebf040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T23:16:02.222 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:01.989+0000 7f8a8e695040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T23:16:02.222 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:02.153+0000 7f8a8e695040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T23:16:02.222 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:02.221+0000 7f8a8e695040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T23:16:02.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-mon[35586]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:02.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/630696217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:16:02.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:01 smithi089 ceph-mon[30933]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:01 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/630696217' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:16:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:02.473 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:02 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:02.130+0000 7eff51ebf040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T23:16:02.485 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:02.289+0000 7f8a8e695040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T23:16:02.485 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:02.374+0000 7f8a8e695040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T23:16:02.485 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:02.436+0000 7f8a8e695040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T23:16:02.985 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:02.495+0000 7f8a8e695040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T23:16:03.473 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:03.015+0000 7eff51ebf040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T23:16:03.473 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:03.081+0000 7eff51ebf040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T23:16:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-mon[35250]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:03.485 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z[37404]: 2024-09-15T23:16:03.273+0000 7f8a8e695040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T23:16:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:03 smithi174 ceph-mon[35586]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:03.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:03.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:03.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:03.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:03.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:03.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:03.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:03 smithi089 ceph-mon[30933]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:04.050 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:03.794+0000 7eff51ebf040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T23:16:04.051 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:03.909+0000 7eff51ebf040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T23:16:04.051 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.049+0000 7eff51ebf040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:16:04.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:04.118+0000 7f2238990640 1 -- 172.21.15.89:0/3736888791 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 14) ==== 92408+0+0 (secure 0 0 0) 0x7f222c0433d0 con 0x7f22341037c0 2024-09-15T23:16:04.329 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-mon[35250]: Standby manager daemon z started 2024-09-15T23:16:04.331 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-mon[35250]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T23:16:04.331 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-mon[35250]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:16:04.331 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-mon[35250]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T23:16:04.331 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-mon[35250]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:16:04.331 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.110+0000 7eff51ebf040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T23:16:04.331 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.259+0000 7eff51ebf040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T23:16:04.332 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.328+0000 7eff51ebf040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T23:16:04.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:04 smithi174 ceph-mon[35586]: Standby manager daemon z started 2024-09-15T23:16:04.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:04 smithi174 ceph-mon[35586]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T23:16:04.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:04 smithi174 ceph-mon[35586]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:16:04.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:04 smithi174 ceph-mon[35586]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T23:16:04.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:04 smithi174 ceph-mon[35586]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:16:04.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:04 smithi089 ceph-mon[30933]: Standby manager daemon z started 2024-09-15T23:16:04.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:04 smithi089 ceph-mon[30933]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T23:16:04.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:04 smithi089 ceph-mon[30933]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:16:04.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:04 smithi089 ceph-mon[30933]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T23:16:04.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:04 smithi089 ceph-mon[30933]: from='mgr.? 172.21.15.174:0/1017276812' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:16:04.611 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.388+0000 7eff51ebf040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T23:16:04.611 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.544+0000 7eff51ebf040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T23:16:04.611 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.609+0000 7eff51ebf040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T23:16:04.877 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.675+0000 7eff51ebf040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T23:16:04.877 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.757+0000 7eff51ebf040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T23:16:04.877 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.816+0000 7eff51ebf040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T23:16:04.877 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:04.875+0000 7eff51ebf040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T23:16:05.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:05 smithi120 ceph-mon[35250]: mgrmap e14: x(active, since 63s), standbys: z 2024-09-15T23:16:05.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T23:16:05.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:05 smithi120 ceph-mon[35250]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:05.381 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:05 smithi089 ceph-mon[30933]: mgrmap e14: x(active, since 63s), standbys: z 2024-09-15T23:16:05.381 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T23:16:05.381 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:05 smithi089 ceph-mon[30933]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:05 smithi174 ceph-mon[35586]: mgrmap e14: x(active, since 63s), standbys: z 2024-09-15T23:16:05.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T23:16:05.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:05 smithi174 ceph-mon[35586]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:05.973 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:16:05 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y[36616]: 2024-09-15T23:16:05.658+0000 7eff51ebf040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T23:16:06.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:06.133+0000 7f2238990640 1 -- 172.21.15.89:0/3736888791 <== mon.0 v2:172.21.15.89:3300/0 8 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f222c04cce0 con 0x7f22341037c0 2024-09-15T23:16:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:06 smithi089 ceph-mon[30933]: Standby manager daemon y started 2024-09-15T23:16:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:06 smithi089 ceph-mon[30933]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T23:16:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:06 smithi089 ceph-mon[30933]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:16:06.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:06 smithi089 ceph-mon[30933]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T23:16:06.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:06 smithi089 ceph-mon[30933]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:16:06.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:06 smithi120 ceph-mon[35250]: Standby manager daemon y started 2024-09-15T23:16:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:06 smithi120 ceph-mon[35250]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T23:16:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:06 smithi120 ceph-mon[35250]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:16:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:06 smithi120 ceph-mon[35250]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T23:16:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:06 smithi120 ceph-mon[35250]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:16:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:06 smithi174 ceph-mon[35586]: Standby manager daemon y started 2024-09-15T23:16:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:06 smithi174 ceph-mon[35586]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T23:16:06.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:06 smithi174 ceph-mon[35586]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:16:06.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:06 smithi174 ceph-mon[35586]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T23:16:06.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:06 smithi174 ceph-mon[35586]: from='mgr.? 172.21.15.120:0/592504949' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:16:06.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:07.452 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:07 smithi089 ceph-mon[30933]: mgrmap e15: x(active, since 65s), standbys: z, y 2024-09-15T23:16:07.452 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T23:16:07.452 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:07 smithi089 ceph-mon[30933]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:07 smithi120 ceph-mon[35250]: mgrmap e15: x(active, since 65s), standbys: z, y 2024-09-15T23:16:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T23:16:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:07 smithi120 ceph-mon[35250]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:07 smithi174 ceph-mon[35586]: mgrmap e15: x(active, since 65s), standbys: z, y 2024-09-15T23:16:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T23:16:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:07 smithi174 ceph-mon[35586]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:10.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:09 smithi120 ceph-mon[35250]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:10.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-15T23:16:10.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:10.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:09 smithi120 ceph-mon[35250]: Deploying daemon osd.0 on smithi089 2024-09-15T23:16:10.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:09 smithi174 ceph-mon[35586]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:10.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-15T23:16:10.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:10.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:09 smithi174 ceph-mon[35586]: Deploying daemon osd.0 on smithi089 2024-09-15T23:16:10.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:09 smithi089 ceph-mon[30933]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:10.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-15T23:16:10.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:10.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:09 smithi089 ceph-mon[30933]: Deploying daemon osd.0 on smithi089 2024-09-15T23:16:12.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:11 smithi120 ceph-mon[35250]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:12.225 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:11 smithi089 ceph-mon[30933]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:12.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:11 smithi174 ceph-mon[35586]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:14.130 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:13 smithi089 ceph-mon[30933]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:14.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:13 smithi120 ceph-mon[35250]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:14.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:13 smithi174 ceph-mon[35586]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:16.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:15 smithi120 ceph-mon[35250]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:16.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:16.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:16.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:16.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:15 smithi174 ceph-mon[35586]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:16.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:16.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:16.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:16.246 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:15 smithi089 ceph-mon[30933]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:16.246 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:16.246 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:16.246 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:18.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:17 smithi120 ceph-mon[35250]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:18.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:17 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:18.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:17 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:18.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:17 smithi174 ceph-mon[35586]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:18.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:17 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:18.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:17 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:17 smithi089 ceph-mon[30933]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:17 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:17 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:19.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:18 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:19.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:18 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:19.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:18 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:19.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:19.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:19.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:19.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:19.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:19.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:19.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.674+0000 7f2238990640 1 -- 172.21.15.89:0/3736888791 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2234062630 con 0x7f221c039500 2024-09-15T23:16:19.675 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 0 on host 'smithi089' 2024-09-15T23:16:19.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 -- 172.21.15.89:0/3736888791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f221c039500 msgr2=0x7f221c03b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:19.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f221c039500 0x7f221c03b9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2234102c60 tx=0x7f2224008040 comp rx=0 tx=0).stop 2024-09-15T23:16:19.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 -- 172.21.15.89:0/3736888791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22341037c0 msgr2=0x7f223419f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:19.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22341037c0 0x7f223419f5f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f222c0098d0 tx=0x7f222c02f840 comp rx=0 tx=0).stop 2024-09-15T23:16:19.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 -- 172.21.15.89:0/3736888791 shutdown_connections 2024-09-15T23:16:19.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f221c039500 0x7f221c03b9c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:19.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22341037c0 0x7f223419f5f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:19.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2234102e00 0x7f2234198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:19.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 --2- 172.21.15.89:0/3736888791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234101c00 0x7f2234198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:19.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 -- 172.21.15.89:0/3736888791 >> 172.21.15.89:0/3736888791 conn(0x7f22340fd390 msgr2=0x7f2234106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:19.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 -- 172.21.15.89:0/3736888791 shutdown_connections 2024-09-15T23:16:19.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:16:19.676+0000 7f2239992640 1 -- 172.21.15.89:0/3736888791 wait complete. 2024-09-15T23:16:20.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:19 smithi120 ceph-mon[35250]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:20.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:20.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:20.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:20.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:19 smithi174 ceph-mon[35586]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:20.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:20.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:20.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:20.238 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:19 smithi089 ceph-mon[30933]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:20.238 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:20.238 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:20.238 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:20.276 DEBUG:teuthology.orchestra.run.smithi089:osd.0> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.0.service 2024-09-15T23:16:20.278 INFO:tasks.cephadm:Deploying osd.1 on smithi120 with /dev/vg_nvme/lv_4... 2024-09-15T23:16:20.279 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-15T23:16:20.515 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:16:21.507 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:16:21 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0[44283]: 2024-09-15T23:16:21.234+0000 7f699850a640 -1 osd.0 0 log_to_monitors true 2024-09-15T23:16:22.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:21 smithi089 ceph-mon[30933]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:22.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:21 smithi089 ceph-mon[30933]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-15T23:16:22.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:21 smithi120 ceph-mon[35250]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:22.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:21 smithi120 ceph-mon[35250]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-15T23:16:22.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:21 smithi174 ceph-mon[35586]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:22.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:21 smithi174 ceph-mon[35586]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-15T23:16:22.782 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:16:22.807 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_4 2024-09-15T23:16:23.052 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:16:23.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:22 smithi174 ceph-mon[35586]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-15T23:16:23.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:22 smithi174 ceph-mon[35586]: osdmap e6: 1 total, 0 up, 1 in 2024-09-15T23:16:23.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:22 smithi174 ceph-mon[35586]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:16:23.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:23.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:23.277 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:22 smithi120 ceph-mon[35250]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-15T23:16:23.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:22 smithi120 ceph-mon[35250]: osdmap e6: 1 total, 0 up, 1 in 2024-09-15T23:16:23.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:22 smithi120 ceph-mon[35250]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:16:23.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:23.278 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:22 smithi089 ceph-mon[30933]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-15T23:16:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:22 smithi089 ceph-mon[30933]: osdmap e6: 1 total, 0 up, 1 in 2024-09-15T23:16:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:22 smithi089 ceph-mon[30933]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:16:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:23.473 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.463+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/2295397234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d241026d0 msgr2=0x7f5d24108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:23.473 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.463+0000 7f5d2c0b4640 1 --2- 172.21.15.120:0/2295397234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d241026d0 0x7f5d24108f60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009ca0 tx=0x7f5d1c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.463+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/2295397234 shutdown_connections 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.463+0000 7f5d2c0b4640 1 --2- 172.21.15.120:0/2295397234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d241026d0 0x7f5d24108f60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.463+0000 7f5d2c0b4640 1 --2- 172.21.15.120:0/2295397234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d24101d10 0x7f5d24102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.463+0000 7f5d2c0b4640 1 --2- 172.21.15.120:0/2295397234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d24100b10 0x7f5d24100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.463+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/2295397234 >> 172.21.15.120:0/2295397234 conn(0x7f5d240fc2a0 msgr2=0x7f5d240fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.463+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/2295397234 shutdown_connections 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.463+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/2295397234 wait complete. 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d2c0b4640 1 Processor -- start 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d2c0b4640 1 -- start start 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d2c0b4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24100b10 0x7f5d2419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d2c0b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d24101d10 0x7f5d2419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d2c0b4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d241026d0 0x7f5d241a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:23.474 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d2c0b4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2410c790 con 0x7f5d24101d10 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d2c0b4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2410c610 con 0x7f5d24100b10 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d2c0b4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2410c910 con 0x7f5d241026d0 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d29e50640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24100b10 0x7f5d2419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d29e50640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24100b10 0x7f5d2419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:41636/0 (socket says 172.21.15.120:41636) 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d29e50640 1 -- 172.21.15.120:0/3306559590 learned_addr learned my addr 172.21.15.120:0/3306559590 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d2964f640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d24101d10 0x7f5d2419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d29e50640 1 -- 172.21.15.120:0/3306559590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d241026d0 msgr2=0x7f5d241a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d29e50640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d241026d0 0x7f5d241a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d29e50640 1 -- 172.21.15.120:0/3306559590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d24101d10 msgr2=0x7f5d2419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d29e50640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d24101d10 0x7f5d2419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d29e50640 1 -- 172.21.15.120:0/3306559590 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d241a1d70 con 0x7f5d24100b10 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.464+0000 7f5d29e50640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24100b10 0x7f5d2419a140 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000e9c0 tx=0x7f5d2000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:23.475 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.465+0000 7f5d1affd640 1 -- 172.21.15.120:0/3306559590 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2000cd80 con 0x7f5d24100b10 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.465+0000 7f5d1affd640 1 -- 172.21.15.120:0/3306559590 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d200035f0 con 0x7f5d24100b10 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.465+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/3306559590 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d241a2060 con 0x7f5d24100b10 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.465+0000 7f5d1affd640 1 -- 172.21.15.120:0/3306559590 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20010690 con 0x7f5d24100b10 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.465+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/3306559590 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d241a25a0 con 0x7f5d24100b10 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.466+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/3306559590 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d241a2860 con 0x7f5d24100b10 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.467+0000 7f5d1affd640 1 -- 172.21.15.120:0/3306559590 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f5d200037a0 con 0x7f5d24100b10 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.467+0000 7f5d1affd640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5d100a31e0 0x7f5d100a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.467+0000 7f5d1affd640 1 -- 172.21.15.120:0/3306559590 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1487+0+0 (secure 0 0 0) 0x7f5d200ce1b0 con 0x7f5d24100b10 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.468+0000 7f5d2964f640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5d100a31e0 0x7f5d100a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.468+0000 7f5d2964f640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5d100a31e0 0x7f5d100a56a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5d2419b660 tx=0x7f5d1400a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:23.476 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.472+0000 7f5d1affd640 1 -- 172.21.15.120:0/3306559590 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5d2009ca60 con 0x7f5d24100b10 2024-09-15T23:16:23.629 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0[44283]: 2024-09-15T23:16:23.363+0000 7f699448d640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T23:16:23.629 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0[44283]: 2024-09-15T23:16:23.369+0000 7f698c2ad640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:16:23.646 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:23.645+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/3306559590 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f5d241a2a30 con 0x7f5d100a31e0 2024-09-15T23:16:24.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:24.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi089 2024-09-15T23:16:24.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:24.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:16:24.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: osdmap e7: 1 total, 0 up, 1 in 2024-09-15T23:16:24.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:16:24.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:24.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi089 to 17087M 2024-09-15T23:16:24.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:24.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:24.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:24.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:24.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:24.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' 2024-09-15T23:16:24.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:16:24.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:16:24.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:24.251 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:24.251 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi089 2024-09-15T23:16:24.251 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:24.251 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:16:24.251 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: osdmap e7: 1 total, 0 up, 1 in 2024-09-15T23:16:24.251 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:16:24.251 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:24.251 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi089 to 17087M 2024-09-15T23:16:24.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:24.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:24.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:24.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:24.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:24.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' 2024-09-15T23:16:24.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:16:24.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:16:24.252 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi089 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: osdmap e7: 1 total, 0 up, 1 in 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi089 to 17087M 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:24.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:24.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:24.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611]' entity='osd.0' 2024-09-15T23:16:24.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:16:24.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:16:24.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:25.221 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:24 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:16:25.222 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:24 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:16:25.222 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:24 smithi120 ceph-mon[35250]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:16:25.222 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:25.222 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:24 smithi120 ceph-mon[35250]: osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] boot 2024-09-15T23:16:25.222 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:24 smithi120 ceph-mon[35250]: osdmap e8: 1 total, 1 up, 1 in 2024-09-15T23:16:25.222 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:25.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:24 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:16:25.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:24 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:16:25.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:24 smithi174 ceph-mon[35586]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:16:25.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:25.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:24 smithi174 ceph-mon[35586]: osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] boot 2024-09-15T23:16:25.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:24 smithi174 ceph-mon[35586]: osdmap e8: 1 total, 1 up, 1 in 2024-09-15T23:16:25.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:24 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:16:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:24 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:16:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:24 smithi089 ceph-mon[30933]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:16:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:25.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:24 smithi089 ceph-mon[30933]: osd.0 [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] boot 2024-09-15T23:16:25.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:24 smithi089 ceph-mon[30933]: osdmap e8: 1 total, 1 up, 1 in 2024-09-15T23:16:25.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:16:26.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:25 smithi174 ceph-mon[35586]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:25 smithi174 ceph-mon[35586]: from='client.? 172.21.15.120:0/2571494453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa999797-98c3-47f1-a08f-71a789176397"}]: dispatch 2024-09-15T23:16:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:25 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa999797-98c3-47f1-a08f-71a789176397"}]: dispatch 2024-09-15T23:16:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:25 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa999797-98c3-47f1-a08f-71a789176397"}]': finished 2024-09-15T23:16:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:25 smithi174 ceph-mon[35586]: osdmap e9: 2 total, 1 up, 2 in 2024-09-15T23:16:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:25 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:25 smithi089 ceph-mon[30933]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:25 smithi089 ceph-mon[30933]: from='client.? 172.21.15.120:0/2571494453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa999797-98c3-47f1-a08f-71a789176397"}]: dispatch 2024-09-15T23:16:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:25 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa999797-98c3-47f1-a08f-71a789176397"}]: dispatch 2024-09-15T23:16:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:25 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa999797-98c3-47f1-a08f-71a789176397"}]': finished 2024-09-15T23:16:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:25 smithi089 ceph-mon[30933]: osdmap e9: 2 total, 1 up, 2 in 2024-09-15T23:16:26.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:25 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:26.439 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:25 smithi120 ceph-mon[35250]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:16:26.439 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:25 smithi120 ceph-mon[35250]: from='client.? 172.21.15.120:0/2571494453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa999797-98c3-47f1-a08f-71a789176397"}]: dispatch 2024-09-15T23:16:26.440 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:25 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa999797-98c3-47f1-a08f-71a789176397"}]: dispatch 2024-09-15T23:16:26.440 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:25 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa999797-98c3-47f1-a08f-71a789176397"}]': finished 2024-09-15T23:16:26.440 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:25 smithi120 ceph-mon[35250]: osdmap e9: 2 total, 1 up, 2 in 2024-09-15T23:16:26.440 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:25 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:27.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:26 smithi174 ceph-mon[35586]: from='client.? 172.21.15.120:0/3782985707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:16:27.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:26 smithi089 ceph-mon[30933]: from='client.? 172.21.15.120:0/3782985707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:16:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:26 smithi120 ceph-mon[35250]: from='client.? 172.21.15.120:0/3782985707' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:16:28.235 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:27 smithi089 ceph-mon[30933]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:28.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:27 smithi174 ceph-mon[35586]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:28.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:27 smithi120 ceph-mon[35250]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:30.268 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:29 smithi120 ceph-mon[35250]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:30.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:29 smithi089 ceph-mon[30933]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:30.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:29 smithi174 ceph-mon[35586]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:32.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:31 smithi120 ceph-mon[35250]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:31 smithi089 ceph-mon[30933]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:32.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:31 smithi174 ceph-mon[35586]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:34.248 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:33 smithi120 ceph-mon[35250]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:34.248 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:33 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-15T23:16:34.248 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:33 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:33 smithi089 ceph-mon[30933]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:33 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-15T23:16:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:33 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:34.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:33 smithi174 ceph-mon[35586]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:33 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-15T23:16:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:33 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:35.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:34 smithi120 ceph-mon[35250]: Deploying daemon osd.1 on smithi120 2024-09-15T23:16:35.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:34 smithi089 ceph-mon[30933]: Deploying daemon osd.1 on smithi120 2024-09-15T23:16:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:34 smithi174 ceph-mon[35586]: Deploying daemon osd.1 on smithi120 2024-09-15T23:16:36.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:36 smithi089 ceph-mon[30933]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:36.352 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:36 smithi120 ceph-mon[35250]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:36 smithi174 ceph-mon[35586]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:38.274 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:38 smithi089 ceph-mon[30933]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:38.357 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:38 smithi120 ceph-mon[35250]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:38 smithi174 ceph-mon[35586]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:40.261 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:40 smithi120 ceph-mon[35250]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:40.261 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:40 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:40.261 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:40 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:40.261 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:40 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:40.288 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:40 smithi174 ceph-mon[35586]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:40.288 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:40 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:40.288 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:40 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:40.288 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:40 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:40.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:40 smithi089 ceph-mon[30933]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:40.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:40 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:40.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:40 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:40.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:40 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:42.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:42 smithi089 ceph-mon[30933]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:42 smithi120 ceph-mon[35250]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:42.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:42 smithi174 ceph-mon[35586]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:43 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:43.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:43 smithi120 ceph-mon[35250]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:43 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:43.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:43 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:43.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:43 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:43.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:43 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:43.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:43 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:43.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:43 smithi174 ceph-mon[35586]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:43.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:43 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:43.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:43 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:43.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:43 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:43.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:43 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:43.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:43 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:43.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:43 smithi089 ceph-mon[30933]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:43.850 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 1 on host 'smithi120' 2024-09-15T23:16:43.850 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.848+0000 7f5d1affd640 1 -- 172.21.15.120:0/3306559590 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5d241a2a30 con 0x7f5d100a31e0 2024-09-15T23:16:43.852 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/3306559590 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5d100a31e0 msgr2=0x7f5d100a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.852 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5d100a31e0 0x7f5d100a56a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5d2419b660 tx=0x7f5d1400a250 comp rx=0 tx=0).stop 2024-09-15T23:16:43.852 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/3306559590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24100b10 msgr2=0x7f5d2419a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.852 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24100b10 0x7f5d2419a140 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000e9c0 tx=0x7f5d2000ee80 comp rx=0 tx=0).stop 2024-09-15T23:16:43.852 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/3306559590 shutdown_connections 2024-09-15T23:16:43.853 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5d100a31e0 0x7f5d100a56a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.853 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d241026d0 0x7f5d241a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.853 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d24101d10 0x7f5d2419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.853 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 --2- 172.21.15.120:0/3306559590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24100b10 0x7f5d2419a140 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.853 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/3306559590 >> 172.21.15.120:0/3306559590 conn(0x7f5d240fc2a0 msgr2=0x7f5d240fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:43.853 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.850+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/3306559590 shutdown_connections 2024-09-15T23:16:43.853 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:16:43.851+0000 7f5d2c0b4640 1 -- 172.21.15.120:0/3306559590 wait complete. 2024-09-15T23:16:44.464 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:44 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:44.464 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:44 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:44.464 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:44 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:44.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:44 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:44.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:44 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:44.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:44 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:44.499 DEBUG:teuthology.orchestra.run.smithi120:osd.1> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.1.service 2024-09-15T23:16:44.502 INFO:tasks.cephadm:Deploying osd.2 on smithi174 with /dev/vg_nvme/lv_4... 2024-09-15T23:16:44.502 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-15T23:16:44.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:44 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:44.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:44 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:44.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:44 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:44.752 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:16:45.170 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:45 smithi174 ceph-mon[35586]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:45 smithi120 ceph-mon[35250]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:45.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:45 smithi089 ceph-mon[30933]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:45.770 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:16:45 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1[39691]: 2024-09-15T23:16:45.525+0000 7ffb81dfb640 -1 osd.1 0 log_to_monitors true 2024-09-15T23:16:46.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:46 smithi174 ceph-mon[35586]: from='osd.1 [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:16:46.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:46 smithi174 ceph-mon[35586]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:16:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:46 smithi120 ceph-mon[35250]: from='osd.1 [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:16:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:46 smithi120 ceph-mon[35250]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:16:46.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:46 smithi089 ceph-mon[30933]: from='osd.1 [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:16:46.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:46 smithi089 ceph-mon[30933]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:16:47.304 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:16:47.330 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:47 smithi174 ceph-mon[35586]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-15T23:16:47.330 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:47 smithi174 ceph-mon[35586]: from='osd.1 [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:16:47.330 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:47 smithi174 ceph-mon[35586]: osdmap e10: 2 total, 1 up, 2 in 2024-09-15T23:16:47.330 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:47 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:47.330 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:47 smithi174 ceph-mon[35586]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:16:47.330 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:47 smithi174 ceph-mon[35586]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:47.331 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_4 2024-09-15T23:16:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:47 smithi120 ceph-mon[35250]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-15T23:16:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:47 smithi120 ceph-mon[35250]: from='osd.1 [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:16:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:47 smithi120 ceph-mon[35250]: osdmap e10: 2 total, 1 up, 2 in 2024-09-15T23:16:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:47 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:47.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:47 smithi120 ceph-mon[35250]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:16:47.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:47 smithi120 ceph-mon[35250]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:47.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:47 smithi089 ceph-mon[30933]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-15T23:16:47.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:47 smithi089 ceph-mon[30933]: from='osd.1 [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:16:47.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:47 smithi089 ceph-mon[30933]: osdmap e10: 2 total, 1 up, 2 in 2024-09-15T23:16:47.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:47 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:47.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:47 smithi089 ceph-mon[30933]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:16:47.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:47 smithi089 ceph-mon[30933]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:47.580 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:16:47.973 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:16:47 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1[39691]: 2024-09-15T23:16:47.485+0000 7ffb7dd7e640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T23:16:47.974 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:16:47 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1[39691]: 2024-09-15T23:16:47.488+0000 7ffb75b9e640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:16:48.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.079+0000 7f61e1af4640 1 -- 172.21.15.174:0/991046191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc1017b0 msgr2=0x7f61dc101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:48.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.079+0000 7f61e1af4640 1 --2- 172.21.15.174:0/991046191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc1017b0 0x7f61dc101c60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f61d4009ca0 tx=0x7f61d402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:16:48.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.079+0000 7f61e1af4640 1 -- 172.21.15.174:0/991046191 shutdown_connections 2024-09-15T23:16:48.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.079+0000 7f61e1af4640 1 --2- 172.21.15.174:0/991046191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc1017b0 0x7f61dc101c60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:48.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.079+0000 7f61e1af4640 1 --2- 172.21.15.174:0/991046191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc100e10 0x7f61dc101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:48.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.079+0000 7f61e1af4640 1 --2- 172.21.15.174:0/991046191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc1004d0 0x7f61dc1008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:48.096 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.079+0000 7f61e1af4640 1 -- 172.21.15.174:0/991046191 >> 172.21.15.174:0/991046191 conn(0x7f61dc0fc280 msgr2=0x7f61dc0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.079+0000 7f61e1af4640 1 -- 172.21.15.174:0/991046191 shutdown_connections 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.079+0000 7f61e1af4640 1 -- 172.21.15.174:0/991046191 wait complete. 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61e1af4640 1 Processor -- start 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61e1af4640 1 -- start start 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61e1af4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc1004d0 0x7f61dc19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61e1af4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100e10 0x7f61dc19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61e1af4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc1017b0 0x7f61dc1a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61e1af4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc10fa40 con 0x7f61dc100e10 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61e1af4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc10f8c0 con 0x7f61dc1017b0 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61e1af4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc10fbc0 con 0x7f61dc1004d0 2024-09-15T23:16:48.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61db7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc1004d0 0x7f61dc19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61db7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc1004d0 0x7f61dc19a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:43368/0 (socket says 172.21.15.174:43368) 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61db7fe640 1 -- 172.21.15.174:0/4042081616 learned_addr learned my addr 172.21.15.174:0/4042081616 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61dbfff640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc1017b0 0x7f61dc1a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61d3fff640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100e10 0x7f61dc19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61db7fe640 1 -- 172.21.15.174:0/4042081616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc1017b0 msgr2=0x7f61dc1a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61db7fe640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc1017b0 0x7f61dc1a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61db7fe640 1 -- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100e10 msgr2=0x7f61dc19a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61db7fe640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100e10 0x7f61dc19a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61db7fe640 1 -- 172.21.15.174:0/4042081616 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61dc1a1cb0 con 0x7f61dc1004d0 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61d3fff640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100e10 0x7f61dc19a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:48.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.080+0000 7f61db7fe640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc1004d0 0x7f61dc19a080 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00d950 tx=0x7f61cc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.081+0000 7f61d97fa640 1 -- 172.21.15.174:0/4042081616 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61cc017070 con 0x7f61dc1004d0 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.081+0000 7f61d97fa640 1 -- 172.21.15.174:0/4042081616 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61cc009c30 con 0x7f61dc1004d0 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.081+0000 7f61d97fa640 1 -- 172.21.15.174:0/4042081616 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61cc00f6b0 con 0x7f61dc1004d0 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.081+0000 7f61e1af4640 1 -- 172.21.15.174:0/4042081616 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61dc1a1fa0 con 0x7f61dc1004d0 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.081+0000 7f61e1af4640 1 -- 172.21.15.174:0/4042081616 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61dc1a23f0 con 0x7f61dc1004d0 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.082+0000 7f61e1af4640 1 -- 172.21.15.174:0/4042081616 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61dc10b220 con 0x7f61dc1004d0 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.083+0000 7f61d97fa640 1 -- 172.21.15.174:0/4042081616 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f61cc0109d0 con 0x7f61dc1004d0 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.084+0000 7f61d97fa640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61c80a31e0 0x7f61c80a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.084+0000 7f61d97fa640 1 -- 172.21.15.174:0/4042081616 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 2029+0+0 (secure 0 0 0) 0x7f61cc0cdb40 con 0x7f61dc1004d0 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.084+0000 7f61d3fff640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61c80a31e0 0x7f61c80a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:48.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.084+0000 7f61d3fff640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61c80a31e0 0x7f61c80a56a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f61c40058c0 tx=0x7f61c4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:48.100 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.092+0000 7f61d97fa640 1 -- 172.21.15.174:0/4042081616 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f61cc09c1d0 con 0x7f61dc1004d0 2024-09-15T23:16:48.274 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:16:48.273+0000 7f61e1af4640 1 -- 172.21.15.174:0/4042081616 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f61dc062630 con 0x7f61c80a31e0 2024-09-15T23:16:48.275 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:16:48.275 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: osdmap e11: 2 total, 1 up, 2 in 2024-09-15T23:16:48.275 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:48.275 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:48.275 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi120 2024-09-15T23:16:48.275 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.275 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.275 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:16:48.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi120 to 17088M 2024-09-15T23:16:48.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:48.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:48.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:48 smithi174 ceph-mon[35586]: from='osd.1 ' entity='osd.1' 2024-09-15T23:16:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: osdmap e11: 2 total, 1 up, 2 in 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi120 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi120 to 17088M 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:48 smithi120 ceph-mon[35250]: from='osd.1 ' entity='osd.1' 2024-09-15T23:16:48.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: osdmap e11: 2 total, 1 up, 2 in 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi120 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi120 to 17088M 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:48.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:16:48.562 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:48 smithi089 ceph-mon[30933]: from='osd.1 ' entity='osd.1' 2024-09-15T23:16:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:16:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:16:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:16:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:16:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:16:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: osd.1 [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] boot 2024-09-15T23:16:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: osdmap e12: 2 total, 2 up, 2 in 2024-09-15T23:16:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:49 smithi120 ceph-mon[35250]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:16:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:16:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:16:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:16:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:16:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: osd.1 [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] boot 2024-09-15T23:16:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: osdmap e12: 2 total, 2 up, 2 in 2024-09-15T23:16:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:49 smithi174 ceph-mon[35586]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:16:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:16:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:49.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:16:49.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:16:49.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:16:49.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:49.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: osd.1 [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] boot 2024-09-15T23:16:49.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: osdmap e12: 2 total, 2 up, 2 in 2024-09-15T23:16:49.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:16:49.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:49 smithi089 ceph-mon[30933]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:16:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:51 smithi120 ceph-mon[35250]: osdmap e13: 2 total, 2 up, 2 in 2024-09-15T23:16:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:51 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/3743507933' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b48eda20-370e-40bc-bdb3-4011a46df4a5"}]: dispatch 2024-09-15T23:16:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:51 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b48eda20-370e-40bc-bdb3-4011a46df4a5"}]: dispatch 2024-09-15T23:16:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:51 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b48eda20-370e-40bc-bdb3-4011a46df4a5"}]': finished 2024-09-15T23:16:51.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:51 smithi120 ceph-mon[35250]: osdmap e14: 3 total, 2 up, 3 in 2024-09-15T23:16:51.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:16:51.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:51 smithi120 ceph-mon[35250]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:51 smithi174 ceph-mon[35586]: osdmap e13: 2 total, 2 up, 2 in 2024-09-15T23:16:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:51 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/3743507933' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b48eda20-370e-40bc-bdb3-4011a46df4a5"}]: dispatch 2024-09-15T23:16:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:51 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b48eda20-370e-40bc-bdb3-4011a46df4a5"}]: dispatch 2024-09-15T23:16:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:51 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b48eda20-370e-40bc-bdb3-4011a46df4a5"}]': finished 2024-09-15T23:16:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:51 smithi174 ceph-mon[35586]: osdmap e14: 3 total, 2 up, 3 in 2024-09-15T23:16:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:16:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:51 smithi174 ceph-mon[35586]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:51 smithi089 ceph-mon[30933]: osdmap e13: 2 total, 2 up, 2 in 2024-09-15T23:16:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:51 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/3743507933' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b48eda20-370e-40bc-bdb3-4011a46df4a5"}]: dispatch 2024-09-15T23:16:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:51 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b48eda20-370e-40bc-bdb3-4011a46df4a5"}]: dispatch 2024-09-15T23:16:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:51 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b48eda20-370e-40bc-bdb3-4011a46df4a5"}]': finished 2024-09-15T23:16:51.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:51 smithi089 ceph-mon[30933]: osdmap e14: 3 total, 2 up, 3 in 2024-09-15T23:16:51.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:16:51.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:51 smithi089 ceph-mon[30933]: pgmap v59: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:52 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/1016731667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:16:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:52 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/1016731667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:16:52.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:52 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/1016731667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:16:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:53 smithi120 ceph-mon[35250]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:53 smithi174 ceph-mon[35586]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:53 smithi089 ceph-mon[30933]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:56.058 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:55 smithi174 ceph-mon[35586]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:55 smithi120 ceph-mon[35250]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:55 smithi089 ceph-mon[30933]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:58.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:57 smithi120 ceph-mon[35250]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:58.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:57 smithi174 ceph-mon[35586]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:57 smithi089 ceph-mon[30933]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:16:59.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-15T23:16:59.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:59.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-15T23:16:59.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:59.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-15T23:16:59.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:00.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:59 smithi120 ceph-mon[35250]: Deploying daemon osd.2 on smithi174 2024-09-15T23:17:00.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:16:59 smithi120 ceph-mon[35250]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:00.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:59 smithi174 ceph-mon[35586]: Deploying daemon osd.2 on smithi174 2024-09-15T23:17:00.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:16:59 smithi174 ceph-mon[35586]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:00.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:59 smithi089 ceph-mon[30933]: Deploying daemon osd.2 on smithi174 2024-09-15T23:17:00.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:16:59 smithi089 ceph-mon[30933]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:02.179 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:01 smithi174 ceph-mon[35586]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:02.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:01 smithi120 ceph-mon[35250]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:01 smithi089 ceph-mon[30933]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:04.098 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:03 smithi174 ceph-mon[35586]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:04.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:03 smithi089 ceph-mon[30933]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:03 smithi120 ceph-mon[35250]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:06.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:05 smithi174 ceph-mon[35586]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:06.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:06.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:06.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:05 smithi089 ceph-mon[30933]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:05 smithi120 ceph-mon[35250]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:08.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:08 smithi089 ceph-mon[30933]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:08.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:08.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:08.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:08.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:08.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:08.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:08 smithi174 ceph-mon[35586]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:08.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:08.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:08.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:08.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:08.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:08 smithi120 ceph-mon[35250]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:08.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:09.025 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.024+0000 7f61d97fa640 1 -- 172.21.15.174:0/4042081616 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f61dc062630 con 0x7f61c80a31e0 2024-09-15T23:17:09.025 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 2 on host 'smithi174' 2024-09-15T23:17:09.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 -- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61c80a31e0 msgr2=0x7f61c80a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:09.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61c80a31e0 0x7f61c80a56a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f61c40058c0 tx=0x7f61c4005850 comp rx=0 tx=0).stop 2024-09-15T23:17:09.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 -- 172.21.15.174:0/4042081616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc1004d0 msgr2=0x7f61dc19a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:09.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc1004d0 0x7f61dc19a080 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f61cc00d950 tx=0x7f61cc00de10 comp rx=0 tx=0).stop 2024-09-15T23:17:09.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 -- 172.21.15.174:0/4042081616 shutdown_connections 2024-09-15T23:17:09.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61c80a31e0 0x7f61c80a56a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:09.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc1017b0 0x7f61dc1a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:09.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100e10 0x7f61dc19a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:09.028 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 --2- 172.21.15.174:0/4042081616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc1004d0 0x7f61dc19a080 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:09.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 -- 172.21.15.174:0/4042081616 >> 172.21.15.174:0/4042081616 conn(0x7f61dc0fc280 msgr2=0x7f61dc0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:09.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 -- 172.21.15.174:0/4042081616 shutdown_connections 2024-09-15T23:17:09.029 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:17:09.026+0000 7f61e1af4640 1 -- 172.21.15.174:0/4042081616 wait complete. 2024-09-15T23:17:09.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:09.328 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:09.649 DEBUG:teuthology.orchestra.run.smithi174:osd.2> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.2.service 2024-09-15T23:17:09.652 INFO:tasks.cephadm:Deploying osd.3 on smithi089 with /dev/vg_nvme/lv_3... 2024-09-15T23:17:09.652 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-15T23:17:09.909 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:10.181 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:10 smithi174 ceph-mon[35586]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:10.182 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:10 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:10.182 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:10 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:10.276 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:10 smithi089 ceph-mon[30933]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:10.276 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:10 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:10.276 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:10 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:10.374 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:10 smithi120 ceph-mon[35250]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:10.375 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:10 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:10.375 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:10 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:11.486 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:11 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2[40538]: 2024-09-15T23:17:11.300+0000 7f86854c7640 -1 osd.2 0 log_to_monitors true 2024-09-15T23:17:12.266 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:12 smithi089 ceph-mon[30933]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:12.267 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:12 smithi089 ceph-mon[30933]: from='osd.2 [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:17:12.267 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:12 smithi089 ceph-mon[30933]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:17:12.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:12 smithi174 ceph-mon[35586]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:12.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:12 smithi174 ceph-mon[35586]: from='osd.2 [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:17:12.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:12 smithi174 ceph-mon[35586]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:17:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:12 smithi120 ceph-mon[35250]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:12 smithi120 ceph-mon[35250]: from='osd.2 [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:17:12.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:12 smithi120 ceph-mon[35250]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:17:12.474 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:17:12.503 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi089:vg_nvme/lv_3 2024-09-15T23:17:12.758 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:13.159 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-15T23:17:13.159 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: osdmap e15: 3 total, 2 up, 3 in 2024-09-15T23:17:13.159 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:13.159 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='osd.2 [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:17:13.159 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:17:13.159 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.159 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.160 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:17:13.160 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.160 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:13.160 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:13.160 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:13 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.242+0000 7f147cf6c640 1 -- 172.21.15.89:0/4114730746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14781037b0 msgr2=0x7f147810a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:13.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.242+0000 7f147cf6c640 1 --2- 172.21.15.89:0/4114730746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14781037b0 0x7f147810a040 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1470009ca0 tx=0x7f147002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:17:13.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.242+0000 7f147cf6c640 1 -- 172.21.15.89:0/4114730746 shutdown_connections 2024-09-15T23:17:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.242+0000 7f147cf6c640 1 --2- 172.21.15.89:0/4114730746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14781037b0 0x7f147810a040 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.242+0000 7f147cf6c640 1 --2- 172.21.15.89:0/4114730746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1478102df0 0x7f1478103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.242+0000 7f147cf6c640 1 --2- 172.21.15.89:0/4114730746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1478101bf0 0x7f1478101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.242+0000 7f147cf6c640 1 -- 172.21.15.89:0/4114730746 >> 172.21.15.89:0/4114730746 conn(0x7f14780fd3c0 msgr2=0x7f14780ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.242+0000 7f147cf6c640 1 -- 172.21.15.89:0/4114730746 shutdown_connections 2024-09-15T23:17:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.242+0000 7f147cf6c640 1 -- 172.21.15.89:0/4114730746 wait complete. 2024-09-15T23:17:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.243+0000 7f147cf6c640 1 Processor -- start 2024-09-15T23:17:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.243+0000 7f147cf6c640 1 -- start start 2024-09-15T23:17:13.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.243+0000 7f147cf6c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1478101bf0 0x7f1478197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.243+0000 7f147cf6c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1478102df0 0x7f1478198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.243+0000 7f147cf6c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14781037b0 0x7f147819f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.243+0000 7f147cf6c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f147810d8f0 con 0x7f14781037b0 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.243+0000 7f147cf6c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f147810d770 con 0x7f1478102df0 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.243+0000 7f147cf6c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f147810da70 con 0x7f1478101bf0 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.243+0000 7f1476d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14781037b0 0x7f147819f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f1476d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14781037b0 0x7f147819f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39776/0 (socket says 172.21.15.89:39776) 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f1476d9d640 1 -- 172.21.15.89:0/21207618 learned_addr learned my addr 172.21.15.89:0/21207618 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f147659c640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1478101bf0 0x7f1478197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f1475d9b640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1478102df0 0x7f1478198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:13.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f1476d9d640 1 -- 172.21.15.89:0/21207618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1478101bf0 msgr2=0x7f1478197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:13.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f1476d9d640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1478101bf0 0x7f1478197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:13.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f1476d9d640 1 -- 172.21.15.89:0/21207618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1478102df0 msgr2=0x7f1478198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:13.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f1476d9d640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1478102df0 0x7f1478198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:13.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f1476d9d640 1 -- 172.21.15.89:0/21207618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147819fb10 con 0x7f14781037b0 2024-09-15T23:17:13.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f1476d9d640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14781037b0 0x7f147819f4a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1470005e30 tx=0x7f1470031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:13.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f14677fe640 1 -- 172.21.15.89:0/21207618 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470003b70 con 0x7f14781037b0 2024-09-15T23:17:13.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f147cf6c640 1 -- 172.21.15.89:0/21207618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f147819fda0 con 0x7f14781037b0 2024-09-15T23:17:13.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f14677fe640 1 -- 172.21.15.89:0/21207618 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1470003d10 con 0x7f14781037b0 2024-09-15T23:17:13.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f14677fe640 1 -- 172.21.15.89:0/21207618 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147002fd30 con 0x7f14781037b0 2024-09-15T23:17:13.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.244+0000 7f147cf6c640 1 -- 172.21.15.89:0/21207618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14781a02d0 con 0x7f14781037b0 2024-09-15T23:17:13.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.245+0000 7f147cf6c640 1 -- 172.21.15.89:0/21207618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1478108f20 con 0x7f14781037b0 2024-09-15T23:17:13.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.246+0000 7f14677fe640 1 -- 172.21.15.89:0/21207618 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f1470004050 con 0x7f14781037b0 2024-09-15T23:17:13.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.247+0000 7f14677fe640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14600a31e0 0x7f14600a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:13.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.247+0000 7f147659c640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14600a31e0 0x7f14600a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:13.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.247+0000 7f14677fe640 1 -- 172.21.15.89:0/21207618 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 2571+0+0 (secure 0 0 0) 0x7f14700ef330 con 0x7f14781037b0 2024-09-15T23:17:13.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.247+0000 7f147659c640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14600a31e0 0x7f14600a56a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1478102c50 tx=0x7f1468009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:13.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.251+0000 7f14677fe640 1 -- 172.21.15.89:0/21207618 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f14700bd8c0 con 0x7f14781037b0 2024-09-15T23:17:13.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:13.429+0000 7f147cf6c640 1 -- 172.21.15.89:0/21207618 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1478062630 con 0x7f14600a31e0 2024-09-15T23:17:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-15T23:17:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: osdmap e15: 3 total, 2 up, 3 in 2024-09-15T23:17:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:13.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='osd.2 [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:17:13.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:17:13.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:17:13.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:13.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:13.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:13 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.485 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2[40538]: 2024-09-15T23:17:13.034+0000 7f868144a640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T23:17:13.486 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2[40538]: 2024-09-15T23:17:13.042+0000 7f867d272640 -1 osd.2 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:17:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-15T23:17:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: osdmap e15: 3 total, 2 up, 3 in 2024-09-15T23:17:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='osd.2 [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:17:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:17:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:17:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:13.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:13.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:13.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:13 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:14.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi174 2024-09-15T23:17:14.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi174 to 17088M 2024-09-15T23:17:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:17:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: osdmap e16: 3 total, 2 up, 3 in 2024-09-15T23:17:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:17:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:17:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:17:14.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:14.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi174 2024-09-15T23:17:14.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi174 to 17088M 2024-09-15T23:17:14.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:14.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:17:14.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: osdmap e16: 3 total, 2 up, 3 in 2024-09-15T23:17:14.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:14.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:14.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:17:14.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:17:14.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:17:14.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:14 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi174 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi174 to 17088M 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: osdmap e16: 3 total, 2 up, 3 in 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:17:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:14 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:15 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:17:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:15 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:17:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:15 smithi120 ceph-mon[35250]: osd.2 [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] boot 2024-09-15T23:17:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:15 smithi120 ceph-mon[35250]: osdmap e17: 3 total, 3 up, 3 in 2024-09-15T23:17:15.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:15 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:17:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:15 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:17:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:15 smithi174 ceph-mon[35586]: osd.2 [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] boot 2024-09-15T23:17:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:15 smithi174 ceph-mon[35586]: osdmap e17: 3 total, 3 up, 3 in 2024-09-15T23:17:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:15.523 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:15 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:17:15.523 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:15 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:17:15.523 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:15 smithi089 ceph-mon[30933]: osd.2 [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] boot 2024-09-15T23:17:15.523 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:15 smithi089 ceph-mon[30933]: osdmap e17: 3 total, 3 up, 3 in 2024-09-15T23:17:15.523 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:17:16.410 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:17:16 smithi089 sudo[48689]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:17:16.410 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:17:16 smithi089 sudo[48689]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:16.410 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:17:16 smithi089 sudo[48689]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:16.410 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:17:16 smithi089 sudo[48689]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:16.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:16 smithi089 ceph-mon[30933]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:16.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:16 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:17:16.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:16 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-15T23:17:16.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:16 smithi089 ceph-mon[30933]: osdmap e18: 3 total, 3 up, 3 in 2024-09-15T23:17:16.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:16 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:17:16.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:16 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2025152088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "783de02c-1c46-44ed-98c4-10fb142648a2"}]: dispatch 2024-09-15T23:17:16.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:16 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:17:16.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:16 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2025152088' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "783de02c-1c46-44ed-98c4-10fb142648a2"}]': finished 2024-09-15T23:17:16.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:16 smithi089 ceph-mon[30933]: osdmap e19: 4 total, 3 up, 4 in 2024-09-15T23:17:16.411 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:16 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:16 smithi120 ceph-mon[35250]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:16 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:17:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:16 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-15T23:17:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:16 smithi120 ceph-mon[35250]: osdmap e18: 3 total, 3 up, 3 in 2024-09-15T23:17:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:16 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:17:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:16 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2025152088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "783de02c-1c46-44ed-98c4-10fb142648a2"}]: dispatch 2024-09-15T23:17:16.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:16 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:17:16.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:16 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2025152088' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "783de02c-1c46-44ed-98c4-10fb142648a2"}]': finished 2024-09-15T23:17:16.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:16 smithi120 ceph-mon[35250]: osdmap e19: 4 total, 3 up, 4 in 2024-09-15T23:17:16.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:16 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:16 smithi174 ceph-mon[35586]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:17:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:16 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:17:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:16 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-15T23:17:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:16 smithi174 ceph-mon[35586]: osdmap e18: 3 total, 3 up, 3 in 2024-09-15T23:17:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:16 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:17:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:16 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2025152088' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "783de02c-1c46-44ed-98c4-10fb142648a2"}]: dispatch 2024-09-15T23:17:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:16 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:17:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:16 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2025152088' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "783de02c-1c46-44ed-98c4-10fb142648a2"}]': finished 2024-09-15T23:17:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:16 smithi174 ceph-mon[35586]: osdmap e19: 4 total, 3 up, 4 in 2024-09-15T23:17:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:16 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:16.810 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:17:16 smithi089 sudo[48700]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:17:16.810 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:17:16 smithi089 sudo[48700]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:16.810 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:17:16 smithi089 sudo[48700]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:16.810 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:17:16 smithi089 sudo[48700]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:16.987 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:17:16 smithi120 sudo[43831]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:17:16.988 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:17:16 smithi120 sudo[43831]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:16.988 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:17:16 smithi120 sudo[43831]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:16.988 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:17:16 smithi120 sudo[43831]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:17 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/215764873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:17:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:17 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:17 smithi120 ceph-mon[35250]: osdmap e20: 4 total, 3 up, 4 in 2024-09-15T23:17:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:17 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:17.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:17 smithi120 ceph-mon[35250]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:17.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:17 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:17.474 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:17:16 smithi120 sudo[43834]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:17:17.474 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:17:16 smithi120 sudo[43834]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:17.474 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:17:16 smithi120 sudo[43834]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:17.474 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:17:16 smithi120 sudo[43834]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:17.485 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:17 smithi174 sudo[44575]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:17:17.485 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:17 smithi174 sudo[44575]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:17.485 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:17 smithi174 sudo[44575]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:17.485 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:17 smithi174 sudo[44575]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:17.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:17 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/215764873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:17:17.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:17 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:17.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:17 smithi174 ceph-mon[35586]: osdmap e20: 4 total, 3 up, 4 in 2024-09-15T23:17:17.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:17 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:17.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:17 smithi174 ceph-mon[35586]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:17.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:17 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:17.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:17 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/215764873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:17:17.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:17 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:17.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:17 smithi089 ceph-mon[30933]: osdmap e20: 4 total, 3 up, 4 in 2024-09-15T23:17:17.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:17 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:17.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:17 smithi089 ceph-mon[30933]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:17.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:17 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:17.985 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:17 smithi174 sudo[44578]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:17:17.985 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:17 smithi174 sudo[44578]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:17.986 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:17 smithi174 sudo[44578]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:17.986 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:17:17 smithi174 sudo[44578]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:18.035 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:17 smithi089 sudo[48927]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:17:18.035 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:17 smithi089 sudo[48927]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:18.035 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:17 smithi089 sudo[48927]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:18.035 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:17 smithi089 sudo[48927]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:18.050+0000 7f14677fe640 1 -- 172.21.15.89:0/21207618 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f14700b9830 con 0x7f14781037b0 2024-09-15T23:17:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 sudo[49113]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:17:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 sudo[49113]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 sudo[49113]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 sudo[49113]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:18.535 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 sudo[44581]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:17:18.536 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 sudo[44581]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:18.536 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 sudo[44581]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:18.536 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 sudo[44581]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:18.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:18.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:17:18.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:17:18.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:17:18.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:17:18.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:17:18.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 ceph-mon[30933]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-15T23:17:18.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:17:18.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:17:18.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:17:18.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 sudo[44584]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:17:18.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 sudo[44584]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:18.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 sudo[44584]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:18.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 sudo[44584]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:18.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:18.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 ceph-mon[35586]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:17:18.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 ceph-mon[35586]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:17:18.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:17:18.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:17:18.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:17:18.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 ceph-mon[35586]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-15T23:17:18.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:17:18.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:17:18.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:17:19.066 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:18 smithi120 sudo[43837]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:17:19.066 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:18 smithi120 sudo[43837]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:19.066 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:18 smithi120 sudo[43837]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:19.066 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:18 smithi120 sudo[43837]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 sudo[43840]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:17:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 sudo[43840]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:17:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 sudo[43840]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:17:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 sudo[43840]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:17:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:17:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:17:19.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:17:19.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:17:19.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:17:19.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-15T23:17:19.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:17:19.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:17:19.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:17:19.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:19 smithi089 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:17:19.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:19 smithi089 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:17:19.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:19 smithi089 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:17:19.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:17:19.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:17:19.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:17:19.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:19 smithi089 ceph-mon[30933]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:19.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:19 smithi089 ceph-mon[30933]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:17:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:17:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:17:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:17:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:17:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:17:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:17:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:19 smithi120 ceph-mon[35250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:17:19.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:19 smithi174 ceph-mon[35586]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:17:19.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:19 smithi174 ceph-mon[35586]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:17:19.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:19 smithi174 ceph-mon[35586]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:17:19.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:17:19.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:17:19.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:17:19.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:19 smithi174 ceph-mon[35586]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:19.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:19 smithi174 ceph-mon[35586]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:17:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:21 smithi120 ceph-mon[35250]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:22.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:21 smithi174 ceph-mon[35586]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:22.243 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:21 smithi089 ceph-mon[30933]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:23.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:23 smithi174 ceph-mon[35586]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:24.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:23 smithi120 ceph-mon[35250]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:24.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:23 smithi089 ceph-mon[30933]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:25.091 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-15T23:17:25.091 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:25.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-15T23:17:25.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-15T23:17:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:26.170 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:25 smithi089 ceph-mon[30933]: Deploying daemon osd.3 on smithi089 2024-09-15T23:17:26.170 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:25 smithi089 ceph-mon[30933]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:26.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:25 smithi120 ceph-mon[35250]: Deploying daemon osd.3 on smithi089 2024-09-15T23:17:26.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:25 smithi120 ceph-mon[35250]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:26.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:25 smithi174 ceph-mon[35586]: Deploying daemon osd.3 on smithi089 2024-09-15T23:17:26.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:25 smithi174 ceph-mon[35586]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:28.149 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:27 smithi089 ceph-mon[30933]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:28.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:27 smithi174 ceph-mon[35586]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:28.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:27 smithi120 ceph-mon[35250]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:30.211 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:29 smithi089 ceph-mon[30933]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:30.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:29 smithi174 ceph-mon[35586]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:29 smithi120 ceph-mon[35250]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:31.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:31.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:31.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:31.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:31.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:31.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:32.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:31 smithi174 ceph-mon[35586]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:31 smithi089 ceph-mon[30933]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:31 smithi120 ceph-mon[35250]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:34.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:33 smithi174 ceph-mon[35586]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:34.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:33 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:34.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:33 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:34.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:33 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:34.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:33 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:34.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:33 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:33 smithi089 ceph-mon[30933]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:33 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:33 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:34.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:33 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:34.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:33 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:34.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:33 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:33 smithi120 ceph-mon[35250]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:33 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:33 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:33 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:33 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:33 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:34.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.879+0000 7f14677fe640 1 -- 172.21.15.89:0/21207618 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1478062630 con 0x7f14600a31e0 2024-09-15T23:17:34.880 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 3 on host 'smithi089' 2024-09-15T23:17:34.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 -- 172.21.15.89:0/21207618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14600a31e0 msgr2=0x7f14600a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:34.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14600a31e0 0x7f14600a56a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1478102c50 tx=0x7f1468009110 comp rx=0 tx=0).stop 2024-09-15T23:17:34.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 -- 172.21.15.89:0/21207618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14781037b0 msgr2=0x7f147819f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:34.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14781037b0 0x7f147819f4a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1470005e30 tx=0x7f1470031ee0 comp rx=0 tx=0).stop 2024-09-15T23:17:34.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 -- 172.21.15.89:0/21207618 shutdown_connections 2024-09-15T23:17:34.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14600a31e0 0x7f14600a56a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:34.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14781037b0 0x7f147819f4a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:34.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1478102df0 0x7f1478198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:34.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 --2- 172.21.15.89:0/21207618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1478101bf0 0x7f1478197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:34.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 -- 172.21.15.89:0/21207618 >> 172.21.15.89:0/21207618 conn(0x7f14780fd3c0 msgr2=0x7f14781060f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:34.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 -- 172.21.15.89:0/21207618 shutdown_connections 2024-09-15T23:17:34.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:17:34.882+0000 7f147cf6c640 1 -- 172.21.15.89:0/21207618 wait complete. 2024-09-15T23:17:35.218 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:34 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:35.218 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:34 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:35.219 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:34 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:35.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:34 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:35.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:34 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:35.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:34 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:34 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:34 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:34 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:35.495 DEBUG:teuthology.orchestra.run.smithi089:osd.3> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.3.service 2024-09-15T23:17:35.497 INFO:tasks.cephadm:Deploying osd.4 on smithi120 with /dev/vg_nvme/lv_3... 2024-09-15T23:17:35.497 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-15T23:17:35.750 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:17:36.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:35 smithi089 ceph-mon[30933]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:36.163 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:35 smithi120 ceph-mon[35250]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:35 smithi174 ceph-mon[35586]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:36.810 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:17:36 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3[50711]: 2024-09-15T23:17:36.635+0000 7f4edd82d640 -1 osd.3 0 log_to_monitors true 2024-09-15T23:17:37.023 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:36 smithi120 ceph-mon[35250]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-15T23:17:37.088 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:36 smithi089 ceph-mon[30933]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-15T23:17:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:36 smithi174 ceph-mon[35586]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-15T23:17:38.078 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:17:38.103 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_3 2024-09-15T23:17:38.109 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:37 smithi089 ceph-mon[30933]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:38.109 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:37 smithi089 ceph-mon[30933]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-15T23:17:38.109 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:37 smithi089 ceph-mon[30933]: osdmap e21: 4 total, 3 up, 4 in 2024-09-15T23:17:38.109 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:37 smithi089 ceph-mon[30933]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:17:38.109 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:37 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:38.115 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:37 smithi120 ceph-mon[35250]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:38.115 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:37 smithi120 ceph-mon[35250]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-15T23:17:38.115 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:37 smithi120 ceph-mon[35250]: osdmap e21: 4 total, 3 up, 4 in 2024-09-15T23:17:38.115 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:37 smithi120 ceph-mon[35250]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:17:38.116 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:37 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:38.353 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:17:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:37 smithi174 ceph-mon[35586]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:37 smithi174 ceph-mon[35586]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-15T23:17:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:37 smithi174 ceph-mon[35586]: osdmap e21: 4 total, 3 up, 4 in 2024-09-15T23:17:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:37 smithi174 ceph-mon[35586]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:17:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:37 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:38.810 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3[50711]: 2024-09-15T23:17:38.400+0000 7f4ed9fc4640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T23:17:38.810 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3[50711]: 2024-09-15T23:17:38.404+0000 7f4ed15d0640 -1 osd.3 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:17:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.888+0000 7fef60e2f640 1 -- 172.21.15.120:0/1289441506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5c101bf0 msgr2=0x7fef5c101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.888+0000 7fef60e2f640 1 --2- 172.21.15.120:0/1289441506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5c101bf0 0x7fef5c101ff0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fef54009900 tx=0x7fef5402a3f0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.889+0000 7fef60e2f640 1 -- 172.21.15.120:0/1289441506 shutdown_connections 2024-09-15T23:17:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.889+0000 7fef60e2f640 1 --2- 172.21.15.120:0/1289441506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef5c1037b0 0x7fef5c10a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.889+0000 7fef60e2f640 1 --2- 172.21.15.120:0/1289441506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef5c102df0 0x7fef5c103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.889+0000 7fef60e2f640 1 --2- 172.21.15.120:0/1289441506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5c101bf0 0x7fef5c101ff0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.889+0000 7fef60e2f640 1 -- 172.21.15.120:0/1289441506 >> 172.21.15.120:0/1289441506 conn(0x7fef5c0fd3c0 msgr2=0x7fef5c0ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.889+0000 7fef60e2f640 1 -- 172.21.15.120:0/1289441506 shutdown_connections 2024-09-15T23:17:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.889+0000 7fef60e2f640 1 -- 172.21.15.120:0/1289441506 wait complete. 2024-09-15T23:17:38.890 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.889+0000 7fef60e2f640 1 Processor -- start 2024-09-15T23:17:38.891 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.889+0000 7fef60e2f640 1 -- start start 2024-09-15T23:17:38.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef60e2f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5c101bf0 0x7fef5c075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef60e2f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef5c102df0 0x7fef5c0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef60e2f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef5c1037b0 0x7fef5c074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef60e2f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10d850 con 0x7fef5c1037b0 2024-09-15T23:17:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef60e2f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10d6d0 con 0x7fef5c101bf0 2024-09-15T23:17:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef60e2f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef5c10d9d0 con 0x7fef5c102df0 2024-09-15T23:17:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef5a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5c101bf0 0x7fef5c075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef5a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5c101bf0 0x7fef5c075f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:44760/0 (socket says 172.21.15.120:44760) 2024-09-15T23:17:38.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef5a59c640 1 -- 172.21.15.120:0/768595808 learned_addr learned my addr 172.21.15.120:0/768595808 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef59d9b640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef5c102df0 0x7fef5c0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef5a59c640 1 -- 172.21.15.120:0/768595808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef5c102df0 msgr2=0x7fef5c0745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef5a59c640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef5c102df0 0x7fef5c0745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef5a59c640 1 -- 172.21.15.120:0/768595808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef5c1037b0 msgr2=0x7fef5c074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef5a59c640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef5c1037b0 0x7fef5c074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef5a59c640 1 -- 172.21.15.120:0/768595808 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef5c0753e0 con 0x7fef5c101bf0 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef59d9b640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef5c102df0 0x7fef5c0745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef5a59c640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5c101bf0 0x7fef5c075f90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fef54031610 tx=0x7fef54004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef537fe640 1 -- 172.21.15.120:0/768595808 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef5403f030 con 0x7fef5c101bf0 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef537fe640 1 -- 172.21.15.120:0/768595808 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef54003ad0 con 0x7fef5c101bf0 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.890+0000 7fef537fe640 1 -- 172.21.15.120:0/768595808 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef54035070 con 0x7fef5c101bf0 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.891+0000 7fef60e2f640 1 -- 172.21.15.120:0/768595808 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef5c1a2000 con 0x7fef5c101bf0 2024-09-15T23:17:38.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.891+0000 7fef60e2f640 1 -- 172.21.15.120:0/768595808 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef5c1a2460 con 0x7fef5c101bf0 2024-09-15T23:17:38.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.892+0000 7fef60e2f640 1 -- 172.21.15.120:0/768595808 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef5c108ff0 con 0x7fef5c101bf0 2024-09-15T23:17:38.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.893+0000 7fef537fe640 1 -- 172.21.15.120:0/768595808 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fef54042030 con 0x7fef5c101bf0 2024-09-15T23:17:38.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.894+0000 7fef537fe640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef480a3240 0x7fef480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.894+0000 7fef537fe640 1 -- 172.21.15.120:0/768595808 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 3702+0+0 (secure 0 0 0) 0x7fef540eedf0 con 0x7fef5c101bf0 2024-09-15T23:17:38.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.894+0000 7fef59d9b640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef480a3240 0x7fef480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.895+0000 7fef59d9b640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef480a3240 0x7fef480a5700 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fef5c075d10 tx=0x7fef44005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:38.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:38.898+0000 7fef537fe640 1 -- 172.21.15.120:0/768595808 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fef540bcd10 con 0x7fef5c101bf0 2024-09-15T23:17:39.079 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:17:39.078+0000 7fef60e2f640 1 -- 172.21.15.120:0/768595808 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fef5c062630 con 0x7fef480a3240 2024-09-15T23:17:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:17:39.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: osdmap e22: 4 total, 3 up, 4 in 2024-09-15T23:17:39.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:39.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:39.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:17:39.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:17:39.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:39.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:39.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' 2024-09-15T23:17:39.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] boot 2024-09-15T23:17:39.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: osdmap e23: 4 total, 4 up, 4 in 2024-09-15T23:17:39.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:39.358 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:17:39.358 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: osdmap e22: 4 total, 3 up, 4 in 2024-09-15T23:17:39.358 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:39.358 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:39.359 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.359 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.359 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:17:39.359 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:17:39.359 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.360 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:39.360 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:39.360 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.360 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' 2024-09-15T23:17:39.360 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] boot 2024-09-15T23:17:39.360 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: osdmap e23: 4 total, 4 up, 4 in 2024-09-15T23:17:39.361 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:39.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:17:39.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: osdmap e22: 4 total, 3 up, 4 in 2024-09-15T23:17:39.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:39.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:39.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:17:39.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:17:39.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:39.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:39.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:39.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139]' entity='osd.3' 2024-09-15T23:17:39.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: osd.3 [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] boot 2024-09-15T23:17:39.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: osdmap e23: 4 total, 4 up, 4 in 2024-09-15T23:17:39.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:17:40.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:39 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:17:40.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:39 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:17:40.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:39 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi089 2024-09-15T23:17:40.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:39 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi089 to 8543M 2024-09-15T23:17:40.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:39 smithi089 ceph-mon[30933]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:40.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:17:40.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:17:40.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:40.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:39 smithi089 ceph-mon[30933]: osdmap e24: 4 total, 4 up, 4 in 2024-09-15T23:17:40.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:39 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:17:40.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:39 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:17:40.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:39 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi089 2024-09-15T23:17:40.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:39 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi089 to 8543M 2024-09-15T23:17:40.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:39 smithi174 ceph-mon[35586]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:40.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:17:40.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:17:40.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:40.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:39 smithi174 ceph-mon[35586]: osdmap e24: 4 total, 4 up, 4 in 2024-09-15T23:17:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:39 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:17:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:39 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:17:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:39 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi089 2024-09-15T23:17:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:39 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi089 to 8543M 2024-09-15T23:17:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:39 smithi120 ceph-mon[35250]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:17:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:39 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:17:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:39 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:17:40.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:39 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:40.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:39 smithi120 ceph-mon[35250]: osdmap e24: 4 total, 4 up, 4 in 2024-09-15T23:17:41.167 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:41 smithi120 ceph-mon[35250]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:17:41.167 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:41 smithi120 ceph-mon[35250]: osdmap e25: 4 total, 4 up, 4 in 2024-09-15T23:17:41.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:41 smithi089 ceph-mon[30933]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:17:41.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:41 smithi089 ceph-mon[30933]: osdmap e25: 4 total, 4 up, 4 in 2024-09-15T23:17:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:41 smithi174 ceph-mon[35586]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:17:41.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:41 smithi174 ceph-mon[35586]: osdmap e25: 4 total, 4 up, 4 in 2024-09-15T23:17:42.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:42 smithi089 ceph-mon[30933]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:42.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:42 smithi089 ceph-mon[30933]: osdmap e26: 4 total, 4 up, 4 in 2024-09-15T23:17:42.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:42 smithi089 ceph-mon[30933]: from='client.? 172.21.15.120:0/3592996990' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "425a48e4-145a-4f5c-88af-d7be7fdd3c51"}]: dispatch 2024-09-15T23:17:42.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:42 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "425a48e4-145a-4f5c-88af-d7be7fdd3c51"}]: dispatch 2024-09-15T23:17:42.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:42 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "425a48e4-145a-4f5c-88af-d7be7fdd3c51"}]': finished 2024-09-15T23:17:42.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:42 smithi089 ceph-mon[30933]: osdmap e27: 5 total, 4 up, 5 in 2024-09-15T23:17:42.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:42 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:17:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:42 smithi120 ceph-mon[35250]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:42 smithi120 ceph-mon[35250]: osdmap e26: 4 total, 4 up, 4 in 2024-09-15T23:17:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:42 smithi120 ceph-mon[35250]: from='client.? 172.21.15.120:0/3592996990' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "425a48e4-145a-4f5c-88af-d7be7fdd3c51"}]: dispatch 2024-09-15T23:17:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:42 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "425a48e4-145a-4f5c-88af-d7be7fdd3c51"}]: dispatch 2024-09-15T23:17:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:42 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "425a48e4-145a-4f5c-88af-d7be7fdd3c51"}]': finished 2024-09-15T23:17:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:42 smithi120 ceph-mon[35250]: osdmap e27: 5 total, 4 up, 5 in 2024-09-15T23:17:42.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:42 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:17:42.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:42 smithi174 ceph-mon[35586]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:42.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:42 smithi174 ceph-mon[35586]: osdmap e26: 4 total, 4 up, 4 in 2024-09-15T23:17:42.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:42 smithi174 ceph-mon[35586]: from='client.? 172.21.15.120:0/3592996990' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "425a48e4-145a-4f5c-88af-d7be7fdd3c51"}]: dispatch 2024-09-15T23:17:42.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:42 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "425a48e4-145a-4f5c-88af-d7be7fdd3c51"}]: dispatch 2024-09-15T23:17:42.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:42 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "425a48e4-145a-4f5c-88af-d7be7fdd3c51"}]': finished 2024-09-15T23:17:42.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:42 smithi174 ceph-mon[35586]: osdmap e27: 5 total, 4 up, 5 in 2024-09-15T23:17:42.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:42 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:17:43.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:43 smithi089 ceph-mon[30933]: from='client.? 172.21.15.120:0/293583639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:17:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:43 smithi120 ceph-mon[35250]: from='client.? 172.21.15.120:0/293583639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:17:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:43 smithi174 ceph-mon[35586]: from='client.? 172.21.15.120:0/293583639' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:17:44.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:44 smithi089 ceph-mon[30933]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:44.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:44 smithi120 ceph-mon[35250]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:44.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:44 smithi174 ceph-mon[35586]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:46.289 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:46 smithi120 ceph-mon[35250]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-09-15T23:17:46.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:46 smithi089 ceph-mon[30933]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-09-15T23:17:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:46 smithi174 ceph-mon[35586]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-09-15T23:17:48.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:48 smithi089 ceph-mon[30933]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-15T23:17:48.428 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:48 smithi120 ceph-mon[35250]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-15T23:17:48.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:48 smithi174 ceph-mon[35586]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-15T23:17:50.070 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:50 smithi120 ceph-mon[35250]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:17:50.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:50 smithi089 ceph-mon[30933]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:17:50.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:50 smithi174 ceph-mon[35586]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:17:51.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-15T23:17:51.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:51.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-15T23:17:51.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-15T23:17:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:52.122 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:52 smithi120 ceph-mon[35250]: Deploying daemon osd.4 on smithi120 2024-09-15T23:17:52.122 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:52 smithi120 ceph-mon[35250]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-15T23:17:52.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:52 smithi089 ceph-mon[30933]: Deploying daemon osd.4 on smithi120 2024-09-15T23:17:52.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:52 smithi089 ceph-mon[30933]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-15T23:17:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:52 smithi174 ceph-mon[35586]: Deploying daemon osd.4 on smithi120 2024-09-15T23:17:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:52 smithi174 ceph-mon[35586]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-15T23:17:54.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:54 smithi120 ceph-mon[35250]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-09-15T23:17:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:54 smithi089 ceph-mon[30933]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-09-15T23:17:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:54 smithi174 ceph-mon[35586]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-09-15T23:17:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:56 smithi120 ceph-mon[35250]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:17:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:56 smithi089 ceph-mon[30933]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:17:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:56 smithi174 ceph-mon[35586]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:17:57.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:57.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:57.313 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:57.335 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:57.335 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:57.335 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:57.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:57.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:17:58.275 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:58 smithi120 ceph-mon[35250]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:58 smithi089 ceph-mon[30933]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:58.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:58 smithi174 ceph-mon[35586]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:59.188 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:17:59 smithi120 ceph-mon[35250]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:59.454 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:17:59 smithi089 ceph-mon[30933]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:17:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:17:59 smithi174 ceph-mon[35586]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:18:00.567 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:00.568 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:00.568 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:00.568 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:00.568 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:00.611 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.609+0000 7fef537fe640 1 -- 172.21.15.120:0/768595808 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fef5c062630 con 0x7fef480a3240 2024-09-15T23:18:00.612 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 4 on host 'smithi120' 2024-09-15T23:18:00.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 -- 172.21.15.120:0/768595808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef480a3240 msgr2=0x7fef480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.613 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef480a3240 0x7fef480a5700 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fef5c075d10 tx=0x7fef44005f90 comp rx=0 tx=0).stop 2024-09-15T23:18:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 -- 172.21.15.120:0/768595808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5c101bf0 msgr2=0x7fef5c075f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5c101bf0 0x7fef5c075f90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fef54031610 tx=0x7fef54004270 comp rx=0 tx=0).stop 2024-09-15T23:18:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 -- 172.21.15.120:0/768595808 shutdown_connections 2024-09-15T23:18:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef480a3240 0x7fef480a5700 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef5c1037b0 0x7fef5c074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef5c102df0 0x7fef5c0745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 --2- 172.21.15.120:0/768595808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef5c101bf0 0x7fef5c075f90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 -- 172.21.15.120:0/768595808 >> 172.21.15.120:0/768595808 conn(0x7fef5c0fd3c0 msgr2=0x7fef5c1061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 -- 172.21.15.120:0/768595808 shutdown_connections 2024-09-15T23:18:00.614 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:00.612+0000 7fef60e2f640 1 -- 172.21.15.120:0/768595808 wait complete. 2024-09-15T23:18:00.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:00.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:00.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:00.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:00.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:00.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:00.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:01.249 DEBUG:teuthology.orchestra.run.smithi120:osd.4> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.4.service 2024-09-15T23:18:01.292 INFO:tasks.cephadm:Deploying osd.5 on smithi174 with /dev/vg_nvme/lv_3... 2024-09-15T23:18:01.292 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-15T23:18:01.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:01.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:01.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:01.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:01 smithi120 ceph-mon[35250]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:18:01.546 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:18:01.667 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:01.667 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:01.668 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:01.668 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:01 smithi174 ceph-mon[35586]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:18:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:01 smithi089 ceph-mon[30933]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:18:02.924 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:02 smithi120 ceph-mon[35250]: from='osd.4 [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:18:02.924 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:02 smithi120 ceph-mon[35250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:18:02.925 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:18:02 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4[46581]: 2024-09-15T23:18:02.639+0000 7f0f364ad640 -1 osd.4 0 log_to_monitors true 2024-09-15T23:18:02.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:02 smithi174 ceph-mon[35586]: from='osd.4 [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:18:02.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:02 smithi174 ceph-mon[35586]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:18:03.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:02 smithi089 ceph-mon[30933]: from='osd.4 [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:18:03.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:02 smithi089 ceph-mon[30933]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:18:03.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:03 smithi120 ceph-mon[35250]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-15T23:18:03.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:03 smithi120 ceph-mon[35250]: osdmap e28: 5 total, 4 up, 5 in 2024-09-15T23:18:03.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:03.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:03 smithi120 ceph-mon[35250]: from='osd.4 [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:18:03.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:03 smithi120 ceph-mon[35250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:18:03.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:03 smithi120 ceph-mon[35250]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:18:03.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:03 smithi174 ceph-mon[35586]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-15T23:18:03.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:03 smithi174 ceph-mon[35586]: osdmap e28: 5 total, 4 up, 5 in 2024-09-15T23:18:03.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:03.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:03 smithi174 ceph-mon[35586]: from='osd.4 [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:18:03.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:03 smithi174 ceph-mon[35586]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:18:03.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:03 smithi174 ceph-mon[35586]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:18:04.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:03 smithi089 ceph-mon[30933]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-15T23:18:04.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:03 smithi089 ceph-mon[30933]: osdmap e28: 5 total, 4 up, 5 in 2024-09-15T23:18:04.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:04.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:03 smithi089 ceph-mon[30933]: from='osd.4 [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:18:04.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:03 smithi089 ceph-mon[30933]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:18:04.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:03 smithi089 ceph-mon[30933]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:18:04.355 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:18:04.382 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_3 2024-09-15T23:18:04.473 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4[46581]: 2024-09-15T23:18:04.083+0000 7f0f32430640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T23:18:04.473 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4[46581]: 2024-09-15T23:18:04.088+0000 7f0f2a250640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:18:04.636 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:18:04.921 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: osdmap e29: 5 total, 4 up, 5 in 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='osd.4 ' entity='osd.4' 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:04.922 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:04.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:18:04.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: osdmap e29: 5 total, 4 up, 5 in 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='osd.4 ' entity='osd.4' 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:04.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:05.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:18:05.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: osdmap e29: 5 total, 4 up, 5 in 2024-09-15T23:18:05.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:05.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:05.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:05.062 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:05.062 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:05.062 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='osd.4 ' entity='osd.4' 2024-09-15T23:18:05.062 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:05.062 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:05.062 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:05.063 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:05.063 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:05.063 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:05.130 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.116+0000 7f5335584640 1 -- 172.21.15.174:0/1639409563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5330101780 msgr2=0x7f5330109020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:05.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.116+0000 7f5335584640 1 --2- 172.21.15.174:0/1639409563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5330101780 0x7f5330109020 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5328009ca0 tx=0x7f532802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:18:05.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.116+0000 7f5335584640 1 -- 172.21.15.174:0/1639409563 shutdown_connections 2024-09-15T23:18:05.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.116+0000 7f5335584640 1 --2- 172.21.15.174:0/1639409563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5330101780 0x7f5330109020 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:05.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.116+0000 7f5335584640 1 --2- 172.21.15.174:0/1639409563 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5330100dc0 0x7f5330101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:05.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.116+0000 7f5335584640 1 --2- 172.21.15.174:0/1639409563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5330102160 0x7f5330102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:05.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.116+0000 7f5335584640 1 -- 172.21.15.174:0/1639409563 >> 172.21.15.174:0/1639409563 conn(0x7f53300fba80 msgr2=0x7f53300fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:05.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.116+0000 7f5335584640 1 -- 172.21.15.174:0/1639409563 shutdown_connections 2024-09-15T23:18:05.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.116+0000 7f5335584640 1 -- 172.21.15.174:0/1639409563 wait complete. 2024-09-15T23:18:05.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5335584640 1 Processor -- start 2024-09-15T23:18:05.131 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5335584640 1 -- start start 2024-09-15T23:18:05.132 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5335584640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5330100dc0 0x7f5330197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:05.132 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5335584640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5330101780 0x7f5330198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:05.132 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5335584640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5330102160 0x7f533019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:05.132 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5335584640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f533010c780 con 0x7f5330100dc0 2024-09-15T23:18:05.132 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5335584640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f533010c600 con 0x7f5330102160 2024-09-15T23:18:05.132 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5335584640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f533010c900 con 0x7f5330101780 2024-09-15T23:18:05.132 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5327fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5330101780 0x7f5330198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5327fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5330101780 0x7f5330198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:50600/0 (socket says 172.21.15.174:50600) 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5327fff640 1 -- 172.21.15.174:0/1559403394 learned_addr learned my addr 172.21.15.174:0/1559403394 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f532effd640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5330100dc0 0x7f5330197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f532f7fe640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5330102160 0x7f533019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5327fff640 1 -- 172.21.15.174:0/1559403394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5330102160 msgr2=0x7f533019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5327fff640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5330102160 0x7f533019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5327fff640 1 -- 172.21.15.174:0/1559403394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5330100dc0 msgr2=0x7f5330197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5327fff640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5330100dc0 0x7f5330197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.117+0000 7f5327fff640 1 -- 172.21.15.174:0/1559403394 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533019fb30 con 0x7f5330101780 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.118+0000 7f5327fff640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5330101780 0x7f5330198440 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f531800d770 tx=0x7f531800dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.118+0000 7f532cff9640 1 -- 172.21.15.174:0/1559403394 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5318017070 con 0x7f5330101780 2024-09-15T23:18:05.133 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.118+0000 7f5335584640 1 -- 172.21.15.174:0/1559403394 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533019fe20 con 0x7f5330101780 2024-09-15T23:18:05.134 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.118+0000 7f532cff9640 1 -- 172.21.15.174:0/1559403394 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5318009c30 con 0x7f5330101780 2024-09-15T23:18:05.134 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.118+0000 7f532cff9640 1 -- 172.21.15.174:0/1559403394 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5318003dd0 con 0x7f5330101780 2024-09-15T23:18:05.134 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.118+0000 7f5335584640 1 -- 172.21.15.174:0/1559403394 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53301a02d0 con 0x7f5330101780 2024-09-15T23:18:05.134 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.119+0000 7f5335584640 1 -- 172.21.15.174:0/1559403394 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53301a0500 con 0x7f5330101780 2024-09-15T23:18:05.134 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.120+0000 7f532cff9640 1 -- 172.21.15.174:0/1559403394 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5318004000 con 0x7f5330101780 2024-09-15T23:18:05.134 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.120+0000 7f532cff9640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f531c0a3270 0x7f531c0a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:05.134 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.121+0000 7f532cff9640 1 -- 172.21.15.174:0/1559403394 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 4101+0+0 (secure 0 0 0) 0x7f53180ce200 con 0x7f5330101780 2024-09-15T23:18:05.134 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.121+0000 7f532effd640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f531c0a3270 0x7f531c0a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:05.134 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.121+0000 7f532effd640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f531c0a3270 0x7f531c0a5730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5320005fd0 tx=0x7f532000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:05.134 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.125+0000 7f532cff9640 1 -- 172.21.15.174:0/1559403394 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f531809bf90 con 0x7f5330101780 2024-09-15T23:18:05.308 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:05.306+0000 7f5335584640 1 -- 172.21.15.174:0/1559403394 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f53301004b0 con 0x7f531c0a3270 2024-09-15T23:18:05.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:18:05.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:18:05.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi120 2024-09-15T23:18:05.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi120 to 8544M 2024-09-15T23:18:05.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:18:05.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: osd.4 [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] boot 2024-09-15T23:18:05.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: osdmap e30: 5 total, 5 up, 5 in 2024-09-15T23:18:05.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:05.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:18:05.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:18:05.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:06.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:18:06.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:18:06.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi120 2024-09-15T23:18:06.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi120 to 8544M 2024-09-15T23:18:06.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:18:06.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: osd.4 [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] boot 2024-09-15T23:18:06.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: osdmap e30: 5 total, 5 up, 5 in 2024-09-15T23:18:06.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:06.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:18:06.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:18:06.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:06.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:18:06.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:18:06.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi120 2024-09-15T23:18:06.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi120 to 8544M 2024-09-15T23:18:06.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-09-15T23:18:06.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: osd.4 [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] boot 2024-09-15T23:18:06.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: osdmap e30: 5 total, 5 up, 5 in 2024-09-15T23:18:06.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:18:06.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:18:06.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:18:06.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:06.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:06 smithi174 ceph-mon[35586]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:18:06.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:06 smithi174 ceph-mon[35586]: osdmap e31: 5 total, 5 up, 5 in 2024-09-15T23:18:07.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:06 smithi089 ceph-mon[30933]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:18:07.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:06 smithi089 ceph-mon[30933]: osdmap e31: 5 total, 5 up, 5 in 2024-09-15T23:18:07.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:06 smithi120 ceph-mon[35250]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:18:07.247 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:06 smithi120 ceph-mon[35250]: osdmap e31: 5 total, 5 up, 5 in 2024-09-15T23:18:07.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:07 smithi174 ceph-mon[35586]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:08.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:07 smithi089 ceph-mon[30933]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:08.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:07 smithi120 ceph-mon[35250]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:08.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:08 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/3544592287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0548511d-1178-448e-8549-0a934b57cfdc"}]: dispatch 2024-09-15T23:18:08.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:08 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0548511d-1178-448e-8549-0a934b57cfdc"}]: dispatch 2024-09-15T23:18:08.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:08 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0548511d-1178-448e-8549-0a934b57cfdc"}]': finished 2024-09-15T23:18:08.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:08 smithi174 ceph-mon[35586]: osdmap e32: 6 total, 5 up, 6 in 2024-09-15T23:18:08.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:08.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:08 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/3825699564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:18:09.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:08 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/3544592287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0548511d-1178-448e-8549-0a934b57cfdc"}]: dispatch 2024-09-15T23:18:09.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:08 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0548511d-1178-448e-8549-0a934b57cfdc"}]: dispatch 2024-09-15T23:18:09.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:08 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0548511d-1178-448e-8549-0a934b57cfdc"}]': finished 2024-09-15T23:18:09.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:08 smithi089 ceph-mon[30933]: osdmap e32: 6 total, 5 up, 6 in 2024-09-15T23:18:09.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:09.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:08 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/3825699564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:18:09.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:08 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/3544592287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0548511d-1178-448e-8549-0a934b57cfdc"}]: dispatch 2024-09-15T23:18:09.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:08 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0548511d-1178-448e-8549-0a934b57cfdc"}]: dispatch 2024-09-15T23:18:09.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:08 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0548511d-1178-448e-8549-0a934b57cfdc"}]': finished 2024-09-15T23:18:09.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:08 smithi120 ceph-mon[35250]: osdmap e32: 6 total, 5 up, 6 in 2024-09-15T23:18:09.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:09.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:08 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/3825699564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:18:09.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:09 smithi089 ceph-mon[30933]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:10.124 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:09 smithi120 ceph-mon[35250]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:10.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:09 smithi174 ceph-mon[35586]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:12.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:11 smithi174 ceph-mon[35586]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:12.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:11 smithi089 ceph-mon[30933]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:11 smithi120 ceph-mon[35250]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:14.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:13 smithi174 ceph-mon[35586]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:14.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:13 smithi089 ceph-mon[30933]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:14.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:13 smithi120 ceph-mon[35250]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:16.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:15 smithi174 ceph-mon[35586]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:16.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:15 smithi089 ceph-mon[30933]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:15 smithi120 ceph-mon[35250]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:17.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:16 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-15T23:18:17.079 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:16 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:17.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:16 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-15T23:18:17.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:16 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:16 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-15T23:18:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:16 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:18.133 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:17 smithi174 ceph-mon[35586]: Deploying daemon osd.5 on smithi174 2024-09-15T23:18:18.133 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:17 smithi174 ceph-mon[35586]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:17 smithi089 ceph-mon[30933]: Deploying daemon osd.5 on smithi174 2024-09-15T23:18:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:17 smithi089 ceph-mon[30933]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:18.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:17 smithi120 ceph-mon[35250]: Deploying daemon osd.5 on smithi174 2024-09-15T23:18:18.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:17 smithi120 ceph-mon[35250]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:20.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:19 smithi174 ceph-mon[35586]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:20.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:19 smithi089 ceph-mon[30933]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:19 smithi120 ceph-mon[35250]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:22.291 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:21 smithi174 ceph-mon[35586]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:21 smithi089 ceph-mon[30933]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:22.340 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:21 smithi120 ceph-mon[35250]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:23.261 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:23.262 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:23.262 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:24.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:23 smithi174 ceph-mon[35586]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:24.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:23 smithi089 ceph-mon[30933]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:23 smithi120 ceph-mon[35250]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:25.924 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.922+0000 7f532cff9640 1 -- 172.21.15.174:0/1559403394 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f53301004b0 con 0x7f531c0a3270 2024-09-15T23:18:25.925 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 5 on host 'smithi174' 2024-09-15T23:18:25.926 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 -- 172.21.15.174:0/1559403394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f531c0a3270 msgr2=0x7f531c0a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:25.926 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f531c0a3270 0x7f531c0a5730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5320005fd0 tx=0x7f532000b390 comp rx=0 tx=0).stop 2024-09-15T23:18:25.926 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 -- 172.21.15.174:0/1559403394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5330101780 msgr2=0x7f5330198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:25.926 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5330101780 0x7f5330198440 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f531800d770 tx=0x7f531800dc30 comp rx=0 tx=0).stop 2024-09-15T23:18:25.926 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 -- 172.21.15.174:0/1559403394 shutdown_connections 2024-09-15T23:18:25.926 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f531c0a3270 0x7f531c0a5730 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:25.927 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5330102160 0x7f533019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:25.927 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5330101780 0x7f5330198440 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:25.927 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 --2- 172.21.15.174:0/1559403394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5330100dc0 0x7f5330197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:25.927 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 -- 172.21.15.174:0/1559403394 >> 172.21.15.174:0/1559403394 conn(0x7f53300fba80 msgr2=0x7f53300ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:25.927 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 -- 172.21.15.174:0/1559403394 shutdown_connections 2024-09-15T23:18:25.927 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:18:25.925+0000 7f5335584640 1 -- 172.21.15.174:0/1559403394 wait complete. 2024-09-15T23:18:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:26 smithi174 ceph-mon[35586]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:26 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:26 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:26 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:26 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:26 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:26 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:26 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:26 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:26 smithi089 ceph-mon[30933]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:26 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:26 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:26 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:26.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:26 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:26.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:26 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:26 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:26.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:26 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:26 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:26 smithi120 ceph-mon[35250]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:26 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:26 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:26 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:26.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:26 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:26.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:26 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:26 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:26.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:26 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:26 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:26.615 DEBUG:teuthology.orchestra.run.smithi174:osd.5> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.5.service 2024-09-15T23:18:26.620 INFO:tasks.cephadm:Deploying osd.6 on smithi089 with /dev/vg_nvme/lv_2... 2024-09-15T23:18:26.620 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-15T23:18:26.878 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:28.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:28 smithi089 ceph-mon[30933]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:28.374 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:28 smithi174 ceph-mon[35586]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:28.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:28 smithi120 ceph-mon[35250]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:29.012 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:18:28 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:18:28.883+0000 7fe8b659d640 -1 osd.5 0 log_to_monitors true 2024-09-15T23:18:29.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:29 smithi089 ceph-mon[30933]: from='osd.5 [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:18:29.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:29 smithi089 ceph-mon[30933]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:18:29.436 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:29 smithi174 ceph-mon[35586]: from='osd.5 [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:18:29.436 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:29 smithi174 ceph-mon[35586]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:18:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:29 smithi120 ceph-mon[35250]: from='osd.5 [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:18:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:29 smithi120 ceph-mon[35250]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:18:29.525 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:18:29.555 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi089:vg_nvme/lv_2 2024-09-15T23:18:29.810 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:30.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:30.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-15T23:18:30.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='osd.5 [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:18:30.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: osdmap e33: 6 total, 5 up, 6 in 2024-09-15T23:18:30.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:30.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:18:30.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:30.038 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:30.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:30.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:30.039 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:30 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.234+0000 7f0948fae640 1 -- 172.21.15.89:0/4105298618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944102690 msgr2=0x7f0944108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.234+0000 7f0948fae640 1 --2- 172.21.15.89:0/4105298618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944102690 0x7f0944108f20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f093c009ca0 tx=0x7f093c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.234+0000 7f0948fae640 1 -- 172.21.15.89:0/4105298618 shutdown_connections 2024-09-15T23:18:30.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.234+0000 7f0948fae640 1 --2- 172.21.15.89:0/4105298618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944102690 0x7f0944108f20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.234+0000 7f0948fae640 1 --2- 172.21.15.89:0/4105298618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0944101cd0 0x7f0944102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.234+0000 7f0948fae640 1 --2- 172.21.15.89:0/4105298618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0944100ad0 0x7f0944100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.234+0000 7f0948fae640 1 -- 172.21.15.89:0/4105298618 >> 172.21.15.89:0/4105298618 conn(0x7f09440fc280 msgr2=0x7f09440fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:30.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.234+0000 7f0948fae640 1 -- 172.21.15.89:0/4105298618 shutdown_connections 2024-09-15T23:18:30.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.234+0000 7f0948fae640 1 -- 172.21.15.89:0/4105298618 wait complete. 2024-09-15T23:18:30.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f0948fae640 1 Processor -- start 2024-09-15T23:18:30.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f0948fae640 1 -- start start 2024-09-15T23:18:30.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f0948fae640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944100ad0 0x7f094406aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f0948fae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0944101cd0 0x7f094419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f0948fae640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0944102690 0x7f094419f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f0948fae640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f094410c7b0 con 0x7f0944100ad0 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f0948fae640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f094410c630 con 0x7f0944102690 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f0948fae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f094410c930 con 0x7f0944101cd0 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f094259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944100ad0 0x7f094406aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f094259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944100ad0 0x7f094406aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44292/0 (socket says 172.21.15.89:44292) 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.235+0000 7f094259c640 1 -- 172.21.15.89:0/4006247756 learned_addr learned my addr 172.21.15.89:0/4006247756 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f0942d9d640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0944102690 0x7f094419f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f0941d9b640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0944101cd0 0x7f094419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f094259c640 1 -- 172.21.15.89:0/4006247756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0944101cd0 msgr2=0x7f094419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f094259c640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0944101cd0 0x7f094419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f094259c640 1 -- 172.21.15.89:0/4006247756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0944102690 msgr2=0x7f094419f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f094259c640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0944102690 0x7f094419f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f094259c640 1 -- 172.21.15.89:0/4006247756 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094419fb00 con 0x7f0944100ad0 2024-09-15T23:18:30.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f094259c640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944100ad0 0x7f094406aad0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f093400c970 tx=0x7f093400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:30.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f09337fe640 1 -- 172.21.15.89:0/4006247756 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0934007c10 con 0x7f0944100ad0 2024-09-15T23:18:30.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f0948fae640 1 -- 172.21.15.89:0/4006247756 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f094419fd90 con 0x7f0944100ad0 2024-09-15T23:18:30.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f09337fe640 1 -- 172.21.15.89:0/4006247756 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0934007db0 con 0x7f0944100ad0 2024-09-15T23:18:30.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f09337fe640 1 -- 172.21.15.89:0/4006247756 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093400f8a0 con 0x7f0944100ad0 2024-09-15T23:18:30.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.236+0000 7f0948fae640 1 -- 172.21.15.89:0/4006247756 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09441a02d0 con 0x7f0944100ad0 2024-09-15T23:18:30.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.237+0000 7f0948fae640 1 -- 172.21.15.89:0/4006247756 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0944107e20 con 0x7f0944100ad0 2024-09-15T23:18:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.238+0000 7f09337fe640 1 -- 172.21.15.89:0/4006247756 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f093400fa40 con 0x7f0944100ad0 2024-09-15T23:18:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.239+0000 7f09337fe640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f092c0a3240 0x7f092c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.239+0000 7f0941d9b640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f092c0a3240 0x7f092c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.239+0000 7f09337fe640 1 -- 172.21.15.89:0/4006247756 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4253+0+0 (secure 0 0 0) 0x7f09340cefe0 con 0x7f0944100ad0 2024-09-15T23:18:30.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.240+0000 7f0941d9b640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f092c0a3240 0x7f092c0a5700 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f094406b050 tx=0x7f0938008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:30.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.243+0000 7f09337fe640 1 -- 172.21.15.89:0/4006247756 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f093409cda0 con 0x7f0944100ad0 2024-09-15T23:18:30.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:30.421+0000 7f0948fae640 1 -- 172.21.15.89:0/4006247756 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0944062630 con 0x7f092c0a3240 2024-09-15T23:18:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:30.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-15T23:18:30.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='osd.5 [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:18:30.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: osdmap e33: 6 total, 5 up, 6 in 2024-09-15T23:18:30.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:30.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:18:30.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:30.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:30.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:30.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:30.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:30 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.486 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:18:30.021+0000 7fe8b2520640 -1 osd.5 0 waiting for initial osdmap 2024-09-15T23:18:30.486 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:18:30.027+0000 7fe8ae348640 -1 osd.5 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:18:30.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:30.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-15T23:18:30.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='osd.5 [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:18:30.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: osdmap e33: 6 total, 5 up, 6 in 2024-09-15T23:18:30.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:30.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:18:30.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:30.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:30.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:30.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:30.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:30.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:30 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:31.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:31 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi174 2024-09-15T23:18:31.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:31 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi174 to 8544M 2024-09-15T23:18:31.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:31 smithi089 ceph-mon[30933]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:18:31.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:31 smithi089 ceph-mon[30933]: osdmap e34: 6 total, 5 up, 6 in 2024-09-15T23:18:31.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:31.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:31.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:18:31.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:18:31.062 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:31 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi174 2024-09-15T23:18:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:31 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi174 to 8544M 2024-09-15T23:18:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:31 smithi120 ceph-mon[35250]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:18:31.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:31 smithi120 ceph-mon[35250]: osdmap e34: 6 total, 5 up, 6 in 2024-09-15T23:18:31.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:31.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:31.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:18:31.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:18:31.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:31 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi174 2024-09-15T23:18:31.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:31 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi174 to 8544M 2024-09-15T23:18:31.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:31 smithi174 ceph-mon[35586]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:18:31.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:31 smithi174 ceph-mon[35586]: osdmap e34: 6 total, 5 up, 6 in 2024-09-15T23:18:31.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:31.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:31.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:18:31.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:18:31.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:32.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:32 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:18:32.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:32 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:18:32.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:32 smithi089 ceph-mon[30933]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:18:32.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:32 smithi089 ceph-mon[30933]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:32.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:32 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:32.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:32 smithi089 ceph-mon[30933]: osd.5 [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] boot 2024-09-15T23:18:32.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:32 smithi089 ceph-mon[30933]: osdmap e35: 6 total, 6 up, 6 in 2024-09-15T23:18:32.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:32 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:32 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:18:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:32 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:18:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:32 smithi120 ceph-mon[35250]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:18:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:32 smithi120 ceph-mon[35250]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:32.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:32 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:32.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:32 smithi120 ceph-mon[35250]: osd.5 [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] boot 2024-09-15T23:18:32.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:32 smithi120 ceph-mon[35250]: osdmap e35: 6 total, 6 up, 6 in 2024-09-15T23:18:32.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:32 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:32.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:32 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:18:32.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:32 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:18:32.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:32 smithi174 ceph-mon[35586]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:18:32.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:32 smithi174 ceph-mon[35586]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:18:32.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:32 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:32.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:32 smithi174 ceph-mon[35586]: osd.5 [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] boot 2024-09-15T23:18:32.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:32 smithi174 ceph-mon[35586]: osdmap e35: 6 total, 6 up, 6 in 2024-09-15T23:18:32.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:32 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:18:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:33 smithi089 ceph-mon[30933]: osdmap e36: 6 total, 6 up, 6 in 2024-09-15T23:18:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:33 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/91645988' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df2eab94-90c7-4847-998f-bccdef2861e1"}]: dispatch 2024-09-15T23:18:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:33 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/91645988' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df2eab94-90c7-4847-998f-bccdef2861e1"}]': finished 2024-09-15T23:18:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:33 smithi089 ceph-mon[30933]: osdmap e37: 7 total, 6 up, 7 in 2024-09-15T23:18:33.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:33 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:33 smithi120 ceph-mon[35250]: osdmap e36: 6 total, 6 up, 6 in 2024-09-15T23:18:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:33 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/91645988' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df2eab94-90c7-4847-998f-bccdef2861e1"}]: dispatch 2024-09-15T23:18:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:33 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/91645988' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df2eab94-90c7-4847-998f-bccdef2861e1"}]': finished 2024-09-15T23:18:33.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:33 smithi120 ceph-mon[35250]: osdmap e37: 7 total, 6 up, 7 in 2024-09-15T23:18:33.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:33 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:33 smithi174 ceph-mon[35586]: osdmap e36: 6 total, 6 up, 6 in 2024-09-15T23:18:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:33 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/91645988' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df2eab94-90c7-4847-998f-bccdef2861e1"}]: dispatch 2024-09-15T23:18:33.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:33 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/91645988' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df2eab94-90c7-4847-998f-bccdef2861e1"}]': finished 2024-09-15T23:18:33.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:33 smithi174 ceph-mon[35586]: osdmap e37: 7 total, 6 up, 7 in 2024-09-15T23:18:33.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:33 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:34 smithi089 ceph-mon[30933]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:34 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/115648776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:18:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:34 smithi089 ceph-mon[30933]: osdmap e38: 7 total, 6 up, 7 in 2024-09-15T23:18:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:34 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:34 smithi120 ceph-mon[35250]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:34 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/115648776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:18:34.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:34 smithi120 ceph-mon[35250]: osdmap e38: 7 total, 6 up, 7 in 2024-09-15T23:18:34.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:34 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:34 smithi174 ceph-mon[35586]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:34 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/115648776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:18:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:34 smithi174 ceph-mon[35586]: osdmap e38: 7 total, 6 up, 7 in 2024-09-15T23:18:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:34 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:35.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:35 smithi120 ceph-mon[35250]: osdmap e39: 7 total, 6 up, 7 in 2024-09-15T23:18:35.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:35 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:35.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:35 smithi120 ceph-mon[35250]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-09-15T23:18:35.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:35 smithi174 ceph-mon[35586]: osdmap e39: 7 total, 6 up, 7 in 2024-09-15T23:18:35.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:35 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:35.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:35 smithi174 ceph-mon[35586]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-09-15T23:18:35.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:35 smithi089 ceph-mon[30933]: osdmap e39: 7 total, 6 up, 7 in 2024-09-15T23:18:35.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:35 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:35.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:35 smithi089 ceph-mon[30933]: pgmap v136: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-09-15T23:18:38.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:37 smithi174 ceph-mon[35586]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-15T23:18:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:37 smithi089 ceph-mon[30933]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-15T23:18:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:37 smithi120 ceph-mon[35250]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-15T23:18:40.058 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:39 smithi089 ceph-mon[30933]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-15T23:18:40.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:39 smithi174 ceph-mon[35586]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-15T23:18:40.275 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:39 smithi120 ceph-mon[35250]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-15T23:18:42.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:41 smithi174 ceph-mon[35586]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:18:42.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-15T23:18:42.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:42.254 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:41 smithi089 ceph-mon[30933]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:18:42.254 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-15T23:18:42.255 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:41 smithi120 ceph-mon[35250]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:18:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-15T23:18:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:43.124 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:42 smithi089 ceph-mon[30933]: Deploying daemon osd.6 on smithi089 2024-09-15T23:18:43.247 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:42 smithi174 ceph-mon[35586]: Deploying daemon osd.6 on smithi089 2024-09-15T23:18:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:42 smithi120 ceph-mon[35250]: Deploying daemon osd.6 on smithi089 2024-09-15T23:18:44.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:43 smithi089 ceph-mon[30933]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-15T23:18:44.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:43 smithi120 ceph-mon[35250]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-15T23:18:44.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:43 smithi174 ceph-mon[35586]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-15T23:18:46.254 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:45 smithi089 ceph-mon[30933]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:45 smithi120 ceph-mon[35250]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:45 smithi174 ceph-mon[35586]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:48.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:47 smithi089 ceph-mon[30933]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:48.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:47 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:48.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:47 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:48.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:47 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:47 smithi120 ceph-mon[35250]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:47 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:47 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:47 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:48.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:47 smithi174 ceph-mon[35586]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:48.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:47 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:48.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:47 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:48.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:47 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:50.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:49 smithi089 ceph-mon[30933]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:50.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:50.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:50.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:49 smithi120 ceph-mon[35250]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:50.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:50.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:50.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:49 smithi174 ceph-mon[35586]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:50.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:50.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:50.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.917+0000 7f09337fe640 1 -- 172.21.15.89:0/4006247756 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0944062630 con 0x7f092c0a3240 2024-09-15T23:18:50.919 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 6 on host 'smithi089' 2024-09-15T23:18:50.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 -- 172.21.15.89:0/4006247756 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f092c0a3240 msgr2=0x7f092c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:50.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f092c0a3240 0x7f092c0a5700 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f094406b050 tx=0x7f0938008040 comp rx=0 tx=0).stop 2024-09-15T23:18:50.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 -- 172.21.15.89:0/4006247756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944100ad0 msgr2=0x7f094406aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:50.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944100ad0 0x7f094406aad0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f093400c970 tx=0x7f093400ce30 comp rx=0 tx=0).stop 2024-09-15T23:18:50.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 -- 172.21.15.89:0/4006247756 shutdown_connections 2024-09-15T23:18:50.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f092c0a3240 0x7f092c0a5700 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:50.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0944102690 0x7f094419f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:50.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0944101cd0 0x7f094419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:50.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 --2- 172.21.15.89:0/4006247756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0944100ad0 0x7f094406aad0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:50.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 -- 172.21.15.89:0/4006247756 >> 172.21.15.89:0/4006247756 conn(0x7f09440fc280 msgr2=0x7f0944104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:50.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 -- 172.21.15.89:0/4006247756 shutdown_connections 2024-09-15T23:18:50.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:18:50.919+0000 7f0948fae640 1 -- 172.21.15.89:0/4006247756 wait complete. 2024-09-15T23:18:51.257 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:51.258 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:51.258 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:51.258 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:51.258 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:51.258 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:51.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:51.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:51.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:51.594 DEBUG:teuthology.orchestra.run.smithi089:osd.6> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.6.service 2024-09-15T23:18:51.597 INFO:tasks.cephadm:Deploying osd.7 on smithi120 with /dev/vg_nvme/lv_2... 2024-09-15T23:18:51.598 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-15T23:18:51.842 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:18:52.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:52 smithi120 ceph-mon[35250]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:52.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:52 smithi089 ceph-mon[30933]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:52 smithi174 ceph-mon[35586]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:53.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:53 smithi089 ceph-mon[30933]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-15T23:18:53.061 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:18:52 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:18:52.963+0000 7f7196d03640 -1 osd.6 0 log_to_monitors true 2024-09-15T23:18:53.203 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:53 smithi120 ceph-mon[35250]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-15T23:18:53.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:53 smithi174 ceph-mon[35586]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-15T23:18:54.060 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:18:54 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:18:54.013+0000 7f7192c86640 -1 osd.6 0 waiting for initial osdmap 2024-09-15T23:18:54.061 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:18:54 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:18:54.022+0000 7f718eaae640 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:18:54.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:54 smithi089 ceph-mon[30933]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:54.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:54 smithi089 ceph-mon[30933]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-15T23:18:54.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:54 smithi089 ceph-mon[30933]: osdmap e40: 7 total, 6 up, 7 in 2024-09-15T23:18:54.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:54 smithi089 ceph-mon[30933]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:18:54.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:54.253 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:54 smithi120 ceph-mon[35250]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:54.253 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:54 smithi120 ceph-mon[35250]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-15T23:18:54.253 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:54 smithi120 ceph-mon[35250]: osdmap e40: 7 total, 6 up, 7 in 2024-09-15T23:18:54.256 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:54 smithi120 ceph-mon[35250]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:18:54.256 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:54.256 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:18:54.283 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_2 2024-09-15T23:18:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:54 smithi174 ceph-mon[35586]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:54.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:54 smithi174 ceph-mon[35586]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-15T23:18:54.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:54 smithi174 ceph-mon[35586]: osdmap e40: 7 total, 6 up, 7 in 2024-09-15T23:18:54.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:54 smithi174 ceph-mon[35586]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:18:54.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:54.533 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:18:55.048 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.041+0000 7fb0d26fc640 1 -- 172.21.15.120:0/615362381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0cc100ad0 msgr2=0x7fb0cc100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.041+0000 7fb0d26fc640 1 --2- 172.21.15.120:0/615362381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0cc100ad0 0x7fb0cc100ed0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc009900 tx=0x7fb0bc02a410 comp rx=0 tx=0).stop 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.041+0000 7fb0d26fc640 1 -- 172.21.15.120:0/615362381 shutdown_connections 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.041+0000 7fb0d26fc640 1 --2- 172.21.15.120:0/615362381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0cc102690 0x7fb0cc108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.041+0000 7fb0d26fc640 1 --2- 172.21.15.120:0/615362381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0cc101cd0 0x7fb0cc102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.041+0000 7fb0d26fc640 1 --2- 172.21.15.120:0/615362381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0cc100ad0 0x7fb0cc100ed0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.041+0000 7fb0d26fc640 1 -- 172.21.15.120:0/615362381 >> 172.21.15.120:0/615362381 conn(0x7fb0cc0fc280 msgr2=0x7fb0cc0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.041+0000 7fb0d26fc640 1 -- 172.21.15.120:0/615362381 shutdown_connections 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.041+0000 7fb0d26fc640 1 -- 172.21.15.120:0/615362381 wait complete. 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.041+0000 7fb0d26fc640 1 Processor -- start 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d26fc640 1 -- start start 2024-09-15T23:18:55.049 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d26fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0cc100ad0 0x7fb0cc196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:55.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d26fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0cc101cd0 0x7fb0cc197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:55.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d26fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0cc102690 0x7fb0cc19e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:55.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d26fc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc10c760 con 0x7fb0cc100ad0 2024-09-15T23:18:55.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d26fc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc10c5e0 con 0x7fb0cc102690 2024-09-15T23:18:55.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d26fc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc10c8e0 con 0x7fb0cc101cd0 2024-09-15T23:18:55.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d0c99640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0cc102690 0x7fb0cc19e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:55.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d0c99640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0cc102690 0x7fb0cc19e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:34064/0 (socket says 172.21.15.120:34064) 2024-09-15T23:18:55.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d0c99640 1 -- 172.21.15.120:0/1432823114 learned_addr learned my addr 172.21.15.120:0/1432823114 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-09-15T23:18:55.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0c3fff640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0cc101cd0 0x7fb0cc197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:55.050 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0cbfff640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0cc100ad0 0x7fb0cc196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d0c99640 1 -- 172.21.15.120:0/1432823114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0cc101cd0 msgr2=0x7fb0cc197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d0c99640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0cc101cd0 0x7fb0cc197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d0c99640 1 -- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0cc100ad0 msgr2=0x7fb0cc196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d0c99640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0cc100ad0 0x7fb0cc196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d0c99640 1 -- 172.21.15.120:0/1432823114 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0cc19ea30 con 0x7fb0cc102690 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0cbfff640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0cc100ad0 0x7fb0cc196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0d0c99640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0cc102690 0x7fb0cc19e3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400ebb0 tx=0x7fb0c4007590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0c9ffb640 1 -- 172.21.15.120:0/1432823114 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c4007ca0 con 0x7fb0cc102690 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.042+0000 7fb0c9ffb640 1 -- 172.21.15.120:0/1432823114 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0c4007e40 con 0x7fb0cc102690 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.043+0000 7fb0c9ffb640 1 -- 172.21.15.120:0/1432823114 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c40058a0 con 0x7fb0cc102690 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.043+0000 7fb0d26fc640 1 -- 172.21.15.120:0/1432823114 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0cc069f70 con 0x7fb0cc102690 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.043+0000 7fb0d26fc640 1 -- 172.21.15.120:0/1432823114 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0cc06a500 con 0x7fb0cc102690 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.044+0000 7fb0d26fc640 1 -- 172.21.15.120:0/1432823114 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0cc100ed0 con 0x7fb0cc102690 2024-09-15T23:18:55.051 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.045+0000 7fb0c9ffb640 1 -- 172.21.15.120:0/1432823114 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fb0c4004a40 con 0x7fb0cc102690 2024-09-15T23:18:55.052 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.045+0000 7fb0c9ffb640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0b80a32f0 0x7fb0b80a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:55.052 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.046+0000 7fb0c9ffb640 1 -- 172.21.15.120:0/1432823114 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4965+0+0 (secure 0 0 0) 0x7fb0c40d00b0 con 0x7fb0cc102690 2024-09-15T23:18:55.052 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.046+0000 7fb0cbfff640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0b80a32f0 0x7fb0b80a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:55.052 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.046+0000 7fb0cbfff640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0b80a32f0 0x7fb0b80a57b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc000c00 tx=0x7fb0bc031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:55.052 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.050+0000 7fb0c9ffb640 1 -- 172.21.15.120:0/1432823114 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb0c409dae0 con 0x7fb0cc102690 2024-09-15T23:18:55.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: osdmap e41: 7 total, 6 up, 7 in 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:55.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:55.250 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:18:55.249+0000 7fb0d26fc640 1 -- 172.21.15.120:0/1432823114 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb0cc06a760 con 0x7fb0b80a32f0 2024-09-15T23:18:55.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: osdmap e41: 7 total, 6 up, 7 in 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:55.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: osdmap e41: 7 total, 6 up, 7 in 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:55.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:55.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:18:55.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:18:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:18:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi089 2024-09-15T23:18:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi089 to 5695M 2024-09-15T23:18:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] boot 2024-09-15T23:18:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: osdmap e42: 7 total, 7 up, 7 in 2024-09-15T23:18:56.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:56.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:18:56.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:18:56.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:18:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:18:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi089 2024-09-15T23:18:56.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi089 to 5695M 2024-09-15T23:18:56.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:56.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] boot 2024-09-15T23:18:56.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: osdmap e42: 7 total, 7 up, 7 in 2024-09-15T23:18:56.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:56.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:18:56.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:18:56.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:18:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:18:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi089 2024-09-15T23:18:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi089 to 5695M 2024-09-15T23:18:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:18:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: osd.6 [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] boot 2024-09-15T23:18:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: osdmap e42: 7 total, 7 up, 7 in 2024-09-15T23:18:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:18:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:18:56.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:18:56.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:57.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:57 smithi089 ceph-mon[30933]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:18:57.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:57 smithi089 ceph-mon[30933]: osdmap e43: 7 total, 7 up, 7 in 2024-09-15T23:18:57.374 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:57 smithi120 ceph-mon[35250]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:18:57.374 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:57 smithi120 ceph-mon[35250]: osdmap e43: 7 total, 7 up, 7 in 2024-09-15T23:18:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:57 smithi174 ceph-mon[35586]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:18:57.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:57 smithi174 ceph-mon[35586]: osdmap e43: 7 total, 7 up, 7 in 2024-09-15T23:18:58.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:58 smithi120 ceph-mon[35250]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:18:58.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:58 smithi120 ceph-mon[35250]: from='client.? 172.21.15.120:0/3571886156' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696fd052-0624-47fc-bdf6-bcfc6f69ff74"}]: dispatch 2024-09-15T23:18:58.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:58 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696fd052-0624-47fc-bdf6-bcfc6f69ff74"}]: dispatch 2024-09-15T23:18:58.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:58 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "696fd052-0624-47fc-bdf6-bcfc6f69ff74"}]': finished 2024-09-15T23:18:58.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:58 smithi120 ceph-mon[35250]: osdmap e44: 8 total, 7 up, 8 in 2024-09-15T23:18:58.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:18:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:58 smithi089 ceph-mon[30933]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:18:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:58 smithi089 ceph-mon[30933]: from='client.? 172.21.15.120:0/3571886156' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696fd052-0624-47fc-bdf6-bcfc6f69ff74"}]: dispatch 2024-09-15T23:18:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:58 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696fd052-0624-47fc-bdf6-bcfc6f69ff74"}]: dispatch 2024-09-15T23:18:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:58 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "696fd052-0624-47fc-bdf6-bcfc6f69ff74"}]': finished 2024-09-15T23:18:58.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:58 smithi089 ceph-mon[30933]: osdmap e44: 8 total, 7 up, 8 in 2024-09-15T23:18:58.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:18:58.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:58 smithi174 ceph-mon[35586]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:18:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:58 smithi174 ceph-mon[35586]: from='client.? 172.21.15.120:0/3571886156' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696fd052-0624-47fc-bdf6-bcfc6f69ff74"}]: dispatch 2024-09-15T23:18:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:58 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "696fd052-0624-47fc-bdf6-bcfc6f69ff74"}]: dispatch 2024-09-15T23:18:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:58 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "696fd052-0624-47fc-bdf6-bcfc6f69ff74"}]': finished 2024-09-15T23:18:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:58 smithi174 ceph-mon[35586]: osdmap e44: 8 total, 7 up, 8 in 2024-09-15T23:18:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:18:59.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:18:59 smithi089 ceph-mon[30933]: from='client.? 172.21.15.120:0/1240650583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:18:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:18:59 smithi120 ceph-mon[35250]: from='client.? 172.21.15.120:0/1240650583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:18:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:18:59 smithi174 ceph-mon[35586]: from='client.? 172.21.15.120:0/1240650583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:19:00.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:00 smithi089 ceph-mon[30933]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:00 smithi120 ceph-mon[35250]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:00 smithi174 ceph-mon[35586]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:02 smithi089 ceph-mon[30933]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:02.351 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:02 smithi120 ceph-mon[35250]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:02.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:02 smithi174 ceph-mon[35586]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:04.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:04 smithi089 ceph-mon[30933]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:04.440 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:04 smithi120 ceph-mon[35250]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:04.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:04 smithi174 ceph-mon[35586]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:06.307 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:06 smithi120 ceph-mon[35250]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:06.308 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-15T23:19:06.308 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:06 smithi089 ceph-mon[30933]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-15T23:19:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:06 smithi174 ceph-mon[35586]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:06.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-15T23:19:06.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:07.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:07 smithi089 ceph-mon[30933]: Deploying daemon osd.7 on smithi120 2024-09-15T23:19:07.384 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:07 smithi120 ceph-mon[35250]: Deploying daemon osd.7 on smithi120 2024-09-15T23:19:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:07 smithi174 ceph-mon[35586]: Deploying daemon osd.7 on smithi120 2024-09-15T23:19:08.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:08 smithi089 ceph-mon[30933]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:08.448 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:08 smithi120 ceph-mon[35250]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:08.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:08 smithi174 ceph-mon[35586]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:10.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:10 smithi089 ceph-mon[30933]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:10.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:10 smithi120 ceph-mon[35250]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:10 smithi174 ceph-mon[35586]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:11.464 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:11 smithi120 ceph-mon[35250]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:11 smithi174 ceph-mon[35586]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:11.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:11 smithi089 ceph-mon[30933]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:12.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:12 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:12.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:12 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:12.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:12 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:12.431 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:12 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:12.431 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:12 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:12.432 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:12 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:12.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:12 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:12.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:12 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:12.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:12 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:13.321 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:13 smithi120 ceph-mon[35250]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:13 smithi174 ceph-mon[35586]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:13.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:13 smithi089 ceph-mon[30933]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:15.777 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:15.777 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:15.777 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:15.778 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:15.778 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:15.778 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:15 smithi120 ceph-mon[35250]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:15.778 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.776+0000 7fb0c9ffb640 1 -- 172.21.15.120:0/1432823114 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb0cc06a760 con 0x7fb0b80a32f0 2024-09-15T23:19:15.778 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 7 on host 'smithi120' 2024-09-15T23:19:15.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 -- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0b80a32f0 msgr2=0x7fb0b80a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0b80a32f0 0x7fb0b80a57b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc000c00 tx=0x7fb0bc031400 comp rx=0 tx=0).stop 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 -- 172.21.15.120:0/1432823114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0cc102690 msgr2=0x7fb0cc19e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0cc102690 0x7fb0cc19e3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0c400ebb0 tx=0x7fb0c4007590 comp rx=0 tx=0).stop 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 -- 172.21.15.120:0/1432823114 shutdown_connections 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0b80a32f0 0x7fb0b80a57b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0cc102690 0x7fb0cc19e3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0cc101cd0 0x7fb0cc197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 --2- 172.21.15.120:0/1432823114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0cc100ad0 0x7fb0cc196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 -- 172.21.15.120:0/1432823114 >> 172.21.15.120:0/1432823114 conn(0x7fb0cc0fc280 msgr2=0x7fb0cc1050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 -- 172.21.15.120:0/1432823114 shutdown_connections 2024-09-15T23:19:15.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:19:15.779+0000 7fb0d26fc640 1 -- 172.21.15.120:0/1432823114 wait complete. 2024-09-15T23:19:15.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:15.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:15.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:15.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:15.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:15.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:15 smithi174 ceph-mon[35586]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:16.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:16.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:16.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:16.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:16.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:16.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:15 smithi089 ceph-mon[30933]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:16.397 DEBUG:teuthology.orchestra.run.smithi120:osd.7> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.7.service 2024-09-15T23:19:16.400 INFO:tasks.cephadm:Deploying osd.8 on smithi174 with /dev/vg_nvme/lv_2... 2024-09-15T23:19:16.400 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-15T23:19:16.654 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:19:16.961 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:16 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:16.961 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:16 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:16.962 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:16 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:16.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:16 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:16.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:16 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:16.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:16 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:17.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:16 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:17.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:16 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:17.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:16 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:17.940 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:17 smithi120 ceph-mon[35250]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:17.940 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:19:17 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:19:17.702+0000 7f72e6a3f640 -1 osd.7 0 log_to_monitors true 2024-09-15T23:19:17.977 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:17 smithi174 ceph-mon[35586]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:18.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:17 smithi089 ceph-mon[30933]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:18.954 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:18 smithi174 ceph-mon[35586]: from='osd.7 [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:19:18.955 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:18 smithi174 ceph-mon[35586]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:19:18.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:18 smithi120 ceph-mon[35250]: from='osd.7 [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:19:18.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:18 smithi120 ceph-mon[35250]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:19:19.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:18 smithi089 ceph-mon[30933]: from='osd.7 [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:19:19.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:18 smithi089 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:19:19.283 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:19:19.309 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_2 2024-09-15T23:19:19.563 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:19:19.846 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: osdmap e45: 8 total, 7 up, 8 in 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='osd.7 [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:19.847 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:19.848 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:19.848 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-15T23:19:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: osdmap e45: 8 total, 7 up, 8 in 2024-09-15T23:19:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='osd.7 [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:19:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:19:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:19.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:20.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-15T23:19:20.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: osdmap e45: 8 total, 7 up, 8 in 2024-09-15T23:19:20.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:20.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='osd.7 [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:20.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.051+0000 7f24207bb640 1 -- 172.21.15.174:0/3641771008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2418107920 msgr2=0x7f2418109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.051+0000 7f24207bb640 1 --2- 172.21.15.174:0/3641771008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2418107920 0x7f2418109d10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2410009ca0 tx=0x7f241002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.051+0000 7f24207bb640 1 -- 172.21.15.174:0/3641771008 shutdown_connections 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.051+0000 7f24207bb640 1 --2- 172.21.15.174:0/3641771008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2418107920 0x7f2418109d10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.051+0000 7f24207bb640 1 --2- 172.21.15.174:0/3641771008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2418104ff0 0x7f24181073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.051+0000 7f24207bb640 1 --2- 172.21.15.174:0/3641771008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24181026c0 0x7f2418104ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.051+0000 7f24207bb640 1 -- 172.21.15.174:0/3641771008 >> 172.21.15.174:0/3641771008 conn(0x7f24180fc280 msgr2=0x7f24180fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.051+0000 7f24207bb640 1 -- 172.21.15.174:0/3641771008 shutdown_connections 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.051+0000 7f24207bb640 1 -- 172.21.15.174:0/3641771008 wait complete. 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f24207bb640 1 Processor -- start 2024-09-15T23:19:20.076 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f24207bb640 1 -- start start 2024-09-15T23:19:20.077 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f24207bb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24181026c0 0x7f2418102020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:20.077 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f24207bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2418104ff0 0x7f2418100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:20.077 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f24207bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2418107920 0x7f2418100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:20.079 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f24207bb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f241810d860 con 0x7f2418107920 2024-09-15T23:19:20.079 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f24207bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f241810d6e0 con 0x7f2418104ff0 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f24207bb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f241810d9e0 con 0x7f24181026c0 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f241e557640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24181026c0 0x7f2418102020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f241e557640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24181026c0 0x7f2418102020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:46386/0 (socket says 172.21.15.174:46386) 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f241e557640 1 -- 172.21.15.174:0/1178350873 learned_addr learned my addr 172.21.15.174:0/1178350873 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.052+0000 7f241dd56640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2418104ff0 0x7f2418100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f241ed58640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2418107920 0x7f2418100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f241e557640 1 -- 172.21.15.174:0/1178350873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2418104ff0 msgr2=0x7f2418100670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f241e557640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2418104ff0 0x7f2418100670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f241e557640 1 -- 172.21.15.174:0/1178350873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2418107920 msgr2=0x7f2418100bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f241e557640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2418107920 0x7f2418100bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f241e557640 1 -- 172.21.15.174:0/1178350873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2418101470 con 0x7f24181026c0 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f241e557640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24181026c0 0x7f2418102020 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f241400c920 tx=0x7f241400cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:20.080 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f240f7fe640 1 -- 172.21.15.174:0/1178350873 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414007c70 con 0x7f24181026c0 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f240f7fe640 1 -- 172.21.15.174:0/1178350873 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2414007e10 con 0x7f24181026c0 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f240f7fe640 1 -- 172.21.15.174:0/1178350873 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414018890 con 0x7f24181026c0 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f24207bb640 1 -- 172.21.15.174:0/1178350873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24181a1fc0 con 0x7f24181026c0 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.053+0000 7f24207bb640 1 -- 172.21.15.174:0/1178350873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24181a2410 con 0x7f24181026c0 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.054+0000 7f24207bb640 1 -- 172.21.15.174:0/1178350873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24180fdd70 con 0x7f24181026c0 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.055+0000 7f240f7fe640 1 -- 172.21.15.174:0/1178350873 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2414018a50 con 0x7f24181026c0 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.056+0000 7f240f7fe640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24040a32c0 0x7f24040a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.056+0000 7f240f7fe640 1 -- 172.21.15.174:0/1178350873 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5117+0+0 (secure 0 0 0) 0x7f24140cf000 con 0x7f24181026c0 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.056+0000 7f241dd56640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24040a32c0 0x7f24040a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.057+0000 7f241dd56640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24040a32c0 0x7f24040a5780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2418101da0 tx=0x7f240800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:20.081 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.063+0000 7f240f7fe640 1 -- 172.21.15.174:0/1178350873 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2414014030 con 0x7f24181026c0 2024-09-15T23:19:20.243 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:20.242+0000 7f24207bb640 1 -- 172.21.15.174:0/1178350873 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f2418062630 con 0x7f24040a32c0 2024-09-15T23:19:20.473 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:19:20.093+0000 7f72e29c2640 -1 osd.7 0 waiting for initial osdmap 2024-09-15T23:19:20.473 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:19:20.100+0000 7f72da7e2640 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:19:20.929 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:19:20.929 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:19:20.929 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi120 2024-09-15T23:19:20.929 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi120 to 5696M 2024-09-15T23:19:20.929 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:19:20.929 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: osdmap e46: 8 total, 7 up, 8 in 2024-09-15T23:19:20.929 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:20.930 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:20.930 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: from='osd.7 ' entity='osd.7' 2024-09-15T23:19:20.930 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:19:20.930 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:19:20.930 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:20.930 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:20 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:20.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:19:20.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:19:20.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi120 2024-09-15T23:19:20.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi120 to 5696M 2024-09-15T23:19:20.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:19:20.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: osdmap e46: 8 total, 7 up, 8 in 2024-09-15T23:19:20.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:20.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:20.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: from='osd.7 ' entity='osd.7' 2024-09-15T23:19:20.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:19:20.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:19:20.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:20.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:20 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:21.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:19:21.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:19:21.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi120 2024-09-15T23:19:21.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi120 to 5696M 2024-09-15T23:19:21.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:19:21.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: osdmap e46: 8 total, 7 up, 8 in 2024-09-15T23:19:21.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:21.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:21.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: from='osd.7 ' entity='osd.7' 2024-09-15T23:19:21.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:19:21.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:19:21.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:21.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:20 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:21.941 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:21 smithi174 ceph-mon[35586]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:19:21.941 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:21 smithi174 ceph-mon[35586]: osd.7 [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] boot 2024-09-15T23:19:21.941 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:21 smithi174 ceph-mon[35586]: osdmap e47: 8 total, 8 up, 8 in 2024-09-15T23:19:21.941 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:21 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:21.941 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:21 smithi174 ceph-mon[35586]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:22.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:21 smithi089 ceph-mon[30933]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:19:22.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:21 smithi089 ceph-mon[30933]: osd.7 [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] boot 2024-09-15T23:19:22.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:21 smithi089 ceph-mon[30933]: osdmap e47: 8 total, 8 up, 8 in 2024-09-15T23:19:22.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:21 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:22.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:21 smithi089 ceph-mon[30933]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:21 smithi120 ceph-mon[35250]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:19:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:21 smithi120 ceph-mon[35250]: osd.7 [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] boot 2024-09-15T23:19:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:21 smithi120 ceph-mon[35250]: osdmap e47: 8 total, 8 up, 8 in 2024-09-15T23:19:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:21 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:19:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:21 smithi120 ceph-mon[35250]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:19:22.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:22 smithi174 ceph-mon[35586]: osdmap e48: 8 total, 8 up, 8 in 2024-09-15T23:19:23.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:22 smithi089 ceph-mon[30933]: osdmap e48: 8 total, 8 up, 8 in 2024-09-15T23:19:23.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:22 smithi120 ceph-mon[35250]: osdmap e48: 8 total, 8 up, 8 in 2024-09-15T23:19:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:23 smithi174 ceph-mon[35586]: osdmap e49: 8 total, 8 up, 8 in 2024-09-15T23:19:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:23 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/554542689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f51541d-5288-4ed5-9780-516f0f26744b"}]: dispatch 2024-09-15T23:19:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:23 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f51541d-5288-4ed5-9780-516f0f26744b"}]: dispatch 2024-09-15T23:19:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:23 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f51541d-5288-4ed5-9780-516f0f26744b"}]': finished 2024-09-15T23:19:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:23 smithi174 ceph-mon[35586]: osdmap e50: 9 total, 8 up, 9 in 2024-09-15T23:19:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:23 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:23 smithi174 ceph-mon[35586]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:23.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:23 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/1225257936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:19:24.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:23 smithi089 ceph-mon[30933]: osdmap e49: 8 total, 8 up, 8 in 2024-09-15T23:19:24.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:23 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/554542689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f51541d-5288-4ed5-9780-516f0f26744b"}]: dispatch 2024-09-15T23:19:24.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:23 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f51541d-5288-4ed5-9780-516f0f26744b"}]: dispatch 2024-09-15T23:19:24.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:23 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f51541d-5288-4ed5-9780-516f0f26744b"}]': finished 2024-09-15T23:19:24.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:23 smithi089 ceph-mon[30933]: osdmap e50: 9 total, 8 up, 9 in 2024-09-15T23:19:24.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:23 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:24.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:23 smithi089 ceph-mon[30933]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:24.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:23 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/1225257936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:19:24.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:23 smithi120 ceph-mon[35250]: osdmap e49: 8 total, 8 up, 8 in 2024-09-15T23:19:24.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:23 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/554542689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f51541d-5288-4ed5-9780-516f0f26744b"}]: dispatch 2024-09-15T23:19:24.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:23 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f51541d-5288-4ed5-9780-516f0f26744b"}]: dispatch 2024-09-15T23:19:24.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:23 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f51541d-5288-4ed5-9780-516f0f26744b"}]': finished 2024-09-15T23:19:24.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:23 smithi120 ceph-mon[35250]: osdmap e50: 9 total, 8 up, 9 in 2024-09-15T23:19:24.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:23 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:24.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:23 smithi120 ceph-mon[35250]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:24.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:23 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/1225257936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:19:25.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:24 smithi120 ceph-mon[35250]: osdmap e51: 9 total, 8 up, 9 in 2024-09-15T23:19:25.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:25.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:24 smithi174 ceph-mon[35586]: osdmap e51: 9 total, 8 up, 9 in 2024-09-15T23:19:25.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:24 smithi089 ceph-mon[30933]: osdmap e51: 9 total, 8 up, 9 in 2024-09-15T23:19:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:26.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:25 smithi120 ceph-mon[35250]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:26.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:25 smithi174 ceph-mon[35586]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:25 smithi089 ceph-mon[30933]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:28.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:28 smithi089 ceph-mon[30933]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:28.343 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:28 smithi174 ceph-mon[35586]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:28.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:28 smithi120 ceph-mon[35250]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:30.151 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:30 smithi174 ceph-mon[35586]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:30.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:30 smithi089 ceph-mon[30933]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:30 smithi120 ceph-mon[35250]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:32.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:32 smithi174 ceph-mon[35586]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:32.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:32 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-15T23:19:32.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:32 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:32 smithi089 ceph-mon[30933]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:32 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-15T23:19:32.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:32 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:32 smithi120 ceph-mon[35250]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:32 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-15T23:19:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:32 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:33.182 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:33 smithi174 ceph-mon[35586]: Deploying daemon osd.8 on smithi174 2024-09-15T23:19:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:33 smithi089 ceph-mon[30933]: Deploying daemon osd.8 on smithi174 2024-09-15T23:19:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:33 smithi120 ceph-mon[35250]: Deploying daemon osd.8 on smithi174 2024-09-15T23:19:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:34 smithi089 ceph-mon[30933]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:34.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:34 smithi174 ceph-mon[35586]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:34 smithi120 ceph-mon[35250]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:36.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:36 smithi089 ceph-mon[30933]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:36.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:36 smithi174 ceph-mon[35586]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:36 smithi120 ceph-mon[35250]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:38.022 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:38 smithi174 ceph-mon[35586]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:38.022 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:38.022 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:38.022 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:38 smithi089 ceph-mon[30933]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:38 smithi120 ceph-mon[35250]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:38.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:38.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:38.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:40.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:40 smithi174 ceph-mon[35586]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:40.275 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:40 smithi120 ceph-mon[35250]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:40.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:40 smithi089 ceph-mon[30933]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:41.036 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 8 on host 'smithi174' 2024-09-15T23:19:41.037 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.035+0000 7f240f7fe640 1 -- 172.21.15.174:0/1178350873 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2418062630 con 0x7f24040a32c0 2024-09-15T23:19:41.038 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 -- 172.21.15.174:0/1178350873 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24040a32c0 msgr2=0x7f24040a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24040a32c0 0x7f24040a5780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2418101da0 tx=0x7f240800b410 comp rx=0 tx=0).stop 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 -- 172.21.15.174:0/1178350873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24181026c0 msgr2=0x7f2418102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24181026c0 0x7f2418102020 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f241400c920 tx=0x7f241400cde0 comp rx=0 tx=0).stop 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 -- 172.21.15.174:0/1178350873 shutdown_connections 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24040a32c0 0x7f24040a5780 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2418107920 0x7f2418100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2418104ff0 0x7f2418100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 --2- 172.21.15.174:0/1178350873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24181026c0 0x7f2418102020 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 -- 172.21.15.174:0/1178350873 >> 172.21.15.174:0/1178350873 conn(0x7f24180fc280 msgr2=0x7f2418102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.037+0000 7f24207bb640 1 -- 172.21.15.174:0/1178350873 shutdown_connections 2024-09-15T23:19:41.039 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:19:41.038+0000 7f24207bb640 1 -- 172.21.15.174:0/1178350873 wait complete. 2024-09-15T23:19:41.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:41.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:41.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:41 smithi174 ceph-mon[35586]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:41.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:41.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:41 smithi120 ceph-mon[35250]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:41.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:41.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:41 smithi089 ceph-mon[30933]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:41.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:41.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:41.743 DEBUG:teuthology.orchestra.run.smithi174:osd.8> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.8.service 2024-09-15T23:19:41.745 INFO:tasks.cephadm:Deploying osd.9 on smithi089 with /dev/vg_nvme/lv_1... 2024-09-15T23:19:41.745 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-15T23:19:42.004 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:43.236 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:19:43 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:19:43.195+0000 7f5f51f8a640 -1 osd.8 0 log_to_monitors true 2024-09-15T23:19:44.264 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:44 smithi089 ceph-mon[30933]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:44.264 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:44 smithi089 ceph-mon[30933]: from='osd.8 [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:19:44.264 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:44 smithi089 ceph-mon[30933]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:19:44.313 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:44 smithi174 ceph-mon[35586]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:44.313 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:44 smithi174 ceph-mon[35586]: from='osd.8 [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:19:44.313 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:44 smithi174 ceph-mon[35586]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:19:44.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:44 smithi120 ceph-mon[35250]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:44.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:44 smithi120 ceph-mon[35250]: from='osd.8 [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:19:44.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:44 smithi120 ceph-mon[35250]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:19:45.026 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:19:45.051 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi089:vg_nvme/lv_1 2024-09-15T23:19:45.276 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:19:45.020+0000 7f5f4df0d640 -1 osd.8 0 waiting for initial osdmap 2024-09-15T23:19:45.276 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:19:45.028+0000 7f5f49d35640 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:19:45.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-15T23:19:45.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: osdmap e52: 9 total, 8 up, 9 in 2024-09-15T23:19:45.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:45.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='osd.8 [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:19:45.276 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:19:45.277 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.277 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.277 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:45.277 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:45.277 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:45.277 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.277 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:45.277 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:45.277 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.301 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:45.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-15T23:19:45.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: osdmap e52: 9 total, 8 up, 9 in 2024-09-15T23:19:45.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:45.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='osd.8 [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:19:45.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:19:45.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:45.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:45.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:45.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:45.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:45.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-15T23:19:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: osdmap e52: 9 total, 8 up, 9 in 2024-09-15T23:19:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='osd.8 [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:45.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:19:45.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.767+0000 7fc565f69640 1 -- 172.21.15.89:0/3836537746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc560104f80 msgr2=0x7fc560105400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:45.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.767+0000 7fc565f69640 1 --2- 172.21.15.89:0/3836537746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc560104f80 0x7fc560105400 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc554009900 tx=0x7fc55402a410 comp rx=0 tx=0).stop 2024-09-15T23:19:45.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.768+0000 7fc565f69640 1 -- 172.21.15.89:0/3836537746 shutdown_connections 2024-09-15T23:19:45.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.768+0000 7fc565f69640 1 --2- 172.21.15.89:0/3836537746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560105940 0x7fc56010c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:45.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.768+0000 7fc565f69640 1 --2- 172.21.15.89:0/3836537746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc560104f80 0x7fc560105400 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.768+0000 7fc565f69640 1 --2- 172.21.15.89:0/3836537746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc560103e20 0x7fc560104220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.768+0000 7fc565f69640 1 -- 172.21.15.89:0/3836537746 >> 172.21.15.89:0/3836537746 conn(0x7fc5600ff5b0 msgr2=0x7fc5601019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.768+0000 7fc565f69640 1 -- 172.21.15.89:0/3836537746 shutdown_connections 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.768+0000 7fc565f69640 1 -- 172.21.15.89:0/3836537746 wait complete. 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.768+0000 7fc565f69640 1 Processor -- start 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.768+0000 7fc565f69640 1 -- start start 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc565f69640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560103e20 0x7fc56019a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc565f69640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc560104f80 0x7fc56019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc565f69640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc560105940 0x7fc5601a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc565f69640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010fb20 con 0x7fc560105940 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc565f69640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010f9a0 con 0x7fc560103e20 2024-09-15T23:19:45.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc565f69640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010fca0 con 0x7fc560104f80 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc560105940 0x7fc5601a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc560105940 0x7fc5601a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48564/0 (socket says 172.21.15.89:48564) 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55ffff640 1 -- 172.21.15.89:0/742292156 learned_addr learned my addr 172.21.15.89:0/742292156 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55effd640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc560104f80 0x7fc56019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55f7fe640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560103e20 0x7fc56019a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55ffff640 1 -- 172.21.15.89:0/742292156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc560104f80 msgr2=0x7fc56019a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55ffff640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc560104f80 0x7fc56019a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55ffff640 1 -- 172.21.15.89:0/742292156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560103e20 msgr2=0x7fc56019a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55ffff640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560103e20 0x7fc56019a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55ffff640 1 -- 172.21.15.89:0/742292156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5601a1d70 con 0x7fc560105940 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55ffff640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc560105940 0x7fc5601a1700 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc558009fd0 tx=0x7fc55800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55cff9640 1 -- 172.21.15.89:0/742292156 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558017070 con 0x7fc560105940 2024-09-15T23:19:45.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc565f69640 1 -- 172.21.15.89:0/742292156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5601a2060 con 0x7fc560105940 2024-09-15T23:19:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55cff9640 1 -- 172.21.15.89:0/742292156 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc558003e50 con 0x7fc560105940 2024-09-15T23:19:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc55cff9640 1 -- 172.21.15.89:0/742292156 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5580106c0 con 0x7fc560105940 2024-09-15T23:19:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.769+0000 7fc565f69640 1 -- 172.21.15.89:0/742292156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5601a2520 con 0x7fc560105940 2024-09-15T23:19:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.770+0000 7fc565f69640 1 -- 172.21.15.89:0/742292156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc56010b260 con 0x7fc560105940 2024-09-15T23:19:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.772+0000 7fc55cff9640 1 -- 172.21.15.89:0/742292156 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fc558003970 con 0x7fc560105940 2024-09-15T23:19:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.772+0000 7fc55cff9640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5480a3270 0x7fc5480a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.772+0000 7fc55f7fe640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5480a3270 0x7fc5480a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.772+0000 7fc55cff9640 1 -- 172.21.15.89:0/742292156 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5549+0+0 (secure 0 0 0) 0x7fc5580ceaf0 con 0x7fc560105940 2024-09-15T23:19:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.773+0000 7fc55f7fe640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5480a3270 0x7fc5480a5730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc550009bd0 tx=0x7fc550005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:45.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.777+0000 7fc55cff9640 1 -- 172.21.15.89:0/742292156 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc55809bc90 con 0x7fc560105940 2024-09-15T23:19:45.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:19:45.950+0000 7fc565f69640 1 -- 172.21.15.89:0/742292156 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc560062630 con 0x7fc5480a3270 2024-09-15T23:19:46.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:46 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi174 2024-09-15T23:19:46.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:46 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi174 to 5696M 2024-09-15T23:19:46.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:46 smithi089 ceph-mon[30933]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:46.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:46 smithi089 ceph-mon[30933]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:19:46.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:46 smithi089 ceph-mon[30933]: osdmap e53: 9 total, 8 up, 9 in 2024-09-15T23:19:46.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:46 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:46.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:46 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:46.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:46 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:19:46.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:46 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:19:46.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:46 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:46 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi174 2024-09-15T23:19:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:46 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi174 to 5696M 2024-09-15T23:19:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:46 smithi120 ceph-mon[35250]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:46 smithi120 ceph-mon[35250]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:19:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:46 smithi120 ceph-mon[35250]: osdmap e53: 9 total, 8 up, 9 in 2024-09-15T23:19:46.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:46 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:46.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:46 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:46.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:46 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:19:46.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:46 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:19:46.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:46 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:46 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi174 2024-09-15T23:19:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:46 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi174 to 5696M 2024-09-15T23:19:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:46 smithi174 ceph-mon[35586]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:46 smithi174 ceph-mon[35586]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:19:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:46 smithi174 ceph-mon[35586]: osdmap e53: 9 total, 8 up, 9 in 2024-09-15T23:19:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:46 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:46 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:46 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:19:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:46 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:19:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:46 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:47.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:47 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:19:47.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:47 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:19:47.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:47 smithi089 ceph-mon[30933]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:19:47.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:47 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:47.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:47 smithi089 ceph-mon[30933]: osd.8 [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] boot 2024-09-15T23:19:47.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:47 smithi089 ceph-mon[30933]: osdmap e54: 9 total, 9 up, 9 in 2024-09-15T23:19:47.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:47 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:47 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:19:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:47 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:19:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:47 smithi120 ceph-mon[35250]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:19:47.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:47 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:47.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:47 smithi120 ceph-mon[35250]: osd.8 [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] boot 2024-09-15T23:19:47.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:47 smithi120 ceph-mon[35250]: osdmap e54: 9 total, 9 up, 9 in 2024-09-15T23:19:47.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:47 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:47.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:47 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:19:47.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:47 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:19:47.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:47 smithi174 ceph-mon[35586]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:19:47.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:47 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:47.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:47 smithi174 ceph-mon[35586]: osd.8 [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] boot 2024-09-15T23:19:47.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:47 smithi174 ceph-mon[35586]: osdmap e54: 9 total, 9 up, 9 in 2024-09-15T23:19:47.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:47 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:19:48.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:48 smithi089 ceph-mon[30933]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:48.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:48 smithi089 ceph-mon[30933]: osdmap e55: 9 total, 9 up, 9 in 2024-09-15T23:19:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:48 smithi120 ceph-mon[35250]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:48 smithi120 ceph-mon[35250]: osdmap e55: 9 total, 9 up, 9 in 2024-09-15T23:19:48.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:48 smithi174 ceph-mon[35586]: pgmap v187: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:19:48.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:48 smithi174 ceph-mon[35586]: osdmap e55: 9 total, 9 up, 9 in 2024-09-15T23:19:49.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:49 smithi089 ceph-mon[30933]: osdmap e56: 9 total, 9 up, 9 in 2024-09-15T23:19:49.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:49 smithi089 ceph-mon[30933]: osdmap e57: 9 total, 9 up, 9 in 2024-09-15T23:19:49.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:49 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/501126540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81930b2a-3a54-4442-9c89-316644a3adce"}]: dispatch 2024-09-15T23:19:49.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:49 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/501126540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81930b2a-3a54-4442-9c89-316644a3adce"}]': finished 2024-09-15T23:19:49.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:49 smithi089 ceph-mon[30933]: osdmap e58: 10 total, 9 up, 10 in 2024-09-15T23:19:49.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:19:49.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:49 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2178837252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:19:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:49 smithi120 ceph-mon[35250]: osdmap e56: 9 total, 9 up, 9 in 2024-09-15T23:19:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:49 smithi120 ceph-mon[35250]: osdmap e57: 9 total, 9 up, 9 in 2024-09-15T23:19:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:49 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/501126540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81930b2a-3a54-4442-9c89-316644a3adce"}]: dispatch 2024-09-15T23:19:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:49 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/501126540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81930b2a-3a54-4442-9c89-316644a3adce"}]': finished 2024-09-15T23:19:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:49 smithi120 ceph-mon[35250]: osdmap e58: 10 total, 9 up, 10 in 2024-09-15T23:19:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:19:49.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:49 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2178837252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:19:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:49 smithi174 ceph-mon[35586]: osdmap e56: 9 total, 9 up, 9 in 2024-09-15T23:19:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:49 smithi174 ceph-mon[35586]: osdmap e57: 9 total, 9 up, 9 in 2024-09-15T23:19:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:49 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/501126540' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81930b2a-3a54-4442-9c89-316644a3adce"}]: dispatch 2024-09-15T23:19:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:49 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/501126540' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81930b2a-3a54-4442-9c89-316644a3adce"}]': finished 2024-09-15T23:19:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:49 smithi174 ceph-mon[35586]: osdmap e58: 10 total, 9 up, 10 in 2024-09-15T23:19:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:19:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:49 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2178837252' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:19:50.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:50 smithi089 ceph-mon[30933]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:19:50.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:50 smithi120 ceph-mon[35250]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:19:50.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:50 smithi174 ceph-mon[35586]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:19:52.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:52 smithi089 ceph-mon[30933]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:19:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:52 smithi120 ceph-mon[35250]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:19:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:52 smithi174 ceph-mon[35586]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:19:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:54 smithi089 ceph-mon[30933]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-15T23:19:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:54 smithi120 ceph-mon[35250]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-15T23:19:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:54 smithi174 ceph-mon[35586]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-15T23:19:56.231 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:56 smithi089 ceph-mon[30933]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-15T23:19:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:56 smithi120 ceph-mon[35250]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-15T23:19:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:56 smithi174 ceph-mon[35586]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-15T23:19:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:58 smithi089 ceph-mon[30933]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-15T23:19:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-15T23:19:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:58 smithi120 ceph-mon[35250]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-15T23:19:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-15T23:19:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:58.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:58 smithi174 ceph-mon[35586]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-15T23:19:58.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-15T23:19:58.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:59.272 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:19:59 smithi089 ceph-mon[30933]: Deploying daemon osd.9 on smithi089 2024-09-15T23:19:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:19:59 smithi120 ceph-mon[35250]: Deploying daemon osd.9 on smithi089 2024-09-15T23:19:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:19:59 smithi174 ceph-mon[35586]: Deploying daemon osd.9 on smithi089 2024-09-15T23:20:00.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:00 smithi089 ceph-mon[30933]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-15T23:20:00.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:00 smithi089 ceph-mon[30933]: overall HEALTH_OK 2024-09-15T23:20:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:00 smithi120 ceph-mon[35250]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-15T23:20:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:00 smithi120 ceph-mon[35250]: overall HEALTH_OK 2024-09-15T23:20:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:00 smithi174 ceph-mon[35586]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-15T23:20:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:00 smithi174 ceph-mon[35586]: overall HEALTH_OK 2024-09-15T23:20:01.091 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:01 smithi089 ceph-mon[30933]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:20:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:01 smithi120 ceph-mon[35250]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:20:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:01 smithi174 ceph-mon[35586]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:20:04.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:04 smithi089 ceph-mon[30933]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:20:04.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:04.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:04.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:04 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:04 smithi120 ceph-mon[35250]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:20:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:04 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:04.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:04 smithi174 ceph-mon[35586]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:20:04.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:04.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:04.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:04 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:06.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:06 smithi089 ceph-mon[30933]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:06 smithi120 ceph-mon[35250]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:06 smithi174 ceph-mon[35586]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:06.861 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 9 on host 'smithi089' 2024-09-15T23:20:06.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.860+0000 7fc55cff9640 1 -- 172.21.15.89:0/742292156 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc560062630 con 0x7fc5480a3270 2024-09-15T23:20:06.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.862+0000 7fc565f69640 1 -- 172.21.15.89:0/742292156 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5480a3270 msgr2=0x7fc5480a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.862+0000 7fc565f69640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5480a3270 0x7fc5480a5730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc550009bd0 tx=0x7fc550005f90 comp rx=0 tx=0).stop 2024-09-15T23:20:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.862+0000 7fc565f69640 1 -- 172.21.15.89:0/742292156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc560105940 msgr2=0x7fc5601a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.862+0000 7fc565f69640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc560105940 0x7fc5601a1700 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc558009fd0 tx=0x7fc55800b260 comp rx=0 tx=0).stop 2024-09-15T23:20:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.863+0000 7fc565f69640 1 -- 172.21.15.89:0/742292156 shutdown_connections 2024-09-15T23:20:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.863+0000 7fc565f69640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5480a3270 0x7fc5480a5730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.863+0000 7fc565f69640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc560105940 0x7fc5601a1700 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.863+0000 7fc565f69640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc560104f80 0x7fc56019a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.863+0000 7fc565f69640 1 --2- 172.21.15.89:0/742292156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560103e20 0x7fc56019a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:06.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.863+0000 7fc565f69640 1 -- 172.21.15.89:0/742292156 >> 172.21.15.89:0/742292156 conn(0x7fc5600ff5b0 msgr2=0x7fc560108430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.863+0000 7fc565f69640 1 -- 172.21.15.89:0/742292156 shutdown_connections 2024-09-15T23:20:06.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:06.863+0000 7fc565f69640 1 -- 172.21.15.89:0/742292156 wait complete. 2024-09-15T23:20:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:07 smithi120 ceph-mon[35250]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:07 smithi174 ceph-mon[35586]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:07.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:07.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:07 smithi089 ceph-mon[30933]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:07.595 DEBUG:teuthology.orchestra.run.smithi089:osd.9> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.9.service 2024-09-15T23:20:07.597 INFO:tasks.cephadm:Deploying osd.10 on smithi120 with /dev/vg_nvme/lv_1... 2024-09-15T23:20:07.597 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-15T23:20:07.854 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:20:09.561 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:20:09 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:20:09.265+0000 7f79434c2640 -1 osd.9 0 log_to_monitors true 2024-09-15T23:20:10.159 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:10 smithi120 ceph-mon[35250]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:10.162 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:10 smithi120 ceph-mon[35250]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-15T23:20:10.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:10 smithi089 ceph-mon[30933]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:10.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:10 smithi089 ceph-mon[30933]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-15T23:20:10.427 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-09-15T23:20:10.456 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi120:vg_nvme/lv_1 2024-09-15T23:20:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:10 smithi174 ceph-mon[35586]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:10 smithi174 ceph-mon[35586]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-15T23:20:10.714 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:20:11.211 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.179+0000 7fd2be993640 1 -- 172.21.15.120:0/3724497747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2b81037c0 msgr2=0x7fd2b810a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:11.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.179+0000 7fd2be993640 1 --2- 172.21.15.120:0/3724497747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2b81037c0 0x7fd2b810a050 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0009ca0 tx=0x7fd2b002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.179+0000 7fd2be993640 1 -- 172.21.15.120:0/3724497747 shutdown_connections 2024-09-15T23:20:11.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.179+0000 7fd2be993640 1 --2- 172.21.15.120:0/3724497747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2b81037c0 0x7fd2b810a050 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.179+0000 7fd2be993640 1 --2- 172.21.15.120:0/3724497747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2b8102e00 0x7fd2b8103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.179+0000 7fd2be993640 1 --2- 172.21.15.120:0/3724497747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2b8101c00 0x7fd2b8102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.179+0000 7fd2be993640 1 -- 172.21.15.120:0/3724497747 >> 172.21.15.120:0/3724497747 conn(0x7fd2b80fd3b0 msgr2=0x7fd2b80ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:11.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.179+0000 7fd2be993640 1 -- 172.21.15.120:0/3724497747 shutdown_connections 2024-09-15T23:20:11.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.179+0000 7fd2be993640 1 -- 172.21.15.120:0/3724497747 wait complete. 2024-09-15T23:20:11.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2be993640 1 Processor -- start 2024-09-15T23:20:11.212 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2be993640 1 -- start start 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2be993640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2b8101c00 0x7fd2b8075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2be993640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2b8102e00 0x7fd2b80745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2be993640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2b81037c0 0x7fd2b8074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2be993640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b810d8d0 con 0x7fd2b8102e00 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2be993640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b810d750 con 0x7fd2b81037c0 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2be993640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b810da50 con 0x7fd2b8101c00 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2bcf30640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2b81037c0 0x7fd2b8074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2b7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2b8101c00 0x7fd2b8075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2bcf30640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2b81037c0 0x7fd2b8074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:52034/0 (socket says 172.21.15.120:52034) 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.180+0000 7fd2bcf30640 1 -- 172.21.15.120:0/1312969854 learned_addr learned my addr 172.21.15.120:0/1312969854 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2b77fe640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2b8102e00 0x7fd2b80745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2bcf30640 1 -- 172.21.15.120:0/1312969854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2b8101c00 msgr2=0x7fd2b8075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:11.213 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2bcf30640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2b8101c00 0x7fd2b8075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2bcf30640 1 -- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2b8102e00 msgr2=0x7fd2b80745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2bcf30640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2b8102e00 0x7fd2b80745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2bcf30640 1 -- 172.21.15.120:0/1312969854 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b80753e0 con 0x7fd2b81037c0 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2b7fff640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2b8101c00 0x7fd2b8075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2b77fe640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2b8102e00 0x7fd2b80745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2bcf30640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2b81037c0 0x7fd2b8074b20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2b002f540 tx=0x7fd2b0031750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2b57fa640 1 -- 172.21.15.120:0/1312969854 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0003b70 con 0x7fd2b81037c0 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2b57fa640 1 -- 172.21.15.120:0/1312969854 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2b0003d10 con 0x7fd2b81037c0 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2be993640 1 -- 172.21.15.120:0/1312969854 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b81a1fd0 con 0x7fd2b81037c0 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2b57fa640 1 -- 172.21.15.120:0/1312969854 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b002fb50 con 0x7fd2b81037c0 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.181+0000 7fd2be993640 1 -- 172.21.15.120:0/1312969854 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b81a24d0 con 0x7fd2b81037c0 2024-09-15T23:20:11.214 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.183+0000 7fd2be993640 1 -- 172.21.15.120:0/1312969854 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2b8108f30 con 0x7fd2b81037c0 2024-09-15T23:20:11.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.183+0000 7fd2b57fa640 1 -- 172.21.15.120:0/1312969854 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd2b002fcf0 con 0x7fd2b81037c0 2024-09-15T23:20:11.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.184+0000 7fd2b57fa640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2a00a32c0 0x7fd2a00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:11.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.184+0000 7fd2b57fa640 1 -- 172.21.15.120:0/1312969854 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5981+0+0 (secure 0 0 0) 0x7fd2b00f03d0 con 0x7fd2b81037c0 2024-09-15T23:20:11.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.184+0000 7fd2b7fff640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2a00a32c0 0x7fd2a00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:11.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.185+0000 7fd2b7fff640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2a00a32c0 0x7fd2a00a5780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8102c60 tx=0x7fd2a8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:11.215 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.188+0000 7fd2b57fa640 1 -- 172.21.15.120:0/1312969854 <== mon.2 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd2b00bda10 con 0x7fd2b81037c0 2024-09-15T23:20:11.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:20:11.017+0000 7f793f445640 -1 osd.9 0 waiting for initial osdmap 2024-09-15T23:20:11.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:20:11.028+0000 7f793b26d640 -1 osd.9 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: osdmap e59: 10 total, 9 up, 10 in 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:11.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:11.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:11.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:11.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:11.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:11 smithi089 ceph-mon[30933]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:20:11.400 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:11.399+0000 7fd2be993640 1 -- 172.21.15.120:0/1312969854 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd2b8062630 con 0x7fd2a00a32c0 2024-09-15T23:20:11.400 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: osdmap e59: 10 total, 9 up, 10 in 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:11.401 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.402 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:11.402 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:11.402 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:11.402 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.402 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:11 smithi120 ceph-mon[35250]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:20:11.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-15T23:20:11.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: osdmap e59: 10 total, 9 up, 10 in 2024-09-15T23:20:11.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-09-15T23:20:11.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:11.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:11.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:11.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:11.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:11.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:11.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:11.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:11.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:11 smithi174 ceph-mon[35586]: from='osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-09-15T23:20:12.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:12 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi089 2024-09-15T23:20:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:12 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi089 to 4271M 2024-09-15T23:20:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:12 smithi089 ceph-mon[30933]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:12 smithi089 ceph-mon[30933]: osdmap e60: 10 total, 9 up, 10 in 2024-09-15T23:20:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:12 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:12 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:12 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:20:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:12 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:20:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:12 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:12.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:12 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi089 2024-09-15T23:20:12.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:12 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi089 to 4271M 2024-09-15T23:20:12.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:12 smithi174 ceph-mon[35586]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:12.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:12 smithi174 ceph-mon[35586]: osdmap e60: 10 total, 9 up, 10 in 2024-09-15T23:20:12.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:12 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:12.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:12 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:12.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:12 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:20:12.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:12 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:20:12.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:12 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:12 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi089 2024-09-15T23:20:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:12 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi089 to 4271M 2024-09-15T23:20:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:12 smithi120 ceph-mon[35250]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:12 smithi120 ceph-mon[35250]: osdmap e60: 10 total, 9 up, 10 in 2024-09-15T23:20:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:12 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:12.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:12 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:12.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:12 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:20:12.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:12 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:20:12.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:12 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:13.139 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:13 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:20:13.140 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:13 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:20:13.140 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:13 smithi120 ceph-mon[35250]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:20:13.140 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:13 smithi120 ceph-mon[35250]: osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] boot 2024-09-15T23:20:13.140 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:13 smithi120 ceph-mon[35250]: osdmap e61: 10 total, 10 up, 10 in 2024-09-15T23:20:13.140 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:13 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:13.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:13 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:20:13.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:13 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:20:13.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:13 smithi089 ceph-mon[30933]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:20:13.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:13 smithi089 ceph-mon[30933]: osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] boot 2024-09-15T23:20:13.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:13 smithi089 ceph-mon[30933]: osdmap e61: 10 total, 10 up, 10 in 2024-09-15T23:20:13.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:13 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:13 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:20:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:13 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:20:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:13 smithi174 ceph-mon[35586]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:20:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:13 smithi174 ceph-mon[35586]: osd.9 [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] boot 2024-09-15T23:20:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:13 smithi174 ceph-mon[35586]: osdmap e61: 10 total, 10 up, 10 in 2024-09-15T23:20:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:13 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:20:14.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:14 smithi089 ceph-mon[30933]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:14.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:14 smithi089 ceph-mon[30933]: osdmap e62: 10 total, 10 up, 10 in 2024-09-15T23:20:14.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:14 smithi089 ceph-mon[30933]: from='client.? 172.21.15.120:0/1260101385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586ff85c-4fa6-415d-8843-7e57328f9e41"}]: dispatch 2024-09-15T23:20:14.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:14 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586ff85c-4fa6-415d-8843-7e57328f9e41"}]: dispatch 2024-09-15T23:20:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:14 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "586ff85c-4fa6-415d-8843-7e57328f9e41"}]': finished 2024-09-15T23:20:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:14 smithi089 ceph-mon[30933]: osdmap e63: 11 total, 10 up, 11 in 2024-09-15T23:20:14.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:14 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:14.433 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:14 smithi120 ceph-mon[35250]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:14.433 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:14 smithi120 ceph-mon[35250]: osdmap e62: 10 total, 10 up, 10 in 2024-09-15T23:20:14.433 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:14 smithi120 ceph-mon[35250]: from='client.? 172.21.15.120:0/1260101385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586ff85c-4fa6-415d-8843-7e57328f9e41"}]: dispatch 2024-09-15T23:20:14.433 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:14 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586ff85c-4fa6-415d-8843-7e57328f9e41"}]: dispatch 2024-09-15T23:20:14.433 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:14 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "586ff85c-4fa6-415d-8843-7e57328f9e41"}]': finished 2024-09-15T23:20:14.434 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:14 smithi120 ceph-mon[35250]: osdmap e63: 11 total, 10 up, 11 in 2024-09-15T23:20:14.434 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:14 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:14.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:14 smithi174 ceph-mon[35586]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:20:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:14 smithi174 ceph-mon[35586]: osdmap e62: 10 total, 10 up, 10 in 2024-09-15T23:20:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:14 smithi174 ceph-mon[35586]: from='client.? 172.21.15.120:0/1260101385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586ff85c-4fa6-415d-8843-7e57328f9e41"}]: dispatch 2024-09-15T23:20:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:14 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "586ff85c-4fa6-415d-8843-7e57328f9e41"}]: dispatch 2024-09-15T23:20:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:14 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "586ff85c-4fa6-415d-8843-7e57328f9e41"}]': finished 2024-09-15T23:20:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:14 smithi174 ceph-mon[35586]: osdmap e63: 11 total, 10 up, 11 in 2024-09-15T23:20:14.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:14 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:15.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:15 smithi089 ceph-mon[30933]: from='client.? 172.21.15.120:0/4213385101' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:20:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:15 smithi120 ceph-mon[35250]: from='client.? 172.21.15.120:0/4213385101' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:20:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:15 smithi174 ceph-mon[35586]: from='client.? 172.21.15.120:0/4213385101' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:20:16.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:16 smithi089 ceph-mon[30933]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:16 smithi120 ceph-mon[35250]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:16 smithi174 ceph-mon[35586]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 290 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:18.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:18 smithi120 ceph-mon[35250]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:18 smithi089 ceph-mon[30933]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:18.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:18 smithi174 ceph-mon[35586]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:20.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:20 smithi089 ceph-mon[30933]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:20 smithi120 ceph-mon[35250]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:20.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:20 smithi174 ceph-mon[35586]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:22 smithi120 ceph-mon[35250]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-15T23:20:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:22 smithi089 ceph-mon[30933]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-15T23:20:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:22 smithi174 ceph-mon[35586]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-15T23:20:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:23 smithi089 ceph-mon[30933]: Deploying daemon osd.10 on smithi120 2024-09-15T23:20:23.348 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:23 smithi120 ceph-mon[35250]: Deploying daemon osd.10 on smithi120 2024-09-15T23:20:23.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:23 smithi174 ceph-mon[35586]: Deploying daemon osd.10 on smithi120 2024-09-15T23:20:24.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:24 smithi120 ceph-mon[35250]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:24.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:24 smithi089 ceph-mon[30933]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:24.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:24 smithi174 ceph-mon[35586]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:25.376 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:25 smithi120 ceph-mon[35250]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:25 smithi174 ceph-mon[35586]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:25.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:25 smithi089 ceph-mon[30933]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:28.277 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:28 smithi089 ceph-mon[30933]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:28.277 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:28 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:28.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:28 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:28.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:28 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:28.334 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:28 smithi120 ceph-mon[35250]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:28.334 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:28 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:28.334 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:28 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:28.334 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:28 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:28.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:28 smithi174 ceph-mon[35586]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:28.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:28 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:28.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:28 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:28.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:28 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:30.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:30 smithi120 ceph-mon[35250]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:30.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:30 smithi089 ceph-mon[30933]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:30.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:30 smithi174 ceph-mon[35586]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:31.437 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.435+0000 7fd2b57fa640 1 -- 172.21.15.120:0/1312969854 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fd2b8062630 con 0x7fd2a00a32c0 2024-09-15T23:20:31.437 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 10 on host 'smithi120' 2024-09-15T23:20:31.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 -- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2a00a32c0 msgr2=0x7fd2a00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:31.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2a00a32c0 0x7fd2a00a5780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8102c60 tx=0x7fd2a8005f90 comp rx=0 tx=0).stop 2024-09-15T23:20:31.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 -- 172.21.15.120:0/1312969854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2b81037c0 msgr2=0x7fd2b8074b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:31.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2b81037c0 0x7fd2b8074b20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd2b002f540 tx=0x7fd2b0031750 comp rx=0 tx=0).stop 2024-09-15T23:20:31.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 -- 172.21.15.120:0/1312969854 shutdown_connections 2024-09-15T23:20:31.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2a00a32c0 0x7fd2a00a5780 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:31.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd2b81037c0 0x7fd2b8074b20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:31.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd2b8102e00 0x7fd2b80745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:31.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 --2- 172.21.15.120:0/1312969854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2b8101c00 0x7fd2b8075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:31.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 -- 172.21.15.120:0/1312969854 >> 172.21.15.120:0/1312969854 conn(0x7fd2b80fd3b0 msgr2=0x7fd2b8106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:31.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 -- 172.21.15.120:0/1312969854 shutdown_connections 2024-09-15T23:20:31.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-09-15T23:20:31.438+0000 7fd2be993640 1 -- 172.21.15.120:0/1312969854 wait complete. 2024-09-15T23:20:32.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:32.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:32.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:31 smithi089 ceph-mon[30933]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:32.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:32.062 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.062 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:31 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.075 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.075 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.075 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:32.075 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:32.075 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.075 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:31 smithi120 ceph-mon[35250]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:32.075 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:32.075 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.076 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:31 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.103 DEBUG:teuthology.orchestra.run.smithi120:osd.10> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.10.service 2024-09-15T23:20:32.106 INFO:tasks.cephadm:Deploying osd.11 on smithi174 with /dev/vg_nvme/lv_1... 2024-09-15T23:20:32.106 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-15T23:20:32.134 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.134 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.134 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:32.135 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:32.135 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.135 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:31 smithi174 ceph-mon[35586]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:32.135 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:32.135 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.135 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:31 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:32.361 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:20:33.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:20:33 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:20:33.646+0000 7f7e14009640 -1 osd.10 0 log_to_monitors true 2024-09-15T23:20:34.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:34 smithi174 ceph-mon[35586]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:34.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:34 smithi174 ceph-mon[35586]: from='osd.10 [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:20:34.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:34 smithi174 ceph-mon[35586]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:20:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:34 smithi089 ceph-mon[30933]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:34.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:34 smithi089 ceph-mon[30933]: from='osd.10 [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:20:34.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:34 smithi089 ceph-mon[30933]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:20:34.339 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:34 smithi120 ceph-mon[35250]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:34.339 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:34 smithi120 ceph-mon[35250]: from='osd.10 [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:20:34.339 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:34 smithi120 ceph-mon[35250]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:20:35.141 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-15T23:20:35.167 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_1 2024-09-15T23:20:35.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:35 smithi089 ceph-mon[30933]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-15T23:20:35.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:35 smithi089 ceph-mon[30933]: osdmap e64: 11 total, 10 up, 11 in 2024-09-15T23:20:35.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:35 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:35.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:35 smithi089 ceph-mon[30933]: from='osd.10 [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:20:35.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:35 smithi089 ceph-mon[30933]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:20:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:20:35.473 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:20:35 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:20:35.033+0000 7f7e0ff8c640 -1 osd.10 0 waiting for initial osdmap 2024-09-15T23:20:35.473 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:20:35 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:20:35.045+0000 7f7e0bdb4640 -1 osd.10 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:20:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:35 smithi120 ceph-mon[35250]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-15T23:20:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:35 smithi120 ceph-mon[35250]: osdmap e64: 11 total, 10 up, 11 in 2024-09-15T23:20:35.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:35 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:35.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:35 smithi120 ceph-mon[35250]: from='osd.10 [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:20:35.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:35 smithi120 ceph-mon[35250]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:20:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:35 smithi174 ceph-mon[35586]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-15T23:20:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:35 smithi174 ceph-mon[35586]: osdmap e64: 11 total, 10 up, 11 in 2024-09-15T23:20:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:35 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:35 smithi174 ceph-mon[35586]: from='osd.10 [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:20:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:35 smithi174 ceph-mon[35586]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-09-15T23:20:35.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 -- 172.21.15.174:0/1980255888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc3c1026d0 msgr2=0x7efc3c108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:35.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 --2- 172.21.15.174:0/1980255888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc3c1026d0 0x7efc3c108f60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efc34009ca0 tx=0x7efc3402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 -- 172.21.15.174:0/1980255888 shutdown_connections 2024-09-15T23:20:35.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 --2- 172.21.15.174:0/1980255888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc3c1026d0 0x7efc3c108f60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 --2- 172.21.15.174:0/1980255888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc3c101d10 0x7efc3c102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.941 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 --2- 172.21.15.174:0/1980255888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c100b10 0x7efc3c100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 -- 172.21.15.174:0/1980255888 >> 172.21.15.174:0/1980255888 conn(0x7efc3c0fc2a0 msgr2=0x7efc3c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 -- 172.21.15.174:0/1980255888 shutdown_connections 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 -- 172.21.15.174:0/1980255888 wait complete. 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 Processor -- start 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.931+0000 7efc4233a640 1 -- start start 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc4233a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc3c100b10 0x7efc3c197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc4233a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc3c101d10 0x7efc3c198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc4233a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c1026d0 0x7efc3c19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc4233a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c10d880 con 0x7efc3c1026d0 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc4233a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c10d700 con 0x7efc3c100b10 2024-09-15T23:20:35.942 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc4233a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c10da00 con 0x7efc3c101d10 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc3c101d10 0x7efc3c198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc3c101d10 0x7efc3c198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:40636/0 (socket says 172.21.15.174:40636) 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3b7fe640 1 -- 172.21.15.174:0/4276959321 learned_addr learned my addr 172.21.15.174:0/4276959321 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3bfff640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc3c100b10 0x7efc3c197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc408d7640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c1026d0 0x7efc3c19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3b7fe640 1 -- 172.21.15.174:0/4276959321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc3c100b10 msgr2=0x7efc3c197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3b7fe640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc3c100b10 0x7efc3c197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3b7fe640 1 -- 172.21.15.174:0/4276959321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c1026d0 msgr2=0x7efc3c19f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3b7fe640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c1026d0 0x7efc3c19f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3b7fe640 1 -- 172.21.15.174:0/4276959321 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3c19fa50 con 0x7efc3c101d10 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3bfff640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc3c100b10 0x7efc3c197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc3b7fe640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc3c101d10 0x7efc3c198360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efc3000c890 tx=0x7efc3000cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:35.943 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc397fa640 1 -- 172.21.15.174:0/4276959321 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc30004220 con 0x7efc3c101d10 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc397fa640 1 -- 172.21.15.174:0/4276959321 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc300043c0 con 0x7efc3c101d10 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.932+0000 7efc4233a640 1 -- 172.21.15.174:0/4276959321 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc3c19fd40 con 0x7efc3c101d10 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.933+0000 7efc397fa640 1 -- 172.21.15.174:0/4276959321 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc300049e0 con 0x7efc3c101d10 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.933+0000 7efc4233a640 1 -- 172.21.15.174:0/4276959321 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc3c1a0220 con 0x7efc3c101d10 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.934+0000 7efc4233a640 1 -- 172.21.15.174:0/4276959321 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc3c107e40 con 0x7efc3c101d10 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.935+0000 7efc397fa640 1 -- 172.21.15.174:0/4276959321 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7efc300079b0 con 0x7efc3c101d10 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.936+0000 7efc397fa640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc240a3270 0x7efc240a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.936+0000 7efc397fa640 1 -- 172.21.15.174:0/4276959321 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 6414+0+0 (secure 0 0 0) 0x7efc300cf280 con 0x7efc3c101d10 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.936+0000 7efc3bfff640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc240a3270 0x7efc240a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.937+0000 7efc3bfff640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc240a3270 0x7efc240a5730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efc3c101b70 tx=0x7efc2c00a5e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:35.944 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:35.939+0000 7efc397fa640 1 -- 172.21.15.174:0/4276959321 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efc30014030 con 0x7efc3c101d10 2024-09-15T23:20:36.119 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:36.117+0000 7efc4233a640 1 -- 172.21.15.174:0/4276959321 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7efc3c062630 con 0x7efc240a3270 2024-09-15T23:20:36.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:36.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:20:36.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: osdmap e65: 11 total, 10 up, 11 in 2024-09-15T23:20:36.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:36.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:36.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi120 2024-09-15T23:20:36.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:36.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:36.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:36 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:36.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: osdmap e65: 11 total, 10 up, 11 in 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi120 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:36 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:20:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-09-15T23:20:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: osdmap e65: 11 total, 10 up, 11 in 2024-09-15T23:20:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi120 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:36 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:37.224 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:37 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:20:37.224 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:37 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:20:37.224 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:37 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi120 to 4272M 2024-09-15T23:20:37.224 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:37 smithi174 ceph-mon[35586]: osd.10 [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] boot 2024-09-15T23:20:37.224 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:37 smithi174 ceph-mon[35586]: osdmap e66: 11 total, 11 up, 11 in 2024-09-15T23:20:37.225 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:37 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:37.225 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:37 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:20:37.225 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:37 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:20:37.225 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:37 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:37.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:37 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:20:37.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:37 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:20:37.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:37 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi120 to 4272M 2024-09-15T23:20:37.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:37 smithi089 ceph-mon[30933]: osd.10 [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] boot 2024-09-15T23:20:37.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:37 smithi089 ceph-mon[30933]: osdmap e66: 11 total, 11 up, 11 in 2024-09-15T23:20:37.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:37 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:37.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:37 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:20:37.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:37 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:20:37.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:37 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:37 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:20:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:37 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:20:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:37 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi120 to 4272M 2024-09-15T23:20:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:37 smithi120 ceph-mon[35250]: osd.10 [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] boot 2024-09-15T23:20:37.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:37 smithi120 ceph-mon[35250]: osdmap e66: 11 total, 11 up, 11 in 2024-09-15T23:20:37.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:37 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:20:37.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:37 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:20:37.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:37 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:20:37.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:37 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:38.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:38 smithi174 ceph-mon[35586]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:20:38.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:38 smithi174 ceph-mon[35586]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:38.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:38 smithi174 ceph-mon[35586]: osdmap e67: 11 total, 11 up, 11 in 2024-09-15T23:20:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:38 smithi089 ceph-mon[30933]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:20:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:38 smithi089 ceph-mon[30933]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:38 smithi089 ceph-mon[30933]: osdmap e67: 11 total, 11 up, 11 in 2024-09-15T23:20:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:38 smithi120 ceph-mon[35250]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:20:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:38 smithi120 ceph-mon[35250]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:38 smithi120 ceph-mon[35250]: osdmap e67: 11 total, 11 up, 11 in 2024-09-15T23:20:39.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:39 smithi174 ceph-mon[35586]: osdmap e68: 11 total, 11 up, 11 in 2024-09-15T23:20:39.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:39 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/2267949349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06c5dd00-db2a-43e0-98e9-61468771c3ea"}]: dispatch 2024-09-15T23:20:39.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:39 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06c5dd00-db2a-43e0-98e9-61468771c3ea"}]: dispatch 2024-09-15T23:20:39.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:39 smithi174 ceph-mon[35586]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06c5dd00-db2a-43e0-98e9-61468771c3ea"}]': finished 2024-09-15T23:20:39.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:39 smithi174 ceph-mon[35586]: osdmap e69: 12 total, 11 up, 12 in 2024-09-15T23:20:39.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:20:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:39 smithi089 ceph-mon[30933]: osdmap e68: 11 total, 11 up, 11 in 2024-09-15T23:20:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:39 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/2267949349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06c5dd00-db2a-43e0-98e9-61468771c3ea"}]: dispatch 2024-09-15T23:20:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:39 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06c5dd00-db2a-43e0-98e9-61468771c3ea"}]: dispatch 2024-09-15T23:20:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:39 smithi089 ceph-mon[30933]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06c5dd00-db2a-43e0-98e9-61468771c3ea"}]': finished 2024-09-15T23:20:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:39 smithi089 ceph-mon[30933]: osdmap e69: 12 total, 11 up, 12 in 2024-09-15T23:20:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:20:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:39 smithi120 ceph-mon[35250]: osdmap e68: 11 total, 11 up, 11 in 2024-09-15T23:20:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:39 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/2267949349' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06c5dd00-db2a-43e0-98e9-61468771c3ea"}]: dispatch 2024-09-15T23:20:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:39 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06c5dd00-db2a-43e0-98e9-61468771c3ea"}]: dispatch 2024-09-15T23:20:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:39 smithi120 ceph-mon[35250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06c5dd00-db2a-43e0-98e9-61468771c3ea"}]': finished 2024-09-15T23:20:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:39 smithi120 ceph-mon[35250]: osdmap e69: 12 total, 11 up, 12 in 2024-09-15T23:20:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:39 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:20:40.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:40 smithi089 ceph-mon[30933]: pgmap v228: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:40.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:40 smithi089 ceph-mon[30933]: from='client.? 172.21.15.174:0/3624934388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:20:40.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:40 smithi089 ceph-mon[30933]: osdmap e70: 12 total, 11 up, 12 in 2024-09-15T23:20:40.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:40 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:20:40.375 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:40 smithi120 ceph-mon[35250]: pgmap v228: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:40.375 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:40 smithi120 ceph-mon[35250]: from='client.? 172.21.15.174:0/3624934388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:20:40.375 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:40 smithi120 ceph-mon[35250]: osdmap e70: 12 total, 11 up, 12 in 2024-09-15T23:20:40.375 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:40 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:20:40.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:40 smithi174 ceph-mon[35586]: pgmap v228: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:40.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:40 smithi174 ceph-mon[35586]: from='client.? 172.21.15.174:0/3624934388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:20:40.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:40 smithi174 ceph-mon[35586]: osdmap e70: 12 total, 11 up, 12 in 2024-09-15T23:20:40.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:40 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:20:41.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:41 smithi120 ceph-mon[35250]: osdmap e71: 12 total, 11 up, 12 in 2024-09-15T23:20:41.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:20:41.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:41 smithi120 ceph-mon[35250]: pgmap v231: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:41.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:41 smithi174 ceph-mon[35586]: osdmap e71: 12 total, 11 up, 12 in 2024-09-15T23:20:41.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:20:41.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:41 smithi174 ceph-mon[35586]: pgmap v231: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:42.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:41 smithi089 ceph-mon[30933]: osdmap e71: 12 total, 11 up, 12 in 2024-09-15T23:20:42.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:20:42.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:41 smithi089 ceph-mon[30933]: pgmap v231: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:44.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:44 smithi174 ceph-mon[35586]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-15T23:20:44.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:44 smithi089 ceph-mon[30933]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-15T23:20:44.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:44 smithi120 ceph-mon[35250]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-15T23:20:46.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:46 smithi174 ceph-mon[35586]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-15T23:20:46.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:46 smithi089 ceph-mon[30933]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-15T23:20:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:46 smithi120 ceph-mon[35250]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-09-15T23:20:48.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:48 smithi174 ceph-mon[35586]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:20:48.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-15T23:20:48.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:48 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:48.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:48 smithi089 ceph-mon[30933]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:20:48.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-15T23:20:48.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:48 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:48 smithi120 ceph-mon[35250]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:20:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-15T23:20:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:48 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:49.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:49 smithi089 ceph-mon[30933]: Deploying daemon osd.11 on smithi174 2024-09-15T23:20:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:49 smithi120 ceph-mon[35250]: Deploying daemon osd.11 on smithi174 2024-09-15T23:20:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:49 smithi174 ceph-mon[35586]: Deploying daemon osd.11 on smithi174 2024-09-15T23:20:50.214 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:50 smithi174 ceph-mon[35586]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-15T23:20:50.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:50 smithi089 ceph-mon[30933]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-15T23:20:50.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:50 smithi120 ceph-mon[35250]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-15T23:20:52.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:52 smithi089 ceph-mon[30933]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-15T23:20:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:52 smithi120 ceph-mon[35250]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-15T23:20:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:52 smithi174 ceph-mon[35586]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-15T23:20:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:54 smithi089 ceph-mon[30933]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:20:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:54 smithi120 ceph-mon[35250]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:20:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:54.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:54 smithi174 ceph-mon[35586]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:20:54.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:54.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:54.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:56.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:56 smithi174 ceph-mon[35586]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:56 smithi089 ceph-mon[30933]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:56.439 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 11 on host 'smithi174' 2024-09-15T23:20:56.439 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.438+0000 7efc397fa640 1 -- 172.21.15.174:0/4276959321 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7efc3c062630 con 0x7efc240a3270 2024-09-15T23:20:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 -- 172.21.15.174:0/4276959321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc240a3270 msgr2=0x7efc240a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc240a3270 0x7efc240a5730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efc3c101b70 tx=0x7efc2c00a5e0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 -- 172.21.15.174:0/4276959321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc3c101d10 msgr2=0x7efc3c198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc3c101d10 0x7efc3c198360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efc3000c890 tx=0x7efc3000cd50 comp rx=0 tx=0).stop 2024-09-15T23:20:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 -- 172.21.15.174:0/4276959321 shutdown_connections 2024-09-15T23:20:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc240a3270 0x7efc240a5730 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c1026d0 0x7efc3c19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc3c101d10 0x7efc3c198360 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 --2- 172.21.15.174:0/4276959321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc3c100b10 0x7efc3c197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.442 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 -- 172.21.15.174:0/4276959321 >> 172.21.15.174:0/4276959321 conn(0x7efc3c0fc2a0 msgr2=0x7efc3c105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:56.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.440+0000 7efc4233a640 1 -- 172.21.15.174:0/4276959321 shutdown_connections 2024-09-15T23:20:56.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:20:56.441+0000 7efc4233a640 1 -- 172.21.15.174:0/4276959321 wait complete. 2024-09-15T23:20:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:56 smithi120 ceph-mon[35250]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:57.066 DEBUG:teuthology.orchestra.run.smithi174:osd.11> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.11.service 2024-09-15T23:20:57.068 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-15T23:20:57.068 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-15T23:20:57.333 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:57.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:57 smithi120 ceph-mon[35250]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:57.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:57 smithi089 ceph-mon[30933]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:57.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.727 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:57 smithi174 ceph-mon[35586]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:57.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:20:57.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.809+0000 7f76bf987640 1 -- 172.21.15.89:0/3001986868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b8101c20 msgr2=0x7f76b8102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:57.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.809+0000 7f76bf987640 1 --2- 172.21.15.89:0/3001986868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b8101c20 0x7f76b8102020 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009900 tx=0x7f76a802a410 comp rx=0 tx=0).stop 2024-09-15T23:20:57.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.809+0000 7f76bf987640 1 -- 172.21.15.89:0/3001986868 shutdown_connections 2024-09-15T23:20:57.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.809+0000 7f76bf987640 1 --2- 172.21.15.89:0/3001986868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b81037e0 0x7f76b810a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:57.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.809+0000 7f76bf987640 1 --2- 172.21.15.89:0/3001986868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76b8102e20 0x7f76b81032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:57.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.809+0000 7f76bf987640 1 --2- 172.21.15.89:0/3001986868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b8101c20 0x7f76b8102020 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:57.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.809+0000 7f76bf987640 1 -- 172.21.15.89:0/3001986868 >> 172.21.15.89:0/3001986868 conn(0x7f76b80fd3b0 msgr2=0x7f76b80ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:57.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.809+0000 7f76bf987640 1 -- 172.21.15.89:0/3001986868 shutdown_connections 2024-09-15T23:20:57.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.809+0000 7f76bf987640 1 -- 172.21.15.89:0/3001986868 wait complete. 2024-09-15T23:20:57.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bf987640 1 Processor -- start 2024-09-15T23:20:57.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bf987640 1 -- start start 2024-09-15T23:20:57.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bf987640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b8101c20 0x7f76b8197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:57.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bf987640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76b8102e20 0x7f76b8198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:57.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bf987640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b81037e0 0x7f76b819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:57.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bf987640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f76b810d8a0 con 0x7f76b81037e0 2024-09-15T23:20:57.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bf987640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f76b810d720 con 0x7f76b8101c20 2024-09-15T23:20:57.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bf987640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f76b810da20 con 0x7f76b8102e20 2024-09-15T23:20:57.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bdf24640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b81037e0 0x7f76b819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:57.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bdf24640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b81037e0 0x7f76b819f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42806/0 (socket says 172.21.15.89:42806) 2024-09-15T23:20:57.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bdf24640 1 -- 172.21.15.89:0/1906309443 learned_addr learned my addr 172.21.15.89:0/1906309443 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:20:57.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bd723640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b8101c20 0x7f76b8197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:57.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bcf22640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76b8102e20 0x7f76b8198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:57.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bdf24640 1 -- 172.21.15.89:0/1906309443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76b8102e20 msgr2=0x7f76b8198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:57.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bdf24640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76b8102e20 0x7f76b8198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:57.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bdf24640 1 -- 172.21.15.89:0/1906309443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b8101c20 msgr2=0x7f76b8197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:57.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bdf24640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b8101c20 0x7f76b8197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:57.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bdf24640 1 -- 172.21.15.89:0/1906309443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b819faf0 con 0x7f76b81037e0 2024-09-15T23:20:57.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bd723640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b8101c20 0x7f76b8197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:57.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.810+0000 7f76bcf22640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76b8102e20 0x7f76b8198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:57.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.811+0000 7f76bdf24640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b81037e0 0x7f76b819f480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f76b000cce0 tx=0x7f76b000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:57.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.811+0000 7f76ae7fc640 1 -- 172.21.15.89:0/1906309443 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b000f450 con 0x7f76b81037e0 2024-09-15T23:20:57.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.811+0000 7f76bf987640 1 -- 172.21.15.89:0/1906309443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b819fde0 con 0x7f76b81037e0 2024-09-15T23:20:57.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.811+0000 7f76ae7fc640 1 -- 172.21.15.89:0/1906309443 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76b000fad0 con 0x7f76b81037e0 2024-09-15T23:20:57.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.811+0000 7f76ae7fc640 1 -- 172.21.15.89:0/1906309443 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0018c10 con 0x7f76b81037e0 2024-09-15T23:20:57.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.811+0000 7f76bf987640 1 -- 172.21.15.89:0/1906309443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b81a0290 con 0x7f76b81037e0 2024-09-15T23:20:57.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.812+0000 7f76bf987640 1 -- 172.21.15.89:0/1906309443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76b8108ff0 con 0x7f76b81037e0 2024-09-15T23:20:57.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.813+0000 7f76ae7fc640 1 -- 172.21.15.89:0/1906309443 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f76b000f5f0 con 0x7f76b81037e0 2024-09-15T23:20:57.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.814+0000 7f76ae7fc640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76a40a3270 0x7f76a40a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:57.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.814+0000 7f76ae7fc640 1 -- 172.21.15.89:0/1906309443 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 6809+0+0 (secure 0 0 0) 0x7f76b00cfa70 con 0x7f76b81037e0 2024-09-15T23:20:57.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.814+0000 7f76bd723640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76a40a3270 0x7f76a40a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:57.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.814+0000 7f76bd723640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76a40a3270 0x7f76a40a5730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f76a8006010 tx=0x7f76a8032dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:57.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.818+0000 7f76ae7fc640 1 -- 172.21.15.89:0/1906309443 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f76b0014030 con 0x7f76b81037e0 2024-09-15T23:20:57.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.989+0000 7f76bf987640 1 -- 172.21.15.89:0/1906309443 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f76b802d9c0 con 0x7f76b81037e0 2024-09-15T23:20:57.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.989+0000 7f76ae7fc640 1 -- 172.21.15.89:0/1906309443 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v71) ==== 74+0+133 (secure 0 0 0) 0x7f76b009cd70 con 0x7f76b81037e0 2024-09-15T23:20:57.991 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:20:57.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 -- 172.21.15.89:0/1906309443 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76a40a3270 msgr2=0x7f76a40a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:57.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76a40a3270 0x7f76a40a5730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f76a8006010 tx=0x7f76a8032dd0 comp rx=0 tx=0).stop 2024-09-15T23:20:57.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 -- 172.21.15.89:0/1906309443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b81037e0 msgr2=0x7f76b819f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:57.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b81037e0 0x7f76b819f480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f76b000cce0 tx=0x7f76b000a620 comp rx=0 tx=0).stop 2024-09-15T23:20:57.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 -- 172.21.15.89:0/1906309443 shutdown_connections 2024-09-15T23:20:57.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76a40a3270 0x7f76a40a5730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:57.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76b81037e0 0x7f76b819f480 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:57.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76b8102e20 0x7f76b8198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:57.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 --2- 172.21.15.89:0/1906309443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f76b8101c20 0x7f76b8197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:57.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 -- 172.21.15.89:0/1906309443 >> 172.21.15.89:0/1906309443 conn(0x7f76b80fd3b0 msgr2=0x7f76b81061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:57.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 -- 172.21.15.89:0/1906309443 shutdown_connections 2024-09-15T23:20:57.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:20:57.992+0000 7f76bf987640 1 -- 172.21.15.89:0/1906309443 wait complete. 2024-09-15T23:20:58.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:58 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1906309443' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:20:58.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:58 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1906309443' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:20:58.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:58 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1906309443' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:20:58.844 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":71,"num_osds":12,"num_up_osds":11,"osd_up_since":1726442436,"num_in_osds":12,"osd_in_since":1726442438,"num_remapped_pgs":0} 2024-09-15T23:20:59.081 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:20:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:20:58.975+0000 7f8531970640 -1 osd.11 0 log_to_monitors true 2024-09-15T23:20:59.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:59 smithi120 ceph-mon[35250]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:59.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:59 smithi120 ceph-mon[35250]: from='osd.11 [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:20:59.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:20:59 smithi120 ceph-mon[35250]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:20:59.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:59 smithi174 ceph-mon[35586]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:59.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:59 smithi174 ceph-mon[35586]: from='osd.11 [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:20:59.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:20:59 smithi174 ceph-mon[35586]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:20:59.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:59 smithi089 ceph-mon[30933]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:20:59.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:59 smithi089 ceph-mon[30933]: from='osd.11 [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:20:59.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:20:59 smithi089 ceph-mon[30933]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:20:59.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-15T23:21:00.094 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:00.554 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-15T23:21:00.554 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: osdmap e72: 12 total, 11 up, 12 in 2024-09-15T23:21:00.554 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:00.554 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='osd.11 [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:21:00.554 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:00.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:00.556 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.633+0000 7f580384f640 1 -- 172.21.15.89:0/902358249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57fc100b10 msgr2=0x7f57fc100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:00.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.633+0000 7f580384f640 1 --2- 172.21.15.89:0/902358249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57fc100b10 0x7f57fc100f10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009900 tx=0x7f57f002a410 comp rx=0 tx=0).stop 2024-09-15T23:21:00.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.633+0000 7f580384f640 1 -- 172.21.15.89:0/902358249 shutdown_connections 2024-09-15T23:21:00.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.633+0000 7f580384f640 1 --2- 172.21.15.89:0/902358249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57fc1026d0 0x7f57fc108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:00.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.633+0000 7f580384f640 1 --2- 172.21.15.89:0/902358249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc101d10 0x7f57fc102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:00.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.633+0000 7f580384f640 1 --2- 172.21.15.89:0/902358249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57fc100b10 0x7f57fc100f10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.633+0000 7f580384f640 1 -- 172.21.15.89:0/902358249 >> 172.21.15.89:0/902358249 conn(0x7f57fc0fc2a0 msgr2=0x7f57fc0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.633+0000 7f580384f640 1 -- 172.21.15.89:0/902358249 shutdown_connections 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.633+0000 7f580384f640 1 -- 172.21.15.89:0/902358249 wait complete. 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f580384f640 1 Processor -- start 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f580384f640 1 -- start start 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f580384f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57fc100b10 0x7f57fc19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f580384f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57fc101d10 0x7f57fc19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f580384f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc1026d0 0x7f57fc1a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f580384f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc10c7a0 con 0x7f57fc101d10 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f580384f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc10c620 con 0x7f57fc1026d0 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f580384f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc10c920 con 0x7f57fc100b10 2024-09-15T23:21:00.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f5800dea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57fc101d10 0x7f57fc19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f5800dea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57fc101d10 0x7f57fc19a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42832/0 (socket says 172.21.15.89:42832) 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f5800dea640 1 -- 172.21.15.89:0/3683762748 learned_addr learned my addr 172.21.15.89:0/3683762748 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f58015eb640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57fc100b10 0x7f57fc19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.634+0000 7f5801dec640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc1026d0 0x7f57fc1a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f5800dea640 1 -- 172.21.15.89:0/3683762748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57fc100b10 msgr2=0x7f57fc19a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f5800dea640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57fc100b10 0x7f57fc19a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f5800dea640 1 -- 172.21.15.89:0/3683762748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc1026d0 msgr2=0x7f57fc1a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f5800dea640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc1026d0 0x7f57fc1a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f5800dea640 1 -- 172.21.15.89:0/3683762748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57fc1a1d60 con 0x7f57fc101d10 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f5801dec640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc1026d0 0x7f57fc1a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f5800dea640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57fc101d10 0x7f57fc19a670 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57f800d950 tx=0x7f57f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f57ee7fc640 1 -- 172.21.15.89:0/3683762748 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f8017030 con 0x7f57fc101d10 2024-09-15T23:21:00.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f57ee7fc640 1 -- 172.21.15.89:0/3683762748 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57f8004020 con 0x7f57fc101d10 2024-09-15T23:21:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f580384f640 1 -- 172.21.15.89:0/3683762748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57fc1a1ff0 con 0x7f57fc101d10 2024-09-15T23:21:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f57ee7fc640 1 -- 172.21.15.89:0/3683762748 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f80047a0 con 0x7f57fc101d10 2024-09-15T23:21:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.635+0000 7f580384f640 1 -- 172.21.15.89:0/3683762748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57fc1a2500 con 0x7f57fc101d10 2024-09-15T23:21:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.636+0000 7f580384f640 1 -- 172.21.15.89:0/3683762748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57fc100f40 con 0x7f57fc101d10 2024-09-15T23:21:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.637+0000 7f57ee7fc640 1 -- 172.21.15.89:0/3683762748 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f57f80041d0 con 0x7f57fc101d10 2024-09-15T23:21:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.638+0000 7f57ee7fc640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57e80a3270 0x7f57e80a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.638+0000 7f58015eb640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57e80a3270 0x7f57e80a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.638+0000 7f57ee7fc640 1 -- 172.21.15.89:0/3683762748 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6847+0+0 (secure 0 0 0) 0x7f57f80ced80 con 0x7f57fc101d10 2024-09-15T23:21:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.638+0000 7f58015eb640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57e80a3270 0x7f57e80a5730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f57f0005ab0 tx=0x7f57f0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:00.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.642+0000 7f57ee7fc640 1 -- 172.21.15.89:0/3683762748 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f57fc100f40 con 0x7f57fc101d10 2024-09-15T23:21:00.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: osdmap e72: 12 total, 11 up, 12 in 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='osd.11 [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.725 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:00.725 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:00.725 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.725 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:00.725 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:00.725 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:00.725 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: osdmap e72: 12 total, 11 up, 12 in 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='osd.11 [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:00.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:00.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:00.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:00.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:00.737 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:00.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.807+0000 7f580384f640 1 -- 172.21.15.89:0/3683762748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f57fc1a2a90 con 0x7f57fc101d10 2024-09-15T23:21:00.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.807+0000 7f57ee7fc640 1 -- 172.21.15.89:0/3683762748 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) ==== 74+0+133 (secure 0 0 0) 0x7f57f809c060 con 0x7f57fc101d10 2024-09-15T23:21:00.808 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 -- 172.21.15.89:0/3683762748 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57e80a3270 msgr2=0x7f57e80a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57e80a3270 0x7f57e80a5730 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f57f0005ab0 tx=0x7f57f0000bc0 comp rx=0 tx=0).stop 2024-09-15T23:21:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 -- 172.21.15.89:0/3683762748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57fc101d10 msgr2=0x7f57fc19a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57fc101d10 0x7f57fc19a670 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57f800d950 tx=0x7f57f800de10 comp rx=0 tx=0).stop 2024-09-15T23:21:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 -- 172.21.15.89:0/3683762748 shutdown_connections 2024-09-15T23:21:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f57e80a3270 0x7f57e80a5730 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:00.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57fc1026d0 0x7f57fc1a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:00.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f57fc101d10 0x7f57fc19a670 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:00.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 --2- 172.21.15.89:0/3683762748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57fc100b10 0x7f57fc19a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:00.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 -- 172.21.15.89:0/3683762748 >> 172.21.15.89:0/3683762748 conn(0x7f57fc0fc2a0 msgr2=0x7f57fc06a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:00.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 -- 172.21.15.89:0/3683762748 shutdown_connections 2024-09-15T23:21:00.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:00.810+0000 7f580384f640 1 -- 172.21.15.89:0/3683762748 wait complete. 2024-09-15T23:21:01.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:21:00.823+0000 7f852d8f3640 -1 osd.11 0 waiting for initial osdmap 2024-09-15T23:21:01.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:21:00 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:21:00.839+0000 7f8525713640 -1 osd.11 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:21:01.495 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":73,"num_osds":12,"num_up_osds":11,"osd_up_since":1726442436,"num_in_osds":12,"osd_in_since":1726442438,"num_remapped_pgs":0} 2024-09-15T23:21:01.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:01 smithi120 ceph-mon[35250]: Detected new or changed devices on smithi174 2024-09-15T23:21:01.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:01 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi174 to 4272M 2024-09-15T23:21:01.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:01 smithi120 ceph-mon[35250]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:21:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:01 smithi120 ceph-mon[35250]: osdmap e73: 12 total, 11 up, 12 in 2024-09-15T23:21:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:01 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3683762748' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:21:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:01 smithi120 ceph-mon[35250]: from='osd.11 ' entity='osd.11' 2024-09-15T23:21:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:01 smithi120 ceph-mon[35250]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:21:01.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:01 smithi174 ceph-mon[35586]: Detected new or changed devices on smithi174 2024-09-15T23:21:01.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:01 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi174 to 4272M 2024-09-15T23:21:01.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:01 smithi174 ceph-mon[35586]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:21:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:01 smithi174 ceph-mon[35586]: osdmap e73: 12 total, 11 up, 12 in 2024-09-15T23:21:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:01 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3683762748' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:21:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:01 smithi174 ceph-mon[35586]: from='osd.11 ' entity='osd.11' 2024-09-15T23:21:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:01 smithi174 ceph-mon[35586]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:21:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:01 smithi089 ceph-mon[30933]: Detected new or changed devices on smithi174 2024-09-15T23:21:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:01 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi174 to 4272M 2024-09-15T23:21:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:01 smithi089 ceph-mon[30933]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-09-15T23:21:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:01 smithi089 ceph-mon[30933]: osdmap e73: 12 total, 11 up, 12 in 2024-09-15T23:21:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:01.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:01 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3683762748' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:21:01.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:01 smithi089 ceph-mon[30933]: from='osd.11 ' entity='osd.11' 2024-09-15T23:21:01.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:01 smithi089 ceph-mon[30933]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:21:02.496 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-15T23:21:02.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:02 smithi120 ceph-mon[35250]: purged_snaps scrub starts 2024-09-15T23:21:02.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:02 smithi120 ceph-mon[35250]: purged_snaps scrub ok 2024-09-15T23:21:02.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:02.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:02 smithi120 ceph-mon[35250]: osd.11 [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] boot 2024-09-15T23:21:02.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:02 smithi120 ceph-mon[35250]: osdmap e74: 12 total, 12 up, 12 in 2024-09-15T23:21:02.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:02.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:02 smithi174 ceph-mon[35586]: purged_snaps scrub starts 2024-09-15T23:21:02.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:02 smithi174 ceph-mon[35586]: purged_snaps scrub ok 2024-09-15T23:21:02.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:02.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:02 smithi174 ceph-mon[35586]: osd.11 [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] boot 2024-09-15T23:21:02.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:02 smithi174 ceph-mon[35586]: osdmap e74: 12 total, 12 up, 12 in 2024-09-15T23:21:02.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:02.748 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:02.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:02 smithi089 ceph-mon[30933]: purged_snaps scrub starts 2024-09-15T23:21:02.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:02 smithi089 ceph-mon[30933]: purged_snaps scrub ok 2024-09-15T23:21:02.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:02.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:02 smithi089 ceph-mon[30933]: osd.11 [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] boot 2024-09-15T23:21:02.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:02 smithi089 ceph-mon[30933]: osdmap e74: 12 total, 12 up, 12 in 2024-09-15T23:21:02.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:21:03.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 -- 172.21.15.89:0/3333316126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5474100d90 msgr2=0x7f54741011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:03.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 --2- 172.21.15.89:0/3333316126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5474100d90 0x7f54741011f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5464009900 tx=0x7f546402a410 comp rx=0 tx=0).stop 2024-09-15T23:21:03.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 -- 172.21.15.89:0/3333316126 shutdown_connections 2024-09-15T23:21:03.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 --2- 172.21.15.89:0/3333316126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5474101730 0x7f5474101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 --2- 172.21.15.89:0/3333316126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5474100d90 0x7f54741011f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 --2- 172.21.15.89:0/3333316126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5474100450 0x7f5474100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 -- 172.21.15.89:0/3333316126 >> 172.21.15.89:0/3333316126 conn(0x7f54740fc280 msgr2=0x7f54740fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 -- 172.21.15.89:0/3333316126 shutdown_connections 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 -- 172.21.15.89:0/3333316126 wait complete. 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 Processor -- start 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 -- start start 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5474100450 0x7f5474199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5474100d90 0x7f547419a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5474101730 0x7f54741a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f547410fa20 con 0x7f5474100450 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f547410f8a0 con 0x7f5474101730 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.225+0000 7f547b6d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f547410fba0 con 0x7f5474100d90 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479473640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5474100450 0x7f5474199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:03.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479473640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5474100450 0x7f5474199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42862/0 (socket says 172.21.15.89:42862) 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479473640 1 -- 172.21.15.89:0/685664468 learned_addr learned my addr 172.21.15.89:0/685664468 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5478c72640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5474100d90 0x7f547419a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479c74640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5474101730 0x7f54741a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479473640 1 -- 172.21.15.89:0/685664468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5474100d90 msgr2=0x7f547419a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479473640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5474100d90 0x7f547419a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479473640 1 -- 172.21.15.89:0/685664468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5474101730 msgr2=0x7f54741a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479473640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5474101730 0x7f54741a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479473640 1 -- 172.21.15.89:0/685664468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54741a1c00 con 0x7f5474100450 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479c74640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5474101730 0x7f54741a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f5479473640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5474100450 0x7f5474199fd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f547000c920 tx=0x7f547000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f546a7fc640 1 -- 172.21.15.89:0/685664468 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5470007c70 con 0x7f5474100450 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f547b6d7640 1 -- 172.21.15.89:0/685664468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54741a1ef0 con 0x7f5474100450 2024-09-15T23:21:03.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f546a7fc640 1 -- 172.21.15.89:0/685664468 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5470007e10 con 0x7f5474100450 2024-09-15T23:21:03.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f546a7fc640 1 -- 172.21.15.89:0/685664468 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5470018890 con 0x7f5474100450 2024-09-15T23:21:03.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.226+0000 7f547b6d7640 1 -- 172.21.15.89:0/685664468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54741a2350 con 0x7f5474100450 2024-09-15T23:21:03.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.227+0000 7f547b6d7640 1 -- 172.21.15.89:0/685664468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f547410b1a0 con 0x7f5474100450 2024-09-15T23:21:03.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.228+0000 7f546a7fc640 1 -- 172.21.15.89:0/685664468 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5470018a30 con 0x7f5474100450 2024-09-15T23:21:03.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.229+0000 7f546a7fc640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54600a3270 0x7f54600a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:03.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.229+0000 7f5478c72640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54600a3270 0x7f54600a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:03.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.229+0000 7f546a7fc640 1 -- 172.21.15.89:0/685664468 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f54700cf590 con 0x7f5474100450 2024-09-15T23:21:03.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.229+0000 7f5478c72640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54600a3270 0x7f54600a5730 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f54640314e0 tx=0x7f5464000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:03.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.233+0000 7f546a7fc640 1 -- 172.21.15.89:0/685664468 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5470014030 con 0x7f5474100450 2024-09-15T23:21:03.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.398+0000 7f547b6d7640 1 -- 172.21.15.89:0/685664468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f547402da20 con 0x7f5474100450 2024-09-15T23:21:03.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.399+0000 7f546a7fc640 1 -- 172.21.15.89:0/685664468 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) ==== 74+0+133 (secure 0 0 0) 0x7f547009c750 con 0x7f5474100450 2024-09-15T23:21:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:03.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 -- 172.21.15.89:0/685664468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54600a3270 msgr2=0x7f54600a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:03.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54600a3270 0x7f54600a5730 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f54640314e0 tx=0x7f5464000bc0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 -- 172.21.15.89:0/685664468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5474100450 msgr2=0x7f5474199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:03.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5474100450 0x7f5474199fd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f547000c920 tx=0x7f547000cde0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 -- 172.21.15.89:0/685664468 shutdown_connections 2024-09-15T23:21:03.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54600a3270 0x7f54600a5730 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5474101730 0x7f54741a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5474100d90 0x7f547419a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 --2- 172.21.15.89:0/685664468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5474100450 0x7f5474199fd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:03.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 -- 172.21.15.89:0/685664468 >> 172.21.15.89:0/685664468 conn(0x7f54740fc280 msgr2=0x7f54740fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:03.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 -- 172.21.15.89:0/685664468 shutdown_connections 2024-09-15T23:21:03.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:03.401+0000 7f547b6d7640 1 -- 172.21.15.89:0/685664468 wait complete. 2024-09-15T23:21:04.069 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":75,"num_osds":12,"num_up_osds":12,"osd_up_since":1726442461,"num_in_osds":12,"osd_in_since":1726442438,"num_remapped_pgs":0} 2024-09-15T23:21:04.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:21:04.092 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:03 smithi089 ceph-mon[30933]: osdmap e75: 12 total, 12 up, 12 in 2024-09-15T23:21:04.092 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:03 smithi089 ceph-mon[30933]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:04.092 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:03 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/685664468' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:21:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:03 smithi120 ceph-mon[35250]: osdmap e75: 12 total, 12 up, 12 in 2024-09-15T23:21:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:03 smithi120 ceph-mon[35250]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:03 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/685664468' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:21:04.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:03 smithi174 ceph-mon[35586]: osdmap e75: 12 total, 12 up, 12 in 2024-09-15T23:21:04.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:03 smithi174 ceph-mon[35586]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:04.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:03 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/685664468' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:21:04.313 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/1788096498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a010c180 msgr2=0x7fe7a010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 --2- 172.21.15.89:0/1788096498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a010c180 0x7fe7a010e600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe798009ca0 tx=0x7fe79802a950 comp rx=0 tx=0).stop 2024-09-15T23:21:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/1788096498 shutdown_connections 2024-09-15T23:21:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 --2- 172.21.15.89:0/1788096498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a010c180 0x7fe7a010e600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 --2- 172.21.15.89:0/1788096498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7a01036c0 0x7fe7a010bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 --2- 172.21.15.89:0/1788096498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a0102da0 0x7fe7a0103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/1788096498 >> 172.21.15.89:0/1788096498 conn(0x7fe7a00fc7a0 msgr2=0x7fe7a00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/1788096498 shutdown_connections 2024-09-15T23:21:04.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/1788096498 wait complete. 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 Processor -- start 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.791+0000 7fe7a7ff5640 1 -- start start 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a7ff5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7a0102da0 0x7fe7a0198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a7ff5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a01036c0 0x7fe7a01989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a7ff5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a010c180 0x7fe7a019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a7ff5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a0111fc0 con 0x7fe7a01036c0 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a7ff5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a0111e40 con 0x7fe7a010c180 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a7ff5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a0112140 con 0x7fe7a0102da0 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5590640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a01036c0 0x7fe7a01989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5590640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a01036c0 0x7fe7a01989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42880/0 (socket says 172.21.15.89:42880) 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5590640 1 -- 172.21.15.89:0/424908273 learned_addr learned my addr 172.21.15.89:0/424908273 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:04.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5d91640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7a0102da0 0x7fe7a0198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a6592640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a010c180 0x7fe7a019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5590640 1 -- 172.21.15.89:0/424908273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7a0102da0 msgr2=0x7fe7a0198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5590640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7a0102da0 0x7fe7a0198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5590640 1 -- 172.21.15.89:0/424908273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a010c180 msgr2=0x7fe7a019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5590640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a010c180 0x7fe7a019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5590640 1 -- 172.21.15.89:0/424908273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a019d410 con 0x7fe7a01036c0 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5d91640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7a0102da0 0x7fe7a0198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a6592640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a010c180 0x7fe7a019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a5590640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a01036c0 0x7fe7a01989a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe79000c970 tx=0x7fe79000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe796ffd640 1 -- 172.21.15.89:0/424908273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790007bb0 con 0x7fe7a01036c0 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe796ffd640 1 -- 172.21.15.89:0/424908273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe790007d50 con 0x7fe7a01036c0 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/424908273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a019d700 con 0x7fe7a01036c0 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.792+0000 7fe796ffd640 1 -- 172.21.15.89:0/424908273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79000f450 con 0x7fe7a01036c0 2024-09-15T23:21:04.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.793+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/424908273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a01a4fe0 con 0x7fe7a01036c0 2024-09-15T23:21:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.793+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/424908273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7a0103180 con 0x7fe7a01036c0 2024-09-15T23:21:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.794+0000 7fe796ffd640 1 -- 172.21.15.89:0/424908273 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe790004fe0 con 0x7fe7a01036c0 2024-09-15T23:21:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.795+0000 7fe796ffd640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe78c0a32c0 0x7fe78c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.796+0000 7fe7a5d91640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe78c0a32c0 0x7fe78c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.796+0000 7fe796ffd640 1 -- 172.21.15.89:0/424908273 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fe7900cf710 con 0x7fe7a01036c0 2024-09-15T23:21:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.796+0000 7fe7a5d91640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe78c0a32c0 0x7fe78c0a5780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe79c006fd0 tx=0x7fe79c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:04.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.799+0000 7fe796ffd640 1 -- 172.21.15.89:0/424908273 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe790014030 con 0x7fe7a01036c0 2024-09-15T23:21:04.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.965+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/424908273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe7a01997c0 con 0x7fe7a01036c0 2024-09-15T23:21:04.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.966+0000 7fe796ffd640 1 -- 172.21.15.89:0/424908273 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+18455 (secure 0 0 0) 0x7fe7a01997c0 con 0x7fe7a01036c0 2024-09-15T23:21:04.968 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:04.968 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":75,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:21:02.832036+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:21:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/424908273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe78c0a32c0 msgr2=0x7fe78c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe78c0a32c0 0x7fe78c0a5780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe79c006fd0 tx=0x7fe79c008040 comp rx=0 tx=0).stop 2024-09-15T23:21:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/424908273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a01036c0 msgr2=0x7fe7a01989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a01036c0 0x7fe7a01989a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe79000c970 tx=0x7fe79000ce30 comp rx=0 tx=0).stop 2024-09-15T23:21:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/424908273 shutdown_connections 2024-09-15T23:21:04.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe78c0a32c0 0x7fe78c0a5780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:04.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7a010c180 0x7fe7a019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:04.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7a01036c0 0x7fe7a01989a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:04.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 --2- 172.21.15.89:0/424908273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7a0102da0 0x7fe7a0198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:04.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/424908273 >> 172.21.15.89:0/424908273 conn(0x7fe7a00fc7a0 msgr2=0x7fe7a00fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:04.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/424908273 shutdown_connections 2024-09-15T23:21:04.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:04.968+0000 7fe7a7ff5640 1 -- 172.21.15.89:0/424908273 wait complete. 2024-09-15T23:21:05.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:05 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/424908273' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:21:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:05 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/424908273' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:21:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:05 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/424908273' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:21:05.751 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T23:17:15.186397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-15T23:21:05.751 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-15T23:21:06.039 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:06 smithi089 ceph-mon[30933]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:06 smithi120 ceph-mon[35250]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:06 smithi174 ceph-mon[35586]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:06.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.535+0000 7fdb7593f640 1 -- 172.21.15.89:0/1009416414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb70102790 msgr2=0x7fdb701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:06.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.535+0000 7fdb7593f640 1 --2- 172.21.15.89:0/1009416414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb70102790 0x7fdb701067e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdb64009900 tx=0x7fdb6402a410 comp rx=0 tx=0).stop 2024-09-15T23:21:06.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.535+0000 7fdb7593f640 1 -- 172.21.15.89:0/1009416414 shutdown_connections 2024-09-15T23:21:06.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.535+0000 7fdb7593f640 1 --2- 172.21.15.89:0/1009416414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb70106f10 0x7fdb7010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:06.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.535+0000 7fdb7593f640 1 --2- 172.21.15.89:0/1009416414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb70102790 0x7fdb701067e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:06.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.535+0000 7fdb7593f640 1 --2- 172.21.15.89:0/1009416414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb70101de0 0x7fdb701021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:06.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.535+0000 7fdb7593f640 1 -- 172.21.15.89:0/1009416414 >> 172.21.15.89:0/1009416414 conn(0x7fdb700fd8c0 msgr2=0x7fdb700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:06.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.535+0000 7fdb7593f640 1 -- 172.21.15.89:0/1009416414 shutdown_connections 2024-09-15T23:21:06.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.535+0000 7fdb7593f640 1 -- 172.21.15.89:0/1009416414 wait complete. 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb7593f640 1 Processor -- start 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb7593f640 1 -- start start 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb7593f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb70101de0 0x7fdb70198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb7593f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb70102790 0x7fdb701989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb7593f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb70106f10 0x7fdb7019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb7593f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdb70111f90 con 0x7fdb70106f10 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb7593f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdb70111e10 con 0x7fdb70102790 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb7593f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdb70112110 con 0x7fdb70101de0 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb6f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb70106f10 0x7fdb7019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb6f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb70106f10 0x7fdb7019cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33332/0 (socket says 172.21.15.89:33332) 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb6f7fe640 1 -- 172.21.15.89:0/2056443498 learned_addr learned my addr 172.21.15.89:0/2056443498 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb6effd640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb70101de0 0x7fdb70198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:06.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb6e7fc640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb70102790 0x7fdb701989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb6f7fe640 1 -- 172.21.15.89:0/2056443498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb70101de0 msgr2=0x7fdb70198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb6f7fe640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb70101de0 0x7fdb70198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb6f7fe640 1 -- 172.21.15.89:0/2056443498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb70102790 msgr2=0x7fdb701989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb6f7fe640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb70102790 0x7fdb701989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.536+0000 7fdb6f7fe640 1 -- 172.21.15.89:0/2056443498 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7019d420 con 0x7fdb70106f10 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.537+0000 7fdb6f7fe640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb70106f10 0x7fdb7019cd40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdb6800b970 tx=0x7fdb6800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.537+0000 7fdb7493d640 1 -- 172.21.15.89:0/2056443498 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68019070 con 0x7fdb70106f10 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.537+0000 7fdb7593f640 1 -- 172.21.15.89:0/2056443498 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7019d710 con 0x7fdb70106f10 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.537+0000 7fdb7493d640 1 -- 172.21.15.89:0/2056443498 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb6800cb40 con 0x7fdb70106f10 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.537+0000 7fdb7493d640 1 -- 172.21.15.89:0/2056443498 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68007610 con 0x7fdb70106f10 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.537+0000 7fdb7593f640 1 -- 172.21.15.89:0/2056443498 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb701a4ff0 con 0x7fdb70106f10 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.538+0000 7fdb7593f640 1 -- 172.21.15.89:0/2056443498 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb701021c0 con 0x7fdb70106f10 2024-09-15T23:21:06.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.539+0000 7fdb7493d640 1 -- 172.21.15.89:0/2056443498 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fdb6801c070 con 0x7fdb70106f10 2024-09-15T23:21:06.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.539+0000 7fdb7493d640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdb580a3240 0x7fdb580a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:06.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.540+0000 7fdb6effd640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdb580a3240 0x7fdb580a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:06.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.540+0000 7fdb7493d640 1 -- 172.21.15.89:0/2056443498 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fdb680ceab0 con 0x7fdb70106f10 2024-09-15T23:21:06.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.540+0000 7fdb6effd640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdb580a3240 0x7fdb580a5700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdb600058c0 tx=0x7fdb6000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:06.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.544+0000 7fdb7493d640 1 -- 172.21.15.89:0/2056443498 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdb6809bc70 con 0x7fdb70106f10 2024-09-15T23:21:06.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.711+0000 7fdb7593f640 1 -- 172.21.15.89:0/2056443498 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdb7010a0f0 con 0x7fdb70106f10 2024-09-15T23:21:06.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.711+0000 7fdb7493d640 1 -- 172.21.15.89:0/2056443498 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) ==== 93+0+10 (secure 0 0 0) 0x7fdb680bb070 con 0x7fdb70106f10 2024-09-15T23:21:06.714 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.713+0000 7fdb7593f640 1 -- 172.21.15.89:0/2056443498 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdb580a3240 msgr2=0x7fdb580a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.713+0000 7fdb7593f640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdb580a3240 0x7fdb580a5700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdb600058c0 tx=0x7fdb6000a4c0 comp rx=0 tx=0).stop 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.713+0000 7fdb7593f640 1 -- 172.21.15.89:0/2056443498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb70106f10 msgr2=0x7fdb7019cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.713+0000 7fdb7593f640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb70106f10 0x7fdb7019cd40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdb6800b970 tx=0x7fdb6800be30 comp rx=0 tx=0).stop 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.713+0000 7fdb7593f640 1 -- 172.21.15.89:0/2056443498 shutdown_connections 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.713+0000 7fdb7593f640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdb580a3240 0x7fdb580a5700 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.713+0000 7fdb7593f640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdb70106f10 0x7fdb7019cd40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.713+0000 7fdb7593f640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb70102790 0x7fdb701989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.713+0000 7fdb7593f640 1 --2- 172.21.15.89:0/2056443498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb70101de0 0x7fdb70198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.713+0000 7fdb7593f640 1 -- 172.21.15.89:0/2056443498 >> 172.21.15.89:0/2056443498 conn(0x7fdb700fd8c0 msgr2=0x7fdb700fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:06.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.714+0000 7fdb7593f640 1 -- 172.21.15.89:0/2056443498 shutdown_connections 2024-09-15T23:21:06.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:06.714+0000 7fdb7593f640 1 -- 172.21.15.89:0/2056443498 wait complete. 2024-09-15T23:21:07.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:07 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2056443498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:21:07.465 INFO:tasks.cephadm:Adding mds.a on smithi089 2024-09-15T23:21:07.465 INFO:tasks.cephadm:Adding mds.d on smithi089 2024-09-15T23:21:07.465 INFO:tasks.cephadm:Adding mds.g on smithi089 2024-09-15T23:21:07.465 INFO:tasks.cephadm:Adding mds.j on smithi089 2024-09-15T23:21:07.465 INFO:tasks.cephadm:Adding mds.b on smithi120 2024-09-15T23:21:07.465 INFO:tasks.cephadm:Adding mds.e on smithi120 2024-09-15T23:21:07.465 INFO:tasks.cephadm:Adding mds.h on smithi120 2024-09-15T23:21:07.466 INFO:tasks.cephadm:Adding mds.k on smithi120 2024-09-15T23:21:07.466 INFO:tasks.cephadm:Adding mds.c on smithi174 2024-09-15T23:21:07.466 INFO:tasks.cephadm:Adding mds.f on smithi174 2024-09-15T23:21:07.466 INFO:tasks.cephadm:Adding mds.i on smithi174 2024-09-15T23:21:07.466 INFO:tasks.cephadm:Adding mds.l on smithi174 2024-09-15T23:21:07.466 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph orch apply mds all '12;smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l' 2024-09-15T23:21:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:07 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2056443498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:21:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:07 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2056443498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:21:07.724 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:21:08.121 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:08 smithi174 ceph-mon[35586]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:08.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.205+0000 7f2896b02640 1 -- 172.21.15.174:0/3109649494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2890101cd0 msgr2=0x7f2890102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:08.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.205+0000 7f2896b02640 1 --2- 172.21.15.174:0/3109649494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2890101cd0 0x7f2890102150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f287c009900 tx=0x7f287c02a410 comp rx=0 tx=0).stop 2024-09-15T23:21:08.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.205+0000 7f2896b02640 1 -- 172.21.15.174:0/3109649494 shutdown_connections 2024-09-15T23:21:08.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.205+0000 7f2896b02640 1 --2- 172.21.15.174:0/3109649494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2890102690 0x7f2890108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:08.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.205+0000 7f2896b02640 1 --2- 172.21.15.174:0/3109649494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2890101cd0 0x7f2890102150 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:08.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.205+0000 7f2896b02640 1 --2- 172.21.15.174:0/3109649494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2890100ad0 0x7f2890100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:08.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.205+0000 7f2896b02640 1 -- 172.21.15.174:0/3109649494 >> 172.21.15.174:0/3109649494 conn(0x7f28900fc280 msgr2=0x7f28900fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:08.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.205+0000 7f2896b02640 1 -- 172.21.15.174:0/3109649494 shutdown_connections 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.205+0000 7f2896b02640 1 -- 172.21.15.174:0/3109649494 wait complete. 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f2896b02640 1 Processor -- start 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f2896b02640 1 -- start start 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f2896b02640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2890100ad0 0x7f2890197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f2896b02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2890101cd0 0x7f2890198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f2896b02640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2890102690 0x7f289019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f2896b02640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f289010c780 con 0x7f2890101cd0 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f2896b02640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f289010c600 con 0x7f2890100ad0 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f2896b02640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f289010c900 con 0x7f2890102690 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f289509f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2890102690 0x7f289019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f289509f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2890102690 0x7f289019f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:42744/0 (socket says 172.21.15.174:42744) 2024-09-15T23:21:08.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f289509f640 1 -- 172.21.15.174:0/3351526369 learned_addr learned my addr 172.21.15.174:0/3351526369 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f289489e640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2890100ad0 0x7f2890197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f289509f640 1 -- 172.21.15.174:0/3351526369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2890100ad0 msgr2=0x7f2890197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f289509f640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2890100ad0 0x7f2890197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f289509f640 1 -- 172.21.15.174:0/3351526369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2890101cd0 msgr2=0x7f2890198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f289509f640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2890101cd0 0x7f2890198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.206+0000 7f289509f640 1 -- 172.21.15.174:0/3351526369 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f289019fb10 con 0x7f2890102690 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.207+0000 7f289509f640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2890102690 0x7f289019f4a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f288800eaa0 tx=0x7f288800ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.207+0000 7f288dffb640 1 -- 172.21.15.174:0/3351526369 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288800cdb0 con 0x7f2890102690 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.207+0000 7f2896b02640 1 -- 172.21.15.174:0/3351526369 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f289019fe00 con 0x7f2890102690 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.207+0000 7f288dffb640 1 -- 172.21.15.174:0/3351526369 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2888005020 con 0x7f2890102690 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.207+0000 7f288dffb640 1 -- 172.21.15.174:0/3351526369 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28880116b0 con 0x7f2890102690 2024-09-15T23:21:08.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.207+0000 7f2896b02640 1 -- 172.21.15.174:0/3351526369 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28901a0310 con 0x7f2890102690 2024-09-15T23:21:08.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.208+0000 7f2896b02640 1 -- 172.21.15.174:0/3351526369 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28901a05a0 con 0x7f2890102690 2024-09-15T23:21:08.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.209+0000 7f288dffb640 1 -- 172.21.15.174:0/3351526369 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2888005560 con 0x7f2890102690 2024-09-15T23:21:08.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.210+0000 7f288dffb640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28800a32f0 0x7f28800a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:08.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.210+0000 7f288dffb640 1 -- 172.21.15.174:0/3351526369 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f28880cf9c0 con 0x7f2890102690 2024-09-15T23:21:08.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.210+0000 7f289489e640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28800a32f0 0x7f28800a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:08.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.211+0000 7f289489e640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28800a32f0 0x7f28800a57b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2884005fd0 tx=0x7f288400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:08.216 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.213+0000 7f288dffb640 1 -- 172.21.15.174:0/3351526369 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2888016030 con 0x7f2890102690 2024-09-15T23:21:08.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:08 smithi089 ceph-mon[30933]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:08.393 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.392+0000 7f2896b02640 1 -- 172.21.15.174:0/3351526369 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}) -- 0x7f289006a680 con 0x7f28800a32f0 2024-09-15T23:21:08.401 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.400+0000 7f288dffb640 1 -- 172.21.15.174:0/3351526369 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f289006a680 con 0x7f28800a32f0 2024-09-15T23:21:08.401 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mds.all update... 2024-09-15T23:21:08.404 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 -- 172.21.15.174:0/3351526369 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28800a32f0 msgr2=0x7f28800a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:08.404 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28800a32f0 0x7f28800a57b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2884005fd0 tx=0x7f288400b410 comp rx=0 tx=0).stop 2024-09-15T23:21:08.404 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 -- 172.21.15.174:0/3351526369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2890102690 msgr2=0x7f289019f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:08.404 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2890102690 0x7f289019f4a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f288800eaa0 tx=0x7f288800ef60 comp rx=0 tx=0).stop 2024-09-15T23:21:08.404 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 -- 172.21.15.174:0/3351526369 shutdown_connections 2024-09-15T23:21:08.404 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28800a32f0 0x7f28800a57b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:08.404 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2890102690 0x7f289019f4a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:08.404 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2890101cd0 0x7f2890198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:08.405 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 --2- 172.21.15.174:0/3351526369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2890100ad0 0x7f2890197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:08.405 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 -- 172.21.15.174:0/3351526369 >> 172.21.15.174:0/3351526369 conn(0x7f28900fc280 msgr2=0x7f2890069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:08.405 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 -- 172.21.15.174:0/3351526369 shutdown_connections 2024-09-15T23:21:08.405 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-15T23:21:08.402+0000 7f2896b02640 1 -- 172.21.15.174:0/3351526369 wait complete. 2024-09-15T23:21:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:08 smithi120 ceph-mon[35250]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:09.162 DEBUG:teuthology.orchestra.run.smithi089:mds.a> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.a.service 2024-09-15T23:21:09.164 DEBUG:teuthology.orchestra.run.smithi089:mds.d> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.d.service 2024-09-15T23:21:09.206 DEBUG:teuthology.orchestra.run.smithi089:mds.g> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.g.service 2024-09-15T23:21:09.251 DEBUG:teuthology.orchestra.run.smithi089:mds.j> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.j.service 2024-09-15T23:21:09.293 DEBUG:teuthology.orchestra.run.smithi120:mds.b> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.b.service 2024-09-15T23:21:09.296 DEBUG:teuthology.orchestra.run.smithi120:mds.e> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.e.service 2024-09-15T23:21:09.341 DEBUG:teuthology.orchestra.run.smithi120:mds.h> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.h.service 2024-09-15T23:21:09.384 DEBUG:teuthology.orchestra.run.smithi120:mds.k> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.k.service 2024-09-15T23:21:09.428 DEBUG:teuthology.orchestra.run.smithi174:mds.c> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.c.service 2024-09-15T23:21:09.431 DEBUG:teuthology.orchestra.run.smithi174:mds.f> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.f.service 2024-09-15T23:21:09.474 DEBUG:teuthology.orchestra.run.smithi174:mds.i> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.i.service 2024-09-15T23:21:09.517 DEBUG:teuthology.orchestra.run.smithi174:mds.l> sudo journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.l.service 2024-09-15T23:21:09.559 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-15T23:21:09.560 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:09.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:21:09.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: Saving service mds.all spec with placement smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-09-15T23:21:09.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:09.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:09.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:09.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:09.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:09.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:09.582 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:09.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:09.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: Deploying daemon mds.l on smithi174 2024-09-15T23:21:09.583 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:09 smithi089 ceph-mon[30933]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:09.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:21:09.698 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: Saving service mds.all spec with placement smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-09-15T23:21:09.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:09.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:09.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:09.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:09.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:09.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:09.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:09.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:09.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: Deploying daemon mds.l on smithi174 2024-09-15T23:21:09.699 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:09 smithi174 ceph-mon[35586]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:21:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: Saving service mds.all spec with placement smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-09-15T23:21:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:09.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:09.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:09.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:09.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: Deploying daemon mds.l on smithi174 2024-09-15T23:21:09.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:09 smithi120 ceph-mon[35250]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:09.801 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.294+0000 7fd05e866640 1 -- 172.21.15.89:0/975536315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd058102de0 msgr2=0x7fd0581031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.294+0000 7fd05e866640 1 --2- 172.21.15.89:0/975536315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd058102de0 0x7fd0581031c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd048009900 tx=0x7fd04802a480 comp rx=0 tx=0).stop 2024-09-15T23:21:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.294+0000 7fd05e866640 1 -- 172.21.15.89:0/975536315 shutdown_connections 2024-09-15T23:21:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.294+0000 7fd05e866640 1 --2- 172.21.15.89:0/975536315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd05810c180 0x7fd05810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.294+0000 7fd05e866640 1 --2- 172.21.15.89:0/975536315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd058103700 0x7fd05810bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.294+0000 7fd05e866640 1 --2- 172.21.15.89:0/975536315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd058102de0 0x7fd0581031c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.294+0000 7fd05e866640 1 -- 172.21.15.89:0/975536315 >> 172.21.15.89:0/975536315 conn(0x7fd0580fc7a0 msgr2=0x7fd0580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.294+0000 7fd05e866640 1 -- 172.21.15.89:0/975536315 shutdown_connections 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.294+0000 7fd05e866640 1 -- 172.21.15.89:0/975536315 wait complete. 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.295+0000 7fd05e866640 1 Processor -- start 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.295+0000 7fd05e866640 1 -- start start 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.295+0000 7fd05e866640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd058102de0 0x7fd058100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.295+0000 7fd05e866640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd058103700 0x7fd058100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.295+0000 7fd05e866640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05810c180 0x7fd0581014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.295+0000 7fd05e866640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd058111f30 con 0x7fd05810c180 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.295+0000 7fd05e866640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd058111db0 con 0x7fd058102de0 2024-09-15T23:21:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.295+0000 7fd05e866640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581120b0 con 0x7fd058103700 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05ce03640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05810c180 0x7fd0581014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05ce03640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05810c180 0x7fd0581014c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33358/0 (socket says 172.21.15.89:33358) 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05ce03640 1 -- 172.21.15.89:0/64544591 learned_addr learned my addr 172.21.15.89:0/64544591 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd0577fe640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd058103700 0x7fd058100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd057fff640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd058102de0 0x7fd058100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05ce03640 1 -- 172.21.15.89:0/64544591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd058103700 msgr2=0x7fd058100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05ce03640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd058103700 0x7fd058100f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05ce03640 1 -- 172.21.15.89:0/64544591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd058102de0 msgr2=0x7fd058100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05ce03640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd058102de0 0x7fd058100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05ce03640 1 -- 172.21.15.89:0/64544591 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0581aae20 con 0x7fd05810c180 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd057fff640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd058102de0 0x7fd058100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05ce03640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05810c180 0x7fd0581014c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd05000cad0 tx=0x7fd05000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd0557fa640 1 -- 172.21.15.89:0/64544591 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05000ae20 con 0x7fd05810c180 2024-09-15T23:21:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd0557fa640 1 -- 172.21.15.89:0/64544591 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd050005020 con 0x7fd05810c180 2024-09-15T23:21:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd0557fa640 1 -- 172.21.15.89:0/64544591 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05000f6f0 con 0x7fd05810c180 2024-09-15T23:21:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05e866640 1 -- 172.21.15.89:0/64544591 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0581aaff0 con 0x7fd05810c180 2024-09-15T23:21:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.296+0000 7fd05e866640 1 -- 172.21.15.89:0/64544591 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0581ab4b0 con 0x7fd05810c180 2024-09-15T23:21:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.297+0000 7fd05e866640 1 -- 172.21.15.89:0/64544591 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd05806afd0 con 0x7fd05810c180 2024-09-15T23:21:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.298+0000 7fd0557fa640 1 -- 172.21.15.89:0/64544591 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd0500051d0 con 0x7fd05810c180 2024-09-15T23:21:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.299+0000 7fd0557fa640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a32c0 0x7fd0440a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.299+0000 7fd057fff640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a32c0 0x7fd0440a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.300+0000 7fd0557fa640 1 -- 172.21.15.89:0/64544591 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fd0500cf8a0 con 0x7fd05810c180 2024-09-15T23:21:10.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.300+0000 7fd057fff640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a32c0 0x7fd0440a5780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd048005ab0 tx=0x7fd048005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:10.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.303+0000 7fd0557fa640 1 -- 172.21.15.89:0/64544591 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd050014030 con 0x7fd05810c180 2024-09-15T23:21:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.512+0000 7fd05e866640 1 -- 172.21.15.89:0/64544591 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0581031c0 con 0x7fd05810c180 2024-09-15T23:21:10.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.512+0000 7fd0557fa640 1 -- 172.21.15.89:0/64544591 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7fd05009ca40 con 0x7fd05810c180 2024-09-15T23:21:10.514 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:10.514 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"btime":"2024-09-15T23:14:08:706210+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-15T23:21:10.514 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 1 2024-09-15T23:21:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 -- 172.21.15.89:0/64544591 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a32c0 msgr2=0x7fd0440a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a32c0 0x7fd0440a5780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd048005ab0 tx=0x7fd048005a20 comp rx=0 tx=0).stop 2024-09-15T23:21:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 -- 172.21.15.89:0/64544591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05810c180 msgr2=0x7fd0581014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05810c180 0x7fd0581014c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd05000cad0 tx=0x7fd05000cf90 comp rx=0 tx=0).stop 2024-09-15T23:21:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 -- 172.21.15.89:0/64544591 shutdown_connections 2024-09-15T23:21:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a32c0 0x7fd0440a5780 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05810c180 0x7fd0581014c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd058103700 0x7fd058100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 --2- 172.21.15.89:0/64544591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd058102de0 0x7fd058100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 -- 172.21.15.89:0/64544591 >> 172.21.15.89:0/64544591 conn(0x7fd0580fc7a0 msgr2=0x7fd0580fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 -- 172.21.15.89:0/64544591 shutdown_connections 2024-09-15T23:21:10.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:10.515+0000 7fd05e866640 1 -- 172.21.15.89:0/64544591 wait complete. 2024-09-15T23:21:10.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:10 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/64544591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:10.987 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:21:10 smithi174 systemd[1]: Starting Ceph mds.l for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:11.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:10 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/64544591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:11.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:10 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/64544591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:11.317 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:21:11 smithi174 podman[65206]: 2024-09-15 23:21:10.953044615 +0000 UTC m=+0.016024755 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:11.317 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:21:11 smithi174 podman[65206]: 2024-09-15 23:21:11.054692085 +0000 UTC m=+0.117672219 container create 61681fc05d8b109552d0c3bf8c495143e22698ac274a14b3c56322f03b38fa59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-8921c8e) 2024-09-15T23:21:11.317 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:21:11 smithi174 podman[65206]: 2024-09-15 23:21:11.290806738 +0000 UTC m=+0.353786869 container init 61681fc05d8b109552d0c3bf8c495143e22698ac274a14b3c56322f03b38fa59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-15T23:21:11.317 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:21:11 smithi174 podman[65206]: 2024-09-15 23:21:11.293616643 +0000 UTC m=+0.356596781 container start 61681fc05d8b109552d0c3bf8c495143e22698ac274a14b3c56322f03b38fa59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-15T23:21:11.317 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mds[65226]: -- 172.21.15.174:0/3506173151 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56111959b860 con 0x561119579400 2024-09-15T23:21:11.736 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l[65222]: starting mds.l at 2024-09-15T23:21:11.737 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:21:11 smithi174 bash[65206]: 61681fc05d8b109552d0c3bf8c495143e22698ac274a14b3c56322f03b38fa59 2024-09-15T23:21:11.737 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:21:11 smithi174 systemd[1]: Started Ceph mds.l for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:12.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:11 smithi120 ceph-mon[35250]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:12.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:11 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6832/2504210581,v1:172.21.15.174:6833/2504210581] up:boot 2024-09-15T23:21:12.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:11 smithi120 ceph-mon[35250]: fsmap 1 up:standby 2024-09-15T23:21:12.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T23:21:12.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:12.147 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:12.148 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:12.148 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:12.148 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:12.148 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:11 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:12.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mon[35586]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:12.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6832/2504210581,v1:172.21.15.174:6833/2504210581] up:boot 2024-09-15T23:21:12.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mon[35586]: fsmap 1 up:standby 2024-09-15T23:21:12.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T23:21:12.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:12.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:12.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:12.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:12.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:12.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:11 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:12.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:11 smithi089 ceph-mon[30933]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:12.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:11 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6832/2504210581,v1:172.21.15.174:6833/2504210581] up:boot 2024-09-15T23:21:12.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:11 smithi089 ceph-mon[30933]: fsmap 1 up:standby 2024-09-15T23:21:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T23:21:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:12.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:11 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:13.085 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:12 smithi120 ceph-mon[35250]: Deploying daemon mds.k on smithi120 2024-09-15T23:21:13.138 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:12 smithi174 ceph-mon[35586]: Deploying daemon mds.k on smithi120 2024-09-15T23:21:13.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:12 smithi089 ceph-mon[30933]: Deploying daemon mds.k on smithi120 2024-09-15T23:21:13.358 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:13.446 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:21:13 smithi120 systemd[1]: Starting Ceph mds.k for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:13.613 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:13.723 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:21:13 smithi120 podman[64257]: 2024-09-15 23:21:13.558971916 +0000 UTC m=+0.015687602 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:13.724 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:21:13 smithi120 podman[64257]: 2024-09-15 23:21:13.688271597 +0000 UTC m=+0.144987276 container create 8429948d1f28c1ff791272ef3c8abfbb53722252d651bc31da208560efdd7925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k, ceph=True, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-15T23:21:14.017 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:13 smithi089 ceph-mon[30933]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:14.092 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:13 smithi120 ceph-mon[35250]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:14.092 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:21:13 smithi120 podman[64257]: 2024-09-15 23:21:13.991788414 +0000 UTC m=+0.448504094 container init 8429948d1f28c1ff791272ef3c8abfbb53722252d651bc31da208560efdd7925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True) 2024-09-15T23:21:14.092 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:21:13 smithi120 podman[64257]: 2024-09-15 23:21:13.994728465 +0000 UTC m=+0.451444144 container start 8429948d1f28c1ff791272ef3c8abfbb53722252d651bc31da208560efdd7925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-15T23:21:14.092 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:21:14 smithi120 ceph-mds[64276]: -- 172.21.15.120:0/3664252925 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55df9962d860 con 0x55df9960a800 2024-09-15T23:21:14.093 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:21:14 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k[64272]: starting mds.k at 2024-09-15T23:21:14.093 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:21:14 smithi120 bash[64257]: 8429948d1f28c1ff791272ef3c8abfbb53722252d651bc31da208560efdd7925 2024-09-15T23:21:14.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 -- 172.21.15.89:0/2924463989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f957c102ef0 msgr2=0x7f957c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:14.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 --2- 172.21.15.89:0/2924463989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f957c102ef0 0x7f957c10da40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9574009ca0 tx=0x7f957402a950 comp rx=0 tx=0).stop 2024-09-15T23:21:14.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 -- 172.21.15.89:0/2924463989 shutdown_connections 2024-09-15T23:21:14.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 --2- 172.21.15.89:0/2924463989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f957c102ef0 0x7f957c10da40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:14.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 --2- 172.21.15.89:0/2924463989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f957c102570 0x7f957c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:14.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 --2- 172.21.15.89:0/2924463989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f957c100e20 0x7f957c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:14.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 -- 172.21.15.89:0/2924463989 >> 172.21.15.89:0/2924463989 conn(0x7f957c0fc7d0 msgr2=0x7f957c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:14.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 -- 172.21.15.89:0/2924463989 shutdown_connections 2024-09-15T23:21:14.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 -- 172.21.15.89:0/2924463989 wait complete. 2024-09-15T23:21:14.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 Processor -- start 2024-09-15T23:21:14.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.095+0000 7f958472f640 1 -- start start 2024-09-15T23:21:14.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f958472f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f957c100e20 0x7f957c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:14.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f958472f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f957c102570 0x7f957c198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:14.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f958472f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f957c102ef0 0x7f957c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:14.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f958472f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f957c1120a0 con 0x7f957c100e20 2024-09-15T23:21:14.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f958472f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f957c111f20 con 0x7f957c102ef0 2024-09-15T23:21:14.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f958472f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f957c112220 con 0x7f957c102570 2024-09-15T23:21:14.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95824cb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f957c100e20 0x7f957c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:14.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95824cb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f957c100e20 0x7f957c1984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33380/0 (socket says 172.21.15.89:33380) 2024-09-15T23:21:14.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95824cb640 1 -- 172.21.15.89:0/787868493 learned_addr learned my addr 172.21.15.89:0/787868493 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:14.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f9582ccc640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f957c102ef0 0x7f957c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:14.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f9581cca640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f957c102570 0x7f957c198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:14.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95824cb640 1 -- 172.21.15.89:0/787868493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f957c102570 msgr2=0x7f957c198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:14.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95824cb640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f957c102570 0x7f957c198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:14.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95824cb640 1 -- 172.21.15.89:0/787868493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f957c102ef0 msgr2=0x7f957c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:14.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95824cb640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f957c102ef0 0x7f957c19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:14.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95824cb640 1 -- 172.21.15.89:0/787868493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957c19d490 con 0x7f957c100e20 2024-09-15T23:21:14.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f9581cca640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f957c102570 0x7f957c198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:14.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95824cb640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f957c100e20 0x7f957c1984e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f957800e9f0 tx=0x7f957800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:14.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95737fe640 1 -- 172.21.15.89:0/787868493 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957800cd80 con 0x7f957c100e20 2024-09-15T23:21:14.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95737fe640 1 -- 172.21.15.89:0/787868493 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9578010910 con 0x7f957c100e20 2024-09-15T23:21:14.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f958472f640 1 -- 172.21.15.89:0/787868493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957c19d780 con 0x7f957c100e20 2024-09-15T23:21:14.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f95737fe640 1 -- 172.21.15.89:0/787868493 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9578018990 con 0x7f957c100e20 2024-09-15T23:21:14.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.096+0000 7f958472f640 1 -- 172.21.15.89:0/787868493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f957c1a5060 con 0x7f957c100e20 2024-09-15T23:21:14.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.097+0000 7f958472f640 1 -- 172.21.15.89:0/787868493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f957c101200 con 0x7f957c100e20 2024-09-15T23:21:14.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.098+0000 7f95737fe640 1 -- 172.21.15.89:0/787868493 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f9578010ac0 con 0x7f957c100e20 2024-09-15T23:21:14.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.099+0000 7f95737fe640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f95680a32c0 0x7f95680a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:14.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.099+0000 7f9581cca640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f95680a32c0 0x7f95680a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:14.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.099+0000 7f95737fe640 1 -- 172.21.15.89:0/787868493 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f95780cfc50 con 0x7f957c100e20 2024-09-15T23:21:14.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.100+0000 7f9581cca640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f95680a32c0 0x7f95680a5780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f956c006fd0 tx=0x7f956c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:14.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.102+0000 7f95737fe640 1 -- 172.21.15.89:0/787868493 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f957809cd70 con 0x7f957c100e20 2024-09-15T23:21:14.151 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:13 smithi174 ceph-mon[35586]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:14.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.322+0000 7f958472f640 1 -- 172.21.15.89:0/787868493 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f957c199840 con 0x7f957c100e20 2024-09-15T23:21:14.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.322+0000 7f95737fe640 1 -- 172.21.15.89:0/787868493 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7f957809c670 con 0x7f957c100e20 2024-09-15T23:21:14.324 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:14.324 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":2,"btime":"2024-09-15T23:21:11:321608+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-15T23:21:14.324 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 2 2024-09-15T23:21:14.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 -- 172.21.15.89:0/787868493 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f95680a32c0 msgr2=0x7f95680a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:14.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f95680a32c0 0x7f95680a5780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f956c006fd0 tx=0x7f956c008040 comp rx=0 tx=0).stop 2024-09-15T23:21:14.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 -- 172.21.15.89:0/787868493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f957c100e20 msgr2=0x7f957c1984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:14.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f957c100e20 0x7f957c1984e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f957800e9f0 tx=0x7f957800eeb0 comp rx=0 tx=0).stop 2024-09-15T23:21:14.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 -- 172.21.15.89:0/787868493 shutdown_connections 2024-09-15T23:21:14.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f95680a32c0 0x7f95680a5780 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:14.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f957c102ef0 0x7f957c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:14.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f957c102570 0x7f957c198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:14.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 --2- 172.21.15.89:0/787868493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f957c100e20 0x7f957c1984e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:14.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 -- 172.21.15.89:0/787868493 >> 172.21.15.89:0/787868493 conn(0x7f957c0fc7d0 msgr2=0x7f957c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:14.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 -- 172.21.15.89:0/787868493 shutdown_connections 2024-09-15T23:21:14.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:14.326+0000 7f958472f640 1 -- 172.21.15.89:0/787868493 wait complete. 2024-09-15T23:21:14.473 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:21:14 smithi120 systemd[1]: Started Ceph mds.k for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:15.431 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:15.432 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:15.432 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:15.432 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:15.432 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:15.432 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:15.432 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: Deploying daemon mds.j on smithi089 2024-09-15T23:21:15.433 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/787868493' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:15.433 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.120:6832/4147766006,v1:172.21.15.120:6833/4147766006] up:boot 2024-09-15T23:21:15.433 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: fsmap 2 up:standby 2024-09-15T23:21:15.433 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T23:21:15.433 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:15 smithi089 ceph-mon[30933]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:15.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:15.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: Deploying daemon mds.j on smithi089 2024-09-15T23:21:15.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/787868493' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:15.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.120:6832/4147766006,v1:172.21.15.120:6833/4147766006] up:boot 2024-09-15T23:21:15.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: fsmap 2 up:standby 2024-09-15T23:21:15.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T23:21:15.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:15 smithi120 ceph-mon[35250]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: Deploying daemon mds.j on smithi089 2024-09-15T23:21:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/787868493' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.120:6832/4147766006,v1:172.21.15.120:6833/4147766006] up:boot 2024-09-15T23:21:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: fsmap 2 up:standby 2024-09-15T23:21:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T23:21:15.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:15 smithi174 ceph-mon[35586]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:17.058 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:21:16 smithi089 systemd[1]: Starting Ceph mds.j for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:17.108 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:17.310 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:21:17 smithi089 podman[69768]: 2024-09-15 23:21:16.959984624 +0000 UTC m=+0.015824434 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:17.310 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:21:17 smithi089 podman[69768]: 2024-09-15 23:21:17.07499018 +0000 UTC m=+0.130829988 container create 1c1226337f697cef385c0e316a9c13c539dc324f9feb9d9f95dd1c13c8aad683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-15T23:21:17.473 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:17.597 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:21:17 smithi089 podman[69768]: 2024-09-15 23:21:17.327982717 +0000 UTC m=+0.383822527 container init 1c1226337f697cef385c0e316a9c13c539dc324f9feb9d9f95dd1c13c8aad683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-j, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-15T23:21:17.598 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:21:17 smithi089 podman[69768]: 2024-09-15 23:21:17.33125767 +0000 UTC m=+0.387097482 container start 1c1226337f697cef385c0e316a9c13c539dc324f9feb9d9f95dd1c13c8aad683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-j, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-15T23:21:17.598 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:21:17 smithi089 ceph-mds[69818]: -- 172.21.15.89:0/3802380719 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561381bff860 con 0x561381bdc800 2024-09-15T23:21:17.598 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:21:17 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-j[69801]: starting mds.j at 2024-09-15T23:21:17.598 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:21:17 smithi089 bash[69768]: 1c1226337f697cef385c0e316a9c13c539dc324f9feb9d9f95dd1c13c8aad683 2024-09-15T23:21:17.598 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:21:17 smithi089 systemd[1]: Started Ceph mds.j for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:18.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 -- 172.21.15.89:0/1021771870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2028100a80 msgr2=0x7f2028100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:18.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 --2- 172.21.15.89:0/1021771870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2028100a80 0x7f2028100ee0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2014009930 tx=0x7f201402a420 comp rx=0 tx=0).stop 2024-09-15T23:21:18.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 -- 172.21.15.89:0/1021771870 shutdown_connections 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 --2- 172.21.15.89:0/1021771870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2028101420 0x7f202810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 --2- 172.21.15.89:0/1021771870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2028100a80 0x7f2028100ee0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 --2- 172.21.15.89:0/1021771870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2028106a80 0x7f2028106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 -- 172.21.15.89:0/1021771870 >> 172.21.15.89:0/1021771870 conn(0x7f20280fc7a0 msgr2=0x7f20280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 -- 172.21.15.89:0/1021771870 shutdown_connections 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 -- 172.21.15.89:0/1021771870 wait complete. 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 Processor -- start 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 -- start start 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2028100a80 0x7f2028198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2028101420 0x7f2028198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2028106a80 0x7f202819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:18.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2028110e80 con 0x7f2028106a80 2024-09-15T23:21:18.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2028110d00 con 0x7f2028101420 2024-09-15T23:21:18.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.996+0000 7f202ef40640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2028111000 con 0x7f2028100a80 2024-09-15T23:21:18.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202d4dd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2028106a80 0x7f202819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:18.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202d4dd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2028106a80 0x7f202819ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59906/0 (socket says 172.21.15.89:59906) 2024-09-15T23:21:18.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202d4dd640 1 -- 172.21.15.89:0/3189267453 learned_addr learned my addr 172.21.15.89:0/3189267453 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:18.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202ccdc640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2028100a80 0x7f2028198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:18.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f2027fff640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2028101420 0x7f2028198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:18.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202d4dd640 1 -- 172.21.15.89:0/3189267453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2028100a80 msgr2=0x7f2028198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:18.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202d4dd640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2028100a80 0x7f2028198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:18.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202d4dd640 1 -- 172.21.15.89:0/3189267453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2028101420 msgr2=0x7f2028198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202d4dd640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2028101420 0x7f2028198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202d4dd640 1 -- 172.21.15.89:0/3189267453 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202819d3d0 con 0x7f2028106a80 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202d4dd640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2028106a80 0x7f202819ccf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f202000eaa0 tx=0x7f202000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f2025ffb640 1 -- 172.21.15.89:0/3189267453 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202000cdb0 con 0x7f2028106a80 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f2025ffb640 1 -- 172.21.15.89:0/3189267453 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2020005020 con 0x7f2028106a80 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f202ef40640 1 -- 172.21.15.89:0/3189267453 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202819d6c0 con 0x7f2028106a80 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.997+0000 7f2025ffb640 1 -- 172.21.15.89:0/3189267453 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20200116b0 con 0x7f2028106a80 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.998+0000 7f202ef40640 1 -- 172.21.15.89:0/3189267453 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20281a4fa0 con 0x7f2028106a80 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.998+0000 7f202ef40640 1 -- 172.21.15.89:0/3189267453 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2028100ee0 con 0x7f2028106a80 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:17.999+0000 7f2025ffb640 1 -- 172.21.15.89:0/3189267453 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2020005560 con 0x7f2028106a80 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.000+0000 7f2025ffb640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20180a3240 0x7f20180a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:18.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.000+0000 7f202ccdc640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20180a3240 0x7f20180a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:18.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.000+0000 7f2025ffb640 1 -- 172.21.15.89:0/3189267453 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f20200cf6e0 con 0x7f2028106a80 2024-09-15T23:21:18.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.001+0000 7f202ccdc640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20180a3240 0x7f20180a5700 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f201c0058c0 tx=0x7f201c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:18.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.004+0000 7f2025ffb640 1 -- 172.21.15.89:0/3189267453 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2020016030 con 0x7f2028106a80 2024-09-15T23:21:18.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:18 smithi089 ceph-mon[30933]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:18.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:18.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:18.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:18.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:18.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:18.201 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:18 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:18.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.223+0000 7f202ef40640 1 -- 172.21.15.89:0/3189267453 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20281a5260 con 0x7f2028106a80 2024-09-15T23:21:18.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.223+0000 7f2025ffb640 1 -- 172.21.15.89:0/3189267453 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3091 (secure 0 0 0) 0x7f202009c8a0 con 0x7f2028106a80 2024-09-15T23:21:18.224 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:18.224 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":4,"btime":"2024-09-15T23:21:18:022998+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-15T23:21:18.225 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 4 2024-09-15T23:21:18.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.225+0000 7f202ef40640 1 -- 172.21.15.89:0/3189267453 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20180a3240 msgr2=0x7f20180a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:18.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.225+0000 7f202ef40640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20180a3240 0x7f20180a5700 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f201c0058c0 tx=0x7f201c00a2d0 comp rx=0 tx=0).stop 2024-09-15T23:21:18.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.225+0000 7f202ef40640 1 -- 172.21.15.89:0/3189267453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2028106a80 msgr2=0x7f202819ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:18.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.225+0000 7f202ef40640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2028106a80 0x7f202819ccf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f202000eaa0 tx=0x7f202000ef60 comp rx=0 tx=0).stop 2024-09-15T23:21:18.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.225+0000 7f202ef40640 1 -- 172.21.15.89:0/3189267453 shutdown_connections 2024-09-15T23:21:18.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.225+0000 7f202ef40640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20180a3240 0x7f20180a5700 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:18.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.225+0000 7f202ef40640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2028106a80 0x7f202819ccf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:18.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.225+0000 7f202ef40640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2028101420 0x7f2028198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:18.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.225+0000 7f202ef40640 1 --2- 172.21.15.89:0/3189267453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2028100a80 0x7f2028198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:18.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.225+0000 7f202ef40640 1 -- 172.21.15.89:0/3189267453 >> 172.21.15.89:0/3189267453 conn(0x7f20280fc7a0 msgr2=0x7f2028104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:18.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.226+0000 7f202ef40640 1 -- 172.21.15.89:0/3189267453 shutdown_connections 2024-09-15T23:21:18.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:18.226+0000 7f202ef40640 1 -- 172.21.15.89:0/3189267453 wait complete. 2024-09-15T23:21:18.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:18 smithi120 ceph-mon[35250]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:18.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:18 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:18.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:18 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:18.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:18 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:18.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:18 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:18.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:18 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:18.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:18 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:18.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:18 smithi174 ceph-mon[35586]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:18.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:18.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:18.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:18.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:18.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:18.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:18 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:19.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:19 smithi089 ceph-mon[30933]: Deploying daemon mds.d on smithi089 2024-09-15T23:21:19.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:19 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.89:6834/3816006418,v1:172.21.15.89:6835/3816006418] up:boot 2024-09-15T23:21:19.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:19 smithi089 ceph-mon[30933]: fsmap 3 up:standby 2024-09-15T23:21:19.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:19 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T23:21:19.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:19 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3189267453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:19 smithi120 ceph-mon[35250]: Deploying daemon mds.d on smithi089 2024-09-15T23:21:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:19 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.89:6834/3816006418,v1:172.21.15.89:6835/3816006418] up:boot 2024-09-15T23:21:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:19 smithi120 ceph-mon[35250]: fsmap 3 up:standby 2024-09-15T23:21:19.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:19 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T23:21:19.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:19 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3189267453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:19.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:19 smithi174 ceph-mon[35586]: Deploying daemon mds.d on smithi089 2024-09-15T23:21:19.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:19 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.89:6834/3816006418,v1:172.21.15.89:6835/3816006418] up:boot 2024-09-15T23:21:19.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:19 smithi174 ceph-mon[35586]: fsmap 3 up:standby 2024-09-15T23:21:19.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:19 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T23:21:19.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:19 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3189267453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:20.296 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:20 smithi089 ceph-mon[30933]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:20 smithi120 ceph-mon[35250]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:20.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:20 smithi174 ceph-mon[35586]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:20.560 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:21:20 smithi089 systemd[1]: Starting Ceph mds.d for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:20.971 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:21:20 smithi089 podman[70187]: 2024-09-15 23:21:20.595288223 +0000 UTC m=+0.015874052 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:20.971 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:21:20 smithi089 podman[70187]: 2024-09-15 23:21:20.717838734 +0000 UTC m=+0.138424566 container create 6da1be8a1e9c37eb267a5c2fa980d5b8116d516c7d2aed02fa81ba137c86549f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0) 2024-09-15T23:21:20.971 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:21:20 smithi089 podman[70187]: 2024-09-15 23:21:20.970724547 +0000 UTC m=+0.391310374 container init 6da1be8a1e9c37eb267a5c2fa980d5b8116d516c7d2aed02fa81ba137c86549f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-15T23:21:21.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:21.290 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:21.310 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:21:20 smithi089 podman[70187]: 2024-09-15 23:21:20.973880677 +0000 UTC m=+0.394466518 container start 6da1be8a1e9c37eb267a5c2fa980d5b8116d516c7d2aed02fa81ba137c86549f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-d, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-15T23:21:21.310 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:21:20 smithi089 ceph-mds[70206]: -- 172.21.15.89:0/269337641 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fa20f17860 con 0x55fa20ef5400 2024-09-15T23:21:21.310 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:21:20 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-d[70202]: starting mds.d at 2024-09-15T23:21:21.310 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:21:21 smithi089 bash[70187]: 6da1be8a1e9c37eb267a5c2fa980d5b8116d516c7d2aed02fa81ba137c86549f 2024-09-15T23:21:21.310 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:21:21 smithi089 systemd[1]: Started Ceph mds.d for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.764+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/182625695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d4101420 msgr2=0x7fc7d410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.764+0000 7fc7dbf5f640 1 --2- 172.21.15.89:0/182625695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d4101420 0x7fc7d410d950 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009cd0 tx=0x7fc7cc02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:21:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.764+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/182625695 shutdown_connections 2024-09-15T23:21:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.764+0000 7fc7dbf5f640 1 --2- 172.21.15.89:0/182625695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d4101420 0x7fc7d410d950 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.764+0000 7fc7dbf5f640 1 --2- 172.21.15.89:0/182625695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d4100a80 0x7fc7d4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.764+0000 7fc7dbf5f640 1 --2- 172.21.15.89:0/182625695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4106a80 0x7fc7d4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.764+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/182625695 >> 172.21.15.89:0/182625695 conn(0x7fc7d40fc7a0 msgr2=0x7fc7d40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:21.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.764+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/182625695 shutdown_connections 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.764+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/182625695 wait complete. 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7dbf5f640 1 Processor -- start 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7dbf5f640 1 -- start start 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7dbf5f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d4100a80 0x7fc7d41993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7dbf5f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101420 0x7fc7d4199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7dbf5f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d4106a80 0x7fc7d41a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7dbf5f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4110f20 con 0x7fc7d4106a80 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7dbf5f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4110da0 con 0x7fc7d4100a80 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7dbf5f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d41110a0 con 0x7fc7d4101420 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7da4fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d4106a80 0x7fc7d41a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7da4fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d4106a80 0x7fc7d41a4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59950/0 (socket says 172.21.15.89:59950) 2024-09-15T23:21:21.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7da4fc640 1 -- 172.21.15.89:0/2881474710 learned_addr learned my addr 172.21.15.89:0/2881474710 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7d9cfb640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d4100a80 0x7fc7d41993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7da4fc640 1 -- 172.21.15.89:0/2881474710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101420 msgr2=0x7fc7d4199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7d94fa640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101420 0x7fc7d4199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7da4fc640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101420 0x7fc7d4199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7da4fc640 1 -- 172.21.15.89:0/2881474710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d4100a80 msgr2=0x7fc7d41993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7da4fc640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d4100a80 0x7fc7d41993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7da4fc640 1 -- 172.21.15.89:0/2881474710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d41a49b0 con 0x7fc7d4106a80 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.765+0000 7fc7d94fa640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101420 0x7fc7d4199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.766+0000 7fc7da4fc640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d4106a80 0x7fc7d41a4470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc006010 tx=0x7fc7cc003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.766+0000 7fc7caffd640 1 -- 172.21.15.89:0/2881474710 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc004450 con 0x7fc7d4106a80 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.766+0000 7fc7caffd640 1 -- 172.21.15.89:0/2881474710 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7cc003e10 con 0x7fc7d4106a80 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.766+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/2881474710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d41a4b80 con 0x7fc7d4106a80 2024-09-15T23:21:21.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.766+0000 7fc7caffd640 1 -- 172.21.15.89:0/2881474710 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc046480 con 0x7fc7d4106a80 2024-09-15T23:21:21.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.766+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/2881474710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d41a5030 con 0x7fc7d4106a80 2024-09-15T23:21:21.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.767+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/2881474710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d4100f60 con 0x7fc7d4106a80 2024-09-15T23:21:21.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.769+0000 7fc7caffd640 1 -- 172.21.15.89:0/2881474710 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fc7cc0045f0 con 0x7fc7d4106a80 2024-09-15T23:21:21.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.769+0000 7fc7caffd640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7c00a3240 0x7fc7c00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:21.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.769+0000 7fc7d9cfb640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7c00a3240 0x7fc7c00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:21.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.770+0000 7fc7caffd640 1 -- 172.21.15.89:0/2881474710 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fc7cc02e030 con 0x7fc7d4106a80 2024-09-15T23:21:21.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.770+0000 7fc7d9cfb640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7c00a3240 0x7fc7c00a5700 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0009990 tx=0x7fc7d0009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:21.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.773+0000 7fc7caffd640 1 -- 172.21.15.89:0/2881474710 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc7cc04b060 con 0x7fc7d4106a80 2024-09-15T23:21:21.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.974+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/2881474710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7d4109010 con 0x7fc7d4106a80 2024-09-15T23:21:21.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.975+0000 7fc7caffd640 1 -- 172.21.15.89:0/2881474710 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3916 (secure 0 0 0) 0x7fc7cc0be130 con 0x7fc7d4106a80 2024-09-15T23:21:21.976 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:21.976 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":5,"btime":"2024-09-15T23:21:21:035242+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-15T23:21:21.976 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 5 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/2881474710 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7c00a3240 msgr2=0x7fc7c00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7c00a3240 0x7fc7c00a5700 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0009990 tx=0x7fc7d0009110 comp rx=0 tx=0).stop 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/2881474710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d4106a80 msgr2=0x7fc7d41a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d4106a80 0x7fc7d41a4470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc006010 tx=0x7fc7cc003c20 comp rx=0 tx=0).stop 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/2881474710 shutdown_connections 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7c00a3240 0x7fc7c00a5700 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d4106a80 0x7fc7d41a4470 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d4101420 0x7fc7d4199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 --2- 172.21.15.89:0/2881474710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d4100a80 0x7fc7d41993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/2881474710 >> 172.21.15.89:0/2881474710 conn(0x7fc7d40fc7a0 msgr2=0x7fc7d40fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:21.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/2881474710 shutdown_connections 2024-09-15T23:21:21.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:21.977+0000 7fc7dbf5f640 1 -- 172.21.15.89:0/2881474710 wait complete. 2024-09-15T23:21:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.89:6836/3164152358,v1:172.21.15.89:6837/3164152358] up:boot 2024-09-15T23:21:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: fsmap 4 up:standby 2024-09-15T23:21:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T23:21:22.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:22.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:22.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:22.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:22.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:22.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:22.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:22 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2881474710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.89:6836/3164152358,v1:172.21.15.89:6837/3164152358] up:boot 2024-09-15T23:21:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: fsmap 4 up:standby 2024-09-15T23:21:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T23:21:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:22.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:22 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2881474710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.89:6836/3164152358,v1:172.21.15.89:6837/3164152358] up:boot 2024-09-15T23:21:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: fsmap 4 up:standby 2024-09-15T23:21:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T23:21:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:22.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:22 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2881474710' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:23.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:23 smithi120 ceph-mon[35250]: Deploying daemon mds.e on smithi120 2024-09-15T23:21:23.224 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:21:23 smithi120 systemd[1]: Starting Ceph mds.e for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:23 smithi089 ceph-mon[30933]: Deploying daemon mds.e on smithi120 2024-09-15T23:21:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:23 smithi174 ceph-mon[35586]: Deploying daemon mds.e on smithi120 2024-09-15T23:21:23.641 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:21:23 smithi120 podman[64557]: 2024-09-15 23:21:23.27246461 +0000 UTC m=+0.015830739 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:23.641 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:21:23 smithi120 podman[64557]: 2024-09-15 23:21:23.404688078 +0000 UTC m=+0.148054244 container create 8d6ea269aa73ad06ae128858712eeb43b20f3aacc69600cfb1d2ed69109d0ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-15T23:21:23.641 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:21:23 smithi120 podman[64557]: 2024-09-15 23:21:23.641042327 +0000 UTC m=+0.384408448 container init 8d6ea269aa73ad06ae128858712eeb43b20f3aacc69600cfb1d2ed69109d0ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-15T23:21:23.973 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:21:23 smithi120 podman[64557]: 2024-09-15 23:21:23.643871002 +0000 UTC m=+0.387237131 container start 8d6ea269aa73ad06ae128858712eeb43b20f3aacc69600cfb1d2ed69109d0ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-15T23:21:23.973 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:21:23 smithi120 ceph-mds[64577]: -- 172.21.15.120:0/2913999795 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5591390d5860 con 0x5591390b3000 2024-09-15T23:21:23.973 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:21:23 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e[64573]: starting mds.e at 2024-09-15T23:21:23.973 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:21:23 smithi120 bash[64557]: 8d6ea269aa73ad06ae128858712eeb43b20f3aacc69600cfb1d2ed69109d0ac4 2024-09-15T23:21:23.973 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:21:23 smithi120 systemd[1]: Started Ceph mds.e for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:24.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:24 smithi089 ceph-mon[30933]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:24.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:24.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:24 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:24.362 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:24 smithi174 ceph-mon[35586]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:24.362 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:24.362 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:24.363 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:24.363 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:24.363 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:24.363 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:24 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:24 smithi120 ceph-mon[35250]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:24.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:24 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:24.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:25.047 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:25.168 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:25 smithi089 ceph-mon[30933]: Deploying daemon mds.f on smithi174 2024-09-15T23:21:25.168 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:25 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.120:6834/1429316223,v1:172.21.15.120:6835/1429316223] up:boot 2024-09-15T23:21:25.168 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:25 smithi089 ceph-mon[30933]: fsmap 5 up:standby 2024-09-15T23:21:25.168 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:25 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:21:25.215 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:25 smithi174 ceph-mon[35586]: Deploying daemon mds.f on smithi174 2024-09-15T23:21:25.215 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:25 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.120:6834/1429316223,v1:172.21.15.120:6835/1429316223] up:boot 2024-09-15T23:21:25.215 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:25 smithi174 ceph-mon[35586]: fsmap 5 up:standby 2024-09-15T23:21:25.216 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:25 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:21:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:25 smithi120 ceph-mon[35250]: Deploying daemon mds.f on smithi174 2024-09-15T23:21:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:25 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.120:6834/1429316223,v1:172.21.15.120:6835/1429316223] up:boot 2024-09-15T23:21:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:25 smithi120 ceph-mon[35250]: fsmap 5 up:standby 2024-09-15T23:21:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:25 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:21:25.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3110142913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c6410c170 msgr2=0x7f5c6410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 --2- 172.21.15.89:0/3110142913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c6410c170 0x7f5c6410e610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009ca0 tx=0x7f5c5c02a950 comp rx=0 tx=0).stop 2024-09-15T23:21:25.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3110142913 shutdown_connections 2024-09-15T23:21:25.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 --2- 172.21.15.89:0/3110142913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c6410c170 0x7f5c6410e610 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 --2- 172.21.15.89:0/3110142913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c64103660 0x7f5c6410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 --2- 172.21.15.89:0/3110142913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c64102d40 0x7f5c64103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3110142913 >> 172.21.15.89:0/3110142913 conn(0x7f5c640fc7a0 msgr2=0x7f5c640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3110142913 shutdown_connections 2024-09-15T23:21:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3110142913 wait complete. 2024-09-15T23:21:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 Processor -- start 2024-09-15T23:21:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 -- start start 2024-09-15T23:21:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c64102d40 0x7f5c64198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c64103660 0x7f5c641989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.557+0000 7f5c6a3b7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c6410c170 0x7f5c6419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c6a3b7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64111fb0 con 0x7f5c64103660 2024-09-15T23:21:25.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c6a3b7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64111e30 con 0x7f5c6410c170 2024-09-15T23:21:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c6a3b7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64112130 con 0x7f5c64102d40 2024-09-15T23:21:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c5bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c64103660 0x7f5c641989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c5bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c64103660 0x7f5c641989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48516/0 (socket says 172.21.15.89:48516) 2024-09-15T23:21:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c5bfff640 1 -- 172.21.15.89:0/3351635595 learned_addr learned my addr 172.21.15.89:0/3351635595 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c63fff640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c64102d40 0x7f5c64198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c68954640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c6410c170 0x7f5c6419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c5bfff640 1 -- 172.21.15.89:0/3351635595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c64102d40 msgr2=0x7f5c64198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c5bfff640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c64102d40 0x7f5c64198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c5bfff640 1 -- 172.21.15.89:0/3351635595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c6410c170 msgr2=0x7f5c6419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c5bfff640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c6410c170 0x7f5c6419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c5bfff640 1 -- 172.21.15.89:0/3351635595 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6419d420 con 0x7f5c64103660 2024-09-15T23:21:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c68954640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c6410c170 0x7f5c6419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c5bfff640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c64103660 0x7f5c641989b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00d950 tx=0x7f5c4c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c61ffb640 1 -- 172.21.15.89:0/3351635595 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c017030 con 0x7f5c64103660 2024-09-15T23:21:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c61ffb640 1 -- 172.21.15.89:0/3351635595 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c4c004020 con 0x7f5c64103660 2024-09-15T23:21:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c61ffb640 1 -- 172.21.15.89:0/3351635595 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c0047a0 con 0x7f5c64103660 2024-09-15T23:21:25.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3351635595 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6419d710 con 0x7f5c64103660 2024-09-15T23:21:25.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.558+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3351635595 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c641a4ff0 con 0x7f5c64103660 2024-09-15T23:21:25.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.559+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3351635595 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c64103120 con 0x7f5c64103660 2024-09-15T23:21:25.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.560+0000 7f5c61ffb640 1 -- 172.21.15.89:0/3351635595 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5c4c0041d0 con 0x7f5c64103660 2024-09-15T23:21:25.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.561+0000 7f5c61ffb640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5c500a32c0 0x7f5c500a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.561+0000 7f5c63fff640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5c500a32c0 0x7f5c500a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.561+0000 7f5c61ffb640 1 -- 172.21.15.89:0/3351635595 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f5c4c0cedf0 con 0x7f5c64103660 2024-09-15T23:21:25.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.561+0000 7f5c63fff640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5c500a32c0 0x7f5c500a5780 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5c54009890 tx=0x7f5c54008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:25.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.565+0000 7f5c61ffb640 1 -- 172.21.15.89:0/3351635595 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5c4c09bfb0 con 0x7f5c64103660 2024-09-15T23:21:25.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.766+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3351635595 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c641997d0 con 0x7f5c64103660 2024-09-15T23:21:25.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.767+0000 7f5c61ffb640 1 -- 172.21.15.89:0/3351635595 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4744 (secure 0 0 0) 0x7f5c4c09b8b0 con 0x7f5c64103660 2024-09-15T23:21:25.768 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:25.768 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":6,"btime":"2024-09-15T23:21:24:046590+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-09-15T23:21:25.769 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 6 2024-09-15T23:21:25.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3351635595 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5c500a32c0 msgr2=0x7f5c500a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5c500a32c0 0x7f5c500a5780 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5c54009890 tx=0x7f5c54008040 comp rx=0 tx=0).stop 2024-09-15T23:21:25.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3351635595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c64103660 msgr2=0x7f5c641989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c64103660 0x7f5c641989b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c00d950 tx=0x7f5c4c00de10 comp rx=0 tx=0).stop 2024-09-15T23:21:25.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3351635595 shutdown_connections 2024-09-15T23:21:25.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5c500a32c0 0x7f5c500a5780 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c6410c170 0x7f5c6419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5c64103660 0x7f5c641989b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 --2- 172.21.15.89:0/3351635595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c64102d40 0x7f5c64198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3351635595 >> 172.21.15.89:0/3351635595 conn(0x7f5c640fc7a0 msgr2=0x7f5c640fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:25.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3351635595 shutdown_connections 2024-09-15T23:21:25.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:25.769+0000 7f5c6a3b7640 1 -- 172.21.15.89:0/3351635595 wait complete. 2024-09-15T23:21:25.908 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:21:25 smithi174 systemd[1]: Starting Ceph mds.f for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:26.180 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:26 smithi174 ceph-mon[35586]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:26.180 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:26 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3351635595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:26.180 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:21:25 smithi174 podman[65513]: 2024-09-15 23:21:25.810243884 +0000 UTC m=+0.015890660 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:26.180 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:21:25 smithi174 podman[65513]: 2024-09-15 23:21:25.925521647 +0000 UTC m=+0.131168455 container create c946c964cccb66a5df4d96f1a40fd6034f3309cc2118267b544c8d4828cb23ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-15T23:21:26.180 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:21:26 smithi174 podman[65513]: 2024-09-15 23:21:26.153128606 +0000 UTC m=+0.358775371 container init c946c964cccb66a5df4d96f1a40fd6034f3309cc2118267b544c8d4828cb23ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-15T23:21:26.180 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:21:26 smithi174 podman[65513]: 2024-09-15 23:21:26.156120924 +0000 UTC m=+0.361767688 container start c946c964cccb66a5df4d96f1a40fd6034f3309cc2118267b544c8d4828cb23ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-15T23:21:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:26 smithi089 ceph-mon[30933]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:26 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3351635595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:26 smithi120 ceph-mon[35250]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:26 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3351635595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:26.485 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:21:26 smithi174 ceph-mds[65532]: -- 172.21.15.174:0/3965955463 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fc70603860 con 0x55fc705e1400 2024-09-15T23:21:26.485 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:21:26 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f[65528]: starting mds.f at 2024-09-15T23:21:26.485 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:21:26 smithi174 bash[65513]: c946c964cccb66a5df4d96f1a40fd6034f3309cc2118267b544c8d4828cb23ef 2024-09-15T23:21:26.486 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:21:26 smithi174 systemd[1]: Started Ceph mds.f for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:27.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:27.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:27.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:27.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:27.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:27.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:27.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: Deploying daemon mds.c on smithi174 2024-09-15T23:21:27.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:27.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] up:boot 2024-09-15T23:21:27.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: fsmap 6 up:standby 2024-09-15T23:21:27.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:21:27.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:27.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:27.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:27.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:27.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:27.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:27.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: Deploying daemon mds.c on smithi174 2024-09-15T23:21:27.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:27.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] up:boot 2024-09-15T23:21:27.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: fsmap 6 up:standby 2024-09-15T23:21:27.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:21:27.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:27.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:27.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:27.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:27.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:27.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:27.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: Deploying daemon mds.c on smithi174 2024-09-15T23:21:27.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:27.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] up:boot 2024-09-15T23:21:27.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: fsmap 6 up:standby 2024-09-15T23:21:27.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:21:28.468 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:21:28 smithi174 systemd[1]: Starting Ceph mds.c for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:28.482 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:28.733 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:21:28 smithi174 podman[65814]: 2024-09-15 23:21:28.370053699 +0000 UTC m=+0.015967435 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:28.733 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:21:28 smithi174 podman[65814]: 2024-09-15 23:21:28.488280264 +0000 UTC m=+0.134193990 container create aa363d291d7c32c25010730f07dbe57beef319b2fd2761570040011806e1b2ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-15T23:21:28.733 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:21:28 smithi174 podman[65814]: 2024-09-15 23:21:28.732675446 +0000 UTC m=+0.378589172 container init aa363d291d7c32c25010730f07dbe57beef319b2fd2761570040011806e1b2ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:21:28.738 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:28.986 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:21:28 smithi174 podman[65814]: 2024-09-15 23:21:28.735605254 +0000 UTC m=+0.381518989 container start aa363d291d7c32c25010730f07dbe57beef319b2fd2761570040011806e1b2ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-15T23:21:28.986 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:21:28 smithi174 ceph-mds[65833]: -- 172.21.15.174:0/2725474146 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56130bd13860 con 0x56130bcf1400 2024-09-15T23:21:28.986 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:21:28 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c[65829]: starting mds.c at 2024-09-15T23:21:28.986 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:21:28 smithi174 bash[65814]: aa363d291d7c32c25010730f07dbe57beef319b2fd2761570040011806e1b2ee 2024-09-15T23:21:28.986 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:21:28 smithi174 systemd[1]: Started Ceph mds.c for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:29.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.221+0000 7f845d24e640 1 -- 172.21.15.89:0/2077740034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8458100e20 msgr2=0x7f8458101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.221+0000 7f845d24e640 1 --2- 172.21.15.89:0/2077740034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8458100e20 0x7f8458101200 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8448009900 tx=0x7f844802a480 comp rx=0 tx=0).stop 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.221+0000 7f845d24e640 1 -- 172.21.15.89:0/2077740034 shutdown_connections 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.221+0000 7f845d24e640 1 --2- 172.21.15.89:0/2077740034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458102ef0 0x7f845810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.221+0000 7f845d24e640 1 --2- 172.21.15.89:0/2077740034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458102570 0x7f84581029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.221+0000 7f845d24e640 1 --2- 172.21.15.89:0/2077740034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8458100e20 0x7f8458101200 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.221+0000 7f845d24e640 1 -- 172.21.15.89:0/2077740034 >> 172.21.15.89:0/2077740034 conn(0x7f84580fc7d0 msgr2=0x7f84580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.221+0000 7f845d24e640 1 -- 172.21.15.89:0/2077740034 shutdown_connections 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.221+0000 7f845d24e640 1 -- 172.21.15.89:0/2077740034 wait complete. 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f845d24e640 1 Processor -- start 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f845d24e640 1 -- start start 2024-09-15T23:21:29.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f845d24e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458100e20 0x7f8458198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f845d24e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8458102570 0x7f84581989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f845d24e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458102ef0 0x7f845819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f845d24e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8458110f80 con 0x7f8458102570 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f845d24e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8458110e00 con 0x7f8458100e20 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f845d24e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8458111100 con 0x7f8458102ef0 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f844ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8458102570 0x7f84581989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f844ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8458102570 0x7f84581989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48528/0 (socket says 172.21.15.89:48528) 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f844ffff640 1 -- 172.21.15.89:0/2731676669 learned_addr learned my addr 172.21.15.89:0/2731676669 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.222+0000 7f8456d9d640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458100e20 0x7f8458198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f845759e640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458102ef0 0x7f845819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f844ffff640 1 -- 172.21.15.89:0/2731676669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458102ef0 msgr2=0x7f845819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f844ffff640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458102ef0 0x7f845819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:29.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f844ffff640 1 -- 172.21.15.89:0/2731676669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458100e20 msgr2=0x7f8458198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f844ffff640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458100e20 0x7f8458198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f844ffff640 1 -- 172.21.15.89:0/2731676669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845819d430 con 0x7f8458102570 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f845759e640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458102ef0 0x7f845819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f844ffff640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8458102570 0x7f84581989c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f844000d950 tx=0x7f844000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f8454d99640 1 -- 172.21.15.89:0/2731676669 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440017070 con 0x7f8458102570 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f845d24e640 1 -- 172.21.15.89:0/2731676669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f845819d720 con 0x7f8458102570 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f8454d99640 1 -- 172.21.15.89:0/2731676669 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8440009c30 con 0x7f8458102570 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f8454d99640 1 -- 172.21.15.89:0/2731676669 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844000f6b0 con 0x7f8458102570 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.223+0000 7f845d24e640 1 -- 172.21.15.89:0/2731676669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84581a5000 con 0x7f8458102570 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.224+0000 7f845d24e640 1 -- 172.21.15.89:0/2731676669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8458101200 con 0x7f8458102570 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.225+0000 7f8454d99640 1 -- 172.21.15.89:0/2731676669 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f84400109d0 con 0x7f8458102570 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.226+0000 7f8454d99640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84440a3240 0x7f84440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:29.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.226+0000 7f8456d9d640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84440a3240 0x7f84440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:29.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.226+0000 7f8454d99640 1 -- 172.21.15.89:0/2731676669 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f84400cf0e0 con 0x7f8458102570 2024-09-15T23:21:29.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.226+0000 7f8456d9d640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84440a3240 0x7f84440a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8448009230 tx=0x7f84480058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:29.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.230+0000 7f8454d99640 1 -- 172.21.15.89:0/2731676669 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8440009de0 con 0x7f8458102570 2024-09-15T23:21:29.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.437+0000 7f845d24e640 1 -- 172.21.15.89:0/2731676669 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84581a51d0 con 0x7f8458102570 2024-09-15T23:21:29.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.438+0000 7f8454d99640 1 -- 172.21.15.89:0/2731676669 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6397 (secure 0 0 0) 0x7f844009c2a0 con 0x7f8458102570 2024-09-15T23:21:29.439 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:29.439 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":8,"btime":"2024-09-15T23:21:28:813920+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-15T23:21:29.439 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 8 2024-09-15T23:21:29.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.441+0000 7f845d24e640 1 -- 172.21.15.89:0/2731676669 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84440a3240 msgr2=0x7f84440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:29.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.441+0000 7f845d24e640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84440a3240 0x7f84440a5700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8448009230 tx=0x7f84480058d0 comp rx=0 tx=0).stop 2024-09-15T23:21:29.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.441+0000 7f845d24e640 1 -- 172.21.15.89:0/2731676669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8458102570 msgr2=0x7f84581989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:29.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.441+0000 7f845d24e640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8458102570 0x7f84581989c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f844000d950 tx=0x7f844000de10 comp rx=0 tx=0).stop 2024-09-15T23:21:29.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.442+0000 7f845d24e640 1 -- 172.21.15.89:0/2731676669 shutdown_connections 2024-09-15T23:21:29.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.442+0000 7f845d24e640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f84440a3240 0x7f84440a5700 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:29.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.442+0000 7f845d24e640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458102ef0 0x7f845819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:29.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.442+0000 7f845d24e640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8458102570 0x7f84581989c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:29.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.442+0000 7f845d24e640 1 --2- 172.21.15.89:0/2731676669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458100e20 0x7f8458198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:29.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.442+0000 7f845d24e640 1 -- 172.21.15.89:0/2731676669 >> 172.21.15.89:0/2731676669 conn(0x7f84580fc7d0 msgr2=0x7f845806a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:29.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.442+0000 7f845d24e640 1 -- 172.21.15.89:0/2731676669 shutdown_connections 2024-09-15T23:21:29.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:29.442+0000 7f845d24e640 1 -- 172.21.15.89:0/2731676669 wait complete. 2024-09-15T23:21:29.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] up:boot 2024-09-15T23:21:29.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: fsmap 7 up:standby 2024-09-15T23:21:29.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:21:29.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:29.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:29.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:29.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:29.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:29.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:29.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: Deploying daemon mds.h on smithi120 2024-09-15T23:21:29.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:29.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:29 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2731676669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] up:boot 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: fsmap 7 up:standby 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: Deploying daemon mds.h on smithi120 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:30.070 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:29 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2731676669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:30.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] up:boot 2024-09-15T23:21:30.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: fsmap 7 up:standby 2024-09-15T23:21:30.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:21:30.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:30.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:30.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:30.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:30.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:30.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:30.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: Deploying daemon mds.h on smithi120 2024-09-15T23:21:30.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:30.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:29 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2731676669' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:30.977 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:21:30 smithi120 systemd[1]: Starting Ceph mds.h for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:31.415 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:21:31 smithi120 podman[64863]: 2024-09-15 23:21:31.024940013 +0000 UTC m=+0.015970096 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:31.416 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:21:31 smithi120 podman[64863]: 2024-09-15 23:21:31.153927294 +0000 UTC m=+0.144957367 container create 3b11d314aa275c9bd5bd48d1685c769e159ae4027b489c9076de96ec37adef0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-15T23:21:31.416 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:21:31 smithi120 podman[64863]: 2024-09-15 23:21:31.415172434 +0000 UTC m=+0.406202511 container init 3b11d314aa275c9bd5bd48d1685c769e159ae4027b489c9076de96ec37adef0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e) 2024-09-15T23:21:31.723 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:21:31 smithi120 podman[64863]: 2024-09-15 23:21:31.418218813 +0000 UTC m=+0.409248887 container start 3b11d314aa275c9bd5bd48d1685c769e159ae4027b489c9076de96ec37adef0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-15T23:21:31.723 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:21:31 smithi120 ceph-mds[64882]: -- 172.21.15.120:0/150529121 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f706873860 con 0x55f706851400 2024-09-15T23:21:31.723 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:21:31 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h[64878]: starting mds.h at 2024-09-15T23:21:31.723 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:21:31 smithi120 bash[64863]: 3b11d314aa275c9bd5bd48d1685c769e159ae4027b489c9076de96ec37adef0f 2024-09-15T23:21:31.724 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:21:31 smithi120 systemd[1]: Started Ceph mds.h for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:32.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:32 smithi089 ceph-mon[30933]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:32 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:32 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:32 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:32 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:32 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:32.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:32 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:32.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:32 smithi174 ceph-mon[35586]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:32.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:32 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:32.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:32 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:32.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:32 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:32.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:32 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:32.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:32 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:32.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:32 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:32.357 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:32 smithi120 ceph-mon[35250]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:32 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:32 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:32 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:32 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:32 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:32.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:32 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.843+0000 7f8847765640 1 -- 172.21.15.89:0/21885765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8840106a80 msgr2=0x7f8840106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.843+0000 7f8847765640 1 --2- 172.21.15.89:0/21885765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8840106a80 0x7f8840106e60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8830009930 tx=0x7f883002a420 comp rx=0 tx=0).stop 2024-09-15T23:21:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.843+0000 7f8847765640 1 -- 172.21.15.89:0/21885765 shutdown_connections 2024-09-15T23:21:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.843+0000 7f8847765640 1 --2- 172.21.15.89:0/21885765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8840101420 0x7f884010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.843+0000 7f8847765640 1 --2- 172.21.15.89:0/21885765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8840100a80 0x7f8840100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.843+0000 7f8847765640 1 --2- 172.21.15.89:0/21885765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8840106a80 0x7f8840106e60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.843+0000 7f8847765640 1 -- 172.21.15.89:0/21885765 >> 172.21.15.89:0/21885765 conn(0x7f88400fc7a0 msgr2=0x7f88400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.843+0000 7f8847765640 1 -- 172.21.15.89:0/21885765 shutdown_connections 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.843+0000 7f8847765640 1 -- 172.21.15.89:0/21885765 wait complete. 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8847765640 1 Processor -- start 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8847765640 1 -- start start 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8847765640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8840100a80 0x7f88401974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8847765640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8840101420 0x7f8840197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8847765640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8840106a80 0x7f884019bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8847765640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8840110f20 con 0x7f8840106a80 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8847765640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8840110da0 con 0x7f8840101420 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8847765640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f88401110a0 con 0x7f8840100a80 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845d02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8840106a80 0x7f884019bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845d02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8840106a80 0x7f884019bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48552/0 (socket says 172.21.15.89:48552) 2024-09-15T23:21:32.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845d02640 1 -- 172.21.15.89:0/3728639719 learned_addr learned my addr 172.21.15.89:0/3728639719 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845501640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8840100a80 0x7f88401974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845d02640 1 -- 172.21.15.89:0/3728639719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8840100a80 msgr2=0x7f88401974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845d02640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8840100a80 0x7f88401974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845d02640 1 -- 172.21.15.89:0/3728639719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8840101420 msgr2=0x7f8840197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845d02640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8840101420 0x7f8840197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845d02640 1 -- 172.21.15.89:0/3728639719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884019c4a0 con 0x7f8840106a80 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845501640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8840100a80 0x7f88401974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.844+0000 7f8845d02640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8840106a80 0x7f884019bdc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f883800b970 tx=0x7f883800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.845+0000 7f88367fc640 1 -- 172.21.15.89:0/3728639719 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838019070 con 0x7f8840106a80 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.845+0000 7f88367fc640 1 -- 172.21.15.89:0/3728639719 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f883800cb40 con 0x7f8840106a80 2024-09-15T23:21:32.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.845+0000 7f8847765640 1 -- 172.21.15.89:0/3728639719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f884019c790 con 0x7f8840106a80 2024-09-15T23:21:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.845+0000 7f88367fc640 1 -- 172.21.15.89:0/3728639719 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8838007650 con 0x7f8840106a80 2024-09-15T23:21:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.845+0000 7f8847765640 1 -- 172.21.15.89:0/3728639719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88401a4070 con 0x7f8840106a80 2024-09-15T23:21:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.846+0000 7f8847765640 1 -- 172.21.15.89:0/3728639719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8840100f60 con 0x7f8840106a80 2024-09-15T23:21:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.847+0000 7f88367fc640 1 -- 172.21.15.89:0/3728639719 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f883801c070 con 0x7f8840106a80 2024-09-15T23:21:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.848+0000 7f88367fc640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f882c0a3240 0x7f882c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.848+0000 7f8845501640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f882c0a3240 0x7f882c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.848+0000 7f88367fc640 1 -- 172.21.15.89:0/3728639719 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f88380ceff0 con 0x7f8840106a80 2024-09-15T23:21:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.848+0000 7f8845501640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f882c0a3240 0x7f882c0a5700 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8830000c00 tx=0x7f8830032d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:32.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:32.851+0000 7f88367fc640 1 -- 172.21.15.89:0/3728639719 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f883809c130 con 0x7f8840106a80 2024-09-15T23:21:33.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.054+0000 7f8847765640 1 -- 172.21.15.89:0/3728639719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8840198850 con 0x7f8840106a80 2024-09-15T23:21:33.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.054+0000 7f88367fc640 1 -- 172.21.15.89:0/3728639719 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7222 (secure 0 0 0) 0x7f8840198850 con 0x7f8840106a80 2024-09-15T23:21:33.056 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:33.056 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":9,"btime":"2024-09-15T23:21:32:026496+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-15T23:21:33.056 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 9 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.056+0000 7f8847765640 1 -- 172.21.15.89:0/3728639719 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f882c0a3240 msgr2=0x7f882c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.056+0000 7f8847765640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f882c0a3240 0x7f882c0a5700 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8830000c00 tx=0x7f8830032d50 comp rx=0 tx=0).stop 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.056+0000 7f8847765640 1 -- 172.21.15.89:0/3728639719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8840106a80 msgr2=0x7f884019bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.056+0000 7f8847765640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8840106a80 0x7f884019bdc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f883800b970 tx=0x7f883800be30 comp rx=0 tx=0).stop 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.056+0000 7f8847765640 1 -- 172.21.15.89:0/3728639719 shutdown_connections 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.056+0000 7f8847765640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f882c0a3240 0x7f882c0a5700 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.056+0000 7f8847765640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8840106a80 0x7f884019bdc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.056+0000 7f8847765640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8840101420 0x7f8840197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.056+0000 7f8847765640 1 --2- 172.21.15.89:0/3728639719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8840100a80 0x7f88401974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.056+0000 7f8847765640 1 -- 172.21.15.89:0/3728639719 >> 172.21.15.89:0/3728639719 conn(0x7f88400fc7a0 msgr2=0x7f88400fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:33.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.057+0000 7f8847765640 1 -- 172.21.15.89:0/3728639719 shutdown_connections 2024-09-15T23:21:33.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:33.057+0000 7f8847765640 1 -- 172.21.15.89:0/3728639719 wait complete. 2024-09-15T23:21:33.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:33 smithi174 ceph-mon[35586]: Deploying daemon mds.i on smithi174 2024-09-15T23:21:33.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:33 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] up:boot 2024-09-15T23:21:33.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:33 smithi174 ceph-mon[35586]: fsmap 8 up:standby 2024-09-15T23:21:33.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:33 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:21:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:33 smithi089 ceph-mon[30933]: Deploying daemon mds.i on smithi174 2024-09-15T23:21:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:33 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] up:boot 2024-09-15T23:21:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:33 smithi089 ceph-mon[30933]: fsmap 8 up:standby 2024-09-15T23:21:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:33 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:21:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:33 smithi120 ceph-mon[35250]: Deploying daemon mds.i on smithi174 2024-09-15T23:21:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:33 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] up:boot 2024-09-15T23:21:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:33 smithi120 ceph-mon[35250]: fsmap 8 up:standby 2024-09-15T23:21:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:33 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:21:33.685 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:21:33 smithi174 systemd[1]: Starting Ceph mds.i for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:33.954 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:21:33 smithi174 podman[66113]: 2024-09-15 23:21:33.587492162 +0000 UTC m=+0.015876478 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:33.955 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:21:33 smithi174 podman[66113]: 2024-09-15 23:21:33.709707642 +0000 UTC m=+0.138091992 container create 7e318973bd9e35db1ca9517756205a5c420fb44e83b88ba1198a96d9cfe3dea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-15T23:21:33.955 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:21:33 smithi174 podman[66113]: 2024-09-15 23:21:33.954135428 +0000 UTC m=+0.382519737 container init 7e318973bd9e35db1ca9517756205a5c420fb44e83b88ba1198a96d9cfe3dea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:21:34.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:34 smithi174 ceph-mon[35586]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:34.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:34 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3728639719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:34.236 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:21:33 smithi174 podman[66113]: 2024-09-15 23:21:33.956962579 +0000 UTC m=+0.385346888 container start 7e318973bd9e35db1ca9517756205a5c420fb44e83b88ba1198a96d9cfe3dea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-15T23:21:34.236 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:21:33 smithi174 ceph-mds[66132]: -- 172.21.15.174:0/3790876763 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d53c3cb860 con 0x55d53c3a9000 2024-09-15T23:21:34.236 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:21:33 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i[66128]: starting mds.i at 2024-09-15T23:21:34.236 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:21:34 smithi174 bash[66113]: 7e318973bd9e35db1ca9517756205a5c420fb44e83b88ba1198a96d9cfe3dea8 2024-09-15T23:21:34.236 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:21:34 smithi174 systemd[1]: Started Ceph mds.i for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:34 smithi089 ceph-mon[30933]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:34.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:34 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3728639719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:34 smithi120 ceph-mon[35250]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:34 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3728639719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:35.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:35 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] up:boot 2024-09-15T23:21:35.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:35 smithi089 ceph-mon[30933]: fsmap 9 up:standby 2024-09-15T23:21:35.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:35 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:21:35.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:35 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:35.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:35 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:35.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:35 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:35.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:35 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:35.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:35 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:35.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:35 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:35.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:35 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] up:boot 2024-09-15T23:21:35.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:35 smithi120 ceph-mon[35250]: fsmap 9 up:standby 2024-09-15T23:21:35.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:35 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:21:35.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:35 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:35.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:35 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:35.477 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:35 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:35.477 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:35 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:35.477 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:35 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:35.477 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:35 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:35 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] up:boot 2024-09-15T23:21:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:35 smithi174 ceph-mon[35586]: fsmap 9 up:standby 2024-09-15T23:21:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:35 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:21:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:35 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:35 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:35 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:35 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:35 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:35 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:35.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:36.082 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:36.206 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:36 smithi089 ceph-mon[30933]: Deploying daemon mds.g on smithi089 2024-09-15T23:21:36.206 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:36 smithi089 ceph-mon[30933]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:36.206 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:21:36 smithi089 systemd[1]: Starting Ceph mds.g for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:36 smithi120 ceph-mon[35250]: Deploying daemon mds.g on smithi089 2024-09-15T23:21:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:36 smithi120 ceph-mon[35250]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:36 smithi174 ceph-mon[35586]: Deploying daemon mds.g on smithi089 2024-09-15T23:21:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:36 smithi174 ceph-mon[35586]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:36.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.584+0000 7ff307678640 1 -- 172.21.15.89:0/1383098629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3001027a0 msgr2=0x7ff3001067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.584+0000 7ff307678640 1 --2- 172.21.15.89:0/1383098629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3001027a0 0x7ff3001067f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009900 tx=0x7ff2f002a410 comp rx=0 tx=0).stop 2024-09-15T23:21:36.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.584+0000 7ff307678640 1 -- 172.21.15.89:0/1383098629 shutdown_connections 2024-09-15T23:21:36.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.584+0000 7ff307678640 1 --2- 172.21.15.89:0/1383098629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff300106f20 0x7ff30010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.584+0000 7ff307678640 1 --2- 172.21.15.89:0/1383098629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3001027a0 0x7ff3001067f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.584+0000 7ff307678640 1 --2- 172.21.15.89:0/1383098629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300101df0 0x7ff3001021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.584+0000 7ff307678640 1 -- 172.21.15.89:0/1383098629 >> 172.21.15.89:0/1383098629 conn(0x7ff3000fd8b0 msgr2=0x7ff3000ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.584+0000 7ff307678640 1 -- 172.21.15.89:0/1383098629 shutdown_connections 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.584+0000 7ff307678640 1 -- 172.21.15.89:0/1383098629 wait complete. 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff307678640 1 Processor -- start 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff307678640 1 -- start start 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff307678640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300101df0 0x7ff300198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff307678640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3001027a0 0x7ff300198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff307678640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff300106f20 0x7ff30019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff307678640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff300111f80 con 0x7ff300106f20 2024-09-15T23:21:36.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff307678640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff300111e00 con 0x7ff300101df0 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff307678640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff300112100 con 0x7ff3001027a0 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff305c15640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff300106f20 0x7ff30019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff305414640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300101df0 0x7ff300198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff305c15640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff300106f20 0x7ff30019ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59544/0 (socket says 172.21.15.89:59544) 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff305c15640 1 -- 172.21.15.89:0/2238938639 learned_addr learned my addr 172.21.15.89:0/2238938639 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.585+0000 7ff304c13640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3001027a0 0x7ff300198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff305c15640 1 -- 172.21.15.89:0/2238938639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3001027a0 msgr2=0x7ff300198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff305c15640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3001027a0 0x7ff300198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff305c15640 1 -- 172.21.15.89:0/2238938639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300101df0 msgr2=0x7ff300198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff305c15640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300101df0 0x7ff300198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff305c15640 1 -- 172.21.15.89:0/2238938639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30019d3b0 con 0x7ff300106f20 2024-09-15T23:21:36.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff305c15640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff300106f20 0x7ff30019ccd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009fd0 tx=0x7ff2f800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff2f67fc640 1 -- 172.21.15.89:0/2238938639 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8017070 con 0x7ff300106f20 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff2f67fc640 1 -- 172.21.15.89:0/2238938639 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2f8003e50 con 0x7ff300106f20 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff2f67fc640 1 -- 172.21.15.89:0/2238938639 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80106c0 con 0x7ff300106f20 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff307678640 1 -- 172.21.15.89:0/2238938639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff30019d6a0 con 0x7ff300106f20 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.586+0000 7ff307678640 1 -- 172.21.15.89:0/2238938639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff300074420 con 0x7ff300106f20 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.587+0000 7ff307678640 1 -- 172.21.15.89:0/2238938639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3001032d0 con 0x7ff300106f20 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.588+0000 7ff2f67fc640 1 -- 172.21.15.89:0/2238938639 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff2f8003970 con 0x7ff300106f20 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.589+0000 7ff2f67fc640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2ec0a3240 0x7ff2ec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.589+0000 7ff305414640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2ec0a3240 0x7ff2ec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.589+0000 7ff2f67fc640 1 -- 172.21.15.89:0/2238938639 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7ff2f80cf1e0 con 0x7ff300106f20 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.589+0000 7ff305414640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2ec0a3240 0x7ff2ec0a5700 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc005d00 tx=0x7ff2fc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:36.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.593+0000 7ff2f67fc640 1 -- 172.21.15.89:0/2238938639 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff2f809c3a0 con 0x7ff300106f20 2024-09-15T23:21:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.814+0000 7ff307678640 1 -- 172.21.15.89:0/2238938639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff30010a160 con 0x7ff300106f20 2024-09-15T23:21:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.814+0000 7ff2f67fc640 1 -- 172.21.15.89:0/2238938639 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8052 (secure 0 0 0) 0x7ff2f809bca0 con 0x7ff300106f20 2024-09-15T23:21:36.815 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:36.816 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":10,"btime":"2024-09-15T23:21:34:039410+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-15T23:21:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 10 2024-09-15T23:21:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 -- 172.21.15.89:0/2238938639 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2ec0a3240 msgr2=0x7ff2ec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2ec0a3240 0x7ff2ec0a5700 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc005d00 tx=0x7ff2fc00b410 comp rx=0 tx=0).stop 2024-09-15T23:21:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 -- 172.21.15.89:0/2238938639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff300106f20 msgr2=0x7ff30019ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff300106f20 0x7ff30019ccd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009fd0 tx=0x7ff2f800b260 comp rx=0 tx=0).stop 2024-09-15T23:21:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 -- 172.21.15.89:0/2238938639 shutdown_connections 2024-09-15T23:21:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2ec0a3240 0x7ff2ec0a5700 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff300106f20 0x7ff30019ccd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3001027a0 0x7ff300198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 --2- 172.21.15.89:0/2238938639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff300101df0 0x7ff300198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 -- 172.21.15.89:0/2238938639 >> 172.21.15.89:0/2238938639 conn(0x7ff3000fd8b0 msgr2=0x7ff3000fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 -- 172.21.15.89:0/2238938639 shutdown_connections 2024-09-15T23:21:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:36.817+0000 7ff307678640 1 -- 172.21.15.89:0/2238938639 wait complete. 2024-09-15T23:21:37.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:37 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2238938639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:37.279 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:21:36 smithi089 podman[71048]: 2024-09-15 23:21:36.727005298 +0000 UTC m=+0.015859790 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:37.279 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:21:36 smithi089 podman[71048]: 2024-09-15 23:21:36.855381479 +0000 UTC m=+0.144236000 container create d5083778b9544e1e61ae957879e3c29f5564e6fe76ca10bff48b9c959bf57825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-15T23:21:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:37 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2238938639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:37 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2238938639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:38.055 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:21:37 smithi089 podman[71048]: 2024-09-15 23:21:37.719138913 +0000 UTC m=+1.007993393 container init d5083778b9544e1e61ae957879e3c29f5564e6fe76ca10bff48b9c959bf57825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-15T23:21:38.055 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:21:37 smithi089 podman[71048]: 2024-09-15 23:21:37.721992951 +0000 UTC m=+1.010847431 container start d5083778b9544e1e61ae957879e3c29f5564e6fe76ca10bff48b9c959bf57825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-15T23:21:38.055 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:21:37 smithi089 ceph-mds[71084]: -- 172.21.15.89:0/554302909 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555bf317b860 con 0x555bf3159000 2024-09-15T23:21:38.055 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:21:37 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g[71076]: starting mds.g at 2024-09-15T23:21:38.055 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:21:37 smithi089 bash[71048]: d5083778b9544e1e61ae957879e3c29f5564e6fe76ca10bff48b9c959bf57825 2024-09-15T23:21:38.055 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:21:37 smithi089 systemd[1]: Started Ceph mds.g for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:38 smithi089 ceph-mon[30933]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:38.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:38.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:38.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:38 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:38 smithi120 ceph-mon[35250]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:38.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:38.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:38 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:38 smithi174 ceph-mon[35586]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:38 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:39.319 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:39 smithi089 ceph-mon[30933]: Deploying daemon mds.a on smithi089 2024-09-15T23:21:39.319 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:39 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.89:6838/2895111363,v1:172.21.15.89:6839/2895111363] up:boot 2024-09-15T23:21:39.319 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:39 smithi089 ceph-mon[30933]: fsmap 10 up:standby 2024-09-15T23:21:39.320 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:39 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T23:21:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:39 smithi120 ceph-mon[35250]: Deploying daemon mds.a on smithi089 2024-09-15T23:21:39.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:39 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.89:6838/2895111363,v1:172.21.15.89:6839/2895111363] up:boot 2024-09-15T23:21:39.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:39 smithi120 ceph-mon[35250]: fsmap 10 up:standby 2024-09-15T23:21:39.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:39 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T23:21:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:39 smithi174 ceph-mon[35586]: Deploying daemon mds.a on smithi089 2024-09-15T23:21:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:39 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.89:6838/2895111363,v1:172.21.15.89:6839/2895111363] up:boot 2024-09-15T23:21:39.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:39 smithi174 ceph-mon[35586]: fsmap 10 up:standby 2024-09-15T23:21:39.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:39 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T23:21:39.609 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:40.019 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:21:39 smithi089 systemd[1]: Starting Ceph mds.a for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:40.273 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:40 smithi089 ceph-mon[30933]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:40.273 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:21:40 smithi089 podman[71385]: 2024-09-15 23:21:39.922041614 +0000 UTC m=+0.016285003 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:40.273 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:21:40 smithi089 podman[71385]: 2024-09-15 23:21:40.03556143 +0000 UTC m=+0.129804849 container create ccb0f0bc2bb76ad6836d084eebb037c9289fe8a66c538e5312e02dee6f69a040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-15T23:21:40.274 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:21:40 smithi089 podman[71385]: 2024-09-15 23:21:40.273172759 +0000 UTC m=+0.367416136 container init ccb0f0bc2bb76ad6836d084eebb037c9289fe8a66c538e5312e02dee6f69a040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-09-15T23:21:40.406 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:40 smithi120 ceph-mon[35250]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:40.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:40 smithi174 ceph-mon[35586]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:40.528 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:21:40 smithi089 podman[71385]: 2024-09-15 23:21:40.276466695 +0000 UTC m=+0.370710078 container start ccb0f0bc2bb76ad6836d084eebb037c9289fe8a66c538e5312e02dee6f69a040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:21:40.528 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:21:40 smithi089 ceph-mds[71430]: -- 172.21.15.89:0/1675834149 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563d5352f860 con 0x563d5350c800 2024-09-15T23:21:40.528 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:21:40 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a[71412]: starting mds.a at 2024-09-15T23:21:40.528 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:21:40 smithi089 bash[71385]: ccb0f0bc2bb76ad6836d084eebb037c9289fe8a66c538e5312e02dee6f69a040 2024-09-15T23:21:40.528 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:21:40 smithi089 systemd[1]: Started Ceph mds.a for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:40.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 -- 172.21.15.89:0/4123745702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68f8101df0 msgr2=0x7f68f81021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:40.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 --2- 172.21.15.89:0/4123745702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68f8101df0 0x7f68f81021d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f68e8009900 tx=0x7f68e802a410 comp rx=0 tx=0).stop 2024-09-15T23:21:40.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 -- 172.21.15.89:0/4123745702 shutdown_connections 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 --2- 172.21.15.89:0/4123745702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68f8106f20 0x7f68f810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 --2- 172.21.15.89:0/4123745702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68f81027a0 0x7f68f81067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 --2- 172.21.15.89:0/4123745702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68f8101df0 0x7f68f81021d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 -- 172.21.15.89:0/4123745702 >> 172.21.15.89:0/4123745702 conn(0x7f68f80fd8b0 msgr2=0x7f68f80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 -- 172.21.15.89:0/4123745702 shutdown_connections 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 -- 172.21.15.89:0/4123745702 wait complete. 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 Processor -- start 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 -- start start 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68f8101df0 0x7f68f81983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68f81027a0 0x7f68f8198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68f8106f20 0x7f68f819ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f68f8111f40 con 0x7f68f8106f20 2024-09-15T23:21:40.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f68f8111dc0 con 0x7f68f81027a0 2024-09-15T23:21:40.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.924+0000 7f68fda60640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f68f81120c0 con 0x7f68f8101df0 2024-09-15T23:21:40.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68f8106f20 0x7f68f819ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:40.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68f8106f20 0x7f68f819ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59606/0 (socket says 172.21.15.89:59606) 2024-09-15T23:21:40.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68f77fe640 1 -- 172.21.15.89:0/1643363794 learned_addr learned my addr 172.21.15.89:0/1643363794 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:40.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68f6ffd640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68f8101df0 0x7f68f81983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:40.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68effff640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68f81027a0 0x7f68f8198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68f77fe640 1 -- 172.21.15.89:0/1643363794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68f8101df0 msgr2=0x7f68f81983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68f77fe640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68f8101df0 0x7f68f81983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68f77fe640 1 -- 172.21.15.89:0/1643363794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68f81027a0 msgr2=0x7f68f8198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68f77fe640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68f81027a0 0x7f68f8198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68f77fe640 1 -- 172.21.15.89:0/1643363794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f819d390 con 0x7f68f8106f20 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68f77fe640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68f8106f20 0x7f68f819ccb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f68f0009fd0 tx=0x7f68f000def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68fca5e640 1 -- 172.21.15.89:0/1643363794 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f0017070 con 0x7f68f8106f20 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68fda60640 1 -- 172.21.15.89:0/1643363794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f819d680 con 0x7f68f8106f20 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68fca5e640 1 -- 172.21.15.89:0/1643363794 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68f0003e50 con 0x7f68f8106f20 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68fca5e640 1 -- 172.21.15.89:0/1643363794 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f0010460 con 0x7f68f8106f20 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.925+0000 7f68fda60640 1 -- 172.21.15.89:0/1643363794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f81a4f60 con 0x7f68f8106f20 2024-09-15T23:21:40.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.926+0000 7f68fda60640 1 -- 172.21.15.89:0/1643363794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f81021d0 con 0x7f68f8106f20 2024-09-15T23:21:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.927+0000 7f68fca5e640 1 -- 172.21.15.89:0/1643363794 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f68f0003970 con 0x7f68f8106f20 2024-09-15T23:21:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.928+0000 7f68fca5e640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f68e40a3240 0x7f68e40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.928+0000 7f68f6ffd640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f68e40a3240 0x7f68e40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.928+0000 7f68fca5e640 1 -- 172.21.15.89:0/1643363794 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f68f00cf080 con 0x7f68f8106f20 2024-09-15T23:21:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.929+0000 7f68f6ffd640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f68e40a3240 0x7f68e40a5700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68e8009230 tx=0x7f68e80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:40.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:40.932+0000 7f68fca5e640 1 -- 172.21.15.89:0/1643363794 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f68f009c240 con 0x7f68f8106f20 2024-09-15T23:21:41.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.137+0000 7f68fda60640 1 -- 172.21.15.89:0/1643363794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68f810a100 con 0x7f68f8106f20 2024-09-15T23:21:41.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.137+0000 7f68fca5e640 1 -- 172.21.15.89:0/1643363794 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9704 (secure 0 0 0) 0x7f68f009bb40 con 0x7f68f8106f20 2024-09-15T23:21:41.139 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:41.139 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":12,"btime":"2024-09-15T23:21:41:065930+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-09-15T23:21:41.139 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 12 2024-09-15T23:21:41.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 -- 172.21.15.89:0/1643363794 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f68e40a3240 msgr2=0x7f68e40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:41.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f68e40a3240 0x7f68e40a5700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f68e8009230 tx=0x7f68e80058d0 comp rx=0 tx=0).stop 2024-09-15T23:21:41.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 -- 172.21.15.89:0/1643363794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68f8106f20 msgr2=0x7f68f819ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:41.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68f8106f20 0x7f68f819ccb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f68f0009fd0 tx=0x7f68f000def0 comp rx=0 tx=0).stop 2024-09-15T23:21:41.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 -- 172.21.15.89:0/1643363794 shutdown_connections 2024-09-15T23:21:41.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f68e40a3240 0x7f68e40a5700 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:41.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f68f8106f20 0x7f68f819ccb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:41.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f68f81027a0 0x7f68f8198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:41.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 --2- 172.21.15.89:0/1643363794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68f8101df0 0x7f68f81983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:41.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 -- 172.21.15.89:0/1643363794 >> 172.21.15.89:0/1643363794 conn(0x7f68f80fd8b0 msgr2=0x7f68f80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:41.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 -- 172.21.15.89:0/1643363794 shutdown_connections 2024-09-15T23:21:41.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:41.139+0000 7f68fda60640 1 -- 172.21.15.89:0/1643363794 wait complete. 2024-09-15T23:21:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: Deploying daemon mds.b on smithi120 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.89:6840/3365004630,v1:172.21.15.89:6841/3365004630] up:boot 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: fsmap 11 up:standby 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T23:21:41.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:41 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1643363794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: Deploying daemon mds.b on smithi120 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.89:6840/3365004630,v1:172.21.15.89:6841/3365004630] up:boot 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: fsmap 11 up:standby 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T23:21:41.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:41 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1643363794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:41.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:41.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:41.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:41.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:21:41.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:21:41.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:41.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: Deploying daemon mds.b on smithi120 2024-09-15T23:21:41.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:41.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.89:6840/3365004630,v1:172.21.15.89:6841/3365004630] up:boot 2024-09-15T23:21:41.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: fsmap 11 up:standby 2024-09-15T23:21:41.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T23:21:41.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:41 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1643363794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:42.692 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:21:42 smithi120 systemd[1]: Starting Ceph mds.b for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:21:42.965 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:21:42 smithi120 podman[65160]: 2024-09-15 23:21:42.595229325 +0000 UTC m=+0.016011872 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:21:42.965 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:21:42 smithi120 podman[65160]: 2024-09-15 23:21:42.728415118 +0000 UTC m=+0.149197665 container create c4a65ae0ee3143258f855ad0d6c16f1f1878a5b0205eb7287e65db759abcc92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-15T23:21:42.965 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:21:42 smithi120 podman[65160]: 2024-09-15 23:21:42.964707809 +0000 UTC m=+0.385490356 container init c4a65ae0ee3143258f855ad0d6c16f1f1878a5b0205eb7287e65db759abcc92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-15T23:21:43.223 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:21:42 smithi120 podman[65160]: 2024-09-15 23:21:42.967656185 +0000 UTC m=+0.388438739 container start c4a65ae0ee3143258f855ad0d6c16f1f1878a5b0205eb7287e65db759abcc92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-09-15T23:21:43.223 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:21:42 smithi120 ceph-mds[65179]: -- 172.21.15.120:0/2240910173 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556a3cf3860 con 0x5556a3cd1400 2024-09-15T23:21:43.224 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:21:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b[65175]: starting mds.b at 2024-09-15T23:21:43.224 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:21:43 smithi120 bash[65160]: c4a65ae0ee3143258f855ad0d6c16f1f1878a5b0205eb7287e65db759abcc92d 2024-09-15T23:21:43.224 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:21:43 smithi120 systemd[1]: Started Ceph mds.b for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:21:43.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:44.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:44 smithi120 ceph-mon[35250]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:44.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:44 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] up:boot 2024-09-15T23:21:44.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:44 smithi120 ceph-mon[35250]: fsmap 12 up:standby 2024-09-15T23:21:44.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:44 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:21:44.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:44 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:44 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:44 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:44 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:44 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:44 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:44.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:44 smithi174 ceph-mon[35586]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:44.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:44 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] up:boot 2024-09-15T23:21:44.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:44 smithi174 ceph-mon[35586]: fsmap 12 up:standby 2024-09-15T23:21:44.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:44 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:21:44.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:44 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:44 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:44 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:44 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:44 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:44 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:44.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:44 smithi089 ceph-mon[30933]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:44.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:44 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] up:boot 2024-09-15T23:21:44.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:44 smithi089 ceph-mon[30933]: fsmap 12 up:standby 2024-09-15T23:21:44.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:44 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:21:44.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:44 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:44 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:44 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:44 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:44 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:44.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:44 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:44.444 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:44.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 -- 172.21.15.89:0/3076564413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144c102dc0 msgr2=0x7f144c1031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:44.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 --2- 172.21.15.89:0/3076564413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144c102dc0 0x7f144c1031a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f143c009900 tx=0x7f143c02a480 comp rx=0 tx=0).stop 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 -- 172.21.15.89:0/3076564413 shutdown_connections 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 --2- 172.21.15.89:0/3076564413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f144c10c1a0 0x7f144c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 --2- 172.21.15.89:0/3076564413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f144c1036e0 0x7f144c10bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 --2- 172.21.15.89:0/3076564413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144c102dc0 0x7f144c1031a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 -- 172.21.15.89:0/3076564413 >> 172.21.15.89:0/3076564413 conn(0x7f144c0fc7a0 msgr2=0x7f144c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 -- 172.21.15.89:0/3076564413 shutdown_connections 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 -- 172.21.15.89:0/3076564413 wait complete. 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 Processor -- start 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.917+0000 7f145244c640 1 -- start start 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f145244c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144c102dc0 0x7f144c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f145244c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f144c1036e0 0x7f144c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f145244c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f144c10c1a0 0x7f144c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:44.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f145244c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f144c111fb0 con 0x7f144c102dc0 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f145244c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f144c111e30 con 0x7f144c1036e0 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f145244c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f144c112130 con 0x7f144c10c1a0 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f144bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144c102dc0 0x7f144c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f144bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144c102dc0 0x7f144c198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59628/0 (socket says 172.21.15.89:59628) 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f144bfff640 1 -- 172.21.15.89:0/487435259 learned_addr learned my addr 172.21.15.89:0/487435259 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f14509e9640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f144c10c1a0 0x7f144c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f144bfff640 1 -- 172.21.15.89:0/487435259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f144c10c1a0 msgr2=0x7f144c19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f144bfff640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f144c10c1a0 0x7f144c19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f144bfff640 1 -- 172.21.15.89:0/487435259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f144c1036e0 msgr2=0x7f144c1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f144bfff640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f144c1036e0 0x7f144c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f144bfff640 1 -- 172.21.15.89:0/487435259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c19d420 con 0x7f144c102dc0 2024-09-15T23:21:44.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f144bfff640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144c102dc0 0x7f144c198470 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f143c006010 tx=0x7f143c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f1449ffb640 1 -- 172.21.15.89:0/487435259 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c03f030 con 0x7f144c102dc0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f1449ffb640 1 -- 172.21.15.89:0/487435259 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f143c003ce0 con 0x7f144c102dc0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f145244c640 1 -- 172.21.15.89:0/487435259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144c19d6b0 con 0x7f144c102dc0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f1449ffb640 1 -- 172.21.15.89:0/487435259 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c035070 con 0x7f144c102dc0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.918+0000 7f145244c640 1 -- 172.21.15.89:0/487435259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f144c1a4ff0 con 0x7f144c102dc0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.920+0000 7f145244c640 1 -- 172.21.15.89:0/487435259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f144c1031a0 con 0x7f144c102dc0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.921+0000 7f1449ffb640 1 -- 172.21.15.89:0/487435259 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f143c042030 con 0x7f144c102dc0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.921+0000 7f1449ffb640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14380a32c0 0x7f14380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.922+0000 7f1443fff640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14380a32c0 0x7f14380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.922+0000 7f1449ffb640 1 -- 172.21.15.89:0/487435259 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f143c0efa00 con 0x7f144c102dc0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.922+0000 7f1443fff640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14380a32c0 0x7f14380a5780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f144c0feda0 tx=0x7f1434006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:44.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:44.926+0000 7f1449ffb640 1 -- 172.21.15.89:0/487435259 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f143c0bcc80 con 0x7f144c102dc0 2024-09-15T23:21:45.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.145+0000 7f145244c640 1 -- 172.21.15.89:0/487435259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f144c107250 con 0x7f144c102dc0 2024-09-15T23:21:45.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.146+0000 7f1449ffb640 1 -- 172.21.15.89:0/487435259 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10533 (secure 0 0 0) 0x7f143c0bc580 con 0x7f144c102dc0 2024-09-15T23:21:45.147 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:45.147 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":13,"btime":"2024-09-15T23:21:43:045963+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24452,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-09-15T23:21:45.147 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 13 2024-09-15T23:21:45.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 -- 172.21.15.89:0/487435259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14380a32c0 msgr2=0x7f14380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:45.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14380a32c0 0x7f14380a5780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f144c0feda0 tx=0x7f1434006c90 comp rx=0 tx=0).stop 2024-09-15T23:21:45.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 -- 172.21.15.89:0/487435259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144c102dc0 msgr2=0x7f144c198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:45.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144c102dc0 0x7f144c198470 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f143c006010 tx=0x7f143c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:21:45.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 -- 172.21.15.89:0/487435259 shutdown_connections 2024-09-15T23:21:45.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f14380a32c0 0x7f14380a5780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:45.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f144c10c1a0 0x7f144c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:45.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f144c1036e0 0x7f144c1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:45.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 --2- 172.21.15.89:0/487435259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f144c102dc0 0x7f144c198470 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:45.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 -- 172.21.15.89:0/487435259 >> 172.21.15.89:0/487435259 conn(0x7f144c0fc7a0 msgr2=0x7f144c0fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:45.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 -- 172.21.15.89:0/487435259 shutdown_connections 2024-09-15T23:21:45.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:45.148+0000 7f145244c640 1 -- 172.21.15.89:0/487435259 wait complete. 2024-09-15T23:21:45.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:45.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/487435259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:45.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:45.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:45.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:45 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:45.877 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-09-15T23:21:45.971 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.971 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.972 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.972 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.972 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:45.972 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/487435259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:45.972 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.972 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.972 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:45.972 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:45.972 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:45.972 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:45 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:46.057 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:46.057 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:46.057 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:46.057 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:46.057 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:46.057 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/487435259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:46.057 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:46.057 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:46.057 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:46.057 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:46.058 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:46.058 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:45 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:46.775 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:47.192 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:46 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.192 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:46 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.192 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:46 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.192 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:46 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.192 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:46 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:46 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:46 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:46 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:46 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:46 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:46 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:46 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:46 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:46 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:46 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:47.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.270+0000 7f13068e5640 1 -- 172.21.15.89:0/1286694150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300102570 msgr2=0x7f13001029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:47.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.270+0000 7f13068e5640 1 --2- 172.21.15.89:0/1286694150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300102570 0x7f13001029b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f12e8009900 tx=0x7f12e802a480 comp rx=0 tx=0).stop 2024-09-15T23:21:47.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.270+0000 7f13068e5640 1 -- 172.21.15.89:0/1286694150 shutdown_connections 2024-09-15T23:21:47.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.270+0000 7f13068e5640 1 --2- 172.21.15.89:0/1286694150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1300102ef0 0x7f130010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:47.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.270+0000 7f13068e5640 1 --2- 172.21.15.89:0/1286694150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300102570 0x7f13001029b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:47.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.270+0000 7f13068e5640 1 --2- 172.21.15.89:0/1286694150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1300100e20 0x7f1300101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:47.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.270+0000 7f13068e5640 1 -- 172.21.15.89:0/1286694150 >> 172.21.15.89:0/1286694150 conn(0x7f13000fc7d0 msgr2=0x7f13000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:47.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.270+0000 7f13068e5640 1 -- 172.21.15.89:0/1286694150 shutdown_connections 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.270+0000 7f13068e5640 1 -- 172.21.15.89:0/1286694150 wait complete. 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f13068e5640 1 Processor -- start 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f13068e5640 1 -- start start 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f13068e5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1300100e20 0x7f1300198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f13068e5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1300102570 0x7f1300198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f13068e5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300102ef0 0x7f130019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f13068e5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1300110f80 con 0x7f1300102ef0 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f13068e5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1300110e00 con 0x7f1300100e20 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f13068e5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1300111100 con 0x7f1300102570 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f1304e82640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300102ef0 0x7f130019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f12fffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1300100e20 0x7f1300198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f1304e82640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300102ef0 0x7f130019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60264/0 (socket says 172.21.15.89:60264) 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f12fffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1300100e20 0x7f1300198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:54872/0 (socket says 172.21.15.89:54872) 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f1304e82640 1 -- 172.21.15.89:0/3243555966 learned_addr learned my addr 172.21.15.89:0/3243555966 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:47.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f12ff7fe640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1300102570 0x7f1300198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f1304e82640 1 -- 172.21.15.89:0/3243555966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1300102570 msgr2=0x7f1300198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f1304e82640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1300102570 0x7f1300198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f1304e82640 1 -- 172.21.15.89:0/3243555966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1300100e20 msgr2=0x7f1300198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.271+0000 7f1304e82640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1300100e20 0x7f1300198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.272+0000 7f1304e82640 1 -- 172.21.15.89:0/3243555966 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130019d4b0 con 0x7f1300102ef0 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.272+0000 7f12fffff640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1300100e20 0x7f1300198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.272+0000 7f1304e82640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300102ef0 0x7f130019cdd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f12f800cce0 tx=0x7f12f800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.272+0000 7f12fd7fa640 1 -- 172.21.15.89:0/3243555966 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f800f450 con 0x7f1300102ef0 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.272+0000 7f13068e5640 1 -- 172.21.15.89:0/3243555966 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130019d7a0 con 0x7f1300102ef0 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.272+0000 7f12fd7fa640 1 -- 172.21.15.89:0/3243555966 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12f800fad0 con 0x7f1300102ef0 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.272+0000 7f12fd7fa640 1 -- 172.21.15.89:0/3243555966 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f8018c10 con 0x7f1300102ef0 2024-09-15T23:21:47.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.272+0000 7f13068e5640 1 -- 172.21.15.89:0/3243555966 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13001a5080 con 0x7f1300102ef0 2024-09-15T23:21:47.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.273+0000 7f13068e5640 1 -- 172.21.15.89:0/3243555966 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1300101200 con 0x7f1300102ef0 2024-09-15T23:21:47.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.274+0000 7f12fd7fa640 1 -- 172.21.15.89:0/3243555966 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f12f800f5f0 con 0x7f1300102ef0 2024-09-15T23:21:47.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.275+0000 7f12fd7fa640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12ec0a3240 0x7f12ec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:47.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.275+0000 7f12fffff640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12ec0a3240 0x7f12ec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:47.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.275+0000 7f12fd7fa640 1 -- 172.21.15.89:0/3243555966 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7f12f80cf9f0 con 0x7f1300102ef0 2024-09-15T23:21:47.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.275+0000 7f12fffff640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12ec0a3240 0x7f12ec0a5700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f12f0005fd0 tx=0x7f12f000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:47.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.279+0000 7f12fd7fa640 1 -- 172.21.15.89:0/3243555966 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f12f8014030 con 0x7f1300102ef0 2024-09-15T23:21:47.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.455+0000 7f13068e5640 1 -- 172.21.15.89:0/3243555966 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f130010a0a0 con 0x7f1300102ef0 2024-09-15T23:21:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:21:47.457 INFO:teuthology.orchestra.run.smithi089.stdout:[ 2024-09-15T23:21:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:21:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: "poolnum": 1, 2024-09-15T23:21:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: "poolname": ".mgr" 2024-09-15T23:21:47.457 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:21:47.457 INFO:teuthology.orchestra.run.smithi089.stdout:] 2024-09-15T23:21:47.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.455+0000 7f12fd7fa640 1 -- 172.21.15.89:0/3243555966 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v75) ==== 84+0+65 (secure 0 0 0) 0x7f12f809cbb0 con 0x7f1300102ef0 2024-09-15T23:21:47.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.458+0000 7f13068e5640 1 -- 172.21.15.89:0/3243555966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12ec0a3240 msgr2=0x7f12ec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.458+0000 7f13068e5640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12ec0a3240 0x7f12ec0a5700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f12f0005fd0 tx=0x7f12f000b390 comp rx=0 tx=0).stop 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.458+0000 7f13068e5640 1 -- 172.21.15.89:0/3243555966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300102ef0 msgr2=0x7f130019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.458+0000 7f13068e5640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300102ef0 0x7f130019cdd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f12f800cce0 tx=0x7f12f800a620 comp rx=0 tx=0).stop 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.459+0000 7f13068e5640 1 -- 172.21.15.89:0/3243555966 shutdown_connections 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.459+0000 7f13068e5640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12ec0a3240 0x7f12ec0a5700 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.459+0000 7f13068e5640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1300102ef0 0x7f130019cdd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.459+0000 7f13068e5640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1300102570 0x7f1300198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.459+0000 7f13068e5640 1 --2- 172.21.15.89:0/3243555966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1300100e20 0x7f1300198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.459+0000 7f13068e5640 1 -- 172.21.15.89:0/3243555966 >> 172.21.15.89:0/3243555966 conn(0x7f13000fc7d0 msgr2=0x7f130006a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.459+0000 7f13068e5640 1 -- 172.21.15.89:0/3243555966 shutdown_connections 2024-09-15T23:21:47.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:47.459+0000 7f13068e5640 1 -- 172.21.15.89:0/3243555966 wait complete. 2024-09-15T23:21:48.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:47 smithi089 ceph-mon[30933]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:48.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:47 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3243555966' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-15T23:21:48.141 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-15T23:21:48.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-15T23:21:48.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:47 smithi120 ceph-mon[35250]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:48.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:47 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3243555966' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-15T23:21:48.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:47 smithi174 ceph-mon[35586]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:48.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:47 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3243555966' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-15T23:21:48.413 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:48.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 -- 172.21.15.89:0/1928768977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa51c106a80 msgr2=0x7fa51c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:48.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 --2- 172.21.15.89:0/1928768977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa51c106a80 0x7fa51c106e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa518009930 tx=0x7fa51802a420 comp rx=0 tx=0).stop 2024-09-15T23:21:48.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 -- 172.21.15.89:0/1928768977 shutdown_connections 2024-09-15T23:21:48.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 --2- 172.21.15.89:0/1928768977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa51c101420 0x7fa51c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 --2- 172.21.15.89:0/1928768977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa51c100a80 0x7fa51c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 --2- 172.21.15.89:0/1928768977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa51c106a80 0x7fa51c106e60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 -- 172.21.15.89:0/1928768977 >> 172.21.15.89:0/1928768977 conn(0x7fa51c0fc7a0 msgr2=0x7fa51c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 -- 172.21.15.89:0/1928768977 shutdown_connections 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 -- 172.21.15.89:0/1928768977 wait complete. 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 Processor -- start 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 -- start start 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa51c100a80 0x7fa51c199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.951+0000 7fa523b1f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa51c101420 0x7fa51c1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa523b1f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa51c106a80 0x7fa51c1a4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa523b1f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c110f20 con 0x7fa51c106a80 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa523b1f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c110da0 con 0x7fa51c101420 2024-09-15T23:21:48.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa523b1f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa51c1110a0 con 0x7fa51c100a80 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa5220bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa51c106a80 0x7fa51c1a4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa5220bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa51c106a80 0x7fa51c1a4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60276/0 (socket says 172.21.15.89:60276) 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa5220bc640 1 -- 172.21.15.89:0/3202063904 learned_addr learned my addr 172.21.15.89:0/3202063904 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa5218bb640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa51c100a80 0x7fa51c199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa5220bc640 1 -- 172.21.15.89:0/3202063904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa51c100a80 msgr2=0x7fa51c199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa5220bc640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa51c100a80 0x7fa51c199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa5220bc640 1 -- 172.21.15.89:0/3202063904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa51c101420 msgr2=0x7fa51c1999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa5220bc640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa51c101420 0x7fa51c1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa5220bc640 1 -- 172.21.15.89:0/3202063904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa51c1a4a50 con 0x7fa51c106a80 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa5220bc640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa51c106a80 0x7fa51c1a4510 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa51400b970 tx=0x7fa51400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa50effd640 1 -- 172.21.15.89:0/3202063904 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514019070 con 0x7fa51c106a80 2024-09-15T23:21:48.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa50effd640 1 -- 172.21.15.89:0/3202063904 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa51400cb40 con 0x7fa51c106a80 2024-09-15T23:21:48.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa523b1f640 1 -- 172.21.15.89:0/3202063904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa51c1a4c20 con 0x7fa51c106a80 2024-09-15T23:21:48.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa50effd640 1 -- 172.21.15.89:0/3202063904 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa514007650 con 0x7fa51c106a80 2024-09-15T23:21:48.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.952+0000 7fa523b1f640 1 -- 172.21.15.89:0/3202063904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa51c1a5130 con 0x7fa51c106a80 2024-09-15T23:21:48.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.953+0000 7fa523b1f640 1 -- 172.21.15.89:0/3202063904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa51c100f60 con 0x7fa51c106a80 2024-09-15T23:21:48.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.954+0000 7fa50effd640 1 -- 172.21.15.89:0/3202063904 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa51401c070 con 0x7fa51c106a80 2024-09-15T23:21:48.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.955+0000 7fa50effd640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa5080a3240 0x7fa5080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:48.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.955+0000 7fa50effd640 1 -- 172.21.15.89:0/3202063904 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7127+0+0 (secure 0 0 0) 0x7fa5140ceab0 con 0x7fa51c106a80 2024-09-15T23:21:48.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.955+0000 7fa5218bb640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa5080a3240 0x7fa5080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:48.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.955+0000 7fa5218bb640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa5080a3240 0x7fa5080a5700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa518000c00 tx=0x7fa518031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:48.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:48.959+0000 7fa50effd640 1 -- 172.21.15.89:0/3202063904 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa51409bc70 con 0x7fa51c106a80 2024-09-15T23:21:49.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.126+0000 7fa523b1f640 1 -- 172.21.15.89:0/3202063904 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa51c108ff0 con 0x7fa51c106a80 2024-09-15T23:21:49.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.489+0000 7fa50effd640 1 -- 172.21.15.89:0/3202063904 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v76) ==== 138+0+0 (secure 0 0 0) 0x7fa5140bb070 con 0x7fa51c106a80 2024-09-15T23:21:49.491 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' created 2024-09-15T23:21:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 -- 172.21.15.89:0/3202063904 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa5080a3240 msgr2=0x7fa5080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa5080a3240 0x7fa5080a5700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa518000c00 tx=0x7fa518031490 comp rx=0 tx=0).stop 2024-09-15T23:21:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 -- 172.21.15.89:0/3202063904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa51c106a80 msgr2=0x7fa51c1a4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa51c106a80 0x7fa51c1a4510 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa51400b970 tx=0x7fa51400be30 comp rx=0 tx=0).stop 2024-09-15T23:21:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 -- 172.21.15.89:0/3202063904 shutdown_connections 2024-09-15T23:21:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa5080a3240 0x7fa5080a5700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa51c106a80 0x7fa51c1a4510 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa51c101420 0x7fa51c1999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 --2- 172.21.15.89:0/3202063904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa51c100a80 0x7fa51c199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 -- 172.21.15.89:0/3202063904 >> 172.21.15.89:0/3202063904 conn(0x7fa51c0fc7a0 msgr2=0x7fa51c0fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:49.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 -- 172.21.15.89:0/3202063904 shutdown_connections 2024-09-15T23:21:49.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:49.491+0000 7fa523b1f640 1 -- 172.21.15.89:0/3202063904 wait complete. 2024-09-15T23:21:49.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:49.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:49.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:49.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:49.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:49.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:49 smithi120 ceph-mon[35250]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:49.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:49 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3202063904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-15T23:21:49.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:49.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:49.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:49.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:49.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:49.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:49 smithi174 ceph-mon[35586]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:49.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:49 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3202063904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-15T23:21:49.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:49.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:49.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:49.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:49.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:21:49.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:49 smithi089 ceph-mon[30933]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:49.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:49 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3202063904' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-15T23:21:50.187 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-15T23:21:50.432 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:50.804 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:50 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3202063904' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-15T23:21:50.804 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:50 smithi089 ceph-mon[30933]: osdmap e76: 12 total, 12 up, 12 in 2024-09-15T23:21:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.888+0000 7f7bcf925640 1 -- 172.21.15.89:0/620560746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc8100b30 msgr2=0x7f7bc8100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.888+0000 7f7bcf925640 1 --2- 172.21.15.89:0/620560746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc8100b30 0x7f7bc8100f10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4009900 tx=0x7f7bc402a470 comp rx=0 tx=0).stop 2024-09-15T23:21:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.888+0000 7f7bcf925640 1 -- 172.21.15.89:0/620560746 shutdown_connections 2024-09-15T23:21:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.888+0000 7f7bcf925640 1 --2- 172.21.15.89:0/620560746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc8105f40 0x7f7bc81063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.888+0000 7f7bcf925640 1 --2- 172.21.15.89:0/620560746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc8101450 0x7f7bc8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.888+0000 7f7bcf925640 1 --2- 172.21.15.89:0/620560746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc8100b30 0x7f7bc8100f10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.888+0000 7f7bcf925640 1 -- 172.21.15.89:0/620560746 >> 172.21.15.89:0/620560746 conn(0x7f7bc80fc7a0 msgr2=0x7f7bc80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.888+0000 7f7bcf925640 1 -- 172.21.15.89:0/620560746 shutdown_connections 2024-09-15T23:21:50.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.888+0000 7f7bcf925640 1 -- 172.21.15.89:0/620560746 wait complete. 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bcf925640 1 Processor -- start 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bcf925640 1 -- start start 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bcf925640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc8100b30 0x7f7bc81a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bcf925640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc8101450 0x7f7bc81a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bcf925640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc8105f40 0x7f7bc819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bcf925640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc8118540 con 0x7f7bc8101450 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bcf925640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc81183c0 con 0x7f7bc8100b30 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bcf925640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc81186c0 con 0x7f7bc8105f40 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bccec0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc8101450 0x7f7bc81a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bccec0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc8101450 0x7f7bc81a4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60286/0 (socket says 172.21.15.89:60286) 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.889+0000 7f7bccec0640 1 -- 172.21.15.89:0/1957881315 learned_addr learned my addr 172.21.15.89:0/1957881315 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:50.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bcd6c1640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc8100b30 0x7f7bc81a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bcdec2640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc8105f40 0x7f7bc819e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bccec0640 1 -- 172.21.15.89:0/1957881315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc8105f40 msgr2=0x7f7bc819e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bccec0640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc8105f40 0x7f7bc819e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bccec0640 1 -- 172.21.15.89:0/1957881315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc8100b30 msgr2=0x7f7bc81a4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bccec0640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc8100b30 0x7f7bc81a4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bccec0640 1 -- 172.21.15.89:0/1957881315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc819f020 con 0x7f7bc8101450 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bccec0640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc8101450 0x7f7bc81a4c70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7bb800d750 tx=0x7f7bb800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bbe7fc640 1 -- 172.21.15.89:0/1957881315 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8017070 con 0x7f7bc8101450 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bbe7fc640 1 -- 172.21.15.89:0/1957881315 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bb8005020 con 0x7f7bc8101450 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bcf925640 1 -- 172.21.15.89:0/1957881315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bc819f310 con 0x7f7bc8101450 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bbe7fc640 1 -- 172.21.15.89:0/1957881315 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8004d00 con 0x7f7bc8101450 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.890+0000 7f7bcf925640 1 -- 172.21.15.89:0/1957881315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bc81ab600 con 0x7f7bc8101450 2024-09-15T23:21:50.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.891+0000 7f7bcf925640 1 -- 172.21.15.89:0/1957881315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bc8105920 con 0x7f7bc8101450 2024-09-15T23:21:50.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.892+0000 7f7bbe7fc640 1 -- 172.21.15.89:0/1957881315 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f7bb80051d0 con 0x7f7bc8101450 2024-09-15T23:21:50.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.893+0000 7f7bbe7fc640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7bb40a32c0 0x7f7bb40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:50.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.893+0000 7f7bcd6c1640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7bb40a32c0 0x7f7bb40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:50.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.893+0000 7f7bbe7fc640 1 -- 172.21.15.89:0/1957881315 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7497+0+0 (secure 0 0 0) 0x7f7bb80cf2f0 con 0x7f7bc8101450 2024-09-15T23:21:50.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.893+0000 7f7bcd6c1640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7bb40a32c0 0x7f7bb40a5780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7bc819fa30 tx=0x7f7bc40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:50.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:50.896+0000 7f7bbe7fc640 1 -- 172.21.15.89:0/1957881315 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7bb809c3c0 con 0x7f7bc8101450 2024-09-15T23:21:50.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:50 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3202063904' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-15T23:21:50.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:50 smithi120 ceph-mon[35250]: osdmap e76: 12 total, 12 up, 12 in 2024-09-15T23:21:50.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:50 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3202063904' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-15T23:21:50.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:50 smithi174 ceph-mon[35586]: osdmap e76: 12 total, 12 up, 12 in 2024-09-15T23:21:51.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.065+0000 7f7bcf925640 1 -- 172.21.15.89:0/1957881315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f7bc81ab830 con 0x7f7bc8101450 2024-09-15T23:21:51.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.502+0000 7f7bbe7fc640 1 -- 172.21.15.89:0/1957881315 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v78) ==== 171+0+0 (secure 0 0 0) 0x7f7bb809bcc0 con 0x7f7bc8101450 2024-09-15T23:21:51.504 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' created 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.504+0000 7f7bcf925640 1 -- 172.21.15.89:0/1957881315 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7bb40a32c0 msgr2=0x7f7bb40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.504+0000 7f7bcf925640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7bb40a32c0 0x7f7bb40a5780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7bc819fa30 tx=0x7f7bc40058d0 comp rx=0 tx=0).stop 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.504+0000 7f7bcf925640 1 -- 172.21.15.89:0/1957881315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc8101450 msgr2=0x7f7bc81a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.504+0000 7f7bcf925640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc8101450 0x7f7bc81a4c70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7bb800d750 tx=0x7f7bb800dc10 comp rx=0 tx=0).stop 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.504+0000 7f7bcf925640 1 -- 172.21.15.89:0/1957881315 shutdown_connections 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.504+0000 7f7bcf925640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7bb40a32c0 0x7f7bb40a5780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.504+0000 7f7bcf925640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc8105f40 0x7f7bc819e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.504+0000 7f7bcf925640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bc8101450 0x7f7bc81a4c70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.504+0000 7f7bcf925640 1 --2- 172.21.15.89:0/1957881315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7bc8100b30 0x7f7bc81a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.504+0000 7f7bcf925640 1 -- 172.21.15.89:0/1957881315 >> 172.21.15.89:0/1957881315 conn(0x7f7bc80fc7a0 msgr2=0x7f7bc8103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:51.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.505+0000 7f7bcf925640 1 -- 172.21.15.89:0/1957881315 shutdown_connections 2024-09-15T23:21:51.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:51.505+0000 7f7bcf925640 1 -- 172.21.15.89:0/1957881315 wait complete. 2024-09-15T23:21:51.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:51 smithi089 ceph-mon[30933]: osdmap e77: 12 total, 12 up, 12 in 2024-09-15T23:21:51.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:51 smithi089 ceph-mon[30933]: pgmap v272: 33 pgs: 1 creating+activating, 3 creating+peering, 26 unknown, 3 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:51.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:51 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1957881315' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-15T23:21:51.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:51 smithi120 ceph-mon[35250]: osdmap e77: 12 total, 12 up, 12 in 2024-09-15T23:21:51.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:51 smithi120 ceph-mon[35250]: pgmap v272: 33 pgs: 1 creating+activating, 3 creating+peering, 26 unknown, 3 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:51.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:51 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1957881315' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-15T23:21:51.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:51 smithi174 ceph-mon[35586]: osdmap e77: 12 total, 12 up, 12 in 2024-09-15T23:21:51.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:51 smithi174 ceph-mon[35586]: pgmap v272: 33 pgs: 1 creating+activating, 3 creating+peering, 26 unknown, 3 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:51.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:51 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1957881315' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-15T23:21:52.154 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-15T23:21:52.398 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:52.772 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:52 smithi089 ceph-mon[30933]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:21:52.772 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:52 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1957881315' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-15T23:21:52.772 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:52 smithi089 ceph-mon[30933]: osdmap e78: 12 total, 12 up, 12 in 2024-09-15T23:21:52.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.851+0000 7f11ffb65640 1 -- 172.21.15.89:0/4224941148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f8074590 msgr2=0x7f11f8074970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.851+0000 7f11ffb65640 1 --2- 172.21.15.89:0/4224941148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f8074590 0x7f11f8074970 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f11f4009900 tx=0x7f11f402a480 comp rx=0 tx=0).stop 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.851+0000 7f11ffb65640 1 -- 172.21.15.89:0/4224941148 shutdown_connections 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.851+0000 7f11ffb65640 1 --2- 172.21.15.89:0/4224941148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11f8112b90 0x7f11f8114f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.851+0000 7f11ffb65640 1 --2- 172.21.15.89:0/4224941148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11f8074eb0 0x7f11f8112650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.851+0000 7f11ffb65640 1 --2- 172.21.15.89:0/4224941148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f8074590 0x7f11f8074970 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.851+0000 7f11ffb65640 1 -- 172.21.15.89:0/4224941148 >> 172.21.15.89:0/4224941148 conn(0x7f11f80fd8c0 msgr2=0x7f11f80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.851+0000 7f11ffb65640 1 -- 172.21.15.89:0/4224941148 shutdown_connections 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.851+0000 7f11ffb65640 1 -- 172.21.15.89:0/4224941148 wait complete. 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11ffb65640 1 Processor -- start 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11ffb65640 1 -- start start 2024-09-15T23:21:52.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11ffb65640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11f8074590 0x7f11f81a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11ffb65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f8074eb0 0x7f11f81a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11ffb65640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11f8112b90 0x7f11f819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11ffb65640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f11f81185c0 con 0x7f11f8074eb0 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11ffb65640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f11f8118440 con 0x7f11f8074590 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11ffb65640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11f8118740 con 0x7f11f8112b90 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11fd100640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f8074eb0 0x7f11f81a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11fd100640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f8074eb0 0x7f11f81a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60304/0 (socket says 172.21.15.89:60304) 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.852+0000 7f11fd100640 1 -- 172.21.15.89:0/2411146542 learned_addr learned my addr 172.21.15.89:0/2411146542 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11fd901640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11f8074590 0x7f11f81a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11fe102640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11f8112b90 0x7f11f819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11fd100640 1 -- 172.21.15.89:0/2411146542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11f8112b90 msgr2=0x7f11f819e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11fd100640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11f8112b90 0x7f11f819e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11fd100640 1 -- 172.21.15.89:0/2411146542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11f8074590 msgr2=0x7f11f81a4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:52.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11fd100640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11f8074590 0x7f11f81a4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11fd100640 1 -- 172.21.15.89:0/2411146542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f819f0f0 con 0x7f11f8074eb0 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11fd901640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11f8074590 0x7f11f81a4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11fe102640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11f8112b90 0x7f11f819e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11fd100640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f8074eb0 0x7f11f81a4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11e800b4a0 tx=0x7f11e800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11eeffd640 1 -- 172.21.15.89:0/2411146542 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e8014070 con 0x7f11f8074eb0 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11ffb65640 1 -- 172.21.15.89:0/2411146542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f819f3e0 con 0x7f11f8074eb0 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11eeffd640 1 -- 172.21.15.89:0/2411146542 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e8010af0 con 0x7f11f8074eb0 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11eeffd640 1 -- 172.21.15.89:0/2411146542 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e8017d10 con 0x7f11f8074eb0 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.853+0000 7f11ffb65640 1 -- 172.21.15.89:0/2411146542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f81ab6d0 con 0x7f11f8074eb0 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.854+0000 7f11ffb65640 1 -- 172.21.15.89:0/2411146542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f8074970 con 0x7f11f8074eb0 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.855+0000 7f11eeffd640 1 -- 172.21.15.89:0/2411146542 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f11e8017070 con 0x7f11f8074eb0 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.856+0000 7f11eeffd640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11e40a3240 0x7f11e40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.856+0000 7f11fd901640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11e40a3240 0x7f11e40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:52.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.857+0000 7f11eeffd640 1 -- 172.21.15.89:0/2411146542 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7879+0+0 (secure 0 0 0) 0x7f11e80ceff0 con 0x7f11f8074eb0 2024-09-15T23:21:52.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.857+0000 7f11fd901640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11e40a3240 0x7f11e40a5700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f11f4000c00 tx=0x7f11f4005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:52.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:52.860+0000 7f11eeffd640 1 -- 172.21.15.89:0/2411146542 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f11e809bec0 con 0x7f11f8074eb0 2024-09-15T23:21:52.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:52 smithi120 ceph-mon[35250]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:21:52.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:52 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1957881315' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-15T23:21:52.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:52 smithi120 ceph-mon[35250]: osdmap e78: 12 total, 12 up, 12 in 2024-09-15T23:21:52.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:52 smithi174 ceph-mon[35586]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:21:52.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:52 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1957881315' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-15T23:21:52.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:52 smithi174 ceph-mon[35586]: osdmap e78: 12 total, 12 up, 12 in 2024-09-15T23:21:53.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.063+0000 7f11ffb65640 1 -- 172.21.15.89:0/2411146542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f11f8105030 con 0x7f11f8074eb0 2024-09-15T23:21:53.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.517+0000 7f11eeffd640 1 -- 172.21.15.89:0/2411146542 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f11e809b7c0 con 0x7f11f8074eb0 2024-09-15T23:21:53.519 INFO:teuthology.orchestra.run.smithi089.stderr:new fs with metadata pool 2 and data pool 3 2024-09-15T23:21:53.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 -- 172.21.15.89:0/2411146542 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11e40a3240 msgr2=0x7f11e40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:53.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11e40a3240 0x7f11e40a5700 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f11f4000c00 tx=0x7f11f4005a20 comp rx=0 tx=0).stop 2024-09-15T23:21:53.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 -- 172.21.15.89:0/2411146542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f8074eb0 msgr2=0x7f11f81a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:53.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f8074eb0 0x7f11f81a4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11e800b4a0 tx=0x7f11e800b960 comp rx=0 tx=0).stop 2024-09-15T23:21:53.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 -- 172.21.15.89:0/2411146542 shutdown_connections 2024-09-15T23:21:53.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11e40a3240 0x7f11e40a5700 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:53.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11f8112b90 0x7f11f819e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:53.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11f8074eb0 0x7f11f81a4d40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:53.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 --2- 172.21.15.89:0/2411146542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11f8074590 0x7f11f81a4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:53.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 -- 172.21.15.89:0/2411146542 >> 172.21.15.89:0/2411146542 conn(0x7f11f80fd8c0 msgr2=0x7f11f80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:53.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 -- 172.21.15.89:0/2411146542 shutdown_connections 2024-09-15T23:21:53.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:53.519+0000 7f11ffb65640 1 -- 172.21.15.89:0/2411146542 wait complete. 2024-09-15T23:21:53.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:53 smithi089 ceph-mon[30933]: osdmap e79: 12 total, 12 up, 12 in 2024-09-15T23:21:53.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:53 smithi089 ceph-mon[30933]: pgmap v275: 97 pgs: 1 creating+activating, 27 creating+peering, 46 unknown, 23 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:53.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:53 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2411146542' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-15T23:21:53.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:53 smithi089 ceph-mon[30933]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-15T23:21:53.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:53 smithi089 ceph-mon[30933]: osdmap e80: 12 total, 12 up, 12 in 2024-09-15T23:21:53.854 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:53 smithi120 ceph-mon[35250]: osdmap e79: 12 total, 12 up, 12 in 2024-09-15T23:21:53.854 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:53 smithi120 ceph-mon[35250]: pgmap v275: 97 pgs: 1 creating+activating, 27 creating+peering, 46 unknown, 23 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:53.854 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:53 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2411146542' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-15T23:21:53.854 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:53 smithi120 ceph-mon[35250]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-15T23:21:53.854 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:53 smithi120 ceph-mon[35250]: osdmap e80: 12 total, 12 up, 12 in 2024-09-15T23:21:53.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:53 smithi174 ceph-mon[35586]: osdmap e79: 12 total, 12 up, 12 in 2024-09-15T23:21:53.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:53 smithi174 ceph-mon[35586]: pgmap v275: 97 pgs: 1 creating+activating, 27 creating+peering, 46 unknown, 23 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:53.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:53 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2411146542' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-15T23:21:53.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:53 smithi174 ceph-mon[35586]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-15T23:21:53.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:53 smithi174 ceph-mon[35586]: osdmap e80: 12 total, 12 up, 12 in 2024-09-15T23:21:54.256 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-15T23:21:54.257 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-15T23:21:54.502 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:54.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:54 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2411146542' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-15T23:21:54.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:54 smithi120 ceph-mon[35250]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-15T23:21:54.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:54 smithi120 ceph-mon[35250]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-15T23:21:54.788 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:54 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2411146542' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-15T23:21:54.788 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:54 smithi089 ceph-mon[30933]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-15T23:21:54.788 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:54 smithi089 ceph-mon[30933]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-15T23:21:54.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.974+0000 7f5000dd2640 1 -- 172.21.15.89:0/205915870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc106a80 msgr2=0x7f4ffc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:54.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.974+0000 7f5000dd2640 1 --2- 172.21.15.89:0/205915870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc106a80 0x7f4ffc106e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009930 tx=0x7f4fec02a420 comp rx=0 tx=0).stop 2024-09-15T23:21:54.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.974+0000 7f5000dd2640 1 -- 172.21.15.89:0/205915870 shutdown_connections 2024-09-15T23:21:54.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.974+0000 7f5000dd2640 1 --2- 172.21.15.89:0/205915870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ffc101420 0x7f4ffc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:54.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.974+0000 7f5000dd2640 1 --2- 172.21.15.89:0/205915870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ffc100a80 0x7f4ffc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:54.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.974+0000 7f5000dd2640 1 --2- 172.21.15.89:0/205915870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc106a80 0x7f4ffc106e60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.974+0000 7f5000dd2640 1 -- 172.21.15.89:0/205915870 >> 172.21.15.89:0/205915870 conn(0x7f4ffc0fc7a0 msgr2=0x7f4ffc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.974+0000 7f5000dd2640 1 -- 172.21.15.89:0/205915870 shutdown_connections 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.974+0000 7f5000dd2640 1 -- 172.21.15.89:0/205915870 wait complete. 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f5000dd2640 1 Processor -- start 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f5000dd2640 1 -- start start 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f5000dd2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ffc100a80 0x7f4ffc1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f5000dd2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc101420 0x7f4ffc198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f5000dd2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ffc106a80 0x7f4ffc19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f5000dd2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc110ea0 con 0x7f4ffc101420 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f5000dd2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc110d20 con 0x7f4ffc100a80 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f5000dd2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc111020 con 0x7f4ffc106a80 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f4ff9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc101420 0x7f4ffc198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f4ffa59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ffc100a80 0x7f4ffc1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:54.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f4ff9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc101420 0x7f4ffc198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60326/0 (socket says 172.21.15.89:60326) 2024-09-15T23:21:54.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f4ff9d9b640 1 -- 172.21.15.89:0/3785559801 learned_addr learned my addr 172.21.15.89:0/3785559801 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:54.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.975+0000 7f4ffad9d640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ffc106a80 0x7f4ffc19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:54.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f4ff9d9b640 1 -- 172.21.15.89:0/3785559801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ffc106a80 msgr2=0x7f4ffc19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:54.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f4ff9d9b640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ffc106a80 0x7f4ffc19cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:54.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f4ff9d9b640 1 -- 172.21.15.89:0/3785559801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ffc100a80 msgr2=0x7f4ffc1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:54.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f4ff9d9b640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ffc100a80 0x7f4ffc1984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:54.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f4ff9d9b640 1 -- 172.21.15.89:0/3785559801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ffc19d470 con 0x7f4ffc101420 2024-09-15T23:21:54.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f4ffad9d640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ffc106a80 0x7f4ffc19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:54.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f4ff9d9b640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc101420 0x7f4ffc198a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000b4a0 tx=0x7f4ff000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:54.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f4feb7fe640 1 -- 172.21.15.89:0/3785559801 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0014070 con 0x7f4ffc101420 2024-09-15T23:21:54.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f4feb7fe640 1 -- 172.21.15.89:0/3785559801 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ff0010af0 con 0x7f4ffc101420 2024-09-15T23:21:54.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f5000dd2640 1 -- 172.21.15.89:0/3785559801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ffc19d760 con 0x7f4ffc101420 2024-09-15T23:21:54.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f4feb7fe640 1 -- 172.21.15.89:0/3785559801 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0017d10 con 0x7f4ffc101420 2024-09-15T23:21:54.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.976+0000 7f5000dd2640 1 -- 172.21.15.89:0/3785559801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ffc1a5040 con 0x7f4ffc101420 2024-09-15T23:21:54.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.977+0000 7f5000dd2640 1 -- 172.21.15.89:0/3785559801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ffc100ee0 con 0x7f4ffc101420 2024-09-15T23:21:54.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.978+0000 7f4feb7fe640 1 -- 172.21.15.89:0/3785559801 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f4ff0017070 con 0x7f4ffc101420 2024-09-15T23:21:54.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.979+0000 7f4feb7fe640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4fe40a3240 0x7f4fe40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:54.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.979+0000 7f4ffa59c640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4fe40a3240 0x7f4fe40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:54.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.979+0000 7f4feb7fe640 1 -- 172.21.15.89:0/3785559801 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7979+0+0 (secure 0 0 0) 0x7f4ff00cefa0 con 0x7f4ffc101420 2024-09-15T23:21:54.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.979+0000 7f4ffa59c640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4fe40a3240 0x7f4fe40a5700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009900 tx=0x7f4fec0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:54.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:54.982+0000 7f4feb7fe640 1 -- 172.21.15.89:0/3785559801 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4ff009be10 con 0x7f4ffc101420 2024-09-15T23:21:54.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:54 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2411146542' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-15T23:21:54.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:54 smithi174 ceph-mon[35586]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-15T23:21:54.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:54 smithi174 ceph-mon[35586]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-15T23:21:55.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.159+0000 7f5000dd2640 1 -- 172.21.15.89:0/3785559801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f4ffc199820 con 0x7f4ffc101420 2024-09-15T23:21:55.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.535+0000 7f4feb7fe640 1 -- 172.21.15.89:0/3785559801 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v81) ==== 157+0+0 (secure 0 0 0) 0x7f4ffc199820 con 0x7f4ffc101420 2024-09-15T23:21:55.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 -- 172.21.15.89:0/3785559801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4fe40a3240 msgr2=0x7f4fe40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4fe40a3240 0x7f4fe40a5700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009900 tx=0x7f4fec0058d0 comp rx=0 tx=0).stop 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 -- 172.21.15.89:0/3785559801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc101420 msgr2=0x7f4ffc198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc101420 0x7f4ffc198a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000b4a0 tx=0x7f4ff000b960 comp rx=0 tx=0).stop 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 -- 172.21.15.89:0/3785559801 shutdown_connections 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4fe40a3240 0x7f4fe40a5700 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ffc106a80 0x7f4ffc19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ffc101420 0x7f4ffc198a00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 --2- 172.21.15.89:0/3785559801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ffc100a80 0x7f4ffc1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 -- 172.21.15.89:0/3785559801 >> 172.21.15.89:0/3785559801 conn(0x7f4ffc0fc7a0 msgr2=0x7f4ffc06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 -- 172.21.15.89:0/3785559801 shutdown_connections 2024-09-15T23:21:55.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:55.538+0000 7f5000dd2640 1 -- 172.21.15.89:0/3785559801 wait complete. 2024-09-15T23:21:55.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:55 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] up:active 2024-09-15T23:21:55.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:55 smithi089 ceph-mon[30933]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:21:55.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:55 smithi089 ceph-mon[30933]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:21:55.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:55 smithi089 ceph-mon[30933]: pgmap v277: 97 pgs: 1 creating+activating, 27 creating+peering, 24 unknown, 45 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:55.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:55 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3785559801' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T23:21:55.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:55 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] up:active 2024-09-15T23:21:55.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:55 smithi120 ceph-mon[35250]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:21:55.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:55 smithi120 ceph-mon[35250]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:21:55.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:55 smithi120 ceph-mon[35250]: pgmap v277: 97 pgs: 1 creating+activating, 27 creating+peering, 24 unknown, 45 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:55.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:55 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3785559801' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T23:21:55.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:55 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] up:active 2024-09-15T23:21:55.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:55 smithi174 ceph-mon[35586]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:21:55.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:55 smithi174 ceph-mon[35586]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:21:55.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:55 smithi174 ceph-mon[35586]: pgmap v277: 97 pgs: 1 creating+activating, 27 creating+peering, 24 unknown, 45 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:21:55.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:55 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3785559801' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T23:21:56.248 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-15T23:21:56.490 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:56.795 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:56 smithi089 ceph-mon[30933]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-15T23:21:56.795 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:56 smithi089 ceph-mon[30933]: Cluster is now healthy 2024-09-15T23:21:56.795 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:56 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3785559801' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-15T23:21:56.795 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:56 smithi089 ceph-mon[30933]: osdmap e81: 12 total, 12 up, 12 in 2024-09-15T23:21:56.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:56 smithi120 ceph-mon[35250]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-15T23:21:56.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:56 smithi120 ceph-mon[35250]: Cluster is now healthy 2024-09-15T23:21:56.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:56 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3785559801' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-15T23:21:56.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:56 smithi120 ceph-mon[35250]: osdmap e81: 12 total, 12 up, 12 in 2024-09-15T23:21:56.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:56 smithi174 ceph-mon[35586]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-15T23:21:56.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:56 smithi174 ceph-mon[35586]: Cluster is now healthy 2024-09-15T23:21:56.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:56 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3785559801' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-15T23:21:56.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:56 smithi174 ceph-mon[35586]: osdmap e81: 12 total, 12 up, 12 in 2024-09-15T23:21:56.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/370413831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c78100310 msgr2=0x7f9c781006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:56.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 --2- 172.21.15.89:0/370413831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c78100310 0x7f9c781006f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009900 tx=0x7f9c6c02a410 comp rx=0 tx=0).stop 2024-09-15T23:21:56.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/370413831 shutdown_connections 2024-09-15T23:21:56.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 --2- 172.21.15.89:0/370413831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c78106f30 0x7f9c7810eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 --2- 172.21.15.89:0/370413831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c78100cc0 0x7f9c78106800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 --2- 172.21.15.89:0/370413831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c78100310 0x7f9c781006f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/370413831 >> 172.21.15.89:0/370413831 conn(0x7f9c78077210 msgr2=0x7f9c780fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/370413831 shutdown_connections 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/370413831 wait complete. 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 Processor -- start 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.980+0000 7f9c7ed9f640 1 -- start start 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c7ed9f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c78100310 0x7f9c78101290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c7ed9f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c78100cc0 0x7f9c781017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c7ed9f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c78106f30 0x7f9c78101d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c7ed9f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9c78112030 con 0x7f9c78100cc0 2024-09-15T23:21:56.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c7ed9f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9c78111eb0 con 0x7f9c78106f30 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c7ed9f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9c781121b0 con 0x7f9c78100310 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c77fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c78100cc0 0x7f9c781017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c77fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c78100cc0 0x7f9c781017d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57360/0 (socket says 172.21.15.89:57360) 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c77fff640 1 -- 172.21.15.89:0/730627582 learned_addr learned my addr 172.21.15.89:0/730627582 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c7cb3b640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c78100310 0x7f9c78101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c7d33c640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c78106f30 0x7f9c78101d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c77fff640 1 -- 172.21.15.89:0/730627582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c78100310 msgr2=0x7f9c78101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c77fff640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c78100310 0x7f9c78101290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c77fff640 1 -- 172.21.15.89:0/730627582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c78106f30 msgr2=0x7f9c78101d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c77fff640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c78106f30 0x7f9c78101d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c77fff640 1 -- 172.21.15.89:0/730627582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c781a6a70 con 0x7f9c78100cc0 2024-09-15T23:21:56.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c77fff640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c78100cc0 0x7f9c781017d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400b4a0 tx=0x7f9c6400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c75ffb640 1 -- 172.21.15.89:0/730627582 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c64014070 con 0x7f9c78100cc0 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c75ffb640 1 -- 172.21.15.89:0/730627582 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c64010af0 con 0x7f9c78100cc0 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c75ffb640 1 -- 172.21.15.89:0/730627582 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c64017d10 con 0x7f9c78100cc0 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.981+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/730627582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c781a6d60 con 0x7f9c78100cc0 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.982+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/730627582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c781a7240 con 0x7f9c78100cc0 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.983+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/730627582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c78100770 con 0x7f9c78100cc0 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.984+0000 7f9c75ffb640 1 -- 172.21.15.89:0/730627582 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f9c64017070 con 0x7f9c78100cc0 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.984+0000 7f9c75ffb640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c680a3240 0x7f9c680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.984+0000 7f9c7cb3b640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c680a3240 0x7f9c680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.984+0000 7f9c75ffb640 1 -- 172.21.15.89:0/730627582 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8277+0+0 (secure 0 0 0) 0x7f9c640cecb0 con 0x7f9c78100cc0 2024-09-15T23:21:56.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.985+0000 7f9c7cb3b640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c680a3240 0x7f9c680a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c000c00 tx=0x7f9c6c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:56.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:56.988+0000 7f9c75ffb640 1 -- 172.21.15.89:0/730627582 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9c6409b9a0 con 0x7f9c78100cc0 2024-09-15T23:21:57.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:57.154+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/730627582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9c7810a110 con 0x7f9c78100cc0 2024-09-15T23:21:57.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:57 smithi089 ceph-mon[30933]: pgmap v279: 97 pgs: 16 creating+peering, 81 active+clean; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-09-15T23:21:57.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:57 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/730627582' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-15T23:21:57.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:57 smithi120 ceph-mon[35250]: pgmap v279: 97 pgs: 16 creating+peering, 81 active+clean; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-09-15T23:21:57.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:57 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/730627582' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-15T23:21:57.988 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:57 smithi174 ceph-mon[35586]: pgmap v279: 97 pgs: 16 creating+peering, 81 active+clean; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 2 op/s 2024-09-15T23:21:57.989 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:57 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/730627582' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-15T23:21:58.473 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:21:58 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b[65175]: 2024-09-15T23:21:58.053+0000 7f9e6c009640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-15T23:21:58.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.543+0000 7f9c75ffb640 1 -- 172.21.15.89:0/730627582 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v83) ==== 228+0+0 (secure 0 0 0) 0x7f9c6409b2a0 con 0x7f9c78100cc0 2024-09-15T23:21:58.545 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data_ec' created 2024-09-15T23:21:58.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/730627582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c680a3240 msgr2=0x7f9c680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:58.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c680a3240 0x7f9c680a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c000c00 tx=0x7f9c6c0058d0 comp rx=0 tx=0).stop 2024-09-15T23:21:58.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/730627582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c78100cc0 msgr2=0x7f9c781017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:58.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c78100cc0 0x7f9c781017d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9c6400b4a0 tx=0x7f9c6400b960 comp rx=0 tx=0).stop 2024-09-15T23:21:58.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/730627582 shutdown_connections 2024-09-15T23:21:58.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c680a3240 0x7f9c680a5700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:58.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c78106f30 0x7f9c78101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:58.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c78100cc0 0x7f9c781017d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:58.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 --2- 172.21.15.89:0/730627582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c78100310 0x7f9c78101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:58.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/730627582 >> 172.21.15.89:0/730627582 conn(0x7f9c78077210 msgr2=0x7f9c781051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:58.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/730627582 shutdown_connections 2024-09-15T23:21:58.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:58.546+0000 7f9c7ed9f640 1 -- 172.21.15.89:0/730627582 wait complete. 2024-09-15T23:21:58.812 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:58 smithi089 ceph-mon[30933]: osdmap e82: 12 total, 12 up, 12 in 2024-09-15T23:21:58.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:58 smithi120 ceph-mon[35250]: osdmap e82: 12 total, 12 up, 12 in 2024-09-15T23:21:58.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:58 smithi174 ceph-mon[35586]: osdmap e82: 12 total, 12 up, 12 in 2024-09-15T23:21:59.247 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-15T23:21:59.496 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:59.903 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:59 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/730627582' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-15T23:21:59.903 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:59 smithi089 ceph-mon[30933]: osdmap e83: 12 total, 12 up, 12 in 2024-09-15T23:21:59.903 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:59 smithi089 ceph-mon[30933]: pgmap v282: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-09-15T23:21:59.903 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:21:59 smithi089 ceph-mon[30933]: osdmap e84: 12 total, 12 up, 12 in 2024-09-15T23:21:59.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:59 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/730627582' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-15T23:21:59.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:59 smithi120 ceph-mon[35250]: osdmap e83: 12 total, 12 up, 12 in 2024-09-15T23:21:59.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:59 smithi120 ceph-mon[35250]: pgmap v282: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-09-15T23:21:59.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:21:59 smithi120 ceph-mon[35250]: osdmap e84: 12 total, 12 up, 12 in 2024-09-15T23:21:59.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:59 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/730627582' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-15T23:21:59.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:59 smithi174 ceph-mon[35586]: osdmap e83: 12 total, 12 up, 12 in 2024-09-15T23:21:59.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:59 smithi174 ceph-mon[35586]: pgmap v282: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-09-15T23:21:59.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:21:59 smithi174 ceph-mon[35586]: osdmap e84: 12 total, 12 up, 12 in 2024-09-15T23:22:00.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.986+0000 7effa964e640 1 -- 172.21.15.89:0/1136735961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effa410c170 msgr2=0x7effa410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.986+0000 7effa964e640 1 --2- 172.21.15.89:0/1136735961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effa410c170 0x7effa410e610 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009ca0 tx=0x7eff9c02a950 comp rx=0 tx=0).stop 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.986+0000 7effa964e640 1 -- 172.21.15.89:0/1136735961 shutdown_connections 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.986+0000 7effa964e640 1 --2- 172.21.15.89:0/1136735961 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effa410c170 0x7effa410e610 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.986+0000 7effa964e640 1 --2- 172.21.15.89:0/1136735961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa4103660 0x7effa410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.986+0000 7effa964e640 1 --2- 172.21.15.89:0/1136735961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa4102d40 0x7effa4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.986+0000 7effa964e640 1 -- 172.21.15.89:0/1136735961 >> 172.21.15.89:0/1136735961 conn(0x7effa40fc7a0 msgr2=0x7effa40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.986+0000 7effa964e640 1 -- 172.21.15.89:0/1136735961 shutdown_connections 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.986+0000 7effa964e640 1 -- 172.21.15.89:0/1136735961 wait complete. 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa964e640 1 Processor -- start 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa964e640 1 -- start start 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa964e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effa4102d40 0x7effa4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa964e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa4103660 0x7effa4198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:00.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa964e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa410c170 0x7effa419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa964e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7effa4111f70 con 0x7effa4102d40 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa964e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7effa4111df0 con 0x7effa410c170 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa964e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7effa41120f0 con 0x7effa4103660 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effa4102d40 0x7effa4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effa4102d40 0x7effa4198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57370/0 (socket says 172.21.15.89:57370) 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa2ffd640 1 -- 172.21.15.89:0/2709346609 learned_addr learned my addr 172.21.15.89:0/2709346609 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7eff9a7fc640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa4103660 0x7effa4198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.987+0000 7effa37fe640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa410c170 0x7effa419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa2ffd640 1 -- 172.21.15.89:0/2709346609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa4103660 msgr2=0x7effa4198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa2ffd640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa4103660 0x7effa4198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa2ffd640 1 -- 172.21.15.89:0/2709346609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa410c170 msgr2=0x7effa419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:00.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa2ffd640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa410c170 0x7effa419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa2ffd640 1 -- 172.21.15.89:0/2709346609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa419d3c0 con 0x7effa4102d40 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7eff9a7fc640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa4103660 0x7effa4198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa37fe640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa410c170 0x7effa419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa2ffd640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effa4102d40 0x7effa4198410 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff9400d770 tx=0x7eff9400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa0ff9640 1 -- 172.21.15.89:0/2709346609 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94017070 con 0x7effa4102d40 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa964e640 1 -- 172.21.15.89:0/2709346609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effa419d6b0 con 0x7effa4102d40 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa0ff9640 1 -- 172.21.15.89:0/2709346609 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff94005500 con 0x7effa4102d40 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa0ff9640 1 -- 172.21.15.89:0/2709346609 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94004e00 con 0x7effa4102d40 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.988+0000 7effa964e640 1 -- 172.21.15.89:0/2709346609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effa41a4f90 con 0x7effa4102d40 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.989+0000 7effa964e640 1 -- 172.21.15.89:0/2709346609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effa4103120 con 0x7effa4102d40 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.990+0000 7effa0ff9640 1 -- 172.21.15.89:0/2709346609 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7eff94004fa0 con 0x7effa4102d40 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.991+0000 7effa0ff9640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff900a3240 0x7eff900a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.991+0000 7eff9a7fc640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff900a3240 0x7eff900a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:00.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.991+0000 7effa0ff9640 1 -- 172.21.15.89:0/2709346609 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8770+0+0 (secure 0 0 0) 0x7eff940cf9c0 con 0x7effa4102d40 2024-09-15T23:22:00.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.991+0000 7eff9a7fc640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff900a3240 0x7eff900a5700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7eff8c005fd0 tx=0x7eff8c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:00.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:21:59.994+0000 7effa0ff9640 1 -- 172.21.15.89:0/2709346609 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7eff9409c4c0 con 0x7effa4102d40 2024-09-15T23:22:00.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.159+0000 7effa964e640 1 -- 172.21.15.89:0/2709346609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7effa41a5230 con 0x7effa4102d40 2024-09-15T23:22:00.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.547+0000 7effa0ff9640 1 -- 172.21.15.89:0/2709346609 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) ==== 169+0+0 (secure 0 0 0) 0x7eff9409bdc0 con 0x7effa4102d40 2024-09-15T23:22:00.548 INFO:teuthology.orchestra.run.smithi089.stderr:set pool 4 allow_ec_overwrites to true 2024-09-15T23:22:00.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 -- 172.21.15.89:0/2709346609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff900a3240 msgr2=0x7eff900a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:00.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff900a3240 0x7eff900a5700 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7eff8c005fd0 tx=0x7eff8c005850 comp rx=0 tx=0).stop 2024-09-15T23:22:00.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 -- 172.21.15.89:0/2709346609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effa4102d40 msgr2=0x7effa4198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:00.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effa4102d40 0x7effa4198410 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff9400d770 tx=0x7eff9400dc30 comp rx=0 tx=0).stop 2024-09-15T23:22:00.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 -- 172.21.15.89:0/2709346609 shutdown_connections 2024-09-15T23:22:00.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff900a3240 0x7eff900a5700 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:00.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa410c170 0x7effa419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:00.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effa4103660 0x7effa4198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:00.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 --2- 172.21.15.89:0/2709346609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effa4102d40 0x7effa4198410 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:00.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 -- 172.21.15.89:0/2709346609 >> 172.21.15.89:0/2709346609 conn(0x7effa40fc7a0 msgr2=0x7effa40fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:00.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 -- 172.21.15.89:0/2709346609 shutdown_connections 2024-09-15T23:22:00.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:00.550+0000 7effa964e640 1 -- 172.21.15.89:0/2709346609 wait complete. 2024-09-15T23:22:00.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:00 smithi089 ceph-mon[30933]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:22:00.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:00 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2709346609' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-15T23:22:00.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:00 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2709346609' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-15T23:22:00.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:00 smithi089 ceph-mon[30933]: osdmap e85: 12 total, 12 up, 12 in 2024-09-15T23:22:00.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:00 smithi120 ceph-mon[35250]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:22:00.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:00 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2709346609' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-15T23:22:00.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:00 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2709346609' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-15T23:22:00.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:00 smithi120 ceph-mon[35250]: osdmap e85: 12 total, 12 up, 12 in 2024-09-15T23:22:00.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:00 smithi174 ceph-mon[35586]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:22:00.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:00 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2709346609' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-15T23:22:00.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:00 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2709346609' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-15T23:22:00.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:00 smithi174 ceph-mon[35586]: osdmap e85: 12 total, 12 up, 12 in 2024-09-15T23:22:01.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-15T23:22:01.531 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:01.831 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:01 smithi089 ceph-mon[30933]: pgmap v285: 129 pgs: 3 creating+peering, 16 unknown, 110 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-09-15T23:22:01.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:01 smithi120 ceph-mon[35250]: pgmap v285: 129 pgs: 3 creating+peering, 16 unknown, 110 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-09-15T23:22:01.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:01 smithi174 ceph-mon[35586]: pgmap v285: 129 pgs: 3 creating+peering, 16 unknown, 110 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-09-15T23:22:02.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 -- 172.21.15.89:0/3020771616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4120102d40 msgr2=0x7f4120103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 --2- 172.21.15.89:0/3020771616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4120102d40 0x7f4120103120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4110009ca0 tx=0x7f411002a950 comp rx=0 tx=0).stop 2024-09-15T23:22:02.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 -- 172.21.15.89:0/3020771616 shutdown_connections 2024-09-15T23:22:02.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 --2- 172.21.15.89:0/3020771616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f412010c170 0x7f412010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 --2- 172.21.15.89:0/3020771616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4120103660 0x7f412010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 --2- 172.21.15.89:0/3020771616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4120102d40 0x7f4120103120 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 -- 172.21.15.89:0/3020771616 >> 172.21.15.89:0/3020771616 conn(0x7f41200fc7a0 msgr2=0x7f41200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:02.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 -- 172.21.15.89:0/3020771616 shutdown_connections 2024-09-15T23:22:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 -- 172.21.15.89:0/3020771616 wait complete. 2024-09-15T23:22:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 Processor -- start 2024-09-15T23:22:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.038+0000 7f4125fa3640 1 -- start start 2024-09-15T23:22:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f4125fa3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4120102d40 0x7f41201009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f4125fa3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4120103660 0x7f4120100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f4125fa3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f412010c170 0x7f4120101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f4125fa3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4120111f70 con 0x7f4120102d40 2024-09-15T23:22:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f4125fa3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4120111df0 con 0x7f4120103660 2024-09-15T23:22:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f4125fa3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f41201120f0 con 0x7f412010c170 2024-09-15T23:22:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4120102d40 0x7f41201009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4120102d40 0x7f41201009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57400/0 (socket says 172.21.15.89:57400) 2024-09-15T23:22:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411f7fe640 1 -- 172.21.15.89:0/3530204561 learned_addr learned my addr 172.21.15.89:0/3530204561 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411effd640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4120103660 0x7f4120100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411ffff640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f412010c170 0x7f4120101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411f7fe640 1 -- 172.21.15.89:0/3530204561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f412010c170 msgr2=0x7f4120101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411f7fe640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f412010c170 0x7f4120101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411f7fe640 1 -- 172.21.15.89:0/3530204561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4120103660 msgr2=0x7f4120100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411f7fe640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4120103660 0x7f4120100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411f7fe640 1 -- 172.21.15.89:0/3530204561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41201a6890 con 0x7f4120102d40 2024-09-15T23:22:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411ffff640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f412010c170 0x7f4120101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411effd640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4120103660 0x7f4120100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:02.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411f7fe640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4120102d40 0x7f41201009a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4110005cc0 tx=0x7f411002f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:02.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.039+0000 7f411cff9640 1 -- 172.21.15.89:0/3530204561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110034040 con 0x7f4120102d40 2024-09-15T23:22:02.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.040+0000 7f411cff9640 1 -- 172.21.15.89:0/3530204561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4110030920 con 0x7f4120102d40 2024-09-15T23:22:02.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.040+0000 7f4125fa3640 1 -- 172.21.15.89:0/3530204561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41201a6b20 con 0x7f4120102d40 2024-09-15T23:22:02.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.040+0000 7f411cff9640 1 -- 172.21.15.89:0/3530204561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110004830 con 0x7f4120102d40 2024-09-15T23:22:02.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.040+0000 7f4125fa3640 1 -- 172.21.15.89:0/3530204561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41201a7000 con 0x7f4120102d40 2024-09-15T23:22:02.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.040+0000 7f4125fa3640 1 -- 172.21.15.89:0/3530204561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4120103120 con 0x7f4120102d40 2024-09-15T23:22:02.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.042+0000 7f411cff9640 1 -- 172.21.15.89:0/3530204561 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f4110037020 con 0x7f4120102d40 2024-09-15T23:22:02.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.042+0000 7f411cff9640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41080a3240 0x7f41080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.042+0000 7f411effd640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41080a3240 0x7f41080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.042+0000 7f411cff9640 1 -- 172.21.15.89:0/3530204561 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8770+0+0 (secure 0 0 0) 0x7f41100f0c80 con 0x7f4120102d40 2024-09-15T23:22:02.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.043+0000 7f411effd640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41080a3240 0x7f41080a5700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f41140058c0 tx=0x7f4114005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:02.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.045+0000 7f411cff9640 1 -- 172.21.15.89:0/3530204561 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f41100bd780 con 0x7f4120102d40 2024-09-15T23:22:02.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.245+0000 7f4125fa3640 1 -- 172.21.15.89:0/3530204561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f4120101ed0 con 0x7f4120102d40 2024-09-15T23:22:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.581+0000 7f411cff9640 1 -- 172.21.15.89:0/3530204561 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f4120101ed0 con 0x7f4120102d40 2024-09-15T23:22:02.582 INFO:teuthology.orchestra.run.smithi089.stderr:added data pool 4 to fsmap 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.583+0000 7f4125fa3640 1 -- 172.21.15.89:0/3530204561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41080a3240 msgr2=0x7f41080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.583+0000 7f4125fa3640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41080a3240 0x7f41080a5700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f41140058c0 tx=0x7f4114005850 comp rx=0 tx=0).stop 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.583+0000 7f4125fa3640 1 -- 172.21.15.89:0/3530204561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4120102d40 msgr2=0x7f41201009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.583+0000 7f4125fa3640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4120102d40 0x7f41201009a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4110005cc0 tx=0x7f411002f630 comp rx=0 tx=0).stop 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.583+0000 7f4125fa3640 1 -- 172.21.15.89:0/3530204561 shutdown_connections 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.583+0000 7f4125fa3640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41080a3240 0x7f41080a5700 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.583+0000 7f4125fa3640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f412010c170 0x7f4120101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.583+0000 7f4125fa3640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4120103660 0x7f4120100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.583+0000 7f4125fa3640 1 --2- 172.21.15.89:0/3530204561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4120102d40 0x7f41201009a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.583+0000 7f4125fa3640 1 -- 172.21.15.89:0/3530204561 >> 172.21.15.89:0/3530204561 conn(0x7f41200fc7a0 msgr2=0x7f41200fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:02.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.584+0000 7f4125fa3640 1 -- 172.21.15.89:0/3530204561 shutdown_connections 2024-09-15T23:22:02.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:02.584+0000 7f4125fa3640 1 -- 172.21.15.89:0/3530204561 wait complete. 2024-09-15T23:22:02.960 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:02 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3530204561' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-15T23:22:02.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:02 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3530204561' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-15T23:22:02.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:02 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3530204561' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-15T23:22:03.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:22:03.527 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:03.907 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:03 smithi089 ceph-mon[30933]: osdmap e86: 12 total, 12 up, 12 in 2024-09-15T23:22:03.907 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:03 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3530204561' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-15T23:22:03.907 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:03 smithi089 ceph-mon[30933]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:22:03.907 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:03 smithi089 ceph-mon[30933]: pgmap v287: 129 pgs: 3 creating+peering, 4 unknown, 122 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:03.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:03 smithi120 ceph-mon[35250]: osdmap e86: 12 total, 12 up, 12 in 2024-09-15T23:22:03.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:03 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3530204561' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-15T23:22:03.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:03 smithi120 ceph-mon[35250]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:22:03.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:03 smithi120 ceph-mon[35250]: pgmap v287: 129 pgs: 3 creating+peering, 4 unknown, 122 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:03.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:03 smithi174 ceph-mon[35586]: osdmap e86: 12 total, 12 up, 12 in 2024-09-15T23:22:03.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:03 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3530204561' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-15T23:22:03.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:03 smithi174 ceph-mon[35586]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:22:03.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:03 smithi174 ceph-mon[35586]: pgmap v287: 129 pgs: 3 creating+peering, 4 unknown, 122 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 -- 172.21.15.89:0/3939991405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41d8102ef0 msgr2=0x7f41d810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 --2- 172.21.15.89:0/3939991405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41d8102ef0 0x7f41d810da40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009900 tx=0x7f41c802a480 comp rx=0 tx=0).stop 2024-09-15T23:22:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 -- 172.21.15.89:0/3939991405 shutdown_connections 2024-09-15T23:22:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 --2- 172.21.15.89:0/3939991405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41d8102ef0 0x7f41d810da40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 --2- 172.21.15.89:0/3939991405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d8102570 0x7f41d81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 --2- 172.21.15.89:0/3939991405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8100e20 0x7f41d8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 -- 172.21.15.89:0/3939991405 >> 172.21.15.89:0/3939991405 conn(0x7f41d80fc7d0 msgr2=0x7f41d80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 -- 172.21.15.89:0/3939991405 shutdown_connections 2024-09-15T23:22:04.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 -- 172.21.15.89:0/3939991405 wait complete. 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 Processor -- start 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.985+0000 7f41df5c6640 1 -- start start 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41df5c6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41d8100e20 0x7f41d81972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41df5c6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d8102570 0x7f41d81977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41df5c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8102ef0 0x7f41d819bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41df5c6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f41d8110f80 con 0x7f41d8100e20 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41df5c6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41d8110e00 con 0x7f41d8102570 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41df5c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f41d8111100 con 0x7f41d8102ef0 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41dd362640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41d8100e20 0x7f41d81972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41dd362640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41d8100e20 0x7f41d81972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57426/0 (socket says 172.21.15.89:57426) 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41dd362640 1 -- 172.21.15.89:0/1474476910 learned_addr learned my addr 172.21.15.89:0/1474476910 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:04.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41dcb61640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d8102570 0x7f41d81977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41ddb63640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8102ef0 0x7f41d819bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41dd362640 1 -- 172.21.15.89:0/1474476910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8102ef0 msgr2=0x7f41d819bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41dd362640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8102ef0 0x7f41d819bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41dd362640 1 -- 172.21.15.89:0/1474476910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d8102570 msgr2=0x7f41d81977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41dd362640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d8102570 0x7f41d81977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41dd362640 1 -- 172.21.15.89:0/1474476910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d819c250 con 0x7f41d8100e20 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41ddb63640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8102ef0 0x7f41d819bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41dd362640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41d8100e20 0x7f41d81972a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f41d000c890 tx=0x7f41d000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41d67fc640 1 -- 172.21.15.89:0/1474476910 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d0004220 con 0x7f41d8100e20 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41df5c6640 1 -- 172.21.15.89:0/1474476910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41d819c540 con 0x7f41d8100e20 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41d67fc640 1 -- 172.21.15.89:0/1474476910 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41d00043c0 con 0x7f41d8100e20 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41d67fc640 1 -- 172.21.15.89:0/1474476910 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d0004a00 con 0x7f41d8100e20 2024-09-15T23:22:04.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.986+0000 7f41df5c6640 1 -- 172.21.15.89:0/1474476910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41d81a3e20 con 0x7f41d8100e20 2024-09-15T23:22:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.987+0000 7f41df5c6640 1 -- 172.21.15.89:0/1474476910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41d8101200 con 0x7f41d8100e20 2024-09-15T23:22:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.988+0000 7f41d67fc640 1 -- 172.21.15.89:0/1474476910 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f41d00079b0 con 0x7f41d8100e20 2024-09-15T23:22:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.989+0000 7f41d67fc640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41c00a3240 0x7f41c00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.989+0000 7f41dcb61640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41c00a3240 0x7f41c00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.989+0000 7f41d67fc640 1 -- 172.21.15.89:0/1474476910 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f41d00cf900 con 0x7f41d8100e20 2024-09-15T23:22:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.989+0000 7f41dcb61640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41c00a3240 0x7f41c00a5700 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f41cc007350 tx=0x7f41cc0072e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:04.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:03.993+0000 7f41d67fc640 1 -- 172.21.15.89:0/1474476910 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f41d0014030 con 0x7f41d8100e20 2024-09-15T23:22:04.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.194+0000 7f41df5c6640 1 -- 172.21.15.89:0/1474476910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41d8198600 con 0x7f41d8100e20 2024-09-15T23:22:04.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.195+0000 7f41d67fc640 1 -- 172.21.15.89:0/1474476910 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11851 (secure 0 0 0) 0x7f41d009c3e0 con 0x7f41d8100e20 2024-09-15T23:22:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:04.196 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"btime":"2024-09-15T23:22:02:573423+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:22:02.246771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24452,"qdb_cluster":[24452]},"id":1}]} 2024-09-15T23:22:04.197 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 -- 172.21.15.89:0/1474476910 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41c00a3240 msgr2=0x7f41c00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41c00a3240 0x7f41c00a5700 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f41cc007350 tx=0x7f41cc0072e0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 -- 172.21.15.89:0/1474476910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41d8100e20 msgr2=0x7f41d81972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41d8100e20 0x7f41d81972a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f41d000c890 tx=0x7f41d000cd50 comp rx=0 tx=0).stop 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 -- 172.21.15.89:0/1474476910 shutdown_connections 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41c00a3240 0x7f41c00a5700 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41d8102ef0 0x7f41d819bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41d8102570 0x7f41d81977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 --2- 172.21.15.89:0/1474476910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41d8100e20 0x7f41d81972a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 -- 172.21.15.89:0/1474476910 >> 172.21.15.89:0/1474476910 conn(0x7f41d80fc7d0 msgr2=0x7f41d81067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 -- 172.21.15.89:0/1474476910 shutdown_connections 2024-09-15T23:22:04.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:04.197+0000 7f41df5c6640 1 -- 172.21.15.89:0/1474476910 wait complete. 2024-09-15T23:22:04.871 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:04 smithi089 ceph-mon[30933]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-15T23:22:04.871 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:04 smithi089 ceph-mon[30933]: Cluster is now healthy 2024-09-15T23:22:04.871 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:04 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1474476910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:04.898 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:22:04.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:04 smithi120 ceph-mon[35250]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-15T23:22:04.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:04 smithi120 ceph-mon[35250]: Cluster is now healthy 2024-09-15T23:22:04.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:04 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1474476910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:04.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:04 smithi174 ceph-mon[35586]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-15T23:22:04.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:04 smithi174 ceph-mon[35586]: Cluster is now healthy 2024-09-15T23:22:04.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:04 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1474476910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:05.143 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.640+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/3524074454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e341055b0 msgr2=0x7f6e3410d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.640+0000 7f6e3b6aa640 1 --2- 172.21.15.89:0/3524074454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e341055b0 0x7f6e3410d130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c009ca0 tx=0x7f6e2c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:22:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.640+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/3524074454 shutdown_connections 2024-09-15T23:22:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.640+0000 7f6e3b6aa640 1 --2- 172.21.15.89:0/3524074454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e341055b0 0x7f6e3410d130 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.640+0000 7f6e3b6aa640 1 --2- 172.21.15.89:0/3524074454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e34100e30 0x7f6e34104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.640+0000 7f6e3b6aa640 1 --2- 172.21.15.89:0/3524074454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e34100480 0x7f6e34100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.640+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/3524074454 >> 172.21.15.89:0/3524074454 conn(0x7f6e34077210 msgr2=0x7f6e340fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:05.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.640+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/3524074454 shutdown_connections 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.640+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/3524074454 wait complete. 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e3b6aa640 1 Processor -- start 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e3b6aa640 1 -- start start 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e3b6aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e34100480 0x7f6e3419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e3b6aa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e34100e30 0x7f6e3419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e3b6aa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e341055b0 0x7f6e3419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e3b6aa640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3410fd90 con 0x7f6e34100480 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e3b6aa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3410fc10 con 0x7f6e34100e30 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e3b6aa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3410ff10 con 0x7f6e341055b0 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e39446640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e34100480 0x7f6e3419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e39446640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e34100480 0x7f6e3419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39746/0 (socket says 172.21.15.89:39746) 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e39446640 1 -- 172.21.15.89:0/2023567962 learned_addr learned my addr 172.21.15.89:0/2023567962 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:05.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e38c45640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e34100e30 0x7f6e3419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e39446640 1 -- 172.21.15.89:0/2023567962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e341055b0 msgr2=0x7f6e3419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e39446640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e341055b0 0x7f6e3419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e39446640 1 -- 172.21.15.89:0/2023567962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e34100e30 msgr2=0x7f6e3419aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e39446640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e34100e30 0x7f6e3419aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e39446640 1 -- 172.21.15.89:0/2023567962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3419f610 con 0x7f6e34100480 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.641+0000 7f6e38c45640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e34100e30 0x7f6e3419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.642+0000 7f6e39446640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e34100480 0x7f6e3419a660 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000c890 tx=0x7f6e3000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.642+0000 7f6e2a7fc640 1 -- 172.21.15.89:0/2023567962 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30004220 con 0x7f6e34100480 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.642+0000 7f6e2a7fc640 1 -- 172.21.15.89:0/2023567962 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e300043c0 con 0x7f6e34100480 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.642+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/2023567962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e3419f900 con 0x7f6e34100480 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.642+0000 7f6e2a7fc640 1 -- 172.21.15.89:0/2023567962 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e300049e0 con 0x7f6e34100480 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.642+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/2023567962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e341a71e0 con 0x7f6e34100480 2024-09-15T23:22:05.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.643+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/2023567962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e34100860 con 0x7f6e34100480 2024-09-15T23:22:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.644+0000 7f6e2a7fc640 1 -- 172.21.15.89:0/2023567962 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6e300079b0 con 0x7f6e34100480 2024-09-15T23:22:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.644+0000 7f6e2a7fc640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6e200a3240 0x7f6e200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.645+0000 7f6e38c45640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6e200a3240 0x7f6e200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.645+0000 7f6e2a7fc640 1 -- 172.21.15.89:0/2023567962 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f6e300cfc90 con 0x7f6e34100480 2024-09-15T23:22:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.645+0000 7f6e38c45640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6e200a3240 0x7f6e200a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6e24005fd0 tx=0x7f6e2400c540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:05.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.648+0000 7f6e2a7fc640 1 -- 172.21.15.89:0/2023567962 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6e30014030 con 0x7f6e34100480 2024-09-15T23:22:05.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:05 smithi089 ceph-mon[30933]: pgmap v288: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:05.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.815+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/2023567962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e3419b9c0 con 0x7f6e34100480 2024-09-15T23:22:05.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.816+0000 7f6e2a7fc640 1 -- 172.21.15.89:0/2023567962 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24199 (secure 0 0 0) 0x7f6e3009c770 con 0x7f6e34100480 2024-09-15T23:22:05.818 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:05.818 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":86,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:22:02.246640+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:22:05.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/2023567962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6e200a3240 msgr2=0x7f6e200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:05.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6e200a3240 0x7f6e200a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6e24005fd0 tx=0x7f6e2400c540 comp rx=0 tx=0).stop 2024-09-15T23:22:05.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/2023567962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e34100480 msgr2=0x7f6e3419a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:05.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e34100480 0x7f6e3419a660 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6e3000c890 tx=0x7f6e3000cd50 comp rx=0 tx=0).stop 2024-09-15T23:22:05.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/2023567962 shutdown_connections 2024-09-15T23:22:05.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6e200a3240 0x7f6e200a5700 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:05.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e341055b0 0x7f6e3419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:05.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6e34100e30 0x7f6e3419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:05.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 --2- 172.21.15.89:0/2023567962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6e34100480 0x7f6e3419a660 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:05.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/2023567962 >> 172.21.15.89:0/2023567962 conn(0x7f6e34077210 msgr2=0x7f6e34102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:05.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/2023567962 shutdown_connections 2024-09-15T23:22:05.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:05.818+0000 7f6e3b6aa640 1 -- 172.21.15.89:0/2023567962 wait complete. 2024-09-15T23:22:05.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:05 smithi120 ceph-mon[35250]: pgmap v288: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:05.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:05 smithi174 ceph-mon[35586]: pgmap v288: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:06.543 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:22:06.789 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:06.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:06 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2023567962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:06.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:06 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2023567962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:06.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:06 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2023567962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:07.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.264+0000 7f81fe32b640 1 -- 172.21.15.89:0/3381205959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81f8101de0 msgr2=0x7f81f81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.264+0000 7f81fe32b640 1 --2- 172.21.15.89:0/3381205959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81f8101de0 0x7f81f81021c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009900 tx=0x7f81e802a410 comp rx=0 tx=0).stop 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.264+0000 7f81fe32b640 1 -- 172.21.15.89:0/3381205959 shutdown_connections 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.264+0000 7f81fe32b640 1 --2- 172.21.15.89:0/3381205959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f8106f10 0x7f81f810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.264+0000 7f81fe32b640 1 --2- 172.21.15.89:0/3381205959 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f8102790 0x7f81f81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.264+0000 7f81fe32b640 1 --2- 172.21.15.89:0/3381205959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81f8101de0 0x7f81f81021c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.264+0000 7f81fe32b640 1 -- 172.21.15.89:0/3381205959 >> 172.21.15.89:0/3381205959 conn(0x7f81f80fd8c0 msgr2=0x7f81f80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.264+0000 7f81fe32b640 1 -- 172.21.15.89:0/3381205959 shutdown_connections 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.264+0000 7f81fe32b640 1 -- 172.21.15.89:0/3381205959 wait complete. 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fe32b640 1 Processor -- start 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fe32b640 1 -- start start 2024-09-15T23:22:07.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fe32b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f8101de0 0x7f81f81984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fe32b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f8102790 0x7f81f8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fe32b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81f8106f10 0x7f81f819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fe32b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f81f8111f90 con 0x7f81f8106f10 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fe32b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81f8111e10 con 0x7f81f8102790 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fe32b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f81f8112110 con 0x7f81f8101de0 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fc8c8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81f8106f10 0x7f81f819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fc8c8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81f8106f10 0x7f81f819cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39776/0 (socket says 172.21.15.89:39776) 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.265+0000 7f81fc8c8640 1 -- 172.21.15.89:0/1125988768 learned_addr learned my addr 172.21.15.89:0/1125988768 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81f77fe640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f8102790 0x7f81f8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81f7fff640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f8101de0 0x7f81f81984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81fc8c8640 1 -- 172.21.15.89:0/1125988768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f8101de0 msgr2=0x7f81f81984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:07.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81fc8c8640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f8101de0 0x7f81f81984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81fc8c8640 1 -- 172.21.15.89:0/1125988768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f8102790 msgr2=0x7f81f8198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81fc8c8640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f8102790 0x7f81f8198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81fc8c8640 1 -- 172.21.15.89:0/1125988768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f819d480 con 0x7f81f8106f10 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81f7fff640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f8101de0 0x7f81f81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81f77fe640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f8102790 0x7f81f8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81fc8c8640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81f8106f10 0x7f81f819cda0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f81f000cad0 tx=0x7f81f000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81f57fa640 1 -- 172.21.15.89:0/1125988768 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f000ae20 con 0x7f81f8106f10 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81fe32b640 1 -- 172.21.15.89:0/1125988768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81f819d770 con 0x7f81f8106f10 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81f57fa640 1 -- 172.21.15.89:0/1125988768 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81f0005020 con 0x7f81f8106f10 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81f57fa640 1 -- 172.21.15.89:0/1125988768 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f000f6f0 con 0x7f81f8106f10 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.266+0000 7f81fe32b640 1 -- 172.21.15.89:0/1125988768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81f81a5050 con 0x7f81f8106f10 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.267+0000 7f81fe32b640 1 -- 172.21.15.89:0/1125988768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81f81021c0 con 0x7f81f8106f10 2024-09-15T23:22:07.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.268+0000 7f81f57fa640 1 -- 172.21.15.89:0/1125988768 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f81f00051d0 con 0x7f81f8106f10 2024-09-15T23:22:07.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.269+0000 7f81f57fa640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81e00a3240 0x7f81e00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:07.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.269+0000 7f81f7fff640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81e00a3240 0x7f81e00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:07.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.269+0000 7f81f57fa640 1 -- 172.21.15.89:0/1125988768 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f81f00cfca0 con 0x7f81f8106f10 2024-09-15T23:22:07.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.269+0000 7f81f7fff640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81e00a3240 0x7f81e00a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f81e8000c00 tx=0x7f81e8032d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:07.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.273+0000 7f81f57fa640 1 -- 172.21.15.89:0/1125988768 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f81f0014030 con 0x7f81f8106f10 2024-09-15T23:22:07.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.441+0000 7f81fe32b640 1 -- 172.21.15.89:0/1125988768 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81f810a150 con 0x7f81f8106f10 2024-09-15T23:22:07.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.444+0000 7f81f57fa640 1 -- 172.21.15.89:0/1125988768 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24199 (secure 0 0 0) 0x7f81f009c780 con 0x7f81f8106f10 2024-09-15T23:22:07.445 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:07.445 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":86,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:22:02.246640+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:22:07.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 -- 172.21.15.89:0/1125988768 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81e00a3240 msgr2=0x7f81e00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:07.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81e00a3240 0x7f81e00a5700 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f81e8000c00 tx=0x7f81e8032d50 comp rx=0 tx=0).stop 2024-09-15T23:22:07.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 -- 172.21.15.89:0/1125988768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81f8106f10 msgr2=0x7f81f819cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:07.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81f8106f10 0x7f81f819cda0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f81f000cad0 tx=0x7f81f000cf90 comp rx=0 tx=0).stop 2024-09-15T23:22:07.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 -- 172.21.15.89:0/1125988768 shutdown_connections 2024-09-15T23:22:07.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81e00a3240 0x7f81e00a5700 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81f8106f10 0x7f81f819cda0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f8102790 0x7f81f8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 --2- 172.21.15.89:0/1125988768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f8101de0 0x7f81f81984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 -- 172.21.15.89:0/1125988768 >> 172.21.15.89:0/1125988768 conn(0x7f81f80fd8c0 msgr2=0x7f81f80fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:07.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 -- 172.21.15.89:0/1125988768 shutdown_connections 2024-09-15T23:22:07.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:07.447+0000 7f81fe32b640 1 -- 172.21.15.89:0/1125988768 wait complete. 2024-09-15T23:22:07.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:07 smithi089 ceph-mon[30933]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:07.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:07 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1125988768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:07.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:07 smithi120 ceph-mon[35250]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:07.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:07 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1125988768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:07.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:07 smithi174 ceph-mon[35586]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:07.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:07 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1125988768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:08.194 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:22:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:08.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.898+0000 7f569cea4640 1 -- 172.21.15.89:0/3837607194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5698100a80 msgr2=0x7f5698100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:08.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.898+0000 7f569cea4640 1 --2- 172.21.15.89:0/3837607194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5698100a80 0x7f5698100ee0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f568c009930 tx=0x7f568c02a420 comp rx=0 tx=0).stop 2024-09-15T23:22:08.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.898+0000 7f569cea4640 1 -- 172.21.15.89:0/3837607194 shutdown_connections 2024-09-15T23:22:08.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.898+0000 7f569cea4640 1 --2- 172.21.15.89:0/3837607194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5698101420 0x7f569810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:08.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.898+0000 7f569cea4640 1 --2- 172.21.15.89:0/3837607194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5698100a80 0x7f5698100ee0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:08.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.898+0000 7f569cea4640 1 --2- 172.21.15.89:0/3837607194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5698106a80 0x7f5698106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:08.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.898+0000 7f569cea4640 1 -- 172.21.15.89:0/3837607194 >> 172.21.15.89:0/3837607194 conn(0x7f56980fc7a0 msgr2=0x7f56980febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:08.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.898+0000 7f569cea4640 1 -- 172.21.15.89:0/3837607194 shutdown_connections 2024-09-15T23:22:08.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.898+0000 7f569cea4640 1 -- 172.21.15.89:0/3837607194 wait complete. 2024-09-15T23:22:08.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f569cea4640 1 Processor -- start 2024-09-15T23:22:08.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f569cea4640 1 -- start start 2024-09-15T23:22:08.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f569cea4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5698100a80 0x7f5698196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:08.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f569cea4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5698101420 0x7f56981974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:08.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f569cea4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5698106a80 0x7f5698197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:08.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f569cea4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5698110e80 con 0x7f5698106a80 2024-09-15T23:22:08.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f569cea4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5698110d00 con 0x7f5698100a80 2024-09-15T23:22:08.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f569cea4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5698111000 con 0x7f5698101420 2024-09-15T23:22:08.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f5696d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5698106a80 0x7f5698197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:08.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f5696d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5698106a80 0x7f5698197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39792/0 (socket says 172.21.15.89:39792) 2024-09-15T23:22:08.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.899+0000 7f5696d9d640 1 -- 172.21.15.89:0/2128392795 learned_addr learned my addr 172.21.15.89:0/2128392795 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:08.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f569659c640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5698100a80 0x7f5698196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:08.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f5696d9d640 1 -- 172.21.15.89:0/2128392795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5698101420 msgr2=0x7f56981974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:08.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f5696d9d640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5698101420 0x7f56981974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:08.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f5696d9d640 1 -- 172.21.15.89:0/2128392795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5698100a80 msgr2=0x7f5698196fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:08.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f5696d9d640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5698100a80 0x7f5698196fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f5696d9d640 1 -- 172.21.15.89:0/2128392795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56981a6810 con 0x7f5698106a80 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f5696d9d640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5698106a80 0x7f5698197a20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f569000cce0 tx=0x7f569000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f56877fe640 1 -- 172.21.15.89:0/2128392795 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569000f450 con 0x7f5698106a80 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f56877fe640 1 -- 172.21.15.89:0/2128392795 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f569000fad0 con 0x7f5698106a80 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f569cea4640 1 -- 172.21.15.89:0/2128392795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56981a6b00 con 0x7f5698106a80 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f56877fe640 1 -- 172.21.15.89:0/2128392795 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5690018c10 con 0x7f5698106a80 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.900+0000 7f569cea4640 1 -- 172.21.15.89:0/2128392795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56981a6ff0 con 0x7f5698106a80 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.901+0000 7f569cea4640 1 -- 172.21.15.89:0/2128392795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5698100ee0 con 0x7f5698106a80 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.902+0000 7f56877fe640 1 -- 172.21.15.89:0/2128392795 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f569000f5f0 con 0x7f5698106a80 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.903+0000 7f56877fe640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56800a3240 0x7f56800a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.903+0000 7f569659c640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56800a3240 0x7f56800a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:08.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.903+0000 7f56877fe640 1 -- 172.21.15.89:0/2128392795 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f56900d02e0 con 0x7f5698106a80 2024-09-15T23:22:08.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.903+0000 7f569659c640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56800a3240 0x7f56800a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f568800cac0 tx=0x7f568800c4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:08.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:08.907+0000 7f56877fe640 1 -- 172.21.15.89:0/2128392795 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5690014030 con 0x7f5698106a80 2024-09-15T23:22:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.109+0000 7f569cea4640 1 -- 172.21.15.89:0/2128392795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56981a7440 con 0x7f5698106a80 2024-09-15T23:22:09.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.109+0000 7f56877fe640 1 -- 172.21.15.89:0/2128392795 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11851 (secure 0 0 0) 0x7f569009cdc0 con 0x7f5698106a80 2024-09-15T23:22:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:09.111 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"btime":"2024-09-15T23:22:02:573423+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24389,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24424,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24428,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24430,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24436,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:22:02.246771+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24452},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24452,"qdb_cluster":[24452]},"id":1}]} 2024-09-15T23:22:09.111 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 17 2024-09-15T23:22:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.111+0000 7f569cea4640 1 -- 172.21.15.89:0/2128392795 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56800a3240 msgr2=0x7f56800a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.111+0000 7f569cea4640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56800a3240 0x7f56800a5700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f568800cac0 tx=0x7f568800c4a0 comp rx=0 tx=0).stop 2024-09-15T23:22:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.111+0000 7f569cea4640 1 -- 172.21.15.89:0/2128392795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5698106a80 msgr2=0x7f5698197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.111+0000 7f569cea4640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5698106a80 0x7f5698197a20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f569000cce0 tx=0x7f569000a620 comp rx=0 tx=0).stop 2024-09-15T23:22:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.112+0000 7f569cea4640 1 -- 172.21.15.89:0/2128392795 shutdown_connections 2024-09-15T23:22:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.112+0000 7f569cea4640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56800a3240 0x7f56800a5700 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.112+0000 7f569cea4640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5698106a80 0x7f5698197a20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.112+0000 7f569cea4640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5698101420 0x7f56981974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.112+0000 7f569cea4640 1 --2- 172.21.15.89:0/2128392795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5698100a80 0x7f5698196fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:09.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.112+0000 7f569cea4640 1 -- 172.21.15.89:0/2128392795 >> 172.21.15.89:0/2128392795 conn(0x7f56980fc7a0 msgr2=0x7f56980fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.112+0000 7f569cea4640 1 -- 172.21.15.89:0/2128392795 shutdown_connections 2024-09-15T23:22:09.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:09.112+0000 7f569cea4640 1 -- 172.21.15.89:0/2128392795 wait complete. 2024-09-15T23:22:09.848 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:22:10.097 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:10.217 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:10 smithi089 ceph-mon[30933]: pgmap v290: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:10.217 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:10 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2128392795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:10.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:10 smithi120 ceph-mon[35250]: pgmap v290: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:10.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:10 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2128392795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:10 smithi174 ceph-mon[35586]: pgmap v290: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:10 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2128392795' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:10.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.627+0000 7fe114b2e640 1 -- 172.21.15.89:0/1631903077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe110102d30 msgr2=0x7fe110103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.627+0000 7fe114b2e640 1 --2- 172.21.15.89:0/1631903077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe110102d30 0x7fe110103110 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe100009cd0 tx=0x7fe10002a910 comp rx=0 tx=0).stop 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.627+0000 7fe114b2e640 1 -- 172.21.15.89:0/1631903077 shutdown_connections 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.627+0000 7fe114b2e640 1 --2- 172.21.15.89:0/1631903077 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1101082a0 0x7fe11010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.627+0000 7fe114b2e640 1 --2- 172.21.15.89:0/1631903077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1101036e0 0x7fe110107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.627+0000 7fe114b2e640 1 --2- 172.21.15.89:0/1631903077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe110102d30 0x7fe110103110 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.627+0000 7fe114b2e640 1 -- 172.21.15.89:0/1631903077 >> 172.21.15.89:0/1631903077 conn(0x7fe1100fc7d0 msgr2=0x7fe1100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.627+0000 7fe114b2e640 1 -- 172.21.15.89:0/1631903077 shutdown_connections 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.627+0000 7fe114b2e640 1 -- 172.21.15.89:0/1631903077 wait complete. 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe114b2e640 1 Processor -- start 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe114b2e640 1 -- start start 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe114b2e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe110102d30 0x7fe1101985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe114b2e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1101036e0 0x7fe110198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe114b2e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1101082a0 0x7fe11019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe114b2e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101120e0 con 0x7fe1101082a0 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe114b2e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe110111f60 con 0x7fe110102d30 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe114b2e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe110112260 con 0x7fe1101036e0 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1101082a0 0x7fe11019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1101082a0 0x7fe11019ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39810/0 (socket says 172.21.15.89:39810) 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10ed9d640 1 -- 172.21.15.89:0/3715224228 learned_addr learned my addr 172.21.15.89:0/3715224228 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10e59c640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe110102d30 0x7fe1101985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10dd9b640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1101036e0 0x7fe110198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10ed9d640 1 -- 172.21.15.89:0/3715224228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1101036e0 msgr2=0x7fe110198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10ed9d640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1101036e0 0x7fe110198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10ed9d640 1 -- 172.21.15.89:0/3715224228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe110102d30 msgr2=0x7fe1101985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10ed9d640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe110102d30 0x7fe1101985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10ed9d640 1 -- 172.21.15.89:0/3715224228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11019d590 con 0x7fe1101082a0 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10e59c640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe110102d30 0x7fe1101985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe10ed9d640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1101082a0 0x7fe11019ceb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe10800d750 tx=0x7fe10800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe1077fe640 1 -- 172.21.15.89:0/3715224228 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108017070 con 0x7fe1101082a0 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.628+0000 7fe1077fe640 1 -- 172.21.15.89:0/3715224228 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe108005500 con 0x7fe1101082a0 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.629+0000 7fe114b2e640 1 -- 172.21.15.89:0/3715224228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11019d880 con 0x7fe1101082a0 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.629+0000 7fe1077fe640 1 -- 172.21.15.89:0/3715224228 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe108004d90 con 0x7fe1101082a0 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.629+0000 7fe114b2e640 1 -- 172.21.15.89:0/3715224228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1101a5160 con 0x7fe1101082a0 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.629+0000 7fe114b2e640 1 -- 172.21.15.89:0/3715224228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe11006a850 con 0x7fe1101082a0 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.631+0000 7fe1077fe640 1 -- 172.21.15.89:0/3715224228 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe108004f30 con 0x7fe1101082a0 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.632+0000 7fe1077fe640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe0fc0a32c0 0x7fe0fc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.632+0000 7fe10e59c640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe0fc0a32c0 0x7fe0fc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.632+0000 7fe1077fe640 1 -- 172.21.15.89:0/3715224228 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fe1080cf9f0 con 0x7fe1101082a0 2024-09-15T23:22:10.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.633+0000 7fe10e59c640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe0fc0a32c0 0x7fe0fc0a5780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe100000f80 tx=0x7fe100005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:10.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.635+0000 7fe1077fe640 1 -- 172.21.15.89:0/3715224228 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe10809c550 con 0x7fe1101082a0 2024-09-15T23:22:10.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.803+0000 7fe114b2e640 1 -- 172.21.15.89:0/3715224228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe11010a2b0 con 0x7fe1101082a0 2024-09-15T23:22:10.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.805+0000 7fe1077fe640 1 -- 172.21.15.89:0/3715224228 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24199 (secure 0 0 0) 0x7fe10809be50 con 0x7fe1101082a0 2024-09-15T23:22:10.807 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:10.807 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":86,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:22:02.246640+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.807+0000 7fe114b2e640 1 -- 172.21.15.89:0/3715224228 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe0fc0a32c0 msgr2=0x7fe0fc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.807+0000 7fe114b2e640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe0fc0a32c0 0x7fe0fc0a5780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe100000f80 tx=0x7fe100005ef0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.807+0000 7fe114b2e640 1 -- 172.21.15.89:0/3715224228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1101082a0 msgr2=0x7fe11019ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.807+0000 7fe114b2e640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1101082a0 0x7fe11019ceb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe10800d750 tx=0x7fe10800dc10 comp rx=0 tx=0).stop 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.808+0000 7fe114b2e640 1 -- 172.21.15.89:0/3715224228 shutdown_connections 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.808+0000 7fe114b2e640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe0fc0a32c0 0x7fe0fc0a5780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.808+0000 7fe114b2e640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1101082a0 0x7fe11019ceb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.808+0000 7fe114b2e640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1101036e0 0x7fe110198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.808+0000 7fe114b2e640 1 --2- 172.21.15.89:0/3715224228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe110102d30 0x7fe1101985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.808+0000 7fe114b2e640 1 -- 172.21.15.89:0/3715224228 >> 172.21.15.89:0/3715224228 conn(0x7fe1100fc7d0 msgr2=0x7fe110105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.808+0000 7fe114b2e640 1 -- 172.21.15.89:0/3715224228 shutdown_connections 2024-09-15T23:22:10.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:10.808+0000 7fe114b2e640 1 -- 172.21.15.89:0/3715224228 wait complete. 2024-09-15T23:22:11.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:11 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3715224228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:11.374 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:11 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3715224228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:11 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3715224228' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:11.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-15T23:21:53.066091+0000', 'modified': '2024-09-15T23:22:02.246771+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24452}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24452': {'gid': 24452, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.120:6839/1554218591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 1554218591}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 1554218591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24452, 'qdb_cluster': [24452]} 2024-09-15T23:22:11.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-15T23:22:11.636 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T23:22:11.885 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:12.177 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:12 smithi089 ceph-mon[30933]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:12.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 -- 172.21.15.89:0/3079864931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedd8106f10 msgr2=0x7fedd810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:12.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 --2- 172.21.15.89:0/3079864931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedd8106f10 0x7fedd810ea90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009ca0 tx=0x7fedd002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 -- 172.21.15.89:0/3079864931 shutdown_connections 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 --2- 172.21.15.89:0/3079864931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedd8106f10 0x7fedd810ea90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 --2- 172.21.15.89:0/3079864931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd8102790 0x7fedd81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 --2- 172.21.15.89:0/3079864931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd8101de0 0x7fedd81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 -- 172.21.15.89:0/3079864931 >> 172.21.15.89:0/3079864931 conn(0x7fedd80fd8c0 msgr2=0x7fedd80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 -- 172.21.15.89:0/3079864931 shutdown_connections 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 -- 172.21.15.89:0/3079864931 wait complete. 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 Processor -- start 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.379+0000 7feddeec2640 1 -- start start 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddeec2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd8101de0 0x7fedd8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddeec2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd8102790 0x7fedd81988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddeec2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedd8106f10 0x7fedd819cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddeec2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8111f70 con 0x7fedd8106f10 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddeec2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8111df0 con 0x7fedd8101de0 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddeec2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fedd81120f0 con 0x7fedd8102790 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddd45f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedd8106f10 0x7fedd819cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddd45f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedd8106f10 0x7fedd819cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39838/0 (socket says 172.21.15.89:39838) 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddd45f640 1 -- 172.21.15.89:0/2086939319 learned_addr learned my addr 172.21.15.89:0/2086939319 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddcc5e640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd8101de0 0x7fedd8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7fedd7fff640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd8102790 0x7fedd81988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddd45f640 1 -- 172.21.15.89:0/2086939319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd8102790 msgr2=0x7fedd81988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddd45f640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd8102790 0x7fedd81988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddd45f640 1 -- 172.21.15.89:0/2086939319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd8101de0 msgr2=0x7fedd8198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:12.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddd45f640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd8101de0 0x7fedd8198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddd45f640 1 -- 172.21.15.89:0/2086939319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd819d340 con 0x7fedd8106f10 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7fedd7fff640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd8102790 0x7fedd81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddd45f640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedd8106f10 0x7fedd819cc60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fedd0031a20 tx=0x7fedd0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7fedd5ffb640 1 -- 172.21.15.89:0/2086939319 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0003c30 con 0x7fedd8106f10 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7feddeec2640 1 -- 172.21.15.89:0/2086939319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd819d5d0 con 0x7fedd8106f10 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7fedd5ffb640 1 -- 172.21.15.89:0/2086939319 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedd0003dd0 con 0x7fedd8106f10 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.380+0000 7fedd5ffb640 1 -- 172.21.15.89:0/2086939319 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd002fcf0 con 0x7fedd8106f10 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.381+0000 7feddeec2640 1 -- 172.21.15.89:0/2086939319 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd81a4f10 con 0x7fedd8106f10 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fedd5ffb640 1 -- 172.21.15.89:0/2086939319 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fedd0004050 con 0x7fedd8106f10 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fede129b740 1 Processor -- start 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fede129b740 1 -- start start 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fede129b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026e3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fede129b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fede129b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 0x5652026f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fede129b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56520264a090 con 0x5652026e38d0 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fede129b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5652025f1a70 con 0x5652026e41f0 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fede129b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565202646050 con 0x5652026f2fc0 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026e3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026e3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39848/0 (socket says 172.21.15.89:39848) 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fedd7fff640 1 -- 172.21.15.89:0/3586411734 learned_addr learned my addr 172.21.15.89:0/3586411734 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.383+0000 7fedd5ffb640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fedc80a3240 0x7fedc80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedd7fff640 1 --2- 172.21.15.89:0/3586411734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 0x5652026f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7feddcc5e640 1 --2- 172.21.15.89:0/3586411734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7feddd45f640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fedc80a3240 0x7fedc80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedd7fff640 1 -- 172.21.15.89:0/3586411734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 msgr2=0x5652026f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedd7fff640 1 --2- 172.21.15.89:0/3586411734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 0x5652026f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedd7fff640 1 -- 172.21.15.89:0/3586411734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 msgr2=0x5652026f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedd7fff640 1 --2- 172.21.15.89:0/3586411734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedd7fff640 1 -- 172.21.15.89:0/3586411734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652025fe130 con 0x5652026e38d0 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedd5ffb640 1 -- 172.21.15.89:0/2086939319 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fedd00f0fa0 con 0x7fedd8106f10 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedd7fff640 1 --2- 172.21.15.89:0/3586411734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026e3cb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009f90 tx=0x7fedc40095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcdc51d255912d43 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7feddd45f640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fedc80a3240 0x7fedc80a5700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fedd00b9870 tx=0x7fedd00e8150 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedc2ffd640 1 -- 172.21.15.89:0/3586411734 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc40150f0 con 0x5652026e38d0 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedc2ffd640 1 -- 172.21.15.89:0/3586411734 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc40035f0 con 0x5652026e38d0 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.384+0000 7fedc2ffd640 1 -- 172.21.15.89:0/3586411734 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4013720 con 0x5652026e38d0 2024-09-15T23:22:12.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 -- 172.21.15.89:0/3586411734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 msgr2=0x5652026e3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 --2- 172.21.15.89:0/3586411734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026e3cb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009f90 tx=0x7fedc40095b0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 -- 172.21.15.89:0/3586411734 shutdown_connections 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 --2- 172.21.15.89:0/3586411734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 0x5652026f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 --2- 172.21.15.89:0/3586411734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 --2- 172.21.15.89:0/3586411734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026e3cb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 -- 172.21.15.89:0/3586411734 >> 172.21.15.89:0/3586411734 conn(0x5652026dff60 msgr2=0x5652026e0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 -- 172.21.15.89:0/3586411734 shutdown_connections 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 -- 172.21.15.89:0/3586411734 wait complete. 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 -- start start 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.385+0000 7fede129b740 1 Processor -- start 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fede129b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fede129b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026efbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fede129b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 0x5652026f0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fede129b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5652026f5bf0 con 0x5652026e38d0 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fede129b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565202604f20 con 0x5652026e41f0 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fede129b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5652026f5d70 con 0x5652026f2fc0 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fede129b740 10 client.0 _ll_register_callbacks cb 0x5652025ce030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39856/0 (socket says 172.21.15.89:39856) 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedd7fff640 1 -- 172.21.15.89:0/1510610121 learned_addr learned my addr 172.21.15.89:0/1510610121 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7feddcc5e640 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026efbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedd7fff640 1 -- 172.21.15.89:0/1510610121 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 msgr2=0x5652026f0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedd7fff640 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 0x5652026f0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedd7fff640 1 -- 172.21.15.89:0/1510610121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 msgr2=0x5652026efbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:12.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedd7fff640 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026efbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedd7fff640 1 -- 172.21.15.89:0/1510610121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652023bb2b0 con 0x5652026e38d0 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedd7fff640 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026f28a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fedc4005940 tx=0x7fedc4015300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f431e7961ff27ffa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedb7fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedb7fff640 1 -- 172.21.15.89:0/1510610121 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4004230 con 0x5652026e38d0 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fedb7fff640 1 -- 172.21.15.89:0/1510610121 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fedc40043d0 con 0x5652026e38d0 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.386+0000 7fede129b740 1 -- 172.21.15.89:0/1510610121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedc40035f0 con 0x5652026e38d0 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fedb7fff640 1 -- 172.21.15.89:0/1510610121 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fedc40037a0 con 0x5652026e38d0 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fede129b740 10 client.14724 fetch_fsmap learned FSMap version 17 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fede129b740 1 -- 172.21.15.89:0/1510610121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5652027015a0 con 0x5652026e38d0 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fedb7fff640 1 -- 172.21.15.89:0/1510610121 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc40035f0 con 0x5652026e38d0 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fedb7fff640 1 -- 172.21.15.89:0/1510610121 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7fedc400c070 con 0x5652026e38d0 2024-09-15T23:22:12.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fede129b740 10 client.14724 fetch_fsmap finished waiting for FSMap version 17 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fede129b740 10 client.14724 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fede129b740 20 client.14724 populate_metadata read hostname 'smithi089' 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fede129b740 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565202703830 0x565202704ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fede129b740 4 client.14724 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7fede129b740 1 -- 172.21.15.89:0/1510610121 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565202657600 con 0x565202703830 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.387+0000 7feddcc5e640 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565202703830 0x565202704ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.388+0000 7fedb7fff640 10 client.14724 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.388+0000 7feddcc5e640 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565202703830 0x565202704ca0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6013e37b73295d59 server_cookie=1699f020ce60a560 in_seq=0 out_seq=0 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fedb7fff640 1 -- 172.21.15.89:0/1510610121 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565202657600 con 0x565202703830 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fedb7fff640 10 client.14724 handle_command_reply: tid=0 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 client.14724 shutdown 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 -- 172.21.15.89:0/1510610121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 msgr2=0x5652026f28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:12.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026f28a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fedc4005940 tx=0x7fedc4015300 comp rx=0 tx=0).stop 2024-09-15T23:22:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 -- 172.21.15.89:0/1510610121 shutdown_connections 2024-09-15T23:22:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565202703830 0x565202704ca0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 0x5652026f0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026efbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fedb7fff640 0 client.14724 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:22:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 --2- 172.21.15.89:0/1510610121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x5652026f28a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 -- 172.21.15.89:0/1510610121 >> 172.21.15.89:0/1510610121 conn(0x5652026dff60 msgr2=0x5652026e5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 -- 172.21.15.89:0/1510610121 shutdown_connections 2024-09-15T23:22:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 1 -- 172.21.15.89:0/1510610121 wait complete. 2024-09-15T23:22:12.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.389+0000 7fede129b740 20 client.14724 trim_cache size 0 max 16384 2024-09-15T23:22:12.403 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:12 smithi120 ceph-mon[35250]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:12.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.406+0000 7fede129b740 1 Processor -- start 2024-09-15T23:22:12.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.406+0000 7fede129b740 1 -- start start 2024-09-15T23:22:12.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fede129b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x565202704ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fede129b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026e0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fede129b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 0x5652026ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fede129b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x565202702fe0 con 0x5652026e38d0 2024-09-15T23:22:12.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fede129b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5652027036b0 con 0x5652026e41f0 2024-09-15T23:22:12.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fede129b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565202703d80 con 0x5652026f2fc0 2024-09-15T23:22:12.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x565202704ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x565202704ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39860/0 (socket says 172.21.15.89:39860) 2024-09-15T23:22:12.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedd7fff640 1 -- 172.21.15.89:0/3459185169 learned_addr learned my addr 172.21.15.89:0/3459185169 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:12.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7feddcc5e640 1 --2- 172.21.15.89:0/3459185169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026e0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedd7fff640 1 -- 172.21.15.89:0/3459185169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 msgr2=0x5652026ee4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:12.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedd7fff640 1 --2- 172.21.15.89:0/3459185169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 0x5652026ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedd7fff640 1 -- 172.21.15.89:0/3459185169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 msgr2=0x5652026e0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:12.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedd7fff640 1 --2- 172.21.15.89:0/3459185169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026e0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedd7fff640 1 -- 172.21.15.89:0/3459185169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652026e0880 con 0x5652026e38d0 2024-09-15T23:22:12.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedd7fff640 1 --2- 172.21.15.89:0/3459185169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x565202704ca0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fedc4000c00 tx=0x7fedc4003670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b545adcb0e2cf93 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:12.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedc27fc640 1 -- 172.21.15.89:0/3459185169 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4039070 con 0x5652026e38d0 2024-09-15T23:22:12.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedc27fc640 1 -- 172.21.15.89:0/3459185169 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc4027070 con 0x5652026e38d0 2024-09-15T23:22:12.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.407+0000 7fedc27fc640 1 -- 172.21.15.89:0/3459185169 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4015e60 con 0x5652026e38d0 2024-09-15T23:22:12.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 -- 172.21.15.89:0/3459185169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 msgr2=0x565202704ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:12.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 --2- 172.21.15.89:0/3459185169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x565202704ca0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fedc4000c00 tx=0x7fedc4003670 comp rx=0 tx=0).stop 2024-09-15T23:22:12.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 -- 172.21.15.89:0/3459185169 shutdown_connections 2024-09-15T23:22:12.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 --2- 172.21.15.89:0/3459185169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026f2fc0 0x5652026ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 --2- 172.21.15.89:0/3459185169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652026e0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 --2- 172.21.15.89:0/3459185169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026e38d0 0x565202704ca0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 -- 172.21.15.89:0/3459185169 >> 172.21.15.89:0/3459185169 conn(0x5652026dff60 msgr2=0x5652026dadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 -- 172.21.15.89:0/3459185169 shutdown_connections 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 -- 172.21.15.89:0/3459185169 wait complete. 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 -- start start 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.408+0000 7fede129b740 1 Processor -- start 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fede129b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026e38d0 0x565202705f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fede129b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652027064d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fede129b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026f2fc0 0x565202706a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fede129b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x565202704b20 con 0x5652026f2fc0 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fede129b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565202704450 con 0x5652026e41f0 2024-09-15T23:22:12.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fede129b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5652027051f0 con 0x5652026e38d0 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fede129b740 10 client.0 _ll_register_callbacks cb 0x7fedc4015420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026f2fc0 0x565202706a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026f2fc0 0x565202706a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39870/0 (socket says 172.21.15.89:39870) 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedd7fff640 1 -- 172.21.15.89:0/1906341641 learned_addr learned my addr 172.21.15.89:0/1906341641 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7feddcc5e640 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652027064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedd7fff640 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026e38d0 0x565202705f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedd7fff640 1 -- 172.21.15.89:0/1906341641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026e38d0 msgr2=0x565202705f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedd7fff640 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026e38d0 0x565202705f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedd7fff640 1 -- 172.21.15.89:0/1906341641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 msgr2=0x5652027064d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedd7fff640 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652027064d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedd7fff640 1 -- 172.21.15.89:0/1906341641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56520276c200 con 0x5652026f2fc0 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedd7fff640 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026f2fc0 0x565202706a30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fedc40094b0 tx=0x7fedc4005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc03177a0deb9905 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:12.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedb7fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedb7fff640 1 -- 172.21.15.89:0/1906341641 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4039040 con 0x5652026f2fc0 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fedb7fff640 1 -- 172.21.15.89:0/1906341641 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fedc4004230 con 0x5652026f2fc0 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.409+0000 7fede129b740 1 -- 172.21.15.89:0/1906341641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedc4027070 con 0x5652026f2fc0 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fedb7fff640 1 -- 172.21.15.89:0/1906341641 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fedc4009510 con 0x5652026f2fc0 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fede129b740 10 client.14730 fetch_fsmap learned FSMap version 17 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fede129b740 1 -- 172.21.15.89:0/1906341641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56520276d500 con 0x5652026f2fc0 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fedb7fff640 1 -- 172.21.15.89:0/1906341641 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc4027070 con 0x5652026f2fc0 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fedb7fff640 1 -- 172.21.15.89:0/1906341641 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7fedc400c070 con 0x5652026f2fc0 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fede129b740 10 client.14730 fetch_fsmap finished waiting for FSMap version 17 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fede129b740 10 client.14730 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fede129b740 20 client.14730 populate_metadata read hostname 'smithi089' 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fede129b740 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565202771c70 0x565202770980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fede129b740 4 client.14730 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:22:12.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7fede129b740 1 -- 172.21.15.89:0/1906341641 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "status"}) -- 0x565202651b10 con 0x565202771c70 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.410+0000 7feddcc5e640 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565202771c70 0x565202770980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fedb7fff640 10 client.14730 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7feddcc5e640 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565202771c70 0x565202770980 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=633d17f372ebf347 server_cookie=6d47f26667d83d78 in_seq=0 out_seq=0 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fedb7fff640 1 -- 172.21.15.89:0/1906341641 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x565202651b10 con 0x565202771c70 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fedb7fff640 10 client.14730 handle_command_reply: tid=0 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 client.14730 shutdown 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 -- 172.21.15.89:0/1906341641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026f2fc0 msgr2=0x565202706a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026f2fc0 0x565202706a30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fedc40094b0 tx=0x7fedc4005bb0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 -- 172.21.15.89:0/1906341641 shutdown_connections 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565202771c70 0x565202770980 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652026f2fc0 0x565202706a30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fedb7fff640 0 client.14730 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652026e41f0 0x5652027064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 --2- 172.21.15.89:0/1906341641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652026e38d0 0x565202705f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 -- 172.21.15.89:0/1906341641 >> 172.21.15.89:0/1906341641 conn(0x5652026dff60 msgr2=0x5652026eaa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:12.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 -- 172.21.15.89:0/1906341641 shutdown_connections 2024-09-15T23:22:12.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 1 -- 172.21.15.89:0/1906341641 wait complete. 2024-09-15T23:22:12.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.411+0000 7fede129b740 20 client.14730 trim_cache size 0 max 16384 2024-09-15T23:22:12.419 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:22:12.419 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:22:12.419 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-09-15T23:22:12.419 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24452, 2024-09-15T23:22:12.419 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:22:12.419 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:22:12.420 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:22:12.420 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 18.891677286, 2024-09-15T23:22:12.420 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 17, 2024-09-15T23:22:12.420 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 86, 2024-09-15T23:22:12.420 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:22:12.420 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 29.418601836000001 2024-09-15T23:22:12.420 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 -- 172.21.15.89:0/2086939319 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fedc80a3240 msgr2=0x7fedc80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fedc80a3240 0x7fedc80a5700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fedd00b9870 tx=0x7fedd00e8150 comp rx=0 tx=0).stop 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 -- 172.21.15.89:0/2086939319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedd8106f10 msgr2=0x7fedd819cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedd8106f10 0x7fedd819cc60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fedd0031a20 tx=0x7fedd0031a50 comp rx=0 tx=0).stop 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 -- 172.21.15.89:0/2086939319 shutdown_connections 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fedc80a3240 0x7fedc80a5700 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedd8106f10 0x7fedd819cc60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd8102790 0x7fedd81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 --2- 172.21.15.89:0/2086939319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedd8101de0 0x7fedd8198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 -- 172.21.15.89:0/2086939319 >> 172.21.15.89:0/2086939319 conn(0x7fedd80fd8c0 msgr2=0x7fedd80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 -- 172.21.15.89:0/2086939319 shutdown_connections 2024-09-15T23:22:12.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:12.412+0000 7feddeec2640 1 -- 172.21.15.89:0/2086939319 wait complete. 2024-09-15T23:22:12.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:12 smithi174 ceph-mon[35586]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:13.101 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-15T23:22:13.102 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:22:13.102 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-15T23:22:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:13.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.905+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/1541947776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d58102790 msgr2=0x7f8d581067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.905+0000 7f8d5eeb6640 1 --2- 172.21.15.89:0/1541947776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d58102790 0x7f8d581067e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8d44009900 tx=0x7f8d4402a410 comp rx=0 tx=0).stop 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.905+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/1541947776 shutdown_connections 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.905+0000 7f8d5eeb6640 1 --2- 172.21.15.89:0/1541947776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d58106f10 0x7f8d5810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.905+0000 7f8d5eeb6640 1 --2- 172.21.15.89:0/1541947776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d58102790 0x7f8d581067e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.905+0000 7f8d5eeb6640 1 --2- 172.21.15.89:0/1541947776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d58101de0 0x7f8d581021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.905+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/1541947776 >> 172.21.15.89:0/1541947776 conn(0x7f8d580fd8c0 msgr2=0x7f8d580ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.905+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/1541947776 shutdown_connections 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.905+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/1541947776 wait complete. 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5eeb6640 1 Processor -- start 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5eeb6640 1 -- start start 2024-09-15T23:22:13.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5eeb6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d58101de0 0x7f8d581983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5eeb6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d58102790 0x7f8d58198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5eeb6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d58106f10 0x7f8d5819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5eeb6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8d58111f70 con 0x7f8d58106f10 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5eeb6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8d58111df0 con 0x7f8d58101de0 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5eeb6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d581120f0 con 0x7f8d58102790 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5d453640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d58106f10 0x7f8d5819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5d453640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d58106f10 0x7f8d5819cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39886/0 (socket says 172.21.15.89:39886) 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5d453640 1 -- 172.21.15.89:0/3775952905 learned_addr learned my addr 172.21.15.89:0/3775952905 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5cc52640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d58101de0 0x7f8d581983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d57fff640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d58102790 0x7f8d58198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5d453640 1 -- 172.21.15.89:0/3775952905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d58102790 msgr2=0x7f8d58198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5d453640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d58102790 0x7f8d58198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5d453640 1 -- 172.21.15.89:0/3775952905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d58101de0 msgr2=0x7f8d581983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5d453640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d58101de0 0x7f8d581983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d5d453640 1 -- 172.21.15.89:0/3775952905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d5819d380 con 0x7f8d58106f10 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.906+0000 7f8d57fff640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d58102790 0x7f8d58198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.907+0000 7f8d5d453640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d58106f10 0x7f8d5819cca0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000cce0 tx=0x7f8d5000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.907+0000 7f8d55ffb640 1 -- 172.21.15.89:0/3775952905 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5000f450 con 0x7f8d58106f10 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.907+0000 7f8d55ffb640 1 -- 172.21.15.89:0/3775952905 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d5000fad0 con 0x7f8d58106f10 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.907+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/3775952905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d5819d670 con 0x7f8d58106f10 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.907+0000 7f8d55ffb640 1 -- 172.21.15.89:0/3775952905 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d50018c10 con 0x7f8d58106f10 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.907+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/3775952905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d581a4f50 con 0x7f8d58106f10 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.908+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/3775952905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d581021c0 con 0x7f8d58106f10 2024-09-15T23:22:13.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.909+0000 7f8d55ffb640 1 -- 172.21.15.89:0/3775952905 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f8d5000f5f0 con 0x7f8d58106f10 2024-09-15T23:22:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.910+0000 7f8d55ffb640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d480a3240 0x7f8d480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.910+0000 7f8d55ffb640 1 -- 172.21.15.89:0/3775952905 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f8d500d01e0 con 0x7f8d58106f10 2024-09-15T23:22:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.910+0000 7f8d5cc52640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d480a3240 0x7f8d480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.910+0000 7f8d5cc52640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d480a3240 0x7f8d480a5700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c005fd0 tx=0x7f8d4c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:13.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:13.914+0000 7f8d55ffb640 1 -- 172.21.15.89:0/3775952905 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8d50014030 con 0x7f8d58106f10 2024-09-15T23:22:14.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.117+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/3775952905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8d5810a1b0 con 0x7f8d58106f10 2024-09-15T23:22:14.119 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:14 smithi089 ceph-mon[30933]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:14.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.118+0000 7f8d55ffb640 1 -- 172.21.15.89:0/3775952905 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f8d5009cc60 con 0x7f8d58106f10 2024-09-15T23:22:14.119 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-15T23:22:14.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/3775952905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d480a3240 msgr2=0x7f8d480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:14.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d480a3240 0x7f8d480a5700 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c005fd0 tx=0x7f8d4c00b390 comp rx=0 tx=0).stop 2024-09-15T23:22:14.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/3775952905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d58106f10 msgr2=0x7f8d5819cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:14.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d58106f10 0x7f8d5819cca0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000cce0 tx=0x7f8d5000a620 comp rx=0 tx=0).stop 2024-09-15T23:22:14.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/3775952905 shutdown_connections 2024-09-15T23:22:14.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d480a3240 0x7f8d480a5700 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:14.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d58106f10 0x7f8d5819cca0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:14.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d58102790 0x7f8d58198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:14.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 --2- 172.21.15.89:0/3775952905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d58101de0 0x7f8d581983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:14.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/3775952905 >> 172.21.15.89:0/3775952905 conn(0x7f8d580fd8c0 msgr2=0x7f8d580fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:14.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/3775952905 shutdown_connections 2024-09-15T23:22:14.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:14.120+0000 7f8d5eeb6640 1 -- 172.21.15.89:0/3775952905 wait complete. 2024-09-15T23:22:14.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:14 smithi120 ceph-mon[35250]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:14.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:14 smithi174 ceph-mon[35586]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:14.938 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-15T23:22:14.938 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-15T23:22:14.938 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-09-15T23:22:14.938 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-15T23:22:14.938 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:22:14.938 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-15T23:22:14.938 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-15T23:22:14.938 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-15T23:22:14.939 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-15T23:22:14.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:14.939 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-09-15T23:22:14.956 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-15T23:22:14.956 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-15T23:22:14.956 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-09-15T23:22:14.956 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:22:14.956 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-09-15T23:22:14.956 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:22:14.956 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:22:14.956 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42315sec preferred_lft 42315sec 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link noprefixroute 2024-09-15T23:22:14.957 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:22:14.958 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-15T23:22:14.958 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:22:14.958 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-09-15T23:22:14.958 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add name ceph-brx type bridge 2024-09-15T23:22:14.958 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr flush dev ceph-brx 2024-09-15T23:22:14.959 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set ceph-brx up 2024-09-15T23:22:14.959 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-15T23:22:14.959 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-09-15T23:22:15.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:22:15.159 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:15 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3775952905' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:22:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:22:15.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:15.168 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-15T23:22:15.194 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-09-15T23:22:15.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:15.196 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-09-15T23:22:15.272 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-09-15T23:22:15.272 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-15T23:22:15.272 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-15T23:22:15.273 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-15T23:22:15.273 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-15T23:22:15.273 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:22:15.273 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-09-15T23:22:15.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-15T23:22:15.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-15T23:22:15.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-15T23:22:15.274 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-09-15T23:22:15.303 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:15 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3775952905' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:22:15.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:22:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:15 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3775952905' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:22:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:22:15.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:15.551 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-09-15T23:22:15.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:15.568 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2024-09-15T23:22:15.624 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:22:15.624 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-09-15T23:22:15.624 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:22:15.624 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-15T23:22:15.624 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-09-15T23:22:15.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:22:15.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:22:15.741 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-15T23:22:15.741 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:22:15.741 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-09-15T23:22:15.741 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-15T23:22:15.741 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-15T23:22:15.741 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-15T23:22:15.741 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-15T23:22:15.741 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-15T23:22:15.742 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-09-15T23:22:15.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:22:15.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:22:15.876 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:22:15.876 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-09-15T23:22:15.876 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-09-15T23:22:15.876 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.0 master ceph-brx 2024-09-15T23:22:15.876 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-09-15T23:22:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:22:15.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:22:15.952 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-15T23:22:15.952 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T23:22:15.952 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:22:15.967 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-15T23:22:15.968 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T23:22:15.968 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:22:16.023 DEBUG:teuthology.orchestra.run.smithi089:> sudo modprobe fuse 2024-09-15T23:22:16.089 DEBUG:teuthology.orchestra.run.smithi089:> cat /proc/self/mounts | awk '{print $2}' 2024-09-15T23:22:16.146 INFO:teuthology.orchestra.run.smithi089.stdout:/proc 2024-09-15T23:22:16.146 INFO:teuthology.orchestra.run.smithi089.stdout:/sys 2024-09-15T23:22:16.146 INFO:teuthology.orchestra.run.smithi089.stdout:/dev 2024-09-15T23:22:16.146 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/security 2024-09-15T23:22:16.146 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/shm 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/pts 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/run 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/cgroup 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/pstore 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/bpf 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/ 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/selinux 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/hugepages 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/mqueue 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/debug 2024-09-15T23:22:16.147 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/tracing 2024-09-15T23:22:16.148 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/fuse/connections 2024-09-15T23:22:16.148 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/config 2024-09-15T23:22:16.148 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-sysctl.service 2024-09-15T23:22:16.148 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-15T23:22:16.148 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-15T23:22:16.148 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/nfs/rpc_pipefs 2024-09-15T23:22:16.148 INFO:teuthology.orchestra.run.smithi089.stdout:/run/user/1000 2024-09-15T23:22:16.148 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/c74bd21383dd0a4cd09dc3ab8b9aa80bfc3cdb8ef29a7bda04ba5d8230e734fe/merged 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/e32a2c1615268d17d7dd8f71c700fdf06e7ddfdf77b34efcdac1fa09960a99bd/merged 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/run/user/0 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/d2384accabc5bac7b4d8a0582af713f719257b33305eb412b562271c45903b58/merged 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/efcfd97affcb1727a2698e26eb2c238713148d5ee0a4275a6c586b19a5ea9cb8/merged 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/efe957fea7930099211b0eaadeb834df2acd71bff749a7ae5be847594329d97c/merged 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/8cf904a8ee5d8ec7691df48e795bc3ec8c327bca6cded1058f791bd7db747c67/merged 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/1ceea88b7a93b7816509dd45157cd9621e6e74616ee31b910708665c9c1b3d96/merged 2024-09-15T23:22:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/0ac2cc46bff27289232763fd3d0b46d1ecaf5e6a22facd567818ea8d7f8273c0/merged 2024-09-15T23:22:16.150 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/767004550edce67c20bf3d088bc32164374d4afa10b5f9d6ea4265b24214235a/merged 2024-09-15T23:22:16.150 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/895f3f0b9e463fce3d78de6f1fcb585ce14afa71a5010d65d94e1667b78809ce/merged 2024-09-15T23:22:16.150 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns 2024-09-15T23:22:16.150 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:22:16.150 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:22:16.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:16.150 DEBUG:teuthology.orchestra.run.smithi089:> ls /sys/fs/fuse/connections 2024-09-15T23:22:16.202 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-15T23:22:16.203 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-15T23:22:16.245 DEBUG:teuthology.orchestra.run.smithi089:> sudo modprobe fuse 2024-09-15T23:22:16.270 DEBUG:teuthology.orchestra.run.smithi089:> cat /proc/self/mounts | awk '{print $2}' 2024-09-15T23:22:16.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:16 smithi089 ceph-mon[30933]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:16.326 INFO:teuthology.orchestra.run.smithi089.stdout:/proc 2024-09-15T23:22:16.326 INFO:teuthology.orchestra.run.smithi089.stdout:/sys 2024-09-15T23:22:16.326 INFO:teuthology.orchestra.run.smithi089.stdout:/dev 2024-09-15T23:22:16.326 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/security 2024-09-15T23:22:16.326 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/shm 2024-09-15T23:22:16.326 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/pts 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/run 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/cgroup 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/pstore 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/bpf 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/ 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/selinux 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/hugepages 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/dev/mqueue 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/debug 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/tracing 2024-09-15T23:22:16.327 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/fs/fuse/connections 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/sys/kernel/config 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-sysctl.service 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/nfs/rpc_pipefs 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/run/user/1000 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/ceph 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/c74bd21383dd0a4cd09dc3ab8b9aa80bfc3cdb8ef29a7bda04ba5d8230e734fe/merged 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/e32a2c1615268d17d7dd8f71c700fdf06e7ddfdf77b34efcdac1fa09960a99bd/merged 2024-09-15T23:22:16.328 INFO:teuthology.orchestra.run.smithi089.stdout:/run/user/0 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/d2384accabc5bac7b4d8a0582af713f719257b33305eb412b562271c45903b58/merged 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/efcfd97affcb1727a2698e26eb2c238713148d5ee0a4275a6c586b19a5ea9cb8/merged 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/efe957fea7930099211b0eaadeb834df2acd71bff749a7ae5be847594329d97c/merged 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/8cf904a8ee5d8ec7691df48e795bc3ec8c327bca6cded1058f791bd7db747c67/merged 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/1ceea88b7a93b7816509dd45157cd9621e6e74616ee31b910708665c9c1b3d96/merged 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/0ac2cc46bff27289232763fd3d0b46d1ecaf5e6a22facd567818ea8d7f8273c0/merged 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/767004550edce67c20bf3d088bc32164374d4afa10b5f9d6ea4265b24214235a/merged 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/var/lib/containers/storage/overlay/895f3f0b9e463fce3d78de6f1fcb585ce14afa71a5010d65d94e1667b78809ce/merged 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:22:16.329 INFO:teuthology.orchestra.run.smithi089.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:22:16.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:16.330 DEBUG:teuthology.orchestra.run.smithi089:> ls /sys/fs/fuse/connections 2024-09-15T23:22:16.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi089.stderr:2024-09-15T23:22:16.362+0000 7f4d5671f480 -1 init, newargv = 0x55c0e486dc00 newargc=15 2024-09-15T23:22:16.363 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi089.stderr:ceph-fuse[74562]: starting ceph client 2024-09-15T23:22:16.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi089.stderr:ceph-fuse[74562]: starting fuse 2024-09-15T23:22:16.378 INFO:teuthology.orchestra.run.smithi089.stdout:120 2024-09-15T23:22:16.379 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2024-09-15T23:22:16.379 DEBUG:teuthology.orchestra.run.smithi089:> sudo stdin-killer -- python3 -c ' 2024-09-15T23:22:16.379 DEBUG:teuthology.orchestra.run.smithi089:> import glob 2024-09-15T23:22:16.379 DEBUG:teuthology.orchestra.run.smithi089:> import re 2024-09-15T23:22:16.379 DEBUG:teuthology.orchestra.run.smithi089:> import os 2024-09-15T23:22:16.379 DEBUG:teuthology.orchestra.run.smithi089:> import subprocess 2024-09-15T23:22:16.379 DEBUG:teuthology.orchestra.run.smithi089:> 2024-09-15T23:22:16.379 DEBUG:teuthology.orchestra.run.smithi089:> def _find_admin_socket(client_name): 2024-09-15T23:22:16.379 DEBUG:teuthology.orchestra.run.smithi089:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> files = glob.glob(asok_path) 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> # Given a non-glob path, it better be there 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> if "*" not in asok_path: 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> assert(len(files) == 1) 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> return files[0] 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> for f in files: 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-15T23:22:16.380 DEBUG:teuthology.orchestra.run.smithi089:> contents = proc_f.read() 2024-09-15T23:22:16.381 DEBUG:teuthology.orchestra.run.smithi089:> if mountpoint in contents: 2024-09-15T23:22:16.381 DEBUG:teuthology.orchestra.run.smithi089:> return f 2024-09-15T23:22:16.381 DEBUG:teuthology.orchestra.run.smithi089:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-15T23:22:16.381 DEBUG:teuthology.orchestra.run.smithi089:> 2024-09-15T23:22:16.381 DEBUG:teuthology.orchestra.run.smithi089:> print(_find_admin_socket("client.admin")) 2024-09-15T23:22:16.381 DEBUG:teuthology.orchestra.run.smithi089:> ' 2024-09-15T23:22:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:16 smithi120 ceph-mon[35250]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:16 smithi174 ceph-mon[35586]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:16.496 INFO:teuthology.orchestra.run.smithi089.stdout:/var/run/ceph/ceph-client.admin.74562.asok 2024-09-15T23:22:16.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:22:16.507 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.74562.asok 2024-09-15T23:22:16.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:16.507 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.74562.asok status 2024-09-15T23:22:16.641 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:22:16.641 INFO:teuthology.orchestra.run.smithi089.stdout: "metadata": { 2024-09-15T23:22:16.641 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_sha1": "8921c8ef947807b717db041d85f0761b7914d5fc", 2024-09-15T23:22:16.641 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T23:22:16.641 INFO:teuthology.orchestra.run.smithi089.stdout: "entity_id": "admin", 2024-09-15T23:22:16.641 INFO:teuthology.orchestra.run.smithi089.stdout: "hostname": "smithi089", 2024-09-15T23:22:16.642 INFO:teuthology.orchestra.run.smithi089.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-15T23:22:16.642 INFO:teuthology.orchestra.run.smithi089.stdout: "pid": "74562", 2024-09-15T23:22:16.642 INFO:teuthology.orchestra.run.smithi089.stdout: "root": "/" 2024-09-15T23:22:16.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:22:16.642 INFO:teuthology.orchestra.run.smithi089.stdout: "dentry_count": 0, 2024-09-15T23:22:16.642 INFO:teuthology.orchestra.run.smithi089.stdout: "dentry_pinned_count": 0, 2024-09-15T23:22:16.643 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 14742, 2024-09-15T23:22:16.643 INFO:teuthology.orchestra.run.smithi089.stdout: "inst": { 2024-09-15T23:22:16.643 INFO:teuthology.orchestra.run.smithi089.stdout: "name": { 2024-09-15T23:22:16.643 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "client", 2024-09-15T23:22:16.643 INFO:teuthology.orchestra.run.smithi089.stdout: "num": 14742 2024-09-15T23:22:16.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:22:16.644 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": { 2024-09-15T23:22:16.644 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-15T23:22:16.644 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "192.168.0.1:0", 2024-09-15T23:22:16.644 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1578909788 2024-09-15T23:22:16.644 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:22:16.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:22:16.645 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": { 2024-09-15T23:22:16.645 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-15T23:22:16.645 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "192.168.0.1:0", 2024-09-15T23:22:16.645 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1578909788 2024-09-15T23:22:16.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:22:16.645 INFO:teuthology.orchestra.run.smithi089.stdout: "inst_str": "client.14742 192.168.0.1:0/1578909788", 2024-09-15T23:22:16.645 INFO:teuthology.orchestra.run.smithi089.stdout: "addr_str": "192.168.0.1:0/1578909788", 2024-09-15T23:22:16.646 INFO:teuthology.orchestra.run.smithi089.stdout: "inode_count": 1, 2024-09-15T23:22:16.646 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_epoch": 17, 2024-09-15T23:22:16.646 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_epoch": 86, 2024-09-15T23:22:16.646 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_epoch_barrier": 0, 2024-09-15T23:22:16.646 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklisted": false, 2024-09-15T23:22:16.646 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs" 2024-09-15T23:22:16.646 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:22:16.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:16.649 DEBUG:teuthology.orchestra.run.smithi089:> stat --file-system '--printf=%T 2024-09-15T23:22:16.649 DEBUG:teuthology.orchestra.run.smithi089:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:22:16.665 INFO:teuthology.orchestra.run.smithi089.stdout:fuseblk 2024-09-15T23:22:16.665 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:22:16.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:16.666 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:22:16.732 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-15T23:22:16.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T23:22:16.844 INFO:teuthology.orchestra.run.smithi089.stdout:# file: . 2024-09-15T23:22:16.844 INFO:teuthology.orchestra.run.smithi089.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-15T23:22:16.844 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:16.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:22:16.850 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:22:16.850 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T23:22:16.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-15T23:22:17.156 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:17.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.646+0000 7ff29648d640 1 -- 172.21.15.89:0/3592547596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff290108150 msgr2=0x7ff29010ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:17.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.646+0000 7ff29648d640 1 --2- 172.21.15.89:0/3592547596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff290108150 0x7ff29010ea80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff288009ca0 tx=0x7ff28802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.646+0000 7ff29648d640 1 -- 172.21.15.89:0/3592547596 shutdown_connections 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.646+0000 7ff29648d640 1 --2- 172.21.15.89:0/3592547596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff290108150 0x7ff29010ea80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.646+0000 7ff29648d640 1 --2- 172.21.15.89:0/3592547596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2901012d0 0x7ff290107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.646+0000 7ff29648d640 1 --2- 172.21.15.89:0/3592547596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290100920 0x7ff290100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.646+0000 7ff29648d640 1 -- 172.21.15.89:0/3592547596 >> 172.21.15.89:0/3592547596 conn(0x7ff290075f70 msgr2=0x7ff290076380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.646+0000 7ff29648d640 1 -- 172.21.15.89:0/3592547596 shutdown_connections 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.646+0000 7ff29648d640 1 -- 172.21.15.89:0/3592547596 wait complete. 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.647+0000 7ff29648d640 1 Processor -- start 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.647+0000 7ff29648d640 1 -- start start 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff29648d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff290100920 0x7ff2901983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff29648d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2901012d0 0x7ff290198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff29648d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290108150 0x7ff29019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:17.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff29648d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff290111f80 con 0x7ff290100920 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff29648d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff290111e00 con 0x7ff2901012d0 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff29648d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff290112100 con 0x7ff290108150 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff290100920 0x7ff2901983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff290100920 0x7ff2901983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43716/0 (socket says 172.21.15.89:43716) 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28ffff640 1 -- 172.21.15.89:0/1571377115 learned_addr learned my addr 172.21.15.89:0/1571377115 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28f7fe640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2901012d0 0x7ff290198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff294a2a640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290108150 0x7ff29019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28ffff640 1 -- 172.21.15.89:0/1571377115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290108150 msgr2=0x7ff29019cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28ffff640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290108150 0x7ff29019cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28ffff640 1 -- 172.21.15.89:0/1571377115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2901012d0 msgr2=0x7ff290198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28ffff640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2901012d0 0x7ff290198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28ffff640 1 -- 172.21.15.89:0/1571377115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff29019d370 con 0x7ff290100920 2024-09-15T23:22:17.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff294a2a640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290108150 0x7ff29019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28ffff640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff290100920 0x7ff2901983c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff28000b4a0 tx=0x7ff28000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28d7fa640 1 -- 172.21.15.89:0/1571377115 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280015020 con 0x7ff290100920 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff29648d640 1 -- 172.21.15.89:0/1571377115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff29019d660 con 0x7ff290100920 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28d7fa640 1 -- 172.21.15.89:0/1571377115 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff280011d10 con 0x7ff290100920 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff28d7fa640 1 -- 172.21.15.89:0/1571377115 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28000f2e0 con 0x7ff290100920 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.648+0000 7ff29648d640 1 -- 172.21.15.89:0/1571377115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2901a4f40 con 0x7ff290100920 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.649+0000 7ff29648d640 1 -- 172.21.15.89:0/1571377115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff29006a110 con 0x7ff290100920 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.650+0000 7ff28d7fa640 1 -- 172.21.15.89:0/1571377115 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff280018020 con 0x7ff290100920 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.651+0000 7ff28d7fa640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2780a3240 0x7ff2780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.651+0000 7ff28d7fa640 1 -- 172.21.15.89:0/1571377115 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7ff2800cf490 con 0x7ff290100920 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.651+0000 7ff28f7fe640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2780a3240 0x7ff2780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.652+0000 7ff28f7fe640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2780a3240 0x7ff2780a5700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff290107990 tx=0x7ff28400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:17.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.655+0000 7ff28d7fa640 1 -- 172.21.15.89:0/1571377115 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff28001f020 con 0x7ff290100920 2024-09-15T23:22:17.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.833+0000 7ff29648d640 1 -- 172.21.15.89:0/1571377115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff2901a5140 con 0x7ff290100920 2024-09-15T23:22:17.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.833+0000 7ff28d7fa640 1 -- 172.21.15.89:0/1571377115 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) ==== 81+0+589 (secure 0 0 0) 0x7ff28009bf70 con 0x7ff290100920 2024-09-15T23:22:17.836 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6801/429862840 2024-09-16T23:15:00.884149+0000 2024-09-15T23:22:17.837 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6800/429862840 2024-09-16T23:15:00.884149+0000 2024-09-15T23:22:17.837 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1174639743 2024-09-16T23:15:00.884149+0000 2024-09-15T23:22:17.837 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3889269108 2024-09-16T23:15:00.884149+0000 2024-09-15T23:22:17.837 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/4118871044 2024-09-16T23:15:00.884149+0000 2024-09-15T23:22:17.837 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/379727762 2024-09-16T23:14:32.424565+0000 2024-09-15T23:22:17.837 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6801/4057509998 2024-09-16T23:14:32.424565+0000 2024-09-15T23:22:17.837 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6800/4057509998 2024-09-16T23:14:32.424565+0000 2024-09-15T23:22:17.837 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3796595264 2024-09-16T23:14:32.424565+0000 2024-09-15T23:22:17.837 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1030688282 2024-09-16T23:14:32.424565+0000 2024-09-15T23:22:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:listed 10 entries 2024-09-15T23:22:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 -- 172.21.15.89:0/1571377115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2780a3240 msgr2=0x7ff2780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2780a3240 0x7ff2780a5700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff290107990 tx=0x7ff28400a2d0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 -- 172.21.15.89:0/1571377115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff290100920 msgr2=0x7ff2901983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff290100920 0x7ff2901983c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff28000b4a0 tx=0x7ff28000b8a0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 -- 172.21.15.89:0/1571377115 shutdown_connections 2024-09-15T23:22:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2780a3240 0x7ff2780a5700 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff290108150 0x7ff29019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2901012d0 0x7ff290198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 --2- 172.21.15.89:0/1571377115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff290100920 0x7ff2901983c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:17.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 -- 172.21.15.89:0/1571377115 >> 172.21.15.89:0/1571377115 conn(0x7ff290075f70 msgr2=0x7ff2900ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:17.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 -- 172.21.15.89:0/1571377115 shutdown_connections 2024-09-15T23:22:17.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:17.835+0000 7ff29648d640 1 -- 172.21.15.89:0/1571377115 wait complete. 2024-09-15T23:22:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:18 smithi089 ceph-mon[30933]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:18 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1571377115' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:22:18.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:18 smithi120 ceph-mon[35250]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:18.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:18 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1571377115' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:22:18.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:18 smithi174 ceph-mon[35586]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:18.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:18 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1571377115' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:22:18.519 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:22:18.519 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T23:22:18.537 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-15T23:22:18.825 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.290+0000 7f38dd648640 1 -- 172.21.15.89:0/4126129770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 msgr2=0x7f38d81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.290+0000 7f38dd648640 1 --2- 172.21.15.89:0/4126129770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d81067e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009900 tx=0x7f38c002a410 comp rx=0 tx=0).stop 2024-09-15T23:22:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.290+0000 7f38dd648640 1 -- 172.21.15.89:0/4126129770 shutdown_connections 2024-09-15T23:22:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.290+0000 7f38dd648640 1 --2- 172.21.15.89:0/4126129770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8106f10 0x7f38d810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.290+0000 7f38dd648640 1 --2- 172.21.15.89:0/4126129770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d81067e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.290+0000 7f38dd648640 1 --2- 172.21.15.89:0/4126129770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8101de0 0x7f38d81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:19.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.290+0000 7f38dd648640 1 -- 172.21.15.89:0/4126129770 >> 172.21.15.89:0/4126129770 conn(0x7f38d80fd8c0 msgr2=0x7f38d80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.290+0000 7f38dd648640 1 -- 172.21.15.89:0/4126129770 shutdown_connections 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.290+0000 7f38dd648640 1 -- 172.21.15.89:0/4126129770 wait complete. 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38dd648640 1 Processor -- start 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38dd648640 1 -- start start 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38dd648640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8101de0 0x7f38d81985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38dd648640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d8198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38dd648640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8106f10 0x7f38d819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38dd648640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f38d8112010 con 0x7f38d8102790 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38dd648640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f38d8111e90 con 0x7f38d8101de0 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38dd648640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f38d8112190 con 0x7f38d8106f10 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d8198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:19.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d8198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43740/0 (socket says 172.21.15.89:43740) 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38cffff640 1 -- 172.21.15.89:0/2078140979 learned_addr learned my addr 172.21.15.89:0/2078140979 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38d6ffd640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8101de0 0x7f38d81985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38cffff640 1 -- 172.21.15.89:0/2078140979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8106f10 msgr2=0x7f38d819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38cffff640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8106f10 0x7f38d819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38cffff640 1 -- 172.21.15.89:0/2078140979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8101de0 msgr2=0x7f38d81985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38cffff640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8101de0 0x7f38d81985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.291+0000 7f38cffff640 1 -- 172.21.15.89:0/2078140979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d819d560 con 0x7f38d8102790 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.292+0000 7f38cffff640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d8198af0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f38c0005ab0 tx=0x7f38c00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.292+0000 7f38d4ff9640 1 -- 172.21.15.89:0/2078140979 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c003f030 con 0x7f38d8102790 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.292+0000 7f38d4ff9640 1 -- 172.21.15.89:0/2078140979 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c0003ce0 con 0x7f38d8102790 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.292+0000 7f38dd648640 1 -- 172.21.15.89:0/2078140979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d819d7f0 con 0x7f38d8102790 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.292+0000 7f38d4ff9640 1 -- 172.21.15.89:0/2078140979 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c0035070 con 0x7f38d8102790 2024-09-15T23:22:19.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.292+0000 7f38dd648640 1 -- 172.21.15.89:0/2078140979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d81a5130 con 0x7f38d8102790 2024-09-15T23:22:19.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.293+0000 7f38dd648640 1 -- 172.21.15.89:0/2078140979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d8102240 con 0x7f38d8102790 2024-09-15T23:22:19.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.294+0000 7f38d4ff9640 1 -- 172.21.15.89:0/2078140979 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f38c0042030 con 0x7f38d8102790 2024-09-15T23:22:19.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.294+0000 7f38d4ff9640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a32c0 0x7f38c40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:19.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.294+0000 7f38d6ffd640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a32c0 0x7f38c40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:19.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.295+0000 7f38d4ff9640 1 -- 172.21.15.89:0/2078140979 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f38c00f0430 con 0x7f38d8102790 2024-09-15T23:22:19.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.295+0000 7f38d6ffd640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a32c0 0x7f38c40a5780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009990 tx=0x7f38c8009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:19.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.299+0000 7f38d4ff9640 1 -- 172.21.15.89:0/2078140979 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f38c00bcf10 con 0x7f38d8102790 2024-09-15T23:22:19.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.475+0000 7f38dd648640 1 -- 172.21.15.89:0/2078140979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f38d802d7c0 con 0x7f38d8102790 2024-09-15T23:22:19.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.475+0000 7f38d4ff9640 1 -- 172.21.15.89:0/2078140979 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) ==== 81+0+589 (secure 0 0 0) 0x7f38c00bc810 con 0x7f38d8102790 2024-09-15T23:22:19.477 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6801/429862840 2024-09-16T23:15:00.884149+0000 2024-09-15T23:22:19.477 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6800/429862840 2024-09-16T23:15:00.884149+0000 2024-09-15T23:22:19.477 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1174639743 2024-09-16T23:15:00.884149+0000 2024-09-15T23:22:19.477 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3889269108 2024-09-16T23:15:00.884149+0000 2024-09-15T23:22:19.477 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/4118871044 2024-09-16T23:15:00.884149+0000 2024-09-15T23:22:19.477 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/379727762 2024-09-16T23:14:32.424565+0000 2024-09-15T23:22:19.477 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6801/4057509998 2024-09-16T23:14:32.424565+0000 2024-09-15T23:22:19.477 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6800/4057509998 2024-09-16T23:14:32.424565+0000 2024-09-15T23:22:19.477 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3796595264 2024-09-16T23:14:32.424565+0000 2024-09-15T23:22:19.477 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1030688282 2024-09-16T23:14:32.424565+0000 2024-09-15T23:22:19.478 INFO:teuthology.orchestra.run.smithi089.stderr:listed 10 entries 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.477+0000 7f38dd648640 1 -- 172.21.15.89:0/2078140979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a32c0 msgr2=0x7f38c40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.477+0000 7f38dd648640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a32c0 0x7f38c40a5780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009990 tx=0x7f38c8009110 comp rx=0 tx=0).stop 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.477+0000 7f38dd648640 1 -- 172.21.15.89:0/2078140979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 msgr2=0x7f38d8198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.477+0000 7f38dd648640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d8198af0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f38c0005ab0 tx=0x7f38c00042f0 comp rx=0 tx=0).stop 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.478+0000 7f38dd648640 1 -- 172.21.15.89:0/2078140979 shutdown_connections 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.478+0000 7f38dd648640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a32c0 0x7f38c40a5780 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.478+0000 7f38dd648640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8106f10 0x7f38d819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.478+0000 7f38dd648640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d8198af0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.478+0000 7f38dd648640 1 --2- 172.21.15.89:0/2078140979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8101de0 0x7f38d81985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.478+0000 7f38dd648640 1 -- 172.21.15.89:0/2078140979 >> 172.21.15.89:0/2078140979 conn(0x7f38d80fd8c0 msgr2=0x7f38d80fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:19.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.478+0000 7f38dd648640 1 -- 172.21.15.89:0/2078140979 shutdown_connections 2024-09-15T23:22:19.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:19.478+0000 7f38dd648640 1 -- 172.21.15.89:0/2078140979 wait complete. 2024-09-15T23:22:20.149 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:20 smithi089 ceph-mon[30933]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:20.149 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:20 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2078140979' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:22:20.177 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi089.front.sepia.ceph.com... 2024-09-15T23:22:20.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:20.177 DEBUG:teuthology.orchestra.run.smithi089:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:22:20.202 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi089.stderr:ceph-fuse[74562]: fuse finished with error 0 and tester_r 0 2024-09-15T23:22:20.205 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T23:22:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:20 smithi120 ceph-mon[35250]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:20 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2078140979' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:22:20.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:20 smithi174 ceph-mon[35586]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:20.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:20 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2078140979' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:22:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:22 smithi120 ceph-mon[35250]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:22.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:22 smithi174 ceph-mon[35586]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:22.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:22 smithi089 ceph-mon[30933]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:22:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:24 smithi120 ceph-mon[35250]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:24.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:24 smithi174 ceph-mon[35586]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:24.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:24 smithi089 ceph-mon[30933]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:26.307 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-09-15T23:22:26.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:22:26.307 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-15T23:22:26.333 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:22:26.399 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:26 smithi089 ceph-mon[30933]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:26 smithi120 ceph-mon[35250]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:26 smithi174 ceph-mon[35586]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:26.631 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.124+0000 7f53c674f640 1 -- 172.21.15.89:0/3708787765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c01082a0 msgr2=0x7f53c010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.124+0000 7f53c674f640 1 --2- 172.21.15.89:0/3708787765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c01082a0 0x7f53c010ec40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009cd0 tx=0x7f53b802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.124+0000 7f53c674f640 1 -- 172.21.15.89:0/3708787765 shutdown_connections 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.124+0000 7f53c674f640 1 --2- 172.21.15.89:0/3708787765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c01082a0 0x7f53c010ec40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.124+0000 7f53c674f640 1 --2- 172.21.15.89:0/3708787765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c01036e0 0x7f53c0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.124+0000 7f53c674f640 1 --2- 172.21.15.89:0/3708787765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c0102d30 0x7f53c0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.124+0000 7f53c674f640 1 -- 172.21.15.89:0/3708787765 >> 172.21.15.89:0/3708787765 conn(0x7f53c00fc7d0 msgr2=0x7f53c00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.124+0000 7f53c674f640 1 -- 172.21.15.89:0/3708787765 shutdown_connections 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.124+0000 7f53c674f640 1 -- 172.21.15.89:0/3708787765 wait complete. 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53c674f640 1 Processor -- start 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53c674f640 1 -- start start 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53c674f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c0102d30 0x7f53c0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53c674f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c01036e0 0x7f53c0198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:27.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53c674f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c01082a0 0x7f53c019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53c674f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f53c01120a0 con 0x7f53c0102d30 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53c674f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f53c0111f20 con 0x7f53c01082a0 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53c674f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f53c0112220 con 0x7f53c01036e0 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c0102d30 0x7f53c0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c0102d30 0x7f53c0198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46238/0 (socket says 172.21.15.89:46238) 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53bffff640 1 -- 172.21.15.89:0/2321386022 learned_addr learned my addr 172.21.15.89:0/2321386022 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53c4cec640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c01082a0 0x7f53c019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53b7fff640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c01036e0 0x7f53c0198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53bffff640 1 -- 172.21.15.89:0/2321386022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c01036e0 msgr2=0x7f53c0198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53bffff640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c01036e0 0x7f53c0198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53bffff640 1 -- 172.21.15.89:0/2321386022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c01082a0 msgr2=0x7f53c019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53bffff640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c01082a0 0x7f53c019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53bffff640 1 -- 172.21.15.89:0/2321386022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c019d4e0 con 0x7f53c0102d30 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53b7fff640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c01036e0 0x7f53c0198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53c4cec640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c01082a0 0x7f53c019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.125+0000 7f53bffff640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c0102d30 0x7f53c0198530 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f53b000d770 tx=0x7f53b000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.126+0000 7f53bdffb640 1 -- 172.21.15.89:0/2321386022 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b0017070 con 0x7f53c0102d30 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.126+0000 7f53bdffb640 1 -- 172.21.15.89:0/2321386022 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b0005500 con 0x7f53c0102d30 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.126+0000 7f53bdffb640 1 -- 172.21.15.89:0/2321386022 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b0004e00 con 0x7f53c0102d30 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.126+0000 7f53c674f640 1 -- 172.21.15.89:0/2321386022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c019d7d0 con 0x7f53c0102d30 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.126+0000 7f53c674f640 1 -- 172.21.15.89:0/2321386022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c01a50b0 con 0x7f53c0102d30 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.127+0000 7f53c674f640 1 -- 172.21.15.89:0/2321386022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53c0103110 con 0x7f53c0102d30 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.128+0000 7f53bdffb640 1 -- 172.21.15.89:0/2321386022 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f53b0004fa0 con 0x7f53c0102d30 2024-09-15T23:22:27.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.128+0000 7f53bdffb640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53ac0a3240 0x7f53ac0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:27.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.129+0000 7f53b7fff640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53ac0a3240 0x7f53ac0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:27.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.129+0000 7f53bdffb640 1 -- 172.21.15.89:0/2321386022 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f53b00cf680 con 0x7f53c0102d30 2024-09-15T23:22:27.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.129+0000 7f53b7fff640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53ac0a3240 0x7f53ac0a5700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f53a8005fd0 tx=0x7f53a80094d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:27.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.133+0000 7f53bdffb640 1 -- 172.21.15.89:0/2321386022 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f53b009c160 con 0x7f53c0102d30 2024-09-15T23:22:27.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.300+0000 7f53c674f640 1 -- 172.21.15.89:0/2321386022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53c010a2b0 con 0x7f53c0102d30 2024-09-15T23:22:27.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.302+0000 7f53bdffb640 1 -- 172.21.15.89:0/2321386022 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24199 (secure 0 0 0) 0x7f53b009ba60 con 0x7f53c0102d30 2024-09-15T23:22:27.304 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:27.304 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":86,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:22:02.246640+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:22:27.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 -- 172.21.15.89:0/2321386022 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53ac0a3240 msgr2=0x7f53ac0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:27.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53ac0a3240 0x7f53ac0a5700 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f53a8005fd0 tx=0x7f53a80094d0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 -- 172.21.15.89:0/2321386022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c0102d30 msgr2=0x7f53c0198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:27.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c0102d30 0x7f53c0198530 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f53b000d770 tx=0x7f53b000dc30 comp rx=0 tx=0).stop 2024-09-15T23:22:27.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 -- 172.21.15.89:0/2321386022 shutdown_connections 2024-09-15T23:22:27.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53ac0a3240 0x7f53ac0a5700 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c01082a0 0x7f53c019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c01036e0 0x7f53c0198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 --2- 172.21.15.89:0/2321386022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c0102d30 0x7f53c0198530 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:27.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 -- 172.21.15.89:0/2321386022 >> 172.21.15.89:0/2321386022 conn(0x7f53c00fc7d0 msgr2=0x7f53c0105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:27.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 -- 172.21.15.89:0/2321386022 shutdown_connections 2024-09-15T23:22:27.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:27.304+0000 7f53c674f640 1 -- 172.21.15.89:0/2321386022 wait complete. 2024-09-15T23:22:28.157 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:22:28.444 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:28.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:28 smithi120 ceph-mon[35250]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:28.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:28 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2321386022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:28.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:28 smithi174 ceph-mon[35586]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:28.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:28 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2321386022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:28.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:28 smithi089 ceph-mon[30933]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:28.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:28 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2321386022' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.915+0000 7f345135a640 1 -- 172.21.15.89:0/1380914378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344c106a60 msgr2=0x7f344c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.915+0000 7f345135a640 1 --2- 172.21.15.89:0/1380914378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344c106a60 0x7f344c106e40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f343c009cd0 tx=0x7f343c02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:22:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.915+0000 7f345135a640 1 -- 172.21.15.89:0/1380914378 shutdown_connections 2024-09-15T23:22:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.915+0000 7f345135a640 1 --2- 172.21.15.89:0/1380914378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344c101400 0x7f344c10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.915+0000 7f345135a640 1 --2- 172.21.15.89:0/1380914378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f344c100a60 0x7f344c100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.915+0000 7f345135a640 1 --2- 172.21.15.89:0/1380914378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344c106a60 0x7f344c106e40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.915+0000 7f345135a640 1 -- 172.21.15.89:0/1380914378 >> 172.21.15.89:0/1380914378 conn(0x7f344c0fc780 msgr2=0x7f344c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.915+0000 7f345135a640 1 -- 172.21.15.89:0/1380914378 shutdown_connections 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.915+0000 7f345135a640 1 -- 172.21.15.89:0/1380914378 wait complete. 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f345135a640 1 Processor -- start 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f345135a640 1 -- start start 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f345135a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344c100a60 0x7f344c197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f345135a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344c101400 0x7f344c197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f345135a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f344c106a60 0x7f344c19bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f345135a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f344c110e20 con 0x7f344c101400 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f345135a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f344c110ca0 con 0x7f344c100a60 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f345135a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f344c110fa0 con 0x7f344c106a60 2024-09-15T23:22:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344c101400 0x7f344c197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344c101400 0x7f344c197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46262/0 (socket says 172.21.15.89:46262) 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344a7fc640 1 -- 172.21.15.89:0/2674965628 learned_addr learned my addr 172.21.15.89:0/2674965628 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344affd640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344c100a60 0x7f344c197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344b7fe640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f344c106a60 0x7f344c19bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344a7fc640 1 -- 172.21.15.89:0/2674965628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f344c106a60 msgr2=0x7f344c19bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344a7fc640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f344c106a60 0x7f344c19bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344a7fc640 1 -- 172.21.15.89:0/2674965628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344c100a60 msgr2=0x7f344c197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344a7fc640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344c100a60 0x7f344c197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344a7fc640 1 -- 172.21.15.89:0/2674965628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344c19c2b0 con 0x7f344c101400 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344b7fe640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f344c106a60 0x7f344c19bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.916+0000 7f344affd640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344c100a60 0x7f344c197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.917+0000 7f344a7fc640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344c101400 0x7f344c197840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3434007b50 tx=0x7f343400e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:28.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.917+0000 7f3443fff640 1 -- 172.21.15.89:0/2674965628 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434007c40 con 0x7f344c101400 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.917+0000 7f3443fff640 1 -- 172.21.15.89:0/2674965628 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3434007de0 con 0x7f344c101400 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.917+0000 7f345135a640 1 -- 172.21.15.89:0/2674965628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344c19c5a0 con 0x7f344c101400 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.917+0000 7f3443fff640 1 -- 172.21.15.89:0/2674965628 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343400ae60 con 0x7f344c101400 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.917+0000 7f345135a640 1 -- 172.21.15.89:0/2674965628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344c06a790 con 0x7f344c101400 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.918+0000 7f345135a640 1 -- 172.21.15.89:0/2674965628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f344c100ec0 con 0x7f344c101400 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.919+0000 7f3443fff640 1 -- 172.21.15.89:0/2674965628 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f343400a440 con 0x7f344c101400 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.919+0000 7f3443fff640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f34380a3240 0x7f34380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.920+0000 7f3443fff640 1 -- 172.21.15.89:0/2674965628 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f34340d1100 con 0x7f344c101400 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.920+0000 7f344affd640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f34380a3240 0x7f34380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.920+0000 7f344affd640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f34380a3240 0x7f34380a5700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f343c000f80 tx=0x7f343c0093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:28.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:28.924+0000 7f3443fff640 1 -- 172.21.15.89:0/2674965628 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3434023060 con 0x7f344c101400 2024-09-15T23:22:29.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.094+0000 7f345135a640 1 -- 172.21.15.89:0/2674965628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f344c06b1e0 con 0x7f344c101400 2024-09-15T23:22:29.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.095+0000 7f3443fff640 1 -- 172.21.15.89:0/2674965628 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24199 (secure 0 0 0) 0x7f343409dc70 con 0x7f344c101400 2024-09-15T23:22:29.096 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:29.097 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":86,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:22:02.246640+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:22:29.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 -- 172.21.15.89:0/2674965628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f34380a3240 msgr2=0x7f34380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f34380a3240 0x7f34380a5700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f343c000f80 tx=0x7f343c0093f0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 -- 172.21.15.89:0/2674965628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344c101400 msgr2=0x7f344c197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344c101400 0x7f344c197840 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3434007b50 tx=0x7f343400e420 comp rx=0 tx=0).stop 2024-09-15T23:22:29.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 -- 172.21.15.89:0/2674965628 shutdown_connections 2024-09-15T23:22:29.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f34380a3240 0x7f34380a5700 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f344c106a60 0x7f344c19bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344c101400 0x7f344c197840 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 --2- 172.21.15.89:0/2674965628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344c100a60 0x7f344c197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 -- 172.21.15.89:0/2674965628 >> 172.21.15.89:0/2674965628 conn(0x7f344c0fc780 msgr2=0x7f344c0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:29.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 -- 172.21.15.89:0/2674965628 shutdown_connections 2024-09-15T23:22:29.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:29.097+0000 7f345135a640 1 -- 172.21.15.89:0/2674965628 wait complete. 2024-09-15T23:22:29.938 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-09-15T23:22:30.192 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:30.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:30 smithi089 ceph-mon[30933]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:30.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:30 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2674965628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:30 smithi120 ceph-mon[35250]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:30 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2674965628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:30.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:30 smithi174 ceph-mon[35586]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:30.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:30 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2674965628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:30.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.704+0000 7f55e084f640 1 -- 172.21.15.89:0/198206566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55dc106f10 msgr2=0x7f55dc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:30.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.704+0000 7f55e084f640 1 --2- 172.21.15.89:0/198206566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55dc106f10 0x7f55dc10ea90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f55d4009ca0 tx=0x7f55d402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:22:30.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.704+0000 7f55e084f640 1 -- 172.21.15.89:0/198206566 shutdown_connections 2024-09-15T23:22:30.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.704+0000 7f55e084f640 1 --2- 172.21.15.89:0/198206566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55dc106f10 0x7f55dc10ea90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:30.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.704+0000 7f55e084f640 1 --2- 172.21.15.89:0/198206566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55dc102790 0x7f55dc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:30.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.704+0000 7f55e084f640 1 --2- 172.21.15.89:0/198206566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55dc101de0 0x7f55dc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:30.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.704+0000 7f55e084f640 1 -- 172.21.15.89:0/198206566 >> 172.21.15.89:0/198206566 conn(0x7f55dc0fd8c0 msgr2=0x7f55dc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:30.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.704+0000 7f55e084f640 1 -- 172.21.15.89:0/198206566 shutdown_connections 2024-09-15T23:22:30.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.704+0000 7f55e084f640 1 -- 172.21.15.89:0/198206566 wait complete. 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55e084f640 1 Processor -- start 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55e084f640 1 -- start start 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55e084f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55dc101de0 0x7f55dc1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55e084f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55dc102790 0x7f55dc198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55e084f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55dc106f10 0x7f55dc19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55e084f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc112010 con 0x7f55dc106f10 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55e084f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc111e90 con 0x7f55dc102790 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55e084f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc112190 con 0x7f55dc101de0 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55dad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55dc106f10 0x7f55dc19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55dad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55dc106f10 0x7f55dc19cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46284/0 (socket says 172.21.15.89:46284) 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55dad9d640 1 -- 172.21.15.89:0/3023647958 learned_addr learned my addr 172.21.15.89:0/3023647958 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.705+0000 7f55d9d9b640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55dc102790 0x7f55dc198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:30.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55da59c640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55dc101de0 0x7f55dc1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55dad9d640 1 -- 172.21.15.89:0/3023647958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55dc101de0 msgr2=0x7f55dc1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55dad9d640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55dc101de0 0x7f55dc1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55dad9d640 1 -- 172.21.15.89:0/3023647958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55dc102790 msgr2=0x7f55dc198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55dad9d640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55dc102790 0x7f55dc198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55dad9d640 1 -- 172.21.15.89:0/3023647958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55dc19d5a0 con 0x7f55dc106f10 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55dad9d640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55dc106f10 0x7f55dc19cec0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f55d4031a60 tx=0x7f55d4031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55cb7fe640 1 -- 172.21.15.89:0/3023647958 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4003c30 con 0x7f55dc106f10 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55cb7fe640 1 -- 172.21.15.89:0/3023647958 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55d4003dd0 con 0x7f55dc106f10 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55e084f640 1 -- 172.21.15.89:0/3023647958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55dc19d830 con 0x7f55dc106f10 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55cb7fe640 1 -- 172.21.15.89:0/3023647958 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d40047e0 con 0x7f55dc106f10 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.706+0000 7f55e084f640 1 -- 172.21.15.89:0/3023647958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55dc1a5170 con 0x7f55dc106f10 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.707+0000 7f55e084f640 1 -- 172.21.15.89:0/3023647958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55dc102240 con 0x7f55dc106f10 2024-09-15T23:22:30.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.708+0000 7f55cb7fe640 1 -- 172.21.15.89:0/3023647958 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f55d402f790 con 0x7f55dc106f10 2024-09-15T23:22:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.709+0000 7f55cb7fe640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f55c40a3240 0x7f55c40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.709+0000 7f55da59c640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f55c40a3240 0x7f55c40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.709+0000 7f55cb7fe640 1 -- 172.21.15.89:0/3023647958 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f55d40f1200 con 0x7f55dc106f10 2024-09-15T23:22:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.709+0000 7f55da59c640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f55c40a3240 0x7f55c40a5700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00a810 tx=0x7f55cc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:30.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.713+0000 7f55cb7fe640 1 -- 172.21.15.89:0/3023647958 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f55d40bdce0 con 0x7f55dc106f10 2024-09-15T23:22:30.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:30.912+0000 7f55e084f640 1 -- 172.21.15.89:0/3023647958 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f55dc10a150 con 0x7f55dc106f10 2024-09-15T23:22:31.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.081+0000 7f55cb7fe640 1 -- 172.21.15.89:0/3023647958 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f55d40bd5e0 con 0x7f55dc106f10 2024-09-15T23:22:31.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.083+0000 7f55e084f640 1 -- 172.21.15.89:0/3023647958 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f55c40a3240 msgr2=0x7f55c40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:31.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.083+0000 7f55e084f640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f55c40a3240 0x7f55c40a5700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00a810 tx=0x7f55cc008040 comp rx=0 tx=0).stop 2024-09-15T23:22:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.083+0000 7f55e084f640 1 -- 172.21.15.89:0/3023647958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55dc106f10 msgr2=0x7f55dc19cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.083+0000 7f55e084f640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55dc106f10 0x7f55dc19cec0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f55d4031a60 tx=0x7f55d4031a90 comp rx=0 tx=0).stop 2024-09-15T23:22:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.083+0000 7f55e084f640 1 -- 172.21.15.89:0/3023647958 shutdown_connections 2024-09-15T23:22:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.083+0000 7f55e084f640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f55c40a3240 0x7f55c40a5700 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.083+0000 7f55e084f640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55dc106f10 0x7f55dc19cec0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.083+0000 7f55e084f640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55dc102790 0x7f55dc198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.083+0000 7f55e084f640 1 --2- 172.21.15.89:0/3023647958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55dc101de0 0x7f55dc1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.083+0000 7f55e084f640 1 -- 172.21.15.89:0/3023647958 >> 172.21.15.89:0/3023647958 conn(0x7f55dc0fd8c0 msgr2=0x7f55dc0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.084+0000 7f55e084f640 1 -- 172.21.15.89:0/3023647958 shutdown_connections 2024-09-15T23:22:31.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:31.084+0000 7f55e084f640 1 -- 172.21.15.89:0/3023647958 wait complete. 2024-09-15T23:22:31.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:31 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3023647958' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-15T23:22:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:31 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3023647958' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-15T23:22:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:31 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3023647958' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-15T23:22:31.879 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-09-15T23:22:31.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-09-15T23:22:32.127 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:32.247 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:32 smithi089 ceph-mon[30933]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:32.247 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:32 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3023647958' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-15T23:22:32.247 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:32 smithi089 ceph-mon[30933]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:22:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:32 smithi120 ceph-mon[35250]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:32 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3023647958' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-15T23:22:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:32 smithi120 ceph-mon[35250]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:22:32.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:32 smithi174 ceph-mon[35586]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:32.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:32 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3023647958' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-15T23:22:32.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:32 smithi174 ceph-mon[35586]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:22:32.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.622+0000 7f42ccdde640 1 -- 172.21.15.89:0/757874403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42c8106a80 msgr2=0x7f42c8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:32.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.622+0000 7f42ccdde640 1 --2- 172.21.15.89:0/757874403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42c8106a80 0x7f42c8106e60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f42b8009930 tx=0x7f42b802a420 comp rx=0 tx=0).stop 2024-09-15T23:22:32.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.622+0000 7f42ccdde640 1 -- 172.21.15.89:0/757874403 shutdown_connections 2024-09-15T23:22:32.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.622+0000 7f42ccdde640 1 --2- 172.21.15.89:0/757874403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c8101420 0x7f42c810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:32.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.622+0000 7f42ccdde640 1 --2- 172.21.15.89:0/757874403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c8100a80 0x7f42c8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:32.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.622+0000 7f42ccdde640 1 --2- 172.21.15.89:0/757874403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42c8106a80 0x7f42c8106e60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:32.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.622+0000 7f42ccdde640 1 -- 172.21.15.89:0/757874403 >> 172.21.15.89:0/757874403 conn(0x7f42c80fc7a0 msgr2=0x7f42c80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:32.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.622+0000 7f42ccdde640 1 -- 172.21.15.89:0/757874403 shutdown_connections 2024-09-15T23:22:32.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.622+0000 7f42ccdde640 1 -- 172.21.15.89:0/757874403 wait complete. 2024-09-15T23:22:32.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42ccdde640 1 Processor -- start 2024-09-15T23:22:32.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42ccdde640 1 -- start start 2024-09-15T23:22:32.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42ccdde640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c8100a80 0x7f42c8198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:32.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42ccdde640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42c8101420 0x7f42c8198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:32.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42ccdde640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c8106a80 0x7f42c819cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:32.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42ccdde640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42c8110f20 con 0x7f42c8101420 2024-09-15T23:22:32.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42ccdde640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f42c8110da0 con 0x7f42c8106a80 2024-09-15T23:22:32.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42ccdde640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42c81110a0 con 0x7f42c8100a80 2024-09-15T23:22:32.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42c8101420 0x7f42c8198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:32.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42c8101420 0x7f42c8198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46290/0 (socket says 172.21.15.89:46290) 2024-09-15T23:22:32.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42bffff640 1 -- 172.21.15.89:0/1994049223 learned_addr learned my addr 172.21.15.89:0/1994049223 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:32.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42c659c640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c8100a80 0x7f42c8198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:32.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42bffff640 1 -- 172.21.15.89:0/1994049223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c8100a80 msgr2=0x7f42c8198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:32.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42bffff640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c8100a80 0x7f42c8198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:32.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42c6d9d640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c8106a80 0x7f42c819cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:32.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.623+0000 7f42bffff640 1 -- 172.21.15.89:0/1994049223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c8106a80 msgr2=0x7f42c819cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:32.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.624+0000 7f42bffff640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c8106a80 0x7f42c819cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:32.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.624+0000 7f42bffff640 1 -- 172.21.15.89:0/1994049223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c819d5c0 con 0x7f42c8101420 2024-09-15T23:22:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.624+0000 7f42c6d9d640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c8106a80 0x7f42c819cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.624+0000 7f42bffff640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42c8101420 0x7f42c8198b50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f42b000b4a0 tx=0x7f42b000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.624+0000 7f42bf7fe640 1 -- 172.21.15.89:0/1994049223 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0014070 con 0x7f42c8101420 2024-09-15T23:22:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.624+0000 7f42bf7fe640 1 -- 172.21.15.89:0/1994049223 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42b0010af0 con 0x7f42c8101420 2024-09-15T23:22:32.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.624+0000 7f42ccdde640 1 -- 172.21.15.89:0/1994049223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c819d8b0 con 0x7f42c8101420 2024-09-15T23:22:32.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.624+0000 7f42bf7fe640 1 -- 172.21.15.89:0/1994049223 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b0017d10 con 0x7f42c8101420 2024-09-15T23:22:32.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.624+0000 7f42ccdde640 1 -- 172.21.15.89:0/1994049223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c81a5190 con 0x7f42c8101420 2024-09-15T23:22:32.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.625+0000 7f42ccdde640 1 -- 172.21.15.89:0/1994049223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c8100f60 con 0x7f42c8101420 2024-09-15T23:22:32.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.626+0000 7f42bf7fe640 1 -- 172.21.15.89:0/1994049223 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f42b0017070 con 0x7f42c8101420 2024-09-15T23:22:32.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.627+0000 7f42bf7fe640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42b40a3240 0x7f42b40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:32.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.627+0000 7f42c659c640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42b40a3240 0x7f42b40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:32.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.627+0000 7f42bf7fe640 1 -- 172.21.15.89:0/1994049223 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f42b00ceff0 con 0x7f42c8101420 2024-09-15T23:22:32.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.627+0000 7f42c659c640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42b40a3240 0x7f42b40a5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f42b8000c00 tx=0x7f42b8031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:32.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.631+0000 7f42bf7fe640 1 -- 172.21.15.89:0/1994049223 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f42b009bab0 con 0x7f42c8101420 2024-09-15T23:22:32.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:32.831+0000 7f42ccdde640 1 -- 172.21.15.89:0/1994049223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f42c8109a70 con 0x7f42c8101420 2024-09-15T23:22:33.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.079+0000 7f42bf7fe640 1 -- 172.21.15.89:0/1994049223 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f42b009b3b0 con 0x7f42c8101420 2024-09-15T23:22:33.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 -- 172.21.15.89:0/1994049223 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42b40a3240 msgr2=0x7f42b40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42b40a3240 0x7f42b40a5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f42b8000c00 tx=0x7f42b8031490 comp rx=0 tx=0).stop 2024-09-15T23:22:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 -- 172.21.15.89:0/1994049223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42c8101420 msgr2=0x7f42c8198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42c8101420 0x7f42c8198b50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f42b000b4a0 tx=0x7f42b000b960 comp rx=0 tx=0).stop 2024-09-15T23:22:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 -- 172.21.15.89:0/1994049223 shutdown_connections 2024-09-15T23:22:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42b40a3240 0x7f42b40a5700 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42c8106a80 0x7f42c819cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f42c8101420 0x7f42c8198b50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:33.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 --2- 172.21.15.89:0/1994049223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42c8100a80 0x7f42c8198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:33.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 -- 172.21.15.89:0/1994049223 >> 172.21.15.89:0/1994049223 conn(0x7f42c80fc7a0 msgr2=0x7f42c806aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:33.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 -- 172.21.15.89:0/1994049223 shutdown_connections 2024-09-15T23:22:33.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:33.081+0000 7f42ccdde640 1 -- 172.21.15.89:0/1994049223 wait complete. 2024-09-15T23:22:33.085 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:33 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1994049223' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-15T23:22:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:33 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1994049223' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-15T23:22:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:33 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1994049223' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-09-15T23:22:33.811 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-09-15T23:22:34.057 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:34.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:34 smithi089 ceph-mon[30933]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:34.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:34 smithi089 ceph-mon[30933]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-15T23:22:34.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:34 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1994049223' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-15T23:22:34.342 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:34 smithi089 ceph-mon[30933]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-15T23:22:34.342 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:34 smithi089 ceph-mon[30933]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:22:34.342 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:34 smithi089 ceph-mon[30933]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-15T23:22:34.342 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:34 smithi089 ceph-mon[30933]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-15T23:22:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:34 smithi120 ceph-mon[35250]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:34 smithi120 ceph-mon[35250]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-15T23:22:34.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:34 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1994049223' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-15T23:22:34.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:34 smithi120 ceph-mon[35250]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-15T23:22:34.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:34 smithi120 ceph-mon[35250]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:22:34.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:34 smithi120 ceph-mon[35250]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-15T23:22:34.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:34 smithi120 ceph-mon[35250]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-15T23:22:34.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:34 smithi174 ceph-mon[35586]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:22:34.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:34 smithi174 ceph-mon[35586]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-15T23:22:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:34 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1994049223' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-09-15T23:22:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:34 smithi174 ceph-mon[35586]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-15T23:22:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:34 smithi174 ceph-mon[35586]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:22:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:34 smithi174 ceph-mon[35586]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-09-15T23:22:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:34 smithi174 ceph-mon[35586]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-09-15T23:22:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.530+0000 7fde77ed4640 1 -- 172.21.15.89:0/465670607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde70100b30 msgr2=0x7fde70100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:34.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.530+0000 7fde77ed4640 1 --2- 172.21.15.89:0/465670607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde70100b30 0x7fde70100f10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009900 tx=0x7fde6c02a470 comp rx=0 tx=0).stop 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.530+0000 7fde77ed4640 1 -- 172.21.15.89:0/465670607 shutdown_connections 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.530+0000 7fde77ed4640 1 --2- 172.21.15.89:0/465670607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde70105f40 0x7fde701063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.530+0000 7fde77ed4640 1 --2- 172.21.15.89:0/465670607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde70101450 0x7fde70105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 --2- 172.21.15.89:0/465670607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde70100b30 0x7fde70100f10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 -- 172.21.15.89:0/465670607 >> 172.21.15.89:0/465670607 conn(0x7fde700fc7a0 msgr2=0x7fde700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 -- 172.21.15.89:0/465670607 shutdown_connections 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 -- 172.21.15.89:0/465670607 wait complete. 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 Processor -- start 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 -- start start 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde70100b30 0x7fde701a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde70101450 0x7fde701a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde70105f40 0x7fde7019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:34.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fde701185c0 con 0x7fde70105f40 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fde70118440 con 0x7fde70101450 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.531+0000 7fde77ed4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fde70118740 con 0x7fde70100b30 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde76471640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde70105f40 0x7fde7019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde76471640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde70105f40 0x7fde7019e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46308/0 (socket says 172.21.15.89:46308) 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde76471640 1 -- 172.21.15.89:0/560418252 learned_addr learned my addr 172.21.15.89:0/560418252 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde75c70640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde70100b30 0x7fde701a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde76471640 1 -- 172.21.15.89:0/560418252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde70100b30 msgr2=0x7fde701a4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde76471640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde70100b30 0x7fde701a4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde76471640 1 -- 172.21.15.89:0/560418252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde70101450 msgr2=0x7fde701a4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde7546f640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde70101450 0x7fde701a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde76471640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde70101450 0x7fde701a4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde76471640 1 -- 172.21.15.89:0/560418252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde7019f080 con 0x7fde70105f40 2024-09-15T23:22:34.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde76471640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde70105f40 0x7fde7019e950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fde6800cad0 tx=0x7fde6800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde66ffd640 1 -- 172.21.15.89:0/560418252 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6800ae20 con 0x7fde70105f40 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde66ffd640 1 -- 172.21.15.89:0/560418252 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde68005020 con 0x7fde70105f40 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde77ed4640 1 -- 172.21.15.89:0/560418252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde7019f370 con 0x7fde70105f40 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde66ffd640 1 -- 172.21.15.89:0/560418252 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6800f6f0 con 0x7fde70105f40 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.532+0000 7fde77ed4640 1 -- 172.21.15.89:0/560418252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde701ab660 con 0x7fde70105f40 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.533+0000 7fde77ed4640 1 -- 172.21.15.89:0/560418252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde70105920 con 0x7fde70105f40 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.534+0000 7fde66ffd640 1 -- 172.21.15.89:0/560418252 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fde680051d0 con 0x7fde70105f40 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.535+0000 7fde66ffd640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde5c0a32c0 0x7fde5c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.535+0000 7fde75c70640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde5c0a32c0 0x7fde5c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.535+0000 7fde66ffd640 1 -- 172.21.15.89:0/560418252 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fde680cfb60 con 0x7fde70105f40 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.535+0000 7fde75c70640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde5c0a32c0 0x7fde5c0a5780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009230 tx=0x7fde6c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:34.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.539+0000 7fde66ffd640 1 -- 172.21.15.89:0/560418252 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fde68014030 con 0x7fde70105f40 2024-09-15T23:22:34.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:34.755+0000 7fde77ed4640 1 -- 172.21.15.89:0/560418252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fde701aba30 con 0x7fde70105f40 2024-09-15T23:22:35.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.102+0000 7fde66ffd640 1 -- 172.21.15.89:0/560418252 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7fde6809c640 con 0x7fde70105f40 2024-09-15T23:22:35.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 -- 172.21.15.89:0/560418252 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde5c0a32c0 msgr2=0x7fde5c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:35.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde5c0a32c0 0x7fde5c0a5780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009230 tx=0x7fde6c0058d0 comp rx=0 tx=0).stop 2024-09-15T23:22:35.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 -- 172.21.15.89:0/560418252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde70105f40 msgr2=0x7fde7019e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:35.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde70105f40 0x7fde7019e950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fde6800cad0 tx=0x7fde6800cf90 comp rx=0 tx=0).stop 2024-09-15T23:22:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 -- 172.21.15.89:0/560418252 shutdown_connections 2024-09-15T23:22:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde5c0a32c0 0x7fde5c0a5780 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde70105f40 0x7fde7019e950 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde70101450 0x7fde701a4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 --2- 172.21.15.89:0/560418252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde70100b30 0x7fde701a4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 -- 172.21.15.89:0/560418252 >> 172.21.15.89:0/560418252 conn(0x7fde700fc7a0 msgr2=0x7fde7010f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 -- 172.21.15.89:0/560418252 shutdown_connections 2024-09-15T23:22:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:35.105+0000 7fde77ed4640 1 -- 172.21.15.89:0/560418252 wait complete. 2024-09-15T23:22:35.440 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:35 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] up:active 2024-09-15T23:22:35.441 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:35 smithi089 ceph-mon[30933]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-15T23:22:35.441 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:35 smithi089 ceph-mon[30933]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-15T23:22:35.441 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:35 smithi089 ceph-mon[30933]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-09-15T23:22:35.441 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:35 smithi089 ceph-mon[30933]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-15T23:22:35.441 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:35 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/560418252' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-15T23:22:35.441 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:35 smithi089 ceph-mon[30933]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-15T23:22:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:35 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] up:active 2024-09-15T23:22:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:35 smithi120 ceph-mon[35250]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-15T23:22:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:35 smithi120 ceph-mon[35250]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-15T23:22:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:35 smithi120 ceph-mon[35250]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-09-15T23:22:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:35 smithi120 ceph-mon[35250]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-15T23:22:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:35 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/560418252' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-15T23:22:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:35 smithi120 ceph-mon[35250]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-15T23:22:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:35 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] up:active 2024-09-15T23:22:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:35 smithi174 ceph-mon[35586]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-15T23:22:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:35 smithi174 ceph-mon[35586]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-09-15T23:22:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:35 smithi174 ceph-mon[35586]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-09-15T23:22:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:35 smithi174 ceph-mon[35586]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-09-15T23:22:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:35 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/560418252' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-15T23:22:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:35 smithi174 ceph-mon[35586]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-09-15T23:22:35.846 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-09-15T23:22:36.094 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:36.363 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:36 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/560418252' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-15T23:22:36.363 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:36 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] up:active 2024-09-15T23:22:36.363 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:36 smithi089 ceph-mon[30933]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-15T23:22:36.363 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:36 smithi089 ceph-mon[30933]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-09-15T23:22:36.363 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:36 smithi089 ceph-mon[30933]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-09-15T23:22:36.364 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:36 smithi089 ceph-mon[30933]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-09-15T23:22:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:36 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/560418252' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-15T23:22:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:36 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] up:active 2024-09-15T23:22:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:36 smithi120 ceph-mon[35250]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-15T23:22:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:36 smithi120 ceph-mon[35250]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-09-15T23:22:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:36 smithi120 ceph-mon[35250]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-09-15T23:22:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:36 smithi120 ceph-mon[35250]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-09-15T23:22:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:36 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/560418252' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-15T23:22:36.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:36 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] up:active 2024-09-15T23:22:36.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:36 smithi174 ceph-mon[35586]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-09-15T23:22:36.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:36 smithi174 ceph-mon[35586]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-09-15T23:22:36.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:36 smithi174 ceph-mon[35586]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-09-15T23:22:36.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:36 smithi174 ceph-mon[35586]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-09-15T23:22:36.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.585+0000 7fb2825a3640 1 -- 172.21.15.89:0/622525803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb27c1082a0 msgr2=0x7fb27c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.585+0000 7fb2825a3640 1 --2- 172.21.15.89:0/622525803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb27c1082a0 0x7fb27c10ec40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb274009cd0 tx=0x7fb27402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.585+0000 7fb2825a3640 1 -- 172.21.15.89:0/622525803 shutdown_connections 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.585+0000 7fb2825a3640 1 --2- 172.21.15.89:0/622525803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb27c1082a0 0x7fb27c10ec40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.585+0000 7fb2825a3640 1 --2- 172.21.15.89:0/622525803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c1036e0 0x7fb27c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.585+0000 7fb2825a3640 1 --2- 172.21.15.89:0/622525803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb27c102d30 0x7fb27c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.585+0000 7fb2825a3640 1 -- 172.21.15.89:0/622525803 >> 172.21.15.89:0/622525803 conn(0x7fb27c0fc7d0 msgr2=0x7fb27c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.585+0000 7fb2825a3640 1 -- 172.21.15.89:0/622525803 shutdown_connections 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.586+0000 7fb2825a3640 1 -- 172.21.15.89:0/622525803 wait complete. 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.586+0000 7fb2825a3640 1 Processor -- start 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.586+0000 7fb2825a3640 1 -- start start 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2825a3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb27c102d30 0x7fb27c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2825a3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb27c1036e0 0x7fb27c198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:36.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2825a3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c1082a0 0x7fb27c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2825a3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1120a0 con 0x7fb27c1036e0 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2825a3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c111f20 con 0x7fb27c102d30 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2825a3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c112220 con 0x7fb27c1082a0 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb27c1036e0 0x7fb27c198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb27c1036e0 0x7fb27c198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36616/0 (socket says 172.21.15.89:36616) 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27b7fe640 1 -- 172.21.15.89:0/418612137 learned_addr learned my addr 172.21.15.89:0/418612137 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb280b40640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c1082a0 0x7fb27c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27bfff640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb27c102d30 0x7fb27c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27b7fe640 1 -- 172.21.15.89:0/418612137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c1082a0 msgr2=0x7fb27c19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27b7fe640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c1082a0 0x7fb27c19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27b7fe640 1 -- 172.21.15.89:0/418612137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb27c102d30 msgr2=0x7fb27c198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:36.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27b7fe640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb27c102d30 0x7fb27c198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27b7fe640 1 -- 172.21.15.89:0/418612137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c19d500 con 0x7fb27c1036e0 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27bfff640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb27c102d30 0x7fb27c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb27b7fe640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb27c1036e0 0x7fb27c198a90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb27000d750 tx=0x7fb27000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2797fa640 1 -- 172.21.15.89:0/418612137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270017070 con 0x7fb27c1036e0 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2797fa640 1 -- 172.21.15.89:0/418612137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb270005500 con 0x7fb27c1036e0 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2797fa640 1 -- 172.21.15.89:0/418612137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270004e00 con 0x7fb27c1036e0 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2825a3640 1 -- 172.21.15.89:0/418612137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27c19d7f0 con 0x7fb27c1036e0 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.587+0000 7fb2825a3640 1 -- 172.21.15.89:0/418612137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27c1a50d0 con 0x7fb27c1036e0 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.588+0000 7fb2825a3640 1 -- 172.21.15.89:0/418612137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb27c103110 con 0x7fb27c1036e0 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.589+0000 7fb2797fa640 1 -- 172.21.15.89:0/418612137 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fb270004fa0 con 0x7fb27c1036e0 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.590+0000 7fb2797fa640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb2640a32c0 0x7fb2640a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:36.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.590+0000 7fb27bfff640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb2640a32c0 0x7fb2640a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:36.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.591+0000 7fb2797fa640 1 -- 172.21.15.89:0/418612137 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fb2700cf420 con 0x7fb27c1036e0 2024-09-15T23:22:36.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.591+0000 7fb27bfff640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb2640a32c0 0x7fb2640a5780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0058c0 tx=0x7fb26c005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:36.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.595+0000 7fb2797fa640 1 -- 172.21.15.89:0/418612137 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb27009bf00 con 0x7fb27c1036e0 2024-09-15T23:22:36.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:36.797+0000 7fb2825a3640 1 -- 172.21.15.89:0/418612137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fb27c10a2e0 con 0x7fb27c1036e0 2024-09-15T23:22:37.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.126+0000 7fb2797fa640 1 -- 172.21.15.89:0/418612137 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v27) ==== 115+0+0 (secure 0 0 0) 0x7fb27009b800 con 0x7fb27c1036e0 2024-09-15T23:22:37.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.128+0000 7fb2825a3640 1 -- 172.21.15.89:0/418612137 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb2640a32c0 msgr2=0x7fb2640a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:37.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.128+0000 7fb2825a3640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb2640a32c0 0x7fb2640a5780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0058c0 tx=0x7fb26c005ec0 comp rx=0 tx=0).stop 2024-09-15T23:22:37.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.128+0000 7fb2825a3640 1 -- 172.21.15.89:0/418612137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb27c1036e0 msgr2=0x7fb27c198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:37.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.128+0000 7fb2825a3640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb27c1036e0 0x7fb27c198a90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb27000d750 tx=0x7fb27000dc10 comp rx=0 tx=0).stop 2024-09-15T23:22:37.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.129+0000 7fb2825a3640 1 -- 172.21.15.89:0/418612137 shutdown_connections 2024-09-15T23:22:37.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.129+0000 7fb2825a3640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb2640a32c0 0x7fb2640a5780 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:37.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.129+0000 7fb2825a3640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c1082a0 0x7fb27c19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:37.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.129+0000 7fb2825a3640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb27c1036e0 0x7fb27c198a90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:37.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.129+0000 7fb2825a3640 1 --2- 172.21.15.89:0/418612137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb27c102d30 0x7fb27c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:37.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.129+0000 7fb2825a3640 1 -- 172.21.15.89:0/418612137 >> 172.21.15.89:0/418612137 conn(0x7fb27c0fc7d0 msgr2=0x7fb27c106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:37.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.129+0000 7fb2825a3640 1 -- 172.21.15.89:0/418612137 shutdown_connections 2024-09-15T23:22:37.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:37.129+0000 7fb2825a3640 1 -- 172.21.15.89:0/418612137 wait complete. 2024-09-15T23:22:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:37 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] up:active 2024-09-15T23:22:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:37 smithi120 ceph-mon[35250]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-15T23:22:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:37 smithi120 ceph-mon[35250]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-15T23:22:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:37 smithi120 ceph-mon[35250]: Cluster is now healthy 2024-09-15T23:22:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:37 smithi120 ceph-mon[35250]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-09-15T23:22:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:37 smithi120 ceph-mon[35250]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-15T23:22:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:37 smithi120 ceph-mon[35250]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-09-15T23:22:37.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:37 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/418612137' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-15T23:22:37.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:37 smithi120 ceph-mon[35250]: pgmap v304: 129 pgs: 129 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 5 op/s 2024-09-15T23:22:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:37 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] up:active 2024-09-15T23:22:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:37 smithi174 ceph-mon[35586]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-15T23:22:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:37 smithi174 ceph-mon[35586]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-15T23:22:37.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:37 smithi174 ceph-mon[35586]: Cluster is now healthy 2024-09-15T23:22:37.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:37 smithi174 ceph-mon[35586]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-09-15T23:22:37.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:37 smithi174 ceph-mon[35586]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-15T23:22:37.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:37 smithi174 ceph-mon[35586]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-09-15T23:22:37.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:37 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/418612137' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-15T23:22:37.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:37 smithi174 ceph-mon[35586]: pgmap v304: 129 pgs: 129 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 5 op/s 2024-09-15T23:22:37.502 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:37 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] up:active 2024-09-15T23:22:37.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:37 smithi089 ceph-mon[30933]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-09-15T23:22:37.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:37 smithi089 ceph-mon[30933]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-15T23:22:37.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:37 smithi089 ceph-mon[30933]: Cluster is now healthy 2024-09-15T23:22:37.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:37 smithi089 ceph-mon[30933]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-09-15T23:22:37.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:37 smithi089 ceph-mon[30933]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-09-15T23:22:37.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:37 smithi089 ceph-mon[30933]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-09-15T23:22:37.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:37 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/418612137' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-15T23:22:37.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:37 smithi089 ceph-mon[30933]: pgmap v304: 129 pgs: 129 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 5 op/s 2024-09-15T23:22:37.960 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-09-15T23:22:37.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-09-15T23:22:38.209 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:38 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/418612137' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-15T23:22:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:38 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] up:active 2024-09-15T23:22:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:38 smithi089 ceph-mon[30933]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-15T23:22:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:38 smithi089 ceph-mon[30933]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T23:22:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:38 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/418612137' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-15T23:22:38.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:38 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] up:active 2024-09-15T23:22:38.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:38 smithi120 ceph-mon[35250]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-15T23:22:38.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:38 smithi120 ceph-mon[35250]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T23:22:38.474 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:22:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b[65175]: 2024-09-15T23:22:38.055+0000 7f9e6c009640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-09-15T23:22:38.474 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:22:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b[65175]: 2024-09-15T23:22:38.055+0000 7f9e6c009640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-09-15T23:22:38.474 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:22:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b[65175]: 2024-09-15T23:22:38.055+0000 7f9e6c009640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-09-15T23:22:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:38 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/418612137' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-15T23:22:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:38 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] up:active 2024-09-15T23:22:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:38 smithi174 ceph-mon[35586]: fsmap cephfs:5 7 up:standby 5 up:active 2024-09-15T23:22:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:38 smithi174 ceph-mon[35586]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T23:22:38.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 -- 172.21.15.89:0/429340882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d24102790 msgr2=0x7f0d241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 --2- 172.21.15.89:0/429340882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d24102790 0x7f0d241067e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009900 tx=0x7f0d1002a410 comp rx=0 tx=0).stop 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 -- 172.21.15.89:0/429340882 shutdown_connections 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 --2- 172.21.15.89:0/429340882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d24106f10 0x7f0d2410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 --2- 172.21.15.89:0/429340882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d24102790 0x7f0d241067e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 --2- 172.21.15.89:0/429340882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d24101de0 0x7f0d241021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 -- 172.21.15.89:0/429340882 >> 172.21.15.89:0/429340882 conn(0x7f0d240fd8c0 msgr2=0x7f0d240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 -- 172.21.15.89:0/429340882 shutdown_connections 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 -- 172.21.15.89:0/429340882 wait complete. 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 Processor -- start 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.710+0000 7f0d2afd8640 1 -- start start 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d2afd8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d24101de0 0x7f0d2419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:38.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d2afd8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d24102790 0x7f0d2419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d2afd8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d24106f10 0x7f0d2419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d2afd8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d24111f90 con 0x7f0d24102790 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d2afd8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0d24111e10 con 0x7f0d24101de0 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d2afd8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0d24112110 con 0x7f0d24106f10 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d23fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d24102790 0x7f0d2419abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d23fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d24102790 0x7f0d2419abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36626/0 (socket says 172.21.15.89:36626) 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d23fff640 1 -- 172.21.15.89:0/4121142501 learned_addr learned my addr 172.21.15.89:0/4121142501 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d29575640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d24106f10 0x7f0d2419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d28d74640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d24101de0 0x7f0d2419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d23fff640 1 -- 172.21.15.89:0/4121142501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d24106f10 msgr2=0x7f0d2419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d23fff640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d24106f10 0x7f0d2419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d23fff640 1 -- 172.21.15.89:0/4121142501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d24101de0 msgr2=0x7f0d2419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:38.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d23fff640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d24101de0 0x7f0d2419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d23fff640 1 -- 172.21.15.89:0/4121142501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2419f660 con 0x7f0d24102790 2024-09-15T23:22:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d28d74640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d24101de0 0x7f0d2419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d23fff640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d24102790 0x7f0d2419abf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0d10005ab0 tx=0x7f0d100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d21ffb640 1 -- 172.21.15.89:0/4121142501 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1003f030 con 0x7f0d24102790 2024-09-15T23:22:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d21ffb640 1 -- 172.21.15.89:0/4121142501 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d10003ce0 con 0x7f0d24102790 2024-09-15T23:22:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d2afd8640 1 -- 172.21.15.89:0/4121142501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d2419f8f0 con 0x7f0d24102790 2024-09-15T23:22:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.711+0000 7f0d21ffb640 1 -- 172.21.15.89:0/4121142501 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10035070 con 0x7f0d24102790 2024-09-15T23:22:38.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.712+0000 7f0d2afd8640 1 -- 172.21.15.89:0/4121142501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d241a7230 con 0x7f0d24102790 2024-09-15T23:22:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.713+0000 7f0d2afd8640 1 -- 172.21.15.89:0/4121142501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d241032c0 con 0x7f0d24102790 2024-09-15T23:22:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.714+0000 7f0d21ffb640 1 -- 172.21.15.89:0/4121142501 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f0d10042030 con 0x7f0d24102790 2024-09-15T23:22:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.715+0000 7f0d21ffb640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d140a3240 0x7f0d140a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.715+0000 7f0d28d74640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d140a3240 0x7f0d140a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.715+0000 7f0d21ffb640 1 -- 172.21.15.89:0/4121142501 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f0d100f04f0 con 0x7f0d24102790 2024-09-15T23:22:38.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.715+0000 7f0d28d74640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d140a3240 0x7f0d140a5700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0d180058c0 tx=0x7f0d18005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:38.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.719+0000 7f0d21ffb640 1 -- 172.21.15.89:0/4121142501 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0d100bcfd0 con 0x7f0d24102790 2024-09-15T23:22:38.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.907+0000 7f0d2afd8640 1 -- 172.21.15.89:0/4121142501 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0d2410de00 con 0x7f0d140a3240 2024-09-15T23:22:38.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.932+0000 7f0d21ffb640 1 -- 172.21.15.89:0/4121142501 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0d2410de00 con 0x7f0d140a3240 2024-09-15T23:22:38.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 -- 172.21.15.89:0/4121142501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d140a3240 msgr2=0x7f0d140a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:38.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d140a3240 0x7f0d140a5700 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0d180058c0 tx=0x7f0d18005ec0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 -- 172.21.15.89:0/4121142501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d24102790 msgr2=0x7f0d2419abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:38.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d24102790 0x7f0d2419abf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0d10005ab0 tx=0x7f0d100042f0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 -- 172.21.15.89:0/4121142501 shutdown_connections 2024-09-15T23:22:38.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d140a3240 0x7f0d140a5700 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d24106f10 0x7f0d2419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d24102790 0x7f0d2419abf0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 --2- 172.21.15.89:0/4121142501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d24101de0 0x7f0d2419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:38.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 -- 172.21.15.89:0/4121142501 >> 172.21.15.89:0/4121142501 conn(0x7f0d240fd8c0 msgr2=0x7f0d240fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:38.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 -- 172.21.15.89:0/4121142501 shutdown_connections 2024-09-15T23:22:38.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:38.934+0000 7f0d2afd8640 1 -- 172.21.15.89:0/4121142501 wait complete. 2024-09-15T23:22:39.299 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:39 smithi089 ceph-mon[30933]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:39.299 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:39 smithi089 ceph-mon[30933]: pgmap v305: 129 pgs: 129 active+clean; 586 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-15T23:22:39.300 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:22:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:22:38.912+0000 7f8abb86a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:22:39.300 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:22:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:22:38.912+0000 7f8abb86a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:22:39.300 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:22:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:22:38.912+0000 7f8abb86a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:22:39.300 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:22:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:22:38.912+0000 7f8abb86a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:22:39.300 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:22:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:22:38.912+0000 7f8abb86a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:22:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:39 smithi120 ceph-mon[35250]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:39 smithi120 ceph-mon[35250]: pgmap v305: 129 pgs: 129 active+clean; 586 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-15T23:22:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:39 smithi174 ceph-mon[35586]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:39 smithi174 ceph-mon[35586]: pgmap v305: 129 pgs: 129 active+clean; 586 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 3.2 KiB/s wr, 11 op/s 2024-09-15T23:22:39.646 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-15T23:22:39.892 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:40.272 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:40 smithi089 ceph-mon[30933]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-15T23:22:40.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.356+0000 7f820ed7c640 1 -- 172.21.15.89:0/57100988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8208102790 msgr2=0x7f82081067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:40.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.356+0000 7f820ed7c640 1 --2- 172.21.15.89:0/57100988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8208102790 0x7f82081067e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009ca0 tx=0x7f81f802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.356+0000 7f820ed7c640 1 -- 172.21.15.89:0/57100988 shutdown_connections 2024-09-15T23:22:40.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.356+0000 7f820ed7c640 1 --2- 172.21.15.89:0/57100988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8208106f10 0x7f820810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.356+0000 7f820ed7c640 1 --2- 172.21.15.89:0/57100988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8208102790 0x7f82081067e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.356+0000 7f820ed7c640 1 --2- 172.21.15.89:0/57100988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8208101de0 0x7f82081021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.356+0000 7f820ed7c640 1 -- 172.21.15.89:0/57100988 >> 172.21.15.89:0/57100988 conn(0x7f82080fd8c0 msgr2=0x7f82080ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.356+0000 7f820ed7c640 1 -- 172.21.15.89:0/57100988 shutdown_connections 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.356+0000 7f820ed7c640 1 -- 172.21.15.89:0/57100988 wait complete. 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f820ed7c640 1 Processor -- start 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f820ed7c640 1 -- start start 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f820ed7c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8208101de0 0x7f82081984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f820ed7c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8208102790 0x7f8208198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f820ed7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8208106f10 0x7f820819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f820ed7c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8208111f70 con 0x7f8208102790 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f820ed7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8208111df0 con 0x7f8208106f10 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f820ed7c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82081120f0 con 0x7f8208101de0 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f8207fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8208102790 0x7f8208198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f8207fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8208102790 0x7f8208198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36666/0 (socket says 172.21.15.89:36666) 2024-09-15T23:22:40.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f8207fff640 1 -- 172.21.15.89:0/1888371515 learned_addr learned my addr 172.21.15.89:0/1888371515 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f820d319640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8208106f10 0x7f820819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f820cb18640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8208101de0 0x7f82081984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f8207fff640 1 -- 172.21.15.89:0/1888371515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8208101de0 msgr2=0x7f82081984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f8207fff640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8208101de0 0x7f82081984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f8207fff640 1 -- 172.21.15.89:0/1888371515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8208106f10 msgr2=0x7f820819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f8207fff640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8208106f10 0x7f820819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f8207fff640 1 -- 172.21.15.89:0/1888371515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820819d480 con 0x7f8208102790 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.357+0000 7f8207fff640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8208102790 0x7f8208198a10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f81f802ade0 tx=0x7f81f80319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.358+0000 7f8205ffb640 1 -- 172.21.15.89:0/1888371515 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f802fa20 con 0x7f8208102790 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.358+0000 7f8205ffb640 1 -- 172.21.15.89:0/1888371515 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81f802fbc0 con 0x7f8208102790 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.358+0000 7f820ed7c640 1 -- 172.21.15.89:0/1888371515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820819d710 con 0x7f8208102790 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.358+0000 7f8205ffb640 1 -- 172.21.15.89:0/1888371515 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f80466e0 con 0x7f8208102790 2024-09-15T23:22:40.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.358+0000 7f820ed7c640 1 -- 172.21.15.89:0/1888371515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82081a5050 con 0x7f8208102790 2024-09-15T23:22:40.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.359+0000 7f820ed7c640 1 -- 172.21.15.89:0/1888371515 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82081021c0 con 0x7f8208102790 2024-09-15T23:22:40.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.360+0000 7f8205ffb640 1 -- 172.21.15.89:0/1888371515 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f81f8046940 con 0x7f8208102790 2024-09-15T23:22:40.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.360+0000 7f8205ffb640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81f40a32d0 0x7f81f40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:40.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.361+0000 7f820cb18640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81f40a32d0 0x7f81f40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:40.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.361+0000 7f8205ffb640 1 -- 172.21.15.89:0/1888371515 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f81f80f10b0 con 0x7f8208102790 2024-09-15T23:22:40.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.361+0000 7f820cb18640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81f40a32d0 0x7f81f40a5790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8200009aa0 tx=0x7f8200005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:40.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.365+0000 7f8205ffb640 1 -- 172.21.15.89:0/1888371515 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f81f80bdb40 con 0x7f8208102790 2024-09-15T23:22:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:40 smithi120 ceph-mon[35250]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-15T23:22:40.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:40 smithi174 ceph-mon[35586]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-15T23:22:40.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.561+0000 7f820ed7c640 1 -- 172.21.15.89:0/1888371515 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f8208062630 con 0x7f81f40a32d0 2024-09-15T23:22:40.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.585+0000 7f8205ffb640 1 -- 172.21.15.89:0/1888371515 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8208062630 con 0x7f81f40a32d0 2024-09-15T23:22:40.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 -- 172.21.15.89:0/1888371515 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81f40a32d0 msgr2=0x7f81f40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:40.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81f40a32d0 0x7f81f40a5790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8200009aa0 tx=0x7f8200005f90 comp rx=0 tx=0).stop 2024-09-15T23:22:40.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 -- 172.21.15.89:0/1888371515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8208102790 msgr2=0x7f8208198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:40.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8208102790 0x7f8208198a10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f81f802ade0 tx=0x7f81f80319f0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 -- 172.21.15.89:0/1888371515 shutdown_connections 2024-09-15T23:22:40.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81f40a32d0 0x7f81f40a5790 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8208106f10 0x7f820819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8208102790 0x7f8208198a10 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 --2- 172.21.15.89:0/1888371515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8208101de0 0x7f82081984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:40.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 -- 172.21.15.89:0/1888371515 >> 172.21.15.89:0/1888371515 conn(0x7f82080fd8c0 msgr2=0x7f82080fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:40.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 -- 172.21.15.89:0/1888371515 shutdown_connections 2024-09-15T23:22:40.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:40.587+0000 7f820ed7c640 1 -- 172.21.15.89:0/1888371515 wait complete. 2024-09-15T23:22:41.295 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:41 smithi089 ceph-mon[30933]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:41.295 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:41 smithi089 ceph-mon[30933]: pgmap v306: 129 pgs: 129 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 14 op/s 2024-09-15T23:22:41.323 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-15T23:22:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:41 smithi120 ceph-mon[35250]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:41 smithi120 ceph-mon[35250]: pgmap v306: 129 pgs: 129 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 14 op/s 2024-09-15T23:22:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:41 smithi174 ceph-mon[35586]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:41 smithi174 ceph-mon[35586]: pgmap v306: 129 pgs: 129 active+clean; 587 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3.5 KiB/s wr, 14 op/s 2024-09-15T23:22:41.567 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:42.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.040+0000 7f459ae78640 1 -- 172.21.15.89:0/1798790821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4594106f10 msgr2=0x7f459410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.040+0000 7f459ae78640 1 --2- 172.21.15.89:0/1798790821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4594106f10 0x7f459410ea90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f458c009ca0 tx=0x7f458c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.040+0000 7f459ae78640 1 -- 172.21.15.89:0/1798790821 shutdown_connections 2024-09-15T23:22:42.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.040+0000 7f459ae78640 1 --2- 172.21.15.89:0/1798790821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4594106f10 0x7f459410ea90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.040+0000 7f459ae78640 1 --2- 172.21.15.89:0/1798790821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4594102790 0x7f45941067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.040+0000 7f459ae78640 1 --2- 172.21.15.89:0/1798790821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594101de0 0x7f45941021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.040+0000 7f459ae78640 1 -- 172.21.15.89:0/1798790821 >> 172.21.15.89:0/1798790821 conn(0x7f45940fd8c0 msgr2=0x7f45940ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:42.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.040+0000 7f459ae78640 1 -- 172.21.15.89:0/1798790821 shutdown_connections 2024-09-15T23:22:42.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.040+0000 7f459ae78640 1 -- 172.21.15.89:0/1798790821 wait complete. 2024-09-15T23:22:42.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f459ae78640 1 Processor -- start 2024-09-15T23:22:42.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f459ae78640 1 -- start start 2024-09-15T23:22:42.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f459ae78640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594101de0 0x7f45941983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f459ae78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4594102790 0x7f4594198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f459ae78640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4594106f10 0x7f459419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f459ae78640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4594111f70 con 0x7f4594102790 2024-09-15T23:22:42.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f459ae78640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4594111df0 con 0x7f4594106f10 2024-09-15T23:22:42.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f459ae78640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45941120f0 con 0x7f4594101de0 2024-09-15T23:22:42.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f4593fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4594102790 0x7f4594198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f4593fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4594102790 0x7f4594198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36686/0 (socket says 172.21.15.89:36686) 2024-09-15T23:22:42.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f4593fff640 1 -- 172.21.15.89:0/1129787232 learned_addr learned my addr 172.21.15.89:0/1129787232 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:42.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f4599415640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4594106f10 0x7f459419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f4598c14640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594101de0 0x7f45941983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.041+0000 7f4593fff640 1 -- 172.21.15.89:0/1129787232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594101de0 msgr2=0x7f45941983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.042+0000 7f4593fff640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594101de0 0x7f45941983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.042+0000 7f4593fff640 1 -- 172.21.15.89:0/1129787232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4594106f10 msgr2=0x7f459419ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.042+0000 7f4593fff640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4594106f10 0x7f459419ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.042+0000 7f4593fff640 1 -- 172.21.15.89:0/1129787232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459419d3a0 con 0x7f4594102790 2024-09-15T23:22:42.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.042+0000 7f4593fff640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4594102790 0x7f4594198930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f458000d950 tx=0x7f458000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:42.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.042+0000 7f4591ffb640 1 -- 172.21.15.89:0/1129787232 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580017030 con 0x7f4594102790 2024-09-15T23:22:42.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.042+0000 7f4591ffb640 1 -- 172.21.15.89:0/1129787232 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4580004020 con 0x7f4594102790 2024-09-15T23:22:42.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.042+0000 7f4591ffb640 1 -- 172.21.15.89:0/1129787232 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45800047a0 con 0x7f4594102790 2024-09-15T23:22:42.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.042+0000 7f459ae78640 1 -- 172.21.15.89:0/1129787232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459419d690 con 0x7f4594102790 2024-09-15T23:22:42.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.042+0000 7f459ae78640 1 -- 172.21.15.89:0/1129787232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45941a4f70 con 0x7f4594102790 2024-09-15T23:22:42.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.043+0000 7f459ae78640 1 -- 172.21.15.89:0/1129787232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45941021c0 con 0x7f4594102790 2024-09-15T23:22:42.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.044+0000 7f4591ffb640 1 -- 172.21.15.89:0/1129787232 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f45800041d0 con 0x7f4594102790 2024-09-15T23:22:42.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.045+0000 7f4591ffb640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45840a3350 0x7f45840a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.045+0000 7f4591ffb640 1 -- 172.21.15.89:0/1129787232 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f45800cf5e0 con 0x7f4594102790 2024-09-15T23:22:42.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.045+0000 7f4598c14640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45840a3350 0x7f45840a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.045+0000 7f4598c14640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45840a3350 0x7f45840a5810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4588005fd0 tx=0x7f4588005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:42.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.049+0000 7f4591ffb640 1 -- 172.21.15.89:0/1129787232 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f458009bf60 con 0x7f4594102790 2024-09-15T23:22:42.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.241+0000 7f459ae78640 1 -- 172.21.15.89:0/1129787232 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f4594062630 con 0x7f45840a3350 2024-09-15T23:22:42.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.247+0000 7f4591ffb640 1 -- 172.21.15.89:0/1129787232 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4594062630 con 0x7f45840a3350 2024-09-15T23:22:42.250 INFO:teuthology.orchestra.run.smithi089.stdout:/volumes/qa/sv_0/b0a16533-89cb-4d8a-ba16-61ecf19e20d9 2024-09-15T23:22:42.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 -- 172.21.15.89:0/1129787232 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45840a3350 msgr2=0x7f45840a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45840a3350 0x7f45840a5810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4588005fd0 tx=0x7f4588005850 comp rx=0 tx=0).stop 2024-09-15T23:22:42.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 -- 172.21.15.89:0/1129787232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4594102790 msgr2=0x7f4594198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4594102790 0x7f4594198930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f458000d950 tx=0x7f458000de10 comp rx=0 tx=0).stop 2024-09-15T23:22:42.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 -- 172.21.15.89:0/1129787232 shutdown_connections 2024-09-15T23:22:42.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45840a3350 0x7f45840a5810 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4594106f10 0x7f459419ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4594102790 0x7f4594198930 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 --2- 172.21.15.89:0/1129787232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594101de0 0x7f45941983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 -- 172.21.15.89:0/1129787232 >> 172.21.15.89:0/1129787232 conn(0x7f45940fd8c0 msgr2=0x7f45940feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:42.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 -- 172.21.15.89:0/1129787232 shutdown_connections 2024-09-15T23:22:42.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:42.249+0000 7f459ae78640 1 -- 172.21.15.89:0/1129787232 wait complete. 2024-09-15T23:22:42.958 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-15T23:22:43.203 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:43.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.695+0000 7fbbd8b26640 1 -- 172.21.15.89:0/3514874933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd4106f10 msgr2=0x7fbbd410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:43.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.695+0000 7fbbd8b26640 1 --2- 172.21.15.89:0/3514874933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd4106f10 0x7fbbd410ea90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc009ca0 tx=0x7fbbcc02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:22:43.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.695+0000 7fbbd8b26640 1 -- 172.21.15.89:0/3514874933 shutdown_connections 2024-09-15T23:22:43.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.695+0000 7fbbd8b26640 1 --2- 172.21.15.89:0/3514874933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd4106f10 0x7fbbd410ea90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:43.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.695+0000 7fbbd8b26640 1 --2- 172.21.15.89:0/3514874933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd4102790 0x7fbbd41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.695+0000 7fbbd8b26640 1 --2- 172.21.15.89:0/3514874933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4101de0 0x7fbbd41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.695+0000 7fbbd8b26640 1 -- 172.21.15.89:0/3514874933 >> 172.21.15.89:0/3514874933 conn(0x7fbbd40fd8c0 msgr2=0x7fbbd40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.695+0000 7fbbd8b26640 1 -- 172.21.15.89:0/3514874933 shutdown_connections 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.695+0000 7fbbd8b26640 1 -- 172.21.15.89:0/3514874933 wait complete. 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd8b26640 1 Processor -- start 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd8b26640 1 -- start start 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd8b26640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4101de0 0x7fbbd4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd8b26640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd4102790 0x7fbbd41989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd8b26640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd4106f10 0x7fbbd419cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd8b26640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd4111f90 con 0x7fbbd4102790 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd8b26640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd4111e10 con 0x7fbbd4106f10 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd8b26640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd4112110 con 0x7fbbd4101de0 2024-09-15T23:22:43.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd4102790 0x7fbbd41989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd4102790 0x7fbbd41989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36716/0 (socket says 172.21.15.89:36716) 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd1d9b640 1 -- 172.21.15.89:0/4108248501 learned_addr learned my addr 172.21.15.89:0/4108248501 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd259c640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4101de0 0x7fbbd4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.696+0000 7fbbd2d9d640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd4106f10 0x7fbbd419cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbd1d9b640 1 -- 172.21.15.89:0/4108248501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4101de0 msgr2=0x7fbbd4198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbd1d9b640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4101de0 0x7fbbd4198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbd1d9b640 1 -- 172.21.15.89:0/4108248501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd4106f10 msgr2=0x7fbbd419cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbd1d9b640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd4106f10 0x7fbbd419cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbd1d9b640 1 -- 172.21.15.89:0/4108248501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd419d410 con 0x7fbbd4102790 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbd2d9d640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd4106f10 0x7fbbd419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbd1d9b640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd4102790 0x7fbbd41989a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbbc800c890 tx=0x7fbbc800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbc37fe640 1 -- 172.21.15.89:0/4108248501 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8004220 con 0x7fbbd4102790 2024-09-15T23:22:43.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbd8b26640 1 -- 172.21.15.89:0/4108248501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd419d700 con 0x7fbbd4102790 2024-09-15T23:22:43.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbc37fe640 1 -- 172.21.15.89:0/4108248501 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbc80043c0 con 0x7fbbd4102790 2024-09-15T23:22:43.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbc37fe640 1 -- 172.21.15.89:0/4108248501 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8004a00 con 0x7fbbd4102790 2024-09-15T23:22:43.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.697+0000 7fbbd8b26640 1 -- 172.21.15.89:0/4108248501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbd41a4fe0 con 0x7fbbd4102790 2024-09-15T23:22:43.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.698+0000 7fbbd8b26640 1 -- 172.21.15.89:0/4108248501 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbd41021c0 con 0x7fbbd4102790 2024-09-15T23:22:43.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.699+0000 7fbbc37fe640 1 -- 172.21.15.89:0/4108248501 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fbbc80079b0 con 0x7fbbd4102790 2024-09-15T23:22:43.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.700+0000 7fbbc37fe640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32d0 0x7fbbbc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:43.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.700+0000 7fbbd259c640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32d0 0x7fbbbc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:43.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.700+0000 7fbbc37fe640 1 -- 172.21.15.89:0/4108248501 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fbbc80cf9b0 con 0x7fbbd4102790 2024-09-15T23:22:43.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.700+0000 7fbbd259c640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32d0 0x7fbbbc0a5790 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4009990 tx=0x7fbbc4009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:43.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.704+0000 7fbbc37fe640 1 -- 172.21.15.89:0/4108248501 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbbc8014030 con 0x7fbbd4102790 2024-09-15T23:22:43.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.893+0000 7fbbd8b26640 1 -- 172.21.15.89:0/4108248501 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fbbd4062630 con 0x7fbbbc0a32d0 2024-09-15T23:22:43.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.917+0000 7fbbc37fe640 1 -- 172.21.15.89:0/4108248501 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbbd4062630 con 0x7fbbbc0a32d0 2024-09-15T23:22:43.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 -- 172.21.15.89:0/4108248501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32d0 msgr2=0x7fbbbc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:43.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32d0 0x7fbbbc0a5790 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4009990 tx=0x7fbbc4009110 comp rx=0 tx=0).stop 2024-09-15T23:22:43.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 -- 172.21.15.89:0/4108248501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd4102790 msgr2=0x7fbbd41989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:43.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd4102790 0x7fbbd41989a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbbc800c890 tx=0x7fbbc800cc90 comp rx=0 tx=0).stop 2024-09-15T23:22:43.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 -- 172.21.15.89:0/4108248501 shutdown_connections 2024-09-15T23:22:43.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32d0 0x7fbbbc0a5790 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:43.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd4106f10 0x7fbbd419cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:43.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd4102790 0x7fbbd41989a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:43.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 --2- 172.21.15.89:0/4108248501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd4101de0 0x7fbbd4198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:43.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 -- 172.21.15.89:0/4108248501 >> 172.21.15.89:0/4108248501 conn(0x7fbbd40fd8c0 msgr2=0x7fbbd40fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:43.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 -- 172.21.15.89:0/4108248501 shutdown_connections 2024-09-15T23:22:43.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:43.919+0000 7fbbd8b26640 1 -- 172.21.15.89:0/4108248501 wait complete. 2024-09-15T23:22:44.224 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:44 smithi089 ceph-mon[30933]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:44.224 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:44 smithi089 ceph-mon[30933]: pgmap v307: 129 pgs: 129 active+clean; 607 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-09-15T23:22:44.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:44 smithi120 ceph-mon[35250]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:44.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:44 smithi120 ceph-mon[35250]: pgmap v307: 129 pgs: 129 active+clean; 607 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-09-15T23:22:44.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:44 smithi174 ceph-mon[35586]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:44.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:44 smithi174 ceph-mon[35586]: pgmap v307: 129 pgs: 129 active+clean; 607 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 6.2 KiB/s wr, 20 op/s 2024-09-15T23:22:44.628 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-15T23:22:44.871 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:45.272 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:45 smithi089 ceph-mon[30933]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.356+0000 7f8dc0de8640 1 -- 172.21.15.89:0/3744427347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dbc101de0 msgr2=0x7f8dbc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.356+0000 7f8dc0de8640 1 --2- 172.21.15.89:0/3744427347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dbc101de0 0x7f8dbc1021c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009900 tx=0x7f8dac02a3f0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.356+0000 7f8dc0de8640 1 -- 172.21.15.89:0/3744427347 shutdown_connections 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.356+0000 7f8dc0de8640 1 --2- 172.21.15.89:0/3744427347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dbc106f10 0x7f8dbc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.356+0000 7f8dc0de8640 1 --2- 172.21.15.89:0/3744427347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8dbc102790 0x7f8dbc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.356+0000 7f8dc0de8640 1 --2- 172.21.15.89:0/3744427347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dbc101de0 0x7f8dbc1021c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.356+0000 7f8dc0de8640 1 -- 172.21.15.89:0/3744427347 >> 172.21.15.89:0/3744427347 conn(0x7f8dbc0fd8c0 msgr2=0x7f8dbc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.356+0000 7f8dc0de8640 1 -- 172.21.15.89:0/3744427347 shutdown_connections 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.356+0000 7f8dc0de8640 1 -- 172.21.15.89:0/3744427347 wait complete. 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8dc0de8640 1 Processor -- start 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8dc0de8640 1 -- start start 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8dc0de8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dbc101de0 0x7f8dbc198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8dc0de8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dbc102790 0x7f8dbc198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8dc0de8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8dbc106f10 0x7f8dbc19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8dc0de8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc111f70 con 0x7f8dbc102790 2024-09-15T23:22:45.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8dc0de8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc111df0 con 0x7f8dbc106f10 2024-09-15T23:22:45.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8dc0de8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc1120f0 con 0x7f8dbc101de0 2024-09-15T23:22:45.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8db9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dbc102790 0x7f8dbc198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:45.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8db9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dbc102790 0x7f8dbc198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58358/0 (socket says 172.21.15.89:58358) 2024-09-15T23:22:45.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.357+0000 7f8db9d9b640 1 -- 172.21.15.89:0/1356669907 learned_addr learned my addr 172.21.15.89:0/1356669907 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:45.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8dbad9d640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8dbc106f10 0x7f8dbc19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:45.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8dba59c640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dbc101de0 0x7f8dbc198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:45.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8db9d9b640 1 -- 172.21.15.89:0/1356669907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dbc101de0 msgr2=0x7f8dbc198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:45.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8db9d9b640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dbc101de0 0x7f8dbc198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8db9d9b640 1 -- 172.21.15.89:0/1356669907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8dbc106f10 msgr2=0x7f8dbc19cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8db9d9b640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8dbc106f10 0x7f8dbc19cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8db9d9b640 1 -- 172.21.15.89:0/1356669907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dbc19d2b0 con 0x7f8dbc102790 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8db9d9b640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dbc102790 0x7f8dbc198840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8db000c920 tx=0x7f8db000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8dab7fe640 1 -- 172.21.15.89:0/1356669907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0007c70 con 0x7f8dbc102790 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8dc0de8640 1 -- 172.21.15.89:0/1356669907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dbc19d5a0 con 0x7f8dbc102790 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8dab7fe640 1 -- 172.21.15.89:0/1356669907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db0007e10 con 0x7f8dbc102790 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8dab7fe640 1 -- 172.21.15.89:0/1356669907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0018890 con 0x7f8dbc102790 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.358+0000 7f8dc0de8640 1 -- 172.21.15.89:0/1356669907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dbc1a4e80 con 0x7f8dbc102790 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.359+0000 7f8dc0de8640 1 -- 172.21.15.89:0/1356669907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dbc1021c0 con 0x7f8dbc102790 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.360+0000 7f8dab7fe640 1 -- 172.21.15.89:0/1356669907 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8db0018a70 con 0x7f8dbc102790 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.361+0000 7f8dab7fe640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da40a32d0 0x7f8da40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:45.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.361+0000 7f8dba59c640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da40a32d0 0x7f8da40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:45.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.361+0000 7f8dab7fe640 1 -- 172.21.15.89:0/1356669907 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f8db00cff40 con 0x7f8dbc102790 2024-09-15T23:22:45.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.361+0000 7f8dba59c640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da40a32d0 0x7f8da40a5790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8dac0096c0 tx=0x7f8dac031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:45.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.365+0000 7f8dab7fe640 1 -- 172.21.15.89:0/1356669907 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8db0014030 con 0x7f8dbc102790 2024-09-15T23:22:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:45 smithi120 ceph-mon[35250]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:45.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:45 smithi174 ceph-mon[35586]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:45.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.558+0000 7f8dc0de8640 1 -- 172.21.15.89:0/1356669907 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f8dbc062630 con 0x7f8da40a32d0 2024-09-15T23:22:45.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.564+0000 7f8dab7fe640 1 -- 172.21.15.89:0/1356669907 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8dbc062630 con 0x7f8da40a32d0 2024-09-15T23:22:45.565 INFO:teuthology.orchestra.run.smithi089.stdout:/volumes/qa/sv_1/50a69b32-a4a2-441c-ac79-60380a9adee5 2024-09-15T23:22:45.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 -- 172.21.15.89:0/1356669907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da40a32d0 msgr2=0x7f8da40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:45.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da40a32d0 0x7f8da40a5790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8dac0096c0 tx=0x7f8dac031490 comp rx=0 tx=0).stop 2024-09-15T23:22:45.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 -- 172.21.15.89:0/1356669907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dbc102790 msgr2=0x7f8dbc198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:45.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dbc102790 0x7f8dbc198840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8db000c920 tx=0x7f8db000cde0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 -- 172.21.15.89:0/1356669907 shutdown_connections 2024-09-15T23:22:45.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da40a32d0 0x7f8da40a5790 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8dbc106f10 0x7f8dbc19cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8dbc102790 0x7f8dbc198840 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 --2- 172.21.15.89:0/1356669907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8dbc101de0 0x7f8dbc198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:45.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 -- 172.21.15.89:0/1356669907 >> 172.21.15.89:0/1356669907 conn(0x7f8dbc0fd8c0 msgr2=0x7f8dbc0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:45.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 -- 172.21.15.89:0/1356669907 shutdown_connections 2024-09-15T23:22:45.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:45.566+0000 7f8dc0de8640 1 -- 172.21.15.89:0/1356669907 wait complete. 2024-09-15T23:22:46.237 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:46 smithi089 ceph-mon[30933]: pgmap v308: 129 pgs: 129 active+clean; 607 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 6.3 KiB/s wr, 20 op/s 2024-09-15T23:22:46.265 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:22:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:46 smithi120 ceph-mon[35250]: pgmap v308: 129 pgs: 129 active+clean; 607 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 6.3 KiB/s wr, 20 op/s 2024-09-15T23:22:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:46 smithi174 ceph-mon[35586]: pgmap v308: 129 pgs: 129 active+clean; 607 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 6.3 KiB/s wr, 20 op/s 2024-09-15T23:22:46.509 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:47.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.977+0000 7feaecae7640 1 -- 172.21.15.89:0/68871846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae810c1a0 msgr2=0x7feae810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:47.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.977+0000 7feaecae7640 1 --2- 172.21.15.89:0/68871846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae810c1a0 0x7feae810e610 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feae0009ca0 tx=0x7feae002a950 comp rx=0 tx=0).stop 2024-09-15T23:22:47.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.977+0000 7feaecae7640 1 -- 172.21.15.89:0/68871846 shutdown_connections 2024-09-15T23:22:47.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.977+0000 7feaecae7640 1 --2- 172.21.15.89:0/68871846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae810c1a0 0x7feae810e610 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:47.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.977+0000 7feaecae7640 1 --2- 172.21.15.89:0/68871846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae8103720 0x7feae810bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:47.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.977+0000 7feaecae7640 1 --2- 172.21.15.89:0/68871846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae8102e00 0x7feae81031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:47.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.977+0000 7feaecae7640 1 -- 172.21.15.89:0/68871846 >> 172.21.15.89:0/68871846 conn(0x7feae80fc7a0 msgr2=0x7feae80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.977+0000 7feaecae7640 1 -- 172.21.15.89:0/68871846 shutdown_connections 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.977+0000 7feaecae7640 1 -- 172.21.15.89:0/68871846 wait complete. 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feaecae7640 1 Processor -- start 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feaecae7640 1 -- start start 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feaecae7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae8102e00 0x7feae819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feaecae7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae8103720 0x7feae819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feaecae7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae810c1a0 0x7feae819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feaecae7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feae8112030 con 0x7feae810c1a0 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feaecae7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feae8111eb0 con 0x7feae8103720 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feaecae7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feae81121b0 con 0x7feae8102e00 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae810c1a0 0x7feae819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae810c1a0 0x7feae819f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58374/0 (socket says 172.21.15.89:58374) 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae6d9d640 1 -- 172.21.15.89:0/3931921203 learned_addr learned my addr 172.21.15.89:0/3931921203 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:47.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae5d9b640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae8103720 0x7feae819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae659c640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae8102e00 0x7feae819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae6d9d640 1 -- 172.21.15.89:0/3931921203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae8102e00 msgr2=0x7feae819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae6d9d640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae8102e00 0x7feae819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae6d9d640 1 -- 172.21.15.89:0/3931921203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae8103720 msgr2=0x7feae819ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae6d9d640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae8103720 0x7feae819ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae6d9d640 1 -- 172.21.15.89:0/3931921203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae819f6e0 con 0x7feae810c1a0 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.978+0000 7feae6d9d640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae810c1a0 0x7feae819f000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feae0006010 tx=0x7feae002f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.979+0000 7fead77fe640 1 -- 172.21.15.89:0/3931921203 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0034040 con 0x7feae810c1a0 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.979+0000 7fead77fe640 1 -- 172.21.15.89:0/3931921203 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feae0003e00 con 0x7feae810c1a0 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.979+0000 7feaecae7640 1 -- 172.21.15.89:0/3931921203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae819f970 con 0x7feae810c1a0 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.979+0000 7fead77fe640 1 -- 172.21.15.89:0/3931921203 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae00456e0 con 0x7feae810c1a0 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.979+0000 7feaecae7640 1 -- 172.21.15.89:0/3931921203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae81a72b0 con 0x7feae810c1a0 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.980+0000 7feaecae7640 1 -- 172.21.15.89:0/3931921203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feae806afb0 con 0x7feae810c1a0 2024-09-15T23:22:47.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.981+0000 7fead77fe640 1 -- 172.21.15.89:0/3931921203 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7feae0037020 con 0x7feae810c1a0 2024-09-15T23:22:47.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.981+0000 7fead77fe640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a32d0 0x7fead00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:47.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.982+0000 7feae659c640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a32d0 0x7fead00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:47.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.982+0000 7fead77fe640 1 -- 172.21.15.89:0/3931921203 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7feae00f0d40 con 0x7feae810c1a0 2024-09-15T23:22:47.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.982+0000 7feae659c640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a32d0 0x7fead00a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fead8009890 tx=0x7fead8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:47.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:46.986+0000 7fead77fe640 1 -- 172.21.15.89:0/3931921203 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7feae00bd7d0 con 0x7feae810c1a0 2024-09-15T23:22:47.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.195+0000 7feaecae7640 1 -- 172.21.15.89:0/3931921203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feae81a7480 con 0x7feae810c1a0 2024-09-15T23:22:47.197 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:47 smithi089 ceph-mon[30933]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:47.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.196+0000 7fead77fe640 1 -- 172.21.15.89:0/3931921203 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7feae00bd0d0 con 0x7feae810c1a0 2024-09-15T23:22:47.197 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:47.198 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":28,"btime":"2024-09-15T23:22:37:131197+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:22:37.131175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24452,24436,24430,24428,24424]},"id":1}]} 2024-09-15T23:22:47.198 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 28 2024-09-15T23:22:47.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 -- 172.21.15.89:0/3931921203 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a32d0 msgr2=0x7fead00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:47.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a32d0 0x7fead00a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fead8009890 tx=0x7fead8008040 comp rx=0 tx=0).stop 2024-09-15T23:22:47.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 -- 172.21.15.89:0/3931921203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae810c1a0 msgr2=0x7feae819f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:47.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae810c1a0 0x7feae819f000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feae0006010 tx=0x7feae002f710 comp rx=0 tx=0).stop 2024-09-15T23:22:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 -- 172.21.15.89:0/3931921203 shutdown_connections 2024-09-15T23:22:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a32d0 0x7fead00a5790 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae810c1a0 0x7feae819f000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae8103720 0x7feae819ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 --2- 172.21.15.89:0/3931921203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae8102e00 0x7feae819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 -- 172.21.15.89:0/3931921203 >> 172.21.15.89:0/3931921203 conn(0x7feae80fc7a0 msgr2=0x7feae8101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 -- 172.21.15.89:0/3931921203 shutdown_connections 2024-09-15T23:22:47.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:47.198+0000 7feaecae7640 1 -- 172.21.15.89:0/3931921203 wait complete. 2024-09-15T23:22:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:47 smithi120 ceph-mon[35250]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:47.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:47 smithi174 ceph-mon[35586]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:22:47.886 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:22:48.130 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:48.251 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:48 smithi089 ceph-mon[30933]: pgmap v309: 129 pgs: 129 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.0 KiB/s wr, 24 op/s 2024-09-15T23:22:48.251 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:48 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3931921203' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:48 smithi120 ceph-mon[35250]: pgmap v309: 129 pgs: 129 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.0 KiB/s wr, 24 op/s 2024-09-15T23:22:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:48 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3931921203' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:48.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:48 smithi174 ceph-mon[35586]: pgmap v309: 129 pgs: 129 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.0 KiB/s wr, 24 op/s 2024-09-15T23:22:48.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:48 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3931921203' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.746+0000 7f303bf87640 1 -- 172.21.15.89:0/1431339689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3034101df0 msgr2=0x7f30341021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.746+0000 7f303bf87640 1 --2- 172.21.15.89:0/1431339689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3034101df0 0x7f30341021d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3030009900 tx=0x7f303002a410 comp rx=0 tx=0).stop 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.746+0000 7f303bf87640 1 -- 172.21.15.89:0/1431339689 shutdown_connections 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.746+0000 7f303bf87640 1 --2- 172.21.15.89:0/1431339689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3034106f20 0x7f303410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.746+0000 7f303bf87640 1 --2- 172.21.15.89:0/1431339689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30341027a0 0x7f30341067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.746+0000 7f303bf87640 1 --2- 172.21.15.89:0/1431339689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3034101df0 0x7f30341021d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.746+0000 7f303bf87640 1 -- 172.21.15.89:0/1431339689 >> 172.21.15.89:0/1431339689 conn(0x7f30340fd8b0 msgr2=0x7f30340ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.746+0000 7f303bf87640 1 -- 172.21.15.89:0/1431339689 shutdown_connections 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.746+0000 7f303bf87640 1 -- 172.21.15.89:0/1431339689 wait complete. 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303bf87640 1 Processor -- start 2024-09-15T23:22:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303bf87640 1 -- start start 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303bf87640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3034101df0 0x7f30340744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303bf87640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30341027a0 0x7f3034074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303bf87640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3034106f20 0x7f3034074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303bf87640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3034111f80 con 0x7f3034106f20 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303bf87640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3034111e00 con 0x7f3034101df0 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303bf87640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3034112100 con 0x7f30341027a0 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303a524640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3034106f20 0x7f3034074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303a524640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3034106f20 0x7f3034074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58394/0 (socket says 172.21.15.89:58394) 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f3039d23640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3034101df0 0x7f30340744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303a524640 1 -- 172.21.15.89:0/786350566 learned_addr learned my addr 172.21.15.89:0/786350566 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f3039522640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30341027a0 0x7f3034074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:48.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303a524640 1 -- 172.21.15.89:0/786350566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30341027a0 msgr2=0x7f3034074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303a524640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30341027a0 0x7f3034074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303a524640 1 -- 172.21.15.89:0/786350566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3034101df0 msgr2=0x7f30340744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303a524640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3034101df0 0x7f30340744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.747+0000 7f303a524640 1 -- 172.21.15.89:0/786350566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30341a6900 con 0x7f3034106f20 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.748+0000 7f303a524640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3034106f20 0x7f3034074f70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f302c00eaa0 tx=0x7f302c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.748+0000 7f302affd640 1 -- 172.21.15.89:0/786350566 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c00cdb0 con 0x7f3034106f20 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.748+0000 7f302affd640 1 -- 172.21.15.89:0/786350566 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f302c005020 con 0x7f3034106f20 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.748+0000 7f303bf87640 1 -- 172.21.15.89:0/786350566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30341a6bf0 con 0x7f3034106f20 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.748+0000 7f302affd640 1 -- 172.21.15.89:0/786350566 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c0116b0 con 0x7f3034106f20 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.748+0000 7f303bf87640 1 -- 172.21.15.89:0/786350566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30341a70e0 con 0x7f3034106f20 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.749+0000 7f303bf87640 1 -- 172.21.15.89:0/786350566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30341021d0 con 0x7f3034106f20 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.751+0000 7f302affd640 1 -- 172.21.15.89:0/786350566 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f302c005560 con 0x7f3034106f20 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.751+0000 7f302affd640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30200a32d0 0x7f30200a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:48.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.751+0000 7f302affd640 1 -- 172.21.15.89:0/786350566 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f302c0d01d0 con 0x7f3034106f20 2024-09-15T23:22:48.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.751+0000 7f3039d23640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30200a32d0 0x7f30200a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:48.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.752+0000 7f3039d23640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30200a32d0 0x7f30200a5790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3030009230 tx=0x7f30300058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:48.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.755+0000 7f302affd640 1 -- 172.21.15.89:0/786350566 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f302c016030 con 0x7f3034106f20 2024-09-15T23:22:48.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.939+0000 7f303bf87640 1 -- 172.21.15.89:0/786350566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3034075a20 con 0x7f3034106f20 2024-09-15T23:22:48.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.940+0000 7f302affd640 1 -- 172.21.15.89:0/786350566 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24199 (secure 0 0 0) 0x7f3034075a20 con 0x7f3034106f20 2024-09-15T23:22:48.942 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:48.942 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":86,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:22:02.246640+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:22:48.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 -- 172.21.15.89:0/786350566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30200a32d0 msgr2=0x7f30200a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:48.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30200a32d0 0x7f30200a5790 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3030009230 tx=0x7f30300058d0 comp rx=0 tx=0).stop 2024-09-15T23:22:48.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 -- 172.21.15.89:0/786350566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3034106f20 msgr2=0x7f3034074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:48.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3034106f20 0x7f3034074f70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f302c00eaa0 tx=0x7f302c00ef60 comp rx=0 tx=0).stop 2024-09-15T23:22:48.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 -- 172.21.15.89:0/786350566 shutdown_connections 2024-09-15T23:22:48.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30200a32d0 0x7f30200a5790 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:48.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3034106f20 0x7f3034074f70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:48.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30341027a0 0x7f3034074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:48.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 --2- 172.21.15.89:0/786350566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3034101df0 0x7f30340744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:48.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 -- 172.21.15.89:0/786350566 >> 172.21.15.89:0/786350566 conn(0x7f30340fd8b0 msgr2=0x7f30340febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:48.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 -- 172.21.15.89:0/786350566 shutdown_connections 2024-09-15T23:22:48.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:48.943+0000 7f303bf87640 1 -- 172.21.15.89:0/786350566 wait complete. 2024-09-15T23:22:49.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:49 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:22:49.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:49 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/786350566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:49.356 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:49 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:22:49.356 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:49 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/786350566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:49.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:49 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:22:49.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:49 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/786350566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:49.682 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-15T23:22:49.682 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:22:50.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:50 smithi089 ceph-mon[30933]: pgmap v310: 129 pgs: 129 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.0 KiB/s wr, 20 op/s 2024-09-15T23:22:50.384 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:50.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:50 smithi120 ceph-mon[35250]: pgmap v310: 129 pgs: 129 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.0 KiB/s wr, 20 op/s 2024-09-15T23:22:50.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:50 smithi174 ceph-mon[35586]: pgmap v310: 129 pgs: 129 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.0 KiB/s wr, 20 op/s 2024-09-15T23:22:50.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.867+0000 7f671a0a8640 1 -- 172.21.15.89:0/2062863586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6714108160 msgr2=0x7f671410eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:50.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.867+0000 7f671a0a8640 1 --2- 172.21.15.89:0/2062863586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6714108160 0x7f671410eb00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f670c009cd0 tx=0x7f670c02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:22:50.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.867+0000 7f671a0a8640 1 -- 172.21.15.89:0/2062863586 shutdown_connections 2024-09-15T23:22:50.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.867+0000 7f671a0a8640 1 --2- 172.21.15.89:0/2062863586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6714108160 0x7f671410eb00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.867+0000 7f671a0a8640 1 --2- 172.21.15.89:0/2062863586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6714101290 0x7f6714107a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.867+0000 7f671a0a8640 1 --2- 172.21.15.89:0/2062863586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67141008e0 0x7f6714100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.867+0000 7f671a0a8640 1 -- 172.21.15.89:0/2062863586 >> 172.21.15.89:0/2062863586 conn(0x7f6714077210 msgr2=0x7f6714077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.867+0000 7f671a0a8640 1 -- 172.21.15.89:0/2062863586 shutdown_connections 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.867+0000 7f671a0a8640 1 -- 172.21.15.89:0/2062863586 wait complete. 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f671a0a8640 1 Processor -- start 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f671a0a8640 1 -- start start 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f671a0a8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67141008e0 0x7f6714198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f671a0a8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6714101290 0x7f67141989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f671a0a8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6714108160 0x7f671419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f671a0a8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6714111ff0 con 0x7f6714101290 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f671a0a8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6714111e70 con 0x7f6714108160 2024-09-15T23:22:50.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f671a0a8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6714112170 con 0x7f67141008e0 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f6712ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6714101290 0x7f67141989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f6712ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6714101290 0x7f67141989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58404/0 (socket says 172.21.15.89:58404) 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.868+0000 7f6712ffd640 1 -- 172.21.15.89:0/4036994805 learned_addr learned my addr 172.21.15.89:0/4036994805 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f6713fff640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6714108160 0x7f671419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f6712ffd640 1 -- 172.21.15.89:0/4036994805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67141008e0 msgr2=0x7f6714198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f67137fe640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67141008e0 0x7f6714198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f6712ffd640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67141008e0 0x7f6714198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f6712ffd640 1 -- 172.21.15.89:0/4036994805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6714108160 msgr2=0x7f671419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f6712ffd640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6714108160 0x7f671419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f6712ffd640 1 -- 172.21.15.89:0/4036994805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671419d440 con 0x7f6714101290 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f67137fe640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67141008e0 0x7f6714198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:50.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f6712ffd640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6714101290 0x7f67141989d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f670800d750 tx=0x7f670800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f6710ff9640 1 -- 172.21.15.89:0/4036994805 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708017070 con 0x7f6714101290 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f6710ff9640 1 -- 172.21.15.89:0/4036994805 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6708009c30 con 0x7f6714101290 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f671a0a8640 1 -- 172.21.15.89:0/4036994805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671419d730 con 0x7f6714101290 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f6710ff9640 1 -- 172.21.15.89:0/4036994805 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670800f620 con 0x7f6714101290 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.869+0000 7f671a0a8640 1 -- 172.21.15.89:0/4036994805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67141a5010 con 0x7f6714101290 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.870+0000 7f671a0a8640 1 -- 172.21.15.89:0/4036994805 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6714100cc0 con 0x7f6714101290 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.871+0000 7f6710ff9640 1 -- 172.21.15.89:0/4036994805 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f670801a070 con 0x7f6714101290 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.872+0000 7f6710ff9640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f66fc0a32d0 0x7f66fc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.872+0000 7f67137fe640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f66fc0a32d0 0x7f66fc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.872+0000 7f6710ff9640 1 -- 172.21.15.89:0/4036994805 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f67080cf640 con 0x7f6714101290 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.872+0000 7f67137fe640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f66fc0a32d0 0x7f66fc0a5790 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6704009b40 tx=0x7f6704005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:50.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:50.876+0000 7f6710ff9640 1 -- 172.21.15.89:0/4036994805 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6708009de0 con 0x7f6714101290 2024-09-15T23:22:51.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.051+0000 7f671a0a8640 1 -- 172.21.15.89:0/4036994805 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6714062630 con 0x7f66fc0a32d0 2024-09-15T23:22:51.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.065+0000 7f6710ff9640 1 -- 172.21.15.89:0/4036994805 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340635 (secure 0 0 0) 0x7f6714062630 con 0x7f66fc0a32d0 2024-09-15T23:22:51.069 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:51.071 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-15T23:22:51.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 -- 172.21.15.89:0/4036994805 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f66fc0a32d0 msgr2=0x7f66fc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:51.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f66fc0a32d0 0x7f66fc0a5790 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6704009b40 tx=0x7f6704005e50 comp rx=0 tx=0).stop 2024-09-15T23:22:51.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 -- 172.21.15.89:0/4036994805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6714101290 msgr2=0x7f67141989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:51.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6714101290 0x7f67141989d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f670800d750 tx=0x7f670800dc10 comp rx=0 tx=0).stop 2024-09-15T23:22:51.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 -- 172.21.15.89:0/4036994805 shutdown_connections 2024-09-15T23:22:51.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f66fc0a32d0 0x7f66fc0a5790 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:51.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6714108160 0x7f671419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:51.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6714101290 0x7f67141989d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:51.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 --2- 172.21.15.89:0/4036994805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67141008e0 0x7f6714198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:51.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 -- 172.21.15.89:0/4036994805 >> 172.21.15.89:0/4036994805 conn(0x7f6714077210 msgr2=0x7f67140ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:51.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.068+0000 7f671a0a8640 1 -- 172.21.15.89:0/4036994805 shutdown_connections 2024-09-15T23:22:51.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:51.069+0000 7f671a0a8640 1 -- 172.21.15.89:0/4036994805 wait complete. 2024-09-15T23:22:51.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:51 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:51 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:51 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:51.772 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-15T23:22:50.993221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":640762,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":266,"num_read_kb":232,"num_write":416,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":341,"ondisk_log_size":341,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360944,"kb_used_data":11696,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580880,"statfs":{"total":1151940427776,"available":1151570821120,"internally_reserved":0,"allocated":11976704,"data_stored":7705008,"data_compressed":318738,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40806,"num_objects":15,"num_object_clones":0,"num_object_copies":47,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":138,"num_read_kb":121,"num_write":52,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003292"},"pg_stats":[{"pgid":"2.1f","version":"86'13","reported_seq":54,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:46.166936+0000","last_change":"2024-09-15T23:21:50.513610+0000","last_active":"2024-09-15T23:22:46.166936+0000","last_peered":"2024-09-15T23:22:46.166936+0000","last_clean":"2024-09-15T23:22:46.166936+0000","last_became_active":"2024-09-15T23:21:50.513266+0000","last_became_peered":"2024-09-15T23:21:50.513266+0000","last_unstale":"2024-09-15T23:22:46.166936+0000","last_undegraded":"2024-09-15T23:22:46.166936+0000","last_fullsized":"2024-09-15T23:22:46.166936+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:16:48.692801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576117+0000","last_change":"2024-09-15T23:21:52.540248+0000","last_active":"2024-09-15T23:22:02.576117+0000","last_peered":"2024-09-15T23:22:02.576117+0000","last_clean":"2024-09-15T23:22:02.576117+0000","last_became_active":"2024-09-15T23:21:52.539945+0000","last_became_peered":"2024-09-15T23:21:52.539945+0000","last_unstale":"2024-09-15T23:22:02.576117+0000","last_undegraded":"2024-09-15T23:22:02.576117+0000","last_fullsized":"2024-09-15T23:22:02.576117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:33.402153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634558+0000","last_change":"2024-09-15T23:21:59.567246+0000","last_active":"2024-09-15T23:22:02.634558+0000","last_peered":"2024-09-15T23:22:02.634558+0000","last_clean":"2024-09-15T23:22:02.634558+0000","last_became_active":"2024-09-15T23:21:59.566832+0000","last_became_peered":"2024-09-15T23:21:59.566832+0000","last_unstale":"2024-09-15T23:22:02.634558+0000","last_undegraded":"2024-09-15T23:22:02.634558+0000","last_fullsized":"2024-09-15T23:22:02.634558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:19.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576462+0000","last_change":"2024-09-15T23:21:50.561767+0000","last_active":"2024-09-15T23:22:02.576462+0000","last_peered":"2024-09-15T23:22:02.576462+0000","last_clean":"2024-09-15T23:22:02.576462+0000","last_became_active":"2024-09-15T23:21:50.561385+0000","last_became_peered":"2024-09-15T23:21:50.561385+0000","last_unstale":"2024-09-15T23:22:02.576462+0000","last_undegraded":"2024-09-15T23:22:02.576462+0000","last_fullsized":"2024-09-15T23:22:02.576462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:11.897884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.923183+0000","last_change":"2024-09-15T23:21:52.534874+0000","last_active":"2024-09-15T23:22:02.923183+0000","last_peered":"2024-09-15T23:22:02.923183+0000","last_clean":"2024-09-15T23:22:02.923183+0000","last_became_active":"2024-09-15T23:21:52.534334+0000","last_became_peered":"2024-09-15T23:21:52.534334+0000","last_unstale":"2024-09-15T23:22:02.923183+0000","last_undegraded":"2024-09-15T23:22:02.923183+0000","last_fullsized":"2024-09-15T23:22:02.923183+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:15.972648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632188+0000","last_change":"2024-09-15T23:21:59.559907+0000","last_active":"2024-09-15T23:22:02.632188+0000","last_peered":"2024-09-15T23:22:02.632188+0000","last_clean":"2024-09-15T23:22:02.632188+0000","last_became_active":"2024-09-15T23:21:59.559440+0000","last_became_peered":"2024-09-15T23:21:59.559440+0000","last_unstale":"2024-09-15T23:22:02.632188+0000","last_undegraded":"2024-09-15T23:22:02.632188+0000","last_fullsized":"2024-09-15T23:22:02.632188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:00.755112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:37.131384+0000","last_change":"2024-09-15T23:21:50.515424+0000","last_active":"2024-09-15T23:22:37.131384+0000","last_peered":"2024-09-15T23:22:37.131384+0000","last_clean":"2024-09-15T23:22:37.131384+0000","last_became_active":"2024-09-15T23:21:50.514966+0000","last_became_peered":"2024-09-15T23:21:50.514966+0000","last_unstale":"2024-09-15T23:22:37.131384+0000","last_undegraded":"2024-09-15T23:22:37.131384+0000","last_fullsized":"2024-09-15T23:22:37.131384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:57.586575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632356+0000","last_change":"2024-09-15T23:21:52.535189+0000","last_active":"2024-09-15T23:22:02.632356+0000","last_peered":"2024-09-15T23:22:02.632356+0000","last_clean":"2024-09-15T23:22:02.632356+0000","last_became_active":"2024-09-15T23:21:52.534807+0000","last_became_peered":"2024-09-15T23:21:52.534807+0000","last_unstale":"2024-09-15T23:22:02.632356+0000","last_undegraded":"2024-09-15T23:22:02.632356+0000","last_fullsized":"2024-09-15T23:22:02.632356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:30.654382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632608+0000","last_change":"2024-09-15T23:21:59.564675+0000","last_active":"2024-09-15T23:22:02.632608+0000","last_peered":"2024-09-15T23:22:02.632608+0000","last_clean":"2024-09-15T23:22:02.632608+0000","last_became_active":"2024-09-15T23:21:59.564271+0000","last_became_peered":"2024-09-15T23:21:59.564271+0000","last_unstale":"2024-09-15T23:22:02.632608+0000","last_undegraded":"2024-09-15T23:22:02.632608+0000","last_fullsized":"2024-09-15T23:22:02.632608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:01.055369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.136685+0000","last_change":"2024-09-15T23:21:50.507790+0000","last_active":"2024-09-15T23:22:36.136685+0000","last_peered":"2024-09-15T23:22:36.136685+0000","last_clean":"2024-09-15T23:22:36.136685+0000","last_became_active":"2024-09-15T23:21:50.507398+0000","last_became_peered":"2024-09-15T23:21:50.507398+0000","last_unstale":"2024-09-15T23:22:36.136685+0000","last_undegraded":"2024-09-15T23:22:36.136685+0000","last_fullsized":"2024-09-15T23:22:36.136685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:58:00.001279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634061+0000","last_change":"2024-09-15T23:21:52.537179+0000","last_active":"2024-09-15T23:22:02.634061+0000","last_peered":"2024-09-15T23:22:02.634061+0000","last_clean":"2024-09-15T23:22:02.634061+0000","last_became_active":"2024-09-15T23:21:52.536841+0000","last_became_peered":"2024-09-15T23:21:52.536841+0000","last_unstale":"2024-09-15T23:22:02.634061+0000","last_undegraded":"2024-09-15T23:22:02.634061+0000","last_fullsized":"2024-09-15T23:22:02.634061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:47.455700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632181+0000","last_change":"2024-09-15T23:21:59.564734+0000","last_active":"2024-09-15T23:22:02.632181+0000","last_peered":"2024-09-15T23:22:02.632181+0000","last_clean":"2024-09-15T23:22:02.632181+0000","last_became_active":"2024-09-15T23:21:59.564339+0000","last_became_peered":"2024-09-15T23:21:59.564339+0000","last_unstale":"2024-09-15T23:22:02.632181+0000","last_undegraded":"2024-09-15T23:22:02.632181+0000","last_fullsized":"2024-09-15T23:22:02.632181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:44.498480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.578855+0000","last_change":"2024-09-15T23:21:52.538025+0000","last_active":"2024-09-15T23:22:02.578855+0000","last_peered":"2024-09-15T23:22:02.578855+0000","last_clean":"2024-09-15T23:22:02.578855+0000","last_became_active":"2024-09-15T23:21:52.537736+0000","last_became_peered":"2024-09-15T23:21:52.537736+0000","last_unstale":"2024-09-15T23:22:02.578855+0000","last_undegraded":"2024-09-15T23:22:02.578855+0000","last_fullsized":"2024-09-15T23:22:02.578855+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:31.916888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:33.097914+0000","last_change":"2024-09-15T23:21:50.516024+0000","last_active":"2024-09-15T23:22:33.097914+0000","last_peered":"2024-09-15T23:22:33.097914+0000","last_clean":"2024-09-15T23:22:33.097914+0000","last_became_active":"2024-09-15T23:21:50.515590+0000","last_became_peered":"2024-09-15T23:21:50.515590+0000","last_unstale":"2024-09-15T23:22:33.097914+0000","last_undegraded":"2024-09-15T23:22:33.097914+0000","last_fullsized":"2024-09-15T23:22:33.097914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:47:38.849366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632272+0000","last_change":"2024-09-15T23:21:59.681971+0000","last_active":"2024-09-15T23:22:02.632272+0000","last_peered":"2024-09-15T23:22:02.632272+0000","last_clean":"2024-09-15T23:22:02.632272+0000","last_became_active":"2024-09-15T23:21:59.681432+0000","last_became_peered":"2024-09-15T23:21:59.681432+0000","last_unstale":"2024-09-15T23:22:02.632272+0000","last_undegraded":"2024-09-15T23:22:02.632272+0000","last_fullsized":"2024-09-15T23:22:02.632272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:26.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.575941+0000","last_change":"2024-09-15T23:21:52.541281+0000","last_active":"2024-09-15T23:22:02.575941+0000","last_peered":"2024-09-15T23:22:02.575941+0000","last_clean":"2024-09-15T23:22:02.575941+0000","last_became_active":"2024-09-15T23:21:52.540772+0000","last_became_peered":"2024-09-15T23:21:52.540772+0000","last_unstale":"2024-09-15T23:22:02.575941+0000","last_undegraded":"2024-09-15T23:22:02.575941+0000","last_fullsized":"2024-09-15T23:22:02.575941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:23:44.444904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:35.119894+0000","last_change":"2024-09-15T23:21:50.508844+0000","last_active":"2024-09-15T23:22:35.119894+0000","last_peered":"2024-09-15T23:22:35.119894+0000","last_clean":"2024-09-15T23:22:35.119894+0000","last_became_active":"2024-09-15T23:21:50.508364+0000","last_became_peered":"2024-09-15T23:21:50.508364+0000","last_unstale":"2024-09-15T23:22:35.119894+0000","last_undegraded":"2024-09-15T23:22:35.119894+0000","last_fullsized":"2024-09-15T23:22:35.119894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:45.129711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633100+0000","last_change":"2024-09-15T23:21:59.562114+0000","last_active":"2024-09-15T23:22:02.633100+0000","last_peered":"2024-09-15T23:22:02.633100+0000","last_clean":"2024-09-15T23:22:02.633100+0000","last_became_active":"2024-09-15T23:21:59.561652+0000","last_became_peered":"2024-09-15T23:21:59.561652+0000","last_unstale":"2024-09-15T23:22:02.633100+0000","last_undegraded":"2024-09-15T23:22:02.633100+0000","last_fullsized":"2024-09-15T23:22:02.633100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:51.781886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632984+0000","last_change":"2024-09-15T23:21:52.531678+0000","last_active":"2024-09-15T23:22:02.632984+0000","last_peered":"2024-09-15T23:22:02.632984+0000","last_clean":"2024-09-15T23:22:02.632984+0000","last_became_active":"2024-09-15T23:21:52.531327+0000","last_became_peered":"2024-09-15T23:21:52.531327+0000","last_unstale":"2024-09-15T23:22:02.632984+0000","last_undegraded":"2024-09-15T23:22:02.632984+0000","last_fullsized":"2024-09-15T23:22:02.632984+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:07.152626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.134997+0000","last_change":"2024-09-15T23:21:50.513197+0000","last_active":"2024-09-15T23:22:36.134997+0000","last_peered":"2024-09-15T23:22:36.134997+0000","last_clean":"2024-09-15T23:22:36.134997+0000","last_became_active":"2024-09-15T23:21:50.512794+0000","last_became_peered":"2024-09-15T23:21:50.512794+0000","last_unstale":"2024-09-15T23:22:36.134997+0000","last_undegraded":"2024-09-15T23:22:36.134997+0000","last_fullsized":"2024-09-15T23:22:36.134997+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:05.457467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633044+0000","last_change":"2024-09-15T23:21:59.565669+0000","last_active":"2024-09-15T23:22:02.633044+0000","last_peered":"2024-09-15T23:22:02.633044+0000","last_clean":"2024-09-15T23:22:02.633044+0000","last_became_active":"2024-09-15T23:21:59.565290+0000","last_became_peered":"2024-09-15T23:21:59.565290+0000","last_unstale":"2024-09-15T23:22:02.633044+0000","last_undegraded":"2024-09-15T23:22:02.633044+0000","last_fullsized":"2024-09-15T23:22:02.633044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:04.861560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576604+0000","last_change":"2024-09-15T23:21:52.529643+0000","last_active":"2024-09-15T23:22:02.576604+0000","last_peered":"2024-09-15T23:22:02.576604+0000","last_clean":"2024-09-15T23:22:02.576604+0000","last_became_active":"2024-09-15T23:21:52.529304+0000","last_became_peered":"2024-09-15T23:21:52.529304+0000","last_unstale":"2024-09-15T23:22:02.576604+0000","last_undegraded":"2024-09-15T23:22:02.576604+0000","last_fullsized":"2024-09-15T23:22:02.576604+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:18:37.758403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:33.097341+0000","last_change":"2024-09-15T23:21:50.562114+0000","last_active":"2024-09-15T23:22:33.097341+0000","last_peered":"2024-09-15T23:22:33.097341+0000","last_clean":"2024-09-15T23:22:33.097341+0000","last_became_active":"2024-09-15T23:21:50.561772+0000","last_became_peered":"2024-09-15T23:21:50.561772+0000","last_unstale":"2024-09-15T23:22:33.097341+0000","last_undegraded":"2024-09-15T23:22:33.097341+0000","last_fullsized":"2024-09-15T23:22:33.097341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:14.806646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633256+0000","last_change":"2024-09-15T23:21:59.681933+0000","last_active":"2024-09-15T23:22:02.633256+0000","last_peered":"2024-09-15T23:22:02.633256+0000","last_clean":"2024-09-15T23:22:02.633256+0000","last_became_active":"2024-09-15T23:21:59.681528+0000","last_became_peered":"2024-09-15T23:21:59.681528+0000","last_unstale":"2024-09-15T23:22:02.633256+0000","last_undegraded":"2024-09-15T23:22:02.633256+0000","last_fullsized":"2024-09-15T23:22:02.633256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:46.199540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.923082+0000","last_change":"2024-09-15T23:21:52.529548+0000","last_active":"2024-09-15T23:22:02.923082+0000","last_peered":"2024-09-15T23:22:02.923082+0000","last_clean":"2024-09-15T23:22:02.923082+0000","last_became_active":"2024-09-15T23:21:52.529253+0000","last_became_peered":"2024-09-15T23:21:52.529253+0000","last_unstale":"2024-09-15T23:22:02.923082+0000","last_undegraded":"2024-09-15T23:22:02.923082+0000","last_fullsized":"2024-09-15T23:22:02.923082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:22.813395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.134592+0000","last_change":"2024-09-15T23:21:50.767759+0000","last_active":"2024-09-15T23:22:36.134592+0000","last_peered":"2024-09-15T23:22:36.134592+0000","last_clean":"2024-09-15T23:22:36.134592+0000","last_became_active":"2024-09-15T23:21:50.767388+0000","last_became_peered":"2024-09-15T23:21:50.767388+0000","last_unstale":"2024-09-15T23:22:36.134592+0000","last_undegraded":"2024-09-15T23:22:36.134592+0000","last_fullsized":"2024-09-15T23:22:36.134592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:36.713769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634571+0000","last_change":"2024-09-15T23:21:59.557951+0000","last_active":"2024-09-15T23:22:02.634571+0000","last_peered":"2024-09-15T23:22:02.634571+0000","last_clean":"2024-09-15T23:22:02.634571+0000","last_became_active":"2024-09-15T23:21:59.557490+0000","last_became_peered":"2024-09-15T23:21:59.557490+0000","last_unstale":"2024-09-15T23:22:02.634571+0000","last_undegraded":"2024-09-15T23:22:02.634571+0000","last_fullsized":"2024-09-15T23:22:02.634571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:17.147389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576306+0000","last_change":"2024-09-15T23:21:52.531638+0000","last_active":"2024-09-15T23:22:02.576306+0000","last_peered":"2024-09-15T23:22:02.576306+0000","last_clean":"2024-09-15T23:22:02.576306+0000","last_became_active":"2024-09-15T23:21:52.531301+0000","last_became_peered":"2024-09-15T23:21:52.531301+0000","last_unstale":"2024-09-15T23:22:02.576306+0000","last_undegraded":"2024-09-15T23:22:02.576306+0000","last_fullsized":"2024-09-15T23:22:02.576306+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.436286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'12","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:44.532224+0000","last_change":"2024-09-15T23:21:50.508470+0000","last_active":"2024-09-15T23:22:44.532224+0000","last_peered":"2024-09-15T23:22:44.532224+0000","last_clean":"2024-09-15T23:22:44.532224+0000","last_became_active":"2024-09-15T23:21:50.508060+0000","last_became_peered":"2024-09-15T23:21:50.508060+0000","last_unstale":"2024-09-15T23:22:44.532224+0000","last_undegraded":"2024-09-15T23:22:44.532224+0000","last_fullsized":"2024-09-15T23:22:44.532224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:59.542252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.575941+0000","last_change":"2024-09-15T23:21:59.682019+0000","last_active":"2024-09-15T23:22:02.575941+0000","last_peered":"2024-09-15T23:22:02.575941+0000","last_clean":"2024-09-15T23:22:02.575941+0000","last_became_active":"2024-09-15T23:21:59.681467+0000","last_became_peered":"2024-09-15T23:21:59.681467+0000","last_unstale":"2024-09-15T23:22:02.575941+0000","last_undegraded":"2024-09-15T23:22:02.575941+0000","last_fullsized":"2024-09-15T23:22:02.575941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:27.650382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632526+0000","last_change":"2024-09-15T23:21:52.536345+0000","last_active":"2024-09-15T23:22:02.632526+0000","last_peered":"2024-09-15T23:22:02.632526+0000","last_clean":"2024-09-15T23:22:02.632526+0000","last_became_active":"2024-09-15T23:21:52.536027+0000","last_became_peered":"2024-09-15T23:21:52.536027+0000","last_unstale":"2024-09-15T23:22:02.632526+0000","last_undegraded":"2024-09-15T23:22:02.632526+0000","last_fullsized":"2024-09-15T23:22:02.632526+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:16.435912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.135178+0000","last_change":"2024-09-15T23:21:50.562945+0000","last_active":"2024-09-15T23:22:36.135178+0000","last_peered":"2024-09-15T23:22:36.135178+0000","last_clean":"2024-09-15T23:22:36.135178+0000","last_became_active":"2024-09-15T23:21:50.561632+0000","last_became_peered":"2024-09-15T23:21:50.561632+0000","last_unstale":"2024-09-15T23:22:36.135178+0000","last_undegraded":"2024-09-15T23:22:36.135178+0000","last_fullsized":"2024-09-15T23:22:36.135178+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:44.545489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633201+0000","last_change":"2024-09-15T23:21:59.564793+0000","last_active":"2024-09-15T23:22:02.633201+0000","last_peered":"2024-09-15T23:22:02.633201+0000","last_clean":"2024-09-15T23:22:02.633201+0000","last_became_active":"2024-09-15T23:21:59.564140+0000","last_became_peered":"2024-09-15T23:21:59.564140+0000","last_unstale":"2024-09-15T23:22:02.633201+0000","last_undegraded":"2024-09-15T23:22:02.633201+0000","last_fullsized":"2024-09-15T23:22:02.633201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:41.464962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632773+0000","last_change":"2024-09-15T23:21:52.533521+0000","last_active":"2024-09-15T23:22:02.632773+0000","last_peered":"2024-09-15T23:22:02.632773+0000","last_clean":"2024-09-15T23:22:02.632773+0000","last_became_active":"2024-09-15T23:21:52.532195+0000","last_became_peered":"2024-09-15T23:21:52.532195+0000","last_unstale":"2024-09-15T23:22:02.632773+0000","last_undegraded":"2024-09-15T23:22:02.632773+0000","last_fullsized":"2024-09-15T23:22:02.632773+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:14.229018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633039+0000","last_change":"2024-09-15T23:21:50.562893+0000","last_active":"2024-09-15T23:22:02.633039+0000","last_peered":"2024-09-15T23:22:02.633039+0000","last_clean":"2024-09-15T23:22:02.633039+0000","last_became_active":"2024-09-15T23:21:50.562516+0000","last_became_peered":"2024-09-15T23:21:50.562516+0000","last_unstale":"2024-09-15T23:22:02.633039+0000","last_undegraded":"2024-09-15T23:22:02.633039+0000","last_fullsized":"2024-09-15T23:22:02.633039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:47:47.442458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576698+0000","last_change":"2024-09-15T23:21:59.682091+0000","last_active":"2024-09-15T23:22:02.576698+0000","last_peered":"2024-09-15T23:22:02.576698+0000","last_clean":"2024-09-15T23:22:02.576698+0000","last_became_active":"2024-09-15T23:21:59.681574+0000","last_became_peered":"2024-09-15T23:21:59.681574+0000","last_unstale":"2024-09-15T23:22:02.576698+0000","last_undegraded":"2024-09-15T23:22:02.576698+0000","last_fullsized":"2024-09-15T23:22:02.576698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:04.991608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633167+0000","last_change":"2024-09-15T23:21:52.533811+0000","last_active":"2024-09-15T23:22:02.633167+0000","last_peered":"2024-09-15T23:22:02.633167+0000","last_clean":"2024-09-15T23:22:02.633167+0000","last_became_active":"2024-09-15T23:21:52.533487+0000","last_became_peered":"2024-09-15T23:21:52.533487+0000","last_unstale":"2024-09-15T23:22:02.633167+0000","last_undegraded":"2024-09-15T23:22:02.633167+0000","last_fullsized":"2024-09-15T23:22:02.633167+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:44.193212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:45.166348+0000","last_change":"2024-09-15T23:21:50.510883+0000","last_active":"2024-09-15T23:22:45.166348+0000","last_peered":"2024-09-15T23:22:45.166348+0000","last_clean":"2024-09-15T23:22:45.166348+0000","last_became_active":"2024-09-15T23:21:50.510478+0000","last_became_peered":"2024-09-15T23:21:50.510478+0000","last_unstale":"2024-09-15T23:22:45.166348+0000","last_undegraded":"2024-09-15T23:22:45.166348+0000","last_fullsized":"2024-09-15T23:22:45.166348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:35.454885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632424+0000","last_change":"2024-09-15T23:21:59.561816+0000","last_active":"2024-09-15T23:22:02.632424+0000","last_peered":"2024-09-15T23:22:02.632424+0000","last_clean":"2024-09-15T23:22:02.632424+0000","last_became_active":"2024-09-15T23:21:59.561434+0000","last_became_peered":"2024-09-15T23:21:59.561434+0000","last_unstale":"2024-09-15T23:22:02.632424+0000","last_undegraded":"2024-09-15T23:22:02.632424+0000","last_fullsized":"2024-09-15T23:22:02.632424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:59.945802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632948+0000","last_change":"2024-09-15T23:21:52.536886+0000","last_active":"2024-09-15T23:22:02.632948+0000","last_peered":"2024-09-15T23:22:02.632948+0000","last_clean":"2024-09-15T23:22:02.632948+0000","last_became_active":"2024-09-15T23:21:52.536570+0000","last_became_peered":"2024-09-15T23:21:52.536570+0000","last_unstale":"2024-09-15T23:22:02.632948+0000","last_undegraded":"2024-09-15T23:22:02.632948+0000","last_fullsized":"2024-09-15T23:22:02.632948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:05:25.330311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:44.150819+0000","last_change":"2024-09-15T23:21:50.561684+0000","last_active":"2024-09-15T23:22:44.150819+0000","last_peered":"2024-09-15T23:22:44.150819+0000","last_clean":"2024-09-15T23:22:44.150819+0000","last_became_active":"2024-09-15T23:21:50.561335+0000","last_became_peered":"2024-09-15T23:21:50.561335+0000","last_unstale":"2024-09-15T23:22:44.150819+0000","last_undegraded":"2024-09-15T23:22:44.150819+0000","last_fullsized":"2024-09-15T23:22:44.150819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:23.897250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632723+0000","last_change":"2024-09-15T23:21:59.681858+0000","last_active":"2024-09-15T23:22:02.632723+0000","last_peered":"2024-09-15T23:22:02.632723+0000","last_clean":"2024-09-15T23:22:02.632723+0000","last_became_active":"2024-09-15T23:21:59.681441+0000","last_became_peered":"2024-09-15T23:21:59.681441+0000","last_unstale":"2024-09-15T23:22:02.632723+0000","last_undegraded":"2024-09-15T23:22:02.632723+0000","last_fullsized":"2024-09-15T23:22:02.632723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:28.513719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632034+0000","last_change":"2024-09-15T23:21:52.536141+0000","last_active":"2024-09-15T23:22:02.632034+0000","last_peered":"2024-09-15T23:22:02.632034+0000","last_clean":"2024-09-15T23:22:02.632034+0000","last_became_active":"2024-09-15T23:21:52.535832+0000","last_became_peered":"2024-09-15T23:21:52.535832+0000","last_unstale":"2024-09-15T23:22:02.632034+0000","last_undegraded":"2024-09-15T23:22:02.632034+0000","last_fullsized":"2024-09-15T23:22:02.632034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:21:44.341412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:34.103482+0000","last_change":"2024-09-15T23:21:50.514689+0000","last_active":"2024-09-15T23:22:34.103482+0000","last_peered":"2024-09-15T23:22:34.103482+0000","last_clean":"2024-09-15T23:22:34.103482+0000","last_became_active":"2024-09-15T23:21:50.513261+0000","last_became_peered":"2024-09-15T23:21:50.513261+0000","last_unstale":"2024-09-15T23:22:34.103482+0000","last_undegraded":"2024-09-15T23:22:34.103482+0000","last_fullsized":"2024-09-15T23:22:34.103482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:30.132991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633489+0000","last_change":"2024-09-15T23:21:59.682118+0000","last_active":"2024-09-15T23:22:02.633489+0000","last_peered":"2024-09-15T23:22:02.633489+0000","last_clean":"2024-09-15T23:22:02.633489+0000","last_became_active":"2024-09-15T23:21:59.681500+0000","last_became_peered":"2024-09-15T23:21:59.681500+0000","last_unstale":"2024-09-15T23:22:02.633489+0000","last_undegraded":"2024-09-15T23:22:02.633489+0000","last_fullsized":"2024-09-15T23:22:02.633489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:58.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576134+0000","last_change":"2024-09-15T23:21:52.540458+0000","last_active":"2024-09-15T23:22:02.576134+0000","last_peered":"2024-09-15T23:22:02.576134+0000","last_clean":"2024-09-15T23:22:02.576134+0000","last_became_active":"2024-09-15T23:21:52.540112+0000","last_became_peered":"2024-09-15T23:21:52.540112+0000","last_unstale":"2024-09-15T23:22:02.576134+0000","last_undegraded":"2024-09-15T23:22:02.576134+0000","last_fullsized":"2024-09-15T23:22:02.576134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:15.240785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:37.137630+0000","last_change":"2024-09-15T23:21:50.513124+0000","last_active":"2024-09-15T23:22:37.137630+0000","last_peered":"2024-09-15T23:22:37.137630+0000","last_clean":"2024-09-15T23:22:37.137630+0000","last_became_active":"2024-09-15T23:21:50.512705+0000","last_became_peered":"2024-09-15T23:21:50.512705+0000","last_unstale":"2024-09-15T23:22:37.137630+0000","last_undegraded":"2024-09-15T23:22:37.137630+0000","last_fullsized":"2024-09-15T23:22:37.137630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.417937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":345,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.578940+0000","last_change":"2024-09-15T23:20:40.656652+0000","last_active":"2024-09-15T23:22:02.578940+0000","last_peered":"2024-09-15T23:22:02.578940+0000","last_clean":"2024-09-15T23:22:02.578940+0000","last_became_active":"2024-09-15T23:20:40.656266+0000","last_became_peered":"2024-09-15T23:20:40.656266+0000","last_unstale":"2024-09-15T23:22:02.578940+0000","last_undegraded":"2024-09-15T23:22:02.578940+0000","last_fullsized":"2024-09-15T23:22:02.578940+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_clean_scrub_stamp":"2024-09-15T23:17:15.186850+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:42.317006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.579288+0000","last_change":"2024-09-15T23:21:59.563777+0000","last_active":"2024-09-15T23:22:02.579288+0000","last_peered":"2024-09-15T23:22:02.579288+0000","last_clean":"2024-09-15T23:22:02.579288+0000","last_became_active":"2024-09-15T23:21:59.563320+0000","last_became_peered":"2024-09-15T23:21:59.563320+0000","last_unstale":"2024-09-15T23:22:02.579288+0000","last_undegraded":"2024-09-15T23:22:02.579288+0000","last_fullsized":"2024-09-15T23:22:02.579288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:25:07.791923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.579367+0000","last_change":"2024-09-15T23:21:52.539842+0000","last_active":"2024-09-15T23:22:02.579367+0000","last_peered":"2024-09-15T23:22:02.579367+0000","last_clean":"2024-09-15T23:22:02.579367+0000","last_became_active":"2024-09-15T23:21:52.538683+0000","last_became_peered":"2024-09-15T23:21:52.538683+0000","last_unstale":"2024-09-15T23:22:02.579367+0000","last_undegraded":"2024-09-15T23:22:02.579367+0000","last_fullsized":"2024-09-15T23:22:02.579367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:43.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:48.158017+0000","last_change":"2024-09-15T23:21:50.767601+0000","last_active":"2024-09-15T23:22:48.158017+0000","last_peered":"2024-09-15T23:22:48.158017+0000","last_clean":"2024-09-15T23:22:48.158017+0000","last_became_active":"2024-09-15T23:21:50.767149+0000","last_became_peered":"2024-09-15T23:21:50.767149+0000","last_unstale":"2024-09-15T23:22:48.158017+0000","last_undegraded":"2024-09-15T23:22:48.158017+0000","last_fullsized":"2024-09-15T23:22:48.158017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:23.636551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632234+0000","last_change":"2024-09-15T23:21:59.563069+0000","last_active":"2024-09-15T23:22:02.632234+0000","last_peered":"2024-09-15T23:22:02.632234+0000","last_clean":"2024-09-15T23:22:02.632234+0000","last_became_active":"2024-09-15T23:21:59.562658+0000","last_became_peered":"2024-09-15T23:21:59.562658+0000","last_unstale":"2024-09-15T23:22:02.632234+0000","last_undegraded":"2024-09-15T23:22:02.632234+0000","last_fullsized":"2024-09-15T23:22:02.632234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:23.067198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.579146+0000","last_change":"2024-09-15T23:21:52.535289+0000","last_active":"2024-09-15T23:22:02.579146+0000","last_peered":"2024-09-15T23:22:02.579146+0000","last_clean":"2024-09-15T23:22:02.579146+0000","last_became_active":"2024-09-15T23:21:52.534885+0000","last_became_peered":"2024-09-15T23:21:52.534885+0000","last_unstale":"2024-09-15T23:22:02.579146+0000","last_undegraded":"2024-09-15T23:22:02.579146+0000","last_fullsized":"2024-09-15T23:22:02.579146+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:42.293786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.135986+0000","last_change":"2024-09-15T23:21:50.513246+0000","last_active":"2024-09-15T23:22:36.135986+0000","last_peered":"2024-09-15T23:22:36.135986+0000","last_clean":"2024-09-15T23:22:36.135986+0000","last_became_active":"2024-09-15T23:21:50.512740+0000","last_became_peered":"2024-09-15T23:21:50.512740+0000","last_unstale":"2024-09-15T23:22:36.135986+0000","last_undegraded":"2024-09-15T23:22:36.135986+0000","last_fullsized":"2024-09-15T23:22:36.135986+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:59.201970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922863+0000","last_change":"2024-09-15T23:21:59.568565+0000","last_active":"2024-09-15T23:22:02.922863+0000","last_peered":"2024-09-15T23:22:02.922863+0000","last_clean":"2024-09-15T23:22:02.922863+0000","last_became_active":"2024-09-15T23:21:59.567789+0000","last_became_peered":"2024-09-15T23:21:59.567789+0000","last_unstale":"2024-09-15T23:22:02.922863+0000","last_undegraded":"2024-09-15T23:22:02.922863+0000","last_fullsized":"2024-09-15T23:22:02.922863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:02.106971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.579365+0000","last_change":"2024-09-15T23:21:52.533727+0000","last_active":"2024-09-15T23:22:02.579365+0000","last_peered":"2024-09-15T23:22:02.579365+0000","last_clean":"2024-09-15T23:22:02.579365+0000","last_became_active":"2024-09-15T23:21:52.532350+0000","last_became_peered":"2024-09-15T23:21:52.532350+0000","last_unstale":"2024-09-15T23:22:02.579365+0000","last_undegraded":"2024-09-15T23:22:02.579365+0000","last_fullsized":"2024-09-15T23:22:02.579365+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:03.624837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.134380+0000","last_change":"2024-09-15T23:21:50.509519+0000","last_active":"2024-09-15T23:22:36.134380+0000","last_peered":"2024-09-15T23:22:36.134380+0000","last_clean":"2024-09-15T23:22:36.134380+0000","last_became_active":"2024-09-15T23:21:50.509224+0000","last_became_peered":"2024-09-15T23:21:50.509224+0000","last_unstale":"2024-09-15T23:22:36.134380+0000","last_undegraded":"2024-09-15T23:22:36.134380+0000","last_fullsized":"2024-09-15T23:22:36.134380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:39:57.526821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632669+0000","last_change":"2024-09-15T23:21:59.566429+0000","last_active":"2024-09-15T23:22:02.632669+0000","last_peered":"2024-09-15T23:22:02.632669+0000","last_clean":"2024-09-15T23:22:02.632669+0000","last_became_active":"2024-09-15T23:21:59.565835+0000","last_became_peered":"2024-09-15T23:21:59.565835+0000","last_unstale":"2024-09-15T23:22:02.632669+0000","last_undegraded":"2024-09-15T23:22:02.632669+0000","last_fullsized":"2024-09-15T23:22:02.632669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:38.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633755+0000","last_change":"2024-09-15T23:21:52.539544+0000","last_active":"2024-09-15T23:22:02.633755+0000","last_peered":"2024-09-15T23:22:02.633755+0000","last_clean":"2024-09-15T23:22:02.633755+0000","last_became_active":"2024-09-15T23:21:52.539203+0000","last_became_peered":"2024-09-15T23:21:52.539203+0000","last_unstale":"2024-09-15T23:22:02.633755+0000","last_undegraded":"2024-09-15T23:22:02.633755+0000","last_fullsized":"2024-09-15T23:22:02.633755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.193965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.099173+0000","last_change":"2024-09-15T23:21:50.513543+0000","last_active":"2024-09-15T23:22:36.099173+0000","last_peered":"2024-09-15T23:22:36.099173+0000","last_clean":"2024-09-15T23:22:36.099173+0000","last_became_active":"2024-09-15T23:21:50.512812+0000","last_became_peered":"2024-09-15T23:21:50.512812+0000","last_unstale":"2024-09-15T23:22:36.099173+0000","last_undegraded":"2024-09-15T23:22:36.099173+0000","last_fullsized":"2024-09-15T23:22:36.099173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:50.954096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632914+0000","last_change":"2024-09-15T23:21:59.570163+0000","last_active":"2024-09-15T23:22:02.632914+0000","last_peered":"2024-09-15T23:22:02.632914+0000","last_clean":"2024-09-15T23:22:02.632914+0000","last_became_active":"2024-09-15T23:21:59.569805+0000","last_became_peered":"2024-09-15T23:21:59.569805+0000","last_unstale":"2024-09-15T23:22:02.632914+0000","last_undegraded":"2024-09-15T23:22:02.632914+0000","last_fullsized":"2024-09-15T23:22:02.632914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:28.530817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633132+0000","last_change":"2024-09-15T23:21:52.537850+0000","last_active":"2024-09-15T23:22:02.633132+0000","last_peered":"2024-09-15T23:22:02.633132+0000","last_clean":"2024-09-15T23:22:02.633132+0000","last_became_active":"2024-09-15T23:21:52.536579+0000","last_became_peered":"2024-09-15T23:21:52.536579+0000","last_unstale":"2024-09-15T23:22:02.633132+0000","last_undegraded":"2024-09-15T23:22:02.633132+0000","last_fullsized":"2024-09-15T23:22:02.633132+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:06.353615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.134639+0000","last_change":"2024-09-15T23:21:50.562221+0000","last_active":"2024-09-15T23:22:36.134639+0000","last_peered":"2024-09-15T23:22:36.134639+0000","last_clean":"2024-09-15T23:22:36.134639+0000","last_became_active":"2024-09-15T23:21:50.561788+0000","last_became_peered":"2024-09-15T23:21:50.561788+0000","last_unstale":"2024-09-15T23:22:36.134639+0000","last_undegraded":"2024-09-15T23:22:36.134639+0000","last_fullsized":"2024-09-15T23:22:36.134639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.853812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632398+0000","last_change":"2024-09-15T23:21:59.558621+0000","last_active":"2024-09-15T23:22:02.632398+0000","last_peered":"2024-09-15T23:22:02.632398+0000","last_clean":"2024-09-15T23:22:02.632398+0000","last_became_active":"2024-09-15T23:21:59.558248+0000","last_became_peered":"2024-09-15T23:21:59.558248+0000","last_unstale":"2024-09-15T23:22:02.632398+0000","last_undegraded":"2024-09-15T23:22:02.632398+0000","last_fullsized":"2024-09-15T23:22:02.632398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:49.854301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.578700+0000","last_change":"2024-09-15T23:21:52.539566+0000","last_active":"2024-09-15T23:22:02.578700+0000","last_peered":"2024-09-15T23:22:02.578700+0000","last_clean":"2024-09-15T23:22:02.578700+0000","last_became_active":"2024-09-15T23:21:52.539219+0000","last_became_peered":"2024-09-15T23:21:52.539219+0000","last_unstale":"2024-09-15T23:22:02.578700+0000","last_undegraded":"2024-09-15T23:22:02.578700+0000","last_fullsized":"2024-09-15T23:22:02.578700+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:41.266308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":52,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:49.158954+0000","last_change":"2024-09-15T23:21:50.516650+0000","last_active":"2024-09-15T23:22:49.158954+0000","last_peered":"2024-09-15T23:22:49.158954+0000","last_clean":"2024-09-15T23:22:49.158954+0000","last_became_active":"2024-09-15T23:21:50.516297+0000","last_became_peered":"2024-09-15T23:21:50.516297+0000","last_unstale":"2024-09-15T23:22:49.158954+0000","last_undegraded":"2024-09-15T23:22:49.158954+0000","last_fullsized":"2024-09-15T23:22:49.158954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:36.928701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633089+0000","last_change":"2024-09-15T23:21:59.568087+0000","last_active":"2024-09-15T23:22:02.633089+0000","last_peered":"2024-09-15T23:22:02.633089+0000","last_clean":"2024-09-15T23:22:02.633089+0000","last_became_active":"2024-09-15T23:21:59.567693+0000","last_became_peered":"2024-09-15T23:21:59.567693+0000","last_unstale":"2024-09-15T23:22:02.633089+0000","last_undegraded":"2024-09-15T23:22:02.633089+0000","last_fullsized":"2024-09-15T23:22:02.633089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.231873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.578860+0000","last_change":"2024-09-15T23:21:52.531090+0000","last_active":"2024-09-15T23:22:02.578860+0000","last_peered":"2024-09-15T23:22:02.578860+0000","last_clean":"2024-09-15T23:22:02.578860+0000","last_became_active":"2024-09-15T23:21:52.530738+0000","last_became_peered":"2024-09-15T23:21:52.530738+0000","last_unstale":"2024-09-15T23:22:02.578860+0000","last_undegraded":"2024-09-15T23:22:02.578860+0000","last_fullsized":"2024-09-15T23:22:02.578860+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:35.325128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.135607+0000","last_change":"2024-09-15T23:21:50.514100+0000","last_active":"2024-09-15T23:22:36.135607+0000","last_peered":"2024-09-15T23:22:36.135607+0000","last_clean":"2024-09-15T23:22:36.135607+0000","last_became_active":"2024-09-15T23:21:50.513629+0000","last_became_peered":"2024-09-15T23:21:50.513629+0000","last_unstale":"2024-09-15T23:22:36.135607+0000","last_undegraded":"2024-09-15T23:22:36.135607+0000","last_fullsized":"2024-09-15T23:22:36.135607+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.187544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633409+0000","last_change":"2024-09-15T23:21:59.562804+0000","last_active":"2024-09-15T23:22:02.633409+0000","last_peered":"2024-09-15T23:22:02.633409+0000","last_clean":"2024-09-15T23:22:02.633409+0000","last_became_active":"2024-09-15T23:21:59.562368+0000","last_became_peered":"2024-09-15T23:21:59.562368+0000","last_unstale":"2024-09-15T23:22:02.633409+0000","last_undegraded":"2024-09-15T23:22:02.633409+0000","last_fullsized":"2024-09-15T23:22:02.633409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:21.137333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632598+0000","last_change":"2024-09-15T23:21:52.530562+0000","last_active":"2024-09-15T23:22:02.632598+0000","last_peered":"2024-09-15T23:22:02.632598+0000","last_clean":"2024-09-15T23:22:02.632598+0000","last_became_active":"2024-09-15T23:21:52.530280+0000","last_became_peered":"2024-09-15T23:21:52.530280+0000","last_unstale":"2024-09-15T23:22:02.632598+0000","last_undegraded":"2024-09-15T23:22:02.632598+0000","last_fullsized":"2024-09-15T23:22:02.632598+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:26.835651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:37.137418+0000","last_change":"2024-09-15T23:21:50.513179+0000","last_active":"2024-09-15T23:22:37.137418+0000","last_peered":"2024-09-15T23:22:37.137418+0000","last_clean":"2024-09-15T23:22:37.137418+0000","last_became_active":"2024-09-15T23:21:50.512790+0000","last_became_peered":"2024-09-15T23:21:50.512790+0000","last_unstale":"2024-09-15T23:22:37.137418+0000","last_undegraded":"2024-09-15T23:22:37.137418+0000","last_fullsized":"2024-09-15T23:22:37.137418+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:56.408198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634066+0000","last_change":"2024-09-15T23:21:59.567895+0000","last_active":"2024-09-15T23:22:02.634066+0000","last_peered":"2024-09-15T23:22:02.634066+0000","last_clean":"2024-09-15T23:22:02.634066+0000","last_became_active":"2024-09-15T23:21:59.567500+0000","last_became_peered":"2024-09-15T23:21:59.567500+0000","last_unstale":"2024-09-15T23:22:02.634066+0000","last_undegraded":"2024-09-15T23:22:02.634066+0000","last_fullsized":"2024-09-15T23:22:02.634066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:09:41.526437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.577102+0000","last_change":"2024-09-15T23:21:52.530970+0000","last_active":"2024-09-15T23:22:02.577102+0000","last_peered":"2024-09-15T23:22:02.577102+0000","last_clean":"2024-09-15T23:22:02.577102+0000","last_became_active":"2024-09-15T23:21:52.530636+0000","last_became_peered":"2024-09-15T23:21:52.530636+0000","last_unstale":"2024-09-15T23:22:02.577102+0000","last_undegraded":"2024-09-15T23:22:02.577102+0000","last_fullsized":"2024-09-15T23:22:02.577102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:33.374658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"86'27","reported_seq":81,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:45.155953+0000","last_change":"2024-09-15T23:21:50.518449+0000","last_active":"2024-09-15T23:22:45.155953+0000","last_peered":"2024-09-15T23:22:45.155953+0000","last_clean":"2024-09-15T23:22:45.155953+0000","last_became_active":"2024-09-15T23:21:50.518099+0000","last_became_peered":"2024-09-15T23:21:50.518099+0000","last_unstale":"2024-09-15T23:22:45.155953+0000","last_undegraded":"2024-09-15T23:22:45.155953+0000","last_fullsized":"2024-09-15T23:22:45.155953+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:43.792971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43345,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":54,"num_write":29,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632825+0000","last_change":"2024-09-15T23:21:59.564939+0000","last_active":"2024-09-15T23:22:02.632825+0000","last_peered":"2024-09-15T23:22:02.632825+0000","last_clean":"2024-09-15T23:22:02.632825+0000","last_became_active":"2024-09-15T23:21:59.564484+0000","last_became_peered":"2024-09-15T23:21:59.564484+0000","last_unstale":"2024-09-15T23:22:02.632825+0000","last_undegraded":"2024-09-15T23:22:02.632825+0000","last_fullsized":"2024-09-15T23:22:02.632825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:10:46.585306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632385+0000","last_change":"2024-09-15T23:21:52.530541+0000","last_active":"2024-09-15T23:22:02.632385+0000","last_peered":"2024-09-15T23:22:02.632385+0000","last_clean":"2024-09-15T23:22:02.632385+0000","last_became_active":"2024-09-15T23:21:52.530217+0000","last_became_peered":"2024-09-15T23:21:52.530217+0000","last_unstale":"2024-09-15T23:22:02.632385+0000","last_undegraded":"2024-09-15T23:22:02.632385+0000","last_fullsized":"2024-09-15T23:22:02.632385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:14.864563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:35.120742+0000","last_change":"2024-09-15T23:21:50.562203+0000","last_active":"2024-09-15T23:22:35.120742+0000","last_peered":"2024-09-15T23:22:35.120742+0000","last_clean":"2024-09-15T23:22:35.120742+0000","last_became_active":"2024-09-15T23:21:50.561827+0000","last_became_peered":"2024-09-15T23:21:50.561827+0000","last_unstale":"2024-09-15T23:22:35.120742+0000","last_undegraded":"2024-09-15T23:22:35.120742+0000","last_fullsized":"2024-09-15T23:22:35.120742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:22.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632891+0000","last_change":"2024-09-15T23:21:59.561013+0000","last_active":"2024-09-15T23:22:02.632891+0000","last_peered":"2024-09-15T23:22:02.632891+0000","last_clean":"2024-09-15T23:22:02.632891+0000","last_became_active":"2024-09-15T23:21:59.560665+0000","last_became_peered":"2024-09-15T23:21:59.560665+0000","last_unstale":"2024-09-15T23:22:02.632891+0000","last_undegraded":"2024-09-15T23:22:02.632891+0000","last_fullsized":"2024-09-15T23:22:02.632891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:00.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.923197+0000","last_change":"2024-09-15T23:21:52.529922+0000","last_active":"2024-09-15T23:22:02.923197+0000","last_peered":"2024-09-15T23:22:02.923197+0000","last_clean":"2024-09-15T23:22:02.923197+0000","last_became_active":"2024-09-15T23:21:52.529179+0000","last_became_peered":"2024-09-15T23:21:52.529179+0000","last_unstale":"2024-09-15T23:22:02.923197+0000","last_undegraded":"2024-09-15T23:22:02.923197+0000","last_fullsized":"2024-09-15T23:22:02.923197+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.077658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.135438+0000","last_change":"2024-09-15T23:21:50.514420+0000","last_active":"2024-09-15T23:22:36.135438+0000","last_peered":"2024-09-15T23:22:36.135438+0000","last_clean":"2024-09-15T23:22:36.135438+0000","last_became_active":"2024-09-15T23:21:50.514007+0000","last_became_peered":"2024-09-15T23:21:50.514007+0000","last_unstale":"2024-09-15T23:22:36.135438+0000","last_undegraded":"2024-09-15T23:22:36.135438+0000","last_fullsized":"2024-09-15T23:22:36.135438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:36.845674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632389+0000","last_change":"2024-09-15T23:21:59.565978+0000","last_active":"2024-09-15T23:22:02.632389+0000","last_peered":"2024-09-15T23:22:02.632389+0000","last_clean":"2024-09-15T23:22:02.632389+0000","last_became_active":"2024-09-15T23:21:59.565604+0000","last_became_peered":"2024-09-15T23:21:59.565604+0000","last_unstale":"2024-09-15T23:22:02.632389+0000","last_undegraded":"2024-09-15T23:22:02.632389+0000","last_fullsized":"2024-09-15T23:22:02.632389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:47.089504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632772+0000","last_change":"2024-09-15T23:21:52.543511+0000","last_active":"2024-09-15T23:22:02.632772+0000","last_peered":"2024-09-15T23:22:02.632772+0000","last_clean":"2024-09-15T23:22:02.632772+0000","last_became_active":"2024-09-15T23:21:52.543169+0000","last_became_peered":"2024-09-15T23:21:52.543169+0000","last_unstale":"2024-09-15T23:22:02.632772+0000","last_undegraded":"2024-09-15T23:22:02.632772+0000","last_fullsized":"2024-09-15T23:22:02.632772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:05.696772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":52,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:45.152204+0000","last_change":"2024-09-15T23:21:50.518425+0000","last_active":"2024-09-15T23:22:45.152204+0000","last_peered":"2024-09-15T23:22:45.152204+0000","last_clean":"2024-09-15T23:22:45.152204+0000","last_became_active":"2024-09-15T23:21:50.518052+0000","last_became_peered":"2024-09-15T23:21:50.518052+0000","last_unstale":"2024-09-15T23:22:45.152204+0000","last_undegraded":"2024-09-15T23:22:45.152204+0000","last_fullsized":"2024-09-15T23:22:45.152204+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:02.243869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.575942+0000","last_change":"2024-09-15T23:21:59.570310+0000","last_active":"2024-09-15T23:22:02.575942+0000","last_peered":"2024-09-15T23:22:02.575942+0000","last_clean":"2024-09-15T23:22:02.575942+0000","last_became_active":"2024-09-15T23:21:59.569909+0000","last_became_peered":"2024-09-15T23:21:59.569909+0000","last_unstale":"2024-09-15T23:22:02.575942+0000","last_undegraded":"2024-09-15T23:22:02.575942+0000","last_fullsized":"2024-09-15T23:22:02.575942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:20:27.515537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.923015+0000","last_change":"2024-09-15T23:21:59.568208+0000","last_active":"2024-09-15T23:22:02.923015+0000","last_peered":"2024-09-15T23:22:02.923015+0000","last_clean":"2024-09-15T23:22:02.923015+0000","last_became_active":"2024-09-15T23:21:59.567858+0000","last_became_peered":"2024-09-15T23:21:59.567858+0000","last_unstale":"2024-09-15T23:22:02.923015+0000","last_undegraded":"2024-09-15T23:22:02.923015+0000","last_fullsized":"2024-09-15T23:22:02.923015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:34:42.895247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632517+0000","last_change":"2024-09-15T23:21:52.532679+0000","last_active":"2024-09-15T23:22:02.632517+0000","last_peered":"2024-09-15T23:22:02.632517+0000","last_clean":"2024-09-15T23:22:02.632517+0000","last_became_active":"2024-09-15T23:21:52.532345+0000","last_became_peered":"2024-09-15T23:21:52.532345+0000","last_unstale":"2024-09-15T23:22:02.632517+0000","last_undegraded":"2024-09-15T23:22:02.632517+0000","last_fullsized":"2024-09-15T23:22:02.632517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:45.943653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.135672+0000","last_change":"2024-09-15T23:21:50.510836+0000","last_active":"2024-09-15T23:22:36.135672+0000","last_peered":"2024-09-15T23:22:36.135672+0000","last_clean":"2024-09-15T23:22:36.135672+0000","last_became_active":"2024-09-15T23:21:50.510514+0000","last_became_peered":"2024-09-15T23:21:50.510514+0000","last_unstale":"2024-09-15T23:22:36.135672+0000","last_undegraded":"2024-09-15T23:22:36.135672+0000","last_fullsized":"2024-09-15T23:22:36.135672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:06.404612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633774+0000","last_change":"2024-09-15T23:21:59.568286+0000","last_active":"2024-09-15T23:22:02.633774+0000","last_peered":"2024-09-15T23:22:02.633774+0000","last_clean":"2024-09-15T23:22:02.633774+0000","last_became_active":"2024-09-15T23:21:59.567785+0000","last_became_peered":"2024-09-15T23:21:59.567785+0000","last_unstale":"2024-09-15T23:22:02.633774+0000","last_undegraded":"2024-09-15T23:22:02.633774+0000","last_fullsized":"2024-09-15T23:22:02.633774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:04.491823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.575768+0000","last_change":"2024-09-15T23:21:52.526766+0000","last_active":"2024-09-15T23:22:02.575768+0000","last_peered":"2024-09-15T23:22:02.575768+0000","last_clean":"2024-09-15T23:22:02.575768+0000","last_became_active":"2024-09-15T23:21:52.526446+0000","last_became_peered":"2024-09-15T23:21:52.526446+0000","last_unstale":"2024-09-15T23:22:02.575768+0000","last_undegraded":"2024-09-15T23:22:02.575768+0000","last_fullsized":"2024-09-15T23:22:02.575768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:21:37.004730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:34.103702+0000","last_change":"2024-09-15T23:21:50.512928+0000","last_active":"2024-09-15T23:22:34.103702+0000","last_peered":"2024-09-15T23:22:34.103702+0000","last_clean":"2024-09-15T23:22:34.103702+0000","last_became_active":"2024-09-15T23:21:50.512558+0000","last_became_peered":"2024-09-15T23:21:50.512558+0000","last_unstale":"2024-09-15T23:22:34.103702+0000","last_undegraded":"2024-09-15T23:22:34.103702+0000","last_fullsized":"2024-09-15T23:22:34.103702+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:08.653084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:35.119796+0000","last_change":"2024-09-15T23:21:50.516920+0000","last_active":"2024-09-15T23:22:35.119796+0000","last_peered":"2024-09-15T23:22:35.119796+0000","last_clean":"2024-09-15T23:22:35.119796+0000","last_became_active":"2024-09-15T23:21:50.516573+0000","last_became_peered":"2024-09-15T23:21:50.516573+0000","last_unstale":"2024-09-15T23:22:35.119796+0000","last_undegraded":"2024-09-15T23:22:35.119796+0000","last_fullsized":"2024-09-15T23:22:35.119796+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.367903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633182+0000","last_change":"2024-09-15T23:21:52.536696+0000","last_active":"2024-09-15T23:22:02.633182+0000","last_peered":"2024-09-15T23:22:02.633182+0000","last_clean":"2024-09-15T23:22:02.633182+0000","last_became_active":"2024-09-15T23:21:52.536338+0000","last_became_peered":"2024-09-15T23:21:52.536338+0000","last_unstale":"2024-09-15T23:22:02.633182+0000","last_undegraded":"2024-09-15T23:22:02.633182+0000","last_fullsized":"2024-09-15T23:22:02.633182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:05.152749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634559+0000","last_change":"2024-09-15T23:21:59.562968+0000","last_active":"2024-09-15T23:22:02.634559+0000","last_peered":"2024-09-15T23:22:02.634559+0000","last_clean":"2024-09-15T23:22:02.634559+0000","last_became_active":"2024-09-15T23:21:59.562579+0000","last_became_peered":"2024-09-15T23:21:59.562579+0000","last_unstale":"2024-09-15T23:22:02.634559+0000","last_undegraded":"2024-09-15T23:22:02.634559+0000","last_fullsized":"2024-09-15T23:22:02.634559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:02.853057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:45.152988+0000","last_change":"2024-09-15T23:21:50.513944+0000","last_active":"2024-09-15T23:22:45.152988+0000","last_peered":"2024-09-15T23:22:45.152988+0000","last_clean":"2024-09-15T23:22:45.152988+0000","last_became_active":"2024-09-15T23:21:50.513484+0000","last_became_peered":"2024-09-15T23:21:50.513484+0000","last_unstale":"2024-09-15T23:22:45.152988+0000","last_undegraded":"2024-09-15T23:22:45.152988+0000","last_fullsized":"2024-09-15T23:22:45.152988+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:35.719344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576737+0000","last_change":"2024-09-15T23:21:52.539467+0000","last_active":"2024-09-15T23:22:02.576737+0000","last_peered":"2024-09-15T23:22:02.576737+0000","last_clean":"2024-09-15T23:22:02.576737+0000","last_became_active":"2024-09-15T23:21:52.538998+0000","last_became_peered":"2024-09-15T23:21:52.538998+0000","last_unstale":"2024-09-15T23:22:02.576737+0000","last_undegraded":"2024-09-15T23:22:02.576737+0000","last_fullsized":"2024-09-15T23:22:02.576737+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:22.213857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634927+0000","last_change":"2024-09-15T23:21:59.564842+0000","last_active":"2024-09-15T23:22:02.634927+0000","last_peered":"2024-09-15T23:22:02.634927+0000","last_clean":"2024-09-15T23:22:02.634927+0000","last_became_active":"2024-09-15T23:21:59.564408+0000","last_became_peered":"2024-09-15T23:21:59.564408+0000","last_unstale":"2024-09-15T23:22:02.634927+0000","last_undegraded":"2024-09-15T23:22:02.634927+0000","last_fullsized":"2024-09-15T23:22:02.634927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:17.909020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632850+0000","last_change":"2024-09-15T23:21:52.539731+0000","last_active":"2024-09-15T23:22:02.632850+0000","last_peered":"2024-09-15T23:22:02.632850+0000","last_clean":"2024-09-15T23:22:02.632850+0000","last_became_active":"2024-09-15T23:21:52.539414+0000","last_became_peered":"2024-09-15T23:21:52.539414+0000","last_unstale":"2024-09-15T23:22:02.632850+0000","last_undegraded":"2024-09-15T23:22:02.632850+0000","last_fullsized":"2024-09-15T23:22:02.632850+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:49.660931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634836+0000","last_change":"2024-09-15T23:21:52.532769+0000","last_active":"2024-09-15T23:22:02.634836+0000","last_peered":"2024-09-15T23:22:02.634836+0000","last_clean":"2024-09-15T23:22:02.634836+0000","last_became_active":"2024-09-15T23:21:52.532426+0000","last_became_peered":"2024-09-15T23:21:52.532426+0000","last_unstale":"2024-09-15T23:22:02.634836+0000","last_undegraded":"2024-09-15T23:22:02.634836+0000","last_fullsized":"2024-09-15T23:22:02.634836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:09.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634291+0000","last_change":"2024-09-15T23:21:52.533148+0000","last_active":"2024-09-15T23:22:02.634291+0000","last_peered":"2024-09-15T23:22:02.634291+0000","last_clean":"2024-09-15T23:22:02.634291+0000","last_became_active":"2024-09-15T23:21:52.532823+0000","last_became_peered":"2024-09-15T23:21:52.532823+0000","last_unstale":"2024-09-15T23:22:02.634291+0000","last_undegraded":"2024-09-15T23:22:02.634291+0000","last_fullsized":"2024-09-15T23:22:02.634291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:01.833945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922805+0000","last_change":"2024-09-15T23:21:52.536790+0000","last_active":"2024-09-15T23:22:02.922805+0000","last_peered":"2024-09-15T23:22:02.922805+0000","last_clean":"2024-09-15T23:22:02.922805+0000","last_became_active":"2024-09-15T23:21:52.536515+0000","last_became_peered":"2024-09-15T23:21:52.536515+0000","last_unstale":"2024-09-15T23:22:02.922805+0000","last_undegraded":"2024-09-15T23:22:02.922805+0000","last_fullsized":"2024-09-15T23:22:02.922805+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:37.207759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632499+0000","last_change":"2024-09-15T23:21:52.530181+0000","last_active":"2024-09-15T23:22:02.632499+0000","last_peered":"2024-09-15T23:22:02.632499+0000","last_clean":"2024-09-15T23:22:02.632499+0000","last_became_active":"2024-09-15T23:21:52.529790+0000","last_became_peered":"2024-09-15T23:21:52.529790+0000","last_unstale":"2024-09-15T23:22:02.632499+0000","last_undegraded":"2024-09-15T23:22:02.632499+0000","last_fullsized":"2024-09-15T23:22:02.632499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:54:02.579290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633603+0000","last_change":"2024-09-15T23:21:52.540628+0000","last_active":"2024-09-15T23:22:02.633603+0000","last_peered":"2024-09-15T23:22:02.633603+0000","last_clean":"2024-09-15T23:22:02.633603+0000","last_became_active":"2024-09-15T23:21:52.540308+0000","last_became_peered":"2024-09-15T23:21:52.540308+0000","last_unstale":"2024-09-15T23:22:02.633603+0000","last_undegraded":"2024-09-15T23:22:02.633603+0000","last_fullsized":"2024-09-15T23:22:02.633603+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:16.660637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634688+0000","last_change":"2024-09-15T23:21:52.533780+0000","last_active":"2024-09-15T23:22:02.634688+0000","last_peered":"2024-09-15T23:22:02.634688+0000","last_clean":"2024-09-15T23:22:02.634688+0000","last_became_active":"2024-09-15T23:21:52.533354+0000","last_became_peered":"2024-09-15T23:21:52.533354+0000","last_unstale":"2024-09-15T23:22:02.634688+0000","last_undegraded":"2024-09-15T23:22:02.634688+0000","last_fullsized":"2024-09-15T23:22:02.634688+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.381059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576254+0000","last_change":"2024-09-15T23:21:52.536406+0000","last_active":"2024-09-15T23:22:02.576254+0000","last_peered":"2024-09-15T23:22:02.576254+0000","last_clean":"2024-09-15T23:22:02.576254+0000","last_became_active":"2024-09-15T23:21:52.536091+0000","last_became_peered":"2024-09-15T23:21:52.536091+0000","last_unstale":"2024-09-15T23:22:02.576254+0000","last_undegraded":"2024-09-15T23:22:02.576254+0000","last_fullsized":"2024-09-15T23:22:02.576254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:30.452201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632813+0000","last_change":"2024-09-15T23:21:52.536333+0000","last_active":"2024-09-15T23:22:02.632813+0000","last_peered":"2024-09-15T23:22:02.632813+0000","last_clean":"2024-09-15T23:22:02.632813+0000","last_became_active":"2024-09-15T23:21:52.534677+0000","last_became_peered":"2024-09-15T23:21:52.534677+0000","last_unstale":"2024-09-15T23:22:02.632813+0000","last_undegraded":"2024-09-15T23:22:02.632813+0000","last_fullsized":"2024-09-15T23:22:02.632813+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:05.287914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576070+0000","last_change":"2024-09-15T23:21:52.536442+0000","last_active":"2024-09-15T23:22:02.576070+0000","last_peered":"2024-09-15T23:22:02.576070+0000","last_clean":"2024-09-15T23:22:02.576070+0000","last_became_active":"2024-09-15T23:21:52.536183+0000","last_became_peered":"2024-09-15T23:21:52.536183+0000","last_unstale":"2024-09-15T23:22:02.576070+0000","last_undegraded":"2024-09-15T23:22:02.576070+0000","last_fullsized":"2024-09-15T23:22:02.576070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:41.641370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632778+0000","last_change":"2024-09-15T23:21:52.540106+0000","last_active":"2024-09-15T23:22:02.632778+0000","last_peered":"2024-09-15T23:22:02.632778+0000","last_clean":"2024-09-15T23:22:02.632778+0000","last_became_active":"2024-09-15T23:21:52.539777+0000","last_became_peered":"2024-09-15T23:21:52.539777+0000","last_unstale":"2024-09-15T23:22:02.632778+0000","last_undegraded":"2024-09-15T23:22:02.632778+0000","last_fullsized":"2024-09-15T23:22:02.632778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:47:01.691391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632365+0000","last_change":"2024-09-15T23:21:52.531171+0000","last_active":"2024-09-15T23:22:02.632365+0000","last_peered":"2024-09-15T23:22:02.632365+0000","last_clean":"2024-09-15T23:22:02.632365+0000","last_became_active":"2024-09-15T23:21:52.530724+0000","last_became_peered":"2024-09-15T23:21:52.530724+0000","last_unstale":"2024-09-15T23:22:02.632365+0000","last_undegraded":"2024-09-15T23:22:02.632365+0000","last_fullsized":"2024-09-15T23:22:02.632365+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:11.009119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633517+0000","last_change":"2024-09-15T23:21:52.530433+0000","last_active":"2024-09-15T23:22:02.633517+0000","last_peered":"2024-09-15T23:22:02.633517+0000","last_clean":"2024-09-15T23:22:02.633517+0000","last_became_active":"2024-09-15T23:21:52.530058+0000","last_became_peered":"2024-09-15T23:21:52.530058+0000","last_unstale":"2024-09-15T23:22:02.633517+0000","last_undegraded":"2024-09-15T23:22:02.633517+0000","last_fullsized":"2024-09-15T23:22:02.633517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:51.208820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634695+0000","last_change":"2024-09-15T23:21:52.534639+0000","last_active":"2024-09-15T23:22:02.634695+0000","last_peered":"2024-09-15T23:22:02.634695+0000","last_clean":"2024-09-15T23:22:02.634695+0000","last_became_active":"2024-09-15T23:21:52.534362+0000","last_became_peered":"2024-09-15T23:21:52.534362+0000","last_unstale":"2024-09-15T23:22:02.634695+0000","last_undegraded":"2024-09-15T23:22:02.634695+0000","last_fullsized":"2024-09-15T23:22:02.634695+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:58:12.023501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633033+0000","last_change":"2024-09-15T23:21:52.532758+0000","last_active":"2024-09-15T23:22:02.633033+0000","last_peered":"2024-09-15T23:22:02.633033+0000","last_clean":"2024-09-15T23:22:02.633033+0000","last_became_active":"2024-09-15T23:21:52.532412+0000","last_became_peered":"2024-09-15T23:21:52.532412+0000","last_unstale":"2024-09-15T23:22:02.633033+0000","last_undegraded":"2024-09-15T23:22:02.633033+0000","last_fullsized":"2024-09-15T23:22:02.633033+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:46.285711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633931+0000","last_change":"2024-09-15T23:21:52.533928+0000","last_active":"2024-09-15T23:22:02.633931+0000","last_peered":"2024-09-15T23:22:02.633931+0000","last_clean":"2024-09-15T23:22:02.633931+0000","last_became_active":"2024-09-15T23:21:52.533578+0000","last_became_peered":"2024-09-15T23:21:52.533578+0000","last_unstale":"2024-09-15T23:22:02.633931+0000","last_undegraded":"2024-09-15T23:22:02.633931+0000","last_fullsized":"2024-09-15T23:22:02.633931+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:49.476316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.631981+0000","last_change":"2024-09-15T23:21:52.533417+0000","last_active":"2024-09-15T23:22:02.631981+0000","last_peered":"2024-09-15T23:22:02.631981+0000","last_clean":"2024-09-15T23:22:02.631981+0000","last_became_active":"2024-09-15T23:21:52.533085+0000","last_became_peered":"2024-09-15T23:21:52.533085+0000","last_unstale":"2024-09-15T23:22:02.631981+0000","last_undegraded":"2024-09-15T23:22:02.631981+0000","last_fullsized":"2024-09-15T23:22:02.631981+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:20.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922732+0000","last_change":"2024-09-15T23:21:52.542340+0000","last_active":"2024-09-15T23:22:02.922732+0000","last_peered":"2024-09-15T23:22:02.922732+0000","last_clean":"2024-09-15T23:22:02.922732+0000","last_became_active":"2024-09-15T23:21:52.542048+0000","last_became_peered":"2024-09-15T23:21:52.542048+0000","last_unstale":"2024-09-15T23:22:02.922732+0000","last_undegraded":"2024-09-15T23:22:02.922732+0000","last_fullsized":"2024-09-15T23:22:02.922732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:57.557420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922634+0000","last_change":"2024-09-15T23:21:52.535576+0000","last_active":"2024-09-15T23:22:02.922634+0000","last_peered":"2024-09-15T23:22:02.922634+0000","last_clean":"2024-09-15T23:22:02.922634+0000","last_became_active":"2024-09-15T23:21:52.535223+0000","last_became_peered":"2024-09-15T23:21:52.535223+0000","last_unstale":"2024-09-15T23:22:02.922634+0000","last_undegraded":"2024-09-15T23:22:02.922634+0000","last_fullsized":"2024-09-15T23:22:02.922634+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:45.874927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632336+0000","last_change":"2024-09-15T23:21:52.519339+0000","last_active":"2024-09-15T23:22:02.632336+0000","last_peered":"2024-09-15T23:22:02.632336+0000","last_clean":"2024-09-15T23:22:02.632336+0000","last_became_active":"2024-09-15T23:21:52.518897+0000","last_became_peered":"2024-09-15T23:21:52.518897+0000","last_unstale":"2024-09-15T23:22:02.632336+0000","last_undegraded":"2024-09-15T23:22:02.632336+0000","last_fullsized":"2024-09-15T23:22:02.632336+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:30:50.169838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633340+0000","last_change":"2024-09-15T23:21:52.537012+0000","last_active":"2024-09-15T23:22:02.633340+0000","last_peered":"2024-09-15T23:22:02.633340+0000","last_clean":"2024-09-15T23:22:02.633340+0000","last_became_active":"2024-09-15T23:21:52.536245+0000","last_became_peered":"2024-09-15T23:21:52.536245+0000","last_unstale":"2024-09-15T23:22:02.633340+0000","last_undegraded":"2024-09-15T23:22:02.633340+0000","last_fullsized":"2024-09-15T23:22:02.633340+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:31.150491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.579327+0000","last_change":"2024-09-15T23:21:52.540431+0000","last_active":"2024-09-15T23:22:02.579327+0000","last_peered":"2024-09-15T23:22:02.579327+0000","last_clean":"2024-09-15T23:22:02.579327+0000","last_became_active":"2024-09-15T23:21:52.540104+0000","last_became_peered":"2024-09-15T23:21:52.540104+0000","last_unstale":"2024-09-15T23:22:02.579327+0000","last_undegraded":"2024-09-15T23:22:02.579327+0000","last_fullsized":"2024-09-15T23:22:02.579327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.550258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633402+0000","last_change":"2024-09-15T23:21:52.537590+0000","last_active":"2024-09-15T23:22:02.633402+0000","last_peered":"2024-09-15T23:22:02.633402+0000","last_clean":"2024-09-15T23:22:02.633402+0000","last_became_active":"2024-09-15T23:21:52.533804+0000","last_became_peered":"2024-09-15T23:21:52.533804+0000","last_unstale":"2024-09-15T23:22:02.633402+0000","last_undegraded":"2024-09-15T23:22:02.633402+0000","last_fullsized":"2024-09-15T23:22:02.633402+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:48.449090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632751+0000","last_change":"2024-09-15T23:21:52.536130+0000","last_active":"2024-09-15T23:22:02.632751+0000","last_peered":"2024-09-15T23:22:02.632751+0000","last_clean":"2024-09-15T23:22:02.632751+0000","last_became_active":"2024-09-15T23:21:52.535336+0000","last_became_peered":"2024-09-15T23:21:52.535336+0000","last_unstale":"2024-09-15T23:22:02.632751+0000","last_undegraded":"2024-09-15T23:22:02.632751+0000","last_fullsized":"2024-09-15T23:22:02.632751+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:52.581994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.575723+0000","last_change":"2024-09-15T23:21:52.540485+0000","last_active":"2024-09-15T23:22:02.575723+0000","last_peered":"2024-09-15T23:22:02.575723+0000","last_clean":"2024-09-15T23:22:02.575723+0000","last_became_active":"2024-09-15T23:21:52.540178+0000","last_became_peered":"2024-09-15T23:21:52.540178+0000","last_unstale":"2024-09-15T23:22:02.575723+0000","last_undegraded":"2024-09-15T23:22:02.575723+0000","last_fullsized":"2024-09-15T23:22:02.575723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:40:22.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.578696+0000","last_change":"2024-09-15T23:21:52.538845+0000","last_active":"2024-09-15T23:22:02.578696+0000","last_peered":"2024-09-15T23:22:02.578696+0000","last_clean":"2024-09-15T23:22:02.578696+0000","last_became_active":"2024-09-15T23:21:52.538511+0000","last_became_peered":"2024-09-15T23:21:52.538511+0000","last_unstale":"2024-09-15T23:22:02.578696+0000","last_undegraded":"2024-09-15T23:22:02.578696+0000","last_fullsized":"2024-09-15T23:22:02.578696+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:00.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633869+0000","last_change":"2024-09-15T23:21:52.533707+0000","last_active":"2024-09-15T23:22:02.633869+0000","last_peered":"2024-09-15T23:22:02.633869+0000","last_clean":"2024-09-15T23:22:02.633869+0000","last_became_active":"2024-09-15T23:21:52.533387+0000","last_became_peered":"2024-09-15T23:21:52.533387+0000","last_unstale":"2024-09-15T23:22:02.633869+0000","last_undegraded":"2024-09-15T23:22:02.633869+0000","last_fullsized":"2024-09-15T23:22:02.633869+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:50.585181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576690+0000","last_change":"2024-09-15T23:21:52.530524+0000","last_active":"2024-09-15T23:22:02.576690+0000","last_peered":"2024-09-15T23:22:02.576690+0000","last_clean":"2024-09-15T23:22:02.576690+0000","last_became_active":"2024-09-15T23:21:52.530203+0000","last_became_peered":"2024-09-15T23:21:52.530203+0000","last_unstale":"2024-09-15T23:22:02.576690+0000","last_undegraded":"2024-09-15T23:22:02.576690+0000","last_fullsized":"2024-09-15T23:22:02.576690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:50.560211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633222+0000","last_change":"2024-09-15T23:21:52.537256+0000","last_active":"2024-09-15T23:22:02.633222+0000","last_peered":"2024-09-15T23:22:02.633222+0000","last_clean":"2024-09-15T23:22:02.633222+0000","last_became_active":"2024-09-15T23:21:52.536731+0000","last_became_peered":"2024-09-15T23:21:52.536731+0000","last_unstale":"2024-09-15T23:22:02.633222+0000","last_undegraded":"2024-09-15T23:22:02.633222+0000","last_fullsized":"2024-09-15T23:22:02.633222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:18.815398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922847+0000","last_change":"2024-09-15T23:21:52.537728+0000","last_active":"2024-09-15T23:22:02.922847+0000","last_peered":"2024-09-15T23:22:02.922847+0000","last_clean":"2024-09-15T23:22:02.922847+0000","last_became_active":"2024-09-15T23:21:52.537434+0000","last_became_peered":"2024-09-15T23:21:52.537434+0000","last_unstale":"2024-09-15T23:22:02.922847+0000","last_undegraded":"2024-09-15T23:22:02.922847+0000","last_fullsized":"2024-09-15T23:22:02.922847+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:25.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922992+0000","last_change":"2024-09-15T23:21:52.534780+0000","last_active":"2024-09-15T23:22:02.922992+0000","last_peered":"2024-09-15T23:22:02.922992+0000","last_clean":"2024-09-15T23:22:02.922992+0000","last_became_active":"2024-09-15T23:21:52.534399+0000","last_became_peered":"2024-09-15T23:21:52.534399+0000","last_unstale":"2024-09-15T23:22:02.922992+0000","last_undegraded":"2024-09-15T23:22:02.922992+0000","last_fullsized":"2024-09-15T23:22:02.922992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:47.344129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633912+0000","last_change":"2024-09-15T23:21:52.533403+0000","last_active":"2024-09-15T23:22:02.633912+0000","last_peered":"2024-09-15T23:22:02.633912+0000","last_clean":"2024-09-15T23:22:02.633912+0000","last_became_active":"2024-09-15T23:21:52.533069+0000","last_became_peered":"2024-09-15T23:21:52.533069+0000","last_unstale":"2024-09-15T23:22:02.633912+0000","last_undegraded":"2024-09-15T23:22:02.633912+0000","last_fullsized":"2024-09-15T23:22:02.633912+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:38.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":50160,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":140,"num_read_kb":123,"num_write":233,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":177136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":246033,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579928,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":482288,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841564,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1112842,"data_compressed":53449,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005090,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":486314,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234023,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":482596,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462956,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":1079095,"data_compressed":53449,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626481,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":477722,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855414,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":1074284,"data_compressed":53449,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018940,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":482189,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247872,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":477430,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444102,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":535645,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607627,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":535425,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":479178,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":58425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":45474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:22:51.781 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:22:52.051 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:52.322 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:52 smithi089 ceph-mon[30933]: pgmap v311: 129 pgs: 129 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 KiB/s wr, 15 op/s 2024-09-15T23:22:52.322 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:52 smithi089 ceph-mon[30933]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:22:52.322 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:52 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:52.322 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:52 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:52 smithi120 ceph-mon[35250]: pgmap v311: 129 pgs: 129 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 KiB/s wr, 15 op/s 2024-09-15T23:22:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:52 smithi120 ceph-mon[35250]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:22:52.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:52 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:52.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:52 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:52 smithi174 ceph-mon[35586]: pgmap v311: 129 pgs: 129 active+clean; 626 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 KiB/s wr, 15 op/s 2024-09-15T23:22:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:52 smithi174 ceph-mon[35586]: from='client.14844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:22:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:52 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:52.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:52 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:52.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.556+0000 7f496e920640 1 -- 172.21.15.89:0/756436589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968101420 msgr2=0x7f496810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:52.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.556+0000 7f496e920640 1 --2- 172.21.15.89:0/756436589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968101420 0x7f496810d950 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4960009cd0 tx=0x7f496002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:22:52.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.556+0000 7f496e920640 1 -- 172.21.15.89:0/756436589 shutdown_connections 2024-09-15T23:22:52.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.556+0000 7f496e920640 1 --2- 172.21.15.89:0/756436589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968101420 0x7f496810d950 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.556+0000 7f496e920640 1 --2- 172.21.15.89:0/756436589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968100a80 0x7f4968100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.556+0000 7f496e920640 1 --2- 172.21.15.89:0/756436589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4968106a80 0x7f4968106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.556+0000 7f496e920640 1 -- 172.21.15.89:0/756436589 >> 172.21.15.89:0/756436589 conn(0x7f49680fc7a0 msgr2=0x7f49680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.556+0000 7f496e920640 1 -- 172.21.15.89:0/756436589 shutdown_connections 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.556+0000 7f496e920640 1 -- 172.21.15.89:0/756436589 wait complete. 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.557+0000 7f496e920640 1 Processor -- start 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.557+0000 7f496e920640 1 -- start start 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.557+0000 7f496e920640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968100a80 0x7f49681983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.557+0000 7f496e920640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4968101420 0x7f4968198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.557+0000 7f496e920640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106a80 0x7f496819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.557+0000 7f496e920640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4968110e80 con 0x7f4968106a80 2024-09-15T23:22:52.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.557+0000 7f496e920640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4968110d00 con 0x7f4968101420 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.557+0000 7f496e920640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4968111000 con 0x7f4968100a80 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496cebd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106a80 0x7f496819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496cebd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106a80 0x7f496819ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58426/0 (socket says 172.21.15.89:58426) 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496cebd640 1 -- 172.21.15.89:0/2437776814 learned_addr learned my addr 172.21.15.89:0/2437776814 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f4967fff640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968100a80 0x7f49681983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f49677fe640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4968101420 0x7f4968198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496cebd640 1 -- 172.21.15.89:0/2437776814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968100a80 msgr2=0x7f49681983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496cebd640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968100a80 0x7f49681983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496cebd640 1 -- 172.21.15.89:0/2437776814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4968101420 msgr2=0x7f4968198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496cebd640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4968101420 0x7f4968198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496cebd640 1 -- 172.21.15.89:0/2437776814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496819d3a0 con 0x7f4968106a80 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f4967fff640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968100a80 0x7f49681983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496cebd640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106a80 0x7f496819ccc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4960005cc0 tx=0x7f4960003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:52.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f49657fa640 1 -- 172.21.15.89:0/2437776814 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960004040 con 0x7f4968106a80 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f49657fa640 1 -- 172.21.15.89:0/2437776814 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49600046c0 con 0x7f4968106a80 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496e920640 1 -- 172.21.15.89:0/2437776814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496819d630 con 0x7f4968106a80 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f49657fa640 1 -- 172.21.15.89:0/2437776814 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4960046480 con 0x7f4968106a80 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.558+0000 7f496e920640 1 -- 172.21.15.89:0/2437776814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49681a4f70 con 0x7f4968106a80 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.559+0000 7f496e920640 1 -- 172.21.15.89:0/2437776814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4968100ee0 con 0x7f4968106a80 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.560+0000 7f49657fa640 1 -- 172.21.15.89:0/2437776814 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f49600305a0 con 0x7f4968106a80 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.561+0000 7f49657fa640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49500a3350 0x7f49500a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.561+0000 7f49657fa640 1 -- 172.21.15.89:0/2437776814 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f496002e030 con 0x7f4968106a80 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.561+0000 7f4967fff640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49500a3350 0x7f49500a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.561+0000 7f4967fff640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49500a3350 0x7f49500a5810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4958005e00 tx=0x7f4958005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:52.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.567+0000 7f49657fa640 1 -- 172.21.15.89:0/2437776814 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f496004b060 con 0x7f4968106a80 2024-09-15T23:22:52.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.733+0000 7f496e920640 1 -- 172.21.15.89:0/2437776814 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49680fdd70 con 0x7f49500a3350 2024-09-15T23:22:52.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.741+0000 7f49657fa640 1 -- 172.21.15.89:0/2437776814 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340635 (secure 0 0 0) 0x7f49680fdd70 con 0x7f49500a3350 2024-09-15T23:22:52.745 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:52.746 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-15T23:22:52.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 -- 172.21.15.89:0/2437776814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49500a3350 msgr2=0x7f49500a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:52.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49500a3350 0x7f49500a5810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4958005e00 tx=0x7f4958005c70 comp rx=0 tx=0).stop 2024-09-15T23:22:52.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 -- 172.21.15.89:0/2437776814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106a80 msgr2=0x7f496819ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:52.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106a80 0x7f496819ccc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4960005cc0 tx=0x7f4960003970 comp rx=0 tx=0).stop 2024-09-15T23:22:52.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 -- 172.21.15.89:0/2437776814 shutdown_connections 2024-09-15T23:22:52.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49500a3350 0x7f49500a5810 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:52.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106a80 0x7f496819ccc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:52.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4968101420 0x7f4968198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:52.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 --2- 172.21.15.89:0/2437776814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968100a80 0x7f49681983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:52.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 -- 172.21.15.89:0/2437776814 >> 172.21.15.89:0/2437776814 conn(0x7f49680fc7a0 msgr2=0x7f4968104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:52.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 -- 172.21.15.89:0/2437776814 shutdown_connections 2024-09-15T23:22:52.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:52.744+0000 7f496e920640 1 -- 172.21.15.89:0/2437776814 wait complete. 2024-09-15T23:22:53.437 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:53 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:22:53.437 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:53 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:22:53.437 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:53 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:53.466 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-15T23:22:50.993221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":640762,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":266,"num_read_kb":232,"num_write":416,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":341,"ondisk_log_size":341,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360944,"kb_used_data":11696,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580880,"statfs":{"total":1151940427776,"available":1151570821120,"internally_reserved":0,"allocated":11976704,"data_stored":7705008,"data_compressed":318738,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40806,"num_objects":15,"num_object_clones":0,"num_object_copies":47,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":138,"num_read_kb":121,"num_write":52,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003292"},"pg_stats":[{"pgid":"2.1f","version":"86'13","reported_seq":54,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:46.166936+0000","last_change":"2024-09-15T23:21:50.513610+0000","last_active":"2024-09-15T23:22:46.166936+0000","last_peered":"2024-09-15T23:22:46.166936+0000","last_clean":"2024-09-15T23:22:46.166936+0000","last_became_active":"2024-09-15T23:21:50.513266+0000","last_became_peered":"2024-09-15T23:21:50.513266+0000","last_unstale":"2024-09-15T23:22:46.166936+0000","last_undegraded":"2024-09-15T23:22:46.166936+0000","last_fullsized":"2024-09-15T23:22:46.166936+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:16:48.692801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576117+0000","last_change":"2024-09-15T23:21:52.540248+0000","last_active":"2024-09-15T23:22:02.576117+0000","last_peered":"2024-09-15T23:22:02.576117+0000","last_clean":"2024-09-15T23:22:02.576117+0000","last_became_active":"2024-09-15T23:21:52.539945+0000","last_became_peered":"2024-09-15T23:21:52.539945+0000","last_unstale":"2024-09-15T23:22:02.576117+0000","last_undegraded":"2024-09-15T23:22:02.576117+0000","last_fullsized":"2024-09-15T23:22:02.576117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:33.402153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634558+0000","last_change":"2024-09-15T23:21:59.567246+0000","last_active":"2024-09-15T23:22:02.634558+0000","last_peered":"2024-09-15T23:22:02.634558+0000","last_clean":"2024-09-15T23:22:02.634558+0000","last_became_active":"2024-09-15T23:21:59.566832+0000","last_became_peered":"2024-09-15T23:21:59.566832+0000","last_unstale":"2024-09-15T23:22:02.634558+0000","last_undegraded":"2024-09-15T23:22:02.634558+0000","last_fullsized":"2024-09-15T23:22:02.634558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:19.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576462+0000","last_change":"2024-09-15T23:21:50.561767+0000","last_active":"2024-09-15T23:22:02.576462+0000","last_peered":"2024-09-15T23:22:02.576462+0000","last_clean":"2024-09-15T23:22:02.576462+0000","last_became_active":"2024-09-15T23:21:50.561385+0000","last_became_peered":"2024-09-15T23:21:50.561385+0000","last_unstale":"2024-09-15T23:22:02.576462+0000","last_undegraded":"2024-09-15T23:22:02.576462+0000","last_fullsized":"2024-09-15T23:22:02.576462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:11.897884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.923183+0000","last_change":"2024-09-15T23:21:52.534874+0000","last_active":"2024-09-15T23:22:02.923183+0000","last_peered":"2024-09-15T23:22:02.923183+0000","last_clean":"2024-09-15T23:22:02.923183+0000","last_became_active":"2024-09-15T23:21:52.534334+0000","last_became_peered":"2024-09-15T23:21:52.534334+0000","last_unstale":"2024-09-15T23:22:02.923183+0000","last_undegraded":"2024-09-15T23:22:02.923183+0000","last_fullsized":"2024-09-15T23:22:02.923183+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:15.972648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632188+0000","last_change":"2024-09-15T23:21:59.559907+0000","last_active":"2024-09-15T23:22:02.632188+0000","last_peered":"2024-09-15T23:22:02.632188+0000","last_clean":"2024-09-15T23:22:02.632188+0000","last_became_active":"2024-09-15T23:21:59.559440+0000","last_became_peered":"2024-09-15T23:21:59.559440+0000","last_unstale":"2024-09-15T23:22:02.632188+0000","last_undegraded":"2024-09-15T23:22:02.632188+0000","last_fullsized":"2024-09-15T23:22:02.632188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:00.755112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":42,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:37.131384+0000","last_change":"2024-09-15T23:21:50.515424+0000","last_active":"2024-09-15T23:22:37.131384+0000","last_peered":"2024-09-15T23:22:37.131384+0000","last_clean":"2024-09-15T23:22:37.131384+0000","last_became_active":"2024-09-15T23:21:50.514966+0000","last_became_peered":"2024-09-15T23:21:50.514966+0000","last_unstale":"2024-09-15T23:22:37.131384+0000","last_undegraded":"2024-09-15T23:22:37.131384+0000","last_fullsized":"2024-09-15T23:22:37.131384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:57.586575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632356+0000","last_change":"2024-09-15T23:21:52.535189+0000","last_active":"2024-09-15T23:22:02.632356+0000","last_peered":"2024-09-15T23:22:02.632356+0000","last_clean":"2024-09-15T23:22:02.632356+0000","last_became_active":"2024-09-15T23:21:52.534807+0000","last_became_peered":"2024-09-15T23:21:52.534807+0000","last_unstale":"2024-09-15T23:22:02.632356+0000","last_undegraded":"2024-09-15T23:22:02.632356+0000","last_fullsized":"2024-09-15T23:22:02.632356+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:30.654382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632608+0000","last_change":"2024-09-15T23:21:59.564675+0000","last_active":"2024-09-15T23:22:02.632608+0000","last_peered":"2024-09-15T23:22:02.632608+0000","last_clean":"2024-09-15T23:22:02.632608+0000","last_became_active":"2024-09-15T23:21:59.564271+0000","last_became_peered":"2024-09-15T23:21:59.564271+0000","last_unstale":"2024-09-15T23:22:02.632608+0000","last_undegraded":"2024-09-15T23:22:02.632608+0000","last_fullsized":"2024-09-15T23:22:02.632608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:01.055369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.136685+0000","last_change":"2024-09-15T23:21:50.507790+0000","last_active":"2024-09-15T23:22:36.136685+0000","last_peered":"2024-09-15T23:22:36.136685+0000","last_clean":"2024-09-15T23:22:36.136685+0000","last_became_active":"2024-09-15T23:21:50.507398+0000","last_became_peered":"2024-09-15T23:21:50.507398+0000","last_unstale":"2024-09-15T23:22:36.136685+0000","last_undegraded":"2024-09-15T23:22:36.136685+0000","last_fullsized":"2024-09-15T23:22:36.136685+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:58:00.001279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634061+0000","last_change":"2024-09-15T23:21:52.537179+0000","last_active":"2024-09-15T23:22:02.634061+0000","last_peered":"2024-09-15T23:22:02.634061+0000","last_clean":"2024-09-15T23:22:02.634061+0000","last_became_active":"2024-09-15T23:21:52.536841+0000","last_became_peered":"2024-09-15T23:21:52.536841+0000","last_unstale":"2024-09-15T23:22:02.634061+0000","last_undegraded":"2024-09-15T23:22:02.634061+0000","last_fullsized":"2024-09-15T23:22:02.634061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:47.455700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632181+0000","last_change":"2024-09-15T23:21:59.564734+0000","last_active":"2024-09-15T23:22:02.632181+0000","last_peered":"2024-09-15T23:22:02.632181+0000","last_clean":"2024-09-15T23:22:02.632181+0000","last_became_active":"2024-09-15T23:21:59.564339+0000","last_became_peered":"2024-09-15T23:21:59.564339+0000","last_unstale":"2024-09-15T23:22:02.632181+0000","last_undegraded":"2024-09-15T23:22:02.632181+0000","last_fullsized":"2024-09-15T23:22:02.632181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:44.498480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.578855+0000","last_change":"2024-09-15T23:21:52.538025+0000","last_active":"2024-09-15T23:22:02.578855+0000","last_peered":"2024-09-15T23:22:02.578855+0000","last_clean":"2024-09-15T23:22:02.578855+0000","last_became_active":"2024-09-15T23:21:52.537736+0000","last_became_peered":"2024-09-15T23:21:52.537736+0000","last_unstale":"2024-09-15T23:22:02.578855+0000","last_undegraded":"2024-09-15T23:22:02.578855+0000","last_fullsized":"2024-09-15T23:22:02.578855+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:31.916888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:33.097914+0000","last_change":"2024-09-15T23:21:50.516024+0000","last_active":"2024-09-15T23:22:33.097914+0000","last_peered":"2024-09-15T23:22:33.097914+0000","last_clean":"2024-09-15T23:22:33.097914+0000","last_became_active":"2024-09-15T23:21:50.515590+0000","last_became_peered":"2024-09-15T23:21:50.515590+0000","last_unstale":"2024-09-15T23:22:33.097914+0000","last_undegraded":"2024-09-15T23:22:33.097914+0000","last_fullsized":"2024-09-15T23:22:33.097914+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:47:38.849366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632272+0000","last_change":"2024-09-15T23:21:59.681971+0000","last_active":"2024-09-15T23:22:02.632272+0000","last_peered":"2024-09-15T23:22:02.632272+0000","last_clean":"2024-09-15T23:22:02.632272+0000","last_became_active":"2024-09-15T23:21:59.681432+0000","last_became_peered":"2024-09-15T23:21:59.681432+0000","last_unstale":"2024-09-15T23:22:02.632272+0000","last_undegraded":"2024-09-15T23:22:02.632272+0000","last_fullsized":"2024-09-15T23:22:02.632272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:26.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.575941+0000","last_change":"2024-09-15T23:21:52.541281+0000","last_active":"2024-09-15T23:22:02.575941+0000","last_peered":"2024-09-15T23:22:02.575941+0000","last_clean":"2024-09-15T23:22:02.575941+0000","last_became_active":"2024-09-15T23:21:52.540772+0000","last_became_peered":"2024-09-15T23:21:52.540772+0000","last_unstale":"2024-09-15T23:22:02.575941+0000","last_undegraded":"2024-09-15T23:22:02.575941+0000","last_fullsized":"2024-09-15T23:22:02.575941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:23:44.444904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:35.119894+0000","last_change":"2024-09-15T23:21:50.508844+0000","last_active":"2024-09-15T23:22:35.119894+0000","last_peered":"2024-09-15T23:22:35.119894+0000","last_clean":"2024-09-15T23:22:35.119894+0000","last_became_active":"2024-09-15T23:21:50.508364+0000","last_became_peered":"2024-09-15T23:21:50.508364+0000","last_unstale":"2024-09-15T23:22:35.119894+0000","last_undegraded":"2024-09-15T23:22:35.119894+0000","last_fullsized":"2024-09-15T23:22:35.119894+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:45.129711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633100+0000","last_change":"2024-09-15T23:21:59.562114+0000","last_active":"2024-09-15T23:22:02.633100+0000","last_peered":"2024-09-15T23:22:02.633100+0000","last_clean":"2024-09-15T23:22:02.633100+0000","last_became_active":"2024-09-15T23:21:59.561652+0000","last_became_peered":"2024-09-15T23:21:59.561652+0000","last_unstale":"2024-09-15T23:22:02.633100+0000","last_undegraded":"2024-09-15T23:22:02.633100+0000","last_fullsized":"2024-09-15T23:22:02.633100+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:51.781886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632984+0000","last_change":"2024-09-15T23:21:52.531678+0000","last_active":"2024-09-15T23:22:02.632984+0000","last_peered":"2024-09-15T23:22:02.632984+0000","last_clean":"2024-09-15T23:22:02.632984+0000","last_became_active":"2024-09-15T23:21:52.531327+0000","last_became_peered":"2024-09-15T23:21:52.531327+0000","last_unstale":"2024-09-15T23:22:02.632984+0000","last_undegraded":"2024-09-15T23:22:02.632984+0000","last_fullsized":"2024-09-15T23:22:02.632984+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:07.152626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":36,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.134997+0000","last_change":"2024-09-15T23:21:50.513197+0000","last_active":"2024-09-15T23:22:36.134997+0000","last_peered":"2024-09-15T23:22:36.134997+0000","last_clean":"2024-09-15T23:22:36.134997+0000","last_became_active":"2024-09-15T23:21:50.512794+0000","last_became_peered":"2024-09-15T23:21:50.512794+0000","last_unstale":"2024-09-15T23:22:36.134997+0000","last_undegraded":"2024-09-15T23:22:36.134997+0000","last_fullsized":"2024-09-15T23:22:36.134997+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:05.457467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633044+0000","last_change":"2024-09-15T23:21:59.565669+0000","last_active":"2024-09-15T23:22:02.633044+0000","last_peered":"2024-09-15T23:22:02.633044+0000","last_clean":"2024-09-15T23:22:02.633044+0000","last_became_active":"2024-09-15T23:21:59.565290+0000","last_became_peered":"2024-09-15T23:21:59.565290+0000","last_unstale":"2024-09-15T23:22:02.633044+0000","last_undegraded":"2024-09-15T23:22:02.633044+0000","last_fullsized":"2024-09-15T23:22:02.633044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:04.861560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576604+0000","last_change":"2024-09-15T23:21:52.529643+0000","last_active":"2024-09-15T23:22:02.576604+0000","last_peered":"2024-09-15T23:22:02.576604+0000","last_clean":"2024-09-15T23:22:02.576604+0000","last_became_active":"2024-09-15T23:21:52.529304+0000","last_became_peered":"2024-09-15T23:21:52.529304+0000","last_unstale":"2024-09-15T23:22:02.576604+0000","last_undegraded":"2024-09-15T23:22:02.576604+0000","last_fullsized":"2024-09-15T23:22:02.576604+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:18:37.758403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:33.097341+0000","last_change":"2024-09-15T23:21:50.562114+0000","last_active":"2024-09-15T23:22:33.097341+0000","last_peered":"2024-09-15T23:22:33.097341+0000","last_clean":"2024-09-15T23:22:33.097341+0000","last_became_active":"2024-09-15T23:21:50.561772+0000","last_became_peered":"2024-09-15T23:21:50.561772+0000","last_unstale":"2024-09-15T23:22:33.097341+0000","last_undegraded":"2024-09-15T23:22:33.097341+0000","last_fullsized":"2024-09-15T23:22:33.097341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:14.806646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633256+0000","last_change":"2024-09-15T23:21:59.681933+0000","last_active":"2024-09-15T23:22:02.633256+0000","last_peered":"2024-09-15T23:22:02.633256+0000","last_clean":"2024-09-15T23:22:02.633256+0000","last_became_active":"2024-09-15T23:21:59.681528+0000","last_became_peered":"2024-09-15T23:21:59.681528+0000","last_unstale":"2024-09-15T23:22:02.633256+0000","last_undegraded":"2024-09-15T23:22:02.633256+0000","last_fullsized":"2024-09-15T23:22:02.633256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:46.199540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.923082+0000","last_change":"2024-09-15T23:21:52.529548+0000","last_active":"2024-09-15T23:22:02.923082+0000","last_peered":"2024-09-15T23:22:02.923082+0000","last_clean":"2024-09-15T23:22:02.923082+0000","last_became_active":"2024-09-15T23:21:52.529253+0000","last_became_peered":"2024-09-15T23:21:52.529253+0000","last_unstale":"2024-09-15T23:22:02.923082+0000","last_undegraded":"2024-09-15T23:22:02.923082+0000","last_fullsized":"2024-09-15T23:22:02.923082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:22.813395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.134592+0000","last_change":"2024-09-15T23:21:50.767759+0000","last_active":"2024-09-15T23:22:36.134592+0000","last_peered":"2024-09-15T23:22:36.134592+0000","last_clean":"2024-09-15T23:22:36.134592+0000","last_became_active":"2024-09-15T23:21:50.767388+0000","last_became_peered":"2024-09-15T23:21:50.767388+0000","last_unstale":"2024-09-15T23:22:36.134592+0000","last_undegraded":"2024-09-15T23:22:36.134592+0000","last_fullsized":"2024-09-15T23:22:36.134592+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:36.713769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634571+0000","last_change":"2024-09-15T23:21:59.557951+0000","last_active":"2024-09-15T23:22:02.634571+0000","last_peered":"2024-09-15T23:22:02.634571+0000","last_clean":"2024-09-15T23:22:02.634571+0000","last_became_active":"2024-09-15T23:21:59.557490+0000","last_became_peered":"2024-09-15T23:21:59.557490+0000","last_unstale":"2024-09-15T23:22:02.634571+0000","last_undegraded":"2024-09-15T23:22:02.634571+0000","last_fullsized":"2024-09-15T23:22:02.634571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:17.147389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576306+0000","last_change":"2024-09-15T23:21:52.531638+0000","last_active":"2024-09-15T23:22:02.576306+0000","last_peered":"2024-09-15T23:22:02.576306+0000","last_clean":"2024-09-15T23:22:02.576306+0000","last_became_active":"2024-09-15T23:21:52.531301+0000","last_became_peered":"2024-09-15T23:21:52.531301+0000","last_unstale":"2024-09-15T23:22:02.576306+0000","last_undegraded":"2024-09-15T23:22:02.576306+0000","last_fullsized":"2024-09-15T23:22:02.576306+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.436286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'12","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:44.532224+0000","last_change":"2024-09-15T23:21:50.508470+0000","last_active":"2024-09-15T23:22:44.532224+0000","last_peered":"2024-09-15T23:22:44.532224+0000","last_clean":"2024-09-15T23:22:44.532224+0000","last_became_active":"2024-09-15T23:21:50.508060+0000","last_became_peered":"2024-09-15T23:21:50.508060+0000","last_unstale":"2024-09-15T23:22:44.532224+0000","last_undegraded":"2024-09-15T23:22:44.532224+0000","last_fullsized":"2024-09-15T23:22:44.532224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:59.542252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.575941+0000","last_change":"2024-09-15T23:21:59.682019+0000","last_active":"2024-09-15T23:22:02.575941+0000","last_peered":"2024-09-15T23:22:02.575941+0000","last_clean":"2024-09-15T23:22:02.575941+0000","last_became_active":"2024-09-15T23:21:59.681467+0000","last_became_peered":"2024-09-15T23:21:59.681467+0000","last_unstale":"2024-09-15T23:22:02.575941+0000","last_undegraded":"2024-09-15T23:22:02.575941+0000","last_fullsized":"2024-09-15T23:22:02.575941+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:27.650382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632526+0000","last_change":"2024-09-15T23:21:52.536345+0000","last_active":"2024-09-15T23:22:02.632526+0000","last_peered":"2024-09-15T23:22:02.632526+0000","last_clean":"2024-09-15T23:22:02.632526+0000","last_became_active":"2024-09-15T23:21:52.536027+0000","last_became_peered":"2024-09-15T23:21:52.536027+0000","last_unstale":"2024-09-15T23:22:02.632526+0000","last_undegraded":"2024-09-15T23:22:02.632526+0000","last_fullsized":"2024-09-15T23:22:02.632526+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:16.435912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":34,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.135178+0000","last_change":"2024-09-15T23:21:50.562945+0000","last_active":"2024-09-15T23:22:36.135178+0000","last_peered":"2024-09-15T23:22:36.135178+0000","last_clean":"2024-09-15T23:22:36.135178+0000","last_became_active":"2024-09-15T23:21:50.561632+0000","last_became_peered":"2024-09-15T23:21:50.561632+0000","last_unstale":"2024-09-15T23:22:36.135178+0000","last_undegraded":"2024-09-15T23:22:36.135178+0000","last_fullsized":"2024-09-15T23:22:36.135178+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:44.545489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633201+0000","last_change":"2024-09-15T23:21:59.564793+0000","last_active":"2024-09-15T23:22:02.633201+0000","last_peered":"2024-09-15T23:22:02.633201+0000","last_clean":"2024-09-15T23:22:02.633201+0000","last_became_active":"2024-09-15T23:21:59.564140+0000","last_became_peered":"2024-09-15T23:21:59.564140+0000","last_unstale":"2024-09-15T23:22:02.633201+0000","last_undegraded":"2024-09-15T23:22:02.633201+0000","last_fullsized":"2024-09-15T23:22:02.633201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:41.464962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632773+0000","last_change":"2024-09-15T23:21:52.533521+0000","last_active":"2024-09-15T23:22:02.632773+0000","last_peered":"2024-09-15T23:22:02.632773+0000","last_clean":"2024-09-15T23:22:02.632773+0000","last_became_active":"2024-09-15T23:21:52.532195+0000","last_became_peered":"2024-09-15T23:21:52.532195+0000","last_unstale":"2024-09-15T23:22:02.632773+0000","last_undegraded":"2024-09-15T23:22:02.632773+0000","last_fullsized":"2024-09-15T23:22:02.632773+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:14.229018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633039+0000","last_change":"2024-09-15T23:21:50.562893+0000","last_active":"2024-09-15T23:22:02.633039+0000","last_peered":"2024-09-15T23:22:02.633039+0000","last_clean":"2024-09-15T23:22:02.633039+0000","last_became_active":"2024-09-15T23:21:50.562516+0000","last_became_peered":"2024-09-15T23:21:50.562516+0000","last_unstale":"2024-09-15T23:22:02.633039+0000","last_undegraded":"2024-09-15T23:22:02.633039+0000","last_fullsized":"2024-09-15T23:22:02.633039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:47:47.442458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576698+0000","last_change":"2024-09-15T23:21:59.682091+0000","last_active":"2024-09-15T23:22:02.576698+0000","last_peered":"2024-09-15T23:22:02.576698+0000","last_clean":"2024-09-15T23:22:02.576698+0000","last_became_active":"2024-09-15T23:21:59.681574+0000","last_became_peered":"2024-09-15T23:21:59.681574+0000","last_unstale":"2024-09-15T23:22:02.576698+0000","last_undegraded":"2024-09-15T23:22:02.576698+0000","last_fullsized":"2024-09-15T23:22:02.576698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:04.991608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633167+0000","last_change":"2024-09-15T23:21:52.533811+0000","last_active":"2024-09-15T23:22:02.633167+0000","last_peered":"2024-09-15T23:22:02.633167+0000","last_clean":"2024-09-15T23:22:02.633167+0000","last_became_active":"2024-09-15T23:21:52.533487+0000","last_became_peered":"2024-09-15T23:21:52.533487+0000","last_unstale":"2024-09-15T23:22:02.633167+0000","last_undegraded":"2024-09-15T23:22:02.633167+0000","last_fullsized":"2024-09-15T23:22:02.633167+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:44.193212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:45.166348+0000","last_change":"2024-09-15T23:21:50.510883+0000","last_active":"2024-09-15T23:22:45.166348+0000","last_peered":"2024-09-15T23:22:45.166348+0000","last_clean":"2024-09-15T23:22:45.166348+0000","last_became_active":"2024-09-15T23:21:50.510478+0000","last_became_peered":"2024-09-15T23:21:50.510478+0000","last_unstale":"2024-09-15T23:22:45.166348+0000","last_undegraded":"2024-09-15T23:22:45.166348+0000","last_fullsized":"2024-09-15T23:22:45.166348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:35.454885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632424+0000","last_change":"2024-09-15T23:21:59.561816+0000","last_active":"2024-09-15T23:22:02.632424+0000","last_peered":"2024-09-15T23:22:02.632424+0000","last_clean":"2024-09-15T23:22:02.632424+0000","last_became_active":"2024-09-15T23:21:59.561434+0000","last_became_peered":"2024-09-15T23:21:59.561434+0000","last_unstale":"2024-09-15T23:22:02.632424+0000","last_undegraded":"2024-09-15T23:22:02.632424+0000","last_fullsized":"2024-09-15T23:22:02.632424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:59.945802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632948+0000","last_change":"2024-09-15T23:21:52.536886+0000","last_active":"2024-09-15T23:22:02.632948+0000","last_peered":"2024-09-15T23:22:02.632948+0000","last_clean":"2024-09-15T23:22:02.632948+0000","last_became_active":"2024-09-15T23:21:52.536570+0000","last_became_peered":"2024-09-15T23:21:52.536570+0000","last_unstale":"2024-09-15T23:22:02.632948+0000","last_undegraded":"2024-09-15T23:22:02.632948+0000","last_fullsized":"2024-09-15T23:22:02.632948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:05:25.330311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":45,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:44.150819+0000","last_change":"2024-09-15T23:21:50.561684+0000","last_active":"2024-09-15T23:22:44.150819+0000","last_peered":"2024-09-15T23:22:44.150819+0000","last_clean":"2024-09-15T23:22:44.150819+0000","last_became_active":"2024-09-15T23:21:50.561335+0000","last_became_peered":"2024-09-15T23:21:50.561335+0000","last_unstale":"2024-09-15T23:22:44.150819+0000","last_undegraded":"2024-09-15T23:22:44.150819+0000","last_fullsized":"2024-09-15T23:22:44.150819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:23.897250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632723+0000","last_change":"2024-09-15T23:21:59.681858+0000","last_active":"2024-09-15T23:22:02.632723+0000","last_peered":"2024-09-15T23:22:02.632723+0000","last_clean":"2024-09-15T23:22:02.632723+0000","last_became_active":"2024-09-15T23:21:59.681441+0000","last_became_peered":"2024-09-15T23:21:59.681441+0000","last_unstale":"2024-09-15T23:22:02.632723+0000","last_undegraded":"2024-09-15T23:22:02.632723+0000","last_fullsized":"2024-09-15T23:22:02.632723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:28.513719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632034+0000","last_change":"2024-09-15T23:21:52.536141+0000","last_active":"2024-09-15T23:22:02.632034+0000","last_peered":"2024-09-15T23:22:02.632034+0000","last_clean":"2024-09-15T23:22:02.632034+0000","last_became_active":"2024-09-15T23:21:52.535832+0000","last_became_peered":"2024-09-15T23:21:52.535832+0000","last_unstale":"2024-09-15T23:22:02.632034+0000","last_undegraded":"2024-09-15T23:22:02.632034+0000","last_fullsized":"2024-09-15T23:22:02.632034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:21:44.341412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:34.103482+0000","last_change":"2024-09-15T23:21:50.514689+0000","last_active":"2024-09-15T23:22:34.103482+0000","last_peered":"2024-09-15T23:22:34.103482+0000","last_clean":"2024-09-15T23:22:34.103482+0000","last_became_active":"2024-09-15T23:21:50.513261+0000","last_became_peered":"2024-09-15T23:21:50.513261+0000","last_unstale":"2024-09-15T23:22:34.103482+0000","last_undegraded":"2024-09-15T23:22:34.103482+0000","last_fullsized":"2024-09-15T23:22:34.103482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:30.132991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633489+0000","last_change":"2024-09-15T23:21:59.682118+0000","last_active":"2024-09-15T23:22:02.633489+0000","last_peered":"2024-09-15T23:22:02.633489+0000","last_clean":"2024-09-15T23:22:02.633489+0000","last_became_active":"2024-09-15T23:21:59.681500+0000","last_became_peered":"2024-09-15T23:21:59.681500+0000","last_unstale":"2024-09-15T23:22:02.633489+0000","last_undegraded":"2024-09-15T23:22:02.633489+0000","last_fullsized":"2024-09-15T23:22:02.633489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:58.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576134+0000","last_change":"2024-09-15T23:21:52.540458+0000","last_active":"2024-09-15T23:22:02.576134+0000","last_peered":"2024-09-15T23:22:02.576134+0000","last_clean":"2024-09-15T23:22:02.576134+0000","last_became_active":"2024-09-15T23:21:52.540112+0000","last_became_peered":"2024-09-15T23:21:52.540112+0000","last_unstale":"2024-09-15T23:22:02.576134+0000","last_undegraded":"2024-09-15T23:22:02.576134+0000","last_fullsized":"2024-09-15T23:22:02.576134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:15.240785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:37.137630+0000","last_change":"2024-09-15T23:21:50.513124+0000","last_active":"2024-09-15T23:22:37.137630+0000","last_peered":"2024-09-15T23:22:37.137630+0000","last_clean":"2024-09-15T23:22:37.137630+0000","last_became_active":"2024-09-15T23:21:50.512705+0000","last_became_peered":"2024-09-15T23:21:50.512705+0000","last_unstale":"2024-09-15T23:22:37.137630+0000","last_undegraded":"2024-09-15T23:22:37.137630+0000","last_fullsized":"2024-09-15T23:22:37.137630+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.417937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":345,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.578940+0000","last_change":"2024-09-15T23:20:40.656652+0000","last_active":"2024-09-15T23:22:02.578940+0000","last_peered":"2024-09-15T23:22:02.578940+0000","last_clean":"2024-09-15T23:22:02.578940+0000","last_became_active":"2024-09-15T23:20:40.656266+0000","last_became_peered":"2024-09-15T23:20:40.656266+0000","last_unstale":"2024-09-15T23:22:02.578940+0000","last_undegraded":"2024-09-15T23:22:02.578940+0000","last_fullsized":"2024-09-15T23:22:02.578940+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_clean_scrub_stamp":"2024-09-15T23:17:15.186850+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:42.317006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.579288+0000","last_change":"2024-09-15T23:21:59.563777+0000","last_active":"2024-09-15T23:22:02.579288+0000","last_peered":"2024-09-15T23:22:02.579288+0000","last_clean":"2024-09-15T23:22:02.579288+0000","last_became_active":"2024-09-15T23:21:59.563320+0000","last_became_peered":"2024-09-15T23:21:59.563320+0000","last_unstale":"2024-09-15T23:22:02.579288+0000","last_undegraded":"2024-09-15T23:22:02.579288+0000","last_fullsized":"2024-09-15T23:22:02.579288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:25:07.791923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.579367+0000","last_change":"2024-09-15T23:21:52.539842+0000","last_active":"2024-09-15T23:22:02.579367+0000","last_peered":"2024-09-15T23:22:02.579367+0000","last_clean":"2024-09-15T23:22:02.579367+0000","last_became_active":"2024-09-15T23:21:52.538683+0000","last_became_peered":"2024-09-15T23:21:52.538683+0000","last_unstale":"2024-09-15T23:22:02.579367+0000","last_undegraded":"2024-09-15T23:22:02.579367+0000","last_fullsized":"2024-09-15T23:22:02.579367+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:43.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:48.158017+0000","last_change":"2024-09-15T23:21:50.767601+0000","last_active":"2024-09-15T23:22:48.158017+0000","last_peered":"2024-09-15T23:22:48.158017+0000","last_clean":"2024-09-15T23:22:48.158017+0000","last_became_active":"2024-09-15T23:21:50.767149+0000","last_became_peered":"2024-09-15T23:21:50.767149+0000","last_unstale":"2024-09-15T23:22:48.158017+0000","last_undegraded":"2024-09-15T23:22:48.158017+0000","last_fullsized":"2024-09-15T23:22:48.158017+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:23.636551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632234+0000","last_change":"2024-09-15T23:21:59.563069+0000","last_active":"2024-09-15T23:22:02.632234+0000","last_peered":"2024-09-15T23:22:02.632234+0000","last_clean":"2024-09-15T23:22:02.632234+0000","last_became_active":"2024-09-15T23:21:59.562658+0000","last_became_peered":"2024-09-15T23:21:59.562658+0000","last_unstale":"2024-09-15T23:22:02.632234+0000","last_undegraded":"2024-09-15T23:22:02.632234+0000","last_fullsized":"2024-09-15T23:22:02.632234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:23.067198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.579146+0000","last_change":"2024-09-15T23:21:52.535289+0000","last_active":"2024-09-15T23:22:02.579146+0000","last_peered":"2024-09-15T23:22:02.579146+0000","last_clean":"2024-09-15T23:22:02.579146+0000","last_became_active":"2024-09-15T23:21:52.534885+0000","last_became_peered":"2024-09-15T23:21:52.534885+0000","last_unstale":"2024-09-15T23:22:02.579146+0000","last_undegraded":"2024-09-15T23:22:02.579146+0000","last_fullsized":"2024-09-15T23:22:02.579146+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:42.293786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.135986+0000","last_change":"2024-09-15T23:21:50.513246+0000","last_active":"2024-09-15T23:22:36.135986+0000","last_peered":"2024-09-15T23:22:36.135986+0000","last_clean":"2024-09-15T23:22:36.135986+0000","last_became_active":"2024-09-15T23:21:50.512740+0000","last_became_peered":"2024-09-15T23:21:50.512740+0000","last_unstale":"2024-09-15T23:22:36.135986+0000","last_undegraded":"2024-09-15T23:22:36.135986+0000","last_fullsized":"2024-09-15T23:22:36.135986+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:59.201970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922863+0000","last_change":"2024-09-15T23:21:59.568565+0000","last_active":"2024-09-15T23:22:02.922863+0000","last_peered":"2024-09-15T23:22:02.922863+0000","last_clean":"2024-09-15T23:22:02.922863+0000","last_became_active":"2024-09-15T23:21:59.567789+0000","last_became_peered":"2024-09-15T23:21:59.567789+0000","last_unstale":"2024-09-15T23:22:02.922863+0000","last_undegraded":"2024-09-15T23:22:02.922863+0000","last_fullsized":"2024-09-15T23:22:02.922863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:02.106971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.579365+0000","last_change":"2024-09-15T23:21:52.533727+0000","last_active":"2024-09-15T23:22:02.579365+0000","last_peered":"2024-09-15T23:22:02.579365+0000","last_clean":"2024-09-15T23:22:02.579365+0000","last_became_active":"2024-09-15T23:21:52.532350+0000","last_became_peered":"2024-09-15T23:21:52.532350+0000","last_unstale":"2024-09-15T23:22:02.579365+0000","last_undegraded":"2024-09-15T23:22:02.579365+0000","last_fullsized":"2024-09-15T23:22:02.579365+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:03.624837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.134380+0000","last_change":"2024-09-15T23:21:50.509519+0000","last_active":"2024-09-15T23:22:36.134380+0000","last_peered":"2024-09-15T23:22:36.134380+0000","last_clean":"2024-09-15T23:22:36.134380+0000","last_became_active":"2024-09-15T23:21:50.509224+0000","last_became_peered":"2024-09-15T23:21:50.509224+0000","last_unstale":"2024-09-15T23:22:36.134380+0000","last_undegraded":"2024-09-15T23:22:36.134380+0000","last_fullsized":"2024-09-15T23:22:36.134380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:39:57.526821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632669+0000","last_change":"2024-09-15T23:21:59.566429+0000","last_active":"2024-09-15T23:22:02.632669+0000","last_peered":"2024-09-15T23:22:02.632669+0000","last_clean":"2024-09-15T23:22:02.632669+0000","last_became_active":"2024-09-15T23:21:59.565835+0000","last_became_peered":"2024-09-15T23:21:59.565835+0000","last_unstale":"2024-09-15T23:22:02.632669+0000","last_undegraded":"2024-09-15T23:22:02.632669+0000","last_fullsized":"2024-09-15T23:22:02.632669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:38.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633755+0000","last_change":"2024-09-15T23:21:52.539544+0000","last_active":"2024-09-15T23:22:02.633755+0000","last_peered":"2024-09-15T23:22:02.633755+0000","last_clean":"2024-09-15T23:22:02.633755+0000","last_became_active":"2024-09-15T23:21:52.539203+0000","last_became_peered":"2024-09-15T23:21:52.539203+0000","last_unstale":"2024-09-15T23:22:02.633755+0000","last_undegraded":"2024-09-15T23:22:02.633755+0000","last_fullsized":"2024-09-15T23:22:02.633755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.193965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":32,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.099173+0000","last_change":"2024-09-15T23:21:50.513543+0000","last_active":"2024-09-15T23:22:36.099173+0000","last_peered":"2024-09-15T23:22:36.099173+0000","last_clean":"2024-09-15T23:22:36.099173+0000","last_became_active":"2024-09-15T23:21:50.512812+0000","last_became_peered":"2024-09-15T23:21:50.512812+0000","last_unstale":"2024-09-15T23:22:36.099173+0000","last_undegraded":"2024-09-15T23:22:36.099173+0000","last_fullsized":"2024-09-15T23:22:36.099173+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:50.954096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632914+0000","last_change":"2024-09-15T23:21:59.570163+0000","last_active":"2024-09-15T23:22:02.632914+0000","last_peered":"2024-09-15T23:22:02.632914+0000","last_clean":"2024-09-15T23:22:02.632914+0000","last_became_active":"2024-09-15T23:21:59.569805+0000","last_became_peered":"2024-09-15T23:21:59.569805+0000","last_unstale":"2024-09-15T23:22:02.632914+0000","last_undegraded":"2024-09-15T23:22:02.632914+0000","last_fullsized":"2024-09-15T23:22:02.632914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:28.530817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633132+0000","last_change":"2024-09-15T23:21:52.537850+0000","last_active":"2024-09-15T23:22:02.633132+0000","last_peered":"2024-09-15T23:22:02.633132+0000","last_clean":"2024-09-15T23:22:02.633132+0000","last_became_active":"2024-09-15T23:21:52.536579+0000","last_became_peered":"2024-09-15T23:21:52.536579+0000","last_unstale":"2024-09-15T23:22:02.633132+0000","last_undegraded":"2024-09-15T23:22:02.633132+0000","last_fullsized":"2024-09-15T23:22:02.633132+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:06.353615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":47,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.134639+0000","last_change":"2024-09-15T23:21:50.562221+0000","last_active":"2024-09-15T23:22:36.134639+0000","last_peered":"2024-09-15T23:22:36.134639+0000","last_clean":"2024-09-15T23:22:36.134639+0000","last_became_active":"2024-09-15T23:21:50.561788+0000","last_became_peered":"2024-09-15T23:21:50.561788+0000","last_unstale":"2024-09-15T23:22:36.134639+0000","last_undegraded":"2024-09-15T23:22:36.134639+0000","last_fullsized":"2024-09-15T23:22:36.134639+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.853812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632398+0000","last_change":"2024-09-15T23:21:59.558621+0000","last_active":"2024-09-15T23:22:02.632398+0000","last_peered":"2024-09-15T23:22:02.632398+0000","last_clean":"2024-09-15T23:22:02.632398+0000","last_became_active":"2024-09-15T23:21:59.558248+0000","last_became_peered":"2024-09-15T23:21:59.558248+0000","last_unstale":"2024-09-15T23:22:02.632398+0000","last_undegraded":"2024-09-15T23:22:02.632398+0000","last_fullsized":"2024-09-15T23:22:02.632398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:49.854301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.578700+0000","last_change":"2024-09-15T23:21:52.539566+0000","last_active":"2024-09-15T23:22:02.578700+0000","last_peered":"2024-09-15T23:22:02.578700+0000","last_clean":"2024-09-15T23:22:02.578700+0000","last_became_active":"2024-09-15T23:21:52.539219+0000","last_became_peered":"2024-09-15T23:21:52.539219+0000","last_unstale":"2024-09-15T23:22:02.578700+0000","last_undegraded":"2024-09-15T23:22:02.578700+0000","last_fullsized":"2024-09-15T23:22:02.578700+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:41.266308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":52,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:49.158954+0000","last_change":"2024-09-15T23:21:50.516650+0000","last_active":"2024-09-15T23:22:49.158954+0000","last_peered":"2024-09-15T23:22:49.158954+0000","last_clean":"2024-09-15T23:22:49.158954+0000","last_became_active":"2024-09-15T23:21:50.516297+0000","last_became_peered":"2024-09-15T23:21:50.516297+0000","last_unstale":"2024-09-15T23:22:49.158954+0000","last_undegraded":"2024-09-15T23:22:49.158954+0000","last_fullsized":"2024-09-15T23:22:49.158954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:36.928701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633089+0000","last_change":"2024-09-15T23:21:59.568087+0000","last_active":"2024-09-15T23:22:02.633089+0000","last_peered":"2024-09-15T23:22:02.633089+0000","last_clean":"2024-09-15T23:22:02.633089+0000","last_became_active":"2024-09-15T23:21:59.567693+0000","last_became_peered":"2024-09-15T23:21:59.567693+0000","last_unstale":"2024-09-15T23:22:02.633089+0000","last_undegraded":"2024-09-15T23:22:02.633089+0000","last_fullsized":"2024-09-15T23:22:02.633089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.231873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.578860+0000","last_change":"2024-09-15T23:21:52.531090+0000","last_active":"2024-09-15T23:22:02.578860+0000","last_peered":"2024-09-15T23:22:02.578860+0000","last_clean":"2024-09-15T23:22:02.578860+0000","last_became_active":"2024-09-15T23:21:52.530738+0000","last_became_peered":"2024-09-15T23:21:52.530738+0000","last_unstale":"2024-09-15T23:22:02.578860+0000","last_undegraded":"2024-09-15T23:22:02.578860+0000","last_fullsized":"2024-09-15T23:22:02.578860+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:35.325128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.135607+0000","last_change":"2024-09-15T23:21:50.514100+0000","last_active":"2024-09-15T23:22:36.135607+0000","last_peered":"2024-09-15T23:22:36.135607+0000","last_clean":"2024-09-15T23:22:36.135607+0000","last_became_active":"2024-09-15T23:21:50.513629+0000","last_became_peered":"2024-09-15T23:21:50.513629+0000","last_unstale":"2024-09-15T23:22:36.135607+0000","last_undegraded":"2024-09-15T23:22:36.135607+0000","last_fullsized":"2024-09-15T23:22:36.135607+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.187544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633409+0000","last_change":"2024-09-15T23:21:59.562804+0000","last_active":"2024-09-15T23:22:02.633409+0000","last_peered":"2024-09-15T23:22:02.633409+0000","last_clean":"2024-09-15T23:22:02.633409+0000","last_became_active":"2024-09-15T23:21:59.562368+0000","last_became_peered":"2024-09-15T23:21:59.562368+0000","last_unstale":"2024-09-15T23:22:02.633409+0000","last_undegraded":"2024-09-15T23:22:02.633409+0000","last_fullsized":"2024-09-15T23:22:02.633409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:21.137333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632598+0000","last_change":"2024-09-15T23:21:52.530562+0000","last_active":"2024-09-15T23:22:02.632598+0000","last_peered":"2024-09-15T23:22:02.632598+0000","last_clean":"2024-09-15T23:22:02.632598+0000","last_became_active":"2024-09-15T23:21:52.530280+0000","last_became_peered":"2024-09-15T23:21:52.530280+0000","last_unstale":"2024-09-15T23:22:02.632598+0000","last_undegraded":"2024-09-15T23:22:02.632598+0000","last_fullsized":"2024-09-15T23:22:02.632598+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:26.835651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:37.137418+0000","last_change":"2024-09-15T23:21:50.513179+0000","last_active":"2024-09-15T23:22:37.137418+0000","last_peered":"2024-09-15T23:22:37.137418+0000","last_clean":"2024-09-15T23:22:37.137418+0000","last_became_active":"2024-09-15T23:21:50.512790+0000","last_became_peered":"2024-09-15T23:21:50.512790+0000","last_unstale":"2024-09-15T23:22:37.137418+0000","last_undegraded":"2024-09-15T23:22:37.137418+0000","last_fullsized":"2024-09-15T23:22:37.137418+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:56.408198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634066+0000","last_change":"2024-09-15T23:21:59.567895+0000","last_active":"2024-09-15T23:22:02.634066+0000","last_peered":"2024-09-15T23:22:02.634066+0000","last_clean":"2024-09-15T23:22:02.634066+0000","last_became_active":"2024-09-15T23:21:59.567500+0000","last_became_peered":"2024-09-15T23:21:59.567500+0000","last_unstale":"2024-09-15T23:22:02.634066+0000","last_undegraded":"2024-09-15T23:22:02.634066+0000","last_fullsized":"2024-09-15T23:22:02.634066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:09:41.526437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.577102+0000","last_change":"2024-09-15T23:21:52.530970+0000","last_active":"2024-09-15T23:22:02.577102+0000","last_peered":"2024-09-15T23:22:02.577102+0000","last_clean":"2024-09-15T23:22:02.577102+0000","last_became_active":"2024-09-15T23:21:52.530636+0000","last_became_peered":"2024-09-15T23:21:52.530636+0000","last_unstale":"2024-09-15T23:22:02.577102+0000","last_undegraded":"2024-09-15T23:22:02.577102+0000","last_fullsized":"2024-09-15T23:22:02.577102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:33.374658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"86'27","reported_seq":81,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:45.155953+0000","last_change":"2024-09-15T23:21:50.518449+0000","last_active":"2024-09-15T23:22:45.155953+0000","last_peered":"2024-09-15T23:22:45.155953+0000","last_clean":"2024-09-15T23:22:45.155953+0000","last_became_active":"2024-09-15T23:21:50.518099+0000","last_became_peered":"2024-09-15T23:21:50.518099+0000","last_unstale":"2024-09-15T23:22:45.155953+0000","last_undegraded":"2024-09-15T23:22:45.155953+0000","last_fullsized":"2024-09-15T23:22:45.155953+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:43.792971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43345,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":54,"num_write":29,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632825+0000","last_change":"2024-09-15T23:21:59.564939+0000","last_active":"2024-09-15T23:22:02.632825+0000","last_peered":"2024-09-15T23:22:02.632825+0000","last_clean":"2024-09-15T23:22:02.632825+0000","last_became_active":"2024-09-15T23:21:59.564484+0000","last_became_peered":"2024-09-15T23:21:59.564484+0000","last_unstale":"2024-09-15T23:22:02.632825+0000","last_undegraded":"2024-09-15T23:22:02.632825+0000","last_fullsized":"2024-09-15T23:22:02.632825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:10:46.585306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632385+0000","last_change":"2024-09-15T23:21:52.530541+0000","last_active":"2024-09-15T23:22:02.632385+0000","last_peered":"2024-09-15T23:22:02.632385+0000","last_clean":"2024-09-15T23:22:02.632385+0000","last_became_active":"2024-09-15T23:21:52.530217+0000","last_became_peered":"2024-09-15T23:21:52.530217+0000","last_unstale":"2024-09-15T23:22:02.632385+0000","last_undegraded":"2024-09-15T23:22:02.632385+0000","last_fullsized":"2024-09-15T23:22:02.632385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:14.864563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:35.120742+0000","last_change":"2024-09-15T23:21:50.562203+0000","last_active":"2024-09-15T23:22:35.120742+0000","last_peered":"2024-09-15T23:22:35.120742+0000","last_clean":"2024-09-15T23:22:35.120742+0000","last_became_active":"2024-09-15T23:21:50.561827+0000","last_became_peered":"2024-09-15T23:21:50.561827+0000","last_unstale":"2024-09-15T23:22:35.120742+0000","last_undegraded":"2024-09-15T23:22:35.120742+0000","last_fullsized":"2024-09-15T23:22:35.120742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:22.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632891+0000","last_change":"2024-09-15T23:21:59.561013+0000","last_active":"2024-09-15T23:22:02.632891+0000","last_peered":"2024-09-15T23:22:02.632891+0000","last_clean":"2024-09-15T23:22:02.632891+0000","last_became_active":"2024-09-15T23:21:59.560665+0000","last_became_peered":"2024-09-15T23:21:59.560665+0000","last_unstale":"2024-09-15T23:22:02.632891+0000","last_undegraded":"2024-09-15T23:22:02.632891+0000","last_fullsized":"2024-09-15T23:22:02.632891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:00.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.923197+0000","last_change":"2024-09-15T23:21:52.529922+0000","last_active":"2024-09-15T23:22:02.923197+0000","last_peered":"2024-09-15T23:22:02.923197+0000","last_clean":"2024-09-15T23:22:02.923197+0000","last_became_active":"2024-09-15T23:21:52.529179+0000","last_became_peered":"2024-09-15T23:21:52.529179+0000","last_unstale":"2024-09-15T23:22:02.923197+0000","last_undegraded":"2024-09-15T23:22:02.923197+0000","last_fullsized":"2024-09-15T23:22:02.923197+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.077658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.135438+0000","last_change":"2024-09-15T23:21:50.514420+0000","last_active":"2024-09-15T23:22:36.135438+0000","last_peered":"2024-09-15T23:22:36.135438+0000","last_clean":"2024-09-15T23:22:36.135438+0000","last_became_active":"2024-09-15T23:21:50.514007+0000","last_became_peered":"2024-09-15T23:21:50.514007+0000","last_unstale":"2024-09-15T23:22:36.135438+0000","last_undegraded":"2024-09-15T23:22:36.135438+0000","last_fullsized":"2024-09-15T23:22:36.135438+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:36.845674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632389+0000","last_change":"2024-09-15T23:21:59.565978+0000","last_active":"2024-09-15T23:22:02.632389+0000","last_peered":"2024-09-15T23:22:02.632389+0000","last_clean":"2024-09-15T23:22:02.632389+0000","last_became_active":"2024-09-15T23:21:59.565604+0000","last_became_peered":"2024-09-15T23:21:59.565604+0000","last_unstale":"2024-09-15T23:22:02.632389+0000","last_undegraded":"2024-09-15T23:22:02.632389+0000","last_fullsized":"2024-09-15T23:22:02.632389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:47.089504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632772+0000","last_change":"2024-09-15T23:21:52.543511+0000","last_active":"2024-09-15T23:22:02.632772+0000","last_peered":"2024-09-15T23:22:02.632772+0000","last_clean":"2024-09-15T23:22:02.632772+0000","last_became_active":"2024-09-15T23:21:52.543169+0000","last_became_peered":"2024-09-15T23:21:52.543169+0000","last_unstale":"2024-09-15T23:22:02.632772+0000","last_undegraded":"2024-09-15T23:22:02.632772+0000","last_fullsized":"2024-09-15T23:22:02.632772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:05.696772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":52,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:45.152204+0000","last_change":"2024-09-15T23:21:50.518425+0000","last_active":"2024-09-15T23:22:45.152204+0000","last_peered":"2024-09-15T23:22:45.152204+0000","last_clean":"2024-09-15T23:22:45.152204+0000","last_became_active":"2024-09-15T23:21:50.518052+0000","last_became_peered":"2024-09-15T23:21:50.518052+0000","last_unstale":"2024-09-15T23:22:45.152204+0000","last_undegraded":"2024-09-15T23:22:45.152204+0000","last_fullsized":"2024-09-15T23:22:45.152204+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:02.243869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.575942+0000","last_change":"2024-09-15T23:21:59.570310+0000","last_active":"2024-09-15T23:22:02.575942+0000","last_peered":"2024-09-15T23:22:02.575942+0000","last_clean":"2024-09-15T23:22:02.575942+0000","last_became_active":"2024-09-15T23:21:59.569909+0000","last_became_peered":"2024-09-15T23:21:59.569909+0000","last_unstale":"2024-09-15T23:22:02.575942+0000","last_undegraded":"2024-09-15T23:22:02.575942+0000","last_fullsized":"2024-09-15T23:22:02.575942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:20:27.515537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.923015+0000","last_change":"2024-09-15T23:21:59.568208+0000","last_active":"2024-09-15T23:22:02.923015+0000","last_peered":"2024-09-15T23:22:02.923015+0000","last_clean":"2024-09-15T23:22:02.923015+0000","last_became_active":"2024-09-15T23:21:59.567858+0000","last_became_peered":"2024-09-15T23:21:59.567858+0000","last_unstale":"2024-09-15T23:22:02.923015+0000","last_undegraded":"2024-09-15T23:22:02.923015+0000","last_fullsized":"2024-09-15T23:22:02.923015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:34:42.895247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632517+0000","last_change":"2024-09-15T23:21:52.532679+0000","last_active":"2024-09-15T23:22:02.632517+0000","last_peered":"2024-09-15T23:22:02.632517+0000","last_clean":"2024-09-15T23:22:02.632517+0000","last_became_active":"2024-09-15T23:21:52.532345+0000","last_became_peered":"2024-09-15T23:21:52.532345+0000","last_unstale":"2024-09-15T23:22:02.632517+0000","last_undegraded":"2024-09-15T23:22:02.632517+0000","last_fullsized":"2024-09-15T23:22:02.632517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:45.943653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:36.135672+0000","last_change":"2024-09-15T23:21:50.510836+0000","last_active":"2024-09-15T23:22:36.135672+0000","last_peered":"2024-09-15T23:22:36.135672+0000","last_clean":"2024-09-15T23:22:36.135672+0000","last_became_active":"2024-09-15T23:21:50.510514+0000","last_became_peered":"2024-09-15T23:21:50.510514+0000","last_unstale":"2024-09-15T23:22:36.135672+0000","last_undegraded":"2024-09-15T23:22:36.135672+0000","last_fullsized":"2024-09-15T23:22:36.135672+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:06.404612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633774+0000","last_change":"2024-09-15T23:21:59.568286+0000","last_active":"2024-09-15T23:22:02.633774+0000","last_peered":"2024-09-15T23:22:02.633774+0000","last_clean":"2024-09-15T23:22:02.633774+0000","last_became_active":"2024-09-15T23:21:59.567785+0000","last_became_peered":"2024-09-15T23:21:59.567785+0000","last_unstale":"2024-09-15T23:22:02.633774+0000","last_undegraded":"2024-09-15T23:22:02.633774+0000","last_fullsized":"2024-09-15T23:22:02.633774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:04.491823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.575768+0000","last_change":"2024-09-15T23:21:52.526766+0000","last_active":"2024-09-15T23:22:02.575768+0000","last_peered":"2024-09-15T23:22:02.575768+0000","last_clean":"2024-09-15T23:22:02.575768+0000","last_became_active":"2024-09-15T23:21:52.526446+0000","last_became_peered":"2024-09-15T23:21:52.526446+0000","last_unstale":"2024-09-15T23:22:02.575768+0000","last_undegraded":"2024-09-15T23:22:02.575768+0000","last_fullsized":"2024-09-15T23:22:02.575768+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:21:37.004730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:34.103702+0000","last_change":"2024-09-15T23:21:50.512928+0000","last_active":"2024-09-15T23:22:34.103702+0000","last_peered":"2024-09-15T23:22:34.103702+0000","last_clean":"2024-09-15T23:22:34.103702+0000","last_became_active":"2024-09-15T23:21:50.512558+0000","last_became_peered":"2024-09-15T23:21:50.512558+0000","last_unstale":"2024-09-15T23:22:34.103702+0000","last_undegraded":"2024-09-15T23:22:34.103702+0000","last_fullsized":"2024-09-15T23:22:34.103702+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:08.653084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:35.119796+0000","last_change":"2024-09-15T23:21:50.516920+0000","last_active":"2024-09-15T23:22:35.119796+0000","last_peered":"2024-09-15T23:22:35.119796+0000","last_clean":"2024-09-15T23:22:35.119796+0000","last_became_active":"2024-09-15T23:21:50.516573+0000","last_became_peered":"2024-09-15T23:21:50.516573+0000","last_unstale":"2024-09-15T23:22:35.119796+0000","last_undegraded":"2024-09-15T23:22:35.119796+0000","last_fullsized":"2024-09-15T23:22:35.119796+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.367903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633182+0000","last_change":"2024-09-15T23:21:52.536696+0000","last_active":"2024-09-15T23:22:02.633182+0000","last_peered":"2024-09-15T23:22:02.633182+0000","last_clean":"2024-09-15T23:22:02.633182+0000","last_became_active":"2024-09-15T23:21:52.536338+0000","last_became_peered":"2024-09-15T23:21:52.536338+0000","last_unstale":"2024-09-15T23:22:02.633182+0000","last_undegraded":"2024-09-15T23:22:02.633182+0000","last_fullsized":"2024-09-15T23:22:02.633182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:05.152749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634559+0000","last_change":"2024-09-15T23:21:59.562968+0000","last_active":"2024-09-15T23:22:02.634559+0000","last_peered":"2024-09-15T23:22:02.634559+0000","last_clean":"2024-09-15T23:22:02.634559+0000","last_became_active":"2024-09-15T23:21:59.562579+0000","last_became_peered":"2024-09-15T23:21:59.562579+0000","last_unstale":"2024-09-15T23:22:02.634559+0000","last_undegraded":"2024-09-15T23:22:02.634559+0000","last_fullsized":"2024-09-15T23:22:02.634559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:02.853057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:45.152988+0000","last_change":"2024-09-15T23:21:50.513944+0000","last_active":"2024-09-15T23:22:45.152988+0000","last_peered":"2024-09-15T23:22:45.152988+0000","last_clean":"2024-09-15T23:22:45.152988+0000","last_became_active":"2024-09-15T23:21:50.513484+0000","last_became_peered":"2024-09-15T23:21:50.513484+0000","last_unstale":"2024-09-15T23:22:45.152988+0000","last_undegraded":"2024-09-15T23:22:45.152988+0000","last_fullsized":"2024-09-15T23:22:45.152988+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:35.719344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576737+0000","last_change":"2024-09-15T23:21:52.539467+0000","last_active":"2024-09-15T23:22:02.576737+0000","last_peered":"2024-09-15T23:22:02.576737+0000","last_clean":"2024-09-15T23:22:02.576737+0000","last_became_active":"2024-09-15T23:21:52.538998+0000","last_became_peered":"2024-09-15T23:21:52.538998+0000","last_unstale":"2024-09-15T23:22:02.576737+0000","last_undegraded":"2024-09-15T23:22:02.576737+0000","last_fullsized":"2024-09-15T23:22:02.576737+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:22.213857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634927+0000","last_change":"2024-09-15T23:21:59.564842+0000","last_active":"2024-09-15T23:22:02.634927+0000","last_peered":"2024-09-15T23:22:02.634927+0000","last_clean":"2024-09-15T23:22:02.634927+0000","last_became_active":"2024-09-15T23:21:59.564408+0000","last_became_peered":"2024-09-15T23:21:59.564408+0000","last_unstale":"2024-09-15T23:22:02.634927+0000","last_undegraded":"2024-09-15T23:22:02.634927+0000","last_fullsized":"2024-09-15T23:22:02.634927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:17.909020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632850+0000","last_change":"2024-09-15T23:21:52.539731+0000","last_active":"2024-09-15T23:22:02.632850+0000","last_peered":"2024-09-15T23:22:02.632850+0000","last_clean":"2024-09-15T23:22:02.632850+0000","last_became_active":"2024-09-15T23:21:52.539414+0000","last_became_peered":"2024-09-15T23:21:52.539414+0000","last_unstale":"2024-09-15T23:22:02.632850+0000","last_undegraded":"2024-09-15T23:22:02.632850+0000","last_fullsized":"2024-09-15T23:22:02.632850+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:49.660931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634836+0000","last_change":"2024-09-15T23:21:52.532769+0000","last_active":"2024-09-15T23:22:02.634836+0000","last_peered":"2024-09-15T23:22:02.634836+0000","last_clean":"2024-09-15T23:22:02.634836+0000","last_became_active":"2024-09-15T23:21:52.532426+0000","last_became_peered":"2024-09-15T23:21:52.532426+0000","last_unstale":"2024-09-15T23:22:02.634836+0000","last_undegraded":"2024-09-15T23:22:02.634836+0000","last_fullsized":"2024-09-15T23:22:02.634836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:09.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634291+0000","last_change":"2024-09-15T23:21:52.533148+0000","last_active":"2024-09-15T23:22:02.634291+0000","last_peered":"2024-09-15T23:22:02.634291+0000","last_clean":"2024-09-15T23:22:02.634291+0000","last_became_active":"2024-09-15T23:21:52.532823+0000","last_became_peered":"2024-09-15T23:21:52.532823+0000","last_unstale":"2024-09-15T23:22:02.634291+0000","last_undegraded":"2024-09-15T23:22:02.634291+0000","last_fullsized":"2024-09-15T23:22:02.634291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:01.833945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922805+0000","last_change":"2024-09-15T23:21:52.536790+0000","last_active":"2024-09-15T23:22:02.922805+0000","last_peered":"2024-09-15T23:22:02.922805+0000","last_clean":"2024-09-15T23:22:02.922805+0000","last_became_active":"2024-09-15T23:21:52.536515+0000","last_became_peered":"2024-09-15T23:21:52.536515+0000","last_unstale":"2024-09-15T23:22:02.922805+0000","last_undegraded":"2024-09-15T23:22:02.922805+0000","last_fullsized":"2024-09-15T23:22:02.922805+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:37.207759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632499+0000","last_change":"2024-09-15T23:21:52.530181+0000","last_active":"2024-09-15T23:22:02.632499+0000","last_peered":"2024-09-15T23:22:02.632499+0000","last_clean":"2024-09-15T23:22:02.632499+0000","last_became_active":"2024-09-15T23:21:52.529790+0000","last_became_peered":"2024-09-15T23:21:52.529790+0000","last_unstale":"2024-09-15T23:22:02.632499+0000","last_undegraded":"2024-09-15T23:22:02.632499+0000","last_fullsized":"2024-09-15T23:22:02.632499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:54:02.579290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633603+0000","last_change":"2024-09-15T23:21:52.540628+0000","last_active":"2024-09-15T23:22:02.633603+0000","last_peered":"2024-09-15T23:22:02.633603+0000","last_clean":"2024-09-15T23:22:02.633603+0000","last_became_active":"2024-09-15T23:21:52.540308+0000","last_became_peered":"2024-09-15T23:21:52.540308+0000","last_unstale":"2024-09-15T23:22:02.633603+0000","last_undegraded":"2024-09-15T23:22:02.633603+0000","last_fullsized":"2024-09-15T23:22:02.633603+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:16.660637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634688+0000","last_change":"2024-09-15T23:21:52.533780+0000","last_active":"2024-09-15T23:22:02.634688+0000","last_peered":"2024-09-15T23:22:02.634688+0000","last_clean":"2024-09-15T23:22:02.634688+0000","last_became_active":"2024-09-15T23:21:52.533354+0000","last_became_peered":"2024-09-15T23:21:52.533354+0000","last_unstale":"2024-09-15T23:22:02.634688+0000","last_undegraded":"2024-09-15T23:22:02.634688+0000","last_fullsized":"2024-09-15T23:22:02.634688+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.381059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576254+0000","last_change":"2024-09-15T23:21:52.536406+0000","last_active":"2024-09-15T23:22:02.576254+0000","last_peered":"2024-09-15T23:22:02.576254+0000","last_clean":"2024-09-15T23:22:02.576254+0000","last_became_active":"2024-09-15T23:21:52.536091+0000","last_became_peered":"2024-09-15T23:21:52.536091+0000","last_unstale":"2024-09-15T23:22:02.576254+0000","last_undegraded":"2024-09-15T23:22:02.576254+0000","last_fullsized":"2024-09-15T23:22:02.576254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:30.452201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632813+0000","last_change":"2024-09-15T23:21:52.536333+0000","last_active":"2024-09-15T23:22:02.632813+0000","last_peered":"2024-09-15T23:22:02.632813+0000","last_clean":"2024-09-15T23:22:02.632813+0000","last_became_active":"2024-09-15T23:21:52.534677+0000","last_became_peered":"2024-09-15T23:21:52.534677+0000","last_unstale":"2024-09-15T23:22:02.632813+0000","last_undegraded":"2024-09-15T23:22:02.632813+0000","last_fullsized":"2024-09-15T23:22:02.632813+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:05.287914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576070+0000","last_change":"2024-09-15T23:21:52.536442+0000","last_active":"2024-09-15T23:22:02.576070+0000","last_peered":"2024-09-15T23:22:02.576070+0000","last_clean":"2024-09-15T23:22:02.576070+0000","last_became_active":"2024-09-15T23:21:52.536183+0000","last_became_peered":"2024-09-15T23:21:52.536183+0000","last_unstale":"2024-09-15T23:22:02.576070+0000","last_undegraded":"2024-09-15T23:22:02.576070+0000","last_fullsized":"2024-09-15T23:22:02.576070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:41.641370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632778+0000","last_change":"2024-09-15T23:21:52.540106+0000","last_active":"2024-09-15T23:22:02.632778+0000","last_peered":"2024-09-15T23:22:02.632778+0000","last_clean":"2024-09-15T23:22:02.632778+0000","last_became_active":"2024-09-15T23:21:52.539777+0000","last_became_peered":"2024-09-15T23:21:52.539777+0000","last_unstale":"2024-09-15T23:22:02.632778+0000","last_undegraded":"2024-09-15T23:22:02.632778+0000","last_fullsized":"2024-09-15T23:22:02.632778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:47:01.691391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632365+0000","last_change":"2024-09-15T23:21:52.531171+0000","last_active":"2024-09-15T23:22:02.632365+0000","last_peered":"2024-09-15T23:22:02.632365+0000","last_clean":"2024-09-15T23:22:02.632365+0000","last_became_active":"2024-09-15T23:21:52.530724+0000","last_became_peered":"2024-09-15T23:21:52.530724+0000","last_unstale":"2024-09-15T23:22:02.632365+0000","last_undegraded":"2024-09-15T23:22:02.632365+0000","last_fullsized":"2024-09-15T23:22:02.632365+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:11.009119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633517+0000","last_change":"2024-09-15T23:21:52.530433+0000","last_active":"2024-09-15T23:22:02.633517+0000","last_peered":"2024-09-15T23:22:02.633517+0000","last_clean":"2024-09-15T23:22:02.633517+0000","last_became_active":"2024-09-15T23:21:52.530058+0000","last_became_peered":"2024-09-15T23:21:52.530058+0000","last_unstale":"2024-09-15T23:22:02.633517+0000","last_undegraded":"2024-09-15T23:22:02.633517+0000","last_fullsized":"2024-09-15T23:22:02.633517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:51.208820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.634695+0000","last_change":"2024-09-15T23:21:52.534639+0000","last_active":"2024-09-15T23:22:02.634695+0000","last_peered":"2024-09-15T23:22:02.634695+0000","last_clean":"2024-09-15T23:22:02.634695+0000","last_became_active":"2024-09-15T23:21:52.534362+0000","last_became_peered":"2024-09-15T23:21:52.534362+0000","last_unstale":"2024-09-15T23:22:02.634695+0000","last_undegraded":"2024-09-15T23:22:02.634695+0000","last_fullsized":"2024-09-15T23:22:02.634695+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:58:12.023501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633033+0000","last_change":"2024-09-15T23:21:52.532758+0000","last_active":"2024-09-15T23:22:02.633033+0000","last_peered":"2024-09-15T23:22:02.633033+0000","last_clean":"2024-09-15T23:22:02.633033+0000","last_became_active":"2024-09-15T23:21:52.532412+0000","last_became_peered":"2024-09-15T23:21:52.532412+0000","last_unstale":"2024-09-15T23:22:02.633033+0000","last_undegraded":"2024-09-15T23:22:02.633033+0000","last_fullsized":"2024-09-15T23:22:02.633033+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:46.285711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633931+0000","last_change":"2024-09-15T23:21:52.533928+0000","last_active":"2024-09-15T23:22:02.633931+0000","last_peered":"2024-09-15T23:22:02.633931+0000","last_clean":"2024-09-15T23:22:02.633931+0000","last_became_active":"2024-09-15T23:21:52.533578+0000","last_became_peered":"2024-09-15T23:21:52.533578+0000","last_unstale":"2024-09-15T23:22:02.633931+0000","last_undegraded":"2024-09-15T23:22:02.633931+0000","last_fullsized":"2024-09-15T23:22:02.633931+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:49.476316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.631981+0000","last_change":"2024-09-15T23:21:52.533417+0000","last_active":"2024-09-15T23:22:02.631981+0000","last_peered":"2024-09-15T23:22:02.631981+0000","last_clean":"2024-09-15T23:22:02.631981+0000","last_became_active":"2024-09-15T23:21:52.533085+0000","last_became_peered":"2024-09-15T23:21:52.533085+0000","last_unstale":"2024-09-15T23:22:02.631981+0000","last_undegraded":"2024-09-15T23:22:02.631981+0000","last_fullsized":"2024-09-15T23:22:02.631981+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:20.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922732+0000","last_change":"2024-09-15T23:21:52.542340+0000","last_active":"2024-09-15T23:22:02.922732+0000","last_peered":"2024-09-15T23:22:02.922732+0000","last_clean":"2024-09-15T23:22:02.922732+0000","last_became_active":"2024-09-15T23:21:52.542048+0000","last_became_peered":"2024-09-15T23:21:52.542048+0000","last_unstale":"2024-09-15T23:22:02.922732+0000","last_undegraded":"2024-09-15T23:22:02.922732+0000","last_fullsized":"2024-09-15T23:22:02.922732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:57.557420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922634+0000","last_change":"2024-09-15T23:21:52.535576+0000","last_active":"2024-09-15T23:22:02.922634+0000","last_peered":"2024-09-15T23:22:02.922634+0000","last_clean":"2024-09-15T23:22:02.922634+0000","last_became_active":"2024-09-15T23:21:52.535223+0000","last_became_peered":"2024-09-15T23:21:52.535223+0000","last_unstale":"2024-09-15T23:22:02.922634+0000","last_undegraded":"2024-09-15T23:22:02.922634+0000","last_fullsized":"2024-09-15T23:22:02.922634+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:45.874927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632336+0000","last_change":"2024-09-15T23:21:52.519339+0000","last_active":"2024-09-15T23:22:02.632336+0000","last_peered":"2024-09-15T23:22:02.632336+0000","last_clean":"2024-09-15T23:22:02.632336+0000","last_became_active":"2024-09-15T23:21:52.518897+0000","last_became_peered":"2024-09-15T23:21:52.518897+0000","last_unstale":"2024-09-15T23:22:02.632336+0000","last_undegraded":"2024-09-15T23:22:02.632336+0000","last_fullsized":"2024-09-15T23:22:02.632336+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:30:50.169838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633340+0000","last_change":"2024-09-15T23:21:52.537012+0000","last_active":"2024-09-15T23:22:02.633340+0000","last_peered":"2024-09-15T23:22:02.633340+0000","last_clean":"2024-09-15T23:22:02.633340+0000","last_became_active":"2024-09-15T23:21:52.536245+0000","last_became_peered":"2024-09-15T23:21:52.536245+0000","last_unstale":"2024-09-15T23:22:02.633340+0000","last_undegraded":"2024-09-15T23:22:02.633340+0000","last_fullsized":"2024-09-15T23:22:02.633340+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:31.150491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.579327+0000","last_change":"2024-09-15T23:21:52.540431+0000","last_active":"2024-09-15T23:22:02.579327+0000","last_peered":"2024-09-15T23:22:02.579327+0000","last_clean":"2024-09-15T23:22:02.579327+0000","last_became_active":"2024-09-15T23:21:52.540104+0000","last_became_peered":"2024-09-15T23:21:52.540104+0000","last_unstale":"2024-09-15T23:22:02.579327+0000","last_undegraded":"2024-09-15T23:22:02.579327+0000","last_fullsized":"2024-09-15T23:22:02.579327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.550258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633402+0000","last_change":"2024-09-15T23:21:52.537590+0000","last_active":"2024-09-15T23:22:02.633402+0000","last_peered":"2024-09-15T23:22:02.633402+0000","last_clean":"2024-09-15T23:22:02.633402+0000","last_became_active":"2024-09-15T23:21:52.533804+0000","last_became_peered":"2024-09-15T23:21:52.533804+0000","last_unstale":"2024-09-15T23:22:02.633402+0000","last_undegraded":"2024-09-15T23:22:02.633402+0000","last_fullsized":"2024-09-15T23:22:02.633402+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:48.449090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.632751+0000","last_change":"2024-09-15T23:21:52.536130+0000","last_active":"2024-09-15T23:22:02.632751+0000","last_peered":"2024-09-15T23:22:02.632751+0000","last_clean":"2024-09-15T23:22:02.632751+0000","last_became_active":"2024-09-15T23:21:52.535336+0000","last_became_peered":"2024-09-15T23:21:52.535336+0000","last_unstale":"2024-09-15T23:22:02.632751+0000","last_undegraded":"2024-09-15T23:22:02.632751+0000","last_fullsized":"2024-09-15T23:22:02.632751+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:52.581994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.575723+0000","last_change":"2024-09-15T23:21:52.540485+0000","last_active":"2024-09-15T23:22:02.575723+0000","last_peered":"2024-09-15T23:22:02.575723+0000","last_clean":"2024-09-15T23:22:02.575723+0000","last_became_active":"2024-09-15T23:21:52.540178+0000","last_became_peered":"2024-09-15T23:21:52.540178+0000","last_unstale":"2024-09-15T23:22:02.575723+0000","last_undegraded":"2024-09-15T23:22:02.575723+0000","last_fullsized":"2024-09-15T23:22:02.575723+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:40:22.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.578696+0000","last_change":"2024-09-15T23:21:52.538845+0000","last_active":"2024-09-15T23:22:02.578696+0000","last_peered":"2024-09-15T23:22:02.578696+0000","last_clean":"2024-09-15T23:22:02.578696+0000","last_became_active":"2024-09-15T23:21:52.538511+0000","last_became_peered":"2024-09-15T23:21:52.538511+0000","last_unstale":"2024-09-15T23:22:02.578696+0000","last_undegraded":"2024-09-15T23:22:02.578696+0000","last_fullsized":"2024-09-15T23:22:02.578696+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:00.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633869+0000","last_change":"2024-09-15T23:21:52.533707+0000","last_active":"2024-09-15T23:22:02.633869+0000","last_peered":"2024-09-15T23:22:02.633869+0000","last_clean":"2024-09-15T23:22:02.633869+0000","last_became_active":"2024-09-15T23:21:52.533387+0000","last_became_peered":"2024-09-15T23:21:52.533387+0000","last_unstale":"2024-09-15T23:22:02.633869+0000","last_undegraded":"2024-09-15T23:22:02.633869+0000","last_fullsized":"2024-09-15T23:22:02.633869+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:53:50.585181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.576690+0000","last_change":"2024-09-15T23:21:52.530524+0000","last_active":"2024-09-15T23:22:02.576690+0000","last_peered":"2024-09-15T23:22:02.576690+0000","last_clean":"2024-09-15T23:22:02.576690+0000","last_became_active":"2024-09-15T23:21:52.530203+0000","last_became_peered":"2024-09-15T23:21:52.530203+0000","last_unstale":"2024-09-15T23:22:02.576690+0000","last_undegraded":"2024-09-15T23:22:02.576690+0000","last_fullsized":"2024-09-15T23:22:02.576690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:50.560211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633222+0000","last_change":"2024-09-15T23:21:52.537256+0000","last_active":"2024-09-15T23:22:02.633222+0000","last_peered":"2024-09-15T23:22:02.633222+0000","last_clean":"2024-09-15T23:22:02.633222+0000","last_became_active":"2024-09-15T23:21:52.536731+0000","last_became_peered":"2024-09-15T23:21:52.536731+0000","last_unstale":"2024-09-15T23:22:02.633222+0000","last_undegraded":"2024-09-15T23:22:02.633222+0000","last_fullsized":"2024-09-15T23:22:02.633222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:18.815398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922847+0000","last_change":"2024-09-15T23:21:52.537728+0000","last_active":"2024-09-15T23:22:02.922847+0000","last_peered":"2024-09-15T23:22:02.922847+0000","last_clean":"2024-09-15T23:22:02.922847+0000","last_became_active":"2024-09-15T23:21:52.537434+0000","last_became_peered":"2024-09-15T23:21:52.537434+0000","last_unstale":"2024-09-15T23:22:02.922847+0000","last_undegraded":"2024-09-15T23:22:02.922847+0000","last_fullsized":"2024-09-15T23:22:02.922847+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:25.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.922992+0000","last_change":"2024-09-15T23:21:52.534780+0000","last_active":"2024-09-15T23:22:02.922992+0000","last_peered":"2024-09-15T23:22:02.922992+0000","last_clean":"2024-09-15T23:22:02.922992+0000","last_became_active":"2024-09-15T23:21:52.534399+0000","last_became_peered":"2024-09-15T23:21:52.534399+0000","last_unstale":"2024-09-15T23:22:02.922992+0000","last_undegraded":"2024-09-15T23:22:02.922992+0000","last_fullsized":"2024-09-15T23:22:02.922992+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:47.344129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-15T23:22:02.633912+0000","last_change":"2024-09-15T23:21:52.533403+0000","last_active":"2024-09-15T23:22:02.633912+0000","last_peered":"2024-09-15T23:22:02.633912+0000","last_clean":"2024-09-15T23:22:02.633912+0000","last_became_active":"2024-09-15T23:21:52.533069+0000","last_became_peered":"2024-09-15T23:21:52.533069+0000","last_unstale":"2024-09-15T23:22:02.633912+0000","last_undegraded":"2024-09-15T23:22:02.633912+0000","last_fullsized":"2024-09-15T23:22:02.633912+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:38.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":50160,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":140,"num_read_kb":123,"num_write":233,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":177136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":202,"ondisk_log_size":202,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":246033,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579928,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":954368,"data_stored":482288,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841564,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1112842,"data_compressed":53449,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005090,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":486314,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234023,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":482596,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462956,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":1079095,"data_compressed":53449,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626481,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":477722,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855414,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":1074284,"data_compressed":53449,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018940,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":482189,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247872,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":477430,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444102,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":535645,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607627,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":535425,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":479178,"data_compressed":17599,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":58425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":45474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:22:53.472 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-15T23:22:53.473 INFO:tasks.cephadm:Setting up client nodes... 2024-09-15T23:22:53.473 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-15T23:22:53.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:53 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:22:53.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:53 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:22:53.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:53 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:53 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:22:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:53 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:22:53.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:53 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:22:53.722 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:54.099 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:54 smithi089 ceph-mon[30933]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:22:54.100 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:54 smithi089 ceph-mon[30933]: pgmap v312: 129 pgs: 129 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 KiB/s wr, 16 op/s 2024-09-15T23:22:54.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.189+0000 7fbd73447640 1 -- 172.21.15.89:0/1420029309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd6c1026d0 msgr2=0x7fbd6c108f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.189+0000 7fbd73447640 1 --2- 172.21.15.89:0/1420029309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd6c1026d0 0x7fbd6c108f60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009ca0 tx=0x7fbd6402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.189+0000 7fbd73447640 1 -- 172.21.15.89:0/1420029309 shutdown_connections 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.189+0000 7fbd73447640 1 --2- 172.21.15.89:0/1420029309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd6c1026d0 0x7fbd6c108f60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.189+0000 7fbd73447640 1 --2- 172.21.15.89:0/1420029309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd6c101d10 0x7fbd6c102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.189+0000 7fbd73447640 1 --2- 172.21.15.89:0/1420029309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd6c100b10 0x7fbd6c100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.189+0000 7fbd73447640 1 -- 172.21.15.89:0/1420029309 >> 172.21.15.89:0/1420029309 conn(0x7fbd6c0fc2a0 msgr2=0x7fbd6c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.189+0000 7fbd73447640 1 -- 172.21.15.89:0/1420029309 shutdown_connections 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.189+0000 7fbd73447640 1 -- 172.21.15.89:0/1420029309 wait complete. 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd73447640 1 Processor -- start 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd73447640 1 -- start start 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd73447640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd6c100b10 0x7fbd6c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:54.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd73447640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd6c101d10 0x7fbd6c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd73447640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd6c1026d0 0x7fbd6c19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd73447640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c10c890 con 0x7fbd6c101d10 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd73447640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c10c710 con 0x7fbd6c100b10 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd73447640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c10ca10 con 0x7fbd6c1026d0 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd709e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd6c101d10 0x7fbd6c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd709e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd6c101d10 0x7fbd6c1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58444/0 (socket says 172.21.15.89:58444) 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd709e2640 1 -- 172.21.15.89:0/3860595941 learned_addr learned my addr 172.21.15.89:0/3860595941 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd711e3640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd6c100b10 0x7fbd6c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd719e4640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd6c1026d0 0x7fbd6c19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd709e2640 1 -- 172.21.15.89:0/3860595941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd6c1026d0 msgr2=0x7fbd6c19f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd709e2640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd6c1026d0 0x7fbd6c19f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd709e2640 1 -- 172.21.15.89:0/3860595941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd6c100b10 msgr2=0x7fbd6c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:54.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd709e2640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd6c100b10 0x7fbd6c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.190+0000 7fbd709e2640 1 -- 172.21.15.89:0/3860595941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6c19fcb0 con 0x7fbd6c101d10 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.191+0000 7fbd719e4640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd6c1026d0 0x7fbd6c19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.191+0000 7fbd709e2640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd6c101d10 0x7fbd6c1985c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800b4a0 tx=0x7fbd6800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.191+0000 7fbd627fc640 1 -- 172.21.15.89:0/3860595941 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68015020 con 0x7fbd6c101d10 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.191+0000 7fbd627fc640 1 -- 172.21.15.89:0/3860595941 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd68011d10 con 0x7fbd6c101d10 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.191+0000 7fbd73447640 1 -- 172.21.15.89:0/3860595941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd6c19ffa0 con 0x7fbd6c101d10 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.191+0000 7fbd627fc640 1 -- 172.21.15.89:0/3860595941 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6800f2e0 con 0x7fbd6c101d10 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.191+0000 7fbd73447640 1 -- 172.21.15.89:0/3860595941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd6c1a04d0 con 0x7fbd6c101d10 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.192+0000 7fbd73447640 1 -- 172.21.15.89:0/3860595941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd6c1a07c0 con 0x7fbd6c101d10 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.193+0000 7fbd627fc640 1 -- 172.21.15.89:0/3860595941 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fbd68018020 con 0x7fbd6c101d10 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.194+0000 7fbd627fc640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd580a3300 0x7fbd580a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.194+0000 7fbd627fc640 1 -- 172.21.15.89:0/3860595941 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fbd680cf430 con 0x7fbd6c101d10 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.194+0000 7fbd711e3640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd580a3300 0x7fbd580a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:54.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.194+0000 7fbd711e3640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd580a3300 0x7fbd580a57c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c101b70 tx=0x7fbd5c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:54.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.198+0000 7fbd627fc640 1 -- 172.21.15.89:0/3860595941 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbd6801f020 con 0x7fbd6c101d10 2024-09-15T23:22:54.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.426+0000 7fbd73447640 1 -- 172.21.15.89:0/3860595941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fbd6c1a0990 con 0x7fbd6c101d10 2024-09-15T23:22:54.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.428+0000 7fbd627fc640 1 -- 172.21.15.89:0/3860595941 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fbd6809bec0 con 0x7fbd6c101d10 2024-09-15T23:22:54.430 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2024-09-15T23:22:54.430 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBObOdmaV56GRAAPJuBkRlKcwc6eZYgQZCJCw== 2024-09-15T23:22:54.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 -- 172.21.15.89:0/3860595941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd580a3300 msgr2=0x7fbd580a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:54.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd580a3300 0x7fbd580a57c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c101b70 tx=0x7fbd5c009110 comp rx=0 tx=0).stop 2024-09-15T23:22:54.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 -- 172.21.15.89:0/3860595941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd6c101d10 msgr2=0x7fbd6c1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:54.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd6c101d10 0x7fbd6c1985c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800b4a0 tx=0x7fbd6800b8a0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 -- 172.21.15.89:0/3860595941 shutdown_connections 2024-09-15T23:22:54.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd580a3300 0x7fbd580a57c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd6c1026d0 0x7fbd6c19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd6c101d10 0x7fbd6c1985c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 --2- 172.21.15.89:0/3860595941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd6c100b10 0x7fbd6c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 -- 172.21.15.89:0/3860595941 >> 172.21.15.89:0/3860595941 conn(0x7fbd6c0fc2a0 msgr2=0x7fbd6c06a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 -- 172.21.15.89:0/3860595941 shutdown_connections 2024-09-15T23:22:54.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:54.430+0000 7fbd73447640 1 -- 172.21.15.89:0/3860595941 wait complete. 2024-09-15T23:22:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:54 smithi120 ceph-mon[35250]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:22:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:54 smithi120 ceph-mon[35250]: pgmap v312: 129 pgs: 129 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 KiB/s wr, 16 op/s 2024-09-15T23:22:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:54 smithi174 ceph-mon[35586]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:22:54.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:54 smithi174 ceph-mon[35586]: pgmap v312: 129 pgs: 129 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 KiB/s wr, 16 op/s 2024-09-15T23:22:55.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:55 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3860595941' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-15T23:22:55.172 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:55 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3860595941' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-15T23:22:55.200 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:22:55.200 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-15T23:22:55.200 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T23:22:55.238 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T23:22:55.238 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-15T23:22:55.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-15T23:22:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:55 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3860595941' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-15T23:22:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:55 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3860595941' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-15T23:22:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:55 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3860595941' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-15T23:22:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:55 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3860595941' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-15T23:22:55.520 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:56.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.995+0000 7fbae4608640 1 -- 172.21.15.89:0/28461142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbadc106f20 msgr2=0x7fbadc10eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.995+0000 7fbae4608640 1 --2- 172.21.15.89:0/28461142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbadc106f20 0x7fbadc10eaa0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009ca0 tx=0x7fbad402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.995+0000 7fbae4608640 1 -- 172.21.15.89:0/28461142 shutdown_connections 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.995+0000 7fbae4608640 1 --2- 172.21.15.89:0/28461142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbadc106f20 0x7fbadc10eaa0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.995+0000 7fbae4608640 1 --2- 172.21.15.89:0/28461142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbadc1027a0 0x7fbadc1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.995+0000 7fbae4608640 1 --2- 172.21.15.89:0/28461142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbadc101df0 0x7fbadc1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.995+0000 7fbae4608640 1 -- 172.21.15.89:0/28461142 >> 172.21.15.89:0/28461142 conn(0x7fbadc0fd8f0 msgr2=0x7fbadc0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.995+0000 7fbae4608640 1 -- 172.21.15.89:0/28461142 shutdown_connections 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.995+0000 7fbae4608640 1 -- 172.21.15.89:0/28461142 wait complete. 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae4608640 1 Processor -- start 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae4608640 1 -- start start 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae4608640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbadc101df0 0x7fbadc198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae4608640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbadc1027a0 0x7fbadc1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae4608640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbadc106f20 0x7fbadc19cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae4608640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc111f80 con 0x7fbadc106f20 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae4608640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc111e00 con 0x7fbadc101df0 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae4608640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc112100 con 0x7fbadc1027a0 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae2ba5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbadc106f20 0x7fbadc19cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae2ba5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbadc106f20 0x7fbadc19cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60966/0 (socket says 172.21.15.89:60966) 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae23a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbadc101df0 0x7fbadc198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae2ba5640 1 -- 172.21.15.89:0/3420773921 learned_addr learned my addr 172.21.15.89:0/3420773921 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.996+0000 7fbae1ba3640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbadc1027a0 0x7fbadc1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbae2ba5640 1 -- 172.21.15.89:0/3420773921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbadc1027a0 msgr2=0x7fbadc1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbae2ba5640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbadc1027a0 0x7fbadc1989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbae2ba5640 1 -- 172.21.15.89:0/3420773921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbadc101df0 msgr2=0x7fbadc198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbae2ba5640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbadc101df0 0x7fbadc198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbae2ba5640 1 -- 172.21.15.89:0/3420773921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc19d430 con 0x7fbadc106f20 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbae2ba5640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbadc106f20 0x7fbadc19cd50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbad4031a20 tx=0x7fbad4031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbad37fe640 1 -- 172.21.15.89:0/3420773921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad4003c30 con 0x7fbadc106f20 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbae4608640 1 -- 172.21.15.89:0/3420773921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbadc19d6c0 con 0x7fbadc106f20 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbad37fe640 1 -- 172.21.15.89:0/3420773921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbad4003dd0 con 0x7fbadc106f20 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbad37fe640 1 -- 172.21.15.89:0/3420773921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad402fcf0 con 0x7fbadc106f20 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.997+0000 7fbae4608640 1 -- 172.21.15.89:0/3420773921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbadc1a5000 con 0x7fbadc106f20 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:55.998+0000 7fbae4608640 1 -- 172.21.15.89:0/3420773921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbadc1021d0 con 0x7fbadc106f20 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.000+0000 7fbad37fe640 1 -- 172.21.15.89:0/3420773921 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fbad4004050 con 0x7fbadc106f20 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.000+0000 7fbad37fe640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbac80a32d0 0x7fbac80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.000+0000 7fbae23a4640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbac80a32d0 0x7fbac80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:56.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.000+0000 7fbad37fe640 1 -- 172.21.15.89:0/3420773921 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7fbad40f10d0 con 0x7fbadc106f20 2024-09-15T23:22:56.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.001+0000 7fbae23a4640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbac80a32d0 0x7fbac80a5790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbad800bbe0 tx=0x7fbad800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:56.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.003+0000 7fbad37fe640 1 -- 172.21.15.89:0/3420773921 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbad40bdbf0 con 0x7fbadc106f20 2024-09-15T23:22:56.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.215+0000 7fbae4608640 1 -- 172.21.15.89:0/3420773921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fbadc10a160 con 0x7fbadc106f20 2024-09-15T23:22:56.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:56 smithi089 ceph-mon[30933]: pgmap v313: 129 pgs: 129 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.4 KiB/s wr, 11 op/s 2024-09-15T23:22:56.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.219+0000 7fbad37fe640 1 -- 172.21.15.89:0/3420773921 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264050 (secure 0 0 0) 0x7fbad402fe90 con 0x7fbadc106f20 2024-09-15T23:22:56.223 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:56.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.222+0000 7fbae4608640 1 -- 172.21.15.89:0/3420773921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbac80a32d0 msgr2=0x7fbac80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:56.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.222+0000 7fbae4608640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbac80a32d0 0x7fbac80a5790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbad800bbe0 tx=0x7fbad800b390 comp rx=0 tx=0).stop 2024-09-15T23:22:56.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.222+0000 7fbae4608640 1 -- 172.21.15.89:0/3420773921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbadc106f20 msgr2=0x7fbadc19cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.222+0000 7fbae4608640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbadc106f20 0x7fbadc19cd50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbad4031a20 tx=0x7fbad4031a50 comp rx=0 tx=0).stop 2024-09-15T23:22:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.222+0000 7fbae4608640 1 -- 172.21.15.89:0/3420773921 shutdown_connections 2024-09-15T23:22:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.222+0000 7fbae4608640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbac80a32d0 0x7fbac80a5790 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.222+0000 7fbae4608640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbadc106f20 0x7fbadc19cd50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.222+0000 7fbae4608640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbadc1027a0 0x7fbadc1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.222+0000 7fbae4608640 1 --2- 172.21.15.89:0/3420773921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbadc101df0 0x7fbadc198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.222+0000 7fbae4608640 1 -- 172.21.15.89:0/3420773921 >> 172.21.15.89:0/3420773921 conn(0x7fbadc0fd8f0 msgr2=0x7fbadc0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.223+0000 7fbae4608640 1 -- 172.21.15.89:0/3420773921 shutdown_connections 2024-09-15T23:22:56.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:56.223+0000 7fbae4608640 1 -- 172.21.15.89:0/3420773921 wait complete. 2024-09-15T23:22:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:56 smithi120 ceph-mon[35250]: pgmap v313: 129 pgs: 129 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.4 KiB/s wr, 11 op/s 2024-09-15T23:22:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:56 smithi174 ceph-mon[35586]: pgmap v313: 129 pgs: 129 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.4 KiB/s wr, 11 op/s 2024-09-15T23:22:56.942 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":3594493718},{"type":"v1","addr":"172.21.15.89:6801","nonce":3594493718}]},"active_addr":"172.21.15.89:6801/3594493718","active_change":"2024-09-15T23:15:00.884253+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24122,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.89:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3695758091}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3157497713}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":1267831777}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":2878808738}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":4237408073}]}]} 2024-09-15T23:22:56.947 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-15T23:22:56.947 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-15T23:22:56.948 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:22:57.195 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:57.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:57 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3420773921' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-15T23:22:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:57 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3420773921' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-15T23:22:57.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:57 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3420773921' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-15T23:22:57.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 -- 172.21.15.89:0/197026940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5968100350 msgr2=0x7f5968100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 --2- 172.21.15.89:0/197026940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5968100350 0x7f5968100730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5958009930 tx=0x7f595802a420 comp rx=0 tx=0).stop 2024-09-15T23:22:57.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 -- 172.21.15.89:0/197026940 shutdown_connections 2024-09-15T23:22:57.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 --2- 172.21.15.89:0/197026940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5968106fa0 0x7f596810ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 --2- 172.21.15.89:0/197026940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5968100d00 0x7f5968106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 --2- 172.21.15.89:0/197026940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5968100350 0x7f5968100730 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 -- 172.21.15.89:0/197026940 >> 172.21.15.89:0/197026940 conn(0x7f59680fbee0 msgr2=0x7f59680fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 -- 172.21.15.89:0/197026940 shutdown_connections 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 -- 172.21.15.89:0/197026940 wait complete. 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 Processor -- start 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.631+0000 7f596fac5640 1 -- start start 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596fac5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5968100350 0x7f5968198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596fac5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5968100d00 0x7f5968198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596fac5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5968106fa0 0x7f596819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596fac5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5968111fc0 con 0x7f5968100d00 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596fac5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5968111e40 con 0x7f5968106fa0 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596fac5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5968112140 con 0x7f5968100350 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596d060640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5968100d00 0x7f5968198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596d060640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5968100d00 0x7f5968198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60986/0 (socket says 172.21.15.89:60986) 2024-09-15T23:22:57.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596d060640 1 -- 172.21.15.89:0/2508567855 learned_addr learned my addr 172.21.15.89:0/2508567855 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596d861640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5968100350 0x7f5968198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596e062640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5968106fa0 0x7f596819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596d060640 1 -- 172.21.15.89:0/2508567855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5968100350 msgr2=0x7f5968198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596d060640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5968100350 0x7f5968198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596d060640 1 -- 172.21.15.89:0/2508567855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5968106fa0 msgr2=0x7f596819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596d060640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5968106fa0 0x7f596819cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596d060640 1 -- 172.21.15.89:0/2508567855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f596819d3f0 con 0x7f5968100d00 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596e062640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5968106fa0 0x7f596819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596d060640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5968100d00 0x7f5968198980 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f596400c970 tx=0x7f596400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f595effd640 1 -- 172.21.15.89:0/2508567855 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964014070 con 0x7f5968100d00 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f595effd640 1 -- 172.21.15.89:0/2508567855 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5964004440 con 0x7f5968100d00 2024-09-15T23:22:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f596fac5640 1 -- 172.21.15.89:0/2508567855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f596819d6e0 con 0x7f5968100d00 2024-09-15T23:22:57.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.632+0000 7f595effd640 1 -- 172.21.15.89:0/2508567855 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596400f450 con 0x7f5968100d00 2024-09-15T23:22:57.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.633+0000 7f596fac5640 1 -- 172.21.15.89:0/2508567855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59681a4fc0 con 0x7f5968100d00 2024-09-15T23:22:57.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.633+0000 7f596fac5640 1 -- 172.21.15.89:0/2508567855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5968100730 con 0x7f5968100d00 2024-09-15T23:22:57.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.635+0000 7f595effd640 1 -- 172.21.15.89:0/2508567855 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f5964017070 con 0x7f5968100d00 2024-09-15T23:22:57.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.635+0000 7f595effd640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59540a32d0 0x7f59540a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.635+0000 7f596d861640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59540a32d0 0x7f59540a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.636+0000 7f595effd640 1 -- 172.21.15.89:0/2508567855 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f59640d0020 con 0x7f5968100d00 2024-09-15T23:22:57.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.636+0000 7f596d861640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59540a32d0 0x7f59540a5790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5958000c00 tx=0x7f5958031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:57.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.638+0000 7f595effd640 1 -- 172.21.15.89:0/2508567855 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f596401e020 con 0x7f5968100d00 2024-09-15T23:22:57.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.807+0000 7f596fac5640 1 -- 172.21.15.89:0/2508567855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59681997a0 con 0x7f5968100d00 2024-09-15T23:22:57.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.808+0000 7f595effd640 1 -- 172.21.15.89:0/2508567855 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24199 (secure 0 0 0) 0x7f59681997a0 con 0x7f5968100d00 2024-09-15T23:22:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:57.810 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":86,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:22:02.246640+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.810+0000 7f596fac5640 1 -- 172.21.15.89:0/2508567855 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59540a32d0 msgr2=0x7f59540a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.810+0000 7f596fac5640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59540a32d0 0x7f59540a5790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5958000c00 tx=0x7f5958031490 comp rx=0 tx=0).stop 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.810+0000 7f596fac5640 1 -- 172.21.15.89:0/2508567855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5968100d00 msgr2=0x7f5968198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.810+0000 7f596fac5640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5968100d00 0x7f5968198980 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f596400c970 tx=0x7f596400ce30 comp rx=0 tx=0).stop 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.811+0000 7f596fac5640 1 -- 172.21.15.89:0/2508567855 shutdown_connections 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.811+0000 7f596fac5640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59540a32d0 0x7f59540a5790 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.811+0000 7f596fac5640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5968106fa0 0x7f596819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.811+0000 7f596fac5640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5968100d00 0x7f5968198980 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.811+0000 7f596fac5640 1 --2- 172.21.15.89:0/2508567855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5968100350 0x7f5968198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.811+0000 7f596fac5640 1 -- 172.21.15.89:0/2508567855 >> 172.21.15.89:0/2508567855 conn(0x7f59680fbee0 msgr2=0x7f5968106040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.811+0000 7f596fac5640 1 -- 172.21.15.89:0/2508567855 shutdown_connections 2024-09-15T23:22:57.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:57.811+0000 7f596fac5640 1 -- 172.21.15.89:0/2508567855 wait complete. 2024-09-15T23:22:58.335 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:58 smithi089 ceph-mon[30933]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-15T23:22:58.335 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:58 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2508567855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:58 smithi120 ceph-mon[35250]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-15T23:22:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:58 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2508567855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:58.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:58 smithi174 ceph-mon[35586]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-15T23:22:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:58 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2508567855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:22:58.522 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-15T23:22:58.522 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:22:58.771 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:59.169 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:22:59 smithi089 ceph-mon[30933]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.251+0000 7f6fecff1640 1 -- 172.21.15.89:0/3319515644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe8108300 msgr2=0x7f6fe810ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.251+0000 7f6fecff1640 1 --2- 172.21.15.89:0/3319515644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe8108300 0x7f6fe810ec70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009cd0 tx=0x7f6fe002a8f0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.252+0000 7f6fecff1640 1 -- 172.21.15.89:0/3319515644 shutdown_connections 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.252+0000 7f6fecff1640 1 --2- 172.21.15.89:0/3319515644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe8108300 0x7f6fe810ec70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.252+0000 7f6fecff1640 1 --2- 172.21.15.89:0/3319515644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fe8103780 0x7f6fe8107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.252+0000 7f6fecff1640 1 --2- 172.21.15.89:0/3319515644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe8102dd0 0x7f6fe81031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.252+0000 7f6fecff1640 1 -- 172.21.15.89:0/3319515644 >> 172.21.15.89:0/3319515644 conn(0x7f6fe80fc7d0 msgr2=0x7f6fe80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.252+0000 7f6fecff1640 1 -- 172.21.15.89:0/3319515644 shutdown_connections 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.252+0000 7f6fecff1640 1 -- 172.21.15.89:0/3319515644 wait complete. 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.252+0000 7f6fecff1640 1 Processor -- start 2024-09-15T23:22:59.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fecff1640 1 -- start start 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fecff1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe8102dd0 0x7f6fe81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fecff1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe8103780 0x7f6fe8198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fecff1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fe8108300 0x7f6fe819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fecff1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81120f0 con 0x7f6fe8103780 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fecff1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8111f70 con 0x7f6fe8108300 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fecff1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8112270 con 0x7f6fe8102dd0 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe8103780 0x7f6fe8198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe8103780 0x7f6fe8198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60992/0 (socket says 172.21.15.89:60992) 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe5d9b640 1 -- 172.21.15.89:0/3015957550 learned_addr learned my addr 172.21.15.89:0/3015957550 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe659c640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe8102dd0 0x7f6fe81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe6d9d640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fe8108300 0x7f6fe819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe5d9b640 1 -- 172.21.15.89:0/3015957550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe8102dd0 msgr2=0x7f6fe81985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe5d9b640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe8102dd0 0x7f6fe81985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe5d9b640 1 -- 172.21.15.89:0/3015957550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fe8108300 msgr2=0x7f6fe819ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe5d9b640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fe8108300 0x7f6fe819ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe5d9b640 1 -- 172.21.15.89:0/3015957550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe819d570 con 0x7f6fe8103780 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fe5d9b640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe8103780 0x7f6fe8198b00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0006fd0 tx=0x7f6fd000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.253+0000 7f6fdf7fe640 1 -- 172.21.15.89:0/3015957550 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0011830 con 0x7f6fe8103780 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.254+0000 7f6fdf7fe640 1 -- 172.21.15.89:0/3015957550 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fd0016070 con 0x7f6fe8103780 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.254+0000 7f6fdf7fe640 1 -- 172.21.15.89:0/3015957550 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000f2e0 con 0x7f6fe8103780 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.254+0000 7f6fecff1640 1 -- 172.21.15.89:0/3015957550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe819d860 con 0x7f6fe8103780 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.254+0000 7f6fecff1640 1 -- 172.21.15.89:0/3015957550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe81a5140 con 0x7f6fe8103780 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.255+0000 7f6fecff1640 1 -- 172.21.15.89:0/3015957550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe81031b0 con 0x7f6fe8103780 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.256+0000 7f6fdf7fe640 1 -- 172.21.15.89:0/3015957550 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f6fd001b460 con 0x7f6fe8103780 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.256+0000 7f6fdf7fe640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6fd40a32d0 0x7f6fd40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.257+0000 7f6fe659c640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6fd40a32d0 0x7f6fd40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.257+0000 7f6fdf7fe640 1 -- 172.21.15.89:0/3015957550 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8802+0+0 (secure 0 0 0) 0x7f6fd00cf270 con 0x7f6fe8103780 2024-09-15T23:22:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.257+0000 7f6fe659c640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6fd40a32d0 0x7f6fd40a5790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80058c0 tx=0x7f6fd800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:59.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.261+0000 7f6fdf7fe640 1 -- 172.21.15.89:0/3015957550 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6fd009bd00 con 0x7f6fe8103780 2024-09-15T23:22:59.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.435+0000 7f6fecff1640 1 -- 172.21.15.89:0/3015957550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6fe810a290 con 0x7f6fe8103780 2024-09-15T23:22:59.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.437+0000 7f6fdf7fe640 1 -- 172.21.15.89:0/3015957550 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+24199 (secure 0 0 0) 0x7f6fd009b600 con 0x7f6fe8103780 2024-09-15T23:22:59.438 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:22:59.438 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":86,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:22:02.246640+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:22:59.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 -- 172.21.15.89:0/3015957550 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6fd40a32d0 msgr2=0x7f6fd40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6fd40a32d0 0x7f6fd40a5790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80058c0 tx=0x7f6fd800a2d0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 -- 172.21.15.89:0/3015957550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe8103780 msgr2=0x7f6fe8198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe8103780 0x7f6fe8198b00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0006fd0 tx=0x7f6fd000b8e0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 -- 172.21.15.89:0/3015957550 shutdown_connections 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6fd40a32d0 0x7f6fd40a5790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6fe8108300 0x7f6fe819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fe8103780 0x7f6fe8198b00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 --2- 172.21.15.89:0/3015957550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe8102dd0 0x7f6fe81985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 -- 172.21.15.89:0/3015957550 >> 172.21.15.89:0/3015957550 conn(0x7f6fe80fc7d0 msgr2=0x7f6fe8105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 -- 172.21.15.89:0/3015957550 shutdown_connections 2024-09-15T23:22:59.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:22:59.439+0000 7f6fecff1640 1 -- 172.21.15.89:0/3015957550 wait complete. 2024-09-15T23:22:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:22:59 smithi120 ceph-mon[35250]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-09-15T23:22:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:22:59 smithi174 ceph-mon[35586]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-09-15T23:23:00.170 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-15T23:23:00.171 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-15T23:23:00.171 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-15T23:23:00.171 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-15T23:23:00.172 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-15T23:23:00.172 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-15T23:23:00.173 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-15T23:23:00.173 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-15T23:23:00.174 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-15T23:23:00.174 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-15T23:23:00.175 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-15T23:23:00.175 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-15T23:23:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:00 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3015957550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:23:00.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:00 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3015957550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:23:00.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:00 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3015957550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:23:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:00.845 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:01.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-09-15T23:23:01.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-09-15T23:23:01.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-15T23:23:01.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-09-15T23:23:01.279 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-09-15T23:23:01.280 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-09-15T23:23:01.280 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-09-15T23:23:01.280 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-09-15T23:23:01.280 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-09-15T23:23:01.280 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-09-15T23:23:01.280 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:01 smithi089 ceph-mon[30933]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-09-15T23:23:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-09-15T23:23:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-09-15T23:23:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-15T23:23:01.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-09-15T23:23:01.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-09-15T23:23:01.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-09-15T23:23:01.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-09-15T23:23:01.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-09-15T23:23:01.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-09-15T23:23:01.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-09-15T23:23:01.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:01 smithi120 ceph-mon[35250]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-09-15T23:23:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-09-15T23:23:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-09-15T23:23:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-15T23:23:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-09-15T23:23:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-09-15T23:23:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-09-15T23:23:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-09-15T23:23:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-09-15T23:23:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-09-15T23:23:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-09-15T23:23:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:01 smithi174 ceph-mon[35586]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-09-15T23:23:01.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/467552442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f34106f10 msgr2=0x7f8f3410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:01.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 --2- 172.21.15.89:0/467552442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f34106f10 0x7f8f3410ea90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c009ca0 tx=0x7f8f2c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:23:01.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/467552442 shutdown_connections 2024-09-15T23:23:01.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 --2- 172.21.15.89:0/467552442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f34106f10 0x7f8f3410ea90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:01.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 --2- 172.21.15.89:0/467552442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f34102790 0x7f8f341067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:01.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 --2- 172.21.15.89:0/467552442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f34101de0 0x7f8f341021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:01.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/467552442 >> 172.21.15.89:0/467552442 conn(0x7f8f340fd8c0 msgr2=0x7f8f340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:01.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/467552442 shutdown_connections 2024-09-15T23:23:01.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/467552442 wait complete. 2024-09-15T23:23:01.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 Processor -- start 2024-09-15T23:23:01.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.739+0000 7f8f3b1fb640 1 -- start start 2024-09-15T23:23:01.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f3b1fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f34101de0 0x7f8f340744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:01.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f3b1fb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f34102790 0x7f8f34074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:01.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f3b1fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f34106f10 0x7f8f34074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:01.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f3b1fb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8f34111f70 con 0x7f8f34106f10 2024-09-15T23:23:01.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f3b1fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8f34111df0 con 0x7f8f34101de0 2024-09-15T23:23:01.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f3b1fb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8f341120f0 con 0x7f8f34102790 2024-09-15T23:23:01.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f39798640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f34106f10 0x7f8f34074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:01.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f39798640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f34106f10 0x7f8f34074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32778/0 (socket says 172.21.15.89:32778) 2024-09-15T23:23:01.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f39798640 1 -- 172.21.15.89:0/2296248309 learned_addr learned my addr 172.21.15.89:0/2296248309 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:01.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f38f97640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f34101de0 0x7f8f340744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:01.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f33fff640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f34102790 0x7f8f34074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:01.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f39798640 1 -- 172.21.15.89:0/2296248309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f34102790 msgr2=0x7f8f34074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:01.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f39798640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f34102790 0x7f8f34074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:01.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f39798640 1 -- 172.21.15.89:0/2296248309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f34101de0 msgr2=0x7f8f340744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:01.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f39798640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f34101de0 0x7f8f340744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:01.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f39798640 1 -- 172.21.15.89:0/2296248309 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f341a68a0 con 0x7f8f34106f10 2024-09-15T23:23:01.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f38f97640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f34101de0 0x7f8f340744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:01.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f33fff640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f34102790 0x7f8f34074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:01.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f39798640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f34106f10 0x7f8f34074f70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c031a60 tx=0x7f8f2c031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:01.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f31ffb640 1 -- 172.21.15.89:0/2296248309 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c003c30 con 0x7f8f34106f10 2024-09-15T23:23:01.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f31ffb640 1 -- 172.21.15.89:0/2296248309 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f2c003dd0 con 0x7f8f34106f10 2024-09-15T23:23:01.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f341a6b30 con 0x7f8f34106f10 2024-09-15T23:23:01.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.740+0000 7f8f31ffb640 1 -- 172.21.15.89:0/2296248309 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0047e0 con 0x7f8f34106f10 2024-09-15T23:23:01.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.741+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f341a70c0 con 0x7f8f34106f10 2024-09-15T23:23:01.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.741+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8f340fee30 con 0x7f8f34106f10 2024-09-15T23:23:01.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.743+0000 7f8f31ffb640 1 -- 172.21.15.89:0/2296248309 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8f2c02f790 con 0x7f8f34106f10 2024-09-15T23:23:01.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.743+0000 7f8f31ffb640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8f200a3350 0x7f8f200a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.743+0000 7f8f38f97640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8f200a3350 0x7f8f200a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.743+0000 7f8f31ffb640 1 -- 172.21.15.89:0/2296248309 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9092+0+0 (secure 0 0 0) 0x7f8f2c0f1320 con 0x7f8f34106f10 2024-09-15T23:23:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.743+0000 7f8f31ffb640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] conn(0x7f8f200ae3f0 0x7f8f200b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.743+0000 7f8f31ffb640 1 -- 172.21.15.89:0/2296248309 --> [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f200b0f60 con 0x7f8f200ae3f0 2024-09-15T23:23:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.743+0000 7f8f31ffb640 1 -- 172.21.15.89:0/2296248309 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f8f2c0f1800 con 0x7f8f34106f10 2024-09-15T23:23:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.744+0000 7f8f38f97640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8f200a3350 0x7f8f200a5810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00abe0 tx=0x7f8f1c00a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.744+0000 7f8f33fff640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] conn(0x7f8f200ae3f0 0x7f8f200b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.744+0000 7f8f33fff640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] conn(0x7f8f200ae3f0 0x7f8f200b0890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.745+0000 7f8f31ffb640 1 -- 172.21.15.89:0/2296248309 <== osd.1 v2:172.21.15.120:6800/2529464017 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8f200b0f60 con 0x7f8f200ae3f0 2024-09-15T23:23:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.758+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 --> [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8f341032c0 con 0x7f8f200ae3f0 2024-09-15T23:23:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.759+0000 7f8f31ffb640 1 -- 172.21.15.89:0/2296248309 <== osd.1 v2:172.21.15.120:6800/2529464017 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8f341032c0 con 0x7f8f200ae3f0 2024-09-15T23:23:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.759+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 >> [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] conn(0x7f8f200ae3f0 msgr2=0x7f8f200b0890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.759+0000 7f8f3b1fb640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] conn(0x7f8f200ae3f0 0x7f8f200b0890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.760+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8f200a3350 msgr2=0x7f8f200a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.760+0000 7f8f3b1fb640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8f200a3350 0x7f8f200a5810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00abe0 tx=0x7f8f1c00a5c0 comp rx=0 tx=0).stop 2024-09-15T23:23:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.760+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f34106f10 msgr2=0x7f8f34074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.760+0000 7f8f3b1fb640 1 --2- 172.21.15.89:0/2296248309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f34106f10 0x7f8f34074f70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c031a60 tx=0x7f8f2c031a90 comp rx=0 tx=0).stop 2024-09-15T23:23:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.760+0000 7f8f39798640 1 -- 172.21.15.89:0/2296248309 reap_dead start 2024-09-15T23:23:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.760+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 shutdown_connections 2024-09-15T23:23:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.760+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 >> 172.21.15.89:0/2296248309 conn(0x7f8f340fd8c0 msgr2=0x7f8f340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.760+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 shutdown_connections 2024-09-15T23:23:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:01.760+0000 7f8f3b1fb640 1 -- 172.21.15.89:0/2296248309 wait complete. 2024-09-15T23:23:02.447 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-09-15T23:23:02.447 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-09-15T23:23:02.447 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-15T23:23:02.447 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-09-15T23:23:02.448 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-09-15T23:23:02.448 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-09-15T23:23:02.448 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-09-15T23:23:02.448 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-09-15T23:23:02.448 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-09-15T23:23:02.448 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-09-15T23:23:02.448 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:02 smithi089 ceph-mon[30933]: osdmap e87: 12 total, 12 up, 12 in 2024-09-15T23:23:02.469 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:02.470 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:02.470 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:02.470 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:02.471 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-09-15T23:23:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-09-15T23:23:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-15T23:23:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-09-15T23:23:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-09-15T23:23:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-09-15T23:23:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-09-15T23:23:02.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-09-15T23:23:02.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-09-15T23:23:02.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-09-15T23:23:02.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:02 smithi120 ceph-mon[35250]: osdmap e87: 12 total, 12 up, 12 in 2024-09-15T23:23:02.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-09-15T23:23:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-09-15T23:23:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-15T23:23:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-09-15T23:23:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-09-15T23:23:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-09-15T23:23:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-09-15T23:23:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-09-15T23:23:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-09-15T23:23:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-09-15T23:23:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:02 smithi174 ceph-mon[35586]: osdmap e87: 12 total, 12 up, 12 in 2024-09-15T23:23:02.491 INFO:teuthology.orchestra.run.smithi089.stdout:51539607630 2024-09-15T23:23:02.491 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-15T23:23:02.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.655+0000 7f8609e02640 1 -- 172.21.15.89:0/2986068370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86041036e0 msgr2=0x7f8604107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:02.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.655+0000 7f8609e02640 1 --2- 172.21.15.89:0/2986068370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86041036e0 0x7f8604107b90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009930 tx=0x7f85f802a420 comp rx=0 tx=0).stop 2024-09-15T23:23:02.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.655+0000 7f8609e02640 1 -- 172.21.15.89:0/2986068370 shutdown_connections 2024-09-15T23:23:02.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.655+0000 7f8609e02640 1 --2- 172.21.15.89:0/2986068370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86041082a0 0x7f860410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:02.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.655+0000 7f8609e02640 1 --2- 172.21.15.89:0/2986068370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86041036e0 0x7f8604107b90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:02.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.655+0000 7f8609e02640 1 --2- 172.21.15.89:0/2986068370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8604102d30 0x7f8604103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.655+0000 7f8609e02640 1 -- 172.21.15.89:0/2986068370 >> 172.21.15.89:0/2986068370 conn(0x7f86040fc7d0 msgr2=0x7f86040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.655+0000 7f8609e02640 1 -- 172.21.15.89:0/2986068370 shutdown_connections 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.655+0000 7f8609e02640 1 -- 172.21.15.89:0/2986068370 wait complete. 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.656+0000 7f8609e02640 1 Processor -- start 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.656+0000 7f8609e02640 1 -- start start 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.656+0000 7f8609e02640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8604102d30 0x7f8604198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.656+0000 7f8609e02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86041036e0 0x7f8604198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.656+0000 7f8609e02640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86041082a0 0x7f860419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.656+0000 7f8609e02640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f86041120a0 con 0x7f86041036e0 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.656+0000 7f8609e02640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8604111f20 con 0x7f8604102d30 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.656+0000 7f8609e02640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8604112220 con 0x7f86041082a0 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86041036e0 0x7f8604198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:02.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86041036e0 0x7f8604198b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32794/0 (socket says 172.21.15.89:32794) 2024-09-15T23:23:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8602ffd640 1 -- 172.21.15.89:0/3266345044 learned_addr learned my addr 172.21.15.89:0/3266345044 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8603fff640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86041082a0 0x7f860419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f86037fe640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8604102d30 0x7f8604198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8602ffd640 1 -- 172.21.15.89:0/3266345044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86041082a0 msgr2=0x7f860419cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8602ffd640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86041082a0 0x7f860419cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:02.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8602ffd640 1 -- 172.21.15.89:0/3266345044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8604102d30 msgr2=0x7f8604198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8602ffd640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8604102d30 0x7f8604198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8602ffd640 1 -- 172.21.15.89:0/3266345044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860419d5f0 con 0x7f86041036e0 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f86037fe640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8604102d30 0x7f8604198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8602ffd640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86041036e0 0x7f8604198b80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f85f8005940 tx=0x7f85f80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8600ff9640 1 -- 172.21.15.89:0/3266345044 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f803f030 con 0x7f86041036e0 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860419d880 con 0x7f86041036e0 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8600ff9640 1 -- 172.21.15.89:0/3266345044 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85f80044c0 con 0x7f86041036e0 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8600ff9640 1 -- 172.21.15.89:0/3266345044 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8035070 con 0x7f86041036e0 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.657+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86041a51c0 con 0x7f86041036e0 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.658+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8604062900 con 0x7f86041036e0 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.660+0000 7f8600ff9640 1 -- 172.21.15.89:0/3266345044 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f85f8042030 con 0x7f86041036e0 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.661+0000 7f8600ff9640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85ec0a32d0 0x7f85ec0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:02.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.661+0000 7f8600ff9640 1 -- 172.21.15.89:0/3266345044 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f85f80f0860 con 0x7f86041036e0 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.661+0000 7f86037fe640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85ec0a32d0 0x7f85ec0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.661+0000 7f8600ff9640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] conn(0x7f85ec0ae370 0x7f85ec0b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.661+0000 7f8600ff9640 1 -- 172.21.15.89:0/3266345044 --> [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f85ec0b0ee0 con 0x7f85ec0ae370 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.661+0000 7f8600ff9640 1 -- 172.21.15.89:0/3266345044 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f85f80f0cb0 con 0x7f86041036e0 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.662+0000 7f86037fe640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85ec0a32d0 0x7f85ec0a5790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009c20 tx=0x7f85f4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.662+0000 7f8603fff640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] conn(0x7f85ec0ae370 0x7f85ec0b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.662+0000 7f8603fff640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] conn(0x7f85ec0ae370 0x7f85ec0b0810 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.663+0000 7f8600ff9640 1 -- 172.21.15.89:0/3266345044 <== osd.5 v2:172.21.15.174:6808/3950370591 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f85ec0b0ee0 con 0x7f85ec0ae370 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.676+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 --> [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8604062d00 con 0x7f85ec0ae370 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.677+0000 7f8600ff9640 1 -- 172.21.15.89:0/3266345044 <== osd.5 v2:172.21.15.174:6808/3950370591 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8604062d00 con 0x7f85ec0ae370 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 >> [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] conn(0x7f85ec0ae370 msgr2=0x7f85ec0b0810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8609e02640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] conn(0x7f85ec0ae370 0x7f85ec0b0810 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:02.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85ec0a32d0 msgr2=0x7f85ec0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8609e02640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85ec0a32d0 0x7f85ec0a5790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009c20 tx=0x7f85f4005f90 comp rx=0 tx=0).stop 2024-09-15T23:23:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86041036e0 msgr2=0x7f8604198b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8609e02640 1 --2- 172.21.15.89:0/3266345044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86041036e0 0x7f8604198b80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f85f8005940 tx=0x7f85f80040c0 comp rx=0 tx=0).stop 2024-09-15T23:23:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8603fff640 1 -- 172.21.15.89:0/3266345044 reap_dead start 2024-09-15T23:23:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 shutdown_connections 2024-09-15T23:23:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 >> 172.21.15.89:0/3266345044 conn(0x7f86040fc7d0 msgr2=0x7f8604106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 shutdown_connections 2024-09-15T23:23:02.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:02.678+0000 7f8609e02640 1 -- 172.21.15.89:0/3266345044 wait complete. 2024-09-15T23:23:02.689 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:02.689 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:02.690 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:02.691 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:03.017 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:03.430 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:03 smithi089 ceph-mon[30933]: osdmap e88: 12 total, 12 up, 12 in 2024-09-15T23:23:03.430 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:03 smithi089 ceph-mon[30933]: pgmap v319: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1023 B/s wr, 12 op/s 2024-09-15T23:23:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:03 smithi120 ceph-mon[35250]: osdmap e88: 12 total, 12 up, 12 in 2024-09-15T23:23:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:03 smithi120 ceph-mon[35250]: pgmap v319: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1023 B/s wr, 12 op/s 2024-09-15T23:23:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:03 smithi174 ceph-mon[35586]: osdmap e88: 12 total, 12 up, 12 in 2024-09-15T23:23:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:03 smithi174 ceph-mon[35586]: pgmap v319: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1023 B/s wr, 12 op/s 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 -- 172.21.15.89:0/3172364175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344010c180 msgr2=0x7f344010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 --2- 172.21.15.89:0/3172364175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344010c180 0x7f344010e600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3430009900 tx=0x7f343002a480 comp rx=0 tx=0).stop 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 -- 172.21.15.89:0/3172364175 shutdown_connections 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 --2- 172.21.15.89:0/3172364175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f344010c180 0x7f344010e600 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 --2- 172.21.15.89:0/3172364175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34401036c0 0x7f344010bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 --2- 172.21.15.89:0/3172364175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3440102da0 0x7f3440103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 -- 172.21.15.89:0/3172364175 >> 172.21.15.89:0/3172364175 conn(0x7f34400fc7a0 msgr2=0x7f34400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 -- 172.21.15.89:0/3172364175 shutdown_connections 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 -- 172.21.15.89:0/3172364175 wait complete. 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 Processor -- start 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.905+0000 7f344870d640 1 -- start start 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f344870d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3440102da0 0x7f3440198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f344870d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34401036c0 0x7f34401989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f344870d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344010c180 0x7f344019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f344870d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3440111fa0 con 0x7f34401036c0 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f344870d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3440111e20 con 0x7f344010c180 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f344870d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3440112120 con 0x7f3440102da0 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f3445ca8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34401036c0 0x7f34401989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f3445ca8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34401036c0 0x7f34401989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32814/0 (socket says 172.21.15.89:32814) 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f3445ca8640 1 -- 172.21.15.89:0/652920521 learned_addr learned my addr 172.21.15.89:0/652920521 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f34464a9640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3440102da0 0x7f3440198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f3446caa640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344010c180 0x7f344019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f3445ca8640 1 -- 172.21.15.89:0/652920521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3440102da0 msgr2=0x7f3440198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f3445ca8640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3440102da0 0x7f3440198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f3445ca8640 1 -- 172.21.15.89:0/652920521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344010c180 msgr2=0x7f344019cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f3445ca8640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f344010c180 0x7f344019cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f3445ca8640 1 -- 172.21.15.89:0/652920521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344019d430 con 0x7f34401036c0 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f34464a9640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3440102da0 0x7f3440198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f3445ca8640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34401036c0 0x7f34401989c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f342800cce0 tx=0x7f342800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f343f7fe640 1 -- 172.21.15.89:0/652920521 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428004b20 con 0x7f34401036c0 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f343f7fe640 1 -- 172.21.15.89:0/652920521 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3428004cc0 con 0x7f34401036c0 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344019d720 con 0x7f34401036c0 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f343f7fe640 1 -- 172.21.15.89:0/652920521 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342800f7d0 con 0x7f34401036c0 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.906+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34401a5000 con 0x7f34401036c0 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.907+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3440062900 con 0x7f34401036c0 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.909+0000 7f343f7fe640 1 -- 172.21.15.89:0/652920521 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f342800f970 con 0x7f34401036c0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.909+0000 7f343f7fe640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f342c0a32d0 0x7f342c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.909+0000 7f34464a9640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f342c0a32d0 0x7f342c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.909+0000 7f343f7fe640 1 -- 172.21.15.89:0/652920521 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f34280d0340 con 0x7f34401036c0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.910+0000 7f343f7fe640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] conn(0x7f342c0ae4a0 0x7f342c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.910+0000 7f343f7fe640 1 -- 172.21.15.89:0/652920521 --> [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f342c0b1010 con 0x7f342c0ae4a0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.910+0000 7f343f7fe640 1 -- 172.21.15.89:0/652920521 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f34280d0790 con 0x7f34401036c0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.910+0000 7f3446caa640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] conn(0x7f342c0ae4a0 0x7f342c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.910+0000 7f34464a9640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f342c0a32d0 0x7f342c0a5790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3438009930 tx=0x7f3438008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.910+0000 7f3446caa640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] conn(0x7f342c0ae4a0 0x7f342c0b0940 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.911+0000 7f343f7fe640 1 -- 172.21.15.89:0/652920521 <== osd.0 v2:172.21.15.89:6802/261479611 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f342c0b1010 con 0x7f342c0ae4a0 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.924+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 --> [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3440062d80 con 0x7f342c0ae4a0 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f343f7fe640 1 -- 172.21.15.89:0/652920521 <== osd.0 v2:172.21.15.89:6802/261479611 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3440062d80 con 0x7f342c0ae4a0 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] conn(0x7f342c0ae4a0 msgr2=0x7f342c0b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f344870d640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] conn(0x7f342c0ae4a0 0x7f342c0b0940 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f342c0a32d0 msgr2=0x7f342c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f344870d640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f342c0a32d0 0x7f342c0a5790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3438009930 tx=0x7f3438008040 comp rx=0 tx=0).stop 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34401036c0 msgr2=0x7f34401989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f344870d640 1 --2- 172.21.15.89:0/652920521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f34401036c0 0x7f34401989c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f342800cce0 tx=0x7f342800a620 comp rx=0 tx=0).stop 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f3446caa640 1 -- 172.21.15.89:0/652920521 reap_dead start 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 shutdown_connections 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 >> 172.21.15.89:0/652920521 conn(0x7f34400fc7a0 msgr2=0x7f34400fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 shutdown_connections 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:03.925+0000 7f344870d640 1 -- 172.21.15.89:0/652920521 wait complete. 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.113+0000 7f1e36034640 1 -- 172.21.15.89:0/1011030835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e30102d40 msgr2=0x7f1e30103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.113+0000 7f1e36034640 1 --2- 172.21.15.89:0/1011030835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e30102d40 0x7f1e30103120 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009900 tx=0x7f1e1802a480 comp rx=0 tx=0).stop 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.113+0000 7f1e36034640 1 -- 172.21.15.89:0/1011030835 shutdown_connections 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.113+0000 7f1e36034640 1 --2- 172.21.15.89:0/1011030835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e3010c170 0x7f1e3010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.113+0000 7f1e36034640 1 --2- 172.21.15.89:0/1011030835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e30103660 0x7f1e3010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.113+0000 7f1e36034640 1 --2- 172.21.15.89:0/1011030835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e30102d40 0x7f1e30103120 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.113+0000 7f1e36034640 1 -- 172.21.15.89:0/1011030835 >> 172.21.15.89:0/1011030835 conn(0x7f1e300fc7a0 msgr2=0x7f1e300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.113+0000 7f1e36034640 1 -- 172.21.15.89:0/1011030835 shutdown_connections 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.113+0000 7f1e36034640 1 -- 172.21.15.89:0/1011030835 wait complete. 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.114+0000 7f1e36034640 1 Processor -- start 2024-09-15T23:23:04.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.114+0000 7f1e36034640 1 -- start start 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.114+0000 7f1e36034640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e30102d40 0x7f1e30198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e36034640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e30103660 0x7f1e30198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e36034640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e3010c170 0x7f1e3019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e36034640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1e30111f70 con 0x7f1e30103660 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e36034640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1e30111df0 con 0x7f1e3010c170 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e36034640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1e301120f0 con 0x7f1e30102d40 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e30103660 0x7f1e30198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e30103660 0x7f1e30198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:32830/0 (socket says 172.21.15.89:32830) 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2effd640 1 -- 172.21.15.89:0/1111601824 learned_addr learned my addr 172.21.15.89:0/1111601824 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2f7fe640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e30102d40 0x7f1e30198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2effd640 1 -- 172.21.15.89:0/1111601824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e30102d40 msgr2=0x7f1e30198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2effd640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e30102d40 0x7f1e30198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:04.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2effd640 1 -- 172.21.15.89:0/1111601824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e3010c170 msgr2=0x7f1e3019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2ffff640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e3010c170 0x7f1e3019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2effd640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e3010c170 0x7f1e3019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2effd640 1 -- 172.21.15.89:0/1111601824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e3019d400 con 0x7f1e30103660 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2ffff640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e3010c170 0x7f1e3019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2effd640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e30103660 0x7f1e30198990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000d750 tx=0x7f1e2000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2cff9640 1 -- 172.21.15.89:0/1111601824 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e20017070 con 0x7f1e30103660 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2cff9640 1 -- 172.21.15.89:0/1111601824 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e20005020 con 0x7f1e30103660 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e3019d6f0 con 0x7f1e30103660 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e2cff9640 1 -- 172.21.15.89:0/1111601824 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e20004d00 con 0x7f1e30103660 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.115+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e301a4fd0 con 0x7f1e30103660 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.116+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1e30062980 con 0x7f1e30103660 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.117+0000 7f1e2cff9640 1 -- 172.21.15.89:0/1111601824 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1e20005560 con 0x7f1e30103660 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.118+0000 7f1e2cff9640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1e1c0a32d0 0x7f1e1c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:04.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.118+0000 7f1e2cff9640 1 -- 172.21.15.89:0/1111601824 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f1e200cfc60 con 0x7f1e30103660 2024-09-15T23:23:04.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.118+0000 7f1e2f7fe640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1e1c0a32d0 0x7f1e1c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:04.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.118+0000 7f1e2cff9640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] conn(0x7f1e1c0ae4a0 0x7f1e1c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:04.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.118+0000 7f1e2cff9640 1 -- 172.21.15.89:0/1111601824 --> [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e20022e50 con 0x7f1e1c0ae4a0 2024-09-15T23:23:04.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.118+0000 7f1e2cff9640 1 -- 172.21.15.89:0/1111601824 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f1e200d00b0 con 0x7f1e30103660 2024-09-15T23:23:04.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.119+0000 7f1e2f7fe640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1e1c0a32d0 0x7f1e1c0a5790 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009230 tx=0x7f1e180058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:04.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.119+0000 7f1e2ffff640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] conn(0x7f1e1c0ae4a0 0x7f1e1c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:04.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.119+0000 7f1e2ffff640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] conn(0x7f1e1c0ae4a0 0x7f1e1c0b0940 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:04.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.120+0000 7f1e2cff9640 1 -- 172.21.15.89:0/1111601824 <== osd.10 v2:172.21.15.120:6824/254855304 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e20022e50 con 0x7f1e1c0ae4a0 2024-09-15T23:23:04.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.134+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 --> [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1e30062e00 con 0x7f1e1c0ae4a0 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.134+0000 7f1e2cff9640 1 -- 172.21.15.89:0/1111601824 <== osd.10 v2:172.21.15.120:6824/254855304 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1e30062e00 con 0x7f1e1c0ae4a0 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 >> [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] conn(0x7f1e1c0ae4a0 msgr2=0x7f1e1c0b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e36034640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] conn(0x7f1e1c0ae4a0 0x7f1e1c0b0940 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1e1c0a32d0 msgr2=0x7f1e1c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e36034640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1e1c0a32d0 0x7f1e1c0a5790 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009230 tx=0x7f1e180058d0 comp rx=0 tx=0).stop 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e30103660 msgr2=0x7f1e30198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e36034640 1 --2- 172.21.15.89:0/1111601824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e30103660 0x7f1e30198990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000d750 tx=0x7f1e2000dc10 comp rx=0 tx=0).stop 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e2ffff640 1 -- 172.21.15.89:0/1111601824 reap_dead start 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 shutdown_connections 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 >> 172.21.15.89:0/1111601824 conn(0x7f1e300fc7a0 msgr2=0x7f1e300fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 shutdown_connections 2024-09-15T23:23:04.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:04.135+0000 7f1e36034640 1 -- 172.21.15.89:0/1111601824 wait complete. 2024-09-15T23:23:04.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:04 smithi089 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-15T23:23:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:04 smithi120 ceph-mon[35250]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-15T23:23:04.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:04 smithi174 ceph-mon[35586]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-15T23:23:04.875 INFO:teuthology.orchestra.run.smithi089.stdout:150323855418 2024-09-15T23:23:04.875 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-15T23:23:05.352 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:05 smithi089 ceph-mon[30933]: pgmap v320: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 383 B/s wr, 9 op/s 2024-09-15T23:23:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:05 smithi120 ceph-mon[35250]: pgmap v320: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 383 B/s wr, 9 op/s 2024-09-15T23:23:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:05 smithi174 ceph-mon[35586]: pgmap v320: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 383 B/s wr, 9 op/s 2024-09-15T23:23:05.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.504+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1387728092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f9c106f10 msgr2=0x7f2f9c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:05.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.504+0000 7f2fa29a0640 1 --2- 172.21.15.89:0/1387728092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f9c106f10 0x7f2f9c10ea90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009ca0 tx=0x7f2f9402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:23:05.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.504+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1387728092 shutdown_connections 2024-09-15T23:23:05.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.504+0000 7f2fa29a0640 1 --2- 172.21.15.89:0/1387728092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f9c106f10 0x7f2f9c10ea90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:05.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.504+0000 7f2fa29a0640 1 --2- 172.21.15.89:0/1387728092 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f9c102790 0x7f2f9c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:05.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.504+0000 7f2fa29a0640 1 --2- 172.21.15.89:0/1387728092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f9c101de0 0x7f2f9c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:05.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.504+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1387728092 >> 172.21.15.89:0/1387728092 conn(0x7f2f9c0fd8c0 msgr2=0x7f2f9c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:05.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.504+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1387728092 shutdown_connections 2024-09-15T23:23:05.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.504+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1387728092 wait complete. 2024-09-15T23:23:05.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2fa29a0640 1 Processor -- start 2024-09-15T23:23:05.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2fa29a0640 1 -- start start 2024-09-15T23:23:05.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2fa29a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f9c101de0 0x7f2f9c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:05.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2fa29a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f9c102790 0x7f2f9c198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:05.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2fa29a0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f9c106f10 0x7f2f9c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:05.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2fa29a0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c111f70 con 0x7f2f9c102790 2024-09-15T23:23:05.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2fa29a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c111df0 con 0x7f2f9c101de0 2024-09-15T23:23:05.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2fa29a0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c1120f0 con 0x7f2f9c106f10 2024-09-15T23:23:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2f93fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f9c102790 0x7f2f9c198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2f93fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f9c102790 0x7f2f9c198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52268/0 (socket says 172.21.15.89:52268) 2024-09-15T23:23:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2f93fff640 1 -- 172.21.15.89:0/1162060923 learned_addr learned my addr 172.21.15.89:0/1162060923 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2f9bfff640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f9c101de0 0x7f2f9c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2f93fff640 1 -- 172.21.15.89:0/1162060923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f9c106f10 msgr2=0x7f2f9c19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:05.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2f93fff640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f9c106f10 0x7f2f9c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:05.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2f93fff640 1 -- 172.21.15.89:0/1162060923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f9c101de0 msgr2=0x7f2f9c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:05.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2f93fff640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f9c101de0 0x7f2f9c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:05.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.505+0000 7f2f93fff640 1 -- 172.21.15.89:0/1162060923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c19d480 con 0x7f2f9c102790 2024-09-15T23:23:05.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.506+0000 7f2f93fff640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f9c102790 0x7f2f9c198a10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400d770 tx=0x7f2f8400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:05.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.506+0000 7f2f99ffb640 1 -- 172.21.15.89:0/1162060923 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84017070 con 0x7f2f9c102790 2024-09-15T23:23:05.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.506+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9c19d770 con 0x7f2f9c102790 2024-09-15T23:23:05.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.506+0000 7f2f99ffb640 1 -- 172.21.15.89:0/1162060923 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f84005500 con 0x7f2f9c102790 2024-09-15T23:23:05.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.506+0000 7f2f99ffb640 1 -- 172.21.15.89:0/1162060923 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84004e00 con 0x7f2f9c102790 2024-09-15T23:23:05.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.506+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9c1a5050 con 0x7f2f9c102790 2024-09-15T23:23:05.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.507+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2f9c062900 con 0x7f2f9c102790 2024-09-15T23:23:05.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.508+0000 7f2f99ffb640 1 -- 172.21.15.89:0/1162060923 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2f84004fa0 con 0x7f2f9c102790 2024-09-15T23:23:05.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.509+0000 7f2f99ffb640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f880a32d0 0x7f2f880a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:05.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.509+0000 7f2f9bfff640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f880a32d0 0x7f2f880a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:05.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.509+0000 7f2f99ffb640 1 -- 172.21.15.89:0/1162060923 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f2f840cfc00 con 0x7f2f9c102790 2024-09-15T23:23:05.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.509+0000 7f2f99ffb640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] conn(0x7f2f880ae4a0 0x7f2f880b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:05.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.509+0000 7f2f99ffb640 1 -- 172.21.15.89:0/1162060923 --> [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f84001d30 con 0x7f2f880ae4a0 2024-09-15T23:23:05.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.509+0000 7f2f99ffb640 1 -- 172.21.15.89:0/1162060923 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f2f840d0050 con 0x7f2f9c102790 2024-09-15T23:23:05.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.509+0000 7f2fa0f3d640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] conn(0x7f2f880ae4a0 0x7f2f880b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:05.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.509+0000 7f2f9bfff640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f880a32d0 0x7f2f880a5790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0058c0 tx=0x7f2f8c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:05.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.509+0000 7f2fa0f3d640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] conn(0x7f2f880ae4a0 0x7f2f880b0940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:05.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.510+0000 7f2f99ffb640 1 -- 172.21.15.89:0/1162060923 <== osd.9 v2:172.21.15.89:6826/2707132462 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2f84001d30 con 0x7f2f880ae4a0 2024-09-15T23:23:05.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.523+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 --> [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2f9c062d00 con 0x7f2f880ae4a0 2024-09-15T23:23:05.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2f99ffb640 1 -- 172.21.15.89:0/1162060923 <== osd.9 v2:172.21.15.89:6826/2707132462 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2f9c062d00 con 0x7f2f880ae4a0 2024-09-15T23:23:05.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] conn(0x7f2f880ae4a0 msgr2=0x7f2f880b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:05.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa29a0640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] conn(0x7f2f880ae4a0 0x7f2f880b0940 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:05.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f880a32d0 msgr2=0x7f2f880a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:05.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa29a0640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f880a32d0 0x7f2f880a5790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0058c0 tx=0x7f2f8c00a2d0 comp rx=0 tx=0).stop 2024-09-15T23:23:05.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f9c102790 msgr2=0x7f2f9c198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:05.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa29a0640 1 --2- 172.21.15.89:0/1162060923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f9c102790 0x7f2f9c198a10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400d770 tx=0x7f2f8400dc30 comp rx=0 tx=0).stop 2024-09-15T23:23:05.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa0f3d640 1 -- 172.21.15.89:0/1162060923 reap_dead start 2024-09-15T23:23:05.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 shutdown_connections 2024-09-15T23:23:05.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 >> 172.21.15.89:0/1162060923 conn(0x7f2f9c0fd8c0 msgr2=0x7f2f9c0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:05.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 shutdown_connections 2024-09-15T23:23:05.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:05.524+0000 7f2fa29a0640 1 -- 172.21.15.89:0/1162060923 wait complete. 2024-09-15T23:23:06.439 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:06 smithi089 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-15T23:23:06.439 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:06 smithi089 ceph-mon[30933]: Cluster is now healthy 2024-09-15T23:23:06.471 INFO:teuthology.orchestra.run.smithi089.stdout:34359738451 2024-09-15T23:23:06.471 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-15T23:23:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:06 smithi120 ceph-mon[35250]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-15T23:23:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:06 smithi120 ceph-mon[35250]: Cluster is now healthy 2024-09-15T23:23:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:06 smithi174 ceph-mon[35586]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-15T23:23:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:06 smithi174 ceph-mon[35586]: Cluster is now healthy 2024-09-15T23:23:07.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:07 smithi089 ceph-mon[30933]: pgmap v321: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 383 B/s wr, 12 op/s 2024-09-15T23:23:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:07 smithi120 ceph-mon[35250]: pgmap v321: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 383 B/s wr, 12 op/s 2024-09-15T23:23:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:07 smithi174 ceph-mon[35586]: pgmap v321: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 383 B/s wr, 12 op/s 2024-09-15T23:23:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.612+0000 7fc7387e4640 1 -- 172.21.15.89:0/4248008636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730102790 msgr2=0x7fc7301067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.612+0000 7fc7387e4640 1 --2- 172.21.15.89:0/4248008636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730102790 0x7fc7301067e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc720009900 tx=0x7fc72002a410 comp rx=0 tx=0).stop 2024-09-15T23:23:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.612+0000 7fc7387e4640 1 -- 172.21.15.89:0/4248008636 shutdown_connections 2024-09-15T23:23:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.612+0000 7fc7387e4640 1 --2- 172.21.15.89:0/4248008636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc730106f10 0x7fc73010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.612+0000 7fc7387e4640 1 --2- 172.21.15.89:0/4248008636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730102790 0x7fc7301067e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:07.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.612+0000 7fc7387e4640 1 --2- 172.21.15.89:0/4248008636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc730101de0 0x7fc7301021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.612+0000 7fc7387e4640 1 -- 172.21.15.89:0/4248008636 >> 172.21.15.89:0/4248008636 conn(0x7fc7300fd8c0 msgr2=0x7fc7300ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.612+0000 7fc7387e4640 1 -- 172.21.15.89:0/4248008636 shutdown_connections 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.613+0000 7fc7387e4640 1 -- 172.21.15.89:0/4248008636 wait complete. 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.613+0000 7fc7387e4640 1 Processor -- start 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.613+0000 7fc7387e4640 1 -- start start 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.613+0000 7fc7387e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc730101de0 0x7fc7300744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.613+0000 7fc7387e4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730102790 0x7fc730074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.613+0000 7fc7387e4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc730106f10 0x7fc730074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.613+0000 7fc7387e4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc730111f90 con 0x7fc730102790 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.613+0000 7fc7387e4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc730111e10 con 0x7fc730106f10 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.613+0000 7fc7387e4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc730112110 con 0x7fc730101de0 2024-09-15T23:23:07.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.613+0000 7fc735d7f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730102790 0x7fc730074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc735d7f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730102790 0x7fc730074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52276/0 (socket says 172.21.15.89:52276) 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc735d7f640 1 -- 172.21.15.89:0/1675907160 learned_addr learned my addr 172.21.15.89:0/1675907160 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc736d81640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc730106f10 0x7fc730074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc736580640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc730101de0 0x7fc7300744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc735d7f640 1 -- 172.21.15.89:0/1675907160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc730101de0 msgr2=0x7fc7300744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc735d7f640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc730101de0 0x7fc7300744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc735d7f640 1 -- 172.21.15.89:0/1675907160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc730106f10 msgr2=0x7fc730074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc735d7f640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc730106f10 0x7fc730074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc735d7f640 1 -- 172.21.15.89:0/1675907160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7301a6950 con 0x7fc730102790 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc736580640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc730101de0 0x7fc7300744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc735d7f640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730102790 0x7fc730074a30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc720006010 tx=0x7fc7200042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc7277fe640 1 -- 172.21.15.89:0/1675907160 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72003f030 con 0x7fc730102790 2024-09-15T23:23:07.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7301a6be0 con 0x7fc730102790 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc7277fe640 1 -- 172.21.15.89:0/1675907160 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc720003ce0 con 0x7fc730102790 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc7277fe640 1 -- 172.21.15.89:0/1675907160 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc720035070 con 0x7fc730102790 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.614+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7301a70c0 con 0x7fc730102790 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.615+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc730062880 con 0x7fc730102790 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.617+0000 7fc7277fe640 1 -- 172.21.15.89:0/1675907160 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc720042030 con 0x7fc730102790 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.617+0000 7fc7277fe640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc71c0a32d0 0x7fc71c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.617+0000 7fc736580640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc71c0a32d0 0x7fc71c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.617+0000 7fc7277fe640 1 -- 172.21.15.89:0/1675907160 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fc7200f0a40 con 0x7fc730102790 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.618+0000 7fc7277fe640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] conn(0x7fc71c0ae4a0 0x7fc71c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.618+0000 7fc7277fe640 1 -- 172.21.15.89:0/1675907160 --> [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc720001d30 con 0x7fc71c0ae4a0 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.618+0000 7fc7277fe640 1 -- 172.21.15.89:0/1675907160 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fc7200f0e90 con 0x7fc730102790 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.618+0000 7fc736d81640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] conn(0x7fc71c0ae4a0 0x7fc71c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.618+0000 7fc736580640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc71c0a32d0 0x7fc71c0a5790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009930 tx=0x7fc72c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:07.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.618+0000 7fc736d81640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] conn(0x7fc71c0ae4a0 0x7fc71c0b0940 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:07.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.619+0000 7fc7277fe640 1 -- 172.21.15.89:0/1675907160 <== osd.6 v2:172.21.15.89:6818/2734649081 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc720001d30 con 0x7fc71c0ae4a0 2024-09-15T23:23:07.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.632+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 --> [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc730062d00 con 0x7fc71c0ae4a0 2024-09-15T23:23:07.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.632+0000 7fc7277fe640 1 -- 172.21.15.89:0/1675907160 <== osd.6 v2:172.21.15.89:6818/2734649081 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc730062d00 con 0x7fc71c0ae4a0 2024-09-15T23:23:07.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] conn(0x7fc71c0ae4a0 msgr2=0x7fc71c0b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:07.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc7387e4640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] conn(0x7fc71c0ae4a0 0x7fc71c0b0940 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:07.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc71c0a32d0 msgr2=0x7fc71c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:07.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc7387e4640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc71c0a32d0 0x7fc71c0a5790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009930 tx=0x7fc72c008040 comp rx=0 tx=0).stop 2024-09-15T23:23:07.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730102790 msgr2=0x7fc730074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:07.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc7387e4640 1 --2- 172.21.15.89:0/1675907160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc730102790 0x7fc730074a30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc720006010 tx=0x7fc7200042f0 comp rx=0 tx=0).stop 2024-09-15T23:23:07.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc736d81640 1 -- 172.21.15.89:0/1675907160 reap_dead start 2024-09-15T23:23:07.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 shutdown_connections 2024-09-15T23:23:07.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 >> 172.21.15.89:0/1675907160 conn(0x7fc7300fd8c0 msgr2=0x7fc7300fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:07.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 shutdown_connections 2024-09-15T23:23:07.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:07.633+0000 7fc7387e4640 1 -- 172.21.15.89:0/1675907160 wait complete. 2024-09-15T23:23:08.057 INFO:teuthology.orchestra.run.smithi089.stdout:283467841568 2024-09-15T23:23:08.057 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-15T23:23:08.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.681+0000 7f1e92f3e640 1 -- 172.21.15.89:0/265067004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e8c101df0 msgr2=0x7f1e8c1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.681+0000 7f1e92f3e640 1 --2- 172.21.15.89:0/265067004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e8c101df0 0x7f1e8c1021d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009900 tx=0x7f1e8002a410 comp rx=0 tx=0).stop 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.681+0000 7f1e92f3e640 1 -- 172.21.15.89:0/265067004 shutdown_connections 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.681+0000 7f1e92f3e640 1 --2- 172.21.15.89:0/265067004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c106f20 0x7f1e8c10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.681+0000 7f1e92f3e640 1 --2- 172.21.15.89:0/265067004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e8c1027a0 0x7f1e8c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.681+0000 7f1e92f3e640 1 --2- 172.21.15.89:0/265067004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e8c101df0 0x7f1e8c1021d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.681+0000 7f1e92f3e640 1 -- 172.21.15.89:0/265067004 >> 172.21.15.89:0/265067004 conn(0x7f1e8c0fd8b0 msgr2=0x7f1e8c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.681+0000 7f1e92f3e640 1 -- 172.21.15.89:0/265067004 shutdown_connections 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.681+0000 7f1e92f3e640 1 -- 172.21.15.89:0/265067004 wait complete. 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e92f3e640 1 Processor -- start 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e92f3e640 1 -- start start 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e92f3e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c101df0 0x7f1e8c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e92f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e8c1027a0 0x7f1e8c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e92f3e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e8c106f20 0x7f1e8c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e92f3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8c111f80 con 0x7f1e8c1027a0 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e92f3e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8c111e00 con 0x7f1e8c101df0 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e92f3e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8c112100 con 0x7f1e8c106f20 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e8bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e8c1027a0 0x7f1e8c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e8bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e8c1027a0 0x7f1e8c074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52308/0 (socket says 172.21.15.89:52308) 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e90cda640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c101df0 0x7f1e8c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.682+0000 7f1e8bfff640 1 -- 172.21.15.89:0/3016439160 learned_addr learned my addr 172.21.15.89:0/3016439160 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e8bfff640 1 -- 172.21.15.89:0/3016439160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e8c106f20 msgr2=0x7f1e8c074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e8bfff640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e8c106f20 0x7f1e8c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e8bfff640 1 -- 172.21.15.89:0/3016439160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c101df0 msgr2=0x7f1e8c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e8bfff640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e8c101df0 0x7f1e8c0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e8bfff640 1 -- 172.21.15.89:0/3016439160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e8c1a6990 con 0x7f1e8c1027a0 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e8bfff640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e8c1027a0 0x7f1e8c074a30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800c890 tx=0x7f1e7800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:08.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e89ffb640 1 -- 172.21.15.89:0/3016439160 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e78004220 con 0x7f1e8c1027a0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e89ffb640 1 -- 172.21.15.89:0/3016439160 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e780043c0 con 0x7f1e8c1027a0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e8c1a6c80 con 0x7f1e8c1027a0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e89ffb640 1 -- 172.21.15.89:0/3016439160 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e780049e0 con 0x7f1e8c1027a0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.683+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e8c1a71c0 con 0x7f1e8c1027a0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.684+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1e8c062900 con 0x7f1e8c1027a0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.685+0000 7f1e89ffb640 1 -- 172.21.15.89:0/3016439160 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1e780079b0 con 0x7f1e8c1027a0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.686+0000 7f1e89ffb640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1e7c0a32d0 0x7f1e7c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.686+0000 7f1e90cda640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1e7c0a32d0 0x7f1e7c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.686+0000 7f1e89ffb640 1 -- 172.21.15.89:0/3016439160 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f1e780cfe60 con 0x7f1e8c1027a0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.686+0000 7f1e89ffb640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] conn(0x7f1e7c0ae4a0 0x7f1e7c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.686+0000 7f1e89ffb640 1 -- 172.21.15.89:0/3016439160 --> [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e78012ec0 con 0x7f1e7c0ae4a0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.686+0000 7f1e89ffb640 1 -- 172.21.15.89:0/3016439160 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f1e780d2520 con 0x7f1e8c1027a0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.686+0000 7f1e914db640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] conn(0x7f1e7c0ae4a0 0x7f1e7c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.686+0000 7f1e90cda640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1e7c0a32d0 0x7f1e7c0a5790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1e80000c00 tx=0x7f1e800058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:08.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.686+0000 7f1e914db640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] conn(0x7f1e7c0ae4a0 0x7f1e7c0b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:08.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.687+0000 7f1e89ffb640 1 -- 172.21.15.89:0/3016439160 <== osd.3 v2:172.21.15.89:6810/2725889139 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e78012ec0 con 0x7f1e7c0ae4a0 2024-09-15T23:23:08.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.700+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 --> [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1e8c062d00 con 0x7f1e7c0ae4a0 2024-09-15T23:23:08.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e89ffb640 1 -- 172.21.15.89:0/3016439160 <== osd.3 v2:172.21.15.89:6810/2725889139 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1e8c062d00 con 0x7f1e7c0ae4a0 2024-09-15T23:23:08.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] conn(0x7f1e7c0ae4a0 msgr2=0x7f1e7c0b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e92f3e640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] conn(0x7f1e7c0ae4a0 0x7f1e7c0b0940 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1e7c0a32d0 msgr2=0x7f1e7c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e92f3e640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1e7c0a32d0 0x7f1e7c0a5790 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1e80000c00 tx=0x7f1e800058d0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e8c1027a0 msgr2=0x7f1e8c074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e92f3e640 1 --2- 172.21.15.89:0/3016439160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e8c1027a0 0x7f1e8c074a30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1e7800c890 tx=0x7f1e7800cd50 comp rx=0 tx=0).stop 2024-09-15T23:23:08.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e914db640 1 -- 172.21.15.89:0/3016439160 reap_dead start 2024-09-15T23:23:08.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 shutdown_connections 2024-09-15T23:23:08.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 >> 172.21.15.89:0/3016439160 conn(0x7f1e8c0fd8b0 msgr2=0x7f1e8c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:08.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 shutdown_connections 2024-09-15T23:23:08.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.701+0000 7f1e92f3e640 1 -- 172.21.15.89:0/3016439160 wait complete. 2024-09-15T23:23:09.403 INFO:teuthology.orchestra.run.smithi089.stdout:98784247876 2024-09-15T23:23:09.403 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-15T23:23:09.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.564+0000 7fa05a391640 1 -- 172.21.15.89:0/2251382361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa054100e20 msgr2=0x7fa054101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.564+0000 7fa05a391640 1 --2- 172.21.15.89:0/2251382361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa054100e20 0x7fa054101200 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa044009900 tx=0x7fa04402a480 comp rx=0 tx=0).stop 2024-09-15T23:23:09.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.564+0000 7fa05a391640 1 -- 172.21.15.89:0/2251382361 shutdown_connections 2024-09-15T23:23:09.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.564+0000 7fa05a391640 1 --2- 172.21.15.89:0/2251382361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa054102ef0 0x7fa05410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.564+0000 7fa05a391640 1 --2- 172.21.15.89:0/2251382361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa054102570 0x7fa0541029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.564+0000 7fa05a391640 1 --2- 172.21.15.89:0/2251382361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa054100e20 0x7fa054101200 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.564+0000 7fa05a391640 1 -- 172.21.15.89:0/2251382361 >> 172.21.15.89:0/2251382361 conn(0x7fa0540fc7d0 msgr2=0x7fa0540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.564+0000 7fa05a391640 1 -- 172.21.15.89:0/2251382361 shutdown_connections 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.564+0000 7fa05a391640 1 -- 172.21.15.89:0/2251382361 wait complete. 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05a391640 1 Processor -- start 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05a391640 1 -- start start 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05a391640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa054100e20 0x7fa054199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05a391640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa054102570 0x7fa054199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05a391640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa054102ef0 0x7fa0541a4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05a391640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa054110f80 con 0x7fa054102ef0 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05a391640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa054110e00 con 0x7fa054102570 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05a391640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa054111100 con 0x7fa054100e20 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05892e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa054102ef0 0x7fa0541a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05892e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa054102ef0 0x7fa0541a4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52314/0 (socket says 172.21.15.89:52314) 2024-09-15T23:23:09.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.565+0000 7fa05892e640 1 -- 172.21.15.89:0/3944816898 learned_addr learned my addr 172.21.15.89:0/3944816898 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa053fff640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa054100e20 0x7fa054199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa0537fe640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa054102570 0x7fa054199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa05892e640 1 -- 172.21.15.89:0/3944816898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa054100e20 msgr2=0x7fa054199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa05892e640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa054100e20 0x7fa054199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa05892e640 1 -- 172.21.15.89:0/3944816898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa054102570 msgr2=0x7fa054199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa05892e640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa054102570 0x7fa054199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa05892e640 1 -- 172.21.15.89:0/3944816898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0541a49d0 con 0x7fa054102ef0 2024-09-15T23:23:09.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa0537fe640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa054102570 0x7fa054199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa05892e640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa054102ef0 0x7fa0541a4490 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00eaa0 tx=0x7fa04c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa0517fa640 1 -- 172.21.15.89:0/3944816898 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c00cdb0 con 0x7fa054102ef0 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0541a4ba0 con 0x7fa054102ef0 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa0517fa640 1 -- 172.21.15.89:0/3944816898 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa04c005020 con 0x7fa054102ef0 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa0517fa640 1 -- 172.21.15.89:0/3944816898 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c0116b0 con 0x7fa054102ef0 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.566+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0541a5060 con 0x7fa054102ef0 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.567+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa0540628c0 con 0x7fa054102ef0 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.569+0000 7fa0517fa640 1 -- 172.21.15.89:0/3944816898 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa04c005560 con 0x7fa054102ef0 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.569+0000 7fa0517fa640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa03c0a32d0 0x7fa03c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.570+0000 7fa053fff640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa03c0a32d0 0x7fa03c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.570+0000 7fa0517fa640 1 -- 172.21.15.89:0/3944816898 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fa04c0d0470 con 0x7fa054102ef0 2024-09-15T23:23:09.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.570+0000 7fa0517fa640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] conn(0x7fa03c0ae4a0 0x7fa03c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.570+0000 7fa0517fa640 1 -- 172.21.15.89:0/3944816898 --> [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa04c014ec0 con 0x7fa03c0ae4a0 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.570+0000 7fa0517fa640 1 -- 172.21.15.89:0/3944816898 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fa04c0d0820 con 0x7fa054102ef0 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.570+0000 7fa053fff640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa03c0a32d0 0x7fa03c0a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa044009230 tx=0x7fa0440058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.570+0000 7fa0537fe640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] conn(0x7fa03c0ae4a0 0x7fa03c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.571+0000 7fa0537fe640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] conn(0x7fa03c0ae4a0 0x7fa03c0b0940 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.571+0000 7fa0517fa640 1 -- 172.21.15.89:0/3944816898 <== osd.11 v2:172.21.15.174:6824/246219377 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa04c014ec0 con 0x7fa03c0ae4a0 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.585+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 --> [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa054062d40 con 0x7fa03c0ae4a0 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa0517fa640 1 -- 172.21.15.89:0/3944816898 <== osd.11 v2:172.21.15.174:6824/246219377 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa054062d40 con 0x7fa03c0ae4a0 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 >> [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] conn(0x7fa03c0ae4a0 msgr2=0x7fa03c0b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05a391640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] conn(0x7fa03c0ae4a0 0x7fa03c0b0940 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa03c0a32d0 msgr2=0x7fa03c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05a391640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa03c0a32d0 0x7fa03c0a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa044009230 tx=0x7fa0440058d0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa054102ef0 msgr2=0x7fa0541a4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05a391640 1 --2- 172.21.15.89:0/3944816898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa054102ef0 0x7fa0541a4490 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00eaa0 tx=0x7fa04c00ef60 comp rx=0 tx=0).stop 2024-09-15T23:23:09.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05892e640 1 -- 172.21.15.89:0/3944816898 reap_dead start 2024-09-15T23:23:09.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 shutdown_connections 2024-09-15T23:23:09.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 >> 172.21.15.89:0/3944816898 conn(0x7fa0540fc7d0 msgr2=0x7fa0540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 shutdown_connections 2024-09-15T23:23:09.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:09.586+0000 7fa05a391640 1 -- 172.21.15.89:0/3944816898 wait complete. 2024-09-15T23:23:10.285 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:10 smithi089 ceph-mon[30933]: pgmap v322: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 383 B/s wr, 11 op/s 2024-09-15T23:23:10.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 -- 172.21.15.89:0/870922706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c81008d0 msgr2=0x7f53c8100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 --2- 172.21.15.89:0/870922706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c81008d0 0x7f53c8100d30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009930 tx=0x7f53b002a420 comp rx=0 tx=0).stop 2024-09-15T23:23:10.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 -- 172.21.15.89:0/870922706 shutdown_connections 2024-09-15T23:23:10.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 --2- 172.21.15.89:0/870922706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c8101270 0x7f53c810d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 --2- 172.21.15.89:0/870922706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c81008d0 0x7f53c8100d30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 --2- 172.21.15.89:0/870922706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c81068d0 0x7f53c8106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 -- 172.21.15.89:0/870922706 >> 172.21.15.89:0/870922706 conn(0x7f53c80fc610 msgr2=0x7f53c80fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 -- 172.21.15.89:0/870922706 shutdown_connections 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 -- 172.21.15.89:0/870922706 wait complete. 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 Processor -- start 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.284+0000 7f53cd39f640 1 -- start start 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53cd39f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c81008d0 0x7f53c810e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53cd39f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c8101270 0x7f53c810e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53cd39f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c81068d0 0x7f53c810ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53cd39f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8111df0 con 0x7f53c81008d0 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53cd39f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8111c70 con 0x7f53c8101270 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53cd39f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f53c8111f70 con 0x7f53c81068d0 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53c7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c81008d0 0x7f53c810e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53c7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c81008d0 0x7f53c810e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52344/0 (socket says 172.21.15.89:52344) 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53c7fff640 1 -- 172.21.15.89:0/3403467263 learned_addr learned my addr 172.21.15.89:0/3403467263 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53bffff640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c8101270 0x7f53c810e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53ccb9e640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c81068d0 0x7f53c810ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53c7fff640 1 -- 172.21.15.89:0/3403467263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c81068d0 msgr2=0x7f53c810ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53c7fff640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53c81068d0 0x7f53c810ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53c7fff640 1 -- 172.21.15.89:0/3403467263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c8101270 msgr2=0x7f53c810e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53c7fff640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f53c8101270 0x7f53c810e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53c7fff640 1 -- 172.21.15.89:0/3403467263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c81a6730 con 0x7f53c81008d0 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53c7fff640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c81008d0 0x7f53c810e230 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f53b800d950 tx=0x7f53b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.285+0000 7f53c5ffb640 1 -- 172.21.15.89:0/3403467263 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b8017030 con 0x7f53c81008d0 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.286+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c81a6a20 con 0x7f53c81008d0 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.286+0000 7f53c5ffb640 1 -- 172.21.15.89:0/3403467263 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b8004020 con 0x7f53c81008d0 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.286+0000 7f53c5ffb640 1 -- 172.21.15.89:0/3403467263 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b80047a0 con 0x7f53c81008d0 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.286+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c81a6f80 con 0x7f53c81008d0 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.286+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f53c8062900 con 0x7f53c81008d0 2024-09-15T23:23:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.288+0000 7f53c5ffb640 1 -- 172.21.15.89:0/3403467263 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f53b80041d0 con 0x7f53c81008d0 2024-09-15T23:23:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.288+0000 7f53c5ffb640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53b40a32d0 0x7f53b40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.288+0000 7f53bffff640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53b40a32d0 0x7f53b40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.288+0000 7f53c5ffb640 1 -- 172.21.15.89:0/3403467263 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f53b80cfe50 con 0x7f53c81008d0 2024-09-15T23:23:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.288+0000 7f53c5ffb640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] conn(0x7f53b40ae4a0 0x7f53b40b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.288+0000 7f53c5ffb640 1 -- 172.21.15.89:0/3403467263 --> [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53b8021e70 con 0x7f53b40ae4a0 2024-09-15T23:23:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.288+0000 7f53c5ffb640 1 -- 172.21.15.89:0/3403467263 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f53b80d02a0 con 0x7f53c81008d0 2024-09-15T23:23:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.289+0000 7f53bffff640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53b40a32d0 0x7f53b40a5790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009900 tx=0x7f53b00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.289+0000 7f53ccb9e640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] conn(0x7f53b40ae4a0 0x7f53b40b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.289+0000 7f53ccb9e640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] conn(0x7f53b40ae4a0 0x7f53b40b0940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.290+0000 7f53c5ffb640 1 -- 172.21.15.89:0/3403467263 <== osd.7 v2:172.21.15.120:6816/831027547 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53b8021e70 con 0x7f53b40ae4a0 2024-09-15T23:23:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.304+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 --> [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f53c8062d80 con 0x7f53b40ae4a0 2024-09-15T23:23:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53c5ffb640 1 -- 172.21.15.89:0/3403467263 <== osd.7 v2:172.21.15.120:6816/831027547 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f53c8062d80 con 0x7f53b40ae4a0 2024-09-15T23:23:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 >> [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] conn(0x7f53b40ae4a0 msgr2=0x7f53b40b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53cd39f640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] conn(0x7f53b40ae4a0 0x7f53b40b0940 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53b40a32d0 msgr2=0x7f53b40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53cd39f640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53b40a32d0 0x7f53b40a5790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009900 tx=0x7f53b00058d0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c81008d0 msgr2=0x7f53c810e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53cd39f640 1 --2- 172.21.15.89:0/3403467263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53c81008d0 0x7f53c810e230 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f53b800d950 tx=0x7f53b800de10 comp rx=0 tx=0).stop 2024-09-15T23:23:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53ccb9e640 1 -- 172.21.15.89:0/3403467263 reap_dead start 2024-09-15T23:23:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 shutdown_connections 2024-09-15T23:23:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 >> 172.21.15.89:0/3403467263 conn(0x7f53c80fc610 msgr2=0x7f53c80fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 shutdown_connections 2024-09-15T23:23:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:10.305+0000 7f53cd39f640 1 -- 172.21.15.89:0/3403467263 wait complete. 2024-09-15T23:23:10.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:10 smithi120 ceph-mon[35250]: pgmap v322: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 383 B/s wr, 11 op/s 2024-09-15T23:23:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:10 smithi174 ceph-mon[35586]: pgmap v322: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 383 B/s wr, 11 op/s 2024-09-15T23:23:10.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.579+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/1099856756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ecc105f40 msgr2=0x7f3ecc1063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.579+0000 7f3ed3c9e640 1 --2- 172.21.15.89:0/1099856756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ecc105f40 0x7f3ecc1063d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009ca0 tx=0x7f3ec402a940 comp rx=0 tx=0).stop 2024-09-15T23:23:10.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.579+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/1099856756 shutdown_connections 2024-09-15T23:23:10.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.579+0000 7f3ed3c9e640 1 --2- 172.21.15.89:0/1099856756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ecc105f40 0x7f3ecc1063d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.579+0000 7f3ed3c9e640 1 --2- 172.21.15.89:0/1099856756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ecc101450 0x7f3ecc105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.579+0000 7f3ed3c9e640 1 --2- 172.21.15.89:0/1099856756 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ecc100b30 0x7f3ecc100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.579+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/1099856756 >> 172.21.15.89:0/1099856756 conn(0x7f3ecc0fc7a0 msgr2=0x7f3ecc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:10.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.579+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/1099856756 shutdown_connections 2024-09-15T23:23:10.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/1099856756 wait complete. 2024-09-15T23:23:10.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed3c9e640 1 Processor -- start 2024-09-15T23:23:10.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed3c9e640 1 -- start start 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed3c9e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ecc100b30 0x7f3ecc1a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed3c9e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ecc101450 0x7f3ecc1a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed3c9e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ecc105f40 0x7f3ecc19e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed3c9e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc1185d0 con 0x7f3ecc100b30 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed3c9e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc118450 con 0x7f3ecc101450 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed3c9e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc118750 con 0x7f3ecc105f40 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed1a3a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ecc100b30 0x7f3ecc1a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed1a3a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ecc100b30 0x7f3ecc1a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52290/0 (socket says 172.21.15.89:52290) 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.580+0000 7f3ed1a3a640 1 -- 172.21.15.89:0/373888717 learned_addr learned my addr 172.21.15.89:0/373888717 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ed1a3a640 1 -- 172.21.15.89:0/373888717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ecc105f40 msgr2=0x7f3ecc19e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ed1a3a640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ecc105f40 0x7f3ecc19e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ed1a3a640 1 -- 172.21.15.89:0/373888717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ecc101450 msgr2=0x7f3ecc1a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ed1239640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ecc101450 0x7f3ecc1a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ed1a3a640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ecc101450 0x7f3ecc1a4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ed1a3a640 1 -- 172.21.15.89:0/373888717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ecc19efc0 con 0x7f3ecc100b30 2024-09-15T23:23:10.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ed1a3a640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ecc100b30 0x7f3ecc1a46a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800b4a0 tx=0x7f3ec800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ec2ffd640 1 -- 172.21.15.89:0/373888717 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8014070 con 0x7f3ecc100b30 2024-09-15T23:23:10.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ec2ffd640 1 -- 172.21.15.89:0/373888717 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ec8010af0 con 0x7f3ecc100b30 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ecc19f2b0 con 0x7f3ecc100b30 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ec2ffd640 1 -- 172.21.15.89:0/373888717 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8017d10 con 0x7f3ecc100b30 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.581+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ecc1ab500 con 0x7f3ecc100b30 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.582+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3ecc105330 con 0x7f3ecc100b30 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.583+0000 7f3ec2ffd640 1 -- 172.21.15.89:0/373888717 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3ec8017070 con 0x7f3ecc100b30 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.584+0000 7f3ec2ffd640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3eb80a32d0 0x7f3eb80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.584+0000 7f3ed1239640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3eb80a32d0 0x7f3eb80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.584+0000 7f3ec2ffd640 1 -- 172.21.15.89:0/373888717 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f3ec80cf510 con 0x7f3ecc100b30 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.584+0000 7f3ec2ffd640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] conn(0x7f3eb80ae4a0 0x7f3eb80b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.584+0000 7f3ec2ffd640 1 -- 172.21.15.89:0/373888717 --> [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ec801ae50 con 0x7f3eb80ae4a0 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.584+0000 7f3ec2ffd640 1 -- 172.21.15.89:0/373888717 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f3ec80cf960 con 0x7f3ecc100b30 2024-09-15T23:23:10.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.584+0000 7f3ed1239640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3eb80a32d0 0x7f3eb80a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc005fd0 tx=0x7f3ebc005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.584+0000 7f3ed223b640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] conn(0x7f3eb80ae4a0 0x7f3eb80b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.585+0000 7f3ed223b640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] conn(0x7f3eb80ae4a0 0x7f3eb80b0940 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.586+0000 7f3ec2ffd640 1 -- 172.21.15.89:0/373888717 <== osd.8 v2:172.21.15.174:6816/2158836361 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ec801ae50 con 0x7f3eb80ae4a0 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.599+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 --> [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3ecc06b070 con 0x7f3eb80ae4a0 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.599+0000 7f3ec2ffd640 1 -- 172.21.15.89:0/373888717 <== osd.8 v2:172.21.15.174:6816/2158836361 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3ecc06b070 con 0x7f3eb80ae4a0 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 >> [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] conn(0x7f3eb80ae4a0 msgr2=0x7f3eb80b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed3c9e640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] conn(0x7f3eb80ae4a0 0x7f3eb80b0940 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3eb80a32d0 msgr2=0x7f3eb80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed3c9e640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3eb80a32d0 0x7f3eb80a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc005fd0 tx=0x7f3ebc005d60 comp rx=0 tx=0).stop 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ecc100b30 msgr2=0x7f3ecc1a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed3c9e640 1 --2- 172.21.15.89:0/373888717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ecc100b30 0x7f3ecc1a46a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3ec800b4a0 tx=0x7f3ec800b960 comp rx=0 tx=0).stop 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed223b640 1 -- 172.21.15.89:0/373888717 reap_dead start 2024-09-15T23:23:10.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 shutdown_connections 2024-09-15T23:23:10.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 >> 172.21.15.89:0/373888717 conn(0x7f3ecc0fc7a0 msgr2=0x7f3ecc1038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:10.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 shutdown_connections 2024-09-15T23:23:10.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:08.600+0000 7f3ed3c9e640 1 -- 172.21.15.89:0/373888717 wait complete. 2024-09-15T23:23:11.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.254+0000 7f4acc03a640 1 -- 172.21.15.89:0/978053984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac4100350 msgr2=0x7f4ac4100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.254+0000 7f4acc03a640 1 --2- 172.21.15.89:0/978053984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac4100350 0x7f4ac4100730 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0009930 tx=0x7f4ac002a420 comp rx=0 tx=0).stop 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.254+0000 7f4acc03a640 1 -- 172.21.15.89:0/978053984 shutdown_connections 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.254+0000 7f4acc03a640 1 --2- 172.21.15.89:0/978053984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac4106f50 0x7f4ac410eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.254+0000 7f4acc03a640 1 --2- 172.21.15.89:0/978053984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ac4100d00 0x7f4ac4106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.254+0000 7f4acc03a640 1 --2- 172.21.15.89:0/978053984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac4100350 0x7f4ac4100730 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.254+0000 7f4acc03a640 1 -- 172.21.15.89:0/978053984 >> 172.21.15.89:0/978053984 conn(0x7f4ac4077210 msgr2=0x7f4ac40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.254+0000 7f4acc03a640 1 -- 172.21.15.89:0/978053984 shutdown_connections 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4acc03a640 1 -- 172.21.15.89:0/978053984 wait complete. 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4acc03a640 1 Processor -- start 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4acc03a640 1 -- start start 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4acc03a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ac4100350 0x7f4ac41012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4acc03a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac4100d00 0x7f4ac4101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:11.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4acc03a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac4106f50 0x7f4ac4101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4acc03a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac4112000 con 0x7f4ac4100d00 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4acc03a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac4111e80 con 0x7f4ac4106f50 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4acc03a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac4112180 con 0x7f4ac4100350 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4ac95d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac4100d00 0x7f4ac4101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4ac95d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac4100d00 0x7f4ac4101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52352/0 (socket says 172.21.15.89:52352) 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.255+0000 7f4ac95d5640 1 -- 172.21.15.89:0/2317362651 learned_addr learned my addr 172.21.15.89:0/2317362651 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4ac9dd6640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ac4100350 0x7f4ac41012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4ac95d5640 1 -- 172.21.15.89:0/2317362651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ac4100350 msgr2=0x7f4ac41012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4ac95d5640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ac4100350 0x7f4ac41012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4ac95d5640 1 -- 172.21.15.89:0/2317362651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac4106f50 msgr2=0x7f4ac4101d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4ac95d5640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ac4106f50 0x7f4ac4101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4ac95d5640 1 -- 172.21.15.89:0/2317362651 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac41a6a20 con 0x7f4ac4100d00 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4ac95d5640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac4100d00 0x7f4ac4101810 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400b450 tx=0x7f4ab400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4abaffd640 1 -- 172.21.15.89:0/2317362651 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4015020 con 0x7f4ac4100d00 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4abaffd640 1 -- 172.21.15.89:0/2317362651 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ab4011d70 con 0x7f4ac4100d00 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac41a6d10 con 0x7f4ac4100d00 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4abaffd640 1 -- 172.21.15.89:0/2317362651 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab400f340 con 0x7f4ac4100d00 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.256+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac41a71a0 con 0x7f4ac4100d00 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.257+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4ac4062900 con 0x7f4ac4100d00 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.258+0000 7f4abaffd640 1 -- 172.21.15.89:0/2317362651 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4ab4018020 con 0x7f4ac4100d00 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.259+0000 7f4abaffd640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ab00a32d0 0x7f4ab00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.259+0000 7f4ac9dd6640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ab00a32d0 0x7f4ab00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.259+0000 7f4abaffd640 1 -- 172.21.15.89:0/2317362651 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f4ab40cff20 con 0x7f4ac4100d00 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.259+0000 7f4abaffd640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] conn(0x7f4ab00ae4a0 0x7f4ab00b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:11.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.259+0000 7f4abaffd640 1 -- 172.21.15.89:0/2317362651 --> [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ab4021e70 con 0x7f4ab00ae4a0 2024-09-15T23:23:11.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.259+0000 7f4abaffd640 1 -- 172.21.15.89:0/2317362651 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f4ab40d0370 con 0x7f4ac4100d00 2024-09-15T23:23:11.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.259+0000 7f4ac9dd6640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ab00a32d0 0x7f4ab00a5790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0000c00 tx=0x7f4ac0031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:11.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.259+0000 7f4aca5d7640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] conn(0x7f4ab00ae4a0 0x7f4ab00b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:11.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.260+0000 7f4aca5d7640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] conn(0x7f4ab00ae4a0 0x7f4ab00b0940 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:11.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.261+0000 7f4abaffd640 1 -- 172.21.15.89:0/2317362651 <== osd.4 v2:172.21.15.120:6808/135794330 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4ab4021e70 con 0x7f4ab00ae4a0 2024-09-15T23:23:11.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.274+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 --> [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4ac4062d80 con 0x7f4ab00ae4a0 2024-09-15T23:23:11.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4abaffd640 1 -- 172.21.15.89:0/2317362651 <== osd.4 v2:172.21.15.120:6808/135794330 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4ac4062d80 con 0x7f4ab00ae4a0 2024-09-15T23:23:11.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 >> [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] conn(0x7f4ab00ae4a0 msgr2=0x7f4ab00b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4acc03a640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] conn(0x7f4ab00ae4a0 0x7f4ab00b0940 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ab00a32d0 msgr2=0x7f4ab00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4acc03a640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ab00a32d0 0x7f4ab00a5790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0000c00 tx=0x7f4ac0031490 comp rx=0 tx=0).stop 2024-09-15T23:23:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac4100d00 msgr2=0x7f4ac4101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4acc03a640 1 --2- 172.21.15.89:0/2317362651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ac4100d00 0x7f4ac4101810 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400b450 tx=0x7f4ab400b910 comp rx=0 tx=0).stop 2024-09-15T23:23:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4aca5d7640 1 -- 172.21.15.89:0/2317362651 reap_dead start 2024-09-15T23:23:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 shutdown_connections 2024-09-15T23:23:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 >> 172.21.15.89:0/2317362651 conn(0x7f4ac4077210 msgr2=0x7f4ac41052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 shutdown_connections 2024-09-15T23:23:11.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.275+0000 7f4acc03a640 1 -- 172.21.15.89:0/2317362651 wait complete. 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.706+0000 7fcdcefbb640 1 -- 172.21.15.89:0/2934469732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdc8102790 msgr2=0x7fcdc81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.706+0000 7fcdcefbb640 1 --2- 172.21.15.89:0/2934469732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdc8102790 0x7fcdc81067e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8009900 tx=0x7fcdb802a410 comp rx=0 tx=0).stop 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.706+0000 7fcdcefbb640 1 -- 172.21.15.89:0/2934469732 shutdown_connections 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.706+0000 7fcdcefbb640 1 --2- 172.21.15.89:0/2934469732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc8106f10 0x7fcdc810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.706+0000 7fcdcefbb640 1 --2- 172.21.15.89:0/2934469732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdc8102790 0x7fcdc81067e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.706+0000 7fcdcefbb640 1 --2- 172.21.15.89:0/2934469732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdc8101de0 0x7fcdc81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.706+0000 7fcdcefbb640 1 -- 172.21.15.89:0/2934469732 >> 172.21.15.89:0/2934469732 conn(0x7fcdc80fd8c0 msgr2=0x7fcdc80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.706+0000 7fcdcefbb640 1 -- 172.21.15.89:0/2934469732 shutdown_connections 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.706+0000 7fcdcefbb640 1 -- 172.21.15.89:0/2934469732 wait complete. 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdcefbb640 1 Processor -- start 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdcefbb640 1 -- start start 2024-09-15T23:23:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdcefbb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdc8101de0 0x7fcdc80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdcefbb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc8102790 0x7fcdc8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdcefbb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdc8106f10 0x7fcdc8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdcefbb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc8111f70 con 0x7fcdc8101de0 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdcefbb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc8111df0 con 0x7fcdc8106f10 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdcefbb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc81120f0 con 0x7fcdc8102790 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdccd57640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdc8101de0 0x7fcdc80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdccd57640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdc8101de0 0x7fcdc80744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52358/0 (socket says 172.21.15.89:52358) 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdccd57640 1 -- 172.21.15.89:0/64844457 learned_addr learned my addr 172.21.15.89:0/64844457 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdcd558640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdc8106f10 0x7fcdc8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdccd57640 1 -- 172.21.15.89:0/64844457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc8102790 msgr2=0x7fcdc8074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdccd57640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc8102790 0x7fcdc8074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdccd57640 1 -- 172.21.15.89:0/64844457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdc8106f10 msgr2=0x7fcdc8074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdccd57640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcdc8106f10 0x7fcdc8074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdccd57640 1 -- 172.21.15.89:0/64844457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc81a6820 con 0x7fcdc8101de0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.707+0000 7fcdccd57640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdc8101de0 0x7fcdc80744f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400c920 tx=0x7fcdc400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.708+0000 7fcdbdffb640 1 -- 172.21.15.89:0/64844457 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4007c70 con 0x7fcdc8101de0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.708+0000 7fcdbdffb640 1 -- 172.21.15.89:0/64844457 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdc4007e10 con 0x7fcdc8101de0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.708+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc81a6b10 con 0x7fcdc8101de0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.708+0000 7fcdbdffb640 1 -- 172.21.15.89:0/64844457 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4018890 con 0x7fcdc8101de0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.708+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc81a70a0 con 0x7fcdc8101de0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.709+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcdc80628c0 con 0x7fcdc8101de0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.710+0000 7fcdbdffb640 1 -- 172.21.15.89:0/64844457 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fcdc4018a30 con 0x7fcdc8101de0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.711+0000 7fcdbdffb640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcdb00a3350 0x7fcdb00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.711+0000 7fcdbffff640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcdb00a3350 0x7fcdb00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.711+0000 7fcdbdffb640 1 -- 172.21.15.89:0/64844457 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fcdc40d0320 con 0x7fcdc8101de0 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.711+0000 7fcdbdffb640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] conn(0x7fcdb00ae520 0x7fcdb00b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.711+0000 7fcdbdffb640 1 -- 172.21.15.89:0/64844457 --> [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcdb00b1090 con 0x7fcdb00ae520 2024-09-15T23:23:11.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.711+0000 7fcdbdffb640 1 -- 172.21.15.89:0/64844457 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fcdc40d0800 con 0x7fcdc8101de0 2024-09-15T23:23:11.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.711+0000 7fcdbffff640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcdb00a3350 0x7fcdb00a5810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcdb802a930 tx=0x7fcdb8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:11.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.712+0000 7fcdcd558640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] conn(0x7fcdb00ae520 0x7fcdb00b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:11.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.712+0000 7fcdcd558640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] conn(0x7fcdb00ae520 0x7fcdb00b09c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:11.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.713+0000 7fcdbdffb640 1 -- 172.21.15.89:0/64844457 <== osd.2 v2:172.21.15.174:6800/3193519940 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcdb00b1090 con 0x7fcdb00ae520 2024-09-15T23:23:11.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.726+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 --> [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcdc8062cc0 con 0x7fcdb00ae520 2024-09-15T23:23:11.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.727+0000 7fcdbdffb640 1 -- 172.21.15.89:0/64844457 <== osd.2 v2:172.21.15.174:6800/3193519940 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcdc8062cc0 con 0x7fcdb00ae520 2024-09-15T23:23:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.727+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 >> [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] conn(0x7fcdb00ae520 msgr2=0x7fcdb00b09c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.727+0000 7fcdcefbb640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] conn(0x7fcdb00ae520 0x7fcdb00b09c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.727+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcdb00a3350 msgr2=0x7fcdb00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.727+0000 7fcdcefbb640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcdb00a3350 0x7fcdb00a5810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcdb802a930 tx=0x7fcdb8000bc0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.727+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdc8101de0 msgr2=0x7fcdc80744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.727+0000 7fcdcefbb640 1 --2- 172.21.15.89:0/64844457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdc8101de0 0x7fcdc80744f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400c920 tx=0x7fcdc400cde0 comp rx=0 tx=0).stop 2024-09-15T23:23:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.727+0000 7fcdcd558640 1 -- 172.21.15.89:0/64844457 reap_dead start 2024-09-15T23:23:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.728+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 shutdown_connections 2024-09-15T23:23:11.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.728+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 >> 172.21.15.89:0/64844457 conn(0x7fcdc80fd8c0 msgr2=0x7fcdc80fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:11.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.728+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 shutdown_connections 2024-09-15T23:23:11.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:11.728+0000 7fcdcefbb640 1 -- 172.21.15.89:0/64844457 wait complete. 2024-09-15T23:23:12.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:12 smithi089 ceph-mon[30933]: pgmap v323: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 8 op/s 2024-09-15T23:23:12.426 INFO:teuthology.orchestra.run.smithi089.stdout:180388626486 2024-09-15T23:23:12.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-15T23:23:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:12 smithi120 ceph-mon[35250]: pgmap v323: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 8 op/s 2024-09-15T23:23:12.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:12 smithi174 ceph-mon[35586]: pgmap v323: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 8 op/s 2024-09-15T23:23:12.869 INFO:teuthology.orchestra.run.smithi089.stdout:201863462961 2024-09-15T23:23:12.869 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-15T23:23:13.433 INFO:teuthology.orchestra.run.smithi089.stdout:231928234028 2024-09-15T23:23:13.433 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-15T23:23:13.875 INFO:teuthology.orchestra.run.smithi089.stdout:261993005094 2024-09-15T23:23:13.876 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-15T23:23:14.287 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:14 smithi089 ceph-mon[30933]: pgmap v324: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:14.312 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:14 smithi174 ceph-mon[35586]: pgmap v324: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:14.324 INFO:teuthology.orchestra.run.smithi089.stdout:317827579933 2024-09-15T23:23:14.324 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-15T23:23:14.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:14 smithi120 ceph-mon[35250]: pgmap v324: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:14.766 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:14.812 INFO:teuthology.orchestra.run.smithi089.stdout:128849018945 2024-09-15T23:23:14.812 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-15T23:23:14.822 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:15.297 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:15.297 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:15.298 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:15.298 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:15.336 INFO:teuthology.orchestra.run.smithi089.stdout:73014444107 2024-09-15T23:23:15.337 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-15T23:23:16.082 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:16 smithi089 ceph-mon[30933]: pgmap v325: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:16.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.293+0000 7f06f212e640 1 -- 172.21.15.89:0/1703320623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06ec103660 msgr2=0x7f06ec10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.293+0000 7f06f212e640 1 --2- 172.21.15.89:0/1703320623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06ec103660 0x7f06ec10bb50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f06e0009900 tx=0x7f06e002a480 comp rx=0 tx=0).stop 2024-09-15T23:23:16.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.293+0000 7f06f212e640 1 -- 172.21.15.89:0/1703320623 shutdown_connections 2024-09-15T23:23:16.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.293+0000 7f06f212e640 1 --2- 172.21.15.89:0/1703320623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ec10c170 0x7f06ec10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.293+0000 7f06f212e640 1 --2- 172.21.15.89:0/1703320623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06ec103660 0x7f06ec10bb50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.293+0000 7f06f212e640 1 --2- 172.21.15.89:0/1703320623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec102d40 0x7f06ec103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.293+0000 7f06f212e640 1 -- 172.21.15.89:0/1703320623 >> 172.21.15.89:0/1703320623 conn(0x7f06ec0fc7a0 msgr2=0x7f06ec0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:16.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.293+0000 7f06f212e640 1 -- 172.21.15.89:0/1703320623 shutdown_connections 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.293+0000 7f06f212e640 1 -- 172.21.15.89:0/1703320623 wait complete. 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06f212e640 1 Processor -- start 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06f212e640 1 -- start start 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06f212e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ec102d40 0x7f06ec198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06f212e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec103660 0x7f06ec198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06f212e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06ec10c170 0x7f06ec19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06f212e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec111f70 con 0x7f06ec10c170 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06f212e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec111df0 con 0x7f06ec103660 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06f212e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec1120f0 con 0x7f06ec102d40 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06ec10c170 0x7f06ec19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06ec10c170 0x7f06ec19cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58726/0 (socket says 172.21.15.89:58726) 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06ebfff640 1 -- 172.21.15.89:0/3456864123 learned_addr learned my addr 172.21.15.89:0/3456864123 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:16.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06eaffd640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec103660 0x7f06ec198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06eb7fe640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ec102d40 0x7f06ec198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06ebfff640 1 -- 172.21.15.89:0/3456864123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ec102d40 msgr2=0x7f06ec198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06ebfff640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ec102d40 0x7f06ec198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06ebfff640 1 -- 172.21.15.89:0/3456864123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec103660 msgr2=0x7f06ec198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06ebfff640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec103660 0x7f06ec198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06ebfff640 1 -- 172.21.15.89:0/3456864123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06ec19d3e0 con 0x7f06ec10c170 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.294+0000 7f06eb7fe640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ec102d40 0x7f06ec198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.295+0000 7f06ebfff640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06ec10c170 0x7f06ec19cd00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f06e400d950 tx=0x7f06e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.295+0000 7f06e8ff9640 1 -- 172.21.15.89:0/3456864123 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4017030 con 0x7f06ec10c170 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.295+0000 7f06e8ff9640 1 -- 172.21.15.89:0/3456864123 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e4004020 con 0x7f06ec10c170 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.295+0000 7f06f212e640 1 -- 172.21.15.89:0/3456864123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06ec19d6d0 con 0x7f06ec10c170 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.295+0000 7f06e8ff9640 1 -- 172.21.15.89:0/3456864123 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4003c00 con 0x7f06ec10c170 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.295+0000 7f06f212e640 1 -- 172.21.15.89:0/3456864123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06ec1a4fb0 con 0x7f06ec10c170 2024-09-15T23:23:16.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.296+0000 7f06f212e640 1 -- 172.21.15.89:0/3456864123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06ec103120 con 0x7f06ec10c170 2024-09-15T23:23:16.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.297+0000 7f06e8ff9640 1 -- 172.21.15.89:0/3456864123 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f06e401a070 con 0x7f06ec10c170 2024-09-15T23:23:16.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.297+0000 7f06e8ff9640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f06d40a3350 0x7f06d40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.298+0000 7f06eb7fe640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f06d40a3350 0x7f06d40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.298+0000 7f06e8ff9640 1 -- 172.21.15.89:0/3456864123 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f06e40cf5d0 con 0x7f06ec10c170 2024-09-15T23:23:16.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.298+0000 7f06eb7fe640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f06d40a3350 0x7f06d40a5810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f06dc005fd0 tx=0x7f06dc0093e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:16.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.302+0000 7f06e8ff9640 1 -- 172.21.15.89:0/3456864123 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f06e4021070 con 0x7f06ec10c170 2024-09-15T23:23:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:16 smithi120 ceph-mon[35250]: pgmap v325: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:16.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.472+0000 7f06f212e640 1 -- 172.21.15.89:0/3456864123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f06ec02d7c0 con 0x7f06ec10c170 2024-09-15T23:23:16.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.473+0000 7f06e8ff9640 1 -- 172.21.15.89:0/3456864123 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f06e409be30 con 0x7f06ec10c170 2024-09-15T23:23:16.475 INFO:teuthology.orchestra.run.smithi089.stdout:51539607633 2024-09-15T23:23:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 -- 172.21.15.89:0/3456864123 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f06d40a3350 msgr2=0x7f06d40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f06d40a3350 0x7f06d40a5810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f06dc005fd0 tx=0x7f06dc0093e0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 -- 172.21.15.89:0/3456864123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06ec10c170 msgr2=0x7f06ec19cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06ec10c170 0x7f06ec19cd00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f06e400d950 tx=0x7f06e400de10 comp rx=0 tx=0).stop 2024-09-15T23:23:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 -- 172.21.15.89:0/3456864123 shutdown_connections 2024-09-15T23:23:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f06d40a3350 0x7f06d40a5810 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06ec10c170 0x7f06ec19cd00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06ec103660 0x7f06ec198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 --2- 172.21.15.89:0/3456864123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06ec102d40 0x7f06ec198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 -- 172.21.15.89:0/3456864123 >> 172.21.15.89:0/3456864123 conn(0x7f06ec0fc7a0 msgr2=0x7f06ec0fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:16.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 -- 172.21.15.89:0/3456864123 shutdown_connections 2024-09-15T23:23:16.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:16.475+0000 7f06f212e640 1 -- 172.21.15.89:0/3456864123 wait complete. 2024-09-15T23:23:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:16 smithi174 ceph-mon[35586]: pgmap v325: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:17.283 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:17 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3456864123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-15T23:23:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.111+0000 7ff70fd4a640 1 -- 172.21.15.89:0/4020017948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff708100e20 msgr2=0x7ff708101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.111+0000 7ff70fd4a640 1 --2- 172.21.15.89:0/4020017948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff708100e20 0x7ff708101200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff704009900 tx=0x7ff70402a480 comp rx=0 tx=0).stop 2024-09-15T23:23:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.111+0000 7ff70fd4a640 1 -- 172.21.15.89:0/4020017948 shutdown_connections 2024-09-15T23:23:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.111+0000 7ff70fd4a640 1 --2- 172.21.15.89:0/4020017948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff708102ef0 0x7ff70810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.111+0000 7ff70fd4a640 1 --2- 172.21.15.89:0/4020017948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff708102570 0x7ff7081029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.111+0000 7ff70fd4a640 1 --2- 172.21.15.89:0/4020017948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff708100e20 0x7ff708101200 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.111+0000 7ff70fd4a640 1 -- 172.21.15.89:0/4020017948 >> 172.21.15.89:0/4020017948 conn(0x7ff7080fc7d0 msgr2=0x7ff7080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:17.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70fd4a640 1 -- 172.21.15.89:0/4020017948 shutdown_connections 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70fd4a640 1 -- 172.21.15.89:0/4020017948 wait complete. 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70fd4a640 1 Processor -- start 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70fd4a640 1 -- start start 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70fd4a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff708100e20 0x7ff7080fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70fd4a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff708102570 0x7ff7080fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70fd4a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff708102ef0 0x7ff7080fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70fd4a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff708111020 con 0x7ff708100e20 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70fd4a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff708110ea0 con 0x7ff708102570 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70fd4a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff7081111a0 con 0x7ff708102ef0 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70dae6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff708100e20 0x7ff7080fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70dae6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff708100e20 0x7ff7080fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58742/0 (socket says 172.21.15.89:58742) 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.112+0000 7ff70dae6640 1 -- 172.21.15.89:0/3830066628 learned_addr learned my addr 172.21.15.89:0/3830066628 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:17.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff70d2e5640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff708102570 0x7ff7080fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:17.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff70dae6640 1 -- 172.21.15.89:0/3830066628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff708102ef0 msgr2=0x7ff7080fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:17.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff70dae6640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff708102ef0 0x7ff7080fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:17.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff70dae6640 1 -- 172.21.15.89:0/3830066628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff708102570 msgr2=0x7ff7080fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:17.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff70dae6640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff708102570 0x7ff7080fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:17.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff70dae6640 1 -- 172.21.15.89:0/3830066628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7081a6ba0 con 0x7ff708100e20 2024-09-15T23:23:17.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff70d2e5640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff708102570 0x7ff7080fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:17.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff70dae6640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff708100e20 0x7ff7080fcc80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff704006010 tx=0x7ff704003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:17.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff6feffd640 1 -- 172.21.15.89:0/3830066628 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70403f070 con 0x7ff708100e20 2024-09-15T23:23:17.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff70fd4a640 1 -- 172.21.15.89:0/3830066628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7081a6e30 con 0x7ff708100e20 2024-09-15T23:23:17.435 INFO:teuthology.orchestra.run.smithi089.stdout:180388626488 2024-09-15T23:23:17.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff6feffd640 1 -- 172.21.15.89:0/3830066628 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff704031e30 con 0x7ff708100e20 2024-09-15T23:23:17.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff6feffd640 1 -- 172.21.15.89:0/3830066628 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704035030 con 0x7ff708100e20 2024-09-15T23:23:17.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.113+0000 7ff70fd4a640 1 -- 172.21.15.89:0/3830066628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7081a72e0 con 0x7ff708100e20 2024-09-15T23:23:17.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.114+0000 7ff70fd4a640 1 -- 172.21.15.89:0/3830066628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff708101280 con 0x7ff708100e20 2024-09-15T23:23:17.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.116+0000 7ff6feffd640 1 -- 172.21.15.89:0/3830066628 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff704042070 con 0x7ff708100e20 2024-09-15T23:23:17.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.116+0000 7ff6feffd640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6f40a32d0 0x7ff6f40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:17.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.117+0000 7ff6feffd640 1 -- 172.21.15.89:0/3830066628 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7ff7040f0dc0 con 0x7ff708100e20 2024-09-15T23:23:17.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.117+0000 7ff70d2e5640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6f40a32d0 0x7ff6f40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:17.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.117+0000 7ff70d2e5640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6f40a32d0 0x7ff6f40a5790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff7080fe710 tx=0x7ff6f8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.120+0000 7ff6feffd640 1 -- 172.21.15.89:0/3830066628 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff7040bd6b0 con 0x7ff708100e20 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.307+0000 7ff70fd4a640 1 -- 172.21.15.89:0/3830066628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7ff70810a0a0 con 0x7ff708100e20 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.308+0000 7ff6feffd640 1 -- 172.21.15.89:0/3830066628 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff7040bcfb0 con 0x7ff708100e20 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.309+0000 7ff70fd4a640 1 -- 172.21.15.89:0/3830066628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6f40a32d0 msgr2=0x7ff6f40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.309+0000 7ff70fd4a640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6f40a32d0 0x7ff6f40a5790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff7080fe710 tx=0x7ff6f8008040 comp rx=0 tx=0).stop 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.309+0000 7ff70fd4a640 1 -- 172.21.15.89:0/3830066628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff708100e20 msgr2=0x7ff7080fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.309+0000 7ff70fd4a640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff708100e20 0x7ff7080fcc80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff704006010 tx=0x7ff704003c80 comp rx=0 tx=0).stop 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.310+0000 7ff70fd4a640 1 -- 172.21.15.89:0/3830066628 shutdown_connections 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.310+0000 7ff70fd4a640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6f40a32d0 0x7ff6f40a5790 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.310+0000 7ff70fd4a640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff708102ef0 0x7ff7080fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.310+0000 7ff70fd4a640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff708102570 0x7ff7080fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.310+0000 7ff70fd4a640 1 --2- 172.21.15.89:0/3830066628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff708100e20 0x7ff7080fcc80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.310+0000 7ff70fd4a640 1 -- 172.21.15.89:0/3830066628 >> 172.21.15.89:0/3830066628 conn(0x7ff7080fc7d0 msgr2=0x7ff70806a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:17.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.310+0000 7ff70fd4a640 1 -- 172.21.15.89:0/3830066628 shutdown_connections 2024-09-15T23:23:17.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:17.310+0000 7ff70fd4a640 1 -- 172.21.15.89:0/3830066628 wait complete. 2024-09-15T23:23:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:17 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3456864123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-15T23:23:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:17 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3456864123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-15T23:23:18.198 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:18 smithi089 ceph-mon[30933]: pgmap v326: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-15T23:23:18.198 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:18 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3830066628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-15T23:23:18.225 INFO:tasks.cephadm.ceph_manager.ceph:need seq 51539607630 got 51539607633 for osd.1 2024-09-15T23:23:18.225 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:18.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:18 smithi120 ceph-mon[35250]: pgmap v326: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-15T23:23:18.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:18 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3830066628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-15T23:23:18.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:18 smithi174 ceph-mon[35586]: pgmap v326: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-09-15T23:23:18.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:18 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3830066628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-15T23:23:19.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.136+0000 7f4268f3b640 1 -- 172.21.15.89:0/2663794430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4264100a80 msgr2=0x7f4264100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.136+0000 7f4268f3b640 1 --2- 172.21.15.89:0/2663794430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4264100a80 0x7f4264100ee0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f424c009930 tx=0x7f424c02a420 comp rx=0 tx=0).stop 2024-09-15T23:23:19.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.136+0000 7f4268f3b640 1 -- 172.21.15.89:0/2663794430 shutdown_connections 2024-09-15T23:23:19.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.136+0000 7f4268f3b640 1 --2- 172.21.15.89:0/2663794430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4264101420 0x7f426410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.136+0000 7f4268f3b640 1 --2- 172.21.15.89:0/2663794430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4264100a80 0x7f4264100ee0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.136+0000 7f4268f3b640 1 --2- 172.21.15.89:0/2663794430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4264106a80 0x7f4264106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.136+0000 7f4268f3b640 1 -- 172.21.15.89:0/2663794430 >> 172.21.15.89:0/2663794430 conn(0x7f42640fc7a0 msgr2=0x7f42640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:19.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.136+0000 7f4268f3b640 1 -- 172.21.15.89:0/2663794430 shutdown_connections 2024-09-15T23:23:19.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.136+0000 7f4268f3b640 1 -- 172.21.15.89:0/2663794430 wait complete. 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f4268f3b640 1 Processor -- start 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f4268f3b640 1 -- start start 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f4268f3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4264100a80 0x7f4264197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f4268f3b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4264101420 0x7f4264197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f4268f3b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4264106a80 0x7f426419bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f4268f3b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4264110e80 con 0x7f4264100a80 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f4268f3b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4264110d00 con 0x7f4264101420 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f4268f3b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4264111000 con 0x7f4264106a80 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f426259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4264100a80 0x7f4264197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f426259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4264100a80 0x7f4264197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58758/0 (socket says 172.21.15.89:58758) 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f426259c640 1 -- 172.21.15.89:0/16864099 learned_addr learned my addr 172.21.15.89:0/16864099 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:19.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f425bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4264101420 0x7f4264197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f4262d9d640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4264106a80 0x7f426419bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f426259c640 1 -- 172.21.15.89:0/16864099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4264106a80 msgr2=0x7f426419bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f426259c640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4264106a80 0x7f426419bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f426259c640 1 -- 172.21.15.89:0/16864099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4264101420 msgr2=0x7f4264197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f426259c640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4264101420 0x7f4264197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f426259c640 1 -- 172.21.15.89:0/16864099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426419c2b0 con 0x7f4264100a80 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.137+0000 7f4262d9d640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4264106a80 0x7f426419bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.138+0000 7f426259c640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4264100a80 0x7f4264197300 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f425400c970 tx=0x7f425400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.138+0000 7f425b7fe640 1 -- 172.21.15.89:0/16864099 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254007bb0 con 0x7f4264100a80 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.138+0000 7f4268f3b640 1 -- 172.21.15.89:0/16864099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f426419c5a0 con 0x7f4264100a80 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.138+0000 7f425b7fe640 1 -- 172.21.15.89:0/16864099 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4254007d50 con 0x7f4264100a80 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.138+0000 7f425b7fe640 1 -- 172.21.15.89:0/16864099 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425400f450 con 0x7f4264100a80 2024-09-15T23:23:19.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.138+0000 7f4268f3b640 1 -- 172.21.15.89:0/16864099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42641a3e80 con 0x7f4264100a80 2024-09-15T23:23:19.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.139+0000 7f4268f3b640 1 -- 172.21.15.89:0/16864099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4264100ee0 con 0x7f4264100a80 2024-09-15T23:23:19.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.140+0000 7f425b7fe640 1 -- 172.21.15.89:0/16864099 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4254004fe0 con 0x7f4264100a80 2024-09-15T23:23:19.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.141+0000 7f425b7fe640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42500a32d0 0x7f42500a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.141+0000 7f425b7fe640 1 -- 172.21.15.89:0/16864099 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f42540cff10 con 0x7f4264100a80 2024-09-15T23:23:19.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.141+0000 7f425bfff640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42500a32d0 0x7f42500a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.141+0000 7f425bfff640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42500a32d0 0x7f42500a5790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f424c009900 tx=0x7f424c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:19.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.144+0000 7f425b7fe640 1 -- 172.21.15.89:0/16864099 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4254014030 con 0x7f4264100a80 2024-09-15T23:23:19.300 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626486 got 180388626488 for osd.6 2024-09-15T23:23:19.300 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:19.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.315+0000 7f4268f3b640 1 -- 172.21.15.89:0/16864099 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f4264109050 con 0x7f4264100a80 2024-09-15T23:23:19.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.315+0000 7f425b7fe640 1 -- 172.21.15.89:0/16864099 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f425409c770 con 0x7f4264100a80 2024-09-15T23:23:19.317 INFO:teuthology.orchestra.run.smithi089.stdout:150323855421 2024-09-15T23:23:19.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 -- 172.21.15.89:0/16864099 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42500a32d0 msgr2=0x7f42500a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42500a32d0 0x7f42500a5790 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f424c009900 tx=0x7f424c0058d0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 -- 172.21.15.89:0/16864099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4264100a80 msgr2=0x7f4264197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4264100a80 0x7f4264197300 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f425400c970 tx=0x7f425400ce30 comp rx=0 tx=0).stop 2024-09-15T23:23:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 -- 172.21.15.89:0/16864099 shutdown_connections 2024-09-15T23:23:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f42500a32d0 0x7f42500a5790 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4264106a80 0x7f426419bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4264101420 0x7f4264197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 --2- 172.21.15.89:0/16864099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4264100a80 0x7f4264197300 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 -- 172.21.15.89:0/16864099 >> 172.21.15.89:0/16864099 conn(0x7f42640fc7a0 msgr2=0x7f42640fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 -- 172.21.15.89:0/16864099 shutdown_connections 2024-09-15T23:23:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.317+0000 7f4268f3b640 1 -- 172.21.15.89:0/16864099 wait complete. 2024-09-15T23:23:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 -- 172.21.15.89:0/3792330252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa78100e20 msgr2=0x7ffa78101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 --2- 172.21.15.89:0/3792330252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa78100e20 0x7ffa78101200 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009900 tx=0x7ffa6802a480 comp rx=0 tx=0).stop 2024-09-15T23:23:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 -- 172.21.15.89:0/3792330252 shutdown_connections 2024-09-15T23:23:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 --2- 172.21.15.89:0/3792330252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa78102ef0 0x7ffa7810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 --2- 172.21.15.89:0/3792330252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa78102570 0x7ffa781029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 --2- 172.21.15.89:0/3792330252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa78100e20 0x7ffa78101200 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 -- 172.21.15.89:0/3792330252 >> 172.21.15.89:0/3792330252 conn(0x7ffa780fc7d0 msgr2=0x7ffa780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 -- 172.21.15.89:0/3792330252 shutdown_connections 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 -- 172.21.15.89:0/3792330252 wait complete. 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 Processor -- start 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.502+0000 7ffa7de35640 1 -- start start 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa7de35640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa78100e20 0x7ffa78199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa7de35640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa78102570 0x7ffa781999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa7de35640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa78102ef0 0x7ffa78199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa7de35640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffa78110f50 con 0x7ffa78102ef0 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa7de35640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa78110dd0 con 0x7ffa78100e20 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa7de35640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa781110d0 con 0x7ffa78102570 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa77fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa78102ef0 0x7ffa78199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa77fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa78102ef0 0x7ffa78199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58788/0 (socket says 172.21.15.89:58788) 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa77fff640 1 -- 172.21.15.89:0/2411176622 learned_addr learned my addr 172.21.15.89:0/2411176622 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa777fe640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa78100e20 0x7ffa78199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa6effd640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa78102570 0x7ffa781999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa77fff640 1 -- 172.21.15.89:0/2411176622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa78102570 msgr2=0x7ffa781999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa77fff640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa78102570 0x7ffa781999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa77fff640 1 -- 172.21.15.89:0/2411176622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa78100e20 msgr2=0x7ffa78199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa77fff640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa78100e20 0x7ffa78199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa77fff640 1 -- 172.21.15.89:0/2411176622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa781a4860 con 0x7ffa78102ef0 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa777fe640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa78100e20 0x7ffa78199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.503+0000 7ffa77fff640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa78102ef0 0x7ffa78199f20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000eaa0 tx=0x7ffa7000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.504+0000 7ffa757fa640 1 -- 172.21.15.89:0/2411176622 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7000cdb0 con 0x7ffa78102ef0 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.504+0000 7ffa757fa640 1 -- 172.21.15.89:0/2411176622 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa70005020 con 0x7ffa78102ef0 2024-09-15T23:23:19.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.504+0000 7ffa7de35640 1 -- 172.21.15.89:0/2411176622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa781a4b50 con 0x7ffa78102ef0 2024-09-15T23:23:19.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.504+0000 7ffa757fa640 1 -- 172.21.15.89:0/2411176622 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa700116b0 con 0x7ffa78102ef0 2024-09-15T23:23:19.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.504+0000 7ffa7de35640 1 -- 172.21.15.89:0/2411176622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa781a5040 con 0x7ffa78102ef0 2024-09-15T23:23:19.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.505+0000 7ffa7de35640 1 -- 172.21.15.89:0/2411176622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa78101200 con 0x7ffa78102ef0 2024-09-15T23:23:19.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.506+0000 7ffa757fa640 1 -- 172.21.15.89:0/2411176622 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ffa70005560 con 0x7ffa78102ef0 2024-09-15T23:23:19.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.507+0000 7ffa757fa640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffa640a32d0 0x7ffa640a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.507+0000 7ffa777fe640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffa640a32d0 0x7ffa640a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.507+0000 7ffa757fa640 1 -- 172.21.15.89:0/2411176622 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7ffa700d0450 con 0x7ffa78102ef0 2024-09-15T23:23:19.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.507+0000 7ffa777fe640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffa640a32d0 0x7ffa640a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffa68000c00 tx=0x7ffa68004100 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:19.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.510+0000 7ffa757fa640 1 -- 172.21.15.89:0/2411176622 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffa70016030 con 0x7ffa78102ef0 2024-09-15T23:23:19.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.686+0000 7ffa7de35640 1 -- 172.21.15.89:0/2411176622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7ffa7806a560 con 0x7ffa78102ef0 2024-09-15T23:23:19.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.686+0000 7ffa757fa640 1 -- 172.21.15.89:0/2411176622 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ffa7009ccb0 con 0x7ffa78102ef0 2024-09-15T23:23:19.689 INFO:teuthology.orchestra.run.smithi089.stdout:261993005097 2024-09-15T23:23:19.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 -- 172.21.15.89:0/2411176622 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffa640a32d0 msgr2=0x7ffa640a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffa640a32d0 0x7ffa640a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffa68000c00 tx=0x7ffa68004100 comp rx=0 tx=0).stop 2024-09-15T23:23:19.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 -- 172.21.15.89:0/2411176622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa78102ef0 msgr2=0x7ffa78199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa78102ef0 0x7ffa78199f20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffa7000eaa0 tx=0x7ffa7000ef60 comp rx=0 tx=0).stop 2024-09-15T23:23:19.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 -- 172.21.15.89:0/2411176622 shutdown_connections 2024-09-15T23:23:19.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffa640a32d0 0x7ffa640a5790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa78102ef0 0x7ffa78199f20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa78102570 0x7ffa781999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 --2- 172.21.15.89:0/2411176622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa78100e20 0x7ffa78199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 -- 172.21.15.89:0/2411176622 >> 172.21.15.89:0/2411176622 conn(0x7ffa780fc7d0 msgr2=0x7ffa780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 -- 172.21.15.89:0/2411176622 shutdown_connections 2024-09-15T23:23:19.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.689+0000 7ffa7de35640 1 -- 172.21.15.89:0/2411176622 wait complete. 2024-09-15T23:23:19.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.716+0000 7fb10fb3d640 1 -- 172.21.15.89:0/2621299689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb108106f20 msgr2=0x7fb10810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.716+0000 7fb10fb3d640 1 --2- 172.21.15.89:0/2621299689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb108106f20 0x7fb10810eaa0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb100009ca0 tx=0x7fb10002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.716+0000 7fb10fb3d640 1 -- 172.21.15.89:0/2621299689 shutdown_connections 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.716+0000 7fb10fb3d640 1 --2- 172.21.15.89:0/2621299689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb108106f20 0x7fb10810eaa0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.716+0000 7fb10fb3d640 1 --2- 172.21.15.89:0/2621299689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1081027a0 0x7fb1081067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.716+0000 7fb10fb3d640 1 --2- 172.21.15.89:0/2621299689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb108101df0 0x7fb1081021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.716+0000 7fb10fb3d640 1 -- 172.21.15.89:0/2621299689 >> 172.21.15.89:0/2621299689 conn(0x7fb1080fd8f0 msgr2=0x7fb1080ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.716+0000 7fb10fb3d640 1 -- 172.21.15.89:0/2621299689 shutdown_connections 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.716+0000 7fb10fb3d640 1 -- 172.21.15.89:0/2621299689 wait complete. 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.717+0000 7fb10fb3d640 1 Processor -- start 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.717+0000 7fb10fb3d640 1 -- start start 2024-09-15T23:23:19.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.717+0000 7fb10fb3d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb108101df0 0x7fb1081983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.717+0000 7fb10fb3d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1081027a0 0x7fb108198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.717+0000 7fb10fb3d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb108106f20 0x7fb10819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.717+0000 7fb10fb3d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb108111fa0 con 0x7fb1081027a0 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.717+0000 7fb10fb3d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb108111e20 con 0x7fb108106f20 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.717+0000 7fb10fb3d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb108112120 con 0x7fb108101df0 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10d0d8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1081027a0 0x7fb108198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10d0d8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1081027a0 0x7fb108198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58814/0 (socket says 172.21.15.89:58814) 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10d0d8640 1 -- 172.21.15.89:0/3313467345 learned_addr learned my addr 172.21.15.89:0/3313467345 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10d8d9640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb108101df0 0x7fb1081983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10e0da640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb108106f20 0x7fb10819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10d0d8640 1 -- 172.21.15.89:0/3313467345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb108101df0 msgr2=0x7fb1081983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10d0d8640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb108101df0 0x7fb1081983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10d0d8640 1 -- 172.21.15.89:0/3313467345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb108106f20 msgr2=0x7fb10819cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10d0d8640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb108106f20 0x7fb10819cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10d0d8640 1 -- 172.21.15.89:0/3313467345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10819d380 con 0x7fb1081027a0 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.718+0000 7fb10d0d8640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1081027a0 0x7fb108198910 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800c920 tx=0x7fb0f800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.719+0000 7fb0feffd640 1 -- 172.21.15.89:0/3313467345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8007c70 con 0x7fb1081027a0 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.719+0000 7fb0feffd640 1 -- 172.21.15.89:0/3313467345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0f8007e10 con 0x7fb1081027a0 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.719+0000 7fb10fb3d640 1 -- 172.21.15.89:0/3313467345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10819d670 con 0x7fb1081027a0 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.719+0000 7fb0feffd640 1 -- 172.21.15.89:0/3313467345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8018890 con 0x7fb1081027a0 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.719+0000 7fb10fb3d640 1 -- 172.21.15.89:0/3313467345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1081a4f50 con 0x7fb1081027a0 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.720+0000 7fb10fb3d640 1 -- 172.21.15.89:0/3313467345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1081021d0 con 0x7fb1081027a0 2024-09-15T23:23:19.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.721+0000 7fb0feffd640 1 -- 172.21.15.89:0/3313467345 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb0f8018a70 con 0x7fb1081027a0 2024-09-15T23:23:19.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.721+0000 7fb0feffd640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0f40a32d0 0x7fb0f40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:19.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.722+0000 7fb10d8d9640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0f40a32d0 0x7fb0f40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:19.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.722+0000 7fb0feffd640 1 -- 172.21.15.89:0/3313467345 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fb0f80d0020 con 0x7fb1081027a0 2024-09-15T23:23:19.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.722+0000 7fb10d8d9640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0f40a32d0 0x7fb0f40a5790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb104009890 tx=0x7fb104008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:19.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.726+0000 7fb0feffd640 1 -- 172.21.15.89:0/3313467345 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb0f8014030 con 0x7fb1081027a0 2024-09-15T23:23:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.901+0000 7fb10fb3d640 1 -- 172.21.15.89:0/3313467345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb10810a160 con 0x7fb1081027a0 2024-09-15T23:23:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.902+0000 7fb0feffd640 1 -- 172.21.15.89:0/3313467345 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb0f800de20 con 0x7fb1081027a0 2024-09-15T23:23:19.904 INFO:teuthology.orchestra.run.smithi089.stdout:98784247878 2024-09-15T23:23:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.903+0000 7fb10fb3d640 1 -- 172.21.15.89:0/3313467345 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0f40a32d0 msgr2=0x7fb0f40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.903+0000 7fb10fb3d640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0f40a32d0 0x7fb0f40a5790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb104009890 tx=0x7fb104008040 comp rx=0 tx=0).stop 2024-09-15T23:23:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.904+0000 7fb10fb3d640 1 -- 172.21.15.89:0/3313467345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1081027a0 msgr2=0x7fb108198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.904+0000 7fb10fb3d640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1081027a0 0x7fb108198910 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800c920 tx=0x7fb0f800cde0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.904+0000 7fb10fb3d640 1 -- 172.21.15.89:0/3313467345 shutdown_connections 2024-09-15T23:23:19.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.904+0000 7fb10fb3d640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0f40a32d0 0x7fb0f40a5790 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.904+0000 7fb10fb3d640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb108106f20 0x7fb10819cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.904+0000 7fb10fb3d640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1081027a0 0x7fb108198910 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.904+0000 7fb10fb3d640 1 --2- 172.21.15.89:0/3313467345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb108101df0 0x7fb1081983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.904+0000 7fb10fb3d640 1 -- 172.21.15.89:0/3313467345 >> 172.21.15.89:0/3313467345 conn(0x7fb1080fd8f0 msgr2=0x7fb1080fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.904+0000 7fb10fb3d640 1 -- 172.21.15.89:0/3313467345 shutdown_connections 2024-09-15T23:23:19.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:19.904+0000 7fb10fb3d640 1 -- 172.21.15.89:0/3313467345 wait complete. 2024-09-15T23:23:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 -- 172.21.15.89:0/2244255900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f70101de0 msgr2=0x7f4f701021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 --2- 172.21.15.89:0/2244255900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f70101de0 0x7f4f701021c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4f68009900 tx=0x7f4f6802a410 comp rx=0 tx=0).stop 2024-09-15T23:23:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 -- 172.21.15.89:0/2244255900 shutdown_connections 2024-09-15T23:23:20.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 --2- 172.21.15.89:0/2244255900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f70106f10 0x7f4f7010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 --2- 172.21.15.89:0/2244255900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f70102790 0x7f4f701067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 --2- 172.21.15.89:0/2244255900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f70101de0 0x7f4f701021c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 -- 172.21.15.89:0/2244255900 >> 172.21.15.89:0/2244255900 conn(0x7f4f700fd8c0 msgr2=0x7f4f700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 -- 172.21.15.89:0/2244255900 shutdown_connections 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 -- 172.21.15.89:0/2244255900 wait complete. 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 Processor -- start 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.023+0000 7f4f76d74640 1 -- start start 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f76d74640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f70101de0 0x7f4f700744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f76d74640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f70102790 0x7f4f70074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f76d74640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f70106f10 0x7f4f70074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:20.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f76d74640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4f70111fb0 con 0x7f4f70102790 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f76d74640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4f70111e30 con 0x7f4f70101de0 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f76d74640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4f70112130 con 0x7f4f70106f10 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f70102790 0x7f4f70074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f70102790 0x7f4f70074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58826/0 (socket says 172.21.15.89:58826) 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6ffff640 1 -- 172.21.15.89:0/1828691955 learned_addr learned my addr 172.21.15.89:0/1828691955 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f74b10640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f70101de0 0x7f4f700744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6ffff640 1 -- 172.21.15.89:0/1828691955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f70106f10 msgr2=0x7f4f70074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6ffff640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f70106f10 0x7f4f70074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6ffff640 1 -- 172.21.15.89:0/1828691955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f70101de0 msgr2=0x7f4f700744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6ffff640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f70101de0 0x7f4f700744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:20.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6ffff640 1 -- 172.21.15.89:0/1828691955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f701a6970 con 0x7f4f70102790 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6ffff640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f70102790 0x7f4f70074a30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000ea80 tx=0x7f4f6000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6dffb640 1 -- 172.21.15.89:0/1828691955 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6000ce40 con 0x7f4f70102790 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6dffb640 1 -- 172.21.15.89:0/1828691955 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f60005020 con 0x7f4f70102790 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f76d74640 1 -- 172.21.15.89:0/1828691955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f701a6c60 con 0x7f4f70102790 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f6dffb640 1 -- 172.21.15.89:0/1828691955 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60011760 con 0x7f4f70102790 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.024+0000 7f4f76d74640 1 -- 172.21.15.89:0/1828691955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f701a7150 con 0x7f4f70102790 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.026+0000 7f4f76d74640 1 -- 172.21.15.89:0/1828691955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f701021c0 con 0x7f4f70102790 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.027+0000 7f4f6dffb640 1 -- 172.21.15.89:0/1828691955 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4f600051d0 con 0x7f4f70102790 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.027+0000 7f4f6dffb640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4f5c0a32d0 0x7f4f5c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.027+0000 7f4f74b10640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4f5c0a32d0 0x7f4f5c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.027+0000 7f4f6dffb640 1 -- 172.21.15.89:0/1828691955 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f4f600d04d0 con 0x7f4f70102790 2024-09-15T23:23:20.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.028+0000 7f4f74b10640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4f5c0a32d0 0x7f4f5c0a5790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4f680096c0 tx=0x7f4f68000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:20.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.031+0000 7f4f6dffb640 1 -- 172.21.15.89:0/1828691955 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4f60016030 con 0x7f4f70102790 2024-09-15T23:23:20.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.202+0000 7f4f76d74640 1 -- 172.21.15.89:0/1828691955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f4f70075a20 con 0x7f4f70102790 2024-09-15T23:23:20.204 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:20 smithi089 ceph-mon[30933]: pgmap v327: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-15T23:23:20.204 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:20 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/16864099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-15T23:23:20.204 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:20 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2411176622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-15T23:23:20.205 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:20 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3313467345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-15T23:23:20.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.202+0000 7f4f6dffb640 1 -- 172.21.15.89:0/1828691955 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4f6009cd30 con 0x7f4f70102790 2024-09-15T23:23:20.205 INFO:teuthology.orchestra.run.smithi089.stdout:231928234030 2024-09-15T23:23:20.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 -- 172.21.15.89:0/1828691955 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4f5c0a32d0 msgr2=0x7f4f5c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4f5c0a32d0 0x7f4f5c0a5790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4f680096c0 tx=0x7f4f68000bc0 comp rx=0 tx=0).stop 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 -- 172.21.15.89:0/1828691955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f70102790 msgr2=0x7f4f70074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f70102790 0x7f4f70074a30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000ea80 tx=0x7f4f6000ef40 comp rx=0 tx=0).stop 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 -- 172.21.15.89:0/1828691955 shutdown_connections 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4f5c0a32d0 0x7f4f5c0a5790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4f70106f10 0x7f4f70074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f70102790 0x7f4f70074a30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 --2- 172.21.15.89:0/1828691955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f70101de0 0x7f4f700744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 -- 172.21.15.89:0/1828691955 >> 172.21.15.89:0/1828691955 conn(0x7f4f700fd8c0 msgr2=0x7f4f700fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 -- 172.21.15.89:0/1828691955 shutdown_connections 2024-09-15T23:23:20.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:20.204+0000 7f4f76d74640 1 -- 172.21.15.89:0/1828691955 wait complete. 2024-09-15T23:23:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:20 smithi120 ceph-mon[35250]: pgmap v327: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-15T23:23:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:20 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/16864099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-15T23:23:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:20 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2411176622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-15T23:23:20.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:20 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3313467345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-15T23:23:20.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:20 smithi174 ceph-mon[35586]: pgmap v327: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-09-15T23:23:20.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:20 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/16864099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-15T23:23:20.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:20 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2411176622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-15T23:23:20.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:20 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3313467345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-15T23:23:20.765 INFO:tasks.cephadm.ceph_manager.ceph:need seq 98784247876 got 98784247878 for osd.3 2024-09-15T23:23:20.765 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:21.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:21 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1828691955' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-15T23:23:21.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:21 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1828691955' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-15T23:23:21.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:21 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1828691955' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-15T23:23:21.668 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855418 got 150323855421 for osd.5 2024-09-15T23:23:21.668 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:22.115 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:22.126 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234028 got 231928234030 for osd.8 2024-09-15T23:23:22.126 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:22 smithi120 ceph-mon[35250]: pgmap v328: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-09-15T23:23:22.477 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:22 smithi089 ceph-mon[30933]: pgmap v328: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-09-15T23:23:22.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:22 smithi174 ceph-mon[35586]: pgmap v328: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-09-15T23:23:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:22.855 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005094 got 261993005097 for osd.9 2024-09-15T23:23:22.855 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.472+0000 7fd096227640 1 -- 172.21.15.89:0/1638462500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd090102ef0 msgr2=0x7fd09010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:23.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.472+0000 7fd096227640 1 --2- 172.21.15.89:0/1638462500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd090102ef0 0x7fd09010da40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd088009ca0 tx=0x7fd08802a950 comp rx=0 tx=0).stop 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.472+0000 7fd096227640 1 -- 172.21.15.89:0/1638462500 shutdown_connections 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.472+0000 7fd096227640 1 --2- 172.21.15.89:0/1638462500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd090102ef0 0x7fd09010da40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.472+0000 7fd096227640 1 --2- 172.21.15.89:0/1638462500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd090102570 0x7fd0901029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.472+0000 7fd096227640 1 --2- 172.21.15.89:0/1638462500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd090100e20 0x7fd090101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.472+0000 7fd096227640 1 -- 172.21.15.89:0/1638462500 >> 172.21.15.89:0/1638462500 conn(0x7fd0900fc7d0 msgr2=0x7fd0900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.472+0000 7fd096227640 1 -- 172.21.15.89:0/1638462500 shutdown_connections 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.472+0000 7fd096227640 1 -- 172.21.15.89:0/1638462500 wait complete. 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.472+0000 7fd096227640 1 Processor -- start 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd096227640 1 -- start start 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd096227640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd090100e20 0x7fd090197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd096227640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd090102570 0x7fd090197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:23.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd096227640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd090102ef0 0x7fd09019bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd096227640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd090110fa0 con 0x7fd090102ef0 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd096227640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd090110e20 con 0x7fd090100e20 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd096227640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd090111120 con 0x7fd090102570 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd090102ef0 0x7fd09019bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd090102ef0 0x7fd09019bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58846/0 (socket says 172.21.15.89:58846) 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08ffff640 1 -- 172.21.15.89:0/1142102429 learned_addr learned my addr 172.21.15.89:0/1142102429 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd087fff640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd090102570 0x7fd090197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08f7fe640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd090100e20 0x7fd090197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08ffff640 1 -- 172.21.15.89:0/1142102429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd090102570 msgr2=0x7fd090197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08ffff640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd090102570 0x7fd090197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08ffff640 1 -- 172.21.15.89:0/1142102429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd090100e20 msgr2=0x7fd090197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:23.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08ffff640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd090100e20 0x7fd090197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08ffff640 1 -- 172.21.15.89:0/1142102429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09019c3f0 con 0x7fd090102ef0 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08f7fe640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd090100e20 0x7fd090197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.473+0000 7fd08ffff640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd090102ef0 0x7fd09019bd10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd08802ae70 tx=0x7fd08802f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.474+0000 7fd08d7fa640 1 -- 172.21.15.89:0/1142102429 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088034040 con 0x7fd090102ef0 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.474+0000 7fd08d7fa640 1 -- 172.21.15.89:0/1142102429 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd088030920 con 0x7fd090102ef0 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.474+0000 7fd096227640 1 -- 172.21.15.89:0/1142102429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09019c680 con 0x7fd090102ef0 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.474+0000 7fd08d7fa640 1 -- 172.21.15.89:0/1142102429 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088045480 con 0x7fd090102ef0 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.474+0000 7fd096227640 1 -- 172.21.15.89:0/1142102429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0901a3fc0 con 0x7fd090102ef0 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.475+0000 7fd096227640 1 -- 172.21.15.89:0/1142102429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0901021b0 con 0x7fd090102ef0 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.476+0000 7fd08d7fa640 1 -- 172.21.15.89:0/1142102429 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd088037020 con 0x7fd090102ef0 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.476+0000 7fd08d7fa640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd07c0a3350 0x7fd07c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:23.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.476+0000 7fd08f7fe640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd07c0a3350 0x7fd07c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.476+0000 7fd08d7fa640 1 -- 172.21.15.89:0/1142102429 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fd0880f0ba0 con 0x7fd090102ef0 2024-09-15T23:23:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.477+0000 7fd08f7fe640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd07c0a3350 0x7fd07c0a5810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd080006fd0 tx=0x7fd080008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:23.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.480+0000 7fd08d7fa640 1 -- 172.21.15.89:0/1142102429 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd0880bd510 con 0x7fd090102ef0 2024-09-15T23:23:23.500 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:23.500 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:23.500 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:23.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.660+0000 7fd096227640 1 -- 172.21.15.89:0/1142102429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fd0901987d0 con 0x7fd090102ef0 2024-09-15T23:23:23.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.660+0000 7fd08d7fa640 1 -- 172.21.15.89:0/1142102429 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd0901987d0 con 0x7fd090102ef0 2024-09-15T23:23:23.661 INFO:teuthology.orchestra.run.smithi089.stdout:283467841572 2024-09-15T23:23:23.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.662+0000 7fd096227640 1 -- 172.21.15.89:0/1142102429 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd07c0a3350 msgr2=0x7fd07c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:23.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.662+0000 7fd096227640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd07c0a3350 0x7fd07c0a5810 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd080006fd0 tx=0x7fd080008040 comp rx=0 tx=0).stop 2024-09-15T23:23:23.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.662+0000 7fd096227640 1 -- 172.21.15.89:0/1142102429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd090102ef0 msgr2=0x7fd09019bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:23.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.662+0000 7fd096227640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd090102ef0 0x7fd09019bd10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd08802ae70 tx=0x7fd08802f5d0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.662+0000 7fd096227640 1 -- 172.21.15.89:0/1142102429 shutdown_connections 2024-09-15T23:23:23.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.662+0000 7fd096227640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd07c0a3350 0x7fd07c0a5810 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.662+0000 7fd096227640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd090102ef0 0x7fd09019bd10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.662+0000 7fd096227640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd090102570 0x7fd090197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.662+0000 7fd096227640 1 --2- 172.21.15.89:0/1142102429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd090100e20 0x7fd090197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.663+0000 7fd096227640 1 -- 172.21.15.89:0/1142102429 >> 172.21.15.89:0/1142102429 conn(0x7fd0900fc7d0 msgr2=0x7fd090106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:23.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.663+0000 7fd096227640 1 -- 172.21.15.89:0/1142102429 shutdown_connections 2024-09-15T23:23:23.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.663+0000 7fd096227640 1 -- 172.21.15.89:0/1142102429 wait complete. 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.677+0000 7fd50b471640 1 -- 172.21.15.89:0/4004106538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd504101420 msgr2=0x7fd50410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.677+0000 7fd50b471640 1 --2- 172.21.15.89:0/4004106538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd504101420 0x7fd50410d950 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009cd0 tx=0x7fd4fc02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.677+0000 7fd50b471640 1 -- 172.21.15.89:0/4004106538 shutdown_connections 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.677+0000 7fd50b471640 1 --2- 172.21.15.89:0/4004106538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd504101420 0x7fd50410d950 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.677+0000 7fd50b471640 1 --2- 172.21.15.89:0/4004106538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd504100a80 0x7fd504100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.677+0000 7fd50b471640 1 --2- 172.21.15.89:0/4004106538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd504106a80 0x7fd504106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.677+0000 7fd50b471640 1 -- 172.21.15.89:0/4004106538 >> 172.21.15.89:0/4004106538 conn(0x7fd5040fc7a0 msgr2=0x7fd5040febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.677+0000 7fd50b471640 1 -- 172.21.15.89:0/4004106538 shutdown_connections 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.677+0000 7fd50b471640 1 -- 172.21.15.89:0/4004106538 wait complete. 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd50b471640 1 Processor -- start 2024-09-15T23:23:23.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd50b471640 1 -- start start 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd50b471640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd504100a80 0x7fd504198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd50b471640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd504101420 0x7fd504198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd50b471640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd504106a80 0x7fd50419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd50b471640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd504110e80 con 0x7fd504101420 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd50b471640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd504110d00 con 0x7fd504100a80 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd50b471640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd504111000 con 0x7fd504106a80 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd508a0c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd504101420 0x7fd504198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd508a0c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd504101420 0x7fd504198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58870/0 (socket says 172.21.15.89:58870) 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd508a0c640 1 -- 172.21.15.89:0/3707217707 learned_addr learned my addr 172.21.15.89:0/3707217707 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.678+0000 7fd50920d640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd504100a80 0x7fd504198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd509a0e640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd504106a80 0x7fd50419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd508a0c640 1 -- 172.21.15.89:0/3707217707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd504106a80 msgr2=0x7fd50419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:23.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd508a0c640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd504106a80 0x7fd50419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd508a0c640 1 -- 172.21.15.89:0/3707217707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd504100a80 msgr2=0x7fd504198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd508a0c640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd504100a80 0x7fd504198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd508a0c640 1 -- 172.21.15.89:0/3707217707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50419d2b0 con 0x7fd504101420 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd508a0c640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd504101420 0x7fd504198840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400c920 tx=0x7fd4f400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd4fa7fc640 1 -- 172.21.15.89:0/3707217707 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4007c70 con 0x7fd504101420 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd4fa7fc640 1 -- 172.21.15.89:0/3707217707 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4f4007e10 con 0x7fd504101420 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd50b471640 1 -- 172.21.15.89:0/3707217707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd50419d5a0 con 0x7fd504101420 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd4fa7fc640 1 -- 172.21.15.89:0/3707217707 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f4018890 con 0x7fd504101420 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.679+0000 7fd50b471640 1 -- 172.21.15.89:0/3707217707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5041a4e80 con 0x7fd504101420 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.681+0000 7fd50b471640 1 -- 172.21.15.89:0/3707217707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd504100ee0 con 0x7fd504101420 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.681+0000 7fd4fa7fc640 1 -- 172.21.15.89:0/3707217707 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd4f4018a70 con 0x7fd504101420 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.682+0000 7fd4fa7fc640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4f00a3350 0x7fd4f00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.683+0000 7fd4fa7fc640 1 -- 172.21.15.89:0/3707217707 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fd4f40cffb0 con 0x7fd504101420 2024-09-15T23:23:23.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.683+0000 7fd50920d640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4f00a3350 0x7fd4f00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:23.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.683+0000 7fd50920d640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4f00a3350 0x7fd4f00a5810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd5000058c0 tx=0x7fd50000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:23.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.687+0000 7fd4fa7fc640 1 -- 172.21.15.89:0/3707217707 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd4f4014030 con 0x7fd504101420 2024-09-15T23:23:23.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.858+0000 7fd50b471640 1 -- 172.21.15.89:0/3707217707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd504109a70 con 0x7fd504101420 2024-09-15T23:23:23.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.859+0000 7fd4fa7fc640 1 -- 172.21.15.89:0/3707217707 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd4f409c7b0 con 0x7fd504101420 2024-09-15T23:23:23.860 INFO:teuthology.orchestra.run.smithi089.stdout:34359738455 2024-09-15T23:23:23.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 -- 172.21.15.89:0/3707217707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4f00a3350 msgr2=0x7fd4f00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:23.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4f00a3350 0x7fd4f00a5810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd5000058c0 tx=0x7fd50000a2d0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 -- 172.21.15.89:0/3707217707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd504101420 msgr2=0x7fd504198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:23.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd504101420 0x7fd504198840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400c920 tx=0x7fd4f400cde0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 -- 172.21.15.89:0/3707217707 shutdown_connections 2024-09-15T23:23:23.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4f00a3350 0x7fd4f00a5810 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd504106a80 0x7fd50419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd504101420 0x7fd504198840 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 --2- 172.21.15.89:0/3707217707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd504100a80 0x7fd504198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:23.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 -- 172.21.15.89:0/3707217707 >> 172.21.15.89:0/3707217707 conn(0x7fd5040fc7a0 msgr2=0x7fd50406a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:23.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 -- 172.21.15.89:0/3707217707 shutdown_connections 2024-09-15T23:23:23.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:23.862+0000 7fd50b471640 1 -- 172.21.15.89:0/3707217707 wait complete. 2024-09-15T23:23:24.215 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:24 smithi089 ceph-mon[30933]: pgmap v329: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-09-15T23:23:24.215 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:24 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1142102429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-15T23:23:24.216 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:24 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3707217707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-15T23:23:24.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.214+0000 7f8945b04640 1 -- 172.21.15.89:0/3906504385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8940103740 msgr2=0x7f8940107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.214+0000 7f8945b04640 1 --2- 172.21.15.89:0/3906504385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8940103740 0x7f8940107b80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8930009930 tx=0x7f893002a420 comp rx=0 tx=0).stop 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.214+0000 7f8945b04640 1 -- 172.21.15.89:0/3906504385 shutdown_connections 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.214+0000 7f8945b04640 1 --2- 172.21.15.89:0/3906504385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8940108290 0x7f894010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.214+0000 7f8945b04640 1 --2- 172.21.15.89:0/3906504385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8940103740 0x7f8940107b80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.214+0000 7f8945b04640 1 --2- 172.21.15.89:0/3906504385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8940102d90 0x7f8940103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.214+0000 7f8945b04640 1 -- 172.21.15.89:0/3906504385 >> 172.21.15.89:0/3906504385 conn(0x7f89400fc7d0 msgr2=0x7f89400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.214+0000 7f8945b04640 1 -- 172.21.15.89:0/3906504385 shutdown_connections 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.214+0000 7f8945b04640 1 -- 172.21.15.89:0/3906504385 wait complete. 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.215+0000 7f8945b04640 1 Processor -- start 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.215+0000 7f8945b04640 1 -- start start 2024-09-15T23:23:24.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f8945b04640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8940102d90 0x7f89401a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f8945b04640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8940103740 0x7f89401a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f8945b04640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8940108290 0x7f894019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f8945b04640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f89401120b0 con 0x7f8940102d90 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f8945b04640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8940111f30 con 0x7f8940108290 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f8945b04640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8940112230 con 0x7f8940103740 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8940102d90 0x7f89401a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8940102d90 0x7f89401a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58890/0 (socket says 172.21.15.89:58890) 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893f7fe640 1 -- 172.21.15.89:0/2566238394 learned_addr learned my addr 172.21.15.89:0/2566238394 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893effd640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8940103740 0x7f89401a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893ffff640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8940108290 0x7f894019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893f7fe640 1 -- 172.21.15.89:0/2566238394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8940103740 msgr2=0x7f89401a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893f7fe640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8940103740 0x7f89401a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:24.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893f7fe640 1 -- 172.21.15.89:0/2566238394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8940108290 msgr2=0x7f894019a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893f7fe640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8940108290 0x7f894019a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893f7fe640 1 -- 172.21.15.89:0/2566238394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894019aef0 con 0x7f8940102d90 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893ffff640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8940108290 0x7f894019a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.216+0000 7f893f7fe640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8940102d90 0x7f89401a05b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f893400d950 tx=0x7f893400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.217+0000 7f893cff9640 1 -- 172.21.15.89:0/2566238394 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934017070 con 0x7f8940102d90 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.217+0000 7f8945b04640 1 -- 172.21.15.89:0/2566238394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f894019b1e0 con 0x7f8940102d90 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.217+0000 7f893cff9640 1 -- 172.21.15.89:0/2566238394 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8934009c30 con 0x7f8940102d90 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.217+0000 7f893cff9640 1 -- 172.21.15.89:0/2566238394 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893400f6b0 con 0x7f8940102d90 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.217+0000 7f8945b04640 1 -- 172.21.15.89:0/2566238394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89401a7530 con 0x7f8940102d90 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.218+0000 7f8945b04640 1 -- 172.21.15.89:0/2566238394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f894006a7f0 con 0x7f8940102d90 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.219+0000 7f893cff9640 1 -- 172.21.15.89:0/2566238394 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f89340109d0 con 0x7f8940102d90 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.219+0000 7f893cff9640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89280a32d0 0x7f89280a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:24.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.219+0000 7f893cff9640 1 -- 172.21.15.89:0/2566238394 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f89340cfa70 con 0x7f8940102d90 2024-09-15T23:23:24.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.219+0000 7f893effd640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89280a32d0 0x7f89280a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:24.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.220+0000 7f893effd640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89280a32d0 0x7f89280a5790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8930004770 tx=0x7f89300058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:24.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.224+0000 7f893cff9640 1 -- 172.21.15.89:0/2566238394 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8934009de0 con 0x7f8940102d90 2024-09-15T23:23:24.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.395+0000 7f8945b04640 1 -- 172.21.15.89:0/2566238394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f894019b520 con 0x7f8940102d90 2024-09-15T23:23:24.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.395+0000 7f893cff9640 1 -- 172.21.15.89:0/2566238394 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f893409c2d0 con 0x7f8940102d90 2024-09-15T23:23:24.397 INFO:teuthology.orchestra.run.smithi089.stdout:73014444109 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.397+0000 7f8945b04640 1 -- 172.21.15.89:0/2566238394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89280a32d0 msgr2=0x7f89280a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.397+0000 7f8945b04640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89280a32d0 0x7f89280a5790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8930004770 tx=0x7f89300058d0 comp rx=0 tx=0).stop 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.397+0000 7f8945b04640 1 -- 172.21.15.89:0/2566238394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8940102d90 msgr2=0x7f89401a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.397+0000 7f8945b04640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8940102d90 0x7f89401a05b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f893400d950 tx=0x7f893400de10 comp rx=0 tx=0).stop 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.398+0000 7f8945b04640 1 -- 172.21.15.89:0/2566238394 shutdown_connections 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.398+0000 7f8945b04640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89280a32d0 0x7f89280a5790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.398+0000 7f8945b04640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8940108290 0x7f894019a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.398+0000 7f8945b04640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8940103740 0x7f89401a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.398+0000 7f8945b04640 1 --2- 172.21.15.89:0/2566238394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8940102d90 0x7f89401a05b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.398+0000 7f8945b04640 1 -- 172.21.15.89:0/2566238394 >> 172.21.15.89:0/2566238394 conn(0x7f89400fc7d0 msgr2=0x7f8940100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:24.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.398+0000 7f8945b04640 1 -- 172.21.15.89:0/2566238394 shutdown_connections 2024-09-15T23:23:24.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:24.398+0000 7f8945b04640 1 -- 172.21.15.89:0/2566238394 wait complete. 2024-09-15T23:23:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:24 smithi120 ceph-mon[35250]: pgmap v329: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-09-15T23:23:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:24 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1142102429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-15T23:23:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:24 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3707217707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-15T23:23:24.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:24 smithi174 ceph-mon[35586]: pgmap v329: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-09-15T23:23:24.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:24 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1142102429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-15T23:23:24.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:24 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3707217707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-15T23:23:24.935 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738455 for osd.0 2024-09-15T23:23:24.935 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:25 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2566238394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-15T23:23:25.382 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:25 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2566238394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-15T23:23:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:25 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2566238394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-15T23:23:25.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.892+0000 7f2380dba640 1 -- 172.21.15.89:0/2727024000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237c100d00 msgr2=0x7f237c106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.892+0000 7f2380dba640 1 --2- 172.21.15.89:0/2727024000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237c100d00 0x7f237c106840 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2364009930 tx=0x7f236402a420 comp rx=0 tx=0).stop 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.893+0000 7f2380dba640 1 -- 172.21.15.89:0/2727024000 shutdown_connections 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.893+0000 7f2380dba640 1 --2- 172.21.15.89:0/2727024000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237c106f50 0x7f237c10eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.893+0000 7f2380dba640 1 --2- 172.21.15.89:0/2727024000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237c100d00 0x7f237c106840 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.893+0000 7f2380dba640 1 --2- 172.21.15.89:0/2727024000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f237c100350 0x7f237c100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.893+0000 7f2380dba640 1 -- 172.21.15.89:0/2727024000 >> 172.21.15.89:0/2727024000 conn(0x7f237c077210 msgr2=0x7f237c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.893+0000 7f2380dba640 1 -- 172.21.15.89:0/2727024000 shutdown_connections 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.893+0000 7f2380dba640 1 -- 172.21.15.89:0/2727024000 wait complete. 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.893+0000 7f2380dba640 1 Processor -- start 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.893+0000 7f2380dba640 1 -- start start 2024-09-15T23:23:25.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.893+0000 7f2380dba640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237c100350 0x7f237c19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f2380dba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237c100d00 0x7f237c19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f2380dba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f237c106f50 0x7f237c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f2380dba640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f237c111fa0 con 0x7f237c100350 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f2380dba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f237c111e20 con 0x7f237c106f50 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f2380dba640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f237c112120 con 0x7f237c100d00 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f237a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237c100350 0x7f237c19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f237a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237c100350 0x7f237c19a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60504/0 (socket says 172.21.15.89:60504) 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f237a59c640 1 -- 172.21.15.89:0/1283196796 learned_addr learned my addr 172.21.15.89:0/1283196796 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f2379d9b640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237c100d00 0x7f237c19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f237a59c640 1 -- 172.21.15.89:0/1283196796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237c100d00 msgr2=0x7f237c19abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f237a59c640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237c100d00 0x7f237c19abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f237a59c640 1 -- 172.21.15.89:0/1283196796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f237c106f50 msgr2=0x7f237c19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:25.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f237a59c640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f237c106f50 0x7f237c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:25.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f237a59c640 1 -- 172.21.15.89:0/1283196796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237c19f640 con 0x7f237c100350 2024-09-15T23:23:25.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f237a59c640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237c100350 0x7f237c19a690 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f236c00c890 tx=0x7f236c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:25.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f23737fe640 1 -- 172.21.15.89:0/1283196796 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c004220 con 0x7f237c100350 2024-09-15T23:23:25.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f2380dba640 1 -- 172.21.15.89:0/1283196796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f237c19f930 con 0x7f237c100350 2024-09-15T23:23:25.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f23737fe640 1 -- 172.21.15.89:0/1283196796 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f236c0043c0 con 0x7f237c100350 2024-09-15T23:23:25.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f23737fe640 1 -- 172.21.15.89:0/1283196796 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c004a00 con 0x7f237c100350 2024-09-15T23:23:25.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.894+0000 7f2380dba640 1 -- 172.21.15.89:0/1283196796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f237c1a7210 con 0x7f237c100350 2024-09-15T23:23:25.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.895+0000 7f2380dba640 1 -- 172.21.15.89:0/1283196796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f237c100730 con 0x7f237c100350 2024-09-15T23:23:25.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.897+0000 7f23737fe640 1 -- 172.21.15.89:0/1283196796 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f236c0079b0 con 0x7f237c100350 2024-09-15T23:23:25.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.897+0000 7f23737fe640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23680a32d0 0x7f23680a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:25.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.898+0000 7f2379d9b640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23680a32d0 0x7f23680a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:25.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.898+0000 7f23737fe640 1 -- 172.21.15.89:0/1283196796 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f236c0cffb0 con 0x7f237c100350 2024-09-15T23:23:25.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.898+0000 7f2379d9b640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23680a32d0 0x7f23680a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2364006010 tx=0x7f2364031690 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:25.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:25.900+0000 7f23737fe640 1 -- 172.21.15.89:0/1283196796 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f236c014030 con 0x7f237c100350 2024-09-15T23:23:26.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.070+0000 7f2380dba640 1 -- 172.21.15.89:0/1283196796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f237c19b9f0 con 0x7f237c100350 2024-09-15T23:23:26.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.070+0000 7f23737fe640 1 -- 172.21.15.89:0/1283196796 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f236c09c810 con 0x7f237c100350 2024-09-15T23:23:26.071 INFO:teuthology.orchestra.run.smithi089.stdout:128849018947 2024-09-15T23:23:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 -- 172.21.15.89:0/1283196796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23680a32d0 msgr2=0x7f23680a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23680a32d0 0x7f23680a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2364006010 tx=0x7f2364031690 comp rx=0 tx=0).stop 2024-09-15T23:23:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 -- 172.21.15.89:0/1283196796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237c100350 msgr2=0x7f237c19a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:26.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237c100350 0x7f237c19a690 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f236c00c890 tx=0x7f236c00cd50 comp rx=0 tx=0).stop 2024-09-15T23:23:26.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 -- 172.21.15.89:0/1283196796 shutdown_connections 2024-09-15T23:23:26.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23680a32d0 0x7f23680a5790 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:26.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f237c106f50 0x7f237c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:26.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237c100d00 0x7f237c19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:26.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 --2- 172.21.15.89:0/1283196796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f237c100350 0x7f237c19a690 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:26.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 -- 172.21.15.89:0/1283196796 >> 172.21.15.89:0/1283196796 conn(0x7f237c077210 msgr2=0x7f237c105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:26.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 -- 172.21.15.89:0/1283196796 shutdown_connections 2024-09-15T23:23:26.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.072+0000 7f2380dba640 1 -- 172.21.15.89:0/1283196796 wait complete. 2024-09-15T23:23:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:26 smithi120 ceph-mon[35250]: pgmap v330: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:26 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1283196796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-15T23:23:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:26 smithi174 ceph-mon[35586]: pgmap v330: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:26 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1283196796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-15T23:23:26.523 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:26 smithi089 ceph-mon[30933]: pgmap v330: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:26.523 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:26 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1283196796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-15T23:23:26.826 INFO:tasks.cephadm.ceph_manager.ceph:need seq 73014444107 got 73014444109 for osd.2 2024-09-15T23:23:26.826 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.987+0000 7fc99de00640 1 -- 172.21.15.89:0/982667621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc998101420 msgr2=0x7fc99810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.987+0000 7fc99de00640 1 --2- 172.21.15.89:0/982667621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc998101420 0x7fc99810d950 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc990009cd0 tx=0x7fc99002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:23:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.987+0000 7fc99de00640 1 -- 172.21.15.89:0/982667621 shutdown_connections 2024-09-15T23:23:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.987+0000 7fc99de00640 1 --2- 172.21.15.89:0/982667621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc998101420 0x7fc99810d950 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.987+0000 7fc99de00640 1 --2- 172.21.15.89:0/982667621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc998100a80 0x7fc998100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.987+0000 7fc99de00640 1 --2- 172.21.15.89:0/982667621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc998106a80 0x7fc998106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.987+0000 7fc99de00640 1 -- 172.21.15.89:0/982667621 >> 172.21.15.89:0/982667621 conn(0x7fc9980fc7a0 msgr2=0x7fc9980febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.987+0000 7fc99de00640 1 -- 172.21.15.89:0/982667621 shutdown_connections 2024-09-15T23:23:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.987+0000 7fc99de00640 1 -- 172.21.15.89:0/982667621 wait complete. 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc99de00640 1 Processor -- start 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc99de00640 1 -- start start 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc99de00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc998100a80 0x7fc9981982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc99de00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc998101420 0x7fc998198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc99de00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc998106a80 0x7fc99819cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc99de00640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc998110e80 con 0x7fc998106a80 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc99de00640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc998110d00 con 0x7fc998101420 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc99de00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc998111000 con 0x7fc998100a80 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc997fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc998106a80 0x7fc99819cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc997fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc998106a80 0x7fc99819cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60516/0 (socket says 172.21.15.89:60516) 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc997fff640 1 -- 172.21.15.89:0/873781364 learned_addr learned my addr 172.21.15.89:0/873781364 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc98ffff640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc998101420 0x7fc998198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc9977fe640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc998100a80 0x7fc9981982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc997fff640 1 -- 172.21.15.89:0/873781364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc998100a80 msgr2=0x7fc9981982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc997fff640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc998100a80 0x7fc9981982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc997fff640 1 -- 172.21.15.89:0/873781364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc998101420 msgr2=0x7fc998198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.988+0000 7fc997fff640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc998101420 0x7fc998198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.989+0000 7fc997fff640 1 -- 172.21.15.89:0/873781364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99819d280 con 0x7fc998106a80 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.989+0000 7fc9977fe640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc998100a80 0x7fc9981982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.989+0000 7fc997fff640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc998106a80 0x7fc99819cba0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc990006010 tx=0x7fc990003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.989+0000 7fc9957fa640 1 -- 172.21.15.89:0/873781364 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990004450 con 0x7fc998106a80 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.989+0000 7fc99de00640 1 -- 172.21.15.89:0/873781364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99819d510 con 0x7fc998106a80 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.989+0000 7fc9957fa640 1 -- 172.21.15.89:0/873781364 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc990003e10 con 0x7fc998106a80 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.989+0000 7fc9957fa640 1 -- 172.21.15.89:0/873781364 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990046480 con 0x7fc998106a80 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.989+0000 7fc99de00640 1 -- 172.21.15.89:0/873781364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9981a4e50 con 0x7fc998106a80 2024-09-15T23:23:26.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.990+0000 7fc99de00640 1 -- 172.21.15.89:0/873781364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc998100ee0 con 0x7fc998106a80 2024-09-15T23:23:26.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.991+0000 7fc9957fa640 1 -- 172.21.15.89:0/873781364 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc9900045f0 con 0x7fc998106a80 2024-09-15T23:23:26.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.992+0000 7fc9957fa640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9840a32d0 0x7fc9840a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:26.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.992+0000 7fc9977fe640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9840a32d0 0x7fc9840a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:26.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.992+0000 7fc9957fa640 1 -- 172.21.15.89:0/873781364 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fc99002e030 con 0x7fc998106a80 2024-09-15T23:23:26.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.992+0000 7fc9977fe640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9840a32d0 0x7fc9840a5790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc988005e10 tx=0x7fc988005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:26.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:26.995+0000 7fc9957fa640 1 -- 172.21.15.89:0/873781364 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc99004b060 con 0x7fc998106a80 2024-09-15T23:23:27.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.168+0000 7fc99de00640 1 -- 172.21.15.89:0/873781364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fc9981a50b0 con 0x7fc998106a80 2024-09-15T23:23:27.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.169+0000 7fc9957fa640 1 -- 172.21.15.89:0/873781364 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc9900be160 con 0x7fc998106a80 2024-09-15T23:23:27.170 INFO:teuthology.orchestra.run.smithi089.stdout:201863462965 2024-09-15T23:23:27.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 -- 172.21.15.89:0/873781364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9840a32d0 msgr2=0x7fc9840a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:27.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9840a32d0 0x7fc9840a5790 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc988005e10 tx=0x7fc988005da0 comp rx=0 tx=0).stop 2024-09-15T23:23:27.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 -- 172.21.15.89:0/873781364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc998106a80 msgr2=0x7fc99819cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:27.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc998106a80 0x7fc99819cba0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc990006010 tx=0x7fc990003c20 comp rx=0 tx=0).stop 2024-09-15T23:23:27.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 -- 172.21.15.89:0/873781364 shutdown_connections 2024-09-15T23:23:27.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9840a32d0 0x7fc9840a5790 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:27.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc998106a80 0x7fc99819cba0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:27.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc998101420 0x7fc998198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:27.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 --2- 172.21.15.89:0/873781364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc998100a80 0x7fc9981982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:27.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 -- 172.21.15.89:0/873781364 >> 172.21.15.89:0/873781364 conn(0x7fc9980fc7a0 msgr2=0x7fc998104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:27.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 -- 172.21.15.89:0/873781364 shutdown_connections 2024-09-15T23:23:27.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:27.171+0000 7fc99de00640 1 -- 172.21.15.89:0/873781364 wait complete. 2024-09-15T23:23:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:27 smithi120 ceph-mon[35250]: pgmap v331: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:27.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:27 smithi174 ceph-mon[35586]: pgmap v331: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:27.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:27 smithi089 ceph-mon[30933]: pgmap v331: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:27.617 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841568 got 283467841572 for osd.10 2024-09-15T23:23:27.618 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:28.045 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018945 got 128849018947 for osd.4 2024-09-15T23:23:28.046 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:28.306 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:28 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/873781364' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-15T23:23:28.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:28 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/873781364' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-15T23:23:28.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:28 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/873781364' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-15T23:23:29.095 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462961 got 201863462965 for osd.7 2024-09-15T23:23:29.096 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:29.352 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:29 smithi089 ceph-mon[30933]: pgmap v332: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:29.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.349+0000 7efc353e2640 1 -- 172.21.15.89:0/1853258734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc30100a90 msgr2=0x7efc30100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.349+0000 7efc353e2640 1 --2- 172.21.15.89:0/1853258734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc30100a90 0x7efc30100ef0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7efc24009930 tx=0x7efc2402a420 comp rx=0 tx=0).stop 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.349+0000 7efc353e2640 1 -- 172.21.15.89:0/1853258734 shutdown_connections 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.349+0000 7efc353e2640 1 --2- 172.21.15.89:0/1853258734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc30101430 0x7efc3010d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.349+0000 7efc353e2640 1 --2- 172.21.15.89:0/1853258734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc30100a90 0x7efc30100ef0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.349+0000 7efc353e2640 1 --2- 172.21.15.89:0/1853258734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc30106a90 0x7efc30106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.349+0000 7efc353e2640 1 -- 172.21.15.89:0/1853258734 >> 172.21.15.89:0/1853258734 conn(0x7efc300fc7d0 msgr2=0x7efc300febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.349+0000 7efc353e2640 1 -- 172.21.15.89:0/1853258734 shutdown_connections 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.349+0000 7efc353e2640 1 -- 172.21.15.89:0/1853258734 wait complete. 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.350+0000 7efc353e2640 1 Processor -- start 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.350+0000 7efc353e2640 1 -- start start 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.350+0000 7efc353e2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc30100a90 0x7efc3019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:29.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.350+0000 7efc353e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc30101430 0x7efc3019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.350+0000 7efc353e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc30106a90 0x7efc3019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc353e2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efc30110e90 con 0x7efc30106a90 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc353e2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efc30110d10 con 0x7efc30100a90 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc353e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efc30111010 con 0x7efc30101430 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc30106a90 0x7efc3019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc30106a90 0x7efc3019eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60538/0 (socket says 172.21.15.89:60538) 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2f7fe640 1 -- 172.21.15.89:0/1032401796 learned_addr learned my addr 172.21.15.89:0/1032401796 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2e7fc640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc30101430 0x7efc3019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2effd640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc30100a90 0x7efc3019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2f7fe640 1 -- 172.21.15.89:0/1032401796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc30101430 msgr2=0x7efc3019ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2f7fe640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc30101430 0x7efc3019ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:29.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2f7fe640 1 -- 172.21.15.89:0/1032401796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc30100a90 msgr2=0x7efc3019a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2f7fe640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc30100a90 0x7efc3019a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2f7fe640 1 -- 172.21.15.89:0/1032401796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3019f590 con 0x7efc30106a90 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc2f7fe640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc30106a90 0x7efc3019eeb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efc2800cce0 tx=0x7efc2800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc17fff640 1 -- 172.21.15.89:0/1032401796 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2800f450 con 0x7efc30106a90 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc353e2640 1 -- 172.21.15.89:0/1032401796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc3019f880 con 0x7efc30106a90 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc17fff640 1 -- 172.21.15.89:0/1032401796 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc2800fad0 con 0x7efc30106a90 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.351+0000 7efc17fff640 1 -- 172.21.15.89:0/1032401796 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc28018c10 con 0x7efc30106a90 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.352+0000 7efc353e2640 1 -- 172.21.15.89:0/1032401796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc301a7160 con 0x7efc30106a90 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.352+0000 7efc353e2640 1 -- 172.21.15.89:0/1032401796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc30100ef0 con 0x7efc30106a90 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.354+0000 7efc17fff640 1 -- 172.21.15.89:0/1032401796 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7efc2800f5f0 con 0x7efc30106a90 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.354+0000 7efc17fff640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc180a32d0 0x7efc180a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.354+0000 7efc2effd640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc180a32d0 0x7efc180a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:29.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.354+0000 7efc17fff640 1 -- 172.21.15.89:0/1032401796 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7efc280d0600 con 0x7efc30106a90 2024-09-15T23:23:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.355+0000 7efc2effd640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc180a32d0 0x7efc180a5790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efc20005fd0 tx=0x7efc2000b510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:29.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.358+0000 7efc17fff640 1 -- 172.21.15.89:0/1032401796 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efc28014030 con 0x7efc30106a90 2024-09-15T23:23:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:29 smithi120 ceph-mon[35250]: pgmap v332: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:29 smithi174 ceph-mon[35586]: pgmap v332: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:29.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.532+0000 7efc353e2640 1 -- 172.21.15.89:0/1032401796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7efc301a7400 con 0x7efc30106a90 2024-09-15T23:23:29.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.532+0000 7efc17fff640 1 -- 172.21.15.89:0/1032401796 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7efc2809cef0 con 0x7efc30106a90 2024-09-15T23:23:29.534 INFO:teuthology.orchestra.run.smithi089.stdout:317827579937 2024-09-15T23:23:29.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.535+0000 7efc353e2640 1 -- 172.21.15.89:0/1032401796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc180a32d0 msgr2=0x7efc180a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:29.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.535+0000 7efc353e2640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc180a32d0 0x7efc180a5790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efc20005fd0 tx=0x7efc2000b510 comp rx=0 tx=0).stop 2024-09-15T23:23:29.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.535+0000 7efc353e2640 1 -- 172.21.15.89:0/1032401796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc30106a90 msgr2=0x7efc3019eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:29.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.535+0000 7efc353e2640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc30106a90 0x7efc3019eeb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efc2800cce0 tx=0x7efc2800a620 comp rx=0 tx=0).stop 2024-09-15T23:23:29.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.535+0000 7efc353e2640 1 -- 172.21.15.89:0/1032401796 shutdown_connections 2024-09-15T23:23:29.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.535+0000 7efc353e2640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efc180a32d0 0x7efc180a5790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:29.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.535+0000 7efc353e2640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc30106a90 0x7efc3019eeb0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:29.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.535+0000 7efc353e2640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc30101430 0x7efc3019ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:29.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.535+0000 7efc353e2640 1 --2- 172.21.15.89:0/1032401796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc30100a90 0x7efc3019a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:29.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.535+0000 7efc353e2640 1 -- 172.21.15.89:0/1032401796 >> 172.21.15.89:0/1032401796 conn(0x7efc300fc7d0 msgr2=0x7efc3010c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:29.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.536+0000 7efc353e2640 1 -- 172.21.15.89:0/1032401796 shutdown_connections 2024-09-15T23:23:29.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:29.536+0000 7efc353e2640 1 -- 172.21.15.89:0/1032401796 wait complete. 2024-09-15T23:23:30.190 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:30 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1032401796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-15T23:23:30.218 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579933 got 317827579937 for osd.11 2024-09-15T23:23:30.218 DEBUG:teuthology.parallel:result is None 2024-09-15T23:23:30.218 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-15T23:23:30.219 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:23:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:30 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1032401796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-15T23:23:30.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:30 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1032401796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-15T23:23:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/623225824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8db8102df0 msgr2=0x7f8db81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 --2- 172.21.15.89:0/623225824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8db8102df0 0x7f8db81031d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8db0009900 tx=0x7f8db002a480 comp rx=0 tx=0).stop 2024-09-15T23:23:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/623225824 shutdown_connections 2024-09-15T23:23:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 --2- 172.21.15.89:0/623225824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8db810c190 0x7f8db810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 --2- 172.21.15.89:0/623225824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8103710 0x7f8db810bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 --2- 172.21.15.89:0/623225824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8db8102df0 0x7f8db81031d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/623225824 >> 172.21.15.89:0/623225824 conn(0x7f8db80fc7d0 msgr2=0x7f8db80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/623225824 shutdown_connections 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/623225824 wait complete. 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 Processor -- start 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.024+0000 7f8dbfe8a640 1 -- start start 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbfe8a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8db8102df0 0x7f8db819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbfe8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8103710 0x7f8db819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbfe8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8db810c190 0x7f8db819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbfe8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8db8111f80 con 0x7f8db810c190 2024-09-15T23:23:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbfe8a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8db8111e00 con 0x7f8db8103710 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbfe8a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8db8112100 con 0x7f8db8102df0 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbe427640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8db810c190 0x7f8db819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbe427640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8db810c190 0x7f8db819eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60552/0 (socket says 172.21.15.89:60552) 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbe427640 1 -- 172.21.15.89:0/3695638729 learned_addr learned my addr 172.21.15.89:0/3695638729 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbd425640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8103710 0x7f8db819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbdc26640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8db8102df0 0x7f8db819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbe427640 1 -- 172.21.15.89:0/3695638729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8db8102df0 msgr2=0x7f8db819a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbe427640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8db8102df0 0x7f8db819a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbe427640 1 -- 172.21.15.89:0/3695638729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8103710 msgr2=0x7f8db819ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbe427640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8103710 0x7f8db819ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbe427640 1 -- 172.21.15.89:0/3695638729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db819f590 con 0x7f8db810c190 2024-09-15T23:23:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbd425640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8103710 0x7f8db819ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbdc26640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8db8102df0 0x7f8db819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbe427640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8db810c190 0x7f8db819eeb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8da800eaa0 tx=0x7f8da800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8db6ffd640 1 -- 172.21.15.89:0/3695638729 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da800cdb0 con 0x7f8db810c190 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8db6ffd640 1 -- 172.21.15.89:0/3695638729 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8da8005020 con 0x7f8db810c190 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/3695638729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db819f880 con 0x7f8db810c190 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8db6ffd640 1 -- 172.21.15.89:0/3695638729 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da80116b0 con 0x7f8db810c190 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.025+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/3695638729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db81a7160 con 0x7f8db810c190 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.027+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/3695638729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db81031d0 con 0x7f8db810c190 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.027+0000 7f8db6ffd640 1 -- 172.21.15.89:0/3695638729 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f8da8005560 con 0x7f8db810c190 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.028+0000 7f8db6ffd640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da00a32d0 0x7f8da00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.028+0000 7f8dbdc26640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da00a32d0 0x7f8da00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.028+0000 7f8db6ffd640 1 -- 172.21.15.89:0/3695638729 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f8da80d0090 con 0x7f8db810c190 2024-09-15T23:23:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.029+0000 7f8dbdc26640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da00a32d0 0x7f8da00a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8db0005ab0 tx=0x7f8db0005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.032+0000 7f8db6ffd640 1 -- 172.21.15.89:0/3695638729 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8da8016030 con 0x7f8db810c190 2024-09-15T23:23:31.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.199+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/3695638729 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8db81021b0 con 0x7f8da00a32d0 2024-09-15T23:23:31.200 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:31 smithi089 ceph-mon[30933]: pgmap v333: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:23:31.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.207+0000 7f8db6ffd640 1 -- 172.21.15.89:0/3695638729 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340640 (secure 0 0 0) 0x7f8db81021b0 con 0x7f8da00a32d0 2024-09-15T23:23:31.212 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:23:31.215 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-15T23:23:31.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/3695638729 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da00a32d0 msgr2=0x7f8da00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da00a32d0 0x7f8da00a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8db0005ab0 tx=0x7f8db0005a20 comp rx=0 tx=0).stop 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/3695638729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8db810c190 msgr2=0x7f8db819eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8db810c190 0x7f8db819eeb0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8da800eaa0 tx=0x7f8da800ef60 comp rx=0 tx=0).stop 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/3695638729 shutdown_connections 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8da00a32d0 0x7f8da00a5790 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8db810c190 0x7f8db819eeb0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8db8103710 0x7f8db819ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 --2- 172.21.15.89:0/3695638729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8db8102df0 0x7f8db819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/3695638729 >> 172.21.15.89:0/3695638729 conn(0x7f8db80fc7d0 msgr2=0x7f8db810d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/3695638729 shutdown_connections 2024-09-15T23:23:31.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:31.210+0000 7f8dbfe8a640 1 -- 172.21.15.89:0/3695638729 wait complete. 2024-09-15T23:23:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:31 smithi120 ceph-mon[35250]: pgmap v333: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:23:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:31 smithi174 ceph-mon[35586]: pgmap v333: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:23:32.071 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-15T23:23:31.004639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659493,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":669,"num_read_kb":453,"num_write":430,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361332,"kb_used_data":12008,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580492,"statfs":{"total":1151940427776,"available":1151570423808,"internally_reserved":0,"allocated":12296192,"data_stored":7961513,"data_compressed":390162,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003641"},"pg_stats":[{"pgid":"2.1f","version":"86'13","reported_seq":98,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:26.214921+0000","last_change":"2024-09-15T23:21:50.513610+0000","last_active":"2024-09-15T23:23:26.214921+0000","last_peered":"2024-09-15T23:23:26.214921+0000","last_clean":"2024-09-15T23:23:26.214921+0000","last_became_active":"2024-09-15T23:21:50.513266+0000","last_became_peered":"2024-09-15T23:21:50.513266+0000","last_unstale":"2024-09-15T23:23:26.214921+0000","last_undegraded":"2024-09-15T23:23:26.214921+0000","last_fullsized":"2024-09-15T23:23:26.214921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:16:48.692801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381072+0000","last_change":"2024-09-15T23:21:52.540248+0000","last_active":"2024-09-15T23:23:02.381072+0000","last_peered":"2024-09-15T23:23:02.381072+0000","last_clean":"2024-09-15T23:23:02.381072+0000","last_became_active":"2024-09-15T23:21:52.539945+0000","last_became_peered":"2024-09-15T23:21:52.539945+0000","last_unstale":"2024-09-15T23:23:02.381072+0000","last_undegraded":"2024-09-15T23:23:02.381072+0000","last_fullsized":"2024-09-15T23:23:02.381072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:33.402153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144420+0000","last_change":"2024-09-15T23:21:59.567246+0000","last_active":"2024-09-15T23:23:02.144420+0000","last_peered":"2024-09-15T23:23:02.144420+0000","last_clean":"2024-09-15T23:23:02.144420+0000","last_became_active":"2024-09-15T23:21:59.566832+0000","last_became_peered":"2024-09-15T23:21:59.566832+0000","last_unstale":"2024-09-15T23:23:02.144420+0000","last_undegraded":"2024-09-15T23:23:02.144420+0000","last_fullsized":"2024-09-15T23:23:02.144420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:19.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381315+0000","last_change":"2024-09-15T23:21:50.561767+0000","last_active":"2024-09-15T23:23:02.381315+0000","last_peered":"2024-09-15T23:23:02.381315+0000","last_clean":"2024-09-15T23:23:02.381315+0000","last_became_active":"2024-09-15T23:21:50.561385+0000","last_became_peered":"2024-09-15T23:21:50.561385+0000","last_unstale":"2024-09-15T23:23:02.381315+0000","last_undegraded":"2024-09-15T23:23:02.381315+0000","last_fullsized":"2024-09-15T23:23:02.381315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:11.897884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381058+0000","last_change":"2024-09-15T23:21:52.534874+0000","last_active":"2024-09-15T23:23:02.381058+0000","last_peered":"2024-09-15T23:23:02.381058+0000","last_clean":"2024-09-15T23:23:02.381058+0000","last_became_active":"2024-09-15T23:21:52.534334+0000","last_became_peered":"2024-09-15T23:21:52.534334+0000","last_unstale":"2024-09-15T23:23:02.381058+0000","last_undegraded":"2024-09-15T23:23:02.381058+0000","last_fullsized":"2024-09-15T23:23:02.381058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:15.972648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.178766+0000","last_change":"2024-09-15T23:21:59.559907+0000","last_active":"2024-09-15T23:23:02.178766+0000","last_peered":"2024-09-15T23:23:02.178766+0000","last_clean":"2024-09-15T23:23:02.178766+0000","last_became_active":"2024-09-15T23:21:59.559440+0000","last_became_peered":"2024-09-15T23:21:59.559440+0000","last_unstale":"2024-09-15T23:23:02.178766+0000","last_undegraded":"2024-09-15T23:23:02.178766+0000","last_fullsized":"2024-09-15T23:23:02.178766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:00.755112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275999+0000","last_change":"2024-09-15T23:21:50.515424+0000","last_active":"2024-09-15T23:23:02.275999+0000","last_peered":"2024-09-15T23:23:02.275999+0000","last_clean":"2024-09-15T23:23:02.275999+0000","last_became_active":"2024-09-15T23:21:50.514966+0000","last_became_peered":"2024-09-15T23:21:50.514966+0000","last_unstale":"2024-09-15T23:23:02.275999+0000","last_undegraded":"2024-09-15T23:23:02.275999+0000","last_fullsized":"2024-09-15T23:23:02.275999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:57.586575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148438+0000","last_change":"2024-09-15T23:21:52.535189+0000","last_active":"2024-09-15T23:23:02.148438+0000","last_peered":"2024-09-15T23:23:02.148438+0000","last_clean":"2024-09-15T23:23:02.148438+0000","last_became_active":"2024-09-15T23:21:52.534807+0000","last_became_peered":"2024-09-15T23:21:52.534807+0000","last_unstale":"2024-09-15T23:23:02.148438+0000","last_undegraded":"2024-09-15T23:23:02.148438+0000","last_fullsized":"2024-09-15T23:23:02.148438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:30.654382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179082+0000","last_change":"2024-09-15T23:21:59.564675+0000","last_active":"2024-09-15T23:23:02.179082+0000","last_peered":"2024-09-15T23:23:02.179082+0000","last_clean":"2024-09-15T23:23:02.179082+0000","last_became_active":"2024-09-15T23:21:59.564271+0000","last_became_peered":"2024-09-15T23:21:59.564271+0000","last_unstale":"2024-09-15T23:23:02.179082+0000","last_undegraded":"2024-09-15T23:23:02.179082+0000","last_fullsized":"2024-09-15T23:23:02.179082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:01.055369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.276101+0000","last_change":"2024-09-15T23:21:50.507790+0000","last_active":"2024-09-15T23:23:02.276101+0000","last_peered":"2024-09-15T23:23:02.276101+0000","last_clean":"2024-09-15T23:23:02.276101+0000","last_became_active":"2024-09-15T23:21:50.507398+0000","last_became_peered":"2024-09-15T23:21:50.507398+0000","last_unstale":"2024-09-15T23:23:02.276101+0000","last_undegraded":"2024-09-15T23:23:02.276101+0000","last_fullsized":"2024-09-15T23:23:02.276101+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:58:00.001279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148799+0000","last_change":"2024-09-15T23:21:52.537179+0000","last_active":"2024-09-15T23:23:02.148799+0000","last_peered":"2024-09-15T23:23:02.148799+0000","last_clean":"2024-09-15T23:23:02.148799+0000","last_became_active":"2024-09-15T23:21:52.536841+0000","last_became_peered":"2024-09-15T23:21:52.536841+0000","last_unstale":"2024-09-15T23:23:02.148799+0000","last_undegraded":"2024-09-15T23:23:02.148799+0000","last_fullsized":"2024-09-15T23:23:02.148799+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:47.455700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.178914+0000","last_change":"2024-09-15T23:21:59.564734+0000","last_active":"2024-09-15T23:23:02.178914+0000","last_peered":"2024-09-15T23:23:02.178914+0000","last_clean":"2024-09-15T23:23:02.178914+0000","last_became_active":"2024-09-15T23:21:59.564339+0000","last_became_peered":"2024-09-15T23:21:59.564339+0000","last_unstale":"2024-09-15T23:23:02.178914+0000","last_undegraded":"2024-09-15T23:23:02.178914+0000","last_fullsized":"2024-09-15T23:23:02.178914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:44.498480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144977+0000","last_change":"2024-09-15T23:21:52.538025+0000","last_active":"2024-09-15T23:23:02.144977+0000","last_peered":"2024-09-15T23:23:02.144977+0000","last_clean":"2024-09-15T23:23:02.144977+0000","last_became_active":"2024-09-15T23:21:52.537736+0000","last_became_peered":"2024-09-15T23:21:52.537736+0000","last_unstale":"2024-09-15T23:23:02.144977+0000","last_undegraded":"2024-09-15T23:23:02.144977+0000","last_fullsized":"2024-09-15T23:23:02.144977+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:31.916888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275698+0000","last_change":"2024-09-15T23:21:50.516024+0000","last_active":"2024-09-15T23:23:02.275698+0000","last_peered":"2024-09-15T23:23:02.275698+0000","last_clean":"2024-09-15T23:23:02.275698+0000","last_became_active":"2024-09-15T23:21:50.515590+0000","last_became_peered":"2024-09-15T23:21:50.515590+0000","last_unstale":"2024-09-15T23:23:02.275698+0000","last_undegraded":"2024-09-15T23:23:02.275698+0000","last_fullsized":"2024-09-15T23:23:02.275698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:47:38.849366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179081+0000","last_change":"2024-09-15T23:21:59.681971+0000","last_active":"2024-09-15T23:23:02.179081+0000","last_peered":"2024-09-15T23:23:02.179081+0000","last_clean":"2024-09-15T23:23:02.179081+0000","last_became_active":"2024-09-15T23:21:59.681432+0000","last_became_peered":"2024-09-15T23:21:59.681432+0000","last_unstale":"2024-09-15T23:23:02.179081+0000","last_undegraded":"2024-09-15T23:23:02.179081+0000","last_fullsized":"2024-09-15T23:23:02.179081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:26.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381423+0000","last_change":"2024-09-15T23:21:52.541281+0000","last_active":"2024-09-15T23:23:02.381423+0000","last_peered":"2024-09-15T23:23:02.381423+0000","last_clean":"2024-09-15T23:23:02.381423+0000","last_became_active":"2024-09-15T23:21:52.540772+0000","last_became_peered":"2024-09-15T23:21:52.540772+0000","last_unstale":"2024-09-15T23:23:02.381423+0000","last_undegraded":"2024-09-15T23:23:02.381423+0000","last_fullsized":"2024-09-15T23:23:02.381423+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:23:44.444904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144530+0000","last_change":"2024-09-15T23:21:50.508844+0000","last_active":"2024-09-15T23:23:02.144530+0000","last_peered":"2024-09-15T23:23:02.144530+0000","last_clean":"2024-09-15T23:23:02.144530+0000","last_became_active":"2024-09-15T23:21:50.508364+0000","last_became_peered":"2024-09-15T23:21:50.508364+0000","last_unstale":"2024-09-15T23:23:02.144530+0000","last_undegraded":"2024-09-15T23:23:02.144530+0000","last_fullsized":"2024-09-15T23:23:02.144530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:45.129711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179363+0000","last_change":"2024-09-15T23:21:59.562114+0000","last_active":"2024-09-15T23:23:02.179363+0000","last_peered":"2024-09-15T23:23:02.179363+0000","last_clean":"2024-09-15T23:23:02.179363+0000","last_became_active":"2024-09-15T23:21:59.561652+0000","last_became_peered":"2024-09-15T23:21:59.561652+0000","last_unstale":"2024-09-15T23:23:02.179363+0000","last_undegraded":"2024-09-15T23:23:02.179363+0000","last_fullsized":"2024-09-15T23:23:02.179363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:51.781886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146548+0000","last_change":"2024-09-15T23:21:52.531678+0000","last_active":"2024-09-15T23:23:02.146548+0000","last_peered":"2024-09-15T23:23:02.146548+0000","last_clean":"2024-09-15T23:23:02.146548+0000","last_became_active":"2024-09-15T23:21:52.531327+0000","last_became_peered":"2024-09-15T23:21:52.531327+0000","last_unstale":"2024-09-15T23:23:02.146548+0000","last_undegraded":"2024-09-15T23:23:02.146548+0000","last_fullsized":"2024-09-15T23:23:02.146548+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:07.152626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381782+0000","last_change":"2024-09-15T23:21:50.513197+0000","last_active":"2024-09-15T23:23:02.381782+0000","last_peered":"2024-09-15T23:23:02.381782+0000","last_clean":"2024-09-15T23:23:02.381782+0000","last_became_active":"2024-09-15T23:21:50.512794+0000","last_became_peered":"2024-09-15T23:21:50.512794+0000","last_unstale":"2024-09-15T23:23:02.381782+0000","last_undegraded":"2024-09-15T23:23:02.381782+0000","last_fullsized":"2024-09-15T23:23:02.381782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:05.457467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143792+0000","last_change":"2024-09-15T23:21:59.565669+0000","last_active":"2024-09-15T23:23:02.143792+0000","last_peered":"2024-09-15T23:23:02.143792+0000","last_clean":"2024-09-15T23:23:02.143792+0000","last_became_active":"2024-09-15T23:21:59.565290+0000","last_became_peered":"2024-09-15T23:21:59.565290+0000","last_unstale":"2024-09-15T23:23:02.143792+0000","last_undegraded":"2024-09-15T23:23:02.143792+0000","last_fullsized":"2024-09-15T23:23:02.143792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:04.861560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275448+0000","last_change":"2024-09-15T23:21:52.529643+0000","last_active":"2024-09-15T23:23:02.275448+0000","last_peered":"2024-09-15T23:23:02.275448+0000","last_clean":"2024-09-15T23:23:02.275448+0000","last_became_active":"2024-09-15T23:21:52.529304+0000","last_became_peered":"2024-09-15T23:21:52.529304+0000","last_unstale":"2024-09-15T23:23:02.275448+0000","last_undegraded":"2024-09-15T23:23:02.275448+0000","last_fullsized":"2024-09-15T23:23:02.275448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:18:37.758403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148047+0000","last_change":"2024-09-15T23:21:50.562114+0000","last_active":"2024-09-15T23:23:02.148047+0000","last_peered":"2024-09-15T23:23:02.148047+0000","last_clean":"2024-09-15T23:23:02.148047+0000","last_became_active":"2024-09-15T23:21:50.561772+0000","last_became_peered":"2024-09-15T23:21:50.561772+0000","last_unstale":"2024-09-15T23:23:02.148047+0000","last_undegraded":"2024-09-15T23:23:02.148047+0000","last_fullsized":"2024-09-15T23:23:02.148047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:14.806646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179172+0000","last_change":"2024-09-15T23:21:59.681933+0000","last_active":"2024-09-15T23:23:02.179172+0000","last_peered":"2024-09-15T23:23:02.179172+0000","last_clean":"2024-09-15T23:23:02.179172+0000","last_became_active":"2024-09-15T23:21:59.681528+0000","last_became_peered":"2024-09-15T23:21:59.681528+0000","last_unstale":"2024-09-15T23:23:02.179172+0000","last_undegraded":"2024-09-15T23:23:02.179172+0000","last_fullsized":"2024-09-15T23:23:02.179172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:46.199540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381069+0000","last_change":"2024-09-15T23:21:52.529548+0000","last_active":"2024-09-15T23:23:02.381069+0000","last_peered":"2024-09-15T23:23:02.381069+0000","last_clean":"2024-09-15T23:23:02.381069+0000","last_became_active":"2024-09-15T23:21:52.529253+0000","last_became_peered":"2024-09-15T23:21:52.529253+0000","last_unstale":"2024-09-15T23:23:02.381069+0000","last_undegraded":"2024-09-15T23:23:02.381069+0000","last_fullsized":"2024-09-15T23:23:02.381069+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:22.813395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179650+0000","last_change":"2024-09-15T23:21:50.767759+0000","last_active":"2024-09-15T23:23:02.179650+0000","last_peered":"2024-09-15T23:23:02.179650+0000","last_clean":"2024-09-15T23:23:02.179650+0000","last_became_active":"2024-09-15T23:21:50.767388+0000","last_became_peered":"2024-09-15T23:21:50.767388+0000","last_unstale":"2024-09-15T23:23:02.179650+0000","last_undegraded":"2024-09-15T23:23:02.179650+0000","last_fullsized":"2024-09-15T23:23:02.179650+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:36.713769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144546+0000","last_change":"2024-09-15T23:21:59.557951+0000","last_active":"2024-09-15T23:23:02.144546+0000","last_peered":"2024-09-15T23:23:02.144546+0000","last_clean":"2024-09-15T23:23:02.144546+0000","last_became_active":"2024-09-15T23:21:59.557490+0000","last_became_peered":"2024-09-15T23:21:59.557490+0000","last_unstale":"2024-09-15T23:23:02.144546+0000","last_undegraded":"2024-09-15T23:23:02.144546+0000","last_fullsized":"2024-09-15T23:23:02.144546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:17.147389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381781+0000","last_change":"2024-09-15T23:21:52.531638+0000","last_active":"2024-09-15T23:23:02.381781+0000","last_peered":"2024-09-15T23:23:02.381781+0000","last_clean":"2024-09-15T23:23:02.381781+0000","last_became_active":"2024-09-15T23:21:52.531301+0000","last_became_peered":"2024-09-15T23:21:52.531301+0000","last_unstale":"2024-09-15T23:23:02.381781+0000","last_undegraded":"2024-09-15T23:23:02.381781+0000","last_fullsized":"2024-09-15T23:23:02.381781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.436286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'15","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143896+0000","last_change":"2024-09-15T23:21:50.508470+0000","last_active":"2024-09-15T23:23:02.143896+0000","last_peered":"2024-09-15T23:23:02.143896+0000","last_clean":"2024-09-15T23:23:02.143896+0000","last_became_active":"2024-09-15T23:21:50.508060+0000","last_became_peered":"2024-09-15T23:21:50.508060+0000","last_unstale":"2024-09-15T23:23:02.143896+0000","last_undegraded":"2024-09-15T23:23:02.143896+0000","last_fullsized":"2024-09-15T23:23:02.143896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:59.542252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381349+0000","last_change":"2024-09-15T23:21:59.682019+0000","last_active":"2024-09-15T23:23:02.381349+0000","last_peered":"2024-09-15T23:23:02.381349+0000","last_clean":"2024-09-15T23:23:02.381349+0000","last_became_active":"2024-09-15T23:21:59.681467+0000","last_became_peered":"2024-09-15T23:21:59.681467+0000","last_unstale":"2024-09-15T23:23:02.381349+0000","last_undegraded":"2024-09-15T23:23:02.381349+0000","last_fullsized":"2024-09-15T23:23:02.381349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:27.650382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148067+0000","last_change":"2024-09-15T23:21:52.536345+0000","last_active":"2024-09-15T23:23:02.148067+0000","last_peered":"2024-09-15T23:23:02.148067+0000","last_clean":"2024-09-15T23:23:02.148067+0000","last_became_active":"2024-09-15T23:21:52.536027+0000","last_became_peered":"2024-09-15T23:21:52.536027+0000","last_unstale":"2024-09-15T23:23:02.148067+0000","last_undegraded":"2024-09-15T23:23:02.148067+0000","last_fullsized":"2024-09-15T23:23:02.148067+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:16.435912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.276164+0000","last_change":"2024-09-15T23:21:50.562945+0000","last_active":"2024-09-15T23:23:02.276164+0000","last_peered":"2024-09-15T23:23:02.276164+0000","last_clean":"2024-09-15T23:23:02.276164+0000","last_became_active":"2024-09-15T23:21:50.561632+0000","last_became_peered":"2024-09-15T23:21:50.561632+0000","last_unstale":"2024-09-15T23:23:02.276164+0000","last_undegraded":"2024-09-15T23:23:02.276164+0000","last_fullsized":"2024-09-15T23:23:02.276164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:44.545489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146498+0000","last_change":"2024-09-15T23:21:59.564793+0000","last_active":"2024-09-15T23:23:02.146498+0000","last_peered":"2024-09-15T23:23:02.146498+0000","last_clean":"2024-09-15T23:23:02.146498+0000","last_became_active":"2024-09-15T23:21:59.564140+0000","last_became_peered":"2024-09-15T23:21:59.564140+0000","last_unstale":"2024-09-15T23:23:02.146498+0000","last_undegraded":"2024-09-15T23:23:02.146498+0000","last_fullsized":"2024-09-15T23:23:02.146498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:41.464962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146857+0000","last_change":"2024-09-15T23:21:52.533521+0000","last_active":"2024-09-15T23:23:02.146857+0000","last_peered":"2024-09-15T23:23:02.146857+0000","last_clean":"2024-09-15T23:23:02.146857+0000","last_became_active":"2024-09-15T23:21:52.532195+0000","last_became_peered":"2024-09-15T23:21:52.532195+0000","last_unstale":"2024-09-15T23:23:02.146857+0000","last_undegraded":"2024-09-15T23:23:02.146857+0000","last_fullsized":"2024-09-15T23:23:02.146857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:14.229018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143658+0000","last_change":"2024-09-15T23:21:50.562893+0000","last_active":"2024-09-15T23:23:02.143658+0000","last_peered":"2024-09-15T23:23:02.143658+0000","last_clean":"2024-09-15T23:23:02.143658+0000","last_became_active":"2024-09-15T23:21:50.562516+0000","last_became_peered":"2024-09-15T23:21:50.562516+0000","last_unstale":"2024-09-15T23:23:02.143658+0000","last_undegraded":"2024-09-15T23:23:02.143658+0000","last_fullsized":"2024-09-15T23:23:02.143658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:47:47.442458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275687+0000","last_change":"2024-09-15T23:21:59.682091+0000","last_active":"2024-09-15T23:23:02.275687+0000","last_peered":"2024-09-15T23:23:02.275687+0000","last_clean":"2024-09-15T23:23:02.275687+0000","last_became_active":"2024-09-15T23:21:59.681574+0000","last_became_peered":"2024-09-15T23:21:59.681574+0000","last_unstale":"2024-09-15T23:23:02.275687+0000","last_undegraded":"2024-09-15T23:23:02.275687+0000","last_fullsized":"2024-09-15T23:23:02.275687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:04.991608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146611+0000","last_change":"2024-09-15T23:21:52.533811+0000","last_active":"2024-09-15T23:23:02.146611+0000","last_peered":"2024-09-15T23:23:02.146611+0000","last_clean":"2024-09-15T23:23:02.146611+0000","last_became_active":"2024-09-15T23:21:52.533487+0000","last_became_peered":"2024-09-15T23:21:52.533487+0000","last_unstale":"2024-09-15T23:23:02.146611+0000","last_undegraded":"2024-09-15T23:23:02.146611+0000","last_fullsized":"2024-09-15T23:23:02.146611+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:44.193212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:25.214164+0000","last_change":"2024-09-15T23:21:50.510883+0000","last_active":"2024-09-15T23:23:25.214164+0000","last_peered":"2024-09-15T23:23:25.214164+0000","last_clean":"2024-09-15T23:23:25.214164+0000","last_became_active":"2024-09-15T23:21:50.510478+0000","last_became_peered":"2024-09-15T23:21:50.510478+0000","last_unstale":"2024-09-15T23:23:25.214164+0000","last_undegraded":"2024-09-15T23:23:25.214164+0000","last_fullsized":"2024-09-15T23:23:25.214164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:35.454885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147693+0000","last_change":"2024-09-15T23:21:59.561816+0000","last_active":"2024-09-15T23:23:02.147693+0000","last_peered":"2024-09-15T23:23:02.147693+0000","last_clean":"2024-09-15T23:23:02.147693+0000","last_became_active":"2024-09-15T23:21:59.561434+0000","last_became_peered":"2024-09-15T23:21:59.561434+0000","last_unstale":"2024-09-15T23:23:02.147693+0000","last_undegraded":"2024-09-15T23:23:02.147693+0000","last_fullsized":"2024-09-15T23:23:02.147693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:59.945802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.277204+0000","last_change":"2024-09-15T23:23:02.277204+0000","last_active":"2024-09-15T23:23:02.277204+0000","last_peered":"2024-09-15T23:23:02.277204+0000","last_clean":"2024-09-15T23:23:02.277204+0000","last_became_active":"2024-09-15T23:23:02.276770+0000","last_became_peered":"2024-09-15T23:23:02.276770+0000","last_unstale":"2024-09-15T23:23:02.277204+0000","last_undegraded":"2024-09-15T23:23:02.277204+0000","last_fullsized":"2024-09-15T23:23:02.277204+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:11.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:24.207917+0000","last_change":"2024-09-15T23:21:50.561684+0000","last_active":"2024-09-15T23:23:24.207917+0000","last_peered":"2024-09-15T23:23:24.207917+0000","last_clean":"2024-09-15T23:23:24.207917+0000","last_became_active":"2024-09-15T23:21:50.561335+0000","last_became_peered":"2024-09-15T23:21:50.561335+0000","last_unstale":"2024-09-15T23:23:24.207917+0000","last_undegraded":"2024-09-15T23:23:24.207917+0000","last_fullsized":"2024-09-15T23:23:24.207917+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:23.897250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148009+0000","last_change":"2024-09-15T23:21:59.681858+0000","last_active":"2024-09-15T23:23:02.148009+0000","last_peered":"2024-09-15T23:23:02.148009+0000","last_clean":"2024-09-15T23:23:02.148009+0000","last_became_active":"2024-09-15T23:21:59.681441+0000","last_became_peered":"2024-09-15T23:21:59.681441+0000","last_unstale":"2024-09-15T23:23:02.148009+0000","last_undegraded":"2024-09-15T23:23:02.148009+0000","last_fullsized":"2024-09-15T23:23:02.148009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:28.513719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145851+0000","last_change":"2024-09-15T23:21:52.536141+0000","last_active":"2024-09-15T23:23:02.145851+0000","last_peered":"2024-09-15T23:23:02.145851+0000","last_clean":"2024-09-15T23:23:02.145851+0000","last_became_active":"2024-09-15T23:21:52.535832+0000","last_became_peered":"2024-09-15T23:21:52.535832+0000","last_unstale":"2024-09-15T23:23:02.145851+0000","last_undegraded":"2024-09-15T23:23:02.145851+0000","last_fullsized":"2024-09-15T23:23:02.145851+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:21:44.341412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144659+0000","last_change":"2024-09-15T23:21:50.514689+0000","last_active":"2024-09-15T23:23:02.144659+0000","last_peered":"2024-09-15T23:23:02.144659+0000","last_clean":"2024-09-15T23:23:02.144659+0000","last_became_active":"2024-09-15T23:21:50.513261+0000","last_became_peered":"2024-09-15T23:21:50.513261+0000","last_unstale":"2024-09-15T23:23:02.144659+0000","last_undegraded":"2024-09-15T23:23:02.144659+0000","last_fullsized":"2024-09-15T23:23:02.144659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:30.132991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149120+0000","last_change":"2024-09-15T23:21:59.682118+0000","last_active":"2024-09-15T23:23:02.149120+0000","last_peered":"2024-09-15T23:23:02.149120+0000","last_clean":"2024-09-15T23:23:02.149120+0000","last_became_active":"2024-09-15T23:21:59.681500+0000","last_became_peered":"2024-09-15T23:21:59.681500+0000","last_unstale":"2024-09-15T23:23:02.149120+0000","last_undegraded":"2024-09-15T23:23:02.149120+0000","last_fullsized":"2024-09-15T23:23:02.149120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:58.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381664+0000","last_change":"2024-09-15T23:21:52.540458+0000","last_active":"2024-09-15T23:23:02.381664+0000","last_peered":"2024-09-15T23:23:02.381664+0000","last_clean":"2024-09-15T23:23:02.381664+0000","last_became_active":"2024-09-15T23:21:52.540112+0000","last_became_peered":"2024-09-15T23:21:52.540112+0000","last_unstale":"2024-09-15T23:23:02.381664+0000","last_undegraded":"2024-09-15T23:23:02.381664+0000","last_fullsized":"2024-09-15T23:23:02.381664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:15.240785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179608+0000","last_change":"2024-09-15T23:21:50.513124+0000","last_active":"2024-09-15T23:23:02.179608+0000","last_peered":"2024-09-15T23:23:02.179608+0000","last_clean":"2024-09-15T23:23:02.179608+0000","last_became_active":"2024-09-15T23:21:50.512705+0000","last_became_peered":"2024-09-15T23:21:50.512705+0000","last_unstale":"2024-09-15T23:23:02.179608+0000","last_undegraded":"2024-09-15T23:23:02.179608+0000","last_fullsized":"2024-09-15T23:23:02.179608+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.417937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":350,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144804+0000","last_change":"2024-09-15T23:20:40.656652+0000","last_active":"2024-09-15T23:23:02.144804+0000","last_peered":"2024-09-15T23:23:02.144804+0000","last_clean":"2024-09-15T23:23:02.144804+0000","last_became_active":"2024-09-15T23:20:40.656266+0000","last_became_peered":"2024-09-15T23:20:40.656266+0000","last_unstale":"2024-09-15T23:23:02.144804+0000","last_undegraded":"2024-09-15T23:23:02.144804+0000","last_fullsized":"2024-09-15T23:23:02.144804+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_clean_scrub_stamp":"2024-09-15T23:17:15.186850+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:42.317006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145235+0000","last_change":"2024-09-15T23:21:59.563777+0000","last_active":"2024-09-15T23:23:02.145235+0000","last_peered":"2024-09-15T23:23:02.145235+0000","last_clean":"2024-09-15T23:23:02.145235+0000","last_became_active":"2024-09-15T23:21:59.563320+0000","last_became_peered":"2024-09-15T23:21:59.563320+0000","last_unstale":"2024-09-15T23:23:02.145235+0000","last_undegraded":"2024-09-15T23:23:02.145235+0000","last_fullsized":"2024-09-15T23:23:02.145235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:25:07.791923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145457+0000","last_change":"2024-09-15T23:21:52.539842+0000","last_active":"2024-09-15T23:23:02.145457+0000","last_peered":"2024-09-15T23:23:02.145457+0000","last_clean":"2024-09-15T23:23:02.145457+0000","last_became_active":"2024-09-15T23:21:52.538683+0000","last_became_peered":"2024-09-15T23:21:52.538683+0000","last_unstale":"2024-09-15T23:23:02.145457+0000","last_undegraded":"2024-09-15T23:23:02.145457+0000","last_fullsized":"2024-09-15T23:23:02.145457+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:43.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:28.214060+0000","last_change":"2024-09-15T23:21:50.767601+0000","last_active":"2024-09-15T23:23:28.214060+0000","last_peered":"2024-09-15T23:23:28.214060+0000","last_clean":"2024-09-15T23:23:28.214060+0000","last_became_active":"2024-09-15T23:21:50.767149+0000","last_became_peered":"2024-09-15T23:21:50.767149+0000","last_unstale":"2024-09-15T23:23:28.214060+0000","last_undegraded":"2024-09-15T23:23:28.214060+0000","last_fullsized":"2024-09-15T23:23:28.214060+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:23.636551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148132+0000","last_change":"2024-09-15T23:21:59.563069+0000","last_active":"2024-09-15T23:23:02.148132+0000","last_peered":"2024-09-15T23:23:02.148132+0000","last_clean":"2024-09-15T23:23:02.148132+0000","last_became_active":"2024-09-15T23:21:59.562658+0000","last_became_peered":"2024-09-15T23:21:59.562658+0000","last_unstale":"2024-09-15T23:23:02.148132+0000","last_undegraded":"2024-09-15T23:23:02.148132+0000","last_fullsized":"2024-09-15T23:23:02.148132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:23.067198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145096+0000","last_change":"2024-09-15T23:21:52.535289+0000","last_active":"2024-09-15T23:23:02.145096+0000","last_peered":"2024-09-15T23:23:02.145096+0000","last_clean":"2024-09-15T23:23:02.145096+0000","last_became_active":"2024-09-15T23:21:52.534885+0000","last_became_peered":"2024-09-15T23:21:52.534885+0000","last_unstale":"2024-09-15T23:23:02.145096+0000","last_undegraded":"2024-09-15T23:23:02.145096+0000","last_fullsized":"2024-09-15T23:23:02.145096+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:42.293786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149082+0000","last_change":"2024-09-15T23:21:50.513246+0000","last_active":"2024-09-15T23:23:02.149082+0000","last_peered":"2024-09-15T23:23:02.149082+0000","last_clean":"2024-09-15T23:23:02.149082+0000","last_became_active":"2024-09-15T23:21:50.512740+0000","last_became_peered":"2024-09-15T23:21:50.512740+0000","last_unstale":"2024-09-15T23:23:02.149082+0000","last_undegraded":"2024-09-15T23:23:02.149082+0000","last_fullsized":"2024-09-15T23:23:02.149082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:59.201970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381169+0000","last_change":"2024-09-15T23:21:59.568565+0000","last_active":"2024-09-15T23:23:02.381169+0000","last_peered":"2024-09-15T23:23:02.381169+0000","last_clean":"2024-09-15T23:23:02.381169+0000","last_became_active":"2024-09-15T23:21:59.567789+0000","last_became_peered":"2024-09-15T23:21:59.567789+0000","last_unstale":"2024-09-15T23:23:02.381169+0000","last_undegraded":"2024-09-15T23:23:02.381169+0000","last_fullsized":"2024-09-15T23:23:02.381169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:02.106971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145412+0000","last_change":"2024-09-15T23:21:52.533727+0000","last_active":"2024-09-15T23:23:02.145412+0000","last_peered":"2024-09-15T23:23:02.145412+0000","last_clean":"2024-09-15T23:23:02.145412+0000","last_became_active":"2024-09-15T23:21:52.532350+0000","last_became_peered":"2024-09-15T23:21:52.532350+0000","last_unstale":"2024-09-15T23:23:02.145412+0000","last_undegraded":"2024-09-15T23:23:02.145412+0000","last_fullsized":"2024-09-15T23:23:02.145412+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:03.624837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148178+0000","last_change":"2024-09-15T23:21:50.509519+0000","last_active":"2024-09-15T23:23:02.148178+0000","last_peered":"2024-09-15T23:23:02.148178+0000","last_clean":"2024-09-15T23:23:02.148178+0000","last_became_active":"2024-09-15T23:21:50.509224+0000","last_became_peered":"2024-09-15T23:21:50.509224+0000","last_unstale":"2024-09-15T23:23:02.148178+0000","last_undegraded":"2024-09-15T23:23:02.148178+0000","last_fullsized":"2024-09-15T23:23:02.148178+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:39:57.526821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147867+0000","last_change":"2024-09-15T23:21:59.566429+0000","last_active":"2024-09-15T23:23:02.147867+0000","last_peered":"2024-09-15T23:23:02.147867+0000","last_clean":"2024-09-15T23:23:02.147867+0000","last_became_active":"2024-09-15T23:21:59.565835+0000","last_became_peered":"2024-09-15T23:21:59.565835+0000","last_unstale":"2024-09-15T23:23:02.147867+0000","last_undegraded":"2024-09-15T23:23:02.147867+0000","last_fullsized":"2024-09-15T23:23:02.147867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:38.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146720+0000","last_change":"2024-09-15T23:21:52.539544+0000","last_active":"2024-09-15T23:23:02.146720+0000","last_peered":"2024-09-15T23:23:02.146720+0000","last_clean":"2024-09-15T23:23:02.146720+0000","last_became_active":"2024-09-15T23:21:52.539203+0000","last_became_peered":"2024-09-15T23:21:52.539203+0000","last_unstale":"2024-09-15T23:23:02.146720+0000","last_undegraded":"2024-09-15T23:23:02.146720+0000","last_fullsized":"2024-09-15T23:23:02.146720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.193965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.276035+0000","last_change":"2024-09-15T23:21:50.513543+0000","last_active":"2024-09-15T23:23:02.276035+0000","last_peered":"2024-09-15T23:23:02.276035+0000","last_clean":"2024-09-15T23:23:02.276035+0000","last_became_active":"2024-09-15T23:21:50.512812+0000","last_became_peered":"2024-09-15T23:21:50.512812+0000","last_unstale":"2024-09-15T23:23:02.276035+0000","last_undegraded":"2024-09-15T23:23:02.276035+0000","last_fullsized":"2024-09-15T23:23:02.276035+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:50.954096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148065+0000","last_change":"2024-09-15T23:21:59.570163+0000","last_active":"2024-09-15T23:23:02.148065+0000","last_peered":"2024-09-15T23:23:02.148065+0000","last_clean":"2024-09-15T23:23:02.148065+0000","last_became_active":"2024-09-15T23:21:59.569805+0000","last_became_peered":"2024-09-15T23:21:59.569805+0000","last_unstale":"2024-09-15T23:23:02.148065+0000","last_undegraded":"2024-09-15T23:23:02.148065+0000","last_fullsized":"2024-09-15T23:23:02.148065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:28.530817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148358+0000","last_change":"2024-09-15T23:21:52.537850+0000","last_active":"2024-09-15T23:23:02.148358+0000","last_peered":"2024-09-15T23:23:02.148358+0000","last_clean":"2024-09-15T23:23:02.148358+0000","last_became_active":"2024-09-15T23:21:52.536579+0000","last_became_peered":"2024-09-15T23:21:52.536579+0000","last_unstale":"2024-09-15T23:23:02.148358+0000","last_undegraded":"2024-09-15T23:23:02.148358+0000","last_fullsized":"2024-09-15T23:23:02.148358+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:06.353615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275622+0000","last_change":"2024-09-15T23:21:50.562221+0000","last_active":"2024-09-15T23:23:02.275622+0000","last_peered":"2024-09-15T23:23:02.275622+0000","last_clean":"2024-09-15T23:23:02.275622+0000","last_became_active":"2024-09-15T23:21:50.561788+0000","last_became_peered":"2024-09-15T23:21:50.561788+0000","last_unstale":"2024-09-15T23:23:02.275622+0000","last_undegraded":"2024-09-15T23:23:02.275622+0000","last_fullsized":"2024-09-15T23:23:02.275622+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.853812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.383003+0000","last_change":"2024-09-15T23:23:02.383003+0000","last_active":"2024-09-15T23:23:02.383003+0000","last_peered":"2024-09-15T23:23:02.383003+0000","last_clean":"2024-09-15T23:23:02.383003+0000","last_became_active":"2024-09-15T23:23:02.382629+0000","last_became_peered":"2024-09-15T23:23:02.382629+0000","last_unstale":"2024-09-15T23:23:02.383003+0000","last_undegraded":"2024-09-15T23:23:02.383003+0000","last_fullsized":"2024-09-15T23:23:02.383003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:50.114689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144569+0000","last_change":"2024-09-15T23:21:52.539566+0000","last_active":"2024-09-15T23:23:02.144569+0000","last_peered":"2024-09-15T23:23:02.144569+0000","last_clean":"2024-09-15T23:23:02.144569+0000","last_became_active":"2024-09-15T23:21:52.539219+0000","last_became_peered":"2024-09-15T23:21:52.539219+0000","last_unstale":"2024-09-15T23:23:02.144569+0000","last_undegraded":"2024-09-15T23:23:02.144569+0000","last_fullsized":"2024-09-15T23:23:02.144569+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:41.266308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:29.215228+0000","last_change":"2024-09-15T23:21:50.516650+0000","last_active":"2024-09-15T23:23:29.215228+0000","last_peered":"2024-09-15T23:23:29.215228+0000","last_clean":"2024-09-15T23:23:29.215228+0000","last_became_active":"2024-09-15T23:21:50.516297+0000","last_became_peered":"2024-09-15T23:21:50.516297+0000","last_unstale":"2024-09-15T23:23:29.215228+0000","last_undegraded":"2024-09-15T23:23:29.215228+0000","last_fullsized":"2024-09-15T23:23:29.215228+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:36.928701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143833+0000","last_change":"2024-09-15T23:21:59.568087+0000","last_active":"2024-09-15T23:23:02.143833+0000","last_peered":"2024-09-15T23:23:02.143833+0000","last_clean":"2024-09-15T23:23:02.143833+0000","last_became_active":"2024-09-15T23:21:59.567693+0000","last_became_peered":"2024-09-15T23:21:59.567693+0000","last_unstale":"2024-09-15T23:23:02.143833+0000","last_undegraded":"2024-09-15T23:23:02.143833+0000","last_fullsized":"2024-09-15T23:23:02.143833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.231873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144909+0000","last_change":"2024-09-15T23:21:52.531090+0000","last_active":"2024-09-15T23:23:02.144909+0000","last_peered":"2024-09-15T23:23:02.144909+0000","last_clean":"2024-09-15T23:23:02.144909+0000","last_became_active":"2024-09-15T23:21:52.530738+0000","last_became_peered":"2024-09-15T23:21:52.530738+0000","last_unstale":"2024-09-15T23:23:02.144909+0000","last_undegraded":"2024-09-15T23:23:02.144909+0000","last_fullsized":"2024-09-15T23:23:02.144909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:35.325128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149158+0000","last_change":"2024-09-15T23:21:50.514100+0000","last_active":"2024-09-15T23:23:02.149158+0000","last_peered":"2024-09-15T23:23:02.149158+0000","last_clean":"2024-09-15T23:23:02.149158+0000","last_became_active":"2024-09-15T23:21:50.513629+0000","last_became_peered":"2024-09-15T23:21:50.513629+0000","last_unstale":"2024-09-15T23:23:02.149158+0000","last_undegraded":"2024-09-15T23:23:02.149158+0000","last_fullsized":"2024-09-15T23:23:02.149158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.187544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145706+0000","last_change":"2024-09-15T23:21:59.562804+0000","last_active":"2024-09-15T23:23:02.145706+0000","last_peered":"2024-09-15T23:23:02.145706+0000","last_clean":"2024-09-15T23:23:02.145706+0000","last_became_active":"2024-09-15T23:21:59.562368+0000","last_became_peered":"2024-09-15T23:21:59.562368+0000","last_unstale":"2024-09-15T23:23:02.145706+0000","last_undegraded":"2024-09-15T23:23:02.145706+0000","last_fullsized":"2024-09-15T23:23:02.145706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:21.137333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146705+0000","last_change":"2024-09-15T23:21:52.530562+0000","last_active":"2024-09-15T23:23:02.146705+0000","last_peered":"2024-09-15T23:23:02.146705+0000","last_clean":"2024-09-15T23:23:02.146705+0000","last_became_active":"2024-09-15T23:21:52.530280+0000","last_became_peered":"2024-09-15T23:21:52.530280+0000","last_unstale":"2024-09-15T23:23:02.146705+0000","last_undegraded":"2024-09-15T23:23:02.146705+0000","last_fullsized":"2024-09-15T23:23:02.146705+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:26.835651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144799+0000","last_change":"2024-09-15T23:21:50.513179+0000","last_active":"2024-09-15T23:23:02.144799+0000","last_peered":"2024-09-15T23:23:02.144799+0000","last_clean":"2024-09-15T23:23:02.144799+0000","last_became_active":"2024-09-15T23:21:50.512790+0000","last_became_peered":"2024-09-15T23:21:50.512790+0000","last_unstale":"2024-09-15T23:23:02.144799+0000","last_undegraded":"2024-09-15T23:23:02.144799+0000","last_fullsized":"2024-09-15T23:23:02.144799+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:56.408198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.383378+0000","last_change":"2024-09-15T23:23:02.382964+0000","last_active":"2024-09-15T23:23:02.383378+0000","last_peered":"2024-09-15T23:23:02.383378+0000","last_clean":"2024-09-15T23:23:02.383378+0000","last_became_active":"2024-09-15T23:23:02.382556+0000","last_became_peered":"2024-09-15T23:23:02.382556+0000","last_unstale":"2024-09-15T23:23:02.383378+0000","last_undegraded":"2024-09-15T23:23:02.383378+0000","last_fullsized":"2024-09-15T23:23:02.383378+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:49.211591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275921+0000","last_change":"2024-09-15T23:21:52.530970+0000","last_active":"2024-09-15T23:23:02.275921+0000","last_peered":"2024-09-15T23:23:02.275921+0000","last_clean":"2024-09-15T23:23:02.275921+0000","last_became_active":"2024-09-15T23:21:52.530636+0000","last_became_peered":"2024-09-15T23:21:52.530636+0000","last_unstale":"2024-09-15T23:23:02.275921+0000","last_undegraded":"2024-09-15T23:23:02.275921+0000","last_fullsized":"2024-09-15T23:23:02.275921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:33.374658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"86'34","reported_seq":175,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:25.209167+0000","last_change":"2024-09-15T23:21:50.518449+0000","last_active":"2024-09-15T23:23:25.209167+0000","last_peered":"2024-09-15T23:23:25.209167+0000","last_clean":"2024-09-15T23:23:25.209167+0000","last_became_active":"2024-09-15T23:21:50.518099+0000","last_became_peered":"2024-09-15T23:21:50.518099+0000","last_unstale":"2024-09-15T23:23:25.209167+0000","last_undegraded":"2024-09-15T23:23:25.209167+0000","last_fullsized":"2024-09-15T23:23:25.209167+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:43.792971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62076,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":107,"num_read_kb":115,"num_write":36,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.277900+0000","last_change":"2024-09-15T23:23:02.277457+0000","last_active":"2024-09-15T23:23:02.277900+0000","last_peered":"2024-09-15T23:23:02.277900+0000","last_clean":"2024-09-15T23:23:02.277900+0000","last_became_active":"2024-09-15T23:23:02.276960+0000","last_became_peered":"2024-09-15T23:23:02.276960+0000","last_unstale":"2024-09-15T23:23:02.277900+0000","last_undegraded":"2024-09-15T23:23:02.277900+0000","last_fullsized":"2024-09-15T23:23:02.277900+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:15:28.823804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148672+0000","last_change":"2024-09-15T23:21:52.530541+0000","last_active":"2024-09-15T23:23:02.148672+0000","last_peered":"2024-09-15T23:23:02.148672+0000","last_clean":"2024-09-15T23:23:02.148672+0000","last_became_active":"2024-09-15T23:21:52.530217+0000","last_became_peered":"2024-09-15T23:21:52.530217+0000","last_unstale":"2024-09-15T23:23:02.148672+0000","last_undegraded":"2024-09-15T23:23:02.148672+0000","last_fullsized":"2024-09-15T23:23:02.148672+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:14.864563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149346+0000","last_change":"2024-09-15T23:21:50.562203+0000","last_active":"2024-09-15T23:23:02.149346+0000","last_peered":"2024-09-15T23:23:02.149346+0000","last_clean":"2024-09-15T23:23:02.149346+0000","last_became_active":"2024-09-15T23:21:50.561827+0000","last_became_peered":"2024-09-15T23:21:50.561827+0000","last_unstale":"2024-09-15T23:23:02.149346+0000","last_undegraded":"2024-09-15T23:23:02.149346+0000","last_fullsized":"2024-09-15T23:23:02.149346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:22.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179436+0000","last_change":"2024-09-15T23:21:59.561013+0000","last_active":"2024-09-15T23:23:02.179436+0000","last_peered":"2024-09-15T23:23:02.179436+0000","last_clean":"2024-09-15T23:23:02.179436+0000","last_became_active":"2024-09-15T23:21:59.560665+0000","last_became_peered":"2024-09-15T23:21:59.560665+0000","last_unstale":"2024-09-15T23:23:02.179436+0000","last_undegraded":"2024-09-15T23:23:02.179436+0000","last_fullsized":"2024-09-15T23:23:02.179436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:00.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381152+0000","last_change":"2024-09-15T23:21:52.529922+0000","last_active":"2024-09-15T23:23:02.381152+0000","last_peered":"2024-09-15T23:23:02.381152+0000","last_clean":"2024-09-15T23:23:02.381152+0000","last_became_active":"2024-09-15T23:21:52.529179+0000","last_became_peered":"2024-09-15T23:21:52.529179+0000","last_unstale":"2024-09-15T23:23:02.381152+0000","last_undegraded":"2024-09-15T23:23:02.381152+0000","last_fullsized":"2024-09-15T23:23:02.381152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.077658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145411+0000","last_change":"2024-09-15T23:21:50.514420+0000","last_active":"2024-09-15T23:23:02.145411+0000","last_peered":"2024-09-15T23:23:02.145411+0000","last_clean":"2024-09-15T23:23:02.145411+0000","last_became_active":"2024-09-15T23:21:50.514007+0000","last_became_peered":"2024-09-15T23:21:50.514007+0000","last_unstale":"2024-09-15T23:23:02.145411+0000","last_undegraded":"2024-09-15T23:23:02.145411+0000","last_fullsized":"2024-09-15T23:23:02.145411+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:36.845674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143335+0000","last_change":"2024-09-15T23:21:59.565978+0000","last_active":"2024-09-15T23:23:02.143335+0000","last_peered":"2024-09-15T23:23:02.143335+0000","last_clean":"2024-09-15T23:23:02.143335+0000","last_became_active":"2024-09-15T23:21:59.565604+0000","last_became_peered":"2024-09-15T23:21:59.565604+0000","last_unstale":"2024-09-15T23:23:02.143335+0000","last_undegraded":"2024-09-15T23:23:02.143335+0000","last_fullsized":"2024-09-15T23:23:02.143335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:47.089504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148587+0000","last_change":"2024-09-15T23:21:52.543511+0000","last_active":"2024-09-15T23:23:02.148587+0000","last_peered":"2024-09-15T23:23:02.148587+0000","last_clean":"2024-09-15T23:23:02.148587+0000","last_became_active":"2024-09-15T23:21:52.543169+0000","last_became_peered":"2024-09-15T23:21:52.543169+0000","last_unstale":"2024-09-15T23:23:02.148587+0000","last_undegraded":"2024-09-15T23:23:02.148587+0000","last_fullsized":"2024-09-15T23:23:02.148587+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:05.696772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:25.209347+0000","last_change":"2024-09-15T23:21:50.518425+0000","last_active":"2024-09-15T23:23:25.209347+0000","last_peered":"2024-09-15T23:23:25.209347+0000","last_clean":"2024-09-15T23:23:25.209347+0000","last_became_active":"2024-09-15T23:21:50.518052+0000","last_became_peered":"2024-09-15T23:21:50.518052+0000","last_unstale":"2024-09-15T23:23:25.209347+0000","last_undegraded":"2024-09-15T23:23:25.209347+0000","last_fullsized":"2024-09-15T23:23:25.209347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:02.243869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380690+0000","last_change":"2024-09-15T23:21:59.570310+0000","last_active":"2024-09-15T23:23:02.380690+0000","last_peered":"2024-09-15T23:23:02.380690+0000","last_clean":"2024-09-15T23:23:02.380690+0000","last_became_active":"2024-09-15T23:21:59.569909+0000","last_became_peered":"2024-09-15T23:21:59.569909+0000","last_unstale":"2024-09-15T23:23:02.380690+0000","last_undegraded":"2024-09-15T23:23:02.380690+0000","last_fullsized":"2024-09-15T23:23:02.380690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:20:27.515537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380837+0000","last_change":"2024-09-15T23:21:59.568208+0000","last_active":"2024-09-15T23:23:02.380837+0000","last_peered":"2024-09-15T23:23:02.380837+0000","last_clean":"2024-09-15T23:23:02.380837+0000","last_became_active":"2024-09-15T23:21:59.567858+0000","last_became_peered":"2024-09-15T23:21:59.567858+0000","last_unstale":"2024-09-15T23:23:02.380837+0000","last_undegraded":"2024-09-15T23:23:02.380837+0000","last_fullsized":"2024-09-15T23:23:02.380837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:34:42.895247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143446+0000","last_change":"2024-09-15T23:21:52.532679+0000","last_active":"2024-09-15T23:23:02.143446+0000","last_peered":"2024-09-15T23:23:02.143446+0000","last_clean":"2024-09-15T23:23:02.143446+0000","last_became_active":"2024-09-15T23:21:52.532345+0000","last_became_peered":"2024-09-15T23:21:52.532345+0000","last_unstale":"2024-09-15T23:23:02.143446+0000","last_undegraded":"2024-09-15T23:23:02.143446+0000","last_fullsized":"2024-09-15T23:23:02.143446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:45.943653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148716+0000","last_change":"2024-09-15T23:21:50.510836+0000","last_active":"2024-09-15T23:23:02.148716+0000","last_peered":"2024-09-15T23:23:02.148716+0000","last_clean":"2024-09-15T23:23:02.148716+0000","last_became_active":"2024-09-15T23:21:50.510514+0000","last_became_peered":"2024-09-15T23:21:50.510514+0000","last_unstale":"2024-09-15T23:23:02.148716+0000","last_undegraded":"2024-09-15T23:23:02.148716+0000","last_fullsized":"2024-09-15T23:23:02.148716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:06.404612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.383436+0000","last_change":"2024-09-15T23:23:02.383436+0000","last_active":"2024-09-15T23:23:02.383436+0000","last_peered":"2024-09-15T23:23:02.383436+0000","last_clean":"2024-09-15T23:23:02.383436+0000","last_became_active":"2024-09-15T23:23:02.382982+0000","last_became_peered":"2024-09-15T23:23:02.382982+0000","last_unstale":"2024-09-15T23:23:02.383436+0000","last_undegraded":"2024-09-15T23:23:02.383436+0000","last_fullsized":"2024-09-15T23:23:02.383436+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:55:14.076604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381259+0000","last_change":"2024-09-15T23:21:52.526766+0000","last_active":"2024-09-15T23:23:02.381259+0000","last_peered":"2024-09-15T23:23:02.381259+0000","last_clean":"2024-09-15T23:23:02.381259+0000","last_became_active":"2024-09-15T23:21:52.526446+0000","last_became_peered":"2024-09-15T23:21:52.526446+0000","last_unstale":"2024-09-15T23:23:02.381259+0000","last_undegraded":"2024-09-15T23:23:02.381259+0000","last_fullsized":"2024-09-15T23:23:02.381259+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:21:37.004730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145965+0000","last_change":"2024-09-15T23:21:50.512928+0000","last_active":"2024-09-15T23:23:02.145965+0000","last_peered":"2024-09-15T23:23:02.145965+0000","last_clean":"2024-09-15T23:23:02.145965+0000","last_became_active":"2024-09-15T23:21:50.512558+0000","last_became_peered":"2024-09-15T23:21:50.512558+0000","last_unstale":"2024-09-15T23:23:02.145965+0000","last_undegraded":"2024-09-15T23:23:02.145965+0000","last_fullsized":"2024-09-15T23:23:02.145965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:08.653084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147478+0000","last_change":"2024-09-15T23:21:50.516920+0000","last_active":"2024-09-15T23:23:02.147478+0000","last_peered":"2024-09-15T23:23:02.147478+0000","last_clean":"2024-09-15T23:23:02.147478+0000","last_became_active":"2024-09-15T23:21:50.516573+0000","last_became_peered":"2024-09-15T23:21:50.516573+0000","last_unstale":"2024-09-15T23:23:02.147478+0000","last_undegraded":"2024-09-15T23:23:02.147478+0000","last_fullsized":"2024-09-15T23:23:02.147478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.367903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148514+0000","last_change":"2024-09-15T23:21:52.536696+0000","last_active":"2024-09-15T23:23:02.148514+0000","last_peered":"2024-09-15T23:23:02.148514+0000","last_clean":"2024-09-15T23:23:02.148514+0000","last_became_active":"2024-09-15T23:21:52.536338+0000","last_became_peered":"2024-09-15T23:21:52.536338+0000","last_unstale":"2024-09-15T23:23:02.148514+0000","last_undegraded":"2024-09-15T23:23:02.148514+0000","last_fullsized":"2024-09-15T23:23:02.148514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:05.152749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144628+0000","last_change":"2024-09-15T23:21:59.562968+0000","last_active":"2024-09-15T23:23:02.144628+0000","last_peered":"2024-09-15T23:23:02.144628+0000","last_clean":"2024-09-15T23:23:02.144628+0000","last_became_active":"2024-09-15T23:21:59.562579+0000","last_became_peered":"2024-09-15T23:21:59.562579+0000","last_unstale":"2024-09-15T23:23:02.144628+0000","last_undegraded":"2024-09-15T23:23:02.144628+0000","last_fullsized":"2024-09-15T23:23:02.144628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:02.853057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:25.210180+0000","last_change":"2024-09-15T23:21:50.513944+0000","last_active":"2024-09-15T23:23:25.210180+0000","last_peered":"2024-09-15T23:23:25.210180+0000","last_clean":"2024-09-15T23:23:25.210180+0000","last_became_active":"2024-09-15T23:21:50.513484+0000","last_became_peered":"2024-09-15T23:21:50.513484+0000","last_unstale":"2024-09-15T23:23:25.210180+0000","last_undegraded":"2024-09-15T23:23:25.210180+0000","last_fullsized":"2024-09-15T23:23:25.210180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:35.719344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275739+0000","last_change":"2024-09-15T23:21:52.539467+0000","last_active":"2024-09-15T23:23:02.275739+0000","last_peered":"2024-09-15T23:23:02.275739+0000","last_clean":"2024-09-15T23:23:02.275739+0000","last_became_active":"2024-09-15T23:21:52.538998+0000","last_became_peered":"2024-09-15T23:21:52.538998+0000","last_unstale":"2024-09-15T23:23:02.275739+0000","last_undegraded":"2024-09-15T23:23:02.275739+0000","last_fullsized":"2024-09-15T23:23:02.275739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:22.213857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145115+0000","last_change":"2024-09-15T23:21:59.564842+0000","last_active":"2024-09-15T23:23:02.145115+0000","last_peered":"2024-09-15T23:23:02.145115+0000","last_clean":"2024-09-15T23:23:02.145115+0000","last_became_active":"2024-09-15T23:21:59.564408+0000","last_became_peered":"2024-09-15T23:21:59.564408+0000","last_unstale":"2024-09-15T23:23:02.145115+0000","last_undegraded":"2024-09-15T23:23:02.145115+0000","last_fullsized":"2024-09-15T23:23:02.145115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:17.909020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149420+0000","last_change":"2024-09-15T23:23:02.149420+0000","last_active":"2024-09-15T23:23:02.149420+0000","last_peered":"2024-09-15T23:23:02.149420+0000","last_clean":"2024-09-15T23:23:02.149420+0000","last_became_active":"2024-09-15T23:23:02.149003+0000","last_became_peered":"2024-09-15T23:23:02.149003+0000","last_unstale":"2024-09-15T23:23:02.149420+0000","last_undegraded":"2024-09-15T23:23:02.149420+0000","last_fullsized":"2024-09-15T23:23:02.149420+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:45.490322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144804+0000","last_change":"2024-09-15T23:21:52.532769+0000","last_active":"2024-09-15T23:23:02.144804+0000","last_peered":"2024-09-15T23:23:02.144804+0000","last_clean":"2024-09-15T23:23:02.144804+0000","last_became_active":"2024-09-15T23:21:52.532426+0000","last_became_peered":"2024-09-15T23:21:52.532426+0000","last_unstale":"2024-09-15T23:23:02.144804+0000","last_undegraded":"2024-09-15T23:23:02.144804+0000","last_fullsized":"2024-09-15T23:23:02.144804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:09.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144463+0000","last_change":"2024-09-15T23:21:52.533148+0000","last_active":"2024-09-15T23:23:02.144463+0000","last_peered":"2024-09-15T23:23:02.144463+0000","last_clean":"2024-09-15T23:23:02.144463+0000","last_became_active":"2024-09-15T23:21:52.532823+0000","last_became_peered":"2024-09-15T23:21:52.532823+0000","last_unstale":"2024-09-15T23:23:02.144463+0000","last_undegraded":"2024-09-15T23:23:02.144463+0000","last_fullsized":"2024-09-15T23:23:02.144463+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:01.833945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381268+0000","last_change":"2024-09-15T23:21:52.536790+0000","last_active":"2024-09-15T23:23:02.381268+0000","last_peered":"2024-09-15T23:23:02.381268+0000","last_clean":"2024-09-15T23:23:02.381268+0000","last_became_active":"2024-09-15T23:21:52.536515+0000","last_became_peered":"2024-09-15T23:21:52.536515+0000","last_unstale":"2024-09-15T23:23:02.381268+0000","last_undegraded":"2024-09-15T23:23:02.381268+0000","last_fullsized":"2024-09-15T23:23:02.381268+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:37.207759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.382842+0000","last_change":"2024-09-15T23:23:02.382842+0000","last_active":"2024-09-15T23:23:02.382842+0000","last_peered":"2024-09-15T23:23:02.382842+0000","last_clean":"2024-09-15T23:23:02.382842+0000","last_became_active":"2024-09-15T23:23:02.382455+0000","last_became_peered":"2024-09-15T23:23:02.382455+0000","last_unstale":"2024-09-15T23:23:02.382842+0000","last_undegraded":"2024-09-15T23:23:02.382842+0000","last_fullsized":"2024-09-15T23:23:02.382842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:23.617641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147039+0000","last_change":"2024-09-15T23:21:52.540628+0000","last_active":"2024-09-15T23:23:02.147039+0000","last_peered":"2024-09-15T23:23:02.147039+0000","last_clean":"2024-09-15T23:23:02.147039+0000","last_became_active":"2024-09-15T23:21:52.540308+0000","last_became_peered":"2024-09-15T23:21:52.540308+0000","last_unstale":"2024-09-15T23:23:02.147039+0000","last_undegraded":"2024-09-15T23:23:02.147039+0000","last_fullsized":"2024-09-15T23:23:02.147039+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:16.660637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146562+0000","last_change":"2024-09-15T23:21:52.533780+0000","last_active":"2024-09-15T23:23:02.146562+0000","last_peered":"2024-09-15T23:23:02.146562+0000","last_clean":"2024-09-15T23:23:02.146562+0000","last_became_active":"2024-09-15T23:21:52.533354+0000","last_became_peered":"2024-09-15T23:21:52.533354+0000","last_unstale":"2024-09-15T23:23:02.146562+0000","last_undegraded":"2024-09-15T23:23:02.146562+0000","last_fullsized":"2024-09-15T23:23:02.146562+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.381059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381783+0000","last_change":"2024-09-15T23:21:52.536406+0000","last_active":"2024-09-15T23:23:02.381783+0000","last_peered":"2024-09-15T23:23:02.381783+0000","last_clean":"2024-09-15T23:23:02.381783+0000","last_became_active":"2024-09-15T23:21:52.536091+0000","last_became_peered":"2024-09-15T23:21:52.536091+0000","last_unstale":"2024-09-15T23:23:02.381783+0000","last_undegraded":"2024-09-15T23:23:02.381783+0000","last_fullsized":"2024-09-15T23:23:02.381783+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:30.452201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148544+0000","last_change":"2024-09-15T23:21:52.536333+0000","last_active":"2024-09-15T23:23:02.148544+0000","last_peered":"2024-09-15T23:23:02.148544+0000","last_clean":"2024-09-15T23:23:02.148544+0000","last_became_active":"2024-09-15T23:21:52.534677+0000","last_became_peered":"2024-09-15T23:21:52.534677+0000","last_unstale":"2024-09-15T23:23:02.148544+0000","last_undegraded":"2024-09-15T23:23:02.148544+0000","last_fullsized":"2024-09-15T23:23:02.148544+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:05.287914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380781+0000","last_change":"2024-09-15T23:21:52.536442+0000","last_active":"2024-09-15T23:23:02.380781+0000","last_peered":"2024-09-15T23:23:02.380781+0000","last_clean":"2024-09-15T23:23:02.380781+0000","last_became_active":"2024-09-15T23:21:52.536183+0000","last_became_peered":"2024-09-15T23:21:52.536183+0000","last_unstale":"2024-09-15T23:23:02.380781+0000","last_undegraded":"2024-09-15T23:23:02.380781+0000","last_fullsized":"2024-09-15T23:23:02.380781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:41.641370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.277365+0000","last_change":"2024-09-15T23:23:02.277365+0000","last_active":"2024-09-15T23:23:02.277365+0000","last_peered":"2024-09-15T23:23:02.277365+0000","last_clean":"2024-09-15T23:23:02.277365+0000","last_became_active":"2024-09-15T23:23:02.276905+0000","last_became_peered":"2024-09-15T23:23:02.276905+0000","last_unstale":"2024-09-15T23:23:02.277365+0000","last_undegraded":"2024-09-15T23:23:02.277365+0000","last_fullsized":"2024-09-15T23:23:02.277365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:21.100657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146982+0000","last_change":"2024-09-15T23:21:52.531171+0000","last_active":"2024-09-15T23:23:02.146982+0000","last_peered":"2024-09-15T23:23:02.146982+0000","last_clean":"2024-09-15T23:23:02.146982+0000","last_became_active":"2024-09-15T23:21:52.530724+0000","last_became_peered":"2024-09-15T23:21:52.530724+0000","last_unstale":"2024-09-15T23:23:02.146982+0000","last_undegraded":"2024-09-15T23:23:02.146982+0000","last_fullsized":"2024-09-15T23:23:02.146982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:11.009119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145814+0000","last_change":"2024-09-15T23:21:52.530433+0000","last_active":"2024-09-15T23:23:02.145814+0000","last_peered":"2024-09-15T23:23:02.145814+0000","last_clean":"2024-09-15T23:23:02.145814+0000","last_became_active":"2024-09-15T23:21:52.530058+0000","last_became_peered":"2024-09-15T23:21:52.530058+0000","last_unstale":"2024-09-15T23:23:02.145814+0000","last_undegraded":"2024-09-15T23:23:02.145814+0000","last_fullsized":"2024-09-15T23:23:02.145814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:51.208820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146177+0000","last_change":"2024-09-15T23:21:52.534639+0000","last_active":"2024-09-15T23:23:02.146177+0000","last_peered":"2024-09-15T23:23:02.146177+0000","last_clean":"2024-09-15T23:23:02.146177+0000","last_became_active":"2024-09-15T23:21:52.534362+0000","last_became_peered":"2024-09-15T23:21:52.534362+0000","last_unstale":"2024-09-15T23:23:02.146177+0000","last_undegraded":"2024-09-15T23:23:02.146177+0000","last_fullsized":"2024-09-15T23:23:02.146177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:58:12.023501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143861+0000","last_change":"2024-09-15T23:21:52.532758+0000","last_active":"2024-09-15T23:23:02.143861+0000","last_peered":"2024-09-15T23:23:02.143861+0000","last_clean":"2024-09-15T23:23:02.143861+0000","last_became_active":"2024-09-15T23:21:52.532412+0000","last_became_peered":"2024-09-15T23:21:52.532412+0000","last_unstale":"2024-09-15T23:23:02.143861+0000","last_undegraded":"2024-09-15T23:23:02.143861+0000","last_fullsized":"2024-09-15T23:23:02.143861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:46.285711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146969+0000","last_change":"2024-09-15T23:21:52.533928+0000","last_active":"2024-09-15T23:23:02.146969+0000","last_peered":"2024-09-15T23:23:02.146969+0000","last_clean":"2024-09-15T23:23:02.146969+0000","last_became_active":"2024-09-15T23:21:52.533578+0000","last_became_peered":"2024-09-15T23:21:52.533578+0000","last_unstale":"2024-09-15T23:23:02.146969+0000","last_undegraded":"2024-09-15T23:23:02.146969+0000","last_fullsized":"2024-09-15T23:23:02.146969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:49.476316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147896+0000","last_change":"2024-09-15T23:21:52.533417+0000","last_active":"2024-09-15T23:23:02.147896+0000","last_peered":"2024-09-15T23:23:02.147896+0000","last_clean":"2024-09-15T23:23:02.147896+0000","last_became_active":"2024-09-15T23:21:52.533085+0000","last_became_peered":"2024-09-15T23:21:52.533085+0000","last_unstale":"2024-09-15T23:23:02.147896+0000","last_undegraded":"2024-09-15T23:23:02.147896+0000","last_fullsized":"2024-09-15T23:23:02.147896+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:20.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380911+0000","last_change":"2024-09-15T23:21:52.542340+0000","last_active":"2024-09-15T23:23:02.380911+0000","last_peered":"2024-09-15T23:23:02.380911+0000","last_clean":"2024-09-15T23:23:02.380911+0000","last_became_active":"2024-09-15T23:21:52.542048+0000","last_became_peered":"2024-09-15T23:21:52.542048+0000","last_unstale":"2024-09-15T23:23:02.380911+0000","last_undegraded":"2024-09-15T23:23:02.380911+0000","last_fullsized":"2024-09-15T23:23:02.380911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:57.557420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380816+0000","last_change":"2024-09-15T23:21:52.535576+0000","last_active":"2024-09-15T23:23:02.380816+0000","last_peered":"2024-09-15T23:23:02.380816+0000","last_clean":"2024-09-15T23:23:02.380816+0000","last_became_active":"2024-09-15T23:21:52.535223+0000","last_became_peered":"2024-09-15T23:21:52.535223+0000","last_unstale":"2024-09-15T23:23:02.380816+0000","last_undegraded":"2024-09-15T23:23:02.380816+0000","last_fullsized":"2024-09-15T23:23:02.380816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:45.874927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.150556+0000","last_change":"2024-09-15T23:23:02.150556+0000","last_active":"2024-09-15T23:23:02.150556+0000","last_peered":"2024-09-15T23:23:02.150556+0000","last_clean":"2024-09-15T23:23:02.150556+0000","last_became_active":"2024-09-15T23:23:02.150253+0000","last_became_peered":"2024-09-15T23:23:02.150253+0000","last_unstale":"2024-09-15T23:23:02.150556+0000","last_undegraded":"2024-09-15T23:23:02.150556+0000","last_fullsized":"2024-09-15T23:23:02.150556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:47.950356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179566+0000","last_change":"2024-09-15T23:21:52.537012+0000","last_active":"2024-09-15T23:23:02.179566+0000","last_peered":"2024-09-15T23:23:02.179566+0000","last_clean":"2024-09-15T23:23:02.179566+0000","last_became_active":"2024-09-15T23:21:52.536245+0000","last_became_peered":"2024-09-15T23:21:52.536245+0000","last_unstale":"2024-09-15T23:23:02.179566+0000","last_undegraded":"2024-09-15T23:23:02.179566+0000","last_fullsized":"2024-09-15T23:23:02.179566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:31.150491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145395+0000","last_change":"2024-09-15T23:21:52.540431+0000","last_active":"2024-09-15T23:23:02.145395+0000","last_peered":"2024-09-15T23:23:02.145395+0000","last_clean":"2024-09-15T23:23:02.145395+0000","last_became_active":"2024-09-15T23:21:52.540104+0000","last_became_peered":"2024-09-15T23:21:52.540104+0000","last_unstale":"2024-09-15T23:23:02.145395+0000","last_undegraded":"2024-09-15T23:23:02.145395+0000","last_fullsized":"2024-09-15T23:23:02.145395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.550258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147173+0000","last_change":"2024-09-15T23:21:52.537590+0000","last_active":"2024-09-15T23:23:02.147173+0000","last_peered":"2024-09-15T23:23:02.147173+0000","last_clean":"2024-09-15T23:23:02.147173+0000","last_became_active":"2024-09-15T23:21:52.533804+0000","last_became_peered":"2024-09-15T23:21:52.533804+0000","last_unstale":"2024-09-15T23:23:02.147173+0000","last_undegraded":"2024-09-15T23:23:02.147173+0000","last_fullsized":"2024-09-15T23:23:02.147173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:48.449090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148967+0000","last_change":"2024-09-15T23:21:52.536130+0000","last_active":"2024-09-15T23:23:02.148967+0000","last_peered":"2024-09-15T23:23:02.148967+0000","last_clean":"2024-09-15T23:23:02.148967+0000","last_became_active":"2024-09-15T23:21:52.535336+0000","last_became_peered":"2024-09-15T23:21:52.535336+0000","last_unstale":"2024-09-15T23:23:02.148967+0000","last_undegraded":"2024-09-15T23:23:02.148967+0000","last_fullsized":"2024-09-15T23:23:02.148967+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:52.581994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380738+0000","last_change":"2024-09-15T23:21:52.540485+0000","last_active":"2024-09-15T23:23:02.380738+0000","last_peered":"2024-09-15T23:23:02.380738+0000","last_clean":"2024-09-15T23:23:02.380738+0000","last_became_active":"2024-09-15T23:21:52.540178+0000","last_became_peered":"2024-09-15T23:21:52.540178+0000","last_unstale":"2024-09-15T23:23:02.380738+0000","last_undegraded":"2024-09-15T23:23:02.380738+0000","last_fullsized":"2024-09-15T23:23:02.380738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:40:22.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144658+0000","last_change":"2024-09-15T23:21:52.538845+0000","last_active":"2024-09-15T23:23:02.144658+0000","last_peered":"2024-09-15T23:23:02.144658+0000","last_clean":"2024-09-15T23:23:02.144658+0000","last_became_active":"2024-09-15T23:21:52.538511+0000","last_became_peered":"2024-09-15T23:21:52.538511+0000","last_unstale":"2024-09-15T23:23:02.144658+0000","last_undegraded":"2024-09-15T23:23:02.144658+0000","last_fullsized":"2024-09-15T23:23:02.144658+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:00.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149590+0000","last_change":"2024-09-15T23:23:02.149590+0000","last_active":"2024-09-15T23:23:02.149590+0000","last_peered":"2024-09-15T23:23:02.149590+0000","last_clean":"2024-09-15T23:23:02.149590+0000","last_became_active":"2024-09-15T23:23:02.149184+0000","last_became_peered":"2024-09-15T23:23:02.149184+0000","last_unstale":"2024-09-15T23:23:02.149590+0000","last_undegraded":"2024-09-15T23:23:02.149590+0000","last_fullsized":"2024-09-15T23:23:02.149590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:05:29.407175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275569+0000","last_change":"2024-09-15T23:21:52.530524+0000","last_active":"2024-09-15T23:23:02.275569+0000","last_peered":"2024-09-15T23:23:02.275569+0000","last_clean":"2024-09-15T23:23:02.275569+0000","last_became_active":"2024-09-15T23:21:52.530203+0000","last_became_peered":"2024-09-15T23:21:52.530203+0000","last_unstale":"2024-09-15T23:23:02.275569+0000","last_undegraded":"2024-09-15T23:23:02.275569+0000","last_fullsized":"2024-09-15T23:23:02.275569+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:50.560211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179469+0000","last_change":"2024-09-15T23:21:52.537256+0000","last_active":"2024-09-15T23:23:02.179469+0000","last_peered":"2024-09-15T23:23:02.179469+0000","last_clean":"2024-09-15T23:23:02.179469+0000","last_became_active":"2024-09-15T23:21:52.536731+0000","last_became_peered":"2024-09-15T23:21:52.536731+0000","last_unstale":"2024-09-15T23:23:02.179469+0000","last_undegraded":"2024-09-15T23:23:02.179469+0000","last_fullsized":"2024-09-15T23:23:02.179469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:18.815398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381161+0000","last_change":"2024-09-15T23:21:52.537728+0000","last_active":"2024-09-15T23:23:02.381161+0000","last_peered":"2024-09-15T23:23:02.381161+0000","last_clean":"2024-09-15T23:23:02.381161+0000","last_became_active":"2024-09-15T23:21:52.537434+0000","last_became_peered":"2024-09-15T23:21:52.537434+0000","last_unstale":"2024-09-15T23:23:02.381161+0000","last_undegraded":"2024-09-15T23:23:02.381161+0000","last_fullsized":"2024-09-15T23:23:02.381161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:25.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380967+0000","last_change":"2024-09-15T23:21:52.534780+0000","last_active":"2024-09-15T23:23:02.380967+0000","last_peered":"2024-09-15T23:23:02.380967+0000","last_clean":"2024-09-15T23:23:02.380967+0000","last_became_active":"2024-09-15T23:21:52.534399+0000","last_became_peered":"2024-09-15T23:21:52.534399+0000","last_unstale":"2024-09-15T23:23:02.380967+0000","last_undegraded":"2024-09-15T23:23:02.380967+0000","last_fullsized":"2024-09-15T23:23:02.380967+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:47.344129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146885+0000","last_change":"2024-09-15T23:21:52.533403+0000","last_active":"2024-09-15T23:23:02.146885+0000","last_peered":"2024-09-15T23:23:02.146885+0000","last_clean":"2024-09-15T23:23:02.146885+0000","last_became_active":"2024-09-15T23:21:52.533069+0000","last_became_peered":"2024-09-15T23:21:52.533069+0000","last_unstale":"2024-09-15T23:23:02.146885+0000","last_undegraded":"2024-09-15T23:23:02.146885+0000","last_fullsized":"2024-09-15T23:23:02.146885+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:38.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68891,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":543,"num_read_kb":344,"num_write":247,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":246033,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579937,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":978944,"data_stored":501202,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841573,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1126400,"data_stored":1150487,"data_compressed":59401,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005099,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1007616,"data_stored":505228,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234032,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":999424,"data_stored":501510,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462965,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1052672,"data_stored":1098009,"data_compressed":59401,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626490,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":978944,"data_stored":496636,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855423,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1060864,"data_stored":1093198,"data_compressed":59401,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018949,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":987136,"data_stored":501103,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247881,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":991232,"data_stored":496344,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444111,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":559962,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607636,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1044480,"data_stored":559742,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1003520,"data_stored":498092,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":64205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:23:32.079 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:23:32.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:32 smithi089 ceph-mon[30933]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:23:32.330 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:32 smithi120 ceph-mon[35250]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:23:32.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:32 smithi174 ceph-mon[35586]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:23:32.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 -- 172.21.15.89:0/3619471367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0101de0 msgr2=0x7f21d01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 --2- 172.21.15.89:0/3619471367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0101de0 0x7f21d01021c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f21c0009900 tx=0x7f21c002a410 comp rx=0 tx=0).stop 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 -- 172.21.15.89:0/3619471367 shutdown_connections 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 --2- 172.21.15.89:0/3619471367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d0106f10 0x7f21d010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 --2- 172.21.15.89:0/3619471367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21d0102790 0x7f21d01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 --2- 172.21.15.89:0/3619471367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0101de0 0x7f21d01021c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 -- 172.21.15.89:0/3619471367 >> 172.21.15.89:0/3619471367 conn(0x7f21d00fd8c0 msgr2=0x7f21d00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 -- 172.21.15.89:0/3619471367 shutdown_connections 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 -- 172.21.15.89:0/3619471367 wait complete. 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 Processor -- start 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.799+0000 7f21d5cce640 1 -- start start 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21d5cce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d0101de0 0x7f21d01984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21d5cce640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0102790 0x7f21d0198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:32.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21d5cce640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21d0106f10 0x7f21d019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21d5cce640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f21d0111f90 con 0x7f21d0102790 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21d5cce640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f21d0111e10 con 0x7f21d0101de0 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21d5cce640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f21d0112110 con 0x7f21d0106f10 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21c6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0102790 0x7f21d0198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21c6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0102790 0x7f21d0198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60566/0 (socket says 172.21.15.89:60566) 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21c6ffd640 1 -- 172.21.15.89:0/2943058676 learned_addr learned my addr 172.21.15.89:0/2943058676 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21cf7fe640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d0101de0 0x7f21d01984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21c6ffd640 1 -- 172.21.15.89:0/2943058676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21d0106f10 msgr2=0x7f21d019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21c6ffd640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21d0106f10 0x7f21d019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21c6ffd640 1 -- 172.21.15.89:0/2943058676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d0101de0 msgr2=0x7f21d01984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21c6ffd640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d0101de0 0x7f21d01984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21c6ffd640 1 -- 172.21.15.89:0/2943058676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d019d4a0 con 0x7f21d0102790 2024-09-15T23:23:32.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21cf7fe640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d0101de0 0x7f21d01984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.800+0000 7f21c6ffd640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0102790 0x7f21d0198a30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f21b800b4a0 tx=0x7f21b800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.801+0000 7f21cd7fa640 1 -- 172.21.15.89:0/2943058676 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b8014070 con 0x7f21d0102790 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.801+0000 7f21cd7fa640 1 -- 172.21.15.89:0/2943058676 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21b8010af0 con 0x7f21d0102790 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.801+0000 7f21d5cce640 1 -- 172.21.15.89:0/2943058676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d019d790 con 0x7f21d0102790 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.801+0000 7f21cd7fa640 1 -- 172.21.15.89:0/2943058676 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b8017d10 con 0x7f21d0102790 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.801+0000 7f21d5cce640 1 -- 172.21.15.89:0/2943058676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d01a5070 con 0x7f21d0102790 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.802+0000 7f21d5cce640 1 -- 172.21.15.89:0/2943058676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d01021c0 con 0x7f21d0102790 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.803+0000 7f21cd7fa640 1 -- 172.21.15.89:0/2943058676 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f21b8017070 con 0x7f21d0102790 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.804+0000 7f21cd7fa640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21bc0a32d0 0x7f21bc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.804+0000 7f21cf7fe640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21bc0a32d0 0x7f21bc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.804+0000 7f21cd7fa640 1 -- 172.21.15.89:0/2943058676 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f21b80cf580 con 0x7f21d0102790 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.805+0000 7f21cf7fe640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21bc0a32d0 0x7f21bc0a5790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f21c0000c00 tx=0x7f21c0032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:32.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.808+0000 7f21cd7fa640 1 -- 172.21.15.89:0/2943058676 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f21b809bef0 con 0x7f21d0102790 2024-09-15T23:23:32.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.968+0000 7f21d5cce640 1 -- 172.21.15.89:0/2943058676 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f21d0062630 con 0x7f21bc0a32d0 2024-09-15T23:23:32.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.976+0000 7f21cd7fa640 1 -- 172.21.15.89:0/2943058676 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340640 (secure 0 0 0) 0x7f21d0062630 con 0x7f21bc0a32d0 2024-09-15T23:23:32.981 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:23:32.983 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-15T23:23:32.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.979+0000 7f21d5cce640 1 -- 172.21.15.89:0/2943058676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21bc0a32d0 msgr2=0x7f21bc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:32.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.979+0000 7f21d5cce640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21bc0a32d0 0x7f21bc0a5790 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f21c0000c00 tx=0x7f21c0032c70 comp rx=0 tx=0).stop 2024-09-15T23:23:32.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.979+0000 7f21d5cce640 1 -- 172.21.15.89:0/2943058676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0102790 msgr2=0x7f21d0198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:32.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.979+0000 7f21d5cce640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0102790 0x7f21d0198a30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f21b800b4a0 tx=0x7f21b800b960 comp rx=0 tx=0).stop 2024-09-15T23:23:32.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.980+0000 7f21d5cce640 1 -- 172.21.15.89:0/2943058676 shutdown_connections 2024-09-15T23:23:32.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.980+0000 7f21d5cce640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21bc0a32d0 0x7f21bc0a5790 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:32.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.980+0000 7f21d5cce640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21d0106f10 0x7f21d019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:32.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.980+0000 7f21d5cce640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f21d0102790 0x7f21d0198a30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:32.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.980+0000 7f21d5cce640 1 --2- 172.21.15.89:0/2943058676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21d0101de0 0x7f21d01984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:32.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.980+0000 7f21d5cce640 1 -- 172.21.15.89:0/2943058676 >> 172.21.15.89:0/2943058676 conn(0x7f21d00fd8c0 msgr2=0x7f21d00fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:32.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.980+0000 7f21d5cce640 1 -- 172.21.15.89:0/2943058676 shutdown_connections 2024-09-15T23:23:32.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:32.980+0000 7f21d5cce640 1 -- 172.21.15.89:0/2943058676 wait complete. 2024-09-15T23:23:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:33 smithi089 ceph-mon[30933]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:23:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:33 smithi089 ceph-mon[30933]: pgmap v334: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:33 smithi120 ceph-mon[35250]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:23:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:33 smithi120 ceph-mon[35250]: pgmap v334: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:33 smithi174 ceph-mon[35586]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:23:33.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:33 smithi174 ceph-mon[35586]: pgmap v334: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:33.711 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-15T23:23:31.004639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659493,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":669,"num_read_kb":453,"num_write":430,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361332,"kb_used_data":12008,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580492,"statfs":{"total":1151940427776,"available":1151570423808,"internally_reserved":0,"allocated":12296192,"data_stored":7961513,"data_compressed":390162,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003641"},"pg_stats":[{"pgid":"2.1f","version":"86'13","reported_seq":98,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:26.214921+0000","last_change":"2024-09-15T23:21:50.513610+0000","last_active":"2024-09-15T23:23:26.214921+0000","last_peered":"2024-09-15T23:23:26.214921+0000","last_clean":"2024-09-15T23:23:26.214921+0000","last_became_active":"2024-09-15T23:21:50.513266+0000","last_became_peered":"2024-09-15T23:21:50.513266+0000","last_unstale":"2024-09-15T23:23:26.214921+0000","last_undegraded":"2024-09-15T23:23:26.214921+0000","last_fullsized":"2024-09-15T23:23:26.214921+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:16:48.692801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381072+0000","last_change":"2024-09-15T23:21:52.540248+0000","last_active":"2024-09-15T23:23:02.381072+0000","last_peered":"2024-09-15T23:23:02.381072+0000","last_clean":"2024-09-15T23:23:02.381072+0000","last_became_active":"2024-09-15T23:21:52.539945+0000","last_became_peered":"2024-09-15T23:21:52.539945+0000","last_unstale":"2024-09-15T23:23:02.381072+0000","last_undegraded":"2024-09-15T23:23:02.381072+0000","last_fullsized":"2024-09-15T23:23:02.381072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:33.402153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144420+0000","last_change":"2024-09-15T23:21:59.567246+0000","last_active":"2024-09-15T23:23:02.144420+0000","last_peered":"2024-09-15T23:23:02.144420+0000","last_clean":"2024-09-15T23:23:02.144420+0000","last_became_active":"2024-09-15T23:21:59.566832+0000","last_became_peered":"2024-09-15T23:21:59.566832+0000","last_unstale":"2024-09-15T23:23:02.144420+0000","last_undegraded":"2024-09-15T23:23:02.144420+0000","last_fullsized":"2024-09-15T23:23:02.144420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:19.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381315+0000","last_change":"2024-09-15T23:21:50.561767+0000","last_active":"2024-09-15T23:23:02.381315+0000","last_peered":"2024-09-15T23:23:02.381315+0000","last_clean":"2024-09-15T23:23:02.381315+0000","last_became_active":"2024-09-15T23:21:50.561385+0000","last_became_peered":"2024-09-15T23:21:50.561385+0000","last_unstale":"2024-09-15T23:23:02.381315+0000","last_undegraded":"2024-09-15T23:23:02.381315+0000","last_fullsized":"2024-09-15T23:23:02.381315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:11.897884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381058+0000","last_change":"2024-09-15T23:21:52.534874+0000","last_active":"2024-09-15T23:23:02.381058+0000","last_peered":"2024-09-15T23:23:02.381058+0000","last_clean":"2024-09-15T23:23:02.381058+0000","last_became_active":"2024-09-15T23:21:52.534334+0000","last_became_peered":"2024-09-15T23:21:52.534334+0000","last_unstale":"2024-09-15T23:23:02.381058+0000","last_undegraded":"2024-09-15T23:23:02.381058+0000","last_fullsized":"2024-09-15T23:23:02.381058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:15.972648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.178766+0000","last_change":"2024-09-15T23:21:59.559907+0000","last_active":"2024-09-15T23:23:02.178766+0000","last_peered":"2024-09-15T23:23:02.178766+0000","last_clean":"2024-09-15T23:23:02.178766+0000","last_became_active":"2024-09-15T23:21:59.559440+0000","last_became_peered":"2024-09-15T23:21:59.559440+0000","last_unstale":"2024-09-15T23:23:02.178766+0000","last_undegraded":"2024-09-15T23:23:02.178766+0000","last_fullsized":"2024-09-15T23:23:02.178766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:00.755112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275999+0000","last_change":"2024-09-15T23:21:50.515424+0000","last_active":"2024-09-15T23:23:02.275999+0000","last_peered":"2024-09-15T23:23:02.275999+0000","last_clean":"2024-09-15T23:23:02.275999+0000","last_became_active":"2024-09-15T23:21:50.514966+0000","last_became_peered":"2024-09-15T23:21:50.514966+0000","last_unstale":"2024-09-15T23:23:02.275999+0000","last_undegraded":"2024-09-15T23:23:02.275999+0000","last_fullsized":"2024-09-15T23:23:02.275999+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:57.586575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148438+0000","last_change":"2024-09-15T23:21:52.535189+0000","last_active":"2024-09-15T23:23:02.148438+0000","last_peered":"2024-09-15T23:23:02.148438+0000","last_clean":"2024-09-15T23:23:02.148438+0000","last_became_active":"2024-09-15T23:21:52.534807+0000","last_became_peered":"2024-09-15T23:21:52.534807+0000","last_unstale":"2024-09-15T23:23:02.148438+0000","last_undegraded":"2024-09-15T23:23:02.148438+0000","last_fullsized":"2024-09-15T23:23:02.148438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:30.654382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179082+0000","last_change":"2024-09-15T23:21:59.564675+0000","last_active":"2024-09-15T23:23:02.179082+0000","last_peered":"2024-09-15T23:23:02.179082+0000","last_clean":"2024-09-15T23:23:02.179082+0000","last_became_active":"2024-09-15T23:21:59.564271+0000","last_became_peered":"2024-09-15T23:21:59.564271+0000","last_unstale":"2024-09-15T23:23:02.179082+0000","last_undegraded":"2024-09-15T23:23:02.179082+0000","last_fullsized":"2024-09-15T23:23:02.179082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:01.055369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.276101+0000","last_change":"2024-09-15T23:21:50.507790+0000","last_active":"2024-09-15T23:23:02.276101+0000","last_peered":"2024-09-15T23:23:02.276101+0000","last_clean":"2024-09-15T23:23:02.276101+0000","last_became_active":"2024-09-15T23:21:50.507398+0000","last_became_peered":"2024-09-15T23:21:50.507398+0000","last_unstale":"2024-09-15T23:23:02.276101+0000","last_undegraded":"2024-09-15T23:23:02.276101+0000","last_fullsized":"2024-09-15T23:23:02.276101+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:58:00.001279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148799+0000","last_change":"2024-09-15T23:21:52.537179+0000","last_active":"2024-09-15T23:23:02.148799+0000","last_peered":"2024-09-15T23:23:02.148799+0000","last_clean":"2024-09-15T23:23:02.148799+0000","last_became_active":"2024-09-15T23:21:52.536841+0000","last_became_peered":"2024-09-15T23:21:52.536841+0000","last_unstale":"2024-09-15T23:23:02.148799+0000","last_undegraded":"2024-09-15T23:23:02.148799+0000","last_fullsized":"2024-09-15T23:23:02.148799+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:47.455700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.178914+0000","last_change":"2024-09-15T23:21:59.564734+0000","last_active":"2024-09-15T23:23:02.178914+0000","last_peered":"2024-09-15T23:23:02.178914+0000","last_clean":"2024-09-15T23:23:02.178914+0000","last_became_active":"2024-09-15T23:21:59.564339+0000","last_became_peered":"2024-09-15T23:21:59.564339+0000","last_unstale":"2024-09-15T23:23:02.178914+0000","last_undegraded":"2024-09-15T23:23:02.178914+0000","last_fullsized":"2024-09-15T23:23:02.178914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:44.498480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144977+0000","last_change":"2024-09-15T23:21:52.538025+0000","last_active":"2024-09-15T23:23:02.144977+0000","last_peered":"2024-09-15T23:23:02.144977+0000","last_clean":"2024-09-15T23:23:02.144977+0000","last_became_active":"2024-09-15T23:21:52.537736+0000","last_became_peered":"2024-09-15T23:21:52.537736+0000","last_unstale":"2024-09-15T23:23:02.144977+0000","last_undegraded":"2024-09-15T23:23:02.144977+0000","last_fullsized":"2024-09-15T23:23:02.144977+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:31.916888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275698+0000","last_change":"2024-09-15T23:21:50.516024+0000","last_active":"2024-09-15T23:23:02.275698+0000","last_peered":"2024-09-15T23:23:02.275698+0000","last_clean":"2024-09-15T23:23:02.275698+0000","last_became_active":"2024-09-15T23:21:50.515590+0000","last_became_peered":"2024-09-15T23:21:50.515590+0000","last_unstale":"2024-09-15T23:23:02.275698+0000","last_undegraded":"2024-09-15T23:23:02.275698+0000","last_fullsized":"2024-09-15T23:23:02.275698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:47:38.849366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179081+0000","last_change":"2024-09-15T23:21:59.681971+0000","last_active":"2024-09-15T23:23:02.179081+0000","last_peered":"2024-09-15T23:23:02.179081+0000","last_clean":"2024-09-15T23:23:02.179081+0000","last_became_active":"2024-09-15T23:21:59.681432+0000","last_became_peered":"2024-09-15T23:21:59.681432+0000","last_unstale":"2024-09-15T23:23:02.179081+0000","last_undegraded":"2024-09-15T23:23:02.179081+0000","last_fullsized":"2024-09-15T23:23:02.179081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:26.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381423+0000","last_change":"2024-09-15T23:21:52.541281+0000","last_active":"2024-09-15T23:23:02.381423+0000","last_peered":"2024-09-15T23:23:02.381423+0000","last_clean":"2024-09-15T23:23:02.381423+0000","last_became_active":"2024-09-15T23:21:52.540772+0000","last_became_peered":"2024-09-15T23:21:52.540772+0000","last_unstale":"2024-09-15T23:23:02.381423+0000","last_undegraded":"2024-09-15T23:23:02.381423+0000","last_fullsized":"2024-09-15T23:23:02.381423+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:23:44.444904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144530+0000","last_change":"2024-09-15T23:21:50.508844+0000","last_active":"2024-09-15T23:23:02.144530+0000","last_peered":"2024-09-15T23:23:02.144530+0000","last_clean":"2024-09-15T23:23:02.144530+0000","last_became_active":"2024-09-15T23:21:50.508364+0000","last_became_peered":"2024-09-15T23:21:50.508364+0000","last_unstale":"2024-09-15T23:23:02.144530+0000","last_undegraded":"2024-09-15T23:23:02.144530+0000","last_fullsized":"2024-09-15T23:23:02.144530+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:45.129711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179363+0000","last_change":"2024-09-15T23:21:59.562114+0000","last_active":"2024-09-15T23:23:02.179363+0000","last_peered":"2024-09-15T23:23:02.179363+0000","last_clean":"2024-09-15T23:23:02.179363+0000","last_became_active":"2024-09-15T23:21:59.561652+0000","last_became_peered":"2024-09-15T23:21:59.561652+0000","last_unstale":"2024-09-15T23:23:02.179363+0000","last_undegraded":"2024-09-15T23:23:02.179363+0000","last_fullsized":"2024-09-15T23:23:02.179363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:51.781886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146548+0000","last_change":"2024-09-15T23:21:52.531678+0000","last_active":"2024-09-15T23:23:02.146548+0000","last_peered":"2024-09-15T23:23:02.146548+0000","last_clean":"2024-09-15T23:23:02.146548+0000","last_became_active":"2024-09-15T23:21:52.531327+0000","last_became_peered":"2024-09-15T23:21:52.531327+0000","last_unstale":"2024-09-15T23:23:02.146548+0000","last_undegraded":"2024-09-15T23:23:02.146548+0000","last_fullsized":"2024-09-15T23:23:02.146548+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:07.152626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381782+0000","last_change":"2024-09-15T23:21:50.513197+0000","last_active":"2024-09-15T23:23:02.381782+0000","last_peered":"2024-09-15T23:23:02.381782+0000","last_clean":"2024-09-15T23:23:02.381782+0000","last_became_active":"2024-09-15T23:21:50.512794+0000","last_became_peered":"2024-09-15T23:21:50.512794+0000","last_unstale":"2024-09-15T23:23:02.381782+0000","last_undegraded":"2024-09-15T23:23:02.381782+0000","last_fullsized":"2024-09-15T23:23:02.381782+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:05.457467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143792+0000","last_change":"2024-09-15T23:21:59.565669+0000","last_active":"2024-09-15T23:23:02.143792+0000","last_peered":"2024-09-15T23:23:02.143792+0000","last_clean":"2024-09-15T23:23:02.143792+0000","last_became_active":"2024-09-15T23:21:59.565290+0000","last_became_peered":"2024-09-15T23:21:59.565290+0000","last_unstale":"2024-09-15T23:23:02.143792+0000","last_undegraded":"2024-09-15T23:23:02.143792+0000","last_fullsized":"2024-09-15T23:23:02.143792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:04.861560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275448+0000","last_change":"2024-09-15T23:21:52.529643+0000","last_active":"2024-09-15T23:23:02.275448+0000","last_peered":"2024-09-15T23:23:02.275448+0000","last_clean":"2024-09-15T23:23:02.275448+0000","last_became_active":"2024-09-15T23:21:52.529304+0000","last_became_peered":"2024-09-15T23:21:52.529304+0000","last_unstale":"2024-09-15T23:23:02.275448+0000","last_undegraded":"2024-09-15T23:23:02.275448+0000","last_fullsized":"2024-09-15T23:23:02.275448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:18:37.758403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148047+0000","last_change":"2024-09-15T23:21:50.562114+0000","last_active":"2024-09-15T23:23:02.148047+0000","last_peered":"2024-09-15T23:23:02.148047+0000","last_clean":"2024-09-15T23:23:02.148047+0000","last_became_active":"2024-09-15T23:21:50.561772+0000","last_became_peered":"2024-09-15T23:21:50.561772+0000","last_unstale":"2024-09-15T23:23:02.148047+0000","last_undegraded":"2024-09-15T23:23:02.148047+0000","last_fullsized":"2024-09-15T23:23:02.148047+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:14.806646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179172+0000","last_change":"2024-09-15T23:21:59.681933+0000","last_active":"2024-09-15T23:23:02.179172+0000","last_peered":"2024-09-15T23:23:02.179172+0000","last_clean":"2024-09-15T23:23:02.179172+0000","last_became_active":"2024-09-15T23:21:59.681528+0000","last_became_peered":"2024-09-15T23:21:59.681528+0000","last_unstale":"2024-09-15T23:23:02.179172+0000","last_undegraded":"2024-09-15T23:23:02.179172+0000","last_fullsized":"2024-09-15T23:23:02.179172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:46.199540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381069+0000","last_change":"2024-09-15T23:21:52.529548+0000","last_active":"2024-09-15T23:23:02.381069+0000","last_peered":"2024-09-15T23:23:02.381069+0000","last_clean":"2024-09-15T23:23:02.381069+0000","last_became_active":"2024-09-15T23:21:52.529253+0000","last_became_peered":"2024-09-15T23:21:52.529253+0000","last_unstale":"2024-09-15T23:23:02.381069+0000","last_undegraded":"2024-09-15T23:23:02.381069+0000","last_fullsized":"2024-09-15T23:23:02.381069+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:22.813395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179650+0000","last_change":"2024-09-15T23:21:50.767759+0000","last_active":"2024-09-15T23:23:02.179650+0000","last_peered":"2024-09-15T23:23:02.179650+0000","last_clean":"2024-09-15T23:23:02.179650+0000","last_became_active":"2024-09-15T23:21:50.767388+0000","last_became_peered":"2024-09-15T23:21:50.767388+0000","last_unstale":"2024-09-15T23:23:02.179650+0000","last_undegraded":"2024-09-15T23:23:02.179650+0000","last_fullsized":"2024-09-15T23:23:02.179650+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:36.713769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144546+0000","last_change":"2024-09-15T23:21:59.557951+0000","last_active":"2024-09-15T23:23:02.144546+0000","last_peered":"2024-09-15T23:23:02.144546+0000","last_clean":"2024-09-15T23:23:02.144546+0000","last_became_active":"2024-09-15T23:21:59.557490+0000","last_became_peered":"2024-09-15T23:21:59.557490+0000","last_unstale":"2024-09-15T23:23:02.144546+0000","last_undegraded":"2024-09-15T23:23:02.144546+0000","last_fullsized":"2024-09-15T23:23:02.144546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:17.147389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381781+0000","last_change":"2024-09-15T23:21:52.531638+0000","last_active":"2024-09-15T23:23:02.381781+0000","last_peered":"2024-09-15T23:23:02.381781+0000","last_clean":"2024-09-15T23:23:02.381781+0000","last_became_active":"2024-09-15T23:21:52.531301+0000","last_became_peered":"2024-09-15T23:21:52.531301+0000","last_unstale":"2024-09-15T23:23:02.381781+0000","last_undegraded":"2024-09-15T23:23:02.381781+0000","last_fullsized":"2024-09-15T23:23:02.381781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.436286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'15","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143896+0000","last_change":"2024-09-15T23:21:50.508470+0000","last_active":"2024-09-15T23:23:02.143896+0000","last_peered":"2024-09-15T23:23:02.143896+0000","last_clean":"2024-09-15T23:23:02.143896+0000","last_became_active":"2024-09-15T23:21:50.508060+0000","last_became_peered":"2024-09-15T23:21:50.508060+0000","last_unstale":"2024-09-15T23:23:02.143896+0000","last_undegraded":"2024-09-15T23:23:02.143896+0000","last_fullsized":"2024-09-15T23:23:02.143896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:59.542252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381349+0000","last_change":"2024-09-15T23:21:59.682019+0000","last_active":"2024-09-15T23:23:02.381349+0000","last_peered":"2024-09-15T23:23:02.381349+0000","last_clean":"2024-09-15T23:23:02.381349+0000","last_became_active":"2024-09-15T23:21:59.681467+0000","last_became_peered":"2024-09-15T23:21:59.681467+0000","last_unstale":"2024-09-15T23:23:02.381349+0000","last_undegraded":"2024-09-15T23:23:02.381349+0000","last_fullsized":"2024-09-15T23:23:02.381349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:27.650382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148067+0000","last_change":"2024-09-15T23:21:52.536345+0000","last_active":"2024-09-15T23:23:02.148067+0000","last_peered":"2024-09-15T23:23:02.148067+0000","last_clean":"2024-09-15T23:23:02.148067+0000","last_became_active":"2024-09-15T23:21:52.536027+0000","last_became_peered":"2024-09-15T23:21:52.536027+0000","last_unstale":"2024-09-15T23:23:02.148067+0000","last_undegraded":"2024-09-15T23:23:02.148067+0000","last_fullsized":"2024-09-15T23:23:02.148067+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:16.435912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.276164+0000","last_change":"2024-09-15T23:21:50.562945+0000","last_active":"2024-09-15T23:23:02.276164+0000","last_peered":"2024-09-15T23:23:02.276164+0000","last_clean":"2024-09-15T23:23:02.276164+0000","last_became_active":"2024-09-15T23:21:50.561632+0000","last_became_peered":"2024-09-15T23:21:50.561632+0000","last_unstale":"2024-09-15T23:23:02.276164+0000","last_undegraded":"2024-09-15T23:23:02.276164+0000","last_fullsized":"2024-09-15T23:23:02.276164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:44.545489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146498+0000","last_change":"2024-09-15T23:21:59.564793+0000","last_active":"2024-09-15T23:23:02.146498+0000","last_peered":"2024-09-15T23:23:02.146498+0000","last_clean":"2024-09-15T23:23:02.146498+0000","last_became_active":"2024-09-15T23:21:59.564140+0000","last_became_peered":"2024-09-15T23:21:59.564140+0000","last_unstale":"2024-09-15T23:23:02.146498+0000","last_undegraded":"2024-09-15T23:23:02.146498+0000","last_fullsized":"2024-09-15T23:23:02.146498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:41.464962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146857+0000","last_change":"2024-09-15T23:21:52.533521+0000","last_active":"2024-09-15T23:23:02.146857+0000","last_peered":"2024-09-15T23:23:02.146857+0000","last_clean":"2024-09-15T23:23:02.146857+0000","last_became_active":"2024-09-15T23:21:52.532195+0000","last_became_peered":"2024-09-15T23:21:52.532195+0000","last_unstale":"2024-09-15T23:23:02.146857+0000","last_undegraded":"2024-09-15T23:23:02.146857+0000","last_fullsized":"2024-09-15T23:23:02.146857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:14.229018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143658+0000","last_change":"2024-09-15T23:21:50.562893+0000","last_active":"2024-09-15T23:23:02.143658+0000","last_peered":"2024-09-15T23:23:02.143658+0000","last_clean":"2024-09-15T23:23:02.143658+0000","last_became_active":"2024-09-15T23:21:50.562516+0000","last_became_peered":"2024-09-15T23:21:50.562516+0000","last_unstale":"2024-09-15T23:23:02.143658+0000","last_undegraded":"2024-09-15T23:23:02.143658+0000","last_fullsized":"2024-09-15T23:23:02.143658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:47:47.442458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275687+0000","last_change":"2024-09-15T23:21:59.682091+0000","last_active":"2024-09-15T23:23:02.275687+0000","last_peered":"2024-09-15T23:23:02.275687+0000","last_clean":"2024-09-15T23:23:02.275687+0000","last_became_active":"2024-09-15T23:21:59.681574+0000","last_became_peered":"2024-09-15T23:21:59.681574+0000","last_unstale":"2024-09-15T23:23:02.275687+0000","last_undegraded":"2024-09-15T23:23:02.275687+0000","last_fullsized":"2024-09-15T23:23:02.275687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:04.991608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146611+0000","last_change":"2024-09-15T23:21:52.533811+0000","last_active":"2024-09-15T23:23:02.146611+0000","last_peered":"2024-09-15T23:23:02.146611+0000","last_clean":"2024-09-15T23:23:02.146611+0000","last_became_active":"2024-09-15T23:21:52.533487+0000","last_became_peered":"2024-09-15T23:21:52.533487+0000","last_unstale":"2024-09-15T23:23:02.146611+0000","last_undegraded":"2024-09-15T23:23:02.146611+0000","last_fullsized":"2024-09-15T23:23:02.146611+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:44.193212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:25.214164+0000","last_change":"2024-09-15T23:21:50.510883+0000","last_active":"2024-09-15T23:23:25.214164+0000","last_peered":"2024-09-15T23:23:25.214164+0000","last_clean":"2024-09-15T23:23:25.214164+0000","last_became_active":"2024-09-15T23:21:50.510478+0000","last_became_peered":"2024-09-15T23:21:50.510478+0000","last_unstale":"2024-09-15T23:23:25.214164+0000","last_undegraded":"2024-09-15T23:23:25.214164+0000","last_fullsized":"2024-09-15T23:23:25.214164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:35.454885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147693+0000","last_change":"2024-09-15T23:21:59.561816+0000","last_active":"2024-09-15T23:23:02.147693+0000","last_peered":"2024-09-15T23:23:02.147693+0000","last_clean":"2024-09-15T23:23:02.147693+0000","last_became_active":"2024-09-15T23:21:59.561434+0000","last_became_peered":"2024-09-15T23:21:59.561434+0000","last_unstale":"2024-09-15T23:23:02.147693+0000","last_undegraded":"2024-09-15T23:23:02.147693+0000","last_fullsized":"2024-09-15T23:23:02.147693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:59.945802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.277204+0000","last_change":"2024-09-15T23:23:02.277204+0000","last_active":"2024-09-15T23:23:02.277204+0000","last_peered":"2024-09-15T23:23:02.277204+0000","last_clean":"2024-09-15T23:23:02.277204+0000","last_became_active":"2024-09-15T23:23:02.276770+0000","last_became_peered":"2024-09-15T23:23:02.276770+0000","last_unstale":"2024-09-15T23:23:02.277204+0000","last_undegraded":"2024-09-15T23:23:02.277204+0000","last_fullsized":"2024-09-15T23:23:02.277204+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:11.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:24.207917+0000","last_change":"2024-09-15T23:21:50.561684+0000","last_active":"2024-09-15T23:23:24.207917+0000","last_peered":"2024-09-15T23:23:24.207917+0000","last_clean":"2024-09-15T23:23:24.207917+0000","last_became_active":"2024-09-15T23:21:50.561335+0000","last_became_peered":"2024-09-15T23:21:50.561335+0000","last_unstale":"2024-09-15T23:23:24.207917+0000","last_undegraded":"2024-09-15T23:23:24.207917+0000","last_fullsized":"2024-09-15T23:23:24.207917+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:23.897250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148009+0000","last_change":"2024-09-15T23:21:59.681858+0000","last_active":"2024-09-15T23:23:02.148009+0000","last_peered":"2024-09-15T23:23:02.148009+0000","last_clean":"2024-09-15T23:23:02.148009+0000","last_became_active":"2024-09-15T23:21:59.681441+0000","last_became_peered":"2024-09-15T23:21:59.681441+0000","last_unstale":"2024-09-15T23:23:02.148009+0000","last_undegraded":"2024-09-15T23:23:02.148009+0000","last_fullsized":"2024-09-15T23:23:02.148009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:28.513719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145851+0000","last_change":"2024-09-15T23:21:52.536141+0000","last_active":"2024-09-15T23:23:02.145851+0000","last_peered":"2024-09-15T23:23:02.145851+0000","last_clean":"2024-09-15T23:23:02.145851+0000","last_became_active":"2024-09-15T23:21:52.535832+0000","last_became_peered":"2024-09-15T23:21:52.535832+0000","last_unstale":"2024-09-15T23:23:02.145851+0000","last_undegraded":"2024-09-15T23:23:02.145851+0000","last_fullsized":"2024-09-15T23:23:02.145851+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:21:44.341412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144659+0000","last_change":"2024-09-15T23:21:50.514689+0000","last_active":"2024-09-15T23:23:02.144659+0000","last_peered":"2024-09-15T23:23:02.144659+0000","last_clean":"2024-09-15T23:23:02.144659+0000","last_became_active":"2024-09-15T23:21:50.513261+0000","last_became_peered":"2024-09-15T23:21:50.513261+0000","last_unstale":"2024-09-15T23:23:02.144659+0000","last_undegraded":"2024-09-15T23:23:02.144659+0000","last_fullsized":"2024-09-15T23:23:02.144659+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:30:30.132991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149120+0000","last_change":"2024-09-15T23:21:59.682118+0000","last_active":"2024-09-15T23:23:02.149120+0000","last_peered":"2024-09-15T23:23:02.149120+0000","last_clean":"2024-09-15T23:23:02.149120+0000","last_became_active":"2024-09-15T23:21:59.681500+0000","last_became_peered":"2024-09-15T23:21:59.681500+0000","last_unstale":"2024-09-15T23:23:02.149120+0000","last_undegraded":"2024-09-15T23:23:02.149120+0000","last_fullsized":"2024-09-15T23:23:02.149120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:58.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381664+0000","last_change":"2024-09-15T23:21:52.540458+0000","last_active":"2024-09-15T23:23:02.381664+0000","last_peered":"2024-09-15T23:23:02.381664+0000","last_clean":"2024-09-15T23:23:02.381664+0000","last_became_active":"2024-09-15T23:21:52.540112+0000","last_became_peered":"2024-09-15T23:21:52.540112+0000","last_unstale":"2024-09-15T23:23:02.381664+0000","last_undegraded":"2024-09-15T23:23:02.381664+0000","last_fullsized":"2024-09-15T23:23:02.381664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:15.240785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179608+0000","last_change":"2024-09-15T23:21:50.513124+0000","last_active":"2024-09-15T23:23:02.179608+0000","last_peered":"2024-09-15T23:23:02.179608+0000","last_clean":"2024-09-15T23:23:02.179608+0000","last_became_active":"2024-09-15T23:21:50.512705+0000","last_became_peered":"2024-09-15T23:21:50.512705+0000","last_unstale":"2024-09-15T23:23:02.179608+0000","last_undegraded":"2024-09-15T23:23:02.179608+0000","last_fullsized":"2024-09-15T23:23:02.179608+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.417937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":350,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144804+0000","last_change":"2024-09-15T23:20:40.656652+0000","last_active":"2024-09-15T23:23:02.144804+0000","last_peered":"2024-09-15T23:23:02.144804+0000","last_clean":"2024-09-15T23:23:02.144804+0000","last_became_active":"2024-09-15T23:20:40.656266+0000","last_became_peered":"2024-09-15T23:20:40.656266+0000","last_unstale":"2024-09-15T23:23:02.144804+0000","last_undegraded":"2024-09-15T23:23:02.144804+0000","last_fullsized":"2024-09-15T23:23:02.144804+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_clean_scrub_stamp":"2024-09-15T23:17:15.186850+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:42.317006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145235+0000","last_change":"2024-09-15T23:21:59.563777+0000","last_active":"2024-09-15T23:23:02.145235+0000","last_peered":"2024-09-15T23:23:02.145235+0000","last_clean":"2024-09-15T23:23:02.145235+0000","last_became_active":"2024-09-15T23:21:59.563320+0000","last_became_peered":"2024-09-15T23:21:59.563320+0000","last_unstale":"2024-09-15T23:23:02.145235+0000","last_undegraded":"2024-09-15T23:23:02.145235+0000","last_fullsized":"2024-09-15T23:23:02.145235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:25:07.791923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145457+0000","last_change":"2024-09-15T23:21:52.539842+0000","last_active":"2024-09-15T23:23:02.145457+0000","last_peered":"2024-09-15T23:23:02.145457+0000","last_clean":"2024-09-15T23:23:02.145457+0000","last_became_active":"2024-09-15T23:21:52.538683+0000","last_became_peered":"2024-09-15T23:21:52.538683+0000","last_unstale":"2024-09-15T23:23:02.145457+0000","last_undegraded":"2024-09-15T23:23:02.145457+0000","last_fullsized":"2024-09-15T23:23:02.145457+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:43.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:28.214060+0000","last_change":"2024-09-15T23:21:50.767601+0000","last_active":"2024-09-15T23:23:28.214060+0000","last_peered":"2024-09-15T23:23:28.214060+0000","last_clean":"2024-09-15T23:23:28.214060+0000","last_became_active":"2024-09-15T23:21:50.767149+0000","last_became_peered":"2024-09-15T23:21:50.767149+0000","last_unstale":"2024-09-15T23:23:28.214060+0000","last_undegraded":"2024-09-15T23:23:28.214060+0000","last_fullsized":"2024-09-15T23:23:28.214060+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:23.636551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148132+0000","last_change":"2024-09-15T23:21:59.563069+0000","last_active":"2024-09-15T23:23:02.148132+0000","last_peered":"2024-09-15T23:23:02.148132+0000","last_clean":"2024-09-15T23:23:02.148132+0000","last_became_active":"2024-09-15T23:21:59.562658+0000","last_became_peered":"2024-09-15T23:21:59.562658+0000","last_unstale":"2024-09-15T23:23:02.148132+0000","last_undegraded":"2024-09-15T23:23:02.148132+0000","last_fullsized":"2024-09-15T23:23:02.148132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:23.067198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145096+0000","last_change":"2024-09-15T23:21:52.535289+0000","last_active":"2024-09-15T23:23:02.145096+0000","last_peered":"2024-09-15T23:23:02.145096+0000","last_clean":"2024-09-15T23:23:02.145096+0000","last_became_active":"2024-09-15T23:21:52.534885+0000","last_became_peered":"2024-09-15T23:21:52.534885+0000","last_unstale":"2024-09-15T23:23:02.145096+0000","last_undegraded":"2024-09-15T23:23:02.145096+0000","last_fullsized":"2024-09-15T23:23:02.145096+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:42.293786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149082+0000","last_change":"2024-09-15T23:21:50.513246+0000","last_active":"2024-09-15T23:23:02.149082+0000","last_peered":"2024-09-15T23:23:02.149082+0000","last_clean":"2024-09-15T23:23:02.149082+0000","last_became_active":"2024-09-15T23:21:50.512740+0000","last_became_peered":"2024-09-15T23:21:50.512740+0000","last_unstale":"2024-09-15T23:23:02.149082+0000","last_undegraded":"2024-09-15T23:23:02.149082+0000","last_fullsized":"2024-09-15T23:23:02.149082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:59.201970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381169+0000","last_change":"2024-09-15T23:21:59.568565+0000","last_active":"2024-09-15T23:23:02.381169+0000","last_peered":"2024-09-15T23:23:02.381169+0000","last_clean":"2024-09-15T23:23:02.381169+0000","last_became_active":"2024-09-15T23:21:59.567789+0000","last_became_peered":"2024-09-15T23:21:59.567789+0000","last_unstale":"2024-09-15T23:23:02.381169+0000","last_undegraded":"2024-09-15T23:23:02.381169+0000","last_fullsized":"2024-09-15T23:23:02.381169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:02.106971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145412+0000","last_change":"2024-09-15T23:21:52.533727+0000","last_active":"2024-09-15T23:23:02.145412+0000","last_peered":"2024-09-15T23:23:02.145412+0000","last_clean":"2024-09-15T23:23:02.145412+0000","last_became_active":"2024-09-15T23:21:52.532350+0000","last_became_peered":"2024-09-15T23:21:52.532350+0000","last_unstale":"2024-09-15T23:23:02.145412+0000","last_undegraded":"2024-09-15T23:23:02.145412+0000","last_fullsized":"2024-09-15T23:23:02.145412+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:03.624837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148178+0000","last_change":"2024-09-15T23:21:50.509519+0000","last_active":"2024-09-15T23:23:02.148178+0000","last_peered":"2024-09-15T23:23:02.148178+0000","last_clean":"2024-09-15T23:23:02.148178+0000","last_became_active":"2024-09-15T23:21:50.509224+0000","last_became_peered":"2024-09-15T23:21:50.509224+0000","last_unstale":"2024-09-15T23:23:02.148178+0000","last_undegraded":"2024-09-15T23:23:02.148178+0000","last_fullsized":"2024-09-15T23:23:02.148178+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:39:57.526821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147867+0000","last_change":"2024-09-15T23:21:59.566429+0000","last_active":"2024-09-15T23:23:02.147867+0000","last_peered":"2024-09-15T23:23:02.147867+0000","last_clean":"2024-09-15T23:23:02.147867+0000","last_became_active":"2024-09-15T23:21:59.565835+0000","last_became_peered":"2024-09-15T23:21:59.565835+0000","last_unstale":"2024-09-15T23:23:02.147867+0000","last_undegraded":"2024-09-15T23:23:02.147867+0000","last_fullsized":"2024-09-15T23:23:02.147867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:38.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146720+0000","last_change":"2024-09-15T23:21:52.539544+0000","last_active":"2024-09-15T23:23:02.146720+0000","last_peered":"2024-09-15T23:23:02.146720+0000","last_clean":"2024-09-15T23:23:02.146720+0000","last_became_active":"2024-09-15T23:21:52.539203+0000","last_became_peered":"2024-09-15T23:21:52.539203+0000","last_unstale":"2024-09-15T23:23:02.146720+0000","last_undegraded":"2024-09-15T23:23:02.146720+0000","last_fullsized":"2024-09-15T23:23:02.146720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.193965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.276035+0000","last_change":"2024-09-15T23:21:50.513543+0000","last_active":"2024-09-15T23:23:02.276035+0000","last_peered":"2024-09-15T23:23:02.276035+0000","last_clean":"2024-09-15T23:23:02.276035+0000","last_became_active":"2024-09-15T23:21:50.512812+0000","last_became_peered":"2024-09-15T23:21:50.512812+0000","last_unstale":"2024-09-15T23:23:02.276035+0000","last_undegraded":"2024-09-15T23:23:02.276035+0000","last_fullsized":"2024-09-15T23:23:02.276035+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:50.954096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148065+0000","last_change":"2024-09-15T23:21:59.570163+0000","last_active":"2024-09-15T23:23:02.148065+0000","last_peered":"2024-09-15T23:23:02.148065+0000","last_clean":"2024-09-15T23:23:02.148065+0000","last_became_active":"2024-09-15T23:21:59.569805+0000","last_became_peered":"2024-09-15T23:21:59.569805+0000","last_unstale":"2024-09-15T23:23:02.148065+0000","last_undegraded":"2024-09-15T23:23:02.148065+0000","last_fullsized":"2024-09-15T23:23:02.148065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:28.530817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148358+0000","last_change":"2024-09-15T23:21:52.537850+0000","last_active":"2024-09-15T23:23:02.148358+0000","last_peered":"2024-09-15T23:23:02.148358+0000","last_clean":"2024-09-15T23:23:02.148358+0000","last_became_active":"2024-09-15T23:21:52.536579+0000","last_became_peered":"2024-09-15T23:21:52.536579+0000","last_unstale":"2024-09-15T23:23:02.148358+0000","last_undegraded":"2024-09-15T23:23:02.148358+0000","last_fullsized":"2024-09-15T23:23:02.148358+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:06.353615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275622+0000","last_change":"2024-09-15T23:21:50.562221+0000","last_active":"2024-09-15T23:23:02.275622+0000","last_peered":"2024-09-15T23:23:02.275622+0000","last_clean":"2024-09-15T23:23:02.275622+0000","last_became_active":"2024-09-15T23:21:50.561788+0000","last_became_peered":"2024-09-15T23:21:50.561788+0000","last_unstale":"2024-09-15T23:23:02.275622+0000","last_undegraded":"2024-09-15T23:23:02.275622+0000","last_fullsized":"2024-09-15T23:23:02.275622+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.853812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.383003+0000","last_change":"2024-09-15T23:23:02.383003+0000","last_active":"2024-09-15T23:23:02.383003+0000","last_peered":"2024-09-15T23:23:02.383003+0000","last_clean":"2024-09-15T23:23:02.383003+0000","last_became_active":"2024-09-15T23:23:02.382629+0000","last_became_peered":"2024-09-15T23:23:02.382629+0000","last_unstale":"2024-09-15T23:23:02.383003+0000","last_undegraded":"2024-09-15T23:23:02.383003+0000","last_fullsized":"2024-09-15T23:23:02.383003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:50.114689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144569+0000","last_change":"2024-09-15T23:21:52.539566+0000","last_active":"2024-09-15T23:23:02.144569+0000","last_peered":"2024-09-15T23:23:02.144569+0000","last_clean":"2024-09-15T23:23:02.144569+0000","last_became_active":"2024-09-15T23:21:52.539219+0000","last_became_peered":"2024-09-15T23:21:52.539219+0000","last_unstale":"2024-09-15T23:23:02.144569+0000","last_undegraded":"2024-09-15T23:23:02.144569+0000","last_fullsized":"2024-09-15T23:23:02.144569+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:41.266308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:29.215228+0000","last_change":"2024-09-15T23:21:50.516650+0000","last_active":"2024-09-15T23:23:29.215228+0000","last_peered":"2024-09-15T23:23:29.215228+0000","last_clean":"2024-09-15T23:23:29.215228+0000","last_became_active":"2024-09-15T23:21:50.516297+0000","last_became_peered":"2024-09-15T23:21:50.516297+0000","last_unstale":"2024-09-15T23:23:29.215228+0000","last_undegraded":"2024-09-15T23:23:29.215228+0000","last_fullsized":"2024-09-15T23:23:29.215228+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:16:36.928701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":981,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143833+0000","last_change":"2024-09-15T23:21:59.568087+0000","last_active":"2024-09-15T23:23:02.143833+0000","last_peered":"2024-09-15T23:23:02.143833+0000","last_clean":"2024-09-15T23:23:02.143833+0000","last_became_active":"2024-09-15T23:21:59.567693+0000","last_became_peered":"2024-09-15T23:21:59.567693+0000","last_unstale":"2024-09-15T23:23:02.143833+0000","last_undegraded":"2024-09-15T23:23:02.143833+0000","last_fullsized":"2024-09-15T23:23:02.143833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.231873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144909+0000","last_change":"2024-09-15T23:21:52.531090+0000","last_active":"2024-09-15T23:23:02.144909+0000","last_peered":"2024-09-15T23:23:02.144909+0000","last_clean":"2024-09-15T23:23:02.144909+0000","last_became_active":"2024-09-15T23:21:52.530738+0000","last_became_peered":"2024-09-15T23:21:52.530738+0000","last_unstale":"2024-09-15T23:23:02.144909+0000","last_undegraded":"2024-09-15T23:23:02.144909+0000","last_fullsized":"2024-09-15T23:23:02.144909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:35.325128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149158+0000","last_change":"2024-09-15T23:21:50.514100+0000","last_active":"2024-09-15T23:23:02.149158+0000","last_peered":"2024-09-15T23:23:02.149158+0000","last_clean":"2024-09-15T23:23:02.149158+0000","last_became_active":"2024-09-15T23:21:50.513629+0000","last_became_peered":"2024-09-15T23:21:50.513629+0000","last_unstale":"2024-09-15T23:23:02.149158+0000","last_undegraded":"2024-09-15T23:23:02.149158+0000","last_fullsized":"2024-09-15T23:23:02.149158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.187544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145706+0000","last_change":"2024-09-15T23:21:59.562804+0000","last_active":"2024-09-15T23:23:02.145706+0000","last_peered":"2024-09-15T23:23:02.145706+0000","last_clean":"2024-09-15T23:23:02.145706+0000","last_became_active":"2024-09-15T23:21:59.562368+0000","last_became_peered":"2024-09-15T23:21:59.562368+0000","last_unstale":"2024-09-15T23:23:02.145706+0000","last_undegraded":"2024-09-15T23:23:02.145706+0000","last_fullsized":"2024-09-15T23:23:02.145706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:21.137333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146705+0000","last_change":"2024-09-15T23:21:52.530562+0000","last_active":"2024-09-15T23:23:02.146705+0000","last_peered":"2024-09-15T23:23:02.146705+0000","last_clean":"2024-09-15T23:23:02.146705+0000","last_became_active":"2024-09-15T23:21:52.530280+0000","last_became_peered":"2024-09-15T23:21:52.530280+0000","last_unstale":"2024-09-15T23:23:02.146705+0000","last_undegraded":"2024-09-15T23:23:02.146705+0000","last_fullsized":"2024-09-15T23:23:02.146705+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:26.835651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144799+0000","last_change":"2024-09-15T23:21:50.513179+0000","last_active":"2024-09-15T23:23:02.144799+0000","last_peered":"2024-09-15T23:23:02.144799+0000","last_clean":"2024-09-15T23:23:02.144799+0000","last_became_active":"2024-09-15T23:21:50.512790+0000","last_became_peered":"2024-09-15T23:21:50.512790+0000","last_unstale":"2024-09-15T23:23:02.144799+0000","last_undegraded":"2024-09-15T23:23:02.144799+0000","last_fullsized":"2024-09-15T23:23:02.144799+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:56.408198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.383378+0000","last_change":"2024-09-15T23:23:02.382964+0000","last_active":"2024-09-15T23:23:02.383378+0000","last_peered":"2024-09-15T23:23:02.383378+0000","last_clean":"2024-09-15T23:23:02.383378+0000","last_became_active":"2024-09-15T23:23:02.382556+0000","last_became_peered":"2024-09-15T23:23:02.382556+0000","last_unstale":"2024-09-15T23:23:02.383378+0000","last_undegraded":"2024-09-15T23:23:02.383378+0000","last_fullsized":"2024-09-15T23:23:02.383378+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:49.211591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275921+0000","last_change":"2024-09-15T23:21:52.530970+0000","last_active":"2024-09-15T23:23:02.275921+0000","last_peered":"2024-09-15T23:23:02.275921+0000","last_clean":"2024-09-15T23:23:02.275921+0000","last_became_active":"2024-09-15T23:21:52.530636+0000","last_became_peered":"2024-09-15T23:21:52.530636+0000","last_unstale":"2024-09-15T23:23:02.275921+0000","last_undegraded":"2024-09-15T23:23:02.275921+0000","last_fullsized":"2024-09-15T23:23:02.275921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:33.374658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"86'34","reported_seq":175,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:25.209167+0000","last_change":"2024-09-15T23:21:50.518449+0000","last_active":"2024-09-15T23:23:25.209167+0000","last_peered":"2024-09-15T23:23:25.209167+0000","last_clean":"2024-09-15T23:23:25.209167+0000","last_became_active":"2024-09-15T23:21:50.518099+0000","last_became_peered":"2024-09-15T23:21:50.518099+0000","last_unstale":"2024-09-15T23:23:25.209167+0000","last_undegraded":"2024-09-15T23:23:25.209167+0000","last_fullsized":"2024-09-15T23:23:25.209167+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:43.792971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62076,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":107,"num_read_kb":115,"num_write":36,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.277900+0000","last_change":"2024-09-15T23:23:02.277457+0000","last_active":"2024-09-15T23:23:02.277900+0000","last_peered":"2024-09-15T23:23:02.277900+0000","last_clean":"2024-09-15T23:23:02.277900+0000","last_became_active":"2024-09-15T23:23:02.276960+0000","last_became_peered":"2024-09-15T23:23:02.276960+0000","last_unstale":"2024-09-15T23:23:02.277900+0000","last_undegraded":"2024-09-15T23:23:02.277900+0000","last_fullsized":"2024-09-15T23:23:02.277900+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:15:28.823804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148672+0000","last_change":"2024-09-15T23:21:52.530541+0000","last_active":"2024-09-15T23:23:02.148672+0000","last_peered":"2024-09-15T23:23:02.148672+0000","last_clean":"2024-09-15T23:23:02.148672+0000","last_became_active":"2024-09-15T23:21:52.530217+0000","last_became_peered":"2024-09-15T23:21:52.530217+0000","last_unstale":"2024-09-15T23:23:02.148672+0000","last_undegraded":"2024-09-15T23:23:02.148672+0000","last_fullsized":"2024-09-15T23:23:02.148672+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:14.864563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149346+0000","last_change":"2024-09-15T23:21:50.562203+0000","last_active":"2024-09-15T23:23:02.149346+0000","last_peered":"2024-09-15T23:23:02.149346+0000","last_clean":"2024-09-15T23:23:02.149346+0000","last_became_active":"2024-09-15T23:21:50.561827+0000","last_became_peered":"2024-09-15T23:21:50.561827+0000","last_unstale":"2024-09-15T23:23:02.149346+0000","last_undegraded":"2024-09-15T23:23:02.149346+0000","last_fullsized":"2024-09-15T23:23:02.149346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:22.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179436+0000","last_change":"2024-09-15T23:21:59.561013+0000","last_active":"2024-09-15T23:23:02.179436+0000","last_peered":"2024-09-15T23:23:02.179436+0000","last_clean":"2024-09-15T23:23:02.179436+0000","last_became_active":"2024-09-15T23:21:59.560665+0000","last_became_peered":"2024-09-15T23:21:59.560665+0000","last_unstale":"2024-09-15T23:23:02.179436+0000","last_undegraded":"2024-09-15T23:23:02.179436+0000","last_fullsized":"2024-09-15T23:23:02.179436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:00.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381152+0000","last_change":"2024-09-15T23:21:52.529922+0000","last_active":"2024-09-15T23:23:02.381152+0000","last_peered":"2024-09-15T23:23:02.381152+0000","last_clean":"2024-09-15T23:23:02.381152+0000","last_became_active":"2024-09-15T23:21:52.529179+0000","last_became_peered":"2024-09-15T23:21:52.529179+0000","last_unstale":"2024-09-15T23:23:02.381152+0000","last_undegraded":"2024-09-15T23:23:02.381152+0000","last_fullsized":"2024-09-15T23:23:02.381152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.077658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145411+0000","last_change":"2024-09-15T23:21:50.514420+0000","last_active":"2024-09-15T23:23:02.145411+0000","last_peered":"2024-09-15T23:23:02.145411+0000","last_clean":"2024-09-15T23:23:02.145411+0000","last_became_active":"2024-09-15T23:21:50.514007+0000","last_became_peered":"2024-09-15T23:21:50.514007+0000","last_unstale":"2024-09-15T23:23:02.145411+0000","last_undegraded":"2024-09-15T23:23:02.145411+0000","last_fullsized":"2024-09-15T23:23:02.145411+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:36.845674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143335+0000","last_change":"2024-09-15T23:21:59.565978+0000","last_active":"2024-09-15T23:23:02.143335+0000","last_peered":"2024-09-15T23:23:02.143335+0000","last_clean":"2024-09-15T23:23:02.143335+0000","last_became_active":"2024-09-15T23:21:59.565604+0000","last_became_peered":"2024-09-15T23:21:59.565604+0000","last_unstale":"2024-09-15T23:23:02.143335+0000","last_undegraded":"2024-09-15T23:23:02.143335+0000","last_fullsized":"2024-09-15T23:23:02.143335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:47.089504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148587+0000","last_change":"2024-09-15T23:21:52.543511+0000","last_active":"2024-09-15T23:23:02.148587+0000","last_peered":"2024-09-15T23:23:02.148587+0000","last_clean":"2024-09-15T23:23:02.148587+0000","last_became_active":"2024-09-15T23:21:52.543169+0000","last_became_peered":"2024-09-15T23:21:52.543169+0000","last_unstale":"2024-09-15T23:23:02.148587+0000","last_undegraded":"2024-09-15T23:23:02.148587+0000","last_fullsized":"2024-09-15T23:23:02.148587+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:05.696772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:25.209347+0000","last_change":"2024-09-15T23:21:50.518425+0000","last_active":"2024-09-15T23:23:25.209347+0000","last_peered":"2024-09-15T23:23:25.209347+0000","last_clean":"2024-09-15T23:23:25.209347+0000","last_became_active":"2024-09-15T23:21:50.518052+0000","last_became_peered":"2024-09-15T23:21:50.518052+0000","last_unstale":"2024-09-15T23:23:25.209347+0000","last_undegraded":"2024-09-15T23:23:25.209347+0000","last_fullsized":"2024-09-15T23:23:25.209347+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:02.243869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380690+0000","last_change":"2024-09-15T23:21:59.570310+0000","last_active":"2024-09-15T23:23:02.380690+0000","last_peered":"2024-09-15T23:23:02.380690+0000","last_clean":"2024-09-15T23:23:02.380690+0000","last_became_active":"2024-09-15T23:21:59.569909+0000","last_became_peered":"2024-09-15T23:21:59.569909+0000","last_unstale":"2024-09-15T23:23:02.380690+0000","last_undegraded":"2024-09-15T23:23:02.380690+0000","last_fullsized":"2024-09-15T23:23:02.380690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:20:27.515537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380837+0000","last_change":"2024-09-15T23:21:59.568208+0000","last_active":"2024-09-15T23:23:02.380837+0000","last_peered":"2024-09-15T23:23:02.380837+0000","last_clean":"2024-09-15T23:23:02.380837+0000","last_became_active":"2024-09-15T23:21:59.567858+0000","last_became_peered":"2024-09-15T23:21:59.567858+0000","last_unstale":"2024-09-15T23:23:02.380837+0000","last_undegraded":"2024-09-15T23:23:02.380837+0000","last_fullsized":"2024-09-15T23:23:02.380837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:34:42.895247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143446+0000","last_change":"2024-09-15T23:21:52.532679+0000","last_active":"2024-09-15T23:23:02.143446+0000","last_peered":"2024-09-15T23:23:02.143446+0000","last_clean":"2024-09-15T23:23:02.143446+0000","last_became_active":"2024-09-15T23:21:52.532345+0000","last_became_peered":"2024-09-15T23:21:52.532345+0000","last_unstale":"2024-09-15T23:23:02.143446+0000","last_undegraded":"2024-09-15T23:23:02.143446+0000","last_fullsized":"2024-09-15T23:23:02.143446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:45.943653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148716+0000","last_change":"2024-09-15T23:21:50.510836+0000","last_active":"2024-09-15T23:23:02.148716+0000","last_peered":"2024-09-15T23:23:02.148716+0000","last_clean":"2024-09-15T23:23:02.148716+0000","last_became_active":"2024-09-15T23:21:50.510514+0000","last_became_peered":"2024-09-15T23:21:50.510514+0000","last_unstale":"2024-09-15T23:23:02.148716+0000","last_undegraded":"2024-09-15T23:23:02.148716+0000","last_fullsized":"2024-09-15T23:23:02.148716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:06.404612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.383436+0000","last_change":"2024-09-15T23:23:02.383436+0000","last_active":"2024-09-15T23:23:02.383436+0000","last_peered":"2024-09-15T23:23:02.383436+0000","last_clean":"2024-09-15T23:23:02.383436+0000","last_became_active":"2024-09-15T23:23:02.382982+0000","last_became_peered":"2024-09-15T23:23:02.382982+0000","last_unstale":"2024-09-15T23:23:02.383436+0000","last_undegraded":"2024-09-15T23:23:02.383436+0000","last_fullsized":"2024-09-15T23:23:02.383436+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:55:14.076604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381259+0000","last_change":"2024-09-15T23:21:52.526766+0000","last_active":"2024-09-15T23:23:02.381259+0000","last_peered":"2024-09-15T23:23:02.381259+0000","last_clean":"2024-09-15T23:23:02.381259+0000","last_became_active":"2024-09-15T23:21:52.526446+0000","last_became_peered":"2024-09-15T23:21:52.526446+0000","last_unstale":"2024-09-15T23:23:02.381259+0000","last_undegraded":"2024-09-15T23:23:02.381259+0000","last_fullsized":"2024-09-15T23:23:02.381259+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:21:37.004730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145965+0000","last_change":"2024-09-15T23:21:50.512928+0000","last_active":"2024-09-15T23:23:02.145965+0000","last_peered":"2024-09-15T23:23:02.145965+0000","last_clean":"2024-09-15T23:23:02.145965+0000","last_became_active":"2024-09-15T23:21:50.512558+0000","last_became_peered":"2024-09-15T23:21:50.512558+0000","last_unstale":"2024-09-15T23:23:02.145965+0000","last_undegraded":"2024-09-15T23:23:02.145965+0000","last_fullsized":"2024-09-15T23:23:02.145965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:08.653084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147478+0000","last_change":"2024-09-15T23:21:50.516920+0000","last_active":"2024-09-15T23:23:02.147478+0000","last_peered":"2024-09-15T23:23:02.147478+0000","last_clean":"2024-09-15T23:23:02.147478+0000","last_became_active":"2024-09-15T23:21:50.516573+0000","last_became_peered":"2024-09-15T23:21:50.516573+0000","last_unstale":"2024-09-15T23:23:02.147478+0000","last_undegraded":"2024-09-15T23:23:02.147478+0000","last_fullsized":"2024-09-15T23:23:02.147478+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.367903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148514+0000","last_change":"2024-09-15T23:21:52.536696+0000","last_active":"2024-09-15T23:23:02.148514+0000","last_peered":"2024-09-15T23:23:02.148514+0000","last_clean":"2024-09-15T23:23:02.148514+0000","last_became_active":"2024-09-15T23:21:52.536338+0000","last_became_peered":"2024-09-15T23:21:52.536338+0000","last_unstale":"2024-09-15T23:23:02.148514+0000","last_undegraded":"2024-09-15T23:23:02.148514+0000","last_fullsized":"2024-09-15T23:23:02.148514+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:05.152749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144628+0000","last_change":"2024-09-15T23:21:59.562968+0000","last_active":"2024-09-15T23:23:02.144628+0000","last_peered":"2024-09-15T23:23:02.144628+0000","last_clean":"2024-09-15T23:23:02.144628+0000","last_became_active":"2024-09-15T23:21:59.562579+0000","last_became_peered":"2024-09-15T23:21:59.562579+0000","last_unstale":"2024-09-15T23:23:02.144628+0000","last_undegraded":"2024-09-15T23:23:02.144628+0000","last_fullsized":"2024-09-15T23:23:02.144628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:02.853057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:25.210180+0000","last_change":"2024-09-15T23:21:50.513944+0000","last_active":"2024-09-15T23:23:25.210180+0000","last_peered":"2024-09-15T23:23:25.210180+0000","last_clean":"2024-09-15T23:23:25.210180+0000","last_became_active":"2024-09-15T23:21:50.513484+0000","last_became_peered":"2024-09-15T23:21:50.513484+0000","last_unstale":"2024-09-15T23:23:25.210180+0000","last_undegraded":"2024-09-15T23:23:25.210180+0000","last_fullsized":"2024-09-15T23:23:25.210180+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:35.719344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275739+0000","last_change":"2024-09-15T23:21:52.539467+0000","last_active":"2024-09-15T23:23:02.275739+0000","last_peered":"2024-09-15T23:23:02.275739+0000","last_clean":"2024-09-15T23:23:02.275739+0000","last_became_active":"2024-09-15T23:21:52.538998+0000","last_became_peered":"2024-09-15T23:21:52.538998+0000","last_unstale":"2024-09-15T23:23:02.275739+0000","last_undegraded":"2024-09-15T23:23:02.275739+0000","last_fullsized":"2024-09-15T23:23:02.275739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:22.213857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145115+0000","last_change":"2024-09-15T23:21:59.564842+0000","last_active":"2024-09-15T23:23:02.145115+0000","last_peered":"2024-09-15T23:23:02.145115+0000","last_clean":"2024-09-15T23:23:02.145115+0000","last_became_active":"2024-09-15T23:21:59.564408+0000","last_became_peered":"2024-09-15T23:21:59.564408+0000","last_unstale":"2024-09-15T23:23:02.145115+0000","last_undegraded":"2024-09-15T23:23:02.145115+0000","last_fullsized":"2024-09-15T23:23:02.145115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:17.909020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149420+0000","last_change":"2024-09-15T23:23:02.149420+0000","last_active":"2024-09-15T23:23:02.149420+0000","last_peered":"2024-09-15T23:23:02.149420+0000","last_clean":"2024-09-15T23:23:02.149420+0000","last_became_active":"2024-09-15T23:23:02.149003+0000","last_became_peered":"2024-09-15T23:23:02.149003+0000","last_unstale":"2024-09-15T23:23:02.149420+0000","last_undegraded":"2024-09-15T23:23:02.149420+0000","last_fullsized":"2024-09-15T23:23:02.149420+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:45.490322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144804+0000","last_change":"2024-09-15T23:21:52.532769+0000","last_active":"2024-09-15T23:23:02.144804+0000","last_peered":"2024-09-15T23:23:02.144804+0000","last_clean":"2024-09-15T23:23:02.144804+0000","last_became_active":"2024-09-15T23:21:52.532426+0000","last_became_peered":"2024-09-15T23:21:52.532426+0000","last_unstale":"2024-09-15T23:23:02.144804+0000","last_undegraded":"2024-09-15T23:23:02.144804+0000","last_fullsized":"2024-09-15T23:23:02.144804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:09.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144463+0000","last_change":"2024-09-15T23:21:52.533148+0000","last_active":"2024-09-15T23:23:02.144463+0000","last_peered":"2024-09-15T23:23:02.144463+0000","last_clean":"2024-09-15T23:23:02.144463+0000","last_became_active":"2024-09-15T23:21:52.532823+0000","last_became_peered":"2024-09-15T23:21:52.532823+0000","last_unstale":"2024-09-15T23:23:02.144463+0000","last_undegraded":"2024-09-15T23:23:02.144463+0000","last_fullsized":"2024-09-15T23:23:02.144463+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:01.833945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381268+0000","last_change":"2024-09-15T23:21:52.536790+0000","last_active":"2024-09-15T23:23:02.381268+0000","last_peered":"2024-09-15T23:23:02.381268+0000","last_clean":"2024-09-15T23:23:02.381268+0000","last_became_active":"2024-09-15T23:21:52.536515+0000","last_became_peered":"2024-09-15T23:21:52.536515+0000","last_unstale":"2024-09-15T23:23:02.381268+0000","last_undegraded":"2024-09-15T23:23:02.381268+0000","last_fullsized":"2024-09-15T23:23:02.381268+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:37.207759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.382842+0000","last_change":"2024-09-15T23:23:02.382842+0000","last_active":"2024-09-15T23:23:02.382842+0000","last_peered":"2024-09-15T23:23:02.382842+0000","last_clean":"2024-09-15T23:23:02.382842+0000","last_became_active":"2024-09-15T23:23:02.382455+0000","last_became_peered":"2024-09-15T23:23:02.382455+0000","last_unstale":"2024-09-15T23:23:02.382842+0000","last_undegraded":"2024-09-15T23:23:02.382842+0000","last_fullsized":"2024-09-15T23:23:02.382842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:23.617641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147039+0000","last_change":"2024-09-15T23:21:52.540628+0000","last_active":"2024-09-15T23:23:02.147039+0000","last_peered":"2024-09-15T23:23:02.147039+0000","last_clean":"2024-09-15T23:23:02.147039+0000","last_became_active":"2024-09-15T23:21:52.540308+0000","last_became_peered":"2024-09-15T23:21:52.540308+0000","last_unstale":"2024-09-15T23:23:02.147039+0000","last_undegraded":"2024-09-15T23:23:02.147039+0000","last_fullsized":"2024-09-15T23:23:02.147039+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:16.660637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146562+0000","last_change":"2024-09-15T23:21:52.533780+0000","last_active":"2024-09-15T23:23:02.146562+0000","last_peered":"2024-09-15T23:23:02.146562+0000","last_clean":"2024-09-15T23:23:02.146562+0000","last_became_active":"2024-09-15T23:21:52.533354+0000","last_became_peered":"2024-09-15T23:21:52.533354+0000","last_unstale":"2024-09-15T23:23:02.146562+0000","last_undegraded":"2024-09-15T23:23:02.146562+0000","last_fullsized":"2024-09-15T23:23:02.146562+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.381059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381783+0000","last_change":"2024-09-15T23:21:52.536406+0000","last_active":"2024-09-15T23:23:02.381783+0000","last_peered":"2024-09-15T23:23:02.381783+0000","last_clean":"2024-09-15T23:23:02.381783+0000","last_became_active":"2024-09-15T23:21:52.536091+0000","last_became_peered":"2024-09-15T23:21:52.536091+0000","last_unstale":"2024-09-15T23:23:02.381783+0000","last_undegraded":"2024-09-15T23:23:02.381783+0000","last_fullsized":"2024-09-15T23:23:02.381783+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:30.452201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148544+0000","last_change":"2024-09-15T23:21:52.536333+0000","last_active":"2024-09-15T23:23:02.148544+0000","last_peered":"2024-09-15T23:23:02.148544+0000","last_clean":"2024-09-15T23:23:02.148544+0000","last_became_active":"2024-09-15T23:21:52.534677+0000","last_became_peered":"2024-09-15T23:21:52.534677+0000","last_unstale":"2024-09-15T23:23:02.148544+0000","last_undegraded":"2024-09-15T23:23:02.148544+0000","last_fullsized":"2024-09-15T23:23:02.148544+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:05.287914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380781+0000","last_change":"2024-09-15T23:21:52.536442+0000","last_active":"2024-09-15T23:23:02.380781+0000","last_peered":"2024-09-15T23:23:02.380781+0000","last_clean":"2024-09-15T23:23:02.380781+0000","last_became_active":"2024-09-15T23:21:52.536183+0000","last_became_peered":"2024-09-15T23:21:52.536183+0000","last_unstale":"2024-09-15T23:23:02.380781+0000","last_undegraded":"2024-09-15T23:23:02.380781+0000","last_fullsized":"2024-09-15T23:23:02.380781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:41.641370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.277365+0000","last_change":"2024-09-15T23:23:02.277365+0000","last_active":"2024-09-15T23:23:02.277365+0000","last_peered":"2024-09-15T23:23:02.277365+0000","last_clean":"2024-09-15T23:23:02.277365+0000","last_became_active":"2024-09-15T23:23:02.276905+0000","last_became_peered":"2024-09-15T23:23:02.276905+0000","last_unstale":"2024-09-15T23:23:02.277365+0000","last_undegraded":"2024-09-15T23:23:02.277365+0000","last_fullsized":"2024-09-15T23:23:02.277365+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:21.100657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146982+0000","last_change":"2024-09-15T23:21:52.531171+0000","last_active":"2024-09-15T23:23:02.146982+0000","last_peered":"2024-09-15T23:23:02.146982+0000","last_clean":"2024-09-15T23:23:02.146982+0000","last_became_active":"2024-09-15T23:21:52.530724+0000","last_became_peered":"2024-09-15T23:21:52.530724+0000","last_unstale":"2024-09-15T23:23:02.146982+0000","last_undegraded":"2024-09-15T23:23:02.146982+0000","last_fullsized":"2024-09-15T23:23:02.146982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:11.009119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145814+0000","last_change":"2024-09-15T23:21:52.530433+0000","last_active":"2024-09-15T23:23:02.145814+0000","last_peered":"2024-09-15T23:23:02.145814+0000","last_clean":"2024-09-15T23:23:02.145814+0000","last_became_active":"2024-09-15T23:21:52.530058+0000","last_became_peered":"2024-09-15T23:21:52.530058+0000","last_unstale":"2024-09-15T23:23:02.145814+0000","last_undegraded":"2024-09-15T23:23:02.145814+0000","last_fullsized":"2024-09-15T23:23:02.145814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:51.208820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146177+0000","last_change":"2024-09-15T23:21:52.534639+0000","last_active":"2024-09-15T23:23:02.146177+0000","last_peered":"2024-09-15T23:23:02.146177+0000","last_clean":"2024-09-15T23:23:02.146177+0000","last_became_active":"2024-09-15T23:21:52.534362+0000","last_became_peered":"2024-09-15T23:21:52.534362+0000","last_unstale":"2024-09-15T23:23:02.146177+0000","last_undegraded":"2024-09-15T23:23:02.146177+0000","last_fullsized":"2024-09-15T23:23:02.146177+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:58:12.023501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.143861+0000","last_change":"2024-09-15T23:21:52.532758+0000","last_active":"2024-09-15T23:23:02.143861+0000","last_peered":"2024-09-15T23:23:02.143861+0000","last_clean":"2024-09-15T23:23:02.143861+0000","last_became_active":"2024-09-15T23:21:52.532412+0000","last_became_peered":"2024-09-15T23:21:52.532412+0000","last_unstale":"2024-09-15T23:23:02.143861+0000","last_undegraded":"2024-09-15T23:23:02.143861+0000","last_fullsized":"2024-09-15T23:23:02.143861+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:46.285711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146969+0000","last_change":"2024-09-15T23:21:52.533928+0000","last_active":"2024-09-15T23:23:02.146969+0000","last_peered":"2024-09-15T23:23:02.146969+0000","last_clean":"2024-09-15T23:23:02.146969+0000","last_became_active":"2024-09-15T23:21:52.533578+0000","last_became_peered":"2024-09-15T23:21:52.533578+0000","last_unstale":"2024-09-15T23:23:02.146969+0000","last_undegraded":"2024-09-15T23:23:02.146969+0000","last_fullsized":"2024-09-15T23:23:02.146969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:49.476316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147896+0000","last_change":"2024-09-15T23:21:52.533417+0000","last_active":"2024-09-15T23:23:02.147896+0000","last_peered":"2024-09-15T23:23:02.147896+0000","last_clean":"2024-09-15T23:23:02.147896+0000","last_became_active":"2024-09-15T23:21:52.533085+0000","last_became_peered":"2024-09-15T23:21:52.533085+0000","last_unstale":"2024-09-15T23:23:02.147896+0000","last_undegraded":"2024-09-15T23:23:02.147896+0000","last_fullsized":"2024-09-15T23:23:02.147896+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:20.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380911+0000","last_change":"2024-09-15T23:21:52.542340+0000","last_active":"2024-09-15T23:23:02.380911+0000","last_peered":"2024-09-15T23:23:02.380911+0000","last_clean":"2024-09-15T23:23:02.380911+0000","last_became_active":"2024-09-15T23:21:52.542048+0000","last_became_peered":"2024-09-15T23:21:52.542048+0000","last_unstale":"2024-09-15T23:23:02.380911+0000","last_undegraded":"2024-09-15T23:23:02.380911+0000","last_fullsized":"2024-09-15T23:23:02.380911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:57.557420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380816+0000","last_change":"2024-09-15T23:21:52.535576+0000","last_active":"2024-09-15T23:23:02.380816+0000","last_peered":"2024-09-15T23:23:02.380816+0000","last_clean":"2024-09-15T23:23:02.380816+0000","last_became_active":"2024-09-15T23:21:52.535223+0000","last_became_peered":"2024-09-15T23:21:52.535223+0000","last_unstale":"2024-09-15T23:23:02.380816+0000","last_undegraded":"2024-09-15T23:23:02.380816+0000","last_fullsized":"2024-09-15T23:23:02.380816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:45.874927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.150556+0000","last_change":"2024-09-15T23:23:02.150556+0000","last_active":"2024-09-15T23:23:02.150556+0000","last_peered":"2024-09-15T23:23:02.150556+0000","last_clean":"2024-09-15T23:23:02.150556+0000","last_became_active":"2024-09-15T23:23:02.150253+0000","last_became_peered":"2024-09-15T23:23:02.150253+0000","last_unstale":"2024-09-15T23:23:02.150556+0000","last_undegraded":"2024-09-15T23:23:02.150556+0000","last_fullsized":"2024-09-15T23:23:02.150556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:47.950356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179566+0000","last_change":"2024-09-15T23:21:52.537012+0000","last_active":"2024-09-15T23:23:02.179566+0000","last_peered":"2024-09-15T23:23:02.179566+0000","last_clean":"2024-09-15T23:23:02.179566+0000","last_became_active":"2024-09-15T23:21:52.536245+0000","last_became_peered":"2024-09-15T23:21:52.536245+0000","last_unstale":"2024-09-15T23:23:02.179566+0000","last_undegraded":"2024-09-15T23:23:02.179566+0000","last_fullsized":"2024-09-15T23:23:02.179566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:31.150491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.145395+0000","last_change":"2024-09-15T23:21:52.540431+0000","last_active":"2024-09-15T23:23:02.145395+0000","last_peered":"2024-09-15T23:23:02.145395+0000","last_clean":"2024-09-15T23:23:02.145395+0000","last_became_active":"2024-09-15T23:21:52.540104+0000","last_became_peered":"2024-09-15T23:21:52.540104+0000","last_unstale":"2024-09-15T23:23:02.145395+0000","last_undegraded":"2024-09-15T23:23:02.145395+0000","last_fullsized":"2024-09-15T23:23:02.145395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.550258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.147173+0000","last_change":"2024-09-15T23:21:52.537590+0000","last_active":"2024-09-15T23:23:02.147173+0000","last_peered":"2024-09-15T23:23:02.147173+0000","last_clean":"2024-09-15T23:23:02.147173+0000","last_became_active":"2024-09-15T23:21:52.533804+0000","last_became_peered":"2024-09-15T23:21:52.533804+0000","last_unstale":"2024-09-15T23:23:02.147173+0000","last_undegraded":"2024-09-15T23:23:02.147173+0000","last_fullsized":"2024-09-15T23:23:02.147173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:48.449090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.148967+0000","last_change":"2024-09-15T23:21:52.536130+0000","last_active":"2024-09-15T23:23:02.148967+0000","last_peered":"2024-09-15T23:23:02.148967+0000","last_clean":"2024-09-15T23:23:02.148967+0000","last_became_active":"2024-09-15T23:21:52.535336+0000","last_became_peered":"2024-09-15T23:21:52.535336+0000","last_unstale":"2024-09-15T23:23:02.148967+0000","last_undegraded":"2024-09-15T23:23:02.148967+0000","last_fullsized":"2024-09-15T23:23:02.148967+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:52.581994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380738+0000","last_change":"2024-09-15T23:21:52.540485+0000","last_active":"2024-09-15T23:23:02.380738+0000","last_peered":"2024-09-15T23:23:02.380738+0000","last_clean":"2024-09-15T23:23:02.380738+0000","last_became_active":"2024-09-15T23:21:52.540178+0000","last_became_peered":"2024-09-15T23:21:52.540178+0000","last_unstale":"2024-09-15T23:23:02.380738+0000","last_undegraded":"2024-09-15T23:23:02.380738+0000","last_fullsized":"2024-09-15T23:23:02.380738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:40:22.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.144658+0000","last_change":"2024-09-15T23:21:52.538845+0000","last_active":"2024-09-15T23:23:02.144658+0000","last_peered":"2024-09-15T23:23:02.144658+0000","last_clean":"2024-09-15T23:23:02.144658+0000","last_became_active":"2024-09-15T23:21:52.538511+0000","last_became_peered":"2024-09-15T23:21:52.538511+0000","last_unstale":"2024-09-15T23:23:02.144658+0000","last_undegraded":"2024-09-15T23:23:02.144658+0000","last_fullsized":"2024-09-15T23:23:02.144658+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:00.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.149590+0000","last_change":"2024-09-15T23:23:02.149590+0000","last_active":"2024-09-15T23:23:02.149590+0000","last_peered":"2024-09-15T23:23:02.149590+0000","last_clean":"2024-09-15T23:23:02.149590+0000","last_became_active":"2024-09-15T23:23:02.149184+0000","last_became_peered":"2024-09-15T23:23:02.149184+0000","last_unstale":"2024-09-15T23:23:02.149590+0000","last_undegraded":"2024-09-15T23:23:02.149590+0000","last_fullsized":"2024-09-15T23:23:02.149590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:05:29.407175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.275569+0000","last_change":"2024-09-15T23:21:52.530524+0000","last_active":"2024-09-15T23:23:02.275569+0000","last_peered":"2024-09-15T23:23:02.275569+0000","last_clean":"2024-09-15T23:23:02.275569+0000","last_became_active":"2024-09-15T23:21:52.530203+0000","last_became_peered":"2024-09-15T23:21:52.530203+0000","last_unstale":"2024-09-15T23:23:02.275569+0000","last_undegraded":"2024-09-15T23:23:02.275569+0000","last_fullsized":"2024-09-15T23:23:02.275569+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:50.560211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.179469+0000","last_change":"2024-09-15T23:21:52.537256+0000","last_active":"2024-09-15T23:23:02.179469+0000","last_peered":"2024-09-15T23:23:02.179469+0000","last_clean":"2024-09-15T23:23:02.179469+0000","last_became_active":"2024-09-15T23:21:52.536731+0000","last_became_peered":"2024-09-15T23:21:52.536731+0000","last_unstale":"2024-09-15T23:23:02.179469+0000","last_undegraded":"2024-09-15T23:23:02.179469+0000","last_fullsized":"2024-09-15T23:23:02.179469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:18.815398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.381161+0000","last_change":"2024-09-15T23:21:52.537728+0000","last_active":"2024-09-15T23:23:02.381161+0000","last_peered":"2024-09-15T23:23:02.381161+0000","last_clean":"2024-09-15T23:23:02.381161+0000","last_became_active":"2024-09-15T23:21:52.537434+0000","last_became_peered":"2024-09-15T23:21:52.537434+0000","last_unstale":"2024-09-15T23:23:02.381161+0000","last_undegraded":"2024-09-15T23:23:02.381161+0000","last_fullsized":"2024-09-15T23:23:02.381161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:25.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.380967+0000","last_change":"2024-09-15T23:21:52.534780+0000","last_active":"2024-09-15T23:23:02.380967+0000","last_peered":"2024-09-15T23:23:02.380967+0000","last_clean":"2024-09-15T23:23:02.380967+0000","last_became_active":"2024-09-15T23:21:52.534399+0000","last_became_peered":"2024-09-15T23:21:52.534399+0000","last_unstale":"2024-09-15T23:23:02.380967+0000","last_undegraded":"2024-09-15T23:23:02.380967+0000","last_fullsized":"2024-09-15T23:23:02.380967+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:47.344129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:23:02.146885+0000","last_change":"2024-09-15T23:21:52.533403+0000","last_active":"2024-09-15T23:23:02.146885+0000","last_peered":"2024-09-15T23:23:02.146885+0000","last_clean":"2024-09-15T23:23:02.146885+0000","last_became_active":"2024-09-15T23:21:52.533069+0000","last_became_peered":"2024-09-15T23:21:52.533069+0000","last_unstale":"2024-09-15T23:23:02.146885+0000","last_undegraded":"2024-09-15T23:23:02.146885+0000","last_fullsized":"2024-09-15T23:23:02.146885+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:38.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68891,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":543,"num_read_kb":344,"num_write":247,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":246033,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579937,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":978944,"data_stored":501202,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841573,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1126400,"data_stored":1150487,"data_compressed":59401,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005099,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1007616,"data_stored":505228,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234032,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":999424,"data_stored":501510,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462965,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1052672,"data_stored":1098009,"data_compressed":59401,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626490,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":978944,"data_stored":496636,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855423,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1060864,"data_stored":1093198,"data_compressed":59401,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018949,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":987136,"data_stored":501103,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247881,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":991232,"data_stored":496344,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444111,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":559962,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607636,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1044480,"data_stored":559742,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1003520,"data_stored":498092,"data_compressed":23551,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":64205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:23:33.717 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-15T23:23:33.717 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T23:23:33.717 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-15T23:23:33.717 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-15T23:23:33.963 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.450+0000 7f4ce9d84640 1 -- 172.21.15.89:0/652611499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ce410c170 msgr2=0x7f4ce410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.450+0000 7f4ce9d84640 1 --2- 172.21.15.89:0/652611499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ce410c170 0x7f4ce410e610 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009ca0 tx=0x7f4cdc02a950 comp rx=0 tx=0).stop 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.450+0000 7f4ce9d84640 1 -- 172.21.15.89:0/652611499 shutdown_connections 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.450+0000 7f4ce9d84640 1 --2- 172.21.15.89:0/652611499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ce410c170 0x7f4ce410e610 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.450+0000 7f4ce9d84640 1 --2- 172.21.15.89:0/652611499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce4103660 0x7f4ce410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.450+0000 7f4ce9d84640 1 --2- 172.21.15.89:0/652611499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce4102d40 0x7f4ce4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.450+0000 7f4ce9d84640 1 -- 172.21.15.89:0/652611499 >> 172.21.15.89:0/652611499 conn(0x7f4ce40fc7a0 msgr2=0x7f4ce40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.450+0000 7f4ce9d84640 1 -- 172.21.15.89:0/652611499 shutdown_connections 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.450+0000 7f4ce9d84640 1 -- 172.21.15.89:0/652611499 wait complete. 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce9d84640 1 Processor -- start 2024-09-15T23:23:34.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce9d84640 1 -- start start 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce9d84640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce4102d40 0x7f4ce4198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce9d84640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce4103660 0x7f4ce4198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce9d84640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ce410c170 0x7f4ce419cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce9d84640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce4111f30 con 0x7f4ce410c170 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce9d84640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce4111db0 con 0x7f4ce4102d40 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce9d84640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41120b0 con 0x7f4ce4103660 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ce410c170 0x7f4ce419cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ce410c170 0x7f4ce419cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60584/0 (socket says 172.21.15.89:60584) 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce3fff640 1 -- 172.21.15.89:0/364279174 learned_addr learned my addr 172.21.15.89:0/364279174 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.451+0000 7f4ce2ffd640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce4103660 0x7f4ce4198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce37fe640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce4102d40 0x7f4ce4198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce3fff640 1 -- 172.21.15.89:0/364279174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce4103660 msgr2=0x7f4ce4198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:34.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce3fff640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce4103660 0x7f4ce4198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce3fff640 1 -- 172.21.15.89:0/364279174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce4102d40 msgr2=0x7f4ce4198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce3fff640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce4102d40 0x7f4ce4198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce3fff640 1 -- 172.21.15.89:0/364279174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce419d2e0 con 0x7f4ce410c170 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce37fe640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce4102d40 0x7f4ce4198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce3fff640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ce410c170 0x7f4ce419cc00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc006010 tx=0x7f4cdc02f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce0ff9640 1 -- 172.21.15.89:0/364279174 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc034040 con 0x7f4ce410c170 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce0ff9640 1 -- 172.21.15.89:0/364279174 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cdc003e00 con 0x7f4ce410c170 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce9d84640 1 -- 172.21.15.89:0/364279174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce419d570 con 0x7f4ce410c170 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce0ff9640 1 -- 172.21.15.89:0/364279174 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0456e0 con 0x7f4ce410c170 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.452+0000 7f4ce9d84640 1 -- 172.21.15.89:0/364279174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce41a4eb0 con 0x7f4ce410c170 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.453+0000 7f4ce9d84640 1 -- 172.21.15.89:0/364279174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce4103120 con 0x7f4ce410c170 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.454+0000 7f4ce0ff9640 1 -- 172.21.15.89:0/364279174 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4cdc037020 con 0x7f4ce410c170 2024-09-15T23:23:34.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.455+0000 7f4ce0ff9640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ccc0a3350 0x7f4ccc0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:34.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.455+0000 7f4ce37fe640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ccc0a3350 0x7f4ccc0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:34.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.455+0000 7f4ce0ff9640 1 -- 172.21.15.89:0/364279174 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f4cdc0f0ec0 con 0x7f4ce410c170 2024-09-15T23:23:34.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.455+0000 7f4ce37fe640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ccc0a3350 0x7f4ccc0a5810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4cd400baa0 tx=0x7f4cd4005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:34.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.459+0000 7f4ce0ff9640 1 -- 172.21.15.89:0/364279174 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4cdc0bd830 con 0x7f4ce410c170 2024-09-15T23:23:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.663+0000 7f4ce9d84640 1 -- 172.21.15.89:0/364279174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4ce41a5080 con 0x7f4ce410c170 2024-09-15T23:23:34.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.663+0000 7f4ce0ff9640 1 -- 172.21.15.89:0/364279174 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f4cdc0bd130 con 0x7f4ce410c170 2024-09-15T23:23:34.665 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:23:34.665 INFO:teuthology.orchestra.run.smithi089.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T23:23:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.665+0000 7f4ce9d84640 1 -- 172.21.15.89:0/364279174 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ccc0a3350 msgr2=0x7f4ccc0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.665+0000 7f4ce9d84640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ccc0a3350 0x7f4ccc0a5810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4cd400baa0 tx=0x7f4cd4005d90 comp rx=0 tx=0).stop 2024-09-15T23:23:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.665+0000 7f4ce9d84640 1 -- 172.21.15.89:0/364279174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ce410c170 msgr2=0x7f4ce419cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.665+0000 7f4ce9d84640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ce410c170 0x7f4ce419cc00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc006010 tx=0x7f4cdc02f710 comp rx=0 tx=0).stop 2024-09-15T23:23:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.665+0000 7f4ce9d84640 1 -- 172.21.15.89:0/364279174 shutdown_connections 2024-09-15T23:23:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.665+0000 7f4ce9d84640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ccc0a3350 0x7f4ccc0a5810 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.665+0000 7f4ce9d84640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ce410c170 0x7f4ce419cc00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.665+0000 7f4ce9d84640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ce4103660 0x7f4ce4198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:34.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.665+0000 7f4ce9d84640 1 --2- 172.21.15.89:0/364279174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ce4102d40 0x7f4ce4198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:34.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.665+0000 7f4ce9d84640 1 -- 172.21.15.89:0/364279174 >> 172.21.15.89:0/364279174 conn(0x7f4ce40fc7a0 msgr2=0x7f4ce40fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:34.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.666+0000 7f4ce9d84640 1 -- 172.21.15.89:0/364279174 shutdown_connections 2024-09-15T23:23:34.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:34.666+0000 7f4ce9d84640 1 -- 172.21.15.89:0/364279174 wait complete. 2024-09-15T23:23:34.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:34 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/364279174' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-15T23:23:34.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:34 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/364279174' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-15T23:23:34.993 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:34 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/364279174' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-15T23:23:35.320 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-15T23:23:35.321 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:23:35.568 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:35.973 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:35 smithi089 ceph-mon[30933]: pgmap v335: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:35.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:35 smithi174 ceph-mon[35586]: pgmap v335: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:36.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.052+0000 7f7417e59640 1 -- 172.21.15.89:0/3298270387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7410102780 msgr2=0x7f74101067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:36.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.052+0000 7f7417e59640 1 --2- 172.21.15.89:0/3298270387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7410102780 0x7f74101067d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7400009900 tx=0x7f740002a410 comp rx=0 tx=0).stop 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.052+0000 7f7417e59640 1 -- 172.21.15.89:0/3298270387 shutdown_connections 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.052+0000 7f7417e59640 1 --2- 172.21.15.89:0/3298270387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7410106f00 0x7f741010ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.052+0000 7f7417e59640 1 --2- 172.21.15.89:0/3298270387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7410102780 0x7f74101067d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.052+0000 7f7417e59640 1 --2- 172.21.15.89:0/3298270387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7410101dd0 0x7f74101021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.052+0000 7f7417e59640 1 -- 172.21.15.89:0/3298270387 >> 172.21.15.89:0/3298270387 conn(0x7f74100fd8b0 msgr2=0x7f74100ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.052+0000 7f7417e59640 1 -- 172.21.15.89:0/3298270387 shutdown_connections 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.052+0000 7f7417e59640 1 -- 172.21.15.89:0/3298270387 wait complete. 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.052+0000 7f7417e59640 1 Processor -- start 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f7417e59640 1 -- start start 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f7417e59640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7410101dd0 0x7f74101983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f7417e59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7410102780 0x7f7410198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:36.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f7417e59640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7410106f00 0x7f741019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f7417e59640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7410111f60 con 0x7f7410106f00 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f7417e59640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7410111de0 con 0x7f7410101dd0 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f7417e59640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f74101120e0 con 0x7f7410102780 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f74163f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7410106f00 0x7f741019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f74163f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7410106f00 0x7f741019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33538/0 (socket says 172.21.15.89:33538) 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f74163f6640 1 -- 172.21.15.89:0/2504469911 learned_addr learned my addr 172.21.15.89:0/2504469911 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f7415bf5640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7410101dd0 0x7f74101983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f74153f4640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7410102780 0x7f7410198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f74163f6640 1 -- 172.21.15.89:0/2504469911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7410102780 msgr2=0x7f7410198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f74163f6640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7410102780 0x7f7410198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f74163f6640 1 -- 172.21.15.89:0/2504469911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7410101dd0 msgr2=0x7f74101983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f74163f6640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7410101dd0 0x7f74101983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f74163f6640 1 -- 172.21.15.89:0/2504469911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741019d370 con 0x7f7410106f00 2024-09-15T23:23:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f7415bf5640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7410101dd0 0x7f74101983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:36.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.053+0000 7f74163f6640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7410106f00 0x7f741019cc90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f740800eaa0 tx=0x7f740800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:36.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.054+0000 7f7406ffd640 1 -- 172.21.15.89:0/2504469911 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740800cdb0 con 0x7f7410106f00 2024-09-15T23:23:36.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.054+0000 7f7406ffd640 1 -- 172.21.15.89:0/2504469911 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7408005020 con 0x7f7410106f00 2024-09-15T23:23:36.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.054+0000 7f7417e59640 1 -- 172.21.15.89:0/2504469911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741019d660 con 0x7f7410106f00 2024-09-15T23:23:36.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.054+0000 7f7406ffd640 1 -- 172.21.15.89:0/2504469911 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74080116b0 con 0x7f7410106f00 2024-09-15T23:23:36.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.054+0000 7f7417e59640 1 -- 172.21.15.89:0/2504469911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74101a4f40 con 0x7f7410106f00 2024-09-15T23:23:36.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.055+0000 7f7417e59640 1 -- 172.21.15.89:0/2504469911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74101021b0 con 0x7f7410106f00 2024-09-15T23:23:36.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.056+0000 7f7406ffd640 1 -- 172.21.15.89:0/2504469911 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7408005560 con 0x7f7410106f00 2024-09-15T23:23:36.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.056+0000 7f7406ffd640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73fc0a32d0 0x7f73fc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:36.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.057+0000 7f7415bf5640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73fc0a32d0 0x7f73fc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:36.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.057+0000 7f7406ffd640 1 -- 172.21.15.89:0/2504469911 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f74080d0420 con 0x7f7410106f00 2024-09-15T23:23:36.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.057+0000 7f7415bf5640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73fc0a32d0 0x7f73fc0a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f740c00abe0 tx=0x7f740c00a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:36.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.060+0000 7f7406ffd640 1 -- 172.21.15.89:0/2504469911 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7408016030 con 0x7f7410106f00 2024-09-15T23:23:36.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:35 smithi120 ceph-mon[35250]: pgmap v335: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.260+0000 7f7417e59640 1 -- 172.21.15.89:0/2504469911 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7410199720 con 0x7f7410106f00 2024-09-15T23:23:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.260+0000 7f7406ffd640 1 -- 172.21.15.89:0/2504469911 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7f7410199720 con 0x7f7410106f00 2024-09-15T23:23:36.262 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:23:36.262 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":28,"btime":"2024-09-15T23:22:37:131197+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:22:37.131175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24452,24436,24430,24428,24424]},"id":1}]} 2024-09-15T23:23:36.262 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 28 2024-09-15T23:23:36.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.262+0000 7f7417e59640 1 -- 172.21.15.89:0/2504469911 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73fc0a32d0 msgr2=0x7f73fc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:36.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.262+0000 7f7417e59640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73fc0a32d0 0x7f73fc0a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f740c00abe0 tx=0x7f740c00a5c0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.262+0000 7f7417e59640 1 -- 172.21.15.89:0/2504469911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7410106f00 msgr2=0x7f741019cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:36.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.262+0000 7f7417e59640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7410106f00 0x7f741019cc90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f740800eaa0 tx=0x7f740800ef60 comp rx=0 tx=0).stop 2024-09-15T23:23:36.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.263+0000 7f7417e59640 1 -- 172.21.15.89:0/2504469911 shutdown_connections 2024-09-15T23:23:36.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.263+0000 7f7417e59640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73fc0a32d0 0x7f73fc0a5790 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.263+0000 7f7417e59640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7410106f00 0x7f741019cc90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.263+0000 7f7417e59640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7410102780 0x7f7410198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.263+0000 7f7417e59640 1 --2- 172.21.15.89:0/2504469911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7410101dd0 0x7f74101983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.263+0000 7f7417e59640 1 -- 172.21.15.89:0/2504469911 >> 172.21.15.89:0/2504469911 conn(0x7f74100fd8b0 msgr2=0x7f74100febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:36.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.263+0000 7f7417e59640 1 -- 172.21.15.89:0/2504469911 shutdown_connections 2024-09-15T23:23:36.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:36.263+0000 7f7417e59640 1 -- 172.21.15.89:0/2504469911 wait complete. 2024-09-15T23:23:36.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:36 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2504469911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:36.982 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:23:36.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:36 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2504469911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:37.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:36 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2504469911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:37.227 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3860003641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a4100a80 msgr2=0x7f92a4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 --2- 172.21.15.89:0/3860003641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a4100a80 0x7f92a4100ee0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9298009930 tx=0x7f929802a420 comp rx=0 tx=0).stop 2024-09-15T23:23:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3860003641 shutdown_connections 2024-09-15T23:23:37.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 --2- 172.21.15.89:0/3860003641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a4101420 0x7f92a410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 --2- 172.21.15.89:0/3860003641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a4100a80 0x7f92a4100ee0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 --2- 172.21.15.89:0/3860003641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4106a80 0x7f92a4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3860003641 >> 172.21.15.89:0/3860003641 conn(0x7f92a40fc7a0 msgr2=0x7f92a40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3860003641 shutdown_connections 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3860003641 wait complete. 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 Processor -- start 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.718+0000 7f92ac2e6640 1 -- start start 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92ac2e6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4100a80 0x7f92a41984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92ac2e6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a4101420 0x7f92a41989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92ac2e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a4106a80 0x7f92a419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92ac2e6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f92a4110ea0 con 0x7f92a4101420 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92ac2e6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f92a4110d20 con 0x7f92a4100a80 2024-09-15T23:23:37.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92ac2e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f92a4111020 con 0x7f92a4106a80 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a9881640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a4101420 0x7f92a41989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a9881640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a4101420 0x7f92a41989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33544/0 (socket says 172.21.15.89:33544) 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a9881640 1 -- 172.21.15.89:0/3216343251 learned_addr learned my addr 172.21.15.89:0/3216343251 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92aa082640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4100a80 0x7f92a41984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92aa883640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a4106a80 0x7f92a419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a9881640 1 -- 172.21.15.89:0/3216343251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a4106a80 msgr2=0x7f92a419cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a9881640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a4106a80 0x7f92a419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a9881640 1 -- 172.21.15.89:0/3216343251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4100a80 msgr2=0x7f92a41984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a9881640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4100a80 0x7f92a41984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a9881640 1 -- 172.21.15.89:0/3216343251 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a419d460 con 0x7f92a4101420 2024-09-15T23:23:37.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a9881640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a4101420 0x7f92a41989f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9298009a50 tx=0x7f9298004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a37fe640 1 -- 172.21.15.89:0/3216343251 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929803f030 con 0x7f92a4101420 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a37fe640 1 -- 172.21.15.89:0/3216343251 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9298003ce0 con 0x7f92a4101420 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92a37fe640 1 -- 172.21.15.89:0/3216343251 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9298035070 con 0x7f92a4101420 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3216343251 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a419d6f0 con 0x7f92a4101420 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.719+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3216343251 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a41a5030 con 0x7f92a4101420 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.720+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3216343251 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a41021c0 con 0x7f92a4101420 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.722+0000 7f92a37fe640 1 -- 172.21.15.89:0/3216343251 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f9298042030 con 0x7f92a4101420 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.722+0000 7f92a37fe640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f928c0a32d0 0x7f928c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.722+0000 7f92aa082640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f928c0a32d0 0x7f928c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.722+0000 7f92a37fe640 1 -- 172.21.15.89:0/3216343251 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f92980f08a0 con 0x7f92a4101420 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.723+0000 7f92aa082640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f928c0a32d0 0x7f928c0a5790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f929c0058c0 tx=0x7f929c00a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:37.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.726+0000 7f92a37fe640 1 -- 172.21.15.89:0/3216343251 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f92980bd100 con 0x7f92a4101420 2024-09-15T23:23:37.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.926+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3216343251 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92a41a5200 con 0x7f92a4101420 2024-09-15T23:23:37.927 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:37 smithi089 ceph-mon[30933]: pgmap v336: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:37.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.927+0000 7f92a37fe640 1 -- 172.21.15.89:0/3216343251 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7f92980bca00 con 0x7f92a4101420 2024-09-15T23:23:37.928 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:23:37.928 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":28,"btime":"2024-09-15T23:22:37:131197+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:22:37.131175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24452,24436,24430,24428,24424]},"id":1}]} 2024-09-15T23:23:37.929 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 28 2024-09-15T23:23:37.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3216343251 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f928c0a32d0 msgr2=0x7f928c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:37.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f928c0a32d0 0x7f928c0a5790 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f929c0058c0 tx=0x7f929c00a4c0 comp rx=0 tx=0).stop 2024-09-15T23:23:37.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3216343251 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a4101420 msgr2=0x7f92a41989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:37.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a4101420 0x7f92a41989f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9298009a50 tx=0x7f9298004270 comp rx=0 tx=0).stop 2024-09-15T23:23:37.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3216343251 shutdown_connections 2024-09-15T23:23:37.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f928c0a32d0 0x7f928c0a5790 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:37.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92a4106a80 0x7f92a419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:37.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92a4101420 0x7f92a41989f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:37.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 --2- 172.21.15.89:0/3216343251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92a4100a80 0x7f92a41984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:37.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3216343251 >> 172.21.15.89:0/3216343251 conn(0x7f92a40fc7a0 msgr2=0x7f92a4104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:37.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3216343251 shutdown_connections 2024-09-15T23:23:37.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:37.929+0000 7f92ac2e6640 1 -- 172.21.15.89:0/3216343251 wait complete. 2024-09-15T23:23:37.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:37 smithi174 ceph-mon[35586]: pgmap v336: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:38.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:37 smithi120 ceph-mon[35250]: pgmap v336: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:38.717 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:23:38.964 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:38.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:38 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3216343251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:39.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:38 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3216343251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:39.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:38 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3216343251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.437+0000 7f12cd7a8640 1 -- 172.21.15.89:0/2815028164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12c8101420 msgr2=0x7f12c810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:39.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.437+0000 7f12cd7a8640 1 --2- 172.21.15.89:0/2815028164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12c8101420 0x7f12c810d950 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009cd0 tx=0x7f12c002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.437+0000 7f12cd7a8640 1 -- 172.21.15.89:0/2815028164 shutdown_connections 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.437+0000 7f12cd7a8640 1 --2- 172.21.15.89:0/2815028164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12c8101420 0x7f12c810d950 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.437+0000 7f12cd7a8640 1 --2- 172.21.15.89:0/2815028164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12c8100a80 0x7f12c8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.437+0000 7f12cd7a8640 1 --2- 172.21.15.89:0/2815028164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12c8106a80 0x7f12c8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.437+0000 7f12cd7a8640 1 -- 172.21.15.89:0/2815028164 >> 172.21.15.89:0/2815028164 conn(0x7f12c80fc7a0 msgr2=0x7f12c80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.437+0000 7f12cd7a8640 1 -- 172.21.15.89:0/2815028164 shutdown_connections 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.437+0000 7f12cd7a8640 1 -- 172.21.15.89:0/2815028164 wait complete. 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12cd7a8640 1 Processor -- start 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12cd7a8640 1 -- start start 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12cd7a8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12c8100a80 0x7f12c8197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12cd7a8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12c8101420 0x7f12c81979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12cd7a8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12c8106a80 0x7f12c819bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:39.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12cd7a8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f12c8110f20 con 0x7f12c8100a80 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12cd7a8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f12c8110da0 con 0x7f12c8101420 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12cd7a8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f12c81110a0 con 0x7f12c8106a80 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12c6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12c8100a80 0x7f12c8197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12c6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12c8100a80 0x7f12c8197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33562/0 (socket says 172.21.15.89:33562) 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12c6ffd640 1 -- 172.21.15.89:0/625475258 learned_addr learned my addr 172.21.15.89:0/625475258 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12c77fe640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12c8106a80 0x7f12c819bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.438+0000 7f12c67fc640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12c8101420 0x7f12c81979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12c6ffd640 1 -- 172.21.15.89:0/625475258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12c8106a80 msgr2=0x7f12c819bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12c6ffd640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12c8106a80 0x7f12c819bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12c6ffd640 1 -- 172.21.15.89:0/625475258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12c8101420 msgr2=0x7f12c81979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12c6ffd640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12c8101420 0x7f12c81979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12c6ffd640 1 -- 172.21.15.89:0/625475258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c819c430 con 0x7f12c8100a80 2024-09-15T23:23:39.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12c77fe640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12c8106a80 0x7f12c819bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12c6ffd640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12c8100a80 0x7f12c8197480 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f12b800b4a0 tx=0x7f12b800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12affff640 1 -- 172.21.15.89:0/625475258 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8014070 con 0x7f12c8100a80 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12cd7a8640 1 -- 172.21.15.89:0/625475258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c819c720 con 0x7f12c8100a80 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12affff640 1 -- 172.21.15.89:0/625475258 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12b8010af0 con 0x7f12c8100a80 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12affff640 1 -- 172.21.15.89:0/625475258 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8017d10 con 0x7f12c8100a80 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.439+0000 7f12cd7a8640 1 -- 172.21.15.89:0/625475258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c81a4000 con 0x7f12c8100a80 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.440+0000 7f12cd7a8640 1 -- 172.21.15.89:0/625475258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c8100f60 con 0x7f12c8100a80 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.441+0000 7f12affff640 1 -- 172.21.15.89:0/625475258 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f12b8017070 con 0x7f12c8100a80 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.442+0000 7f12affff640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12b00a32d0 0x7f12b00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.442+0000 7f12c67fc640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12b00a32d0 0x7f12b00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.442+0000 7f12affff640 1 -- 172.21.15.89:0/625475258 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f12b80cf1a0 con 0x7f12c8100a80 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.442+0000 7f12c67fc640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12b00a32d0 0x7f12b00a5790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f12c80fedb0 tx=0x7f12bc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:39.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.446+0000 7f12affff640 1 -- 172.21.15.89:0/625475258 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f12b809bb10 con 0x7f12c8100a80 2024-09-15T23:23:39.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.612+0000 7f12cd7a8640 1 -- 172.21.15.89:0/625475258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12c8109040 con 0x7f12c8100a80 2024-09-15T23:23:39.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.614+0000 7f12affff640 1 -- 172.21.15.89:0/625475258 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24672 (secure 0 0 0) 0x7f12b809b410 con 0x7f12c8100a80 2024-09-15T23:23:39.615 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:23:39.615 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":88,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:23:02.140424+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:23:39.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 -- 172.21.15.89:0/625475258 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12b00a32d0 msgr2=0x7f12b00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:39.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12b00a32d0 0x7f12b00a5790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f12c80fedb0 tx=0x7f12bc008040 comp rx=0 tx=0).stop 2024-09-15T23:23:39.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 -- 172.21.15.89:0/625475258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12c8100a80 msgr2=0x7f12c8197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:39.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12c8100a80 0x7f12c8197480 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f12b800b4a0 tx=0x7f12b800b960 comp rx=0 tx=0).stop 2024-09-15T23:23:39.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 -- 172.21.15.89:0/625475258 shutdown_connections 2024-09-15T23:23:39.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12b00a32d0 0x7f12b00a5790 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:39.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12c8106a80 0x7f12c819bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:39.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12c8101420 0x7f12c81979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:39.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 --2- 172.21.15.89:0/625475258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12c8100a80 0x7f12c8197480 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:39.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 -- 172.21.15.89:0/625475258 >> 172.21.15.89:0/625475258 conn(0x7f12c80fc7a0 msgr2=0x7f12c80fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:39.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 -- 172.21.15.89:0/625475258 shutdown_connections 2024-09-15T23:23:39.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:39.616+0000 7f12cd7a8640 1 -- 172.21.15.89:0/625475258 wait complete. 2024-09-15T23:23:39.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:39 smithi089 ceph-mon[30933]: pgmap v337: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:39.980 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:39 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/625475258' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:23:40.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:39 smithi120 ceph-mon[35250]: pgmap v337: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:40.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:39 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/625475258' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:23:40.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:39 smithi174 ceph-mon[35586]: pgmap v337: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:40.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:39 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/625475258' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:23:40.428 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:23:40.673 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:41.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.165+0000 7f056e93d640 1 -- 172.21.15.89:0/2095645104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05681027c0 msgr2=0x7f0568106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:41.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.165+0000 7f056e93d640 1 --2- 172.21.15.89:0/2095645104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05681027c0 0x7f0568106810 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0550009900 tx=0x7f055002a410 comp rx=0 tx=0).stop 2024-09-15T23:23:41.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.165+0000 7f056e93d640 1 -- 172.21.15.89:0/2095645104 shutdown_connections 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.165+0000 7f056e93d640 1 --2- 172.21.15.89:0/2095645104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0568106f40 0x7f056810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.165+0000 7f056e93d640 1 --2- 172.21.15.89:0/2095645104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05681027c0 0x7f0568106810 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.165+0000 7f056e93d640 1 --2- 172.21.15.89:0/2095645104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0568101e10 0x7f05681021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.165+0000 7f056e93d640 1 -- 172.21.15.89:0/2095645104 >> 172.21.15.89:0/2095645104 conn(0x7f05680fd8d0 msgr2=0x7f05680ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.165+0000 7f056e93d640 1 -- 172.21.15.89:0/2095645104 shutdown_connections 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.165+0000 7f056e93d640 1 -- 172.21.15.89:0/2095645104 wait complete. 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f056e93d640 1 Processor -- start 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f056e93d640 1 -- start start 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f056e93d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0568101e10 0x7f05681984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f056e93d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05681027c0 0x7f0568198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f056e93d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0568106f40 0x7f056819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f056e93d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0568111fc0 con 0x7f0568101e10 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f056e93d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0568111e40 con 0x7f05681027c0 2024-09-15T23:23:41.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f056e93d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0568112140 con 0x7f0568106f40 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f0567fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0568101e10 0x7f05681984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f0567fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0568101e10 0x7f05681984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33574/0 (socket says 172.21.15.89:33574) 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f0567fff640 1 -- 172.21.15.89:0/3179847725 learned_addr learned my addr 172.21.15.89:0/3179847725 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f055ffff640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05681027c0 0x7f0568198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f0567fff640 1 -- 172.21.15.89:0/3179847725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0568106f40 msgr2=0x7f056819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f0567fff640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0568106f40 0x7f056819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f0567fff640 1 -- 172.21.15.89:0/3179847725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05681027c0 msgr2=0x7f0568198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.166+0000 7f0567fff640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05681027c0 0x7f0568198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.167+0000 7f0567fff640 1 -- 172.21.15.89:0/3179847725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056819d490 con 0x7f0568101e10 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.167+0000 7f0567fff640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0568101e10 0x7f05681984e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f055800b4a0 tx=0x7f055800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.167+0000 7f0565ffb640 1 -- 172.21.15.89:0/3179847725 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558014070 con 0x7f0568101e10 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.167+0000 7f0565ffb640 1 -- 172.21.15.89:0/3179847725 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0558010af0 con 0x7f0568101e10 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.167+0000 7f056e93d640 1 -- 172.21.15.89:0/3179847725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f056819d780 con 0x7f0568101e10 2024-09-15T23:23:41.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.167+0000 7f0565ffb640 1 -- 172.21.15.89:0/3179847725 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558017d10 con 0x7f0568101e10 2024-09-15T23:23:41.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.167+0000 7f056e93d640 1 -- 172.21.15.89:0/3179847725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05681a5060 con 0x7f0568101e10 2024-09-15T23:23:41.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.168+0000 7f056e93d640 1 -- 172.21.15.89:0/3179847725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05681021f0 con 0x7f0568101e10 2024-09-15T23:23:41.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.170+0000 7f0565ffb640 1 -- 172.21.15.89:0/3179847725 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0558017070 con 0x7f0568101e10 2024-09-15T23:23:41.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.170+0000 7f0565ffb640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f05540a32d0 0x7f05540a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:41.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.171+0000 7f055ffff640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f05540a32d0 0x7f05540a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:41.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.171+0000 7f0565ffb640 1 -- 172.21.15.89:0/3179847725 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f05580cf4d0 con 0x7f0568101e10 2024-09-15T23:23:41.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.171+0000 7f055ffff640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f05540a32d0 0x7f05540a5790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0550000c00 tx=0x7f05500058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:41.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.174+0000 7f0565ffb640 1 -- 172.21.15.89:0/3179847725 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f055809be40 con 0x7f0568101e10 2024-09-15T23:23:41.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.373+0000 7f056e93d640 1 -- 172.21.15.89:0/3179847725 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0568199840 con 0x7f0568101e10 2024-09-15T23:23:41.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.374+0000 7f0565ffb640 1 -- 172.21.15.89:0/3179847725 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7f055809b740 con 0x7f0568101e10 2024-09-15T23:23:41.375 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:23:41.375 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":28,"btime":"2024-09-15T23:22:37:131197+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:22:37.131175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24452,24436,24430,24428,24424]},"id":1}]} 2024-09-15T23:23:41.376 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 28 2024-09-15T23:23:41.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 -- 172.21.15.89:0/3179847725 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f05540a32d0 msgr2=0x7f05540a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:41.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f05540a32d0 0x7f05540a5790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0550000c00 tx=0x7f05500058d0 comp rx=0 tx=0).stop 2024-09-15T23:23:41.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 -- 172.21.15.89:0/3179847725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0568101e10 msgr2=0x7f05681984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:41.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0568101e10 0x7f05681984e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f055800b4a0 tx=0x7f055800b960 comp rx=0 tx=0).stop 2024-09-15T23:23:41.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 -- 172.21.15.89:0/3179847725 shutdown_connections 2024-09-15T23:23:41.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f05540a32d0 0x7f05540a5790 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:41.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0568106f40 0x7f056819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:41.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f05681027c0 0x7f0568198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:41.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 --2- 172.21.15.89:0/3179847725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0568101e10 0x7f05681984e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:41.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 -- 172.21.15.89:0/3179847725 >> 172.21.15.89:0/3179847725 conn(0x7f05680fd8d0 msgr2=0x7f05680fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:41.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 -- 172.21.15.89:0/3179847725 shutdown_connections 2024-09-15T23:23:41.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:41.376+0000 7f056e93d640 1 -- 172.21.15.89:0/3179847725 wait complete. 2024-09-15T23:23:42.133 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:42 smithi089 ceph-mon[30933]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:23:42.134 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:42 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3179847725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:42.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-15T23:21:53.066091+0000', 'modified': '2024-09-15T23:22:37.131175+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24452, 'mds_1': 24436, 'mds_2': 24430, 'mds_3': 24428, 'mds_4': 24424}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14610': {'gid': 14610, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.89:6839/2895111363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6838', 'nonce': 2895111363}, {'type': 'v1', 'addr': '172.21.15.89:6839', 'nonce': 2895111363}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14616': {'gid': 14616, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.89:6841/3365004630', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6840', 'nonce': 3365004630}, {'type': 'v1', 'addr': '172.21.15.89:6841', 'nonce': 3365004630}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24389': {'gid': 24389, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6833/4147766006', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6832', 'nonce': 4147766006}, {'type': 'v1', 'addr': '172.21.15.120:6833', 'nonce': 4147766006}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24403': {'gid': 24403, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6833/2504210581', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6832', 'nonce': 2504210581}, {'type': 'v1', 'addr': '172.21.15.174:6833', 'nonce': 2504210581}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24404': {'gid': 24404, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1429316223', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1429316223}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1429316223}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.174:6835/733541815', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 733541815}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 733541815}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24428': {'gid': 24428, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.120:6837/991556551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 991556551}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 991556551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.174:6837/3626951914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 3626951914}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 3626951914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.174:6839/3380568300', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 3380568300}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 3380568300}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.120:6839/1554218591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 1554218591}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 1554218591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24452, 'qdb_cluster': [24452, 24436, 24430, 24428, 24424]} 2024-09-15T23:23:42.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-15T23:23:42.164 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-15T23:23:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:42 smithi120 ceph-mon[35250]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:23:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:42 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3179847725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:42.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:42 smithi174 ceph-mon[35586]: pgmap v338: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:23:42.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:42 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3179847725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:42.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.932+0000 7f2b60f46640 1 -- 172.21.15.89:0/1975222796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b5c102790 msgr2=0x7f2b5c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.932+0000 7f2b60f46640 1 --2- 172.21.15.89:0/1975222796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b5c102790 0x7f2b5c1067e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2b50009900 tx=0x7f2b5002a410 comp rx=0 tx=0).stop 2024-09-15T23:23:42.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.932+0000 7f2b60f46640 1 -- 172.21.15.89:0/1975222796 shutdown_connections 2024-09-15T23:23:42.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.932+0000 7f2b60f46640 1 --2- 172.21.15.89:0/1975222796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b5c106f10 0x7f2b5c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.933+0000 7f2b60f46640 1 --2- 172.21.15.89:0/1975222796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b5c102790 0x7f2b5c1067e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.933+0000 7f2b60f46640 1 --2- 172.21.15.89:0/1975222796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b5c101de0 0x7f2b5c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.933+0000 7f2b60f46640 1 -- 172.21.15.89:0/1975222796 >> 172.21.15.89:0/1975222796 conn(0x7f2b5c0fd8c0 msgr2=0x7f2b5c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.933+0000 7f2b60f46640 1 -- 172.21.15.89:0/1975222796 shutdown_connections 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.933+0000 7f2b60f46640 1 -- 172.21.15.89:0/1975222796 wait complete. 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.933+0000 7f2b60f46640 1 Processor -- start 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.933+0000 7f2b60f46640 1 -- start start 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b60f46640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b5c101de0 0x7f2b5c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b60f46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b5c102790 0x7f2b5c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b60f46640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b5c106f10 0x7f2b5c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b60f46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c112010 con 0x7f2b5c102790 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b60f46640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c111e90 con 0x7f2b5c106f10 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b60f46640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5c112190 con 0x7f2b5c101de0 2024-09-15T23:23:42.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b5c102790 0x7f2b5c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b5c102790 0x7f2b5c198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33586/0 (socket says 172.21.15.89:33586) 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b59d9b640 1 -- 172.21.15.89:0/1803283619 learned_addr learned my addr 172.21.15.89:0/1803283619 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b5a59c640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b5c101de0 0x7f2b5c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b59d9b640 1 -- 172.21.15.89:0/1803283619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b5c101de0 msgr2=0x7f2b5c1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b5c101de0 0x7f2b5c1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b59d9b640 1 -- 172.21.15.89:0/1803283619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b5c106f10 msgr2=0x7f2b5c19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b5c106f10 0x7f2b5c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b59d9b640 1 -- 172.21.15.89:0/1803283619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b5c19d560 con 0x7f2b5c102790 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b5c102790 0x7f2b5c198af0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2b50005ab0 tx=0x7f2b500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b4b7fe640 1 -- 172.21.15.89:0/1803283619 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5003f030 con 0x7f2b5c102790 2024-09-15T23:23:42.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b4b7fe640 1 -- 172.21.15.89:0/1803283619 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b50003ce0 con 0x7f2b5c102790 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b4b7fe640 1 -- 172.21.15.89:0/1803283619 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50035070 con 0x7f2b5c102790 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b60f46640 1 -- 172.21.15.89:0/1803283619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b5c19d7f0 con 0x7f2b5c102790 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.934+0000 7f2b60f46640 1 -- 172.21.15.89:0/1803283619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b5c1a5130 con 0x7f2b5c102790 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.936+0000 7f2b4b7fe640 1 -- 172.21.15.89:0/1803283619 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2b50042030 con 0x7f2b5c102790 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b6331f740 1 Processor -- start 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b6331f740 1 -- start start 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d059c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d068a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d06b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b6331f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56014cfc0090 con 0x56014d059890 2024-09-15T23:23:42.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b6331f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56014cf67a70 con 0x56014d05a1b0 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b6331f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56014cfbc050 con 0x56014d068fc0 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b4b7fe640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b440a3350 0x7f2b440a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d059c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d059c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33602/0 (socket says 172.21.15.89:33602) 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b5ad9d640 1 -- 172.21.15.89:0/857980999 learned_addr learned my addr 172.21.15.89:0/857980999 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.937+0000 7f2b5ad9d640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b440a3350 0x7f2b440a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b4b7fe640 1 -- 172.21.15.89:0/1803283619 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f2b500f0740 con 0x7f2b5c102790 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b5a59c640 1 --2- 172.21.15.89:0/857980999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d068a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/857980999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d06b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b5ad9d640 1 -- 172.21.15.89:0/857980999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 msgr2=0x56014d06b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b5ad9d640 1 --2- 172.21.15.89:0/857980999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d06b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b5ad9d640 1 -- 172.21.15.89:0/857980999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 msgr2=0x56014d068a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b5ad9d640 1 --2- 172.21.15.89:0/857980999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d068a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b5ad9d640 1 -- 172.21.15.89:0/857980999 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56014cf74130 con 0x56014d059890 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b5ad9d640 1 --2- 172.21.15.89:0/857980999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d059c70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2b540127a0 tx=0x7f2b54012c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b8519e970c6062 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b5ad9d640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b440a3350 0x7f2b440a5810 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2b540145b0 tx=0x7f2b54019000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b48ff9640 1 -- 172.21.15.89:0/857980999 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b54014c50 con 0x56014d059890 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b48ff9640 1 -- 172.21.15.89:0/857980999 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b54014df0 con 0x56014d059890 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.938+0000 7f2b48ff9640 1 -- 172.21.15.89:0/857980999 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b540199c0 con 0x56014d059890 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.939+0000 7f2b6331f740 1 -- 172.21.15.89:0/857980999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 msgr2=0x56014d059c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.939+0000 7f2b6331f740 1 --2- 172.21.15.89:0/857980999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d059c70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2b540127a0 tx=0x7f2b54012c60 comp rx=0 tx=0).stop 2024-09-15T23:23:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.939+0000 7f2b6331f740 1 -- 172.21.15.89:0/857980999 shutdown_connections 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.939+0000 7f2b6331f740 1 --2- 172.21.15.89:0/857980999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d06b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.939+0000 7f2b6331f740 1 --2- 172.21.15.89:0/857980999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d068a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.939+0000 7f2b6331f740 1 --2- 172.21.15.89:0/857980999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d059c70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.939+0000 7f2b6331f740 1 -- 172.21.15.89:0/857980999 >> 172.21.15.89:0/857980999 conn(0x56014d055f60 msgr2=0x56014d056340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.939+0000 7f2b6331f740 1 -- 172.21.15.89:0/857980999 shutdown_connections 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.939+0000 7f2b6331f740 1 -- 172.21.15.89:0/857980999 wait complete. 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.939+0000 7f2b6331f740 1 -- start start 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b6331f740 1 Processor -- start 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d059890 0x56014d068910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d065c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d068fc0 0x56014d0661c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b6331f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56014d06bb30 con 0x56014d068fc0 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b6331f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56014cf7af20 con 0x56014d05a1b0 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b6331f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56014d06bcb0 con 0x56014d059890 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b6331f740 10 client.0 _ll_register_callbacks cb 0x56014cf44030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d068fc0 0x56014d0661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b5ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d068fc0 0x56014d0661c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33604/0 (socket says 172.21.15.89:33604) 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b5ad9d640 1 -- 172.21.15.89:0/1354113565 learned_addr learned my addr 172.21.15.89:0/1354113565 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b5a59c640 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d059890 0x56014d068910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d065c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b5ad9d640 1 -- 172.21.15.89:0/1354113565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d059890 msgr2=0x56014d068910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b5ad9d640 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d059890 0x56014d068910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b5ad9d640 1 -- 172.21.15.89:0/1354113565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 msgr2=0x56014d065c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b5ad9d640 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d065c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b5ad9d640 1 -- 172.21.15.89:0/1354113565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56014cd312b0 con 0x56014d068fc0 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d065c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b5ad9d640 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d068fc0 0x56014d0661c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2b540091f0 tx=0x7f2b5400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2917b337022149 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.940+0000 7f2b41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b41ffb640 1 -- 172.21.15.89:0/1354113565 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b54030060 con 0x56014d068fc0 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b41ffb640 1 -- 172.21.15.89:0/1354113565 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f2b540198c0 con 0x56014d068fc0 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b6331f740 1 -- 172.21.15.89:0/1354113565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b54014df0 con 0x56014d068fc0 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b41ffb640 1 -- 172.21.15.89:0/1354113565 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2b54037050 con 0x56014d068fc0 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b6331f740 10 client.15072 fetch_fsmap learned FSMap version 28 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b6331f740 1 -- 172.21.15.89:0/1354113565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56014d077540 con 0x56014d068fc0 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b41ffb640 1 -- 172.21.15.89:0/1354113565 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b54014df0 con 0x56014d068fc0 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b41ffb640 1 -- 172.21.15.89:0/1354113565 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f2b5402b780 con 0x56014d068fc0 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b6331f740 10 client.15072 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b6331f740 10 client.15072 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b6331f740 20 client.15072 populate_metadata read hostname 'smithi089' 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b6331f740 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56014d079810 0x56014d07ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b6331f740 4 client.15072 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b6331f740 1 -- 172.21.15.89:0/1354113565 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56014cfcd600 con 0x56014d079810 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.941+0000 7f2b5a59c640 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56014d079810 0x56014d07ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.942+0000 7f2b41ffb640 10 client.15072 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:23:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.942+0000 7f2b5a59c640 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56014d079810 0x56014d07ac80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=13b0f81a401d9d03 server_cookie=231d0537000f4e37 in_seq=0 out_seq=0 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b41ffb640 1 -- 172.21.15.89:0/1354113565 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56014cfcd600 con 0x56014d079810 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b41ffb640 10 client.15072 handle_command_reply: tid=0 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 client.15072 shutdown 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 -- 172.21.15.89:0/1354113565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d068fc0 msgr2=0x56014d0661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d068fc0 0x56014d0661c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2b540091f0 tx=0x7f2b5400bed0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 -- 172.21.15.89:0/1354113565 shutdown_connections 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56014d079810 0x56014d07ac80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d068fc0 0x56014d0661c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b41ffb640 0 client.15072 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d065c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 --2- 172.21.15.89:0/1354113565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d059890 0x56014d068910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 -- 172.21.15.89:0/1354113565 >> 172.21.15.89:0/1354113565 conn(0x56014d055f60 msgr2=0x56014d05bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 -- 172.21.15.89:0/1354113565 shutdown_connections 2024-09-15T23:23:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 1 -- 172.21.15.89:0/1354113565 wait complete. 2024-09-15T23:23:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.943+0000 7f2b6331f740 20 client.15072 trim_cache size 0 max 16384 2024-09-15T23:23:42.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.960+0000 7f2b6331f740 1 Processor -- start 2024-09-15T23:23:42.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b6331f740 1 -- start start 2024-09-15T23:23:42.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d059890 0x56014d079140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d05a1b0 0x56014d079ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d050ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b6331f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56014d078fc0 con 0x56014d05a1b0 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b6331f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56014d079690 con 0x56014d059890 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b6331f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56014d079d60 con 0x56014d068fc0 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d05a1b0 0x56014d079ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d05a1b0 0x56014d079ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33606/0 (socket says 172.21.15.89:33606) 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b59d9b640 1 -- 172.21.15.89:0/2655170772 learned_addr learned my addr 172.21.15.89:0/2655170772 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b59d9b640 1 -- 172.21.15.89:0/2655170772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 msgr2=0x56014d050ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/2655170772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d050ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b59d9b640 1 -- 172.21.15.89:0/2655170772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d059890 msgr2=0x56014d079140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b5a59c640 1 --2- 172.21.15.89:0/2655170772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d059890 0x56014d079140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/2655170772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d059890 0x56014d079140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b59d9b640 1 -- 172.21.15.89:0/2655170772 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56014d06bca0 con 0x56014d05a1b0 2024-09-15T23:23:42.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b5a59c640 1 --2- 172.21.15.89:0/2655170772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d059890 0x56014d079140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:42.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/2655170772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d05a1b0 0x56014d079ee0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2b500bc830 tx=0x7f2b5003b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd77ad4a4285716 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:42.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.961+0000 7f2b43fff640 1 -- 172.21.15.89:0/2655170772 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50049040 con 0x56014d05a1b0 2024-09-15T23:23:42.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b43fff640 1 -- 172.21.15.89:0/2655170772 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b500bc8a0 con 0x56014d05a1b0 2024-09-15T23:23:42.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b43fff640 1 -- 172.21.15.89:0/2655170772 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b500b8f10 con 0x56014d05a1b0 2024-09-15T23:23:42.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b6331f740 1 -- 172.21.15.89:0/2655170772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d05a1b0 msgr2=0x56014d079ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b6331f740 1 --2- 172.21.15.89:0/2655170772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d05a1b0 0x56014d079ee0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2b500bc830 tx=0x7f2b5003b800 comp rx=0 tx=0).stop 2024-09-15T23:23:42.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b6331f740 1 -- 172.21.15.89:0/2655170772 shutdown_connections 2024-09-15T23:23:42.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b6331f740 1 --2- 172.21.15.89:0/2655170772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d050ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b6331f740 1 --2- 172.21.15.89:0/2655170772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d05a1b0 0x56014d079ee0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b6331f740 1 --2- 172.21.15.89:0/2655170772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d059890 0x56014d079140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b6331f740 1 -- 172.21.15.89:0/2655170772 >> 172.21.15.89:0/2655170772 conn(0x56014d055f60 msgr2=0x56014d061240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:42.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b6331f740 1 -- 172.21.15.89:0/2655170772 shutdown_connections 2024-09-15T23:23:42.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b6331f740 1 -- 172.21.15.89:0/2655170772 wait complete. 2024-09-15T23:23:42.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.962+0000 7f2b6331f740 1 -- start start 2024-09-15T23:23:42.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b6331f740 1 Processor -- start 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d083230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d083770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b6331f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d07c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b6331f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56014d07ab00 con 0x56014d059890 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b6331f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56014d07a430 con 0x56014d05a1b0 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b6331f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56014d07b1d0 con 0x56014d068fc0 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b6331f740 10 client.0 _ll_register_callbacks cb 0x7f2b4c00b180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d083230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d083230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33614/0 (socket says 172.21.15.89:33614) 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5a59c640 1 -- 172.21.15.89:0/2370482410 learned_addr learned my addr 172.21.15.89:0/2370482410 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d083770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5ad9d640 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d07c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5a59c640 1 -- 172.21.15.89:0/2370482410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 msgr2=0x56014d07c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5a59c640 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d07c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5a59c640 1 -- 172.21.15.89:0/2370482410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 msgr2=0x56014d083770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5a59c640 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d083770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5a59c640 1 -- 172.21.15.89:0/2370482410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56014d066530 con 0x56014d059890 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5ad9d640 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d07c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b5a59c640 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d083230 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006e10 tx=0x7f2b4c017810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d440a0c46d657b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.963+0000 7f2b41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b41ffb640 1 -- 172.21.15.89:0/2370482410 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c015420 con 0x56014d059890 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b41ffb640 1 -- 172.21.15.89:0/2370482410 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f2b4c0155c0 con 0x56014d059890 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b6331f740 1 -- 172.21.15.89:0/2370482410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b500bc8a0 con 0x56014d059890 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b41ffb640 1 -- 172.21.15.89:0/2370482410 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2b4c02c050 con 0x56014d059890 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b6331f740 10 client.15078 fetch_fsmap learned FSMap version 28 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b6331f740 1 -- 172.21.15.89:0/2370482410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56014d081770 con 0x56014d059890 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b41ffb640 1 -- 172.21.15.89:0/2370482410 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b500bc8a0 con 0x56014d059890 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b41ffb640 1 -- 172.21.15.89:0/2370482410 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f2b4c024840 con 0x56014d059890 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b6331f740 10 client.15078 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b6331f740 10 client.15078 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b6331f740 20 client.15078 populate_metadata read hostname 'smithi089' 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b6331f740 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56014d079810 0x56014d0e71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b6331f740 4 client.15078 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.964+0000 7f2b6331f740 1 -- 172.21.15.89:0/2370482410 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "status"}) -- 0x56014cfc7b10 con 0x56014d079810 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56014d079810 0x56014d0e71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b59d9b640 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56014d079810 0x56014d0e71f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=da9c68cd3addb6d server_cookie=18ab33493cbbcab4 in_seq=0 out_seq=0 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b41ffb640 10 client.15078 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b41ffb640 1 -- 172.21.15.89:0/2370482410 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56014cfc7b10 con 0x56014d079810 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b41ffb640 10 client.15078 handle_command_reply: tid=0 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 client.15078 shutdown 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 -- 172.21.15.89:0/2370482410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 msgr2=0x56014d083230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d083230 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006e10 tx=0x7f2b4c017810 comp rx=0 tx=0).stop 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 -- 172.21.15.89:0/2370482410 shutdown_connections 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56014d079810 0x56014d0e71f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56014d068fc0 0x56014d07c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b41ffb640 0 client.15078 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56014d05a1b0 0x56014d083770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 --2- 172.21.15.89:0/2370482410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56014d059890 0x56014d083230 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 -- 172.21.15.89:0/2370482410 >> 172.21.15.89:0/2370482410 conn(0x56014d055f60 msgr2=0x56014d060a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 -- 172.21.15.89:0/2370482410 shutdown_connections 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.965+0000 7f2b6331f740 1 -- 172.21.15.89:0/2370482410 wait complete. 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b6331f740 20 client.15078 trim_cache size 0 max 16384 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 4, 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24424, 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 66.840048730000007, 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 28, 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 88, 2024-09-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 136.78416788999999 2024-09-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 -- 172.21.15.89:0/1803283619 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b440a3350 msgr2=0x7f2b440a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b440a3350 0x7f2b440a5810 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2b540145b0 tx=0x7f2b54019000 comp rx=0 tx=0).stop 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 -- 172.21.15.89:0/1803283619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b5c102790 msgr2=0x7f2b5c198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b5c102790 0x7f2b5c198af0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2b50005ab0 tx=0x7f2b500042f0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 -- 172.21.15.89:0/1803283619 shutdown_connections 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b440a3350 0x7f2b440a5810 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b5c106f10 0x7f2b5c19ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b5c102790 0x7f2b5c198af0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 --2- 172.21.15.89:0/1803283619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b5c101de0 0x7f2b5c1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 -- 172.21.15.89:0/1803283619 >> 172.21.15.89:0/1803283619 conn(0x7f2b5c0fd8c0 msgr2=0x7f2b5c0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 -- 172.21.15.89:0/1803283619 shutdown_connections 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:42.966+0000 7f2b60f46640 1 -- 172.21.15.89:0/1803283619 wait complete. 2024-09-15T23:23:43.848 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-15T23:23:44.093 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:44.213 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:44 smithi089 ceph-mon[30933]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:44.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:44 smithi120 ceph-mon[35250]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:44.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:44 smithi174 ceph-mon[35586]: pgmap v339: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:44.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.627+0000 7f4a6e49e640 1 -- 172.21.15.89:0/2032721826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68101de0 msgr2=0x7f4a681021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:44.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.627+0000 7f4a6e49e640 1 --2- 172.21.15.89:0/2032721826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68101de0 0x7f4a681021c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009900 tx=0x7f4a5802a410 comp rx=0 tx=0).stop 2024-09-15T23:23:44.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.627+0000 7f4a6e49e640 1 -- 172.21.15.89:0/2032721826 shutdown_connections 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.627+0000 7f4a6e49e640 1 --2- 172.21.15.89:0/2032721826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a68106f10 0x7f4a6810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.627+0000 7f4a6e49e640 1 --2- 172.21.15.89:0/2032721826 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a68102790 0x7f4a681067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.627+0000 7f4a6e49e640 1 --2- 172.21.15.89:0/2032721826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68101de0 0x7f4a681021c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.627+0000 7f4a6e49e640 1 -- 172.21.15.89:0/2032721826 >> 172.21.15.89:0/2032721826 conn(0x7f4a680fd8c0 msgr2=0x7f4a680ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.627+0000 7f4a6e49e640 1 -- 172.21.15.89:0/2032721826 shutdown_connections 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.627+0000 7f4a6e49e640 1 -- 172.21.15.89:0/2032721826 wait complete. 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.627+0000 7f4a6e49e640 1 Processor -- start 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a6e49e640 1 -- start start 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a6e49e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68101de0 0x7f4a680744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a6e49e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a68102790 0x7f4a68074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a6e49e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a68106f10 0x7f4a68074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a6e49e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a68111fb0 con 0x7f4a68101de0 2024-09-15T23:23:44.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a6e49e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4a68111e30 con 0x7f4a68102790 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a6e49e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a68112130 con 0x7f4a68106f10 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a67fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68101de0 0x7f4a680744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a67fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68101de0 0x7f4a680744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33628/0 (socket says 172.21.15.89:33628) 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a67fff640 1 -- 172.21.15.89:0/3627382837 learned_addr learned my addr 172.21.15.89:0/3627382837 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a677fe640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a68102790 0x7f4a68074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a6ca3b640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a68106f10 0x7f4a68074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a67fff640 1 -- 172.21.15.89:0/3627382837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a68106f10 msgr2=0x7f4a68074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a67fff640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a68106f10 0x7f4a68074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a67fff640 1 -- 172.21.15.89:0/3627382837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a68102790 msgr2=0x7f4a68074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a67fff640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a68102790 0x7f4a68074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a67fff640 1 -- 172.21.15.89:0/3627382837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a681a6950 con 0x7f4a68101de0 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.628+0000 7f4a6ca3b640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a68106f10 0x7f4a68074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:44.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.629+0000 7f4a67fff640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68101de0 0x7f4a680744f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4a58005ab0 tx=0x7f4a580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.629+0000 7f4a657fa640 1 -- 172.21.15.89:0/3627382837 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5803f030 con 0x7f4a68101de0 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.629+0000 7f4a6e49e640 1 -- 172.21.15.89:0/3627382837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a681a6be0 con 0x7f4a68101de0 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.629+0000 7f4a657fa640 1 -- 172.21.15.89:0/3627382837 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a58003ce0 con 0x7f4a68101de0 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.629+0000 7f4a657fa640 1 -- 172.21.15.89:0/3627382837 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58035070 con 0x7f4a68101de0 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.629+0000 7f4a6e49e640 1 -- 172.21.15.89:0/3627382837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a681a7110 con 0x7f4a68101de0 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.631+0000 7f4a657fa640 1 -- 172.21.15.89:0/3627382837 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4a58042030 con 0x7f4a68101de0 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.631+0000 7f4a70877740 1 Processor -- start 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.631+0000 7f4a70877740 1 -- start start 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a70877740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f6c9d0 0x55ee05f6cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a70877740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6d2f0 0x55ee05f77670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a70877740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f77bb0 0x55ee05f79fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a70877740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee05ed3090 con 0x55ee05f77bb0 2024-09-15T23:23:44.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a70877740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ee05e7aa70 con 0x55ee05f6d2f0 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a70877740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ee05ecf050 con 0x55ee05f6c9d0 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f77bb0 0x55ee05f79fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a657fa640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a500a32d0 0x7f4a500a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f77bb0 0x55ee05f79fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33642/0 (socket says 172.21.15.89:33642) 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a677fe640 1 -- 172.21.15.89:0/84681032 learned_addr learned my addr 172.21.15.89:0/84681032 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a6ca3b640 1 --2- 172.21.15.89:0/84681032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f6c9d0 0x55ee05f6cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a6ca3b640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a500a32d0 0x7f4a500a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a657fa640 1 -- 172.21.15.89:0/3627382837 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f4a580f0710 con 0x7f4a68101de0 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a67fff640 1 --2- 172.21.15.89:0/84681032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6d2f0 0x55ee05f77670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a677fe640 1 -- 172.21.15.89:0/84681032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f6c9d0 msgr2=0x55ee05f6cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a677fe640 1 --2- 172.21.15.89:0/84681032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f6c9d0 0x55ee05f6cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a677fe640 1 -- 172.21.15.89:0/84681032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6d2f0 msgr2=0x55ee05f77670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a677fe640 1 --2- 172.21.15.89:0/84681032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6d2f0 0x55ee05f77670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a677fe640 1 -- 172.21.15.89:0/84681032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee05e87130 con 0x55ee05f77bb0 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a677fe640 1 --2- 172.21.15.89:0/84681032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f77bb0 0x55ee05f79fa0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00abe0 tx=0x7f4a5c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804ab338cf5006f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a6ca3b640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a500a32d0 0x7f4a500a5790 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000e870 tx=0x7f4a60008220 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.632+0000 7f4a4e7fc640 1 -- 172.21.15.89:0/84681032 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c01d070 con 0x55ee05f77bb0 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a4e7fc640 1 -- 172.21.15.89:0/84681032 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c018b20 con 0x55ee05f77bb0 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a4e7fc640 1 -- 172.21.15.89:0/84681032 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c010ad0 con 0x55ee05f77bb0 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a70877740 1 -- 172.21.15.89:0/84681032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f77bb0 msgr2=0x55ee05f79fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a70877740 1 --2- 172.21.15.89:0/84681032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f77bb0 0x55ee05f79fa0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00abe0 tx=0x7f4a5c00f790 comp rx=0 tx=0).stop 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a70877740 1 -- 172.21.15.89:0/84681032 shutdown_connections 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a70877740 1 --2- 172.21.15.89:0/84681032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f77bb0 0x55ee05f79fa0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a70877740 1 --2- 172.21.15.89:0/84681032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6d2f0 0x55ee05f77670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a70877740 1 --2- 172.21.15.89:0/84681032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f6c9d0 0x55ee05f6cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a70877740 1 -- 172.21.15.89:0/84681032 >> 172.21.15.89:0/84681032 conn(0x55ee05f68f60 msgr2=0x55ee05f69340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a70877740 1 -- 172.21.15.89:0/84681032 shutdown_connections 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.633+0000 7f4a70877740 1 -- 172.21.15.89:0/84681032 wait complete. 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a70877740 1 -- start start 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a70877740 1 Processor -- start 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a70877740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f6c9d0 0x55ee05f773b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a70877740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f74700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a70877740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f77bb0 0x55ee05f74c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a70877740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee05f7a780 con 0x55ee05f6d2f0 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a70877740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ee05e8df20 con 0x55ee05f77bb0 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a70877740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ee05f7a900 con 0x55ee05f6c9d0 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a70877740 10 client.0 _ll_register_callbacks cb 0x55ee05e57030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f74700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f74700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33654/0 (socket says 172.21.15.89:33654) 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a677fe640 1 -- 172.21.15.89:0/4058085707 learned_addr learned my addr 172.21.15.89:0/4058085707 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a6ca3b640 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f77bb0 0x55ee05f74c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a67fff640 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f6c9d0 0x55ee05f773b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a677fe640 1 -- 172.21.15.89:0/4058085707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f6c9d0 msgr2=0x55ee05f773b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a677fe640 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f6c9d0 0x55ee05f773b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a677fe640 1 -- 172.21.15.89:0/4058085707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f77bb0 msgr2=0x55ee05f74c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a677fe640 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f77bb0 0x55ee05f74c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.634+0000 7f4a677fe640 1 -- 172.21.15.89:0/4058085707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee05c442b0 con 0x55ee05f6d2f0 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a6ca3b640 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f77bb0 0x55ee05f74c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a677fe640 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f74700 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00af20 tx=0x7f4a5c0186f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767c54d9df5dd6cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a477fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:23:44.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a477fe640 1 -- 172.21.15.89:0/4058085707 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c01d050 con 0x55ee05f6d2f0 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a477fe640 1 -- 172.21.15.89:0/4058085707 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f4a5c005040 con 0x55ee05f6d2f0 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a70877740 1 -- 172.21.15.89:0/4058085707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a5c018b20 con 0x55ee05f6d2f0 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a477fe640 1 -- 172.21.15.89:0/4058085707 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4a5c005240 con 0x55ee05f6d2f0 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a70877740 10 client.15090 fetch_fsmap learned FSMap version 28 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a70877740 1 -- 172.21.15.89:0/4058085707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ee05f85f80 con 0x55ee05f6d2f0 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a477fe640 1 -- 172.21.15.89:0/4058085707 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c018b20 con 0x55ee05f6d2f0 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a477fe640 1 -- 172.21.15.89:0/4058085707 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f4a5c004730 con 0x55ee05f6d2f0 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a70877740 10 client.15090 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a70877740 10 client.15090 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a70877740 20 client.15090 populate_metadata read hostname 'smithi089' 2024-09-15T23:23:44.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.635+0000 7f4a70877740 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55ee05f881f0 0x55ee05f89660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.636+0000 7f4a70877740 4 client.15090 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.636+0000 7f4a70877740 1 -- 172.21.15.89:0/4058085707 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee05ee0600 con 0x55ee05f881f0 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.636+0000 7f4a67fff640 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55ee05f881f0 0x55ee05f89660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.636+0000 7f4a67fff640 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55ee05f881f0 0x55ee05f89660 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=24898f0573f3d0de server_cookie=c3fd0e63a5866cc9 in_seq=0 out_seq=0 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.636+0000 7f4a477fe640 10 client.15090 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a477fe640 1 -- 172.21.15.89:0/4058085707 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ee05ee0600 con 0x55ee05f881f0 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a477fe640 10 client.15090 handle_command_reply: tid=0 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a70877740 1 client.15090 shutdown 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a70877740 1 -- 172.21.15.89:0/4058085707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 msgr2=0x55ee05f74700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a70877740 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f74700 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00af20 tx=0x7f4a5c0186f0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a70877740 1 -- 172.21.15.89:0/4058085707 shutdown_connections 2024-09-15T23:23:44.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a70877740 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55ee05f881f0 0x55ee05f89660 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a70877740 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f77bb0 0x55ee05f74c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a70877740 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f74700 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a70877740 1 --2- 172.21.15.89:0/4058085707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f6c9d0 0x55ee05f773b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.638+0000 7f4a70877740 1 -- 172.21.15.89:0/4058085707 >> 172.21.15.89:0/4058085707 conn(0x55ee05f68f60 msgr2=0x55ee05f6ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.637+0000 7f4a477fe640 0 client.15090 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:23:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.638+0000 7f4a70877740 1 -- 172.21.15.89:0/4058085707 shutdown_connections 2024-09-15T23:23:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.638+0000 7f4a70877740 1 -- 172.21.15.89:0/4058085707 wait complete. 2024-09-15T23:23:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.638+0000 7f4a70877740 20 client.15090 trim_cache size 0 max 16384 2024-09-15T23:23:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a70877740 1 Processor -- start 2024-09-15T23:23:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a70877740 1 -- start start 2024-09-15T23:23:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a70877740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6c9d0 0x55ee05f87b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a70877740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f88f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a70877740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 0x55ee05f6e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a70877740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee05f879a0 con 0x55ee05f6d2f0 2024-09-15T23:23:44.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a70877740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ee05f88070 con 0x55ee05f6c9d0 2024-09-15T23:23:44.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a70877740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ee05f88740 con 0x55ee05f77bb0 2024-09-15T23:23:44.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f88f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f88f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33656/0 (socket says 172.21.15.89:33656) 2024-09-15T23:23:44.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.655+0000 7f4a677fe640 1 -- 172.21.15.89:0/1715226056 learned_addr learned my addr 172.21.15.89:0/1715226056 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:44.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a6ca3b640 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 0x55ee05f6e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a67fff640 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6c9d0 0x55ee05f87b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a677fe640 1 -- 172.21.15.89:0/1715226056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 msgr2=0x55ee05f6e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:44.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a677fe640 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 0x55ee05f6e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a677fe640 1 -- 172.21.15.89:0/1715226056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6c9d0 msgr2=0x55ee05f87b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:44.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a677fe640 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6c9d0 0x55ee05f87b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a677fe640 1 -- 172.21.15.89:0/1715226056 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee05f6d120 con 0x55ee05f6d2f0 2024-09-15T23:23:44.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a67fff640 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6c9d0 0x55ee05f87b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:44.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a6ca3b640 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 0x55ee05f6e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:44.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a677fe640 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f88f90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c005e70 tx=0x7f4a5c018760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b05fff9a1b9ac1c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:44.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a4dffb640 1 -- 172.21.15.89:0/1715226056 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c01d040 con 0x55ee05f6d2f0 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a4dffb640 1 -- 172.21.15.89:0/1715226056 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c03b070 con 0x55ee05f6d2f0 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.656+0000 7f4a4dffb640 1 -- 172.21.15.89:0/1715226056 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c005020 con 0x55ee05f6d2f0 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 -- 172.21.15.89:0/1715226056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 msgr2=0x55ee05f88f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f88f90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c005e70 tx=0x7f4a5c018760 comp rx=0 tx=0).stop 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 -- 172.21.15.89:0/1715226056 shutdown_connections 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 0x55ee05f6e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6d2f0 0x55ee05f88f90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 --2- 172.21.15.89:0/1715226056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6c9d0 0x55ee05f87b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 -- 172.21.15.89:0/1715226056 >> 172.21.15.89:0/1715226056 conn(0x55ee05f68f60 msgr2=0x55ee05f6fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 -- 172.21.15.89:0/1715226056 shutdown_connections 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 -- 172.21.15.89:0/1715226056 wait complete. 2024-09-15T23:23:44.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 -- start start 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.657+0000 7f4a70877740 1 Processor -- start 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a70877740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6c9d0 0x55ee05f92140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a70877740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6d2f0 0x55ee05f8a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a70877740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 0x55ee05f8ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a70877740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee05f894e0 con 0x55ee05f6c9d0 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a70877740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ee05f88e10 con 0x55ee05f6d2f0 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a70877740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ee05f89bb0 con 0x55ee05f77bb0 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a70877740 10 client.0 _ll_register_callbacks cb 0x7f4a5c01b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a67fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6c9d0 0x55ee05f92140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a67fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6c9d0 0x55ee05f92140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33662/0 (socket says 172.21.15.89:33662) 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a67fff640 1 -- 172.21.15.89:0/365040998 learned_addr learned my addr 172.21.15.89:0/365040998 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a6ca3b640 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 0x55ee05f8ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a677fe640 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6d2f0 0x55ee05f8a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a67fff640 1 -- 172.21.15.89:0/365040998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 msgr2=0x55ee05f8ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a67fff640 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 0x55ee05f8ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a67fff640 1 -- 172.21.15.89:0/365040998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6d2f0 msgr2=0x55ee05f8a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a67fff640 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6d2f0 0x55ee05f8a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a67fff640 1 -- 172.21.15.89:0/365040998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee05f8b670 con 0x55ee05f6c9d0 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a67fff640 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6c9d0 0x55ee05f92140 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4a58031550 tx=0x7f4a580b8510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476a52410cd459a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.658+0000 7f4a477fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a477fe640 1 -- 172.21.15.89:0/365040998 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a58049040 con 0x55ee05f6c9d0 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a477fe640 1 -- 172.21.15.89:0/365040998 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f4a580edfb0 con 0x55ee05f6c9d0 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a70877740 1 -- 172.21.15.89:0/365040998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a5c03b070 con 0x55ee05f6c9d0 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a477fe640 1 -- 172.21.15.89:0/365040998 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4a580bc870 con 0x55ee05f6c9d0 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a70877740 10 client.15096 fetch_fsmap learned FSMap version 28 2024-09-15T23:23:44.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a70877740 1 -- 172.21.15.89:0/365040998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ee05ff1960 con 0x55ee05f6c9d0 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a477fe640 1 -- 172.21.15.89:0/365040998 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c03b070 con 0x55ee05f6c9d0 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a477fe640 1 -- 172.21.15.89:0/365040998 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f4a5c019ce0 con 0x55ee05f6c9d0 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a70877740 10 client.15096 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a70877740 10 client.15096 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a70877740 20 client.15096 populate_metadata read hostname 'smithi089' 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a70877740 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55ee05ff6580 0x55ee05ff5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a70877740 4 client.15096 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.659+0000 7f4a70877740 1 -- 172.21.15.89:0/365040998 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "status"}) -- 0x55ee05edab10 con 0x55ee05ff6580 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a677fe640 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55ee05ff6580 0x55ee05ff5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a677fe640 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55ee05ff6580 0x55ee05ff5290 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9918d202f4a6a673 server_cookie=37f9f3d5960778ee in_seq=0 out_seq=0 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a477fe640 10 client.15096 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a477fe640 1 -- 172.21.15.89:0/365040998 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ee05edab10 con 0x55ee05ff6580 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a477fe640 10 client.15096 handle_command_reply: tid=0 2024-09-15T23:23:44.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a70877740 1 client.15096 shutdown 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a70877740 1 -- 172.21.15.89:0/365040998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6c9d0 msgr2=0x55ee05f92140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a70877740 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6c9d0 0x55ee05f92140 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4a58031550 tx=0x7f4a580b8510 comp rx=0 tx=0).stop 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a70877740 1 -- 172.21.15.89:0/365040998 shutdown_connections 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a70877740 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55ee05ff6580 0x55ee05ff5290 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a70877740 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee05f77bb0 0x55ee05f8ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a70877740 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee05f6d2f0 0x55ee05f8a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a477fe640 0 client.15096 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a70877740 1 --2- 172.21.15.89:0/365040998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee05f6c9d0 0x55ee05f92140 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.660+0000 7f4a70877740 1 -- 172.21.15.89:0/365040998 >> 172.21.15.89:0/365040998 conn(0x55ee05f68f60 msgr2=0x55ee05f677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a70877740 1 -- 172.21.15.89:0/365040998 shutdown_connections 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a70877740 1 -- 172.21.15.89:0/365040998 wait complete. 2024-09-15T23:23:44.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a70877740 20 client.15096 trim_cache size 0 max 16384 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 3, 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24428, 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 69.549623992999997, 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 28, 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 88, 2024-09-15T23:23:44.671 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:23:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 133.21719328899999 2024-09-15T23:23:44.672 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:23:44.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a6e49e640 1 -- 172.21.15.89:0/3627382837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a500a32d0 msgr2=0x7f4a500a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:44.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a6e49e640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a500a32d0 0x7f4a500a5790 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000e870 tx=0x7f4a60008220 comp rx=0 tx=0).stop 2024-09-15T23:23:44.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a6e49e640 1 -- 172.21.15.89:0/3627382837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68101de0 msgr2=0x7f4a680744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:44.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a6e49e640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68101de0 0x7f4a680744f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4a58005ab0 tx=0x7f4a580042f0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a6e49e640 1 -- 172.21.15.89:0/3627382837 shutdown_connections 2024-09-15T23:23:44.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a6e49e640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a500a32d0 0x7f4a500a5790 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a6e49e640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a68106f10 0x7f4a68074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a6e49e640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a68102790 0x7f4a68074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a6e49e640 1 --2- 172.21.15.89:0/3627382837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a68101de0 0x7f4a680744f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:44.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.661+0000 7f4a6e49e640 1 -- 172.21.15.89:0/3627382837 >> 172.21.15.89:0/3627382837 conn(0x7f4a680fd8c0 msgr2=0x7f4a680fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:44.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.662+0000 7f4a6e49e640 1 -- 172.21.15.89:0/3627382837 shutdown_connections 2024-09-15T23:23:44.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:44.662+0000 7f4a6e49e640 1 -- 172.21.15.89:0/3627382837 wait complete. 2024-09-15T23:23:45.583 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-15T23:23:45.830 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:46.250 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:46 smithi089 ceph-mon[30933]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:46.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.329+0000 7fedc03b6640 1 -- 172.21.15.89:0/3302405206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8101420 msgr2=0x7fedb810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:46.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.329+0000 7fedc03b6640 1 --2- 172.21.15.89:0/3302405206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8101420 0x7fedb810d950 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009cd0 tx=0x7fedb002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.329+0000 7fedc03b6640 1 -- 172.21.15.89:0/3302405206 shutdown_connections 2024-09-15T23:23:46.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.329+0000 7fedc03b6640 1 --2- 172.21.15.89:0/3302405206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8101420 0x7fedb810d950 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.329+0000 7fedc03b6640 1 --2- 172.21.15.89:0/3302405206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedb8100a80 0x7fedb8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.329+0000 7fedc03b6640 1 --2- 172.21.15.89:0/3302405206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb8106a80 0x7fedb8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.329+0000 7fedc03b6640 1 -- 172.21.15.89:0/3302405206 >> 172.21.15.89:0/3302405206 conn(0x7fedb80fc7a0 msgr2=0x7fedb80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.329+0000 7fedc03b6640 1 -- 172.21.15.89:0/3302405206 shutdown_connections 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.329+0000 7fedc03b6640 1 -- 172.21.15.89:0/3302405206 wait complete. 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedc03b6640 1 Processor -- start 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedc03b6640 1 -- start start 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedc03b6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8100a80 0x7fedb819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedc03b6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb8101420 0x7fedb819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedc03b6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedb8106a80 0x7fedb819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedc03b6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fedb8110ea0 con 0x7fedb8100a80 2024-09-15T23:23:46.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedc03b6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fedb8110d20 con 0x7fedb8106a80 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedc03b6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fedb8111020 con 0x7fedb8101420 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbe152640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8100a80 0x7fedb819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbe152640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8100a80 0x7fedb819a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38970/0 (socket says 172.21.15.89:38970) 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbe152640 1 -- 172.21.15.89:0/2404560145 learned_addr learned my addr 172.21.15.89:0/2404560145 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbe953640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedb8106a80 0x7fedb819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbd951640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb8101420 0x7fedb819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbe152640 1 -- 172.21.15.89:0/2404560145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb8101420 msgr2=0x7fedb819ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbe152640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb8101420 0x7fedb819ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbe152640 1 -- 172.21.15.89:0/2404560145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedb8106a80 msgr2=0x7fedb819eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbe152640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedb8106a80 0x7fedb819eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbe152640 1 -- 172.21.15.89:0/2404560145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb819f5c0 con 0x7fedb8100a80 2024-09-15T23:23:46.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbe953640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedb8106a80 0x7fedb819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.330+0000 7fedbd951640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb8101420 0x7fedb819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.331+0000 7fedbe152640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8100a80 0x7fedb819a610 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fedb400b4a0 tx=0x7fedb400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.331+0000 7fedab7fe640 1 -- 172.21.15.89:0/2404560145 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb4014070 con 0x7fedb8100a80 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.331+0000 7fedc03b6640 1 -- 172.21.15.89:0/2404560145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb819f8b0 con 0x7fedb8100a80 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.331+0000 7fedab7fe640 1 -- 172.21.15.89:0/2404560145 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb4010af0 con 0x7fedb8100a80 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.331+0000 7fedab7fe640 1 -- 172.21.15.89:0/2404560145 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb4017d10 con 0x7fedb8100a80 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.331+0000 7fedc03b6640 1 -- 172.21.15.89:0/2404560145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb81a7190 con 0x7fedb8100a80 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.333+0000 7fedab7fe640 1 -- 172.21.15.89:0/2404560145 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fedb4017070 con 0x7fedb8100a80 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.333+0000 7fedc278f740 1 Processor -- start 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.333+0000 7fedc278f740 1 -- start start 2024-09-15T23:23:46.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedc278f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f77db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedc278f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f82670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedc278f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f82bb0 0x564a76f84fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedc278f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564a76ede090 con 0x564a76f779d0 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedab7fe640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feda40a32d0 0x7feda40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedc278f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564a76e85a70 con 0x564a76f782f0 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedc278f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564a76eda050 con 0x564a76f82bb0 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe953640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f77db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe953640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f77db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38974/0 (socket says 172.21.15.89:38974) 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe953640 1 -- 172.21.15.89:0/1621623872 learned_addr learned my addr 172.21.15.89:0/1621623872 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe953640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feda40a32d0 0x7feda40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedab7fe640 1 -- 172.21.15.89:0/2404560145 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fedb40cf170 con 0x7fedb8100a80 2024-09-15T23:23:46.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe152640 1 --2- 172.21.15.89:0/1621623872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f82670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbd951640 1 --2- 172.21.15.89:0/1621623872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f82bb0 0x564a76f84fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe953640 1 -- 172.21.15.89:0/1621623872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f82bb0 msgr2=0x564a76f84fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe953640 1 --2- 172.21.15.89:0/1621623872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f82bb0 0x564a76f84fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe953640 1 -- 172.21.15.89:0/1621623872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 msgr2=0x564a76f82670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe953640 1 --2- 172.21.15.89:0/1621623872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f82670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe953640 1 -- 172.21.15.89:0/1621623872 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a76e92130 con 0x564a76f779d0 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.334+0000 7fedbe953640 1 --2- 172.21.15.89:0/1621623872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f77db0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fedb0031d00 tx=0x7fedb0031d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f76e8c2b17bd5c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7fedbe953640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feda40a32d0 0x7feda40a5790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fedb0031da0 tx=0x7fedb0048000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7feda8dea640 1 -- 172.21.15.89:0/1621623872 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb004e070 con 0x564a76f779d0 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7feda8dea640 1 -- 172.21.15.89:0/1621623872 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb0036040 con 0x564a76f779d0 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7feda8dea640 1 -- 172.21.15.89:0/1621623872 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb00307e0 con 0x564a76f779d0 2024-09-15T23:23:46.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7fedc278f740 1 -- 172.21.15.89:0/1621623872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 msgr2=0x564a76f77db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7fedc278f740 1 --2- 172.21.15.89:0/1621623872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f77db0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fedb0031d00 tx=0x7fedb0031d30 comp rx=0 tx=0).stop 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7fedc278f740 1 -- 172.21.15.89:0/1621623872 shutdown_connections 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7fedc278f740 1 --2- 172.21.15.89:0/1621623872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f82bb0 0x564a76f84fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7fedc278f740 1 --2- 172.21.15.89:0/1621623872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f82670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7fedc278f740 1 --2- 172.21.15.89:0/1621623872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f77db0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.335+0000 7fedc278f740 1 -- 172.21.15.89:0/1621623872 >> 172.21.15.89:0/1621623872 conn(0x564a76f73f60 msgr2=0x564a76f74340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 1 -- 172.21.15.89:0/1621623872 shutdown_connections 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 1 -- 172.21.15.89:0/1621623872 wait complete. 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 1 -- start start 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 1 Processor -- start 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f779d0 0x564a76f824f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f7f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f82bb0 0x564a76f7fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564a76f85780 con 0x564a76f82bb0 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564a76e98f20 con 0x564a76f782f0 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564a76f85900 con 0x564a76f779d0 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedc278f740 10 client.0 _ll_register_callbacks cb 0x564a76e62030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedbe953640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f82bb0 0x564a76f7fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedbe953640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f82bb0 0x564a76f7fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38982/0 (socket says 172.21.15.89:38982) 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedbe953640 1 -- 172.21.15.89:0/3705835448 learned_addr learned my addr 172.21.15.89:0/3705835448 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.336+0000 7fedbd951640 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f7f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedbe152640 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f779d0 0x564a76f824f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedbe953640 1 -- 172.21.15.89:0/3705835448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f779d0 msgr2=0x564a76f824f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedbe953640 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f779d0 0x564a76f824f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedbe953640 1 -- 172.21.15.89:0/3705835448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 msgr2=0x564a76f7f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedbe953640 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f7f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedbe953640 1 -- 172.21.15.89:0/3705835448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a76c4f2b0 con 0x564a76f82bb0 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedbe953640 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f82bb0 0x564a76f7fd80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fedb00307d0 tx=0x7fedb0005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0934a85c9b99395 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7feda1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7feda1ffb640 1 -- 172.21.15.89:0/3705835448 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0057070 con 0x564a76f82bb0 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7feda1ffb640 1 -- 172.21.15.89:0/3705835448 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fedb0004310 con 0x564a76f82bb0 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedc278f740 1 -- 172.21.15.89:0/3705835448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedb0036040 con 0x564a76f82bb0 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7feda1ffb640 1 -- 172.21.15.89:0/3705835448 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fedb004e050 con 0x564a76f82bb0 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedc278f740 10 client.15108 fetch_fsmap learned FSMap version 28 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedc278f740 1 -- 172.21.15.89:0/3705835448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564a76f91130 con 0x564a76f82bb0 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7feda1ffb640 1 -- 172.21.15.89:0/3705835448 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb0036040 con 0x564a76f82bb0 2024-09-15T23:23:46.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7feda1ffb640 1 -- 172.21.15.89:0/3705835448 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fedb0030070 con 0x564a76f82bb0 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.337+0000 7fedc278f740 10 client.15108 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.338+0000 7fedc278f740 10 client.15108 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.338+0000 7fedc278f740 20 client.15108 populate_metadata read hostname 'smithi089' 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.338+0000 7fedc278f740 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564a76f93370 0x564a76f947e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.338+0000 7fedc278f740 4 client.15108 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.338+0000 7fedc278f740 1 -- 172.21.15.89:0/3705835448 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a76eeb600 con 0x564a76f93370 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.338+0000 7fedbe152640 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564a76f93370 0x564a76f947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.338+0000 7feda1ffb640 10 client.15108 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.338+0000 7fedbe152640 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564a76f93370 0x564a76f947e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=21668ade20b463bb server_cookie=5efcbb0d48c7626f in_seq=0 out_seq=0 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7feda1ffb640 1 -- 172.21.15.89:0/3705835448 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564a76eeb600 con 0x564a76f93370 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7feda1ffb640 10 client.15108 handle_command_reply: tid=0 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 client.15108 shutdown 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 -- 172.21.15.89:0/3705835448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f82bb0 msgr2=0x564a76f7fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:46.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f82bb0 0x564a76f7fd80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fedb00307d0 tx=0x7fedb0005f80 comp rx=0 tx=0).stop 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 -- 172.21.15.89:0/3705835448 shutdown_connections 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f82bb0 0x564a76f7fd80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f7f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564a76f93370 0x564a76f947e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 --2- 172.21.15.89:0/3705835448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f779d0 0x564a76f824f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7feda1ffb640 0 client.15108 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 -- 172.21.15.89:0/3705835448 >> 172.21.15.89:0/3705835448 conn(0x564a76f73f60 msgr2=0x564a76f79c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 -- 172.21.15.89:0/3705835448 shutdown_connections 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.339+0000 7fedc278f740 1 -- 172.21.15.89:0/3705835448 wait complete. 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.340+0000 7fedc278f740 20 client.15108 trim_cache size 0 max 16384 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedc278f740 1 Processor -- start 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedc278f740 1 -- start start 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedc278f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f947e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedc278f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f782f0 0x564a76f9cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedc278f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f82bb0 0x564a76f9d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedc278f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564a76f92b20 con 0x564a76f779d0 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedc278f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564a76f931f0 con 0x564a76f82bb0 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedc278f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564a76f938c0 con 0x564a76f782f0 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedbe152640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedbe152640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f947e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38994/0 (socket says 172.21.15.89:38994) 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedbe152640 1 -- 172.21.15.89:0/759641144 learned_addr learned my addr 172.21.15.89:0/759641144 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedbe953640 1 --2- 172.21.15.89:0/759641144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f82bb0 0x564a76f9d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedbe152640 1 -- 172.21.15.89:0/759641144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f782f0 msgr2=0x564a76f9cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedbe152640 1 --2- 172.21.15.89:0/759641144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f782f0 0x564a76f9cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedbe152640 1 -- 172.21.15.89:0/759641144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f82bb0 msgr2=0x564a76f9d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedbe152640 1 --2- 172.21.15.89:0/759641144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f82bb0 0x564a76f9d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedbe152640 1 -- 172.21.15.89:0/759641144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a76f85310 con 0x564a76f779d0 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.357+0000 7fedbe953640 1 --2- 172.21.15.89:0/759641144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f82bb0 0x564a76f9d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7fedbe152640 1 --2- 172.21.15.89:0/759641144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f947e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fedb4097840 tx=0x7fedb4012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c7499e644820db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7feda3fff640 1 -- 172.21.15.89:0/759641144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb40b3070 con 0x564a76f779d0 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7feda3fff640 1 -- 172.21.15.89:0/759641144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb40a4050 con 0x564a76f779d0 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7feda3fff640 1 -- 172.21.15.89:0/759641144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb401e040 con 0x564a76f779d0 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7fedc278f740 1 -- 172.21.15.89:0/759641144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 msgr2=0x564a76f947e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7fedc278f740 1 --2- 172.21.15.89:0/759641144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f947e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fedb4097840 tx=0x7fedb4012040 comp rx=0 tx=0).stop 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7fedc278f740 1 -- 172.21.15.89:0/759641144 shutdown_connections 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7fedc278f740 1 --2- 172.21.15.89:0/759641144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f82bb0 0x564a76f9d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7fedc278f740 1 --2- 172.21.15.89:0/759641144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f782f0 0x564a76f9cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7fedc278f740 1 --2- 172.21.15.89:0/759641144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f947e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7fedc278f740 1 -- 172.21.15.89:0/759641144 >> 172.21.15.89:0/759641144 conn(0x564a76f73f60 msgr2=0x564a76f6ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:46.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7fedc278f740 1 -- 172.21.15.89:0/759641144 shutdown_connections 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.358+0000 7fedc278f740 1 -- 172.21.15.89:0/759641144 wait complete. 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedc278f740 1 -- start start 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedc278f740 1 Processor -- start 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedc278f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f9cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedc278f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f982d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedc278f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f82bb0 0x564a76f98810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedc278f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564a76f94660 con 0x564a76f779d0 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedc278f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564a76f93f90 con 0x564a76f782f0 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedc278f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564a76f94d30 con 0x564a76f82bb0 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedc278f740 10 client.0 _ll_register_callbacks cb 0x7fedb4017eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedbe152640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f9cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedbe152640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f9cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39008/0 (socket says 172.21.15.89:39008) 2024-09-15T23:23:46.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedbe152640 1 -- 172.21.15.89:0/162105769 learned_addr learned my addr 172.21.15.89:0/162105769 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedbe152640 1 -- 172.21.15.89:0/162105769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f82bb0 msgr2=0x564a76f98810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedbe152640 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f82bb0 0x564a76f98810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedbe152640 1 -- 172.21.15.89:0/162105769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 msgr2=0x564a76f982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedbe152640 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.359+0000 7fedbe152640 1 -- 172.21.15.89:0/162105769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a76f98dd0 con 0x564a76f779d0 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7fedbe152640 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f9cfa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fedb409b370 tx=0x7fedb4017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888efa16234fe3fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7feda1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7feda1ffb640 1 -- 172.21.15.89:0/162105769 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb4065060 con 0x564a76f779d0 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7feda1ffb640 1 -- 172.21.15.89:0/162105769 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fedb4010ca0 con 0x564a76f779d0 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7fedc278f740 1 -- 172.21.15.89:0/162105769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedb40a4050 con 0x564a76f779d0 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7feda1ffb640 1 -- 172.21.15.89:0/162105769 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fedb401e030 con 0x564a76f779d0 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7fedc278f740 10 client.15114 fetch_fsmap learned FSMap version 28 2024-09-15T23:23:46.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7fedc278f740 1 -- 172.21.15.89:0/162105769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564a76ffd7a0 con 0x564a76f779d0 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7feda1ffb640 1 -- 172.21.15.89:0/162105769 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb40a4050 con 0x564a76f779d0 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7feda1ffb640 1 -- 172.21.15.89:0/162105769 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fedb4097a50 con 0x564a76f779d0 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7fedc278f740 10 client.15114 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7fedc278f740 10 client.15114 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7fedc278f740 20 client.15114 populate_metadata read hostname 'smithi089' 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7fedc278f740 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564a77001fe0 0x564a77000cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7fedc278f740 4 client.15114 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.360+0000 7fedc278f740 1 -- 172.21.15.89:0/162105769 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "status"}) -- 0x564a76ee5b10 con 0x564a77001fe0 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedbd951640 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564a77001fe0 0x564a77000cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedbd951640 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564a77001fe0 0x564a77000cf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b42f640855e7d59 server_cookie=61c35404b6764353 in_seq=0 out_seq=0 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7feda1ffb640 10 client.15114 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7feda1ffb640 1 -- 172.21.15.89:0/162105769 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564a76ee5b10 con 0x564a77001fe0 2024-09-15T23:23:46.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7feda1ffb640 10 client.15114 handle_command_reply: tid=0 2024-09-15T23:23:46.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 client.15114 shutdown 2024-09-15T23:23:46.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 -- 172.21.15.89:0/162105769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 msgr2=0x564a76f9cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:46.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f9cfa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fedb409b370 tx=0x7fedb4017730 comp rx=0 tx=0).stop 2024-09-15T23:23:46.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 -- 172.21.15.89:0/162105769 shutdown_connections 2024-09-15T23:23:46.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564a77001fe0 0x564a77000cf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.434 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:23:46.434 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:23:46.434 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 2, 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24430, 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 72.267211145000005, 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 28, 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 88, 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 137.600731475 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:23:46.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a76f82bb0 0x564a76f98810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a76f782f0 0x564a76f982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7feda1ffb640 0 client.15114 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 --2- 172.21.15.89:0/162105769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a76f779d0 0x564a76f9cfa0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 -- 172.21.15.89:0/162105769 >> 172.21.15.89:0/162105769 conn(0x564a76f73f60 msgr2=0x564a76f80eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 -- 172.21.15.89:0/162105769 shutdown_connections 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.361+0000 7fedc278f740 1 -- 172.21.15.89:0/162105769 wait complete. 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc278f740 20 client.15114 trim_cache size 0 max 16384 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 -- 172.21.15.89:0/2404560145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feda40a32d0 msgr2=0x7feda40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feda40a32d0 0x7feda40a5790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fedb0031da0 tx=0x7fedb0048000 comp rx=0 tx=0).stop 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 -- 172.21.15.89:0/2404560145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8100a80 msgr2=0x7fedb819a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8100a80 0x7fedb819a610 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fedb400b4a0 tx=0x7fedb400b960 comp rx=0 tx=0).stop 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 -- 172.21.15.89:0/2404560145 shutdown_connections 2024-09-15T23:23:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feda40a32d0 0x7feda40a5790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedb8106a80 0x7fedb819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb8101420 0x7fedb819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 --2- 172.21.15.89:0/2404560145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedb8100a80 0x7fedb819a610 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:46.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 -- 172.21.15.89:0/2404560145 >> 172.21.15.89:0/2404560145 conn(0x7fedb80fc7a0 msgr2=0x7fedb806a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:46.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 -- 172.21.15.89:0/2404560145 shutdown_connections 2024-09-15T23:23:46.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:46.362+0000 7fedc03b6640 1 -- 172.21.15.89:0/2404560145 wait complete. 2024-09-15T23:23:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:46 smithi120 ceph-mon[35250]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:46 smithi174 ceph-mon[35586]: pgmap v340: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:47.150 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-15T23:23:47.394 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:47.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 -- 172.21.15.89:0/302282650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf4101610 msgr2=0x7f0cf41019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:47.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 --2- 172.21.15.89:0/302282650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf4101610 0x7f0cf41019f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009900 tx=0x7f0ce402a410 comp rx=0 tx=0).stop 2024-09-15T23:23:47.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 -- 172.21.15.89:0/302282650 shutdown_connections 2024-09-15T23:23:47.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 --2- 172.21.15.89:0/302282650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4105e40 0x7f0cf410d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 --2- 172.21.15.89:0/302282650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cf4101fc0 0x7f0cf4105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 --2- 172.21.15.89:0/302282650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf4101610 0x7f0cf41019f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 -- 172.21.15.89:0/302282650 >> 172.21.15.89:0/302282650 conn(0x7f0cf40fc5c0 msgr2=0x7f0cf40fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 -- 172.21.15.89:0/302282650 shutdown_connections 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 -- 172.21.15.89:0/302282650 wait complete. 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 Processor -- start 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.887+0000 7f0cfc339640 1 -- start start 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfc339640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cf4101610 0x7f0cf41971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfc339640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4101fc0 0x7f0cf41976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfc339640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf4105e40 0x7f0cf4197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfc339640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf4110ee0 con 0x7f0cf4105e40 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfc339640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf4110d60 con 0x7f0cf4101fc0 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfc339640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf4111060 con 0x7f0cf4101610 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfa8d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf4105e40 0x7f0cf4197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfa8d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf4105e40 0x7f0cf4197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39014/0 (socket says 172.21.15.89:39014) 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfa8d6640 1 -- 172.21.15.89:0/2892167763 learned_addr learned my addr 172.21.15.89:0/2892167763 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfa0d5640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cf4101610 0x7f0cf41971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4101fc0 0x7f0cf41976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfa8d6640 1 -- 172.21.15.89:0/2892167763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cf4101610 msgr2=0x7f0cf41971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfa8d6640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cf4101610 0x7f0cf41971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfa8d6640 1 -- 172.21.15.89:0/2892167763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4101fc0 msgr2=0x7f0cf41976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfa8d6640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4101fc0 0x7f0cf41976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfa8d6640 1 -- 172.21.15.89:0/2892167763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf41a6a10 con 0x7f0cf4105e40 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4101fc0 0x7f0cf41976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfa8d6640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf4105e40 0x7f0cf4197c20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00b970 tx=0x7f0cec00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0ceb7fe640 1 -- 172.21.15.89:0/2892167763 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec019070 con 0x7f0cf4105e40 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfc339640 1 -- 172.21.15.89:0/2892167763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf41a6d00 con 0x7f0cf4105e40 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0ceb7fe640 1 -- 172.21.15.89:0/2892167763 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cec00cb40 con 0x7f0cf4105e40 2024-09-15T23:23:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0ceb7fe640 1 -- 172.21.15.89:0/2892167763 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec007650 con 0x7f0cf4105e40 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.888+0000 7f0cfc339640 1 -- 172.21.15.89:0/2892167763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf41a71f0 con 0x7f0cf4105e40 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0ceb7fe640 1 -- 172.21.15.89:0/2892167763 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0cec01c070 con 0x7f0cf4105e40 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfe712740 1 Processor -- start 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfe712740 1 -- start start 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763556db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d763561670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d763561bb0 0x55d763563fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfe712740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d7634bd090 con 0x55d7635572f0 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfe712740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d763464a70 con 0x55d763561bb0 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfe712740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d7634b9050 con 0x55d7635569d0 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0ceb7fe640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ce00a32d0 0x7f0ce00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfa0d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d763561670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfa0d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d763561670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39016/0 (socket says 172.21.15.89:39016) 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.891+0000 7f0cfa0d5640 1 -- 172.21.15.89:0/2115504190 learned_addr learned my addr 172.21.15.89:0/2115504190 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cfa8d6640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ce00a32d0 0x7f0ce00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/2115504190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d763561bb0 0x55d763563fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/2115504190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763556db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0ceb7fe640 1 -- 172.21.15.89:0/2892167763 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f0cec0cf290 con 0x7f0cf4105e40 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cfa0d5640 1 -- 172.21.15.89:0/2115504190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 msgr2=0x55d763556db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cfa0d5640 1 --2- 172.21.15.89:0/2115504190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763556db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cfa0d5640 1 -- 172.21.15.89:0/2115504190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d763561bb0 msgr2=0x55d763563fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cfa0d5640 1 --2- 172.21.15.89:0/2115504190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d763561bb0 0x55d763563fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cfa0d5640 1 -- 172.21.15.89:0/2115504190 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d763471130 con 0x55d7635572f0 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/2115504190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d763561bb0 0x55d763563fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cfa0d5640 1 --2- 172.21.15.89:0/2115504190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d763561670 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009900 tx=0x7f0ce40316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9c403f440ee722 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0cfa8d6640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ce00a32d0 0x7f0ce00a5790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0cec097f80 tx=0x7f0cec009970 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0ce8dea640 1 -- 172.21.15.89:0/2115504190 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4040070 con 0x55d7635572f0 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0ce8dea640 1 -- 172.21.15.89:0/2115504190 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ce4005020 con 0x55d7635572f0 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.892+0000 7f0ce8dea640 1 -- 172.21.15.89:0/2115504190 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4035040 con 0x55d7635572f0 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 -- 172.21.15.89:0/2115504190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 msgr2=0x55d763561670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 --2- 172.21.15.89:0/2115504190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d763561670 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009900 tx=0x7f0ce40316e0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 -- 172.21.15.89:0/2115504190 shutdown_connections 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 --2- 172.21.15.89:0/2115504190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d763561bb0 0x55d763563fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 --2- 172.21.15.89:0/2115504190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d763561670 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 --2- 172.21.15.89:0/2115504190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763556db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 -- 172.21.15.89:0/2115504190 >> 172.21.15.89:0/2115504190 conn(0x55d763552f60 msgr2=0x55d763553340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 -- 172.21.15.89:0/2115504190 shutdown_connections 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 -- 172.21.15.89:0/2115504190 wait complete. 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 -- start start 2024-09-15T23:23:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.893+0000 7f0cfe712740 1 Processor -- start 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763561350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d76355e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d763561bb0 0x55d76355ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfe712740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d763564780 con 0x55d7635572f0 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfe712740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d763477f20 con 0x55d763561bb0 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfe712740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d763564900 con 0x55d7635569d0 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfe712740 10 client.0 _ll_register_callbacks cb 0x55d763441030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfa0d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d76355e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfa0d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d76355e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39024/0 (socket says 172.21.15.89:39024) 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfa0d5640 1 -- 172.21.15.89:0/3866878328 learned_addr learned my addr 172.21.15.89:0/3866878328 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763561350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfa0d5640 1 -- 172.21.15.89:0/3866878328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 msgr2=0x55d763561350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfa0d5640 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763561350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfa0d5640 1 -- 172.21.15.89:0/3866878328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d763561bb0 msgr2=0x55d76355ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfa0d5640 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d763561bb0 0x55d76355ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfa0d5640 1 -- 172.21.15.89:0/3866878328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d76322e2b0 con 0x55d7635572f0 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfa0d5640 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d76355e6a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0ce402a930 tx=0x7f0ce4009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f2babf4dbcc1ae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cddffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cddffb640 1 -- 172.21.15.89:0/3866878328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce404c020 con 0x55d7635572f0 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cddffb640 1 -- 172.21.15.89:0/3866878328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f0ce40048f0 con 0x55d7635572f0 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.894+0000 7f0cfe712740 1 -- 172.21.15.89:0/3866878328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ce4005020 con 0x55d7635572f0 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cddffb640 1 -- 172.21.15.89:0/3866878328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0ce4057050 con 0x55d7635572f0 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cfe712740 10 client.15126 fetch_fsmap learned FSMap version 28 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cfe712740 1 -- 172.21.15.89:0/3866878328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d76356ffc0 con 0x55d7635572f0 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cddffb640 1 -- 172.21.15.89:0/3866878328 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ce4005020 con 0x55d7635572f0 2024-09-15T23:23:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cddffb640 1 -- 172.21.15.89:0/3866878328 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f0ce4044400 con 0x55d7635572f0 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cfe712740 10 client.15126 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cfe712740 10 client.15126 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cfe712740 20 client.15126 populate_metadata read hostname 'smithi089' 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cfe712740 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d763572120 0x55d763573590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cfe712740 4 client.15126 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cfe712740 1 -- 172.21.15.89:0/3866878328 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7634ca600 con 0x55d763572120 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.895+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d763572120 0x55d763573590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.896+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d763572120 0x55d763573590 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6fe9cb417c990079 server_cookie=703e61e05e22785 in_seq=0 out_seq=0 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.896+0000 7f0cddffb640 10 client.15126 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cddffb640 1 -- 172.21.15.89:0/3866878328 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d7634ca600 con 0x55d763572120 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cddffb640 10 client.15126 handle_command_reply: tid=0 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 client.15126 shutdown 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 -- 172.21.15.89:0/3866878328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 msgr2=0x55d76355e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d76355e6a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0ce402a930 tx=0x7f0ce4009520 comp rx=0 tx=0).stop 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 -- 172.21.15.89:0/3866878328 shutdown_connections 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d763572120 0x55d763573590 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d763561bb0 0x55d76355ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d7635572f0 0x55d76355e6a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cddffb640 0 client.15126 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 --2- 172.21.15.89:0/3866878328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763561350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 -- 172.21.15.89:0/3866878328 >> 172.21.15.89:0/3866878328 conn(0x55d763552f60 msgr2=0x55d763558c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 -- 172.21.15.89:0/3866878328 shutdown_connections 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 1 -- 172.21.15.89:0/3866878328 wait complete. 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.897+0000 7f0cfe712740 20 client.15126 trim_cache size 0 max 16384 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.914+0000 7f0cfe712740 1 Processor -- start 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cfe712740 1 -- start start 2024-09-15T23:23:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763573590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7635572f0 0x55d763560500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d76355b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cfe712740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d7635718d0 con 0x55d763561bb0 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cfe712740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d763571fa0 con 0x55d7635572f0 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cfe712740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d763572670 con 0x55d7635569d0 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cf98d4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d76355b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cf98d4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d76355b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39034/0 (socket says 172.21.15.89:39034) 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cf98d4640 1 -- 172.21.15.89:0/3400221561 learned_addr learned my addr 172.21.15.89:0/3400221561 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cf98d4640 1 -- 172.21.15.89:0/3400221561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 msgr2=0x55d763573590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/3400221561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763573590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cf98d4640 1 -- 172.21.15.89:0/3400221561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7635572f0 msgr2=0x55d763560500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/3400221561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7635572f0 0x55d763560500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cf98d4640 1 -- 172.21.15.89:0/3400221561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d763557120 con 0x55d763561bb0 2024-09-15T23:23:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/3400221561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d76355b900 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000be20 tx=0x7f0cf00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d7d7cf1bd6b692 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cdffff640 1 -- 172.21.15.89:0/3400221561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0005020 con 0x55d763561bb0 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cdffff640 1 -- 172.21.15.89:0/3400221561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cf00047a0 con 0x55d763561bb0 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.915+0000 7f0cdffff640 1 -- 172.21.15.89:0/3400221561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf001a6f0 con 0x55d763561bb0 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 -- 172.21.15.89:0/3400221561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 msgr2=0x55d76355b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 --2- 172.21.15.89:0/3400221561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d76355b900 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000be20 tx=0x7f0cf00118d0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 -- 172.21.15.89:0/3400221561 shutdown_connections 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 --2- 172.21.15.89:0/3400221561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d76355b900 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 --2- 172.21.15.89:0/3400221561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7635572f0 0x55d763560500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 --2- 172.21.15.89:0/3400221561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635569d0 0x55d763573590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 -- 172.21.15.89:0/3400221561 >> 172.21.15.89:0/3400221561 conn(0x55d763552f60 msgr2=0x55d76355fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 -- 172.21.15.89:0/3400221561 shutdown_connections 2024-09-15T23:23:47.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 -- 172.21.15.89:0/3400221561 wait complete. 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 -- start start 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.916+0000 7f0cfe712740 1 Processor -- start 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7635569d0 0x55d763577950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635572f0 0x55d763577e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cfe712740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d7635db830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cfe712740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d763573410 con 0x55d763561bb0 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cfe712740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d763572d40 con 0x55d7635569d0 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cfe712740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d763573ae0 con 0x55d7635572f0 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cfe712740 10 client.0 _ll_register_callbacks cb 0x7f0ce40315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cf98d4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d7635db830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cf98d4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d7635db830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39040/0 (socket says 172.21.15.89:39040) 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cf98d4640 1 -- 172.21.15.89:0/94958882 learned_addr learned my addr 172.21.15.89:0/94958882 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cfa0d5640 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635572f0 0x55d763577e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cf98d4640 1 -- 172.21.15.89:0/94958882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635572f0 msgr2=0x55d763577e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635572f0 0x55d763577e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cf98d4640 1 -- 172.21.15.89:0/94958882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7635569d0 msgr2=0x55d763577950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7635569d0 0x55d763577950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cf98d4640 1 -- 172.21.15.89:0/94958882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d763578790 con 0x55d763561bb0 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cf98d4640 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d7635db830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000bb80 tx=0x7f0cf001a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d719e45a6dcdea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cddffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cddffb640 1 -- 172.21.15.89:0/94958882 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf002a020 con 0x55d763561bb0 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cddffb640 1 -- 172.21.15.89:0/94958882 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f0cf0004b60 con 0x55d763561bb0 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.917+0000 7f0cfe712740 1 -- 172.21.15.89:0/94958882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf00047a0 con 0x55d763561bb0 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cddffb640 1 -- 172.21.15.89:0/94958882 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0cf0035050 con 0x55d763561bb0 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cfe712740 10 client.15132 fetch_fsmap learned FSMap version 28 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cfe712740 1 -- 172.21.15.89:0/94958882 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d7635dcea0 con 0x55d763561bb0 2024-09-15T23:23:47.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cddffb640 1 -- 172.21.15.89:0/94958882 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cf00047a0 con 0x55d763561bb0 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cddffb640 1 -- 172.21.15.89:0/94958882 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f0cf0024470 con 0x55d763561bb0 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cfe712740 10 client.15132 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cfe712740 10 client.15132 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cfe712740 20 client.15132 populate_metadata read hostname 'smithi089' 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cfe712740 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d7635e1720 0x55d7635e0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cfe712740 4 client.15132 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cfe712740 1 -- 172.21.15.89:0/94958882 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "status"}) -- 0x55d7634c4b10 con 0x55d7635e1720 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.918+0000 7f0cfa0d5640 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d7635e1720 0x55d7635e0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cddffb640 10 client.15132 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfa0d5640 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d7635e1720 0x55d7635e0430 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c8cc55651ac6bca7 server_cookie=6050b7d525690f5 in_seq=0 out_seq=0 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cddffb640 1 -- 172.21.15.89:0/94958882 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d7634c4b10 con 0x55d7635e1720 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cddffb640 10 client.15132 handle_command_reply: tid=0 2024-09-15T23:23:47.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 client.15132 shutdown 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 -- 172.21.15.89:0/94958882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 msgr2=0x55d7635db830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d7635db830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000bb80 tx=0x7f0cf001a630 comp rx=0 tx=0).stop 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 -- 172.21.15.89:0/94958882 shutdown_connections 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d7635e1720 0x55d7635e0430 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d763561bb0 0x55d7635db830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7635572f0 0x55d763577e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 --2- 172.21.15.89:0/94958882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d7635569d0 0x55d763577950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 -- 172.21.15.89:0/94958882 >> 172.21.15.89:0/94958882 conn(0x55d763552f60 msgr2=0x55d763552000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cddffb640 0 client.15132 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 -- 172.21.15.89:0/94958882 shutdown_connections 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 1 -- 172.21.15.89:0/94958882 wait complete. 2024-09-15T23:23:47.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.919+0000 7f0cfe712740 20 client.15132 trim_cache size 0 max 16384 2024-09-15T23:23:47.937 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:23:47.937 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:23:47.937 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 1, 2024-09-15T23:23:47.937 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24436, 2024-09-15T23:23:47.937 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:23:47.937 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:23:47.937 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:23:47.937 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 74.831377287999999, 2024-09-15T23:23:47.937 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 28, 2024-09-15T23:23:47.938 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 88, 2024-09-15T23:23:47.938 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:23:47.938 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 133.93738984800001 2024-09-15T23:23:47.938 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 -- 172.21.15.89:0/2892167763 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ce00a32d0 msgr2=0x7f0ce00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ce00a32d0 0x7f0ce00a5790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0cec097f80 tx=0x7f0cec009970 comp rx=0 tx=0).stop 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 -- 172.21.15.89:0/2892167763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf4105e40 msgr2=0x7f0cf4197c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf4105e40 0x7f0cf4197c20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00b970 tx=0x7f0cec00be30 comp rx=0 tx=0).stop 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 -- 172.21.15.89:0/2892167763 shutdown_connections 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ce00a32d0 0x7f0ce00a5790 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cf4105e40 0x7f0cf4197c20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4101fc0 0x7f0cf41976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 --2- 172.21.15.89:0/2892167763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cf4101610 0x7f0cf41971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 -- 172.21.15.89:0/2892167763 >> 172.21.15.89:0/2892167763 conn(0x7f0cf40fc5c0 msgr2=0x7f0cf40fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 -- 172.21.15.89:0/2892167763 shutdown_connections 2024-09-15T23:23:47.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:47.920+0000 7f0cfc339640 1 -- 172.21.15.89:0/2892167763 wait complete. 2024-09-15T23:23:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:48 smithi120 ceph-mon[35250]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:48.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:48 smithi174 ceph-mon[35586]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:48.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:48 smithi089 ceph-mon[30933]: pgmap v341: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:48.746 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T23:23:48.994 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:49.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.490+0000 7f4b5aca0640 1 -- 172.21.15.89:0/353414042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b54100a80 msgr2=0x7f4b54100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.490+0000 7f4b5aca0640 1 --2- 172.21.15.89:0/353414042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b54100a80 0x7f4b54100ee0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009930 tx=0x7f4b4402a420 comp rx=0 tx=0).stop 2024-09-15T23:23:49.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.490+0000 7f4b5aca0640 1 -- 172.21.15.89:0/353414042 shutdown_connections 2024-09-15T23:23:49.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.490+0000 7f4b5aca0640 1 --2- 172.21.15.89:0/353414042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b54101420 0x7f4b5410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.490+0000 7f4b5aca0640 1 --2- 172.21.15.89:0/353414042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b54100a80 0x7f4b54100ee0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.490+0000 7f4b5aca0640 1 --2- 172.21.15.89:0/353414042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b54106a80 0x7f4b54106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.490+0000 7f4b5aca0640 1 -- 172.21.15.89:0/353414042 >> 172.21.15.89:0/353414042 conn(0x7f4b540fc7a0 msgr2=0x7f4b540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.490+0000 7f4b5aca0640 1 -- 172.21.15.89:0/353414042 shutdown_connections 2024-09-15T23:23:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.490+0000 7f4b5aca0640 1 -- 172.21.15.89:0/353414042 wait complete. 2024-09-15T23:23:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b5aca0640 1 Processor -- start 2024-09-15T23:23:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b5aca0640 1 -- start start 2024-09-15T23:23:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b5aca0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b54100a80 0x7f4b541984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b5aca0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b54101420 0x7f4b54198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b5aca0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b54106a80 0x7f4b5419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b5aca0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4b54110ea0 con 0x7f4b54101420 2024-09-15T23:23:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b5aca0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4b54110d20 con 0x7f4b54106a80 2024-09-15T23:23:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b5aca0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4b54111020 con 0x7f4b54100a80 2024-09-15T23:23:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b54101420 0x7f4b54198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b54101420 0x7f4b54198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39054/0 (socket says 172.21.15.89:39054) 2024-09-15T23:23:49.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b4bfff640 1 -- 172.21.15.89:0/3259033617 learned_addr learned my addr 172.21.15.89:0/3259033617 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:49.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b5923d640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b54106a80 0x7f4b5419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b54100a80 0x7f4b541984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b4bfff640 1 -- 172.21.15.89:0/3259033617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b54100a80 msgr2=0x7f4b541984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b54100a80 0x7f4b541984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b4bfff640 1 -- 172.21.15.89:0/3259033617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b54106a80 msgr2=0x7f4b5419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b54106a80 0x7f4b5419cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b4bfff640 1 -- 172.21.15.89:0/3259033617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b5419d490 con 0x7f4b54101420 2024-09-15T23:23:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b5923d640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b54106a80 0x7f4b5419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.491+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b54100a80 0x7f4b541984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:49.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.492+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b54101420 0x7f4b54198a20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009230 tx=0x7f4b44004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.492+0000 7f4b49ffb640 1 -- 172.21.15.89:0/3259033617 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4403f030 con 0x7f4b54101420 2024-09-15T23:23:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.492+0000 7f4b49ffb640 1 -- 172.21.15.89:0/3259033617 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b44003ad0 con 0x7f4b54101420 2024-09-15T23:23:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.492+0000 7f4b5aca0640 1 -- 172.21.15.89:0/3259033617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b5419d720 con 0x7f4b54101420 2024-09-15T23:23:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.492+0000 7f4b49ffb640 1 -- 172.21.15.89:0/3259033617 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44035070 con 0x7f4b54101420 2024-09-15T23:23:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.492+0000 7f4b5aca0640 1 -- 172.21.15.89:0/3259033617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b541a5060 con 0x7f4b54101420 2024-09-15T23:23:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.494+0000 7f4b49ffb640 1 -- 172.21.15.89:0/3259033617 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4b44042030 con 0x7f4b54101420 2024-09-15T23:23:49.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.494+0000 7f4b5d079740 1 Processor -- start 2024-09-15T23:23:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.494+0000 7f4b5d079740 1 -- start start 2024-09-15T23:23:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075119d0 0x555607511db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 0x55560751c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560751cbb0 0x55560751efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b5d079740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555607478090 con 0x55560751cbb0 2024-09-15T23:23:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b5d079740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55560741fa70 con 0x5556075122f0 2024-09-15T23:23:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b5d079740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555607474050 con 0x5556075119d0 2024-09-15T23:23:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560751cbb0 0x55560751efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560751cbb0 0x55560751efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39056/0 (socket says 172.21.15.89:39056) 2024-09-15T23:23:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b4bfff640 1 -- 172.21.15.89:0/2136920214 learned_addr learned my addr 172.21.15.89:0/2136920214 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:49.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b5923d640 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075119d0 0x555607511db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 0x55560751c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b49ffb640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b3c0a3350 0x7f4b3c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b4bfff640 1 -- 172.21.15.89:0/2136920214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075119d0 msgr2=0x555607511db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075119d0 0x555607511db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b4bfff640 1 -- 172.21.15.89:0/2136920214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 msgr2=0x55560751c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 0x55560751c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b4bfff640 1 -- 172.21.15.89:0/2136920214 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55560742c130 con 0x55560751cbb0 2024-09-15T23:23:49.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b5923d640 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075119d0 0x555607511db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:49.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 0x55560751c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:49.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b5923d640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b3c0a3350 0x7f4b3c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b49ffb640 1 -- 172.21.15.89:0/3259033617 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f4b440f0330 con 0x7f4b54101420 2024-09-15T23:23:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.495+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560751cbb0 0x55560751efa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4b440ed8c0 tx=0x7f4b4404b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f782eb0d0a2368c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b3affd640 1 -- 172.21.15.89:0/2136920214 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b440bbdf0 con 0x55560751cbb0 2024-09-15T23:23:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b3affd640 1 -- 172.21.15.89:0/2136920214 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b440a57e0 con 0x55560751cbb0 2024-09-15T23:23:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b3affd640 1 -- 172.21.15.89:0/2136920214 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b440a6f60 con 0x55560751cbb0 2024-09-15T23:23:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b5d079740 1 -- 172.21.15.89:0/2136920214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560751cbb0 msgr2=0x55560751efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b5923d640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b3c0a3350 0x7f4b3c0a5810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00a5a0 tx=0x7f4b4c00c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b5d079740 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560751cbb0 0x55560751efa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4b440ed8c0 tx=0x7f4b4404b000 comp rx=0 tx=0).stop 2024-09-15T23:23:49.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b5d079740 1 -- 172.21.15.89:0/2136920214 shutdown_connections 2024-09-15T23:23:49.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b5d079740 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560751cbb0 0x55560751efa0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b5d079740 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 0x55560751c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b5d079740 1 --2- 172.21.15.89:0/2136920214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075119d0 0x555607511db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b5d079740 1 -- 172.21.15.89:0/2136920214 >> 172.21.15.89:0/2136920214 conn(0x55560750df60 msgr2=0x55560750e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b5d079740 1 -- 172.21.15.89:0/2136920214 shutdown_connections 2024-09-15T23:23:49.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.496+0000 7f4b5d079740 1 -- 172.21.15.89:0/2136920214 wait complete. 2024-09-15T23:23:49.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b5d079740 1 -- start start 2024-09-15T23:23:49.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b5d079740 1 Processor -- start 2024-09-15T23:23:49.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075119d0 0x55560751c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075122f0 0x555607519710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560751cbb0 0x555607519c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b5d079740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55560751f540 con 0x5556075122f0 2024-09-15T23:23:49.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b5d079740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555607432f20 con 0x5556075119d0 2024-09-15T23:23:49.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b5d079740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55560751f6c0 con 0x55560751cbb0 2024-09-15T23:23:49.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b5d079740 10 client.0 _ll_register_callbacks cb 0x5556073fc030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:49.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075122f0 0x555607519710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075122f0 0x555607519710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39060/0 (socket says 172.21.15.89:39060) 2024-09-15T23:23:49.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b4bfff640 1 -- 172.21.15.89:0/1010325605 learned_addr learned my addr 172.21.15.89:0/1010325605 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:49.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.497+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075119d0 0x55560751c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b4bfff640 1 -- 172.21.15.89:0/1010325605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560751cbb0 msgr2=0x555607519c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560751cbb0 0x555607519c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b4bfff640 1 -- 172.21.15.89:0/1010325605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075119d0 msgr2=0x55560751c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075119d0 0x55560751c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b4bfff640 1 -- 172.21.15.89:0/1010325605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556071e92b0 con 0x5556075122f0 2024-09-15T23:23:49.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075122f0 0x555607519710 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4b440c4bd0 tx=0x7f4b4402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14bc3c56d14a4d12 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b33fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:23:49.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b33fff640 1 -- 172.21.15.89:0/1010325605 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44049070 con 0x5556075122f0 2024-09-15T23:23:49.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b33fff640 1 -- 172.21.15.89:0/1010325605 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f4b440f0120 con 0x5556075122f0 2024-09-15T23:23:49.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b5d079740 1 -- 172.21.15.89:0/1010325605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b440a57e0 con 0x5556075122f0 2024-09-15T23:23:49.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b33fff640 1 -- 172.21.15.89:0/1010325605 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b440bcb90 con 0x5556075122f0 2024-09-15T23:23:49.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b5d079740 10 client.15144 fetch_fsmap learned FSMap version 28 2024-09-15T23:23:49.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b5d079740 1 -- 172.21.15.89:0/1010325605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55560751f540 con 0x5556075122f0 2024-09-15T23:23:49.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b33fff640 1 -- 172.21.15.89:0/1010325605 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b440a57e0 con 0x5556075122f0 2024-09-15T23:23:49.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b33fff640 1 -- 172.21.15.89:0/1010325605 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f4b440b8b00 con 0x5556075122f0 2024-09-15T23:23:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b5d079740 10 client.15144 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:23:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b5d079740 10 client.15144 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:23:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b5d079740 20 client.15144 populate_metadata read hostname 'smithi089' 2024-09-15T23:23:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55560752d030 0x55560752e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b5d079740 4 client.15144 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.498+0000 7f4b5d079740 1 -- 172.21.15.89:0/1010325605 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555607485600 con 0x55560752d030 2024-09-15T23:23:49.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.499+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55560752d030 0x55560752e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.499+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55560752d030 0x55560752e4a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae16840b5816cd96 server_cookie=f1d510376ee6e0bd in_seq=0 out_seq=0 2024-09-15T23:23:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.499+0000 7f4b33fff640 10 client.15144 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:23:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b33fff640 1 -- 172.21.15.89:0/1010325605 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555607485600 con 0x55560752d030 2024-09-15T23:23:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b33fff640 10 client.15144 handle_command_reply: tid=0 2024-09-15T23:23:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 client.15144 shutdown 2024-09-15T23:23:49.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 -- 172.21.15.89:0/1010325605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075122f0 msgr2=0x555607519710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075122f0 0x555607519710 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4b440c4bd0 tx=0x7f4b4402a420 comp rx=0 tx=0).stop 2024-09-15T23:23:49.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 -- 172.21.15.89:0/1010325605 shutdown_connections 2024-09-15T23:23:49.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55560752d030 0x55560752e4a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560751cbb0 0x555607519c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075122f0 0x555607519710 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b33fff640 0 client.15144 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:23:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1010325605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075119d0 0x55560751c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 -- 172.21.15.89:0/1010325605 >> 172.21.15.89:0/1010325605 conn(0x55560750df60 msgr2=0x555607513c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 -- 172.21.15.89:0/1010325605 shutdown_connections 2024-09-15T23:23:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 1 -- 172.21.15.89:0/1010325605 wait complete. 2024-09-15T23:23:49.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.500+0000 7f4b5d079740 20 client.15144 trim_cache size 0 max 16384 2024-09-15T23:23:49.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b5d079740 1 Processor -- start 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b5d079740 1 -- start start 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x55560752e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075122f0 0x55560750e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560751cbb0 0x555607514de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b5d079740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55560752c7e0 con 0x5556075119d0 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b5d079740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55560752ceb0 con 0x55560751cbb0 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b5d079740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55560752d580 con 0x5556075122f0 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b58a3c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x55560752e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b58a3c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x55560752e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39076/0 (socket says 172.21.15.89:39076) 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.518+0000 7f4b58a3c640 1 -- 172.21.15.89:0/2192797506 learned_addr learned my addr 172.21.15.89:0/2192797506 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:49.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075122f0 0x55560750e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b58a3c640 1 -- 172.21.15.89:0/2192797506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075122f0 msgr2=0x55560750e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b5923d640 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560751cbb0 0x555607514de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075122f0 0x55560750e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b58a3c640 1 -- 172.21.15.89:0/2192797506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560751cbb0 msgr2=0x555607514de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560751cbb0 0x555607514de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b58a3c640 1 -- 172.21.15.89:0/2192797506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55560751c730 con 0x5556075119d0 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075122f0 0x55560750e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b5923d640 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560751cbb0 0x555607514de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x55560752e4a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4b50000c00 tx=0x7f4b500094b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e879e9b395dcc6b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b3a7fc640 1 -- 172.21.15.89:0/2192797506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5001f070 con 0x5556075119d0 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b3a7fc640 1 -- 172.21.15.89:0/2192797506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b50004220 con 0x5556075119d0 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.519+0000 7f4b3a7fc640 1 -- 172.21.15.89:0/2192797506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5001b520 con 0x5556075119d0 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 -- 172.21.15.89:0/2192797506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 msgr2=0x55560752e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x55560752e4a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4b50000c00 tx=0x7f4b500094b0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 -- 172.21.15.89:0/2192797506 shutdown_connections 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560751cbb0 0x555607514de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5556075122f0 0x55560750e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 --2- 172.21.15.89:0/2192797506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x55560752e4a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 -- 172.21.15.89:0/2192797506 >> 172.21.15.89:0/2192797506 conn(0x55560750df60 msgr2=0x555607517fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 -- 172.21.15.89:0/2192797506 shutdown_connections 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 -- 172.21.15.89:0/2192797506 wait complete. 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 -- start start 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.520+0000 7f4b5d079740 1 Processor -- start 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x555607536a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 0x555607536f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b5d079740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560751cbb0 0x55560752fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b5d079740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55560752e320 con 0x5556075119d0 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b5d079740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55560752dc50 con 0x5556075122f0 2024-09-15T23:23:49.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b5d079740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55560752e9f0 con 0x55560751cbb0 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b5d079740 10 client.0 _ll_register_callbacks cb 0x7f4b440b85f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b58a3c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x555607536a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b58a3c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x555607536a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39084/0 (socket says 172.21.15.89:39084) 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b58a3c640 1 -- 172.21.15.89:0/1589906809 learned_addr learned my addr 172.21.15.89:0/1589906809 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b5923d640 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560751cbb0 0x55560752fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 0x555607536f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b58a3c640 1 -- 172.21.15.89:0/1589906809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560751cbb0 msgr2=0x55560752fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560751cbb0 0x55560752fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b58a3c640 1 -- 172.21.15.89:0/1589906809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 msgr2=0x555607536f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 0x555607536f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b58a3c640 1 -- 172.21.15.89:0/1589906809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556075305e0 con 0x5556075119d0 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 0x555607536f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:49.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b58a3c640 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x555607536a20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4b50005940 tx=0x7f4b50004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655c0f196857d870 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b33fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b33fff640 1 -- 172.21.15.89:0/1589906809 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5001f040 con 0x5556075119d0 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.521+0000 7f4b33fff640 1 -- 172.21.15.89:0/1589906809 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f4b50003b00 con 0x5556075119d0 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b5d079740 1 -- 172.21.15.89:0/1589906809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b50004220 con 0x5556075119d0 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b33fff640 1 -- 172.21.15.89:0/1589906809 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b50035050 con 0x5556075119d0 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b5d079740 10 client.15150 fetch_fsmap learned FSMap version 28 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b5d079740 1 -- 172.21.15.89:0/1589906809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555607535030 con 0x5556075119d0 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b33fff640 1 -- 172.21.15.89:0/1589906809 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b50004220 con 0x5556075119d0 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b33fff640 1 -- 172.21.15.89:0/1589906809 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f4b5000f040 con 0x5556075119d0 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b5d079740 10 client.15150 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b5d079740 10 client.15150 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b5d079740 20 client.15150 populate_metadata read hostname 'smithi089' 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55560759bc50 0x55560759a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b5d079740 4 client.15150 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.522+0000 7f4b5d079740 1 -- 172.21.15.89:0/1589906809 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "status"}) -- 0x55560747fb10 con 0x55560759bc50 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55560759bc50 0x55560759a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b33fff640 10 client.15150 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b4bfff640 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55560759bc50 0x55560759a960 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e515e88c6e6c589 server_cookie=d6d9910af01ffb0d in_seq=0 out_seq=0 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b33fff640 1 -- 172.21.15.89:0/1589906809 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55560747fb10 con 0x55560759bc50 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b33fff640 10 client.15150 handle_command_reply: tid=0 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b5d079740 1 client.15150 shutdown 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b5d079740 1 -- 172.21.15.89:0/1589906809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 msgr2=0x555607536a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x555607536a20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4b50005940 tx=0x7f4b50004500 comp rx=0 tx=0).stop 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b5d079740 1 -- 172.21.15.89:0/1589906809 shutdown_connections 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55560759bc50 0x55560759a960 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560751cbb0 0x55560752fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556075122f0 0x555607536f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b33fff640 0 client.15150 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:23:49.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b5d079740 1 --2- 172.21.15.89:0/1589906809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5556075119d0 0x555607536a20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.523+0000 7f4b5d079740 1 -- 172.21.15.89:0/1589906809 >> 172.21.15.89:0/1589906809 conn(0x55560750df60 msgr2=0x555607518800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5d079740 1 -- 172.21.15.89:0/1589906809 shutdown_connections 2024-09-15T23:23:49.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5d079740 1 -- 172.21.15.89:0/1589906809 wait complete. 2024-09-15T23:23:49.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5d079740 20 client.15150 trim_cache size 0 max 16384 2024-09-15T23:23:49.547 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:23:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:23:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-09-15T23:23:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24452, 2024-09-15T23:23:49.547 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:23:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:23:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:23:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 116.003964064, 2024-09-15T23:23:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 28, 2024-09-15T23:23:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 88, 2024-09-15T23:23:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:23:49.548 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 126.530885865 2024-09-15T23:23:49.548 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:23:49.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5aca0640 1 -- 172.21.15.89:0/3259033617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b3c0a3350 msgr2=0x7f4b3c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5aca0640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b3c0a3350 0x7f4b3c0a5810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c00a5a0 tx=0x7f4b4c00c040 comp rx=0 tx=0).stop 2024-09-15T23:23:49.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5aca0640 1 -- 172.21.15.89:0/3259033617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b54101420 msgr2=0x7f4b54198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5aca0640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b54101420 0x7f4b54198a20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009230 tx=0x7f4b44004270 comp rx=0 tx=0).stop 2024-09-15T23:23:49.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5aca0640 1 -- 172.21.15.89:0/3259033617 shutdown_connections 2024-09-15T23:23:49.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5aca0640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b3c0a3350 0x7f4b3c0a5810 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5aca0640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b54106a80 0x7f4b5419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5aca0640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b54101420 0x7f4b54198a20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5aca0640 1 --2- 172.21.15.89:0/3259033617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b54100a80 0x7f4b541984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.524+0000 7f4b5aca0640 1 -- 172.21.15.89:0/3259033617 >> 172.21.15.89:0/3259033617 conn(0x7f4b540fc7a0 msgr2=0x7f4b5406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.525+0000 7f4b5aca0640 1 -- 172.21.15.89:0/3259033617 shutdown_connections 2024-09-15T23:23:49.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:49.525+0000 7f4b5aca0640 1 -- 172.21.15.89:0/3259033617 wait complete. 2024-09-15T23:23:50.451 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:50 smithi089 ceph-mon[30933]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:50.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:50 smithi120 ceph-mon[35250]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:50.481 INFO:tasks.cephadm:Setup complete, yielding 2024-09-15T23:23:50.482 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-15T23:23:50.493 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi089.front.sepia.ceph.com 2024-09-15T23:23:50.494 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- bash -c 'ceph orch status' 2024-09-15T23:23:50.494 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:50 smithi174 ceph-mon[35586]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:23:50.742 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:51.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.267+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2348552087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc1c107900 msgr2=0x7fdc1c109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.267+0000 7fdc22bc4640 1 --2- 172.21.15.89:0/2348552087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc1c107900 0x7fdc1c109d50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009ca0 tx=0x7fdc1402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.267+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2348552087 shutdown_connections 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.267+0000 7fdc22bc4640 1 --2- 172.21.15.89:0/2348552087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc1c107900 0x7fdc1c109d50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.267+0000 7fdc22bc4640 1 --2- 172.21.15.89:0/2348552087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c104fd0 0x7fdc1c1073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.267+0000 7fdc22bc4640 1 --2- 172.21.15.89:0/2348552087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc1c1026a0 0x7fdc1c104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.267+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2348552087 >> 172.21.15.89:0/2348552087 conn(0x7fdc1c0fc2a0 msgr2=0x7fdc1c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.267+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2348552087 shutdown_connections 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.267+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2348552087 wait complete. 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc22bc4640 1 Processor -- start 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc22bc4640 1 -- start start 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc22bc4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc1c1026a0 0x7fdc1c102000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc22bc4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c104fd0 0x7fdc1c100650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc22bc4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc1c107900 0x7fdc1c100b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc22bc4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c10d8c0 con 0x7fdc1c107900 2024-09-15T23:23:51.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc22bc4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c10d740 con 0x7fdc1c104fd0 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc22bc4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c10da40 con 0x7fdc1c1026a0 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc21161640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc1c107900 0x7fdc1c100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc21161640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc1c107900 0x7fdc1c100b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39088/0 (socket says 172.21.15.89:39088) 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc21161640 1 -- 172.21.15.89:0/2266396002 learned_addr learned my addr 172.21.15.89:0/2266396002 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc1bfff640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c104fd0 0x7fdc1c100650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc20960640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc1c1026a0 0x7fdc1c102000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc21161640 1 -- 172.21.15.89:0/2266396002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc1c1026a0 msgr2=0x7fdc1c102000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc21161640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc1c1026a0 0x7fdc1c102000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc21161640 1 -- 172.21.15.89:0/2266396002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c104fd0 msgr2=0x7fdc1c100650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc21161640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c104fd0 0x7fdc1c100650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc21161640 1 -- 172.21.15.89:0/2266396002 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c101450 con 0x7fdc1c107900 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.268+0000 7fdc1bfff640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c104fd0 0x7fdc1c100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.269+0000 7fdc21161640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc1c107900 0x7fdc1c100b90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdc14031ae0 tx=0x7fdc14031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:51.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.269+0000 7fdc19ffb640 1 -- 172.21.15.89:0/2266396002 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14003c30 con 0x7fdc1c107900 2024-09-15T23:23:51.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.269+0000 7fdc19ffb640 1 -- 172.21.15.89:0/2266396002 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc14003dd0 con 0x7fdc1c107900 2024-09-15T23:23:51.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.269+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2266396002 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c1a2020 con 0x7fdc1c107900 2024-09-15T23:23:51.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.269+0000 7fdc19ffb640 1 -- 172.21.15.89:0/2266396002 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1402fcb0 con 0x7fdc1c107900 2024-09-15T23:23:51.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.269+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2266396002 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c1a2530 con 0x7fdc1c107900 2024-09-15T23:23:51.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.270+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2266396002 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc1c0fdd90 con 0x7fdc1c107900 2024-09-15T23:23:51.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.271+0000 7fdc19ffb640 1 -- 172.21.15.89:0/2266396002 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fdc14004050 con 0x7fdc1c107900 2024-09-15T23:23:51.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.272+0000 7fdc19ffb640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc0c0a32d0 0x7fdc0c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.272+0000 7fdc20960640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc0c0a32d0 0x7fdc0c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.272+0000 7fdc19ffb640 1 -- 172.21.15.89:0/2266396002 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fdc140f1250 con 0x7fdc1c107900 2024-09-15T23:23:51.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.272+0000 7fdc20960640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc0c0a32d0 0x7fdc0c0a5790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdc10005fd0 tx=0x7fdc10005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:51.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.275+0000 7fdc19ffb640 1 -- 172.21.15.89:0/2266396002 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdc140bdbc0 con 0x7fdc1c107900 2024-09-15T23:23:51.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.454+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2266396002 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fdc1c062630 con 0x7fdc0c0a32d0 2024-09-15T23:23:51.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.455+0000 7fdc19ffb640 1 -- 172.21.15.89:0/2266396002 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fdc1c062630 con 0x7fdc0c0a32d0 2024-09-15T23:23:51.456 INFO:teuthology.orchestra.run.smithi089.stdout:Backend: cephadm 2024-09-15T23:23:51.456 INFO:teuthology.orchestra.run.smithi089.stdout:Available: Yes 2024-09-15T23:23:51.457 INFO:teuthology.orchestra.run.smithi089.stdout:Paused: No 2024-09-15T23:23:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.457+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2266396002 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc0c0a32d0 msgr2=0x7fdc0c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.457+0000 7fdc22bc4640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc0c0a32d0 0x7fdc0c0a5790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdc10005fd0 tx=0x7fdc10005850 comp rx=0 tx=0).stop 2024-09-15T23:23:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.457+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2266396002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc1c107900 msgr2=0x7fdc1c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.457+0000 7fdc22bc4640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc1c107900 0x7fdc1c100b90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdc14031ae0 tx=0x7fdc14031ee0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.458+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2266396002 shutdown_connections 2024-09-15T23:23:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.458+0000 7fdc22bc4640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc0c0a32d0 0x7fdc0c0a5790 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.458+0000 7fdc22bc4640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc1c107900 0x7fdc1c100b90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.458+0000 7fdc22bc4640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c104fd0 0x7fdc1c100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.458+0000 7fdc22bc4640 1 --2- 172.21.15.89:0/2266396002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc1c1026a0 0x7fdc1c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.458+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2266396002 >> 172.21.15.89:0/2266396002 conn(0x7fdc1c0fc2a0 msgr2=0x7fdc1c102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.458+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2266396002 shutdown_connections 2024-09-15T23:23:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:51.458+0000 7fdc22bc4640 1 -- 172.21.15.89:0/2266396002 wait complete. 2024-09-15T23:23:52.175 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-15T23:23:52.422 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:52 smithi120 ceph-mon[35250]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:23:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:52 smithi174 ceph-mon[35586]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:23:52.543 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:52 smithi089 ceph-mon[30933]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.923+0000 7fe4c8fad640 1 -- 172.21.15.89:0/54391413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4c41026b0 msgr2=0x7fe4c4108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.923+0000 7fe4c8fad640 1 --2- 172.21.15.89:0/54391413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4c41026b0 0x7fe4c4108f40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009900 tx=0x7fe4b402a410 comp rx=0 tx=0).stop 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 -- 172.21.15.89:0/54391413 shutdown_connections 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 --2- 172.21.15.89:0/54391413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4c41026b0 0x7fe4c4108f40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 --2- 172.21.15.89:0/54391413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4c4101cf0 0x7fe4c4102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 --2- 172.21.15.89:0/54391413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4c4100af0 0x7fe4c4100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 -- 172.21.15.89:0/54391413 >> 172.21.15.89:0/54391413 conn(0x7fe4c40fc2a0 msgr2=0x7fe4c40fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 -- 172.21.15.89:0/54391413 shutdown_connections 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 -- 172.21.15.89:0/54391413 wait complete. 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 Processor -- start 2024-09-15T23:23:52.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 -- start start 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4c4100af0 0x7fe4c406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.924+0000 7fe4c8fad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4c4101cf0 0x7fe4c419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c8fad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4c41026b0 0x7fe4c419f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c8fad640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c410c770 con 0x7fe4c4101cf0 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c8fad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c410c5f0 con 0x7fe4c4100af0 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c8fad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c410c8f0 con 0x7fe4c41026b0 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4c4101cf0 0x7fe4c419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4c4101cf0 0x7fe4c419cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39096/0 (socket says 172.21.15.89:39096) 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c1d9b640 1 -- 172.21.15.89:0/1215547927 learned_addr learned my addr 172.21.15.89:0/1215547927 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c259c640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4c4100af0 0x7fe4c406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:52.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c2d9d640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4c41026b0 0x7fe4c419f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c1d9b640 1 -- 172.21.15.89:0/1215547927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4c41026b0 msgr2=0x7fe4c419f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c1d9b640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4c41026b0 0x7fe4c419f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c1d9b640 1 -- 172.21.15.89:0/1215547927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4c4100af0 msgr2=0x7fe4c406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c1d9b640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4c4100af0 0x7fe4c406b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c1d9b640 1 -- 172.21.15.89:0/1215547927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c419fb50 con 0x7fe4c4101cf0 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c1d9b640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4c4101cf0 0x7fe4c419cc20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00e9f0 tx=0x7fe4ac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4bb7fe640 1 -- 172.21.15.89:0/1215547927 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac00cd80 con 0x7fe4c4101cf0 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4bb7fe640 1 -- 172.21.15.89:0/1215547927 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4ac005500 con 0x7fe4c4101cf0 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4bb7fe640 1 -- 172.21.15.89:0/1215547927 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac010690 con 0x7fe4c4101cf0 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c8fad640 1 -- 172.21.15.89:0/1215547927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c419fde0 con 0x7fe4c4101cf0 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.925+0000 7fe4c8fad640 1 -- 172.21.15.89:0/1215547927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c41a0240 con 0x7fe4c4101cf0 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.926+0000 7fe4c8fad640 1 -- 172.21.15.89:0/1215547927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c4107ef0 con 0x7fe4c4101cf0 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.927+0000 7fe4bb7fe640 1 -- 172.21.15.89:0/1215547927 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fe4ac0046c0 con 0x7fe4c4101cf0 2024-09-15T23:23:52.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.928+0000 7fe4bb7fe640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4b00a32d0 0x7fe4b00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:52.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.928+0000 7fe4c259c640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4b00a32d0 0x7fe4b00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:52.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.928+0000 7fe4bb7fe640 1 -- 172.21.15.89:0/1215547927 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fe4ac0d0160 con 0x7fe4c4101cf0 2024-09-15T23:23:52.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.929+0000 7fe4c259c640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4b00a32d0 0x7fe4b00a5790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4101b50 tx=0x7fe4bc009720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:52.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:52.932+0000 7fe4bb7fe640 1 -- 172.21.15.89:0/1215547927 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe4ac09c9c0 con 0x7fe4c4101cf0 2024-09-15T23:23:53.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.134+0000 7fe4c8fad640 1 -- 172.21.15.89:0/1215547927 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fe4c4062630 con 0x7fe4b00a32d0 2024-09-15T23:23:53.135 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:53 smithi089 ceph-mon[30933]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:53.135 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:53 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:23:53.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.147+0000 7fe4bb7fe640 1 -- 172.21.15.89:0/1215547927 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4061 (secure 0 0 0) 0x7fe4c4062630 con 0x7fe4b00a32d0 2024-09-15T23:23:53.148 INFO:teuthology.orchestra.run.smithi089.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-15T23:23:53.148 INFO:teuthology.orchestra.run.smithi089.stdout:mds.a smithi089 running (2m) 61s ago 2m 15.6M - 19.1.1-326-g8921c8ef f1455bd0219c ccb0f0bc2bb7 2024-09-15T23:23:53.148 INFO:teuthology.orchestra.run.smithi089.stdout:mds.b smithi120 running (2m) 63s ago 2m 25.9M - 19.1.1-326-g8921c8ef f1455bd0219c c4a65ae0ee31 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mds.c smithi174 running (2m) 63s ago 2m 16.5M - 19.1.1-326-g8921c8ef f1455bd0219c aa363d291d7c 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mds.d smithi089 running (2m) 61s ago 2m 13.4M - 19.1.1-326-g8921c8ef f1455bd0219c 6da1be8a1e9c 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mds.e smithi120 running (2m) 63s ago 2m 15.8M - 19.1.1-326-g8921c8ef f1455bd0219c 8d6ea269aa73 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mds.f smithi174 running (2m) 63s ago 2m 19.6M - 19.1.1-326-g8921c8ef f1455bd0219c c946c964cccb 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mds.g smithi089 running (2m) 61s ago 2m 15.5M - 19.1.1-326-g8921c8ef f1455bd0219c d5083778b954 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mds.h smithi120 running (2m) 63s ago 2m 18.5M - 19.1.1-326-g8921c8ef f1455bd0219c 3b11d314aa27 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mds.i smithi174 running (2m) 63s ago 2m 19.5M - 19.1.1-326-g8921c8ef f1455bd0219c 7e318973bd9e 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mds.j smithi089 running (2m) 61s ago 2m 16.8M - 19.1.1-326-g8921c8ef f1455bd0219c 1c1226337f69 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mds.k smithi120 running (2m) 63s ago 2m 15.6M - 19.1.1-326-g8921c8ef f1455bd0219c 8429948d1f28 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mds.l smithi174 running (2m) 63s ago 2m 15.7M - 19.1.1-326-g8921c8ef f1455bd0219c 61681fc05d8b 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.x smithi089 *:9283,8765 running (9m) 61s ago 9m 506M - 19.1.1-326-g8921c8ef f1455bd0219c 9504be6ed986 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.y smithi120 *:8443,8765 running (7m) 63s ago 7m 444M - 19.1.1-326-g8921c8ef f1455bd0219c 5dccf040190a 2024-09-15T23:23:53.149 INFO:teuthology.orchestra.run.smithi089.stdout:mgr.z smithi174 *:8443,8765 running (7m) 63s ago 7m 443M - 19.1.1-326-g8921c8ef f1455bd0219c 86570c43e948 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:mon.a smithi089 running (9m) 61s ago 9m 49.5M 2048M 19.1.1-326-g8921c8ef f1455bd0219c 587b46d28206 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:mon.b smithi120 running (8m) 63s ago 8m 41.3M 2048M 19.1.1-326-g8921c8ef f1455bd0219c fc28fc27ee92 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:mon.c smithi174 running (8m) 63s ago 8m 41.9M 2048M 19.1.1-326-g8921c8ef f1455bd0219c 04c6697b62dd 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0 smithi089 running (7m) 61s ago 7m 57.0M 4271M 19.1.1-326-g8921c8ef f1455bd0219c 2cfb4bf4b3b6 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1 smithi120 running (7m) 63s ago 7m 58.2M 4272M 19.1.1-326-g8921c8ef f1455bd0219c fb5469c71234 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2 smithi174 running (6m) 63s ago 6m 48.0M 4272M 19.1.1-326-g8921c8ef f1455bd0219c a82184329c17 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3 smithi089 running (6m) 61s ago 6m 56.9M 4271M 19.1.1-326-g8921c8ef f1455bd0219c 6cf4778962f9 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4 smithi120 running (5m) 63s ago 5m 52.2M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 7b64fb3ff88e 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5 smithi174 running (5m) 63s ago 5m 45.7M 4272M 19.1.1-326-g8921c8ef f1455bd0219c e7dcaa9d2229 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6 smithi089 running (5m) 61s ago 5m 41.5M 4271M 19.1.1-326-g8921c8ef f1455bd0219c 914a4cfe76f5 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7 smithi120 running (4m) 63s ago 4m 58.8M 4272M 19.1.1-326-g8921c8ef f1455bd0219c ec285f96aaff 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.8 smithi174 running (4m) 63s ago 4m 42.9M 4272M 19.1.1-326-g8921c8ef f1455bd0219c b3d67fe4ecaf 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.9 smithi089 running (3m) 61s ago 3m 41.8M 4271M 19.1.1-326-g8921c8ef f1455bd0219c 64698b35e1d8 2024-09-15T23:23:53.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.10 smithi120 running (3m) 63s ago 3m 45.9M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 136840411a9e 2024-09-15T23:23:53.151 INFO:teuthology.orchestra.run.smithi089.stdout:osd.11 smithi174 running (3m) 63s ago 3m 52.9M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 70559f4c28d2 2024-09-15T23:23:53.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 -- 172.21.15.89:0/1215547927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4b00a32d0 msgr2=0x7fe4b00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:53.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4b00a32d0 0x7fe4b00a5790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4101b50 tx=0x7fe4bc009720 comp rx=0 tx=0).stop 2024-09-15T23:23:53.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 -- 172.21.15.89:0/1215547927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4c4101cf0 msgr2=0x7fe4c419cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:53.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4c4101cf0 0x7fe4c419cc20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00e9f0 tx=0x7fe4ac00edf0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 -- 172.21.15.89:0/1215547927 shutdown_connections 2024-09-15T23:23:53.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4b00a32d0 0x7fe4b00a5790 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4c41026b0 0x7fe4c419f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4c4101cf0 0x7fe4c419cc20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 --2- 172.21.15.89:0/1215547927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4c4100af0 0x7fe4c406b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 -- 172.21.15.89:0/1215547927 >> 172.21.15.89:0/1215547927 conn(0x7fe4c40fc2a0 msgr2=0x7fe4c41050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:53.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 -- 172.21.15.89:0/1215547927 shutdown_connections 2024-09-15T23:23:53.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:53.149+0000 7fe4c8fad640 1 -- 172.21.15.89:0/1215547927 wait complete. 2024-09-15T23:23:53.332 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:53 smithi120 ceph-mon[35250]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:53.332 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:53 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:23:53.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:53 smithi174 ceph-mon[35586]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:53.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:53 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:23:54.191 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:54 smithi089 ceph-mon[30933]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:54.192 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:54.192 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:54 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:54 smithi120 ceph-mon[35250]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:54 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:54 smithi174 ceph-mon[35586]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:54 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:54.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ls' 2024-09-15T23:23:54.919 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:55.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:55 smithi089 ceph-mon[30933]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:55.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:55.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:55 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:55.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.395+0000 7f630cfd7640 1 -- 172.21.15.89:0/3880472522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6308101c00 msgr2=0x7f6308102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:55.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.395+0000 7f630cfd7640 1 --2- 172.21.15.89:0/3880472522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6308101c00 0x7f6308102000 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009900 tx=0x7f62f802a410 comp rx=0 tx=0).stop 2024-09-15T23:23:55.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.395+0000 7f630cfd7640 1 -- 172.21.15.89:0/3880472522 shutdown_connections 2024-09-15T23:23:55.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.395+0000 7f630cfd7640 1 --2- 172.21.15.89:0/3880472522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63081037c0 0x7f630810a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.395+0000 7f630cfd7640 1 --2- 172.21.15.89:0/3880472522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6308102e00 0x7f6308103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.395+0000 7f630cfd7640 1 --2- 172.21.15.89:0/3880472522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6308101c00 0x7f6308102000 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.395+0000 7f630cfd7640 1 -- 172.21.15.89:0/3880472522 >> 172.21.15.89:0/3880472522 conn(0x7f63080fd390 msgr2=0x7f63080ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.395+0000 7f630cfd7640 1 -- 172.21.15.89:0/3880472522 shutdown_connections 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.395+0000 7f630cfd7640 1 -- 172.21.15.89:0/3880472522 wait complete. 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.396+0000 7f630cfd7640 1 Processor -- start 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.396+0000 7f630cfd7640 1 -- start start 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.396+0000 7f630cfd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6308101c00 0x7f6308197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.396+0000 7f630cfd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6308102e00 0x7f6308198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.396+0000 7f630cfd7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63081037c0 0x7f630819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.396+0000 7f630cfd7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f630810d890 con 0x7f6308101c00 2024-09-15T23:23:55.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.396+0000 7f630cfd7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f630810d710 con 0x7f63081037c0 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.396+0000 7f630cfd7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f630810da10 con 0x7f6308102e00 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.396+0000 7f630659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6308101c00 0x7f6308197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f630659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6308101c00 0x7f6308197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41412/0 (socket says 172.21.15.89:41412) 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f630659c640 1 -- 172.21.15.89:0/2646476984 learned_addr learned my addr 172.21.15.89:0/2646476984 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f6305d9b640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6308102e00 0x7f6308198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f6306d9d640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63081037c0 0x7f630819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f630659c640 1 -- 172.21.15.89:0/2646476984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6308102e00 msgr2=0x7f6308198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f630659c640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6308102e00 0x7f6308198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f630659c640 1 -- 172.21.15.89:0/2646476984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63081037c0 msgr2=0x7f630819f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f630659c640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63081037c0 0x7f630819f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f630659c640 1 -- 172.21.15.89:0/2646476984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630819fb30 con 0x7f6308101c00 2024-09-15T23:23:55.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f630659c640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6308101c00 0x7f6308197f00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f62f8031610 tx=0x7f62f80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f62f77fe640 1 -- 172.21.15.89:0/2646476984 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f803f030 con 0x7f6308101c00 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f630cfd7640 1 -- 172.21.15.89:0/2646476984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630819fdc0 con 0x7f6308101c00 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f62f77fe640 1 -- 172.21.15.89:0/2646476984 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62f8003ce0 con 0x7f6308101c00 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f62f77fe640 1 -- 172.21.15.89:0/2646476984 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8035070 con 0x7f6308101c00 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.397+0000 7f630cfd7640 1 -- 172.21.15.89:0/2646476984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63081a0210 con 0x7f6308101c00 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.398+0000 7f630cfd7640 1 -- 172.21.15.89:0/2646476984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63081090a0 con 0x7f6308101c00 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.400+0000 7f62f77fe640 1 -- 172.21.15.89:0/2646476984 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f62f8042030 con 0x7f6308101c00 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.400+0000 7f62f77fe640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62f00a3300 0x7f62f00a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.400+0000 7f6305d9b640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62f00a3300 0x7f62f00a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.400+0000 7f62f77fe640 1 -- 172.21.15.89:0/2646476984 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f62f80f0640 con 0x7f6308101c00 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.401+0000 7f6305d9b640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62f00a3300 0x7f62f00a57c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f62fc005fd0 tx=0x7f62fc005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:55.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.403+0000 7f62f77fe640 1 -- 172.21.15.89:0/2646476984 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f62f80bcea0 con 0x7f6308101c00 2024-09-15T23:23:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:55 smithi120 ceph-mon[35250]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:55 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:55 smithi174 ceph-mon[35586]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:55 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:55.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.585+0000 7f630cfd7640 1 -- 172.21.15.89:0/2646476984 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f6308062630 con 0x7f62f00a3300 2024-09-15T23:23:55.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.594+0000 7f62f77fe640 1 -- 172.21.15.89:0/2646476984 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f6308062630 con 0x7f62f00a3300 2024-09-15T23:23:55.596 INFO:teuthology.orchestra.run.smithi089.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-15T23:23:55.596 INFO:teuthology.orchestra.run.smithi089.stdout:mds.all 12/12 1s ago 2m smithi089=a;smithi089=d;smithi089=g;smithi089=j;smithi120=b;smithi120=e;smithi120=h;smithi120=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-09-15T23:23:55.596 INFO:teuthology.orchestra.run.smithi089.stdout:mgr 3/3 1s ago 8m smithi089=x;smithi120=y;smithi174=z;count:3 2024-09-15T23:23:55.596 INFO:teuthology.orchestra.run.smithi089.stdout:mon 3/3 1s ago 8m smithi089:172.21.15.89=a;smithi120:172.21.15.120=b;smithi174:172.21.15.174=c;count:3 2024-09-15T23:23:55.596 INFO:teuthology.orchestra.run.smithi089.stdout:osd 12 1s ago - 2024-09-15T23:23:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 -- 172.21.15.89:0/2646476984 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62f00a3300 msgr2=0x7f62f00a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62f00a3300 0x7f62f00a57c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f62fc005fd0 tx=0x7f62fc005850 comp rx=0 tx=0).stop 2024-09-15T23:23:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 -- 172.21.15.89:0/2646476984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6308101c00 msgr2=0x7f6308197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:55.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6308101c00 0x7f6308197f00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f62f8031610 tx=0x7f62f80043b0 comp rx=0 tx=0).stop 2024-09-15T23:23:55.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 -- 172.21.15.89:0/2646476984 shutdown_connections 2024-09-15T23:23:55.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62f00a3300 0x7f62f00a57c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:55.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63081037c0 0x7f630819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:55.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6308102e00 0x7f6308198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:55.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 --2- 172.21.15.89:0/2646476984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6308101c00 0x7f6308197f00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:55.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 -- 172.21.15.89:0/2646476984 >> 172.21.15.89:0/2646476984 conn(0x7f63080fd390 msgr2=0x7f6308106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:55.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.596+0000 7f630cfd7640 1 -- 172.21.15.89:0/2646476984 shutdown_connections 2024-09-15T23:23:55.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:55.597+0000 7f630cfd7640 1 -- 172.21.15.89:0/2646476984 wait complete. 2024-09-15T23:23:56.277 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:56 smithi089 ceph-mon[30933]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:56.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:56.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:56.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:23:56.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:23:56.278 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:56.305 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-15T23:23:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:56 smithi120 ceph-mon[35250]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:56.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:23:56.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:23:56.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:56 smithi174 ceph-mon[35586]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:23:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:23:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:23:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:23:56.550 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:57.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 -- 172.21.15.89:0/1200214386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67781037b0 msgr2=0x7f677810a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:57.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 --2- 172.21.15.89:0/1200214386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67781037b0 0x7f677810a040 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6770009ca0 tx=0x7f677002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 -- 172.21.15.89:0/1200214386 shutdown_connections 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 --2- 172.21.15.89:0/1200214386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67781037b0 0x7f677810a040 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 --2- 172.21.15.89:0/1200214386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6778102df0 0x7f6778103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 --2- 172.21.15.89:0/1200214386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6778101bf0 0x7f6778101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 -- 172.21.15.89:0/1200214386 >> 172.21.15.89:0/1200214386 conn(0x7f67780fd3c0 msgr2=0x7f67780ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 -- 172.21.15.89:0/1200214386 shutdown_connections 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 -- 172.21.15.89:0/1200214386 wait complete. 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 Processor -- start 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.012+0000 7f67803a5640 1 -- start start 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f67803a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6778101bf0 0x7f6778075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f67803a5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6778102df0 0x7f67780745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:57.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f67803a5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67781037b0 0x7f6778074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f67803a5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f677810d8f0 con 0x7f67781037b0 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f67803a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f677810d770 con 0x7f6778101bf0 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f67803a5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f677810da70 con 0x7f6778102df0 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e942640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67781037b0 0x7f6778074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e942640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67781037b0 0x7f6778074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41436/0 (socket says 172.21.15.89:41436) 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e942640 1 -- 172.21.15.89:0/698862770 learned_addr learned my addr 172.21.15.89:0/698862770 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677d940640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6778102df0 0x7f67780745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e141640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6778101bf0 0x7f6778075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e942640 1 -- 172.21.15.89:0/698862770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6778102df0 msgr2=0x7f67780745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e942640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6778102df0 0x7f67780745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e942640 1 -- 172.21.15.89:0/698862770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6778101bf0 msgr2=0x7f6778075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e942640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6778101bf0 0x7f6778075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e942640 1 -- 172.21.15.89:0/698862770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67780753e0 con 0x7f67781037b0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e141640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6778101bf0 0x7f6778075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f677e942640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67781037b0 0x7f6778074b20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6770005e30 tx=0x7f6770031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.013+0000 7f676f7fe640 1 -- 172.21.15.89:0/698862770 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770003c30 con 0x7f67781037b0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.014+0000 7f676f7fe640 1 -- 172.21.15.89:0/698862770 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6770003dd0 con 0x7f67781037b0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.014+0000 7f67803a5640 1 -- 172.21.15.89:0/698862770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67781a2020 con 0x7f67781037b0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.014+0000 7f676f7fe640 1 -- 172.21.15.89:0/698862770 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677002fde0 con 0x7f67781037b0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.014+0000 7f67803a5640 1 -- 172.21.15.89:0/698862770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67781a2600 con 0x7f67781037b0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.014+0000 7f67803a5640 1 -- 172.21.15.89:0/698862770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6778108f20 con 0x7f67781037b0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.016+0000 7f676f7fe640 1 -- 172.21.15.89:0/698862770 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f6770004050 con 0x7f67781037b0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.017+0000 7f676f7fe640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67640a3350 0x7f67640a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.017+0000 7f676f7fe640 1 -- 172.21.15.89:0/698862770 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7f67700f1190 con 0x7f67781037b0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.017+0000 7f677e141640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67640a3350 0x7f67640a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.017+0000 7f677e141640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67640a3350 0x7f67640a5810 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6774006fd0 tx=0x7f6774008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:57.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.019+0000 7f676f7fe640 1 -- 172.21.15.89:0/698862770 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f67700bdcb0 con 0x7f67781037b0 2024-09-15T23:23:57.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.196+0000 7f67803a5640 1 -- 172.21.15.89:0/698862770 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f6778062630 con 0x7f67640a3350 2024-09-15T23:23:57.198 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:57 smithi089 ceph-mon[30933]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:57.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.197+0000 7f676f7fe640 1 -- 172.21.15.89:0/698862770 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f6778062630 con 0x7f67640a3350 2024-09-15T23:23:57.198 INFO:teuthology.orchestra.run.smithi089.stdout:HOST ADDR LABELS STATUS 2024-09-15T23:23:57.199 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089 172.21.15.89 2024-09-15T23:23:57.199 INFO:teuthology.orchestra.run.smithi089.stdout:smithi120 172.21.15.120 2024-09-15T23:23:57.199 INFO:teuthology.orchestra.run.smithi089.stdout:smithi174 172.21.15.174 2024-09-15T23:23:57.199 INFO:teuthology.orchestra.run.smithi089.stdout:3 hosts in cluster 2024-09-15T23:23:57.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.199+0000 7f67803a5640 1 -- 172.21.15.89:0/698862770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67640a3350 msgr2=0x7f67640a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:57.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.199+0000 7f67803a5640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67640a3350 0x7f67640a5810 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6774006fd0 tx=0x7f6774008040 comp rx=0 tx=0).stop 2024-09-15T23:23:57.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.199+0000 7f67803a5640 1 -- 172.21.15.89:0/698862770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67781037b0 msgr2=0x7f6778074b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:57.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.199+0000 7f67803a5640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67781037b0 0x7f6778074b20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6770005e30 tx=0x7f6770031ee0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.199+0000 7f67803a5640 1 -- 172.21.15.89:0/698862770 shutdown_connections 2024-09-15T23:23:57.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.200+0000 7f67803a5640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67640a3350 0x7f67640a5810 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.200+0000 7f67803a5640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67781037b0 0x7f6778074b20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.200+0000 7f67803a5640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6778102df0 0x7f67780745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.200+0000 7f67803a5640 1 --2- 172.21.15.89:0/698862770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6778101bf0 0x7f6778075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:57.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.200+0000 7f67803a5640 1 -- 172.21.15.89:0/698862770 >> 172.21.15.89:0/698862770 conn(0x7f67780fd3c0 msgr2=0x7f67781060f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:57.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.200+0000 7f67803a5640 1 -- 172.21.15.89:0/698862770 shutdown_connections 2024-09-15T23:23:57.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:57.200+0000 7f67803a5640 1 -- 172.21.15.89:0/698862770 wait complete. 2024-09-15T23:23:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:57 smithi120 ceph-mon[35250]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:57 smithi174 ceph-mon[35586]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:57.922 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-15T23:23:58.179 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:58.337 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:58 smithi089 ceph-mon[30933]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:58 smithi120 ceph-mon[35250]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:58.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:58 smithi174 ceph-mon[35586]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:23:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.662+0000 7feb3c63f640 1 -- 172.21.15.89:0/2273638505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb341079a0 msgr2=0x7feb34109d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.662+0000 7feb3c63f640 1 --2- 172.21.15.89:0/2273638505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb341079a0 0x7feb34109d90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009ca0 tx=0x7feb2c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:23:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.662+0000 7feb3c63f640 1 -- 172.21.15.89:0/2273638505 shutdown_connections 2024-09-15T23:23:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.662+0000 7feb3c63f640 1 --2- 172.21.15.89:0/2273638505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb341079a0 0x7feb34109d90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.662+0000 7feb3c63f640 1 --2- 172.21.15.89:0/2273638505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb34105070 0x7feb34107460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.662+0000 7feb3c63f640 1 --2- 172.21.15.89:0/2273638505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb34102740 0x7feb34104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.662+0000 7feb3c63f640 1 -- 172.21.15.89:0/2273638505 >> 172.21.15.89:0/2273638505 conn(0x7feb340fc2a0 msgr2=0x7feb340fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.662+0000 7feb3c63f640 1 -- 172.21.15.89:0/2273638505 shutdown_connections 2024-09-15T23:23:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.662+0000 7feb3c63f640 1 -- 172.21.15.89:0/2273638505 wait complete. 2024-09-15T23:23:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.662+0000 7feb3c63f640 1 Processor -- start 2024-09-15T23:23:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3c63f640 1 -- start start 2024-09-15T23:23:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3c63f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb34102740 0x7feb34197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3c63f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb34105070 0x7feb34198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3c63f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb341079a0 0x7feb3419f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3c63f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feb3410d8c0 con 0x7feb341079a0 2024-09-15T23:23:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3c63f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feb3410d740 con 0x7feb34102740 2024-09-15T23:23:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3c63f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb3410da40 con 0x7feb34105070 2024-09-15T23:23:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3abdc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb341079a0 0x7feb3419f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3abdc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb341079a0 0x7feb3419f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41462/0 (socket says 172.21.15.89:41462) 2024-09-15T23:23:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3abdc640 1 -- 172.21.15.89:0/1844135905 learned_addr learned my addr 172.21.15.89:0/1844135905 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:23:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb39bda640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb34105070 0x7feb34198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3a3db640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb34102740 0x7feb34197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:58.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3abdc640 1 -- 172.21.15.89:0/1844135905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb34105070 msgr2=0x7feb34198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:58.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3abdc640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb34105070 0x7feb34198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:58.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3abdc640 1 -- 172.21.15.89:0/1844135905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb34102740 msgr2=0x7feb34197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:58.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3abdc640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb34102740 0x7feb34197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:58.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.663+0000 7feb3abdc640 1 -- 172.21.15.89:0/1844135905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3419fa60 con 0x7feb341079a0 2024-09-15T23:23:58.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.664+0000 7feb3abdc640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb341079a0 0x7feb3419f3f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0098d0 tx=0x7feb2c02f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:58.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.664+0000 7feb2b7fe640 1 -- 172.21.15.89:0/1844135905 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c02fc30 con 0x7feb341079a0 2024-09-15T23:23:58.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.664+0000 7feb3c63f640 1 -- 172.21.15.89:0/1844135905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb3419fcf0 con 0x7feb341079a0 2024-09-15T23:23:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.664+0000 7feb2b7fe640 1 -- 172.21.15.89:0/1844135905 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb2c003e50 con 0x7feb341079a0 2024-09-15T23:23:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.664+0000 7feb2b7fe640 1 -- 172.21.15.89:0/1844135905 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2c004810 con 0x7feb341079a0 2024-09-15T23:23:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.664+0000 7feb3c63f640 1 -- 172.21.15.89:0/1844135905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb341a01d0 con 0x7feb341079a0 2024-09-15T23:23:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.665+0000 7feb3c63f640 1 -- 172.21.15.89:0/1844135905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb340fdd90 con 0x7feb341079a0 2024-09-15T23:23:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.666+0000 7feb2b7fe640 1 -- 172.21.15.89:0/1844135905 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7feb2c003970 con 0x7feb341079a0 2024-09-15T23:23:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.667+0000 7feb2b7fe640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb200a3300 0x7feb200a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.667+0000 7feb3a3db640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb200a3300 0x7feb200a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:58.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.667+0000 7feb2b7fe640 1 -- 172.21.15.89:0/1844135905 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7feb2c0f0fb0 con 0x7feb341079a0 2024-09-15T23:23:58.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.667+0000 7feb3a3db640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb200a3300 0x7feb200a57c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7feb300058c0 tx=0x7feb3000a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:58.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.671+0000 7feb2b7fe640 1 -- 172.21.15.89:0/1844135905 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7feb2c0bda40 con 0x7feb341079a0 2024-09-15T23:23:58.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.845+0000 7feb3c63f640 1 -- 172.21.15.89:0/1844135905 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7feb34062630 con 0x7feb200a3300 2024-09-15T23:23:58.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.846+0000 7feb2b7fe640 1 -- 172.21.15.89:0/1844135905 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7feb34062630 con 0x7feb200a3300 2024-09-15T23:23:58.848 INFO:teuthology.orchestra.run.smithi089.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-15T23:23:58.848 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040126400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-15T23:23:58.848 INFO:teuthology.orchestra.run.smithi089.stdout:smithi120 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400XV400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-15T23:23:58.848 INFO:teuthology.orchestra.run.smithi089.stdout:smithi174 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050074400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-15T23:23:58.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 -- 172.21.15.89:0/1844135905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb200a3300 msgr2=0x7feb200a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:58.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb200a3300 0x7feb200a57c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7feb300058c0 tx=0x7feb3000a250 comp rx=0 tx=0).stop 2024-09-15T23:23:58.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 -- 172.21.15.89:0/1844135905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb341079a0 msgr2=0x7feb3419f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:58.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb341079a0 0x7feb3419f3f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0098d0 tx=0x7feb2c02f840 comp rx=0 tx=0).stop 2024-09-15T23:23:58.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 -- 172.21.15.89:0/1844135905 shutdown_connections 2024-09-15T23:23:58.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb200a3300 0x7feb200a57c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:58.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb341079a0 0x7feb3419f3f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:58.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb34105070 0x7feb34198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:58.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 --2- 172.21.15.89:0/1844135905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb34102740 0x7feb34197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:58.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 -- 172.21.15.89:0/1844135905 >> 172.21.15.89:0/1844135905 conn(0x7feb340fc2a0 msgr2=0x7feb34102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:58.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.848+0000 7feb3c63f640 1 -- 172.21.15.89:0/1844135905 shutdown_connections 2024-09-15T23:23:58.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:23:58.849+0000 7feb3c63f640 1 -- 172.21.15.89:0/1844135905 wait complete. 2024-09-15T23:23:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:23:59 smithi120 ceph-mon[35250]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:23:59 smithi174 ceph-mon[35586]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:59.537 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:23:59 smithi089 ceph-mon[30933]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:23:59.565 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-15T23:23:59.573 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi089.front.sepia.ceph.com 2024-09-15T23:23:59.574 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- bash -c 'ceph fs dump' 2024-09-15T23:23:59.829 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:00.397 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:00 smithi089 ceph-mon[30933]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:24:00.397 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:00 smithi089 ceph-mon[30933]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:24:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.395+0000 7fca56d06640 1 -- 172.21.15.89:0/2665069428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca50102690 msgr2=0x7fca50108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.395+0000 7fca56d06640 1 --2- 172.21.15.89:0/2665069428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca50102690 0x7fca50108f20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fca48009ca0 tx=0x7fca4802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.395+0000 7fca56d06640 1 -- 172.21.15.89:0/2665069428 shutdown_connections 2024-09-15T23:24:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.395+0000 7fca56d06640 1 --2- 172.21.15.89:0/2665069428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca50102690 0x7fca50108f20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.395+0000 7fca56d06640 1 --2- 172.21.15.89:0/2665069428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca50101cd0 0x7fca50102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.395+0000 7fca56d06640 1 --2- 172.21.15.89:0/2665069428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50100ad0 0x7fca50100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.395+0000 7fca56d06640 1 -- 172.21.15.89:0/2665069428 >> 172.21.15.89:0/2665069428 conn(0x7fca500fc280 msgr2=0x7fca500fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.395+0000 7fca56d06640 1 -- 172.21.15.89:0/2665069428 shutdown_connections 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.395+0000 7fca56d06640 1 -- 172.21.15.89:0/2665069428 wait complete. 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.395+0000 7fca56d06640 1 Processor -- start 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca56d06640 1 -- start start 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca56d06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca50100ad0 0x7fca5006ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca56d06640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50101cd0 0x7fca5019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca56d06640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca50102690 0x7fca5019f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca56d06640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fca5010c7d0 con 0x7fca50100ad0 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca56d06640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fca5010c650 con 0x7fca50102690 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca56d06640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fca5010c950 con 0x7fca50101cd0 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca54aa2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca50100ad0 0x7fca5006ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca54aa2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca50100ad0 0x7fca5006ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41482/0 (socket says 172.21.15.89:41482) 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca54aa2640 1 -- 172.21.15.89:0/1179520303 learned_addr learned my addr 172.21.15.89:0/1179520303 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca552a3640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca50102690 0x7fca5019f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca4ffff640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50101cd0 0x7fca5019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca54aa2640 1 -- 172.21.15.89:0/1179520303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50101cd0 msgr2=0x7fca5019cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca54aa2640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50101cd0 0x7fca5019cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca54aa2640 1 -- 172.21.15.89:0/1179520303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca50102690 msgr2=0x7fca5019f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca54aa2640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca50102690 0x7fca5019f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.396+0000 7fca54aa2640 1 -- 172.21.15.89:0/1179520303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5019fb20 con 0x7fca50100ad0 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.397+0000 7fca54aa2640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca50100ad0 0x7fca5006ab10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fca4400b450 tx=0x7fca4400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.397+0000 7fca4dffb640 1 -- 172.21.15.89:0/1179520303 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca44015020 con 0x7fca50100ad0 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.397+0000 7fca56d06640 1 -- 172.21.15.89:0/1179520303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca5019fe10 con 0x7fca50100ad0 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.397+0000 7fca4dffb640 1 -- 172.21.15.89:0/1179520303 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca44011d70 con 0x7fca50100ad0 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.397+0000 7fca4dffb640 1 -- 172.21.15.89:0/1179520303 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4400f340 con 0x7fca50100ad0 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.397+0000 7fca56d06640 1 -- 172.21.15.89:0/1179520303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca501a0270 con 0x7fca50100ad0 2024-09-15T23:24:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.398+0000 7fca56d06640 1 -- 172.21.15.89:0/1179520303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca50107ea0 con 0x7fca50100ad0 2024-09-15T23:24:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.399+0000 7fca4dffb640 1 -- 172.21.15.89:0/1179520303 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fca44018020 con 0x7fca50100ad0 2024-09-15T23:24:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.400+0000 7fca4dffb640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca400a32d0 0x7fca400a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.400+0000 7fca4dffb640 1 -- 172.21.15.89:0/1179520303 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9092+0+0 (secure 0 0 0) 0x7fca440cf750 con 0x7fca50100ad0 2024-09-15T23:24:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.400+0000 7fca4ffff640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca400a32d0 0x7fca400a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.401+0000 7fca4ffff640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca400a32d0 0x7fca400a5790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fca5006a890 tx=0x7fca3c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.404+0000 7fca4dffb640 1 -- 172.21.15.89:0/1179520303 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fca4401f020 con 0x7fca50100ad0 2024-09-15T23:24:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:00 smithi120 ceph-mon[35250]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:24:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:00 smithi120 ceph-mon[35250]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:24:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:00 smithi174 ceph-mon[35586]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:24:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:00 smithi174 ceph-mon[35586]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-09-15T23:24:00.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.615+0000 7fca56d06640 1 -- 172.21.15.89:0/1179520303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fca5002d970 con 0x7fca50100ad0 2024-09-15T23:24:00.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.616+0000 7fca4dffb640 1 -- 172.21.15.89:0/1179520303 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3138 (secure 0 0 0) 0x7fca4400d040 con 0x7fca50100ad0 2024-09-15T23:24:00.618 INFO:teuthology.orchestra.run.smithi089.stdout:e28 2024-09-15T23:24:00.618 INFO:teuthology.orchestra.run.smithi089.stdout:btime 2024-09-15T23:22:37:131197+0000 2024-09-15T23:24:00.618 INFO:teuthology.orchestra.run.smithi089.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-15T23:24:00.618 INFO:teuthology.orchestra.run.smithi089.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:legacy client fscid: 1 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:Filesystem 'cephfs' (1) 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:fs_name cephfs 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 28 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-09-15T23:21:53.066091+0000 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2024-09-15T23:22:37.131175+0000 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:tableserver 0 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:root 0 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:session_timeout 300 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:session_autoclose 300 2024-09-15T23:24:00.619 INFO:teuthology.orchestra.run.smithi089.stdout:max_file_size 1099511627776 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:max_xattr_size 65536 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:required_client_features {} 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure 0 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:last_failure_osd_epoch 0 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:max_mds 5 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:in 0,1,2,3,4 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:up {0=24452,1=24436,2=24430,3=24428,4=24424} 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:failed 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:damaged 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:stopped 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:data_pools [3,4] 2024-09-15T23:24:00.620 INFO:teuthology.orchestra.run.smithi089.stdout:metadata_pool 2 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:inline_data disabled 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:balancer 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:bal_rank_mask -1 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:standby_count_wanted 0 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:qdb_cluster leader: 24452 members: 24452,24436,24430,24428,24424 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.b{0:24452} state up:active seq 4 addr [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.e{0:24404} state up:standby-replay seq 1 addr [v2:172.21.15.120:6834/1429316223,v1:172.21.15.120:6835/1429316223] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.i{1:24436} state up:active seq 16 addr [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.l{1:24403} state up:standby-replay seq 1 addr [v2:172.21.15.174:6832/2504210581,v1:172.21.15.174:6833/2504210581] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.c{2:24430} state up:active seq 18 addr [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.k{2:24389} state up:standby-replay seq 1 addr [v2:172.21.15.120:6832/4147766006,v1:172.21.15.120:6833/4147766006] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.621 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.h{3:24428} state up:active seq 17 addr [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.622 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.a{3:14616} state up:standby-replay seq 1 addr [v2:172.21.15.89:6840/3365004630,v1:172.21.15.89:6841/3365004630] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.622 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.f{4:24424} state up:active seq 19 addr [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.622 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.g{4:14610} state up:standby-replay seq 1 addr [v2:172.21.15.89:6838/2895111363,v1:172.21.15.89:6839/2895111363] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.622 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:00.622 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:00.622 INFO:teuthology.orchestra.run.smithi089.stdout:Standby daemons: 2024-09-15T23:24:00.622 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:00.622 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.j{-1:14538} state up:standby seq 1 addr [v2:172.21.15.89:6834/3816006418,v1:172.21.15.89:6835/3816006418] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.622 INFO:teuthology.orchestra.run.smithi089.stdout:[mds.d{-1:14550} state up:standby seq 1 addr [v2:172.21.15.89:6836/3164152358,v1:172.21.15.89:6837/3164152358] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:24:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 28 2024-09-15T23:24:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 -- 172.21.15.89:0/1179520303 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca400a32d0 msgr2=0x7fca400a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca400a32d0 0x7fca400a5790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fca5006a890 tx=0x7fca3c00b410 comp rx=0 tx=0).stop 2024-09-15T23:24:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 -- 172.21.15.89:0/1179520303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca50100ad0 msgr2=0x7fca5006ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca50100ad0 0x7fca5006ab10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fca4400b450 tx=0x7fca4400b910 comp rx=0 tx=0).stop 2024-09-15T23:24:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 -- 172.21.15.89:0/1179520303 shutdown_connections 2024-09-15T23:24:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca400a32d0 0x7fca400a5790 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca50102690 0x7fca5019f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50101cd0 0x7fca5019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 --2- 172.21.15.89:0/1179520303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca50100ad0 0x7fca5006ab10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 -- 172.21.15.89:0/1179520303 >> 172.21.15.89:0/1179520303 conn(0x7fca500fc280 msgr2=0x7fca50105070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 -- 172.21.15.89:0/1179520303 shutdown_connections 2024-09-15T23:24:00.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:00.619+0000 7fca56d06640 1 -- 172.21.15.89:0/1179520303 wait complete. 2024-09-15T23:24:01.146 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:01 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1179520303' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-15T23:24:01.377 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- bash -c 'ceph osd dump' 2024-09-15T23:24:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:01 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1179520303' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-15T23:24:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:01 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1179520303' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-15T23:24:01.624 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:02.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.110+0000 7f818d855640 1 -- 172.21.15.89:0/1864303207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81881037c0 msgr2=0x7f818810a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.110+0000 7f818d855640 1 --2- 172.21.15.89:0/1864303207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81881037c0 0x7f818810a050 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8180009ca0 tx=0x7f818002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.110+0000 7f818d855640 1 -- 172.21.15.89:0/1864303207 shutdown_connections 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.110+0000 7f818d855640 1 --2- 172.21.15.89:0/1864303207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81881037c0 0x7f818810a050 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.110+0000 7f818d855640 1 --2- 172.21.15.89:0/1864303207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8188102e00 0x7f8188103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.110+0000 7f818d855640 1 --2- 172.21.15.89:0/1864303207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101c00 0x7f8188102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.110+0000 7f818d855640 1 -- 172.21.15.89:0/1864303207 >> 172.21.15.89:0/1864303207 conn(0x7f81880fd390 msgr2=0x7f81880ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.110+0000 7f818d855640 1 -- 172.21.15.89:0/1864303207 shutdown_connections 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.110+0000 7f818d855640 1 -- 172.21.15.89:0/1864303207 wait complete. 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.111+0000 7f818d855640 1 Processor -- start 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.111+0000 7f818d855640 1 -- start start 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.112+0000 7f818d855640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101c00 0x7f8188197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.112+0000 7f818d855640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8188102e00 0x7f81881983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.112+0000 7f818d855640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81881037c0 0x7f818819f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.112+0000 7f818d855640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f818810d880 con 0x7f81881037c0 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.112+0000 7f818d855640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f818810d700 con 0x7f8188102e00 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.112+0000 7f818d855640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f818810da00 con 0x7f8188101c00 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.112+0000 7f81877fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81881037c0 0x7f818819f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.112+0000 7f81877fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81881037c0 0x7f818819f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41506/0 (socket says 172.21.15.89:41506) 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.112+0000 7f8186ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101c00 0x7f8188197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.112+0000 7f81877fe640 1 -- 172.21.15.89:0/2277167820 learned_addr learned my addr 172.21.15.89:0/2277167820 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.113+0000 7f8186ffd640 1 -- 172.21.15.89:0/2277167820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8188102e00 msgr2=0x7f81881983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.113+0000 7f8186ffd640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8188102e00 0x7f81881983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.113+0000 7f8186ffd640 1 -- 172.21.15.89:0/2277167820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81881037c0 msgr2=0x7f818819f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.113+0000 7f8186ffd640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81881037c0 0x7f818819f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.113+0000 7f8186ffd640 1 -- 172.21.15.89:0/2277167820 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818819faa0 con 0x7f8188101c00 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.113+0000 7f81877fe640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81881037c0 0x7f818819f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.113+0000 7f8186ffd640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101c00 0x7f8188197e70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8188102c60 tx=0x7f817800dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.120+0000 7f818c853640 1 -- 172.21.15.89:0/2277167820 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8178017070 con 0x7f8188101c00 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.120+0000 7f818c853640 1 -- 172.21.15.89:0/2277167820 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8178010460 con 0x7f8188101c00 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.120+0000 7f818c853640 1 -- 172.21.15.89:0/2277167820 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817800f620 con 0x7f8188101c00 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.120+0000 7f818d855640 1 -- 172.21.15.89:0/2277167820 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8188074480 con 0x7f8188101c00 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.120+0000 7f818d855640 1 -- 172.21.15.89:0/2277167820 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8188074970 con 0x7f8188101c00 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.121+0000 7f818d855640 1 -- 172.21.15.89:0/2277167820 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8188108fb0 con 0x7f8188101c00 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.122+0000 7f818c853640 1 -- 172.21.15.89:0/2277167820 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f817801a070 con 0x7f8188101c00 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.123+0000 7f818c853640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81700a3270 0x7f81700a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.123+0000 7f81867fc640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81700a3270 0x7f81700a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.123+0000 7f818c853640 1 -- 172.21.15.89:0/2277167820 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9237+0+0 (secure 0 0 0) 0x7f81780cf680 con 0x7f8188101c00 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.123+0000 7f81867fc640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81700a3270 0x7f81700a5730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f817c0058c0 tx=0x7f817c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:02.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.126+0000 7f818c853640 1 -- 172.21.15.89:0/2277167820 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f817809bfb0 con 0x7f8188101c00 2024-09-15T23:24:02.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.295+0000 7f818d855640 1 -- 172.21.15.89:0/2277167820 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f818802c640 con 0x7f8188101c00 2024-09-15T23:24:02.298 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:02 smithi089 ceph-mon[30933]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:24:02.298 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:02 smithi089 ceph-mon[30933]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-15T23:24:02.298 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-09-15T23:24:02.298 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-09-15T23:24:02.298 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-09-15T23:24:02.298 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]: dispatch 2024-09-15T23:24:02.298 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-09-15T23:24:02.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.296+0000 7f818c853640 1 -- 172.21.15.89:0/2277167820 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) ==== 56+0+5724 (secure 0 0 0) 0x7f817809b8b0 con 0x7f8188101c00 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 89 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-09-15T23:14:08.706430+0000 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2024-09-15T23:24:02.104679+0000 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:crush_version 27 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:full_ratio 0.95 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:backfillfull_ratio 0.9 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:nearfull_ratio 0.85 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:require_min_compat_client luminous 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:min_compat_client luminous 2024-09-15T23:24:02.299 INFO:teuthology.orchestra.run.smithi089.stdout:require_osd_release squid 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:stretch_mode_enabled false 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 20 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:max_osd 12 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6802/261479611,v1:172.21.15.89:6803/261479611] [v2:172.21.15.89:6804/261479611,v1:172.21.15.89:6805/261479611] exists,up e292c3de-4e37-4c40-9085-a85a262fcf6b 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1 up in weight 1 up_from 12 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6800/2529464017,v1:172.21.15.120:6801/2529464017] [v2:172.21.15.120:6802/2529464017,v1:172.21.15.120:6803/2529464017] exists,up fa999797-98c3-47f1-a08f-71a789176397 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2 up in weight 1 up_from 17 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6800/3193519940,v1:172.21.15.174:6801/3193519940] [v2:172.21.15.174:6802/3193519940,v1:172.21.15.174:6803/3193519940] exists,up b48eda20-370e-40bc-bdb3-4011a46df4a5 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3 up in weight 1 up_from 23 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6810/2725889139,v1:172.21.15.89:6811/2725889139] [v2:172.21.15.89:6812/2725889139,v1:172.21.15.89:6813/2725889139] exists,up 783de02c-1c46-44ed-98c4-10fb142648a2 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4 up in weight 1 up_from 30 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6808/135794330,v1:172.21.15.120:6809/135794330] [v2:172.21.15.120:6810/135794330,v1:172.21.15.120:6811/135794330] exists,up 425a48e4-145a-4f5c-88af-d7be7fdd3c51 2024-09-15T23:24:02.300 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5 up in weight 1 up_from 35 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6808/3950370591,v1:172.21.15.174:6809/3950370591] [v2:172.21.15.174:6810/3950370591,v1:172.21.15.174:6811/3950370591] exists,up 0548511d-1178-448e-8549-0a934b57cfdc 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6 up in weight 1 up_from 42 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6818/2734649081,v1:172.21.15.89:6819/2734649081] [v2:172.21.15.89:6820/2734649081,v1:172.21.15.89:6821/2734649081] exists,up df2eab94-90c7-4847-998f-bccdef2861e1 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7 up in weight 1 up_from 47 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6816/831027547,v1:172.21.15.120:6817/831027547] [v2:172.21.15.120:6818/831027547,v1:172.21.15.120:6819/831027547] exists,up 696fd052-0624-47fc-bdf6-bcfc6f69ff74 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:osd.8 up in weight 1 up_from 54 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6816/2158836361,v1:172.21.15.174:6817/2158836361] [v2:172.21.15.174:6818/2158836361,v1:172.21.15.174:6819/2158836361] exists,up 3f51541d-5288-4ed5-9780-516f0f26744b 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:osd.9 up in weight 1 up_from 61 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6826/2707132462,v1:172.21.15.89:6827/2707132462] [v2:172.21.15.89:6828/2707132462,v1:172.21.15.89:6829/2707132462] exists,up 81930b2a-3a54-4442-9c89-316644a3adce 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:osd.10 up in weight 1 up_from 66 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6824/254855304,v1:172.21.15.120:6825/254855304] [v2:172.21.15.120:6826/254855304,v1:172.21.15.120:6827/254855304] exists,up 586ff85c-4fa6-415d-8843-7e57328f9e41 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:osd.11 up in weight 1 up_from 74 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6824/246219377,v1:172.21.15.174:6825/246219377] [v2:172.21.15.174:6826/246219377,v1:172.21.15.174:6827/246219377] exists,up 06c5dd00-db2a-43e0-98e9-61468771c3ea 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 2.0 [8,4] 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 2.b [8,11] 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 2.e [2,1] 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 2.11 [7,11] 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 2.14 [2,4] 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.0 [4,8] 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.20 [5,9] 2024-09-15T23:24:02.301 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.24 [4,9] 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.2a [3,8] 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.33 [5,9] 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 3.3a [4,6] 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 4.12 [9,10] 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 4.15 [5,10] 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 4.16 [9,8] 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:pg_upmap_items 4.1f [9,8] 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:6801/429862840 expires 2024-09-16T23:15:00.884149+0000 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:6800/429862840 expires 2024-09-16T23:15:00.884149+0000 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/1174639743 expires 2024-09-16T23:15:00.884149+0000 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/3889269108 expires 2024-09-16T23:15:00.884149+0000 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/4118871044 expires 2024-09-16T23:15:00.884149+0000 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/379727762 expires 2024-09-16T23:14:32.424565+0000 2024-09-15T23:24:02.302 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:6801/4057509998 expires 2024-09-16T23:14:32.424565+0000 2024-09-15T23:24:02.303 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:6800/4057509998 expires 2024-09-16T23:14:32.424565+0000 2024-09-15T23:24:02.303 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/3796595264 expires 2024-09-16T23:14:32.424565+0000 2024-09-15T23:24:02.303 INFO:teuthology.orchestra.run.smithi089.stdout:blocklist 172.21.15.89:0/1030688282 expires 2024-09-16T23:14:32.424565+0000 2024-09-15T23:24:02.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 -- 172.21.15.89:0/2277167820 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81700a3270 msgr2=0x7f81700a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81700a3270 0x7f81700a5730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f817c0058c0 tx=0x7f817c005850 comp rx=0 tx=0).stop 2024-09-15T23:24:02.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 -- 172.21.15.89:0/2277167820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101c00 msgr2=0x7f8188197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101c00 0x7f8188197e70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8188102c60 tx=0x7f817800dbb0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 -- 172.21.15.89:0/2277167820 shutdown_connections 2024-09-15T23:24:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81700a3270 0x7f81700a5730 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81881037c0 0x7f818819f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8188102e00 0x7f81881983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 --2- 172.21.15.89:0/2277167820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101c00 0x7f8188197e70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 -- 172.21.15.89:0/2277167820 >> 172.21.15.89:0/2277167820 conn(0x7f81880fd390 msgr2=0x7f8188106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 -- 172.21.15.89:0/2277167820 shutdown_connections 2024-09-15T23:24:02.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:02.299+0000 7f818d855640 1 -- 172.21.15.89:0/2277167820 wait complete. 2024-09-15T23:24:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:02 smithi120 ceph-mon[35250]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:24:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:02 smithi120 ceph-mon[35250]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-15T23:24:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-09-15T23:24:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-09-15T23:24:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-09-15T23:24:02.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]: dispatch 2024-09-15T23:24:02.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-09-15T23:24:02.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:02 smithi174 ceph-mon[35586]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-09-15T23:24:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:02 smithi174 ceph-mon[35586]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-15T23:24:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-09-15T23:24:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-09-15T23:24:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-09-15T23:24:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]: dispatch 2024-09-15T23:24:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-09-15T23:24:03.018 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-15T23:24:03.029 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:03.283 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:03.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-09-15T23:24:03.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-09-15T23:24:03.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-09-15T23:24:03.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]': finished 2024-09-15T23:24:03.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-09-15T23:24:03.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:03 smithi089 ceph-mon[30933]: osdmap e89: 12 total, 12 up, 12 in 2024-09-15T23:24:03.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:03 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2277167820' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-15T23:24:03.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:03 smithi089 ceph-mon[30933]: pgmap v350: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 409 B/s wr, 10 op/s 2024-09-15T23:24:03.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:03 smithi089 ceph-mon[30933]: osdmap e90: 12 total, 12 up, 12 in 2024-09-15T23:24:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-09-15T23:24:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-09-15T23:24:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-09-15T23:24:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]': finished 2024-09-15T23:24:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-09-15T23:24:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:03 smithi120 ceph-mon[35250]: osdmap e89: 12 total, 12 up, 12 in 2024-09-15T23:24:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:03 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2277167820' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-15T23:24:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:03 smithi120 ceph-mon[35250]: pgmap v350: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 409 B/s wr, 10 op/s 2024-09-15T23:24:03.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:03 smithi120 ceph-mon[35250]: osdmap e90: 12 total, 12 up, 12 in 2024-09-15T23:24:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-09-15T23:24:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-09-15T23:24:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-09-15T23:24:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.11", "id": [7, 11]}]': finished 2024-09-15T23:24:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-09-15T23:24:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:03 smithi174 ceph-mon[35586]: osdmap e89: 12 total, 12 up, 12 in 2024-09-15T23:24:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:03 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2277167820' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-15T23:24:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:03 smithi174 ceph-mon[35586]: pgmap v350: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 409 B/s wr, 10 op/s 2024-09-15T23:24:03.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:03 smithi174 ceph-mon[35586]: osdmap e90: 12 total, 12 up, 12 in 2024-09-15T23:24:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.753+0000 7f16266f8640 1 -- 172.21.15.89:0/3659349494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1620101420 msgr2=0x7f162010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.753+0000 7f16266f8640 1 --2- 172.21.15.89:0/3659349494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1620101420 0x7f162010d950 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1618009cd0 tx=0x7f161802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:24:03.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.753+0000 7f16266f8640 1 -- 172.21.15.89:0/3659349494 shutdown_connections 2024-09-15T23:24:03.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.753+0000 7f16266f8640 1 --2- 172.21.15.89:0/3659349494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1620101420 0x7f162010d950 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:03.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.753+0000 7f16266f8640 1 --2- 172.21.15.89:0/3659349494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1620100a80 0x7f1620100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:03.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.753+0000 7f16266f8640 1 --2- 172.21.15.89:0/3659349494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620106a80 0x7f1620106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:03.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.753+0000 7f16266f8640 1 -- 172.21.15.89:0/3659349494 >> 172.21.15.89:0/3659349494 conn(0x7f16200fc7a0 msgr2=0x7f16200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:03.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.753+0000 7f16266f8640 1 -- 172.21.15.89:0/3659349494 shutdown_connections 2024-09-15T23:24:03.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.753+0000 7f16266f8640 1 -- 172.21.15.89:0/3659349494 wait complete. 2024-09-15T23:24:03.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f16266f8640 1 Processor -- start 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f16266f8640 1 -- start start 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f16266f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620100a80 0x7f16201983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f16266f8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1620101420 0x7f1620198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f16266f8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1620106a80 0x7f162019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f16266f8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1620110e80 con 0x7f1620101420 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f16266f8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1620110d00 con 0x7f1620106a80 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f16266f8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1620111000 con 0x7f1620100a80 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f161f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1620101420 0x7f1620198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f161f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1620101420 0x7f1620198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41538/0 (socket says 172.21.15.89:41538) 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f161f7fe640 1 -- 172.21.15.89:0/3003836034 learned_addr learned my addr 172.21.15.89:0/3003836034 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f1624c95640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1620106a80 0x7f162019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:03.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.754+0000 7f161ffff640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620100a80 0x7f16201983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f161f7fe640 1 -- 172.21.15.89:0/3003836034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620100a80 msgr2=0x7f16201983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f161f7fe640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620100a80 0x7f16201983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f161f7fe640 1 -- 172.21.15.89:0/3003836034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1620106a80 msgr2=0x7f162019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f161f7fe640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1620106a80 0x7f162019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f161f7fe640 1 -- 172.21.15.89:0/3003836034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f162019d380 con 0x7f1620101420 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f161ffff640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620100a80 0x7f16201983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f161f7fe640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1620101420 0x7f1620198910 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f161400c890 tx=0x7f161400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f161d7fa640 1 -- 172.21.15.89:0/3003836034 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614004220 con 0x7f1620101420 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f161d7fa640 1 -- 172.21.15.89:0/3003836034 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16140043c0 con 0x7f1620101420 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f16266f8640 1 -- 172.21.15.89:0/3003836034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f162019d670 con 0x7f1620101420 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f161d7fa640 1 -- 172.21.15.89:0/3003836034 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16140049e0 con 0x7f1620101420 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.755+0000 7f16266f8640 1 -- 172.21.15.89:0/3003836034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16201a4f50 con 0x7f1620101420 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.756+0000 7f16266f8640 1 -- 172.21.15.89:0/3003836034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1620100ee0 con 0x7f1620101420 2024-09-15T23:24:03.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.757+0000 7f161d7fa640 1 -- 172.21.15.89:0/3003836034 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f16140079b0 con 0x7f1620101420 2024-09-15T23:24:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.758+0000 7f161d7fa640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a3240 0x7f16080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.758+0000 7f161d7fa640 1 -- 172.21.15.89:0/3003836034 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f16140cfe10 con 0x7f1620101420 2024-09-15T23:24:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.758+0000 7f161ffff640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a3240 0x7f16080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.758+0000 7f161ffff640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a3240 0x7f16080a5700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f161000baa0 tx=0x7f1610005d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:03.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.762+0000 7f161d7fa640 1 -- 172.21.15.89:0/3003836034 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1614014030 con 0x7f1620101420 2024-09-15T23:24:03.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.966+0000 7f16266f8640 1 -- 172.21.15.89:0/3003836034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16201a51f0 con 0x7f1620101420 2024-09-15T23:24:03.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.967+0000 7f161d7fa640 1 -- 172.21.15.89:0/3003836034 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7f161409c740 con 0x7f1620101420 2024-09-15T23:24:03.972 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:03.972 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":28,"btime":"2024-09-15T23:22:37:131197+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:22:37.131175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24452,24436,24430,24428,24424]},"id":1}]} 2024-09-15T23:24:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 28 2024-09-15T23:24:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 -- 172.21.15.89:0/3003836034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a3240 msgr2=0x7f16080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a3240 0x7f16080a5700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f161000baa0 tx=0x7f1610005d20 comp rx=0 tx=0).stop 2024-09-15T23:24:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 -- 172.21.15.89:0/3003836034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1620101420 msgr2=0x7f1620198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1620101420 0x7f1620198910 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f161400c890 tx=0x7f161400cd50 comp rx=0 tx=0).stop 2024-09-15T23:24:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 -- 172.21.15.89:0/3003836034 shutdown_connections 2024-09-15T23:24:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a3240 0x7f16080a5700 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1620106a80 0x7f162019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:03.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1620101420 0x7f1620198910 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:03.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 --2- 172.21.15.89:0/3003836034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620100a80 0x7f16201983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:03.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 -- 172.21.15.89:0/3003836034 >> 172.21.15.89:0/3003836034 conn(0x7f16200fc7a0 msgr2=0x7f162006a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:03.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 -- 172.21.15.89:0/3003836034 shutdown_connections 2024-09-15T23:24:03.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:03.969+0000 7f16266f8640 1 -- 172.21.15.89:0/3003836034 wait complete. 2024-09-15T23:24:04.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:04 smithi089 ceph-mon[30933]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-15T23:24:04.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:04 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3003836034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:04 smithi120 ceph-mon[35250]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-15T23:24:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:04 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3003836034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:04.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:04 smithi174 ceph-mon[35586]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-15T23:24:04.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:04 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3003836034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:04.673 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:04.920 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:05.307 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:05 smithi089 ceph-mon[30933]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-15T23:24:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:05 smithi120 ceph-mon[35250]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-15T23:24:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:05 smithi174 ceph-mon[35586]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-15T23:24:05.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.489+0000 7f96cf727640 1 -- 172.21.15.89:0/4189195596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96c810c180 msgr2=0x7f96c810e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:05.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.489+0000 7f96cf727640 1 --2- 172.21.15.89:0/4189195596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96c810c180 0x7f96c810e600 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009ca0 tx=0x7f96c002a950 comp rx=0 tx=0).stop 2024-09-15T23:24:05.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.489+0000 7f96cf727640 1 -- 172.21.15.89:0/4189195596 shutdown_connections 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.489+0000 7f96cf727640 1 --2- 172.21.15.89:0/4189195596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96c810c180 0x7f96c810e600 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.489+0000 7f96cf727640 1 --2- 172.21.15.89:0/4189195596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c81036c0 0x7f96c810bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.489+0000 7f96cf727640 1 --2- 172.21.15.89:0/4189195596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c8102da0 0x7f96c8103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.489+0000 7f96cf727640 1 -- 172.21.15.89:0/4189195596 >> 172.21.15.89:0/4189195596 conn(0x7f96c80fc7a0 msgr2=0x7f96c80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.489+0000 7f96cf727640 1 -- 172.21.15.89:0/4189195596 shutdown_connections 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.489+0000 7f96cf727640 1 -- 172.21.15.89:0/4189195596 wait complete. 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cf727640 1 Processor -- start 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cf727640 1 -- start start 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cf727640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c8102da0 0x7f96c8198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cf727640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c81036c0 0x7f96c81989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cf727640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96c810c180 0x7f96c819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:05.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cf727640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f96c8111fa0 con 0x7f96c810c180 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cf727640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f96c8111e20 con 0x7f96c8102da0 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cf727640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f96c8112120 con 0x7f96c81036c0 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cdcc4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96c810c180 0x7f96c819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cdcc4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96c810c180 0x7f96c819cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43452/0 (socket says 172.21.15.89:43452) 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cdcc4640 1 -- 172.21.15.89:0/2118128376 learned_addr learned my addr 172.21.15.89:0/2118128376 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cd4c3640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c8102da0 0x7f96c8198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cccc2640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c81036c0 0x7f96c81989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cdcc4640 1 -- 172.21.15.89:0/2118128376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c81036c0 msgr2=0x7f96c81989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cdcc4640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c81036c0 0x7f96c81989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cdcc4640 1 -- 172.21.15.89:0/2118128376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c8102da0 msgr2=0x7f96c8198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cdcc4640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c8102da0 0x7f96c8198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cdcc4640 1 -- 172.21.15.89:0/2118128376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c819d430 con 0x7f96c810c180 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cccc2640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c81036c0 0x7f96c81989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:05.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.490+0000 7f96cd4c3640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c8102da0 0x7f96c8198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.491+0000 7f96cdcc4640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96c810c180 0x7f96c819cd50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f96c002ae70 tx=0x7f96c002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.491+0000 7f96be7fc640 1 -- 172.21.15.89:0/2118128376 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c0034040 con 0x7f96c810c180 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.491+0000 7f96cf727640 1 -- 172.21.15.89:0/2118128376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c819d6c0 con 0x7f96c810c180 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.491+0000 7f96be7fc640 1 -- 172.21.15.89:0/2118128376 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96c0030920 con 0x7f96c810c180 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.491+0000 7f96cf727640 1 -- 172.21.15.89:0/2118128376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c81a5000 con 0x7f96c810c180 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.491+0000 7f96be7fc640 1 -- 172.21.15.89:0/2118128376 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c0045480 con 0x7f96c810c180 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.492+0000 7f96cf727640 1 -- 172.21.15.89:0/2118128376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96c8103180 con 0x7f96c810c180 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.493+0000 7f96be7fc640 1 -- 172.21.15.89:0/2118128376 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f96c0037020 con 0x7f96c810c180 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.493+0000 7f96be7fc640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f96b40a3240 0x7f96b40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.494+0000 7f96cd4c3640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f96b40a3240 0x7f96b40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.494+0000 7f96be7fc640 1 -- 172.21.15.89:0/2118128376 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f96c00f0e80 con 0x7f96c810c180 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.494+0000 7f96cd4c3640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f96b40a3240 0x7f96b40a5700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009930 tx=0x7f96c4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:05.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.496+0000 7f96be7fc640 1 -- 172.21.15.89:0/2118128376 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f96c00bd840 con 0x7f96c810c180 2024-09-15T23:24:05.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.708+0000 7f96cf727640 1 -- 172.21.15.89:0/2118128376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96c8107230 con 0x7f96c810c180 2024-09-15T23:24:05.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.709+0000 7f96be7fc640 1 -- 172.21.15.89:0/2118128376 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11986 (secure 0 0 0) 0x7f96c00bd140 con 0x7f96c810c180 2024-09-15T23:24:05.711 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:05.711 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":28,"btime":"2024-09-15T23:22:37:131197+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:22:37.131175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24452,24436,24430,24428,24424]},"id":1}]} 2024-09-15T23:24:05.711 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 28 2024-09-15T23:24:05.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 -- 172.21.15.89:0/2118128376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f96b40a3240 msgr2=0x7f96b40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:05.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f96b40a3240 0x7f96b40a5700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009930 tx=0x7f96c4008040 comp rx=0 tx=0).stop 2024-09-15T23:24:05.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 -- 172.21.15.89:0/2118128376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96c810c180 msgr2=0x7f96c819cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:05.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96c810c180 0x7f96c819cd50 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f96c002ae70 tx=0x7f96c002f5b0 comp rx=0 tx=0).stop 2024-09-15T23:24:05.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 -- 172.21.15.89:0/2118128376 shutdown_connections 2024-09-15T23:24:05.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f96b40a3240 0x7f96b40a5700 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:05.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96c810c180 0x7f96c819cd50 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:05.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c81036c0 0x7f96c81989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:05.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 --2- 172.21.15.89:0/2118128376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f96c8102da0 0x7f96c8198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:05.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 -- 172.21.15.89:0/2118128376 >> 172.21.15.89:0/2118128376 conn(0x7f96c80fc7a0 msgr2=0x7f96c80fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:05.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 -- 172.21.15.89:0/2118128376 shutdown_connections 2024-09-15T23:24:05.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:05.711+0000 7f96cf727640 1 -- 172.21.15.89:0/2118128376 wait complete. 2024-09-15T23:24:06.368 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:06 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2118128376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:06.396 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:24:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:06 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2118128376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:06 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2118128376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:06.642 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.179+0000 7f117b6ec640 1 -- 172.21.15.89:0/290917116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1174106f70 msgr2=0x7f117410eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.179+0000 7f117b6ec640 1 --2- 172.21.15.89:0/290917116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1174106f70 0x7f117410eab0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f116c009900 tx=0x7f116c02a410 comp rx=0 tx=0).stop 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.179+0000 7f117b6ec640 1 -- 172.21.15.89:0/290917116 shutdown_connections 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.180+0000 7f117b6ec640 1 --2- 172.21.15.89:0/290917116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1174106f70 0x7f117410eab0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.180+0000 7f117b6ec640 1 --2- 172.21.15.89:0/290917116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1174100eb0 0x7f1174106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.180+0000 7f117b6ec640 1 --2- 172.21.15.89:0/290917116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1174100500 0x7f11741008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.180+0000 7f117b6ec640 1 -- 172.21.15.89:0/290917116 >> 172.21.15.89:0/290917116 conn(0x7f11740fc5c0 msgr2=0x7f11740fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.180+0000 7f117b6ec640 1 -- 172.21.15.89:0/290917116 shutdown_connections 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.180+0000 7f117b6ec640 1 -- 172.21.15.89:0/290917116 wait complete. 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.180+0000 7f117b6ec640 1 Processor -- start 2024-09-15T23:24:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.180+0000 7f117b6ec640 1 -- start start 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f117b6ec640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1174100500 0x7f11741984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f117b6ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1174100eb0 0x7f11741989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f117b6ec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1174106f70 0x7f117419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f117b6ec640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1174112010 con 0x7f1174106f70 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f117b6ec640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1174111e90 con 0x7f1174100eb0 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f117b6ec640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1174112190 con 0x7f1174100500 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1179c89640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1174106f70 0x7f117419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1179c89640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1174106f70 0x7f117419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43474/0 (socket says 172.21.15.89:43474) 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1179c89640 1 -- 172.21.15.89:0/1264342485 learned_addr learned my addr 172.21.15.89:0/1264342485 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1179488640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1174100500 0x7f11741984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1178c87640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1174100eb0 0x7f11741989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1179c89640 1 -- 172.21.15.89:0/1264342485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1174100500 msgr2=0x7f11741984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1179c89640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1174100500 0x7f11741984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1179c89640 1 -- 172.21.15.89:0/1264342485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1174100eb0 msgr2=0x7f11741989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1179c89640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1174100eb0 0x7f11741989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1179c89640 1 -- 172.21.15.89:0/1264342485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117419d460 con 0x7f1174106f70 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.181+0000 7f1179c89640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1174106f70 0x7f117419cd80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f116c009230 tx=0x7f116c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.182+0000 7f11727fc640 1 -- 172.21.15.89:0/1264342485 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c03f030 con 0x7f1174106f70 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.182+0000 7f11727fc640 1 -- 172.21.15.89:0/1264342485 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f116c003ce0 con 0x7f1174106f70 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.182+0000 7f117b6ec640 1 -- 172.21.15.89:0/1264342485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f117419d6f0 con 0x7f1174106f70 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.182+0000 7f11727fc640 1 -- 172.21.15.89:0/1264342485 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c035070 con 0x7f1174106f70 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.182+0000 7f117b6ec640 1 -- 172.21.15.89:0/1264342485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11741a5030 con 0x7f1174106f70 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.183+0000 7f117b6ec640 1 -- 172.21.15.89:0/1264342485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11741008e0 con 0x7f1174106f70 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.184+0000 7f11727fc640 1 -- 172.21.15.89:0/1264342485 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f116c042030 con 0x7f1174106f70 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.184+0000 7f11727fc640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f115c0a32c0 0x7f115c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.185+0000 7f1179488640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f115c0a32c0 0x7f115c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.185+0000 7f11727fc640 1 -- 172.21.15.89:0/1264342485 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f116c0f04e0 con 0x7f1174106f70 2024-09-15T23:24:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.185+0000 7f1179488640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f115c0a32c0 0x7f115c0a5780 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1174069fc0 tx=0x7f116400d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.189+0000 7f11727fc640 1 -- 172.21.15.89:0/1264342485 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f116c0bc7c0 con 0x7f1174106f70 2024-09-15T23:24:07.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.356+0000 7f117b6ec640 1 -- 172.21.15.89:0/1264342485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f117410a0e0 con 0x7f1174106f70 2024-09-15T23:24:07.357 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:07 smithi089 ceph-mon[30933]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 639 B/s wr, 10 op/s; 7.7 KiB/s, 1 objects/s recovering 2024-09-15T23:24:07.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.357+0000 7f11727fc640 1 -- 172.21.15.89:0/1264342485 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24962 (secure 0 0 0) 0x7f116c0c0ac0 con 0x7f1174106f70 2024-09-15T23:24:07.359 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:07.359 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":90,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:24:03.106312+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.11","mappings":[{"from":7,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:24:07.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 -- 172.21.15.89:0/1264342485 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f115c0a32c0 msgr2=0x7f115c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:07.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f115c0a32c0 0x7f115c0a5780 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1174069fc0 tx=0x7f116400d040 comp rx=0 tx=0).stop 2024-09-15T23:24:07.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 -- 172.21.15.89:0/1264342485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1174106f70 msgr2=0x7f117419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:07.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1174106f70 0x7f117419cd80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f116c009230 tx=0x7f116c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:24:07.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 -- 172.21.15.89:0/1264342485 shutdown_connections 2024-09-15T23:24:07.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f115c0a32c0 0x7f115c0a5780 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:07.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1174106f70 0x7f117419cd80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:07.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1174100eb0 0x7f11741989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:07.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 --2- 172.21.15.89:0/1264342485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1174100500 0x7f11741984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:07.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 -- 172.21.15.89:0/1264342485 >> 172.21.15.89:0/1264342485 conn(0x7f11740fc5c0 msgr2=0x7f11741057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:07.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 -- 172.21.15.89:0/1264342485 shutdown_connections 2024-09-15T23:24:07.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:07.360+0000 7f117b6ec640 1 -- 172.21.15.89:0/1264342485 wait complete. 2024-09-15T23:24:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:07 smithi120 ceph-mon[35250]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 639 B/s wr, 10 op/s; 7.7 KiB/s, 1 objects/s recovering 2024-09-15T23:24:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:07 smithi174 ceph-mon[35586]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 639 B/s wr, 10 op/s; 7.7 KiB/s, 1 objects/s recovering 2024-09-15T23:24:08.055 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-15T23:21:53.066091+0000', 'modified': '2024-09-15T23:22:37.131175+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24452, 'mds_1': 24436, 'mds_2': 24430, 'mds_3': 24428, 'mds_4': 24424}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14610': {'gid': 14610, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.89:6839/2895111363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6838', 'nonce': 2895111363}, {'type': 'v1', 'addr': '172.21.15.89:6839', 'nonce': 2895111363}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14616': {'gid': 14616, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.89:6841/3365004630', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6840', 'nonce': 3365004630}, {'type': 'v1', 'addr': '172.21.15.89:6841', 'nonce': 3365004630}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24389': {'gid': 24389, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6833/4147766006', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6832', 'nonce': 4147766006}, {'type': 'v1', 'addr': '172.21.15.120:6833', 'nonce': 4147766006}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24403': {'gid': 24403, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6833/2504210581', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6832', 'nonce': 2504210581}, {'type': 'v1', 'addr': '172.21.15.174:6833', 'nonce': 2504210581}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24404': {'gid': 24404, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1429316223', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1429316223}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1429316223}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.174:6835/733541815', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 733541815}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 733541815}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24428': {'gid': 24428, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.120:6837/991556551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 991556551}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 991556551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.174:6837/3626951914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 3626951914}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 3626951914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.174:6839/3380568300', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 3380568300}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 3380568300}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.120:6839/1554218591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 1554218591}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 1554218591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24452, 'qdb_cluster': [24452, 24436, 24430, 24428, 24424]} 2024-09-15T23:24:08.055 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-15T23:24:08.055 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-15T23:24:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:08.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:08 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1264342485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:08 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1264342485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:08.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:08 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1264342485' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:08.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.798+0000 7f4cb728d640 1 -- 172.21.15.89:0/3446042163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb0102790 msgr2=0x7f4cb01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:08.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.798+0000 7f4cb728d640 1 --2- 172.21.15.89:0/3446042163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb0102790 0x7f4cb01067e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009900 tx=0x7f4ca002a410 comp rx=0 tx=0).stop 2024-09-15T23:24:08.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.798+0000 7f4cb728d640 1 -- 172.21.15.89:0/3446042163 shutdown_connections 2024-09-15T23:24:08.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.798+0000 7f4cb728d640 1 --2- 172.21.15.89:0/3446042163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb0106f10 0x7f4cb010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.798+0000 7f4cb728d640 1 --2- 172.21.15.89:0/3446042163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb0102790 0x7f4cb01067e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.798+0000 7f4cb728d640 1 --2- 172.21.15.89:0/3446042163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb0101de0 0x7f4cb01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.798+0000 7f4cb728d640 1 -- 172.21.15.89:0/3446042163 >> 172.21.15.89:0/3446042163 conn(0x7f4cb00fd8c0 msgr2=0x7f4cb00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.798+0000 7f4cb728d640 1 -- 172.21.15.89:0/3446042163 shutdown_connections 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.798+0000 7f4cb728d640 1 -- 172.21.15.89:0/3446042163 wait complete. 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb728d640 1 Processor -- start 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb728d640 1 -- start start 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb728d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb0101de0 0x7f4cb0198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb728d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb0102790 0x7f4cb01989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb728d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb0106f10 0x7f4cb019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb728d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb0111f70 con 0x7f4cb0102790 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb728d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb0111df0 con 0x7f4cb0106f10 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb728d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb01120f0 con 0x7f4cb0101de0 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb4828640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb0102790 0x7f4cb01989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb4828640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb0102790 0x7f4cb01989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43500/0 (socket says 172.21.15.89:43500) 2024-09-15T23:24:08.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb4828640 1 -- 172.21.15.89:0/1073273381 learned_addr learned my addr 172.21.15.89:0/1073273381 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb5029640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb0101de0 0x7f4cb0198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb4828640 1 -- 172.21.15.89:0/1073273381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb0101de0 msgr2=0x7f4cb0198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb4828640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb0101de0 0x7f4cb0198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb4828640 1 -- 172.21.15.89:0/1073273381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb0106f10 msgr2=0x7f4cb019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb582a640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb0106f10 0x7f4cb019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb4828640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb0106f10 0x7f4cb019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb4828640 1 -- 172.21.15.89:0/1073273381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb019d440 con 0x7f4cb0102790 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.799+0000 7f4cb4828640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb0102790 0x7f4cb01989d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0006010 tx=0x7f4ca00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.800+0000 7f4ca67fc640 1 -- 172.21.15.89:0/1073273381 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca003f030 con 0x7f4cb0102790 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.800+0000 7f4ca67fc640 1 -- 172.21.15.89:0/1073273381 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca0003ce0 con 0x7f4cb0102790 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.800+0000 7f4ca67fc640 1 -- 172.21.15.89:0/1073273381 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0035070 con 0x7f4cb0102790 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.800+0000 7f4cb728d640 1 -- 172.21.15.89:0/1073273381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb019d6d0 con 0x7f4cb0102790 2024-09-15T23:24:08.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.800+0000 7f4cb728d640 1 -- 172.21.15.89:0/1073273381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb01a5010 con 0x7f4cb0102790 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.802+0000 7f4ca67fc640 1 -- 172.21.15.89:0/1073273381 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4ca0042030 con 0x7f4cb0102790 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.802+0000 7f4cb9666740 1 Processor -- start 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.802+0000 7f4cb9666740 1 -- start start 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.802+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced12fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.802+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced13a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4ca67fc640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4c9c0a32c0 0x7f4c9c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced13cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb9666740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558ced096090 con 0x558ced1302f0 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb9666740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ced03da70 con 0x558ced12f9d0 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb9666740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558ced092050 con 0x558ced13abb0 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb5029640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced13a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb4828640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4c9c0a32c0 0x7f4c9c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb5029640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced13a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43516/0 (socket says 172.21.15.89:43516) 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb5029640 1 -- 172.21.15.89:0/2585751033 learned_addr learned my addr 172.21.15.89:0/2585751033 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4ca67fc640 1 -- 172.21.15.89:0/1073273381 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4ca00f0340 con 0x7f4cb0102790 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb582a640 1 --2- 172.21.15.89:0/2585751033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced12fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb582a640 1 --2- 172.21.15.89:0/2585751033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced13cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb5029640 1 -- 172.21.15.89:0/2585751033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 msgr2=0x558ced13cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb5029640 1 --2- 172.21.15.89:0/2585751033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced13cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb5029640 1 -- 172.21.15.89:0/2585751033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 msgr2=0x558ced12fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb5029640 1 --2- 172.21.15.89:0/2585751033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced12fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb5029640 1 -- 172.21.15.89:0/2585751033 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ced04a130 con 0x558ced1302f0 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb582a640 1 --2- 172.21.15.89:0/2585751033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced13cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb4828640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4c9c0a32c0 0x7f4c9c0a5780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0005ab0 tx=0x7f4ca0087520 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:08.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4cb5029640 1 --2- 172.21.15.89:0/2585751033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced13a670 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00bcf0 tx=0x7f4cac00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68cdc4e8173f56cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4c9b7fe640 1 -- 172.21.15.89:0/2585751033 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac015070 con 0x558ced1302f0 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4c9b7fe640 1 -- 172.21.15.89:0/2585751033 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cac01a940 con 0x558ced1302f0 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.803+0000 7f4c9b7fe640 1 -- 172.21.15.89:0/2585751033 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac0119d0 con 0x558ced1302f0 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.804+0000 7f4cb9666740 1 -- 172.21.15.89:0/2585751033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 msgr2=0x558ced13a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.804+0000 7f4cb9666740 1 --2- 172.21.15.89:0/2585751033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced13a670 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00bcf0 tx=0x7f4cac00f730 comp rx=0 tx=0).stop 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.804+0000 7f4cb9666740 1 -- 172.21.15.89:0/2585751033 shutdown_connections 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.804+0000 7f4cb9666740 1 --2- 172.21.15.89:0/2585751033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced13cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.804+0000 7f4cb9666740 1 --2- 172.21.15.89:0/2585751033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced13a670 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.804+0000 7f4cb9666740 1 --2- 172.21.15.89:0/2585751033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced12fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.804+0000 7f4cb9666740 1 -- 172.21.15.89:0/2585751033 >> 172.21.15.89:0/2585751033 conn(0x558ced12bf60 msgr2=0x558ced12c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.804+0000 7f4cb9666740 1 -- 172.21.15.89:0/2585751033 shutdown_connections 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.804+0000 7f4cb9666740 1 -- 172.21.15.89:0/2585751033 wait complete. 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb9666740 1 -- start start 2024-09-15T23:24:08.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb9666740 1 Processor -- start 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced13a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1377c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced137d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb9666740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558ced13d780 con 0x558ced1302f0 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb9666740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ced050f20 con 0x558ced12f9d0 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb9666740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558ced13d900 con 0x558ced13abb0 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb9666740 10 client.0 _ll_register_callbacks cb 0x558ced01a030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb5029640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb5029640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1377c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43526/0 (socket says 172.21.15.89:43526) 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb5029640 1 -- 172.21.15.89:0/250971564 learned_addr learned my addr 172.21.15.89:0/250971564 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:08.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.805+0000 7f4cb582a640 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced13a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb582a640 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced137d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb5029640 1 -- 172.21.15.89:0/250971564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 msgr2=0x558ced137d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb5029640 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced137d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb5029640 1 -- 172.21.15.89:0/250971564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 msgr2=0x558ced13a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb5029640 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced13a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb5029640 1 -- 172.21.15.89:0/250971564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cece072b0 con 0x558ced1302f0 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb582a640 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced13a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb5029640 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1377c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00b840 tx=0x7f4cac00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e3280d872935f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4c98ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4c98ff9640 1 -- 172.21.15.89:0/250971564 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac028060 con 0x558ced1302f0 2024-09-15T23:24:08.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4c98ff9640 1 -- 172.21.15.89:0/250971564 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4cac01aaf0 con 0x558ced1302f0 2024-09-15T23:24:08.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb9666740 1 -- 172.21.15.89:0/250971564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cac01a940 con 0x558ced1302f0 2024-09-15T23:24:08.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4c98ff9640 1 -- 172.21.15.89:0/250971564 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4cac02f050 con 0x558ced1302f0 2024-09-15T23:24:08.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb9666740 10 client.15222 fetch_fsmap learned FSMap version 28 2024-09-15T23:24:08.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb9666740 1 -- 172.21.15.89:0/250971564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558ced149040 con 0x558ced1302f0 2024-09-15T23:24:08.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4c98ff9640 1 -- 172.21.15.89:0/250971564 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cac01a940 con 0x558ced1302f0 2024-09-15T23:24:08.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4c98ff9640 1 -- 172.21.15.89:0/250971564 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f4cac011a80 con 0x558ced1302f0 2024-09-15T23:24:08.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb9666740 10 client.15222 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:24:08.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb9666740 10 client.15222 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T23:24:08.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.806+0000 7f4cb9666740 20 client.15222 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.807+0000 7f4cb9666740 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558ced14b320 0x558ced14c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.807+0000 7f4cb9666740 4 client.15222 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.807+0000 7f4cb9666740 1 -- 172.21.15.89:0/250971564 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ced0a3600 con 0x558ced14b320 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.807+0000 7f4cb582a640 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558ced14b320 0x558ced14c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.807+0000 7f4cb582a640 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558ced14b320 0x558ced14c790 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c54ac95f4cad2d52 server_cookie=2c1dfe4399c8ba90 in_seq=0 out_seq=0 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.807+0000 7f4c98ff9640 10 client.15222 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4c98ff9640 1 -- 172.21.15.89:0/250971564 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558ced0a3600 con 0x558ced14b320 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4c98ff9640 10 client.15222 handle_command_reply: tid=0 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4cb9666740 1 client.15222 shutdown 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4cb9666740 1 -- 172.21.15.89:0/250971564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 msgr2=0x558ced1377c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4cb9666740 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1377c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00b840 tx=0x7f4cac00b990 comp rx=0 tx=0).stop 2024-09-15T23:24:08.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4cb9666740 1 -- 172.21.15.89:0/250971564 shutdown_connections 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4cb9666740 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558ced14b320 0x558ced14c790 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4cb9666740 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced137d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4cb9666740 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1377c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4cb9666740 1 --2- 172.21.15.89:0/250971564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced13a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4c98ff9640 0 client.15222 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4cb9666740 1 -- 172.21.15.89:0/250971564 >> 172.21.15.89:0/250971564 conn(0x558ced12bf60 msgr2=0x558ced131c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.808+0000 7f4cb9666740 1 -- 172.21.15.89:0/250971564 shutdown_connections 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.809+0000 7f4cb9666740 1 -- 172.21.15.89:0/250971564 wait complete. 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.809+0000 7f4cb9666740 20 client.15222 trim_cache size 0 max 16384 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb9666740 1 Processor -- start 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb9666740 1 -- start start 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced12f9d0 0x558ced14b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced1302f0 0x558ced14ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced13abb0 0x558ced139620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb9666740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558ced14aad0 con 0x558ced13abb0 2024-09-15T23:24:08.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb9666740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ced14b1a0 con 0x558ced1302f0 2024-09-15T23:24:08.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb9666740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558ced14b870 con 0x558ced12f9d0 2024-09-15T23:24:08.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb582a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced13abb0 0x558ced139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb582a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced13abb0 0x558ced139620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43536/0 (socket says 172.21.15.89:43536) 2024-09-15T23:24:08.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb582a640 1 -- 172.21.15.89:0/1503205906 learned_addr learned my addr 172.21.15.89:0/1503205906 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:08.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb5029640 1 --2- 172.21.15.89:0/1503205906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced1302f0 0x558ced14ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb582a640 1 -- 172.21.15.89:0/1503205906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced12f9d0 msgr2=0x558ced14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:08.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb582a640 1 --2- 172.21.15.89:0/1503205906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced12f9d0 0x558ced14b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb582a640 1 -- 172.21.15.89:0/1503205906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced1302f0 msgr2=0x558ced14ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb582a640 1 --2- 172.21.15.89:0/1503205906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced1302f0 0x558ced14ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb582a640 1 -- 172.21.15.89:0/1503205906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ced1382e0 con 0x558ced13abb0 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.826+0000 7f4cb5029640 1 --2- 172.21.15.89:0/1503205906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced1302f0 0x558ced14ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4cb582a640 1 --2- 172.21.15.89:0/1503205906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced13abb0 0x558ced139620 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8005cc0 tx=0x7f4ca8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5776adff0394243 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4c9affd640 1 -- 172.21.15.89:0/1503205906 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8004b20 con 0x558ced13abb0 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4c9affd640 1 -- 172.21.15.89:0/1503205906 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca8004cc0 con 0x558ced13abb0 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4c9affd640 1 -- 172.21.15.89:0/1503205906 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8005440 con 0x558ced13abb0 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4cb9666740 1 -- 172.21.15.89:0/1503205906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced13abb0 msgr2=0x558ced139620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4cb9666740 1 --2- 172.21.15.89:0/1503205906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced13abb0 0x558ced139620 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8005cc0 tx=0x7f4ca8011a30 comp rx=0 tx=0).stop 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4cb9666740 1 -- 172.21.15.89:0/1503205906 shutdown_connections 2024-09-15T23:24:08.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4cb9666740 1 --2- 172.21.15.89:0/1503205906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced13abb0 0x558ced139620 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4cb9666740 1 --2- 172.21.15.89:0/1503205906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced1302f0 0x558ced14ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4cb9666740 1 --2- 172.21.15.89:0/1503205906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced12f9d0 0x558ced14b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4cb9666740 1 -- 172.21.15.89:0/1503205906 >> 172.21.15.89:0/1503205906 conn(0x558ced12bf60 msgr2=0x558ced138f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4cb9666740 1 -- 172.21.15.89:0/1503205906 shutdown_connections 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.827+0000 7f4cb9666740 1 -- 172.21.15.89:0/1503205906 wait complete. 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb9666740 1 -- start start 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb9666740 1 Processor -- start 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced150060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1505a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb9666740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced1ada60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb9666740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558ced14c610 con 0x558ced1302f0 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb9666740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ced14bf40 con 0x558ced12f9d0 2024-09-15T23:24:08.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb9666740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558ced14cce0 con 0x558ced13abb0 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb9666740 10 client.0 _ll_register_callbacks cb 0x7f4cac00bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb5029640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb5029640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1505a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43542/0 (socket says 172.21.15.89:43542) 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb5029640 1 -- 172.21.15.89:0/735589615 learned_addr learned my addr 172.21.15.89:0/735589615 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.828+0000 7f4cb582a640 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb582a640 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced1ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb5029640 1 -- 172.21.15.89:0/735589615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 msgr2=0x558ced1ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb5029640 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced1ada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb5029640 1 -- 172.21.15.89:0/735589615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 msgr2=0x558ced150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb5029640 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb5029640 1 -- 172.21.15.89:0/735589615 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ced1ae020 con 0x558ced1302f0 2024-09-15T23:24:08.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb582a640 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb5029640 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1505a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4cac006010 tx=0x7f4cac0048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937f2a1db551e983 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4c98ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4c98ff9640 1 -- 172.21.15.89:0/735589615 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac015070 con 0x558ced1302f0 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4c98ff9640 1 -- 172.21.15.89:0/735589615 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4cac011720 con 0x558ced1302f0 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb9666740 1 -- 172.21.15.89:0/735589615 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ca8004cc0 con 0x558ced1302f0 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4c98ff9640 1 -- 172.21.15.89:0/735589615 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4cac041050 con 0x558ced1302f0 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb9666740 10 client.15228 fetch_fsmap learned FSMap version 28 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4cb9666740 1 -- 172.21.15.89:0/735589615 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558ced1b5400 con 0x558ced1302f0 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.829+0000 7f4c98ff9640 1 -- 172.21.15.89:0/735589615 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca8004cc0 con 0x558ced1302f0 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.830+0000 7f4c98ff9640 1 -- 172.21.15.89:0/735589615 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f4cac024450 con 0x558ced1302f0 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.830+0000 7f4cb9666740 10 client.15228 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:24:08.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.830+0000 7f4cb9666740 10 client.15228 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.830+0000 7f4cb9666740 20 client.15228 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.830+0000 7f4cb9666740 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558ced1ba030 0x558ced1b8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.830+0000 7f4cb9666740 4 client.15228 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.830+0000 7f4cb9666740 1 -- 172.21.15.89:0/735589615 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "status"}) -- 0x558ced09db10 con 0x558ced1ba030 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.830+0000 7f4cb582a640 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558ced1ba030 0x558ced1b8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.830+0000 7f4c98ff9640 10 client.15228 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.830+0000 7f4cb582a640 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558ced1ba030 0x558ced1b8d40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a57d254a60445c54 server_cookie=e08fdb3186c3694a in_seq=0 out_seq=0 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4c98ff9640 1 -- 172.21.15.89:0/735589615 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558ced09db10 con 0x558ced1ba030 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4c98ff9640 10 client.15228 handle_command_reply: tid=0 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 client.15228 shutdown 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 -- 172.21.15.89:0/735589615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 msgr2=0x558ced1505a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1505a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4cac006010 tx=0x7f4cac0048a0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 -- 172.21.15.89:0/735589615 shutdown_connections 2024-09-15T23:24:08.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558ced1ba030 0x558ced1b8d40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ced13abb0 0x558ced1ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ced1302f0 0x558ced1505a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 --2- 172.21.15.89:0/735589615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ced12f9d0 0x558ced150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 -- 172.21.15.89:0/735589615 >> 172.21.15.89:0/735589615 conn(0x558ced12bf60 msgr2=0x558ced12b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:08.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4c98ff9640 0 client.15228 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:24:08.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 -- 172.21.15.89:0/735589615 shutdown_connections 2024-09-15T23:24:08.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 1 -- 172.21.15.89:0/735589615 wait complete. 2024-09-15T23:24:08.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb9666740 20 client.15228 trim_cache size 0 max 16384 2024-09-15T23:24:08.850 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:08.850 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 4, 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24424, 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 92.705391726000002, 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 28, 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 90, 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 162.64951009399999 2024-09-15T23:24:08.851 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:08.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb728d640 1 -- 172.21.15.89:0/1073273381 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4c9c0a32c0 msgr2=0x7f4c9c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:08.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb728d640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4c9c0a32c0 0x7f4c9c0a5780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0005ab0 tx=0x7f4ca0087520 comp rx=0 tx=0).stop 2024-09-15T23:24:08.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb728d640 1 -- 172.21.15.89:0/1073273381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb0102790 msgr2=0x7f4cb01989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:08.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.831+0000 7f4cb728d640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb0102790 0x7f4cb01989d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0006010 tx=0x7f4ca00042f0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.832+0000 7f4cb728d640 1 -- 172.21.15.89:0/1073273381 shutdown_connections 2024-09-15T23:24:08.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.832+0000 7f4cb728d640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4c9c0a32c0 0x7f4c9c0a5780 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.832+0000 7f4cb728d640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb0106f10 0x7f4cb019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.832+0000 7f4cb728d640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb0102790 0x7f4cb01989d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.832+0000 7f4cb728d640 1 --2- 172.21.15.89:0/1073273381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb0101de0 0x7f4cb0198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:08.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.832+0000 7f4cb728d640 1 -- 172.21.15.89:0/1073273381 >> 172.21.15.89:0/1073273381 conn(0x7f4cb00fd8c0 msgr2=0x7f4cb00fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:08.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.832+0000 7f4cb728d640 1 -- 172.21.15.89:0/1073273381 shutdown_connections 2024-09-15T23:24:08.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:08.832+0000 7f4cb728d640 1 -- 172.21.15.89:0/1073273381 wait complete. 2024-09-15T23:24:09.453 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:09 smithi089 ceph-mon[30933]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 639 B/s wr, 10 op/s; 7.7 KiB/s, 1 objects/s recovering 2024-09-15T23:24:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:09 smithi120 ceph-mon[35250]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 639 B/s wr, 10 op/s; 7.7 KiB/s, 1 objects/s recovering 2024-09-15T23:24:09.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:09 smithi174 ceph-mon[35586]: pgmap v354: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 639 B/s wr, 10 op/s; 7.7 KiB/s, 1 objects/s recovering 2024-09-15T23:24:09.519 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-15T23:24:09.766 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.276+0000 7f123d137640 1 -- 172.21.15.89:0/3992764316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1238100a80 msgr2=0x7f1238100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.276+0000 7f123d137640 1 --2- 172.21.15.89:0/3992764316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1238100a80 0x7f1238100ee0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f122c009930 tx=0x7f122c02a420 comp rx=0 tx=0).stop 2024-09-15T23:24:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.276+0000 7f123d137640 1 -- 172.21.15.89:0/3992764316 shutdown_connections 2024-09-15T23:24:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.276+0000 7f123d137640 1 --2- 172.21.15.89:0/3992764316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1238101420 0x7f123810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.276+0000 7f123d137640 1 --2- 172.21.15.89:0/3992764316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1238100a80 0x7f1238100ee0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.276+0000 7f123d137640 1 --2- 172.21.15.89:0/3992764316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1238106a80 0x7f1238106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.276+0000 7f123d137640 1 -- 172.21.15.89:0/3992764316 >> 172.21.15.89:0/3992764316 conn(0x7f12380fc7a0 msgr2=0x7f12380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.276+0000 7f123d137640 1 -- 172.21.15.89:0/3992764316 shutdown_connections 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.276+0000 7f123d137640 1 -- 172.21.15.89:0/3992764316 wait complete. 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.276+0000 7f123d137640 1 Processor -- start 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f123d137640 1 -- start start 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f123d137640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1238100a80 0x7f1238199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f123d137640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1238101420 0x7f12381997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f123d137640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1238106a80 0x7f12381a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f123d137640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1238110ea0 con 0x7f1238100a80 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f123d137640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1238110d20 con 0x7f1238101420 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f123d137640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1238111020 con 0x7f1238106a80 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f1236d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1238100a80 0x7f1238199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f1236d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1238100a80 0x7f1238199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43556/0 (socket says 172.21.15.89:43556) 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f1236d9d640 1 -- 172.21.15.89:0/3044251730 learned_addr learned my addr 172.21.15.89:0/3044251730 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f123659c640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1238101420 0x7f12381997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f123759e640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1238106a80 0x7f12381a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f1236d9d640 1 -- 172.21.15.89:0/3044251730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1238106a80 msgr2=0x7f12381a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f1236d9d640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1238106a80 0x7f12381a4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f1236d9d640 1 -- 172.21.15.89:0/3044251730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1238101420 msgr2=0x7f12381997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f1236d9d640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1238101420 0x7f12381997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f1236d9d640 1 -- 172.21.15.89:0/3044251730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12381a4850 con 0x7f1238100a80 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.277+0000 7f123659c640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1238101420 0x7f12381997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.278+0000 7f1236d9d640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1238100a80 0x7f1238199290 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f122800c920 tx=0x7f122800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.278+0000 7f121ffff640 1 -- 172.21.15.89:0/3044251730 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1228007c70 con 0x7f1238100a80 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.278+0000 7f123d137640 1 -- 172.21.15.89:0/3044251730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12381a4a20 con 0x7f1238100a80 2024-09-15T23:24:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.278+0000 7f121ffff640 1 -- 172.21.15.89:0/3044251730 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1228007e10 con 0x7f1238100a80 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.278+0000 7f121ffff640 1 -- 172.21.15.89:0/3044251730 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1228018890 con 0x7f1238100a80 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.278+0000 7f123d137640 1 -- 172.21.15.89:0/3044251730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12381a4ee0 con 0x7f1238100a80 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.280+0000 7f121ffff640 1 -- 172.21.15.89:0/3044251730 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1228018a70 con 0x7f1238100a80 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.280+0000 7f123f510740 1 Processor -- start 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.280+0000 7f123f510740 1 -- start start 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f123f510740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83db99d0 0x557f83db9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f123f510740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dc4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f123f510740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dc4bb0 0x557f83dc6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f123f510740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557f83d20090 con 0x557f83dba2f0 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f123f510740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f83cc7a70 con 0x557f83dc4bb0 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f123f510740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557f83d1c050 con 0x557f83db99d0 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f121ffff640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12200a3240 0x7f12200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f1236d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dc4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f1236d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dc4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43572/0 (socket says 172.21.15.89:43572) 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f1236d9d640 1 -- 172.21.15.89:0/3058618110 learned_addr learned my addr 172.21.15.89:0/3058618110 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f123759e640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12200a3240 0x7f12200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f123759e640 1 --2- 172.21.15.89:0/3058618110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83db99d0 0x557f83db9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f121ffff640 1 -- 172.21.15.89:0/3044251730 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f12280cff00 con 0x7f1238100a80 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f123659c640 1 --2- 172.21.15.89:0/3058618110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dc4bb0 0x557f83dc6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f1236d9d640 1 -- 172.21.15.89:0/3058618110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83db99d0 msgr2=0x557f83db9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f1236d9d640 1 --2- 172.21.15.89:0/3058618110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83db99d0 0x557f83db9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f1236d9d640 1 -- 172.21.15.89:0/3058618110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dc4bb0 msgr2=0x557f83dc6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f1236d9d640 1 --2- 172.21.15.89:0/3058618110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dc4bb0 0x557f83dc6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f1236d9d640 1 -- 172.21.15.89:0/3058618110 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f83cd4130 con 0x557f83dba2f0 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f123759e640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12200a3240 0x7f12200a5700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f12300144c0 tx=0x7f1230001b60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.281+0000 7f1236d9d640 1 --2- 172.21.15.89:0/3058618110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dc4670 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1228098b00 tx=0x7f122800ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87926b3b9d34c84e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f121d7fa640 1 -- 172.21.15.89:0/3058618110 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122809c830 con 0x557f83dba2f0 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f121d7fa640 1 -- 172.21.15.89:0/3058618110 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f122800de20 con 0x557f83dba2f0 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f121d7fa640 1 -- 172.21.15.89:0/3058618110 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12280bc0f0 con 0x557f83dba2f0 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f123f510740 1 -- 172.21.15.89:0/3058618110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 msgr2=0x557f83dc4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f123f510740 1 --2- 172.21.15.89:0/3058618110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dc4670 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1228098b00 tx=0x7f122800ff30 comp rx=0 tx=0).stop 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f123f510740 1 -- 172.21.15.89:0/3058618110 shutdown_connections 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f123f510740 1 --2- 172.21.15.89:0/3058618110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dc4bb0 0x557f83dc6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f123f510740 1 --2- 172.21.15.89:0/3058618110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dc4670 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f123f510740 1 --2- 172.21.15.89:0/3058618110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83db99d0 0x557f83db9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f123f510740 1 -- 172.21.15.89:0/3058618110 >> 172.21.15.89:0/3058618110 conn(0x557f83db5f60 msgr2=0x557f83db6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f123f510740 1 -- 172.21.15.89:0/3058618110 shutdown_connections 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.282+0000 7f123f510740 1 -- 172.21.15.89:0/3058618110 wait complete. 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123f510740 1 -- start start 2024-09-15T23:24:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123f510740 1 Processor -- start 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123f510740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dc4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123f510740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dba2f0 0x557f83dc1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123f510740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 0x557f83dc1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123f510740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557f83dc7780 con 0x557f83db99d0 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123f510740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f83cdaf20 con 0x557f83dba2f0 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123f510740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557f83dc7900 con 0x557f83dc4bb0 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123f510740 10 client.0 _ll_register_callbacks cb 0x557f83ca4030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f1236d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dc4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f1236d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dc4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43574/0 (socket says 172.21.15.89:43574) 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f1236d9d640 1 -- 172.21.15.89:0/309310124 learned_addr learned my addr 172.21.15.89:0/309310124 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123659c640 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dba2f0 0x557f83dc1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123759e640 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 0x557f83dc1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f1236d9d640 1 -- 172.21.15.89:0/309310124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 msgr2=0x557f83dc1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f1236d9d640 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 0x557f83dc1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f1236d9d640 1 -- 172.21.15.89:0/309310124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dba2f0 msgr2=0x557f83dc1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f1236d9d640 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dba2f0 0x557f83dc1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f1236d9d640 1 -- 172.21.15.89:0/309310124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f83a912b0 con 0x557f83db99d0 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123659c640 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dba2f0 0x557f83dc1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.283+0000 7f123759e640 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 0x557f83dc1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f1236d9d640 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dc4500 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f122809c0c0 tx=0x7f12280a4da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cccb15c99eddcb4d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f121aff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f121aff5640 1 -- 172.21.15.89:0/309310124 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122807e030 con 0x557f83db99d0 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f121aff5640 1 -- 172.21.15.89:0/309310124 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f12280bc0f0 con 0x557f83db99d0 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f123f510740 1 -- 172.21.15.89:0/309310124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f122800de20 con 0x557f83db99d0 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f121aff5640 1 -- 172.21.15.89:0/309310124 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f12280a04e0 con 0x557f83db99d0 2024-09-15T23:24:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f123f510740 10 client.15240 fetch_fsmap learned FSMap version 28 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f123f510740 1 -- 172.21.15.89:0/309310124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f83dd3150 con 0x557f83db99d0 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f121aff5640 1 -- 172.21.15.89:0/309310124 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f122800de20 con 0x557f83db99d0 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f121aff5640 1 -- 172.21.15.89:0/309310124 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f122809b580 con 0x557f83db99d0 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f123f510740 10 client.15240 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f123f510740 10 client.15240 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f123f510740 20 client.15240 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f123f510740 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557f83dd44a0 0x557f83dd65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f123f510740 4 client.15240 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f123f510740 1 -- 172.21.15.89:0/309310124 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f83d2d600 con 0x557f83dd44a0 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.284+0000 7f123659c640 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557f83dd44a0 0x557f83dd65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.285+0000 7f121aff5640 10 client.15240 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.285+0000 7f123659c640 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557f83dd44a0 0x557f83dd65a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b5bfd5da33e56848 server_cookie=4b5518a2d2359a57 in_seq=0 out_seq=0 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f121aff5640 1 -- 172.21.15.89:0/309310124 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557f83d2d600 con 0x557f83dd44a0 2024-09-15T23:24:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f121aff5640 10 client.15240 handle_command_reply: tid=0 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 client.15240 shutdown 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 -- 172.21.15.89:0/309310124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 msgr2=0x557f83dc4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dc4500 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f122809c0c0 tx=0x7f12280a4da0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 -- 172.21.15.89:0/309310124 shutdown_connections 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557f83dd44a0 0x557f83dd65a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 0x557f83dc1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dba2f0 0x557f83dc1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 --2- 172.21.15.89:0/309310124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dc4500 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f121aff5640 0 client.15240 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 -- 172.21.15.89:0/309310124 >> 172.21.15.89:0/309310124 conn(0x557f83db5f60 msgr2=0x557f83dbbc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 -- 172.21.15.89:0/309310124 shutdown_connections 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 1 -- 172.21.15.89:0/309310124 wait complete. 2024-09-15T23:24:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.286+0000 7f123f510740 20 client.15240 trim_cache size 0 max 16384 2024-09-15T23:24:10.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123f510740 1 Processor -- start 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123f510740 1 -- start start 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123f510740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83db99d0 0x557f83dd5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123f510740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dd5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123f510740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 0x557f83dd5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123f510740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557f83dd48e0 con 0x557f83dba2f0 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123f510740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f83dd4fb0 con 0x557f83db99d0 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123f510740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557f83dd5680 con 0x557f83dc4bb0 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dd5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dd5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43580/0 (socket says 172.21.15.89:43580) 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123659c640 1 -- 172.21.15.89:0/2941137238 learned_addr learned my addr 172.21.15.89:0/2941137238 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123759e640 1 --2- 172.21.15.89:0/2941137238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 0x557f83dd5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f1236d9d640 1 --2- 172.21.15.89:0/2941137238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83db99d0 0x557f83dd5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123659c640 1 -- 172.21.15.89:0/2941137238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 msgr2=0x557f83dd5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123659c640 1 --2- 172.21.15.89:0/2941137238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 0x557f83dd5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123659c640 1 -- 172.21.15.89:0/2941137238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83db99d0 msgr2=0x557f83dd5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123659c640 1 --2- 172.21.15.89:0/2941137238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83db99d0 0x557f83dd5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.305+0000 7f123659c640 1 -- 172.21.15.89:0/2941137238 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f83dc7890 con 0x557f83dba2f0 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.306+0000 7f123659c640 1 --2- 172.21.15.89:0/2941137238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dd5130 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f122c000c00 tx=0x7f122c0314e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b01413ef2c17d28 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.306+0000 7f121cff9640 1 -- 172.21.15.89:0/2941137238 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122c048070 con 0x557f83dba2f0 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.306+0000 7f121cff9640 1 -- 172.21.15.89:0/2941137238 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f122c005500 con 0x557f83dba2f0 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.306+0000 7f121cff9640 1 -- 172.21.15.89:0/2941137238 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122c035040 con 0x557f83dba2f0 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 -- 172.21.15.89:0/2941137238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 msgr2=0x557f83dd5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 --2- 172.21.15.89:0/2941137238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dd5130 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f122c000c00 tx=0x7f122c0314e0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 -- 172.21.15.89:0/2941137238 shutdown_connections 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 --2- 172.21.15.89:0/2941137238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dc4bb0 0x557f83dd5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 --2- 172.21.15.89:0/2941137238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83dba2f0 0x557f83dd5130 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 --2- 172.21.15.89:0/2941137238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83db99d0 0x557f83dd5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 -- 172.21.15.89:0/2941137238 >> 172.21.15.89:0/2941137238 conn(0x557f83db5f60 msgr2=0x557f83dc0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 -- 172.21.15.89:0/2941137238 shutdown_connections 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 -- 172.21.15.89:0/2941137238 wait complete. 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 -- start start 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 Processor -- start 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dd9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dba2f0 0x557f83dda4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dc4bb0 0x557f83dde600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557f83dd6420 con 0x557f83db99d0 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f83dd5d50 con 0x557f83dc4bb0 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557f83dd6af0 con 0x557f83dba2f0 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.307+0000 7f123f510740 10 client.0 _ll_register_callbacks cb 0x7f122800a560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f1236d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dd9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f1236d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dd9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43588/0 (socket says 172.21.15.89:43588) 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f1236d9d640 1 -- 172.21.15.89:0/3780270469 learned_addr learned my addr 172.21.15.89:0/3780270469 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:10.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f123759e640 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dc4bb0 0x557f83dde600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f123659c640 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dba2f0 0x557f83dda4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f1236d9d640 1 -- 172.21.15.89:0/3780270469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dba2f0 msgr2=0x557f83dda4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f1236d9d640 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dba2f0 0x557f83dda4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f1236d9d640 1 -- 172.21.15.89:0/3780270469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dc4bb0 msgr2=0x557f83dde600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f1236d9d640 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dc4bb0 0x557f83dde600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f1236d9d640 1 -- 172.21.15.89:0/3780270469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f83ddec70 con 0x557f83db99d0 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f123659c640 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dba2f0 0x557f83dda4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f1236d9d640 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dd9f80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1228007590 tx=0x7f1228020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f48a65af8216b84 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f121aff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f121aff5640 1 -- 172.21.15.89:0/3780270469 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122807e040 con 0x557f83db99d0 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f121aff5640 1 -- 172.21.15.89:0/3780270469 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f122809b580 con 0x557f83db99d0 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f123f510740 1 -- 172.21.15.89:0/3780270469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f122c005500 con 0x557f83db99d0 2024-09-15T23:24:10.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f121aff5640 1 -- 172.21.15.89:0/3780270469 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f122800de20 con 0x557f83db99d0 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f123f510740 10 client.15246 fetch_fsmap learned FSMap version 28 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f123f510740 1 -- 172.21.15.89:0/3780270469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f83e3f2f0 con 0x557f83db99d0 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.308+0000 7f121aff5640 1 -- 172.21.15.89:0/3780270469 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f122c005500 con 0x557f83db99d0 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.309+0000 7f121aff5640 1 -- 172.21.15.89:0/3780270469 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f12280a4910 con 0x557f83db99d0 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.309+0000 7f123f510740 10 client.15246 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.309+0000 7f123f510740 10 client.15246 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.309+0000 7f123f510740 20 client.15246 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.309+0000 7f123f510740 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557f83e43c60 0x557f83e42970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.309+0000 7f123f510740 4 client.15246 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.309+0000 7f123f510740 1 -- 172.21.15.89:0/3780270469 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "status"}) -- 0x557f83d27b10 con 0x557f83e43c60 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.309+0000 7f123659c640 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557f83e43c60 0x557f83e42970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.309+0000 7f123659c640 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557f83e43c60 0x557f83e42970 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=862283b4ef620674 server_cookie=8e33d54b21236616 in_seq=0 out_seq=0 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.309+0000 7f121aff5640 10 client.15246 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f121aff5640 1 -- 172.21.15.89:0/3780270469 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557f83d27b10 con 0x557f83e43c60 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f121aff5640 10 client.15246 handle_command_reply: tid=0 2024-09-15T23:24:10.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 client.15246 shutdown 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 -- 172.21.15.89:0/3780270469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 msgr2=0x557f83dd9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dd9f80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1228007590 tx=0x7f1228020000 comp rx=0 tx=0).stop 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 -- 172.21.15.89:0/3780270469 shutdown_connections 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557f83e43c60 0x557f83e42970 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f83dc4bb0 0x557f83dde600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f83dba2f0 0x557f83dda4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 --2- 172.21.15.89:0/3780270469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f83db99d0 0x557f83dd9f80 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 -- 172.21.15.89:0/3780270469 >> 172.21.15.89:0/3780270469 conn(0x557f83db5f60 msgr2=0x557f83dc0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f121aff5640 0 client.15246 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 -- 172.21.15.89:0/3780270469 shutdown_connections 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 1 -- 172.21.15.89:0/3780270469 wait complete. 2024-09-15T23:24:10.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123f510740 20 client.15246 trim_cache size 0 max 16384 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 3, 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24428, 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 95.199086605000005, 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 28, 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 90, 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:24:10.320 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 158.86666525000001 2024-09-15T23:24:10.321 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:10.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123d137640 1 -- 172.21.15.89:0/3044251730 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12200a3240 msgr2=0x7f12200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:10.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123d137640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12200a3240 0x7f12200a5700 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f12300144c0 tx=0x7f1230001b60 comp rx=0 tx=0).stop 2024-09-15T23:24:10.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123d137640 1 -- 172.21.15.89:0/3044251730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1238100a80 msgr2=0x7f1238199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:10.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.310+0000 7f123d137640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1238100a80 0x7f1238199290 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f122800c920 tx=0x7f122800cde0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.311+0000 7f123d137640 1 -- 172.21.15.89:0/3044251730 shutdown_connections 2024-09-15T23:24:10.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.311+0000 7f123d137640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f12200a3240 0x7f12200a5700 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.311+0000 7f123d137640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1238106a80 0x7f12381a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.311+0000 7f123d137640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1238101420 0x7f12381997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.311+0000 7f123d137640 1 --2- 172.21.15.89:0/3044251730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1238100a80 0x7f1238199290 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:10.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.311+0000 7f123d137640 1 -- 172.21.15.89:0/3044251730 >> 172.21.15.89:0/3044251730 conn(0x7f12380fc7a0 msgr2=0x7f12380fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:10.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.311+0000 7f123d137640 1 -- 172.21.15.89:0/3044251730 shutdown_connections 2024-09-15T23:24:10.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:10.311+0000 7f123d137640 1 -- 172.21.15.89:0/3044251730 wait complete. 2024-09-15T23:24:10.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:10 smithi120 ceph-mon[35250]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-15T23:24:10.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:10 smithi120 ceph-mon[35250]: Cluster is now healthy 2024-09-15T23:24:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:10 smithi174 ceph-mon[35586]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-15T23:24:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:10 smithi174 ceph-mon[35586]: Cluster is now healthy 2024-09-15T23:24:10.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:10 smithi089 ceph-mon[30933]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-15T23:24:10.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:10 smithi089 ceph-mon[30933]: Cluster is now healthy 2024-09-15T23:24:11.031 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-15T23:24:11.277 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:11.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:11 smithi089 ceph-mon[30933]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 114 B/s wr, 6 op/s; 7.0 KiB/s, 1 objects/s recovering 2024-09-15T23:24:11.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:11 smithi120 ceph-mon[35250]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 114 B/s wr, 6 op/s; 7.0 KiB/s, 1 objects/s recovering 2024-09-15T23:24:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:11 smithi174 ceph-mon[35586]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 114 B/s wr, 6 op/s; 7.0 KiB/s, 1 objects/s recovering 2024-09-15T23:24:11.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 -- 172.21.15.89:0/1286396295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd81027a0 msgr2=0x7f1cd81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:11.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 --2- 172.21.15.89:0/1286396295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd81027a0 0x7f1cd81067f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009900 tx=0x7f1ccc02a410 comp rx=0 tx=0).stop 2024-09-15T23:24:11.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 -- 172.21.15.89:0/1286396295 shutdown_connections 2024-09-15T23:24:11.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 --2- 172.21.15.89:0/1286396295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd8106f20 0x7f1cd810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 --2- 172.21.15.89:0/1286396295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd81027a0 0x7f1cd81067f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 --2- 172.21.15.89:0/1286396295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cd8101df0 0x7f1cd81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 -- 172.21.15.89:0/1286396295 >> 172.21.15.89:0/1286396295 conn(0x7f1cd80fd8f0 msgr2=0x7f1cd80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 -- 172.21.15.89:0/1286396295 shutdown_connections 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 -- 172.21.15.89:0/1286396295 wait complete. 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 Processor -- start 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.778+0000 7f1cdf67c640 1 -- start start 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdf67c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd8101df0 0x7f1cd8198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdf67c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd81027a0 0x7f1cd8198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdf67c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cd8106f20 0x7f1cd819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdf67c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd8111f80 con 0x7f1cd81027a0 2024-09-15T23:24:11.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdf67c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd8111e00 con 0x7f1cd8106f20 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdf67c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd8112100 con 0x7f1cd8101df0 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdcc17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd81027a0 0x7f1cd8198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdcc17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd81027a0 0x7f1cd8198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43602/0 (socket says 172.21.15.89:43602) 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdcc17640 1 -- 172.21.15.89:0/950822800 learned_addr learned my addr 172.21.15.89:0/950822800 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cddc19640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cd8106f20 0x7f1cd819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdd418640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd8101df0 0x7f1cd8198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdcc17640 1 -- 172.21.15.89:0/950822800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd8101df0 msgr2=0x7f1cd8198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd8101df0 0x7f1cd8198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdcc17640 1 -- 172.21.15.89:0/950822800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cd8106f20 msgr2=0x7f1cd819ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cd8106f20 0x7f1cd819ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdcc17640 1 -- 172.21.15.89:0/950822800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd819d3b0 con 0x7f1cd81027a0 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdd418640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd8101df0 0x7f1cd8198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:11.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd81027a0 0x7f1cd8198940 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc006010 tx=0x7f1ccc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cd67fc640 1 -- 172.21.15.89:0/950822800 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc03f030 con 0x7f1cd81027a0 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cd67fc640 1 -- 172.21.15.89:0/950822800 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ccc003ce0 con 0x7f1cd81027a0 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdf67c640 1 -- 172.21.15.89:0/950822800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd819d640 con 0x7f1cd81027a0 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cd67fc640 1 -- 172.21.15.89:0/950822800 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc035070 con 0x7f1cd81027a0 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.779+0000 7f1cdf67c640 1 -- 172.21.15.89:0/950822800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd81a4f80 con 0x7f1cd81027a0 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.782+0000 7f1cd67fc640 1 -- 172.21.15.89:0/950822800 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1ccc042030 con 0x7f1cd81027a0 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.782+0000 7f1ce1a55740 1 Processor -- start 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.782+0000 7f1ce1a55740 1 -- start start 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99349d0 0x555bd9934db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cd67fc640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cc00a3240 0x7f1cc00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99352f0 0x555bd993f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 0x555bd9941fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555bd989b090 con 0x555bd99352f0 2024-09-15T23:24:11.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555bd9842a70 con 0x555bd99349d0 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555bd9897050 con 0x555bd993fbb0 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdcc17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99352f0 0x555bd993f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdd418640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cc00a3240 0x7f1cc00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdcc17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99352f0 0x555bd993f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43618/0 (socket says 172.21.15.89:43618) 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cddc19640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99349d0 0x555bd9934db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdcc17640 1 -- 172.21.15.89:0/2316363842 learned_addr learned my addr 172.21.15.89:0/2316363842 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cd67fc640 1 -- 172.21.15.89:0/950822800 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1ccc0f0420 con 0x7f1cd81027a0 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cddc19640 1 --2- 172.21.15.89:0/2316363842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 0x555bd9941fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdcc17640 1 -- 172.21.15.89:0/2316363842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 msgr2=0x555bd9941fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/2316363842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 0x555bd9941fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdcc17640 1 -- 172.21.15.89:0/2316363842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99349d0 msgr2=0x555bd9934db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/2316363842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99349d0 0x555bd9934db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdcc17640 1 -- 172.21.15.89:0/2316363842 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bd984f130 con 0x555bd99352f0 2024-09-15T23:24:11.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdd418640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cc00a3240 0x7f1cc00a5700 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8005fd0 tx=0x7f1cc80093e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.783+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/2316363842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99352f0 0x555bd993f670 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc0b9080 tx=0x7f1ccc0b1b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21b841e709344bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1cbf7fe640 1 -- 172.21.15.89:0/2316363842 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc049020 con 0x555bd99352f0 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1cbf7fe640 1 -- 172.21.15.89:0/2316363842 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ccc03b580 con 0x555bd99352f0 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1cbf7fe640 1 -- 172.21.15.89:0/2316363842 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc0044e0 con 0x555bd99352f0 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2316363842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99352f0 msgr2=0x555bd993f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/2316363842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99352f0 0x555bd993f670 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc0b9080 tx=0x7f1ccc0b1b90 comp rx=0 tx=0).stop 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2316363842 shutdown_connections 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/2316363842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 0x555bd9941fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/2316363842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99352f0 0x555bd993f670 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/2316363842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99349d0 0x555bd9934db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2316363842 >> 172.21.15.89:0/2316363842 conn(0x555bd9930f60 msgr2=0x555bd9931340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2316363842 shutdown_connections 2024-09-15T23:24:11.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.784+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2316363842 wait complete. 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1ce1a55740 1 -- start start 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1ce1a55740 1 Processor -- start 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd993f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd99352f0 0x555bd993c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd993fbb0 0x555bd993ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555bd9942540 con 0x555bd99349d0 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555bd9855f20 con 0x555bd993fbb0 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555bd99426c0 con 0x555bd99352f0 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1ce1a55740 10 client.0 _ll_register_callbacks cb 0x555bd981f030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1cdd418640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd993f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1cdd418640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd993f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43626/0 (socket says 172.21.15.89:43626) 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1cdd418640 1 -- 172.21.15.89:0/3530730264 learned_addr learned my addr 172.21.15.89:0/3530730264 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:11.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.785+0000 7f1cddc19640 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd993fbb0 0x555bd993ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd99352f0 0x555bd993c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cdd418640 1 -- 172.21.15.89:0/3530730264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd99352f0 msgr2=0x555bd993c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cdd418640 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd99352f0 0x555bd993c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cdd418640 1 -- 172.21.15.89:0/3530730264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd993fbb0 msgr2=0x555bd993ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cdd418640 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd993fbb0 0x555bd993ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cdd418640 1 -- 172.21.15.89:0/3530730264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bd960c2b0 con 0x555bd99349d0 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cdd418640 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd993f3e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009760 tx=0x7f1cc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4cfa93d87a53187 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cbcff9640 1 -- 172.21.15.89:0/3530730264 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8022070 con 0x555bd99349d0 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1ce1a55740 1 -- 172.21.15.89:0/3530730264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555bd98a8600 con 0x555bd99349d0 2024-09-15T23:24:11.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cbcff9640 1 -- 172.21.15.89:0/3530730264 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1cc80108a0 con 0x555bd99349d0 2024-09-15T23:24:11.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cbcff9640 1 -- 172.21.15.89:0/3530730264 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1cc802e050 con 0x555bd99349d0 2024-09-15T23:24:11.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1ce1a55740 10 client.15258 fetch_fsmap learned FSMap version 28 2024-09-15T23:24:11.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1ce1a55740 1 -- 172.21.15.89:0/3530730264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555bd994ddf0 con 0x555bd99349d0 2024-09-15T23:24:11.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cbcff9640 1 -- 172.21.15.89:0/3530730264 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555bd98a8600 con 0x555bd99349d0 2024-09-15T23:24:11.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.786+0000 7f1cbcff9640 1 -- 172.21.15.89:0/3530730264 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f1cc80263b0 con 0x555bd99349d0 2024-09-15T23:24:11.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.787+0000 7f1ce1a55740 10 client.15258 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:24:11.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.787+0000 7f1ce1a55740 10 client.15258 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T23:24:11.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.787+0000 7f1ce1a55740 20 client.15258 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:11.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.787+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x555bd9950470 0x555bd99518e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.787+0000 7f1ce1a55740 4 client.15258 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:11.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.787+0000 7f1ce1a55740 1 -- 172.21.15.89:0/3530730264 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bd98a2b10 con 0x555bd9950470 2024-09-15T23:24:11.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.787+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x555bd9950470 0x555bd99518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.787+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x555bd9950470 0x555bd99518e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=965473d9abb470c3 server_cookie=82e4851146287bb8 in_seq=0 out_seq=0 2024-09-15T23:24:11.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.787+0000 7f1cbcff9640 10 client.15258 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:24:11.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1cbcff9640 1 -- 172.21.15.89:0/3530730264 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555bd98a2b10 con 0x555bd9950470 2024-09-15T23:24:11.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1cbcff9640 10 client.15258 handle_command_reply: tid=0 2024-09-15T23:24:11.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 client.15258 shutdown 2024-09-15T23:24:11.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 -- 172.21.15.89:0/3530730264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 msgr2=0x555bd993f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:11.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd993f3e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009760 tx=0x7f1cc8004a40 comp rx=0 tx=0).stop 2024-09-15T23:24:11.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 -- 172.21.15.89:0/3530730264 shutdown_connections 2024-09-15T23:24:11.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd993fbb0 0x555bd993ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x555bd9950470 0x555bd99518e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd99352f0 0x555bd993c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/3530730264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd993f3e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 -- 172.21.15.89:0/3530730264 >> 172.21.15.89:0/3530730264 conn(0x555bd9930f60 msgr2=0x555bd9936c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:11.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1cbcff9640 0 client.15258 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:24:11.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 -- 172.21.15.89:0/3530730264 shutdown_connections 2024-09-15T23:24:11.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 1 -- 172.21.15.89:0/3530730264 wait complete. 2024-09-15T23:24:11.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.788+0000 7f1ce1a55740 20 client.15258 trim_cache size 0 max 16384 2024-09-15T23:24:11.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1ce1a55740 1 Processor -- start 2024-09-15T23:24:11.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1ce1a55740 1 -- start start 2024-09-15T23:24:11.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd9950470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 0x555bd994b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 0x555bd9959bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555bd994fc20 con 0x555bd99349d0 2024-09-15T23:24:11.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555bd99502f0 con 0x555bd99352f0 2024-09-15T23:24:11.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555bd99509c0 con 0x555bd993fbb0 2024-09-15T23:24:11.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1cdd418640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd9950470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1cdd418640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd9950470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43634/0 (socket says 172.21.15.89:43634) 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1cdd418640 1 -- 172.21.15.89:0/1387727266 learned_addr learned my addr 172.21.15.89:0/1387727266 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 0x555bd994b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1cdd418640 1 -- 172.21.15.89:0/1387727266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 msgr2=0x555bd9959bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.806+0000 7f1cddc19640 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 0x555bd9959bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.807+0000 7f1cdd418640 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 0x555bd9959bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.807+0000 7f1cdd418640 1 -- 172.21.15.89:0/1387727266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 msgr2=0x555bd994b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.807+0000 7f1cdd418640 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 0x555bd994b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.807+0000 7f1cdd418640 1 -- 172.21.15.89:0/1387727266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bd9942300 con 0x555bd99349d0 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.807+0000 7f1cddc19640 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 0x555bd9959bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.807+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 0x555bd994b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:11.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.807+0000 7f1cdd418640 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd9950470 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009710 tx=0x7f1cc8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41ddfcefa2cf87e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.807+0000 7f1cbeffd640 1 -- 172.21.15.89:0/1387727266 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8036070 con 0x555bd99349d0 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.807+0000 7f1cbeffd640 1 -- 172.21.15.89:0/1387727266 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cc8003e50 con 0x555bd99349d0 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.807+0000 7f1cbeffd640 1 -- 172.21.15.89:0/1387727266 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8022050 con 0x555bd99349d0 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 -- 172.21.15.89:0/1387727266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 msgr2=0x555bd9950470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd9950470 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009710 tx=0x7f1cc8000f40 comp rx=0 tx=0).stop 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 -- 172.21.15.89:0/1387727266 shutdown_connections 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd993fbb0 0x555bd9959bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 0x555bd994b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/1387727266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd99349d0 0x555bd9950470 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 -- 172.21.15.89:0/1387727266 >> 172.21.15.89:0/1387727266 conn(0x555bd9930f60 msgr2=0x555bd993b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 -- 172.21.15.89:0/1387727266 shutdown_connections 2024-09-15T23:24:11.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 -- 172.21.15.89:0/1387727266 wait complete. 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 -- start start 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.808+0000 7f1ce1a55740 1 Processor -- start 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd99349d0 0x555bd9954970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 0x555bd9954eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1ce1a55740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd993fbb0 0x555bd99553f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555bd9951760 con 0x555bd993fbb0 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555bd9951090 con 0x555bd99352f0 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1ce1a55740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555bd9951e30 con 0x555bd99349d0 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1ce1a55740 10 client.0 _ll_register_callbacks cb 0x7f1cc8015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cddc19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd993fbb0 0x555bd99553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cddc19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd993fbb0 0x555bd99553f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43640/0 (socket says 172.21.15.89:43640) 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cddc19640 1 -- 172.21.15.89:0/2409247721 learned_addr learned my addr 172.21.15.89:0/2409247721 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:11.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cdd418640 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd99349d0 0x555bd9954970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 0x555bd9954eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cddc19640 1 -- 172.21.15.89:0/2409247721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd99349d0 msgr2=0x555bd9954970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cddc19640 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd99349d0 0x555bd9954970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cddc19640 1 -- 172.21.15.89:0/2409247721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 msgr2=0x555bd9954eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cddc19640 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 0x555bd9954eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cddc19640 1 -- 172.21.15.89:0/2409247721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bd99b8ba0 con 0x555bd993fbb0 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cdcc17640 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 0x555bd9954eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cddc19640 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd993fbb0 0x555bd99553f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0012a20 tx=0x7f1cd0012ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a078dd463c31947 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cbcff9640 1 -- 172.21.15.89:0/2409247721 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd000fa20 con 0x555bd993fbb0 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1cbcff9640 1 -- 172.21.15.89:0/2409247721 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1cd000fbc0 con 0x555bd993fbb0 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.809+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2409247721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cc8003e50 con 0x555bd993fbb0 2024-09-15T23:24:11.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1cbcff9640 1 -- 172.21.15.89:0/2409247721 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1cd0029050 con 0x555bd993fbb0 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1ce1a55740 10 client.15264 fetch_fsmap learned FSMap version 28 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2409247721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555bd99b9dd0 con 0x555bd993fbb0 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1cbcff9640 1 -- 172.21.15.89:0/2409247721 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cc8003e50 con 0x555bd993fbb0 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1cbcff9640 1 -- 172.21.15.89:0/2409247721 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f1cd0005380 con 0x555bd993fbb0 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1ce1a55740 10 client.15264 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1ce1a55740 10 client.15264 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1ce1a55740 20 client.15264 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x555bd99bea50 0x555bd99bd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1ce1a55740 4 client.15264 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2409247721 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "status"}) -- 0x555bd981f0e0 con 0x555bd99bea50 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.810+0000 7f1cdd418640 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x555bd99bea50 0x555bd99bd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1cbcff9640 10 client.15264 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1cdd418640 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x555bd99bea50 0x555bd99bd760 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1da6f32500afe01 server_cookie=71ad4ba489cec7e6 in_seq=0 out_seq=0 2024-09-15T23:24:11.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1cbcff9640 1 -- 172.21.15.89:0/2409247721 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x555bd981f0e0 con 0x555bd99bea50 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1cbcff9640 10 client.15264 handle_command_reply: tid=0 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 client.15264 shutdown 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2409247721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd993fbb0 msgr2=0x555bd99553f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd993fbb0 0x555bd99553f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0012a20 tx=0x7f1cd0012ee0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2409247721 shutdown_connections 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555bd993fbb0 0x555bd99553f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555bd99352f0 0x555bd9954eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x555bd99bea50 0x555bd99bd760 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 --2- 172.21.15.89:0/2409247721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555bd99349d0 0x555bd9954970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1cbcff9640 0 client.15264 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2409247721 >> 172.21.15.89:0/2409247721 conn(0x555bd9930f60 msgr2=0x555bd993b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2409247721 shutdown_connections 2024-09-15T23:24:11.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 1 -- 172.21.15.89:0/2409247721 wait complete. 2024-09-15T23:24:11.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.811+0000 7f1ce1a55740 20 client.15264 trim_cache size 0 max 16384 2024-09-15T23:24:11.823 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:24:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 2, 2024-09-15T23:24:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24430, 2024-09-15T23:24:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:24:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:24:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:24:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 97.716971981, 2024-09-15T23:24:11.823 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 28, 2024-09-15T23:24:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 90, 2024-09-15T23:24:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:24:11.824 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 163.05049192600001 2024-09-15T23:24:11.824 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:11.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 -- 172.21.15.89:0/950822800 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cc00a3240 msgr2=0x7f1cc00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cc00a3240 0x7f1cc00a5700 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8005fd0 tx=0x7f1cc80093e0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 -- 172.21.15.89:0/950822800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd81027a0 msgr2=0x7f1cd8198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd81027a0 0x7f1cd8198940 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc006010 tx=0x7f1ccc0042f0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 -- 172.21.15.89:0/950822800 shutdown_connections 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cc00a3240 0x7f1cc00a5700 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cd8106f20 0x7f1cd819ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1cd81027a0 0x7f1cd8198940 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 --2- 172.21.15.89:0/950822800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd8101df0 0x7f1cd8198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 -- 172.21.15.89:0/950822800 >> 172.21.15.89:0/950822800 conn(0x7f1cd80fd8f0 msgr2=0x7f1cd80fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 -- 172.21.15.89:0/950822800 shutdown_connections 2024-09-15T23:24:11.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:11.812+0000 7f1cdf67c640 1 -- 172.21.15.89:0/950822800 wait complete. 2024-09-15T23:24:12.551 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-15T23:24:12.797 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:13.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.280+0000 7f0c6408b640 1 -- 172.21.15.89:0/3344817542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c5c100e20 msgr2=0x7f0c5c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 --2- 172.21.15.89:0/3344817542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c5c100e20 0x7f0c5c101200 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009900 tx=0x7f0c5002a480 comp rx=0 tx=0).stop 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 -- 172.21.15.89:0/3344817542 shutdown_connections 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 --2- 172.21.15.89:0/3344817542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c5c102ef0 0x7f0c5c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 --2- 172.21.15.89:0/3344817542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c5c102570 0x7f0c5c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 --2- 172.21.15.89:0/3344817542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c5c100e20 0x7f0c5c101200 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 -- 172.21.15.89:0/3344817542 >> 172.21.15.89:0/3344817542 conn(0x7f0c5c0fc7d0 msgr2=0x7f0c5c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 -- 172.21.15.89:0/3344817542 shutdown_connections 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 -- 172.21.15.89:0/3344817542 wait complete. 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 Processor -- start 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 -- start start 2024-09-15T23:24:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.281+0000 7f0c6408b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c5c100e20 0x7f0c5c0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c6408b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c5c102570 0x7f0c5c0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c6408b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c5c102ef0 0x7f0c5c0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c6408b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c110fc0 con 0x7f0c5c100e20 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c6408b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c110e40 con 0x7f0c5c102ef0 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c6408b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c111140 con 0x7f0c5c102570 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c61e27640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c5c100e20 0x7f0c5c0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c61e27640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c5c100e20 0x7f0c5c0fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43670/0 (socket says 172.21.15.89:43670) 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c61e27640 1 -- 172.21.15.89:0/4188572069 learned_addr learned my addr 172.21.15.89:0/4188572069 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c61626640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c5c102570 0x7f0c5c0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c62628640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c5c102ef0 0x7f0c5c0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c61e27640 1 -- 172.21.15.89:0/4188572069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c5c102570 msgr2=0x7f0c5c0fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c61e27640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c5c102570 0x7f0c5c0fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c61e27640 1 -- 172.21.15.89:0/4188572069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c5c102ef0 msgr2=0x7f0c5c0fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c61e27640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c5c102ef0 0x7f0c5c0fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c61e27640 1 -- 172.21.15.89:0/4188572069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c1a6ad0 con 0x7f0c5c100e20 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c62628640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c5c102ef0 0x7f0c5c0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c61e27640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c5c100e20 0x7f0c5c0fcc80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0c50006010 tx=0x7f0c50003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c4effd640 1 -- 172.21.15.89:0/4188572069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5003f070 con 0x7f0c5c100e20 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c4effd640 1 -- 172.21.15.89:0/4188572069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c50031c30 con 0x7f0c5c100e20 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c6408b640 1 -- 172.21.15.89:0/4188572069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5c1a6d60 con 0x7f0c5c100e20 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c4effd640 1 -- 172.21.15.89:0/4188572069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50035030 con 0x7f0c5c100e20 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.282+0000 7f0c6408b640 1 -- 172.21.15.89:0/4188572069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c5c1a71e0 con 0x7f0c5c100e20 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.285+0000 7f0c4effd640 1 -- 172.21.15.89:0/4188572069 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0c50042070 con 0x7f0c5c100e20 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.285+0000 7f0c66464740 1 Processor -- start 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.285+0000 7f0c66464740 1 -- start start 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c66464740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 0x564b2512ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c66464740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512e2f0 0x564b25138670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c66464740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b2513afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c66464740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b25094090 con 0x564b25138bb0 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c66464740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b2503ba70 con 0x564b2512d9d0 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c66464740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b25090050 con 0x564b2512e2f0 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c4effd640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0c480a3240 0x7f0c480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61626640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b2513afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61626640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b2513afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43674/0 (socket says 172.21.15.89:43674) 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61626640 1 -- 172.21.15.89:0/2115059177 learned_addr learned my addr 172.21.15.89:0/2115059177 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c62628640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0c480a3240 0x7f0c480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c4effd640 1 -- 172.21.15.89:0/4188572069 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0c500f05c0 con 0x7f0c5c100e20 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2115059177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512e2f0 0x564b25138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c62628640 1 --2- 172.21.15.89:0/2115059177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 0x564b2512ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61626640 1 -- 172.21.15.89:0/2115059177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512e2f0 msgr2=0x564b25138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61626640 1 --2- 172.21.15.89:0/2115059177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512e2f0 0x564b25138670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61626640 1 -- 172.21.15.89:0/2115059177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 msgr2=0x564b2512ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61626640 1 --2- 172.21.15.89:0/2115059177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 0x564b2512ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61626640 1 -- 172.21.15.89:0/2115059177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b25048130 con 0x564b25138bb0 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2115059177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512e2f0 0x564b25138670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c61626640 1 --2- 172.21.15.89:0/2115059177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b2513afa0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800b740 tx=0x7f0c5800bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3609ccf1a2b58d09 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c62628640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0c480a3240 0x7f0c480a5700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400e870 tx=0x7f0c540094d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.286+0000 7f0c47fff640 1 -- 172.21.15.89:0/2115059177 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c58003dd0 con 0x564b25138bb0 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c47fff640 1 -- 172.21.15.89:0/2115059177 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c58015070 con 0x564b25138bb0 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c47fff640 1 -- 172.21.15.89:0/2115059177 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c58004e00 con 0x564b25138bb0 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c66464740 1 -- 172.21.15.89:0/2115059177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 msgr2=0x564b2513afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c66464740 1 --2- 172.21.15.89:0/2115059177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b2513afa0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800b740 tx=0x7f0c5800bc00 comp rx=0 tx=0).stop 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c66464740 1 -- 172.21.15.89:0/2115059177 shutdown_connections 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c66464740 1 --2- 172.21.15.89:0/2115059177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b2513afa0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c66464740 1 --2- 172.21.15.89:0/2115059177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512e2f0 0x564b25138670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c66464740 1 --2- 172.21.15.89:0/2115059177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 0x564b2512ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c66464740 1 -- 172.21.15.89:0/2115059177 >> 172.21.15.89:0/2115059177 conn(0x564b25129f60 msgr2=0x564b2512a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c66464740 1 -- 172.21.15.89:0/2115059177 shutdown_connections 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.287+0000 7f0c66464740 1 -- 172.21.15.89:0/2115059177 wait complete. 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c66464740 1 -- start start 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c66464740 1 Processor -- start 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c66464740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 0x564b25138340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c66464740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512e2f0 0x564b25135690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c66464740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b25135bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c66464740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b2513b780 con 0x564b25138bb0 2024-09-15T23:24:13.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c66464740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b2504ef20 con 0x564b2512e2f0 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c66464740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b2513b900 con 0x564b2512d9d0 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c66464740 10 client.0 _ll_register_callbacks cb 0x564b25018030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c62628640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b25135bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c62628640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b25135bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43686/0 (socket says 172.21.15.89:43686) 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.288+0000 7f0c62628640 1 -- 172.21.15.89:0/2931347136 learned_addr learned my addr 172.21.15.89:0/2931347136 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c61626640 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512e2f0 0x564b25135690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 0x564b25138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c62628640 1 -- 172.21.15.89:0/2931347136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 msgr2=0x564b25138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c62628640 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 0x564b25138340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c62628640 1 -- 172.21.15.89:0/2931347136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512e2f0 msgr2=0x564b25135690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c62628640 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512e2f0 0x564b25135690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c62628640 1 -- 172.21.15.89:0/2931347136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b24e052b0 con 0x564b25138bb0 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 0x564b25138340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c62628640 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b25135bd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400cb10 tx=0x7f0c54004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a54d16dc3b092ee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c457fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c457fa640 1 -- 172.21.15.89:0/2931347136 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c54023070 con 0x564b25138bb0 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c457fa640 1 -- 172.21.15.89:0/2931347136 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0c5400dd70 con 0x564b25138bb0 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c66464740 1 -- 172.21.15.89:0/2931347136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c58015070 con 0x564b25138bb0 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c457fa640 1 -- 172.21.15.89:0/2931347136 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0c5402f050 con 0x564b25138bb0 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c66464740 10 client.15276 fetch_fsmap learned FSMap version 28 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c66464740 1 -- 172.21.15.89:0/2931347136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564b25147000 con 0x564b25138bb0 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.289+0000 7f0c457fa640 1 -- 172.21.15.89:0/2931347136 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c58015070 con 0x564b25138bb0 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.290+0000 7f0c457fa640 1 -- 172.21.15.89:0/2931347136 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f0c5401a050 con 0x564b25138bb0 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.290+0000 7f0c66464740 10 client.15276 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.290+0000 7f0c66464740 10 client.15276 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:24:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.290+0000 7f0c66464740 20 client.15276 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.290+0000 7f0c66464740 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x564b25149660 0x564b2514aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.290+0000 7f0c66464740 4 client.15276 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.290+0000 7f0c66464740 1 -- 172.21.15.89:0/2931347136 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b250a1600 con 0x564b25149660 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.290+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x564b25149660 0x564b2514aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.290+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x564b25149660 0x564b2514aad0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f6bee069001277 server_cookie=941fb5670b035a7 in_seq=0 out_seq=0 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.290+0000 7f0c457fa640 10 client.15276 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c457fa640 1 -- 172.21.15.89:0/2931347136 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564b250a1600 con 0x564b25149660 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c457fa640 10 client.15276 handle_command_reply: tid=0 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c66464740 1 client.15276 shutdown 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c66464740 1 -- 172.21.15.89:0/2931347136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 msgr2=0x564b25135bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c66464740 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b25135bd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400cb10 tx=0x7f0c54004b50 comp rx=0 tx=0).stop 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c66464740 1 -- 172.21.15.89:0/2931347136 shutdown_connections 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c66464740 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x564b25149660 0x564b2514aad0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c66464740 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b25138bb0 0x564b25135bd0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c66464740 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512e2f0 0x564b25135690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c457fa640 0 client.15276 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:24:13.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c66464740 1 --2- 172.21.15.89:0/2931347136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 0x564b25138340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.291+0000 7f0c66464740 1 -- 172.21.15.89:0/2931347136 >> 172.21.15.89:0/2931347136 conn(0x564b25129f60 msgr2=0x564b2512fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:13.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.292+0000 7f0c66464740 1 -- 172.21.15.89:0/2931347136 shutdown_connections 2024-09-15T23:24:13.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.292+0000 7f0c66464740 1 -- 172.21.15.89:0/2931347136 wait complete. 2024-09-15T23:24:13.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.292+0000 7f0c66464740 20 client.15276 trim_cache size 0 max 16384 2024-09-15T23:24:13.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.309+0000 7f0c66464740 1 Processor -- start 2024-09-15T23:24:13.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.309+0000 7f0c66464740 1 -- start start 2024-09-15T23:24:13.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c66464740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 0x564b25135650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c66464740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b25135b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c66464740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b25138bb0 0x564b25133f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c66464740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b25148e10 con 0x564b2512e2f0 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c66464740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b251494e0 con 0x564b25138bb0 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c66464740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b25149bb0 con 0x564b2512d9d0 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c61626640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b25135b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c61626640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b25135b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43700/0 (socket says 172.21.15.89:43700) 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c61626640 1 -- 172.21.15.89:0/2750188681 learned_addr learned my addr 172.21.15.89:0/2750188681 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2750188681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 0x564b25135650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c62628640 1 --2- 172.21.15.89:0/2750188681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b25138bb0 0x564b25133f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c61626640 1 -- 172.21.15.89:0/2750188681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 msgr2=0x564b25135650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c61626640 1 --2- 172.21.15.89:0/2750188681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 0x564b25135650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c61626640 1 -- 172.21.15.89:0/2750188681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b25138bb0 msgr2=0x564b25133f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c61626640 1 --2- 172.21.15.89:0/2750188681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b25138bb0 0x564b25133f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c61626640 1 -- 172.21.15.89:0/2750188681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b251386b0 con 0x564b2512e2f0 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c61626640 1 --2- 172.21.15.89:0/2750188681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b25135b90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x564b2512b2e0 tx=0x7f0c5800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c50a461339210b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c477fe640 1 -- 172.21.15.89:0/2750188681 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c58003bd0 con 0x564b2512e2f0 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c477fe640 1 -- 172.21.15.89:0/2750188681 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c58029070 con 0x564b2512e2f0 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.310+0000 7f0c477fe640 1 -- 172.21.15.89:0/2750188681 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c58011440 con 0x564b2512e2f0 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 -- 172.21.15.89:0/2750188681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 msgr2=0x564b25135b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 --2- 172.21.15.89:0/2750188681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b25135b90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x564b2512b2e0 tx=0x7f0c5800be50 comp rx=0 tx=0).stop 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 -- 172.21.15.89:0/2750188681 shutdown_connections 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 --2- 172.21.15.89:0/2750188681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b25138bb0 0x564b25133f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 --2- 172.21.15.89:0/2750188681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b25135b90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 --2- 172.21.15.89:0/2750188681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2512d9d0 0x564b25135650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 -- 172.21.15.89:0/2750188681 >> 172.21.15.89:0/2750188681 conn(0x564b25129f60 msgr2=0x564b2512a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 -- 172.21.15.89:0/2750188681 shutdown_connections 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 -- 172.21.15.89:0/2750188681 wait complete. 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 -- start start 2024-09-15T23:24:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.311+0000 7f0c66464740 1 Processor -- start 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c66464740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 0x564b2514bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c66464740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b2514c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c66464740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b25138bb0 0x564b2514c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c66464740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b2514a950 con 0x564b2512e2f0 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c66464740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b2514a280 con 0x564b2512d9d0 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c66464740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b2514b020 con 0x564b25138bb0 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c66464740 10 client.0 _ll_register_callbacks cb 0x7f0c54009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c61626640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b2514c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c61626640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b2514c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43704/0 (socket says 172.21.15.89:43704) 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c61626640 1 -- 172.21.15.89:0/2346082759 learned_addr learned my addr 172.21.15.89:0/2346082759 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 0x564b2514bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c62628640 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b25138bb0 0x564b2514c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c61626640 1 -- 172.21.15.89:0/2346082759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b25138bb0 msgr2=0x564b2514c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c61626640 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b25138bb0 0x564b2514c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c61626640 1 -- 172.21.15.89:0/2346082759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 msgr2=0x564b2514bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c61626640 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 0x564b2514bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c61626640 1 -- 172.21.15.89:0/2346082759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b251b2000 con 0x564b2512e2f0 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 0x564b2514bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.312+0000 7f0c62628640 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b25138bb0 0x564b2514c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c61626640 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b2514c300 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0c58005d00 tx=0x7f0c58004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c79e737dc98fe3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c457fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c457fa640 1 -- 172.21.15.89:0/2346082759 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c58024470 con 0x564b2512e2f0 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c457fa640 1 -- 172.21.15.89:0/2346082759 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0c58024610 con 0x564b2512e2f0 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c66464740 1 -- 172.21.15.89:0/2346082759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c58029070 con 0x564b2512e2f0 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c457fa640 1 -- 172.21.15.89:0/2346082759 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0c5803c050 con 0x564b2512e2f0 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c66464740 10 client.15282 fetch_fsmap learned FSMap version 28 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c66464740 1 -- 172.21.15.89:0/2346082759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564b251b32e0 con 0x564b2512e2f0 2024-09-15T23:24:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c457fa640 1 -- 172.21.15.89:0/2346082759 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c58029070 con 0x564b2512e2f0 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c457fa640 1 -- 172.21.15.89:0/2346082759 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f0c58011440 con 0x564b2512e2f0 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c66464740 10 client.15282 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c66464740 10 client.15282 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c66464740 20 client.15282 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.313+0000 7f0c66464740 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x564b251b7780 0x564b251b6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.314+0000 7f0c66464740 4 client.15282 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.314+0000 7f0c66464740 1 -- 172.21.15.89:0/2346082759 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "status"}) -- 0x564b2509bb10 con 0x564b251b7780 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.314+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x564b251b7780 0x564b251b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.314+0000 7f0c61e27640 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x564b251b7780 0x564b251b6490 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5df7a4a562d7af53 server_cookie=1ac2793b4040dacd in_seq=0 out_seq=0 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.314+0000 7f0c457fa640 10 client.15282 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.314+0000 7f0c457fa640 1 -- 172.21.15.89:0/2346082759 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x564b2509bb10 con 0x564b251b7780 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.314+0000 7f0c457fa640 10 client.15282 handle_command_reply: tid=0 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.314+0000 7f0c66464740 1 client.15282 shutdown 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.314+0000 7f0c66464740 1 -- 172.21.15.89:0/2346082759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 msgr2=0x564b2514c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.314+0000 7f0c66464740 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b2514c300 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0c58005d00 tx=0x7f0c58004310 comp rx=0 tx=0).stop 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c66464740 1 -- 172.21.15.89:0/2346082759 shutdown_connections 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c66464740 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x564b251b7780 0x564b251b6490 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c66464740 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b25138bb0 0x564b2514c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c66464740 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2512e2f0 0x564b2514c300 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c457fa640 0 client.15282 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c66464740 1 --2- 172.21.15.89:0/2346082759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2512d9d0 0x564b2514bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c66464740 1 -- 172.21.15.89:0/2346082759 >> 172.21.15.89:0/2346082759 conn(0x564b25129f60 msgr2=0x564b25135650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c66464740 1 -- 172.21.15.89:0/2346082759 shutdown_connections 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c66464740 1 -- 172.21.15.89:0/2346082759 wait complete. 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c66464740 20 client.15282 trim_cache size 0 max 16384 2024-09-15T23:24:13.322 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 1, 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24436, 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 100.22672661, 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 28, 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 90, 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 159.332728081 2024-09-15T23:24:13.323 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:13.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 -- 172.21.15.89:0/4188572069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0c480a3240 msgr2=0x7f0c480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:13.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0c480a3240 0x7f0c480a5700 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400e870 tx=0x7f0c540094d0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 -- 172.21.15.89:0/4188572069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c5c100e20 msgr2=0x7f0c5c0fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:13.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c5c100e20 0x7f0c5c0fcc80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0c50006010 tx=0x7f0c50003c10 comp rx=0 tx=0).stop 2024-09-15T23:24:13.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 -- 172.21.15.89:0/4188572069 shutdown_connections 2024-09-15T23:24:13.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0c480a3240 0x7f0c480a5700 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c5c102ef0 0x7f0c5c0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c5c102570 0x7f0c5c0fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 --2- 172.21.15.89:0/4188572069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c5c100e20 0x7f0c5c0fcc80 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:13.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 -- 172.21.15.89:0/4188572069 >> 172.21.15.89:0/4188572069 conn(0x7f0c5c0fc7d0 msgr2=0x7f0c5c06a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:13.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 -- 172.21.15.89:0/4188572069 shutdown_connections 2024-09-15T23:24:13.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:13.315+0000 7f0c6408b640 1 -- 172.21.15.89:0/4188572069 wait complete. 2024-09-15T23:24:14.039 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T23:24:14.291 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:14.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:14 smithi089 ceph-mon[30933]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 102 B/s wr, 9 op/s; 6.2 KiB/s, 1 objects/s recovering 2024-09-15T23:24:14.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:14 smithi120 ceph-mon[35250]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 102 B/s wr, 9 op/s; 6.2 KiB/s, 1 objects/s recovering 2024-09-15T23:24:14.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:14 smithi174 ceph-mon[35586]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 102 B/s wr, 9 op/s; 6.2 KiB/s, 1 objects/s recovering 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.797+0000 7fd96a5c1640 1 -- 172.21.15.89:0/3714687423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd964106f20 msgr2=0x7fd96410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.797+0000 7fd96a5c1640 1 --2- 172.21.15.89:0/3714687423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd964106f20 0x7fd96410eaa0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009ca0 tx=0x7fd95c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.797+0000 7fd96a5c1640 1 -- 172.21.15.89:0/3714687423 shutdown_connections 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.797+0000 7fd96a5c1640 1 --2- 172.21.15.89:0/3714687423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd964106f20 0x7fd96410eaa0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.797+0000 7fd96a5c1640 1 --2- 172.21.15.89:0/3714687423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9641027a0 0x7fd9641067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.797+0000 7fd96a5c1640 1 --2- 172.21.15.89:0/3714687423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd964101df0 0x7fd9641021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.797+0000 7fd96a5c1640 1 -- 172.21.15.89:0/3714687423 >> 172.21.15.89:0/3714687423 conn(0x7fd9640fd8b0 msgr2=0x7fd9640ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.797+0000 7fd96a5c1640 1 -- 172.21.15.89:0/3714687423 shutdown_connections 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.797+0000 7fd96a5c1640 1 -- 172.21.15.89:0/3714687423 wait complete. 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd96a5c1640 1 Processor -- start 2024-09-15T23:24:14.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd96a5c1640 1 -- start start 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd96a5c1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd964101df0 0x7fd964198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd96a5c1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9641027a0 0x7fd964198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd96a5c1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd964106f20 0x7fd96419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd96a5c1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd964111f80 con 0x7fd964101df0 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd96a5c1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd964111e00 con 0x7fd964106f20 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd96a5c1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd964112100 con 0x7fd9641027a0 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd963fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd964101df0 0x7fd964198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd963fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd964101df0 0x7fd964198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43720/0 (socket says 172.21.15.89:43720) 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.798+0000 7fd963fff640 1 -- 172.21.15.89:0/187542749 learned_addr learned my addr 172.21.15.89:0/187542749 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd963fff640 1 -- 172.21.15.89:0/187542749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9641027a0 msgr2=0x7fd964198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd963fff640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9641027a0 0x7fd964198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd963fff640 1 -- 172.21.15.89:0/187542749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd964106f20 msgr2=0x7fd96419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd963fff640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd964106f20 0x7fd96419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd963fff640 1 -- 172.21.15.89:0/187542749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96419d2b0 con 0x7fd964101df0 2024-09-15T23:24:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd963fff640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd964101df0 0x7fd964198300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd95400c920 tx=0x7fd95400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd9617fa640 1 -- 172.21.15.89:0/187542749 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954007c70 con 0x7fd964101df0 2024-09-15T23:24:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd9617fa640 1 -- 172.21.15.89:0/187542749 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd954007e10 con 0x7fd964101df0 2024-09-15T23:24:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd96a5c1640 1 -- 172.21.15.89:0/187542749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96419d5a0 con 0x7fd964101df0 2024-09-15T23:24:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd9617fa640 1 -- 172.21.15.89:0/187542749 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954018890 con 0x7fd964101df0 2024-09-15T23:24:14.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.799+0000 7fd96a5c1640 1 -- 172.21.15.89:0/187542749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9641a4e80 con 0x7fd964101df0 2024-09-15T23:24:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.801+0000 7fd9617fa640 1 -- 172.21.15.89:0/187542749 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd954018a70 con 0x7fd964101df0 2024-09-15T23:24:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd96c99a740 1 Processor -- start 2024-09-15T23:24:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd96c99a740 1 -- start start 2024-09-15T23:24:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 0x557a22f69db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f74670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f76fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd96c99a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557a22ed0090 con 0x557a22f6a2f0 2024-09-15T23:24:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd96c99a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557a22e77a70 con 0x557a22f74bb0 2024-09-15T23:24:14.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd96c99a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557a22ecc050 con 0x557a22f699d0 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd963fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f74670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd963fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f74670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43724/0 (socket says 172.21.15.89:43724) 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd963fff640 1 -- 172.21.15.89:0/813027850 learned_addr learned my addr 172.21.15.89:0/813027850 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd9637fe640 1 --2- 172.21.15.89:0/813027850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f76fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd9617fa640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd94c0a3240 0x7fd94c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd968b5e640 1 --2- 172.21.15.89:0/813027850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 0x557a22f69db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd963fff640 1 -- 172.21.15.89:0/813027850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 msgr2=0x557a22f69db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd963fff640 1 --2- 172.21.15.89:0/813027850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 0x557a22f69db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd963fff640 1 -- 172.21.15.89:0/813027850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 msgr2=0x557a22f76fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd963fff640 1 --2- 172.21.15.89:0/813027850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f76fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd963fff640 1 -- 172.21.15.89:0/813027850 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a22e84130 con 0x557a22f6a2f0 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd968b5e640 1 --2- 172.21.15.89:0/813027850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 0x557a22f69db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.802+0000 7fd968b5e640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd94c0a3240 0x7fd94c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.803+0000 7fd9617fa640 1 -- 172.21.15.89:0/187542749 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd9540cff00 con 0x7fd964101df0 2024-09-15T23:24:14.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.803+0000 7fd963fff640 1 --2- 172.21.15.89:0/813027850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f74670 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd95409be30 tx=0x7fd954014400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dedfb19fe491ad5a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:14.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.803+0000 7fd94a7fc640 1 -- 172.21.15.89:0/813027850 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9540d36f0 con 0x557a22f6a2f0 2024-09-15T23:24:14.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.803+0000 7fd94a7fc640 1 -- 172.21.15.89:0/813027850 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd954066fd0 con 0x557a22f6a2f0 2024-09-15T23:24:14.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.803+0000 7fd94a7fc640 1 -- 172.21.15.89:0/813027850 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9540a4da0 con 0x557a22f6a2f0 2024-09-15T23:24:14.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.803+0000 7fd968b5e640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd94c0a3240 0x7fd94c0a5700 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd95c02f540 tx=0x7fd95c040040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:14.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 -- 172.21.15.89:0/813027850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 msgr2=0x557a22f74670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:14.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 --2- 172.21.15.89:0/813027850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f74670 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd95409be30 tx=0x7fd954014400 comp rx=0 tx=0).stop 2024-09-15T23:24:14.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 -- 172.21.15.89:0/813027850 shutdown_connections 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 --2- 172.21.15.89:0/813027850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f76fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 --2- 172.21.15.89:0/813027850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f74670 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 --2- 172.21.15.89:0/813027850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 0x557a22f69db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 -- 172.21.15.89:0/813027850 >> 172.21.15.89:0/813027850 conn(0x557a22f65f60 msgr2=0x557a22f66340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 -- 172.21.15.89:0/813027850 shutdown_connections 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 -- 172.21.15.89:0/813027850 wait complete. 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 -- start start 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.804+0000 7fd96c99a740 1 Processor -- start 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f74360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 0x557a22f716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f71bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd96c99a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557a22f77720 con 0x557a22f699d0 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd96c99a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557a22e8af20 con 0x557a22f74bb0 2024-09-15T23:24:14.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd96c99a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557a22f778a0 con 0x557a22f6a2f0 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd96c99a740 10 client.0 _ll_register_callbacks cb 0x557a22e54030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd963fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f74360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd963fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f74360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43738/0 (socket says 172.21.15.89:43738) 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd963fff640 1 -- 172.21.15.89:0/565964798 learned_addr learned my addr 172.21.15.89:0/565964798 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd9637fe640 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 0x557a22f716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd968b5e640 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f71bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd963fff640 1 -- 172.21.15.89:0/565964798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 msgr2=0x557a22f716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd963fff640 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 0x557a22f716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd963fff640 1 -- 172.21.15.89:0/565964798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 msgr2=0x557a22f71bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd963fff640 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f71bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd963fff640 1 -- 172.21.15.89:0/565964798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a22c412b0 con 0x557a22f699d0 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd9637fe640 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 0x557a22f716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:14.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd963fff640 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f74360 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd9540cd660 tx=0x7fd954000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b4292d79288144 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd9437fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd9437fe640 1 -- 172.21.15.89:0/565964798 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95409c830 con 0x557a22f699d0 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd9437fe640 1 -- 172.21.15.89:0/565964798 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd95406d0d0 con 0x557a22f699d0 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.805+0000 7fd96c99a740 1 -- 172.21.15.89:0/565964798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd954066fd0 con 0x557a22f699d0 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd9437fe640 1 -- 172.21.15.89:0/565964798 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd95400de20 con 0x557a22f699d0 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd96c99a740 10 client.15294 fetch_fsmap learned FSMap version 28 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd96c99a740 1 -- 172.21.15.89:0/565964798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557a22f82f60 con 0x557a22f699d0 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd9437fe640 1 -- 172.21.15.89:0/565964798 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd954066fd0 con 0x557a22f699d0 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd9437fe640 1 -- 172.21.15.89:0/565964798 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd954078ce0 con 0x557a22f699d0 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd96c99a740 10 client.15294 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd96c99a740 10 client.15294 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd96c99a740 20 client.15294 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd96c99a740 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557a22f851b0 0x557a22f86620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd96c99a740 4 client.15294 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd96c99a740 1 -- 172.21.15.89:0/565964798 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a22edd600 con 0x557a22f851b0 2024-09-15T23:24:14.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.806+0000 7fd9637fe640 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557a22f851b0 0x557a22f86620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.807+0000 7fd9637fe640 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557a22f851b0 0x557a22f86620 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=585e16b1b0d106b2 server_cookie=9fca561d3a906c7a in_seq=0 out_seq=0 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.807+0000 7fd9437fe640 10 client.15294 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.807+0000 7fd9437fe640 1 -- 172.21.15.89:0/565964798 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557a22edd600 con 0x557a22f851b0 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.807+0000 7fd9437fe640 10 client.15294 handle_command_reply: tid=0 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.807+0000 7fd96c99a740 1 client.15294 shutdown 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 1 -- 172.21.15.89:0/565964798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 msgr2=0x557a22f74360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f74360 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd9540cd660 tx=0x7fd954000bc0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 1 -- 172.21.15.89:0/565964798 shutdown_connections 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557a22f851b0 0x557a22f86620 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f71bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 0x557a22f716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd9437fe640 0 client.15294 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 1 --2- 172.21.15.89:0/565964798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f74360 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 1 -- 172.21.15.89:0/565964798 >> 172.21.15.89:0/565964798 conn(0x557a22f65f60 msgr2=0x557a22f6bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:14.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 1 -- 172.21.15.89:0/565964798 shutdown_connections 2024-09-15T23:24:14.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 1 -- 172.21.15.89:0/565964798 wait complete. 2024-09-15T23:24:14.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.808+0000 7fd96c99a740 20 client.15294 trim_cache size 0 max 16384 2024-09-15T23:24:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd96c99a740 1 Processor -- start 2024-09-15T23:24:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd96c99a740 1 -- start start 2024-09-15T23:24:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 0x557a22f86620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f8e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f8ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd96c99a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557a22f84960 con 0x557a22f6a2f0 2024-09-15T23:24:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd96c99a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557a22f85030 con 0x557a22f74bb0 2024-09-15T23:24:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd96c99a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557a22f85700 con 0x557a22f699d0 2024-09-15T23:24:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd9637fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f8e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd9637fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f8e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43754/0 (socket says 172.21.15.89:43754) 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.825+0000 7fd9637fe640 1 -- 172.21.15.89:0/4220583600 learned_addr learned my addr 172.21.15.89:0/4220583600 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd968b5e640 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f8ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd963fff640 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 0x557a22f86620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd9637fe640 1 -- 172.21.15.89:0/4220583600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 msgr2=0x557a22f86620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd9637fe640 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 0x557a22f86620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd9637fe640 1 -- 172.21.15.89:0/4220583600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 msgr2=0x557a22f8ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd9637fe640 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f8ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd9637fe640 1 -- 172.21.15.89:0/4220583600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a22f65d90 con 0x557a22f6a2f0 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd963fff640 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 0x557a22f86620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd968b5e640 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f8ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd9637fe640 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f8e900 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x557a22f672e0 tx=0x7fd95800a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff28310c52bb6afa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd949ffb640 1 -- 172.21.15.89:0/4220583600 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9580130d0 con 0x557a22f6a2f0 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd949ffb640 1 -- 172.21.15.89:0/4220583600 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd958013270 con 0x557a22f6a2f0 2024-09-15T23:24:14.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.826+0000 7fd949ffb640 1 -- 172.21.15.89:0/4220583600 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95801a330 con 0x557a22f6a2f0 2024-09-15T23:24:14.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 -- 172.21.15.89:0/4220583600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 msgr2=0x557a22f8e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:14.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f8e900 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x557a22f672e0 tx=0x7fd95800a250 comp rx=0 tx=0).stop 2024-09-15T23:24:14.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 -- 172.21.15.89:0/4220583600 shutdown_connections 2024-09-15T23:24:14.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f8ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f6a2f0 0x557a22f8e900 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 --2- 172.21.15.89:0/4220583600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f699d0 0x557a22f86620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 -- 172.21.15.89:0/4220583600 >> 172.21.15.89:0/4220583600 conn(0x557a22f65f60 msgr2=0x557a22f73620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:14.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 -- 172.21.15.89:0/4220583600 shutdown_connections 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 -- 172.21.15.89:0/4220583600 wait complete. 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 -- start start 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.827+0000 7fd96c99a740 1 Processor -- start 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f8ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 0x557a22f8a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd96c99a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f8a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd96c99a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557a22f864a0 con 0x557a22f699d0 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd96c99a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557a22f85dd0 con 0x557a22f74bb0 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd96c99a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557a22f86b70 con 0x557a22f6a2f0 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd96c99a740 10 client.0 _ll_register_callbacks cb 0x7fd958005ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd963fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f8ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd963fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f8ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43768/0 (socket says 172.21.15.89:43768) 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd963fff640 1 -- 172.21.15.89:0/3425222266 learned_addr learned my addr 172.21.15.89:0/3425222266 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:14.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd9637fe640 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 0x557a22f8a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd968b5e640 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f8a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd963fff640 1 -- 172.21.15.89:0/3425222266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 msgr2=0x557a22f8a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd963fff640 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 0x557a22f8a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd963fff640 1 -- 172.21.15.89:0/3425222266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 msgr2=0x557a22f8a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd963fff640 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f8a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd963fff640 1 -- 172.21.15.89:0/3425222266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a22f69800 con 0x557a22f699d0 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd968b5e640 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f8a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd9637fe640 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 0x557a22f8a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd963fff640 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f8ef20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd95400c8f0 tx=0x7fd95400a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77985120838cd7d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd9437fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd9437fe640 1 -- 172.21.15.89:0/3425222266 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9540983d0 con 0x557a22f699d0 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd9437fe640 1 -- 172.21.15.89:0/3425222266 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd954082380 con 0x557a22f699d0 2024-09-15T23:24:14.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd96c99a740 1 -- 172.21.15.89:0/3425222266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd958013270 con 0x557a22f699d0 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd9437fe640 1 -- 172.21.15.89:0/3425222266 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd95409c830 con 0x557a22f699d0 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd96c99a740 10 client.15300 fetch_fsmap learned FSMap version 28 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.828+0000 7fd96c99a740 1 -- 172.21.15.89:0/3425222266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557a22fef540 con 0x557a22f699d0 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd9437fe640 1 -- 172.21.15.89:0/3425222266 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd958013270 con 0x557a22f699d0 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd9437fe640 1 -- 172.21.15.89:0/3425222266 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd95409c390 con 0x557a22f699d0 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd96c99a740 10 client.15300 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd96c99a740 10 client.15300 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd96c99a740 20 client.15300 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd96c99a740 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557a22ff3ed0 0x557a22ff2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd96c99a740 4 client.15300 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd96c99a740 1 -- 172.21.15.89:0/3425222266 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "status"}) -- 0x557a22ed7b10 con 0x557a22ff3ed0 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd9637fe640 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557a22ff3ed0 0x557a22ff2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd9437fe640 10 client.15300 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.829+0000 7fd9637fe640 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557a22ff3ed0 0x557a22ff2be0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5eb3f912f87c203b server_cookie=ceb53999fcdfd909 in_seq=0 out_seq=0 2024-09-15T23:24:14.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd9437fe640 1 -- 172.21.15.89:0/3425222266 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557a22ed7b10 con 0x557a22ff3ed0 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd9437fe640 10 client.15300 handle_command_reply: tid=0 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 client.15300 shutdown 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 -- 172.21.15.89:0/3425222266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 msgr2=0x557a22f8ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f8ef20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd95400c8f0 tx=0x7fd95400a410 comp rx=0 tx=0).stop 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 -- 172.21.15.89:0/3425222266 shutdown_connections 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557a22ff3ed0 0x557a22ff2be0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557a22f74bb0 0x557a22f8a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557a22f6a2f0 0x557a22f8a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 --2- 172.21.15.89:0/3425222266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557a22f699d0 0x557a22f8ef20 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd9437fe640 0 client.15300 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 -- 172.21.15.89:0/3425222266 >> 172.21.15.89:0/3425222266 conn(0x557a22f65f60 msgr2=0x557a22f7fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 -- 172.21.15.89:0/3425222266 shutdown_connections 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 1 -- 172.21.15.89:0/3425222266 wait complete. 2024-09-15T23:24:14.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.830+0000 7fd96c99a740 20 client.15300 trim_cache size 0 max 16384 2024-09-15T23:24:14.837 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:14.840 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:24:14.840 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-09-15T23:24:14.840 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24452, 2024-09-15T23:24:14.840 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:24:14.840 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:24:14.840 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:24:14.840 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 141.31046210599999, 2024-09-15T23:24:14.840 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 28, 2024-09-15T23:24:14.841 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 90, 2024-09-15T23:24:14.841 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 86, 2024-09-15T23:24:14.841 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 151.837384719 2024-09-15T23:24:14.841 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:14.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 -- 172.21.15.89:0/187542749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd94c0a3240 msgr2=0x7fd94c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:14.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd94c0a3240 0x7fd94c0a5700 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd95c02f540 tx=0x7fd95c040040 comp rx=0 tx=0).stop 2024-09-15T23:24:14.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 -- 172.21.15.89:0/187542749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd964101df0 msgr2=0x7fd964198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd964101df0 0x7fd964198300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd95400c920 tx=0x7fd95400cde0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 -- 172.21.15.89:0/187542749 shutdown_connections 2024-09-15T23:24:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd94c0a3240 0x7fd94c0a5700 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd964106f20 0x7fd96419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9641027a0 0x7fd964198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 --2- 172.21.15.89:0/187542749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd964101df0 0x7fd964198300 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 -- 172.21.15.89:0/187542749 >> 172.21.15.89:0/187542749 conn(0x7fd9640fd8b0 msgr2=0x7fd9640fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 -- 172.21.15.89:0/187542749 shutdown_connections 2024-09-15T23:24:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:14.831+0000 7fd96a5c1640 1 -- 172.21.15.89:0/187542749 wait complete. 2024-09-15T23:24:15.569 INFO:teuthology.run_tasks:Running task sequential... 2024-09-15T23:24:15.579 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-15T23:24:15.579 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-15T23:24:15.579 INFO:teuthology.task.print:Enabling mgr modules 2024-09-15T23:24:15.579 INFO:teuthology.run_tasks:Running task kclient... 2024-09-15T23:24:15.602 INFO:tasks.kclient:Mounting kernel clients... 2024-09-15T23:24:15.603 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-15T23:24:15.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:24:15.603 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-09-15T23:24:15.619 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-15T23:24:15.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:24:15.620 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:24:15.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:24:15.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2024-09-15T23:24:15.782 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-09-15T23:24:15.782 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-15T23:24:15.782 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-15T23:24:15.783 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-15T23:24:16.030 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:16.338 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:16 smithi089 ceph-mon[30933]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-09-15T23:24:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:16 smithi120 ceph-mon[35250]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-09-15T23:24:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:16 smithi174 ceph-mon[35586]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-09-15T23:24:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 -- 172.21.15.89:0/3803708609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368101410 msgr2=0x7fe36810d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 --2- 172.21.15.89:0/3803708609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368101410 0x7fe36810d940 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe360009cd0 tx=0x7fe36002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:24:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 -- 172.21.15.89:0/3803708609 shutdown_connections 2024-09-15T23:24:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 --2- 172.21.15.89:0/3803708609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368101410 0x7fe36810d940 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 --2- 172.21.15.89:0/3803708609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe368100a70 0x7fe368100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 --2- 172.21.15.89:0/3803708609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe368106a70 0x7fe368106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 -- 172.21.15.89:0/3803708609 >> 172.21.15.89:0/3803708609 conn(0x7fe3680fc790 msgr2=0x7fe3680febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:16.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 -- 172.21.15.89:0/3803708609 shutdown_connections 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 -- 172.21.15.89:0/3803708609 wait complete. 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 Processor -- start 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 -- start start 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.560+0000 7fe3703ee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe368100a70 0x7fe368199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe3703ee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368101410 0x7fe3681997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe3703ee640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe368106a70 0x7fe3681a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe3703ee640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe368110e70 con 0x7fe368101410 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe3703ee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe368110cf0 con 0x7fe368100a70 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe3703ee640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe368110ff0 con 0x7fe368106a70 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36d989640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368101410 0x7fe3681997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36d989640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368101410 0x7fe3681997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35644/0 (socket says 172.21.15.89:35644) 2024-09-15T23:24:16.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36d989640 1 -- 172.21.15.89:0/3710439693 learned_addr learned my addr 172.21.15.89:0/3710439693 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36e98b640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe368106a70 0x7fe3681a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36e18a640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe368100a70 0x7fe368199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36d989640 1 -- 172.21.15.89:0/3710439693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe368106a70 msgr2=0x7fe3681a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36d989640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe368106a70 0x7fe3681a42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36d989640 1 -- 172.21.15.89:0/3710439693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe368100a70 msgr2=0x7fe368199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36d989640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe368100a70 0x7fe368199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36d989640 1 -- 172.21.15.89:0/3710439693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3681a4830 con 0x7fe368101410 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe36d989640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368101410 0x7fe3681997b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe36400d770 tx=0x7fe36400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe35b7fe640 1 -- 172.21.15.89:0/3710439693 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364017070 con 0x7fe368101410 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe3703ee640 1 -- 172.21.15.89:0/3710439693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3681a4a00 con 0x7fe368101410 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe35b7fe640 1 -- 172.21.15.89:0/3710439693 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe364005500 con 0x7fe368101410 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe35b7fe640 1 -- 172.21.15.89:0/3710439693 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364004e00 con 0x7fe368101410 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.561+0000 7fe3703ee640 1 -- 172.21.15.89:0/3710439693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3681a4f40 con 0x7fe368101410 2024-09-15T23:24:16.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.562+0000 7fe3703ee640 1 -- 172.21.15.89:0/3710439693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe368100ed0 con 0x7fe368101410 2024-09-15T23:24:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.564+0000 7fe35b7fe640 1 -- 172.21.15.89:0/3710439693 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fe364004fa0 con 0x7fe368101410 2024-09-15T23:24:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.564+0000 7fe35b7fe640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3540a3240 0x7fe3540a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.564+0000 7fe36e18a640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3540a3240 0x7fe3540a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.564+0000 7fe35b7fe640 1 -- 172.21.15.89:0/3710439693 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe3640cfb00 con 0x7fe368101410 2024-09-15T23:24:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.565+0000 7fe36e18a640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3540a3240 0x7fe3540a5700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe35c00bb60 tx=0x7fe35c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:16.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.567+0000 7fe35b7fe640 1 -- 172.21.15.89:0/3710439693 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe36409bde0 con 0x7fe368101410 2024-09-15T23:24:16.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.769+0000 7fe3703ee640 1 -- 172.21.15.89:0/3710439693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe36806adc0 con 0x7fe368101410 2024-09-15T23:24:16.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.770+0000 7fe35b7fe640 1 -- 172.21.15.89:0/3710439693 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7fe36806adc0 con 0x7fe368101410 2024-09-15T23:24:16.771 INFO:teuthology.orchestra.run.smithi089.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-15T23:24:16.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.772+0000 7fe3703ee640 1 -- 172.21.15.89:0/3710439693 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3540a3240 msgr2=0x7fe3540a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.772+0000 7fe3703ee640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3540a3240 0x7fe3540a5700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe35c00bb60 tx=0x7fe35c00b390 comp rx=0 tx=0).stop 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.772+0000 7fe3703ee640 1 -- 172.21.15.89:0/3710439693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368101410 msgr2=0x7fe3681997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.772+0000 7fe3703ee640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368101410 0x7fe3681997b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe36400d770 tx=0x7fe36400dc30 comp rx=0 tx=0).stop 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.772+0000 7fe3703ee640 1 -- 172.21.15.89:0/3710439693 shutdown_connections 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.772+0000 7fe3703ee640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3540a3240 0x7fe3540a5700 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.772+0000 7fe3703ee640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe368106a70 0x7fe3681a42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.772+0000 7fe3703ee640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe368101410 0x7fe3681997b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.772+0000 7fe3703ee640 1 --2- 172.21.15.89:0/3710439693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe368100a70 0x7fe368199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.772+0000 7fe3703ee640 1 -- 172.21.15.89:0/3710439693 >> 172.21.15.89:0/3710439693 conn(0x7fe3680fc790 msgr2=0x7fe3680fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.773+0000 7fe3703ee640 1 -- 172.21.15.89:0/3710439693 shutdown_connections 2024-09-15T23:24:16.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:16.773+0000 7fe3703ee640 1 -- 172.21.15.89:0/3710439693 wait complete. 2024-09-15T23:24:17.435 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:17 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3710439693' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:24:17.462 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-15T23:24:17.462 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-15T23:24:17.462 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi089.front.sepia.ceph.com 2024-09-15T23:24:17.462 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-15T23:24:17.462 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-15T23:24:17.462 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-15T23:24:17.462 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-15T23:24:17.462 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-15T23:24:17.462 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-15T23:24:17.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:24:17.463 DEBUG:teuthology.orchestra.run.smithi089:> ip addr 2024-09-15T23:24:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:17 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3710439693' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:24:17.479 INFO:teuthology.orchestra.run.smithi089.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-15T23:24:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-15T23:24:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: inet 127.0.0.1/8 scope host lo 2024-09-15T23:24:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:24:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 ::1/128 scope host 2024-09-15T23:24:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:24:17.479 INFO:teuthology.orchestra.run.smithi089.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:24:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c6 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:24:17.479 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f0 2024-09-15T23:24:17.479 INFO:teuthology.orchestra.run.smithi089.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:24:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:d8:a4:c7 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:24:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: altname enp7s0f1 2024-09-15T23:24:17.480 INFO:teuthology.orchestra.run.smithi089.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:24:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:00 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:24:17.480 INFO:teuthology.orchestra.run.smithi089.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-15T23:24:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: link/ether 0c:c4:7a:8f:d5:01 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:24:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: inet 172.21.15.89/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-15T23:24:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft 42192sec preferred_lft 42192sec 2024-09-15T23:24:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: inet6 fe80::ec4:7aff:fe8f:d501/64 scope link noprefixroute 2024-09-15T23:24:17.480 INFO:teuthology.orchestra.run.smithi089.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:24:17.480 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-15T23:24:17.481 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:24:17.481 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-09-15T23:24:17.481 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add name ceph-brx type bridge 2024-09-15T23:24:17.481 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr flush dev ceph-brx 2024-09-15T23:24:17.481 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set ceph-brx up 2024-09-15T23:24:17.481 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-15T23:24:17.481 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-09-15T23:24:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:17 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3710439693' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:24:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:24:17.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:24:17.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:24:17.619 DEBUG:teuthology.orchestra.run.smithi089:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-15T23:24:17.642 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-09-15T23:24:17.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:24:17.644 DEBUG:teuthology.orchestra.run.smithi089:> route 2024-09-15T23:24:17.700 INFO:teuthology.orchestra.run.smithi089.stdout:Kernel IP routing table 2024-09-15T23:24:17.700 INFO:teuthology.orchestra.run.smithi089.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-15T23:24:17.700 INFO:teuthology.orchestra.run.smithi089.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-15T23:24:17.701 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-15T23:24:17.701 INFO:teuthology.orchestra.run.smithi089.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-15T23:24:17.701 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:24:17.701 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-09-15T23:24:17.701 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-15T23:24:17.701 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-15T23:24:17.701 DEBUG:teuthology.orchestra.run.smithi089:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-15T23:24:17.701 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-09-15T23:24:17.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:24:17.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:24:17.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:24:17.828 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-09-15T23:24:17.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:24:17.845 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list-id 2024-09-15T23:24:17.900 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:24:17.901 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-09-15T23:24:17.901 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-15T23:24:17.901 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-15T23:24:17.901 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-09-15T23:24:17.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:24:18.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:24:18.017 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-15T23:24:18.018 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:24:18.018 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-09-15T23:24:18.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-15T23:24:18.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-15T23:24:18.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-15T23:24:18.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-15T23:24:18.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-15T23:24:18.018 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-09-15T23:24:18.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:24:18.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:24:18.131 DEBUG:teuthology.orchestra.run.smithi089:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:24:18.131 DEBUG:teuthology.orchestra.run.smithi089:> set -e 2024-09-15T23:24:18.132 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set brx.0 up 2024-09-15T23:24:18.132 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link set dev brx.0 master ceph-brx 2024-09-15T23:24:18.132 DEBUG:teuthology.orchestra.run.smithi089:> ') 2024-09-15T23:24:18.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:24:18.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:24:18.209 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T23:24:18.209 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-15T23:24:18.224 INFO:teuthology.orchestra.run.smithi089.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-15T23:24:18.224 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T23:24:18.224 DEBUG:teuthology.orchestra.run.smithi089:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-15T23:24:18.279 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-09-15T23:24:18.279 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-09-15T23:24:18.279 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/50a69b32-a4a2-441c-ac79-60380a9adee5 2024-09-15T23:24:18.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:24:18.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/50a69b32-a4a2-441c-ac79-60380a9adee5 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-09-15T23:24:18.342 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:18 smithi089 ceph-mon[30933]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-09-15T23:24:18.363 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-09-15T23:24:18.364 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-09-15T23:24:18.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:18 smithi120 ceph-mon[35250]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-09-15T23:24:18.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:18 smithi174 ceph-mon[35586]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 85 B/s wr, 11 op/s; 5.2 KiB/s, 1 objects/s recovering 2024-09-15T23:24:18.498 INFO:teuthology.orchestra.run.smithi089.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-09-15T23:24:18.498 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-09-15T23:24:18.498 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: resolved to: "172.21.15.89:3300,172.21.15.120:3300,172.21.15.174:3300" 2024-09-15T23:24:18.499 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: trying mount with new device syntax: 0@1b4e699c-73b8-11ef-bceb-c7b262605968.cephfs=/volumes/qa/sv_1/50a69b32-a4a2-441c-ac79-60380a9adee5 2024-09-15T23:24:18.499 INFO:teuthology.orchestra.run.smithi089.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.89:3300/172.21.15.120:3300/172.21.15.174:3300" will pass to kernel 2024-09-15T23:24:18.499 INFO:teuthology.orchestra.run.smithi089.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-15T23:24:18.500 INFO:teuthology.orchestra.run.smithi089.stdout: You just mounted a file system that supports labels which does not 2024-09-15T23:24:18.500 INFO:teuthology.orchestra.run.smithi089.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-15T23:24:18.500 INFO:teuthology.orchestra.run.smithi089.stdout: applications will generate AVC messages and not be allowed access to 2024-09-15T23:24:18.500 INFO:teuthology.orchestra.run.smithi089.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-15T23:24:18.501 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-15T23:24:18.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:24:18.501 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-15T23:24:18.568 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-15T23:24:18.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T23:24:18.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:24:18.665 INFO:teuthology.orchestra.run.smithi089.stdout:client24790 2024-09-15T23:24:18.665 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:24:18.665 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T23:24:18.684 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-15T23:24:18.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T23:24:18.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:24:18.779 INFO:teuthology.orchestra.run.smithi089.stdout:client24790 2024-09-15T23:24:18.780 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-15T23:24:18.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T23:24:18.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:24:18.836 INFO:teuthology.orchestra.run.smithi089.stdout:client24790 2024-09-15T23:24:18.836 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968.client24790/status') 2024-09-15T23:24:18.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:24:18.894 INFO:teuthology.orchestra.run.smithi089.stdout:instance: client.24790 (3)172.21.15.89:0/2275896927 2024-09-15T23:24:18.894 INFO:teuthology.orchestra.run.smithi089.stdout:blocklisted: false 2024-09-15T23:24:18.895 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records in 2024-09-15T23:24:18.895 INFO:teuthology.orchestra.run.smithi089.stderr:0+1 records out 2024-09-15T23:24:18.895 INFO:teuthology.orchestra.run.smithi089.stderr:71 bytes copied, 2.1703e-05 s, 3.3 MB/s 2024-09-15T23:24:18.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:24:18.902 INFO:teuthology.run_tasks:Running task exec... 2024-09-15T23:24:18.918 INFO:teuthology.task.exec:Executing custom commands... 2024-09-15T23:24:18.918 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi089.front.sepia.ceph.com 2024-09-15T23:24:18.918 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-09-15T23:24:19.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 -- 172.21.15.89:0/3506001578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f511c102b40 msgr2=0x7f511c102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:19.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 --2- 172.21.15.89:0/3506001578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f511c102b40 0x7f511c102fc0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f510c009900 tx=0x7f510c02a410 comp rx=0 tx=0).stop 2024-09-15T23:24:19.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 -- 172.21.15.89:0/3506001578 shutdown_connections 2024-09-15T23:24:19.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 --2- 172.21.15.89:0/3506001578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511c103500 0x7f511c109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 --2- 172.21.15.89:0/3506001578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f511c102b40 0x7f511c102fc0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 --2- 172.21.15.89:0/3506001578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f511c101950 0x7f511c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 -- 172.21.15.89:0/3506001578 >> 172.21.15.89:0/3506001578 conn(0x7f511c0fd0d0 msgr2=0x7f511c0ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 -- 172.21.15.89:0/3506001578 shutdown_connections 2024-09-15T23:24:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 -- 172.21.15.89:0/3506001578 wait complete. 2024-09-15T23:24:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 Processor -- start 2024-09-15T23:24:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.019+0000 7f512199f640 1 -- start start 2024-09-15T23:24:19.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512199f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f511c101950 0x7f511c197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512199f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511c102b40 0x7f511c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512199f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f511c103500 0x7f511c19f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512199f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f511c10d6d0 con 0x7f511c103500 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512199f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f511c10d550 con 0x7f511c102b40 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512199f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f511c10d850 con 0x7f511c101950 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512119e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f511c103500 0x7f511c19f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512119e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f511c103500 0x7f511c19f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35656/0 (socket says 172.21.15.89:35656) 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512119e640 1 -- 172.21.15.89:0/2015550061 learned_addr learned my addr 172.21.15.89:0/2015550061 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512099d640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f511c101950 0x7f511c197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f5113fff640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511c102b40 0x7f511c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512119e640 1 -- 172.21.15.89:0/2015550061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f511c101950 msgr2=0x7f511c197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512119e640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f511c101950 0x7f511c197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512119e640 1 -- 172.21.15.89:0/2015550061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511c102b40 msgr2=0x7f511c198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:19.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512119e640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511c102b40 0x7f511c198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:19.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512119e640 1 -- 172.21.15.89:0/2015550061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f511c19fa20 con 0x7f511c103500 2024-09-15T23:24:19.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512099d640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f511c101950 0x7f511c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:19.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512119e640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f511c103500 0x7f511c19f3b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f511400b970 tx=0x7f511400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:19.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f5111ffb640 1 -- 172.21.15.89:0/2015550061 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5114019070 con 0x7f511c103500 2024-09-15T23:24:19.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f5111ffb640 1 -- 172.21.15.89:0/2015550061 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f511400cb40 con 0x7f511c103500 2024-09-15T23:24:19.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f512199f640 1 -- 172.21.15.89:0/2015550061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f511c19fd10 con 0x7f511c103500 2024-09-15T23:24:19.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.020+0000 7f5111ffb640 1 -- 172.21.15.89:0/2015550061 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5114007650 con 0x7f511c103500 2024-09-15T23:24:19.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.021+0000 7f512199f640 1 -- 172.21.15.89:0/2015550061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f511c1a0200 con 0x7f511c103500 2024-09-15T23:24:19.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.022+0000 7f512199f640 1 -- 172.21.15.89:0/2015550061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f511c108e80 con 0x7f511c103500 2024-09-15T23:24:19.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.023+0000 7f5111ffb640 1 -- 172.21.15.89:0/2015550061 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f511401c070 con 0x7f511c103500 2024-09-15T23:24:19.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.023+0000 7f5111ffb640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51040a3240 0x7f51040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:19.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.023+0000 7f512099d640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51040a3240 0x7f51040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:19.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.023+0000 7f5111ffb640 1 -- 172.21.15.89:0/2015550061 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f51140cf390 con 0x7f511c103500 2024-09-15T23:24:19.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.024+0000 7f512099d640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51040a3240 0x7f51040a5700 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f511c1029a0 tx=0x7f5118008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:19.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.027+0000 7f5111ffb640 1 -- 172.21.15.89:0/2015550061 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f511409bcc0 con 0x7f511c103500 2024-09-15T23:24:19.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:19.234+0000 7f512199f640 1 -- 172.21.15.89:0/2015550061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f511c062df0 con 0x7f511c103500 2024-09-15T23:24:20.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.089+0000 7f5111ffb640 1 -- 172.21.15.89:0/2015550061 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7f511409b5c0 con 0x7f511c103500 2024-09-15T23:24:20.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 -- 172.21.15.89:0/2015550061 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51040a3240 msgr2=0x7f51040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51040a3240 0x7f51040a5700 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f511c1029a0 tx=0x7f5118008040 comp rx=0 tx=0).stop 2024-09-15T23:24:20.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 -- 172.21.15.89:0/2015550061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f511c103500 msgr2=0x7f511c19f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f511c103500 0x7f511c19f3b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f511400b970 tx=0x7f511400be30 comp rx=0 tx=0).stop 2024-09-15T23:24:20.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 -- 172.21.15.89:0/2015550061 shutdown_connections 2024-09-15T23:24:20.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51040a3240 0x7f51040a5700 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f511c103500 0x7f511c19f3b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511c102b40 0x7f511c198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 --2- 172.21.15.89:0/2015550061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f511c101950 0x7f511c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 -- 172.21.15.89:0/2015550061 >> 172.21.15.89:0/2015550061 conn(0x7f511c0fd0d0 msgr2=0x7f511c106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:20.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 -- 172.21.15.89:0/2015550061 shutdown_connections 2024-09-15T23:24:20.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.092+0000 7f512199f640 1 -- 172.21.15.89:0/2015550061 wait complete. 2024-09-15T23:24:20.105 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-09-15T23:24:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 -- 172.21.15.89:0/577881993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a0101650 msgr2=0x7f22a0108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 --2- 172.21.15.89:0/577881993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a0101650 0x7f22a0108ef0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2298009ca0 tx=0x7f229802a8b0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 -- 172.21.15.89:0/577881993 shutdown_connections 2024-09-15T23:24:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 --2- 172.21.15.89:0/577881993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a0101650 0x7f22a0108ef0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 --2- 172.21.15.89:0/577881993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a0100c90 0x7f22a0101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 --2- 172.21.15.89:0/577881993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a01020d0 0x7f22a01024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 -- 172.21.15.89:0/577881993 >> 172.21.15.89:0/577881993 conn(0x7f22a00fb970 msgr2=0x7f22a00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:20.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 -- 172.21.15.89:0/577881993 shutdown_connections 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 -- 172.21.15.89:0/577881993 wait complete. 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 Processor -- start 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.210+0000 7f22a54d7640 1 -- start start 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f22a54d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a0100c90 0x7f22a0196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f22a54d7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a0101650 0x7f22a01972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f22a54d7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a01020d0 0x7f22a019e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f22a54d7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f22a010c7d0 con 0x7f22a0101650 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f22a54d7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f22a010c650 con 0x7f22a01020d0 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f22a54d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f22a010c950 con 0x7f22a0100c90 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f229e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a0101650 0x7f22a01972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f229e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a0101650 0x7f22a01972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35676/0 (socket says 172.21.15.89:35676) 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f229e7fc640 1 -- 172.21.15.89:0/563273 learned_addr learned my addr 172.21.15.89:0/563273 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f229effd640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a0100c90 0x7f22a0196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f229e7fc640 1 -- 172.21.15.89:0/563273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a0100c90 msgr2=0x7f22a0196db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f229e7fc640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a0100c90 0x7f22a0196db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f229e7fc640 1 -- 172.21.15.89:0/563273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a01020d0 msgr2=0x7f22a019e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f229e7fc640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a01020d0 0x7f22a019e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f229e7fc640 1 -- 172.21.15.89:0/563273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a019e9c0 con 0x7f22a0101650 2024-09-15T23:24:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f229e7fc640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a0101650 0x7f22a01972f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f228800d950 tx=0x7f228800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f2297fff640 1 -- 172.21.15.89:0/563273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288017070 con 0x7f22a0101650 2024-09-15T23:24:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f2297fff640 1 -- 172.21.15.89:0/563273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f228800be10 con 0x7f22a0101650 2024-09-15T23:24:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f22a54d7640 1 -- 172.21.15.89:0/563273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a019ecb0 con 0x7f22a0101650 2024-09-15T23:24:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f2297fff640 1 -- 172.21.15.89:0/563273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228800f620 con 0x7f22a0101650 2024-09-15T23:24:20.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.211+0000 7f22a54d7640 1 -- 172.21.15.89:0/563273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a019f1b0 con 0x7f22a0101650 2024-09-15T23:24:20.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.213+0000 7f22a54d7640 1 -- 172.21.15.89:0/563273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a0107d00 con 0x7f22a0101650 2024-09-15T23:24:20.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.214+0000 7f2297fff640 1 -- 172.21.15.89:0/563273 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f228801a070 con 0x7f22a0101650 2024-09-15T23:24:20.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.214+0000 7f2297fff640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f228c0a3370 0x7f228c0a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.214+0000 7f229effd640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f228c0a3370 0x7f228c0a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.214+0000 7f2297fff640 1 -- 172.21.15.89:0/563273 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f22880cf7d0 con 0x7f22a0101650 2024-09-15T23:24:20.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.215+0000 7f229effd640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f228c0a3370 0x7f228c0a5830 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f22a0198280 tx=0x7f2290009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:20.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.218+0000 7f2297fff640 1 -- 172.21.15.89:0/563273 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f228809c100 con 0x7f22a0101650 2024-09-15T23:24:20.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.408+0000 7f22a54d7640 1 -- 172.21.15.89:0/563273 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f22a0062790 con 0x7f228c0a3370 2024-09-15T23:24:20.415 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:20 smithi089 ceph-mon[30933]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T23:24:20.415 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:20 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2015550061' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-15T23:24:20.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.425+0000 7f2297fff640 1 -- 172.21.15.89:0/563273 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f22a0062790 con 0x7f228c0a3370 2024-09-15T23:24:20.426 INFO:teuthology.orchestra.run.smithi089.stdout:{} 2024-09-15T23:24:20.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 -- 172.21.15.89:0/563273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f228c0a3370 msgr2=0x7f228c0a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f228c0a3370 0x7f228c0a5830 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f22a0198280 tx=0x7f2290009110 comp rx=0 tx=0).stop 2024-09-15T23:24:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 -- 172.21.15.89:0/563273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a0101650 msgr2=0x7f22a01972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a0101650 0x7f22a01972f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f228800d950 tx=0x7f228800dd50 comp rx=0 tx=0).stop 2024-09-15T23:24:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 -- 172.21.15.89:0/563273 shutdown_connections 2024-09-15T23:24:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f228c0a3370 0x7f228c0a5830 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a01020d0 0x7f22a019e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22a0101650 0x7f22a01972f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 --2- 172.21.15.89:0/563273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22a0100c90 0x7f22a0196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 -- 172.21.15.89:0/563273 >> 172.21.15.89:0/563273 conn(0x7f22a00fb970 msgr2=0x7f22a0104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 -- 172.21.15.89:0/563273 shutdown_connections 2024-09-15T23:24:20.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.427+0000 7f22a54d7640 1 -- 172.21.15.89:0/563273 wait complete. 2024-09-15T23:24:20.439 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-15T23:24:20.454 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-09-15T23:24:20.470 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T23:24:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:20 smithi120 ceph-mon[35250]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T23:24:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:20 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2015550061' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-15T23:24:20.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:20 smithi174 ceph-mon[35586]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T23:24:20.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:20 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2015550061' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-09-15T23:24:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.569+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2971278384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25b41025c0 msgr2=0x7f25b4108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.569+0000 7f25ba7cf640 1 --2- 172.21.15.89:0/2971278384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25b41025c0 0x7f25b4108e50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f25ac009ca0 tx=0x7f25ac02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.569+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2971278384 shutdown_connections 2024-09-15T23:24:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.569+0000 7f25ba7cf640 1 --2- 172.21.15.89:0/2971278384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25b41025c0 0x7f25b4108e50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.569+0000 7f25ba7cf640 1 --2- 172.21.15.89:0/2971278384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b4101c00 0x7f25b4102080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.569+0000 7f25ba7cf640 1 --2- 172.21.15.89:0/2971278384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b4100a10 0x7f25b4100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.569+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2971278384 >> 172.21.15.89:0/2971278384 conn(0x7f25b40fc170 msgr2=0x7f25b40fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:20.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.569+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2971278384 shutdown_connections 2024-09-15T23:24:20.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.570+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2971278384 wait complete. 2024-09-15T23:24:20.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.570+0000 7f25ba7cf640 1 Processor -- start 2024-09-15T23:24:20.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.570+0000 7f25ba7cf640 1 -- start start 2024-09-15T23:24:20.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.570+0000 7f25ba7cf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b4100a10 0x7f25b4069ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.570+0000 7f25ba7cf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25b4101c00 0x7f25b419cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.570+0000 7f25ba7cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b41025c0 0x7f25b419f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.570+0000 7f25ba7cf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f25b410c690 con 0x7f25b4101c00 2024-09-15T23:24:20.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.570+0000 7f25ba7cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f25b410c510 con 0x7f25b41025c0 2024-09-15T23:24:20.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.570+0000 7f25ba7cf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25b410c810 con 0x7f25b4100a10 2024-09-15T23:24:20.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.570+0000 7f25b37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25b4101c00 0x7f25b419cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25b4101c00 0x7f25b419cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35710/0 (socket says 172.21.15.89:35710) 2024-09-15T23:24:20.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b37fe640 1 -- 172.21.15.89:0/2932893349 learned_addr learned my addr 172.21.15.89:0/2932893349 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:20.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b3fff640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b4100a10 0x7f25b4069ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b37fe640 1 -- 172.21.15.89:0/2932893349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b4100a10 msgr2=0x7f25b4069ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b8d6c640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b41025c0 0x7f25b419f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b37fe640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b4100a10 0x7f25b4069ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b37fe640 1 -- 172.21.15.89:0/2932893349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b41025c0 msgr2=0x7f25b419f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b37fe640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b41025c0 0x7f25b419f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b37fe640 1 -- 172.21.15.89:0/2932893349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b419fa70 con 0x7f25b4101c00 2024-09-15T23:24:20.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b3fff640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b4100a10 0x7f25b4069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:20.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b8d6c640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b41025c0 0x7f25b419f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:20.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b37fe640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25b4101c00 0x7f25b419cb40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f25a800d770 tx=0x7f25a800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:20.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b17fa640 1 -- 172.21.15.89:0/2932893349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8017070 con 0x7f25b4101c00 2024-09-15T23:24:20.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b17fa640 1 -- 172.21.15.89:0/2932893349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25a8005500 con 0x7f25b4101c00 2024-09-15T23:24:20.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2932893349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b419fd00 con 0x7f25b4101c00 2024-09-15T23:24:20.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25b17fa640 1 -- 172.21.15.89:0/2932893349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8004e00 con 0x7f25b4101c00 2024-09-15T23:24:20.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.571+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2932893349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b41a01a0 con 0x7f25b4101c00 2024-09-15T23:24:20.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.572+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2932893349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b4107e00 con 0x7f25b4101c00 2024-09-15T23:24:20.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.573+0000 7f25b17fa640 1 -- 172.21.15.89:0/2932893349 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f25a8004fa0 con 0x7f25b4101c00 2024-09-15T23:24:20.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.574+0000 7f25b17fa640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f259c0a3250 0x7f259c0a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.574+0000 7f25b3fff640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f259c0a3250 0x7f259c0a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.574+0000 7f25b17fa640 1 -- 172.21.15.89:0/2932893349 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f25a80cf830 con 0x7f25b4101c00 2024-09-15T23:24:20.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.575+0000 7f25b3fff640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f259c0a3250 0x7f259c0a5710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f25b4101a60 tx=0x7f25a4005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:20.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.578+0000 7f25b17fa640 1 -- 172.21.15.89:0/2932893349 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f25a809c160 con 0x7f25b4101c00 2024-09-15T23:24:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.745+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2932893349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25b4062e90 con 0x7f25b4101c00 2024-09-15T23:24:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.746+0000 7f25b17fa640 1 -- 172.21.15.89:0/2932893349 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24962 (secure 0 0 0) 0x7f25a809ba60 con 0x7f25b4101c00 2024-09-15T23:24:20.748 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:20.748 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":90,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:24:03.106312+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.11","mappings":[{"from":7,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:24:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.748+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2932893349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f259c0a3250 msgr2=0x7f259c0a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.748+0000 7f25ba7cf640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f259c0a3250 0x7f259c0a5710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f25b4101a60 tx=0x7f25a4005e70 comp rx=0 tx=0).stop 2024-09-15T23:24:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.748+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2932893349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25b4101c00 msgr2=0x7f25b419cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.748+0000 7f25ba7cf640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25b4101c00 0x7f25b419cb40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f25a800d770 tx=0x7f25a800dc30 comp rx=0 tx=0).stop 2024-09-15T23:24:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.749+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2932893349 shutdown_connections 2024-09-15T23:24:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.749+0000 7f25ba7cf640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f259c0a3250 0x7f259c0a5710 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.749+0000 7f25ba7cf640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25b41025c0 0x7f25b419f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.749+0000 7f25ba7cf640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f25b4101c00 0x7f25b419cb40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.749+0000 7f25ba7cf640 1 --2- 172.21.15.89:0/2932893349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b4100a10 0x7f25b4069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.749+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2932893349 >> 172.21.15.89:0/2932893349 conn(0x7f25b40fc170 msgr2=0x7f25b4104fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.749+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2932893349 shutdown_connections 2024-09-15T23:24:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.749+0000 7f25ba7cf640 1 -- 172.21.15.89:0/2932893349 wait complete. 2024-09-15T23:24:20.760 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:24:20 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:24:20.413+0000 7f8abb86a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:24:20.760 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:24:20 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:24:20.413+0000 7f8abb86a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:24:20.760 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:24:20 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:24:20.413+0000 7f8abb86a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:24:20.760 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:24:20 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:24:20.413+0000 7f8abb86a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:24:20.760 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:24:20 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:24:20.413+0000 7f8abb86a640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:24:20.762 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T23:17:15.186397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-15T23:21:49.360728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-15T23:21:51.296727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-15T23:21:57.778828+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-15T23:24:20.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T23:24:20.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.863+0000 7f365dc50640 1 -- 172.21.15.89:0/515802427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3658104ee0 msgr2=0x7f36581072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.863+0000 7f365dc50640 1 --2- 172.21.15.89:0/515802427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3658104ee0 0x7f36581072d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f364c009900 tx=0x7f364c02a410 comp rx=0 tx=0).stop 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.863+0000 7f365dc50640 1 -- 172.21.15.89:0/515802427 shutdown_connections 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.863+0000 7f365dc50640 1 --2- 172.21.15.89:0/515802427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3658107810 0x7f3658109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.863+0000 7f365dc50640 1 --2- 172.21.15.89:0/515802427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3658104ee0 0x7f36581072d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.863+0000 7f365dc50640 1 --2- 172.21.15.89:0/515802427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36581025b0 0x7f36581049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.863+0000 7f365dc50640 1 -- 172.21.15.89:0/515802427 >> 172.21.15.89:0/515802427 conn(0x7f36580fc1b0 msgr2=0x7f36580fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.863+0000 7f365dc50640 1 -- 172.21.15.89:0/515802427 shutdown_connections 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.863+0000 7f365dc50640 1 -- 172.21.15.89:0/515802427 wait complete. 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f365dc50640 1 Processor -- start 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f365dc50640 1 -- start start 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f365dc50640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36581025b0 0x7f3658197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f365dc50640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3658104ee0 0x7f36581984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f365dc50640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3658107810 0x7f365819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f365dc50640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f365810d840 con 0x7f36581025b0 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f365dc50640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f365810d6c0 con 0x7f3658104ee0 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f365dc50640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f365810d9c0 con 0x7f3658107810 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f36577fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36581025b0 0x7f3658197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f36577fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36581025b0 0x7f3658197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35734/0 (socket says 172.21.15.89:35734) 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f36577fe640 1 -- 172.21.15.89:0/1092388797 learned_addr learned my addr 172.21.15.89:0/1092388797 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f3656ffd640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3658104ee0 0x7f36581984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f3657fff640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3658107810 0x7f365819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f36577fe640 1 -- 172.21.15.89:0/1092388797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3658107810 msgr2=0x7f365819f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f36577fe640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3658107810 0x7f365819f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f36577fe640 1 -- 172.21.15.89:0/1092388797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3658104ee0 msgr2=0x7f36581984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:20.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f36577fe640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3658104ee0 0x7f36581984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:20.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f36577fe640 1 -- 172.21.15.89:0/1092388797 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365819fb80 con 0x7f36581025b0 2024-09-15T23:24:20.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.864+0000 7f3656ffd640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3658104ee0 0x7f36581984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:20.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.865+0000 7f36577fe640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36581025b0 0x7f3658197f60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f364800d950 tx=0x7f364800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:20.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.865+0000 7f3654ff9640 1 -- 172.21.15.89:0/1092388797 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648017070 con 0x7f36581025b0 2024-09-15T23:24:20.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.865+0000 7f3654ff9640 1 -- 172.21.15.89:0/1092388797 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3648009c30 con 0x7f36581025b0 2024-09-15T23:24:20.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.865+0000 7f365dc50640 1 -- 172.21.15.89:0/1092388797 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f365819fe70 con 0x7f36581025b0 2024-09-15T23:24:20.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.865+0000 7f3654ff9640 1 -- 172.21.15.89:0/1092388797 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364800f6b0 con 0x7f36581025b0 2024-09-15T23:24:20.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.865+0000 7f365dc50640 1 -- 172.21.15.89:0/1092388797 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36581a02e0 con 0x7f36581025b0 2024-09-15T23:24:20.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.866+0000 7f365dc50640 1 -- 172.21.15.89:0/1092388797 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36580fdca0 con 0x7f36581025b0 2024-09-15T23:24:20.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.867+0000 7f3654ff9640 1 -- 172.21.15.89:0/1092388797 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f36480109d0 con 0x7f36581025b0 2024-09-15T23:24:20.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.868+0000 7f3654ff9640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36400a32d0 0x7f36400a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:20.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.868+0000 7f3656ffd640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36400a32d0 0x7f36400a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:20.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.868+0000 7f3654ff9640 1 -- 172.21.15.89:0/1092388797 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f36480cf9a0 con 0x7f36581025b0 2024-09-15T23:24:20.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.868+0000 7f3656ffd640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36400a32d0 0x7f36400a5790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f364c006010 tx=0x7f364c0315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:20.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:20.871+0000 7f3654ff9640 1 -- 172.21.15.89:0/1092388797 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3648009de0 con 0x7f36581025b0 2024-09-15T23:24:21.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.039+0000 7f365dc50640 1 -- 172.21.15.89:0/1092388797 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3658062df0 con 0x7f36581025b0 2024-09-15T23:24:21.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.040+0000 7f3654ff9640 1 -- 172.21.15.89:0/1092388797 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f364809c2d0 con 0x7f36581025b0 2024-09-15T23:24:21.041 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-09-15T23:24:21.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 -- 172.21.15.89:0/1092388797 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36400a32d0 msgr2=0x7f36400a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36400a32d0 0x7f36400a5790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f364c006010 tx=0x7f364c0315b0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 -- 172.21.15.89:0/1092388797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36581025b0 msgr2=0x7f3658197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36581025b0 0x7f3658197f60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f364800d950 tx=0x7f364800de10 comp rx=0 tx=0).stop 2024-09-15T23:24:21.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 -- 172.21.15.89:0/1092388797 shutdown_connections 2024-09-15T23:24:21.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36400a32d0 0x7f36400a5790 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3658107810 0x7f365819f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3658104ee0 0x7f36581984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 --2- 172.21.15.89:0/1092388797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36581025b0 0x7f3658197f60 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 -- 172.21.15.89:0/1092388797 >> 172.21.15.89:0/1092388797 conn(0x7f36580fc1b0 msgr2=0x7f3658102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 -- 172.21.15.89:0/1092388797 shutdown_connections 2024-09-15T23:24:21.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.042+0000 7f365dc50640 1 -- 172.21.15.89:0/1092388797 wait complete. 2024-09-15T23:24:21.054 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-15T23:24:21.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.159+0000 7f17baea6640 1 -- 172.21.15.89:0/2195637293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17b41025c0 msgr2=0x7f17b4108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.159+0000 7f17baea6640 1 --2- 172.21.15.89:0/2195637293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17b41025c0 0x7f17b4108e50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f17ac009ca0 tx=0x7f17ac02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.159+0000 7f17baea6640 1 -- 172.21.15.89:0/2195637293 shutdown_connections 2024-09-15T23:24:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.159+0000 7f17baea6640 1 --2- 172.21.15.89:0/2195637293 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17b41025c0 0x7f17b4108e50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.159+0000 7f17baea6640 1 --2- 172.21.15.89:0/2195637293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17b4101c00 0x7f17b4102080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.159+0000 7f17baea6640 1 --2- 172.21.15.89:0/2195637293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17b4100a10 0x7f17b4100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.159+0000 7f17baea6640 1 -- 172.21.15.89:0/2195637293 >> 172.21.15.89:0/2195637293 conn(0x7f17b40fc190 msgr2=0x7f17b40fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.159+0000 7f17baea6640 1 -- 172.21.15.89:0/2195637293 shutdown_connections 2024-09-15T23:24:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.159+0000 7f17baea6640 1 -- 172.21.15.89:0/2195637293 wait complete. 2024-09-15T23:24:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17baea6640 1 Processor -- start 2024-09-15T23:24:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17baea6640 1 -- start start 2024-09-15T23:24:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17baea6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17b4100a10 0x7f17b4196d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17baea6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17b4101c00 0x7f17b4197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17baea6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17b41025c0 0x7f17b419e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17baea6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f17b410c670 con 0x7f17b41025c0 2024-09-15T23:24:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17baea6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f17b410c4f0 con 0x7f17b4100a10 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17baea6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f17b410c7f0 con 0x7f17b4101c00 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b9443640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17b41025c0 0x7f17b419e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b9443640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17b41025c0 0x7f17b419e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35758/0 (socket says 172.21.15.89:35758) 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b9443640 1 -- 172.21.15.89:0/1384358921 learned_addr learned my addr 172.21.15.89:0/1384358921 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b3fff640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17b4101c00 0x7f17b4197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b8c42640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17b4100a10 0x7f17b4196d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b9443640 1 -- 172.21.15.89:0/1384358921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17b4101c00 msgr2=0x7f17b4197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b9443640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17b4101c00 0x7f17b4197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b9443640 1 -- 172.21.15.89:0/1384358921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17b4100a10 msgr2=0x7f17b4196d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b9443640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17b4100a10 0x7f17b4196d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b9443640 1 -- 172.21.15.89:0/1384358921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b419e920 con 0x7f17b41025c0 2024-09-15T23:24:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.160+0000 7f17b8c42640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17b4100a10 0x7f17b4196d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.161+0000 7f17b9443640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17b41025c0 0x7f17b419e280 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f17ac031ae0 tx=0x7f17ac031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.161+0000 7f17b1ffb640 1 -- 172.21.15.89:0/1384358921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac003c30 con 0x7f17b41025c0 2024-09-15T23:24:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.161+0000 7f17b1ffb640 1 -- 172.21.15.89:0/1384358921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17ac003dd0 con 0x7f17b41025c0 2024-09-15T23:24:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.161+0000 7f17baea6640 1 -- 172.21.15.89:0/1384358921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b419ebb0 con 0x7f17b41025c0 2024-09-15T23:24:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.161+0000 7f17b1ffb640 1 -- 172.21.15.89:0/1384358921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac02fcb0 con 0x7f17b41025c0 2024-09-15T23:24:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.161+0000 7f17baea6640 1 -- 172.21.15.89:0/1384358921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b419f100 con 0x7f17b41025c0 2024-09-15T23:24:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.162+0000 7f17baea6640 1 -- 172.21.15.89:0/1384358921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b4107e00 con 0x7f17b41025c0 2024-09-15T23:24:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.163+0000 7f17b1ffb640 1 -- 172.21.15.89:0/1384358921 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f17ac004050 con 0x7f17b41025c0 2024-09-15T23:24:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.164+0000 7f17b1ffb640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17a40a32e0 0x7f17a40a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.164+0000 7f17b8c42640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17a40a32e0 0x7f17a40a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.164+0000 7f17b1ffb640 1 -- 172.21.15.89:0/1384358921 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f17ac0f1380 con 0x7f17b41025c0 2024-09-15T23:24:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.164+0000 7f17b8c42640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17a40a32e0 0x7f17a40a57a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f17b41981e0 tx=0x7f17a8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:21.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.167+0000 7f17b1ffb640 1 -- 172.21.15.89:0/1384358921 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f17ac0bdc60 con 0x7f17b41025c0 2024-09-15T23:24:21.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:21 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2015550061' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-15T23:24:21.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:21 smithi089 ceph-mon[30933]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T23:24:21.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:21 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2932893349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:21.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:21 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1092388797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:24:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.332+0000 7f17baea6640 1 -- 172.21.15.89:0/1384358921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f17b402da30 con 0x7f17b41025c0 2024-09-15T23:24:21.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.333+0000 7f17b1ffb640 1 -- 172.21.15.89:0/1384358921 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f17ac0bd560 con 0x7f17b41025c0 2024-09-15T23:24:21.335 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-09-15T23:24:21.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 -- 172.21.15.89:0/1384358921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17a40a32e0 msgr2=0x7f17a40a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17a40a32e0 0x7f17a40a57a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f17b41981e0 tx=0x7f17a8005f90 comp rx=0 tx=0).stop 2024-09-15T23:24:21.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 -- 172.21.15.89:0/1384358921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17b41025c0 msgr2=0x7f17b419e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17b41025c0 0x7f17b419e280 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f17ac031ae0 tx=0x7f17ac031ee0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 -- 172.21.15.89:0/1384358921 shutdown_connections 2024-09-15T23:24:21.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17a40a32e0 0x7f17a40a57a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17b41025c0 0x7f17b419e280 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17b4101c00 0x7f17b4197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 --2- 172.21.15.89:0/1384358921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17b4100a10 0x7f17b4196d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 -- 172.21.15.89:0/1384358921 >> 172.21.15.89:0/1384358921 conn(0x7f17b40fc190 msgr2=0x7f17b4104fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:21.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 -- 172.21.15.89:0/1384358921 shutdown_connections 2024-09-15T23:24:21.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.335+0000 7f17baea6640 1 -- 172.21.15.89:0/1384358921 wait complete. 2024-09-15T23:24:21.348 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-15T23:24:21.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.450+0000 7f75e4ea5640 1 -- 172.21.15.89:0/2554182716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e0102d10 msgr2=0x7f75e0103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.450+0000 7f75e4ea5640 1 --2- 172.21.15.89:0/2554182716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e0102d10 0x7f75e0103190 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f75d4009900 tx=0x7f75d402a410 comp rx=0 tx=0).stop 2024-09-15T23:24:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.450+0000 7f75e4ea5640 1 -- 172.21.15.89:0/2554182716 shutdown_connections 2024-09-15T23:24:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.450+0000 7f75e4ea5640 1 --2- 172.21.15.89:0/2554182716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75e01036d0 0x7f75e0109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.450+0000 7f75e4ea5640 1 --2- 172.21.15.89:0/2554182716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e0102d10 0x7f75e0103190 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.450+0000 7f75e4ea5640 1 --2- 172.21.15.89:0/2554182716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e0101b20 0x7f75e0101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.450+0000 7f75e4ea5640 1 -- 172.21.15.89:0/2554182716 >> 172.21.15.89:0/2554182716 conn(0x7f75e00fd2e0 msgr2=0x7f75e00ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.450+0000 7f75e4ea5640 1 -- 172.21.15.89:0/2554182716 shutdown_connections 2024-09-15T23:24:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.450+0000 7f75e4ea5640 1 -- 172.21.15.89:0/2554182716 wait complete. 2024-09-15T23:24:21.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75e4ea5640 1 Processor -- start 2024-09-15T23:24:21.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75e4ea5640 1 -- start start 2024-09-15T23:24:21.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75e4ea5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e0101b20 0x7f75e0075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75e4ea5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75e0102d10 0x7f75e00745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75e4ea5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e01036d0 0x7f75e0074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75e4ea5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f75e010d7a0 con 0x7f75e0101b20 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75e4ea5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f75e010d620 con 0x7f75e01036d0 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75e4ea5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f75e010d920 con 0x7f75e0102d10 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75de59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e0101b20 0x7f75e0075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75de59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e0101b20 0x7f75e0075fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35764/0 (socket says 172.21.15.89:35764) 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75de59c640 1 -- 172.21.15.89:0/3199086075 learned_addr learned my addr 172.21.15.89:0/3199086075 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75ddd9b640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75e0102d10 0x7f75e00745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75ded9d640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e01036d0 0x7f75e0074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75de59c640 1 -- 172.21.15.89:0/3199086075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75e0102d10 msgr2=0x7f75e00745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75de59c640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75e0102d10 0x7f75e00745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.455 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75de59c640 1 -- 172.21.15.89:0/3199086075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e01036d0 msgr2=0x7f75e0074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75de59c640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e01036d0 0x7f75e0074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75de59c640 1 -- 172.21.15.89:0/3199086075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e00753b0 con 0x7f75e0101b20 2024-09-15T23:24:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.451+0000 7f75ded9d640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e01036d0 0x7f75e0074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.452+0000 7f75de59c640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e0101b20 0x7f75e0075fa0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f75d000e9c0 tx=0x7f75d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.452+0000 7f75cf7fe640 1 -- 172.21.15.89:0/3199086075 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d000cd80 con 0x7f75e0101b20 2024-09-15T23:24:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.452+0000 7f75cf7fe640 1 -- 172.21.15.89:0/3199086075 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75d00035f0 con 0x7f75e0101b20 2024-09-15T23:24:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.452+0000 7f75e4ea5640 1 -- 172.21.15.89:0/3199086075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e01a1f70 con 0x7f75e0101b20 2024-09-15T23:24:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.452+0000 7f75cf7fe640 1 -- 172.21.15.89:0/3199086075 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d0010690 con 0x7f75e0101b20 2024-09-15T23:24:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.452+0000 7f75e4ea5640 1 -- 172.21.15.89:0/3199086075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e01a2370 con 0x7f75e0101b20 2024-09-15T23:24:21.456 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.453+0000 7f75e4ea5640 1 -- 172.21.15.89:0/3199086075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75e0108f10 con 0x7f75e0101b20 2024-09-15T23:24:21.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.454+0000 7f75cf7fe640 1 -- 172.21.15.89:0/3199086075 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f75d00037a0 con 0x7f75e0101b20 2024-09-15T23:24:21.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.455+0000 7f75cf7fe640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75c80a32e0 0x7f75c80a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.455+0000 7f75ddd9b640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75c80a32e0 0x7f75c80a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.455+0000 7f75cf7fe640 1 -- 172.21.15.89:0/3199086075 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f75d00d0340 con 0x7f75e0101b20 2024-09-15T23:24:21.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.455+0000 7f75ddd9b640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75c80a32e0 0x7f75c80a57a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f75d4000c00 tx=0x7f75d40315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:21.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.459+0000 7f75cf7fe640 1 -- 172.21.15.89:0/3199086075 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f75d009cb10 con 0x7f75e0101b20 2024-09-15T23:24:21.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:21 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2015550061' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-15T23:24:21.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:21 smithi120 ceph-mon[35250]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T23:24:21.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:21 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2932893349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:21.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:21 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1092388797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:24:21.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:21 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2015550061' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-09-15T23:24:21.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:21 smithi174 ceph-mon[35586]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-09-15T23:24:21.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:21 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2932893349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:21.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:21 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1092388797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:24:21.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.626+0000 7f75e4ea5640 1 -- 172.21.15.89:0/3199086075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f75e002da30 con 0x7f75e0101b20 2024-09-15T23:24:21.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.627+0000 7f75cf7fe640 1 -- 172.21.15.89:0/3199086075 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f75d009c410 con 0x7f75e0101b20 2024-09-15T23:24:21.628 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-09-15T23:24:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 -- 172.21.15.89:0/3199086075 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75c80a32e0 msgr2=0x7f75c80a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75c80a32e0 0x7f75c80a57a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f75d4000c00 tx=0x7f75d40315d0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 -- 172.21.15.89:0/3199086075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e0101b20 msgr2=0x7f75e0075fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e0101b20 0x7f75e0075fa0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f75d000e9c0 tx=0x7f75d000ee80 comp rx=0 tx=0).stop 2024-09-15T23:24:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 -- 172.21.15.89:0/3199086075 shutdown_connections 2024-09-15T23:24:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75c80a32e0 0x7f75c80a57a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75e01036d0 0x7f75e0074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75e0102d10 0x7f75e00745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 --2- 172.21.15.89:0/3199086075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f75e0101b20 0x7f75e0075fa0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 -- 172.21.15.89:0/3199086075 >> 172.21.15.89:0/3199086075 conn(0x7f75e00fd2e0 msgr2=0x7f75e01060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:21.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 -- 172.21.15.89:0/3199086075 shutdown_connections 2024-09-15T23:24:21.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.629+0000 7f75e4ea5640 1 -- 172.21.15.89:0/3199086075 wait complete. 2024-09-15T23:24:21.641 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-15T23:24:21.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 -- 172.21.15.89:0/1056721753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd998104db0 msgr2=0x7fd9981071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 --2- 172.21.15.89:0/1056721753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd998104db0 0x7fd9981071a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd984009900 tx=0x7fd98402a410 comp rx=0 tx=0).stop 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 -- 172.21.15.89:0/1056721753 shutdown_connections 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 --2- 172.21.15.89:0/1056721753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9981076e0 0x7fd998109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 --2- 172.21.15.89:0/1056721753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd998104db0 0x7fd9981071a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 --2- 172.21.15.89:0/1056721753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd998102480 0x7fd998104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 -- 172.21.15.89:0/1056721753 >> 172.21.15.89:0/1056721753 conn(0x7fd9980fc0a0 msgr2=0x7fd9980fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 -- 172.21.15.89:0/1056721753 shutdown_connections 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 -- 172.21.15.89:0/1056721753 wait complete. 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 Processor -- start 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.743+0000 7fd99df9f640 1 -- start start 2024-09-15T23:24:21.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd99df9f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd998102480 0x7fd998197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd99df9f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd998104db0 0x7fd998198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd99df9f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9981076e0 0x7fd99819f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd99df9f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd99810d690 con 0x7fd998104db0 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd99df9f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd99810d510 con 0x7fd998102480 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd99df9f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd99810d810 con 0x7fd9981076e0 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd997fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd998104db0 0x7fd998198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd997fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd998104db0 0x7fd998198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35784/0 (socket says 172.21.15.89:35784) 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd997fff640 1 -- 172.21.15.89:0/2340783171 learned_addr learned my addr 172.21.15.89:0/2340783171 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd99cf9d640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd998102480 0x7fd998197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd99d79e640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9981076e0 0x7fd99819f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd997fff640 1 -- 172.21.15.89:0/2340783171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9981076e0 msgr2=0x7fd99819f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd997fff640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9981076e0 0x7fd99819f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd997fff640 1 -- 172.21.15.89:0/2340783171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd998102480 msgr2=0x7fd998197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd997fff640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd998102480 0x7fd998197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd997fff640 1 -- 172.21.15.89:0/2340783171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99819f950 con 0x7fd998104db0 2024-09-15T23:24:21.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd997fff640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd998104db0 0x7fd998198280 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd984031610 tx=0x7fd9840043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:21.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd995ffb640 1 -- 172.21.15.89:0/2340783171 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98403f030 con 0x7fd998104db0 2024-09-15T23:24:21.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd995ffb640 1 -- 172.21.15.89:0/2340783171 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd984003ce0 con 0x7fd998104db0 2024-09-15T23:24:21.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd99df9f640 1 -- 172.21.15.89:0/2340783171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99819fbe0 con 0x7fd998104db0 2024-09-15T23:24:21.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.744+0000 7fd995ffb640 1 -- 172.21.15.89:0/2340783171 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984035070 con 0x7fd998104db0 2024-09-15T23:24:21.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.745+0000 7fd99df9f640 1 -- 172.21.15.89:0/2340783171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9981a00d0 con 0x7fd998104db0 2024-09-15T23:24:21.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.745+0000 7fd99df9f640 1 -- 172.21.15.89:0/2340783171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9980fdb90 con 0x7fd998104db0 2024-09-15T23:24:21.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.747+0000 7fd995ffb640 1 -- 172.21.15.89:0/2340783171 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fd984042030 con 0x7fd998104db0 2024-09-15T23:24:21.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.747+0000 7fd995ffb640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9880a32e0 0x7fd9880a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:21.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.747+0000 7fd99cf9d640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9880a32e0 0x7fd9880a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:21.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.747+0000 7fd995ffb640 1 -- 172.21.15.89:0/2340783171 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd9840f0570 con 0x7fd998104db0 2024-09-15T23:24:21.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.748+0000 7fd99cf9d640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9880a32e0 0x7fd9880a57a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd98c005e10 tx=0x7fd98c005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:21.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.751+0000 7fd995ffb640 1 -- 172.21.15.89:0/2340783171 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd9840bcd40 con 0x7fd998104db0 2024-09-15T23:24:21.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.917+0000 7fd99df9f640 1 -- 172.21.15.89:0/2340783171 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd99802da10 con 0x7fd998104db0 2024-09-15T23:24:21.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.917+0000 7fd995ffb640 1 -- 172.21.15.89:0/2340783171 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7fd9840bc640 con 0x7fd998104db0 2024-09-15T23:24:21.918 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-09-15T23:24:21.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 -- 172.21.15.89:0/2340783171 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9880a32e0 msgr2=0x7fd9880a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9880a32e0 0x7fd9880a57a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd98c005e10 tx=0x7fd98c005da0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 -- 172.21.15.89:0/2340783171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd998104db0 msgr2=0x7fd998198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:21.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd998104db0 0x7fd998198280 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd984031610 tx=0x7fd9840043b0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 -- 172.21.15.89:0/2340783171 shutdown_connections 2024-09-15T23:24:21.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9880a32e0 0x7fd9880a57a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9981076e0 0x7fd99819f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd998104db0 0x7fd998198280 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 --2- 172.21.15.89:0/2340783171 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd998102480 0x7fd998197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:21.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 -- 172.21.15.89:0/2340783171 >> 172.21.15.89:0/2340783171 conn(0x7fd9980fc0a0 msgr2=0x7fd998102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:21.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 -- 172.21.15.89:0/2340783171 shutdown_connections 2024-09-15T23:24:21.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:21.919+0000 7fd99df9f640 1 -- 172.21.15.89:0/2340783171 wait complete. 2024-09-15T23:24:21.932 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-09-15T23:24:21.932 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:22.182 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:22.303 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:22 smithi089 ceph-mon[30933]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:24:22.304 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:22 smithi089 ceph-mon[30933]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T23:24:22.304 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:22 smithi089 ceph-mon[30933]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-15T23:24:22.304 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:22 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1384358921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:24:22.304 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:22 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3199086075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:24:22.304 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:22 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2340783171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:24:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:22 smithi120 ceph-mon[35250]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:24:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:22 smithi120 ceph-mon[35250]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T23:24:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:22 smithi120 ceph-mon[35250]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-15T23:24:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:22 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1384358921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:24:22.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:22 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3199086075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:24:22.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:22 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2340783171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:24:22.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:22 smithi174 ceph-mon[35586]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:24:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:22 smithi174 ceph-mon[35586]: pgmap v360: 129 pgs: 129 active+clean; 645 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T23:24:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:22 smithi174 ceph-mon[35586]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-09-15T23:24:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:22 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1384358921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:24:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:22 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3199086075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:24:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:22 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2340783171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:24:22.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 -- 172.21.15.89:0/1055333147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7f8108160 msgr2=0x7fc7f810eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:22.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 --2- 172.21.15.89:0/1055333147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7f8108160 0x7fc7f810eb00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009cd0 tx=0x7fc7f002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 -- 172.21.15.89:0/1055333147 shutdown_connections 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 --2- 172.21.15.89:0/1055333147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7f8108160 0x7fc7f810eb00 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 --2- 172.21.15.89:0/1055333147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7f8101290 0x7fc7f8107a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 --2- 172.21.15.89:0/1055333147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f81008e0 0x7fc7f8100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 -- 172.21.15.89:0/1055333147 >> 172.21.15.89:0/1055333147 conn(0x7fc7f8077210 msgr2=0x7fc7f8077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 -- 172.21.15.89:0/1055333147 shutdown_connections 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 -- 172.21.15.89:0/1055333147 wait complete. 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 Processor -- start 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.705+0000 7fc800585640 1 -- start start 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc800585640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7f81008e0 0x7fc7f8198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc800585640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f8101290 0x7fc7f81989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc800585640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7f8108160 0x7fc7f819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc800585640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f8112030 con 0x7fc7f81008e0 2024-09-15T23:24:22.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc800585640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f8111eb0 con 0x7fc7f8108160 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc800585640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81121b0 con 0x7fc7f8101290 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7fe321640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7f81008e0 0x7fc7f8198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7fe321640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7f81008e0 0x7fc7f8198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35804/0 (socket says 172.21.15.89:35804) 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7fe321640 1 -- 172.21.15.89:0/3782975980 learned_addr learned my addr 172.21.15.89:0/3782975980 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7fdb20640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f8101290 0x7fc7f81989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7feb22640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7f8108160 0x7fc7f819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7fe321640 1 -- 172.21.15.89:0/3782975980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f8101290 msgr2=0x7fc7f81989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7fe321640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f8101290 0x7fc7f81989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7fe321640 1 -- 172.21.15.89:0/3782975980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7f8108160 msgr2=0x7fc7f819cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7fe321640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7f8108160 0x7fc7f819cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7fe321640 1 -- 172.21.15.89:0/3782975980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f819d420 con 0x7fc7f81008e0 2024-09-15T23:24:22.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7fe321640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7f81008e0 0x7fc7f8198470 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400b4a0 tx=0x7fc7f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:22.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7ef7fe640 1 -- 172.21.15.89:0/3782975980 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4014070 con 0x7fc7f81008e0 2024-09-15T23:24:22.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.706+0000 7fc7ef7fe640 1 -- 172.21.15.89:0/3782975980 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7f4010af0 con 0x7fc7f81008e0 2024-09-15T23:24:22.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.707+0000 7fc800585640 1 -- 172.21.15.89:0/3782975980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f819d710 con 0x7fc7f81008e0 2024-09-15T23:24:22.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.707+0000 7fc7ef7fe640 1 -- 172.21.15.89:0/3782975980 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4017d10 con 0x7fc7f81008e0 2024-09-15T23:24:22.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.707+0000 7fc800585640 1 -- 172.21.15.89:0/3782975980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f81a4ff0 con 0x7fc7f81008e0 2024-09-15T23:24:22.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.708+0000 7fc800585640 1 -- 172.21.15.89:0/3782975980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f8100cc0 con 0x7fc7f81008e0 2024-09-15T23:24:22.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.709+0000 7fc7ef7fe640 1 -- 172.21.15.89:0/3782975980 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fc7f4017070 con 0x7fc7f81008e0 2024-09-15T23:24:22.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.709+0000 7fc7ef7fe640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7e40a32d0 0x7fc7e40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:22.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.710+0000 7fc7fdb20640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7e40a32d0 0x7fc7e40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:22.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.710+0000 7fc7ef7fe640 1 -- 172.21.15.89:0/3782975980 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc7f40cf1a0 con 0x7fc7f81008e0 2024-09-15T23:24:22.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.710+0000 7fc7fdb20640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7e40a32d0 0x7fc7e40a5790 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8005e10 tx=0x7fc7e8005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:22.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.714+0000 7fc7ef7fe640 1 -- 172.21.15.89:0/3782975980 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc7f409ba80 con 0x7fc7f81008e0 2024-09-15T23:24:22.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.911+0000 7fc800585640 1 -- 172.21.15.89:0/3782975980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7f81a52f0 con 0x7fc7f81008e0 2024-09-15T23:24:22.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.912+0000 7fc7ef7fe640 1 -- 172.21.15.89:0/3782975980 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fc7f409b380 con 0x7fc7f81008e0 2024-09-15T23:24:22.913 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:22.914 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:24:22.914 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:24:22.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 -- 172.21.15.89:0/3782975980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7e40a32d0 msgr2=0x7fc7e40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:22.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7e40a32d0 0x7fc7e40a5790 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8005e10 tx=0x7fc7e8005da0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 -- 172.21.15.89:0/3782975980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7f81008e0 msgr2=0x7fc7f8198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:22.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7f81008e0 0x7fc7f8198470 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400b4a0 tx=0x7fc7f400b960 comp rx=0 tx=0).stop 2024-09-15T23:24:22.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 -- 172.21.15.89:0/3782975980 shutdown_connections 2024-09-15T23:24:22.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7e40a32d0 0x7fc7e40a5790 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7f8108160 0x7fc7f819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7f8101290 0x7fc7f81989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 --2- 172.21.15.89:0/3782975980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7f81008e0 0x7fc7f8198470 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:22.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 -- 172.21.15.89:0/3782975980 >> 172.21.15.89:0/3782975980 conn(0x7fc7f8077210 msgr2=0x7fc7f80ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:22.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 -- 172.21.15.89:0/3782975980 shutdown_connections 2024-09-15T23:24:22.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:22.915+0000 7fc800585640 1 -- 172.21.15.89:0/3782975980 wait complete. 2024-09-15T23:24:23.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:23 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3782975980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:23 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3782975980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:23.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:23 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3782975980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:23.649 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-09-15T23:24:23.649 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-09-15T23:24:23.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T23:24:23.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.750+0000 7f5165da6640 1 -- 172.21.15.89:0/220534366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5160101c20 msgr2=0x7f51601020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.750+0000 7f5165da6640 1 --2- 172.21.15.89:0/220534366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5160101c20 0x7f51601020a0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5154009900 tx=0x7f515402a410 comp rx=0 tx=0).stop 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.750+0000 7f5165da6640 1 -- 172.21.15.89:0/220534366 shutdown_connections 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.750+0000 7f5165da6640 1 --2- 172.21.15.89:0/220534366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51601025e0 0x7f5160108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.750+0000 7f5165da6640 1 --2- 172.21.15.89:0/220534366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5160101c20 0x7f51601020a0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.750+0000 7f5165da6640 1 --2- 172.21.15.89:0/220534366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5160100a30 0x7f5160100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.750+0000 7f5165da6640 1 -- 172.21.15.89:0/220534366 >> 172.21.15.89:0/220534366 conn(0x7f51600fc190 msgr2=0x7f51600fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.750+0000 7f5165da6640 1 -- 172.21.15.89:0/220534366 shutdown_connections 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.750+0000 7f5165da6640 1 -- 172.21.15.89:0/220534366 wait complete. 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.750+0000 7f5165da6640 1 Processor -- start 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f5165da6640 1 -- start start 2024-09-15T23:24:23.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f5165da6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5160100a30 0x7f516006af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f5165da6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5160101c20 0x7f516019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f5165da6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51601025e0 0x7f516019f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f5165da6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f516010c700 con 0x7f5160101c20 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f5165da6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f516010c580 con 0x7f5160100a30 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f5165da6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f516010c880 con 0x7f51601025e0 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f515effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5160101c20 0x7f516019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f515effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5160101c20 0x7f516019cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35822/0 (socket says 172.21.15.89:35822) 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f515effd640 1 -- 172.21.15.89:0/306822691 learned_addr learned my addr 172.21.15.89:0/306822691 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f515ffff640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51601025e0 0x7f516019f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f515f7fe640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5160100a30 0x7f516006af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f515effd640 1 -- 172.21.15.89:0/306822691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51601025e0 msgr2=0x7f516019f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:23.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f515effd640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51601025e0 0x7f516019f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f515effd640 1 -- 172.21.15.89:0/306822691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5160100a30 msgr2=0x7f516006af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f515effd640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5160100a30 0x7f516006af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.751+0000 7f515effd640 1 -- 172.21.15.89:0/306822691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516019fba0 con 0x7f5160101c20 2024-09-15T23:24:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.752+0000 7f515effd640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5160101c20 0x7f516019cc70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f51540314e0 tx=0x7f5154032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.752+0000 7f515cff9640 1 -- 172.21.15.89:0/306822691 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515403f030 con 0x7f5160101c20 2024-09-15T23:24:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.752+0000 7f5165da6640 1 -- 172.21.15.89:0/306822691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f516019fdd0 con 0x7f5160101c20 2024-09-15T23:24:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.752+0000 7f515cff9640 1 -- 172.21.15.89:0/306822691 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5154032e50 con 0x7f5160101c20 2024-09-15T23:24:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.752+0000 7f515cff9640 1 -- 172.21.15.89:0/306822691 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5154035070 con 0x7f5160101c20 2024-09-15T23:24:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.752+0000 7f5165da6640 1 -- 172.21.15.89:0/306822691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51601a0260 con 0x7f5160101c20 2024-09-15T23:24:23.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.753+0000 7f5165da6640 1 -- 172.21.15.89:0/306822691 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5160107e70 con 0x7f5160101c20 2024-09-15T23:24:23.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.754+0000 7f515cff9640 1 -- 172.21.15.89:0/306822691 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f5154042030 con 0x7f5160101c20 2024-09-15T23:24:23.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.755+0000 7f515cff9640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51480a32e0 0x7f51480a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.755+0000 7f515f7fe640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51480a32e0 0x7f51480a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.755+0000 7f515cff9640 1 -- 172.21.15.89:0/306822691 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f51540f0880 con 0x7f5160101c20 2024-09-15T23:24:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.755+0000 7f515f7fe640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51480a32e0 0x7f51480a57a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5150005fd0 tx=0x7f5150005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:23.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.758+0000 7f515cff9640 1 -- 172.21.15.89:0/306822691 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f51540bd050 con 0x7f5160101c20 2024-09-15T23:24:23.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.922+0000 7f5165da6640 1 -- 172.21.15.89:0/306822691 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5160062630 con 0x7f51480a32e0 2024-09-15T23:24:23.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.930+0000 7f515cff9640 1 -- 172.21.15.89:0/306822691 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340821 (secure 0 0 0) 0x7f5160062630 con 0x7f51480a32e0 2024-09-15T23:24:23.936 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:23.939 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-15T23:24:23.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 -- 172.21.15.89:0/306822691 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51480a32e0 msgr2=0x7f51480a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:23.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51480a32e0 0x7f51480a57a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5150005fd0 tx=0x7f5150005850 comp rx=0 tx=0).stop 2024-09-15T23:24:23.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 -- 172.21.15.89:0/306822691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5160101c20 msgr2=0x7f516019cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:23.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5160101c20 0x7f516019cc70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f51540314e0 tx=0x7f5154032c60 comp rx=0 tx=0).stop 2024-09-15T23:24:23.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 -- 172.21.15.89:0/306822691 shutdown_connections 2024-09-15T23:24:23.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51480a32e0 0x7f51480a57a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:23.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51601025e0 0x7f516019f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:23.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5160101c20 0x7f516019cc70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:23.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 --2- 172.21.15.89:0/306822691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5160100a30 0x7f516006af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:23.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 -- 172.21.15.89:0/306822691 >> 172.21.15.89:0/306822691 conn(0x7f51600fc190 msgr2=0x7f5160105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:23.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 -- 172.21.15.89:0/306822691 shutdown_connections 2024-09-15T23:24:23.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:23.934+0000 7f5165da6640 1 -- 172.21.15.89:0/306822691 wait complete. 2024-09-15T23:24:23.946 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-15T23:24:23.021695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666658,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1197,"num_read_kb":720,"num_write":446,"num_write_kb":3098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1244183,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369880,"kb_used_data":12264,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571944,"statfs":{"total":1151940427776,"available":1151561670656,"internally_reserved":0,"allocated":12558336,"data_stored":8199791,"data_compressed":471759,"data_compressed_allocated":638976,"data_compressed_original":2813952,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6775,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":137,"num_read_kb":72,"num_write":9,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003289"},"pg_stats":[{"pgid":"2.1f","version":"86'13","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:16.275939+0000","last_change":"2024-09-15T23:21:50.513610+0000","last_active":"2024-09-15T23:24:16.275939+0000","last_peered":"2024-09-15T23:24:16.275939+0000","last_clean":"2024-09-15T23:24:16.275939+0000","last_became_active":"2024-09-15T23:21:50.513266+0000","last_became_peered":"2024-09-15T23:21:50.513266+0000","last_unstale":"2024-09-15T23:24:16.275939+0000","last_undegraded":"2024-09-15T23:24:16.275939+0000","last_fullsized":"2024-09-15T23:24:16.275939+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:16:48.692801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112578+0000","last_change":"2024-09-15T23:21:52.540248+0000","last_active":"2024-09-15T23:24:03.112578+0000","last_peered":"2024-09-15T23:24:03.112578+0000","last_clean":"2024-09-15T23:24:03.112578+0000","last_became_active":"2024-09-15T23:21:52.539945+0000","last_became_peered":"2024-09-15T23:21:52.539945+0000","last_unstale":"2024-09-15T23:24:03.112578+0000","last_undegraded":"2024-09-15T23:24:03.112578+0000","last_fullsized":"2024-09-15T23:24:03.112578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:33.402153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112360+0000","last_change":"2024-09-15T23:21:59.567246+0000","last_active":"2024-09-15T23:24:03.112360+0000","last_peered":"2024-09-15T23:24:03.112360+0000","last_clean":"2024-09-15T23:24:03.112360+0000","last_became_active":"2024-09-15T23:21:59.566832+0000","last_became_peered":"2024-09-15T23:21:59.566832+0000","last_unstale":"2024-09-15T23:24:03.112360+0000","last_undegraded":"2024-09-15T23:24:03.112360+0000","last_fullsized":"2024-09-15T23:24:03.112360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:19.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112747+0000","last_change":"2024-09-15T23:21:50.561767+0000","last_active":"2024-09-15T23:24:03.112747+0000","last_peered":"2024-09-15T23:24:03.112747+0000","last_clean":"2024-09-15T23:24:03.112747+0000","last_became_active":"2024-09-15T23:21:50.561385+0000","last_became_peered":"2024-09-15T23:21:50.561385+0000","last_unstale":"2024-09-15T23:24:03.112747+0000","last_undegraded":"2024-09-15T23:24:03.112747+0000","last_fullsized":"2024-09-15T23:24:03.112747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:11.897884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194008+0000","last_change":"2024-09-15T23:21:52.534874+0000","last_active":"2024-09-15T23:24:03.194008+0000","last_peered":"2024-09-15T23:24:03.194008+0000","last_clean":"2024-09-15T23:24:03.194008+0000","last_became_active":"2024-09-15T23:21:52.534334+0000","last_became_peered":"2024-09-15T23:21:52.534334+0000","last_unstale":"2024-09-15T23:24:03.194008+0000","last_undegraded":"2024-09-15T23:24:03.194008+0000","last_fullsized":"2024-09-15T23:24:03.194008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:15.972648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110384+0000","last_change":"2024-09-15T23:21:59.559907+0000","last_active":"2024-09-15T23:24:03.110384+0000","last_peered":"2024-09-15T23:24:03.110384+0000","last_clean":"2024-09-15T23:24:03.110384+0000","last_became_active":"2024-09-15T23:21:59.559440+0000","last_became_peered":"2024-09-15T23:21:59.559440+0000","last_unstale":"2024-09-15T23:24:03.110384+0000","last_undegraded":"2024-09-15T23:24:03.110384+0000","last_fullsized":"2024-09-15T23:24:03.110384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:00.755112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117658+0000","last_change":"2024-09-15T23:21:50.515424+0000","last_active":"2024-09-15T23:24:03.117658+0000","last_peered":"2024-09-15T23:24:03.117658+0000","last_clean":"2024-09-15T23:24:03.117658+0000","last_became_active":"2024-09-15T23:21:50.514966+0000","last_became_peered":"2024-09-15T23:21:50.514966+0000","last_unstale":"2024-09-15T23:24:03.117658+0000","last_undegraded":"2024-09-15T23:24:03.117658+0000","last_fullsized":"2024-09-15T23:24:03.117658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:57.586575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117341+0000","last_change":"2024-09-15T23:21:52.535189+0000","last_active":"2024-09-15T23:24:03.117341+0000","last_peered":"2024-09-15T23:24:03.117341+0000","last_clean":"2024-09-15T23:24:03.117341+0000","last_became_active":"2024-09-15T23:21:52.534807+0000","last_became_peered":"2024-09-15T23:21:52.534807+0000","last_unstale":"2024-09-15T23:24:03.117341+0000","last_undegraded":"2024-09-15T23:24:03.117341+0000","last_fullsized":"2024-09-15T23:24:03.117341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:30.654382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110773+0000","last_change":"2024-09-15T23:21:59.564675+0000","last_active":"2024-09-15T23:24:03.110773+0000","last_peered":"2024-09-15T23:24:03.110773+0000","last_clean":"2024-09-15T23:24:03.110773+0000","last_became_active":"2024-09-15T23:21:59.564271+0000","last_became_peered":"2024-09-15T23:21:59.564271+0000","last_unstale":"2024-09-15T23:24:03.110773+0000","last_undegraded":"2024-09-15T23:24:03.110773+0000","last_fullsized":"2024-09-15T23:24:03.110773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:01.055369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117219+0000","last_change":"2024-09-15T23:21:50.507790+0000","last_active":"2024-09-15T23:24:03.117219+0000","last_peered":"2024-09-15T23:24:03.117219+0000","last_clean":"2024-09-15T23:24:03.117219+0000","last_became_active":"2024-09-15T23:21:50.507398+0000","last_became_peered":"2024-09-15T23:21:50.507398+0000","last_unstale":"2024-09-15T23:24:03.117219+0000","last_undegraded":"2024-09-15T23:24:03.117219+0000","last_fullsized":"2024-09-15T23:24:03.117219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:58:00.001279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116935+0000","last_change":"2024-09-15T23:21:52.537179+0000","last_active":"2024-09-15T23:24:03.116935+0000","last_peered":"2024-09-15T23:24:03.116935+0000","last_clean":"2024-09-15T23:24:03.116935+0000","last_became_active":"2024-09-15T23:21:52.536841+0000","last_became_peered":"2024-09-15T23:21:52.536841+0000","last_unstale":"2024-09-15T23:24:03.116935+0000","last_undegraded":"2024-09-15T23:24:03.116935+0000","last_fullsized":"2024-09-15T23:24:03.116935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:47.455700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110441+0000","last_change":"2024-09-15T23:21:59.564734+0000","last_active":"2024-09-15T23:24:03.110441+0000","last_peered":"2024-09-15T23:24:03.110441+0000","last_clean":"2024-09-15T23:24:03.110441+0000","last_became_active":"2024-09-15T23:21:59.564339+0000","last_became_peered":"2024-09-15T23:21:59.564339+0000","last_unstale":"2024-09-15T23:24:03.110441+0000","last_undegraded":"2024-09-15T23:24:03.110441+0000","last_fullsized":"2024-09-15T23:24:03.110441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:44.498480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113970+0000","last_change":"2024-09-15T23:21:52.538025+0000","last_active":"2024-09-15T23:24:03.113970+0000","last_peered":"2024-09-15T23:24:03.113970+0000","last_clean":"2024-09-15T23:24:03.113970+0000","last_became_active":"2024-09-15T23:21:52.537736+0000","last_became_peered":"2024-09-15T23:21:52.537736+0000","last_unstale":"2024-09-15T23:24:03.113970+0000","last_undegraded":"2024-09-15T23:24:03.113970+0000","last_fullsized":"2024-09-15T23:24:03.113970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:31.916888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.127260+0000","last_change":"2024-09-15T23:24:03.127260+0000","last_active":"2024-09-15T23:24:03.127260+0000","last_peered":"2024-09-15T23:24:03.127260+0000","last_clean":"2024-09-15T23:24:03.127260+0000","last_became_active":"2024-09-15T23:24:03.119384+0000","last_became_peered":"2024-09-15T23:24:03.119384+0000","last_unstale":"2024-09-15T23:24:03.127260+0000","last_undegraded":"2024-09-15T23:24:03.127260+0000","last_fullsized":"2024-09-15T23:24:03.127260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:35.325048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110625+0000","last_change":"2024-09-15T23:21:59.681971+0000","last_active":"2024-09-15T23:24:03.110625+0000","last_peered":"2024-09-15T23:24:03.110625+0000","last_clean":"2024-09-15T23:24:03.110625+0000","last_became_active":"2024-09-15T23:21:59.681432+0000","last_became_peered":"2024-09-15T23:21:59.681432+0000","last_unstale":"2024-09-15T23:24:03.110625+0000","last_undegraded":"2024-09-15T23:24:03.110625+0000","last_fullsized":"2024-09-15T23:24:03.110625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:26.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112179+0000","last_change":"2024-09-15T23:21:52.541281+0000","last_active":"2024-09-15T23:24:03.112179+0000","last_peered":"2024-09-15T23:24:03.112179+0000","last_clean":"2024-09-15T23:24:03.112179+0000","last_became_active":"2024-09-15T23:21:52.540772+0000","last_became_peered":"2024-09-15T23:21:52.540772+0000","last_unstale":"2024-09-15T23:24:03.112179+0000","last_undegraded":"2024-09-15T23:24:03.112179+0000","last_fullsized":"2024-09-15T23:24:03.112179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:23:44.444904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112700+0000","last_change":"2024-09-15T23:21:50.508844+0000","last_active":"2024-09-15T23:24:03.112700+0000","last_peered":"2024-09-15T23:24:03.112700+0000","last_clean":"2024-09-15T23:24:03.112700+0000","last_became_active":"2024-09-15T23:21:50.508364+0000","last_became_peered":"2024-09-15T23:21:50.508364+0000","last_unstale":"2024-09-15T23:24:03.112700+0000","last_undegraded":"2024-09-15T23:24:03.112700+0000","last_fullsized":"2024-09-15T23:24:03.112700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:45.129711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110720+0000","last_change":"2024-09-15T23:21:59.562114+0000","last_active":"2024-09-15T23:24:03.110720+0000","last_peered":"2024-09-15T23:24:03.110720+0000","last_clean":"2024-09-15T23:24:03.110720+0000","last_became_active":"2024-09-15T23:21:59.561652+0000","last_became_peered":"2024-09-15T23:21:59.561652+0000","last_unstale":"2024-09-15T23:24:03.110720+0000","last_undegraded":"2024-09-15T23:24:03.110720+0000","last_fullsized":"2024-09-15T23:24:03.110720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:51.781886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.531678+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.531327+0000","last_became_peered":"2024-09-15T23:21:52.531327+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:07.152626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112193+0000","last_change":"2024-09-15T23:21:50.513197+0000","last_active":"2024-09-15T23:24:03.112193+0000","last_peered":"2024-09-15T23:24:03.112193+0000","last_clean":"2024-09-15T23:24:03.112193+0000","last_became_active":"2024-09-15T23:21:50.512794+0000","last_became_peered":"2024-09-15T23:21:50.512794+0000","last_unstale":"2024-09-15T23:24:03.112193+0000","last_undegraded":"2024-09-15T23:24:03.112193+0000","last_fullsized":"2024-09-15T23:24:03.112193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:05.457467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194628+0000","last_change":"2024-09-15T23:21:59.565669+0000","last_active":"2024-09-15T23:24:03.194628+0000","last_peered":"2024-09-15T23:24:03.194628+0000","last_clean":"2024-09-15T23:24:03.194628+0000","last_became_active":"2024-09-15T23:21:59.565290+0000","last_became_peered":"2024-09-15T23:21:59.565290+0000","last_unstale":"2024-09-15T23:24:03.194628+0000","last_undegraded":"2024-09-15T23:24:03.194628+0000","last_fullsized":"2024-09-15T23:24:03.194628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:04.861560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116394+0000","last_change":"2024-09-15T23:21:52.529643+0000","last_active":"2024-09-15T23:24:03.116394+0000","last_peered":"2024-09-15T23:24:03.116394+0000","last_clean":"2024-09-15T23:24:03.116394+0000","last_became_active":"2024-09-15T23:21:52.529304+0000","last_became_peered":"2024-09-15T23:21:52.529304+0000","last_unstale":"2024-09-15T23:24:03.116394+0000","last_undegraded":"2024-09-15T23:24:03.116394+0000","last_fullsized":"2024-09-15T23:24:03.116394+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:18:37.758403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117052+0000","last_change":"2024-09-15T23:21:50.562114+0000","last_active":"2024-09-15T23:24:03.117052+0000","last_peered":"2024-09-15T23:24:03.117052+0000","last_clean":"2024-09-15T23:24:03.117052+0000","last_became_active":"2024-09-15T23:21:50.561772+0000","last_became_peered":"2024-09-15T23:21:50.561772+0000","last_unstale":"2024-09-15T23:24:03.117052+0000","last_undegraded":"2024-09-15T23:24:03.117052+0000","last_fullsized":"2024-09-15T23:24:03.117052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:14.806646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110672+0000","last_change":"2024-09-15T23:21:59.681933+0000","last_active":"2024-09-15T23:24:03.110672+0000","last_peered":"2024-09-15T23:24:03.110672+0000","last_clean":"2024-09-15T23:24:03.110672+0000","last_became_active":"2024-09-15T23:21:59.681528+0000","last_became_peered":"2024-09-15T23:21:59.681528+0000","last_unstale":"2024-09-15T23:24:03.110672+0000","last_undegraded":"2024-09-15T23:24:03.110672+0000","last_fullsized":"2024-09-15T23:24:03.110672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:46.199540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194326+0000","last_change":"2024-09-15T23:21:52.529548+0000","last_active":"2024-09-15T23:24:03.194326+0000","last_peered":"2024-09-15T23:24:03.194326+0000","last_clean":"2024-09-15T23:24:03.194326+0000","last_became_active":"2024-09-15T23:21:52.529253+0000","last_became_peered":"2024-09-15T23:21:52.529253+0000","last_unstale":"2024-09-15T23:24:03.194326+0000","last_undegraded":"2024-09-15T23:24:03.194326+0000","last_fullsized":"2024-09-15T23:24:03.194326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:22.813395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111022+0000","last_change":"2024-09-15T23:21:50.767759+0000","last_active":"2024-09-15T23:24:03.111022+0000","last_peered":"2024-09-15T23:24:03.111022+0000","last_clean":"2024-09-15T23:24:03.111022+0000","last_became_active":"2024-09-15T23:21:50.767388+0000","last_became_peered":"2024-09-15T23:21:50.767388+0000","last_unstale":"2024-09-15T23:24:03.111022+0000","last_undegraded":"2024-09-15T23:24:03.111022+0000","last_fullsized":"2024-09-15T23:24:03.111022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:36.713769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112512+0000","last_change":"2024-09-15T23:21:59.557951+0000","last_active":"2024-09-15T23:24:03.112512+0000","last_peered":"2024-09-15T23:24:03.112512+0000","last_clean":"2024-09-15T23:24:03.112512+0000","last_became_active":"2024-09-15T23:21:59.557490+0000","last_became_peered":"2024-09-15T23:21:59.557490+0000","last_unstale":"2024-09-15T23:24:03.112512+0000","last_undegraded":"2024-09-15T23:24:03.112512+0000","last_fullsized":"2024-09-15T23:24:03.112512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:17.147389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112859+0000","last_change":"2024-09-15T23:21:52.531638+0000","last_active":"2024-09-15T23:24:03.112859+0000","last_peered":"2024-09-15T23:24:03.112859+0000","last_clean":"2024-09-15T23:24:03.112859+0000","last_became_active":"2024-09-15T23:21:52.531301+0000","last_became_peered":"2024-09-15T23:21:52.531301+0000","last_unstale":"2024-09-15T23:24:03.112859+0000","last_undegraded":"2024-09-15T23:24:03.112859+0000","last_fullsized":"2024-09-15T23:24:03.112859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.436286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'16","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194603+0000","last_change":"2024-09-15T23:21:50.508470+0000","last_active":"2024-09-15T23:24:03.194603+0000","last_peered":"2024-09-15T23:24:03.194603+0000","last_clean":"2024-09-15T23:24:03.194603+0000","last_became_active":"2024-09-15T23:21:50.508060+0000","last_became_peered":"2024-09-15T23:21:50.508060+0000","last_unstale":"2024-09-15T23:24:03.194603+0000","last_undegraded":"2024-09-15T23:24:03.194603+0000","last_fullsized":"2024-09-15T23:24:03.194603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:59.542252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111992+0000","last_change":"2024-09-15T23:21:59.682019+0000","last_active":"2024-09-15T23:24:03.111992+0000","last_peered":"2024-09-15T23:24:03.111992+0000","last_clean":"2024-09-15T23:24:03.111992+0000","last_became_active":"2024-09-15T23:21:59.681467+0000","last_became_peered":"2024-09-15T23:21:59.681467+0000","last_unstale":"2024-09-15T23:24:03.111992+0000","last_undegraded":"2024-09-15T23:24:03.111992+0000","last_fullsized":"2024-09-15T23:24:03.111992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:27.650382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109766+0000","last_change":"2024-09-15T23:21:52.536345+0000","last_active":"2024-09-15T23:24:03.109766+0000","last_peered":"2024-09-15T23:24:03.109766+0000","last_clean":"2024-09-15T23:24:03.109766+0000","last_became_active":"2024-09-15T23:21:52.536027+0000","last_became_peered":"2024-09-15T23:21:52.536027+0000","last_unstale":"2024-09-15T23:24:03.109766+0000","last_undegraded":"2024-09-15T23:24:03.109766+0000","last_fullsized":"2024-09-15T23:24:03.109766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:16.435912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117716+0000","last_change":"2024-09-15T23:21:50.562945+0000","last_active":"2024-09-15T23:24:03.117716+0000","last_peered":"2024-09-15T23:24:03.117716+0000","last_clean":"2024-09-15T23:24:03.117716+0000","last_became_active":"2024-09-15T23:21:50.561632+0000","last_became_peered":"2024-09-15T23:21:50.561632+0000","last_unstale":"2024-09-15T23:24:03.117716+0000","last_undegraded":"2024-09-15T23:24:03.117716+0000","last_fullsized":"2024-09-15T23:24:03.117716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:44.545489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115268+0000","last_change":"2024-09-15T23:21:59.564793+0000","last_active":"2024-09-15T23:24:03.115268+0000","last_peered":"2024-09-15T23:24:03.115268+0000","last_clean":"2024-09-15T23:24:03.115268+0000","last_became_active":"2024-09-15T23:21:59.564140+0000","last_became_peered":"2024-09-15T23:21:59.564140+0000","last_unstale":"2024-09-15T23:24:03.115268+0000","last_undegraded":"2024-09-15T23:24:03.115268+0000","last_fullsized":"2024-09-15T23:24:03.115268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:41.464962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113451+0000","last_change":"2024-09-15T23:21:52.533521+0000","last_active":"2024-09-15T23:24:03.113451+0000","last_peered":"2024-09-15T23:24:03.113451+0000","last_clean":"2024-09-15T23:24:03.113451+0000","last_became_active":"2024-09-15T23:21:52.532195+0000","last_became_peered":"2024-09-15T23:21:52.532195+0000","last_unstale":"2024-09-15T23:24:03.113451+0000","last_undegraded":"2024-09-15T23:24:03.113451+0000","last_fullsized":"2024-09-15T23:24:03.113451+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:14.229018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194243+0000","last_change":"2024-09-15T23:21:50.562893+0000","last_active":"2024-09-15T23:24:03.194243+0000","last_peered":"2024-09-15T23:24:03.194243+0000","last_clean":"2024-09-15T23:24:03.194243+0000","last_became_active":"2024-09-15T23:21:50.562516+0000","last_became_peered":"2024-09-15T23:21:50.562516+0000","last_unstale":"2024-09-15T23:24:03.194243+0000","last_undegraded":"2024-09-15T23:24:03.194243+0000","last_fullsized":"2024-09-15T23:24:03.194243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:47:47.442458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117856+0000","last_change":"2024-09-15T23:21:59.682091+0000","last_active":"2024-09-15T23:24:03.117856+0000","last_peered":"2024-09-15T23:24:03.117856+0000","last_clean":"2024-09-15T23:24:03.117856+0000","last_became_active":"2024-09-15T23:21:59.681574+0000","last_became_peered":"2024-09-15T23:21:59.681574+0000","last_unstale":"2024-09-15T23:24:03.117856+0000","last_undegraded":"2024-09-15T23:24:03.117856+0000","last_fullsized":"2024-09-15T23:24:03.117856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:04.991608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114426+0000","last_change":"2024-09-15T23:21:52.533811+0000","last_active":"2024-09-15T23:24:03.114426+0000","last_peered":"2024-09-15T23:24:03.114426+0000","last_clean":"2024-09-15T23:24:03.114426+0000","last_became_active":"2024-09-15T23:21:52.533487+0000","last_became_peered":"2024-09-15T23:21:52.533487+0000","last_unstale":"2024-09-15T23:24:03.114426+0000","last_undegraded":"2024-09-15T23:24:03.114426+0000","last_fullsized":"2024-09-15T23:24:03.114426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:44.193212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.281895+0000","last_change":"2024-09-15T23:21:50.510883+0000","last_active":"2024-09-15T23:24:20.281895+0000","last_peered":"2024-09-15T23:24:20.281895+0000","last_clean":"2024-09-15T23:24:20.281895+0000","last_became_active":"2024-09-15T23:21:50.510478+0000","last_became_peered":"2024-09-15T23:21:50.510478+0000","last_unstale":"2024-09-15T23:24:20.281895+0000","last_undegraded":"2024-09-15T23:24:20.281895+0000","last_fullsized":"2024-09-15T23:24:20.281895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:35.454885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109705+0000","last_change":"2024-09-15T23:21:59.561816+0000","last_active":"2024-09-15T23:24:03.109705+0000","last_peered":"2024-09-15T23:24:03.109705+0000","last_clean":"2024-09-15T23:24:03.109705+0000","last_became_active":"2024-09-15T23:21:59.561434+0000","last_became_peered":"2024-09-15T23:21:59.561434+0000","last_unstale":"2024-09-15T23:24:03.109705+0000","last_undegraded":"2024-09-15T23:24:03.109705+0000","last_fullsized":"2024-09-15T23:24:03.109705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:59.945802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117955+0000","last_change":"2024-09-15T23:23:02.277204+0000","last_active":"2024-09-15T23:24:03.117955+0000","last_peered":"2024-09-15T23:24:03.117955+0000","last_clean":"2024-09-15T23:24:03.117955+0000","last_became_active":"2024-09-15T23:23:02.276770+0000","last_became_peered":"2024-09-15T23:23:02.276770+0000","last_unstale":"2024-09-15T23:24:03.117955+0000","last_undegraded":"2024-09-15T23:24:03.117955+0000","last_fullsized":"2024-09-15T23:24:03.117955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:11.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.145290+0000","last_change":"2024-09-15T23:21:50.561684+0000","last_active":"2024-09-15T23:24:20.145290+0000","last_peered":"2024-09-15T23:24:20.145290+0000","last_clean":"2024-09-15T23:24:20.145290+0000","last_became_active":"2024-09-15T23:21:50.561335+0000","last_became_peered":"2024-09-15T23:21:50.561335+0000","last_unstale":"2024-09-15T23:24:20.145290+0000","last_undegraded":"2024-09-15T23:24:20.145290+0000","last_fullsized":"2024-09-15T23:24:20.145290+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:23.897250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109863+0000","last_change":"2024-09-15T23:21:59.681858+0000","last_active":"2024-09-15T23:24:03.109863+0000","last_peered":"2024-09-15T23:24:03.109863+0000","last_clean":"2024-09-15T23:24:03.109863+0000","last_became_active":"2024-09-15T23:21:59.681441+0000","last_became_peered":"2024-09-15T23:21:59.681441+0000","last_unstale":"2024-09-15T23:24:03.109863+0000","last_undegraded":"2024-09-15T23:24:03.109863+0000","last_fullsized":"2024-09-15T23:24:03.109863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:28.513719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109320+0000","last_change":"2024-09-15T23:21:52.536141+0000","last_active":"2024-09-15T23:24:03.109320+0000","last_peered":"2024-09-15T23:24:03.109320+0000","last_clean":"2024-09-15T23:24:03.109320+0000","last_became_active":"2024-09-15T23:21:52.535832+0000","last_became_peered":"2024-09-15T23:21:52.535832+0000","last_unstale":"2024-09-15T23:24:03.109320+0000","last_undegraded":"2024-09-15T23:24:03.109320+0000","last_fullsized":"2024-09-15T23:24:03.109320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:21:44.341412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.123671+0000","last_change":"2024-09-15T23:24:03.123671+0000","last_active":"2024-09-15T23:24:03.123671+0000","last_peered":"2024-09-15T23:24:03.123671+0000","last_clean":"2024-09-15T23:24:03.123671+0000","last_became_active":"2024-09-15T23:24:03.119585+0000","last_became_peered":"2024-09-15T23:24:03.119585+0000","last_unstale":"2024-09-15T23:24:03.123671+0000","last_undegraded":"2024-09-15T23:24:03.123671+0000","last_fullsized":"2024-09-15T23:24:03.123671+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:30.734156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118359+0000","last_change":"2024-09-15T23:21:59.682118+0000","last_active":"2024-09-15T23:24:03.118359+0000","last_peered":"2024-09-15T23:24:03.118359+0000","last_clean":"2024-09-15T23:24:03.118359+0000","last_became_active":"2024-09-15T23:21:59.681500+0000","last_became_peered":"2024-09-15T23:21:59.681500+0000","last_unstale":"2024-09-15T23:24:03.118359+0000","last_undegraded":"2024-09-15T23:24:03.118359+0000","last_fullsized":"2024-09-15T23:24:03.118359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:58.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112298+0000","last_change":"2024-09-15T23:21:52.540458+0000","last_active":"2024-09-15T23:24:03.112298+0000","last_peered":"2024-09-15T23:24:03.112298+0000","last_clean":"2024-09-15T23:24:03.112298+0000","last_became_active":"2024-09-15T23:21:52.540112+0000","last_became_peered":"2024-09-15T23:21:52.540112+0000","last_unstale":"2024-09-15T23:24:03.112298+0000","last_undegraded":"2024-09-15T23:24:03.112298+0000","last_fullsized":"2024-09-15T23:24:03.112298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:15.240785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111097+0000","last_change":"2024-09-15T23:21:50.513124+0000","last_active":"2024-09-15T23:24:03.111097+0000","last_peered":"2024-09-15T23:24:03.111097+0000","last_clean":"2024-09-15T23:24:03.111097+0000","last_became_active":"2024-09-15T23:21:50.512705+0000","last_became_peered":"2024-09-15T23:21:50.512705+0000","last_unstale":"2024-09-15T23:24:03.111097+0000","last_undegraded":"2024-09-15T23:24:03.111097+0000","last_fullsized":"2024-09-15T23:24:03.111097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.417937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":354,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113701+0000","last_change":"2024-09-15T23:20:40.656652+0000","last_active":"2024-09-15T23:24:03.113701+0000","last_peered":"2024-09-15T23:24:03.113701+0000","last_clean":"2024-09-15T23:24:03.113701+0000","last_became_active":"2024-09-15T23:20:40.656266+0000","last_became_peered":"2024-09-15T23:20:40.656266+0000","last_unstale":"2024-09-15T23:24:03.113701+0000","last_undegraded":"2024-09-15T23:24:03.113701+0000","last_fullsized":"2024-09-15T23:24:03.113701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_clean_scrub_stamp":"2024-09-15T23:17:15.186850+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:42.317006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113728+0000","last_change":"2024-09-15T23:21:59.563777+0000","last_active":"2024-09-15T23:24:03.113728+0000","last_peered":"2024-09-15T23:24:03.113728+0000","last_clean":"2024-09-15T23:24:03.113728+0000","last_became_active":"2024-09-15T23:21:59.563320+0000","last_became_peered":"2024-09-15T23:21:59.563320+0000","last_unstale":"2024-09-15T23:24:03.113728+0000","last_undegraded":"2024-09-15T23:24:03.113728+0000","last_fullsized":"2024-09-15T23:24:03.113728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:25:07.791923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114145+0000","last_change":"2024-09-15T23:21:52.539842+0000","last_active":"2024-09-15T23:24:03.114145+0000","last_peered":"2024-09-15T23:24:03.114145+0000","last_clean":"2024-09-15T23:24:03.114145+0000","last_became_active":"2024-09-15T23:21:52.538683+0000","last_became_peered":"2024-09-15T23:21:52.538683+0000","last_unstale":"2024-09-15T23:24:03.114145+0000","last_undegraded":"2024-09-15T23:24:03.114145+0000","last_fullsized":"2024-09-15T23:24:03.114145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:43.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:18.280187+0000","last_change":"2024-09-15T23:21:50.767601+0000","last_active":"2024-09-15T23:24:18.280187+0000","last_peered":"2024-09-15T23:24:18.280187+0000","last_clean":"2024-09-15T23:24:18.280187+0000","last_became_active":"2024-09-15T23:21:50.767149+0000","last_became_peered":"2024-09-15T23:21:50.767149+0000","last_unstale":"2024-09-15T23:24:18.280187+0000","last_undegraded":"2024-09-15T23:24:18.280187+0000","last_fullsized":"2024-09-15T23:24:18.280187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:23.636551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117168+0000","last_change":"2024-09-15T23:21:59.563069+0000","last_active":"2024-09-15T23:24:03.117168+0000","last_peered":"2024-09-15T23:24:03.117168+0000","last_clean":"2024-09-15T23:24:03.117168+0000","last_became_active":"2024-09-15T23:21:59.562658+0000","last_became_peered":"2024-09-15T23:21:59.562658+0000","last_unstale":"2024-09-15T23:24:03.117168+0000","last_undegraded":"2024-09-15T23:24:03.117168+0000","last_fullsized":"2024-09-15T23:24:03.117168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:23.067198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113535+0000","last_change":"2024-09-15T23:21:52.535289+0000","last_active":"2024-09-15T23:24:03.113535+0000","last_peered":"2024-09-15T23:24:03.113535+0000","last_clean":"2024-09-15T23:24:03.113535+0000","last_became_active":"2024-09-15T23:21:52.534885+0000","last_became_peered":"2024-09-15T23:21:52.534885+0000","last_unstale":"2024-09-15T23:24:03.113535+0000","last_undegraded":"2024-09-15T23:24:03.113535+0000","last_fullsized":"2024-09-15T23:24:03.113535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:42.293786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118612+0000","last_change":"2024-09-15T23:21:50.513246+0000","last_active":"2024-09-15T23:24:03.118612+0000","last_peered":"2024-09-15T23:24:03.118612+0000","last_clean":"2024-09-15T23:24:03.118612+0000","last_became_active":"2024-09-15T23:21:50.512740+0000","last_became_peered":"2024-09-15T23:21:50.512740+0000","last_unstale":"2024-09-15T23:24:03.118612+0000","last_undegraded":"2024-09-15T23:24:03.118612+0000","last_fullsized":"2024-09-15T23:24:03.118612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:59.201970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193756+0000","last_change":"2024-09-15T23:21:59.568565+0000","last_active":"2024-09-15T23:24:03.193756+0000","last_peered":"2024-09-15T23:24:03.193756+0000","last_clean":"2024-09-15T23:24:03.193756+0000","last_became_active":"2024-09-15T23:21:59.567789+0000","last_became_peered":"2024-09-15T23:21:59.567789+0000","last_unstale":"2024-09-15T23:24:03.193756+0000","last_undegraded":"2024-09-15T23:24:03.193756+0000","last_fullsized":"2024-09-15T23:24:03.193756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:02.106971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114106+0000","last_change":"2024-09-15T23:21:52.533727+0000","last_active":"2024-09-15T23:24:03.114106+0000","last_peered":"2024-09-15T23:24:03.114106+0000","last_clean":"2024-09-15T23:24:03.114106+0000","last_became_active":"2024-09-15T23:21:52.532350+0000","last_became_peered":"2024-09-15T23:21:52.532350+0000","last_unstale":"2024-09-15T23:24:03.114106+0000","last_undegraded":"2024-09-15T23:24:03.114106+0000","last_fullsized":"2024-09-15T23:24:03.114106+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:03.624837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.124878+0000","last_change":"2024-09-15T23:24:03.124430+0000","last_active":"2024-09-15T23:24:03.124878+0000","last_peered":"2024-09-15T23:24:03.124878+0000","last_clean":"2024-09-15T23:24:03.124878+0000","last_became_active":"2024-09-15T23:24:03.119042+0000","last_became_peered":"2024-09-15T23:24:03.119042+0000","last_unstale":"2024-09-15T23:24:03.124878+0000","last_undegraded":"2024-09-15T23:24:03.124878+0000","last_fullsized":"2024-09-15T23:24:03.124878+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:52.065543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117798+0000","last_change":"2024-09-15T23:21:59.566429+0000","last_active":"2024-09-15T23:24:03.117798+0000","last_peered":"2024-09-15T23:24:03.117798+0000","last_clean":"2024-09-15T23:24:03.117798+0000","last_became_active":"2024-09-15T23:21:59.565835+0000","last_became_peered":"2024-09-15T23:21:59.565835+0000","last_unstale":"2024-09-15T23:24:03.117798+0000","last_undegraded":"2024-09-15T23:24:03.117798+0000","last_fullsized":"2024-09-15T23:24:03.117798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:38.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116276+0000","last_change":"2024-09-15T23:21:52.539544+0000","last_active":"2024-09-15T23:24:03.116276+0000","last_peered":"2024-09-15T23:24:03.116276+0000","last_clean":"2024-09-15T23:24:03.116276+0000","last_became_active":"2024-09-15T23:21:52.539203+0000","last_became_peered":"2024-09-15T23:21:52.539203+0000","last_unstale":"2024-09-15T23:24:03.116276+0000","last_undegraded":"2024-09-15T23:24:03.116276+0000","last_fullsized":"2024-09-15T23:24:03.116276+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.193965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:07.046123+0000","last_change":"2024-09-15T23:21:50.513543+0000","last_active":"2024-09-15T23:24:07.046123+0000","last_peered":"2024-09-15T23:24:07.046123+0000","last_clean":"2024-09-15T23:24:07.046123+0000","last_became_active":"2024-09-15T23:21:50.512812+0000","last_became_peered":"2024-09-15T23:21:50.512812+0000","last_unstale":"2024-09-15T23:24:07.046123+0000","last_undegraded":"2024-09-15T23:24:07.046123+0000","last_fullsized":"2024-09-15T23:24:07.046123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:50.954096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117689+0000","last_change":"2024-09-15T23:21:59.570163+0000","last_active":"2024-09-15T23:24:03.117689+0000","last_peered":"2024-09-15T23:24:03.117689+0000","last_clean":"2024-09-15T23:24:03.117689+0000","last_became_active":"2024-09-15T23:21:59.569805+0000","last_became_peered":"2024-09-15T23:21:59.569805+0000","last_unstale":"2024-09-15T23:24:03.117689+0000","last_undegraded":"2024-09-15T23:24:03.117689+0000","last_fullsized":"2024-09-15T23:24:03.117689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:28.530817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117877+0000","last_change":"2024-09-15T23:21:52.537850+0000","last_active":"2024-09-15T23:24:03.117877+0000","last_peered":"2024-09-15T23:24:03.117877+0000","last_clean":"2024-09-15T23:24:03.117877+0000","last_became_active":"2024-09-15T23:21:52.536579+0000","last_became_peered":"2024-09-15T23:21:52.536579+0000","last_unstale":"2024-09-15T23:24:03.117877+0000","last_undegraded":"2024-09-15T23:24:03.117877+0000","last_fullsized":"2024-09-15T23:24:03.117877+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:06.353615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118588+0000","last_change":"2024-09-15T23:21:50.562221+0000","last_active":"2024-09-15T23:24:03.118588+0000","last_peered":"2024-09-15T23:24:03.118588+0000","last_clean":"2024-09-15T23:24:03.118588+0000","last_became_active":"2024-09-15T23:21:50.561788+0000","last_became_peered":"2024-09-15T23:21:50.561788+0000","last_unstale":"2024-09-15T23:24:03.118588+0000","last_undegraded":"2024-09-15T23:24:03.118588+0000","last_fullsized":"2024-09-15T23:24:03.118588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.853812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109810+0000","last_change":"2024-09-15T23:23:02.383003+0000","last_active":"2024-09-15T23:24:03.109810+0000","last_peered":"2024-09-15T23:24:03.109810+0000","last_clean":"2024-09-15T23:24:03.109810+0000","last_became_active":"2024-09-15T23:23:02.382629+0000","last_became_peered":"2024-09-15T23:23:02.382629+0000","last_unstale":"2024-09-15T23:24:03.109810+0000","last_undegraded":"2024-09-15T23:24:03.109810+0000","last_fullsized":"2024-09-15T23:24:03.109810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:50.114689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113567+0000","last_change":"2024-09-15T23:21:52.539566+0000","last_active":"2024-09-15T23:24:03.113567+0000","last_peered":"2024-09-15T23:24:03.113567+0000","last_clean":"2024-09-15T23:24:03.113567+0000","last_became_active":"2024-09-15T23:21:52.539219+0000","last_became_peered":"2024-09-15T23:21:52.539219+0000","last_unstale":"2024-09-15T23:24:03.113567+0000","last_undegraded":"2024-09-15T23:24:03.113567+0000","last_fullsized":"2024-09-15T23:24:03.113567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:41.266308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":167,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.145816+0000","last_change":"2024-09-15T23:24:03.127667+0000","last_active":"2024-09-15T23:24:20.145816+0000","last_peered":"2024-09-15T23:24:20.145816+0000","last_clean":"2024-09-15T23:24:20.145816+0000","last_became_active":"2024-09-15T23:24:03.121613+0000","last_became_peered":"2024-09-15T23:24:03.121613+0000","last_unstale":"2024-09-15T23:24:20.145816+0000","last_undegraded":"2024-09-15T23:24:20.145816+0000","last_fullsized":"2024-09-15T23:24:20.145816+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:54:24.166895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":4,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1122,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194597+0000","last_change":"2024-09-15T23:21:59.568087+0000","last_active":"2024-09-15T23:24:03.194597+0000","last_peered":"2024-09-15T23:24:03.194597+0000","last_clean":"2024-09-15T23:24:03.194597+0000","last_became_active":"2024-09-15T23:21:59.567693+0000","last_became_peered":"2024-09-15T23:21:59.567693+0000","last_unstale":"2024-09-15T23:24:03.194597+0000","last_undegraded":"2024-09-15T23:24:03.194597+0000","last_fullsized":"2024-09-15T23:24:03.194597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.231873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114396+0000","last_change":"2024-09-15T23:21:52.531090+0000","last_active":"2024-09-15T23:24:03.114396+0000","last_peered":"2024-09-15T23:24:03.114396+0000","last_clean":"2024-09-15T23:24:03.114396+0000","last_became_active":"2024-09-15T23:21:52.530738+0000","last_became_peered":"2024-09-15T23:21:52.530738+0000","last_unstale":"2024-09-15T23:24:03.114396+0000","last_undegraded":"2024-09-15T23:24:03.114396+0000","last_fullsized":"2024-09-15T23:24:03.114396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:35.325128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117423+0000","last_change":"2024-09-15T23:21:50.514100+0000","last_active":"2024-09-15T23:24:03.117423+0000","last_peered":"2024-09-15T23:24:03.117423+0000","last_clean":"2024-09-15T23:24:03.117423+0000","last_became_active":"2024-09-15T23:21:50.513629+0000","last_became_peered":"2024-09-15T23:21:50.513629+0000","last_unstale":"2024-09-15T23:24:03.117423+0000","last_undegraded":"2024-09-15T23:24:03.117423+0000","last_fullsized":"2024-09-15T23:24:03.117423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.187544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115275+0000","last_change":"2024-09-15T23:21:59.562804+0000","last_active":"2024-09-15T23:24:03.115275+0000","last_peered":"2024-09-15T23:24:03.115275+0000","last_clean":"2024-09-15T23:24:03.115275+0000","last_became_active":"2024-09-15T23:21:59.562368+0000","last_became_peered":"2024-09-15T23:21:59.562368+0000","last_unstale":"2024-09-15T23:24:03.115275+0000","last_undegraded":"2024-09-15T23:24:03.115275+0000","last_fullsized":"2024-09-15T23:24:03.115275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:21.137333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109909+0000","last_change":"2024-09-15T23:21:52.530562+0000","last_active":"2024-09-15T23:24:03.109909+0000","last_peered":"2024-09-15T23:24:03.109909+0000","last_clean":"2024-09-15T23:24:03.109909+0000","last_became_active":"2024-09-15T23:21:52.530280+0000","last_became_peered":"2024-09-15T23:21:52.530280+0000","last_unstale":"2024-09-15T23:24:03.109909+0000","last_undegraded":"2024-09-15T23:24:03.109909+0000","last_fullsized":"2024-09-15T23:24:03.109909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:26.835651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113202+0000","last_change":"2024-09-15T23:21:50.513179+0000","last_active":"2024-09-15T23:24:03.113202+0000","last_peered":"2024-09-15T23:24:03.113202+0000","last_clean":"2024-09-15T23:24:03.113202+0000","last_became_active":"2024-09-15T23:21:50.512790+0000","last_became_peered":"2024-09-15T23:21:50.512790+0000","last_unstale":"2024-09-15T23:24:03.113202+0000","last_undegraded":"2024-09-15T23:24:03.113202+0000","last_fullsized":"2024-09-15T23:24:03.113202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:56.408198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113316+0000","last_change":"2024-09-15T23:23:02.382964+0000","last_active":"2024-09-15T23:24:03.113316+0000","last_peered":"2024-09-15T23:24:03.113316+0000","last_clean":"2024-09-15T23:24:03.113316+0000","last_became_active":"2024-09-15T23:23:02.382556+0000","last_became_peered":"2024-09-15T23:23:02.382556+0000","last_unstale":"2024-09-15T23:24:03.113316+0000","last_undegraded":"2024-09-15T23:24:03.113316+0000","last_fullsized":"2024-09-15T23:24:03.113316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:49.211591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117484+0000","last_change":"2024-09-15T23:21:52.530970+0000","last_active":"2024-09-15T23:24:03.117484+0000","last_peered":"2024-09-15T23:24:03.117484+0000","last_clean":"2024-09-15T23:24:03.117484+0000","last_became_active":"2024-09-15T23:21:52.530636+0000","last_became_peered":"2024-09-15T23:21:52.530636+0000","last_unstale":"2024-09-15T23:24:03.117484+0000","last_undegraded":"2024-09-15T23:24:03.117484+0000","last_fullsized":"2024-09-15T23:24:03.117484+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:33.374658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'40","reported_seq":316,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.146278+0000","last_change":"2024-09-15T23:24:03.131527+0000","last_active":"2024-09-15T23:24:20.146278+0000","last_peered":"2024-09-15T23:24:20.146278+0000","last_clean":"2024-09-15T23:24:20.146278+0000","last_became_active":"2024-09-15T23:24:03.119053+0000","last_became_peered":"2024-09-15T23:24:03.119053+0000","last_unstale":"2024-09-15T23:24:20.146278+0000","last_undegraded":"2024-09-15T23:24:20.146278+0000","last_fullsized":"2024-09-15T23:24:20.146278+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:21.870435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68399,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":214,"num_read_kb":169,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":62325,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113841+0000","last_change":"2024-09-15T23:23:02.277457+0000","last_active":"2024-09-15T23:24:03.113841+0000","last_peered":"2024-09-15T23:24:03.113841+0000","last_clean":"2024-09-15T23:24:03.113841+0000","last_became_active":"2024-09-15T23:23:02.276960+0000","last_became_peered":"2024-09-15T23:23:02.276960+0000","last_unstale":"2024-09-15T23:24:03.113841+0000","last_undegraded":"2024-09-15T23:24:03.113841+0000","last_fullsized":"2024-09-15T23:24:03.113841+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:15:28.823804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117534+0000","last_change":"2024-09-15T23:21:52.530541+0000","last_active":"2024-09-15T23:24:03.117534+0000","last_peered":"2024-09-15T23:24:03.117534+0000","last_clean":"2024-09-15T23:24:03.117534+0000","last_became_active":"2024-09-15T23:21:52.530217+0000","last_became_peered":"2024-09-15T23:21:52.530217+0000","last_unstale":"2024-09-15T23:24:03.117534+0000","last_undegraded":"2024-09-15T23:24:03.117534+0000","last_fullsized":"2024-09-15T23:24:03.117534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:14.864563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118500+0000","last_change":"2024-09-15T23:21:50.562203+0000","last_active":"2024-09-15T23:24:03.118500+0000","last_peered":"2024-09-15T23:24:03.118500+0000","last_clean":"2024-09-15T23:24:03.118500+0000","last_became_active":"2024-09-15T23:21:50.561827+0000","last_became_peered":"2024-09-15T23:21:50.561827+0000","last_unstale":"2024-09-15T23:24:03.118500+0000","last_undegraded":"2024-09-15T23:24:03.118500+0000","last_fullsized":"2024-09-15T23:24:03.118500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:22.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111024+0000","last_change":"2024-09-15T23:21:59.561013+0000","last_active":"2024-09-15T23:24:03.111024+0000","last_peered":"2024-09-15T23:24:03.111024+0000","last_clean":"2024-09-15T23:24:03.111024+0000","last_became_active":"2024-09-15T23:21:59.560665+0000","last_became_peered":"2024-09-15T23:21:59.560665+0000","last_unstale":"2024-09-15T23:24:03.111024+0000","last_undegraded":"2024-09-15T23:24:03.111024+0000","last_fullsized":"2024-09-15T23:24:03.111024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:00.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194109+0000","last_change":"2024-09-15T23:21:52.529922+0000","last_active":"2024-09-15T23:24:03.194109+0000","last_peered":"2024-09-15T23:24:03.194109+0000","last_clean":"2024-09-15T23:24:03.194109+0000","last_became_active":"2024-09-15T23:21:52.529179+0000","last_became_peered":"2024-09-15T23:21:52.529179+0000","last_unstale":"2024-09-15T23:24:03.194109+0000","last_undegraded":"2024-09-15T23:24:03.194109+0000","last_fullsized":"2024-09-15T23:24:03.194109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.077658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113481+0000","last_change":"2024-09-15T23:21:50.514420+0000","last_active":"2024-09-15T23:24:03.113481+0000","last_peered":"2024-09-15T23:24:03.113481+0000","last_clean":"2024-09-15T23:24:03.113481+0000","last_became_active":"2024-09-15T23:21:50.514007+0000","last_became_peered":"2024-09-15T23:21:50.514007+0000","last_unstale":"2024-09-15T23:24:03.113481+0000","last_undegraded":"2024-09-15T23:24:03.113481+0000","last_fullsized":"2024-09-15T23:24:03.113481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:36.845674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193335+0000","last_change":"2024-09-15T23:21:59.565978+0000","last_active":"2024-09-15T23:24:03.193335+0000","last_peered":"2024-09-15T23:24:03.193335+0000","last_clean":"2024-09-15T23:24:03.193335+0000","last_became_active":"2024-09-15T23:21:59.565604+0000","last_became_peered":"2024-09-15T23:21:59.565604+0000","last_unstale":"2024-09-15T23:24:03.193335+0000","last_undegraded":"2024-09-15T23:24:03.193335+0000","last_fullsized":"2024-09-15T23:24:03.193335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:47.089504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118064+0000","last_change":"2024-09-15T23:21:52.543511+0000","last_active":"2024-09-15T23:24:03.118064+0000","last_peered":"2024-09-15T23:24:03.118064+0000","last_clean":"2024-09-15T23:24:03.118064+0000","last_became_active":"2024-09-15T23:21:52.543169+0000","last_became_peered":"2024-09-15T23:21:52.543169+0000","last_unstale":"2024-09-15T23:24:03.118064+0000","last_undegraded":"2024-09-15T23:24:03.118064+0000","last_fullsized":"2024-09-15T23:24:03.118064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:05.696772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":252,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.282085+0000","last_change":"2024-09-15T23:21:50.518425+0000","last_active":"2024-09-15T23:24:20.282085+0000","last_peered":"2024-09-15T23:24:20.282085+0000","last_clean":"2024-09-15T23:24:20.282085+0000","last_became_active":"2024-09-15T23:21:50.518052+0000","last_became_peered":"2024-09-15T23:21:50.518052+0000","last_unstale":"2024-09-15T23:24:20.282085+0000","last_undegraded":"2024-09-15T23:24:20.282085+0000","last_fullsized":"2024-09-15T23:24:20.282085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:02.243869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112035+0000","last_change":"2024-09-15T23:21:59.570310+0000","last_active":"2024-09-15T23:24:03.112035+0000","last_peered":"2024-09-15T23:24:03.112035+0000","last_clean":"2024-09-15T23:24:03.112035+0000","last_became_active":"2024-09-15T23:21:59.569909+0000","last_became_peered":"2024-09-15T23:21:59.569909+0000","last_unstale":"2024-09-15T23:24:03.112035+0000","last_undegraded":"2024-09-15T23:24:03.112035+0000","last_fullsized":"2024-09-15T23:24:03.112035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:20:27.515537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194191+0000","last_change":"2024-09-15T23:21:59.568208+0000","last_active":"2024-09-15T23:24:03.194191+0000","last_peered":"2024-09-15T23:24:03.194191+0000","last_clean":"2024-09-15T23:24:03.194191+0000","last_became_active":"2024-09-15T23:21:59.567858+0000","last_became_peered":"2024-09-15T23:21:59.567858+0000","last_unstale":"2024-09-15T23:24:03.194191+0000","last_undegraded":"2024-09-15T23:24:03.194191+0000","last_fullsized":"2024-09-15T23:24:03.194191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:34:42.895247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193532+0000","last_change":"2024-09-15T23:21:52.532679+0000","last_active":"2024-09-15T23:24:03.193532+0000","last_peered":"2024-09-15T23:24:03.193532+0000","last_clean":"2024-09-15T23:24:03.193532+0000","last_became_active":"2024-09-15T23:21:52.532345+0000","last_became_peered":"2024-09-15T23:21:52.532345+0000","last_unstale":"2024-09-15T23:24:03.193532+0000","last_undegraded":"2024-09-15T23:24:03.193532+0000","last_fullsized":"2024-09-15T23:24:03.193532+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:45.943653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118135+0000","last_change":"2024-09-15T23:21:50.510836+0000","last_active":"2024-09-15T23:24:03.118135+0000","last_peered":"2024-09-15T23:24:03.118135+0000","last_clean":"2024-09-15T23:24:03.118135+0000","last_became_active":"2024-09-15T23:21:50.510514+0000","last_became_peered":"2024-09-15T23:21:50.510514+0000","last_unstale":"2024-09-15T23:24:03.118135+0000","last_undegraded":"2024-09-15T23:24:03.118135+0000","last_fullsized":"2024-09-15T23:24:03.118135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:06.404612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116423+0000","last_change":"2024-09-15T23:23:02.383436+0000","last_active":"2024-09-15T23:24:03.116423+0000","last_peered":"2024-09-15T23:24:03.116423+0000","last_clean":"2024-09-15T23:24:03.116423+0000","last_became_active":"2024-09-15T23:23:02.382982+0000","last_became_peered":"2024-09-15T23:23:02.382982+0000","last_unstale":"2024-09-15T23:24:03.116423+0000","last_undegraded":"2024-09-15T23:24:03.116423+0000","last_fullsized":"2024-09-15T23:24:03.116423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:55:14.076604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111923+0000","last_change":"2024-09-15T23:21:52.526766+0000","last_active":"2024-09-15T23:24:03.111923+0000","last_peered":"2024-09-15T23:24:03.111923+0000","last_clean":"2024-09-15T23:24:03.111923+0000","last_became_active":"2024-09-15T23:21:52.526446+0000","last_became_peered":"2024-09-15T23:21:52.526446+0000","last_unstale":"2024-09-15T23:24:03.111923+0000","last_undegraded":"2024-09-15T23:24:03.111923+0000","last_fullsized":"2024-09-15T23:24:03.111923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:21:37.004730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109384+0000","last_change":"2024-09-15T23:21:50.512928+0000","last_active":"2024-09-15T23:24:03.109384+0000","last_peered":"2024-09-15T23:24:03.109384+0000","last_clean":"2024-09-15T23:24:03.109384+0000","last_became_active":"2024-09-15T23:21:50.512558+0000","last_became_peered":"2024-09-15T23:21:50.512558+0000","last_unstale":"2024-09-15T23:24:03.109384+0000","last_undegraded":"2024-09-15T23:24:03.109384+0000","last_fullsized":"2024-09-15T23:24:03.109384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:08.653084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114309+0000","last_change":"2024-09-15T23:21:50.516920+0000","last_active":"2024-09-15T23:24:03.114309+0000","last_peered":"2024-09-15T23:24:03.114309+0000","last_clean":"2024-09-15T23:24:03.114309+0000","last_became_active":"2024-09-15T23:21:50.516573+0000","last_became_peered":"2024-09-15T23:21:50.516573+0000","last_unstale":"2024-09-15T23:24:03.114309+0000","last_undegraded":"2024-09-15T23:24:03.114309+0000","last_fullsized":"2024-09-15T23:24:03.114309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.367903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117266+0000","last_change":"2024-09-15T23:21:52.536696+0000","last_active":"2024-09-15T23:24:03.117266+0000","last_peered":"2024-09-15T23:24:03.117266+0000","last_clean":"2024-09-15T23:24:03.117266+0000","last_became_active":"2024-09-15T23:21:52.536338+0000","last_became_peered":"2024-09-15T23:21:52.536338+0000","last_unstale":"2024-09-15T23:24:03.117266+0000","last_undegraded":"2024-09-15T23:24:03.117266+0000","last_fullsized":"2024-09-15T23:24:03.117266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:05.152749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112825+0000","last_change":"2024-09-15T23:21:59.562968+0000","last_active":"2024-09-15T23:24:03.112825+0000","last_peered":"2024-09-15T23:24:03.112825+0000","last_clean":"2024-09-15T23:24:03.112825+0000","last_became_active":"2024-09-15T23:21:59.562579+0000","last_became_peered":"2024-09-15T23:21:59.562579+0000","last_unstale":"2024-09-15T23:24:03.112825+0000","last_undegraded":"2024-09-15T23:24:03.112825+0000","last_fullsized":"2024-09-15T23:24:03.112825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:02.853057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:21.284870+0000","last_change":"2024-09-15T23:21:50.513944+0000","last_active":"2024-09-15T23:24:21.284870+0000","last_peered":"2024-09-15T23:24:21.284870+0000","last_clean":"2024-09-15T23:24:21.284870+0000","last_became_active":"2024-09-15T23:21:50.513484+0000","last_became_peered":"2024-09-15T23:21:50.513484+0000","last_unstale":"2024-09-15T23:24:21.284870+0000","last_undegraded":"2024-09-15T23:24:21.284870+0000","last_fullsized":"2024-09-15T23:24:21.284870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:35.719344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118061+0000","last_change":"2024-09-15T23:21:52.539467+0000","last_active":"2024-09-15T23:24:03.118061+0000","last_peered":"2024-09-15T23:24:03.118061+0000","last_clean":"2024-09-15T23:24:03.118061+0000","last_became_active":"2024-09-15T23:21:52.538998+0000","last_became_peered":"2024-09-15T23:21:52.538998+0000","last_unstale":"2024-09-15T23:24:03.118061+0000","last_undegraded":"2024-09-15T23:24:03.118061+0000","last_fullsized":"2024-09-15T23:24:03.118061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:22.213857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112854+0000","last_change":"2024-09-15T23:21:59.564842+0000","last_active":"2024-09-15T23:24:03.112854+0000","last_peered":"2024-09-15T23:24:03.112854+0000","last_clean":"2024-09-15T23:24:03.112854+0000","last_became_active":"2024-09-15T23:21:59.564408+0000","last_became_peered":"2024-09-15T23:21:59.564408+0000","last_unstale":"2024-09-15T23:24:03.112854+0000","last_undegraded":"2024-09-15T23:24:03.112854+0000","last_fullsized":"2024-09-15T23:24:03.112854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:17.909020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115745+0000","last_change":"2024-09-15T23:23:02.149420+0000","last_active":"2024-09-15T23:24:03.115745+0000","last_peered":"2024-09-15T23:24:03.115745+0000","last_clean":"2024-09-15T23:24:03.115745+0000","last_became_active":"2024-09-15T23:23:02.149003+0000","last_became_peered":"2024-09-15T23:23:02.149003+0000","last_unstale":"2024-09-15T23:24:03.115745+0000","last_undegraded":"2024-09-15T23:24:03.115745+0000","last_fullsized":"2024-09-15T23:24:03.115745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:45.490322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.532769+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532426+0000","last_became_peered":"2024-09-15T23:21:52.532426+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:09.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.533148+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532823+0000","last_became_peered":"2024-09-15T23:21:52.532823+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:01.833945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193803+0000","last_change":"2024-09-15T23:21:52.536790+0000","last_active":"2024-09-15T23:24:03.193803+0000","last_peered":"2024-09-15T23:24:03.193803+0000","last_clean":"2024-09-15T23:24:03.193803+0000","last_became_active":"2024-09-15T23:21:52.536515+0000","last_became_peered":"2024-09-15T23:21:52.536515+0000","last_unstale":"2024-09-15T23:24:03.193803+0000","last_undegraded":"2024-09-15T23:24:03.193803+0000","last_fullsized":"2024-09-15T23:24:03.193803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:37.207759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109800+0000","last_change":"2024-09-15T23:23:02.382842+0000","last_active":"2024-09-15T23:24:03.109800+0000","last_peered":"2024-09-15T23:24:03.109800+0000","last_clean":"2024-09-15T23:24:03.109800+0000","last_became_active":"2024-09-15T23:23:02.382455+0000","last_became_peered":"2024-09-15T23:23:02.382455+0000","last_unstale":"2024-09-15T23:24:03.109800+0000","last_undegraded":"2024-09-15T23:24:03.109800+0000","last_fullsized":"2024-09-15T23:24:03.109800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:23.617641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.540628+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.540308+0000","last_became_peered":"2024-09-15T23:21:52.540308+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:16.660637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113381+0000","last_change":"2024-09-15T23:21:52.533780+0000","last_active":"2024-09-15T23:24:03.113381+0000","last_peered":"2024-09-15T23:24:03.113381+0000","last_clean":"2024-09-15T23:24:03.113381+0000","last_became_active":"2024-09-15T23:21:52.533354+0000","last_became_peered":"2024-09-15T23:21:52.533354+0000","last_unstale":"2024-09-15T23:24:03.113381+0000","last_undegraded":"2024-09-15T23:24:03.113381+0000","last_fullsized":"2024-09-15T23:24:03.113381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.381059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112801+0000","last_change":"2024-09-15T23:21:52.536406+0000","last_active":"2024-09-15T23:24:03.112801+0000","last_peered":"2024-09-15T23:24:03.112801+0000","last_clean":"2024-09-15T23:24:03.112801+0000","last_became_active":"2024-09-15T23:21:52.536091+0000","last_became_peered":"2024-09-15T23:21:52.536091+0000","last_unstale":"2024-09-15T23:24:03.112801+0000","last_undegraded":"2024-09-15T23:24:03.112801+0000","last_fullsized":"2024-09-15T23:24:03.112801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:30.452201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117879+0000","last_change":"2024-09-15T23:21:52.536333+0000","last_active":"2024-09-15T23:24:03.117879+0000","last_peered":"2024-09-15T23:24:03.117879+0000","last_clean":"2024-09-15T23:24:03.117879+0000","last_became_active":"2024-09-15T23:21:52.534677+0000","last_became_peered":"2024-09-15T23:21:52.534677+0000","last_unstale":"2024-09-15T23:24:03.117879+0000","last_undegraded":"2024-09-15T23:24:03.117879+0000","last_fullsized":"2024-09-15T23:24:03.117879+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:05.287914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112116+0000","last_change":"2024-09-15T23:21:52.536442+0000","last_active":"2024-09-15T23:24:03.112116+0000","last_peered":"2024-09-15T23:24:03.112116+0000","last_clean":"2024-09-15T23:24:03.112116+0000","last_became_active":"2024-09-15T23:21:52.536183+0000","last_became_peered":"2024-09-15T23:21:52.536183+0000","last_unstale":"2024-09-15T23:24:03.112116+0000","last_undegraded":"2024-09-15T23:24:03.112116+0000","last_fullsized":"2024-09-15T23:24:03.112116+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:41.641370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117398+0000","last_change":"2024-09-15T23:23:02.277365+0000","last_active":"2024-09-15T23:24:03.117398+0000","last_peered":"2024-09-15T23:24:03.117398+0000","last_clean":"2024-09-15T23:24:03.117398+0000","last_became_active":"2024-09-15T23:23:02.276905+0000","last_became_peered":"2024-09-15T23:23:02.276905+0000","last_unstale":"2024-09-15T23:24:03.117398+0000","last_undegraded":"2024-09-15T23:24:03.117398+0000","last_fullsized":"2024-09-15T23:24:03.117398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:21.100657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109788+0000","last_change":"2024-09-15T23:21:52.531171+0000","last_active":"2024-09-15T23:24:03.109788+0000","last_peered":"2024-09-15T23:24:03.109788+0000","last_clean":"2024-09-15T23:24:03.109788+0000","last_became_active":"2024-09-15T23:21:52.530724+0000","last_became_peered":"2024-09-15T23:21:52.530724+0000","last_unstale":"2024-09-15T23:24:03.109788+0000","last_undegraded":"2024-09-15T23:24:03.109788+0000","last_fullsized":"2024-09-15T23:24:03.109788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:11.009119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115400+0000","last_change":"2024-09-15T23:21:52.530433+0000","last_active":"2024-09-15T23:24:03.115400+0000","last_peered":"2024-09-15T23:24:03.115400+0000","last_clean":"2024-09-15T23:24:03.115400+0000","last_became_active":"2024-09-15T23:21:52.530058+0000","last_became_peered":"2024-09-15T23:21:52.530058+0000","last_unstale":"2024-09-15T23:24:03.115400+0000","last_undegraded":"2024-09-15T23:24:03.115400+0000","last_fullsized":"2024-09-15T23:24:03.115400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:51.208820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113218+0000","last_change":"2024-09-15T23:21:52.534639+0000","last_active":"2024-09-15T23:24:03.113218+0000","last_peered":"2024-09-15T23:24:03.113218+0000","last_clean":"2024-09-15T23:24:03.113218+0000","last_became_active":"2024-09-15T23:21:52.534362+0000","last_became_peered":"2024-09-15T23:21:52.534362+0000","last_unstale":"2024-09-15T23:24:03.113218+0000","last_undegraded":"2024-09-15T23:24:03.113218+0000","last_fullsized":"2024-09-15T23:24:03.113218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:58:12.023501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194630+0000","last_change":"2024-09-15T23:21:52.532758+0000","last_active":"2024-09-15T23:24:03.194630+0000","last_peered":"2024-09-15T23:24:03.194630+0000","last_clean":"2024-09-15T23:24:03.194630+0000","last_became_active":"2024-09-15T23:21:52.532412+0000","last_became_peered":"2024-09-15T23:21:52.532412+0000","last_unstale":"2024-09-15T23:24:03.194630+0000","last_undegraded":"2024-09-15T23:24:03.194630+0000","last_fullsized":"2024-09-15T23:24:03.194630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:46.285711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116737+0000","last_change":"2024-09-15T23:21:52.533928+0000","last_active":"2024-09-15T23:24:03.116737+0000","last_peered":"2024-09-15T23:24:03.116737+0000","last_clean":"2024-09-15T23:24:03.116737+0000","last_became_active":"2024-09-15T23:21:52.533578+0000","last_became_peered":"2024-09-15T23:21:52.533578+0000","last_unstale":"2024-09-15T23:24:03.116737+0000","last_undegraded":"2024-09-15T23:24:03.116737+0000","last_fullsized":"2024-09-15T23:24:03.116737+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:49.476316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116894+0000","last_change":"2024-09-15T23:21:52.533417+0000","last_active":"2024-09-15T23:24:03.116894+0000","last_peered":"2024-09-15T23:24:03.116894+0000","last_clean":"2024-09-15T23:24:03.116894+0000","last_became_active":"2024-09-15T23:21:52.533085+0000","last_became_peered":"2024-09-15T23:21:52.533085+0000","last_unstale":"2024-09-15T23:24:03.116894+0000","last_undegraded":"2024-09-15T23:24:03.116894+0000","last_fullsized":"2024-09-15T23:24:03.116894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:20.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193763+0000","last_change":"2024-09-15T23:21:52.542340+0000","last_active":"2024-09-15T23:24:03.193763+0000","last_peered":"2024-09-15T23:24:03.193763+0000","last_clean":"2024-09-15T23:24:03.193763+0000","last_became_active":"2024-09-15T23:21:52.542048+0000","last_became_peered":"2024-09-15T23:21:52.542048+0000","last_unstale":"2024-09-15T23:24:03.193763+0000","last_undegraded":"2024-09-15T23:24:03.193763+0000","last_fullsized":"2024-09-15T23:24:03.193763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:57.557420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193941+0000","last_change":"2024-09-15T23:21:52.535576+0000","last_active":"2024-09-15T23:24:03.193941+0000","last_peered":"2024-09-15T23:24:03.193941+0000","last_clean":"2024-09-15T23:24:03.193941+0000","last_became_active":"2024-09-15T23:21:52.535223+0000","last_became_peered":"2024-09-15T23:21:52.535223+0000","last_unstale":"2024-09-15T23:24:03.193941+0000","last_undegraded":"2024-09-15T23:24:03.193941+0000","last_fullsized":"2024-09-15T23:24:03.193941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:45.874927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109691+0000","last_change":"2024-09-15T23:23:02.150556+0000","last_active":"2024-09-15T23:24:03.109691+0000","last_peered":"2024-09-15T23:24:03.109691+0000","last_clean":"2024-09-15T23:24:03.109691+0000","last_became_active":"2024-09-15T23:23:02.150253+0000","last_became_peered":"2024-09-15T23:23:02.150253+0000","last_unstale":"2024-09-15T23:24:03.109691+0000","last_undegraded":"2024-09-15T23:24:03.109691+0000","last_fullsized":"2024-09-15T23:24:03.109691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:47.950356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110875+0000","last_change":"2024-09-15T23:21:52.537012+0000","last_active":"2024-09-15T23:24:03.110875+0000","last_peered":"2024-09-15T23:24:03.110875+0000","last_clean":"2024-09-15T23:24:03.110875+0000","last_became_active":"2024-09-15T23:21:52.536245+0000","last_became_peered":"2024-09-15T23:21:52.536245+0000","last_unstale":"2024-09-15T23:24:03.110875+0000","last_undegraded":"2024-09-15T23:24:03.110875+0000","last_fullsized":"2024-09-15T23:24:03.110875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:31.150491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114034+0000","last_change":"2024-09-15T23:21:52.540431+0000","last_active":"2024-09-15T23:24:03.114034+0000","last_peered":"2024-09-15T23:24:03.114034+0000","last_clean":"2024-09-15T23:24:03.114034+0000","last_became_active":"2024-09-15T23:21:52.540104+0000","last_became_peered":"2024-09-15T23:21:52.540104+0000","last_unstale":"2024-09-15T23:24:03.114034+0000","last_undegraded":"2024-09-15T23:24:03.114034+0000","last_fullsized":"2024-09-15T23:24:03.114034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.550258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114876+0000","last_change":"2024-09-15T23:21:52.537590+0000","last_active":"2024-09-15T23:24:03.114876+0000","last_peered":"2024-09-15T23:24:03.114876+0000","last_clean":"2024-09-15T23:24:03.114876+0000","last_became_active":"2024-09-15T23:21:52.533804+0000","last_became_peered":"2024-09-15T23:21:52.533804+0000","last_unstale":"2024-09-15T23:24:03.114876+0000","last_undegraded":"2024-09-15T23:24:03.114876+0000","last_fullsized":"2024-09-15T23:24:03.114876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:48.449090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117952+0000","last_change":"2024-09-15T23:21:52.536130+0000","last_active":"2024-09-15T23:24:03.117952+0000","last_peered":"2024-09-15T23:24:03.117952+0000","last_clean":"2024-09-15T23:24:03.117952+0000","last_became_active":"2024-09-15T23:21:52.535336+0000","last_became_peered":"2024-09-15T23:21:52.535336+0000","last_unstale":"2024-09-15T23:24:03.117952+0000","last_undegraded":"2024-09-15T23:24:03.117952+0000","last_fullsized":"2024-09-15T23:24:03.117952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:52.581994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111827+0000","last_change":"2024-09-15T23:21:52.540485+0000","last_active":"2024-09-15T23:24:03.111827+0000","last_peered":"2024-09-15T23:24:03.111827+0000","last_clean":"2024-09-15T23:24:03.111827+0000","last_became_active":"2024-09-15T23:21:52.540178+0000","last_became_peered":"2024-09-15T23:21:52.540178+0000","last_unstale":"2024-09-15T23:24:03.111827+0000","last_undegraded":"2024-09-15T23:24:03.111827+0000","last_fullsized":"2024-09-15T23:24:03.111827+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:40:22.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113883+0000","last_change":"2024-09-15T23:21:52.538845+0000","last_active":"2024-09-15T23:24:03.113883+0000","last_peered":"2024-09-15T23:24:03.113883+0000","last_clean":"2024-09-15T23:24:03.113883+0000","last_became_active":"2024-09-15T23:21:52.538511+0000","last_became_peered":"2024-09-15T23:21:52.538511+0000","last_unstale":"2024-09-15T23:24:03.113883+0000","last_undegraded":"2024-09-15T23:24:03.113883+0000","last_fullsized":"2024-09-15T23:24:03.113883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:00.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114107+0000","last_change":"2024-09-15T23:23:02.149590+0000","last_active":"2024-09-15T23:24:03.114107+0000","last_peered":"2024-09-15T23:24:03.114107+0000","last_clean":"2024-09-15T23:24:03.114107+0000","last_became_active":"2024-09-15T23:23:02.149184+0000","last_became_peered":"2024-09-15T23:23:02.149184+0000","last_unstale":"2024-09-15T23:24:03.114107+0000","last_undegraded":"2024-09-15T23:24:03.114107+0000","last_fullsized":"2024-09-15T23:24:03.114107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:05:29.407175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117530+0000","last_change":"2024-09-15T23:21:52.530524+0000","last_active":"2024-09-15T23:24:03.117530+0000","last_peered":"2024-09-15T23:24:03.117530+0000","last_clean":"2024-09-15T23:24:03.117530+0000","last_became_active":"2024-09-15T23:21:52.530203+0000","last_became_peered":"2024-09-15T23:21:52.530203+0000","last_unstale":"2024-09-15T23:24:03.117530+0000","last_undegraded":"2024-09-15T23:24:03.117530+0000","last_fullsized":"2024-09-15T23:24:03.117530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:50.560211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110690+0000","last_change":"2024-09-15T23:21:52.537256+0000","last_active":"2024-09-15T23:24:03.110690+0000","last_peered":"2024-09-15T23:24:03.110690+0000","last_clean":"2024-09-15T23:24:03.110690+0000","last_became_active":"2024-09-15T23:21:52.536731+0000","last_became_peered":"2024-09-15T23:21:52.536731+0000","last_unstale":"2024-09-15T23:24:03.110690+0000","last_undegraded":"2024-09-15T23:24:03.110690+0000","last_fullsized":"2024-09-15T23:24:03.110690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:18.815398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194364+0000","last_change":"2024-09-15T23:21:52.537728+0000","last_active":"2024-09-15T23:24:03.194364+0000","last_peered":"2024-09-15T23:24:03.194364+0000","last_clean":"2024-09-15T23:24:03.194364+0000","last_became_active":"2024-09-15T23:21:52.537434+0000","last_became_peered":"2024-09-15T23:21:52.537434+0000","last_unstale":"2024-09-15T23:24:03.194364+0000","last_undegraded":"2024-09-15T23:24:03.194364+0000","last_fullsized":"2024-09-15T23:24:03.194364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:25.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194218+0000","last_change":"2024-09-15T23:21:52.534780+0000","last_active":"2024-09-15T23:24:03.194218+0000","last_peered":"2024-09-15T23:24:03.194218+0000","last_clean":"2024-09-15T23:24:03.194218+0000","last_became_active":"2024-09-15T23:21:52.534399+0000","last_became_peered":"2024-09-15T23:21:52.534399+0000","last_unstale":"2024-09-15T23:24:03.194218+0000","last_undegraded":"2024-09-15T23:24:03.194218+0000","last_fullsized":"2024-09-15T23:24:03.194218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:47.344129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116592+0000","last_change":"2024-09-15T23:21:52.533403+0000","last_active":"2024-09-15T23:24:03.116592+0000","last_peered":"2024-09-15T23:24:03.116592+0000","last_clean":"2024-09-15T23:24:03.116592+0000","last_became_active":"2024-09-15T23:21:52.533069+0000","last_became_peered":"2024-09-15T23:21:52.533069+0000","last_unstale":"2024-09-15T23:24:03.116592+0000","last_undegraded":"2024-09-15T23:24:03.116592+0000","last_fullsized":"2024-09-15T23:24:03.116592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:38.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76056,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1071,"num_read_kb":611,"num_write":263,"num_write_kb":158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":63447,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":222402,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":246033,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579948,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":521776,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841584,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1159168,"data_stored":1176403,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005109,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1032192,"data_stored":524287,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234043,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":520569,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462976,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1073152,"data_stored":1116395,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626501,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1003520,"data_stored":515695,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855434,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1112565,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018959,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":582936,"data_compressed":39742,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247891,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1015808,"data_stored":515403,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444121,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1019904,"data_stored":517253,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607646,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":579358,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1028096,"data_stored":517151,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":70026,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":71062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:24:23.953 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T23:24:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/95427016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cb41025e0 msgr2=0x7f3cb4108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 --2- 172.21.15.89:0/95427016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cb41025e0 0x7f3cb4108e70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009ca0 tx=0x7f3cac02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/95427016 shutdown_connections 2024-09-15T23:24:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 --2- 172.21.15.89:0/95427016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cb41025e0 0x7f3cb4108e70 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 --2- 172.21.15.89:0/95427016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cb4101c20 0x7f3cb41020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 --2- 172.21.15.89:0/95427016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cb4100a30 0x7f3cb4100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/95427016 >> 172.21.15.89:0/95427016 conn(0x7f3cb40fc190 msgr2=0x7f3cb40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/95427016 shutdown_connections 2024-09-15T23:24:24.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/95427016 wait complete. 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 Processor -- start 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.086+0000 7f3cbbf9f640 1 -- start start 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cbbf9f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cb4100a30 0x7f3cb410b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cbbf9f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cb4101c20 0x7f3cb41099f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cbbf9f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cb41025e0 0x7f3cb4109f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cbbf9f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb410d800 con 0x7f3cb4101c20 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cbbf9f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb410d680 con 0x7f3cb4100a30 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cbbf9f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb410d980 con 0x7f3cb41025e0 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb953a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cb4101c20 0x7f3cb41099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb953a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cb4101c20 0x7f3cb41099f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35838/0 (socket says 172.21.15.89:35838) 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb953a640 1 -- 172.21.15.89:0/2894639184 learned_addr learned my addr 172.21.15.89:0/2894639184 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cba53c640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cb41025e0 0x7f3cb4109f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb953a640 1 -- 172.21.15.89:0/2894639184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cb41025e0 msgr2=0x7f3cb4109f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb9d3b640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cb4100a30 0x7f3cb410b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb953a640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cb41025e0 0x7f3cb4109f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb953a640 1 -- 172.21.15.89:0/2894639184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cb4100a30 msgr2=0x7f3cb410b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb953a640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cb4100a30 0x7f3cb410b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb953a640 1 -- 172.21.15.89:0/2894639184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb410a7c0 con 0x7f3cb4101c20 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb9d3b640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cb4100a30 0x7f3cb410b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cb953a640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cb4101c20 0x7f3cb41099f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800d770 tx=0x7f3ca800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3ca6ffd640 1 -- 172.21.15.89:0/2894639184 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8017070 con 0x7f3cb4101c20 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3ca6ffd640 1 -- 172.21.15.89:0/2894639184 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ca8005500 con 0x7f3cb4101c20 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3ca6ffd640 1 -- 172.21.15.89:0/2894639184 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8004e00 con 0x7f3cb4101c20 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/2894639184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb41a1f60 con 0x7f3cb4101c20 2024-09-15T23:24:24.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.087+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/2894639184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb41a2370 con 0x7f3cb4101c20 2024-09-15T23:24:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.088+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/2894639184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb4107e20 con 0x7f3cb4101c20 2024-09-15T23:24:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.090+0000 7f3ca6ffd640 1 -- 172.21.15.89:0/2894639184 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f3ca8004fa0 con 0x7f3cb4101c20 2024-09-15T23:24:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.091+0000 7f3ca6ffd640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3ca00a3360 0x7f3ca00a5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.091+0000 7f3cb9d3b640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3ca00a3360 0x7f3ca00a5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.091+0000 7f3ca6ffd640 1 -- 172.21.15.89:0/2894639184 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3ca80cfc80 con 0x7f3cb4101c20 2024-09-15T23:24:24.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.091+0000 7f3cb9d3b640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3ca00a3360 0x7f3ca00a5820 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4101a80 tx=0x7f3cb0005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:24.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.094+0000 7f3ca6ffd640 1 -- 172.21.15.89:0/2894639184 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3ca809c4d0 con 0x7f3cb4101c20 2024-09-15T23:24:24.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.258+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/2894639184 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3cb4062630 con 0x7f3ca00a3360 2024-09-15T23:24:24.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.266+0000 7f3ca6ffd640 1 -- 172.21.15.89:0/2894639184 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340821 (secure 0 0 0) 0x7f3cb4062630 con 0x7f3ca00a3360 2024-09-15T23:24:24.271 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:24.272 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-15T23:24:24.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/2894639184 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3ca00a3360 msgr2=0x7f3ca00a5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3ca00a3360 0x7f3ca00a5820 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4101a80 tx=0x7f3cb0005d30 comp rx=0 tx=0).stop 2024-09-15T23:24:24.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/2894639184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cb4101c20 msgr2=0x7f3cb41099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cb4101c20 0x7f3cb41099f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3ca800d770 tx=0x7f3ca800dc30 comp rx=0 tx=0).stop 2024-09-15T23:24:24.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/2894639184 shutdown_connections 2024-09-15T23:24:24.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3ca00a3360 0x7f3ca00a5820 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3cb41025e0 0x7f3cb4109f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cb4101c20 0x7f3cb41099f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 --2- 172.21.15.89:0/2894639184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3cb4100a30 0x7f3cb410b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/2894639184 >> 172.21.15.89:0/2894639184 conn(0x7f3cb40fc190 msgr2=0x7f3cb4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:24.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/2894639184 shutdown_connections 2024-09-15T23:24:24.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.270+0000 7f3cbbf9f640 1 -- 172.21.15.89:0/2894639184 wait complete. 2024-09-15T23:24:24.280 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:24 smithi089 ceph-mon[30933]: pgmap v361: 129 pgs: 129 active+clean; 651 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-09-15T23:24:24.282 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-15T23:24:23.021695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666658,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1197,"num_read_kb":720,"num_write":446,"num_write_kb":3098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1244183,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369880,"kb_used_data":12264,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571944,"statfs":{"total":1151940427776,"available":1151561670656,"internally_reserved":0,"allocated":12558336,"data_stored":8199791,"data_compressed":471759,"data_compressed_allocated":638976,"data_compressed_original":2813952,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6775,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":137,"num_read_kb":72,"num_write":9,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003289"},"pg_stats":[{"pgid":"2.1f","version":"86'13","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:16.275939+0000","last_change":"2024-09-15T23:21:50.513610+0000","last_active":"2024-09-15T23:24:16.275939+0000","last_peered":"2024-09-15T23:24:16.275939+0000","last_clean":"2024-09-15T23:24:16.275939+0000","last_became_active":"2024-09-15T23:21:50.513266+0000","last_became_peered":"2024-09-15T23:21:50.513266+0000","last_unstale":"2024-09-15T23:24:16.275939+0000","last_undegraded":"2024-09-15T23:24:16.275939+0000","last_fullsized":"2024-09-15T23:24:16.275939+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:16:48.692801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112578+0000","last_change":"2024-09-15T23:21:52.540248+0000","last_active":"2024-09-15T23:24:03.112578+0000","last_peered":"2024-09-15T23:24:03.112578+0000","last_clean":"2024-09-15T23:24:03.112578+0000","last_became_active":"2024-09-15T23:21:52.539945+0000","last_became_peered":"2024-09-15T23:21:52.539945+0000","last_unstale":"2024-09-15T23:24:03.112578+0000","last_undegraded":"2024-09-15T23:24:03.112578+0000","last_fullsized":"2024-09-15T23:24:03.112578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:33.402153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112360+0000","last_change":"2024-09-15T23:21:59.567246+0000","last_active":"2024-09-15T23:24:03.112360+0000","last_peered":"2024-09-15T23:24:03.112360+0000","last_clean":"2024-09-15T23:24:03.112360+0000","last_became_active":"2024-09-15T23:21:59.566832+0000","last_became_peered":"2024-09-15T23:21:59.566832+0000","last_unstale":"2024-09-15T23:24:03.112360+0000","last_undegraded":"2024-09-15T23:24:03.112360+0000","last_fullsized":"2024-09-15T23:24:03.112360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:19.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112747+0000","last_change":"2024-09-15T23:21:50.561767+0000","last_active":"2024-09-15T23:24:03.112747+0000","last_peered":"2024-09-15T23:24:03.112747+0000","last_clean":"2024-09-15T23:24:03.112747+0000","last_became_active":"2024-09-15T23:21:50.561385+0000","last_became_peered":"2024-09-15T23:21:50.561385+0000","last_unstale":"2024-09-15T23:24:03.112747+0000","last_undegraded":"2024-09-15T23:24:03.112747+0000","last_fullsized":"2024-09-15T23:24:03.112747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:11.897884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194008+0000","last_change":"2024-09-15T23:21:52.534874+0000","last_active":"2024-09-15T23:24:03.194008+0000","last_peered":"2024-09-15T23:24:03.194008+0000","last_clean":"2024-09-15T23:24:03.194008+0000","last_became_active":"2024-09-15T23:21:52.534334+0000","last_became_peered":"2024-09-15T23:21:52.534334+0000","last_unstale":"2024-09-15T23:24:03.194008+0000","last_undegraded":"2024-09-15T23:24:03.194008+0000","last_fullsized":"2024-09-15T23:24:03.194008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:15.972648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110384+0000","last_change":"2024-09-15T23:21:59.559907+0000","last_active":"2024-09-15T23:24:03.110384+0000","last_peered":"2024-09-15T23:24:03.110384+0000","last_clean":"2024-09-15T23:24:03.110384+0000","last_became_active":"2024-09-15T23:21:59.559440+0000","last_became_peered":"2024-09-15T23:21:59.559440+0000","last_unstale":"2024-09-15T23:24:03.110384+0000","last_undegraded":"2024-09-15T23:24:03.110384+0000","last_fullsized":"2024-09-15T23:24:03.110384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:00.755112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117658+0000","last_change":"2024-09-15T23:21:50.515424+0000","last_active":"2024-09-15T23:24:03.117658+0000","last_peered":"2024-09-15T23:24:03.117658+0000","last_clean":"2024-09-15T23:24:03.117658+0000","last_became_active":"2024-09-15T23:21:50.514966+0000","last_became_peered":"2024-09-15T23:21:50.514966+0000","last_unstale":"2024-09-15T23:24:03.117658+0000","last_undegraded":"2024-09-15T23:24:03.117658+0000","last_fullsized":"2024-09-15T23:24:03.117658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:57.586575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117341+0000","last_change":"2024-09-15T23:21:52.535189+0000","last_active":"2024-09-15T23:24:03.117341+0000","last_peered":"2024-09-15T23:24:03.117341+0000","last_clean":"2024-09-15T23:24:03.117341+0000","last_became_active":"2024-09-15T23:21:52.534807+0000","last_became_peered":"2024-09-15T23:21:52.534807+0000","last_unstale":"2024-09-15T23:24:03.117341+0000","last_undegraded":"2024-09-15T23:24:03.117341+0000","last_fullsized":"2024-09-15T23:24:03.117341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:30.654382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110773+0000","last_change":"2024-09-15T23:21:59.564675+0000","last_active":"2024-09-15T23:24:03.110773+0000","last_peered":"2024-09-15T23:24:03.110773+0000","last_clean":"2024-09-15T23:24:03.110773+0000","last_became_active":"2024-09-15T23:21:59.564271+0000","last_became_peered":"2024-09-15T23:21:59.564271+0000","last_unstale":"2024-09-15T23:24:03.110773+0000","last_undegraded":"2024-09-15T23:24:03.110773+0000","last_fullsized":"2024-09-15T23:24:03.110773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:01.055369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117219+0000","last_change":"2024-09-15T23:21:50.507790+0000","last_active":"2024-09-15T23:24:03.117219+0000","last_peered":"2024-09-15T23:24:03.117219+0000","last_clean":"2024-09-15T23:24:03.117219+0000","last_became_active":"2024-09-15T23:21:50.507398+0000","last_became_peered":"2024-09-15T23:21:50.507398+0000","last_unstale":"2024-09-15T23:24:03.117219+0000","last_undegraded":"2024-09-15T23:24:03.117219+0000","last_fullsized":"2024-09-15T23:24:03.117219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:58:00.001279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116935+0000","last_change":"2024-09-15T23:21:52.537179+0000","last_active":"2024-09-15T23:24:03.116935+0000","last_peered":"2024-09-15T23:24:03.116935+0000","last_clean":"2024-09-15T23:24:03.116935+0000","last_became_active":"2024-09-15T23:21:52.536841+0000","last_became_peered":"2024-09-15T23:21:52.536841+0000","last_unstale":"2024-09-15T23:24:03.116935+0000","last_undegraded":"2024-09-15T23:24:03.116935+0000","last_fullsized":"2024-09-15T23:24:03.116935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:47.455700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110441+0000","last_change":"2024-09-15T23:21:59.564734+0000","last_active":"2024-09-15T23:24:03.110441+0000","last_peered":"2024-09-15T23:24:03.110441+0000","last_clean":"2024-09-15T23:24:03.110441+0000","last_became_active":"2024-09-15T23:21:59.564339+0000","last_became_peered":"2024-09-15T23:21:59.564339+0000","last_unstale":"2024-09-15T23:24:03.110441+0000","last_undegraded":"2024-09-15T23:24:03.110441+0000","last_fullsized":"2024-09-15T23:24:03.110441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:44.498480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113970+0000","last_change":"2024-09-15T23:21:52.538025+0000","last_active":"2024-09-15T23:24:03.113970+0000","last_peered":"2024-09-15T23:24:03.113970+0000","last_clean":"2024-09-15T23:24:03.113970+0000","last_became_active":"2024-09-15T23:21:52.537736+0000","last_became_peered":"2024-09-15T23:21:52.537736+0000","last_unstale":"2024-09-15T23:24:03.113970+0000","last_undegraded":"2024-09-15T23:24:03.113970+0000","last_fullsized":"2024-09-15T23:24:03.113970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:31.916888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.127260+0000","last_change":"2024-09-15T23:24:03.127260+0000","last_active":"2024-09-15T23:24:03.127260+0000","last_peered":"2024-09-15T23:24:03.127260+0000","last_clean":"2024-09-15T23:24:03.127260+0000","last_became_active":"2024-09-15T23:24:03.119384+0000","last_became_peered":"2024-09-15T23:24:03.119384+0000","last_unstale":"2024-09-15T23:24:03.127260+0000","last_undegraded":"2024-09-15T23:24:03.127260+0000","last_fullsized":"2024-09-15T23:24:03.127260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:35.325048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110625+0000","last_change":"2024-09-15T23:21:59.681971+0000","last_active":"2024-09-15T23:24:03.110625+0000","last_peered":"2024-09-15T23:24:03.110625+0000","last_clean":"2024-09-15T23:24:03.110625+0000","last_became_active":"2024-09-15T23:21:59.681432+0000","last_became_peered":"2024-09-15T23:21:59.681432+0000","last_unstale":"2024-09-15T23:24:03.110625+0000","last_undegraded":"2024-09-15T23:24:03.110625+0000","last_fullsized":"2024-09-15T23:24:03.110625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:26.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112179+0000","last_change":"2024-09-15T23:21:52.541281+0000","last_active":"2024-09-15T23:24:03.112179+0000","last_peered":"2024-09-15T23:24:03.112179+0000","last_clean":"2024-09-15T23:24:03.112179+0000","last_became_active":"2024-09-15T23:21:52.540772+0000","last_became_peered":"2024-09-15T23:21:52.540772+0000","last_unstale":"2024-09-15T23:24:03.112179+0000","last_undegraded":"2024-09-15T23:24:03.112179+0000","last_fullsized":"2024-09-15T23:24:03.112179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:23:44.444904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112700+0000","last_change":"2024-09-15T23:21:50.508844+0000","last_active":"2024-09-15T23:24:03.112700+0000","last_peered":"2024-09-15T23:24:03.112700+0000","last_clean":"2024-09-15T23:24:03.112700+0000","last_became_active":"2024-09-15T23:21:50.508364+0000","last_became_peered":"2024-09-15T23:21:50.508364+0000","last_unstale":"2024-09-15T23:24:03.112700+0000","last_undegraded":"2024-09-15T23:24:03.112700+0000","last_fullsized":"2024-09-15T23:24:03.112700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:45.129711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110720+0000","last_change":"2024-09-15T23:21:59.562114+0000","last_active":"2024-09-15T23:24:03.110720+0000","last_peered":"2024-09-15T23:24:03.110720+0000","last_clean":"2024-09-15T23:24:03.110720+0000","last_became_active":"2024-09-15T23:21:59.561652+0000","last_became_peered":"2024-09-15T23:21:59.561652+0000","last_unstale":"2024-09-15T23:24:03.110720+0000","last_undegraded":"2024-09-15T23:24:03.110720+0000","last_fullsized":"2024-09-15T23:24:03.110720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:51.781886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.531678+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.531327+0000","last_became_peered":"2024-09-15T23:21:52.531327+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:07.152626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112193+0000","last_change":"2024-09-15T23:21:50.513197+0000","last_active":"2024-09-15T23:24:03.112193+0000","last_peered":"2024-09-15T23:24:03.112193+0000","last_clean":"2024-09-15T23:24:03.112193+0000","last_became_active":"2024-09-15T23:21:50.512794+0000","last_became_peered":"2024-09-15T23:21:50.512794+0000","last_unstale":"2024-09-15T23:24:03.112193+0000","last_undegraded":"2024-09-15T23:24:03.112193+0000","last_fullsized":"2024-09-15T23:24:03.112193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:05.457467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194628+0000","last_change":"2024-09-15T23:21:59.565669+0000","last_active":"2024-09-15T23:24:03.194628+0000","last_peered":"2024-09-15T23:24:03.194628+0000","last_clean":"2024-09-15T23:24:03.194628+0000","last_became_active":"2024-09-15T23:21:59.565290+0000","last_became_peered":"2024-09-15T23:21:59.565290+0000","last_unstale":"2024-09-15T23:24:03.194628+0000","last_undegraded":"2024-09-15T23:24:03.194628+0000","last_fullsized":"2024-09-15T23:24:03.194628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:04.861560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116394+0000","last_change":"2024-09-15T23:21:52.529643+0000","last_active":"2024-09-15T23:24:03.116394+0000","last_peered":"2024-09-15T23:24:03.116394+0000","last_clean":"2024-09-15T23:24:03.116394+0000","last_became_active":"2024-09-15T23:21:52.529304+0000","last_became_peered":"2024-09-15T23:21:52.529304+0000","last_unstale":"2024-09-15T23:24:03.116394+0000","last_undegraded":"2024-09-15T23:24:03.116394+0000","last_fullsized":"2024-09-15T23:24:03.116394+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:18:37.758403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117052+0000","last_change":"2024-09-15T23:21:50.562114+0000","last_active":"2024-09-15T23:24:03.117052+0000","last_peered":"2024-09-15T23:24:03.117052+0000","last_clean":"2024-09-15T23:24:03.117052+0000","last_became_active":"2024-09-15T23:21:50.561772+0000","last_became_peered":"2024-09-15T23:21:50.561772+0000","last_unstale":"2024-09-15T23:24:03.117052+0000","last_undegraded":"2024-09-15T23:24:03.117052+0000","last_fullsized":"2024-09-15T23:24:03.117052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:14.806646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110672+0000","last_change":"2024-09-15T23:21:59.681933+0000","last_active":"2024-09-15T23:24:03.110672+0000","last_peered":"2024-09-15T23:24:03.110672+0000","last_clean":"2024-09-15T23:24:03.110672+0000","last_became_active":"2024-09-15T23:21:59.681528+0000","last_became_peered":"2024-09-15T23:21:59.681528+0000","last_unstale":"2024-09-15T23:24:03.110672+0000","last_undegraded":"2024-09-15T23:24:03.110672+0000","last_fullsized":"2024-09-15T23:24:03.110672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:46.199540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194326+0000","last_change":"2024-09-15T23:21:52.529548+0000","last_active":"2024-09-15T23:24:03.194326+0000","last_peered":"2024-09-15T23:24:03.194326+0000","last_clean":"2024-09-15T23:24:03.194326+0000","last_became_active":"2024-09-15T23:21:52.529253+0000","last_became_peered":"2024-09-15T23:21:52.529253+0000","last_unstale":"2024-09-15T23:24:03.194326+0000","last_undegraded":"2024-09-15T23:24:03.194326+0000","last_fullsized":"2024-09-15T23:24:03.194326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:22.813395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111022+0000","last_change":"2024-09-15T23:21:50.767759+0000","last_active":"2024-09-15T23:24:03.111022+0000","last_peered":"2024-09-15T23:24:03.111022+0000","last_clean":"2024-09-15T23:24:03.111022+0000","last_became_active":"2024-09-15T23:21:50.767388+0000","last_became_peered":"2024-09-15T23:21:50.767388+0000","last_unstale":"2024-09-15T23:24:03.111022+0000","last_undegraded":"2024-09-15T23:24:03.111022+0000","last_fullsized":"2024-09-15T23:24:03.111022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:36.713769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112512+0000","last_change":"2024-09-15T23:21:59.557951+0000","last_active":"2024-09-15T23:24:03.112512+0000","last_peered":"2024-09-15T23:24:03.112512+0000","last_clean":"2024-09-15T23:24:03.112512+0000","last_became_active":"2024-09-15T23:21:59.557490+0000","last_became_peered":"2024-09-15T23:21:59.557490+0000","last_unstale":"2024-09-15T23:24:03.112512+0000","last_undegraded":"2024-09-15T23:24:03.112512+0000","last_fullsized":"2024-09-15T23:24:03.112512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:17.147389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112859+0000","last_change":"2024-09-15T23:21:52.531638+0000","last_active":"2024-09-15T23:24:03.112859+0000","last_peered":"2024-09-15T23:24:03.112859+0000","last_clean":"2024-09-15T23:24:03.112859+0000","last_became_active":"2024-09-15T23:21:52.531301+0000","last_became_peered":"2024-09-15T23:21:52.531301+0000","last_unstale":"2024-09-15T23:24:03.112859+0000","last_undegraded":"2024-09-15T23:24:03.112859+0000","last_fullsized":"2024-09-15T23:24:03.112859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.436286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'16","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194603+0000","last_change":"2024-09-15T23:21:50.508470+0000","last_active":"2024-09-15T23:24:03.194603+0000","last_peered":"2024-09-15T23:24:03.194603+0000","last_clean":"2024-09-15T23:24:03.194603+0000","last_became_active":"2024-09-15T23:21:50.508060+0000","last_became_peered":"2024-09-15T23:21:50.508060+0000","last_unstale":"2024-09-15T23:24:03.194603+0000","last_undegraded":"2024-09-15T23:24:03.194603+0000","last_fullsized":"2024-09-15T23:24:03.194603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:59.542252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111992+0000","last_change":"2024-09-15T23:21:59.682019+0000","last_active":"2024-09-15T23:24:03.111992+0000","last_peered":"2024-09-15T23:24:03.111992+0000","last_clean":"2024-09-15T23:24:03.111992+0000","last_became_active":"2024-09-15T23:21:59.681467+0000","last_became_peered":"2024-09-15T23:21:59.681467+0000","last_unstale":"2024-09-15T23:24:03.111992+0000","last_undegraded":"2024-09-15T23:24:03.111992+0000","last_fullsized":"2024-09-15T23:24:03.111992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:27.650382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109766+0000","last_change":"2024-09-15T23:21:52.536345+0000","last_active":"2024-09-15T23:24:03.109766+0000","last_peered":"2024-09-15T23:24:03.109766+0000","last_clean":"2024-09-15T23:24:03.109766+0000","last_became_active":"2024-09-15T23:21:52.536027+0000","last_became_peered":"2024-09-15T23:21:52.536027+0000","last_unstale":"2024-09-15T23:24:03.109766+0000","last_undegraded":"2024-09-15T23:24:03.109766+0000","last_fullsized":"2024-09-15T23:24:03.109766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:16.435912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117716+0000","last_change":"2024-09-15T23:21:50.562945+0000","last_active":"2024-09-15T23:24:03.117716+0000","last_peered":"2024-09-15T23:24:03.117716+0000","last_clean":"2024-09-15T23:24:03.117716+0000","last_became_active":"2024-09-15T23:21:50.561632+0000","last_became_peered":"2024-09-15T23:21:50.561632+0000","last_unstale":"2024-09-15T23:24:03.117716+0000","last_undegraded":"2024-09-15T23:24:03.117716+0000","last_fullsized":"2024-09-15T23:24:03.117716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:44.545489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115268+0000","last_change":"2024-09-15T23:21:59.564793+0000","last_active":"2024-09-15T23:24:03.115268+0000","last_peered":"2024-09-15T23:24:03.115268+0000","last_clean":"2024-09-15T23:24:03.115268+0000","last_became_active":"2024-09-15T23:21:59.564140+0000","last_became_peered":"2024-09-15T23:21:59.564140+0000","last_unstale":"2024-09-15T23:24:03.115268+0000","last_undegraded":"2024-09-15T23:24:03.115268+0000","last_fullsized":"2024-09-15T23:24:03.115268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:41.464962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113451+0000","last_change":"2024-09-15T23:21:52.533521+0000","last_active":"2024-09-15T23:24:03.113451+0000","last_peered":"2024-09-15T23:24:03.113451+0000","last_clean":"2024-09-15T23:24:03.113451+0000","last_became_active":"2024-09-15T23:21:52.532195+0000","last_became_peered":"2024-09-15T23:21:52.532195+0000","last_unstale":"2024-09-15T23:24:03.113451+0000","last_undegraded":"2024-09-15T23:24:03.113451+0000","last_fullsized":"2024-09-15T23:24:03.113451+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:14.229018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194243+0000","last_change":"2024-09-15T23:21:50.562893+0000","last_active":"2024-09-15T23:24:03.194243+0000","last_peered":"2024-09-15T23:24:03.194243+0000","last_clean":"2024-09-15T23:24:03.194243+0000","last_became_active":"2024-09-15T23:21:50.562516+0000","last_became_peered":"2024-09-15T23:21:50.562516+0000","last_unstale":"2024-09-15T23:24:03.194243+0000","last_undegraded":"2024-09-15T23:24:03.194243+0000","last_fullsized":"2024-09-15T23:24:03.194243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:47:47.442458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117856+0000","last_change":"2024-09-15T23:21:59.682091+0000","last_active":"2024-09-15T23:24:03.117856+0000","last_peered":"2024-09-15T23:24:03.117856+0000","last_clean":"2024-09-15T23:24:03.117856+0000","last_became_active":"2024-09-15T23:21:59.681574+0000","last_became_peered":"2024-09-15T23:21:59.681574+0000","last_unstale":"2024-09-15T23:24:03.117856+0000","last_undegraded":"2024-09-15T23:24:03.117856+0000","last_fullsized":"2024-09-15T23:24:03.117856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:04.991608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114426+0000","last_change":"2024-09-15T23:21:52.533811+0000","last_active":"2024-09-15T23:24:03.114426+0000","last_peered":"2024-09-15T23:24:03.114426+0000","last_clean":"2024-09-15T23:24:03.114426+0000","last_became_active":"2024-09-15T23:21:52.533487+0000","last_became_peered":"2024-09-15T23:21:52.533487+0000","last_unstale":"2024-09-15T23:24:03.114426+0000","last_undegraded":"2024-09-15T23:24:03.114426+0000","last_fullsized":"2024-09-15T23:24:03.114426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:44.193212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.281895+0000","last_change":"2024-09-15T23:21:50.510883+0000","last_active":"2024-09-15T23:24:20.281895+0000","last_peered":"2024-09-15T23:24:20.281895+0000","last_clean":"2024-09-15T23:24:20.281895+0000","last_became_active":"2024-09-15T23:21:50.510478+0000","last_became_peered":"2024-09-15T23:21:50.510478+0000","last_unstale":"2024-09-15T23:24:20.281895+0000","last_undegraded":"2024-09-15T23:24:20.281895+0000","last_fullsized":"2024-09-15T23:24:20.281895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:35.454885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109705+0000","last_change":"2024-09-15T23:21:59.561816+0000","last_active":"2024-09-15T23:24:03.109705+0000","last_peered":"2024-09-15T23:24:03.109705+0000","last_clean":"2024-09-15T23:24:03.109705+0000","last_became_active":"2024-09-15T23:21:59.561434+0000","last_became_peered":"2024-09-15T23:21:59.561434+0000","last_unstale":"2024-09-15T23:24:03.109705+0000","last_undegraded":"2024-09-15T23:24:03.109705+0000","last_fullsized":"2024-09-15T23:24:03.109705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:59.945802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117955+0000","last_change":"2024-09-15T23:23:02.277204+0000","last_active":"2024-09-15T23:24:03.117955+0000","last_peered":"2024-09-15T23:24:03.117955+0000","last_clean":"2024-09-15T23:24:03.117955+0000","last_became_active":"2024-09-15T23:23:02.276770+0000","last_became_peered":"2024-09-15T23:23:02.276770+0000","last_unstale":"2024-09-15T23:24:03.117955+0000","last_undegraded":"2024-09-15T23:24:03.117955+0000","last_fullsized":"2024-09-15T23:24:03.117955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:11.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.145290+0000","last_change":"2024-09-15T23:21:50.561684+0000","last_active":"2024-09-15T23:24:20.145290+0000","last_peered":"2024-09-15T23:24:20.145290+0000","last_clean":"2024-09-15T23:24:20.145290+0000","last_became_active":"2024-09-15T23:21:50.561335+0000","last_became_peered":"2024-09-15T23:21:50.561335+0000","last_unstale":"2024-09-15T23:24:20.145290+0000","last_undegraded":"2024-09-15T23:24:20.145290+0000","last_fullsized":"2024-09-15T23:24:20.145290+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:23.897250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109863+0000","last_change":"2024-09-15T23:21:59.681858+0000","last_active":"2024-09-15T23:24:03.109863+0000","last_peered":"2024-09-15T23:24:03.109863+0000","last_clean":"2024-09-15T23:24:03.109863+0000","last_became_active":"2024-09-15T23:21:59.681441+0000","last_became_peered":"2024-09-15T23:21:59.681441+0000","last_unstale":"2024-09-15T23:24:03.109863+0000","last_undegraded":"2024-09-15T23:24:03.109863+0000","last_fullsized":"2024-09-15T23:24:03.109863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:28.513719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109320+0000","last_change":"2024-09-15T23:21:52.536141+0000","last_active":"2024-09-15T23:24:03.109320+0000","last_peered":"2024-09-15T23:24:03.109320+0000","last_clean":"2024-09-15T23:24:03.109320+0000","last_became_active":"2024-09-15T23:21:52.535832+0000","last_became_peered":"2024-09-15T23:21:52.535832+0000","last_unstale":"2024-09-15T23:24:03.109320+0000","last_undegraded":"2024-09-15T23:24:03.109320+0000","last_fullsized":"2024-09-15T23:24:03.109320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:21:44.341412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.123671+0000","last_change":"2024-09-15T23:24:03.123671+0000","last_active":"2024-09-15T23:24:03.123671+0000","last_peered":"2024-09-15T23:24:03.123671+0000","last_clean":"2024-09-15T23:24:03.123671+0000","last_became_active":"2024-09-15T23:24:03.119585+0000","last_became_peered":"2024-09-15T23:24:03.119585+0000","last_unstale":"2024-09-15T23:24:03.123671+0000","last_undegraded":"2024-09-15T23:24:03.123671+0000","last_fullsized":"2024-09-15T23:24:03.123671+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:30.734156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118359+0000","last_change":"2024-09-15T23:21:59.682118+0000","last_active":"2024-09-15T23:24:03.118359+0000","last_peered":"2024-09-15T23:24:03.118359+0000","last_clean":"2024-09-15T23:24:03.118359+0000","last_became_active":"2024-09-15T23:21:59.681500+0000","last_became_peered":"2024-09-15T23:21:59.681500+0000","last_unstale":"2024-09-15T23:24:03.118359+0000","last_undegraded":"2024-09-15T23:24:03.118359+0000","last_fullsized":"2024-09-15T23:24:03.118359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:58.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112298+0000","last_change":"2024-09-15T23:21:52.540458+0000","last_active":"2024-09-15T23:24:03.112298+0000","last_peered":"2024-09-15T23:24:03.112298+0000","last_clean":"2024-09-15T23:24:03.112298+0000","last_became_active":"2024-09-15T23:21:52.540112+0000","last_became_peered":"2024-09-15T23:21:52.540112+0000","last_unstale":"2024-09-15T23:24:03.112298+0000","last_undegraded":"2024-09-15T23:24:03.112298+0000","last_fullsized":"2024-09-15T23:24:03.112298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:15.240785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111097+0000","last_change":"2024-09-15T23:21:50.513124+0000","last_active":"2024-09-15T23:24:03.111097+0000","last_peered":"2024-09-15T23:24:03.111097+0000","last_clean":"2024-09-15T23:24:03.111097+0000","last_became_active":"2024-09-15T23:21:50.512705+0000","last_became_peered":"2024-09-15T23:21:50.512705+0000","last_unstale":"2024-09-15T23:24:03.111097+0000","last_undegraded":"2024-09-15T23:24:03.111097+0000","last_fullsized":"2024-09-15T23:24:03.111097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.417937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":354,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113701+0000","last_change":"2024-09-15T23:20:40.656652+0000","last_active":"2024-09-15T23:24:03.113701+0000","last_peered":"2024-09-15T23:24:03.113701+0000","last_clean":"2024-09-15T23:24:03.113701+0000","last_became_active":"2024-09-15T23:20:40.656266+0000","last_became_peered":"2024-09-15T23:20:40.656266+0000","last_unstale":"2024-09-15T23:24:03.113701+0000","last_undegraded":"2024-09-15T23:24:03.113701+0000","last_fullsized":"2024-09-15T23:24:03.113701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_clean_scrub_stamp":"2024-09-15T23:17:15.186850+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:42.317006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113728+0000","last_change":"2024-09-15T23:21:59.563777+0000","last_active":"2024-09-15T23:24:03.113728+0000","last_peered":"2024-09-15T23:24:03.113728+0000","last_clean":"2024-09-15T23:24:03.113728+0000","last_became_active":"2024-09-15T23:21:59.563320+0000","last_became_peered":"2024-09-15T23:21:59.563320+0000","last_unstale":"2024-09-15T23:24:03.113728+0000","last_undegraded":"2024-09-15T23:24:03.113728+0000","last_fullsized":"2024-09-15T23:24:03.113728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:25:07.791923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114145+0000","last_change":"2024-09-15T23:21:52.539842+0000","last_active":"2024-09-15T23:24:03.114145+0000","last_peered":"2024-09-15T23:24:03.114145+0000","last_clean":"2024-09-15T23:24:03.114145+0000","last_became_active":"2024-09-15T23:21:52.538683+0000","last_became_peered":"2024-09-15T23:21:52.538683+0000","last_unstale":"2024-09-15T23:24:03.114145+0000","last_undegraded":"2024-09-15T23:24:03.114145+0000","last_fullsized":"2024-09-15T23:24:03.114145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:43.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:18.280187+0000","last_change":"2024-09-15T23:21:50.767601+0000","last_active":"2024-09-15T23:24:18.280187+0000","last_peered":"2024-09-15T23:24:18.280187+0000","last_clean":"2024-09-15T23:24:18.280187+0000","last_became_active":"2024-09-15T23:21:50.767149+0000","last_became_peered":"2024-09-15T23:21:50.767149+0000","last_unstale":"2024-09-15T23:24:18.280187+0000","last_undegraded":"2024-09-15T23:24:18.280187+0000","last_fullsized":"2024-09-15T23:24:18.280187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:23.636551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117168+0000","last_change":"2024-09-15T23:21:59.563069+0000","last_active":"2024-09-15T23:24:03.117168+0000","last_peered":"2024-09-15T23:24:03.117168+0000","last_clean":"2024-09-15T23:24:03.117168+0000","last_became_active":"2024-09-15T23:21:59.562658+0000","last_became_peered":"2024-09-15T23:21:59.562658+0000","last_unstale":"2024-09-15T23:24:03.117168+0000","last_undegraded":"2024-09-15T23:24:03.117168+0000","last_fullsized":"2024-09-15T23:24:03.117168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:23.067198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113535+0000","last_change":"2024-09-15T23:21:52.535289+0000","last_active":"2024-09-15T23:24:03.113535+0000","last_peered":"2024-09-15T23:24:03.113535+0000","last_clean":"2024-09-15T23:24:03.113535+0000","last_became_active":"2024-09-15T23:21:52.534885+0000","last_became_peered":"2024-09-15T23:21:52.534885+0000","last_unstale":"2024-09-15T23:24:03.113535+0000","last_undegraded":"2024-09-15T23:24:03.113535+0000","last_fullsized":"2024-09-15T23:24:03.113535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:42.293786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118612+0000","last_change":"2024-09-15T23:21:50.513246+0000","last_active":"2024-09-15T23:24:03.118612+0000","last_peered":"2024-09-15T23:24:03.118612+0000","last_clean":"2024-09-15T23:24:03.118612+0000","last_became_active":"2024-09-15T23:21:50.512740+0000","last_became_peered":"2024-09-15T23:21:50.512740+0000","last_unstale":"2024-09-15T23:24:03.118612+0000","last_undegraded":"2024-09-15T23:24:03.118612+0000","last_fullsized":"2024-09-15T23:24:03.118612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:59.201970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193756+0000","last_change":"2024-09-15T23:21:59.568565+0000","last_active":"2024-09-15T23:24:03.193756+0000","last_peered":"2024-09-15T23:24:03.193756+0000","last_clean":"2024-09-15T23:24:03.193756+0000","last_became_active":"2024-09-15T23:21:59.567789+0000","last_became_peered":"2024-09-15T23:21:59.567789+0000","last_unstale":"2024-09-15T23:24:03.193756+0000","last_undegraded":"2024-09-15T23:24:03.193756+0000","last_fullsized":"2024-09-15T23:24:03.193756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:02.106971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114106+0000","last_change":"2024-09-15T23:21:52.533727+0000","last_active":"2024-09-15T23:24:03.114106+0000","last_peered":"2024-09-15T23:24:03.114106+0000","last_clean":"2024-09-15T23:24:03.114106+0000","last_became_active":"2024-09-15T23:21:52.532350+0000","last_became_peered":"2024-09-15T23:21:52.532350+0000","last_unstale":"2024-09-15T23:24:03.114106+0000","last_undegraded":"2024-09-15T23:24:03.114106+0000","last_fullsized":"2024-09-15T23:24:03.114106+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:03.624837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.124878+0000","last_change":"2024-09-15T23:24:03.124430+0000","last_active":"2024-09-15T23:24:03.124878+0000","last_peered":"2024-09-15T23:24:03.124878+0000","last_clean":"2024-09-15T23:24:03.124878+0000","last_became_active":"2024-09-15T23:24:03.119042+0000","last_became_peered":"2024-09-15T23:24:03.119042+0000","last_unstale":"2024-09-15T23:24:03.124878+0000","last_undegraded":"2024-09-15T23:24:03.124878+0000","last_fullsized":"2024-09-15T23:24:03.124878+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:52.065543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117798+0000","last_change":"2024-09-15T23:21:59.566429+0000","last_active":"2024-09-15T23:24:03.117798+0000","last_peered":"2024-09-15T23:24:03.117798+0000","last_clean":"2024-09-15T23:24:03.117798+0000","last_became_active":"2024-09-15T23:21:59.565835+0000","last_became_peered":"2024-09-15T23:21:59.565835+0000","last_unstale":"2024-09-15T23:24:03.117798+0000","last_undegraded":"2024-09-15T23:24:03.117798+0000","last_fullsized":"2024-09-15T23:24:03.117798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:38.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116276+0000","last_change":"2024-09-15T23:21:52.539544+0000","last_active":"2024-09-15T23:24:03.116276+0000","last_peered":"2024-09-15T23:24:03.116276+0000","last_clean":"2024-09-15T23:24:03.116276+0000","last_became_active":"2024-09-15T23:21:52.539203+0000","last_became_peered":"2024-09-15T23:21:52.539203+0000","last_unstale":"2024-09-15T23:24:03.116276+0000","last_undegraded":"2024-09-15T23:24:03.116276+0000","last_fullsized":"2024-09-15T23:24:03.116276+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.193965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:07.046123+0000","last_change":"2024-09-15T23:21:50.513543+0000","last_active":"2024-09-15T23:24:07.046123+0000","last_peered":"2024-09-15T23:24:07.046123+0000","last_clean":"2024-09-15T23:24:07.046123+0000","last_became_active":"2024-09-15T23:21:50.512812+0000","last_became_peered":"2024-09-15T23:21:50.512812+0000","last_unstale":"2024-09-15T23:24:07.046123+0000","last_undegraded":"2024-09-15T23:24:07.046123+0000","last_fullsized":"2024-09-15T23:24:07.046123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:50.954096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117689+0000","last_change":"2024-09-15T23:21:59.570163+0000","last_active":"2024-09-15T23:24:03.117689+0000","last_peered":"2024-09-15T23:24:03.117689+0000","last_clean":"2024-09-15T23:24:03.117689+0000","last_became_active":"2024-09-15T23:21:59.569805+0000","last_became_peered":"2024-09-15T23:21:59.569805+0000","last_unstale":"2024-09-15T23:24:03.117689+0000","last_undegraded":"2024-09-15T23:24:03.117689+0000","last_fullsized":"2024-09-15T23:24:03.117689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:28.530817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117877+0000","last_change":"2024-09-15T23:21:52.537850+0000","last_active":"2024-09-15T23:24:03.117877+0000","last_peered":"2024-09-15T23:24:03.117877+0000","last_clean":"2024-09-15T23:24:03.117877+0000","last_became_active":"2024-09-15T23:21:52.536579+0000","last_became_peered":"2024-09-15T23:21:52.536579+0000","last_unstale":"2024-09-15T23:24:03.117877+0000","last_undegraded":"2024-09-15T23:24:03.117877+0000","last_fullsized":"2024-09-15T23:24:03.117877+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:06.353615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118588+0000","last_change":"2024-09-15T23:21:50.562221+0000","last_active":"2024-09-15T23:24:03.118588+0000","last_peered":"2024-09-15T23:24:03.118588+0000","last_clean":"2024-09-15T23:24:03.118588+0000","last_became_active":"2024-09-15T23:21:50.561788+0000","last_became_peered":"2024-09-15T23:21:50.561788+0000","last_unstale":"2024-09-15T23:24:03.118588+0000","last_undegraded":"2024-09-15T23:24:03.118588+0000","last_fullsized":"2024-09-15T23:24:03.118588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.853812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109810+0000","last_change":"2024-09-15T23:23:02.383003+0000","last_active":"2024-09-15T23:24:03.109810+0000","last_peered":"2024-09-15T23:24:03.109810+0000","last_clean":"2024-09-15T23:24:03.109810+0000","last_became_active":"2024-09-15T23:23:02.382629+0000","last_became_peered":"2024-09-15T23:23:02.382629+0000","last_unstale":"2024-09-15T23:24:03.109810+0000","last_undegraded":"2024-09-15T23:24:03.109810+0000","last_fullsized":"2024-09-15T23:24:03.109810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:50.114689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113567+0000","last_change":"2024-09-15T23:21:52.539566+0000","last_active":"2024-09-15T23:24:03.113567+0000","last_peered":"2024-09-15T23:24:03.113567+0000","last_clean":"2024-09-15T23:24:03.113567+0000","last_became_active":"2024-09-15T23:21:52.539219+0000","last_became_peered":"2024-09-15T23:21:52.539219+0000","last_unstale":"2024-09-15T23:24:03.113567+0000","last_undegraded":"2024-09-15T23:24:03.113567+0000","last_fullsized":"2024-09-15T23:24:03.113567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:41.266308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":167,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.145816+0000","last_change":"2024-09-15T23:24:03.127667+0000","last_active":"2024-09-15T23:24:20.145816+0000","last_peered":"2024-09-15T23:24:20.145816+0000","last_clean":"2024-09-15T23:24:20.145816+0000","last_became_active":"2024-09-15T23:24:03.121613+0000","last_became_peered":"2024-09-15T23:24:03.121613+0000","last_unstale":"2024-09-15T23:24:20.145816+0000","last_undegraded":"2024-09-15T23:24:20.145816+0000","last_fullsized":"2024-09-15T23:24:20.145816+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:54:24.166895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":4,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1122,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194597+0000","last_change":"2024-09-15T23:21:59.568087+0000","last_active":"2024-09-15T23:24:03.194597+0000","last_peered":"2024-09-15T23:24:03.194597+0000","last_clean":"2024-09-15T23:24:03.194597+0000","last_became_active":"2024-09-15T23:21:59.567693+0000","last_became_peered":"2024-09-15T23:21:59.567693+0000","last_unstale":"2024-09-15T23:24:03.194597+0000","last_undegraded":"2024-09-15T23:24:03.194597+0000","last_fullsized":"2024-09-15T23:24:03.194597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.231873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114396+0000","last_change":"2024-09-15T23:21:52.531090+0000","last_active":"2024-09-15T23:24:03.114396+0000","last_peered":"2024-09-15T23:24:03.114396+0000","last_clean":"2024-09-15T23:24:03.114396+0000","last_became_active":"2024-09-15T23:21:52.530738+0000","last_became_peered":"2024-09-15T23:21:52.530738+0000","last_unstale":"2024-09-15T23:24:03.114396+0000","last_undegraded":"2024-09-15T23:24:03.114396+0000","last_fullsized":"2024-09-15T23:24:03.114396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:35.325128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117423+0000","last_change":"2024-09-15T23:21:50.514100+0000","last_active":"2024-09-15T23:24:03.117423+0000","last_peered":"2024-09-15T23:24:03.117423+0000","last_clean":"2024-09-15T23:24:03.117423+0000","last_became_active":"2024-09-15T23:21:50.513629+0000","last_became_peered":"2024-09-15T23:21:50.513629+0000","last_unstale":"2024-09-15T23:24:03.117423+0000","last_undegraded":"2024-09-15T23:24:03.117423+0000","last_fullsized":"2024-09-15T23:24:03.117423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.187544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115275+0000","last_change":"2024-09-15T23:21:59.562804+0000","last_active":"2024-09-15T23:24:03.115275+0000","last_peered":"2024-09-15T23:24:03.115275+0000","last_clean":"2024-09-15T23:24:03.115275+0000","last_became_active":"2024-09-15T23:21:59.562368+0000","last_became_peered":"2024-09-15T23:21:59.562368+0000","last_unstale":"2024-09-15T23:24:03.115275+0000","last_undegraded":"2024-09-15T23:24:03.115275+0000","last_fullsized":"2024-09-15T23:24:03.115275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:21.137333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109909+0000","last_change":"2024-09-15T23:21:52.530562+0000","last_active":"2024-09-15T23:24:03.109909+0000","last_peered":"2024-09-15T23:24:03.109909+0000","last_clean":"2024-09-15T23:24:03.109909+0000","last_became_active":"2024-09-15T23:21:52.530280+0000","last_became_peered":"2024-09-15T23:21:52.530280+0000","last_unstale":"2024-09-15T23:24:03.109909+0000","last_undegraded":"2024-09-15T23:24:03.109909+0000","last_fullsized":"2024-09-15T23:24:03.109909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:26.835651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113202+0000","last_change":"2024-09-15T23:21:50.513179+0000","last_active":"2024-09-15T23:24:03.113202+0000","last_peered":"2024-09-15T23:24:03.113202+0000","last_clean":"2024-09-15T23:24:03.113202+0000","last_became_active":"2024-09-15T23:21:50.512790+0000","last_became_peered":"2024-09-15T23:21:50.512790+0000","last_unstale":"2024-09-15T23:24:03.113202+0000","last_undegraded":"2024-09-15T23:24:03.113202+0000","last_fullsized":"2024-09-15T23:24:03.113202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:56.408198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113316+0000","last_change":"2024-09-15T23:23:02.382964+0000","last_active":"2024-09-15T23:24:03.113316+0000","last_peered":"2024-09-15T23:24:03.113316+0000","last_clean":"2024-09-15T23:24:03.113316+0000","last_became_active":"2024-09-15T23:23:02.382556+0000","last_became_peered":"2024-09-15T23:23:02.382556+0000","last_unstale":"2024-09-15T23:24:03.113316+0000","last_undegraded":"2024-09-15T23:24:03.113316+0000","last_fullsized":"2024-09-15T23:24:03.113316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:49.211591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117484+0000","last_change":"2024-09-15T23:21:52.530970+0000","last_active":"2024-09-15T23:24:03.117484+0000","last_peered":"2024-09-15T23:24:03.117484+0000","last_clean":"2024-09-15T23:24:03.117484+0000","last_became_active":"2024-09-15T23:21:52.530636+0000","last_became_peered":"2024-09-15T23:21:52.530636+0000","last_unstale":"2024-09-15T23:24:03.117484+0000","last_undegraded":"2024-09-15T23:24:03.117484+0000","last_fullsized":"2024-09-15T23:24:03.117484+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:33.374658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'40","reported_seq":316,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.146278+0000","last_change":"2024-09-15T23:24:03.131527+0000","last_active":"2024-09-15T23:24:20.146278+0000","last_peered":"2024-09-15T23:24:20.146278+0000","last_clean":"2024-09-15T23:24:20.146278+0000","last_became_active":"2024-09-15T23:24:03.119053+0000","last_became_peered":"2024-09-15T23:24:03.119053+0000","last_unstale":"2024-09-15T23:24:20.146278+0000","last_undegraded":"2024-09-15T23:24:20.146278+0000","last_fullsized":"2024-09-15T23:24:20.146278+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:21.870435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68399,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":214,"num_read_kb":169,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":62325,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113841+0000","last_change":"2024-09-15T23:23:02.277457+0000","last_active":"2024-09-15T23:24:03.113841+0000","last_peered":"2024-09-15T23:24:03.113841+0000","last_clean":"2024-09-15T23:24:03.113841+0000","last_became_active":"2024-09-15T23:23:02.276960+0000","last_became_peered":"2024-09-15T23:23:02.276960+0000","last_unstale":"2024-09-15T23:24:03.113841+0000","last_undegraded":"2024-09-15T23:24:03.113841+0000","last_fullsized":"2024-09-15T23:24:03.113841+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:15:28.823804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117534+0000","last_change":"2024-09-15T23:21:52.530541+0000","last_active":"2024-09-15T23:24:03.117534+0000","last_peered":"2024-09-15T23:24:03.117534+0000","last_clean":"2024-09-15T23:24:03.117534+0000","last_became_active":"2024-09-15T23:21:52.530217+0000","last_became_peered":"2024-09-15T23:21:52.530217+0000","last_unstale":"2024-09-15T23:24:03.117534+0000","last_undegraded":"2024-09-15T23:24:03.117534+0000","last_fullsized":"2024-09-15T23:24:03.117534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:14.864563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118500+0000","last_change":"2024-09-15T23:21:50.562203+0000","last_active":"2024-09-15T23:24:03.118500+0000","last_peered":"2024-09-15T23:24:03.118500+0000","last_clean":"2024-09-15T23:24:03.118500+0000","last_became_active":"2024-09-15T23:21:50.561827+0000","last_became_peered":"2024-09-15T23:21:50.561827+0000","last_unstale":"2024-09-15T23:24:03.118500+0000","last_undegraded":"2024-09-15T23:24:03.118500+0000","last_fullsized":"2024-09-15T23:24:03.118500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:22.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111024+0000","last_change":"2024-09-15T23:21:59.561013+0000","last_active":"2024-09-15T23:24:03.111024+0000","last_peered":"2024-09-15T23:24:03.111024+0000","last_clean":"2024-09-15T23:24:03.111024+0000","last_became_active":"2024-09-15T23:21:59.560665+0000","last_became_peered":"2024-09-15T23:21:59.560665+0000","last_unstale":"2024-09-15T23:24:03.111024+0000","last_undegraded":"2024-09-15T23:24:03.111024+0000","last_fullsized":"2024-09-15T23:24:03.111024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:00.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194109+0000","last_change":"2024-09-15T23:21:52.529922+0000","last_active":"2024-09-15T23:24:03.194109+0000","last_peered":"2024-09-15T23:24:03.194109+0000","last_clean":"2024-09-15T23:24:03.194109+0000","last_became_active":"2024-09-15T23:21:52.529179+0000","last_became_peered":"2024-09-15T23:21:52.529179+0000","last_unstale":"2024-09-15T23:24:03.194109+0000","last_undegraded":"2024-09-15T23:24:03.194109+0000","last_fullsized":"2024-09-15T23:24:03.194109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.077658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113481+0000","last_change":"2024-09-15T23:21:50.514420+0000","last_active":"2024-09-15T23:24:03.113481+0000","last_peered":"2024-09-15T23:24:03.113481+0000","last_clean":"2024-09-15T23:24:03.113481+0000","last_became_active":"2024-09-15T23:21:50.514007+0000","last_became_peered":"2024-09-15T23:21:50.514007+0000","last_unstale":"2024-09-15T23:24:03.113481+0000","last_undegraded":"2024-09-15T23:24:03.113481+0000","last_fullsized":"2024-09-15T23:24:03.113481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:36.845674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193335+0000","last_change":"2024-09-15T23:21:59.565978+0000","last_active":"2024-09-15T23:24:03.193335+0000","last_peered":"2024-09-15T23:24:03.193335+0000","last_clean":"2024-09-15T23:24:03.193335+0000","last_became_active":"2024-09-15T23:21:59.565604+0000","last_became_peered":"2024-09-15T23:21:59.565604+0000","last_unstale":"2024-09-15T23:24:03.193335+0000","last_undegraded":"2024-09-15T23:24:03.193335+0000","last_fullsized":"2024-09-15T23:24:03.193335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:47.089504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118064+0000","last_change":"2024-09-15T23:21:52.543511+0000","last_active":"2024-09-15T23:24:03.118064+0000","last_peered":"2024-09-15T23:24:03.118064+0000","last_clean":"2024-09-15T23:24:03.118064+0000","last_became_active":"2024-09-15T23:21:52.543169+0000","last_became_peered":"2024-09-15T23:21:52.543169+0000","last_unstale":"2024-09-15T23:24:03.118064+0000","last_undegraded":"2024-09-15T23:24:03.118064+0000","last_fullsized":"2024-09-15T23:24:03.118064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:05.696772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":252,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.282085+0000","last_change":"2024-09-15T23:21:50.518425+0000","last_active":"2024-09-15T23:24:20.282085+0000","last_peered":"2024-09-15T23:24:20.282085+0000","last_clean":"2024-09-15T23:24:20.282085+0000","last_became_active":"2024-09-15T23:21:50.518052+0000","last_became_peered":"2024-09-15T23:21:50.518052+0000","last_unstale":"2024-09-15T23:24:20.282085+0000","last_undegraded":"2024-09-15T23:24:20.282085+0000","last_fullsized":"2024-09-15T23:24:20.282085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:02.243869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112035+0000","last_change":"2024-09-15T23:21:59.570310+0000","last_active":"2024-09-15T23:24:03.112035+0000","last_peered":"2024-09-15T23:24:03.112035+0000","last_clean":"2024-09-15T23:24:03.112035+0000","last_became_active":"2024-09-15T23:21:59.569909+0000","last_became_peered":"2024-09-15T23:21:59.569909+0000","last_unstale":"2024-09-15T23:24:03.112035+0000","last_undegraded":"2024-09-15T23:24:03.112035+0000","last_fullsized":"2024-09-15T23:24:03.112035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:20:27.515537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194191+0000","last_change":"2024-09-15T23:21:59.568208+0000","last_active":"2024-09-15T23:24:03.194191+0000","last_peered":"2024-09-15T23:24:03.194191+0000","last_clean":"2024-09-15T23:24:03.194191+0000","last_became_active":"2024-09-15T23:21:59.567858+0000","last_became_peered":"2024-09-15T23:21:59.567858+0000","last_unstale":"2024-09-15T23:24:03.194191+0000","last_undegraded":"2024-09-15T23:24:03.194191+0000","last_fullsized":"2024-09-15T23:24:03.194191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:34:42.895247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193532+0000","last_change":"2024-09-15T23:21:52.532679+0000","last_active":"2024-09-15T23:24:03.193532+0000","last_peered":"2024-09-15T23:24:03.193532+0000","last_clean":"2024-09-15T23:24:03.193532+0000","last_became_active":"2024-09-15T23:21:52.532345+0000","last_became_peered":"2024-09-15T23:21:52.532345+0000","last_unstale":"2024-09-15T23:24:03.193532+0000","last_undegraded":"2024-09-15T23:24:03.193532+0000","last_fullsized":"2024-09-15T23:24:03.193532+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:45.943653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118135+0000","last_change":"2024-09-15T23:21:50.510836+0000","last_active":"2024-09-15T23:24:03.118135+0000","last_peered":"2024-09-15T23:24:03.118135+0000","last_clean":"2024-09-15T23:24:03.118135+0000","last_became_active":"2024-09-15T23:21:50.510514+0000","last_became_peered":"2024-09-15T23:21:50.510514+0000","last_unstale":"2024-09-15T23:24:03.118135+0000","last_undegraded":"2024-09-15T23:24:03.118135+0000","last_fullsized":"2024-09-15T23:24:03.118135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:06.404612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116423+0000","last_change":"2024-09-15T23:23:02.383436+0000","last_active":"2024-09-15T23:24:03.116423+0000","last_peered":"2024-09-15T23:24:03.116423+0000","last_clean":"2024-09-15T23:24:03.116423+0000","last_became_active":"2024-09-15T23:23:02.382982+0000","last_became_peered":"2024-09-15T23:23:02.382982+0000","last_unstale":"2024-09-15T23:24:03.116423+0000","last_undegraded":"2024-09-15T23:24:03.116423+0000","last_fullsized":"2024-09-15T23:24:03.116423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:55:14.076604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111923+0000","last_change":"2024-09-15T23:21:52.526766+0000","last_active":"2024-09-15T23:24:03.111923+0000","last_peered":"2024-09-15T23:24:03.111923+0000","last_clean":"2024-09-15T23:24:03.111923+0000","last_became_active":"2024-09-15T23:21:52.526446+0000","last_became_peered":"2024-09-15T23:21:52.526446+0000","last_unstale":"2024-09-15T23:24:03.111923+0000","last_undegraded":"2024-09-15T23:24:03.111923+0000","last_fullsized":"2024-09-15T23:24:03.111923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:21:37.004730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109384+0000","last_change":"2024-09-15T23:21:50.512928+0000","last_active":"2024-09-15T23:24:03.109384+0000","last_peered":"2024-09-15T23:24:03.109384+0000","last_clean":"2024-09-15T23:24:03.109384+0000","last_became_active":"2024-09-15T23:21:50.512558+0000","last_became_peered":"2024-09-15T23:21:50.512558+0000","last_unstale":"2024-09-15T23:24:03.109384+0000","last_undegraded":"2024-09-15T23:24:03.109384+0000","last_fullsized":"2024-09-15T23:24:03.109384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:08.653084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114309+0000","last_change":"2024-09-15T23:21:50.516920+0000","last_active":"2024-09-15T23:24:03.114309+0000","last_peered":"2024-09-15T23:24:03.114309+0000","last_clean":"2024-09-15T23:24:03.114309+0000","last_became_active":"2024-09-15T23:21:50.516573+0000","last_became_peered":"2024-09-15T23:21:50.516573+0000","last_unstale":"2024-09-15T23:24:03.114309+0000","last_undegraded":"2024-09-15T23:24:03.114309+0000","last_fullsized":"2024-09-15T23:24:03.114309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.367903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117266+0000","last_change":"2024-09-15T23:21:52.536696+0000","last_active":"2024-09-15T23:24:03.117266+0000","last_peered":"2024-09-15T23:24:03.117266+0000","last_clean":"2024-09-15T23:24:03.117266+0000","last_became_active":"2024-09-15T23:21:52.536338+0000","last_became_peered":"2024-09-15T23:21:52.536338+0000","last_unstale":"2024-09-15T23:24:03.117266+0000","last_undegraded":"2024-09-15T23:24:03.117266+0000","last_fullsized":"2024-09-15T23:24:03.117266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:05.152749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112825+0000","last_change":"2024-09-15T23:21:59.562968+0000","last_active":"2024-09-15T23:24:03.112825+0000","last_peered":"2024-09-15T23:24:03.112825+0000","last_clean":"2024-09-15T23:24:03.112825+0000","last_became_active":"2024-09-15T23:21:59.562579+0000","last_became_peered":"2024-09-15T23:21:59.562579+0000","last_unstale":"2024-09-15T23:24:03.112825+0000","last_undegraded":"2024-09-15T23:24:03.112825+0000","last_fullsized":"2024-09-15T23:24:03.112825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:02.853057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:21.284870+0000","last_change":"2024-09-15T23:21:50.513944+0000","last_active":"2024-09-15T23:24:21.284870+0000","last_peered":"2024-09-15T23:24:21.284870+0000","last_clean":"2024-09-15T23:24:21.284870+0000","last_became_active":"2024-09-15T23:21:50.513484+0000","last_became_peered":"2024-09-15T23:21:50.513484+0000","last_unstale":"2024-09-15T23:24:21.284870+0000","last_undegraded":"2024-09-15T23:24:21.284870+0000","last_fullsized":"2024-09-15T23:24:21.284870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:35.719344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118061+0000","last_change":"2024-09-15T23:21:52.539467+0000","last_active":"2024-09-15T23:24:03.118061+0000","last_peered":"2024-09-15T23:24:03.118061+0000","last_clean":"2024-09-15T23:24:03.118061+0000","last_became_active":"2024-09-15T23:21:52.538998+0000","last_became_peered":"2024-09-15T23:21:52.538998+0000","last_unstale":"2024-09-15T23:24:03.118061+0000","last_undegraded":"2024-09-15T23:24:03.118061+0000","last_fullsized":"2024-09-15T23:24:03.118061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:22.213857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112854+0000","last_change":"2024-09-15T23:21:59.564842+0000","last_active":"2024-09-15T23:24:03.112854+0000","last_peered":"2024-09-15T23:24:03.112854+0000","last_clean":"2024-09-15T23:24:03.112854+0000","last_became_active":"2024-09-15T23:21:59.564408+0000","last_became_peered":"2024-09-15T23:21:59.564408+0000","last_unstale":"2024-09-15T23:24:03.112854+0000","last_undegraded":"2024-09-15T23:24:03.112854+0000","last_fullsized":"2024-09-15T23:24:03.112854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:17.909020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115745+0000","last_change":"2024-09-15T23:23:02.149420+0000","last_active":"2024-09-15T23:24:03.115745+0000","last_peered":"2024-09-15T23:24:03.115745+0000","last_clean":"2024-09-15T23:24:03.115745+0000","last_became_active":"2024-09-15T23:23:02.149003+0000","last_became_peered":"2024-09-15T23:23:02.149003+0000","last_unstale":"2024-09-15T23:24:03.115745+0000","last_undegraded":"2024-09-15T23:24:03.115745+0000","last_fullsized":"2024-09-15T23:24:03.115745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:45.490322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.532769+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532426+0000","last_became_peered":"2024-09-15T23:21:52.532426+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:09.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.533148+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532823+0000","last_became_peered":"2024-09-15T23:21:52.532823+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:01.833945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193803+0000","last_change":"2024-09-15T23:21:52.536790+0000","last_active":"2024-09-15T23:24:03.193803+0000","last_peered":"2024-09-15T23:24:03.193803+0000","last_clean":"2024-09-15T23:24:03.193803+0000","last_became_active":"2024-09-15T23:21:52.536515+0000","last_became_peered":"2024-09-15T23:21:52.536515+0000","last_unstale":"2024-09-15T23:24:03.193803+0000","last_undegraded":"2024-09-15T23:24:03.193803+0000","last_fullsized":"2024-09-15T23:24:03.193803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:37.207759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109800+0000","last_change":"2024-09-15T23:23:02.382842+0000","last_active":"2024-09-15T23:24:03.109800+0000","last_peered":"2024-09-15T23:24:03.109800+0000","last_clean":"2024-09-15T23:24:03.109800+0000","last_became_active":"2024-09-15T23:23:02.382455+0000","last_became_peered":"2024-09-15T23:23:02.382455+0000","last_unstale":"2024-09-15T23:24:03.109800+0000","last_undegraded":"2024-09-15T23:24:03.109800+0000","last_fullsized":"2024-09-15T23:24:03.109800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:23.617641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.540628+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.540308+0000","last_became_peered":"2024-09-15T23:21:52.540308+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:16.660637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113381+0000","last_change":"2024-09-15T23:21:52.533780+0000","last_active":"2024-09-15T23:24:03.113381+0000","last_peered":"2024-09-15T23:24:03.113381+0000","last_clean":"2024-09-15T23:24:03.113381+0000","last_became_active":"2024-09-15T23:21:52.533354+0000","last_became_peered":"2024-09-15T23:21:52.533354+0000","last_unstale":"2024-09-15T23:24:03.113381+0000","last_undegraded":"2024-09-15T23:24:03.113381+0000","last_fullsized":"2024-09-15T23:24:03.113381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.381059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112801+0000","last_change":"2024-09-15T23:21:52.536406+0000","last_active":"2024-09-15T23:24:03.112801+0000","last_peered":"2024-09-15T23:24:03.112801+0000","last_clean":"2024-09-15T23:24:03.112801+0000","last_became_active":"2024-09-15T23:21:52.536091+0000","last_became_peered":"2024-09-15T23:21:52.536091+0000","last_unstale":"2024-09-15T23:24:03.112801+0000","last_undegraded":"2024-09-15T23:24:03.112801+0000","last_fullsized":"2024-09-15T23:24:03.112801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:30.452201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117879+0000","last_change":"2024-09-15T23:21:52.536333+0000","last_active":"2024-09-15T23:24:03.117879+0000","last_peered":"2024-09-15T23:24:03.117879+0000","last_clean":"2024-09-15T23:24:03.117879+0000","last_became_active":"2024-09-15T23:21:52.534677+0000","last_became_peered":"2024-09-15T23:21:52.534677+0000","last_unstale":"2024-09-15T23:24:03.117879+0000","last_undegraded":"2024-09-15T23:24:03.117879+0000","last_fullsized":"2024-09-15T23:24:03.117879+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:05.287914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112116+0000","last_change":"2024-09-15T23:21:52.536442+0000","last_active":"2024-09-15T23:24:03.112116+0000","last_peered":"2024-09-15T23:24:03.112116+0000","last_clean":"2024-09-15T23:24:03.112116+0000","last_became_active":"2024-09-15T23:21:52.536183+0000","last_became_peered":"2024-09-15T23:21:52.536183+0000","last_unstale":"2024-09-15T23:24:03.112116+0000","last_undegraded":"2024-09-15T23:24:03.112116+0000","last_fullsized":"2024-09-15T23:24:03.112116+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:41.641370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117398+0000","last_change":"2024-09-15T23:23:02.277365+0000","last_active":"2024-09-15T23:24:03.117398+0000","last_peered":"2024-09-15T23:24:03.117398+0000","last_clean":"2024-09-15T23:24:03.117398+0000","last_became_active":"2024-09-15T23:23:02.276905+0000","last_became_peered":"2024-09-15T23:23:02.276905+0000","last_unstale":"2024-09-15T23:24:03.117398+0000","last_undegraded":"2024-09-15T23:24:03.117398+0000","last_fullsized":"2024-09-15T23:24:03.117398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:21.100657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109788+0000","last_change":"2024-09-15T23:21:52.531171+0000","last_active":"2024-09-15T23:24:03.109788+0000","last_peered":"2024-09-15T23:24:03.109788+0000","last_clean":"2024-09-15T23:24:03.109788+0000","last_became_active":"2024-09-15T23:21:52.530724+0000","last_became_peered":"2024-09-15T23:21:52.530724+0000","last_unstale":"2024-09-15T23:24:03.109788+0000","last_undegraded":"2024-09-15T23:24:03.109788+0000","last_fullsized":"2024-09-15T23:24:03.109788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:11.009119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115400+0000","last_change":"2024-09-15T23:21:52.530433+0000","last_active":"2024-09-15T23:24:03.115400+0000","last_peered":"2024-09-15T23:24:03.115400+0000","last_clean":"2024-09-15T23:24:03.115400+0000","last_became_active":"2024-09-15T23:21:52.530058+0000","last_became_peered":"2024-09-15T23:21:52.530058+0000","last_unstale":"2024-09-15T23:24:03.115400+0000","last_undegraded":"2024-09-15T23:24:03.115400+0000","last_fullsized":"2024-09-15T23:24:03.115400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:51.208820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113218+0000","last_change":"2024-09-15T23:21:52.534639+0000","last_active":"2024-09-15T23:24:03.113218+0000","last_peered":"2024-09-15T23:24:03.113218+0000","last_clean":"2024-09-15T23:24:03.113218+0000","last_became_active":"2024-09-15T23:21:52.534362+0000","last_became_peered":"2024-09-15T23:21:52.534362+0000","last_unstale":"2024-09-15T23:24:03.113218+0000","last_undegraded":"2024-09-15T23:24:03.113218+0000","last_fullsized":"2024-09-15T23:24:03.113218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:58:12.023501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194630+0000","last_change":"2024-09-15T23:21:52.532758+0000","last_active":"2024-09-15T23:24:03.194630+0000","last_peered":"2024-09-15T23:24:03.194630+0000","last_clean":"2024-09-15T23:24:03.194630+0000","last_became_active":"2024-09-15T23:21:52.532412+0000","last_became_peered":"2024-09-15T23:21:52.532412+0000","last_unstale":"2024-09-15T23:24:03.194630+0000","last_undegraded":"2024-09-15T23:24:03.194630+0000","last_fullsized":"2024-09-15T23:24:03.194630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:46.285711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116737+0000","last_change":"2024-09-15T23:21:52.533928+0000","last_active":"2024-09-15T23:24:03.116737+0000","last_peered":"2024-09-15T23:24:03.116737+0000","last_clean":"2024-09-15T23:24:03.116737+0000","last_became_active":"2024-09-15T23:21:52.533578+0000","last_became_peered":"2024-09-15T23:21:52.533578+0000","last_unstale":"2024-09-15T23:24:03.116737+0000","last_undegraded":"2024-09-15T23:24:03.116737+0000","last_fullsized":"2024-09-15T23:24:03.116737+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:49.476316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116894+0000","last_change":"2024-09-15T23:21:52.533417+0000","last_active":"2024-09-15T23:24:03.116894+0000","last_peered":"2024-09-15T23:24:03.116894+0000","last_clean":"2024-09-15T23:24:03.116894+0000","last_became_active":"2024-09-15T23:21:52.533085+0000","last_became_peered":"2024-09-15T23:21:52.533085+0000","last_unstale":"2024-09-15T23:24:03.116894+0000","last_undegraded":"2024-09-15T23:24:03.116894+0000","last_fullsized":"2024-09-15T23:24:03.116894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:20.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193763+0000","last_change":"2024-09-15T23:21:52.542340+0000","last_active":"2024-09-15T23:24:03.193763+0000","last_peered":"2024-09-15T23:24:03.193763+0000","last_clean":"2024-09-15T23:24:03.193763+0000","last_became_active":"2024-09-15T23:21:52.542048+0000","last_became_peered":"2024-09-15T23:21:52.542048+0000","last_unstale":"2024-09-15T23:24:03.193763+0000","last_undegraded":"2024-09-15T23:24:03.193763+0000","last_fullsized":"2024-09-15T23:24:03.193763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:57.557420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193941+0000","last_change":"2024-09-15T23:21:52.535576+0000","last_active":"2024-09-15T23:24:03.193941+0000","last_peered":"2024-09-15T23:24:03.193941+0000","last_clean":"2024-09-15T23:24:03.193941+0000","last_became_active":"2024-09-15T23:21:52.535223+0000","last_became_peered":"2024-09-15T23:21:52.535223+0000","last_unstale":"2024-09-15T23:24:03.193941+0000","last_undegraded":"2024-09-15T23:24:03.193941+0000","last_fullsized":"2024-09-15T23:24:03.193941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:45.874927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109691+0000","last_change":"2024-09-15T23:23:02.150556+0000","last_active":"2024-09-15T23:24:03.109691+0000","last_peered":"2024-09-15T23:24:03.109691+0000","last_clean":"2024-09-15T23:24:03.109691+0000","last_became_active":"2024-09-15T23:23:02.150253+0000","last_became_peered":"2024-09-15T23:23:02.150253+0000","last_unstale":"2024-09-15T23:24:03.109691+0000","last_undegraded":"2024-09-15T23:24:03.109691+0000","last_fullsized":"2024-09-15T23:24:03.109691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:47.950356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110875+0000","last_change":"2024-09-15T23:21:52.537012+0000","last_active":"2024-09-15T23:24:03.110875+0000","last_peered":"2024-09-15T23:24:03.110875+0000","last_clean":"2024-09-15T23:24:03.110875+0000","last_became_active":"2024-09-15T23:21:52.536245+0000","last_became_peered":"2024-09-15T23:21:52.536245+0000","last_unstale":"2024-09-15T23:24:03.110875+0000","last_undegraded":"2024-09-15T23:24:03.110875+0000","last_fullsized":"2024-09-15T23:24:03.110875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:31.150491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114034+0000","last_change":"2024-09-15T23:21:52.540431+0000","last_active":"2024-09-15T23:24:03.114034+0000","last_peered":"2024-09-15T23:24:03.114034+0000","last_clean":"2024-09-15T23:24:03.114034+0000","last_became_active":"2024-09-15T23:21:52.540104+0000","last_became_peered":"2024-09-15T23:21:52.540104+0000","last_unstale":"2024-09-15T23:24:03.114034+0000","last_undegraded":"2024-09-15T23:24:03.114034+0000","last_fullsized":"2024-09-15T23:24:03.114034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.550258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114876+0000","last_change":"2024-09-15T23:21:52.537590+0000","last_active":"2024-09-15T23:24:03.114876+0000","last_peered":"2024-09-15T23:24:03.114876+0000","last_clean":"2024-09-15T23:24:03.114876+0000","last_became_active":"2024-09-15T23:21:52.533804+0000","last_became_peered":"2024-09-15T23:21:52.533804+0000","last_unstale":"2024-09-15T23:24:03.114876+0000","last_undegraded":"2024-09-15T23:24:03.114876+0000","last_fullsized":"2024-09-15T23:24:03.114876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:48.449090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117952+0000","last_change":"2024-09-15T23:21:52.536130+0000","last_active":"2024-09-15T23:24:03.117952+0000","last_peered":"2024-09-15T23:24:03.117952+0000","last_clean":"2024-09-15T23:24:03.117952+0000","last_became_active":"2024-09-15T23:21:52.535336+0000","last_became_peered":"2024-09-15T23:21:52.535336+0000","last_unstale":"2024-09-15T23:24:03.117952+0000","last_undegraded":"2024-09-15T23:24:03.117952+0000","last_fullsized":"2024-09-15T23:24:03.117952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:52.581994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111827+0000","last_change":"2024-09-15T23:21:52.540485+0000","last_active":"2024-09-15T23:24:03.111827+0000","last_peered":"2024-09-15T23:24:03.111827+0000","last_clean":"2024-09-15T23:24:03.111827+0000","last_became_active":"2024-09-15T23:21:52.540178+0000","last_became_peered":"2024-09-15T23:21:52.540178+0000","last_unstale":"2024-09-15T23:24:03.111827+0000","last_undegraded":"2024-09-15T23:24:03.111827+0000","last_fullsized":"2024-09-15T23:24:03.111827+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:40:22.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113883+0000","last_change":"2024-09-15T23:21:52.538845+0000","last_active":"2024-09-15T23:24:03.113883+0000","last_peered":"2024-09-15T23:24:03.113883+0000","last_clean":"2024-09-15T23:24:03.113883+0000","last_became_active":"2024-09-15T23:21:52.538511+0000","last_became_peered":"2024-09-15T23:21:52.538511+0000","last_unstale":"2024-09-15T23:24:03.113883+0000","last_undegraded":"2024-09-15T23:24:03.113883+0000","last_fullsized":"2024-09-15T23:24:03.113883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:00.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114107+0000","last_change":"2024-09-15T23:23:02.149590+0000","last_active":"2024-09-15T23:24:03.114107+0000","last_peered":"2024-09-15T23:24:03.114107+0000","last_clean":"2024-09-15T23:24:03.114107+0000","last_became_active":"2024-09-15T23:23:02.149184+0000","last_became_peered":"2024-09-15T23:23:02.149184+0000","last_unstale":"2024-09-15T23:24:03.114107+0000","last_undegraded":"2024-09-15T23:24:03.114107+0000","last_fullsized":"2024-09-15T23:24:03.114107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:05:29.407175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117530+0000","last_change":"2024-09-15T23:21:52.530524+0000","last_active":"2024-09-15T23:24:03.117530+0000","last_peered":"2024-09-15T23:24:03.117530+0000","last_clean":"2024-09-15T23:24:03.117530+0000","last_became_active":"2024-09-15T23:21:52.530203+0000","last_became_peered":"2024-09-15T23:21:52.530203+0000","last_unstale":"2024-09-15T23:24:03.117530+0000","last_undegraded":"2024-09-15T23:24:03.117530+0000","last_fullsized":"2024-09-15T23:24:03.117530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:50.560211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110690+0000","last_change":"2024-09-15T23:21:52.537256+0000","last_active":"2024-09-15T23:24:03.110690+0000","last_peered":"2024-09-15T23:24:03.110690+0000","last_clean":"2024-09-15T23:24:03.110690+0000","last_became_active":"2024-09-15T23:21:52.536731+0000","last_became_peered":"2024-09-15T23:21:52.536731+0000","last_unstale":"2024-09-15T23:24:03.110690+0000","last_undegraded":"2024-09-15T23:24:03.110690+0000","last_fullsized":"2024-09-15T23:24:03.110690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:18.815398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194364+0000","last_change":"2024-09-15T23:21:52.537728+0000","last_active":"2024-09-15T23:24:03.194364+0000","last_peered":"2024-09-15T23:24:03.194364+0000","last_clean":"2024-09-15T23:24:03.194364+0000","last_became_active":"2024-09-15T23:21:52.537434+0000","last_became_peered":"2024-09-15T23:21:52.537434+0000","last_unstale":"2024-09-15T23:24:03.194364+0000","last_undegraded":"2024-09-15T23:24:03.194364+0000","last_fullsized":"2024-09-15T23:24:03.194364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:25.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194218+0000","last_change":"2024-09-15T23:21:52.534780+0000","last_active":"2024-09-15T23:24:03.194218+0000","last_peered":"2024-09-15T23:24:03.194218+0000","last_clean":"2024-09-15T23:24:03.194218+0000","last_became_active":"2024-09-15T23:21:52.534399+0000","last_became_peered":"2024-09-15T23:21:52.534399+0000","last_unstale":"2024-09-15T23:24:03.194218+0000","last_undegraded":"2024-09-15T23:24:03.194218+0000","last_fullsized":"2024-09-15T23:24:03.194218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:47.344129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116592+0000","last_change":"2024-09-15T23:21:52.533403+0000","last_active":"2024-09-15T23:24:03.116592+0000","last_peered":"2024-09-15T23:24:03.116592+0000","last_clean":"2024-09-15T23:24:03.116592+0000","last_became_active":"2024-09-15T23:21:52.533069+0000","last_became_peered":"2024-09-15T23:21:52.533069+0000","last_unstale":"2024-09-15T23:24:03.116592+0000","last_undegraded":"2024-09-15T23:24:03.116592+0000","last_fullsized":"2024-09-15T23:24:03.116592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:38.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76056,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1071,"num_read_kb":611,"num_write":263,"num_write_kb":158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":63447,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":222402,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":246033,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579948,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":521776,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841584,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1159168,"data_stored":1176403,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005109,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1032192,"data_stored":524287,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234043,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":520569,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462976,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1073152,"data_stored":1116395,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626501,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1003520,"data_stored":515695,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855434,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1112565,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018959,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":582936,"data_compressed":39742,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247891,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1015808,"data_stored":515403,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444121,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1019904,"data_stored":517253,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607646,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":579358,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1028096,"data_stored":517151,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":70026,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":71062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:24:24.287 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-09-15T23:24:24.287 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T23:24:24.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 -- 172.21.15.89:0/3360098957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb05c101c00 msgr2=0x7fb05c102080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 --2- 172.21.15.89:0/3360098957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb05c101c00 0x7fb05c102080 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb050009900 tx=0x7fb05002a410 comp rx=0 tx=0).stop 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 -- 172.21.15.89:0/3360098957 shutdown_connections 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 --2- 172.21.15.89:0/3360098957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb05c1025c0 0x7fb05c108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 --2- 172.21.15.89:0/3360098957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb05c101c00 0x7fb05c102080 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 --2- 172.21.15.89:0/3360098957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb05c100a10 0x7fb05c100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 -- 172.21.15.89:0/3360098957 >> 172.21.15.89:0/3360098957 conn(0x7fb05c0fc170 msgr2=0x7fb05c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 -- 172.21.15.89:0/3360098957 shutdown_connections 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 -- 172.21.15.89:0/3360098957 wait complete. 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 Processor -- start 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.422+0000 7fb060b9f640 1 -- start start 2024-09-15T23:24:24.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb060b9f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb05c100a10 0x7fb05c10b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb060b9f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb05c101c00 0x7fb05c1099d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb060b9f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb05c1025c0 0x7fb05c109f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb060b9f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c10d7e0 con 0x7fb05c101c00 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb060b9f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c10d660 con 0x7fb05c100a10 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb060b9f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c10d960 con 0x7fb05c1025c0 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb059d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb05c101c00 0x7fb05c1099d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb059d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb05c101c00 0x7fb05c1099d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35866/0 (socket says 172.21.15.89:35866) 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb059d9b640 1 -- 172.21.15.89:0/3200316034 learned_addr learned my addr 172.21.15.89:0/3200316034 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb05ad9d640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb05c1025c0 0x7fb05c109f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb05a59c640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb05c100a10 0x7fb05c10b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb059d9b640 1 -- 172.21.15.89:0/3200316034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb05c1025c0 msgr2=0x7fb05c109f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb059d9b640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb05c1025c0 0x7fb05c109f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb059d9b640 1 -- 172.21.15.89:0/3200316034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb05c100a10 msgr2=0x7fb05c10b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb059d9b640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb05c100a10 0x7fb05c10b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb059d9b640 1 -- 172.21.15.89:0/3200316034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c10a7a0 con 0x7fb05c101c00 2024-09-15T23:24:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb059d9b640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb05c101c00 0x7fb05c1099d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb050009230 tx=0x7fb050031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb04b7fe640 1 -- 172.21.15.89:0/3200316034 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05003f030 con 0x7fb05c101c00 2024-09-15T23:24:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb04b7fe640 1 -- 172.21.15.89:0/3200316034 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb050003ad0 con 0x7fb05c101c00 2024-09-15T23:24:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb060b9f640 1 -- 172.21.15.89:0/3200316034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05c1a1e80 con 0x7fb05c101c00 2024-09-15T23:24:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb04b7fe640 1 -- 172.21.15.89:0/3200316034 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb050035070 con 0x7fb05c101c00 2024-09-15T23:24:24.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.423+0000 7fb060b9f640 1 -- 172.21.15.89:0/3200316034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c1a22e0 con 0x7fb05c101c00 2024-09-15T23:24:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.424+0000 7fb060b9f640 1 -- 172.21.15.89:0/3200316034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb05c107d30 con 0x7fb05c101c00 2024-09-15T23:24:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.425+0000 7fb04b7fe640 1 -- 172.21.15.89:0/3200316034 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fb050042030 con 0x7fb05c101c00 2024-09-15T23:24:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.426+0000 7fb04b7fe640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0440a32e0 0x7fb0440a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.426+0000 7fb05a59c640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0440a32e0 0x7fb0440a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.426+0000 7fb04b7fe640 1 -- 172.21.15.89:0/3200316034 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb0500f03c0 con 0x7fb05c101c00 2024-09-15T23:24:24.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.427+0000 7fb05a59c640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0440a32e0 0x7fb0440a57a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb04c005c40 tx=0x7fb04c005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:24.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.430+0000 7fb04b7fe640 1 -- 172.21.15.89:0/3200316034 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb0500bcb90 con 0x7fb05c101c00 2024-09-15T23:24:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:24 smithi120 ceph-mon[35250]: pgmap v361: 129 pgs: 129 active+clean; 651 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-09-15T23:24:24.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:24 smithi174 ceph-mon[35586]: pgmap v361: 129 pgs: 129 active+clean; 651 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-09-15T23:24:24.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.597+0000 7fb060b9f640 1 -- 172.21.15.89:0/3200316034 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb05c062630 con 0x7fb0440a32e0 2024-09-15T23:24:24.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.605+0000 7fb04b7fe640 1 -- 172.21.15.89:0/3200316034 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340821 (secure 0 0 0) 0x7fb05c062630 con 0x7fb0440a32e0 2024-09-15T23:24:24.609 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:24.611 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-15T23:24:24.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 -- 172.21.15.89:0/3200316034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0440a32e0 msgr2=0x7fb0440a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0440a32e0 0x7fb0440a57a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb04c005c40 tx=0x7fb04c005bd0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 -- 172.21.15.89:0/3200316034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb05c101c00 msgr2=0x7fb05c1099d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb05c101c00 0x7fb05c1099d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb050009230 tx=0x7fb050031b60 comp rx=0 tx=0).stop 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 -- 172.21.15.89:0/3200316034 shutdown_connections 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0440a32e0 0x7fb0440a57a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb05c1025c0 0x7fb05c109f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb05c101c00 0x7fb05c1099d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 --2- 172.21.15.89:0/3200316034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb05c100a10 0x7fb05c10b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 -- 172.21.15.89:0/3200316034 >> 172.21.15.89:0/3200316034 conn(0x7fb05c0fc170 msgr2=0x7fb05c104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 -- 172.21.15.89:0/3200316034 shutdown_connections 2024-09-15T23:24:24.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:24.608+0000 7fb060b9f640 1 -- 172.21.15.89:0/3200316034 wait complete. 2024-09-15T23:24:24.620 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-15T23:24:23.021695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666658,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1197,"num_read_kb":720,"num_write":446,"num_write_kb":3098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1244183,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369880,"kb_used_data":12264,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571944,"statfs":{"total":1151940427776,"available":1151561670656,"internally_reserved":0,"allocated":12558336,"data_stored":8199791,"data_compressed":471759,"data_compressed_allocated":638976,"data_compressed_original":2813952,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6775,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":137,"num_read_kb":72,"num_write":9,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003289"},"pg_stats":[{"pgid":"2.1f","version":"86'13","reported_seq":153,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:16.275939+0000","last_change":"2024-09-15T23:21:50.513610+0000","last_active":"2024-09-15T23:24:16.275939+0000","last_peered":"2024-09-15T23:24:16.275939+0000","last_clean":"2024-09-15T23:24:16.275939+0000","last_became_active":"2024-09-15T23:21:50.513266+0000","last_became_peered":"2024-09-15T23:21:50.513266+0000","last_unstale":"2024-09-15T23:24:16.275939+0000","last_undegraded":"2024-09-15T23:24:16.275939+0000","last_fullsized":"2024-09-15T23:24:16.275939+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:16:48.692801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112578+0000","last_change":"2024-09-15T23:21:52.540248+0000","last_active":"2024-09-15T23:24:03.112578+0000","last_peered":"2024-09-15T23:24:03.112578+0000","last_clean":"2024-09-15T23:24:03.112578+0000","last_became_active":"2024-09-15T23:21:52.539945+0000","last_became_peered":"2024-09-15T23:21:52.539945+0000","last_unstale":"2024-09-15T23:24:03.112578+0000","last_undegraded":"2024-09-15T23:24:03.112578+0000","last_fullsized":"2024-09-15T23:24:03.112578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:33.402153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112360+0000","last_change":"2024-09-15T23:21:59.567246+0000","last_active":"2024-09-15T23:24:03.112360+0000","last_peered":"2024-09-15T23:24:03.112360+0000","last_clean":"2024-09-15T23:24:03.112360+0000","last_became_active":"2024-09-15T23:21:59.566832+0000","last_became_peered":"2024-09-15T23:21:59.566832+0000","last_unstale":"2024-09-15T23:24:03.112360+0000","last_undegraded":"2024-09-15T23:24:03.112360+0000","last_fullsized":"2024-09-15T23:24:03.112360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:19.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112747+0000","last_change":"2024-09-15T23:21:50.561767+0000","last_active":"2024-09-15T23:24:03.112747+0000","last_peered":"2024-09-15T23:24:03.112747+0000","last_clean":"2024-09-15T23:24:03.112747+0000","last_became_active":"2024-09-15T23:21:50.561385+0000","last_became_peered":"2024-09-15T23:21:50.561385+0000","last_unstale":"2024-09-15T23:24:03.112747+0000","last_undegraded":"2024-09-15T23:24:03.112747+0000","last_fullsized":"2024-09-15T23:24:03.112747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:11.897884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194008+0000","last_change":"2024-09-15T23:21:52.534874+0000","last_active":"2024-09-15T23:24:03.194008+0000","last_peered":"2024-09-15T23:24:03.194008+0000","last_clean":"2024-09-15T23:24:03.194008+0000","last_became_active":"2024-09-15T23:21:52.534334+0000","last_became_peered":"2024-09-15T23:21:52.534334+0000","last_unstale":"2024-09-15T23:24:03.194008+0000","last_undegraded":"2024-09-15T23:24:03.194008+0000","last_fullsized":"2024-09-15T23:24:03.194008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:15.972648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110384+0000","last_change":"2024-09-15T23:21:59.559907+0000","last_active":"2024-09-15T23:24:03.110384+0000","last_peered":"2024-09-15T23:24:03.110384+0000","last_clean":"2024-09-15T23:24:03.110384+0000","last_became_active":"2024-09-15T23:21:59.559440+0000","last_became_peered":"2024-09-15T23:21:59.559440+0000","last_unstale":"2024-09-15T23:24:03.110384+0000","last_undegraded":"2024-09-15T23:24:03.110384+0000","last_fullsized":"2024-09-15T23:24:03.110384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:00.755112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117658+0000","last_change":"2024-09-15T23:21:50.515424+0000","last_active":"2024-09-15T23:24:03.117658+0000","last_peered":"2024-09-15T23:24:03.117658+0000","last_clean":"2024-09-15T23:24:03.117658+0000","last_became_active":"2024-09-15T23:21:50.514966+0000","last_became_peered":"2024-09-15T23:21:50.514966+0000","last_unstale":"2024-09-15T23:24:03.117658+0000","last_undegraded":"2024-09-15T23:24:03.117658+0000","last_fullsized":"2024-09-15T23:24:03.117658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:57.586575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117341+0000","last_change":"2024-09-15T23:21:52.535189+0000","last_active":"2024-09-15T23:24:03.117341+0000","last_peered":"2024-09-15T23:24:03.117341+0000","last_clean":"2024-09-15T23:24:03.117341+0000","last_became_active":"2024-09-15T23:21:52.534807+0000","last_became_peered":"2024-09-15T23:21:52.534807+0000","last_unstale":"2024-09-15T23:24:03.117341+0000","last_undegraded":"2024-09-15T23:24:03.117341+0000","last_fullsized":"2024-09-15T23:24:03.117341+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:30.654382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110773+0000","last_change":"2024-09-15T23:21:59.564675+0000","last_active":"2024-09-15T23:24:03.110773+0000","last_peered":"2024-09-15T23:24:03.110773+0000","last_clean":"2024-09-15T23:24:03.110773+0000","last_became_active":"2024-09-15T23:21:59.564271+0000","last_became_peered":"2024-09-15T23:21:59.564271+0000","last_unstale":"2024-09-15T23:24:03.110773+0000","last_undegraded":"2024-09-15T23:24:03.110773+0000","last_fullsized":"2024-09-15T23:24:03.110773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:01.055369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117219+0000","last_change":"2024-09-15T23:21:50.507790+0000","last_active":"2024-09-15T23:24:03.117219+0000","last_peered":"2024-09-15T23:24:03.117219+0000","last_clean":"2024-09-15T23:24:03.117219+0000","last_became_active":"2024-09-15T23:21:50.507398+0000","last_became_peered":"2024-09-15T23:21:50.507398+0000","last_unstale":"2024-09-15T23:24:03.117219+0000","last_undegraded":"2024-09-15T23:24:03.117219+0000","last_fullsized":"2024-09-15T23:24:03.117219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:58:00.001279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116935+0000","last_change":"2024-09-15T23:21:52.537179+0000","last_active":"2024-09-15T23:24:03.116935+0000","last_peered":"2024-09-15T23:24:03.116935+0000","last_clean":"2024-09-15T23:24:03.116935+0000","last_became_active":"2024-09-15T23:21:52.536841+0000","last_became_peered":"2024-09-15T23:21:52.536841+0000","last_unstale":"2024-09-15T23:24:03.116935+0000","last_undegraded":"2024-09-15T23:24:03.116935+0000","last_fullsized":"2024-09-15T23:24:03.116935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:47.455700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110441+0000","last_change":"2024-09-15T23:21:59.564734+0000","last_active":"2024-09-15T23:24:03.110441+0000","last_peered":"2024-09-15T23:24:03.110441+0000","last_clean":"2024-09-15T23:24:03.110441+0000","last_became_active":"2024-09-15T23:21:59.564339+0000","last_became_peered":"2024-09-15T23:21:59.564339+0000","last_unstale":"2024-09-15T23:24:03.110441+0000","last_undegraded":"2024-09-15T23:24:03.110441+0000","last_fullsized":"2024-09-15T23:24:03.110441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:44.498480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113970+0000","last_change":"2024-09-15T23:21:52.538025+0000","last_active":"2024-09-15T23:24:03.113970+0000","last_peered":"2024-09-15T23:24:03.113970+0000","last_clean":"2024-09-15T23:24:03.113970+0000","last_became_active":"2024-09-15T23:21:52.537736+0000","last_became_peered":"2024-09-15T23:21:52.537736+0000","last_unstale":"2024-09-15T23:24:03.113970+0000","last_undegraded":"2024-09-15T23:24:03.113970+0000","last_fullsized":"2024-09-15T23:24:03.113970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:31.916888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.127260+0000","last_change":"2024-09-15T23:24:03.127260+0000","last_active":"2024-09-15T23:24:03.127260+0000","last_peered":"2024-09-15T23:24:03.127260+0000","last_clean":"2024-09-15T23:24:03.127260+0000","last_became_active":"2024-09-15T23:24:03.119384+0000","last_became_peered":"2024-09-15T23:24:03.119384+0000","last_unstale":"2024-09-15T23:24:03.127260+0000","last_undegraded":"2024-09-15T23:24:03.127260+0000","last_fullsized":"2024-09-15T23:24:03.127260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:35.325048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110625+0000","last_change":"2024-09-15T23:21:59.681971+0000","last_active":"2024-09-15T23:24:03.110625+0000","last_peered":"2024-09-15T23:24:03.110625+0000","last_clean":"2024-09-15T23:24:03.110625+0000","last_became_active":"2024-09-15T23:21:59.681432+0000","last_became_peered":"2024-09-15T23:21:59.681432+0000","last_unstale":"2024-09-15T23:24:03.110625+0000","last_undegraded":"2024-09-15T23:24:03.110625+0000","last_fullsized":"2024-09-15T23:24:03.110625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:26.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112179+0000","last_change":"2024-09-15T23:21:52.541281+0000","last_active":"2024-09-15T23:24:03.112179+0000","last_peered":"2024-09-15T23:24:03.112179+0000","last_clean":"2024-09-15T23:24:03.112179+0000","last_became_active":"2024-09-15T23:21:52.540772+0000","last_became_peered":"2024-09-15T23:21:52.540772+0000","last_unstale":"2024-09-15T23:24:03.112179+0000","last_undegraded":"2024-09-15T23:24:03.112179+0000","last_fullsized":"2024-09-15T23:24:03.112179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:23:44.444904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112700+0000","last_change":"2024-09-15T23:21:50.508844+0000","last_active":"2024-09-15T23:24:03.112700+0000","last_peered":"2024-09-15T23:24:03.112700+0000","last_clean":"2024-09-15T23:24:03.112700+0000","last_became_active":"2024-09-15T23:21:50.508364+0000","last_became_peered":"2024-09-15T23:21:50.508364+0000","last_unstale":"2024-09-15T23:24:03.112700+0000","last_undegraded":"2024-09-15T23:24:03.112700+0000","last_fullsized":"2024-09-15T23:24:03.112700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:45.129711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110720+0000","last_change":"2024-09-15T23:21:59.562114+0000","last_active":"2024-09-15T23:24:03.110720+0000","last_peered":"2024-09-15T23:24:03.110720+0000","last_clean":"2024-09-15T23:24:03.110720+0000","last_became_active":"2024-09-15T23:21:59.561652+0000","last_became_peered":"2024-09-15T23:21:59.561652+0000","last_unstale":"2024-09-15T23:24:03.110720+0000","last_undegraded":"2024-09-15T23:24:03.110720+0000","last_fullsized":"2024-09-15T23:24:03.110720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:51.781886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.531678+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.531327+0000","last_became_peered":"2024-09-15T23:21:52.531327+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:07.152626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112193+0000","last_change":"2024-09-15T23:21:50.513197+0000","last_active":"2024-09-15T23:24:03.112193+0000","last_peered":"2024-09-15T23:24:03.112193+0000","last_clean":"2024-09-15T23:24:03.112193+0000","last_became_active":"2024-09-15T23:21:50.512794+0000","last_became_peered":"2024-09-15T23:21:50.512794+0000","last_unstale":"2024-09-15T23:24:03.112193+0000","last_undegraded":"2024-09-15T23:24:03.112193+0000","last_fullsized":"2024-09-15T23:24:03.112193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:05.457467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194628+0000","last_change":"2024-09-15T23:21:59.565669+0000","last_active":"2024-09-15T23:24:03.194628+0000","last_peered":"2024-09-15T23:24:03.194628+0000","last_clean":"2024-09-15T23:24:03.194628+0000","last_became_active":"2024-09-15T23:21:59.565290+0000","last_became_peered":"2024-09-15T23:21:59.565290+0000","last_unstale":"2024-09-15T23:24:03.194628+0000","last_undegraded":"2024-09-15T23:24:03.194628+0000","last_fullsized":"2024-09-15T23:24:03.194628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:04.861560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116394+0000","last_change":"2024-09-15T23:21:52.529643+0000","last_active":"2024-09-15T23:24:03.116394+0000","last_peered":"2024-09-15T23:24:03.116394+0000","last_clean":"2024-09-15T23:24:03.116394+0000","last_became_active":"2024-09-15T23:21:52.529304+0000","last_became_peered":"2024-09-15T23:21:52.529304+0000","last_unstale":"2024-09-15T23:24:03.116394+0000","last_undegraded":"2024-09-15T23:24:03.116394+0000","last_fullsized":"2024-09-15T23:24:03.116394+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:18:37.758403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117052+0000","last_change":"2024-09-15T23:21:50.562114+0000","last_active":"2024-09-15T23:24:03.117052+0000","last_peered":"2024-09-15T23:24:03.117052+0000","last_clean":"2024-09-15T23:24:03.117052+0000","last_became_active":"2024-09-15T23:21:50.561772+0000","last_became_peered":"2024-09-15T23:21:50.561772+0000","last_unstale":"2024-09-15T23:24:03.117052+0000","last_undegraded":"2024-09-15T23:24:03.117052+0000","last_fullsized":"2024-09-15T23:24:03.117052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:14.806646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110672+0000","last_change":"2024-09-15T23:21:59.681933+0000","last_active":"2024-09-15T23:24:03.110672+0000","last_peered":"2024-09-15T23:24:03.110672+0000","last_clean":"2024-09-15T23:24:03.110672+0000","last_became_active":"2024-09-15T23:21:59.681528+0000","last_became_peered":"2024-09-15T23:21:59.681528+0000","last_unstale":"2024-09-15T23:24:03.110672+0000","last_undegraded":"2024-09-15T23:24:03.110672+0000","last_fullsized":"2024-09-15T23:24:03.110672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:46.199540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194326+0000","last_change":"2024-09-15T23:21:52.529548+0000","last_active":"2024-09-15T23:24:03.194326+0000","last_peered":"2024-09-15T23:24:03.194326+0000","last_clean":"2024-09-15T23:24:03.194326+0000","last_became_active":"2024-09-15T23:21:52.529253+0000","last_became_peered":"2024-09-15T23:21:52.529253+0000","last_unstale":"2024-09-15T23:24:03.194326+0000","last_undegraded":"2024-09-15T23:24:03.194326+0000","last_fullsized":"2024-09-15T23:24:03.194326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:22.813395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111022+0000","last_change":"2024-09-15T23:21:50.767759+0000","last_active":"2024-09-15T23:24:03.111022+0000","last_peered":"2024-09-15T23:24:03.111022+0000","last_clean":"2024-09-15T23:24:03.111022+0000","last_became_active":"2024-09-15T23:21:50.767388+0000","last_became_peered":"2024-09-15T23:21:50.767388+0000","last_unstale":"2024-09-15T23:24:03.111022+0000","last_undegraded":"2024-09-15T23:24:03.111022+0000","last_fullsized":"2024-09-15T23:24:03.111022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:36.713769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112512+0000","last_change":"2024-09-15T23:21:59.557951+0000","last_active":"2024-09-15T23:24:03.112512+0000","last_peered":"2024-09-15T23:24:03.112512+0000","last_clean":"2024-09-15T23:24:03.112512+0000","last_became_active":"2024-09-15T23:21:59.557490+0000","last_became_peered":"2024-09-15T23:21:59.557490+0000","last_unstale":"2024-09-15T23:24:03.112512+0000","last_undegraded":"2024-09-15T23:24:03.112512+0000","last_fullsized":"2024-09-15T23:24:03.112512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:17.147389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112859+0000","last_change":"2024-09-15T23:21:52.531638+0000","last_active":"2024-09-15T23:24:03.112859+0000","last_peered":"2024-09-15T23:24:03.112859+0000","last_clean":"2024-09-15T23:24:03.112859+0000","last_became_active":"2024-09-15T23:21:52.531301+0000","last_became_peered":"2024-09-15T23:21:52.531301+0000","last_unstale":"2024-09-15T23:24:03.112859+0000","last_undegraded":"2024-09-15T23:24:03.112859+0000","last_fullsized":"2024-09-15T23:24:03.112859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.436286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'16","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194603+0000","last_change":"2024-09-15T23:21:50.508470+0000","last_active":"2024-09-15T23:24:03.194603+0000","last_peered":"2024-09-15T23:24:03.194603+0000","last_clean":"2024-09-15T23:24:03.194603+0000","last_became_active":"2024-09-15T23:21:50.508060+0000","last_became_peered":"2024-09-15T23:21:50.508060+0000","last_unstale":"2024-09-15T23:24:03.194603+0000","last_undegraded":"2024-09-15T23:24:03.194603+0000","last_fullsized":"2024-09-15T23:24:03.194603+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:59.542252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111992+0000","last_change":"2024-09-15T23:21:59.682019+0000","last_active":"2024-09-15T23:24:03.111992+0000","last_peered":"2024-09-15T23:24:03.111992+0000","last_clean":"2024-09-15T23:24:03.111992+0000","last_became_active":"2024-09-15T23:21:59.681467+0000","last_became_peered":"2024-09-15T23:21:59.681467+0000","last_unstale":"2024-09-15T23:24:03.111992+0000","last_undegraded":"2024-09-15T23:24:03.111992+0000","last_fullsized":"2024-09-15T23:24:03.111992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:27.650382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109766+0000","last_change":"2024-09-15T23:21:52.536345+0000","last_active":"2024-09-15T23:24:03.109766+0000","last_peered":"2024-09-15T23:24:03.109766+0000","last_clean":"2024-09-15T23:24:03.109766+0000","last_became_active":"2024-09-15T23:21:52.536027+0000","last_became_peered":"2024-09-15T23:21:52.536027+0000","last_unstale":"2024-09-15T23:24:03.109766+0000","last_undegraded":"2024-09-15T23:24:03.109766+0000","last_fullsized":"2024-09-15T23:24:03.109766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:16.435912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117716+0000","last_change":"2024-09-15T23:21:50.562945+0000","last_active":"2024-09-15T23:24:03.117716+0000","last_peered":"2024-09-15T23:24:03.117716+0000","last_clean":"2024-09-15T23:24:03.117716+0000","last_became_active":"2024-09-15T23:21:50.561632+0000","last_became_peered":"2024-09-15T23:21:50.561632+0000","last_unstale":"2024-09-15T23:24:03.117716+0000","last_undegraded":"2024-09-15T23:24:03.117716+0000","last_fullsized":"2024-09-15T23:24:03.117716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:44.545489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115268+0000","last_change":"2024-09-15T23:21:59.564793+0000","last_active":"2024-09-15T23:24:03.115268+0000","last_peered":"2024-09-15T23:24:03.115268+0000","last_clean":"2024-09-15T23:24:03.115268+0000","last_became_active":"2024-09-15T23:21:59.564140+0000","last_became_peered":"2024-09-15T23:21:59.564140+0000","last_unstale":"2024-09-15T23:24:03.115268+0000","last_undegraded":"2024-09-15T23:24:03.115268+0000","last_fullsized":"2024-09-15T23:24:03.115268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:41.464962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113451+0000","last_change":"2024-09-15T23:21:52.533521+0000","last_active":"2024-09-15T23:24:03.113451+0000","last_peered":"2024-09-15T23:24:03.113451+0000","last_clean":"2024-09-15T23:24:03.113451+0000","last_became_active":"2024-09-15T23:21:52.532195+0000","last_became_peered":"2024-09-15T23:21:52.532195+0000","last_unstale":"2024-09-15T23:24:03.113451+0000","last_undegraded":"2024-09-15T23:24:03.113451+0000","last_fullsized":"2024-09-15T23:24:03.113451+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:14.229018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194243+0000","last_change":"2024-09-15T23:21:50.562893+0000","last_active":"2024-09-15T23:24:03.194243+0000","last_peered":"2024-09-15T23:24:03.194243+0000","last_clean":"2024-09-15T23:24:03.194243+0000","last_became_active":"2024-09-15T23:21:50.562516+0000","last_became_peered":"2024-09-15T23:21:50.562516+0000","last_unstale":"2024-09-15T23:24:03.194243+0000","last_undegraded":"2024-09-15T23:24:03.194243+0000","last_fullsized":"2024-09-15T23:24:03.194243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:47:47.442458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117856+0000","last_change":"2024-09-15T23:21:59.682091+0000","last_active":"2024-09-15T23:24:03.117856+0000","last_peered":"2024-09-15T23:24:03.117856+0000","last_clean":"2024-09-15T23:24:03.117856+0000","last_became_active":"2024-09-15T23:21:59.681574+0000","last_became_peered":"2024-09-15T23:21:59.681574+0000","last_unstale":"2024-09-15T23:24:03.117856+0000","last_undegraded":"2024-09-15T23:24:03.117856+0000","last_fullsized":"2024-09-15T23:24:03.117856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:04.991608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114426+0000","last_change":"2024-09-15T23:21:52.533811+0000","last_active":"2024-09-15T23:24:03.114426+0000","last_peered":"2024-09-15T23:24:03.114426+0000","last_clean":"2024-09-15T23:24:03.114426+0000","last_became_active":"2024-09-15T23:21:52.533487+0000","last_became_peered":"2024-09-15T23:21:52.533487+0000","last_unstale":"2024-09-15T23:24:03.114426+0000","last_undegraded":"2024-09-15T23:24:03.114426+0000","last_fullsized":"2024-09-15T23:24:03.114426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:44.193212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.281895+0000","last_change":"2024-09-15T23:21:50.510883+0000","last_active":"2024-09-15T23:24:20.281895+0000","last_peered":"2024-09-15T23:24:20.281895+0000","last_clean":"2024-09-15T23:24:20.281895+0000","last_became_active":"2024-09-15T23:21:50.510478+0000","last_became_peered":"2024-09-15T23:21:50.510478+0000","last_unstale":"2024-09-15T23:24:20.281895+0000","last_undegraded":"2024-09-15T23:24:20.281895+0000","last_fullsized":"2024-09-15T23:24:20.281895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:35.454885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109705+0000","last_change":"2024-09-15T23:21:59.561816+0000","last_active":"2024-09-15T23:24:03.109705+0000","last_peered":"2024-09-15T23:24:03.109705+0000","last_clean":"2024-09-15T23:24:03.109705+0000","last_became_active":"2024-09-15T23:21:59.561434+0000","last_became_peered":"2024-09-15T23:21:59.561434+0000","last_unstale":"2024-09-15T23:24:03.109705+0000","last_undegraded":"2024-09-15T23:24:03.109705+0000","last_fullsized":"2024-09-15T23:24:03.109705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:59.945802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117955+0000","last_change":"2024-09-15T23:23:02.277204+0000","last_active":"2024-09-15T23:24:03.117955+0000","last_peered":"2024-09-15T23:24:03.117955+0000","last_clean":"2024-09-15T23:24:03.117955+0000","last_became_active":"2024-09-15T23:23:02.276770+0000","last_became_peered":"2024-09-15T23:23:02.276770+0000","last_unstale":"2024-09-15T23:24:03.117955+0000","last_undegraded":"2024-09-15T23:24:03.117955+0000","last_fullsized":"2024-09-15T23:24:03.117955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:11.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":150,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.145290+0000","last_change":"2024-09-15T23:21:50.561684+0000","last_active":"2024-09-15T23:24:20.145290+0000","last_peered":"2024-09-15T23:24:20.145290+0000","last_clean":"2024-09-15T23:24:20.145290+0000","last_became_active":"2024-09-15T23:21:50.561335+0000","last_became_peered":"2024-09-15T23:21:50.561335+0000","last_unstale":"2024-09-15T23:24:20.145290+0000","last_undegraded":"2024-09-15T23:24:20.145290+0000","last_fullsized":"2024-09-15T23:24:20.145290+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:23.897250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109863+0000","last_change":"2024-09-15T23:21:59.681858+0000","last_active":"2024-09-15T23:24:03.109863+0000","last_peered":"2024-09-15T23:24:03.109863+0000","last_clean":"2024-09-15T23:24:03.109863+0000","last_became_active":"2024-09-15T23:21:59.681441+0000","last_became_peered":"2024-09-15T23:21:59.681441+0000","last_unstale":"2024-09-15T23:24:03.109863+0000","last_undegraded":"2024-09-15T23:24:03.109863+0000","last_fullsized":"2024-09-15T23:24:03.109863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:28.513719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109320+0000","last_change":"2024-09-15T23:21:52.536141+0000","last_active":"2024-09-15T23:24:03.109320+0000","last_peered":"2024-09-15T23:24:03.109320+0000","last_clean":"2024-09-15T23:24:03.109320+0000","last_became_active":"2024-09-15T23:21:52.535832+0000","last_became_peered":"2024-09-15T23:21:52.535832+0000","last_unstale":"2024-09-15T23:24:03.109320+0000","last_undegraded":"2024-09-15T23:24:03.109320+0000","last_fullsized":"2024-09-15T23:24:03.109320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:21:44.341412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.123671+0000","last_change":"2024-09-15T23:24:03.123671+0000","last_active":"2024-09-15T23:24:03.123671+0000","last_peered":"2024-09-15T23:24:03.123671+0000","last_clean":"2024-09-15T23:24:03.123671+0000","last_became_active":"2024-09-15T23:24:03.119585+0000","last_became_peered":"2024-09-15T23:24:03.119585+0000","last_unstale":"2024-09-15T23:24:03.123671+0000","last_undegraded":"2024-09-15T23:24:03.123671+0000","last_fullsized":"2024-09-15T23:24:03.123671+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:30.734156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118359+0000","last_change":"2024-09-15T23:21:59.682118+0000","last_active":"2024-09-15T23:24:03.118359+0000","last_peered":"2024-09-15T23:24:03.118359+0000","last_clean":"2024-09-15T23:24:03.118359+0000","last_became_active":"2024-09-15T23:21:59.681500+0000","last_became_peered":"2024-09-15T23:21:59.681500+0000","last_unstale":"2024-09-15T23:24:03.118359+0000","last_undegraded":"2024-09-15T23:24:03.118359+0000","last_fullsized":"2024-09-15T23:24:03.118359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:58.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112298+0000","last_change":"2024-09-15T23:21:52.540458+0000","last_active":"2024-09-15T23:24:03.112298+0000","last_peered":"2024-09-15T23:24:03.112298+0000","last_clean":"2024-09-15T23:24:03.112298+0000","last_became_active":"2024-09-15T23:21:52.540112+0000","last_became_peered":"2024-09-15T23:21:52.540112+0000","last_unstale":"2024-09-15T23:24:03.112298+0000","last_undegraded":"2024-09-15T23:24:03.112298+0000","last_fullsized":"2024-09-15T23:24:03.112298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:15.240785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111097+0000","last_change":"2024-09-15T23:21:50.513124+0000","last_active":"2024-09-15T23:24:03.111097+0000","last_peered":"2024-09-15T23:24:03.111097+0000","last_clean":"2024-09-15T23:24:03.111097+0000","last_became_active":"2024-09-15T23:21:50.512705+0000","last_became_peered":"2024-09-15T23:21:50.512705+0000","last_unstale":"2024-09-15T23:24:03.111097+0000","last_undegraded":"2024-09-15T23:24:03.111097+0000","last_fullsized":"2024-09-15T23:24:03.111097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.417937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":354,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113701+0000","last_change":"2024-09-15T23:20:40.656652+0000","last_active":"2024-09-15T23:24:03.113701+0000","last_peered":"2024-09-15T23:24:03.113701+0000","last_clean":"2024-09-15T23:24:03.113701+0000","last_became_active":"2024-09-15T23:20:40.656266+0000","last_became_peered":"2024-09-15T23:20:40.656266+0000","last_unstale":"2024-09-15T23:24:03.113701+0000","last_undegraded":"2024-09-15T23:24:03.113701+0000","last_fullsized":"2024-09-15T23:24:03.113701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_clean_scrub_stamp":"2024-09-15T23:17:15.186850+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:42.317006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113728+0000","last_change":"2024-09-15T23:21:59.563777+0000","last_active":"2024-09-15T23:24:03.113728+0000","last_peered":"2024-09-15T23:24:03.113728+0000","last_clean":"2024-09-15T23:24:03.113728+0000","last_became_active":"2024-09-15T23:21:59.563320+0000","last_became_peered":"2024-09-15T23:21:59.563320+0000","last_unstale":"2024-09-15T23:24:03.113728+0000","last_undegraded":"2024-09-15T23:24:03.113728+0000","last_fullsized":"2024-09-15T23:24:03.113728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:25:07.791923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114145+0000","last_change":"2024-09-15T23:21:52.539842+0000","last_active":"2024-09-15T23:24:03.114145+0000","last_peered":"2024-09-15T23:24:03.114145+0000","last_clean":"2024-09-15T23:24:03.114145+0000","last_became_active":"2024-09-15T23:21:52.538683+0000","last_became_peered":"2024-09-15T23:21:52.538683+0000","last_unstale":"2024-09-15T23:24:03.114145+0000","last_undegraded":"2024-09-15T23:24:03.114145+0000","last_fullsized":"2024-09-15T23:24:03.114145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:43.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":149,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:18.280187+0000","last_change":"2024-09-15T23:21:50.767601+0000","last_active":"2024-09-15T23:24:18.280187+0000","last_peered":"2024-09-15T23:24:18.280187+0000","last_clean":"2024-09-15T23:24:18.280187+0000","last_became_active":"2024-09-15T23:21:50.767149+0000","last_became_peered":"2024-09-15T23:21:50.767149+0000","last_unstale":"2024-09-15T23:24:18.280187+0000","last_undegraded":"2024-09-15T23:24:18.280187+0000","last_fullsized":"2024-09-15T23:24:18.280187+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:23.636551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117168+0000","last_change":"2024-09-15T23:21:59.563069+0000","last_active":"2024-09-15T23:24:03.117168+0000","last_peered":"2024-09-15T23:24:03.117168+0000","last_clean":"2024-09-15T23:24:03.117168+0000","last_became_active":"2024-09-15T23:21:59.562658+0000","last_became_peered":"2024-09-15T23:21:59.562658+0000","last_unstale":"2024-09-15T23:24:03.117168+0000","last_undegraded":"2024-09-15T23:24:03.117168+0000","last_fullsized":"2024-09-15T23:24:03.117168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:23.067198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113535+0000","last_change":"2024-09-15T23:21:52.535289+0000","last_active":"2024-09-15T23:24:03.113535+0000","last_peered":"2024-09-15T23:24:03.113535+0000","last_clean":"2024-09-15T23:24:03.113535+0000","last_became_active":"2024-09-15T23:21:52.534885+0000","last_became_peered":"2024-09-15T23:21:52.534885+0000","last_unstale":"2024-09-15T23:24:03.113535+0000","last_undegraded":"2024-09-15T23:24:03.113535+0000","last_fullsized":"2024-09-15T23:24:03.113535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:42.293786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118612+0000","last_change":"2024-09-15T23:21:50.513246+0000","last_active":"2024-09-15T23:24:03.118612+0000","last_peered":"2024-09-15T23:24:03.118612+0000","last_clean":"2024-09-15T23:24:03.118612+0000","last_became_active":"2024-09-15T23:21:50.512740+0000","last_became_peered":"2024-09-15T23:21:50.512740+0000","last_unstale":"2024-09-15T23:24:03.118612+0000","last_undegraded":"2024-09-15T23:24:03.118612+0000","last_fullsized":"2024-09-15T23:24:03.118612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:59.201970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193756+0000","last_change":"2024-09-15T23:21:59.568565+0000","last_active":"2024-09-15T23:24:03.193756+0000","last_peered":"2024-09-15T23:24:03.193756+0000","last_clean":"2024-09-15T23:24:03.193756+0000","last_became_active":"2024-09-15T23:21:59.567789+0000","last_became_peered":"2024-09-15T23:21:59.567789+0000","last_unstale":"2024-09-15T23:24:03.193756+0000","last_undegraded":"2024-09-15T23:24:03.193756+0000","last_fullsized":"2024-09-15T23:24:03.193756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:02.106971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114106+0000","last_change":"2024-09-15T23:21:52.533727+0000","last_active":"2024-09-15T23:24:03.114106+0000","last_peered":"2024-09-15T23:24:03.114106+0000","last_clean":"2024-09-15T23:24:03.114106+0000","last_became_active":"2024-09-15T23:21:52.532350+0000","last_became_peered":"2024-09-15T23:21:52.532350+0000","last_unstale":"2024-09-15T23:24:03.114106+0000","last_undegraded":"2024-09-15T23:24:03.114106+0000","last_fullsized":"2024-09-15T23:24:03.114106+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:03.624837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.124878+0000","last_change":"2024-09-15T23:24:03.124430+0000","last_active":"2024-09-15T23:24:03.124878+0000","last_peered":"2024-09-15T23:24:03.124878+0000","last_clean":"2024-09-15T23:24:03.124878+0000","last_became_active":"2024-09-15T23:24:03.119042+0000","last_became_peered":"2024-09-15T23:24:03.119042+0000","last_unstale":"2024-09-15T23:24:03.124878+0000","last_undegraded":"2024-09-15T23:24:03.124878+0000","last_fullsized":"2024-09-15T23:24:03.124878+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:52.065543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117798+0000","last_change":"2024-09-15T23:21:59.566429+0000","last_active":"2024-09-15T23:24:03.117798+0000","last_peered":"2024-09-15T23:24:03.117798+0000","last_clean":"2024-09-15T23:24:03.117798+0000","last_became_active":"2024-09-15T23:21:59.565835+0000","last_became_peered":"2024-09-15T23:21:59.565835+0000","last_unstale":"2024-09-15T23:24:03.117798+0000","last_undegraded":"2024-09-15T23:24:03.117798+0000","last_fullsized":"2024-09-15T23:24:03.117798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:38.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116276+0000","last_change":"2024-09-15T23:21:52.539544+0000","last_active":"2024-09-15T23:24:03.116276+0000","last_peered":"2024-09-15T23:24:03.116276+0000","last_clean":"2024-09-15T23:24:03.116276+0000","last_became_active":"2024-09-15T23:21:52.539203+0000","last_became_peered":"2024-09-15T23:21:52.539203+0000","last_unstale":"2024-09-15T23:24:03.116276+0000","last_undegraded":"2024-09-15T23:24:03.116276+0000","last_fullsized":"2024-09-15T23:24:03.116276+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.193965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:07.046123+0000","last_change":"2024-09-15T23:21:50.513543+0000","last_active":"2024-09-15T23:24:07.046123+0000","last_peered":"2024-09-15T23:24:07.046123+0000","last_clean":"2024-09-15T23:24:07.046123+0000","last_became_active":"2024-09-15T23:21:50.512812+0000","last_became_peered":"2024-09-15T23:21:50.512812+0000","last_unstale":"2024-09-15T23:24:07.046123+0000","last_undegraded":"2024-09-15T23:24:07.046123+0000","last_fullsized":"2024-09-15T23:24:07.046123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:50.954096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117689+0000","last_change":"2024-09-15T23:21:59.570163+0000","last_active":"2024-09-15T23:24:03.117689+0000","last_peered":"2024-09-15T23:24:03.117689+0000","last_clean":"2024-09-15T23:24:03.117689+0000","last_became_active":"2024-09-15T23:21:59.569805+0000","last_became_peered":"2024-09-15T23:21:59.569805+0000","last_unstale":"2024-09-15T23:24:03.117689+0000","last_undegraded":"2024-09-15T23:24:03.117689+0000","last_fullsized":"2024-09-15T23:24:03.117689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:28.530817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117877+0000","last_change":"2024-09-15T23:21:52.537850+0000","last_active":"2024-09-15T23:24:03.117877+0000","last_peered":"2024-09-15T23:24:03.117877+0000","last_clean":"2024-09-15T23:24:03.117877+0000","last_became_active":"2024-09-15T23:21:52.536579+0000","last_became_peered":"2024-09-15T23:21:52.536579+0000","last_unstale":"2024-09-15T23:24:03.117877+0000","last_undegraded":"2024-09-15T23:24:03.117877+0000","last_fullsized":"2024-09-15T23:24:03.117877+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:06.353615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118588+0000","last_change":"2024-09-15T23:21:50.562221+0000","last_active":"2024-09-15T23:24:03.118588+0000","last_peered":"2024-09-15T23:24:03.118588+0000","last_clean":"2024-09-15T23:24:03.118588+0000","last_became_active":"2024-09-15T23:21:50.561788+0000","last_became_peered":"2024-09-15T23:21:50.561788+0000","last_unstale":"2024-09-15T23:24:03.118588+0000","last_undegraded":"2024-09-15T23:24:03.118588+0000","last_fullsized":"2024-09-15T23:24:03.118588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.853812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109810+0000","last_change":"2024-09-15T23:23:02.383003+0000","last_active":"2024-09-15T23:24:03.109810+0000","last_peered":"2024-09-15T23:24:03.109810+0000","last_clean":"2024-09-15T23:24:03.109810+0000","last_became_active":"2024-09-15T23:23:02.382629+0000","last_became_peered":"2024-09-15T23:23:02.382629+0000","last_unstale":"2024-09-15T23:24:03.109810+0000","last_undegraded":"2024-09-15T23:24:03.109810+0000","last_fullsized":"2024-09-15T23:24:03.109810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:50.114689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113567+0000","last_change":"2024-09-15T23:21:52.539566+0000","last_active":"2024-09-15T23:24:03.113567+0000","last_peered":"2024-09-15T23:24:03.113567+0000","last_clean":"2024-09-15T23:24:03.113567+0000","last_became_active":"2024-09-15T23:21:52.539219+0000","last_became_peered":"2024-09-15T23:21:52.539219+0000","last_unstale":"2024-09-15T23:24:03.113567+0000","last_undegraded":"2024-09-15T23:24:03.113567+0000","last_fullsized":"2024-09-15T23:24:03.113567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:41.266308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":167,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.145816+0000","last_change":"2024-09-15T23:24:03.127667+0000","last_active":"2024-09-15T23:24:20.145816+0000","last_peered":"2024-09-15T23:24:20.145816+0000","last_clean":"2024-09-15T23:24:20.145816+0000","last_became_active":"2024-09-15T23:24:03.121613+0000","last_became_peered":"2024-09-15T23:24:03.121613+0000","last_unstale":"2024-09-15T23:24:20.145816+0000","last_undegraded":"2024-09-15T23:24:20.145816+0000","last_fullsized":"2024-09-15T23:24:20.145816+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:54:24.166895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":4,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1122,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194597+0000","last_change":"2024-09-15T23:21:59.568087+0000","last_active":"2024-09-15T23:24:03.194597+0000","last_peered":"2024-09-15T23:24:03.194597+0000","last_clean":"2024-09-15T23:24:03.194597+0000","last_became_active":"2024-09-15T23:21:59.567693+0000","last_became_peered":"2024-09-15T23:21:59.567693+0000","last_unstale":"2024-09-15T23:24:03.194597+0000","last_undegraded":"2024-09-15T23:24:03.194597+0000","last_fullsized":"2024-09-15T23:24:03.194597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.231873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114396+0000","last_change":"2024-09-15T23:21:52.531090+0000","last_active":"2024-09-15T23:24:03.114396+0000","last_peered":"2024-09-15T23:24:03.114396+0000","last_clean":"2024-09-15T23:24:03.114396+0000","last_became_active":"2024-09-15T23:21:52.530738+0000","last_became_peered":"2024-09-15T23:21:52.530738+0000","last_unstale":"2024-09-15T23:24:03.114396+0000","last_undegraded":"2024-09-15T23:24:03.114396+0000","last_fullsized":"2024-09-15T23:24:03.114396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:35.325128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117423+0000","last_change":"2024-09-15T23:21:50.514100+0000","last_active":"2024-09-15T23:24:03.117423+0000","last_peered":"2024-09-15T23:24:03.117423+0000","last_clean":"2024-09-15T23:24:03.117423+0000","last_became_active":"2024-09-15T23:21:50.513629+0000","last_became_peered":"2024-09-15T23:21:50.513629+0000","last_unstale":"2024-09-15T23:24:03.117423+0000","last_undegraded":"2024-09-15T23:24:03.117423+0000","last_fullsized":"2024-09-15T23:24:03.117423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.187544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115275+0000","last_change":"2024-09-15T23:21:59.562804+0000","last_active":"2024-09-15T23:24:03.115275+0000","last_peered":"2024-09-15T23:24:03.115275+0000","last_clean":"2024-09-15T23:24:03.115275+0000","last_became_active":"2024-09-15T23:21:59.562368+0000","last_became_peered":"2024-09-15T23:21:59.562368+0000","last_unstale":"2024-09-15T23:24:03.115275+0000","last_undegraded":"2024-09-15T23:24:03.115275+0000","last_fullsized":"2024-09-15T23:24:03.115275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:21.137333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109909+0000","last_change":"2024-09-15T23:21:52.530562+0000","last_active":"2024-09-15T23:24:03.109909+0000","last_peered":"2024-09-15T23:24:03.109909+0000","last_clean":"2024-09-15T23:24:03.109909+0000","last_became_active":"2024-09-15T23:21:52.530280+0000","last_became_peered":"2024-09-15T23:21:52.530280+0000","last_unstale":"2024-09-15T23:24:03.109909+0000","last_undegraded":"2024-09-15T23:24:03.109909+0000","last_fullsized":"2024-09-15T23:24:03.109909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:26.835651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113202+0000","last_change":"2024-09-15T23:21:50.513179+0000","last_active":"2024-09-15T23:24:03.113202+0000","last_peered":"2024-09-15T23:24:03.113202+0000","last_clean":"2024-09-15T23:24:03.113202+0000","last_became_active":"2024-09-15T23:21:50.512790+0000","last_became_peered":"2024-09-15T23:21:50.512790+0000","last_unstale":"2024-09-15T23:24:03.113202+0000","last_undegraded":"2024-09-15T23:24:03.113202+0000","last_fullsized":"2024-09-15T23:24:03.113202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:56.408198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113316+0000","last_change":"2024-09-15T23:23:02.382964+0000","last_active":"2024-09-15T23:24:03.113316+0000","last_peered":"2024-09-15T23:24:03.113316+0000","last_clean":"2024-09-15T23:24:03.113316+0000","last_became_active":"2024-09-15T23:23:02.382556+0000","last_became_peered":"2024-09-15T23:23:02.382556+0000","last_unstale":"2024-09-15T23:24:03.113316+0000","last_undegraded":"2024-09-15T23:24:03.113316+0000","last_fullsized":"2024-09-15T23:24:03.113316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:49.211591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117484+0000","last_change":"2024-09-15T23:21:52.530970+0000","last_active":"2024-09-15T23:24:03.117484+0000","last_peered":"2024-09-15T23:24:03.117484+0000","last_clean":"2024-09-15T23:24:03.117484+0000","last_became_active":"2024-09-15T23:21:52.530636+0000","last_became_peered":"2024-09-15T23:21:52.530636+0000","last_unstale":"2024-09-15T23:24:03.117484+0000","last_undegraded":"2024-09-15T23:24:03.117484+0000","last_fullsized":"2024-09-15T23:24:03.117484+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:33.374658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'40","reported_seq":316,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.146278+0000","last_change":"2024-09-15T23:24:03.131527+0000","last_active":"2024-09-15T23:24:20.146278+0000","last_peered":"2024-09-15T23:24:20.146278+0000","last_clean":"2024-09-15T23:24:20.146278+0000","last_became_active":"2024-09-15T23:24:03.119053+0000","last_became_peered":"2024-09-15T23:24:03.119053+0000","last_unstale":"2024-09-15T23:24:20.146278+0000","last_undegraded":"2024-09-15T23:24:20.146278+0000","last_fullsized":"2024-09-15T23:24:20.146278+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:21.870435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68399,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":214,"num_read_kb":169,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":62325,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113841+0000","last_change":"2024-09-15T23:23:02.277457+0000","last_active":"2024-09-15T23:24:03.113841+0000","last_peered":"2024-09-15T23:24:03.113841+0000","last_clean":"2024-09-15T23:24:03.113841+0000","last_became_active":"2024-09-15T23:23:02.276960+0000","last_became_peered":"2024-09-15T23:23:02.276960+0000","last_unstale":"2024-09-15T23:24:03.113841+0000","last_undegraded":"2024-09-15T23:24:03.113841+0000","last_fullsized":"2024-09-15T23:24:03.113841+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:15:28.823804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117534+0000","last_change":"2024-09-15T23:21:52.530541+0000","last_active":"2024-09-15T23:24:03.117534+0000","last_peered":"2024-09-15T23:24:03.117534+0000","last_clean":"2024-09-15T23:24:03.117534+0000","last_became_active":"2024-09-15T23:21:52.530217+0000","last_became_peered":"2024-09-15T23:21:52.530217+0000","last_unstale":"2024-09-15T23:24:03.117534+0000","last_undegraded":"2024-09-15T23:24:03.117534+0000","last_fullsized":"2024-09-15T23:24:03.117534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:14.864563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118500+0000","last_change":"2024-09-15T23:21:50.562203+0000","last_active":"2024-09-15T23:24:03.118500+0000","last_peered":"2024-09-15T23:24:03.118500+0000","last_clean":"2024-09-15T23:24:03.118500+0000","last_became_active":"2024-09-15T23:21:50.561827+0000","last_became_peered":"2024-09-15T23:21:50.561827+0000","last_unstale":"2024-09-15T23:24:03.118500+0000","last_undegraded":"2024-09-15T23:24:03.118500+0000","last_fullsized":"2024-09-15T23:24:03.118500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:22.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111024+0000","last_change":"2024-09-15T23:21:59.561013+0000","last_active":"2024-09-15T23:24:03.111024+0000","last_peered":"2024-09-15T23:24:03.111024+0000","last_clean":"2024-09-15T23:24:03.111024+0000","last_became_active":"2024-09-15T23:21:59.560665+0000","last_became_peered":"2024-09-15T23:21:59.560665+0000","last_unstale":"2024-09-15T23:24:03.111024+0000","last_undegraded":"2024-09-15T23:24:03.111024+0000","last_fullsized":"2024-09-15T23:24:03.111024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:00.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194109+0000","last_change":"2024-09-15T23:21:52.529922+0000","last_active":"2024-09-15T23:24:03.194109+0000","last_peered":"2024-09-15T23:24:03.194109+0000","last_clean":"2024-09-15T23:24:03.194109+0000","last_became_active":"2024-09-15T23:21:52.529179+0000","last_became_peered":"2024-09-15T23:21:52.529179+0000","last_unstale":"2024-09-15T23:24:03.194109+0000","last_undegraded":"2024-09-15T23:24:03.194109+0000","last_fullsized":"2024-09-15T23:24:03.194109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.077658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113481+0000","last_change":"2024-09-15T23:21:50.514420+0000","last_active":"2024-09-15T23:24:03.113481+0000","last_peered":"2024-09-15T23:24:03.113481+0000","last_clean":"2024-09-15T23:24:03.113481+0000","last_became_active":"2024-09-15T23:21:50.514007+0000","last_became_peered":"2024-09-15T23:21:50.514007+0000","last_unstale":"2024-09-15T23:24:03.113481+0000","last_undegraded":"2024-09-15T23:24:03.113481+0000","last_fullsized":"2024-09-15T23:24:03.113481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:36.845674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193335+0000","last_change":"2024-09-15T23:21:59.565978+0000","last_active":"2024-09-15T23:24:03.193335+0000","last_peered":"2024-09-15T23:24:03.193335+0000","last_clean":"2024-09-15T23:24:03.193335+0000","last_became_active":"2024-09-15T23:21:59.565604+0000","last_became_peered":"2024-09-15T23:21:59.565604+0000","last_unstale":"2024-09-15T23:24:03.193335+0000","last_undegraded":"2024-09-15T23:24:03.193335+0000","last_fullsized":"2024-09-15T23:24:03.193335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:47.089504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118064+0000","last_change":"2024-09-15T23:21:52.543511+0000","last_active":"2024-09-15T23:24:03.118064+0000","last_peered":"2024-09-15T23:24:03.118064+0000","last_clean":"2024-09-15T23:24:03.118064+0000","last_became_active":"2024-09-15T23:21:52.543169+0000","last_became_peered":"2024-09-15T23:21:52.543169+0000","last_unstale":"2024-09-15T23:24:03.118064+0000","last_undegraded":"2024-09-15T23:24:03.118064+0000","last_fullsized":"2024-09-15T23:24:03.118064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:05.696772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":252,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:20.282085+0000","last_change":"2024-09-15T23:21:50.518425+0000","last_active":"2024-09-15T23:24:20.282085+0000","last_peered":"2024-09-15T23:24:20.282085+0000","last_clean":"2024-09-15T23:24:20.282085+0000","last_became_active":"2024-09-15T23:21:50.518052+0000","last_became_peered":"2024-09-15T23:21:50.518052+0000","last_unstale":"2024-09-15T23:24:20.282085+0000","last_undegraded":"2024-09-15T23:24:20.282085+0000","last_fullsized":"2024-09-15T23:24:20.282085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:02.243869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":208,"num_read_kb":208,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112035+0000","last_change":"2024-09-15T23:21:59.570310+0000","last_active":"2024-09-15T23:24:03.112035+0000","last_peered":"2024-09-15T23:24:03.112035+0000","last_clean":"2024-09-15T23:24:03.112035+0000","last_became_active":"2024-09-15T23:21:59.569909+0000","last_became_peered":"2024-09-15T23:21:59.569909+0000","last_unstale":"2024-09-15T23:24:03.112035+0000","last_undegraded":"2024-09-15T23:24:03.112035+0000","last_fullsized":"2024-09-15T23:24:03.112035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:20:27.515537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194191+0000","last_change":"2024-09-15T23:21:59.568208+0000","last_active":"2024-09-15T23:24:03.194191+0000","last_peered":"2024-09-15T23:24:03.194191+0000","last_clean":"2024-09-15T23:24:03.194191+0000","last_became_active":"2024-09-15T23:21:59.567858+0000","last_became_peered":"2024-09-15T23:21:59.567858+0000","last_unstale":"2024-09-15T23:24:03.194191+0000","last_undegraded":"2024-09-15T23:24:03.194191+0000","last_fullsized":"2024-09-15T23:24:03.194191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:34:42.895247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193532+0000","last_change":"2024-09-15T23:21:52.532679+0000","last_active":"2024-09-15T23:24:03.193532+0000","last_peered":"2024-09-15T23:24:03.193532+0000","last_clean":"2024-09-15T23:24:03.193532+0000","last_became_active":"2024-09-15T23:21:52.532345+0000","last_became_peered":"2024-09-15T23:21:52.532345+0000","last_unstale":"2024-09-15T23:24:03.193532+0000","last_undegraded":"2024-09-15T23:24:03.193532+0000","last_fullsized":"2024-09-15T23:24:03.193532+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:45.943653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118135+0000","last_change":"2024-09-15T23:21:50.510836+0000","last_active":"2024-09-15T23:24:03.118135+0000","last_peered":"2024-09-15T23:24:03.118135+0000","last_clean":"2024-09-15T23:24:03.118135+0000","last_became_active":"2024-09-15T23:21:50.510514+0000","last_became_peered":"2024-09-15T23:21:50.510514+0000","last_unstale":"2024-09-15T23:24:03.118135+0000","last_undegraded":"2024-09-15T23:24:03.118135+0000","last_fullsized":"2024-09-15T23:24:03.118135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:06.404612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116423+0000","last_change":"2024-09-15T23:23:02.383436+0000","last_active":"2024-09-15T23:24:03.116423+0000","last_peered":"2024-09-15T23:24:03.116423+0000","last_clean":"2024-09-15T23:24:03.116423+0000","last_became_active":"2024-09-15T23:23:02.382982+0000","last_became_peered":"2024-09-15T23:23:02.382982+0000","last_unstale":"2024-09-15T23:24:03.116423+0000","last_undegraded":"2024-09-15T23:24:03.116423+0000","last_fullsized":"2024-09-15T23:24:03.116423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:55:14.076604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111923+0000","last_change":"2024-09-15T23:21:52.526766+0000","last_active":"2024-09-15T23:24:03.111923+0000","last_peered":"2024-09-15T23:24:03.111923+0000","last_clean":"2024-09-15T23:24:03.111923+0000","last_became_active":"2024-09-15T23:21:52.526446+0000","last_became_peered":"2024-09-15T23:21:52.526446+0000","last_unstale":"2024-09-15T23:24:03.111923+0000","last_undegraded":"2024-09-15T23:24:03.111923+0000","last_fullsized":"2024-09-15T23:24:03.111923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:21:37.004730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109384+0000","last_change":"2024-09-15T23:21:50.512928+0000","last_active":"2024-09-15T23:24:03.109384+0000","last_peered":"2024-09-15T23:24:03.109384+0000","last_clean":"2024-09-15T23:24:03.109384+0000","last_became_active":"2024-09-15T23:21:50.512558+0000","last_became_peered":"2024-09-15T23:21:50.512558+0000","last_unstale":"2024-09-15T23:24:03.109384+0000","last_undegraded":"2024-09-15T23:24:03.109384+0000","last_fullsized":"2024-09-15T23:24:03.109384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:08.653084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114309+0000","last_change":"2024-09-15T23:21:50.516920+0000","last_active":"2024-09-15T23:24:03.114309+0000","last_peered":"2024-09-15T23:24:03.114309+0000","last_clean":"2024-09-15T23:24:03.114309+0000","last_became_active":"2024-09-15T23:21:50.516573+0000","last_became_peered":"2024-09-15T23:21:50.516573+0000","last_unstale":"2024-09-15T23:24:03.114309+0000","last_undegraded":"2024-09-15T23:24:03.114309+0000","last_fullsized":"2024-09-15T23:24:03.114309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.367903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117266+0000","last_change":"2024-09-15T23:21:52.536696+0000","last_active":"2024-09-15T23:24:03.117266+0000","last_peered":"2024-09-15T23:24:03.117266+0000","last_clean":"2024-09-15T23:24:03.117266+0000","last_became_active":"2024-09-15T23:21:52.536338+0000","last_became_peered":"2024-09-15T23:21:52.536338+0000","last_unstale":"2024-09-15T23:24:03.117266+0000","last_undegraded":"2024-09-15T23:24:03.117266+0000","last_fullsized":"2024-09-15T23:24:03.117266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:05.152749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112825+0000","last_change":"2024-09-15T23:21:59.562968+0000","last_active":"2024-09-15T23:24:03.112825+0000","last_peered":"2024-09-15T23:24:03.112825+0000","last_clean":"2024-09-15T23:24:03.112825+0000","last_became_active":"2024-09-15T23:21:59.562579+0000","last_became_peered":"2024-09-15T23:21:59.562579+0000","last_unstale":"2024-09-15T23:24:03.112825+0000","last_undegraded":"2024-09-15T23:24:03.112825+0000","last_fullsized":"2024-09-15T23:24:03.112825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:02.853057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:21.284870+0000","last_change":"2024-09-15T23:21:50.513944+0000","last_active":"2024-09-15T23:24:21.284870+0000","last_peered":"2024-09-15T23:24:21.284870+0000","last_clean":"2024-09-15T23:24:21.284870+0000","last_became_active":"2024-09-15T23:21:50.513484+0000","last_became_peered":"2024-09-15T23:21:50.513484+0000","last_unstale":"2024-09-15T23:24:21.284870+0000","last_undegraded":"2024-09-15T23:24:21.284870+0000","last_fullsized":"2024-09-15T23:24:21.284870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:35.719344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118061+0000","last_change":"2024-09-15T23:21:52.539467+0000","last_active":"2024-09-15T23:24:03.118061+0000","last_peered":"2024-09-15T23:24:03.118061+0000","last_clean":"2024-09-15T23:24:03.118061+0000","last_became_active":"2024-09-15T23:21:52.538998+0000","last_became_peered":"2024-09-15T23:21:52.538998+0000","last_unstale":"2024-09-15T23:24:03.118061+0000","last_undegraded":"2024-09-15T23:24:03.118061+0000","last_fullsized":"2024-09-15T23:24:03.118061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:22.213857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112854+0000","last_change":"2024-09-15T23:21:59.564842+0000","last_active":"2024-09-15T23:24:03.112854+0000","last_peered":"2024-09-15T23:24:03.112854+0000","last_clean":"2024-09-15T23:24:03.112854+0000","last_became_active":"2024-09-15T23:21:59.564408+0000","last_became_peered":"2024-09-15T23:21:59.564408+0000","last_unstale":"2024-09-15T23:24:03.112854+0000","last_undegraded":"2024-09-15T23:24:03.112854+0000","last_fullsized":"2024-09-15T23:24:03.112854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:17.909020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115745+0000","last_change":"2024-09-15T23:23:02.149420+0000","last_active":"2024-09-15T23:24:03.115745+0000","last_peered":"2024-09-15T23:24:03.115745+0000","last_clean":"2024-09-15T23:24:03.115745+0000","last_became_active":"2024-09-15T23:23:02.149003+0000","last_became_peered":"2024-09-15T23:23:02.149003+0000","last_unstale":"2024-09-15T23:24:03.115745+0000","last_undegraded":"2024-09-15T23:24:03.115745+0000","last_fullsized":"2024-09-15T23:24:03.115745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:45.490322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.532769+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532426+0000","last_became_peered":"2024-09-15T23:21:52.532426+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:09.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.533148+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532823+0000","last_became_peered":"2024-09-15T23:21:52.532823+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:01.833945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193803+0000","last_change":"2024-09-15T23:21:52.536790+0000","last_active":"2024-09-15T23:24:03.193803+0000","last_peered":"2024-09-15T23:24:03.193803+0000","last_clean":"2024-09-15T23:24:03.193803+0000","last_became_active":"2024-09-15T23:21:52.536515+0000","last_became_peered":"2024-09-15T23:21:52.536515+0000","last_unstale":"2024-09-15T23:24:03.193803+0000","last_undegraded":"2024-09-15T23:24:03.193803+0000","last_fullsized":"2024-09-15T23:24:03.193803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:37.207759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109800+0000","last_change":"2024-09-15T23:23:02.382842+0000","last_active":"2024-09-15T23:24:03.109800+0000","last_peered":"2024-09-15T23:24:03.109800+0000","last_clean":"2024-09-15T23:24:03.109800+0000","last_became_active":"2024-09-15T23:23:02.382455+0000","last_became_peered":"2024-09-15T23:23:02.382455+0000","last_unstale":"2024-09-15T23:24:03.109800+0000","last_undegraded":"2024-09-15T23:24:03.109800+0000","last_fullsized":"2024-09-15T23:24:03.109800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:23.617641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.540628+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.540308+0000","last_became_peered":"2024-09-15T23:21:52.540308+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:16.660637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113381+0000","last_change":"2024-09-15T23:21:52.533780+0000","last_active":"2024-09-15T23:24:03.113381+0000","last_peered":"2024-09-15T23:24:03.113381+0000","last_clean":"2024-09-15T23:24:03.113381+0000","last_became_active":"2024-09-15T23:21:52.533354+0000","last_became_peered":"2024-09-15T23:21:52.533354+0000","last_unstale":"2024-09-15T23:24:03.113381+0000","last_undegraded":"2024-09-15T23:24:03.113381+0000","last_fullsized":"2024-09-15T23:24:03.113381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.381059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112801+0000","last_change":"2024-09-15T23:21:52.536406+0000","last_active":"2024-09-15T23:24:03.112801+0000","last_peered":"2024-09-15T23:24:03.112801+0000","last_clean":"2024-09-15T23:24:03.112801+0000","last_became_active":"2024-09-15T23:21:52.536091+0000","last_became_peered":"2024-09-15T23:21:52.536091+0000","last_unstale":"2024-09-15T23:24:03.112801+0000","last_undegraded":"2024-09-15T23:24:03.112801+0000","last_fullsized":"2024-09-15T23:24:03.112801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:30.452201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117879+0000","last_change":"2024-09-15T23:21:52.536333+0000","last_active":"2024-09-15T23:24:03.117879+0000","last_peered":"2024-09-15T23:24:03.117879+0000","last_clean":"2024-09-15T23:24:03.117879+0000","last_became_active":"2024-09-15T23:21:52.534677+0000","last_became_peered":"2024-09-15T23:21:52.534677+0000","last_unstale":"2024-09-15T23:24:03.117879+0000","last_undegraded":"2024-09-15T23:24:03.117879+0000","last_fullsized":"2024-09-15T23:24:03.117879+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:05.287914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112116+0000","last_change":"2024-09-15T23:21:52.536442+0000","last_active":"2024-09-15T23:24:03.112116+0000","last_peered":"2024-09-15T23:24:03.112116+0000","last_clean":"2024-09-15T23:24:03.112116+0000","last_became_active":"2024-09-15T23:21:52.536183+0000","last_became_peered":"2024-09-15T23:21:52.536183+0000","last_unstale":"2024-09-15T23:24:03.112116+0000","last_undegraded":"2024-09-15T23:24:03.112116+0000","last_fullsized":"2024-09-15T23:24:03.112116+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:41.641370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117398+0000","last_change":"2024-09-15T23:23:02.277365+0000","last_active":"2024-09-15T23:24:03.117398+0000","last_peered":"2024-09-15T23:24:03.117398+0000","last_clean":"2024-09-15T23:24:03.117398+0000","last_became_active":"2024-09-15T23:23:02.276905+0000","last_became_peered":"2024-09-15T23:23:02.276905+0000","last_unstale":"2024-09-15T23:24:03.117398+0000","last_undegraded":"2024-09-15T23:24:03.117398+0000","last_fullsized":"2024-09-15T23:24:03.117398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:21.100657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109788+0000","last_change":"2024-09-15T23:21:52.531171+0000","last_active":"2024-09-15T23:24:03.109788+0000","last_peered":"2024-09-15T23:24:03.109788+0000","last_clean":"2024-09-15T23:24:03.109788+0000","last_became_active":"2024-09-15T23:21:52.530724+0000","last_became_peered":"2024-09-15T23:21:52.530724+0000","last_unstale":"2024-09-15T23:24:03.109788+0000","last_undegraded":"2024-09-15T23:24:03.109788+0000","last_fullsized":"2024-09-15T23:24:03.109788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:11.009119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115400+0000","last_change":"2024-09-15T23:21:52.530433+0000","last_active":"2024-09-15T23:24:03.115400+0000","last_peered":"2024-09-15T23:24:03.115400+0000","last_clean":"2024-09-15T23:24:03.115400+0000","last_became_active":"2024-09-15T23:21:52.530058+0000","last_became_peered":"2024-09-15T23:21:52.530058+0000","last_unstale":"2024-09-15T23:24:03.115400+0000","last_undegraded":"2024-09-15T23:24:03.115400+0000","last_fullsized":"2024-09-15T23:24:03.115400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:51.208820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113218+0000","last_change":"2024-09-15T23:21:52.534639+0000","last_active":"2024-09-15T23:24:03.113218+0000","last_peered":"2024-09-15T23:24:03.113218+0000","last_clean":"2024-09-15T23:24:03.113218+0000","last_became_active":"2024-09-15T23:21:52.534362+0000","last_became_peered":"2024-09-15T23:21:52.534362+0000","last_unstale":"2024-09-15T23:24:03.113218+0000","last_undegraded":"2024-09-15T23:24:03.113218+0000","last_fullsized":"2024-09-15T23:24:03.113218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:58:12.023501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194630+0000","last_change":"2024-09-15T23:21:52.532758+0000","last_active":"2024-09-15T23:24:03.194630+0000","last_peered":"2024-09-15T23:24:03.194630+0000","last_clean":"2024-09-15T23:24:03.194630+0000","last_became_active":"2024-09-15T23:21:52.532412+0000","last_became_peered":"2024-09-15T23:21:52.532412+0000","last_unstale":"2024-09-15T23:24:03.194630+0000","last_undegraded":"2024-09-15T23:24:03.194630+0000","last_fullsized":"2024-09-15T23:24:03.194630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:46.285711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116737+0000","last_change":"2024-09-15T23:21:52.533928+0000","last_active":"2024-09-15T23:24:03.116737+0000","last_peered":"2024-09-15T23:24:03.116737+0000","last_clean":"2024-09-15T23:24:03.116737+0000","last_became_active":"2024-09-15T23:21:52.533578+0000","last_became_peered":"2024-09-15T23:21:52.533578+0000","last_unstale":"2024-09-15T23:24:03.116737+0000","last_undegraded":"2024-09-15T23:24:03.116737+0000","last_fullsized":"2024-09-15T23:24:03.116737+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:49.476316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116894+0000","last_change":"2024-09-15T23:21:52.533417+0000","last_active":"2024-09-15T23:24:03.116894+0000","last_peered":"2024-09-15T23:24:03.116894+0000","last_clean":"2024-09-15T23:24:03.116894+0000","last_became_active":"2024-09-15T23:21:52.533085+0000","last_became_peered":"2024-09-15T23:21:52.533085+0000","last_unstale":"2024-09-15T23:24:03.116894+0000","last_undegraded":"2024-09-15T23:24:03.116894+0000","last_fullsized":"2024-09-15T23:24:03.116894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:20.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193763+0000","last_change":"2024-09-15T23:21:52.542340+0000","last_active":"2024-09-15T23:24:03.193763+0000","last_peered":"2024-09-15T23:24:03.193763+0000","last_clean":"2024-09-15T23:24:03.193763+0000","last_became_active":"2024-09-15T23:21:52.542048+0000","last_became_peered":"2024-09-15T23:21:52.542048+0000","last_unstale":"2024-09-15T23:24:03.193763+0000","last_undegraded":"2024-09-15T23:24:03.193763+0000","last_fullsized":"2024-09-15T23:24:03.193763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:57.557420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193941+0000","last_change":"2024-09-15T23:21:52.535576+0000","last_active":"2024-09-15T23:24:03.193941+0000","last_peered":"2024-09-15T23:24:03.193941+0000","last_clean":"2024-09-15T23:24:03.193941+0000","last_became_active":"2024-09-15T23:21:52.535223+0000","last_became_peered":"2024-09-15T23:21:52.535223+0000","last_unstale":"2024-09-15T23:24:03.193941+0000","last_undegraded":"2024-09-15T23:24:03.193941+0000","last_fullsized":"2024-09-15T23:24:03.193941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:45.874927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109691+0000","last_change":"2024-09-15T23:23:02.150556+0000","last_active":"2024-09-15T23:24:03.109691+0000","last_peered":"2024-09-15T23:24:03.109691+0000","last_clean":"2024-09-15T23:24:03.109691+0000","last_became_active":"2024-09-15T23:23:02.150253+0000","last_became_peered":"2024-09-15T23:23:02.150253+0000","last_unstale":"2024-09-15T23:24:03.109691+0000","last_undegraded":"2024-09-15T23:24:03.109691+0000","last_fullsized":"2024-09-15T23:24:03.109691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:47.950356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110875+0000","last_change":"2024-09-15T23:21:52.537012+0000","last_active":"2024-09-15T23:24:03.110875+0000","last_peered":"2024-09-15T23:24:03.110875+0000","last_clean":"2024-09-15T23:24:03.110875+0000","last_became_active":"2024-09-15T23:21:52.536245+0000","last_became_peered":"2024-09-15T23:21:52.536245+0000","last_unstale":"2024-09-15T23:24:03.110875+0000","last_undegraded":"2024-09-15T23:24:03.110875+0000","last_fullsized":"2024-09-15T23:24:03.110875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:31.150491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114034+0000","last_change":"2024-09-15T23:21:52.540431+0000","last_active":"2024-09-15T23:24:03.114034+0000","last_peered":"2024-09-15T23:24:03.114034+0000","last_clean":"2024-09-15T23:24:03.114034+0000","last_became_active":"2024-09-15T23:21:52.540104+0000","last_became_peered":"2024-09-15T23:21:52.540104+0000","last_unstale":"2024-09-15T23:24:03.114034+0000","last_undegraded":"2024-09-15T23:24:03.114034+0000","last_fullsized":"2024-09-15T23:24:03.114034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.550258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114876+0000","last_change":"2024-09-15T23:21:52.537590+0000","last_active":"2024-09-15T23:24:03.114876+0000","last_peered":"2024-09-15T23:24:03.114876+0000","last_clean":"2024-09-15T23:24:03.114876+0000","last_became_active":"2024-09-15T23:21:52.533804+0000","last_became_peered":"2024-09-15T23:21:52.533804+0000","last_unstale":"2024-09-15T23:24:03.114876+0000","last_undegraded":"2024-09-15T23:24:03.114876+0000","last_fullsized":"2024-09-15T23:24:03.114876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:48.449090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117952+0000","last_change":"2024-09-15T23:21:52.536130+0000","last_active":"2024-09-15T23:24:03.117952+0000","last_peered":"2024-09-15T23:24:03.117952+0000","last_clean":"2024-09-15T23:24:03.117952+0000","last_became_active":"2024-09-15T23:21:52.535336+0000","last_became_peered":"2024-09-15T23:21:52.535336+0000","last_unstale":"2024-09-15T23:24:03.117952+0000","last_undegraded":"2024-09-15T23:24:03.117952+0000","last_fullsized":"2024-09-15T23:24:03.117952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:52.581994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111827+0000","last_change":"2024-09-15T23:21:52.540485+0000","last_active":"2024-09-15T23:24:03.111827+0000","last_peered":"2024-09-15T23:24:03.111827+0000","last_clean":"2024-09-15T23:24:03.111827+0000","last_became_active":"2024-09-15T23:21:52.540178+0000","last_became_peered":"2024-09-15T23:21:52.540178+0000","last_unstale":"2024-09-15T23:24:03.111827+0000","last_undegraded":"2024-09-15T23:24:03.111827+0000","last_fullsized":"2024-09-15T23:24:03.111827+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:40:22.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113883+0000","last_change":"2024-09-15T23:21:52.538845+0000","last_active":"2024-09-15T23:24:03.113883+0000","last_peered":"2024-09-15T23:24:03.113883+0000","last_clean":"2024-09-15T23:24:03.113883+0000","last_became_active":"2024-09-15T23:21:52.538511+0000","last_became_peered":"2024-09-15T23:21:52.538511+0000","last_unstale":"2024-09-15T23:24:03.113883+0000","last_undegraded":"2024-09-15T23:24:03.113883+0000","last_fullsized":"2024-09-15T23:24:03.113883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:00.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114107+0000","last_change":"2024-09-15T23:23:02.149590+0000","last_active":"2024-09-15T23:24:03.114107+0000","last_peered":"2024-09-15T23:24:03.114107+0000","last_clean":"2024-09-15T23:24:03.114107+0000","last_became_active":"2024-09-15T23:23:02.149184+0000","last_became_peered":"2024-09-15T23:23:02.149184+0000","last_unstale":"2024-09-15T23:24:03.114107+0000","last_undegraded":"2024-09-15T23:24:03.114107+0000","last_fullsized":"2024-09-15T23:24:03.114107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:05:29.407175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117530+0000","last_change":"2024-09-15T23:21:52.530524+0000","last_active":"2024-09-15T23:24:03.117530+0000","last_peered":"2024-09-15T23:24:03.117530+0000","last_clean":"2024-09-15T23:24:03.117530+0000","last_became_active":"2024-09-15T23:21:52.530203+0000","last_became_peered":"2024-09-15T23:21:52.530203+0000","last_unstale":"2024-09-15T23:24:03.117530+0000","last_undegraded":"2024-09-15T23:24:03.117530+0000","last_fullsized":"2024-09-15T23:24:03.117530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:50.560211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110690+0000","last_change":"2024-09-15T23:21:52.537256+0000","last_active":"2024-09-15T23:24:03.110690+0000","last_peered":"2024-09-15T23:24:03.110690+0000","last_clean":"2024-09-15T23:24:03.110690+0000","last_became_active":"2024-09-15T23:21:52.536731+0000","last_became_peered":"2024-09-15T23:21:52.536731+0000","last_unstale":"2024-09-15T23:24:03.110690+0000","last_undegraded":"2024-09-15T23:24:03.110690+0000","last_fullsized":"2024-09-15T23:24:03.110690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:18.815398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194364+0000","last_change":"2024-09-15T23:21:52.537728+0000","last_active":"2024-09-15T23:24:03.194364+0000","last_peered":"2024-09-15T23:24:03.194364+0000","last_clean":"2024-09-15T23:24:03.194364+0000","last_became_active":"2024-09-15T23:21:52.537434+0000","last_became_peered":"2024-09-15T23:21:52.537434+0000","last_unstale":"2024-09-15T23:24:03.194364+0000","last_undegraded":"2024-09-15T23:24:03.194364+0000","last_fullsized":"2024-09-15T23:24:03.194364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:25.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194218+0000","last_change":"2024-09-15T23:21:52.534780+0000","last_active":"2024-09-15T23:24:03.194218+0000","last_peered":"2024-09-15T23:24:03.194218+0000","last_clean":"2024-09-15T23:24:03.194218+0000","last_became_active":"2024-09-15T23:21:52.534399+0000","last_became_peered":"2024-09-15T23:21:52.534399+0000","last_unstale":"2024-09-15T23:24:03.194218+0000","last_undegraded":"2024-09-15T23:24:03.194218+0000","last_fullsized":"2024-09-15T23:24:03.194218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:47.344129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116592+0000","last_change":"2024-09-15T23:21:52.533403+0000","last_active":"2024-09-15T23:24:03.116592+0000","last_peered":"2024-09-15T23:24:03.116592+0000","last_clean":"2024-09-15T23:24:03.116592+0000","last_became_active":"2024-09-15T23:21:52.533069+0000","last_became_peered":"2024-09-15T23:21:52.533069+0000","last_unstale":"2024-09-15T23:24:03.116592+0000","last_undegraded":"2024-09-15T23:24:03.116592+0000","last_fullsized":"2024-09-15T23:24:03.116592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:38.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":76056,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1071,"num_read_kb":611,"num_write":263,"num_write_kb":158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":63447,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":222402,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":246033,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579948,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":521776,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841584,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1159168,"data_stored":1176403,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005109,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1032192,"data_stored":524287,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234043,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":520569,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462976,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1073152,"data_stored":1116395,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626501,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1003520,"data_stored":515695,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855434,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1112565,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018959,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":582936,"data_compressed":39742,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247891,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1015808,"data_stored":515403,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444121,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1019904,"data_stored":517253,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607646,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":579358,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1028096,"data_stored":517151,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":70026,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":71062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:24:24.625 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:24.904 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:25 smithi089 ceph-mon[30933]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:25.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.405+0000 7f1b0cb75640 1 -- 172.21.15.89:0/2945535808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b08101df0 msgr2=0x7f1b081021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:25.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.405+0000 7f1b0cb75640 1 --2- 172.21.15.89:0/2945535808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b08101df0 0x7f1b081021d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1af8009ca0 tx=0x7f1af802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.405+0000 7f1b0cb75640 1 -- 172.21.15.89:0/2945535808 shutdown_connections 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.405+0000 7f1b0cb75640 1 --2- 172.21.15.89:0/2945535808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b08106f20 0x7f1b0810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.405+0000 7f1b0cb75640 1 --2- 172.21.15.89:0/2945535808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b081027a0 0x7f1b081067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.405+0000 7f1b0cb75640 1 --2- 172.21.15.89:0/2945535808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b08101df0 0x7f1b081021d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.405+0000 7f1b0cb75640 1 -- 172.21.15.89:0/2945535808 >> 172.21.15.89:0/2945535808 conn(0x7f1b080fd8b0 msgr2=0x7f1b080ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.406+0000 7f1b0cb75640 1 -- 172.21.15.89:0/2945535808 shutdown_connections 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.406+0000 7f1b0cb75640 1 -- 172.21.15.89:0/2945535808 wait complete. 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.406+0000 7f1b0cb75640 1 Processor -- start 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.406+0000 7f1b0cb75640 1 -- start start 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.406+0000 7f1b0cb75640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b08101df0 0x7f1b0819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.406+0000 7f1b0cb75640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b081027a0 0x7f1b0819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:25.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b0cb75640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b08106f20 0x7f1b0819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b0cb75640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1b08111f40 con 0x7f1b08106f20 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b0cb75640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1b08111dc0 con 0x7f1b08101df0 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b0cb75640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b081120c0 con 0x7f1b081027a0 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b06d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b08106f20 0x7f1b0819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b06d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b08106f20 0x7f1b0819eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55986/0 (socket says 172.21.15.89:55986) 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b06d9d640 1 -- 172.21.15.89:0/3282385782 learned_addr learned my addr 172.21.15.89:0/3282385782 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b0659c640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b08101df0 0x7f1b0819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b05d9b640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b081027a0 0x7f1b0819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b06d9d640 1 -- 172.21.15.89:0/3282385782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b081027a0 msgr2=0x7f1b0819ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b06d9d640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b081027a0 0x7f1b0819ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b06d9d640 1 -- 172.21.15.89:0/3282385782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b08101df0 msgr2=0x7f1b0819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:25.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b06d9d640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b08101df0 0x7f1b0819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b06d9d640 1 -- 172.21.15.89:0/3282385782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0819f5d0 con 0x7f1b08106f20 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b06d9d640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b08106f20 0x7f1b0819eef0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000c970 tx=0x7f1b0000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1af77fe640 1 -- 172.21.15.89:0/3282385782 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b00007bb0 con 0x7f1b08106f20 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b0cb75640 1 -- 172.21.15.89:0/3282385782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b0819f8c0 con 0x7f1b08106f20 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1af77fe640 1 -- 172.21.15.89:0/3282385782 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b00007d50 con 0x7f1b08106f20 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1af77fe640 1 -- 172.21.15.89:0/3282385782 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0000f450 con 0x7f1b08106f20 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.407+0000 7f1b0cb75640 1 -- 172.21.15.89:0/3282385782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b081a71a0 con 0x7f1b08106f20 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.408+0000 7f1b0cb75640 1 -- 172.21.15.89:0/3282385782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b081021d0 con 0x7f1b08106f20 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.410+0000 7f1af77fe640 1 -- 172.21.15.89:0/3282385782 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f1b00004f40 con 0x7f1b08106f20 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.410+0000 7f1af77fe640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1af00a32d0 0x7f1af00a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.410+0000 7f1b0659c640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1af00a32d0 0x7f1af00a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.410+0000 7f1af77fe640 1 -- 172.21.15.89:0/3282385782 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1b000cffb0 con 0x7f1b08106f20 2024-09-15T23:24:25.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.411+0000 7f1b0659c640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1af00a32d0 0x7f1af00a5790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1af802f540 tx=0x7f1af802f310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:25.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.413+0000 7f1af77fe640 1 -- 172.21.15.89:0/3282385782 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1b00014030 con 0x7f1b08106f20 2024-09-15T23:24:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:25 smithi120 ceph-mon[35250]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:25 smithi174 ceph-mon[35586]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:25.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.614+0000 7f1b0cb75640 1 -- 172.21.15.89:0/3282385782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b081a7430 con 0x7f1b08106f20 2024-09-15T23:24:25.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.615+0000 7f1af77fe640 1 -- 172.21.15.89:0/3282385782 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f1b0009c780 con 0x7f1b08106f20 2024-09-15T23:24:25.616 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:25.616 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:24:25.617 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:24:25.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 -- 172.21.15.89:0/3282385782 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1af00a32d0 msgr2=0x7f1af00a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:25.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1af00a32d0 0x7f1af00a5790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1af802f540 tx=0x7f1af802f310 comp rx=0 tx=0).stop 2024-09-15T23:24:25.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 -- 172.21.15.89:0/3282385782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b08106f20 msgr2=0x7f1b0819eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:25.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b08106f20 0x7f1b0819eef0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000c970 tx=0x7f1b0000ce30 comp rx=0 tx=0).stop 2024-09-15T23:24:25.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 -- 172.21.15.89:0/3282385782 shutdown_connections 2024-09-15T23:24:25.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1af00a32d0 0x7f1af00a5790 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:25.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b08106f20 0x7f1b0819eef0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:25.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b081027a0 0x7f1b0819ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:25.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 --2- 172.21.15.89:0/3282385782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b08101df0 0x7f1b0819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:25.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 -- 172.21.15.89:0/3282385782 >> 172.21.15.89:0/3282385782 conn(0x7f1b080fd8b0 msgr2=0x7f1b080feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:25.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 -- 172.21.15.89:0/3282385782 shutdown_connections 2024-09-15T23:24:25.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:25.617+0000 7f1b0cb75640 1 -- 172.21.15.89:0/3282385782 wait complete. 2024-09-15T23:24:26.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:26 smithi089 ceph-mon[30933]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:26.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:26 smithi089 ceph-mon[30933]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:26.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:26 smithi089 ceph-mon[30933]: pgmap v362: 129 pgs: 129 active+clean; 651 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-09-15T23:24:26.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:26 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3282385782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:26.335 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:24:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:26 smithi120 ceph-mon[35250]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:26 smithi120 ceph-mon[35250]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:26 smithi120 ceph-mon[35250]: pgmap v362: 129 pgs: 129 active+clean; 651 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-09-15T23:24:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:26 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3282385782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:26 smithi174 ceph-mon[35586]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:26 smithi174 ceph-mon[35586]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:26 smithi174 ceph-mon[35586]: pgmap v362: 129 pgs: 129 active+clean; 651 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-09-15T23:24:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:26 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3282385782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:26.580 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.065+0000 7efd0855b640 1 -- 172.21.15.89:0/1592159333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd001082a0 msgr2=0x7efd0010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:27.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.065+0000 7efd0855b640 1 --2- 172.21.15.89:0/1592159333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd001082a0 0x7efd0010ec40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efcf8009cd0 tx=0x7efcf802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.065+0000 7efd0855b640 1 -- 172.21.15.89:0/1592159333 shutdown_connections 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.065+0000 7efd0855b640 1 --2- 172.21.15.89:0/1592159333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd001082a0 0x7efd0010ec40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.065+0000 7efd0855b640 1 --2- 172.21.15.89:0/1592159333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd001036e0 0x7efd00107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.065+0000 7efd0855b640 1 --2- 172.21.15.89:0/1592159333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd00102d30 0x7efd00103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.065+0000 7efd0855b640 1 -- 172.21.15.89:0/1592159333 >> 172.21.15.89:0/1592159333 conn(0x7efd000fc7d0 msgr2=0x7efd000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.065+0000 7efd0855b640 1 -- 172.21.15.89:0/1592159333 shutdown_connections 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.065+0000 7efd0855b640 1 -- 172.21.15.89:0/1592159333 wait complete. 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd0855b640 1 Processor -- start 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd0855b640 1 -- start start 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd0855b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd00102d30 0x7efd00198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:27.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd0855b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd001036e0 0x7efd00198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd0855b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd001082a0 0x7efd0019ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd0855b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efd001120a0 con 0x7efd001036e0 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd0855b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efd00111f20 con 0x7efd001082a0 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd0855b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efd00112220 con 0x7efd00102d30 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd05af6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd001036e0 0x7efd00198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd05af6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd001036e0 0x7efd00198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56016/0 (socket says 172.21.15.89:56016) 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd05af6640 1 -- 172.21.15.89:0/518061064 learned_addr learned my addr 172.21.15.89:0/518061064 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd062f7640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd00102d30 0x7efd00198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd06af8640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd001082a0 0x7efd0019ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd05af6640 1 -- 172.21.15.89:0/518061064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd00102d30 msgr2=0x7efd00198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd05af6640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd00102d30 0x7efd00198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd05af6640 1 -- 172.21.15.89:0/518061064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd001082a0 msgr2=0x7efd0019ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:27.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd05af6640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd001082a0 0x7efd0019ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd05af6640 1 -- 172.21.15.89:0/518061064 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0019d4f0 con 0x7efd001036e0 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.066+0000 7efd06af8640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd001082a0 0x7efd0019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.067+0000 7efd05af6640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd001036e0 0x7efd00198a80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00d740 tx=0x7efcfc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.067+0000 7efcf77fe640 1 -- 172.21.15.89:0/518061064 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc017070 con 0x7efd001036e0 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.067+0000 7efd0855b640 1 -- 172.21.15.89:0/518061064 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0019d7e0 con 0x7efd001036e0 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.067+0000 7efcf77fe640 1 -- 172.21.15.89:0/518061064 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcfc00de50 con 0x7efd001036e0 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.067+0000 7efcf77fe640 1 -- 172.21.15.89:0/518061064 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00f6b0 con 0x7efd001036e0 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.067+0000 7efd0855b640 1 -- 172.21.15.89:0/518061064 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd001a50c0 con 0x7efd001036e0 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.068+0000 7efd0855b640 1 -- 172.21.15.89:0/518061064 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd00103110 con 0x7efd001036e0 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.069+0000 7efcf77fe640 1 -- 172.21.15.89:0/518061064 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7efcfc01a070 con 0x7efd001036e0 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.070+0000 7efcf77fe640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efcec0a32d0 0x7efcec0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:27.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.070+0000 7efd062f7640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efcec0a32d0 0x7efcec0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:27.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.070+0000 7efcf77fe640 1 -- 172.21.15.89:0/518061064 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7efcfc0cf7e0 con 0x7efd001036e0 2024-09-15T23:24:27.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.070+0000 7efd062f7640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efcec0a32d0 0x7efcec0a5790 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efcf000bbe0 tx=0x7efcf000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:27.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.073+0000 7efcf77fe640 1 -- 172.21.15.89:0/518061064 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efcfc09bfb0 con 0x7efd001036e0 2024-09-15T23:24:27.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.243+0000 7efd0855b640 1 -- 172.21.15.89:0/518061064 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd001a5290 con 0x7efd001036e0 2024-09-15T23:24:27.244 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:27 smithi089 ceph-mon[30933]: pgmap v363: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-09-15T23:24:27.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.244+0000 7efcf77fe640 1 -- 172.21.15.89:0/518061064 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24962 (secure 0 0 0) 0x7efcfc09b8b0 con 0x7efd001036e0 2024-09-15T23:24:27.246 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:27.246 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":90,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:24:03.106312+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.11","mappings":[{"from":7,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.246+0000 7efd0855b640 1 -- 172.21.15.89:0/518061064 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efcec0a32d0 msgr2=0x7efcec0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.246+0000 7efd0855b640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efcec0a32d0 0x7efcec0a5790 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efcf000bbe0 tx=0x7efcf000b390 comp rx=0 tx=0).stop 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.246+0000 7efd0855b640 1 -- 172.21.15.89:0/518061064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd001036e0 msgr2=0x7efd00198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.246+0000 7efd0855b640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd001036e0 0x7efd00198a80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00d740 tx=0x7efcfc00dc00 comp rx=0 tx=0).stop 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.247+0000 7efd0855b640 1 -- 172.21.15.89:0/518061064 shutdown_connections 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.247+0000 7efd0855b640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efcec0a32d0 0x7efcec0a5790 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.247+0000 7efd0855b640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efd001082a0 0x7efd0019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.247+0000 7efd0855b640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd001036e0 0x7efd00198a80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.247+0000 7efd0855b640 1 --2- 172.21.15.89:0/518061064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd00102d30 0x7efd00198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.247+0000 7efd0855b640 1 -- 172.21.15.89:0/518061064 >> 172.21.15.89:0/518061064 conn(0x7efd000fc7d0 msgr2=0x7efd00105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.247+0000 7efd0855b640 1 -- 172.21.15.89:0/518061064 shutdown_connections 2024-09-15T23:24:27.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:27.247+0000 7efd0855b640 1 -- 172.21.15.89:0/518061064 wait complete. 2024-09-15T23:24:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:27 smithi120 ceph-mon[35250]: pgmap v363: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-09-15T23:24:27.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:27 smithi174 ceph-mon[35586]: pgmap v363: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-09-15T23:24:27.955 DEBUG:tasks.fwd_scrub:Yielding 2024-09-15T23:24:27.956 INFO:teuthology.run_tasks:Running task background_exec... 2024-09-15T23:24:27.966 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi089.front.sepia.ceph.com 2024-09-15T23:24:27.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-09-15T23:24:27.966 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-09-15T23:24:27.967 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:24:27.967 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:24:27.970 INFO:teuthology.run_tasks:Running task quiescer... 2024-09-15T23:24:27.982 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-15T23:24:27.983 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:24:28.229 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:28.241 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:28.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:28 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/518061064' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:28.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:28 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/518061064' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:28.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:28 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/518061064' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:28.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.912+0000 7f2d28f77640 1 -- 172.21.15.89:0/1462540804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d24101de0 msgr2=0x7f2d241021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.912+0000 7f2d28f77640 1 --2- 172.21.15.89:0/1462540804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d24101de0 0x7f2d241021c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009900 tx=0x7f2d1402a410 comp rx=0 tx=0).stop 2024-09-15T23:24:28.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.912+0000 7f2d28f77640 1 -- 172.21.15.89:0/1462540804 shutdown_connections 2024-09-15T23:24:28.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.912+0000 7f2d28f77640 1 --2- 172.21.15.89:0/1462540804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d24106f10 0x7f2d2410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.912+0000 7f2d28f77640 1 --2- 172.21.15.89:0/1462540804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d24102790 0x7f2d241067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.912+0000 7f2d28f77640 1 --2- 172.21.15.89:0/1462540804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d24101de0 0x7f2d241021c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.912+0000 7f2d28f77640 1 -- 172.21.15.89:0/1462540804 >> 172.21.15.89:0/1462540804 conn(0x7f2d240fd8c0 msgr2=0x7f2d240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.912+0000 7f2d28f77640 1 -- 172.21.15.89:0/1462540804 shutdown_connections 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.912+0000 7f2d28f77640 1 -- 172.21.15.89:0/1462540804 wait complete. 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.913+0000 7f2d28f77640 1 Processor -- start 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.913+0000 7f2d28f77640 1 -- start start 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.913+0000 7f2d28f77640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d24101de0 0x7f2d24198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d28f77640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d24102790 0x7f2d241989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d28f77640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d24106f10 0x7f2d2419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d28f77640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d24111f90 con 0x7f2d24101de0 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d28f77640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d24111e10 con 0x7f2d24106f10 2024-09-15T23:24:28.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d28f77640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d24112110 con 0x7f2d24102790 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d24101de0 0x7f2d24198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d24101de0 0x7f2d24198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56030/0 (socket says 172.21.15.89:56030) 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d2259c640 1 -- 172.21.15.89:0/2331773545 learned_addr learned my addr 172.21.15.89:0/2331773545 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d21d9b640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d24102790 0x7f2d241989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d24106f10 0x7f2d2419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d2259c640 1 -- 172.21.15.89:0/2331773545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d24102790 msgr2=0x7f2d241989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d2259c640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d24102790 0x7f2d241989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d2259c640 1 -- 172.21.15.89:0/2331773545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d24106f10 msgr2=0x7f2d2419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d2259c640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d24106f10 0x7f2d2419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d2259c640 1 -- 172.21.15.89:0/2331773545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2419d440 con 0x7f2d24101de0 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d24106f10 0x7f2d2419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d2259c640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d24101de0 0x7f2d24198490 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2d14006010 tx=0x7f2d140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d137fe640 1 -- 172.21.15.89:0/2331773545 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1403f030 con 0x7f2d24101de0 2024-09-15T23:24:28.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d137fe640 1 -- 172.21.15.89:0/2331773545 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d14003ce0 con 0x7f2d24101de0 2024-09-15T23:24:28.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d28f77640 1 -- 172.21.15.89:0/2331773545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d2419d6d0 con 0x7f2d24101de0 2024-09-15T23:24:28.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.914+0000 7f2d137fe640 1 -- 172.21.15.89:0/2331773545 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d14035070 con 0x7f2d24101de0 2024-09-15T23:24:28.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.915+0000 7f2d28f77640 1 -- 172.21.15.89:0/2331773545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d241a5010 con 0x7f2d24101de0 2024-09-15T23:24:28.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d137fe640 1 -- 172.21.15.89:0/2331773545 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f2d14042030 con 0x7f2d24101de0 2024-09-15T23:24:28.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d2b350740 1 Processor -- start 2024-09-15T23:24:28.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d2b350740 1 -- start start 2024-09-15T23:24:28.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d461a40 0x56441d461e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d470c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d471150 0x56441d473540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d2b350740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56441d3c7280 con 0x56441d461a40 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d2b350740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56441d36eb10 con 0x56441d462360 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d2b350740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56441d3c3240 con 0x56441d471150 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d461a40 0x56441d461e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d461a40 0x56441d461e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56036/0 (socket says 172.21.15.89:56036) 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.917+0000 7f2d22d9d640 1 -- 172.21.15.89:0/4207943872 learned_addr learned my addr 172.21.15.89:0/4207943872 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d137fe640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d0c0a32d0 0x7f2d0c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d21d9b640 1 --2- 172.21.15.89:0/4207943872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d471150 0x56441d473540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d2259c640 1 --2- 172.21.15.89:0/4207943872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d470c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d0c0a32d0 0x7f2d0c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d137fe640 1 -- 172.21.15.89:0/2331773545 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2d140f0600 con 0x7f2d24101de0 2024-09-15T23:24:28.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d22d9d640 1 -- 172.21.15.89:0/4207943872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d471150 msgr2=0x56441d473540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/4207943872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d471150 0x56441d473540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d22d9d640 1 -- 172.21.15.89:0/4207943872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 msgr2=0x56441d470c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/4207943872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d470c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d22d9d640 1 -- 172.21.15.89:0/4207943872 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56441d37b1f0 con 0x56441d461a40 2024-09-15T23:24:28.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/4207943872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d461a40 0x56441d461e20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c013950 tx=0x7f2d1c013e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e49fd033ddd653d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d10ff9640 1 -- 172.21.15.89:0/4207943872 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c013420 con 0x56441d461a40 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d0c0a32d0 0x7f2d0c0a5790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00d000 tx=0x7f2d1c01d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d10ff9640 1 -- 172.21.15.89:0/4207943872 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1c011dc0 con 0x56441d461a40 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.918+0000 7f2d10ff9640 1 -- 172.21.15.89:0/4207943872 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c005250 con 0x56441d461a40 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.919+0000 7f2d2b350740 1 -- 172.21.15.89:0/4207943872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d461a40 msgr2=0x56441d461e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.919+0000 7f2d2b350740 1 --2- 172.21.15.89:0/4207943872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d461a40 0x56441d461e20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c013950 tx=0x7f2d1c013e10 comp rx=0 tx=0).stop 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.919+0000 7f2d2b350740 1 -- 172.21.15.89:0/4207943872 shutdown_connections 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.919+0000 7f2d2b350740 1 --2- 172.21.15.89:0/4207943872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d471150 0x56441d473540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.919+0000 7f2d2b350740 1 --2- 172.21.15.89:0/4207943872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d470c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.919+0000 7f2d2b350740 1 --2- 172.21.15.89:0/4207943872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d461a40 0x56441d461e20 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.919+0000 7f2d2b350740 1 -- 172.21.15.89:0/4207943872 >> 172.21.15.89:0/4207943872 conn(0x56441d45e0f0 msgr2=0x56441d45e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.919+0000 7f2d2b350740 1 -- 172.21.15.89:0/4207943872 shutdown_connections 2024-09-15T23:24:28.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.919+0000 7f2d2b350740 1 -- 172.21.15.89:0/4207943872 wait complete. 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d2b350740 1 -- start start 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d2b350740 1 Processor -- start 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d461a40 0x56441d46e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d46e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d46eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d2b350740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56441d473d20 con 0x56441d471150 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d2b350740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56441d381f10 con 0x56441d462360 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d2b350740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56441d473ea0 con 0x56441d461a40 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d2b350740 10 client.0 _ll_register_callbacks cb 0x56441d34aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d46eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d46eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56044/0 (socket says 172.21.15.89:56044) 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d22d9d640 1 -- 172.21.15.89:0/2536234933 learned_addr learned my addr 172.21.15.89:0/2536234933 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:28.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d21d9b640 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d46e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d22d9d640 1 -- 172.21.15.89:0/2536234933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d461a40 msgr2=0x56441d46e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d461a40 0x56441d46e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d22d9d640 1 -- 172.21.15.89:0/2536234933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 msgr2=0x56441d46e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d46e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.920+0000 7f2d22d9d640 1 -- 172.21.15.89:0/2536234933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56441d1386a0 con 0x56441d471150 2024-09-15T23:24:28.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d46eb80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c009890 tx=0x7f2d1c0098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b729f31f6cc74 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:28.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d09ffb640 1 -- 172.21.15.89:0/2536234933 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c01c770 con 0x56441d471150 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d09ffb640 1 -- 172.21.15.89:0/2536234933 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2d1c004a40 con 0x56441d471150 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d2b350740 1 -- 172.21.15.89:0/2536234933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d1c011dc0 con 0x56441d471150 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d09ffb640 1 -- 172.21.15.89:0/2536234933 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2d1c03a050 con 0x56441d471150 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d2b350740 10 client.15402 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d2b350740 1 -- 172.21.15.89:0/2536234933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56441d47fa40 con 0x56441d471150 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d09ffb640 1 -- 172.21.15.89:0/2536234933 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1c011dc0 con 0x56441d471150 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d09ffb640 1 -- 172.21.15.89:0/2536234933 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2d1c01da40 con 0x56441d471150 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d2b350740 10 client.15402 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d2b350740 10 client.15402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d2b350740 20 client.15402 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d2b350740 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56441d480e50 0x56441d483240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d2b350740 4 client.15402 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:28.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.921+0000 7f2d2b350740 1 -- 172.21.15.89:0/2536234933 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56441d3d47f0 con 0x56441d480e50 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.922+0000 7f2d2259c640 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56441d480e50 0x56441d483240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.922+0000 7f2d2259c640 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56441d480e50 0x56441d483240 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15f02e273f0be613 server_cookie=786465d22d21ea16 in_seq=0 out_seq=0 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.922+0000 7f2d09ffb640 10 client.15402 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d09ffb640 1 -- 172.21.15.89:0/2536234933 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56441d3d47f0 con 0x56441d480e50 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d09ffb640 10 client.15402 handle_command_reply: tid=0 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 client.15402 shutdown 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 -- 172.21.15.89:0/2536234933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 msgr2=0x56441d46eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d46eb80 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c009890 tx=0x7f2d1c0098c0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 -- 172.21.15.89:0/2536234933 shutdown_connections 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56441d480e50 0x56441d483240 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d46eb80 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d46e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 --2- 172.21.15.89:0/2536234933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d461a40 0x56441d46e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 -- 172.21.15.89:0/2536234933 >> 172.21.15.89:0/2536234933 conn(0x56441d45e0f0 msgr2=0x56441d463e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d09ffb640 0 client.15402 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:28.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 -- 172.21.15.89:0/2536234933 shutdown_connections 2024-09-15T23:24:28.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 1 -- 172.21.15.89:0/2536234933 wait complete. 2024-09-15T23:24:28.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.923+0000 7f2d2b350740 20 client.15402 trim_cache size 0 max 16384 2024-09-15T23:24:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.940+0000 7f2d2b350740 1 Processor -- start 2024-09-15T23:24:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.940+0000 7f2d2b350740 1 -- start start 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.940+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d461a40 0x56441d46ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.940+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d4888f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.940+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d488e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.940+0000 7f2d2b350740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d241accd0 con 0x56441d471150 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.940+0000 7f2d2b350740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d241ae810 con 0x56441d462360 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.940+0000 7f2d2b350740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d241ad3a0 con 0x56441d461a40 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d488e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d21d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d4888f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d21d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d4888f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:53080/0 (socket says 172.21.15.89:53080) 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d21d9b640 1 -- 172.21.15.89:0/286235952 learned_addr learned my addr 172.21.15.89:0/286235952 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d2259c640 1 --2- 172.21.15.89:0/286235952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d461a40 0x56441d46ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d22d9d640 1 -- 172.21.15.89:0/286235952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d461a40 msgr2=0x56441d46ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/286235952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d461a40 0x56441d46ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d22d9d640 1 -- 172.21.15.89:0/286235952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 msgr2=0x56441d4888f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/286235952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d4888f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d22d9d640 1 -- 172.21.15.89:0/286235952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56441d4709d0 con 0x56441d471150 2024-09-15T23:24:28.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/286235952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d488e30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0093e0 tx=0x7f2d1c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d17eee808576cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d0bfff640 1 -- 172.21.15.89:0/286235952 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c043070 con 0x56441d471150 2024-09-15T23:24:28.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d0bfff640 1 -- 172.21.15.89:0/286235952 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1c011dc0 con 0x56441d471150 2024-09-15T23:24:28.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.941+0000 7f2d0bfff640 1 -- 172.21.15.89:0/286235952 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c027770 con 0x56441d471150 2024-09-15T23:24:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 -- 172.21.15.89:0/286235952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 msgr2=0x56441d488e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 --2- 172.21.15.89:0/286235952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d488e30 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0093e0 tx=0x7f2d1c009b50 comp rx=0 tx=0).stop 2024-09-15T23:24:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 -- 172.21.15.89:0/286235952 shutdown_connections 2024-09-15T23:24:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 --2- 172.21.15.89:0/286235952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d488e30 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 --2- 172.21.15.89:0/286235952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d462360 0x56441d4888f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 --2- 172.21.15.89:0/286235952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d461a40 0x56441d46ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 -- 172.21.15.89:0/286235952 >> 172.21.15.89:0/286235952 conn(0x56441d45e0f0 msgr2=0x56441d46f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 -- 172.21.15.89:0/286235952 shutdown_connections 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 -- 172.21.15.89:0/286235952 wait complete. 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 -- start start 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.942+0000 7f2d2b350740 1 Processor -- start 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d461a40 0x56441d479750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d462360 0x56441d474a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d2b350740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d474fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d2b350740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d241ae140 con 0x56441d471150 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d2b350740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d241ada70 con 0x56441d461a40 2024-09-15T23:24:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d2b350740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d241aacb0 con 0x56441d462360 2024-09-15T23:24:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d2b350740 10 client.0 _ll_register_callbacks cb 0x7f2d140b8a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d474fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d474fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56058/0 (socket says 172.21.15.89:56058) 2024-09-15T23:24:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d22d9d640 1 -- 172.21.15.89:0/426230091 learned_addr learned my addr 172.21.15.89:0/426230091 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d21d9b640 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d462360 0x56441d474a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d2259c640 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d461a40 0x56441d479750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d22d9d640 1 -- 172.21.15.89:0/426230091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d462360 msgr2=0x56441d474a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d462360 0x56441d474a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d22d9d640 1 -- 172.21.15.89:0/426230091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d461a40 msgr2=0x56441d479750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d461a40 0x56441d479750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d22d9d640 1 -- 172.21.15.89:0/426230091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56441d4756b0 con 0x56441d471150 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d2259c640 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d461a40 0x56441d479750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d22d9d640 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d474fc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c01deb0 tx=0x7f2d1c01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6b8c96740859eb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d09ffb640 1 -- 172.21.15.89:0/426230091 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c043070 con 0x56441d471150 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d09ffb640 1 -- 172.21.15.89:0/426230091 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2d1c004bf0 con 0x56441d471150 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.943+0000 7f2d2b350740 1 -- 172.21.15.89:0/426230091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d1c011dc0 con 0x56441d471150 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d09ffb640 1 -- 172.21.15.89:0/426230091 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2d1c052050 con 0x56441d471150 2024-09-15T23:24:28.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d2b350740 10 client.15408 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d2b350740 1 -- 172.21.15.89:0/426230091 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56441d4eb7c0 con 0x56441d471150 2024-09-15T23:24:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d09ffb640 1 -- 172.21.15.89:0/426230091 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d1c011dc0 con 0x56441d471150 2024-09-15T23:24:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d09ffb640 1 -- 172.21.15.89:0/426230091 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2d1c027440 con 0x56441d471150 2024-09-15T23:24:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d2b350740 10 client.15408 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d2b350740 10 client.15408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d2b350740 20 client.15408 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:28.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d2b350740 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56441d4ebe80 0x56441d4ee2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d2b350740 4 client.15408 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d2b350740 1 -- 172.21.15.89:0/426230091 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56441d3ced00 con 0x56441d4ebe80 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.944+0000 7f2d2259c640 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56441d4ebe80 0x56441d4ee2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.945+0000 7f2d09ffb640 10 client.15408 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.945+0000 7f2d2259c640 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56441d4ebe80 0x56441d4ee2a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64afd6171563572c server_cookie=47c07c07bd1d00dd in_seq=0 out_seq=0 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d09ffb640 1 -- 172.21.15.89:0/426230091 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56441d3ced00 con 0x56441d4ebe80 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d09ffb640 10 client.15408 handle_command_reply: tid=0 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d2b350740 1 client.15408 shutdown 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d2b350740 1 -- 172.21.15.89:0/426230091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 msgr2=0x56441d474fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d2b350740 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d474fc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c01deb0 tx=0x7f2d1c01ca00 comp rx=0 tx=0).stop 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d2b350740 1 -- 172.21.15.89:0/426230091 shutdown_connections 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d2b350740 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56441d4ebe80 0x56441d4ee2a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d2b350740 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56441d471150 0x56441d474fc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d2b350740 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56441d462360 0x56441d474a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d2b350740 1 --2- 172.21.15.89:0/426230091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56441d461a40 0x56441d479750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.946+0000 7f2d2b350740 1 -- 172.21.15.89:0/426230091 >> 172.21.15.89:0/426230091 conn(0x56441d45e0f0 msgr2=0x56441d45d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d09ffb640 0 client.15408 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d2b350740 1 -- 172.21.15.89:0/426230091 shutdown_connections 2024-09-15T23:24:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d2b350740 1 -- 172.21.15.89:0/426230091 wait complete. 2024-09-15T23:24:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d2b350740 20 client.15408 trim_cache size 0 max 16384 2024-09-15T23:24:28.955 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:28.955 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:24:28.955 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "eb4cea45-715e-4764-9a2e-7000c8d30f82", 2024-09-15T23:24:28.955 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:24:28.955 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d28f77640 1 -- 172.21.15.89:0/2331773545 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d0c0a32d0 msgr2=0x7f2d0c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d28f77640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d0c0a32d0 0x7f2d0c0a5790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00d000 tx=0x7f2d1c01d000 comp rx=0 tx=0).stop 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d28f77640 1 -- 172.21.15.89:0/2331773545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d24101de0 msgr2=0x7f2d24198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d28f77640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d24101de0 0x7f2d24198490 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2d14006010 tx=0x7f2d140042f0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d28f77640 1 -- 172.21.15.89:0/2331773545 shutdown_connections 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d28f77640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d0c0a32d0 0x7f2d0c0a5790 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d28f77640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d24106f10 0x7f2d2419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d28f77640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d24102790 0x7f2d241989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d28f77640 1 --2- 172.21.15.89:0/2331773545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d24101de0 0x7f2d24198490 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.947+0000 7f2d28f77640 1 -- 172.21.15.89:0/2331773545 >> 172.21.15.89:0/2331773545 conn(0x7f2d240fd8c0 msgr2=0x7f2d240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.948+0000 7f2d28f77640 1 -- 172.21.15.89:0/2331773545 shutdown_connections 2024-09-15T23:24:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:28.948+0000 7f2d28f77640 1 -- 172.21.15.89:0/2331773545 wait complete. 2024-09-15T23:24:29.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:29 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:24:29.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:29 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:29.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:29 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:24:29.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:29 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:29.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:29 smithi089 ceph-mon[30933]: pgmap v364: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:24:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:29 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:24:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:29 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:29 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:24:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:29 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:29 smithi120 ceph-mon[35250]: pgmap v364: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:24:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:29 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:24:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:29 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:29 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:24:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:29 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:29.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:29 smithi174 ceph-mon[35586]: pgmap v364: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:24:29.680 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:24:29.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.814+0000 7f415c541640 1 -- 172.21.15.89:0/793931513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4154102570 msgr2=0x7f41541029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:29.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.814+0000 7f415c541640 1 --2- 172.21.15.89:0/793931513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4154102570 0x7f41541029b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4144009900 tx=0x7f414402a480 comp rx=0 tx=0).stop 2024-09-15T23:24:29.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.814+0000 7f415c541640 1 -- 172.21.15.89:0/793931513 shutdown_connections 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.814+0000 7f415c541640 1 --2- 172.21.15.89:0/793931513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4154102ef0 0x7f415410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.814+0000 7f415c541640 1 --2- 172.21.15.89:0/793931513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4154102570 0x7f41541029b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.814+0000 7f415c541640 1 --2- 172.21.15.89:0/793931513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4154100e20 0x7f4154101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.814+0000 7f415c541640 1 -- 172.21.15.89:0/793931513 >> 172.21.15.89:0/793931513 conn(0x7f41540fc7d0 msgr2=0x7f41540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.814+0000 7f415c541640 1 -- 172.21.15.89:0/793931513 shutdown_connections 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.814+0000 7f415c541640 1 -- 172.21.15.89:0/793931513 wait complete. 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415c541640 1 Processor -- start 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415c541640 1 -- start start 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415c541640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4154100e20 0x7f4154199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415c541640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4154102570 0x7f4154199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415c541640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4154102ef0 0x7f41541a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415c541640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4154110f80 con 0x7f4154102ef0 2024-09-15T23:24:29.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415c541640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4154110e00 con 0x7f4154100e20 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415c541640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4154111100 con 0x7f4154102570 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415aade640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4154102ef0 0x7f41541a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415aade640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4154102ef0 0x7f41541a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56072/0 (socket says 172.21.15.89:56072) 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.815+0000 7f415aade640 1 -- 172.21.15.89:0/1843155208 learned_addr learned my addr 172.21.15.89:0/1843155208 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f4159adc640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4154102570 0x7f4154199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f415a2dd640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4154100e20 0x7f4154199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f415aade640 1 -- 172.21.15.89:0/1843155208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4154102570 msgr2=0x7f4154199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f415aade640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4154102570 0x7f4154199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f415aade640 1 -- 172.21.15.89:0/1843155208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4154100e20 msgr2=0x7f4154199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f415aade640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4154100e20 0x7f4154199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f415aade640 1 -- 172.21.15.89:0/1843155208 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41541a48f0 con 0x7f4154102ef0 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f415a2dd640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4154100e20 0x7f4154199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:29.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f415aade640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4154102ef0 0x7f41541a43b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f414c00eaa0 tx=0x7f414c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f414b7fe640 1 -- 172.21.15.89:0/1843155208 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c00cdb0 con 0x7f4154102ef0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f415c541640 1 -- 172.21.15.89:0/1843155208 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41541a4ac0 con 0x7f4154102ef0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f414b7fe640 1 -- 172.21.15.89:0/1843155208 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f414c005020 con 0x7f4154102ef0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f414b7fe640 1 -- 172.21.15.89:0/1843155208 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c0116b0 con 0x7f4154102ef0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.816+0000 7f415c541640 1 -- 172.21.15.89:0/1843155208 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41541a4f80 con 0x7f4154102ef0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.818+0000 7f415c541640 1 -- 172.21.15.89:0/1843155208 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4154101200 con 0x7f4154102ef0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.820+0000 7f414b7fe640 1 -- 172.21.15.89:0/1843155208 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f414c005560 con 0x7f4154102ef0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.820+0000 7f414b7fe640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41400a32d0 0x7f41400a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.820+0000 7f414b7fe640 1 -- 172.21.15.89:0/1843155208 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f414c0d0390 con 0x7f4154102ef0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.820+0000 7f415a2dd640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41400a32d0 0x7f41400a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.821+0000 7f415a2dd640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41400a32d0 0x7f41400a5790 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f41500058c0 tx=0x7f4150005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:29.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:29.824+0000 7f414b7fe640 1 -- 172.21.15.89:0/1843155208 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f414c016030 con 0x7f4154102ef0 2024-09-15T23:24:29.955 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:30.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.010+0000 7f415c541640 1 -- 172.21.15.89:0/1843155208 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4154106fb0 con 0x7f41400a32d0 2024-09-15T23:24:30.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.019+0000 7f414b7fe640 1 -- 172.21.15.89:0/1843155208 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340822 (secure 0 0 0) 0x7f4154106fb0 con 0x7f41400a32d0 2024-09-15T23:24:30.023 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:30.024 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-15T23:24:30.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 -- 172.21.15.89:0/1843155208 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41400a32d0 msgr2=0x7f41400a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41400a32d0 0x7f41400a5790 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f41500058c0 tx=0x7f4150005ec0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 -- 172.21.15.89:0/1843155208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4154102ef0 msgr2=0x7f41541a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4154102ef0 0x7f41541a43b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f414c00eaa0 tx=0x7f414c00ef60 comp rx=0 tx=0).stop 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 -- 172.21.15.89:0/1843155208 shutdown_connections 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41400a32d0 0x7f41400a5790 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4154102ef0 0x7f41541a43b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4154102570 0x7f4154199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 --2- 172.21.15.89:0/1843155208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4154100e20 0x7f4154199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 -- 172.21.15.89:0/1843155208 >> 172.21.15.89:0/1843155208 conn(0x7f41540fc7d0 msgr2=0x7f41540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 -- 172.21.15.89:0/1843155208 shutdown_connections 2024-09-15T23:24:30.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.022+0000 7f415c541640 1 -- 172.21.15.89:0/1843155208 wait complete. 2024-09-15T23:24:30.234 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:30 smithi089 ceph-mon[30933]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:30 smithi120 ceph-mon[35250]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:30.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:30 smithi174 ceph-mon[35586]: from='client.15414 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:30.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.634+0000 7fe839056640 1 -- 172.21.15.89:0/38947377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8341082a0 msgr2=0x7fe83410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.634+0000 7fe839056640 1 --2- 172.21.15.89:0/38947377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8341082a0 0x7fe83410ec40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe82c009cd0 tx=0x7fe82c02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.634+0000 7fe839056640 1 -- 172.21.15.89:0/38947377 shutdown_connections 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.634+0000 7fe839056640 1 --2- 172.21.15.89:0/38947377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8341082a0 0x7fe83410ec40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.634+0000 7fe839056640 1 --2- 172.21.15.89:0/38947377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8341036e0 0x7fe834107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.635+0000 7fe839056640 1 --2- 172.21.15.89:0/38947377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe834102d30 0x7fe834103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.635+0000 7fe839056640 1 -- 172.21.15.89:0/38947377 >> 172.21.15.89:0/38947377 conn(0x7fe8340fc7d0 msgr2=0x7fe8340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.635+0000 7fe839056640 1 -- 172.21.15.89:0/38947377 shutdown_connections 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.635+0000 7fe839056640 1 -- 172.21.15.89:0/38947377 wait complete. 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.635+0000 7fe839056640 1 Processor -- start 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe839056640 1 -- start start 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe839056640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe834102d30 0x7fe834198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe839056640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8341036e0 0x7fe834198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe839056640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8341082a0 0x7fe83419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe839056640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341120a0 con 0x7fe8341036e0 2024-09-15T23:24:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe839056640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe834111f20 con 0x7fe8341082a0 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe839056640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe834112220 con 0x7fe834102d30 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe83259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8341036e0 0x7fe834198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe83259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8341036e0 0x7fe834198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56078/0 (socket says 172.21.15.89:56078) 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe83259c640 1 -- 172.21.15.89:0/1173145680 learned_addr learned my addr 172.21.15.89:0/1173145680 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe832d9d640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe834102d30 0x7fe834198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe83359e640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8341082a0 0x7fe83419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe83259c640 1 -- 172.21.15.89:0/1173145680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe834102d30 msgr2=0x7fe834198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe83259c640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe834102d30 0x7fe834198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe83259c640 1 -- 172.21.15.89:0/1173145680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8341082a0 msgr2=0x7fe83419ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe83259c640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8341082a0 0x7fe83419ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe83259c640 1 -- 172.21.15.89:0/1173145680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83419d510 con 0x7fe8341036e0 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.636+0000 7fe83359e640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8341082a0 0x7fe83419ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.637+0000 7fe83259c640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8341036e0 0x7fe834198aa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00d750 tx=0x7fe81c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:30.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.637+0000 7fe82bfff640 1 -- 172.21.15.89:0/1173145680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c017070 con 0x7fe8341036e0 2024-09-15T23:24:30.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.637+0000 7fe839056640 1 -- 172.21.15.89:0/1173145680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83419d800 con 0x7fe8341036e0 2024-09-15T23:24:30.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.637+0000 7fe82bfff640 1 -- 172.21.15.89:0/1173145680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe81c005500 con 0x7fe8341036e0 2024-09-15T23:24:30.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.637+0000 7fe82bfff640 1 -- 172.21.15.89:0/1173145680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c004e00 con 0x7fe8341036e0 2024-09-15T23:24:30.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.637+0000 7fe839056640 1 -- 172.21.15.89:0/1173145680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8341a50e0 con 0x7fe8341036e0 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.639+0000 7fe82bfff640 1 -- 172.21.15.89:0/1173145680 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fe81c004fa0 con 0x7fe8341036e0 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.639+0000 7fe83b42f740 1 Processor -- start 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe83b42f740 1 -- start start 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe82bfff640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe8200a32d0 0x7fe8200a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564894314a0 0x556489431880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556489431dc0 0x55648943c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55648943c680 0x55648943ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe83b42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5564893972e0 con 0x5564894314a0 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe83b42f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55648933ece0 con 0x556489431dc0 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe83b42f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5564893932a0 con 0x55648943c680 2024-09-15T23:24:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe82bfff640 1 -- 172.21.15.89:0/1173145680 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe81c0cf7b0 con 0x7fe8341036e0 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe832d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564894314a0 0x556489431880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe832d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564894314a0 0x556489431880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56080/0 (socket says 172.21.15.89:56080) 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe832d9d640 1 -- 172.21.15.89:0/3259112954 learned_addr learned my addr 172.21.15.89:0/3259112954 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe83359e640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe8200a32d0 0x7fe8200a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe83259c640 1 --2- 172.21.15.89:0/3259112954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556489431dc0 0x55648943c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.640+0000 7fe83359e640 1 --2- 172.21.15.89:0/3259112954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55648943c680 0x55648943ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.641+0000 7fe832d9d640 1 -- 172.21.15.89:0/3259112954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55648943c680 msgr2=0x55648943ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.641+0000 7fe832d9d640 1 --2- 172.21.15.89:0/3259112954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55648943c680 0x55648943ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.641+0000 7fe832d9d640 1 -- 172.21.15.89:0/3259112954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556489431dc0 msgr2=0x55648943c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.641+0000 7fe832d9d640 1 --2- 172.21.15.89:0/3259112954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556489431dc0 0x55648943c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.641+0000 7fe832d9d640 1 -- 172.21.15.89:0/3259112954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556489323440 con 0x5564894314a0 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.641+0000 7fe832d9d640 1 --2- 172.21.15.89:0/3259112954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564894314a0 0x556489431880 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe82400ac20 tx=0x7fe82400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fad961a3c21e04 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.641+0000 7fe83359e640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe8200a32d0 0x7fe8200a5790 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe82c005fe0 tx=0x7fe82c005d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.641+0000 7fe8297fa640 1 -- 172.21.15.89:0/3259112954 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82401d070 con 0x5564894314a0 2024-09-15T23:24:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.641+0000 7fe8297fa640 1 -- 172.21.15.89:0/3259112954 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe824018b20 con 0x5564894314a0 2024-09-15T23:24:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.641+0000 7fe8297fa640 1 -- 172.21.15.89:0/3259112954 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824010b90 con 0x5564894314a0 2024-09-15T23:24:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 -- 172.21.15.89:0/3259112954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564894314a0 msgr2=0x556489431880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 --2- 172.21.15.89:0/3259112954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564894314a0 0x556489431880 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe82400ac20 tx=0x7fe82400f790 comp rx=0 tx=0).stop 2024-09-15T23:24:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 -- 172.21.15.89:0/3259112954 shutdown_connections 2024-09-15T23:24:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 --2- 172.21.15.89:0/3259112954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55648943c680 0x55648943ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 --2- 172.21.15.89:0/3259112954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556489431dc0 0x55648943c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 --2- 172.21.15.89:0/3259112954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564894314a0 0x556489431880 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 -- 172.21.15.89:0/3259112954 >> 172.21.15.89:0/3259112954 conn(0x55648942da30 msgr2=0x55648942de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 -- 172.21.15.89:0/3259112954 shutdown_connections 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 -- 172.21.15.89:0/3259112954 wait complete. 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 -- start start 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.642+0000 7fe83b42f740 1 Processor -- start 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x55648943bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489439270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x5564894397b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83b42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55648943f250 con 0x556489431dc0 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83b42f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556489352050 con 0x55648943c680 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83b42f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55648943f3d0 con 0x5564894314a0 2024-09-15T23:24:30.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83b42f740 10 client.0 _ll_register_callbacks cb 0x556489392160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489439270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489439270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56092/0 (socket says 172.21.15.89:56092) 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83259c640 1 -- 172.21.15.89:0/2980576848 learned_addr learned my addr 172.21.15.89:0/2980576848 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe832d9d640 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x55648943bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83259c640 1 -- 172.21.15.89:0/2980576848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 msgr2=0x55648943bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83259c640 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x55648943bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83259c640 1 -- 172.21.15.89:0/2980576848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 msgr2=0x5564894397b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83359e640 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x5564894397b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83259c640 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x5564894397b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83259c640 1 -- 172.21.15.89:0/2980576848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564891083c0 con 0x556489431dc0 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83359e640 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x5564894397b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83259c640 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489439270 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0ccda0 tx=0x7fe81c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab90fd48c9880da1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe81a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:30.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2980576848 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c020040 con 0x556489431dc0 2024-09-15T23:24:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2980576848 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe81c0ccfc0 con 0x556489431dc0 2024-09-15T23:24:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.643+0000 7fe83b42f740 1 -- 172.21.15.89:0/2980576848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe824018b20 con 0x556489431dc0 2024-09-15T23:24:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2980576848 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe81c09fc30 con 0x556489431dc0 2024-09-15T23:24:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe83b42f740 10 client.15426 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe83b42f740 1 -- 172.21.15.89:0/2980576848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55648944ac00 con 0x556489431dc0 2024-09-15T23:24:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2980576848 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe824018b20 con 0x556489431dc0 2024-09-15T23:24:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2980576848 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe81c097ef0 con 0x556489431dc0 2024-09-15T23:24:30.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe83b42f740 10 client.15426 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe83b42f740 10 client.15426 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe83b42f740 20 client.15426 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55648944bfc0 0x55648944e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe83b42f740 4 client.15426 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe83b42f740 1 -- 172.21.15.89:0/2980576848 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564893a4850 con 0x55648944bfc0 2024-09-15T23:24:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.644+0000 7fe832d9d640 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55648944bfc0 0x55648944e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.645+0000 7fe832d9d640 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55648944bfc0 0x55648944e3f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30b59e1f046a4739 server_cookie=327567c4aa0a435b in_seq=0 out_seq=0 2024-09-15T23:24:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.645+0000 7fe81a7fc640 10 client.15426 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.645+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2980576848 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5564893a4850 con 0x55648944bfc0 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.645+0000 7fe81a7fc640 10 client.15426 handle_command_reply: tid=0 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.645+0000 7fe83b42f740 1 client.15426 shutdown 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 1 -- 172.21.15.89:0/2980576848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 msgr2=0x556489439270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489439270 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0ccda0 tx=0x7fe81c0046c0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 1 -- 172.21.15.89:0/2980576848 shutdown_connections 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55648944bfc0 0x55648944e3f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x5564894397b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489439270 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2980576848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x55648943bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 1 -- 172.21.15.89:0/2980576848 >> 172.21.15.89:0/2980576848 conn(0x55648942da30 msgr2=0x55648942cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe81a7fc640 0 client.15426 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 1 -- 172.21.15.89:0/2980576848 shutdown_connections 2024-09-15T23:24:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 1 -- 172.21.15.89:0/2980576848 wait complete. 2024-09-15T23:24:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.646+0000 7fe83b42f740 20 client.15426 trim_cache size 0 max 16384 2024-09-15T23:24:30.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.662+0000 7fe83b42f740 1 Processor -- start 2024-09-15T23:24:30.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83b42f740 1 -- start start 2024-09-15T23:24:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x55648943b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489453cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x556489454200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83b42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341aee80 con 0x556489431dc0 2024-09-15T23:24:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83b42f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341b09c0 con 0x55648943c680 2024-09-15T23:24:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83b42f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341af550 con 0x5564894314a0 2024-09-15T23:24:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489453cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489453cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56100/0 (socket says 172.21.15.89:56100) 2024-09-15T23:24:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83259c640 1 -- 172.21.15.89:0/56832392 learned_addr learned my addr 172.21.15.89:0/56832392 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe832d9d640 1 --2- 172.21.15.89:0/56832392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x55648943b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83359e640 1 --2- 172.21.15.89:0/56832392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x556489454200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83259c640 1 -- 172.21.15.89:0/56832392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 msgr2=0x55648943b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83259c640 1 --2- 172.21.15.89:0/56832392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x55648943b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83259c640 1 -- 172.21.15.89:0/56832392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 msgr2=0x556489454200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83259c640 1 --2- 172.21.15.89:0/56832392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x556489454200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe83259c640 1 -- 172.21.15.89:0/56832392 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564894312d0 con 0x556489431dc0 2024-09-15T23:24:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.663+0000 7fe832d9d640 1 --2- 172.21.15.89:0/56832392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x55648943b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.664+0000 7fe83259c640 1 --2- 172.21.15.89:0/56832392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489453cc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe81c099b30 tx=0x7fe81c09b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9370050470f52394 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.664+0000 7fe828ff9640 1 -- 172.21.15.89:0/56832392 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c05f030 con 0x556489431dc0 2024-09-15T23:24:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.664+0000 7fe828ff9640 1 -- 172.21.15.89:0/56832392 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe81c015070 con 0x556489431dc0 2024-09-15T23:24:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.664+0000 7fe828ff9640 1 -- 172.21.15.89:0/56832392 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c009cc0 con 0x556489431dc0 2024-09-15T23:24:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 -- 172.21.15.89:0/56832392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 msgr2=0x556489453cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 --2- 172.21.15.89:0/56832392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489453cc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe81c099b30 tx=0x7fe81c09b4f0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 -- 172.21.15.89:0/56832392 shutdown_connections 2024-09-15T23:24:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 --2- 172.21.15.89:0/56832392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x556489454200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 --2- 172.21.15.89:0/56832392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489453cc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 --2- 172.21.15.89:0/56832392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x55648943b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 -- 172.21.15.89:0/56832392 >> 172.21.15.89:0/56832392 conn(0x55648942da30 msgr2=0x55648943aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 -- 172.21.15.89:0/56832392 shutdown_connections 2024-09-15T23:24:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 -- 172.21.15.89:0/56832392 wait complete. 2024-09-15T23:24:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 -- start start 2024-09-15T23:24:30.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 Processor -- start 2024-09-15T23:24:30.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x556489444c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489441f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x5564894424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341b02f0 con 0x556489431dc0 2024-09-15T23:24:30.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341afc20 con 0x55648943c680 2024-09-15T23:24:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe8341ace60 con 0x5564894314a0 2024-09-15T23:24:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83b42f740 10 client.0 _ll_register_callbacks cb 0x7fe81c09b1e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.665+0000 7fe83259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489441f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489441f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56114/0 (socket says 172.21.15.89:56114) 2024-09-15T23:24:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83259c640 1 -- 172.21.15.89:0/2389438713 learned_addr learned my addr 172.21.15.89:0/2389438713 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe832d9d640 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x556489444c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83359e640 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x5564894424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83259c640 1 -- 172.21.15.89:0/2389438713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 msgr2=0x556489444c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:30.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83259c640 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x556489444c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83259c640 1 -- 172.21.15.89:0/2389438713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 msgr2=0x5564894424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:30.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83259c640 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x5564894424b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83259c640 1 -- 172.21.15.89:0/2389438713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556489442ba0 con 0x556489431dc0 2024-09-15T23:24:30.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe832d9d640 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x556489444c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:30.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83359e640 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x5564894424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:30.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83259c640 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489441f70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe81c09b580 tx=0x7fe81c0ccd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d915b3acdb4e638 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:30.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe81a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:30.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2389438713 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c05a060 con 0x556489431dc0 2024-09-15T23:24:30.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2389438713 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe81c0ccfc0 con 0x556489431dc0 2024-09-15T23:24:30.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83b42f740 1 -- 172.21.15.89:0/2389438713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe81c015070 con 0x556489431dc0 2024-09-15T23:24:30.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2389438713 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe81c05f030 con 0x556489431dc0 2024-09-15T23:24:30.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83b42f740 10 client.15432 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:30.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.666+0000 7fe83b42f740 1 -- 172.21.15.89:0/2389438713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5564894b6b50 con 0x556489431dc0 2024-09-15T23:24:30.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.667+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2389438713 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe81c015070 con 0x556489431dc0 2024-09-15T23:24:30.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.667+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2389438713 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe81c0a40f0 con 0x556489431dc0 2024-09-15T23:24:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.667+0000 7fe83b42f740 10 client.15432 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.667+0000 7fe83b42f740 10 client.15432 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.667+0000 7fe83b42f740 20 client.15432 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.667+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5564894b6f80 0x5564894b9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.667+0000 7fe83b42f740 4 client.15432 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:24:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.667+0000 7fe83b42f740 1 -- 172.21.15.89:0/2389438713 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55648939ed60 con 0x5564894b6f80 2024-09-15T23:24:30.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.667+0000 7fe832d9d640 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5564894b6f80 0x5564894b9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:30.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe832d9d640 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5564894b6f80 0x5564894b9370 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca7332b17840dbf6 server_cookie=3e86aa6c7c238c4c in_seq=0 out_seq=0 2024-09-15T23:24:30.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe81a7fc640 10 client.15432 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:30.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe81a7fc640 1 -- 172.21.15.89:0/2389438713 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55648939ed60 con 0x5564894b6f80 2024-09-15T23:24:30.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe81a7fc640 10 client.15432 handle_command_reply: tid=0 2024-09-15T23:24:30.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 client.15432 shutdown 2024-09-15T23:24:30.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 -- 172.21.15.89:0/2389438713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 msgr2=0x556489441f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489441f70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe81c09b580 tx=0x7fe81c0ccd30 comp rx=0 tx=0).stop 2024-09-15T23:24:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 -- 172.21.15.89:0/2389438713 shutdown_connections 2024-09-15T23:24:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5564894b6f80 0x5564894b9370 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55648943c680 0x5564894424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556489431dc0 0x556489441f70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 --2- 172.21.15.89:0/2389438713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5564894314a0 0x556489444c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 -- 172.21.15.89:0/2389438713 >> 172.21.15.89:0/2389438713 conn(0x55648942da30 msgr2=0x55648943b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe81a7fc640 0 client.15432 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 -- 172.21.15.89:0/2389438713 shutdown_connections 2024-09-15T23:24:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 1 -- 172.21.15.89:0/2389438713 wait complete. 2024-09-15T23:24:30.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.668+0000 7fe83b42f740 20 client.15432 trim_cache size 0 max 16384 2024-09-15T23:24:30.685 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:24:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:24:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "eb4cea45-715e-4764-9a2e-7000c8d30f82": { 2024-09-15T23:24:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:24:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "eb4cea45-715e-4764-9a2e-7000c8d30f82", 2024-09-15T23:24:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:24:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:24:30.685 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:24:30.685 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:30.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 -- 172.21.15.89:0/1173145680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe8200a32d0 msgr2=0x7fe8200a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe8200a32d0 0x7fe8200a5790 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe82c005fe0 tx=0x7fe82c005d80 comp rx=0 tx=0).stop 2024-09-15T23:24:30.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 -- 172.21.15.89:0/1173145680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8341036e0 msgr2=0x7fe834198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:30.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8341036e0 0x7fe834198aa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00d750 tx=0x7fe81c00dc10 comp rx=0 tx=0).stop 2024-09-15T23:24:30.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 -- 172.21.15.89:0/1173145680 shutdown_connections 2024-09-15T23:24:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe8200a32d0 0x7fe8200a5790 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8341082a0 0x7fe83419ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe8341036e0 0x7fe834198aa0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 --2- 172.21.15.89:0/1173145680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe834102d30 0x7fe834198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 -- 172.21.15.89:0/1173145680 >> 172.21.15.89:0/1173145680 conn(0x7fe8340fc7d0 msgr2=0x7fe834105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 -- 172.21.15.89:0/1173145680 shutdown_connections 2024-09-15T23:24:30.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:30.669+0000 7fe839056640 1 -- 172.21.15.89:0/1173145680 wait complete. 2024-09-15T23:24:31.241 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:31 smithi089 ceph-mon[30933]: pgmap v365: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:24:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:31 smithi120 ceph-mon[35250]: pgmap v365: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:24:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:31 smithi174 ceph-mon[35586]: pgmap v365: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:24:31.580 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-15T23:24:29.023325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669178,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1255,"num_read_kb":760,"num_write":451,"num_write_kb":3103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1244183,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369884,"kb_used_data":12284,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571940,"statfs":{"total":1151940427776,"available":1151561666560,"internally_reserved":0,"allocated":12578816,"data_stored":8219276,"data_compressed":471759,"data_compressed_allocated":638976,"data_compressed_original":2813952,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9295,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":67,"num_write":14,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003217"},"pg_stats":[{"pgid":"2.1f","version":"86'13","reported_seq":163,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:26.289472+0000","last_change":"2024-09-15T23:21:50.513610+0000","last_active":"2024-09-15T23:24:26.289472+0000","last_peered":"2024-09-15T23:24:26.289472+0000","last_clean":"2024-09-15T23:24:26.289472+0000","last_became_active":"2024-09-15T23:21:50.513266+0000","last_became_peered":"2024-09-15T23:21:50.513266+0000","last_unstale":"2024-09-15T23:24:26.289472+0000","last_undegraded":"2024-09-15T23:24:26.289472+0000","last_fullsized":"2024-09-15T23:24:26.289472+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:16:48.692801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112578+0000","last_change":"2024-09-15T23:21:52.540248+0000","last_active":"2024-09-15T23:24:03.112578+0000","last_peered":"2024-09-15T23:24:03.112578+0000","last_clean":"2024-09-15T23:24:03.112578+0000","last_became_active":"2024-09-15T23:21:52.539945+0000","last_became_peered":"2024-09-15T23:21:52.539945+0000","last_unstale":"2024-09-15T23:24:03.112578+0000","last_undegraded":"2024-09-15T23:24:03.112578+0000","last_fullsized":"2024-09-15T23:24:03.112578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:33.402153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112456+0000","last_change":"2024-09-15T23:21:59.567246+0000","last_active":"2024-09-15T23:24:03.112456+0000","last_peered":"2024-09-15T23:24:03.112456+0000","last_clean":"2024-09-15T23:24:03.112456+0000","last_became_active":"2024-09-15T23:21:59.566832+0000","last_became_peered":"2024-09-15T23:21:59.566832+0000","last_unstale":"2024-09-15T23:24:03.112456+0000","last_undegraded":"2024-09-15T23:24:03.112456+0000","last_fullsized":"2024-09-15T23:24:03.112456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:19.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112747+0000","last_change":"2024-09-15T23:21:50.561767+0000","last_active":"2024-09-15T23:24:03.112747+0000","last_peered":"2024-09-15T23:24:03.112747+0000","last_clean":"2024-09-15T23:24:03.112747+0000","last_became_active":"2024-09-15T23:21:50.561385+0000","last_became_peered":"2024-09-15T23:21:50.561385+0000","last_unstale":"2024-09-15T23:24:03.112747+0000","last_undegraded":"2024-09-15T23:24:03.112747+0000","last_fullsized":"2024-09-15T23:24:03.112747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:11.897884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194008+0000","last_change":"2024-09-15T23:21:52.534874+0000","last_active":"2024-09-15T23:24:03.194008+0000","last_peered":"2024-09-15T23:24:03.194008+0000","last_clean":"2024-09-15T23:24:03.194008+0000","last_became_active":"2024-09-15T23:21:52.534334+0000","last_became_peered":"2024-09-15T23:21:52.534334+0000","last_unstale":"2024-09-15T23:24:03.194008+0000","last_undegraded":"2024-09-15T23:24:03.194008+0000","last_fullsized":"2024-09-15T23:24:03.194008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:15.972648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110442+0000","last_change":"2024-09-15T23:21:59.559907+0000","last_active":"2024-09-15T23:24:03.110442+0000","last_peered":"2024-09-15T23:24:03.110442+0000","last_clean":"2024-09-15T23:24:03.110442+0000","last_became_active":"2024-09-15T23:21:59.559440+0000","last_became_peered":"2024-09-15T23:21:59.559440+0000","last_unstale":"2024-09-15T23:24:03.110442+0000","last_undegraded":"2024-09-15T23:24:03.110442+0000","last_fullsized":"2024-09-15T23:24:03.110442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:00.755112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117658+0000","last_change":"2024-09-15T23:21:50.515424+0000","last_active":"2024-09-15T23:24:03.117658+0000","last_peered":"2024-09-15T23:24:03.117658+0000","last_clean":"2024-09-15T23:24:03.117658+0000","last_became_active":"2024-09-15T23:21:50.514966+0000","last_became_peered":"2024-09-15T23:21:50.514966+0000","last_unstale":"2024-09-15T23:24:03.117658+0000","last_undegraded":"2024-09-15T23:24:03.117658+0000","last_fullsized":"2024-09-15T23:24:03.117658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:57.586575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117412+0000","last_change":"2024-09-15T23:21:52.535189+0000","last_active":"2024-09-15T23:24:03.117412+0000","last_peered":"2024-09-15T23:24:03.117412+0000","last_clean":"2024-09-15T23:24:03.117412+0000","last_became_active":"2024-09-15T23:21:52.534807+0000","last_became_peered":"2024-09-15T23:21:52.534807+0000","last_unstale":"2024-09-15T23:24:03.117412+0000","last_undegraded":"2024-09-15T23:24:03.117412+0000","last_fullsized":"2024-09-15T23:24:03.117412+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:30.654382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110773+0000","last_change":"2024-09-15T23:21:59.564675+0000","last_active":"2024-09-15T23:24:03.110773+0000","last_peered":"2024-09-15T23:24:03.110773+0000","last_clean":"2024-09-15T23:24:03.110773+0000","last_became_active":"2024-09-15T23:21:59.564271+0000","last_became_peered":"2024-09-15T23:21:59.564271+0000","last_unstale":"2024-09-15T23:24:03.110773+0000","last_undegraded":"2024-09-15T23:24:03.110773+0000","last_fullsized":"2024-09-15T23:24:03.110773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:01.055369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117219+0000","last_change":"2024-09-15T23:21:50.507790+0000","last_active":"2024-09-15T23:24:03.117219+0000","last_peered":"2024-09-15T23:24:03.117219+0000","last_clean":"2024-09-15T23:24:03.117219+0000","last_became_active":"2024-09-15T23:21:50.507398+0000","last_became_peered":"2024-09-15T23:21:50.507398+0000","last_unstale":"2024-09-15T23:24:03.117219+0000","last_undegraded":"2024-09-15T23:24:03.117219+0000","last_fullsized":"2024-09-15T23:24:03.117219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:58:00.001279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116935+0000","last_change":"2024-09-15T23:21:52.537179+0000","last_active":"2024-09-15T23:24:03.116935+0000","last_peered":"2024-09-15T23:24:03.116935+0000","last_clean":"2024-09-15T23:24:03.116935+0000","last_became_active":"2024-09-15T23:21:52.536841+0000","last_became_peered":"2024-09-15T23:21:52.536841+0000","last_unstale":"2024-09-15T23:24:03.116935+0000","last_undegraded":"2024-09-15T23:24:03.116935+0000","last_fullsized":"2024-09-15T23:24:03.116935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:47.455700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110441+0000","last_change":"2024-09-15T23:21:59.564734+0000","last_active":"2024-09-15T23:24:03.110441+0000","last_peered":"2024-09-15T23:24:03.110441+0000","last_clean":"2024-09-15T23:24:03.110441+0000","last_became_active":"2024-09-15T23:21:59.564339+0000","last_became_peered":"2024-09-15T23:21:59.564339+0000","last_unstale":"2024-09-15T23:24:03.110441+0000","last_undegraded":"2024-09-15T23:24:03.110441+0000","last_fullsized":"2024-09-15T23:24:03.110441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:44.498480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113970+0000","last_change":"2024-09-15T23:21:52.538025+0000","last_active":"2024-09-15T23:24:03.113970+0000","last_peered":"2024-09-15T23:24:03.113970+0000","last_clean":"2024-09-15T23:24:03.113970+0000","last_became_active":"2024-09-15T23:21:52.537736+0000","last_became_peered":"2024-09-15T23:21:52.537736+0000","last_unstale":"2024-09-15T23:24:03.113970+0000","last_undegraded":"2024-09-15T23:24:03.113970+0000","last_fullsized":"2024-09-15T23:24:03.113970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:31.916888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.127260+0000","last_change":"2024-09-15T23:24:03.127260+0000","last_active":"2024-09-15T23:24:03.127260+0000","last_peered":"2024-09-15T23:24:03.127260+0000","last_clean":"2024-09-15T23:24:03.127260+0000","last_became_active":"2024-09-15T23:24:03.119384+0000","last_became_peered":"2024-09-15T23:24:03.119384+0000","last_unstale":"2024-09-15T23:24:03.127260+0000","last_undegraded":"2024-09-15T23:24:03.127260+0000","last_fullsized":"2024-09-15T23:24:03.127260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:35.325048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110625+0000","last_change":"2024-09-15T23:21:59.681971+0000","last_active":"2024-09-15T23:24:03.110625+0000","last_peered":"2024-09-15T23:24:03.110625+0000","last_clean":"2024-09-15T23:24:03.110625+0000","last_became_active":"2024-09-15T23:21:59.681432+0000","last_became_peered":"2024-09-15T23:21:59.681432+0000","last_unstale":"2024-09-15T23:24:03.110625+0000","last_undegraded":"2024-09-15T23:24:03.110625+0000","last_fullsized":"2024-09-15T23:24:03.110625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:26.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112179+0000","last_change":"2024-09-15T23:21:52.541281+0000","last_active":"2024-09-15T23:24:03.112179+0000","last_peered":"2024-09-15T23:24:03.112179+0000","last_clean":"2024-09-15T23:24:03.112179+0000","last_became_active":"2024-09-15T23:21:52.540772+0000","last_became_peered":"2024-09-15T23:21:52.540772+0000","last_unstale":"2024-09-15T23:24:03.112179+0000","last_undegraded":"2024-09-15T23:24:03.112179+0000","last_fullsized":"2024-09-15T23:24:03.112179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:23:44.444904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112700+0000","last_change":"2024-09-15T23:21:50.508844+0000","last_active":"2024-09-15T23:24:03.112700+0000","last_peered":"2024-09-15T23:24:03.112700+0000","last_clean":"2024-09-15T23:24:03.112700+0000","last_became_active":"2024-09-15T23:21:50.508364+0000","last_became_peered":"2024-09-15T23:21:50.508364+0000","last_unstale":"2024-09-15T23:24:03.112700+0000","last_undegraded":"2024-09-15T23:24:03.112700+0000","last_fullsized":"2024-09-15T23:24:03.112700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:45.129711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110720+0000","last_change":"2024-09-15T23:21:59.562114+0000","last_active":"2024-09-15T23:24:03.110720+0000","last_peered":"2024-09-15T23:24:03.110720+0000","last_clean":"2024-09-15T23:24:03.110720+0000","last_became_active":"2024-09-15T23:21:59.561652+0000","last_became_peered":"2024-09-15T23:21:59.561652+0000","last_unstale":"2024-09-15T23:24:03.110720+0000","last_undegraded":"2024-09-15T23:24:03.110720+0000","last_fullsized":"2024-09-15T23:24:03.110720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:51.781886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.531678+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.531327+0000","last_became_peered":"2024-09-15T23:21:52.531327+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:07.152626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112193+0000","last_change":"2024-09-15T23:21:50.513197+0000","last_active":"2024-09-15T23:24:03.112193+0000","last_peered":"2024-09-15T23:24:03.112193+0000","last_clean":"2024-09-15T23:24:03.112193+0000","last_became_active":"2024-09-15T23:21:50.512794+0000","last_became_peered":"2024-09-15T23:21:50.512794+0000","last_unstale":"2024-09-15T23:24:03.112193+0000","last_undegraded":"2024-09-15T23:24:03.112193+0000","last_fullsized":"2024-09-15T23:24:03.112193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:05.457467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194628+0000","last_change":"2024-09-15T23:21:59.565669+0000","last_active":"2024-09-15T23:24:03.194628+0000","last_peered":"2024-09-15T23:24:03.194628+0000","last_clean":"2024-09-15T23:24:03.194628+0000","last_became_active":"2024-09-15T23:21:59.565290+0000","last_became_peered":"2024-09-15T23:21:59.565290+0000","last_unstale":"2024-09-15T23:24:03.194628+0000","last_undegraded":"2024-09-15T23:24:03.194628+0000","last_fullsized":"2024-09-15T23:24:03.194628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:04.861560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116394+0000","last_change":"2024-09-15T23:21:52.529643+0000","last_active":"2024-09-15T23:24:03.116394+0000","last_peered":"2024-09-15T23:24:03.116394+0000","last_clean":"2024-09-15T23:24:03.116394+0000","last_became_active":"2024-09-15T23:21:52.529304+0000","last_became_peered":"2024-09-15T23:21:52.529304+0000","last_unstale":"2024-09-15T23:24:03.116394+0000","last_undegraded":"2024-09-15T23:24:03.116394+0000","last_fullsized":"2024-09-15T23:24:03.116394+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:18:37.758403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117052+0000","last_change":"2024-09-15T23:21:50.562114+0000","last_active":"2024-09-15T23:24:03.117052+0000","last_peered":"2024-09-15T23:24:03.117052+0000","last_clean":"2024-09-15T23:24:03.117052+0000","last_became_active":"2024-09-15T23:21:50.561772+0000","last_became_peered":"2024-09-15T23:21:50.561772+0000","last_unstale":"2024-09-15T23:24:03.117052+0000","last_undegraded":"2024-09-15T23:24:03.117052+0000","last_fullsized":"2024-09-15T23:24:03.117052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:14.806646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110672+0000","last_change":"2024-09-15T23:21:59.681933+0000","last_active":"2024-09-15T23:24:03.110672+0000","last_peered":"2024-09-15T23:24:03.110672+0000","last_clean":"2024-09-15T23:24:03.110672+0000","last_became_active":"2024-09-15T23:21:59.681528+0000","last_became_peered":"2024-09-15T23:21:59.681528+0000","last_unstale":"2024-09-15T23:24:03.110672+0000","last_undegraded":"2024-09-15T23:24:03.110672+0000","last_fullsized":"2024-09-15T23:24:03.110672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:46.199540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194326+0000","last_change":"2024-09-15T23:21:52.529548+0000","last_active":"2024-09-15T23:24:03.194326+0000","last_peered":"2024-09-15T23:24:03.194326+0000","last_clean":"2024-09-15T23:24:03.194326+0000","last_became_active":"2024-09-15T23:21:52.529253+0000","last_became_peered":"2024-09-15T23:21:52.529253+0000","last_unstale":"2024-09-15T23:24:03.194326+0000","last_undegraded":"2024-09-15T23:24:03.194326+0000","last_fullsized":"2024-09-15T23:24:03.194326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:22.813395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111022+0000","last_change":"2024-09-15T23:21:50.767759+0000","last_active":"2024-09-15T23:24:03.111022+0000","last_peered":"2024-09-15T23:24:03.111022+0000","last_clean":"2024-09-15T23:24:03.111022+0000","last_became_active":"2024-09-15T23:21:50.767388+0000","last_became_peered":"2024-09-15T23:21:50.767388+0000","last_unstale":"2024-09-15T23:24:03.111022+0000","last_undegraded":"2024-09-15T23:24:03.111022+0000","last_fullsized":"2024-09-15T23:24:03.111022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:36.713769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112512+0000","last_change":"2024-09-15T23:21:59.557951+0000","last_active":"2024-09-15T23:24:03.112512+0000","last_peered":"2024-09-15T23:24:03.112512+0000","last_clean":"2024-09-15T23:24:03.112512+0000","last_became_active":"2024-09-15T23:21:59.557490+0000","last_became_peered":"2024-09-15T23:21:59.557490+0000","last_unstale":"2024-09-15T23:24:03.112512+0000","last_undegraded":"2024-09-15T23:24:03.112512+0000","last_fullsized":"2024-09-15T23:24:03.112512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:17.147389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112859+0000","last_change":"2024-09-15T23:21:52.531638+0000","last_active":"2024-09-15T23:24:03.112859+0000","last_peered":"2024-09-15T23:24:03.112859+0000","last_clean":"2024-09-15T23:24:03.112859+0000","last_became_active":"2024-09-15T23:21:52.531301+0000","last_became_peered":"2024-09-15T23:21:52.531301+0000","last_unstale":"2024-09-15T23:24:03.112859+0000","last_undegraded":"2024-09-15T23:24:03.112859+0000","last_fullsized":"2024-09-15T23:24:03.112859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.436286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'18","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:18.545294+0000","last_change":"2024-09-15T23:21:50.508470+0000","last_active":"2024-09-15T23:24:18.545294+0000","last_peered":"2024-09-15T23:24:18.545294+0000","last_clean":"2024-09-15T23:24:18.545294+0000","last_became_active":"2024-09-15T23:21:50.508060+0000","last_became_peered":"2024-09-15T23:21:50.508060+0000","last_unstale":"2024-09-15T23:24:18.545294+0000","last_undegraded":"2024-09-15T23:24:18.545294+0000","last_fullsized":"2024-09-15T23:24:18.545294+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:59.542252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111992+0000","last_change":"2024-09-15T23:21:59.682019+0000","last_active":"2024-09-15T23:24:03.111992+0000","last_peered":"2024-09-15T23:24:03.111992+0000","last_clean":"2024-09-15T23:24:03.111992+0000","last_became_active":"2024-09-15T23:21:59.681467+0000","last_became_peered":"2024-09-15T23:21:59.681467+0000","last_unstale":"2024-09-15T23:24:03.111992+0000","last_undegraded":"2024-09-15T23:24:03.111992+0000","last_fullsized":"2024-09-15T23:24:03.111992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:27.650382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109766+0000","last_change":"2024-09-15T23:21:52.536345+0000","last_active":"2024-09-15T23:24:03.109766+0000","last_peered":"2024-09-15T23:24:03.109766+0000","last_clean":"2024-09-15T23:24:03.109766+0000","last_became_active":"2024-09-15T23:21:52.536027+0000","last_became_peered":"2024-09-15T23:21:52.536027+0000","last_unstale":"2024-09-15T23:24:03.109766+0000","last_undegraded":"2024-09-15T23:24:03.109766+0000","last_fullsized":"2024-09-15T23:24:03.109766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:16.435912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117716+0000","last_change":"2024-09-15T23:21:50.562945+0000","last_active":"2024-09-15T23:24:03.117716+0000","last_peered":"2024-09-15T23:24:03.117716+0000","last_clean":"2024-09-15T23:24:03.117716+0000","last_became_active":"2024-09-15T23:21:50.561632+0000","last_became_peered":"2024-09-15T23:21:50.561632+0000","last_unstale":"2024-09-15T23:24:03.117716+0000","last_undegraded":"2024-09-15T23:24:03.117716+0000","last_fullsized":"2024-09-15T23:24:03.117716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:44.545489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115268+0000","last_change":"2024-09-15T23:21:59.564793+0000","last_active":"2024-09-15T23:24:03.115268+0000","last_peered":"2024-09-15T23:24:03.115268+0000","last_clean":"2024-09-15T23:24:03.115268+0000","last_became_active":"2024-09-15T23:21:59.564140+0000","last_became_peered":"2024-09-15T23:21:59.564140+0000","last_unstale":"2024-09-15T23:24:03.115268+0000","last_undegraded":"2024-09-15T23:24:03.115268+0000","last_fullsized":"2024-09-15T23:24:03.115268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:41.464962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113451+0000","last_change":"2024-09-15T23:21:52.533521+0000","last_active":"2024-09-15T23:24:03.113451+0000","last_peered":"2024-09-15T23:24:03.113451+0000","last_clean":"2024-09-15T23:24:03.113451+0000","last_became_active":"2024-09-15T23:21:52.532195+0000","last_became_peered":"2024-09-15T23:21:52.532195+0000","last_unstale":"2024-09-15T23:24:03.113451+0000","last_undegraded":"2024-09-15T23:24:03.113451+0000","last_fullsized":"2024-09-15T23:24:03.113451+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:14.229018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194243+0000","last_change":"2024-09-15T23:21:50.562893+0000","last_active":"2024-09-15T23:24:03.194243+0000","last_peered":"2024-09-15T23:24:03.194243+0000","last_clean":"2024-09-15T23:24:03.194243+0000","last_became_active":"2024-09-15T23:21:50.562516+0000","last_became_peered":"2024-09-15T23:21:50.562516+0000","last_unstale":"2024-09-15T23:24:03.194243+0000","last_undegraded":"2024-09-15T23:24:03.194243+0000","last_fullsized":"2024-09-15T23:24:03.194243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:47:47.442458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117856+0000","last_change":"2024-09-15T23:21:59.682091+0000","last_active":"2024-09-15T23:24:03.117856+0000","last_peered":"2024-09-15T23:24:03.117856+0000","last_clean":"2024-09-15T23:24:03.117856+0000","last_became_active":"2024-09-15T23:21:59.681574+0000","last_became_peered":"2024-09-15T23:21:59.681574+0000","last_unstale":"2024-09-15T23:24:03.117856+0000","last_undegraded":"2024-09-15T23:24:03.117856+0000","last_fullsized":"2024-09-15T23:24:03.117856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:04.991608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114426+0000","last_change":"2024-09-15T23:21:52.533811+0000","last_active":"2024-09-15T23:24:03.114426+0000","last_peered":"2024-09-15T23:24:03.114426+0000","last_clean":"2024-09-15T23:24:03.114426+0000","last_became_active":"2024-09-15T23:21:52.533487+0000","last_became_peered":"2024-09-15T23:21:52.533487+0000","last_unstale":"2024-09-15T23:24:03.114426+0000","last_undegraded":"2024-09-15T23:24:03.114426+0000","last_fullsized":"2024-09-15T23:24:03.114426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:44.193212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:25.288686+0000","last_change":"2024-09-15T23:21:50.510883+0000","last_active":"2024-09-15T23:24:25.288686+0000","last_peered":"2024-09-15T23:24:25.288686+0000","last_clean":"2024-09-15T23:24:25.288686+0000","last_became_active":"2024-09-15T23:21:50.510478+0000","last_became_peered":"2024-09-15T23:21:50.510478+0000","last_unstale":"2024-09-15T23:24:25.288686+0000","last_undegraded":"2024-09-15T23:24:25.288686+0000","last_fullsized":"2024-09-15T23:24:25.288686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:35.454885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109705+0000","last_change":"2024-09-15T23:21:59.561816+0000","last_active":"2024-09-15T23:24:03.109705+0000","last_peered":"2024-09-15T23:24:03.109705+0000","last_clean":"2024-09-15T23:24:03.109705+0000","last_became_active":"2024-09-15T23:21:59.561434+0000","last_became_peered":"2024-09-15T23:21:59.561434+0000","last_unstale":"2024-09-15T23:24:03.109705+0000","last_undegraded":"2024-09-15T23:24:03.109705+0000","last_fullsized":"2024-09-15T23:24:03.109705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:59.945802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117955+0000","last_change":"2024-09-15T23:23:02.277204+0000","last_active":"2024-09-15T23:24:03.117955+0000","last_peered":"2024-09-15T23:24:03.117955+0000","last_clean":"2024-09-15T23:24:03.117955+0000","last_became_active":"2024-09-15T23:23:02.276770+0000","last_became_peered":"2024-09-15T23:23:02.276770+0000","last_unstale":"2024-09-15T23:24:03.117955+0000","last_undegraded":"2024-09-15T23:24:03.117955+0000","last_fullsized":"2024-09-15T23:24:03.117955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:11.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":155,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:24.151376+0000","last_change":"2024-09-15T23:21:50.561684+0000","last_active":"2024-09-15T23:24:24.151376+0000","last_peered":"2024-09-15T23:24:24.151376+0000","last_clean":"2024-09-15T23:24:24.151376+0000","last_became_active":"2024-09-15T23:21:50.561335+0000","last_became_peered":"2024-09-15T23:21:50.561335+0000","last_unstale":"2024-09-15T23:24:24.151376+0000","last_undegraded":"2024-09-15T23:24:24.151376+0000","last_fullsized":"2024-09-15T23:24:24.151376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:23.897250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":109,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109863+0000","last_change":"2024-09-15T23:21:59.681858+0000","last_active":"2024-09-15T23:24:03.109863+0000","last_peered":"2024-09-15T23:24:03.109863+0000","last_clean":"2024-09-15T23:24:03.109863+0000","last_became_active":"2024-09-15T23:21:59.681441+0000","last_became_peered":"2024-09-15T23:21:59.681441+0000","last_unstale":"2024-09-15T23:24:03.109863+0000","last_undegraded":"2024-09-15T23:24:03.109863+0000","last_fullsized":"2024-09-15T23:24:03.109863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:28.513719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109320+0000","last_change":"2024-09-15T23:21:52.536141+0000","last_active":"2024-09-15T23:24:03.109320+0000","last_peered":"2024-09-15T23:24:03.109320+0000","last_clean":"2024-09-15T23:24:03.109320+0000","last_became_active":"2024-09-15T23:21:52.535832+0000","last_became_peered":"2024-09-15T23:21:52.535832+0000","last_unstale":"2024-09-15T23:24:03.109320+0000","last_undegraded":"2024-09-15T23:24:03.109320+0000","last_fullsized":"2024-09-15T23:24:03.109320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:21:44.341412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.123671+0000","last_change":"2024-09-15T23:24:03.123671+0000","last_active":"2024-09-15T23:24:03.123671+0000","last_peered":"2024-09-15T23:24:03.123671+0000","last_clean":"2024-09-15T23:24:03.123671+0000","last_became_active":"2024-09-15T23:24:03.119585+0000","last_became_peered":"2024-09-15T23:24:03.119585+0000","last_unstale":"2024-09-15T23:24:03.123671+0000","last_undegraded":"2024-09-15T23:24:03.123671+0000","last_fullsized":"2024-09-15T23:24:03.123671+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:30.734156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118359+0000","last_change":"2024-09-15T23:21:59.682118+0000","last_active":"2024-09-15T23:24:03.118359+0000","last_peered":"2024-09-15T23:24:03.118359+0000","last_clean":"2024-09-15T23:24:03.118359+0000","last_became_active":"2024-09-15T23:21:59.681500+0000","last_became_peered":"2024-09-15T23:21:59.681500+0000","last_unstale":"2024-09-15T23:24:03.118359+0000","last_undegraded":"2024-09-15T23:24:03.118359+0000","last_fullsized":"2024-09-15T23:24:03.118359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:58.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112298+0000","last_change":"2024-09-15T23:21:52.540458+0000","last_active":"2024-09-15T23:24:03.112298+0000","last_peered":"2024-09-15T23:24:03.112298+0000","last_clean":"2024-09-15T23:24:03.112298+0000","last_became_active":"2024-09-15T23:21:52.540112+0000","last_became_peered":"2024-09-15T23:21:52.540112+0000","last_unstale":"2024-09-15T23:24:03.112298+0000","last_undegraded":"2024-09-15T23:24:03.112298+0000","last_fullsized":"2024-09-15T23:24:03.112298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:15.240785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111097+0000","last_change":"2024-09-15T23:21:50.513124+0000","last_active":"2024-09-15T23:24:03.111097+0000","last_peered":"2024-09-15T23:24:03.111097+0000","last_clean":"2024-09-15T23:24:03.111097+0000","last_became_active":"2024-09-15T23:21:50.512705+0000","last_became_peered":"2024-09-15T23:21:50.512705+0000","last_unstale":"2024-09-15T23:24:03.111097+0000","last_undegraded":"2024-09-15T23:24:03.111097+0000","last_fullsized":"2024-09-15T23:24:03.111097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.417937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":354,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113701+0000","last_change":"2024-09-15T23:20:40.656652+0000","last_active":"2024-09-15T23:24:03.113701+0000","last_peered":"2024-09-15T23:24:03.113701+0000","last_clean":"2024-09-15T23:24:03.113701+0000","last_became_active":"2024-09-15T23:20:40.656266+0000","last_became_peered":"2024-09-15T23:20:40.656266+0000","last_unstale":"2024-09-15T23:24:03.113701+0000","last_undegraded":"2024-09-15T23:24:03.113701+0000","last_fullsized":"2024-09-15T23:24:03.113701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_clean_scrub_stamp":"2024-09-15T23:17:15.186850+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:42.317006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113728+0000","last_change":"2024-09-15T23:21:59.563777+0000","last_active":"2024-09-15T23:24:03.113728+0000","last_peered":"2024-09-15T23:24:03.113728+0000","last_clean":"2024-09-15T23:24:03.113728+0000","last_became_active":"2024-09-15T23:21:59.563320+0000","last_became_peered":"2024-09-15T23:21:59.563320+0000","last_unstale":"2024-09-15T23:24:03.113728+0000","last_undegraded":"2024-09-15T23:24:03.113728+0000","last_fullsized":"2024-09-15T23:24:03.113728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:25:07.791923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114145+0000","last_change":"2024-09-15T23:21:52.539842+0000","last_active":"2024-09-15T23:24:03.114145+0000","last_peered":"2024-09-15T23:24:03.114145+0000","last_clean":"2024-09-15T23:24:03.114145+0000","last_became_active":"2024-09-15T23:21:52.538683+0000","last_became_peered":"2024-09-15T23:21:52.538683+0000","last_unstale":"2024-09-15T23:24:03.114145+0000","last_undegraded":"2024-09-15T23:24:03.114145+0000","last_fullsized":"2024-09-15T23:24:03.114145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:43.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:23.286560+0000","last_change":"2024-09-15T23:21:50.767601+0000","last_active":"2024-09-15T23:24:23.286560+0000","last_peered":"2024-09-15T23:24:23.286560+0000","last_clean":"2024-09-15T23:24:23.286560+0000","last_became_active":"2024-09-15T23:21:50.767149+0000","last_became_peered":"2024-09-15T23:21:50.767149+0000","last_unstale":"2024-09-15T23:24:23.286560+0000","last_undegraded":"2024-09-15T23:24:23.286560+0000","last_fullsized":"2024-09-15T23:24:23.286560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:23.636551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":110,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117168+0000","last_change":"2024-09-15T23:21:59.563069+0000","last_active":"2024-09-15T23:24:03.117168+0000","last_peered":"2024-09-15T23:24:03.117168+0000","last_clean":"2024-09-15T23:24:03.117168+0000","last_became_active":"2024-09-15T23:21:59.562658+0000","last_became_peered":"2024-09-15T23:21:59.562658+0000","last_unstale":"2024-09-15T23:24:03.117168+0000","last_undegraded":"2024-09-15T23:24:03.117168+0000","last_fullsized":"2024-09-15T23:24:03.117168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:23.067198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113535+0000","last_change":"2024-09-15T23:21:52.535289+0000","last_active":"2024-09-15T23:24:03.113535+0000","last_peered":"2024-09-15T23:24:03.113535+0000","last_clean":"2024-09-15T23:24:03.113535+0000","last_became_active":"2024-09-15T23:21:52.534885+0000","last_became_peered":"2024-09-15T23:21:52.534885+0000","last_unstale":"2024-09-15T23:24:03.113535+0000","last_undegraded":"2024-09-15T23:24:03.113535+0000","last_fullsized":"2024-09-15T23:24:03.113535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:42.293786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118612+0000","last_change":"2024-09-15T23:21:50.513246+0000","last_active":"2024-09-15T23:24:03.118612+0000","last_peered":"2024-09-15T23:24:03.118612+0000","last_clean":"2024-09-15T23:24:03.118612+0000","last_became_active":"2024-09-15T23:21:50.512740+0000","last_became_peered":"2024-09-15T23:21:50.512740+0000","last_unstale":"2024-09-15T23:24:03.118612+0000","last_undegraded":"2024-09-15T23:24:03.118612+0000","last_fullsized":"2024-09-15T23:24:03.118612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:59.201970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193756+0000","last_change":"2024-09-15T23:21:59.568565+0000","last_active":"2024-09-15T23:24:03.193756+0000","last_peered":"2024-09-15T23:24:03.193756+0000","last_clean":"2024-09-15T23:24:03.193756+0000","last_became_active":"2024-09-15T23:21:59.567789+0000","last_became_peered":"2024-09-15T23:21:59.567789+0000","last_unstale":"2024-09-15T23:24:03.193756+0000","last_undegraded":"2024-09-15T23:24:03.193756+0000","last_fullsized":"2024-09-15T23:24:03.193756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:02.106971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114106+0000","last_change":"2024-09-15T23:21:52.533727+0000","last_active":"2024-09-15T23:24:03.114106+0000","last_peered":"2024-09-15T23:24:03.114106+0000","last_clean":"2024-09-15T23:24:03.114106+0000","last_became_active":"2024-09-15T23:21:52.532350+0000","last_became_peered":"2024-09-15T23:21:52.532350+0000","last_unstale":"2024-09-15T23:24:03.114106+0000","last_undegraded":"2024-09-15T23:24:03.114106+0000","last_fullsized":"2024-09-15T23:24:03.114106+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:03.624837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.124878+0000","last_change":"2024-09-15T23:24:03.124430+0000","last_active":"2024-09-15T23:24:03.124878+0000","last_peered":"2024-09-15T23:24:03.124878+0000","last_clean":"2024-09-15T23:24:03.124878+0000","last_became_active":"2024-09-15T23:24:03.119042+0000","last_became_peered":"2024-09-15T23:24:03.119042+0000","last_unstale":"2024-09-15T23:24:03.124878+0000","last_undegraded":"2024-09-15T23:24:03.124878+0000","last_fullsized":"2024-09-15T23:24:03.124878+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:52.065543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117798+0000","last_change":"2024-09-15T23:21:59.566429+0000","last_active":"2024-09-15T23:24:03.117798+0000","last_peered":"2024-09-15T23:24:03.117798+0000","last_clean":"2024-09-15T23:24:03.117798+0000","last_became_active":"2024-09-15T23:21:59.565835+0000","last_became_peered":"2024-09-15T23:21:59.565835+0000","last_unstale":"2024-09-15T23:24:03.117798+0000","last_undegraded":"2024-09-15T23:24:03.117798+0000","last_fullsized":"2024-09-15T23:24:03.117798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:38.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116276+0000","last_change":"2024-09-15T23:21:52.539544+0000","last_active":"2024-09-15T23:24:03.116276+0000","last_peered":"2024-09-15T23:24:03.116276+0000","last_clean":"2024-09-15T23:24:03.116276+0000","last_became_active":"2024-09-15T23:21:52.539203+0000","last_became_peered":"2024-09-15T23:21:52.539203+0000","last_unstale":"2024-09-15T23:24:03.116276+0000","last_undegraded":"2024-09-15T23:24:03.116276+0000","last_fullsized":"2024-09-15T23:24:03.116276+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.193965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:07.046123+0000","last_change":"2024-09-15T23:21:50.513543+0000","last_active":"2024-09-15T23:24:07.046123+0000","last_peered":"2024-09-15T23:24:07.046123+0000","last_clean":"2024-09-15T23:24:07.046123+0000","last_became_active":"2024-09-15T23:21:50.512812+0000","last_became_peered":"2024-09-15T23:21:50.512812+0000","last_unstale":"2024-09-15T23:24:07.046123+0000","last_undegraded":"2024-09-15T23:24:07.046123+0000","last_fullsized":"2024-09-15T23:24:07.046123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:50.954096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117689+0000","last_change":"2024-09-15T23:21:59.570163+0000","last_active":"2024-09-15T23:24:03.117689+0000","last_peered":"2024-09-15T23:24:03.117689+0000","last_clean":"2024-09-15T23:24:03.117689+0000","last_became_active":"2024-09-15T23:21:59.569805+0000","last_became_peered":"2024-09-15T23:21:59.569805+0000","last_unstale":"2024-09-15T23:24:03.117689+0000","last_undegraded":"2024-09-15T23:24:03.117689+0000","last_fullsized":"2024-09-15T23:24:03.117689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:28.530817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117877+0000","last_change":"2024-09-15T23:21:52.537850+0000","last_active":"2024-09-15T23:24:03.117877+0000","last_peered":"2024-09-15T23:24:03.117877+0000","last_clean":"2024-09-15T23:24:03.117877+0000","last_became_active":"2024-09-15T23:21:52.536579+0000","last_became_peered":"2024-09-15T23:21:52.536579+0000","last_unstale":"2024-09-15T23:24:03.117877+0000","last_undegraded":"2024-09-15T23:24:03.117877+0000","last_fullsized":"2024-09-15T23:24:03.117877+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:06.353615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118588+0000","last_change":"2024-09-15T23:21:50.562221+0000","last_active":"2024-09-15T23:24:03.118588+0000","last_peered":"2024-09-15T23:24:03.118588+0000","last_clean":"2024-09-15T23:24:03.118588+0000","last_became_active":"2024-09-15T23:21:50.561788+0000","last_became_peered":"2024-09-15T23:21:50.561788+0000","last_unstale":"2024-09-15T23:24:03.118588+0000","last_undegraded":"2024-09-15T23:24:03.118588+0000","last_fullsized":"2024-09-15T23:24:03.118588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.853812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109849+0000","last_change":"2024-09-15T23:23:02.383003+0000","last_active":"2024-09-15T23:24:03.109849+0000","last_peered":"2024-09-15T23:24:03.109849+0000","last_clean":"2024-09-15T23:24:03.109849+0000","last_became_active":"2024-09-15T23:23:02.382629+0000","last_became_peered":"2024-09-15T23:23:02.382629+0000","last_unstale":"2024-09-15T23:24:03.109849+0000","last_undegraded":"2024-09-15T23:24:03.109849+0000","last_fullsized":"2024-09-15T23:24:03.109849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:50.114689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113567+0000","last_change":"2024-09-15T23:21:52.539566+0000","last_active":"2024-09-15T23:24:03.113567+0000","last_peered":"2024-09-15T23:24:03.113567+0000","last_clean":"2024-09-15T23:24:03.113567+0000","last_became_active":"2024-09-15T23:21:52.539219+0000","last_became_peered":"2024-09-15T23:21:52.539219+0000","last_unstale":"2024-09-15T23:24:03.113567+0000","last_undegraded":"2024-09-15T23:24:03.113567+0000","last_fullsized":"2024-09-15T23:24:03.113567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:41.266308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":173,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:24.151855+0000","last_change":"2024-09-15T23:24:03.127667+0000","last_active":"2024-09-15T23:24:24.151855+0000","last_peered":"2024-09-15T23:24:24.151855+0000","last_clean":"2024-09-15T23:24:24.151855+0000","last_became_active":"2024-09-15T23:24:03.121613+0000","last_became_peered":"2024-09-15T23:24:03.121613+0000","last_unstale":"2024-09-15T23:24:24.151855+0000","last_undegraded":"2024-09-15T23:24:24.151855+0000","last_fullsized":"2024-09-15T23:24:24.151855+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:54:24.166895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1122,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194642+0000","last_change":"2024-09-15T23:21:59.568087+0000","last_active":"2024-09-15T23:24:03.194642+0000","last_peered":"2024-09-15T23:24:03.194642+0000","last_clean":"2024-09-15T23:24:03.194642+0000","last_became_active":"2024-09-15T23:21:59.567693+0000","last_became_peered":"2024-09-15T23:21:59.567693+0000","last_unstale":"2024-09-15T23:24:03.194642+0000","last_undegraded":"2024-09-15T23:24:03.194642+0000","last_fullsized":"2024-09-15T23:24:03.194642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.231873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114396+0000","last_change":"2024-09-15T23:21:52.531090+0000","last_active":"2024-09-15T23:24:03.114396+0000","last_peered":"2024-09-15T23:24:03.114396+0000","last_clean":"2024-09-15T23:24:03.114396+0000","last_became_active":"2024-09-15T23:21:52.530738+0000","last_became_peered":"2024-09-15T23:21:52.530738+0000","last_unstale":"2024-09-15T23:24:03.114396+0000","last_undegraded":"2024-09-15T23:24:03.114396+0000","last_fullsized":"2024-09-15T23:24:03.114396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:35.325128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117423+0000","last_change":"2024-09-15T23:21:50.514100+0000","last_active":"2024-09-15T23:24:03.117423+0000","last_peered":"2024-09-15T23:24:03.117423+0000","last_clean":"2024-09-15T23:24:03.117423+0000","last_became_active":"2024-09-15T23:21:50.513629+0000","last_became_peered":"2024-09-15T23:21:50.513629+0000","last_unstale":"2024-09-15T23:24:03.117423+0000","last_undegraded":"2024-09-15T23:24:03.117423+0000","last_fullsized":"2024-09-15T23:24:03.117423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.187544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115275+0000","last_change":"2024-09-15T23:21:59.562804+0000","last_active":"2024-09-15T23:24:03.115275+0000","last_peered":"2024-09-15T23:24:03.115275+0000","last_clean":"2024-09-15T23:24:03.115275+0000","last_became_active":"2024-09-15T23:21:59.562368+0000","last_became_peered":"2024-09-15T23:21:59.562368+0000","last_unstale":"2024-09-15T23:24:03.115275+0000","last_undegraded":"2024-09-15T23:24:03.115275+0000","last_fullsized":"2024-09-15T23:24:03.115275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:21.137333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109909+0000","last_change":"2024-09-15T23:21:52.530562+0000","last_active":"2024-09-15T23:24:03.109909+0000","last_peered":"2024-09-15T23:24:03.109909+0000","last_clean":"2024-09-15T23:24:03.109909+0000","last_became_active":"2024-09-15T23:21:52.530280+0000","last_became_peered":"2024-09-15T23:21:52.530280+0000","last_unstale":"2024-09-15T23:24:03.109909+0000","last_undegraded":"2024-09-15T23:24:03.109909+0000","last_fullsized":"2024-09-15T23:24:03.109909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:26.835651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113202+0000","last_change":"2024-09-15T23:21:50.513179+0000","last_active":"2024-09-15T23:24:03.113202+0000","last_peered":"2024-09-15T23:24:03.113202+0000","last_clean":"2024-09-15T23:24:03.113202+0000","last_became_active":"2024-09-15T23:21:50.512790+0000","last_became_peered":"2024-09-15T23:21:50.512790+0000","last_unstale":"2024-09-15T23:24:03.113202+0000","last_undegraded":"2024-09-15T23:24:03.113202+0000","last_fullsized":"2024-09-15T23:24:03.113202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:56.408198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113316+0000","last_change":"2024-09-15T23:23:02.382964+0000","last_active":"2024-09-15T23:24:03.113316+0000","last_peered":"2024-09-15T23:24:03.113316+0000","last_clean":"2024-09-15T23:24:03.113316+0000","last_became_active":"2024-09-15T23:23:02.382556+0000","last_became_peered":"2024-09-15T23:23:02.382556+0000","last_unstale":"2024-09-15T23:24:03.113316+0000","last_undegraded":"2024-09-15T23:24:03.113316+0000","last_fullsized":"2024-09-15T23:24:03.113316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:49.211591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117484+0000","last_change":"2024-09-15T23:21:52.530970+0000","last_active":"2024-09-15T23:24:03.117484+0000","last_peered":"2024-09-15T23:24:03.117484+0000","last_clean":"2024-09-15T23:24:03.117484+0000","last_became_active":"2024-09-15T23:21:52.530636+0000","last_became_peered":"2024-09-15T23:21:52.530636+0000","last_unstale":"2024-09-15T23:24:03.117484+0000","last_undegraded":"2024-09-15T23:24:03.117484+0000","last_fullsized":"2024-09-15T23:24:03.117484+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:33.374658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'41","reported_seq":329,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:25.153679+0000","last_change":"2024-09-15T23:24:03.131527+0000","last_active":"2024-09-15T23:24:25.153679+0000","last_peered":"2024-09-15T23:24:25.153679+0000","last_clean":"2024-09-15T23:24:25.153679+0000","last_became_active":"2024-09-15T23:24:03.119053+0000","last_became_peered":"2024-09-15T23:24:03.119053+0000","last_unstale":"2024-09-15T23:24:25.153679+0000","last_undegraded":"2024-09-15T23:24:25.153679+0000","last_fullsized":"2024-09-15T23:24:25.153679+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:21.870435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70919,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":226,"num_read_kb":183,"num_write":43,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":62325,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113841+0000","last_change":"2024-09-15T23:23:02.277457+0000","last_active":"2024-09-15T23:24:03.113841+0000","last_peered":"2024-09-15T23:24:03.113841+0000","last_clean":"2024-09-15T23:24:03.113841+0000","last_became_active":"2024-09-15T23:23:02.276960+0000","last_became_peered":"2024-09-15T23:23:02.276960+0000","last_unstale":"2024-09-15T23:24:03.113841+0000","last_undegraded":"2024-09-15T23:24:03.113841+0000","last_fullsized":"2024-09-15T23:24:03.113841+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:15:28.823804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117534+0000","last_change":"2024-09-15T23:21:52.530541+0000","last_active":"2024-09-15T23:24:03.117534+0000","last_peered":"2024-09-15T23:24:03.117534+0000","last_clean":"2024-09-15T23:24:03.117534+0000","last_became_active":"2024-09-15T23:21:52.530217+0000","last_became_peered":"2024-09-15T23:21:52.530217+0000","last_unstale":"2024-09-15T23:24:03.117534+0000","last_undegraded":"2024-09-15T23:24:03.117534+0000","last_fullsized":"2024-09-15T23:24:03.117534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:14.864563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118500+0000","last_change":"2024-09-15T23:21:50.562203+0000","last_active":"2024-09-15T23:24:03.118500+0000","last_peered":"2024-09-15T23:24:03.118500+0000","last_clean":"2024-09-15T23:24:03.118500+0000","last_became_active":"2024-09-15T23:21:50.561827+0000","last_became_peered":"2024-09-15T23:21:50.561827+0000","last_unstale":"2024-09-15T23:24:03.118500+0000","last_undegraded":"2024-09-15T23:24:03.118500+0000","last_fullsized":"2024-09-15T23:24:03.118500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:22.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111024+0000","last_change":"2024-09-15T23:21:59.561013+0000","last_active":"2024-09-15T23:24:03.111024+0000","last_peered":"2024-09-15T23:24:03.111024+0000","last_clean":"2024-09-15T23:24:03.111024+0000","last_became_active":"2024-09-15T23:21:59.560665+0000","last_became_peered":"2024-09-15T23:21:59.560665+0000","last_unstale":"2024-09-15T23:24:03.111024+0000","last_undegraded":"2024-09-15T23:24:03.111024+0000","last_fullsized":"2024-09-15T23:24:03.111024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:00.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194109+0000","last_change":"2024-09-15T23:21:52.529922+0000","last_active":"2024-09-15T23:24:03.194109+0000","last_peered":"2024-09-15T23:24:03.194109+0000","last_clean":"2024-09-15T23:24:03.194109+0000","last_became_active":"2024-09-15T23:21:52.529179+0000","last_became_peered":"2024-09-15T23:21:52.529179+0000","last_unstale":"2024-09-15T23:24:03.194109+0000","last_undegraded":"2024-09-15T23:24:03.194109+0000","last_fullsized":"2024-09-15T23:24:03.194109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.077658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113481+0000","last_change":"2024-09-15T23:21:50.514420+0000","last_active":"2024-09-15T23:24:03.113481+0000","last_peered":"2024-09-15T23:24:03.113481+0000","last_clean":"2024-09-15T23:24:03.113481+0000","last_became_active":"2024-09-15T23:21:50.514007+0000","last_became_peered":"2024-09-15T23:21:50.514007+0000","last_unstale":"2024-09-15T23:24:03.113481+0000","last_undegraded":"2024-09-15T23:24:03.113481+0000","last_fullsized":"2024-09-15T23:24:03.113481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:36.845674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193392+0000","last_change":"2024-09-15T23:21:59.565978+0000","last_active":"2024-09-15T23:24:03.193392+0000","last_peered":"2024-09-15T23:24:03.193392+0000","last_clean":"2024-09-15T23:24:03.193392+0000","last_became_active":"2024-09-15T23:21:59.565604+0000","last_became_peered":"2024-09-15T23:21:59.565604+0000","last_unstale":"2024-09-15T23:24:03.193392+0000","last_undegraded":"2024-09-15T23:24:03.193392+0000","last_fullsized":"2024-09-15T23:24:03.193392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:47.089504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118064+0000","last_change":"2024-09-15T23:21:52.543511+0000","last_active":"2024-09-15T23:24:03.118064+0000","last_peered":"2024-09-15T23:24:03.118064+0000","last_clean":"2024-09-15T23:24:03.118064+0000","last_became_active":"2024-09-15T23:21:52.543169+0000","last_became_peered":"2024-09-15T23:21:52.543169+0000","last_unstale":"2024-09-15T23:24:03.118064+0000","last_undegraded":"2024-09-15T23:24:03.118064+0000","last_fullsized":"2024-09-15T23:24:03.118064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:05.696772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":262,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:25.288749+0000","last_change":"2024-09-15T23:21:50.518425+0000","last_active":"2024-09-15T23:24:25.288749+0000","last_peered":"2024-09-15T23:24:25.288749+0000","last_clean":"2024-09-15T23:24:25.288749+0000","last_became_active":"2024-09-15T23:21:50.518052+0000","last_became_peered":"2024-09-15T23:21:50.518052+0000","last_unstale":"2024-09-15T23:24:25.288749+0000","last_undegraded":"2024-09-15T23:24:25.288749+0000","last_fullsized":"2024-09-15T23:24:25.288749+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:02.243869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":218,"num_read_kb":218,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112035+0000","last_change":"2024-09-15T23:21:59.570310+0000","last_active":"2024-09-15T23:24:03.112035+0000","last_peered":"2024-09-15T23:24:03.112035+0000","last_clean":"2024-09-15T23:24:03.112035+0000","last_became_active":"2024-09-15T23:21:59.569909+0000","last_became_peered":"2024-09-15T23:21:59.569909+0000","last_unstale":"2024-09-15T23:24:03.112035+0000","last_undegraded":"2024-09-15T23:24:03.112035+0000","last_fullsized":"2024-09-15T23:24:03.112035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:20:27.515537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194191+0000","last_change":"2024-09-15T23:21:59.568208+0000","last_active":"2024-09-15T23:24:03.194191+0000","last_peered":"2024-09-15T23:24:03.194191+0000","last_clean":"2024-09-15T23:24:03.194191+0000","last_became_active":"2024-09-15T23:21:59.567858+0000","last_became_peered":"2024-09-15T23:21:59.567858+0000","last_unstale":"2024-09-15T23:24:03.194191+0000","last_undegraded":"2024-09-15T23:24:03.194191+0000","last_fullsized":"2024-09-15T23:24:03.194191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:34:42.895247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193532+0000","last_change":"2024-09-15T23:21:52.532679+0000","last_active":"2024-09-15T23:24:03.193532+0000","last_peered":"2024-09-15T23:24:03.193532+0000","last_clean":"2024-09-15T23:24:03.193532+0000","last_became_active":"2024-09-15T23:21:52.532345+0000","last_became_peered":"2024-09-15T23:21:52.532345+0000","last_unstale":"2024-09-15T23:24:03.193532+0000","last_undegraded":"2024-09-15T23:24:03.193532+0000","last_fullsized":"2024-09-15T23:24:03.193532+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:45.943653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118135+0000","last_change":"2024-09-15T23:21:50.510836+0000","last_active":"2024-09-15T23:24:03.118135+0000","last_peered":"2024-09-15T23:24:03.118135+0000","last_clean":"2024-09-15T23:24:03.118135+0000","last_became_active":"2024-09-15T23:21:50.510514+0000","last_became_peered":"2024-09-15T23:21:50.510514+0000","last_unstale":"2024-09-15T23:24:03.118135+0000","last_undegraded":"2024-09-15T23:24:03.118135+0000","last_fullsized":"2024-09-15T23:24:03.118135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:06.404612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116423+0000","last_change":"2024-09-15T23:23:02.383436+0000","last_active":"2024-09-15T23:24:03.116423+0000","last_peered":"2024-09-15T23:24:03.116423+0000","last_clean":"2024-09-15T23:24:03.116423+0000","last_became_active":"2024-09-15T23:23:02.382982+0000","last_became_peered":"2024-09-15T23:23:02.382982+0000","last_unstale":"2024-09-15T23:24:03.116423+0000","last_undegraded":"2024-09-15T23:24:03.116423+0000","last_fullsized":"2024-09-15T23:24:03.116423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:55:14.076604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111923+0000","last_change":"2024-09-15T23:21:52.526766+0000","last_active":"2024-09-15T23:24:03.111923+0000","last_peered":"2024-09-15T23:24:03.111923+0000","last_clean":"2024-09-15T23:24:03.111923+0000","last_became_active":"2024-09-15T23:21:52.526446+0000","last_became_peered":"2024-09-15T23:21:52.526446+0000","last_unstale":"2024-09-15T23:24:03.111923+0000","last_undegraded":"2024-09-15T23:24:03.111923+0000","last_fullsized":"2024-09-15T23:24:03.111923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:21:37.004730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109384+0000","last_change":"2024-09-15T23:21:50.512928+0000","last_active":"2024-09-15T23:24:03.109384+0000","last_peered":"2024-09-15T23:24:03.109384+0000","last_clean":"2024-09-15T23:24:03.109384+0000","last_became_active":"2024-09-15T23:21:50.512558+0000","last_became_peered":"2024-09-15T23:21:50.512558+0000","last_unstale":"2024-09-15T23:24:03.109384+0000","last_undegraded":"2024-09-15T23:24:03.109384+0000","last_fullsized":"2024-09-15T23:24:03.109384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:08.653084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114309+0000","last_change":"2024-09-15T23:21:50.516920+0000","last_active":"2024-09-15T23:24:03.114309+0000","last_peered":"2024-09-15T23:24:03.114309+0000","last_clean":"2024-09-15T23:24:03.114309+0000","last_became_active":"2024-09-15T23:21:50.516573+0000","last_became_peered":"2024-09-15T23:21:50.516573+0000","last_unstale":"2024-09-15T23:24:03.114309+0000","last_undegraded":"2024-09-15T23:24:03.114309+0000","last_fullsized":"2024-09-15T23:24:03.114309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.367903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117266+0000","last_change":"2024-09-15T23:21:52.536696+0000","last_active":"2024-09-15T23:24:03.117266+0000","last_peered":"2024-09-15T23:24:03.117266+0000","last_clean":"2024-09-15T23:24:03.117266+0000","last_became_active":"2024-09-15T23:21:52.536338+0000","last_became_peered":"2024-09-15T23:21:52.536338+0000","last_unstale":"2024-09-15T23:24:03.117266+0000","last_undegraded":"2024-09-15T23:24:03.117266+0000","last_fullsized":"2024-09-15T23:24:03.117266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:05.152749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112825+0000","last_change":"2024-09-15T23:21:59.562968+0000","last_active":"2024-09-15T23:24:03.112825+0000","last_peered":"2024-09-15T23:24:03.112825+0000","last_clean":"2024-09-15T23:24:03.112825+0000","last_became_active":"2024-09-15T23:21:59.562579+0000","last_became_peered":"2024-09-15T23:21:59.562579+0000","last_unstale":"2024-09-15T23:24:03.112825+0000","last_undegraded":"2024-09-15T23:24:03.112825+0000","last_fullsized":"2024-09-15T23:24:03.112825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:02.853057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":157,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:25.290273+0000","last_change":"2024-09-15T23:21:50.513944+0000","last_active":"2024-09-15T23:24:25.290273+0000","last_peered":"2024-09-15T23:24:25.290273+0000","last_clean":"2024-09-15T23:24:25.290273+0000","last_became_active":"2024-09-15T23:21:50.513484+0000","last_became_peered":"2024-09-15T23:21:50.513484+0000","last_unstale":"2024-09-15T23:24:25.290273+0000","last_undegraded":"2024-09-15T23:24:25.290273+0000","last_fullsized":"2024-09-15T23:24:25.290273+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:35.719344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118061+0000","last_change":"2024-09-15T23:21:52.539467+0000","last_active":"2024-09-15T23:24:03.118061+0000","last_peered":"2024-09-15T23:24:03.118061+0000","last_clean":"2024-09-15T23:24:03.118061+0000","last_became_active":"2024-09-15T23:21:52.538998+0000","last_became_peered":"2024-09-15T23:21:52.538998+0000","last_unstale":"2024-09-15T23:24:03.118061+0000","last_undegraded":"2024-09-15T23:24:03.118061+0000","last_fullsized":"2024-09-15T23:24:03.118061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:22.213857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112854+0000","last_change":"2024-09-15T23:21:59.564842+0000","last_active":"2024-09-15T23:24:03.112854+0000","last_peered":"2024-09-15T23:24:03.112854+0000","last_clean":"2024-09-15T23:24:03.112854+0000","last_became_active":"2024-09-15T23:21:59.564408+0000","last_became_peered":"2024-09-15T23:21:59.564408+0000","last_unstale":"2024-09-15T23:24:03.112854+0000","last_undegraded":"2024-09-15T23:24:03.112854+0000","last_fullsized":"2024-09-15T23:24:03.112854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:17.909020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115745+0000","last_change":"2024-09-15T23:23:02.149420+0000","last_active":"2024-09-15T23:24:03.115745+0000","last_peered":"2024-09-15T23:24:03.115745+0000","last_clean":"2024-09-15T23:24:03.115745+0000","last_became_active":"2024-09-15T23:23:02.149003+0000","last_became_peered":"2024-09-15T23:23:02.149003+0000","last_unstale":"2024-09-15T23:24:03.115745+0000","last_undegraded":"2024-09-15T23:24:03.115745+0000","last_fullsized":"2024-09-15T23:24:03.115745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:45.490322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.532769+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532426+0000","last_became_peered":"2024-09-15T23:21:52.532426+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:09.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.533148+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532823+0000","last_became_peered":"2024-09-15T23:21:52.532823+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:01.833945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193803+0000","last_change":"2024-09-15T23:21:52.536790+0000","last_active":"2024-09-15T23:24:03.193803+0000","last_peered":"2024-09-15T23:24:03.193803+0000","last_clean":"2024-09-15T23:24:03.193803+0000","last_became_active":"2024-09-15T23:21:52.536515+0000","last_became_peered":"2024-09-15T23:21:52.536515+0000","last_unstale":"2024-09-15T23:24:03.193803+0000","last_undegraded":"2024-09-15T23:24:03.193803+0000","last_fullsized":"2024-09-15T23:24:03.193803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:37.207759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109800+0000","last_change":"2024-09-15T23:23:02.382842+0000","last_active":"2024-09-15T23:24:03.109800+0000","last_peered":"2024-09-15T23:24:03.109800+0000","last_clean":"2024-09-15T23:24:03.109800+0000","last_became_active":"2024-09-15T23:23:02.382455+0000","last_became_peered":"2024-09-15T23:23:02.382455+0000","last_unstale":"2024-09-15T23:24:03.109800+0000","last_undegraded":"2024-09-15T23:24:03.109800+0000","last_fullsized":"2024-09-15T23:24:03.109800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:23.617641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.540628+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.540308+0000","last_became_peered":"2024-09-15T23:21:52.540308+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:16.660637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113381+0000","last_change":"2024-09-15T23:21:52.533780+0000","last_active":"2024-09-15T23:24:03.113381+0000","last_peered":"2024-09-15T23:24:03.113381+0000","last_clean":"2024-09-15T23:24:03.113381+0000","last_became_active":"2024-09-15T23:21:52.533354+0000","last_became_peered":"2024-09-15T23:21:52.533354+0000","last_unstale":"2024-09-15T23:24:03.113381+0000","last_undegraded":"2024-09-15T23:24:03.113381+0000","last_fullsized":"2024-09-15T23:24:03.113381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.381059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112801+0000","last_change":"2024-09-15T23:21:52.536406+0000","last_active":"2024-09-15T23:24:03.112801+0000","last_peered":"2024-09-15T23:24:03.112801+0000","last_clean":"2024-09-15T23:24:03.112801+0000","last_became_active":"2024-09-15T23:21:52.536091+0000","last_became_peered":"2024-09-15T23:21:52.536091+0000","last_unstale":"2024-09-15T23:24:03.112801+0000","last_undegraded":"2024-09-15T23:24:03.112801+0000","last_fullsized":"2024-09-15T23:24:03.112801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:30.452201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117879+0000","last_change":"2024-09-15T23:21:52.536333+0000","last_active":"2024-09-15T23:24:03.117879+0000","last_peered":"2024-09-15T23:24:03.117879+0000","last_clean":"2024-09-15T23:24:03.117879+0000","last_became_active":"2024-09-15T23:21:52.534677+0000","last_became_peered":"2024-09-15T23:21:52.534677+0000","last_unstale":"2024-09-15T23:24:03.117879+0000","last_undegraded":"2024-09-15T23:24:03.117879+0000","last_fullsized":"2024-09-15T23:24:03.117879+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:05.287914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112116+0000","last_change":"2024-09-15T23:21:52.536442+0000","last_active":"2024-09-15T23:24:03.112116+0000","last_peered":"2024-09-15T23:24:03.112116+0000","last_clean":"2024-09-15T23:24:03.112116+0000","last_became_active":"2024-09-15T23:21:52.536183+0000","last_became_peered":"2024-09-15T23:21:52.536183+0000","last_unstale":"2024-09-15T23:24:03.112116+0000","last_undegraded":"2024-09-15T23:24:03.112116+0000","last_fullsized":"2024-09-15T23:24:03.112116+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:41.641370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117398+0000","last_change":"2024-09-15T23:23:02.277365+0000","last_active":"2024-09-15T23:24:03.117398+0000","last_peered":"2024-09-15T23:24:03.117398+0000","last_clean":"2024-09-15T23:24:03.117398+0000","last_became_active":"2024-09-15T23:23:02.276905+0000","last_became_peered":"2024-09-15T23:23:02.276905+0000","last_unstale":"2024-09-15T23:24:03.117398+0000","last_undegraded":"2024-09-15T23:24:03.117398+0000","last_fullsized":"2024-09-15T23:24:03.117398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:21.100657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109788+0000","last_change":"2024-09-15T23:21:52.531171+0000","last_active":"2024-09-15T23:24:03.109788+0000","last_peered":"2024-09-15T23:24:03.109788+0000","last_clean":"2024-09-15T23:24:03.109788+0000","last_became_active":"2024-09-15T23:21:52.530724+0000","last_became_peered":"2024-09-15T23:21:52.530724+0000","last_unstale":"2024-09-15T23:24:03.109788+0000","last_undegraded":"2024-09-15T23:24:03.109788+0000","last_fullsized":"2024-09-15T23:24:03.109788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:11.009119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115400+0000","last_change":"2024-09-15T23:21:52.530433+0000","last_active":"2024-09-15T23:24:03.115400+0000","last_peered":"2024-09-15T23:24:03.115400+0000","last_clean":"2024-09-15T23:24:03.115400+0000","last_became_active":"2024-09-15T23:21:52.530058+0000","last_became_peered":"2024-09-15T23:21:52.530058+0000","last_unstale":"2024-09-15T23:24:03.115400+0000","last_undegraded":"2024-09-15T23:24:03.115400+0000","last_fullsized":"2024-09-15T23:24:03.115400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:51.208820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113218+0000","last_change":"2024-09-15T23:21:52.534639+0000","last_active":"2024-09-15T23:24:03.113218+0000","last_peered":"2024-09-15T23:24:03.113218+0000","last_clean":"2024-09-15T23:24:03.113218+0000","last_became_active":"2024-09-15T23:21:52.534362+0000","last_became_peered":"2024-09-15T23:21:52.534362+0000","last_unstale":"2024-09-15T23:24:03.113218+0000","last_undegraded":"2024-09-15T23:24:03.113218+0000","last_fullsized":"2024-09-15T23:24:03.113218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:58:12.023501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194630+0000","last_change":"2024-09-15T23:21:52.532758+0000","last_active":"2024-09-15T23:24:03.194630+0000","last_peered":"2024-09-15T23:24:03.194630+0000","last_clean":"2024-09-15T23:24:03.194630+0000","last_became_active":"2024-09-15T23:21:52.532412+0000","last_became_peered":"2024-09-15T23:21:52.532412+0000","last_unstale":"2024-09-15T23:24:03.194630+0000","last_undegraded":"2024-09-15T23:24:03.194630+0000","last_fullsized":"2024-09-15T23:24:03.194630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:46.285711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116737+0000","last_change":"2024-09-15T23:21:52.533928+0000","last_active":"2024-09-15T23:24:03.116737+0000","last_peered":"2024-09-15T23:24:03.116737+0000","last_clean":"2024-09-15T23:24:03.116737+0000","last_became_active":"2024-09-15T23:21:52.533578+0000","last_became_peered":"2024-09-15T23:21:52.533578+0000","last_unstale":"2024-09-15T23:24:03.116737+0000","last_undegraded":"2024-09-15T23:24:03.116737+0000","last_fullsized":"2024-09-15T23:24:03.116737+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:49.476316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116894+0000","last_change":"2024-09-15T23:21:52.533417+0000","last_active":"2024-09-15T23:24:03.116894+0000","last_peered":"2024-09-15T23:24:03.116894+0000","last_clean":"2024-09-15T23:24:03.116894+0000","last_became_active":"2024-09-15T23:21:52.533085+0000","last_became_peered":"2024-09-15T23:21:52.533085+0000","last_unstale":"2024-09-15T23:24:03.116894+0000","last_undegraded":"2024-09-15T23:24:03.116894+0000","last_fullsized":"2024-09-15T23:24:03.116894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:20.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193763+0000","last_change":"2024-09-15T23:21:52.542340+0000","last_active":"2024-09-15T23:24:03.193763+0000","last_peered":"2024-09-15T23:24:03.193763+0000","last_clean":"2024-09-15T23:24:03.193763+0000","last_became_active":"2024-09-15T23:21:52.542048+0000","last_became_peered":"2024-09-15T23:21:52.542048+0000","last_unstale":"2024-09-15T23:24:03.193763+0000","last_undegraded":"2024-09-15T23:24:03.193763+0000","last_fullsized":"2024-09-15T23:24:03.193763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:57.557420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193941+0000","last_change":"2024-09-15T23:21:52.535576+0000","last_active":"2024-09-15T23:24:03.193941+0000","last_peered":"2024-09-15T23:24:03.193941+0000","last_clean":"2024-09-15T23:24:03.193941+0000","last_became_active":"2024-09-15T23:21:52.535223+0000","last_became_peered":"2024-09-15T23:21:52.535223+0000","last_unstale":"2024-09-15T23:24:03.193941+0000","last_undegraded":"2024-09-15T23:24:03.193941+0000","last_fullsized":"2024-09-15T23:24:03.193941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:45.874927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109691+0000","last_change":"2024-09-15T23:23:02.150556+0000","last_active":"2024-09-15T23:24:03.109691+0000","last_peered":"2024-09-15T23:24:03.109691+0000","last_clean":"2024-09-15T23:24:03.109691+0000","last_became_active":"2024-09-15T23:23:02.150253+0000","last_became_peered":"2024-09-15T23:23:02.150253+0000","last_unstale":"2024-09-15T23:24:03.109691+0000","last_undegraded":"2024-09-15T23:24:03.109691+0000","last_fullsized":"2024-09-15T23:24:03.109691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:47.950356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110875+0000","last_change":"2024-09-15T23:21:52.537012+0000","last_active":"2024-09-15T23:24:03.110875+0000","last_peered":"2024-09-15T23:24:03.110875+0000","last_clean":"2024-09-15T23:24:03.110875+0000","last_became_active":"2024-09-15T23:21:52.536245+0000","last_became_peered":"2024-09-15T23:21:52.536245+0000","last_unstale":"2024-09-15T23:24:03.110875+0000","last_undegraded":"2024-09-15T23:24:03.110875+0000","last_fullsized":"2024-09-15T23:24:03.110875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:31.150491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114034+0000","last_change":"2024-09-15T23:21:52.540431+0000","last_active":"2024-09-15T23:24:03.114034+0000","last_peered":"2024-09-15T23:24:03.114034+0000","last_clean":"2024-09-15T23:24:03.114034+0000","last_became_active":"2024-09-15T23:21:52.540104+0000","last_became_peered":"2024-09-15T23:21:52.540104+0000","last_unstale":"2024-09-15T23:24:03.114034+0000","last_undegraded":"2024-09-15T23:24:03.114034+0000","last_fullsized":"2024-09-15T23:24:03.114034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.550258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114876+0000","last_change":"2024-09-15T23:21:52.537590+0000","last_active":"2024-09-15T23:24:03.114876+0000","last_peered":"2024-09-15T23:24:03.114876+0000","last_clean":"2024-09-15T23:24:03.114876+0000","last_became_active":"2024-09-15T23:21:52.533804+0000","last_became_peered":"2024-09-15T23:21:52.533804+0000","last_unstale":"2024-09-15T23:24:03.114876+0000","last_undegraded":"2024-09-15T23:24:03.114876+0000","last_fullsized":"2024-09-15T23:24:03.114876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:48.449090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117952+0000","last_change":"2024-09-15T23:21:52.536130+0000","last_active":"2024-09-15T23:24:03.117952+0000","last_peered":"2024-09-15T23:24:03.117952+0000","last_clean":"2024-09-15T23:24:03.117952+0000","last_became_active":"2024-09-15T23:21:52.535336+0000","last_became_peered":"2024-09-15T23:21:52.535336+0000","last_unstale":"2024-09-15T23:24:03.117952+0000","last_undegraded":"2024-09-15T23:24:03.117952+0000","last_fullsized":"2024-09-15T23:24:03.117952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:52.581994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111899+0000","last_change":"2024-09-15T23:21:52.540485+0000","last_active":"2024-09-15T23:24:03.111899+0000","last_peered":"2024-09-15T23:24:03.111899+0000","last_clean":"2024-09-15T23:24:03.111899+0000","last_became_active":"2024-09-15T23:21:52.540178+0000","last_became_peered":"2024-09-15T23:21:52.540178+0000","last_unstale":"2024-09-15T23:24:03.111899+0000","last_undegraded":"2024-09-15T23:24:03.111899+0000","last_fullsized":"2024-09-15T23:24:03.111899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:40:22.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113883+0000","last_change":"2024-09-15T23:21:52.538845+0000","last_active":"2024-09-15T23:24:03.113883+0000","last_peered":"2024-09-15T23:24:03.113883+0000","last_clean":"2024-09-15T23:24:03.113883+0000","last_became_active":"2024-09-15T23:21:52.538511+0000","last_became_peered":"2024-09-15T23:21:52.538511+0000","last_unstale":"2024-09-15T23:24:03.113883+0000","last_undegraded":"2024-09-15T23:24:03.113883+0000","last_fullsized":"2024-09-15T23:24:03.113883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:00.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114158+0000","last_change":"2024-09-15T23:23:02.149590+0000","last_active":"2024-09-15T23:24:03.114158+0000","last_peered":"2024-09-15T23:24:03.114158+0000","last_clean":"2024-09-15T23:24:03.114158+0000","last_became_active":"2024-09-15T23:23:02.149184+0000","last_became_peered":"2024-09-15T23:23:02.149184+0000","last_unstale":"2024-09-15T23:24:03.114158+0000","last_undegraded":"2024-09-15T23:24:03.114158+0000","last_fullsized":"2024-09-15T23:24:03.114158+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:05:29.407175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117530+0000","last_change":"2024-09-15T23:21:52.530524+0000","last_active":"2024-09-15T23:24:03.117530+0000","last_peered":"2024-09-15T23:24:03.117530+0000","last_clean":"2024-09-15T23:24:03.117530+0000","last_became_active":"2024-09-15T23:21:52.530203+0000","last_became_peered":"2024-09-15T23:21:52.530203+0000","last_unstale":"2024-09-15T23:24:03.117530+0000","last_undegraded":"2024-09-15T23:24:03.117530+0000","last_fullsized":"2024-09-15T23:24:03.117530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:50.560211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110690+0000","last_change":"2024-09-15T23:21:52.537256+0000","last_active":"2024-09-15T23:24:03.110690+0000","last_peered":"2024-09-15T23:24:03.110690+0000","last_clean":"2024-09-15T23:24:03.110690+0000","last_became_active":"2024-09-15T23:21:52.536731+0000","last_became_peered":"2024-09-15T23:21:52.536731+0000","last_unstale":"2024-09-15T23:24:03.110690+0000","last_undegraded":"2024-09-15T23:24:03.110690+0000","last_fullsized":"2024-09-15T23:24:03.110690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:18.815398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194364+0000","last_change":"2024-09-15T23:21:52.537728+0000","last_active":"2024-09-15T23:24:03.194364+0000","last_peered":"2024-09-15T23:24:03.194364+0000","last_clean":"2024-09-15T23:24:03.194364+0000","last_became_active":"2024-09-15T23:21:52.537434+0000","last_became_peered":"2024-09-15T23:21:52.537434+0000","last_unstale":"2024-09-15T23:24:03.194364+0000","last_undegraded":"2024-09-15T23:24:03.194364+0000","last_fullsized":"2024-09-15T23:24:03.194364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:25.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194218+0000","last_change":"2024-09-15T23:21:52.534780+0000","last_active":"2024-09-15T23:24:03.194218+0000","last_peered":"2024-09-15T23:24:03.194218+0000","last_clean":"2024-09-15T23:24:03.194218+0000","last_became_active":"2024-09-15T23:21:52.534399+0000","last_became_peered":"2024-09-15T23:21:52.534399+0000","last_unstale":"2024-09-15T23:24:03.194218+0000","last_undegraded":"2024-09-15T23:24:03.194218+0000","last_fullsized":"2024-09-15T23:24:03.194218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:47.344129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116592+0000","last_change":"2024-09-15T23:21:52.533403+0000","last_active":"2024-09-15T23:24:03.116592+0000","last_peered":"2024-09-15T23:24:03.116592+0000","last_clean":"2024-09-15T23:24:03.116592+0000","last_became_active":"2024-09-15T23:21:52.533069+0000","last_became_peered":"2024-09-15T23:21:52.533069+0000","last_unstale":"2024-09-15T23:24:03.116592+0000","last_undegraded":"2024-09-15T23:24:03.116592+0000","last_fullsized":"2024-09-15T23:24:03.116592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:38.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":78576,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1129,"num_read_kb":651,"num_write":268,"num_write_kb":163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":63447,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":235728,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":246033,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579949,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":521776,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841585,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1163264,"data_stored":1178923,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005111,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":524287,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234044,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":520569,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462977,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1073152,"data_stored":1116395,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626502,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1003520,"data_stored":515695,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855435,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1112565,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018960,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1044480,"data_stored":591615,"data_compressed":39742,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247892,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":515403,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444122,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1019904,"data_stored":517253,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607647,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":587644,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1028096,"data_stored":517151,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":72671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":72546,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":73582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:24:31.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:24:31.952 INFO:tasks.cephfs.filesystem:scrub status for tag:eb4cea45-715e-4764-9a2e-7000c8d30f82 - {'path': '/', 'tag': 'eb4cea45-715e-4764-9a2e-7000c8d30f82', 'options': 'recursive,force'} 2024-09-15T23:24:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.480+0000 7f64f11e0640 1 -- 172.21.15.89:0/1122559728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64ec101de0 msgr2=0x7f64ec1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.480+0000 7f64f11e0640 1 --2- 172.21.15.89:0/1122559728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64ec101de0 0x7f64ec1021c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009900 tx=0x7f64dc02a410 comp rx=0 tx=0).stop 2024-09-15T23:24:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.480+0000 7f64f11e0640 1 -- 172.21.15.89:0/1122559728 shutdown_connections 2024-09-15T23:24:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.480+0000 7f64f11e0640 1 --2- 172.21.15.89:0/1122559728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ec106f10 0x7f64ec10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.480+0000 7f64f11e0640 1 --2- 172.21.15.89:0/1122559728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec102790 0x7f64ec1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.480+0000 7f64f11e0640 1 --2- 172.21.15.89:0/1122559728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64ec101de0 0x7f64ec1021c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.480+0000 7f64f11e0640 1 -- 172.21.15.89:0/1122559728 >> 172.21.15.89:0/1122559728 conn(0x7f64ec0fd8c0 msgr2=0x7f64ec0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.480+0000 7f64f11e0640 1 -- 172.21.15.89:0/1122559728 shutdown_connections 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.480+0000 7f64f11e0640 1 -- 172.21.15.89:0/1122559728 wait complete. 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64f11e0640 1 Processor -- start 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64f11e0640 1 -- start start 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64f11e0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ec101de0 0x7f64ec1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64f11e0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64ec102790 0x7f64ec198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64f11e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec106f10 0x7f64ec19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64f11e0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec111f90 con 0x7f64ec102790 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64f11e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec111e10 con 0x7f64ec106f10 2024-09-15T23:24:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64f11e0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec112110 con 0x7f64ec101de0 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64ea59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64ec102790 0x7f64ec198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64ea59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64ec102790 0x7f64ec198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56128/0 (socket says 172.21.15.89:56128) 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.481+0000 7f64ea59c640 1 -- 172.21.15.89:0/1694458518 learned_addr learned my addr 172.21.15.89:0/1694458518 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64eb59e640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec106f10 0x7f64ec19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64ead9d640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ec101de0 0x7f64ec1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64ea59c640 1 -- 172.21.15.89:0/1694458518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ec101de0 msgr2=0x7f64ec1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64ea59c640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ec101de0 0x7f64ec1984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64ea59c640 1 -- 172.21.15.89:0/1694458518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec106f10 msgr2=0x7f64ec19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64ea59c640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec106f10 0x7f64ec19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64ea59c640 1 -- 172.21.15.89:0/1694458518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ec19d480 con 0x7f64ec102790 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64eb59e640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec106f10 0x7f64ec19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64ea59c640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64ec102790 0x7f64ec198a10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f64d400b4a0 tx=0x7f64d400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:32.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64e3fff640 1 -- 172.21.15.89:0/1694458518 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d4014070 con 0x7f64ec102790 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64f11e0640 1 -- 172.21.15.89:0/1694458518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ec19d770 con 0x7f64ec102790 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64e3fff640 1 -- 172.21.15.89:0/1694458518 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64d4010af0 con 0x7f64ec102790 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64e3fff640 1 -- 172.21.15.89:0/1694458518 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d4017d10 con 0x7f64ec102790 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.482+0000 7f64f11e0640 1 -- 172.21.15.89:0/1694458518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ec1a5050 con 0x7f64ec102790 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.483+0000 7f64f11e0640 1 -- 172.21.15.89:0/1694458518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64ec1021c0 con 0x7f64ec102790 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.484+0000 7f64e3fff640 1 -- 172.21.15.89:0/1694458518 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f64d4017070 con 0x7f64ec102790 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.485+0000 7f64e3fff640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f64d80a32d0 0x7f64d80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.485+0000 7f64ead9d640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f64d80a32d0 0x7f64d80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.485+0000 7f64e3fff640 1 -- 172.21.15.89:0/1694458518 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f64d40cf200 con 0x7f64ec102790 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.486+0000 7f64ead9d640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f64d80a32d0 0x7f64d80a5790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0096c0 tx=0x7f64dc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:32.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.489+0000 7f64e3fff640 1 -- 172.21.15.89:0/1694458518 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f64d409bae0 con 0x7f64ec102790 2024-09-15T23:24:32.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.659+0000 7f64f11e0640 1 -- 172.21.15.89:0/1694458518 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f64ec062630 con 0x7f64d80a32d0 2024-09-15T23:24:32.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.667+0000 7f64e3fff640 1 -- 172.21.15.89:0/1694458518 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340822 (secure 0 0 0) 0x7f64ec062630 con 0x7f64d80a32d0 2024-09-15T23:24:32.671 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:32.673 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-15T23:24:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 -- 172.21.15.89:0/1694458518 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f64d80a32d0 msgr2=0x7f64d80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f64d80a32d0 0x7f64d80a5790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0096c0 tx=0x7f64dc000bc0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 -- 172.21.15.89:0/1694458518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64ec102790 msgr2=0x7f64ec198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64ec102790 0x7f64ec198a10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f64d400b4a0 tx=0x7f64d400b960 comp rx=0 tx=0).stop 2024-09-15T23:24:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 -- 172.21.15.89:0/1694458518 shutdown_connections 2024-09-15T23:24:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f64d80a32d0 0x7f64d80a5790 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64ec106f10 0x7f64ec19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64ec102790 0x7f64ec198a10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 --2- 172.21.15.89:0/1694458518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64ec101de0 0x7f64ec1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 -- 172.21.15.89:0/1694458518 >> 172.21.15.89:0/1694458518 conn(0x7f64ec0fd8c0 msgr2=0x7f64ec0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:32.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 -- 172.21.15.89:0/1694458518 shutdown_connections 2024-09-15T23:24:32.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:32.670+0000 7f64f11e0640 1 -- 172.21.15.89:0/1694458518 wait complete. 2024-09-15T23:24:33.420 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-15T23:24:31.023864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669178,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1265,"num_read_kb":760,"num_write":451,"num_write_kb":3103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1244183,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369884,"kb_used_data":12284,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571940,"statfs":{"total":1151940427776,"available":1151561666560,"internally_reserved":0,"allocated":12578816,"data_stored":8219276,"data_compressed":471759,"data_compressed_allocated":638976,"data_compressed_original":2813952,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9295,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":62,"num_write":14,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003175"},"pg_stats":[{"pgid":"2.1f","version":"86'13","reported_seq":163,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:26.289472+0000","last_change":"2024-09-15T23:21:50.513610+0000","last_active":"2024-09-15T23:24:26.289472+0000","last_peered":"2024-09-15T23:24:26.289472+0000","last_clean":"2024-09-15T23:24:26.289472+0000","last_became_active":"2024-09-15T23:21:50.513266+0000","last_became_peered":"2024-09-15T23:21:50.513266+0000","last_unstale":"2024-09-15T23:24:26.289472+0000","last_undegraded":"2024-09-15T23:24:26.289472+0000","last_fullsized":"2024-09-15T23:24:26.289472+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:16:48.692801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112578+0000","last_change":"2024-09-15T23:21:52.540248+0000","last_active":"2024-09-15T23:24:03.112578+0000","last_peered":"2024-09-15T23:24:03.112578+0000","last_clean":"2024-09-15T23:24:03.112578+0000","last_became_active":"2024-09-15T23:21:52.539945+0000","last_became_peered":"2024-09-15T23:21:52.539945+0000","last_unstale":"2024-09-15T23:24:03.112578+0000","last_undegraded":"2024-09-15T23:24:03.112578+0000","last_fullsized":"2024-09-15T23:24:03.112578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:33.402153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112456+0000","last_change":"2024-09-15T23:21:59.567246+0000","last_active":"2024-09-15T23:24:03.112456+0000","last_peered":"2024-09-15T23:24:03.112456+0000","last_clean":"2024-09-15T23:24:03.112456+0000","last_became_active":"2024-09-15T23:21:59.566832+0000","last_became_peered":"2024-09-15T23:21:59.566832+0000","last_unstale":"2024-09-15T23:24:03.112456+0000","last_undegraded":"2024-09-15T23:24:03.112456+0000","last_fullsized":"2024-09-15T23:24:03.112456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:19.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112747+0000","last_change":"2024-09-15T23:21:50.561767+0000","last_active":"2024-09-15T23:24:03.112747+0000","last_peered":"2024-09-15T23:24:03.112747+0000","last_clean":"2024-09-15T23:24:03.112747+0000","last_became_active":"2024-09-15T23:21:50.561385+0000","last_became_peered":"2024-09-15T23:21:50.561385+0000","last_unstale":"2024-09-15T23:24:03.112747+0000","last_undegraded":"2024-09-15T23:24:03.112747+0000","last_fullsized":"2024-09-15T23:24:03.112747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:11.897884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194008+0000","last_change":"2024-09-15T23:21:52.534874+0000","last_active":"2024-09-15T23:24:03.194008+0000","last_peered":"2024-09-15T23:24:03.194008+0000","last_clean":"2024-09-15T23:24:03.194008+0000","last_became_active":"2024-09-15T23:21:52.534334+0000","last_became_peered":"2024-09-15T23:21:52.534334+0000","last_unstale":"2024-09-15T23:24:03.194008+0000","last_undegraded":"2024-09-15T23:24:03.194008+0000","last_fullsized":"2024-09-15T23:24:03.194008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:15.972648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110442+0000","last_change":"2024-09-15T23:21:59.559907+0000","last_active":"2024-09-15T23:24:03.110442+0000","last_peered":"2024-09-15T23:24:03.110442+0000","last_clean":"2024-09-15T23:24:03.110442+0000","last_became_active":"2024-09-15T23:21:59.559440+0000","last_became_peered":"2024-09-15T23:21:59.559440+0000","last_unstale":"2024-09-15T23:24:03.110442+0000","last_undegraded":"2024-09-15T23:24:03.110442+0000","last_fullsized":"2024-09-15T23:24:03.110442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:00.755112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117658+0000","last_change":"2024-09-15T23:21:50.515424+0000","last_active":"2024-09-15T23:24:03.117658+0000","last_peered":"2024-09-15T23:24:03.117658+0000","last_clean":"2024-09-15T23:24:03.117658+0000","last_became_active":"2024-09-15T23:21:50.514966+0000","last_became_peered":"2024-09-15T23:21:50.514966+0000","last_unstale":"2024-09-15T23:24:03.117658+0000","last_undegraded":"2024-09-15T23:24:03.117658+0000","last_fullsized":"2024-09-15T23:24:03.117658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:57.586575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117412+0000","last_change":"2024-09-15T23:21:52.535189+0000","last_active":"2024-09-15T23:24:03.117412+0000","last_peered":"2024-09-15T23:24:03.117412+0000","last_clean":"2024-09-15T23:24:03.117412+0000","last_became_active":"2024-09-15T23:21:52.534807+0000","last_became_peered":"2024-09-15T23:21:52.534807+0000","last_unstale":"2024-09-15T23:24:03.117412+0000","last_undegraded":"2024-09-15T23:24:03.117412+0000","last_fullsized":"2024-09-15T23:24:03.117412+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:30.654382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110835+0000","last_change":"2024-09-15T23:21:59.564675+0000","last_active":"2024-09-15T23:24:03.110835+0000","last_peered":"2024-09-15T23:24:03.110835+0000","last_clean":"2024-09-15T23:24:03.110835+0000","last_became_active":"2024-09-15T23:21:59.564271+0000","last_became_peered":"2024-09-15T23:21:59.564271+0000","last_unstale":"2024-09-15T23:24:03.110835+0000","last_undegraded":"2024-09-15T23:24:03.110835+0000","last_fullsized":"2024-09-15T23:24:03.110835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:01.055369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117219+0000","last_change":"2024-09-15T23:21:50.507790+0000","last_active":"2024-09-15T23:24:03.117219+0000","last_peered":"2024-09-15T23:24:03.117219+0000","last_clean":"2024-09-15T23:24:03.117219+0000","last_became_active":"2024-09-15T23:21:50.507398+0000","last_became_peered":"2024-09-15T23:21:50.507398+0000","last_unstale":"2024-09-15T23:24:03.117219+0000","last_undegraded":"2024-09-15T23:24:03.117219+0000","last_fullsized":"2024-09-15T23:24:03.117219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:58:00.001279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116935+0000","last_change":"2024-09-15T23:21:52.537179+0000","last_active":"2024-09-15T23:24:03.116935+0000","last_peered":"2024-09-15T23:24:03.116935+0000","last_clean":"2024-09-15T23:24:03.116935+0000","last_became_active":"2024-09-15T23:21:52.536841+0000","last_became_peered":"2024-09-15T23:21:52.536841+0000","last_unstale":"2024-09-15T23:24:03.116935+0000","last_undegraded":"2024-09-15T23:24:03.116935+0000","last_fullsized":"2024-09-15T23:24:03.116935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:47.455700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110534+0000","last_change":"2024-09-15T23:21:59.564734+0000","last_active":"2024-09-15T23:24:03.110534+0000","last_peered":"2024-09-15T23:24:03.110534+0000","last_clean":"2024-09-15T23:24:03.110534+0000","last_became_active":"2024-09-15T23:21:59.564339+0000","last_became_peered":"2024-09-15T23:21:59.564339+0000","last_unstale":"2024-09-15T23:24:03.110534+0000","last_undegraded":"2024-09-15T23:24:03.110534+0000","last_fullsized":"2024-09-15T23:24:03.110534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:44.498480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113970+0000","last_change":"2024-09-15T23:21:52.538025+0000","last_active":"2024-09-15T23:24:03.113970+0000","last_peered":"2024-09-15T23:24:03.113970+0000","last_clean":"2024-09-15T23:24:03.113970+0000","last_became_active":"2024-09-15T23:21:52.537736+0000","last_became_peered":"2024-09-15T23:21:52.537736+0000","last_unstale":"2024-09-15T23:24:03.113970+0000","last_undegraded":"2024-09-15T23:24:03.113970+0000","last_fullsized":"2024-09-15T23:24:03.113970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:31.916888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.127260+0000","last_change":"2024-09-15T23:24:03.127260+0000","last_active":"2024-09-15T23:24:03.127260+0000","last_peered":"2024-09-15T23:24:03.127260+0000","last_clean":"2024-09-15T23:24:03.127260+0000","last_became_active":"2024-09-15T23:24:03.119384+0000","last_became_peered":"2024-09-15T23:24:03.119384+0000","last_unstale":"2024-09-15T23:24:03.127260+0000","last_undegraded":"2024-09-15T23:24:03.127260+0000","last_fullsized":"2024-09-15T23:24:03.127260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:35.325048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110625+0000","last_change":"2024-09-15T23:21:59.681971+0000","last_active":"2024-09-15T23:24:03.110625+0000","last_peered":"2024-09-15T23:24:03.110625+0000","last_clean":"2024-09-15T23:24:03.110625+0000","last_became_active":"2024-09-15T23:21:59.681432+0000","last_became_peered":"2024-09-15T23:21:59.681432+0000","last_unstale":"2024-09-15T23:24:03.110625+0000","last_undegraded":"2024-09-15T23:24:03.110625+0000","last_fullsized":"2024-09-15T23:24:03.110625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:26.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112179+0000","last_change":"2024-09-15T23:21:52.541281+0000","last_active":"2024-09-15T23:24:03.112179+0000","last_peered":"2024-09-15T23:24:03.112179+0000","last_clean":"2024-09-15T23:24:03.112179+0000","last_became_active":"2024-09-15T23:21:52.540772+0000","last_became_peered":"2024-09-15T23:21:52.540772+0000","last_unstale":"2024-09-15T23:24:03.112179+0000","last_undegraded":"2024-09-15T23:24:03.112179+0000","last_fullsized":"2024-09-15T23:24:03.112179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:23:44.444904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112700+0000","last_change":"2024-09-15T23:21:50.508844+0000","last_active":"2024-09-15T23:24:03.112700+0000","last_peered":"2024-09-15T23:24:03.112700+0000","last_clean":"2024-09-15T23:24:03.112700+0000","last_became_active":"2024-09-15T23:21:50.508364+0000","last_became_peered":"2024-09-15T23:21:50.508364+0000","last_unstale":"2024-09-15T23:24:03.112700+0000","last_undegraded":"2024-09-15T23:24:03.112700+0000","last_fullsized":"2024-09-15T23:24:03.112700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:45.129711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110720+0000","last_change":"2024-09-15T23:21:59.562114+0000","last_active":"2024-09-15T23:24:03.110720+0000","last_peered":"2024-09-15T23:24:03.110720+0000","last_clean":"2024-09-15T23:24:03.110720+0000","last_became_active":"2024-09-15T23:21:59.561652+0000","last_became_peered":"2024-09-15T23:21:59.561652+0000","last_unstale":"2024-09-15T23:24:03.110720+0000","last_undegraded":"2024-09-15T23:24:03.110720+0000","last_fullsized":"2024-09-15T23:24:03.110720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:51.781886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.531678+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.531327+0000","last_became_peered":"2024-09-15T23:21:52.531327+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:07.152626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112193+0000","last_change":"2024-09-15T23:21:50.513197+0000","last_active":"2024-09-15T23:24:03.112193+0000","last_peered":"2024-09-15T23:24:03.112193+0000","last_clean":"2024-09-15T23:24:03.112193+0000","last_became_active":"2024-09-15T23:21:50.512794+0000","last_became_peered":"2024-09-15T23:21:50.512794+0000","last_unstale":"2024-09-15T23:24:03.112193+0000","last_undegraded":"2024-09-15T23:24:03.112193+0000","last_fullsized":"2024-09-15T23:24:03.112193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:05.457467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194628+0000","last_change":"2024-09-15T23:21:59.565669+0000","last_active":"2024-09-15T23:24:03.194628+0000","last_peered":"2024-09-15T23:24:03.194628+0000","last_clean":"2024-09-15T23:24:03.194628+0000","last_became_active":"2024-09-15T23:21:59.565290+0000","last_became_peered":"2024-09-15T23:21:59.565290+0000","last_unstale":"2024-09-15T23:24:03.194628+0000","last_undegraded":"2024-09-15T23:24:03.194628+0000","last_fullsized":"2024-09-15T23:24:03.194628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:04.861560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116394+0000","last_change":"2024-09-15T23:21:52.529643+0000","last_active":"2024-09-15T23:24:03.116394+0000","last_peered":"2024-09-15T23:24:03.116394+0000","last_clean":"2024-09-15T23:24:03.116394+0000","last_became_active":"2024-09-15T23:21:52.529304+0000","last_became_peered":"2024-09-15T23:21:52.529304+0000","last_unstale":"2024-09-15T23:24:03.116394+0000","last_undegraded":"2024-09-15T23:24:03.116394+0000","last_fullsized":"2024-09-15T23:24:03.116394+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:18:37.758403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117052+0000","last_change":"2024-09-15T23:21:50.562114+0000","last_active":"2024-09-15T23:24:03.117052+0000","last_peered":"2024-09-15T23:24:03.117052+0000","last_clean":"2024-09-15T23:24:03.117052+0000","last_became_active":"2024-09-15T23:21:50.561772+0000","last_became_peered":"2024-09-15T23:21:50.561772+0000","last_unstale":"2024-09-15T23:24:03.117052+0000","last_undegraded":"2024-09-15T23:24:03.117052+0000","last_fullsized":"2024-09-15T23:24:03.117052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:14.806646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110672+0000","last_change":"2024-09-15T23:21:59.681933+0000","last_active":"2024-09-15T23:24:03.110672+0000","last_peered":"2024-09-15T23:24:03.110672+0000","last_clean":"2024-09-15T23:24:03.110672+0000","last_became_active":"2024-09-15T23:21:59.681528+0000","last_became_peered":"2024-09-15T23:21:59.681528+0000","last_unstale":"2024-09-15T23:24:03.110672+0000","last_undegraded":"2024-09-15T23:24:03.110672+0000","last_fullsized":"2024-09-15T23:24:03.110672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:46.199540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194326+0000","last_change":"2024-09-15T23:21:52.529548+0000","last_active":"2024-09-15T23:24:03.194326+0000","last_peered":"2024-09-15T23:24:03.194326+0000","last_clean":"2024-09-15T23:24:03.194326+0000","last_became_active":"2024-09-15T23:21:52.529253+0000","last_became_peered":"2024-09-15T23:21:52.529253+0000","last_unstale":"2024-09-15T23:24:03.194326+0000","last_undegraded":"2024-09-15T23:24:03.194326+0000","last_fullsized":"2024-09-15T23:24:03.194326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:22.813395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111022+0000","last_change":"2024-09-15T23:21:50.767759+0000","last_active":"2024-09-15T23:24:03.111022+0000","last_peered":"2024-09-15T23:24:03.111022+0000","last_clean":"2024-09-15T23:24:03.111022+0000","last_became_active":"2024-09-15T23:21:50.767388+0000","last_became_peered":"2024-09-15T23:21:50.767388+0000","last_unstale":"2024-09-15T23:24:03.111022+0000","last_undegraded":"2024-09-15T23:24:03.111022+0000","last_fullsized":"2024-09-15T23:24:03.111022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:36.713769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112512+0000","last_change":"2024-09-15T23:21:59.557951+0000","last_active":"2024-09-15T23:24:03.112512+0000","last_peered":"2024-09-15T23:24:03.112512+0000","last_clean":"2024-09-15T23:24:03.112512+0000","last_became_active":"2024-09-15T23:21:59.557490+0000","last_became_peered":"2024-09-15T23:21:59.557490+0000","last_unstale":"2024-09-15T23:24:03.112512+0000","last_undegraded":"2024-09-15T23:24:03.112512+0000","last_fullsized":"2024-09-15T23:24:03.112512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:17.147389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112859+0000","last_change":"2024-09-15T23:21:52.531638+0000","last_active":"2024-09-15T23:24:03.112859+0000","last_peered":"2024-09-15T23:24:03.112859+0000","last_clean":"2024-09-15T23:24:03.112859+0000","last_became_active":"2024-09-15T23:21:52.531301+0000","last_became_peered":"2024-09-15T23:21:52.531301+0000","last_unstale":"2024-09-15T23:24:03.112859+0000","last_undegraded":"2024-09-15T23:24:03.112859+0000","last_fullsized":"2024-09-15T23:24:03.112859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.436286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'18","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:18.545294+0000","last_change":"2024-09-15T23:21:50.508470+0000","last_active":"2024-09-15T23:24:18.545294+0000","last_peered":"2024-09-15T23:24:18.545294+0000","last_clean":"2024-09-15T23:24:18.545294+0000","last_became_active":"2024-09-15T23:21:50.508060+0000","last_became_peered":"2024-09-15T23:21:50.508060+0000","last_unstale":"2024-09-15T23:24:18.545294+0000","last_undegraded":"2024-09-15T23:24:18.545294+0000","last_fullsized":"2024-09-15T23:24:18.545294+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:59.542252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111992+0000","last_change":"2024-09-15T23:21:59.682019+0000","last_active":"2024-09-15T23:24:03.111992+0000","last_peered":"2024-09-15T23:24:03.111992+0000","last_clean":"2024-09-15T23:24:03.111992+0000","last_became_active":"2024-09-15T23:21:59.681467+0000","last_became_peered":"2024-09-15T23:21:59.681467+0000","last_unstale":"2024-09-15T23:24:03.111992+0000","last_undegraded":"2024-09-15T23:24:03.111992+0000","last_fullsized":"2024-09-15T23:24:03.111992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:27.650382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109766+0000","last_change":"2024-09-15T23:21:52.536345+0000","last_active":"2024-09-15T23:24:03.109766+0000","last_peered":"2024-09-15T23:24:03.109766+0000","last_clean":"2024-09-15T23:24:03.109766+0000","last_became_active":"2024-09-15T23:21:52.536027+0000","last_became_peered":"2024-09-15T23:21:52.536027+0000","last_unstale":"2024-09-15T23:24:03.109766+0000","last_undegraded":"2024-09-15T23:24:03.109766+0000","last_fullsized":"2024-09-15T23:24:03.109766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:16.435912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117716+0000","last_change":"2024-09-15T23:21:50.562945+0000","last_active":"2024-09-15T23:24:03.117716+0000","last_peered":"2024-09-15T23:24:03.117716+0000","last_clean":"2024-09-15T23:24:03.117716+0000","last_became_active":"2024-09-15T23:21:50.561632+0000","last_became_peered":"2024-09-15T23:21:50.561632+0000","last_unstale":"2024-09-15T23:24:03.117716+0000","last_undegraded":"2024-09-15T23:24:03.117716+0000","last_fullsized":"2024-09-15T23:24:03.117716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:44.545489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115268+0000","last_change":"2024-09-15T23:21:59.564793+0000","last_active":"2024-09-15T23:24:03.115268+0000","last_peered":"2024-09-15T23:24:03.115268+0000","last_clean":"2024-09-15T23:24:03.115268+0000","last_became_active":"2024-09-15T23:21:59.564140+0000","last_became_peered":"2024-09-15T23:21:59.564140+0000","last_unstale":"2024-09-15T23:24:03.115268+0000","last_undegraded":"2024-09-15T23:24:03.115268+0000","last_fullsized":"2024-09-15T23:24:03.115268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:41.464962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113451+0000","last_change":"2024-09-15T23:21:52.533521+0000","last_active":"2024-09-15T23:24:03.113451+0000","last_peered":"2024-09-15T23:24:03.113451+0000","last_clean":"2024-09-15T23:24:03.113451+0000","last_became_active":"2024-09-15T23:21:52.532195+0000","last_became_peered":"2024-09-15T23:21:52.532195+0000","last_unstale":"2024-09-15T23:24:03.113451+0000","last_undegraded":"2024-09-15T23:24:03.113451+0000","last_fullsized":"2024-09-15T23:24:03.113451+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:14.229018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194243+0000","last_change":"2024-09-15T23:21:50.562893+0000","last_active":"2024-09-15T23:24:03.194243+0000","last_peered":"2024-09-15T23:24:03.194243+0000","last_clean":"2024-09-15T23:24:03.194243+0000","last_became_active":"2024-09-15T23:21:50.562516+0000","last_became_peered":"2024-09-15T23:21:50.562516+0000","last_unstale":"2024-09-15T23:24:03.194243+0000","last_undegraded":"2024-09-15T23:24:03.194243+0000","last_fullsized":"2024-09-15T23:24:03.194243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:47:47.442458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117856+0000","last_change":"2024-09-15T23:21:59.682091+0000","last_active":"2024-09-15T23:24:03.117856+0000","last_peered":"2024-09-15T23:24:03.117856+0000","last_clean":"2024-09-15T23:24:03.117856+0000","last_became_active":"2024-09-15T23:21:59.681574+0000","last_became_peered":"2024-09-15T23:21:59.681574+0000","last_unstale":"2024-09-15T23:24:03.117856+0000","last_undegraded":"2024-09-15T23:24:03.117856+0000","last_fullsized":"2024-09-15T23:24:03.117856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:04.991608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114426+0000","last_change":"2024-09-15T23:21:52.533811+0000","last_active":"2024-09-15T23:24:03.114426+0000","last_peered":"2024-09-15T23:24:03.114426+0000","last_clean":"2024-09-15T23:24:03.114426+0000","last_became_active":"2024-09-15T23:21:52.533487+0000","last_became_peered":"2024-09-15T23:21:52.533487+0000","last_unstale":"2024-09-15T23:24:03.114426+0000","last_undegraded":"2024-09-15T23:24:03.114426+0000","last_fullsized":"2024-09-15T23:24:03.114426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:44.193212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":154,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:25.288686+0000","last_change":"2024-09-15T23:21:50.510883+0000","last_active":"2024-09-15T23:24:25.288686+0000","last_peered":"2024-09-15T23:24:25.288686+0000","last_clean":"2024-09-15T23:24:25.288686+0000","last_became_active":"2024-09-15T23:21:50.510478+0000","last_became_peered":"2024-09-15T23:21:50.510478+0000","last_unstale":"2024-09-15T23:24:25.288686+0000","last_undegraded":"2024-09-15T23:24:25.288686+0000","last_fullsized":"2024-09-15T23:24:25.288686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:35.454885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109705+0000","last_change":"2024-09-15T23:21:59.561816+0000","last_active":"2024-09-15T23:24:03.109705+0000","last_peered":"2024-09-15T23:24:03.109705+0000","last_clean":"2024-09-15T23:24:03.109705+0000","last_became_active":"2024-09-15T23:21:59.561434+0000","last_became_peered":"2024-09-15T23:21:59.561434+0000","last_unstale":"2024-09-15T23:24:03.109705+0000","last_undegraded":"2024-09-15T23:24:03.109705+0000","last_fullsized":"2024-09-15T23:24:03.109705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:59.945802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117955+0000","last_change":"2024-09-15T23:23:02.277204+0000","last_active":"2024-09-15T23:24:03.117955+0000","last_peered":"2024-09-15T23:24:03.117955+0000","last_clean":"2024-09-15T23:24:03.117955+0000","last_became_active":"2024-09-15T23:23:02.276770+0000","last_became_peered":"2024-09-15T23:23:02.276770+0000","last_unstale":"2024-09-15T23:24:03.117955+0000","last_undegraded":"2024-09-15T23:24:03.117955+0000","last_fullsized":"2024-09-15T23:24:03.117955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:11.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":155,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:24.151376+0000","last_change":"2024-09-15T23:21:50.561684+0000","last_active":"2024-09-15T23:24:24.151376+0000","last_peered":"2024-09-15T23:24:24.151376+0000","last_clean":"2024-09-15T23:24:24.151376+0000","last_became_active":"2024-09-15T23:21:50.561335+0000","last_became_peered":"2024-09-15T23:21:50.561335+0000","last_unstale":"2024-09-15T23:24:24.151376+0000","last_undegraded":"2024-09-15T23:24:24.151376+0000","last_fullsized":"2024-09-15T23:24:24.151376+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:23.897250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":109,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109863+0000","last_change":"2024-09-15T23:21:59.681858+0000","last_active":"2024-09-15T23:24:03.109863+0000","last_peered":"2024-09-15T23:24:03.109863+0000","last_clean":"2024-09-15T23:24:03.109863+0000","last_became_active":"2024-09-15T23:21:59.681441+0000","last_became_peered":"2024-09-15T23:21:59.681441+0000","last_unstale":"2024-09-15T23:24:03.109863+0000","last_undegraded":"2024-09-15T23:24:03.109863+0000","last_fullsized":"2024-09-15T23:24:03.109863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:28.513719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109320+0000","last_change":"2024-09-15T23:21:52.536141+0000","last_active":"2024-09-15T23:24:03.109320+0000","last_peered":"2024-09-15T23:24:03.109320+0000","last_clean":"2024-09-15T23:24:03.109320+0000","last_became_active":"2024-09-15T23:21:52.535832+0000","last_became_peered":"2024-09-15T23:21:52.535832+0000","last_unstale":"2024-09-15T23:24:03.109320+0000","last_undegraded":"2024-09-15T23:24:03.109320+0000","last_fullsized":"2024-09-15T23:24:03.109320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:21:44.341412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.123671+0000","last_change":"2024-09-15T23:24:03.123671+0000","last_active":"2024-09-15T23:24:03.123671+0000","last_peered":"2024-09-15T23:24:03.123671+0000","last_clean":"2024-09-15T23:24:03.123671+0000","last_became_active":"2024-09-15T23:24:03.119585+0000","last_became_peered":"2024-09-15T23:24:03.119585+0000","last_unstale":"2024-09-15T23:24:03.123671+0000","last_undegraded":"2024-09-15T23:24:03.123671+0000","last_fullsized":"2024-09-15T23:24:03.123671+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:30.734156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118359+0000","last_change":"2024-09-15T23:21:59.682118+0000","last_active":"2024-09-15T23:24:03.118359+0000","last_peered":"2024-09-15T23:24:03.118359+0000","last_clean":"2024-09-15T23:24:03.118359+0000","last_became_active":"2024-09-15T23:21:59.681500+0000","last_became_peered":"2024-09-15T23:21:59.681500+0000","last_unstale":"2024-09-15T23:24:03.118359+0000","last_undegraded":"2024-09-15T23:24:03.118359+0000","last_fullsized":"2024-09-15T23:24:03.118359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:58.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112298+0000","last_change":"2024-09-15T23:21:52.540458+0000","last_active":"2024-09-15T23:24:03.112298+0000","last_peered":"2024-09-15T23:24:03.112298+0000","last_clean":"2024-09-15T23:24:03.112298+0000","last_became_active":"2024-09-15T23:21:52.540112+0000","last_became_peered":"2024-09-15T23:21:52.540112+0000","last_unstale":"2024-09-15T23:24:03.112298+0000","last_undegraded":"2024-09-15T23:24:03.112298+0000","last_fullsized":"2024-09-15T23:24:03.112298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:15.240785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111097+0000","last_change":"2024-09-15T23:21:50.513124+0000","last_active":"2024-09-15T23:24:03.111097+0000","last_peered":"2024-09-15T23:24:03.111097+0000","last_clean":"2024-09-15T23:24:03.111097+0000","last_became_active":"2024-09-15T23:21:50.512705+0000","last_became_peered":"2024-09-15T23:21:50.512705+0000","last_unstale":"2024-09-15T23:24:03.111097+0000","last_undegraded":"2024-09-15T23:24:03.111097+0000","last_fullsized":"2024-09-15T23:24:03.111097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.417937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":354,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113701+0000","last_change":"2024-09-15T23:20:40.656652+0000","last_active":"2024-09-15T23:24:03.113701+0000","last_peered":"2024-09-15T23:24:03.113701+0000","last_clean":"2024-09-15T23:24:03.113701+0000","last_became_active":"2024-09-15T23:20:40.656266+0000","last_became_peered":"2024-09-15T23:20:40.656266+0000","last_unstale":"2024-09-15T23:24:03.113701+0000","last_undegraded":"2024-09-15T23:24:03.113701+0000","last_fullsized":"2024-09-15T23:24:03.113701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_clean_scrub_stamp":"2024-09-15T23:17:15.186850+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:42.317006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113728+0000","last_change":"2024-09-15T23:21:59.563777+0000","last_active":"2024-09-15T23:24:03.113728+0000","last_peered":"2024-09-15T23:24:03.113728+0000","last_clean":"2024-09-15T23:24:03.113728+0000","last_became_active":"2024-09-15T23:21:59.563320+0000","last_became_peered":"2024-09-15T23:21:59.563320+0000","last_unstale":"2024-09-15T23:24:03.113728+0000","last_undegraded":"2024-09-15T23:24:03.113728+0000","last_fullsized":"2024-09-15T23:24:03.113728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:25:07.791923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114145+0000","last_change":"2024-09-15T23:21:52.539842+0000","last_active":"2024-09-15T23:24:03.114145+0000","last_peered":"2024-09-15T23:24:03.114145+0000","last_clean":"2024-09-15T23:24:03.114145+0000","last_became_active":"2024-09-15T23:21:52.538683+0000","last_became_peered":"2024-09-15T23:21:52.538683+0000","last_unstale":"2024-09-15T23:24:03.114145+0000","last_undegraded":"2024-09-15T23:24:03.114145+0000","last_fullsized":"2024-09-15T23:24:03.114145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:43.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":159,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:28.293643+0000","last_change":"2024-09-15T23:21:50.767601+0000","last_active":"2024-09-15T23:24:28.293643+0000","last_peered":"2024-09-15T23:24:28.293643+0000","last_clean":"2024-09-15T23:24:28.293643+0000","last_became_active":"2024-09-15T23:21:50.767149+0000","last_became_peered":"2024-09-15T23:21:50.767149+0000","last_unstale":"2024-09-15T23:24:28.293643+0000","last_undegraded":"2024-09-15T23:24:28.293643+0000","last_fullsized":"2024-09-15T23:24:28.293643+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:23.636551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117168+0000","last_change":"2024-09-15T23:21:59.563069+0000","last_active":"2024-09-15T23:24:03.117168+0000","last_peered":"2024-09-15T23:24:03.117168+0000","last_clean":"2024-09-15T23:24:03.117168+0000","last_became_active":"2024-09-15T23:21:59.562658+0000","last_became_peered":"2024-09-15T23:21:59.562658+0000","last_unstale":"2024-09-15T23:24:03.117168+0000","last_undegraded":"2024-09-15T23:24:03.117168+0000","last_fullsized":"2024-09-15T23:24:03.117168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:23.067198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113535+0000","last_change":"2024-09-15T23:21:52.535289+0000","last_active":"2024-09-15T23:24:03.113535+0000","last_peered":"2024-09-15T23:24:03.113535+0000","last_clean":"2024-09-15T23:24:03.113535+0000","last_became_active":"2024-09-15T23:21:52.534885+0000","last_became_peered":"2024-09-15T23:21:52.534885+0000","last_unstale":"2024-09-15T23:24:03.113535+0000","last_undegraded":"2024-09-15T23:24:03.113535+0000","last_fullsized":"2024-09-15T23:24:03.113535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:42.293786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118612+0000","last_change":"2024-09-15T23:21:50.513246+0000","last_active":"2024-09-15T23:24:03.118612+0000","last_peered":"2024-09-15T23:24:03.118612+0000","last_clean":"2024-09-15T23:24:03.118612+0000","last_became_active":"2024-09-15T23:21:50.512740+0000","last_became_peered":"2024-09-15T23:21:50.512740+0000","last_unstale":"2024-09-15T23:24:03.118612+0000","last_undegraded":"2024-09-15T23:24:03.118612+0000","last_fullsized":"2024-09-15T23:24:03.118612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:59.201970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193756+0000","last_change":"2024-09-15T23:21:59.568565+0000","last_active":"2024-09-15T23:24:03.193756+0000","last_peered":"2024-09-15T23:24:03.193756+0000","last_clean":"2024-09-15T23:24:03.193756+0000","last_became_active":"2024-09-15T23:21:59.567789+0000","last_became_peered":"2024-09-15T23:21:59.567789+0000","last_unstale":"2024-09-15T23:24:03.193756+0000","last_undegraded":"2024-09-15T23:24:03.193756+0000","last_fullsized":"2024-09-15T23:24:03.193756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:02.106971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114106+0000","last_change":"2024-09-15T23:21:52.533727+0000","last_active":"2024-09-15T23:24:03.114106+0000","last_peered":"2024-09-15T23:24:03.114106+0000","last_clean":"2024-09-15T23:24:03.114106+0000","last_became_active":"2024-09-15T23:21:52.532350+0000","last_became_peered":"2024-09-15T23:21:52.532350+0000","last_unstale":"2024-09-15T23:24:03.114106+0000","last_undegraded":"2024-09-15T23:24:03.114106+0000","last_fullsized":"2024-09-15T23:24:03.114106+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:03.624837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.124878+0000","last_change":"2024-09-15T23:24:03.124430+0000","last_active":"2024-09-15T23:24:03.124878+0000","last_peered":"2024-09-15T23:24:03.124878+0000","last_clean":"2024-09-15T23:24:03.124878+0000","last_became_active":"2024-09-15T23:24:03.119042+0000","last_became_peered":"2024-09-15T23:24:03.119042+0000","last_unstale":"2024-09-15T23:24:03.124878+0000","last_undegraded":"2024-09-15T23:24:03.124878+0000","last_fullsized":"2024-09-15T23:24:03.124878+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:52.065543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117798+0000","last_change":"2024-09-15T23:21:59.566429+0000","last_active":"2024-09-15T23:24:03.117798+0000","last_peered":"2024-09-15T23:24:03.117798+0000","last_clean":"2024-09-15T23:24:03.117798+0000","last_became_active":"2024-09-15T23:21:59.565835+0000","last_became_peered":"2024-09-15T23:21:59.565835+0000","last_unstale":"2024-09-15T23:24:03.117798+0000","last_undegraded":"2024-09-15T23:24:03.117798+0000","last_fullsized":"2024-09-15T23:24:03.117798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:38.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116276+0000","last_change":"2024-09-15T23:21:52.539544+0000","last_active":"2024-09-15T23:24:03.116276+0000","last_peered":"2024-09-15T23:24:03.116276+0000","last_clean":"2024-09-15T23:24:03.116276+0000","last_became_active":"2024-09-15T23:21:52.539203+0000","last_became_peered":"2024-09-15T23:21:52.539203+0000","last_unstale":"2024-09-15T23:24:03.116276+0000","last_undegraded":"2024-09-15T23:24:03.116276+0000","last_fullsized":"2024-09-15T23:24:03.116276+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.193965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:07.046123+0000","last_change":"2024-09-15T23:21:50.513543+0000","last_active":"2024-09-15T23:24:07.046123+0000","last_peered":"2024-09-15T23:24:07.046123+0000","last_clean":"2024-09-15T23:24:07.046123+0000","last_became_active":"2024-09-15T23:21:50.512812+0000","last_became_peered":"2024-09-15T23:21:50.512812+0000","last_unstale":"2024-09-15T23:24:07.046123+0000","last_undegraded":"2024-09-15T23:24:07.046123+0000","last_fullsized":"2024-09-15T23:24:07.046123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:50.954096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117689+0000","last_change":"2024-09-15T23:21:59.570163+0000","last_active":"2024-09-15T23:24:03.117689+0000","last_peered":"2024-09-15T23:24:03.117689+0000","last_clean":"2024-09-15T23:24:03.117689+0000","last_became_active":"2024-09-15T23:21:59.569805+0000","last_became_peered":"2024-09-15T23:21:59.569805+0000","last_unstale":"2024-09-15T23:24:03.117689+0000","last_undegraded":"2024-09-15T23:24:03.117689+0000","last_fullsized":"2024-09-15T23:24:03.117689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:28.530817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117877+0000","last_change":"2024-09-15T23:21:52.537850+0000","last_active":"2024-09-15T23:24:03.117877+0000","last_peered":"2024-09-15T23:24:03.117877+0000","last_clean":"2024-09-15T23:24:03.117877+0000","last_became_active":"2024-09-15T23:21:52.536579+0000","last_became_peered":"2024-09-15T23:21:52.536579+0000","last_unstale":"2024-09-15T23:24:03.117877+0000","last_undegraded":"2024-09-15T23:24:03.117877+0000","last_fullsized":"2024-09-15T23:24:03.117877+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:06.353615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118588+0000","last_change":"2024-09-15T23:21:50.562221+0000","last_active":"2024-09-15T23:24:03.118588+0000","last_peered":"2024-09-15T23:24:03.118588+0000","last_clean":"2024-09-15T23:24:03.118588+0000","last_became_active":"2024-09-15T23:21:50.561788+0000","last_became_peered":"2024-09-15T23:21:50.561788+0000","last_unstale":"2024-09-15T23:24:03.118588+0000","last_undegraded":"2024-09-15T23:24:03.118588+0000","last_fullsized":"2024-09-15T23:24:03.118588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.853812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109849+0000","last_change":"2024-09-15T23:23:02.383003+0000","last_active":"2024-09-15T23:24:03.109849+0000","last_peered":"2024-09-15T23:24:03.109849+0000","last_clean":"2024-09-15T23:24:03.109849+0000","last_became_active":"2024-09-15T23:23:02.382629+0000","last_became_peered":"2024-09-15T23:23:02.382629+0000","last_unstale":"2024-09-15T23:24:03.109849+0000","last_undegraded":"2024-09-15T23:24:03.109849+0000","last_fullsized":"2024-09-15T23:24:03.109849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:50.114689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113567+0000","last_change":"2024-09-15T23:21:52.539566+0000","last_active":"2024-09-15T23:24:03.113567+0000","last_peered":"2024-09-15T23:24:03.113567+0000","last_clean":"2024-09-15T23:24:03.113567+0000","last_became_active":"2024-09-15T23:21:52.539219+0000","last_became_peered":"2024-09-15T23:21:52.539219+0000","last_unstale":"2024-09-15T23:24:03.113567+0000","last_undegraded":"2024-09-15T23:24:03.113567+0000","last_fullsized":"2024-09-15T23:24:03.113567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:41.266308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:29.159267+0000","last_change":"2024-09-15T23:24:03.127667+0000","last_active":"2024-09-15T23:24:29.159267+0000","last_peered":"2024-09-15T23:24:29.159267+0000","last_clean":"2024-09-15T23:24:29.159267+0000","last_became_active":"2024-09-15T23:24:03.121613+0000","last_became_peered":"2024-09-15T23:24:03.121613+0000","last_unstale":"2024-09-15T23:24:29.159267+0000","last_undegraded":"2024-09-15T23:24:29.159267+0000","last_fullsized":"2024-09-15T23:24:29.159267+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:54:24.166895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1122,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194642+0000","last_change":"2024-09-15T23:21:59.568087+0000","last_active":"2024-09-15T23:24:03.194642+0000","last_peered":"2024-09-15T23:24:03.194642+0000","last_clean":"2024-09-15T23:24:03.194642+0000","last_became_active":"2024-09-15T23:21:59.567693+0000","last_became_peered":"2024-09-15T23:21:59.567693+0000","last_unstale":"2024-09-15T23:24:03.194642+0000","last_undegraded":"2024-09-15T23:24:03.194642+0000","last_fullsized":"2024-09-15T23:24:03.194642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.231873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114396+0000","last_change":"2024-09-15T23:21:52.531090+0000","last_active":"2024-09-15T23:24:03.114396+0000","last_peered":"2024-09-15T23:24:03.114396+0000","last_clean":"2024-09-15T23:24:03.114396+0000","last_became_active":"2024-09-15T23:21:52.530738+0000","last_became_peered":"2024-09-15T23:21:52.530738+0000","last_unstale":"2024-09-15T23:24:03.114396+0000","last_undegraded":"2024-09-15T23:24:03.114396+0000","last_fullsized":"2024-09-15T23:24:03.114396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:35.325128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117423+0000","last_change":"2024-09-15T23:21:50.514100+0000","last_active":"2024-09-15T23:24:03.117423+0000","last_peered":"2024-09-15T23:24:03.117423+0000","last_clean":"2024-09-15T23:24:03.117423+0000","last_became_active":"2024-09-15T23:21:50.513629+0000","last_became_peered":"2024-09-15T23:21:50.513629+0000","last_unstale":"2024-09-15T23:24:03.117423+0000","last_undegraded":"2024-09-15T23:24:03.117423+0000","last_fullsized":"2024-09-15T23:24:03.117423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.187544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115275+0000","last_change":"2024-09-15T23:21:59.562804+0000","last_active":"2024-09-15T23:24:03.115275+0000","last_peered":"2024-09-15T23:24:03.115275+0000","last_clean":"2024-09-15T23:24:03.115275+0000","last_became_active":"2024-09-15T23:21:59.562368+0000","last_became_peered":"2024-09-15T23:21:59.562368+0000","last_unstale":"2024-09-15T23:24:03.115275+0000","last_undegraded":"2024-09-15T23:24:03.115275+0000","last_fullsized":"2024-09-15T23:24:03.115275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:21.137333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109909+0000","last_change":"2024-09-15T23:21:52.530562+0000","last_active":"2024-09-15T23:24:03.109909+0000","last_peered":"2024-09-15T23:24:03.109909+0000","last_clean":"2024-09-15T23:24:03.109909+0000","last_became_active":"2024-09-15T23:21:52.530280+0000","last_became_peered":"2024-09-15T23:21:52.530280+0000","last_unstale":"2024-09-15T23:24:03.109909+0000","last_undegraded":"2024-09-15T23:24:03.109909+0000","last_fullsized":"2024-09-15T23:24:03.109909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:26.835651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113202+0000","last_change":"2024-09-15T23:21:50.513179+0000","last_active":"2024-09-15T23:24:03.113202+0000","last_peered":"2024-09-15T23:24:03.113202+0000","last_clean":"2024-09-15T23:24:03.113202+0000","last_became_active":"2024-09-15T23:21:50.512790+0000","last_became_peered":"2024-09-15T23:21:50.512790+0000","last_unstale":"2024-09-15T23:24:03.113202+0000","last_undegraded":"2024-09-15T23:24:03.113202+0000","last_fullsized":"2024-09-15T23:24:03.113202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:56.408198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113316+0000","last_change":"2024-09-15T23:23:02.382964+0000","last_active":"2024-09-15T23:24:03.113316+0000","last_peered":"2024-09-15T23:24:03.113316+0000","last_clean":"2024-09-15T23:24:03.113316+0000","last_became_active":"2024-09-15T23:23:02.382556+0000","last_became_peered":"2024-09-15T23:23:02.382556+0000","last_unstale":"2024-09-15T23:24:03.113316+0000","last_undegraded":"2024-09-15T23:24:03.113316+0000","last_fullsized":"2024-09-15T23:24:03.113316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:49.211591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117484+0000","last_change":"2024-09-15T23:21:52.530970+0000","last_active":"2024-09-15T23:24:03.117484+0000","last_peered":"2024-09-15T23:24:03.117484+0000","last_clean":"2024-09-15T23:24:03.117484+0000","last_became_active":"2024-09-15T23:21:52.530636+0000","last_became_peered":"2024-09-15T23:21:52.530636+0000","last_unstale":"2024-09-15T23:24:03.117484+0000","last_undegraded":"2024-09-15T23:24:03.117484+0000","last_fullsized":"2024-09-15T23:24:03.117484+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:33.374658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'41","reported_seq":329,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:25.153679+0000","last_change":"2024-09-15T23:24:03.131527+0000","last_active":"2024-09-15T23:24:25.153679+0000","last_peered":"2024-09-15T23:24:25.153679+0000","last_clean":"2024-09-15T23:24:25.153679+0000","last_became_active":"2024-09-15T23:24:03.119053+0000","last_became_peered":"2024-09-15T23:24:03.119053+0000","last_unstale":"2024-09-15T23:24:25.153679+0000","last_undegraded":"2024-09-15T23:24:25.153679+0000","last_fullsized":"2024-09-15T23:24:25.153679+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:21.870435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70919,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":226,"num_read_kb":183,"num_write":43,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":62325,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113841+0000","last_change":"2024-09-15T23:23:02.277457+0000","last_active":"2024-09-15T23:24:03.113841+0000","last_peered":"2024-09-15T23:24:03.113841+0000","last_clean":"2024-09-15T23:24:03.113841+0000","last_became_active":"2024-09-15T23:23:02.276960+0000","last_became_peered":"2024-09-15T23:23:02.276960+0000","last_unstale":"2024-09-15T23:24:03.113841+0000","last_undegraded":"2024-09-15T23:24:03.113841+0000","last_fullsized":"2024-09-15T23:24:03.113841+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:15:28.823804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117534+0000","last_change":"2024-09-15T23:21:52.530541+0000","last_active":"2024-09-15T23:24:03.117534+0000","last_peered":"2024-09-15T23:24:03.117534+0000","last_clean":"2024-09-15T23:24:03.117534+0000","last_became_active":"2024-09-15T23:21:52.530217+0000","last_became_peered":"2024-09-15T23:21:52.530217+0000","last_unstale":"2024-09-15T23:24:03.117534+0000","last_undegraded":"2024-09-15T23:24:03.117534+0000","last_fullsized":"2024-09-15T23:24:03.117534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:14.864563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118500+0000","last_change":"2024-09-15T23:21:50.562203+0000","last_active":"2024-09-15T23:24:03.118500+0000","last_peered":"2024-09-15T23:24:03.118500+0000","last_clean":"2024-09-15T23:24:03.118500+0000","last_became_active":"2024-09-15T23:21:50.561827+0000","last_became_peered":"2024-09-15T23:21:50.561827+0000","last_unstale":"2024-09-15T23:24:03.118500+0000","last_undegraded":"2024-09-15T23:24:03.118500+0000","last_fullsized":"2024-09-15T23:24:03.118500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:22.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111024+0000","last_change":"2024-09-15T23:21:59.561013+0000","last_active":"2024-09-15T23:24:03.111024+0000","last_peered":"2024-09-15T23:24:03.111024+0000","last_clean":"2024-09-15T23:24:03.111024+0000","last_became_active":"2024-09-15T23:21:59.560665+0000","last_became_peered":"2024-09-15T23:21:59.560665+0000","last_unstale":"2024-09-15T23:24:03.111024+0000","last_undegraded":"2024-09-15T23:24:03.111024+0000","last_fullsized":"2024-09-15T23:24:03.111024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:00.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194109+0000","last_change":"2024-09-15T23:21:52.529922+0000","last_active":"2024-09-15T23:24:03.194109+0000","last_peered":"2024-09-15T23:24:03.194109+0000","last_clean":"2024-09-15T23:24:03.194109+0000","last_became_active":"2024-09-15T23:21:52.529179+0000","last_became_peered":"2024-09-15T23:21:52.529179+0000","last_unstale":"2024-09-15T23:24:03.194109+0000","last_undegraded":"2024-09-15T23:24:03.194109+0000","last_fullsized":"2024-09-15T23:24:03.194109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.077658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113481+0000","last_change":"2024-09-15T23:21:50.514420+0000","last_active":"2024-09-15T23:24:03.113481+0000","last_peered":"2024-09-15T23:24:03.113481+0000","last_clean":"2024-09-15T23:24:03.113481+0000","last_became_active":"2024-09-15T23:21:50.514007+0000","last_became_peered":"2024-09-15T23:21:50.514007+0000","last_unstale":"2024-09-15T23:24:03.113481+0000","last_undegraded":"2024-09-15T23:24:03.113481+0000","last_fullsized":"2024-09-15T23:24:03.113481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:36.845674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193392+0000","last_change":"2024-09-15T23:21:59.565978+0000","last_active":"2024-09-15T23:24:03.193392+0000","last_peered":"2024-09-15T23:24:03.193392+0000","last_clean":"2024-09-15T23:24:03.193392+0000","last_became_active":"2024-09-15T23:21:59.565604+0000","last_became_peered":"2024-09-15T23:21:59.565604+0000","last_unstale":"2024-09-15T23:24:03.193392+0000","last_undegraded":"2024-09-15T23:24:03.193392+0000","last_fullsized":"2024-09-15T23:24:03.193392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:47.089504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118064+0000","last_change":"2024-09-15T23:21:52.543511+0000","last_active":"2024-09-15T23:24:03.118064+0000","last_peered":"2024-09-15T23:24:03.118064+0000","last_clean":"2024-09-15T23:24:03.118064+0000","last_became_active":"2024-09-15T23:21:52.543169+0000","last_became_peered":"2024-09-15T23:21:52.543169+0000","last_unstale":"2024-09-15T23:24:03.118064+0000","last_undegraded":"2024-09-15T23:24:03.118064+0000","last_fullsized":"2024-09-15T23:24:03.118064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:05.696772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":262,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:25.288749+0000","last_change":"2024-09-15T23:21:50.518425+0000","last_active":"2024-09-15T23:24:25.288749+0000","last_peered":"2024-09-15T23:24:25.288749+0000","last_clean":"2024-09-15T23:24:25.288749+0000","last_became_active":"2024-09-15T23:21:50.518052+0000","last_became_peered":"2024-09-15T23:21:50.518052+0000","last_unstale":"2024-09-15T23:24:25.288749+0000","last_undegraded":"2024-09-15T23:24:25.288749+0000","last_fullsized":"2024-09-15T23:24:25.288749+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:02.243869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":218,"num_read_kb":218,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112035+0000","last_change":"2024-09-15T23:21:59.570310+0000","last_active":"2024-09-15T23:24:03.112035+0000","last_peered":"2024-09-15T23:24:03.112035+0000","last_clean":"2024-09-15T23:24:03.112035+0000","last_became_active":"2024-09-15T23:21:59.569909+0000","last_became_peered":"2024-09-15T23:21:59.569909+0000","last_unstale":"2024-09-15T23:24:03.112035+0000","last_undegraded":"2024-09-15T23:24:03.112035+0000","last_fullsized":"2024-09-15T23:24:03.112035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:20:27.515537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194191+0000","last_change":"2024-09-15T23:21:59.568208+0000","last_active":"2024-09-15T23:24:03.194191+0000","last_peered":"2024-09-15T23:24:03.194191+0000","last_clean":"2024-09-15T23:24:03.194191+0000","last_became_active":"2024-09-15T23:21:59.567858+0000","last_became_peered":"2024-09-15T23:21:59.567858+0000","last_unstale":"2024-09-15T23:24:03.194191+0000","last_undegraded":"2024-09-15T23:24:03.194191+0000","last_fullsized":"2024-09-15T23:24:03.194191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:34:42.895247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193532+0000","last_change":"2024-09-15T23:21:52.532679+0000","last_active":"2024-09-15T23:24:03.193532+0000","last_peered":"2024-09-15T23:24:03.193532+0000","last_clean":"2024-09-15T23:24:03.193532+0000","last_became_active":"2024-09-15T23:21:52.532345+0000","last_became_peered":"2024-09-15T23:21:52.532345+0000","last_unstale":"2024-09-15T23:24:03.193532+0000","last_undegraded":"2024-09-15T23:24:03.193532+0000","last_fullsized":"2024-09-15T23:24:03.193532+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:45.943653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118135+0000","last_change":"2024-09-15T23:21:50.510836+0000","last_active":"2024-09-15T23:24:03.118135+0000","last_peered":"2024-09-15T23:24:03.118135+0000","last_clean":"2024-09-15T23:24:03.118135+0000","last_became_active":"2024-09-15T23:21:50.510514+0000","last_became_peered":"2024-09-15T23:21:50.510514+0000","last_unstale":"2024-09-15T23:24:03.118135+0000","last_undegraded":"2024-09-15T23:24:03.118135+0000","last_fullsized":"2024-09-15T23:24:03.118135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:06.404612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116423+0000","last_change":"2024-09-15T23:23:02.383436+0000","last_active":"2024-09-15T23:24:03.116423+0000","last_peered":"2024-09-15T23:24:03.116423+0000","last_clean":"2024-09-15T23:24:03.116423+0000","last_became_active":"2024-09-15T23:23:02.382982+0000","last_became_peered":"2024-09-15T23:23:02.382982+0000","last_unstale":"2024-09-15T23:24:03.116423+0000","last_undegraded":"2024-09-15T23:24:03.116423+0000","last_fullsized":"2024-09-15T23:24:03.116423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:55:14.076604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111923+0000","last_change":"2024-09-15T23:21:52.526766+0000","last_active":"2024-09-15T23:24:03.111923+0000","last_peered":"2024-09-15T23:24:03.111923+0000","last_clean":"2024-09-15T23:24:03.111923+0000","last_became_active":"2024-09-15T23:21:52.526446+0000","last_became_peered":"2024-09-15T23:21:52.526446+0000","last_unstale":"2024-09-15T23:24:03.111923+0000","last_undegraded":"2024-09-15T23:24:03.111923+0000","last_fullsized":"2024-09-15T23:24:03.111923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:21:37.004730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109384+0000","last_change":"2024-09-15T23:21:50.512928+0000","last_active":"2024-09-15T23:24:03.109384+0000","last_peered":"2024-09-15T23:24:03.109384+0000","last_clean":"2024-09-15T23:24:03.109384+0000","last_became_active":"2024-09-15T23:21:50.512558+0000","last_became_peered":"2024-09-15T23:21:50.512558+0000","last_unstale":"2024-09-15T23:24:03.109384+0000","last_undegraded":"2024-09-15T23:24:03.109384+0000","last_fullsized":"2024-09-15T23:24:03.109384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:08.653084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114309+0000","last_change":"2024-09-15T23:21:50.516920+0000","last_active":"2024-09-15T23:24:03.114309+0000","last_peered":"2024-09-15T23:24:03.114309+0000","last_clean":"2024-09-15T23:24:03.114309+0000","last_became_active":"2024-09-15T23:21:50.516573+0000","last_became_peered":"2024-09-15T23:21:50.516573+0000","last_unstale":"2024-09-15T23:24:03.114309+0000","last_undegraded":"2024-09-15T23:24:03.114309+0000","last_fullsized":"2024-09-15T23:24:03.114309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.367903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117266+0000","last_change":"2024-09-15T23:21:52.536696+0000","last_active":"2024-09-15T23:24:03.117266+0000","last_peered":"2024-09-15T23:24:03.117266+0000","last_clean":"2024-09-15T23:24:03.117266+0000","last_became_active":"2024-09-15T23:21:52.536338+0000","last_became_peered":"2024-09-15T23:21:52.536338+0000","last_unstale":"2024-09-15T23:24:03.117266+0000","last_undegraded":"2024-09-15T23:24:03.117266+0000","last_fullsized":"2024-09-15T23:24:03.117266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:05.152749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112910+0000","last_change":"2024-09-15T23:21:59.562968+0000","last_active":"2024-09-15T23:24:03.112910+0000","last_peered":"2024-09-15T23:24:03.112910+0000","last_clean":"2024-09-15T23:24:03.112910+0000","last_became_active":"2024-09-15T23:21:59.562579+0000","last_became_peered":"2024-09-15T23:21:59.562579+0000","last_unstale":"2024-09-15T23:24:03.112910+0000","last_undegraded":"2024-09-15T23:24:03.112910+0000","last_fullsized":"2024-09-15T23:24:03.112910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:02.853057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":157,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:25.290273+0000","last_change":"2024-09-15T23:21:50.513944+0000","last_active":"2024-09-15T23:24:25.290273+0000","last_peered":"2024-09-15T23:24:25.290273+0000","last_clean":"2024-09-15T23:24:25.290273+0000","last_became_active":"2024-09-15T23:21:50.513484+0000","last_became_peered":"2024-09-15T23:21:50.513484+0000","last_unstale":"2024-09-15T23:24:25.290273+0000","last_undegraded":"2024-09-15T23:24:25.290273+0000","last_fullsized":"2024-09-15T23:24:25.290273+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:35.719344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118061+0000","last_change":"2024-09-15T23:21:52.539467+0000","last_active":"2024-09-15T23:24:03.118061+0000","last_peered":"2024-09-15T23:24:03.118061+0000","last_clean":"2024-09-15T23:24:03.118061+0000","last_became_active":"2024-09-15T23:21:52.538998+0000","last_became_peered":"2024-09-15T23:21:52.538998+0000","last_unstale":"2024-09-15T23:24:03.118061+0000","last_undegraded":"2024-09-15T23:24:03.118061+0000","last_fullsized":"2024-09-15T23:24:03.118061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:22.213857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113057+0000","last_change":"2024-09-15T23:21:59.564842+0000","last_active":"2024-09-15T23:24:03.113057+0000","last_peered":"2024-09-15T23:24:03.113057+0000","last_clean":"2024-09-15T23:24:03.113057+0000","last_became_active":"2024-09-15T23:21:59.564408+0000","last_became_peered":"2024-09-15T23:21:59.564408+0000","last_unstale":"2024-09-15T23:24:03.113057+0000","last_undegraded":"2024-09-15T23:24:03.113057+0000","last_fullsized":"2024-09-15T23:24:03.113057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:17.909020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115745+0000","last_change":"2024-09-15T23:23:02.149420+0000","last_active":"2024-09-15T23:24:03.115745+0000","last_peered":"2024-09-15T23:24:03.115745+0000","last_clean":"2024-09-15T23:24:03.115745+0000","last_became_active":"2024-09-15T23:23:02.149003+0000","last_became_peered":"2024-09-15T23:23:02.149003+0000","last_unstale":"2024-09-15T23:24:03.115745+0000","last_undegraded":"2024-09-15T23:24:03.115745+0000","last_fullsized":"2024-09-15T23:24:03.115745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:45.490322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.532769+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532426+0000","last_became_peered":"2024-09-15T23:21:52.532426+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:09.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.533148+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532823+0000","last_became_peered":"2024-09-15T23:21:52.532823+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:01.833945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193803+0000","last_change":"2024-09-15T23:21:52.536790+0000","last_active":"2024-09-15T23:24:03.193803+0000","last_peered":"2024-09-15T23:24:03.193803+0000","last_clean":"2024-09-15T23:24:03.193803+0000","last_became_active":"2024-09-15T23:21:52.536515+0000","last_became_peered":"2024-09-15T23:21:52.536515+0000","last_unstale":"2024-09-15T23:24:03.193803+0000","last_undegraded":"2024-09-15T23:24:03.193803+0000","last_fullsized":"2024-09-15T23:24:03.193803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:37.207759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109800+0000","last_change":"2024-09-15T23:23:02.382842+0000","last_active":"2024-09-15T23:24:03.109800+0000","last_peered":"2024-09-15T23:24:03.109800+0000","last_clean":"2024-09-15T23:24:03.109800+0000","last_became_active":"2024-09-15T23:23:02.382455+0000","last_became_peered":"2024-09-15T23:23:02.382455+0000","last_unstale":"2024-09-15T23:24:03.109800+0000","last_undegraded":"2024-09-15T23:24:03.109800+0000","last_fullsized":"2024-09-15T23:24:03.109800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:23.617641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.540628+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.540308+0000","last_became_peered":"2024-09-15T23:21:52.540308+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:16.660637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113381+0000","last_change":"2024-09-15T23:21:52.533780+0000","last_active":"2024-09-15T23:24:03.113381+0000","last_peered":"2024-09-15T23:24:03.113381+0000","last_clean":"2024-09-15T23:24:03.113381+0000","last_became_active":"2024-09-15T23:21:52.533354+0000","last_became_peered":"2024-09-15T23:21:52.533354+0000","last_unstale":"2024-09-15T23:24:03.113381+0000","last_undegraded":"2024-09-15T23:24:03.113381+0000","last_fullsized":"2024-09-15T23:24:03.113381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.381059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112801+0000","last_change":"2024-09-15T23:21:52.536406+0000","last_active":"2024-09-15T23:24:03.112801+0000","last_peered":"2024-09-15T23:24:03.112801+0000","last_clean":"2024-09-15T23:24:03.112801+0000","last_became_active":"2024-09-15T23:21:52.536091+0000","last_became_peered":"2024-09-15T23:21:52.536091+0000","last_unstale":"2024-09-15T23:24:03.112801+0000","last_undegraded":"2024-09-15T23:24:03.112801+0000","last_fullsized":"2024-09-15T23:24:03.112801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:30.452201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117879+0000","last_change":"2024-09-15T23:21:52.536333+0000","last_active":"2024-09-15T23:24:03.117879+0000","last_peered":"2024-09-15T23:24:03.117879+0000","last_clean":"2024-09-15T23:24:03.117879+0000","last_became_active":"2024-09-15T23:21:52.534677+0000","last_became_peered":"2024-09-15T23:21:52.534677+0000","last_unstale":"2024-09-15T23:24:03.117879+0000","last_undegraded":"2024-09-15T23:24:03.117879+0000","last_fullsized":"2024-09-15T23:24:03.117879+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:05.287914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112116+0000","last_change":"2024-09-15T23:21:52.536442+0000","last_active":"2024-09-15T23:24:03.112116+0000","last_peered":"2024-09-15T23:24:03.112116+0000","last_clean":"2024-09-15T23:24:03.112116+0000","last_became_active":"2024-09-15T23:21:52.536183+0000","last_became_peered":"2024-09-15T23:21:52.536183+0000","last_unstale":"2024-09-15T23:24:03.112116+0000","last_undegraded":"2024-09-15T23:24:03.112116+0000","last_fullsized":"2024-09-15T23:24:03.112116+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:41.641370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117398+0000","last_change":"2024-09-15T23:23:02.277365+0000","last_active":"2024-09-15T23:24:03.117398+0000","last_peered":"2024-09-15T23:24:03.117398+0000","last_clean":"2024-09-15T23:24:03.117398+0000","last_became_active":"2024-09-15T23:23:02.276905+0000","last_became_peered":"2024-09-15T23:23:02.276905+0000","last_unstale":"2024-09-15T23:24:03.117398+0000","last_undegraded":"2024-09-15T23:24:03.117398+0000","last_fullsized":"2024-09-15T23:24:03.117398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:21.100657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109788+0000","last_change":"2024-09-15T23:21:52.531171+0000","last_active":"2024-09-15T23:24:03.109788+0000","last_peered":"2024-09-15T23:24:03.109788+0000","last_clean":"2024-09-15T23:24:03.109788+0000","last_became_active":"2024-09-15T23:21:52.530724+0000","last_became_peered":"2024-09-15T23:21:52.530724+0000","last_unstale":"2024-09-15T23:24:03.109788+0000","last_undegraded":"2024-09-15T23:24:03.109788+0000","last_fullsized":"2024-09-15T23:24:03.109788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:11.009119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115400+0000","last_change":"2024-09-15T23:21:52.530433+0000","last_active":"2024-09-15T23:24:03.115400+0000","last_peered":"2024-09-15T23:24:03.115400+0000","last_clean":"2024-09-15T23:24:03.115400+0000","last_became_active":"2024-09-15T23:21:52.530058+0000","last_became_peered":"2024-09-15T23:21:52.530058+0000","last_unstale":"2024-09-15T23:24:03.115400+0000","last_undegraded":"2024-09-15T23:24:03.115400+0000","last_fullsized":"2024-09-15T23:24:03.115400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:51.208820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113218+0000","last_change":"2024-09-15T23:21:52.534639+0000","last_active":"2024-09-15T23:24:03.113218+0000","last_peered":"2024-09-15T23:24:03.113218+0000","last_clean":"2024-09-15T23:24:03.113218+0000","last_became_active":"2024-09-15T23:21:52.534362+0000","last_became_peered":"2024-09-15T23:21:52.534362+0000","last_unstale":"2024-09-15T23:24:03.113218+0000","last_undegraded":"2024-09-15T23:24:03.113218+0000","last_fullsized":"2024-09-15T23:24:03.113218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:58:12.023501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194630+0000","last_change":"2024-09-15T23:21:52.532758+0000","last_active":"2024-09-15T23:24:03.194630+0000","last_peered":"2024-09-15T23:24:03.194630+0000","last_clean":"2024-09-15T23:24:03.194630+0000","last_became_active":"2024-09-15T23:21:52.532412+0000","last_became_peered":"2024-09-15T23:21:52.532412+0000","last_unstale":"2024-09-15T23:24:03.194630+0000","last_undegraded":"2024-09-15T23:24:03.194630+0000","last_fullsized":"2024-09-15T23:24:03.194630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:46.285711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116737+0000","last_change":"2024-09-15T23:21:52.533928+0000","last_active":"2024-09-15T23:24:03.116737+0000","last_peered":"2024-09-15T23:24:03.116737+0000","last_clean":"2024-09-15T23:24:03.116737+0000","last_became_active":"2024-09-15T23:21:52.533578+0000","last_became_peered":"2024-09-15T23:21:52.533578+0000","last_unstale":"2024-09-15T23:24:03.116737+0000","last_undegraded":"2024-09-15T23:24:03.116737+0000","last_fullsized":"2024-09-15T23:24:03.116737+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:49.476316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116894+0000","last_change":"2024-09-15T23:21:52.533417+0000","last_active":"2024-09-15T23:24:03.116894+0000","last_peered":"2024-09-15T23:24:03.116894+0000","last_clean":"2024-09-15T23:24:03.116894+0000","last_became_active":"2024-09-15T23:21:52.533085+0000","last_became_peered":"2024-09-15T23:21:52.533085+0000","last_unstale":"2024-09-15T23:24:03.116894+0000","last_undegraded":"2024-09-15T23:24:03.116894+0000","last_fullsized":"2024-09-15T23:24:03.116894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:20.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193763+0000","last_change":"2024-09-15T23:21:52.542340+0000","last_active":"2024-09-15T23:24:03.193763+0000","last_peered":"2024-09-15T23:24:03.193763+0000","last_clean":"2024-09-15T23:24:03.193763+0000","last_became_active":"2024-09-15T23:21:52.542048+0000","last_became_peered":"2024-09-15T23:21:52.542048+0000","last_unstale":"2024-09-15T23:24:03.193763+0000","last_undegraded":"2024-09-15T23:24:03.193763+0000","last_fullsized":"2024-09-15T23:24:03.193763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:57.557420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193941+0000","last_change":"2024-09-15T23:21:52.535576+0000","last_active":"2024-09-15T23:24:03.193941+0000","last_peered":"2024-09-15T23:24:03.193941+0000","last_clean":"2024-09-15T23:24:03.193941+0000","last_became_active":"2024-09-15T23:21:52.535223+0000","last_became_peered":"2024-09-15T23:21:52.535223+0000","last_unstale":"2024-09-15T23:24:03.193941+0000","last_undegraded":"2024-09-15T23:24:03.193941+0000","last_fullsized":"2024-09-15T23:24:03.193941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:45.874927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109691+0000","last_change":"2024-09-15T23:23:02.150556+0000","last_active":"2024-09-15T23:24:03.109691+0000","last_peered":"2024-09-15T23:24:03.109691+0000","last_clean":"2024-09-15T23:24:03.109691+0000","last_became_active":"2024-09-15T23:23:02.150253+0000","last_became_peered":"2024-09-15T23:23:02.150253+0000","last_unstale":"2024-09-15T23:24:03.109691+0000","last_undegraded":"2024-09-15T23:24:03.109691+0000","last_fullsized":"2024-09-15T23:24:03.109691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:47.950356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110875+0000","last_change":"2024-09-15T23:21:52.537012+0000","last_active":"2024-09-15T23:24:03.110875+0000","last_peered":"2024-09-15T23:24:03.110875+0000","last_clean":"2024-09-15T23:24:03.110875+0000","last_became_active":"2024-09-15T23:21:52.536245+0000","last_became_peered":"2024-09-15T23:21:52.536245+0000","last_unstale":"2024-09-15T23:24:03.110875+0000","last_undegraded":"2024-09-15T23:24:03.110875+0000","last_fullsized":"2024-09-15T23:24:03.110875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:31.150491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114034+0000","last_change":"2024-09-15T23:21:52.540431+0000","last_active":"2024-09-15T23:24:03.114034+0000","last_peered":"2024-09-15T23:24:03.114034+0000","last_clean":"2024-09-15T23:24:03.114034+0000","last_became_active":"2024-09-15T23:21:52.540104+0000","last_became_peered":"2024-09-15T23:21:52.540104+0000","last_unstale":"2024-09-15T23:24:03.114034+0000","last_undegraded":"2024-09-15T23:24:03.114034+0000","last_fullsized":"2024-09-15T23:24:03.114034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.550258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114876+0000","last_change":"2024-09-15T23:21:52.537590+0000","last_active":"2024-09-15T23:24:03.114876+0000","last_peered":"2024-09-15T23:24:03.114876+0000","last_clean":"2024-09-15T23:24:03.114876+0000","last_became_active":"2024-09-15T23:21:52.533804+0000","last_became_peered":"2024-09-15T23:21:52.533804+0000","last_unstale":"2024-09-15T23:24:03.114876+0000","last_undegraded":"2024-09-15T23:24:03.114876+0000","last_fullsized":"2024-09-15T23:24:03.114876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:48.449090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117952+0000","last_change":"2024-09-15T23:21:52.536130+0000","last_active":"2024-09-15T23:24:03.117952+0000","last_peered":"2024-09-15T23:24:03.117952+0000","last_clean":"2024-09-15T23:24:03.117952+0000","last_became_active":"2024-09-15T23:21:52.535336+0000","last_became_peered":"2024-09-15T23:21:52.535336+0000","last_unstale":"2024-09-15T23:24:03.117952+0000","last_undegraded":"2024-09-15T23:24:03.117952+0000","last_fullsized":"2024-09-15T23:24:03.117952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:52.581994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111899+0000","last_change":"2024-09-15T23:21:52.540485+0000","last_active":"2024-09-15T23:24:03.111899+0000","last_peered":"2024-09-15T23:24:03.111899+0000","last_clean":"2024-09-15T23:24:03.111899+0000","last_became_active":"2024-09-15T23:21:52.540178+0000","last_became_peered":"2024-09-15T23:21:52.540178+0000","last_unstale":"2024-09-15T23:24:03.111899+0000","last_undegraded":"2024-09-15T23:24:03.111899+0000","last_fullsized":"2024-09-15T23:24:03.111899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:40:22.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113883+0000","last_change":"2024-09-15T23:21:52.538845+0000","last_active":"2024-09-15T23:24:03.113883+0000","last_peered":"2024-09-15T23:24:03.113883+0000","last_clean":"2024-09-15T23:24:03.113883+0000","last_became_active":"2024-09-15T23:21:52.538511+0000","last_became_peered":"2024-09-15T23:21:52.538511+0000","last_unstale":"2024-09-15T23:24:03.113883+0000","last_undegraded":"2024-09-15T23:24:03.113883+0000","last_fullsized":"2024-09-15T23:24:03.113883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:00.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114158+0000","last_change":"2024-09-15T23:23:02.149590+0000","last_active":"2024-09-15T23:24:03.114158+0000","last_peered":"2024-09-15T23:24:03.114158+0000","last_clean":"2024-09-15T23:24:03.114158+0000","last_became_active":"2024-09-15T23:23:02.149184+0000","last_became_peered":"2024-09-15T23:23:02.149184+0000","last_unstale":"2024-09-15T23:24:03.114158+0000","last_undegraded":"2024-09-15T23:24:03.114158+0000","last_fullsized":"2024-09-15T23:24:03.114158+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:05:29.407175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117530+0000","last_change":"2024-09-15T23:21:52.530524+0000","last_active":"2024-09-15T23:24:03.117530+0000","last_peered":"2024-09-15T23:24:03.117530+0000","last_clean":"2024-09-15T23:24:03.117530+0000","last_became_active":"2024-09-15T23:21:52.530203+0000","last_became_peered":"2024-09-15T23:21:52.530203+0000","last_unstale":"2024-09-15T23:24:03.117530+0000","last_undegraded":"2024-09-15T23:24:03.117530+0000","last_fullsized":"2024-09-15T23:24:03.117530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:50.560211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110690+0000","last_change":"2024-09-15T23:21:52.537256+0000","last_active":"2024-09-15T23:24:03.110690+0000","last_peered":"2024-09-15T23:24:03.110690+0000","last_clean":"2024-09-15T23:24:03.110690+0000","last_became_active":"2024-09-15T23:21:52.536731+0000","last_became_peered":"2024-09-15T23:21:52.536731+0000","last_unstale":"2024-09-15T23:24:03.110690+0000","last_undegraded":"2024-09-15T23:24:03.110690+0000","last_fullsized":"2024-09-15T23:24:03.110690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:18.815398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194364+0000","last_change":"2024-09-15T23:21:52.537728+0000","last_active":"2024-09-15T23:24:03.194364+0000","last_peered":"2024-09-15T23:24:03.194364+0000","last_clean":"2024-09-15T23:24:03.194364+0000","last_became_active":"2024-09-15T23:21:52.537434+0000","last_became_peered":"2024-09-15T23:21:52.537434+0000","last_unstale":"2024-09-15T23:24:03.194364+0000","last_undegraded":"2024-09-15T23:24:03.194364+0000","last_fullsized":"2024-09-15T23:24:03.194364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:25.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194218+0000","last_change":"2024-09-15T23:21:52.534780+0000","last_active":"2024-09-15T23:24:03.194218+0000","last_peered":"2024-09-15T23:24:03.194218+0000","last_clean":"2024-09-15T23:24:03.194218+0000","last_became_active":"2024-09-15T23:21:52.534399+0000","last_became_peered":"2024-09-15T23:21:52.534399+0000","last_unstale":"2024-09-15T23:24:03.194218+0000","last_undegraded":"2024-09-15T23:24:03.194218+0000","last_fullsized":"2024-09-15T23:24:03.194218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:47.344129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116592+0000","last_change":"2024-09-15T23:21:52.533403+0000","last_active":"2024-09-15T23:24:03.116592+0000","last_peered":"2024-09-15T23:24:03.116592+0000","last_clean":"2024-09-15T23:24:03.116592+0000","last_became_active":"2024-09-15T23:21:52.533069+0000","last_became_peered":"2024-09-15T23:21:52.533069+0000","last_unstale":"2024-09-15T23:24:03.116592+0000","last_undegraded":"2024-09-15T23:24:03.116592+0000","last_fullsized":"2024-09-15T23:24:03.116592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:38.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":78576,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1139,"num_read_kb":651,"num_write":268,"num_write_kb":163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":63447,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":235728,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":246033,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579949,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":521776,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841585,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1163264,"data_stored":1178923,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005111,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":524287,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234044,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":520569,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462977,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1073152,"data_stored":1116395,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626502,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1003520,"data_stored":515695,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855435,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1112565,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018961,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1044480,"data_stored":591615,"data_compressed":39742,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247893,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":515403,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444123,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1019904,"data_stored":517253,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607648,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":587644,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1028096,"data_stored":517151,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":72671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":72546,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":73582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:24:33.426 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-15T23:24:33.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:24:33.676 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.188+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/995285179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b28102570 msgr2=0x7f2b281029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.188+0000 7f2b2d4eb640 1 --2- 172.21.15.89:0/995285179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b28102570 0x7f2b281029b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009900 tx=0x7f2b1c02a480 comp rx=0 tx=0).stop 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.188+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/995285179 shutdown_connections 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.188+0000 7f2b2d4eb640 1 --2- 172.21.15.89:0/995285179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b28102ef0 0x7f2b2810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.188+0000 7f2b2d4eb640 1 --2- 172.21.15.89:0/995285179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b28102570 0x7f2b281029b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.188+0000 7f2b2d4eb640 1 --2- 172.21.15.89:0/995285179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b28100e20 0x7f2b28101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.188+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/995285179 >> 172.21.15.89:0/995285179 conn(0x7f2b280fc7d0 msgr2=0x7f2b280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.188+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/995285179 shutdown_connections 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.188+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/995285179 wait complete. 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.189+0000 7f2b2d4eb640 1 Processor -- start 2024-09-15T23:24:34.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.189+0000 7f2b2d4eb640 1 -- start start 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.189+0000 7f2b2d4eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b28100e20 0x7f2b281992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.189+0000 7f2b2d4eb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b28102570 0x7f2b28199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b2d4eb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b28102ef0 0x7f2b281a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b2d4eb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2b28110f50 con 0x7f2b28100e20 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b2d4eb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2b28110dd0 con 0x7f2b28102570 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b2d4eb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2b281110d0 con 0x7f2b28102ef0 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b26ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b28100e20 0x7f2b281992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b26ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b28100e20 0x7f2b281992e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56160/0 (socket says 172.21.15.89:56160) 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b26ffd640 1 -- 172.21.15.89:0/4134837965 learned_addr learned my addr 172.21.15.89:0/4134837965 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b277fe640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b28102ef0 0x7f2b281a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b267fc640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b28102570 0x7f2b28199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b26ffd640 1 -- 172.21.15.89:0/4134837965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b28102ef0 msgr2=0x7f2b281a4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:34.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b26ffd640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b28102ef0 0x7f2b281a4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b26ffd640 1 -- 172.21.15.89:0/4134837965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b28102570 msgr2=0x7f2b28199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b26ffd640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b28102570 0x7f2b28199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b26ffd640 1 -- 172.21.15.89:0/4134837965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b281a48a0 con 0x7f2b28100e20 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b26ffd640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b28100e20 0x7f2b281992e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800c920 tx=0x7f2b1800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b0ffff640 1 -- 172.21.15.89:0/4134837965 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b18007c70 con 0x7f2b28100e20 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b0ffff640 1 -- 172.21.15.89:0/4134837965 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b18007e10 con 0x7f2b28100e20 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/4134837965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b281a4a70 con 0x7f2b28100e20 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b0ffff640 1 -- 172.21.15.89:0/4134837965 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b18018890 con 0x7f2b28100e20 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.190+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/4134837965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b281a4f80 con 0x7f2b28100e20 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.192+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/4134837965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b28101200 con 0x7f2b28100e20 2024-09-15T23:24:34.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.193+0000 7f2b0ffff640 1 -- 172.21.15.89:0/4134837965 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f2b18018a70 con 0x7f2b28100e20 2024-09-15T23:24:34.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.194+0000 7f2b0ffff640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b100a32d0 0x7f2b100a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:34.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.194+0000 7f2b267fc640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b100a32d0 0x7f2b100a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:34.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.194+0000 7f2b0ffff640 1 -- 172.21.15.89:0/4134837965 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2b180cffe0 con 0x7f2b28100e20 2024-09-15T23:24:34.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.194+0000 7f2b267fc640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b100a32d0 0x7f2b100a5790 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2b2806aac0 tx=0x7f2b1c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:34.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.198+0000 7f2b0ffff640 1 -- 172.21.15.89:0/4134837965 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2b18014030 con 0x7f2b28100e20 2024-09-15T23:24:34.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.359+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/4134837965 --> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2b28106f50 con 0x7f2b100a32d0 2024-09-15T23:24:34.361 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:34 smithi089 ceph-mon[30933]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:34.361 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:34 smithi089 ceph-mon[30933]: pgmap v366: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-15T23:24:34.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.367+0000 7f2b0ffff640 1 -- 172.21.15.89:0/4134837965 <== mgr.14150 v2:172.21.15.89:6800/3594493718 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340822 (secure 0 0 0) 0x7f2b28106f50 con 0x7f2b100a32d0 2024-09-15T23:24:34.372 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:34.373 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-15T23:24:34.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/4134837965 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b100a32d0 msgr2=0x7f2b100a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:34.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b100a32d0 0x7f2b100a5790 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2b2806aac0 tx=0x7f2b1c005a20 comp rx=0 tx=0).stop 2024-09-15T23:24:34.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/4134837965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b28100e20 msgr2=0x7f2b281992e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:34.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b28100e20 0x7f2b281992e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2b1800c920 tx=0x7f2b1800cde0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/4134837965 shutdown_connections 2024-09-15T23:24:34.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2b100a32d0 0x7f2b100a5790 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b28102ef0 0x7f2b281a4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2b28102570 0x7f2b28199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 --2- 172.21.15.89:0/4134837965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2b28100e20 0x7f2b281992e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/4134837965 >> 172.21.15.89:0/4134837965 conn(0x7f2b280fc7d0 msgr2=0x7f2b28106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:34.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/4134837965 shutdown_connections 2024-09-15T23:24:34.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:34.371+0000 7f2b2d4eb640 1 -- 172.21.15.89:0/4134837965 wait complete. 2024-09-15T23:24:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:34 smithi120 ceph-mon[35250]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:34 smithi120 ceph-mon[35250]: pgmap v366: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-15T23:24:34.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:34 smithi174 ceph-mon[35586]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:34 smithi174 ceph-mon[35586]: pgmap v366: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-15T23:24:35.044 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-15T23:24:33.024561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669178,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1301,"num_read_kb":781,"num_write":451,"num_write_kb":3103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1244183,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369884,"kb_used_data":12284,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571940,"statfs":{"total":1151940427776,"available":1151561666560,"internally_reserved":0,"allocated":12578816,"data_stored":8219276,"data_compressed":471759,"data_compressed_allocated":638976,"data_compressed_original":2813952,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8902,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":141,"num_read_kb":83,"num_write":13,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003397"},"pg_stats":[{"pgid":"2.1f","version":"86'13","reported_seq":163,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:26.289472+0000","last_change":"2024-09-15T23:21:50.513610+0000","last_active":"2024-09-15T23:24:26.289472+0000","last_peered":"2024-09-15T23:24:26.289472+0000","last_clean":"2024-09-15T23:24:26.289472+0000","last_became_active":"2024-09-15T23:21:50.513266+0000","last_became_peered":"2024-09-15T23:21:50.513266+0000","last_unstale":"2024-09-15T23:24:26.289472+0000","last_undegraded":"2024-09-15T23:24:26.289472+0000","last_fullsized":"2024-09-15T23:24:26.289472+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:16:48.692801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":112,"num_read_kb":112,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112578+0000","last_change":"2024-09-15T23:21:52.540248+0000","last_active":"2024-09-15T23:24:03.112578+0000","last_peered":"2024-09-15T23:24:03.112578+0000","last_clean":"2024-09-15T23:24:03.112578+0000","last_became_active":"2024-09-15T23:21:52.539945+0000","last_became_peered":"2024-09-15T23:21:52.539945+0000","last_unstale":"2024-09-15T23:24:03.112578+0000","last_undegraded":"2024-09-15T23:24:03.112578+0000","last_fullsized":"2024-09-15T23:24:03.112578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:33.402153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112456+0000","last_change":"2024-09-15T23:21:59.567246+0000","last_active":"2024-09-15T23:24:03.112456+0000","last_peered":"2024-09-15T23:24:03.112456+0000","last_clean":"2024-09-15T23:24:03.112456+0000","last_became_active":"2024-09-15T23:21:59.566832+0000","last_became_peered":"2024-09-15T23:21:59.566832+0000","last_unstale":"2024-09-15T23:24:03.112456+0000","last_undegraded":"2024-09-15T23:24:03.112456+0000","last_fullsized":"2024-09-15T23:24:03.112456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:46:19.663081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112747+0000","last_change":"2024-09-15T23:21:50.561767+0000","last_active":"2024-09-15T23:24:03.112747+0000","last_peered":"2024-09-15T23:24:03.112747+0000","last_clean":"2024-09-15T23:24:03.112747+0000","last_became_active":"2024-09-15T23:21:50.561385+0000","last_became_peered":"2024-09-15T23:21:50.561385+0000","last_unstale":"2024-09-15T23:24:03.112747+0000","last_undegraded":"2024-09-15T23:24:03.112747+0000","last_fullsized":"2024-09-15T23:24:03.112747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:51:11.897884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194008+0000","last_change":"2024-09-15T23:21:52.534874+0000","last_active":"2024-09-15T23:24:03.194008+0000","last_peered":"2024-09-15T23:24:03.194008+0000","last_clean":"2024-09-15T23:24:03.194008+0000","last_became_active":"2024-09-15T23:21:52.534334+0000","last_became_peered":"2024-09-15T23:21:52.534334+0000","last_unstale":"2024-09-15T23:24:03.194008+0000","last_undegraded":"2024-09-15T23:24:03.194008+0000","last_fullsized":"2024-09-15T23:24:03.194008+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:15.972648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110442+0000","last_change":"2024-09-15T23:21:59.559907+0000","last_active":"2024-09-15T23:24:03.110442+0000","last_peered":"2024-09-15T23:24:03.110442+0000","last_clean":"2024-09-15T23:24:03.110442+0000","last_became_active":"2024-09-15T23:21:59.559440+0000","last_became_peered":"2024-09-15T23:21:59.559440+0000","last_unstale":"2024-09-15T23:24:03.110442+0000","last_undegraded":"2024-09-15T23:24:03.110442+0000","last_fullsized":"2024-09-15T23:24:03.110442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:00.755112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117658+0000","last_change":"2024-09-15T23:21:50.515424+0000","last_active":"2024-09-15T23:24:03.117658+0000","last_peered":"2024-09-15T23:24:03.117658+0000","last_clean":"2024-09-15T23:24:03.117658+0000","last_became_active":"2024-09-15T23:21:50.514966+0000","last_became_peered":"2024-09-15T23:21:50.514966+0000","last_unstale":"2024-09-15T23:24:03.117658+0000","last_undegraded":"2024-09-15T23:24:03.117658+0000","last_fullsized":"2024-09-15T23:24:03.117658+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:57.586575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117412+0000","last_change":"2024-09-15T23:21:52.535189+0000","last_active":"2024-09-15T23:24:03.117412+0000","last_peered":"2024-09-15T23:24:03.117412+0000","last_clean":"2024-09-15T23:24:03.117412+0000","last_became_active":"2024-09-15T23:21:52.534807+0000","last_became_peered":"2024-09-15T23:21:52.534807+0000","last_unstale":"2024-09-15T23:24:03.117412+0000","last_undegraded":"2024-09-15T23:24:03.117412+0000","last_fullsized":"2024-09-15T23:24:03.117412+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:16:30.654382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110835+0000","last_change":"2024-09-15T23:21:59.564675+0000","last_active":"2024-09-15T23:24:03.110835+0000","last_peered":"2024-09-15T23:24:03.110835+0000","last_clean":"2024-09-15T23:24:03.110835+0000","last_became_active":"2024-09-15T23:21:59.564271+0000","last_became_peered":"2024-09-15T23:21:59.564271+0000","last_unstale":"2024-09-15T23:24:03.110835+0000","last_undegraded":"2024-09-15T23:24:03.110835+0000","last_fullsized":"2024-09-15T23:24:03.110835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:01.055369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117219+0000","last_change":"2024-09-15T23:21:50.507790+0000","last_active":"2024-09-15T23:24:03.117219+0000","last_peered":"2024-09-15T23:24:03.117219+0000","last_clean":"2024-09-15T23:24:03.117219+0000","last_became_active":"2024-09-15T23:21:50.507398+0000","last_became_peered":"2024-09-15T23:21:50.507398+0000","last_unstale":"2024-09-15T23:24:03.117219+0000","last_undegraded":"2024-09-15T23:24:03.117219+0000","last_fullsized":"2024-09-15T23:24:03.117219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:58:00.001279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116935+0000","last_change":"2024-09-15T23:21:52.537179+0000","last_active":"2024-09-15T23:24:03.116935+0000","last_peered":"2024-09-15T23:24:03.116935+0000","last_clean":"2024-09-15T23:24:03.116935+0000","last_became_active":"2024-09-15T23:21:52.536841+0000","last_became_peered":"2024-09-15T23:21:52.536841+0000","last_unstale":"2024-09-15T23:24:03.116935+0000","last_undegraded":"2024-09-15T23:24:03.116935+0000","last_fullsized":"2024-09-15T23:24:03.116935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:44:47.455700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110534+0000","last_change":"2024-09-15T23:21:59.564734+0000","last_active":"2024-09-15T23:24:03.110534+0000","last_peered":"2024-09-15T23:24:03.110534+0000","last_clean":"2024-09-15T23:24:03.110534+0000","last_became_active":"2024-09-15T23:21:59.564339+0000","last_became_peered":"2024-09-15T23:21:59.564339+0000","last_unstale":"2024-09-15T23:24:03.110534+0000","last_undegraded":"2024-09-15T23:24:03.110534+0000","last_fullsized":"2024-09-15T23:24:03.110534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:44.498480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113970+0000","last_change":"2024-09-15T23:21:52.538025+0000","last_active":"2024-09-15T23:24:03.113970+0000","last_peered":"2024-09-15T23:24:03.113970+0000","last_clean":"2024-09-15T23:24:03.113970+0000","last_became_active":"2024-09-15T23:21:52.537736+0000","last_became_peered":"2024-09-15T23:21:52.537736+0000","last_unstale":"2024-09-15T23:24:03.113970+0000","last_undegraded":"2024-09-15T23:24:03.113970+0000","last_fullsized":"2024-09-15T23:24:03.113970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:31.916888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.127260+0000","last_change":"2024-09-15T23:24:03.127260+0000","last_active":"2024-09-15T23:24:03.127260+0000","last_peered":"2024-09-15T23:24:03.127260+0000","last_clean":"2024-09-15T23:24:03.127260+0000","last_became_active":"2024-09-15T23:24:03.119384+0000","last_became_peered":"2024-09-15T23:24:03.119384+0000","last_unstale":"2024-09-15T23:24:03.127260+0000","last_undegraded":"2024-09-15T23:24:03.127260+0000","last_fullsized":"2024-09-15T23:24:03.127260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:35.325048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110625+0000","last_change":"2024-09-15T23:21:59.681971+0000","last_active":"2024-09-15T23:24:03.110625+0000","last_peered":"2024-09-15T23:24:03.110625+0000","last_clean":"2024-09-15T23:24:03.110625+0000","last_became_active":"2024-09-15T23:21:59.681432+0000","last_became_peered":"2024-09-15T23:21:59.681432+0000","last_unstale":"2024-09-15T23:24:03.110625+0000","last_undegraded":"2024-09-15T23:24:03.110625+0000","last_fullsized":"2024-09-15T23:24:03.110625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:26.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112179+0000","last_change":"2024-09-15T23:21:52.541281+0000","last_active":"2024-09-15T23:24:03.112179+0000","last_peered":"2024-09-15T23:24:03.112179+0000","last_clean":"2024-09-15T23:24:03.112179+0000","last_became_active":"2024-09-15T23:21:52.540772+0000","last_became_peered":"2024-09-15T23:21:52.540772+0000","last_unstale":"2024-09-15T23:24:03.112179+0000","last_undegraded":"2024-09-15T23:24:03.112179+0000","last_fullsized":"2024-09-15T23:24:03.112179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:23:44.444904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112700+0000","last_change":"2024-09-15T23:21:50.508844+0000","last_active":"2024-09-15T23:24:03.112700+0000","last_peered":"2024-09-15T23:24:03.112700+0000","last_clean":"2024-09-15T23:24:03.112700+0000","last_became_active":"2024-09-15T23:21:50.508364+0000","last_became_peered":"2024-09-15T23:21:50.508364+0000","last_unstale":"2024-09-15T23:24:03.112700+0000","last_undegraded":"2024-09-15T23:24:03.112700+0000","last_fullsized":"2024-09-15T23:24:03.112700+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:45.129711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110720+0000","last_change":"2024-09-15T23:21:59.562114+0000","last_active":"2024-09-15T23:24:03.110720+0000","last_peered":"2024-09-15T23:24:03.110720+0000","last_clean":"2024-09-15T23:24:03.110720+0000","last_became_active":"2024-09-15T23:21:59.561652+0000","last_became_peered":"2024-09-15T23:21:59.561652+0000","last_unstale":"2024-09-15T23:24:03.110720+0000","last_undegraded":"2024-09-15T23:24:03.110720+0000","last_fullsized":"2024-09-15T23:24:03.110720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:51.781886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.531678+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.531327+0000","last_became_peered":"2024-09-15T23:21:52.531327+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:07.152626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112193+0000","last_change":"2024-09-15T23:21:50.513197+0000","last_active":"2024-09-15T23:24:03.112193+0000","last_peered":"2024-09-15T23:24:03.112193+0000","last_clean":"2024-09-15T23:24:03.112193+0000","last_became_active":"2024-09-15T23:21:50.512794+0000","last_became_peered":"2024-09-15T23:21:50.512794+0000","last_unstale":"2024-09-15T23:24:03.112193+0000","last_undegraded":"2024-09-15T23:24:03.112193+0000","last_fullsized":"2024-09-15T23:24:03.112193+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:05.457467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194628+0000","last_change":"2024-09-15T23:21:59.565669+0000","last_active":"2024-09-15T23:24:03.194628+0000","last_peered":"2024-09-15T23:24:03.194628+0000","last_clean":"2024-09-15T23:24:03.194628+0000","last_became_active":"2024-09-15T23:21:59.565290+0000","last_became_peered":"2024-09-15T23:21:59.565290+0000","last_unstale":"2024-09-15T23:24:03.194628+0000","last_undegraded":"2024-09-15T23:24:03.194628+0000","last_fullsized":"2024-09-15T23:24:03.194628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:04.861560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116394+0000","last_change":"2024-09-15T23:21:52.529643+0000","last_active":"2024-09-15T23:24:03.116394+0000","last_peered":"2024-09-15T23:24:03.116394+0000","last_clean":"2024-09-15T23:24:03.116394+0000","last_became_active":"2024-09-15T23:21:52.529304+0000","last_became_peered":"2024-09-15T23:21:52.529304+0000","last_unstale":"2024-09-15T23:24:03.116394+0000","last_undegraded":"2024-09-15T23:24:03.116394+0000","last_fullsized":"2024-09-15T23:24:03.116394+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:18:37.758403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117052+0000","last_change":"2024-09-15T23:21:50.562114+0000","last_active":"2024-09-15T23:24:03.117052+0000","last_peered":"2024-09-15T23:24:03.117052+0000","last_clean":"2024-09-15T23:24:03.117052+0000","last_became_active":"2024-09-15T23:21:50.561772+0000","last_became_peered":"2024-09-15T23:21:50.561772+0000","last_unstale":"2024-09-15T23:24:03.117052+0000","last_undegraded":"2024-09-15T23:24:03.117052+0000","last_fullsized":"2024-09-15T23:24:03.117052+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:14.806646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110672+0000","last_change":"2024-09-15T23:21:59.681933+0000","last_active":"2024-09-15T23:24:03.110672+0000","last_peered":"2024-09-15T23:24:03.110672+0000","last_clean":"2024-09-15T23:24:03.110672+0000","last_became_active":"2024-09-15T23:21:59.681528+0000","last_became_peered":"2024-09-15T23:21:59.681528+0000","last_unstale":"2024-09-15T23:24:03.110672+0000","last_undegraded":"2024-09-15T23:24:03.110672+0000","last_fullsized":"2024-09-15T23:24:03.110672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:59:46.199540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194326+0000","last_change":"2024-09-15T23:21:52.529548+0000","last_active":"2024-09-15T23:24:03.194326+0000","last_peered":"2024-09-15T23:24:03.194326+0000","last_clean":"2024-09-15T23:24:03.194326+0000","last_became_active":"2024-09-15T23:21:52.529253+0000","last_became_peered":"2024-09-15T23:21:52.529253+0000","last_unstale":"2024-09-15T23:24:03.194326+0000","last_undegraded":"2024-09-15T23:24:03.194326+0000","last_fullsized":"2024-09-15T23:24:03.194326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:22.813395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111022+0000","last_change":"2024-09-15T23:21:50.767759+0000","last_active":"2024-09-15T23:24:03.111022+0000","last_peered":"2024-09-15T23:24:03.111022+0000","last_clean":"2024-09-15T23:24:03.111022+0000","last_became_active":"2024-09-15T23:21:50.767388+0000","last_became_peered":"2024-09-15T23:21:50.767388+0000","last_unstale":"2024-09-15T23:24:03.111022+0000","last_undegraded":"2024-09-15T23:24:03.111022+0000","last_fullsized":"2024-09-15T23:24:03.111022+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:05:36.713769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112512+0000","last_change":"2024-09-15T23:21:59.557951+0000","last_active":"2024-09-15T23:24:03.112512+0000","last_peered":"2024-09-15T23:24:03.112512+0000","last_clean":"2024-09-15T23:24:03.112512+0000","last_became_active":"2024-09-15T23:21:59.557490+0000","last_became_peered":"2024-09-15T23:21:59.557490+0000","last_unstale":"2024-09-15T23:24:03.112512+0000","last_undegraded":"2024-09-15T23:24:03.112512+0000","last_fullsized":"2024-09-15T23:24:03.112512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:13:17.147389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112859+0000","last_change":"2024-09-15T23:21:52.531638+0000","last_active":"2024-09-15T23:24:03.112859+0000","last_peered":"2024-09-15T23:24:03.112859+0000","last_clean":"2024-09-15T23:24:03.112859+0000","last_became_active":"2024-09-15T23:21:52.531301+0000","last_became_peered":"2024-09-15T23:21:52.531301+0000","last_unstale":"2024-09-15T23:24:03.112859+0000","last_undegraded":"2024-09-15T23:24:03.112859+0000","last_fullsized":"2024-09-15T23:24:03.112859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:36.436286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'18","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:18.545294+0000","last_change":"2024-09-15T23:21:50.508470+0000","last_active":"2024-09-15T23:24:18.545294+0000","last_peered":"2024-09-15T23:24:18.545294+0000","last_clean":"2024-09-15T23:24:18.545294+0000","last_became_active":"2024-09-15T23:21:50.508060+0000","last_became_peered":"2024-09-15T23:21:50.508060+0000","last_unstale":"2024-09-15T23:24:18.545294+0000","last_undegraded":"2024-09-15T23:24:18.545294+0000","last_fullsized":"2024-09-15T23:24:18.545294+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:59.542252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111992+0000","last_change":"2024-09-15T23:21:59.682019+0000","last_active":"2024-09-15T23:24:03.111992+0000","last_peered":"2024-09-15T23:24:03.111992+0000","last_clean":"2024-09-15T23:24:03.111992+0000","last_became_active":"2024-09-15T23:21:59.681467+0000","last_became_peered":"2024-09-15T23:21:59.681467+0000","last_unstale":"2024-09-15T23:24:03.111992+0000","last_undegraded":"2024-09-15T23:24:03.111992+0000","last_fullsized":"2024-09-15T23:24:03.111992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:13:27.650382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109766+0000","last_change":"2024-09-15T23:21:52.536345+0000","last_active":"2024-09-15T23:24:03.109766+0000","last_peered":"2024-09-15T23:24:03.109766+0000","last_clean":"2024-09-15T23:24:03.109766+0000","last_became_active":"2024-09-15T23:21:52.536027+0000","last_became_peered":"2024-09-15T23:21:52.536027+0000","last_unstale":"2024-09-15T23:24:03.109766+0000","last_undegraded":"2024-09-15T23:24:03.109766+0000","last_fullsized":"2024-09-15T23:24:03.109766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:16.435912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117716+0000","last_change":"2024-09-15T23:21:50.562945+0000","last_active":"2024-09-15T23:24:03.117716+0000","last_peered":"2024-09-15T23:24:03.117716+0000","last_clean":"2024-09-15T23:24:03.117716+0000","last_became_active":"2024-09-15T23:21:50.561632+0000","last_became_peered":"2024-09-15T23:21:50.561632+0000","last_unstale":"2024-09-15T23:24:03.117716+0000","last_undegraded":"2024-09-15T23:24:03.117716+0000","last_fullsized":"2024-09-15T23:24:03.117716+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:44.545489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115268+0000","last_change":"2024-09-15T23:21:59.564793+0000","last_active":"2024-09-15T23:24:03.115268+0000","last_peered":"2024-09-15T23:24:03.115268+0000","last_clean":"2024-09-15T23:24:03.115268+0000","last_became_active":"2024-09-15T23:21:59.564140+0000","last_became_peered":"2024-09-15T23:21:59.564140+0000","last_unstale":"2024-09-15T23:24:03.115268+0000","last_undegraded":"2024-09-15T23:24:03.115268+0000","last_fullsized":"2024-09-15T23:24:03.115268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:41.464962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113451+0000","last_change":"2024-09-15T23:21:52.533521+0000","last_active":"2024-09-15T23:24:03.113451+0000","last_peered":"2024-09-15T23:24:03.113451+0000","last_clean":"2024-09-15T23:24:03.113451+0000","last_became_active":"2024-09-15T23:21:52.532195+0000","last_became_peered":"2024-09-15T23:21:52.532195+0000","last_unstale":"2024-09-15T23:24:03.113451+0000","last_undegraded":"2024-09-15T23:24:03.113451+0000","last_fullsized":"2024-09-15T23:24:03.113451+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:14.229018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194243+0000","last_change":"2024-09-15T23:21:50.562893+0000","last_active":"2024-09-15T23:24:03.194243+0000","last_peered":"2024-09-15T23:24:03.194243+0000","last_clean":"2024-09-15T23:24:03.194243+0000","last_became_active":"2024-09-15T23:21:50.562516+0000","last_became_peered":"2024-09-15T23:21:50.562516+0000","last_unstale":"2024-09-15T23:24:03.194243+0000","last_undegraded":"2024-09-15T23:24:03.194243+0000","last_fullsized":"2024-09-15T23:24:03.194243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:47:47.442458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117856+0000","last_change":"2024-09-15T23:21:59.682091+0000","last_active":"2024-09-15T23:24:03.117856+0000","last_peered":"2024-09-15T23:24:03.117856+0000","last_clean":"2024-09-15T23:24:03.117856+0000","last_became_active":"2024-09-15T23:21:59.681574+0000","last_became_peered":"2024-09-15T23:21:59.681574+0000","last_unstale":"2024-09-15T23:24:03.117856+0000","last_undegraded":"2024-09-15T23:24:03.117856+0000","last_fullsized":"2024-09-15T23:24:03.117856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:04.991608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114426+0000","last_change":"2024-09-15T23:21:52.533811+0000","last_active":"2024-09-15T23:24:03.114426+0000","last_peered":"2024-09-15T23:24:03.114426+0000","last_clean":"2024-09-15T23:24:03.114426+0000","last_became_active":"2024-09-15T23:21:52.533487+0000","last_became_peered":"2024-09-15T23:21:52.533487+0000","last_unstale":"2024-09-15T23:24:03.114426+0000","last_undegraded":"2024-09-15T23:24:03.114426+0000","last_fullsized":"2024-09-15T23:24:03.114426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:44.193212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":159,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:30.295806+0000","last_change":"2024-09-15T23:21:50.510883+0000","last_active":"2024-09-15T23:24:30.295806+0000","last_peered":"2024-09-15T23:24:30.295806+0000","last_clean":"2024-09-15T23:24:30.295806+0000","last_became_active":"2024-09-15T23:21:50.510478+0000","last_became_peered":"2024-09-15T23:21:50.510478+0000","last_unstale":"2024-09-15T23:24:30.295806+0000","last_undegraded":"2024-09-15T23:24:30.295806+0000","last_fullsized":"2024-09-15T23:24:30.295806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:35.454885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109705+0000","last_change":"2024-09-15T23:21:59.561816+0000","last_active":"2024-09-15T23:24:03.109705+0000","last_peered":"2024-09-15T23:24:03.109705+0000","last_clean":"2024-09-15T23:24:03.109705+0000","last_became_active":"2024-09-15T23:21:59.561434+0000","last_became_peered":"2024-09-15T23:21:59.561434+0000","last_unstale":"2024-09-15T23:24:03.109705+0000","last_undegraded":"2024-09-15T23:24:03.109705+0000","last_fullsized":"2024-09-15T23:24:03.109705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:59.945802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117955+0000","last_change":"2024-09-15T23:23:02.277204+0000","last_active":"2024-09-15T23:24:03.117955+0000","last_peered":"2024-09-15T23:24:03.117955+0000","last_clean":"2024-09-15T23:24:03.117955+0000","last_became_active":"2024-09-15T23:23:02.276770+0000","last_became_peered":"2024-09-15T23:23:02.276770+0000","last_unstale":"2024-09-15T23:24:03.117955+0000","last_undegraded":"2024-09-15T23:24:03.117955+0000","last_fullsized":"2024-09-15T23:24:03.117955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:11.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":160,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:29.158653+0000","last_change":"2024-09-15T23:21:50.561684+0000","last_active":"2024-09-15T23:24:29.158653+0000","last_peered":"2024-09-15T23:24:29.158653+0000","last_clean":"2024-09-15T23:24:29.158653+0000","last_became_active":"2024-09-15T23:21:50.561335+0000","last_became_peered":"2024-09-15T23:21:50.561335+0000","last_unstale":"2024-09-15T23:24:29.158653+0000","last_undegraded":"2024-09-15T23:24:29.158653+0000","last_fullsized":"2024-09-15T23:24:29.158653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:23.897250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109863+0000","last_change":"2024-09-15T23:21:59.681858+0000","last_active":"2024-09-15T23:24:03.109863+0000","last_peered":"2024-09-15T23:24:03.109863+0000","last_clean":"2024-09-15T23:24:03.109863+0000","last_became_active":"2024-09-15T23:21:59.681441+0000","last_became_peered":"2024-09-15T23:21:59.681441+0000","last_unstale":"2024-09-15T23:24:03.109863+0000","last_undegraded":"2024-09-15T23:24:03.109863+0000","last_fullsized":"2024-09-15T23:24:03.109863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:59:28.513719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109320+0000","last_change":"2024-09-15T23:21:52.536141+0000","last_active":"2024-09-15T23:24:03.109320+0000","last_peered":"2024-09-15T23:24:03.109320+0000","last_clean":"2024-09-15T23:24:03.109320+0000","last_became_active":"2024-09-15T23:21:52.535832+0000","last_became_peered":"2024-09-15T23:21:52.535832+0000","last_unstale":"2024-09-15T23:24:03.109320+0000","last_undegraded":"2024-09-15T23:24:03.109320+0000","last_fullsized":"2024-09-15T23:24:03.109320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:21:44.341412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.123671+0000","last_change":"2024-09-15T23:24:03.123671+0000","last_active":"2024-09-15T23:24:03.123671+0000","last_peered":"2024-09-15T23:24:03.123671+0000","last_clean":"2024-09-15T23:24:03.123671+0000","last_became_active":"2024-09-15T23:24:03.119585+0000","last_became_peered":"2024-09-15T23:24:03.119585+0000","last_unstale":"2024-09-15T23:24:03.123671+0000","last_undegraded":"2024-09-15T23:24:03.123671+0000","last_fullsized":"2024-09-15T23:24:03.123671+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:30.734156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118359+0000","last_change":"2024-09-15T23:21:59.682118+0000","last_active":"2024-09-15T23:24:03.118359+0000","last_peered":"2024-09-15T23:24:03.118359+0000","last_clean":"2024-09-15T23:24:03.118359+0000","last_became_active":"2024-09-15T23:21:59.681500+0000","last_became_peered":"2024-09-15T23:21:59.681500+0000","last_unstale":"2024-09-15T23:24:03.118359+0000","last_undegraded":"2024-09-15T23:24:03.118359+0000","last_fullsized":"2024-09-15T23:24:03.118359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:55:58.126308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112298+0000","last_change":"2024-09-15T23:21:52.540458+0000","last_active":"2024-09-15T23:24:03.112298+0000","last_peered":"2024-09-15T23:24:03.112298+0000","last_clean":"2024-09-15T23:24:03.112298+0000","last_became_active":"2024-09-15T23:21:52.540112+0000","last_became_peered":"2024-09-15T23:21:52.540112+0000","last_unstale":"2024-09-15T23:24:03.112298+0000","last_undegraded":"2024-09-15T23:24:03.112298+0000","last_fullsized":"2024-09-15T23:24:03.112298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:15.240785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111097+0000","last_change":"2024-09-15T23:21:50.513124+0000","last_active":"2024-09-15T23:24:03.111097+0000","last_peered":"2024-09-15T23:24:03.111097+0000","last_clean":"2024-09-15T23:24:03.111097+0000","last_became_active":"2024-09-15T23:21:50.512705+0000","last_became_peered":"2024-09-15T23:21:50.512705+0000","last_unstale":"2024-09-15T23:24:03.111097+0000","last_undegraded":"2024-09-15T23:24:03.111097+0000","last_fullsized":"2024-09-15T23:24:03.111097+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:36:44.417937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":354,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113701+0000","last_change":"2024-09-15T23:20:40.656652+0000","last_active":"2024-09-15T23:24:03.113701+0000","last_peered":"2024-09-15T23:24:03.113701+0000","last_clean":"2024-09-15T23:24:03.113701+0000","last_became_active":"2024-09-15T23:20:40.656266+0000","last_became_peered":"2024-09-15T23:20:40.656266+0000","last_unstale":"2024-09-15T23:24:03.113701+0000","last_undegraded":"2024-09-15T23:24:03.113701+0000","last_fullsized":"2024-09-15T23:24:03.113701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:17:15.186850+0000","last_clean_scrub_stamp":"2024-09-15T23:17:15.186850+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:42.317006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113728+0000","last_change":"2024-09-15T23:21:59.563777+0000","last_active":"2024-09-15T23:24:03.113728+0000","last_peered":"2024-09-15T23:24:03.113728+0000","last_clean":"2024-09-15T23:24:03.113728+0000","last_became_active":"2024-09-15T23:21:59.563320+0000","last_became_peered":"2024-09-15T23:21:59.563320+0000","last_unstale":"2024-09-15T23:24:03.113728+0000","last_undegraded":"2024-09-15T23:24:03.113728+0000","last_fullsized":"2024-09-15T23:24:03.113728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:25:07.791923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114145+0000","last_change":"2024-09-15T23:21:52.539842+0000","last_active":"2024-09-15T23:24:03.114145+0000","last_peered":"2024-09-15T23:24:03.114145+0000","last_clean":"2024-09-15T23:24:03.114145+0000","last_became_active":"2024-09-15T23:21:52.538683+0000","last_became_peered":"2024-09-15T23:21:52.538683+0000","last_unstale":"2024-09-15T23:24:03.114145+0000","last_undegraded":"2024-09-15T23:24:03.114145+0000","last_fullsized":"2024-09-15T23:24:03.114145+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:43.375136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":159,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:28.293643+0000","last_change":"2024-09-15T23:21:50.767601+0000","last_active":"2024-09-15T23:24:28.293643+0000","last_peered":"2024-09-15T23:24:28.293643+0000","last_clean":"2024-09-15T23:24:28.293643+0000","last_became_active":"2024-09-15T23:21:50.767149+0000","last_became_peered":"2024-09-15T23:21:50.767149+0000","last_unstale":"2024-09-15T23:24:28.293643+0000","last_undegraded":"2024-09-15T23:24:28.293643+0000","last_fullsized":"2024-09-15T23:24:28.293643+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:53:23.636551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117168+0000","last_change":"2024-09-15T23:21:59.563069+0000","last_active":"2024-09-15T23:24:03.117168+0000","last_peered":"2024-09-15T23:24:03.117168+0000","last_clean":"2024-09-15T23:24:03.117168+0000","last_became_active":"2024-09-15T23:21:59.562658+0000","last_became_peered":"2024-09-15T23:21:59.562658+0000","last_unstale":"2024-09-15T23:24:03.117168+0000","last_undegraded":"2024-09-15T23:24:03.117168+0000","last_fullsized":"2024-09-15T23:24:03.117168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:23.067198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113535+0000","last_change":"2024-09-15T23:21:52.535289+0000","last_active":"2024-09-15T23:24:03.113535+0000","last_peered":"2024-09-15T23:24:03.113535+0000","last_clean":"2024-09-15T23:24:03.113535+0000","last_became_active":"2024-09-15T23:21:52.534885+0000","last_became_peered":"2024-09-15T23:21:52.534885+0000","last_unstale":"2024-09-15T23:24:03.113535+0000","last_undegraded":"2024-09-15T23:24:03.113535+0000","last_fullsized":"2024-09-15T23:24:03.113535+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:42.293786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118612+0000","last_change":"2024-09-15T23:21:50.513246+0000","last_active":"2024-09-15T23:24:03.118612+0000","last_peered":"2024-09-15T23:24:03.118612+0000","last_clean":"2024-09-15T23:24:03.118612+0000","last_became_active":"2024-09-15T23:21:50.512740+0000","last_became_peered":"2024-09-15T23:21:50.512740+0000","last_unstale":"2024-09-15T23:24:03.118612+0000","last_undegraded":"2024-09-15T23:24:03.118612+0000","last_fullsized":"2024-09-15T23:24:03.118612+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:59.201970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193756+0000","last_change":"2024-09-15T23:21:59.568565+0000","last_active":"2024-09-15T23:24:03.193756+0000","last_peered":"2024-09-15T23:24:03.193756+0000","last_clean":"2024-09-15T23:24:03.193756+0000","last_became_active":"2024-09-15T23:21:59.567789+0000","last_became_peered":"2024-09-15T23:21:59.567789+0000","last_unstale":"2024-09-15T23:24:03.193756+0000","last_undegraded":"2024-09-15T23:24:03.193756+0000","last_fullsized":"2024-09-15T23:24:03.193756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:23:02.106971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114106+0000","last_change":"2024-09-15T23:21:52.533727+0000","last_active":"2024-09-15T23:24:03.114106+0000","last_peered":"2024-09-15T23:24:03.114106+0000","last_clean":"2024-09-15T23:24:03.114106+0000","last_became_active":"2024-09-15T23:21:52.532350+0000","last_became_peered":"2024-09-15T23:21:52.532350+0000","last_unstale":"2024-09-15T23:24:03.114106+0000","last_undegraded":"2024-09-15T23:24:03.114106+0000","last_fullsized":"2024-09-15T23:24:03.114106+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:03.624837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.124878+0000","last_change":"2024-09-15T23:24:03.124430+0000","last_active":"2024-09-15T23:24:03.124878+0000","last_peered":"2024-09-15T23:24:03.124878+0000","last_clean":"2024-09-15T23:24:03.124878+0000","last_became_active":"2024-09-15T23:24:03.119042+0000","last_became_peered":"2024-09-15T23:24:03.119042+0000","last_unstale":"2024-09-15T23:24:03.124878+0000","last_undegraded":"2024-09-15T23:24:03.124878+0000","last_fullsized":"2024-09-15T23:24:03.124878+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:13:52.065543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117798+0000","last_change":"2024-09-15T23:21:59.566429+0000","last_active":"2024-09-15T23:24:03.117798+0000","last_peered":"2024-09-15T23:24:03.117798+0000","last_clean":"2024-09-15T23:24:03.117798+0000","last_became_active":"2024-09-15T23:21:59.565835+0000","last_became_peered":"2024-09-15T23:21:59.565835+0000","last_unstale":"2024-09-15T23:24:03.117798+0000","last_undegraded":"2024-09-15T23:24:03.117798+0000","last_fullsized":"2024-09-15T23:24:03.117798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:38.822144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116276+0000","last_change":"2024-09-15T23:21:52.539544+0000","last_active":"2024-09-15T23:24:03.116276+0000","last_peered":"2024-09-15T23:24:03.116276+0000","last_clean":"2024-09-15T23:24:03.116276+0000","last_became_active":"2024-09-15T23:21:52.539203+0000","last_became_peered":"2024-09-15T23:21:52.539203+0000","last_unstale":"2024-09-15T23:24:03.116276+0000","last_undegraded":"2024-09-15T23:24:03.116276+0000","last_fullsized":"2024-09-15T23:24:03.116276+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:56.193965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:07.046123+0000","last_change":"2024-09-15T23:21:50.513543+0000","last_active":"2024-09-15T23:24:07.046123+0000","last_peered":"2024-09-15T23:24:07.046123+0000","last_clean":"2024-09-15T23:24:07.046123+0000","last_became_active":"2024-09-15T23:21:50.512812+0000","last_became_peered":"2024-09-15T23:21:50.512812+0000","last_unstale":"2024-09-15T23:24:07.046123+0000","last_undegraded":"2024-09-15T23:24:07.046123+0000","last_fullsized":"2024-09-15T23:24:07.046123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:50.954096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117689+0000","last_change":"2024-09-15T23:21:59.570163+0000","last_active":"2024-09-15T23:24:03.117689+0000","last_peered":"2024-09-15T23:24:03.117689+0000","last_clean":"2024-09-15T23:24:03.117689+0000","last_became_active":"2024-09-15T23:21:59.569805+0000","last_became_peered":"2024-09-15T23:21:59.569805+0000","last_unstale":"2024-09-15T23:24:03.117689+0000","last_undegraded":"2024-09-15T23:24:03.117689+0000","last_fullsized":"2024-09-15T23:24:03.117689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:06:28.530817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117877+0000","last_change":"2024-09-15T23:21:52.537850+0000","last_active":"2024-09-15T23:24:03.117877+0000","last_peered":"2024-09-15T23:24:03.117877+0000","last_clean":"2024-09-15T23:24:03.117877+0000","last_became_active":"2024-09-15T23:21:52.536579+0000","last_became_peered":"2024-09-15T23:21:52.536579+0000","last_unstale":"2024-09-15T23:24:03.117877+0000","last_undegraded":"2024-09-15T23:24:03.117877+0000","last_fullsized":"2024-09-15T23:24:03.117877+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:06.353615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118588+0000","last_change":"2024-09-15T23:21:50.562221+0000","last_active":"2024-09-15T23:24:03.118588+0000","last_peered":"2024-09-15T23:24:03.118588+0000","last_clean":"2024-09-15T23:24:03.118588+0000","last_became_active":"2024-09-15T23:21:50.561788+0000","last_became_peered":"2024-09-15T23:21:50.561788+0000","last_unstale":"2024-09-15T23:24:03.118588+0000","last_undegraded":"2024-09-15T23:24:03.118588+0000","last_fullsized":"2024-09-15T23:24:03.118588+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:26.853812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109849+0000","last_change":"2024-09-15T23:23:02.383003+0000","last_active":"2024-09-15T23:24:03.109849+0000","last_peered":"2024-09-15T23:24:03.109849+0000","last_clean":"2024-09-15T23:24:03.109849+0000","last_became_active":"2024-09-15T23:23:02.382629+0000","last_became_peered":"2024-09-15T23:23:02.382629+0000","last_unstale":"2024-09-15T23:24:03.109849+0000","last_undegraded":"2024-09-15T23:24:03.109849+0000","last_fullsized":"2024-09-15T23:24:03.109849+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:50.114689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113567+0000","last_change":"2024-09-15T23:21:52.539566+0000","last_active":"2024-09-15T23:24:03.113567+0000","last_peered":"2024-09-15T23:24:03.113567+0000","last_clean":"2024-09-15T23:24:03.113567+0000","last_became_active":"2024-09-15T23:21:52.539219+0000","last_became_peered":"2024-09-15T23:21:52.539219+0000","last_unstale":"2024-09-15T23:24:03.113567+0000","last_undegraded":"2024-09-15T23:24:03.113567+0000","last_fullsized":"2024-09-15T23:24:03.113567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:41.266308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:29.159267+0000","last_change":"2024-09-15T23:24:03.127667+0000","last_active":"2024-09-15T23:24:29.159267+0000","last_peered":"2024-09-15T23:24:29.159267+0000","last_clean":"2024-09-15T23:24:29.159267+0000","last_became_active":"2024-09-15T23:24:03.121613+0000","last_became_peered":"2024-09-15T23:24:03.121613+0000","last_unstale":"2024-09-15T23:24:29.159267+0000","last_undegraded":"2024-09-15T23:24:29.159267+0000","last_fullsized":"2024-09-15T23:24:29.159267+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:54:24.166895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1515,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1122,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,10],"acting":[4,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194642+0000","last_change":"2024-09-15T23:21:59.568087+0000","last_active":"2024-09-15T23:24:03.194642+0000","last_peered":"2024-09-15T23:24:03.194642+0000","last_clean":"2024-09-15T23:24:03.194642+0000","last_became_active":"2024-09-15T23:21:59.567693+0000","last_became_peered":"2024-09-15T23:21:59.567693+0000","last_unstale":"2024-09-15T23:24:03.194642+0000","last_undegraded":"2024-09-15T23:24:03.194642+0000","last_fullsized":"2024-09-15T23:24:03.194642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:04.231873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114396+0000","last_change":"2024-09-15T23:21:52.531090+0000","last_active":"2024-09-15T23:24:03.114396+0000","last_peered":"2024-09-15T23:24:03.114396+0000","last_clean":"2024-09-15T23:24:03.114396+0000","last_became_active":"2024-09-15T23:21:52.530738+0000","last_became_peered":"2024-09-15T23:21:52.530738+0000","last_unstale":"2024-09-15T23:24:03.114396+0000","last_undegraded":"2024-09-15T23:24:03.114396+0000","last_fullsized":"2024-09-15T23:24:03.114396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:26:35.325128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117423+0000","last_change":"2024-09-15T23:21:50.514100+0000","last_active":"2024-09-15T23:24:03.117423+0000","last_peered":"2024-09-15T23:24:03.117423+0000","last_clean":"2024-09-15T23:24:03.117423+0000","last_became_active":"2024-09-15T23:21:50.513629+0000","last_became_peered":"2024-09-15T23:21:50.513629+0000","last_unstale":"2024-09-15T23:24:03.117423+0000","last_undegraded":"2024-09-15T23:24:03.117423+0000","last_fullsized":"2024-09-15T23:24:03.117423+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:52.187544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115275+0000","last_change":"2024-09-15T23:21:59.562804+0000","last_active":"2024-09-15T23:24:03.115275+0000","last_peered":"2024-09-15T23:24:03.115275+0000","last_clean":"2024-09-15T23:24:03.115275+0000","last_became_active":"2024-09-15T23:21:59.562368+0000","last_became_peered":"2024-09-15T23:21:59.562368+0000","last_unstale":"2024-09-15T23:24:03.115275+0000","last_undegraded":"2024-09-15T23:24:03.115275+0000","last_fullsized":"2024-09-15T23:24:03.115275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:21.137333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109909+0000","last_change":"2024-09-15T23:21:52.530562+0000","last_active":"2024-09-15T23:24:03.109909+0000","last_peered":"2024-09-15T23:24:03.109909+0000","last_clean":"2024-09-15T23:24:03.109909+0000","last_became_active":"2024-09-15T23:21:52.530280+0000","last_became_peered":"2024-09-15T23:21:52.530280+0000","last_unstale":"2024-09-15T23:24:03.109909+0000","last_undegraded":"2024-09-15T23:24:03.109909+0000","last_fullsized":"2024-09-15T23:24:03.109909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:36:26.835651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113202+0000","last_change":"2024-09-15T23:21:50.513179+0000","last_active":"2024-09-15T23:24:03.113202+0000","last_peered":"2024-09-15T23:24:03.113202+0000","last_clean":"2024-09-15T23:24:03.113202+0000","last_became_active":"2024-09-15T23:21:50.512790+0000","last_became_peered":"2024-09-15T23:21:50.512790+0000","last_unstale":"2024-09-15T23:24:03.113202+0000","last_undegraded":"2024-09-15T23:24:03.113202+0000","last_fullsized":"2024-09-15T23:24:03.113202+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:56.408198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113316+0000","last_change":"2024-09-15T23:23:02.382964+0000","last_active":"2024-09-15T23:24:03.113316+0000","last_peered":"2024-09-15T23:24:03.113316+0000","last_clean":"2024-09-15T23:24:03.113316+0000","last_became_active":"2024-09-15T23:23:02.382556+0000","last_became_peered":"2024-09-15T23:23:02.382556+0000","last_unstale":"2024-09-15T23:24:03.113316+0000","last_undegraded":"2024-09-15T23:24:03.113316+0000","last_fullsized":"2024-09-15T23:24:03.113316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:43:49.211591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117484+0000","last_change":"2024-09-15T23:21:52.530970+0000","last_active":"2024-09-15T23:24:03.117484+0000","last_peered":"2024-09-15T23:24:03.117484+0000","last_clean":"2024-09-15T23:24:03.117484+0000","last_became_active":"2024-09-15T23:21:52.530636+0000","last_became_peered":"2024-09-15T23:21:52.530636+0000","last_unstale":"2024-09-15T23:24:03.117484+0000","last_undegraded":"2024-09-15T23:24:03.117484+0000","last_fullsized":"2024-09-15T23:24:03.117484+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:33.374658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'41","reported_seq":340,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:30.160742+0000","last_change":"2024-09-15T23:24:03.131527+0000","last_active":"2024-09-15T23:24:30.160742+0000","last_peered":"2024-09-15T23:24:30.160742+0000","last_clean":"2024-09-15T23:24:30.160742+0000","last_became_active":"2024-09-15T23:24:03.119053+0000","last_became_peered":"2024-09-15T23:24:03.119053+0000","last_unstale":"2024-09-15T23:24:30.160742+0000","last_undegraded":"2024-09-15T23:24:30.160742+0000","last_fullsized":"2024-09-15T23:24:30.160742+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:02:21.870435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70919,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":189,"num_write":43,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":62325,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113841+0000","last_change":"2024-09-15T23:23:02.277457+0000","last_active":"2024-09-15T23:24:03.113841+0000","last_peered":"2024-09-15T23:24:03.113841+0000","last_clean":"2024-09-15T23:24:03.113841+0000","last_became_active":"2024-09-15T23:23:02.276960+0000","last_became_peered":"2024-09-15T23:23:02.276960+0000","last_unstale":"2024-09-15T23:24:03.113841+0000","last_undegraded":"2024-09-15T23:24:03.113841+0000","last_fullsized":"2024-09-15T23:24:03.113841+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:15:28.823804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117534+0000","last_change":"2024-09-15T23:21:52.530541+0000","last_active":"2024-09-15T23:24:03.117534+0000","last_peered":"2024-09-15T23:24:03.117534+0000","last_clean":"2024-09-15T23:24:03.117534+0000","last_became_active":"2024-09-15T23:21:52.530217+0000","last_became_peered":"2024-09-15T23:21:52.530217+0000","last_unstale":"2024-09-15T23:24:03.117534+0000","last_undegraded":"2024-09-15T23:24:03.117534+0000","last_fullsized":"2024-09-15T23:24:03.117534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:14.864563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118500+0000","last_change":"2024-09-15T23:21:50.562203+0000","last_active":"2024-09-15T23:24:03.118500+0000","last_peered":"2024-09-15T23:24:03.118500+0000","last_clean":"2024-09-15T23:24:03.118500+0000","last_became_active":"2024-09-15T23:21:50.561827+0000","last_became_peered":"2024-09-15T23:21:50.561827+0000","last_unstale":"2024-09-15T23:24:03.118500+0000","last_undegraded":"2024-09-15T23:24:03.118500+0000","last_fullsized":"2024-09-15T23:24:03.118500+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:22.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111024+0000","last_change":"2024-09-15T23:21:59.561013+0000","last_active":"2024-09-15T23:24:03.111024+0000","last_peered":"2024-09-15T23:24:03.111024+0000","last_clean":"2024-09-15T23:24:03.111024+0000","last_became_active":"2024-09-15T23:21:59.560665+0000","last_became_peered":"2024-09-15T23:21:59.560665+0000","last_unstale":"2024-09-15T23:24:03.111024+0000","last_undegraded":"2024-09-15T23:24:03.111024+0000","last_fullsized":"2024-09-15T23:24:03.111024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:00.538102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194109+0000","last_change":"2024-09-15T23:21:52.529922+0000","last_active":"2024-09-15T23:24:03.194109+0000","last_peered":"2024-09-15T23:24:03.194109+0000","last_clean":"2024-09-15T23:24:03.194109+0000","last_became_active":"2024-09-15T23:21:52.529179+0000","last_became_peered":"2024-09-15T23:21:52.529179+0000","last_unstale":"2024-09-15T23:24:03.194109+0000","last_undegraded":"2024-09-15T23:24:03.194109+0000","last_fullsized":"2024-09-15T23:24:03.194109+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:15.077658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113481+0000","last_change":"2024-09-15T23:21:50.514420+0000","last_active":"2024-09-15T23:24:03.113481+0000","last_peered":"2024-09-15T23:24:03.113481+0000","last_clean":"2024-09-15T23:24:03.113481+0000","last_became_active":"2024-09-15T23:21:50.514007+0000","last_became_peered":"2024-09-15T23:21:50.514007+0000","last_unstale":"2024-09-15T23:24:03.113481+0000","last_undegraded":"2024-09-15T23:24:03.113481+0000","last_fullsized":"2024-09-15T23:24:03.113481+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:36.845674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193392+0000","last_change":"2024-09-15T23:21:59.565978+0000","last_active":"2024-09-15T23:24:03.193392+0000","last_peered":"2024-09-15T23:24:03.193392+0000","last_clean":"2024-09-15T23:24:03.193392+0000","last_became_active":"2024-09-15T23:21:59.565604+0000","last_became_peered":"2024-09-15T23:21:59.565604+0000","last_unstale":"2024-09-15T23:24:03.193392+0000","last_undegraded":"2024-09-15T23:24:03.193392+0000","last_fullsized":"2024-09-15T23:24:03.193392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:47.089504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118064+0000","last_change":"2024-09-15T23:21:52.543511+0000","last_active":"2024-09-15T23:24:03.118064+0000","last_peered":"2024-09-15T23:24:03.118064+0000","last_clean":"2024-09-15T23:24:03.118064+0000","last_became_active":"2024-09-15T23:21:52.543169+0000","last_became_peered":"2024-09-15T23:21:52.543169+0000","last_unstale":"2024-09-15T23:24:03.118064+0000","last_undegraded":"2024-09-15T23:24:03.118064+0000","last_fullsized":"2024-09-15T23:24:03.118064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:08:05.696772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":272,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:30.295930+0000","last_change":"2024-09-15T23:21:50.518425+0000","last_active":"2024-09-15T23:24:30.295930+0000","last_peered":"2024-09-15T23:24:30.295930+0000","last_clean":"2024-09-15T23:24:30.295930+0000","last_became_active":"2024-09-15T23:21:50.518052+0000","last_became_peered":"2024-09-15T23:21:50.518052+0000","last_unstale":"2024-09-15T23:24:30.295930+0000","last_undegraded":"2024-09-15T23:24:30.295930+0000","last_fullsized":"2024-09-15T23:24:30.295930+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:02.243869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":228,"num_read_kb":228,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112035+0000","last_change":"2024-09-15T23:21:59.570310+0000","last_active":"2024-09-15T23:24:03.112035+0000","last_peered":"2024-09-15T23:24:03.112035+0000","last_clean":"2024-09-15T23:24:03.112035+0000","last_became_active":"2024-09-15T23:21:59.569909+0000","last_became_peered":"2024-09-15T23:21:59.569909+0000","last_unstale":"2024-09-15T23:24:03.112035+0000","last_undegraded":"2024-09-15T23:24:03.112035+0000","last_fullsized":"2024-09-15T23:24:03.112035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:20:27.515537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194191+0000","last_change":"2024-09-15T23:21:59.568208+0000","last_active":"2024-09-15T23:24:03.194191+0000","last_peered":"2024-09-15T23:24:03.194191+0000","last_clean":"2024-09-15T23:24:03.194191+0000","last_became_active":"2024-09-15T23:21:59.567858+0000","last_became_peered":"2024-09-15T23:21:59.567858+0000","last_unstale":"2024-09-15T23:24:03.194191+0000","last_undegraded":"2024-09-15T23:24:03.194191+0000","last_fullsized":"2024-09-15T23:24:03.194191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:34:42.895247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193532+0000","last_change":"2024-09-15T23:21:52.532679+0000","last_active":"2024-09-15T23:24:03.193532+0000","last_peered":"2024-09-15T23:24:03.193532+0000","last_clean":"2024-09-15T23:24:03.193532+0000","last_became_active":"2024-09-15T23:21:52.532345+0000","last_became_peered":"2024-09-15T23:21:52.532345+0000","last_unstale":"2024-09-15T23:24:03.193532+0000","last_undegraded":"2024-09-15T23:24:03.193532+0000","last_fullsized":"2024-09-15T23:24:03.193532+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:45.943653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118135+0000","last_change":"2024-09-15T23:21:50.510836+0000","last_active":"2024-09-15T23:24:03.118135+0000","last_peered":"2024-09-15T23:24:03.118135+0000","last_clean":"2024-09-15T23:24:03.118135+0000","last_became_active":"2024-09-15T23:21:50.510514+0000","last_became_peered":"2024-09-15T23:21:50.510514+0000","last_unstale":"2024-09-15T23:24:03.118135+0000","last_undegraded":"2024-09-15T23:24:03.118135+0000","last_fullsized":"2024-09-15T23:24:03.118135+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:06.404612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116423+0000","last_change":"2024-09-15T23:23:02.383436+0000","last_active":"2024-09-15T23:24:03.116423+0000","last_peered":"2024-09-15T23:24:03.116423+0000","last_clean":"2024-09-15T23:24:03.116423+0000","last_became_active":"2024-09-15T23:23:02.382982+0000","last_became_peered":"2024-09-15T23:23:02.382982+0000","last_unstale":"2024-09-15T23:24:03.116423+0000","last_undegraded":"2024-09-15T23:24:03.116423+0000","last_fullsized":"2024-09-15T23:24:03.116423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:55:14.076604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111923+0000","last_change":"2024-09-15T23:21:52.526766+0000","last_active":"2024-09-15T23:24:03.111923+0000","last_peered":"2024-09-15T23:24:03.111923+0000","last_clean":"2024-09-15T23:24:03.111923+0000","last_became_active":"2024-09-15T23:21:52.526446+0000","last_became_peered":"2024-09-15T23:21:52.526446+0000","last_unstale":"2024-09-15T23:24:03.111923+0000","last_undegraded":"2024-09-15T23:24:03.111923+0000","last_fullsized":"2024-09-15T23:24:03.111923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:21:37.004730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109384+0000","last_change":"2024-09-15T23:21:50.512928+0000","last_active":"2024-09-15T23:24:03.109384+0000","last_peered":"2024-09-15T23:24:03.109384+0000","last_clean":"2024-09-15T23:24:03.109384+0000","last_became_active":"2024-09-15T23:21:50.512558+0000","last_became_peered":"2024-09-15T23:21:50.512558+0000","last_unstale":"2024-09-15T23:24:03.109384+0000","last_undegraded":"2024-09-15T23:24:03.109384+0000","last_fullsized":"2024-09-15T23:24:03.109384+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:22:08.653084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114309+0000","last_change":"2024-09-15T23:21:50.516920+0000","last_active":"2024-09-15T23:24:03.114309+0000","last_peered":"2024-09-15T23:24:03.114309+0000","last_clean":"2024-09-15T23:24:03.114309+0000","last_became_active":"2024-09-15T23:21:50.516573+0000","last_became_peered":"2024-09-15T23:21:50.516573+0000","last_unstale":"2024-09-15T23:24:03.114309+0000","last_undegraded":"2024-09-15T23:24:03.114309+0000","last_fullsized":"2024-09-15T23:24:03.114309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.367903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117266+0000","last_change":"2024-09-15T23:21:52.536696+0000","last_active":"2024-09-15T23:24:03.117266+0000","last_peered":"2024-09-15T23:24:03.117266+0000","last_clean":"2024-09-15T23:24:03.117266+0000","last_became_active":"2024-09-15T23:21:52.536338+0000","last_became_peered":"2024-09-15T23:21:52.536338+0000","last_unstale":"2024-09-15T23:24:03.117266+0000","last_undegraded":"2024-09-15T23:24:03.117266+0000","last_fullsized":"2024-09-15T23:24:03.117266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:05.152749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112910+0000","last_change":"2024-09-15T23:21:59.562968+0000","last_active":"2024-09-15T23:24:03.112910+0000","last_peered":"2024-09-15T23:24:03.112910+0000","last_clean":"2024-09-15T23:24:03.112910+0000","last_became_active":"2024-09-15T23:21:59.562579+0000","last_became_peered":"2024-09-15T23:21:59.562579+0000","last_unstale":"2024-09-15T23:24:03.112910+0000","last_undegraded":"2024-09-15T23:24:03.112910+0000","last_fullsized":"2024-09-15T23:24:03.112910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:02.853057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":162,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:30.297312+0000","last_change":"2024-09-15T23:21:50.513944+0000","last_active":"2024-09-15T23:24:30.297312+0000","last_peered":"2024-09-15T23:24:30.297312+0000","last_clean":"2024-09-15T23:24:30.297312+0000","last_became_active":"2024-09-15T23:21:50.513484+0000","last_became_peered":"2024-09-15T23:21:50.513484+0000","last_unstale":"2024-09-15T23:24:30.297312+0000","last_undegraded":"2024-09-15T23:24:30.297312+0000","last_fullsized":"2024-09-15T23:24:30.297312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:49.483230+0000","last_clean_scrub_stamp":"2024-09-15T23:21:49.483230+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:35.719344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.118061+0000","last_change":"2024-09-15T23:21:52.539467+0000","last_active":"2024-09-15T23:24:03.118061+0000","last_peered":"2024-09-15T23:24:03.118061+0000","last_clean":"2024-09-15T23:24:03.118061+0000","last_became_active":"2024-09-15T23:21:52.538998+0000","last_became_peered":"2024-09-15T23:21:52.538998+0000","last_unstale":"2024-09-15T23:24:03.118061+0000","last_undegraded":"2024-09-15T23:24:03.118061+0000","last_fullsized":"2024-09-15T23:24:03.118061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:22.213857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113057+0000","last_change":"2024-09-15T23:21:59.564842+0000","last_active":"2024-09-15T23:24:03.113057+0000","last_peered":"2024-09-15T23:24:03.113057+0000","last_clean":"2024-09-15T23:24:03.113057+0000","last_became_active":"2024-09-15T23:21:59.564408+0000","last_became_peered":"2024-09-15T23:21:59.564408+0000","last_unstale":"2024-09-15T23:24:03.113057+0000","last_undegraded":"2024-09-15T23:24:03.113057+0000","last_fullsized":"2024-09-15T23:24:03.113057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:58.542828+0000","last_clean_scrub_stamp":"2024-09-15T23:21:58.542828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:17.909020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115745+0000","last_change":"2024-09-15T23:23:02.149420+0000","last_active":"2024-09-15T23:24:03.115745+0000","last_peered":"2024-09-15T23:24:03.115745+0000","last_clean":"2024-09-15T23:24:03.115745+0000","last_became_active":"2024-09-15T23:23:02.149003+0000","last_became_peered":"2024-09-15T23:23:02.149003+0000","last_unstale":"2024-09-15T23:24:03.115745+0000","last_undegraded":"2024-09-15T23:24:03.115745+0000","last_fullsized":"2024-09-15T23:24:03.115745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:38:45.490322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.532769+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532426+0000","last_became_peered":"2024-09-15T23:21:52.532426+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:09.519001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112646+0000","last_change":"2024-09-15T23:21:52.533148+0000","last_active":"2024-09-15T23:24:03.112646+0000","last_peered":"2024-09-15T23:24:03.112646+0000","last_clean":"2024-09-15T23:24:03.112646+0000","last_became_active":"2024-09-15T23:21:52.532823+0000","last_became_peered":"2024-09-15T23:21:52.532823+0000","last_unstale":"2024-09-15T23:24:03.112646+0000","last_undegraded":"2024-09-15T23:24:03.112646+0000","last_fullsized":"2024-09-15T23:24:03.112646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:01.833945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193803+0000","last_change":"2024-09-15T23:21:52.536790+0000","last_active":"2024-09-15T23:24:03.193803+0000","last_peered":"2024-09-15T23:24:03.193803+0000","last_clean":"2024-09-15T23:24:03.193803+0000","last_became_active":"2024-09-15T23:21:52.536515+0000","last_became_peered":"2024-09-15T23:21:52.536515+0000","last_unstale":"2024-09-15T23:24:03.193803+0000","last_undegraded":"2024-09-15T23:24:03.193803+0000","last_fullsized":"2024-09-15T23:24:03.193803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:37.207759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109800+0000","last_change":"2024-09-15T23:23:02.382842+0000","last_active":"2024-09-15T23:24:03.109800+0000","last_peered":"2024-09-15T23:24:03.109800+0000","last_clean":"2024-09-15T23:24:03.109800+0000","last_became_active":"2024-09-15T23:23:02.382455+0000","last_became_peered":"2024-09-15T23:23:02.382455+0000","last_unstale":"2024-09-15T23:24:03.109800+0000","last_undegraded":"2024-09-15T23:24:03.109800+0000","last_fullsized":"2024-09-15T23:24:03.109800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:23.617641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115872+0000","last_change":"2024-09-15T23:21:52.540628+0000","last_active":"2024-09-15T23:24:03.115872+0000","last_peered":"2024-09-15T23:24:03.115872+0000","last_clean":"2024-09-15T23:24:03.115872+0000","last_became_active":"2024-09-15T23:21:52.540308+0000","last_became_peered":"2024-09-15T23:21:52.540308+0000","last_unstale":"2024-09-15T23:24:03.115872+0000","last_undegraded":"2024-09-15T23:24:03.115872+0000","last_fullsized":"2024-09-15T23:24:03.115872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:16.660637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113381+0000","last_change":"2024-09-15T23:21:52.533780+0000","last_active":"2024-09-15T23:24:03.113381+0000","last_peered":"2024-09-15T23:24:03.113381+0000","last_clean":"2024-09-15T23:24:03.113381+0000","last_became_active":"2024-09-15T23:21:52.533354+0000","last_became_peered":"2024-09-15T23:21:52.533354+0000","last_unstale":"2024-09-15T23:24:03.113381+0000","last_undegraded":"2024-09-15T23:24:03.113381+0000","last_fullsized":"2024-09-15T23:24:03.113381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:33.381059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112801+0000","last_change":"2024-09-15T23:21:52.536406+0000","last_active":"2024-09-15T23:24:03.112801+0000","last_peered":"2024-09-15T23:24:03.112801+0000","last_clean":"2024-09-15T23:24:03.112801+0000","last_became_active":"2024-09-15T23:21:52.536091+0000","last_became_peered":"2024-09-15T23:21:52.536091+0000","last_unstale":"2024-09-15T23:24:03.112801+0000","last_undegraded":"2024-09-15T23:24:03.112801+0000","last_fullsized":"2024-09-15T23:24:03.112801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:30.452201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117879+0000","last_change":"2024-09-15T23:21:52.536333+0000","last_active":"2024-09-15T23:24:03.117879+0000","last_peered":"2024-09-15T23:24:03.117879+0000","last_clean":"2024-09-15T23:24:03.117879+0000","last_became_active":"2024-09-15T23:21:52.534677+0000","last_became_peered":"2024-09-15T23:21:52.534677+0000","last_unstale":"2024-09-15T23:24:03.117879+0000","last_undegraded":"2024-09-15T23:24:03.117879+0000","last_fullsized":"2024-09-15T23:24:03.117879+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:05.287914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.112116+0000","last_change":"2024-09-15T23:21:52.536442+0000","last_active":"2024-09-15T23:24:03.112116+0000","last_peered":"2024-09-15T23:24:03.112116+0000","last_clean":"2024-09-15T23:24:03.112116+0000","last_became_active":"2024-09-15T23:21:52.536183+0000","last_became_peered":"2024-09-15T23:21:52.536183+0000","last_unstale":"2024-09-15T23:24:03.112116+0000","last_undegraded":"2024-09-15T23:24:03.112116+0000","last_fullsized":"2024-09-15T23:24:03.112116+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:18:41.641370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117398+0000","last_change":"2024-09-15T23:23:02.277365+0000","last_active":"2024-09-15T23:24:03.117398+0000","last_peered":"2024-09-15T23:24:03.117398+0000","last_clean":"2024-09-15T23:24:03.117398+0000","last_became_active":"2024-09-15T23:23:02.276905+0000","last_became_peered":"2024-09-15T23:23:02.276905+0000","last_unstale":"2024-09-15T23:24:03.117398+0000","last_undegraded":"2024-09-15T23:24:03.117398+0000","last_fullsized":"2024-09-15T23:24:03.117398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:13:21.100657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109788+0000","last_change":"2024-09-15T23:21:52.531171+0000","last_active":"2024-09-15T23:24:03.109788+0000","last_peered":"2024-09-15T23:24:03.109788+0000","last_clean":"2024-09-15T23:24:03.109788+0000","last_became_active":"2024-09-15T23:21:52.530724+0000","last_became_peered":"2024-09-15T23:21:52.530724+0000","last_unstale":"2024-09-15T23:24:03.109788+0000","last_undegraded":"2024-09-15T23:24:03.109788+0000","last_fullsized":"2024-09-15T23:24:03.109788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:28:11.009119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.115400+0000","last_change":"2024-09-15T23:21:52.530433+0000","last_active":"2024-09-15T23:24:03.115400+0000","last_peered":"2024-09-15T23:24:03.115400+0000","last_clean":"2024-09-15T23:24:03.115400+0000","last_became_active":"2024-09-15T23:21:52.530058+0000","last_became_peered":"2024-09-15T23:21:52.530058+0000","last_unstale":"2024-09-15T23:24:03.115400+0000","last_undegraded":"2024-09-15T23:24:03.115400+0000","last_fullsized":"2024-09-15T23:24:03.115400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:51.208820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113218+0000","last_change":"2024-09-15T23:21:52.534639+0000","last_active":"2024-09-15T23:24:03.113218+0000","last_peered":"2024-09-15T23:24:03.113218+0000","last_clean":"2024-09-15T23:24:03.113218+0000","last_became_active":"2024-09-15T23:21:52.534362+0000","last_became_peered":"2024-09-15T23:21:52.534362+0000","last_unstale":"2024-09-15T23:24:03.113218+0000","last_undegraded":"2024-09-15T23:24:03.113218+0000","last_fullsized":"2024-09-15T23:24:03.113218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:58:12.023501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194630+0000","last_change":"2024-09-15T23:21:52.532758+0000","last_active":"2024-09-15T23:24:03.194630+0000","last_peered":"2024-09-15T23:24:03.194630+0000","last_clean":"2024-09-15T23:24:03.194630+0000","last_became_active":"2024-09-15T23:21:52.532412+0000","last_became_peered":"2024-09-15T23:21:52.532412+0000","last_unstale":"2024-09-15T23:24:03.194630+0000","last_undegraded":"2024-09-15T23:24:03.194630+0000","last_fullsized":"2024-09-15T23:24:03.194630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:46.285711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116737+0000","last_change":"2024-09-15T23:21:52.533928+0000","last_active":"2024-09-15T23:24:03.116737+0000","last_peered":"2024-09-15T23:24:03.116737+0000","last_clean":"2024-09-15T23:24:03.116737+0000","last_became_active":"2024-09-15T23:21:52.533578+0000","last_became_peered":"2024-09-15T23:21:52.533578+0000","last_unstale":"2024-09-15T23:24:03.116737+0000","last_undegraded":"2024-09-15T23:24:03.116737+0000","last_fullsized":"2024-09-15T23:24:03.116737+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:49.476316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116894+0000","last_change":"2024-09-15T23:21:52.533417+0000","last_active":"2024-09-15T23:24:03.116894+0000","last_peered":"2024-09-15T23:24:03.116894+0000","last_clean":"2024-09-15T23:24:03.116894+0000","last_became_active":"2024-09-15T23:21:52.533085+0000","last_became_peered":"2024-09-15T23:21:52.533085+0000","last_unstale":"2024-09-15T23:24:03.116894+0000","last_undegraded":"2024-09-15T23:24:03.116894+0000","last_fullsized":"2024-09-15T23:24:03.116894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:53:20.034798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193763+0000","last_change":"2024-09-15T23:21:52.542340+0000","last_active":"2024-09-15T23:24:03.193763+0000","last_peered":"2024-09-15T23:24:03.193763+0000","last_clean":"2024-09-15T23:24:03.193763+0000","last_became_active":"2024-09-15T23:21:52.542048+0000","last_became_peered":"2024-09-15T23:21:52.542048+0000","last_unstale":"2024-09-15T23:24:03.193763+0000","last_undegraded":"2024-09-15T23:24:03.193763+0000","last_fullsized":"2024-09-15T23:24:03.193763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:57.557420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.193941+0000","last_change":"2024-09-15T23:21:52.535576+0000","last_active":"2024-09-15T23:24:03.193941+0000","last_peered":"2024-09-15T23:24:03.193941+0000","last_clean":"2024-09-15T23:24:03.193941+0000","last_became_active":"2024-09-15T23:21:52.535223+0000","last_became_peered":"2024-09-15T23:21:52.535223+0000","last_unstale":"2024-09-15T23:24:03.193941+0000","last_undegraded":"2024-09-15T23:24:03.193941+0000","last_fullsized":"2024-09-15T23:24:03.193941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:45.874927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.109691+0000","last_change":"2024-09-15T23:23:02.150556+0000","last_active":"2024-09-15T23:24:03.109691+0000","last_peered":"2024-09-15T23:24:03.109691+0000","last_clean":"2024-09-15T23:24:03.109691+0000","last_became_active":"2024-09-15T23:23:02.150253+0000","last_became_peered":"2024-09-15T23:23:02.150253+0000","last_unstale":"2024-09-15T23:24:03.109691+0000","last_undegraded":"2024-09-15T23:24:03.109691+0000","last_fullsized":"2024-09-15T23:24:03.109691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:47.950356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110875+0000","last_change":"2024-09-15T23:21:52.537012+0000","last_active":"2024-09-15T23:24:03.110875+0000","last_peered":"2024-09-15T23:24:03.110875+0000","last_clean":"2024-09-15T23:24:03.110875+0000","last_became_active":"2024-09-15T23:21:52.536245+0000","last_became_peered":"2024-09-15T23:21:52.536245+0000","last_unstale":"2024-09-15T23:24:03.110875+0000","last_undegraded":"2024-09-15T23:24:03.110875+0000","last_fullsized":"2024-09-15T23:24:03.110875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:20:31.150491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114034+0000","last_change":"2024-09-15T23:21:52.540431+0000","last_active":"2024-09-15T23:24:03.114034+0000","last_peered":"2024-09-15T23:24:03.114034+0000","last_clean":"2024-09-15T23:24:03.114034+0000","last_became_active":"2024-09-15T23:21:52.540104+0000","last_became_peered":"2024-09-15T23:21:52.540104+0000","last_unstale":"2024-09-15T23:24:03.114034+0000","last_undegraded":"2024-09-15T23:24:03.114034+0000","last_fullsized":"2024-09-15T23:24:03.114034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:55.550258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114876+0000","last_change":"2024-09-15T23:21:52.537590+0000","last_active":"2024-09-15T23:24:03.114876+0000","last_peered":"2024-09-15T23:24:03.114876+0000","last_clean":"2024-09-15T23:24:03.114876+0000","last_became_active":"2024-09-15T23:21:52.533804+0000","last_became_peered":"2024-09-15T23:21:52.533804+0000","last_unstale":"2024-09-15T23:24:03.114876+0000","last_undegraded":"2024-09-15T23:24:03.114876+0000","last_fullsized":"2024-09-15T23:24:03.114876+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:16:48.449090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117952+0000","last_change":"2024-09-15T23:21:52.536130+0000","last_active":"2024-09-15T23:24:03.117952+0000","last_peered":"2024-09-15T23:24:03.117952+0000","last_clean":"2024-09-15T23:24:03.117952+0000","last_became_active":"2024-09-15T23:21:52.535336+0000","last_became_peered":"2024-09-15T23:21:52.535336+0000","last_unstale":"2024-09-15T23:24:03.117952+0000","last_undegraded":"2024-09-15T23:24:03.117952+0000","last_fullsized":"2024-09-15T23:24:03.117952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:05:52.581994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.111899+0000","last_change":"2024-09-15T23:21:52.540485+0000","last_active":"2024-09-15T23:24:03.111899+0000","last_peered":"2024-09-15T23:24:03.111899+0000","last_clean":"2024-09-15T23:24:03.111899+0000","last_became_active":"2024-09-15T23:21:52.540178+0000","last_became_peered":"2024-09-15T23:21:52.540178+0000","last_unstale":"2024-09-15T23:24:03.111899+0000","last_undegraded":"2024-09-15T23:24:03.111899+0000","last_fullsized":"2024-09-15T23:24:03.111899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:40:22.375222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.113883+0000","last_change":"2024-09-15T23:21:52.538845+0000","last_active":"2024-09-15T23:24:03.113883+0000","last_peered":"2024-09-15T23:24:03.113883+0000","last_clean":"2024-09-15T23:24:03.113883+0000","last_became_active":"2024-09-15T23:21:52.538511+0000","last_became_peered":"2024-09-15T23:21:52.538511+0000","last_unstale":"2024-09-15T23:24:03.113883+0000","last_undegraded":"2024-09-15T23:24:03.113883+0000","last_fullsized":"2024-09-15T23:24:03.113883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:00.205945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.114158+0000","last_change":"2024-09-15T23:23:02.149590+0000","last_active":"2024-09-15T23:24:03.114158+0000","last_peered":"2024-09-15T23:24:03.114158+0000","last_clean":"2024-09-15T23:24:03.114158+0000","last_became_active":"2024-09-15T23:23:02.149184+0000","last_became_peered":"2024-09-15T23:23:02.149184+0000","last_unstale":"2024-09-15T23:24:03.114158+0000","last_undegraded":"2024-09-15T23:24:03.114158+0000","last_fullsized":"2024-09-15T23:24:03.114158+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:05:29.407175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.117530+0000","last_change":"2024-09-15T23:21:52.530524+0000","last_active":"2024-09-15T23:24:03.117530+0000","last_peered":"2024-09-15T23:24:03.117530+0000","last_clean":"2024-09-15T23:24:03.117530+0000","last_became_active":"2024-09-15T23:21:52.530203+0000","last_became_peered":"2024-09-15T23:21:52.530203+0000","last_unstale":"2024-09-15T23:24:03.117530+0000","last_undegraded":"2024-09-15T23:24:03.117530+0000","last_fullsized":"2024-09-15T23:24:03.117530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:50.560211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.110690+0000","last_change":"2024-09-15T23:21:52.537256+0000","last_active":"2024-09-15T23:24:03.110690+0000","last_peered":"2024-09-15T23:24:03.110690+0000","last_clean":"2024-09-15T23:24:03.110690+0000","last_became_active":"2024-09-15T23:21:52.536731+0000","last_became_peered":"2024-09-15T23:21:52.536731+0000","last_unstale":"2024-09-15T23:24:03.110690+0000","last_undegraded":"2024-09-15T23:24:03.110690+0000","last_fullsized":"2024-09-15T23:24:03.110690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:22:18.815398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194364+0000","last_change":"2024-09-15T23:21:52.537728+0000","last_active":"2024-09-15T23:24:03.194364+0000","last_peered":"2024-09-15T23:24:03.194364+0000","last_clean":"2024-09-15T23:24:03.194364+0000","last_became_active":"2024-09-15T23:21:52.537434+0000","last_became_peered":"2024-09-15T23:21:52.537434+0000","last_unstale":"2024-09-15T23:24:03.194364+0000","last_undegraded":"2024-09-15T23:24:03.194364+0000","last_fullsized":"2024-09-15T23:24:03.194364+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:27:25.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.194218+0000","last_change":"2024-09-15T23:21:52.534780+0000","last_active":"2024-09-15T23:24:03.194218+0000","last_peered":"2024-09-15T23:24:03.194218+0000","last_clean":"2024-09-15T23:24:03.194218+0000","last_became_active":"2024-09-15T23:21:52.534399+0000","last_became_peered":"2024-09-15T23:21:52.534399+0000","last_unstale":"2024-09-15T23:24:03.194218+0000","last_undegraded":"2024-09-15T23:24:03.194218+0000","last_fullsized":"2024-09-15T23:24:03.194218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:47.344129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"88'8","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:24:03.116592+0000","last_change":"2024-09-15T23:21:52.533403+0000","last_active":"2024-09-15T23:24:03.116592+0000","last_peered":"2024-09-15T23:24:03.116592+0000","last_clean":"2024-09-15T23:24:03.116592+0000","last_became_active":"2024-09-15T23:21:52.533069+0000","last_became_peered":"2024-09-15T23:21:52.533069+0000","last_unstale":"2024-09-15T23:24:03.116592+0000","last_undegraded":"2024-09-15T23:24:03.116592+0000","last_fullsized":"2024-09-15T23:24:03.116592+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:21:51.499051+0000","last_clean_scrub_stamp":"2024-09-15T23:21:51.499051+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:38.780965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":78576,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1175,"num_read_kb":672,"num_write":268,"num_write_kb":163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":63447,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":235728,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":246033,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":74,"seq":317827579950,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":521776,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841586,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1163264,"data_stored":1178923,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":61,"seq":261993005111,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":524287,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234045,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":520569,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462978,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1073152,"data_stored":1116395,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626503,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1003520,"data_stored":515695,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":35,"seq":150323855436,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":1112565,"data_compressed":65347,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018961,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1044480,"data_stored":591615,"data_compressed":39742,"data_compressed_allocated":53248,"data_compressed_original":143360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":23,"seq":98784247893,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":515403,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":17,"seq":73014444123,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1019904,"data_stored":517253,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607648,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1081344,"data_stored":587644,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1028096,"data_stored":517151,"data_compressed":29497,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34211,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35850,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":72671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":72546,"data_compressed":10245,"data_compressed_allocated":12288,"data_compressed_original":61440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":73582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:24:35.050 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:35.299 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.756+0000 7f152608f640 1 -- 172.21.15.89:0/1097321463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1520102790 msgr2=0x7f15201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.756+0000 7f152608f640 1 --2- 172.21.15.89:0/1097321463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1520102790 0x7f15201067e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1508009900 tx=0x7f150802a410 comp rx=0 tx=0).stop 2024-09-15T23:24:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.756+0000 7f152608f640 1 -- 172.21.15.89:0/1097321463 shutdown_connections 2024-09-15T23:24:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.756+0000 7f152608f640 1 --2- 172.21.15.89:0/1097321463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1520106f10 0x7f152010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.756+0000 7f152608f640 1 --2- 172.21.15.89:0/1097321463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1520102790 0x7f15201067e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.756+0000 7f152608f640 1 --2- 172.21.15.89:0/1097321463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1520101de0 0x7f15201021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.756+0000 7f152608f640 1 -- 172.21.15.89:0/1097321463 >> 172.21.15.89:0/1097321463 conn(0x7f15200fd8c0 msgr2=0x7f15200ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.756+0000 7f152608f640 1 -- 172.21.15.89:0/1097321463 shutdown_connections 2024-09-15T23:24:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.756+0000 7f152608f640 1 -- 172.21.15.89:0/1097321463 wait complete. 2024-09-15T23:24:35.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.756+0000 7f152608f640 1 Processor -- start 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f152608f640 1 -- start start 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f152608f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1520101de0 0x7f1520198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f152608f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1520102790 0x7f1520198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f152608f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1520106f10 0x7f152019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f152608f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1520111f30 con 0x7f1520102790 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f152608f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1520111db0 con 0x7f1520101de0 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f152608f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f15201120b0 con 0x7f1520106f10 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1520102790 0x7f1520198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1520102790 0x7f1520198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37920/0 (socket says 172.21.15.89:37920) 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151effd640 1 -- 172.21.15.89:0/1635182376 learned_addr learned my addr 172.21.15.89:0/1635182376 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151ffff640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1520106f10 0x7f152019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151f7fe640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1520101de0 0x7f1520198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151effd640 1 -- 172.21.15.89:0/1635182376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1520106f10 msgr2=0x7f152019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151effd640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1520106f10 0x7f152019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151effd640 1 -- 172.21.15.89:0/1635182376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1520101de0 msgr2=0x7f1520198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151effd640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1520101de0 0x7f1520198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151effd640 1 -- 172.21.15.89:0/1635182376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152019d3c0 con 0x7f1520102790 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151effd640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1520102790 0x7f1520198950 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1508005ab0 tx=0x7f15080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.757+0000 7f151cff9640 1 -- 172.21.15.89:0/1635182376 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150803f030 con 0x7f1520102790 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.758+0000 7f152608f640 1 -- 172.21.15.89:0/1635182376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f152019d650 con 0x7f1520102790 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.758+0000 7f151cff9640 1 -- 172.21.15.89:0/1635182376 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1508003ce0 con 0x7f1520102790 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.758+0000 7f151cff9640 1 -- 172.21.15.89:0/1635182376 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508035070 con 0x7f1520102790 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.758+0000 7f152608f640 1 -- 172.21.15.89:0/1635182376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15201a4f90 con 0x7f1520102790 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.759+0000 7f152608f640 1 -- 172.21.15.89:0/1635182376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15201032c0 con 0x7f1520102790 2024-09-15T23:24:35.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.760+0000 7f151cff9640 1 -- 172.21.15.89:0/1635182376 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f1508042030 con 0x7f1520102790 2024-09-15T23:24:35.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.760+0000 7f151cff9640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f150c0a32d0 0x7f150c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.761+0000 7f151f7fe640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f150c0a32d0 0x7f150c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.761+0000 7f151cff9640 1 -- 172.21.15.89:0/1635182376 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f15080f0830 con 0x7f1520102790 2024-09-15T23:24:35.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.761+0000 7f151f7fe640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f150c0a32d0 0x7f150c0a5790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1510005e10 tx=0x7f1510005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:35.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.764+0000 7f151cff9640 1 -- 172.21.15.89:0/1635182376 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f15080bd000 con 0x7f1520102790 2024-09-15T23:24:35.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.967+0000 7f152608f640 1 -- 172.21.15.89:0/1635182376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f152010a0f0 con 0x7f1520102790 2024-09-15T23:24:35.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.968+0000 7f151cff9640 1 -- 172.21.15.89:0/1635182376 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f15080d0e20 con 0x7f1520102790 2024-09-15T23:24:35.969 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:35.969 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:24:35.970 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:24:35.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 -- 172.21.15.89:0/1635182376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f150c0a32d0 msgr2=0x7f150c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f150c0a32d0 0x7f150c0a5790 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1510005e10 tx=0x7f1510005da0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 -- 172.21.15.89:0/1635182376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1520102790 msgr2=0x7f1520198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1520102790 0x7f1520198950 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1508005ab0 tx=0x7f15080042f0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 -- 172.21.15.89:0/1635182376 shutdown_connections 2024-09-15T23:24:35.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f150c0a32d0 0x7f150c0a5790 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1520106f10 0x7f152019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1520102790 0x7f1520198950 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 --2- 172.21.15.89:0/1635182376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1520101de0 0x7f1520198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 -- 172.21.15.89:0/1635182376 >> 172.21.15.89:0/1635182376 conn(0x7f15200fd8c0 msgr2=0x7f15200fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:35.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 -- 172.21.15.89:0/1635182376 shutdown_connections 2024-09-15T23:24:35.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:35.970+0000 7f152608f640 1 -- 172.21.15.89:0/1635182376 wait complete. 2024-09-15T23:24:36.290 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:36 smithi089 ceph-mon[30933]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:36.290 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:36 smithi089 ceph-mon[30933]: pgmap v367: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-15T23:24:36.290 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:36 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1635182376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:36 smithi120 ceph-mon[35250]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:36 smithi120 ceph-mon[35250]: pgmap v367: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-15T23:24:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:36 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1635182376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:36 smithi174 ceph-mon[35586]: from='client.15444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:24:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:36 smithi174 ceph-mon[35586]: pgmap v367: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-15T23:24:36.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:36 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1635182376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:36.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:36.902 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:37.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.389+0000 7faa86f99640 1 -- 172.21.15.89:0/3681036697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa80102570 msgr2=0x7faa801029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.389+0000 7faa86f99640 1 --2- 172.21.15.89:0/3681036697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa80102570 0x7faa801029b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009900 tx=0x7faa6c02a480 comp rx=0 tx=0).stop 2024-09-15T23:24:37.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.389+0000 7faa86f99640 1 -- 172.21.15.89:0/3681036697 shutdown_connections 2024-09-15T23:24:37.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.389+0000 7faa86f99640 1 --2- 172.21.15.89:0/3681036697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa80102ef0 0x7faa8010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.389+0000 7faa86f99640 1 --2- 172.21.15.89:0/3681036697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa80102570 0x7faa801029b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.389+0000 7faa86f99640 1 --2- 172.21.15.89:0/3681036697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa80100e20 0x7faa80101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.389+0000 7faa86f99640 1 -- 172.21.15.89:0/3681036697 >> 172.21.15.89:0/3681036697 conn(0x7faa800fc7d0 msgr2=0x7faa800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:37.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.389+0000 7faa86f99640 1 -- 172.21.15.89:0/3681036697 shutdown_connections 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.389+0000 7faa86f99640 1 -- 172.21.15.89:0/3681036697 wait complete. 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa86f99640 1 Processor -- start 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa86f99640 1 -- start start 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa86f99640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa80100e20 0x7faa80198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa86f99640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa80102570 0x7faa80198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa86f99640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa80102ef0 0x7faa8019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa86f99640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faa80111020 con 0x7faa80102570 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa86f99640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faa80110ea0 con 0x7faa80100e20 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa86f99640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faa801111a0 con 0x7faa80102ef0 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa7ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa80102570 0x7faa80198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa7ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa80102570 0x7faa80198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37932/0 (socket says 172.21.15.89:37932) 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa7ffff640 1 -- 172.21.15.89:0/390004364 learned_addr learned my addr 172.21.15.89:0/390004364 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:37.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.390+0000 7faa84d35640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa80100e20 0x7faa80198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa85536640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa80102ef0 0x7faa8019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa7ffff640 1 -- 172.21.15.89:0/390004364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa80102ef0 msgr2=0x7faa8019cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa7ffff640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa80102ef0 0x7faa8019cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa7ffff640 1 -- 172.21.15.89:0/390004364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa80100e20 msgr2=0x7faa80198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa7ffff640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa80100e20 0x7faa80198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa7ffff640 1 -- 172.21.15.89:0/390004364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8019d600 con 0x7faa80102570 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa85536640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa80102ef0 0x7faa8019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa84d35640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa80100e20 0x7faa80198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa7ffff640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa80102570 0x7faa80198b90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7faa6c006010 tx=0x7faa6c003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa7dffb640 1 -- 172.21.15.89:0/390004364 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c03f070 con 0x7faa80102570 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa7dffb640 1 -- 172.21.15.89:0/390004364 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa6c031e30 con 0x7faa80102570 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa86f99640 1 -- 172.21.15.89:0/390004364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8019d890 con 0x7faa80102570 2024-09-15T23:24:37.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa7dffb640 1 -- 172.21.15.89:0/390004364 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c035030 con 0x7faa80102570 2024-09-15T23:24:37.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.391+0000 7faa86f99640 1 -- 172.21.15.89:0/390004364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa801a51d0 con 0x7faa80102570 2024-09-15T23:24:37.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.392+0000 7faa86f99640 1 -- 172.21.15.89:0/390004364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa80101280 con 0x7faa80102570 2024-09-15T23:24:37.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.394+0000 7faa7dffb640 1 -- 172.21.15.89:0/390004364 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7faa6c042070 con 0x7faa80102570 2024-09-15T23:24:37.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.395+0000 7faa7dffb640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa700a32d0 0x7faa700a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.395+0000 7faa84d35640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa700a32d0 0x7faa700a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.395+0000 7faa7dffb640 1 -- 172.21.15.89:0/390004364 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7faa6c0f0e80 con 0x7faa80102570 2024-09-15T23:24:37.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.395+0000 7faa84d35640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa700a32d0 0x7faa700a5790 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faa74009930 tx=0x7faa74008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:37.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.398+0000 7faa7dffb640 1 -- 172.21.15.89:0/390004364 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7faa6c0bd6e0 con 0x7faa80102570 2024-09-15T23:24:37.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.600+0000 7faa86f99640 1 -- 172.21.15.89:0/390004364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa801a53a0 con 0x7faa80102570 2024-09-15T23:24:37.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.601+0000 7faa7dffb640 1 -- 172.21.15.89:0/390004364 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7faa6c0bcfe0 con 0x7faa80102570 2024-09-15T23:24:37.603 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:37.603 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:24:37.603 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.603+0000 7faa86f99640 1 -- 172.21.15.89:0/390004364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa700a32d0 msgr2=0x7faa700a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.603+0000 7faa86f99640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa700a32d0 0x7faa700a5790 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7faa74009930 tx=0x7faa74008040 comp rx=0 tx=0).stop 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.603+0000 7faa86f99640 1 -- 172.21.15.89:0/390004364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa80102570 msgr2=0x7faa80198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.603+0000 7faa86f99640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa80102570 0x7faa80198b90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7faa6c006010 tx=0x7faa6c003c80 comp rx=0 tx=0).stop 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.603+0000 7faa86f99640 1 -- 172.21.15.89:0/390004364 shutdown_connections 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.603+0000 7faa86f99640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa700a32d0 0x7faa700a5790 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.603+0000 7faa86f99640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa80102ef0 0x7faa8019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.603+0000 7faa86f99640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa80102570 0x7faa80198b90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.603+0000 7faa86f99640 1 --2- 172.21.15.89:0/390004364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa80100e20 0x7faa80198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.603+0000 7faa86f99640 1 -- 172.21.15.89:0/390004364 >> 172.21.15.89:0/390004364 conn(0x7faa800fc7d0 msgr2=0x7faa80069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.604+0000 7faa86f99640 1 -- 172.21.15.89:0/390004364 shutdown_connections 2024-09-15T23:24:37.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:37.604+0000 7faa86f99640 1 -- 172.21.15.89:0/390004364 wait complete. 2024-09-15T23:24:38.263 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:24:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:38 smithi089 ceph-mon[30933]: pgmap v368: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 255 B/s wr, 12 op/s 2024-09-15T23:24:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:38 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/390004364' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:38 smithi120 ceph-mon[35250]: pgmap v368: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 255 B/s wr, 12 op/s 2024-09-15T23:24:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:38 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/390004364' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:38 smithi174 ceph-mon[35586]: pgmap v368: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 255 B/s wr, 12 op/s 2024-09-15T23:24:38.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:38 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/390004364' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:38.509 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:38.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.965+0000 7fa8f7b49640 1 -- 172.21.15.89:0/3725255107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f0100e20 msgr2=0x7fa8f0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:38.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.965+0000 7fa8f7b49640 1 --2- 172.21.15.89:0/3725255107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f0100e20 0x7fa8f0101200 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009900 tx=0x7fa8ec02a480 comp rx=0 tx=0).stop 2024-09-15T23:24:38.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.965+0000 7fa8f7b49640 1 -- 172.21.15.89:0/3725255107 shutdown_connections 2024-09-15T23:24:38.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.965+0000 7fa8f7b49640 1 --2- 172.21.15.89:0/3725255107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f0102ef0 0x7fa8f010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:38.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.965+0000 7fa8f7b49640 1 --2- 172.21.15.89:0/3725255107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8f0102570 0x7fa8f01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.965+0000 7fa8f7b49640 1 --2- 172.21.15.89:0/3725255107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f0100e20 0x7fa8f0101200 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.965+0000 7fa8f7b49640 1 -- 172.21.15.89:0/3725255107 >> 172.21.15.89:0/3725255107 conn(0x7fa8f00fc7d0 msgr2=0x7fa8f00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.965+0000 7fa8f7b49640 1 -- 172.21.15.89:0/3725255107 shutdown_connections 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.965+0000 7fa8f7b49640 1 -- 172.21.15.89:0/3725255107 wait complete. 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f7b49640 1 Processor -- start 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f7b49640 1 -- start start 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f7b49640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8f0100e20 0x7fa8f00fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f7b49640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f0102570 0x7fa8f00fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f7b49640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f0102ef0 0x7fa8f00fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f7b49640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f0110f80 con 0x7fa8f0102ef0 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f7b49640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f0110e00 con 0x7fa8f0100e20 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f7b49640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f0111100 con 0x7fa8f0102570 2024-09-15T23:24:38.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f60e6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f0102ef0 0x7fa8f00fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f60e6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f0102ef0 0x7fa8f00fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37938/0 (socket says 172.21.15.89:37938) 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f60e6640 1 -- 172.21.15.89:0/154123703 learned_addr learned my addr 172.21.15.89:0/154123703 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f58e5640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8f0100e20 0x7fa8f00fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.966+0000 7fa8f50e4640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f0102570 0x7fa8f00fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8f60e6640 1 -- 172.21.15.89:0/154123703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f0102570 msgr2=0x7fa8f00fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8f60e6640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f0102570 0x7fa8f00fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8f60e6640 1 -- 172.21.15.89:0/154123703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8f0100e20 msgr2=0x7fa8f00fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8f60e6640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8f0100e20 0x7fa8f00fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8f60e6640 1 -- 172.21.15.89:0/154123703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f01a6a60 con 0x7fa8f0102ef0 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8f60e6640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f0102ef0 0x7fa8f00fd700 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800eb10 tx=0x7fa8e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8e6ffd640 1 -- 172.21.15.89:0/154123703 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8004b20 con 0x7fa8f0102ef0 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8f7b49640 1 -- 172.21.15.89:0/154123703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f01a6d50 con 0x7fa8f0102ef0 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8e6ffd640 1 -- 172.21.15.89:0/154123703 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8e8004cc0 con 0x7fa8f0102ef0 2024-09-15T23:24:38.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8e6ffd640 1 -- 172.21.15.89:0/154123703 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8010760 con 0x7fa8f0102ef0 2024-09-15T23:24:38.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.967+0000 7fa8f7b49640 1 -- 172.21.15.89:0/154123703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f01a7260 con 0x7fa8f0102ef0 2024-09-15T23:24:38.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.968+0000 7fa8f7b49640 1 -- 172.21.15.89:0/154123703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8f0101200 con 0x7fa8f0102ef0 2024-09-15T23:24:38.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.969+0000 7fa8e6ffd640 1 -- 172.21.15.89:0/154123703 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fa8e8010900 con 0x7fa8f0102ef0 2024-09-15T23:24:38.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.970+0000 7fa8e6ffd640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa8dc0a32d0 0x7fa8dc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:38.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.970+0000 7fa8f58e5640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa8dc0a32d0 0x7fa8dc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:38.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.970+0000 7fa8e6ffd640 1 -- 172.21.15.89:0/154123703 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa8e80d0860 con 0x7fa8f0102ef0 2024-09-15T23:24:38.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.971+0000 7fa8f58e5640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa8dc0a32d0 0x7fa8dc0a5790 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009230 tx=0x7fa8ec0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:38.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:38.974+0000 7fa8e6ffd640 1 -- 172.21.15.89:0/154123703 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa8e809d030 con 0x7fa8f0102ef0 2024-09-15T23:24:39.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.145+0000 7fa8f7b49640 1 -- 172.21.15.89:0/154123703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8f00fe1b0 con 0x7fa8f0102ef0 2024-09-15T23:24:39.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.147+0000 7fa8e6ffd640 1 -- 172.21.15.89:0/154123703 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24962 (secure 0 0 0) 0x7fa8e809c930 con 0x7fa8f0102ef0 2024-09-15T23:24:39.148 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:39.149 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":90,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:24:03.106312+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.11","mappings":[{"from":7,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:24:39.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 -- 172.21.15.89:0/154123703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa8dc0a32d0 msgr2=0x7fa8dc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa8dc0a32d0 0x7fa8dc0a5790 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009230 tx=0x7fa8ec0058d0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 -- 172.21.15.89:0/154123703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f0102ef0 msgr2=0x7fa8f00fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f0102ef0 0x7fa8f00fd700 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa8e800eb10 tx=0x7fa8e800c2d0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 -- 172.21.15.89:0/154123703 shutdown_connections 2024-09-15T23:24:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa8dc0a32d0 0x7fa8dc0a5790 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f0102ef0 0x7fa8f00fd700 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8f0102570 0x7fa8f00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 --2- 172.21.15.89:0/154123703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa8f0100e20 0x7fa8f00fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 -- 172.21.15.89:0/154123703 >> 172.21.15.89:0/154123703 conn(0x7fa8f00fc7d0 msgr2=0x7fa8f0069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.149+0000 7fa8f7b49640 1 -- 172.21.15.89:0/154123703 shutdown_connections 2024-09-15T23:24:39.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:39.150+0000 7fa8f7b49640 1 -- 172.21.15.89:0/154123703 wait complete. 2024-09-15T23:24:39.765 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-09-15T23:24:40.010 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:40.413 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:40 smithi089 ceph-mon[30933]: pgmap v369: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-09-15T23:24:40.413 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:40 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/154123703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:40 smithi120 ceph-mon[35250]: pgmap v369: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-09-15T23:24:40.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:40 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/154123703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:40.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:40 smithi174 ceph-mon[35586]: pgmap v369: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-09-15T23:24:40.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:40 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/154123703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:24:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.102+0000 7f367c4ba640 1 -- 172.21.15.89:0/2245144149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36741078a0 msgr2=0x7f3674109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.102+0000 7f367c4ba640 1 --2- 172.21.15.89:0/2245144149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36741078a0 0x7f3674109c90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f366c009ca0 tx=0x7f366c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.102+0000 7f367c4ba640 1 -- 172.21.15.89:0/2245144149 shutdown_connections 2024-09-15T23:24:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.102+0000 7f367c4ba640 1 --2- 172.21.15.89:0/2245144149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36741078a0 0x7f3674109c90 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.102+0000 7f367c4ba640 1 --2- 172.21.15.89:0/2245144149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3674104f70 0x7f3674107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.102+0000 7f367c4ba640 1 --2- 172.21.15.89:0/2245144149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3674102640 0x7f3674104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.102+0000 7f367c4ba640 1 -- 172.21.15.89:0/2245144149 >> 172.21.15.89:0/2245144149 conn(0x7f36740fc1e0 msgr2=0x7f36740fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.102+0000 7f367c4ba640 1 -- 172.21.15.89:0/2245144149 shutdown_connections 2024-09-15T23:24:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.102+0000 7f367c4ba640 1 -- 172.21.15.89:0/2245144149 wait complete. 2024-09-15T23:24:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.102+0000 7f367c4ba640 1 Processor -- start 2024-09-15T23:24:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367c4ba640 1 -- start start 2024-09-15T23:24:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367c4ba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3674102640 0x7f3674197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367c4ba640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3674104f70 0x7f36741982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367c4ba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36741078a0 0x7f367419f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367c4ba640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f367410d7e0 con 0x7f3674104f70 2024-09-15T23:24:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367c4ba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f367410d660 con 0x7f36741078a0 2024-09-15T23:24:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367c4ba640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f367410d960 con 0x7f3674102640 2024-09-15T23:24:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f3679a55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3674104f70 0x7f36741982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f3679a55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3674104f70 0x7f36741982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37954/0 (socket says 172.21.15.89:37954) 2024-09-15T23:24:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f3679a55640 1 -- 172.21.15.89:0/3796278963 learned_addr learned my addr 172.21.15.89:0/3796278963 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367a256640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3674102640 0x7f3674197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367aa57640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36741078a0 0x7f367419f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f3679a55640 1 -- 172.21.15.89:0/3796278963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3674102640 msgr2=0x7f3674197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f3679a55640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3674102640 0x7f3674197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f3679a55640 1 -- 172.21.15.89:0/3796278963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36741078a0 msgr2=0x7f367419f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f3679a55640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36741078a0 0x7f367419f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f3679a55640 1 -- 172.21.15.89:0/3796278963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f367419f9d0 con 0x7f3674104f70 2024-09-15T23:24:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367aa57640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36741078a0 0x7f367419f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f3679a55640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3674104f70 0x7f36741982f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f366400c970 tx=0x7f366400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f366b7fe640 1 -- 172.21.15.89:0/3796278963 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3664007bb0 con 0x7f3674104f70 2024-09-15T23:24:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367c4ba640 1 -- 172.21.15.89:0/3796278963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f367419fcc0 con 0x7f3674104f70 2024-09-15T23:24:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f366b7fe640 1 -- 172.21.15.89:0/3796278963 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3664007d50 con 0x7f3674104f70 2024-09-15T23:24:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f366b7fe640 1 -- 172.21.15.89:0/3796278963 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366400f450 con 0x7f3674104f70 2024-09-15T23:24:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.103+0000 7f367c4ba640 1 -- 172.21.15.89:0/3796278963 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36741a01c0 con 0x7f3674104f70 2024-09-15T23:24:41.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.106+0000 7f366b7fe640 1 -- 172.21.15.89:0/3796278963 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f36640046c0 con 0x7f3674104f70 2024-09-15T23:24:41.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.106+0000 7f367f259740 1 Processor -- start 2024-09-15T23:24:41.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.106+0000 7f367f259740 1 -- start start 2024-09-15T23:24:41.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367f259740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a1dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367f259740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a22da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367f259740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a23370 0x55b9b2a25830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367f259740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b9b29812f0 con 0x55b9b2a1e2e0 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367f259740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b9b2929270 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367f259740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b9b297d2b0 con 0x55b9b2a23370 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f366b7fe640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36600a3390 0x7f36600a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367a256640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a22da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367a256640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a22da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37964/0 (socket says 172.21.15.89:37964) 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367a256640 1 -- 172.21.15.89:0/696942140 learned_addr learned my addr 172.21.15.89:0/696942140 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f3679a55640 1 --2- 172.21.15.89:0/696942140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a23370 0x55b9b2a25830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367aa57640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36600a3390 0x7f36600a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367aa57640 1 --2- 172.21.15.89:0/696942140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a1dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f366b7fe640 1 -- 172.21.15.89:0/3796278963 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f36640cfec0 con 0x7f3674104f70 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367a256640 1 -- 172.21.15.89:0/696942140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a23370 msgr2=0x55b9b2a25830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367a256640 1 --2- 172.21.15.89:0/696942140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a23370 0x55b9b2a25830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367a256640 1 -- 172.21.15.89:0/696942140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 msgr2=0x55b9b2a1dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367a256640 1 --2- 172.21.15.89:0/696942140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a1dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367a256640 1 -- 172.21.15.89:0/696942140 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9b2928340 con 0x55b9b2a1e2e0 2024-09-15T23:24:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367aa57640 1 --2- 172.21.15.89:0/696942140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a1dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367a256640 1 --2- 172.21.15.89:0/696942140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a22da0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f367000bd10 tx=0x7f36700107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d353d87aff5868 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f367aa57640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36600a3390 0x7f36600a5850 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f366c006010 tx=0x7f366c031bb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.107+0000 7f3668bfe640 1 -- 172.21.15.89:0/696942140 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670004030 con 0x55b9b2a1e2e0 2024-09-15T23:24:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f3668bfe640 1 -- 172.21.15.89:0/696942140 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36700041d0 con 0x55b9b2a1e2e0 2024-09-15T23:24:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f3668bfe640 1 -- 172.21.15.89:0/696942140 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670011840 con 0x55b9b2a1e2e0 2024-09-15T23:24:41.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f367f259740 1 -- 172.21.15.89:0/696942140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1e2e0 msgr2=0x55b9b2a22da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f367f259740 1 --2- 172.21.15.89:0/696942140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a22da0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f367000bd10 tx=0x7f36700107a0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f367f259740 1 -- 172.21.15.89:0/696942140 shutdown_connections 2024-09-15T23:24:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f367f259740 1 --2- 172.21.15.89:0/696942140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a23370 0x55b9b2a25830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f367f259740 1 --2- 172.21.15.89:0/696942140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a22da0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f367f259740 1 --2- 172.21.15.89:0/696942140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a1dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f367f259740 1 -- 172.21.15.89:0/696942140 >> 172.21.15.89:0/696942140 conn(0x55b9b2a19010 msgr2=0x55b9b2a19410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f367f259740 1 -- 172.21.15.89:0/696942140 shutdown_connections 2024-09-15T23:24:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.108+0000 7f367f259740 1 -- 172.21.15.89:0/696942140 wait complete. 2024-09-15T23:24:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367f259740 1 -- start start 2024-09-15T23:24:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367f259740 1 Processor -- start 2024-09-15T23:24:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367f259740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a27bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367f259740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a28100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367f259740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a23370 0x55b9b2a2f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367f259740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b9b2a26340 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367f259740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b9b290a290 con 0x55b9b2a23370 2024-09-15T23:24:41.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367f259740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b9b2a264c0 con 0x55b9b2a1e2e0 2024-09-15T23:24:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367f259740 10 client.0 _ll_register_callbacks cb 0x55b9b293b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367a256640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a27bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367a256640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a27bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37974/0 (socket says 172.21.15.89:37974) 2024-09-15T23:24:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367a256640 1 -- 172.21.15.89:0/1070286114 learned_addr learned my addr 172.21.15.89:0/1070286114 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367aa57640 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a23370 0x55b9b2a2f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f3679a55640 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a28100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367a256640 1 -- 172.21.15.89:0/1070286114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 msgr2=0x55b9b2a28100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367a256640 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a28100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367a256640 1 -- 172.21.15.89:0/1070286114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a23370 msgr2=0x55b9b2a2f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367a256640 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a23370 0x55b9b2a2f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.109+0000 7f367a256640 1 -- 172.21.15.89:0/1070286114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9b2694210 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f367a256640 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a27bc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3670003c90 tx=0x7f3670005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcac22251f41753 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f365dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f365dffb640 1 -- 172.21.15.89:0/1070286114 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670029060 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f365dffb640 1 -- 172.21.15.89:0/1070286114 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3670004380 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f367f259740 1 -- 172.21.15.89:0/1070286114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36700041d0 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f365dffb640 1 -- 172.21.15.89:0/1070286114 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3670030050 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f367f259740 10 client.15474 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f367f259740 1 -- 172.21.15.89:0/1070286114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b9b2760620 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f365dffb640 1 -- 172.21.15.89:0/1070286114 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36700041d0 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f365dffb640 1 -- 172.21.15.89:0/1070286114 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3670004750 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f367f259740 10 client.15474 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f367f259740 10 client.15474 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f367f259740 20 client.15474 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f367f259740 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b9b2a33550 0x55b9b2a35940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f367f259740 4 client.15474 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.110+0000 7f367f259740 1 -- 172.21.15.89:0/1070286114 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9b298ea20 con 0x55b9b2a33550 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.111+0000 7f3679a55640 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b9b2a33550 0x55b9b2a35940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.111+0000 7f365dffb640 10 client.15474 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:41.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.111+0000 7f3679a55640 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b9b2a33550 0x55b9b2a35940 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4862cf657ceefb6 server_cookie=1c3d1069e90d5b42 in_seq=0 out_seq=0 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f365dffb640 1 -- 172.21.15.89:0/1070286114 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b9b298ea20 con 0x55b9b2a33550 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f365dffb640 10 client.15474 handle_command_reply: tid=0 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 client.15474 shutdown 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 -- 172.21.15.89:0/1070286114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1d9a0 msgr2=0x55b9b2a27bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a27bc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3670003c90 tx=0x7f3670005ce0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 -- 172.21.15.89:0/1070286114 shutdown_connections 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b9b2a33550 0x55b9b2a35940 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a23370 0x55b9b2a2f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a28100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f365dffb640 0 client.15474 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 --2- 172.21.15.89:0/1070286114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a27bc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 -- 172.21.15.89:0/1070286114 >> 172.21.15.89:0/1070286114 conn(0x55b9b2a19010 msgr2=0x55b9b2a1a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 -- 172.21.15.89:0/1070286114 shutdown_connections 2024-09-15T23:24:41.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 1 -- 172.21.15.89:0/1070286114 wait complete. 2024-09-15T23:24:41.128 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.112+0000 7f367f259740 20 client.15474 trim_cache size 0 max 16384 2024-09-15T23:24:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367f259740 1 Processor -- start 2024-09-15T23:24:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367f259740 1 -- start start 2024-09-15T23:24:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367f259740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a180b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367f259740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a2f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367f259740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a1a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367f259740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f36741aa930 con 0x55b9b2a23370 2024-09-15T23:24:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367f259740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f36741ac470 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367f259740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36741ab000 con 0x55b9b2a1e2e0 2024-09-15T23:24:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367aa57640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a1a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367aa57640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a1a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37984/0 (socket says 172.21.15.89:37984) 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367aa57640 1 -- 172.21.15.89:0/2728188841 learned_addr learned my addr 172.21.15.89:0/2728188841 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f3679a55640 1 --2- 172.21.15.89:0/2728188841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a2f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.126+0000 7f367a256640 1 --2- 172.21.15.89:0/2728188841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a180b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f367aa57640 1 -- 172.21.15.89:0/2728188841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 msgr2=0x55b9b2a2f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f367aa57640 1 --2- 172.21.15.89:0/2728188841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a2f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f367aa57640 1 -- 172.21.15.89:0/2728188841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 msgr2=0x55b9b2a180b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f367aa57640 1 --2- 172.21.15.89:0/2728188841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a180b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f367aa57640 1 -- 172.21.15.89:0/2728188841 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9b2a1e7c0 con 0x55b9b2a23370 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f367aa57640 1 --2- 172.21.15.89:0/2728188841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a1a400 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f366c009ca0 tx=0x7f366c03f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9466497fbd3022c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f365ffff640 1 -- 172.21.15.89:0/2728188841 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c02fce0 con 0x55b9b2a23370 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f365ffff640 1 -- 172.21.15.89:0/2728188841 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f366c036040 con 0x55b9b2a23370 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f365ffff640 1 -- 172.21.15.89:0/2728188841 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c03c670 con 0x55b9b2a23370 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f367f259740 1 -- 172.21.15.89:0/2728188841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 msgr2=0x55b9b2a1a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.127+0000 7f367f259740 1 --2- 172.21.15.89:0/2728188841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a1a400 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f366c009ca0 tx=0x7f366c03f070 comp rx=0 tx=0).stop 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 -- 172.21.15.89:0/2728188841 shutdown_connections 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 --2- 172.21.15.89:0/2728188841 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a1a400 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 --2- 172.21.15.89:0/2728188841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a2f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 --2- 172.21.15.89:0/2728188841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a180b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 -- 172.21.15.89:0/2728188841 >> 172.21.15.89:0/2728188841 conn(0x55b9b2a19010 msgr2=0x55b9b2a179a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 -- 172.21.15.89:0/2728188841 shutdown_connections 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 -- 172.21.15.89:0/2728188841 wait complete. 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 -- start start 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 Processor -- start 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a2ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a2cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a2df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f36741abda0 con 0x55b9b2a23370 2024-09-15T23:24:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f36741ab6d0 con 0x55b9b2a1d9a0 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36741a8910 con 0x55b9b2a1e2e0 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367f259740 10 client.0 _ll_register_callbacks cb 0x7f36640982e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367aa57640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a2df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367aa57640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a2df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37988/0 (socket says 172.21.15.89:37988) 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367aa57640 1 -- 172.21.15.89:0/1983119602 learned_addr learned my addr 172.21.15.89:0/1983119602 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367a256640 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a2ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367aa57640 1 -- 172.21.15.89:0/1983119602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 msgr2=0x55b9b2a2cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367aa57640 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a2cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367aa57640 1 -- 172.21.15.89:0/1983119602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 msgr2=0x55b9b2a2ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367aa57640 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a2ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.128+0000 7f367aa57640 1 -- 172.21.15.89:0/1983119602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9b2a2e570 con 0x55b9b2a23370 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f367aa57640 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a2df60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f366c02fd10 tx=0x7f366c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616bc4264e878f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f365dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f365dffb640 1 -- 172.21.15.89:0/1983119602 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c056050 con 0x55b9b2a23370 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f365dffb640 1 -- 172.21.15.89:0/1983119602 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f366c03c820 con 0x55b9b2a23370 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f367f259740 1 -- 172.21.15.89:0/1983119602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f366c036040 con 0x55b9b2a23370 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f365dffb640 1 -- 172.21.15.89:0/1983119602 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f366c03ca20 con 0x55b9b2a23370 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f367f259740 10 client.15480 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f367f259740 1 -- 172.21.15.89:0/1983119602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b9b2a9e340 con 0x55b9b2a23370 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f365dffb640 1 -- 172.21.15.89:0/1983119602 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f366c036040 con 0x55b9b2a23370 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f365dffb640 1 -- 172.21.15.89:0/1983119602 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f366c05a910 con 0x55b9b2a23370 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f367f259740 10 client.15480 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f367f259740 10 client.15480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f367f259740 20 client.15480 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f367f259740 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b9b2a9ed20 0x55b9b2aa1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f367f259740 4 client.15480 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:24:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.129+0000 7f367f259740 1 -- 172.21.15.89:0/1983119602 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b9b2988d70 con 0x55b9b2a9ed20 2024-09-15T23:24:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.130+0000 7f367a256640 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b9b2a9ed20 0x55b9b2aa1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.130+0000 7f367a256640 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b9b2a9ed20 0x55b9b2aa1140 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e400ac47c05e1f2d server_cookie=78424a022f0a6fd3 in_seq=0 out_seq=0 2024-09-15T23:24:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.130+0000 7f365dffb640 10 client.15480 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:41.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.143+0000 7f365dffb640 1 -- 172.21.15.89:0/1983119602 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b9b2988d70 con 0x55b9b2a9ed20 2024-09-15T23:24:41.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.143+0000 7f365dffb640 10 client.15480 handle_command_reply: tid=0 2024-09-15T23:24:41.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.143+0000 7f367f259740 1 client.15480 shutdown 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 1 -- 172.21.15.89:0/1983119602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 msgr2=0x55b9b2a2df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a2df60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f366c02fd10 tx=0x7f366c0095c0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 1 -- 172.21.15.89:0/1983119602 shutdown_connections 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b9b2a9ed20 0x55b9b2aa1140 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b9b2a23370 0x55b9b2a2df60 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f365dffb640 0 client.15480 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9b2a1e2e0 0x55b9b2a2cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 1 --2- 172.21.15.89:0/1983119602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b9b2a1d9a0 0x55b9b2a2ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 1 -- 172.21.15.89:0/1983119602 >> 172.21.15.89:0/1983119602 conn(0x55b9b2a19010 msgr2=0x55b9b2a181c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 1 -- 172.21.15.89:0/1983119602 shutdown_connections 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 1 -- 172.21.15.89:0/1983119602 wait complete. 2024-09-15T23:24:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367f259740 20 client.15480 trim_cache size 0 max 16384 2024-09-15T23:24:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:24:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:24:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:24:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:24:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367c4ba640 1 -- 172.21.15.89:0/3796278963 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36600a3390 msgr2=0x7f36600a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367c4ba640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36600a3390 0x7f36600a5850 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f366c006010 tx=0x7f366c031bb0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367c4ba640 1 -- 172.21.15.89:0/3796278963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3674104f70 msgr2=0x7f36741982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367c4ba640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3674104f70 0x7f36741982f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f366400c970 tx=0x7f366400ce30 comp rx=0 tx=0).stop 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367c4ba640 1 -- 172.21.15.89:0/3796278963 shutdown_connections 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367c4ba640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36600a3390 0x7f36600a5850 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367c4ba640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36741078a0 0x7f367419f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367c4ba640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3674104f70 0x7f36741982f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367c4ba640 1 --2- 172.21.15.89:0/3796278963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3674102640 0x7f3674197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.144+0000 7f367c4ba640 1 -- 172.21.15.89:0/3796278963 >> 172.21.15.89:0/3796278963 conn(0x7f36740fc1e0 msgr2=0x7f3674102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.145+0000 7f367c4ba640 1 -- 172.21.15.89:0/3796278963 shutdown_connections 2024-09-15T23:24:41.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:41.145+0000 7f367c4ba640 1 -- 172.21.15.89:0/3796278963 wait complete. 2024-09-15T23:24:41.222 INFO:teuthology.orchestra.run.smithi089.stdout:{"run_dir":"/var/run/ceph"} 2024-09-15T23:24:41.223 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-09-15T23:24:41.223 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 3393 2024-09-15T23:24:41.224 DEBUG:tasks.quiescer:Yielding 2024-09-15T23:24:41.224 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T23:24:41.247 INFO:tasks.workunit:Pulling workunits from ref ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-15T23:24:41.248 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T23:24:41.248 INFO:tasks.workunit:timeout=3h 2024-09-15T23:24:41.248 INFO:tasks.workunit:cleanup=True 2024-09-15T23:24:41.249 DEBUG:teuthology.orchestra.run.smithi089:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T23:24:41.249 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:41.266 INFO:teuthology.orchestra.run.smithi089.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-15T23:24:41.267 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-15T23:24:41.267 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 78h/120d Inode: 1099511628283 Links: 2 2024-09-15T23:24:41.267 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:24:41.267 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-15T23:24:41.267 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-15 23:22:43.898345024 +0000 2024-09-15T23:24:41.268 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-15 23:22:43.898345024 +0000 2024-09-15T23:24:41.268 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-15 23:24:18.564579989 +0000 2024-09-15T23:24:41.268 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-15 23:22:43.898345024 +0000 2024-09-15T23:24:41.268 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-15T23:24:41.268 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-15T23:24:41.340 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-15T23:24:41.398 INFO:tasks.workunit.client.0.smithi089.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T23:24:41.505 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:42.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.996+0000 7fc103478640 1 -- 172.21.15.89:0/494830104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc1036e0 msgr2=0x7fc0fc107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.996+0000 7fc103478640 1 --2- 172.21.15.89:0/494830104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc1036e0 0x7fc0fc107b90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009930 tx=0x7fc0ec02a420 comp rx=0 tx=0).stop 2024-09-15T23:24:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.996+0000 7fc103478640 1 -- 172.21.15.89:0/494830104 shutdown_connections 2024-09-15T23:24:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.996+0000 7fc103478640 1 --2- 172.21.15.89:0/494830104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0fc1082a0 0x7fc0fc10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.996+0000 7fc103478640 1 --2- 172.21.15.89:0/494830104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc1036e0 0x7fc0fc107b90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.996+0000 7fc103478640 1 --2- 172.21.15.89:0/494830104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0fc102d30 0x7fc0fc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.996+0000 7fc103478640 1 -- 172.21.15.89:0/494830104 >> 172.21.15.89:0/494830104 conn(0x7fc0fc0fc7d0 msgr2=0x7fc0fc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:42.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.996+0000 7fc103478640 1 -- 172.21.15.89:0/494830104 shutdown_connections 2024-09-15T23:24:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.996+0000 7fc103478640 1 -- 172.21.15.89:0/494830104 wait complete. 2024-09-15T23:24:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc103478640 1 Processor -- start 2024-09-15T23:24:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc103478640 1 -- start start 2024-09-15T23:24:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc103478640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0fc102d30 0x7fc0fc198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc103478640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc1036e0 0x7fc0fc198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc103478640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0fc1082a0 0x7fc0fc19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc103478640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc1120a0 con 0x7fc0fc1036e0 2024-09-15T23:24:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc103478640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc111f20 con 0x7fc0fc1082a0 2024-09-15T23:24:42.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc103478640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc112220 con 0x7fc0fc102d30 2024-09-15T23:24:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc100a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc1036e0 0x7fc0fc198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc100a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc1036e0 0x7fc0fc198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37998/0 (socket says 172.21.15.89:37998) 2024-09-15T23:24:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc100a13640 1 -- 172.21.15.89:0/3550888170 learned_addr learned my addr 172.21.15.89:0/3550888170 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.997+0000 7fc101a15640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0fc1082a0 0x7fc0fc19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc101214640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0fc102d30 0x7fc0fc198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc100a13640 1 -- 172.21.15.89:0/3550888170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0fc102d30 msgr2=0x7fc0fc198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc100a13640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0fc102d30 0x7fc0fc198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc100a13640 1 -- 172.21.15.89:0/3550888170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0fc1082a0 msgr2=0x7fc0fc19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc100a13640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0fc1082a0 0x7fc0fc19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc100a13640 1 -- 172.21.15.89:0/3550888170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0fc19d500 con 0x7fc0fc1036e0 2024-09-15T23:24:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc100a13640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc1036e0 0x7fc0fc198a90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec005940 tx=0x7fc0ec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc0f27fc640 1 -- 172.21.15.89:0/3550888170 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec03f030 con 0x7fc0fc1036e0 2024-09-15T23:24:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc0f27fc640 1 -- 172.21.15.89:0/3550888170 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0ec003ce0 con 0x7fc0fc1036e0 2024-09-15T23:24:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc103478640 1 -- 172.21.15.89:0/3550888170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0fc19d790 con 0x7fc0fc1036e0 2024-09-15T23:24:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc0f27fc640 1 -- 172.21.15.89:0/3550888170 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec035070 con 0x7fc0fc1036e0 2024-09-15T23:24:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.998+0000 7fc103478640 1 -- 172.21.15.89:0/3550888170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0fc1a50d0 con 0x7fc0fc1036e0 2024-09-15T23:24:42.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:41.999+0000 7fc103478640 1 -- 172.21.15.89:0/3550888170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0fc06a850 con 0x7fc0fc1036e0 2024-09-15T23:24:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.000+0000 7fc0f27fc640 1 -- 172.21.15.89:0/3550888170 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fc0ec042030 con 0x7fc0fc1036e0 2024-09-15T23:24:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.001+0000 7fc0f27fc640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0e80a32d0 0x7fc0e80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.001+0000 7fc101214640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0e80a32d0 0x7fc0e80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.001+0000 7fc0f27fc640 1 -- 172.21.15.89:0/3550888170 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc0ec0f0650 con 0x7fc0fc1036e0 2024-09-15T23:24:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.002+0000 7fc101214640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0e80a32d0 0x7fc0e80a5790 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8005e10 tx=0x7fc0f8005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:42.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.004+0000 7fc0f27fc640 1 -- 172.21.15.89:0/3550888170 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc0ec0bce20 con 0x7fc0fc1036e0 2024-09-15T23:24:42.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.210+0000 7fc103478640 1 -- 172.21.15.89:0/3550888170 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0fc103110 con 0x7fc0fc1036e0 2024-09-15T23:24:42.211 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:42 smithi089 ceph-mon[30933]: pgmap v370: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-09-15T23:24:42.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.211+0000 7fc0f27fc640 1 -- 172.21.15.89:0/3550888170 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fc0fc103110 con 0x7fc0fc1036e0 2024-09-15T23:24:42.212 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:24:42.212 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:24:42.213 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:24:42.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 -- 172.21.15.89:0/3550888170 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0e80a32d0 msgr2=0x7fc0e80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0e80a32d0 0x7fc0e80a5790 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8005e10 tx=0x7fc0f8005da0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 -- 172.21.15.89:0/3550888170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc1036e0 msgr2=0x7fc0fc198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc1036e0 0x7fc0fc198a90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec005940 tx=0x7fc0ec0042f0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 -- 172.21.15.89:0/3550888170 shutdown_connections 2024-09-15T23:24:42.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0e80a32d0 0x7fc0e80a5790 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0fc1082a0 0x7fc0fc19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc0fc1036e0 0x7fc0fc198a90 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 --2- 172.21.15.89:0/3550888170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0fc102d30 0x7fc0fc198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 -- 172.21.15.89:0/3550888170 >> 172.21.15.89:0/3550888170 conn(0x7fc0fc0fc7d0 msgr2=0x7fc0fc105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:42.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 -- 172.21.15.89:0/3550888170 shutdown_connections 2024-09-15T23:24:42.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:42.213+0000 7fc103478640 1 -- 172.21.15.89:0/3550888170 wait complete. 2024-09-15T23:24:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:42 smithi120 ceph-mon[35250]: pgmap v370: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-09-15T23:24:42.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:42 smithi174 ceph-mon[35586]: pgmap v370: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-09-15T23:24:43.001 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-15T23:21:53.066091+0000', 'modified': '2024-09-15T23:24:19.235823+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24452, 'mds_1': 24436, 'mds_2': 24430, 'mds_3': 24428, 'mds_4': 24424}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14610': {'gid': 14610, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.89:6839/2895111363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6838', 'nonce': 2895111363}, {'type': 'v1', 'addr': '172.21.15.89:6839', 'nonce': 2895111363}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14616': {'gid': 14616, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.89:6841/3365004630', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6840', 'nonce': 3365004630}, {'type': 'v1', 'addr': '172.21.15.89:6841', 'nonce': 3365004630}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24389': {'gid': 24389, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6833/4147766006', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6832', 'nonce': 4147766006}, {'type': 'v1', 'addr': '172.21.15.120:6833', 'nonce': 4147766006}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24403': {'gid': 24403, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6833/2504210581', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6832', 'nonce': 2504210581}, {'type': 'v1', 'addr': '172.21.15.174:6833', 'nonce': 2504210581}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24404': {'gid': 24404, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1429316223', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1429316223}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1429316223}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24424': {'gid': 24424, 'name': 'f', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 19, 'addr': '172.21.15.174:6835/733541815', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 733541815}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 733541815}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24428': {'gid': 24428, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.120:6837/991556551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 991556551}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 991556551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24430': {'gid': 24430, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.174:6837/3626951914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 3626951914}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 3626951914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24436': {'gid': 24436, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.174:6839/3380568300', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 3380568300}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 3380568300}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24452': {'gid': 24452, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.120:6839/1554218591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 1554218591}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 1554218591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24452, 'qdb_cluster': [24424, 24428, 24430, 24436, 24452]} 2024-09-15T23:24:43.001 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-09-15T23:24:43.001 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.f status 2024-09-15T23:24:43.302 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:43.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:43 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3550888170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:43.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:43 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:24:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:43 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3550888170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:43 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:24:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:43 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3550888170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:43 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:24:43.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.789+0000 7fc1b4648640 1 -- 172.21.15.89:0/3197437317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac106f10 msgr2=0x7fc1ac10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.789+0000 7fc1b4648640 1 --2- 172.21.15.89:0/3197437317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac106f10 0x7fc1ac10ea90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009ca0 tx=0x7fc19c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.789+0000 7fc1b4648640 1 -- 172.21.15.89:0/3197437317 shutdown_connections 2024-09-15T23:24:43.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.789+0000 7fc1b4648640 1 --2- 172.21.15.89:0/3197437317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac106f10 0x7fc1ac10ea90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.789+0000 7fc1b4648640 1 --2- 172.21.15.89:0/3197437317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1ac102790 0x7fc1ac1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.789+0000 7fc1b4648640 1 --2- 172.21.15.89:0/3197437317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1ac101de0 0x7fc1ac1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.789+0000 7fc1b4648640 1 -- 172.21.15.89:0/3197437317 >> 172.21.15.89:0/3197437317 conn(0x7fc1ac0fd8c0 msgr2=0x7fc1ac0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:43.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.789+0000 7fc1b4648640 1 -- 172.21.15.89:0/3197437317 shutdown_connections 2024-09-15T23:24:43.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.789+0000 7fc1b4648640 1 -- 172.21.15.89:0/3197437317 wait complete. 2024-09-15T23:24:43.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b4648640 1 Processor -- start 2024-09-15T23:24:43.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b4648640 1 -- start start 2024-09-15T23:24:43.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b4648640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1ac101de0 0x7fc1ac198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b4648640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac102790 0x7fc1ac1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b4648640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1ac106f10 0x7fc1ac19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b4648640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ac111f90 con 0x7fc1ac102790 2024-09-15T23:24:43.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b4648640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ac111e10 con 0x7fc1ac101de0 2024-09-15T23:24:43.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b4648640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ac112110 con 0x7fc1ac106f10 2024-09-15T23:24:43.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b1be3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac102790 0x7fc1ac1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b1be3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac102790 0x7fc1ac1989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38012/0 (socket says 172.21.15.89:38012) 2024-09-15T23:24:43.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b1be3640 1 -- 172.21.15.89:0/2386141897 learned_addr learned my addr 172.21.15.89:0/2386141897 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:43.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b23e4640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1ac101de0 0x7fc1ac198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b1be3640 1 -- 172.21.15.89:0/2386141897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1ac106f10 msgr2=0x7fc1ac19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1ac106f10 0x7fc1ac19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b1be3640 1 -- 172.21.15.89:0/2386141897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1ac101de0 msgr2=0x7fc1ac198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.790+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1ac101de0 0x7fc1ac198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.791+0000 7fc1b1be3640 1 -- 172.21.15.89:0/2386141897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1ac19d420 con 0x7fc1ac102790 2024-09-15T23:24:43.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.791+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac102790 0x7fc1ac1989b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000c890 tx=0x7fc1a000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:43.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.791+0000 7fc1ab7fe640 1 -- 172.21.15.89:0/2386141897 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0004220 con 0x7fc1ac102790 2024-09-15T23:24:43.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.791+0000 7fc1b4648640 1 -- 172.21.15.89:0/2386141897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1ac19d710 con 0x7fc1ac102790 2024-09-15T23:24:43.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.791+0000 7fc1ab7fe640 1 -- 172.21.15.89:0/2386141897 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a00043c0 con 0x7fc1ac102790 2024-09-15T23:24:43.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.791+0000 7fc1ab7fe640 1 -- 172.21.15.89:0/2386141897 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0004a00 con 0x7fc1ac102790 2024-09-15T23:24:43.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.791+0000 7fc1b4648640 1 -- 172.21.15.89:0/2386141897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1ac1a4ff0 con 0x7fc1ac102790 2024-09-15T23:24:43.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.793+0000 7fc1ab7fe640 1 -- 172.21.15.89:0/2386141897 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fc1a00079b0 con 0x7fc1ac102790 2024-09-15T23:24:43.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.793+0000 7fc1b6a21740 1 Processor -- start 2024-09-15T23:24:43.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b6a21740 1 -- start start 2024-09-15T23:24:43.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314da9d0 0x5586314dadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314e5bb0 0x5586314e7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558631441090 con 0x5586314db2f0 2024-09-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5586313e8a70 con 0x5586314da9d0 2024-09-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55863143d050 con 0x5586314e5bb0 2024-09-15T23:24:43.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1ab7fe640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1940a32d0 0x7fc1940a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b23e4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b23e4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38020/0 (socket says 172.21.15.89:38020) 2024-09-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b23e4640 1 -- 172.21.15.89:0/3198598818 learned_addr learned my addr 172.21.15.89:0/3198598818 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b2be5640 1 --2- 172.21.15.89:0/3198598818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314da9d0 0x5586314dadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b2be5640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1940a32d0 0x7fc1940a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1ab7fe640 1 -- 172.21.15.89:0/2386141897 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc1a00cfc40 con 0x7fc1ac102790 2024-09-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/3198598818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314e5bb0 0x5586314e7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b23e4640 1 -- 172.21.15.89:0/3198598818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314e5bb0 msgr2=0x5586314e7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b23e4640 1 --2- 172.21.15.89:0/3198598818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314e5bb0 0x5586314e7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b23e4640 1 -- 172.21.15.89:0/3198598818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314da9d0 msgr2=0x5586314dadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b23e4640 1 --2- 172.21.15.89:0/3198598818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314da9d0 0x5586314dadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.794+0000 7fc1b23e4640 1 -- 172.21.15.89:0/3198598818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586313f5130 con 0x5586314db2f0 2024-09-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.795+0000 7fc1b23e4640 1 --2- 172.21.15.89:0/3198598818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e5670 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc1a400b820 tx=0x7fc1a400bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cfa3c62cd6b5a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.795+0000 7fc1b2be5640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1940a32d0 0x7fc1940a5790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc19c02ade0 tx=0x7fc19c005f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.795+0000 7fc1a8ff9640 1 -- 172.21.15.89:0/3198598818 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a400d950 con 0x5586314db2f0 2024-09-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.795+0000 7fc1a8ff9640 1 -- 172.21.15.89:0/3198598818 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a4019070 con 0x5586314db2f0 2024-09-15T23:24:43.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.795+0000 7fc1a8ff9640 1 -- 172.21.15.89:0/3198598818 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a40123e0 con 0x5586314db2f0 2024-09-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3198598818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 msgr2=0x5586314e5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/3198598818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e5670 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc1a400b820 tx=0x7fc1a400bce0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3198598818 shutdown_connections 2024-09-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/3198598818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314e5bb0 0x5586314e7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/3198598818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e5670 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/3198598818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314da9d0 0x5586314dadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3198598818 >> 172.21.15.89:0/3198598818 conn(0x5586314d6f60 msgr2=0x5586314d7340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3198598818 shutdown_connections 2024-09-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3198598818 wait complete. 2024-09-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 -- start start 2024-09-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.796+0000 7fc1b6a21740 1 Processor -- start 2024-09-15T23:24:43.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314da9d0 0x5586314e53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314e5bb0 0x5586314e2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5586314e8780 con 0x5586314db2f0 2024-09-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5586313fbf20 con 0x5586314e5bb0 2024-09-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5586314e8900 con 0x5586314da9d0 2024-09-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b6a21740 10 client.0 _ll_register_callbacks cb 0x5586313c5030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b1be3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b1be3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38036/0 (socket says 172.21.15.89:38036) 2024-09-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b1be3640 1 -- 172.21.15.89:0/816642044 learned_addr learned my addr 172.21.15.89:0/816642044 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:43.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b23e4640 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314da9d0 0x5586314e53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b1be3640 1 -- 172.21.15.89:0/816642044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314da9d0 msgr2=0x5586314e53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314da9d0 0x5586314e53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b1be3640 1 -- 172.21.15.89:0/816642044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314e5bb0 msgr2=0x5586314e2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314e5bb0 0x5586314e2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b1be3640 1 -- 172.21.15.89:0/816642044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586311b22b0 con 0x5586314db2f0 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e2710 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0007590 tx=0x7fc1a009b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982a430e9c8a3551 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc191ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc191ffb640 1 -- 172.21.15.89:0/816642044 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a001d040 con 0x5586314db2f0 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc191ffb640 1 -- 172.21.15.89:0/816642044 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc1a00ba5e0 con 0x5586314db2f0 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b6a21740 1 -- 172.21.15.89:0/816642044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1a4019070 con 0x5586314db2f0 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc191ffb640 1 -- 172.21.15.89:0/816642044 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc1a00a00c0 con 0x5586314db2f0 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b6a21740 10 client.15498 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:43.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.797+0000 7fc1b6a21740 1 -- 172.21.15.89:0/816642044 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5586314f3fe0 con 0x5586314db2f0 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc191ffb640 1 -- 172.21.15.89:0/816642044 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a4019070 con 0x5586314db2f0 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc191ffb640 1 -- 172.21.15.89:0/816642044 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc1a009b160 con 0x5586314db2f0 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc1b6a21740 10 client.15498 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc1b6a21740 10 client.15498 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc1b6a21740 20 client.15498 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5586314f6690 0x5586314f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc1b6a21740 4 client.15498 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc1b6a21740 1 -- 172.21.15.89:0/816642044 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55863144e600 con 0x5586314f6690 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc1b23e4640 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5586314f6690 0x5586314f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc191ffb640 10 client.15498 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.798+0000 7fc1b23e4640 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5586314f6690 0x5586314f7b00 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fb7451e4358b6d9e server_cookie=3f38e6f164a11a75 in_seq=0 out_seq=0 2024-09-15T23:24:43.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc191ffb640 1 -- 172.21.15.89:0/816642044 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55863144e600 con 0x5586314f6690 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc191ffb640 10 client.15498 handle_command_reply: tid=0 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 client.15498 shutdown 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 -- 172.21.15.89:0/816642044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 msgr2=0x5586314e2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e2710 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0007590 tx=0x7fc1a009b950 comp rx=0 tx=0).stop 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 -- 172.21.15.89:0/816642044 shutdown_connections 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5586314f6690 0x5586314f7b00 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314e5bb0 0x5586314e2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc191ffb640 0 client.15498 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314e2710 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/816642044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314da9d0 0x5586314e53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 -- 172.21.15.89:0/816642044 >> 172.21.15.89:0/816642044 conn(0x5586314d6f60 msgr2=0x5586314dcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 -- 172.21.15.89:0/816642044 shutdown_connections 2024-09-15T23:24:43.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.799+0000 7fc1b6a21740 1 -- 172.21.15.89:0/816642044 wait complete. 2024-09-15T23:24:43.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.800+0000 7fc1b6a21740 20 client.15498 trim_cache size 0 max 16384 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b6a21740 1 Processor -- start 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b6a21740 1 -- start start 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314da9d0 0x5586314f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314f7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314e5bb0 0x5586314e1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5586314f5e40 con 0x5586314db2f0 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5586314f6510 con 0x5586314da9d0 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5586314f6be0 con 0x5586314e5bb0 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b1be3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b1be3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314f7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38044/0 (socket says 172.21.15.89:38044) 2024-09-15T23:24:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b1be3640 1 -- 172.21.15.89:0/2043891861 learned_addr learned my addr 172.21.15.89:0/2043891861 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b2be5640 1 --2- 172.21.15.89:0/2043891861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314e5bb0 0x5586314e1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b1be3640 1 -- 172.21.15.89:0/2043891861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314e5bb0 msgr2=0x5586314e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/2043891861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314e5bb0 0x5586314e1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b1be3640 1 -- 172.21.15.89:0/2043891861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314da9d0 msgr2=0x5586314f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/2043891861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314da9d0 0x5586314f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b1be3640 1 -- 172.21.15.89:0/2043891861 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586314d6d90 con 0x5586314db2f0 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.818+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/2043891861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314f7430 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0098170 tx=0x7fc1a009b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9df1ec68741d870 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc193fff640 1 -- 172.21.15.89:0/2043891861 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a001d070 con 0x5586314db2f0 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc193fff640 1 -- 172.21.15.89:0/2043891861 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a00a4980 con 0x5586314db2f0 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc193fff640 1 -- 172.21.15.89:0/2043891861 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a00a1250 con 0x5586314db2f0 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc1b6a21740 1 -- 172.21.15.89:0/2043891861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 msgr2=0x5586314f7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/2043891861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314f7430 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0098170 tx=0x7fc1a009b5c0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc1b6a21740 1 -- 172.21.15.89:0/2043891861 shutdown_connections 2024-09-15T23:24:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/2043891861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314e5bb0 0x5586314e1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/2043891861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314f7430 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/2043891861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314da9d0 0x5586314f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc1b6a21740 1 -- 172.21.15.89:0/2043891861 >> 172.21.15.89:0/2043891861 conn(0x5586314d6f60 msgr2=0x5586314e26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc1b6a21740 1 -- 172.21.15.89:0/2043891861 shutdown_connections 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.819+0000 7fc1b6a21740 1 -- 172.21.15.89:0/2043891861 wait complete. 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b6a21740 1 -- start start 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b6a21740 1 Processor -- start 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314da9d0 0x5586314fa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314fa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b6a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314e5bb0 0x5586314f9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5586314f7980 con 0x5586314db2f0 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5586314f72b0 con 0x5586314e5bb0 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b6a21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5586314f8050 con 0x5586314da9d0 2024-09-15T23:24:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b6a21740 10 client.0 _ll_register_callbacks cb 0x7fc1a0004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b1be3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314fa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b1be3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314fa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38058/0 (socket says 172.21.15.89:38058) 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b1be3640 1 -- 172.21.15.89:0/3498420815 learned_addr learned my addr 172.21.15.89:0/3498420815 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b2be5640 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314e5bb0 0x5586314f9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b23e4640 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314da9d0 0x5586314fa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b1be3640 1 -- 172.21.15.89:0/3498420815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314da9d0 msgr2=0x5586314fa2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314da9d0 0x5586314fa2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b1be3640 1 -- 172.21.15.89:0/3498420815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314e5bb0 msgr2=0x5586314f9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314e5bb0 0x5586314f9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b1be3640 1 -- 172.21.15.89:0/3498420815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586314f9910 con 0x5586314db2f0 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.820+0000 7fc1b2be5640 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314e5bb0 0x5586314f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc1b1be3640 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314fa830 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0099fc0 tx=0x7fc1a00054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d81f83509974b6d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc191ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc191ffb640 1 -- 172.21.15.89:0/3498420815 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a001d070 con 0x5586314db2f0 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc191ffb640 1 -- 172.21.15.89:0/3498420815 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc1a00a1250 con 0x5586314db2f0 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3498420815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1a00a4980 con 0x5586314db2f0 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc191ffb640 1 -- 172.21.15.89:0/3498420815 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc1a00a1490 con 0x5586314db2f0 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc1b6a21740 10 client.15504 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3498420815 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5586314f9910 con 0x5586314db2f0 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc191ffb640 1 -- 172.21.15.89:0/3498420815 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1a00a4980 con 0x5586314db2f0 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc191ffb640 1 -- 172.21.15.89:0/3498420815 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc1a00ba260 con 0x5586314db2f0 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc1b6a21740 10 client.15504 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc1b6a21740 10 client.15504 resolve_mds: resolved name 'f' to daemon mds.f 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc1b6a21740 20 client.15504 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558631568dc0 0x558631567ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc1b6a21740 4 client.15504 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.821+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3498420815 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "status"}) -- 0x558631448b10 con 0x558631568dc0 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b23e4640 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558631568dc0 0x558631567ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b23e4640 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558631568dc0 0x558631567ad0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=58e8b8a3c9c2ad34 server_cookie=c98597781d238fe3 in_seq=0 out_seq=0 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc191ffb640 10 client.15504 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc191ffb640 1 -- 172.21.15.89:0/3498420815 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558631448b10 con 0x558631568dc0 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc191ffb640 10 client.15504 handle_command_reply: tid=0 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b6a21740 1 client.15504 shutdown 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3498420815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 msgr2=0x5586314fa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314fa830 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0099fc0 tx=0x7fc1a00054a0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3498420815 shutdown_connections 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558631568dc0 0x558631567ad0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586314e5bb0 0x5586314f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5586314db2f0 0x5586314fa830 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc191ffb640 0 client.15504 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b6a21740 1 --2- 172.21.15.89:0/3498420815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5586314da9d0 0x5586314fa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.822+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3498420815 >> 172.21.15.89:0/3498420815 conn(0x5586314d6f60 msgr2=0x5586314e26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3498420815 shutdown_connections 2024-09-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b6a21740 1 -- 172.21.15.89:0/3498420815 wait complete. 2024-09-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b6a21740 20 client.15504 trim_cache size 0 max 16384 2024-09-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 4, 2024-09-15T23:24:43.840 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24424, 2024-09-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 127.697150882, 2024-09-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 29, 2024-09-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 90, 2024-09-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 90, 2024-09-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 197.64126775099999 2024-09-15T23:24:43.841 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 -- 172.21.15.89:0/2386141897 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1940a32d0 msgr2=0x7fc1940a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1940a32d0 0x7fc1940a5790 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc19c02ade0 tx=0x7fc19c005f10 comp rx=0 tx=0).stop 2024-09-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 -- 172.21.15.89:0/2386141897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac102790 msgr2=0x7fc1ac1989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac102790 0x7fc1ac1989b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000c890 tx=0x7fc1a000cd50 comp rx=0 tx=0).stop 2024-09-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 -- 172.21.15.89:0/2386141897 shutdown_connections 2024-09-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1940a32d0 0x7fc1940a5790 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1ac106f10 0x7fc1ac19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac102790 0x7fc1ac1989b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 --2- 172.21.15.89:0/2386141897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1ac101de0 0x7fc1ac198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 -- 172.21.15.89:0/2386141897 >> 172.21.15.89:0/2386141897 conn(0x7fc1ac0fd8c0 msgr2=0x7fc1ac0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 -- 172.21.15.89:0/2386141897 shutdown_connections 2024-09-15T23:24:43.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:43.823+0000 7fc1b4648640 1 -- 172.21.15.89:0/2386141897 wait complete. 2024-09-15T23:24:44.375 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:44 smithi120 ceph-mon[35250]: pgmap v371: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-15T23:24:44.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:44 smithi174 ceph-mon[35586]: pgmap v371: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-15T23:24:44.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:44 smithi089 ceph-mon[30933]: pgmap v371: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-15T23:24:44.638 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-15T23:24:44.883 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:45.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.375+0000 7f984d796640 1 -- 172.21.15.89:0/1509887483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9848101420 msgr2=0x7f984810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:45.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.375+0000 7f984d796640 1 --2- 172.21.15.89:0/1509887483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9848101420 0x7f984810d950 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f9840009930 tx=0x7f984002a420 comp rx=0 tx=0).stop 2024-09-15T23:24:45.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.375+0000 7f984d796640 1 -- 172.21.15.89:0/1509887483 shutdown_connections 2024-09-15T23:24:45.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.375+0000 7f984d796640 1 --2- 172.21.15.89:0/1509887483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9848101420 0x7f984810d950 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.375+0000 7f984d796640 1 --2- 172.21.15.89:0/1509887483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848100a80 0x7f9848100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.375+0000 7f984d796640 1 --2- 172.21.15.89:0/1509887483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9848106a80 0x7f9848106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.375+0000 7f984d796640 1 -- 172.21.15.89:0/1509887483 >> 172.21.15.89:0/1509887483 conn(0x7f98480fc7a0 msgr2=0x7f98480febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:45.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.375+0000 7f984d796640 1 -- 172.21.15.89:0/1509887483 shutdown_connections 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.375+0000 7f984d796640 1 -- 172.21.15.89:0/1509887483 wait complete. 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f984d796640 1 Processor -- start 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f984d796640 1 -- start start 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f984d796640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9848100a80 0x7f98481984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f984d796640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9848101420 0x7f98481989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f984d796640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848106a80 0x7f984819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f984d796640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9848110ec0 con 0x7f9848101420 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f984d796640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9848110d40 con 0x7f9848106a80 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f984d796640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9848111040 con 0x7f9848100a80 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f98467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9848101420 0x7f98481989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f98467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9848101420 0x7f98481989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48988/0 (socket says 172.21.15.89:48988) 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f98467fc640 1 -- 172.21.15.89:0/3728853941 learned_addr learned my addr 172.21.15.89:0/3728853941 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:45.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f98477fe640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848106a80 0x7f984819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9848100a80 0x7f98481984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f98467fc640 1 -- 172.21.15.89:0/3728853941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9848100a80 msgr2=0x7f98481984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f98467fc640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9848100a80 0x7f98481984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f98467fc640 1 -- 172.21.15.89:0/3728853941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848106a80 msgr2=0x7f984819cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.376+0000 7f98467fc640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848106a80 0x7f984819cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.377+0000 7f98467fc640 1 -- 172.21.15.89:0/3728853941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f984819d450 con 0x7f9848101420 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.377+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9848100a80 0x7f98481984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.377+0000 7f98467fc640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9848101420 0x7f98481989e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f983c00b4a0 tx=0x7f983c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.377+0000 7f982ffff640 1 -- 172.21.15.89:0/3728853941 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c014070 con 0x7f9848101420 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.377+0000 7f984d796640 1 -- 172.21.15.89:0/3728853941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f984819d740 con 0x7f9848101420 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.377+0000 7f982ffff640 1 -- 172.21.15.89:0/3728853941 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f983c010af0 con 0x7f9848101420 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.377+0000 7f982ffff640 1 -- 172.21.15.89:0/3728853941 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c017d10 con 0x7f9848101420 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.377+0000 7f984d796640 1 -- 172.21.15.89:0/3728853941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98481a5020 con 0x7f9848101420 2024-09-15T23:24:45.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.379+0000 7f982ffff640 1 -- 172.21.15.89:0/3728853941 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f983c017070 con 0x7f9848101420 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f984fb6f740 1 Processor -- start 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f984fb6f740 1 -- start start 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d24eddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d24f8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 0x55c9d24fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f984fb6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c9d2454090 con 0x55c9d24ed9d0 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f984fb6f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c9d23fba70 con 0x55c9d24f8bb0 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f984fb6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c9d2450050 con 0x55c9d24ee2f0 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f98477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d24eddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f982ffff640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98300a32d0 0x7f98300a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f98477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d24eddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48990/0 (socket says 172.21.15.89:48990) 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f98477fe640 1 -- 172.21.15.89:0/3091410799 learned_addr learned my addr 172.21.15.89:0/3091410799 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:45.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f98467fc640 1 --2- 172.21.15.89:0/3091410799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 0x55c9d24fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3091410799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d24f8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f98477fe640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98300a32d0 0x7f98300a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f982ffff640 1 -- 172.21.15.89:0/3728853941 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f983c0cf200 con 0x7f9848101420 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f98477fe640 1 -- 172.21.15.89:0/3091410799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 msgr2=0x55c9d24f8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f98477fe640 1 --2- 172.21.15.89:0/3091410799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d24f8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f98477fe640 1 -- 172.21.15.89:0/3091410799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 msgr2=0x55c9d24fafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f98477fe640 1 --2- 172.21.15.89:0/3091410799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 0x55c9d24fafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.380+0000 7f98477fe640 1 -- 172.21.15.89:0/3091410799 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9d2408130 con 0x55c9d24ed9d0 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.381+0000 7f98477fe640 1 --2- 172.21.15.89:0/3091410799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d24eddb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9840005940 tx=0x7f98400318f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3c031d9ef5158a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.381+0000 7f982d7fa640 1 -- 172.21.15.89:0/3091410799 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840035070 con 0x55c9d24ed9d0 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.381+0000 7f982d7fa640 1 -- 172.21.15.89:0/3091410799 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98400044c0 con 0x55c9d24ed9d0 2024-09-15T23:24:45.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.381+0000 7f982d7fa640 1 -- 172.21.15.89:0/3091410799 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98400035f0 con 0x55c9d24ed9d0 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 -- 172.21.15.89:0/3091410799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 msgr2=0x55c9d24eddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f98477fe640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98300a32d0 0x7f98300a5790 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f984004a000 tx=0x7f9840048000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3091410799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d24eddb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9840005940 tx=0x7f98400318f0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 -- 172.21.15.89:0/3091410799 shutdown_connections 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3091410799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 0x55c9d24fafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3091410799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d24f8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3091410799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d24eddb0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 -- 172.21.15.89:0/3091410799 >> 172.21.15.89:0/3091410799 conn(0x55c9d24e9f60 msgr2=0x55c9d24ea340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 -- 172.21.15.89:0/3091410799 shutdown_connections 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 -- 172.21.15.89:0/3091410799 wait complete. 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 -- start start 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 Processor -- start 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ed9d0 0x55c9d24f83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d24f5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c9d24fb780 con 0x55c9d24f8bb0 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c9d240ef20 con 0x55c9d24ed9d0 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c9d24fb900 con 0x55c9d24ee2f0 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.382+0000 7f984fb6f740 10 client.0 _ll_register_callbacks cb 0x55c9d23d8030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48998/0 (socket says 172.21.15.89:48998) 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98477fe640 1 -- 172.21.15.89:0/3706374138 learned_addr learned my addr 172.21.15.89:0/3706374138 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ed9d0 0x55c9d24f83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98467fc640 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d24f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98477fe640 1 -- 172.21.15.89:0/3706374138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 msgr2=0x55c9d24f5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98477fe640 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d24f5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98477fe640 1 -- 172.21.15.89:0/3706374138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ed9d0 msgr2=0x55c9d24f83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98477fe640 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ed9d0 0x55c9d24f83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98477fe640 1 -- 172.21.15.89:0/3706374138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9d21c52b0 con 0x55c9d24f8bb0 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98467fc640 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d24f5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ed9d0 0x55c9d24f83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f98477fe640 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f5c60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f984002a460 tx=0x7f9840004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159e515bc365f06c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f982aff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f982aff5640 1 -- 172.21.15.89:0/3706374138 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984005b060 con 0x55c9d24f8bb0 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f982aff5640 1 -- 172.21.15.89:0/3706374138 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9840003ca0 con 0x55c9d24f8bb0 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f984fb6f740 1 -- 172.21.15.89:0/3706374138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c9d2461600 con 0x55c9d24f8bb0 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f982aff5640 1 -- 172.21.15.89:0/3706374138 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9840062050 con 0x55c9d24f8bb0 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f984fb6f740 10 client.15516 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:45.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f984fb6f740 1 -- 172.21.15.89:0/3706374138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c9d2507030 con 0x55c9d24f8bb0 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.383+0000 7f982aff5640 1 -- 172.21.15.89:0/3706374138 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c9d2461600 con 0x55c9d24f8bb0 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.384+0000 7f982aff5640 1 -- 172.21.15.89:0/3706374138 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f984003a480 con 0x55c9d24f8bb0 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.384+0000 7f984fb6f740 10 client.15516 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.384+0000 7f984fb6f740 10 client.15516 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.384+0000 7f984fb6f740 20 client.15516 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.384+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c9d25091a0 0x55c9d250a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.384+0000 7f984fb6f740 4 client.15516 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.384+0000 7f984fb6f740 1 -- 172.21.15.89:0/3706374138 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9d245bb10 con 0x55c9d25091a0 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.384+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c9d25091a0 0x55c9d250a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.384+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c9d25091a0 0x55c9d250a610 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b102c3ac14f0567a server_cookie=d395f1cc6416f423 in_seq=0 out_seq=0 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.384+0000 7f982aff5640 10 client.15516 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f982aff5640 1 -- 172.21.15.89:0/3706374138 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c9d245bb10 con 0x55c9d25091a0 2024-09-15T23:24:45.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f982aff5640 10 client.15516 handle_command_reply: tid=0 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f984fb6f740 1 client.15516 shutdown 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f984fb6f740 1 -- 172.21.15.89:0/3706374138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 msgr2=0x55c9d24f5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f5c60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f984002a460 tx=0x7f9840004d00 comp rx=0 tx=0).stop 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f984fb6f740 1 -- 172.21.15.89:0/3706374138 shutdown_connections 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c9d25091a0 0x55c9d250a610 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f5c60 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d24f5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3706374138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ed9d0 0x55c9d24f83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f984fb6f740 1 -- 172.21.15.89:0/3706374138 >> 172.21.15.89:0/3706374138 conn(0x55c9d24e9f60 msgr2=0x55c9d24efc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.385+0000 7f982aff5640 0 client.15516 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.386+0000 7f984fb6f740 1 -- 172.21.15.89:0/3706374138 shutdown_connections 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.386+0000 7f984fb6f740 1 -- 172.21.15.89:0/3706374138 wait complete. 2024-09-15T23:24:45.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.386+0000 7f984fb6f740 20 client.15516 trim_cache size 0 max 16384 2024-09-15T23:24:45.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f984fb6f740 1 Processor -- start 2024-09-15T23:24:45.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f984fb6f740 1 -- start start 2024-09-15T23:24:45.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d2508ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d2509f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f984fb6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c9d2508950 con 0x55c9d24ed9d0 2024-09-15T23:24:45.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f984fb6f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c9d2509020 con 0x55c9d24f8bb0 2024-09-15T23:24:45.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f984fb6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c9d25096f0 con 0x55c9d24ee2f0 2024-09-15T23:24:45.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f9846ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d2508ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f9846ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d2508ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49000/0 (socket says 172.21.15.89:49000) 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f9846ffd640 1 -- 172.21.15.89:0/3546708144 learned_addr learned my addr 172.21.15.89:0/3546708144 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f98477fe640 1 --2- 172.21.15.89:0/3546708144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.404+0000 7f98467fc640 1 --2- 172.21.15.89:0/3546708144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d2509f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.405+0000 7f9846ffd640 1 -- 172.21.15.89:0/3546708144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 msgr2=0x55c9d2509f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.405+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3546708144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d2509f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.405+0000 7f9846ffd640 1 -- 172.21.15.89:0/3546708144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 msgr2=0x55c9d24f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.405+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3546708144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.405+0000 7f9846ffd640 1 -- 172.21.15.89:0/3546708144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9d24ee120 con 0x55c9d24ed9d0 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.405+0000 7f98477fe640 1 --2- 172.21.15.89:0/3546708144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.405+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3546708144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d2508ad0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9838005e60 tx=0x7f9838013900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1aed6f6348dbebd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:45.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.405+0000 7f982cff9640 1 -- 172.21.15.89:0/3546708144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838004b20 con 0x55c9d24ed9d0 2024-09-15T23:24:45.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.405+0000 7f982cff9640 1 -- 172.21.15.89:0/3546708144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9838004cc0 con 0x55c9d24ed9d0 2024-09-15T23:24:45.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.405+0000 7f982cff9640 1 -- 172.21.15.89:0/3546708144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838005440 con 0x55c9d24ed9d0 2024-09-15T23:24:45.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 -- 172.21.15.89:0/3546708144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 msgr2=0x55c9d2508ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3546708144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d2508ad0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9838005e60 tx=0x7f9838013900 comp rx=0 tx=0).stop 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 -- 172.21.15.89:0/3546708144 shutdown_connections 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3546708144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24f8bb0 0x55c9d24f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3546708144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ee2f0 0x55c9d2509f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3546708144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24ed9d0 0x55c9d2508ad0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 -- 172.21.15.89:0/3546708144 >> 172.21.15.89:0/3546708144 conn(0x55c9d24e9f60 msgr2=0x55c9d24f6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 -- 172.21.15.89:0/3546708144 shutdown_connections 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 -- 172.21.15.89:0/3546708144 wait complete. 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 -- start start 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 Processor -- start 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ed9d0 0x55c9d250d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ee2f0 0x55c9d250dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d250e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c9d250a490 con 0x55c9d24f8bb0 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c9d2509dc0 con 0x55c9d24ee2f0 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c9d250ab60 con 0x55c9d24ed9d0 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f984fb6f740 10 client.0 _ll_register_callbacks cb 0x7f983c095f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.406+0000 7f98477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d250e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f9846ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ed9d0 0x55c9d250d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f98477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d250e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49014/0 (socket says 172.21.15.89:49014) 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f9846ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ed9d0 0x55c9d250d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:33782/0 (socket says 172.21.15.89:33782) 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f98477fe640 1 -- 172.21.15.89:0/3986689617 learned_addr learned my addr 172.21.15.89:0/3986689617 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f98467fc640 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ee2f0 0x55c9d250dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f98477fe640 1 -- 172.21.15.89:0/3986689617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ed9d0 msgr2=0x55c9d250d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f98477fe640 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ed9d0 0x55c9d250d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f98477fe640 1 -- 172.21.15.89:0/3986689617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ee2f0 msgr2=0x55c9d250dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f98477fe640 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ee2f0 0x55c9d250dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f98477fe640 1 -- 172.21.15.89:0/3986689617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9d2571a70 con 0x55c9d24f8bb0 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f98477fe640 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d250e2e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f984002a940 tx=0x7f984003afe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdf1f6ba39524a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f982aff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f982aff5640 1 -- 172.21.15.89:0/3986689617 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984006b060 con 0x55c9d24f8bb0 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f982aff5640 1 -- 172.21.15.89:0/3986689617 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f984003aa20 con 0x55c9d24f8bb0 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f984fb6f740 1 -- 172.21.15.89:0/3986689617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9838004cc0 con 0x55c9d24f8bb0 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f982aff5640 1 -- 172.21.15.89:0/3986689617 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9840072050 con 0x55c9d24f8bb0 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f984fb6f740 10 client.15522 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.407+0000 7f984fb6f740 1 -- 172.21.15.89:0/3986689617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c9d2572d50 con 0x55c9d24f8bb0 2024-09-15T23:24:45.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f982aff5640 1 -- 172.21.15.89:0/3986689617 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9838004cc0 con 0x55c9d24f8bb0 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f982aff5640 1 -- 172.21.15.89:0/3986689617 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9840004d70 con 0x55c9d24f8bb0 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f984fb6f740 10 client.15522 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f984fb6f740 10 client.15522 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f984fb6f740 20 client.15522 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c9d2577950 0x55c9d2576660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f984fb6f740 4 client.15522 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f984fb6f740 1 -- 172.21.15.89:0/3986689617 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "status"}) -- 0x55c9d23d80e0 con 0x55c9d2577950 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c9d2577950 0x55c9d2576660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f9846ffd640 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c9d2577950 0x55c9d2576660 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=30f974724eb2f622 server_cookie=2535500a895f2c00 in_seq=0 out_seq=0 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.408+0000 7f982aff5640 10 client.15522 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f982aff5640 1 -- 172.21.15.89:0/3986689617 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c9d23d80e0 con 0x55c9d2577950 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f982aff5640 10 client.15522 handle_command_reply: tid=0 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 client.15522 shutdown 2024-09-15T23:24:45.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 -- 172.21.15.89:0/3986689617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 msgr2=0x55c9d250e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d250e2e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f984002a940 tx=0x7f984003afe0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 -- 172.21.15.89:0/3986689617 shutdown_connections 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c9d2577950 0x55c9d2576660 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9d24f8bb0 0x55c9d250e2e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9d24ee2f0 0x55c9d250dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f982aff5640 0 client.15522 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 --2- 172.21.15.89:0/3986689617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c9d24ed9d0 0x55c9d250d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 -- 172.21.15.89:0/3986689617 >> 172.21.15.89:0/3986689617 conn(0x55c9d24e9f60 msgr2=0x55c9d24f7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 -- 172.21.15.89:0/3986689617 shutdown_connections 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 1 -- 172.21.15.89:0/3986689617 wait complete. 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984fb6f740 20 client.15522 trim_cache size 0 max 16384 2024-09-15T23:24:45.417 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 3, 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24428, 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 130.29915733000001, 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 29, 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 90, 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 90, 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 193.966725698 2024-09-15T23:24:45.418 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:45.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984d796640 1 -- 172.21.15.89:0/3728853941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98300a32d0 msgr2=0x7f98300a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:45.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984d796640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98300a32d0 0x7f98300a5790 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f984004a000 tx=0x7f9840048000 comp rx=0 tx=0).stop 2024-09-15T23:24:45.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984d796640 1 -- 172.21.15.89:0/3728853941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9848101420 msgr2=0x7f98481989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:45.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.409+0000 7f984d796640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9848101420 0x7f98481989e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f983c00b4a0 tx=0x7f983c00b960 comp rx=0 tx=0).stop 2024-09-15T23:24:45.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.410+0000 7f984d796640 1 -- 172.21.15.89:0/3728853941 shutdown_connections 2024-09-15T23:24:45.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.410+0000 7f984d796640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98300a32d0 0x7f98300a5790 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.410+0000 7f984d796640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9848106a80 0x7f984819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.410+0000 7f984d796640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9848101420 0x7f98481989e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.410+0000 7f984d796640 1 --2- 172.21.15.89:0/3728853941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9848100a80 0x7f98481984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:45.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.410+0000 7f984d796640 1 -- 172.21.15.89:0/3728853941 >> 172.21.15.89:0/3728853941 conn(0x7f98480fc7a0 msgr2=0x7f98481047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:45.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.410+0000 7f984d796640 1 -- 172.21.15.89:0/3728853941 shutdown_connections 2024-09-15T23:24:45.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:45.410+0000 7f984d796640 1 -- 172.21.15.89:0/3728853941 wait complete. 2024-09-15T23:24:46.096 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.c status 2024-09-15T23:24:46.341 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:46.462 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:46 smithi089 ceph-mon[30933]: pgmap v372: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-09-15T23:24:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:46 smithi120 ceph-mon[35250]: pgmap v372: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-09-15T23:24:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:46 smithi174 ceph-mon[35586]: pgmap v372: 129 pgs: 129 active+clean; 653 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-09-15T23:24:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.810+0000 7f1f4e674640 1 -- 172.21.15.89:0/2101331511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f481027a0 msgr2=0x7f1f481067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.810+0000 7f1f4e674640 1 --2- 172.21.15.89:0/2101331511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f481027a0 0x7f1f481067f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1f30009900 tx=0x7f1f3002a410 comp rx=0 tx=0).stop 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.810+0000 7f1f4e674640 1 -- 172.21.15.89:0/2101331511 shutdown_connections 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.810+0000 7f1f4e674640 1 --2- 172.21.15.89:0/2101331511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f48106f20 0x7f1f4810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.810+0000 7f1f4e674640 1 --2- 172.21.15.89:0/2101331511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f481027a0 0x7f1f481067f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.810+0000 7f1f4e674640 1 --2- 172.21.15.89:0/2101331511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f48101df0 0x7f1f481021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.810+0000 7f1f4e674640 1 -- 172.21.15.89:0/2101331511 >> 172.21.15.89:0/2101331511 conn(0x7f1f480fd8f0 msgr2=0x7f1f480ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.810+0000 7f1f4e674640 1 -- 172.21.15.89:0/2101331511 shutdown_connections 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.810+0000 7f1f4e674640 1 -- 172.21.15.89:0/2101331511 wait complete. 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f4e674640 1 Processor -- start 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f4e674640 1 -- start start 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f4e674640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f48101df0 0x7f1f481984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f4e674640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f481027a0 0x7f1f481989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f4e674640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f48106f20 0x7f1f4819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f4e674640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1f48111fa0 con 0x7f1f481027a0 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f4e674640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1f48111e20 con 0x7f1f48106f20 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f4e674640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1f48112120 con 0x7f1f48101df0 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f3ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f481027a0 0x7f1f481989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f3ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f481027a0 0x7f1f481989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49040/0 (socket says 172.21.15.89:49040) 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f3ffff640 1 -- 172.21.15.89:0/3015727543 learned_addr learned my addr 172.21.15.89:0/3015727543 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f47fff640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f48101df0 0x7f1f481984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f4cc11640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f48106f20 0x7f1f4819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f3ffff640 1 -- 172.21.15.89:0/3015727543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f48101df0 msgr2=0x7f1f481984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f48101df0 0x7f1f481984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f3ffff640 1 -- 172.21.15.89:0/3015727543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f48106f20 msgr2=0x7f1f4819cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f48106f20 0x7f1f4819cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f3ffff640 1 -- 172.21.15.89:0/3015727543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4819d450 con 0x7f1f481027a0 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.811+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f481027a0 0x7f1f481989e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1f30006010 tx=0x7f1f300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.812+0000 7f1f45ffb640 1 -- 172.21.15.89:0/3015727543 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3003f030 con 0x7f1f481027a0 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.812+0000 7f1f45ffb640 1 -- 172.21.15.89:0/3015727543 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f30003ce0 con 0x7f1f481027a0 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.812+0000 7f1f4e674640 1 -- 172.21.15.89:0/3015727543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f4819d6e0 con 0x7f1f481027a0 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.812+0000 7f1f45ffb640 1 -- 172.21.15.89:0/3015727543 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30035070 con 0x7f1f481027a0 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.812+0000 7f1f4e674640 1 -- 172.21.15.89:0/3015727543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f481a5020 con 0x7f1f481027a0 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.814+0000 7f1f45ffb640 1 -- 172.21.15.89:0/3015727543 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f1f30042030 con 0x7f1f481027a0 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.814+0000 7f1f50a4d740 1 Processor -- start 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.814+0000 7f1f50a4d740 1 -- start start 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e84bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84c2f0 0x564e1e856670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e858fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564e1e7b2090 con 0x564e1e856bb0 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e1e759a70 con 0x564e1e84c2f0 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e1e7ae050 con 0x564e1e84b9d0 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f3ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e858fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f4cc11640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e84bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f3ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e858fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49046/0 (socket says 172.21.15.89:49046) 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f3ffff640 1 -- 172.21.15.89:0/3008606711 learned_addr learned my addr 172.21.15.89:0/3008606711 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f45ffb640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f340a3350 0x7f1f340a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f47fff640 1 --2- 172.21.15.89:0/3008606711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84c2f0 0x564e1e856670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f3ffff640 1 -- 172.21.15.89:0/3008606711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 msgr2=0x564e1e84bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/3008606711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e84bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f3ffff640 1 -- 172.21.15.89:0/3008606711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84c2f0 msgr2=0x564e1e856670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/3008606711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84c2f0 0x564e1e856670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f3ffff640 1 -- 172.21.15.89:0/3008606711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e1e766130 con 0x564e1e856bb0 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f4cc11640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f340a3350 0x7f1f340a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f45ffb640 1 -- 172.21.15.89:0/3015727543 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1f300f0550 con 0x7f1f481027a0 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.815+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/3008606711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e858fa0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1f300a0c40 tx=0x7f1f300870d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74148eab12f47dbe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f4cc11640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f340a3350 0x7f1f340a5810 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1f40005fd0 tx=0x7f1f40004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f3e7fc640 1 -- 172.21.15.89:0/3008606711 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f300bc1c0 con 0x564e1e856bb0 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f3e7fc640 1 -- 172.21.15.89:0/3008606711 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f3009e040 con 0x564e1e856bb0 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f3e7fc640 1 -- 172.21.15.89:0/3008606711 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3003bcb0 con 0x564e1e856bb0 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f50a4d740 1 -- 172.21.15.89:0/3008606711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 msgr2=0x564e1e858fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/3008606711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e858fa0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1f300a0c40 tx=0x7f1f300870d0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f50a4d740 1 -- 172.21.15.89:0/3008606711 shutdown_connections 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/3008606711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e858fa0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/3008606711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84c2f0 0x564e1e856670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/3008606711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e84bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f50a4d740 1 -- 172.21.15.89:0/3008606711 >> 172.21.15.89:0/3008606711 conn(0x564e1e847f60 msgr2=0x564e1e848340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f50a4d740 1 -- 172.21.15.89:0/3008606711 shutdown_connections 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.816+0000 7f1f50a4d740 1 -- 172.21.15.89:0/3008606711 wait complete. 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f50a4d740 1 -- start start 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f50a4d740 1 Processor -- start 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84b9d0 0x564e1e8563c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84c2f0 0x564e1e853710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e853c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564e1e859540 con 0x564e1e856bb0 2024-09-15T23:24:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e1e76cf20 con 0x564e1e84b9d0 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e1e8596c0 con 0x564e1e84c2f0 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f50a4d740 10 client.0 _ll_register_callbacks cb 0x564e1e736030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f4cc11640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e853c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f3ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84c2f0 0x564e1e853710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f4cc11640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e853c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49054/0 (socket says 172.21.15.89:49054) 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.817+0000 7f1f4cc11640 1 -- 172.21.15.89:0/2054021410 learned_addr learned my addr 172.21.15.89:0/2054021410 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f47fff640 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84b9d0 0x564e1e8563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f4cc11640 1 -- 172.21.15.89:0/2054021410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84c2f0 msgr2=0x564e1e853710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f4cc11640 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84c2f0 0x564e1e853710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f4cc11640 1 -- 172.21.15.89:0/2054021410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84b9d0 msgr2=0x564e1e8563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f4cc11640 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84b9d0 0x564e1e8563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f4cc11640 1 -- 172.21.15.89:0/2054021410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e1e5232b0 con 0x564e1e856bb0 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f47fff640 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84b9d0 0x564e1e8563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f4cc11640 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e853c50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1f40013a00 tx=0x7f1f40004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eafd54128f438806 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f277fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f277fe640 1 -- 172.21.15.89:0/2054021410 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4001ed70 con 0x564e1e856bb0 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f277fe640 1 -- 172.21.15.89:0/2054021410 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1f4001f450 con 0x564e1e856bb0 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f50a4d740 1 -- 172.21.15.89:0/2054021410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f3009e040 con 0x564e1e856bb0 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f277fe640 1 -- 172.21.15.89:0/2054021410 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1f4001f650 con 0x564e1e856bb0 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f50a4d740 10 client.15534 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f50a4d740 1 -- 172.21.15.89:0/2054021410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564e1e864da0 con 0x564e1e856bb0 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.818+0000 7f1f277fe640 1 -- 172.21.15.89:0/2054021410 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f3009e040 con 0x564e1e856bb0 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.819+0000 7f1f277fe640 1 -- 172.21.15.89:0/2054021410 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1f4000f040 con 0x564e1e856bb0 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.819+0000 7f1f50a4d740 10 client.15534 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.819+0000 7f1f50a4d740 10 client.15534 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.819+0000 7f1f50a4d740 20 client.15534 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.819+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e1e867390 0x564e1e868800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.819+0000 7f1f50a4d740 4 client.15534 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.819+0000 7f1f50a4d740 1 -- 172.21.15.89:0/2054021410 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e1e7bf600 con 0x564e1e867390 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.819+0000 7f1f47fff640 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e1e867390 0x564e1e868800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f277fe640 10 client.15534 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f47fff640 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e1e867390 0x564e1e868800 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=893105e73e33420a server_cookie=15e6b63a63f0f487 in_seq=0 out_seq=0 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f277fe640 1 -- 172.21.15.89:0/2054021410 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564e1e7bf600 con 0x564e1e867390 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f277fe640 10 client.15534 handle_command_reply: tid=0 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f50a4d740 1 client.15534 shutdown 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f50a4d740 1 -- 172.21.15.89:0/2054021410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 msgr2=0x564e1e853c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e853c50 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1f40013a00 tx=0x7f1f40004ab0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f50a4d740 1 -- 172.21.15.89:0/2054021410 shutdown_connections 2024-09-15T23:24:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e856bb0 0x564e1e853c50 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e1e867390 0x564e1e868800 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84c2f0 0x564e1e853710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/2054021410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e84b9d0 0x564e1e8563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f50a4d740 1 -- 172.21.15.89:0/2054021410 >> 172.21.15.89:0/2054021410 conn(0x564e1e847f60 msgr2=0x564e1e84dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.820+0000 7f1f277fe640 0 client.15534 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:24:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.821+0000 7f1f50a4d740 1 -- 172.21.15.89:0/2054021410 shutdown_connections 2024-09-15T23:24:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.821+0000 7f1f50a4d740 1 -- 172.21.15.89:0/2054021410 wait complete. 2024-09-15T23:24:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.821+0000 7f1f50a4d740 20 client.15534 trim_cache size 0 max 16384 2024-09-15T23:24:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f50a4d740 1 Processor -- start 2024-09-15T23:24:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f50a4d740 1 -- start start 2024-09-15T23:24:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e867390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e862590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e856bb0 0x564e1e852fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564e1e866b40 con 0x564e1e84c2f0 2024-09-15T23:24:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e1e867210 con 0x564e1e856bb0 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e1e8678e0 con 0x564e1e84b9d0 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f3ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e862590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f3ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e862590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49060/0 (socket says 172.21.15.89:49060) 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.838+0000 7f1f3ffff640 1 -- 172.21.15.89:0/1642641406 learned_addr learned my addr 172.21.15.89:0/1642641406 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f4cc11640 1 --2- 172.21.15.89:0/1642641406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e856bb0 0x564e1e852fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f47fff640 1 --2- 172.21.15.89:0/1642641406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e867390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f3ffff640 1 -- 172.21.15.89:0/1642641406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 msgr2=0x564e1e867390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/1642641406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e867390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f3ffff640 1 -- 172.21.15.89:0/1642641406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e856bb0 msgr2=0x564e1e852fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/1642641406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e856bb0 0x564e1e852fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f3ffff640 1 -- 172.21.15.89:0/1642641406 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e1e856730 con 0x564e1e84c2f0 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f47fff640 1 --2- 172.21.15.89:0/1642641406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e867390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/1642641406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e862590 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1f300b9050 tx=0x7f1f30005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1e0ba3b4a6c78d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f3dffb640 1 -- 172.21.15.89:0/1642641406 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30049070 con 0x564e1e84c2f0 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f3dffb640 1 -- 172.21.15.89:0/1642641406 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f3003b580 con 0x564e1e84c2f0 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.839+0000 7f1f3dffb640 1 -- 172.21.15.89:0/1642641406 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3008c420 con 0x564e1e84c2f0 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 -- 172.21.15.89:0/1642641406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 msgr2=0x564e1e862590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/1642641406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e862590 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1f300b9050 tx=0x7f1f30005b50 comp rx=0 tx=0).stop 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 -- 172.21.15.89:0/1642641406 shutdown_connections 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/1642641406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e856bb0 0x564e1e852fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/1642641406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e862590 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/1642641406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e867390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 -- 172.21.15.89:0/1642641406 >> 172.21.15.89:0/1642641406 conn(0x564e1e847f60 msgr2=0x564e1e8536d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 -- 172.21.15.89:0/1642641406 shutdown_connections 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 -- 172.21.15.89:0/1642641406 wait complete. 2024-09-15T23:24:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 -- start start 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 Processor -- start 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e86deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e86e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e856bb0 0x564e1e86e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564e1e868680 con 0x564e1e84c2f0 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e1e867fb0 con 0x564e1e856bb0 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.840+0000 7f1f50a4d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e1e868d50 con 0x564e1e84b9d0 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f50a4d740 10 client.0 _ll_register_callbacks cb 0x7f1f40009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f3ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e86e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f3ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e86e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49062/0 (socket says 172.21.15.89:49062) 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f3ffff640 1 -- 172.21.15.89:0/4011342647 learned_addr learned my addr 172.21.15.89:0/4011342647 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f47fff640 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e86deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f3ffff640 1 -- 172.21.15.89:0/4011342647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 msgr2=0x564e1e86deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f4cc11640 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e856bb0 0x564e1e86e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e86deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f3ffff640 1 -- 172.21.15.89:0/4011342647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e856bb0 msgr2=0x564e1e86e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e856bb0 0x564e1e86e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f3ffff640 1 -- 172.21.15.89:0/4011342647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e1e8d3ac0 con 0x564e1e84c2f0 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f3ffff640 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e86e3f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1f30004650 tx=0x7f1f300bcd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3841b2dd6185c988 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f277fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f277fe640 1 -- 172.21.15.89:0/4011342647 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f30049030 con 0x564e1e84c2f0 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f277fe640 1 -- 172.21.15.89:0/4011342647 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1f300b8cf0 con 0x564e1e84c2f0 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f50a4d740 1 -- 172.21.15.89:0/4011342647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564e1e7b9b10 con 0x564e1e84c2f0 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f277fe640 1 -- 172.21.15.89:0/4011342647 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1f3008c7a0 con 0x564e1e84c2f0 2024-09-15T23:24:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f50a4d740 10 client.15540 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.841+0000 7f1f50a4d740 1 -- 172.21.15.89:0/4011342647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564e1e8d4d00 con 0x564e1e84c2f0 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f277fe640 1 -- 172.21.15.89:0/4011342647 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564e1e7b9b10 con 0x564e1e84c2f0 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f277fe640 1 -- 172.21.15.89:0/4011342647 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1f300edd00 con 0x564e1e84c2f0 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f50a4d740 10 client.15540 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f50a4d740 10 client.15540 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f50a4d740 20 client.15540 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e1e8d9470 0x564e1e8d8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f50a4d740 4 client.15540 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f50a4d740 1 -- 172.21.15.89:0/4011342647 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "status"}) -- 0x564e1e7360e0 con 0x564e1e8d9470 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f47fff640 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e1e8d9470 0x564e1e8d8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f47fff640 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e1e8d9470 0x564e1e8d8180 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ccd2b38986caa4d5 server_cookie=c7b99214b16814c8 in_seq=0 out_seq=0 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.842+0000 7f1f277fe640 10 client.15540 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f277fe640 1 -- 172.21.15.89:0/4011342647 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564e1e7360e0 con 0x564e1e8d9470 2024-09-15T23:24:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f277fe640 10 client.15540 handle_command_reply: tid=0 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 client.15540 shutdown 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 -- 172.21.15.89:0/4011342647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 msgr2=0x564e1e86e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e86e3f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1f30004650 tx=0x7f1f300bcd80 comp rx=0 tx=0).stop 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 -- 172.21.15.89:0/4011342647 shutdown_connections 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e1e856bb0 0x564e1e86e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e1e84c2f0 0x564e1e86e3f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e1e8d9470 0x564e1e8d8180 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 --2- 172.21.15.89:0/4011342647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e1e84b9d0 0x564e1e86deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 -- 172.21.15.89:0/4011342647 >> 172.21.15.89:0/4011342647 conn(0x564e1e847f60 msgr2=0x564e1e866ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f277fe640 0 client.15540 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 -- 172.21.15.89:0/4011342647 shutdown_connections 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 1 -- 172.21.15.89:0/4011342647 wait complete. 2024-09-15T23:24:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f50a4d740 20 client.15540 trim_cache size 0 max 16384 2024-09-15T23:24:46.861 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:24:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 2, 2024-09-15T23:24:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24430, 2024-09-15T23:24:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:24:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:24:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:24:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 132.74967503600001, 2024-09-15T23:24:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 29, 2024-09-15T23:24:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 90, 2024-09-15T23:24:46.862 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 90, 2024-09-15T23:24:46.862 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 198.083195351 2024-09-15T23:24:46.862 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f4e674640 1 -- 172.21.15.89:0/3015727543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f340a3350 msgr2=0x7f1f340a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f4e674640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f340a3350 0x7f1f340a5810 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1f40005fd0 tx=0x7f1f40004a40 comp rx=0 tx=0).stop 2024-09-15T23:24:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f4e674640 1 -- 172.21.15.89:0/3015727543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f481027a0 msgr2=0x7f1f481989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.843+0000 7f1f4e674640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f481027a0 0x7f1f481989e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1f30006010 tx=0x7f1f300042f0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.844+0000 7f1f4e674640 1 -- 172.21.15.89:0/3015727543 shutdown_connections 2024-09-15T23:24:46.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.844+0000 7f1f4e674640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f340a3350 0x7f1f340a5810 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.844+0000 7f1f4e674640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f48106f20 0x7f1f4819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.844+0000 7f1f4e674640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f481027a0 0x7f1f481989e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.844+0000 7f1f4e674640 1 --2- 172.21.15.89:0/3015727543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f48101df0 0x7f1f481984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.844+0000 7f1f4e674640 1 -- 172.21.15.89:0/3015727543 >> 172.21.15.89:0/3015727543 conn(0x7f1f480fd8f0 msgr2=0x7f1f480fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:46.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.844+0000 7f1f4e674640 1 -- 172.21.15.89:0/3015727543 shutdown_connections 2024-09-15T23:24:46.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:46.844+0000 7f1f4e674640 1 -- 172.21.15.89:0/3015727543 wait complete. 2024-09-15T23:24:47.673 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-15T23:24:47.918 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:48.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:48 smithi089 ceph-mon[30933]: pgmap v373: 129 pgs: 129 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.7 KiB/s wr, 16 op/s 2024-09-15T23:24:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:48 smithi120 ceph-mon[35250]: pgmap v373: 129 pgs: 129 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.7 KiB/s wr, 16 op/s 2024-09-15T23:24:48.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2970557564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cd0100e20 msgr2=0x7f6cd0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 --2- 172.21.15.89:0/2970557564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cd0100e20 0x7f6cd0101200 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009900 tx=0x7f6cc002a480 comp rx=0 tx=0).stop 2024-09-15T23:24:48.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2970557564 shutdown_connections 2024-09-15T23:24:48.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 --2- 172.21.15.89:0/2970557564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd0102ef0 0x7f6cd010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 --2- 172.21.15.89:0/2970557564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cd0102570 0x7f6cd01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 --2- 172.21.15.89:0/2970557564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cd0100e20 0x7f6cd0101200 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2970557564 >> 172.21.15.89:0/2970557564 conn(0x7f6cd00fc7d0 msgr2=0x7f6cd00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2970557564 shutdown_connections 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2970557564 wait complete. 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 Processor -- start 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.461+0000 7f6cd81a2640 1 -- start start 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd81a2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cd0100e20 0x7f6cd01993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd81a2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cd0102570 0x7f6cd0199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd81a2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd0102ef0 0x7f6cd01a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd81a2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0110ff0 con 0x7f6cd0100e20 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd81a2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0110e70 con 0x7f6cd0102570 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd81a2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0111170 con 0x7f6cd0102ef0 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd5f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cd0100e20 0x7f6cd01993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd5f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cd0100e20 0x7f6cd01993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49076/0 (socket says 172.21.15.89:49076) 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd5f3e640 1 -- 172.21.15.89:0/2198868713 learned_addr learned my addr 172.21.15.89:0/2198868713 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd673f640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd0102ef0 0x7f6cd01a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd573d640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cd0102570 0x7f6cd0199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd5f3e640 1 -- 172.21.15.89:0/2198868713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd0102ef0 msgr2=0x7f6cd01a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd0102ef0 0x7f6cd01a4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd5f3e640 1 -- 172.21.15.89:0/2198868713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cd0102570 msgr2=0x7f6cd0199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cd0102570 0x7f6cd0199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd5f3e640 1 -- 172.21.15.89:0/2198868713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd01a4980 con 0x7f6cd0100e20 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd673f640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd0102ef0 0x7f6cd01a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd573d640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cd0102570 0x7f6cd0199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cd0100e20 0x7f6cd01993c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0006010 tx=0x7f6cc0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cceffd640 1 -- 172.21.15.89:0/2198868713 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc003f070 con 0x7f6cd0100e20 2024-09-15T23:24:48.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2198868713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd01a4b50 con 0x7f6cd0100e20 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cceffd640 1 -- 172.21.15.89:0/2198868713 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cc0003e30 con 0x7f6cd0100e20 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.462+0000 7f6cceffd640 1 -- 172.21.15.89:0/2198868713 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0035030 con 0x7f6cd0100e20 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.463+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2198868713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd01a5030 con 0x7f6cd0100e20 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.465+0000 7f6cceffd640 1 -- 172.21.15.89:0/2198868713 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f6cc0042070 con 0x7f6cd0100e20 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.465+0000 7f6cda57b740 1 Processor -- start 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.465+0000 7f6cda57b740 1 -- start start 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.465+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8c9d0 0x556190a8cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.465+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a8d2f0 0x556190a97670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a99fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cda57b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5561909f3090 con 0x556190a97bb0 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cda57b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55619099aa70 con 0x556190a8d2f0 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cda57b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5561909ef050 con 0x556190a8c9d0 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cceffd640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cb80a32d0 0x7f6cb80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd573d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a99fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd573d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a99fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49082/0 (socket says 172.21.15.89:49082) 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd573d640 1 -- 172.21.15.89:0/3571663831 learned_addr learned my addr 172.21.15.89:0/3571663831 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd673f640 1 --2- 172.21.15.89:0/3571663831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8c9d0 0x556190a8cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cceffd640 1 -- 172.21.15.89:0/2198868713 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6cc00f0940 con 0x7f6cd0100e20 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/3571663831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a8d2f0 0x556190a97670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd673f640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cb80a32d0 0x7f6cb80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd573d640 1 -- 172.21.15.89:0/3571663831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8c9d0 msgr2=0x556190a8cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd573d640 1 --2- 172.21.15.89:0/3571663831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8c9d0 0x556190a8cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd573d640 1 -- 172.21.15.89:0/3571663831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a8d2f0 msgr2=0x556190a97670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd573d640 1 --2- 172.21.15.89:0/3571663831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a8d2f0 0x556190a97670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd573d640 1 -- 172.21.15.89:0/3571663831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561909a7130 con 0x556190a97bb0 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd573d640 1 --2- 172.21.15.89:0/3571663831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a99fa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4007f80 tx=0x7f6cc400c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590e35ff83bebc4d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cb7fff640 1 -- 172.21.15.89:0/3571663831 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4014070 con 0x556190a97bb0 2024-09-15T23:24:48.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cb7fff640 1 -- 172.21.15.89:0/3571663831 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cc400fd30 con 0x556190a97bb0 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cd673f640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cb80a32d0 0x7f6cb80a5790 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8005d50 tx=0x7f6cc8005ce0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.466+0000 7f6cb7fff640 1 -- 172.21.15.89:0/3571663831 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc400ede0 con 0x556190a97bb0 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.467+0000 7f6cda57b740 1 -- 172.21.15.89:0/3571663831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 msgr2=0x556190a99fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.467+0000 7f6cda57b740 1 --2- 172.21.15.89:0/3571663831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a99fa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4007f80 tx=0x7f6cc400c750 comp rx=0 tx=0).stop 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.467+0000 7f6cda57b740 1 -- 172.21.15.89:0/3571663831 shutdown_connections 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.467+0000 7f6cda57b740 1 --2- 172.21.15.89:0/3571663831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a99fa0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.467+0000 7f6cda57b740 1 --2- 172.21.15.89:0/3571663831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a8d2f0 0x556190a97670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.467+0000 7f6cda57b740 1 --2- 172.21.15.89:0/3571663831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8c9d0 0x556190a8cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.467+0000 7f6cda57b740 1 -- 172.21.15.89:0/3571663831 >> 172.21.15.89:0/3571663831 conn(0x556190a88f60 msgr2=0x556190a89340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.467+0000 7f6cda57b740 1 -- 172.21.15.89:0/3571663831 shutdown_connections 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.467+0000 7f6cda57b740 1 -- 172.21.15.89:0/3571663831 wait complete. 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cda57b740 1 -- start start 2024-09-15T23:24:48.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cda57b740 1 Processor -- start 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a8c9d0 0x556190a97360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8d2f0 0x556190a946b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a94bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cda57b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556190a9a780 con 0x556190a97bb0 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cda57b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5561909adf20 con 0x556190a8c9d0 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cda57b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556190a9a900 con 0x556190a8d2f0 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cda57b740 10 client.0 _ll_register_callbacks cb 0x556190977030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cd673f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a94bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cd673f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a94bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49098/0 (socket says 172.21.15.89:49098) 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cd673f640 1 -- 172.21.15.89:0/1659318573 learned_addr learned my addr 172.21.15.89:0/1659318573 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a8c9d0 0x556190a97360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cd573d640 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8d2f0 0x556190a946b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cd673f640 1 -- 172.21.15.89:0/1659318573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8d2f0 msgr2=0x556190a946b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cd673f640 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8d2f0 0x556190a946b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cd673f640 1 -- 172.21.15.89:0/1659318573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a8c9d0 msgr2=0x556190a97360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.468+0000 7f6cd673f640 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a8c9d0 0x556190a97360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cd673f640 1 -- 172.21.15.89:0/1659318573 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561907642b0 con 0x556190a97bb0 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cd673f640 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a94bf0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8010e50 tx=0x7f6cc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef28c35d7dc518fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1659318573 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc8004220 con 0x556190a97bb0 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1659318573 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6cc80043c0 con 0x556190a97bb0 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cda57b740 1 -- 172.21.15.89:0/1659318573 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cc400fd30 con 0x556190a97bb0 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1659318573 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6cc802e050 con 0x556190a97bb0 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cda57b740 10 client.15552 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cda57b740 1 -- 172.21.15.89:0/1659318573 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556190aa5ff0 con 0x556190a97bb0 2024-09-15T23:24:48.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1659318573 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cc400fd30 con 0x556190a97bb0 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1659318573 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6cc801a070 con 0x556190a97bb0 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.469+0000 7f6cda57b740 10 client.15552 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.470+0000 7f6cda57b740 10 client.15552 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.470+0000 7f6cda57b740 20 client.15552 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.470+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556190aa86c0 0x556190aa9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.470+0000 7f6cda57b740 4 client.15552 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.470+0000 7f6cda57b740 1 -- 172.21.15.89:0/1659318573 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556190a00600 con 0x556190aa86c0 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.470+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556190aa86c0 0x556190aa9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.470+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556190aa86c0 0x556190aa9b30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4bcdf5624e1b68ae server_cookie=110eefd5930be75d in_seq=0 out_seq=0 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.470+0000 7f6cb57fa640 10 client.15552 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1659318573 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556190a00600 con 0x556190aa86c0 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cb57fa640 10 client.15552 handle_command_reply: tid=0 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cda57b740 1 client.15552 shutdown 2024-09-15T23:24:48.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cda57b740 1 -- 172.21.15.89:0/1659318573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 msgr2=0x556190a94bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a94bf0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8010e50 tx=0x7f6cc80046c0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cda57b740 1 -- 172.21.15.89:0/1659318573 shutdown_connections 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556190aa86c0 0x556190aa9b30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a97bb0 0x556190a94bf0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8d2f0 0x556190a946b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1659318573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a8c9d0 0x556190a97360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cb57fa640 0 client.15552 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.471+0000 7f6cda57b740 1 -- 172.21.15.89:0/1659318573 >> 172.21.15.89:0/1659318573 conn(0x556190a88f60 msgr2=0x556190a8ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.472+0000 7f6cda57b740 1 -- 172.21.15.89:0/1659318573 shutdown_connections 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.472+0000 7f6cda57b740 1 -- 172.21.15.89:0/1659318573 wait complete. 2024-09-15T23:24:48.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.472+0000 7f6cda57b740 20 client.15552 trim_cache size 0 max 16384 2024-09-15T23:24:48.493 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:48 smithi174 ceph-mon[35586]: pgmap v373: 129 pgs: 129 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.7 KiB/s wr, 16 op/s 2024-09-15T23:24:48.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.489+0000 7f6cda57b740 1 Processor -- start 2024-09-15T23:24:48.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.489+0000 7f6cda57b740 1 -- start start 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.489+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8c9d0 0x556190aa9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8d2f0 0x556190a93c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a97bb0 0x556190ab2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cda57b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556190aa7e70 con 0x556190a8d2f0 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cda57b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556190aa8540 con 0x556190a97bb0 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cda57b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556190aa8c10 con 0x556190a8c9d0 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cd573d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8d2f0 0x556190a93c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cd573d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8d2f0 0x556190a93c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49110/0 (socket says 172.21.15.89:49110) 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cd573d640 1 -- 172.21.15.89:0/853395522 learned_addr learned my addr 172.21.15.89:0/853395522 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/853395522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8c9d0 0x556190aa9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cd573d640 1 -- 172.21.15.89:0/853395522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8c9d0 msgr2=0x556190aa9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cd573d640 1 --2- 172.21.15.89:0/853395522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8c9d0 0x556190aa9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cd573d640 1 -- 172.21.15.89:0/853395522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a97bb0 msgr2=0x556190ab2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cd573d640 1 --2- 172.21.15.89:0/853395522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a97bb0 0x556190ab2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cd573d640 1 -- 172.21.15.89:0/853395522 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556190a88d90 con 0x556190a8d2f0 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cd573d640 1 --2- 172.21.15.89:0/853395522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8d2f0 0x556190a93c30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4007ea0 tx=0x7f6cc4007b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea058db07d37a06 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cb77fe640 1 -- 172.21.15.89:0/853395522 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4014040 con 0x556190a8d2f0 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cb77fe640 1 -- 172.21.15.89:0/853395522 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cc400fd30 con 0x556190a8d2f0 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.490+0000 7f6cb77fe640 1 -- 172.21.15.89:0/853395522 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4019e00 con 0x556190a8d2f0 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 -- 172.21.15.89:0/853395522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8d2f0 msgr2=0x556190a93c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 --2- 172.21.15.89:0/853395522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8d2f0 0x556190a93c30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4007ea0 tx=0x7f6cc4007b40 comp rx=0 tx=0).stop 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 -- 172.21.15.89:0/853395522 shutdown_connections 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 --2- 172.21.15.89:0/853395522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a97bb0 0x556190ab2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 --2- 172.21.15.89:0/853395522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8d2f0 0x556190a93c30 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 --2- 172.21.15.89:0/853395522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8c9d0 0x556190aa9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 -- 172.21.15.89:0/853395522 >> 172.21.15.89:0/853395522 conn(0x556190a88f60 msgr2=0x556190a94670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 -- 172.21.15.89:0/853395522 shutdown_connections 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 -- 172.21.15.89:0/853395522 wait complete. 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 -- start start 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.491+0000 7f6cda57b740 1 Processor -- start 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8c9d0 0x556190a92380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8d2f0 0x556190ab1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cda57b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a97bb0 0x556190ab2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cda57b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556190aa99b0 con 0x556190a8c9d0 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cda57b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556190aa92e0 con 0x556190a97bb0 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cda57b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556190aaa080 con 0x556190a8d2f0 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd5f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8c9d0 0x556190a92380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cda57b740 10 client.0 _ll_register_callbacks cb 0x7f6cc800ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:48.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd5f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8c9d0 0x556190a92380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49126/0 (socket says 172.21.15.89:49126) 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd5f3e640 1 -- 172.21.15.89:0/1610359335 learned_addr learned my addr 172.21.15.89:0/1610359335 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd573d640 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8d2f0 0x556190ab1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd673f640 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a97bb0 0x556190ab2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd5f3e640 1 -- 172.21.15.89:0/1610359335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8d2f0 msgr2=0x556190ab1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8d2f0 0x556190ab1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd5f3e640 1 -- 172.21.15.89:0/1610359335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a97bb0 msgr2=0x556190ab2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a97bb0 0x556190ab2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd5f3e640 1 -- 172.21.15.89:0/1610359335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556190ab2840 con 0x556190a8c9d0 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cd5f3e640 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8c9d0 0x556190a92380 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00b93e0 tx=0x7f6cc00a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dccffaffc19300fd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.492+0000 7f6cb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1610359335 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0049040 con 0x556190a8c9d0 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1610359335 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6cc003b460 con 0x556190a8c9d0 2024-09-15T23:24:48.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cda57b740 1 -- 172.21.15.89:0/1610359335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cc400fd30 con 0x556190a8c9d0 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1610359335 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6cc00a70b0 con 0x556190a8c9d0 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cda57b740 10 client.15558 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cda57b740 1 -- 172.21.15.89:0/1610359335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556190b16f30 con 0x556190a8c9d0 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1610359335 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cc400fd30 con 0x556190a8c9d0 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1610359335 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6cc00c51f0 con 0x556190a8c9d0 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cda57b740 10 client.15558 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cda57b740 10 client.15558 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cda57b740 20 client.15558 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556190b1baa0 0x556190b1a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cda57b740 4 client.15558 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cda57b740 1 -- 172.21.15.89:0/1610359335 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "status"}) -- 0x5561909fab10 con 0x556190b1baa0 2024-09-15T23:24:48.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.493+0000 7f6cd573d640 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556190b1baa0 0x556190b1a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cb57fa640 10 client.15558 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cd573d640 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556190b1baa0 0x556190b1a7b0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a05c0c11f4cb233c server_cookie=7daac2a6b4f9826c in_seq=0 out_seq=0 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cb57fa640 1 -- 172.21.15.89:0/1610359335 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5561909fab10 con 0x556190b1baa0 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cb57fa640 10 client.15558 handle_command_reply: tid=0 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 client.15558 shutdown 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 -- 172.21.15.89:0/1610359335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8c9d0 msgr2=0x556190a92380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8c9d0 0x556190a92380 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00b93e0 tx=0x7f6cc00a7040 comp rx=0 tx=0).stop 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 -- 172.21.15.89:0/1610359335 shutdown_connections 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556190b1baa0 0x556190b1a7b0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556190a97bb0 0x556190ab2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556190a8d2f0 0x556190ab1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cb57fa640 0 client.15558 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 --2- 172.21.15.89:0/1610359335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556190a8c9d0 0x556190a92380 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 -- 172.21.15.89:0/1610359335 >> 172.21.15.89:0/1610359335 conn(0x556190a88f60 msgr2=0x556190a8f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 -- 172.21.15.89:0/1610359335 shutdown_connections 2024-09-15T23:24:48.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 1 -- 172.21.15.89:0/1610359335 wait complete. 2024-09-15T23:24:48.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.494+0000 7f6cda57b740 20 client.15558 trim_cache size 0 max 16384 2024-09-15T23:24:48.504 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:48.504 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:24:48.504 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 1, 2024-09-15T23:24:48.504 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24436, 2024-09-15T23:24:48.504 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:24:48.504 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:24:48.504 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:24:48.504 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 135.40737680500001, 2024-09-15T23:24:48.505 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 29, 2024-09-15T23:24:48.505 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 90, 2024-09-15T23:24:48.505 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 90, 2024-09-15T23:24:48.505 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 194.51337797599999 2024-09-15T23:24:48.505 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:48.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2198868713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cb80a32d0 msgr2=0x7f6cb80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cb80a32d0 0x7f6cb80a5790 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8005d50 tx=0x7f6cc8005ce0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2198868713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cd0100e20 msgr2=0x7f6cd01993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cd0100e20 0x7f6cd01993c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0006010 tx=0x7f6cc0003c10 comp rx=0 tx=0).stop 2024-09-15T23:24:48.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2198868713 shutdown_connections 2024-09-15T23:24:48.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cb80a32d0 0x7f6cb80a5790 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd0102ef0 0x7f6cd01a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6cd0102570 0x7f6cd0199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 --2- 172.21.15.89:0/2198868713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6cd0100e20 0x7f6cd01993c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2198868713 >> 172.21.15.89:0/2198868713 conn(0x7f6cd00fc7d0 msgr2=0x7f6cd00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2198868713 shutdown_connections 2024-09-15T23:24:48.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:48.495+0000 7f6cd81a2640 1 -- 172.21.15.89:0/2198868713 wait complete. 2024-09-15T23:24:49.268 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T23:24:49.513 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.030+0000 7f4441be4640 1 -- 172.21.15.89:0/702786705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443c102d30 msgr2=0x7f443c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.030+0000 7f4441be4640 1 --2- 172.21.15.89:0/702786705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443c102d30 0x7f443c103110 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f442c009930 tx=0x7f442c02a440 comp rx=0 tx=0).stop 2024-09-15T23:24:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.030+0000 7f4441be4640 1 -- 172.21.15.89:0/702786705 shutdown_connections 2024-09-15T23:24:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.030+0000 7f4441be4640 1 --2- 172.21.15.89:0/702786705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f443c1082a0 0x7f443c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.030+0000 7f4441be4640 1 --2- 172.21.15.89:0/702786705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f443c1036e0 0x7f443c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.030+0000 7f4441be4640 1 --2- 172.21.15.89:0/702786705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443c102d30 0x7f443c103110 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.030+0000 7f4441be4640 1 -- 172.21.15.89:0/702786705 >> 172.21.15.89:0/702786705 conn(0x7f443c0fc7d0 msgr2=0x7f443c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.030+0000 7f4441be4640 1 -- 172.21.15.89:0/702786705 shutdown_connections 2024-09-15T23:24:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.030+0000 7f4441be4640 1 -- 172.21.15.89:0/702786705 wait complete. 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f4441be4640 1 Processor -- start 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f4441be4640 1 -- start start 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f4441be4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f443c102d30 0x7f443c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f4441be4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f443c1036e0 0x7f443c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f4441be4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443c1082a0 0x7f443c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f4441be4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f443c112100 con 0x7f443c1082a0 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f4441be4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f443c111f80 con 0x7f443c102d30 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f4441be4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f443c112280 con 0x7f443c1036e0 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f443bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443c1082a0 0x7f443c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f443bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443c1082a0 0x7f443c19cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49138/0 (socket says 172.21.15.89:49138) 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.031+0000 7f443bfff640 1 -- 172.21.15.89:0/3957901658 learned_addr learned my addr 172.21.15.89:0/3957901658 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f443b7fe640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f443c102d30 0x7f443c198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f443affd640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f443c1036e0 0x7f443c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f443bfff640 1 -- 172.21.15.89:0/3957901658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f443c1036e0 msgr2=0x7f443c198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f443bfff640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f443c1036e0 0x7f443c198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f443bfff640 1 -- 172.21.15.89:0/3957901658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f443c102d30 msgr2=0x7f443c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f443bfff640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f443c102d30 0x7f443c198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f443bfff640 1 -- 172.21.15.89:0/3957901658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443c19d600 con 0x7f443c1082a0 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f443bfff640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443c1082a0 0x7f443c19cf20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f443400cad0 tx=0x7f443400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f4438ff9640 1 -- 172.21.15.89:0/3957901658 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443400ae20 con 0x7f443c1082a0 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f4438ff9640 1 -- 172.21.15.89:0/3957901658 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4434005020 con 0x7f443c1082a0 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f4441be4640 1 -- 172.21.15.89:0/3957901658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443c19d8f0 con 0x7f443c1082a0 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f4438ff9640 1 -- 172.21.15.89:0/3957901658 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443400f6f0 con 0x7f443c1082a0 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.032+0000 7f4441be4640 1 -- 172.21.15.89:0/3957901658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f443c1a51d0 con 0x7f443c1082a0 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.034+0000 7f4438ff9640 1 -- 172.21.15.89:0/3957901658 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f44340051d0 con 0x7f443c1082a0 2024-09-15T23:24:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.034+0000 7f4443fbd740 1 Processor -- start 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f4443fbd740 1 -- start start 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 0x556a031b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 0x556a031c0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f4443fbd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556a0311c090 con 0x556a031c0bb0 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f4443fbd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556a030c3a70 con 0x556a031b59d0 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f4443fbd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556a03118050 con 0x556a031b62f0 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f4438ff9640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f44240a3350 0x7f44240a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031c2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49150/0 (socket says 172.21.15.89:49150) 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443affd640 1 -- 172.21.15.89:0/4232646516 learned_addr learned my addr 172.21.15.89:0/4232646516 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443bfff640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f44240a3350 0x7f44240a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443b7fe640 1 --2- 172.21.15.89:0/4232646516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 0x556a031c0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443affd640 1 --2- 172.21.15.89:0/4232646516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 0x556a031b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f4438ff9640 1 -- 172.21.15.89:0/3957901658 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f44340d0160 con 0x7f443c1082a0 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443affd640 1 -- 172.21.15.89:0/4232646516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 msgr2=0x556a031c0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443affd640 1 --2- 172.21.15.89:0/4232646516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 0x556a031c0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443affd640 1 -- 172.21.15.89:0/4232646516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 msgr2=0x556a031b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443affd640 1 --2- 172.21.15.89:0/4232646516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 0x556a031b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443affd640 1 -- 172.21.15.89:0/4232646516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a030d0130 con 0x556a031c0bb0 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.035+0000 7f443affd640 1 --2- 172.21.15.89:0/4232646516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031c2fa0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f443000ded0 tx=0x7f443000d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622efaf36c009032 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f443bfff640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f44240a3350 0x7f44240a5810 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f443400a6e0 tx=0x7f443406e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f4421ffb640 1 -- 172.21.15.89:0/4232646516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443001c070 con 0x556a031c0bb0 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f4421ffb640 1 -- 172.21.15.89:0/4232646516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4430010040 con 0x556a031c0bb0 2024-09-15T23:24:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f4421ffb640 1 -- 172.21.15.89:0/4232646516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4430016720 con 0x556a031c0bb0 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f4443fbd740 1 -- 172.21.15.89:0/4232646516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 msgr2=0x556a031c2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f4443fbd740 1 --2- 172.21.15.89:0/4232646516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031c2fa0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f443000ded0 tx=0x7f443000d3c0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f4443fbd740 1 -- 172.21.15.89:0/4232646516 shutdown_connections 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f4443fbd740 1 --2- 172.21.15.89:0/4232646516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031c2fa0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f4443fbd740 1 --2- 172.21.15.89:0/4232646516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 0x556a031c0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f4443fbd740 1 --2- 172.21.15.89:0/4232646516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 0x556a031b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.036+0000 7f4443fbd740 1 -- 172.21.15.89:0/4232646516 >> 172.21.15.89:0/4232646516 conn(0x556a031b1f60 msgr2=0x556a031b2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 1 -- 172.21.15.89:0/4232646516 shutdown_connections 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 1 -- 172.21.15.89:0/4232646516 wait complete. 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 1 -- start start 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 1 Processor -- start 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 0x556a031c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b62f0 0x556a031bd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031c0bb0 0x556a031bdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556a031c3780 con 0x556a031b62f0 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556a030d6f20 con 0x556a031b59d0 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556a031c3900 con 0x556a031c0bb0 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.037+0000 7f4443fbd740 10 client.0 _ll_register_callbacks cb 0x556a030a0030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b62f0 0x556a031bd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b62f0 0x556a031bd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49152/0 (socket says 172.21.15.89:49152) 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443b7fe640 1 -- 172.21.15.89:0/2449192131 learned_addr learned my addr 172.21.15.89:0/2449192131 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443affd640 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031c0bb0 0x556a031bdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443affd640 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 0x556a031c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443b7fe640 1 -- 172.21.15.89:0/2449192131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031c0bb0 msgr2=0x556a031bdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443b7fe640 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031c0bb0 0x556a031bdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443b7fe640 1 -- 172.21.15.89:0/2449192131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 msgr2=0x556a031c04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443b7fe640 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 0x556a031c04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443b7fe640 1 -- 172.21.15.89:0/2449192131 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a02e8d2b0 con 0x556a031b62f0 2024-09-15T23:24:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443affd640 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 0x556a031c04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f443b7fe640 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b62f0 0x556a031bd840 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f442c006010 tx=0x7f442c031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecf3ec6d50a588e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f441affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f441affd640 1 -- 172.21.15.89:0/2449192131 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c035040 con 0x556a031b62f0 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f441affd640 1 -- 172.21.15.89:0/2449192131 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f442c004380 con 0x556a031b62f0 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f4443fbd740 1 -- 172.21.15.89:0/2449192131 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4430010040 con 0x556a031b62f0 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f441affd640 1 -- 172.21.15.89:0/2449192131 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f442c0035f0 con 0x556a031b62f0 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f4443fbd740 10 client.15570 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.038+0000 7f4443fbd740 1 -- 172.21.15.89:0/2449192131 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556a031cf130 con 0x556a031b62f0 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f441affd640 1 -- 172.21.15.89:0/2449192131 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4430010040 con 0x556a031b62f0 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f441affd640 1 -- 172.21.15.89:0/2449192131 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f442c03c920 con 0x556a031b62f0 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f4443fbd740 10 client.15570 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f4443fbd740 10 client.15570 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f4443fbd740 20 client.15570 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f4443fbd740 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a031d1790 0x556a031d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f4443fbd740 4 client.15570 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f4443fbd740 1 -- 172.21.15.89:0/2449192131 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a03129600 con 0x556a031d1790 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f443affd640 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a031d1790 0x556a031d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f441affd640 10 client.15570 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.039+0000 7f443affd640 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a031d1790 0x556a031d2c00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1e4c11edaf7f3c server_cookie=339a16a4c421b2ed in_seq=0 out_seq=0 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f441affd640 1 -- 172.21.15.89:0/2449192131 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556a03129600 con 0x556a031d1790 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f441affd640 10 client.15570 handle_command_reply: tid=0 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 client.15570 shutdown 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 -- 172.21.15.89:0/2449192131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b62f0 msgr2=0x556a031bd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b62f0 0x556a031bd840 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f442c006010 tx=0x7f442c031a60 comp rx=0 tx=0).stop 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 -- 172.21.15.89:0/2449192131 shutdown_connections 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a031d1790 0x556a031d2c00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031c0bb0 0x556a031bdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b62f0 0x556a031bd840 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 --2- 172.21.15.89:0/2449192131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b59d0 0x556a031c04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 -- 172.21.15.89:0/2449192131 >> 172.21.15.89:0/2449192131 conn(0x556a031b1f60 msgr2=0x556a031b7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:50.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f441affd640 0 client.15570 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:50.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 -- 172.21.15.89:0/2449192131 shutdown_connections 2024-09-15T23:24:50.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.040+0000 7f4443fbd740 1 -- 172.21.15.89:0/2449192131 wait complete. 2024-09-15T23:24:50.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.041+0000 7f4443fbd740 20 client.15570 trim_cache size 0 max 16384 2024-09-15T23:24:50.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.058+0000 7f4443fbd740 1 Processor -- start 2024-09-15T23:24:50.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.058+0000 7f4443fbd740 1 -- start start 2024-09-15T23:24:50.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.058+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b59d0 0x556a031d1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.058+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 0x556a031daee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.058+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031c0bb0 0x556a031db420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.058+0000 7f4443fbd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556a031d0f40 con 0x556a031b59d0 2024-09-15T23:24:50.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.058+0000 7f4443fbd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556a031d1610 con 0x556a031c0bb0 2024-09-15T23:24:50.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.058+0000 7f4443fbd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556a031d1ce0 con 0x556a031b62f0 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b59d0 0x556a031d1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b59d0 0x556a031d1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49158/0 (socket says 172.21.15.89:49158) 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443affd640 1 -- 172.21.15.89:0/2088551922 learned_addr learned my addr 172.21.15.89:0/2088551922 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443b7fe640 1 --2- 172.21.15.89:0/2088551922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 0x556a031daee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443affd640 1 -- 172.21.15.89:0/2088551922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 msgr2=0x556a031daee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443affd640 1 --2- 172.21.15.89:0/2088551922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 0x556a031daee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443affd640 1 -- 172.21.15.89:0/2088551922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031c0bb0 msgr2=0x556a031db420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443affd640 1 --2- 172.21.15.89:0/2088551922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031c0bb0 0x556a031db420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443affd640 1 -- 172.21.15.89:0/2088551922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a031be360 con 0x556a031b59d0 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443b7fe640 1 --2- 172.21.15.89:0/2088551922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 0x556a031daee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f443affd640 1 --2- 172.21.15.89:0/2088551922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b59d0 0x556a031d1e60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4430006aa0 tx=0x7f4430017ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5195ace72cc3213 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f44217fa640 1 -- 172.21.15.89:0/2088551922 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443001c040 con 0x556a031b59d0 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f44217fa640 1 -- 172.21.15.89:0/2088551922 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4430034070 con 0x556a031b59d0 2024-09-15T23:24:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.059+0000 7f44217fa640 1 -- 172.21.15.89:0/2088551922 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44300268d0 con 0x556a031b59d0 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 -- 172.21.15.89:0/2088551922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b59d0 msgr2=0x556a031d1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 --2- 172.21.15.89:0/2088551922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b59d0 0x556a031d1e60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4430006aa0 tx=0x7f4430017ef0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 -- 172.21.15.89:0/2088551922 shutdown_connections 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 --2- 172.21.15.89:0/2088551922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031c0bb0 0x556a031db420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 --2- 172.21.15.89:0/2088551922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b62f0 0x556a031daee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 --2- 172.21.15.89:0/2088551922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031b59d0 0x556a031d1e60 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 -- 172.21.15.89:0/2088551922 >> 172.21.15.89:0/2088551922 conn(0x556a031b1f60 msgr2=0x556a031bc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 -- 172.21.15.89:0/2088551922 shutdown_connections 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 -- 172.21.15.89:0/2088551922 wait complete. 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 -- start start 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.060+0000 7f4443fbd740 1 Processor -- start 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b59d0 0x556a031d5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b62f0 0x556a031d63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f4443fbd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031d6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f4443fbd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556a031d2a80 con 0x556a031c0bb0 2024-09-15T23:24:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f4443fbd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556a031d23b0 con 0x556a031b62f0 2024-09-15T23:24:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f4443fbd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556a031d3150 con 0x556a031b59d0 2024-09-15T23:24:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f4443fbd740 10 client.0 _ll_register_callbacks cb 0x7f442c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031d6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031d6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49170/0 (socket says 172.21.15.89:49170) 2024-09-15T23:24:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443affd640 1 -- 172.21.15.89:0/3505138629 learned_addr learned my addr 172.21.15.89:0/3505138629 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443b7fe640 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b62f0 0x556a031d63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443affd640 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b59d0 0x556a031d5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443affd640 1 -- 172.21.15.89:0/3505138629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b59d0 msgr2=0x556a031d5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443affd640 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b59d0 0x556a031d5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443affd640 1 -- 172.21.15.89:0/3505138629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b62f0 msgr2=0x556a031d63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443affd640 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b62f0 0x556a031d63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443affd640 1 -- 172.21.15.89:0/3505138629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a0323a180 con 0x556a031c0bb0 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443b7fe640 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b62f0 0x556a031d63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f443affd640 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031d6910 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4430011010 tx=0x7f4430026590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591b097a17ab8d43 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f441affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f441affd640 1 -- 172.21.15.89:0/3505138629 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443003d070 con 0x556a031c0bb0 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f4443fbd740 1 -- 172.21.15.89:0/3505138629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4430034070 con 0x556a031c0bb0 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.061+0000 7f441affd640 1 -- 172.21.15.89:0/3505138629 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4430005bc0 con 0x556a031c0bb0 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f441affd640 1 -- 172.21.15.89:0/3505138629 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4430049050 con 0x556a031c0bb0 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f4443fbd740 10 client.15576 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f4443fbd740 1 -- 172.21.15.89:0/3505138629 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556a0323b350 con 0x556a031c0bb0 2024-09-15T23:24:50.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f441affd640 1 -- 172.21.15.89:0/3505138629 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4430034070 con 0x556a031c0bb0 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f441affd640 1 -- 172.21.15.89:0/3505138629 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4430022040 con 0x556a031c0bb0 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f4443fbd740 10 client.15576 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f4443fbd740 10 client.15576 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f4443fbd740 20 client.15576 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f4443fbd740 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a0323f7a0 0x556a0323e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f4443fbd740 4 client.15576 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f4443fbd740 1 -- 172.21.15.89:0/3505138629 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "status"}) -- 0x556a03123b10 con 0x556a0323f7a0 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.062+0000 7f443b7fe640 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a0323f7a0 0x556a0323e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f441affd640 10 client.15576 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f443b7fe640 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a0323f7a0 0x556a0323e4b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a8623df2f23d518 server_cookie=5c7c57aadccf3de5 in_seq=0 out_seq=0 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f441affd640 1 -- 172.21.15.89:0/3505138629 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x556a03123b10 con 0x556a0323f7a0 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f441affd640 10 client.15576 handle_command_reply: tid=0 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 client.15576 shutdown 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 -- 172.21.15.89:0/3505138629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 msgr2=0x556a031d6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031d6910 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4430011010 tx=0x7f4430026590 comp rx=0 tx=0).stop 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 -- 172.21.15.89:0/3505138629 shutdown_connections 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a0323f7a0 0x556a0323e4b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a031c0bb0 0x556a031d6910 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a031b62f0 0x556a031d63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f441affd640 0 client.15576 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 --2- 172.21.15.89:0/3505138629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a031b59d0 0x556a031d5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 -- 172.21.15.89:0/3505138629 >> 172.21.15.89:0/3505138629 conn(0x556a031b1f60 msgr2=0x556a031bd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 -- 172.21.15.89:0/3505138629 shutdown_connections 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 1 -- 172.21.15.89:0/3505138629 wait complete. 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.063+0000 7f4443fbd740 20 client.15576 trim_cache size 0 max 16384 2024-09-15T23:24:50.072 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_fsid": "1b4e699c-73b8-11ef-bceb-c7b262605968", 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "whoami": 0, 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "id": 24452, 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "want_state": "up:active", 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "state": "up:active", 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "fs_name": "cephfs", 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "rank_uptime": 176.543765968, 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "mdsmap_epoch": 29, 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch": 90, 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_epoch_barrier": 90, 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout: "uptime": 187.070687905 2024-09-15T23:24:50.073 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:24:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 -- 172.21.15.89:0/3957901658 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f44240a3350 msgr2=0x7f44240a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f44240a3350 0x7f44240a5810 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f443400a6e0 tx=0x7f443406e000 comp rx=0 tx=0).stop 2024-09-15T23:24:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 -- 172.21.15.89:0/3957901658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443c1082a0 msgr2=0x7f443c19cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443c1082a0 0x7f443c19cf20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f443400cad0 tx=0x7f443400cf90 comp rx=0 tx=0).stop 2024-09-15T23:24:50.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 -- 172.21.15.89:0/3957901658 shutdown_connections 2024-09-15T23:24:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f44240a3350 0x7f44240a5810 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f443c1082a0 0x7f443c19cf20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f443c1036e0 0x7f443c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 --2- 172.21.15.89:0/3957901658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f443c102d30 0x7f443c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 -- 172.21.15.89:0/3957901658 >> 172.21.15.89:0/3957901658 conn(0x7f443c0fc7d0 msgr2=0x7f443c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 -- 172.21.15.89:0/3957901658 shutdown_connections 2024-09-15T23:24:50.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:24:50.064+0000 7f4441be4640 1 -- 172.21.15.89:0/3957901658 wait complete. 2024-09-15T23:24:50.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:50 smithi089 ceph-mon[30933]: pgmap v374: 129 pgs: 129 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-09-15T23:24:50.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:50 smithi120 ceph-mon[35250]: pgmap v374: 129 pgs: 129 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-09-15T23:24:50.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:50 smithi174 ceph-mon[35586]: pgmap v374: 129 pgs: 129 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-09-15T23:24:50.803 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-09-15T23:24:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:52 smithi120 ceph-mon[35250]: pgmap v375: 129 pgs: 129 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-09-15T23:24:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:52 smithi174 ceph-mon[35586]: pgmap v375: 129 pgs: 129 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-09-15T23:24:52.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:52 smithi089 ceph-mon[30933]: pgmap v375: 129 pgs: 129 active+clean; 664 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.7 KiB/s wr, 14 op/s 2024-09-15T23:24:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.241+0000 7fb06f5d6640 1 -- 172.21.15.89:0/2403665531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb068102600 msgr2=0x7fb068108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.241+0000 7fb06f5d6640 1 --2- 172.21.15.89:0/2403665531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb068102600 0x7fb068108e90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb060009ca0 tx=0x7fb06002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.241+0000 7fb06f5d6640 1 -- 172.21.15.89:0/2403665531 shutdown_connections 2024-09-15T23:24:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.241+0000 7fb06f5d6640 1 --2- 172.21.15.89:0/2403665531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb068102600 0x7fb068108e90 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.244 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.241+0000 7fb06f5d6640 1 --2- 172.21.15.89:0/2403665531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb068101c40 0x7fb0681020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.241+0000 7fb06f5d6640 1 --2- 172.21.15.89:0/2403665531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068100a50 0x7fb068100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.241+0000 7fb06f5d6640 1 -- 172.21.15.89:0/2403665531 >> 172.21.15.89:0/2403665531 conn(0x7fb0680fc1b0 msgr2=0x7fb0680fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.241+0000 7fb06f5d6640 1 -- 172.21.15.89:0/2403665531 shutdown_connections 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.241+0000 7fb06f5d6640 1 -- 172.21.15.89:0/2403665531 wait complete. 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06f5d6640 1 Processor -- start 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06f5d6640 1 -- start start 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06f5d6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb068100a50 0x7fb0681049e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06f5d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068101c40 0x7fb068103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06f5d6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb068102600 0x7fb068103560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06f5d6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb06810c6d0 con 0x7fb068100a50 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06f5d6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb06810c550 con 0x7fb068101c40 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06f5d6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb06810c850 con 0x7fb068102600 2024-09-15T23:24:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06d372640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb068100a50 0x7fb0681049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06d372640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb068100a50 0x7fb0681049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49178/0 (socket says 172.21.15.89:49178) 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06d372640 1 -- 172.21.15.89:0/1924989808 learned_addr learned my addr 172.21.15.89:0/1924989808 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06db73640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb068102600 0x7fb068103560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06d372640 1 -- 172.21.15.89:0/1924989808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb068102600 msgr2=0x7fb068103560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06d372640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb068102600 0x7fb068103560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06d372640 1 -- 172.21.15.89:0/1924989808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068101c40 msgr2=0x7fb068103020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06cb71640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068101c40 0x7fb068103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06d372640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068101c40 0x7fb068103020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06d372640 1 -- 172.21.15.89:0/1924989808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb068103df0 con 0x7fb068100a50 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06db73640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb068102600 0x7fb068103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb06d372640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb068100a50 0x7fb0681049e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb05800d770 tx=0x7fb05800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.242+0000 7fb0667fc640 1 -- 172.21.15.89:0/1924989808 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb058017070 con 0x7fb068100a50 2024-09-15T23:24:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.243+0000 7fb0667fc640 1 -- 172.21.15.89:0/1924989808 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb058009c30 con 0x7fb068100a50 2024-09-15T23:24:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.243+0000 7fb06f5d6640 1 -- 172.21.15.89:0/1924989808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0681a1e40 con 0x7fb068100a50 2024-09-15T23:24:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.243+0000 7fb0667fc640 1 -- 172.21.15.89:0/1924989808 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb058003dd0 con 0x7fb068100a50 2024-09-15T23:24:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.243+0000 7fb06f5d6640 1 -- 172.21.15.89:0/1924989808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0681a2250 con 0x7fb068100a50 2024-09-15T23:24:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.245+0000 7fb0667fc640 1 -- 172.21.15.89:0/1924989808 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fb058003fb0 con 0x7fb068100a50 2024-09-15T23:24:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.245+0000 7fb072450740 1 Processor -- start 2024-09-15T23:24:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.245+0000 7fb072450740 1 -- start start 2024-09-15T23:24:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.245+0000 7fb072450740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119c9a0 0x55f14119cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.245+0000 7fb072450740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119d2e0 0x55f1411a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.245+0000 7fb072450740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f1411a2370 0x55f1411a4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.245+0000 7fb072450740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f1411002f0 con 0x55f14119c9a0 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.245+0000 7fb072450740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f1410a8270 con 0x55f14119d2e0 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.245+0000 7fb072450740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f1410fc2b0 con 0x55f1411a2370 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06db73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119c9a0 0x55f14119cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06db73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119c9a0 0x55f14119cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49194/0 (socket says 172.21.15.89:49194) 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06db73640 1 -- 172.21.15.89:0/1225232634 learned_addr learned my addr 172.21.15.89:0/1225232634 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb0667fc640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0500a3390 0x7fb0500a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06cb71640 1 --2- 172.21.15.89:0/1225232634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f1411a2370 0x55f1411a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06d372640 1 --2- 172.21.15.89:0/1225232634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119d2e0 0x55f1411a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06db73640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0500a3390 0x7fb0500a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb0667fc640 1 -- 172.21.15.89:0/1924989808 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb0580cf830 con 0x7fb068100a50 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06db73640 1 -- 172.21.15.89:0/1225232634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f1411a2370 msgr2=0x55f1411a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:54.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06db73640 1 --2- 172.21.15.89:0/1225232634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f1411a2370 0x55f1411a4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06db73640 1 -- 172.21.15.89:0/1225232634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119d2e0 msgr2=0x55f1411a1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06db73640 1 --2- 172.21.15.89:0/1225232634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119d2e0 0x55f1411a1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06db73640 1 -- 172.21.15.89:0/1225232634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1410a7340 con 0x55f14119c9a0 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.246+0000 7fb06db73640 1 --2- 172.21.15.89:0/1225232634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119c9a0 0x55f14119cda0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb060009430 tx=0x7fb060031f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c5007ee6342c2d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb04f7fe640 1 -- 172.21.15.89:0/1225232634 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb060004b40 con 0x55f14119c9a0 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb04f7fe640 1 -- 172.21.15.89:0/1225232634 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb06004d070 con 0x55f14119c9a0 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb04f7fe640 1 -- 172.21.15.89:0/1225232634 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06002f610 con 0x55f14119c9a0 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb06db73640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0500a3390 0x7fb0500a5850 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb06004c000 tx=0x7fb06004a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb072450740 1 -- 172.21.15.89:0/1225232634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119c9a0 msgr2=0x55f14119cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb072450740 1 --2- 172.21.15.89:0/1225232634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119c9a0 0x55f14119cda0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb060009430 tx=0x7fb060031f40 comp rx=0 tx=0).stop 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb072450740 1 -- 172.21.15.89:0/1225232634 shutdown_connections 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb072450740 1 --2- 172.21.15.89:0/1225232634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f1411a2370 0x55f1411a4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb072450740 1 --2- 172.21.15.89:0/1225232634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119d2e0 0x55f1411a1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb072450740 1 --2- 172.21.15.89:0/1225232634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119c9a0 0x55f14119cda0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb072450740 1 -- 172.21.15.89:0/1225232634 >> 172.21.15.89:0/1225232634 conn(0x55f141198010 msgr2=0x55f141198410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb072450740 1 -- 172.21.15.89:0/1225232634 shutdown_connections 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.247+0000 7fb072450740 1 -- 172.21.15.89:0/1225232634 wait complete. 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.248+0000 7fb072450740 1 -- start start 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.248+0000 7fb072450740 1 Processor -- start 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.248+0000 7fb072450740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119c9a0 0x55f1411a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.248+0000 7fb072450740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411a7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.248+0000 7fb072450740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f1411a2370 0x55f1411ae130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.248+0000 7fb072450740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f1411a5340 con 0x55f14119d2e0 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.248+0000 7fb072450740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f141089290 con 0x55f14119c9a0 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.248+0000 7fb072450740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f1411a54c0 con 0x55f1411a2370 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.248+0000 7fb072450740 10 client.0 _ll_register_callbacks cb 0x55f1410ba140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06cb71640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06d372640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119c9a0 0x55f1411a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06d372640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119c9a0 0x55f1411a6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:35918/0 (socket says 172.21.15.89:35918) 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06cb71640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411a7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49202/0 (socket says 172.21.15.89:49202) 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06d372640 1 -- 172.21.15.89:0/1334416278 learned_addr learned my addr 172.21.15.89:0/1334416278 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06db73640 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f1411a2370 0x55f1411ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06d372640 1 -- 172.21.15.89:0/1334416278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f1411a2370 msgr2=0x55f1411ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06d372640 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f1411a2370 0x55f1411ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06d372640 1 -- 172.21.15.89:0/1334416278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 msgr2=0x55f1411a7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06d372640 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411a7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06d372640 1 -- 172.21.15.89:0/1334416278 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f140e13210 con 0x55f14119c9a0 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06cb71640 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb06d372640 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119c9a0 0x55f1411a6c20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb05809b890 tx=0x7fb05806b570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0c1c01c1d528e2e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb04cff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb04cff9640 1 -- 172.21.15.89:0/1334416278 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb058020040 con 0x55f14119c9a0 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb072450740 1 -- 172.21.15.89:0/1334416278 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb06004d070 con 0x55f14119c9a0 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb04cff9640 1 -- 172.21.15.89:0/1334416278 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb0580042a0 con 0x55f14119c9a0 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb04cff9640 1 -- 172.21.15.89:0/1334416278 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb05806a050 con 0x55f14119c9a0 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb072450740 10 client.25004 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.249+0000 7fb072450740 1 -- 172.21.15.89:0/1334416278 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f140edf620 con 0x55f14119c9a0 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.250+0000 7fb04cff9640 1 -- 172.21.15.89:0/1334416278 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb06004d070 con 0x55f14119c9a0 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.250+0000 7fb04cff9640 1 -- 172.21.15.89:0/1334416278 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb058097f70 con 0x55f14119c9a0 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.250+0000 7fb072450740 10 client.25004 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.250+0000 7fb072450740 10 client.25004 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.250+0000 7fb072450740 20 client.25004 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.250+0000 7fb072450740 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f1411b2820 0x55f1411b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.250+0000 7fb072450740 4 client.25004 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.250+0000 7fb072450740 1 -- 172.21.15.89:0/1334416278 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f14110da20 con 0x55f1411b2820 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.250+0000 7fb06cb71640 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f1411b2820 0x55f1411b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb04cff9640 10 client.25004 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb06cb71640 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f1411b2820 0x55f1411b4c30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c78153dcd954d30 server_cookie=82db860c290f353d in_seq=0 out_seq=0 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb04cff9640 1 -- 172.21.15.89:0/1334416278 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f14110da20 con 0x55f1411b2820 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb04cff9640 10 client.25004 handle_command_reply: tid=0 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb072450740 1 client.25004 shutdown 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb072450740 1 -- 172.21.15.89:0/1334416278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119c9a0 msgr2=0x55f1411a6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb072450740 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119c9a0 0x55f1411a6c20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb05809b890 tx=0x7fb05806b570 comp rx=0 tx=0).stop 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb072450740 1 -- 172.21.15.89:0/1334416278 shutdown_connections 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb072450740 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f1411b2820 0x55f1411b4c30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb072450740 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f1411a2370 0x55f1411ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb072450740 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb04cff9640 0 client.25004 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb072450740 1 --2- 172.21.15.89:0/1334416278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f14119c9a0 0x55f1411a6c20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.251+0000 7fb072450740 1 -- 172.21.15.89:0/1334416278 >> 172.21.15.89:0/1334416278 conn(0x55f141198010 msgr2=0x55f141199820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.252+0000 7fb072450740 1 -- 172.21.15.89:0/1334416278 shutdown_connections 2024-09-15T23:24:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.252+0000 7fb072450740 1 -- 172.21.15.89:0/1334416278 wait complete. 2024-09-15T23:24:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.252+0000 7fb072450740 20 client.25004 trim_cache size 0 max 16384 2024-09-15T23:24:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb072450740 1 Processor -- start 2024-09-15T23:24:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb072450740 1 -- start start 2024-09-15T23:24:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb072450740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 0x55f1411970b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb072450740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb072450740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1411a2370 0x55f1411b9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb072450740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb0681acab0 con 0x55f14119d2e0 2024-09-15T23:24:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb072450740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb0681ae5f0 con 0x55f1411a2370 2024-09-15T23:24:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb072450740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb0681ad180 con 0x55f14119c9a0 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06cb71640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06cb71640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49218/0 (socket says 172.21.15.89:49218) 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06cb71640 1 -- 172.21.15.89:0/3035835504 learned_addr learned my addr 172.21.15.89:0/3035835504 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06d372640 1 --2- 172.21.15.89:0/3035835504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 0x55f1411970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06db73640 1 --2- 172.21.15.89:0/3035835504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1411a2370 0x55f1411b9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06cb71640 1 -- 172.21.15.89:0/3035835504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 msgr2=0x55f1411970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06cb71640 1 --2- 172.21.15.89:0/3035835504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 0x55f1411970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06cb71640 1 -- 172.21.15.89:0/3035835504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1411a2370 msgr2=0x55f1411b9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06cb71640 1 --2- 172.21.15.89:0/3035835504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1411a2370 0x55f1411b9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06cb71640 1 -- 172.21.15.89:0/3035835504 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1411b52b0 con 0x55f14119d2e0 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06d372640 1 --2- 172.21.15.89:0/3035835504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 0x55f1411970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb06cb71640 1 --2- 172.21.15.89:0/3035835504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ae0c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x55f1411b54e0 tx=0x7fb05c004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1b78158c676db4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.266+0000 7fb04effd640 1 -- 172.21.15.89:0/3035835504 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c020020 con 0x55f14119d2e0 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb04effd640 1 -- 172.21.15.89:0/3035835504 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb05c003d00 con 0x55f14119d2e0 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb04effd640 1 -- 172.21.15.89:0/3035835504 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c012870 con 0x55f14119d2e0 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb072450740 1 -- 172.21.15.89:0/3035835504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 msgr2=0x55f1411ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb072450740 1 --2- 172.21.15.89:0/3035835504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ae0c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x55f1411b54e0 tx=0x7fb05c004410 comp rx=0 tx=0).stop 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb072450740 1 -- 172.21.15.89:0/3035835504 shutdown_connections 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb072450740 1 --2- 172.21.15.89:0/3035835504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1411a2370 0x55f1411b9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb072450740 1 --2- 172.21.15.89:0/3035835504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ae0c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb072450740 1 --2- 172.21.15.89:0/3035835504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 0x55f1411970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb072450740 1 -- 172.21.15.89:0/3035835504 >> 172.21.15.89:0/3035835504 conn(0x55f141198010 msgr2=0x55f1411969a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb072450740 1 -- 172.21.15.89:0/3035835504 shutdown_connections 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb072450740 1 -- 172.21.15.89:0/3035835504 wait complete. 2024-09-15T23:24:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.267+0000 7fb072450740 1 -- start start 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb072450740 1 Processor -- start 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb072450740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 0x55f1411ac140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb072450740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ac680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb072450740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1411a2370 0x55f1411ad640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb072450740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb0681adf20 con 0x55f14119d2e0 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb072450740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb0681ad850 con 0x55f1411a2370 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb072450740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb0681aaa90 con 0x55f14119c9a0 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb072450740 10 client.0 _ll_register_callbacks cb 0x7fb0580035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb06cb71640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb06cb71640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ac680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49226/0 (socket says 172.21.15.89:49226) 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb06cb71640 1 -- 172.21.15.89:0/1529763934 learned_addr learned my addr 172.21.15.89:0/1529763934 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb06db73640 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1411a2370 0x55f1411ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb06d372640 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 0x55f1411ac140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb06cb71640 1 -- 172.21.15.89:0/1529763934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 msgr2=0x55f1411ac140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb06cb71640 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 0x55f1411ac140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb06cb71640 1 -- 172.21.15.89:0/1529763934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1411a2370 msgr2=0x55f1411ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb06cb71640 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1411a2370 0x55f1411ad640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.268+0000 7fb06cb71640 1 -- 172.21.15.89:0/1529763934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1411adc50 con 0x55f14119d2e0 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb06cb71640 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ac680 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0124c0 tx=0x7fb05c009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e16cb0b7c7e4666 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb04cff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb04cff9640 1 -- 172.21.15.89:0/1529763934 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c012a00 con 0x55f14119d2e0 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb04cff9640 1 -- 172.21.15.89:0/1529763934 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb05c012ba0 con 0x55f14119d2e0 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb072450740 1 -- 172.21.15.89:0/1529763934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb05c003d00 con 0x55f14119d2e0 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb04cff9640 1 -- 172.21.15.89:0/1529763934 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb05c012da0 con 0x55f14119d2e0 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb072450740 10 client.15594 fetch_fsmap learned FSMap version 29 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb072450740 1 -- 172.21.15.89:0/1529763934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f14121dde0 con 0x55f14119d2e0 2024-09-15T23:24:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb04cff9640 1 -- 172.21.15.89:0/1529763934 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb05c003d00 con 0x55f14119d2e0 2024-09-15T23:24:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb04cff9640 1 -- 172.21.15.89:0/1529763934 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb05c02d030 con 0x55f14119d2e0 2024-09-15T23:24:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb072450740 10 client.15594 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:24:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb072450740 10 client.15594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb072450740 20 client.15594 populate_metadata read hostname 'smithi089' 2024-09-15T23:24:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb072450740 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f14121e710 0x55f141220b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb072450740 4 client.15594 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:24:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb072450740 1 -- 172.21.15.89:0/1529763934 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f141107d70 con 0x55f14121e710 2024-09-15T23:24:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.269+0000 7fb06d372640 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f14121e710 0x55f141220b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.270+0000 7fb04cff9640 10 client.15594 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.270+0000 7fb06d372640 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f14121e710 0x55f141220b30 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1a9594823a6fdf3 server_cookie=321bf83ccc1daf55 in_seq=0 out_seq=0 2024-09-15T23:24:54.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb04cff9640 1 -- 172.21.15.89:0/1529763934 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f141107d70 con 0x55f14121e710 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb04cff9640 10 client.15594 handle_command_reply: tid=0 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 client.15594 shutdown 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 -- 172.21.15.89:0/1529763934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 msgr2=0x55f1411ac680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ac680 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0124c0 tx=0x7fb05c009a40 comp rx=0 tx=0).stop 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 -- 172.21.15.89:0/1529763934 shutdown_connections 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f14121e710 0x55f141220b30 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1411a2370 0x55f1411ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f14119d2e0 0x55f1411ac680 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 --2- 172.21.15.89:0/1529763934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f14119c9a0 0x55f1411ac140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb04cff9640 0 client.15594 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 -- 172.21.15.89:0/1529763934 >> 172.21.15.89:0/1529763934 conn(0x55f141198010 msgr2=0x55f1411996e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 -- 172.21.15.89:0/1529763934 shutdown_connections 2024-09-15T23:24:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 1 -- 172.21.15.89:0/1529763934 wait complete. 2024-09-15T23:24:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.278+0000 7fb072450740 20 client.15594 trim_cache size 0 max 16384 2024-09-15T23:24:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:24:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:24:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:24:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:24:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 -- 172.21.15.89:0/1924989808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0500a3390 msgr2=0x7fb0500a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0500a3390 0x7fb0500a5850 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb06004c000 tx=0x7fb06004a000 comp rx=0 tx=0).stop 2024-09-15T23:24:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 -- 172.21.15.89:0/1924989808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb068100a50 msgr2=0x7fb0681049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb068100a50 0x7fb0681049e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fb05800d770 tx=0x7fb05800dc30 comp rx=0 tx=0).stop 2024-09-15T23:24:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 -- 172.21.15.89:0/1924989808 shutdown_connections 2024-09-15T23:24:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0500a3390 0x7fb0500a5850 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb068102600 0x7fb068103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068101c40 0x7fb068103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 --2- 172.21.15.89:0/1924989808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb068100a50 0x7fb0681049e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 -- 172.21.15.89:0/1924989808 >> 172.21.15.89:0/1924989808 conn(0x7fb0680fc1b0 msgr2=0x7fb0680fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 -- 172.21.15.89:0/1924989808 shutdown_connections 2024-09-15T23:24:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:24:54.279+0000 7fb06f5d6640 1 -- 172.21.15.89:0/1924989808 wait complete. 2024-09-15T23:24:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:54 smithi120 ceph-mon[35250]: pgmap v376: 129 pgs: 129 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2024-09-15T23:24:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:54 smithi174 ceph-mon[35586]: pgmap v376: 129 pgs: 129 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2024-09-15T23:24:54.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:54 smithi089 ceph-mon[30933]: pgmap v376: 129 pgs: 129 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2024-09-15T23:24:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:55 smithi120 ceph-mon[35250]: pgmap v377: 129 pgs: 129 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-09-15T23:24:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:55 smithi174 ceph-mon[35586]: pgmap v377: 129 pgs: 129 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-09-15T23:24:55.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:55 smithi089 ceph-mon[30933]: pgmap v377: 129 pgs: 129 active+clean; 665 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-09-15T23:24:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:24:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:24:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:56 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:24:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:24:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:24:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:56 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:24:56.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:24:56.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:24:56.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:56 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:24:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:24:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:24:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:24:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:24:57 smithi120 ceph-mon[35250]: pgmap v378: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-09-15T23:24:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:24:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:24:57.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:24:57.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:24:57 smithi174 ceph-mon[35586]: pgmap v378: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-09-15T23:24:57.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:24:57.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:24:57.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:24:57.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:24:57 smithi089 ceph-mon[30933]: pgmap v378: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-09-15T23:25:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:00 smithi120 ceph-mon[35250]: pgmap v379: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T23:25:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:00 smithi174 ceph-mon[35586]: pgmap v379: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T23:25:00.541 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:00 smithi089 ceph-mon[30933]: pgmap v379: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T23:25:01.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:25:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:02 smithi089 ceph-mon[30933]: pgmap v380: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T23:25:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:02 smithi120 ceph-mon[35250]: pgmap v380: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T23:25:02.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:02 smithi174 ceph-mon[35586]: pgmap v380: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T23:25:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.741+0000 7eff60dc1640 1 -- 172.21.15.89:0/1262822843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff5c1082a0 msgr2=0x7eff5c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.741+0000 7eff60dc1640 1 --2- 172.21.15.89:0/1262822843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff5c1082a0 0x7eff5c10ec40 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7eff54009cd0 tx=0x7eff5402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.741+0000 7eff60dc1640 1 -- 172.21.15.89:0/1262822843 shutdown_connections 2024-09-15T23:25:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.741+0000 7eff60dc1640 1 --2- 172.21.15.89:0/1262822843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff5c1082a0 0x7eff5c10ec40 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.741+0000 7eff60dc1640 1 --2- 172.21.15.89:0/1262822843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff5c1036e0 0x7eff5c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.741+0000 7eff60dc1640 1 --2- 172.21.15.89:0/1262822843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff5c102d30 0x7eff5c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.741+0000 7eff60dc1640 1 -- 172.21.15.89:0/1262822843 >> 172.21.15.89:0/1262822843 conn(0x7eff5c0fc7d0 msgr2=0x7eff5c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.741+0000 7eff60dc1640 1 -- 172.21.15.89:0/1262822843 shutdown_connections 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.741+0000 7eff60dc1640 1 -- 172.21.15.89:0/1262822843 wait complete. 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff60dc1640 1 Processor -- start 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff60dc1640 1 -- start start 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff60dc1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff5c102d30 0x7eff5c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff60dc1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff5c1036e0 0x7eff5c198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff60dc1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff5c1082a0 0x7eff5c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff60dc1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c112100 con 0x7eff5c1036e0 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff60dc1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c111f80 con 0x7eff5c102d30 2024-09-15T23:25:02.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff60dc1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c112280 con 0x7eff5c1082a0 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff5c1036e0 0x7eff5c198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff5c1036e0 0x7eff5c198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51072/0 (socket says 172.21.15.89:51072) 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.742+0000 7eff59d9b640 1 -- 172.21.15.89:0/3541312671 learned_addr learned my addr 172.21.15.89:0/3541312671 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff5a59c640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff5c102d30 0x7eff5c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff5ad9d640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff5c1082a0 0x7eff5c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff59d9b640 1 -- 172.21.15.89:0/3541312671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff5c1082a0 msgr2=0x7eff5c19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff59d9b640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff5c1082a0 0x7eff5c19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff59d9b640 1 -- 172.21.15.89:0/3541312671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff5c102d30 msgr2=0x7eff5c198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff59d9b640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff5c102d30 0x7eff5c198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff59d9b640 1 -- 172.21.15.89:0/3541312671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff5c19d500 con 0x7eff5c1036e0 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff59d9b640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff5c1036e0 0x7eff5c198a90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7eff4400c8c0 tx=0x7eff4400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:02.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff537fe640 1 -- 172.21.15.89:0/3541312671 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44007bb0 con 0x7eff5c1036e0 2024-09-15T23:25:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff537fe640 1 -- 172.21.15.89:0/3541312671 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff44007d50 con 0x7eff5c1036e0 2024-09-15T23:25:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff60dc1640 1 -- 172.21.15.89:0/3541312671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff5c19d7f0 con 0x7eff5c1036e0 2024-09-15T23:25:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff537fe640 1 -- 172.21.15.89:0/3541312671 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44018790 con 0x7eff5c1036e0 2024-09-15T23:25:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.743+0000 7eff60dc1640 1 -- 172.21.15.89:0/3541312671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff5c1a50d0 con 0x7eff5c1036e0 2024-09-15T23:25:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.745+0000 7eff537fe640 1 -- 172.21.15.89:0/3541312671 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7eff44018970 con 0x7eff5c1036e0 2024-09-15T23:25:02.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.746+0000 7eff537fe640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff480a32d0 0x7eff480a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.746+0000 7eff5a59c640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff480a32d0 0x7eff480a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.746+0000 7eff537fe640 1 -- 172.21.15.89:0/3541312671 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7eff440d0080 con 0x7eff5c1036e0 2024-09-15T23:25:02.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff6319a740 1 Processor -- start 2024-09-15T23:25:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff5a59c640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff480a32d0 0x7eff480a5790 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009990 tx=0x7eff4c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff6319a740 1 -- start start 2024-09-15T23:25:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff6319a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab64a0 0x558053ab6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff6319a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab6dc0 0x558053ac1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff6319a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053ac3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff6319a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558053a1c2e0 con 0x558053ab64a0 2024-09-15T23:25:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff6319a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580539c3ce0 con 0x558053ab6dc0 2024-09-15T23:25:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff6319a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558053a182a0 con 0x558053ac1680 2024-09-15T23:25:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff5a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab64a0 0x558053ab6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff5a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab64a0 0x558053ab6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51074/0 (socket says 172.21.15.89:51074) 2024-09-15T23:25:02.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff5a59c640 1 -- 172.21.15.89:0/653926228 learned_addr learned my addr 172.21.15.89:0/653926228 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:02.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.747+0000 7eff59d9b640 1 --2- 172.21.15.89:0/653926228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab6dc0 0x558053ac1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.748+0000 7eff5ad9d640 1 --2- 172.21.15.89:0/653926228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053ac3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.748+0000 7eff5a59c640 1 -- 172.21.15.89:0/653926228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 msgr2=0x558053ac3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:02.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.748+0000 7eff5a59c640 1 --2- 172.21.15.89:0/653926228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053ac3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.748+0000 7eff5a59c640 1 -- 172.21.15.89:0/653926228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab6dc0 msgr2=0x558053ac1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:02.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.748+0000 7eff5a59c640 1 --2- 172.21.15.89:0/653926228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab6dc0 0x558053ac1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.748+0000 7eff5a59c640 1 -- 172.21.15.89:0/653926228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580539a8440 con 0x558053ab64a0 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.748+0000 7eff5a59c640 1 --2- 172.21.15.89:0/653926228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab64a0 0x558053ab6880 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009fd0 tx=0x7eff4c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be72d45acfdec3d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.748+0000 7eff50ff9640 1 -- 172.21.15.89:0/653926228 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c01b040 con 0x558053ab64a0 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.748+0000 7eff50ff9640 1 -- 172.21.15.89:0/653926228 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff4c024070 con 0x558053ab64a0 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.748+0000 7eff50ff9640 1 -- 172.21.15.89:0/653926228 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4c01f3f0 con 0x558053ab64a0 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 -- 172.21.15.89:0/653926228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab64a0 msgr2=0x558053ab6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 --2- 172.21.15.89:0/653926228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab64a0 0x558053ab6880 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009fd0 tx=0x7eff4c011000 comp rx=0 tx=0).stop 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 -- 172.21.15.89:0/653926228 shutdown_connections 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 --2- 172.21.15.89:0/653926228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053ac3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 --2- 172.21.15.89:0/653926228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab6dc0 0x558053ac1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 --2- 172.21.15.89:0/653926228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab64a0 0x558053ab6880 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 -- 172.21.15.89:0/653926228 >> 172.21.15.89:0/653926228 conn(0x558053ab2a30 msgr2=0x558053ab2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 -- 172.21.15.89:0/653926228 shutdown_connections 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 -- 172.21.15.89:0/653926228 wait complete. 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 -- start start 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.749+0000 7eff6319a740 1 Processor -- start 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff6319a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab64a0 0x558053ac0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff6319a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053abe210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff6319a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053abe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff6319a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558053ac4250 con 0x558053ab6dc0 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff6319a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580539d7050 con 0x558053ab64a0 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff6319a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558053ac43d0 con 0x558053ac1680 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053abe210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053abe210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51086/0 (socket says 172.21.15.89:51086) 2024-09-15T23:25:02.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff59d9b640 1 -- 172.21.15.89:0/2361231802 learned_addr learned my addr 172.21.15.89:0/2361231802 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff6319a740 10 client.0 _ll_register_callbacks cb 0x558053a17160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff5a59c640 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab64a0 0x558053ac0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff5ad9d640 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053abe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff59d9b640 1 -- 172.21.15.89:0/2361231802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 msgr2=0x558053abe750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff59d9b640 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053abe750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff59d9b640 1 -- 172.21.15.89:0/2361231802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab64a0 msgr2=0x558053ac0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff59d9b640 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab64a0 0x558053ac0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff59d9b640 1 -- 172.21.15.89:0/2361231802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55805378d3c0 con 0x558053ab6dc0 2024-09-15T23:25:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff59d9b640 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053abe210 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7eff44007590 tx=0x7eff4409c150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1dcfe634011db05 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:02.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.750+0000 7eff41ffb640 1 -- 172.21.15.89:0/2361231802 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff440b4070 con 0x558053ab6dc0 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff41ffb640 1 -- 172.21.15.89:0/2361231802 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7eff440a0190 con 0x558053ab6dc0 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff6319a740 1 -- 172.21.15.89:0/2361231802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff4c024070 con 0x558053ab6dc0 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff41ffb640 1 -- 172.21.15.89:0/2361231802 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7eff440bc0b0 con 0x558053ab6dc0 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff6319a740 10 client.15606 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff6319a740 1 -- 172.21.15.89:0/2361231802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558053acfc20 con 0x558053ab6dc0 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff41ffb640 1 -- 172.21.15.89:0/2361231802 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff4c024070 con 0x558053ab6dc0 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff41ffb640 1 -- 172.21.15.89:0/2361231802 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7eff440a4930 con 0x558053ab6dc0 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff6319a740 10 client.15606 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff6319a740 10 client.15606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff6319a740 20 client.15606 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff6319a740 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558053ad10e0 0x558053ad34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff6319a740 4 client.15606 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff6319a740 1 -- 172.21.15.89:0/2361231802 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558053a29850 con 0x558053ad10e0 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.751+0000 7eff5a59c640 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558053ad10e0 0x558053ad34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.752+0000 7eff41ffb640 10 client.15606 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.752+0000 7eff5a59c640 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558053ad10e0 0x558053ad34f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95d6412373466400 server_cookie=deadf65d635e846e in_seq=0 out_seq=0 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff41ffb640 1 -- 172.21.15.89:0/2361231802 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558053a29850 con 0x558053ad10e0 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff41ffb640 10 client.15606 handle_command_reply: tid=0 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 client.15606 shutdown 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 -- 172.21.15.89:0/2361231802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 msgr2=0x558053abe210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053abe210 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7eff44007590 tx=0x7eff4409c150 comp rx=0 tx=0).stop 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 -- 172.21.15.89:0/2361231802 shutdown_connections 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558053ad10e0 0x558053ad34f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053abe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff41ffb640 0 client.15606 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:02.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053abe210 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 --2- 172.21.15.89:0/2361231802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab64a0 0x558053ac0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 -- 172.21.15.89:0/2361231802 >> 172.21.15.89:0/2361231802 conn(0x558053ab2a30 msgr2=0x558053ab1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:02.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 -- 172.21.15.89:0/2361231802 shutdown_connections 2024-09-15T23:25:02.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 1 -- 172.21.15.89:0/2361231802 wait complete. 2024-09-15T23:25:02.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.753+0000 7eff6319a740 20 client.15606 trim_cache size 0 max 16384 2024-09-15T23:25:02.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff6319a740 1 Processor -- start 2024-09-15T23:25:02.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff6319a740 1 -- start start 2024-09-15T23:25:02.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff6319a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab64a0 0x558053acd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff6319a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ad8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff6319a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053ad9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff6319a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1aeeb0 con 0x558053ab6dc0 2024-09-15T23:25:02.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff6319a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1b09f0 con 0x558053ab64a0 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff6319a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1af580 con 0x558053ac1680 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ad8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ad8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51096/0 (socket says 172.21.15.89:51096) 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.771+0000 7eff59d9b640 1 -- 172.21.15.89:0/2658867476 learned_addr learned my addr 172.21.15.89:0/2658867476 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff5a59c640 1 --2- 172.21.15.89:0/2658867476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab64a0 0x558053acd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff5ad9d640 1 --2- 172.21.15.89:0/2658867476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053ad9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff59d9b640 1 -- 172.21.15.89:0/2658867476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 msgr2=0x558053ad9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff59d9b640 1 --2- 172.21.15.89:0/2658867476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053ad9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff59d9b640 1 -- 172.21.15.89:0/2658867476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab64a0 msgr2=0x558053acd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff59d9b640 1 --2- 172.21.15.89:0/2658867476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab64a0 0x558053acd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff59d9b640 1 -- 172.21.15.89:0/2658867476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558053ab62d0 con 0x558053ab6dc0 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff59d9b640 1 --2- 172.21.15.89:0/2658867476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ad8dc0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7eff440a4890 tx=0x7eff4409c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b25353da33e27e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff43fff640 1 -- 172.21.15.89:0/2658867476 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff440b4070 con 0x558053ab6dc0 2024-09-15T23:25:02.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff43fff640 1 -- 172.21.15.89:0/2658867476 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff440a0190 con 0x558053ab6dc0 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.772+0000 7eff43fff640 1 -- 172.21.15.89:0/2658867476 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44018c60 con 0x558053ab6dc0 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 -- 172.21.15.89:0/2658867476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 msgr2=0x558053ad8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 --2- 172.21.15.89:0/2658867476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ad8dc0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7eff440a4890 tx=0x7eff4409c050 comp rx=0 tx=0).stop 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 -- 172.21.15.89:0/2658867476 shutdown_connections 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 --2- 172.21.15.89:0/2658867476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ac1680 0x558053ad9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 --2- 172.21.15.89:0/2658867476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ad8dc0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 --2- 172.21.15.89:0/2658867476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ab64a0 0x558053acd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 -- 172.21.15.89:0/2658867476 >> 172.21.15.89:0/2658867476 conn(0x558053ab2a30 msgr2=0x558053abd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 -- 172.21.15.89:0/2658867476 shutdown_connections 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 -- 172.21.15.89:0/2658867476 wait complete. 2024-09-15T23:25:02.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 -- start start 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.773+0000 7eff6319a740 1 Processor -- start 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff6319a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ab64a0 0x558053ac5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff6319a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ac6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff6319a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ac1680 0x558053b34d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff6319a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1b0320 con 0x558053ab6dc0 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff6319a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1afc50 con 0x558053ac1680 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff6319a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1ace90 con 0x558053ab64a0 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff6319a740 10 client.0 _ll_register_callbacks cb 0x7eff4400c810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ac6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff59d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ac6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51112/0 (socket says 172.21.15.89:51112) 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff59d9b640 1 -- 172.21.15.89:0/3361276278 learned_addr learned my addr 172.21.15.89:0/3361276278 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff5a59c640 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ab64a0 0x558053ac5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff59d9b640 1 -- 172.21.15.89:0/3361276278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ab64a0 msgr2=0x558053ac5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff59d9b640 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ab64a0 0x558053ac5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff5ad9d640 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ac1680 0x558053b34d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff59d9b640 1 -- 172.21.15.89:0/3361276278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ac1680 msgr2=0x558053b34d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff59d9b640 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ac1680 0x558053b34d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff59d9b640 1 -- 172.21.15.89:0/3361276278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558053ab3ae0 con 0x558053ab6dc0 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff5ad9d640 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ac1680 0x558053b34d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff59d9b640 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ac6100 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7eff4409a400 tx=0x7eff4409a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01360a71012cb32 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.774+0000 7eff41ffb640 1 -- 172.21.15.89:0/3361276278 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff440b4070 con 0x558053ab6dc0 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff41ffb640 1 -- 172.21.15.89:0/3361276278 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7eff44097ea0 con 0x558053ab6dc0 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff6319a740 1 -- 172.21.15.89:0/3361276278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff440a0190 con 0x558053ab6dc0 2024-09-15T23:25:02.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff41ffb640 1 -- 172.21.15.89:0/3361276278 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7eff44084050 con 0x558053ab6dc0 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff6319a740 10 client.15612 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff6319a740 1 -- 172.21.15.89:0/3361276278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558053b3c520 con 0x558053ab6dc0 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff41ffb640 1 -- 172.21.15.89:0/3361276278 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff440a0190 con 0x558053ab6dc0 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff41ffb640 1 -- 172.21.15.89:0/3361276278 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7eff440a1690 con 0x558053ab6dc0 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff6319a740 10 client.15612 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff6319a740 10 client.15612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff6319a740 20 client.15612 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff6319a740 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558053ad10e0 0x558053b3ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff6319a740 4 client.15612 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff6319a740 1 -- 172.21.15.89:0/3361276278 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558053a23d60 con 0x558053ad10e0 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.775+0000 7eff5a59c640 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558053ad10e0 0x558053b3ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff5a59c640 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558053ad10e0 0x558053b3ea90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae9700d9bd184ba5 server_cookie=1bab89b2470c8322 in_seq=0 out_seq=0 2024-09-15T23:25:02.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff41ffb640 10 client.15612 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff41ffb640 1 -- 172.21.15.89:0/3361276278 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558053a23d60 con 0x558053ad10e0 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff41ffb640 10 client.15612 handle_command_reply: tid=0 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 client.15612 shutdown 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 -- 172.21.15.89:0/3361276278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 msgr2=0x558053ac6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ac6100 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7eff4409a400 tx=0x7eff4409a660 comp rx=0 tx=0).stop 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 -- 172.21.15.89:0/3361276278 shutdown_connections 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558053ad10e0 0x558053b3ea90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558053ac1680 0x558053b34d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558053ab6dc0 0x558053ac6100 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 --2- 172.21.15.89:0/3361276278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558053ab64a0 0x558053ac5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 -- 172.21.15.89:0/3361276278 >> 172.21.15.89:0/3361276278 conn(0x558053ab2a30 msgr2=0x558053abfa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff41ffb640 0 client.15612 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:02.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 -- 172.21.15.89:0/3361276278 shutdown_connections 2024-09-15T23:25:02.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 1 -- 172.21.15.89:0/3361276278 wait complete. 2024-09-15T23:25:02.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.776+0000 7eff6319a740 20 client.15612 trim_cache size 0 max 16384 2024-09-15T23:25:02.788 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:25:02.788 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:25:02.788 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:25:02.788 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:25:02.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 -- 172.21.15.89:0/3541312671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff480a32d0 msgr2=0x7eff480a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:02.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff480a32d0 0x7eff480a5790 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009990 tx=0x7eff4c009110 comp rx=0 tx=0).stop 2024-09-15T23:25:02.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 -- 172.21.15.89:0/3541312671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff5c1036e0 msgr2=0x7eff5c198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:02.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff5c1036e0 0x7eff5c198a90 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7eff4400c8c0 tx=0x7eff4400ccc0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 -- 172.21.15.89:0/3541312671 shutdown_connections 2024-09-15T23:25:02.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff480a32d0 0x7eff480a5790 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff5c1082a0 0x7eff5c19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff5c1036e0 0x7eff5c198a90 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 --2- 172.21.15.89:0/3541312671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff5c102d30 0x7eff5c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:02.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 -- 172.21.15.89:0/3541312671 >> 172.21.15.89:0/3541312671 conn(0x7eff5c0fc7d0 msgr2=0x7eff5c106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:02.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 -- 172.21.15.89:0/3541312671 shutdown_connections 2024-09-15T23:25:02.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:02.777+0000 7eff60dc1640 1 -- 172.21.15.89:0/3541312671 wait complete. 2024-09-15T23:25:03.671 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:25:03.671 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:25:03.934 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:04.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:04 smithi089 ceph-mon[30933]: pgmap v381: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-15T23:25:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:04 smithi120 ceph-mon[35250]: pgmap v381: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-15T23:25:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.458+0000 7ffa18393640 1 -- 172.21.15.89:0/286241386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa10100e20 msgr2=0x7ffa10101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.458+0000 7ffa18393640 1 --2- 172.21.15.89:0/286241386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa10100e20 0x7ffa10101200 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009900 tx=0x7ffa0c02a480 comp rx=0 tx=0).stop 2024-09-15T23:25:04.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.458+0000 7ffa18393640 1 -- 172.21.15.89:0/286241386 shutdown_connections 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.458+0000 7ffa18393640 1 --2- 172.21.15.89:0/286241386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa10102ef0 0x7ffa1010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.458+0000 7ffa18393640 1 --2- 172.21.15.89:0/286241386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa10102570 0x7ffa101029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.458+0000 7ffa18393640 1 --2- 172.21.15.89:0/286241386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa10100e20 0x7ffa10101200 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.458+0000 7ffa18393640 1 -- 172.21.15.89:0/286241386 >> 172.21.15.89:0/286241386 conn(0x7ffa100fc7d0 msgr2=0x7ffa100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.458+0000 7ffa18393640 1 -- 172.21.15.89:0/286241386 shutdown_connections 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.458+0000 7ffa18393640 1 -- 172.21.15.89:0/286241386 wait complete. 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.459+0000 7ffa18393640 1 Processor -- start 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.459+0000 7ffa18393640 1 -- start start 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.459+0000 7ffa18393640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa10100e20 0x7ffa10199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa18393640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa10102570 0x7ffa10199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa18393640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa10102ef0 0x7ffa101a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:04.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa18393640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffa10110fa0 con 0x7ffa10102ef0 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa18393640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa10110e20 con 0x7ffa10102570 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa18393640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa10111120 con 0x7ffa10100e20 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa16930640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa10102ef0 0x7ffa101a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa16930640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa10102ef0 0x7ffa101a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51138/0 (socket says 172.21.15.89:51138) 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa16930640 1 -- 172.21.15.89:0/2443900069 learned_addr learned my addr 172.21.15.89:0/2443900069 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa1612f640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa10100e20 0x7ffa10199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa1592e640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa10102570 0x7ffa10199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa16930640 1 -- 172.21.15.89:0/2443900069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa10100e20 msgr2=0x7ffa10199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa16930640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa10100e20 0x7ffa10199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa16930640 1 -- 172.21.15.89:0/2443900069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa10102570 msgr2=0x7ffa10199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa16930640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa10102570 0x7ffa10199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa16930640 1 -- 172.21.15.89:0/2443900069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa101a4910 con 0x7ffa10102ef0 2024-09-15T23:25:04.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa1612f640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa10100e20 0x7ffa10199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa16930640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa10102ef0 0x7ffa101a43d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800eaa0 tx=0x7ffa0800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa077fe640 1 -- 172.21.15.89:0/2443900069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0800cdb0 con 0x7ffa10102ef0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa18393640 1 -- 172.21.15.89:0/2443900069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa101a4ae0 con 0x7ffa10102ef0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa077fe640 1 -- 172.21.15.89:0/2443900069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa08005020 con 0x7ffa10102ef0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa077fe640 1 -- 172.21.15.89:0/2443900069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa080116b0 con 0x7ffa10102ef0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.460+0000 7ffa18393640 1 -- 172.21.15.89:0/2443900069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa101a4ff0 con 0x7ffa10102ef0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.462+0000 7ffa18393640 1 -- 172.21.15.89:0/2443900069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa10101200 con 0x7ffa10102ef0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.463+0000 7ffa077fe640 1 -- 172.21.15.89:0/2443900069 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7ffa08005560 con 0x7ffa10102ef0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.464+0000 7ffa077fe640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9fc0a32d0 0x7ff9fc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.464+0000 7ffa1612f640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9fc0a32d0 0x7ff9fc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.464+0000 7ffa077fe640 1 -- 172.21.15.89:0/2443900069 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ffa080d04e0 con 0x7ffa10102ef0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.464+0000 7ffa1612f640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9fc0a32d0 0x7ff9fc0a5790 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c000c00 tx=0x7ffa0c005a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:04.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.467+0000 7ffa077fe640 1 -- 172.21.15.89:0/2443900069 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffa08016030 con 0x7ffa10102ef0 2024-09-15T23:25:04.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:04 smithi174 ceph-mon[35586]: pgmap v381: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-09-15T23:25:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.690+0000 7ffa18393640 1 -- 172.21.15.89:0/2443900069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa101a5280 con 0x7ffa10102ef0 2024-09-15T23:25:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.690+0000 7ffa077fe640 1 -- 172.21.15.89:0/2443900069 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7ffa0809ccb0 con 0x7ffa10102ef0 2024-09-15T23:25:04.692 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:25:04.692 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:25:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:25:04.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 -- 172.21.15.89:0/2443900069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9fc0a32d0 msgr2=0x7ff9fc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:04.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9fc0a32d0 0x7ff9fc0a5790 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c000c00 tx=0x7ffa0c005a40 comp rx=0 tx=0).stop 2024-09-15T23:25:04.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 -- 172.21.15.89:0/2443900069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa10102ef0 msgr2=0x7ffa101a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:04.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa10102ef0 0x7ffa101a43d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800eaa0 tx=0x7ffa0800ef60 comp rx=0 tx=0).stop 2024-09-15T23:25:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 -- 172.21.15.89:0/2443900069 shutdown_connections 2024-09-15T23:25:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9fc0a32d0 0x7ff9fc0a5790 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa10102ef0 0x7ffa101a43d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa10102570 0x7ffa10199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 --2- 172.21.15.89:0/2443900069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa10100e20 0x7ffa10199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 -- 172.21.15.89:0/2443900069 >> 172.21.15.89:0/2443900069 conn(0x7ffa100fc7d0 msgr2=0x7ffa100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 -- 172.21.15.89:0/2443900069 shutdown_connections 2024-09-15T23:25:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:04.693+0000 7ffa18393640 1 -- 172.21.15.89:0/2443900069 wait complete. 2024-09-15T23:25:05.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:05 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2443900069' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:05.348 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:25:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:05 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2443900069' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:05 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2443900069' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:05.605 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:05.804 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.2 2024-09-15T23:25:05.805 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:25:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.122+0000 7f71c28d7640 1 -- 172.21.15.89:0/3551387872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71bc102d30 msgr2=0x7f71bc103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.122+0000 7f71c28d7640 1 --2- 172.21.15.89:0/3551387872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71bc102d30 0x7f71bc103110 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009930 tx=0x7f71ac02a420 comp rx=0 tx=0).stop 2024-09-15T23:25:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.122+0000 7f71c28d7640 1 -- 172.21.15.89:0/3551387872 shutdown_connections 2024-09-15T23:25:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.122+0000 7f71c28d7640 1 --2- 172.21.15.89:0/3551387872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71bc1082a0 0x7f71bc10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.122+0000 7f71c28d7640 1 --2- 172.21.15.89:0/3551387872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71bc1036e0 0x7f71bc107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.122+0000 7f71c28d7640 1 --2- 172.21.15.89:0/3551387872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71bc102d30 0x7f71bc103110 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.122+0000 7f71c28d7640 1 -- 172.21.15.89:0/3551387872 >> 172.21.15.89:0/3551387872 conn(0x7f71bc0fc7d0 msgr2=0x7f71bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.122+0000 7f71c28d7640 1 -- 172.21.15.89:0/3551387872 shutdown_connections 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.122+0000 7f71c28d7640 1 -- 172.21.15.89:0/3551387872 wait complete. 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71c28d7640 1 Processor -- start 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71c28d7640 1 -- start start 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71c28d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71bc102d30 0x7f71bc100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71c28d7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71bc1036e0 0x7f71bc100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71c28d7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71bc1082a0 0x7f71bc101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71c28d7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1120a0 con 0x7f71bc1036e0 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71c28d7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc111f20 con 0x7f71bc1082a0 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71c28d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc112220 con 0x7f71bc102d30 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71bc1036e0 0x7f71bc100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71bc1036e0 0x7f71bc100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44242/0 (socket says 172.21.15.89:44242) 2024-09-15T23:25:06.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71b3fff640 1 -- 172.21.15.89:0/1468060438 learned_addr learned my addr 172.21.15.89:0/1468060438 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71c0e74640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71bc1082a0 0x7f71bc101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71bbfff640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71bc102d30 0x7f71bc100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71b3fff640 1 -- 172.21.15.89:0/1468060438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71bc102d30 msgr2=0x7f71bc100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71b3fff640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71bc102d30 0x7f71bc100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71b3fff640 1 -- 172.21.15.89:0/1468060438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71bc1082a0 msgr2=0x7f71bc101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71b3fff640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71bc1082a0 0x7f71bc101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.123+0000 7f71b3fff640 1 -- 172.21.15.89:0/1468060438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71bc1a6a80 con 0x7f71bc1036e0 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.124+0000 7f71b3fff640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71bc1036e0 0x7f71bc100ed0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c970 tx=0x7f71a400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.124+0000 7f71b9ffb640 1 -- 172.21.15.89:0/1468060438 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a4007bb0 con 0x7f71bc1036e0 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.124+0000 7f71b9ffb640 1 -- 172.21.15.89:0/1468060438 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71a4007d50 con 0x7f71bc1036e0 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.124+0000 7f71c28d7640 1 -- 172.21.15.89:0/1468060438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71bc1a6d70 con 0x7f71bc1036e0 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.124+0000 7f71b9ffb640 1 -- 172.21.15.89:0/1468060438 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a400f450 con 0x7f71bc1036e0 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.124+0000 7f71c28d7640 1 -- 172.21.15.89:0/1468060438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71bc1a7250 con 0x7f71bc1036e0 2024-09-15T23:25:06.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.126+0000 7f71b9ffb640 1 -- 172.21.15.89:0/1468060438 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f71a4004f60 con 0x7f71bc1036e0 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.126+0000 7f71c4cb0740 1 Processor -- start 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.126+0000 7f71c4cb0740 1 -- start start 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d25d0 0x5603053d2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053df760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5603053382f0 con 0x5603053d3b40 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5603052dfcf0 con 0x5603053dd370 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5603053342b0 con 0x5603053d25d0 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c0e74640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c0e74640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053d2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44252/0 (socket says 172.21.15.89:44252) 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c0e74640 1 -- 172.21.15.89:0/3000096425 learned_addr learned my addr 172.21.15.89:0/3000096425 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71bbfff640 1 --2- 172.21.15.89:0/3000096425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d25d0 0x5603053d2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71b3fff640 1 --2- 172.21.15.89:0/3000096425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053df760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71b9ffb640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f71a80a32d0 0x7f71a80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c0e74640 1 -- 172.21.15.89:0/3000096425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d25d0 msgr2=0x5603053d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c0e74640 1 --2- 172.21.15.89:0/3000096425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d25d0 0x5603053d2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c0e74640 1 -- 172.21.15.89:0/3000096425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 msgr2=0x5603053df760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c0e74640 1 --2- 172.21.15.89:0/3000096425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053df760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c0e74640 1 -- 172.21.15.89:0/3000096425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603052c43a0 con 0x5603053d3b40 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71c0e74640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f71a80a32d0 0x7f71a80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.127+0000 7f71b9ffb640 1 -- 172.21.15.89:0/1468060438 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f71a40cfc20 con 0x7f71bc1036e0 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.128+0000 7f71c0e74640 1 --2- 172.21.15.89:0/3000096425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053d2090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f71b4010440 tx=0x7f71b4010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1d525fe1a2b74e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.128+0000 7f71b27fc640 1 -- 172.21.15.89:0/3000096425 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b401d400 con 0x5603053d3b40 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.128+0000 7f71b27fc640 1 -- 172.21.15.89:0/3000096425 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b40134c0 con 0x5603053d3b40 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.128+0000 7f71b27fc640 1 -- 172.21.15.89:0/3000096425 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b4011800 con 0x5603053d3b40 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.128+0000 7f71c0e74640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f71a80a32d0 0x7f71a80a5790 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f71b4021000 tx=0x7f71b401e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 -- 172.21.15.89:0/3000096425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 msgr2=0x5603053d2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:06.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/3000096425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053d2090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f71b4010440 tx=0x7f71b4010900 comp rx=0 tx=0).stop 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 -- 172.21.15.89:0/3000096425 shutdown_connections 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/3000096425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053df760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/3000096425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d25d0 0x5603053d2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/3000096425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053d2090 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 -- 172.21.15.89:0/3000096425 >> 172.21.15.89:0/3000096425 conn(0x56030539b310 msgr2=0x5603053cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 -- 172.21.15.89:0/3000096425 shutdown_connections 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 -- 172.21.15.89:0/3000096425 wait complete. 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 -- start start 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.129+0000 7f71c4cb0740 1 Processor -- start 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d25d0 0x5603053dcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053da040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053da650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5603053e0070 con 0x5603053d3b40 2024-09-15T23:25:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5603052f3060 con 0x5603053dd370 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5603053e01f0 con 0x5603053d25d0 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71c4cb0740 10 client.0 _ll_register_callbacks cb 0x5603050468c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053da040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053da040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44254/0 (socket says 172.21.15.89:44254) 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71b3fff640 1 -- 172.21.15.89:0/2828411412 learned_addr learned my addr 172.21.15.89:0/2828411412 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71bbfff640 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d25d0 0x5603053dcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71c0e74640 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053da650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71b3fff640 1 -- 172.21.15.89:0/2828411412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d25d0 msgr2=0x5603053dcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71b3fff640 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d25d0 0x5603053dcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71b3fff640 1 -- 172.21.15.89:0/2828411412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 msgr2=0x5603053da650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71b3fff640 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053da650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71b3fff640 1 -- 172.21.15.89:0/2828411412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603050a93c0 con 0x5603053d3b40 2024-09-15T23:25:06.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71b3fff640 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053da040 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f71a4007590 tx=0x7f71a409b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12f57ce71a3a25 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f719b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f719b7fe640 1 -- 172.21.15.89:0/2828411412 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a401d030 con 0x5603053d3b40 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f719b7fe640 1 -- 172.21.15.89:0/2828411412 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f71a4005250 con 0x5603053d3b40 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2828411412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71b40134c0 con 0x5603053d3b40 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.130+0000 7f719b7fe640 1 -- 172.21.15.89:0/2828411412 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f71a409c3f0 con 0x5603053d3b40 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f71c4cb0740 10 client.15630 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2828411412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5603053ebba0 con 0x5603053d3b40 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f719b7fe640 1 -- 172.21.15.89:0/2828411412 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b40134c0 con 0x5603053d3b40 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f719b7fe640 1 -- 172.21.15.89:0/2828411412 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f71a409b140 con 0x5603053d3b40 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f71c4cb0740 10 client.15630 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f71c4cb0740 10 client.15630 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f71c4cb0740 20 client.15630 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5603053ecfb0 0x5603053ef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f71c4cb0740 4 client.15630 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2828411412 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560305345860 con 0x5603053ecfb0 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.131+0000 7f71bbfff640 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5603053ecfb0 0x5603053ef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.132+0000 7f719b7fe640 10 client.15630 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.132+0000 7f71bbfff640 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5603053ecfb0 0x5603053ef3a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8f5cd42bf16a3751 server_cookie=16a5c3b2df0f83a0 in_seq=0 out_seq=0 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.132+0000 7f719b7fe640 1 -- 172.21.15.89:0/2828411412 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560305345860 con 0x5603053ecfb0 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.132+0000 7f719b7fe640 10 client.15630 handle_command_reply: tid=0 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.132+0000 7f71c4cb0740 1 client.15630 shutdown 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2828411412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 msgr2=0x5603053da040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053da040 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f71a4007590 tx=0x7f71a409b8f0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2828411412 shutdown_connections 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5603053ecfb0 0x5603053ef3a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053da650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d3b40 0x5603053da040 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2828411412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d25d0 0x5603053dcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2828411412 >> 172.21.15.89:0/2828411412 conn(0x56030539b310 msgr2=0x5603053d4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f719b7fe640 0 client.15630 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2828411412 shutdown_connections 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2828411412 wait complete. 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.133+0000 7f71c4cb0740 20 client.15630 trim_cache size 0 max 16384 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.145+0000 7f71c4cb0740 1 Processor -- start 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.145+0000 7f71c4cb0740 1 -- start start 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053e9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d3b40 0x5603053d9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1b11e0 con 0x5603053d25d0 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1b2d20 con 0x5603053dd370 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1b18b0 con 0x5603053d3b40 2024-09-15T23:25:06.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71bbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71bbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053e9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44264/0 (socket says 172.21.15.89:44264) 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71bbfff640 1 -- 172.21.15.89:0/1472946926 learned_addr learned my addr 172.21.15.89:0/1472946926 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71b3fff640 1 --2- 172.21.15.89:0/1472946926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d3b40 0x5603053d9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71c0e74640 1 --2- 172.21.15.89:0/1472946926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71bbfff640 1 -- 172.21.15.89:0/1472946926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d3b40 msgr2=0x5603053d9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71bbfff640 1 --2- 172.21.15.89:0/1472946926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d3b40 0x5603053d9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71bbfff640 1 -- 172.21.15.89:0/1472946926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 msgr2=0x5603053f4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71bbfff640 1 --2- 172.21.15.89:0/1472946926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053f4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71bbfff640 1 -- 172.21.15.89:0/1472946926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603053d5120 con 0x5603053d25d0 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71bbfff640 1 --2- 172.21.15.89:0/1472946926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053e9310 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f71ac005c40 tx=0x7f71ac0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b733e41a8619aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.146+0000 7f71b1ffb640 1 -- 172.21.15.89:0/1472946926 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac048070 con 0x5603053d25d0 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71b1ffb640 1 -- 172.21.15.89:0/1472946926 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71ac005520 con 0x5603053d25d0 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71b1ffb640 1 -- 172.21.15.89:0/1472946926 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac035040 con 0x5603053d25d0 2024-09-15T23:25:06.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71c4cb0740 1 -- 172.21.15.89:0/1472946926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 msgr2=0x5603053e9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/1472946926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053e9310 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f71ac005c40 tx=0x7f71ac0058d0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71c4cb0740 1 -- 172.21.15.89:0/1472946926 shutdown_connections 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/1472946926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053f4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/1472946926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d3b40 0x5603053d9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/1472946926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053e9310 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71c4cb0740 1 -- 172.21.15.89:0/1472946926 >> 172.21.15.89:0/1472946926 conn(0x56030539b310 msgr2=0x5603053d8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71c4cb0740 1 -- 172.21.15.89:0/1472946926 shutdown_connections 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.147+0000 7f71c4cb0740 1 -- 172.21.15.89:0/1472946926 wait complete. 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71c4cb0740 1 -- start start 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71c4cb0740 1 Processor -- start 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053d7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d3b40 0x5603053d7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71c4cb0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053d7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1b2650 con 0x5603053d25d0 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1b1f80 con 0x5603053dd370 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71c4cb0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f71bc1af1c0 con 0x5603053d3b40 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71c4cb0740 10 client.0 _ll_register_callbacks cb 0x7f71a400c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71bbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053d7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71bbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053d7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44278/0 (socket says 172.21.15.89:44278) 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71bbfff640 1 -- 172.21.15.89:0/2807140181 learned_addr learned my addr 172.21.15.89:0/2807140181 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71b3fff640 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d3b40 0x5603053d7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71c0e74640 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053d7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71bbfff640 1 -- 172.21.15.89:0/2807140181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d3b40 msgr2=0x5603053d7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71bbfff640 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d3b40 0x5603053d7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71bbfff640 1 -- 172.21.15.89:0/2807140181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 msgr2=0x5603053d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71bbfff640 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053d7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.148+0000 7f71bbfff640 1 -- 172.21.15.89:0/2807140181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560305456de0 con 0x5603053d25d0 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f71c0e74640 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053d7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f71bbfff640 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053d7200 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f71ac02a940 tx=0x7f71ac0314e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0698ad9a099b1c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f719b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f719b7fe640 1 -- 172.21.15.89:0/2807140181 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac035050 con 0x5603053d25d0 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f719b7fe640 1 -- 172.21.15.89:0/2807140181 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f71ac042040 con 0x5603053d25d0 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2807140181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71ac005520 con 0x5603053d25d0 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f719b7fe640 1 -- 172.21.15.89:0/2807140181 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f71ac059050 con 0x5603053d25d0 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f71c4cb0740 10 client.15636 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2807140181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560305458080 con 0x5603053d25d0 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f719b7fe640 1 -- 172.21.15.89:0/2807140181 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71ac005520 con 0x5603053d25d0 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f719b7fe640 1 -- 172.21.15.89:0/2807140181 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f71ac0315b0 con 0x5603053d25d0 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f71c4cb0740 10 client.15636 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f71c4cb0740 10 client.15636 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.149+0000 7f71c4cb0740 20 client.15636 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:06.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.150+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5603054586c0 0x56030545aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.150+0000 7f71c4cb0740 4 client.15636 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.150+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2807140181 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56030533fd70 con 0x5603054586c0 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.150+0000 7f71b3fff640 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5603054586c0 0x56030545aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.150+0000 7f719b7fe640 10 client.15636 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.150+0000 7f71b3fff640 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5603054586c0 0x56030545aae0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3517058da07fbd28 server_cookie=6e7f0d87f4a83ae6 in_seq=0 out_seq=0 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.150+0000 7f719b7fe640 1 -- 172.21.15.89:0/2807140181 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56030533fd70 con 0x5603054586c0 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.150+0000 7f719b7fe640 10 client.15636 handle_command_reply: tid=0 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 client.15636 shutdown 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2807140181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 msgr2=0x5603053d7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053d7200 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f71ac02a940 tx=0x7f71ac0314e0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2807140181 shutdown_connections 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5603054586c0 0x56030545aae0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5603053dd370 0x5603053d7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5603053d3b40 0x5603053d7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f719b7fe640 0 client.15636 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:06.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 --2- 172.21.15.89:0/2807140181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5603053d25d0 0x5603053d7200 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2807140181 >> 172.21.15.89:0/2807140181 conn(0x56030539b310 msgr2=0x5603053d9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:06.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2807140181 shutdown_connections 2024-09-15T23:25:06.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 1 -- 172.21.15.89:0/2807140181 wait complete. 2024-09-15T23:25:06.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.151+0000 7f71c4cb0740 20 client.15636 trim_cache size 0 max 16384 2024-09-15T23:25:06.168 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:06.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 -- 172.21.15.89:0/1468060438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f71a80a32d0 msgr2=0x7f71a80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:06.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f71a80a32d0 0x7f71a80a5790 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f71b4021000 tx=0x7f71b401e000 comp rx=0 tx=0).stop 2024-09-15T23:25:06.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 -- 172.21.15.89:0/1468060438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71bc1036e0 msgr2=0x7f71bc100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:06.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71bc1036e0 0x7f71bc100ed0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c970 tx=0x7f71a400ce30 comp rx=0 tx=0).stop 2024-09-15T23:25:06.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 -- 172.21.15.89:0/1468060438 shutdown_connections 2024-09-15T23:25:06.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f71a80a32d0 0x7f71a80a5790 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71bc1082a0 0x7f71bc101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71bc1036e0 0x7f71bc100ed0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 --2- 172.21.15.89:0/1468060438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f71bc102d30 0x7f71bc100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 -- 172.21.15.89:0/1468060438 >> 172.21.15.89:0/1468060438 conn(0x7f71bc0fc7d0 msgr2=0x7f71bc105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 -- 172.21.15.89:0/1468060438 shutdown_connections 2024-09-15T23:25:06.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:06.152+0000 7f71c28d7640 1 -- 172.21.15.89:0/1468060438 wait complete. 2024-09-15T23:25:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:06 smithi089 ceph-mon[30933]: pgmap v382: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-09-15T23:25:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:06 smithi120 ceph-mon[35250]: pgmap v382: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-09-15T23:25:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:06 smithi174 ceph-mon[35586]: pgmap v382: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-09-15T23:25:07.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.370+0000 7f45fc711640 1 -- 172.21.15.89:0/1316018956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45f41016f0 msgr2=0x7f45f4101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:07.373 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.370+0000 7f45fc711640 1 --2- 172.21.15.89:0/1316018956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45f41016f0 0x7f45f4101ba0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009ca0 tx=0x7f45ec02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.370+0000 7f45fc711640 1 -- 172.21.15.89:0/1316018956 shutdown_connections 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.370+0000 7f45fc711640 1 --2- 172.21.15.89:0/1316018956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45f41016f0 0x7f45f4101ba0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.370+0000 7f45fc711640 1 --2- 172.21.15.89:0/1316018956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f4100d50 0x7f45f41011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.370+0000 7f45fc711640 1 --2- 172.21.15.89:0/1316018956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f4100410 0x7f45f4100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.370+0000 7f45fc711640 1 -- 172.21.15.89:0/1316018956 >> 172.21.15.89:0/1316018956 conn(0x7f45f40fc1e0 msgr2=0x7f45f40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45fc711640 1 -- 172.21.15.89:0/1316018956 shutdown_connections 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45fc711640 1 -- 172.21.15.89:0/1316018956 wait complete. 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45fc711640 1 Processor -- start 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45fc711640 1 -- start start 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45fc711640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f4100410 0x7f45f4199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45fc711640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f4100d50 0x7f45f419a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45fc711640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45f41016f0 0x7f45f41a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45fc711640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f45f410f950 con 0x7f45f41016f0 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45fc711640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45f410f7d0 con 0x7f45f4100d50 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45fc711640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45f410fad0 con 0x7f45f4100410 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45facae640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45f41016f0 0x7f45f41a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45facae640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45f41016f0 0x7f45f41a14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44296/0 (socket says 172.21.15.89:44296) 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.371+0000 7f45facae640 1 -- 172.21.15.89:0/3665965128 learned_addr learned my addr 172.21.15.89:0/3665965128 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45f9cac640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f4100d50 0x7f45f419a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f4100410 0x7f45f4199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45facae640 1 -- 172.21.15.89:0/3665965128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f4100410 msgr2=0x7f45f4199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45facae640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f4100410 0x7f45f4199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45facae640 1 -- 172.21.15.89:0/3665965128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f4100d50 msgr2=0x7f45f419a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45facae640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f4100d50 0x7f45f419a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45facae640 1 -- 172.21.15.89:0/3665965128 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f41a1b40 con 0x7f45f41016f0 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45facae640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45f41016f0 0x7f45f41a14d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f45ec031a40 tx=0x7f45ec031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45eb7fe640 1 -- 172.21.15.89:0/3665965128 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ec003c30 con 0x7f45f41016f0 2024-09-15T23:25:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45eb7fe640 1 -- 172.21.15.89:0/3665965128 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45ec003dd0 con 0x7f45f41016f0 2024-09-15T23:25:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45fc711640 1 -- 172.21.15.89:0/3665965128 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f41a1dd0 con 0x7f45f41016f0 2024-09-15T23:25:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45eb7fe640 1 -- 172.21.15.89:0/3665965128 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ec004690 con 0x7f45f41016f0 2024-09-15T23:25:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.372+0000 7f45fc711640 1 -- 172.21.15.89:0/3665965128 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f41a2290 con 0x7f45f41016f0 2024-09-15T23:25:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.374+0000 7f45eb7fe640 1 -- 172.21.15.89:0/3665965128 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f45ec02f710 con 0x7f45f41016f0 2024-09-15T23:25:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45ff4a8740 1 Processor -- start 2024-09-15T23:25:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45ff4a8740 1 -- start start 2024-09-15T23:25:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6f99a0 0x5621fb6f9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb6feda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6ff370 0x5621fb701830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5621fb65d2f0 con 0x5621fb6fa2e0 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5621fb605270 con 0x5621fb6ff370 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5621fb6592b0 con 0x5621fb6f99a0 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45eb7fe640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45e00a3390 0x7f45e00a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45fa4ad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb6feda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45fa4ad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb6feda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44306/0 (socket says 172.21.15.89:44306) 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45fa4ad640 1 -- 172.21.15.89:0/896149740 learned_addr learned my addr 172.21.15.89:0/896149740 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.375+0000 7f45f9cac640 1 --2- 172.21.15.89:0/896149740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6f99a0 0x5621fb6f9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45eb7fe640 1 -- 172.21.15.89:0/3665965128 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f45ec0f1170 con 0x7f45f41016f0 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45f9cac640 1 --2- 172.21.15.89:0/896149740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6ff370 0x5621fb701830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45facae640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45e00a3390 0x7f45e00a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45fa4ad640 1 -- 172.21.15.89:0/896149740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6f99a0 msgr2=0x5621fb6f9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/896149740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6f99a0 0x5621fb6f9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45fa4ad640 1 -- 172.21.15.89:0/896149740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6ff370 msgr2=0x5621fb701830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/896149740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6ff370 0x5621fb701830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45fa4ad640 1 -- 172.21.15.89:0/896149740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621fb604340 con 0x5621fb6fa2e0 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/896149740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb6feda0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f45f000bcf0 tx=0x7f45f000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc9e1cfa0ea1be7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45facae640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45e00a3390 0x7f45e00a5850 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f45ec0c5a90 tx=0x7f45ec005c30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45e8ff9640 1 -- 172.21.15.89:0/896149740 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0015070 con 0x5621fb6fa2e0 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45e8ff9640 1 -- 172.21.15.89:0/896149740 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45f001a940 con 0x5621fb6fa2e0 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.376+0000 7f45e8ff9640 1 -- 172.21.15.89:0/896149740 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0011a30 con 0x5621fb6fa2e0 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.377+0000 7f45ff4a8740 1 -- 172.21.15.89:0/896149740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 msgr2=0x5621fb6feda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.377+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/896149740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb6feda0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f45f000bcf0 tx=0x7f45f000f730 comp rx=0 tx=0).stop 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.377+0000 7f45ff4a8740 1 -- 172.21.15.89:0/896149740 shutdown_connections 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.377+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/896149740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6ff370 0x5621fb701830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.377+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/896149740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb6feda0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.377+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/896149740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6f99a0 0x5621fb6f9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.377+0000 7f45ff4a8740 1 -- 172.21.15.89:0/896149740 >> 172.21.15.89:0/896149740 conn(0x5621fb6f5010 msgr2=0x5621fb6f5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.377+0000 7f45ff4a8740 1 -- 172.21.15.89:0/896149740 shutdown_connections 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.377+0000 7f45ff4a8740 1 -- 172.21.15.89:0/896149740 wait complete. 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.377+0000 7f45ff4a8740 1 -- start start 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45ff4a8740 1 Processor -- start 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6f99a0 0x5621fb703bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6fa2e0 0x5621fb704100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6ff370 0x5621fb70b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5621fb702340 con 0x5621fb6ff370 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5621fb5e6290 con 0x5621fb6f99a0 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5621fb7024c0 con 0x5621fb6fa2e0 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45ff4a8740 10 client.0 _ll_register_callbacks cb 0x5621fb617140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45f9cac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6ff370 0x5621fb70b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45fa4ad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6fa2e0 0x5621fb704100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45f9cac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6ff370 0x5621fb70b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44322/0 (socket says 172.21.15.89:44322) 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45f9cac640 1 -- 172.21.15.89:0/1163865089 learned_addr learned my addr 172.21.15.89:0/1163865089 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45f9cac640 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6f99a0 0x5621fb703bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45f9cac640 1 -- 172.21.15.89:0/1163865089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6fa2e0 msgr2=0x5621fb704100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45f9cac640 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6fa2e0 0x5621fb704100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45f9cac640 1 -- 172.21.15.89:0/1163865089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6f99a0 msgr2=0x5621fb703bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45f9cac640 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6f99a0 0x5621fb703bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45f9cac640 1 -- 172.21.15.89:0/1163865089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621fb370210 con 0x5621fb6ff370 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6fa2e0 0x5621fb704100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45f9cac640 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6ff370 0x5621fb70b140 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x5621fb7030f0 tx=0x7f45e400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad792ae725fd5ae7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.378+0000 7f45ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ddffb640 1 -- 172.21.15.89:0/1163865089 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e401c040 con 0x5621fb6ff370 2024-09-15T23:25:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1163865089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45f001a940 con 0x5621fb6ff370 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ddffb640 1 -- 172.21.15.89:0/1163865089 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f45e4004230 con 0x5621fb6ff370 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ddffb640 1 -- 172.21.15.89:0/1163865089 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f45e4028050 con 0x5621fb6ff370 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ff4a8740 10 client.15648 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1163865089 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5621fb43c620 con 0x5621fb6ff370 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ddffb640 1 -- 172.21.15.89:0/1163865089 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45f001a940 con 0x5621fb6ff370 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ddffb640 1 -- 172.21.15.89:0/1163865089 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f45e4011450 con 0x5621fb6ff370 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ff4a8740 10 client.15648 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ff4a8740 10 client.15648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ff4a8740 20 client.15648 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621fb70f840 0x5621fb711c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ff4a8740 4 client.15648 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.379+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1163865089 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621fb66aa20 con 0x5621fb70f840 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.380+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621fb70f840 0x5621fb711c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.380+0000 7f45ddffb640 10 client.15648 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:07.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.380+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621fb70f840 0x5621fb711c70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf42617fc1178060 server_cookie=228bc26200defd10 in_seq=0 out_seq=0 2024-09-15T23:25:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ddffb640 1 -- 172.21.15.89:0/1163865089 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5621fb66aa20 con 0x5621fb70f840 2024-09-15T23:25:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ddffb640 10 client.15648 handle_command_reply: tid=0 2024-09-15T23:25:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 client.15648 shutdown 2024-09-15T23:25:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1163865089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6ff370 msgr2=0x5621fb70b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6ff370 0x5621fb70b140 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x5621fb7030f0 tx=0x7f45e400c7e0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1163865089 shutdown_connections 2024-09-15T23:25:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621fb70f840 0x5621fb711c70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6ff370 0x5621fb70b140 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ddffb640 0 client.15648 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:07.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6fa2e0 0x5621fb704100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/1163865089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6f99a0 0x5621fb703bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1163865089 >> 172.21.15.89:0/1163865089 conn(0x5621fb6f5010 msgr2=0x5621fb6f6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:07.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1163865089 shutdown_connections 2024-09-15T23:25:07.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1163865089 wait complete. 2024-09-15T23:25:07.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.381+0000 7f45ff4a8740 20 client.15648 trim_cache size 0 max 16384 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45ff4a8740 1 Processor -- start 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45ff4a8740 1 -- start start 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6f99a0 0x5621fb6f40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6fa2e0 0x5621fb70b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6ff370 0x5621fb716570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f45f41ac9a0 con 0x5621fb6f99a0 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45f41ae4e0 con 0x5621fb6fa2e0 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45f41ad070 con 0x5621fb6ff370 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45f9cac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6f99a0 0x5621fb6f40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45f9cac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6f99a0 0x5621fb6f40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44334/0 (socket says 172.21.15.89:44334) 2024-09-15T23:25:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45f9cac640 1 -- 172.21.15.89:0/1594564239 learned_addr learned my addr 172.21.15.89:0/1594564239 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45f9cac640 1 --2- 172.21.15.89:0/1594564239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6ff370 0x5621fb716570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/1594564239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6fa2e0 0x5621fb70b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45f9cac640 1 -- 172.21.15.89:0/1594564239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6ff370 msgr2=0x5621fb716570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45f9cac640 1 --2- 172.21.15.89:0/1594564239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6ff370 0x5621fb716570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45f9cac640 1 -- 172.21.15.89:0/1594564239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6fa2e0 msgr2=0x5621fb70b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45f9cac640 1 --2- 172.21.15.89:0/1594564239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6fa2e0 0x5621fb70b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45f9cac640 1 -- 172.21.15.89:0/1594564239 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621fb6f5ca0 con 0x5621fb6f99a0 2024-09-15T23:25:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/1594564239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6fa2e0 0x5621fb70b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.396+0000 7f45f9cac640 1 --2- 172.21.15.89:0/1594564239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6f99a0 0x5621fb6f40b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f45e400fca0 tx=0x7f45e400c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fce83a87f11fc8b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:07.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45dffff640 1 -- 172.21.15.89:0/1594564239 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4011650 con 0x5621fb6f99a0 2024-09-15T23:25:07.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45dffff640 1 -- 172.21.15.89:0/1594564239 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45e4025020 con 0x5621fb6f99a0 2024-09-15T23:25:07.400 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45dffff640 1 -- 172.21.15.89:0/1594564239 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e4021d10 con 0x5621fb6f99a0 2024-09-15T23:25:07.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1594564239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6f99a0 msgr2=0x5621fb6f40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:07.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/1594564239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6f99a0 0x5621fb6f40b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f45e400fca0 tx=0x7f45e400c3b0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1594564239 shutdown_connections 2024-09-15T23:25:07.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/1594564239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6ff370 0x5621fb716570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.403 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/1594564239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6fa2e0 0x5621fb70b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/1594564239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6f99a0 0x5621fb6f40b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1594564239 >> 172.21.15.89:0/1594564239 conn(0x5621fb6f5010 msgr2=0x5621fb6f39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1594564239 shutdown_connections 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.397+0000 7f45ff4a8740 1 -- 172.21.15.89:0/1594564239 wait complete. 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45ff4a8740 1 -- start start 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45ff4a8740 1 Processor -- start 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6f99a0 0x5621fb708800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb708d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45ff4a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6ff370 0x5621fb709d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f45f41ade10 con 0x5621fb6fa2e0 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45f41ad740 con 0x5621fb6ff370 2024-09-15T23:25:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45ff4a8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45f41aa980 con 0x5621fb6f99a0 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45ff4a8740 10 client.0 _ll_register_callbacks cb 0x7f45e4013980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45fa4ad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb708d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45fa4ad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb708d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44344/0 (socket says 172.21.15.89:44344) 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.398+0000 7f45fa4ad640 1 -- 172.21.15.89:0/2226169672 learned_addr learned my addr 172.21.15.89:0/2226169672 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45f9cac640 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6ff370 0x5621fb709d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45fa4ad640 1 -- 172.21.15.89:0/2226169672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6f99a0 msgr2=0x5621fb708800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6f99a0 0x5621fb708800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45fa4ad640 1 -- 172.21.15.89:0/2226169672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6ff370 msgr2=0x5621fb709d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6ff370 0x5621fb709d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45fa4ad640 1 -- 172.21.15.89:0/2226169672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621fb70a360 con 0x5621fb6fa2e0 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45fa4ad640 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb708d40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f45f000bcf0 tx=0x7f45f000b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7d22ed170ba2aed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45ddffb640 1 -- 172.21.15.89:0/2226169672 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0015040 con 0x5621fb6fa2e0 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45ddffb640 1 -- 172.21.15.89:0/2226169672 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f45f0005450 con 0x5621fb6fa2e0 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45ff4a8740 1 -- 172.21.15.89:0/2226169672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45e4025020 con 0x5621fb6fa2e0 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45ddffb640 1 -- 172.21.15.89:0/2226169672 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f45f0034050 con 0x5621fb6fa2e0 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45ff4a8740 10 client.15654 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45ff4a8740 1 -- 172.21.15.89:0/2226169672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5621fb77a550 con 0x5621fb6fa2e0 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45ddffb640 1 -- 172.21.15.89:0/2226169672 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45e4025020 con 0x5621fb6fa2e0 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.399+0000 7f45ddffb640 1 -- 172.21.15.89:0/2226169672 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f45f001a620 con 0x5621fb6fa2e0 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.400+0000 7f45ff4a8740 10 client.15654 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.400+0000 7f45ff4a8740 10 client.15654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.400+0000 7f45ff4a8740 20 client.15654 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.400+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621fb77ab00 0x5621fb77cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.400+0000 7f45ff4a8740 4 client.15654 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:25:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.400+0000 7f45ff4a8740 1 -- 172.21.15.89:0/2226169672 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621fb664d70 con 0x5621fb77ab00 2024-09-15T23:25:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.400+0000 7f45f9cac640 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621fb77ab00 0x5621fb77cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.400+0000 7f45f9cac640 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621fb77ab00 0x5621fb77cef0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9e6ce17d830a6d1 server_cookie=9df1ae26d3fcb9d4 in_seq=0 out_seq=0 2024-09-15T23:25:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.400+0000 7f45ddffb640 10 client.15654 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:07.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ddffb640 1 -- 172.21.15.89:0/2226169672 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621fb664d70 con 0x5621fb77ab00 2024-09-15T23:25:07.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ddffb640 10 client.15654 handle_command_reply: tid=0 2024-09-15T23:25:07.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 client.15654 shutdown 2024-09-15T23:25:07.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 -- 172.21.15.89:0/2226169672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 msgr2=0x5621fb708d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:07.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb708d40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f45f000bcf0 tx=0x7f45f000b7a0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 -- 172.21.15.89:0/2226169672 shutdown_connections 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621fb77ab00 0x5621fb77cef0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621fb6ff370 0x5621fb709d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ddffb640 0 client.15654 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621fb6fa2e0 0x5621fb708d40 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 --2- 172.21.15.89:0/2226169672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621fb6f99a0 0x5621fb708800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 -- 172.21.15.89:0/2226169672 >> 172.21.15.89:0/2226169672 conn(0x5621fb6f5010 msgr2=0x5621fb6f39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 -- 172.21.15.89:0/2226169672 shutdown_connections 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 1 -- 172.21.15.89:0/2226169672 wait complete. 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.406+0000 7f45ff4a8740 20 client.15654 trim_cache size 0 max 16384 2024-09-15T23:25:07.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:25:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:25:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:25:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:25:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 -- 172.21.15.89:0/3665965128 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45e00a3390 msgr2=0x7f45e00a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:07.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45e00a3390 0x7f45e00a5850 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f45ec0c5a90 tx=0x7f45ec005c30 comp rx=0 tx=0).stop 2024-09-15T23:25:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 -- 172.21.15.89:0/3665965128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45f41016f0 msgr2=0x7f45f41a14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45f41016f0 0x7f45f41a14d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f45ec031a40 tx=0x7f45ec031ee0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 -- 172.21.15.89:0/3665965128 shutdown_connections 2024-09-15T23:25:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f45e00a3390 0x7f45e00a5850 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f45f41016f0 0x7f45f41a14d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45f4100d50 0x7f45f419a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 --2- 172.21.15.89:0/3665965128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45f4100410 0x7f45f4199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 -- 172.21.15.89:0/3665965128 >> 172.21.15.89:0/3665965128 conn(0x7f45f40fc1e0 msgr2=0x7f45f40fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 -- 172.21.15.89:0/3665965128 shutdown_connections 2024-09-15T23:25:07.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:07.407+0000 7f45fc711640 1 -- 172.21.15.89:0/3665965128 wait complete. 2024-09-15T23:25:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:08 smithi120 ceph-mon[35250]: pgmap v383: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-09-15T23:25:08.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:08 smithi174 ceph-mon[35586]: pgmap v383: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-09-15T23:25:08.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:08 smithi089 ceph-mon[30933]: pgmap v383: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-09-15T23:25:09.781 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:25:09.858 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:10.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:10 smithi120 ceph-mon[35250]: pgmap v384: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-15T23:25:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:10 smithi174 ceph-mon[35586]: pgmap v384: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-15T23:25:10.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:10 smithi089 ceph-mon[30933]: pgmap v384: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-15T23:25:11.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.970+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2134454589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fcc100e20 msgr2=0x7f1fcc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:11.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.970+0000 7f1fd1bce640 1 --2- 172.21.15.89:0/2134454589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fcc100e20 0x7f1fcc101200 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009900 tx=0x7f1fbc02a480 comp rx=0 tx=0).stop 2024-09-15T23:25:11.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.970+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2134454589 shutdown_connections 2024-09-15T23:25:11.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.970+0000 7f1fd1bce640 1 --2- 172.21.15.89:0/2134454589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fcc102ef0 0x7f1fcc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:11.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.970+0000 7f1fd1bce640 1 --2- 172.21.15.89:0/2134454589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1fcc102570 0x7f1fcc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:11.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.970+0000 7f1fd1bce640 1 --2- 172.21.15.89:0/2134454589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fcc100e20 0x7f1fcc101200 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:11.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.970+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2134454589 >> 172.21.15.89:0/2134454589 conn(0x7f1fcc0fc7d0 msgr2=0x7f1fcc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:11.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.970+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2134454589 shutdown_connections 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.970+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2134454589 wait complete. 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fd1bce640 1 Processor -- start 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fd1bce640 1 -- start start 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fd1bce640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1fcc100e20 0x7f1fcc103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fd1bce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fcc102570 0x7f1fcc103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fd1bce640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fcc102ef0 0x7f1fcc1041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fd1bce640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc110f80 con 0x7f1fcc102ef0 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fd1bce640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc110e00 con 0x7f1fcc102570 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fd1bce640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc111100 con 0x7f1fcc100e20 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fcc102ef0 0x7f1fcc1041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fcbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fcc102ef0 0x7f1fcc1041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44368/0 (socket says 172.21.15.89:44368) 2024-09-15T23:25:11.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fcbfff640 1 -- 172.21.15.89:0/2502239917 learned_addr learned my addr 172.21.15.89:0/2502239917 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.971+0000 7f1fcaffd640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fcc102570 0x7f1fcc103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fcb7fe640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1fcc100e20 0x7f1fcc103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fcbfff640 1 -- 172.21.15.89:0/2502239917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1fcc100e20 msgr2=0x7f1fcc103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fcbfff640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1fcc100e20 0x7f1fcc103750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fcbfff640 1 -- 172.21.15.89:0/2502239917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fcc102570 msgr2=0x7f1fcc103c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fcbfff640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fcc102570 0x7f1fcc103c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fcbfff640 1 -- 172.21.15.89:0/2502239917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fcc1a6a30 con 0x7f1fcc102ef0 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fcaffd640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fcc102570 0x7f1fcc103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fcbfff640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fcc102ef0 0x7f1fcc1041d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009fd0 tx=0x7f1fc400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fc8ff9640 1 -- 172.21.15.89:0/2502239917 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4017070 con 0x7f1fcc102ef0 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2502239917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fcc1a6d20 con 0x7f1fcc102ef0 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fc8ff9640 1 -- 172.21.15.89:0/2502239917 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fc4003e50 con 0x7f1fcc102ef0 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fc8ff9640 1 -- 172.21.15.89:0/2502239917 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc40106c0 con 0x7f1fcc102ef0 2024-09-15T23:25:11.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.972+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2502239917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fcc1a72b0 con 0x7f1fcc102ef0 2024-09-15T23:25:11.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.974+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2502239917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fcc101200 con 0x7f1fcc102ef0 2024-09-15T23:25:11.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.976+0000 7f1fc8ff9640 1 -- 172.21.15.89:0/2502239917 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f1fc4003970 con 0x7f1fcc102ef0 2024-09-15T23:25:11.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.976+0000 7f1fc8ff9640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1fb40a32d0 0x7f1fb40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:11.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.977+0000 7f1fc8ff9640 1 -- 172.21.15.89:0/2502239917 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1fc40cfe20 con 0x7f1fcc102ef0 2024-09-15T23:25:11.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.977+0000 7f1fcb7fe640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1fb40a32d0 0x7f1fb40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:11.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.978+0000 7f1fcb7fe640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1fb40a32d0 0x7f1fb40a5790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009230 tx=0x7f1fbc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:11.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:11.980+0000 7f1fc8ff9640 1 -- 172.21.15.89:0/2502239917 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1fc409bfa0 con 0x7f1fcc102ef0 2024-09-15T23:25:12.022 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.288+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2502239917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1fcc1a74e0 con 0x7f1fcc102ef0 2024-09-15T23:25:12.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.289+0000 7f1fc8ff9640 1 -- 172.21.15.89:0/2502239917 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f1fc40a02a0 con 0x7f1fcc102ef0 2024-09-15T23:25:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:25:12.290 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:25:12.290 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:25:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2502239917 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1fb40a32d0 msgr2=0x7f1fb40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1fb40a32d0 0x7f1fb40a5790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009230 tx=0x7f1fbc0058d0 comp rx=0 tx=0).stop 2024-09-15T23:25:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2502239917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fcc102ef0 msgr2=0x7f1fcc1041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fcc102ef0 0x7f1fcc1041d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009fd0 tx=0x7f1fc400b260 comp rx=0 tx=0).stop 2024-09-15T23:25:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2502239917 shutdown_connections 2024-09-15T23:25:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1fb40a32d0 0x7f1fb40a5790 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fcc102ef0 0x7f1fcc1041d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1fcc102570 0x7f1fcc103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:12.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 --2- 172.21.15.89:0/2502239917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1fcc100e20 0x7f1fcc103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2502239917 >> 172.21.15.89:0/2502239917 conn(0x7f1fcc0fc7d0 msgr2=0x7f1fcc06a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2502239917 shutdown_connections 2024-09-15T23:25:12.293 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:12.291+0000 7f1fd1bce640 1 -- 172.21.15.89:0/2502239917 wait complete. 2024-09-15T23:25:12.354 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:12 smithi089 ceph-mon[30933]: pgmap v385: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T23:25:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:12 smithi120 ceph-mon[35250]: pgmap v385: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T23:25:12.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:12 smithi174 ceph-mon[35586]: pgmap v385: 129 pgs: 129 active+clean; 668 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T23:25:13.078 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 150.2 --await-for 45.0' 2024-09-15T23:25:13.078 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 150.2 --await-for 45.0 2024-09-15T23:25:13.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.299+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2201136151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd134105f30 msgr2=0x7fd1341063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.299+0000 7fd13c7ac640 1 --2- 172.21.15.89:0/2201136151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd134105f30 0x7fd1341063c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009ca0 tx=0x7fd12c02a940 comp rx=0 tx=0).stop 2024-09-15T23:25:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.300+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2201136151 shutdown_connections 2024-09-15T23:25:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.300+0000 7fd13c7ac640 1 --2- 172.21.15.89:0/2201136151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd134105f30 0x7fd1341063c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.300+0000 7fd13c7ac640 1 --2- 172.21.15.89:0/2201136151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd134101440 0x7fd134105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.300+0000 7fd13c7ac640 1 --2- 172.21.15.89:0/2201136151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd134100b20 0x7fd134100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.300+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2201136151 >> 172.21.15.89:0/2201136151 conn(0x7fd1340fc790 msgr2=0x7fd1340febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.301+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2201136151 shutdown_connections 2024-09-15T23:25:13.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.301+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2201136151 wait complete. 2024-09-15T23:25:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.301+0000 7fd13c7ac640 1 Processor -- start 2024-09-15T23:25:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13c7ac640 1 -- start start 2024-09-15T23:25:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13c7ac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd134100b20 0x7fd13406d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13c7ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd134101440 0x7fd13406dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13c7ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd134105f30 0x7fd1340721a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13c7ac640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd13406c220 con 0x7fd134101440 2024-09-15T23:25:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13c7ac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd13406c0a0 con 0x7fd134100b20 2024-09-15T23:25:13.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13c7ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd13406c3a0 con 0x7fd134105f30 2024-09-15T23:25:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13a548640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd134100b20 0x7fd13406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd139d47640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd134101440 0x7fd13406dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13a548640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd134100b20 0x7fd13406d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:35814/0 (socket says 172.21.15.89:35814) 2024-09-15T23:25:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13a548640 1 -- 172.21.15.89:0/2276933248 learned_addr learned my addr 172.21.15.89:0/2276933248 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.302+0000 7fd13ad49640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd134105f30 0x7fd1340721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.303+0000 7fd13a548640 1 -- 172.21.15.89:0/2276933248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd134105f30 msgr2=0x7fd1340721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.303+0000 7fd13a548640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd134105f30 0x7fd1340721a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.303+0000 7fd13a548640 1 -- 172.21.15.89:0/2276933248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd134101440 msgr2=0x7fd13406dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.303+0000 7fd13a548640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd134101440 0x7fd13406dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.303+0000 7fd13a548640 1 -- 172.21.15.89:0/2276933248 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1340728d0 con 0x7fd134100b20 2024-09-15T23:25:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.303+0000 7fd139d47640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd134101440 0x7fd13406dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.303+0000 7fd13a548640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd134100b20 0x7fd13406d860 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd13000c920 tx=0x7fd13000cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.303+0000 7fd12b7fe640 1 -- 172.21.15.89:0/2276933248 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd130007c70 con 0x7fd134100b20 2024-09-15T23:25:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.304+0000 7fd12b7fe640 1 -- 172.21.15.89:0/2276933248 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd130007e10 con 0x7fd134100b20 2024-09-15T23:25:13.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.304+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2276933248 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd134072bc0 con 0x7fd134100b20 2024-09-15T23:25:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.304+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2276933248 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1341b3d50 con 0x7fd134100b20 2024-09-15T23:25:13.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.304+0000 7fd12b7fe640 1 -- 172.21.15.89:0/2276933248 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13000e860 con 0x7fd134100b20 2024-09-15T23:25:13.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.307+0000 7fd12b7fe640 1 -- 172.21.15.89:0/2276933248 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fd13000eae0 con 0x7fd134100b20 2024-09-15T23:25:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.308+0000 7fd13eb85740 1 Processor -- start 2024-09-15T23:25:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.308+0000 7fd13eb85740 1 -- start start 2024-09-15T23:25:13.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.309+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e79913c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.309+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e79913ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.309+0000 7fd13a548640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e79913ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.309+0000 7fd13ad49640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e79913c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.309+0000 7fd13ad49640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e79913c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:47822/0 (socket says 172.21.15.89:47822) 2024-09-15T23:25:13.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.309+0000 7fd13a548640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e79913ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44390/0 (socket says 172.21.15.89:44390) 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.309+0000 7fd12b7fe640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1200a3390 0x7fd1200a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.309+0000 7fd12b7fe640 1 -- 172.21.15.89:0/2276933248 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd1300cfe90 con 0x7fd134100b20 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799147370 0x55e799149760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13ad49640 1 -- 172.21.15.89:0/2436403256 learned_addr learned my addr 172.21.15.89:0/2436403256 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13eb85740 1 -- 172.21.15.89:0/2436403256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e7990a22f0 con 0x55e79913c5d0 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13eb85740 1 -- 172.21.15.89:0/2436403256 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e799049cf0 con 0x55e799147370 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13eb85740 1 -- 172.21.15.89:0/2436403256 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e79909e2b0 con 0x55e79913db40 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd139d47640 1 --2- 172.21.15.89:0/2436403256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799147370 0x55e799149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13ad49640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1200a3390 0x7fd1200a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13a548640 1 -- 172.21.15.89:0/2436403256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 msgr2=0x55e79913c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13a548640 1 --2- 172.21.15.89:0/2436403256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e79913c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13a548640 1 -- 172.21.15.89:0/2436403256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799147370 msgr2=0x55e799149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13a548640 1 --2- 172.21.15.89:0/2436403256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799147370 0x55e799149760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.310+0000 7fd13a548640 1 -- 172.21.15.89:0/2436403256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e79902e3a0 con 0x55e79913c5d0 2024-09-15T23:25:13.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.311+0000 7fd139d47640 1 --2- 172.21.15.89:0/2436403256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799147370 0x55e799149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:13.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.311+0000 7fd13ad49640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1200a3390 0x7fd1200a5850 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd12c02fcc0 tx=0x7fd12c02a940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.311+0000 7fd13a548640 1 --2- 172.21.15.89:0/2436403256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e79913ca10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd1300cd450 tx=0x7fd13009bbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d0b2a6e7c2e26d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.311+0000 7fd128ff9640 1 -- 172.21.15.89:0/2436403256 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1300a0310 con 0x55e79913c5d0 2024-09-15T23:25:13.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.311+0000 7fd128ff9640 1 -- 172.21.15.89:0/2436403256 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1300a4bd0 con 0x55e79913c5d0 2024-09-15T23:25:13.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.311+0000 7fd128ff9640 1 -- 172.21.15.89:0/2436403256 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1300bae00 con 0x55e79913c5d0 2024-09-15T23:25:13.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.312+0000 7fd13eb85740 1 -- 172.21.15.89:0/2436403256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 msgr2=0x55e79913ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.312+0000 7fd13eb85740 1 --2- 172.21.15.89:0/2436403256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e79913ca10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd1300cd450 tx=0x7fd13009bbc0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.312+0000 7fd13eb85740 1 -- 172.21.15.89:0/2436403256 shutdown_connections 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.312+0000 7fd13eb85740 1 --2- 172.21.15.89:0/2436403256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799147370 0x55e799149760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.312+0000 7fd13eb85740 1 --2- 172.21.15.89:0/2436403256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e79913ca10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.312+0000 7fd13eb85740 1 --2- 172.21.15.89:0/2436403256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e79913c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.312+0000 7fd13eb85740 1 -- 172.21.15.89:0/2436403256 >> 172.21.15.89:0/2436403256 conn(0x55e799105310 msgr2=0x55e799138c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.312+0000 7fd13eb85740 1 -- 172.21.15.89:0/2436403256 shutdown_connections 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.312+0000 7fd13eb85740 1 -- 172.21.15.89:0/2436403256 wait complete. 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.313+0000 7fd13eb85740 1 -- start start 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.313+0000 7fd13eb85740 1 Processor -- start 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.313+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e799146d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.313+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e79913db40 0x55e799144040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.313+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 0x55e799144650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.313+0000 7fd13eb85740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e79914a070 con 0x55e79913c5d0 2024-09-15T23:25:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.313+0000 7fd13a548640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e799146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.313+0000 7fd13a548640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e799146d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44400/0 (socket says 172.21.15.89:44400) 2024-09-15T23:25:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.313+0000 7fd13a548640 1 -- 172.21.15.89:0/482081593 learned_addr learned my addr 172.21.15.89:0/482081593 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13eb85740 1 -- 172.21.15.89:0/482081593 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e79905d060 con 0x55e79913db40 2024-09-15T23:25:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13eb85740 1 -- 172.21.15.89:0/482081593 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e79914a1f0 con 0x55e799147370 2024-09-15T23:25:13.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13eb85740 10 client.0 _ll_register_callbacks cb 0x55e798db08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13ad49640 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 0x55e799144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13a548640 1 -- 172.21.15.89:0/482081593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 msgr2=0x55e799144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13a548640 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 0x55e799144650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13a548640 1 -- 172.21.15.89:0/482081593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e79913db40 msgr2=0x55e799144040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13a548640 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e79913db40 0x55e799144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13a548640 1 -- 172.21.15.89:0/482081593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e798e133c0 con 0x55e79913c5d0 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13ad49640 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 0x55e799144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd11dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.314+0000 7fd13a548640 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e799146d90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd13009bc60 tx=0x7fd1300cd8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3022f7decef37823 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.315+0000 7fd11dffb640 1 -- 172.21.15.89:0/482081593 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1300badb0 con 0x55e79913c5d0 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.315+0000 7fd11dffb640 1 -- 172.21.15.89:0/482081593 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd13009a260 con 0x55e79913c5d0 2024-09-15T23:25:13.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.315+0000 7fd13eb85740 1 -- 172.21.15.89:0/482081593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1300a4bd0 con 0x55e79913c5d0 2024-09-15T23:25:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.315+0000 7fd11dffb640 1 -- 172.21.15.89:0/482081593 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd1300badb0 con 0x55e79913c5d0 2024-09-15T23:25:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.315+0000 7fd13eb85740 10 client.15672 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.315+0000 7fd13eb85740 1 -- 172.21.15.89:0/482081593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e799155b50 con 0x55e79913c5d0 2024-09-15T23:25:13.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.315+0000 7fd11dffb640 1 -- 172.21.15.89:0/482081593 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1300a4bd0 con 0x55e79913c5d0 2024-09-15T23:25:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.316+0000 7fd11dffb640 1 -- 172.21.15.89:0/482081593 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd13009f070 con 0x55e79913c5d0 2024-09-15T23:25:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.316+0000 7fd13eb85740 10 client.15672 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:13.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.316+0000 7fd13eb85740 10 client.15672 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.316+0000 7fd13eb85740 20 client.15672 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.316+0000 7fd13eb85740 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e799156ea0 0x55e799159290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.316+0000 7fd13eb85740 4 client.15672 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.316+0000 7fd13eb85740 1 -- 172.21.15.89:0/482081593 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7990af860 con 0x55e799156ea0 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.316+0000 7fd139d47640 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e799156ea0 0x55e799159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.317+0000 7fd139d47640 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e799156ea0 0x55e799159290 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1e095c4d78f0b75d server_cookie=4e743f3c2a192d in_seq=0 out_seq=0 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.317+0000 7fd11dffb640 10 client.15672 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.317+0000 7fd11dffb640 1 -- 172.21.15.89:0/482081593 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e7990af860 con 0x55e799156ea0 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.317+0000 7fd11dffb640 10 client.15672 handle_command_reply: tid=0 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 client.15672 shutdown 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 -- 172.21.15.89:0/482081593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 msgr2=0x55e799146d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e799146d90 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd13009bc60 tx=0x7fd1300cd8d0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 -- 172.21.15.89:0/482081593 shutdown_connections 2024-09-15T23:25:13.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e799156ea0 0x55e799159290 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd11dffb640 0 client.15672 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:13.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 0x55e799144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e79913db40 0x55e799144040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 --2- 172.21.15.89:0/482081593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e79913c5d0 0x55e799146d90 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 -- 172.21.15.89:0/482081593 >> 172.21.15.89:0/482081593 conn(0x55e799105310 msgr2=0x55e79913e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 -- 172.21.15.89:0/482081593 shutdown_connections 2024-09-15T23:25:13.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 1 -- 172.21.15.89:0/482081593 wait complete. 2024-09-15T23:25:13.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.318+0000 7fd13eb85740 20 client.15672 trim_cache size 0 max 16384 2024-09-15T23:25:13.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13eb85740 1 Processor -- start 2024-09-15T23:25:13.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13eb85740 1 -- start start 2024-09-15T23:25:13.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e79913db40 0x55e799144010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 0x55e79915c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799156ea0 0x55e79915c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13eb85740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd11800fdf0 con 0x55e799156ea0 2024-09-15T23:25:13.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13eb85740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd118011930 con 0x55e79913db40 2024-09-15T23:25:13.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13eb85740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd1180104c0 con 0x55e799147370 2024-09-15T23:25:13.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13ad49640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799156ea0 0x55e79915c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13ad49640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799156ea0 0x55e79915c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44414/0 (socket says 172.21.15.89:44414) 2024-09-15T23:25:13.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.343+0000 7fd13ad49640 1 -- 172.21.15.89:0/3964322574 learned_addr learned my addr 172.21.15.89:0/3964322574 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:13.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.344+0000 7fd139d47640 1 --2- 172.21.15.89:0/3964322574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 0x55e79915c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.344+0000 7fd13a548640 1 --2- 172.21.15.89:0/3964322574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e79913db40 0x55e799144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.344+0000 7fd139d47640 1 -- 172.21.15.89:0/3964322574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e79913db40 msgr2=0x55e799144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.344+0000 7fd139d47640 1 --2- 172.21.15.89:0/3964322574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e79913db40 0x55e799144010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.344+0000 7fd139d47640 1 -- 172.21.15.89:0/3964322574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799156ea0 msgr2=0x55e79915c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.344+0000 7fd139d47640 1 --2- 172.21.15.89:0/3964322574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799156ea0 0x55e79915c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.344+0000 7fd139d47640 1 -- 172.21.15.89:0/3964322574 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e799144980 con 0x55e799147370 2024-09-15T23:25:13.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.344+0000 7fd13ad49640 1 --2- 172.21.15.89:0/3964322574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799156ea0 0x55e79915c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:13.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.344+0000 7fd139d47640 1 --2- 172.21.15.89:0/3964322574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 0x55e79915c2b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55e7991438d0 tx=0x7fd124013930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a85230e336c90476 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.345+0000 7fd11ffff640 1 -- 172.21.15.89:0/3964322574 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd124023030 con 0x55e799147370 2024-09-15T23:25:13.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.345+0000 7fd11ffff640 1 -- 172.21.15.89:0/3964322574 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd124005500 con 0x55e799147370 2024-09-15T23:25:13.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.345+0000 7fd11ffff640 1 -- 172.21.15.89:0/3964322574 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12401f440 con 0x55e799147370 2024-09-15T23:25:13.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.346+0000 7fd13eb85740 1 -- 172.21.15.89:0/3964322574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 msgr2=0x55e79915c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.346+0000 7fd13eb85740 1 --2- 172.21.15.89:0/3964322574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 0x55e79915c2b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55e7991438d0 tx=0x7fd124013930 comp rx=0 tx=0).stop 2024-09-15T23:25:13.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.346+0000 7fd13eb85740 1 -- 172.21.15.89:0/3964322574 shutdown_connections 2024-09-15T23:25:13.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.346+0000 7fd13eb85740 1 --2- 172.21.15.89:0/3964322574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799156ea0 0x55e79915c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.346+0000 7fd13eb85740 1 --2- 172.21.15.89:0/3964322574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e799147370 0x55e79915c2b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.346+0000 7fd13eb85740 1 --2- 172.21.15.89:0/3964322574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e79913db40 0x55e799144010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.346+0000 7fd13eb85740 1 -- 172.21.15.89:0/3964322574 >> 172.21.15.89:0/3964322574 conn(0x55e799105310 msgr2=0x55e799143900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.346+0000 7fd13eb85740 1 -- 172.21.15.89:0/3964322574 shutdown_connections 2024-09-15T23:25:13.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.346+0000 7fd13eb85740 1 -- 172.21.15.89:0/3964322574 wait complete. 2024-09-15T23:25:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.346+0000 7fd13eb85740 1 -- start start 2024-09-15T23:25:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13eb85740 1 Processor -- start 2024-09-15T23:25:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e799159f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799147370 0x55e79915a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13eb85740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799156ea0 0x55e79915a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13eb85740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd118011260 con 0x55e799147370 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13eb85740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd118010b90 con 0x55e799156ea0 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13eb85740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd11800dd70 con 0x55e79913db40 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13eb85740 10 client.0 _ll_register_callbacks cb 0x7fd13009bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13a548640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e799159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13a548640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e799159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:47854/0 (socket says 172.21.15.89:47854) 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13a548640 1 -- 172.21.15.89:0/3747465467 learned_addr learned my addr 172.21.15.89:0/3747465467 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd139d47640 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799147370 0x55e79915a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13ad49640 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799156ea0 0x55e79915a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13a548640 1 -- 172.21.15.89:0/3747465467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799156ea0 msgr2=0x55e79915a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13a548640 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799156ea0 0x55e79915a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13a548640 1 -- 172.21.15.89:0/3747465467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799147370 msgr2=0x55e79915a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.347+0000 7fd13a548640 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799147370 0x55e79915a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.348+0000 7fd13a548640 1 -- 172.21.15.89:0/3747465467 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7991c07e0 con 0x55e79913db40 2024-09-15T23:25:13.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.348+0000 7fd13ad49640 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799156ea0 0x55e79915a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:13.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.348+0000 7fd13a548640 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e799159f70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd130096d80 tx=0x7fd130005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99767117781e2040 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.348+0000 7fd11dffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:25:13.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.348+0000 7fd11dffb640 1 -- 172.21.15.89:0/3747465467 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13007f060 con 0x55e79913db40 2024-09-15T23:25:13.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.348+0000 7fd11dffb640 1 -- 172.21.15.89:0/3747465467 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd13007ab90 con 0x55e79913db40 2024-09-15T23:25:13.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.348+0000 7fd13eb85740 1 -- 172.21.15.89:0/3747465467 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd124005500 con 0x55e79913db40 2024-09-15T23:25:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.349+0000 7fd11dffb640 1 -- 172.21.15.89:0/3747465467 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd13009baa0 con 0x55e79913db40 2024-09-15T23:25:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.349+0000 7fd13eb85740 10 client.25066 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.349+0000 7fd13eb85740 1 -- 172.21.15.89:0/3747465467 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e7991c1a00 con 0x55e79913db40 2024-09-15T23:25:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.349+0000 7fd11dffb640 1 -- 172.21.15.89:0/3747465467 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd124005500 con 0x55e79913db40 2024-09-15T23:25:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.350+0000 7fd11dffb640 1 -- 172.21.15.89:0/3747465467 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd13009f070 con 0x55e79913db40 2024-09-15T23:25:13.352 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:13 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2502239917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.350+0000 7fd13eb85740 10 client.25066 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.350+0000 7fd13eb85740 10 client.25066 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:25:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.350+0000 7fd13eb85740 20 client.25066 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.350+0000 7fd13eb85740 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e7991c1d70 0x55e7991c4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.350+0000 7fd13eb85740 4 client.25066 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.350+0000 7fd13eb85740 1 -- 172.21.15.89:0/3747465467 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7990a9d70 con 0x55e7991c1d70 2024-09-15T23:25:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.350+0000 7fd139d47640 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e7991c1d70 0x55e7991c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.351+0000 7fd139d47640 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e7991c1d70 0x55e7991c4190 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=302c0d3bf7823c0e server_cookie=5b41b8f702a7409a in_seq=0 out_seq=0 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.351+0000 7fd11dffb640 10 client.25066 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.351+0000 7fd11dffb640 1 -- 172.21.15.89:0/3747465467 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7990a9d70 con 0x55e7991c1d70 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.351+0000 7fd11dffb640 10 client.25066 handle_command_reply: tid=0 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.351+0000 7fd13eb85740 1 client.25066 shutdown 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 1 -- 172.21.15.89:0/3747465467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 msgr2=0x55e799159f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e799159f70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd130096d80 tx=0x7fd130005e60 comp rx=0 tx=0).stop 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 1 -- 172.21.15.89:0/3747465467 shutdown_connections 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e7991c1d70 0x55e7991c4190 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e799156ea0 0x55e79915a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e799147370 0x55e79915a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 1 --2- 172.21.15.89:0/3747465467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e79913db40 0x55e799159f70 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd130096d80 tx=0x7fd130005e60 comp rx=0 tx=0).stop 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 1 -- 172.21.15.89:0/3747465467 >> 172.21.15.89:0/3747465467 conn(0x55e799105310 msgr2=0x55e79913e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd11dffb640 0 client.25066 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:13.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 1 -- 172.21.15.89:0/3747465467 shutdown_connections 2024-09-15T23:25:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 1 -- 172.21.15.89:0/3747465467 wait complete. 2024-09-15T23:25:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.352+0000 7fd13eb85740 20 client.25066 trim_cache size 0 max 16384 2024-09-15T23:25:13.356 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2276933248 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1200a3390 msgr2=0x7fd1200a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1200a3390 0x7fd1200a5850 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd12c02fcc0 tx=0x7fd12c02a940 comp rx=0 tx=0).stop 2024-09-15T23:25:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2276933248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd134100b20 msgr2=0x7fd13406d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd134100b20 0x7fd13406d860 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd13000c920 tx=0x7fd13000cde0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2276933248 shutdown_connections 2024-09-15T23:25:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1200a3390 0x7fd1200a5850 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd134105f30 0x7fd1340721a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd134101440 0x7fd13406dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 --2- 172.21.15.89:0/2276933248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd134100b20 0x7fd13406d860 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2276933248 >> 172.21.15.89:0/2276933248 conn(0x7fd1340fc790 msgr2=0x7fd1341038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2276933248 shutdown_connections 2024-09-15T23:25:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:13.354+0000 7fd13c7ac640 1 -- 172.21.15.89:0/2276933248 wait complete. 2024-09-15T23:25:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:13 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2502239917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:13 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2502239917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:13.621 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:14.100 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:25:14.341 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:14 smithi089 ceph-mon[30933]: pgmap v386: 129 pgs: 129 active+clean; 669 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-15T23:25:14.374 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:14 smithi120 ceph-mon[35250]: pgmap v386: 129 pgs: 129 active+clean; 669 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-15T23:25:14.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:14 smithi174 ceph-mon[35586]: pgmap v386: 129 pgs: 129 active+clean; 669 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-09-15T23:25:14.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.568+0000 7f8622719640 1 -- 172.21.15.89:0/3335864642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861c106b80 msgr2=0x7f861c106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.568+0000 7f8622719640 1 --2- 172.21.15.89:0/3335864642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861c106b80 0x7f861c106f60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f860c009900 tx=0x7f860c02a410 comp rx=0 tx=0).stop 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.568+0000 7f8622719640 1 -- 172.21.15.89:0/3335864642 shutdown_connections 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.568+0000 7f8622719640 1 --2- 172.21.15.89:0/3335864642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f861c100aa0 0x7f861c10ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.568+0000 7f8622719640 1 --2- 172.21.15.89:0/3335864642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f861c100100 0x7f861c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.568+0000 7f8622719640 1 --2- 172.21.15.89:0/3335864642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861c106b80 0x7f861c106f60 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.568+0000 7f8622719640 1 -- 172.21.15.89:0/3335864642 >> 172.21.15.89:0/3335864642 conn(0x7f861c077230 msgr2=0x7f861c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.568+0000 7f8622719640 1 -- 172.21.15.89:0/3335864642 shutdown_connections 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.568+0000 7f8622719640 1 -- 172.21.15.89:0/3335864642 wait complete. 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.569+0000 7f8622719640 1 Processor -- start 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.569+0000 7f8622719640 1 -- start start 2024-09-15T23:25:14.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.569+0000 7f8622719640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861c100100 0x7f861c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.569+0000 7f8622719640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f861c100aa0 0x7f861c198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.569+0000 7f8622719640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f861c106b80 0x7f861c19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.569+0000 7f861bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861c100100 0x7f861c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.569+0000 7f861bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861c100100 0x7f861c198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44444/0 (socket says 172.21.15.89:44444) 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f861bfff640 1 -- 172.21.15.89:0/1277769698 learned_addr learned my addr 172.21.15.89:0/1277769698 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f861c112010 con 0x7f861c100100 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f861c111e90 con 0x7f861c100aa0 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f861c112190 con 0x7f861c106b80 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f861b7fe640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f861c100aa0 0x7f861c198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f861bfff640 1 -- 172.21.15.89:0/1277769698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f861c106b80 msgr2=0x7f861c19ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f861bfff640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f861c106b80 0x7f861c19ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f861bfff640 1 -- 172.21.15.89:0/1277769698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f861c100aa0 msgr2=0x7f861c198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:14.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f861bfff640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f861c100aa0 0x7f861c198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f861bfff640 1 -- 172.21.15.89:0/1277769698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861c19d520 con 0x7f861c100100 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f861bfff640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861c100100 0x7f861c198570 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f860c006010 tx=0x7f860c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f86197fa640 1 -- 172.21.15.89:0/1277769698 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c03f030 con 0x7f861c100100 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f86197fa640 1 -- 172.21.15.89:0/1277769698 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f860c003ce0 con 0x7f861c100100 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.570+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861c19d7b0 con 0x7f861c100100 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.571+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861c101210 con 0x7f861c100100 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.571+0000 7f86197fa640 1 -- 172.21.15.89:0/1277769698 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c035070 con 0x7f861c100100 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.573+0000 7f86197fa640 1 -- 172.21.15.89:0/1277769698 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f860c042030 con 0x7f861c100100 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f86197fa640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86040a32d0 0x7f86040a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f861b7fe640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86040a32d0 0x7f86040a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f86197fa640 1 -- 172.21.15.89:0/1277769698 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f860c0f04c0 con 0x7f861c100100 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f8624af2740 1 Processor -- start 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f8624af2740 1 -- start start 2024-09-15T23:25:14.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f8624af2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160680 0x55db71160a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f8624af2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160fa0 0x55db71173c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f8624af2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71174150 0x55db71176540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f8624af2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55db710c62a0 con 0x55db71160fa0 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f8624af2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55db7106db10 con 0x55db71160680 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f8624af2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55db710c2260 con 0x55db71174150 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.574+0000 7f861b7fe640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86040a32d0 0x7f86040a5790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f861c0ff810 tx=0x7f861000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160fa0 0x55db71173c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160680 0x55db71160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160680 0x55db71160a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:35854/0 (socket says 172.21.15.89:35854) 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160fa0 0x55db71173c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44456/0 (socket says 172.21.15.89:44456) 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861bfff640 1 -- 172.21.15.89:0/1876107493 learned_addr learned my addr 172.21.15.89:0/1876107493 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f8620cb6640 1 --2- 172.21.15.89:0/1876107493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71174150 0x55db71176540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861bfff640 1 -- 172.21.15.89:0/1876107493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71174150 msgr2=0x55db71176540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:14.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861bfff640 1 --2- 172.21.15.89:0/1876107493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71174150 0x55db71176540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861bfff640 1 -- 172.21.15.89:0/1876107493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160fa0 msgr2=0x55db71173c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861bfff640 1 --2- 172.21.15.89:0/1876107493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160fa0 0x55db71173c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861bfff640 1 -- 172.21.15.89:0/1876107493 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db70e37b90 con 0x55db71160680 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f861bfff640 1 --2- 172.21.15.89:0/1876107493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160680 0x55db71160a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f860c004650 tx=0x7f860c003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feed9b9ec7553097 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f86027fc640 1 -- 172.21.15.89:0/1876107493 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c049040 con 0x55db71160680 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f86027fc640 1 -- 172.21.15.89:0/1876107493 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f860c03b580 con 0x55db71160680 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.575+0000 7f86027fc640 1 -- 172.21.15.89:0/1876107493 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c0c52c0 con 0x55db71160680 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.576+0000 7f8624af2740 1 -- 172.21.15.89:0/1876107493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160680 msgr2=0x55db71160a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.576+0000 7f8624af2740 1 --2- 172.21.15.89:0/1876107493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160680 0x55db71160a60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f860c004650 tx=0x7f860c003f90 comp rx=0 tx=0).stop 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.576+0000 7f8624af2740 1 -- 172.21.15.89:0/1876107493 shutdown_connections 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.576+0000 7f8624af2740 1 --2- 172.21.15.89:0/1876107493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71174150 0x55db71176540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.576+0000 7f8624af2740 1 --2- 172.21.15.89:0/1876107493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160fa0 0x55db71173c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.576+0000 7f8624af2740 1 --2- 172.21.15.89:0/1876107493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160680 0x55db71160a60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.576+0000 7f8624af2740 1 -- 172.21.15.89:0/1876107493 >> 172.21.15.89:0/1876107493 conn(0x55db7115ccf0 msgr2=0x55db7115d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.577+0000 7f8624af2740 1 -- 172.21.15.89:0/1876107493 shutdown_connections 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.577+0000 7f8624af2740 1 -- 172.21.15.89:0/1876107493 wait complete. 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.577+0000 7f8624af2740 1 -- start start 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.577+0000 7f8624af2740 1 Processor -- start 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f8624af2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db711711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f8624af2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 0x55db711716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f8624af2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71174150 0x55db71171c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f8624af2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55db71176a80 con 0x55db71160680 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f8624af2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55db7104e6e0 con 0x55db71160fa0 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f8624af2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55db71176c00 con 0x55db71174150 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f8624af2740 10 client.0 _ll_register_callbacks cb 0x55db71072a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f861bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db711711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f861bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db711711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44464/0 (socket says 172.21.15.89:44464) 2024-09-15T23:25:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f861bfff640 1 -- 172.21.15.89:0/450049013 learned_addr learned my addr 172.21.15.89:0/450049013 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f861b7fe640 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 0x55db711716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f8620cb6640 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71174150 0x55db71171c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f861b7fe640 1 -- 172.21.15.89:0/450049013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71174150 msgr2=0x55db71171c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f861b7fe640 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71174150 0x55db71171c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f861b7fe640 1 -- 172.21.15.89:0/450049013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 msgr2=0x55db711711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f861b7fe640 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db711711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f861b7fe640 1 -- 172.21.15.89:0/450049013 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db71180020 con 0x55db71160fa0 2024-09-15T23:25:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.578+0000 7f861bfff640 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db711711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.579+0000 7f861b7fe640 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 0x55db711716f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8610011790 tx=0x7f8610004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90bec356f51c5c2a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.579+0000 7f85fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.579+0000 7f85fb7fe640 1 -- 172.21.15.89:0/450049013 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861001fbb0 con 0x55db71160fa0 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.579+0000 7f85fb7fe640 1 -- 172.21.15.89:0/450049013 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f861001fd50 con 0x55db71160fa0 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.579+0000 7f8624af2740 1 -- 172.21.15.89:0/450049013 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55db710d39d0 con 0x55db71160fa0 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.579+0000 7f85fb7fe640 1 -- 172.21.15.89:0/450049013 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8610028440 con 0x55db71160fa0 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.579+0000 7f8624af2740 10 client.25082 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.579+0000 7f8624af2740 1 -- 172.21.15.89:0/450049013 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55db711829f0 con 0x55db71160fa0 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.580+0000 7f85fb7fe640 1 -- 172.21.15.89:0/450049013 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55db710d39d0 con 0x55db71160fa0 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.580+0000 7f85fb7fe640 1 -- 172.21.15.89:0/450049013 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8610026070 con 0x55db71160fa0 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.580+0000 7f8624af2740 10 client.25082 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.580+0000 7f8624af2740 10 client.25082 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:25:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.580+0000 7f8624af2740 20 client.25082 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.580+0000 7f8624af2740 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55db711842d0 0x55db71186750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.580+0000 7f8624af2740 4 client.25082 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.580+0000 7f8624af2740 1 -- 172.21.15.89:0/450049013 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db710cdd20 con 0x55db711842d0 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.580+0000 7f861bfff640 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55db711842d0 0x55db71186750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.581+0000 7f85fb7fe640 10 client.25082 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.581+0000 7f861bfff640 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55db711842d0 0x55db71186750 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bd84e6bbca95407 server_cookie=2a9a97756c319777 in_seq=0 out_seq=0 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f85fb7fe640 1 -- 172.21.15.89:0/450049013 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55db710cdd20 con 0x55db711842d0 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f85fb7fe640 10 client.25082 handle_command_reply: tid=0 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 client.25082 shutdown 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 -- 172.21.15.89:0/450049013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 msgr2=0x55db711716f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 0x55db711716f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8610011790 tx=0x7f8610004730 comp rx=0 tx=0).stop 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 -- 172.21.15.89:0/450049013 shutdown_connections 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55db711842d0 0x55db71186750 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71174150 0x55db71171c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 0x55db711716f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 --2- 172.21.15.89:0/450049013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db711711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 -- 172.21.15.89:0/450049013 >> 172.21.15.89:0/450049013 conn(0x55db7115ccf0 msgr2=0x55db7115be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f85fb7fe640 0 client.25082 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 -- 172.21.15.89:0/450049013 shutdown_connections 2024-09-15T23:25:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 1 -- 172.21.15.89:0/450049013 wait complete. 2024-09-15T23:25:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.582+0000 7f8624af2740 20 client.25082 trim_cache size 0 max 16384 2024-09-15T23:25:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f8624af2740 1 Processor -- start 2024-09-15T23:25:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f8624af2740 1 -- start start 2024-09-15T23:25:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f8624af2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db7118c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f8624af2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160fa0 0x55db7118c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f8624af2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71174150 0x55db7116e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f8624af2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1acd40 con 0x55db71160680 2024-09-15T23:25:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f861bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db7118c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f861b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160fa0 0x55db7118c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f861bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db7118c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44472/0 (socket says 172.21.15.89:44472) 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f861b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160fa0 0x55db7118c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:47908/0 (socket says 172.21.15.89:47908) 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f861bfff640 1 -- 172.21.15.89:0/1489054620 learned_addr learned my addr 172.21.15.89:0/1489054620 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f8624af2740 1 -- 172.21.15.89:0/1489054620 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1ae880 con 0x55db71174150 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.614+0000 7f8624af2740 1 -- 172.21.15.89:0/1489054620 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1ad410 con 0x55db71160fa0 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f8620cb6640 1 --2- 172.21.15.89:0/1489054620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71174150 0x55db7116e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f861b7fe640 1 -- 172.21.15.89:0/1489054620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71174150 msgr2=0x55db7116e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f861b7fe640 1 --2- 172.21.15.89:0/1489054620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71174150 0x55db7116e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f861b7fe640 1 -- 172.21.15.89:0/1489054620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 msgr2=0x55db7118c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f861b7fe640 1 --2- 172.21.15.89:0/1489054620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db7118c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f861b7fe640 1 -- 172.21.15.89:0/1489054620 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db71158000 con 0x55db71160fa0 2024-09-15T23:25:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f8620cb6640 1 --2- 172.21.15.89:0/1489054620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71174150 0x55db7116e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f861b7fe640 1 --2- 172.21.15.89:0/1489054620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160fa0 0x55db7118c5a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8610027820 tx=0x7f8610027ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1342ac34af8d7e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f8601ffb640 1 -- 172.21.15.89:0/1489054620 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861003b030 con 0x55db71160fa0 2024-09-15T23:25:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f8601ffb640 1 -- 172.21.15.89:0/1489054620 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8610005500 con 0x55db71160fa0 2024-09-15T23:25:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.615+0000 7f8601ffb640 1 -- 172.21.15.89:0/1489054620 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861001fa60 con 0x55db71160fa0 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.616+0000 7f8624af2740 1 -- 172.21.15.89:0/1489054620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160fa0 msgr2=0x55db7118c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.616+0000 7f8624af2740 1 --2- 172.21.15.89:0/1489054620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160fa0 0x55db7118c5a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8610027820 tx=0x7f8610027ee0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.617+0000 7f8624af2740 1 -- 172.21.15.89:0/1489054620 shutdown_connections 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.617+0000 7f8624af2740 1 --2- 172.21.15.89:0/1489054620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71174150 0x55db7116e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.617+0000 7f8624af2740 1 --2- 172.21.15.89:0/1489054620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160fa0 0x55db7118c5a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.617+0000 7f8624af2740 1 --2- 172.21.15.89:0/1489054620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71160680 0x55db7118c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.617+0000 7f8624af2740 1 -- 172.21.15.89:0/1489054620 >> 172.21.15.89:0/1489054620 conn(0x55db7115ccf0 msgr2=0x55db71173230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.617+0000 7f8624af2740 1 -- 172.21.15.89:0/1489054620 shutdown_connections 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.617+0000 7f8624af2740 1 -- 172.21.15.89:0/1489054620 wait complete. 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.617+0000 7f8624af2740 1 -- start start 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.617+0000 7f8624af2740 1 Processor -- start 2024-09-15T23:25:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8624af2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160680 0x55db7117b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8624af2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 0x55db7117bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8624af2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71174150 0x55db711789b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8624af2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1ae1b0 con 0x55db71174150 2024-09-15T23:25:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8624af2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1adae0 con 0x55db71160fa0 2024-09-15T23:25:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8624af2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1aad20 con 0x55db71160680 2024-09-15T23:25:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8624af2740 10 client.0 _ll_register_callbacks cb 0x7f8610010ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8620cb6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71174150 0x55db711789b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f861bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160680 0x55db7117b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8620cb6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71174150 0x55db711789b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44478/0 (socket says 172.21.15.89:44478) 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8620cb6640 1 -- 172.21.15.89:0/2990989061 learned_addr learned my addr 172.21.15.89:0/2990989061 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f861b7fe640 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 0x55db7117bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f861bfff640 1 -- 172.21.15.89:0/2990989061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 msgr2=0x55db7117bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f861bfff640 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 0x55db7117bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f861bfff640 1 -- 172.21.15.89:0/2990989061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71174150 msgr2=0x55db711789b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f861bfff640 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71174150 0x55db711789b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f861bfff640 1 -- 172.21.15.89:0/2990989061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db7117c720 con 0x55db71160680 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f8620cb6640 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71174150 0x55db711789b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f861bfff640 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160680 0x55db7117b9e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f860c0bc5e0 tx=0x7f860c0058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d07cf0833333f536 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.618+0000 7f85fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.619+0000 7f85fb7fe640 1 -- 172.21.15.89:0/2990989061 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c0da040 con 0x55db71160680 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.619+0000 7f85fb7fe640 1 -- 172.21.15.89:0/2990989061 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f860c0edcd0 con 0x55db71160680 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.619+0000 7f8624af2740 1 -- 172.21.15.89:0/2990989061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8610005500 con 0x55db71160680 2024-09-15T23:25:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.619+0000 7f85fb7fe640 1 -- 172.21.15.89:0/2990989061 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f860c049040 con 0x55db71160680 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.619+0000 7f8624af2740 10 client.25081 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.619+0000 7f8624af2740 1 -- 172.21.15.89:0/2990989061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55db711ef560 con 0x55db71160680 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.619+0000 7f85fb7fe640 1 -- 172.21.15.89:0/2990989061 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8610005500 con 0x55db71160680 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.620+0000 7f85fb7fe640 1 -- 172.21.15.89:0/2990989061 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f860c0b8cc0 con 0x55db71160680 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.620+0000 7f8624af2740 10 client.25081 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.620+0000 7f8624af2740 10 client.25081 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.620+0000 7f8624af2740 20 client.25081 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.620+0000 7f8624af2740 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55db711efb10 0x55db711f1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.620+0000 7f8624af2740 4 client.25081 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.2, "await_for": 45.0}] 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.620+0000 7f8624af2740 1 -- 172.21.15.89:0/2990989061 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.2, "await_for": 45.0}) -- 0x55db71049ec0 con 0x55db711efb10 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.620+0000 7f861b7fe640 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55db711efb10 0x55db711f1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.621+0000 7f85fb7fe640 10 client.25081 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.621+0000 7f861b7fe640 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55db711efb10 0x55db711f1f00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a005690486505a22 server_cookie=d0b49591bcc01720 in_seq=0 out_seq=0 2024-09-15T23:25:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.632+0000 7f85fb7fe640 1 -- 172.21.15.89:0/2990989061 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55db71049ec0 con 0x55db711efb10 2024-09-15T23:25:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.632+0000 7f85fb7fe640 10 client.25081 handle_command_reply: tid=0 2024-09-15T23:25:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.632+0000 7f8624af2740 1 client.25081 shutdown 2024-09-15T23:25:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.632+0000 7f8624af2740 1 -- 172.21.15.89:0/2990989061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160680 msgr2=0x55db7117b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:14.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.632+0000 7f8624af2740 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160680 0x55db7117b9e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f860c0bc5e0 tx=0x7f860c0058d0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.632+0000 7f8624af2740 1 -- 172.21.15.89:0/2990989061 shutdown_connections 2024-09-15T23:25:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.632+0000 7f8624af2740 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55db711efb10 0x55db711f1f00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.632+0000 7f8624af2740 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db71174150 0x55db711789b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.632+0000 7f85fb7fe640 0 client.25081 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.632+0000 7f8624af2740 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db71160fa0 0x55db7117bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.633+0000 7f8624af2740 1 --2- 172.21.15.89:0/2990989061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55db71160680 0x55db7117b9e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.633+0000 7f8624af2740 1 -- 172.21.15.89:0/2990989061 >> 172.21.15.89:0/2990989061 conn(0x55db7115ccf0 msgr2=0x55db7115b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:14.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.633+0000 7f8624af2740 1 -- 172.21.15.89:0/2990989061 shutdown_connections 2024-09-15T23:25:14.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.633+0000 7f8624af2740 1 -- 172.21.15.89:0/2990989061 wait complete. 2024-09-15T23:25:14.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.633+0000 7f8624af2740 20 client.25081 trim_cache size 0 max 16384 2024-09-15T23:25:14.635 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:25:14.635 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:25:14.635 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:25:14.635 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 2, 2024-09-15T23:25:14.635 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:25:14.635 INFO:teuthology.orchestra.run.smithi089.stdout: "90d6d4a2": { 2024-09-15T23:25:14.635 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 2, 2024-09-15T23:25:14.635 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 150.2, 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:25:14.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:25:14.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:25:14.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:25:14.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:25:14.637 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.633+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86040a32d0 msgr2=0x7f86040a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.633+0000 7f8622719640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86040a32d0 0x7f86040a5790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f861c0ff810 tx=0x7f861000b390 comp rx=0 tx=0).stop 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.633+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861c100100 msgr2=0x7f861c198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.634+0000 7f8622719640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861c100100 0x7f861c198570 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f860c006010 tx=0x7f860c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.634+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 shutdown_connections 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.634+0000 7f8622719640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86040a32d0 0x7f86040a5790 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.634+0000 7f8622719640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f861c106b80 0x7f861c19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.634+0000 7f8622719640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f861c100aa0 0x7f861c198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.634+0000 7f8622719640 1 --2- 172.21.15.89:0/1277769698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f861c100100 0x7f861c198570 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.634+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 >> 172.21.15.89:0/1277769698 conn(0x7f861c077230 msgr2=0x7f861c104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:14.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.634+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 shutdown_connections 2024-09-15T23:25:14.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:14.634+0000 7f8622719640 1 -- 172.21.15.89:0/1277769698 wait complete. 2024-09-15T23:25:14.643 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:15.394 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '90d6d4a2', quiesce took 9.6 seconds. Will release after: 20.6 2024-09-15T23:25:15.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.902+0000 7f272f632640 1 -- 172.21.15.89:0/265873842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2728100a90 msgr2=0x7f2728100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.902+0000 7f272f632640 1 --2- 172.21.15.89:0/265873842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2728100a90 0x7f2728100ef0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2718009930 tx=0x7f271802a420 comp rx=0 tx=0).stop 2024-09-15T23:25:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.903+0000 7f272f632640 1 -- 172.21.15.89:0/265873842 shutdown_connections 2024-09-15T23:25:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.903+0000 7f272f632640 1 --2- 172.21.15.89:0/265873842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2728101430 0x7f272810d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.903+0000 7f272f632640 1 --2- 172.21.15.89:0/265873842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2728100a90 0x7f2728100ef0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.903+0000 7f272f632640 1 --2- 172.21.15.89:0/265873842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2728106a90 0x7f2728106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.903+0000 7f272f632640 1 -- 172.21.15.89:0/265873842 >> 172.21.15.89:0/265873842 conn(0x7f27280fc7d0 msgr2=0x7f27280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.903+0000 7f272f632640 1 -- 172.21.15.89:0/265873842 shutdown_connections 2024-09-15T23:25:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.903+0000 7f272f632640 1 -- 172.21.15.89:0/265873842 wait complete. 2024-09-15T23:25:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.903+0000 7f272f632640 1 Processor -- start 2024-09-15T23:25:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272f632640 1 -- start start 2024-09-15T23:25:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272f632640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2728100a90 0x7f27281983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272f632640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2728101430 0x7f2728198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272f632640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2728106a90 0x7f272819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272f632640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2728110e50 con 0x7f2728106a90 2024-09-15T23:25:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272f632640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2728110cd0 con 0x7f2728101430 2024-09-15T23:25:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272f632640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2728110fd0 con 0x7f2728100a90 2024-09-15T23:25:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272d3ce640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2728100a90 0x7f27281983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272cbcd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2728101430 0x7f2728198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272cbcd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2728101430 0x7f2728198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:41828/0 (socket says 172.21.15.89:41828) 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272d3ce640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2728100a90 0x7f27281983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:34100/0 (socket says 172.21.15.89:34100) 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272cbcd640 1 -- 172.21.15.89:0/2774626259 learned_addr learned my addr 172.21.15.89:0/2774626259 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.904+0000 7f272dbcf640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2728106a90 0x7f272819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.905+0000 7f272cbcd640 1 -- 172.21.15.89:0/2774626259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2728100a90 msgr2=0x7f27281983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.905+0000 7f272cbcd640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2728100a90 0x7f27281983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.905+0000 7f272cbcd640 1 -- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2728106a90 msgr2=0x7f272819cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.905+0000 7f272cbcd640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2728106a90 0x7f272819cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.905+0000 7f272cbcd640 1 -- 172.21.15.89:0/2774626259 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272819d380 con 0x7f2728101430 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.905+0000 7f272dbcf640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2728106a90 0x7f272819cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.905+0000 7f272cbcd640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2728101430 0x7f2728198910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2718009230 tx=0x7f2718031b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.905+0000 7f271e7fc640 1 -- 172.21.15.89:0/2774626259 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718035070 con 0x7f2728101430 2024-09-15T23:25:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.905+0000 7f272f632640 1 -- 172.21.15.89:0/2774626259 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f272819d610 con 0x7f2728101430 2024-09-15T23:25:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.905+0000 7f272f632640 1 -- 172.21.15.89:0/2774626259 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27280fccc0 con 0x7f2728101430 2024-09-15T23:25:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.906+0000 7f271e7fc640 1 -- 172.21.15.89:0/2774626259 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f271803dc40 con 0x7f2728101430 2024-09-15T23:25:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.906+0000 7f271e7fc640 1 -- 172.21.15.89:0/2774626259 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718003de0 con 0x7f2728101430 2024-09-15T23:25:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.908+0000 7f2731a0b740 1 Processor -- start 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.908+0000 7f2731a0b740 1 -- start start 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.908+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d526b40 0x55bd9d525090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.908+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d5255d0 0x55bd9d525a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.908+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d530370 0x55bd9d532760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.908+0000 7f2731a0b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bd9d48b2f0 con 0x55bd9d5255d0 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.908+0000 7f2731a0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bd9d432cf0 con 0x55bd9d530370 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.908+0000 7f2731a0b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bd9d4872b0 con 0x55bd9d526b40 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f272d3ce640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d5255d0 0x55bd9d525a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f272d3ce640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d5255d0 0x55bd9d525a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45880/0 (socket says 172.21.15.89:45880) 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f272d3ce640 1 -- 172.21.15.89:0/1032199761 learned_addr learned my addr 172.21.15.89:0/1032199761 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f272dbcf640 1 --2- 172.21.15.89:0/1032199761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d526b40 0x55bd9d525090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f272d3ce640 1 -- 172.21.15.89:0/1032199761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d526b40 msgr2=0x55bd9d525090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f272d3ce640 1 --2- 172.21.15.89:0/1032199761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d526b40 0x55bd9d525090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f272d3ce640 1 -- 172.21.15.89:0/1032199761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d530370 msgr2=0x55bd9d532760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:25:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f272d3ce640 1 --2- 172.21.15.89:0/1032199761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d530370 0x55bd9d532760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f272d3ce640 1 -- 172.21.15.89:0/1032199761 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd9d4173a0 con 0x55bd9d5255d0 2024-09-15T23:25:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f272d3ce640 1 --2- 172.21.15.89:0/1032199761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d5255d0 0x55bd9d525a10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f272400ac90 tx=0x7f272400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c56b6731070858 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.909+0000 7f27137fe640 1 -- 172.21.15.89:0/1032199761 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272401d070 con 0x55bd9d5255d0 2024-09-15T23:25:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.910+0000 7f27137fe640 1 -- 172.21.15.89:0/1032199761 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2724018b20 con 0x55bd9d5255d0 2024-09-15T23:25:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.910+0000 7f27137fe640 1 -- 172.21.15.89:0/1032199761 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2724010bc0 con 0x55bd9d5255d0 2024-09-15T23:25:15.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.910+0000 7f271e7fc640 1 -- 172.21.15.89:0/2774626259 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f271803d760 con 0x7f2728101430 2024-09-15T23:25:15.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.910+0000 7f2731a0b740 1 -- 172.21.15.89:0/1032199761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d5255d0 msgr2=0x55bd9d525a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.910+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1032199761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d5255d0 0x55bd9d525a10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f272400ac90 tx=0x7f272400f790 comp rx=0 tx=0).stop 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.910+0000 7f2731a0b740 1 -- 172.21.15.89:0/1032199761 shutdown_connections 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.910+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1032199761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d530370 0x55bd9d532760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.910+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1032199761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d5255d0 0x55bd9d525a10 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.910+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1032199761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d526b40 0x55bd9d525090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.910+0000 7f2731a0b740 1 -- 172.21.15.89:0/1032199761 >> 172.21.15.89:0/1032199761 conn(0x55bd9d4ee310 msgr2=0x55bd9d521c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.911+0000 7f2731a0b740 1 -- 172.21.15.89:0/1032199761 shutdown_connections 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.911+0000 7f2731a0b740 1 -- 172.21.15.89:0/1032199761 wait complete. 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.911+0000 7f2731a0b740 1 -- start start 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.911+0000 7f2731a0b740 1 Processor -- start 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 0x55bd9d52d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d52d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d52dcb0 0x55bd9d53c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f2731a0b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bd9d533110 con 0x55bd9d530370 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f2731a0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bd9d446060 con 0x55bd9d526b40 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f2731a0b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bd9d533290 con 0x55bd9d52dcb0 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f2731a0b740 10 client.0 _ll_register_callbacks cb 0x55bd9d1998c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f272d3ce640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d52d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f272dbcf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 0x55bd9d52d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f272d3ce640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d52d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45892/0 (socket says 172.21.15.89:45892) 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f272d3ce640 1 -- 172.21.15.89:0/1531639738 learned_addr learned my addr 172.21.15.89:0/1531639738 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f272cbcd640 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d52dcb0 0x55bd9d53c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f271e7fc640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f27140a3260 0x7f27140a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f272dbcf640 1 -- 172.21.15.89:0/1531639738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d52dcb0 msgr2=0x55bd9d53c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f272dbcf640 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d52dcb0 0x55bd9d53c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.912+0000 7f272dbcf640 1 -- 172.21.15.89:0/1531639738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 msgr2=0x55bd9d52d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:15.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f272dbcf640 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d52d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f272dbcf640 1 -- 172.21.15.89:0/1531639738 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd9d1fc3c0 con 0x55bd9d526b40 2024-09-15T23:25:15.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f272dbcf640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f27140a3260 0x7f27140a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f271e7fc640 1 -- 172.21.15.89:0/2774626259 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f27180f4690 con 0x7f2728101430 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f272d3ce640 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d52d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f272dbcf640 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 0x55bd9d52d230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2720011f20 tx=0x7f27200138f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=892ce96ca270571d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f2710ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f2710ff9640 1 -- 172.21.15.89:0/1531639738 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720019040 con 0x55bd9d526b40 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f2731a0b740 1 -- 172.21.15.89:0/1531639738 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2724018b20 con 0x55bd9d526b40 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f2710ff9640 1 -- 172.21.15.89:0/1531639738 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f272001d840 con 0x55bd9d526b40 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f272dbcf640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f27140a3260 0x7f27140a5720 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2720024000 tx=0x7f2720025000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.913+0000 7f2710ff9640 1 -- 172.21.15.89:0/1531639738 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2720010e80 con 0x55bd9d526b40 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.914+0000 7f2731a0b740 10 client.25094 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.914+0000 7f2731a0b740 1 -- 172.21.15.89:0/1531639738 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bd9d53f2a0 con 0x55bd9d526b40 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.914+0000 7f2710ff9640 1 -- 172.21.15.89:0/1531639738 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2724018b20 con 0x55bd9d526b40 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.914+0000 7f2710ff9640 1 -- 172.21.15.89:0/1531639738 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f272000f040 con 0x55bd9d526b40 2024-09-15T23:25:15.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.914+0000 7f2731a0b740 10 client.25094 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.914+0000 7f2731a0b740 10 client.25094 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.914+0000 7f2731a0b740 20 client.25094 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.914+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bd9d540850 0x55bd9d542c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.914+0000 7f2731a0b740 4 client.25094 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.914+0000 7f2731a0b740 1 -- 172.21.15.89:0/1531639738 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd9d498860 con 0x55bd9d540850 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.915+0000 7f272d3ce640 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bd9d540850 0x55bd9d542c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.915+0000 7f272d3ce640 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bd9d540850 0x55bd9d542c90 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b7628a736f57e92f server_cookie=919a403005717ec5 in_seq=0 out_seq=0 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.915+0000 7f2710ff9640 10 client.25094 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2710ff9640 1 -- 172.21.15.89:0/1531639738 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bd9d498860 con 0x55bd9d540850 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2710ff9640 10 client.25094 handle_command_reply: tid=0 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 client.25094 shutdown 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 -- 172.21.15.89:0/1531639738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 msgr2=0x55bd9d52d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 0x55bd9d52d230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2720011f20 tx=0x7f27200138f0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 -- 172.21.15.89:0/1531639738 shutdown_connections 2024-09-15T23:25:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bd9d540850 0x55bd9d542c90 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d52dcb0 0x55bd9d53c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d52d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1531639738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 0x55bd9d52d230 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 -- 172.21.15.89:0/1531639738 >> 172.21.15.89:0/1531639738 conn(0x55bd9d4ee310 msgr2=0x55bd9d527930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2710ff9640 0 client.25094 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 -- 172.21.15.89:0/1531639738 shutdown_connections 2024-09-15T23:25:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 1 -- 172.21.15.89:0/1531639738 wait complete. 2024-09-15T23:25:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.916+0000 7f2731a0b740 20 client.25094 trim_cache size 0 max 16384 2024-09-15T23:25:15.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.929+0000 7f2731a0b740 1 Processor -- start 2024-09-15T23:25:15.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.929+0000 7f2731a0b740 1 -- start start 2024-09-15T23:25:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.929+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d526b40 0x55bd9d52c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.929+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d52dcb0 0x55bd9d53c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.929+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d530370 0x55bd9d53c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f2731a0b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f270c00fcd0 con 0x55bd9d526b40 2024-09-15T23:25:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f2731a0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f270c011810 con 0x55bd9d52dcb0 2024-09-15T23:25:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f2731a0b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f270c0103a0 con 0x55bd9d530370 2024-09-15T23:25:15.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272d3ce640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d526b40 0x55bd9d52c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272d3ce640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d526b40 0x55bd9d52c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45906/0 (socket says 172.21.15.89:45906) 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272d3ce640 1 -- 172.21.15.89:0/1001194122 learned_addr learned my addr 172.21.15.89:0/1001194122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272cbcd640 1 --2- 172.21.15.89:0/1001194122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d52dcb0 0x55bd9d53c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272dbcf640 1 --2- 172.21.15.89:0/1001194122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d530370 0x55bd9d53c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272d3ce640 1 -- 172.21.15.89:0/1001194122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d530370 msgr2=0x55bd9d53c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272d3ce640 1 --2- 172.21.15.89:0/1001194122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d530370 0x55bd9d53c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272d3ce640 1 -- 172.21.15.89:0/1001194122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d52dcb0 msgr2=0x55bd9d53c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272d3ce640 1 --2- 172.21.15.89:0/1001194122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d52dcb0 0x55bd9d53c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272d3ce640 1 -- 172.21.15.89:0/1001194122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd9d522dd0 con 0x55bd9d526b40 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.930+0000 7f272d3ce640 1 --2- 172.21.15.89:0/1001194122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d526b40 0x55bd9d52c210 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f272400abb0 tx=0x7f272400a820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7da50b03c6922d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.931+0000 7f2712ffd640 1 -- 172.21.15.89:0/1001194122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272401d040 con 0x55bd9d526b40 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.931+0000 7f2712ffd640 1 -- 172.21.15.89:0/1001194122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f272400a890 con 0x55bd9d526b40 2024-09-15T23:25:15.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.931+0000 7f2712ffd640 1 -- 172.21.15.89:0/1001194122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2724004660 con 0x55bd9d526b40 2024-09-15T23:25:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.932+0000 7f2731a0b740 1 -- 172.21.15.89:0/1001194122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d526b40 msgr2=0x55bd9d52c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.932+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1001194122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d526b40 0x55bd9d52c210 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f272400abb0 tx=0x7f272400a820 comp rx=0 tx=0).stop 2024-09-15T23:25:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.932+0000 7f2731a0b740 1 -- 172.21.15.89:0/1001194122 shutdown_connections 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.932+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1001194122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d530370 0x55bd9d53c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.932+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1001194122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d52dcb0 0x55bd9d53c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.932+0000 7f2731a0b740 1 --2- 172.21.15.89:0/1001194122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d526b40 0x55bd9d52c210 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.932+0000 7f2731a0b740 1 -- 172.21.15.89:0/1001194122 >> 172.21.15.89:0/1001194122 conn(0x55bd9d4ee310 msgr2=0x55bd9d52bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.932+0000 7f2731a0b740 1 -- 172.21.15.89:0/1001194122 shutdown_connections 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.932+0000 7f2731a0b740 1 -- 172.21.15.89:0/1001194122 wait complete. 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.932+0000 7f2731a0b740 1 -- start start 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f2731a0b740 1 Processor -- start 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 0x55bd9d538ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d52dcb0 0x55bd9d547da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f2731a0b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d5482e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f2731a0b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f270c011140 con 0x55bd9d530370 2024-09-15T23:25:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f2731a0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f270c010a70 con 0x55bd9d526b40 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f2731a0b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f270c00dcb0 con 0x55bd9d52dcb0 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f2731a0b740 10 client.0 _ll_register_callbacks cb 0x7f27200113e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f272dbcf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d5482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f272dbcf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d5482e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45912/0 (socket says 172.21.15.89:45912) 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f272d3ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 0x55bd9d538ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.933+0000 7f272dbcf640 1 -- 172.21.15.89:0/3387720277 learned_addr learned my addr 172.21.15.89:0/3387720277 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.934+0000 7f272dbcf640 1 -- 172.21.15.89:0/3387720277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d52dcb0 msgr2=0x55bd9d547da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.934+0000 7f272dbcf640 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d52dcb0 0x55bd9d547da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.934+0000 7f272dbcf640 1 -- 172.21.15.89:0/3387720277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 msgr2=0x55bd9d538ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.934+0000 7f272dbcf640 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 0x55bd9d538ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.934+0000 7f272dbcf640 1 -- 172.21.15.89:0/3387720277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd9d528120 con 0x55bd9d530370 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.934+0000 7f272dbcf640 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d5482e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2720016040 tx=0x7f2720004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd142f017be62a12 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.934+0000 7f2710ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.934+0000 7f2710ff9640 1 -- 172.21.15.89:0/3387720277 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720019040 con 0x55bd9d530370 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.934+0000 7f2710ff9640 1 -- 172.21.15.89:0/3387720277 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2720011020 con 0x55bd9d530370 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.934+0000 7f2731a0b740 1 -- 172.21.15.89:0/3387720277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bd9d492d70 con 0x55bd9d530370 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2710ff9640 1 -- 172.21.15.89:0/3387720277 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2720011220 con 0x55bd9d530370 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2731a0b740 10 client.15714 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2731a0b740 1 -- 172.21.15.89:0/3387720277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bd9d5263a0 con 0x55bd9d530370 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2710ff9640 1 -- 172.21.15.89:0/3387720277 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bd9d492d70 con 0x55bd9d530370 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2710ff9640 1 -- 172.21.15.89:0/3387720277 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f272001dc20 con 0x55bd9d530370 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2731a0b740 10 client.15714 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2731a0b740 10 client.15714 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2731a0b740 20 client.15714 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2731a0b740 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bd9d5b4130 0x55bd9d5b6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2731a0b740 4 client.15714 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.935+0000 7f2731a0b740 1 -- 172.21.15.89:0/3387720277 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bd9d40f170 con 0x55bd9d5b4130 2024-09-15T23:25:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.936+0000 7f272d3ce640 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bd9d5b4130 0x55bd9d5b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.936+0000 7f272d3ce640 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bd9d5b4130 0x55bd9d5b6550 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc4ec8f23f9e028a server_cookie=f9a76a8655eb7099 in_seq=0 out_seq=0 2024-09-15T23:25:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.936+0000 7f2710ff9640 10 client.15714 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.936+0000 7f2710ff9640 1 -- 172.21.15.89:0/3387720277 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bd9d40f170 con 0x55bd9d5b4130 2024-09-15T23:25:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.936+0000 7f2710ff9640 10 client.15714 handle_command_reply: tid=0 2024-09-15T23:25:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.936+0000 7f2731a0b740 1 client.15714 shutdown 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 1 -- 172.21.15.89:0/3387720277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 msgr2=0x55bd9d5482e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d5482e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2720016040 tx=0x7f2720004a90 comp rx=0 tx=0).stop 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 1 -- 172.21.15.89:0/3387720277 shutdown_connections 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd9d530370 0x55bd9d5482e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bd9d5b4130 0x55bd9d5b6550 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bd9d52dcb0 0x55bd9d547da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2710ff9640 0 client.15714 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 1 --2- 172.21.15.89:0/3387720277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd9d526b40 0x55bd9d538ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 1 -- 172.21.15.89:0/3387720277 >> 172.21.15.89:0/3387720277 conn(0x55bd9d4ee310 msgr2=0x55bd9d52c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 1 -- 172.21.15.89:0/3387720277 shutdown_connections 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 1 -- 172.21.15.89:0/3387720277 wait complete. 2024-09-15T23:25:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.937+0000 7f2731a0b740 20 client.15714 trim_cache size 0 max 16384 2024-09-15T23:25:15.943 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:15.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.938+0000 7f272f632640 1 -- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f27140a3260 msgr2=0x7f27140a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.938+0000 7f272f632640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f27140a3260 0x7f27140a5720 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2720024000 tx=0x7f2720025000 comp rx=0 tx=0).stop 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.938+0000 7f272f632640 1 -- 172.21.15.89:0/2774626259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2728101430 msgr2=0x7f2728198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.938+0000 7f272f632640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2728101430 0x7f2728198910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2718009230 tx=0x7f2718031b50 comp rx=0 tx=0).stop 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.938+0000 7f272f632640 1 -- 172.21.15.89:0/2774626259 shutdown_connections 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.938+0000 7f272f632640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f27140a3260 0x7f27140a5720 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.938+0000 7f272f632640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2728106a90 0x7f272819cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.938+0000 7f272f632640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2728101430 0x7f2728198910 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.939+0000 7f272f632640 1 --2- 172.21.15.89:0/2774626259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2728100a90 0x7f27281983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.939+0000 7f272f632640 1 -- 172.21.15.89:0/2774626259 >> 172.21.15.89:0/2774626259 conn(0x7f27280fc7d0 msgr2=0x7f272806a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.939+0000 7f272f632640 1 -- 172.21.15.89:0/2774626259 shutdown_connections 2024-09-15T23:25:15.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:15.939+0000 7f272f632640 1 -- 172.21.15.89:0/2774626259 wait complete. 2024-09-15T23:25:16.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:16 smithi089 ceph-mon[30933]: pgmap v387: 129 pgs: 129 active+clean; 669 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-15T23:25:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:16 smithi120 ceph-mon[35250]: pgmap v387: 129 pgs: 129 active+clean; 669 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-15T23:25:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:16 smithi174 ceph-mon[35586]: pgmap v387: 129 pgs: 129 active+clean; 669 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-09-15T23:25:16.696 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:25:16.957 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:17.398 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:17 smithi089 ceph-mon[30933]: pgmap v388: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-15T23:25:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:17 smithi120 ceph-mon[35250]: pgmap v388: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-15T23:25:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:17 smithi174 ceph-mon[35586]: pgmap v388: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-15T23:25:17.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.526+0000 7f0484b96640 1 -- 172.21.15.89:0/2130374035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0480102570 msgr2=0x7f04801029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.526+0000 7f0484b96640 1 --2- 172.21.15.89:0/2130374035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0480102570 0x7f04801029b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0474009900 tx=0x7f047402a480 comp rx=0 tx=0).stop 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.526+0000 7f0484b96640 1 -- 172.21.15.89:0/2130374035 shutdown_connections 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.526+0000 7f0484b96640 1 --2- 172.21.15.89:0/2130374035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0480102ef0 0x7f048010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.526+0000 7f0484b96640 1 --2- 172.21.15.89:0/2130374035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0480102570 0x7f04801029b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.526+0000 7f0484b96640 1 --2- 172.21.15.89:0/2130374035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0480100e20 0x7f0480101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.527+0000 7f0484b96640 1 -- 172.21.15.89:0/2130374035 >> 172.21.15.89:0/2130374035 conn(0x7f04800fc7d0 msgr2=0x7f04800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.527+0000 7f0484b96640 1 -- 172.21.15.89:0/2130374035 shutdown_connections 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.527+0000 7f0484b96640 1 -- 172.21.15.89:0/2130374035 wait complete. 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f0484b96640 1 Processor -- start 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f0484b96640 1 -- start start 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f0484b96640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0480100e20 0x7f0480197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f0484b96640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0480102570 0x7f04801978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f0484b96640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0480102ef0 0x7f048019bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f0484b96640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0480110f50 con 0x7f0480102570 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f0484b96640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0480110dd0 con 0x7f0480100e20 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f0484b96640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f04801110d0 con 0x7f0480102ef0 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f047dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0480102570 0x7f04801978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f047dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0480102570 0x7f04801978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45942/0 (socket says 172.21.15.89:45942) 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f047dd9b640 1 -- 172.21.15.89:0/1194118530 learned_addr learned my addr 172.21.15.89:0/1194118530 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.528+0000 7f047e59c640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0480100e20 0x7f0480197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.529+0000 7f047ed9d640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0480102ef0 0x7f048019bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.529+0000 7f047e59c640 1 -- 172.21.15.89:0/1194118530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0480102ef0 msgr2=0x7f048019bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.529+0000 7f047e59c640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0480102ef0 0x7f048019bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.529+0000 7f047e59c640 1 -- 172.21.15.89:0/1194118530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0480102570 msgr2=0x7f04801978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.529+0000 7f047e59c640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0480102570 0x7f04801978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.529+0000 7f047e59c640 1 -- 172.21.15.89:0/1194118530 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048019c360 con 0x7f0480100e20 2024-09-15T23:25:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.529+0000 7f047dd9b640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0480102570 0x7f04801978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.530+0000 7f047e59c640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0480100e20 0x7f0480197390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f047000c970 tx=0x7f047000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.530+0000 7f046f7fe640 1 -- 172.21.15.89:0/1194118530 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470007bb0 con 0x7f0480100e20 2024-09-15T23:25:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.530+0000 7f0484b96640 1 -- 172.21.15.89:0/1194118530 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f048019c650 con 0x7f0480100e20 2024-09-15T23:25:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.530+0000 7f046f7fe640 1 -- 172.21.15.89:0/1194118530 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0470007d50 con 0x7f0480100e20 2024-09-15T23:25:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.530+0000 7f0484b96640 1 -- 172.21.15.89:0/1194118530 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0480069f70 con 0x7f0480100e20 2024-09-15T23:25:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.530+0000 7f046f7fe640 1 -- 172.21.15.89:0/1194118530 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470018820 con 0x7f0480100e20 2024-09-15T23:25:17.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.533+0000 7f0486f6f740 1 Processor -- start 2024-09-15T23:25:17.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.534+0000 7f0486f6f740 1 -- start start 2024-09-15T23:25:17.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.534+0000 7f046f7fe640 1 -- 172.21.15.89:0/1194118530 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f04700189c0 con 0x7f0480100e20 2024-09-15T23:25:17.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.534+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a9b40 0x5580942a8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a85d0 0x5580942a8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 0x5580942b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f0486f6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55809420e2f0 con 0x5580942a9b40 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f0486f6f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580941b5cf0 con 0x5580942a85d0 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f0486f6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55809420a2b0 con 0x5580942b3370 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f047ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a9b40 0x5580942a8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f047ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a9b40 0x5580942a8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45956/0 (socket says 172.21.15.89:45956) 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f047ed9d640 1 -- 172.21.15.89:0/1262294396 learned_addr learned my addr 172.21.15.89:0/1262294396 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f047e59c640 1 --2- 172.21.15.89:0/1262294396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a85d0 0x5580942a8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f046f7fe640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04680a32d0 0x7f04680a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f047ed9d640 1 -- 172.21.15.89:0/1262294396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 msgr2=0x5580942b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:17.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f047dd9b640 1 --2- 172.21.15.89:0/1262294396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 0x5580942b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f047ed9d640 1 --2- 172.21.15.89:0/1262294396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 0x5580942b5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f047ed9d640 1 -- 172.21.15.89:0/1262294396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a85d0 msgr2=0x5580942a8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.535+0000 7f047ed9d640 1 --2- 172.21.15.89:0/1262294396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a85d0 0x5580942a8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.536+0000 7f047ed9d640 1 -- 172.21.15.89:0/1262294396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55809419a3a0 con 0x5580942a9b40 2024-09-15T23:25:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.536+0000 7f046f7fe640 1 -- 172.21.15.89:0/1194118530 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f04700cfd50 con 0x7f0480100e20 2024-09-15T23:25:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.536+0000 7f047ed9d640 1 --2- 172.21.15.89:0/1262294396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a9b40 0x5580942a8090 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0478007ec0 tx=0x7f047800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c00b64cd58032d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.536+0000 7f047ed9d640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04680a32d0 0x7f04680a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.536+0000 7f046cff9640 1 -- 172.21.15.89:0/1262294396 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478010ba0 con 0x5580942a9b40 2024-09-15T23:25:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.536+0000 7f046cff9640 1 -- 172.21.15.89:0/1262294396 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0478010dc0 con 0x5580942a9b40 2024-09-15T23:25:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.536+0000 7f046cff9640 1 -- 172.21.15.89:0/1262294396 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478017960 con 0x5580942a9b40 2024-09-15T23:25:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.537+0000 7f0486f6f740 1 -- 172.21.15.89:0/1262294396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a9b40 msgr2=0x5580942a8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.537+0000 7f047ed9d640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04680a32d0 0x7f04680a5790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f047801a040 tx=0x7f0478017c40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.537+0000 7f0486f6f740 1 --2- 172.21.15.89:0/1262294396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a9b40 0x5580942a8090 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0478007ec0 tx=0x7f047800f790 comp rx=0 tx=0).stop 2024-09-15T23:25:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.537+0000 7f0486f6f740 1 -- 172.21.15.89:0/1262294396 shutdown_connections 2024-09-15T23:25:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.537+0000 7f0486f6f740 1 --2- 172.21.15.89:0/1262294396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 0x5580942b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.537+0000 7f0486f6f740 1 --2- 172.21.15.89:0/1262294396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a85d0 0x5580942a8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.537+0000 7f0486f6f740 1 --2- 172.21.15.89:0/1262294396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a9b40 0x5580942a8090 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.538+0000 7f0486f6f740 1 -- 172.21.15.89:0/1262294396 >> 172.21.15.89:0/1262294396 conn(0x558094271310 msgr2=0x5580942a4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.538+0000 7f0486f6f740 1 -- 172.21.15.89:0/1262294396 shutdown_connections 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.538+0000 7f0486f6f740 1 -- 172.21.15.89:0/1262294396 wait complete. 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.538+0000 7f0486f6f740 1 -- start start 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.538+0000 7f0486f6f740 1 Processor -- start 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942b2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a9b40 0x5580942b0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 0x5580942b0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f0486f6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5580942ba3e0 con 0x5580942a85d0 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f0486f6f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580941c9060 con 0x5580942a9b40 2024-09-15T23:25:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f0486f6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5580942ba560 con 0x5580942b3370 2024-09-15T23:25:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f0486f6f740 10 client.0 _ll_register_callbacks cb 0x558093f1c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f047ed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 0x5580942b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f047ed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 0x5580942b0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:34172/0 (socket says 172.21.15.89:34172) 2024-09-15T23:25:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f047ed9d640 1 -- 172.21.15.89:0/1915610805 learned_addr learned my addr 172.21.15.89:0/1915610805 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.539+0000 7f047e59c640 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942b2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f047ed9d640 1 -- 172.21.15.89:0/1915610805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a9b40 msgr2=0x5580942b0030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:25:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f047ed9d640 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a9b40 0x5580942b0030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f047ed9d640 1 -- 172.21.15.89:0/1915610805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 msgr2=0x5580942b2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f047ed9d640 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942b2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f047ed9d640 1 -- 172.21.15.89:0/1915610805 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558093f7f3c0 con 0x5580942b3370 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f047e59c640 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f047ed9d640 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 0x5580942b0640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0478017440 tx=0x7f0478017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3205635a37667c9f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f0465ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f0465ffb640 1 -- 172.21.15.89:0/1915610805 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478034060 con 0x5580942b3370 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f0465ffb640 1 -- 172.21.15.89:0/1915610805 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0478010ba0 con 0x5580942b3370 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.540+0000 7f0486f6f740 1 -- 172.21.15.89:0/1915610805 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0478010dc0 con 0x5580942b3370 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0465ffb640 1 -- 172.21.15.89:0/1915610805 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f047803b050 con 0x5580942b3370 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0486f6f740 10 client.25099 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0486f6f740 1 -- 172.21.15.89:0/1915610805 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5580942c5ea0 con 0x5580942b3370 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0465ffb640 1 -- 172.21.15.89:0/1915610805 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0478003e50 con 0x5580942b3370 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0465ffb640 1 -- 172.21.15.89:0/1915610805 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f04780050c0 con 0x5580942b3370 2024-09-15T23:25:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0486f6f740 10 client.25099 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0486f6f740 10 client.25099 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0486f6f740 20 client.25099 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0486f6f740 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5580942c7300 0x5580942c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0486f6f740 4 client.25099 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.541+0000 7f0486f6f740 1 -- 172.21.15.89:0/1915610805 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0478010dc0 con 0x5580942c7300 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.542+0000 7f047e59c640 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5580942c7300 0x5580942c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.542+0000 7f047e59c640 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5580942c7300 0x5580942c96f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dcba3e1b1ba1795a server_cookie=96a7c97d689ddcdb in_seq=0 out_seq=0 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.542+0000 7f0465ffb640 10 client.25099 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0465ffb640 1 -- 172.21.15.89:0/1915610805 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f0478010dc0 con 0x5580942c7300 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0465ffb640 10 client.25099 handle_command_reply: tid=0 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0486f6f740 1 client.25099 shutdown 2024-09-15T23:25:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0486f6f740 1 -- 172.21.15.89:0/1915610805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 msgr2=0x5580942b0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0486f6f740 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 0x5580942b0640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0478017440 tx=0x7f0478017000 comp rx=0 tx=0).stop 2024-09-15T23:25:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0486f6f740 1 -- 172.21.15.89:0/1915610805 shutdown_connections 2024-09-15T23:25:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0486f6f740 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5580942c7300 0x5580942c96f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0486f6f740 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942b3370 0x5580942b0640 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0486f6f740 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a9b40 0x5580942b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0486f6f740 1 --2- 172.21.15.89:0/1915610805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942b2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0486f6f740 1 -- 172.21.15.89:0/1915610805 >> 172.21.15.89:0/1915610805 conn(0x558094271310 msgr2=0x5580942aa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.543+0000 7f0465ffb640 0 client.25099 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.544+0000 7f0486f6f740 1 -- 172.21.15.89:0/1915610805 shutdown_connections 2024-09-15T23:25:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.544+0000 7f0486f6f740 1 -- 172.21.15.89:0/1915610805 wait complete. 2024-09-15T23:25:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.544+0000 7f0486f6f740 20 client.25099 trim_cache size 0 max 16384 2024-09-15T23:25:17.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.568+0000 7f0486f6f740 1 Processor -- start 2024-09-15T23:25:17.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.568+0000 7f0486f6f740 1 -- start start 2024-09-15T23:25:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a85d0 0x5580942a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942a9b40 0x5580942c3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942b3370 0x5580942ab120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f0486f6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f046000fd20 con 0x5580942b3370 2024-09-15T23:25:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f0486f6f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0460011860 con 0x5580942a85d0 2024-09-15T23:25:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f0486f6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f04600103f0 con 0x5580942a9b40 2024-09-15T23:25:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f047ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942b3370 0x5580942ab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f047ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942b3370 0x5580942ab120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45976/0 (socket says 172.21.15.89:45976) 2024-09-15T23:25:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f047ed9d640 1 -- 172.21.15.89:0/527198660 learned_addr learned my addr 172.21.15.89:0/527198660 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f047dd9b640 1 --2- 172.21.15.89:0/527198660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942a9b40 0x5580942c3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f047e59c640 1 --2- 172.21.15.89:0/527198660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a85d0 0x5580942a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f047ed9d640 1 -- 172.21.15.89:0/527198660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942a9b40 msgr2=0x5580942c3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f047ed9d640 1 --2- 172.21.15.89:0/527198660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942a9b40 0x5580942c3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f047ed9d640 1 -- 172.21.15.89:0/527198660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a85d0 msgr2=0x5580942a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f047ed9d640 1 --2- 172.21.15.89:0/527198660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a85d0 0x5580942a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.569+0000 7f047ed9d640 1 -- 172.21.15.89:0/527198660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580942a5960 con 0x5580942b3370 2024-09-15T23:25:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.570+0000 7f047ed9d640 1 --2- 172.21.15.89:0/527198660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942b3370 0x5580942ab120 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0478005a80 tx=0x7f047800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de6f89215f794a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.570+0000 7f0467fff640 1 -- 172.21.15.89:0/527198660 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478044070 con 0x5580942b3370 2024-09-15T23:25:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.570+0000 7f0467fff640 1 -- 172.21.15.89:0/527198660 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0478007630 con 0x5580942b3370 2024-09-15T23:25:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.570+0000 7f0467fff640 1 -- 172.21.15.89:0/527198660 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478034040 con 0x5580942b3370 2024-09-15T23:25:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.571+0000 7f0486f6f740 1 -- 172.21.15.89:0/527198660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942b3370 msgr2=0x5580942ab120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.571+0000 7f0486f6f740 1 --2- 172.21.15.89:0/527198660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942b3370 0x5580942ab120 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0478005a80 tx=0x7f047800f790 comp rx=0 tx=0).stop 2024-09-15T23:25:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.571+0000 7f0486f6f740 1 -- 172.21.15.89:0/527198660 shutdown_connections 2024-09-15T23:25:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.571+0000 7f0486f6f740 1 --2- 172.21.15.89:0/527198660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942b3370 0x5580942ab120 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.571+0000 7f0486f6f740 1 --2- 172.21.15.89:0/527198660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942a9b40 0x5580942c3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.571+0000 7f0486f6f740 1 --2- 172.21.15.89:0/527198660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942a85d0 0x5580942a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.571+0000 7f0486f6f740 1 -- 172.21.15.89:0/527198660 >> 172.21.15.89:0/527198660 conn(0x558094271310 msgr2=0x5580942a4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.571+0000 7f0486f6f740 1 -- 172.21.15.89:0/527198660 shutdown_connections 2024-09-15T23:25:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.571+0000 7f0486f6f740 1 -- 172.21.15.89:0/527198660 wait complete. 2024-09-15T23:25:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f0486f6f740 1 -- start start 2024-09-15T23:25:17.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f0486f6f740 1 Processor -- start 2024-09-15T23:25:17.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942ac840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942a9b40 0x5580942acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f0486f6f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942b3370 0x5580942ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f047e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f0486f6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0460011190 con 0x5580942a85d0 2024-09-15T23:25:17.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f0486f6f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0460010ac0 con 0x5580942b3370 2024-09-15T23:25:17.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f0486f6f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f046000dd00 con 0x5580942a9b40 2024-09-15T23:25:17.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f047e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942ac840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45986/0 (socket says 172.21.15.89:45986) 2024-09-15T23:25:17.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f047e59c640 1 -- 172.21.15.89:0/3279681244 learned_addr learned my addr 172.21.15.89:0/3279681244 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f0486f6f740 10 client.0 _ll_register_callbacks cb 0x7f0470098060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f047dd9b640 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942a9b40 0x5580942acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.572+0000 7f047ed9d640 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942b3370 0x5580942ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f047e59c640 1 -- 172.21.15.89:0/3279681244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942a9b40 msgr2=0x5580942acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f047e59c640 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942a9b40 0x5580942acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f047e59c640 1 -- 172.21.15.89:0/3279681244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942b3370 msgr2=0x5580942ad2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f047e59c640 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942b3370 0x5580942ad2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f047e59c640 1 -- 172.21.15.89:0/3279681244 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580942be7d0 con 0x5580942a85d0 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f047e59c640 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942ac840 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0470098220 tx=0x7f0470018cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e360e95c52e9fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f0465ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f0465ffb640 1 -- 172.21.15.89:0/3279681244 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047009bdc0 con 0x5580942a85d0 2024-09-15T23:25:17.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f0465ffb640 1 -- 172.21.15.89:0/3279681244 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f047009c020 con 0x5580942a85d0 2024-09-15T23:25:17.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f0486f6f740 1 -- 172.21.15.89:0/3279681244 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0478007630 con 0x5580942a85d0 2024-09-15T23:25:17.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f0465ffb640 1 -- 172.21.15.89:0/3279681244 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f04700a0170 con 0x5580942a85d0 2024-09-15T23:25:17.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f0486f6f740 10 client.15732 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:17.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.573+0000 7f0486f6f740 1 -- 172.21.15.89:0/3279681244 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5580942bfb10 con 0x5580942a85d0 2024-09-15T23:25:17.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.574+0000 7f0465ffb640 1 -- 172.21.15.89:0/3279681244 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0478007630 con 0x5580942a85d0 2024-09-15T23:25:17.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.574+0000 7f0465ffb640 1 -- 172.21.15.89:0/3279681244 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0478017600 con 0x5580942a85d0 2024-09-15T23:25:17.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.574+0000 7f0486f6f740 10 client.15732 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:17.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.574+0000 7f0486f6f740 10 client.15732 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:25:17.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.574+0000 7f0486f6f740 20 client.15732 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:17.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.574+0000 7f0486f6f740 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5580942c0190 0x558094338550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.574+0000 7f0486f6f740 4 client.15732 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.574+0000 7f0486f6f740 1 -- 172.21.15.89:0/3279681244 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55809421b860 con 0x5580942c0190 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.575+0000 7f047dd9b640 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5580942c0190 0x558094338550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.575+0000 7f047dd9b640 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5580942c0190 0x558094338550 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1442f33a1dd15333 server_cookie=cca980c6e5e870f2 in_seq=0 out_seq=0 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.575+0000 7f0465ffb640 10 client.15732 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.575+0000 7f0465ffb640 1 -- 172.21.15.89:0/3279681244 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55809421b860 con 0x5580942c0190 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.575+0000 7f0465ffb640 10 client.15732 handle_command_reply: tid=0 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.575+0000 7f0486f6f740 1 client.15732 shutdown 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 1 -- 172.21.15.89:0/3279681244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 msgr2=0x5580942ac840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942ac840 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0470098220 tx=0x7f0470018cb0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 1 -- 172.21.15.89:0/3279681244 shutdown_connections 2024-09-15T23:25:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5580942c0190 0x558094338550 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580942b3370 0x5580942ad2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580942a9b40 0x5580942acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 1 --2- 172.21.15.89:0/3279681244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580942a85d0 0x5580942ac840 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 1 -- 172.21.15.89:0/3279681244 >> 172.21.15.89:0/3279681244 conn(0x558094271310 msgr2=0x5580942b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0465ffb640 0 client.15732 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 1 -- 172.21.15.89:0/3279681244 shutdown_connections 2024-09-15T23:25:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 1 -- 172.21.15.89:0/3279681244 wait complete. 2024-09-15T23:25:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.576+0000 7f0486f6f740 20 client.15732 trim_cache size 0 max 16384 2024-09-15T23:25:17.587 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:17.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 -- 172.21.15.89:0/1194118530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04680a32d0 msgr2=0x7f04680a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04680a32d0 0x7f04680a5790 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f047801a040 tx=0x7f0478017c40 comp rx=0 tx=0).stop 2024-09-15T23:25:17.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 -- 172.21.15.89:0/1194118530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0480100e20 msgr2=0x7f0480197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0480100e20 0x7f0480197390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f047000c970 tx=0x7f047000ce30 comp rx=0 tx=0).stop 2024-09-15T23:25:17.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 -- 172.21.15.89:0/1194118530 shutdown_connections 2024-09-15T23:25:17.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04680a32d0 0x7f04680a5790 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0480102ef0 0x7f048019bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0480102570 0x7f04801978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 --2- 172.21.15.89:0/1194118530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0480100e20 0x7f0480197390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 -- 172.21.15.89:0/1194118530 >> 172.21.15.89:0/1194118530 conn(0x7f04800fc7d0 msgr2=0x7f04801068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:17.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 -- 172.21.15.89:0/1194118530 shutdown_connections 2024-09-15T23:25:17.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:17.577+0000 7f0484b96640 1 -- 172.21.15.89:0/1194118530 wait complete. 2024-09-15T23:25:18.327 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:25:18.610 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:19.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.134+0000 7f255a933640 1 -- 172.21.15.89:0/1101385239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2554101e10 msgr2=0x7f25541021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:19.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.134+0000 7f255a933640 1 --2- 172.21.15.89:0/1101385239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2554101e10 0x7f25541021f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2544009900 tx=0x7f254402a410 comp rx=0 tx=0).stop 2024-09-15T23:25:19.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.134+0000 7f255a933640 1 -- 172.21.15.89:0/1101385239 shutdown_connections 2024-09-15T23:25:19.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.134+0000 7f255a933640 1 --2- 172.21.15.89:0/1101385239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2554106f40 0x7f255410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.134+0000 7f255a933640 1 --2- 172.21.15.89:0/1101385239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25541027c0 0x7f2554106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.134+0000 7f255a933640 1 --2- 172.21.15.89:0/1101385239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2554101e10 0x7f25541021f0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.134+0000 7f255a933640 1 -- 172.21.15.89:0/1101385239 >> 172.21.15.89:0/1101385239 conn(0x7f25540fd8d0 msgr2=0x7f25540ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:19.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.135+0000 7f255a933640 1 -- 172.21.15.89:0/1101385239 shutdown_connections 2024-09-15T23:25:19.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.135+0000 7f255a933640 1 -- 172.21.15.89:0/1101385239 wait complete. 2024-09-15T23:25:19.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.135+0000 7f255a933640 1 Processor -- start 2024-09-15T23:25:19.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.135+0000 7f255a933640 1 -- start start 2024-09-15T23:25:19.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f255a933640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2554101e10 0x7f255419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f255a933640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25541027c0 0x7f255419ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f255a933640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2554106f40 0x7f255419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f255a933640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2554111f60 con 0x7f2554106f40 2024-09-15T23:25:19.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f255a933640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2554111de0 con 0x7f2554101e10 2024-09-15T23:25:19.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f255a933640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25541120e0 con 0x7f25541027c0 2024-09-15T23:25:19.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f2558ed0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2554106f40 0x7f255419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f25537fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25541027c0 0x7f255419ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f25537fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25541027c0 0x7f255419ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:34200/0 (socket says 172.21.15.89:34200) 2024-09-15T23:25:19.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f25537fe640 1 -- 172.21.15.89:0/1759638340 learned_addr learned my addr 172.21.15.89:0/1759638340 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:19.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f2553fff640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2554101e10 0x7f255419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f25537fe640 1 -- 172.21.15.89:0/1759638340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2554101e10 msgr2=0x7f255419a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:19.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f25537fe640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2554101e10 0x7f255419a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f25537fe640 1 -- 172.21.15.89:0/1759638340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2554106f40 msgr2=0x7f255419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:19.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f25537fe640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2554106f40 0x7f255419ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.136+0000 7f25537fe640 1 -- 172.21.15.89:0/1759638340 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255419f670 con 0x7f25541027c0 2024-09-15T23:25:19.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.137+0000 7f25537fe640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25541027c0 0x7f255419ac00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f254800d950 tx=0x7f254800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:19.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.137+0000 7f25517fa640 1 -- 172.21.15.89:0/1759638340 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2548017030 con 0x7f25541027c0 2024-09-15T23:25:19.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.137+0000 7f25517fa640 1 -- 172.21.15.89:0/1759638340 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2548004020 con 0x7f25541027c0 2024-09-15T23:25:19.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.137+0000 7f255a933640 1 -- 172.21.15.89:0/1759638340 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f255419f960 con 0x7f25541027c0 2024-09-15T23:25:19.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.137+0000 7f25517fa640 1 -- 172.21.15.89:0/1759638340 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25480047a0 con 0x7f25541027c0 2024-09-15T23:25:19.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.137+0000 7f255a933640 1 -- 172.21.15.89:0/1759638340 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25541a7240 con 0x7f25541027c0 2024-09-15T23:25:19.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f255cd0c740 1 Processor -- start 2024-09-15T23:25:19.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f255cd0c740 1 -- start start 2024-09-15T23:25:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe7b40 0x558c8dfe6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe65d0 0x558c8dfe6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dff1370 0x558c8dff3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f255cd0c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558c8df4c2f0 con 0x558c8dfe65d0 2024-09-15T23:25:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f255cd0c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558c8def3cf0 con 0x558c8dfe7b40 2024-09-15T23:25:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f255cd0c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558c8df482b0 con 0x558c8dff1370 2024-09-15T23:25:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f2558ed0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe7b40 0x558c8dfe6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f2558ed0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe7b40 0x558c8dfe6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:41936/0 (socket says 172.21.15.89:41936) 2024-09-15T23:25:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.140+0000 7f2558ed0640 1 -- 172.21.15.89:0/2439591116 learned_addr learned my addr 172.21.15.89:0/2439591116 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f2558ed0640 1 -- 172.21.15.89:0/2439591116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dff1370 msgr2=0x558c8dff3760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:25:19.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f2553fff640 1 --2- 172.21.15.89:0/2439591116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe65d0 0x558c8dfe6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f2558ed0640 1 --2- 172.21.15.89:0/2439591116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dff1370 0x558c8dff3760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f25517fa640 1 -- 172.21.15.89:0/1759638340 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f25480041d0 con 0x7f25541027c0 2024-09-15T23:25:19.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f2558ed0640 1 -- 172.21.15.89:0/2439591116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe65d0 msgr2=0x558c8dfe6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:19.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f2558ed0640 1 --2- 172.21.15.89:0/2439591116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe65d0 0x558c8dfe6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f2558ed0640 1 -- 172.21.15.89:0/2439591116 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c8ded83a0 con 0x558c8dfe7b40 2024-09-15T23:25:19.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f2553fff640 1 --2- 172.21.15.89:0/2439591116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe65d0 0x558c8dfe6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:19.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f2558ed0640 1 --2- 172.21.15.89:0/2439591116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe7b40 0x558c8dfe6090 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f254c009e30 tx=0x7f254c00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e43494e2df8ca13 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:19.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f253a7fc640 1 -- 172.21.15.89:0/2439591116 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c011640 con 0x558c8dfe7b40 2024-09-15T23:25:19.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f253a7fc640 1 -- 172.21.15.89:0/2439591116 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f254c011cc0 con 0x558c8dfe7b40 2024-09-15T23:25:19.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.141+0000 7f253a7fc640 1 -- 172.21.15.89:0/2439591116 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c01bd70 con 0x558c8dfe7b40 2024-09-15T23:25:19.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.142+0000 7f255cd0c740 1 -- 172.21.15.89:0/2439591116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe7b40 msgr2=0x558c8dfe6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:19.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.142+0000 7f255cd0c740 1 --2- 172.21.15.89:0/2439591116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe7b40 0x558c8dfe6090 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f254c009e30 tx=0x7f254c00f730 comp rx=0 tx=0).stop 2024-09-15T23:25:19.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.142+0000 7f255cd0c740 1 -- 172.21.15.89:0/2439591116 shutdown_connections 2024-09-15T23:25:19.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.142+0000 7f255cd0c740 1 --2- 172.21.15.89:0/2439591116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dff1370 0x558c8dff3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.142+0000 7f255cd0c740 1 --2- 172.21.15.89:0/2439591116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe65d0 0x558c8dfe6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.142+0000 7f255cd0c740 1 --2- 172.21.15.89:0/2439591116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe7b40 0x558c8dfe6090 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.142+0000 7f255cd0c740 1 -- 172.21.15.89:0/2439591116 >> 172.21.15.89:0/2439591116 conn(0x558c8dfaf310 msgr2=0x558c8dfe2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.142+0000 7f255cd0c740 1 -- 172.21.15.89:0/2439591116 shutdown_connections 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.142+0000 7f255cd0c740 1 -- 172.21.15.89:0/2439591116 wait complete. 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f255cd0c740 1 -- start start 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f255cd0c740 1 Processor -- start 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe65d0 0x558c8dfee0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe7b40 0x558c8dfee630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dff1370 0x558c8dfeeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f255cd0c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558c8dff4070 con 0x558c8dfe7b40 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f255cd0c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558c8df07060 con 0x558c8dff1370 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f255cd0c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558c8dff41f0 con 0x558c8dfe65d0 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f255cd0c740 10 client.0 _ll_register_callbacks cb 0x558c8dc5a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f2553fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe7b40 0x558c8dfee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f2553fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe7b40 0x558c8dfee630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46002/0 (socket says 172.21.15.89:46002) 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f2553fff640 1 -- 172.21.15.89:0/1397681516 learned_addr learned my addr 172.21.15.89:0/1397681516 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.143+0000 7f2558ed0640 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe65d0 0x558c8dfee0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f2553fff640 1 -- 172.21.15.89:0/1397681516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe65d0 msgr2=0x558c8dfee0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f2553fff640 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe65d0 0x558c8dfee0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f2553fff640 1 -- 172.21.15.89:0/1397681516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dff1370 msgr2=0x558c8dfeeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f2553fff640 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dff1370 0x558c8dfeeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f2553fff640 1 -- 172.21.15.89:0/1397681516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c8dcbd3c0 con 0x558c8dfe7b40 2024-09-15T23:25:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f2553fff640 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe7b40 0x558c8dfee630 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f254402a930 tx=0x7f25440317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc7353b37d5a381 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f25337fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f25517fa640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f253c0a32e0 0x7f253c0a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f25337fe640 1 -- 172.21.15.89:0/1397681516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544040070 con 0x558c8dfe7b40 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f2558ed0640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f253c0a32e0 0x7f253c0a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f255cd0c740 1 -- 172.21.15.89:0/1397681516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f254c011cc0 con 0x558c8dfe7b40 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f25517fa640 1 -- 172.21.15.89:0/1759638340 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f25480cf890 con 0x7f25541027c0 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.144+0000 7f25337fe640 1 -- 172.21.15.89:0/1397681516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2544004450 con 0x558c8dfe7b40 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.145+0000 7f25337fe640 1 -- 172.21.15.89:0/1397681516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2544035040 con 0x558c8dfe7b40 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.145+0000 7f255cd0c740 10 client.15738 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.145+0000 7f255cd0c740 1 -- 172.21.15.89:0/1397681516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558c8e0083d0 con 0x558c8dfe7b40 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.145+0000 7f2558ed0640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f253c0a32e0 0x7f253c0a57a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f254c0094d0 tx=0x7f254c01b950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.145+0000 7f25337fe640 1 -- 172.21.15.89:0/1397681516 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f254c011cc0 con 0x558c8dfe7b40 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.145+0000 7f25337fe640 1 -- 172.21.15.89:0/1397681516 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2544045750 con 0x558c8dfe7b40 2024-09-15T23:25:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.146+0000 7f255cd0c740 10 client.15738 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.146+0000 7f255cd0c740 10 client.15738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.146+0000 7f255cd0c740 20 client.15738 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.146+0000 7f255cd0c740 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c8e0097e0 0x558c8e00bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.146+0000 7f255cd0c740 4 client.15738 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.146+0000 7f255cd0c740 1 -- 172.21.15.89:0/1397681516 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c8df59860 con 0x558c8e0097e0 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.146+0000 7f25537fe640 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c8e0097e0 0x558c8e00bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.147+0000 7f25337fe640 10 client.15738 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.147+0000 7f25537fe640 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c8e0097e0 0x558c8e00bc00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9776bda68b616c89 server_cookie=f9dc56bf7a18e449 in_seq=0 out_seq=0 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.147+0000 7f25337fe640 1 -- 172.21.15.89:0/1397681516 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558c8df59860 con 0x558c8e0097e0 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.147+0000 7f25337fe640 10 client.15738 handle_command_reply: tid=0 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 client.15738 shutdown 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 -- 172.21.15.89:0/1397681516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe7b40 msgr2=0x558c8dfee630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe7b40 0x558c8dfee630 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f254402a930 tx=0x7f25440317e0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 -- 172.21.15.89:0/1397681516 shutdown_connections 2024-09-15T23:25:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c8e0097e0 0x558c8e00bc00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f25337fe640 0 client.15738 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dff1370 0x558c8dfeeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dfe7b40 0x558c8dfee630 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 --2- 172.21.15.89:0/1397681516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe65d0 0x558c8dfee0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 -- 172.21.15.89:0/1397681516 >> 172.21.15.89:0/1397681516 conn(0x558c8dfaf310 msgr2=0x558c8dfe8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 -- 172.21.15.89:0/1397681516 shutdown_connections 2024-09-15T23:25:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 1 -- 172.21.15.89:0/1397681516 wait complete. 2024-09-15T23:25:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.148+0000 7f255cd0c740 20 client.15738 trim_cache size 0 max 16384 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.161+0000 7f255cd0c740 1 Processor -- start 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.161+0000 7f255cd0c740 1 -- start start 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.161+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe65d0 0x558c8dfee9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.161+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 0x558c8e0114b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.161+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e0119f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.161+0000 7f255cd0c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f25541acd10 con 0x558c8dff1370 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.161+0000 7f255cd0c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f25541ae850 con 0x558c8dfe65d0 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.161+0000 7f255cd0c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25541ad3e0 con 0x558c8dfe7b40 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f2558ed0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e0119f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f2558ed0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e0119f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46006/0 (socket says 172.21.15.89:46006) 2024-09-15T23:25:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f2558ed0640 1 -- 172.21.15.89:0/3580218752 learned_addr learned my addr 172.21.15.89:0/3580218752 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f2553fff640 1 --2- 172.21.15.89:0/3580218752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe65d0 0x558c8dfee9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f25537fe640 1 --2- 172.21.15.89:0/3580218752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 0x558c8e0114b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f2558ed0640 1 -- 172.21.15.89:0/3580218752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 msgr2=0x558c8e0114b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f2558ed0640 1 --2- 172.21.15.89:0/3580218752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 0x558c8e0114b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f2558ed0640 1 -- 172.21.15.89:0/3580218752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe65d0 msgr2=0x558c8dfee9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f2558ed0640 1 --2- 172.21.15.89:0/3580218752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe65d0 0x558c8dfee9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f2558ed0640 1 -- 172.21.15.89:0/3580218752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c8dfe9120 con 0x558c8dff1370 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.162+0000 7f2558ed0640 1 --2- 172.21.15.89:0/3580218752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e0119f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f254c009c70 tx=0x7f254c026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2240a9617d5e811e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.163+0000 7f2539ffb640 1 -- 172.21.15.89:0/3580218752 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c011440 con 0x558c8dff1370 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.163+0000 7f2539ffb640 1 -- 172.21.15.89:0/3580218752 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f254c011ac0 con 0x558c8dff1370 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.163+0000 7f2539ffb640 1 -- 172.21.15.89:0/3580218752 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c033bb0 con 0x558c8dff1370 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.163+0000 7f255cd0c740 1 -- 172.21.15.89:0/3580218752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 msgr2=0x558c8e0119f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.163+0000 7f255cd0c740 1 --2- 172.21.15.89:0/3580218752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e0119f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f254c009c70 tx=0x7f254c026000 comp rx=0 tx=0).stop 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.163+0000 7f255cd0c740 1 -- 172.21.15.89:0/3580218752 shutdown_connections 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.163+0000 7f255cd0c740 1 --2- 172.21.15.89:0/3580218752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e0119f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.163+0000 7f255cd0c740 1 --2- 172.21.15.89:0/3580218752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 0x558c8e0114b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.163+0000 7f255cd0c740 1 --2- 172.21.15.89:0/3580218752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe65d0 0x558c8dfee9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.163+0000 7f255cd0c740 1 -- 172.21.15.89:0/3580218752 >> 172.21.15.89:0/3580218752 conn(0x558c8dfaf310 msgr2=0x558c8dff0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.164+0000 7f255cd0c740 1 -- 172.21.15.89:0/3580218752 shutdown_connections 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.164+0000 7f255cd0c740 1 -- 172.21.15.89:0/3580218752 wait complete. 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.164+0000 7f255cd0c740 1 -- start start 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.164+0000 7f255cd0c740 1 Processor -- start 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.164+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe65d0 0x558c8dffde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.164+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 0x558c8dffe370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.164+0000 7f255cd0c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e073480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.164+0000 7f255cd0c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f25541ae180 con 0x558c8dff1370 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.164+0000 7f255cd0c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f25541adab0 con 0x558c8dfe65d0 2024-09-15T23:25:19.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.164+0000 7f255cd0c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25541aacf0 con 0x558c8dfe7b40 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f255cd0c740 10 client.0 _ll_register_callbacks cb 0x7f2544009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f2558ed0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f2558ed0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e073480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46016/0 (socket says 172.21.15.89:46016) 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f2558ed0640 1 -- 172.21.15.89:0/2790826658 learned_addr learned my addr 172.21.15.89:0/2790826658 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f25537fe640 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 0x558c8dffe370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f2553fff640 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe65d0 0x558c8dffde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f2558ed0640 1 -- 172.21.15.89:0/2790826658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 msgr2=0x558c8dffe370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f2558ed0640 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 0x558c8dffe370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f2558ed0640 1 -- 172.21.15.89:0/2790826658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe65d0 msgr2=0x558c8dffde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f2558ed0640 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe65d0 0x558c8dffde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f2558ed0640 1 -- 172.21.15.89:0/2790826658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c8dffec50 con 0x558c8dff1370 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f25537fe640 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 0x558c8dffe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:19.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f2558ed0640 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e073480 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f254c01bfd0 tx=0x7f254c00ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1891309465d46347 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.165+0000 7f25337fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.166+0000 7f25337fe640 1 -- 172.21.15.89:0/2790826658 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254c011c70 con 0x558c8dff1370 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.166+0000 7f25337fe640 1 -- 172.21.15.89:0/2790826658 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f254c011650 con 0x558c8dff1370 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.166+0000 7f255cd0c740 1 -- 172.21.15.89:0/2790826658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f254c011ac0 con 0x558c8dff1370 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.166+0000 7f25337fe640 1 -- 172.21.15.89:0/2790826658 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f254c011850 con 0x558c8dff1370 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.166+0000 7f255cd0c740 10 client.15744 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.166+0000 7f255cd0c740 1 -- 172.21.15.89:0/2790826658 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558c8e074ad0 con 0x558c8dff1370 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.166+0000 7f25337fe640 1 -- 172.21.15.89:0/2790826658 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f254c011ac0 con 0x558c8dff1370 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.167+0000 7f25337fe640 1 -- 172.21.15.89:0/2790826658 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f254c03b030 con 0x558c8dff1370 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.167+0000 7f255cd0c740 10 client.15744 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.167+0000 7f255cd0c740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.167+0000 7f255cd0c740 20 client.15744 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.167+0000 7f255cd0c740 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c8e074f40 0x558c8e077330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:19.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.167+0000 7f255cd0c740 4 client.15744 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.167+0000 7f255cd0c740 1 -- 172.21.15.89:0/2790826658 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c8df53d70 con 0x558c8e074f40 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.167+0000 7f2553fff640 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c8e074f40 0x558c8e077330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.167+0000 7f25337fe640 10 client.15744 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.167+0000 7f2553fff640 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c8e074f40 0x558c8e077330 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71ca97c2903a22b0 server_cookie=b9b0b47819709fff in_seq=0 out_seq=0 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f25337fe640 1 -- 172.21.15.89:0/2790826658 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c8df53d70 con 0x558c8e074f40 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f25337fe640 10 client.15744 handle_command_reply: tid=0 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 client.15744 shutdown 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 -- 172.21.15.89:0/2790826658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 msgr2=0x558c8e073480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e073480 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f254c01bfd0 tx=0x7f254c00ffd0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 -- 172.21.15.89:0/2790826658 shutdown_connections 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c8e074f40 0x558c8e077330 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c8dff1370 0x558c8e073480 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c8dfe7b40 0x558c8dffe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 --2- 172.21.15.89:0/2790826658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c8dfe65d0 0x558c8dffde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 -- 172.21.15.89:0/2790826658 >> 172.21.15.89:0/2790826658 conn(0x558c8dfaf310 msgr2=0x558c8dfe14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f25337fe640 0 client.15744 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 -- 172.21.15.89:0/2790826658 shutdown_connections 2024-09-15T23:25:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.168+0000 7f255cd0c740 1 -- 172.21.15.89:0/2790826658 wait complete. 2024-09-15T23:25:19.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.169+0000 7f255cd0c740 20 client.15744 trim_cache size 0 max 16384 2024-09-15T23:25:19.183 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.169+0000 7f255a933640 1 -- 172.21.15.89:0/1759638340 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f253c0a32e0 msgr2=0x7f253c0a57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.169+0000 7f255a933640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f253c0a32e0 0x7f253c0a57a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f254c0094d0 tx=0x7f254c01b950 comp rx=0 tx=0).stop 2024-09-15T23:25:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.169+0000 7f255a933640 1 -- 172.21.15.89:0/1759638340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25541027c0 msgr2=0x7f255419ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.169+0000 7f255a933640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25541027c0 0x7f255419ac00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f254800d950 tx=0x7f254800de10 comp rx=0 tx=0).stop 2024-09-15T23:25:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.170+0000 7f255a933640 1 -- 172.21.15.89:0/1759638340 shutdown_connections 2024-09-15T23:25:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.170+0000 7f255a933640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f253c0a32e0 0x7f253c0a57a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.170+0000 7f255a933640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2554106f40 0x7f255419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.170+0000 7f255a933640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25541027c0 0x7f255419ac00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.170+0000 7f255a933640 1 --2- 172.21.15.89:0/1759638340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2554101e10 0x7f255419a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.170+0000 7f255a933640 1 -- 172.21.15.89:0/1759638340 >> 172.21.15.89:0/1759638340 conn(0x7f25540fd8d0 msgr2=0x7f25540fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.170+0000 7f255a933640 1 -- 172.21.15.89:0/1759638340 shutdown_connections 2024-09-15T23:25:19.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:19.170+0000 7f255a933640 1 -- 172.21.15.89:0/1759638340 wait complete. 2024-09-15T23:25:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:20 smithi120 ceph-mon[35250]: pgmap v389: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-15T23:25:20.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:20 smithi174 ceph-mon[35586]: pgmap v389: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-15T23:25:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.539+0000 7f7551cd7640 1 -- 172.21.15.89:0/3241280997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f754c10bbb0 msgr2=0x7f754c10dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.539+0000 7f7551cd7640 1 --2- 172.21.15.89:0/3241280997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f754c10bbb0 0x7f754c10dfa0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f754000aa10 tx=0x7f754002b720 comp rx=0 tx=0).stop 2024-09-15T23:25:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.539+0000 7f7551cd7640 1 -- 172.21.15.89:0/3241280997 shutdown_connections 2024-09-15T23:25:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.539+0000 7f7551cd7640 1 --2- 172.21.15.89:0/3241280997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f754c10e640 0x7f754c110b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.539+0000 7f7551cd7640 1 --2- 172.21.15.89:0/3241280997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f754c10bbb0 0x7f754c10dfa0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.539+0000 7f7551cd7640 1 --2- 172.21.15.89:0/3241280997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f754c1091f0 0x7f754c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.539+0000 7f7551cd7640 1 -- 172.21.15.89:0/3241280997 >> 172.21.15.89:0/3241280997 conn(0x7f754c06ece0 msgr2=0x7f754c071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.539+0000 7f7551cd7640 1 -- 172.21.15.89:0/3241280997 shutdown_connections 2024-09-15T23:25:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.539+0000 7f7551cd7640 1 -- 172.21.15.89:0/3241280997 wait complete. 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.540+0000 7f7551cd7640 1 Processor -- start 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.540+0000 7f7551cd7640 1 -- start start 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.540+0000 7f7551cd7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f754c1091f0 0x7f754c1a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.540+0000 7f7551cd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f754c10e640 0x7f754c1a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.540+0000 7f7551cd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f754c1a7fb0 0x7f754c1aa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.540+0000 7f7551cd7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f754c113900 con 0x7f754c1a7fb0 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.540+0000 7f7551cd7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f754c113780 con 0x7f754c1091f0 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.540+0000 7f7551cd7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f754c113a80 con 0x7f754c10e640 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f754c1a7fb0 0x7f754c1aa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f754c1a7fb0 0x7f754c1aa3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46046/0 (socket says 172.21.15.89:46046) 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754bfff640 1 -- 172.21.15.89:0/3856896870 learned_addr learned my addr 172.21.15.89:0/3856896870 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754affd640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f754c10e640 0x7f754c1a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754b7fe640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f754c1091f0 0x7f754c1a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754affd640 1 -- 172.21.15.89:0/3856896870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f754c1091f0 msgr2=0x7f754c1a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754affd640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f754c1091f0 0x7f754c1a2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754affd640 1 -- 172.21.15.89:0/3856896870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f754c1a7fb0 msgr2=0x7f754c1aa3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754affd640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f754c1a7fb0 0x7f754c1aa3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754affd640 1 -- 172.21.15.89:0/3856896870 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754c1aaa10 con 0x7f754c10e640 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754bfff640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f754c1a7fb0 0x7f754c1aa3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754b7fe640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f754c1091f0 0x7f754c1a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.541+0000 7f754affd640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f754c10e640 0x7f754c1a2f50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f754c073c60 tx=0x7f7540030de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.542+0000 7f7548ff9640 1 -- 172.21.15.89:0/3856896870 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754003f020 con 0x7f754c10e640 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.542+0000 7f7551cd7640 1 -- 172.21.15.89:0/3856896870 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754c1aaca0 con 0x7f754c10e640 2024-09-15T23:25:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.542+0000 7f7548ff9640 1 -- 172.21.15.89:0/3856896870 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7540032400 con 0x7f754c10e640 2024-09-15T23:25:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.542+0000 7f7548ff9640 1 -- 172.21.15.89:0/3856896870 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7540036040 con 0x7f754c10e640 2024-09-15T23:25:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.542+0000 7f7551cd7640 1 -- 172.21.15.89:0/3856896870 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f754c1ab160 con 0x7f754c10e640 2024-09-15T23:25:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f7554a72740 1 Processor -- start 2024-09-15T23:25:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f7548ff9640 1 -- 172.21.15.89:0/3856896870 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f7540042020 con 0x7f754c10e640 2024-09-15T23:25:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f7554a72740 1 -- start start 2024-09-15T23:25:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f7554a72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9b8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f7554a72740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9bdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f7554a72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 0x56443a9c0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f7554a72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56443a91c2f0 con 0x56443a9b89a0 2024-09-15T23:25:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f7554a72740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56443a8c4270 con 0x56443a9be370 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f7554a72740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56443a9182b0 con 0x56443a9b92e0 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f754bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9b8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.546+0000 7f754bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9b8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46052/0 (socket says 172.21.15.89:46052) 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.547+0000 7f754bfff640 1 -- 172.21.15.89:0/189490014 learned_addr learned my addr 172.21.15.89:0/189490014 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.547+0000 7f754b7fe640 1 --2- 172.21.15.89:0/189490014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9bdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.547+0000 7f754affd640 1 --2- 172.21.15.89:0/189490014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 0x56443a9c0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.547+0000 7f754bfff640 1 -- 172.21.15.89:0/189490014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 msgr2=0x56443a9bdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.547+0000 7f754bfff640 1 --2- 172.21.15.89:0/189490014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9bdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.547+0000 7f754bfff640 1 -- 172.21.15.89:0/189490014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 msgr2=0x56443a9c0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.547+0000 7f7548ff9640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75340a3410 0x7f75340a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.548+0000 7f754bfff640 1 --2- 172.21.15.89:0/189490014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 0x56443a9c0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.548+0000 7f754bfff640 1 -- 172.21.15.89:0/189490014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56443a8c3340 con 0x56443a9b89a0 2024-09-15T23:25:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.548+0000 7f754affd640 1 --2- 172.21.15.89:0/189490014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 0x56443a9c0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.548+0000 7f754b7fe640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75340a3410 0x7f75340a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.548+0000 7f7548ff9640 1 -- 172.21.15.89:0/3856896870 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f75400f0390 con 0x7f754c10e640 2024-09-15T23:25:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.548+0000 7f754bfff640 1 --2- 172.21.15.89:0/189490014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9b8da0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f75440118b0 tx=0x7f7544011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db79bb78bc28b53 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.548+0000 7f754b7fe640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75340a3410 0x7f75340a58d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x56443a9b9d10 tx=0x7f753c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.548+0000 7f7531ffb640 1 -- 172.21.15.89:0/189490014 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754401e070 con 0x56443a9b89a0 2024-09-15T23:25:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.549+0000 7f7531ffb640 1 -- 172.21.15.89:0/189490014 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7544011310 con 0x56443a9b89a0 2024-09-15T23:25:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.549+0000 7f7531ffb640 1 -- 172.21.15.89:0/189490014 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544019e20 con 0x56443a9b89a0 2024-09-15T23:25:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.549+0000 7f7554a72740 1 -- 172.21.15.89:0/189490014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 msgr2=0x56443a9b8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.550+0000 7f7554a72740 1 --2- 172.21.15.89:0/189490014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9b8da0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f75440118b0 tx=0x7f7544011d70 comp rx=0 tx=0).stop 2024-09-15T23:25:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.550+0000 7f7554a72740 1 -- 172.21.15.89:0/189490014 shutdown_connections 2024-09-15T23:25:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.550+0000 7f7554a72740 1 --2- 172.21.15.89:0/189490014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 0x56443a9c0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.550+0000 7f7554a72740 1 --2- 172.21.15.89:0/189490014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9bdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.550+0000 7f7554a72740 1 --2- 172.21.15.89:0/189490014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9b8da0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.550+0000 7f7554a72740 1 -- 172.21.15.89:0/189490014 >> 172.21.15.89:0/189490014 conn(0x56443a9b4010 msgr2=0x56443a9b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.550+0000 7f7554a72740 1 -- 172.21.15.89:0/189490014 shutdown_connections 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.550+0000 7f7554a72740 1 -- 172.21.15.89:0/189490014 wait complete. 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.550+0000 7f7554a72740 1 -- start start 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.550+0000 7f7554a72740 1 Processor -- start 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f7554a72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f7554a72740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9c30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f7554a72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 0x56443a9ca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46068/0 (socket says 172.21.15.89:46068) 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f754b7fe640 1 -- 172.21.15.89:0/1175917873 learned_addr learned my addr 172.21.15.89:0/1175917873 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:20.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f7554a72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56443a9c1340 con 0x56443a9b89a0 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f754bfff640 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 0x56443a9ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f754affd640 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9c30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f7554a72740 1 -- 172.21.15.89:0/1175917873 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56443a8a5290 con 0x56443a9be370 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.551+0000 7f7554a72740 1 -- 172.21.15.89:0/1175917873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56443a9c14c0 con 0x56443a9b92e0 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f754affd640 1 -- 172.21.15.89:0/1175917873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 msgr2=0x56443a9ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f754affd640 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 0x56443a9ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f754affd640 1 -- 172.21.15.89:0/1175917873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 msgr2=0x56443a9c2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f7554a72740 10 client.0 _ll_register_callbacks cb 0x56443a8d6140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f754affd640 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f754affd640 1 -- 172.21.15.89:0/1175917873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56443a62f210 con 0x56443a9b92e0 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f754b7fe640 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f7554a72740 1 -- 172.21.15.89:0/1175917873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7544011310 con 0x56443a9b92e0 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f754affd640 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9c30d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f75400ed9f0 tx=0x7f7540030510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=864cadb08f0ddbd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f752affd640 10 client.25132 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f752affd640 1 -- 172.21.15.89:0/1175917873 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75400b87a0 con 0x56443a9b92e0 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f752affd640 1 -- 172.21.15.89:0/1175917873 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f75400325b0 con 0x56443a9b92e0 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f752affd640 1 -- 172.21.15.89:0/1175917873 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7540005d50 con 0x56443a9b92e0 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f7554a72740 10 client.25132 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.552+0000 7f7554a72740 1 -- 172.21.15.89:0/1175917873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=29}) -- 0x56443a6fb620 con 0x56443a9b92e0 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.553+0000 7f752affd640 1 -- 172.21.15.89:0/1175917873 <== mon.1 v2:172.21.15.174:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f754000b040 con 0x56443a9b92e0 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.553+0000 7f7554a72740 10 client.25132 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.553+0000 7f7554a72740 10 client.25132 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.553+0000 7f7554a72740 20 client.25132 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.553+0000 7f7554a72740 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56443a9ce690 0x56443a9d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.553+0000 7f7554a72740 4 client.25132 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.553+0000 7f7554a72740 1 -- 172.21.15.89:0/1175917873 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56443a929a20 con 0x56443a9ce690 2024-09-15T23:25:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.553+0000 7f754b7fe640 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56443a9ce690 0x56443a9d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.554+0000 7f752affd640 10 client.25132 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.554+0000 7f754b7fe640 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56443a9ce690 0x56443a9d0b00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b603c4968abc9f3 server_cookie=615b204645dcd257 in_seq=0 out_seq=0 2024-09-15T23:25:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.554+0000 7f752affd640 1 -- 172.21.15.89:0/1175917873 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56443a929a20 con 0x56443a9ce690 2024-09-15T23:25:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.554+0000 7f752affd640 10 client.25132 handle_command_reply: tid=0 2024-09-15T23:25:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.554+0000 7f7554a72740 1 client.25132 shutdown 2024-09-15T23:25:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 1 -- 172.21.15.89:0/1175917873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 msgr2=0x56443a9c30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9c30d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f75400ed9f0 tx=0x7f7540030510 comp rx=0 tx=0).stop 2024-09-15T23:25:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 1 -- 172.21.15.89:0/1175917873 shutdown_connections 2024-09-15T23:25:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56443a9ce690 0x56443a9d0b00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f752affd640 0 client.25132 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9be370 0x56443a9ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9c30d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 1 --2- 172.21.15.89:0/1175917873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 1 -- 172.21.15.89:0/1175917873 >> 172.21.15.89:0/1175917873 conn(0x56443a9b4010 msgr2=0x56443a9b5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 1 -- 172.21.15.89:0/1175917873 shutdown_connections 2024-09-15T23:25:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 1 -- 172.21.15.89:0/1175917873 wait complete. 2024-09-15T23:25:20.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.555+0000 7f7554a72740 20 client.25132 trim_cache size 0 max 16384 2024-09-15T23:25:20.565 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:20 smithi089 ceph-mon[30933]: pgmap v389: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-15T23:25:20.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.582+0000 7f7554a72740 1 Processor -- start 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.582+0000 7f7554a72740 1 -- start start 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.582+0000 7f7554a72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b89a0 0x56443a9b5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.582+0000 7f7554a72740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.582+0000 7f7554a72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9be370 0x56443a9b5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.582+0000 7f7554a72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b5840 con 0x56443a9be370 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.582+0000 7f754bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9be370 0x56443a9b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b89a0 0x56443a9b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9be370 0x56443a9b5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46082/0 (socket says 172.21.15.89:46082) 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754bfff640 1 -- 172.21.15.89:0/1353549275 learned_addr learned my addr 172.21.15.89:0/1353549275 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754affd640 1 --2- 172.21.15.89:0/1353549275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f7554a72740 1 -- 172.21.15.89:0/1353549275 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b7380 con 0x56443a9b89a0 2024-09-15T23:25:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f7554a72740 1 -- 172.21.15.89:0/1353549275 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b5f10 con 0x56443a9b92e0 2024-09-15T23:25:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754bfff640 1 -- 172.21.15.89:0/1353549275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 msgr2=0x56443a9ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754bfff640 1 --2- 172.21.15.89:0/1353549275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754bfff640 1 -- 172.21.15.89:0/1353549275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b89a0 msgr2=0x56443a9b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754bfff640 1 --2- 172.21.15.89:0/1353549275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b89a0 0x56443a9b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754bfff640 1 -- 172.21.15.89:0/1353549275 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56443a9c1ed0 con 0x56443a9be370 2024-09-15T23:25:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754affd640 1 --2- 172.21.15.89:0/1353549275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754b7fe640 1 --2- 172.21.15.89:0/1353549275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b89a0 0x56443a9b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.583+0000 7f754bfff640 1 --2- 172.21.15.89:0/1353549275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9be370 0x56443a9b5940 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7544009aa0 tx=0x7f7544011110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73a8a1dcb9cc458 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.584+0000 7f75317fa640 1 -- 172.21.15.89:0/1353549275 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754401e040 con 0x56443a9be370 2024-09-15T23:25:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.584+0000 7f75317fa640 1 -- 172.21.15.89:0/1353549275 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f754401f040 con 0x56443a9be370 2024-09-15T23:25:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.584+0000 7f75317fa640 1 -- 172.21.15.89:0/1353549275 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754400fc20 con 0x56443a9be370 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.585+0000 7f7554a72740 1 -- 172.21.15.89:0/1353549275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9be370 msgr2=0x56443a9b5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.585+0000 7f7554a72740 1 --2- 172.21.15.89:0/1353549275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9be370 0x56443a9b5940 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7544009aa0 tx=0x7f7544011110 comp rx=0 tx=0).stop 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.585+0000 7f7554a72740 1 -- 172.21.15.89:0/1353549275 shutdown_connections 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.585+0000 7f7554a72740 1 --2- 172.21.15.89:0/1353549275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9be370 0x56443a9b5940 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.585+0000 7f7554a72740 1 --2- 172.21.15.89:0/1353549275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9b92e0 0x56443a9ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.585+0000 7f7554a72740 1 --2- 172.21.15.89:0/1353549275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b89a0 0x56443a9b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.585+0000 7f7554a72740 1 -- 172.21.15.89:0/1353549275 >> 172.21.15.89:0/1353549275 conn(0x56443a9b4010 msgr2=0x56443a9b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.585+0000 7f7554a72740 1 -- 172.21.15.89:0/1353549275 shutdown_connections 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.585+0000 7f7554a72740 1 -- 172.21.15.89:0/1353549275 wait complete. 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.586+0000 7f7554a72740 1 -- start start 2024-09-15T23:25:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.586+0000 7f7554a72740 1 Processor -- start 2024-09-15T23:25:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.586+0000 7f7554a72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f7554a72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b92e0 0x56443a9c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46096/0 (socket says 172.21.15.89:46096) 2024-09-15T23:25:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f754b7fe640 1 -- 172.21.15.89:0/3718518461 learned_addr learned my addr 172.21.15.89:0/3718518461 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:20.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f7554a72740 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9be370 0x56443a9c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b6cb0 con 0x56443a9b89a0 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b65e0 con 0x56443a9b92e0 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f754c1b3820 con 0x56443a9be370 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f754bfff640 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9be370 0x56443a9c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f754affd640 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b92e0 0x56443a9c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f7554a72740 10 client.0 _ll_register_callbacks cb 0x7f7540005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f754b7fe640 1 -- 172.21.15.89:0/3718518461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9be370 msgr2=0x56443a9c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f754b7fe640 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9be370 0x56443a9c8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f754b7fe640 1 -- 172.21.15.89:0/3718518461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b92e0 msgr2=0x56443a9c7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f754b7fe640 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b92e0 0x56443a9c7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.587+0000 7f754b7fe640 1 -- 172.21.15.89:0/3718518461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56443a9c91d0 con 0x56443a9b89a0 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.588+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f754401f040 con 0x56443a9b89a0 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.588+0000 7f754b7fe640 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c7670 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f753c012ea0 tx=0x7f753c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4e6c19ae000310 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.588+0000 7f752affd640 10 client.15762 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.588+0000 7f752affd640 1 -- 172.21.15.89:0/3718518461 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c00f6e0 con 0x56443a9b89a0 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.588+0000 7f752affd640 1 -- 172.21.15.89:0/3718518461 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f753c00f880 con 0x56443a9b89a0 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.588+0000 7f752affd640 1 -- 172.21.15.89:0/3718518461 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f753c00fa80 con 0x56443a9b89a0 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.588+0000 7f7554a72740 10 client.15762 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.588+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({fsmap=29}) -- 0x56443aa39390 con 0x56443a9b89a0 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.588+0000 7f752affd640 1 -- 172.21.15.89:0/3718518461 <== mon.0 v2:172.21.15.89:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f753c00fc20 con 0x56443a9b89a0 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.589+0000 7f7554a72740 10 client.15762 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.589+0000 7f7554a72740 10 client.15762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.589+0000 7f7554a72740 20 client.15762 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.589+0000 7f7554a72740 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56443aa39b20 0x56443aa3bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.589+0000 7f7554a72740 4 client.15762 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:25:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.589+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56443a923d70 con 0x56443aa39b20 2024-09-15T23:25:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.589+0000 7f754affd640 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56443aa39b20 0x56443aa3bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.589+0000 7f752affd640 10 client.15762 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.589+0000 7f754affd640 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56443aa39b20 0x56443aa3bf10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef0c2df2550c26f0 server_cookie=8d529f66a175e3a2 in_seq=0 out_seq=0 2024-09-15T23:25:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f752affd640 1 -- 172.21.15.89:0/3718518461 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56443a923d70 con 0x56443aa39b20 2024-09-15T23:25:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f752affd640 10 client.15762 handle_command_reply: tid=0 2024-09-15T23:25:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 client.15762 shutdown 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 msgr2=0x56443a9c7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c7670 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f753c012ea0 tx=0x7f753c0046c0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 shutdown_connections 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56443aa39b20 0x56443aa3bf10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56443a9be370 0x56443a9c8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56443a9b92e0 0x56443a9c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 --2- 172.21.15.89:0/3718518461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56443a9b89a0 0x56443a9c7670 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 >> 172.21.15.89:0/3718518461 conn(0x56443a9b4010 msgr2=0x56443a9b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f752affd640 0 client.15762 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 shutdown_connections 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 1 -- 172.21.15.89:0/3718518461 wait complete. 2024-09-15T23:25:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.597+0000 7f7554a72740 20 client.15762 trim_cache size 0 max 16384 2024-09-15T23:25:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:25:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:25:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:25:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:25:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 -- 172.21.15.89:0/3856896870 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75340a3410 msgr2=0x7f75340a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75340a3410 0x7f75340a58d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x56443a9b9d10 tx=0x7f753c005f90 comp rx=0 tx=0).stop 2024-09-15T23:25:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 -- 172.21.15.89:0/3856896870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f754c10e640 msgr2=0x7f754c1a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f754c10e640 0x7f754c1a2f50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f754c073c60 tx=0x7f7540030de0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 -- 172.21.15.89:0/3856896870 shutdown_connections 2024-09-15T23:25:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75340a3410 0x7f75340a58d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f754c1a7fb0 0x7f754c1aa3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f754c10e640 0x7f754c1a2f50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 --2- 172.21.15.89:0/3856896870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f754c1091f0 0x7f754c1a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 -- 172.21.15.89:0/3856896870 >> 172.21.15.89:0/3856896870 conn(0x7f754c06ece0 msgr2=0x7f754c10c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.598+0000 7f7551cd7640 1 -- 172.21.15.89:0/3856896870 shutdown_connections 2024-09-15T23:25:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:20.599+0000 7f7551cd7640 1 -- 172.21.15.89:0/3856896870 wait complete. 2024-09-15T23:25:20.865 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:25:20.865 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:25:21.248 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.814+0000 7fddab044640 1 -- 172.21.15.89:0/1716688557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102570 msgr2=0x7fdda41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.814+0000 7fddab044640 1 --2- 172.21.15.89:0/1716688557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102570 0x7fdda41029b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009890 tx=0x7fdd9002a400 comp rx=0 tx=0).stop 2024-09-15T23:25:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.814+0000 7fddab044640 1 -- 172.21.15.89:0/1716688557 shutdown_connections 2024-09-15T23:25:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.814+0000 7fddab044640 1 --2- 172.21.15.89:0/1716688557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda4102ef0 0x7fdda410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.814+0000 7fddab044640 1 --2- 172.21.15.89:0/1716688557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102570 0x7fdda41029b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.814+0000 7fddab044640 1 --2- 172.21.15.89:0/1716688557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdda4100e20 0x7fdda4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.814+0000 7fddab044640 1 -- 172.21.15.89:0/1716688557 >> 172.21.15.89:0/1716688557 conn(0x7fdda40fc7d0 msgr2=0x7fdda40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.814+0000 7fddab044640 1 -- 172.21.15.89:0/1716688557 shutdown_connections 2024-09-15T23:25:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.814+0000 7fddab044640 1 -- 172.21.15.89:0/1716688557 wait complete. 2024-09-15T23:25:21.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.815+0000 7fddab044640 1 Processor -- start 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.815+0000 7fddab044640 1 -- start start 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.815+0000 7fddab044640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdda4100e20 0x7fdda4197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.815+0000 7fddab044640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102570 0x7fdda4197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fddab044640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda4102ef0 0x7fdda419bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fddab044640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdda4110f80 con 0x7fdda4100e20 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fddab044640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdda4110e00 con 0x7fdda4102570 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fddab044640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdda4111100 con 0x7fdda4102ef0 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fdda3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102570 0x7fdda4197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fdda3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102570 0x7fdda4197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:42016/0 (socket says 172.21.15.89:42016) 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fdda3fff640 1 -- 172.21.15.89:0/1475062990 learned_addr learned my addr 172.21.15.89:0/1475062990 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:21.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fdda95e1640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda4102ef0 0x7fdda419bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fdda8de0640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdda4100e20 0x7fdda4197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fdda95e1640 1 -- 172.21.15.89:0/1475062990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102570 msgr2=0x7fdda4197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fdda95e1640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102570 0x7fdda4197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fdda95e1640 1 -- 172.21.15.89:0/1475062990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdda4100e20 msgr2=0x7fdda4197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fdda95e1640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdda4100e20 0x7fdda4197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.816+0000 7fdda95e1640 1 -- 172.21.15.89:0/1475062990 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda419c1b0 con 0x7fdda4102ef0 2024-09-15T23:25:21.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.817+0000 7fdda95e1640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda4102ef0 0x7fdda419bad0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00cab0 tx=0x7fdd9c00ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.817+0000 7fdda1ffb640 1 -- 172.21.15.89:0/1475062990 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c013d70 con 0x7fdda4102ef0 2024-09-15T23:25:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.817+0000 7fdda1ffb640 1 -- 172.21.15.89:0/1475062990 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd9c00ad70 con 0x7fdda4102ef0 2024-09-15T23:25:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.817+0000 7fddab044640 1 -- 172.21.15.89:0/1475062990 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda419c4a0 con 0x7fdda4102ef0 2024-09-15T23:25:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.817+0000 7fdda1ffb640 1 -- 172.21.15.89:0/1475062990 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c00e3b0 con 0x7fdda4102ef0 2024-09-15T23:25:21.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.818+0000 7fddab044640 1 -- 172.21.15.89:0/1475062990 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda4069f70 con 0x7fdda4102ef0 2024-09-15T23:25:21.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.821+0000 7fddad41d740 1 Processor -- start 2024-09-15T23:25:21.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.821+0000 7fdda1ffb640 1 -- 172.21.15.89:0/1475062990 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fdd9c00a890 con 0x7fdda4102ef0 2024-09-15T23:25:21.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.821+0000 7fddad41d740 1 -- start start 2024-09-15T23:25:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fddad41d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a5178ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fddad41d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a51790480 0x558a5179a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fddad41d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a5179d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fddad41d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558a516f5280 con 0x558a5178fb60 2024-09-15T23:25:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fddad41d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558a5169cb10 con 0x558a51790480 2024-09-15T23:25:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fddad41d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558a516f1240 con 0x558a5179ad40 2024-09-15T23:25:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fdda3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a5178ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fdda3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a5178ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46122/0 (socket says 172.21.15.89:46122) 2024-09-15T23:25:21.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fdda3fff640 1 -- 172.21.15.89:0/1760598114 learned_addr learned my addr 172.21.15.89:0/1760598114 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:21.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fdda3fff640 1 -- 172.21.15.89:0/1760598114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 msgr2=0x558a5179d130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:25:21.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fdda3fff640 1 --2- 172.21.15.89:0/1760598114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a5179d130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fdda3fff640 1 -- 172.21.15.89:0/1760598114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a51790480 msgr2=0x558a5179a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fdda3fff640 1 --2- 172.21.15.89:0/1760598114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a51790480 0x558a5179a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.822+0000 7fdda3fff640 1 -- 172.21.15.89:0/1760598114 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a516a91f0 con 0x558a5178fb60 2024-09-15T23:25:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.823+0000 7fdda3fff640 1 --2- 172.21.15.89:0/1760598114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a5178ff40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdd900094e0 tx=0x7fdd90005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a61b4a75fdbc31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.823+0000 7fdda1ffb640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdd940a3350 0x7fdd940a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.823+0000 7fdd8effd640 1 -- 172.21.15.89:0/1760598114 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90037040 con 0x558a5178fb60 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.823+0000 7fdda1ffb640 1 -- 172.21.15.89:0/1475062990 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdd9c0d4820 con 0x7fdda4102ef0 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.823+0000 7fdd8effd640 1 -- 172.21.15.89:0/1760598114 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd90004230 con 0x558a5178fb60 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.823+0000 7fdd8effd640 1 -- 172.21.15.89:0/1760598114 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90030830 con 0x558a5178fb60 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.823+0000 7fdda8de0640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdd940a3350 0x7fdd940a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.824+0000 7fddad41d740 1 -- 172.21.15.89:0/1760598114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 msgr2=0x558a5178ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.824+0000 7fddad41d740 1 --2- 172.21.15.89:0/1760598114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a5178ff40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdd900094e0 tx=0x7fdd90005750 comp rx=0 tx=0).stop 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.824+0000 7fdda8de0640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdd940a3350 0x7fdd940a5810 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x558a5178cf50 tx=0x7fdd980097a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.824+0000 7fddad41d740 1 -- 172.21.15.89:0/1760598114 shutdown_connections 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.824+0000 7fddad41d740 1 --2- 172.21.15.89:0/1760598114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a5179d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.824+0000 7fddad41d740 1 --2- 172.21.15.89:0/1760598114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a51790480 0x558a5179a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.824+0000 7fddad41d740 1 --2- 172.21.15.89:0/1760598114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a5178ff40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.824+0000 7fddad41d740 1 -- 172.21.15.89:0/1760598114 >> 172.21.15.89:0/1760598114 conn(0x558a5178c0f0 msgr2=0x558a5178c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.825+0000 7fddad41d740 1 -- 172.21.15.89:0/1760598114 shutdown_connections 2024-09-15T23:25:21.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.825+0000 7fddad41d740 1 -- 172.21.15.89:0/1760598114 wait complete. 2024-09-15T23:25:21.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.825+0000 7fddad41d740 1 -- start start 2024-09-15T23:25:21.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.825+0000 7fddad41d740 1 Processor -- start 2024-09-15T23:25:21.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fddad41d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 0x558a51797bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fddad41d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a51790480 0x558a517980f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fddad41d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a51798630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fddad41d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558a5179d9b0 con 0x558a51790480 2024-09-15T23:25:21.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fddad41d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558a516aff10 con 0x558a5178fb60 2024-09-15T23:25:21.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fddad41d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558a5179db30 con 0x558a5179ad40 2024-09-15T23:25:21.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fddad41d740 10 client.0 _ll_register_callbacks cb 0x558a51678a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:21.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fdda3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a51790480 0x558a517980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fdda3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a51790480 0x558a517980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46130/0 (socket says 172.21.15.89:46130) 2024-09-15T23:25:21.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fdda3fff640 1 -- 172.21.15.89:0/1302370095 learned_addr learned my addr 172.21.15.89:0/1302370095 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:21.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fdda8de0640 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 0x558a51797bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fdda95e1640 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a51798630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fdda3fff640 1 -- 172.21.15.89:0/1302370095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 msgr2=0x558a51798630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fdda3fff640 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a51798630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fdda3fff640 1 -- 172.21.15.89:0/1302370095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 msgr2=0x558a51797bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fdda3fff640 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 0x558a51797bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.826+0000 7fdda3fff640 1 -- 172.21.15.89:0/1302370095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a514666a0 con 0x558a51790480 2024-09-15T23:25:21.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.827+0000 7fdda3fff640 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a51790480 0x558a517980f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000b010 tx=0x7fdd90003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b927ea14f2651a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.827+0000 7fdd83fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:21.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.827+0000 7fdd83fff640 1 -- 172.21.15.89:0/1302370095 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90037040 con 0x558a51790480 2024-09-15T23:25:21.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.827+0000 7fdd83fff640 1 -- 172.21.15.89:0/1302370095 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdd900307d0 con 0x558a51790480 2024-09-15T23:25:21.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.827+0000 7fddad41d740 1 -- 172.21.15.89:0/1302370095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd90004230 con 0x558a51790480 2024-09-15T23:25:21.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.827+0000 7fdd83fff640 1 -- 172.21.15.89:0/1302370095 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdd90003a10 con 0x558a51790480 2024-09-15T23:25:21.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.827+0000 7fddad41d740 10 client.15774 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.827+0000 7fddad41d740 1 -- 172.21.15.89:0/1302370095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558a517a9570 con 0x558a51790480 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.828+0000 7fdd83fff640 1 -- 172.21.15.89:0/1302370095 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd90004230 con 0x558a51790480 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.828+0000 7fdd83fff640 1 -- 172.21.15.89:0/1302370095 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fdd90046460 con 0x558a51790480 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.828+0000 7fddad41d740 10 client.15774 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.828+0000 7fddad41d740 10 client.15774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.828+0000 7fddad41d740 20 client.15774 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.828+0000 7fddad41d740 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558a517aa8c0 0x558a517accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.828+0000 7fddad41d740 4 client.15774 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.828+0000 7fddad41d740 1 -- 172.21.15.89:0/1302370095 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a517027f0 con 0x558a517aa8c0 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.828+0000 7fdda8de0640 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558a517aa8c0 0x558a517accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.829+0000 7fdd83fff640 10 client.15774 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.829+0000 7fdda8de0640 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558a517aa8c0 0x558a517accb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da06ffd81af52e37 server_cookie=a7e82c1457adb34 in_seq=0 out_seq=0 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.829+0000 7fdd83fff640 1 -- 172.21.15.89:0/1302370095 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558a517027f0 con 0x558a517aa8c0 2024-09-15T23:25:21.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.829+0000 7fdd83fff640 10 client.15774 handle_command_reply: tid=0 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.829+0000 7fddad41d740 1 client.15774 shutdown 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.829+0000 7fddad41d740 1 -- 172.21.15.89:0/1302370095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a51790480 msgr2=0x558a517980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.829+0000 7fddad41d740 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a51790480 0x558a517980f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000b010 tx=0x7fdd90003e00 comp rx=0 tx=0).stop 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.830+0000 7fddad41d740 1 -- 172.21.15.89:0/1302370095 shutdown_connections 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.830+0000 7fddad41d740 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558a517aa8c0 0x558a517accb0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.830+0000 7fddad41d740 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a51798630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.830+0000 7fddad41d740 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a51790480 0x558a517980f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.830+0000 7fddad41d740 1 --2- 172.21.15.89:0/1302370095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 0x558a51797bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.830+0000 7fddad41d740 1 -- 172.21.15.89:0/1302370095 >> 172.21.15.89:0/1302370095 conn(0x558a5178c0f0 msgr2=0x558a51791e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.830+0000 7fdd83fff640 0 client.15774 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.830+0000 7fddad41d740 1 -- 172.21.15.89:0/1302370095 shutdown_connections 2024-09-15T23:25:21.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.830+0000 7fddad41d740 1 -- 172.21.15.89:0/1302370095 wait complete. 2024-09-15T23:25:21.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.830+0000 7fddad41d740 20 client.15774 trim_cache size 0 max 16384 2024-09-15T23:25:21.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fddad41d740 1 Processor -- start 2024-09-15T23:25:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fddad41d740 1 -- start start 2024-09-15T23:25:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fddad41d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 0x558a517925c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fddad41d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5179ad40 0x558a517a64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fddad41d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a51796ca0 0x558a517af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fddad41d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41aca60 con 0x558a5179ad40 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fddad41d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41ae5a0 con 0x558a5178fb60 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fddad41d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41ad130 con 0x558a51796ca0 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fdda8de0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 0x558a517925c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fdda8de0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 0x558a517925c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:42032/0 (socket says 172.21.15.89:42032) 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fdda8de0640 1 -- 172.21.15.89:0/916507040 learned_addr learned my addr 172.21.15.89:0/916507040 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.847+0000 7fdda3fff640 1 --2- 172.21.15.89:0/916507040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5179ad40 0x558a517a64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdda95e1640 1 --2- 172.21.15.89:0/916507040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a51796ca0 0x558a517af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdda8de0640 1 -- 172.21.15.89:0/916507040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a51796ca0 msgr2=0x558a517af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdda8de0640 1 --2- 172.21.15.89:0/916507040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a51796ca0 0x558a517af8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdda8de0640 1 -- 172.21.15.89:0/916507040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5179ad40 msgr2=0x558a517a64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdda8de0640 1 --2- 172.21.15.89:0/916507040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5179ad40 0x558a517a64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdda8de0640 1 -- 172.21.15.89:0/916507040 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a5178bf20 con 0x558a5178fb60 2024-09-15T23:25:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdda3fff640 1 --2- 172.21.15.89:0/916507040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5179ad40 0x558a517a64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdda8de0640 1 --2- 172.21.15.89:0/916507040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 0x558a517925c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdd98005fd0 tx=0x7fdd98014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7016fcb6a9f82b79 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdd8e7fc640 1 -- 172.21.15.89:0/916507040 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98003d70 con 0x558a5178fb60 2024-09-15T23:25:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdd8e7fc640 1 -- 172.21.15.89:0/916507040 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd9802e070 con 0x558a5178fb60 2024-09-15T23:25:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.848+0000 7fdd8e7fc640 1 -- 172.21.15.89:0/916507040 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9801f6f0 con 0x558a5178fb60 2024-09-15T23:25:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 -- 172.21.15.89:0/916507040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 msgr2=0x558a517925c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 --2- 172.21.15.89:0/916507040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 0x558a517925c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdd98005fd0 tx=0x7fdd98014000 comp rx=0 tx=0).stop 2024-09-15T23:25:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 -- 172.21.15.89:0/916507040 shutdown_connections 2024-09-15T23:25:21.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 --2- 172.21.15.89:0/916507040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a51796ca0 0x558a517af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 --2- 172.21.15.89:0/916507040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5179ad40 0x558a517a64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 --2- 172.21.15.89:0/916507040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a5178fb60 0x558a517925c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 -- 172.21.15.89:0/916507040 >> 172.21.15.89:0/916507040 conn(0x558a5178c0f0 msgr2=0x558a517a5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 -- 172.21.15.89:0/916507040 shutdown_connections 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 -- 172.21.15.89:0/916507040 wait complete. 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 -- start start 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.849+0000 7fddad41d740 1 Processor -- start 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fddad41d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a517ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fddad41d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a51796ca0 0x558a517ade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fddad41d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a517ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fddad41d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41aded0 con 0x558a5178fb60 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda8de0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a517ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fddad41d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41ad800 con 0x558a51796ca0 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fddad41d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41aaa40 con 0x558a5179ad40 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda8de0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a517ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46138/0 (socket says 172.21.15.89:46138) 2024-09-15T23:25:21.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda8de0640 1 -- 172.21.15.89:0/4240634545 learned_addr learned my addr 172.21.15.89:0/4240634545 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda3fff640 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a51796ca0 0x558a517ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fddad41d740 10 client.0 _ll_register_callbacks cb 0x7fdd900096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda95e1640 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a517ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda8de0640 1 -- 172.21.15.89:0/4240634545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 msgr2=0x558a517ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda8de0640 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a517ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda8de0640 1 -- 172.21.15.89:0/4240634545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a51796ca0 msgr2=0x558a517ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda8de0640 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a51796ca0 0x558a517ade70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda8de0640 1 -- 172.21.15.89:0/4240634545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a51787830 con 0x558a5178fb60 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.850+0000 7fdda3fff640 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a51796ca0 0x558a517ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.851+0000 7fdda8de0640 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a517ad930 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fdd980039c0 tx=0x7fdd98009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660c0277a4bd8df6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.851+0000 7fdd83fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:21.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.851+0000 7fdd83fff640 1 -- 172.21.15.89:0/4240634545 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98005030 con 0x558a5178fb60 2024-09-15T23:25:21.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.851+0000 7fdd83fff640 1 -- 172.21.15.89:0/4240634545 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdd980051d0 con 0x558a5178fb60 2024-09-15T23:25:21.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.851+0000 7fddad41d740 1 -- 172.21.15.89:0/4240634545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd9802e070 con 0x558a5178fb60 2024-09-15T23:25:21.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.851+0000 7fdd83fff640 1 -- 172.21.15.89:0/4240634545 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdd980053d0 con 0x558a5178fb60 2024-09-15T23:25:21.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.851+0000 7fddad41d740 10 client.15780 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:21.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.851+0000 7fddad41d740 1 -- 172.21.15.89:0/4240634545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558a518143a0 con 0x558a5178fb60 2024-09-15T23:25:21.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.851+0000 7fdd83fff640 1 -- 172.21.15.89:0/4240634545 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd9802e070 con 0x558a5178fb60 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.851+0000 7fdd83fff640 1 -- 172.21.15.89:0/4240634545 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fdd98004ac0 con 0x558a5178fb60 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.852+0000 7fddad41d740 10 client.15780 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.852+0000 7fddad41d740 10 client.15780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.852+0000 7fddad41d740 20 client.15780 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.852+0000 7fddad41d740 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558a51814ac0 0x558a51816eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.852+0000 7fddad41d740 4 client.15780 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.852+0000 7fddad41d740 1 -- 172.21.15.89:0/4240634545 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558a516fcd00 con 0x558a51814ac0 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.852+0000 7fdda3fff640 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558a51814ac0 0x558a51816eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.852+0000 7fdda3fff640 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558a51814ac0 0x558a51816eb0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96a612be079b0afe server_cookie=f557491a15a73d4f in_seq=0 out_seq=0 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.852+0000 7fdd83fff640 10 client.15780 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fdd83fff640 1 -- 172.21.15.89:0/4240634545 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558a516fcd00 con 0x558a51814ac0 2024-09-15T23:25:21.858 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:25:21.859 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:25:21.859 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "ae319ca9-ffec-40e9-b506-9919dabc677c", 2024-09-15T23:25:21.859 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:25:21.859 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:25:21.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fdd83fff640 10 client.15780 handle_command_reply: tid=0 2024-09-15T23:25:21.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 client.15780 shutdown 2024-09-15T23:25:21.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 -- 172.21.15.89:0/4240634545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 msgr2=0x558a517ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a517ad930 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fdd980039c0 tx=0x7fdd98009e00 comp rx=0 tx=0).stop 2024-09-15T23:25:21.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 -- 172.21.15.89:0/4240634545 shutdown_connections 2024-09-15T23:25:21.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558a51814ac0 0x558a51816eb0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fdd83fff640 0 client.15780 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:21.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558a5179ad40 0x558a517ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558a51796ca0 0x558a517ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 --2- 172.21.15.89:0/4240634545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558a5178fb60 0x558a517ad930 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 -- 172.21.15.89:0/4240634545 >> 172.21.15.89:0/4240634545 conn(0x558a5178c0f0 msgr2=0x558a5178b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 -- 172.21.15.89:0/4240634545 shutdown_connections 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 1 -- 172.21.15.89:0/4240634545 wait complete. 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.854+0000 7fddad41d740 20 client.15780 trim_cache size 0 max 16384 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 -- 172.21.15.89:0/1475062990 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdd940a3350 msgr2=0x7fdd940a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdd940a3350 0x7fdd940a5810 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x558a5178cf50 tx=0x7fdd980097a0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 -- 172.21.15.89:0/1475062990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda4102ef0 msgr2=0x7fdda419bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda4102ef0 0x7fdda419bad0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00cab0 tx=0x7fdd9c00ceb0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 -- 172.21.15.89:0/1475062990 shutdown_connections 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdd940a3350 0x7fdd940a5810 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdda4102ef0 0x7fdda419bad0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102570 0x7fdda4197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 --2- 172.21.15.89:0/1475062990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdda4100e20 0x7fdda4197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 -- 172.21.15.89:0/1475062990 >> 172.21.15.89:0/1475062990 conn(0x7fdda40fc7d0 msgr2=0x7fdda41067c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 -- 172.21.15.89:0/1475062990 shutdown_connections 2024-09-15T23:25:21.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:21.855+0000 7fddab044640 1 -- 172.21.15.89:0/1475062990 wait complete. 2024-09-15T23:25:22.402 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:22 smithi089 ceph-mon[30933]: pgmap v390: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-15T23:25:22.402 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:22 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:25:22.402 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:22 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:22.402 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:22 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:25:22.402 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:22 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:22 smithi120 ceph-mon[35250]: pgmap v390: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-15T23:25:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:22 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:25:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:22 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:22 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:25:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:22 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:22.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:22 smithi174 ceph-mon[35586]: pgmap v390: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-15T23:25:22.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:22 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:25:22.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:22 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:22 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:25:22.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:22 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:22.701 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:25:22.971 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:23.395 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:23 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:25:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:23 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:25:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:23 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.480+0000 7f077bc28640 1 -- 172.21.15.89:0/2743322963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07741082a0 msgr2=0x7f077410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.480+0000 7f077bc28640 1 --2- 172.21.15.89:0/2743322963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07741082a0 0x7f077410ec40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f076c009c10 tx=0x7f076c02a6f0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.480+0000 7f077bc28640 1 -- 172.21.15.89:0/2743322963 shutdown_connections 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.480+0000 7f077bc28640 1 --2- 172.21.15.89:0/2743322963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07741082a0 0x7f077410ec40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.480+0000 7f077bc28640 1 --2- 172.21.15.89:0/2743322963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07741036e0 0x7f0774107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.480+0000 7f077bc28640 1 --2- 172.21.15.89:0/2743322963 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0774102d30 0x7f0774103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.480+0000 7f077bc28640 1 -- 172.21.15.89:0/2743322963 >> 172.21.15.89:0/2743322963 conn(0x7f07740fc7d0 msgr2=0x7f07740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.481+0000 7f077bc28640 1 -- 172.21.15.89:0/2743322963 shutdown_connections 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.481+0000 7f077bc28640 1 -- 172.21.15.89:0/2743322963 wait complete. 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.481+0000 7f077bc28640 1 Processor -- start 2024-09-15T23:25:23.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.481+0000 7f077bc28640 1 -- start start 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f077bc28640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0774102d30 0x7f0774100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f077bc28640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07741036e0 0x7f0774100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f077bc28640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07741082a0 0x7f0774101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f077bc28640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0774112050 con 0x7f07741036e0 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f077bc28640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0774111ed0 con 0x7f07741082a0 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f077bc28640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07741121d0 con 0x7f0774102d30 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07791c3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07741036e0 0x7f0774100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07791c3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07741036e0 0x7f0774100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46160/0 (socket says 172.21.15.89:46160) 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07791c3640 1 -- 172.21.15.89:0/114699176 learned_addr learned my addr 172.21.15.89:0/114699176 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07799c4640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0774102d30 0x7f0774100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f077a1c5640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07741082a0 0x7f0774101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07791c3640 1 -- 172.21.15.89:0/114699176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0774102d30 msgr2=0x7f0774100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07791c3640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0774102d30 0x7f0774100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07791c3640 1 -- 172.21.15.89:0/114699176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07741082a0 msgr2=0x7f0774101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07791c3640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07741082a0 0x7f0774101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07791c3640 1 -- 172.21.15.89:0/114699176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07741aafc0 con 0x7f07741036e0 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07799c4640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0774102d30 0x7f0774100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.482+0000 7f07791c3640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07741036e0 0x7f0774100ed0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f076800e9c0 tx=0x7f076800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.483+0000 7f0766ffd640 1 -- 172.21.15.89:0/114699176 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076800ccc0 con 0x7f07741036e0 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.483+0000 7f0766ffd640 1 -- 172.21.15.89:0/114699176 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0768005500 con 0x7f07741036e0 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.483+0000 7f077bc28640 1 -- 172.21.15.89:0/114699176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07741ab1f0 con 0x7f07741036e0 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.483+0000 7f0766ffd640 1 -- 172.21.15.89:0/114699176 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768010430 con 0x7f07741036e0 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.483+0000 7f077bc28640 1 -- 172.21.15.89:0/114699176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07741ab6e0 con 0x7f07741036e0 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.485+0000 7f0766ffd640 1 -- 172.21.15.89:0/114699176 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f0768005020 con 0x7f07741036e0 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.486+0000 7f0766ffd640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07600a32d0 0x7f07600a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.486+0000 7f077e001740 1 Processor -- start 2024-09-15T23:25:23.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.486+0000 7f07799c4640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07600a32d0 0x7f07600a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.486+0000 7f0766ffd640 1 -- 172.21.15.89:0/114699176 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f07680d0360 con 0x7f07741036e0 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f077e001740 1 -- start start 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f07799c4640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07600a32d0 0x7f07600a5790 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0770005fd0 tx=0x7f0770009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f077e001740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f077e001740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f077e001740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05b0680 0x561db05b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f077e001740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561db050b2e0 con 0x561db05a54a0 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f077e001740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561db04b2ce0 con 0x561db05a5dc0 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f077e001740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561db05072a0 con 0x561db05b0680 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f07799c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f07799c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46172/0 (socket says 172.21.15.89:46172) 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f07799c4640 1 -- 172.21.15.89:0/4012222139 learned_addr learned my addr 172.21.15.89:0/4012222139 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f077a1c5640 1 --2- 172.21.15.89:0/4012222139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05b0680 0x561db05b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.487+0000 7f07791c3640 1 --2- 172.21.15.89:0/4012222139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.488+0000 7f07799c4640 1 -- 172.21.15.89:0/4012222139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05b0680 msgr2=0x561db05b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.488+0000 7f07799c4640 1 --2- 172.21.15.89:0/4012222139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05b0680 0x561db05b2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.488+0000 7f07799c4640 1 -- 172.21.15.89:0/4012222139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 msgr2=0x561db05b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.488+0000 7f07799c4640 1 --2- 172.21.15.89:0/4012222139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05b0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.488+0000 7f07799c4640 1 -- 172.21.15.89:0/4012222139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561db0497440 con 0x561db05a54a0 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.488+0000 7f07799c4640 1 --2- 172.21.15.89:0/4012222139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a5880 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0770009b00 tx=0x7f0770014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d624fa3ed2ca342 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.488+0000 7f075ffff640 1 -- 172.21.15.89:0/4012222139 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077001c7d0 con 0x561db05a54a0 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.488+0000 7f075ffff640 1 -- 172.21.15.89:0/4012222139 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f077001ce50 con 0x561db05a54a0 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.488+0000 7f075ffff640 1 -- 172.21.15.89:0/4012222139 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077000e610 con 0x561db05a54a0 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.489+0000 7f077e001740 1 -- 172.21.15.89:0/4012222139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 msgr2=0x561db05a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.489+0000 7f077e001740 1 --2- 172.21.15.89:0/4012222139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a5880 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0770009b00 tx=0x7f0770014000 comp rx=0 tx=0).stop 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.489+0000 7f077e001740 1 -- 172.21.15.89:0/4012222139 shutdown_connections 2024-09-15T23:25:23.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.489+0000 7f077e001740 1 --2- 172.21.15.89:0/4012222139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05b0680 0x561db05b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.489+0000 7f077e001740 1 --2- 172.21.15.89:0/4012222139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.489+0000 7f077e001740 1 --2- 172.21.15.89:0/4012222139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a5880 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.489+0000 7f077e001740 1 -- 172.21.15.89:0/4012222139 >> 172.21.15.89:0/4012222139 conn(0x561db05a1a30 msgr2=0x561db05a1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.489+0000 7f077e001740 1 -- 172.21.15.89:0/4012222139 shutdown_connections 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.489+0000 7f077e001740 1 -- 172.21.15.89:0/4012222139 wait complete. 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.489+0000 7f077e001740 1 -- start start 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f077e001740 1 Processor -- start 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f077e001740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a54a0 0x561db05b0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f077e001740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05ad300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f077e001740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f07799c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a54a0 0x561db05b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f07799c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a54a0 0x561db05b0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:34388/0 (socket says 172.21.15.89:34388) 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f07799c4640 1 -- 172.21.15.89:0/2222042364 learned_addr learned my addr 172.21.15.89:0/2222042364 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:23.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f077a1c5640 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f077e001740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561db05b3250 con 0x561db05b0680 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f077e001740 1 -- 172.21.15.89:0/2222042364 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561db04c6050 con 0x561db05a5dc0 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f077e001740 1 -- 172.21.15.89:0/2222042364 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561db05b33d0 con 0x561db05a54a0 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.490+0000 7f07791c3640 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f07799c4640 1 -- 172.21.15.89:0/2222042364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 msgr2=0x561db05ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f07799c4640 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f07799c4640 1 -- 172.21.15.89:0/2222042364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 msgr2=0x561db05ad8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f07799c4640 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05ad8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f07799c4640 1 -- 172.21.15.89:0/2222042364 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561db027c3c0 con 0x561db05a54a0 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f077a1c5640 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05ad8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f077e001740 10 client.0 _ll_register_callbacks cb 0x561db0506160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f077e001740 1 -- 172.21.15.89:0/2222042364 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f077001ce50 con 0x561db05a54a0 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f07799c4640 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a54a0 0x561db05b0050 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0770010010 tx=0x7f0770026900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c015a6089a7d96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f075d7fa640 10 client.25162 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f075d7fa640 1 -- 172.21.15.89:0/2222042364 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0770005040 con 0x561db05a54a0 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.491+0000 7f075d7fa640 1 -- 172.21.15.89:0/2222042364 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f07700051e0 con 0x561db05a54a0 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f075d7fa640 1 -- 172.21.15.89:0/2222042364 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0770005420 con 0x561db05a54a0 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f077e001740 10 client.25162 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f077e001740 1 -- 172.21.15.89:0/2222042364 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=29}) -- 0x561db05bea10 con 0x561db05a54a0 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f075d7fa640 1 -- 172.21.15.89:0/2222042364 <== mon.1 v2:172.21.15.174:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f077002f070 con 0x561db05a54a0 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f077e001740 10 client.25162 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f077e001740 10 client.25162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f077e001740 20 client.25162 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f077e001740 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561db05bfd60 0x561db05c2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f077e001740 4 client.25162 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f077e001740 1 -- 172.21.15.89:0/2222042364 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561db0518850 con 0x561db05bfd60 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.492+0000 7f07791c3640 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561db05bfd60 0x561db05c2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.493+0000 7f075d7fa640 10 client.25162 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.493+0000 7f07791c3640 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561db05bfd60 0x561db05c2150 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90a3f951b7fd25f7 server_cookie=9b55d079d07f1c25 in_seq=0 out_seq=0 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.493+0000 7f075d7fa640 1 -- 172.21.15.89:0/2222042364 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561db0518850 con 0x561db05bfd60 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.493+0000 7f075d7fa640 10 client.25162 handle_command_reply: tid=0 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.493+0000 7f077e001740 1 client.25162 shutdown 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 1 -- 172.21.15.89:0/2222042364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a54a0 msgr2=0x561db05b0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a54a0 0x561db05b0050 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0770010010 tx=0x7f0770026900 comp rx=0 tx=0).stop 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 1 -- 172.21.15.89:0/2222042364 shutdown_connections 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561db05bfd60 0x561db05c2150 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05ad8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 1 --2- 172.21.15.89:0/2222042364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a54a0 0x561db05b0050 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 1 -- 172.21.15.89:0/2222042364 >> 172.21.15.89:0/2222042364 conn(0x561db05a1a30 msgr2=0x561db05a0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f075d7fa640 0 client.25162 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:23.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 1 -- 172.21.15.89:0/2222042364 shutdown_connections 2024-09-15T23:25:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 1 -- 172.21.15.89:0/2222042364 wait complete. 2024-09-15T23:25:23.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.494+0000 7f077e001740 20 client.25162 trim_cache size 0 max 16384 2024-09-15T23:25:23.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.511+0000 7f077e001740 1 Processor -- start 2024-09-15T23:25:23.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f077e001740 1 -- start start 2024-09-15T23:25:23.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f077e001740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f077e001740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05c7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f077e001740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05b0680 0x561db05c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f077e001740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f07741b54c0 con 0x561db05a54a0 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f077e001740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07741b7000 con 0x561db05a5dc0 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f077e001740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07741b5b90 con 0x561db05b0680 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f07791c3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f07799c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f07799c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46184/0 (socket says 172.21.15.89:46184) 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.512+0000 7f07799c4640 1 -- 172.21.15.89:0/101261304 learned_addr learned my addr 172.21.15.89:0/101261304 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f077a1c5640 1 --2- 172.21.15.89:0/101261304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05b0680 0x561db05c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f07791c3640 1 -- 172.21.15.89:0/101261304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05b0680 msgr2=0x561db05c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f07791c3640 1 --2- 172.21.15.89:0/101261304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05b0680 0x561db05c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f07791c3640 1 -- 172.21.15.89:0/101261304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 msgr2=0x561db05a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f07791c3640 1 --2- 172.21.15.89:0/101261304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f07791c3640 1 -- 172.21.15.89:0/101261304 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561db05a5bf0 con 0x561db05a5dc0 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f07799c4640 1 --2- 172.21.15.89:0/101261304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f07791c3640 1 --2- 172.21.15.89:0/101261304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05c7840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f07680cd8c0 tx=0x7f0768098040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2616c9daabe1d307 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f075f7fe640 1 -- 172.21.15.89:0/101261304 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680a0780 con 0x561db05a5dc0 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f075f7fe640 1 -- 172.21.15.89:0/101261304 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07680a5040 con 0x561db05a5dc0 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.513+0000 7f075f7fe640 1 -- 172.21.15.89:0/101261304 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680cdb10 con 0x561db05a5dc0 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.514+0000 7f077e001740 1 -- 172.21.15.89:0/101261304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 msgr2=0x561db05c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.514+0000 7f077e001740 1 --2- 172.21.15.89:0/101261304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05c7840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f07680cd8c0 tx=0x7f0768098040 comp rx=0 tx=0).stop 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.514+0000 7f077e001740 1 -- 172.21.15.89:0/101261304 shutdown_connections 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.514+0000 7f077e001740 1 --2- 172.21.15.89:0/101261304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05b0680 0x561db05c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.514+0000 7f077e001740 1 --2- 172.21.15.89:0/101261304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a5dc0 0x561db05c7840 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.514+0000 7f077e001740 1 --2- 172.21.15.89:0/101261304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05a54a0 0x561db05a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.514+0000 7f077e001740 1 -- 172.21.15.89:0/101261304 >> 172.21.15.89:0/101261304 conn(0x561db05a1a30 msgr2=0x561db05a0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.514+0000 7f077e001740 1 -- 172.21.15.89:0/101261304 shutdown_connections 2024-09-15T23:25:23.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.514+0000 7f077e001740 1 -- 172.21.15.89:0/101261304 wait complete. 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.515+0000 7f077e001740 1 -- start start 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.515+0000 7f077e001740 1 Processor -- start 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.515+0000 7f077e001740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a54a0 0x561db05b6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f077e001740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a5dc0 0x561db05b6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f077e001740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05b6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f07799c4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a54a0 0x561db05b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f077a1c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05b6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f077a1c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05b6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46192/0 (socket says 172.21.15.89:46192) 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f07791c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a5dc0 0x561db05b6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f07799c4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a54a0 0x561db05b6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:42098/0 (socket says 172.21.15.89:42098) 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f07741b6930 con 0x561db05b0680 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07741b6260 con 0x561db05a54a0 2024-09-15T23:25:23.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07741b34a0 con 0x561db05a5dc0 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f077e001740 10 client.0 _ll_register_callbacks cb 0x7f0768004950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f077a1c5640 1 -- 172.21.15.89:0/3701399729 learned_addr learned my addr 172.21.15.89:0/3701399729 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f07791c3640 1 -- 172.21.15.89:0/3701399729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a54a0 msgr2=0x561db05b6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f07791c3640 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a54a0 0x561db05b6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f07791c3640 1 -- 172.21.15.89:0/3701399729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 msgr2=0x561db05b6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f07799c4640 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a54a0 0x561db05b6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f07791c3640 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05b6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.516+0000 7f07791c3640 1 -- 172.21.15.89:0/3701399729 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561db0629820 con 0x561db05a5dc0 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.517+0000 7f077a1c5640 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05b6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.517+0000 7f07791c3640 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a5dc0 0x561db05b6850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f07680054d0 tx=0x7f0768005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfe9f6a132d8c13b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.517+0000 7f075d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.517+0000 7f075d7fa640 1 -- 172.21.15.89:0/3701399729 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680cdb10 con 0x561db05a5dc0 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.517+0000 7f075d7fa640 1 -- 172.21.15.89:0/3701399729 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0768098a40 con 0x561db05a5dc0 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.517+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07680a5040 con 0x561db05a5dc0 2024-09-15T23:25:23.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.517+0000 7f075d7fa640 1 -- 172.21.15.89:0/3701399729 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f07680b51b0 con 0x561db05a5dc0 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.517+0000 7f077e001740 10 client.25168 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.517+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561db062aa90 con 0x561db05a5dc0 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.518+0000 7f075d7fa640 1 -- 172.21.15.89:0/3701399729 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07680b5350 con 0x561db05a5dc0 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.518+0000 7f075d7fa640 1 -- 172.21.15.89:0/3701399729 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f07680a4c40 con 0x561db05a5dc0 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.518+0000 7f077e001740 10 client.25168 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.518+0000 7f077e001740 10 client.25168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.518+0000 7f077e001740 20 client.25168 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.518+0000 7f077e001740 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561db062b150 0x561db062d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.518+0000 7f077e001740 4 client.25168 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.518+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f07680a5040 con 0x561db062b150 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.518+0000 7f07799c4640 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561db062b150 0x561db062d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.519+0000 7f075d7fa640 10 client.25168 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.519+0000 7f07799c4640 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561db062b150 0x561db062d540 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a1a220beafc6515 server_cookie=a2e7cf2e36092196 in_seq=0 out_seq=0 2024-09-15T23:25:23.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.519+0000 7f075d7fa640 1 -- 172.21.15.89:0/3701399729 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f07680a5040 con 0x561db062b150 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.519+0000 7f075d7fa640 10 client.25168 handle_command_reply: tid=0 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.519+0000 7f077e001740 1 client.25168 shutdown 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.519+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a5dc0 msgr2=0x561db05b6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.519+0000 7f077e001740 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a5dc0 0x561db05b6850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f07680054d0 tx=0x7f0768005400 comp rx=0 tx=0).stop 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.519+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 shutdown_connections 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.519+0000 7f077e001740 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561db062b150 0x561db062d540 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.519+0000 7f077e001740 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561db05b0680 0x561db05b6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f077e001740 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561db05a5dc0 0x561db05b6850 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f077e001740 1 --2- 172.21.15.89:0/3701399729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561db05a54a0 0x561db05b6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 >> 172.21.15.89:0/3701399729 conn(0x561db05a1a30 msgr2=0x561db05a0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f075d7fa640 0 client.25168 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 shutdown_connections 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f077e001740 1 -- 172.21.15.89:0/3701399729 wait complete. 2024-09-15T23:25:23.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f077e001740 20 client.25168 trim_cache size 0 max 16384 2024-09-15T23:25:23.532 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:25:23.532 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:25:23.532 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:25:23.532 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f077bc28640 1 -- 172.21.15.89:0/114699176 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07600a32d0 msgr2=0x7f07600a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f077bc28640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07600a32d0 0x7f07600a5790 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0770005fd0 tx=0x7f0770009260 comp rx=0 tx=0).stop 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f077bc28640 1 -- 172.21.15.89:0/114699176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07741036e0 msgr2=0x7f0774100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.520+0000 7f077bc28640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07741036e0 0x7f0774100ed0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f076800e9c0 tx=0x7f076800edc0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.521+0000 7f077bc28640 1 -- 172.21.15.89:0/114699176 shutdown_connections 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.521+0000 7f077bc28640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07600a32d0 0x7f07600a5790 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.521+0000 7f077bc28640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07741082a0 0x7f0774101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.521+0000 7f077bc28640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07741036e0 0x7f0774100ed0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.521+0000 7f077bc28640 1 --2- 172.21.15.89:0/114699176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0774102d30 0x7f0774100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.521+0000 7f077bc28640 1 -- 172.21.15.89:0/114699176 >> 172.21.15.89:0/114699176 conn(0x7f07740fc7d0 msgr2=0x7f0774105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.521+0000 7f077bc28640 1 -- 172.21.15.89:0/114699176 shutdown_connections 2024-09-15T23:25:23.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:23.521+0000 7f077bc28640 1 -- 172.21.15.89:0/114699176 wait complete. 2024-09-15T23:25:24.236 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:24 smithi089 ceph-mon[30933]: pgmap v391: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:25:24.272 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:25:24.272 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:25:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:24 smithi120 ceph-mon[35250]: pgmap v391: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:25:24.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:24 smithi174 ceph-mon[35586]: pgmap v391: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:25:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:25.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.130+0000 7f6365d8a640 1 -- 172.21.15.89:0/683433077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63601036e0 msgr2=0x7f6360107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:25.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.130+0000 7f6365d8a640 1 --2- 172.21.15.89:0/683433077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63601036e0 0x7f6360107b90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6354009930 tx=0x7f635402a420 comp rx=0 tx=0).stop 2024-09-15T23:25:25.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.131+0000 7f6365d8a640 1 -- 172.21.15.89:0/683433077 shutdown_connections 2024-09-15T23:25:25.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.131+0000 7f6365d8a640 1 --2- 172.21.15.89:0/683433077 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63601082a0 0x7f636010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:25.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.131+0000 7f6365d8a640 1 --2- 172.21.15.89:0/683433077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63601036e0 0x7f6360107b90 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:25.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.131+0000 7f6365d8a640 1 --2- 172.21.15.89:0/683433077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6360102d30 0x7f6360103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:25.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.131+0000 7f6365d8a640 1 -- 172.21.15.89:0/683433077 >> 172.21.15.89:0/683433077 conn(0x7f63600fc7d0 msgr2=0x7f63600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:25.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.131+0000 7f6365d8a640 1 -- 172.21.15.89:0/683433077 shutdown_connections 2024-09-15T23:25:25.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.131+0000 7f6365d8a640 1 -- 172.21.15.89:0/683433077 wait complete. 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.131+0000 7f6365d8a640 1 Processor -- start 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.132+0000 7f6365d8a640 1 -- start start 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.132+0000 7f6365d8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6360102d30 0x7f6360198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.132+0000 7f6365d8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63601036e0 0x7f6360198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.132+0000 7f6365d8a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63601082a0 0x7f636019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.132+0000 7f6365d8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f63601120a0 con 0x7f63601036e0 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.132+0000 7f6365d8a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6360111f20 con 0x7f6360102d30 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.132+0000 7f6365d8a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6360112220 con 0x7f63601082a0 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.132+0000 7f635effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63601036e0 0x7f6360198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.132+0000 7f635effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63601036e0 0x7f6360198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46214/0 (socket says 172.21.15.89:46214) 2024-09-15T23:25:25.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.132+0000 7f635effd640 1 -- 172.21.15.89:0/1266419712 learned_addr learned my addr 172.21.15.89:0/1266419712 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.133+0000 7f635effd640 1 -- 172.21.15.89:0/1266419712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63601082a0 msgr2=0x7f636019cdf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.133+0000 7f635effd640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63601082a0 0x7f636019cdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.133+0000 7f635effd640 1 -- 172.21.15.89:0/1266419712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6360102d30 msgr2=0x7f6360198520 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.133+0000 7f635effd640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6360102d30 0x7f6360198520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.133+0000 7f635effd640 1 -- 172.21.15.89:0/1266419712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636019d4d0 con 0x7f63601036e0 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.133+0000 7f635effd640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63601036e0 0x7f6360198a60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6354005940 tx=0x7f63540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.133+0000 7f635cff9640 1 -- 172.21.15.89:0/1266419712 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635403f030 con 0x7f63601036e0 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.133+0000 7f6365d8a640 1 -- 172.21.15.89:0/1266419712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f636019d760 con 0x7f63601036e0 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.134+0000 7f6365d8a640 1 -- 172.21.15.89:0/1266419712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63601a50a0 con 0x7f63601036e0 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.134+0000 7f635cff9640 1 -- 172.21.15.89:0/1266419712 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6354003ce0 con 0x7f63601036e0 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.134+0000 7f635cff9640 1 -- 172.21.15.89:0/1266419712 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354049060 con 0x7f63601036e0 2024-09-15T23:25:25.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.135+0000 7f6365d8a640 1 -- 172.21.15.89:0/1266419712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6360103110 con 0x7f63601036e0 2024-09-15T23:25:25.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.136+0000 7f635cff9640 1 -- 172.21.15.89:0/1266419712 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f6354042030 con 0x7f63601036e0 2024-09-15T23:25:25.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.137+0000 7f635cff9640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63480a3350 0x7f63480a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:25.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.137+0000 7f635cff9640 1 -- 172.21.15.89:0/1266419712 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f63540f0530 con 0x7f63601036e0 2024-09-15T23:25:25.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.137+0000 7f635f7fe640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63480a3350 0x7f63480a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:25.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.137+0000 7f635f7fe640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63480a3350 0x7f63480a5810 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f635000a930 tx=0x7f6350005c00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:25.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.144+0000 7f635cff9640 1 -- 172.21.15.89:0/1266419712 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f63540bcd00 con 0x7f63601036e0 2024-09-15T23:25:25.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.356+0000 7f6365d8a640 1 -- 172.21.15.89:0/1266419712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6360199880 con 0x7f63601036e0 2024-09-15T23:25:25.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.357+0000 7f635cff9640 1 -- 172.21.15.89:0/1266419712 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f63540bc600 con 0x7f63601036e0 2024-09-15T23:25:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:25:25.359 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:25:25.359 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:25:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.359+0000 7f6365d8a640 1 -- 172.21.15.89:0/1266419712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63480a3350 msgr2=0x7f63480a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.359+0000 7f6365d8a640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63480a3350 0x7f63480a5810 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f635000a930 tx=0x7f6350005c00 comp rx=0 tx=0).stop 2024-09-15T23:25:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.359+0000 7f6365d8a640 1 -- 172.21.15.89:0/1266419712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63601036e0 msgr2=0x7f6360198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.359+0000 7f6365d8a640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63601036e0 0x7f6360198a60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6354005940 tx=0x7f63540042f0 comp rx=0 tx=0).stop 2024-09-15T23:25:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.359+0000 7f6365d8a640 1 -- 172.21.15.89:0/1266419712 shutdown_connections 2024-09-15T23:25:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.359+0000 7f6365d8a640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63480a3350 0x7f63480a5810 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.359+0000 7f6365d8a640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63601082a0 0x7f636019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.359+0000 7f6365d8a640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f63601036e0 0x7f6360198a60 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:25.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.359+0000 7f6365d8a640 1 --2- 172.21.15.89:0/1266419712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6360102d30 0x7f6360198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:25.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.360+0000 7f6365d8a640 1 -- 172.21.15.89:0/1266419712 >> 172.21.15.89:0/1266419712 conn(0x7f63600fc7d0 msgr2=0x7f6360105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:25.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.360+0000 7f6365d8a640 1 -- 172.21.15.89:0/1266419712 shutdown_connections 2024-09-15T23:25:25.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:25.360+0000 7f6365d8a640 1 -- 172.21.15.89:0/1266419712 wait complete. 2024-09-15T23:25:26.039 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:25:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:26 smithi089 ceph-mon[30933]: pgmap v392: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-15T23:25:26.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:26 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1266419712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:26.383 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:26 smithi120 ceph-mon[35250]: pgmap v392: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-15T23:25:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:26 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1266419712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:26 smithi174 ceph-mon[35586]: pgmap v392: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-09-15T23:25:26.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:26 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1266419712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:26.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.937+0000 7fc78038b640 1 -- 172.21.15.89:0/2187501003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc778102db0 msgr2=0x7fc778103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.937+0000 7fc78038b640 1 --2- 172.21.15.89:0/2187501003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc778102db0 0x7fc778103190 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc774009930 tx=0x7fc77402a420 comp rx=0 tx=0).stop 2024-09-15T23:25:26.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.938+0000 7fc78038b640 1 -- 172.21.15.89:0/2187501003 shutdown_connections 2024-09-15T23:25:26.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.938+0000 7fc78038b640 1 --2- 172.21.15.89:0/2187501003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7781082e0 0x7fc77810ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.938+0000 7fc78038b640 1 --2- 172.21.15.89:0/2187501003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc778103760 0x7fc778107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.938+0000 7fc78038b640 1 --2- 172.21.15.89:0/2187501003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc778102db0 0x7fc778103190 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.938+0000 7fc78038b640 1 -- 172.21.15.89:0/2187501003 >> 172.21.15.89:0/2187501003 conn(0x7fc7780fc7d0 msgr2=0x7fc7780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.938+0000 7fc78038b640 1 -- 172.21.15.89:0/2187501003 shutdown_connections 2024-09-15T23:25:26.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.938+0000 7fc78038b640 1 -- 172.21.15.89:0/2187501003 wait complete. 2024-09-15T23:25:26.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc78038b640 1 Processor -- start 2024-09-15T23:25:26.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc78038b640 1 -- start start 2024-09-15T23:25:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc78038b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc778102db0 0x7fc778198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc78038b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc778103760 0x7fc778198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc78038b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7781082e0 0x7fc77819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc78038b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc778112090 con 0x7fc778102db0 2024-09-15T23:25:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc78038b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc778111f10 con 0x7fc778103760 2024-09-15T23:25:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc78038b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc778112210 con 0x7fc7781082e0 2024-09-15T23:25:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc77e127640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc778102db0 0x7fc778198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc77e127640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc778102db0 0x7fc778198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38874/0 (socket says 172.21.15.89:38874) 2024-09-15T23:25:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.939+0000 7fc77e127640 1 -- 172.21.15.89:0/1803228101 learned_addr learned my addr 172.21.15.89:0/1803228101 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:26.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc77d926640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc778103760 0x7fc778198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc77e928640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7781082e0 0x7fc77819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc77d926640 1 -- 172.21.15.89:0/1803228101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7781082e0 msgr2=0x7fc77819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc77d926640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7781082e0 0x7fc77819cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc77d926640 1 -- 172.21.15.89:0/1803228101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc778102db0 msgr2=0x7fc778198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc77d926640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc778102db0 0x7fc778198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc77d926640 1 -- 172.21.15.89:0/1803228101 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77819d4d0 con 0x7fc778103760 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc77d926640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc778103760 0x7fc778198a60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7780697d0 tx=0x7fc76800cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc76f7fe640 1 -- 172.21.15.89:0/1803228101 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768007bb0 con 0x7fc778103760 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc78038b640 1 -- 172.21.15.89:0/1803228101 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc77819d7c0 con 0x7fc778103760 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc76f7fe640 1 -- 172.21.15.89:0/1803228101 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc768007d50 con 0x7fc778103760 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.940+0000 7fc76f7fe640 1 -- 172.21.15.89:0/1803228101 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768018780 con 0x7fc778103760 2024-09-15T23:25:26.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.941+0000 7fc78038b640 1 -- 172.21.15.89:0/1803228101 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7781a50a0 con 0x7fc778103760 2024-09-15T23:25:26.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.944+0000 7fc782764740 1 Processor -- start 2024-09-15T23:25:26.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.944+0000 7fc76f7fe640 1 -- 172.21.15.89:0/1803228101 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7fc76800f450 con 0x7fc778103760 2024-09-15T23:25:26.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.944+0000 7fc782764740 1 -- start start 2024-09-15T23:25:26.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc782764740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 0x55d6d68ed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc782764740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68ede20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc782764740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 0x55d6d68fab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc782764740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d6d684f2f0 con 0x55d6d68f8780 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc782764740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d6d67f6cf0 con 0x55d6d68ed9e0 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc782764740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d6d684b2b0 con 0x55d6d68eef50 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc77e928640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 0x55d6d68ed4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc77e928640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 0x55d6d68ed4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:47358/0 (socket says 172.21.15.89:47358) 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc77e928640 1 -- 172.21.15.89:0/3421976693 learned_addr learned my addr 172.21.15.89:0/3421976693 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc77d926640 1 --2- 172.21.15.89:0/3421976693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 0x55d6d68fab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc77e127640 1 --2- 172.21.15.89:0/3421976693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68ede20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc77e928640 1 -- 172.21.15.89:0/3421976693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 msgr2=0x55d6d68ede20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc77e928640 1 --2- 172.21.15.89:0/3421976693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68ede20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc77e928640 1 -- 172.21.15.89:0/3421976693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 msgr2=0x55d6d68fab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc77e928640 1 --2- 172.21.15.89:0/3421976693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 0x55d6d68fab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.945+0000 7fc77e928640 1 -- 172.21.15.89:0/3421976693 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6d67db3a0 con 0x55d6d68eef50 2024-09-15T23:25:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.946+0000 7fc77d926640 1 --2- 172.21.15.89:0/3421976693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 0x55d6d68fab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.946+0000 7fc77e928640 1 --2- 172.21.15.89:0/3421976693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 0x55d6d68ed4a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc770009f70 tx=0x7fc770010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=524113ba13e93cbc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.946+0000 7fc76f7fe640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7640a3350 0x7fc7640a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.946+0000 7fc77e127640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7640a3350 0x7fc7640a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.946+0000 7fc76cdea640 1 -- 172.21.15.89:0/3421976693 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc770016070 con 0x55d6d68eef50 2024-09-15T23:25:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.947+0000 7fc76cdea640 1 -- 172.21.15.89:0/3421976693 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc770004d20 con 0x55d6d68eef50 2024-09-15T23:25:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.947+0000 7fc76cdea640 1 -- 172.21.15.89:0/3421976693 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc770012950 con 0x55d6d68eef50 2024-09-15T23:25:26.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.947+0000 7fc76f7fe640 1 -- 172.21.15.89:0/1803228101 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc7680cfe70 con 0x7fc778103760 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.947+0000 7fc782764740 1 -- 172.21.15.89:0/3421976693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 msgr2=0x55d6d68ed4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.947+0000 7fc77e127640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7640a3350 0x7fc7640a5810 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55d6d68eaed0 tx=0x7fc774005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.947+0000 7fc782764740 1 --2- 172.21.15.89:0/3421976693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 0x55d6d68ed4a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc770009f70 tx=0x7fc770010800 comp rx=0 tx=0).stop 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.948+0000 7fc782764740 1 -- 172.21.15.89:0/3421976693 shutdown_connections 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.948+0000 7fc782764740 1 --2- 172.21.15.89:0/3421976693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 0x55d6d68fab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.948+0000 7fc782764740 1 --2- 172.21.15.89:0/3421976693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68ede20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.948+0000 7fc782764740 1 --2- 172.21.15.89:0/3421976693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 0x55d6d68ed4a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.948+0000 7fc782764740 1 -- 172.21.15.89:0/3421976693 >> 172.21.15.89:0/3421976693 conn(0x55d6d68b2310 msgr2=0x55d6d68ea070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.948+0000 7fc782764740 1 -- 172.21.15.89:0/3421976693 shutdown_connections 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.948+0000 7fc782764740 1 -- 172.21.15.89:0/3421976693 wait complete. 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.948+0000 7fc782764740 1 -- start start 2024-09-15T23:25:26.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc782764740 1 Processor -- start 2024-09-15T23:25:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc782764740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68f8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc782764740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc782764740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68f8780 0x55d6d68f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc782764740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d6d68fb480 con 0x55d6d68eef50 2024-09-15T23:25:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc782764740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d6d680a060 con 0x55d6d68ed9e0 2024-09-15T23:25:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc77e127640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68f8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc782764740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d6d68fb600 con 0x55d6d68f8780 2024-09-15T23:25:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc77e127640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68f8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:58894/0 (socket says 172.21.15.89:58894) 2024-09-15T23:25:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc77e127640 1 -- 172.21.15.89:0/3416421940 learned_addr learned my addr 172.21.15.89:0/3416421940 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:26.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc77e928640 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68f8780 0x55d6d68f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc782764740 10 client.0 _ll_register_callbacks cb 0x55d6d655d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.949+0000 7fc77d926640 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.950+0000 7fc77e928640 1 -- 172.21.15.89:0/3416421940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 msgr2=0x55d6d68f8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.950+0000 7fc77e928640 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68f8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.950+0000 7fc77e928640 1 -- 172.21.15.89:0/3416421940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 msgr2=0x55d6d68f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.950+0000 7fc77e928640 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.950+0000 7fc77e928640 1 -- 172.21.15.89:0/3416421940 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6d65c03c0 con 0x55d6d68f8780 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.950+0000 7fc77e127640 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68f8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.950+0000 7fc77d926640 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.950+0000 7fc77e928640 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68f8780 0x55d6d68f5c70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc770009f70 tx=0x7fc770005100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5733c607334e74b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.950+0000 7fc761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:25:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc761ffb640 1 -- 172.21.15.89:0/3416421940 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc770016070 con 0x55d6d68f8780 2024-09-15T23:25:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc782764740 1 -- 172.21.15.89:0/3416421940 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc770004d20 con 0x55d6d68f8780 2024-09-15T23:25:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc761ffb640 1 -- 172.21.15.89:0/3416421940 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc770005480 con 0x55d6d68f8780 2024-09-15T23:25:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc761ffb640 1 -- 172.21.15.89:0/3416421940 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc77002e050 con 0x55d6d68f8780 2024-09-15T23:25:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc782764740 10 client.25183 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc782764740 1 -- 172.21.15.89:0/3416421940 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d6d6906fb0 con 0x55d6d68f8780 2024-09-15T23:25:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc761ffb640 1 -- 172.21.15.89:0/3416421940 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc770004d20 con 0x55d6d68f8780 2024-09-15T23:25:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc761ffb640 1 -- 172.21.15.89:0/3416421940 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc77000f040 con 0x55d6d68f8780 2024-09-15T23:25:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc782764740 10 client.25183 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:26.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc782764740 10 client.25183 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc782764740 20 client.25183 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.951+0000 7fc782764740 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55d6d6908300 0x55d6d690a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.952+0000 7fc782764740 4 client.25183 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.952+0000 7fc782764740 1 -- 172.21.15.89:0/3416421940 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6d685c860 con 0x55d6d6908300 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.952+0000 7fc77e127640 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55d6d6908300 0x55d6d690a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.952+0000 7fc77e127640 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55d6d6908300 0x55d6d690a6f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=118cd1b5220490f9 server_cookie=784ae16716fea068 in_seq=0 out_seq=0 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.952+0000 7fc761ffb640 10 client.25183 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc761ffb640 1 -- 172.21.15.89:0/3416421940 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d6d685c860 con 0x55d6d6908300 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc761ffb640 10 client.25183 handle_command_reply: tid=0 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc782764740 1 client.25183 shutdown 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc782764740 1 -- 172.21.15.89:0/3416421940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68f8780 msgr2=0x55d6d68f5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc782764740 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68f8780 0x55d6d68f5c70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc770009f70 tx=0x7fc770005100 comp rx=0 tx=0).stop 2024-09-15T23:25:26.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc782764740 1 -- 172.21.15.89:0/3416421940 shutdown_connections 2024-09-15T23:25:26.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc782764740 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55d6d6908300 0x55d6d690a6f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc761ffb640 0 client.25183 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:26.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc782764740 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68f8780 0x55d6d68f5c70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc782764740 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc782764740 1 --2- 172.21.15.89:0/3416421940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68f8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc782764740 1 -- 172.21.15.89:0/3416421940 >> 172.21.15.89:0/3416421940 conn(0x55d6d68b2310 msgr2=0x55d6d68efd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.953+0000 7fc782764740 1 -- 172.21.15.89:0/3416421940 shutdown_connections 2024-09-15T23:25:26.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.954+0000 7fc782764740 1 -- 172.21.15.89:0/3416421940 wait complete. 2024-09-15T23:25:26.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.954+0000 7fc782764740 20 client.25183 trim_cache size 0 max 16384 2024-09-15T23:25:26.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.978+0000 7fc782764740 1 Processor -- start 2024-09-15T23:25:26.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.978+0000 7fc782764740 1 -- start start 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.978+0000 7fc782764740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68ed9e0 0x55d6d68f0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.978+0000 7fc782764740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.978+0000 7fc782764740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68f8780 0x55d6d690d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.978+0000 7fc782764740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781acc40 con 0x55d6d68eef50 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.978+0000 7fc782764740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781ae780 con 0x55d6d68f8780 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.978+0000 7fc782764740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781ad310 con 0x55d6d68ed9e0 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77d926640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77d926640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38896/0 (socket says 172.21.15.89:38896) 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77d926640 1 -- 172.21.15.89:0/1514962083 learned_addr learned my addr 172.21.15.89:0/1514962083 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77e127640 1 --2- 172.21.15.89:0/1514962083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68ed9e0 0x55d6d68f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77e928640 1 --2- 172.21.15.89:0/1514962083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68f8780 0x55d6d690d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77d926640 1 -- 172.21.15.89:0/1514962083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68ed9e0 msgr2=0x55d6d68f0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77d926640 1 --2- 172.21.15.89:0/1514962083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68ed9e0 0x55d6d68f0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77d926640 1 -- 172.21.15.89:0/1514962083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68f8780 msgr2=0x55d6d690d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77d926640 1 --2- 172.21.15.89:0/1514962083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68f8780 0x55d6d690d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77d926640 1 -- 172.21.15.89:0/1514962083 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6d68faee0 con 0x55d6d68eef50 2024-09-15T23:25:26.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.979+0000 7fc77d926640 1 --2- 172.21.15.89:0/1514962083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f0a70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc7680cd460 tx=0x7fc768000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eba01e150bbb353 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.980+0000 7fc763fff640 1 -- 172.21.15.89:0/1514962083 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76809b8a0 con 0x55d6d68eef50 2024-09-15T23:25:26.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.980+0000 7fc763fff640 1 -- 172.21.15.89:0/1514962083 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc76809c640 con 0x55d6d68eef50 2024-09-15T23:25:26.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.980+0000 7fc763fff640 1 -- 172.21.15.89:0/1514962083 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76800f740 con 0x55d6d68eef50 2024-09-15T23:25:26.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 -- 172.21.15.89:0/1514962083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 msgr2=0x55d6d68f0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 --2- 172.21.15.89:0/1514962083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f0a70 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc7680cd460 tx=0x7fc768000bc0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 -- 172.21.15.89:0/1514962083 shutdown_connections 2024-09-15T23:25:26.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 --2- 172.21.15.89:0/1514962083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68f8780 0x55d6d690d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 --2- 172.21.15.89:0/1514962083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68eef50 0x55d6d68f0a70 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 --2- 172.21.15.89:0/1514962083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68ed9e0 0x55d6d68f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 -- 172.21.15.89:0/1514962083 >> 172.21.15.89:0/1514962083 conn(0x55d6d68b2310 msgr2=0x55d6d68f45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 -- 172.21.15.89:0/1514962083 shutdown_connections 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 -- 172.21.15.89:0/1514962083 wait complete. 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 -- start start 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.981+0000 7fc782764740 1 Processor -- start 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc782764740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68e7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc782764740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 0x55d6d68e8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc782764740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 0x55d6d690c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc782764740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781ae0b0 con 0x55d6d68f8780 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc782764740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781ad9e0 con 0x55d6d68ed9e0 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc782764740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7781aac20 con 0x55d6d68eef50 2024-09-15T23:25:26.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc782764740 10 client.0 _ll_register_callbacks cb 0x7fc770009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc77e127640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68e7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc77e127640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68e7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:58910/0 (socket says 172.21.15.89:58910) 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc77e127640 1 -- 172.21.15.89:0/2705827594 learned_addr learned my addr 172.21.15.89:0/2705827594 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc77e127640 1 -- 172.21.15.89:0/2705827594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 msgr2=0x55d6d68e8270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc77e127640 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 0x55d6d68e8270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc77e127640 1 -- 172.21.15.89:0/2705827594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 msgr2=0x55d6d690c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.982+0000 7fc77e928640 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 0x55d6d690c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc77e127640 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 0x55d6d690c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc77e127640 1 -- 172.21.15.89:0/2705827594 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6d68e8b70 con 0x55d6d68ed9e0 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc77e928640 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 0x55d6d690c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc77e127640 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68e7d30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc7740041b0 tx=0x7fc774009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11f4492b320e7b7f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:25:26.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc761ffb640 1 -- 172.21.15.89:0/2705827594 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774035050 con 0x55d6d68ed9e0 2024-09-15T23:25:26.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc761ffb640 1 -- 172.21.15.89:0/2705827594 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc7740037f0 con 0x55d6d68ed9e0 2024-09-15T23:25:26.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc782764740 1 -- 172.21.15.89:0/2705827594 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc76809c640 con 0x55d6d68ed9e0 2024-09-15T23:25:26.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc761ffb640 1 -- 172.21.15.89:0/2705827594 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc7740039f0 con 0x55d6d68ed9e0 2024-09-15T23:25:26.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc782764740 10 client.25190 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.983+0000 7fc782764740 1 -- 172.21.15.89:0/2705827594 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d6d690dd90 con 0x55d6d68ed9e0 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.984+0000 7fc761ffb640 1 -- 172.21.15.89:0/2705827594 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc774003b90 con 0x55d6d68ed9e0 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.984+0000 7fc761ffb640 1 -- 172.21.15.89:0/2705827594 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc774059070 con 0x55d6d68ed9e0 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.984+0000 7fc782764740 10 client.25190 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.984+0000 7fc782764740 10 client.25190 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.984+0000 7fc782764740 20 client.25190 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.984+0000 7fc782764740 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55d6d6908300 0x55d6d6979430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.984+0000 7fc782764740 4 client.25190 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.984+0000 7fc782764740 1 -- 172.21.15.89:0/2705827594 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc76809c640 con 0x55d6d6908300 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.985+0000 7fc77d926640 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55d6d6908300 0x55d6d6979430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.985+0000 7fc77d926640 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55d6d6908300 0x55d6d6979430 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=39938ca625a099fd server_cookie=143b3afba84e159a in_seq=0 out_seq=0 2024-09-15T23:25:26.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.985+0000 7fc761ffb640 10 client.25190 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.985+0000 7fc761ffb640 1 -- 172.21.15.89:0/2705827594 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc76809c640 con 0x55d6d6908300 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.985+0000 7fc761ffb640 10 client.25190 handle_command_reply: tid=0 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.985+0000 7fc782764740 1 client.25190 shutdown 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 1 -- 172.21.15.89:0/2705827594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 msgr2=0x55d6d68e7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68e7d30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc7740041b0 tx=0x7fc774009260 comp rx=0 tx=0).stop 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 1 -- 172.21.15.89:0/2705827594 shutdown_connections 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55d6d6908300 0x55d6d6979430 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d6d68f8780 0x55d6d690c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc761ffb640 0 client.25190 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d6d68eef50 0x55d6d68e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 1 --2- 172.21.15.89:0/2705827594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6d68ed9e0 0x55d6d68e7d30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 1 -- 172.21.15.89:0/2705827594 >> 172.21.15.89:0/2705827594 conn(0x55d6d68b2310 msgr2=0x55d6d68f6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 1 -- 172.21.15.89:0/2705827594 shutdown_connections 2024-09-15T23:25:26.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 1 -- 172.21.15.89:0/2705827594 wait complete. 2024-09-15T23:25:26.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.986+0000 7fc782764740 20 client.25190 trim_cache size 0 max 16384 2024-09-15T23:25:26.991 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 -- 172.21.15.89:0/1803228101 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7640a3350 msgr2=0x7fc7640a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7640a3350 0x7fc7640a5810 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x55d6d68eaed0 tx=0x7fc774005a20 comp rx=0 tx=0).stop 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 -- 172.21.15.89:0/1803228101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc778103760 msgr2=0x7fc778198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc778103760 0x7fc778198a60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7780697d0 tx=0x7fc76800cdd0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 -- 172.21.15.89:0/1803228101 shutdown_connections 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7640a3350 0x7fc7640a5810 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7781082e0 0x7fc77819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc778103760 0x7fc778198a60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 --2- 172.21.15.89:0/1803228101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc778102db0 0x7fc778198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 -- 172.21.15.89:0/1803228101 >> 172.21.15.89:0/1803228101 conn(0x7fc7780fc7d0 msgr2=0x7fc778106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 -- 172.21.15.89:0/1803228101 shutdown_connections 2024-09-15T23:25:26.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:26.987+0000 7fc78038b640 1 -- 172.21.15.89:0/1803228101 wait complete. 2024-09-15T23:25:27.773 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:25:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:28.362 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:28 smithi089 ceph-mon[30933]: pgmap v393: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-09-15T23:25:28.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:28 smithi120 ceph-mon[35250]: pgmap v393: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-09-15T23:25:28.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:28 smithi174 ceph-mon[35586]: pgmap v393: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-09-15T23:25:28.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.618+0000 7f79d3b68640 1 -- 172.21.15.89:0/3686114364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc101430 msgr2=0x7f79cc10d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.618+0000 7f79d3b68640 1 --2- 172.21.15.89:0/3686114364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc101430 0x7f79cc10d960 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009930 tx=0x7f79c402a420 comp rx=0 tx=0).stop 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.618+0000 7f79d3b68640 1 -- 172.21.15.89:0/3686114364 shutdown_connections 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.618+0000 7f79d3b68640 1 --2- 172.21.15.89:0/3686114364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc101430 0x7f79cc10d960 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.618+0000 7f79d3b68640 1 --2- 172.21.15.89:0/3686114364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79cc100a90 0x7f79cc100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.618+0000 7f79d3b68640 1 --2- 172.21.15.89:0/3686114364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79cc106a90 0x7f79cc106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.618+0000 7f79d3b68640 1 -- 172.21.15.89:0/3686114364 >> 172.21.15.89:0/3686114364 conn(0x7f79cc0fc7d0 msgr2=0x7f79cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.618+0000 7f79d3b68640 1 -- 172.21.15.89:0/3686114364 shutdown_connections 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.618+0000 7f79d3b68640 1 -- 172.21.15.89:0/3686114364 wait complete. 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.619+0000 7f79d3b68640 1 Processor -- start 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.619+0000 7f79d3b68640 1 -- start start 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.619+0000 7f79d3b68640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79cc100a90 0x7f79cc1a02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.619+0000 7f79d3b68640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc101430 0x7f79cc1a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.619+0000 7f79d3b68640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79cc106a90 0x7f79cc19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.619+0000 7f79d3b68640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc110e90 con 0x7f79cc101430 2024-09-15T23:25:28.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.619+0000 7f79d3b68640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc110d10 con 0x7f79cc100a90 2024-09-15T23:25:28.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.619+0000 7f79d3b68640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc111010 con 0x7f79cc106a90 2024-09-15T23:25:28.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.619+0000 7f79d1103640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc101430 0x7f79cc1a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.619+0000 7f79d1103640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc101430 0x7f79cc1a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38944/0 (socket says 172.21.15.89:38944) 2024-09-15T23:25:28.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.620+0000 7f79d1103640 1 -- 172.21.15.89:0/743676698 learned_addr learned my addr 172.21.15.89:0/743676698 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:28.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.620+0000 7f79d1904640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79cc100a90 0x7f79cc1a02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.620+0000 7f79d2105640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79cc106a90 0x7f79cc19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.620+0000 7f79d1103640 1 -- 172.21.15.89:0/743676698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79cc106a90 msgr2=0x7f79cc19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.620+0000 7f79d1103640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79cc106a90 0x7f79cc19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.620+0000 7f79d1103640 1 -- 172.21.15.89:0/743676698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79cc100a90 msgr2=0x7f79cc1a02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.620+0000 7f79d1103640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79cc100a90 0x7f79cc1a02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.620+0000 7f79d1103640 1 -- 172.21.15.89:0/743676698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79cc19ae90 con 0x7f79cc101430 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.620+0000 7f79d2105640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79cc106a90 0x7f79cc19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.620+0000 7f79d1103640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc101430 0x7f79cc1a0810 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f79b800eab0 tx=0x7f79b800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.621+0000 7f79caffd640 1 -- 172.21.15.89:0/743676698 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b800ce40 con 0x7f79cc101430 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.621+0000 7f79caffd640 1 -- 172.21.15.89:0/743676698 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79b8011950 con 0x7f79cc101430 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.621+0000 7f79caffd640 1 -- 172.21.15.89:0/743676698 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b801aad0 con 0x7f79cc101430 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.621+0000 7f79d3b68640 1 -- 172.21.15.89:0/743676698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79cc19b180 con 0x7f79cc101430 2024-09-15T23:25:28.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.621+0000 7f79d3b68640 1 -- 172.21.15.89:0/743676698 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79cc1a7130 con 0x7f79cc101430 2024-09-15T23:25:28.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.624+0000 7f79d5f41740 1 Processor -- start 2024-09-15T23:25:28.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.624+0000 7f79d5f41740 1 -- start start 2024-09-15T23:25:28.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.624+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a291b40 0x55f56a290090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a290a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a29b370 0x55f56a29d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d5f41740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f56a1f62f0 con 0x55f56a2905d0 2024-09-15T23:25:28.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d5f41740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f56a19dcf0 con 0x55f56a291b40 2024-09-15T23:25:28.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d5f41740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f56a1f22b0 con 0x55f56a29b370 2024-09-15T23:25:28.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d1904640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a290a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d1904640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a290a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38958/0 (socket says 172.21.15.89:38958) 2024-09-15T23:25:28.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d1904640 1 -- 172.21.15.89:0/3316385852 learned_addr learned my addr 172.21.15.89:0/3316385852 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:28.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d2105640 1 --2- 172.21.15.89:0/3316385852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a291b40 0x55f56a290090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d1904640 1 -- 172.21.15.89:0/3316385852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a29b370 msgr2=0x55f56a29d760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d1904640 1 --2- 172.21.15.89:0/3316385852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a29b370 0x55f56a29d760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d1904640 1 -- 172.21.15.89:0/3316385852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a291b40 msgr2=0x55f56a290090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79d1904640 1 --2- 172.21.15.89:0/3316385852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a291b40 0x55f56a290090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.625+0000 7f79caffd640 1 -- 172.21.15.89:0/743676698 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f79b801acb0 con 0x7f79cc101430 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.626+0000 7f79d1904640 1 -- 172.21.15.89:0/3316385852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f56a1823a0 con 0x55f56a2905d0 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.626+0000 7f79d2105640 1 --2- 172.21.15.89:0/3316385852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a291b40 0x55f56a290090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.626+0000 7f79d1904640 1 --2- 172.21.15.89:0/3316385852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a290a10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009cc0 tx=0x7f79c0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ab95103ce14092 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.626+0000 7f79b7fff640 1 -- 172.21.15.89:0/3316385852 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0004230 con 0x55f56a2905d0 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.626+0000 7f79b7fff640 1 -- 172.21.15.89:0/3316385852 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c00043d0 con 0x55f56a2905d0 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.626+0000 7f79b7fff640 1 -- 172.21.15.89:0/3316385852 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c00198e0 con 0x55f56a2905d0 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.627+0000 7f79caffd640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79bc0a3350 0x7f79bc0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.627+0000 7f79caffd640 1 -- 172.21.15.89:0/743676698 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f79b80d07c0 con 0x7f79cc101430 2024-09-15T23:25:28.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.627+0000 7f79d2105640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79bc0a3350 0x7f79bc0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.628+0000 7f79d2105640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79bc0a3350 0x7f79bc0a5810 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f79c4005940 tx=0x7f79c40058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.628+0000 7f79d5f41740 1 -- 172.21.15.89:0/3316385852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 msgr2=0x55f56a290a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.628+0000 7f79d5f41740 1 --2- 172.21.15.89:0/3316385852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a290a10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009cc0 tx=0x7f79c0011810 comp rx=0 tx=0).stop 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.628+0000 7f79d5f41740 1 -- 172.21.15.89:0/3316385852 shutdown_connections 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.628+0000 7f79d5f41740 1 --2- 172.21.15.89:0/3316385852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a29b370 0x55f56a29d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.628+0000 7f79d5f41740 1 --2- 172.21.15.89:0/3316385852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a290a10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.628+0000 7f79d5f41740 1 --2- 172.21.15.89:0/3316385852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a291b40 0x55f56a290090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.628+0000 7f79d5f41740 1 -- 172.21.15.89:0/3316385852 >> 172.21.15.89:0/3316385852 conn(0x55f56a259310 msgr2=0x55f56a28cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.629+0000 7f79d5f41740 1 -- 172.21.15.89:0/3316385852 shutdown_connections 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.629+0000 7f79d5f41740 1 -- 172.21.15.89:0/3316385852 wait complete. 2024-09-15T23:25:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.629+0000 7f79d5f41740 1 -- start start 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.629+0000 7f79d5f41740 1 Processor -- start 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.629+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a2905d0 0x55f56a29adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a291b40 0x55f56a2980a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a29b370 0x55f56a2987e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d5f41740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f56a29e070 con 0x55f56a291b40 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d5f41740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f56a1b1060 con 0x55f56a29b370 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d5f41740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f56a29e1f0 con 0x55f56a2905d0 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d1103640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a291b40 0x55f56a2980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d5f41740 10 client.0 _ll_register_callbacks cb 0x55f569f048c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d1103640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a291b40 0x55f56a2980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38972/0 (socket says 172.21.15.89:38972) 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d1103640 1 -- 172.21.15.89:0/3677134397 learned_addr learned my addr 172.21.15.89:0/3677134397 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d1904640 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a2905d0 0x55f56a29adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d2105640 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a29b370 0x55f56a2987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d1103640 1 -- 172.21.15.89:0/3677134397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a2905d0 msgr2=0x55f56a29adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d1103640 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a2905d0 0x55f56a29adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d1103640 1 -- 172.21.15.89:0/3677134397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a29b370 msgr2=0x55f56a2987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d1103640 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a29b370 0x55f56a2987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d1103640 1 -- 172.21.15.89:0/3677134397 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f569f673c0 con 0x55f56a291b40 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79d1103640 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a291b40 0x55f56a2980a0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f79b80cddb0 tx=0x7f79b809c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253711fc702011f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.630+0000 7f79b57fa640 1 -- 172.21.15.89:0/3677134397 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b8087070 con 0x55f56a291b40 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79b57fa640 1 -- 172.21.15.89:0/3677134397 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f79b8085be0 con 0x55f56a291b40 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79d5f41740 1 -- 172.21.15.89:0/3677134397 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79c00043d0 con 0x55f56a291b40 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79b57fa640 1 -- 172.21.15.89:0/3677134397 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f79b8085e20 con 0x55f56a291b40 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79d5f41740 10 client.15828 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79d5f41740 1 -- 172.21.15.89:0/3677134397 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f56a29e4f0 con 0x55f56a291b40 2024-09-15T23:25:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79b57fa640 1 -- 172.21.15.89:0/3677134397 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c00043d0 con 0x55f56a291b40 2024-09-15T23:25:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79b57fa640 1 -- 172.21.15.89:0/3677134397 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f79b8022070 con 0x55f56a291b40 2024-09-15T23:25:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79d5f41740 10 client.15828 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79d5f41740 10 client.15828 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:25:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79d5f41740 20 client.15828 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79d5f41740 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f56a29f9b0 0x55f56a2a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79d5f41740 4 client.15828 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.631+0000 7f79d5f41740 1 -- 172.21.15.89:0/3677134397 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f56a203860 con 0x55f56a29f9b0 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.632+0000 7f79d1904640 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f56a29f9b0 0x55f56a2a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.632+0000 7f79b57fa640 10 client.15828 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.632+0000 7f79d1904640 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f56a29f9b0 0x55f56a2a1da0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=af1d66ceaa20f63 server_cookie=21f054fccaf66c99 in_seq=0 out_seq=0 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79b57fa640 1 -- 172.21.15.89:0/3677134397 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f56a203860 con 0x55f56a29f9b0 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79b57fa640 10 client.15828 handle_command_reply: tid=0 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 client.15828 shutdown 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 -- 172.21.15.89:0/3677134397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a291b40 msgr2=0x55f56a2980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a291b40 0x55f56a2980a0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f79b80cddb0 tx=0x7f79b809c780 comp rx=0 tx=0).stop 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 -- 172.21.15.89:0/3677134397 shutdown_connections 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f56a29f9b0 0x55f56a2a1da0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a29b370 0x55f56a2987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a291b40 0x55f56a2980a0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 --2- 172.21.15.89:0/3677134397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a2905d0 0x55f56a29adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79b57fa640 0 client.15828 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 -- 172.21.15.89:0/3677134397 >> 172.21.15.89:0/3677134397 conn(0x55f56a259310 msgr2=0x55f56a292930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 -- 172.21.15.89:0/3677134397 shutdown_connections 2024-09-15T23:25:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 1 -- 172.21.15.89:0/3677134397 wait complete. 2024-09-15T23:25:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.633+0000 7f79d5f41740 20 client.15828 trim_cache size 0 max 16384 2024-09-15T23:25:28.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.646+0000 7f79d5f41740 1 Processor -- start 2024-09-15T23:25:28.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.646+0000 7f79d5f41740 1 -- start start 2024-09-15T23:25:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.646+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a2905d0 0x55f56a297310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a291b40 0x55f56a29c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a29b370 0x55f56a29cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d5f41740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1accc0 con 0x55f56a29b370 2024-09-15T23:25:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d5f41740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1ae800 con 0x55f56a2905d0 2024-09-15T23:25:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d5f41740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1ad390 con 0x55f56a291b40 2024-09-15T23:25:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d2105640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a29b370 0x55f56a29cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d2105640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a29b370 0x55f56a29cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38976/0 (socket says 172.21.15.89:38976) 2024-09-15T23:25:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d2105640 1 -- 172.21.15.89:0/571214101 learned_addr learned my addr 172.21.15.89:0/571214101 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d1904640 1 --2- 172.21.15.89:0/571214101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a2905d0 0x55f56a297310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d2105640 1 -- 172.21.15.89:0/571214101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a291b40 msgr2=0x55f56a29c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d2105640 1 --2- 172.21.15.89:0/571214101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a291b40 0x55f56a29c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d2105640 1 -- 172.21.15.89:0/571214101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a2905d0 msgr2=0x55f56a297310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d2105640 1 --2- 172.21.15.89:0/571214101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a2905d0 0x55f56a297310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d2105640 1 -- 172.21.15.89:0/571214101 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f56a2965f0 con 0x55f56a29b370 2024-09-15T23:25:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.647+0000 7f79d2105640 1 --2- 172.21.15.89:0/571214101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a29b370 0x55f56a29cca0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009230 tx=0x7f79c403e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53ada080098ebfe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.648+0000 7f79b77fe640 1 -- 172.21.15.89:0/571214101 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4035040 con 0x55f56a29b370 2024-09-15T23:25:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.648+0000 7f79b77fe640 1 -- 172.21.15.89:0/571214101 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c4005520 con 0x55f56a29b370 2024-09-15T23:25:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.648+0000 7f79b77fe640 1 -- 172.21.15.89:0/571214101 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4004e60 con 0x55f56a29b370 2024-09-15T23:25:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.649+0000 7f79d5f41740 1 -- 172.21.15.89:0/571214101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a29b370 msgr2=0x55f56a29cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.649+0000 7f79d5f41740 1 --2- 172.21.15.89:0/571214101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a29b370 0x55f56a29cca0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009230 tx=0x7f79c403e000 comp rx=0 tx=0).stop 2024-09-15T23:25:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.649+0000 7f79d5f41740 1 -- 172.21.15.89:0/571214101 shutdown_connections 2024-09-15T23:25:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.649+0000 7f79d5f41740 1 --2- 172.21.15.89:0/571214101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a29b370 0x55f56a29cca0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.649+0000 7f79d5f41740 1 --2- 172.21.15.89:0/571214101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a291b40 0x55f56a29c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.649+0000 7f79d5f41740 1 --2- 172.21.15.89:0/571214101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a2905d0 0x55f56a297310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.649+0000 7f79d5f41740 1 -- 172.21.15.89:0/571214101 >> 172.21.15.89:0/571214101 conn(0x55f56a259310 msgr2=0x55f56a292930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.650+0000 7f79d5f41740 1 -- 172.21.15.89:0/571214101 shutdown_connections 2024-09-15T23:25:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.650+0000 7f79d5f41740 1 -- 172.21.15.89:0/571214101 wait complete. 2024-09-15T23:25:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.650+0000 7f79d5f41740 1 -- start start 2024-09-15T23:25:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.650+0000 7f79d5f41740 1 Processor -- start 2024-09-15T23:25:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a2b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a291b40 0x55f56a2b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d5f41740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a29b370 0x55f56a2b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d5f41740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1ae130 con 0x55f56a2905d0 2024-09-15T23:25:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d5f41740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1ada60 con 0x55f56a29b370 2024-09-15T23:25:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d5f41740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc1aad00 con 0x55f56a291b40 2024-09-15T23:25:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d5f41740 10 client.0 _ll_register_callbacks cb 0x7f79b801a400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:28.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1904640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a2b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1904640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a2b30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38990/0 (socket says 172.21.15.89:38990) 2024-09-15T23:25:28.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1904640 1 -- 172.21.15.89:0/584970328 learned_addr learned my addr 172.21.15.89:0/584970328 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d2105640 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a29b370 0x55f56a2b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1103640 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a291b40 0x55f56a2b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1904640 1 -- 172.21.15.89:0/584970328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a291b40 msgr2=0x55f56a2b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1904640 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a291b40 0x55f56a2b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1904640 1 -- 172.21.15.89:0/584970328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a29b370 msgr2=0x55f56a2b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1904640 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a29b370 0x55f56a2b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1904640 1 -- 172.21.15.89:0/584970328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f56a3263d0 con 0x55f56a2905d0 2024-09-15T23:25:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1103640 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a291b40 0x55f56a2b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79d1904640 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a2b30e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009cc0 tx=0x7f79c0019480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407f417cc34cec3e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.651+0000 7f79b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.652+0000 7f79b57fa640 1 -- 172.21.15.89:0/584970328 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c00050b0 con 0x55f56a2905d0 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.652+0000 7f79b57fa640 1 -- 172.21.15.89:0/584970328 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f79c0005250 con 0x55f56a2905d0 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.652+0000 7f79d5f41740 1 -- 172.21.15.89:0/584970328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79c4005520 con 0x55f56a2905d0 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.652+0000 7f79b57fa640 1 -- 172.21.15.89:0/584970328 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f79c0039050 con 0x55f56a2905d0 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.652+0000 7f79d5f41740 10 client.15834 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.652+0000 7f79d5f41740 1 -- 172.21.15.89:0/584970328 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f56a327650 con 0x55f56a2905d0 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.652+0000 7f79b57fa640 1 -- 172.21.15.89:0/584970328 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c4005520 con 0x55f56a2905d0 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.652+0000 7f79b57fa640 1 -- 172.21.15.89:0/584970328 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f79c002e020 con 0x55f56a2905d0 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.653+0000 7f79d5f41740 10 client.15834 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.653+0000 7f79d5f41740 10 client.15834 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.653+0000 7f79d5f41740 20 client.15834 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.653+0000 7f79d5f41740 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f56a327b90 0x55f56a329f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.653+0000 7f79d5f41740 4 client.15834 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.653+0000 7f79d5f41740 1 -- 172.21.15.89:0/584970328 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f56a1fdd70 con 0x55f56a327b90 2024-09-15T23:25:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.654+0000 7f79d1103640 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f56a327b90 0x55f56a329f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.654+0000 7f79d1103640 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f56a327b90 0x55f56a329f80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d419c655fff4c0e9 server_cookie=68add00ab3b53627 in_seq=0 out_seq=0 2024-09-15T23:25:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.654+0000 7f79b57fa640 10 client.15834 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.654+0000 7f79b57fa640 1 -- 172.21.15.89:0/584970328 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f56a1fdd70 con 0x55f56a327b90 2024-09-15T23:25:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.654+0000 7f79b57fa640 10 client.15834 handle_command_reply: tid=0 2024-09-15T23:25:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.654+0000 7f79d5f41740 1 client.15834 shutdown 2024-09-15T23:25:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 1 -- 172.21.15.89:0/584970328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 msgr2=0x55f56a2b30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a2b30e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f79c0009cc0 tx=0x7f79c0019480 comp rx=0 tx=0).stop 2024-09-15T23:25:28.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 1 -- 172.21.15.89:0/584970328 shutdown_connections 2024-09-15T23:25:28.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f56a327b90 0x55f56a329f80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f56a29b370 0x55f56a2b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f56a291b40 0x55f56a2b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 1 --2- 172.21.15.89:0/584970328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f56a2905d0 0x55f56a2b30e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 1 -- 172.21.15.89:0/584970328 >> 172.21.15.89:0/584970328 conn(0x55f56a259310 msgr2=0x55f56a299900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:28.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79b57fa640 0 client.15834 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:28.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 1 -- 172.21.15.89:0/584970328 shutdown_connections 2024-09-15T23:25:28.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 1 -- 172.21.15.89:0/584970328 wait complete. 2024-09-15T23:25:28.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.655+0000 7f79d5f41740 20 client.15834 trim_cache size 0 max 16384 2024-09-15T23:25:28.662 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:28.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.656+0000 7f79d3b68640 1 -- 172.21.15.89:0/743676698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79bc0a3350 msgr2=0x7f79bc0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:28.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.656+0000 7f79d3b68640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79bc0a3350 0x7f79bc0a5810 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f79c4005940 tx=0x7f79c40058d0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.656+0000 7f79d3b68640 1 -- 172.21.15.89:0/743676698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc101430 msgr2=0x7f79cc1a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:28.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.656+0000 7f79d3b68640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc101430 0x7f79cc1a0810 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f79b800eab0 tx=0x7f79b800ef70 comp rx=0 tx=0).stop 2024-09-15T23:25:28.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.656+0000 7f79d3b68640 1 -- 172.21.15.89:0/743676698 shutdown_connections 2024-09-15T23:25:28.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.656+0000 7f79d3b68640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79bc0a3350 0x7f79bc0a5810 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.656+0000 7f79d3b68640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79cc106a90 0x7f79cc19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.656+0000 7f79d3b68640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79cc101430 0x7f79cc1a0810 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.657+0000 7f79d3b68640 1 --2- 172.21.15.89:0/743676698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79cc100a90 0x7f79cc1a02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:28.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.657+0000 7f79d3b68640 1 -- 172.21.15.89:0/743676698 >> 172.21.15.89:0/743676698 conn(0x7f79cc0fc7d0 msgr2=0x7f79cc1045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:28.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.657+0000 7f79d3b68640 1 -- 172.21.15.89:0/743676698 shutdown_connections 2024-09-15T23:25:28.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:28.657+0000 7f79d3b68640 1 -- 172.21.15.89:0/743676698 wait complete. 2024-09-15T23:25:29.367 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:25:29.652 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.177+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/299072217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58074f40 msgr2=0x7f3c5810de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.177+0000 7f3c5fa5f640 1 --2- 172.21.15.89:0/299072217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58074f40 0x7f3c5810de80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009900 tx=0x7f3c4802a480 comp rx=0 tx=0).stop 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.177+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/299072217 shutdown_connections 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.177+0000 7f3c5fa5f640 1 --2- 172.21.15.89:0/299072217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c5810e3c0 0x7f3c58110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.177+0000 7f3c5fa5f640 1 --2- 172.21.15.89:0/299072217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58074f40 0x7f3c5810de80 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.177+0000 7f3c5fa5f640 1 --2- 172.21.15.89:0/299072217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c58074620 0x7f3c58074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.177+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/299072217 >> 172.21.15.89:0/299072217 conn(0x7f3c580fd880 msgr2=0x7f3c580ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.177+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/299072217 shutdown_connections 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.177+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/299072217 wait complete. 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.178+0000 7f3c5fa5f640 1 Processor -- start 2024-09-15T23:25:30.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.178+0000 7f3c5fa5f640 1 -- start start 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.178+0000 7f3c5fa5f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c58074620 0x7f3c5819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.178+0000 7f3c5fa5f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58074f40 0x7f3c5819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.178+0000 7f3c5cffa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58074f40 0x7f3c5819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.178+0000 7f3c5fa5f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5810e3c0 0x7f3c5819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.178+0000 7f3c5cffa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58074f40 0x7f3c5819a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39008/0 (socket says 172.21.15.89:39008) 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.178+0000 7f3c5d7fb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c58074620 0x7f3c5819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.178+0000 7f3c5d7fb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c58074620 0x7f3c5819a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:47466/0 (socket says 172.21.15.89:47466) 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.178+0000 7f3c5d7fb640 1 -- 172.21.15.89:0/660122277 learned_addr learned my addr 172.21.15.89:0/660122277 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c58114140 con 0x7f3c58074f40 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c58113fc0 con 0x7f3c5810e3c0 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c581142c0 con 0x7f3c58074620 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c5dffc640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5810e3c0 0x7f3c5819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c5cffa640 1 -- 172.21.15.89:0/660122277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c58074620 msgr2=0x7f3c5819a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c58074620 0x7f3c5819a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c5cffa640 1 -- 172.21.15.89:0/660122277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5810e3c0 msgr2=0x7f3c5819ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5810e3c0 0x7f3c5819ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c5cffa640 1 -- 172.21.15.89:0/660122277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5819f410 con 0x7f3c58074f40 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58074f40 0x7f3c5819a9a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009850 tx=0x7f3c48030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c4e7fc640 1 -- 172.21.15.89:0/660122277 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4803f070 con 0x7f3c58074f40 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.179+0000 7f3c4e7fc640 1 -- 172.21.15.89:0/660122277 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c48004460 con 0x7f3c58074f40 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.180+0000 7f3c4e7fc640 1 -- 172.21.15.89:0/660122277 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c48035030 con 0x7f3c58074f40 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.180+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5819f6a0 con 0x7f3c58074f40 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.180+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5819fa70 con 0x7f3c58074f40 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.182+0000 7f3c4e7fc640 1 -- 172.21.15.89:0/660122277 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 19) ==== 138406+0+0 (secure 0 0 0) 0x7f3c48042070 con 0x7f3c58074f40 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.183+0000 7f3c4e7fc640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3350 0x7f3c440a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.183+0000 7f3c4e7fc640 1 -- 172.21.15.89:0/660122277 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c480f0760 con 0x7f3c58074f40 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.183+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3350 0x7f3c440a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.183+0000 7f3c61e38740 1 Processor -- start 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.183+0000 7f3c61e38740 1 -- start start 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3350 0x7f3c440a5810 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3c54005fd0 tx=0x7f3c5400b480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b09090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 0x565027b09a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b14370 0x565027b16760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c61e38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x565027a6f2f0 con 0x565027b0ab40 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c61e38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565027a16cf0 con 0x565027b095d0 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c61e38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565027a6b2b0 con 0x565027b14370 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5d7fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b09090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5d7fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b09090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39010/0 (socket says 172.21.15.89:39010) 2024-09-15T23:25:30.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5d7fb640 1 -- 172.21.15.89:0/722361436 learned_addr learned my addr 172.21.15.89:0/722361436 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5dffc640 1 --2- 172.21.15.89:0/722361436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b14370 0x565027b16760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/722361436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 0x565027b09a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5d7fb640 1 -- 172.21.15.89:0/722361436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b14370 msgr2=0x565027b16760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/722361436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b14370 0x565027b16760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5d7fb640 1 -- 172.21.15.89:0/722361436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 msgr2=0x565027b09a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/722361436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 0x565027b09a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.184+0000 7f3c5d7fb640 1 -- 172.21.15.89:0/722361436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650279fb3a0 con 0x565027b0ab40 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.185+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/722361436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b09090 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3c54011010 tx=0x7f3c54013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2e0eb8acdab313 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.185+0000 7f3c437fe640 1 -- 172.21.15.89:0/722361436 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54005220 con 0x565027b0ab40 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.185+0000 7f3c437fe640 1 -- 172.21.15.89:0/722361436 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c54004c80 con 0x565027b0ab40 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.186+0000 7f3c437fe640 1 -- 172.21.15.89:0/722361436 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5401eaa0 con 0x565027b0ab40 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.186+0000 7f3c61e38740 1 -- 172.21.15.89:0/722361436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 msgr2=0x565027b09090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.186+0000 7f3c61e38740 1 --2- 172.21.15.89:0/722361436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b09090 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3c54011010 tx=0x7f3c54013000 comp rx=0 tx=0).stop 2024-09-15T23:25:30.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.186+0000 7f3c61e38740 1 -- 172.21.15.89:0/722361436 shutdown_connections 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.186+0000 7f3c61e38740 1 --2- 172.21.15.89:0/722361436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b14370 0x565027b16760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.186+0000 7f3c61e38740 1 --2- 172.21.15.89:0/722361436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 0x565027b09a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.186+0000 7f3c61e38740 1 --2- 172.21.15.89:0/722361436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b09090 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.186+0000 7f3c61e38740 1 -- 172.21.15.89:0/722361436 >> 172.21.15.89:0/722361436 conn(0x565027ad2310 msgr2=0x565027b05c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.186+0000 7f3c61e38740 1 -- 172.21.15.89:0/722361436 shutdown_connections 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.186+0000 7f3c61e38740 1 -- 172.21.15.89:0/722361436 wait complete. 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.187+0000 7f3c61e38740 1 -- start start 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.187+0000 7f3c61e38740 1 Processor -- start 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.187+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b095d0 0x565027b13c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.187+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b10f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5cffa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b10f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5cffa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b10f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39024/0 (socket says 172.21.15.89:39024) 2024-09-15T23:25:30.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 0x565027b11480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5d7fb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b095d0 0x565027b13c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5cffa640 1 -- 172.21.15.89:0/2268633821 learned_addr learned my addr 172.21.15.89:0/2268633821 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x565027b17110 con 0x565027b0ab40 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565027a2a060 con 0x565027b14370 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565027b17290 con 0x565027b095d0 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5dffc640 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 0x565027b11480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5cffa640 1 -- 172.21.15.89:0/2268633821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b095d0 msgr2=0x565027b13c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b095d0 0x565027b13c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5cffa640 1 -- 172.21.15.89:0/2268633821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 msgr2=0x565027b11480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 0x565027b11480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5cffa640 1 -- 172.21.15.89:0/2268633821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650277e03c0 con 0x565027b0ab40 2024-09-15T23:25:30.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5dffc640 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 0x565027b11480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.188+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b10f40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3c480b91a0 tx=0x7f3c480bc310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94304ef98e62e42b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c61e38740 10 client.0 _ll_register_callbacks cb 0x56502777d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c40ff9640 1 -- 172.21.15.89:0/2268633821 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c480bc7d0 con 0x565027b0ab40 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c40ff9640 1 -- 172.21.15.89:0/2268633821 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c480035f0 con 0x565027b0ab40 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c54004c80 con 0x565027b0ab40 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c40ff9640 1 -- 172.21.15.89:0/2268633821 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c480c0b80 con 0x565027b0ab40 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c61e38740 10 client.15846 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({fsmap=29}) -- 0x565027b22680 con 0x565027b0ab40 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c40ff9640 1 -- 172.21.15.89:0/2268633821 <== mon.0 v2:172.21.15.89:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3c480b8e40 con 0x565027b0ab40 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c61e38740 10 client.15846 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.189+0000 7f3c61e38740 10 client.15846 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:25:30.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.190+0000 7f3c61e38740 20 client.15846 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:30.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.190+0000 7f3c61e38740 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x565027b23dc0 0x565027b26280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.190+0000 7f3c61e38740 4 client.15846 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:30.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.190+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565027a7c860 con 0x565027b23dc0 2024-09-15T23:25:30.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.190+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x565027b23dc0 0x565027b26280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.190+0000 7f3c40ff9640 10 client.15846 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:30.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.190+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x565027b23dc0 0x565027b26280 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b8c24854d345401 server_cookie=c1372f0a97094f37 in_seq=0 out_seq=0 2024-09-15T23:25:30.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c40ff9640 1 -- 172.21.15.89:0/2268633821 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565027a7c860 con 0x565027b23dc0 2024-09-15T23:25:30.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c40ff9640 10 client.15846 handle_command_reply: tid=0 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 client.15846 shutdown 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 msgr2=0x565027b10f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b10f40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3c480b91a0 tx=0x7f3c480bc310 comp rx=0 tx=0).stop 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 shutdown_connections 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 0x565027b11480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b0ab40 0x565027b10f40 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x565027b23dc0 0x565027b26280 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 --2- 172.21.15.89:0/2268633821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b095d0 0x565027b13c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 >> 172.21.15.89:0/2268633821 conn(0x565027ad2310 msgr2=0x565027b0b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c40ff9640 0 client.15846 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 shutdown_connections 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.191+0000 7f3c61e38740 1 -- 172.21.15.89:0/2268633821 wait complete. 2024-09-15T23:25:30.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.192+0000 7f3c61e38740 20 client.15846 trim_cache size 0 max 16384 2024-09-15T23:25:30.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c61e38740 1 Processor -- start 2024-09-15T23:25:30.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c61e38740 1 -- start start 2024-09-15T23:25:30.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 0x565027b20400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 0x565027b2bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b14370 0x565027b2c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c61e38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c581b0fc0 con 0x565027b14370 2024-09-15T23:25:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c61e38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c581b2b00 con 0x565027b095d0 2024-09-15T23:25:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c61e38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c581b1690 con 0x565027b0ab40 2024-09-15T23:25:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c5dffc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b14370 0x565027b2c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c5dffc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b14370 0x565027b2c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39032/0 (socket says 172.21.15.89:39032) 2024-09-15T23:25:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c5dffc640 1 -- 172.21.15.89:0/3550958706 learned_addr learned my addr 172.21.15.89:0/3550958706 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.217+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/3550958706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 0x565027b20400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/3550958706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 0x565027b2bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c5d7fb640 1 -- 172.21.15.89:0/3550958706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 msgr2=0x565027b2bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:30.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/3550958706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 0x565027b2bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c5d7fb640 1 -- 172.21.15.89:0/3550958706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b14370 msgr2=0x565027b2c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:30.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/3550958706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b14370 0x565027b2c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c5d7fb640 1 -- 172.21.15.89:0/3550958706 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565027b06960 con 0x565027b095d0 2024-09-15T23:25:30.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/3550958706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 0x565027b2bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:30.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/3550958706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 0x565027b20400 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3c540058c0 tx=0x7f3c5401e680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2151573d4128141b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:30.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c42ffd640 1 -- 172.21.15.89:0/3550958706 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54004620 con 0x565027b095d0 2024-09-15T23:25:30.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c42ffd640 1 -- 172.21.15.89:0/3550958706 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c5402a070 con 0x565027b095d0 2024-09-15T23:25:30.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.218+0000 7f3c42ffd640 1 -- 172.21.15.89:0/3550958706 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5401e950 con 0x565027b095d0 2024-09-15T23:25:30.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.219+0000 7f3c61e38740 1 -- 172.21.15.89:0/3550958706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 msgr2=0x565027b20400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:30.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.219+0000 7f3c61e38740 1 --2- 172.21.15.89:0/3550958706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 0x565027b20400 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3c540058c0 tx=0x7f3c5401e680 comp rx=0 tx=0).stop 2024-09-15T23:25:30.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.219+0000 7f3c61e38740 1 -- 172.21.15.89:0/3550958706 shutdown_connections 2024-09-15T23:25:30.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.219+0000 7f3c61e38740 1 --2- 172.21.15.89:0/3550958706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b14370 0x565027b2c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.219+0000 7f3c61e38740 1 --2- 172.21.15.89:0/3550958706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 0x565027b2bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.219+0000 7f3c61e38740 1 --2- 172.21.15.89:0/3550958706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b095d0 0x565027b20400 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.219+0000 7f3c61e38740 1 -- 172.21.15.89:0/3550958706 >> 172.21.15.89:0/3550958706 conn(0x565027ad2310 msgr2=0x565027b0f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:30.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.219+0000 7f3c61e38740 1 -- 172.21.15.89:0/3550958706 shutdown_connections 2024-09-15T23:25:30.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.220+0000 7f3c61e38740 1 -- 172.21.15.89:0/3550958706 wait complete. 2024-09-15T23:25:30.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.220+0000 7f3c61e38740 1 -- start start 2024-09-15T23:25:30.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.220+0000 7f3c61e38740 1 Processor -- start 2024-09-15T23:25:30.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.220+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b095d0 0x565027b176b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.220+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 0x565027b17bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.220+0000 7f3c61e38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 0x565027b8cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.220+0000 7f3c61e38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c581b2430 con 0x565027b095d0 2024-09-15T23:25:30.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.220+0000 7f3c61e38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c581b1d60 con 0x565027b14370 2024-09-15T23:25:30.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.220+0000 7f3c61e38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c581aefa0 con 0x565027b0ab40 2024-09-15T23:25:30.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c61e38740 10 client.0 _ll_register_callbacks cb 0x7f3c48005ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:30.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5d7fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b095d0 0x565027b176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5d7fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b095d0 0x565027b176b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39046/0 (socket says 172.21.15.89:39046) 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5d7fb640 1 -- 172.21.15.89:0/222362691 learned_addr learned my addr 172.21.15.89:0/222362691 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 0x565027b17bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5dffc640 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 0x565027b8cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5cffa640 1 -- 172.21.15.89:0/222362691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 msgr2=0x565027b8cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 0x565027b8cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5cffa640 1 -- 172.21.15.89:0/222362691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b095d0 msgr2=0x565027b176b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b095d0 0x565027b176b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5cffa640 1 -- 172.21.15.89:0/222362691 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565027b8d090 con 0x565027b0ab40 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.221+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b095d0 0x565027b176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.222+0000 7f3c5cffa640 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 0x565027b17bf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3c480baa80 tx=0x7f3c48045000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf61a8766b72baf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.222+0000 7f3c40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:25:30.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.222+0000 7f3c40ff9640 1 -- 172.21.15.89:0/222362691 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c480bc7d0 con 0x565027b0ab40 2024-09-15T23:25:30.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c40ff9640 1 -- 172.21.15.89:0/222362691 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c480f03b0 con 0x565027b0ab40 2024-09-15T23:25:30.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c61e38740 1 -- 172.21.15.89:0/222362691 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c5402a070 con 0x565027b0ab40 2024-09-15T23:25:30.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c40ff9640 1 -- 172.21.15.89:0/222362691 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c480bc130 con 0x565027b0ab40 2024-09-15T23:25:30.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c61e38740 10 client.25213 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:30.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c61e38740 1 -- 172.21.15.89:0/222362691 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x565027b8e210 con 0x565027b0ab40 2024-09-15T23:25:30.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c40ff9640 1 -- 172.21.15.89:0/222362691 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c54004c80 con 0x565027b0ab40 2024-09-15T23:25:30.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c40ff9640 1 -- 172.21.15.89:0/222362691 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3c480c4fb0 con 0x565027b0ab40 2024-09-15T23:25:30.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c61e38740 10 client.25213 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:30.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c61e38740 10 client.25213 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:25:30.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c61e38740 20 client.25213 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:30.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c61e38740 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x565027b8e7a0 0x565027b90b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:30.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c61e38740 4 client.25213 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:30.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.223+0000 7f3c61e38740 1 -- 172.21.15.89:0/222362691 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f3c5402a070 con 0x565027b8e7a0 2024-09-15T23:25:30.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.224+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x565027b8e7a0 0x565027b90b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:30.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.224+0000 7f3c40ff9640 10 client.25213 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:30.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.224+0000 7f3c5d7fb640 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x565027b8e7a0 0x565027b90b90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=669688452f40646b server_cookie=6178733c4fcaaf91 in_seq=0 out_seq=0 2024-09-15T23:25:30.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.224+0000 7f3c40ff9640 1 -- 172.21.15.89:0/222362691 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f3c5402a070 con 0x565027b8e7a0 2024-09-15T23:25:30.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.224+0000 7f3c40ff9640 10 client.25213 handle_command_reply: tid=0 2024-09-15T23:25:30.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.224+0000 7f3c61e38740 1 client.25213 shutdown 2024-09-15T23:25:30.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 1 -- 172.21.15.89:0/222362691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 msgr2=0x565027b17bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:30.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 0x565027b17bf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3c480baa80 tx=0x7f3c48045000 comp rx=0 tx=0).stop 2024-09-15T23:25:30.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 1 -- 172.21.15.89:0/222362691 shutdown_connections 2024-09-15T23:25:30.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565027b14370 0x565027b8cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x565027b8e7a0 0x565027b90b90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.231 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:30.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565027b0ab40 0x565027b17bf0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 1 --2- 172.21.15.89:0/222362691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565027b095d0 0x565027b176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 1 -- 172.21.15.89:0/222362691 >> 172.21.15.89:0/222362691 conn(0x565027ad2310 msgr2=0x565027b0b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c40ff9640 0 client.25213 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 1 -- 172.21.15.89:0/222362691 shutdown_connections 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 1 -- 172.21.15.89:0/222362691 wait complete. 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.225+0000 7f3c61e38740 20 client.25213 trim_cache size 0 max 16384 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3350 msgr2=0x7f3c440a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3350 0x7f3c440a5810 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3c54005fd0 tx=0x7f3c5400b480 comp rx=0 tx=0).stop 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58074f40 msgr2=0x7f3c5819a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58074f40 0x7f3c5819a9a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009850 tx=0x7f3c48030650 comp rx=0 tx=0).stop 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 shutdown_connections 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3350 0x7f3c440a5810 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5810e3c0 0x7f3c5819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58074f40 0x7f3c5819a9a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 --2- 172.21.15.89:0/660122277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c58074620 0x7f3c5819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:30.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 >> 172.21.15.89:0/660122277 conn(0x7f3c580fd880 msgr2=0x7f3c580fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:30.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 shutdown_connections 2024-09-15T23:25:30.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:30.226+0000 7f3c5fa5f640 1 -- 172.21.15.89:0/660122277 wait complete. 2024-09-15T23:25:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:30 smithi120 ceph-mon[35250]: pgmap v394: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-15T23:25:30.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:30 smithi174 ceph-mon[35586]: pgmap v394: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-15T23:25:30.523 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:30 smithi089 ceph-mon[30933]: pgmap v394: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 12 op/s 2024-09-15T23:25:30.995 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:25:31.316 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:31.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.844+0000 7f826edf9640 1 -- 172.21.15.89:0/4179112843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8268102570 msgr2=0x7f82681029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.844+0000 7f826edf9640 1 --2- 172.21.15.89:0/4179112843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8268102570 0x7f82681029b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8254009900 tx=0x7f825402a480 comp rx=0 tx=0).stop 2024-09-15T23:25:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.845+0000 7f826edf9640 1 -- 172.21.15.89:0/4179112843 shutdown_connections 2024-09-15T23:25:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.845+0000 7f826edf9640 1 --2- 172.21.15.89:0/4179112843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8268102ef0 0x7f826810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.845+0000 7f826edf9640 1 --2- 172.21.15.89:0/4179112843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8268102570 0x7f82681029b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.845+0000 7f826edf9640 1 --2- 172.21.15.89:0/4179112843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268100e20 0x7f8268101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.845+0000 7f826edf9640 1 -- 172.21.15.89:0/4179112843 >> 172.21.15.89:0/4179112843 conn(0x7f82680fc7d0 msgr2=0x7f82680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.845+0000 7f826edf9640 1 -- 172.21.15.89:0/4179112843 shutdown_connections 2024-09-15T23:25:31.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.845+0000 7f826edf9640 1 -- 172.21.15.89:0/4179112843 wait complete. 2024-09-15T23:25:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.845+0000 7f826edf9640 1 Processor -- start 2024-09-15T23:25:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f826edf9640 1 -- start start 2024-09-15T23:25:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f826edf9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8268100e20 0x7f82681993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f826edf9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102570 0x7f8268199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f826edf9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8268102ef0 0x7f82681a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f826edf9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8268110f80 con 0x7f8268102ef0 2024-09-15T23:25:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f826edf9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8268110e00 con 0x7f8268100e20 2024-09-15T23:25:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f826edf9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8268111100 con 0x7f8268102570 2024-09-15T23:25:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f8267fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102570 0x7f8268199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f826d396640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8268102ef0 0x7f82681a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f8267fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102570 0x7f8268199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:47534/0 (socket says 172.21.15.89:47534) 2024-09-15T23:25:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f8267fff640 1 -- 172.21.15.89:0/1911983029 learned_addr learned my addr 172.21.15.89:0/1911983029 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.846+0000 7f826cb95640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8268100e20 0x7f82681993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.847+0000 7f8267fff640 1 -- 172.21.15.89:0/1911983029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8268100e20 msgr2=0x7f82681993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.847+0000 7f8267fff640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8268100e20 0x7f82681993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.847+0000 7f8267fff640 1 -- 172.21.15.89:0/1911983029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8268102ef0 msgr2=0x7f82681a4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.847+0000 7f8267fff640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8268102ef0 0x7f82681a4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.847+0000 7f8267fff640 1 -- 172.21.15.89:0/1911983029 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82681a4990 con 0x7f8268102570 2024-09-15T23:25:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.847+0000 7f8267fff640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102570 0x7f8268199910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8254006010 tx=0x7f825403ac10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.847+0000 7f8265ffb640 1 -- 172.21.15.89:0/1911983029 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254035070 con 0x7f8268102570 2024-09-15T23:25:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.847+0000 7f8265ffb640 1 -- 172.21.15.89:0/1911983029 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8254003ce0 con 0x7f8268102570 2024-09-15T23:25:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.847+0000 7f826edf9640 1 -- 172.21.15.89:0/1911983029 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82681971c0 con 0x7f8268102570 2024-09-15T23:25:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.848+0000 7f8265ffb640 1 -- 172.21.15.89:0/1911983029 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254031da0 con 0x7f8268102570 2024-09-15T23:25:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.848+0000 7f826edf9640 1 -- 172.21.15.89:0/1911983029 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8268197630 con 0x7f8268102570 2024-09-15T23:25:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.851+0000 7f8265ffb640 1 -- 172.21.15.89:0/1911983029 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8254031660 con 0x7f8268102570 2024-09-15T23:25:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.851+0000 7f82711d2740 1 Processor -- start 2024-09-15T23:25:31.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f82711d2740 1 -- start start 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f82711d2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1caaf8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f82711d2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf85d0 0x55c1caaf8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f82711d2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab05760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f82711d2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c1caa5e2f0 con 0x55c1cab03370 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f82711d2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c1caa05cf0 con 0x55c1caaf9b40 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f82711d2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c1caa5a2b0 con 0x55c1caaf85d0 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f826cb95640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf85d0 0x55c1caaf8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f826d396640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1caaf8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f826d396640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1caaf8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:59026/0 (socket says 172.21.15.89:59026) 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f826cb95640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf85d0 0x55c1caaf8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:47546/0 (socket says 172.21.15.89:47546) 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f826d396640 1 -- 172.21.15.89:0/3590479299 learned_addr learned my addr 172.21.15.89:0/3590479299 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f8265ffb640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82580a3340 0x7f82580a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f8267fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab05760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.852+0000 7f8265ffb640 1 -- 172.21.15.89:0/1911983029 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f82540f3f70 con 0x7f8268102570 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.853+0000 7f826d396640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82580a3340 0x7f82580a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.853+0000 7f826d396640 1 -- 172.21.15.89:0/3590479299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf85d0 msgr2=0x55c1caaf8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.853+0000 7f826d396640 1 --2- 172.21.15.89:0/3590479299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf85d0 0x55c1caaf8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.853+0000 7f826d396640 1 -- 172.21.15.89:0/3590479299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 msgr2=0x55c1cab05760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.853+0000 7f826d396640 1 --2- 172.21.15.89:0/3590479299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab05760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.853+0000 7f826d396640 1 -- 172.21.15.89:0/3590479299 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1ca9ea3a0 con 0x55c1caaf9b40 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.853+0000 7f8267fff640 1 --2- 172.21.15.89:0/3590479299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab05760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.853+0000 7f826d396640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82580a3340 0x7f82580a5800 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f82600117b0 tx=0x7f826001b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.853+0000 7f826d396640 1 --2- 172.21.15.89:0/3590479299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1caaf8090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8260013940 tx=0x7f8260013e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd621547af417263 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.854+0000 7f8252ffd640 1 -- 172.21.15.89:0/3590479299 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8260003b70 con 0x55c1caaf9b40 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.854+0000 7f8252ffd640 1 -- 172.21.15.89:0/3590479299 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826000dda0 con 0x55c1caaf9b40 2024-09-15T23:25:31.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.854+0000 7f8252ffd640 1 -- 172.21.15.89:0/3590479299 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82600046f0 con 0x55c1caaf9b40 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 -- 172.21.15.89:0/3590479299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 msgr2=0x55c1caaf8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 --2- 172.21.15.89:0/3590479299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1caaf8090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8260013940 tx=0x7f8260013e00 comp rx=0 tx=0).stop 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 -- 172.21.15.89:0/3590479299 shutdown_connections 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 --2- 172.21.15.89:0/3590479299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab05760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 --2- 172.21.15.89:0/3590479299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf85d0 0x55c1caaf8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 --2- 172.21.15.89:0/3590479299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1caaf8090 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 -- 172.21.15.89:0/3590479299 >> 172.21.15.89:0/3590479299 conn(0x55c1caac1310 msgr2=0x55c1caaf4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 -- 172.21.15.89:0/3590479299 shutdown_connections 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 -- 172.21.15.89:0/3590479299 wait complete. 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 -- start start 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.855+0000 7f82711d2740 1 Processor -- start 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f82711d2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf85d0 0x55c1cab02d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f82711d2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf9b40 0x55c1cab00010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f82711d2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab00600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f82711d2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c1cab05fb0 con 0x55c1cab03370 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f82711d2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c1caa19060 con 0x55c1caaf85d0 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f82711d2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c1cab06130 con 0x55c1caaf9b40 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f82711d2740 10 client.0 _ll_register_callbacks cb 0x55c1ca76c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f826d396640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab00600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f826cb95640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf85d0 0x55c1cab02d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f826d396640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab00600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39074/0 (socket says 172.21.15.89:39074) 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f826d396640 1 -- 172.21.15.89:0/2242577228 learned_addr learned my addr 172.21.15.89:0/2242577228 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.856+0000 7f8267fff640 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf9b40 0x55c1cab00010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f826d396640 1 -- 172.21.15.89:0/2242577228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf9b40 msgr2=0x55c1cab00010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f826d396640 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf9b40 0x55c1cab00010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f826d396640 1 -- 172.21.15.89:0/2242577228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf85d0 msgr2=0x55c1cab02d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f826d396640 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf85d0 0x55c1cab02d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f826d396640 1 -- 172.21.15.89:0/2242577228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1ca7cf3c0 con 0x55c1cab03370 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f826d396640 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab00600 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8260014010 tx=0x7f8260005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170b85ce8ecdb27b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f8247fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f8247fff640 1 -- 172.21.15.89:0/2242577228 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8260033020 con 0x55c1cab03370 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f8247fff640 1 -- 172.21.15.89:0/2242577228 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8260004c10 con 0x55c1cab03370 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f82711d2740 1 -- 172.21.15.89:0/2242577228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f826000dda0 con 0x55c1cab03370 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.857+0000 7f8247fff640 1 -- 172.21.15.89:0/2242577228 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f826003e050 con 0x55c1cab03370 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f82711d2740 10 client.15861 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f82711d2740 1 -- 172.21.15.89:0/2242577228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c1cab119f0 con 0x55c1cab03370 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f8247fff640 1 -- 172.21.15.89:0/2242577228 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826000dda0 con 0x55c1cab03370 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f8247fff640 1 -- 172.21.15.89:0/2242577228 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8260005160 con 0x55c1cab03370 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f82711d2740 10 client.15861 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f82711d2740 10 client.15861 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f82711d2740 20 client.15861 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:31.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f82711d2740 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c1cab12eb0 0x55c1cab152a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f82711d2740 4 client.15861 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f82711d2740 1 -- 172.21.15.89:0/2242577228 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1caa6b860 con 0x55c1cab12eb0 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.858+0000 7f826cb95640 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c1cab12eb0 0x55c1cab152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.859+0000 7f8247fff640 10 client.15861 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.859+0000 7f826cb95640 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c1cab12eb0 0x55c1cab152a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=da0269d340dd41c0 server_cookie=4ab00c38ad25c50a in_seq=0 out_seq=0 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f8247fff640 1 -- 172.21.15.89:0/2242577228 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c1caa6b860 con 0x55c1cab12eb0 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f8247fff640 10 client.15861 handle_command_reply: tid=0 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 client.15861 shutdown 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 -- 172.21.15.89:0/2242577228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 msgr2=0x55c1cab00600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab00600 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8260014010 tx=0x7f8260005fd0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 -- 172.21.15.89:0/2242577228 shutdown_connections 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c1cab12eb0 0x55c1cab152a0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1cab03370 0x55c1cab00600 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f8247fff640 0 client.15861 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:31.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1caaf9b40 0x55c1cab00010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 --2- 172.21.15.89:0/2242577228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf85d0 0x55c1cab02d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 -- 172.21.15.89:0/2242577228 >> 172.21.15.89:0/2242577228 conn(0x55c1caac1310 msgr2=0x55c1caafa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 -- 172.21.15.89:0/2242577228 shutdown_connections 2024-09-15T23:25:31.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 1 -- 172.21.15.89:0/2242577228 wait complete. 2024-09-15T23:25:31.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.860+0000 7f82711d2740 20 client.15861 trim_cache size 0 max 16384 2024-09-15T23:25:31.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.885+0000 7f82711d2740 1 Processor -- start 2024-09-15T23:25:31.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.885+0000 7f82711d2740 1 -- start start 2024-09-15T23:25:31.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.885+0000 7f82711d2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab0f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f82711d2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1cab1a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f82711d2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1cab03370 0x55c1cab1adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f82711d2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f82681accd0 con 0x55c1caaf85d0 2024-09-15T23:25:31.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f82711d2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f82681ae810 con 0x55c1caaf9b40 2024-09-15T23:25:31.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f82711d2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82681ad3a0 con 0x55c1cab03370 2024-09-15T23:25:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f826cb95640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab0f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f826cb95640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab0f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39088/0 (socket says 172.21.15.89:39088) 2024-09-15T23:25:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f826cb95640 1 -- 172.21.15.89:0/3004273471 learned_addr learned my addr 172.21.15.89:0/3004273471 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f8267fff640 1 --2- 172.21.15.89:0/3004273471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1cab1a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f826d396640 1 --2- 172.21.15.89:0/3004273471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1cab03370 0x55c1cab1adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f826cb95640 1 -- 172.21.15.89:0/3004273471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1cab03370 msgr2=0x55c1cab1adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f826cb95640 1 --2- 172.21.15.89:0/3004273471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1cab03370 0x55c1cab1adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f826cb95640 1 -- 172.21.15.89:0/3004273471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 msgr2=0x55c1cab1a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f826cb95640 1 --2- 172.21.15.89:0/3004273471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1cab1a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f826cb95640 1 -- 172.21.15.89:0/3004273471 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1caaf5960 con 0x55c1caaf85d0 2024-09-15T23:25:31.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.886+0000 7f826cb95640 1 --2- 172.21.15.89:0/3004273471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab0f280 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f825c00b990 tx=0x7f825c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d442020b25ebf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.887+0000 7f82527fc640 1 -- 172.21.15.89:0/3004273471 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c022020 con 0x55c1caaf85d0 2024-09-15T23:25:31.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.887+0000 7f82527fc640 1 -- 172.21.15.89:0/3004273471 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825c003e50 con 0x55c1caaf85d0 2024-09-15T23:25:31.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.887+0000 7f82527fc640 1 -- 172.21.15.89:0/3004273471 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c0119a0 con 0x55c1caaf85d0 2024-09-15T23:25:31.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 -- 172.21.15.89:0/3004273471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 msgr2=0x55c1cab0f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 --2- 172.21.15.89:0/3004273471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab0f280 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f825c00b990 tx=0x7f825c00be50 comp rx=0 tx=0).stop 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 -- 172.21.15.89:0/3004273471 shutdown_connections 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 --2- 172.21.15.89:0/3004273471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1cab03370 0x55c1cab1adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 --2- 172.21.15.89:0/3004273471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1cab1a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 --2- 172.21.15.89:0/3004273471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab0f280 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 -- 172.21.15.89:0/3004273471 >> 172.21.15.89:0/3004273471 conn(0x55c1caac1310 msgr2=0x55c1caaff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 -- 172.21.15.89:0/3004273471 shutdown_connections 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 -- 172.21.15.89:0/3004273471 wait complete. 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 -- start start 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.888+0000 7f82711d2740 1 Processor -- start 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f82711d2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab07080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f82711d2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1cab075c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f826cb95640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab07080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f826cb95640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab07080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39096/0 (socket says 172.21.15.89:39096) 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f826cb95640 1 -- 172.21.15.89:0/3401249287 learned_addr learned my addr 172.21.15.89:0/3401249287 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f82711d2740 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1cab03370 0x55c1cab07b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f82681ae140 con 0x55c1caaf85d0 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f82681ada70 con 0x55c1caaf9b40 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82681aacb0 con 0x55c1cab03370 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f82711d2740 10 client.0 _ll_register_callbacks cb 0x7f8260013710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f826cb95640 1 -- 172.21.15.89:0/3401249287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1cab03370 msgr2=0x55c1cab07b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f826cb95640 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1cab03370 0x55c1cab07b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f826cb95640 1 -- 172.21.15.89:0/3401249287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 msgr2=0x55c1cab075c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.889+0000 7f8267fff640 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1cab075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.890+0000 7f826cb95640 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1cab075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.890+0000 7f826cb95640 1 -- 172.21.15.89:0/3401249287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1cab85050 con 0x55c1caaf85d0 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.890+0000 7f8267fff640 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1cab075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:31.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.890+0000 7f826cb95640 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab07080 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f825c01d650 tx=0x7f825c01d680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78d227387707d7a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.890+0000 7f8247fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:31.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.890+0000 7f8247fff640 1 -- 172.21.15.89:0/3401249287 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c01da30 con 0x55c1caaf85d0 2024-09-15T23:25:31.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.890+0000 7f8247fff640 1 -- 172.21.15.89:0/3401249287 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f825c01dbd0 con 0x55c1caaf85d0 2024-09-15T23:25:31.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.890+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f825c003e50 con 0x55c1caaf85d0 2024-09-15T23:25:31.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.890+0000 7f8247fff640 1 -- 172.21.15.89:0/3401249287 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f825c03a050 con 0x55c1caaf85d0 2024-09-15T23:25:31.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f82711d2740 10 client.15867 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:31.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c1cab86250 con 0x55c1caaf85d0 2024-09-15T23:25:31.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f8247fff640 1 -- 172.21.15.89:0/3401249287 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f825c003e50 con 0x55c1caaf85d0 2024-09-15T23:25:31.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f8247fff640 1 -- 172.21.15.89:0/3401249287 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f825c02f020 con 0x55c1caaf85d0 2024-09-15T23:25:31.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f82711d2740 10 client.15867 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:31.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f82711d2740 10 client.15867 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:25:31.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f82711d2740 20 client.15867 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:31.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f82711d2740 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c1cab12eb0 0x55c1cab88920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f82711d2740 4 client.15867 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:31.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c1caa65d70 con 0x55c1cab12eb0 2024-09-15T23:25:31.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.891+0000 7f8267fff640 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c1cab12eb0 0x55c1cab88920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.892+0000 7f8247fff640 10 client.15867 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:31.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.892+0000 7f8267fff640 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c1cab12eb0 0x55c1cab88920 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=326d5e638aeaea49 server_cookie=bdb261499d5bcd2a in_seq=0 out_seq=0 2024-09-15T23:25:31.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.892+0000 7f8247fff640 1 -- 172.21.15.89:0/3401249287 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c1caa65d70 con 0x55c1cab12eb0 2024-09-15T23:25:31.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.892+0000 7f8247fff640 10 client.15867 handle_command_reply: tid=0 2024-09-15T23:25:31.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.892+0000 7f82711d2740 1 client.15867 shutdown 2024-09-15T23:25:31.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.892+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 msgr2=0x55c1cab07080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.892+0000 7f82711d2740 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab07080 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f825c01d650 tx=0x7f825c01d680 comp rx=0 tx=0).stop 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.892+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 shutdown_connections 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.892+0000 7f82711d2740 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c1cab12eb0 0x55c1cab88920 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.893+0000 7f82711d2740 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1cab03370 0x55c1cab07b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.893+0000 7f82711d2740 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1caaf9b40 0x55c1cab075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.893+0000 7f82711d2740 1 --2- 172.21.15.89:0/3401249287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1caaf85d0 0x55c1cab07080 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.893+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 >> 172.21.15.89:0/3401249287 conn(0x55c1caac1310 msgr2=0x55c1cab01870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.893+0000 7f8247fff640 0 client.15867 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.893+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 shutdown_connections 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.893+0000 7f82711d2740 1 -- 172.21.15.89:0/3401249287 wait complete. 2024-09-15T23:25:31.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.893+0000 7f82711d2740 20 client.15867 trim_cache size 0 max 16384 2024-09-15T23:25:31.900 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:31.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 -- 172.21.15.89:0/1911983029 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82580a3340 msgr2=0x7f82580a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82580a3340 0x7f82580a5800 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f82600117b0 tx=0x7f826001b000 comp rx=0 tx=0).stop 2024-09-15T23:25:31.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 -- 172.21.15.89:0/1911983029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102570 msgr2=0x7f8268199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102570 0x7f8268199910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8254006010 tx=0x7f825403ac10 comp rx=0 tx=0).stop 2024-09-15T23:25:31.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 -- 172.21.15.89:0/1911983029 shutdown_connections 2024-09-15T23:25:31.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82580a3340 0x7f82580a5800 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8268102ef0 0x7f82681a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8268102570 0x7f8268199910 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 --2- 172.21.15.89:0/1911983029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8268100e20 0x7f82681993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 -- 172.21.15.89:0/1911983029 >> 172.21.15.89:0/1911983029 conn(0x7f82680fc7d0 msgr2=0x7f82680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 -- 172.21.15.89:0/1911983029 shutdown_connections 2024-09-15T23:25:31.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:31.894+0000 7f826edf9640 1 -- 172.21.15.89:0/1911983029 wait complete. 2024-09-15T23:25:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:32 smithi120 ceph-mon[35250]: pgmap v395: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 12 op/s 2024-09-15T23:25:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:32 smithi120 ceph-mon[35250]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-09-15T23:25:32.479 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:32 smithi089 ceph-mon[30933]: pgmap v395: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 12 op/s 2024-09-15T23:25:32.480 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:32 smithi089 ceph-mon[30933]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-09-15T23:25:32.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:32 smithi174 ceph-mon[35586]: pgmap v395: 129 pgs: 129 active+clean; 679 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 12 op/s 2024-09-15T23:25:32.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:32 smithi174 ceph-mon[35586]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-09-15T23:25:32.597 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:25:32.879 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:33.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.465+0000 7f7b90827640 1 -- 172.21.15.89:0/3286572941 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b88035020 con 0x7f7b8c1009b0 2024-09-15T23:25:33.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.465+0000 7f7b93a8d640 1 -- 172.21.15.89:0/3286572941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b8c1009b0 msgr2=0x7f7b8c100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.465+0000 7f7b93a8d640 1 --2- 172.21.15.89:0/3286572941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b8c1009b0 0x7f7b8c100d90 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009900 tx=0x7f7b8802a410 comp rx=0 tx=0).stop 2024-09-15T23:25:33.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.466+0000 7f7b93a8d640 1 -- 172.21.15.89:0/3286572941 shutdown_connections 2024-09-15T23:25:33.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.466+0000 7f7b93a8d640 1 --2- 172.21.15.89:0/3286572941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c10e4d0 0x7f7b8c1108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.466+0000 7f7b93a8d640 1 --2- 172.21.15.89:0/3286572941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8c101360 0x7f7b8c10df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.466+0000 7f7b93a8d640 1 --2- 172.21.15.89:0/3286572941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b8c1009b0 0x7f7b8c100d90 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.466+0000 7f7b93a8d640 1 -- 172.21.15.89:0/3286572941 >> 172.21.15.89:0/3286572941 conn(0x7f7b8c0fc7d0 msgr2=0x7f7b8c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.466+0000 7f7b93a8d640 1 -- 172.21.15.89:0/3286572941 shutdown_connections 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.466+0000 7f7b93a8d640 1 -- 172.21.15.89:0/3286572941 wait complete. 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b93a8d640 1 Processor -- start 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b93a8d640 1 -- start start 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b93a8d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c1009b0 0x7f7b8c102c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b93a8d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b8c101360 0x7f7b8c103140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b93a8d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8c10e4d0 0x7f7b8c103680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b93a8d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c114280 con 0x7f7b8c101360 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b93a8d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c114100 con 0x7f7b8c10e4d0 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b93a8d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c114400 con 0x7f7b8c1009b0 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b91028640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b8c101360 0x7f7b8c103140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b91028640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b8c101360 0x7f7b8c103140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39116/0 (socket says 172.21.15.89:39116) 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b91028640 1 -- 172.21.15.89:0/1578329547 learned_addr learned my addr 172.21.15.89:0/1578329547 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:33.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b9202a640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8c10e4d0 0x7f7b8c103680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b91028640 1 -- 172.21.15.89:0/1578329547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c1009b0 msgr2=0x7f7b8c102c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b91028640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c1009b0 0x7f7b8c102c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b91028640 1 -- 172.21.15.89:0/1578329547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8c10e4d0 msgr2=0x7f7b8c103680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.467+0000 7f7b91028640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8c10e4d0 0x7f7b8c103680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.468+0000 7f7b91028640 1 -- 172.21.15.89:0/1578329547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c1aadc0 con 0x7f7b8c101360 2024-09-15T23:25:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.468+0000 7f7b9202a640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8c10e4d0 0x7f7b8c103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.468+0000 7f7b91028640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b8c101360 0x7f7b8c103140 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00ea20 tx=0x7f7b7c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.468+0000 7f7b82ffd640 1 -- 172.21.15.89:0/1578329547 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c00cd80 con 0x7f7b8c101360 2024-09-15T23:25:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.468+0000 7f7b82ffd640 1 -- 172.21.15.89:0/1578329547 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b7c010910 con 0x7f7b8c101360 2024-09-15T23:25:33.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.468+0000 7f7b82ffd640 1 -- 172.21.15.89:0/1578329547 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c018920 con 0x7f7b8c101360 2024-09-15T23:25:33.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.468+0000 7f7b93a8d640 1 -- 172.21.15.89:0/1578329547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8c1ab0b0 con 0x7f7b8c101360 2024-09-15T23:25:33.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.468+0000 7f7b93a8d640 1 -- 172.21.15.89:0/1578329547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b8c1ab530 con 0x7f7b8c101360 2024-09-15T23:25:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.471+0000 7f7b82ffd640 1 -- 172.21.15.89:0/1578329547 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7b7c010ac0 con 0x7f7b8c101360 2024-09-15T23:25:33.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.472+0000 7f7b95e66740 1 Processor -- start 2024-09-15T23:25:33.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.472+0000 7f7b95e66740 1 -- start start 2024-09-15T23:25:33.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.472+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb0b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.472+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb0ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.472+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb16370 0x55a5aeb18760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.472+0000 7f7b95e66740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a5aea712f0 con 0x55a5aeb0cb40 2024-09-15T23:25:33.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.472+0000 7f7b95e66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a5aea18cf0 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.472+0000 7f7b95e66740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a5aea6d2b0 con 0x55a5aeb16370 2024-09-15T23:25:33.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.472+0000 7f7b82ffd640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7b780a32c0 0x7f7b780a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b9202a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb0b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b9202a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb0b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39130/0 (socket says 172.21.15.89:39130) 2024-09-15T23:25:33.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b9202a640 1 -- 172.21.15.89:0/3013872856 learned_addr learned my addr 172.21.15.89:0/3013872856 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:33.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b91028640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7b780a32c0 0x7f7b780a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b91829640 1 --2- 172.21.15.89:0/3013872856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb0ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b9202a640 1 -- 172.21.15.89:0/3013872856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb16370 msgr2=0x55a5aeb18760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b9202a640 1 --2- 172.21.15.89:0/3013872856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb16370 0x55a5aeb18760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b9202a640 1 -- 172.21.15.89:0/3013872856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0b5d0 msgr2=0x55a5aeb0ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b9202a640 1 --2- 172.21.15.89:0/3013872856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb0ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b82ffd640 1 -- 172.21.15.89:0/1578329547 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7b7c0d01a0 con 0x7f7b8c101360 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b9202a640 1 -- 172.21.15.89:0/3013872856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5ae9fd3a0 con 0x55a5aeb0cb40 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b91829640 1 --2- 172.21.15.89:0/3013872856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb0ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b9202a640 1 --2- 172.21.15.89:0/3013872856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb0b090 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7b840103c0 tx=0x7f7b84010880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f22505aef3da7e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.473+0000 7f7b91028640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7b780a32c0 0x7f7b780a5780 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c012020 tx=0x7f7b7c098040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.474+0000 7f7b77fff640 1 -- 172.21.15.89:0/3013872856 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b84016070 con 0x55a5aeb0cb40 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.474+0000 7f7b77fff640 1 -- 172.21.15.89:0/3013872856 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b84004d20 con 0x55a5aeb0cb40 2024-09-15T23:25:33.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.474+0000 7f7b77fff640 1 -- 172.21.15.89:0/3013872856 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b840054a0 con 0x55a5aeb0cb40 2024-09-15T23:25:33.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 -- 172.21.15.89:0/3013872856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0cb40 msgr2=0x55a5aeb0b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 --2- 172.21.15.89:0/3013872856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb0b090 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7b840103c0 tx=0x7f7b84010880 comp rx=0 tx=0).stop 2024-09-15T23:25:33.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 -- 172.21.15.89:0/3013872856 shutdown_connections 2024-09-15T23:25:33.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 --2- 172.21.15.89:0/3013872856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb16370 0x55a5aeb18760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 --2- 172.21.15.89:0/3013872856 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb0ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 --2- 172.21.15.89:0/3013872856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb0b090 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 -- 172.21.15.89:0/3013872856 >> 172.21.15.89:0/3013872856 conn(0x55a5aead4310 msgr2=0x55a5aeb07c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 -- 172.21.15.89:0/3013872856 shutdown_connections 2024-09-15T23:25:33.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 -- 172.21.15.89:0/3013872856 wait complete. 2024-09-15T23:25:33.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 -- start start 2024-09-15T23:25:33.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.475+0000 7f7b95e66740 1 Processor -- start 2024-09-15T23:25:33.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb15d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb13000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb16370 0x55a5aeb135d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b95e66740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a5aeb19070 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b95e66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a5aea2c060 con 0x55a5aeb0cb40 2024-09-15T23:25:33.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b95e66740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a5aeb191f0 con 0x55a5aeb16370 2024-09-15T23:25:33.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b9202a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb15d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b95e66740 10 client.0 _ll_register_callbacks cb 0x55a5ae77f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:33.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b9202a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb15d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39142/0 (socket says 172.21.15.89:39142) 2024-09-15T23:25:33.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b9202a640 1 -- 172.21.15.89:0/2985697401 learned_addr learned my addr 172.21.15.89:0/2985697401 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:33.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb16370 0x55a5aeb135d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b91829640 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb13000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b9202a640 1 -- 172.21.15.89:0/2985697401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb16370 msgr2=0x55a5aeb135d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb16370 0x55a5aeb135d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b9202a640 1 -- 172.21.15.89:0/2985697401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 msgr2=0x55a5aeb13000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb13000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.476+0000 7f7b9202a640 1 -- 172.21.15.89:0/2985697401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5ae7e23c0 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.477+0000 7f7b91829640 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb13000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:33.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.477+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb15d50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7b840092f0 tx=0x7f7b84009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fee90104eb085dd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.477+0000 7f7b757fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:33.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.477+0000 7f7b757fa640 1 -- 172.21.15.89:0/2985697401 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b84016040 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.477+0000 7f7b95e66740 1 -- 172.21.15.89:0/2985697401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b84004d20 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.477+0000 7f7b757fa640 1 -- 172.21.15.89:0/2985697401 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7b84012860 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.477+0000 7f7b757fa640 1 -- 172.21.15.89:0/2985697401 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7b84012a60 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.478+0000 7f7b95e66740 10 client.15879 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:33.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.478+0000 7f7b95e66740 1 -- 172.21.15.89:0/2985697401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a5aeb24b00 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.479+0000 7f7b757fa640 1 -- 172.21.15.89:0/2985697401 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b84004d20 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.479+0000 7f7b757fa640 1 -- 172.21.15.89:0/2985697401 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7b84028020 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.479+0000 7f7b95e66740 10 client.15879 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:33.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.479+0000 7f7b95e66740 10 client.15879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:33.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.479+0000 7f7b95e66740 20 client.15879 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:33.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.479+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a5aeb25eb0 0x55a5aeb282a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.479+0000 7f7b95e66740 4 client.15879 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:33.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.479+0000 7f7b95e66740 1 -- 172.21.15.89:0/2985697401 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5aea7e860 con 0x55a5aeb25eb0 2024-09-15T23:25:33.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.479+0000 7f7b91829640 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a5aeb25eb0 0x55a5aeb282a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.480+0000 7f7b91829640 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a5aeb25eb0 0x55a5aeb282a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=957bf5be2497a7cc server_cookie=d01417326db0f206 in_seq=0 out_seq=0 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.480+0000 7f7b757fa640 10 client.15879 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b757fa640 1 -- 172.21.15.89:0/2985697401 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a5aea7e860 con 0x55a5aeb25eb0 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b757fa640 10 client.15879 handle_command_reply: tid=0 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b95e66740 1 client.15879 shutdown 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b95e66740 1 -- 172.21.15.89:0/2985697401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0b5d0 msgr2=0x55a5aeb15d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb15d50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7b840092f0 tx=0x7f7b84009280 comp rx=0 tx=0).stop 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b95e66740 1 -- 172.21.15.89:0/2985697401 shutdown_connections 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a5aeb25eb0 0x55a5aeb282a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb16370 0x55a5aeb135d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb13000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2985697401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb15d50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b95e66740 1 -- 172.21.15.89:0/2985697401 >> 172.21.15.89:0/2985697401 conn(0x55a5aead4310 msgr2=0x55a5aeb0d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b757fa640 0 client.15879 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.481+0000 7f7b95e66740 1 -- 172.21.15.89:0/2985697401 shutdown_connections 2024-09-15T23:25:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.482+0000 7f7b95e66740 1 -- 172.21.15.89:0/2985697401 wait complete. 2024-09-15T23:25:33.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.482+0000 7f7b95e66740 20 client.15879 trim_cache size 0 max 16384 2024-09-15T23:25:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.494+0000 7f7b95e66740 1 Processor -- start 2024-09-15T23:25:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.494+0000 7f7b95e66740 1 -- start start 2024-09-15T23:25:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb22340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb128e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb12e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b95e66740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b5630 con 0x55a5aeb16370 2024-09-15T23:25:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b95e66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b7170 con 0x55a5aeb0cb40 2024-09-15T23:25:33.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b95e66740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b5d00 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b91829640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb128e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b91829640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb128e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:59102/0 (socket says 172.21.15.89:59102) 2024-09-15T23:25:33.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b91829640 1 -- 172.21.15.89:0/2942570292 learned_addr learned my addr 172.21.15.89:0/2942570292 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:33.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2942570292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb22340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2942570292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb12e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b91829640 1 -- 172.21.15.89:0/2942570292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb0b5d0 msgr2=0x55a5aeb22340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b91829640 1 --2- 172.21.15.89:0/2942570292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb22340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b91829640 1 -- 172.21.15.89:0/2942570292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 msgr2=0x55a5aeb12e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b91829640 1 --2- 172.21.15.89:0/2942570292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb12e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b91829640 1 -- 172.21.15.89:0/2942570292 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5aeb160c0 con 0x55a5aeb0cb40 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.495+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2942570292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb12e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.496+0000 7f7b91829640 1 --2- 172.21.15.89:0/2942570292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb128e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009900 tx=0x7f7b880042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2562e53de9e565ec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.496+0000 7f7b777fe640 1 -- 172.21.15.89:0/2942570292 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8804b070 con 0x55a5aeb0cb40 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.496+0000 7f7b777fe640 1 -- 172.21.15.89:0/2942570292 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b88005520 con 0x55a5aeb0cb40 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.496+0000 7f7b777fe640 1 -- 172.21.15.89:0/2942570292 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b88035040 con 0x55a5aeb0cb40 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.497+0000 7f7b95e66740 1 -- 172.21.15.89:0/2942570292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 msgr2=0x55a5aeb128e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.497+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2942570292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb128e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009900 tx=0x7f7b880042e0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.497+0000 7f7b95e66740 1 -- 172.21.15.89:0/2942570292 shutdown_connections 2024-09-15T23:25:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.497+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2942570292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb12e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.497+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2942570292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb128e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.497+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2942570292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb22340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.497+0000 7f7b95e66740 1 -- 172.21.15.89:0/2942570292 >> 172.21.15.89:0/2942570292 conn(0x55a5aead4310 msgr2=0x55a5aeb06d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.497+0000 7f7b95e66740 1 -- 172.21.15.89:0/2942570292 shutdown_connections 2024-09-15T23:25:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.497+0000 7f7b95e66740 1 -- 172.21.15.89:0/2942570292 wait complete. 2024-09-15T23:25:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.498+0000 7f7b95e66740 1 -- start start 2024-09-15T23:25:33.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.498+0000 7f7b95e66740 1 Processor -- start 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.498+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb889b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.498+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb88ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.498+0000 7f7b95e66740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb1e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.498+0000 7f7b95e66740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b6aa0 con 0x55a5aeb16370 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.498+0000 7f7b95e66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b63d0 con 0x55a5aeb0cb40 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.498+0000 7f7b95e66740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1b3610 con 0x55a5aeb0b5d0 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.498+0000 7f7b95e66740 10 client.0 _ll_register_callbacks cb 0x7f7b84009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.498+0000 7f7b9202a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb1e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b9202a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb1e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39162/0 (socket says 172.21.15.89:39162) 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b9202a640 1 -- 172.21.15.89:0/2173553633 learned_addr learned my addr 172.21.15.89:0/2173553633 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b91829640 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb88ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb889b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b9202a640 1 -- 172.21.15.89:0/2173553633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb0b5d0 msgr2=0x55a5aeb889b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb889b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b9202a640 1 -- 172.21.15.89:0/2173553633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 msgr2=0x55a5aeb88ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb88ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b9202a640 1 -- 172.21.15.89:0/2173553633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5aeb1e7b0 con 0x55a5aeb16370 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b91829640 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb88ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b9202a640 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb1e140 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7b84015040 tx=0x7f7b8401d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583a1ded3e1e9040 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b757fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b757fa640 1 -- 172.21.15.89:0/2173553633 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b84016040 con 0x55a5aeb16370 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b757fa640 1 -- 172.21.15.89:0/2173553633 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7b8401b480 con 0x55a5aeb16370 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.499+0000 7f7b95e66740 1 -- 172.21.15.89:0/2173553633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b88005520 con 0x55a5aeb16370 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b757fa640 1 -- 172.21.15.89:0/2173553633 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7b84042050 con 0x55a5aeb16370 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b95e66740 10 client.15885 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:33.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b95e66740 1 -- 172.21.15.89:0/2173553633 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a5aeb901e0 con 0x55a5aeb16370 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b757fa640 1 -- 172.21.15.89:0/2173553633 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b88005520 con 0x55a5aeb16370 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b757fa640 1 -- 172.21.15.89:0/2173553633 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7b84037040 con 0x55a5aeb16370 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b95e66740 10 client.15885 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b95e66740 10 client.15885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b95e66740 20 client.15885 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a5aeb90960 0x55a5aeb92d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b95e66740 4 client.15885 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.500+0000 7f7b95e66740 1 -- 172.21.15.89:0/2173553633 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5aea78d70 con 0x55a5aeb90960 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.501+0000 7f7b91829640 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a5aeb90960 0x55a5aeb92d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.501+0000 7f7b91829640 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a5aeb90960 0x55a5aeb92d80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cec7a1c2ed7eeff6 server_cookie=e6871f2448022b01 in_seq=0 out_seq=0 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.501+0000 7f7b757fa640 10 client.15885 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.501+0000 7f7b757fa640 1 -- 172.21.15.89:0/2173553633 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5aea78d70 con 0x55a5aeb90960 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.501+0000 7f7b757fa640 10 client.15885 handle_command_reply: tid=0 2024-09-15T23:25:33.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.501+0000 7f7b95e66740 1 client.15885 shutdown 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.501+0000 7f7b95e66740 1 -- 172.21.15.89:0/2173553633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 msgr2=0x55a5aeb1e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.501+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb1e140 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7b84015040 tx=0x7f7b8401d040 comp rx=0 tx=0).stop 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.502+0000 7f7b95e66740 1 -- 172.21.15.89:0/2173553633 shutdown_connections 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.502+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a5aeb90960 0x55a5aeb92d80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.502+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5aeb16370 0x55a5aeb1e140 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.502+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a5aeb0cb40 0x55a5aeb88ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.502+0000 7f7b95e66740 1 --2- 172.21.15.89:0/2173553633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a5aeb0b5d0 0x55a5aeb889b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.502+0000 7f7b95e66740 1 -- 172.21.15.89:0/2173553633 >> 172.21.15.89:0/2173553633 conn(0x55a5aead4310 msgr2=0x55a5aeb0d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.502+0000 7f7b757fa640 0 client.15885 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.502+0000 7f7b95e66740 1 -- 172.21.15.89:0/2173553633 shutdown_connections 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.502+0000 7f7b95e66740 1 -- 172.21.15.89:0/2173553633 wait complete. 2024-09-15T23:25:33.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.502+0000 7f7b95e66740 20 client.15885 trim_cache size 0 max 16384 2024-09-15T23:25:33.508 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:33.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 -- 172.21.15.89:0/1578329547 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7b780a32c0 msgr2=0x7f7b780a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7b780a32c0 0x7f7b780a5780 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c012020 tx=0x7f7b7c098040 comp rx=0 tx=0).stop 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 -- 172.21.15.89:0/1578329547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b8c101360 msgr2=0x7f7b8c103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b8c101360 0x7f7b8c103140 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00ea20 tx=0x7f7b7c00eee0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 -- 172.21.15.89:0/1578329547 shutdown_connections 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7b780a32c0 0x7f7b780a5780 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b8c10e4d0 0x7f7b8c103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b8c101360 0x7f7b8c103140 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 --2- 172.21.15.89:0/1578329547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b8c1009b0 0x7f7b8c102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 -- 172.21.15.89:0/1578329547 >> 172.21.15.89:0/1578329547 conn(0x7f7b8c0fc7d0 msgr2=0x7f7b8c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 -- 172.21.15.89:0/1578329547 shutdown_connections 2024-09-15T23:25:33.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:33.503+0000 7f7b93a8d640 1 -- 172.21.15.89:0/1578329547 wait complete. 2024-09-15T23:25:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.740+0000 7f51f685a640 1 -- 172.21.15.89:0/3980804875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0100a40 msgr2=0x7f51f0100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.740+0000 7f51f685a640 1 --2- 172.21.15.89:0/3980804875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0100a40 0x7f51f0100e40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f51e0009900 tx=0x7f51e002a410 comp rx=0 tx=0).stop 2024-09-15T23:25:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.740+0000 7f51f685a640 1 -- 172.21.15.89:0/3980804875 shutdown_connections 2024-09-15T23:25:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.740+0000 7f51f685a640 1 --2- 172.21.15.89:0/3980804875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f01025f0 0x7f51f0108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.740+0000 7f51f685a640 1 --2- 172.21.15.89:0/3980804875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f0101c30 0x7f51f01020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.740+0000 7f51f685a640 1 --2- 172.21.15.89:0/3980804875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0100a40 0x7f51f0100e40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.740+0000 7f51f685a640 1 -- 172.21.15.89:0/3980804875 >> 172.21.15.89:0/3980804875 conn(0x7f51f00fc1e0 msgr2=0x7f51f00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.740+0000 7f51f685a640 1 -- 172.21.15.89:0/3980804875 shutdown_connections 2024-09-15T23:25:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.740+0000 7f51f685a640 1 -- 172.21.15.89:0/3980804875 wait complete. 2024-09-15T23:25:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.741+0000 7f51f685a640 1 Processor -- start 2024-09-15T23:25:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.741+0000 7f51f685a640 1 -- start start 2024-09-15T23:25:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.741+0000 7f51f685a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0100a40 0x7f51f006a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f685a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f0101c30 0x7f51f019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f685a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01025f0 0x7f51f01a1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f685a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51f010c6f0 con 0x7f51f0101c30 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f0101c30 0x7f51f019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51effff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0100a40 0x7f51f006a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f4df7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01025f0 0x7f51f01a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51effff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0100a40 0x7f51f006a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:47648/0 (socket says 172.21.15.89:47648) 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f4df7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01025f0 0x7f51f01a1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:59144/0 (socket says 172.21.15.89:59144) 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f0101c30 0x7f51f019ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39188/0 (socket says 172.21.15.89:39188) 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51effff640 1 -- 172.21.15.89:0/2579718329 learned_addr learned my addr 172.21.15.89:0/2579718329 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f685a640 1 -- 172.21.15.89:0/2579718329 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51f010c570 con 0x7f51f01025f0 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f685a640 1 -- 172.21.15.89:0/2579718329 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f51f010c870 con 0x7f51f0100a40 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f4df7640 1 -- 172.21.15.89:0/2579718329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0100a40 msgr2=0x7f51f006a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f4df7640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0100a40 0x7f51f006a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f4df7640 1 -- 172.21.15.89:0/2579718329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f0101c30 msgr2=0x7f51f019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f4df7640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f0101c30 0x7f51f019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.742+0000 7f51f4df7640 1 -- 172.21.15.89:0/2579718329 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f01a1c80 con 0x7f51f01025f0 2024-09-15T23:25:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.743+0000 7f51f4df7640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01025f0 0x7f51f01a1490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f51e800ec50 tx=0x7f51e800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.743+0000 7f51ed7fa640 1 -- 172.21.15.89:0/2579718329 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8003b70 con 0x7f51f01025f0 2024-09-15T23:25:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.743+0000 7f51ed7fa640 1 -- 172.21.15.89:0/2579718329 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e8003d10 con 0x7f51f01025f0 2024-09-15T23:25:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.743+0000 7f51ed7fa640 1 -- 172.21.15.89:0/2579718329 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8010890 con 0x7f51f01025f0 2024-09-15T23:25:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.743+0000 7f51f685a640 1 -- 172.21.15.89:0/2579718329 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f01a1f10 con 0x7f51f01025f0 2024-09-15T23:25:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.743+0000 7f51f685a640 1 -- 172.21.15.89:0/2579718329 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f01a2380 con 0x7f51f01025f0 2024-09-15T23:25:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.746+0000 7f51f95f5740 1 Processor -- start 2024-09-15T23:25:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51ed7fa640 1 -- 172.21.15.89:0/2579718329 <== mon.2 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f51e8003ec0 con 0x7f51f01025f0 2024-09-15T23:25:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51f95f5740 1 -- start start 2024-09-15T23:25:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51f95f5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b8351b0 0x55d31b8355b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51f95f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 0x55d31b83a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51f95f5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b83ab80 0x55d31b83d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51f95f5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d31b7902f0 con 0x55d31b8351b0 2024-09-15T23:25:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51f95f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d31b738270 con 0x55d31b835af0 2024-09-15T23:25:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51f95f5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d31b78c2b0 con 0x55d31b83ab80 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b8351b0 0x55d31b8355b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b8351b0 0x55d31b8355b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39192/0 (socket says 172.21.15.89:39192) 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51ef7fe640 1 -- 172.21.15.89:0/2868342559 learned_addr learned my addr 172.21.15.89:0/2868342559 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.748+0000 7f51effff640 1 --2- 172.21.15.89:0/2868342559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 0x55d31b83a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51ef7fe640 1 -- 172.21.15.89:0/2868342559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b83ab80 msgr2=0x55d31b83d040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/2868342559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b83ab80 0x55d31b83d040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51ef7fe640 1 -- 172.21.15.89:0/2868342559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 msgr2=0x55d31b83a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/2868342559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 0x55d31b83a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51ef7fe640 1 -- 172.21.15.89:0/2868342559 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d31b737340 con 0x55d31b8351b0 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51ed7fa640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51d80a3320 0x7f51d80a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51effff640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51d80a3320 0x7f51d80a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/2868342559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b8351b0 0x55d31b8355b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f51e4009c60 tx=0x7f51e4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b887ff6cef2cd7b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51ed7fa640 1 -- 172.21.15.89:0/2579718329 <== mon.2 v2:172.21.15.120:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f51e80d0290 con 0x7f51f01025f0 2024-09-15T23:25:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51d67fc640 1 -- 172.21.15.89:0/2868342559 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e401e070 con 0x55d31b8351b0 2024-09-15T23:25:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51d67fc640 1 -- 172.21.15.89:0/2868342559 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e40198f0 con 0x55d31b8351b0 2024-09-15T23:25:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.749+0000 7f51d67fc640 1 -- 172.21.15.89:0/2868342559 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e400f850 con 0x55d31b8351b0 2024-09-15T23:25:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.750+0000 7f51f95f5740 1 -- 172.21.15.89:0/2868342559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b8351b0 msgr2=0x55d31b8355b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.750+0000 7f51f95f5740 1 --2- 172.21.15.89:0/2868342559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b8351b0 0x55d31b8355b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f51e4009c60 tx=0x7f51e4011810 comp rx=0 tx=0).stop 2024-09-15T23:25:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.750+0000 7f51effff640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51d80a3320 0x7f51d80a57e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f51e0000c00 tx=0x7f51e00315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.751+0000 7f51f95f5740 1 -- 172.21.15.89:0/2868342559 shutdown_connections 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.751+0000 7f51f95f5740 1 --2- 172.21.15.89:0/2868342559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b83ab80 0x55d31b83d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.751+0000 7f51f95f5740 1 --2- 172.21.15.89:0/2868342559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 0x55d31b83a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.751+0000 7f51f95f5740 1 --2- 172.21.15.89:0/2868342559 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b8351b0 0x55d31b8355b0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.751+0000 7f51f95f5740 1 -- 172.21.15.89:0/2868342559 >> 172.21.15.89:0/2868342559 conn(0x55d31b830820 msgr2=0x55d31b830c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.751+0000 7f51f95f5740 1 -- 172.21.15.89:0/2868342559 shutdown_connections 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.751+0000 7f51f95f5740 1 -- 172.21.15.89:0/2868342559 wait complete. 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.751+0000 7f51f95f5740 1 -- start start 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.752+0000 7f51f95f5740 1 Processor -- start 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.752+0000 7f51f95f5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b835af0 0x55d31b83f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.752+0000 7f51f95f5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b83fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.752+0000 7f51ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b83fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.752+0000 7f51ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b83fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39208/0 (socket says 172.21.15.89:39208) 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.752+0000 7f51ef7fe640 1 -- 172.21.15.89:0/316243941 learned_addr learned my addr 172.21.15.89:0/316243941 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.752+0000 7f51f95f5740 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b844be0 0x55d31b846fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51effff640 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b835af0 0x55d31b83f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d31b83db50 con 0x55d31b83ab80 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d31b719290 con 0x55d31b844be0 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d31b83dcd0 con 0x55d31b835af0 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51effff640 1 -- 172.21.15.89:0/316243941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b844be0 msgr2=0x55d31b846fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51effff640 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b844be0 0x55d31b846fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51effff640 1 -- 172.21.15.89:0/316243941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 msgr2=0x55d31b83fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51effff640 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b83fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51effff640 1 -- 172.21.15.89:0/316243941 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d31b4a3210 con 0x55d31b835af0 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b83fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51f95f5740 10 client.0 _ll_register_callbacks cb 0x55d31b74a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51effff640 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b835af0 0x55d31b83f670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f51e0004770 tx=0x7f51e0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d540cc13cd47d3c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.753+0000 7f51cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:25:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.754+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e40198f0 con 0x55d31b835af0 2024-09-15T23:25:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.754+0000 7f51cf7fe640 1 -- 172.21.15.89:0/316243941 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0058050 con 0x55d31b835af0 2024-09-15T23:25:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.754+0000 7f51cf7fe640 1 -- 172.21.15.89:0/316243941 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f51e00035f0 con 0x55d31b835af0 2024-09-15T23:25:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.754+0000 7f51cf7fe640 1 -- 172.21.15.89:0/316243941 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51e004e070 con 0x55d31b835af0 2024-09-15T23:25:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.754+0000 7f51f95f5740 10 client.25249 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.754+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55d31b56f620 con 0x55d31b835af0 2024-09-15T23:25:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.755+0000 7f51cf7fe640 1 -- 172.21.15.89:0/316243941 <== mon.1 v2:172.21.15.174:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f51e0003810 con 0x55d31b835af0 2024-09-15T23:25:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.755+0000 7f51f95f5740 10 client.25249 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.755+0000 7f51f95f5740 10 client.25249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.755+0000 7f51f95f5740 20 client.25249 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.755+0000 7f51f95f5740 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d31b84b540 0x55d31b84d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.755+0000 7f51f95f5740 4 client.25249 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.755+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d31b79da20 con 0x55d31b84b540 2024-09-15T23:25:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.755+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d31b84b540 0x55d31b84d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.756+0000 7f51cf7fe640 10 client.25249 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.756+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d31b84b540 0x55d31b84d9d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c78c7cea128b36d1 server_cookie=b653f0032817f3ba in_seq=0 out_seq=0 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51cf7fe640 1 -- 172.21.15.89:0/316243941 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d31b79da20 con 0x55d31b84b540 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51cf7fe640 10 client.25249 handle_command_reply: tid=0 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 client.25249 shutdown 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b835af0 msgr2=0x55d31b83f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b835af0 0x55d31b83f670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f51e0004770 tx=0x7f51e0009260 comp rx=0 tx=0).stop 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 shutdown_connections 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d31b84b540 0x55d31b84d9d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51cf7fe640 0 client.25249 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b844be0 0x55d31b846fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b83fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 --2- 172.21.15.89:0/316243941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b835af0 0x55d31b83f670 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 >> 172.21.15.89:0/316243941 conn(0x55d31b830820 msgr2=0x55d31b832030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 shutdown_connections 2024-09-15T23:25:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.757+0000 7f51f95f5740 1 -- 172.21.15.89:0/316243941 wait complete. 2024-09-15T23:25:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.758+0000 7f51f95f5740 20 client.25249 trim_cache size 0 max 16384 2024-09-15T23:25:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.785+0000 7f51f95f5740 1 Processor -- start 2024-09-15T23:25:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.785+0000 7f51f95f5740 1 -- start start 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.785+0000 7f51f95f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 0x55d31b830c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.785+0000 7f51f95f5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b831140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.785+0000 7f51f95f5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b844be0 0x55d31b831680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b831140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b831140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39218/0 (socket says 172.21.15.89:39218) 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51ef7fe640 1 -- 172.21.15.89:0/3713799298 learned_addr learned my addr 172.21.15.89:0/3713799298 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51effff640 1 --2- 172.21.15.89:0/3713799298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 0x55d31b830c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51f95f5740 1 -- 172.21.15.89:0/3713799298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01a8590 con 0x55d31b83ab80 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51f95f5740 1 -- 172.21.15.89:0/3713799298 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01aa0d0 con 0x55d31b835af0 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51f95f5740 1 -- 172.21.15.89:0/3713799298 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01a8c60 con 0x55d31b844be0 2024-09-15T23:25:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51f4df7640 1 --2- 172.21.15.89:0/3713799298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b844be0 0x55d31b831680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51effff640 1 -- 172.21.15.89:0/3713799298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b844be0 msgr2=0x55d31b831680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51effff640 1 --2- 172.21.15.89:0/3713799298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b844be0 0x55d31b831680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51effff640 1 -- 172.21.15.89:0/3713799298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 msgr2=0x55d31b831140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51effff640 1 --2- 172.21.15.89:0/3713799298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b831140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51effff640 1 -- 172.21.15.89:0/3713799298 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d31b847d80 con 0x55d31b835af0 2024-09-15T23:25:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/3713799298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b831140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.786+0000 7f51effff640 1 --2- 172.21.15.89:0/3713799298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 0x55d31b830c00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f51e0006010 tx=0x7f51e004a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12fa2b93809f758 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.787+0000 7f51d5ffb640 1 -- 172.21.15.89:0/3713799298 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0058070 con 0x55d31b835af0 2024-09-15T23:25:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.787+0000 7f51d5ffb640 1 -- 172.21.15.89:0/3713799298 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e40198f0 con 0x55d31b835af0 2024-09-15T23:25:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.787+0000 7f51d5ffb640 1 -- 172.21.15.89:0/3713799298 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0035040 con 0x55d31b835af0 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.788+0000 7f51f95f5740 1 -- 172.21.15.89:0/3713799298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 msgr2=0x55d31b830c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.788+0000 7f51f95f5740 1 --2- 172.21.15.89:0/3713799298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 0x55d31b830c00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f51e0006010 tx=0x7f51e004a000 comp rx=0 tx=0).stop 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.788+0000 7f51f95f5740 1 -- 172.21.15.89:0/3713799298 shutdown_connections 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.788+0000 7f51f95f5740 1 --2- 172.21.15.89:0/3713799298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b844be0 0x55d31b831680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.788+0000 7f51f95f5740 1 --2- 172.21.15.89:0/3713799298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b83ab80 0x55d31b831140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.788+0000 7f51f95f5740 1 --2- 172.21.15.89:0/3713799298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b835af0 0x55d31b830c00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.788+0000 7f51f95f5740 1 -- 172.21.15.89:0/3713799298 >> 172.21.15.89:0/3713799298 conn(0x55d31b830820 msgr2=0x55d31b83f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.788+0000 7f51f95f5740 1 -- 172.21.15.89:0/3713799298 shutdown_connections 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.788+0000 7f51f95f5740 1 -- 172.21.15.89:0/3713799298 wait complete. 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.789+0000 7f51f95f5740 1 -- start start 2024-09-15T23:25:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.789+0000 7f51f95f5740 1 Processor -- start 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.790+0000 7f51f95f5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b835af0 0x55d31b85cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.790+0000 7f51f95f5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b83ab80 0x55d31b85d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.790+0000 7f51effff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b835af0 0x55d31b85cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.790+0000 7f51effff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b835af0 0x55d31b85cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39222/0 (socket says 172.21.15.89:39222) 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.790+0000 7f51effff640 1 -- 172.21.15.89:0/360013699 learned_addr learned my addr 172.21.15.89:0/360013699 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.790+0000 7f51f95f5740 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b844be0 0x55d31b85e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.790+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01a9a00 con 0x55d31b835af0 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.790+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01a9330 con 0x55d31b844be0 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.790+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f51f01a6570 con 0x55d31b83ab80 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.790+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b83ab80 0x55d31b85d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51f4df7640 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b844be0 0x55d31b85e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51ef7fe640 1 -- 172.21.15.89:0/360013699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b844be0 msgr2=0x55d31b85e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b844be0 0x55d31b85e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51ef7fe640 1 -- 172.21.15.89:0/360013699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b835af0 msgr2=0x55d31b85cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b835af0 0x55d31b85cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51ef7fe640 1 -- 172.21.15.89:0/360013699 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d31b85e6e0 con 0x55d31b83ab80 2024-09-15T23:25:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51effff640 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b835af0 0x55d31b85cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51f95f5740 10 client.0 _ll_register_callbacks cb 0x7f51e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e40198f0 con 0x55d31b83ab80 2024-09-15T23:25:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51ef7fe640 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b83ab80 0x55d31b85d0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f51e4000c00 tx=0x7f51e4011dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d1f15e0732ee5fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:25:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51cf7fe640 1 -- 172.21.15.89:0/360013699 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e401e040 con 0x55d31b83ab80 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.791+0000 7f51cf7fe640 1 -- 172.21.15.89:0/360013699 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f51e40050b0 con 0x55d31b83ab80 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51cf7fe640 1 -- 172.21.15.89:0/360013699 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51e40052f0 con 0x55d31b83ab80 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51f95f5740 10 client.25252 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55d31b85f7c0 con 0x55d31b83ab80 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51cf7fe640 1 -- 172.21.15.89:0/360013699 <== mon.1 v2:172.21.15.174:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f51e4005490 con 0x55d31b83ab80 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51f95f5740 10 client.25252 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51f95f5740 10 client.25252 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51f95f5740 20 client.25252 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51f95f5740 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d31b8b6130 0x55d31b8b8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51f95f5740 4 client.25252 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d31b797d70 con 0x55d31b8b6130 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.792+0000 7f51effff640 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d31b8b6130 0x55d31b8b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.793+0000 7f51effff640 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d31b8b6130 0x55d31b8b8520 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=586493c484b1a591 server_cookie=a18c969423b3f5ae in_seq=0 out_seq=0 2024-09-15T23:25:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.793+0000 7f51cf7fe640 10 client.25252 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51cf7fe640 1 -- 172.21.15.89:0/360013699 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d31b797d70 con 0x55d31b8b6130 2024-09-15T23:25:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51cf7fe640 10 client.25252 handle_command_reply: tid=0 2024-09-15T23:25:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 client.25252 shutdown 2024-09-15T23:25:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b83ab80 msgr2=0x55d31b85d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b83ab80 0x55d31b85d0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f51e4000c00 tx=0x7f51e4011dc0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 shutdown_connections 2024-09-15T23:25:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d31b8b6130 0x55d31b8b8520 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d31b844be0 0x55d31b85e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d31b83ab80 0x55d31b85d0c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 --2- 172.21.15.89:0/360013699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d31b835af0 0x55d31b85cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51cf7fe640 0 client.25252 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 >> 172.21.15.89:0/360013699 conn(0x55d31b830820 msgr2=0x55d31b8269a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 shutdown_connections 2024-09-15T23:25:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.798+0000 7f51f95f5740 1 -- 172.21.15.89:0/360013699 wait complete. 2024-09-15T23:25:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f95f5740 20 client.25252 trim_cache size 0 max 16384 2024-09-15T23:25:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:25:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:25:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:25:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:25:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 -- 172.21.15.89:0/2579718329 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51d80a3320 msgr2=0x7f51d80a57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51d80a3320 0x7f51d80a57e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f51e0000c00 tx=0x7f51e00315b0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 -- 172.21.15.89:0/2579718329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01025f0 msgr2=0x7f51f01a1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01025f0 0x7f51f01a1490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f51e800ec50 tx=0x7f51e800c6a0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 -- 172.21.15.89:0/2579718329 shutdown_connections 2024-09-15T23:25:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51d80a3320 0x7f51d80a57e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51f01025f0 0x7f51f01a1490 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51f0101c30 0x7f51f019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 --2- 172.21.15.89:0/2579718329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0100a40 0x7f51f006a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:33.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 -- 172.21.15.89:0/2579718329 >> 172.21.15.89:0/2579718329 conn(0x7f51f00fc1e0 msgr2=0x7f51f0104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:33.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 -- 172.21.15.89:0/2579718329 shutdown_connections 2024-09-15T23:25:33.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:33.799+0000 7f51f685a640 1 -- 172.21.15.89:0/2579718329 wait complete. 2024-09-15T23:25:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:34 smithi120 ceph-mon[35250]: pgmap v396: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 14 op/s 2024-09-15T23:25:34.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:34 smithi174 ceph-mon[35586]: pgmap v396: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 14 op/s 2024-09-15T23:25:34.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:34 smithi089 ceph-mon[30933]: pgmap v396: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 14 op/s 2024-09-15T23:25:35.282 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:25:35.283 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:25:35.652 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:35.995 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:25:36.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.238+0000 7f4ab6b31640 1 -- 172.21.15.89:0/1543088858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ab0105560 msgr2=0x7f4ab010d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.238+0000 7f4ab6b31640 1 --2- 172.21.15.89:0/1543088858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ab0105560 0x7f4ab010d0e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800adb0 tx=0x7f4aa802b7c0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.238+0000 7f4ab6b31640 1 -- 172.21.15.89:0/1543088858 shutdown_connections 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.238+0000 7f4ab6b31640 1 --2- 172.21.15.89:0/1543088858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ab0105560 0x7f4ab010d0e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.238+0000 7f4ab6b31640 1 --2- 172.21.15.89:0/1543088858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ab0100de0 0x7f4ab0104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.238+0000 7f4ab6b31640 1 --2- 172.21.15.89:0/1543088858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ab0100430 0x7f4ab0100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.238+0000 7f4ab6b31640 1 -- 172.21.15.89:0/1543088858 >> 172.21.15.89:0/1543088858 conn(0x7f4ab0077230 msgr2=0x7f4ab00fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.238+0000 7f4ab6b31640 1 -- 172.21.15.89:0/1543088858 shutdown_connections 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.238+0000 7f4ab6b31640 1 -- 172.21.15.89:0/1543088858 wait complete. 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.239+0000 7f4ab6b31640 1 Processor -- start 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.239+0000 7f4ab6b31640 1 -- start start 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.240+0000 7f4ab6b31640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ab0100430 0x7f4ab01961f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.240+0000 7f4ab6b31640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ab0100de0 0x7f4ab0196730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.240+0000 7f4ab6b31640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ab0105560 0x7f4ab019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.240+0000 7f4ab6b31640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab010fd90 con 0x7f4ab0100de0 2024-09-15T23:25:36.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.240+0000 7f4ab6b31640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab010fc10 con 0x7f4ab0100430 2024-09-15T23:25:36.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.240+0000 7f4ab6b31640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab010ff10 con 0x7f4ab0105560 2024-09-15T23:25:36.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.240+0000 7f4aaffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ab0100de0 0x7f4ab0196730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.240+0000 7f4aaffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ab0100de0 0x7f4ab0196730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55408/0 (socket says 172.21.15.89:55408) 2024-09-15T23:25:36.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.240+0000 7f4aaffff640 1 -- 172.21.15.89:0/838755680 learned_addr learned my addr 172.21.15.89:0/838755680 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:36.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.241+0000 7f4aaffff640 1 -- 172.21.15.89:0/838755680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ab0105560 msgr2=0x7f4ab019aac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:25:36.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.241+0000 7f4aaffff640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ab0105560 0x7f4ab019aac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.241+0000 7f4aaffff640 1 -- 172.21.15.89:0/838755680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ab0100430 msgr2=0x7f4ab01961f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:36.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.241+0000 7f4aaffff640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ab0100430 0x7f4ab01961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.241+0000 7f4aaffff640 1 -- 172.21.15.89:0/838755680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab019b1a0 con 0x7f4ab0100de0 2024-09-15T23:25:36.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.241+0000 7f4aaffff640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ab0100de0 0x7f4ab0196730 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00e9f0 tx=0x7f4a9c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.242+0000 7f4aadffb640 1 -- 172.21.15.89:0/838755680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c00cd80 con 0x7f4ab0100de0 2024-09-15T23:25:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.242+0000 7f4aadffb640 1 -- 172.21.15.89:0/838755680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c010910 con 0x7f4ab0100de0 2024-09-15T23:25:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.242+0000 7f4aadffb640 1 -- 172.21.15.89:0/838755680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c0188f0 con 0x7f4ab0100de0 2024-09-15T23:25:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.242+0000 7f4ab6b31640 1 -- 172.21.15.89:0/838755680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab019b490 con 0x7f4ab0100de0 2024-09-15T23:25:36.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.242+0000 7f4ab6b31640 1 -- 172.21.15.89:0/838755680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab019b8c0 con 0x7f4ab0100de0 2024-09-15T23:25:36.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.247+0000 7f4aadffb640 1 -- 172.21.15.89:0/838755680 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4a9c018a90 con 0x7f4ab0100de0 2024-09-15T23:25:36.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.248+0000 7f4ab8f0a740 1 Processor -- start 2024-09-15T23:25:36.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.248+0000 7f4aadffb640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4aa00a5550 0x7f4aa00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.248+0000 7f4aadffb640 1 -- 172.21.15.89:0/838755680 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4a9c0d00c0 con 0x7f4ab0100de0 2024-09-15T23:25:36.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.248+0000 7f4ab8f0a740 1 -- start start 2024-09-15T23:25:36.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4ab50ce640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4aa00a5550 0x7f4aa00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546cff40 0x55a1546d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546e33e0 0x55a1546e57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4ab8f0a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a154635280 con 0x55a1546d10e0 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4ab8f0a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a1545dcb10 con 0x55a1546cff40 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4ab8f0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a154631240 con 0x55a1546e33e0 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4ab48cd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4ab48cd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55420/0 (socket says 172.21.15.89:55420) 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4ab48cd640 1 -- 172.21.15.89:0/1628125138 learned_addr learned my addr 172.21.15.89:0/1628125138 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.249+0000 7f4aaffff640 1 --2- 172.21.15.89:0/1628125138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546cff40 0x55a1546d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.250+0000 7f4ab48cd640 1 -- 172.21.15.89:0/1628125138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546e33e0 msgr2=0x55a1546e57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.250+0000 7f4ab48cd640 1 --2- 172.21.15.89:0/1628125138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546e33e0 0x55a1546e57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.250+0000 7f4ab48cd640 1 -- 172.21.15.89:0/1628125138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546cff40 msgr2=0x55a1546d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.250+0000 7f4ab48cd640 1 --2- 172.21.15.89:0/1628125138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546cff40 0x55a1546d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.250+0000 7f4ab48cd640 1 -- 172.21.15.89:0/1628125138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1545e91f0 con 0x55a1546d10e0 2024-09-15T23:25:36.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.250+0000 7f4ab48cd640 1 --2- 172.21.15.89:0/1628125138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546cfa00 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400bb10 tx=0x7f4aa400d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d692b048969f7b0b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.250+0000 7f4ab50ce640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4aa00a5550 0x7f4aa00a7a10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55a1546cd1e0 tx=0x7f4aa8032ba0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.251+0000 7f4a9affd640 1 -- 172.21.15.89:0/1628125138 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4016070 con 0x55a1546d10e0 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.251+0000 7f4a9affd640 1 -- 172.21.15.89:0/1628125138 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4aa4005500 con 0x55a1546d10e0 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.251+0000 7f4a9affd640 1 -- 172.21.15.89:0/1628125138 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4010460 con 0x55a1546d10e0 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.251+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/1628125138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 msgr2=0x55a1546cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.251+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/1628125138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546cfa00 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400bb10 tx=0x7f4aa400d2e0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.252+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/1628125138 shutdown_connections 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.252+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/1628125138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546e33e0 0x55a1546e57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.252+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/1628125138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546cff40 0x55a1546d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.252+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/1628125138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546cfa00 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.252+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/1628125138 >> 172.21.15.89:0/1628125138 conn(0x55a1546cc0f0 msgr2=0x55a1546cc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.252+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/1628125138 shutdown_connections 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.252+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/1628125138 wait complete. 2024-09-15T23:25:36.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.252+0000 7f4ab8f0a740 1 -- start start 2024-09-15T23:25:36.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.252+0000 7f4ab8f0a740 1 Processor -- start 2024-09-15T23:25:36.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546cff40 0x55a1546d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 0x55a1546d3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 0x55a1546d4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab8f0a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a1546e6160 con 0x55a1546e33e0 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab8f0a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a1545eff10 con 0x55a1546d10e0 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab8f0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a1546e62e0 con 0x55a1546cff40 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab8f0a740 10 client.0 _ll_register_callbacks cb 0x55a1545b8a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab50ce640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 0x55a1546d4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab50ce640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 0x55a1546d4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55436/0 (socket says 172.21.15.89:55436) 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab50ce640 1 -- 172.21.15.89:0/345024155 learned_addr learned my addr 172.21.15.89:0/345024155 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab48cd640 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546cff40 0x55a1546d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4aaffff640 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 0x55a1546d3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab50ce640 1 -- 172.21.15.89:0/345024155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546cff40 msgr2=0x55a1546d37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab50ce640 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546cff40 0x55a1546d37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab50ce640 1 -- 172.21.15.89:0/345024155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 msgr2=0x55a1546d3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab50ce640 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 0x55a1546d3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.253+0000 7f4ab50ce640 1 -- 172.21.15.89:0/345024155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1543a66a0 con 0x55a1546e33e0 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.254+0000 7f4ab50ce640 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 0x55a1546d4230 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8032680 tx=0x7f4aa802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd3c6a073dd542e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.254+0000 7f4a8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.254+0000 7f4a8ffff640 1 -- 172.21.15.89:0/345024155 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa804d070 con 0x55a1546e33e0 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.254+0000 7f4a8ffff640 1 -- 172.21.15.89:0/345024155 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4aa8030920 con 0x55a1546e33e0 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.254+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/345024155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4aa4005500 con 0x55a1546e33e0 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.254+0000 7f4a8ffff640 1 -- 172.21.15.89:0/345024155 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4aa8057050 con 0x55a1546e33e0 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.254+0000 7f4ab8f0a740 10 client.15915 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.254+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/345024155 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a1546fa7d0 con 0x55a1546e33e0 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.255+0000 7f4a8ffff640 1 -- 172.21.15.89:0/345024155 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4aa4005500 con 0x55a1546e33e0 2024-09-15T23:25:36.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.255+0000 7f4a8ffff640 1 -- 172.21.15.89:0/345024155 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4aa8049bb0 con 0x55a1546e33e0 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.255+0000 7f4ab8f0a740 10 client.15915 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.255+0000 7f4ab8f0a740 10 client.15915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.255+0000 7f4ab8f0a740 20 client.15915 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.255+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1546fbb70 0x55a1546fdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.255+0000 7f4ab8f0a740 4 client.15915 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.255+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/345024155 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1546427f0 con 0x55a1546fbb70 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.255+0000 7f4ab48cd640 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1546fbb70 0x55a1546fdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.257+0000 7f4a8ffff640 10 client.15915 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.257+0000 7f4ab48cd640 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1546fbb70 0x55a1546fdf60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97884fba0e35a0be server_cookie=c61308ea7f13eef3 in_seq=0 out_seq=0 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.257+0000 7f4a8ffff640 1 -- 172.21.15.89:0/345024155 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a1546427f0 con 0x55a1546fbb70 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.257+0000 7f4a8ffff640 10 client.15915 handle_command_reply: tid=0 2024-09-15T23:25:36.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.257+0000 7f4ab8f0a740 1 client.15915 shutdown 2024-09-15T23:25:36.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.258+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/345024155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 msgr2=0x55a1546d4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.258+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 0x55a1546d4230 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8032680 tx=0x7f4aa802b7c0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.258+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/345024155 shutdown_connections 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.258+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1546fbb70 0x55a1546fdf60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.258+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 0x55a1546d4230 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.258+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 0x55a1546d3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.258+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/345024155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546cff40 0x55a1546d37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.258+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/345024155 >> 172.21.15.89:0/345024155 conn(0x55a1546cc0f0 msgr2=0x55a1546d1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.258+0000 7f4a8ffff640 0 client.15915 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.259+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/345024155 shutdown_connections 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.259+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/345024155 wait complete. 2024-09-15T23:25:36.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.259+0000 7f4ab8f0a740 20 client.15915 trim_cache size 0 max 16384 2024-09-15T23:25:36.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.293+0000 7f4ab8f0a740 1 Processor -- start 2024-09-15T23:25:36.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.293+0000 7f4ab8f0a740 1 -- start start 2024-09-15T23:25:36.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.293+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546cff40 0x55a1546d4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.293+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 0x55a154703770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.293+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 0x55a154703cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.293+0000 7f4ab8f0a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01acd70 con 0x55a1546e33e0 2024-09-15T23:25:36.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.293+0000 7f4ab8f0a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01ae8b0 con 0x55a1546d10e0 2024-09-15T23:25:36.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.293+0000 7f4ab8f0a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01ad440 con 0x55a1546cff40 2024-09-15T23:25:36.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4aaffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 0x55a154703770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4aaffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 0x55a154703770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:44466/0 (socket says 172.21.15.89:44466) 2024-09-15T23:25:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4aaffff640 1 -- 172.21.15.89:0/2251075602 learned_addr learned my addr 172.21.15.89:0/2251075602 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4ab48cd640 1 --2- 172.21.15.89:0/2251075602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546cff40 0x55a1546d4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4ab50ce640 1 --2- 172.21.15.89:0/2251075602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 0x55a154703cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4aaffff640 1 -- 172.21.15.89:0/2251075602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546cff40 msgr2=0x55a1546d4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4aaffff640 1 --2- 172.21.15.89:0/2251075602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546cff40 0x55a1546d4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4aaffff640 1 -- 172.21.15.89:0/2251075602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 msgr2=0x55a154703cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4aaffff640 1 --2- 172.21.15.89:0/2251075602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 0x55a154703cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4aaffff640 1 -- 172.21.15.89:0/2251075602 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1546c65d0 con 0x55a1546d10e0 2024-09-15T23:25:36.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.294+0000 7f4aaffff640 1 --2- 172.21.15.89:0/2251075602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 0x55a154703770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c010fa0 tx=0x7f4a9c0bc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=820d2f425702085 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.295+0000 7f4a9a7fc640 1 -- 172.21.15.89:0/2251075602 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c097e60 con 0x55a1546d10e0 2024-09-15T23:25:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.295+0000 7f4a9a7fc640 1 -- 172.21.15.89:0/2251075602 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c0b5260 con 0x55a1546d10e0 2024-09-15T23:25:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.295+0000 7f4a9a7fc640 1 -- 172.21.15.89:0/2251075602 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c0bc0b0 con 0x55a1546d10e0 2024-09-15T23:25:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.296+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/2251075602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 msgr2=0x55a154703770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.296+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/2251075602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 0x55a154703770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c010fa0 tx=0x7f4a9c0bc040 comp rx=0 tx=0).stop 2024-09-15T23:25:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.296+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/2251075602 shutdown_connections 2024-09-15T23:25:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.296+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/2251075602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546e33e0 0x55a154703cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.296+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/2251075602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546d10e0 0x55a154703770 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.296+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/2251075602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546cff40 0x55a1546d4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.296+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/2251075602 >> 172.21.15.89:0/2251075602 conn(0x55a1546cc0f0 msgr2=0x55a1546d5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:36.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.297+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/2251075602 shutdown_connections 2024-09-15T23:25:36.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.297+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/2251075602 wait complete. 2024-09-15T23:25:36.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.297+0000 7f4ab8f0a740 1 -- start start 2024-09-15T23:25:36.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.297+0000 7f4ab8f0a740 1 Processor -- start 2024-09-15T23:25:36.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546cff40 0x55a1546dd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4ab8f0a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546dd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4aaffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546dd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4aaffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546dd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55458/0 (socket says 172.21.15.89:55458) 2024-09-15T23:25:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4aaffff640 1 -- 172.21.15.89:0/50203326 learned_addr learned my addr 172.21.15.89:0/50203326 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546e33e0 0x55a1546d9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01ae1e0 con 0x55a1546d10e0 2024-09-15T23:25:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01adb10 con 0x55a1546cff40 2024-09-15T23:25:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab01aad50 con 0x55a1546e33e0 2024-09-15T23:25:36.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4ab8f0a740 10 client.0 _ll_register_callbacks cb 0x7f4aa8040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4ab50ce640 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546e33e0 0x55a1546d9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4ab48cd640 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546cff40 0x55a1546dd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4aaffff640 1 -- 172.21.15.89:0/50203326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546e33e0 msgr2=0x55a1546d9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4aaffff640 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546e33e0 0x55a1546d9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4aaffff640 1 -- 172.21.15.89:0/50203326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546cff40 msgr2=0x55a1546dd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4aaffff640 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546cff40 0x55a1546dd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.298+0000 7f4aaffff640 1 -- 172.21.15.89:0/50203326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1546d0950 con 0x55a1546d10e0 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.299+0000 7f4aaffff640 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546dd640 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c01a010 tx=0x7f4a9c009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d537002846fe3067 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.299+0000 7f4a8ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.299+0000 7f4a8ffff640 1 -- 172.21.15.89:0/50203326 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c080060 con 0x55a1546d10e0 2024-09-15T23:25:36.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.299+0000 7f4a8ffff640 1 -- 172.21.15.89:0/50203326 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4a9c0bc0b0 con 0x55a1546d10e0 2024-09-15T23:25:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.299+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a9c0b5260 con 0x55a1546d10e0 2024-09-15T23:25:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4a8ffff640 1 -- 172.21.15.89:0/50203326 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4a9c0b9030 con 0x55a1546d10e0 2024-09-15T23:25:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4ab8f0a740 10 client.15918 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a1546d6920 con 0x55a1546d10e0 2024-09-15T23:25:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4a8ffff640 1 -- 172.21.15.89:0/50203326 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9c0a03f0 con 0x55a1546d10e0 2024-09-15T23:25:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4a8ffff640 1 -- 172.21.15.89:0/50203326 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4a9c099240 con 0x55a1546d10e0 2024-09-15T23:25:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4ab8f0a740 10 client.15918 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4ab8f0a740 10 client.15918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:36.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4ab8f0a740 20 client.15918 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:36.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a154767690 0x55a154769ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:36.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4ab8f0a740 4 client.15918 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:25:36.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.300+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f4a9c0b5260 con 0x55a154767690 2024-09-15T23:25:36.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.301+0000 7f4ab48cd640 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a154767690 0x55a154769ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:36.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.301+0000 7f4ab48cd640 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a154767690 0x55a154769ab0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b1516edfed6c501 server_cookie=79027e64c167fc91 in_seq=0 out_seq=0 2024-09-15T23:25:36.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.301+0000 7f4a8ffff640 10 client.15918 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:36.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.302+0000 7f4a8ffff640 1 -- 172.21.15.89:0/50203326 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f4a9c0b5260 con 0x55a154767690 2024-09-15T23:25:36.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.302+0000 7f4a8ffff640 10 client.15918 handle_command_reply: tid=0 2024-09-15T23:25:36.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.302+0000 7f4ab8f0a740 1 client.15918 shutdown 2024-09-15T23:25:36.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.303+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 msgr2=0x55a1546dd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:36.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.303+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546dd640 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c01a010 tx=0x7f4a9c009590 comp rx=0 tx=0).stop 2024-09-15T23:25:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.303+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 shutdown_connections 2024-09-15T23:25:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.303+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a154767690 0x55a154769ab0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.303+0000 7f4a8ffff640 0 client.15918 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.303+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1546e33e0 0x55a1546d9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.303+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1546d10e0 0x55a1546dd640 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.303+0000 7f4ab8f0a740 1 --2- 172.21.15.89:0/50203326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1546cff40 0x55a1546dd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.303+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 >> 172.21.15.89:0/50203326 conn(0x55a1546cc0f0 msgr2=0x55a1546d4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.304+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 shutdown_connections 2024-09-15T23:25:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.304+0000 7f4ab8f0a740 1 -- 172.21.15.89:0/50203326 wait complete. 2024-09-15T23:25:36.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.304+0000 7f4ab8f0a740 20 client.15918 trim_cache size 0 max 16384 2024-09-15T23:25:36.307 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:25:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:25:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "6b62e615-c830-4618-8dad-4aace43bbcf5", 2024-09-15T23:25:36.307 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:25:36.307 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:25:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 -- 172.21.15.89:0/838755680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4aa00a5550 msgr2=0x7f4aa00a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4aa00a5550 0x7f4aa00a7a10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55a1546cd1e0 tx=0x7f4aa8032ba0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 -- 172.21.15.89:0/838755680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ab0100de0 msgr2=0x7f4ab0196730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ab0100de0 0x7f4ab0196730 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c00e9f0 tx=0x7f4a9c00eeb0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 -- 172.21.15.89:0/838755680 shutdown_connections 2024-09-15T23:25:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4aa00a5550 0x7f4aa00a7a10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ab0105560 0x7f4ab019aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4ab0100de0 0x7f4ab0196730 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 --2- 172.21.15.89:0/838755680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ab0100430 0x7f4ab01961f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:36.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 -- 172.21.15.89:0/838755680 >> 172.21.15.89:0/838755680 conn(0x7f4ab0077230 msgr2=0x7f4ab0103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:36.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 -- 172.21.15.89:0/838755680 shutdown_connections 2024-09-15T23:25:36.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:36.305+0000 7f4ab6b31640 1 -- 172.21.15.89:0/838755680 wait complete. 2024-09-15T23:25:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:36 smithi120 ceph-mon[35250]: pgmap v397: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-15T23:25:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:36 smithi174 ceph-mon[35586]: pgmap v397: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-15T23:25:36.542 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:36.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:36 smithi089 ceph-mon[30933]: pgmap v397: 129 pgs: 129 active+clean; 680 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-15T23:25:37.077 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:25:37.426 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:37 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:25:37.426 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:37 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:37.426 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:37 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:25:37.426 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:37 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:37.427 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:37 smithi089 ceph-mon[30933]: pgmap v398: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-09-15T23:25:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:37 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:25:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:37 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:37 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:25:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:37 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:37 smithi120 ceph-mon[35250]: pgmap v398: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-09-15T23:25:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:37 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:25:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:37 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:37 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:25:37.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:37 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:37.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:37 smithi174 ceph-mon[35586]: pgmap v398: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-09-15T23:25:37.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.716+0000 7fb6254c3640 1 -- 172.21.15.89:0/618024335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6201009b0 msgr2=0x7fb620100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:37.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.716+0000 7fb6254c3640 1 --2- 172.21.15.89:0/618024335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6201009b0 0x7fb620100d90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb610009900 tx=0x7fb61002a410 comp rx=0 tx=0).stop 2024-09-15T23:25:37.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.716+0000 7fb6254c3640 1 -- 172.21.15.89:0/618024335 shutdown_connections 2024-09-15T23:25:37.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.716+0000 7fb6254c3640 1 --2- 172.21.15.89:0/618024335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb62010e4d0 0x7fb6201108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:37.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.716+0000 7fb6254c3640 1 --2- 172.21.15.89:0/618024335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb620101360 0x7fb62010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:37.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.716+0000 7fb6254c3640 1 --2- 172.21.15.89:0/618024335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6201009b0 0x7fb620100d90 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:37.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.716+0000 7fb6254c3640 1 -- 172.21.15.89:0/618024335 >> 172.21.15.89:0/618024335 conn(0x7fb6200fc7d0 msgr2=0x7fb6200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:37.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.717+0000 7fb6254c3640 1 -- 172.21.15.89:0/618024335 shutdown_connections 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.717+0000 7fb6254c3640 1 -- 172.21.15.89:0/618024335 wait complete. 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb6254c3640 1 Processor -- start 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb6254c3640 1 -- start start 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb6254c3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6201009b0 0x7fb6201a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb6254c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb620101360 0x7fb6201a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb6254c3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb62010e4d0 0x7fb62019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb6254c3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb620114230 con 0x7fb62010e4d0 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb6254c3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201140b0 con 0x7fb6201009b0 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb61effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6201009b0 0x7fb6201a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb6254c3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb6201143b0 con 0x7fb620101360 2024-09-15T23:25:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb61effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6201009b0 0x7fb6201a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:44504/0 (socket says 172.21.15.89:44504) 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb61f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb62010e4d0 0x7fb62019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb61effd640 1 -- 172.21.15.89:0/3088009154 learned_addr learned my addr 172.21.15.89:0/3088009154 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.718+0000 7fb61e7fc640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb620101360 0x7fb6201a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb61f7fe640 1 -- 172.21.15.89:0/3088009154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb620101360 msgr2=0x7fb6201a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb61f7fe640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb620101360 0x7fb6201a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb61f7fe640 1 -- 172.21.15.89:0/3088009154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6201009b0 msgr2=0x7fb6201a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb61f7fe640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6201009b0 0x7fb6201a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb61f7fe640 1 -- 172.21.15.89:0/3088009154 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62019acf0 con 0x7fb62010e4d0 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb61effd640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6201009b0 0x7fb6201a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb61f7fe640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb62010e4d0 0x7fb62019a5c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb618009fd0 tx=0x7fb61800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:37.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb607fff640 1 -- 172.21.15.89:0/3088009154 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61800ce20 con 0x7fb62010e4d0 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb6254c3640 1 -- 172.21.15.89:0/3088009154 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62019afe0 con 0x7fb62010e4d0 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb607fff640 1 -- 172.21.15.89:0/3088009154 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb618005020 con 0x7fb62010e4d0 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb607fff640 1 -- 172.21.15.89:0/3088009154 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618010720 con 0x7fb62010e4d0 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.719+0000 7fb6254c3640 1 -- 172.21.15.89:0/3088009154 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6201a72d0 con 0x7fb62010e4d0 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.721+0000 7fb605ffb640 1 -- 172.21.15.89:0/3088009154 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb600005290 con 0x7fb62010e4d0 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.724+0000 7fb607fff640 1 -- 172.21.15.89:0/3088009154 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb618005540 con 0x7fb62010e4d0 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.725+0000 7fb607fff640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6080a3340 0x7fb6080a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.725+0000 7fb607fff640 1 -- 172.21.15.89:0/3088009154 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb6180d0940 con 0x7fb62010e4d0 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.725+0000 7fb61effd640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6080a3340 0x7fb6080a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.726+0000 7fb61effd640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6080a3340 0x7fb6080a5800 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb6100096c0 tx=0x7fb610005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:37.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:37.728+0000 7fb607fff640 1 -- 172.21.15.89:0/3088009154 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb61801d060 con 0x7fb62010e4d0 2024-09-15T23:25:37.791 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:38.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.055+0000 7fb605ffb640 1 -- 172.21.15.89:0/3088009154 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb600005580 con 0x7fb62010e4d0 2024-09-15T23:25:38.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.056+0000 7fb607fff640 1 -- 172.21.15.89:0/3088009154 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fb61809d210 con 0x7fb62010e4d0 2024-09-15T23:25:38.058 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:25:38.058 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:25:38.058 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:25:38.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.059+0000 7fb605ffb640 1 -- 172.21.15.89:0/3088009154 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6080a3340 msgr2=0x7fb6080a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.059+0000 7fb605ffb640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6080a3340 0x7fb6080a5800 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb6100096c0 tx=0x7fb610005a20 comp rx=0 tx=0).stop 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.059+0000 7fb605ffb640 1 -- 172.21.15.89:0/3088009154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb62010e4d0 msgr2=0x7fb62019a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.059+0000 7fb605ffb640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb62010e4d0 0x7fb62019a5c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb618009fd0 tx=0x7fb61800ef10 comp rx=0 tx=0).stop 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.059+0000 7fb605ffb640 1 -- 172.21.15.89:0/3088009154 shutdown_connections 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.059+0000 7fb605ffb640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6080a3340 0x7fb6080a5800 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.059+0000 7fb605ffb640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb62010e4d0 0x7fb62019a5c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.059+0000 7fb605ffb640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb620101360 0x7fb6201a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.059+0000 7fb605ffb640 1 --2- 172.21.15.89:0/3088009154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6201009b0 0x7fb6201a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.059+0000 7fb605ffb640 1 -- 172.21.15.89:0/3088009154 >> 172.21.15.89:0/3088009154 conn(0x7fb6200fc7d0 msgr2=0x7fb6200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.060+0000 7fb605ffb640 1 -- 172.21.15.89:0/3088009154 shutdown_connections 2024-09-15T23:25:38.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.060+0000 7fb605ffb640 1 -- 172.21.15.89:0/3088009154 wait complete. 2024-09-15T23:25:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:38 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:25:38.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:38 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3088009154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:38 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:25:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:38 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3088009154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:38 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:25:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:38 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3088009154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.537+0000 7f62d3a37640 1 -- 172.21.15.89:0/3505086333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62cc10e4d0 msgr2=0x7f62cc1108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.537+0000 7f62d3a37640 1 --2- 172.21.15.89:0/3505086333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62cc10e4d0 0x7f62cc1108c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009ca0 tx=0x7f62c402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.537+0000 7f62d3a37640 1 -- 172.21.15.89:0/3505086333 shutdown_connections 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.537+0000 7f62d3a37640 1 --2- 172.21.15.89:0/3505086333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62cc10e4d0 0x7f62cc1108c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.537+0000 7f62d3a37640 1 --2- 172.21.15.89:0/3505086333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62cc1012d0 0x7f62cc10df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.537+0000 7f62d3a37640 1 --2- 172.21.15.89:0/3505086333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62cc100920 0x7f62cc100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.537+0000 7f62d3a37640 1 -- 172.21.15.89:0/3505086333 >> 172.21.15.89:0/3505086333 conn(0x7f62cc0fc7d0 msgr2=0x7f62cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.537+0000 7f62d3a37640 1 -- 172.21.15.89:0/3505086333 shutdown_connections 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.537+0000 7f62d3a37640 1 -- 172.21.15.89:0/3505086333 wait complete. 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.538+0000 7f62d3a37640 1 Processor -- start 2024-09-15T23:25:38.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.538+0000 7f62d3a37640 1 -- start start 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.538+0000 7f62d3a37640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62cc100920 0x7f62cc19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.538+0000 7f62d3a37640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62cc1012d0 0x7f62cc19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.538+0000 7f62d3a37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62cc10e4d0 0x7f62cc19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.538+0000 7f62d3a37640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc114140 con 0x7f62cc10e4d0 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.538+0000 7f62d3a37640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc113fc0 con 0x7f62cc100920 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.538+0000 7f62d3a37640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc1142c0 con 0x7f62cc1012d0 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d1fd4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62cc10e4d0 0x7f62cc19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d1fd4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62cc10e4d0 0x7f62cc19eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55478/0 (socket says 172.21.15.89:55478) 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d1fd4640 1 -- 172.21.15.89:0/3571525177 learned_addr learned my addr 172.21.15.89:0/3571525177 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62cc1012d0 0x7f62cc19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d17d3640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62cc100920 0x7f62cc19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d0fd2640 1 -- 172.21.15.89:0/3571525177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62cc100920 msgr2=0x7f62cc19a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62cc100920 0x7f62cc19a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d0fd2640 1 -- 172.21.15.89:0/3571525177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62cc10e4d0 msgr2=0x7f62cc19eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62cc10e4d0 0x7f62cc19eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d0fd2640 1 -- 172.21.15.89:0/3571525177 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62cc19f590 con 0x7f62cc1012d0 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d1fd4640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62cc10e4d0 0x7f62cc19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.539+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62cc1012d0 0x7f62cc19ab20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f62bc00d6d0 tx=0x7f62bc00db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.540+0000 7f62c27fc640 1 -- 172.21.15.89:0/3571525177 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc017070 con 0x7f62cc1012d0 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.540+0000 7f62d3a37640 1 -- 172.21.15.89:0/3571525177 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62cc19f880 con 0x7f62cc1012d0 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.540+0000 7f62c27fc640 1 -- 172.21.15.89:0/3571525177 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62bc009820 con 0x7f62cc1012d0 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.540+0000 7f62c27fc640 1 -- 172.21.15.89:0/3571525177 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc004cd0 con 0x7f62cc1012d0 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.540+0000 7f62d3a37640 1 -- 172.21.15.89:0/3571525177 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62cc1a7160 con 0x7f62cc1012d0 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.543+0000 7f62d5e10740 1 Processor -- start 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.543+0000 7f62c27fc640 1 -- 172.21.15.89:0/3571525177 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f62bc0099d0 con 0x7f62cc1012d0 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.543+0000 7f62d5e10740 1 -- start start 2024-09-15T23:25:38.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a792447a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 0x555a79257950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7925a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d5e10740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555a791aa2e0 con 0x555a79244ce0 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d5e10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555a79151ce0 con 0x555a792443c0 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d5e10740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555a791a62a0 con 0x555a79257e90 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d1fd4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a792447a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d1fd4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a792447a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:44526/0 (socket says 172.21.15.89:44526) 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d1fd4640 1 -- 172.21.15.89:0/3021092699 learned_addr learned my addr 172.21.15.89:0/3021092699 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3021092699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7925a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.544+0000 7f62d17d3640 1 --2- 172.21.15.89:0/3021092699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 0x555a79257950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62d1fd4640 1 -- 172.21.15.89:0/3021092699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 msgr2=0x555a7925a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62d1fd4640 1 --2- 172.21.15.89:0/3021092699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7925a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62d1fd4640 1 -- 172.21.15.89:0/3021092699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 msgr2=0x555a79257950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62d1fd4640 1 --2- 172.21.15.89:0/3021092699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 0x555a79257950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62d1fd4640 1 -- 172.21.15.89:0/3021092699 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a79136440 con 0x555a792443c0 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62d1fd4640 1 --2- 172.21.15.89:0/3021092699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a792447a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f62c4004af0 tx=0x7f62c40098d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29889318e4c0fe21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62c27fc640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62b80a3340 0x7f62b80a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62b77fe640 1 -- 172.21.15.89:0/3021092699 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4003bf0 con 0x555a792443c0 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62b77fe640 1 -- 172.21.15.89:0/3021092699 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c4036040 con 0x555a792443c0 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62b77fe640 1 -- 172.21.15.89:0/3021092699 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4004790 con 0x555a792443c0 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62d17d3640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62b80a3340 0x7f62b80a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.545+0000 7f62c27fc640 1 -- 172.21.15.89:0/3571525177 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f62bc0cf830 con 0x7f62cc1012d0 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.546+0000 7f62d5e10740 1 -- 172.21.15.89:0/3021092699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 msgr2=0x555a792447a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.546+0000 7f62d17d3640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62b80a3340 0x7f62b80a5800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f62c8005fd0 tx=0x7f62c8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.546+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3021092699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a792447a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f62c4004af0 tx=0x7f62c40098d0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.546+0000 7f62d5e10740 1 -- 172.21.15.89:0/3021092699 shutdown_connections 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.546+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3021092699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7925a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.546+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3021092699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 0x555a79257950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.546+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3021092699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a792447a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.546+0000 7f62d5e10740 1 -- 172.21.15.89:0/3021092699 >> 172.21.15.89:0/3021092699 conn(0x555a79240a30 msgr2=0x555a79240e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.546+0000 7f62d5e10740 1 -- 172.21.15.89:0/3021092699 shutdown_connections 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.546+0000 7f62d5e10740 1 -- 172.21.15.89:0/3021092699 wait complete. 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.547+0000 7f62d5e10740 1 -- start start 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.547+0000 7f62d5e10740 1 Processor -- start 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.547+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a79257880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.547+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 0x555a79254b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.547+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a79255120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d5e10740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555a7925a820 con 0x555a79244ce0 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d5e10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555a79165050 con 0x555a792443c0 2024-09-15T23:25:38.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d5e10740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555a7925a9a0 con 0x555a79257e90 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d5e10740 10 client.0 _ll_register_callbacks cb 0x555a791a5160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d17d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a79257880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d0fd2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 0x555a79254b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d17d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a79257880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:44532/0 (socket says 172.21.15.89:44532) 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d17d3640 1 -- 172.21.15.89:0/3645883696 learned_addr learned my addr 172.21.15.89:0/3645883696 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d1fd4640 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a79255120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d17d3640 1 -- 172.21.15.89:0/3645883696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 msgr2=0x555a79255120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d17d3640 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a79255120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d17d3640 1 -- 172.21.15.89:0/3645883696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 msgr2=0x555a79254b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d17d3640 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 0x555a79254b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.548+0000 7f62d17d3640 1 -- 172.21.15.89:0/3645883696 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a78f1b3c0 con 0x555a792443c0 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.549+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 0x555a79254b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.549+0000 7f62d17d3640 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a79257880 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f62c800c850 tx=0x7f62c8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7552aaed37b5e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:38.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.549+0000 7f62b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.549+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3645883696 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8023070 con 0x555a792443c0 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.549+0000 7f62d5e10740 1 -- 172.21.15.89:0/3645883696 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c4036040 con 0x555a792443c0 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.549+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3645883696 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f62c801f650 con 0x555a792443c0 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.549+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3645883696 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f62c802f050 con 0x555a792443c0 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.549+0000 7f62d5e10740 10 client.25298 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.549+0000 7f62d5e10740 1 -- 172.21.15.89:0/3645883696 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555a79266320 con 0x555a792443c0 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.549+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3645883696 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c4036040 con 0x555a792443c0 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.550+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3645883696 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f62c800f040 con 0x555a792443c0 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.550+0000 7f62d5e10740 10 client.25298 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.550+0000 7f62d5e10740 10 client.25298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.550+0000 7f62d5e10740 20 client.25298 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.550+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555a79267730 0x555a79269b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.550+0000 7f62d5e10740 4 client.25298 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:38.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.550+0000 7f62d5e10740 1 -- 172.21.15.89:0/3645883696 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a791b7850 con 0x555a79267730 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.551+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555a79267730 0x555a79269b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.551+0000 7f62b4ff9640 10 client.25298 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.551+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555a79267730 0x555a79269b40 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dbc794c0a8d95c5 server_cookie=c776b37c662bf494 in_seq=0 out_seq=0 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3645883696 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555a791b7850 con 0x555a79267730 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62b4ff9640 10 client.25298 handle_command_reply: tid=0 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62d5e10740 1 client.25298 shutdown 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62d5e10740 1 -- 172.21.15.89:0/3645883696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 msgr2=0x555a79257880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a79257880 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f62c800c850 tx=0x7f62c8004730 comp rx=0 tx=0).stop 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62d5e10740 1 -- 172.21.15.89:0/3645883696 shutdown_connections 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555a79267730 0x555a79269b40 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a79255120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62b4ff9640 0 client.25298 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a79244ce0 0x555a79254b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3645883696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a792443c0 0x555a79257880 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.552+0000 7f62d5e10740 1 -- 172.21.15.89:0/3645883696 >> 172.21.15.89:0/3645883696 conn(0x555a79240a30 msgr2=0x555a7923fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:38.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.553+0000 7f62d5e10740 1 -- 172.21.15.89:0/3645883696 shutdown_connections 2024-09-15T23:25:38.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.553+0000 7f62d5e10740 1 -- 172.21.15.89:0/3645883696 wait complete. 2024-09-15T23:25:38.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.553+0000 7f62d5e10740 20 client.25298 trim_cache size 0 max 16384 2024-09-15T23:25:38.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.586+0000 7f62d5e10740 1 Processor -- start 2024-09-15T23:25:38.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.586+0000 7f62d5e10740 1 -- start start 2024-09-15T23:25:38.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.586+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 0x555a79263ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.586+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 0x555a7926f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.586+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7926f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d5e10740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc1acba0 con 0x555a792443c0 2024-09-15T23:25:38.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d5e10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc1ae6e0 con 0x555a79244ce0 2024-09-15T23:25:38.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d5e10740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc1ad270 con 0x555a79257e90 2024-09-15T23:25:38.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d1fd4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7926f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d0fd2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 0x555a7926f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d1fd4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7926f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:56352/0 (socket says 172.21.15.89:56352) 2024-09-15T23:25:38.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d1fd4640 1 -- 172.21.15.89:0/3772819540 learned_addr learned my addr 172.21.15.89:0/3772819540 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:38.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d17d3640 1 --2- 172.21.15.89:0/3772819540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 0x555a79263ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d1fd4640 1 -- 172.21.15.89:0/3772819540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 msgr2=0x555a7926f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:38.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d1fd4640 1 --2- 172.21.15.89:0/3772819540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 0x555a7926f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d1fd4640 1 -- 172.21.15.89:0/3772819540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 msgr2=0x555a79263ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:38.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d1fd4640 1 --2- 172.21.15.89:0/3772819540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 0x555a79263ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d1fd4640 1 -- 172.21.15.89:0/3772819540 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a79253ae0 con 0x555a79257e90 2024-09-15T23:25:38.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.587+0000 7f62d1fd4640 1 --2- 172.21.15.89:0/3772819540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7926f970 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009bf0 tx=0x7f62c40044b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eba4f1398521eaf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:38.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.588+0000 7f62b6ffd640 1 -- 172.21.15.89:0/3772819540 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4051070 con 0x555a79257e90 2024-09-15T23:25:38.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.588+0000 7f62b6ffd640 1 -- 172.21.15.89:0/3772819540 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c4036040 con 0x555a79257e90 2024-09-15T23:25:38.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.588+0000 7f62b6ffd640 1 -- 172.21.15.89:0/3772819540 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4003eb0 con 0x555a79257e90 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.589+0000 7f62d5e10740 1 -- 172.21.15.89:0/3772819540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 msgr2=0x555a7926f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.589+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3772819540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7926f970 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009bf0 tx=0x7f62c40044b0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.589+0000 7f62d5e10740 1 -- 172.21.15.89:0/3772819540 shutdown_connections 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.589+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3772819540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7926f970 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.589+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3772819540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 0x555a7926f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.589+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3772819540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 0x555a79263ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.589+0000 7f62d5e10740 1 -- 172.21.15.89:0/3772819540 >> 172.21.15.89:0/3772819540 conn(0x555a79240a30 msgr2=0x555a79250110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.589+0000 7f62d5e10740 1 -- 172.21.15.89:0/3772819540 shutdown_connections 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.589+0000 7f62d5e10740 1 -- 172.21.15.89:0/3772819540 wait complete. 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.590+0000 7f62d5e10740 1 -- start start 2024-09-15T23:25:38.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.590+0000 7f62d5e10740 1 Processor -- start 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.590+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 0x555a79260480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 0x555a7925b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d17d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 0x555a79260480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d5e10740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7925bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d5e10740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc1ae010 con 0x555a792443c0 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d5e10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc1ad940 con 0x555a79244ce0 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d17d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 0x555a79260480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55512/0 (socket says 172.21.15.89:55512) 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d17d3640 1 -- 172.21.15.89:0/3803917712 learned_addr learned my addr 172.21.15.89:0/3803917712 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 0x555a7925b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d1fd4640 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7925bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d5e10740 1 -- 172.21.15.89:0/3803917712 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc1aaba0 con 0x555a79257e90 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.591+0000 7f62d5e10740 10 client.0 _ll_register_callbacks cb 0x7f62c8016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62d0fd2640 1 -- 172.21.15.89:0/3803917712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 msgr2=0x555a7925bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7925bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62d0fd2640 1 -- 172.21.15.89:0/3803917712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 msgr2=0x555a79260480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 0x555a79260480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62d0fd2640 1 -- 172.21.15.89:0/3803917712 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a79255490 con 0x555a79244ce0 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62d5e10740 1 -- 172.21.15.89:0/3803917712 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c4036040 con 0x555a79244ce0 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62d0fd2640 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 0x555a7925b7b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009610 tx=0x7f62bc086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78e3026c9bddfd1b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62b4ff9640 10 client.25304 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3803917712 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc020040 con 0x555a79244ce0 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3803917712 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f62bc009d40 con 0x555a79244ce0 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3803917712 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f62bc0a4670 con 0x555a79244ce0 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.592+0000 7f62d5e10740 10 client.25304 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:38.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.593+0000 7f62d5e10740 1 -- 172.21.15.89:0/3803917712 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({fsmap=29}) -- 0x555a792d2290 con 0x555a79244ce0 2024-09-15T23:25:38.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.593+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3803917712 <== mon.2 v2:172.21.15.120:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f62bc098070 con 0x555a79244ce0 2024-09-15T23:25:38.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.593+0000 7f62d5e10740 10 client.25304 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:38.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.593+0000 7f62d5e10740 10 client.25304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:38.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.593+0000 7f62d5e10740 20 client.25304 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:38.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.593+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555a79267730 0x555a792d48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:38.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.593+0000 7f62d5e10740 4 client.25304 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:25:38.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.594+0000 7f62d5e10740 1 -- 172.21.15.89:0/3803917712 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555a791b1d60 con 0x555a79267730 2024-09-15T23:25:38.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.594+0000 7f62d17d3640 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555a79267730 0x555a792d48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:38.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.594+0000 7f62d17d3640 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555a79267730 0x555a792d48f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a106038d1f90e01e server_cookie=cc82ea6f9bd234ff in_seq=0 out_seq=0 2024-09-15T23:25:38.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.594+0000 7f62b4ff9640 10 client.25304 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:38.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.594+0000 7f62b4ff9640 1 -- 172.21.15.89:0/3803917712 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555a791b1d60 con 0x555a79267730 2024-09-15T23:25:38.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.594+0000 7f62b4ff9640 10 client.25304 handle_command_reply: tid=0 2024-09-15T23:25:38.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.594+0000 7f62d5e10740 1 client.25304 shutdown 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 1 -- 172.21.15.89:0/3803917712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 msgr2=0x555a7925b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 0x555a7925b7b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009610 tx=0x7f62bc086040 comp rx=0 tx=0).stop 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 1 -- 172.21.15.89:0/3803917712 shutdown_connections 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555a79267730 0x555a792d48f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555a79257e90 0x555a7925bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555a79244ce0 0x555a7925b7b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 1 --2- 172.21.15.89:0/3803917712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a792443c0 0x555a79260480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 1 -- 172.21.15.89:0/3803917712 >> 172.21.15.89:0/3803917712 conn(0x555a79240a30 msgr2=0x555a7923f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62b4ff9640 0 client.25304 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 1 -- 172.21.15.89:0/3803917712 shutdown_connections 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 1 -- 172.21.15.89:0/3803917712 wait complete. 2024-09-15T23:25:38.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.597+0000 7f62d5e10740 20 client.25304 trim_cache size 0 max 16384 2024-09-15T23:25:38.600 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:25:38.600 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:25:38.600 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:25:38.600 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 -- 172.21.15.89:0/3571525177 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62b80a3340 msgr2=0x7f62b80a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62b80a3340 0x7f62b80a5800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f62c8005fd0 tx=0x7f62c8005850 comp rx=0 tx=0).stop 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 -- 172.21.15.89:0/3571525177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62cc1012d0 msgr2=0x7f62cc19ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62cc1012d0 0x7f62cc19ab20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f62bc00d6d0 tx=0x7f62bc00db90 comp rx=0 tx=0).stop 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 -- 172.21.15.89:0/3571525177 shutdown_connections 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62b80a3340 0x7f62b80a5800 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62cc10e4d0 0x7f62cc19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62cc1012d0 0x7f62cc19ab20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 --2- 172.21.15.89:0/3571525177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62cc100920 0x7f62cc19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 -- 172.21.15.89:0/3571525177 >> 172.21.15.89:0/3571525177 conn(0x7f62cc0fc7d0 msgr2=0x7f62cc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 -- 172.21.15.89:0/3571525177 shutdown_connections 2024-09-15T23:25:38.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:38.599+0000 7f62d3a37640 1 -- 172.21.15.89:0/3571525177 wait complete. 2024-09-15T23:25:39.247 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:39 smithi089 ceph-mon[30933]: pgmap v399: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-09-15T23:25:39.276 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:25:39.277 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:25:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:39 smithi120 ceph-mon[35250]: pgmap v399: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-09-15T23:25:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:39 smithi174 ceph-mon[35586]: pgmap v399: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-09-15T23:25:39.816 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 90d6d4a2 --release --await' 2024-09-15T23:25:39.816 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 90d6d4a2 --release --await 2024-09-15T23:25:39.869 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:40.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 -- 172.21.15.89:0/2695166441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf4101de0 msgr2=0x7fddf41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:40.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 --2- 172.21.15.89:0/2695166441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf4101de0 0x7fddf41021c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009900 tx=0x7fdde402a410 comp rx=0 tx=0).stop 2024-09-15T23:25:40.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 -- 172.21.15.89:0/2695166441 shutdown_connections 2024-09-15T23:25:40.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 --2- 172.21.15.89:0/2695166441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddf4106f10 0x7fddf410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:40.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 --2- 172.21.15.89:0/2695166441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddf4102790 0x7fddf41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:40.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 --2- 172.21.15.89:0/2695166441 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf4101de0 0x7fddf41021c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:40.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 -- 172.21.15.89:0/2695166441 >> 172.21.15.89:0/2695166441 conn(0x7fddf40fd880 msgr2=0x7fddf40ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:40.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 -- 172.21.15.89:0/2695166441 shutdown_connections 2024-09-15T23:25:40.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 -- 172.21.15.89:0/2695166441 wait complete. 2024-09-15T23:25:40.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 Processor -- start 2024-09-15T23:25:40.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.589+0000 7fddf945b640 1 -- start start 2024-09-15T23:25:40.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf945b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf4101de0 0x7fddf4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:40.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf945b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddf4102790 0x7fddf4198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:40.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf945b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddf4106f10 0x7fddf419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:40.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf945b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fddf4111f70 con 0x7fddf4101de0 2024-09-15T23:25:40.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf945b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fddf4111df0 con 0x7fddf4102790 2024-09-15T23:25:40.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf945b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fddf41120f0 con 0x7fddf4106f10 2024-09-15T23:25:40.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf4101de0 0x7fddf4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:40.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf4101de0 0x7fddf4198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55536/0 (socket says 172.21.15.89:55536) 2024-09-15T23:25:40.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf2ffd640 1 -- 172.21.15.89:0/3097898712 learned_addr learned my addr 172.21.15.89:0/3097898712 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:40.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf37fe640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddf4106f10 0x7fddf419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:40.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddebfff640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddf4102790 0x7fddf4198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:40.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf2ffd640 1 -- 172.21.15.89:0/3097898712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddf4106f10 msgr2=0x7fddf419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:40.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf2ffd640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddf4106f10 0x7fddf419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:40.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf2ffd640 1 -- 172.21.15.89:0/3097898712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddf4102790 msgr2=0x7fddf4198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:40.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf2ffd640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddf4102790 0x7fddf4198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:40.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf2ffd640 1 -- 172.21.15.89:0/3097898712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf419d3d0 con 0x7fddf4101de0 2024-09-15T23:25:40.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf2ffd640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf4101de0 0x7fddf4198420 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fdde4005ab0 tx=0x7fdde40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:40.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf0ff9640 1 -- 172.21.15.89:0/3097898712 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde403f030 con 0x7fddf4101de0 2024-09-15T23:25:40.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.590+0000 7fddf945b640 1 -- 172.21.15.89:0/3097898712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddf419d660 con 0x7fddf4101de0 2024-09-15T23:25:40.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.591+0000 7fddf0ff9640 1 -- 172.21.15.89:0/3097898712 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdde4003ce0 con 0x7fddf4101de0 2024-09-15T23:25:40.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.591+0000 7fddf0ff9640 1 -- 172.21.15.89:0/3097898712 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde4035070 con 0x7fddf4101de0 2024-09-15T23:25:40.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.591+0000 7fddf945b640 1 -- 172.21.15.89:0/3097898712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddf41a4fa0 con 0x7fddf4101de0 2024-09-15T23:25:40.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.592+0000 7fddf945b640 1 -- 172.21.15.89:0/3097898712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddf41021c0 con 0x7fddf4101de0 2024-09-15T23:25:40.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.593+0000 7fddf0ff9640 1 -- 172.21.15.89:0/3097898712 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fdde4042030 con 0x7fddf4101de0 2024-09-15T23:25:40.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.593+0000 7fddf0ff9640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdde00a3240 0x7fdde00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:40.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.594+0000 7fddebfff640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdde00a3240 0x7fdde00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:40.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.594+0000 7fddf0ff9640 1 -- 172.21.15.89:0/3097898712 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdde40f0860 con 0x7fddf4101de0 2024-09-15T23:25:40.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.594+0000 7fddebfff640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdde00a3240 0x7fdde00a5700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fddf40ffdf0 tx=0x7fdddc005c60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:40.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.597+0000 7fddf0ff9640 1 -- 172.21.15.89:0/3097898712 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdde40bd190 con 0x7fddf4101de0 2024-09-15T23:25:40.740 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:40.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.820+0000 7fddf945b640 1 -- 172.21.15.89:0/3097898712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddf410a0f0 con 0x7fddf4101de0 2024-09-15T23:25:40.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.821+0000 7fddf0ff9640 1 -- 172.21.15.89:0/3097898712 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fdde40bca90 con 0x7fddf4101de0 2024-09-15T23:25:40.823 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:25:40.823 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:25:40.823 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:25:40.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 -- 172.21.15.89:0/3097898712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdde00a3240 msgr2=0x7fdde00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdde00a3240 0x7fdde00a5700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fddf40ffdf0 tx=0x7fdddc005c60 comp rx=0 tx=0).stop 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 -- 172.21.15.89:0/3097898712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf4101de0 msgr2=0x7fddf4198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf4101de0 0x7fddf4198420 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fdde4005ab0 tx=0x7fdde40042f0 comp rx=0 tx=0).stop 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 -- 172.21.15.89:0/3097898712 shutdown_connections 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdde00a3240 0x7fdde00a5700 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddf4106f10 0x7fddf419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddf4102790 0x7fddf4198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 --2- 172.21.15.89:0/3097898712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddf4101de0 0x7fddf4198420 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 -- 172.21.15.89:0/3097898712 >> 172.21.15.89:0/3097898712 conn(0x7fddf40fd880 msgr2=0x7fddf40fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 -- 172.21.15.89:0/3097898712 shutdown_connections 2024-09-15T23:25:40.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:40.823+0000 7fddf945b640 1 -- 172.21.15.89:0/3097898712 wait complete. 2024-09-15T23:25:41.077 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:40 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3097898712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:41.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:40 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3097898712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:41.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:40 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3097898712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:41.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.532+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1653196145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d0100960 msgr2=0x7fc7d0100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:41.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.532+0000 7fc7d81b6640 1 --2- 172.21.15.89:0/1653196145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d0100960 0x7fc7d0100d40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009900 tx=0x7fc7cc02a410 comp rx=0 tx=0).stop 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.532+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1653196145 shutdown_connections 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.532+0000 7fc7d81b6640 1 --2- 172.21.15.89:0/1653196145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d010e510 0x7fc7d0110900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.532+0000 7fc7d81b6640 1 --2- 172.21.15.89:0/1653196145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d0101310 0x7fc7d010dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.532+0000 7fc7d81b6640 1 --2- 172.21.15.89:0/1653196145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d0100960 0x7fc7d0100d40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.532+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1653196145 >> 172.21.15.89:0/1653196145 conn(0x7fc7d00fc7d0 msgr2=0x7fc7d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.532+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1653196145 shutdown_connections 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.532+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1653196145 wait complete. 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d81b6640 1 Processor -- start 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d81b6640 1 -- start start 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d81b6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d0100960 0x7fc7d019a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d81b6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d0101310 0x7fc7d019ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d81b6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d010e510 0x7fc7d019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d81b6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d0114220 con 0x7fc7d010e510 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d81b6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d01140a0 con 0x7fc7d0101310 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d81b6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d01143a0 con 0x7fc7d0100960 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d6753640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d010e510 0x7fc7d019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d6753640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d010e510 0x7fc7d019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55552/0 (socket says 172.21.15.89:55552) 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d6753640 1 -- 172.21.15.89:0/1317173298 learned_addr learned my addr 172.21.15.89:0/1317173298 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d0100960 0x7fc7d019a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d5751640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d0101310 0x7fc7d019ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d6753640 1 -- 172.21.15.89:0/1317173298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d0100960 msgr2=0x7fc7d019a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d6753640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d0100960 0x7fc7d019a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d6753640 1 -- 172.21.15.89:0/1317173298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d0101310 msgr2=0x7fc7d019ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d6753640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d0101310 0x7fc7d019ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.533+0000 7fc7d6753640 1 -- 172.21.15.89:0/1317173298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d019f6f0 con 0x7fc7d010e510 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.534+0000 7fc7d6753640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d010e510 0x7fc7d019f010 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800b9e0 tx=0x7fc7c800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.534+0000 7fc7c6ffd640 1 -- 172.21.15.89:0/1317173298 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8019070 con 0x7fc7d010e510 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.534+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1317173298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d019f9e0 con 0x7fc7d010e510 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.534+0000 7fc7c6ffd640 1 -- 172.21.15.89:0/1317173298 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c8014440 con 0x7fc7d010e510 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.534+0000 7fc7c6ffd640 1 -- 172.21.15.89:0/1317173298 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80125a0 con 0x7fc7d010e510 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.534+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1317173298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d01a72c0 con 0x7fc7d010e510 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.536+0000 7fc7c6ffd640 1 -- 172.21.15.89:0/1317173298 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc7c801c070 con 0x7fc7d010e510 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.537+0000 7fc7c6ffd640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7bc0a3240 0x7fc7bc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.537+0000 7fc7da58f740 1 Processor -- start 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.537+0000 7fc7c6ffd640 1 -- 172.21.15.89:0/1317173298 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc7c80cf2e0 con 0x7fc7d010e510 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.537+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7bc0a3240 0x7fc7bc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.537+0000 7fc7da58f740 1 -- start start 2024-09-15T23:25:41.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.537+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4060 0x555cceac4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.537+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 0x555cceaced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.537+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555ccead1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7da58f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555ccea29620 con 0x555cceacf240 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7da58f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555cce9d0e50 con 0x555cceac4980 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7da58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555ccea255e0 con 0x555cceac4060 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7bc0a3240 0x7fc7bc0a5700 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc000c00 tx=0x7fc7cc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d6753640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555ccead1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d6753640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555ccead1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55564/0 (socket says 172.21.15.89:55564) 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d6753640 1 -- 172.21.15.89:0/2739061022 learned_addr learned my addr 172.21.15.89:0/2739061022 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/2739061022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4060 0x555cceac4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d5751640 1 --2- 172.21.15.89:0/2739061022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 0x555cceaced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d6753640 1 -- 172.21.15.89:0/2739061022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4060 msgr2=0x555cceac4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d6753640 1 --2- 172.21.15.89:0/2739061022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4060 0x555cceac4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d6753640 1 -- 172.21.15.89:0/2739061022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 msgr2=0x555cceaced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:41.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d6753640 1 --2- 172.21.15.89:0/2739061022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 0x555cceaced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d6753640 1 -- 172.21.15.89:0/2739061022 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cce9c1d40 con 0x555cceacf240 2024-09-15T23:25:41.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7d6753640 1 --2- 172.21.15.89:0/2739061022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555ccead1630 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8098f00 tx=0x7fc7c809be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df002e118daaa75 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:41.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7bbfff640 1 -- 172.21.15.89:0/2739061022 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80a50f0 con 0x555cceacf240 2024-09-15T23:25:41.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7bbfff640 1 -- 172.21.15.89:0/2739061022 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c80cce50 con 0x555cceacf240 2024-09-15T23:25:41.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.538+0000 7fc7bbfff640 1 -- 172.21.15.89:0/2739061022 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8098d40 con 0x555cceacf240 2024-09-15T23:25:41.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.539+0000 7fc7da58f740 1 -- 172.21.15.89:0/2739061022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 msgr2=0x555ccead1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:41.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.539+0000 7fc7da58f740 1 --2- 172.21.15.89:0/2739061022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555ccead1630 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8098f00 tx=0x7fc7c809be90 comp rx=0 tx=0).stop 2024-09-15T23:25:41.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.539+0000 7fc7da58f740 1 -- 172.21.15.89:0/2739061022 shutdown_connections 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.539+0000 7fc7da58f740 1 --2- 172.21.15.89:0/2739061022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555ccead1630 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.539+0000 7fc7da58f740 1 --2- 172.21.15.89:0/2739061022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 0x555cceaced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.539+0000 7fc7da58f740 1 --2- 172.21.15.89:0/2739061022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4060 0x555cceac4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.539+0000 7fc7da58f740 1 -- 172.21.15.89:0/2739061022 >> 172.21.15.89:0/2739061022 conn(0x555cceac05f0 msgr2=0x555cceac09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.539+0000 7fc7da58f740 1 -- 172.21.15.89:0/2739061022 shutdown_connections 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.539+0000 7fc7da58f740 1 -- 172.21.15.89:0/2739061022 wait complete. 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.540+0000 7fc7da58f740 1 -- start start 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.540+0000 7fc7da58f740 1 Processor -- start 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 0x555cceacc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4980 0x555cceacc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceaccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7da58f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555ccead1e10 con 0x555cceacf240 2024-09-15T23:25:41.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7da58f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555cce9b1f10 con 0x555cceac4060 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7da58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555ccead1f90 con 0x555cceac4980 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7da58f740 10 client.0 _ll_register_callbacks cb 0x555cce9d5db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d6753640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceaccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d6753640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceaccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55580/0 (socket says 172.21.15.89:55580) 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d6753640 1 -- 172.21.15.89:0/3663794286 learned_addr learned my addr 172.21.15.89:0/3663794286 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 0x555cceacc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d5751640 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4980 0x555cceacc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d6753640 1 -- 172.21.15.89:0/3663794286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4980 msgr2=0x555cceacc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d6753640 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4980 0x555cceacc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d6753640 1 -- 172.21.15.89:0/3663794286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 msgr2=0x555cceacc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d6753640 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 0x555cceacc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d6753640 1 -- 172.21.15.89:0/3663794286 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cce738580 con 0x555cceacf240 2024-09-15T23:25:41.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d5751640 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4980 0x555cceacc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7d6753640 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceaccb90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80099a0 tx=0x7fc7c800c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5951aaf31c17671b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3663794286 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80b8020 con 0x555cceacf240 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3663794286 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc7c80b4260 con 0x555cceacf240 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.541+0000 7fc7da58f740 1 -- 172.21.15.89:0/3663794286 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7c80cce50 con 0x555cceacf240 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3663794286 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc7c80b4460 con 0x555cceacf240 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7da58f740 10 client.15960 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7da58f740 1 -- 172.21.15.89:0/3663794286 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555cce802960 con 0x555cceacf240 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3663794286 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c80cce50 con 0x555cceacf240 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3663794286 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc7c8098280 con 0x555cceacf240 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7da58f740 10 client.15960 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7da58f740 10 client.15960 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:25:41.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7da58f740 20 client.15960 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555cceadec20 0x555cceae1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7da58f740 4 client.15960 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7da58f740 1 -- 172.21.15.89:0/3663794286 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ccea36b90 con 0x555cceadec20 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.542+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555cceadec20 0x555cceae1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.543+0000 7fc7b97fa640 10 client.15960 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.543+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555cceadec20 0x555cceae1010 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdf72b2ee48f5f3a server_cookie=744e29eb9e0d601e in_seq=0 out_seq=0 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3663794286 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555ccea36b90 con 0x555cceadec20 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7b97fa640 10 client.15960 handle_command_reply: tid=0 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 client.15960 shutdown 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 -- 172.21.15.89:0/3663794286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 msgr2=0x555cceaccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceaccb90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80099a0 tx=0x7fc7c800c580 comp rx=0 tx=0).stop 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 -- 172.21.15.89:0/3663794286 shutdown_connections 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555cceadec20 0x555cceae1010 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceaccb90 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7b97fa640 0 client.15960 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:41.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4980 0x555cceacc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3663794286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 0x555cceacc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 -- 172.21.15.89:0/3663794286 >> 172.21.15.89:0/3663794286 conn(0x555cceac05f0 msgr2=0x555cceac6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:41.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 -- 172.21.15.89:0/3663794286 shutdown_connections 2024-09-15T23:25:41.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 1 -- 172.21.15.89:0/3663794286 wait complete. 2024-09-15T23:25:41.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.544+0000 7fc7da58f740 20 client.15960 trim_cache size 0 max 16384 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.580+0000 7fc7da58f740 1 Processor -- start 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.580+0000 7fc7da58f740 1 -- start start 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4060 0x555cceacafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 0x555cceac9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceac98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7da58f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d01b10f0 con 0x555cceacf240 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7da58f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d01b2c30 con 0x555cceac4980 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7da58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d01b17c0 con 0x555cceac4060 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d6753640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceac98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d6753640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceac98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55586/0 (socket says 172.21.15.89:55586) 2024-09-15T23:25:41.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d6753640 1 -- 172.21.15.89:0/2997285410 learned_addr learned my addr 172.21.15.89:0/2997285410 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:41.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d5751640 1 --2- 172.21.15.89:0/2997285410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 0x555cceac9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/2997285410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4060 0x555cceacafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d5751640 1 -- 172.21.15.89:0/2997285410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4060 msgr2=0x555cceacafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:41.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d5751640 1 --2- 172.21.15.89:0/2997285410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4060 0x555cceacafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d5751640 1 -- 172.21.15.89:0/2997285410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 msgr2=0x555cceac98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:41.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d5751640 1 --2- 172.21.15.89:0/2997285410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceac98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d5751640 1 -- 172.21.15.89:0/2997285410 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cceaca660 con 0x555cceac4980 2024-09-15T23:25:41.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.581+0000 7fc7d6753640 1 --2- 172.21.15.89:0/2997285410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceac98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:41.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.582+0000 7fc7d5751640 1 --2- 172.21.15.89:0/2997285410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 0x555cceac9370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x555cceacd750 tx=0x7fc7c000af90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cb8b294b0c7f850 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:41.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.582+0000 7fc7bb7fe640 1 -- 172.21.15.89:0/2997285410 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0012d70 con 0x555cceac4980 2024-09-15T23:25:41.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.582+0000 7fc7bb7fe640 1 -- 172.21.15.89:0/2997285410 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c0005500 con 0x555cceac4980 2024-09-15T23:25:41.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.582+0000 7fc7bb7fe640 1 -- 172.21.15.89:0/2997285410 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000f680 con 0x555cceac4980 2024-09-15T23:25:41.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.583+0000 7fc7da58f740 1 -- 172.21.15.89:0/2997285410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 msgr2=0x555cceac9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:41.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.583+0000 7fc7da58f740 1 --2- 172.21.15.89:0/2997285410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 0x555cceac9370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x555cceacd750 tx=0x7fc7c000af90 comp rx=0 tx=0).stop 2024-09-15T23:25:41.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.583+0000 7fc7da58f740 1 -- 172.21.15.89:0/2997285410 shutdown_connections 2024-09-15T23:25:41.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.583+0000 7fc7da58f740 1 --2- 172.21.15.89:0/2997285410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceacf240 0x555cceac98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.583+0000 7fc7da58f740 1 --2- 172.21.15.89:0/2997285410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4980 0x555cceac9370 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.583+0000 7fc7da58f740 1 --2- 172.21.15.89:0/2997285410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceac4060 0x555cceacafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.583+0000 7fc7da58f740 1 -- 172.21.15.89:0/2997285410 >> 172.21.15.89:0/2997285410 conn(0x555cceac05f0 msgr2=0x555cceaca8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.583+0000 7fc7da58f740 1 -- 172.21.15.89:0/2997285410 shutdown_connections 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.583+0000 7fc7da58f740 1 -- 172.21.15.89:0/2997285410 wait complete. 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.583+0000 7fc7da58f740 1 -- start start 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7da58f740 1 Processor -- start 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 0x555cceae26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceac4980 0x555cceae2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7da58f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceacf240 0x555cceb425a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7da58f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d01b2560 con 0x555cceac4980 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7da58f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d01b1e90 con 0x555cceac4060 2024-09-15T23:25:41.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7da58f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d01af0d0 con 0x555cceacf240 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7da58f740 10 client.0 _ll_register_callbacks cb 0x7fc7c80a08f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7d5f52640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 0x555cceae26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7d5f52640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 0x555cceae26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:44606/0 (socket says 172.21.15.89:44606) 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7d5f52640 1 -- 172.21.15.89:0/3341412600 learned_addr learned my addr 172.21.15.89:0/3341412600 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7d6753640 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceacf240 0x555cceb425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.584+0000 7fc7d5751640 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceac4980 0x555cceae2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7d5f52640 1 -- 172.21.15.89:0/3341412600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceacf240 msgr2=0x555cceb425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceacf240 0x555cceb425a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7d5f52640 1 -- 172.21.15.89:0/3341412600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceac4980 msgr2=0x555cceae2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceac4980 0x555cceae2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7d5f52640 1 -- 172.21.15.89:0/3341412600 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cceb42c40 con 0x555cceac4060 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7d6753640 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceacf240 0x555cceb425a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7d5f52640 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 0x555cceae26c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009230 tx=0x7fc7cc009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cbb211e900cdc46 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:41.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3341412600 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc035040 con 0x555cceac4060 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3341412600 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc7cc003800 con 0x555cceac4060 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7da58f740 1 -- 172.21.15.89:0/3341412600 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7c0005500 con 0x555cceac4060 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3341412600 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc7cc067050 con 0x555cceac4060 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.585+0000 7fc7da58f740 10 client.25319 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.586+0000 7fc7da58f740 1 -- 172.21.15.89:0/3341412600 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555cceb4a070 con 0x555cceac4060 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.586+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3341412600 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7cc056420 con 0x555cceac4060 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.586+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3341412600 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc7cc05e460 con 0x555cceac4060 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.586+0000 7fc7da58f740 10 client.25319 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.586+0000 7fc7da58f740 10 client.25319 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:25:41.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.586+0000 7fc7da58f740 20 client.25319 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:41.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.586+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555cceb4a770 0x555cceb4cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:41.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.586+0000 7fc7da58f740 4 client.25319 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "90d6d4a2", "release": true, "await": true}] 2024-09-15T23:25:41.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.586+0000 7fc7da58f740 1 -- 172.21.15.89:0/3341412600 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "90d6d4a2", "release": true, "await": true}) -- 0x7fc7c0005500 con 0x555cceb4a770 2024-09-15T23:25:41.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.586+0000 7fc7d5751640 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555cceb4a770 0x555cceb4cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:41.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.587+0000 7fc7b97fa640 10 client.25319 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:41.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.587+0000 7fc7d5751640 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555cceb4a770 0x555cceb4cb90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66c0e6fce9f7edc7 server_cookie=eb70f6f7dd0c0f54 in_seq=0 out_seq=0 2024-09-15T23:25:41.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.588+0000 7fc7b97fa640 1 -- 172.21.15.89:0/3341412600 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x7fc7c0005500 con 0x555cceb4a770 2024-09-15T23:25:41.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.588+0000 7fc7b97fa640 10 client.25319 handle_command_reply: tid=0 2024-09-15T23:25:41.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.588+0000 7fc7da58f740 1 client.25319 shutdown 2024-09-15T23:25:41.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.588+0000 7fc7da58f740 1 -- 172.21.15.89:0/3341412600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 msgr2=0x555cceae26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:41.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.588+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 0x555cceae26c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009230 tx=0x7fc7cc009260 comp rx=0 tx=0).stop 2024-09-15T23:25:41.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.589+0000 7fc7da58f740 1 -- 172.21.15.89:0/3341412600 shutdown_connections 2024-09-15T23:25:41.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.589+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555cceb4a770 0x555cceb4cb90 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.589+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555cceacf240 0x555cceb425a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.589+0000 7fc7b97fa640 0 client.25319 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:41.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.589+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cceac4980 0x555cceae2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.589+0000 7fc7da58f740 1 --2- 172.21.15.89:0/3341412600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cceac4060 0x555cceae26c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.589+0000 7fc7da58f740 1 -- 172.21.15.89:0/3341412600 >> 172.21.15.89:0/3341412600 conn(0x555cceac05f0 msgr2=0x555cceacb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:41.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.589+0000 7fc7da58f740 1 -- 172.21.15.89:0/3341412600 shutdown_connections 2024-09-15T23:25:41.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.589+0000 7fc7da58f740 1 -- 172.21.15.89:0/3341412600 wait complete. 2024-09-15T23:25:41.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.589+0000 7fc7da58f740 20 client.25319 trim_cache size 0 max 16384 2024-09-15T23:25:41.593 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 4, 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "90d6d4a2": { 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 4, 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:25:41.594 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 150.2, 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:25:41.595 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:25:41.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.590+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1317173298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7bc0a3240 msgr2=0x7fc7bc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:41.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.590+0000 7fc7d81b6640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7bc0a3240 0x7fc7bc0a5700 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc000c00 tx=0x7fc7cc0058d0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.590+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1317173298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d010e510 msgr2=0x7fc7d019f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:41.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.590+0000 7fc7d81b6640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d010e510 0x7fc7d019f010 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800b9e0 tx=0x7fc7c800bea0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.591+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1317173298 shutdown_connections 2024-09-15T23:25:41.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.591+0000 7fc7d81b6640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc7bc0a3240 0x7fc7bc0a5700 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.591+0000 7fc7d81b6640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc7d010e510 0x7fc7d019f010 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.591+0000 7fc7d81b6640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d0101310 0x7fc7d019ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.591+0000 7fc7d81b6640 1 --2- 172.21.15.89:0/1317173298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d0100960 0x7fc7d019a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:41.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.591+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1317173298 >> 172.21.15.89:0/1317173298 conn(0x7fc7d00fc7d0 msgr2=0x7fc7d010f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:41.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.591+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1317173298 shutdown_connections 2024-09-15T23:25:41.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:41.591+0000 7fc7d81b6640 1 -- 172.21.15.89:0/1317173298 wait complete. 2024-09-15T23:25:41.794 INFO:tasks.workunit.client.0.smithi089.stderr:Updating files: 94% (12143/12889) Updating files: 95% (12245/12889) Updating files: 96% (12374/12889) Updating files: 97% (12503/12889) Updating files: 98% (12632/12889) Updating files: 99% (12761/12889) Updating files: 100% (12889/12889) Updating files: 100% (12889/12889), done. 2024-09-15T23:25:42.135 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:41 smithi089 ceph-mon[30933]: pgmap v400: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-15T23:25:42.163 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:25:42.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:41 smithi120 ceph-mon[35250]: pgmap v400: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-15T23:25:42.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:41 smithi174 ceph-mon[35586]: pgmap v400: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-15T23:25:42.682 INFO:tasks.workunit.client.0.smithi089.stderr:Note: switching to 'ad35db06e8924912f01e32b35576297c0f1352b4'. 2024-09-15T23:25:42.682 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-09-15T23:25:42.683 INFO:tasks.workunit.client.0.smithi089.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T23:25:42.683 INFO:tasks.workunit.client.0.smithi089.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T23:25:42.683 INFO:tasks.workunit.client.0.smithi089.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T23:25:42.683 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-09-15T23:25:42.683 INFO:tasks.workunit.client.0.smithi089.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T23:25:42.683 INFO:tasks.workunit.client.0.smithi089.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T23:25:42.684 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-09-15T23:25:42.684 INFO:tasks.workunit.client.0.smithi089.stderr: git switch -c 2024-09-15T23:25:42.684 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-09-15T23:25:42.684 INFO:tasks.workunit.client.0.smithi089.stderr:Or undo this operation with: 2024-09-15T23:25:42.684 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-09-15T23:25:42.684 INFO:tasks.workunit.client.0.smithi089.stderr: git switch - 2024-09-15T23:25:42.684 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-09-15T23:25:42.684 INFO:tasks.workunit.client.0.smithi089.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T23:25:42.684 INFO:tasks.workunit.client.0.smithi089.stderr: 2024-09-15T23:25:42.684 INFO:tasks.workunit.client.0.smithi089.stderr:HEAD is now at ad35db06e89 Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid 2024-09-15T23:25:42.687 DEBUG:teuthology.orchestra.run.smithi089:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T23:25:42.757 INFO:tasks.workunit.client.0.smithi089.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-15T23:25:42.759 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T23:25:42.759 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-15T23:25:43.083 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '90d6d4a2', total seconds elapsed: 37.3 2024-09-15T23:25:43.152 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:43.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 -- 172.21.15.89:0/1189509211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc934100b30 msgr2=0x7fc934100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:43.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 --2- 172.21.15.89:0/1189509211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc934100b30 0x7fc934100f10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc930009900 tx=0x7fc93002a470 comp rx=0 tx=0).stop 2024-09-15T23:25:43.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 -- 172.21.15.89:0/1189509211 shutdown_connections 2024-09-15T23:25:43.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 --2- 172.21.15.89:0/1189509211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc934105f40 0x7fc9341063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 --2- 172.21.15.89:0/1189509211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc934101450 0x7fc934105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 --2- 172.21.15.89:0/1189509211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc934100b30 0x7fc934100f10 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 -- 172.21.15.89:0/1189509211 >> 172.21.15.89:0/1189509211 conn(0x7fc9340fc7a0 msgr2=0x7fc9340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 -- 172.21.15.89:0/1189509211 shutdown_connections 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 -- 172.21.15.89:0/1189509211 wait complete. 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 Processor -- start 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.805+0000 7fc93b2f1640 1 -- start start 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93b2f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc934100b30 0x7fc9341a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93b2f1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc934101450 0x7fc9341a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93b2f1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc934105f40 0x7fc93419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93b2f1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc9341185d0 con 0x7fc934100b30 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93b2f1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc934118450 con 0x7fc934105f40 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93b2f1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc934118750 con 0x7fc934101450 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93908d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc934100b30 0x7fc9341a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93908d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc934100b30 0x7fc9341a46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55620/0 (socket says 172.21.15.89:55620) 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93908d640 1 -- 172.21.15.89:0/258565888 learned_addr learned my addr 172.21.15.89:0/258565888 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93888c640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc934101450 0x7fc9341a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93988e640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc934105f40 0x7fc93419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93908d640 1 -- 172.21.15.89:0/258565888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc934101450 msgr2=0x7fc9341a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93908d640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc934101450 0x7fc9341a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93908d640 1 -- 172.21.15.89:0/258565888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc934105f40 msgr2=0x7fc93419e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93908d640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc934105f40 0x7fc93419e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93908d640 1 -- 172.21.15.89:0/258565888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc93419efb0 con 0x7fc934100b30 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93988e640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc934105f40 0x7fc93419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.806+0000 7fc93908d640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc934100b30 0x7fc9341a46c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc930005ab0 tx=0x7fc9300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.807+0000 7fc92a7fc640 1 -- 172.21.15.89:0/258565888 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93003f030 con 0x7fc934100b30 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.807+0000 7fc93b2f1640 1 -- 172.21.15.89:0/258565888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc93419f240 con 0x7fc934100b30 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.807+0000 7fc92a7fc640 1 -- 172.21.15.89:0/258565888 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc930003ce0 con 0x7fc934100b30 2024-09-15T23:25:43.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.807+0000 7fc92a7fc640 1 -- 172.21.15.89:0/258565888 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930035070 con 0x7fc934100b30 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.807+0000 7fc93b2f1640 1 -- 172.21.15.89:0/258565888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9341ab590 con 0x7fc934100b30 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.809+0000 7fc92a7fc640 1 -- 172.21.15.89:0/258565888 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc930042030 con 0x7fc934100b30 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.809+0000 7fc93d6ca740 1 Processor -- start 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.809+0000 7fc93d6ca740 1 -- start start 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.809+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9685090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.809+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f96855d0 0x5593f9685a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.809+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9692760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5593f95eb2f0 con 0x5593f9690370 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5593f9592cf0 con 0x5593f9686b40 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5593f95e72b0 con 0x5593f96855d0 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc92a7fc640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9200a3240 0x7fc9200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93888c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9692760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93888c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9692760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55628/0 (socket says 172.21.15.89:55628) 2024-09-15T23:25:43.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93988e640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9200a3240 0x7fc9200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93888c640 1 -- 172.21.15.89:0/3559331500 learned_addr learned my addr 172.21.15.89:0/3559331500 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93988e640 1 --2- 172.21.15.89:0/3559331500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9685090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc92a7fc640 1 -- 172.21.15.89:0/258565888 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc9300f05f0 con 0x7fc934100b30 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93908d640 1 --2- 172.21.15.89:0/3559331500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f96855d0 0x5593f9685a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93888c640 1 -- 172.21.15.89:0/3559331500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f96855d0 msgr2=0x5593f9685a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93888c640 1 --2- 172.21.15.89:0/3559331500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f96855d0 0x5593f9685a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93888c640 1 -- 172.21.15.89:0/3559331500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 msgr2=0x5593f9685090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93888c640 1 --2- 172.21.15.89:0/3559331500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9685090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93888c640 1 -- 172.21.15.89:0/3559331500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593f95773a0 con 0x5593f9690370 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93988e640 1 --2- 172.21.15.89:0/3559331500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9685090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93888c640 1 --2- 172.21.15.89:0/3559331500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9692760 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc92400ac50 tx=0x7fc92400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207199868266d8f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc93988e640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9200a3240 0x7fc9200a5700 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00e8f0 tx=0x7fc92c009380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:43.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.810+0000 7fc91f7fe640 1 -- 172.21.15.89:0/3559331500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92401d070 con 0x5593f9690370 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc91f7fe640 1 -- 172.21.15.89:0/3559331500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc924018b20 con 0x5593f9690370 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc91f7fe640 1 -- 172.21.15.89:0/3559331500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc924010bd0 con 0x5593f9690370 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3559331500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 msgr2=0x5593f9692760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/3559331500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9692760 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fc92400ac50 tx=0x7fc92400f790 comp rx=0 tx=0).stop 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3559331500 shutdown_connections 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/3559331500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9692760 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/3559331500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f96855d0 0x5593f9685a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/3559331500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9685090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3559331500 >> 172.21.15.89:0/3559331500 conn(0x5593f964e310 msgr2=0x5593f9681c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3559331500 shutdown_connections 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.811+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3559331500 wait complete. 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93d6ca740 1 -- start start 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93d6ca740 1 Processor -- start 2024-09-15T23:25:43.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f96855d0 0x5593f968fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9686b40 0x5593f968cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 0x5593f968d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5593f9693070 con 0x5593f9686b40 2024-09-15T23:25:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5593f95a6060 con 0x5593f96855d0 2024-09-15T23:25:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5593f96931f0 con 0x5593f9690370 2024-09-15T23:25:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93888c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9686b40 0x5593f968cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93d6ca740 10 client.0 _ll_register_callbacks cb 0x5593f92f98c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93888c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9686b40 0x5593f968cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55638/0 (socket says 172.21.15.89:55638) 2024-09-15T23:25:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93888c640 1 -- 172.21.15.89:0/1742872676 learned_addr learned my addr 172.21.15.89:0/1742872676 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:43.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.812+0000 7fc93908d640 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f96855d0 0x5593f968fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc93988e640 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 0x5593f968d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc93888c640 1 -- 172.21.15.89:0/1742872676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 msgr2=0x5593f968d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc93888c640 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 0x5593f968d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc93888c640 1 -- 172.21.15.89:0/1742872676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f96855d0 msgr2=0x5593f968fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc93888c640 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f96855d0 0x5593f968fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc93888c640 1 -- 172.21.15.89:0/1742872676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593f935c3c0 con 0x5593f9686b40 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc93888c640 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9686b40 0x5593f968cfe0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc924010770 tx=0x7fc924010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c4eb34834a317c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc91cff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc91cff9640 1 -- 172.21.15.89:0/1742872676 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92401d050 con 0x5593f9686b40 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc91cff9640 1 -- 172.21.15.89:0/1742872676 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc924018640 con 0x5593f9686b40 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc93d6ca740 1 -- 172.21.15.89:0/1742872676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc924018b20 con 0x5593f9686b40 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc91cff9640 1 -- 172.21.15.89:0/1742872676 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc924032050 con 0x5593f9686b40 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc93d6ca740 10 client.15978 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:43.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc93d6ca740 1 -- 172.21.15.89:0/1742872676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5593f969ead0 con 0x5593f9686b40 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.813+0000 7fc91cff9640 1 -- 172.21.15.89:0/1742872676 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc924018b20 con 0x5593f9686b40 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.814+0000 7fc91cff9640 1 -- 172.21.15.89:0/1742872676 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc9240046c0 con 0x5593f9686b40 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.814+0000 7fc93d6ca740 10 client.15978 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.814+0000 7fc93d6ca740 10 client.15978 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.814+0000 7fc93d6ca740 20 client.15978 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.814+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5593f969fed0 0x5593f96a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.814+0000 7fc93d6ca740 4 client.15978 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.814+0000 7fc93d6ca740 1 -- 172.21.15.89:0/1742872676 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593f95f8860 con 0x5593f969fed0 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.814+0000 7fc93908d640 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5593f969fed0 0x5593f96a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc91cff9640 10 client.15978 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc93908d640 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5593f969fed0 0x5593f96a22c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6d0a3ccdc21473fe server_cookie=2b43c3d82eced41d in_seq=0 out_seq=0 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc91cff9640 1 -- 172.21.15.89:0/1742872676 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5593f95f8860 con 0x5593f969fed0 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc91cff9640 10 client.15978 handle_command_reply: tid=0 2024-09-15T23:25:43.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc93d6ca740 1 client.15978 shutdown 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc93d6ca740 1 -- 172.21.15.89:0/1742872676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9686b40 msgr2=0x5593f968cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9686b40 0x5593f968cfe0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc924010770 tx=0x7fc924010c00 comp rx=0 tx=0).stop 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc93d6ca740 1 -- 172.21.15.89:0/1742872676 shutdown_connections 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5593f969fed0 0x5593f96a22c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 0x5593f968d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9686b40 0x5593f968cfe0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/1742872676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f96855d0 0x5593f968fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc91cff9640 0 client.15978 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.815+0000 7fc93d6ca740 1 -- 172.21.15.89:0/1742872676 >> 172.21.15.89:0/1742872676 conn(0x5593f964e310 msgr2=0x5593f9687960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.816+0000 7fc93d6ca740 1 -- 172.21.15.89:0/1742872676 shutdown_connections 2024-09-15T23:25:43.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.816+0000 7fc93d6ca740 1 -- 172.21.15.89:0/1742872676 wait complete. 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.816+0000 7fc93d6ca740 20 client.15978 trim_cache size 0 max 16384 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.828+0000 7fc93d6ca740 1 Processor -- start 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.828+0000 7fc93d6ca740 1 -- start start 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f96855d0 0x5593f969c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9691760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9691ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc9341b5370 con 0x5593f9690370 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc9341b6eb0 con 0x5593f9686b40 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc9341b5a40 con 0x5593f96855d0 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93988e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9691ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93988e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9691ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55646/0 (socket says 172.21.15.89:55646) 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93988e640 1 -- 172.21.15.89:0/773669264 learned_addr learned my addr 172.21.15.89:0/773669264 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93908d640 1 --2- 172.21.15.89:0/773669264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f96855d0 0x5593f969c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93988e640 1 -- 172.21.15.89:0/773669264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f96855d0 msgr2=0x5593f969c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93988e640 1 --2- 172.21.15.89:0/773669264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f96855d0 0x5593f969c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93988e640 1 -- 172.21.15.89:0/773669264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 msgr2=0x5593f9691760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93988e640 1 --2- 172.21.15.89:0/773669264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9691760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93988e640 1 -- 172.21.15.89:0/773669264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593f9686400 con 0x5593f9690370 2024-09-15T23:25:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc93988e640 1 --2- 172.21.15.89:0/773669264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9691ca0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00e650 tx=0x7fc92c008220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59a89afb323f669 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.829+0000 7fc91effd640 1 -- 172.21.15.89:0/773669264 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c00e030 con 0x5593f9690370 2024-09-15T23:25:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc91effd640 1 -- 172.21.15.89:0/773669264 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc92c00e1d0 con 0x5593f9690370 2024-09-15T23:25:43.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc91effd640 1 -- 172.21.15.89:0/773669264 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c01ab90 con 0x5593f9690370 2024-09-15T23:25:43.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc93d6ca740 1 -- 172.21.15.89:0/773669264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 msgr2=0x5593f9691ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:43.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/773669264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9691ca0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00e650 tx=0x7fc92c008220 comp rx=0 tx=0).stop 2024-09-15T23:25:43.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc93d6ca740 1 -- 172.21.15.89:0/773669264 shutdown_connections 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/773669264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f9690370 0x5593f9691ca0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/773669264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9691760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/773669264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f96855d0 0x5593f969c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc93d6ca740 1 -- 172.21.15.89:0/773669264 >> 172.21.15.89:0/773669264 conn(0x5593f964e310 msgr2=0x5593f968b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc93d6ca740 1 -- 172.21.15.89:0/773669264 shutdown_connections 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.830+0000 7fc93d6ca740 1 -- 172.21.15.89:0/773669264 wait complete. 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93d6ca740 1 -- start start 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93d6ca740 1 Processor -- start 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f96855d0 0x5593f9698d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9691840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93d6ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 0x5593f9691fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc9341b67e0 con 0x5593f96855d0 2024-09-15T23:25:43.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc9341b6110 con 0x5593f9686b40 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93d6ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc9341b3350 con 0x5593f9690370 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93d6ca740 10 client.0 _ll_register_callbacks cb 0x7fc92401b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93908d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f96855d0 0x5593f9698d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93908d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f96855d0 0x5593f9698d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55662/0 (socket says 172.21.15.89:55662) 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93908d640 1 -- 172.21.15.89:0/3558554552 learned_addr learned my addr 172.21.15.89:0/3558554552 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.831+0000 7fc93888c640 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9691840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93988e640 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 0x5593f9691fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93908d640 1 -- 172.21.15.89:0/3558554552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 msgr2=0x5593f9691fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93908d640 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 0x5593f9691fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93908d640 1 -- 172.21.15.89:0/3558554552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 msgr2=0x5593f9691840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93908d640 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9691840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93908d640 1 -- 172.21.15.89:0/3558554552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593f9696880 con 0x5593f96855d0 2024-09-15T23:25:43.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93988e640 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 0x5593f9691fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93908d640 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f96855d0 0x5593f9698d60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc930005940 tx=0x7fc930003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c88b6a4d4fcf245 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc91cff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc91cff9640 1 -- 172.21.15.89:0/3558554552 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930049040 con 0x5593f96855d0 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc91cff9640 1 -- 172.21.15.89:0/3558554552 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc93003bd70 con 0x5593f96855d0 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3558554552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc92c00e1d0 con 0x5593f96855d0 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc91cff9640 1 -- 172.21.15.89:0/3558554552 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc9300bc820 con 0x5593f96855d0 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93d6ca740 10 client.15984 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3558554552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5593f970abc0 con 0x5593f96855d0 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.832+0000 7fc91cff9640 1 -- 172.21.15.89:0/3558554552 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc92c00e1d0 con 0x5593f96855d0 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.833+0000 7fc91cff9640 1 -- 172.21.15.89:0/3558554552 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc924019ce0 con 0x5593f96855d0 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.833+0000 7fc93d6ca740 10 client.15984 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.833+0000 7fc93d6ca740 10 client.15984 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.833+0000 7fc93d6ca740 20 client.15984 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:43.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.833+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5593f970b240 0x5593f970d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.833+0000 7fc93d6ca740 4 client.15984 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.833+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3558554552 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5593f95f2d70 con 0x5593f970b240 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.833+0000 7fc93888c640 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5593f970b240 0x5593f970d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.833+0000 7fc91cff9640 10 client.15984 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.833+0000 7fc93888c640 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5593f970b240 0x5593f970d630 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=832fcc7ae1761d04 server_cookie=b4d6b71d25c78c11 in_seq=0 out_seq=0 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc91cff9640 1 -- 172.21.15.89:0/3558554552 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5593f95f2d70 con 0x5593f970b240 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc91cff9640 10 client.15984 handle_command_reply: tid=0 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 client.15984 shutdown 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3558554552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f96855d0 msgr2=0x5593f9698d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f96855d0 0x5593f9698d60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc930005940 tx=0x7fc930003f50 comp rx=0 tx=0).stop 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3558554552 shutdown_connections 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5593f970b240 0x5593f970d630 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5593f9690370 0x5593f9691fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5593f9686b40 0x5593f9691840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 --2- 172.21.15.89:0/3558554552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5593f96855d0 0x5593f9698d60 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc91cff9640 0 client.15984 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:43.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3558554552 >> 172.21.15.89:0/3558554552 conn(0x5593f964e310 msgr2=0x5593f968f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:43.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3558554552 shutdown_connections 2024-09-15T23:25:43.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 1 -- 172.21.15.89:0/3558554552 wait complete. 2024-09-15T23:25:43.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93d6ca740 20 client.15984 trim_cache size 0 max 16384 2024-09-15T23:25:43.850 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:43.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93b2f1640 1 -- 172.21.15.89:0/258565888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9200a3240 msgr2=0x7fc9200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:43.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93b2f1640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9200a3240 0x7fc9200a5700 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00e8f0 tx=0x7fc92c009380 comp rx=0 tx=0).stop 2024-09-15T23:25:43.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93b2f1640 1 -- 172.21.15.89:0/258565888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc934100b30 msgr2=0x7fc9341a46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:43.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.834+0000 7fc93b2f1640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc934100b30 0x7fc9341a46c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc930005ab0 tx=0x7fc9300042f0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.835+0000 7fc93b2f1640 1 -- 172.21.15.89:0/258565888 shutdown_connections 2024-09-15T23:25:43.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.835+0000 7fc93b2f1640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9200a3240 0x7fc9200a5700 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.835+0000 7fc93b2f1640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc934105f40 0x7fc93419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.835+0000 7fc93b2f1640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc934101450 0x7fc9341a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.835+0000 7fc93b2f1640 1 --2- 172.21.15.89:0/258565888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc934100b30 0x7fc9341a46c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:43.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.835+0000 7fc93b2f1640 1 -- 172.21.15.89:0/258565888 >> 172.21.15.89:0/258565888 conn(0x7fc9340fc7a0 msgr2=0x7fc9341038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:43.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.835+0000 7fc93b2f1640 1 -- 172.21.15.89:0/258565888 shutdown_connections 2024-09-15T23:25:43.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:43.835+0000 7fc93b2f1640 1 -- 172.21.15.89:0/258565888 wait complete. 2024-09-15T23:25:44.353 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:44 smithi089 ceph-mon[30933]: pgmap v401: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 597 B/s wr, 13 op/s 2024-09-15T23:25:44.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:44 smithi120 ceph-mon[35250]: pgmap v401: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 597 B/s wr, 13 op/s 2024-09-15T23:25:44.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:44 smithi174 ceph-mon[35586]: pgmap v401: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 597 B/s wr, 13 op/s 2024-09-15T23:25:44.729 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:25:44.732 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-15T23:25:44.827 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-15T23:25:44.867 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T23:25:44.868 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T23:25:44.868 INFO:tasks.workunit.client.0.smithi089.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-15T23:25:44.907 INFO:tasks.workunit.client.0.smithi089.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T23:25:44.914 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:25:44.914 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T23:25:44.930 INFO:tasks.workunit:Running workunits matching fs/test_o_trunc.sh on client.0... 2024-09-15T23:25:44.932 INFO:tasks.workunit:Running workunit fs/test_o_trunc.sh... 2024-09-15T23:25:44.932 DEBUG:teuthology.orchestra.run.smithi089:workunit test fs/test_o_trunc.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-09-15T23:25:44.986 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:44.996 INFO:tasks.workunit.client.0.smithi089.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc.sh 2024-09-15T23:25:44.996 INFO:tasks.workunit.client.0.smithi089.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs 2024-09-15T23:25:44.996 INFO:tasks.workunit.client.0.smithi089.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_o_trunc trunc.foo 600 2024-09-15T23:25:45.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.482+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/2595808145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a2810c170 msgr2=0x7f2a2810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:45.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.482+0000 7f2a2c9eb640 1 --2- 172.21.15.89:0/2595808145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a2810c170 0x7f2a2810e610 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009ca0 tx=0x7f2a2002a950 comp rx=0 tx=0).stop 2024-09-15T23:25:45.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.482+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/2595808145 shutdown_connections 2024-09-15T23:25:45.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.482+0000 7f2a2c9eb640 1 --2- 172.21.15.89:0/2595808145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a2810c170 0x7f2a2810e610 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.482+0000 7f2a2c9eb640 1 --2- 172.21.15.89:0/2595808145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a28103660 0x7f2a2810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.503 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.482+0000 7f2a2c9eb640 1 --2- 172.21.15.89:0/2595808145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a28102d40 0x7f2a28103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.482+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/2595808145 >> 172.21.15.89:0/2595808145 conn(0x7f2a280fc7a0 msgr2=0x7f2a280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.482+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/2595808145 shutdown_connections 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.482+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/2595808145 wait complete. 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a2c9eb640 1 Processor -- start 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a2c9eb640 1 -- start start 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a2c9eb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a28102d40 0x7f2a281983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a2c9eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a28103660 0x7f2a281988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a2c9eb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2810c170 0x7f2a2819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a2c9eb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a28111f30 con 0x7f2a28103660 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a2c9eb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a28111db0 con 0x7f2a28102d40 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a2c9eb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a281120b0 con 0x7f2a2810c170 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a25d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a28103660 0x7f2a281988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a25d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a28103660 0x7f2a281988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54940/0 (socket says 172.21.15.89:54940) 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.483+0000 7f2a25d9b640 1 -- 172.21.15.89:0/532588588 learned_addr learned my addr 172.21.15.89:0/532588588 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2810c170 0x7f2a2819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a2659c640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a28102d40 0x7f2a281983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a25d9b640 1 -- 172.21.15.89:0/532588588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2810c170 msgr2=0x7f2a2819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2810c170 0x7f2a2819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a25d9b640 1 -- 172.21.15.89:0/532588588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a28102d40 msgr2=0x7f2a281983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a28102d40 0x7f2a281983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a25d9b640 1 -- 172.21.15.89:0/532588588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2819d360 con 0x7f2a28103660 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2810c170 0x7f2a2819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a28103660 0x7f2a281988f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00c940 tx=0x7f2a1c00ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a177fe640 1 -- 172.21.15.89:0/532588588 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c004220 con 0x7f2a28103660 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/532588588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2819d650 con 0x7f2a28103660 2024-09-15T23:25:45.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a177fe640 1 -- 172.21.15.89:0/532588588 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a1c0043c0 con 0x7f2a28103660 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a177fe640 1 -- 172.21.15.89:0/532588588 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c004a00 con 0x7f2a28103660 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.484+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/532588588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a281a4f30 con 0x7f2a28103660 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.486+0000 7f2a177fe640 1 -- 172.21.15.89:0/532588588 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2a1c0079b0 con 0x7f2a28103660 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2edc4740 1 Processor -- start 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2edc4740 1 -- start start 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 0x55f2f8a83090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a83a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a8e370 0x55f2f8a90760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f2f89e92f0 con 0x55f2f8a835d0 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f2f8990cf0 con 0x55f2f8a84b40 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f2f89e52b0 con 0x55f2f8a8e370 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a177fe640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a100a3240 0x7f2a100a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a83a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a83a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54956/0 (socket says 172.21.15.89:54956) 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a2659c640 1 -- 172.21.15.89:0/2717057097 learned_addr learned my addr 172.21.15.89:0/2717057097 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.487+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/2717057097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 0x55f2f8a83090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a100a3240 0x7f2a100a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a177fe640 1 -- 172.21.15.89:0/532588588 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2a1c0cfd00 con 0x7f2a28103660 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/2717057097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a8e370 0x55f2f8a90760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a2659c640 1 -- 172.21.15.89:0/2717057097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a8e370 msgr2=0x55f2f8a90760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a2659c640 1 --2- 172.21.15.89:0/2717057097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a8e370 0x55f2f8a90760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a2659c640 1 -- 172.21.15.89:0/2717057097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 msgr2=0x55f2f8a83090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a2659c640 1 --2- 172.21.15.89:0/2717057097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 0x55f2f8a83090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a2659c640 1 -- 172.21.15.89:0/2717057097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2f89753a0 con 0x55f2f8a835d0 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/2717057097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 0x55f2f8a83090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:45.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a2659c640 1 --2- 172.21.15.89:0/2717057097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a83a10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800baa0 tx=0x7f2a1800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53251542029a46f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a100a3240 0x7f2a100a5700 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009a60 tx=0x7f2a200095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a14ff9640 1 -- 172.21.15.89:0/2717057097 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18017420 con 0x55f2f8a835d0 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a14ff9640 1 -- 172.21.15.89:0/2717057097 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a18017aa0 con 0x55f2f8a835d0 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.488+0000 7f2a14ff9640 1 -- 172.21.15.89:0/2717057097 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1800fb40 con 0x55f2f8a835d0 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2717057097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 msgr2=0x55f2f8a83a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2717057097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a83a10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800baa0 tx=0x7f2a1800bf60 comp rx=0 tx=0).stop 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2717057097 shutdown_connections 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2717057097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a8e370 0x55f2f8a90760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2717057097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a83a10 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2717057097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 0x55f2f8a83090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2717057097 >> 172.21.15.89:0/2717057097 conn(0x55f2f8a4c310 msgr2=0x55f2f8a7fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:45.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2717057097 shutdown_connections 2024-09-15T23:25:45.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2717057097 wait complete. 2024-09-15T23:25:45.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 -- start start 2024-09-15T23:25:45.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.489+0000 7f2a2edc4740 1 Processor -- start 2024-09-15T23:25:45.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a8dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 0x55f2f8a8af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 0x55f2f8a8b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f2f8a91070 con 0x55f2f8a835d0 2024-09-15T23:25:45.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f2f89a4060 con 0x55f2f8a8e370 2024-09-15T23:25:45.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f2f8a911f0 con 0x55f2f8a84b40 2024-09-15T23:25:45.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2edc4740 10 client.0 _ll_register_callbacks cb 0x55f2f86f78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:45.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a8dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a8dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54964/0 (socket says 172.21.15.89:54964) 2024-09-15T23:25:45.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2659c640 1 -- 172.21.15.89:0/1336834641 learned_addr learned my addr 172.21.15.89:0/1336834641 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:45.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 0x55f2f8a8af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 0x55f2f8a8b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2659c640 1 -- 172.21.15.89:0/1336834641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 msgr2=0x55f2f8a8af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:45.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2659c640 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 0x55f2f8a8af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2659c640 1 -- 172.21.15.89:0/1336834641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 msgr2=0x55f2f8a8b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2659c640 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 0x55f2f8a8b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2659c640 1 -- 172.21.15.89:0/1336834641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2f875a3c0 con 0x55f2f8a835d0 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2659c640 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a8dce0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800ba70 tx=0x7f2a1800b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc3798535b1d9eb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a0dffb640 1 -- 172.21.15.89:0/1336834641 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18025070 con 0x55f2f8a835d0 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a2edc4740 1 -- 172.21.15.89:0/1336834641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a18017aa0 con 0x55f2f8a835d0 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.490+0000 7f2a0dffb640 1 -- 172.21.15.89:0/1336834641 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2a18017620 con 0x55f2f8a835d0 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a0dffb640 1 -- 172.21.15.89:0/1336834641 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a18031050 con 0x55f2f8a835d0 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a2edc4740 10 client.15996 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a2edc4740 1 -- 172.21.15.89:0/1336834641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f2f8a9ca80 con 0x55f2f8a835d0 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a0dffb640 1 -- 172.21.15.89:0/1336834641 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a18017aa0 con 0x55f2f8a835d0 2024-09-15T23:25:45.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a0dffb640 1 -- 172.21.15.89:0/1336834641 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2a18005450 con 0x55f2f8a835d0 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a2edc4740 10 client.15996 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a2edc4740 10 client.15996 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a2edc4740 20 client.15996 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f2f8a9de30 0x55f2f8aa0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a2edc4740 4 client.15996 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a2edc4740 1 -- 172.21.15.89:0/1336834641 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2f89f6860 con 0x55f2f8a9de30 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.491+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f2f8a9de30 0x55f2f8aa0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.492+0000 7f2a0dffb640 10 client.15996 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.492+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f2f8a9de30 0x55f2f8aa0220 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6c1922f41c28ada8 server_cookie=663ef0bac362b688 in_seq=0 out_seq=0 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.492+0000 7f2a0dffb640 1 -- 172.21.15.89:0/1336834641 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f2f89f6860 con 0x55f2f8a9de30 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.492+0000 7f2a0dffb640 10 client.15996 handle_command_reply: tid=0 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.492+0000 7f2a2edc4740 1 client.15996 shutdown 2024-09-15T23:25:45.512 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 1 -- 172.21.15.89:0/1336834641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 msgr2=0x55f2f8a8dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a8dce0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800ba70 tx=0x7f2a1800b600 comp rx=0 tx=0).stop 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 1 -- 172.21.15.89:0/1336834641 shutdown_connections 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f2f8a9de30 0x55f2f8aa0220 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 0x55f2f8a8b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 0x55f2f8a8af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a0dffb640 0 client.15996 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/1336834641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a8dce0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 1 -- 172.21.15.89:0/1336834641 >> 172.21.15.89:0/1336834641 conn(0x55f2f8a4c310 msgr2=0x55f2f8a85960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 1 -- 172.21.15.89:0/1336834641 shutdown_connections 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 1 -- 172.21.15.89:0/1336834641 wait complete. 2024-09-15T23:25:45.513 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.493+0000 7f2a2edc4740 20 client.15996 trim_cache size 0 max 16384 2024-09-15T23:25:45.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a2edc4740 1 Processor -- start 2024-09-15T23:25:45.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a2edc4740 1 -- start start 2024-09-15T23:25:45.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a835d0 0x55f2f8a9a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 0x55f2f8a7fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a8e370 0x55f2f8a8a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a281aee90 con 0x55f2f8a8e370 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a281b09d0 con 0x55f2f8a84b40 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a281af560 con 0x55f2f8a835d0 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a26d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a8e370 0x55f2f8a8a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a26d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a8e370 0x55f2f8a8a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54980/0 (socket says 172.21.15.89:54980) 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a26d9d640 1 -- 172.21.15.89:0/2363637849 learned_addr learned my addr 172.21.15.89:0/2363637849 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/2363637849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 0x55f2f8a7fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a2659c640 1 --2- 172.21.15.89:0/2363637849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a835d0 0x55f2f8a9a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a26d9d640 1 -- 172.21.15.89:0/2363637849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a835d0 msgr2=0x55f2f8a9a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/2363637849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a835d0 0x55f2f8a9a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a26d9d640 1 -- 172.21.15.89:0/2363637849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 msgr2=0x55f2f8a7fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:45.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/2363637849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 0x55f2f8a7fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a26d9d640 1 -- 172.21.15.89:0/2363637849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2f8a832d0 con 0x55f2f8a8e370 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/2363637849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 0x55f2f8a7fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.506+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/2363637849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a8e370 0x55f2f8a8a870 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2a2002f4a0 tx=0x7f2a20042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd5d4139f57b7b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a0ffff640 1 -- 172.21.15.89:0/2363637849 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20050070 con 0x55f2f8a8e370 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a0ffff640 1 -- 172.21.15.89:0/2363637849 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a20004520 con 0x55f2f8a8e370 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a0ffff640 1 -- 172.21.15.89:0/2363637849 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20034040 con 0x55f2f8a8e370 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2363637849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a8e370 msgr2=0x55f2f8a8a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2363637849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a8e370 0x55f2f8a8a870 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2a2002f4a0 tx=0x7f2a20042000 comp rx=0 tx=0).stop 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2363637849 shutdown_connections 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2363637849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a8e370 0x55f2f8a8a870 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2363637849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a84b40 0x55f2f8a7fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2363637849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a835d0 0x55f2f8a9a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2363637849 >> 172.21.15.89:0/2363637849 conn(0x55f2f8a4c310 msgr2=0x55f2f8a7ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2363637849 shutdown_connections 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.507+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2363637849 wait complete. 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2edc4740 1 -- start start 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2edc4740 1 Processor -- start 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a96950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 0x55f2f8a87b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2edc4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 0x55f2f8a880b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a281b0300 con 0x55f2f8a835d0 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a281afc30 con 0x55f2f8a8e370 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2edc4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a281ace70 con 0x55f2f8a84b40 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2edc4740 10 client.0 _ll_register_callbacks cb 0x7f2a20009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a96950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a96950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54986/0 (socket says 172.21.15.89:54986) 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2659c640 1 -- 172.21.15.89:0/2007488449 learned_addr learned my addr 172.21.15.89:0/2007488449 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 0x55f2f8a87b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 0x55f2f8a880b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2659c640 1 -- 172.21.15.89:0/2007488449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 msgr2=0x55f2f8a87b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2659c640 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 0x55f2f8a87b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2659c640 1 -- 172.21.15.89:0/2007488449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 msgr2=0x55f2f8a880b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2659c640 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 0x55f2f8a880b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a2659c640 1 -- 172.21.15.89:0/2007488449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2f8a886a0 con 0x55f2f8a835d0 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 0x55f2f8a87b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.508+0000 7f2a26d9d640 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 0x55f2f8a880b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a2659c640 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a96950 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800b5d0 tx=0x7f2a1800b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465938477d9829ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:45.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a0dffb640 1 -- 172.21.15.89:0/2007488449 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18025040 con 0x55f2f8a835d0 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2007488449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a20004520 con 0x55f2f8a835d0 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a0dffb640 1 -- 172.21.15.89:0/2007488449 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2a18017830 con 0x55f2f8a835d0 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a0dffb640 1 -- 172.21.15.89:0/2007488449 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a18043050 con 0x55f2f8a835d0 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a2edc4740 10 client.16002 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2007488449 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f2f8b0ca00 con 0x55f2f8a835d0 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a0dffb640 1 -- 172.21.15.89:0/2007488449 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a20004520 con 0x55f2f8a835d0 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a0dffb640 1 -- 172.21.15.89:0/2007488449 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2a18004ce0 con 0x55f2f8a835d0 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a2edc4740 10 client.16002 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a2edc4740 10 client.16002 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a2edc4740 20 client.16002 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f2f8b0d0b0 0x55f2f8b0f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a2edc4740 4 client.16002 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.509+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2007488449 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f2f89f0d70 con 0x55f2f8b0d0b0 2024-09-15T23:25:45.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.510+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f2f8b0d0b0 0x55f2f8b0f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.510+0000 7f2a25d9b640 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f2f8b0d0b0 0x55f2f8b0f4a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=84afb910c68143ea server_cookie=b975f8cbfa6d1a65 in_seq=0 out_seq=0 2024-09-15T23:25:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.510+0000 7f2a0dffb640 10 client.16002 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.510+0000 7f2a0dffb640 1 -- 172.21.15.89:0/2007488449 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f2f89f0d70 con 0x55f2f8b0d0b0 2024-09-15T23:25:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.510+0000 7f2a0dffb640 10 client.16002 handle_command_reply: tid=0 2024-09-15T23:25:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.510+0000 7f2a2edc4740 1 client.16002 shutdown 2024-09-15T23:25:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.510+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2007488449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 msgr2=0x55f2f8a96950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.510+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a96950 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800b5d0 tx=0x7f2a1800b970 comp rx=0 tx=0).stop 2024-09-15T23:25:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.510+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2007488449 shutdown_connections 2024-09-15T23:25:45.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.510+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f2f8b0d0b0 0x55f2f8b0f4a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2f8a8e370 0x55f2f8a880b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2f8a84b40 0x55f2f8a87b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a0dffb640 0 client.16002 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2edc4740 1 --2- 172.21.15.89:0/2007488449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2f8a835d0 0x55f2f8a96950 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2007488449 >> 172.21.15.89:0/2007488449 conn(0x55f2f8a4c310 msgr2=0x55f2f8a85110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2007488449 shutdown_connections 2024-09-15T23:25:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2edc4740 1 -- 172.21.15.89:0/2007488449 wait complete. 2024-09-15T23:25:45.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2edc4740 20 client.16002 trim_cache size 0 max 16384 2024-09-15T23:25:45.525 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:45.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/532588588 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a100a3240 msgr2=0x7f2a100a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:45.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a100a3240 0x7f2a100a5700 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009a60 tx=0x7f2a200095c0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/532588588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a28103660 msgr2=0x7f2a281988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:45.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a28103660 0x7f2a281988f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00c940 tx=0x7f2a1c00ce00 comp rx=0 tx=0).stop 2024-09-15T23:25:45.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/532588588 shutdown_connections 2024-09-15T23:25:45.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a100a3240 0x7f2a100a5700 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a2810c170 0x7f2a2819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a28103660 0x7f2a281988f0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 --2- 172.21.15.89:0/532588588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a28102d40 0x7f2a281983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:45.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/532588588 >> 172.21.15.89:0/532588588 conn(0x7f2a280fc7a0 msgr2=0x7f2a280fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:45.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/532588588 shutdown_connections 2024-09-15T23:25:45.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:45.511+0000 7f2a2c9eb640 1 -- 172.21.15.89:0/532588588 wait complete. 2024-09-15T23:25:46.259 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:46 smithi089 ceph-mon[30933]: pgmap v402: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-15T23:25:46.288 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:25:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:46 smithi120 ceph-mon[35250]: pgmap v402: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-15T23:25:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:46 smithi174 ceph-mon[35586]: pgmap v402: 129 pgs: 129 active+clean; 681 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-09-15T23:25:46.575 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.888+0000 7fe4772a6640 1 -- 172.21.15.89:0/481983704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe470100a40 msgr2=0x7fe470100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.888+0000 7fe4772a6640 1 --2- 172.21.15.89:0/481983704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe470100a40 0x7fe470100e40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009900 tx=0x7fe46c02a410 comp rx=0 tx=0).stop 2024-09-15T23:25:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.888+0000 7fe4772a6640 1 -- 172.21.15.89:0/481983704 shutdown_connections 2024-09-15T23:25:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.888+0000 7fe4772a6640 1 --2- 172.21.15.89:0/481983704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4701025f0 0x7fe470108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.888+0000 7fe4772a6640 1 --2- 172.21.15.89:0/481983704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe470101c30 0x7fe4701020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.888+0000 7fe4772a6640 1 --2- 172.21.15.89:0/481983704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe470100a40 0x7fe470100e40 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.888+0000 7fe4772a6640 1 -- 172.21.15.89:0/481983704 >> 172.21.15.89:0/481983704 conn(0x7fe4700fc1e0 msgr2=0x7fe4700fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.888+0000 7fe4772a6640 1 -- 172.21.15.89:0/481983704 shutdown_connections 2024-09-15T23:25:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.888+0000 7fe4772a6640 1 -- 172.21.15.89:0/481983704 wait complete. 2024-09-15T23:25:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.888+0000 7fe4772a6640 1 Processor -- start 2024-09-15T23:25:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe4772a6640 1 -- start start 2024-09-15T23:25:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe4772a6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe470100a40 0x7fe470196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe4772a6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe470101c30 0x7fe470197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe4772a6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4701025f0 0x7fe47019e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe4772a6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe47010c6c0 con 0x7fe4701025f0 2024-09-15T23:25:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe4772a6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe47010c540 con 0x7fe470101c30 2024-09-15T23:25:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe4772a6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe47010c840 con 0x7fe470100a40 2024-09-15T23:25:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475843640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4701025f0 0x7fe47019e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475843640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4701025f0 0x7fe47019e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55016/0 (socket says 172.21.15.89:55016) 2024-09-15T23:25:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475843640 1 -- 172.21.15.89:0/2708323217 learned_addr learned my addr 172.21.15.89:0/2708323217 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe474841640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe470101c30 0x7fe470197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475042640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe470100a40 0x7fe470196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475843640 1 -- 172.21.15.89:0/2708323217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe470100a40 msgr2=0x7fe470196d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475843640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe470100a40 0x7fe470196d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475843640 1 -- 172.21.15.89:0/2708323217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe470101c30 msgr2=0x7fe470197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475843640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe470101c30 0x7fe470197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475843640 1 -- 172.21.15.89:0/2708323217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47019e910 con 0x7fe4701025f0 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe474841640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe470101c30 0x7fe470197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475042640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe470100a40 0x7fe470196d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe475843640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4701025f0 0x7fe47019e270 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe46800ebb0 tx=0x7fe468007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe4667fc640 1 -- 172.21.15.89:0/2708323217 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe468007ca0 con 0x7fe4701025f0 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe4667fc640 1 -- 172.21.15.89:0/2708323217 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe468007e40 con 0x7fe4701025f0 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.889+0000 7fe4667fc640 1 -- 172.21.15.89:0/2708323217 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4680058a0 con 0x7fe4701025f0 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.890+0000 7fe4772a6640 1 -- 172.21.15.89:0/2708323217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4700698a0 con 0x7fe4701025f0 2024-09-15T23:25:46.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.890+0000 7fe4772a6640 1 -- 172.21.15.89:0/2708323217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe470069e40 con 0x7fe4701025f0 2024-09-15T23:25:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.892+0000 7fe4667fc640 1 -- 172.21.15.89:0/2708323217 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe468004a40 con 0x7fe4701025f0 2024-09-15T23:25:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.892+0000 7fe47a039740 1 Processor -- start 2024-09-15T23:25:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.892+0000 7fe47a039740 1 -- start start 2024-09-15T23:25:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe47a039740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f38f9a0 0x56401f38fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe47a039740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f394da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe47a039740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f395370 0x56401f397830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe47a039740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56401f2f32f0 con 0x56401f3902e0 2024-09-15T23:25:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe47a039740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56401f29b270 con 0x56401f38f9a0 2024-09-15T23:25:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe47a039740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56401f2ef2b0 con 0x56401f395370 2024-09-15T23:25:46.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe475042640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f394da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe4667fc640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe45c0a3370 0x7fe45c0a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe475042640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f394da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55032/0 (socket says 172.21.15.89:55032) 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe475042640 1 -- 172.21.15.89:0/547681946 learned_addr learned my addr 172.21.15.89:0/547681946 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe474841640 1 --2- 172.21.15.89:0/547681946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f38f9a0 0x56401f38fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe474841640 1 --2- 172.21.15.89:0/547681946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f395370 0x56401f397830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe475843640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe45c0a3370 0x7fe45c0a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe4667fc640 1 -- 172.21.15.89:0/2708323217 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe4680d1330 con 0x7fe4701025f0 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe475042640 1 -- 172.21.15.89:0/547681946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f395370 msgr2=0x56401f397830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe475042640 1 --2- 172.21.15.89:0/547681946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f395370 0x56401f397830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe475042640 1 -- 172.21.15.89:0/547681946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f38f9a0 msgr2=0x56401f38fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe475042640 1 --2- 172.21.15.89:0/547681946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f38f9a0 0x56401f38fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.893+0000 7fe475042640 1 -- 172.21.15.89:0/547681946 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56401f29a340 con 0x56401f3902e0 2024-09-15T23:25:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.894+0000 7fe475843640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe45c0a3370 0x7fe45c0a5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe4680a5ca0 tx=0x7fe468099040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.894+0000 7fe475042640 1 --2- 172.21.15.89:0/547681946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f394da0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe46c005940 tx=0x7fe46c032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388e389f2016b9eb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.894+0000 7fe45b7fe640 1 -- 172.21.15.89:0/547681946 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c03f070 con 0x56401f3902e0 2024-09-15T23:25:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.894+0000 7fe45b7fe640 1 -- 172.21.15.89:0/547681946 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe46c0318b0 con 0x56401f3902e0 2024-09-15T23:25:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.894+0000 7fe45b7fe640 1 -- 172.21.15.89:0/547681946 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c035070 con 0x56401f3902e0 2024-09-15T23:25:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 -- 172.21.15.89:0/547681946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 msgr2=0x56401f394da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 --2- 172.21.15.89:0/547681946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f394da0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe46c005940 tx=0x7fe46c032d60 comp rx=0 tx=0).stop 2024-09-15T23:25:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 -- 172.21.15.89:0/547681946 shutdown_connections 2024-09-15T23:25:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 --2- 172.21.15.89:0/547681946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f395370 0x56401f397830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 --2- 172.21.15.89:0/547681946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f394da0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 --2- 172.21.15.89:0/547681946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f38f9a0 0x56401f38fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 -- 172.21.15.89:0/547681946 >> 172.21.15.89:0/547681946 conn(0x56401f38b010 msgr2=0x56401f38b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 -- 172.21.15.89:0/547681946 shutdown_connections 2024-09-15T23:25:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 -- 172.21.15.89:0/547681946 wait complete. 2024-09-15T23:25:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 -- start start 2024-09-15T23:25:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 Processor -- start 2024-09-15T23:25:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f38f9a0 0x56401f399b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f3902e0 0x56401f39a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f395370 0x56401f3a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56401f3982e0 con 0x56401f38f9a0 2024-09-15T23:25:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56401f27c290 con 0x56401f3902e0 2024-09-15T23:25:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.895+0000 7fe47a039740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56401f398460 con 0x56401f395370 2024-09-15T23:25:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe47a039740 10 client.0 _ll_register_callbacks cb 0x56401f2ad140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe474841640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f38f9a0 0x56401f399b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe474841640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f38f9a0 0x56401f399b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55046/0 (socket says 172.21.15.89:55046) 2024-09-15T23:25:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe474841640 1 -- 172.21.15.89:0/3627654541 learned_addr learned my addr 172.21.15.89:0/3627654541 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe475042640 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f3902e0 0x56401f39a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe474841640 1 -- 172.21.15.89:0/3627654541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f395370 msgr2=0x56401f3a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe474841640 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f395370 0x56401f3a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe474841640 1 -- 172.21.15.89:0/3627654541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f3902e0 msgr2=0x56401f39a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe474841640 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f3902e0 0x56401f39a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe474841640 1 -- 172.21.15.89:0/3627654541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56401f006210 con 0x56401f38f9a0 2024-09-15T23:25:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe474841640 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f38f9a0 0x56401f399b70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe46000b830 tx=0x7fe4600117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8baadc7ace7029d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe458ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe458ff9640 1 -- 172.21.15.89:0/3627654541 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe460013e60 con 0x56401f38f9a0 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe458ff9640 1 -- 172.21.15.89:0/3627654541 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe460004220 con 0x56401f38f9a0 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe47a039740 1 -- 172.21.15.89:0/3627654541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe46c0318b0 con 0x56401f38f9a0 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe458ff9640 1 -- 172.21.15.89:0/3627654541 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe460025050 con 0x56401f38f9a0 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe47a039740 10 client.16014 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe47a039740 1 -- 172.21.15.89:0/3627654541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56401f0d2620 con 0x56401f38f9a0 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.896+0000 7fe458ff9640 1 -- 172.21.15.89:0/3627654541 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe46c0318b0 con 0x56401f38f9a0 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.897+0000 7fe458ff9640 1 -- 172.21.15.89:0/3627654541 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe46000f040 con 0x56401f38f9a0 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.897+0000 7fe47a039740 10 client.16014 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.897+0000 7fe47a039740 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.897+0000 7fe47a039740 20 client.16014 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.897+0000 7fe47a039740 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56401f3a5820 0x56401f3a7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.897+0000 7fe47a039740 4 client.16014 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.897+0000 7fe47a039740 1 -- 172.21.15.89:0/3627654541 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56401f300a20 con 0x56401f3a5820 2024-09-15T23:25:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.897+0000 7fe475042640 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56401f3a5820 0x56401f3a7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.897+0000 7fe458ff9640 10 client.16014 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.897+0000 7fe475042640 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56401f3a5820 0x56401f3a7cb0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94112834948ae019 server_cookie=c9d0c18aa9653109 in_seq=0 out_seq=0 2024-09-15T23:25:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe458ff9640 1 -- 172.21.15.89:0/3627654541 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56401f300a20 con 0x56401f3a5820 2024-09-15T23:25:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe458ff9640 10 client.16014 handle_command_reply: tid=0 2024-09-15T23:25:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 client.16014 shutdown 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 -- 172.21.15.89:0/3627654541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f38f9a0 msgr2=0x56401f399b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f38f9a0 0x56401f399b70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe46000b830 tx=0x7fe4600117b0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 -- 172.21.15.89:0/3627654541 shutdown_connections 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56401f3a5820 0x56401f3a7cb0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f395370 0x56401f3a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f3902e0 0x56401f39a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe458ff9640 0 client.16014 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 --2- 172.21.15.89:0/3627654541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f38f9a0 0x56401f399b70 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 -- 172.21.15.89:0/3627654541 >> 172.21.15.89:0/3627654541 conn(0x56401f38b010 msgr2=0x56401f38c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 -- 172.21.15.89:0/3627654541 shutdown_connections 2024-09-15T23:25:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.898+0000 7fe47a039740 1 -- 172.21.15.89:0/3627654541 wait complete. 2024-09-15T23:25:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.899+0000 7fe47a039740 20 client.16014 trim_cache size 0 max 16384 2024-09-15T23:25:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe47a039740 1 Processor -- start 2024-09-15T23:25:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe47a039740 1 -- start start 2024-09-15T23:25:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe47a039740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f38f9a0 0x56401f39a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe47a039740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f3a1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe47a039740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 0x56401f38bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe47a039740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701a8690 con 0x56401f3902e0 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe47a039740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701aa1d0 con 0x56401f395370 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe47a039740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701a8d60 con 0x56401f38f9a0 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe475042640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f3a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe475042640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f3a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55052/0 (socket says 172.21.15.89:55052) 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe475042640 1 -- 172.21.15.89:0/423733980 learned_addr learned my addr 172.21.15.89:0/423733980 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe474841640 1 --2- 172.21.15.89:0/423733980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f38f9a0 0x56401f39a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe474841640 1 --2- 172.21.15.89:0/423733980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 0x56401f38bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe475042640 1 -- 172.21.15.89:0/423733980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f38f9a0 msgr2=0x56401f39a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe475042640 1 --2- 172.21.15.89:0/423733980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f38f9a0 0x56401f39a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe475042640 1 -- 172.21.15.89:0/423733980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 msgr2=0x56401f38bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe475042640 1 --2- 172.21.15.89:0/423733980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 0x56401f38bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe475042640 1 -- 172.21.15.89:0/423733980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56401f3984f0 con 0x56401f3902e0 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe475042640 1 --2- 172.21.15.89:0/423733980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f3a1010 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe46c02a930 tx=0x7fe46c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abac8dfc6c8bfdc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe45affd640 1 -- 172.21.15.89:0/423733980 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c053070 con 0x56401f3902e0 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.913+0000 7fe45affd640 1 -- 172.21.15.89:0/423733980 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe46c031d90 con 0x56401f3902e0 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe45affd640 1 -- 172.21.15.89:0/423733980 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c035040 con 0x56401f3902e0 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 -- 172.21.15.89:0/423733980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 msgr2=0x56401f3a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 --2- 172.21.15.89:0/423733980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f3a1010 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe46c02a930 tx=0x7fe46c031400 comp rx=0 tx=0).stop 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 -- 172.21.15.89:0/423733980 shutdown_connections 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 --2- 172.21.15.89:0/423733980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 0x56401f38bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 --2- 172.21.15.89:0/423733980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f3a1010 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 --2- 172.21.15.89:0/423733980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f38f9a0 0x56401f39a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 -- 172.21.15.89:0/423733980 >> 172.21.15.89:0/423733980 conn(0x56401f38b010 msgr2=0x56401f399920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:46.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 -- 172.21.15.89:0/423733980 shutdown_connections 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 -- 172.21.15.89:0/423733980 wait complete. 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 -- start start 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.914+0000 7fe47a039740 1 Processor -- start 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe47a039740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f38f9a0 0x56401f39e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe47a039740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f39e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe47a039740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 0x56401f39f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe47a039740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701a9b00 con 0x56401f3902e0 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe47a039740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701a9430 con 0x56401f395370 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe47a039740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701a6670 con 0x56401f38f9a0 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe47a039740 10 client.0 _ll_register_callbacks cb 0x7fe460005ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe475042640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f39e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe475042640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f39e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55054/0 (socket says 172.21.15.89:55054) 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe475042640 1 -- 172.21.15.89:0/4185986450 learned_addr learned my addr 172.21.15.89:0/4185986450 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe474841640 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 0x56401f39f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe474841640 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f38f9a0 0x56401f39e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe475042640 1 -- 172.21.15.89:0/4185986450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f38f9a0 msgr2=0x56401f39e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe475042640 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f38f9a0 0x56401f39e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe475042640 1 -- 172.21.15.89:0/4185986450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 msgr2=0x56401f39f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe475042640 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 0x56401f39f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe475042640 1 -- 172.21.15.89:0/4185986450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56401f39ff90 con 0x56401f3902e0 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe474841640 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 0x56401f39f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe475042640 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f39e9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe46c031610 tx=0x7fe46c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7554df2658f0cf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.915+0000 7fe458ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe458ff9640 1 -- 172.21.15.89:0/4185986450 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c035050 con 0x56401f3902e0 2024-09-15T23:25:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe458ff9640 1 -- 172.21.15.89:0/4185986450 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe46c0318b0 con 0x56401f3902e0 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe47a039740 1 -- 172.21.15.89:0/4185986450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe46c031d90 con 0x56401f3902e0 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe458ff9640 1 -- 172.21.15.89:0/4185986450 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe46c067050 con 0x56401f3902e0 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe47a039740 10 client.16020 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe47a039740 1 -- 172.21.15.89:0/4185986450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56401f3a12a0 con 0x56401f3902e0 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe458ff9640 1 -- 172.21.15.89:0/4185986450 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe46c031d90 con 0x56401f3902e0 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe458ff9640 1 -- 172.21.15.89:0/4185986450 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe46c04d070 con 0x56401f3902e0 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe47a039740 10 client.16020 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe47a039740 10 client.16020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe47a039740 20 client.16020 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe47a039740 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56401f3a5820 0x56401f4126a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe47a039740 4 client.16020 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:25:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe47a039740 1 -- 172.21.15.89:0/4185986450 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56401f2fad70 con 0x56401f3a5820 2024-09-15T23:25:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.916+0000 7fe474841640 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56401f3a5820 0x56401f4126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.917+0000 7fe458ff9640 10 client.16020 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.917+0000 7fe474841640 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56401f3a5820 0x56401f4126a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=245110ad7af84334 server_cookie=da1326c4d6637420 in_seq=0 out_seq=0 2024-09-15T23:25:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe458ff9640 1 -- 172.21.15.89:0/4185986450 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56401f2fad70 con 0x56401f3a5820 2024-09-15T23:25:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe458ff9640 10 client.16020 handle_command_reply: tid=0 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 client.16020 shutdown 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 -- 172.21.15.89:0/4185986450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 msgr2=0x56401f39e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f39e9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe46c031610 tx=0x7fe46c0035f0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 -- 172.21.15.89:0/4185986450 shutdown_connections 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56401f3a5820 0x56401f4126a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56401f395370 0x56401f39f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe458ff9640 0 client.16020 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56401f3902e0 0x56401f39e9c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 --2- 172.21.15.89:0/4185986450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56401f38f9a0 0x56401f39e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 -- 172.21.15.89:0/4185986450 >> 172.21.15.89:0/4185986450 conn(0x56401f38b010 msgr2=0x56401f3899a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 -- 172.21.15.89:0/4185986450 shutdown_connections 2024-09-15T23:25:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 1 -- 172.21.15.89:0/4185986450 wait complete. 2024-09-15T23:25:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.926+0000 7fe47a039740 20 client.16020 trim_cache size 0 max 16384 2024-09-15T23:25:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:25:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:25:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:25:46.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:25:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 -- 172.21.15.89:0/2708323217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe45c0a3370 msgr2=0x7fe45c0a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe45c0a3370 0x7fe45c0a5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe4680a5ca0 tx=0x7fe468099040 comp rx=0 tx=0).stop 2024-09-15T23:25:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 -- 172.21.15.89:0/2708323217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4701025f0 msgr2=0x7fe47019e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4701025f0 0x7fe47019e270 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe46800ebb0 tx=0x7fe468007590 comp rx=0 tx=0).stop 2024-09-15T23:25:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 -- 172.21.15.89:0/2708323217 shutdown_connections 2024-09-15T23:25:46.930 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe45c0a3370 0x7fe45c0a5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4701025f0 0x7fe47019e270 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe470101c30 0x7fe470197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 --2- 172.21.15.89:0/2708323217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe470100a40 0x7fe470196d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:46.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 -- 172.21.15.89:0/2708323217 >> 172.21.15.89:0/2708323217 conn(0x7fe4700fc1e0 msgr2=0x7fe470104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:46.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 -- 172.21.15.89:0/2708323217 shutdown_connections 2024-09-15T23:25:46.931 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:25:46.927+0000 7fe4772a6640 1 -- 172.21.15.89:0/2708323217 wait complete. 2024-09-15T23:25:47.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.071+0000 7f2916372640 1 -- 172.21.15.89:0/3737447810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2910101de0 msgr2=0x7f29101021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.071+0000 7f2916372640 1 --2- 172.21.15.89:0/3737447810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2910101de0 0x7f29101021c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2900009900 tx=0x7f290002a410 comp rx=0 tx=0).stop 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.071+0000 7f2916372640 1 -- 172.21.15.89:0/3737447810 shutdown_connections 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.071+0000 7f2916372640 1 --2- 172.21.15.89:0/3737447810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2910106f10 0x7f291010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.071+0000 7f2916372640 1 --2- 172.21.15.89:0/3737447810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2910102790 0x7f29101067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.071+0000 7f2916372640 1 --2- 172.21.15.89:0/3737447810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2910101de0 0x7f29101021c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.071+0000 7f2916372640 1 -- 172.21.15.89:0/3737447810 >> 172.21.15.89:0/3737447810 conn(0x7f29100fd8c0 msgr2=0x7f29100ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.071+0000 7f2916372640 1 -- 172.21.15.89:0/3737447810 shutdown_connections 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.071+0000 7f2916372640 1 -- 172.21.15.89:0/3737447810 wait complete. 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.072+0000 7f2916372640 1 Processor -- start 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.072+0000 7f2916372640 1 -- start start 2024-09-15T23:25:47.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.072+0000 7f2916372640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2910101de0 0x7f29101983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.072+0000 7f2916372640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2910102790 0x7f2910198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.072+0000 7f2916372640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2910106f10 0x7f291019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.072+0000 7f2916372640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2910111f70 con 0x7f2910102790 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.072+0000 7f2916372640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2910111df0 con 0x7f2910106f10 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.072+0000 7f2916372640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f29101120f0 con 0x7f2910101de0 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.072+0000 7f290f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2910102790 0x7f2910198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2910102790 0x7f2910198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55066/0 (socket says 172.21.15.89:55066) 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290f7fe640 1 -- 172.21.15.89:0/849019187 learned_addr learned my addr 172.21.15.89:0/849019187 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290ffff640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2910101de0 0x7f29101983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f291490f640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2910106f10 0x7f291019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290f7fe640 1 -- 172.21.15.89:0/849019187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2910101de0 msgr2=0x7f29101983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290f7fe640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2910101de0 0x7f29101983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290f7fe640 1 -- 172.21.15.89:0/849019187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2910106f10 msgr2=0x7f291019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290f7fe640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2910106f10 0x7f291019cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290f7fe640 1 -- 172.21.15.89:0/849019187 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291019d380 con 0x7f2910102790 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f291490f640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2910106f10 0x7f291019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290f7fe640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2910102790 0x7f2910198910 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f290400d770 tx=0x7f290400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290d7fa640 1 -- 172.21.15.89:0/849019187 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904017070 con 0x7f2910102790 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f2916372640 1 -- 172.21.15.89:0/849019187 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f291019d670 con 0x7f2910102790 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290d7fa640 1 -- 172.21.15.89:0/849019187 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2904005500 con 0x7f2910102790 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f290d7fa640 1 -- 172.21.15.89:0/849019187 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904004e00 con 0x7f2910102790 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.073+0000 7f2916372640 1 -- 172.21.15.89:0/849019187 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29101a4f50 con 0x7f2910102790 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.075+0000 7f290d7fa640 1 -- 172.21.15.89:0/849019187 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2904004fa0 con 0x7f2910102790 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291874b740 1 Processor -- start 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291874b740 1 -- start start 2024-09-15T23:25:47.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f290d7fa640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28f80a3240 0x7f28f80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291874b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55951639bb40 0x55951639a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291874b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639a5d0 0x55951639aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291874b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163a7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291874b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5595163002f0 con 0x5595163a5370 2024-09-15T23:25:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291874b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5595162a7cf0 con 0x55951639a5d0 2024-09-15T23:25:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291874b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5595162fc2b0 con 0x55951639bb40 2024-09-15T23:25:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291490f640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28f80a3240 0x7f28f80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f290d7fa640 1 -- 172.21.15.89:0/849019187 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f29040cf790 con 0x7f2910102790 2024-09-15T23:25:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291490f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291490f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163a7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55068/0 (socket says 172.21.15.89:55068) 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f291490f640 1 -- 172.21.15.89:0/1598278078 learned_addr learned my addr 172.21.15.89:0/1598278078 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f290ffff640 1 --2- 172.21.15.89:0/1598278078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55951639bb40 0x55951639a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.076+0000 7f290f7fe640 1 --2- 172.21.15.89:0/1598278078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639a5d0 0x55951639aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.077+0000 7f291490f640 1 -- 172.21.15.89:0/1598278078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55951639bb40 msgr2=0x55951639a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.077+0000 7f291490f640 1 --2- 172.21.15.89:0/1598278078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55951639bb40 0x55951639a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.077+0000 7f291490f640 1 -- 172.21.15.89:0/1598278078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639a5d0 msgr2=0x55951639aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.077+0000 7f291490f640 1 --2- 172.21.15.89:0/1598278078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639a5d0 0x55951639aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.077+0000 7f291490f640 1 -- 172.21.15.89:0/1598278078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55951628c3a0 con 0x5595163a5370 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.077+0000 7f291490f640 1 --2- 172.21.15.89:0/1598278078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163a7760 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f290800fc60 tx=0x7f2908019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197c862d16a82bcd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.077+0000 7f291490f640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28f80a3240 0x7f28f80a5700 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2908012ae0 tx=0x7f290800ad30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.077+0000 7f28f67fc640 1 -- 172.21.15.89:0/1598278078 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908022070 con 0x5595163a5370 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.077+0000 7f28f67fc640 1 -- 172.21.15.89:0/1598278078 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f290800dde0 con 0x5595163a5370 2024-09-15T23:25:47.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.077+0000 7f28f67fc640 1 -- 172.21.15.89:0/1598278078 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29080105d0 con 0x5595163a5370 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 -- 172.21.15.89:0/1598278078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 msgr2=0x5595163a7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 --2- 172.21.15.89:0/1598278078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163a7760 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f290800fc60 tx=0x7f2908019000 comp rx=0 tx=0).stop 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 -- 172.21.15.89:0/1598278078 shutdown_connections 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 --2- 172.21.15.89:0/1598278078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163a7760 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 --2- 172.21.15.89:0/1598278078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639a5d0 0x55951639aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 --2- 172.21.15.89:0/1598278078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55951639bb40 0x55951639a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 -- 172.21.15.89:0/1598278078 >> 172.21.15.89:0/1598278078 conn(0x559516363310 msgr2=0x559516396c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 -- 172.21.15.89:0/1598278078 shutdown_connections 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 -- 172.21.15.89:0/1598278078 wait complete. 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 -- start start 2024-09-15T23:25:47.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.078+0000 7f291874b740 1 Processor -- start 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f291874b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639a5d0 0x5595163a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f291874b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639bb40 0x5595163a2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f291874b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5595163a5370 0x5595163a2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f291874b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5595163a8010 con 0x55951639bb40 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f291874b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5595162bb060 con 0x55951639a5d0 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f291874b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5595163a8190 con 0x5595163a5370 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f291874b740 10 client.0 _ll_register_callbacks cb 0x55951600e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f290f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639bb40 0x5595163a2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f290f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639bb40 0x5595163a2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55082/0 (socket says 172.21.15.89:55082) 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f290f7fe640 1 -- 172.21.15.89:0/2718812543 learned_addr learned my addr 172.21.15.89:0/2718812543 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f291490f640 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5595163a5370 0x5595163a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f290f7fe640 1 -- 172.21.15.89:0/2718812543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5595163a5370 msgr2=0x5595163a2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f290f7fe640 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5595163a5370 0x5595163a2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f290f7fe640 1 -- 172.21.15.89:0/2718812543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639a5d0 msgr2=0x5595163a4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f290f7fe640 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639a5d0 0x5595163a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f290f7fe640 1 -- 172.21.15.89:0/2718812543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595160713c0 con 0x55951639bb40 2024-09-15T23:25:47.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f290f7fe640 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639bb40 0x5595163a2010 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f290409b950 tx=0x7f29040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bc406f9d671a75 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:47.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f28ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:47.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f28ef7fe640 1 -- 172.21.15.89:0/2718812543 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904020040 con 0x55951639bb40 2024-09-15T23:25:47.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f28ef7fe640 1 -- 172.21.15.89:0/2718812543 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2904099cc0 con 0x55951639bb40 2024-09-15T23:25:47.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f291874b740 1 -- 172.21.15.89:0/2718812543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f290800dde0 con 0x55951639bb40 2024-09-15T23:25:47.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f28ef7fe640 1 -- 172.21.15.89:0/2718812543 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2904066f70 con 0x55951639bb40 2024-09-15T23:25:47.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.079+0000 7f291874b740 10 client.16032 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:47.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.080+0000 7f291874b740 1 -- 172.21.15.89:0/2718812543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5595163b3b00 con 0x55951639bb40 2024-09-15T23:25:47.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.080+0000 7f28ef7fe640 1 -- 172.21.15.89:0/2718812543 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f290800dde0 con 0x55951639bb40 2024-09-15T23:25:47.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.080+0000 7f28ef7fe640 1 -- 172.21.15.89:0/2718812543 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2904098030 con 0x55951639bb40 2024-09-15T23:25:47.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.080+0000 7f291874b740 10 client.16032 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:47.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.080+0000 7f291874b740 10 client.16032 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:25:47.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.080+0000 7f291874b740 20 client.16032 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:47.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.080+0000 7f291874b740 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5595163b4f60 0x5595163b7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.080+0000 7f291874b740 4 client.16032 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.080+0000 7f291874b740 1 -- 172.21.15.89:0/2718812543 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55951630d860 con 0x5595163b4f60 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.080+0000 7f290ffff640 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5595163b4f60 0x5595163b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.081+0000 7f28ef7fe640 10 client.16032 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.081+0000 7f290ffff640 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5595163b4f60 0x5595163b7390 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=36f885949032561c server_cookie=c49d4f6ad51fce73 in_seq=0 out_seq=0 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.081+0000 7f28ef7fe640 1 -- 172.21.15.89:0/2718812543 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55951630d860 con 0x5595163b4f60 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.081+0000 7f28ef7fe640 10 client.16032 handle_command_reply: tid=0 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.081+0000 7f291874b740 1 client.16032 shutdown 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 1 -- 172.21.15.89:0/2718812543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639bb40 msgr2=0x5595163a2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639bb40 0x5595163a2010 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f290409b950 tx=0x7f29040046c0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 1 -- 172.21.15.89:0/2718812543 shutdown_connections 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5595163b4f60 0x5595163b7390 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5595163a5370 0x5595163a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639bb40 0x5595163a2010 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f28ef7fe640 0 client.16032 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:47.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 1 --2- 172.21.15.89:0/2718812543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639a5d0 0x5595163a4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 1 -- 172.21.15.89:0/2718812543 >> 172.21.15.89:0/2718812543 conn(0x559516363310 msgr2=0x55951639c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:47.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 1 -- 172.21.15.89:0/2718812543 shutdown_connections 2024-09-15T23:25:47.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 1 -- 172.21.15.89:0/2718812543 wait complete. 2024-09-15T23:25:47.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.082+0000 7f291874b740 20 client.16032 trim_cache size 0 max 16384 2024-09-15T23:25:47.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291874b740 1 Processor -- start 2024-09-15T23:25:47.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291874b740 1 -- start start 2024-09-15T23:25:47.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291874b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55951639a5d0 0x55951639d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291874b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 0x55951639d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291874b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163bcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291874b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f29101aede0 con 0x5595163a5370 2024-09-15T23:25:47.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291874b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f29101b0920 con 0x55951639bb40 2024-09-15T23:25:47.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291874b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f29101af4b0 con 0x55951639a5d0 2024-09-15T23:25:47.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291490f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163bcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291490f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163bcc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55096/0 (socket says 172.21.15.89:55096) 2024-09-15T23:25:47.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291490f640 1 -- 172.21.15.89:0/2067893868 learned_addr learned my addr 172.21.15.89:0/2067893868 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:47.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f290ffff640 1 --2- 172.21.15.89:0/2067893868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55951639a5d0 0x55951639d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291490f640 1 -- 172.21.15.89:0/2067893868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55951639a5d0 msgr2=0x55951639d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291490f640 1 --2- 172.21.15.89:0/2067893868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55951639a5d0 0x55951639d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291490f640 1 -- 172.21.15.89:0/2067893868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 msgr2=0x55951639d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f290f7fe640 1 --2- 172.21.15.89:0/2067893868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 0x55951639d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291490f640 1 --2- 172.21.15.89:0/2067893868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 0x55951639d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f291490f640 1 -- 172.21.15.89:0/2067893868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595163b7a10 con 0x5595163a5370 2024-09-15T23:25:47.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.095+0000 7f290f7fe640 1 --2- 172.21.15.89:0/2067893868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 0x55951639d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:47.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f291490f640 1 --2- 172.21.15.89:0/2067893868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163bcc60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f290800f730 tx=0x7f2908012830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1228ed0832d29b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:47.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f28f5ffb640 1 -- 172.21.15.89:0/2067893868 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2908022040 con 0x5595163a5370 2024-09-15T23:25:47.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f28f5ffb640 1 -- 172.21.15.89:0/2067893868 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2908034070 con 0x5595163a5370 2024-09-15T23:25:47.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f28f5ffb640 1 -- 172.21.15.89:0/2067893868 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290801eb80 con 0x5595163a5370 2024-09-15T23:25:47.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f291874b740 1 -- 172.21.15.89:0/2067893868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 msgr2=0x5595163bcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f291874b740 1 --2- 172.21.15.89:0/2067893868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163bcc60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f290800f730 tx=0x7f2908012830 comp rx=0 tx=0).stop 2024-09-15T23:25:47.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f291874b740 1 -- 172.21.15.89:0/2067893868 shutdown_connections 2024-09-15T23:25:47.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f291874b740 1 --2- 172.21.15.89:0/2067893868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5595163a5370 0x5595163bcc60 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f291874b740 1 --2- 172.21.15.89:0/2067893868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 0x55951639d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f291874b740 1 --2- 172.21.15.89:0/2067893868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55951639a5d0 0x55951639d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.096+0000 7f291874b740 1 -- 172.21.15.89:0/2067893868 >> 172.21.15.89:0/2067893868 conn(0x559516363310 msgr2=0x5595163a4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 1 -- 172.21.15.89:0/2067893868 shutdown_connections 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 1 -- 172.21.15.89:0/2067893868 wait complete. 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 1 -- start start 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 1 Processor -- start 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639a5d0 0x5595163a9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 0x5595163a9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5595163a5370 0x55951641ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f29101b0250 con 0x55951639a5d0 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f29101afb80 con 0x55951639bb40 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f29101acdc0 con 0x5595163a5370 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f290ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639a5d0 0x5595163a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f291874b740 10 client.0 _ll_register_callbacks cb 0x7f290400d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:47.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f290ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639a5d0 0x5595163a9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55112/0 (socket says 172.21.15.89:55112) 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.097+0000 7f290ffff640 1 -- 172.21.15.89:0/275860484 learned_addr learned my addr 172.21.15.89:0/275860484 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f290f7fe640 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 0x5595163a9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f290ffff640 1 -- 172.21.15.89:0/275860484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5595163a5370 msgr2=0x55951641ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f290ffff640 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5595163a5370 0x55951641ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f290ffff640 1 -- 172.21.15.89:0/275860484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 msgr2=0x5595163a9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f290ffff640 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 0x5595163a9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f290ffff640 1 -- 172.21.15.89:0/275860484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55951641f1f0 con 0x55951639a5d0 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f290f7fe640 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 0x5595163a9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f290ffff640 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639a5d0 0x5595163a9940 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2900033000 tx=0x7f2900031a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a453a27a4c541b03 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f28ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f28ef7fe640 1 -- 172.21.15.89:0/275860484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900035040 con 0x55951639a5d0 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f28ef7fe640 1 -- 172.21.15.89:0/275860484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2900004450 con 0x55951639a5d0 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f291874b740 1 -- 172.21.15.89:0/275860484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2908034070 con 0x55951639a5d0 2024-09-15T23:25:47.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f28ef7fe640 1 -- 172.21.15.89:0/275860484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2900053050 con 0x55951639a5d0 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f291874b740 10 client.16038 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f291874b740 1 -- 172.21.15.89:0/275860484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559516420480 con 0x55951639a5d0 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.098+0000 7f28ef7fe640 1 -- 172.21.15.89:0/275860484 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2908034070 con 0x55951639a5d0 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f28ef7fe640 1 -- 172.21.15.89:0/275860484 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f290003d690 con 0x55951639a5d0 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f291874b740 10 client.16038 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f291874b740 10 client.16038 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f291874b740 20 client.16038 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f291874b740 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559516420b50 0x559516422f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f291874b740 4 client.16038 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f291874b740 1 -- 172.21.15.89:0/275860484 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559516307d70 con 0x559516420b50 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f290f7fe640 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559516420b50 0x559516422f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f28ef7fe640 10 client.16038 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f290f7fe640 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559516420b50 0x559516422f80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ca84cd847fb1a42 server_cookie=c6c27b65795ff835 in_seq=0 out_seq=0 2024-09-15T23:25:47.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f28ef7fe640 1 -- 172.21.15.89:0/275860484 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559516307d70 con 0x559516420b50 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f28ef7fe640 10 client.16038 handle_command_reply: tid=0 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.099+0000 7f291874b740 1 client.16038 shutdown 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 1 -- 172.21.15.89:0/275860484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639a5d0 msgr2=0x5595163a9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639a5d0 0x5595163a9940 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2900033000 tx=0x7f2900031a00 comp rx=0 tx=0).stop 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 1 -- 172.21.15.89:0/275860484 shutdown_connections 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559516420b50 0x559516422f80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5595163a5370 0x55951641ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55951639bb40 0x5595163a9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 1 --2- 172.21.15.89:0/275860484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55951639a5d0 0x5595163a9940 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 1 -- 172.21.15.89:0/275860484 >> 172.21.15.89:0/275860484 conn(0x559516363310 msgr2=0x5595163954e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f28ef7fe640 0 client.16038 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 1 -- 172.21.15.89:0/275860484 shutdown_connections 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 1 -- 172.21.15.89:0/275860484 wait complete. 2024-09-15T23:25:47.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f291874b740 20 client.16038 trim_cache size 0 max 16384 2024-09-15T23:25:47.112 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:47.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f2916372640 1 -- 172.21.15.89:0/849019187 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28f80a3240 msgr2=0x7f28f80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f2916372640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28f80a3240 0x7f28f80a5700 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2908012ae0 tx=0x7f290800ad30 comp rx=0 tx=0).stop 2024-09-15T23:25:47.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f2916372640 1 -- 172.21.15.89:0/849019187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2910102790 msgr2=0x7f2910198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.100+0000 7f2916372640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2910102790 0x7f2910198910 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f290400d770 tx=0x7f290400dc30 comp rx=0 tx=0).stop 2024-09-15T23:25:47.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.101+0000 7f2916372640 1 -- 172.21.15.89:0/849019187 shutdown_connections 2024-09-15T23:25:47.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.101+0000 7f2916372640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28f80a3240 0x7f28f80a5700 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.101+0000 7f2916372640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2910106f10 0x7f291019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.101+0000 7f2916372640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2910102790 0x7f2910198910 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.101+0000 7f2916372640 1 --2- 172.21.15.89:0/849019187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2910101de0 0x7f29101983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.101+0000 7f2916372640 1 -- 172.21.15.89:0/849019187 >> 172.21.15.89:0/849019187 conn(0x7f29100fd8c0 msgr2=0x7f29100febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:47.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.101+0000 7f2916372640 1 -- 172.21.15.89:0/849019187 shutdown_connections 2024-09-15T23:25:47.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:47.101+0000 7f2916372640 1 -- 172.21.15.89:0/849019187 wait complete. 2024-09-15T23:25:47.808 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:25:48.054 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:48.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:48 smithi089 ceph-mon[30933]: pgmap v403: 129 pgs: 129 active+clean; 704 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-09-15T23:25:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:48 smithi120 ceph-mon[35250]: pgmap v403: 129 pgs: 129 active+clean; 704 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-09-15T23:25:48.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:48 smithi174 ceph-mon[35586]: pgmap v403: 129 pgs: 129 active+clean; 704 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-09-15T23:25:48.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.539+0000 7f5282ab3640 1 -- 172.21.15.89:0/1077696151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f527c10c170 msgr2=0x7f527c10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.539+0000 7f5282ab3640 1 --2- 172.21.15.89:0/1077696151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f527c10c170 0x7f527c10e610 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f5264009900 tx=0x7f526402a480 comp rx=0 tx=0).stop 2024-09-15T23:25:48.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.539+0000 7f5282ab3640 1 -- 172.21.15.89:0/1077696151 shutdown_connections 2024-09-15T23:25:48.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.539+0000 7f5282ab3640 1 --2- 172.21.15.89:0/1077696151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f527c10c170 0x7f527c10e610 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.539+0000 7f5282ab3640 1 --2- 172.21.15.89:0/1077696151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f527c103660 0x7f527c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.539+0000 7f5282ab3640 1 --2- 172.21.15.89:0/1077696151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f527c102d40 0x7f527c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.539+0000 7f5282ab3640 1 -- 172.21.15.89:0/1077696151 >> 172.21.15.89:0/1077696151 conn(0x7f527c0fc7a0 msgr2=0x7f527c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.539+0000 7f5282ab3640 1 -- 172.21.15.89:0/1077696151 shutdown_connections 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.539+0000 7f5282ab3640 1 -- 172.21.15.89:0/1077696151 wait complete. 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5282ab3640 1 Processor -- start 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5282ab3640 1 -- start start 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5282ab3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f527c102d40 0x7f527c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5282ab3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f527c103660 0x7f527c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5282ab3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f527c10c170 0x7f527c19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5282ab3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f527c111f70 con 0x7f527c10c170 2024-09-15T23:25:48.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5282ab3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f527c111df0 con 0x7f527c102d40 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5282ab3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f527c1120f0 con 0x7f527c103660 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5281050640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f527c10c170 0x7f527c19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5281050640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f527c10c170 0x7f527c19cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55126/0 (socket says 172.21.15.89:55126) 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5281050640 1 -- 172.21.15.89:0/151660342 learned_addr learned my addr 172.21.15.89:0/151660342 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f528084f640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f527c102d40 0x7f527c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f527bfff640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f527c103660 0x7f527c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5281050640 1 -- 172.21.15.89:0/151660342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f527c103660 msgr2=0x7f527c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5281050640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f527c103660 0x7f527c198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5281050640 1 -- 172.21.15.89:0/151660342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f527c102d40 msgr2=0x7f527c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5281050640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f527c102d40 0x7f527c198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f5281050640 1 -- 172.21.15.89:0/151660342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f527c19d3c0 con 0x7f527c10c170 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.540+0000 7f527bfff640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f527c103660 0x7f527c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.541+0000 7f5281050640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f527c10c170 0x7f527c19cce0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5264005ab0 tx=0x7f52640042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:48.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.541+0000 7f5279ffb640 1 -- 172.21.15.89:0/151660342 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526403f030 con 0x7f527c10c170 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.541+0000 7f5279ffb640 1 -- 172.21.15.89:0/151660342 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5264003ce0 con 0x7f527c10c170 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.541+0000 7f5282ab3640 1 -- 172.21.15.89:0/151660342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f527c19d650 con 0x7f527c10c170 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.541+0000 7f5279ffb640 1 -- 172.21.15.89:0/151660342 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5264035070 con 0x7f527c10c170 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.541+0000 7f5282ab3640 1 -- 172.21.15.89:0/151660342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f527c1a4f90 con 0x7f527c10c170 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.543+0000 7f5279ffb640 1 -- 172.21.15.89:0/151660342 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5264042030 con 0x7f527c10c170 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5284e8c740 1 Processor -- start 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5284e8c740 1 -- start start 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5279ffb640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52680a3240 0x7f52680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a3d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a3da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a48370 0x55e8c2a4a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5284e8c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e8c29a32f0 con 0x55e8c2a48370 2024-09-15T23:25:48.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5284e8c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e8c294acf0 con 0x55e8c2a3eb40 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5284e8c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e8c299f2b0 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f527bfff640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52680a3240 0x7f52680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5281050640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a48370 0x55e8c2a4a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5281050640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a48370 0x55e8c2a4a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55136/0 (socket says 172.21.15.89:55136) 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5281050640 1 -- 172.21.15.89:0/2600561093 learned_addr learned my addr 172.21.15.89:0/2600561093 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5279ffb640 1 -- 172.21.15.89:0/151660342 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f52640f0510 con 0x7f527c10c170 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f528084f640 1 --2- 172.21.15.89:0/2600561093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a3d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f527bfff640 1 --2- 172.21.15.89:0/2600561093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a3da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5281050640 1 -- 172.21.15.89:0/2600561093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3d5d0 msgr2=0x55e8c2a3da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5281050640 1 --2- 172.21.15.89:0/2600561093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a3da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5281050640 1 -- 172.21.15.89:0/2600561093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a3eb40 msgr2=0x55e8c2a3d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5281050640 1 --2- 172.21.15.89:0/2600561093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a3d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5281050640 1 -- 172.21.15.89:0/2600561093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8c292f3a0 con 0x55e8c2a48370 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f527bfff640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52680a3240 0x7f52680a5700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f526c00afd0 tx=0x7f526c013b20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.544+0000 7f5281050640 1 --2- 172.21.15.89:0/2600561093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a48370 0x55e8c2a4a760 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f526404b000 tx=0x7f526403b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec1912dd141e5de server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5272ffd640 1 -- 172.21.15.89:0/2600561093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5264049040 con 0x55e8c2a48370 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5272ffd640 1 -- 172.21.15.89:0/2600561093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52640bce40 con 0x55e8c2a48370 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5272ffd640 1 -- 172.21.15.89:0/2600561093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52640ba7e0 con 0x55e8c2a48370 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5284e8c740 1 -- 172.21.15.89:0/2600561093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a48370 msgr2=0x55e8c2a4a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5284e8c740 1 --2- 172.21.15.89:0/2600561093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a48370 0x55e8c2a4a760 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f526404b000 tx=0x7f526403b6b0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5284e8c740 1 -- 172.21.15.89:0/2600561093 shutdown_connections 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5284e8c740 1 --2- 172.21.15.89:0/2600561093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a48370 0x55e8c2a4a760 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5284e8c740 1 --2- 172.21.15.89:0/2600561093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a3da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5284e8c740 1 --2- 172.21.15.89:0/2600561093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a3d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5284e8c740 1 -- 172.21.15.89:0/2600561093 >> 172.21.15.89:0/2600561093 conn(0x55e8c2a06310 msgr2=0x55e8c2a39c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:48.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5284e8c740 1 -- 172.21.15.89:0/2600561093 shutdown_connections 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.545+0000 7f5284e8c740 1 -- 172.21.15.89:0/2600561093 wait complete. 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f5284e8c740 1 -- start start 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f5284e8c740 1 Processor -- start 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a47ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a44f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 0x55e8c2a454f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f5284e8c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e8c2a4ae30 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f5284e8c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e8c295e060 con 0x55e8c2a48370 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f5284e8c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e8c2a4afb0 con 0x55e8c2a3eb40 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f5284e8c740 10 client.0 _ll_register_callbacks cb 0x55e8c26b18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f528084f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a47ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f528084f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a47ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55144/0 (socket says 172.21.15.89:55144) 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f528084f640 1 -- 172.21.15.89:0/494034923 learned_addr learned my addr 172.21.15.89:0/494034923 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:48.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f5281050640 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 0x55e8c2a454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.546+0000 7f527bfff640 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a44f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f528084f640 1 -- 172.21.15.89:0/494034923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3eb40 msgr2=0x55e8c2a44f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f528084f640 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a44f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f528084f640 1 -- 172.21.15.89:0/494034923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 msgr2=0x55e8c2a454f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f528084f640 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 0x55e8c2a454f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f528084f640 1 -- 172.21.15.89:0/494034923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8c27143c0 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f528084f640 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a47ce0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f527400ba60 tx=0x7f527400bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ffb4bafd68aa9f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f525bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f525bfff640 1 -- 172.21.15.89:0/494034923 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274005020 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f525bfff640 1 -- 172.21.15.89:0/494034923 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f52740051c0 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f5284e8c740 1 -- 172.21.15.89:0/494034923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52640bce40 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f525bfff640 1 -- 172.21.15.89:0/494034923 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5274024050 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f5284e8c740 10 client.16050 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f5284e8c740 1 -- 172.21.15.89:0/494034923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e8c2a567e0 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f525bfff640 1 -- 172.21.15.89:0/494034923 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52640bce40 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f525bfff640 1 -- 172.21.15.89:0/494034923 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f527401b590 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f5284e8c740 10 client.16050 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f5284e8c740 10 client.16050 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.547+0000 7f5284e8c740 20 client.16050 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.548+0000 7f5284e8c740 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55e8c2a57ca0 0x55e8c2a5a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.548+0000 7f5284e8c740 4 client.16050 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.548+0000 7f5284e8c740 1 -- 172.21.15.89:0/494034923 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8c29b0860 con 0x55e8c2a57ca0 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.548+0000 7f527bfff640 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55e8c2a57ca0 0x55e8c2a5a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.548+0000 7f527bfff640 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55e8c2a57ca0 0x55e8c2a5a0f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b4b70a5de74d5d1 server_cookie=43ace1478742012f in_seq=0 out_seq=0 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.548+0000 7f525bfff640 10 client.16050 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f525bfff640 1 -- 172.21.15.89:0/494034923 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e8c29b0860 con 0x55e8c2a57ca0 2024-09-15T23:25:48.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f525bfff640 10 client.16050 handle_command_reply: tid=0 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f5284e8c740 1 client.16050 shutdown 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f5284e8c740 1 -- 172.21.15.89:0/494034923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 msgr2=0x55e8c2a47ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f5284e8c740 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a47ce0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f527400ba60 tx=0x7f527400bf20 comp rx=0 tx=0).stop 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f5284e8c740 1 -- 172.21.15.89:0/494034923 shutdown_connections 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f5284e8c740 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55e8c2a57ca0 0x55e8c2a5a0f0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f5284e8c740 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 0x55e8c2a454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f5284e8c740 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a44f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f5284e8c740 1 --2- 172.21.15.89:0/494034923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a47ce0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f525bfff640 0 client.16050 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.549+0000 7f5284e8c740 1 -- 172.21.15.89:0/494034923 >> 172.21.15.89:0/494034923 conn(0x55e8c2a06310 msgr2=0x55e8c2a3f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.550+0000 7f5284e8c740 1 -- 172.21.15.89:0/494034923 shutdown_connections 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.550+0000 7f5284e8c740 1 -- 172.21.15.89:0/494034923 wait complete. 2024-09-15T23:25:48.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.550+0000 7f5284e8c740 20 client.16050 trim_cache size 0 max 16384 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f5284e8c740 1 Processor -- start 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f5284e8c740 1 -- start start 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a44850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a38bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 0x55e8c2a39130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f5284e8c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f527c1aee20 con 0x55e8c2a3eb40 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f5284e8c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f527c1b0960 con 0x55e8c2a48370 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f5284e8c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f527c1af4f0 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f527bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a38bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f527bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a38bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55146/0 (socket says 172.21.15.89:55146) 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.562+0000 7f527bfff640 1 -- 172.21.15.89:0/1364471384 learned_addr learned my addr 172.21.15.89:0/1364471384 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f5281050640 1 --2- 172.21.15.89:0/1364471384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 0x55e8c2a39130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f528084f640 1 --2- 172.21.15.89:0/1364471384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a44850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f527bfff640 1 -- 172.21.15.89:0/1364471384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3d5d0 msgr2=0x55e8c2a44850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f527bfff640 1 --2- 172.21.15.89:0/1364471384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a44850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f527bfff640 1 -- 172.21.15.89:0/1364471384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 msgr2=0x55e8c2a39130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f527bfff640 1 --2- 172.21.15.89:0/1364471384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 0x55e8c2a39130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f527bfff640 1 -- 172.21.15.89:0/1364471384 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8c2a434e0 con 0x55e8c2a3eb40 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f5281050640 1 --2- 172.21.15.89:0/1364471384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 0x55e8c2a39130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f527bfff640 1 --2- 172.21.15.89:0/1364471384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a38bf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f526c005cc0 tx=0x7f526c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254b9c1ef8bfa784 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f52727fc640 1 -- 172.21.15.89:0/1364471384 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c004b20 con 0x55e8c2a3eb40 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f52727fc640 1 -- 172.21.15.89:0/1364471384 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f526c004cc0 con 0x55e8c2a3eb40 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.563+0000 7f52727fc640 1 -- 172.21.15.89:0/1364471384 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c005280 con 0x55e8c2a3eb40 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 -- 172.21.15.89:0/1364471384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3eb40 msgr2=0x55e8c2a38bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 --2- 172.21.15.89:0/1364471384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a38bf0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f526c005cc0 tx=0x7f526c01c000 comp rx=0 tx=0).stop 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 -- 172.21.15.89:0/1364471384 shutdown_connections 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 --2- 172.21.15.89:0/1364471384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a48370 0x55e8c2a39130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 --2- 172.21.15.89:0/1364471384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a38bf0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 --2- 172.21.15.89:0/1364471384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a44850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 -- 172.21.15.89:0/1364471384 >> 172.21.15.89:0/1364471384 conn(0x55e8c2a06310 msgr2=0x55e8c2a384e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 -- 172.21.15.89:0/1364471384 shutdown_connections 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 -- 172.21.15.89:0/1364471384 wait complete. 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 -- start start 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 Processor -- start 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a507f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a41b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a48370 0x55e8c2a42050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f527c1b0290 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f527c1afbc0 con 0x55e8c2a3eb40 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f527c1ace00 con 0x55e8c2a48370 2024-09-15T23:25:48.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f5284e8c740 10 client.0 _ll_register_callbacks cb 0x7f527400b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.564+0000 7f528084f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f528084f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a507f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55162/0 (socket says 172.21.15.89:55162) 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f528084f640 1 -- 172.21.15.89:0/3100935585 learned_addr learned my addr 172.21.15.89:0/3100935585 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f527bfff640 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a41b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f5281050640 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a48370 0x55e8c2a42050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f528084f640 1 -- 172.21.15.89:0/3100935585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a48370 msgr2=0x55e8c2a42050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f528084f640 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a48370 0x55e8c2a42050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f528084f640 1 -- 172.21.15.89:0/3100935585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a3eb40 msgr2=0x55e8c2a41b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f528084f640 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a41b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f528084f640 1 -- 172.21.15.89:0/3100935585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8c2a42690 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f5281050640 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a48370 0x55e8c2a42050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f528084f640 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a507f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5274000c00 tx=0x7f5274003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5562e54cc9ee0d71 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:48.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f525bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f525bfff640 1 -- 172.21.15.89:0/3100935585 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274021040 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f525bfff640 1 -- 172.21.15.89:0/3100935585 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f527401b470 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f5284e8c740 1 -- 172.21.15.89:0/3100935585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f526c004cc0 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f525bfff640 1 -- 172.21.15.89:0/3100935585 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5274038050 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f5284e8c740 10 client.16056 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.565+0000 7f5284e8c740 1 -- 172.21.15.89:0/3100935585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e8c2ac27a0 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.566+0000 7f525bfff640 1 -- 172.21.15.89:0/3100935585 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f526c004cc0 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.566+0000 7f525bfff640 1 -- 172.21.15.89:0/3100935585 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f52740049a0 con 0x55e8c2a3d5d0 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.566+0000 7f5284e8c740 10 client.16056 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.566+0000 7f5284e8c740 10 client.16056 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.566+0000 7f5284e8c740 20 client.16056 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.566+0000 7f5284e8c740 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55e8c2ac2f50 0x55e8c2ac5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.566+0000 7f5284e8c740 4 client.16056 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:48.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.566+0000 7f5284e8c740 1 -- 172.21.15.89:0/3100935585 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8c29aad70 con 0x55e8c2ac2f50 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.566+0000 7f527bfff640 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55e8c2ac2f50 0x55e8c2ac5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f525bfff640 10 client.16056 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f527bfff640 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55e8c2ac2f50 0x55e8c2ac5370 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f6efc26a11ce4012 server_cookie=53959ecf398572d1 in_seq=0 out_seq=0 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f525bfff640 1 -- 172.21.15.89:0/3100935585 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8c29aad70 con 0x55e8c2ac2f50 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f525bfff640 10 client.16056 handle_command_reply: tid=0 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 client.16056 shutdown 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 -- 172.21.15.89:0/3100935585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 msgr2=0x55e8c2a507f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a507f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5274000c00 tx=0x7f5274003ee0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 -- 172.21.15.89:0/3100935585 shutdown_connections 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55e8c2ac2f50 0x55e8c2ac5370 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8c2a48370 0x55e8c2a42050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f525bfff640 0 client.16056 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8c2a3eb40 0x55e8c2a41b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 --2- 172.21.15.89:0/3100935585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8c2a3d5d0 0x55e8c2a507f0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 -- 172.21.15.89:0/3100935585 >> 172.21.15.89:0/3100935585 conn(0x55e8c2a06310 msgr2=0x55e8c2a384e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:48.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 -- 172.21.15.89:0/3100935585 shutdown_connections 2024-09-15T23:25:48.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 1 -- 172.21.15.89:0/3100935585 wait complete. 2024-09-15T23:25:48.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.567+0000 7f5284e8c740 20 client.16056 trim_cache size 0 max 16384 2024-09-15T23:25:48.584 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:48.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 -- 172.21.15.89:0/151660342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52680a3240 msgr2=0x7f52680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52680a3240 0x7f52680a5700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f526c00afd0 tx=0x7f526c013b20 comp rx=0 tx=0).stop 2024-09-15T23:25:48.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 -- 172.21.15.89:0/151660342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f527c10c170 msgr2=0x7f527c19cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f527c10c170 0x7f527c19cce0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5264005ab0 tx=0x7f52640042f0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 -- 172.21.15.89:0/151660342 shutdown_connections 2024-09-15T23:25:48.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52680a3240 0x7f52680a5700 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f527c10c170 0x7f527c19cce0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f527c103660 0x7f527c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 --2- 172.21.15.89:0/151660342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f527c102d40 0x7f527c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 -- 172.21.15.89:0/151660342 >> 172.21.15.89:0/151660342 conn(0x7f527c0fc7a0 msgr2=0x7f527c0fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:48.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 -- 172.21.15.89:0/151660342 shutdown_connections 2024-09-15T23:25:48.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:48.568+0000 7f5282ab3640 1 -- 172.21.15.89:0/151660342 wait complete. 2024-09-15T23:25:49.267 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:25:49.514 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:50.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 -- 172.21.15.89:0/325426177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2438101420 msgr2=0x7f243810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:50.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 --2- 172.21.15.89:0/325426177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2438101420 0x7f243810d950 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2430009cd0 tx=0x7f243002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 -- 172.21.15.89:0/325426177 shutdown_connections 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 --2- 172.21.15.89:0/325426177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2438101420 0x7f243810d950 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 --2- 172.21.15.89:0/325426177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2438100a80 0x7f2438100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 --2- 172.21.15.89:0/325426177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2438106a80 0x7f2438106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 -- 172.21.15.89:0/325426177 >> 172.21.15.89:0/325426177 conn(0x7f24380fc7a0 msgr2=0x7f24380febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 -- 172.21.15.89:0/325426177 shutdown_connections 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 -- 172.21.15.89:0/325426177 wait complete. 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 Processor -- start 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.006+0000 7f243ee46640 1 -- start start 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243ee46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2438100a80 0x7f24381a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243ee46640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2438101420 0x7f24381a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243ee46640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2438106a80 0x7f243819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243ee46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f243810f900 con 0x7f2438100a80 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243ee46640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f243810f780 con 0x7f2438106a80 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243ee46640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f243810fa80 con 0x7f2438101420 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243cbe2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2438100a80 0x7f24381a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243cbe2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2438100a80 0x7f24381a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55172/0 (socket says 172.21.15.89:55172) 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243cbe2640 1 -- 172.21.15.89:0/507577368 learned_addr learned my addr 172.21.15.89:0/507577368 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f2437fff640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2438101420 0x7f24381a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243d3e3640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2438106a80 0x7f243819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243cbe2640 1 -- 172.21.15.89:0/507577368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2438101420 msgr2=0x7f24381a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243cbe2640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2438101420 0x7f24381a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243cbe2640 1 -- 172.21.15.89:0/507577368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2438106a80 msgr2=0x7f243819a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:50.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243cbe2640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2438106a80 0x7f243819a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243cbe2640 1 -- 172.21.15.89:0/507577368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f243819acf0 con 0x7f2438100a80 2024-09-15T23:25:50.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243d3e3640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2438106a80 0x7f243819a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:50.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f243cbe2640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2438100a80 0x7f24381a0400 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f242c00b4a0 tx=0x7f242c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:50.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.007+0000 7f2435ffb640 1 -- 172.21.15.89:0/507577368 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c015020 con 0x7f2438100a80 2024-09-15T23:25:50.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.008+0000 7f2435ffb640 1 -- 172.21.15.89:0/507577368 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f242c011d10 con 0x7f2438100a80 2024-09-15T23:25:50.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.008+0000 7f243ee46640 1 -- 172.21.15.89:0/507577368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f243819afe0 con 0x7f2438100a80 2024-09-15T23:25:50.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.008+0000 7f2435ffb640 1 -- 172.21.15.89:0/507577368 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c00f2e0 con 0x7f2438100a80 2024-09-15T23:25:50.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.008+0000 7f243ee46640 1 -- 172.21.15.89:0/507577368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24381a72d0 con 0x7f2438100a80 2024-09-15T23:25:50.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.010+0000 7f2435ffb640 1 -- 172.21.15.89:0/507577368 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f242c018020 con 0x7f2438100a80 2024-09-15T23:25:50.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.010+0000 7f244121f740 1 Processor -- start 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.010+0000 7f244121f740 1 -- start start 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.010+0000 7f2435ffb640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24280a3240 0x7f24280a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.010+0000 7f244121f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505ca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.010+0000 7f244121f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505caa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.010+0000 7f244121f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 0x55c0505d7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.010+0000 7f244121f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c0505302f0 con 0x55c0505cbb40 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.010+0000 7f244121f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c0504d7cf0 con 0x55c0505ca5d0 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.010+0000 7f244121f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c05052c2b0 con 0x55c0505d5370 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243d3e3640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24280a3240 0x7f24280a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f2435ffb640 1 -- 172.21.15.89:0/507577368 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f242c0cf5d0 con 0x7f2438100a80 2024-09-15T23:25:50.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243cbe2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243cbe2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505ca090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55182/0 (socket says 172.21.15.89:55182) 2024-09-15T23:25:50.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243cbe2640 1 -- 172.21.15.89:0/714994202 learned_addr learned my addr 172.21.15.89:0/714994202 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:50.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f2437fff640 1 --2- 172.21.15.89:0/714994202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243d3e3640 1 --2- 172.21.15.89:0/714994202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 0x55c0505d7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243cbe2640 1 -- 172.21.15.89:0/714994202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 msgr2=0x55c0505d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:50.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243cbe2640 1 --2- 172.21.15.89:0/714994202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 0x55c0505d7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243cbe2640 1 -- 172.21.15.89:0/714994202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 msgr2=0x55c0505caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:50.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243cbe2640 1 --2- 172.21.15.89:0/714994202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505caa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243cbe2640 1 -- 172.21.15.89:0/714994202 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0504bc3a0 con 0x55c0505cbb40 2024-09-15T23:25:50.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f2437fff640 1 --2- 172.21.15.89:0/714994202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:50.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243cbe2640 1 --2- 172.21.15.89:0/714994202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505ca090 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f242c09b790 tx=0x7f242c09b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea6a3c701fc8a64 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:50.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f243d3e3640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24280a3240 0x7f24280a5700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2430006010 tx=0x7f2430031800 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:50.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f2422ffd640 1 -- 172.21.15.89:0/714994202 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c078c30 con 0x55c0505cbb40 2024-09-15T23:25:50.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f2422ffd640 1 -- 172.21.15.89:0/714994202 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f242c09bf00 con 0x55c0505cbb40 2024-09-15T23:25:50.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.011+0000 7f2422ffd640 1 -- 172.21.15.89:0/714994202 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242c0092b0 con 0x55c0505cbb40 2024-09-15T23:25:50.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 -- 172.21.15.89:0/714994202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 msgr2=0x55c0505ca090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:50.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 --2- 172.21.15.89:0/714994202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505ca090 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f242c09b790 tx=0x7f242c09b400 comp rx=0 tx=0).stop 2024-09-15T23:25:50.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 -- 172.21.15.89:0/714994202 shutdown_connections 2024-09-15T23:25:50.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 --2- 172.21.15.89:0/714994202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 0x55c0505d7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 --2- 172.21.15.89:0/714994202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 --2- 172.21.15.89:0/714994202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505ca090 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 -- 172.21.15.89:0/714994202 >> 172.21.15.89:0/714994202 conn(0x55c050593310 msgr2=0x55c0505c6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 -- 172.21.15.89:0/714994202 shutdown_connections 2024-09-15T23:25:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 -- 172.21.15.89:0/714994202 wait complete. 2024-09-15T23:25:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 -- start start 2024-09-15T23:25:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.012+0000 7f244121f740 1 Processor -- start 2024-09-15T23:25:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f244121f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505d4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f244121f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f244121f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 0x55c0505d2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f244121f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c0505d8010 con 0x55c0505cbb40 2024-09-15T23:25:50.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f244121f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c0504eb060 con 0x55c0505ca5d0 2024-09-15T23:25:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f244121f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c0505d8190 con 0x55c0505d5370 2024-09-15T23:25:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f244121f740 10 client.0 _ll_register_callbacks cb 0x55c05023e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f2437fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f2437fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505d2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55188/0 (socket says 172.21.15.89:55188) 2024-09-15T23:25:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f2437fff640 1 -- 172.21.15.89:0/3846822175 learned_addr learned my addr 172.21.15.89:0/3846822175 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f243cbe2640 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505d4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f243d3e3640 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 0x55c0505d2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f2437fff640 1 -- 172.21.15.89:0/3846822175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 msgr2=0x55c0505d2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f2437fff640 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 0x55c0505d2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f2437fff640 1 -- 172.21.15.89:0/3846822175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 msgr2=0x55c0505d4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f2437fff640 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505d4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f2437fff640 1 -- 172.21.15.89:0/3846822175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0502a13c0 con 0x55c0505cbb40 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f243cbe2640 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505d4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f243d3e3640 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 0x55c0505d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f2437fff640 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505d2010 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2424009e80 tx=0x7f2424011630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f8a2581ef3b7b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.013+0000 7f2417fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f2417fff640 1 -- 172.21.15.89:0/3846822175 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2424019070 con 0x55c0505cbb40 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f2417fff640 1 -- 172.21.15.89:0/3846822175 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2424015660 con 0x55c0505cbb40 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f244121f740 1 -- 172.21.15.89:0/3846822175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f242c09bf00 con 0x55c0505cbb40 2024-09-15T23:25:50.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f2417fff640 1 -- 172.21.15.89:0/3846822175 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2424025050 con 0x55c0505cbb40 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f244121f740 10 client.16068 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f244121f740 1 -- 172.21.15.89:0/3846822175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c0505e3b10 con 0x55c0505cbb40 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f2417fff640 1 -- 172.21.15.89:0/3846822175 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f242c09bf00 con 0x55c0505cbb40 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f2417fff640 1 -- 172.21.15.89:0/3846822175 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2424014a80 con 0x55c0505cbb40 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f244121f740 10 client.16068 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f244121f740 10 client.16068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f244121f740 20 client.16068 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f244121f740 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0505e4f70 0x55c0505e73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f244121f740 4 client.16068 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.014+0000 7f244121f740 1 -- 172.21.15.89:0/3846822175 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c05053d860 con 0x55c0505e4f70 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.015+0000 7f243cbe2640 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0505e4f70 0x55c0505e73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.015+0000 7f243cbe2640 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0505e4f70 0x55c0505e73d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2d8125883dbffd8 server_cookie=1778e42ea5b65c51 in_seq=0 out_seq=0 2024-09-15T23:25:50.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.015+0000 7f2417fff640 10 client.16068 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:50.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f2417fff640 1 -- 172.21.15.89:0/3846822175 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c05053d860 con 0x55c0505e4f70 2024-09-15T23:25:50.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f2417fff640 10 client.16068 handle_command_reply: tid=0 2024-09-15T23:25:50.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 client.16068 shutdown 2024-09-15T23:25:50.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 -- 172.21.15.89:0/3846822175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 msgr2=0x55c0505d2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:50.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505d2010 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2424009e80 tx=0x7f2424011630 comp rx=0 tx=0).stop 2024-09-15T23:25:50.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 -- 172.21.15.89:0/3846822175 shutdown_connections 2024-09-15T23:25:50.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0505e4f70 0x55c0505e73d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505d5370 0x55c0505d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505cbb40 0x55c0505d2010 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f2417fff640 0 client.16068 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 --2- 172.21.15.89:0/3846822175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505d4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 -- 172.21.15.89:0/3846822175 >> 172.21.15.89:0/3846822175 conn(0x55c050593310 msgr2=0x55c0505cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:50.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 -- 172.21.15.89:0/3846822175 shutdown_connections 2024-09-15T23:25:50.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 1 -- 172.21.15.89:0/3846822175 wait complete. 2024-09-15T23:25:50.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.016+0000 7f244121f740 20 client.16068 trim_cache size 0 max 16384 2024-09-15T23:25:50.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.029+0000 7f244121f740 1 Processor -- start 2024-09-15T23:25:50.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.029+0000 7f244121f740 1 -- start start 2024-09-15T23:25:50.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.029+0000 7f244121f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505ca5d0 0x55c0505cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.029+0000 7f244121f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505cbb40 0x55c0505cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.029+0000 7f244121f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c0505ecca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.029+0000 7f244121f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f24381b1260 con 0x55c0505d5370 2024-09-15T23:25:50.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.029+0000 7f244121f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f24381b2da0 con 0x55c0505cbb40 2024-09-15T23:25:50.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.029+0000 7f244121f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f24381b1930 con 0x55c0505ca5d0 2024-09-15T23:25:50.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243d3e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c0505ecca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243d3e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c0505ecca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55190/0 (socket says 172.21.15.89:55190) 2024-09-15T23:25:50.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243d3e3640 1 -- 172.21.15.89:0/3508545179 learned_addr learned my addr 172.21.15.89:0/3508545179 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:50.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243cbe2640 1 --2- 172.21.15.89:0/3508545179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505ca5d0 0x55c0505cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f2437fff640 1 --2- 172.21.15.89:0/3508545179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505cbb40 0x55c0505cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243d3e3640 1 -- 172.21.15.89:0/3508545179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505ca5d0 msgr2=0x55c0505cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:50.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243d3e3640 1 --2- 172.21.15.89:0/3508545179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505ca5d0 0x55c0505cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243d3e3640 1 -- 172.21.15.89:0/3508545179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505cbb40 msgr2=0x55c0505cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:50.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243d3e3640 1 --2- 172.21.15.89:0/3508545179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505cbb40 0x55c0505cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243d3e3640 1 -- 172.21.15.89:0/3508545179 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0505cdc00 con 0x55c0505d5370 2024-09-15T23:25:50.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243cbe2640 1 --2- 172.21.15.89:0/3508545179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505ca5d0 0x55c0505cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:50.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f243d3e3640 1 --2- 172.21.15.89:0/3508545179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c0505ecca0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2430009a40 tx=0x7f2430043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109c1ff4b17bacdd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:50.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f24227fc640 1 -- 172.21.15.89:0/3508545179 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243004e070 con 0x55c0505d5370 2024-09-15T23:25:50.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f24227fc640 1 -- 172.21.15.89:0/3508545179 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2430036040 con 0x55c0505d5370 2024-09-15T23:25:50.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.030+0000 7f24227fc640 1 -- 172.21.15.89:0/3508545179 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430030db0 con 0x55c0505d5370 2024-09-15T23:25:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 -- 172.21.15.89:0/3508545179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 msgr2=0x55c0505ecca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 --2- 172.21.15.89:0/3508545179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c0505ecca0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2430009a40 tx=0x7f2430043000 comp rx=0 tx=0).stop 2024-09-15T23:25:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 -- 172.21.15.89:0/3508545179 shutdown_connections 2024-09-15T23:25:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 --2- 172.21.15.89:0/3508545179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c0505ecca0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 --2- 172.21.15.89:0/3508545179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505cbb40 0x55c0505cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 --2- 172.21.15.89:0/3508545179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505ca5d0 0x55c0505cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 -- 172.21.15.89:0/3508545179 >> 172.21.15.89:0/3508545179 conn(0x55c050593310 msgr2=0x55c0505d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 -- 172.21.15.89:0/3508545179 shutdown_connections 2024-09-15T23:25:50.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 -- 172.21.15.89:0/3508545179 wait complete. 2024-09-15T23:25:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 -- start start 2024-09-15T23:25:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.031+0000 7f244121f740 1 Processor -- start 2024-09-15T23:25:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f244121f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505d9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f244121f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505cbb40 0x55c0505d9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f244121f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c05064ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f244121f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f24381b26d0 con 0x55c0505d5370 2024-09-15T23:25:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f244121f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f24381b2000 con 0x55c0505ca5d0 2024-09-15T23:25:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f244121f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f24381af240 con 0x55c0505cbb40 2024-09-15T23:25:50.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f244121f740 10 client.0 _ll_register_callbacks cb 0x7f24240093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f243d3e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c05064ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f243d3e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c05064ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55204/0 (socket says 172.21.15.89:55204) 2024-09-15T23:25:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f243d3e3640 1 -- 172.21.15.89:0/220046175 learned_addr learned my addr 172.21.15.89:0/220046175 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:50.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f243cbe2640 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505d9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f2437fff640 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505cbb40 0x55c0505d9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f243d3e3640 1 -- 172.21.15.89:0/220046175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505cbb40 msgr2=0x55c0505d9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f243d3e3640 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505cbb40 0x55c0505d9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f243d3e3640 1 -- 172.21.15.89:0/220046175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 msgr2=0x55c0505d9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:50.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f243d3e3640 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505d9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.032+0000 7f243d3e3640 1 -- 172.21.15.89:0/220046175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c05064f260 con 0x55c0505d5370 2024-09-15T23:25:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f243d3e3640 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c05064ec70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2430009430 tx=0x7f2430005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c90a41985c93d4b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f2417fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:50.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f2417fff640 1 -- 172.21.15.89:0/220046175 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430057030 con 0x55c0505d5370 2024-09-15T23:25:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f2417fff640 1 -- 172.21.15.89:0/220046175 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f243002e040 con 0x55c0505d5370 2024-09-15T23:25:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f244121f740 1 -- 172.21.15.89:0/220046175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2430036040 con 0x55c0505d5370 2024-09-15T23:25:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f2417fff640 1 -- 172.21.15.89:0/220046175 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f243004e070 con 0x55c0505d5370 2024-09-15T23:25:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f244121f740 10 client.16074 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:50.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f244121f740 1 -- 172.21.15.89:0/220046175 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c050650520 con 0x55c0505d5370 2024-09-15T23:25:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f2417fff640 1 -- 172.21.15.89:0/220046175 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2430036040 con 0x55c0505d5370 2024-09-15T23:25:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f2417fff640 1 -- 172.21.15.89:0/220046175 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2430049040 con 0x55c0505d5370 2024-09-15T23:25:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f244121f740 10 client.16074 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:50.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f244121f740 10 client.16074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:50.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f244121f740 20 client.16074 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:50.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f244121f740 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c050650b90 0x55c050652f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:50.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f244121f740 4 client.16074 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:50.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.033+0000 7f244121f740 1 -- 172.21.15.89:0/220046175 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c050537d70 con 0x55c050650b90 2024-09-15T23:25:50.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f243cbe2640 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c050650b90 0x55c050652f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:50.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f243cbe2640 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c050650b90 0x55c050652f80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5975b0a1156b647 server_cookie=ee1aabbef99c2895 in_seq=0 out_seq=0 2024-09-15T23:25:50.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f2417fff640 10 client.16074 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:50.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f2417fff640 1 -- 172.21.15.89:0/220046175 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c050537d70 con 0x55c050650b90 2024-09-15T23:25:50.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f2417fff640 10 client.16074 handle_command_reply: tid=0 2024-09-15T23:25:50.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f244121f740 1 client.16074 shutdown 2024-09-15T23:25:50.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f244121f740 1 -- 172.21.15.89:0/220046175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 msgr2=0x55c05064ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:50.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f244121f740 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c05064ec70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2430009430 tx=0x7f2430005ef0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f244121f740 1 -- 172.21.15.89:0/220046175 shutdown_connections 2024-09-15T23:25:50.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f244121f740 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c050650b90 0x55c050652f80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f244121f740 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0505d5370 0x55c05064ec70 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f2417fff640 0 client.16074 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:50.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f244121f740 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0505cbb40 0x55c0505d9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f244121f740 1 --2- 172.21.15.89:0/220046175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0505ca5d0 0x55c0505d9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.034+0000 7f244121f740 1 -- 172.21.15.89:0/220046175 >> 172.21.15.89:0/220046175 conn(0x55c050593310 msgr2=0x55c0505c5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f244121f740 1 -- 172.21.15.89:0/220046175 shutdown_connections 2024-09-15T23:25:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f244121f740 1 -- 172.21.15.89:0/220046175 wait complete. 2024-09-15T23:25:50.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f244121f740 20 client.16074 trim_cache size 0 max 16384 2024-09-15T23:25:50.066 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 -- 172.21.15.89:0/507577368 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24280a3240 msgr2=0x7f24280a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:50.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24280a3240 0x7f24280a5700 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2430006010 tx=0x7f2430031800 comp rx=0 tx=0).stop 2024-09-15T23:25:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 -- 172.21.15.89:0/507577368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2438100a80 msgr2=0x7f24381a0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2438100a80 0x7f24381a0400 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f242c00b4a0 tx=0x7f242c00b8a0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 -- 172.21.15.89:0/507577368 shutdown_connections 2024-09-15T23:25:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24280a3240 0x7f24280a5700 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2438106a80 0x7f243819a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2438101420 0x7f24381a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 --2- 172.21.15.89:0/507577368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2438100a80 0x7f24381a0400 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 -- 172.21.15.89:0/507577368 >> 172.21.15.89:0/507577368 conn(0x7f24380fc7a0 msgr2=0x7f24380fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 -- 172.21.15.89:0/507577368 shutdown_connections 2024-09-15T23:25:50.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:50.035+0000 7f243ee46640 1 -- 172.21.15.89:0/507577368 wait complete. 2024-09-15T23:25:50.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:50 smithi089 ceph-mon[30933]: pgmap v404: 129 pgs: 129 active+clean; 704 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-09-15T23:25:50.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:50 smithi120 ceph-mon[35250]: pgmap v404: 129 pgs: 129 active+clean; 704 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-09-15T23:25:50.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:50 smithi174 ceph-mon[35586]: pgmap v404: 129 pgs: 129 active+clean; 704 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-09-15T23:25:51.787 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:25:51.787 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:25:52.041 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:52.422 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:52 smithi089 ceph-mon[30933]: pgmap v405: 129 pgs: 129 active+clean; 704 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.0 KiB/s wr, 12 op/s 2024-09-15T23:25:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:52 smithi120 ceph-mon[35250]: pgmap v405: 129 pgs: 129 active+clean; 704 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.0 KiB/s wr, 12 op/s 2024-09-15T23:25:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:52 smithi174 ceph-mon[35586]: pgmap v405: 129 pgs: 129 active+clean; 704 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.0 KiB/s wr, 12 op/s 2024-09-15T23:25:52.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 -- 172.21.15.89:0/3190931570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d01036e0 msgr2=0x7f80d0107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:52.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 --2- 172.21.15.89:0/3190931570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d01036e0 0x7f80d0107b90 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009930 tx=0x7f80b802a420 comp rx=0 tx=0).stop 2024-09-15T23:25:52.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 -- 172.21.15.89:0/3190931570 shutdown_connections 2024-09-15T23:25:52.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 --2- 172.21.15.89:0/3190931570 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80d01082a0 0x7f80d010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 --2- 172.21.15.89:0/3190931570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d01036e0 0x7f80d0107b90 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 --2- 172.21.15.89:0/3190931570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80d0102d30 0x7f80d0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 -- 172.21.15.89:0/3190931570 >> 172.21.15.89:0/3190931570 conn(0x7f80d00fc7d0 msgr2=0x7f80d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:52.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 -- 172.21.15.89:0/3190931570 shutdown_connections 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 -- 172.21.15.89:0/3190931570 wait complete. 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 Processor -- start 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.505+0000 7f80d6279640 1 -- start start 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80d6279640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d0102d30 0x7f80d01986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80d6279640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80d01036e0 0x7f80d0198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80d6279640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80d01082a0 0x7f80d019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80d6279640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01120c0 con 0x7f80d0102d30 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80d6279640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f80d0111f40 con 0x7f80d01036e0 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80d6279640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f80d0112240 con 0x7f80d01082a0 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d0102d30 0x7f80d01986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d0102d30 0x7f80d01986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55222/0 (socket says 172.21.15.89:55222) 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cffff640 1 -- 172.21.15.89:0/2163183138 learned_addr learned my addr 172.21.15.89:0/2163183138 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80d4816640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80d01082a0 0x7f80d019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cffff640 1 -- 172.21.15.89:0/2163183138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80d01082a0 msgr2=0x7f80d019cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cffff640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80d01082a0 0x7f80d019cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cffff640 1 -- 172.21.15.89:0/2163183138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80d01036e0 msgr2=0x7f80d0198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cffff640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80d01036e0 0x7f80d0198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cffff640 1 -- 172.21.15.89:0/2163183138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d019d650 con 0x7f80d0102d30 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cffff640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d0102d30 0x7f80d01986a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f80c000d950 tx=0x7f80c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cdffb640 1 -- 172.21.15.89:0/2163183138 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c0017070 con 0x7f80d0102d30 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cdffb640 1 -- 172.21.15.89:0/2163183138 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c0009c30 con 0x7f80d0102d30 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80d6279640 1 -- 172.21.15.89:0/2163183138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d019d940 con 0x7f80d0102d30 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80cdffb640 1 -- 172.21.15.89:0/2163183138 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c000f6b0 con 0x7f80d0102d30 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.506+0000 7f80d6279640 1 -- 172.21.15.89:0/2163183138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d01a5220 con 0x7f80d0102d30 2024-09-15T23:25:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.509+0000 7f80cdffb640 1 -- 172.21.15.89:0/2163183138 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f80c00109d0 con 0x7f80d0102d30 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.509+0000 7f80d8652740 1 Processor -- start 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.509+0000 7f80d8652740 1 -- start start 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80d8652740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d86f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80d8652740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d91800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80d8652740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d94130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80d8652740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557063cec280 con 0x557063d91d40 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80d8652740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557063c93b10 con 0x557063d86b60 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80d8652740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557063ce8240 con 0x557063d87480 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80c7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d94130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80c7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d94130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55224/0 (socket says 172.21.15.89:55224) 2024-09-15T23:25:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80c7fff640 1 -- 172.21.15.89:0/258555376 learned_addr learned my addr 172.21.15.89:0/258555376 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80d4816640 1 --2- 172.21.15.89:0/258555376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d86f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80cffff640 1 --2- 172.21.15.89:0/258555376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d91800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80cdffb640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f80bc0a3240 0x7f80bc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80c7fff640 1 -- 172.21.15.89:0/258555376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 msgr2=0x557063d91800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80c7fff640 1 --2- 172.21.15.89:0/258555376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d91800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80c7fff640 1 -- 172.21.15.89:0/258555376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 msgr2=0x557063d86f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80c7fff640 1 --2- 172.21.15.89:0/258555376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d86f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80c7fff640 1 -- 172.21.15.89:0/258555376 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557063ca01f0 con 0x557063d91d40 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80d4816640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f80bc0a3240 0x7f80bc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80cdffb640 1 -- 172.21.15.89:0/2163183138 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f80c00cfa40 con 0x7f80d0102d30 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.510+0000 7f80c7fff640 1 --2- 172.21.15.89:0/258555376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d94130 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f80b8033000 tx=0x7f80b8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1b5f8d7fcf68f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80c67fc640 1 -- 172.21.15.89:0/258555376 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8040070 con 0x557063d91d40 2024-09-15T23:25:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80d4816640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f80bc0a3240 0x7f80bc0a5700 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80c800a4e0 tx=0x7f80c800c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80c67fc640 1 -- 172.21.15.89:0/258555376 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80b8004040 con 0x557063d91d40 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80c67fc640 1 -- 172.21.15.89:0/258555376 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8035040 con 0x557063d91d40 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80d8652740 1 -- 172.21.15.89:0/258555376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 msgr2=0x557063d94130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80d8652740 1 --2- 172.21.15.89:0/258555376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d94130 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f80b8033000 tx=0x7f80b8005d50 comp rx=0 tx=0).stop 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80d8652740 1 -- 172.21.15.89:0/258555376 shutdown_connections 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80d8652740 1 --2- 172.21.15.89:0/258555376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d94130 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80d8652740 1 --2- 172.21.15.89:0/258555376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d91800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80d8652740 1 --2- 172.21.15.89:0/258555376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d86f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80d8652740 1 -- 172.21.15.89:0/258555376 >> 172.21.15.89:0/258555376 conn(0x557063d830f0 msgr2=0x557063d834d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80d8652740 1 -- 172.21.15.89:0/258555376 shutdown_connections 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.511+0000 7f80d8652740 1 -- 172.21.15.89:0/258555376 wait complete. 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d8652740 1 -- start start 2024-09-15T23:25:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d8652740 1 Processor -- start 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d8652740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d8eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d8652740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d8f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d8652740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d8f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d8652740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557063d94910 con 0x557063d91d40 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d8652740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557063ca6f10 con 0x557063d86b60 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d8652740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557063d94a90 con 0x557063d87480 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d8652740 10 client.0 _ll_register_callbacks cb 0x557063c6fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d4816640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d8f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d4816640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d8f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55240/0 (socket says 172.21.15.89:55240) 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80d4816640 1 -- 172.21.15.89:0/1977664934 learned_addr learned my addr 172.21.15.89:0/1977664934 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.512+0000 7f80cffff640 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d8eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80c7fff640 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d8f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80d4816640 1 -- 172.21.15.89:0/1977664934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 msgr2=0x557063d8f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80d4816640 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d8f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80d4816640 1 -- 172.21.15.89:0/1977664934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 msgr2=0x557063d8eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80d4816640 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d8eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80d4816640 1 -- 172.21.15.89:0/1977664934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557063a5d6a0 con 0x557063d91d40 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80c7fff640 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d8f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80d4816640 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d8f5f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f80c8019ea0 tx=0x7f80c800e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4515f90e36cd7582 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80af7fe640 1 -- 172.21.15.89:0/1977664934 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8027060 con 0x557063d91d40 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80d8652740 1 -- 172.21.15.89:0/1977664934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80b8004040 con 0x557063d91d40 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80af7fe640 1 -- 172.21.15.89:0/1977664934 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f80c80190d0 con 0x557063d91d40 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80af7fe640 1 -- 172.21.15.89:0/1977664934 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f80c80192d0 con 0x557063d91d40 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80d8652740 10 client.16086 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80d8652740 1 -- 172.21.15.89:0/1977664934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557063da0510 con 0x557063d91d40 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.513+0000 7f80af7fe640 1 -- 172.21.15.89:0/1977664934 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80b8004040 con 0x557063d91d40 2024-09-15T23:25:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.514+0000 7f80af7fe640 1 -- 172.21.15.89:0/1977664934 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f80c8022650 con 0x557063d91d40 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.514+0000 7f80d8652740 10 client.16086 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.514+0000 7f80d8652740 10 client.16086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.514+0000 7f80d8652740 20 client.16086 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.514+0000 7f80d8652740 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557063da19d0 0x557063da3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.514+0000 7f80d8652740 4 client.16086 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.514+0000 7f80d8652740 1 -- 172.21.15.89:0/1977664934 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557063cf97f0 con 0x557063da19d0 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.514+0000 7f80cffff640 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557063da19d0 0x557063da3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.514+0000 7f80af7fe640 10 client.16086 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.514+0000 7f80cffff640 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557063da19d0 0x557063da3e90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea56b7139cc3a8a5 server_cookie=ed64cc07338e54aa in_seq=0 out_seq=0 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80af7fe640 1 -- 172.21.15.89:0/1977664934 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557063cf97f0 con 0x557063da19d0 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80af7fe640 10 client.16086 handle_command_reply: tid=0 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 client.16086 shutdown 2024-09-15T23:25:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 -- 172.21.15.89:0/1977664934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 msgr2=0x557063d8f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d8f5f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f80c8019ea0 tx=0x7f80c800e040 comp rx=0 tx=0).stop 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 -- 172.21.15.89:0/1977664934 shutdown_connections 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557063da19d0 0x557063da3e90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d8f5f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d8f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 --2- 172.21.15.89:0/1977664934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d8eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80af7fe640 0 client.16086 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 -- 172.21.15.89:0/1977664934 >> 172.21.15.89:0/1977664934 conn(0x557063d830f0 msgr2=0x557063d88dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 -- 172.21.15.89:0/1977664934 shutdown_connections 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.515+0000 7f80d8652740 1 -- 172.21.15.89:0/1977664934 wait complete. 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.516+0000 7f80d8652740 20 client.16086 trim_cache size 0 max 16384 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.532+0000 7f80d8652740 1 Processor -- start 2024-09-15T23:25:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.532+0000 7f80d8652740 1 -- start start 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.532+0000 7f80d8652740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d88cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.532+0000 7f80d8652740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d9a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d8652740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d9a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d8652740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01aee00 con 0x557063d91d40 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d8652740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01b0940 con 0x557063d86b60 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d8652740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01af4d0 con 0x557063d87480 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d4816640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d9a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d4816640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d9a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55252/0 (socket says 172.21.15.89:55252) 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d4816640 1 -- 172.21.15.89:0/2346847228 learned_addr learned my addr 172.21.15.89:0/2346847228 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80c7fff640 1 --2- 172.21.15.89:0/2346847228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d9a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80cffff640 1 --2- 172.21.15.89:0/2346847228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d88cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d4816640 1 -- 172.21.15.89:0/2346847228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 msgr2=0x557063d9a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d4816640 1 --2- 172.21.15.89:0/2346847228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d9a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d4816640 1 -- 172.21.15.89:0/2346847228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 msgr2=0x557063d88cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d4816640 1 --2- 172.21.15.89:0/2346847228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d88cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d4816640 1 -- 172.21.15.89:0/2346847228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557063d8db40 con 0x557063d91d40 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80cffff640 1 --2- 172.21.15.89:0/2346847228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d88cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80d4816640 1 --2- 172.21.15.89:0/2346847228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d9a600 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f80c800ae10 tx=0x7f80c8019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c2b614237741ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80c5ffb640 1 -- 172.21.15.89:0/2346847228 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c80223f0 con 0x557063d91d40 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80c5ffb640 1 -- 172.21.15.89:0/2346847228 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c8022590 con 0x557063d91d40 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.533+0000 7f80c5ffb640 1 -- 172.21.15.89:0/2346847228 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c80073e0 con 0x557063d91d40 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.534+0000 7f80d8652740 1 -- 172.21.15.89:0/2346847228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 msgr2=0x557063d9a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.534+0000 7f80d8652740 1 --2- 172.21.15.89:0/2346847228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d9a600 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f80c800ae10 tx=0x7f80c8019260 comp rx=0 tx=0).stop 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.534+0000 7f80d8652740 1 -- 172.21.15.89:0/2346847228 shutdown_connections 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.534+0000 7f80d8652740 1 --2- 172.21.15.89:0/2346847228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d91d40 0x557063d9a600 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.534+0000 7f80d8652740 1 --2- 172.21.15.89:0/2346847228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d9a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.534+0000 7f80d8652740 1 --2- 172.21.15.89:0/2346847228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d86b60 0x557063d88cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.534+0000 7f80d8652740 1 -- 172.21.15.89:0/2346847228 >> 172.21.15.89:0/2346847228 conn(0x557063d830f0 msgr2=0x557063d885b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.534+0000 7f80d8652740 1 -- 172.21.15.89:0/2346847228 shutdown_connections 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.534+0000 7f80d8652740 1 -- 172.21.15.89:0/2346847228 wait complete. 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.534+0000 7f80d8652740 1 -- start start 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80d8652740 1 Processor -- start 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80d8652740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d86b60 0x557063d99ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80d8652740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d9a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80d8652740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d91d40 0x557063d96cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80d8652740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01b0270 con 0x557063d86b60 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d86b60 0x557063d99ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80d8652740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01afba0 con 0x557063d91d40 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80d8652740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f80d01acde0 con 0x557063d87480 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d86b60 0x557063d99ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55254/0 (socket says 172.21.15.89:55254) 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80cffff640 1 -- 172.21.15.89:0/1264104740 learned_addr learned my addr 172.21.15.89:0/1264104740 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:52.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80d8652740 10 client.0 _ll_register_callbacks cb 0x7f80c8009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80c7fff640 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d9a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.535+0000 7f80d4816640 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d91d40 0x557063d96cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80cffff640 1 -- 172.21.15.89:0/1264104740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 msgr2=0x557063d9a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80cffff640 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d9a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80cffff640 1 -- 172.21.15.89:0/1264104740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d91d40 msgr2=0x557063d96cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80cffff640 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d91d40 0x557063d96cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80cffff640 1 -- 172.21.15.89:0/1264104740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557063d973a0 con 0x557063d86b60 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80c7fff640 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d9a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80cffff640 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d86b60 0x557063d99ec0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f80c00cd150 tx=0x7f80c000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cec2041b0c37866 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80af7fe640 1 -- 172.21.15.89:0/1264104740 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c0020040 con 0x557063d86b60 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80af7fe640 1 -- 172.21.15.89:0/1264104740 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f80c00982e0 con 0x557063d86b60 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80d8652740 1 -- 172.21.15.89:0/1264104740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80c8022590 con 0x557063d86b60 2024-09-15T23:25:52.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80af7fe640 1 -- 172.21.15.89:0/1264104740 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f80c009bc70 con 0x557063d86b60 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80d8652740 10 client.16092 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.536+0000 7f80d8652740 1 -- 172.21.15.89:0/1264104740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557063e15900 con 0x557063d86b60 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.537+0000 7f80af7fe640 1 -- 172.21.15.89:0/1264104740 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c8022590 con 0x557063d86b60 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.537+0000 7f80af7fe640 1 -- 172.21.15.89:0/1264104740 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f80c009ab30 con 0x557063d86b60 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.537+0000 7f80d8652740 10 client.16092 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.537+0000 7f80d8652740 10 client.16092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.537+0000 7f80d8652740 20 client.16092 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.537+0000 7f80d8652740 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557063e15f80 0x557063e183a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.537+0000 7f80d8652740 4 client.16092 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.537+0000 7f80d8652740 1 -- 172.21.15.89:0/1264104740 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557063cf3d00 con 0x557063e15f80 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.537+0000 7f80c7fff640 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557063e15f80 0x557063e183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.538+0000 7f80af7fe640 10 client.16092 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.538+0000 7f80c7fff640 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557063e15f80 0x557063e183a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8e53a439f17230a server_cookie=33d91c4f963f95f6 in_seq=0 out_seq=0 2024-09-15T23:25:52.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80af7fe640 1 -- 172.21.15.89:0/1264104740 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557063cf3d00 con 0x557063e15f80 2024-09-15T23:25:52.563 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:25:52.563 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:25:52.563 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "a6fb18a9-bb4b-44d6-b475-c8addf565b22", 2024-09-15T23:25:52.563 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:25:52.563 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:25:52.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80af7fe640 10 client.16092 handle_command_reply: tid=0 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80d8652740 1 client.16092 shutdown 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80d8652740 1 -- 172.21.15.89:0/1264104740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d86b60 msgr2=0x557063d99ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80d8652740 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d86b60 0x557063d99ec0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f80c00cd150 tx=0x7f80c000bf60 comp rx=0 tx=0).stop 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80d8652740 1 -- 172.21.15.89:0/1264104740 shutdown_connections 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80d8652740 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557063e15f80 0x557063e183a0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80d8652740 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557063d91d40 0x557063d96cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80d8652740 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557063d87480 0x557063d9a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80af7fe640 0 client.16092 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80d8652740 1 --2- 172.21.15.89:0/1264104740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557063d86b60 0x557063d99ec0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.539+0000 7f80d8652740 1 -- 172.21.15.89:0/1264104740 >> 172.21.15.89:0/1264104740 conn(0x557063d830f0 msgr2=0x557063d8f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.540+0000 7f80d8652740 1 -- 172.21.15.89:0/1264104740 shutdown_connections 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.540+0000 7f80d8652740 1 -- 172.21.15.89:0/1264104740 wait complete. 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.540+0000 7f80d8652740 20 client.16092 trim_cache size 0 max 16384 2024-09-15T23:25:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 -- 172.21.15.89:0/2163183138 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f80bc0a3240 msgr2=0x7f80bc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f80bc0a3240 0x7f80bc0a5700 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80c800a4e0 tx=0x7f80c800c040 comp rx=0 tx=0).stop 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 -- 172.21.15.89:0/2163183138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d0102d30 msgr2=0x7f80d01986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d0102d30 0x7f80d01986a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f80c000d950 tx=0x7f80c000de10 comp rx=0 tx=0).stop 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 -- 172.21.15.89:0/2163183138 shutdown_connections 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f80bc0a3240 0x7f80bc0a5700 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80d01082a0 0x7f80d019cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f80d01036e0 0x7f80d0198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 --2- 172.21.15.89:0/2163183138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80d0102d30 0x7f80d01986a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 -- 172.21.15.89:0/2163183138 >> 172.21.15.89:0/2163183138 conn(0x7f80d00fc7d0 msgr2=0x7f80d0106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 -- 172.21.15.89:0/2163183138 shutdown_connections 2024-09-15T23:25:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:52.541+0000 7f80d6279640 1 -- 172.21.15.89:0/2163183138 wait complete. 2024-09-15T23:25:53.331 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:25:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:53 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:25:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:53 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:53 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:25:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:53 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:53.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:53 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:25:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:53 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:25:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:53 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:53 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:25:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:53 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:53.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:53 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:25:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:53 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:25:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:53 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:53 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:25:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:53 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:53 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:25:53.577 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:54.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.081+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/4287249083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c0074590 msgr2=0x7fe3c0074970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:54.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.081+0000 7fe3c7cc9640 1 --2- 172.21.15.89:0/4287249083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c0074590 0x7fe3c0074970 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009900 tx=0x7fe3bc02a480 comp rx=0 tx=0).stop 2024-09-15T23:25:54.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.081+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/4287249083 shutdown_connections 2024-09-15T23:25:54.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.081+0000 7fe3c7cc9640 1 --2- 172.21.15.89:0/4287249083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c0112b90 0x7fe3c0114f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.081+0000 7fe3c7cc9640 1 --2- 172.21.15.89:0/4287249083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c0074eb0 0x7fe3c0112650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.081+0000 7fe3c7cc9640 1 --2- 172.21.15.89:0/4287249083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c0074590 0x7fe3c0074970 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.081+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/4287249083 >> 172.21.15.89:0/4287249083 conn(0x7fe3c00fd8c0 msgr2=0x7fe3c00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.081+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/4287249083 shutdown_connections 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.081+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/4287249083 wait complete. 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c7cc9640 1 Processor -- start 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c7cc9640 1 -- start start 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c7cc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c0074590 0x7fe3c01a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c7cc9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c0074eb0 0x7fe3c01a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c7cc9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c0112b90 0x7fe3c019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c7cc9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c0118600 con 0x7fe3c0112b90 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c7cc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c0118480 con 0x7fe3c0074590 2024-09-15T23:25:54.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c7cc9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c0118780 con 0x7fe3c0074eb0 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c6266640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c0112b90 0x7fe3c019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c6266640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c0112b90 0x7fe3c019e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55280/0 (socket says 172.21.15.89:55280) 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c6266640 1 -- 172.21.15.89:0/120811471 learned_addr learned my addr 172.21.15.89:0/120811471 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c5264640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c0074eb0 0x7fe3c01a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c0074590 0x7fe3c01a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c6266640 1 -- 172.21.15.89:0/120811471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c0074eb0 msgr2=0x7fe3c01a4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c6266640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c0074eb0 0x7fe3c01a4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c6266640 1 -- 172.21.15.89:0/120811471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c0074590 msgr2=0x7fe3c01a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c6266640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c0074590 0x7fe3c01a47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.082+0000 7fe3c6266640 1 -- 172.21.15.89:0/120811471 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c019f0c0 con 0x7fe3c0112b90 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.083+0000 7fe3c6266640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c0112b90 0x7fe3c019e990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800b9e0 tx=0x7fe3b800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.083+0000 7fe3b6ffd640 1 -- 172.21.15.89:0/120811471 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8019070 con 0x7fe3c0112b90 2024-09-15T23:25:54.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.083+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/120811471 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c019f3b0 con 0x7fe3c0112b90 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.083+0000 7fe3b6ffd640 1 -- 172.21.15.89:0/120811471 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b8014440 con 0x7fe3c0112b90 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.083+0000 7fe3b6ffd640 1 -- 172.21.15.89:0/120811471 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80125c0 con 0x7fe3c0112b90 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.083+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/120811471 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c01ab6a0 con 0x7fe3c0112b90 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.085+0000 7fe3b6ffd640 1 -- 172.21.15.89:0/120811471 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe3b801c070 con 0x7fe3c0112b90 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.085+0000 7fe3ca0a2740 1 Processor -- start 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.085+0000 7fe3ca0a2740 1 -- start start 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe82434a0 0x55cbe8243880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe824e680 0x55cbe8250a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cbe81a92e0 con 0x55cbe8243dc0 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cbe8150ce0 con 0x55cbe82434a0 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cbe81a52a0 con 0x55cbe824e680 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3b6ffd640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3ac0a32c0 0x7fe3ac0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5a65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5a65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55296/0 (socket says 172.21.15.89:55296) 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5a65640 1 -- 172.21.15.89:0/1806895208 learned_addr learned my addr 172.21.15.89:0/1806895208 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5264640 1 --2- 172.21.15.89:0/1806895208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe82434a0 0x55cbe8243880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5264640 1 --2- 172.21.15.89:0/1806895208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe824e680 0x55cbe8250a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c6266640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3ac0a32c0 0x7fe3ac0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3b6ffd640 1 -- 172.21.15.89:0/120811471 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe3b80cf210 con 0x7fe3c0112b90 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5a65640 1 -- 172.21.15.89:0/1806895208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe824e680 msgr2=0x55cbe8250a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/1806895208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe824e680 0x55cbe8250a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5a65640 1 -- 172.21.15.89:0/1806895208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe82434a0 msgr2=0x55cbe8243880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/1806895208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe82434a0 0x55cbe8243880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5a65640 1 -- 172.21.15.89:0/1806895208 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbe8135440 con 0x55cbe8243dc0 2024-09-15T23:25:54.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/1806895208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824e140 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc02a9a0 tx=0x7fe3bc032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc96783043030247 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:54.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.086+0000 7fe3c6266640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3ac0a32c0 0x7fe3ac0a5780 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe3b809b480 tx=0x7fe3b8065f30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:54.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3abfff640 1 -- 172.21.15.89:0/1806895208 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc03f070 con 0x55cbe8243dc0 2024-09-15T23:25:54.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3abfff640 1 -- 172.21.15.89:0/1806895208 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3bc0317b0 con 0x55cbe8243dc0 2024-09-15T23:25:54.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3abfff640 1 -- 172.21.15.89:0/1806895208 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc035070 con 0x55cbe8243dc0 2024-09-15T23:25:54.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/1806895208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 msgr2=0x55cbe824e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:54.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/1806895208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824e140 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc02a9a0 tx=0x7fe3bc032d60 comp rx=0 tx=0).stop 2024-09-15T23:25:54.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/1806895208 shutdown_connections 2024-09-15T23:25:54.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/1806895208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe824e680 0x55cbe8250a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/1806895208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824e140 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/1806895208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe82434a0 0x55cbe8243880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/1806895208 >> 172.21.15.89:0/1806895208 conn(0x55cbe823fa30 msgr2=0x55cbe823fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.087+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/1806895208 shutdown_connections 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/1806895208 wait complete. 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3ca0a2740 1 -- start start 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3ca0a2740 1 Processor -- start 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe824df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe824e680 0x55cbe824b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cbe8251250 con 0x55cbe8243dc0 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cbe8164050 con 0x55cbe824e680 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cbe82513d0 con 0x55cbe82434a0 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3ca0a2740 10 client.0 _ll_register_callbacks cb 0x55cbe81a4160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3c5a65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3c5a65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55312/0 (socket says 172.21.15.89:55312) 2024-09-15T23:25:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.088+0000 7fe3c5a65640 1 -- 172.21.15.89:0/416423736 learned_addr learned my addr 172.21.15.89:0/416423736 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3c5264640 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe824e680 0x55cbe824b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3c5264640 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe824df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3c5a65640 1 -- 172.21.15.89:0/416423736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 msgr2=0x55cbe824df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe824df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3c5a65640 1 -- 172.21.15.89:0/416423736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe824e680 msgr2=0x55cbe824b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe824e680 0x55cbe824b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3c5a65640 1 -- 172.21.15.89:0/416423736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbe7f1a3c0 con 0x55cbe8243dc0 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3c5264640 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe824e680 0x55cbe824b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824b230 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009690 tx=0x7fe3bc02ab00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1379332e2639aaec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3a97fa640 1 -- 172.21.15.89:0/416423736 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc04a020 con 0x55cbe8243dc0 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3a97fa640 1 -- 172.21.15.89:0/416423736 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe3bc004450 con 0x55cbe8243dc0 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/416423736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3bc0317b0 con 0x55cbe8243dc0 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3a97fa640 1 -- 172.21.15.89:0/416423736 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe3bc055050 con 0x55cbe8243dc0 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3ca0a2740 10 client.16104 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/416423736 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cbe825cbf0 con 0x55cbe8243dc0 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3a97fa640 1 -- 172.21.15.89:0/416423736 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3bc0317b0 con 0x55cbe8243dc0 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.089+0000 7fe3a97fa640 1 -- 172.21.15.89:0/416423736 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe3bc03b7b0 con 0x55cbe8243dc0 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.090+0000 7fe3ca0a2740 10 client.16104 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:54.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.090+0000 7fe3ca0a2740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.090+0000 7fe3ca0a2740 20 client.16104 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.090+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbe825df40 0x55cbe8260330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.090+0000 7fe3ca0a2740 4 client.16104 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.090+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/416423736 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbe81b6850 con 0x55cbe825df40 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.090+0000 7fe3c5264640 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbe825df40 0x55cbe8260330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.090+0000 7fe3a97fa640 10 client.16104 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.090+0000 7fe3c5264640 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbe825df40 0x55cbe8260330 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2da1ffbf16b35c6 server_cookie=3085e6dc8bd8464f in_seq=0 out_seq=0 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3a97fa640 1 -- 172.21.15.89:0/416423736 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cbe81b6850 con 0x55cbe825df40 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3a97fa640 10 client.16104 handle_command_reply: tid=0 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 client.16104 shutdown 2024-09-15T23:25:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/416423736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 msgr2=0x55cbe824b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824b230 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc009690 tx=0x7fe3bc02ab00 comp rx=0 tx=0).stop 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/416423736 shutdown_connections 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbe825df40 0x55cbe8260330 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe824e680 0x55cbe824b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe824b230 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/416423736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe824df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3a97fa640 0 client.16104 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/416423736 >> 172.21.15.89:0/416423736 conn(0x55cbe823fa30 msgr2=0x55cbe823eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/416423736 shutdown_connections 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/416423736 wait complete. 2024-09-15T23:25:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.091+0000 7fe3ca0a2740 20 client.16104 trim_cache size 0 max 16384 2024-09-15T23:25:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3ca0a2740 1 Processor -- start 2024-09-15T23:25:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3ca0a2740 1 -- start start 2024-09-15T23:25:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe823fa30 0x55cbe8263350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe8263890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe8248490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b5550 con 0x55cbe8243dc0 2024-09-15T23:25:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b7090 con 0x55cbe823fa30 2024-09-15T23:25:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b5c20 con 0x55cbe82434a0 2024-09-15T23:25:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3c5264640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe8248490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3c5264640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe8248490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55316/0 (socket says 172.21.15.89:55316) 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3c5264640 1 -- 172.21.15.89:0/3774714253 learned_addr learned my addr 172.21.15.89:0/3774714253 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3c5264640 1 --2- 172.21.15.89:0/3774714253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe823fa30 0x55cbe8263350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.108+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/3774714253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe8263890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3c5264640 1 -- 172.21.15.89:0/3774714253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 msgr2=0x55cbe8263890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3c5264640 1 --2- 172.21.15.89:0/3774714253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe8263890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3c5264640 1 -- 172.21.15.89:0/3774714253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe823fa30 msgr2=0x55cbe8263350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3c5264640 1 --2- 172.21.15.89:0/3774714253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe823fa30 0x55cbe8263350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3c5264640 1 -- 172.21.15.89:0/3774714253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbe824e2f0 con 0x55cbe8243dc0 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3c5264640 1 --2- 172.21.15.89:0/3774714253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe8248490 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe3c01a0180 tx=0x7fe3b000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca2d72f232f86fd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3ab7fe640 1 -- 172.21.15.89:0/3774714253 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0003e10 con 0x55cbe8243dc0 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3ab7fe640 1 -- 172.21.15.89:0/3774714253 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b0026070 con 0x55cbe8243dc0 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3ab7fe640 1 -- 172.21.15.89:0/3774714253 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0004e10 con 0x55cbe8243dc0 2024-09-15T23:25:54.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/3774714253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 msgr2=0x55cbe8248490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.109+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/3774714253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe8248490 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe3c01a0180 tx=0x7fe3b000b390 comp rx=0 tx=0).stop 2024-09-15T23:25:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.110+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/3774714253 shutdown_connections 2024-09-15T23:25:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.110+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/3774714253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe8248490 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.110+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/3774714253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe8263890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.110+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/3774714253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe823fa30 0x55cbe8263350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.110+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/3774714253 >> 172.21.15.89:0/3774714253 conn(0x55cbe8246c20 msgr2=0x55cbe824d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.110+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/3774714253 shutdown_connections 2024-09-15T23:25:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.110+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/3774714253 wait complete. 2024-09-15T23:25:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.110+0000 7fe3ca0a2740 1 -- start start 2024-09-15T23:25:54.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.110+0000 7fe3ca0a2740 1 Processor -- start 2024-09-15T23:25:54.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.110+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe823fa30 0x55cbe8261a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe8261f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3ca0a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe82c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b69c0 con 0x55cbe8243dc0 2024-09-15T23:25:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b62f0 con 0x55cbe823fa30 2024-09-15T23:25:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3ca0a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c01b3530 con 0x55cbe82434a0 2024-09-15T23:25:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3ca0a2740 10 client.0 _ll_register_callbacks cb 0x7fe3b000ba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5264640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe82c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5264640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe82c18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55330/0 (socket says 172.21.15.89:55330) 2024-09-15T23:25:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5264640 1 -- 172.21.15.89:0/2291094899 learned_addr learned my addr 172.21.15.89:0/2291094899 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5264640 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe823fa30 0x55cbe8261a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe8261f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5264640 1 -- 172.21.15.89:0/2291094899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 msgr2=0x55cbe8261f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5264640 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe8261f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5264640 1 -- 172.21.15.89:0/2291094899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe823fa30 msgr2=0x55cbe8261a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5264640 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe823fa30 0x55cbe8261a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5264640 1 -- 172.21.15.89:0/2291094899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbe82c1f60 con 0x55cbe8243dc0 2024-09-15T23:25:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3c5264640 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe82c18c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0004420 tx=0x7fe3b0005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25cc15d1dc292d20 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3a97fa640 1 -- 172.21.15.89:0/2291094899 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0012720 con 0x55cbe8243dc0 2024-09-15T23:25:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/2291094899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3b0026070 con 0x55cbe8243dc0 2024-09-15T23:25:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.111+0000 7fe3a97fa640 1 -- 172.21.15.89:0/2291094899 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe3b00128c0 con 0x55cbe8243dc0 2024-09-15T23:25:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3a97fa640 1 -- 172.21.15.89:0/2291094899 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe3b0042050 con 0x55cbe8243dc0 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3ca0a2740 10 client.16110 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/2291094899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cbe82c9300 con 0x55cbe8243dc0 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3a97fa640 1 -- 172.21.15.89:0/2291094899 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b0026070 con 0x55cbe8243dc0 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3a97fa640 1 -- 172.21.15.89:0/2291094899 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe3b001e070 con 0x55cbe8243dc0 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3ca0a2740 10 client.16110 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3ca0a2740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3ca0a2740 20 client.16110 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbe824e680 0x55cbe82cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3ca0a2740 4 client.16110 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/2291094899 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cbe81b0d60 con 0x55cbe824e680 2024-09-15T23:25:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.112+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbe824e680 0x55cbe82cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3c5a65640 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbe824e680 0x55cbe82cb890 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b768ae99159d8c8c server_cookie=959e11d9a2d5da62 in_seq=0 out_seq=0 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3a97fa640 10 client.16110 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3a97fa640 1 -- 172.21.15.89:0/2291094899 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cbe81b0d60 con 0x55cbe824e680 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3a97fa640 10 client.16110 handle_command_reply: tid=0 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 client.16110 shutdown 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/2291094899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 msgr2=0x55cbe82c18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe82c18c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0004420 tx=0x7fe3b0005f70 comp rx=0 tx=0).stop 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/2291094899 shutdown_connections 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbe824e680 0x55cbe82cb890 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbe8243dc0 0x55cbe82c18c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3a97fa640 0 client.16110 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:25:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbe82434a0 0x55cbe8261f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 --2- 172.21.15.89:0/2291094899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbe823fa30 0x55cbe8261a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/2291094899 >> 172.21.15.89:0/2291094899 conn(0x55cbe8246c20 msgr2=0x55cbe8244e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/2291094899 shutdown_connections 2024-09-15T23:25:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 1 -- 172.21.15.89:0/2291094899 wait complete. 2024-09-15T23:25:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.113+0000 7fe3ca0a2740 20 client.16110 trim_cache size 0 max 16384 2024-09-15T23:25:54.133 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:25:54.133 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:25:54.133 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:25:54.133 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:25:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/120811471 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3ac0a32c0 msgr2=0x7fe3ac0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3ac0a32c0 0x7fe3ac0a5780 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe3b809b480 tx=0x7fe3b8065f30 comp rx=0 tx=0).stop 2024-09-15T23:25:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/120811471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c0112b90 msgr2=0x7fe3c019e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c0112b90 0x7fe3c019e990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800b9e0 tx=0x7fe3b800bea0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/120811471 shutdown_connections 2024-09-15T23:25:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3ac0a32c0 0x7fe3ac0a5780 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3c0112b90 0x7fe3c019e990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c0074eb0 0x7fe3c01a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 --2- 172.21.15.89:0/120811471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3c0074590 0x7fe3c01a47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/120811471 >> 172.21.15.89:0/120811471 conn(0x7fe3c00fd8c0 msgr2=0x7fe3c00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/120811471 shutdown_connections 2024-09-15T23:25:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:54.114+0000 7fe3c7cc9640 1 -- 172.21.15.89:0/120811471 wait complete. 2024-09-15T23:25:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:54 smithi089 ceph-mon[30933]: pgmap v406: 129 pgs: 129 active+clean; 763 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 KiB/s wr, 16 op/s 2024-09-15T23:25:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:54 smithi120 ceph-mon[35250]: pgmap v406: 129 pgs: 129 active+clean; 763 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 KiB/s wr, 16 op/s 2024-09-15T23:25:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:54 smithi174 ceph-mon[35586]: pgmap v406: 129 pgs: 129 active+clean; 763 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.1 KiB/s wr, 16 op/s 2024-09-15T23:25:54.844 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:25:54.844 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:25:55.088 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:55.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.588+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/1316504228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a84106a80 msgr2=0x7f2a84106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:55.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.588+0000 7f2a8c6bf640 1 --2- 172.21.15.89:0/1316504228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a84106a80 0x7f2a84106e60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009930 tx=0x7f2a7402a420 comp rx=0 tx=0).stop 2024-09-15T23:25:55.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.588+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/1316504228 shutdown_connections 2024-09-15T23:25:55.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.588+0000 7f2a8c6bf640 1 --2- 172.21.15.89:0/1316504228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a84101420 0x7f2a8410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:55.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.588+0000 7f2a8c6bf640 1 --2- 172.21.15.89:0/1316504228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a84100a80 0x7f2a84100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:55.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.588+0000 7f2a8c6bf640 1 --2- 172.21.15.89:0/1316504228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a84106a80 0x7f2a84106e60 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:55.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.588+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/1316504228 >> 172.21.15.89:0/1316504228 conn(0x7f2a840fc7a0 msgr2=0x7f2a840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:55.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.588+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/1316504228 shutdown_connections 2024-09-15T23:25:55.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.588+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/1316504228 wait complete. 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8c6bf640 1 Processor -- start 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8c6bf640 1 -- start start 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8c6bf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a84100a80 0x7f2a84198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8c6bf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a84101420 0x7f2a84198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8c6bf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a84106a80 0x7f2a8419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8c6bf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84110ea0 con 0x7f2a84100a80 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8c6bf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84110d20 con 0x7f2a84101420 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8c6bf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a84111020 con 0x7f2a84106a80 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8a45b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a84100a80 0x7f2a84198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8a45b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a84100a80 0x7f2a84198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36590/0 (socket says 172.21.15.89:36590) 2024-09-15T23:25:55.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8a45b640 1 -- 172.21.15.89:0/75126833 learned_addr learned my addr 172.21.15.89:0/75126833 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a89c5a640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a84101420 0x7f2a84198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8a45b640 1 -- 172.21.15.89:0/75126833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a84106a80 msgr2=0x7f2a8419cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8ac5c640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a84106a80 0x7f2a8419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8a45b640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a84106a80 0x7f2a8419cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8a45b640 1 -- 172.21.15.89:0/75126833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a84101420 msgr2=0x7f2a84198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8a45b640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a84101420 0x7f2a84198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8a45b640 1 -- 172.21.15.89:0/75126833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8419d4d0 con 0x7f2a84100a80 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a89c5a640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a84101420 0x7f2a84198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.589+0000 7f2a8a45b640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a84100a80 0x7f2a84198520 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009a50 tx=0x7f2a74004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.590+0000 7f2a7b7fe640 1 -- 172.21.15.89:0/75126833 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7403f030 con 0x7f2a84100a80 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.590+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/75126833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a8419d760 con 0x7f2a84100a80 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.590+0000 7f2a7b7fe640 1 -- 172.21.15.89:0/75126833 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a74003ce0 con 0x7f2a84100a80 2024-09-15T23:25:55.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.590+0000 7f2a7b7fe640 1 -- 172.21.15.89:0/75126833 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a74035070 con 0x7f2a84100a80 2024-09-15T23:25:55.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.590+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/75126833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a841a50a0 con 0x7f2a84100a80 2024-09-15T23:25:55.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.591+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/75126833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a841021c0 con 0x7f2a84100a80 2024-09-15T23:25:55.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.592+0000 7f2a7b7fe640 1 -- 172.21.15.89:0/75126833 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2a74042030 con 0x7f2a84100a80 2024-09-15T23:25:55.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.592+0000 7f2a7b7fe640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a700a3240 0x7f2a700a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:55.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.593+0000 7f2a89c5a640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a700a3240 0x7f2a700a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:55.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.593+0000 7f2a7b7fe640 1 -- 172.21.15.89:0/75126833 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2a740f07f0 con 0x7f2a84100a80 2024-09-15T23:25:55.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.593+0000 7f2a89c5a640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a700a3240 0x7f2a700a5700 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2a80006fd0 tx=0x7f2a80008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:55.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.596+0000 7f2a7b7fe640 1 -- 172.21.15.89:0/75126833 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2a740bd120 con 0x7f2a84100a80 2024-09-15T23:25:55.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.797+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/75126833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a84100ee0 con 0x7f2a84100a80 2024-09-15T23:25:56.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.797+0000 7f2a7b7fe640 1 -- 172.21.15.89:0/75126833 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f2a740bca20 con 0x7f2a84100a80 2024-09-15T23:25:56.074 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:25:56.074 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/75126833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a700a3240 msgr2=0x7f2a700a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a700a3240 0x7f2a700a5700 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2a80006fd0 tx=0x7f2a80008040 comp rx=0 tx=0).stop 2024-09-15T23:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/75126833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a84100a80 msgr2=0x7f2a84198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a84100a80 0x7f2a84198520 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009a50 tx=0x7f2a74004270 comp rx=0 tx=0).stop 2024-09-15T23:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/75126833 shutdown_connections 2024-09-15T23:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a700a3240 0x7f2a700a5700 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a84106a80 0x7f2a8419cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:56.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a84101420 0x7f2a84198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:56.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 --2- 172.21.15.89:0/75126833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a84100a80 0x7f2a84198520 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:56.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/75126833 >> 172.21.15.89:0/75126833 conn(0x7f2a840fc7a0 msgr2=0x7f2a8406aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:56.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/75126833 shutdown_connections 2024-09-15T23:25:56.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:55.800+0000 7f2a8c6bf640 1 -- 172.21.15.89:0/75126833 wait complete. 2024-09-15T23:25:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:56 smithi120 ceph-mon[35250]: pgmap v407: 129 pgs: 129 active+clean; 763 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.1 KiB/s wr, 14 op/s 2024-09-15T23:25:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:56 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/75126833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:56.483 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:56 smithi089 ceph-mon[30933]: pgmap v407: 129 pgs: 129 active+clean; 763 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.1 KiB/s wr, 14 op/s 2024-09-15T23:25:56.483 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:56 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/75126833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:56 smithi174 ceph-mon[35586]: pgmap v407: 129 pgs: 129 active+clean; 763 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.1 KiB/s wr, 14 op/s 2024-09-15T23:25:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:56 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/75126833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:56.511 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:25:56.768 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:57.180 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:25:57.180 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.181 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:57 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 -- 172.21.15.89:0/581334853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10ac101df0 msgr2=0x7f10ac1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:57.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 --2- 172.21.15.89:0/581334853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10ac101df0 0x7f10ac1021d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009900 tx=0x7f10a002a410 comp rx=0 tx=0).stop 2024-09-15T23:25:57.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 -- 172.21.15.89:0/581334853 shutdown_connections 2024-09-15T23:25:57.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 --2- 172.21.15.89:0/581334853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10ac106f20 0x7f10ac10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 --2- 172.21.15.89:0/581334853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ac1027a0 0x7f10ac1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 --2- 172.21.15.89:0/581334853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10ac101df0 0x7f10ac1021d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 -- 172.21.15.89:0/581334853 >> 172.21.15.89:0/581334853 conn(0x7f10ac0fd8f0 msgr2=0x7f10ac0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 -- 172.21.15.89:0/581334853 shutdown_connections 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 -- 172.21.15.89:0/581334853 wait complete. 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 Processor -- start 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.263+0000 7f10b1a77640 1 -- start start 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10b1a77640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10ac101df0 0x7f10ac198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10b1a77640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10ac1027a0 0x7f10ac198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10b1a77640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ac106f20 0x7f10ac19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10b1a77640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac111f80 con 0x7f10ac101df0 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10b1a77640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac111e00 con 0x7f10ac106f20 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10b1a77640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac112100 con 0x7f10ac1027a0 2024-09-15T23:25:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10ab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10ac101df0 0x7f10ac198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10ab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10ac101df0 0x7f10ac198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36600/0 (socket says 172.21.15.89:36600) 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10ab7fe640 1 -- 172.21.15.89:0/1242716618 learned_addr learned my addr 172.21.15.89:0/1242716618 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10abfff640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ac106f20 0x7f10ac19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10aaffd640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10ac1027a0 0x7f10ac198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10ab7fe640 1 -- 172.21.15.89:0/1242716618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10ac1027a0 msgr2=0x7f10ac198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10ab7fe640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10ac1027a0 0x7f10ac198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10ab7fe640 1 -- 172.21.15.89:0/1242716618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ac106f20 msgr2=0x7f10ac19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10ab7fe640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ac106f20 0x7f10ac19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10ab7fe640 1 -- 172.21.15.89:0/1242716618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac19d400 con 0x7f10ac101df0 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10abfff640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ac106f20 0x7f10ac19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.264+0000 7f10ab7fe640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10ac101df0 0x7f10ac198450 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f10a0006010 tx=0x7f10a00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.265+0000 7f10a8ff9640 1 -- 172.21.15.89:0/1242716618 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a003f030 con 0x7f10ac101df0 2024-09-15T23:25:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.265+0000 7f10b1a77640 1 -- 172.21.15.89:0/1242716618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ac19d690 con 0x7f10ac101df0 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.265+0000 7f10a8ff9640 1 -- 172.21.15.89:0/1242716618 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a0003ce0 con 0x7f10ac101df0 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.265+0000 7f10a8ff9640 1 -- 172.21.15.89:0/1242716618 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0035070 con 0x7f10ac101df0 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.265+0000 7f10b1a77640 1 -- 172.21.15.89:0/1242716618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ac1a4fd0 con 0x7f10ac101df0 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.267+0000 7f10b3e50740 1 Processor -- start 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.267+0000 7f10b3e50740 1 -- start start 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.267+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb353b40 0x5651eb352090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.267+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb3525d0 0x5651eb352a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.267+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb363b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.267+0000 7f10b3e50740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5651eb2b82f0 con 0x5651eb353b40 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.267+0000 7f10b3e50740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5651eb25fcf0 con 0x5651eb361780 2024-09-15T23:25:57.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.267+0000 7f10b3e50740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5651eb2b42b0 con 0x5651eb3525d0 2024-09-15T23:25:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10abfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb353b40 0x5651eb352090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10abfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb353b40 0x5651eb352090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36602/0 (socket says 172.21.15.89:36602) 2024-09-15T23:25:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10abfff640 1 -- 172.21.15.89:0/1551288488 learned_addr learned my addr 172.21.15.89:0/1551288488 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10a8ff9640 1 -- 172.21.15.89:0/1242716618 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f10a0042030 con 0x7f10ac101df0 2024-09-15T23:25:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10abfff640 1 -- 172.21.15.89:0/1551288488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb3525d0 msgr2=0x5651eb352a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:25:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10aaffd640 1 --2- 172.21.15.89:0/1551288488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb363b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10abfff640 1 --2- 172.21.15.89:0/1551288488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb3525d0 0x5651eb352a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10abfff640 1 -- 172.21.15.89:0/1551288488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 msgr2=0x5651eb363b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:57.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10abfff640 1 --2- 172.21.15.89:0/1551288488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb363b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10abfff640 1 -- 172.21.15.89:0/1551288488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651eb2443a0 con 0x5651eb353b40 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10aaffd640 1 --2- 172.21.15.89:0/1551288488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb363b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10abfff640 1 --2- 172.21.15.89:0/1551288488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb353b40 0x5651eb352090 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f10a4010710 tx=0x7f10a4010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a51c8b7555f9a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f1091ffb640 1 -- 172.21.15.89:0/1551288488 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a400e8d0 con 0x5651eb353b40 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f1091ffb640 1 -- 172.21.15.89:0/1551288488 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a4012450 con 0x5651eb353b40 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f1091ffb640 1 -- 172.21.15.89:0/1551288488 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a401c670 con 0x5651eb353b40 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.268+0000 7f10a8ff9640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f10940a3240 0x7f10940a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10ab7fe640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f10940a3240 0x7f10940a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10a8ff9640 1 -- 172.21.15.89:0/1242716618 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f10a00eff10 con 0x7f10ac101df0 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10ab7fe640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f10940a3240 0x7f10940a5700 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f10a00c4940 tx=0x7f10a0086d00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10b3e50740 1 -- 172.21.15.89:0/1551288488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb353b40 msgr2=0x5651eb352090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10b3e50740 1 --2- 172.21.15.89:0/1551288488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb353b40 0x5651eb352090 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f10a4010710 tx=0x7f10a4010bd0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10b3e50740 1 -- 172.21.15.89:0/1551288488 shutdown_connections 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10b3e50740 1 --2- 172.21.15.89:0/1551288488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb363b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10b3e50740 1 --2- 172.21.15.89:0/1551288488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb3525d0 0x5651eb352a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10b3e50740 1 --2- 172.21.15.89:0/1551288488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb353b40 0x5651eb352090 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10b3e50740 1 -- 172.21.15.89:0/1551288488 >> 172.21.15.89:0/1551288488 conn(0x5651eb31b310 msgr2=0x5651eb34ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10b3e50740 1 -- 172.21.15.89:0/1551288488 shutdown_connections 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.269+0000 7f10b3e50740 1 -- 172.21.15.89:0/1551288488 wait complete. 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.270+0000 7f10b3e50740 1 -- start start 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.270+0000 7f10b3e50740 1 Processor -- start 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.270+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb3525d0 0x5651eb3611d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.270+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb353b40 0x5651eb35e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.270+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb361780 0x5651eb35ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.270+0000 7f10b3e50740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5651eb364480 con 0x5651eb361780 2024-09-15T23:25:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.270+0000 7f10b3e50740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5651eb273060 con 0x5651eb353b40 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.270+0000 7f10b3e50740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5651eb364600 con 0x5651eb3525d0 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.270+0000 7f10b3e50740 10 client.0 _ll_register_callbacks cb 0x5651eafc68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10abfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb361780 0x5651eb35ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10abfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb361780 0x5651eb35ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36608/0 (socket says 172.21.15.89:36608) 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10abfff640 1 -- 172.21.15.89:0/2751947235 learned_addr learned my addr 172.21.15.89:0/2751947235 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10abfff640 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb3525d0 0x5651eb3611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10aaffd640 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb353b40 0x5651eb35e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10abfff640 1 -- 172.21.15.89:0/2751947235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb3525d0 msgr2=0x5651eb3611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10abfff640 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb3525d0 0x5651eb3611d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10abfff640 1 -- 172.21.15.89:0/2751947235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb353b40 msgr2=0x5651eb35e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10abfff640 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb353b40 0x5651eb35e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10abfff640 1 -- 172.21.15.89:0/2751947235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651eb0293c0 con 0x5651eb361780 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10aaffd640 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb353b40 0x5651eb35e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10abfff640 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb361780 0x5651eb35ec10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f10a40049d0 tx=0x7f10a4012e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69131f14a8702d73 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f108affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f108affd640 1 -- 172.21.15.89:0/2751947235 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a400eb80 con 0x5651eb361780 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f108affd640 1 -- 172.21.15.89:0/2751947235 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f10a400ed20 con 0x5651eb361780 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10b3e50740 1 -- 172.21.15.89:0/2751947235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10a4012450 con 0x5651eb361780 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f108affd640 1 -- 172.21.15.89:0/2751947235 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f10a402b420 con 0x5651eb361780 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10b3e50740 10 client.16128 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.271+0000 7f10b3e50740 1 -- 172.21.15.89:0/2751947235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5651eb36ff00 con 0x5651eb361780 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.272+0000 7f108affd640 1 -- 172.21.15.89:0/2751947235 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a4012450 con 0x5651eb361780 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.272+0000 7f108affd640 1 -- 172.21.15.89:0/2751947235 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f10a400f040 con 0x5651eb361780 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.272+0000 7f10b3e50740 10 client.16128 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.272+0000 7f10b3e50740 10 client.16128 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.272+0000 7f10b3e50740 20 client.16128 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.272+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651eb371300 0x5651eb3736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.272+0000 7f10b3e50740 4 client.16128 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.272+0000 7f10b3e50740 1 -- 172.21.15.89:0/2751947235 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651eb2c5860 con 0x5651eb371300 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.272+0000 7f10aaffd640 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651eb371300 0x5651eb3736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.272+0000 7f108affd640 10 client.16128 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.273+0000 7f10aaffd640 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651eb371300 0x5651eb3736f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6856fc0d2c213607 server_cookie=9e46ef7a8d0f9874 in_seq=0 out_seq=0 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.273+0000 7f108affd640 1 -- 172.21.15.89:0/2751947235 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5651eb2c5860 con 0x5651eb371300 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.273+0000 7f108affd640 10 client.16128 handle_command_reply: tid=0 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.273+0000 7f10b3e50740 1 client.16128 shutdown 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.273+0000 7f10b3e50740 1 -- 172.21.15.89:0/2751947235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb361780 msgr2=0x5651eb35ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.273+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb361780 0x5651eb35ec10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f10a40049d0 tx=0x7f10a4012e70 comp rx=0 tx=0).stop 2024-09-15T23:25:57.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.274+0000 7f10b3e50740 1 -- 172.21.15.89:0/2751947235 shutdown_connections 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.274+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651eb371300 0x5651eb3736f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.274+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb361780 0x5651eb35ec10 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.274+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb353b40 0x5651eb35e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.274+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2751947235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb3525d0 0x5651eb3611d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.274+0000 7f10b3e50740 1 -- 172.21.15.89:0/2751947235 >> 172.21.15.89:0/2751947235 conn(0x5651eb31b310 msgr2=0x5651eb354960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.274+0000 7f108affd640 0 client.16128 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.274+0000 7f10b3e50740 1 -- 172.21.15.89:0/2751947235 shutdown_connections 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.274+0000 7f10b3e50740 1 -- 172.21.15.89:0/2751947235 wait complete. 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.274+0000 7f10b3e50740 20 client.16128 trim_cache size 0 max 16384 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.286+0000 7f10b3e50740 1 Processor -- start 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10b3e50740 1 -- start start 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb35dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb353b40 0x5651eb34e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb34ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10b3e50740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac1acc90 con 0x5651eb3525d0 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10b3e50740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac1ae7d0 con 0x5651eb361780 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10b3e50740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac1ad360 con 0x5651eb353b40 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10abfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb35dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10abfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb35dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36616/0 (socket says 172.21.15.89:36616) 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10abfff640 1 -- 172.21.15.89:0/1405947198 learned_addr learned my addr 172.21.15.89:0/1405947198 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10aaffd640 1 --2- 172.21.15.89:0/1405947198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb353b40 0x5651eb34e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10abfff640 1 -- 172.21.15.89:0/1405947198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb353b40 msgr2=0x5651eb34e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10abfff640 1 --2- 172.21.15.89:0/1405947198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb353b40 0x5651eb34e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10abfff640 1 -- 172.21.15.89:0/1405947198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 msgr2=0x5651eb34ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10abfff640 1 --2- 172.21.15.89:0/1405947198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb34ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10abfff640 1 -- 172.21.15.89:0/1405947198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651eb355120 con 0x5651eb3525d0 2024-09-15T23:25:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10abfff640 1 --2- 172.21.15.89:0/1405947198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb35dd40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009770 tx=0x7f10a4012ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf8851795f59796 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10917fa640 1 -- 172.21.15.89:0/1405947198 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a4012450 con 0x5651eb3525d0 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10917fa640 1 -- 172.21.15.89:0/1405947198 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a400e6c0 con 0x5651eb3525d0 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.287+0000 7f10917fa640 1 -- 172.21.15.89:0/1405947198 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a402b5c0 con 0x5651eb3525d0 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.288+0000 7f10b3e50740 1 -- 172.21.15.89:0/1405947198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 msgr2=0x5651eb35dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.288+0000 7f10b3e50740 1 --2- 172.21.15.89:0/1405947198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb35dd40 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009770 tx=0x7f10a4012ee0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.288+0000 7f10b3e50740 1 -- 172.21.15.89:0/1405947198 shutdown_connections 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.288+0000 7f10b3e50740 1 --2- 172.21.15.89:0/1405947198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb34ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.288+0000 7f10b3e50740 1 --2- 172.21.15.89:0/1405947198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb353b40 0x5651eb34e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.288+0000 7f10b3e50740 1 --2- 172.21.15.89:0/1405947198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb35dd40 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.288+0000 7f10b3e50740 1 -- 172.21.15.89:0/1405947198 >> 172.21.15.89:0/1405947198 conn(0x5651eb31b310 msgr2=0x5651eb359720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.288+0000 7f10b3e50740 1 -- 172.21.15.89:0/1405947198 shutdown_connections 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.288+0000 7f10b3e50740 1 -- 172.21.15.89:0/1405947198 wait complete. 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.288+0000 7f10b3e50740 1 -- start start 2024-09-15T23:25:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10b3e50740 1 Processor -- start 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb373c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb353b40 0x5651eb374170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10b3e50740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb3d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10b3e50740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac1ae100 con 0x5651eb3525d0 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10b3e50740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac1ada30 con 0x5651eb361780 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10b3e50740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac1aac70 con 0x5651eb353b40 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10b3e50740 10 client.0 _ll_register_callbacks cb 0x7f10a4016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10abfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb373c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10abfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb373c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36632/0 (socket says 172.21.15.89:36632) 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10abfff640 1 -- 172.21.15.89:0/2051962865 learned_addr learned my addr 172.21.15.89:0/2051962865 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10aaffd640 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb353b40 0x5651eb374170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10abfff640 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb3d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10abfff640 1 -- 172.21.15.89:0/2051962865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb353b40 msgr2=0x5651eb374170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10abfff640 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb353b40 0x5651eb374170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10abfff640 1 -- 172.21.15.89:0/2051962865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 msgr2=0x5651eb3d2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10abfff640 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb3d2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.289+0000 7f10abfff640 1 -- 172.21.15.89:0/2051962865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651eb3d3290 con 0x5651eb3525d0 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f10abfff640 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb373c30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f10a4005d70 tx=0x7f10a4012aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=309cfa94619b2197 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f108affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f108affd640 1 -- 172.21.15.89:0/2051962865 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a400eae0 con 0x5651eb3525d0 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f10b3e50740 1 -- 172.21.15.89:0/2051962865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10a400e6c0 con 0x5651eb3525d0 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f108affd640 1 -- 172.21.15.89:0/2051962865 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f10a403d040 con 0x5651eb3525d0 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f108affd640 1 -- 172.21.15.89:0/2051962865 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f10a40049a0 con 0x5651eb3525d0 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f10b3e50740 10 client.16134 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f10b3e50740 1 -- 172.21.15.89:0/2051962865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5651eb3db800 con 0x5651eb3525d0 2024-09-15T23:25:57.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f108affd640 1 -- 172.21.15.89:0/2051962865 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a400e6c0 con 0x5651eb3525d0 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f108affd640 1 -- 172.21.15.89:0/2051962865 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f10a4040020 con 0x5651eb3525d0 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f10b3e50740 10 client.16134 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f10b3e50740 10 client.16134 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f10b3e50740 20 client.16134 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651eb3dbc20 0x5651eb3de010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f10b3e50740 4 client.16134 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.290+0000 7f10b3e50740 1 -- 172.21.15.89:0/2051962865 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5651eb2bfd70 con 0x5651eb3dbc20 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10aaffd640 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651eb3dbc20 0x5651eb3de010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f108affd640 10 client.16134 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10aaffd640 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651eb3dbc20 0x5651eb3de010 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4add47fd5fcedb2b server_cookie=2e0fd2fee98c2d42 in_seq=0 out_seq=0 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f108affd640 1 -- 172.21.15.89:0/2051962865 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5651eb2bfd70 con 0x5651eb3dbc20 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f108affd640 10 client.16134 handle_command_reply: tid=0 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10b3e50740 1 client.16134 shutdown 2024-09-15T23:25:57.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10b3e50740 1 -- 172.21.15.89:0/2051962865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 msgr2=0x5651eb373c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb373c30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f10a4005d70 tx=0x7f10a4012aa0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10b3e50740 1 -- 172.21.15.89:0/2051962865 shutdown_connections 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651eb3dbc20 0x5651eb3de010 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f108affd640 0 client.16134 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651eb361780 0x5651eb3d2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651eb353b40 0x5651eb374170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10b3e50740 1 --2- 172.21.15.89:0/2051962865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651eb3525d0 0x5651eb373c30 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10b3e50740 1 -- 172.21.15.89:0/2051962865 >> 172.21.15.89:0/2051962865 conn(0x5651eb31b310 msgr2=0x5651eb35fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.291+0000 7f10b3e50740 1 -- 172.21.15.89:0/2051962865 shutdown_connections 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b3e50740 1 -- 172.21.15.89:0/2051962865 wait complete. 2024-09-15T23:25:57.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b3e50740 20 client.16134 trim_cache size 0 max 16384 2024-09-15T23:25:57.311 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:57.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 -- 172.21.15.89:0/1242716618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f10940a3240 msgr2=0x7f10940a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f10940a3240 0x7f10940a5700 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f10a00c4940 tx=0x7f10a0086d00 comp rx=0 tx=0).stop 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 -- 172.21.15.89:0/1242716618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10ac101df0 msgr2=0x7f10ac198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10ac101df0 0x7f10ac198450 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f10a0006010 tx=0x7f10a00042f0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 -- 172.21.15.89:0/1242716618 shutdown_connections 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f10940a3240 0x7f10940a5700 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10ac106f20 0x7f10ac19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10ac1027a0 0x7f10ac198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 --2- 172.21.15.89:0/1242716618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10ac101df0 0x7f10ac198450 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 -- 172.21.15.89:0/1242716618 >> 172.21.15.89:0/1242716618 conn(0x7f10ac0fd8f0 msgr2=0x7f10ac0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 -- 172.21.15.89:0/1242716618 shutdown_connections 2024-09-15T23:25:57.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:57.292+0000 7f10b1a77640 1 -- 172.21.15.89:0/1242716618 wait complete. 2024-09-15T23:25:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:25:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:57 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:25:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:57 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:58.038 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:25:58.286 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:58 smithi089 ceph-mon[30933]: pgmap v408: 129 pgs: 129 active+clean; 811 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 20 op/s 2024-09-15T23:25:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:25:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:25:58.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:58 smithi120 ceph-mon[35250]: pgmap v408: 129 pgs: 129 active+clean; 811 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 20 op/s 2024-09-15T23:25:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:25:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:25:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:58.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:58 smithi174 ceph-mon[35586]: pgmap v408: 129 pgs: 129 active+clean; 811 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 20 op/s 2024-09-15T23:25:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:25:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:25:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.773+0000 7f30f1ad9640 1 -- 172.21.15.89:0/3276886266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec103660 msgr2=0x7f30ec10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.773+0000 7f30f1ad9640 1 --2- 172.21.15.89:0/3276886266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec103660 0x7f30ec10bb50 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009900 tx=0x7f30dc02a480 comp rx=0 tx=0).stop 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.773+0000 7f30f1ad9640 1 -- 172.21.15.89:0/3276886266 shutdown_connections 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.773+0000 7f30f1ad9640 1 --2- 172.21.15.89:0/3276886266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec10c170 0x7f30ec10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.773+0000 7f30f1ad9640 1 --2- 172.21.15.89:0/3276886266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec103660 0x7f30ec10bb50 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.773+0000 7f30f1ad9640 1 --2- 172.21.15.89:0/3276886266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec102d40 0x7f30ec103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.773+0000 7f30f1ad9640 1 -- 172.21.15.89:0/3276886266 >> 172.21.15.89:0/3276886266 conn(0x7f30ec0fc7a0 msgr2=0x7f30ec0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.773+0000 7f30f1ad9640 1 -- 172.21.15.89:0/3276886266 shutdown_connections 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.773+0000 7f30f1ad9640 1 -- 172.21.15.89:0/3276886266 wait complete. 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.773+0000 7f30f1ad9640 1 Processor -- start 2024-09-15T23:25:58.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30f1ad9640 1 -- start start 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30f1ad9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec102d40 0x7f30ec198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30f1ad9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec103660 0x7f30ec1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30f1ad9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec10c170 0x7f30ec19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30f1ad9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec111f70 con 0x7f30ec10c170 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30f1ad9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec111df0 con 0x7f30ec103660 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30f1ad9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1120f0 con 0x7f30ec102d40 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec10c170 0x7f30ec19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec10c170 0x7f30ec19cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36644/0 (socket says 172.21.15.89:36644) 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30ebfff640 1 -- 172.21.15.89:0/1134240951 learned_addr learned my addr 172.21.15.89:0/1134240951 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec102d40 0x7f30ec198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30eaffd640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec103660 0x7f30ec1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30ebfff640 1 -- 172.21.15.89:0/1134240951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec102d40 msgr2=0x7f30ec198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30ebfff640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec102d40 0x7f30ec198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30ebfff640 1 -- 172.21.15.89:0/1134240951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec103660 msgr2=0x7f30ec1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30ebfff640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec103660 0x7f30ec1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30ebfff640 1 -- 172.21.15.89:0/1134240951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec19d420 con 0x7f30ec10c170 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.774+0000 7f30ebfff640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec10c170 0x7f30ec19cd40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f30e400cad0 tx=0x7f30e400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.775+0000 7f30e8ff9640 1 -- 172.21.15.89:0/1134240951 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e400ae20 con 0x7f30ec10c170 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.775+0000 7f30e8ff9640 1 -- 172.21.15.89:0/1134240951 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30e4005020 con 0x7f30ec10c170 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.775+0000 7f30f1ad9640 1 -- 172.21.15.89:0/1134240951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ec19d710 con 0x7f30ec10c170 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.775+0000 7f30e8ff9640 1 -- 172.21.15.89:0/1134240951 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e400f6f0 con 0x7f30ec10c170 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.775+0000 7f30f1ad9640 1 -- 172.21.15.89:0/1134240951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ec1a4ff0 con 0x7f30ec10c170 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.777+0000 7f30e8ff9640 1 -- 172.21.15.89:0/1134240951 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f30e40051d0 con 0x7f30ec10c170 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.777+0000 7f30f3eb2740 1 Processor -- start 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.777+0000 7f30f3eb2740 1 -- start start 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 0x564b460d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460d2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460df760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b460382f0 con 0x564b460d25d0 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b45fdfcf0 con 0x564b460d3b40 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b460342b0 con 0x564b460dd370 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460d2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460d2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36652/0 (socket says 172.21.15.89:36652) 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eb7fe640 1 -- 172.21.15.89:0/2939980864 learned_addr learned my addr 172.21.15.89:0/2939980864 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eaffd640 1 --2- 172.21.15.89:0/2939980864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460df760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eaffd640 1 --2- 172.21.15.89:0/2939980864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 0x564b460d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30e8ff9640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d80a3240 0x7f30d80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eb7fe640 1 -- 172.21.15.89:0/2939980864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 msgr2=0x564b460df760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/2939980864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460df760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eb7fe640 1 -- 172.21.15.89:0/2939980864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 msgr2=0x564b460d2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/2939980864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 0x564b460d2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eb7fe640 1 -- 172.21.15.89:0/2939980864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b45fc43a0 con 0x564b460d25d0 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eaffd640 1 --2- 172.21.15.89:0/2939980864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 0x564b460d2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30ebfff640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d80a3240 0x7f30d80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/2939980864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460d2a10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f30d400b760 tx=0x7f30d400bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5543083372941771 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30e8ff9640 1 -- 172.21.15.89:0/1134240951 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f30e40cfd30 con 0x7f30ec10c170 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.778+0000 7f30e1ffb640 1 -- 172.21.15.89:0/2939980864 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4003dd0 con 0x564b460d25d0 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30e1ffb640 1 -- 172.21.15.89:0/2939980864 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d4015070 con 0x564b460d25d0 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30e1ffb640 1 -- 172.21.15.89:0/2939980864 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4011440 con 0x564b460d25d0 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2939980864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 msgr2=0x564b460d2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30ebfff640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d80a3240 0x7f30d80a5700 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f30e4098930 tx=0x7f30e406bbd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/2939980864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460d2a10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f30d400b760 tx=0x7f30d400bc20 comp rx=0 tx=0).stop 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2939980864 shutdown_connections 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/2939980864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460df760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/2939980864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460d2a10 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/2939980864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 0x564b460d2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2939980864 >> 172.21.15.89:0/2939980864 conn(0x564b4609b310 msgr2=0x564b460cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:58.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2939980864 shutdown_connections 2024-09-15T23:25:58.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.779+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2939980864 wait complete. 2024-09-15T23:25:58.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30f3eb2740 1 -- start start 2024-09-15T23:25:58.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30f3eb2740 1 Processor -- start 2024-09-15T23:25:58.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460dcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460d3b40 0x564b460da000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460dd370 0x564b460da5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b460e0070 con 0x564b460d25d0 2024-09-15T23:25:58.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b45ff3060 con 0x564b460dd370 2024-09-15T23:25:58.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b460e01f0 con 0x564b460d3b40 2024-09-15T23:25:58.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30f3eb2740 10 client.0 _ll_register_callbacks cb 0x564b45d468c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:58.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30eaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460dcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30eaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460dcd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36662/0 (socket says 172.21.15.89:36662) 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30eaffd640 1 -- 172.21.15.89:0/931790887 learned_addr learned my addr 172.21.15.89:0/931790887 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.780+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460d3b40 0x564b460da000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30eaffd640 1 -- 172.21.15.89:0/931790887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460d3b40 msgr2=0x564b460da000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30eaffd640 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460d3b40 0x564b460da000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30eaffd640 1 -- 172.21.15.89:0/931790887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460dd370 msgr2=0x564b460da5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30eaffd640 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460dd370 0x564b460da5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30eaffd640 1 -- 172.21.15.89:0/931790887 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b45da93c0 con 0x564b460d25d0 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30eaffd640 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460dcd50 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009900 tx=0x7f30dc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74bb50d998bfdca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30caffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30caffd640 1 -- 172.21.15.89:0/931790887 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc035040 con 0x564b460d25d0 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30f3eb2740 1 -- 172.21.15.89:0/931790887 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30d4015070 con 0x564b460d25d0 2024-09-15T23:25:58.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30caffd640 1 -- 172.21.15.89:0/931790887 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f30dc004450 con 0x564b460d25d0 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30caffd640 1 -- 172.21.15.89:0/931790887 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f30dc051050 con 0x564b460d25d0 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30f3eb2740 10 client.16146 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30f3eb2740 1 -- 172.21.15.89:0/931790887 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b460ebb40 con 0x564b460d25d0 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30caffd640 1 -- 172.21.15.89:0/931790887 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d4015070 con 0x564b460d25d0 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.781+0000 7f30caffd640 1 -- 172.21.15.89:0/931790887 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f30dc03a040 con 0x564b460d25d0 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.782+0000 7f30f3eb2740 10 client.16146 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.782+0000 7f30f3eb2740 10 client.16146 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.782+0000 7f30f3eb2740 20 client.16146 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.782+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x564b460ecf50 0x564b460ef340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.782+0000 7f30f3eb2740 4 client.16146 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.782+0000 7f30f3eb2740 1 -- 172.21.15.89:0/931790887 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b46045860 con 0x564b460ecf50 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.782+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x564b460ecf50 0x564b460ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.782+0000 7f30caffd640 10 client.16146 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:58.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.782+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x564b460ecf50 0x564b460ef340 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4d2af248c8667835 server_cookie=e0fe4400a64fa291 in_seq=0 out_seq=0 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30caffd640 1 -- 172.21.15.89:0/931790887 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564b46045860 con 0x564b460ecf50 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30caffd640 10 client.16146 handle_command_reply: tid=0 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 client.16146 shutdown 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 -- 172.21.15.89:0/931790887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 msgr2=0x564b460dcd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460dcd50 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009900 tx=0x7f30dc005dc0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 -- 172.21.15.89:0/931790887 shutdown_connections 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x564b460ecf50 0x564b460ef340 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460dd370 0x564b460da5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30caffd640 0 client.16146 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460d3b40 0x564b460da000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/931790887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460dcd50 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 -- 172.21.15.89:0/931790887 >> 172.21.15.89:0/931790887 conn(0x564b4609b310 msgr2=0x564b460d4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:58.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 -- 172.21.15.89:0/931790887 shutdown_connections 2024-09-15T23:25:58.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.783+0000 7f30f3eb2740 1 -- 172.21.15.89:0/931790887 wait complete. 2024-09-15T23:25:58.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.784+0000 7f30f3eb2740 20 client.16146 trim_cache size 0 max 16384 2024-09-15T23:25:58.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.802+0000 7f30f3eb2740 1 Processor -- start 2024-09-15T23:25:58.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.802+0000 7f30f3eb2740 1 -- start start 2024-09-15T23:25:58.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.802+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d25d0 0x564b460cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.802+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d3b40 0x564b460d5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.802+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460d5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.802+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1aef60 con 0x564b460d3b40 2024-09-15T23:25:58.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.802+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1b0aa0 con 0x564b460d25d0 2024-09-15T23:25:58.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.802+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1af630 con 0x564b460dd370 2024-09-15T23:25:58.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.802+0000 7f30eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d3b40 0x564b460d5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d3b40 0x564b460d5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36664/0 (socket says 172.21.15.89:36664) 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30eb7fe640 1 -- 172.21.15.89:0/1314268989 learned_addr learned my addr 172.21.15.89:0/1314268989 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30eaffd640 1 --2- 172.21.15.89:0/1314268989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460d5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30eaffd640 1 --2- 172.21.15.89:0/1314268989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d25d0 0x564b460cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30eb7fe640 1 -- 172.21.15.89:0/1314268989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 msgr2=0x564b460d5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/1314268989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460d5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30eb7fe640 1 -- 172.21.15.89:0/1314268989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d25d0 msgr2=0x564b460cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/1314268989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d25d0 0x564b460cdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30eb7fe640 1 -- 172.21.15.89:0/1314268989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b460ef6b0 con 0x564b460d3b40 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/1314268989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d3b40 0x564b460d5120 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x564b460cfac0 tx=0x7f30d4004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb28740a6d164b6a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30e17fa640 1 -- 172.21.15.89:0/1314268989 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4003dd0 con 0x564b460d3b40 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30e17fa640 1 -- 172.21.15.89:0/1314268989 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d4015030 con 0x564b460d3b40 2024-09-15T23:25:58.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.803+0000 7f30e17fa640 1 -- 172.21.15.89:0/1314268989 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4019680 con 0x564b460d3b40 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.804+0000 7f30f3eb2740 1 -- 172.21.15.89:0/1314268989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d3b40 msgr2=0x564b460d5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.804+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/1314268989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d3b40 0x564b460d5120 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x564b460cfac0 tx=0x7f30d4004a80 comp rx=0 tx=0).stop 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.804+0000 7f30f3eb2740 1 -- 172.21.15.89:0/1314268989 shutdown_connections 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.804+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/1314268989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460d5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.804+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/1314268989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d3b40 0x564b460d5120 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.804+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/1314268989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d25d0 0x564b460cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.804+0000 7f30f3eb2740 1 -- 172.21.15.89:0/1314268989 >> 172.21.15.89:0/1314268989 conn(0x564b4609b310 msgr2=0x564b460cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.804+0000 7f30f3eb2740 1 -- 172.21.15.89:0/1314268989 shutdown_connections 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.804+0000 7f30f3eb2740 1 -- 172.21.15.89:0/1314268989 wait complete. 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.804+0000 7f30f3eb2740 1 -- start start 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30f3eb2740 1 Processor -- start 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 0x564b460d6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30f3eb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460d7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1b03d0 con 0x564b460d25d0 2024-09-15T23:25:58.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1afd00 con 0x564b460d3b40 2024-09-15T23:25:58.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30f3eb2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1acf40 con 0x564b460dd370 2024-09-15T23:25:58.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30f3eb2740 10 client.0 _ll_register_callbacks cb 0x7f30dc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:58.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460e5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36672/0 (socket says 172.21.15.89:36672) 2024-09-15T23:25:58.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eaffd640 1 -- 172.21.15.89:0/2900506622 learned_addr learned my addr 172.21.15.89:0/2900506622 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:25:58.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eaffd640 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460d7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 0x564b460d6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eaffd640 1 -- 172.21.15.89:0/2900506622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 msgr2=0x564b460d7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eaffd640 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460d7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eaffd640 1 -- 172.21.15.89:0/2900506622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 msgr2=0x564b460d6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eaffd640 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 0x564b460d6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eaffd640 1 -- 172.21.15.89:0/2900506622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b460d7bc0 con 0x564b460d25d0 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30eaffd640 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460e5cf0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f30dc033000 tx=0x7f30dc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306e9756176975d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.805+0000 7f30caffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30caffd640 1 -- 172.21.15.89:0/2900506622 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc05d060 con 0x564b460d25d0 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30caffd640 1 -- 172.21.15.89:0/2900506622 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f30dc004460 con 0x564b460d25d0 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2900506622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30d4015030 con 0x564b460d25d0 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30caffd640 1 -- 172.21.15.89:0/2900506622 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f30dc064050 con 0x564b460d25d0 2024-09-15T23:25:58.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30f3eb2740 10 client.16152 fetch_fsmap learned FSMap version 29 2024-09-15T23:25:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2900506622 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b46157cb0 con 0x564b460d25d0 2024-09-15T23:25:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30caffd640 1 -- 172.21.15.89:0/2900506622 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d4015030 con 0x564b460d25d0 2024-09-15T23:25:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30caffd640 1 -- 172.21.15.89:0/2900506622 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f30dc003b40 con 0x564b460d25d0 2024-09-15T23:25:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30f3eb2740 10 client.16152 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:25:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30f3eb2740 10 client.16152 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:25:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30f3eb2740 20 client.16152 populate_metadata read hostname 'smithi089' 2024-09-15T23:25:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x564b460ecf50 0x564b4615a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30f3eb2740 4 client.16152 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2900506622 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564b4603fd70 con 0x564b460ecf50 2024-09-15T23:25:58.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.806+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x564b460ecf50 0x564b4615a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:58.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30eb7fe640 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x564b460ecf50 0x564b4615a440 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3db285cc30145d8 server_cookie=17873f738290cbac in_seq=0 out_seq=0 2024-09-15T23:25:58.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30caffd640 10 client.16152 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:58.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30caffd640 1 -- 172.21.15.89:0/2900506622 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564b4603fd70 con 0x564b460ecf50 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30caffd640 10 client.16152 handle_command_reply: tid=0 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 client.16152 shutdown 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2900506622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 msgr2=0x564b460e5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460e5cf0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f30dc033000 tx=0x7f30dc005dc0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2900506622 shutdown_connections 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x564b460ecf50 0x564b4615a440 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b460dd370 0x564b460d7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b460d3b40 0x564b460d6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 --2- 172.21.15.89:0/2900506622 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b460d25d0 0x564b460e5cf0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2900506622 >> 172.21.15.89:0/2900506622 conn(0x564b4609b310 msgr2=0x564b460cdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30caffd640 0 client.16152 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2900506622 shutdown_connections 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 1 -- 172.21.15.89:0/2900506622 wait complete. 2024-09-15T23:25:58.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.807+0000 7f30f3eb2740 20 client.16152 trim_cache size 0 max 16384 2024-09-15T23:25:58.816 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:25:58.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 -- 172.21.15.89:0/1134240951 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d80a3240 msgr2=0x7f30d80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d80a3240 0x7f30d80a5700 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f30e4098930 tx=0x7f30e406bbd0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 -- 172.21.15.89:0/1134240951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec10c170 msgr2=0x7f30ec19cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec10c170 0x7f30ec19cd40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f30e400cad0 tx=0x7f30e400cf90 comp rx=0 tx=0).stop 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 -- 172.21.15.89:0/1134240951 shutdown_connections 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d80a3240 0x7f30d80a5700 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec10c170 0x7f30ec19cd40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec103660 0x7f30ec1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 --2- 172.21.15.89:0/1134240951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec102d40 0x7f30ec198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 -- 172.21.15.89:0/1134240951 >> 172.21.15.89:0/1134240951 conn(0x7f30ec0fc7a0 msgr2=0x7f30ec0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 -- 172.21.15.89:0/1134240951 shutdown_connections 2024-09-15T23:25:58.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:25:58.808+0000 7f30f1ad9640 1 -- 172.21.15.89:0/1134240951 wait complete. 2024-09-15T23:25:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:25:59 smithi120 ceph-mon[35250]: pgmap v409: 129 pgs: 129 active+clean; 811 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:25:59.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:25:59 smithi174 ceph-mon[35586]: pgmap v409: 129 pgs: 129 active+clean; 811 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:25:59.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:25:59 smithi089 ceph-mon[30933]: pgmap v409: 129 pgs: 129 active+clean; 811 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:25:59.617 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:25:59.863 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:00.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.012+0000 7f1d0abd8640 1 -- 172.21.15.89:0/2272888190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d041001d0 msgr2=0x7f1d041005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.012+0000 7f1d0abd8640 1 --2- 172.21.15.89:0/2272888190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d041001d0 0x7f1d041005d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009ca0 tx=0x7f1cf402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.012+0000 7f1d0abd8640 1 -- 172.21.15.89:0/2272888190 shutdown_connections 2024-09-15T23:26:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.012+0000 7f1d0abd8640 1 --2- 172.21.15.89:0/2272888190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d04101d80 0x7f1d04108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.012+0000 7f1d0abd8640 1 --2- 172.21.15.89:0/2272888190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d041013c0 0x7f1d04101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.012+0000 7f1d0abd8640 1 --2- 172.21.15.89:0/2272888190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d041001d0 0x7f1d041005d0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.012+0000 7f1d0abd8640 1 -- 172.21.15.89:0/2272888190 >> 172.21.15.89:0/2272888190 conn(0x7f1d040fb970 msgr2=0x7f1d040fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.012+0000 7f1d0abd8640 1 -- 172.21.15.89:0/2272888190 shutdown_connections 2024-09-15T23:26:00.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.012+0000 7f1d0abd8640 1 -- 172.21.15.89:0/2272888190 wait complete. 2024-09-15T23:26:00.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d0abd8640 1 Processor -- start 2024-09-15T23:26:00.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d0abd8640 1 -- start start 2024-09-15T23:26:00.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d0abd8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d041001d0 0x7f1d04104160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d0abd8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d041013c0 0x7f1d041027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d0abd8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d04101d80 0x7f1d04102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d0abd8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0410b600 con 0x7f1d041013c0 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d0abd8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0410b480 con 0x7f1d04101d80 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d0abd8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0410b780 con 0x7f1d041001d0 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d03fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d041013c0 0x7f1d041027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d03fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d041013c0 0x7f1d041027a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36680/0 (socket says 172.21.15.89:36680) 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.013+0000 7f1d03fff640 1 -- 172.21.15.89:0/3831445986 learned_addr learned my addr 172.21.15.89:0/3831445986 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d08974640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d041001d0 0x7f1d04104160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d09175640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d04101d80 0x7f1d04102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d03fff640 1 -- 172.21.15.89:0/3831445986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d041001d0 msgr2=0x7f1d04104160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d03fff640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d041001d0 0x7f1d04104160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d03fff640 1 -- 172.21.15.89:0/3831445986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d04101d80 msgr2=0x7f1d04102ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d03fff640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d04101d80 0x7f1d04102ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d03fff640 1 -- 172.21.15.89:0/3831445986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d04103570 con 0x7f1d041013c0 2024-09-15T23:26:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d03fff640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d041013c0 0x7f1d041027a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c890 tx=0x7f1cec00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d01ffb640 1 -- 172.21.15.89:0/3831445986 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec004220 con 0x7f1d041013c0 2024-09-15T23:26:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d01ffb640 1 -- 172.21.15.89:0/3831445986 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cec0043c0 con 0x7f1d041013c0 2024-09-15T23:26:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d0abd8640 1 -- 172.21.15.89:0/3831445986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d041a1ef0 con 0x7f1d041013c0 2024-09-15T23:26:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d01ffb640 1 -- 172.21.15.89:0/3831445986 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0049e0 con 0x7f1d041013c0 2024-09-15T23:26:00.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.014+0000 7f1d0abd8640 1 -- 172.21.15.89:0/3831445986 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d041a2360 con 0x7f1d041013c0 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.016+0000 7f1d01ffb640 1 -- 172.21.15.89:0/3831445986 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1cec0079b0 con 0x7f1d041013c0 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d0da4c740 1 Processor -- start 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d01ffb640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cf00a3370 0x7f1cf00a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d0da4c740 1 -- start start 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d09175640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cf00a3370 0x7f1cf00a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d01ffb640 1 -- 172.21.15.89:0/3831445986 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1cec0cfd90 con 0x7f1d041013c0 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c8738da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87392e0 0x5581c873dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c873e370 0x5581c8740830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5581c869c2f0 con 0x5581c87389a0 2024-09-15T23:26:00.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5581c8644270 con 0x5581c873e370 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5581c86982b0 con 0x5581c87392e0 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d08974640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c8738da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d08974640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c8738da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36696/0 (socket says 172.21.15.89:36696) 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d08974640 1 -- 172.21.15.89:0/2647181620 learned_addr learned my addr 172.21.15.89:0/2647181620 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d09175640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cf00a3370 0x7f1cf00a5830 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc005fd0 tx=0x7f1cfc00f460 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d09175640 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c873e370 0x5581c8740830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.017+0000 7f1d03fff640 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87392e0 0x5581c873dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1d08974640 1 -- 172.21.15.89:0/2647181620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87392e0 msgr2=0x5581c873dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1d08974640 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87392e0 0x5581c873dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1d08974640 1 -- 172.21.15.89:0/2647181620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c873e370 msgr2=0x5581c8740830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1d08974640 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c873e370 0x5581c8740830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1d08974640 1 -- 172.21.15.89:0/2647181620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c8643340 con 0x5581c87389a0 2024-09-15T23:26:00.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1d09175640 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c873e370 0x5581c8740830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1d03fff640 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87392e0 0x5581c873dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1d08974640 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c8738da0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009ca0 tx=0x7f1cf4009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88fdc44f74fc573 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1cfaffd640 1 -- 172.21.15.89:0/2647181620 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4031650 con 0x5581c87389a0 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1cfaffd640 1 -- 172.21.15.89:0/2647181620 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cf4036050 con 0x5581c87389a0 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.018+0000 7f1cfaffd640 1 -- 172.21.15.89:0/2647181620 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4004480 con 0x5581c87389a0 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2647181620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 msgr2=0x5581c8738da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c8738da0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009ca0 tx=0x7f1cf4009930 comp rx=0 tx=0).stop 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2647181620 shutdown_connections 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c873e370 0x5581c8740830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87392e0 0x5581c873dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2647181620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c8738da0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2647181620 >> 172.21.15.89:0/2647181620 conn(0x5581c8734010 msgr2=0x5581c8734410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2647181620 shutdown_connections 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2647181620 wait complete. 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 -- start start 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 Processor -- start 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87389a0 0x5581c8742930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8742e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c873e370 0x5581c8749eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5581c8741160 con 0x5581c873e370 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5581c8625290 con 0x5581c87392e0 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5581c87412e0 con 0x5581c87389a0 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d0da4c740 10 client.0 _ll_register_callbacks cb 0x5581c8656140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.019+0000 7f1d09175640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c873e370 0x5581c8749eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d09175640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c873e370 0x5581c8749eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36698/0 (socket says 172.21.15.89:36698) 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d09175640 1 -- 172.21.15.89:0/3188158041 learned_addr learned my addr 172.21.15.89:0/3188158041 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:00.030 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d08974640 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87389a0 0x5581c8742930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d03fff640 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8742e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d09175640 1 -- 172.21.15.89:0/3188158041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87389a0 msgr2=0x5581c8742930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d09175640 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87389a0 0x5581c8742930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d09175640 1 -- 172.21.15.89:0/3188158041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 msgr2=0x5581c8742e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d09175640 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8742e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d09175640 1 -- 172.21.15.89:0/3188158041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581c83af210 con 0x5581c873e370 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d03fff640 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8742e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d09175640 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c873e370 0x5581c8749eb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc005d00 tx=0x7f1cfc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bfd54ccb9a0cf37 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1ce3fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1ce3fff640 1 -- 172.21.15.89:0/3188158041 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc015030 con 0x5581c873e370 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1ce3fff640 1 -- 172.21.15.89:0/3188158041 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1cfc01c630 con 0x5581c873e370 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d0da4c740 1 -- 172.21.15.89:0/3188158041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cf4036050 con 0x5581c873e370 2024-09-15T23:26:00.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1ce3fff640 1 -- 172.21.15.89:0/3188158041 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1cfc02e050 con 0x5581c873e370 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d0da4c740 10 client.16164 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1d0da4c740 1 -- 172.21.15.89:0/3188158041 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5581c847b620 con 0x5581c873e370 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.020+0000 7f1ce3fff640 1 -- 172.21.15.89:0/3188158041 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cf4036050 con 0x5581c873e370 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.021+0000 7f1ce3fff640 1 -- 172.21.15.89:0/3188158041 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1cfc00b040 con 0x5581c873e370 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.021+0000 7f1d0da4c740 10 client.16164 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.021+0000 7f1d0da4c740 10 client.16164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.021+0000 7f1d0da4c740 20 client.16164 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.021+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5581c874e5c0 0x5581c87509f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.021+0000 7f1d0da4c740 4 client.16164 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.021+0000 7f1d0da4c740 1 -- 172.21.15.89:0/3188158041 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581c86a9a20 con 0x5581c874e5c0 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.021+0000 7f1d08974640 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5581c874e5c0 0x5581c87509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.021+0000 7f1ce3fff640 10 client.16164 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:00.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.021+0000 7f1d08974640 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5581c874e5c0 0x5581c87509f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd9681fa52079192 server_cookie=e80d23b3cd14ec8c in_seq=0 out_seq=0 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1ce3fff640 1 -- 172.21.15.89:0/3188158041 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5581c86a9a20 con 0x5581c874e5c0 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1ce3fff640 10 client.16164 handle_command_reply: tid=0 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 client.16164 shutdown 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 -- 172.21.15.89:0/3188158041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c873e370 msgr2=0x5581c8749eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c873e370 0x5581c8749eb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc005d00 tx=0x7f1cfc004730 comp rx=0 tx=0).stop 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 -- 172.21.15.89:0/3188158041 shutdown_connections 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5581c874e5c0 0x5581c87509f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c873e370 0x5581c8749eb0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8742e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/3188158041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c87389a0 0x5581c8742930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1ce3fff640 0 client.16164 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 -- 172.21.15.89:0/3188158041 >> 172.21.15.89:0/3188158041 conn(0x5581c8734010 msgr2=0x5581c8735820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 -- 172.21.15.89:0/3188158041 shutdown_connections 2024-09-15T23:26:00.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 1 -- 172.21.15.89:0/3188158041 wait complete. 2024-09-15T23:26:00.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.022+0000 7f1d0da4c740 20 client.16164 trim_cache size 0 max 16384 2024-09-15T23:26:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d0da4c740 1 Processor -- start 2024-09-15T23:26:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d0da4c740 1 -- start start 2024-09-15T23:26:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87330b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8749e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c873e370 0x5581c87552f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1d041aca40 con 0x5581c87389a0 2024-09-15T23:26:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1d041ae580 con 0x5581c87392e0 2024-09-15T23:26:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1d041ad110 con 0x5581c873e370 2024-09-15T23:26:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d08974640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87330b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d08974640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87330b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36702/0 (socket says 172.21.15.89:36702) 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d08974640 1 -- 172.21.15.89:0/2673314396 learned_addr learned my addr 172.21.15.89:0/2673314396 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d03fff640 1 --2- 172.21.15.89:0/2673314396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8749e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.037+0000 7f1d09175640 1 --2- 172.21.15.89:0/2673314396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c873e370 0x5581c87552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.038+0000 7f1d08974640 1 -- 172.21.15.89:0/2673314396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c873e370 msgr2=0x5581c87552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.038+0000 7f1d08974640 1 --2- 172.21.15.89:0/2673314396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c873e370 0x5581c87552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.038+0000 7f1d08974640 1 -- 172.21.15.89:0/2673314396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 msgr2=0x5581c8749e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.038+0000 7f1d08974640 1 --2- 172.21.15.89:0/2673314396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8749e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.038+0000 7f1d08974640 1 -- 172.21.15.89:0/2673314396 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581c874cee0 con 0x5581c87389a0 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.038+0000 7f1d08974640 1 --2- 172.21.15.89:0/2673314396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87330b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4031ae0 tx=0x7f1cf4009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60bef989b110318 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.038+0000 7f1cfa7fc640 1 -- 172.21.15.89:0/2673314396 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4054070 con 0x5581c87389a0 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.038+0000 7f1cfa7fc640 1 -- 172.21.15.89:0/2673314396 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cf4036040 con 0x5581c87389a0 2024-09-15T23:26:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.038+0000 7f1cfa7fc640 1 -- 172.21.15.89:0/2673314396 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4004440 con 0x5581c87389a0 2024-09-15T23:26:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2673314396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 msgr2=0x5581c87330b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2673314396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87330b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4031ae0 tx=0x7f1cf4009a20 comp rx=0 tx=0).stop 2024-09-15T23:26:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2673314396 shutdown_connections 2024-09-15T23:26:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2673314396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c873e370 0x5581c87552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2673314396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8749e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2673314396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87330b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2673314396 >> 172.21.15.89:0/2673314396 conn(0x5581c8734010 msgr2=0x5581c87329a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2673314396 shutdown_connections 2024-09-15T23:26:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2673314396 wait complete. 2024-09-15T23:26:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 -- start start 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 Processor -- start 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87477b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8747cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c873e370 0x5581c8748cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1d041adeb0 con 0x5581c87389a0 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1d041ad7e0 con 0x5581c87392e0 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1d041aaa20 con 0x5581c873e370 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d0da4c740 10 client.0 _ll_register_callbacks cb 0x7f1cfc00d450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.039+0000 7f1d08974640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d08974640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87477b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36704/0 (socket says 172.21.15.89:36704) 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d08974640 1 -- 172.21.15.89:0/2639178798 learned_addr learned my addr 172.21.15.89:0/2639178798 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d09175640 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c873e370 0x5581c8748cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d03fff640 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8747cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d08974640 1 -- 172.21.15.89:0/2639178798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c873e370 msgr2=0x5581c8748cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d08974640 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c873e370 0x5581c8748cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d08974640 1 -- 172.21.15.89:0/2639178798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 msgr2=0x5581c8747cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d08974640 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8747cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d08974640 1 -- 172.21.15.89:0/2639178798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581c87492c0 con 0x5581c87389a0 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d03fff640 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8747cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d08974640 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87477b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009bf0 tx=0x7f1cf4031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8364f809261e9fec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1ce3fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1ce3fff640 1 -- 172.21.15.89:0/2639178798 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4054070 con 0x5581c87389a0 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2639178798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cf4036040 con 0x5581c87389a0 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1ce3fff640 1 -- 172.21.15.89:0/2639178798 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1cf4003b80 con 0x5581c87389a0 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1ce3fff640 1 -- 172.21.15.89:0/2639178798 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1cf4003d80 con 0x5581c87389a0 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d0da4c740 10 client.16170 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:00.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2639178798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5581c87b93f0 con 0x5581c87389a0 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.040+0000 7f1ce3fff640 1 -- 172.21.15.89:0/2639178798 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cf4036040 con 0x5581c87389a0 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.041+0000 7f1ce3fff640 1 -- 172.21.15.89:0/2639178798 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1cf4042070 con 0x5581c87389a0 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.041+0000 7f1d0da4c740 10 client.16170 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.041+0000 7f1d0da4c740 10 client.16170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.041+0000 7f1d0da4c740 20 client.16170 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.041+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5581c87b9a10 0x5581c87bbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.041+0000 7f1d0da4c740 4 client.16170 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.041+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2639178798 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5581c86a3d70 con 0x5581c87b9a10 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.041+0000 7f1d03fff640 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5581c87b9a10 0x5581c87bbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.041+0000 7f1ce3fff640 10 client.16170 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:00.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.041+0000 7f1d03fff640 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5581c87b9a10 0x5581c87bbe30 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b57ab6e9c11564fc server_cookie=313f7fa23aab80a4 in_seq=0 out_seq=0 2024-09-15T23:26:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1ce3fff640 1 -- 172.21.15.89:0/2639178798 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5581c86a3d70 con 0x5581c87b9a10 2024-09-15T23:26:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1ce3fff640 10 client.16170 handle_command_reply: tid=0 2024-09-15T23:26:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 client.16170 shutdown 2024-09-15T23:26:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2639178798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 msgr2=0x5581c87477b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87477b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4009bf0 tx=0x7f1cf4031ee0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2639178798 shutdown_connections 2024-09-15T23:26:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5581c87b9a10 0x5581c87bbe30 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5581c873e370 0x5581c8748cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1ce3fff640 0 client.16170 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5581c87392e0 0x5581c8747cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 --2- 172.21.15.89:0/2639178798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581c87389a0 0x5581c87477b0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2639178798 >> 172.21.15.89:0/2639178798 conn(0x5581c8734010 msgr2=0x5581c87329a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2639178798 shutdown_connections 2024-09-15T23:26:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 1 -- 172.21.15.89:0/2639178798 wait complete. 2024-09-15T23:26:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0da4c740 20 client.16170 trim_cache size 0 max 16384 2024-09-15T23:26:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:26:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:26:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:26:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:26:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 -- 172.21.15.89:0/3831445986 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cf00a3370 msgr2=0x7f1cf00a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cf00a3370 0x7f1cf00a5830 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc005fd0 tx=0x7f1cfc00f460 comp rx=0 tx=0).stop 2024-09-15T23:26:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 -- 172.21.15.89:0/3831445986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d041013c0 msgr2=0x7f1d041027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d041013c0 0x7f1d041027a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c890 tx=0x7f1cec00cd50 comp rx=0 tx=0).stop 2024-09-15T23:26:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 -- 172.21.15.89:0/3831445986 shutdown_connections 2024-09-15T23:26:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1cf00a3370 0x7f1cf00a5830 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d04101d80 0x7f1d04102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d041013c0 0x7f1d041027a0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 --2- 172.21.15.89:0/3831445986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d041001d0 0x7f1d04104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 -- 172.21.15.89:0/3831445986 >> 172.21.15.89:0/3831445986 conn(0x7f1d040fb970 msgr2=0x7f1d041068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 -- 172.21.15.89:0/3831445986 shutdown_connections 2024-09-15T23:26:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:00.055+0000 7f1d0abd8640 1 -- 172.21.15.89:0/3831445986 wait complete. 2024-09-15T23:26:00.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.365+0000 7f69cade0640 1 -- 172.21.15.89:0/1310719510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4100a80 msgr2=0x7f69c4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.365+0000 7f69cade0640 1 --2- 172.21.15.89:0/1310719510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4100a80 0x7f69c4100ee0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009930 tx=0x7f69b002a420 comp rx=0 tx=0).stop 2024-09-15T23:26:00.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.365+0000 7f69cade0640 1 -- 172.21.15.89:0/1310719510 shutdown_connections 2024-09-15T23:26:00.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.365+0000 7f69cade0640 1 --2- 172.21.15.89:0/1310719510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69c4101420 0x7f69c410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.365+0000 7f69cade0640 1 --2- 172.21.15.89:0/1310719510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4100a80 0x7f69c4100ee0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.365+0000 7f69cade0640 1 --2- 172.21.15.89:0/1310719510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69c4106a80 0x7f69c4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.365+0000 7f69cade0640 1 -- 172.21.15.89:0/1310719510 >> 172.21.15.89:0/1310719510 conn(0x7f69c40fc7a0 msgr2=0x7f69c40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.365+0000 7f69cade0640 1 -- 172.21.15.89:0/1310719510 shutdown_connections 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.365+0000 7f69cade0640 1 -- 172.21.15.89:0/1310719510 wait complete. 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69cade0640 1 Processor -- start 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69cade0640 1 -- start start 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69cade0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69c4100a80 0x7f69c41992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69cade0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69c4101420 0x7f69c4199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69cade0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4106a80 0x7f69c41a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69cade0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f69c4110e80 con 0x7f69c4106a80 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69cade0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69c4110d00 con 0x7f69c4101420 2024-09-15T23:26:00.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69cade0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f69c4111000 con 0x7f69c4100a80 2024-09-15T23:26:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69c937d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4106a80 0x7f69c41a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69c937d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4106a80 0x7f69c41a4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36718/0 (socket says 172.21.15.89:36718) 2024-09-15T23:26:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69c937d640 1 -- 172.21.15.89:0/798066822 learned_addr learned my addr 172.21.15.89:0/798066822 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69c3fff640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69c4101420 0x7f69c4199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69c8b7c640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69c4100a80 0x7f69c41992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69c937d640 1 -- 172.21.15.89:0/798066822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69c4100a80 msgr2=0x7f69c41992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69c937d640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69c4100a80 0x7f69c41992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69c937d640 1 -- 172.21.15.89:0/798066822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69c4101420 msgr2=0x7f69c4199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69c937d640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69c4101420 0x7f69c4199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.366+0000 7f69c937d640 1 -- 172.21.15.89:0/798066822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c41a4890 con 0x7f69c4106a80 2024-09-15T23:26:00.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.367+0000 7f69c937d640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4106a80 0x7f69c41a4350 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00eaa0 tx=0x7f69bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.367+0000 7f69c1ffb640 1 -- 172.21.15.89:0/798066822 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc00cdb0 con 0x7f69c4106a80 2024-09-15T23:26:00.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.367+0000 7f69c1ffb640 1 -- 172.21.15.89:0/798066822 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69bc005020 con 0x7f69c4106a80 2024-09-15T23:26:00.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.367+0000 7f69cade0640 1 -- 172.21.15.89:0/798066822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c41a4a60 con 0x7f69c4106a80 2024-09-15T23:26:00.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.367+0000 7f69c1ffb640 1 -- 172.21.15.89:0/798066822 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0116b0 con 0x7f69c4106a80 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.367+0000 7f69cade0640 1 -- 172.21.15.89:0/798066822 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c41a4f20 con 0x7f69c4106a80 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.369+0000 7f69c1ffb640 1 -- 172.21.15.89:0/798066822 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f69bc005560 con 0x7f69c4106a80 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69cd1b9740 1 Processor -- start 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69cd1b9740 1 -- start start 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c1ffb640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69b40a3240 0x7f69b40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6b0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e5fb6162f0 con 0x55e5fb6bf780 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e5fb5bdcf0 con 0x55e5fb6b1b40 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e5fb6122b0 con 0x55e5fb6b05d0 2024-09-15T23:26:00.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c1ffb640 1 -- 172.21.15.89:0/798066822 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f69bc0cfec0 con 0x7f69c4106a80 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c937d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c3fff640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69b40a3240 0x7f69b40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c937d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36734/0 (socket says 172.21.15.89:36734) 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c937d640 1 -- 172.21.15.89:0/1847785494 learned_addr learned my addr 172.21.15.89:0/1847785494 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c8b7c640 1 --2- 172.21.15.89:0/1847785494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c3fff640 1 --2- 172.21.15.89:0/1847785494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c937d640 1 -- 172.21.15.89:0/1847785494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 msgr2=0x55e5fb6b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c937d640 1 --2- 172.21.15.89:0/1847785494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c937d640 1 -- 172.21.15.89:0/1847785494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b1b40 msgr2=0x55e5fb6b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c937d640 1 --2- 172.21.15.89:0/1847785494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6b0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c937d640 1 -- 172.21.15.89:0/1847785494 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5fb5a23a0 con 0x55e5fb6bf780 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.370+0000 7f69c3fff640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69b40a3240 0x7f69b40a5700 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009900 tx=0x7f69b0031800 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69c937d640 1 --2- 172.21.15.89:0/1847785494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c1b70 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f69bc098b70 tx=0x7f69bc09c0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181ef970d299480f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69aeffd640 1 -- 172.21.15.89:0/1847785494 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc01f040 con 0x55e5fb6bf780 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69aeffd640 1 -- 172.21.15.89:0/1847785494 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69bc0a4d60 con 0x55e5fb6bf780 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69aeffd640 1 -- 172.21.15.89:0/1847785494 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc098810 con 0x55e5fb6bf780 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69cd1b9740 1 -- 172.21.15.89:0/1847785494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 msgr2=0x55e5fb6c1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/1847785494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c1b70 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f69bc098b70 tx=0x7f69bc09c0a0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69cd1b9740 1 -- 172.21.15.89:0/1847785494 shutdown_connections 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/1847785494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c1b70 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/1847785494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/1847785494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69cd1b9740 1 -- 172.21.15.89:0/1847785494 >> 172.21.15.89:0/1847785494 conn(0x55e5fb679310 msgr2=0x55e5fb6acc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.371+0000 7f69cd1b9740 1 -- 172.21.15.89:0/1847785494 shutdown_connections 2024-09-15T23:26:00.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69cd1b9740 1 -- 172.21.15.89:0/1847785494 wait complete. 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69cd1b9740 1 -- start start 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69cd1b9740 1 Processor -- start 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6bf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6bc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6bf780 0x55e5fb6bc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e5fb6c2420 con 0x55e5fb6b1b40 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e5fb5d1060 con 0x55e5fb6bf780 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e5fb6c25a0 con 0x55e5fb6b05d0 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69cd1b9740 10 client.0 _ll_register_callbacks cb 0x55e5fb3248c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69c3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6bc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69c3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6bc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36744/0 (socket says 172.21.15.89:36744) 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.372+0000 7f69c3fff640 1 -- 172.21.15.89:0/16206427 learned_addr learned my addr 172.21.15.89:0/16206427 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69c937d640 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6bf780 0x55e5fb6bc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69c8b7c640 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6bf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69c3fff640 1 -- 172.21.15.89:0/16206427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 msgr2=0x55e5fb6bf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69c3fff640 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6bf110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69c3fff640 1 -- 172.21.15.89:0/16206427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6bf780 msgr2=0x55e5fb6bc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69c3fff640 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6bf780 0x55e5fb6bc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69c3fff640 1 -- 172.21.15.89:0/16206427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5fb3873c0 con 0x55e5fb6b1b40 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69c937d640 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6bf780 0x55e5fb6bc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69c3fff640 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6bc3c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f69b0000c00 tx=0x7f69b002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30c7a07c41240d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69a3fff640 1 -- 172.21.15.89:0/16206427 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b004e070 con 0x55e5fb6b1b40 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69cd1b9740 1 -- 172.21.15.89:0/16206427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69bc0a4d60 con 0x55e5fb6b1b40 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69a3fff640 1 -- 172.21.15.89:0/16206427 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f69b0003b80 con 0x55e5fb6b1b40 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69a3fff640 1 -- 172.21.15.89:0/16206427 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f69b005a050 con 0x55e5fb6b1b40 2024-09-15T23:26:00.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69cd1b9740 10 client.16182 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69cd1b9740 1 -- 172.21.15.89:0/16206427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e5fb6cdea0 con 0x55e5fb6b1b40 2024-09-15T23:26:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.373+0000 7f69a3fff640 1 -- 172.21.15.89:0/16206427 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69bc0a4d60 con 0x55e5fb6b1b40 2024-09-15T23:26:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.374+0000 7f69a3fff640 1 -- 172.21.15.89:0/16206427 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f69b003d480 con 0x55e5fb6b1b40 2024-09-15T23:26:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.374+0000 7f69cd1b9740 10 client.16182 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.374+0000 7f69cd1b9740 10 client.16182 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:26:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.374+0000 7f69cd1b9740 20 client.16182 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.374+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55e5fb6cf2b0 0x55e5fb6d1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.374+0000 7f69cd1b9740 4 client.16182 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.374+0000 7f69cd1b9740 1 -- 172.21.15.89:0/16206427 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5fb623860 con 0x55e5fb6cf2b0 2024-09-15T23:26:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.374+0000 7f69c8b7c640 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55e5fb6cf2b0 0x55e5fb6d1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.374+0000 7f69a3fff640 10 client.16182 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:26:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.374+0000 7f69c8b7c640 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55e5fb6cf2b0 0x55e5fb6d1730 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cb4d92caf68db0b5 server_cookie=a516be455bf30579 in_seq=0 out_seq=0 2024-09-15T23:26:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69a3fff640 1 -- 172.21.15.89:0/16206427 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e5fb623860 con 0x55e5fb6cf2b0 2024-09-15T23:26:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69a3fff640 10 client.16182 handle_command_reply: tid=0 2024-09-15T23:26:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69cd1b9740 1 client.16182 shutdown 2024-09-15T23:26:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69cd1b9740 1 -- 172.21.15.89:0/16206427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6b1b40 msgr2=0x55e5fb6bc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6bc3c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f69b0000c00 tx=0x7f69b002a420 comp rx=0 tx=0).stop 2024-09-15T23:26:00.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69cd1b9740 1 -- 172.21.15.89:0/16206427 shutdown_connections 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6bf780 0x55e5fb6bc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6bc3c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55e5fb6cf2b0 0x55e5fb6d1730 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/16206427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6bf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69a3fff640 0 client.16182 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.375+0000 7f69cd1b9740 1 -- 172.21.15.89:0/16206427 >> 172.21.15.89:0/16206427 conn(0x55e5fb679310 msgr2=0x55e5fb6b2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.376+0000 7f69cd1b9740 1 -- 172.21.15.89:0/16206427 shutdown_connections 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.376+0000 7f69cd1b9740 1 -- 172.21.15.89:0/16206427 wait complete. 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.376+0000 7f69cd1b9740 20 client.16182 trim_cache size 0 max 16384 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.388+0000 7f69cd1b9740 1 Processor -- start 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.388+0000 7f69cd1b9740 1 -- start start 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6ace10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6d7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41acd60 con 0x55e5fb6bf780 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41ae8a0 con 0x55e5fb6b05d0 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41ad430 con 0x55e5fb6b1b40 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69c937d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6d7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69c937d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6d7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36754/0 (socket says 172.21.15.89:36754) 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69c937d640 1 -- 172.21.15.89:0/4211931316 learned_addr learned my addr 172.21.15.89:0/4211931316 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69c937d640 1 -- 172.21.15.89:0/4211931316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b1b40 msgr2=0x55e5fb6cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69c937d640 1 --2- 172.21.15.89:0/4211931316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69c937d640 1 -- 172.21.15.89:0/4211931316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b05d0 msgr2=0x55e5fb6ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69c937d640 1 --2- 172.21.15.89:0/4211931316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69c937d640 1 -- 172.21.15.89:0/4211931316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5fb6b03d0 con 0x55e5fb6bf780 2024-09-15T23:26:00.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69c937d640 1 --2- 172.21.15.89:0/4211931316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6d7000 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f69bc09c130 tx=0x7f69bc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408e98e8a1787586 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69ae7fc640 1 -- 172.21.15.89:0/4211931316 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc01f040 con 0x55e5fb6bf780 2024-09-15T23:26:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69ae7fc640 1 -- 172.21.15.89:0/4211931316 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69bc09a240 con 0x55e5fb6bf780 2024-09-15T23:26:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.389+0000 7f69ae7fc640 1 -- 172.21.15.89:0/4211931316 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc09b5f0 con 0x55e5fb6bf780 2024-09-15T23:26:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 -- 172.21.15.89:0/4211931316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 msgr2=0x55e5fb6d7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/4211931316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6d7000 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f69bc09c130 tx=0x7f69bc005850 comp rx=0 tx=0).stop 2024-09-15T23:26:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 -- 172.21.15.89:0/4211931316 shutdown_connections 2024-09-15T23:26:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/4211931316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6d7000 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/4211931316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/4211931316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 -- 172.21.15.89:0/4211931316 >> 172.21.15.89:0/4211931316 conn(0x55e5fb679310 msgr2=0x55e5fb6ac700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 -- 172.21.15.89:0/4211931316 shutdown_connections 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 -- 172.21.15.89:0/4211931316 wait complete. 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 -- start start 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.390+0000 7f69cd1b9740 1 Processor -- start 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6c3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69cd1b9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41ae1d0 con 0x55e5fb6bf780 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41adb00 con 0x55e5fb6b1b40 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69cd1b9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f69c41aad40 con 0x55e5fb6b05d0 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69cd1b9740 10 client.0 _ll_register_callbacks cb 0x7f69b0031a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:00.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c937d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c937d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36768/0 (socket says 172.21.15.89:36768) 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c937d640 1 -- 172.21.15.89:0/3769801199 learned_addr learned my addr 172.21.15.89:0/3769801199 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c8b7c640 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c3fff640 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c937d640 1 -- 172.21.15.89:0/3769801199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 msgr2=0x55e5fb6c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c937d640 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6c3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c937d640 1 -- 172.21.15.89:0/3769801199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b1b40 msgr2=0x55e5fb6c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c937d640 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6c3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c937d640 1 -- 172.21.15.89:0/3769801199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5fb738fd0 con 0x55e5fb6bf780 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69c937d640 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c3ff0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009270 tx=0x7f69bc0054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e740cea7027ef09e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69a3fff640 1 -- 172.21.15.89:0/3769801199 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc084060 con 0x55e5fb6bf780 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69a3fff640 1 -- 172.21.15.89:0/3769801199 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f69bc09b5f0 con 0x55e5fb6bf780 2024-09-15T23:26:00.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.391+0000 7f69cd1b9740 1 -- 172.21.15.89:0/3769801199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69bc09a240 con 0x55e5fb6bf780 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69a3fff640 1 -- 172.21.15.89:0/3769801199 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f69bc01f040 con 0x55e5fb6bf780 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69cd1b9740 10 client.16188 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69cd1b9740 1 -- 172.21.15.89:0/3769801199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e5fb73a290 con 0x55e5fb6bf780 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69a3fff640 1 -- 172.21.15.89:0/3769801199 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69bc09a240 con 0x55e5fb6bf780 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69a3fff640 1 -- 172.21.15.89:0/3769801199 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f69bc0b6070 con 0x55e5fb6bf780 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69cd1b9740 10 client.16188 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69cd1b9740 10 client.16188 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69cd1b9740 20 client.16188 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55e5fb73a510 0x55e5fb73c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69cd1b9740 4 client.16188 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69cd1b9740 1 -- 172.21.15.89:0/3769801199 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e5fb61dd70 con 0x55e5fb73a510 2024-09-15T23:26:00.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.392+0000 7f69c8b7c640 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55e5fb73a510 0x55e5fb73c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69c8b7c640 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55e5fb73a510 0x55e5fb73c900 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6f34e1a934c4702a server_cookie=1b0d4c0a1fc1b7f0 in_seq=0 out_seq=0 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69a3fff640 10 client.16188 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69a3fff640 1 -- 172.21.15.89:0/3769801199 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e5fb61dd70 con 0x55e5fb73a510 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69a3fff640 10 client.16188 handle_command_reply: tid=0 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 client.16188 shutdown 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 -- 172.21.15.89:0/3769801199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 msgr2=0x55e5fb6c3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c3ff0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009270 tx=0x7f69bc0054d0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 -- 172.21.15.89:0/3769801199 shutdown_connections 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fb6bf780 0x55e5fb6c3ff0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fb6b1b40 0x55e5fb6c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55e5fb73a510 0x55e5fb73c900 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 --2- 172.21.15.89:0/3769801199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fb6b05d0 0x55e5fb6c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 -- 172.21.15.89:0/3769801199 >> 172.21.15.89:0/3769801199 conn(0x55e5fb679310 msgr2=0x55e5fb6bdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69a3fff640 0 client.16188 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:26:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 -- 172.21.15.89:0/3769801199 shutdown_connections 2024-09-15T23:26:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.393+0000 7f69cd1b9740 1 -- 172.21.15.89:0/3769801199 wait complete. 2024-09-15T23:26:00.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cd1b9740 20 client.16188 trim_cache size 0 max 16384 2024-09-15T23:26:00.413 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:26:00.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 -- 172.21.15.89:0/798066822 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69b40a3240 msgr2=0x7f69b40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69b40a3240 0x7f69b40a5700 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009900 tx=0x7f69b0031800 comp rx=0 tx=0).stop 2024-09-15T23:26:00.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 -- 172.21.15.89:0/798066822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4106a80 msgr2=0x7f69c41a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:00.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4106a80 0x7f69c41a4350 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00eaa0 tx=0x7f69bc00ef60 comp rx=0 tx=0).stop 2024-09-15T23:26:00.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 -- 172.21.15.89:0/798066822 shutdown_connections 2024-09-15T23:26:00.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69b40a3240 0x7f69b40a5700 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4106a80 0x7f69c41a4350 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69c4101420 0x7f69c4199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 --2- 172.21.15.89:0/798066822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69c4100a80 0x7f69c41992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:00.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 -- 172.21.15.89:0/798066822 >> 172.21.15.89:0/798066822 conn(0x7f69c40fc7a0 msgr2=0x7f69c40fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:00.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 -- 172.21.15.89:0/798066822 shutdown_connections 2024-09-15T23:26:00.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:00.394+0000 7f69cade0640 1 -- 172.21.15.89:0/798066822 wait complete. 2024-09-15T23:26:01.202 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:26:01.453 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.920+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/885031537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d4102df0 msgr2=0x7fd1d41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.920+0000 7fd1d8a2b640 1 --2- 172.21.15.89:0/885031537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d4102df0 0x7fd1d41031d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009900 tx=0x7fd1c402a410 comp rx=0 tx=0).stop 2024-09-15T23:26:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.920+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/885031537 shutdown_connections 2024-09-15T23:26:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.920+0000 7fd1d8a2b640 1 --2- 172.21.15.89:0/885031537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1d410c2f0 0x7fd1d410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.920+0000 7fd1d8a2b640 1 --2- 172.21.15.89:0/885031537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d41037a0 0x7fd1d410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.920+0000 7fd1d8a2b640 1 --2- 172.21.15.89:0/885031537 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d4102df0 0x7fd1d41031d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.920+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/885031537 >> 172.21.15.89:0/885031537 conn(0x7fd1d40fc7d0 msgr2=0x7fd1d40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.920+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/885031537 shutdown_connections 2024-09-15T23:26:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.920+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/885031537 wait complete. 2024-09-15T23:26:01.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.920+0000 7fd1d8a2b640 1 Processor -- start 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d8a2b640 1 -- start start 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d8a2b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1d4102df0 0x7fd1d4198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d8a2b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d41037a0 0x7fd1d4198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d8a2b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d410c2f0 0x7fd1d419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d8a2b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d4112110 con 0x7fd1d41037a0 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d8a2b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d4111f90 con 0x7fd1d410c2f0 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d8a2b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d4112290 con 0x7fd1d4102df0 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1cbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d41037a0 0x7fd1d4198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1cbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d41037a0 0x7fd1d4198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36792/0 (socket says 172.21.15.89:36792) 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1cbfff640 1 -- 172.21.15.89:0/2426004582 learned_addr learned my addr 172.21.15.89:0/2426004582 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:01.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d2d9d640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d410c2f0 0x7fd1d419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d259c640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1d4102df0 0x7fd1d4198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1cbfff640 1 -- 172.21.15.89:0/2426004582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1d4102df0 msgr2=0x7fd1d4198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1d4102df0 0x7fd1d4198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1cbfff640 1 -- 172.21.15.89:0/2426004582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d410c2f0 msgr2=0x7fd1d419cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d410c2f0 0x7fd1d419cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1cbfff640 1 -- 172.21.15.89:0/2426004582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d419d5c0 con 0x7fd1d41037a0 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d2d9d640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d410c2f0 0x7fd1d419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1d259c640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1d4102df0 0x7fd1d4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d41037a0 0x7fd1d4198b50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00c970 tx=0x7fd1bc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.921+0000 7fd1cb7fe640 1 -- 172.21.15.89:0/2426004582 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc007bb0 con 0x7fd1d41037a0 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.922+0000 7fd1cb7fe640 1 -- 172.21.15.89:0/2426004582 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1bc007d50 con 0x7fd1d41037a0 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.922+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/2426004582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d419d8b0 con 0x7fd1d41037a0 2024-09-15T23:26:01.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.922+0000 7fd1cb7fe640 1 -- 172.21.15.89:0/2426004582 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc00f450 con 0x7fd1d41037a0 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.922+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/2426004582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d41a5190 con 0x7fd1d41037a0 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.924+0000 7fd1cb7fe640 1 -- 172.21.15.89:0/2426004582 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd1bc004f60 con 0x7fd1d41037a0 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.924+0000 7fd1dae04740 1 Processor -- start 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.924+0000 7fd1dae04740 1 -- start start 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cb7fe640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1c00a3240 0x7fd1c00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f528f1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f528f1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f52902b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1dae04740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556f528572f0 con 0x556f528f15d0 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1dae04740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556f527fecf0 con 0x556f528f2b40 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1dae04740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556f528532b0 con 0x556f52900780 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1d2d9d640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1c00a3240 0x7fd1c00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f528f1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f528f1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36802/0 (socket says 172.21.15.89:36802) 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cbfff640 1 -- 172.21.15.89:0/4152775976 learned_addr learned my addr 172.21.15.89:0/4152775976 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cb7fe640 1 -- 172.21.15.89:0/2426004582 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd1bc0cfb00 con 0x7fd1d41037a0 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1d259c640 1 --2- 172.21.15.89:0/4152775976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f528f1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1d2d9d640 1 --2- 172.21.15.89:0/4152775976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f52902b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cbfff640 1 -- 172.21.15.89:0/4152775976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 msgr2=0x556f52902b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/4152775976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f52902b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cbfff640 1 -- 172.21.15.89:0/4152775976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 msgr2=0x556f528f1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/4152775976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f528f1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cbfff640 1 -- 172.21.15.89:0/4152775976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f527e33a0 con 0x556f528f15d0 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1d2d9d640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1c00a3240 0x7fd1c00a5700 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc005fd0 tx=0x7fd1cc014af0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:01.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/4152775976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f528f1a10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00c940 tx=0x7fd1bc01baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c37254ba00e17f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1c8ff9640 1 -- 172.21.15.89:0/4152775976 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc0b4f30 con 0x556f528f15d0 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.925+0000 7fd1c8ff9640 1 -- 172.21.15.89:0/4152775976 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1bc005600 con 0x556f528f15d0 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.926+0000 7fd1c8ff9640 1 -- 172.21.15.89:0/4152775976 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc009c50 con 0x556f528f15d0 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.926+0000 7fd1dae04740 1 -- 172.21.15.89:0/4152775976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 msgr2=0x556f528f1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.926+0000 7fd1dae04740 1 --2- 172.21.15.89:0/4152775976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f528f1a10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00c940 tx=0x7fd1bc01baf0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.926+0000 7fd1dae04740 1 -- 172.21.15.89:0/4152775976 shutdown_connections 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.926+0000 7fd1dae04740 1 --2- 172.21.15.89:0/4152775976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f52902b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.926+0000 7fd1dae04740 1 --2- 172.21.15.89:0/4152775976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f528f1a10 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.926+0000 7fd1dae04740 1 --2- 172.21.15.89:0/4152775976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f528f1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.926+0000 7fd1dae04740 1 -- 172.21.15.89:0/4152775976 >> 172.21.15.89:0/4152775976 conn(0x556f528ba310 msgr2=0x556f528edc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.926+0000 7fd1dae04740 1 -- 172.21.15.89:0/4152775976 shutdown_connections 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.926+0000 7fd1dae04740 1 -- 172.21.15.89:0/4152775976 wait complete. 2024-09-15T23:26:01.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1dae04740 1 -- start start 2024-09-15T23:26:01.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1dae04740 1 Processor -- start 2024-09-15T23:26:01.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529001a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f528fd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f528fda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1dae04740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556f52903420 con 0x556f528f15d0 2024-09-15T23:26:01.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1dae04740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556f52812060 con 0x556f528f2b40 2024-09-15T23:26:01.957 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:26:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1dae04740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556f529035a0 con 0x556f52900780 2024-09-15T23:26:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1dae04740 10 client.0 _ll_register_callbacks cb 0x556f525658c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1d259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1d259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529001a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36804/0 (socket says 172.21.15.89:36804) 2024-09-15T23:26:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1d259c640 1 -- 172.21.15.89:0/997946686 learned_addr learned my addr 172.21.15.89:0/997946686 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1d2d9d640 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f528fda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.927+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f528fd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1d259c640 1 -- 172.21.15.89:0/997946686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 msgr2=0x556f528fda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1d259c640 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f528fda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1d259c640 1 -- 172.21.15.89:0/997946686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 msgr2=0x556f528fd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1d259c640 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f528fd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1d259c640 1 -- 172.21.15.89:0/997946686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f525c83c0 con 0x556f528f15d0 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1d2d9d640 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f528fda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f528fd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1d259c640 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529001a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4032fd0 tx=0x7fd1c4031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91af04b0ac000d7f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/997946686 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c403f070 con 0x556f528f15d0 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/997946686 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd1c4005030 con 0x556f528f15d0 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1dae04740 1 -- 172.21.15.89:0/997946686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556f52864860 con 0x556f528f15d0 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/997946686 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd1c404c050 con 0x556f528f15d0 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1dae04740 10 client.16200 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:01.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1dae04740 1 -- 172.21.15.89:0/997946686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f5290eef0 con 0x556f528f15d0 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/997946686 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556f52864860 con 0x556f528f15d0 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/997946686 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd1c4004c50 con 0x556f528f15d0 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1dae04740 10 client.16200 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1dae04740 10 client.16200 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1dae04740 20 client.16200 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1dae04740 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556f529103b0 0x556f529127a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1dae04740 4 client.16200 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.928+0000 7fd1dae04740 1 -- 172.21.15.89:0/997946686 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f5285ed70 con 0x556f529103b0 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.929+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556f529103b0 0x556f529127a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.929+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556f529103b0 0x556f529127a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b8ace541db656d6 server_cookie=9f23dc595a21ed27 in_seq=0 out_seq=0 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.929+0000 7fd1b9ffb640 10 client.16200 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/997946686 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556f5285ed70 con 0x556f529103b0 2024-09-15T23:26:01.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1b9ffb640 10 client.16200 handle_command_reply: tid=0 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 client.16200 shutdown 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 -- 172.21.15.89:0/997946686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 msgr2=0x556f529001a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529001a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4032fd0 tx=0x7fd1c4031920 comp rx=0 tx=0).stop 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 -- 172.21.15.89:0/997946686 shutdown_connections 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556f529103b0 0x556f529127a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f528fda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f528fd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1b9ffb640 0 client.16200 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 --2- 172.21.15.89:0/997946686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529001a0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 -- 172.21.15.89:0/997946686 >> 172.21.15.89:0/997946686 conn(0x556f528ba310 msgr2=0x556f528f3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 -- 172.21.15.89:0/997946686 shutdown_connections 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 1 -- 172.21.15.89:0/997946686 wait complete. 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.930+0000 7fd1dae04740 20 client.16200 trim_cache size 0 max 16384 2024-09-15T23:26:01.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1dae04740 1 Processor -- start 2024-09-15T23:26:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1dae04740 1 -- start start 2024-09-15T23:26:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f528f15d0 0x556f528fc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f2b40 0x556f5290c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f52900780 0x556f528ecbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1dae04740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41acd80 con 0x556f528f2b40 2024-09-15T23:26:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1dae04740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41ae8c0 con 0x556f52900780 2024-09-15T23:26:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1dae04740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41ad450 con 0x556f528f15d0 2024-09-15T23:26:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1cbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f2b40 0x556f5290c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1cbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f2b40 0x556f5290c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36814/0 (socket says 172.21.15.89:36814) 2024-09-15T23:26:01.962 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1cbfff640 1 -- 172.21.15.89:0/3644891807 learned_addr learned my addr 172.21.15.89:0/3644891807 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1cbfff640 1 -- 172.21.15.89:0/3644891807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f528f15d0 msgr2=0x556f528fc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1d259c640 1 --2- 172.21.15.89:0/3644891807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f528f15d0 0x556f528fc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/3644891807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f528f15d0 0x556f528fc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1cbfff640 1 -- 172.21.15.89:0/3644891807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f52900780 msgr2=0x556f528ecbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/3644891807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f52900780 0x556f528ecbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.943+0000 7fd1cbfff640 1 -- 172.21.15.89:0/3644891807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f528fb9a0 con 0x556f528f2b40 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/3644891807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f2b40 0x556f5290c6c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc098700 tx=0x7fd1bc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb9fc082fbb920d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1bbfff640 1 -- 172.21.15.89:0/3644891807 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc01d040 con 0x556f528f2b40 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1bbfff640 1 -- 172.21.15.89:0/3644891807 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1bc09a030 con 0x556f528f2b40 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1bbfff640 1 -- 172.21.15.89:0/3644891807 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc005250 con 0x556f528f2b40 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1dae04740 1 -- 172.21.15.89:0/3644891807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f2b40 msgr2=0x556f5290c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1dae04740 1 --2- 172.21.15.89:0/3644891807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f2b40 0x556f5290c6c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc098700 tx=0x7fd1bc000bc0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1dae04740 1 -- 172.21.15.89:0/3644891807 shutdown_connections 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1dae04740 1 --2- 172.21.15.89:0/3644891807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f52900780 0x556f528ecbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1dae04740 1 --2- 172.21.15.89:0/3644891807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f2b40 0x556f5290c6c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1dae04740 1 --2- 172.21.15.89:0/3644891807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f528f15d0 0x556f528fc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1dae04740 1 -- 172.21.15.89:0/3644891807 >> 172.21.15.89:0/3644891807 conn(0x556f528ba310 msgr2=0x556f528ec4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1dae04740 1 -- 172.21.15.89:0/3644891807 shutdown_connections 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.944+0000 7fd1dae04740 1 -- 172.21.15.89:0/3644891807 wait complete. 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1dae04740 1 -- start start 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1dae04740 1 Processor -- start 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f52904c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1dae04740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f52905140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1dae04740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41ae1f0 con 0x556f528f15d0 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1dae04740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41adb20 con 0x556f528f2b40 2024-09-15T23:26:01.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1dae04740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d41aad60 con 0x556f52900780 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1dae04740 10 client.0 _ll_register_callbacks cb 0x7fd1c4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1d259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1d259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36820/0 (socket says 172.21.15.89:36820) 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1d259c640 1 -- 172.21.15.89:0/1991696379 learned_addr learned my addr 172.21.15.89:0/1991696379 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f52904c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1d259c640 1 -- 172.21.15.89:0/1991696379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 msgr2=0x556f52905140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1d2d9d640 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f52905140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1d259c640 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f52905140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1d259c640 1 -- 172.21.15.89:0/1991696379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 msgr2=0x556f52904c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1d259c640 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f52904c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.945+0000 7fd1d259c640 1 -- 172.21.15.89:0/1991696379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f5297a1a0 con 0x556f528f15d0 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1d259c640 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529046c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4033000 tx=0x7fd1c4031480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86db0ad911e08399 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:01.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/1991696379 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c403f030 con 0x556f528f15d0 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/1991696379 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd1c4004310 con 0x556f528f15d0 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1dae04740 1 -- 172.21.15.89:0/1991696379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1bc09a030 con 0x556f528f15d0 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/1991696379 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd1c405e050 con 0x556f528f15d0 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1dae04740 10 client.16206 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1dae04740 1 -- 172.21.15.89:0/1991696379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f5297b440 con 0x556f528f15d0 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/1991696379 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1bc09a030 con 0x556f528f15d0 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/1991696379 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd1c4003830 con 0x556f528f15d0 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1dae04740 10 client.16206 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1dae04740 10 client.16206 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1dae04740 20 client.16206 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1dae04740 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556f5297bbe0 0x556f5297e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1dae04740 4 client.16206 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:01.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.946+0000 7fd1dae04740 1 -- 172.21.15.89:0/1991696379 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f527db170 con 0x556f5297bbe0 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556f5297bbe0 0x556f5297e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1cbfff640 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556f5297bbe0 0x556f5297e000 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d943f4249396af66 server_cookie=1c2e8845f52f94cb in_seq=0 out_seq=0 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1b9ffb640 10 client.16206 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1b9ffb640 1 -- 172.21.15.89:0/1991696379 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f527db170 con 0x556f5297bbe0 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1b9ffb640 10 client.16206 handle_command_reply: tid=0 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 client.16206 shutdown 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 -- 172.21.15.89:0/1991696379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 msgr2=0x556f529046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529046c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4033000 tx=0x7fd1c4031480 comp rx=0 tx=0).stop 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 -- 172.21.15.89:0/1991696379 shutdown_connections 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556f5297bbe0 0x556f5297e000 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556f52900780 0x556f52905140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f528f2b40 0x556f52904c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1b9ffb640 0 client.16206 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:26:01.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 --2- 172.21.15.89:0/1991696379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556f528f15d0 0x556f529046c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 -- 172.21.15.89:0/1991696379 >> 172.21.15.89:0/1991696379 conn(0x556f528ba310 msgr2=0x556f528ecd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 -- 172.21.15.89:0/1991696379 shutdown_connections 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.947+0000 7fd1dae04740 1 -- 172.21.15.89:0/1991696379 wait complete. 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1dae04740 20 client.16206 trim_cache size 0 max 16384 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/2426004582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1c00a3240 msgr2=0x7fd1c00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1c00a3240 0x7fd1c00a5700 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc005fd0 tx=0x7fd1cc014af0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/2426004582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d41037a0 msgr2=0x7fd1d4198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d41037a0 0x7fd1d4198b50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00c970 tx=0x7fd1bc00ce30 comp rx=0 tx=0).stop 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/2426004582 shutdown_connections 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1c00a3240 0x7fd1c00a5700 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1d410c2f0 0x7fd1d419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1d41037a0 0x7fd1d4198b50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 --2- 172.21.15.89:0/2426004582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1d4102df0 0x7fd1d4198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:01.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/2426004582 >> 172.21.15.89:0/2426004582 conn(0x7fd1d40fc7d0 msgr2=0x7fd1d40fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:01.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/2426004582 shutdown_connections 2024-09-15T23:26:01.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:01.948+0000 7fd1d8a2b640 1 -- 172.21.15.89:0/2426004582 wait complete. 2024-09-15T23:26:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:02 smithi120 ceph-mon[35250]: pgmap v410: 129 pgs: 129 active+clean; 811 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-15T23:26:02.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:02 smithi174 ceph-mon[35586]: pgmap v410: 129 pgs: 129 active+clean; 811 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-15T23:26:02.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:02 smithi089 ceph-mon[30933]: pgmap v410: 129 pgs: 129 active+clean; 811 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-15T23:26:02.678 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:26:02.925 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:03.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.407+0000 7f9e882dc640 1 -- 172.21.15.89:0/3917488163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e80103780 msgr2=0x7f9e80107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.407+0000 7f9e882dc640 1 --2- 172.21.15.89:0/3917488163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e80103780 0x7f9e80107bc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9e70009930 tx=0x7f9e7002a420 comp rx=0 tx=0).stop 2024-09-15T23:26:03.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 -- 172.21.15.89:0/3917488163 shutdown_connections 2024-09-15T23:26:03.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 --2- 172.21.15.89:0/3917488163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e80108300 0x7f9e8010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 --2- 172.21.15.89:0/3917488163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e80103780 0x7f9e80107bc0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 --2- 172.21.15.89:0/3917488163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e80102dd0 0x7f9e801031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 -- 172.21.15.89:0/3917488163 >> 172.21.15.89:0/3917488163 conn(0x7f9e800fc7d0 msgr2=0x7f9e800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 -- 172.21.15.89:0/3917488163 shutdown_connections 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 -- 172.21.15.89:0/3917488163 wait complete. 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 Processor -- start 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 -- start start 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e80102dd0 0x7f9e80198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e80103780 0x7f9e80198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e80108300 0x7f9e8019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9e801120f0 con 0x7f9e80108300 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e80111f70 con 0x7f9e80102dd0 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.408+0000 7f9e882dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e80112270 con 0x7f9e80103780 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e86879640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e80108300 0x7f9e8019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e86879640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e80108300 0x7f9e8019cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36828/0 (socket says 172.21.15.89:36828) 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e86879640 1 -- 172.21.15.89:0/3385341668 learned_addr learned my addr 172.21.15.89:0/3385341668 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e85877640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e80103780 0x7f9e80198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e86078640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e80102dd0 0x7f9e80198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e86879640 1 -- 172.21.15.89:0/3385341668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e80103780 msgr2=0x7f9e80198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e86879640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e80103780 0x7f9e80198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e86879640 1 -- 172.21.15.89:0/3385341668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e80102dd0 msgr2=0x7f9e80198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e86879640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e80102dd0 0x7f9e80198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e86879640 1 -- 172.21.15.89:0/3385341668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8019d600 con 0x7f9e80108300 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e86879640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e80108300 0x7f9e8019cf20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800eaa0 tx=0x7f9e7800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e777fe640 1 -- 172.21.15.89:0/3385341668 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7800cdb0 con 0x7f9e80108300 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e777fe640 1 -- 172.21.15.89:0/3385341668 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e78005020 con 0x7f9e80108300 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e882dc640 1 -- 172.21.15.89:0/3385341668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e8019d8f0 con 0x7f9e80108300 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e777fe640 1 -- 172.21.15.89:0/3385341668 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e780116b0 con 0x7f9e80108300 2024-09-15T23:26:03.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.409+0000 7f9e882dc640 1 -- 172.21.15.89:0/3385341668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e801a51d0 con 0x7f9e80108300 2024-09-15T23:26:03.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.412+0000 7f9e777fe640 1 -- 172.21.15.89:0/3385341668 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9e78005560 con 0x7f9e80108300 2024-09-15T23:26:03.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.412+0000 7f9e8a6b5740 1 Processor -- start 2024-09-15T23:26:03.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.412+0000 7f9e8a6b5740 1 -- start start 2024-09-15T23:26:03.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.412+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 0x55757730a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730a5d0 0x55757730aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577319780 0x55757731bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5575772702f0 con 0x55757730a5d0 2024-09-15T23:26:03.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557577217cf0 con 0x557577319780 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55757726c2b0 con 0x55757730bb40 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e777fe640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e6c0a32c0 0x7f9e6c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86078640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730a5d0 0x55757730aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86078640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730a5d0 0x55757730aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36844/0 (socket says 172.21.15.89:36844) 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86078640 1 -- 172.21.15.89:0/3050618124 learned_addr learned my addr 172.21.15.89:0/3050618124 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e85877640 1 --2- 172.21.15.89:0/3050618124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577319780 0x55757731bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86879640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e6c0a32c0 0x7f9e6c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e85877640 1 --2- 172.21.15.89:0/3050618124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 0x55757730a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e777fe640 1 -- 172.21.15.89:0/3385341668 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9e780cffe0 con 0x7f9e80108300 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86078640 1 -- 172.21.15.89:0/3050618124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 msgr2=0x55757730a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86078640 1 --2- 172.21.15.89:0/3050618124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 0x55757730a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86078640 1 -- 172.21.15.89:0/3050618124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577319780 msgr2=0x55757731bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86078640 1 --2- 172.21.15.89:0/3050618124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577319780 0x55757731bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86078640 1 -- 172.21.15.89:0/3050618124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575771fc3a0 con 0x55757730a5d0 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e85877640 1 --2- 172.21.15.89:0/3050618124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 0x55757730a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86078640 1 --2- 172.21.15.89:0/3050618124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730a5d0 0x55757730aa10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c00b820 tx=0x7f9e7c00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5590a5cd40a97445 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.413+0000 7f9e86879640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e6c0a32c0 0x7f9e6c0a5780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9e7801f3c0 tx=0x7f9e7801c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.414+0000 7f9e74dea640 1 -- 172.21.15.89:0/3050618124 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c00d950 con 0x55757730a5d0 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.414+0000 7f9e74dea640 1 -- 172.21.15.89:0/3050618124 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e7c019070 con 0x55757730a5d0 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.414+0000 7f9e74dea640 1 -- 172.21.15.89:0/3050618124 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c0123e0 con 0x55757730a5d0 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.414+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/3050618124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730a5d0 msgr2=0x55757730aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.414+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/3050618124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730a5d0 0x55757730aa10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c00b820 tx=0x7f9e7c00bce0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.415+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/3050618124 shutdown_connections 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.415+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/3050618124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577319780 0x55757731bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.415+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/3050618124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730a5d0 0x55757730aa10 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.415+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/3050618124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 0x55757730a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.415+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/3050618124 >> 172.21.15.89:0/3050618124 conn(0x5575772d3310 msgr2=0x557577306c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.415+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/3050618124 shutdown_connections 2024-09-15T23:26:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.415+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/3050618124 wait complete. 2024-09-15T23:26:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.415+0000 7f9e8a6b5740 1 -- start start 2024-09-15T23:26:03.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.415+0000 7f9e8a6b5740 1 Processor -- start 2024-09-15T23:26:03.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 0x557577319120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 0x5575773163d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557577319780 0x557577316970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55757731c480 con 0x557577319780 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55757722b060 con 0x55757730a5d0 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55757731c600 con 0x55757730bb40 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e8a6b5740 10 client.0 _ll_register_callbacks cb 0x557576f7e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e85877640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557577319780 0x557577316970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e85877640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557577319780 0x557577316970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36856/0 (socket says 172.21.15.89:36856) 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e85877640 1 -- 172.21.15.89:0/2506906444 learned_addr learned my addr 172.21.15.89:0/2506906444 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e86078640 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 0x5575773163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e85877640 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 0x557577319120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e85877640 1 -- 172.21.15.89:0/2506906444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 msgr2=0x5575773163d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e85877640 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 0x5575773163d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e85877640 1 -- 172.21.15.89:0/2506906444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 msgr2=0x557577319120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e85877640 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 0x557577319120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e85877640 1 -- 172.21.15.89:0/2506906444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557576fe13c0 con 0x557577319780 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e85877640 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557577319780 0x557577316970 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9e700057c0 tx=0x7f9e7002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7dda122ef88367f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e69ffb640 1 -- 172.21.15.89:0/2506906444 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70035040 con 0x557577319780 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/2506906444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e7c019070 con 0x557577319780 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.416+0000 7f9e69ffb640 1 -- 172.21.15.89:0/2506906444 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9e70004250 con 0x557577319780 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e69ffb640 1 -- 172.21.15.89:0/2506906444 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e70004450 con 0x557577319780 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e8a6b5740 10 client.16218 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/2506906444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557577327f20 con 0x557577319780 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e69ffb640 1 -- 172.21.15.89:0/2506906444 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e7c019070 con 0x557577319780 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e69ffb640 1 -- 172.21.15.89:0/2506906444 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9e7004a020 con 0x557577319780 2024-09-15T23:26:03.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e8a6b5740 10 client.16218 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e8a6b5740 10 client.16218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e8a6b5740 20 client.16218 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557577329270 0x55757732b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e8a6b5740 4 client.16218 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/2506906444 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55757727d860 con 0x557577329270 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.417+0000 7f9e86078640 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557577329270 0x55757732b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.418+0000 7f9e69ffb640 10 client.16218 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.418+0000 7f9e86078640 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557577329270 0x55757732b660 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adcd7663e0fc2460 server_cookie=eb5000277538056d in_seq=0 out_seq=0 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.418+0000 7f9e69ffb640 1 -- 172.21.15.89:0/2506906444 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55757727d860 con 0x557577329270 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.418+0000 7f9e69ffb640 10 client.16218 handle_command_reply: tid=0 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.418+0000 7f9e8a6b5740 1 client.16218 shutdown 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/2506906444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557577319780 msgr2=0x557577316970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557577319780 0x557577316970 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9e700057c0 tx=0x7f9e7002a420 comp rx=0 tx=0).stop 2024-09-15T23:26:03.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/2506906444 shutdown_connections 2024-09-15T23:26:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557577329270 0x55757732b660 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557577319780 0x557577316970 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730bb40 0x5575773163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/2506906444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 0x557577319120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/2506906444 >> 172.21.15.89:0/2506906444 conn(0x5575772d3310 msgr2=0x55757730c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e69ffb640 0 client.16218 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/2506906444 shutdown_connections 2024-09-15T23:26:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/2506906444 wait complete. 2024-09-15T23:26:03.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.419+0000 7f9e8a6b5740 20 client.16218 trim_cache size 0 max 16384 2024-09-15T23:26:03.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e8a6b5740 1 Processor -- start 2024-09-15T23:26:03.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e8a6b5740 1 -- start start 2024-09-15T23:26:03.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 0x557577325720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x55757731b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557577319780 0x55757731ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9e801acdb0 con 0x55757730bb40 2024-09-15T23:26:03.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e801ae8f0 con 0x55757730a5d0 2024-09-15T23:26:03.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e801ad480 con 0x557577319780 2024-09-15T23:26:03.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e86078640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x55757731b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e86078640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x55757731b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36860/0 (socket says 172.21.15.89:36860) 2024-09-15T23:26:03.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.432+0000 7f9e86078640 1 -- 172.21.15.89:0/1761396978 learned_addr learned my addr 172.21.15.89:0/1761396978 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:03.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e85877640 1 --2- 172.21.15.89:0/1761396978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557577319780 0x55757731ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e85877640 1 --2- 172.21.15.89:0/1761396978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 0x557577325720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e86078640 1 -- 172.21.15.89:0/1761396978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557577319780 msgr2=0x55757731ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e86078640 1 --2- 172.21.15.89:0/1761396978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557577319780 0x55757731ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e86078640 1 -- 172.21.15.89:0/1761396978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 msgr2=0x557577325720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e86078640 1 --2- 172.21.15.89:0/1761396978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 0x557577325720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e86078640 1 -- 172.21.15.89:0/1761396978 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55757730b750 con 0x55757730bb40 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e85877640 1 --2- 172.21.15.89:0/1761396978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 0x557577325720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e86078640 1 --2- 172.21.15.89:0/1761396978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x55757731b780 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x557577307ac0 tx=0x7f9e7c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b9f673c8deed5e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e6bfff640 1 -- 172.21.15.89:0/1761396978 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c02b020 con 0x55757730bb40 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e6bfff640 1 -- 172.21.15.89:0/1761396978 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e7c019040 con 0x55757730bb40 2024-09-15T23:26:03.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.433+0000 7f9e6bfff640 1 -- 172.21.15.89:0/1761396978 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c027870 con 0x55757730bb40 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1761396978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 msgr2=0x55757731b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/1761396978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x55757731b780 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x557577307ac0 tx=0x7f9e7c008040 comp rx=0 tx=0).stop 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1761396978 shutdown_connections 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/1761396978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557577319780 0x55757731ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/1761396978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x55757731b780 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/1761396978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55757730a5d0 0x557577325720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1761396978 >> 172.21.15.89:0/1761396978 conn(0x5575772d3310 msgr2=0x557577316390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1761396978 shutdown_connections 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1761396978 wait complete. 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 -- start start 2024-09-15T23:26:03.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.434+0000 7f9e8a6b5740 1 Processor -- start 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730a5d0 0x557577321f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x557577313000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e8a6b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577319780 0x557577313540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9e801ae220 con 0x55757730bb40 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e801adb50 con 0x557577319780 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e8a6b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e801aad90 con 0x55757730a5d0 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e8a6b5740 10 client.0 _ll_register_callbacks cb 0x7f9e70032df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e86078640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x557577313000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e86078640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x557577313000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36864/0 (socket says 172.21.15.89:36864) 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e86078640 1 -- 172.21.15.89:0/1591216716 learned_addr learned my addr 172.21.15.89:0/1591216716 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e85877640 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577319780 0x557577313540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e85877640 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730a5d0 0x557577321f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e86078640 1 -- 172.21.15.89:0/1591216716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730a5d0 msgr2=0x557577321f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e86078640 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730a5d0 0x557577321f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e86078640 1 -- 172.21.15.89:0/1591216716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577319780 msgr2=0x557577313540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e86078640 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577319780 0x557577313540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e86078640 1 -- 172.21.15.89:0/1591216716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557577313c80 con 0x55757730bb40 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e86078640 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x557577313000 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c008010 tx=0x7f9e7c00b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc460edd85f980c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e69ffb640 1 -- 172.21.15.89:0/1591216716 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c02b030 con 0x55757730bb40 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e69ffb640 1 -- 172.21.15.89:0/1591216716 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9e7c00db50 con 0x55757730bb40 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.435+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1591216716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e7c019040 con 0x55757730bb40 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e69ffb640 1 -- 172.21.15.89:0/1591216716 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e7c040050 con 0x55757730bb40 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e8a6b5740 10 client.16224 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1591216716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557577393eb0 con 0x55757730bb40 2024-09-15T23:26:03.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e69ffb640 1 -- 172.21.15.89:0/1591216716 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e7c019040 con 0x55757730bb40 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e69ffb640 1 -- 172.21.15.89:0/1591216716 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9e7c013070 con 0x55757730bb40 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e8a6b5740 10 client.16224 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e8a6b5740 10 client.16224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e8a6b5740 20 client.16224 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557577394320 0x557577396710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e8a6b5740 4 client.16224 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1591216716 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55757731bf50 con 0x557577394320 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.436+0000 7f9e85877640 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557577394320 0x557577396710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e69ffb640 10 client.16224 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e85877640 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557577394320 0x557577396710 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4015108f4dc718e server_cookie=68bb27eb0b887e9a in_seq=0 out_seq=0 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e69ffb640 1 -- 172.21.15.89:0/1591216716 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55757731bf50 con 0x557577394320 2024-09-15T23:26:03.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e69ffb640 10 client.16224 handle_command_reply: tid=0 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 client.16224 shutdown 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1591216716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 msgr2=0x557577313000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x557577313000 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c008010 tx=0x7f9e7c00b3a0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1591216716 shutdown_connections 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557577394320 0x557577396710 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e69ffb640 0 client.16224 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557577319780 0x557577313540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55757730bb40 0x557577313000 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 --2- 172.21.15.89:0/1591216716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55757730a5d0 0x557577321f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1591216716 >> 172.21.15.89:0/1591216716 conn(0x5575772d3310 msgr2=0x557577314c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1591216716 shutdown_connections 2024-09-15T23:26:03.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 1 -- 172.21.15.89:0/1591216716 wait complete. 2024-09-15T23:26:03.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.437+0000 7f9e8a6b5740 20 client.16224 trim_cache size 0 max 16384 2024-09-15T23:26:03.445 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 -- 172.21.15.89:0/3385341668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e6c0a32c0 msgr2=0x7f9e6c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e6c0a32c0 0x7f9e6c0a5780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9e7801f3c0 tx=0x7f9e7801c040 comp rx=0 tx=0).stop 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 -- 172.21.15.89:0/3385341668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e80108300 msgr2=0x7f9e8019cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e80108300 0x7f9e8019cf20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800eaa0 tx=0x7f9e7800ef60 comp rx=0 tx=0).stop 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 -- 172.21.15.89:0/3385341668 shutdown_connections 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e6c0a32c0 0x7f9e6c0a5780 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e80108300 0x7f9e8019cf20 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e80103780 0x7f9e80198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 --2- 172.21.15.89:0/3385341668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e80102dd0 0x7f9e80198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 -- 172.21.15.89:0/3385341668 >> 172.21.15.89:0/3385341668 conn(0x7f9e800fc7d0 msgr2=0x7f9e80105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 -- 172.21.15.89:0/3385341668 shutdown_connections 2024-09-15T23:26:03.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:03.438+0000 7f9e882dc640 1 -- 172.21.15.89:0/3385341668 wait complete. 2024-09-15T23:26:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:04 smithi120 ceph-mon[35250]: pgmap v411: 129 pgs: 129 active+clean; 881 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:26:04.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:04 smithi174 ceph-mon[35586]: pgmap v411: 129 pgs: 129 active+clean; 881 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:26:04.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:04 smithi089 ceph-mon[30933]: pgmap v411: 129 pgs: 129 active+clean; 881 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:26:05.349 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:26:05.349 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:26:05.610 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:06.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 -- 172.21.15.89:0/2990056811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4101027a0 msgr2=0x7fc4101067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:06.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 --2- 172.21.15.89:0/2990056811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4101027a0 0x7fc4101067f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc404009900 tx=0x7fc40402a410 comp rx=0 tx=0).stop 2024-09-15T23:26:06.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 -- 172.21.15.89:0/2990056811 shutdown_connections 2024-09-15T23:26:06.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 --2- 172.21.15.89:0/2990056811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc410106f20 0x7fc41010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 --2- 172.21.15.89:0/2990056811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4101027a0 0x7fc4101067f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 --2- 172.21.15.89:0/2990056811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc410101df0 0x7fc4101021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 -- 172.21.15.89:0/2990056811 >> 172.21.15.89:0/2990056811 conn(0x7fc4100fd8f0 msgr2=0x7fc4100ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:06.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 -- 172.21.15.89:0/2990056811 shutdown_connections 2024-09-15T23:26:06.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 -- 172.21.15.89:0/2990056811 wait complete. 2024-09-15T23:26:06.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 Processor -- start 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.089+0000 7fc41866b640 1 -- start start 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc41866b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc410101df0 0x7fc4100744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc41866b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4101027a0 0x7fc410074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc41866b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc410106f20 0x7fc410074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc41866b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc410111f80 con 0x7fc4101027a0 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc41866b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc410111e00 con 0x7fc410101df0 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc41866b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc410112100 con 0x7fc410106f20 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc415c06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4101027a0 0x7fc410074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc415c06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4101027a0 0x7fc410074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55276/0 (socket says 172.21.15.89:55276) 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc415c06640 1 -- 172.21.15.89:0/4126804661 learned_addr learned my addr 172.21.15.89:0/4126804661 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc416c08640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc410106f20 0x7fc410074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc416407640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc410101df0 0x7fc4100744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc415c06640 1 -- 172.21.15.89:0/4126804661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc410106f20 msgr2=0x7fc410074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc415c06640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc410106f20 0x7fc410074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc415c06640 1 -- 172.21.15.89:0/4126804661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc410101df0 msgr2=0x7fc4100744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc415c06640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc410101df0 0x7fc4100744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc415c06640 1 -- 172.21.15.89:0/4126804661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4101a6950 con 0x7fc4101027a0 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.090+0000 7fc415c06640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4101027a0 0x7fc410074a30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc404006010 tx=0x7fc4040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.091+0000 7fc4037fe640 1 -- 172.21.15.89:0/4126804661 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40403f030 con 0x7fc4101027a0 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.091+0000 7fc41866b640 1 -- 172.21.15.89:0/4126804661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4101a6be0 con 0x7fc4101027a0 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.091+0000 7fc4037fe640 1 -- 172.21.15.89:0/4126804661 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc404003ce0 con 0x7fc4101027a0 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.091+0000 7fc4037fe640 1 -- 172.21.15.89:0/4126804661 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404035070 con 0x7fc4101027a0 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.091+0000 7fc41866b640 1 -- 172.21.15.89:0/4126804661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4101a70c0 con 0x7fc4101027a0 2024-09-15T23:26:06.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.093+0000 7fc4037fe640 1 -- 172.21.15.89:0/4126804661 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc404042030 con 0x7fc4101027a0 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.093+0000 7fc41aa44740 1 Processor -- start 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc41aa44740 1 -- start start 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053ab60 0x55c67053af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c670545800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c670548130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc41aa44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c6704a0280 con 0x55c670545d40 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc41aa44740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c670447b10 con 0x55c67053b480 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc41aa44740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c67049c240 con 0x55c67053ab60 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc415c06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c670548130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc415c06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c670548130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55284/0 (socket says 172.21.15.89:55284) 2024-09-15T23:26:06.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc415c06640 1 -- 172.21.15.89:0/558331759 learned_addr learned my addr 172.21.15.89:0/558331759 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc416c08640 1 --2- 172.21.15.89:0/558331759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053ab60 0x55c67053af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc416407640 1 --2- 172.21.15.89:0/558331759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c670545800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc415c06640 1 -- 172.21.15.89:0/558331759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053ab60 msgr2=0x55c67053af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc415c06640 1 --2- 172.21.15.89:0/558331759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053ab60 0x55c67053af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc415c06640 1 -- 172.21.15.89:0/558331759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 msgr2=0x55c670545800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc415c06640 1 --2- 172.21.15.89:0/558331759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c670545800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc4037fe640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3fc0a3240 0x7fc3fc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.094+0000 7fc415c06640 1 -- 172.21.15.89:0/558331759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6704541f0 con 0x55c670545d40 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.095+0000 7fc416c08640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3fc0a3240 0x7fc3fc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.095+0000 7fc415c06640 1 --2- 172.21.15.89:0/558331759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c670548130 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc4040ba370 tx=0x7fc4040f3f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2406cf8ae62c85c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.095+0000 7fc4037fe640 1 -- 172.21.15.89:0/4126804661 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc4040f0160 con 0x7fc4101027a0 2024-09-15T23:26:06.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.095+0000 7fc400ff9640 1 -- 172.21.15.89:0/558331759 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404049070 con 0x55c670545d40 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.095+0000 7fc400ff9640 1 -- 172.21.15.89:0/558331759 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc40403b580 con 0x55c670545d40 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.095+0000 7fc400ff9640 1 -- 172.21.15.89:0/558331759 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4040bc2d0 con 0x55c670545d40 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 -- 172.21.15.89:0/558331759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 msgr2=0x55c670548130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc416c08640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3fc0a3240 0x7fc3fc0a5700 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc40800ae90 tx=0x7fc40800a4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 --2- 172.21.15.89:0/558331759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c670548130 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc4040ba370 tx=0x7fc4040f3f90 comp rx=0 tx=0).stop 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 -- 172.21.15.89:0/558331759 shutdown_connections 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 --2- 172.21.15.89:0/558331759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c670548130 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 --2- 172.21.15.89:0/558331759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c670545800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 --2- 172.21.15.89:0/558331759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053ab60 0x55c67053af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 -- 172.21.15.89:0/558331759 >> 172.21.15.89:0/558331759 conn(0x55c6705370f0 msgr2=0x55c6705374d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:06.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 -- 172.21.15.89:0/558331759 shutdown_connections 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 -- 172.21.15.89:0/558331759 wait complete. 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 -- start start 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 Processor -- start 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c670542bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c670543100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 0x55c670543640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c6705486d0 con 0x55c67053ab60 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c67045af10 con 0x55c67053b480 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c670548850 con 0x55c670545d40 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc41aa44740 10 client.0 _ll_register_callbacks cb 0x55c670423a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.096+0000 7fc416407640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c670542bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc416407640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c670542bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55288/0 (socket says 172.21.15.89:55288) 2024-09-15T23:26:06.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc416407640 1 -- 172.21.15.89:0/1930829212 learned_addr learned my addr 172.21.15.89:0/1930829212 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:06.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc415c06640 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c670543100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc416407640 1 -- 172.21.15.89:0/1930829212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 msgr2=0x55c670543640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:06.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc416c08640 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 0x55c670543640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc416407640 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 0x55c670543640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc416407640 1 -- 172.21.15.89:0/1930829212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 msgr2=0x55c670543100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:06.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc416407640 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c670543100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc416407640 1 -- 172.21.15.89:0/1930829212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6702116a0 con 0x55c67053ab60 2024-09-15T23:26:06.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc416c08640 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 0x55c670543640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:06.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc415c06640 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c670543100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:06.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc416407640 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c670542bc0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00b920 tx=0x7fc40c00bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40213c4d8baefe7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc3f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/1930829212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c01b440 con 0x55c67053ab60 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/1930829212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc40c01b5e0 con 0x55c67053ab60 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc41aa44740 1 -- 172.21.15.89:0/1930829212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c6704ad7f0 con 0x55c67053ab60 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/1930829212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc40c01b820 con 0x55c67053ab60 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc41aa44740 10 client.16236 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.097+0000 7fc41aa44740 1 -- 172.21.15.89:0/1930829212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c670554250 con 0x55c67053ab60 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.098+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/1930829212 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c6704ad7f0 con 0x55c67053ab60 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.098+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/1930829212 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc40c019020 con 0x55c67053ab60 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.098+0000 7fc41aa44740 10 client.16236 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.098+0000 7fc41aa44740 10 client.16236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.098+0000 7fc41aa44740 20 client.16236 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.098+0000 7fc41aa44740 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c670555710 0x55c670557bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.098+0000 7fc41aa44740 4 client.16236 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:06.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.098+0000 7fc41aa44740 1 -- 172.21.15.89:0/1930829212 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6704a7d00 con 0x55c670555710 2024-09-15T23:26:06.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.098+0000 7fc415c06640 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c670555710 0x55c670557bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc415c06640 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c670555710 0x55c670557bd0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e528f544335c6eda server_cookie=2f6191431f89fa44 in_seq=0 out_seq=0 2024-09-15T23:26:06.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc3f9ffb640 10 client.16236 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:06.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/1930829212 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c6704a7d00 con 0x55c670555710 2024-09-15T23:26:06.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc3f9ffb640 10 client.16236 handle_command_reply: tid=0 2024-09-15T23:26:06.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc41aa44740 1 client.16236 shutdown 2024-09-15T23:26:06.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc41aa44740 1 -- 172.21.15.89:0/1930829212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 msgr2=0x55c670542bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:06.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc41aa44740 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c670542bc0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00b920 tx=0x7fc40c00bd20 comp rx=0 tx=0).stop 2024-09-15T23:26:06.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc41aa44740 1 -- 172.21.15.89:0/1930829212 shutdown_connections 2024-09-15T23:26:06.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc41aa44740 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c670555710 0x55c670557bd0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc41aa44740 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 0x55c670543640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc3f9ffb640 0 client.16236 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:06.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc41aa44740 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c670543100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc41aa44740 1 --2- 172.21.15.89:0/1930829212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c670542bc0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc41aa44740 1 -- 172.21.15.89:0/1930829212 >> 172.21.15.89:0/1930829212 conn(0x55c6705370f0 msgr2=0x55c67053cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:06.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.099+0000 7fc41aa44740 1 -- 172.21.15.89:0/1930829212 shutdown_connections 2024-09-15T23:26:06.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.100+0000 7fc41aa44740 1 -- 172.21.15.89:0/1930829212 wait complete. 2024-09-15T23:26:06.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.100+0000 7fc41aa44740 20 client.16236 trim_cache size 0 max 16384 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc41aa44740 1 Processor -- start 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc41aa44740 1 -- start start 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c67053ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c67055d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 0x55c67055da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc41aa44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc4101b1070 con 0x55c67053ab60 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc41aa44740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4101b2bb0 con 0x55c67053b480 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc41aa44740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4101b1740 con 0x55c670545d40 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc416407640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c67053ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc416407640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c67053ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55294/0 (socket says 172.21.15.89:55294) 2024-09-15T23:26:06.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc416407640 1 -- 172.21.15.89:0/2137502568 learned_addr learned my addr 172.21.15.89:0/2137502568 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc415c06640 1 --2- 172.21.15.89:0/2137502568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c67055d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc416c08640 1 --2- 172.21.15.89:0/2137502568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 0x55c67055da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc416407640 1 -- 172.21.15.89:0/2137502568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 msgr2=0x55c67055da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc416407640 1 --2- 172.21.15.89:0/2137502568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 0x55c67055da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc416407640 1 -- 172.21.15.89:0/2137502568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 msgr2=0x55c67055d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc416407640 1 --2- 172.21.15.89:0/2137502568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c67055d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.117+0000 7fc416407640 1 -- 172.21.15.89:0/2137502568 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c670536f20 con 0x55c67053ab60 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.118+0000 7fc416407640 1 --2- 172.21.15.89:0/2137502568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c67053ccc0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00c010 tx=0x7fc40c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a1489b54c6f5a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.118+0000 7fc3fbfff640 1 -- 172.21.15.89:0/2137502568 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c01b440 con 0x55c67053ab60 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.118+0000 7fc3fbfff640 1 -- 172.21.15.89:0/2137502568 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc40c01b5e0 con 0x55c67053ab60 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.118+0000 7fc3fbfff640 1 -- 172.21.15.89:0/2137502568 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c005040 con 0x55c67053ab60 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 -- 172.21.15.89:0/2137502568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 msgr2=0x55c67053ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:06.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 --2- 172.21.15.89:0/2137502568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c67053ccc0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00c010 tx=0x7fc40c0039c0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 -- 172.21.15.89:0/2137502568 shutdown_connections 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 --2- 172.21.15.89:0/2137502568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c670545d40 0x55c67055da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 --2- 172.21.15.89:0/2137502568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053b480 0x55c67055d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 --2- 172.21.15.89:0/2137502568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c67053ab60 0x55c67053ccc0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 -- 172.21.15.89:0/2137502568 >> 172.21.15.89:0/2137502568 conn(0x55c6705370f0 msgr2=0x55c67053c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 -- 172.21.15.89:0/2137502568 shutdown_connections 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 -- 172.21.15.89:0/2137502568 wait complete. 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 -- start start 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 Processor -- start 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053ab60 0x55c67054d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053b480 0x55c67054dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c67054a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc4101b24e0 con 0x55c670545d40 2024-09-15T23:26:06.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4101b1e10 con 0x55c67053ab60 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4101af050 con 0x55c67053b480 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc41aa44740 10 client.0 _ll_register_callbacks cb 0x7fc40c004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc416c08640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c67054a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc416c08640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c67054a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55310/0 (socket says 172.21.15.89:55310) 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.119+0000 7fc416c08640 1 -- 172.21.15.89:0/2689231688 learned_addr learned my addr 172.21.15.89:0/2689231688 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc416407640 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053ab60 0x55c67054d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc415c06640 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053b480 0x55c67054dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc416c08640 1 -- 172.21.15.89:0/2689231688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053b480 msgr2=0x55c67054dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc416c08640 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053b480 0x55c67054dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc416c08640 1 -- 172.21.15.89:0/2689231688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053ab60 msgr2=0x55c67054d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc416c08640 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053ab60 0x55c67054d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc416c08640 1 -- 172.21.15.89:0/2689231688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c67054ae90 con 0x55c670545d40 2024-09-15T23:26:06.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc415c06640 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053b480 0x55c67054dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc416c08640 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c67054a950 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc40801bcd0 tx=0x7fc408004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20c4b4e221a688d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc3f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/2689231688 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40802c060 con 0x55c670545d40 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/2689231688 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc40800fb30 con 0x55c670545d40 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc41aa44740 1 -- 172.21.15.89:0/2689231688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc40c01b5e0 con 0x55c670545d40 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/2689231688 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc408033050 con 0x55c670545d40 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc41aa44740 10 client.16242 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.120+0000 7fc41aa44740 1 -- 172.21.15.89:0/2689231688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c6705c0be0 con 0x55c670545d40 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/2689231688 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc40c01b5e0 con 0x55c670545d40 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/2689231688 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc408026070 con 0x55c670545d40 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc41aa44740 10 client.16242 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc41aa44740 10 client.16242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc41aa44740 20 client.16242 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:06.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc41aa44740 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c6705c1390 0x55c6705c37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc41aa44740 4 client.16242 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc41aa44740 1 -- 172.21.15.89:0/2689231688 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c670424110 con 0x55c6705c1390 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc416407640 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c6705c1390 0x55c6705c37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc3f9ffb640 10 client.16242 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.121+0000 7fc416407640 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c6705c1390 0x55c6705c37b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12ef736381439a56 server_cookie=f2b6fd08cc648ab4 in_seq=0 out_seq=0 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc3f9ffb640 1 -- 172.21.15.89:0/2689231688 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c670424110 con 0x55c6705c1390 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc3f9ffb640 10 client.16242 handle_command_reply: tid=0 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 client.16242 shutdown 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 -- 172.21.15.89:0/2689231688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 msgr2=0x55c67054a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c67054a950 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc40801bcd0 tx=0x7fc408004ab0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 -- 172.21.15.89:0/2689231688 shutdown_connections 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c6705c1390 0x55c6705c37b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c670545d40 0x55c67054a950 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c67053b480 0x55c67054dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 --2- 172.21.15.89:0/2689231688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c67053ab60 0x55c67054d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc3f9ffb640 0 client.16242 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 -- 172.21.15.89:0/2689231688 >> 172.21.15.89:0/2689231688 conn(0x55c6705370f0 msgr2=0x55c670541b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 -- 172.21.15.89:0/2689231688 shutdown_connections 2024-09-15T23:26:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 1 -- 172.21.15.89:0/2689231688 wait complete. 2024-09-15T23:26:06.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.123+0000 7fc41aa44740 20 client.16242 trim_cache size 0 max 16384 2024-09-15T23:26:06.131 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:26:06.131 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:26:06.131 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "ed1b8e5e-c872-47c7-aa92-0ce57ebadf47", 2024-09-15T23:26:06.132 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:26:06.132 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:26:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 -- 172.21.15.89:0/4126804661 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3fc0a3240 msgr2=0x7fc3fc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3fc0a3240 0x7fc3fc0a5700 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc40800ae90 tx=0x7fc40800a4d0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 -- 172.21.15.89:0/4126804661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4101027a0 msgr2=0x7fc410074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4101027a0 0x7fc410074a30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc404006010 tx=0x7fc4040042f0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 -- 172.21.15.89:0/4126804661 shutdown_connections 2024-09-15T23:26:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3fc0a3240 0x7fc3fc0a5700 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc410106f20 0x7fc410074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4101027a0 0x7fc410074a30 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 --2- 172.21.15.89:0/4126804661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc410101df0 0x7fc4100744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 -- 172.21.15.89:0/4126804661 >> 172.21.15.89:0/4126804661 conn(0x7fc4100fd8f0 msgr2=0x7fc4100fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 -- 172.21.15.89:0/4126804661 shutdown_connections 2024-09-15T23:26:06.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:06.124+0000 7fc41866b640 1 -- 172.21.15.89:0/4126804661 wait complete. 2024-09-15T23:26:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:06 smithi089 ceph-mon[30933]: pgmap v412: 129 pgs: 129 active+clean; 881 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:06 smithi120 ceph-mon[35250]: pgmap v412: 129 pgs: 129 active+clean; 881 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:06 smithi174 ceph-mon[35586]: pgmap v412: 129 pgs: 129 active+clean; 881 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:06.901 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:26:07.150 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:07.271 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:07 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:26:07.271 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:07 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:07.271 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:07 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:26:07.271 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:07 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:07 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:26:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:07 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:07 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:26:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:07 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:07 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:26:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:07 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:07 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:26:07.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:07 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:07.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.662+0000 7f8c8a185640 1 -- 172.21.15.89:0/909469330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84100680 msgr2=0x7f8c84100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.662+0000 7f8c8a185640 1 --2- 172.21.15.89:0/909469330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84100680 0x7f8c84100ae0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c009900 tx=0x7f8c6c02a410 comp rx=0 tx=0).stop 2024-09-15T23:26:07.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.662+0000 7f8c8a185640 1 -- 172.21.15.89:0/909469330 shutdown_connections 2024-09-15T23:26:07.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.662+0000 7f8c8a185640 1 --2- 172.21.15.89:0/909469330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c84101020 0x7f8c8410d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.662+0000 7f8c8a185640 1 --2- 172.21.15.89:0/909469330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84100680 0x7f8c84100ae0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.662+0000 7f8c8a185640 1 --2- 172.21.15.89:0/909469330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c84105220 0x7f8c84105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.662+0000 7f8c8a185640 1 -- 172.21.15.89:0/909469330 >> 172.21.15.89:0/909469330 conn(0x7f8c84077210 msgr2=0x7f8c84077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.662+0000 7f8c8a185640 1 -- 172.21.15.89:0/909469330 shutdown_connections 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.662+0000 7f8c8a185640 1 -- 172.21.15.89:0/909469330 wait complete. 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c8a185640 1 Processor -- start 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c8a185640 1 -- start start 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c8a185640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c84100680 0x7f8c84197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c8a185640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c84101020 0x7f8c84197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c8a185640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84105220 0x7f8c8419bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c8a185640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84110fb0 con 0x7f8c84105220 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c8a185640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84110e30 con 0x7f8c84101020 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c8a185640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84111130 con 0x7f8c84100680 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c83fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84105220 0x7f8c8419bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c83fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84105220 0x7f8c8419bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55318/0 (socket says 172.21.15.89:55318) 2024-09-15T23:26:07.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c83fff640 1 -- 172.21.15.89:0/2702808004 learned_addr learned my addr 172.21.15.89:0/2702808004 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.663+0000 7f8c837fe640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c84100680 0x7f8c84197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c84101020 0x7f8c84197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c83fff640 1 -- 172.21.15.89:0/2702808004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c84100680 msgr2=0x7f8c84197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c83fff640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c84100680 0x7f8c84197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c83fff640 1 -- 172.21.15.89:0/2702808004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c84101020 msgr2=0x7f8c84197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c83fff640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c84101020 0x7f8c84197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c83fff640 1 -- 172.21.15.89:0/2702808004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8419c300 con 0x7f8c84105220 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c83fff640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84105220 0x7f8c8419bc20 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009fd0 tx=0x7f8c7c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c80ff9640 1 -- 172.21.15.89:0/2702808004 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c011830 con 0x7f8c84105220 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c8a185640 1 -- 172.21.15.89:0/2702808004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c8419c5f0 con 0x7f8c84105220 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c80ff9640 1 -- 172.21.15.89:0/2702808004 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c7c016070 con 0x7f8c84105220 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c80ff9640 1 -- 172.21.15.89:0/2702808004 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c00f2e0 con 0x7f8c84105220 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.664+0000 7f8c8a185640 1 -- 172.21.15.89:0/2702808004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c841a3ed0 con 0x7f8c84105220 2024-09-15T23:26:07.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.666+0000 7f8c80ff9640 1 -- 172.21.15.89:0/2702808004 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8c7c0119d0 con 0x7f8c84105220 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.666+0000 7f8c8c55e740 1 Processor -- start 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.666+0000 7f8c8c55e740 1 -- start start 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c04e320 0x56517c04e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c05b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c05b890 0x56517c05dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56517bfb42e0 con 0x56517c04ec40 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56517bf5bce0 con 0x56517c05b890 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56517bfb02a0 con 0x56517c04e320 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c05b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c05b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55322/0 (socket says 172.21.15.89:55322) 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c837fe640 1 -- 172.21.15.89:0/2379375783 learned_addr learned my addr 172.21.15.89:0/2379375783 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:07.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2379375783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c04e320 0x56517c04e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2379375783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c05b890 0x56517c05dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c80ff9640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8c700a3240 0x7f8c700a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c837fe640 1 -- 172.21.15.89:0/2379375783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c04e320 msgr2=0x56517c04e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c837fe640 1 --2- 172.21.15.89:0/2379375783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c04e320 0x56517c04e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c837fe640 1 -- 172.21.15.89:0/2379375783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c05b890 msgr2=0x56517c05dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c837fe640 1 --2- 172.21.15.89:0/2379375783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c05b890 0x56517c05dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c837fe640 1 -- 172.21.15.89:0/2379375783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56517bf40440 con 0x56517c04ec40 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2379375783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c05b890 0x56517c05dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c83fff640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8c700a3240 0x7f8c700a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c80ff9640 1 -- 172.21.15.89:0/2702808004 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8c7c0cf420 con 0x7f8c84105220 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.667+0000 7f8c837fe640 1 --2- 172.21.15.89:0/2379375783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c05b350 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8c74007ed0 tx=0x7f8c7400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6340b4354cece45f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:07.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c79ffb640 1 -- 172.21.15.89:0/2379375783 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74014070 con 0x56517c04ec40 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c83fff640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8c700a3240 0x7f8c700a5700 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c097af0 tx=0x7f8c7c0664c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c79ffb640 1 -- 172.21.15.89:0/2379375783 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c74004d20 con 0x56517c04ec40 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c79ffb640 1 -- 172.21.15.89:0/2379375783 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c740054e0 con 0x56517c04ec40 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2379375783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 msgr2=0x56517c05b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2379375783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c05b350 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8c74007ed0 tx=0x7f8c7400e720 comp rx=0 tx=0).stop 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2379375783 shutdown_connections 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2379375783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c05b890 0x56517c05dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2379375783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c05b350 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2379375783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c04e320 0x56517c04e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2379375783 >> 172.21.15.89:0/2379375783 conn(0x56517c04aa30 msgr2=0x56517c04ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2379375783 shutdown_connections 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.668+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2379375783 wait complete. 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c8c55e740 1 -- start start 2024-09-15T23:26:07.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c8c55e740 1 Processor -- start 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 0x56517c05b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 0x56517c058500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c058af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56517c05e4c0 con 0x56517c04e320 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56517bf6f050 con 0x56517c04ec40 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56517c05e640 con 0x56517c05b890 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c8c55e740 10 client.0 _ll_register_callbacks cb 0x56517bfaf160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c82ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 0x56517c05b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c82ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 0x56517c05b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55332/0 (socket says 172.21.15.89:55332) 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.669+0000 7f8c82ffd640 1 -- 172.21.15.89:0/1771922946 learned_addr learned my addr 172.21.15.89:0/1771922946 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c837fe640 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 0x56517c058500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c058af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c82ffd640 1 -- 172.21.15.89:0/1771922946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 msgr2=0x56517c058af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c058af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c82ffd640 1 -- 172.21.15.89:0/1771922946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 msgr2=0x56517c058500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 0x56517c058500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c82ffd640 1 -- 172.21.15.89:0/1771922946 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56517bd253c0 con 0x56517c04e320 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 0x56517c05b250 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c02a930 tx=0x7f8c6c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8507b8ee67df92cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c62ffd640 1 -- 172.21.15.89:0/1771922946 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c041070 con 0x56517c04e320 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c62ffd640 1 -- 172.21.15.89:0/1771922946 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8c6c031740 con 0x56517c04e320 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c8c55e740 1 -- 172.21.15.89:0/1771922946 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c74004d20 con 0x56517c04e320 2024-09-15T23:26:07.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c62ffd640 1 -- 172.21.15.89:0/1771922946 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c6c051050 con 0x56517c04e320 2024-09-15T23:26:07.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c8c55e740 10 client.16254 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:07.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.670+0000 7f8c8c55e740 1 -- 172.21.15.89:0/1771922946 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56517c069f20 con 0x56517c04e320 2024-09-15T23:26:07.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c62ffd640 1 -- 172.21.15.89:0/1771922946 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c74004d20 con 0x56517c04e320 2024-09-15T23:26:07.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c62ffd640 1 -- 172.21.15.89:0/1771922946 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8c6c03d9a0 con 0x56517c04e320 2024-09-15T23:26:07.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c8c55e740 10 client.16254 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:07.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c8c55e740 10 client.16254 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:07.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c8c55e740 20 client.16254 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:07.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56517c06b3e0 0x56517c06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c8c55e740 4 client.16254 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:07.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c8c55e740 1 -- 172.21.15.89:0/1771922946 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56517bfc1850 con 0x56517c06b3e0 2024-09-15T23:26:07.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c837fe640 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56517c06b3e0 0x56517c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c62ffd640 10 client.16254 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:07.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.671+0000 7f8c837fe640 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56517c06b3e0 0x56517c06d7d0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=450c8673de6d454f server_cookie=3be546742354bbea in_seq=0 out_seq=0 2024-09-15T23:26:07.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c62ffd640 1 -- 172.21.15.89:0/1771922946 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56517bfc1850 con 0x56517c06b3e0 2024-09-15T23:26:07.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c62ffd640 10 client.16254 handle_command_reply: tid=0 2024-09-15T23:26:07.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c8c55e740 1 client.16254 shutdown 2024-09-15T23:26:07.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c8c55e740 1 -- 172.21.15.89:0/1771922946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 msgr2=0x56517c05b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 0x56517c05b250 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c02a930 tx=0x7f8c6c004040 comp rx=0 tx=0).stop 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c8c55e740 1 -- 172.21.15.89:0/1771922946 shutdown_connections 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56517c06b3e0 0x56517c06d7d0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c058af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 0x56517c058500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c62ffd640 0 client.16254 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/1771922946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 0x56517c05b250 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.672+0000 7f8c8c55e740 1 -- 172.21.15.89:0/1771922946 >> 172.21.15.89:0/1771922946 conn(0x56517c04aa30 msgr2=0x56517c0506f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.673+0000 7f8c8c55e740 1 -- 172.21.15.89:0/1771922946 shutdown_connections 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.673+0000 7f8c8c55e740 1 -- 172.21.15.89:0/1771922946 wait complete. 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.673+0000 7f8c8c55e740 20 client.16254 trim_cache size 0 max 16384 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c8c55e740 1 Processor -- start 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c8c55e740 1 -- start start 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04e320 0x56517c067720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c056dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c073060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c841acc80 con 0x56517c04ec40 2024-09-15T23:26:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c841ae7c0 con 0x56517c04e320 2024-09-15T23:26:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c841ad350 con 0x56517c05b890 2024-09-15T23:26:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c056dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c056dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55334/0 (socket says 172.21.15.89:55334) 2024-09-15T23:26:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c837fe640 1 -- 172.21.15.89:0/2160136237 learned_addr learned my addr 172.21.15.89:0/2160136237 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2160136237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04e320 0x56517c067720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2160136237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c837fe640 1 -- 172.21.15.89:0/2160136237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 msgr2=0x56517c073060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c837fe640 1 --2- 172.21.15.89:0/2160136237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c073060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c837fe640 1 -- 172.21.15.89:0/2160136237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04e320 msgr2=0x56517c067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c837fe640 1 --2- 172.21.15.89:0/2160136237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04e320 0x56517c067720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c837fe640 1 -- 172.21.15.89:0/2160136237 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56517c0574b0 con 0x56517c04ec40 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2160136237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04e320 0x56517c067720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.690+0000 7f8c837fe640 1 --2- 172.21.15.89:0/2160136237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c056dd0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x56517c04f3f0 tx=0x7f8c74010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac288acc02c140c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c797fa640 1 -- 172.21.15.89:0/2160136237 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74029020 con 0x56517c04ec40 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c797fa640 1 -- 172.21.15.89:0/2160136237 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c74010b20 con 0x56517c04ec40 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c797fa640 1 -- 172.21.15.89:0/2160136237 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74014050 con 0x56517c04ec40 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2160136237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 msgr2=0x56517c056dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2160136237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c056dd0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x56517c04f3f0 tx=0x7f8c74010f90 comp rx=0 tx=0).stop 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2160136237 shutdown_connections 2024-09-15T23:26:07.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2160136237 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c073060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2160136237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04ec40 0x56517c056dd0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2160136237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04e320 0x56517c067720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2160136237 >> 172.21.15.89:0/2160136237 conn(0x56517c04aa30 msgr2=0x56517c053b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2160136237 shutdown_connections 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2160136237 wait complete. 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.691+0000 7f8c8c55e740 1 -- start start 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c8c55e740 1 Processor -- start 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 0x56517c063d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 0x56517c05f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c8c55e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c05f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c841ae0f0 con 0x56517c04e320 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c841ada20 con 0x56517c04ec40 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c8c55e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c841aac60 con 0x56517c05b890 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c8c55e740 10 client.0 _ll_register_callbacks cb 0x7f8c6c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c837fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 0x56517c05f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c82ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 0x56517c063d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c837fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 0x56517c05f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:47782/0 (socket says 172.21.15.89:47782) 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.692+0000 7f8c837fe640 1 -- 172.21.15.89:0/2264122939 learned_addr learned my addr 172.21.15.89:0/2264122939 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c05f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c837fe640 1 -- 172.21.15.89:0/2264122939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 msgr2=0x56517c05f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c837fe640 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c05f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c837fe640 1 -- 172.21.15.89:0/2264122939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 msgr2=0x56517c063d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c837fe640 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 0x56517c063d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c837fe640 1 -- 172.21.15.89:0/2264122939 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56517c058e60 con 0x56517c04ec40 2024-09-15T23:26:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c05f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c837fe640 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 0x56517c05f130 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400ec20 tx=0x7f8c74010d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c73a90308ed1b5b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:26:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c62ffd640 1 -- 172.21.15.89:0/2264122939 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74029050 con 0x56517c04ec40 2024-09-15T23:26:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2264122939 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c74010b20 con 0x56517c04ec40 2024-09-15T23:26:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c62ffd640 1 -- 172.21.15.89:0/2264122939 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8c74003970 con 0x56517c04ec40 2024-09-15T23:26:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c62ffd640 1 -- 172.21.15.89:0/2264122939 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c74003b70 con 0x56517c04ec40 2024-09-15T23:26:07.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c8c55e740 10 client.25502 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2264122939 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56517c0d5be0 con 0x56517c04ec40 2024-09-15T23:26:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.693+0000 7f8c62ffd640 1 -- 172.21.15.89:0/2264122939 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c74010b20 con 0x56517c04ec40 2024-09-15T23:26:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c62ffd640 1 -- 172.21.15.89:0/2264122939 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8c7402e6b0 con 0x56517c04ec40 2024-09-15T23:26:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c8c55e740 10 client.25502 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c8c55e740 10 client.25502 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c8c55e740 20 client.25502 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56517c0d6190 0x56517c0d85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c8c55e740 4 client.25502 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:26:07.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2264122939 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56517bfbbd60 con 0x56517c0d6190 2024-09-15T23:26:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56517c0d6190 0x56517c0d85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c62ffd640 10 client.25502 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c82ffd640 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56517c0d6190 0x56517c0d85b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e80b78fcf257938 server_cookie=abf013190d2e0dc8 in_seq=0 out_seq=0 2024-09-15T23:26:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c62ffd640 1 -- 172.21.15.89:0/2264122939 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56517bfbbd60 con 0x56517c0d6190 2024-09-15T23:26:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.694+0000 7f8c62ffd640 10 client.25502 handle_command_reply: tid=0 2024-09-15T23:26:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 client.25502 shutdown 2024-09-15T23:26:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2264122939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 msgr2=0x56517c05f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 0x56517c05f130 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400ec20 tx=0x7f8c74010d50 comp rx=0 tx=0).stop 2024-09-15T23:26:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2264122939 shutdown_connections 2024-09-15T23:26:07.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56517c0d6190 0x56517c0d85b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56517c05b890 0x56517c05f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56517c04ec40 0x56517c05f130 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c62ffd640 0 client.25502 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:07.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 --2- 172.21.15.89:0/2264122939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56517c04e320 0x56517c063d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2264122939 >> 172.21.15.89:0/2264122939 conn(0x56517c04aa30 msgr2=0x56517c053b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:07.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2264122939 shutdown_connections 2024-09-15T23:26:07.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 1 -- 172.21.15.89:0/2264122939 wait complete. 2024-09-15T23:26:07.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8c55e740 20 client.25502 trim_cache size 0 max 16384 2024-09-15T23:26:07.716 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:26:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:26:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:26:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "ed1b8e5e-c872-47c7-aa92-0ce57ebadf47": { 2024-09-15T23:26:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:26:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "ed1b8e5e-c872-47c7-aa92-0ce57ebadf47", 2024-09-15T23:26:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:26:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:26:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:26:07.717 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:26:07.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8a185640 1 -- 172.21.15.89:0/2702808004 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8c700a3240 msgr2=0x7f8c700a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8a185640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8c700a3240 0x7f8c700a5700 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c097af0 tx=0x7f8c7c0664c0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8a185640 1 -- 172.21.15.89:0/2702808004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84105220 msgr2=0x7f8c8419bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8a185640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84105220 0x7f8c8419bc20 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c009fd0 tx=0x7f8c7c00b8e0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8a185640 1 -- 172.21.15.89:0/2702808004 shutdown_connections 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8a185640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8c700a3240 0x7f8c700a5700 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8a185640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c84105220 0x7f8c8419bc20 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8a185640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c84101020 0x7f8c84197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.695+0000 7f8c8a185640 1 --2- 172.21.15.89:0/2702808004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c84100680 0x7f8c84197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.696+0000 7f8c8a185640 1 -- 172.21.15.89:0/2702808004 >> 172.21.15.89:0/2702808004 conn(0x7f8c84077210 msgr2=0x7f8c840fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.696+0000 7f8c8a185640 1 -- 172.21.15.89:0/2702808004 shutdown_connections 2024-09-15T23:26:07.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:07.696+0000 7f8c8a185640 1 -- 172.21.15.89:0/2702808004 wait complete. 2024-09-15T23:26:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:08 smithi120 ceph-mon[35250]: pgmap v413: 129 pgs: 129 active+clean; 938 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 18 KiB/s wr, 21 op/s 2024-09-15T23:26:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:08 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:26:08.490 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:08 smithi174 ceph-mon[35586]: pgmap v413: 129 pgs: 129 active+clean; 938 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 18 KiB/s wr, 21 op/s 2024-09-15T23:26:08.490 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:08 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:26:08.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:08 smithi089 ceph-mon[30933]: pgmap v413: 129 pgs: 129 active+clean; 938 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 18 KiB/s wr, 21 op/s 2024-09-15T23:26:08.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:08 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:26:08.811 INFO:tasks.cephfs.filesystem:scrub status for tag:ed1b8e5e-c872-47c7-aa92-0ce57ebadf47 - {'path': '/', 'tag': 'ed1b8e5e-c872-47c7-aa92-0ce57ebadf47', 'options': 'recursive,force'} 2024-09-15T23:26:10.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:10 smithi120 ceph-mon[35250]: pgmap v414: 129 pgs: 129 active+clean; 938 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:26:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:10 smithi174 ceph-mon[35586]: pgmap v414: 129 pgs: 129 active+clean; 938 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:26:10.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:10 smithi089 ceph-mon[30933]: pgmap v414: 129 pgs: 129 active+clean; 938 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:26:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:12 smithi120 ceph-mon[35250]: pgmap v415: 129 pgs: 129 active+clean; 938 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:26:12.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:12 smithi174 ceph-mon[35586]: pgmap v415: 129 pgs: 129 active+clean; 938 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:26:12.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:12 smithi089 ceph-mon[30933]: pgmap v415: 129 pgs: 129 active+clean; 938 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:26:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.152+0000 7fa7a513a640 1 -- 172.21.15.89:0/3705697278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7a0100a40 msgr2=0x7fa7a0100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.152+0000 7fa7a513a640 1 --2- 172.21.15.89:0/3705697278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7a0100a40 0x7fa7a0100e40 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fa790009900 tx=0x7fa79002a410 comp rx=0 tx=0).stop 2024-09-15T23:26:13.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.152+0000 7fa7a513a640 1 -- 172.21.15.89:0/3705697278 shutdown_connections 2024-09-15T23:26:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.152+0000 7fa7a513a640 1 --2- 172.21.15.89:0/3705697278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7a01025f0 0x7fa7a0108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.152+0000 7fa7a513a640 1 --2- 172.21.15.89:0/3705697278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7a0101c30 0x7fa7a01020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.152+0000 7fa7a513a640 1 --2- 172.21.15.89:0/3705697278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7a0100a40 0x7fa7a0100e40 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.152+0000 7fa7a513a640 1 -- 172.21.15.89:0/3705697278 >> 172.21.15.89:0/3705697278 conn(0x7fa7a00fc1e0 msgr2=0x7fa7a00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.152+0000 7fa7a513a640 1 -- 172.21.15.89:0/3705697278 shutdown_connections 2024-09-15T23:26:13.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.152+0000 7fa7a513a640 1 -- 172.21.15.89:0/3705697278 wait complete. 2024-09-15T23:26:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa7a513a640 1 Processor -- start 2024-09-15T23:26:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa7a513a640 1 -- start start 2024-09-15T23:26:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa7a513a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7a0100a40 0x7fa7a006a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa7a513a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7a0101c30 0x7fa7a019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa7a513a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7a01025f0 0x7fa7a019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa7a513a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010c6c0 con 0x7fa7a0101c30 2024-09-15T23:26:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa7a513a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010c540 con 0x7fa7a01025f0 2024-09-15T23:26:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa7a513a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010c840 con 0x7fa7a0100a40 2024-09-15T23:26:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa79e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7a0101c30 0x7fa7a019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa79e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7a0101c30 0x7fa7a019caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55360/0 (socket says 172.21.15.89:55360) 2024-09-15T23:26:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa79e59c640 1 -- 172.21.15.89:0/834918186 learned_addr learned my addr 172.21.15.89:0/834918186 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.153+0000 7fa79ed9d640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7a0100a40 0x7fa7a006a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa79f59e640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7a01025f0 0x7fa7a019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa79e59c640 1 -- 172.21.15.89:0/834918186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7a0100a40 msgr2=0x7fa7a006a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa79e59c640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7a0100a40 0x7fa7a006a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa79e59c640 1 -- 172.21.15.89:0/834918186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7a01025f0 msgr2=0x7fa7a019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa79e59c640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7a01025f0 0x7fa7a019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa79e59c640 1 -- 172.21.15.89:0/834918186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a019f9d0 con 0x7fa7a0101c30 2024-09-15T23:26:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa79e59c640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7a0101c30 0x7fa7a019caa0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa79400c920 tx=0x7fa79400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa787fff640 1 -- 172.21.15.89:0/834918186 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794007c70 con 0x7fa7a0101c30 2024-09-15T23:26:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa787fff640 1 -- 172.21.15.89:0/834918186 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa794007e10 con 0x7fa7a0101c30 2024-09-15T23:26:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa7a513a640 1 -- 172.21.15.89:0/834918186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a019fc60 con 0x7fa7a0101c30 2024-09-15T23:26:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa787fff640 1 -- 172.21.15.89:0/834918186 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794018890 con 0x7fa7a0101c30 2024-09-15T23:26:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.154+0000 7fa7a513a640 1 -- 172.21.15.89:0/834918186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a01a01b0 con 0x7fa7a0101c30 2024-09-15T23:26:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.156+0000 7fa787fff640 1 -- 172.21.15.89:0/834918186 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa794018a70 con 0x7fa7a0101c30 2024-09-15T23:26:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.157+0000 7fa7a7ed4740 1 Processor -- start 2024-09-15T23:26:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.157+0000 7fa7a7ed4740 1 -- start start 2024-09-15T23:26:13.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.157+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f5290da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.157+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f5295da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.157+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 0x5555f5298830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.157+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5555f51f42f0 con 0x5555f52909a0 2024-09-15T23:26:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.157+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5555f519c270 con 0x5555f52912e0 2024-09-15T23:26:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.157+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5555f51f02b0 con 0x5555f5296370 2024-09-15T23:26:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.157+0000 7fa787fff640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7880a32f0 0x7fa7880a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.157+0000 7fa79f59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f5290da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79f59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f5290da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55362/0 (socket says 172.21.15.89:55362) 2024-09-15T23:26:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79f59e640 1 -- 172.21.15.89:0/2167262296 learned_addr learned my addr 172.21.15.89:0/2167262296 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79e59c640 1 --2- 172.21.15.89:0/2167262296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 0x5555f5298830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79f59e640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7880a32f0 0x7fa7880a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79ed9d640 1 --2- 172.21.15.89:0/2167262296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f5295da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa787fff640 1 -- 172.21.15.89:0/834918186 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa7940cff60 con 0x7fa7a0101c30 2024-09-15T23:26:13.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79f59e640 1 -- 172.21.15.89:0/2167262296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 msgr2=0x5555f5298830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79f59e640 1 --2- 172.21.15.89:0/2167262296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 0x5555f5298830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79f59e640 1 -- 172.21.15.89:0/2167262296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 msgr2=0x5555f5295da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79f59e640 1 --2- 172.21.15.89:0/2167262296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f5295da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79f59e640 1 -- 172.21.15.89:0/2167262296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555f519b340 con 0x5555f52909a0 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa79f59e640 1 --2- 172.21.15.89:0/2167262296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f5290da0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa7980149c0 tx=0x7fa798014e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57235191c2f4aa2f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa7857fa640 1 -- 172.21.15.89:0/2167262296 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79801eb40 con 0x5555f52909a0 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa7857fa640 1 -- 172.21.15.89:0/2167262296 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa798023070 con 0x5555f52909a0 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.158+0000 7fa7857fa640 1 -- 172.21.15.89:0/2167262296 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798013b40 con 0x5555f52909a0 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/2167262296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 msgr2=0x5555f5290da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/2167262296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f5290da0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa7980149c0 tx=0x7fa798014e80 comp rx=0 tx=0).stop 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa79f59e640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7880a32f0 0x7fa7880a57b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa798022000 tx=0x7fa798020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/2167262296 shutdown_connections 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/2167262296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 0x5555f5298830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/2167262296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f5295da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/2167262296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f5290da0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/2167262296 >> 172.21.15.89:0/2167262296 conn(0x5555f528c010 msgr2=0x5555f528c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/2167262296 shutdown_connections 2024-09-15T23:26:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/2167262296 wait complete. 2024-09-15T23:26:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.159+0000 7fa7a7ed4740 1 -- start start 2024-09-15T23:26:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa7a7ed4740 1 Processor -- start 2024-09-15T23:26:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f529ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f529b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 0x5555f52a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5555f5299340 con 0x5555f52909a0 2024-09-15T23:26:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5555f517d290 con 0x5555f52912e0 2024-09-15T23:26:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5555f52994c0 con 0x5555f5296370 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa7a7ed4740 10 client.0 _ll_register_callbacks cb 0x5555f51ae140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f529ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f529ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55376/0 (socket says 172.21.15.89:55376) 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79ed9d640 1 -- 172.21.15.89:0/1920224589 learned_addr learned my addr 172.21.15.89:0/1920224589 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79e59c640 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f529b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79f59e640 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 0x5555f52a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79ed9d640 1 -- 172.21.15.89:0/1920224589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 msgr2=0x5555f52a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79ed9d640 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 0x5555f52a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79ed9d640 1 -- 172.21.15.89:0/1920224589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 msgr2=0x5555f529b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79ed9d640 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f529b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79ed9d640 1 -- 172.21.15.89:0/1920224589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555f4f07210 con 0x5555f52909a0 2024-09-15T23:26:13.180 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79f59e640 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 0x5555f52a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa79ed9d640 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f529ab70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa790009900 tx=0x7fa790031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef05ad74ff6d9c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.160+0000 7fa782ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa782ff5640 1 -- 172.21.15.89:0/1920224589 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790035070 con 0x5555f52909a0 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa782ff5640 1 -- 172.21.15.89:0/1920224589 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa7900042c0 con 0x5555f52909a0 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/1920224589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa798023070 con 0x5555f52909a0 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa782ff5640 1 -- 172.21.15.89:0/1920224589 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7900044c0 con 0x5555f52909a0 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa7a7ed4740 10 client.16272 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/1920224589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5555f4fd3620 con 0x5555f52909a0 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa782ff5640 1 -- 172.21.15.89:0/1920224589 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa798023070 con 0x5555f52909a0 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa782ff5640 1 -- 172.21.15.89:0/1920224589 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa7900035f0 con 0x5555f52909a0 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa7a7ed4740 10 client.16272 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa7a7ed4740 10 client.16272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa7a7ed4740 20 client.16272 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:13.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555f52a6830 0x5555f52a8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa7a7ed4740 4 client.16272 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.161+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/1920224589 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555f5201a20 con 0x5555f52a6830 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.162+0000 7fa79e59c640 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555f52a6830 0x5555f52a8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.162+0000 7fa782ff5640 10 client.16272 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.162+0000 7fa79e59c640 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555f52a6830 0x5555f52a8cc0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4261f0d7ce8f2f7b server_cookie=96fd775059551531 in_seq=0 out_seq=0 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa782ff5640 1 -- 172.21.15.89:0/1920224589 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5555f5201a20 con 0x5555f52a6830 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa782ff5640 10 client.16272 handle_command_reply: tid=0 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 client.16272 shutdown 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/1920224589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 msgr2=0x5555f529ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f529ab70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa790009900 tx=0x7fa790031a20 comp rx=0 tx=0).stop 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/1920224589 shutdown_connections 2024-09-15T23:26:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555f52a6830 0x5555f52a8cc0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f5296370 0x5555f52a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f529b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/1920224589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52909a0 0x5555f529ab70 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa782ff5640 0 client.16272 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/1920224589 >> 172.21.15.89:0/1920224589 conn(0x5555f528c010 msgr2=0x5555f528d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/1920224589 shutdown_connections 2024-09-15T23:26:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/1920224589 wait complete. 2024-09-15T23:26:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.163+0000 7fa7a7ed4740 20 client.16272 trim_cache size 0 max 16384 2024-09-15T23:26:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa7a7ed4740 1 Processor -- start 2024-09-15T23:26:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa7a7ed4740 1 -- start start 2024-09-15T23:26:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f52909a0 0x5555f528cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f52a2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f5296370 0x5555f528d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01aa700 con 0x5555f5296370 2024-09-15T23:26:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01ac240 con 0x5555f52912e0 2024-09-15T23:26:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01aadd0 con 0x5555f52909a0 2024-09-15T23:26:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa79f59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f5296370 0x5555f528d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa79f59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f5296370 0x5555f528d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55380/0 (socket says 172.21.15.89:55380) 2024-09-15T23:26:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa79f59e640 1 -- 172.21.15.89:0/353853107 learned_addr learned my addr 172.21.15.89:0/353853107 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa79e59c640 1 --2- 172.21.15.89:0/353853107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f52a2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.178+0000 7fa79ed9d640 1 --2- 172.21.15.89:0/353853107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f52909a0 0x5555f528cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.179+0000 7fa79f59e640 1 -- 172.21.15.89:0/353853107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f52909a0 msgr2=0x5555f528cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.179+0000 7fa79f59e640 1 --2- 172.21.15.89:0/353853107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f52909a0 0x5555f528cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.179+0000 7fa79f59e640 1 -- 172.21.15.89:0/353853107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 msgr2=0x5555f52a2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.179+0000 7fa79f59e640 1 --2- 172.21.15.89:0/353853107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f52a2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.179+0000 7fa79f59e640 1 -- 172.21.15.89:0/353853107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555f528a650 con 0x5555f5296370 2024-09-15T23:26:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.179+0000 7fa79f59e640 1 --2- 172.21.15.89:0/353853107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f5296370 0x5555f528d460 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa798013cb0 tx=0x7fa798014fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=939c280c13a41a6c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.179+0000 7fa784ff9640 1 -- 172.21.15.89:0/353853107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798013de0 con 0x5555f5296370 2024-09-15T23:26:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.179+0000 7fa784ff9640 1 -- 172.21.15.89:0/353853107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa798023070 con 0x5555f5296370 2024-09-15T23:26:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.179+0000 7fa784ff9640 1 -- 172.21.15.89:0/353853107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79801eba0 con 0x5555f5296370 2024-09-15T23:26:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/353853107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f5296370 msgr2=0x5555f528d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/353853107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f5296370 0x5555f528d460 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa798013cb0 tx=0x7fa798014fb0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.192 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/353853107 shutdown_connections 2024-09-15T23:26:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/353853107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f5296370 0x5555f528d460 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/353853107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f52912e0 0x5555f52a2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/353853107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f52909a0 0x5555f528cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/353853107 >> 172.21.15.89:0/353853107 conn(0x5555f528c010 msgr2=0x5555f529a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/353853107 shutdown_connections 2024-09-15T23:26:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/353853107 wait complete. 2024-09-15T23:26:13.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 -- start start 2024-09-15T23:26:13.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 Processor -- start 2024-09-15T23:26:13.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f52909a0 0x5555f529f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52912e0 0x5555f529fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5296370 0x5555f52a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01abb70 con 0x5555f52912e0 2024-09-15T23:26:13.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01ab4a0 con 0x5555f5296370 2024-09-15T23:26:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01a86e0 con 0x5555f52909a0 2024-09-15T23:26:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa7a7ed4740 10 client.0 _ll_register_callbacks cb 0x7fa790009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.180+0000 7fa79e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52912e0 0x5555f529fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52912e0 0x5555f529fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55396/0 (socket says 172.21.15.89:55396) 2024-09-15T23:26:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79e59c640 1 -- 172.21.15.89:0/314438759 learned_addr learned my addr 172.21.15.89:0/314438759 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:13.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79f59e640 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5296370 0x5555f52a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79ed9d640 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f52909a0 0x5555f529f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79e59c640 1 -- 172.21.15.89:0/314438759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f52909a0 msgr2=0x5555f529f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79e59c640 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f52909a0 0x5555f529f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79e59c640 1 -- 172.21.15.89:0/314438759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5296370 msgr2=0x5555f52a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79e59c640 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5296370 0x5555f52a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79e59c640 1 -- 172.21.15.89:0/314438759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555f52a12d0 con 0x5555f52912e0 2024-09-15T23:26:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79f59e640 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5296370 0x5555f52a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa79e59c640 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52912e0 0x5555f529fcb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa79409c850 tx=0x7fa794015aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bece5b4a406053f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa782ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa782ff5640 1 -- 172.21.15.89:0/314438759 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7940a4da0 con 0x5555f52912e0 2024-09-15T23:26:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa782ff5640 1 -- 172.21.15.89:0/314438759 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa794018d60 con 0x5555f52912e0 2024-09-15T23:26:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/314438759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa798023070 con 0x5555f52912e0 2024-09-15T23:26:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa782ff5640 1 -- 172.21.15.89:0/314438759 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa794014210 con 0x5555f52912e0 2024-09-15T23:26:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa7a7ed4740 10 client.16278 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/314438759 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5555f5311730 con 0x5555f52912e0 2024-09-15T23:26:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.181+0000 7fa782ff5640 1 -- 172.21.15.89:0/314438759 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa798023070 con 0x5555f52912e0 2024-09-15T23:26:13.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.182+0000 7fa782ff5640 1 -- 172.21.15.89:0/314438759 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa7940987a0 con 0x5555f52912e0 2024-09-15T23:26:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.182+0000 7fa7a7ed4740 10 client.16278 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.182+0000 7fa7a7ed4740 10 client.16278 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.182+0000 7fa7a7ed4740 20 client.16278 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.182+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555f5312030 0x5555f5314450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.182+0000 7fa7a7ed4740 4 client.16278 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:26:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.182+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/314438759 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5555f51fbd70 con 0x5555f5312030 2024-09-15T23:26:13.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.182+0000 7fa79ed9d640 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555f5312030 0x5555f5314450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.182+0000 7fa782ff5640 10 client.16278 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.182+0000 7fa79ed9d640 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555f5312030 0x5555f5314450 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2307f2853166d0ad server_cookie=882959f484fe7563 in_seq=0 out_seq=0 2024-09-15T23:26:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa782ff5640 1 -- 172.21.15.89:0/314438759 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5555f51fbd70 con 0x5555f5312030 2024-09-15T23:26:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa782ff5640 10 client.16278 handle_command_reply: tid=0 2024-09-15T23:26:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa7a7ed4740 1 client.16278 shutdown 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/314438759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52912e0 msgr2=0x5555f529fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52912e0 0x5555f529fcb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa79409c850 tx=0x7fa794015aa0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/314438759 shutdown_connections 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555f5312030 0x5555f5314450 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5296370 0x5555f52a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa782ff5640 0 client.16278 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555f52912e0 0x5555f529fcb0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa7a7ed4740 1 --2- 172.21.15.89:0/314438759 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555f52909a0 0x5555f529f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/314438759 >> 172.21.15.89:0/314438759 conn(0x5555f528c010 msgr2=0x5555f528a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.196+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/314438759 shutdown_connections 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a7ed4740 1 -- 172.21.15.89:0/314438759 wait complete. 2024-09-15T23:26:13.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a7ed4740 20 client.16278 trim_cache size 0 max 16384 2024-09-15T23:26:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:26:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:26:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:26:13.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 -- 172.21.15.89:0/834918186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7880a32f0 msgr2=0x7fa7880a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7880a32f0 0x7fa7880a57b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa798022000 tx=0x7fa798020000 comp rx=0 tx=0).stop 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 -- 172.21.15.89:0/834918186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7a0101c30 msgr2=0x7fa7a019caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7a0101c30 0x7fa7a019caa0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa79400c920 tx=0x7fa79400cde0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 -- 172.21.15.89:0/834918186 shutdown_connections 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7880a32f0 0x7fa7880a57b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7a01025f0 0x7fa7a019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7a0101c30 0x7fa7a019caa0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 --2- 172.21.15.89:0/834918186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7a0100a40 0x7fa7a006a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 -- 172.21.15.89:0/834918186 >> 172.21.15.89:0/834918186 conn(0x7fa7a00fc1e0 msgr2=0x7fa7a0104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:13.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 -- 172.21.15.89:0/834918186 shutdown_connections 2024-09-15T23:26:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:13.197+0000 7fa7a513a640 1 -- 172.21.15.89:0/834918186 wait complete. 2024-09-15T23:26:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:13 smithi120 ceph-mon[35250]: pgmap v416: 129 pgs: 129 active+clean; 997 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:26:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:13 smithi174 ceph-mon[35586]: pgmap v416: 129 pgs: 129 active+clean; 997 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:26:13.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:13 smithi089 ceph-mon[30933]: pgmap v416: 129 pgs: 129 active+clean; 997 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:26:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:16 smithi120 ceph-mon[35250]: pgmap v417: 129 pgs: 129 active+clean; 997 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:26:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:16 smithi174 ceph-mon[35586]: pgmap v417: 129 pgs: 129 active+clean; 997 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:26:16.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:16 smithi089 ceph-mon[30933]: pgmap v417: 129 pgs: 129 active+clean; 997 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:26:18.403 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:18 smithi120 ceph-mon[35250]: pgmap v418: 129 pgs: 129 active+clean; 1.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:26:18.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:18 smithi174 ceph-mon[35586]: pgmap v418: 129 pgs: 129 active+clean; 1.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:26:18.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:18 smithi089 ceph-mon[30933]: pgmap v418: 129 pgs: 129 active+clean; 1.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:26:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:20 smithi120 ceph-mon[35250]: pgmap v419: 129 pgs: 129 active+clean; 1.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:26:20.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:20 smithi174 ceph-mon[35586]: pgmap v419: 129 pgs: 129 active+clean; 1.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:26:20.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:20 smithi089 ceph-mon[30933]: pgmap v419: 129 pgs: 129 active+clean; 1.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:26:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:22 smithi120 ceph-mon[35250]: pgmap v420: 129 pgs: 129 active+clean; 1.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:26:22.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:22 smithi174 ceph-mon[35586]: pgmap v420: 129 pgs: 129 active+clean; 1.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:26:22.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:22 smithi089 ceph-mon[30933]: pgmap v420: 129 pgs: 129 active+clean; 1.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:26:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:24 smithi120 ceph-mon[35250]: pgmap v421: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:26:24.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:24 smithi174 ceph-mon[35586]: pgmap v421: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:26:24.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:24 smithi089 ceph-mon[30933]: pgmap v421: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:26:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.294+0000 7f7264586640 1 -- 172.21.15.89:0/3760769657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c101c50 msgr2=0x7f725c1020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.294+0000 7f7264586640 1 --2- 172.21.15.89:0/3760769657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c101c50 0x7f725c1020d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f724c009900 tx=0x7f724c02a410 comp rx=0 tx=0).stop 2024-09-15T23:26:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.294+0000 7f7264586640 1 -- 172.21.15.89:0/3760769657 shutdown_connections 2024-09-15T23:26:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.294+0000 7f7264586640 1 --2- 172.21.15.89:0/3760769657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f725c102610 0x7f725c108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.294+0000 7f7264586640 1 --2- 172.21.15.89:0/3760769657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c101c50 0x7f725c1020d0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.294+0000 7f7264586640 1 --2- 172.21.15.89:0/3760769657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f725c100a60 0x7f725c100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.294+0000 7f7264586640 1 -- 172.21.15.89:0/3760769657 >> 172.21.15.89:0/3760769657 conn(0x7f725c0fc1e0 msgr2=0x7f725c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.295+0000 7f7264586640 1 -- 172.21.15.89:0/3760769657 shutdown_connections 2024-09-15T23:26:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.295+0000 7f7264586640 1 -- 172.21.15.89:0/3760769657 wait complete. 2024-09-15T23:26:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.295+0000 7f7264586640 1 Processor -- start 2024-09-15T23:26:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.295+0000 7f7264586640 1 -- start start 2024-09-15T23:26:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.295+0000 7f7264586640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f725c100a60 0x7f725c196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.295+0000 7f7264586640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f725c101c50 0x7f725c1972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.295+0000 7f7264586640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c102610 0x7f725c19e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.295+0000 7f7264586640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f725c10c760 con 0x7f725c102610 2024-09-15T23:26:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.295+0000 7f7264586640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f725c10c5e0 con 0x7f725c100a60 2024-09-15T23:26:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.295+0000 7f7264586640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f725c10c8e0 con 0x7f725c101c50 2024-09-15T23:26:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262b23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c102610 0x7f725c19e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262b23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c102610 0x7f725c19e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46742/0 (socket says 172.21.15.89:46742) 2024-09-15T23:26:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262b23640 1 -- 172.21.15.89:0/2291646495 learned_addr learned my addr 172.21.15.89:0/2291646495 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262322640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f725c100a60 0x7f725c196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7261b21640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f725c101c50 0x7f725c1972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262b23640 1 -- 172.21.15.89:0/2291646495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f725c101c50 msgr2=0x7f725c1972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:26.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262b23640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f725c101c50 0x7f725c1972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262b23640 1 -- 172.21.15.89:0/2291646495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f725c100a60 msgr2=0x7f725c196db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262b23640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f725c100a60 0x7f725c196db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262b23640 1 -- 172.21.15.89:0/2291646495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725c19e9c0 con 0x7f725c102610 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7261b21640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f725c101c50 0x7f725c1972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262322640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f725c100a60 0x7f725c196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7262b23640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c102610 0x7f725c19e320 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f725400eaa0 tx=0x7f725400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f72537fe640 1 -- 172.21.15.89:0/2291646495 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725400cdb0 con 0x7f725c102610 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f72537fe640 1 -- 172.21.15.89:0/2291646495 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7254005020 con 0x7f725c102610 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7264586640 1 -- 172.21.15.89:0/2291646495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f725c19ecb0 con 0x7f725c102610 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f72537fe640 1 -- 172.21.15.89:0/2291646495 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72540116b0 con 0x7f725c102610 2024-09-15T23:26:26.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.296+0000 7f7264586640 1 -- 172.21.15.89:0/2291646495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f725c19f1b0 con 0x7f725c102610 2024-09-15T23:26:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.299+0000 7f72537fe640 1 -- 172.21.15.89:0/2291646495 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7254005560 con 0x7f725c102610 2024-09-15T23:26:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.299+0000 7f72673fe740 1 Processor -- start 2024-09-15T23:26:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.299+0000 7f72673fe740 1 -- start start 2024-09-15T23:26:26.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f72673fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67db3bb0 0x55ad67db3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f72673fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67db8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f72673fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db9580 0x55ad67dbba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f72673fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ad67d152f0 con 0x55ad67db9580 2024-09-15T23:26:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f72673fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ad67cbd270 con 0x55ad67db44f0 2024-09-15T23:26:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f72673fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ad67d112b0 con 0x55ad67db3bb0 2024-09-15T23:26:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7261b21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db9580 0x55ad67dbba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7261b21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db9580 0x55ad67dbba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46748/0 (socket says 172.21.15.89:46748) 2024-09-15T23:26:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7261b21640 1 -- 172.21.15.89:0/1546717250 learned_addr learned my addr 172.21.15.89:0/1546717250 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:26.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f72537fe640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72480a3370 0x7f72480a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7261b21640 1 --2- 172.21.15.89:0/1546717250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67db3bb0 0x55ad67db3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7262322640 1 --2- 172.21.15.89:0/1546717250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67db8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7262b23640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72480a3370 0x7f72480a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7261b21640 1 -- 172.21.15.89:0/1546717250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67db3bb0 msgr2=0x55ad67db3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7261b21640 1 --2- 172.21.15.89:0/1546717250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67db3bb0 0x55ad67db3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7261b21640 1 -- 172.21.15.89:0/1546717250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 msgr2=0x55ad67db8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7261b21640 1 --2- 172.21.15.89:0/1546717250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67db8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7261b21640 1 -- 172.21.15.89:0/1546717250 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad67cbc340 con 0x55ad67db9580 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f72537fe640 1 -- 172.21.15.89:0/2291646495 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f72540d0060 con 0x7f725c102610 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.300+0000 7f7261b21640 1 --2- 172.21.15.89:0/1546717250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db9580 0x55ad67dbba40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f724c005c20 tx=0x7f724c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b8f2342b679b15 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f7250bfe640 1 -- 172.21.15.89:0/1546717250 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c035040 con 0x55ad67db9580 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f7262b23640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72480a3370 0x7f72480a5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f72540cd770 tx=0x7f725406af40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f7250bfe640 1 -- 172.21.15.89:0/1546717250 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f724c004230 con 0x55ad67db9580 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f7250bfe640 1 -- 172.21.15.89:0/1546717250 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c03c670 con 0x55ad67db9580 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f72673fe740 1 -- 172.21.15.89:0/1546717250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db9580 msgr2=0x55ad67dbba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f72673fe740 1 --2- 172.21.15.89:0/1546717250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db9580 0x55ad67dbba40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f724c005c20 tx=0x7f724c031400 comp rx=0 tx=0).stop 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f72673fe740 1 -- 172.21.15.89:0/1546717250 shutdown_connections 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f72673fe740 1 --2- 172.21.15.89:0/1546717250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db9580 0x55ad67dbba40 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f72673fe740 1 --2- 172.21.15.89:0/1546717250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67db8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.312 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f72673fe740 1 --2- 172.21.15.89:0/1546717250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67db3bb0 0x55ad67db3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.301+0000 7f72673fe740 1 -- 172.21.15.89:0/1546717250 >> 172.21.15.89:0/1546717250 conn(0x55ad67dad010 msgr2=0x55ad67dad410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 1 -- 172.21.15.89:0/1546717250 shutdown_connections 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 1 -- 172.21.15.89:0/1546717250 wait complete. 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 1 -- start start 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 1 Processor -- start 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dbddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67dbe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67db9580 0x55ad67dc5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ad67dbc4f0 con 0x55ad67db3bb0 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ad67c9e290 con 0x55ad67db44f0 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ad67dbc670 con 0x55ad67db9580 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f72673fe740 10 client.0 _ll_register_callbacks cb 0x55ad67ccf140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f7261b21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dbddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.313 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f7261b21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dbddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46752/0 (socket says 172.21.15.89:46752) 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f7261b21640 1 -- 172.21.15.89:0/1640784435 learned_addr learned my addr 172.21.15.89:0/1640784435 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.302+0000 7f7261b21640 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67db9580 0x55ad67dc5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7262322640 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67dbe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7261b21640 1 -- 172.21.15.89:0/1640784435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67db9580 msgr2=0x55ad67dc5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7261b21640 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67db9580 0x55ad67dc5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7261b21640 1 -- 172.21.15.89:0/1640784435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 msgr2=0x55ad67dbe310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7261b21640 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67dbe310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7261b21640 1 -- 172.21.15.89:0/1640784435 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad67a28210 con 0x55ad67db3bb0 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7245ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7261b21640 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dbddd0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f724c006010 tx=0x7f724c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c31e3faae49c4d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7245ffb640 1 -- 172.21.15.89:0/1640784435 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c035070 con 0x55ad67db3bb0 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7245ffb640 1 -- 172.21.15.89:0/1640784435 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f724c04c470 con 0x55ad67db3bb0 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f72673fe740 1 -- 172.21.15.89:0/1640784435 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f724c004230 con 0x55ad67db3bb0 2024-09-15T23:26:26.314 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7245ffb640 1 -- 172.21.15.89:0/1640784435 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f724c003a80 con 0x55ad67db3bb0 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f72673fe740 10 client.16290 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f72673fe740 1 -- 172.21.15.89:0/1640784435 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ad67af4620 con 0x55ad67db3bb0 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7245ffb640 1 -- 172.21.15.89:0/1640784435 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f724c004230 con 0x55ad67db3bb0 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.303+0000 7f7245ffb640 1 -- 172.21.15.89:0/1640784435 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f724c03a040 con 0x55ad67db3bb0 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.304+0000 7f72673fe740 10 client.16290 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.304+0000 7f72673fe740 10 client.16290 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.304+0000 7f72673fe740 20 client.16290 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.304+0000 7f72673fe740 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ad67dc9650 0x55ad67dcba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.304+0000 7f72673fe740 4 client.16290 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.304+0000 7f72673fe740 1 -- 172.21.15.89:0/1640784435 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad67d22a20 con 0x55ad67dc9650 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.304+0000 7f7262322640 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ad67dc9650 0x55ad67dcba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.304+0000 7f7262322640 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ad67dc9650 0x55ad67dcba40 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81456822e25c119d server_cookie=6aaa49ac78ee4705 in_seq=0 out_seq=0 2024-09-15T23:26:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.304+0000 7f7245ffb640 10 client.16290 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f7245ffb640 1 -- 172.21.15.89:0/1640784435 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ad67d22a20 con 0x55ad67dc9650 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f7245ffb640 10 client.16290 handle_command_reply: tid=0 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 client.16290 shutdown 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 -- 172.21.15.89:0/1640784435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 msgr2=0x55ad67dbddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dbddd0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f724c006010 tx=0x7f724c005c90 comp rx=0 tx=0).stop 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 -- 172.21.15.89:0/1640784435 shutdown_connections 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ad67dc9650 0x55ad67dcba40 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67db9580 0x55ad67dc5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67dbe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f7245ffb640 0 client.16290 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 --2- 172.21.15.89:0/1640784435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dbddd0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 -- 172.21.15.89:0/1640784435 >> 172.21.15.89:0/1640784435 conn(0x55ad67dad010 msgr2=0x55ad67dae820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 -- 172.21.15.89:0/1640784435 shutdown_connections 2024-09-15T23:26:26.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 1 -- 172.21.15.89:0/1640784435 wait complete. 2024-09-15T23:26:26.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.305+0000 7f72673fe740 20 client.16290 trim_cache size 0 max 16384 2024-09-15T23:26:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f72673fe740 1 Processor -- start 2024-09-15T23:26:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f72673fe740 1 -- start start 2024-09-15T23:26:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f72673fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67dad010 0x55ad67dac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f72673fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db3bb0 0x55ad67dc52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f72673fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db44f0 0x55ad67dbe1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f72673fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f725c1a87e0 con 0x55ad67db44f0 2024-09-15T23:26:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f72673fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f725c1aa320 con 0x55ad67db3bb0 2024-09-15T23:26:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f72673fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f725c1a8eb0 con 0x55ad67dad010 2024-09-15T23:26:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7261b21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db44f0 0x55ad67dbe1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7261b21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db44f0 0x55ad67dbe1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46768/0 (socket says 172.21.15.89:46768) 2024-09-15T23:26:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7261b21640 1 -- 172.21.15.89:0/4043161832 learned_addr learned my addr 172.21.15.89:0/4043161832 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7262322640 1 --2- 172.21.15.89:0/4043161832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db3bb0 0x55ad67dc52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7261b21640 1 --2- 172.21.15.89:0/4043161832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67dad010 0x55ad67dac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7261b21640 1 -- 172.21.15.89:0/4043161832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67dad010 msgr2=0x55ad67dac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7261b21640 1 --2- 172.21.15.89:0/4043161832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67dad010 0x55ad67dac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7261b21640 1 -- 172.21.15.89:0/4043161832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db3bb0 msgr2=0x55ad67dc52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7261b21640 1 --2- 172.21.15.89:0/4043161832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db3bb0 0x55ad67dc52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7261b21640 1 -- 172.21.15.89:0/4043161832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad67db89e0 con 0x55ad67db44f0 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.320+0000 7f7261b21640 1 --2- 172.21.15.89:0/4043161832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db44f0 0x55ad67dbe1f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f724c009840 tx=0x7f724c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67fe772146860574 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f7247fff640 1 -- 172.21.15.89:0/4043161832 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c066070 con 0x55ad67db44f0 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f7247fff640 1 -- 172.21.15.89:0/4043161832 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f724c051070 con 0x55ad67db44f0 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f7247fff640 1 -- 172.21.15.89:0/4043161832 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c042070 con 0x55ad67db44f0 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 -- 172.21.15.89:0/4043161832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db44f0 msgr2=0x55ad67dbe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 --2- 172.21.15.89:0/4043161832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db44f0 0x55ad67dbe1f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f724c009840 tx=0x7f724c009260 comp rx=0 tx=0).stop 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 -- 172.21.15.89:0/4043161832 shutdown_connections 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 --2- 172.21.15.89:0/4043161832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db44f0 0x55ad67dbe1f0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 --2- 172.21.15.89:0/4043161832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db3bb0 0x55ad67dc52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 --2- 172.21.15.89:0/4043161832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67dad010 0x55ad67dac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 -- 172.21.15.89:0/4043161832 >> 172.21.15.89:0/4043161832 conn(0x55ad67db90d0 msgr2=0x55ad67dae6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 -- 172.21.15.89:0/4043161832 shutdown_connections 2024-09-15T23:26:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 -- 172.21.15.89:0/4043161832 wait complete. 2024-09-15T23:26:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 -- start start 2024-09-15T23:26:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.321+0000 7f72673fe740 1 Processor -- start 2024-09-15T23:26:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f72673fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67dad010 0x55ad67dc2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f72673fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dc3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f72673fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67dc4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f72673fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f725c1a9c50 con 0x55ad67db3bb0 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f72673fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f725c1a9580 con 0x55ad67db44f0 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f72673fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f725c1a67c0 con 0x55ad67dad010 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f72673fe740 10 client.0 _ll_register_callbacks cb 0x7f72580071c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7262322640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dc3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7262322640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dc3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46776/0 (socket says 172.21.15.89:46776) 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7262322640 1 -- 172.21.15.89:0/1573245488 learned_addr learned my addr 172.21.15.89:0/1573245488 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7261b21640 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67dad010 0x55ad67dc2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7261b21640 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67dc4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7262322640 1 -- 172.21.15.89:0/1573245488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67dad010 msgr2=0x55ad67dc2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7262322640 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67dad010 0x55ad67dc2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7262322640 1 -- 172.21.15.89:0/1573245488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 msgr2=0x55ad67dc4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:26.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7262322640 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67dc4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7262322640 1 -- 172.21.15.89:0/1573245488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad67dc48a0 con 0x55ad67db3bb0 2024-09-15T23:26:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7261b21640 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67dc4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7262322640 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dc3280 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x55ad67db4f20 tx=0x7f72580157f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1bd4aad1e05126 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7245ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7245ffb640 1 -- 172.21.15.89:0/1573245488 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258024410 con 0x55ad67db3bb0 2024-09-15T23:26:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f7245ffb640 1 -- 172.21.15.89:0/1573245488 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f72580245b0 con 0x55ad67db3bb0 2024-09-15T23:26:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.322+0000 7f72673fe740 1 -- 172.21.15.89:0/1573245488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f724c051070 con 0x55ad67db3bb0 2024-09-15T23:26:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f7245ffb640 1 -- 172.21.15.89:0/1573245488 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f725802d050 con 0x55ad67db3bb0 2024-09-15T23:26:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f72673fe740 10 client.16296 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f72673fe740 1 -- 172.21.15.89:0/1573245488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ad67e34580 con 0x55ad67db3bb0 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f7245ffb640 1 -- 172.21.15.89:0/1573245488 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f724c051070 con 0x55ad67db3bb0 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f7245ffb640 1 -- 172.21.15.89:0/1573245488 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7258014950 con 0x55ad67db3bb0 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f72673fe740 10 client.16296 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f72673fe740 10 client.16296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f72673fe740 20 client.16296 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f72673fe740 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ad67db9580 0x55ad67e36f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f72673fe740 4 client.16296 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f72673fe740 1 -- 172.21.15.89:0/1573245488 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ad67d1cd70 con 0x55ad67db9580 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.323+0000 7f7261b21640 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ad67db9580 0x55ad67e36f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.324+0000 7f7261b21640 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ad67db9580 0x55ad67e36f50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3db8a71d552273a server_cookie=708bd01693e765b1 in_seq=0 out_seq=0 2024-09-15T23:26:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.324+0000 7f7245ffb640 10 client.16296 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f7245ffb640 1 -- 172.21.15.89:0/1573245488 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ad67d1cd70 con 0x55ad67db9580 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f7245ffb640 10 client.16296 handle_command_reply: tid=0 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 client.16296 shutdown 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 -- 172.21.15.89:0/1573245488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 msgr2=0x55ad67dc3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dc3280 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x55ad67db4f20 tx=0x7f72580157f0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 -- 172.21.15.89:0/1573245488 shutdown_connections 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ad67db9580 0x55ad67e36f50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad67db44f0 0x55ad67dc4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f7245ffb640 0 client.16296 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad67db3bb0 0x55ad67dc3280 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 --2- 172.21.15.89:0/1573245488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad67dad010 0x55ad67dc2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 -- 172.21.15.89:0/1573245488 >> 172.21.15.89:0/1573245488 conn(0x55ad67db90d0 msgr2=0x55ad67dab9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 -- 172.21.15.89:0/1573245488 shutdown_connections 2024-09-15T23:26:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 1 -- 172.21.15.89:0/1573245488 wait complete. 2024-09-15T23:26:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f72673fe740 20 client.16296 trim_cache size 0 max 16384 2024-09-15T23:26:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:26:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:26:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:26:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:26:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f7264586640 1 -- 172.21.15.89:0/2291646495 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72480a3370 msgr2=0x7f72480a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f7264586640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72480a3370 0x7f72480a5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f72540cd770 tx=0x7f725406af40 comp rx=0 tx=0).stop 2024-09-15T23:26:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f7264586640 1 -- 172.21.15.89:0/2291646495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c102610 msgr2=0x7f725c19e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.337+0000 7f7264586640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c102610 0x7f725c19e320 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f725400eaa0 tx=0x7f725400ef60 comp rx=0 tx=0).stop 2024-09-15T23:26:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.338+0000 7f7264586640 1 -- 172.21.15.89:0/2291646495 shutdown_connections 2024-09-15T23:26:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.338+0000 7f7264586640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72480a3370 0x7f72480a5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.338+0000 7f7264586640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f725c102610 0x7f725c19e320 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.338+0000 7f7264586640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f725c101c50 0x7f725c1972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.338+0000 7f7264586640 1 --2- 172.21.15.89:0/2291646495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f725c100a60 0x7f725c196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.338+0000 7f7264586640 1 -- 172.21.15.89:0/2291646495 >> 172.21.15.89:0/2291646495 conn(0x7f725c0fc1e0 msgr2=0x7f725c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.338+0000 7f7264586640 1 -- 172.21.15.89:0/2291646495 shutdown_connections 2024-09-15T23:26:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:26.338+0000 7f7264586640 1 -- 172.21.15.89:0/2291646495 wait complete. 2024-09-15T23:26:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:26 smithi120 ceph-mon[35250]: pgmap v422: 129 pgs: 129 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:26:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:26 smithi174 ceph-mon[35586]: pgmap v422: 129 pgs: 129 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:26:26.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:26 smithi089 ceph-mon[30933]: pgmap v422: 129 pgs: 129 active+clean; 1.1 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:26:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:27 smithi120 ceph-mon[35250]: pgmap v423: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:26:27.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:27 smithi174 ceph-mon[35586]: pgmap v423: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:26:27.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:27 smithi089 ceph-mon[30933]: pgmap v423: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:26:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:30 smithi120 ceph-mon[35250]: pgmap v424: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:30.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:30 smithi174 ceph-mon[35586]: pgmap v424: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:30.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:30 smithi089 ceph-mon[30933]: pgmap v424: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:32 smithi120 ceph-mon[35250]: pgmap v425: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:32.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:32 smithi174 ceph-mon[35586]: pgmap v425: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:32.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:32 smithi089 ceph-mon[30933]: pgmap v425: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:34 smithi120 ceph-mon[35250]: pgmap v426: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:26:34.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:34 smithi174 ceph-mon[35586]: pgmap v426: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:26:34.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:34 smithi089 ceph-mon[30933]: pgmap v426: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:26:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:36 smithi120 ceph-mon[35250]: pgmap v427: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:26:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:36 smithi174 ceph-mon[35586]: pgmap v427: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:26:36.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:36 smithi089 ceph-mon[30933]: pgmap v427: 129 pgs: 129 active+clean; 1.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:26:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:38 smithi120 ceph-mon[35250]: pgmap v428: 129 pgs: 129 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:26:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:38 smithi174 ceph-mon[35586]: pgmap v428: 129 pgs: 129 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:26:38.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:38 smithi089 ceph-mon[30933]: pgmap v428: 129 pgs: 129 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:26:38.813 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:26:39.073 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:39.169 INFO:tasks.quiescer.fs.[cephfs]:Total duration (39.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T23:26:39.170 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.55 2024-09-15T23:26:39.170 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:26:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.428+0000 7ff36e90a640 1 -- 172.21.15.89:0/3821307937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff368100a40 msgr2=0x7ff368100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.428+0000 7ff36e90a640 1 --2- 172.21.15.89:0/3821307937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff368100a40 0x7ff368100e40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff358009900 tx=0x7ff35802a410 comp rx=0 tx=0).stop 2024-09-15T23:26:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.428+0000 7ff36e90a640 1 -- 172.21.15.89:0/3821307937 shutdown_connections 2024-09-15T23:26:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.428+0000 7ff36e90a640 1 --2- 172.21.15.89:0/3821307937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3681025f0 0x7ff368108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.428+0000 7ff36e90a640 1 --2- 172.21.15.89:0/3821307937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff368101c30 0x7ff3681020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.428+0000 7ff36e90a640 1 --2- 172.21.15.89:0/3821307937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff368100a40 0x7ff368100e40 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.428+0000 7ff36e90a640 1 -- 172.21.15.89:0/3821307937 >> 172.21.15.89:0/3821307937 conn(0x7ff3680fc1e0 msgr2=0x7ff3680fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.428+0000 7ff36e90a640 1 -- 172.21.15.89:0/3821307937 shutdown_connections 2024-09-15T23:26:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.428+0000 7ff36e90a640 1 -- 172.21.15.89:0/3821307937 wait complete. 2024-09-15T23:26:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36e90a640 1 Processor -- start 2024-09-15T23:26:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36e90a640 1 -- start start 2024-09-15T23:26:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36e90a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff368100a40 0x7ff368197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36e90a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff368101c30 0x7ff3681984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36e90a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3681025f0 0x7ff36819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36e90a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff36810c7f0 con 0x7ff3681025f0 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36e90a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff36810c670 con 0x7ff368101c30 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36e90a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff36810c970 con 0x7ff368100a40 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36cea7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3681025f0 0x7ff36819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36cea7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3681025f0 0x7ff36819f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46346/0 (socket says 172.21.15.89:46346) 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff36cea7640 1 -- 172.21.15.89:0/1977620807 learned_addr learned my addr 172.21.15.89:0/1977620807 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff3677fe640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff368101c30 0x7ff3681984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.429+0000 7ff367fff640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff368100a40 0x7ff368197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff36cea7640 1 -- 172.21.15.89:0/1977620807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff368100a40 msgr2=0x7ff368197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff36cea7640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff368100a40 0x7ff368197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff36cea7640 1 -- 172.21.15.89:0/1977620807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff368101c30 msgr2=0x7ff3681984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff36cea7640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff368101c30 0x7ff3681984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff36cea7640 1 -- 172.21.15.89:0/1977620807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36819fb80 con 0x7ff3681025f0 2024-09-15T23:26:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff367fff640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff368100a40 0x7ff368197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff36cea7640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3681025f0 0x7ff36819f510 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff36000b970 tx=0x7ff36000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff3657fa640 1 -- 172.21.15.89:0/1977620807 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff360019070 con 0x7ff3681025f0 2024-09-15T23:26:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff3657fa640 1 -- 172.21.15.89:0/1977620807 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff36000cb40 con 0x7ff3681025f0 2024-09-15T23:26:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff36e90a640 1 -- 172.21.15.89:0/1977620807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff36819fe70 con 0x7ff3681025f0 2024-09-15T23:26:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff3657fa640 1 -- 172.21.15.89:0/1977620807 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff360007650 con 0x7ff3681025f0 2024-09-15T23:26:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.430+0000 7ff36e90a640 1 -- 172.21.15.89:0/1977620807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3681a0360 con 0x7ff3681025f0 2024-09-15T23:26:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.432+0000 7ff3657fa640 1 -- 172.21.15.89:0/1977620807 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff36001c070 con 0x7ff3681025f0 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3716a2740 1 Processor -- start 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3716a2740 1 -- start start 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 0x562e3820a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820a6f0 0x562e3820f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820f780 0x562e38211c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3716a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562e381692f0 con 0x562e3820f780 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3716a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562e38111270 con 0x562e38209db0 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3657fa640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff3500a32f0 0x7ff3500a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3716a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562e381652b0 con 0x562e3820a6f0 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820f780 0x562e38211c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff36cea7640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff3500a32f0 0x7ff3500a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820f780 0x562e38211c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46350/0 (socket says 172.21.15.89:46350) 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3677fe640 1 -- 172.21.15.89:0/196989348 learned_addr learned my addr 172.21.15.89:0/196989348 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3677fe640 1 --2- 172.21.15.89:0/196989348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 0x562e3820a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3657fa640 1 -- 172.21.15.89:0/1977620807 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff3600cf3f0 con 0x7ff3681025f0 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff367fff640 1 --2- 172.21.15.89:0/196989348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820a6f0 0x562e3820f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3677fe640 1 -- 172.21.15.89:0/196989348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820a6f0 msgr2=0x562e3820f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3677fe640 1 --2- 172.21.15.89:0/196989348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820a6f0 0x562e3820f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3677fe640 1 -- 172.21.15.89:0/196989348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 msgr2=0x562e3820a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3677fe640 1 --2- 172.21.15.89:0/196989348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 0x562e3820a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.433+0000 7ff3677fe640 1 -- 172.21.15.89:0/196989348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e38110340 con 0x562e3820f780 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.434+0000 7ff3677fe640 1 --2- 172.21.15.89:0/196989348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820f780 0x562e38211c40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff35c016530 tx=0x7ff35c00a2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ee5dc464e40918 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.434+0000 7ff36cea7640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff3500a32f0 0x7ff3500a57b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff3600ccaa0 tx=0x7ff360066000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.434+0000 7ff34e7fc640 1 -- 172.21.15.89:0/196989348 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c018430 con 0x562e3820f780 2024-09-15T23:26:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.434+0000 7ff34e7fc640 1 -- 172.21.15.89:0/196989348 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff35c018ab0 con 0x562e3820f780 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.434+0000 7ff34e7fc640 1 -- 172.21.15.89:0/196989348 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c017a00 con 0x562e3820f780 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.434+0000 7ff3716a2740 1 -- 172.21.15.89:0/196989348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820f780 msgr2=0x562e38211c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.434+0000 7ff3716a2740 1 --2- 172.21.15.89:0/196989348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820f780 0x562e38211c40 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff35c016530 tx=0x7ff35c00a2b0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 -- 172.21.15.89:0/196989348 shutdown_connections 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 --2- 172.21.15.89:0/196989348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820f780 0x562e38211c40 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 --2- 172.21.15.89:0/196989348 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820a6f0 0x562e3820f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 --2- 172.21.15.89:0/196989348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 0x562e3820a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 -- 172.21.15.89:0/196989348 >> 172.21.15.89:0/196989348 conn(0x562e38201010 msgr2=0x562e38201410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 -- 172.21.15.89:0/196989348 shutdown_connections 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 -- 172.21.15.89:0/196989348 wait complete. 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 -- start start 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 Processor -- start 2024-09-15T23:26:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 0x562e38213f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e382144b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820f780 0x562e3821b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562e38212690 con 0x562e3820a6f0 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562e380f2290 con 0x562e38209db0 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562e38212810 con 0x562e3820f780 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff3716a2740 10 client.0 _ll_register_callbacks cb 0x562e38123140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff367fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e382144b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff367fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e382144b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46356/0 (socket says 172.21.15.89:46356) 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.435+0000 7ff367fff640 1 -- 172.21.15.89:0/2737704721 learned_addr learned my addr 172.21.15.89:0/2737704721 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff3677fe640 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 0x562e38213f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff367fff640 1 -- 172.21.15.89:0/2737704721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820f780 msgr2=0x562e3821b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff367fff640 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820f780 0x562e3821b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff367fff640 1 -- 172.21.15.89:0/2737704721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 msgr2=0x562e38213f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff367fff640 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 0x562e38213f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff367fff640 1 -- 172.21.15.89:0/2737704721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e37e7c210 con 0x562e3820a6f0 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff3677fe640 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 0x562e38213f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff367fff640 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e382144b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff358009900 tx=0x7ff358005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d497a2270bd44c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff3477fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff3477fe640 1 -- 172.21.15.89:0/2737704721 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff358035030 con 0x562e3820a6f0 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff3477fe640 1 -- 172.21.15.89:0/2737704721 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff3580318b0 con 0x562e3820a6f0 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff3716a2740 1 -- 172.21.15.89:0/2737704721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff35c018ab0 con 0x562e3820a6f0 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff3477fe640 1 -- 172.21.15.89:0/2737704721 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff358005650 con 0x562e3820a6f0 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff3716a2740 10 client.16308 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff3716a2740 1 -- 172.21.15.89:0/2737704721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e37f48620 con 0x562e3820a6f0 2024-09-15T23:26:39.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.436+0000 7ff3477fe640 1 -- 172.21.15.89:0/2737704721 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff35c018ab0 con 0x562e3820a6f0 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.437+0000 7ff3477fe640 1 -- 172.21.15.89:0/2737704721 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff3580035f0 con 0x562e3820a6f0 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.437+0000 7ff3716a2740 10 client.16308 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.437+0000 7ff3716a2740 10 client.16308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.437+0000 7ff3716a2740 20 client.16308 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.437+0000 7ff3716a2740 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562e3821fc00 0x562e38222080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.437+0000 7ff3716a2740 4 client.16308 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.437+0000 7ff3716a2740 1 -- 172.21.15.89:0/2737704721 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e38176a20 con 0x562e3821fc00 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.437+0000 7ff3677fe640 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562e3821fc00 0x562e38222080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.437+0000 7ff3477fe640 10 client.16308 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.437+0000 7ff3677fe640 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562e3821fc00 0x562e38222080 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab654f19e2c5e884 server_cookie=d80891f2d52cf76 in_seq=0 out_seq=0 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3477fe640 1 -- 172.21.15.89:0/2737704721 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562e38176a20 con 0x562e3821fc00 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3477fe640 10 client.16308 handle_command_reply: tid=0 2024-09-15T23:26:39.445 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 client.16308 shutdown 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 -- 172.21.15.89:0/2737704721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 msgr2=0x562e382144b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e382144b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff358009900 tx=0x7ff358005250 comp rx=0 tx=0).stop 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 -- 172.21.15.89:0/2737704721 shutdown_connections 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562e3821fc00 0x562e38222080 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820f780 0x562e3821b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3477fe640 0 client.16308 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e382144b0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 --2- 172.21.15.89:0/2737704721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e38209db0 0x562e38213f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 -- 172.21.15.89:0/2737704721 >> 172.21.15.89:0/2737704721 conn(0x562e38201010 msgr2=0x562e38202820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 -- 172.21.15.89:0/2737704721 shutdown_connections 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 1 -- 172.21.15.89:0/2737704721 wait complete. 2024-09-15T23:26:39.446 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.438+0000 7ff3716a2740 20 client.16308 trim_cache size 0 max 16384 2024-09-15T23:26:39.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.452+0000 7ff3716a2740 1 Processor -- start 2024-09-15T23:26:39.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.452+0000 7ff3716a2740 1 -- start start 2024-09-15T23:26:39.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.452+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e38209db0 0x562e382000b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.452+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e3821b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.452+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3820f780 0x562e38202400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.452+0000 7ff3716a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff3681a87e0 con 0x562e3820a6f0 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.452+0000 7ff3716a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff3681aa320 con 0x562e3820f780 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.452+0000 7ff3716a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff3681a8eb0 con 0x562e38209db0 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff367fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e3821b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff367fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e3821b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46358/0 (socket says 172.21.15.89:46358) 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff367fff640 1 -- 172.21.15.89:0/3118122035 learned_addr learned my addr 172.21.15.89:0/3118122035 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff3677fe640 1 --2- 172.21.15.89:0/3118122035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e38209db0 0x562e382000b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff367fff640 1 -- 172.21.15.89:0/3118122035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e38209db0 msgr2=0x562e382000b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff367fff640 1 --2- 172.21.15.89:0/3118122035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e38209db0 0x562e382000b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff367fff640 1 -- 172.21.15.89:0/3118122035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3820f780 msgr2=0x562e38202400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff367fff640 1 --2- 172.21.15.89:0/3118122035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3820f780 0x562e38202400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff367fff640 1 -- 172.21.15.89:0/3118122035 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e38200e30 con 0x562e3820a6f0 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff3677fe640 1 --2- 172.21.15.89:0/3118122035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e38209db0 0x562e382000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff367fff640 1 --2- 172.21.15.89:0/3118122035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e3821b4b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff358009850 tx=0x7ff35802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0868e4121b3c82c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff34dffb640 1 -- 172.21.15.89:0/3118122035 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff358055070 con 0x562e3820a6f0 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff34dffb640 1 -- 172.21.15.89:0/3118122035 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff358009520 con 0x562e3820a6f0 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.453+0000 7ff34dffb640 1 -- 172.21.15.89:0/3118122035 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff358035070 con 0x562e3820a6f0 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 -- 172.21.15.89:0/3118122035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 msgr2=0x562e3821b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 --2- 172.21.15.89:0/3118122035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e3821b4b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff358009850 tx=0x7ff35802a410 comp rx=0 tx=0).stop 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 -- 172.21.15.89:0/3118122035 shutdown_connections 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 --2- 172.21.15.89:0/3118122035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3820f780 0x562e38202400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 --2- 172.21.15.89:0/3118122035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e3820a6f0 0x562e3821b4b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 --2- 172.21.15.89:0/3118122035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e38209db0 0x562e382000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 -- 172.21.15.89:0/3118122035 >> 172.21.15.89:0/3118122035 conn(0x562e38201010 msgr2=0x562e381ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 -- 172.21.15.89:0/3118122035 shutdown_connections 2024-09-15T23:26:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 -- 172.21.15.89:0/3118122035 wait complete. 2024-09-15T23:26:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 -- start start 2024-09-15T23:26:39.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 Processor -- start 2024-09-15T23:26:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e38209db0 0x562e382189e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3820a6f0 0x562e38218f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820f780 0x562e38219e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff3681a9c50 con 0x562e38209db0 2024-09-15T23:26:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff3681a9580 con 0x562e3820a6f0 2024-09-15T23:26:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.454+0000 7ff3716a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff3681a67c0 con 0x562e3820f780 2024-09-15T23:26:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3716a2740 10 client.0 _ll_register_callbacks cb 0x7ff358009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e38209db0 0x562e382189e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e38209db0 0x562e382189e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46374/0 (socket says 172.21.15.89:46374) 2024-09-15T23:26:39.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3677fe640 1 -- 172.21.15.89:0/1787444728 learned_addr learned my addr 172.21.15.89:0/1787444728 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff367fff640 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3820a6f0 0x562e38218f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3677fe640 1 -- 172.21.15.89:0/1787444728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820f780 msgr2=0x562e38219e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3677fe640 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820f780 0x562e38219e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3677fe640 1 -- 172.21.15.89:0/1787444728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3820a6f0 msgr2=0x562e38218f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3677fe640 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3820a6f0 0x562e38218f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3677fe640 1 -- 172.21.15.89:0/1787444728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e3821a480 con 0x562e38209db0 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3677fe640 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e38209db0 0x562e382189e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff35c016e80 tx=0x7ff35c00a1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c0b26b5b0462c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3477fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3477fe640 1 -- 172.21.15.89:0/1787444728 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c032060 con 0x562e38209db0 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3716a2740 1 -- 172.21.15.89:0/1787444728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff358009520 con 0x562e38209db0 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3477fe640 1 -- 172.21.15.89:0/1787444728 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff35c018430 con 0x562e38209db0 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3477fe640 1 -- 172.21.15.89:0/1787444728 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff35c039050 con 0x562e38209db0 2024-09-15T23:26:39.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3716a2740 10 client.16314 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:39.463 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3716a2740 1 -- 172.21.15.89:0/1787444728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e3821b7d0 con 0x562e38209db0 2024-09-15T23:26:39.463 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.455+0000 7ff3477fe640 1 -- 172.21.15.89:0/1787444728 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff358009520 con 0x562e38209db0 2024-09-15T23:26:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.456+0000 7ff3477fe640 1 -- 172.21.15.89:0/1787444728 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff35c012050 con 0x562e38209db0 2024-09-15T23:26:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.456+0000 7ff3716a2740 10 client.16314 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.456+0000 7ff3716a2740 10 client.16314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.456+0000 7ff3716a2740 20 client.16314 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.456+0000 7ff3716a2740 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562e3828adb0 0x562e3828d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.456+0000 7ff3716a2740 4 client.16314 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:26:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.456+0000 7ff3716a2740 1 -- 172.21.15.89:0/1787444728 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e38170d70 con 0x562e3828adb0 2024-09-15T23:26:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.456+0000 7ff367fff640 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562e3828adb0 0x562e3828d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.456+0000 7ff3477fe640 10 client.16314 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:39.465 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.456+0000 7ff367fff640 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562e3828adb0 0x562e3828d1d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5686c395f90baab6 server_cookie=edcdac837cbeee93 in_seq=0 out_seq=0 2024-09-15T23:26:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3477fe640 1 -- 172.21.15.89:0/1787444728 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e38170d70 con 0x562e3828adb0 2024-09-15T23:26:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3477fe640 10 client.16314 handle_command_reply: tid=0 2024-09-15T23:26:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 client.16314 shutdown 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 -- 172.21.15.89:0/1787444728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e38209db0 msgr2=0x562e382189e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e38209db0 0x562e382189e0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff35c016e80 tx=0x7ff35c00a1b0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 -- 172.21.15.89:0/1787444728 shutdown_connections 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562e3828adb0 0x562e3828d1d0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562e3820f780 0x562e38219e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3477fe640 0 client.16314 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e3820a6f0 0x562e38218f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 --2- 172.21.15.89:0/1787444728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562e38209db0 0x562e382189e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 -- 172.21.15.89:0/1787444728 >> 172.21.15.89:0/1787444728 conn(0x562e38201010 msgr2=0x562e382001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 -- 172.21.15.89:0/1787444728 shutdown_connections 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 1 -- 172.21.15.89:0/1787444728 wait complete. 2024-09-15T23:26:39.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.469+0000 7ff3716a2740 20 client.16314 trim_cache size 0 max 16384 2024-09-15T23:26:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:26:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:26:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:26:39.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:26:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 -- 172.21.15.89:0/1977620807 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff3500a32f0 msgr2=0x7ff3500a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff3500a32f0 0x7ff3500a57b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff3600ccaa0 tx=0x7ff360066000 comp rx=0 tx=0).stop 2024-09-15T23:26:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 -- 172.21.15.89:0/1977620807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3681025f0 msgr2=0x7ff36819f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3681025f0 0x7ff36819f510 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff36000b970 tx=0x7ff36000be30 comp rx=0 tx=0).stop 2024-09-15T23:26:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 -- 172.21.15.89:0/1977620807 shutdown_connections 2024-09-15T23:26:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff3500a32f0 0x7ff3500a57b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3681025f0 0x7ff36819f510 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff368101c30 0x7ff3681984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 --2- 172.21.15.89:0/1977620807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff368100a40 0x7ff368197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 -- 172.21.15.89:0/1977620807 >> 172.21.15.89:0/1977620807 conn(0x7ff3680fc1e0 msgr2=0x7ff3680fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 -- 172.21.15.89:0/1977620807 shutdown_connections 2024-09-15T23:26:39.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:39.470+0000 7ff36e90a640 1 -- 172.21.15.89:0/1977620807 wait complete. 2024-09-15T23:26:39.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:39 smithi120 ceph-mon[35250]: pgmap v429: 129 pgs: 129 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:26:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:39 smithi174 ceph-mon[35586]: pgmap v429: 129 pgs: 129 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:26:39.493 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:39 smithi089 ceph-mon[30933]: pgmap v429: 129 pgs: 129 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:26:39.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.575+0000 7fd53f663640 1 -- 172.21.15.89:0/1920634059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd538102790 msgr2=0x7fd5381067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.575+0000 7fd53f663640 1 --2- 172.21.15.89:0/1920634059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd538102790 0x7fd5381067e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd534009900 tx=0x7fd53402a3f0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.575+0000 7fd53f663640 1 -- 172.21.15.89:0/1920634059 shutdown_connections 2024-09-15T23:26:39.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.575+0000 7fd53f663640 1 --2- 172.21.15.89:0/1920634059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd538106f10 0x7fd53810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.575+0000 7fd53f663640 1 --2- 172.21.15.89:0/1920634059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd538102790 0x7fd5381067e0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.575+0000 7fd53f663640 1 --2- 172.21.15.89:0/1920634059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd538101de0 0x7fd5381021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.575+0000 7fd53f663640 1 -- 172.21.15.89:0/1920634059 >> 172.21.15.89:0/1920634059 conn(0x7fd5380fd8c0 msgr2=0x7fd5380ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.575+0000 7fd53f663640 1 -- 172.21.15.89:0/1920634059 shutdown_connections 2024-09-15T23:26:39.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.575+0000 7fd53f663640 1 -- 172.21.15.89:0/1920634059 wait complete. 2024-09-15T23:26:39.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53f663640 1 Processor -- start 2024-09-15T23:26:39.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53f663640 1 -- start start 2024-09-15T23:26:39.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53f663640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd538101de0 0x7fd5380744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53f663640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd538102790 0x7fd538074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53f663640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd538106f10 0x7fd538074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53f663640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd538111f30 con 0x7fd538106f10 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53f663640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd538111db0 con 0x7fd538102790 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53f663640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd5381120b0 con 0x7fd538101de0 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53dc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd538106f10 0x7fd538074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53dc00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd538106f10 0x7fd538074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46390/0 (socket says 172.21.15.89:46390) 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53dc00640 1 -- 172.21.15.89:0/784859864 learned_addr learned my addr 172.21.15.89:0/784859864 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd538101de0 0x7fd5380744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd538102790 0x7fd538074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53dc00640 1 -- 172.21.15.89:0/784859864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd538101de0 msgr2=0x7fd5380744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53dc00640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd538101de0 0x7fd5380744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53dc00640 1 -- 172.21.15.89:0/784859864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd538102790 msgr2=0x7fd538074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53dc00640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd538102790 0x7fd538074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.576+0000 7fd53dc00640 1 -- 172.21.15.89:0/784859864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5381a6900 con 0x7fd538106f10 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.577+0000 7fd53dc00640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd538106f10 0x7fd538074f70 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd53000eaa0 tx=0x7fd53000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.577+0000 7fd52e7fc640 1 -- 172.21.15.89:0/784859864 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53000cdb0 con 0x7fd538106f10 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.577+0000 7fd53f663640 1 -- 172.21.15.89:0/784859864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5381a6bf0 con 0x7fd538106f10 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.577+0000 7fd52e7fc640 1 -- 172.21.15.89:0/784859864 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd530005020 con 0x7fd538106f10 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.577+0000 7fd52e7fc640 1 -- 172.21.15.89:0/784859864 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5300116b0 con 0x7fd538106f10 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.577+0000 7fd53f663640 1 -- 172.21.15.89:0/784859864 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5381a7100 con 0x7fd538106f10 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.579+0000 7fd52e7fc640 1 -- 172.21.15.89:0/784859864 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd530005560 con 0x7fd538106f10 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd541a3c740 1 Processor -- start 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd541a3c740 1 -- start start 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e498880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 0x55ec2e4a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd541a3c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ec2e3fe2e0 con 0x55ec2e4984a0 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd541a3c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ec2e3a5ce0 con 0x55ec2e4a3680 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd541a3c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ec2e3fa2a0 con 0x55ec2e498dc0 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd52e7fc640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5240a3240 0x7fd5240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53cbfe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e498880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53cbfe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e498880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46404/0 (socket says 172.21.15.89:46404) 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53cbfe640 1 -- 172.21.15.89:0/4036181494 learned_addr learned my addr 172.21.15.89:0/4036181494 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53dc00640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5240a3240 0x7fd5240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/4036181494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/4036181494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 0x55ec2e4a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd52e7fc640 1 -- 172.21.15.89:0/784859864 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd5300cfe30 con 0x7fd538106f10 2024-09-15T23:26:39.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53cbfe640 1 -- 172.21.15.89:0/4036181494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 msgr2=0x55ec2e4a3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/4036181494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53cbfe640 1 -- 172.21.15.89:0/4036181494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 msgr2=0x55ec2e4a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/4036181494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 0x55ec2e4a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.580+0000 7fd53cbfe640 1 -- 172.21.15.89:0/4036181494 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec2e38a440 con 0x55ec2e4984a0 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.581+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/4036181494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e498880 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd534005ec0 tx=0x7fd534000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392cabcf62d5966d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.581+0000 7fd53dc00640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5240a3240 0x7fd5240a5700 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd530098ae0 tx=0x7fd5300c6e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.581+0000 7fd5237fe640 1 -- 172.21.15.89:0/4036181494 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534044020 con 0x55ec2e4984a0 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.581+0000 7fd5237fe640 1 -- 172.21.15.89:0/4036181494 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd534004230 con 0x55ec2e4984a0 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.581+0000 7fd5237fe640 1 -- 172.21.15.89:0/4036181494 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534035040 con 0x55ec2e4984a0 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 -- 172.21.15.89:0/4036181494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 msgr2=0x55ec2e498880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 --2- 172.21.15.89:0/4036181494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e498880 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd534005ec0 tx=0x7fd534000bc0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 -- 172.21.15.89:0/4036181494 shutdown_connections 2024-09-15T23:26:39.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 --2- 172.21.15.89:0/4036181494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 0x55ec2e4a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 --2- 172.21.15.89:0/4036181494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 --2- 172.21.15.89:0/4036181494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e498880 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 -- 172.21.15.89:0/4036181494 >> 172.21.15.89:0/4036181494 conn(0x55ec2e494a30 msgr2=0x55ec2e494e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 -- 172.21.15.89:0/4036181494 shutdown_connections 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 -- 172.21.15.89:0/4036181494 wait complete. 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 -- start start 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 Processor -- start 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4a3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 0x55ec2e4a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ec2e4a6250 con 0x55ec2e4984a0 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ec2e3b9050 con 0x55ec2e4a3680 2024-09-15T23:26:39.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ec2e4a63d0 con 0x55ec2e498dc0 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.582+0000 7fd541a3c740 10 client.0 _ll_register_callbacks cb 0x55ec2e3f9160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd53cbfe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd53cbfe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4a3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46406/0 (socket says 172.21.15.89:46406) 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd53cbfe640 1 -- 172.21.15.89:0/3552367766 learned_addr learned my addr 172.21.15.89:0/3552367766 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd53cbfe640 1 -- 172.21.15.89:0/3552367766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 msgr2=0x55ec2e4a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd53cbfe640 1 -- 172.21.15.89:0/3552367766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 msgr2=0x55ec2e4a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 0x55ec2e4a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd53cbfe640 1 -- 172.21.15.89:0/3552367766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec2e16f3c0 con 0x55ec2e4984a0 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4a3040 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd534003e30 tx=0x7fd5340041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969d9d46310cefeb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd520ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:39.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd520ff9640 1 -- 172.21.15.89:0/3552367766 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534044050 con 0x55ec2e4984a0 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd520ff9640 1 -- 172.21.15.89:0/3552367766 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd5340037f0 con 0x55ec2e4984a0 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd541a3c740 1 -- 172.21.15.89:0/3552367766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd534004230 con 0x55ec2e4984a0 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd520ff9640 1 -- 172.21.15.89:0/3552367766 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd534056050 con 0x55ec2e4984a0 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd541a3c740 10 client.16326 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd541a3c740 1 -- 172.21.15.89:0/3552367766 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ec2e4b1ca0 con 0x55ec2e4984a0 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.583+0000 7fd520ff9640 1 -- 172.21.15.89:0/3552367766 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd534004230 con 0x55ec2e4984a0 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.584+0000 7fd520ff9640 1 -- 172.21.15.89:0/3552367766 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd534004750 con 0x55ec2e4984a0 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.584+0000 7fd541a3c740 10 client.16326 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.584+0000 7fd541a3c740 10 client.16326 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.584+0000 7fd541a3c740 20 client.16326 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.584+0000 7fd541a3c740 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ec2e4b2ff0 0x55ec2e4b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.584+0000 7fd541a3c740 4 client.16326 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.584+0000 7fd541a3c740 1 -- 172.21.15.89:0/3552367766 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec2e40b850 con 0x55ec2e4b2ff0 2024-09-15T23:26:39.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.584+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ec2e4b2ff0 0x55ec2e4b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd520ff9640 10 client.16326 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:39.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ec2e4b2ff0 0x55ec2e4b53e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f96a06e95a33d15 server_cookie=ace1cab003ff6566 in_seq=0 out_seq=0 2024-09-15T23:26:39.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd520ff9640 1 -- 172.21.15.89:0/3552367766 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ec2e40b850 con 0x55ec2e4b2ff0 2024-09-15T23:26:39.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd520ff9640 10 client.16326 handle_command_reply: tid=0 2024-09-15T23:26:39.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd541a3c740 1 client.16326 shutdown 2024-09-15T23:26:39.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd541a3c740 1 -- 172.21.15.89:0/3552367766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 msgr2=0x55ec2e4a3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd541a3c740 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4a3040 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd534003e30 tx=0x7fd5340041c0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd541a3c740 1 -- 172.21.15.89:0/3552367766 shutdown_connections 2024-09-15T23:26:39.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd541a3c740 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ec2e4b2ff0 0x55ec2e4b53e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd541a3c740 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 0x55ec2e4a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd541a3c740 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd520ff9640 0 client.16326 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:39.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd541a3c740 1 --2- 172.21.15.89:0/3552367766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4a3040 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.585+0000 7fd541a3c740 1 -- 172.21.15.89:0/3552367766 >> 172.21.15.89:0/3552367766 conn(0x55ec2e494a30 msgr2=0x55ec2e49a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.586+0000 7fd541a3c740 1 -- 172.21.15.89:0/3552367766 shutdown_connections 2024-09-15T23:26:39.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.586+0000 7fd541a3c740 1 -- 172.21.15.89:0/3552367766 wait complete. 2024-09-15T23:26:39.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.586+0000 7fd541a3c740 20 client.16326 trim_cache size 0 max 16384 2024-09-15T23:26:39.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd541a3c740 1 Processor -- start 2024-09-15T23:26:39.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd541a3c740 1 -- start start 2024-09-15T23:26:39.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e498dc0 0x55ec2e49ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e4a3680 0x55ec2e4ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd541a3c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd5381b0fa0 con 0x55ec2e4984a0 2024-09-15T23:26:39.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd541a3c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd5381b2ae0 con 0x55ec2e498dc0 2024-09-15T23:26:39.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd541a3c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd5381b1670 con 0x55ec2e4a3680 2024-09-15T23:26:39.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53cbfe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53cbfe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46416/0 (socket says 172.21.15.89:46416) 2024-09-15T23:26:39.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53cbfe640 1 -- 172.21.15.89:0/250964477 learned_addr learned my addr 172.21.15.89:0/250964477 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:39.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/250964477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e498dc0 0x55ec2e49ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53cbfe640 1 -- 172.21.15.89:0/250964477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e4a3680 msgr2=0x55ec2e4ba9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/250964477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e4a3680 0x55ec2e4ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53cbfe640 1 -- 172.21.15.89:0/250964477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e498dc0 msgr2=0x55ec2e49ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/250964477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e498dc0 0x55ec2e49ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53cbfe640 1 -- 172.21.15.89:0/250964477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec2e49f2a0 con 0x55ec2e4984a0 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/250964477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e498dc0 0x55ec2e49ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.603+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/250964477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4af4e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd5340096c0 tx=0x7fd534005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705b10d493f1566b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd522ffd640 1 -- 172.21.15.89:0/250964477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534044070 con 0x55ec2e4984a0 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd522ffd640 1 -- 172.21.15.89:0/250964477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd53404d070 con 0x55ec2e4984a0 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd522ffd640 1 -- 172.21.15.89:0/250964477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534035040 con 0x55ec2e4984a0 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd541a3c740 1 -- 172.21.15.89:0/250964477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 msgr2=0x55ec2e4af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd541a3c740 1 --2- 172.21.15.89:0/250964477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4af4e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd5340096c0 tx=0x7fd534005c00 comp rx=0 tx=0).stop 2024-09-15T23:26:39.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd541a3c740 1 -- 172.21.15.89:0/250964477 shutdown_connections 2024-09-15T23:26:39.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd541a3c740 1 --2- 172.21.15.89:0/250964477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e4a3680 0x55ec2e4ba9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd541a3c740 1 --2- 172.21.15.89:0/250964477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e498dc0 0x55ec2e49ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd541a3c740 1 --2- 172.21.15.89:0/250964477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4af4e0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd541a3c740 1 -- 172.21.15.89:0/250964477 >> 172.21.15.89:0/250964477 conn(0x55ec2e494a30 msgr2=0x55ec2e49b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd541a3c740 1 -- 172.21.15.89:0/250964477 shutdown_connections 2024-09-15T23:26:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.604+0000 7fd541a3c740 1 -- 172.21.15.89:0/250964477 wait complete. 2024-09-15T23:26:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd541a3c740 1 -- start start 2024-09-15T23:26:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd541a3c740 1 Processor -- start 2024-09-15T23:26:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4abf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd541a3c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 0x55ec2e4a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd541a3c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd5381b2410 con 0x55ec2e4984a0 2024-09-15T23:26:39.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd541a3c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd5381b1d40 con 0x55ec2e4a3680 2024-09-15T23:26:39.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd541a3c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd5381aef80 con 0x55ec2e498dc0 2024-09-15T23:26:39.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd541a3c740 10 client.0 _ll_register_callbacks cb 0x7fd52800b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:39.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd53cbfe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4abf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd53cbfe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4abf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46424/0 (socket says 172.21.15.89:46424) 2024-09-15T23:26:39.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd53cbfe640 1 -- 172.21.15.89:0/1985603105 learned_addr learned my addr 172.21.15.89:0/1985603105 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:39.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd53cbfe640 1 -- 172.21.15.89:0/1985603105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 msgr2=0x55ec2e4a72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd53cbfe640 1 -- 172.21.15.89:0/1985603105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 msgr2=0x55ec2e4a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:39.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 0x55ec2e4a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.605+0000 7fd53cbfe640 1 -- 172.21.15.89:0/1985603105 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec2e4a33b0 con 0x55ec2e4984a0 2024-09-15T23:26:39.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd53cbfe640 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4abf60 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd534005ec0 tx=0x7fd5340058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b680ec2aba47852 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd520ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd520ff9640 1 -- 172.21.15.89:0/1985603105 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534035070 con 0x55ec2e4984a0 2024-09-15T23:26:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd520ff9640 1 -- 172.21.15.89:0/1985603105 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd534031630 con 0x55ec2e4984a0 2024-09-15T23:26:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd541a3c740 1 -- 172.21.15.89:0/1985603105 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd53404d070 con 0x55ec2e4984a0 2024-09-15T23:26:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd520ff9640 1 -- 172.21.15.89:0/1985603105 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd53406a050 con 0x55ec2e4984a0 2024-09-15T23:26:39.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd541a3c740 10 client.16332 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:39.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd541a3c740 1 -- 172.21.15.89:0/1985603105 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ec2e51d980 con 0x55ec2e4984a0 2024-09-15T23:26:39.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd520ff9640 1 -- 172.21.15.89:0/1985603105 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd53404d070 con 0x55ec2e4984a0 2024-09-15T23:26:39.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd520ff9640 1 -- 172.21.15.89:0/1985603105 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd534048ab0 con 0x55ec2e4984a0 2024-09-15T23:26:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd541a3c740 10 client.16332 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd541a3c740 10 client.16332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd541a3c740 20 client.16332 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd541a3c740 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ec2e51df90 0x55ec2e5203b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.606+0000 7fd541a3c740 4 client.16332 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:26:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.607+0000 7fd541a3c740 1 -- 172.21.15.89:0/1985603105 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ec2e405d60 con 0x55ec2e51df90 2024-09-15T23:26:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.607+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ec2e51df90 0x55ec2e5203b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.607+0000 7fd53d3ff640 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ec2e51df90 0x55ec2e5203b0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8598e8cc5d56038c server_cookie=e8e32ed912a856bf in_seq=0 out_seq=0 2024-09-15T23:26:39.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.607+0000 7fd520ff9640 10 client.16332 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.607+0000 7fd520ff9640 1 -- 172.21.15.89:0/1985603105 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ec2e405d60 con 0x55ec2e51df90 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.607+0000 7fd520ff9640 10 client.16332 handle_command_reply: tid=0 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.607+0000 7fd541a3c740 1 client.16332 shutdown 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.607+0000 7fd541a3c740 1 -- 172.21.15.89:0/1985603105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 msgr2=0x55ec2e4abf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.607+0000 7fd541a3c740 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4abf60 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd534005ec0 tx=0x7fd5340058c0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd541a3c740 1 -- 172.21.15.89:0/1985603105 shutdown_connections 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd541a3c740 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ec2e51df90 0x55ec2e5203b0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd541a3c740 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec2e4a3680 0x55ec2e4a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd541a3c740 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ec2e498dc0 0x55ec2e4a72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd520ff9640 0 client.16332 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd541a3c740 1 --2- 172.21.15.89:0/1985603105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ec2e4984a0 0x55ec2e4abf60 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd541a3c740 1 -- 172.21.15.89:0/1985603105 >> 172.21.15.89:0/1985603105 conn(0x55ec2e494a30 msgr2=0x55ec2e493b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd541a3c740 1 -- 172.21.15.89:0/1985603105 shutdown_connections 2024-09-15T23:26:39.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd541a3c740 1 -- 172.21.15.89:0/1985603105 wait complete. 2024-09-15T23:26:39.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd541a3c740 20 client.16332 trim_cache size 0 max 16384 2024-09-15T23:26:39.631 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:26:39.631 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:26:39.631 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:26:39.631 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:26:39.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd53f663640 1 -- 172.21.15.89:0/784859864 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5240a3240 msgr2=0x7fd5240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd53f663640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5240a3240 0x7fd5240a5700 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd530098ae0 tx=0x7fd5300c6e30 comp rx=0 tx=0).stop 2024-09-15T23:26:39.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd53f663640 1 -- 172.21.15.89:0/784859864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd538106f10 msgr2=0x7fd538074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd53f663640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd538106f10 0x7fd538074f70 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd53000eaa0 tx=0x7fd53000ef60 comp rx=0 tx=0).stop 2024-09-15T23:26:39.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd53f663640 1 -- 172.21.15.89:0/784859864 shutdown_connections 2024-09-15T23:26:39.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd53f663640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5240a3240 0x7fd5240a5700 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd53f663640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd538106f10 0x7fd538074f70 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.608+0000 7fd53f663640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd538102790 0x7fd538074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.609+0000 7fd53f663640 1 --2- 172.21.15.89:0/784859864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd538101de0 0x7fd5380744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.609+0000 7fd53f663640 1 -- 172.21.15.89:0/784859864 >> 172.21.15.89:0/784859864 conn(0x7fd5380fd8c0 msgr2=0x7fd5380fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.609+0000 7fd53f663640 1 -- 172.21.15.89:0/784859864 shutdown_connections 2024-09-15T23:26:39.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:39.609+0000 7fd53f663640 1 -- 172.21.15.89:0/784859864 wait complete. 2024-09-15T23:26:40.469 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:26:40.469 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:26:40.485 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:41.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 -- 172.21.15.89:0/1065300053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c50101e10 msgr2=0x7f9c501021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:41.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 --2- 172.21.15.89:0/1065300053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c50101e10 0x7f9c501021f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9c40009ca0 tx=0x7f9c4002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:26:41.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 -- 172.21.15.89:0/1065300053 shutdown_connections 2024-09-15T23:26:41.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 --2- 172.21.15.89:0/1065300053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c50106f40 0x7f9c5010eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:41.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 --2- 172.21.15.89:0/1065300053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c501027c0 0x7f9c50106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:41.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 --2- 172.21.15.89:0/1065300053 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c50101e10 0x7f9c501021f0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:41.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 -- 172.21.15.89:0/1065300053 >> 172.21.15.89:0/1065300053 conn(0x7f9c500fd8d0 msgr2=0x7f9c500ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:41.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 -- 172.21.15.89:0/1065300053 shutdown_connections 2024-09-15T23:26:41.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 -- 172.21.15.89:0/1065300053 wait complete. 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 Processor -- start 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.137+0000 7f9c5782d640 1 -- start start 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c5782d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c50101e10 0x7f9c50198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c5782d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c501027c0 0x7f9c50198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c5782d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c50106f40 0x7f9c5019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c5782d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9c50111fa0 con 0x7f9c501027c0 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c5782d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9c50111e20 con 0x7f9c50101e10 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c5782d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9c50112120 con 0x7f9c50106f40 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c54dc8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c501027c0 0x7f9c50198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c54dc8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c501027c0 0x7f9c50198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46440/0 (socket says 172.21.15.89:46440) 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c54dc8640 1 -- 172.21.15.89:0/580336697 learned_addr learned my addr 172.21.15.89:0/580336697 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c555c9640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c50101e10 0x7f9c50198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c55dca640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c50106f40 0x7f9c5019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c54dc8640 1 -- 172.21.15.89:0/580336697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c50106f40 msgr2=0x7f9c5019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c54dc8640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c50106f40 0x7f9c5019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c54dc8640 1 -- 172.21.15.89:0/580336697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c50101e10 msgr2=0x7f9c50198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c54dc8640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c50101e10 0x7f9c50198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c54dc8640 1 -- 172.21.15.89:0/580336697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c5019d3c0 con 0x7f9c501027c0 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c54dc8640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c501027c0 0x7f9c50198950 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400d950 tx=0x7f9c4400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c4e7fc640 1 -- 172.21.15.89:0/580336697 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c44017070 con 0x7f9c501027c0 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c4e7fc640 1 -- 172.21.15.89:0/580336697 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c44009c30 con 0x7f9c501027c0 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c4e7fc640 1 -- 172.21.15.89:0/580336697 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4400f6b0 con 0x7f9c501027c0 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.138+0000 7f9c5782d640 1 -- 172.21.15.89:0/580336697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c5019d6b0 con 0x7f9c501027c0 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.139+0000 7f9c5782d640 1 -- 172.21.15.89:0/580336697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c501a4f90 con 0x7f9c501027c0 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.139+0000 7f9c5782d640 1 -- 172.21.15.89:0/580336697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c501021f0 con 0x7f9c501027c0 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.141+0000 7f9c4e7fc640 1 -- 172.21.15.89:0/580336697 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9c440109d0 con 0x7f9c501027c0 2024-09-15T23:26:41.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.141+0000 7f9c4e7fc640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c380a32c0 0x7f9c380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:41.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.141+0000 7f9c555c9640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c380a32c0 0x7f9c380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:41.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.141+0000 7f9c4e7fc640 1 -- 172.21.15.89:0/580336697 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9c440cf910 con 0x7f9c501027c0 2024-09-15T23:26:41.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.142+0000 7f9c555c9640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c380a32c0 0x7f9c380a5780 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9c40006010 tx=0x7f9c40009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:41.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.145+0000 7f9c4e7fc640 1 -- 172.21.15.89:0/580336697 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9c44009de0 con 0x7f9c501027c0 2024-09-15T23:26:41.195 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:41.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.361+0000 7f9c5782d640 1 -- 172.21.15.89:0/580336697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c5010a180 con 0x7f9c501027c0 2024-09-15T23:26:41.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.362+0000 7f9c4e7fc640 1 -- 172.21.15.89:0/580336697 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f9c4409c220 con 0x7f9c501027c0 2024-09-15T23:26:41.363 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:26:41.363 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:26:41.364 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:26:41.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 -- 172.21.15.89:0/580336697 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c380a32c0 msgr2=0x7f9c380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:41.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c380a32c0 0x7f9c380a5780 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9c40006010 tx=0x7f9c40009360 comp rx=0 tx=0).stop 2024-09-15T23:26:41.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 -- 172.21.15.89:0/580336697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c501027c0 msgr2=0x7f9c50198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:41.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c501027c0 0x7f9c50198950 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400d950 tx=0x7f9c4400de10 comp rx=0 tx=0).stop 2024-09-15T23:26:41.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 -- 172.21.15.89:0/580336697 shutdown_connections 2024-09-15T23:26:41.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9c380a32c0 0x7f9c380a5780 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:41.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c50106f40 0x7f9c5019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:41.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c501027c0 0x7f9c50198950 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:41.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 --2- 172.21.15.89:0/580336697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c50101e10 0x7f9c50198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:41.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 -- 172.21.15.89:0/580336697 >> 172.21.15.89:0/580336697 conn(0x7f9c500fd8d0 msgr2=0x7f9c500fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:41.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 -- 172.21.15.89:0/580336697 shutdown_connections 2024-09-15T23:26:41.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:41.364+0000 7f9c5782d640 1 -- 172.21.15.89:0/580336697 wait complete. 2024-09-15T23:26:42.282 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0' 2024-09-15T23:26:42.282 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0 2024-09-15T23:26:42.442 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:42 smithi089 ceph-mon[30933]: pgmap v430: 129 pgs: 129 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:42.443 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:42 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/580336697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:42 smithi120 ceph-mon[35250]: pgmap v430: 129 pgs: 129 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:42 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/580336697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:42.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:42 smithi174 ceph-mon[35586]: pgmap v430: 129 pgs: 129 active+clean; 1.3 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:26:42.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:42 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/580336697' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:42.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.521+0000 7f6227576640 1 -- 172.21.15.89:0/2371352887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6220100e20 msgr2=0x7f6220101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.521+0000 7f6227576640 1 --2- 172.21.15.89:0/2371352887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6220100e20 0x7f6220101200 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f621c009900 tx=0x7f621c02a480 comp rx=0 tx=0).stop 2024-09-15T23:26:42.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.521+0000 7f6227576640 1 -- 172.21.15.89:0/2371352887 shutdown_connections 2024-09-15T23:26:42.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.521+0000 7f6227576640 1 --2- 172.21.15.89:0/2371352887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6220102ef0 0x7f622010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.521+0000 7f6227576640 1 --2- 172.21.15.89:0/2371352887 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6220102570 0x7f62201029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.521+0000 7f6227576640 1 --2- 172.21.15.89:0/2371352887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6220100e20 0x7f6220101200 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.521+0000 7f6227576640 1 -- 172.21.15.89:0/2371352887 >> 172.21.15.89:0/2371352887 conn(0x7f62200fc7d0 msgr2=0x7f62200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:42.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.521+0000 7f6227576640 1 -- 172.21.15.89:0/2371352887 shutdown_connections 2024-09-15T23:26:42.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6227576640 1 -- 172.21.15.89:0/2371352887 wait complete. 2024-09-15T23:26:42.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6227576640 1 Processor -- start 2024-09-15T23:26:42.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6227576640 1 -- start start 2024-09-15T23:26:42.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6227576640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6220100e20 0x7f6220103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6227576640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6220102570 0x7f6220103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6227576640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6220102ef0 0x7f62201041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6227576640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6220110fd0 con 0x7f6220100e20 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6227576640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6220110e50 con 0x7f6220102570 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6227576640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6220111150 con 0x7f6220102ef0 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6225312640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6220100e20 0x7f6220103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6225312640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6220100e20 0x7f6220103750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46458/0 (socket says 172.21.15.89:46458) 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.522+0000 7f6225312640 1 -- 172.21.15.89:0/3524627786 learned_addr learned my addr 172.21.15.89:0/3524627786 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f6224b11640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6220102570 0x7f6220103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f6225312640 1 -- 172.21.15.89:0/3524627786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6220102ef0 msgr2=0x7f62201041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f6225312640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6220102ef0 0x7f62201041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f6225312640 1 -- 172.21.15.89:0/3524627786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6220102570 msgr2=0x7f6220103c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f6225312640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6220102570 0x7f6220103c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f6225312640 1 -- 172.21.15.89:0/3524627786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62201a6a00 con 0x7f6220100e20 2024-09-15T23:26:42.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f6224b11640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6220102570 0x7f6220103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f6225312640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6220100e20 0x7f6220103750 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f621c02aa70 tx=0x7f621c003b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f62167fc640 1 -- 172.21.15.89:0/3524627786 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c03f070 con 0x7f6220100e20 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f62167fc640 1 -- 172.21.15.89:0/3524627786 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f621c031c20 con 0x7f6220100e20 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f6227576640 1 -- 172.21.15.89:0/3524627786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62201a6c90 con 0x7f6220100e20 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f62167fc640 1 -- 172.21.15.89:0/3524627786 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c035030 con 0x7f6220100e20 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.523+0000 7f6227576640 1 -- 172.21.15.89:0/3524627786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62201a7170 con 0x7f6220100e20 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.524+0000 7f6227576640 1 -- 172.21.15.89:0/3524627786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6220101200 con 0x7f6220100e20 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.525+0000 7f62167fc640 1 -- 172.21.15.89:0/3524627786 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f621c042070 con 0x7f6220100e20 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.526+0000 7f62167fc640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f620c0a3240 0x7f620c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.526+0000 7f6224b11640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f620c0a3240 0x7f620c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.526+0000 7f62167fc640 1 -- 172.21.15.89:0/3524627786 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f621c0f0580 con 0x7f6220100e20 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.526+0000 7f6224b11640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f620c0a3240 0x7f620c0a5700 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f62201051e0 tx=0x7f6210005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:42.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.530+0000 7f62167fc640 1 -- 172.21.15.89:0/3524627786 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f621c031dd0 con 0x7f6220100e20 2024-09-15T23:26:42.617 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:42.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.744+0000 7f6227576640 1 -- 172.21.15.89:0/3524627786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62201a73e0 con 0x7f6220100e20 2024-09-15T23:26:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.745+0000 7f62167fc640 1 -- 172.21.15.89:0/3524627786 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f621c0bceb0 con 0x7f6220100e20 2024-09-15T23:26:42.746 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:26:42.746 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:26:42.746 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:26:42.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 -- 172.21.15.89:0/3524627786 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f620c0a3240 msgr2=0x7f620c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f620c0a3240 0x7f620c0a5700 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f62201051e0 tx=0x7f6210005e50 comp rx=0 tx=0).stop 2024-09-15T23:26:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 -- 172.21.15.89:0/3524627786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6220100e20 msgr2=0x7f6220103750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6220100e20 0x7f6220103750 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f621c02aa70 tx=0x7f621c003b60 comp rx=0 tx=0).stop 2024-09-15T23:26:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 -- 172.21.15.89:0/3524627786 shutdown_connections 2024-09-15T23:26:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f620c0a3240 0x7f620c0a5700 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6220102ef0 0x7f62201041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6220102570 0x7f6220103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 --2- 172.21.15.89:0/3524627786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6220100e20 0x7f6220103750 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 -- 172.21.15.89:0/3524627786 >> 172.21.15.89:0/3524627786 conn(0x7f62200fc7d0 msgr2=0x7f622006a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 -- 172.21.15.89:0/3524627786 shutdown_connections 2024-09-15T23:26:42.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:42.747+0000 7f6227576640 1 -- 172.21.15.89:0/3524627786 wait complete. 2024-09-15T23:26:43.445 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:43 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3524627786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:43.476 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:43 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3524627786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:43 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3524627786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:43.533 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:26:43.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.735+0000 7fe7fa75b640 1 -- 172.21.15.89:0/455461197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f4101420 msgr2=0x7fe7f410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.735+0000 7fe7fa75b640 1 --2- 172.21.15.89:0/455461197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f4101420 0x7fe7f410d950 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009cd0 tx=0x7fe7ec02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.735+0000 7fe7fa75b640 1 -- 172.21.15.89:0/455461197 shutdown_connections 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.735+0000 7fe7fa75b640 1 --2- 172.21.15.89:0/455461197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f4101420 0x7fe7f410d950 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.735+0000 7fe7fa75b640 1 --2- 172.21.15.89:0/455461197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f4100a80 0x7fe7f4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.735+0000 7fe7fa75b640 1 --2- 172.21.15.89:0/455461197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7f4106a80 0x7fe7f4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.735+0000 7fe7fa75b640 1 -- 172.21.15.89:0/455461197 >> 172.21.15.89:0/455461197 conn(0x7fe7f40fc7a0 msgr2=0x7fe7f40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.735+0000 7fe7fa75b640 1 -- 172.21.15.89:0/455461197 shutdown_connections 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.735+0000 7fe7fa75b640 1 -- 172.21.15.89:0/455461197 wait complete. 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.735+0000 7fe7fa75b640 1 Processor -- start 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7fa75b640 1 -- start start 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7fa75b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f4100a80 0x7fe7f4102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7fa75b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f4101420 0x7fe7f4102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7fa75b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7f4106a80 0x7fe7f41030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7fa75b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f4110e80 con 0x7fe7f4101420 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7fa75b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f4110d00 con 0x7fe7f4100a80 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7fa75b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f4111000 con 0x7fe7f4106a80 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f4101420 0x7fe7f4102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f4101420 0x7fe7f4102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46468/0 (socket says 172.21.15.89:46468) 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f37fe640 1 -- 172.21.15.89:0/3615613668 learned_addr learned my addr 172.21.15.89:0/3615613668 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f4100a80 0x7fe7f4102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f8cf8640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7f4106a80 0x7fe7f41030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f37fe640 1 -- 172.21.15.89:0/3615613668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7f4106a80 msgr2=0x7fe7f41030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7f4106a80 0x7fe7f41030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f37fe640 1 -- 172.21.15.89:0/3615613668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f4100a80 msgr2=0x7fe7f4102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f4100a80 0x7fe7f4102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f37fe640 1 -- 172.21.15.89:0/3615613668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f41a67d0 con 0x7fe7f4101420 2024-09-15T23:26:43.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.736+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f4100a80 0x7fe7f4102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.737+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f4101420 0x7fe7f4102b90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800c920 tx=0x7fe7e800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.737+0000 7fe7f17fa640 1 -- 172.21.15.89:0/3615613668 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8007c70 con 0x7fe7f4101420 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.737+0000 7fe7fa75b640 1 -- 172.21.15.89:0/3615613668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f41a6ac0 con 0x7fe7f4101420 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.737+0000 7fe7f17fa640 1 -- 172.21.15.89:0/3615613668 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e8007e10 con 0x7fe7f4101420 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.737+0000 7fe7f17fa640 1 -- 172.21.15.89:0/3615613668 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8018890 con 0x7fe7f4101420 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.737+0000 7fe7fa75b640 1 -- 172.21.15.89:0/3615613668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f41a7050 con 0x7fe7f4101420 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.739+0000 7fe7f17fa640 1 -- 172.21.15.89:0/3615613668 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe7e8018a70 con 0x7fe7f4101420 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.739+0000 7fe7fcb34740 1 Processor -- start 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.739+0000 7fe7fcb34740 1 -- start start 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768d7850 0x5580768d7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d8170 0x5580768e24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768e2a30 0x5580768e4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55807683c160 con 0x5580768d8170 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580767e3720 con 0x5580768e2a30 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558076838120 con 0x5580768d7850 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d8170 0x5580768e24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d8170 0x5580768e24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46484/0 (socket says 172.21.15.89:46484) 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f3fff640 1 -- 172.21.15.89:0/3694940082 learned_addr learned my addr 172.21.15.89:0/3694940082 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f8cf8640 1 --2- 172.21.15.89:0/3694940082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768d7850 0x5580768d7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f17fa640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7dc0a3240 0x7fe7dc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/3694940082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768e2a30 0x5580768e4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f8cf8640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7dc0a3240 0x7fe7dc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f3fff640 1 -- 172.21.15.89:0/3694940082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768d7850 msgr2=0x5580768d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/3694940082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768d7850 0x5580768d7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f3fff640 1 -- 172.21.15.89:0/3694940082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768e2a30 msgr2=0x5580768e4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/3694940082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768e2a30 0x5580768e4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f3fff640 1 -- 172.21.15.89:0/3694940082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580765adba0 con 0x5580768d8170 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f17fa640 1 -- 172.21.15.89:0/3615613668 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe7e80cff00 con 0x7fe7f4101420 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.740+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/3694940082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d8170 0x5580768e24f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4008be0 tx=0x7fe7e400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a47d0dad8c10f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7f8cf8640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7dc0a3240 0x7fe7dc0a5700 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec02ae00 tx=0x7fe7ec02a8e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7da7fc640 1 -- 172.21.15.89:0/3694940082 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4014070 con 0x5580768d8170 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7da7fc640 1 -- 172.21.15.89:0/3694940082 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e40047a0 con 0x5580768d8170 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7fcb34740 1 -- 172.21.15.89:0/3694940082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d8170 msgr2=0x5580768e24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/3694940082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d8170 0x5580768e24f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4008be0 tx=0x7fe7e400e720 comp rx=0 tx=0).stop 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7fcb34740 1 -- 172.21.15.89:0/3694940082 shutdown_connections 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/3694940082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768e2a30 0x5580768e4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/3694940082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d8170 0x5580768e24f0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/3694940082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768d7850 0x5580768d7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7fcb34740 1 -- 172.21.15.89:0/3694940082 >> 172.21.15.89:0/3694940082 conn(0x5580768d3de0 msgr2=0x5580768d41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7fcb34740 1 -- 172.21.15.89:0/3694940082 shutdown_connections 2024-09-15T23:26:43.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.741+0000 7fe7fcb34740 1 -- 172.21.15.89:0/3694940082 wait complete. 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7fcb34740 1 -- start start 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7fcb34740 1 Processor -- start 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768d7850 0x5580768df8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768dfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768e2a30 0x5580768e0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5580768e5650 con 0x5580768e2a30 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580767c4900 con 0x5580768d8170 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5580768e57d0 con 0x5580768d7850 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7fcb34740 10 client.0 _ll_register_callbacks cb 0x5580767c2780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7f8cf8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768e2a30 0x5580768e0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7f8cf8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768e2a30 0x5580768e0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46498/0 (socket says 172.21.15.89:46498) 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7f8cf8640 1 -- 172.21.15.89:0/1754220692 learned_addr learned my addr 172.21.15.89:0/1754220692 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:43.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.742+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768d7850 0x5580768df8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7f8cf8640 1 -- 172.21.15.89:0/1754220692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768d7850 msgr2=0x5580768df8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7f8cf8640 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768d7850 0x5580768df8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7f8cf8640 1 -- 172.21.15.89:0/1754220692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 msgr2=0x5580768dfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768dfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7f8cf8640 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768dfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7f8cf8640 1 -- 172.21.15.89:0/1754220692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558076804c50 con 0x5580768e2a30 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768dfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7f8cf8640 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768e2a30 0x5580768e0370 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec0315c0 tx=0x7fe7ec0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0fda5a7b519ffc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7d37fe640 1 -- 172.21.15.89:0/1754220692 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec04f020 con 0x5580768e2a30 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7d37fe640 1 -- 172.21.15.89:0/1754220692 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe7ec0305a0 con 0x5580768e2a30 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7fcb34740 1 -- 172.21.15.89:0/1754220692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7e40047a0 con 0x5580768e2a30 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7d37fe640 1 -- 172.21.15.89:0/1754220692 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe7ec004b40 con 0x5580768e2a30 2024-09-15T23:26:43.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7fcb34740 10 client.16356 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7fcb34740 1 -- 172.21.15.89:0/1754220692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5580768f1370 con 0x5580768e2a30 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.743+0000 7fe7d37fe640 1 -- 172.21.15.89:0/1754220692 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e40047a0 con 0x5580768e2a30 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.744+0000 7fe7d37fe640 1 -- 172.21.15.89:0/1754220692 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe7ec030070 con 0x5580768e2a30 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.744+0000 7fe7fcb34740 10 client.16356 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.744+0000 7fe7fcb34740 10 client.16356 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.744+0000 7fe7fcb34740 20 client.16356 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.744+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580768f2830 0x5580768f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.744+0000 7fe7fcb34740 4 client.16356 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.744+0000 7fe7fcb34740 1 -- 172.21.15.89:0/1754220692 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558076849890 con 0x5580768f2830 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.744+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580768f2830 0x5580768f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.745+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580768f2830 0x5580768f4c20 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91270a04ef6bd4e1 server_cookie=ed164b0470d8c221 in_seq=0 out_seq=0 2024-09-15T23:26:43.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.745+0000 7fe7d37fe640 10 client.16356 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.745+0000 7fe7d37fe640 1 -- 172.21.15.89:0/1754220692 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558076849890 con 0x5580768f2830 2024-09-15T23:26:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.745+0000 7fe7d37fe640 10 client.16356 handle_command_reply: tid=0 2024-09-15T23:26:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.745+0000 7fe7fcb34740 1 client.16356 shutdown 2024-09-15T23:26:43.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 1 -- 172.21.15.89:0/1754220692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768e2a30 msgr2=0x5580768e0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768e2a30 0x5580768e0370 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec0315c0 tx=0x7fe7ec0095c0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 1 -- 172.21.15.89:0/1754220692 shutdown_connections 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580768f2830 0x5580768f4c20 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768e2a30 0x5580768e0370 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7d37fe640 0 client.16356 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768dfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/1754220692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768d7850 0x5580768df8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 1 -- 172.21.15.89:0/1754220692 >> 172.21.15.89:0/1754220692 conn(0x5580768d3de0 msgr2=0x5580768d9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 1 -- 172.21.15.89:0/1754220692 shutdown_connections 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 1 -- 172.21.15.89:0/1754220692 wait complete. 2024-09-15T23:26:43.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.746+0000 7fe7fcb34740 20 client.16356 trim_cache size 0 max 16384 2024-09-15T23:26:43.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.764+0000 7fe7fcb34740 1 Processor -- start 2024-09-15T23:26:43.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.764+0000 7fe7fcb34740 1 -- start start 2024-09-15T23:26:43.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.764+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768e17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.764+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768de1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.764+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 0x5580768fa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.764+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f41b0f60 con 0x5580768d7850 2024-09-15T23:26:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.764+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f41b2aa0 con 0x5580768d8170 2024-09-15T23:26:43.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.764+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f41b1630 con 0x5580768e2a30 2024-09-15T23:26:43.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.764+0000 7fe7f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768e17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768e17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46506/0 (socket says 172.21.15.89:46506) 2024-09-15T23:26:43.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f3fff640 1 -- 172.21.15.89:0/3438936583 learned_addr learned my addr 172.21.15.89:0/3438936583 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:43.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f8cf8640 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 0x5580768fa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768de1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f3fff640 1 -- 172.21.15.89:0/3438936583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 msgr2=0x5580768fa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 0x5580768fa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f3fff640 1 -- 172.21.15.89:0/3438936583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 msgr2=0x5580768de1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768de1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f3fff640 1 -- 172.21.15.89:0/3438936583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580768f4f90 con 0x5580768d7850 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f8cf8640 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 0x5580768fa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768de1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768e17c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400ef90 tx=0x7fe7e4008ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3ab6d96a8a84f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7d9ffb640 1 -- 172.21.15.89:0/3438936583 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4014040 con 0x5580768d7850 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7d9ffb640 1 -- 172.21.15.89:0/3438936583 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e4005510 con 0x5580768d7850 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.765+0000 7fe7d9ffb640 1 -- 172.21.15.89:0/3438936583 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e40247e0 con 0x5580768d7850 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.766+0000 7fe7fcb34740 1 -- 172.21.15.89:0/3438936583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 msgr2=0x5580768e17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.766+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768e17c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400ef90 tx=0x7fe7e4008ee0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.766+0000 7fe7fcb34740 1 -- 172.21.15.89:0/3438936583 shutdown_connections 2024-09-15T23:26:43.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.766+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 0x5580768fa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.766+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768de1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.766+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/3438936583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768e17c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.766+0000 7fe7fcb34740 1 -- 172.21.15.89:0/3438936583 >> 172.21.15.89:0/3438936583 conn(0x5580768d3de0 msgr2=0x5580768d9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:43.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.766+0000 7fe7fcb34740 1 -- 172.21.15.89:0/3438936583 shutdown_connections 2024-09-15T23:26:43.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.766+0000 7fe7fcb34740 1 -- 172.21.15.89:0/3438936583 wait complete. 2024-09-15T23:26:43.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.766+0000 7fe7fcb34740 1 -- start start 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7fcb34740 1 Processor -- start 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768eade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768eb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7fcb34740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 0x5580768e7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f41b23d0 con 0x5580768d7850 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f41b1d00 con 0x5580768d8170 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7fcb34740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f41aef40 con 0x5580768e2a30 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7fcb34740 10 client.0 _ll_register_callbacks cb 0x7fe7ec0314d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768eade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768eade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46518/0 (socket says 172.21.15.89:46518) 2024-09-15T23:26:43.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7f3fff640 1 -- 172.21.15.89:0/633058814 learned_addr learned my addr 172.21.15.89:0/633058814 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7f8cf8640 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 0x5580768e7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7f3fff640 1 -- 172.21.15.89:0/633058814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 msgr2=0x5580768e7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 0x5580768e7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7f3fff640 1 -- 172.21.15.89:0/633058814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 msgr2=0x5580768eb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768eb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.767+0000 7fe7f3fff640 1 -- 172.21.15.89:0/633058814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580768e7ba0 con 0x5580768d7850 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7f3fff640 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768eade0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400e760 tx=0x7fe7e4005310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca6cb99c26717f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7d37fe640 1 -- 172.21.15.89:0/633058814 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e40037f0 con 0x5580768d7850 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7d37fe640 1 -- 172.21.15.89:0/633058814 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe7e4003990 con 0x5580768d7850 2024-09-15T23:26:43.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7fcb34740 1 -- 172.21.15.89:0/633058814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7e4005510 con 0x5580768d7850 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7d37fe640 1 -- 172.21.15.89:0/633058814 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe7e4003b90 con 0x5580768d7850 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7fcb34740 10 client.16362 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7fcb34740 1 -- 172.21.15.89:0/633058814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55807695dfc0 con 0x5580768d7850 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7d37fe640 1 -- 172.21.15.89:0/633058814 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7e4005510 con 0x5580768d7850 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7d37fe640 1 -- 172.21.15.89:0/633058814 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe7e4036030 con 0x5580768d7850 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7fcb34740 10 client.16362 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7fcb34740 10 client.16362 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7fcb34740 20 client.16362 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580768f2830 0x558076960500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7fcb34740 4 client.16362 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}] 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.768+0000 7fe7fcb34740 1 -- 172.21.15.89:0/633058814 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}) -- 0x558076843be0 con 0x5580768f2830 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.769+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580768f2830 0x558076960500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.769+0000 7fe7d37fe640 10 client.16362 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:43.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.769+0000 7fe7f37fe640 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580768f2830 0x558076960500 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab1dd4a03a7012f6 server_cookie=d8302f5aa62caf8d in_seq=0 out_seq=0 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7d37fe640 1 -- 172.21.15.89:0/633058814 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x558076843be0 con 0x5580768f2830 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7d37fe640 10 client.16362 handle_command_reply: tid=0 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7fcb34740 1 client.16362 shutdown 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7fcb34740 1 -- 172.21.15.89:0/633058814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 msgr2=0x5580768eade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768eade0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe7e400e760 tx=0x7fe7e4005310 comp rx=0 tx=0).stop 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7fcb34740 1 -- 172.21.15.89:0/633058814 shutdown_connections 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580768f2830 0x558076960500 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7d37fe640 0 client.16362 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580768e2a30 0x5580768e7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580768d8170 0x5580768eb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7fcb34740 1 --2- 172.21.15.89:0/633058814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580768d7850 0x5580768eade0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.784+0000 7fe7fcb34740 1 -- 172.21.15.89:0/633058814 >> 172.21.15.89:0/633058814 conn(0x5580768d3de0 msgr2=0x5580768d9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:43.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fcb34740 1 -- 172.21.15.89:0/633058814 shutdown_connections 2024-09-15T23:26:43.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fcb34740 1 -- 172.21.15.89:0/633058814 wait complete. 2024-09-15T23:26:43.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fcb34740 20 client.16362 trim_cache size 0 max 16384 2024-09-15T23:26:43.787 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:26:43.787 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:26:43.787 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:26:43.787 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 6, 2024-09-15T23:26:43.787 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:26:43.787 INFO:teuthology.orchestra.run.smithi089.stdout: "67c52a99": { 2024-09-15T23:26:43.787 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 6, 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 139.6, 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:26:43.788 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:26:43.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:26:43.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:26:43.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:26:43.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:26:43.789 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:26:43.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 -- 172.21.15.89:0/3615613668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7dc0a3240 msgr2=0x7fe7dc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7dc0a3240 0x7fe7dc0a5700 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec02ae00 tx=0x7fe7ec02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 -- 172.21.15.89:0/3615613668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f4101420 msgr2=0x7fe7f4102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f4101420 0x7fe7f4102b90 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800c920 tx=0x7fe7e800cde0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 -- 172.21.15.89:0/3615613668 shutdown_connections 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7dc0a3240 0x7fe7dc0a5700 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7f4106a80 0x7fe7f41030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7f4101420 0x7fe7f4102b90 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 --2- 172.21.15.89:0/3615613668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe7f4100a80 0x7fe7f4102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 -- 172.21.15.89:0/3615613668 >> 172.21.15.89:0/3615613668 conn(0x7fe7f40fc7a0 msgr2=0x7fe7f40fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.785+0000 7fe7fa75b640 1 -- 172.21.15.89:0/3615613668 shutdown_connections 2024-09-15T23:26:43.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:43.786+0000 7fe7fa75b640 1 -- 172.21.15.89:0/3615613668 wait complete. 2024-09-15T23:26:44.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:44 smithi120 ceph-mon[35250]: pgmap v431: 129 pgs: 129 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 20 op/s 2024-09-15T23:26:44.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:44 smithi174 ceph-mon[35586]: pgmap v431: 129 pgs: 129 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 20 op/s 2024-09-15T23:26:44.493 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:44 smithi089 ceph-mon[30933]: pgmap v431: 129 pgs: 129 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 20 op/s 2024-09-15T23:26:44.499 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:44.527 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '67c52a99', quiesce took 5.4 seconds. Will release after: 14.15 2024-09-15T23:26:45.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.027+0000 7f2acbe6f640 1 -- 172.21.15.89:0/1239747313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ac4105f40 msgr2=0x7f2ac41063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.027+0000 7f2acbe6f640 1 --2- 172.21.15.89:0/1239747313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ac4105f40 0x7f2ac41063d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009ca0 tx=0x7f2abc02a940 comp rx=0 tx=0).stop 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.027+0000 7f2acbe6f640 1 -- 172.21.15.89:0/1239747313 shutdown_connections 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.027+0000 7f2acbe6f640 1 --2- 172.21.15.89:0/1239747313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ac4105f40 0x7f2ac41063d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.027+0000 7f2acbe6f640 1 --2- 172.21.15.89:0/1239747313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac4101450 0x7f2ac4105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.027+0000 7f2acbe6f640 1 --2- 172.21.15.89:0/1239747313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4100b30 0x7f2ac4100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.027+0000 7f2acbe6f640 1 -- 172.21.15.89:0/1239747313 >> 172.21.15.89:0/1239747313 conn(0x7f2ac40fc7a0 msgr2=0x7f2ac40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.027+0000 7f2acbe6f640 1 -- 172.21.15.89:0/1239747313 shutdown_connections 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.027+0000 7f2acbe6f640 1 -- 172.21.15.89:0/1239747313 wait complete. 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2acbe6f640 1 Processor -- start 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2acbe6f640 1 -- start start 2024-09-15T23:26:45.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2acbe6f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac4100b30 0x7f2ac41a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2acbe6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ac4101450 0x7f2ac41a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2acbe6f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4105f40 0x7f2ac419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2acbe6f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac4118640 con 0x7f2ac4101450 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2acbe6f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac41184c0 con 0x7f2ac4105f40 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2acbe6f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac41187c0 con 0x7f2ac4100b30 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2ac940a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ac4101450 0x7f2ac41a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2ac940a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ac4101450 0x7f2ac41a4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46540/0 (socket says 172.21.15.89:46540) 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.028+0000 7f2ac940a640 1 -- 172.21.15.89:0/170389534 learned_addr learned my addr 172.21.15.89:0/170389534 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2aca40c640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4105f40 0x7f2ac419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac4100b30 0x7f2ac41a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2ac940a640 1 -- 172.21.15.89:0/170389534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac4100b30 msgr2=0x7f2ac41a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2ac940a640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac4100b30 0x7f2ac41a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2ac940a640 1 -- 172.21.15.89:0/170389534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4105f40 msgr2=0x7f2ac419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2ac940a640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4105f40 0x7f2ac419e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2ac940a640 1 -- 172.21.15.89:0/170389534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac419f100 con 0x7f2ac4101450 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2aca40c640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4105f40 0x7f2ac419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2ac940a640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ac4101450 0x7f2ac41a4d50 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400b3f0 tx=0x7f2ab400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2abaffd640 1 -- 172.21.15.89:0/170389534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4015020 con 0x7f2ac4101450 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2abaffd640 1 -- 172.21.15.89:0/170389534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ab4011d10 con 0x7f2ac4101450 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2abaffd640 1 -- 172.21.15.89:0/170389534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab400f2e0 con 0x7f2ac4101450 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2acbe6f640 1 -- 172.21.15.89:0/170389534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac419f3f0 con 0x7f2ac4101450 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.029+0000 7f2acbe6f640 1 -- 172.21.15.89:0/170389534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac41ab6e0 con 0x7f2ac4101450 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.031+0000 7f2abaffd640 1 -- 172.21.15.89:0/170389534 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2ab4018020 con 0x7f2ac4101450 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ace248740 1 Processor -- start 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ace248740 1 -- start start 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2abaffd640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2ab00a3240 0x7f2ab00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ace248740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ace248740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628ad5d0 0x564a628ada10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ace248740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628b8370 0x564a628ba760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ace248740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564a628132f0 con 0x564a628aeb40 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ace248740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564a627bacf0 con 0x564a628b8370 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ace248740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564a6280f2b0 con 0x564a628ad5d0 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ac9c0b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2aca40c640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2ab00a3240 0x7f2ab00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ac9c0b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628ad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46542/0 (socket says 172.21.15.89:46542) 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ac9c0b640 1 -- 172.21.15.89:0/518975671 learned_addr learned my addr 172.21.15.89:0/518975671 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2abaffd640 1 -- 172.21.15.89:0/170389534 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2ab40cf640 con 0x7f2ac4101450 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2ac940a640 1 --2- 172.21.15.89:0/518975671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628ad5d0 0x564a628ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.032+0000 7f2aca40c640 1 --2- 172.21.15.89:0/518975671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628b8370 0x564a628ba760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2ac9c0b640 1 -- 172.21.15.89:0/518975671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628ad5d0 msgr2=0x564a628ada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/518975671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628ad5d0 0x564a628ada10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2ac9c0b640 1 -- 172.21.15.89:0/518975671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628b8370 msgr2=0x564a628ba760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/518975671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628b8370 0x564a628ba760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2ac9c0b640 1 -- 172.21.15.89:0/518975671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a6279f3a0 con 0x564a628aeb40 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2aca40c640 1 --2- 172.21.15.89:0/518975671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628b8370 0x564a628ba760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2aca40c640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2ab00a3240 0x7f2ab00a5700 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009a60 tx=0x7f2abc02f820 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/518975671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628ad090 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000b7f0 tx=0x7f2ac000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67155c71ba7c7f4a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2aaffff640 1 -- 172.21.15.89:0/518975671 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000d950 con 0x564a628aeb40 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2aaffff640 1 -- 172.21.15.89:0/518975671 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac0018070 con 0x564a628aeb40 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.033+0000 7f2aaffff640 1 -- 172.21.15.89:0/518975671 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac00123e0 con 0x564a628aeb40 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 -- 172.21.15.89:0/518975671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 msgr2=0x564a628ad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 --2- 172.21.15.89:0/518975671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628ad090 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000b7f0 tx=0x7f2ac000bcb0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 -- 172.21.15.89:0/518975671 shutdown_connections 2024-09-15T23:26:45.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 --2- 172.21.15.89:0/518975671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628b8370 0x564a628ba760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 --2- 172.21.15.89:0/518975671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628ad5d0 0x564a628ada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 --2- 172.21.15.89:0/518975671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628ad090 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 -- 172.21.15.89:0/518975671 >> 172.21.15.89:0/518975671 conn(0x564a62876310 msgr2=0x564a628a9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 -- 172.21.15.89:0/518975671 shutdown_connections 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 -- 172.21.15.89:0/518975671 wait complete. 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 -- start start 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.034+0000 7f2ace248740 1 Processor -- start 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2ace248740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 0x564a628b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2ace248740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628aeb40 0x564a628b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2ace248740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a628b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2ace248740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564a628bb070 con 0x564a628b8370 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2ace248740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564a627ce060 con 0x564a628ad5d0 2024-09-15T23:26:45.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2ace248740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564a628bb1f0 con 0x564a628aeb40 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2ace248740 10 client.0 _ll_register_callbacks cb 0x564a625218c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2aca40c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a628b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2aca40c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a628b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46556/0 (socket says 172.21.15.89:46556) 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2aca40c640 1 -- 172.21.15.89:0/733866557 learned_addr learned my addr 172.21.15.89:0/733866557 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2ac940a640 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628aeb40 0x564a628b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 0x564a628b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2aca40c640 1 -- 172.21.15.89:0/733866557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628aeb40 msgr2=0x564a628b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2aca40c640 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628aeb40 0x564a628b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2aca40c640 1 -- 172.21.15.89:0/733866557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 msgr2=0x564a628b7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2aca40c640 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 0x564a628b7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2aca40c640 1 -- 172.21.15.89:0/733866557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a625843c0 con 0x564a628b8370 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 0x564a628b7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2aca40c640 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a628b54d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009ca0 tx=0x7f2abc02a940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4365e208a139c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:45.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.035+0000 7f2aad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2aad7fa640 1 -- 172.21.15.89:0/733866557 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc034040 con 0x564a628b8370 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2aad7fa640 1 -- 172.21.15.89:0/733866557 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2abc030640 con 0x564a628b8370 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2ace248740 1 -- 172.21.15.89:0/733866557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ac0018070 con 0x564a628b8370 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2aad7fa640 1 -- 172.21.15.89:0/733866557 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2abc058050 con 0x564a628b8370 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2ace248740 10 client.16374 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2ace248740 1 -- 172.21.15.89:0/733866557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564a628c6a80 con 0x564a628b8370 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2aad7fa640 1 -- 172.21.15.89:0/733866557 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac0018070 con 0x564a628b8370 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2aad7fa640 1 -- 172.21.15.89:0/733866557 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2abc02f070 con 0x564a628b8370 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2ace248740 10 client.16374 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2ace248740 10 client.16374 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2ace248740 20 client.16374 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:45.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2ace248740 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x564a628c7f40 0x564a628ca400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2ace248740 4 client.16374 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2ace248740 1 -- 172.21.15.89:0/733866557 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a62820860 con 0x564a628c7f40 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.036+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x564a628c7f40 0x564a628ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.037+0000 7f2aad7fa640 10 client.16374 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.037+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x564a628c7f40 0x564a628ca400 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=de6bd40df8b1eaaf server_cookie=a4fbd20e0d4a9878 in_seq=0 out_seq=0 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2aad7fa640 1 -- 172.21.15.89:0/733866557 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564a62820860 con 0x564a628c7f40 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2aad7fa640 10 client.16374 handle_command_reply: tid=0 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 client.16374 shutdown 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 -- 172.21.15.89:0/733866557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 msgr2=0x564a628b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a628b54d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009ca0 tx=0x7f2abc02a940 comp rx=0 tx=0).stop 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 -- 172.21.15.89:0/733866557 shutdown_connections 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x564a628c7f40 0x564a628ca400 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a628b54d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628aeb40 0x564a628b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 --2- 172.21.15.89:0/733866557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 0x564a628b7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2aad7fa640 0 client.16374 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:26:45.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 -- 172.21.15.89:0/733866557 >> 172.21.15.89:0/733866557 conn(0x564a62876310 msgr2=0x564a628af960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:45.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 -- 172.21.15.89:0/733866557 shutdown_connections 2024-09-15T23:26:45.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 1 -- 172.21.15.89:0/733866557 wait complete. 2024-09-15T23:26:45.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.038+0000 7f2ace248740 20 client.16374 trim_cache size 0 max 16384 2024-09-15T23:26:45.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ace248740 1 Processor -- start 2024-09-15T23:26:45.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ace248740 1 -- start start 2024-09-15T23:26:45.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ace248740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 0x564a628b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ace248740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628b6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ace248740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628b8370 0x564a628b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ace248740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac41b5600 con 0x564a628aeb40 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ace248740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac41b7140 con 0x564a628ad5d0 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ace248740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac41b5cd0 con 0x564a628b8370 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ac940a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628b6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ac940a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628b6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46562/0 (socket says 172.21.15.89:46562) 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ac940a640 1 -- 172.21.15.89:0/3065050872 learned_addr learned my addr 172.21.15.89:0/3065050872 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/3065050872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 0x564a628b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2aca40c640 1 --2- 172.21.15.89:0/3065050872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628b8370 0x564a628b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ac940a640 1 -- 172.21.15.89:0/3065050872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628b8370 msgr2=0x564a628b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ac940a640 1 --2- 172.21.15.89:0/3065050872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628b8370 0x564a628b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ac940a640 1 -- 172.21.15.89:0/3065050872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 msgr2=0x564a628b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ac940a640 1 --2- 172.21.15.89:0/3065050872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 0x564a628b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.051+0000 7f2ac940a640 1 -- 172.21.15.89:0/3065050872 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a628add50 con 0x564a628aeb40 2024-09-15T23:26:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.052+0000 7f2ac940a640 1 --2- 172.21.15.89:0/3065050872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628b6ee0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40cd0c0 tx=0x7f2ab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf0a6bd42e700de server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.052+0000 7f2aaf7fe640 1 -- 172.21.15.89:0/3065050872 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab409bf70 con 0x564a628aeb40 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.052+0000 7f2aaf7fe640 1 -- 172.21.15.89:0/3065050872 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ab409b870 con 0x564a628aeb40 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.052+0000 7f2aaf7fe640 1 -- 172.21.15.89:0/3065050872 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab40999a0 con 0x564a628aeb40 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 -- 172.21.15.89:0/3065050872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 msgr2=0x564a628b6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 --2- 172.21.15.89:0/3065050872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628b6ee0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40cd0c0 tx=0x7f2ab40046c0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 -- 172.21.15.89:0/3065050872 shutdown_connections 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 --2- 172.21.15.89:0/3065050872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628b8370 0x564a628b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 --2- 172.21.15.89:0/3065050872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628aeb40 0x564a628b6ee0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 --2- 172.21.15.89:0/3065050872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628ad5d0 0x564a628b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 -- 172.21.15.89:0/3065050872 >> 172.21.15.89:0/3065050872 conn(0x564a62876310 msgr2=0x564a628b67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 -- 172.21.15.89:0/3065050872 shutdown_connections 2024-09-15T23:26:45.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 -- 172.21.15.89:0/3065050872 wait complete. 2024-09-15T23:26:45.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 -- start start 2024-09-15T23:26:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.053+0000 7f2ace248740 1 Processor -- start 2024-09-15T23:26:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2ace248740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628ad5d0 0x564a628bc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2ace248740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628aeb40 0x564a628bcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2ace248740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a62931cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2ace248740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac41b6a70 con 0x564a628b8370 2024-09-15T23:26:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2ace248740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac41b63a0 con 0x564a628aeb40 2024-09-15T23:26:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2ace248740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac41b35e0 con 0x564a628ad5d0 2024-09-15T23:26:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2ace248740 10 client.0 _ll_register_callbacks cb 0x7f2abc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aca40c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a62931cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aca40c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a62931cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46568/0 (socket says 172.21.15.89:46568) 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aca40c640 1 -- 172.21.15.89:0/2894051435 learned_addr learned my addr 172.21.15.89:0/2894051435 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2ac9c0b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628ad5d0 0x564a628bc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2ac940a640 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628aeb40 0x564a628bcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aca40c640 1 -- 172.21.15.89:0/2894051435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628ad5d0 msgr2=0x564a628bc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aca40c640 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628ad5d0 0x564a628bc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aca40c640 1 -- 172.21.15.89:0/2894051435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628aeb40 msgr2=0x564a628bcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aca40c640 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628aeb40 0x564a628bcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aca40c640 1 -- 172.21.15.89:0/2894051435 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a62932230 con 0x564a628b8370 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aca40c640 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a62931cf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2abc005cc0 tx=0x7f2abc02f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf8e933d54c66bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:45.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aad7fa640 1 -- 172.21.15.89:0/2894051435 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc061060 con 0x564a628b8370 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2aad7fa640 1 -- 172.21.15.89:0/2894051435 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2abc03db00 con 0x564a628b8370 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.054+0000 7f2ace248740 1 -- 172.21.15.89:0/2894051435 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ab409b870 con 0x564a628b8370 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2aad7fa640 1 -- 172.21.15.89:0/2894051435 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2abc034040 con 0x564a628b8370 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2ace248740 10 client.16380 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2ace248740 1 -- 172.21.15.89:0/2894051435 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564a629334b0 con 0x564a628b8370 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2aad7fa640 1 -- 172.21.15.89:0/2894051435 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ab409b870 con 0x564a628b8370 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2aad7fa640 1 -- 172.21.15.89:0/2894051435 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2abc02f040 con 0x564a628b8370 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2ace248740 10 client.16380 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2ace248740 10 client.16380 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2ace248740 20 client.16380 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2ace248740 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x564a62933bd0 0x564a62935fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:45.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2ace248740 4 client.16380 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2ace248740 1 -- 172.21.15.89:0/2894051435 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564a6281ad70 con 0x564a62933bd0 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.055+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x564a62933bd0 0x564a62935fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ac9c0b640 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x564a62933bd0 0x564a62935fc0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c948034508809fe6 server_cookie=f220c599429a755f in_seq=0 out_seq=0 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2aad7fa640 10 client.16380 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2aad7fa640 1 -- 172.21.15.89:0/2894051435 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564a6281ad70 con 0x564a62933bd0 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2aad7fa640 10 client.16380 handle_command_reply: tid=0 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 client.16380 shutdown 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 -- 172.21.15.89:0/2894051435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 msgr2=0x564a62931cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a62931cf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2abc005cc0 tx=0x7f2abc02f6c0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 -- 172.21.15.89:0/2894051435 shutdown_connections 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x564a62933bd0 0x564a62935fc0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564a628b8370 0x564a62931cf0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564a628aeb40 0x564a628bcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 --2- 172.21.15.89:0/2894051435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564a628ad5d0 0x564a628bc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2aad7fa640 0 client.16380 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:26:45.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 -- 172.21.15.89:0/2894051435 >> 172.21.15.89:0/2894051435 conn(0x564a62876310 msgr2=0x564a628b6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:45.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 -- 172.21.15.89:0/2894051435 shutdown_connections 2024-09-15T23:26:45.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 1 -- 172.21.15.89:0/2894051435 wait complete. 2024-09-15T23:26:45.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.056+0000 7f2ace248740 20 client.16380 trim_cache size 0 max 16384 2024-09-15T23:26:45.070 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:26:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 -- 172.21.15.89:0/170389534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2ab00a3240 msgr2=0x7f2ab00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2ab00a3240 0x7f2ab00a5700 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009a60 tx=0x7f2abc02f820 comp rx=0 tx=0).stop 2024-09-15T23:26:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 -- 172.21.15.89:0/170389534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ac4101450 msgr2=0x7f2ac41a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ac4101450 0x7f2ac41a4d50 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400b3f0 tx=0x7f2ab400b8b0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 -- 172.21.15.89:0/170389534 shutdown_connections 2024-09-15T23:26:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2ab00a3240 0x7f2ab00a5700 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ac4105f40 0x7f2ac419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ac4101450 0x7f2ac41a4d50 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 --2- 172.21.15.89:0/170389534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ac4100b30 0x7f2ac41a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:45.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 -- 172.21.15.89:0/170389534 >> 172.21.15.89:0/170389534 conn(0x7f2ac40fc7a0 msgr2=0x7f2ac410f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:45.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 -- 172.21.15.89:0/170389534 shutdown_connections 2024-09-15T23:26:45.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:45.057+0000 7f2acbe6f640 1 -- 172.21.15.89:0/170389534 wait complete. 2024-09-15T23:26:45.863 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:26:46.115 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:46.383 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:46 smithi089 ceph-mon[30933]: pgmap v432: 129 pgs: 129 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:26:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:46 smithi120 ceph-mon[35250]: pgmap v432: 129 pgs: 129 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:26:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:46 smithi174 ceph-mon[35586]: pgmap v432: 129 pgs: 129 active+clean; 1.4 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:26:46.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 -- 172.21.15.89:0/3887219935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc50100eb0 msgr2=0x7fdc501067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 --2- 172.21.15.89:0/3887219935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc50100eb0 0x7fdc501067a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdc44009900 tx=0x7fdc4402a410 comp rx=0 tx=0).stop 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 -- 172.21.15.89:0/3887219935 shutdown_connections 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 --2- 172.21.15.89:0/3887219935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc50106ed0 0x7fdc5010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 --2- 172.21.15.89:0/3887219935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc50100eb0 0x7fdc501067a0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 --2- 172.21.15.89:0/3887219935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc50100500 0x7fdc501008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 -- 172.21.15.89:0/3887219935 >> 172.21.15.89:0/3887219935 conn(0x7fdc500fc5c0 msgr2=0x7fdc500fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 -- 172.21.15.89:0/3887219935 shutdown_connections 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 -- 172.21.15.89:0/3887219935 wait complete. 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 Processor -- start 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.591+0000 7fdc56681640 1 -- start start 2024-09-15T23:26:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc56681640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc50100500 0x7fdc501984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc56681640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc50100eb0 0x7fdc501989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc56681640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc50106ed0 0x7fdc5019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc56681640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc50111f80 con 0x7fdc50100eb0 2024-09-15T23:26:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc56681640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc50111e00 con 0x7fdc50100500 2024-09-15T23:26:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc56681640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdc50112100 con 0x7fdc50106ed0 2024-09-15T23:26:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc50100eb0 0x7fdc501989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc50100eb0 0x7fdc501989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59724/0 (socket says 172.21.15.89:59724) 2024-09-15T23:26:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4f7fe640 1 -- 172.21.15.89:0/3019812625 learned_addr learned my addr 172.21.15.89:0/3019812625 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc54c1e640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc50106ed0 0x7fdc5019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc50100500 0x7fdc501984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4f7fe640 1 -- 172.21.15.89:0/3019812625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc50106ed0 msgr2=0x7fdc5019cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc50106ed0 0x7fdc5019cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4f7fe640 1 -- 172.21.15.89:0/3019812625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc50100500 msgr2=0x7fdc501984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc50100500 0x7fdc501984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4f7fe640 1 -- 172.21.15.89:0/3019812625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc5019d450 con 0x7fdc50100eb0 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc50100eb0 0x7fdc501989e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdc44006010 tx=0x7fdc440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4d7fa640 1 -- 172.21.15.89:0/3019812625 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4403f030 con 0x7fdc50100eb0 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc56681640 1 -- 172.21.15.89:0/3019812625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc5019d6e0 con 0x7fdc50100eb0 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.592+0000 7fdc4d7fa640 1 -- 172.21.15.89:0/3019812625 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc44003ce0 con 0x7fdc50100eb0 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.593+0000 7fdc4d7fa640 1 -- 172.21.15.89:0/3019812625 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc44035070 con 0x7fdc50100eb0 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.593+0000 7fdc56681640 1 -- 172.21.15.89:0/3019812625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc501a5020 con 0x7fdc50100eb0 2024-09-15T23:26:46.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.595+0000 7fdc4d7fa640 1 -- 172.21.15.89:0/3019812625 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fdc44042030 con 0x7fdc50100eb0 2024-09-15T23:26:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.595+0000 7fdc58a5a740 1 Processor -- start 2024-09-15T23:26:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.595+0000 7fdc58a5a740 1 -- start start 2024-09-15T23:26:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.595+0000 7fdc4d7fa640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc380a3240 0x7fdc380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.595+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5dbb40 0x55826f5da090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.595+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5da5d0 0x55826f5daa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.595+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5e5370 0x55826f5e7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.595+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55826f5402f0 con 0x55826f5dbb40 2024-09-15T23:26:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.595+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55826f4e7cf0 con 0x55826f5da5d0 2024-09-15T23:26:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.595+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55826f53c2b0 con 0x55826f5e5370 2024-09-15T23:26:46.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4d7fa640 1 -- 172.21.15.89:0/3019812625 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdc440f03d0 con 0x7fdc50100eb0 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc54c1e640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc380a3240 0x7fdc380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5dbb40 0x55826f5da090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5dbb40 0x55826f5da090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59728/0 (socket says 172.21.15.89:59728) 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4ffff640 1 -- 172.21.15.89:0/2103260687 learned_addr learned my addr 172.21.15.89:0/2103260687 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/2103260687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5da5d0 0x55826f5daa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc54c1e640 1 --2- 172.21.15.89:0/2103260687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5e5370 0x55826f5e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4ffff640 1 -- 172.21.15.89:0/2103260687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5e5370 msgr2=0x55826f5e7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/2103260687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5e5370 0x55826f5e7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4ffff640 1 -- 172.21.15.89:0/2103260687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5da5d0 msgr2=0x55826f5daa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/2103260687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5da5d0 0x55826f5daa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4ffff640 1 -- 172.21.15.89:0/2103260687 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55826f4cc3a0 con 0x55826f5dbb40 2024-09-15T23:26:46.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc54c1e640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc380a3240 0x7fdc380a5700 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdc480092f0 tx=0x7fdc48009f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:46.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/2103260687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5dbb40 0x55826f5da090 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000cc60 tx=0x7fdc4000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c43906fbe72759d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:46.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc367fc640 1 -- 172.21.15.89:0/2103260687 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc40014070 con 0x55826f5dbb40 2024-09-15T23:26:46.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc367fc640 1 -- 172.21.15.89:0/2103260687 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc400047a0 con 0x55826f5dbb40 2024-09-15T23:26:46.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.596+0000 7fdc367fc640 1 -- 172.21.15.89:0/2103260687 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc40010720 con 0x55826f5dbb40 2024-09-15T23:26:46.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.597+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2103260687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5dbb40 msgr2=0x55826f5da090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:46.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.597+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/2103260687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5dbb40 0x55826f5da090 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000cc60 tx=0x7fdc4000e720 comp rx=0 tx=0).stop 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.597+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2103260687 shutdown_connections 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.597+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/2103260687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5e5370 0x55826f5e7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.597+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/2103260687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5da5d0 0x55826f5daa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.597+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/2103260687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5dbb40 0x55826f5da090 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.597+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2103260687 >> 172.21.15.89:0/2103260687 conn(0x55826f5a3310 msgr2=0x55826f5d6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.597+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2103260687 shutdown_connections 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.597+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2103260687 wait complete. 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.597+0000 7fdc58a5a740 1 -- start start 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc58a5a740 1 Processor -- start 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5da5d0 0x55826f5e4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5e2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5e5370 0x55826f5e2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55826f5e8070 con 0x55826f5e5370 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55826f4fb060 con 0x55826f5da5d0 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55826f5e81f0 con 0x55826f5dbb40 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc58a5a740 10 client.0 _ll_register_callbacks cb 0x55826f24e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc54c1e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5e5370 0x55826f5e2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc54c1e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5e5370 0x55826f5e2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59742/0 (socket says 172.21.15.89:59742) 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc54c1e640 1 -- 172.21.15.89:0/1180195366 learned_addr learned my addr 172.21.15.89:0/1180195366 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5da5d0 0x55826f5e4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5e2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc54c1e640 1 -- 172.21.15.89:0/1180195366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 msgr2=0x55826f5e2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc54c1e640 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5e2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc54c1e640 1 -- 172.21.15.89:0/1180195366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5da5d0 msgr2=0x55826f5e4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc54c1e640 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5da5d0 0x55826f5e4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc54c1e640 1 -- 172.21.15.89:0/1180195366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55826f2b13c0 con 0x55826f5e5370 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.598+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5e2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc54c1e640 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5e5370 0x55826f5e2800 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fdc48012900 tx=0x7fdc48004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9217a3875385fd11 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc2f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/1180195366 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc48025070 con 0x55826f5e5370 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/1180195366 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdc4800fcf0 con 0x55826f5e5370 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc58a5a740 1 -- 172.21.15.89:0/1180195366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc400047a0 con 0x55826f5e5370 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/1180195366 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdc48031050 con 0x55826f5e5370 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc58a5a740 10 client.16392 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc58a5a740 1 -- 172.21.15.89:0/1180195366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55826f5f3b80 con 0x55826f5e5370 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/1180195366 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc400047a0 con 0x55826f5e5370 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/1180195366 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fdc4801e040 con 0x55826f5e5370 2024-09-15T23:26:46.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc58a5a740 10 client.16392 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc58a5a740 10 client.16392 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:26:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc58a5a740 20 client.16392 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55826f5f4f90 0x55826f5f7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc58a5a740 4 client.16392 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.599+0000 7fdc58a5a740 1 -- 172.21.15.89:0/1180195366 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55826f54d860 con 0x55826f5f4f90 2024-09-15T23:26:46.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.600+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55826f5f4f90 0x55826f5f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.601+0000 7fdc2f7fe640 10 client.16392 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:26:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.601+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55826f5f4f90 0x55826f5f7430 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ca9a58678bdfb05 server_cookie=7df9ec09ce09225b in_seq=0 out_seq=0 2024-09-15T23:26:46.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/1180195366 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55826f54d860 con 0x55826f5f4f90 2024-09-15T23:26:46.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc2f7fe640 10 client.16392 handle_command_reply: tid=0 2024-09-15T23:26:46.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 client.16392 shutdown 2024-09-15T23:26:46.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 -- 172.21.15.89:0/1180195366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5e5370 msgr2=0x55826f5e2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:46.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5e5370 0x55826f5e2800 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fdc48012900 tx=0x7fdc48004ab0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 -- 172.21.15.89:0/1180195366 shutdown_connections 2024-09-15T23:26:46.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55826f5f4f90 0x55826f5f7430 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5e5370 0x55826f5e2800 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5e2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/1180195366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5da5d0 0x55826f5e4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 -- 172.21.15.89:0/1180195366 >> 172.21.15.89:0/1180195366 conn(0x55826f5a3310 msgr2=0x55826f5dc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc2f7fe640 0 client.16392 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:26:46.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 -- 172.21.15.89:0/1180195366 shutdown_connections 2024-09-15T23:26:46.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 1 -- 172.21.15.89:0/1180195366 wait complete. 2024-09-15T23:26:46.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.602+0000 7fdc58a5a740 20 client.16392 trim_cache size 0 max 16384 2024-09-15T23:26:46.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc58a5a740 1 Processor -- start 2024-09-15T23:26:46.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc58a5a740 1 -- start start 2024-09-15T23:26:46.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5dd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5dd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5e5370 0x55826f5fcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501aee20 con 0x55826f5da5d0 2024-09-15T23:26:46.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501b0960 con 0x55826f5e5370 2024-09-15T23:26:46.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501af4f0 con 0x55826f5dbb40 2024-09-15T23:26:46.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5dd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5dd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59746/0 (socket says 172.21.15.89:59746) 2024-09-15T23:26:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc4ffff640 1 -- 172.21.15.89:0/907937990 learned_addr learned my addr 172.21.15.89:0/907937990 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.615+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/907937990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5dd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.616+0000 7fdc4ffff640 1 -- 172.21.15.89:0/907937990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 msgr2=0x55826f5dd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.616+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/907937990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5dd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.616+0000 7fdc4ffff640 1 -- 172.21.15.89:0/907937990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5e5370 msgr2=0x55826f5fcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.616+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/907937990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5e5370 0x55826f5fcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.616+0000 7fdc4ffff640 1 -- 172.21.15.89:0/907937990 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55826f5d7960 con 0x55826f5da5d0 2024-09-15T23:26:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.616+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/907937990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5dd120 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000ef40 tx=0x7fdc4000c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e1145d0a7d3c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.616+0000 7fdc35ffb640 1 -- 172.21.15.89:0/907937990 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc40014040 con 0x55826f5da5d0 2024-09-15T23:26:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.616+0000 7fdc35ffb640 1 -- 172.21.15.89:0/907937990 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc40004220 con 0x55826f5da5d0 2024-09-15T23:26:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.616+0000 7fdc35ffb640 1 -- 172.21.15.89:0/907937990 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc400107e0 con 0x55826f5da5d0 2024-09-15T23:26:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 -- 172.21.15.89:0/907937990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 msgr2=0x55826f5dd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:46.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/907937990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5dd120 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000ef40 tx=0x7fdc4000c550 comp rx=0 tx=0).stop 2024-09-15T23:26:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 -- 172.21.15.89:0/907937990 shutdown_connections 2024-09-15T23:26:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/907937990 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5e5370 0x55826f5fcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/907937990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5dd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/907937990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5dd120 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 -- 172.21.15.89:0/907937990 >> 172.21.15.89:0/907937990 conn(0x55826f5a3310 msgr2=0x55826f5f6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 -- 172.21.15.89:0/907937990 shutdown_connections 2024-09-15T23:26:46.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 -- 172.21.15.89:0/907937990 wait complete. 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 -- start start 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.617+0000 7fdc58a5a740 1 Processor -- start 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5e9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5e9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc58a5a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5e5370 0x55826f5e9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501b0290 con 0x55826f5da5d0 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501afbc0 con 0x55826f5e5370 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc58a5a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdc501ace00 con 0x55826f5dbb40 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc58a5a740 10 client.0 _ll_register_callbacks cb 0x7fdc48014f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5e9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5e9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59748/0 (socket says 172.21.15.89:59748) 2024-09-15T23:26:46.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4ffff640 1 -- 172.21.15.89:0/2577506276 learned_addr learned my addr 172.21.15.89:0/2577506276 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5e9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4ffff640 1 -- 172.21.15.89:0/2577506276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 msgr2=0x55826f5e9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5e9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4ffff640 1 -- 172.21.15.89:0/2577506276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5e5370 msgr2=0x55826f5e9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc54c1e640 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5e5370 0x55826f5e9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5e5370 0x55826f5e9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4ffff640 1 -- 172.21.15.89:0/2577506276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55826f663130 con 0x55826f5da5d0 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5e9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc54c1e640 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5e5370 0x55826f5e9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc4ffff640 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5e9130 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000cb80 tx=0x7fdc4000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffc2b0368e76ed62 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc2f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:46.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/2577506276 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc40032070 con 0x55826f5da5d0 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/2577506276 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdc40003a00 con 0x55826f5da5d0 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.618+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2577506276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc40004220 con 0x55826f5da5d0 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/2577506276 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdc4003f050 con 0x55826f5da5d0 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc58a5a740 10 client.16398 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2577506276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55826f664370 con 0x55826f5da5d0 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/2577506276 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc40004220 con 0x55826f5da5d0 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/2577506276 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fdc40004a00 con 0x55826f5da5d0 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc58a5a740 10 client.16398 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc58a5a740 10 client.16398 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc58a5a740 20 client.16398 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55826f664890 0x55826f666c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:46.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc58a5a740 4 client.16398 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2577506276 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55826f547d70 con 0x55826f664890 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.619+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55826f664890 0x55826f666c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc2f7fe640 10 client.16398 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc4f7fe640 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55826f664890 0x55826f666c80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b58664cb3e700cb7 server_cookie=bf913309ad19b095 in_seq=0 out_seq=0 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc2f7fe640 1 -- 172.21.15.89:0/2577506276 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55826f547d70 con 0x55826f664890 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc2f7fe640 10 client.16398 handle_command_reply: tid=0 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 client.16398 shutdown 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2577506276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 msgr2=0x55826f5e9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5e9130 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000cb80 tx=0x7fdc4000e720 comp rx=0 tx=0).stop 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2577506276 shutdown_connections 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55826f664890 0x55826f666c80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc2f7fe640 0 client.16398 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:26:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55826f5e5370 0x55826f5e9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55826f5dbb40 0x55826f5e9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 --2- 172.21.15.89:0/2577506276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55826f5da5d0 0x55826f5e9130 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2577506276 >> 172.21.15.89:0/2577506276 conn(0x55826f5a3310 msgr2=0x55826f5e38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2577506276 shutdown_connections 2024-09-15T23:26:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 1 -- 172.21.15.89:0/2577506276 wait complete. 2024-09-15T23:26:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.620+0000 7fdc58a5a740 20 client.16398 trim_cache size 0 max 16384 2024-09-15T23:26:46.634 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:26:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 -- 172.21.15.89:0/3019812625 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc380a3240 msgr2=0x7fdc380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc380a3240 0x7fdc380a5700 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdc480092f0 tx=0x7fdc48009f90 comp rx=0 tx=0).stop 2024-09-15T23:26:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 -- 172.21.15.89:0/3019812625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc50100eb0 msgr2=0x7fdc501989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc50100eb0 0x7fdc501989e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdc44006010 tx=0x7fdc440042f0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 -- 172.21.15.89:0/3019812625 shutdown_connections 2024-09-15T23:26:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc380a3240 0x7fdc380a5700 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc50106ed0 0x7fdc5019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc50100eb0 0x7fdc501989e0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 --2- 172.21.15.89:0/3019812625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc50100500 0x7fdc501984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 -- 172.21.15.89:0/3019812625 >> 172.21.15.89:0/3019812625 conn(0x7fdc500fc5c0 msgr2=0x7fdc50105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 -- 172.21.15.89:0/3019812625 shutdown_connections 2024-09-15T23:26:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:46.621+0000 7fdc56681640 1 -- 172.21.15.89:0/3019812625 wait complete. 2024-09-15T23:26:47.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:47 smithi089 ceph-mon[30933]: pgmap v433: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:26:47.446 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:26:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:47 smithi120 ceph-mon[35250]: pgmap v433: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:26:47.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:47 smithi174 ceph-mon[35586]: pgmap v433: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:26:47.696 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.166+0000 7f6282d96640 1 -- 172.21.15.89:0/263768545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f627c100e20 msgr2=0x7f627c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.166+0000 7f6282d96640 1 --2- 172.21.15.89:0/263768545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f627c100e20 0x7f627c101200 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6270009900 tx=0x7f627002a480 comp rx=0 tx=0).stop 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.166+0000 7f6282d96640 1 -- 172.21.15.89:0/263768545 shutdown_connections 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.166+0000 7f6282d96640 1 --2- 172.21.15.89:0/263768545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f627c102ef0 0x7f627c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.166+0000 7f6282d96640 1 --2- 172.21.15.89:0/263768545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c102570 0x7f627c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.166+0000 7f6282d96640 1 --2- 172.21.15.89:0/263768545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f627c100e20 0x7f627c101200 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.166+0000 7f6282d96640 1 -- 172.21.15.89:0/263768545 >> 172.21.15.89:0/263768545 conn(0x7f627c0fc7d0 msgr2=0x7f627c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.166+0000 7f6282d96640 1 -- 172.21.15.89:0/263768545 shutdown_connections 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.166+0000 7f6282d96640 1 -- 172.21.15.89:0/263768545 wait complete. 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6282d96640 1 Processor -- start 2024-09-15T23:26:48.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6282d96640 1 -- start start 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6282d96640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c100e20 0x7f627c197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6282d96640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f627c102570 0x7f627c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6282d96640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f627c102ef0 0x7f627c19be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6282d96640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f627c111020 con 0x7f627c102570 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6282d96640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f627c110ea0 con 0x7f627c102ef0 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6282d96640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1111a0 con 0x7f627c100e20 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f627bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f627c102570 0x7f627c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f627bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f627c102570 0x7f627c197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59768/0 (socket says 172.21.15.89:59768) 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f627bfff640 1 -- 172.21.15.89:0/1447478718 learned_addr learned my addr 172.21.15.89:0/1447478718 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6280b32640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c100e20 0x7f627c197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6281333640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f627c102ef0 0x7f627c19be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f627bfff640 1 -- 172.21.15.89:0/1447478718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c100e20 msgr2=0x7f627c197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:48.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f627bfff640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c100e20 0x7f627c197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f627bfff640 1 -- 172.21.15.89:0/1447478718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f627c102ef0 msgr2=0x7f627c19be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f627bfff640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f627c102ef0 0x7f627c19be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f627bfff640 1 -- 172.21.15.89:0/1447478718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627c19c530 con 0x7f627c102570 2024-09-15T23:26:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6280b32640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c100e20 0x7f627c197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f6281333640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f627c102ef0 0x7f627c19be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.167+0000 7f627bfff640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f627c102570 0x7f627c197ac0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f626800b4a0 tx=0x7f626800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.168+0000 7f6279ffb640 1 -- 172.21.15.89:0/1447478718 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268014070 con 0x7f627c102570 2024-09-15T23:26:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.168+0000 7f6282d96640 1 -- 172.21.15.89:0/1447478718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f627c19c820 con 0x7f627c102570 2024-09-15T23:26:48.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.168+0000 7f6279ffb640 1 -- 172.21.15.89:0/1447478718 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6268010af0 con 0x7f627c102570 2024-09-15T23:26:48.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.168+0000 7f6279ffb640 1 -- 172.21.15.89:0/1447478718 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268017d10 con 0x7f627c102570 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.168+0000 7f6282d96640 1 -- 172.21.15.89:0/1447478718 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f627c1a4100 con 0x7f627c102570 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.170+0000 7f6279ffb640 1 -- 172.21.15.89:0/1447478718 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6268017070 con 0x7f627c102570 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f628516f740 1 Processor -- start 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f628516f740 1 -- start start 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f628516f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaeea090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f628516f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaeea5d0 0x556acaeeaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f6279ffb640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f626c0a3240 0x7f626c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f628516f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaef5370 0x556acaef7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f628516f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556acae502f0 con 0x556acaeea5d0 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f628516f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556acadf7cf0 con 0x556acaeebb40 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f628516f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556acae4c2b0 con 0x556acaef5370 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f6280b32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaeea5d0 0x556acaeeaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f6280b32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaeea5d0 0x556acaeeaa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59770/0 (socket says 172.21.15.89:59770) 2024-09-15T23:26:48.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f6280b32640 1 -- 172.21.15.89:0/1924593603 learned_addr learned my addr 172.21.15.89:0/1924593603 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f627bfff640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f626c0a3240 0x7f626c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f6281333640 1 --2- 172.21.15.89:0/1924593603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaeea090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f6279ffb640 1 -- 172.21.15.89:0/1447478718 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f62680cf250 con 0x7f627c102570 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f6281333640 1 --2- 172.21.15.89:0/1924593603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaef5370 0x556acaef7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f6280b32640 1 -- 172.21.15.89:0/1924593603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaef5370 msgr2=0x556acaef7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.171+0000 7f6280b32640 1 --2- 172.21.15.89:0/1924593603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaef5370 0x556acaef7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.172+0000 7f6280b32640 1 -- 172.21.15.89:0/1924593603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 msgr2=0x556acaeea090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.172+0000 7f6280b32640 1 --2- 172.21.15.89:0/1924593603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaeea090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.172+0000 7f6280b32640 1 -- 172.21.15.89:0/1924593603 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556acaddc3a0 con 0x556acaeea5d0 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.172+0000 7f6281333640 1 --2- 172.21.15.89:0/1924593603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaef5370 0x556acaef7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.172+0000 7f6280b32640 1 --2- 172.21.15.89:0/1924593603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaeea5d0 0x556acaeeaa10 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f62700314e0 tx=0x7f6270004180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7dc4f94229be3f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.172+0000 7f627bfff640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f626c0a3240 0x7f626c0a5700 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6268097880 tx=0x7f6268090900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:48.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.172+0000 7f6266ffd640 1 -- 172.21.15.89:0/1924593603 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270035070 con 0x556acaeea5d0 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.172+0000 7f6266ffd640 1 -- 172.21.15.89:0/1924593603 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6270004370 con 0x556acaeea5d0 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.172+0000 7f6266ffd640 1 -- 172.21.15.89:0/1924593603 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270031a00 con 0x556acaeea5d0 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 -- 172.21.15.89:0/1924593603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaeea5d0 msgr2=0x556acaeeaa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 --2- 172.21.15.89:0/1924593603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaeea5d0 0x556acaeeaa10 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f62700314e0 tx=0x7f6270004180 comp rx=0 tx=0).stop 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 -- 172.21.15.89:0/1924593603 shutdown_connections 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 --2- 172.21.15.89:0/1924593603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaef5370 0x556acaef7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 --2- 172.21.15.89:0/1924593603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaeea5d0 0x556acaeeaa10 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 --2- 172.21.15.89:0/1924593603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaeea090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 -- 172.21.15.89:0/1924593603 >> 172.21.15.89:0/1924593603 conn(0x556acaeb3310 msgr2=0x556acaee6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 -- 172.21.15.89:0/1924593603 shutdown_connections 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 -- 172.21.15.89:0/1924593603 wait complete. 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 -- start start 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 Processor -- start 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeea5d0 0x556acaef4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaef1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaef24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556acaef8010 con 0x556acaef5370 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556acae0b060 con 0x556acaeebb40 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.173+0000 7f628516f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556acaef8190 con 0x556acaeea5d0 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f628516f740 10 client.0 _ll_register_callbacks cb 0x556acab5e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6281333640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaef24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6281333640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaef24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59778/0 (socket says 172.21.15.89:59778) 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6281333640 1 -- 172.21.15.89:0/4247946478 learned_addr learned my addr 172.21.15.89:0/4247946478 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6281333640 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeea5d0 0x556acaef4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6280b32640 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaef1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6281333640 1 -- 172.21.15.89:0/4247946478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeea5d0 msgr2=0x556acaef4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:48.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6281333640 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeea5d0 0x556acaef4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6281333640 1 -- 172.21.15.89:0/4247946478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 msgr2=0x556acaef1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6281333640 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaef1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6281333640 1 -- 172.21.15.89:0/4247946478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556acabc13c0 con 0x556acaef5370 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6280b32640 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaef1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f6281333640 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaef24f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6274015a00 tx=0x7f6274004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f7f1ff4284d05e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f625bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f625bfff640 1 -- 172.21.15.89:0/4247946478 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740151d0 con 0x556acaef5370 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f628516f740 1 -- 172.21.15.89:0/4247946478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6270004370 con 0x556acaef5370 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f625bfff640 1 -- 172.21.15.89:0/4247946478 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6274012b90 con 0x556acaef5370 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f625bfff640 1 -- 172.21.15.89:0/4247946478 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6274025050 con 0x556acaef5370 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f628516f740 10 client.16410 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.174+0000 7f628516f740 1 -- 172.21.15.89:0/4247946478 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556acaf03a80 con 0x556acaef5370 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.175+0000 7f625bfff640 1 -- 172.21.15.89:0/4247946478 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6270004370 con 0x556acaef5370 2024-09-15T23:26:48.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.175+0000 7f625bfff640 1 -- 172.21.15.89:0/4247946478 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6274010040 con 0x556acaef5370 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.175+0000 7f628516f740 10 client.16410 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.175+0000 7f628516f740 10 client.16410 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.175+0000 7f628516f740 20 client.16410 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.175+0000 7f628516f740 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x556acaf04e90 0x556acaf072c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.175+0000 7f628516f740 4 client.16410 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.175+0000 7f628516f740 1 -- 172.21.15.89:0/4247946478 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556acae5d860 con 0x556acaf04e90 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.175+0000 7f6280b32640 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x556acaf04e90 0x556acaf072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f625bfff640 10 client.16410 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f6280b32640 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x556acaf04e90 0x556acaf072c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5ccdd657a830633b server_cookie=86a10cd7542210c9 in_seq=0 out_seq=0 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f625bfff640 1 -- 172.21.15.89:0/4247946478 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556acae5d860 con 0x556acaf04e90 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f625bfff640 10 client.16410 handle_command_reply: tid=0 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f628516f740 1 client.16410 shutdown 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f628516f740 1 -- 172.21.15.89:0/4247946478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 msgr2=0x556acaef24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f628516f740 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaef24f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6274015a00 tx=0x7f6274004a40 comp rx=0 tx=0).stop 2024-09-15T23:26:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f628516f740 1 -- 172.21.15.89:0/4247946478 shutdown_connections 2024-09-15T23:26:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f628516f740 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaef24f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f628516f740 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaef1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f628516f740 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x556acaf04e90 0x556acaf072c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f628516f740 1 --2- 172.21.15.89:0/4247946478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeea5d0 0x556acaef4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f625bfff640 0 client.16410 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:26:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.176+0000 7f628516f740 1 -- 172.21.15.89:0/4247946478 >> 172.21.15.89:0/4247946478 conn(0x556acaeb3310 msgr2=0x556acaeec930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.177+0000 7f628516f740 1 -- 172.21.15.89:0/4247946478 shutdown_connections 2024-09-15T23:26:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.177+0000 7f628516f740 1 -- 172.21.15.89:0/4247946478 wait complete. 2024-09-15T23:26:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.177+0000 7f628516f740 20 client.16410 trim_cache size 0 max 16384 2024-09-15T23:26:48.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f628516f740 1 Processor -- start 2024-09-15T23:26:48.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f628516f740 1 -- start start 2024-09-15T23:26:48.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f628516f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeea5d0 0x556acaf012b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f628516f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeebb40 0x556acaefd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f628516f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaefdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f628516f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1acd80 con 0x556acaef5370 2024-09-15T23:26:48.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f628516f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1ae8c0 con 0x556acaeea5d0 2024-09-15T23:26:48.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f628516f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1ad450 con 0x556acaeebb40 2024-09-15T23:26:48.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6281333640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaefdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6281333640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaefdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59792/0 (socket says 172.21.15.89:59792) 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6281333640 1 -- 172.21.15.89:0/4052751271 learned_addr learned my addr 172.21.15.89:0/4052751271 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6280b32640 1 --2- 172.21.15.89:0/4052751271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeebb40 0x556acaefd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6281333640 1 --2- 172.21.15.89:0/4052751271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeea5d0 0x556acaf012b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6281333640 1 -- 172.21.15.89:0/4052751271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeebb40 msgr2=0x556acaefd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6281333640 1 --2- 172.21.15.89:0/4052751271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeebb40 0x556acaefd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6281333640 1 -- 172.21.15.89:0/4052751271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeea5d0 msgr2=0x556acaf012b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6281333640 1 --2- 172.21.15.89:0/4052751271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeea5d0 0x556acaf012b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6281333640 1 -- 172.21.15.89:0/4052751271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556acaee5310 con 0x556acaef5370 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.190+0000 7f6281333640 1 --2- 172.21.15.89:0/4052751271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaefdd40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6274015040 tx=0x7f627400a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a343feb8fd1c22b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f62667fc640 1 -- 172.21.15.89:0/4052751271 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627402d070 con 0x556acaef5370 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f62667fc640 1 -- 172.21.15.89:0/4052751271 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6274022030 con 0x556acaef5370 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f62667fc640 1 -- 172.21.15.89:0/4052751271 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627401d910 con 0x556acaef5370 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f628516f740 1 -- 172.21.15.89:0/4052751271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 msgr2=0x556acaefdd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:48.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f628516f740 1 --2- 172.21.15.89:0/4052751271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaefdd40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6274015040 tx=0x7f627400a2c0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f628516f740 1 -- 172.21.15.89:0/4052751271 shutdown_connections 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f628516f740 1 --2- 172.21.15.89:0/4052751271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaefdd40 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f628516f740 1 --2- 172.21.15.89:0/4052751271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeebb40 0x556acaefd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f628516f740 1 --2- 172.21.15.89:0/4052751271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeea5d0 0x556acaf012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f628516f740 1 -- 172.21.15.89:0/4052751271 >> 172.21.15.89:0/4052751271 conn(0x556acaeb3310 msgr2=0x556acaef0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f628516f740 1 -- 172.21.15.89:0/4052751271 shutdown_connections 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.191+0000 7f628516f740 1 -- 172.21.15.89:0/4052751271 wait complete. 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f628516f740 1 -- start start 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f628516f740 1 Processor -- start 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f628516f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeea5d0 0x556acaeef540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f628516f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaeefa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f628516f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaf0c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f628516f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1ae1f0 con 0x556acaef5370 2024-09-15T23:26:48.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f628516f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1adb20 con 0x556acaeebb40 2024-09-15T23:26:48.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f628516f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1aad60 con 0x556acaeea5d0 2024-09-15T23:26:48.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f628516f740 10 client.0 _ll_register_callbacks cb 0x7f627400a9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:48.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f6281333640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaf0c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f6281333640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaf0c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59798/0 (socket says 172.21.15.89:59798) 2024-09-15T23:26:48.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f6281333640 1 -- 172.21.15.89:0/1837049533 learned_addr learned my addr 172.21.15.89:0/1837049533 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:48.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.192+0000 7f6281333640 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeea5d0 0x556acaeef540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f6280b32640 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaeefa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f6281333640 1 -- 172.21.15.89:0/1837049533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeea5d0 msgr2=0x556acaeef540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f6281333640 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeea5d0 0x556acaeef540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f6281333640 1 -- 172.21.15.89:0/1837049533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 msgr2=0x556acaeefa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f6281333640 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaeefa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f6281333640 1 -- 172.21.15.89:0/1837049533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556acaf0c950 con 0x556acaef5370 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f6280b32640 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaeefa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f6281333640 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaf0c170 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6274004a80 tx=0x7f6274013d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4956b48c917567f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f625bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f625bfff640 1 -- 172.21.15.89:0/1837049533 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627402d070 con 0x556acaef5370 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f625bfff640 1 -- 172.21.15.89:0/1837049533 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6274012e00 con 0x556acaef5370 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f628516f740 1 -- 172.21.15.89:0/1837049533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6274022030 con 0x556acaef5370 2024-09-15T23:26:48.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f625bfff640 1 -- 172.21.15.89:0/1837049533 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f627401d6f0 con 0x556acaef5370 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f628516f740 10 client.16416 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f628516f740 1 -- 172.21.15.89:0/1837049533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556acaf73610 con 0x556acaef5370 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.193+0000 7f625bfff640 1 -- 172.21.15.89:0/1837049533 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6274022030 con 0x556acaef5370 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f625bfff640 1 -- 172.21.15.89:0/1837049533 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6274031400 con 0x556acaef5370 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f628516f740 10 client.16416 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f628516f740 10 client.16416 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f628516f740 20 client.16416 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f628516f740 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x556acaf73b70 0x556acaf75f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f628516f740 4 client.16416 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f628516f740 1 -- 172.21.15.89:0/1837049533 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556acae57d70 con 0x556acaf73b70 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f6280b32640 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x556acaf73b70 0x556acaf75f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f625bfff640 10 client.16416 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f6280b32640 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x556acaf73b70 0x556acaf75f60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6f4b6a31c34fcf9 server_cookie=d9122ce18c3bccd5 in_seq=0 out_seq=0 2024-09-15T23:26:48.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f625bfff640 1 -- 172.21.15.89:0/1837049533 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556acae57d70 con 0x556acaf73b70 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f625bfff640 10 client.16416 handle_command_reply: tid=0 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.194+0000 7f628516f740 1 client.16416 shutdown 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 1 -- 172.21.15.89:0/1837049533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 msgr2=0x556acaf0c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaf0c170 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6274004a80 tx=0x7f6274013d80 comp rx=0 tx=0).stop 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 1 -- 172.21.15.89:0/1837049533 shutdown_connections 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556acaef5370 0x556acaf0c170 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556acaeebb40 0x556acaeefa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x556acaf73b70 0x556acaf75f60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 1 --2- 172.21.15.89:0/1837049533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556acaeea5d0 0x556acaeef540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 1 -- 172.21.15.89:0/1837049533 >> 172.21.15.89:0/1837049533 conn(0x556acaeb3310 msgr2=0x556acaef06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f625bfff640 0 client.16416 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 1 -- 172.21.15.89:0/1837049533 shutdown_connections 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 1 -- 172.21.15.89:0/1837049533 wait complete. 2024-09-15T23:26:48.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f628516f740 20 client.16416 trim_cache size 0 max 16384 2024-09-15T23:26:48.204 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:26:48.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.195+0000 7f6282d96640 1 -- 172.21.15.89:0/1447478718 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f626c0a3240 msgr2=0x7f626c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:48.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f626c0a3240 0x7f626c0a5700 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6268097880 tx=0x7f6268090900 comp rx=0 tx=0).stop 2024-09-15T23:26:48.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 -- 172.21.15.89:0/1447478718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f627c102570 msgr2=0x7f627c197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:48.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f627c102570 0x7f627c197ac0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f626800b4a0 tx=0x7f626800b960 comp rx=0 tx=0).stop 2024-09-15T23:26:48.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 -- 172.21.15.89:0/1447478718 shutdown_connections 2024-09-15T23:26:48.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f626c0a3240 0x7f626c0a5700 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f627c102ef0 0x7f627c19be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f627c102570 0x7f627c197ac0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 --2- 172.21.15.89:0/1447478718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c100e20 0x7f627c197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:48.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 -- 172.21.15.89:0/1447478718 >> 172.21.15.89:0/1447478718 conn(0x7f627c0fc7d0 msgr2=0x7f627c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:48.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 -- 172.21.15.89:0/1447478718 shutdown_connections 2024-09-15T23:26:48.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:48.196+0000 7f6282d96640 1 -- 172.21.15.89:0/1447478718 wait complete. 2024-09-15T23:26:48.972 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:26:49.219 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:49.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.685+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3097917605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb00101420 msgr2=0x7fcb0010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.685+0000 7fcb05f7d640 1 --2- 172.21.15.89:0/3097917605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb00101420 0x7fcb0010d950 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009cd0 tx=0x7fcaf802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.685+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3097917605 shutdown_connections 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.685+0000 7fcb05f7d640 1 --2- 172.21.15.89:0/3097917605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb00101420 0x7fcb0010d950 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.685+0000 7fcb05f7d640 1 --2- 172.21.15.89:0/3097917605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb00100a80 0x7fcb00100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.685+0000 7fcb05f7d640 1 --2- 172.21.15.89:0/3097917605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb00106a80 0x7fcb00106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.685+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3097917605 >> 172.21.15.89:0/3097917605 conn(0x7fcb000fc7a0 msgr2=0x7fcb000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.685+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3097917605 shutdown_connections 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.686+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3097917605 wait complete. 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.686+0000 7fcb05f7d640 1 Processor -- start 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.686+0000 7fcb05f7d640 1 -- start start 2024-09-15T23:26:49.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.686+0000 7fcb05f7d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb00100a80 0x7fcb001993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.686+0000 7fcb05f7d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb00101420 0x7fcb001998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.686+0000 7fcb05f7d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb00106a80 0x7fcb001a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.686+0000 7fcb05f7d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcb00110f20 con 0x7fcb00106a80 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.686+0000 7fcb05f7d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcb00110da0 con 0x7fcb00100a80 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.686+0000 7fcb05f7d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcb001110a0 con 0x7fcb00101420 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb00106a80 0x7fcb001a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb00106a80 0x7fcb001a4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59814/0 (socket says 172.21.15.89:59814) 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafffff640 1 -- 172.21.15.89:0/3802019780 learned_addr learned my addr 172.21.15.89:0/3802019780 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb00100a80 0x7fcb001993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb00101420 0x7fcb001998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafffff640 1 -- 172.21.15.89:0/3802019780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb00101420 msgr2=0x7fcb001998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafffff640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb00101420 0x7fcb001998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafffff640 1 -- 172.21.15.89:0/3802019780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb00100a80 msgr2=0x7fcb001993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:49.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafffff640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb00100a80 0x7fcb001993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafffff640 1 -- 172.21.15.89:0/3802019780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb001a4970 con 0x7fcb00106a80 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb00100a80 0x7fcb001993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb00101420 0x7fcb001998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafffff640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb00106a80 0x7fcb001a4430 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8006010 tx=0x7fcaf8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafd7fa640 1 -- 172.21.15.89:0/3802019780 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8004450 con 0x7fcb00106a80 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3802019780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb001a4b40 con 0x7fcb00106a80 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafd7fa640 1 -- 172.21.15.89:0/3802019780 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaf8003e10 con 0x7fcb00106a80 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcafd7fa640 1 -- 172.21.15.89:0/3802019780 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80466e0 con 0x7fcb00106a80 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.687+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3802019780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb001a5070 con 0x7fcb00106a80 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcafd7fa640 1 -- 172.21.15.89:0/3802019780 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcaf80305a0 con 0x7fcb00106a80 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcb08356740 1 Processor -- start 2024-09-15T23:26:49.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcb08356740 1 -- start start 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcb08356740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf21090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcb08356740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf21a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcb08356740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf2c370 0x55c5daf2e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcb08356740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c5dae872f0 con 0x55c5daf215d0 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcb08356740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c5dae2ecf0 con 0x55c5daf22b40 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcb08356740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c5dae832b0 con 0x55c5daf2c370 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcaff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf21a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcaff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf21a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59824/0 (socket says 172.21.15.89:59824) 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcaff7fe640 1 -- 172.21.15.89:0/209983137 learned_addr learned my addr 172.21.15.89:0/209983137 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/209983137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf21090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcaff7fe640 1 -- 172.21.15.89:0/209983137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf2c370 msgr2=0x55c5daf2e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/209983137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf2c370 0x55c5daf2e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcaff7fe640 1 -- 172.21.15.89:0/209983137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 msgr2=0x55c5daf21090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/209983137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf21090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcaff7fe640 1 -- 172.21.15.89:0/209983137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5dae133a0 con 0x55c5daf215d0 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/209983137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf21090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.690+0000 7fcafd7fa640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcaec0a3240 0x7fcaec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.691+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/209983137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf21a10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0007850 tx=0x7fcaf0007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23cb5d08e964e5f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.691+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcaec0a3240 0x7fcaec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.691+0000 7fcafd7fa640 1 -- 172.21.15.89:0/3802019780 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcaf802e030 con 0x7fcb00106a80 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.691+0000 7fcaf5ffb640 1 -- 172.21.15.89:0/209983137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf000d950 con 0x55c5daf215d0 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.691+0000 7fcaf5ffb640 1 -- 172.21.15.89:0/209983137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaf0018070 con 0x55c5daf215d0 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.691+0000 7fcaf5ffb640 1 -- 172.21.15.89:0/209983137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf00123e0 con 0x55c5daf215d0 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.691+0000 7fcb08356740 1 -- 172.21.15.89:0/209983137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 msgr2=0x55c5daf21a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:49.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.691+0000 7fcb08356740 1 --2- 172.21.15.89:0/209983137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf21a10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0007850 tx=0x7fcaf0007d10 comp rx=0 tx=0).stop 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.691+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcaec0a3240 0x7fcaec0a5700 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcae8006d30 tx=0x7fcae8006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 -- 172.21.15.89:0/209983137 shutdown_connections 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 --2- 172.21.15.89:0/209983137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf2c370 0x55c5daf2e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 --2- 172.21.15.89:0/209983137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf21a10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 --2- 172.21.15.89:0/209983137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf21090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 -- 172.21.15.89:0/209983137 >> 172.21.15.89:0/209983137 conn(0x55c5daeea310 msgr2=0x55c5daf1dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 -- 172.21.15.89:0/209983137 shutdown_connections 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 -- 172.21.15.89:0/209983137 wait complete. 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 -- start start 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 Processor -- start 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 0x55c5daf2bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf28f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf294f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c5daf2f070 con 0x55c5daf2c370 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c5dae42060 con 0x55c5daf22b40 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c5daf2f1f0 con 0x55c5daf215d0 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcb08356740 10 client.0 _ll_register_callbacks cb 0x55c5dab958c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.692+0000 7fcafffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf294f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcafffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf294f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59832/0 (socket says 172.21.15.89:59832) 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcafffff640 1 -- 172.21.15.89:0/188894607 learned_addr learned my addr 172.21.15.89:0/188894607 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 0x55c5daf2bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf28f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcafffff640 1 -- 172.21.15.89:0/188894607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 msgr2=0x55c5daf2bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcafffff640 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 0x55c5daf2bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcafffff640 1 -- 172.21.15.89:0/188894607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 msgr2=0x55c5daf28f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcafffff640 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf28f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcafffff640 1 -- 172.21.15.89:0/188894607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5dabf83c0 con 0x55c5daf2c370 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 0x55c5daf2bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcafffff640 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf294f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80bd910 tx=0x7fcaf80be140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec48b7b38335512 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcadeffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcadeffd640 1 -- 172.21.15.89:0/188894607 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80c65f0 con 0x55c5daf2c370 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcadeffd640 1 -- 172.21.15.89:0/188894607 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcaf80bbc80 con 0x55c5daf2c370 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcb08356740 1 -- 172.21.15.89:0/188894607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcaf0018070 con 0x55c5daf2c370 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcadeffd640 1 -- 172.21.15.89:0/188894607 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcaf8030e80 con 0x55c5daf2c370 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcb08356740 10 client.16428 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcb08356740 1 -- 172.21.15.89:0/188894607 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c5daf3aab0 con 0x55c5daf2c370 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.693+0000 7fcadeffd640 1 -- 172.21.15.89:0/188894607 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaf0018070 con 0x55c5daf2c370 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.694+0000 7fcadeffd640 1 -- 172.21.15.89:0/188894607 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcaf804e070 con 0x55c5daf2c370 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.694+0000 7fcb08356740 10 client.16428 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.694+0000 7fcb08356740 10 client.16428 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:26:49.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.694+0000 7fcb08356740 20 client.16428 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.694+0000 7fcb08356740 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c5daf3bea0 0x55c5daf3e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.694+0000 7fcb08356740 4 client.16428 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.694+0000 7fcb08356740 1 -- 172.21.15.89:0/188894607 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5dae94860 con 0x55c5daf3bea0 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.694+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c5daf3bea0 0x55c5daf3e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.694+0000 7fcadeffd640 10 client.16428 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.694+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c5daf3bea0 0x55c5daf3e2d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e6baa1e2b2cd71c7 server_cookie=d6b1b88600cf5572 in_seq=0 out_seq=0 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcadeffd640 1 -- 172.21.15.89:0/188894607 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c5dae94860 con 0x55c5daf3bea0 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcadeffd640 10 client.16428 handle_command_reply: tid=0 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 client.16428 shutdown 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 -- 172.21.15.89:0/188894607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 msgr2=0x55c5daf294f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:49.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf294f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80bd910 tx=0x7fcaf80be140 comp rx=0 tx=0).stop 2024-09-15T23:26:49.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 -- 172.21.15.89:0/188894607 shutdown_connections 2024-09-15T23:26:49.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c5daf3bea0 0x55c5daf3e2d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf294f0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf28f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 --2- 172.21.15.89:0/188894607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 0x55c5daf2bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcadeffd640 0 client.16428 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:26:49.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 -- 172.21.15.89:0/188894607 >> 172.21.15.89:0/188894607 conn(0x55c5daeea310 msgr2=0x55c5daf23930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:49.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 -- 172.21.15.89:0/188894607 shutdown_connections 2024-09-15T23:26:49.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.695+0000 7fcb08356740 1 -- 172.21.15.89:0/188894607 wait complete. 2024-09-15T23:26:49.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.696+0000 7fcb08356740 20 client.16428 trim_cache size 0 max 16384 2024-09-15T23:26:49.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.708+0000 7fcb08356740 1 Processor -- start 2024-09-15T23:26:49.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.708+0000 7fcb08356740 1 -- start start 2024-09-15T23:26:49.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcb08356740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 0x55c5daf38340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcb08356740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf20bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcb08356740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf34890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcb08356740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcb001aedf0 con 0x55c5daf2c370 2024-09-15T23:26:49.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcb08356740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcb001b0930 con 0x55c5daf22b40 2024-09-15T23:26:49.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcb08356740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcb001af4c0 con 0x55c5daf215d0 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcafffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf34890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcafffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf34890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59838/0 (socket says 172.21.15.89:59838) 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcafffff640 1 -- 172.21.15.89:0/1824302929 learned_addr learned my addr 172.21.15.89:0/1824302929 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/1824302929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 0x55c5daf38340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/1824302929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf20bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcafffff640 1 -- 172.21.15.89:0/1824302929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 msgr2=0x55c5daf38340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcafffff640 1 --2- 172.21.15.89:0/1824302929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 0x55c5daf38340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcafffff640 1 -- 172.21.15.89:0/1824302929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 msgr2=0x55c5daf20bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcafffff640 1 --2- 172.21.15.89:0/1824302929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf20bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcafffff640 1 -- 172.21.15.89:0/1824302929 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5daf1c310 con 0x55c5daf2c370 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/1824302929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf20bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:49.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcafffff640 1 --2- 172.21.15.89:0/1824302929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf34890 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80ba350 tx=0x7fcaf8030b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4919c46550f3c5c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:49.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcaf57fa640 1 -- 172.21.15.89:0/1824302929 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80c65f0 con 0x55c5daf2c370 2024-09-15T23:26:49.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcaf57fa640 1 -- 172.21.15.89:0/1824302929 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaf8036070 con 0x55c5daf2c370 2024-09-15T23:26:49.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.709+0000 7fcaf57fa640 1 -- 172.21.15.89:0/1824302929 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80bbc80 con 0x55c5daf2c370 2024-09-15T23:26:49.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.710+0000 7fcb08356740 1 -- 172.21.15.89:0/1824302929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 msgr2=0x55c5daf34890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:49.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.710+0000 7fcb08356740 1 --2- 172.21.15.89:0/1824302929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf34890 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcaf80ba350 tx=0x7fcaf8030b80 comp rx=0 tx=0).stop 2024-09-15T23:26:49.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.710+0000 7fcb08356740 1 -- 172.21.15.89:0/1824302929 shutdown_connections 2024-09-15T23:26:49.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.710+0000 7fcb08356740 1 --2- 172.21.15.89:0/1824302929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf2c370 0x55c5daf34890 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.710+0000 7fcb08356740 1 --2- 172.21.15.89:0/1824302929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf20bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.710+0000 7fcb08356740 1 --2- 172.21.15.89:0/1824302929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf215d0 0x55c5daf38340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.710+0000 7fcb08356740 1 -- 172.21.15.89:0/1824302929 >> 172.21.15.89:0/1824302929 conn(0x55c5daeea310 msgr2=0x55c5daf27860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:49.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.710+0000 7fcb08356740 1 -- 172.21.15.89:0/1824302929 shutdown_connections 2024-09-15T23:26:49.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.710+0000 7fcb08356740 1 -- 172.21.15.89:0/1824302929 wait complete. 2024-09-15T23:26:49.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.710+0000 7fcb08356740 1 -- start start 2024-09-15T23:26:49.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcb08356740 1 Processor -- start 2024-09-15T23:26:49.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcb08356740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf263f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcb08356740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf26930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcb08356740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf2c370 0x55c5daf42b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcb08356740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcb001b0260 con 0x55c5daf215d0 2024-09-15T23:26:49.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcb08356740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcb001afb90 con 0x55c5daf22b40 2024-09-15T23:26:49.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcb08356740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcb001acdd0 con 0x55c5daf2c370 2024-09-15T23:26:49.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcb08356740 10 client.0 _ll_register_callbacks cb 0x7fcaf80095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:49.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcaff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf263f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcaff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf263f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59840/0 (socket says 172.21.15.89:59840) 2024-09-15T23:26:49.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcaff7fe640 1 -- 172.21.15.89:0/293623898 learned_addr learned my addr 172.21.15.89:0/293623898 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:49.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf26930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcafffff640 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf2c370 0x55c5daf42b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcaff7fe640 1 -- 172.21.15.89:0/293623898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf2c370 msgr2=0x55c5daf42b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:49.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf2c370 0x55c5daf42b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcaff7fe640 1 -- 172.21.15.89:0/293623898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 msgr2=0x55c5daf26930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:49.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf26930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcaff7fe640 1 -- 172.21.15.89:0/293623898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5daf430f0 con 0x55c5daf215d0 2024-09-15T23:26:49.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf26930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:49.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.711+0000 7fcafffff640 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf2c370 0x55c5daf42b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:49.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcaff7fe640 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf263f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0007fd0 tx=0x7fcaf00139f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e752dfa64683375 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:49.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcadeffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:49.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcadeffd640 1 -- 172.21.15.89:0/293623898 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf002a030 con 0x55c5daf215d0 2024-09-15T23:26:49.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcadeffd640 1 -- 172.21.15.89:0/293623898 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcaf000d950 con 0x55c5daf215d0 2024-09-15T23:26:49.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcb08356740 1 -- 172.21.15.89:0/293623898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcaf8036070 con 0x55c5daf215d0 2024-09-15T23:26:49.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcadeffd640 1 -- 172.21.15.89:0/293623898 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcaf0034050 con 0x55c5daf215d0 2024-09-15T23:26:49.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcb08356740 10 client.16434 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:49.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcb08356740 1 -- 172.21.15.89:0/293623898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c5daf44340 con 0x55c5daf215d0 2024-09-15T23:26:49.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcadeffd640 1 -- 172.21.15.89:0/293623898 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaf8036070 con 0x55c5daf215d0 2024-09-15T23:26:49.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcadeffd640 1 -- 172.21.15.89:0/293623898 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcaf0026a50 con 0x55c5daf215d0 2024-09-15T23:26:49.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcb08356740 10 client.16434 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:49.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcb08356740 10 client.16434 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:26:49.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcb08356740 20 client.16434 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:49.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcb08356740 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c5dafabba0 0x55c5dafadf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:49.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcb08356740 4 client.16434 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:49.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.712+0000 7fcb08356740 1 -- 172.21.15.89:0/293623898 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c5dae8ed70 con 0x55c5dafabba0 2024-09-15T23:26:49.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.713+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c5dafabba0 0x55c5dafadf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:49.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.713+0000 7fcadeffd640 10 client.16434 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:26:49.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.713+0000 7fcaf6ffd640 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c5dafabba0 0x55c5dafadf90 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=81a23511fa0dddb8 server_cookie=d239303688373b61 in_seq=0 out_seq=0 2024-09-15T23:26:49.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.713+0000 7fcadeffd640 1 -- 172.21.15.89:0/293623898 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c5dae8ed70 con 0x55c5dafabba0 2024-09-15T23:26:49.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.713+0000 7fcadeffd640 10 client.16434 handle_command_reply: tid=0 2024-09-15T23:26:49.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.713+0000 7fcb08356740 1 client.16434 shutdown 2024-09-15T23:26:49.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 1 -- 172.21.15.89:0/293623898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 msgr2=0x55c5daf263f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:49.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf263f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0007fd0 tx=0x7fcaf00139f0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 1 -- 172.21.15.89:0/293623898 shutdown_connections 2024-09-15T23:26:49.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c5dafabba0 0x55c5dafadf90 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5daf2c370 0x55c5daf42b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcadeffd640 0 client.16434 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:26:49.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5daf22b40 0x55c5daf26930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 1 --2- 172.21.15.89:0/293623898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5daf215d0 0x55c5daf263f0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 1 -- 172.21.15.89:0/293623898 >> 172.21.15.89:0/293623898 conn(0x55c5daeea310 msgr2=0x55c5daf27cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:49.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 1 -- 172.21.15.89:0/293623898 shutdown_connections 2024-09-15T23:26:49.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 1 -- 172.21.15.89:0/293623898 wait complete. 2024-09-15T23:26:49.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb08356740 20 client.16434 trim_cache size 0 max 16384 2024-09-15T23:26:49.733 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:26:49.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3802019780 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcaec0a3240 msgr2=0x7fcaec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:49.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb05f7d640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcaec0a3240 0x7fcaec0a5700 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcae8006d30 tx=0x7fcae8006cc0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3802019780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb00106a80 msgr2=0x7fcb001a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:49.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.714+0000 7fcb05f7d640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb00106a80 0x7fcb001a4430 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8006010 tx=0x7fcaf8003c20 comp rx=0 tx=0).stop 2024-09-15T23:26:49.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.715+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3802019780 shutdown_connections 2024-09-15T23:26:49.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.715+0000 7fcb05f7d640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcaec0a3240 0x7fcaec0a5700 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.715+0000 7fcb05f7d640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcb00106a80 0x7fcb001a4430 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.715+0000 7fcb05f7d640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb00101420 0x7fcb001998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.715+0000 7fcb05f7d640 1 --2- 172.21.15.89:0/3802019780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb00100a80 0x7fcb001993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:49.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.715+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3802019780 >> 172.21.15.89:0/3802019780 conn(0x7fcb000fc7a0 msgr2=0x7fcb000fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:49.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.715+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3802019780 shutdown_connections 2024-09-15T23:26:49.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:49.715+0000 7fcb05f7d640 1 -- 172.21.15.89:0/3802019780 wait complete. 2024-09-15T23:26:50.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:50 smithi120 ceph-mon[35250]: pgmap v434: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:26:50.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:50 smithi174 ceph-mon[35586]: pgmap v434: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:26:50.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:50 smithi089 ceph-mon[30933]: pgmap v434: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:26:50.660 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:26:50.912 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:51.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.391+0000 7f8a0430d640 1 -- 172.21.15.89:0/2310385149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89fc106a80 msgr2=0x7f89fc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:51.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.391+0000 7f8a0430d640 1 --2- 172.21.15.89:0/2310385149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89fc106a80 0x7f89fc106e60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f89f8009930 tx=0x7f89f802a420 comp rx=0 tx=0).stop 2024-09-15T23:26:51.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.391+0000 7f8a0430d640 1 -- 172.21.15.89:0/2310385149 shutdown_connections 2024-09-15T23:26:51.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.391+0000 7f8a0430d640 1 --2- 172.21.15.89:0/2310385149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89fc101420 0x7f89fc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.391+0000 7f8a0430d640 1 --2- 172.21.15.89:0/2310385149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89fc100a80 0x7f89fc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.391+0000 7f8a0430d640 1 --2- 172.21.15.89:0/2310385149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89fc106a80 0x7f89fc106e60 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.391+0000 7f8a0430d640 1 -- 172.21.15.89:0/2310385149 >> 172.21.15.89:0/2310385149 conn(0x7f89fc0fc7a0 msgr2=0x7f89fc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.391+0000 7f8a0430d640 1 -- 172.21.15.89:0/2310385149 shutdown_connections 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.391+0000 7f8a0430d640 1 -- 172.21.15.89:0/2310385149 wait complete. 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a0430d640 1 Processor -- start 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a0430d640 1 -- start start 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a0430d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89fc100a80 0x7f89fc198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a0430d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89fc101420 0x7f89fc198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a0430d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89fc106a80 0x7f89fc19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a0430d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc110ea0 con 0x7f89fc106a80 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a0430d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc110d20 con 0x7f89fc100a80 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a0430d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc111020 con 0x7f89fc101420 2024-09-15T23:26:51.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a028aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89fc106a80 0x7f89fc19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a028aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89fc106a80 0x7f89fc19cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59856/0 (socket says 172.21.15.89:59856) 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a028aa640 1 -- 172.21.15.89:0/748852527 learned_addr learned my addr 172.21.15.89:0/748852527 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a020a9640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89fc100a80 0x7f89fc198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a018a8640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89fc101420 0x7f89fc198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a028aa640 1 -- 172.21.15.89:0/748852527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89fc101420 msgr2=0x7f89fc198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a028aa640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89fc101420 0x7f89fc198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a028aa640 1 -- 172.21.15.89:0/748852527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89fc100a80 msgr2=0x7f89fc198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a028aa640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89fc100a80 0x7f89fc198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.392+0000 7f8a028aa640 1 -- 172.21.15.89:0/748852527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89fc19d4d0 con 0x7f89fc106a80 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.393+0000 7f8a028aa640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89fc106a80 0x7f89fc19cdf0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f89f400cce0 tx=0x7f89f400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.393+0000 7f89f37fe640 1 -- 172.21.15.89:0/748852527 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f400f450 con 0x7f89fc106a80 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.393+0000 7f89f37fe640 1 -- 172.21.15.89:0/748852527 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f400fad0 con 0x7f89fc106a80 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.393+0000 7f8a0430d640 1 -- 172.21.15.89:0/748852527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89fc19d7c0 con 0x7f89fc106a80 2024-09-15T23:26:51.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.393+0000 7f89f37fe640 1 -- 172.21.15.89:0/748852527 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f4018b60 con 0x7f89fc106a80 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.393+0000 7f8a0430d640 1 -- 172.21.15.89:0/748852527 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89fc1a50a0 con 0x7f89fc106a80 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.395+0000 7f89f37fe640 1 -- 172.21.15.89:0/748852527 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f89f400fc80 con 0x7f89fc106a80 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.395+0000 7f8a066e6740 1 Processor -- start 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a066e6740 1 -- start start 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x558520298090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x558520298a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202a3370 0x5585202a5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a066e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5585201fe2f0 con 0x5585202985d0 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a066e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5585201a5cf0 con 0x5585202a3370 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a066e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5585201fa2b0 con 0x558520299b40 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f89f37fe640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89e80a32c0 0x7f89e80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a020a9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x558520298a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a020a9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x558520298a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59858/0 (socket says 172.21.15.89:59858) 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a020a9640 1 -- 172.21.15.89:0/27764712 learned_addr learned my addr 172.21.15.89:0/27764712 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a028aa640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89e80a32c0 0x7f89e80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a018a8640 1 --2- 172.21.15.89:0/27764712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x558520298090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f89f37fe640 1 -- 172.21.15.89:0/748852527 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f89f40d00f0 con 0x7f89fc106a80 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a018a8640 1 --2- 172.21.15.89:0/27764712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202a3370 0x5585202a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a020a9640 1 -- 172.21.15.89:0/27764712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 msgr2=0x558520298090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a020a9640 1 --2- 172.21.15.89:0/27764712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x558520298090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a020a9640 1 -- 172.21.15.89:0/27764712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202a3370 msgr2=0x5585202a5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a020a9640 1 --2- 172.21.15.89:0/27764712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202a3370 0x5585202a5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.396+0000 7f8a020a9640 1 -- 172.21.15.89:0/27764712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55852018a3a0 con 0x5585202985d0 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.397+0000 7f8a020a9640 1 --2- 172.21.15.89:0/27764712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x558520298a10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f89f802a940 tx=0x7f89f80316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db60e637ffedc74 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.397+0000 7f8a028aa640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89e80a32c0 0x7f89e80a5780 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f89f409c2b0 tx=0x7f89f406f1a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:51.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.397+0000 7f89f0dea640 1 -- 172.21.15.89:0/27764712 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8035040 con 0x5585202985d0 2024-09-15T23:26:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.397+0000 7f89f0dea640 1 -- 172.21.15.89:0/27764712 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f8005020 con 0x5585202985d0 2024-09-15T23:26:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.397+0000 7f89f0dea640 1 -- 172.21.15.89:0/27764712 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f803fbe0 con 0x5585202985d0 2024-09-15T23:26:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.397+0000 7f8a066e6740 1 -- 172.21.15.89:0/27764712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 msgr2=0x558520298a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.397+0000 7f8a066e6740 1 --2- 172.21.15.89:0/27764712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x558520298a10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f89f802a940 tx=0x7f89f80316e0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 -- 172.21.15.89:0/27764712 shutdown_connections 2024-09-15T23:26:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 --2- 172.21.15.89:0/27764712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202a3370 0x5585202a5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 --2- 172.21.15.89:0/27764712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x558520298a10 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 --2- 172.21.15.89:0/27764712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x558520298090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 -- 172.21.15.89:0/27764712 >> 172.21.15.89:0/27764712 conn(0x558520261310 msgr2=0x558520294c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 -- 172.21.15.89:0/27764712 shutdown_connections 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 -- 172.21.15.89:0/27764712 wait complete. 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 -- start start 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 Processor -- start 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x5585202a0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202a3370 0x5585202a0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5585202a6010 con 0x5585202985d0 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5585201b9060 con 0x5585202a3370 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5585202a6190 con 0x558520299b40 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a066e6740 10 client.0 _ll_register_callbacks cb 0x55851ff0c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a018a8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a018a8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202a2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59860/0 (socket says 172.21.15.89:59860) 2024-09-15T23:26:51.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.398+0000 7f8a018a8640 1 -- 172.21.15.89:0/1385560021 learned_addr learned my addr 172.21.15.89:0/1385560021 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:51.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a020a9640 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x5585202a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a018a8640 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202a3370 0x5585202a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a018a8640 1 -- 172.21.15.89:0/1385560021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 msgr2=0x5585202a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:51.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a018a8640 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x5585202a0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a018a8640 1 -- 172.21.15.89:0/1385560021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202a3370 msgr2=0x5585202a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:51.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a018a8640 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202a3370 0x5585202a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a018a8640 1 -- 172.21.15.89:0/1385560021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55851ff6f3c0 con 0x5585202985d0 2024-09-15T23:26:51.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a018a8640 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202a2d60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f89ec01ae10 tx=0x7f89ec009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2201f23ed695a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:51.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f89e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:51.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f89e5ffb640 1 -- 172.21.15.89:0/1385560021 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec015e00 con 0x5585202985d0 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f89e5ffb640 1 -- 172.21.15.89:0/1385560021 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f89ec01a0d0 con 0x5585202985d0 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a066e6740 1 -- 172.21.15.89:0/1385560021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89f8005020 con 0x5585202985d0 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f89e5ffb640 1 -- 172.21.15.89:0/1385560021 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f89ec02a050 con 0x5585202985d0 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a066e6740 10 client.16446 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f8a066e6740 1 -- 172.21.15.89:0/1385560021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5585202b1ab0 con 0x5585202985d0 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.399+0000 7f89e5ffb640 1 -- 172.21.15.89:0/1385560021 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f8005020 con 0x5585202985d0 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.400+0000 7f89e5ffb640 1 -- 172.21.15.89:0/1385560021 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f89ec025440 con 0x5585202985d0 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.400+0000 7f8a066e6740 10 client.16446 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.400+0000 7f8a066e6740 10 client.16446 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.400+0000 7f8a066e6740 20 client.16446 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.400+0000 7f8a066e6740 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5585202b2f70 0x5585202b5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.400+0000 7f8a066e6740 4 client.16446 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.400+0000 7f8a066e6740 1 -- 172.21.15.89:0/1385560021 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55852020b860 con 0x5585202b2f70 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.400+0000 7f8a020a9640 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5585202b2f70 0x5585202b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.400+0000 7f8a020a9640 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5585202b2f70 0x5585202b5360 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5868aa3b1d5ec756 server_cookie=e1a704b531b3b4eb in_seq=0 out_seq=0 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.400+0000 7f89e5ffb640 10 client.16446 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f89e5ffb640 1 -- 172.21.15.89:0/1385560021 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55852020b860 con 0x5585202b2f70 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f89e5ffb640 10 client.16446 handle_command_reply: tid=0 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f8a066e6740 1 client.16446 shutdown 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f8a066e6740 1 -- 172.21.15.89:0/1385560021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 msgr2=0x5585202a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f8a066e6740 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202a2d60 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f89ec01ae10 tx=0x7f89ec009110 comp rx=0 tx=0).stop 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f8a066e6740 1 -- 172.21.15.89:0/1385560021 shutdown_connections 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f8a066e6740 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5585202b2f70 0x5585202b5360 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f8a066e6740 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202a3370 0x5585202a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f8a066e6740 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x5585202a0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f8a066e6740 1 --2- 172.21.15.89:0/1385560021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202a2d60 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f8a066e6740 1 -- 172.21.15.89:0/1385560021 >> 172.21.15.89:0/1385560021 conn(0x558520261310 msgr2=0x55852029a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.401+0000 7f89e5ffb640 0 client.16446 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.402+0000 7f8a066e6740 1 -- 172.21.15.89:0/1385560021 shutdown_connections 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.402+0000 7f8a066e6740 1 -- 172.21.15.89:0/1385560021 wait complete. 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.402+0000 7f8a066e6740 20 client.16446 trim_cache size 0 max 16384 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.414+0000 7f8a066e6740 1 Processor -- start 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.414+0000 7f8a066e6740 1 -- start start 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.414+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202985d0 0x558520293bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.414+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x55852029b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.414+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202a3370 0x55852029b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.414+0000 7f8a066e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1acc60 con 0x5585202a3370 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.414+0000 7f8a066e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1ae7a0 con 0x5585202985d0 2024-09-15T23:26:51.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.414+0000 7f8a066e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1ad330 con 0x558520299b40 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a018a8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202a3370 0x55852029b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a018a8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202a3370 0x55852029b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59872/0 (socket says 172.21.15.89:59872) 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a018a8640 1 -- 172.21.15.89:0/4280108831 learned_addr learned my addr 172.21.15.89:0/4280108831 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a018a8640 1 --2- 172.21.15.89:0/4280108831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202985d0 0x558520293bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a020a9640 1 --2- 172.21.15.89:0/4280108831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x55852029b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a018a8640 1 -- 172.21.15.89:0/4280108831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 msgr2=0x55852029b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a018a8640 1 --2- 172.21.15.89:0/4280108831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x55852029b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a018a8640 1 -- 172.21.15.89:0/4280108831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202985d0 msgr2=0x558520293bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a018a8640 1 --2- 172.21.15.89:0/4280108831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202985d0 0x558520293bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a018a8640 1 -- 172.21.15.89:0/4280108831 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585202a3090 con 0x5585202a3370 2024-09-15T23:26:51.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a020a9640 1 --2- 172.21.15.89:0/4280108831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x55852029b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f8a018a8640 1 --2- 172.21.15.89:0/4280108831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202a3370 0x55852029b660 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f89ec01a750 tx=0x7f89ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb143a2beda0638 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f89e7fff640 1 -- 172.21.15.89:0/4280108831 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec009720 con 0x5585202a3370 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f89e7fff640 1 -- 172.21.15.89:0/4280108831 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89ec009b10 con 0x5585202a3370 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.415+0000 7f89e7fff640 1 -- 172.21.15.89:0/4280108831 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec018930 con 0x5585202a3370 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 -- 172.21.15.89:0/4280108831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202a3370 msgr2=0x55852029b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 --2- 172.21.15.89:0/4280108831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202a3370 0x55852029b660 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f89ec01a750 tx=0x7f89ec0046c0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 -- 172.21.15.89:0/4280108831 shutdown_connections 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 --2- 172.21.15.89:0/4280108831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202a3370 0x55852029b660 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 --2- 172.21.15.89:0/4280108831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558520299b40 0x55852029b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 --2- 172.21.15.89:0/4280108831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585202985d0 0x558520293bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 -- 172.21.15.89:0/4280108831 >> 172.21.15.89:0/4280108831 conn(0x558520261310 msgr2=0x5585202934e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 -- 172.21.15.89:0/4280108831 shutdown_connections 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 -- 172.21.15.89:0/4280108831 wait complete. 2024-09-15T23:26:51.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 -- start start 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 Processor -- start 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558520299b40 0x5585202abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5585202a3370 0x55852029c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1ae0d0 con 0x5585202985d0 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1ada00 con 0x558520299b40 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.416+0000 7f8a066e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1aac40 con 0x5585202a3370 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a066e6740 10 client.0 _ll_register_callbacks cb 0x7f89ec01aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a018a8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a018a8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202ab6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59874/0 (socket says 172.21.15.89:59874) 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a018a8640 1 -- 172.21.15.89:0/4221745582 learned_addr learned my addr 172.21.15.89:0/4221745582 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a020a9640 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558520299b40 0x5585202abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a018a8640 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5585202a3370 0x55852029c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a018a8640 1 -- 172.21.15.89:0/4221745582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5585202a3370 msgr2=0x55852029c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a018a8640 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5585202a3370 0x55852029c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a018a8640 1 -- 172.21.15.89:0/4221745582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558520299b40 msgr2=0x5585202abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a018a8640 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558520299b40 0x5585202abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a018a8640 1 -- 172.21.15.89:0/4221745582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55852029d190 con 0x5585202985d0 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a018a8640 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202ab6b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009150 tx=0x7f89ec025c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e576cc871f03f188 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f89e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f89e5ffb640 1 -- 172.21.15.89:0/4221745582 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ec032040 con 0x5585202985d0 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f89e5ffb640 1 -- 172.21.15.89:0/4221745582 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f89ec01a0d0 con 0x5585202985d0 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.417+0000 7f8a066e6740 1 -- 172.21.15.89:0/4221745582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89ec009b10 con 0x5585202985d0 2024-09-15T23:26:51.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f89e5ffb640 1 -- 172.21.15.89:0/4221745582 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f89ec01a2d0 con 0x5585202985d0 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f8a066e6740 10 client.16452 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f8a066e6740 1 -- 172.21.15.89:0/4221745582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55852031d6d0 con 0x5585202985d0 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f89e5ffb640 1 -- 172.21.15.89:0/4221745582 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89ec009b10 con 0x5585202985d0 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f89e5ffb640 1 -- 172.21.15.89:0/4221745582 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f89ec036020 con 0x5585202985d0 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f8a066e6740 10 client.16452 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f8a066e6740 10 client.16452 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f8a066e6740 20 client.16452 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f8a066e6740 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55852031dce0 0x5585203200d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f8a066e6740 4 client.16452 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f8a066e6740 1 -- 172.21.15.89:0/4221745582 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558520205d70 con 0x55852031dce0 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.418+0000 7f8a020a9640 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55852031dce0 0x5585203200d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a020a9640 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55852031dce0 0x5585203200d0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef8692e29a1424cd server_cookie=a51c9825dc21e1e4 in_seq=0 out_seq=0 2024-09-15T23:26:51.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f89e5ffb640 10 client.16452 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f89e5ffb640 1 -- 172.21.15.89:0/4221745582 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558520205d70 con 0x55852031dce0 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f89e5ffb640 10 client.16452 handle_command_reply: tid=0 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 client.16452 shutdown 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 -- 172.21.15.89:0/4221745582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 msgr2=0x5585202ab6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202ab6b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009150 tx=0x7f89ec025c00 comp rx=0 tx=0).stop 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 -- 172.21.15.89:0/4221745582 shutdown_connections 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55852031dce0 0x5585203200d0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5585202a3370 0x55852029c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558520299b40 0x5585202abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 --2- 172.21.15.89:0/4221745582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585202985d0 0x5585202ab6b0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 -- 172.21.15.89:0/4221745582 >> 172.21.15.89:0/4221745582 conn(0x558520261310 msgr2=0x558520293d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:51.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f89e5ffb640 0 client.16452 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:51.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 -- 172.21.15.89:0/4221745582 shutdown_connections 2024-09-15T23:26:51.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 1 -- 172.21.15.89:0/4221745582 wait complete. 2024-09-15T23:26:51.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.419+0000 7f8a066e6740 20 client.16452 trim_cache size 0 max 16384 2024-09-15T23:26:51.438 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:26:51.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 -- 172.21.15.89:0/748852527 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89e80a32c0 msgr2=0x7f89e80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:51.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89e80a32c0 0x7f89e80a5780 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f89f409c2b0 tx=0x7f89f406f1a0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 -- 172.21.15.89:0/748852527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89fc106a80 msgr2=0x7f89fc19cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:51.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89fc106a80 0x7f89fc19cdf0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f89f400cce0 tx=0x7f89f400a620 comp rx=0 tx=0).stop 2024-09-15T23:26:51.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 -- 172.21.15.89:0/748852527 shutdown_connections 2024-09-15T23:26:51.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89e80a32c0 0x7f89e80a5780 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f89fc106a80 0x7f89fc19cdf0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89fc101420 0x7f89fc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 --2- 172.21.15.89:0/748852527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89fc100a80 0x7f89fc198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:51.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 -- 172.21.15.89:0/748852527 >> 172.21.15.89:0/748852527 conn(0x7f89fc0fc7a0 msgr2=0x7f89fc06aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:51.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 -- 172.21.15.89:0/748852527 shutdown_connections 2024-09-15T23:26:51.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:51.420+0000 7f8a0430d640 1 -- 172.21.15.89:0/748852527 wait complete. 2024-09-15T23:26:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:52 smithi120 ceph-mon[35250]: pgmap v435: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:26:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:52 smithi174 ceph-mon[35586]: pgmap v435: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:26:52.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:52 smithi089 ceph-mon[30933]: pgmap v435: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:26:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.565+0000 7f91e5513640 1 -- 172.21.15.89:0/3357272858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91e0100a40 msgr2=0x7f91e0100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.565+0000 7f91e5513640 1 --2- 172.21.15.89:0/3357272858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91e0100a40 0x7f91e0100e40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009900 tx=0x7f91d002a410 comp rx=0 tx=0).stop 2024-09-15T23:26:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.565+0000 7f91e5513640 1 -- 172.21.15.89:0/3357272858 shutdown_connections 2024-09-15T23:26:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.565+0000 7f91e5513640 1 --2- 172.21.15.89:0/3357272858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e01025f0 0x7f91e0108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.565+0000 7f91e5513640 1 --2- 172.21.15.89:0/3357272858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91e0101c30 0x7f91e01020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.565+0000 7f91e5513640 1 --2- 172.21.15.89:0/3357272858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91e0100a40 0x7f91e0100e40 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.565+0000 7f91e5513640 1 -- 172.21.15.89:0/3357272858 >> 172.21.15.89:0/3357272858 conn(0x7f91e00fc1e0 msgr2=0x7f91e00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.566+0000 7f91e5513640 1 -- 172.21.15.89:0/3357272858 shutdown_connections 2024-09-15T23:26:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.566+0000 7f91e5513640 1 -- 172.21.15.89:0/3357272858 wait complete. 2024-09-15T23:26:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.566+0000 7f91e5513640 1 Processor -- start 2024-09-15T23:26:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.566+0000 7f91e5513640 1 -- start start 2024-09-15T23:26:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91e5513640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e0100a40 0x7f91e006a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91e5513640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91e0101c30 0x7f91e019caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91e5513640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91e01025f0 0x7f91e019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91e5513640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f91e010c6c0 con 0x7f91e0101c30 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91e5513640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f91e010c540 con 0x7f91e01025f0 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91e5513640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f91e010c840 con 0x7f91e0100a40 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91de7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91e0101c30 0x7f91e019caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91de7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91e0101c30 0x7f91e019caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59884/0 (socket says 172.21.15.89:59884) 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91de7fc640 1 -- 172.21.15.89:0/545614387 learned_addr learned my addr 172.21.15.89:0/545614387 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91deffd640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e0100a40 0x7f91e006a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91df7fe640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91e01025f0 0x7f91e019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91de7fc640 1 -- 172.21.15.89:0/545614387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e0100a40 msgr2=0x7f91e006a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91de7fc640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e0100a40 0x7f91e006a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91de7fc640 1 -- 172.21.15.89:0/545614387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91e01025f0 msgr2=0x7f91e019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91de7fc640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91e01025f0 0x7f91e019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91de7fc640 1 -- 172.21.15.89:0/545614387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e019f9d0 con 0x7f91e0101c30 2024-09-15T23:26:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91df7fe640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91e01025f0 0x7f91e019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91de7fc640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91e0101c30 0x7f91e019caa0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f91d400c920 tx=0x7f91d400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91c7fff640 1 -- 172.21.15.89:0/545614387 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4007c70 con 0x7f91e0101c30 2024-09-15T23:26:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91c7fff640 1 -- 172.21.15.89:0/545614387 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d4007e10 con 0x7f91e0101c30 2024-09-15T23:26:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91e5513640 1 -- 172.21.15.89:0/545614387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e019fc60 con 0x7f91e0101c30 2024-09-15T23:26:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.567+0000 7f91c7fff640 1 -- 172.21.15.89:0/545614387 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d4018890 con 0x7f91e0101c30 2024-09-15T23:26:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.568+0000 7f91e5513640 1 -- 172.21.15.89:0/545614387 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e01a01b0 con 0x7f91e0101c30 2024-09-15T23:26:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.570+0000 7f91c7fff640 1 -- 172.21.15.89:0/545614387 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f91d4018a70 con 0x7f91e0101c30 2024-09-15T23:26:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.570+0000 7f91e82a4740 1 Processor -- start 2024-09-15T23:26:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.570+0000 7f91e82a4740 1 -- start start 2024-09-15T23:26:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.570+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f25171f9a0 0x55f25171fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.570+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2517202e0 0x55f251724da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.570+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f251725370 0x55f251727830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.570+0000 7f91e82a4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f2516832f0 con 0x55f2517202e0 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.570+0000 7f91e82a4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f25162b270 con 0x55f251725370 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.570+0000 7f91e82a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f25167f2b0 con 0x55f25171f9a0 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91c7fff640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f91c80a32f0 0x7f91c80a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91deffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2517202e0 0x55f251724da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91deffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2517202e0 0x55f251724da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59888/0 (socket says 172.21.15.89:59888) 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91deffd640 1 -- 172.21.15.89:0/2657169364 learned_addr learned my addr 172.21.15.89:0/2657169364 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91df7fe640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f91c80a32f0 0x7f91c80a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91df7fe640 1 --2- 172.21.15.89:0/2657169364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f25171f9a0 0x55f25171fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91c7fff640 1 -- 172.21.15.89:0/545614387 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f91d40cff00 con 0x7f91e0101c30 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91de7fc640 1 --2- 172.21.15.89:0/2657169364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f251725370 0x55f251727830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91deffd640 1 -- 172.21.15.89:0/2657169364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f25171f9a0 msgr2=0x55f25171fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91deffd640 1 --2- 172.21.15.89:0/2657169364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f25171f9a0 0x55f25171fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91deffd640 1 -- 172.21.15.89:0/2657169364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f251725370 msgr2=0x55f251727830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91deffd640 1 --2- 172.21.15.89:0/2657169364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f251725370 0x55f251727830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91deffd640 1 -- 172.21.15.89:0/2657169364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f25162a340 con 0x55f2517202e0 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91df7fe640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f91c80a32f0 0x7f91c80a57b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009fd0 tx=0x7f91d8009d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91deffd640 1 --2- 172.21.15.89:0/2657169364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2517202e0 0x55f251724da0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f91d0031440 tx=0x7f91d0032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3993e61354c96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91c57fa640 1 -- 172.21.15.89:0/2657169364 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d003f070 con 0x55f2517202e0 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91c57fa640 1 -- 172.21.15.89:0/2657169364 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d00318b0 con 0x55f2517202e0 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.571+0000 7f91c57fa640 1 -- 172.21.15.89:0/2657169364 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0035070 con 0x55f2517202e0 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.572+0000 7f91e82a4740 1 -- 172.21.15.89:0/2657169364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2517202e0 msgr2=0x55f251724da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.572+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2657169364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2517202e0 0x55f251724da0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f91d0031440 tx=0x7f91d0032d60 comp rx=0 tx=0).stop 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.572+0000 7f91e82a4740 1 -- 172.21.15.89:0/2657169364 shutdown_connections 2024-09-15T23:26:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.572+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2657169364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f251725370 0x55f251727830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.572+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2657169364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2517202e0 0x55f251724da0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.572+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2657169364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f25171f9a0 0x55f25171fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.572+0000 7f91e82a4740 1 -- 172.21.15.89:0/2657169364 >> 172.21.15.89:0/2657169364 conn(0x55f25171b010 msgr2=0x55f25171b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.572+0000 7f91e82a4740 1 -- 172.21.15.89:0/2657169364 shutdown_connections 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.572+0000 7f91e82a4740 1 -- 172.21.15.89:0/2657169364 wait complete. 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91e82a4740 1 -- start start 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91e82a4740 1 Processor -- start 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 0x55f25172a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 0x55f251731110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91e82a4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f2517282e0 con 0x55f25171f9a0 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91e82a4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f25160c290 con 0x55f2517202e0 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91e82a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f251728460 con 0x55f251725370 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91e82a4740 10 client.0 _ll_register_callbacks cb 0x55f25163d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91deffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91deffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59900/0 (socket says 172.21.15.89:59900) 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91deffd640 1 -- 172.21.15.89:0/2839631311 learned_addr learned my addr 172.21.15.89:0/2839631311 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91df7fe640 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 0x55f251731110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91de7fc640 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 0x55f25172a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91deffd640 1 -- 172.21.15.89:0/2839631311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 msgr2=0x55f251731110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91deffd640 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 0x55f251731110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91deffd640 1 -- 172.21.15.89:0/2839631311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 msgr2=0x55f25172a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91deffd640 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 0x55f25172a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91deffd640 1 -- 172.21.15.89:0/2839631311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f251396210 con 0x55f25171f9a0 2024-09-15T23:26:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91de7fc640 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 0x55f25172a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91deffd640 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729b90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f91d00314e0 tx=0x7f91d0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d7fe5b659ab911 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.573+0000 7f91c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91c2ff5640 1 -- 172.21.15.89:0/2839631311 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d004a020 con 0x55f25171f9a0 2024-09-15T23:26:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91c2ff5640 1 -- 172.21.15.89:0/2839631311 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f91d0003c20 con 0x55f25171f9a0 2024-09-15T23:26:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91e82a4740 1 -- 172.21.15.89:0/2839631311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91d00318b0 con 0x55f25171f9a0 2024-09-15T23:26:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91c2ff5640 1 -- 172.21.15.89:0/2839631311 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f91d0055050 con 0x55f25171f9a0 2024-09-15T23:26:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91e82a4740 10 client.16464 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91e82a4740 1 -- 172.21.15.89:0/2839631311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f251462620 con 0x55f25171f9a0 2024-09-15T23:26:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91c2ff5640 1 -- 172.21.15.89:0/2839631311 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d00318b0 con 0x55f25171f9a0 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91c2ff5640 1 -- 172.21.15.89:0/2839631311 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f91d003bc10 con 0x55f25171f9a0 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91e82a4740 10 client.16464 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91e82a4740 10 client.16464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91e82a4740 20 client.16464 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f251735460 0x55f251737870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91e82a4740 4 client.16464 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.574+0000 7f91e82a4740 1 -- 172.21.15.89:0/2839631311 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f251690a20 con 0x55f251735460 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.575+0000 7f91de7fc640 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f251735460 0x55f251737870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.575+0000 7f91c2ff5640 10 client.16464 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.575+0000 7f91de7fc640 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f251735460 0x55f251737870 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ab328efa5bd64e3 server_cookie=3de5bd23f432db32 in_seq=0 out_seq=0 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91c2ff5640 1 -- 172.21.15.89:0/2839631311 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f251690a20 con 0x55f251735460 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91c2ff5640 10 client.16464 handle_command_reply: tid=0 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 client.16464 shutdown 2024-09-15T23:26:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 -- 172.21.15.89:0/2839631311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 msgr2=0x55f251729b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729b90 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f91d00314e0 tx=0x7f91d0003820 comp rx=0 tx=0).stop 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 -- 172.21.15.89:0/2839631311 shutdown_connections 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f251735460 0x55f251737870 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 0x55f251731110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 0x55f25172a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91c2ff5640 0 client.16464 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2839631311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729b90 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 -- 172.21.15.89:0/2839631311 >> 172.21.15.89:0/2839631311 conn(0x55f25171b010 msgr2=0x55f25171c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 -- 172.21.15.89:0/2839631311 shutdown_connections 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 1 -- 172.21.15.89:0/2839631311 wait complete. 2024-09-15T23:26:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.576+0000 7f91e82a4740 20 client.16464 trim_cache size 0 max 16384 2024-09-15T23:26:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.590+0000 7f91e82a4740 1 Processor -- start 2024-09-15T23:26:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.590+0000 7f91e82a4740 1 -- start start 2024-09-15T23:26:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.590+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.590+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2517202e0 0x55f2517310d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.590+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f251725370 0x55f25171c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.590+0000 7f91e82a4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01aa720 con 0x55f25171f9a0 2024-09-15T23:26:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.590+0000 7f91e82a4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01ac260 con 0x55f251725370 2024-09-15T23:26:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.590+0000 7f91e82a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01aadf0 con 0x55f2517202e0 2024-09-15T23:26:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.590+0000 7f91deffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91deffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59912/0 (socket says 172.21.15.89:59912) 2024-09-15T23:26:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91deffd640 1 -- 172.21.15.89:0/2950690524 learned_addr learned my addr 172.21.15.89:0/2950690524 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91deffd640 1 -- 172.21.15.89:0/2950690524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2517202e0 msgr2=0x55f2517310d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91deffd640 1 --2- 172.21.15.89:0/2950690524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2517202e0 0x55f2517310d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91deffd640 1 -- 172.21.15.89:0/2950690524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f251725370 msgr2=0x55f25171c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91deffd640 1 --2- 172.21.15.89:0/2950690524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f251725370 0x55f25171c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91deffd640 1 -- 172.21.15.89:0/2950690524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f251719650 con 0x55f25171f9a0 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91deffd640 1 --2- 172.21.15.89:0/2950690524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729fb0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f91d002a930 tx=0x7f91d003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14ca188b6d2f222 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91c4ff9640 1 -- 172.21.15.89:0/2950690524 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d004a070 con 0x55f25171f9a0 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91c4ff9640 1 -- 172.21.15.89:0/2950690524 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d0003e20 con 0x55f25171f9a0 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.591+0000 7f91c4ff9640 1 -- 172.21.15.89:0/2950690524 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d003f070 con 0x55f25171f9a0 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 -- 172.21.15.89:0/2950690524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 msgr2=0x55f251729fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2950690524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729fb0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f91d002a930 tx=0x7f91d003e000 comp rx=0 tx=0).stop 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 -- 172.21.15.89:0/2950690524 shutdown_connections 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2950690524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f251725370 0x55f25171c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2950690524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f2517202e0 0x55f2517310d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 --2- 172.21.15.89:0/2950690524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f251729fb0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 -- 172.21.15.89:0/2950690524 >> 172.21.15.89:0/2950690524 conn(0x55f25171b010 msgr2=0x55f25171a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 -- 172.21.15.89:0/2950690524 shutdown_connections 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 -- 172.21.15.89:0/2950690524 wait complete. 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 -- start start 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 Processor -- start 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f25172ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 0x55f25172f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 0x55f251730280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01abb90 con 0x55f25171f9a0 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01ab4c0 con 0x55f2517202e0 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f91e01a8700 con 0x55f251725370 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91e82a4740 10 client.0 _ll_register_callbacks cb 0x7f91d8014650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91deffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f25172ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91deffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f25172ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59922/0 (socket says 172.21.15.89:59922) 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91deffd640 1 -- 172.21.15.89:0/1708568429 learned_addr learned my addr 172.21.15.89:0/1708568429 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.592+0000 7f91de7fc640 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 0x55f25172f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91df7fe640 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 0x55f251730280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91deffd640 1 -- 172.21.15.89:0/1708568429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 msgr2=0x55f251730280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91deffd640 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 0x55f251730280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91deffd640 1 -- 172.21.15.89:0/1708568429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 msgr2=0x55f25172f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91deffd640 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 0x55f25172f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91deffd640 1 -- 172.21.15.89:0/1708568429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f251730890 con 0x55f25171f9a0 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91df7fe640 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 0x55f251730280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91deffd640 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f25172ed30 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009900 tx=0x7f91d0032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3695ae36df6b3c30 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91c2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91c2ff5640 1 -- 172.21.15.89:0/1708568429 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d005e060 con 0x55f25171f9a0 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91c2ff5640 1 -- 172.21.15.89:0/1708568429 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f91d00055c0 con 0x55f25171f9a0 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91e82a4740 1 -- 172.21.15.89:0/1708568429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91d0003e20 con 0x55f25171f9a0 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91c2ff5640 1 -- 172.21.15.89:0/1708568429 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f91d0065050 con 0x55f25171f9a0 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91e82a4740 10 client.16470 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91e82a4740 1 -- 172.21.15.89:0/1708568429 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f2517a0670 con 0x55f25171f9a0 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91c2ff5640 1 -- 172.21.15.89:0/1708568429 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d0003e20 con 0x55f25171f9a0 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91c2ff5640 1 -- 172.21.15.89:0/1708568429 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f91d00046c0 con 0x55f25171f9a0 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91e82a4740 10 client.16470 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91e82a4740 10 client.16470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91e82a4740 20 client.16470 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91e82a4740 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f251735460 0x55f2517a3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91e82a4740 4 client.16470 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.593+0000 7f91e82a4740 1 -- 172.21.15.89:0/1708568429 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f25168ad70 con 0x55f251735460 2024-09-15T23:26:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.594+0000 7f91de7fc640 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f251735460 0x55f2517a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.594+0000 7f91c2ff5640 10 client.16470 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.594+0000 7f91de7fc640 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f251735460 0x55f2517a3030 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2013a5f953d887a server_cookie=170a5042700d9402 in_seq=0 out_seq=0 2024-09-15T23:26:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91c2ff5640 1 -- 172.21.15.89:0/1708568429 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f25168ad70 con 0x55f251735460 2024-09-15T23:26:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91c2ff5640 10 client.16470 handle_command_reply: tid=0 2024-09-15T23:26:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 client.16470 shutdown 2024-09-15T23:26:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 -- 172.21.15.89:0/1708568429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 msgr2=0x55f25172ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f25172ed30 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009900 tx=0x7f91d0032d60 comp rx=0 tx=0).stop 2024-09-15T23:26:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 -- 172.21.15.89:0/1708568429 shutdown_connections 2024-09-15T23:26:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f251735460 0x55f2517a3030 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f251725370 0x55f251730280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2517202e0 0x55f25172f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 --2- 172.21.15.89:0/1708568429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f25171f9a0 0x55f25172ed30 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91c2ff5640 0 client.16470 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 -- 172.21.15.89:0/1708568429 >> 172.21.15.89:0/1708568429 conn(0x55f25171b010 msgr2=0x55f25171b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 -- 172.21.15.89:0/1708568429 shutdown_connections 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 1 -- 172.21.15.89:0/1708568429 wait complete. 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.607+0000 7f91e82a4740 20 client.16470 trim_cache size 0 max 16384 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:26:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:26:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 -- 172.21.15.89:0/545614387 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f91c80a32f0 msgr2=0x7f91c80a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f91c80a32f0 0x7f91c80a57b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009fd0 tx=0x7f91d8009d60 comp rx=0 tx=0).stop 2024-09-15T23:26:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 -- 172.21.15.89:0/545614387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91e0101c30 msgr2=0x7f91e019caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91e0101c30 0x7f91e019caa0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f91d400c920 tx=0x7f91d400cde0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 -- 172.21.15.89:0/545614387 shutdown_connections 2024-09-15T23:26:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f91c80a32f0 0x7f91c80a57b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f91e01025f0 0x7f91e019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f91e0101c30 0x7f91e019caa0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 --2- 172.21.15.89:0/545614387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e0100a40 0x7f91e006a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 -- 172.21.15.89:0/545614387 >> 172.21.15.89:0/545614387 conn(0x7f91e00fc1e0 msgr2=0x7f91e0104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 -- 172.21.15.89:0/545614387 shutdown_connections 2024-09-15T23:26:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:26:52.608+0000 7f91e5513640 1 -- 172.21.15.89:0/545614387 wait complete. 2024-09-15T23:26:53.347 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:26:53.348 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:26:53.608 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.106+0000 7fc992400640 1 -- 172.21.15.89:0/1909595276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c102790 msgr2=0x7fc98c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.106+0000 7fc992400640 1 --2- 172.21.15.89:0/1909595276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c102790 0x7fc98c1067e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc980009900 tx=0x7fc98002a410 comp rx=0 tx=0).stop 2024-09-15T23:26:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.106+0000 7fc992400640 1 -- 172.21.15.89:0/1909595276 shutdown_connections 2024-09-15T23:26:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.106+0000 7fc992400640 1 --2- 172.21.15.89:0/1909595276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc98c106f10 0x7fc98c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.106+0000 7fc992400640 1 --2- 172.21.15.89:0/1909595276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c102790 0x7fc98c1067e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.106+0000 7fc992400640 1 --2- 172.21.15.89:0/1909595276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc98c101de0 0x7fc98c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.106+0000 7fc992400640 1 -- 172.21.15.89:0/1909595276 >> 172.21.15.89:0/1909595276 conn(0x7fc98c0fd8c0 msgr2=0x7fc98c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:54.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.106+0000 7fc992400640 1 -- 172.21.15.89:0/1909595276 shutdown_connections 2024-09-15T23:26:54.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.106+0000 7fc992400640 1 -- 172.21.15.89:0/1909595276 wait complete. 2024-09-15T23:26:54.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc992400640 1 Processor -- start 2024-09-15T23:26:54.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc992400640 1 -- start start 2024-09-15T23:26:54.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc992400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc98c101de0 0x7fc98c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc992400640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c102790 0x7fc98c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc992400640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc98c106f10 0x7fc98c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc992400640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c111f90 con 0x7fc98c102790 2024-09-15T23:26:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc992400640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c111e10 con 0x7fc98c106f10 2024-09-15T23:26:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc992400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c112110 con 0x7fc98c101de0 2024-09-15T23:26:54.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc98b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c102790 0x7fc98c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc98b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c102790 0x7fc98c1989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59930/0 (socket says 172.21.15.89:59930) 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc98b7fe640 1 -- 172.21.15.89:0/1671472968 learned_addr learned my addr 172.21.15.89:0/1671472968 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc98bfff640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc98c101de0 0x7fc98c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.107+0000 7fc99099d640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc98c106f10 0x7fc98c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc98b7fe640 1 -- 172.21.15.89:0/1671472968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc98c101de0 msgr2=0x7fc98c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc98b7fe640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc98c101de0 0x7fc98c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc98b7fe640 1 -- 172.21.15.89:0/1671472968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc98c106f10 msgr2=0x7fc98c19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc98b7fe640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc98c106f10 0x7fc98c19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc98b7fe640 1 -- 172.21.15.89:0/1671472968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98c19d420 con 0x7fc98c102790 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc99099d640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc98c106f10 0x7fc98c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc98b7fe640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c102790 0x7fc98c1989b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc980006010 tx=0x7fc9800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:54.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc9897fa640 1 -- 172.21.15.89:0/1671472968 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98003f030 con 0x7fc98c102790 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc992400640 1 -- 172.21.15.89:0/1671472968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc98c19d6b0 con 0x7fc98c102790 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc9897fa640 1 -- 172.21.15.89:0/1671472968 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc980003ce0 con 0x7fc98c102790 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc9897fa640 1 -- 172.21.15.89:0/1671472968 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980035070 con 0x7fc98c102790 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.108+0000 7fc992400640 1 -- 172.21.15.89:0/1671472968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc98c1a4ff0 con 0x7fc98c102790 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.110+0000 7fc9897fa640 1 -- 172.21.15.89:0/1671472968 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc980042030 con 0x7fc98c102790 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.110+0000 7fc9947d9740 1 Processor -- start 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.110+0000 7fc9947d9740 1 -- start start 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 0x5622588fae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258909c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x56225890c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc9947d9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562258860280 con 0x56225890a150 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc9947d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562258807b10 con 0x5622588faa60 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc9947d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56225885c240 con 0x5622588fb380 2024-09-15T23:26:54.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x56225890c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258909c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x56225890c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59944/0 (socket says 172.21.15.89:59944) 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98b7fe640 1 -- 172.21.15.89:0/3155710323 learned_addr learned my addr 172.21.15.89:0/3155710323 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc99099d640 1 --2- 172.21.15.89:0/3155710323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 0x5622588fae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98b7fe640 1 -- 172.21.15.89:0/3155710323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 msgr2=0x562258909c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98b7fe640 1 --2- 172.21.15.89:0/3155710323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258909c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98b7fe640 1 -- 172.21.15.89:0/3155710323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 msgr2=0x5622588fae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98b7fe640 1 --2- 172.21.15.89:0/3155710323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 0x5622588fae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc9897fa640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3240 0x7fc9740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98b7fe640 1 -- 172.21.15.89:0/3155710323 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622588141f0 con 0x56225890a150 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc99099d640 1 --2- 172.21.15.89:0/3155710323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 0x5622588fae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98bfff640 1 --2- 172.21.15.89:0/3155710323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258909c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:54.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc99099d640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3240 0x7fc9740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc98b7fe640 1 --2- 172.21.15.89:0/3155710323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x56225890c540 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc9800eda10 tx=0x7fc9800b8950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26bf8be137f33d15 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.111+0000 7fc9897fa640 1 -- 172.21.15.89:0/1671472968 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc9800f03c0 con 0x7fc98c102790 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9727fc640 1 -- 172.21.15.89:0/3155710323 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980049020 con 0x56225890a150 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9727fc640 1 -- 172.21.15.89:0/3155710323 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9800bc6b0 con 0x56225890a150 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9727fc640 1 -- 172.21.15.89:0/3155710323 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9800bc480 con 0x56225890a150 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9947d9740 1 -- 172.21.15.89:0/3155710323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 msgr2=0x56225890c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc99099d640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3240 0x7fc9740a5700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc98400a510 tx=0x7fc9840096f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9947d9740 1 --2- 172.21.15.89:0/3155710323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x56225890c540 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc9800eda10 tx=0x7fc9800b8950 comp rx=0 tx=0).stop 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9947d9740 1 -- 172.21.15.89:0/3155710323 shutdown_connections 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9947d9740 1 --2- 172.21.15.89:0/3155710323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x56225890c540 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9947d9740 1 --2- 172.21.15.89:0/3155710323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258909c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9947d9740 1 --2- 172.21.15.89:0/3155710323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 0x5622588fae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9947d9740 1 -- 172.21.15.89:0/3155710323 >> 172.21.15.89:0/3155710323 conn(0x5622588f70f0 msgr2=0x5622588f74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:54.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.112+0000 7fc9947d9740 1 -- 172.21.15.89:0/3155710323 shutdown_connections 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc9947d9740 1 -- 172.21.15.89:0/3155710323 wait complete. 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc9947d9740 1 -- start start 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc9947d9740 1 Processor -- start 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 0x562258907050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258907590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x562258907ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc9947d9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56225890cb40 con 0x56225890a150 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc9947d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56225881af10 con 0x5622588faa60 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc9947d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56225890ccc0 con 0x5622588fb380 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc9947d9740 10 client.0 _ll_register_callbacks cb 0x5622587e3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc99099d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x562258907ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc99099d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x562258907ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59958/0 (socket says 172.21.15.89:59958) 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.113+0000 7fc99099d640 1 -- 172.21.15.89:0/65557405 learned_addr learned my addr 172.21.15.89:0/65557405 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:54.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc98bfff640 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 0x562258907050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc98b7fe640 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258907590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc99099d640 1 -- 172.21.15.89:0/65557405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 msgr2=0x562258907590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc99099d640 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258907590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc99099d640 1 -- 172.21.15.89:0/65557405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 msgr2=0x562258907050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc99099d640 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 0x562258907050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc99099d640 1 -- 172.21.15.89:0/65557405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622585d16a0 con 0x56225890a150 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc98bfff640 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 0x562258907050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc99099d640 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x562258907ad0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc984007de0 tx=0x7fc984004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d52065cb2fe57b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc96b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc96b7fe640 1 -- 172.21.15.89:0/65557405 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984028060 con 0x56225890a150 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc96b7fe640 1 -- 172.21.15.89:0/65557405 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc98401a930 con 0x56225890a150 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc9947d9740 1 -- 172.21.15.89:0/65557405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56225886d7f0 con 0x56225890a150 2024-09-15T23:26:54.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc96b7fe640 1 -- 172.21.15.89:0/65557405 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc98402f050 con 0x56225890a150 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc9947d9740 10 client.16482 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc9947d9740 1 -- 172.21.15.89:0/65557405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5622589186f0 con 0x56225890a150 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.114+0000 7fc96b7fe640 1 -- 172.21.15.89:0/65557405 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56225886d7f0 con 0x56225890a150 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.115+0000 7fc96b7fe640 1 -- 172.21.15.89:0/65557405 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc984023bc0 con 0x56225890a150 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.115+0000 7fc9947d9740 10 client.16482 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.115+0000 7fc9947d9740 10 client.16482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.115+0000 7fc9947d9740 20 client.16482 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.115+0000 7fc9947d9740 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562258919bb0 0x56225891c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.115+0000 7fc9947d9740 4 client.16482 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.115+0000 7fc9947d9740 1 -- 172.21.15.89:0/65557405 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562258867d00 con 0x562258919bb0 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.115+0000 7fc98bfff640 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562258919bb0 0x56225891c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.115+0000 7fc96b7fe640 10 client.16482 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.115+0000 7fc98bfff640 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562258919bb0 0x56225891c000 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdb2eb2fe4145ba6 server_cookie=46f8d8d6dae5c0e6 in_seq=0 out_seq=0 2024-09-15T23:26:54.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.116+0000 7fc96b7fe640 1 -- 172.21.15.89:0/65557405 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562258867d00 con 0x562258919bb0 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.116+0000 7fc96b7fe640 10 client.16482 handle_command_reply: tid=0 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.116+0000 7fc9947d9740 1 client.16482 shutdown 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 1 -- 172.21.15.89:0/65557405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 msgr2=0x562258907ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x562258907ad0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc984007de0 tx=0x7fc984004b50 comp rx=0 tx=0).stop 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 1 -- 172.21.15.89:0/65557405 shutdown_connections 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562258919bb0 0x56225891c000 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x562258907ad0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258907590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 1 --2- 172.21.15.89:0/65557405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588faa60 0x562258907050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc96b7fe640 0 client.16482 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 1 -- 172.21.15.89:0/65557405 >> 172.21.15.89:0/65557405 conn(0x5622588f70f0 msgr2=0x5622588fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 1 -- 172.21.15.89:0/65557405 shutdown_connections 2024-09-15T23:26:54.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 1 -- 172.21.15.89:0/65557405 wait complete. 2024-09-15T23:26:54.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.117+0000 7fc9947d9740 20 client.16482 trim_cache size 0 max 16384 2024-09-15T23:26:54.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc9947d9740 1 Processor -- start 2024-09-15T23:26:54.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc9947d9740 1 -- start start 2024-09-15T23:26:54.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622588faa60 0x562258921920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258921e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56225890a150 0x5622589042b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc9947d9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1acc90 con 0x5622588faa60 2024-09-15T23:26:54.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc9947d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1ae7d0 con 0x56225890a150 2024-09-15T23:26:54.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc9947d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1ad360 con 0x5622588fb380 2024-09-15T23:26:54.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622588faa60 0x562258921920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622588faa60 0x562258921920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59962/0 (socket says 172.21.15.89:59962) 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98bfff640 1 -- 172.21.15.89:0/701899410 learned_addr learned my addr 172.21.15.89:0/701899410 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98bfff640 1 -- 172.21.15.89:0/701899410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 msgr2=0x562258921e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98b7fe640 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258921e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98bfff640 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258921e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98bfff640 1 -- 172.21.15.89:0/701899410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56225890a150 msgr2=0x5622589042b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc99099d640 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56225890a150 0x5622589042b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98bfff640 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56225890a150 0x5622589042b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98bfff640 1 -- 172.21.15.89:0/701899410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622588f1680 con 0x5622588faa60 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98b7fe640 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258921e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc99099d640 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56225890a150 0x5622589042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.135+0000 7fc98bfff640 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622588faa60 0x562258921920 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00bd90 tx=0x7fc97c00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a015d882f41e0e9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc971ffb640 1 -- 172.21.15.89:0/701899410 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c004220 con 0x5622588faa60 2024-09-15T23:26:54.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc971ffb640 1 -- 172.21.15.89:0/701899410 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc97c0043c0 con 0x5622588faa60 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc971ffb640 1 -- 172.21.15.89:0/701899410 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c003850 con 0x5622588faa60 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc9947d9740 1 -- 172.21.15.89:0/701899410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622588faa60 msgr2=0x562258921920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc9947d9740 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622588faa60 0x562258921920 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00bd90 tx=0x7fc97c00b410 comp rx=0 tx=0).stop 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc9947d9740 1 -- 172.21.15.89:0/701899410 shutdown_connections 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc9947d9740 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56225890a150 0x5622589042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc9947d9740 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588fb380 0x562258921e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc9947d9740 1 --2- 172.21.15.89:0/701899410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622588faa60 0x562258921920 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc9947d9740 1 -- 172.21.15.89:0/701899410 >> 172.21.15.89:0/701899410 conn(0x5622588f70f0 msgr2=0x562258908fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc9947d9740 1 -- 172.21.15.89:0/701899410 shutdown_connections 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.136+0000 7fc9947d9740 1 -- 172.21.15.89:0/701899410 wait complete. 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc9947d9740 1 -- start start 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc9947d9740 1 Processor -- start 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588faa60 0x56225890dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588fb380 0x56225890e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc9947d9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x5622589838f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc9947d9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1ae100 con 0x56225890a150 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc9947d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1ada30 con 0x5622588fb380 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc9947d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1aac70 con 0x5622588faa60 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc9947d9740 10 client.0 _ll_register_callbacks cb 0x7fc984009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc99099d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x5622589838f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc99099d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x5622589838f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59972/0 (socket says 172.21.15.89:59972) 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc99099d640 1 -- 172.21.15.89:0/159266552 learned_addr learned my addr 172.21.15.89:0/159266552 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.137+0000 7fc98b7fe640 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588fb380 0x56225890e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc98bfff640 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588faa60 0x56225890dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc99099d640 1 -- 172.21.15.89:0/159266552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588faa60 msgr2=0x56225890dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc99099d640 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588faa60 0x56225890dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc99099d640 1 -- 172.21.15.89:0/159266552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588fb380 msgr2=0x56225890e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc99099d640 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588fb380 0x56225890e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc99099d640 1 -- 172.21.15.89:0/159266552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622588f8280 con 0x56225890a150 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc98bfff640 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588faa60 0x56225890dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc99099d640 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x5622589838f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc984009d40 tx=0x7fc984007ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454fe6ad6c24c459 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc96b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc96b7fe640 1 -- 172.21.15.89:0/159266552 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984038060 con 0x56225890a150 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc96b7fe640 1 -- 172.21.15.89:0/159266552 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc984018920 con 0x56225890a150 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc9947d9740 1 -- 172.21.15.89:0/159266552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc97c0043c0 con 0x56225890a150 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc96b7fe640 1 -- 172.21.15.89:0/159266552 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc984028040 con 0x56225890a150 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc9947d9740 10 client.16488 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc9947d9740 1 -- 172.21.15.89:0/159266552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562258984d80 con 0x56225890a150 2024-09-15T23:26:54.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.138+0000 7fc96b7fe640 1 -- 172.21.15.89:0/159266552 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc97c0043c0 con 0x56225890a150 2024-09-15T23:26:54.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.139+0000 7fc96b7fe640 1 -- 172.21.15.89:0/159266552 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc984023500 con 0x56225890a150 2024-09-15T23:26:54.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.139+0000 7fc9947d9740 10 client.16488 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:54.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.139+0000 7fc9947d9740 10 client.16488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:54.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.139+0000 7fc9947d9740 20 client.16488 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:54.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.139+0000 7fc9947d9740 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622589854d0 0x5622589878f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:54.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.139+0000 7fc9947d9740 4 client.16488 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:26:54.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.139+0000 7fc9947d9740 1 -- 172.21.15.89:0/159266552 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5622587e4110 con 0x5622589854d0 2024-09-15T23:26:54.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.139+0000 7fc98bfff640 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622589854d0 0x5622589878f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:54.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.139+0000 7fc96b7fe640 10 client.16488 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:54.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.139+0000 7fc98bfff640 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622589854d0 0x5622589878f0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=911a81cc0217495a server_cookie=3053fc38f25bdc3 in_seq=0 out_seq=0 2024-09-15T23:26:54.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.140+0000 7fc96b7fe640 1 -- 172.21.15.89:0/159266552 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5622587e4110 con 0x5622589854d0 2024-09-15T23:26:54.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.140+0000 7fc96b7fe640 10 client.16488 handle_command_reply: tid=0 2024-09-15T23:26:54.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.140+0000 7fc9947d9740 1 client.16488 shutdown 2024-09-15T23:26:54.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 1 -- 172.21.15.89:0/159266552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 msgr2=0x5622589838f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:54.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x5622589838f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc984009d40 tx=0x7fc984007ac0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 1 -- 172.21.15.89:0/159266552 shutdown_connections 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622589854d0 0x5622589878f0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56225890a150 0x5622589838f0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622588fb380 0x56225890e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 1 --2- 172.21.15.89:0/159266552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622588faa60 0x56225890dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc96b7fe640 0 client.16488 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 1 -- 172.21.15.89:0/159266552 >> 172.21.15.89:0/159266552 conn(0x5622588f70f0 msgr2=0x5622588fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 1 -- 172.21.15.89:0/159266552 shutdown_connections 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 1 -- 172.21.15.89:0/159266552 wait complete. 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc9947d9740 20 client.16488 trim_cache size 0 max 16384 2024-09-15T23:26:54.148 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:26:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:26:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "f01ebea3-45df-4474-a870-64dc01292b4a", 2024-09-15T23:26:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:26:54.149 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:26:54.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc992400640 1 -- 172.21.15.89:0/1671472968 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3240 msgr2=0x7fc9740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:54.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc992400640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3240 0x7fc9740a5700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc98400a510 tx=0x7fc9840096f0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc992400640 1 -- 172.21.15.89:0/1671472968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c102790 msgr2=0x7fc98c1989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:54.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.141+0000 7fc992400640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c102790 0x7fc98c1989b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc980006010 tx=0x7fc9800042f0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.142+0000 7fc992400640 1 -- 172.21.15.89:0/1671472968 shutdown_connections 2024-09-15T23:26:54.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.142+0000 7fc992400640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3240 0x7fc9740a5700 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.142+0000 7fc992400640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc98c106f10 0x7fc98c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.142+0000 7fc992400640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc98c102790 0x7fc98c1989b0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.142+0000 7fc992400640 1 --2- 172.21.15.89:0/1671472968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc98c101de0 0x7fc98c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:54.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.142+0000 7fc992400640 1 -- 172.21.15.89:0/1671472968 >> 172.21.15.89:0/1671472968 conn(0x7fc98c0fd8c0 msgr2=0x7fc98c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:54.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.142+0000 7fc992400640 1 -- 172.21.15.89:0/1671472968 shutdown_connections 2024-09-15T23:26:54.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:54.142+0000 7fc992400640 1 -- 172.21.15.89:0/1671472968 wait complete. 2024-09-15T23:26:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:54 smithi089 ceph-mon[30933]: pgmap v436: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:26:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:54 smithi120 ceph-mon[35250]: pgmap v436: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:26:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:54 smithi174 ceph-mon[35586]: pgmap v436: 129 pgs: 129 active+clean; 1.4 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:26:54.988 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:26:55.241 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:55.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:55 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:26:55.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:55 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:55.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:55 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:26:55.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:55 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:55 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:26:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:55 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:55.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:55 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:26:55.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:55 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:55 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:26:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:55 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:55 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:26:55.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:55 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:55.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 -- 172.21.15.89:0/1323508680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fcc102d80 msgr2=0x7f8fcc103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:55.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 --2- 172.21.15.89:0/1323508680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fcc102d80 0x7f8fcc103160 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc009900 tx=0x7f8fbc02a480 comp rx=0 tx=0).stop 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 -- 172.21.15.89:0/1323508680 shutdown_connections 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 --2- 172.21.15.89:0/1323508680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc10c160 0x7f8fcc10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 --2- 172.21.15.89:0/1323508680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fcc1036a0 0x7f8fcc10bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 --2- 172.21.15.89:0/1323508680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fcc102d80 0x7f8fcc103160 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 -- 172.21.15.89:0/1323508680 >> 172.21.15.89:0/1323508680 conn(0x7f8fcc0fc780 msgr2=0x7f8fcc0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 -- 172.21.15.89:0/1323508680 shutdown_connections 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 -- 172.21.15.89:0/1323508680 wait complete. 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 Processor -- start 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.731+0000 7f8fd2381640 1 -- start start 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fd2381640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fcc102d80 0x7f8fcc1009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fd2381640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fcc1036a0 0x7f8fcc100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fd2381640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc10c160 0x7f8fcc101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fd2381640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc111f40 con 0x7f8fcc1036a0 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fd2381640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc111dc0 con 0x7f8fcc10c160 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fd2381640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc1120c0 con 0x7f8fcc102d80 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fcc1036a0 0x7f8fcc100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fcc1036a0 0x7f8fcc100f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35266/0 (socket says 172.21.15.89:35266) 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/1768920534 learned_addr learned my addr 172.21.15.89:0/1768920534 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fcc102d80 0x7f8fcc1009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fd091e640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc10c160 0x7f8fcc101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/1768920534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fcc102d80 msgr2=0x7f8fcc1009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fcc102d80 0x7f8fcc1009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/1768920534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc10c160 msgr2=0x7f8fcc101460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc10c160 0x7f8fcc101460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/1768920534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fcc1a6950 con 0x7f8fcc1036a0 2024-09-15T23:26:55.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fcc102d80 0x7f8fcc1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fd091e640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc10c160 0x7f8fcc101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fcc1036a0 0x7f8fcc100f20 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400d770 tx=0x7f8fb400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fc97fa640 1 -- 172.21.15.89:0/1768920534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4017070 con 0x7f8fcc1036a0 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fc97fa640 1 -- 172.21.15.89:0/1768920534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fb4005500 con 0x7f8fcc1036a0 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.732+0000 7f8fd2381640 1 -- 172.21.15.89:0/1768920534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fcc1a6c40 con 0x7f8fcc1036a0 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.733+0000 7f8fc97fa640 1 -- 172.21.15.89:0/1768920534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4004e00 con 0x7f8fcc1036a0 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.733+0000 7f8fd2381640 1 -- 172.21.15.89:0/1768920534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fcc1a7130 con 0x7f8fcc1036a0 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.735+0000 7f8fc97fa640 1 -- 172.21.15.89:0/1768920534 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8fb4004fa0 con 0x7f8fcc1036a0 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.735+0000 7f8fd475a740 1 Processor -- start 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fd475a740 1 -- start start 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe60880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe6b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe6b680 0x56205fe6da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fd475a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56205fdc62e0 con 0x56205fe60dc0 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fd475a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56205fd6dce0 con 0x56205fe604a0 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fd475a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56205fdc22a0 con 0x56205fe6b680 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fcbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe6b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fcbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe6b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35280/0 (socket says 172.21.15.89:35280) 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fcbfff640 1 -- 172.21.15.89:0/3720449394 learned_addr learned my addr 172.21.15.89:0/3720449394 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fd091e640 1 --2- 172.21.15.89:0/3720449394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe60880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/3720449394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe6b680 0x56205fe6da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fcbfff640 1 -- 172.21.15.89:0/3720449394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe6b680 msgr2=0x56205fe6da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/3720449394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe6b680 0x56205fe6da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fcbfff640 1 -- 172.21.15.89:0/3720449394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 msgr2=0x56205fe60880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:55.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/3720449394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe60880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fcbfff640 1 -- 172.21.15.89:0/3720449394 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56205fd52440 con 0x56205fe60dc0 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fd091e640 1 --2- 172.21.15.89:0/3720449394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe60880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fc97fa640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8fb80a3240 0x7f8fb80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/3720449394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe6b140 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc033000 tx=0x7f8fbc032cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6ea57fc0adcdd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fd091e640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8fb80a3240 0x7f8fb80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.736+0000 7f8fc97fa640 1 -- 172.21.15.89:0/1768920534 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8fb40cf830 con 0x7f8fcc1036a0 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fc27fc640 1 -- 172.21.15.89:0/3720449394 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc03f070 con 0x56205fe60dc0 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fc27fc640 1 -- 172.21.15.89:0/3720449394 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fbc0317b0 con 0x56205fe60dc0 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fc27fc640 1 -- 172.21.15.89:0/3720449394 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc035070 con 0x56205fe60dc0 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fd475a740 1 -- 172.21.15.89:0/3720449394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 msgr2=0x56205fe6b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fd091e640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8fb80a3240 0x7f8fb80a5700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4005fd0 tx=0x7f8fc4009530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fd475a740 1 --2- 172.21.15.89:0/3720449394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe6b140 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc033000 tx=0x7f8fbc032cb0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fd475a740 1 -- 172.21.15.89:0/3720449394 shutdown_connections 2024-09-15T23:26:55.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fd475a740 1 --2- 172.21.15.89:0/3720449394 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe6b680 0x56205fe6da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fd475a740 1 --2- 172.21.15.89:0/3720449394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe6b140 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fd475a740 1 --2- 172.21.15.89:0/3720449394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe60880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fd475a740 1 -- 172.21.15.89:0/3720449394 >> 172.21.15.89:0/3720449394 conn(0x56205fe5ca30 msgr2=0x56205fe5ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:55.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fd475a740 1 -- 172.21.15.89:0/3720449394 shutdown_connections 2024-09-15T23:26:55.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.737+0000 7f8fd475a740 1 -- 172.21.15.89:0/3720449394 wait complete. 2024-09-15T23:26:55.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fd475a740 1 -- start start 2024-09-15T23:26:55.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fd475a740 1 Processor -- start 2024-09-15T23:26:55.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe6afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe68270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe6b680 0x56205fe687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fd475a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56205fe6e250 con 0x56205fe60dc0 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fd475a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56205fd81050 con 0x56205fe604a0 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fd475a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56205fe6e3d0 con 0x56205fe6b680 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fd475a740 10 client.0 _ll_register_callbacks cb 0x56205fdc1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fcb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe68270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fcb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe68270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35286/0 (socket says 172.21.15.89:35286) 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/386444540 learned_addr learned my addr 172.21.15.89:0/386444540 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.738+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe6afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fd091e640 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe6b680 0x56205fe687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/386444540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe6b680 msgr2=0x56205fe687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe6b680 0x56205fe687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/386444540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 msgr2=0x56205fe6afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe6afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/386444540 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56205fb373c0 con 0x56205fe60dc0 2024-09-15T23:26:55.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe68270 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40ccf40 tx=0x7f8fb409b3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2470ebd9d155220c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:55.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:55.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fab7fe640 1 -- 172.21.15.89:0/386444540 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4020070 con 0x56205fe60dc0 2024-09-15T23:26:55.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fab7fe640 1 -- 172.21.15.89:0/386444540 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8fb4009cc0 con 0x56205fe60dc0 2024-09-15T23:26:55.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fd475a740 1 -- 172.21.15.89:0/386444540 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fbc0317b0 con 0x56205fe60dc0 2024-09-15T23:26:55.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fab7fe640 1 -- 172.21.15.89:0/386444540 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8fb409ba60 con 0x56205fe60dc0 2024-09-15T23:26:55.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fd475a740 10 client.16500 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:55.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fd475a740 1 -- 172.21.15.89:0/386444540 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56205fe79c80 con 0x56205fe60dc0 2024-09-15T23:26:55.763 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fab7fe640 1 -- 172.21.15.89:0/386444540 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fbc0317b0 con 0x56205fe60dc0 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.739+0000 7f8fab7fe640 1 -- 172.21.15.89:0/386444540 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8fb40cfce0 con 0x56205fe60dc0 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.740+0000 7f8fd475a740 10 client.16500 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.740+0000 7f8fd475a740 10 client.16500 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.740+0000 7f8fd475a740 20 client.16500 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.740+0000 7f8fd475a740 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56205fe7b090 0x56205fe7d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.740+0000 7f8fd475a740 4 client.16500 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.740+0000 7f8fd475a740 1 -- 172.21.15.89:0/386444540 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56205fdd3850 con 0x56205fe7b090 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.740+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56205fe7b090 0x56205fe7d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.740+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56205fe7b090 0x56205fe7d4a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a86411fa62231bd4 server_cookie=6daf028942e48f5a in_seq=0 out_seq=0 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.740+0000 7f8fab7fe640 10 client.16500 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fab7fe640 1 -- 172.21.15.89:0/386444540 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56205fdd3850 con 0x56205fe7b090 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fab7fe640 10 client.16500 handle_command_reply: tid=0 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 client.16500 shutdown 2024-09-15T23:26:55.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 -- 172.21.15.89:0/386444540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 msgr2=0x56205fe68270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:55.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe68270 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40ccf40 tx=0x7f8fb409b3c0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 -- 172.21.15.89:0/386444540 shutdown_connections 2024-09-15T23:26:55.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56205fe7b090 0x56205fe7d4a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe6b680 0x56205fe687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe68270 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 --2- 172.21.15.89:0/386444540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe6afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fab7fe640 0 client.16500 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:55.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 -- 172.21.15.89:0/386444540 >> 172.21.15.89:0/386444540 conn(0x56205fe5ca30 msgr2=0x56205fe62760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:55.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 -- 172.21.15.89:0/386444540 shutdown_connections 2024-09-15T23:26:55.765 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 1 -- 172.21.15.89:0/386444540 wait complete. 2024-09-15T23:26:55.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.741+0000 7f8fd475a740 20 client.16500 trim_cache size 0 max 16384 2024-09-15T23:26:55.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd475a740 1 Processor -- start 2024-09-15T23:26:55.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd475a740 1 -- start start 2024-09-15T23:26:55.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe5ca30 0x56205fe66b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe82d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe832b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd475a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc1b0fa0 con 0x56205fe60dc0 2024-09-15T23:26:55.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd475a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc1b2ae0 con 0x56205fe604a0 2024-09-15T23:26:55.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd475a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc1b1670 con 0x56205fe5ca30 2024-09-15T23:26:55.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd091e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe832b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd091e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe832b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35298/0 (socket says 172.21.15.89:35298) 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd091e640 1 -- 172.21.15.89:0/717994454 learned_addr learned my addr 172.21.15.89:0/717994454 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe5ca30 0x56205fe66b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe82d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd091e640 1 -- 172.21.15.89:0/717994454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe5ca30 msgr2=0x56205fe66b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd091e640 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe5ca30 0x56205fe66b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd091e640 1 -- 172.21.15.89:0/717994454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 msgr2=0x56205fe82d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd091e640 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe82d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fd091e640 1 -- 172.21.15.89:0/717994454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56205fe67220 con 0x56205fe60dc0 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe82d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.759+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe5ca30 0x56205fe66b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.760+0000 7f8fd091e640 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe832b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40128c0 tx=0x7f8fc4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e1d3bf60a3e7af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.760+0000 7f8fc1ffb640 1 -- 172.21.15.89:0/717994454 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc401eb30 con 0x56205fe60dc0 2024-09-15T23:26:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.760+0000 7f8fc1ffb640 1 -- 172.21.15.89:0/717994454 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fc4012e40 con 0x56205fe60dc0 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.760+0000 7f8fc1ffb640 1 -- 172.21.15.89:0/717994454 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4027630 con 0x56205fe60dc0 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 -- 172.21.15.89:0/717994454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 msgr2=0x56205fe832b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe832b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40128c0 tx=0x7f8fc4014000 comp rx=0 tx=0).stop 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 -- 172.21.15.89:0/717994454 shutdown_connections 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe60dc0 0x56205fe832b0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe604a0 0x56205fe82d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 --2- 172.21.15.89:0/717994454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe5ca30 0x56205fe66b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 -- 172.21.15.89:0/717994454 >> 172.21.15.89:0/717994454 conn(0x56205fe68910 msgr2=0x56205fe6a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 -- 172.21.15.89:0/717994454 shutdown_connections 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 -- 172.21.15.89:0/717994454 wait complete. 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 -- start start 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 Processor -- start 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe5ca30 0x56205fe71710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe604a0 0x56205fe71c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe60dc0 0x56205fe72190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.761+0000 7f8fd475a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc1b2410 con 0x56205fe604a0 2024-09-15T23:26:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fd475a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc1b1d40 con 0x56205fe5ca30 2024-09-15T23:26:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fd475a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8fcc1aef80 con 0x56205fe60dc0 2024-09-15T23:26:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fd475a740 10 client.0 _ll_register_callbacks cb 0x7f8fb400de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fcb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe604a0 0x56205fe71c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fcb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe604a0 0x56205fe71c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35314/0 (socket says 172.21.15.89:35314) 2024-09-15T23:26:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/414171745 learned_addr learned my addr 172.21.15.89:0/414171745 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fd091e640 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe60dc0 0x56205fe72190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/414171745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe60dc0 msgr2=0x56205fe72190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe60dc0 0x56205fe72190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe5ca30 0x56205fe71710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/414171745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe5ca30 msgr2=0x56205fe71710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe5ca30 0x56205fe71710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fcb7fe640 1 -- 172.21.15.89:0/414171745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56205fee4ff0 con 0x56205fe604a0 2024-09-15T23:26:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fcb7fe640 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe604a0 0x56205fe71c50 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4099d10 tx=0x7f8fb4099f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a86449b5fedd17f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:26:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fab7fe640 1 -- 172.21.15.89:0/414171745 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4020070 con 0x56205fe604a0 2024-09-15T23:26:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fab7fe640 1 -- 172.21.15.89:0/414171745 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8fb40977b0 con 0x56205fe604a0 2024-09-15T23:26:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fd475a740 1 -- 172.21.15.89:0/414171745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56205fdcdd60 con 0x56205fe604a0 2024-09-15T23:26:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fab7fe640 1 -- 172.21.15.89:0/414171745 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8fb4097d00 con 0x56205fe604a0 2024-09-15T23:26:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fd475a740 10 client.16506 fetch_fsmap learned FSMap version 29 2024-09-15T23:26:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.762+0000 7f8fd475a740 1 -- 172.21.15.89:0/414171745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56205fee6250 con 0x56205fe604a0 2024-09-15T23:26:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.763+0000 7f8fab7fe640 1 -- 172.21.15.89:0/414171745 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56205fdcdd60 con 0x56205fe604a0 2024-09-15T23:26:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.763+0000 7f8fab7fe640 1 -- 172.21.15.89:0/414171745 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8fb40ba240 con 0x56205fe604a0 2024-09-15T23:26:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.763+0000 7f8fd475a740 10 client.16506 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:26:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.763+0000 7f8fd475a740 10 client.16506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.763+0000 7f8fd475a740 20 client.16506 populate_metadata read hostname 'smithi089' 2024-09-15T23:26:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.763+0000 7f8fd475a740 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56205fe6b680 0x56205fee87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.763+0000 7f8fd475a740 4 client.16506 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:26:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.763+0000 7f8fd475a740 1 -- 172.21.15.89:0/414171745 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56205fd4a210 con 0x56205fe6b680 2024-09-15T23:26:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.763+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56205fe6b680 0x56205fee87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fcbfff640 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56205fe6b680 0x56205fee87f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=311b22049440d878 server_cookie=43a808790ae9a9f9 in_seq=0 out_seq=0 2024-09-15T23:26:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fab7fe640 10 client.16506 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fab7fe640 1 -- 172.21.15.89:0/414171745 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56205fd4a210 con 0x56205fe6b680 2024-09-15T23:26:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fab7fe640 10 client.16506 handle_command_reply: tid=0 2024-09-15T23:26:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 client.16506 shutdown 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 -- 172.21.15.89:0/414171745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe604a0 msgr2=0x56205fe71c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe604a0 0x56205fe71c50 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4099d10 tx=0x7f8fb4099f70 comp rx=0 tx=0).stop 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 -- 172.21.15.89:0/414171745 shutdown_connections 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56205fe6b680 0x56205fee87f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56205fe60dc0 0x56205fe72190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56205fe604a0 0x56205fe71c50 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 --2- 172.21.15.89:0/414171745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56205fe5ca30 0x56205fe71710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 -- 172.21.15.89:0/414171745 >> 172.21.15.89:0/414171745 conn(0x56205fe68910 msgr2=0x56205fe5bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fab7fe640 0 client.16506 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 -- 172.21.15.89:0/414171745 shutdown_connections 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 1 -- 172.21.15.89:0/414171745 wait complete. 2024-09-15T23:26:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.764+0000 7f8fd475a740 20 client.16506 trim_cache size 0 max 16384 2024-09-15T23:26:55.779 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:26:55.779 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:26:55.779 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:26:55.779 INFO:teuthology.orchestra.run.smithi089.stdout: "f01ebea3-45df-4474-a870-64dc01292b4a": { 2024-09-15T23:26:55.779 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:26:55.779 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "f01ebea3-45df-4474-a870-64dc01292b4a", 2024-09-15T23:26:55.779 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:26:55.779 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:26:55.779 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:26:55.780 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:26:55.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 -- 172.21.15.89:0/1768920534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8fb80a3240 msgr2=0x7f8fb80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:55.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8fb80a3240 0x7f8fb80a5700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4005fd0 tx=0x7f8fc4009530 comp rx=0 tx=0).stop 2024-09-15T23:26:55.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 -- 172.21.15.89:0/1768920534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fcc1036a0 msgr2=0x7f8fcc100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:55.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fcc1036a0 0x7f8fcc100f20 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400d770 tx=0x7f8fb400dc30 comp rx=0 tx=0).stop 2024-09-15T23:26:55.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 -- 172.21.15.89:0/1768920534 shutdown_connections 2024-09-15T23:26:55.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8fb80a3240 0x7f8fb80a5700 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8fcc10c160 0x7f8fcc101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8fcc1036a0 0x7f8fcc100f20 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 --2- 172.21.15.89:0/1768920534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fcc102d80 0x7f8fcc1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:55.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 -- 172.21.15.89:0/1768920534 >> 172.21.15.89:0/1768920534 conn(0x7f8fcc0fc780 msgr2=0x7f8fcc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:55.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 -- 172.21.15.89:0/1768920534 shutdown_connections 2024-09-15T23:26:55.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:55.765+0000 7f8fd2381640 1 -- 172.21.15.89:0/1768920534 wait complete. 2024-09-15T23:26:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:56 smithi120 ceph-mon[35250]: pgmap v437: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-09-15T23:26:56.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:56 smithi174 ceph-mon[35586]: pgmap v437: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-09-15T23:26:56.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:56 smithi089 ceph-mon[30933]: pgmap v437: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-09-15T23:26:56.646 INFO:tasks.cephfs.filesystem:scrub status for tag:f01ebea3-45df-4474-a870-64dc01292b4a - {'path': '/', 'tag': 'f01ebea3-45df-4474-a870-64dc01292b4a', 'options': 'recursive,force'} 2024-09-15T23:26:57.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:57 smithi120 ceph-mon[35250]: pgmap v438: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2024-09-15T23:26:57.489 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:57 smithi174 ceph-mon[35586]: pgmap v438: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2024-09-15T23:26:57.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:57 smithi089 ceph-mon[30933]: pgmap v438: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.8 KiB/s wr, 17 op/s 2024-09-15T23:26:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:26:58.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:26:58.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:26:58.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:26:58.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:58 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:26:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:26:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:26:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:26:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:26:58.487 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:58 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:26:58.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:26:58.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:26:58.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:26:58.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:26:58.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:58 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:26:58.678 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:26:58.930 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:59.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:26:59 smithi089 ceph-mon[30933]: pgmap v439: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.1 KiB/s wr, 13 op/s 2024-09-15T23:26:59.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.432+0000 7fd401e59640 1 -- 172.21.15.89:0/663599337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3fc106f10 msgr2=0x7fd3fc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.432+0000 7fd401e59640 1 --2- 172.21.15.89:0/663599337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3fc106f10 0x7fd3fc10ea90 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009ca0 tx=0x7fd3f402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.432+0000 7fd401e59640 1 -- 172.21.15.89:0/663599337 shutdown_connections 2024-09-15T23:26:59.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.432+0000 7fd401e59640 1 --2- 172.21.15.89:0/663599337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3fc106f10 0x7fd3fc10ea90 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.432+0000 7fd401e59640 1 --2- 172.21.15.89:0/663599337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3fc102790 0x7fd3fc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.432+0000 7fd401e59640 1 --2- 172.21.15.89:0/663599337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3fc101de0 0x7fd3fc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.432+0000 7fd401e59640 1 -- 172.21.15.89:0/663599337 >> 172.21.15.89:0/663599337 conn(0x7fd3fc0fd8c0 msgr2=0x7fd3fc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:59.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.433+0000 7fd401e59640 1 -- 172.21.15.89:0/663599337 shutdown_connections 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.433+0000 7fd401e59640 1 -- 172.21.15.89:0/663599337 wait complete. 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.433+0000 7fd401e59640 1 Processor -- start 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.433+0000 7fd401e59640 1 -- start start 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.433+0000 7fd401e59640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3fc101de0 0x7fd3fc198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.433+0000 7fd401e59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3fc102790 0x7fd3fc1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.433+0000 7fd401e59640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3fc106f10 0x7fd3fc19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.433+0000 7fd401e59640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc111f90 con 0x7fd3fc106f10 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.433+0000 7fd401e59640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc111e10 con 0x7fd3fc101de0 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.433+0000 7fd401e59640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc112110 con 0x7fd3fc102790 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3fbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3fc106f10 0x7fd3fc19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3fbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3fc106f10 0x7fd3fc19cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35326/0 (socket says 172.21.15.89:35326) 2024-09-15T23:26:59.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3fbfff640 1 -- 172.21.15.89:0/2333649259 learned_addr learned my addr 172.21.15.89:0/2333649259 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3faffd640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3fc102790 0x7fd3fc1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3fb7fe640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3fc101de0 0x7fd3fc198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3fbfff640 1 -- 172.21.15.89:0/2333649259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3fc102790 msgr2=0x7fd3fc1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3fbfff640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3fc102790 0x7fd3fc1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3fbfff640 1 -- 172.21.15.89:0/2333649259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3fc101de0 msgr2=0x7fd3fc198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3fbfff640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3fc101de0 0x7fd3fc198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3fbfff640 1 -- 172.21.15.89:0/2333649259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3fc19d420 con 0x7fd3fc106f10 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3fbfff640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3fc106f10 0x7fd3fc19cd40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4031a20 tx=0x7fd3f4031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3f8ff9640 1 -- 172.21.15.89:0/2333649259 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4003c30 con 0x7fd3fc106f10 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3f8ff9640 1 -- 172.21.15.89:0/2333649259 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f4003dd0 con 0x7fd3fc106f10 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd401e59640 1 -- 172.21.15.89:0/2333649259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3fc19d6b0 con 0x7fd3fc106f10 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd3f8ff9640 1 -- 172.21.15.89:0/2333649259 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f402fcf0 con 0x7fd3fc106f10 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.434+0000 7fd401e59640 1 -- 172.21.15.89:0/2333649259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3fc1a4ff0 con 0x7fd3fc106f10 2024-09-15T23:26:59.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.435+0000 7fd401e59640 1 -- 172.21.15.89:0/2333649259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3fc1021c0 con 0x7fd3fc106f10 2024-09-15T23:26:59.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.436+0000 7fd3f8ff9640 1 -- 172.21.15.89:0/2333649259 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd3f4004b40 con 0x7fd3fc106f10 2024-09-15T23:26:59.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.437+0000 7fd3f8ff9640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd3e40a3240 0x7fd3e40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.437+0000 7fd3fb7fe640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd3e40a3240 0x7fd3e40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.437+0000 7fd3f8ff9640 1 -- 172.21.15.89:0/2333649259 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd3f40f1280 con 0x7fd3fc106f10 2024-09-15T23:26:59.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.437+0000 7fd3fb7fe640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd3e40a3240 0x7fd3e40a5700 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009890 tx=0x7fd3ec008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:59.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.440+0000 7fd3f8ff9640 1 -- 172.21.15.89:0/2333649259 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd3f40bdc40 con 0x7fd3fc106f10 2024-09-15T23:26:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:26:59 smithi120 ceph-mon[35250]: pgmap v439: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.1 KiB/s wr, 13 op/s 2024-09-15T23:26:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:26:59 smithi174 ceph-mon[35586]: pgmap v439: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.1 KiB/s wr, 13 op/s 2024-09-15T23:26:59.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.639+0000 7fd401e59640 1 -- 172.21.15.89:0/2333649259 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3fc1997d0 con 0x7fd3fc106f10 2024-09-15T23:26:59.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.640+0000 7fd3f8ff9640 1 -- 172.21.15.89:0/2333649259 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fd3fc1997d0 con 0x7fd3fc106f10 2024-09-15T23:26:59.644 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:26:59.644 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:26:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:26:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 -- 172.21.15.89:0/2333649259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd3e40a3240 msgr2=0x7fd3e40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd3e40a3240 0x7fd3e40a5700 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009890 tx=0x7fd3ec008040 comp rx=0 tx=0).stop 2024-09-15T23:26:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 -- 172.21.15.89:0/2333649259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3fc106f10 msgr2=0x7fd3fc19cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3fc106f10 0x7fd3fc19cd40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4031a20 tx=0x7fd3f4031a50 comp rx=0 tx=0).stop 2024-09-15T23:26:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 -- 172.21.15.89:0/2333649259 shutdown_connections 2024-09-15T23:26:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd3e40a3240 0x7fd3e40a5700 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd3fc106f10 0x7fd3fc19cd40 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3fc102790 0x7fd3fc1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 --2- 172.21.15.89:0/2333649259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd3fc101de0 0x7fd3fc198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 -- 172.21.15.89:0/2333649259 >> 172.21.15.89:0/2333649259 conn(0x7fd3fc0fd8c0 msgr2=0x7fd3fc0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:59.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 -- 172.21.15.89:0/2333649259 shutdown_connections 2024-09-15T23:26:59.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:26:59.642+0000 7fd401e59640 1 -- 172.21.15.89:0/2333649259 wait complete. 2024-09-15T23:27:00.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:00 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2333649259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:00.382 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 67c52a99 --release --await' 2024-09-15T23:27:00.383 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 67c52a99 --release --await 2024-09-15T23:27:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:00 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2333649259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:00 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2333649259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:00.629 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.163+0000 7f02825ff640 1 -- 172.21.15.89:0/546065642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c102d30 msgr2=0x7f027c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.163+0000 7f02825ff640 1 --2- 172.21.15.89:0/546065642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c102d30 0x7f027c103110 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f026c009930 tx=0x7f026c02a420 comp rx=0 tx=0).stop 2024-09-15T23:27:01.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.163+0000 7f02825ff640 1 -- 172.21.15.89:0/546065642 shutdown_connections 2024-09-15T23:27:01.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.163+0000 7f02825ff640 1 --2- 172.21.15.89:0/546065642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f027c1082a0 0x7f027c10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.163+0000 7f02825ff640 1 --2- 172.21.15.89:0/546065642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f027c1036e0 0x7f027c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.163+0000 7f02825ff640 1 --2- 172.21.15.89:0/546065642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c102d30 0x7f027c103110 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.163+0000 7f02825ff640 1 -- 172.21.15.89:0/546065642 >> 172.21.15.89:0/546065642 conn(0x7f027c0fc7d0 msgr2=0x7f027c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.163+0000 7f02825ff640 1 -- 172.21.15.89:0/546065642 shutdown_connections 2024-09-15T23:27:01.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.163+0000 7f02825ff640 1 -- 172.21.15.89:0/546065642 wait complete. 2024-09-15T23:27:01.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f02825ff640 1 Processor -- start 2024-09-15T23:27:01.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f02825ff640 1 -- start start 2024-09-15T23:27:01.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f02825ff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f027c102d30 0x7f027c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f02825ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f027c1036e0 0x7f027c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f02825ff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c1082a0 0x7f027c19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f02825ff640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f027c1120a0 con 0x7f027c1082a0 2024-09-15T23:27:01.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f02825ff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f027c111f20 con 0x7f027c1036e0 2024-09-15T23:27:01.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f02825ff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f027c112220 con 0x7f027c102d30 2024-09-15T23:27:01.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0280b9c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c1082a0 0x7f027c19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0280b9c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c1082a0 0x7f027c19ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35332/0 (socket says 172.21.15.89:35332) 2024-09-15T23:27:01.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0280b9c640 1 -- 172.21.15.89:0/3928640383 learned_addr learned my addr 172.21.15.89:0/3928640383 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:01.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0273fff640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f027c1036e0 0x7f027c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f027bfff640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f027c102d30 0x7f027c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0280b9c640 1 -- 172.21.15.89:0/3928640383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f027c102d30 msgr2=0x7f027c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0280b9c640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f027c102d30 0x7f027c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0280b9c640 1 -- 172.21.15.89:0/3928640383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f027c1036e0 msgr2=0x7f027c198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0280b9c640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f027c1036e0 0x7f027c198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0280b9c640 1 -- 172.21.15.89:0/3928640383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027c19d590 con 0x7f027c1082a0 2024-09-15T23:27:01.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0273fff640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f027c1036e0 0x7f027c198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:01.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f027bfff640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f027c102d30 0x7f027c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:01.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.164+0000 7f0280b9c640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c1082a0 0x7f027c19ceb0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f027400cad0 tx=0x7f027400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.165+0000 7f0279ffb640 1 -- 172.21.15.89:0/3928640383 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027400ae20 con 0x7f027c1082a0 2024-09-15T23:27:01.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.165+0000 7f0279ffb640 1 -- 172.21.15.89:0/3928640383 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0274005020 con 0x7f027c1082a0 2024-09-15T23:27:01.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.165+0000 7f02825ff640 1 -- 172.21.15.89:0/3928640383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f027c19d880 con 0x7f027c1082a0 2024-09-15T23:27:01.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.165+0000 7f0279ffb640 1 -- 172.21.15.89:0/3928640383 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027400f6f0 con 0x7f027c1082a0 2024-09-15T23:27:01.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.165+0000 7f02825ff640 1 -- 172.21.15.89:0/3928640383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f027c1a5160 con 0x7f027c1082a0 2024-09-15T23:27:01.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.167+0000 7f0279ffb640 1 -- 172.21.15.89:0/3928640383 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f02740051d0 con 0x7f027c1082a0 2024-09-15T23:27:01.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.167+0000 7f02849d8740 1 Processor -- start 2024-09-15T23:27:01.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f02849d8740 1 -- start start 2024-09-15T23:27:01.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f02849d8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x5578518fc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f02849d8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x55785190b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f02849d8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 0x55785190da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f02849d8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557851861620 con 0x5578518fbf40 2024-09-15T23:27:01.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f02849d8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557851808e50 con 0x55785190b650 2024-09-15T23:27:01.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f02849d8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55785185d5e0 con 0x5578518fc860 2024-09-15T23:27:01.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0279ffb640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f02680a32c0 0x7f02680a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0273fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x5578518fc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0273fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x5578518fc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35346/0 (socket says 172.21.15.89:35346) 2024-09-15T23:27:01.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0273fff640 1 -- 172.21.15.89:0/2610925897 learned_addr learned my addr 172.21.15.89:0/2610925897 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:01.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0279ffb640 1 -- 172.21.15.89:0/3928640383 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f02740cfd30 con 0x7f027c1082a0 2024-09-15T23:27:01.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0280b9c640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f02680a32c0 0x7f02680a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f027bfff640 1 --2- 172.21.15.89:0/2610925897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x55785190b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0273fff640 1 --2- 172.21.15.89:0/2610925897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 0x55785190da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0273fff640 1 -- 172.21.15.89:0/2610925897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 msgr2=0x55785190b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0273fff640 1 --2- 172.21.15.89:0/2610925897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x55785190b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0273fff640 1 -- 172.21.15.89:0/2610925897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 msgr2=0x55785190da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0273fff640 1 --2- 172.21.15.89:0/2610925897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 0x55785190da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0273fff640 1 -- 172.21.15.89:0/2610925897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578517f9d40 con 0x5578518fbf40 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f027bfff640 1 --2- 172.21.15.89:0/2610925897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x55785190b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.168+0000 7f0273fff640 1 --2- 172.21.15.89:0/2610925897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x5578518fc320 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f0264005d70 tx=0x7f026400c600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e611861d79fc50 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f0280b9c640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f02680a32c0 0x7f02680a5780 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f02740a46a0 tx=0x7f0274005b40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02727fc640 1 -- 172.21.15.89:0/2610925897 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264004220 con 0x5578518fbf40 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02727fc640 1 -- 172.21.15.89:0/2610925897 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0264016030 con 0x5578518fbf40 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02727fc640 1 -- 172.21.15.89:0/2610925897 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02640037f0 con 0x5578518fbf40 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02849d8740 1 -- 172.21.15.89:0/2610925897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 msgr2=0x5578518fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02849d8740 1 --2- 172.21.15.89:0/2610925897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x5578518fc320 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f0264005d70 tx=0x7f026400c600 comp rx=0 tx=0).stop 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02849d8740 1 -- 172.21.15.89:0/2610925897 shutdown_connections 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02849d8740 1 --2- 172.21.15.89:0/2610925897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 0x55785190da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02849d8740 1 --2- 172.21.15.89:0/2610925897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x55785190b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02849d8740 1 --2- 172.21.15.89:0/2610925897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x5578518fc320 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02849d8740 1 -- 172.21.15.89:0/2610925897 >> 172.21.15.89:0/2610925897 conn(0x5578518f85f0 msgr2=0x5578518f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02849d8740 1 -- 172.21.15.89:0/2610925897 shutdown_connections 2024-09-15T23:27:01.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.169+0000 7f02849d8740 1 -- 172.21.15.89:0/2610925897 wait complete. 2024-09-15T23:27:01.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f02849d8740 1 -- start start 2024-09-15T23:27:01.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f02849d8740 1 Processor -- start 2024-09-15T23:27:01.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f02849d8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fbf40 0x557851908550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f02849d8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578518fc860 0x557851908a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f02849d8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55785190b650 0x557851908fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f02849d8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55785190e220 con 0x55785190b650 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f02849d8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5578517e9f10 con 0x5578518fc860 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f02849d8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55785190e3a0 con 0x5578518fbf40 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f02849d8740 10 client.0 _ll_register_callbacks cb 0x55785180ddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f0273fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55785190b650 0x557851908fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f0273fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55785190b650 0x557851908fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35354/0 (socket says 172.21.15.89:35354) 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.170+0000 7f0273fff640 1 -- 172.21.15.89:0/799165145 learned_addr learned my addr 172.21.15.89:0/799165145 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f027bfff640 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578518fc860 0x557851908a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f0273fff640 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fbf40 0x557851908550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f0273fff640 1 -- 172.21.15.89:0/799165145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fbf40 msgr2=0x557851908550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f0273fff640 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fbf40 0x557851908550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f0273fff640 1 -- 172.21.15.89:0/799165145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578518fc860 msgr2=0x557851908a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f0273fff640 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578518fc860 0x557851908a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f0273fff640 1 -- 172.21.15.89:0/799165145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557851570580 con 0x55785190b650 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f0273fff640 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55785190b650 0x557851908fd0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f0264003730 tx=0x7f0264012ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f076829ca71b9f87 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f025b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f025b7fe640 1 -- 172.21.15.89:0/799165145 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264024040 con 0x55785190b650 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f025b7fe640 1 -- 172.21.15.89:0/799165145 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0264012860 con 0x55785190b650 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f02849d8740 1 -- 172.21.15.89:0/799165145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0264016030 con 0x55785190b650 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f025b7fe640 1 -- 172.21.15.89:0/799165145 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0264033050 con 0x55785190b650 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f02849d8740 10 client.16524 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.171+0000 7f02849d8740 1 -- 172.21.15.89:0/799165145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55785163a960 con 0x55785190b650 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f025b7fe640 1 -- 172.21.15.89:0/799165145 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0264016030 con 0x55785190b650 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f025b7fe640 1 -- 172.21.15.89:0/799165145 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f026401b470 con 0x55785190b650 2024-09-15T23:27:01.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f02849d8740 10 client.16524 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f02849d8740 10 client.16524 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f02849d8740 20 client.16524 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f02849d8740 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55785191b150 0x55785191d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f02849d8740 4 client.16524 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f02849d8740 1 -- 172.21.15.89:0/799165145 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55785186eb90 con 0x55785191b150 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f027bfff640 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55785191b150 0x55785191d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f025b7fe640 10 client.16524 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.172+0000 7f027bfff640 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55785191b150 0x55785191d540 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=349e253c22c6ba9 server_cookie=622df03563343b45 in_seq=0 out_seq=0 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f025b7fe640 1 -- 172.21.15.89:0/799165145 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55785186eb90 con 0x55785191b150 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f025b7fe640 10 client.16524 handle_command_reply: tid=0 2024-09-15T23:27:01.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f02849d8740 1 client.16524 shutdown 2024-09-15T23:27:01.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f02849d8740 1 -- 172.21.15.89:0/799165145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55785190b650 msgr2=0x557851908fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f02849d8740 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55785190b650 0x557851908fd0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f0264003730 tx=0x7f0264012ee0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f02849d8740 1 -- 172.21.15.89:0/799165145 shutdown_connections 2024-09-15T23:27:01.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f02849d8740 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55785191b150 0x55785191d540 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f02849d8740 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55785190b650 0x557851908fd0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f02849d8740 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578518fc860 0x557851908a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f025b7fe640 0 client.16524 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:01.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f02849d8740 1 --2- 172.21.15.89:0/799165145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fbf40 0x557851908550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f02849d8740 1 -- 172.21.15.89:0/799165145 >> 172.21.15.89:0/799165145 conn(0x5578518f85f0 msgr2=0x5578518fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.173+0000 7f02849d8740 1 -- 172.21.15.89:0/799165145 shutdown_connections 2024-09-15T23:27:01.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.174+0000 7f02849d8740 1 -- 172.21.15.89:0/799165145 wait complete. 2024-09-15T23:27:01.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.174+0000 7f02849d8740 20 client.16524 trim_cache size 0 max 16384 2024-09-15T23:27:01.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f02849d8740 1 Processor -- start 2024-09-15T23:27:01.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f02849d8740 1 -- start start 2024-09-15T23:27:01.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f02849d8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851907ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f02849d8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x557851908c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f02849d8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 0x557851922b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f02849d8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f027c1acc30 con 0x5578518fbf40 2024-09-15T23:27:01.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f02849d8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f027c1ae770 con 0x55785190b650 2024-09-15T23:27:01.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f02849d8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f027c1ad300 con 0x5578518fc860 2024-09-15T23:27:01.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f0273fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851907ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f0273fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851907ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35362/0 (socket says 172.21.15.89:35362) 2024-09-15T23:27:01.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f0273fff640 1 -- 172.21.15.89:0/2928211656 learned_addr learned my addr 172.21.15.89:0/2928211656 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f027bfff640 1 --2- 172.21.15.89:0/2928211656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x557851908c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f0273fff640 1 -- 172.21.15.89:0/2928211656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 msgr2=0x557851908c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f0273fff640 1 --2- 172.21.15.89:0/2928211656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x557851908c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f0273fff640 1 -- 172.21.15.89:0/2928211656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 msgr2=0x557851922b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.190+0000 7f0273fff640 1 --2- 172.21.15.89:0/2928211656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 0x557851922b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f0273fff640 1 -- 172.21.15.89:0/2928211656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578518f9550 con 0x5578518fbf40 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f027bfff640 1 --2- 172.21.15.89:0/2928211656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x557851908c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f0273fff640 1 --2- 172.21.15.89:0/2928211656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851907ab0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f0264031040 tx=0x7f0264010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8446cfaef8c0e8d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f0271ffb640 1 -- 172.21.15.89:0/2928211656 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026403c070 con 0x5578518fbf40 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f0271ffb640 1 -- 172.21.15.89:0/2928211656 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0264016040 con 0x5578518fbf40 2024-09-15T23:27:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f0271ffb640 1 -- 172.21.15.89:0/2928211656 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026401b470 con 0x5578518fbf40 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f02849d8740 1 -- 172.21.15.89:0/2928211656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 msgr2=0x557851907ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f02849d8740 1 --2- 172.21.15.89:0/2928211656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851907ab0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f0264031040 tx=0x7f0264010ec0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f02849d8740 1 -- 172.21.15.89:0/2928211656 shutdown_connections 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f02849d8740 1 --2- 172.21.15.89:0/2928211656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 0x557851922b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f02849d8740 1 --2- 172.21.15.89:0/2928211656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x557851908c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f02849d8740 1 --2- 172.21.15.89:0/2928211656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851907ab0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.191+0000 7f02849d8740 1 -- 172.21.15.89:0/2928211656 >> 172.21.15.89:0/2928211656 conn(0x5578518f85f0 msgr2=0x557851908510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.192+0000 7f02849d8740 1 -- 172.21.15.89:0/2928211656 shutdown_connections 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.192+0000 7f02849d8740 1 -- 172.21.15.89:0/2928211656 wait complete. 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.192+0000 7f02849d8740 1 -- start start 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.192+0000 7f02849d8740 1 Processor -- start 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.192+0000 7f02849d8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851905650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.192+0000 7f02849d8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x557851905b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.192+0000 7f02849d8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 0x5578519060d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.192+0000 7f02849d8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f027c1ae0a0 con 0x5578518fbf40 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.192+0000 7f02849d8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f027c1ad9d0 con 0x55785190b650 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.192+0000 7f02849d8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f027c1aac10 con 0x5578518fc860 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f02849d8740 10 client.0 _ll_register_callbacks cb 0x7f02640035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f0273fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851905650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f0273fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851905650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35370/0 (socket says 172.21.15.89:35370) 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f0273fff640 1 -- 172.21.15.89:0/3293648561 learned_addr learned my addr 172.21.15.89:0/3293648561 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f027bfff640 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x557851905b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f0273fff640 1 -- 172.21.15.89:0/3293648561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 msgr2=0x557851905b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f0273fff640 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x557851905b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f0273fff640 1 -- 172.21.15.89:0/3293648561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 msgr2=0x5578519060d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f0273fff640 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 0x5578519060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f0273fff640 1 -- 172.21.15.89:0/3293648561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557851984be0 con 0x5578518fbf40 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f027bfff640 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x557851905b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f0273fff640 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851905650 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f026401bb10 tx=0x7f026401bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7081e23cfc88f77 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f025b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f025b7fe640 1 -- 172.21.15.89:0/3293648561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026403c070 con 0x5578518fbf40 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f025b7fe640 1 -- 172.21.15.89:0/3293648561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0264005040 con 0x5578518fbf40 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f02849d8740 1 -- 172.21.15.89:0/3293648561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0264016040 con 0x5578518fbf40 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f025b7fe640 1 -- 172.21.15.89:0/3293648561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0264049050 con 0x5578518fbf40 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f02849d8740 10 client.16530 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.193+0000 7f02849d8740 1 -- 172.21.15.89:0/3293648561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557851985e70 con 0x5578518fbf40 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f025b7fe640 1 -- 172.21.15.89:0/3293648561 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0264016040 con 0x5578518fbf40 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f025b7fe640 1 -- 172.21.15.89:0/3293648561 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0264005240 con 0x5578518fbf40 2024-09-15T23:27:01.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f02849d8740 10 client.16530 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f02849d8740 10 client.16530 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f02849d8740 20 client.16530 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f02849d8740 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5578519864d0 0x5578519888c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f02849d8740 4 client.16530 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "67c52a99", "release": true, "await": true}] 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f02849d8740 1 -- 172.21.15.89:0/3293648561 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "67c52a99", "release": true, "await": true}) -- 0x5578518690a0 con 0x5578519864d0 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f027bfff640 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5578519864d0 0x5578519888c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f025b7fe640 10 client.16530 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.194+0000 7f027bfff640 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5578519864d0 0x5578519888c0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a396b03b65cba2e server_cookie=e831bf70135b66af in_seq=0 out_seq=0 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f025b7fe640 1 -- 172.21.15.89:0/3293648561 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5578518690a0 con 0x5578519864d0 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f025b7fe640 10 client.16530 handle_command_reply: tid=0 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 client.16530 shutdown 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 -- 172.21.15.89:0/3293648561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 msgr2=0x557851905650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851905650 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f026401bb10 tx=0x7f026401bd10 comp rx=0 tx=0).stop 2024-09-15T23:27:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 -- 172.21.15.89:0/3293648561 shutdown_connections 2024-09-15T23:27:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5578519864d0 0x5578519888c0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55785190b650 0x5578519060d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5578518fc860 0x557851905b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f025b7fe640 0 client.16530 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 --2- 172.21.15.89:0/3293648561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5578518fbf40 0x557851905650 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 -- 172.21.15.89:0/3293648561 >> 172.21.15.89:0/3293648561 conn(0x5578518f85f0 msgr2=0x557851908d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 -- 172.21.15.89:0/3293648561 shutdown_connections 2024-09-15T23:27:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.197+0000 7f02849d8740 1 -- 172.21.15.89:0/3293648561 wait complete. 2024-09-15T23:27:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02849d8740 20 client.16530 trim_cache size 0 max 16384 2024-09-15T23:27:01.207 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:27:01.207 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:27:01.207 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 8, 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "67c52a99": { 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 8, 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 139.6, 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:27:01.208 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:27:01.209 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:27:01.209 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:27:01.209 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:27:01.209 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:27:01.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:01.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:01.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:01.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:01.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:01.209 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:27:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 -- 172.21.15.89:0/3928640383 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f02680a32c0 msgr2=0x7f02680a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f02680a32c0 0x7f02680a5780 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f02740a46a0 tx=0x7f0274005b40 comp rx=0 tx=0).stop 2024-09-15T23:27:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 -- 172.21.15.89:0/3928640383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c1082a0 msgr2=0x7f027c19ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c1082a0 0x7f027c19ceb0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f027400cad0 tx=0x7f027400cf90 comp rx=0 tx=0).stop 2024-09-15T23:27:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 -- 172.21.15.89:0/3928640383 shutdown_connections 2024-09-15T23:27:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f02680a32c0 0x7f02680a5780 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f027c1082a0 0x7f027c19ceb0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f027c1036e0 0x7f027c198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 --2- 172.21.15.89:0/3928640383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f027c102d30 0x7f027c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 -- 172.21.15.89:0/3928640383 >> 172.21.15.89:0/3928640383 conn(0x7f027c0fc7d0 msgr2=0x7f027c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 -- 172.21.15.89:0/3928640383 shutdown_connections 2024-09-15T23:27:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:01.198+0000 7f02825ff640 1 -- 172.21.15.89:0/3928640383 wait complete. 2024-09-15T23:27:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:01 smithi120 ceph-mon[35250]: pgmap v440: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.1 KiB/s wr, 14 op/s 2024-09-15T23:27:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:01 smithi174 ceph-mon[35586]: pgmap v440: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.1 KiB/s wr, 14 op/s 2024-09-15T23:27:01.522 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:01 smithi089 ceph-mon[30933]: pgmap v440: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.1 KiB/s wr, 14 op/s 2024-09-15T23:27:01.919 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '67c52a99', total seconds elapsed: 22.7 2024-09-15T23:27:01.919 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.55 2024-09-15T23:27:01.920 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:27:02.169 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:02.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.689+0000 7f5a01ecc640 1 -- 172.21.15.89:0/2401266671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59fc102790 msgr2=0x7f59fc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.689+0000 7f5a01ecc640 1 --2- 172.21.15.89:0/2401266671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59fc102790 0x7f59fc1067e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f59f0009900 tx=0x7f59f002a410 comp rx=0 tx=0).stop 2024-09-15T23:27:02.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.689+0000 7f5a01ecc640 1 -- 172.21.15.89:0/2401266671 shutdown_connections 2024-09-15T23:27:02.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.689+0000 7f5a01ecc640 1 --2- 172.21.15.89:0/2401266671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc106f10 0x7f59fc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.689+0000 7f5a01ecc640 1 --2- 172.21.15.89:0/2401266671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59fc102790 0x7f59fc1067e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.689+0000 7f5a01ecc640 1 --2- 172.21.15.89:0/2401266671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59fc101de0 0x7f59fc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.689+0000 7f5a01ecc640 1 -- 172.21.15.89:0/2401266671 >> 172.21.15.89:0/2401266671 conn(0x7f59fc0fd8c0 msgr2=0x7f59fc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.689+0000 7f5a01ecc640 1 -- 172.21.15.89:0/2401266671 shutdown_connections 2024-09-15T23:27:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.689+0000 7f5a01ecc640 1 -- 172.21.15.89:0/2401266671 wait complete. 2024-09-15T23:27:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f5a01ecc640 1 Processor -- start 2024-09-15T23:27:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f5a01ecc640 1 -- start start 2024-09-15T23:27:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f5a01ecc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59fc101de0 0x7f59fc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f5a01ecc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc102790 0x7f59fc198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f5a01ecc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59fc106f10 0x7f59fc19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f5a01ecc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc111f70 con 0x7f59fc106f10 2024-09-15T23:27:02.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f5a01ecc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc111df0 con 0x7f59fc101de0 2024-09-15T23:27:02.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f5a01ecc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc1120f0 con 0x7f59fc102790 2024-09-15T23:27:02.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f59fbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59fc106f10 0x7f59fc19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f59fbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59fc106f10 0x7f59fc19cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35380/0 (socket says 172.21.15.89:35380) 2024-09-15T23:27:02.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.690+0000 7f59fbfff640 1 -- 172.21.15.89:0/656037598 learned_addr learned my addr 172.21.15.89:0/656037598 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:02.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59fb7fe640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59fc101de0 0x7f59fc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59faffd640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc102790 0x7f59fc198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59fbfff640 1 -- 172.21.15.89:0/656037598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc102790 msgr2=0x7f59fc198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59fbfff640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc102790 0x7f59fc198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59fbfff640 1 -- 172.21.15.89:0/656037598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59fc101de0 msgr2=0x7f59fc198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59fbfff640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59fc101de0 0x7f59fc198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59fbfff640 1 -- 172.21.15.89:0/656037598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc19d3c0 con 0x7f59fc106f10 2024-09-15T23:27:02.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59fbfff640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59fc106f10 0x7f59fc19cce0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f59f400d750 tx=0x7f59f400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:02.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59f8ff9640 1 -- 172.21.15.89:0/656037598 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f4017070 con 0x7f59fc106f10 2024-09-15T23:27:02.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59f8ff9640 1 -- 172.21.15.89:0/656037598 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59f4005500 con 0x7f59fc106f10 2024-09-15T23:27:02.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f5a01ecc640 1 -- 172.21.15.89:0/656037598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59fc19d6b0 con 0x7f59fc106f10 2024-09-15T23:27:02.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f59f8ff9640 1 -- 172.21.15.89:0/656037598 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f4004d90 con 0x7f59fc106f10 2024-09-15T23:27:02.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.691+0000 7f5a01ecc640 1 -- 172.21.15.89:0/656037598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59fc1a4f90 con 0x7f59fc106f10 2024-09-15T23:27:02.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.692+0000 7f5a01ecc640 1 -- 172.21.15.89:0/656037598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59fc1021c0 con 0x7f59fc106f10 2024-09-15T23:27:02.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.693+0000 7f59f8ff9640 1 -- 172.21.15.89:0/656037598 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f59f4004f30 con 0x7f59fc106f10 2024-09-15T23:27:02.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.694+0000 7f59f8ff9640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59e40a32c0 0x7f59e40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.694+0000 7f59fb7fe640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59e40a32c0 0x7f59e40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.694+0000 7f59f8ff9640 1 -- 172.21.15.89:0/656037598 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f59f40cf5b0 con 0x7f59fc106f10 2024-09-15T23:27:02.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.695+0000 7f59fb7fe640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59e40a32c0 0x7f59e40a5780 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f59ec005fd0 tx=0x7f59ec009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:02.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.698+0000 7f59f8ff9640 1 -- 172.21.15.89:0/656037598 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f59f409bee0 con 0x7f59fc106f10 2024-09-15T23:27:02.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.898+0000 7f5a01ecc640 1 -- 172.21.15.89:0/656037598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59fc10a150 con 0x7f59fc106f10 2024-09-15T23:27:02.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.899+0000 7f59f8ff9640 1 -- 172.21.15.89:0/656037598 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f59f409b7e0 con 0x7f59fc106f10 2024-09-15T23:27:02.901 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:27:02.901 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:27:02.902 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:27:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 -- 172.21.15.89:0/656037598 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59e40a32c0 msgr2=0x7f59e40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59e40a32c0 0x7f59e40a5780 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f59ec005fd0 tx=0x7f59ec009680 comp rx=0 tx=0).stop 2024-09-15T23:27:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 -- 172.21.15.89:0/656037598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59fc106f10 msgr2=0x7f59fc19cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59fc106f10 0x7f59fc19cce0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f59f400d750 tx=0x7f59f400dc10 comp rx=0 tx=0).stop 2024-09-15T23:27:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 -- 172.21.15.89:0/656037598 shutdown_connections 2024-09-15T23:27:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59e40a32c0 0x7f59e40a5780 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59fc106f10 0x7f59fc19cce0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59fc102790 0x7f59fc198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 --2- 172.21.15.89:0/656037598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59fc101de0 0x7f59fc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 -- 172.21.15.89:0/656037598 >> 172.21.15.89:0/656037598 conn(0x7f59fc0fd8c0 msgr2=0x7f59fc0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 -- 172.21.15.89:0/656037598 shutdown_connections 2024-09-15T23:27:02.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:02.901+0000 7f5a01ecc640 1 -- 172.21.15.89:0/656037598 wait complete. 2024-09-15T23:27:03.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:02 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/656037598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:03.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:02 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/656037598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:03.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:02 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/656037598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:03.654 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0' 2024-09-15T23:27:03.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0 2024-09-15T23:27:03.900 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:03 smithi120 ceph-mon[35250]: pgmap v441: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-09-15T23:27:04.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:03 smithi174 ceph-mon[35586]: pgmap v441: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-09-15T23:27:04.273 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:03 smithi089 ceph-mon[30933]: pgmap v441: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-09-15T23:27:04.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.360+0000 7f98c85f3640 1 -- 172.21.15.89:0/3048507515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98c0101420 msgr2=0x7f98c010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.360+0000 7f98c85f3640 1 --2- 172.21.15.89:0/3048507515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98c0101420 0x7f98c010d950 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009cd0 tx=0x7f98b802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.360+0000 7f98c85f3640 1 -- 172.21.15.89:0/3048507515 shutdown_connections 2024-09-15T23:27:04.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.360+0000 7f98c85f3640 1 --2- 172.21.15.89:0/3048507515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98c0101420 0x7f98c010d950 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.360+0000 7f98c85f3640 1 --2- 172.21.15.89:0/3048507515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c0100a80 0x7f98c0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.360+0000 7f98c85f3640 1 --2- 172.21.15.89:0/3048507515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c0106a80 0x7f98c0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.360+0000 7f98c85f3640 1 -- 172.21.15.89:0/3048507515 >> 172.21.15.89:0/3048507515 conn(0x7f98c00fc7a0 msgr2=0x7f98c00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.360+0000 7f98c85f3640 1 -- 172.21.15.89:0/3048507515 shutdown_connections 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.360+0000 7f98c85f3640 1 -- 172.21.15.89:0/3048507515 wait complete. 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c85f3640 1 Processor -- start 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c85f3640 1 -- start start 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c85f3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c0100a80 0x7f98c0197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c85f3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98c0101420 0x7f98c01975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c85f3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c0106a80 0x7f98c0197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c85f3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98c0110e80 con 0x7f98c0101420 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c85f3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98c0110d00 con 0x7f98c0106a80 2024-09-15T23:27:04.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c85f3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98c0111000 con 0x7f98c0100a80 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c5b8e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98c0101420 0x7f98c01975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c5b8e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98c0101420 0x7f98c01975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35390/0 (socket says 172.21.15.89:35390) 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c5b8e640 1 -- 172.21.15.89:0/3236430903 learned_addr learned my addr 172.21.15.89:0/3236430903 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.361+0000 7f98c638f640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c0100a80 0x7f98c0197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98c5b8e640 1 -- 172.21.15.89:0/3236430903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c0100a80 msgr2=0x7f98c0197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98c6b90640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c0106a80 0x7f98c0197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c0100a80 0x7f98c0197080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98c5b8e640 1 -- 172.21.15.89:0/3236430903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c0106a80 msgr2=0x7f98c0197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c0106a80 0x7f98c0197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98c5b8e640 1 -- 172.21.15.89:0/3236430903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c01a68f0 con 0x7f98c0101420 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98c0101420 0x7f98c01975c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f98b000c890 tx=0x7f98b000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98b77fe640 1 -- 172.21.15.89:0/3236430903 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b0004220 con 0x7f98c0101420 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98c85f3640 1 -- 172.21.15.89:0/3236430903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c01a6be0 con 0x7f98c0101420 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98b77fe640 1 -- 172.21.15.89:0/3236430903 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98b00043c0 con 0x7f98c0101420 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98b77fe640 1 -- 172.21.15.89:0/3236430903 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b0004a00 con 0x7f98c0101420 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.362+0000 7f98c85f3640 1 -- 172.21.15.89:0/3236430903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c01a7170 con 0x7f98c0101420 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.364+0000 7f98b77fe640 1 -- 172.21.15.89:0/3236430903 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f98b00079b0 con 0x7f98c0101420 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98ca9cc740 1 Processor -- start 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98ca9cc740 1 -- start start 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587524ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x5558752554f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 0x555875257e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5558751af160 con 0x55587524a850 2024-09-15T23:27:04.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555875156720 con 0x55587524b170 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5558751ab120 con 0x555875255a30 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98c6b90640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587524ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98c6b90640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587524ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35400/0 (socket says 172.21.15.89:35400) 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98c6b90640 1 -- 172.21.15.89:0/3467302714 learned_addr learned my addr 172.21.15.89:0/3467302714 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98c638f640 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x5558752554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98b77fe640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98ac0a3240 0x7f98ac0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 0x555875257e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98c6b90640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98ac0a3240 0x7f98ac0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98c638f640 1 -- 172.21.15.89:0/3467302714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 msgr2=0x555875257e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98c638f640 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 0x555875257e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98c638f640 1 -- 172.21.15.89:0/3467302714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 msgr2=0x55587524ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.365+0000 7f98c638f640 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587524ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.366+0000 7f98c638f640 1 -- 172.21.15.89:0/3467302714 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555874f20ba0 con 0x55587524b170 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.366+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 0x555875257e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.366+0000 7f98c6b90640 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587524ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.366+0000 7f98b77fe640 1 -- 172.21.15.89:0/3236430903 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f98b00cfa00 con 0x7f98c0101420 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.366+0000 7f98c638f640 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x5558752554f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0045b0 tx=0x7f98bc00bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11bd08ee5c36f44a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.366+0000 7f98c6b90640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98ac0a3240 0x7f98ac0a5700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f98b802ae00 tx=0x7f98b802a8e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.366+0000 7f98b4ff9640 1 -- 172.21.15.89:0/3467302714 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc003bd0 con 0x55587524b170 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.366+0000 7f98b4ff9640 1 -- 172.21.15.89:0/3467302714 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98bc015070 con 0x55587524b170 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.366+0000 7f98b4ff9640 1 -- 172.21.15.89:0/3467302714 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc004d70 con 0x55587524b170 2024-09-15T23:27:04.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 -- 172.21.15.89:0/3467302714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 msgr2=0x5558752554f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x5558752554f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0045b0 tx=0x7f98bc00bb20 comp rx=0 tx=0).stop 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 -- 172.21.15.89:0/3467302714 shutdown_connections 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 0x555875257e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x5558752554f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/3467302714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587524ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 -- 172.21.15.89:0/3467302714 >> 172.21.15.89:0/3467302714 conn(0x555875246de0 msgr2=0x5558752471c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 -- 172.21.15.89:0/3467302714 shutdown_connections 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 -- 172.21.15.89:0/3467302714 wait complete. 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 -- start start 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 Processor -- start 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 0x555875252870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524b170 0x555875252db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555875255a30 0x5558752532f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555875258600 con 0x55587524b170 2024-09-15T23:27:04.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555875137900 con 0x555875255a30 2024-09-15T23:27:04.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.367+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555875258780 con 0x55587524a850 2024-09-15T23:27:04.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c5b8e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524b170 0x555875252db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98ca9cc740 10 client.0 _ll_register_callbacks cb 0x555875135780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:04.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c5b8e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524b170 0x555875252db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35412/0 (socket says 172.21.15.89:35412) 2024-09-15T23:27:04.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c5b8e640 1 -- 172.21.15.89:0/2250615697 learned_addr learned my addr 172.21.15.89:0/2250615697 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:04.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c638f640 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 0x555875252870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c6b90640 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555875255a30 0x5558752532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c5b8e640 1 -- 172.21.15.89:0/2250615697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 msgr2=0x555875252870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 0x555875252870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c5b8e640 1 -- 172.21.15.89:0/2250615697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555875255a30 msgr2=0x5558752532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555875255a30 0x5558752532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c5b8e640 1 -- 172.21.15.89:0/2250615697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555875177c50 con 0x55587524b170 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524b170 0x555875252db0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f98b00865b0 tx=0x7f98b006a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f3c28ca304586db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2250615697 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b001d040 con 0x55587524b170 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2250615697 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f98b00861e0 con 0x55587524b170 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2250615697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98bc015070 con 0x55587524b170 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2250615697 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f98b0086040 con 0x55587524b170 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98ca9cc740 10 client.16548 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:04.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.368+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2250615697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555875264230 con 0x55587524b170 2024-09-15T23:27:04.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2250615697 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98bc015070 con 0x55587524b170 2024-09-15T23:27:04.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2250615697 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f98b009b080 con 0x55587524b170 2024-09-15T23:27:04.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98ca9cc740 10 client.16548 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:04.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98ca9cc740 10 client.16548 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:27:04.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98ca9cc740 20 client.16548 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:04.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5558752656f0 0x555875267ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98ca9cc740 4 client.16548 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:04.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2250615697 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558751bc890 con 0x5558752656f0 2024-09-15T23:27:04.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98c638f640 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5558752656f0 0x555875267ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98a9ffb640 10 client.16548 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.369+0000 7f98c638f640 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5558752656f0 0x555875267ae0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2c6cdd4e8ce4324 server_cookie=9b3f3cda3c6dd173 in_seq=0 out_seq=0 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2250615697 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5558751bc890 con 0x5558752656f0 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98a9ffb640 10 client.16548 handle_command_reply: tid=0 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 client.16548 shutdown 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2250615697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524b170 msgr2=0x555875252db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524b170 0x555875252db0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f98b00865b0 tx=0x7f98b006a970 comp rx=0 tx=0).stop 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2250615697 shutdown_connections 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5558752656f0 0x555875267ae0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555875255a30 0x5558752532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524b170 0x555875252db0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98a9ffb640 0 client.16548 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:04.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2250615697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 0x555875252870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2250615697 >> 172.21.15.89:0/2250615697 conn(0x555875246de0 msgr2=0x55587524cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2250615697 shutdown_connections 2024-09-15T23:27:04.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.370+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2250615697 wait complete. 2024-09-15T23:27:04.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.371+0000 7f98ca9cc740 20 client.16548 trim_cache size 0 max 16384 2024-09-15T23:27:04.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.387+0000 7f98ca9cc740 1 Processor -- start 2024-09-15T23:27:04.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98ca9cc740 1 -- start start 2024-09-15T23:27:04.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 0x5558752517b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x55587526d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555875255a30 0x55587526d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98c01b10a0 con 0x555875255a30 2024-09-15T23:27:04.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98c01b2be0 con 0x55587524b170 2024-09-15T23:27:04.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98c01b1770 con 0x55587524a850 2024-09-15T23:27:04.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c6b90640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555875255a30 0x55587526d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c6b90640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555875255a30 0x55587526d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35420/0 (socket says 172.21.15.89:35420) 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c6b90640 1 -- 172.21.15.89:0/3440415505 learned_addr learned my addr 172.21.15.89:0/3440415505 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c638f640 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 0x5558752517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x55587526d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c6b90640 1 -- 172.21.15.89:0/3440415505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 msgr2=0x5558752517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c6b90640 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 0x5558752517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c6b90640 1 -- 172.21.15.89:0/3440415505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 msgr2=0x55587526d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c6b90640 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x55587526d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c6b90640 1 -- 172.21.15.89:0/3440415505 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555875246c10 con 0x555875255a30 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c638f640 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 0x5558752517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x55587526d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.388+0000 7f98c6b90640 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555875255a30 0x55587526d790 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f98b8031a20 tx=0x7f98b8005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9b6efe58be2757 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98abfff640 1 -- 172.21.15.89:0/3440415505 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8004240 con 0x555875255a30 2024-09-15T23:27:04.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98abfff640 1 -- 172.21.15.89:0/3440415505 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98b8036040 con 0x555875255a30 2024-09-15T23:27:04.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98abfff640 1 -- 172.21.15.89:0/3440415505 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b803f440 con 0x555875255a30 2024-09-15T23:27:04.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98ca9cc740 1 -- 172.21.15.89:0/3440415505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555875255a30 msgr2=0x55587526d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555875255a30 0x55587526d790 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f98b8031a20 tx=0x7f98b8005da0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98ca9cc740 1 -- 172.21.15.89:0/3440415505 shutdown_connections 2024-09-15T23:27:04.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555875255a30 0x55587526d790 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x55587526d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/3440415505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55587524a850 0x5558752517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98ca9cc740 1 -- 172.21.15.89:0/3440415505 >> 172.21.15.89:0/3440415505 conn(0x555875246de0 msgr2=0x5558752548f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98ca9cc740 1 -- 172.21.15.89:0/3440415505 shutdown_connections 2024-09-15T23:27:04.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.389+0000 7f98ca9cc740 1 -- 172.21.15.89:0/3440415505 wait complete. 2024-09-15T23:27:04.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98ca9cc740 1 -- start start 2024-09-15T23:27:04.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98ca9cc740 1 Processor -- start 2024-09-15T23:27:04.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587525dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x5558752589c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98ca9cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 0x555875258f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98c01b2510 con 0x55587524a850 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98c01b1e40 con 0x55587524b170 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98ca9cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98c01af080 con 0x555875255a30 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98ca9cc740 10 client.0 _ll_register_callbacks cb 0x7f98b0004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98c638f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587525dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98c638f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587525dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35436/0 (socket says 172.21.15.89:35436) 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98c638f640 1 -- 172.21.15.89:0/2352504714 learned_addr learned my addr 172.21.15.89:0/2352504714 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x5558752589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98c638f640 1 -- 172.21.15.89:0/2352504714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 msgr2=0x555875258f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.390+0000 7f98c6b90640 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 0x555875258f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98c638f640 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 0x555875258f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98c638f640 1 -- 172.21.15.89:0/2352504714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 msgr2=0x5558752589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98c638f640 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x5558752589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98c638f640 1 -- 172.21.15.89:0/2352504714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555875247a70 con 0x55587524a850 2024-09-15T23:27:04.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x5558752589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:04.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98c6b90640 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 0x555875258f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:04.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98c638f640 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587525dfe0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f98bc004500 tx=0x7f98bc00b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3ef0825346aaad server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:04.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2352504714 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc011440 con 0x55587524a850 2024-09-15T23:27:04.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2352504714 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f98bc0115e0 con 0x55587524a850 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2352504714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98b8036040 con 0x55587524a850 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2352504714 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f98bc011820 con 0x55587524a850 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98ca9cc740 10 client.16554 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2352504714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5558752423f0 con 0x55587524a850 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2352504714 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98b8036040 con 0x55587524a850 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2352504714 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f98bc02fa20 con 0x55587524a850 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98ca9cc740 10 client.16554 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98ca9cc740 10 client.16554 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98ca9cc740 20 client.16554 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5558752d1420 0x5558752d3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98ca9cc740 4 client.16554 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}] 2024-09-15T23:27:04.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.391+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2352504714 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}) -- 0x5558751b6be0 con 0x5558752d1420 2024-09-15T23:27:04.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.392+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5558752d1420 0x5558752d3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.392+0000 7f98a9ffb640 10 client.16554 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:04.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.392+0000 7f98c5b8e640 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5558752d1420 0x5558752d3840 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5c2f39c4d074ef7 server_cookie=4d7ff3dfe0c974d8 in_seq=0 out_seq=0 2024-09-15T23:27:04.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98a9ffb640 1 -- 172.21.15.89:0/2352504714 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5558751b6be0 con 0x5558752d1420 2024-09-15T23:27:04.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98a9ffb640 10 client.16554 handle_command_reply: tid=0 2024-09-15T23:27:04.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 client.16554 shutdown 2024-09-15T23:27:04.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2352504714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 msgr2=0x55587525dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587525dfe0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f98bc004500 tx=0x7f98bc00b420 comp rx=0 tx=0).stop 2024-09-15T23:27:04.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2352504714 shutdown_connections 2024-09-15T23:27:04.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5558752d1420 0x5558752d3840 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555875255a30 0x555875258f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55587524b170 0x5558752589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98a9ffb640 0 client.16554 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:04.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 --2- 172.21.15.89:0/2352504714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55587524a850 0x55587525dfe0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2352504714 >> 172.21.15.89:0/2352504714 conn(0x555875246de0 msgr2=0x5558752456a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2352504714 shutdown_connections 2024-09-15T23:27:04.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 1 -- 172.21.15.89:0/2352504714 wait complete. 2024-09-15T23:27:04.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.404+0000 7f98ca9cc740 20 client.16554 trim_cache size 0 max 16384 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 10, 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: "52838e2a": { 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 10, 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:27:04.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 139.6, 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:04.408 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:27:04.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 -- 172.21.15.89:0/3236430903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98ac0a3240 msgr2=0x7f98ac0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98ac0a3240 0x7f98ac0a5700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f98b802ae00 tx=0x7f98b802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 -- 172.21.15.89:0/3236430903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98c0101420 msgr2=0x7f98c01975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98c0101420 0x7f98c01975c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f98b000c890 tx=0x7f98b000cd50 comp rx=0 tx=0).stop 2024-09-15T23:27:04.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 -- 172.21.15.89:0/3236430903 shutdown_connections 2024-09-15T23:27:04.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98ac0a3240 0x7f98ac0a5700 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f98c0106a80 0x7f98c0197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98c0101420 0x7f98c01975c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 --2- 172.21.15.89:0/3236430903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c0100a80 0x7f98c0197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 -- 172.21.15.89:0/3236430903 >> 172.21.15.89:0/3236430903 conn(0x7f98c00fc7a0 msgr2=0x7f98c00fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 -- 172.21.15.89:0/3236430903 shutdown_connections 2024-09-15T23:27:04.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:04.405+0000 7f98c85f3640 1 -- 172.21.15.89:0/3236430903 wait complete. 2024-09-15T23:27:05.211 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '52838e2a', quiesce took 3.3 seconds. Will release after: 16.25 2024-09-15T23:27:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.706+0000 7f4bd0f68640 1 -- 172.21.15.89:0/2541793917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bcc102610 msgr2=0x7f4bcc108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.706+0000 7f4bd0f68640 1 --2- 172.21.15.89:0/2541793917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bcc102610 0x7f4bcc108ea0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009ca0 tx=0x7f4bc402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.706+0000 7f4bd0f68640 1 -- 172.21.15.89:0/2541793917 shutdown_connections 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.706+0000 7f4bd0f68640 1 --2- 172.21.15.89:0/2541793917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bcc102610 0x7f4bcc108ea0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.706+0000 7f4bd0f68640 1 --2- 172.21.15.89:0/2541793917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bcc101c50 0x7f4bcc1020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.706+0000 7f4bd0f68640 1 --2- 172.21.15.89:0/2541793917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc100a60 0x7f4bcc100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.706+0000 7f4bd0f68640 1 -- 172.21.15.89:0/2541793917 >> 172.21.15.89:0/2541793917 conn(0x7f4bcc0fc1e0 msgr2=0x7f4bcc0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.706+0000 7f4bd0f68640 1 -- 172.21.15.89:0/2541793917 shutdown_connections 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.706+0000 7f4bd0f68640 1 -- 172.21.15.89:0/2541793917 wait complete. 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bd0f68640 1 Processor -- start 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bd0f68640 1 -- start start 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bd0f68640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc100a60 0x7f4bcc196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bd0f68640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bcc101c50 0x7f4bcc1971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bd0f68640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bcc102610 0x7f4bcc19e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bd0f68640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc10c6e0 con 0x7f4bcc101c50 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bd0f68640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc10c560 con 0x7f4bcc100a60 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bd0f68640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc10c860 con 0x7f4bcc102610 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bc9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bcc101c50 0x7f4bcc1971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bc9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bcc101c50 0x7f4bcc1971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45596/0 (socket says 172.21.15.89:45596) 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bc9d9b640 1 -- 172.21.15.89:0/636733199 learned_addr learned my addr 172.21.15.89:0/636733199 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bcc102610 0x7f4bcc19e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.707+0000 7f4bca59c640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc100a60 0x7f4bcc196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bc9d9b640 1 -- 172.21.15.89:0/636733199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bcc102610 msgr2=0x7f4bcc19e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bc9d9b640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bcc102610 0x7f4bcc19e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bc9d9b640 1 -- 172.21.15.89:0/636733199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc100a60 msgr2=0x7f4bcc196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bc9d9b640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc100a60 0x7f4bcc196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bc9d9b640 1 -- 172.21.15.89:0/636733199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc19e8c0 con 0x7f4bcc101c50 2024-09-15T23:27:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bc9d9b640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bcc101c50 0x7f4bcc1971c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000d770 tx=0x7f4bc000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bbb7fe640 1 -- 172.21.15.89:0/636733199 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0017070 con 0x7f4bcc101c50 2024-09-15T23:27:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bbb7fe640 1 -- 172.21.15.89:0/636733199 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bc0005500 con 0x7f4bcc101c50 2024-09-15T23:27:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bd0f68640 1 -- 172.21.15.89:0/636733199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bcc19ebb0 con 0x7f4bcc101c50 2024-09-15T23:27:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bbb7fe640 1 -- 172.21.15.89:0/636733199 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0004e00 con 0x7f4bcc101c50 2024-09-15T23:27:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.708+0000 7f4bd0f68640 1 -- 172.21.15.89:0/636733199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bcc19f0b0 con 0x7f4bcc101c50 2024-09-15T23:27:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.710+0000 7f4bbb7fe640 1 -- 172.21.15.89:0/636733199 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4bc0004fa0 con 0x7f4bcc101c50 2024-09-15T23:27:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bd3dd9740 1 Processor -- start 2024-09-15T23:27:05.718 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bd3dd9740 1 -- start start 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f59b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 0x55c074f5a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5a6a0 0x55c074f6acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c074ec22f0 con 0x55c074f5b300 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c074e6a270 con 0x55c074f5b700 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c074ebe2b0 con 0x55c074f5a6a0 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bbb7fe640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bb40a32f0 0x7f4bb40a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bcad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f59b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.711+0000 7f4bcad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f59b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45612/0 (socket says 172.21.15.89:45612) 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bcad9d640 1 -- 172.21.15.89:0/2791157791 learned_addr learned my addr 172.21.15.89:0/2791157791 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bb40a32f0 0x7f4bb40a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.719 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bca59c640 1 --2- 172.21.15.89:0/2791157791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 0x55c074f5a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bbb7fe640 1 -- 172.21.15.89:0/636733199 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4bc00cf620 con 0x7f4bcc101c50 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bc9d9b640 1 --2- 172.21.15.89:0/2791157791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5a6a0 0x55c074f6acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bcad9d640 1 -- 172.21.15.89:0/2791157791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5a6a0 msgr2=0x55c074f6acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/2791157791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5a6a0 0x55c074f6acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bcad9d640 1 -- 172.21.15.89:0/2791157791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 msgr2=0x55c074f5a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/2791157791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 0x55c074f5a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bcad9d640 1 -- 172.21.15.89:0/2791157791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c074e69340 con 0x55c074f5b300 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bca59c640 1 --2- 172.21.15.89:0/2791157791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 0x55c074f5a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/2791157791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f59b40 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009ca0 tx=0x7f4bc4031650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905ac2f6ac02de5e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bb40a32f0 0x7f4bb40a57b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4035000 tx=0x7f4bc4036000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bb8ff9640 1 -- 172.21.15.89:0/2791157791 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc404d070 con 0x55c074f5b300 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bb8ff9640 1 -- 172.21.15.89:0/2791157791 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bc4036070 con 0x55c074f5b300 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.712+0000 7f4bb8ff9640 1 -- 172.21.15.89:0/2791157791 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4049640 con 0x55c074f5b300 2024-09-15T23:27:05.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.713+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/2791157791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 msgr2=0x55c074f59b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.713+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/2791157791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f59b40 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009ca0 tx=0x7f4bc4031650 comp rx=0 tx=0).stop 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.713+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/2791157791 shutdown_connections 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.713+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/2791157791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5a6a0 0x55c074f6acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.713+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/2791157791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 0x55c074f5a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.713+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/2791157791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f59b40 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.713+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/2791157791 >> 172.21.15.89:0/2791157791 conn(0x55c074f5c220 msgr2=0x55c074f5c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.713+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/2791157791 shutdown_connections 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.713+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/2791157791 wait complete. 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bd3dd9740 1 -- start start 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bd3dd9740 1 Processor -- start 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5a6a0 0x55c074f6ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5b300 0x55c074f6d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f743c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c074f6b510 con 0x55c074f5b700 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c074e4b290 con 0x55c074f5a6a0 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c074f6b690 con 0x55c074f5b300 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bd3dd9740 10 client.0 _ll_register_callbacks cb 0x55c074e7c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bcad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f743c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bcad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f743c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45616/0 (socket says 172.21.15.89:45616) 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bcad9d640 1 -- 172.21.15.89:0/1404612520 learned_addr learned my addr 172.21.15.89:0/1404612520 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bca59c640 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5a6a0 0x55c074f6ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bc9d9b640 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5b300 0x55c074f6d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bcad9d640 1 -- 172.21.15.89:0/1404612520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5b300 msgr2=0x55c074f6d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5b300 0x55c074f6d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bcad9d640 1 -- 172.21.15.89:0/1404612520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5a6a0 msgr2=0x55c074f6ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5a6a0 0x55c074f6ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bcad9d640 1 -- 172.21.15.89:0/1404612520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c074bd5210 con 0x55c074f5b700 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bc9d9b640 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5b300 0x55c074f6d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f743c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4bc402a900 tx=0x7f4bc40309e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728986aefbae3951 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.714+0000 7f4bb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/1404612520 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc404d070 con 0x55c074f5b700 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/1404612520 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4bc403b400 con 0x55c074f5b700 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/1404612520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc4036070 con 0x55c074f5b700 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/1404612520 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4bc4066050 con 0x55c074f5b700 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bd3dd9740 10 client.16566 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/1404612520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c074ca1620 con 0x55c074f5b700 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/1404612520 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bc4036070 con 0x55c074f5b700 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/1404612520 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4bc4049400 con 0x55c074f5b700 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bd3dd9740 10 client.16566 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:05.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bd3dd9740 10 client.16566 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bd3dd9740 20 client.16566 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c074f788b0 0x55c074f7aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bd3dd9740 4 client.16566 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.715+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/1404612520 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c074ecfa20 con 0x55c074f788b0 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.716+0000 7f4bca59c640 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c074f788b0 0x55c074f7aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.716+0000 7f4bb1ffb640 10 client.16566 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.716+0000 7f4bca59c640 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c074f788b0 0x55c074f7aca0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b81aabb6dd9fca0 server_cookie=304c7f999b6923b6 in_seq=0 out_seq=0 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/1404612520 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c074ecfa20 con 0x55c074f788b0 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bb1ffb640 10 client.16566 handle_command_reply: tid=0 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 client.16566 shutdown 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/1404612520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 msgr2=0x55c074f743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f743c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4bc402a900 tx=0x7f4bc40309e0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/1404612520 shutdown_connections 2024-09-15T23:27:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c074f788b0 0x55c074f7aca0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f743c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5b300 0x55c074f6d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/1404612520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5a6a0 0x55c074f6ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bb1ffb640 0 client.16566 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/1404612520 >> 172.21.15.89:0/1404612520 conn(0x55c074f5c220 msgr2=0x55c074f5d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/1404612520 shutdown_connections 2024-09-15T23:27:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/1404612520 wait complete. 2024-09-15T23:27:05.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.717+0000 7f4bd3dd9740 20 client.16566 trim_cache size 0 max 16384 2024-09-15T23:27:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bd3dd9740 1 Processor -- start 2024-09-15T23:27:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bd3dd9740 1 -- start start 2024-09-15T23:27:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5a6a0 0x55c074f6ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f74380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 0x55c074f599b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1a84e0 con 0x55c074f5b300 2024-09-15T23:27:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1aa020 con 0x55c074f5b700 2024-09-15T23:27:05.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1a8bb0 con 0x55c074f5a6a0 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bc9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f74380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bc9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f74380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45630/0 (socket says 172.21.15.89:45630) 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.731+0000 7f4bc9d9b640 1 -- 172.21.15.89:0/3053977331 learned_addr learned my addr 172.21.15.89:0/3053977331 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bc9d9b640 1 -- 172.21.15.89:0/3053977331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5a6a0 msgr2=0x55c074f6ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bc9d9b640 1 --2- 172.21.15.89:0/3053977331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5a6a0 0x55c074f6ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bc9d9b640 1 -- 172.21.15.89:0/3053977331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 msgr2=0x55c074f599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/3053977331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 0x55c074f599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bc9d9b640 1 --2- 172.21.15.89:0/3053977331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 0x55c074f599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bc9d9b640 1 -- 172.21.15.89:0/3053977331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c074f6b730 con 0x55c074f5b300 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/3053977331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 0x55c074f599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bc9d9b640 1 --2- 172.21.15.89:0/3053977331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f74380 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00982d0 tx=0x7f4bc0009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf3fb770e385990 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bb3fff640 1 -- 172.21.15.89:0/3053977331 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0020040 con 0x55c074f5b300 2024-09-15T23:27:05.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bb3fff640 1 -- 172.21.15.89:0/3053977331 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bc009b900 con 0x55c074f5b300 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bb3fff640 1 -- 172.21.15.89:0/3053977331 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0009cc0 con 0x55c074f5b300 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/3053977331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 msgr2=0x55c074f74380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/3053977331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f74380 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00982d0 tx=0x7f4bc0009c30 comp rx=0 tx=0).stop 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/3053977331 shutdown_connections 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/3053977331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5b700 0x55c074f599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/3053977331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b300 0x55c074f74380 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.732+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/3053977331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5a6a0 0x55c074f6ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/3053977331 >> 172.21.15.89:0/3053977331 conn(0x55c074f5c220 msgr2=0x55c074f5d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/3053977331 shutdown_connections 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/3053977331 wait complete. 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 -- start start 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 Processor -- start 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5a6a0 0x55c074f719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5b300 0x55c074f71f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f72f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1a9950 con 0x55c074f5b700 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1a9280 con 0x55c074f5a6a0 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc1a64c0 con 0x55c074f5b300 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bd3dd9740 10 client.0 _ll_register_callbacks cb 0x7f4bbc004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bcad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f72f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bcad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f72f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45640/0 (socket says 172.21.15.89:45640) 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bcad9d640 1 -- 172.21.15.89:0/498182073 learned_addr learned my addr 172.21.15.89:0/498182073 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bca59c640 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5a6a0 0x55c074f719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bcad9d640 1 -- 172.21.15.89:0/498182073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5b300 msgr2=0x55c074f71f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5b300 0x55c074f71f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bcad9d640 1 -- 172.21.15.89:0/498182073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5a6a0 msgr2=0x55c074f719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5a6a0 0x55c074f719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.733+0000 7f4bcad9d640 1 -- 172.21.15.89:0/498182073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c074f73550 con 0x55c074f5b700 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bcad9d640 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f72f40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f4bc402ade0 tx=0x7f4bc402f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10bce8aa81aad39b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/498182073 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc404d070 con 0x55c074f5b700 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/498182073 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4bc403b400 con 0x55c074f5b700 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/498182073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc009b900 con 0x55c074f5b700 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/498182073 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4bc4049e60 con 0x55c074f5b700 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bd3dd9740 10 client.16572 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/498182073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c074fe3580 con 0x55c074f5b700 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/498182073 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bc009b900 con 0x55c074f5b700 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/498182073 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4bc4049400 con 0x55c074f5b700 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bd3dd9740 10 client.16572 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bd3dd9740 10 client.16572 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:05.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bd3dd9740 20 client.16572 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c074fe3dc0 0x55c074fe61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bd3dd9740 4 client.16572 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:27:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.734+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/498182073 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c074ec9d70 con 0x55c074fe3dc0 2024-09-15T23:27:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.735+0000 7f4bca59c640 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c074fe3dc0 0x55c074fe61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.735+0000 7f4bb1ffb640 10 client.16572 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:05.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.735+0000 7f4bca59c640 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c074fe3dc0 0x55c074fe61e0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90a39a09871fc327 server_cookie=e284263a99c6b6e5 in_seq=0 out_seq=0 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bb1ffb640 1 -- 172.21.15.89:0/498182073 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c074ec9d70 con 0x55c074fe3dc0 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bb1ffb640 10 client.16572 handle_command_reply: tid=0 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 client.16572 shutdown 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/498182073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 msgr2=0x55c074f72f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f72f40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f4bc402ade0 tx=0x7f4bc402f480 comp rx=0 tx=0).stop 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/498182073 shutdown_connections 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c074fe3dc0 0x55c074fe61e0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c074f5b700 0x55c074f72f40 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bb1ffb640 0 client.16572 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c074f5b300 0x55c074f71f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 --2- 172.21.15.89:0/498182073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c074f5a6a0 0x55c074f719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/498182073 >> 172.21.15.89:0/498182073 conn(0x55c074f5c220 msgr2=0x55c074f589a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/498182073 shutdown_connections 2024-09-15T23:27:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 1 -- 172.21.15.89:0/498182073 wait complete. 2024-09-15T23:27:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.746+0000 7f4bd3dd9740 20 client.16572 trim_cache size 0 max 16384 2024-09-15T23:27:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:27:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:27:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:27:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:27:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 -- 172.21.15.89:0/636733199 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bb40a32f0 msgr2=0x7f4bb40a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bb40a32f0 0x7f4bb40a57b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4035000 tx=0x7f4bc4036000 comp rx=0 tx=0).stop 2024-09-15T23:27:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 -- 172.21.15.89:0/636733199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bcc101c50 msgr2=0x7f4bcc1971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bcc101c50 0x7f4bcc1971c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000d770 tx=0x7f4bc000dc30 comp rx=0 tx=0).stop 2024-09-15T23:27:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 -- 172.21.15.89:0/636733199 shutdown_connections 2024-09-15T23:27:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bb40a32f0 0x7f4bb40a57b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bcc102610 0x7f4bcc19e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4bcc101c50 0x7f4bcc1971c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 --2- 172.21.15.89:0/636733199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bcc100a60 0x7f4bcc196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 -- 172.21.15.89:0/636733199 >> 172.21.15.89:0/636733199 conn(0x7f4bcc0fc1e0 msgr2=0x7f4bcc104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 -- 172.21.15.89:0/636733199 shutdown_connections 2024-09-15T23:27:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:05.747+0000 7f4bd0f68640 1 -- 172.21.15.89:0/636733199 wait complete. 2024-09-15T23:27:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:06 smithi120 ceph-mon[35250]: pgmap v442: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-15T23:27:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:06 smithi174 ceph-mon[35586]: pgmap v442: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-15T23:27:06.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:06 smithi089 ceph-mon[30933]: pgmap v442: 129 pgs: 129 active+clean; 1.4 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-15T23:27:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:08 smithi120 ceph-mon[35250]: pgmap v443: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-09-15T23:27:08.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:08 smithi174 ceph-mon[35586]: pgmap v443: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-09-15T23:27:08.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:08 smithi089 ceph-mon[30933]: pgmap v443: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.2 KiB/s wr, 18 op/s 2024-09-15T23:27:10.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:10 smithi120 ceph-mon[35250]: pgmap v444: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-09-15T23:27:10.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:10 smithi174 ceph-mon[35586]: pgmap v444: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-09-15T23:27:10.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:10 smithi089 ceph-mon[30933]: pgmap v444: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-09-15T23:27:11.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:11 smithi120 ceph-mon[35250]: pgmap v445: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.8 KiB/s wr, 14 op/s 2024-09-15T23:27:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:11 smithi174 ceph-mon[35586]: pgmap v445: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.8 KiB/s wr, 14 op/s 2024-09-15T23:27:11.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:11 smithi089 ceph-mon[30933]: pgmap v445: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.8 KiB/s wr, 14 op/s 2024-09-15T23:27:14.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:14 smithi120 ceph-mon[35250]: pgmap v446: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-09-15T23:27:14.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:14 smithi174 ceph-mon[35586]: pgmap v446: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-09-15T23:27:14.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:14 smithi089 ceph-mon[30933]: pgmap v446: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-09-15T23:27:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:16 smithi120 ceph-mon[35250]: pgmap v447: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.6 KiB/s wr, 12 op/s 2024-09-15T23:27:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:16 smithi174 ceph-mon[35586]: pgmap v447: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.6 KiB/s wr, 12 op/s 2024-09-15T23:27:16.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:16 smithi089 ceph-mon[30933]: pgmap v447: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.6 KiB/s wr, 12 op/s 2024-09-15T23:27:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:17 smithi120 ceph-mon[35250]: pgmap v448: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-09-15T23:27:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:17 smithi174 ceph-mon[35586]: pgmap v448: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-09-15T23:27:17.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:17 smithi089 ceph-mon[30933]: pgmap v448: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-09-15T23:27:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.843+0000 7fd58f84c640 1 -- 172.21.15.89:0/376082127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd588101c40 msgr2=0x7fd5881020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.843+0000 7fd58f84c640 1 --2- 172.21.15.89:0/376082127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd588101c40 0x7fd5881020c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009900 tx=0x7fd57c02a410 comp rx=0 tx=0).stop 2024-09-15T23:27:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.843+0000 7fd58f84c640 1 -- 172.21.15.89:0/376082127 shutdown_connections 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.843+0000 7fd58f84c640 1 --2- 172.21.15.89:0/376082127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd588102600 0x7fd588108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.843+0000 7fd58f84c640 1 --2- 172.21.15.89:0/376082127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd588101c40 0x7fd5881020c0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.843+0000 7fd58f84c640 1 --2- 172.21.15.89:0/376082127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd588100a50 0x7fd588100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.843+0000 7fd58f84c640 1 -- 172.21.15.89:0/376082127 >> 172.21.15.89:0/376082127 conn(0x7fd5880fc1b0 msgr2=0x7fd5880fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.843+0000 7fd58f84c640 1 -- 172.21.15.89:0/376082127 shutdown_connections 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.843+0000 7fd58f84c640 1 -- 172.21.15.89:0/376082127 wait complete. 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58f84c640 1 Processor -- start 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58f84c640 1 -- start start 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58f84c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd588100a50 0x7fd588197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58f84c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd588101c40 0x7fd5881982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58f84c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd588102600 0x7fd58819f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58f84c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd58810d820 con 0x7fd588101c40 2024-09-15T23:27:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58f84c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd58810d6a0 con 0x7fd588100a50 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58f84c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd58810d9a0 con 0x7fd588102600 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58cde7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd588101c40 0x7fd5881982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58cde7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd588101c40 0x7fd5881982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45988/0 (socket says 172.21.15.89:45988) 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58cde7640 1 -- 172.21.15.89:0/2811303789 learned_addr learned my addr 172.21.15.89:0/2811303789 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd588100a50 0x7fd588197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58dde9640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd588102600 0x7fd58819f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58cde7640 1 -- 172.21.15.89:0/2811303789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd588102600 msgr2=0x7fd58819f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58cde7640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd588102600 0x7fd58819f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58cde7640 1 -- 172.21.15.89:0/2811303789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd588100a50 msgr2=0x7fd588197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58cde7640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd588100a50 0x7fd588197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.844+0000 7fd58cde7640 1 -- 172.21.15.89:0/2811303789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58819f9d0 con 0x7fd588101c40 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.845+0000 7fd58cde7640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd588101c40 0x7fd5881982f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009230 tx=0x7fd57c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.845+0000 7fd57a7fc640 1 -- 172.21.15.89:0/2811303789 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c03f030 con 0x7fd588101c40 2024-09-15T23:27:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.845+0000 7fd58f84c640 1 -- 172.21.15.89:0/2811303789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd58819fc60 con 0x7fd588101c40 2024-09-15T23:27:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.845+0000 7fd57a7fc640 1 -- 172.21.15.89:0/2811303789 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd57c003ce0 con 0x7fd588101c40 2024-09-15T23:27:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.845+0000 7fd57a7fc640 1 -- 172.21.15.89:0/2811303789 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c035070 con 0x7fd588101c40 2024-09-15T23:27:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.845+0000 7fd58f84c640 1 -- 172.21.15.89:0/2811303789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5881a01a0 con 0x7fd588101c40 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.847+0000 7fd57a7fc640 1 -- 172.21.15.89:0/2811303789 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd57c042030 con 0x7fd588101c40 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd5925e6740 1 Processor -- start 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd5925e6740 1 -- start start 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223ce7cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce7d2e0 0x56223ce81da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce84830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd5925e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56223cde02f0 con 0x56223ce82370 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd5925e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56223cd88270 con 0x56223ce7d2e0 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd5925e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56223cddc2b0 con 0x56223ce7c9a0 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd58cde7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce84830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.854 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd58cde7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce84830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45994/0 (socket says 172.21.15.89:45994) 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd58cde7640 1 -- 172.21.15.89:0/3084013222 learned_addr learned my addr 172.21.15.89:0/3084013222 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd58dde9640 1 --2- 172.21.15.89:0/3084013222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223ce7cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd57a7fc640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5740a32f0 0x7fd5740a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.848+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/3084013222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce7d2e0 0x56223ce81da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd58dde9640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5740a32f0 0x7fd5740a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd58cde7640 1 -- 172.21.15.89:0/3084013222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 msgr2=0x56223ce7cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd58cde7640 1 --2- 172.21.15.89:0/3084013222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223ce7cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd58cde7640 1 -- 172.21.15.89:0/3084013222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce7d2e0 msgr2=0x56223ce81da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd58cde7640 1 --2- 172.21.15.89:0/3084013222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce7d2e0 0x56223ce81da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd58cde7640 1 -- 172.21.15.89:0/3084013222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56223cd87340 con 0x56223ce82370 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd57a7fc640 1 -- 172.21.15.89:0/2811303789 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd57c0f06f0 con 0x7fd588101c40 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd58cde7640 1 --2- 172.21.15.89:0/3084013222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce84830 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0b8d80 tx=0x7fd57c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8dbbcf381389768 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd58dde9640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5740a32f0 0x7fd5740a57b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd580005fd0 tx=0x7fd580004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:18.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd5737fe640 1 -- 172.21.15.89:0/3084013222 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c049070 con 0x56223ce82370 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd5737fe640 1 -- 172.21.15.89:0/3084013222 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd57c0bd020 con 0x56223ce82370 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.849+0000 7fd5737fe640 1 -- 172.21.15.89:0/3084013222 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0c0cd0 con 0x56223ce82370 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 -- 172.21.15.89:0/3084013222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 msgr2=0x56223ce84830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 --2- 172.21.15.89:0/3084013222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce84830 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0b8d80 tx=0x7fd57c009260 comp rx=0 tx=0).stop 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 -- 172.21.15.89:0/3084013222 shutdown_connections 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 --2- 172.21.15.89:0/3084013222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce84830 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 --2- 172.21.15.89:0/3084013222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce7d2e0 0x56223ce81da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 --2- 172.21.15.89:0/3084013222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223ce7cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 -- 172.21.15.89:0/3084013222 >> 172.21.15.89:0/3084013222 conn(0x56223ce78010 msgr2=0x56223ce78410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 -- 172.21.15.89:0/3084013222 shutdown_connections 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 -- 172.21.15.89:0/3084013222 wait complete. 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 -- start start 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.850+0000 7fd5925e6740 1 Processor -- start 2024-09-15T23:27:18.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223ce86880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce7d2e0 0x56223ce86dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce8dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd5925e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56223ce85100 con 0x56223ce82370 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd5925e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56223cd69290 con 0x56223ce7d2e0 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd5925e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56223ce85280 con 0x56223ce7c9a0 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd5925e6740 10 client.0 _ll_register_callbacks cb 0x56223cd9a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58dde9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce8dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58dde9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce8dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46010/0 (socket says 172.21.15.89:46010) 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58dde9640 1 -- 172.21.15.89:0/3762051825 learned_addr learned my addr 172.21.15.89:0/3762051825 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58cde7640 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce7d2e0 0x56223ce86dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223ce86880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58dde9640 1 -- 172.21.15.89:0/3762051825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 msgr2=0x56223ce86880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58dde9640 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223ce86880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58dde9640 1 -- 172.21.15.89:0/3762051825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce7d2e0 msgr2=0x56223ce86dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58dde9640 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce7d2e0 0x56223ce86dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58dde9640 1 -- 172.21.15.89:0/3762051825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56223caf3210 con 0x56223ce82370 2024-09-15T23:27:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd58dde9640 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce8dd90 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd5800133f0 tx=0x7fd580004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48913bebcba6259b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.851+0000 7fd570ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:18.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd570ff9640 1 -- 172.21.15.89:0/3762051825 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580019070 con 0x56223ce82370 2024-09-15T23:27:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd570ff9640 1 -- 172.21.15.89:0/3762051825 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd580003970 con 0x56223ce82370 2024-09-15T23:27:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd5925e6740 1 -- 172.21.15.89:0/3762051825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd57c0bd020 con 0x56223ce82370 2024-09-15T23:27:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd570ff9640 1 -- 172.21.15.89:0/3762051825 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd580003b70 con 0x56223ce82370 2024-09-15T23:27:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd5925e6740 10 client.16584 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd5925e6740 1 -- 172.21.15.89:0/3762051825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56223cbbf620 con 0x56223ce82370 2024-09-15T23:27:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd570ff9640 1 -- 172.21.15.89:0/3762051825 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd57c0bd020 con 0x56223ce82370 2024-09-15T23:27:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd570ff9640 1 -- 172.21.15.89:0/3762051825 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd58001f070 con 0x56223ce82370 2024-09-15T23:27:18.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd5925e6740 10 client.16584 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:18.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd5925e6740 10 client.16584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:18.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd5925e6740 20 client.16584 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:18.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd5925e6740 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56223ce92560 0x56223ce94a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd5925e6740 4 client.16584 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:18.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.852+0000 7fd5925e6740 1 -- 172.21.15.89:0/3762051825 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56223cdeda20 con 0x56223ce92560 2024-09-15T23:27:18.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.853+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56223ce92560 0x56223ce94a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.853+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56223ce92560 0x56223ce94a20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5708605a71bd552 server_cookie=e9f93c0e2bcc971 in_seq=0 out_seq=0 2024-09-15T23:27:18.861 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.853+0000 7fd570ff9640 10 client.16584 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:18.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd570ff9640 1 -- 172.21.15.89:0/3762051825 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56223cdeda20 con 0x56223ce92560 2024-09-15T23:27:18.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd570ff9640 10 client.16584 handle_command_reply: tid=0 2024-09-15T23:27:18.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 client.16584 shutdown 2024-09-15T23:27:18.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 -- 172.21.15.89:0/3762051825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 msgr2=0x56223ce8dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:18.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce8dd90 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd5800133f0 tx=0x7fd580004a40 comp rx=0 tx=0).stop 2024-09-15T23:27:18.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 -- 172.21.15.89:0/3762051825 shutdown_connections 2024-09-15T23:27:18.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56223ce92560 0x56223ce94a20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce82370 0x56223ce8dd90 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce7d2e0 0x56223ce86dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 --2- 172.21.15.89:0/3762051825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223ce86880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd570ff9640 0 client.16584 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:18.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 -- 172.21.15.89:0/3762051825 >> 172.21.15.89:0/3762051825 conn(0x56223ce78010 msgr2=0x56223ce79820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:18.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 -- 172.21.15.89:0/3762051825 shutdown_connections 2024-09-15T23:27:18.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 1 -- 172.21.15.89:0/3762051825 wait complete. 2024-09-15T23:27:18.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.854+0000 7fd5925e6740 20 client.16584 trim_cache size 0 max 16384 2024-09-15T23:27:18.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd5925e6740 1 Processor -- start 2024-09-15T23:27:18.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd5925e6740 1 -- start start 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7c9a0 0x56223ce78ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7d2e0 0x56223ce79400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce82370 0x56223ce79940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd5925e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd5881aa8b0 con 0x56223ce7c9a0 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd5925e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd5881ac3f0 con 0x56223ce82370 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd5925e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd5881aaf80 con 0x56223ce7d2e0 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd58d5e8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7c9a0 0x56223ce78ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd58d5e8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7c9a0 0x56223ce78ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46022/0 (socket says 172.21.15.89:46022) 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd58d5e8640 1 -- 172.21.15.89:0/593293679 learned_addr learned my addr 172.21.15.89:0/593293679 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd58d5e8640 1 -- 172.21.15.89:0/593293679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7d2e0 msgr2=0x56223ce79400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/593293679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7d2e0 0x56223ce79400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd58d5e8640 1 -- 172.21.15.89:0/593293679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce82370 msgr2=0x56223ce79940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/593293679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce82370 0x56223ce79940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd58d5e8640 1 -- 172.21.15.89:0/593293679 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56223ce7d7c0 con 0x56223ce7c9a0 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.869+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/593293679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7c9a0 0x56223ce78ec0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x56223ce78e90 tx=0x7fd58400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a85bc133a238d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd572ffd640 1 -- 172.21.15.89:0/593293679 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584026440 con 0x56223ce7c9a0 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd572ffd640 1 -- 172.21.15.89:0/593293679 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd584026ac0 con 0x56223ce7c9a0 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd572ffd640 1 -- 172.21.15.89:0/593293679 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584011b60 con 0x56223ce7c9a0 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd5925e6740 1 -- 172.21.15.89:0/593293679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7c9a0 msgr2=0x56223ce78ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd5925e6740 1 --2- 172.21.15.89:0/593293679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7c9a0 0x56223ce78ec0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x56223ce78e90 tx=0x7fd58400b390 comp rx=0 tx=0).stop 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd5925e6740 1 -- 172.21.15.89:0/593293679 shutdown_connections 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd5925e6740 1 --2- 172.21.15.89:0/593293679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce82370 0x56223ce79940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd5925e6740 1 --2- 172.21.15.89:0/593293679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7d2e0 0x56223ce79400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd5925e6740 1 --2- 172.21.15.89:0/593293679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7c9a0 0x56223ce78ec0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd5925e6740 1 -- 172.21.15.89:0/593293679 >> 172.21.15.89:0/593293679 conn(0x56223ce78010 msgr2=0x56223ce86880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd5925e6740 1 -- 172.21.15.89:0/593293679 shutdown_connections 2024-09-15T23:27:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.870+0000 7fd5925e6740 1 -- 172.21.15.89:0/593293679 wait complete. 2024-09-15T23:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd5925e6740 1 -- start start 2024-09-15T23:27:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd5925e6740 1 Processor -- start 2024-09-15T23:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223cef8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7d2e0 0x56223cef8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd5925e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce82370 0x56223ceffdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd5925e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd5881abd20 con 0x56223ce7d2e0 2024-09-15T23:27:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd5925e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd5881ab650 con 0x56223ce82370 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd5925e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd5881a8890 con 0x56223ce7c9a0 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd5925e6740 10 client.0 _ll_register_callbacks cb 0x7fd580009f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58cde7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7d2e0 0x56223cef8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58cde7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7d2e0 0x56223cef8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46030/0 (socket says 172.21.15.89:46030) 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58cde7640 1 -- 172.21.15.89:0/4077206245 learned_addr learned my addr 172.21.15.89:0/4077206245 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223cef8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58cde7640 1 -- 172.21.15.89:0/4077206245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 msgr2=0x56223cef8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58dde9640 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce82370 0x56223ceffdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58cde7640 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223cef8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58cde7640 1 -- 172.21.15.89:0/4077206245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce82370 msgr2=0x56223ceffdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58cde7640 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce82370 0x56223ceffdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58cde7640 1 -- 172.21.15.89:0/4077206245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56223cf003e0 con 0x56223ce7d2e0 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.871+0000 7fd58dde9640 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce82370 0x56223ceffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd58cde7640 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7d2e0 0x56223cef8d90 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0b92f0 tx=0x7fd57c0f64b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58e53150fab2e9d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd570ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd570ff9640 1 -- 172.21.15.89:0/4077206245 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c049070 con 0x56223ce7d2e0 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd5925e6740 1 -- 172.21.15.89:0/4077206245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd584026ac0 con 0x56223ce7d2e0 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd570ff9640 1 -- 172.21.15.89:0/4077206245 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd57c0ba9c0 con 0x56223ce7d2e0 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd570ff9640 1 -- 172.21.15.89:0/4077206245 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd57c0bc280 con 0x56223ce7d2e0 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd5925e6740 10 client.16590 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd5925e6740 1 -- 172.21.15.89:0/4077206245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56223cf01760 con 0x56223ce7d2e0 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd570ff9640 1 -- 172.21.15.89:0/4077206245 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd584026ac0 con 0x56223ce7d2e0 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd570ff9640 1 -- 172.21.15.89:0/4077206245 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd57c0f3ce0 con 0x56223ce7d2e0 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd5925e6740 10 client.16590 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd5925e6740 10 client.16590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd5925e6740 20 client.16590 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd5925e6740 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56223cf01fc0 0x56223cf043e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd5925e6740 4 client.16590 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:27:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.872+0000 7fd5925e6740 1 -- 172.21.15.89:0/4077206245 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56223cde7d70 con 0x56223cf01fc0 2024-09-15T23:27:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.873+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56223cf01fc0 0x56223cf043e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.873+0000 7fd58d5e8640 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56223cf01fc0 0x56223cf043e0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76fd2cadf30db698 server_cookie=16e96bba9199ec90 in_seq=0 out_seq=0 2024-09-15T23:27:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.873+0000 7fd570ff9640 10 client.16590 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd570ff9640 1 -- 172.21.15.89:0/4077206245 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56223cde7d70 con 0x56223cf01fc0 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd570ff9640 10 client.16590 handle_command_reply: tid=0 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 client.16590 shutdown 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 -- 172.21.15.89:0/4077206245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7d2e0 msgr2=0x56223cef8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7d2e0 0x56223cef8d90 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0b92f0 tx=0x7fd57c0f64b0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 -- 172.21.15.89:0/4077206245 shutdown_connections 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56223cf01fc0 0x56223cf043e0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56223ce82370 0x56223ceffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd570ff9640 0 client.16590 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56223ce7d2e0 0x56223cef8d90 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 --2- 172.21.15.89:0/4077206245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56223ce7c9a0 0x56223cef8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.885 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 -- 172.21.15.89:0/4077206245 >> 172.21.15.89:0/4077206245 conn(0x56223ce78010 msgr2=0x56223ce769a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 -- 172.21.15.89:0/4077206245 shutdown_connections 2024-09-15T23:27:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.883+0000 7fd5925e6740 1 -- 172.21.15.89:0/4077206245 wait complete. 2024-09-15T23:27:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd5925e6740 20 client.16590 trim_cache size 0 max 16384 2024-09-15T23:27:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:27:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:27:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:27:18.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:27:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 -- 172.21.15.89:0/2811303789 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5740a32f0 msgr2=0x7fd5740a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5740a32f0 0x7fd5740a57b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd580005fd0 tx=0x7fd580004ab0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 -- 172.21.15.89:0/2811303789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd588101c40 msgr2=0x7fd5881982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd588101c40 0x7fd5881982f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009230 tx=0x7fd57c004360 comp rx=0 tx=0).stop 2024-09-15T23:27:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 -- 172.21.15.89:0/2811303789 shutdown_connections 2024-09-15T23:27:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd5740a32f0 0x7fd5740a57b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd588102600 0x7fd58819f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd588101c40 0x7fd5881982f0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 --2- 172.21.15.89:0/2811303789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd588100a50 0x7fd588197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 -- 172.21.15.89:0/2811303789 >> 172.21.15.89:0/2811303789 conn(0x7fd5880fc1b0 msgr2=0x7fd588104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 -- 172.21.15.89:0/2811303789 shutdown_connections 2024-09-15T23:27:18.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:18.884+0000 7fd58f84c640 1 -- 172.21.15.89:0/2811303789 wait complete. 2024-09-15T23:27:20.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:20 smithi120 ceph-mon[35250]: pgmap v449: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 10 op/s 2024-09-15T23:27:20.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:20 smithi174 ceph-mon[35586]: pgmap v449: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 10 op/s 2024-09-15T23:27:20.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:20 smithi089 ceph-mon[30933]: pgmap v449: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 10 op/s 2024-09-15T23:27:21.463 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:27:21.721 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 -- 172.21.15.89:0/199722257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b20102790 msgr2=0x7f0b201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 --2- 172.21.15.89:0/199722257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b20102790 0x7f0b201067e0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009900 tx=0x7f0b0802a410 comp rx=0 tx=0).stop 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 -- 172.21.15.89:0/199722257 shutdown_connections 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 --2- 172.21.15.89:0/199722257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b20106f10 0x7f0b2010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 --2- 172.21.15.89:0/199722257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b20102790 0x7f0b201067e0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 --2- 172.21.15.89:0/199722257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b20101de0 0x7f0b201021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 -- 172.21.15.89:0/199722257 >> 172.21.15.89:0/199722257 conn(0x7f0b200fd8c0 msgr2=0x7f0b200ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 -- 172.21.15.89:0/199722257 shutdown_connections 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 -- 172.21.15.89:0/199722257 wait complete. 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 Processor -- start 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.220+0000 7f0b25941640 1 -- start start 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b25941640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b20101de0 0x7f0b20198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b25941640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b20102790 0x7f0b201989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b25941640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b20106f10 0x7f0b2019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b25941640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0b20111f70 con 0x7f0b20102790 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b25941640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0b20111df0 con 0x7f0b20101de0 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b25941640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0b201120f0 con 0x7f0b20106f10 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b167fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b20102790 0x7f0b201989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b167fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b20102790 0x7f0b201989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46036/0 (socket says 172.21.15.89:46036) 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b167fc640 1 -- 172.21.15.89:0/1801295520 learned_addr learned my addr 172.21.15.89:0/1801295520 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b1f7fe640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b20106f10 0x7f0b2019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b1effd640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b20101de0 0x7f0b20198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b167fc640 1 -- 172.21.15.89:0/1801295520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b20106f10 msgr2=0x7f0b2019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b167fc640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b20106f10 0x7f0b2019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b167fc640 1 -- 172.21.15.89:0/1801295520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b20101de0 msgr2=0x7f0b20198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b167fc640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b20101de0 0x7f0b20198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b167fc640 1 -- 172.21.15.89:0/1801295520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b2019d440 con 0x7f0b20102790 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.221+0000 7f0b167fc640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b20102790 0x7f0b201989d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0b08006010 tx=0x7f0b080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.222+0000 7f0b2493f640 1 -- 172.21.15.89:0/1801295520 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0803f030 con 0x7f0b20102790 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.222+0000 7f0b2493f640 1 -- 172.21.15.89:0/1801295520 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b08003ce0 con 0x7f0b20102790 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.222+0000 7f0b25941640 1 -- 172.21.15.89:0/1801295520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b2019d6d0 con 0x7f0b20102790 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.222+0000 7f0b2493f640 1 -- 172.21.15.89:0/1801295520 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08035070 con 0x7f0b20102790 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.222+0000 7f0b25941640 1 -- 172.21.15.89:0/1801295520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b201a5010 con 0x7f0b20102790 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.223+0000 7f0b25941640 1 -- 172.21.15.89:0/1801295520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b201021c0 con 0x7f0b20102790 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.224+0000 7f0b2493f640 1 -- 172.21.15.89:0/1801295520 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0b08042030 con 0x7f0b20102790 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.224+0000 7f0b2493f640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b0c0a32c0 0x7f0b0c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.225+0000 7f0b1effd640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b0c0a32c0 0x7f0b0c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.225+0000 7f0b2493f640 1 -- 172.21.15.89:0/1801295520 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0b080f0290 con 0x7f0b20102790 2024-09-15T23:27:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.225+0000 7f0b1effd640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b0c0a32c0 0x7f0b0c0a5780 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0b100058c0 tx=0x7f0b1000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.229+0000 7f0b2493f640 1 -- 172.21.15.89:0/1801295520 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0b080bcc80 con 0x7f0b20102790 2024-09-15T23:27:22.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.429+0000 7f0b25941640 1 -- 172.21.15.89:0/1801295520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b2010a0f0 con 0x7f0b20102790 2024-09-15T23:27:22.431 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:22 smithi089 ceph-mon[30933]: pgmap v450: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 10 op/s 2024-09-15T23:27:22.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.430+0000 7f0b2493f640 1 -- 172.21.15.89:0/1801295520 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f0b080bc580 con 0x7f0b20102790 2024-09-15T23:27:22.432 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:27:22.432 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:27:22.432 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:27:22.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 -- 172.21.15.89:0/1801295520 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b0c0a32c0 msgr2=0x7f0b0c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b0c0a32c0 0x7f0b0c0a5780 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0b100058c0 tx=0x7f0b1000a4c0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 -- 172.21.15.89:0/1801295520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b20102790 msgr2=0x7f0b201989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b20102790 0x7f0b201989d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0b08006010 tx=0x7f0b080042f0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 -- 172.21.15.89:0/1801295520 shutdown_connections 2024-09-15T23:27:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b0c0a32c0 0x7f0b0c0a5780 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b20106f10 0x7f0b2019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b20102790 0x7f0b201989d0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 --2- 172.21.15.89:0/1801295520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b20101de0 0x7f0b20198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 -- 172.21.15.89:0/1801295520 >> 172.21.15.89:0/1801295520 conn(0x7f0b200fd8c0 msgr2=0x7f0b200fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 -- 172.21.15.89:0/1801295520 shutdown_connections 2024-09-15T23:27:22.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:22.432+0000 7f0b25941640 1 -- 172.21.15.89:0/1801295520 wait complete. 2024-09-15T23:27:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:22 smithi120 ceph-mon[35250]: pgmap v450: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 10 op/s 2024-09-15T23:27:22.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:22 smithi174 ceph-mon[35586]: pgmap v450: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 10 op/s 2024-09-15T23:27:23.243 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 52838e2a --release --await' 2024-09-15T23:27:23.243 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 52838e2a --release --await 2024-09-15T23:27:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:23 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1801295520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:23 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1801295520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:23.488 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:23.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:23 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1801295520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:23.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.971+0000 7f6a1db94640 1 -- 172.21.15.89:0/4251341542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a18102790 msgr2=0x7f6a181067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:23.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.971+0000 7f6a1db94640 1 --2- 172.21.15.89:0/4251341542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a18102790 0x7f6a181067e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c009900 tx=0x7f6a0c02a410 comp rx=0 tx=0).stop 2024-09-15T23:27:23.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.971+0000 7f6a1db94640 1 -- 172.21.15.89:0/4251341542 shutdown_connections 2024-09-15T23:27:23.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.971+0000 7f6a1db94640 1 --2- 172.21.15.89:0/4251341542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a18106f10 0x7f6a1810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.971+0000 7f6a1db94640 1 --2- 172.21.15.89:0/4251341542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a18102790 0x7f6a181067e0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.971+0000 7f6a1db94640 1 --2- 172.21.15.89:0/4251341542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a18101de0 0x7f6a181021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.971+0000 7f6a1db94640 1 -- 172.21.15.89:0/4251341542 >> 172.21.15.89:0/4251341542 conn(0x7f6a180fd8c0 msgr2=0x7f6a180ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.971+0000 7f6a1db94640 1 -- 172.21.15.89:0/4251341542 shutdown_connections 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.971+0000 7f6a1db94640 1 -- 172.21.15.89:0/4251341542 wait complete. 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a1db94640 1 Processor -- start 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a1db94640 1 -- start start 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a1db94640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a18101de0 0x7f6a18198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a1db94640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a18102790 0x7f6a181988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a1db94640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a18106f10 0x7f6a1819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a1db94640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a18111f70 con 0x7f6a18102790 2024-09-15T23:27:23.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a1db94640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6a18111df0 con 0x7f6a18101de0 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a1db94640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181120f0 con 0x7f6a18106f10 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a16ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a18102790 0x7f6a181988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a16ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a18102790 0x7f6a181988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46052/0 (socket says 172.21.15.89:46052) 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.972+0000 7f6a16ffd640 1 -- 172.21.15.89:0/2727217385 learned_addr learned my addr 172.21.15.89:0/2727217385 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a17fff640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a18106f10 0x7f6a1819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a177fe640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a18101de0 0x7f6a18198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a16ffd640 1 -- 172.21.15.89:0/2727217385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a18106f10 msgr2=0x7f6a1819cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a18106f10 0x7f6a1819cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a16ffd640 1 -- 172.21.15.89:0/2727217385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a18101de0 msgr2=0x7f6a18198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a18101de0 0x7f6a18198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a16ffd640 1 -- 172.21.15.89:0/2727217385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a1819d330 con 0x7f6a18102790 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a18102790 0x7f6a181988c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c006010 tx=0x7f6a0c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a14ff9640 1 -- 172.21.15.89:0/2727217385 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c03f030 con 0x7f6a18102790 2024-09-15T23:27:23.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a1db94640 1 -- 172.21.15.89:0/2727217385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a1819d5c0 con 0x7f6a18102790 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a14ff9640 1 -- 172.21.15.89:0/2727217385 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a0c003ce0 con 0x7f6a18102790 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a14ff9640 1 -- 172.21.15.89:0/2727217385 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c035070 con 0x7f6a18102790 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.973+0000 7f6a1db94640 1 -- 172.21.15.89:0/2727217385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a181a4f00 con 0x7f6a18102790 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a14ff9640 1 -- 172.21.15.89:0/2727217385 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6a0c042030 con 0x7f6a18102790 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a1ff6d740 1 Processor -- start 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a1ff6d740 1 -- start start 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88060 0x5590e4f88440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f92d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4f95630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5590e4eed620 con 0x5590e4f88980 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5590e4e94e50 con 0x5590e4f93240 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5590e4ee95e0 con 0x5590e4f88060 2024-09-15T23:27:23.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a177fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f92d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a177fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f92d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46064/0 (socket says 172.21.15.89:46064) 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a177fe640 1 -- 172.21.15.89:0/3950025436 learned_addr learned my addr 172.21.15.89:0/3950025436 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a17fff640 1 --2- 172.21.15.89:0/3950025436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88060 0x5590e4f88440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a14ff9640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a000a3240 0x7f6a000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/3950025436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4f95630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a177fe640 1 -- 172.21.15.89:0/3950025436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88060 msgr2=0x5590e4f88440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a177fe640 1 --2- 172.21.15.89:0/3950025436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88060 0x5590e4f88440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a177fe640 1 -- 172.21.15.89:0/3950025436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 msgr2=0x5590e4f95630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a177fe640 1 --2- 172.21.15.89:0/3950025436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4f95630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a177fe640 1 -- 172.21.15.89:0/3950025436 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590e4e85d40 con 0x5590e4f88980 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.976+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/3950025436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4f95630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.977+0000 7f6a17fff640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a000a3240 0x7f6a000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.977+0000 7f6a14ff9640 1 -- 172.21.15.89:0/2727217385 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6a0c0f03d0 con 0x7f6a18102790 2024-09-15T23:27:23.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.977+0000 7f6a177fe640 1 --2- 172.21.15.89:0/3950025436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f92d00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800cc60 tx=0x7f6a0800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83fba213eaef44 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.977+0000 7f69fdffb640 1 -- 172.21.15.89:0/3950025436 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a08014070 con 0x5590e4f88980 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.977+0000 7f69fdffb640 1 -- 172.21.15.89:0/3950025436 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a080047a0 con 0x5590e4f88980 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.977+0000 7f69fdffb640 1 -- 172.21.15.89:0/3950025436 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a08010760 con 0x5590e4f88980 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/3950025436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 msgr2=0x5590e4f92d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/3950025436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f92d00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800cc60 tx=0x7f6a0800e720 comp rx=0 tx=0).stop 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a17fff640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a000a3240 0x7f6a000a5700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6a10005fd0 tx=0x7f6a10004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/3950025436 shutdown_connections 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/3950025436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4f95630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/3950025436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f92d00 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/3950025436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88060 0x5590e4f88440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/3950025436 >> 172.21.15.89:0/3950025436 conn(0x5590e4f845f0 msgr2=0x5590e4f849d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/3950025436 shutdown_connections 2024-09-15T23:27:23.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/3950025436 wait complete. 2024-09-15T23:27:23.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 -- start start 2024-09-15T23:27:23.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.978+0000 7f6a1ff6d740 1 Processor -- start 2024-09-15T23:27:23.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88060 0x5590e4f900e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f90620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4f90b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5590e4f95e10 con 0x5590e4f88980 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5590e4e75f10 con 0x5590e4f93240 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5590e4f95f90 con 0x5590e4f88060 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a1ff6d740 10 client.0 _ll_register_callbacks cb 0x5590e4e99db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a16ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f90620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a16ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f90620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46080/0 (socket says 172.21.15.89:46080) 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a16ffd640 1 -- 172.21.15.89:0/4108730578 learned_addr learned my addr 172.21.15.89:0/4108730578 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a177fe640 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88060 0x5590e4f900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a17fff640 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4f90b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a16ffd640 1 -- 172.21.15.89:0/4108730578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88060 msgr2=0x5590e4f900e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88060 0x5590e4f900e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a16ffd640 1 -- 172.21.15.89:0/4108730578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 msgr2=0x5590e4f90b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4f90b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a16ffd640 1 -- 172.21.15.89:0/4108730578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590e4bfc580 con 0x5590e4f88980 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a17fff640 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4f90b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f90620 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0bc460 tx=0x7f6a0c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33bc83906bd1b1a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f69f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f69f6ffd640 1 -- 172.21.15.89:0/4108730578 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c049070 con 0x5590e4f88980 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/4108730578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a080047a0 con 0x5590e4f88980 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.979+0000 7f69f6ffd640 1 -- 172.21.15.89:0/4108730578 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6a0c0f0120 con 0x5590e4f88980 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f69f6ffd640 1 -- 172.21.15.89:0/4108730578 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a0c0bc0c0 con 0x5590e4f88980 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f6a1ff6d740 10 client.16608 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/4108730578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5590e4cc6960 con 0x5590e4f88980 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f69f6ffd640 1 -- 172.21.15.89:0/4108730578 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a080047a0 con 0x5590e4f88980 2024-09-15T23:27:23.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f69f6ffd640 1 -- 172.21.15.89:0/4108730578 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6a0c0c4f40 con 0x5590e4f88980 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f6a1ff6d740 10 client.16608 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f6a1ff6d740 10 client.16608 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f6a1ff6d740 20 client.16608 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5590e4fa2b60 0x5590e4fa4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f6a1ff6d740 4 client.16608 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/4108730578 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590e4efab90 con 0x5590e4fa2b60 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.980+0000 7f6a177fe640 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5590e4fa2b60 0x5590e4fa4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.981+0000 7f69f6ffd640 10 client.16608 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.981+0000 7f6a177fe640 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5590e4fa2b60 0x5590e4fa4fd0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9816727f73194f7 server_cookie=158b496500d23df2 in_seq=0 out_seq=0 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.981+0000 7f69f6ffd640 1 -- 172.21.15.89:0/4108730578 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5590e4efab90 con 0x5590e4fa2b60 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.981+0000 7f69f6ffd640 10 client.16608 handle_command_reply: tid=0 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.981+0000 7f6a1ff6d740 1 client.16608 shutdown 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/4108730578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 msgr2=0x5590e4f90620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:23.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f90620 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0bc460 tx=0x7f6a0c009560 comp rx=0 tx=0).stop 2024-09-15T23:27:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/4108730578 shutdown_connections 2024-09-15T23:27:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5590e4fa2b60 0x5590e4fa4fd0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4f90b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88980 0x5590e4f90620 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/4108730578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88060 0x5590e4f900e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f69f6ffd640 0 client.16608 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/4108730578 >> 172.21.15.89:0/4108730578 conn(0x5590e4f845f0 msgr2=0x5590e4f8a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/4108730578 shutdown_connections 2024-09-15T23:27:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/4108730578 wait complete. 2024-09-15T23:27:23.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.982+0000 7f6a1ff6d740 20 client.16608 trim_cache size 0 max 16384 2024-09-15T23:27:24.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.998+0000 7f6a1ff6d740 1 Processor -- start 2024-09-15T23:27:24.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.998+0000 7f6a1ff6d740 1 -- start start 2024-09-15T23:27:24.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f83640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:24.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 0x5590e4faa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:24.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4faade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:24.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181aedb0 con 0x5590e4f88060 2024-09-15T23:27:24.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181b08f0 con 0x5590e4f93240 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181af480 con 0x5590e4f88980 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a177fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f83640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a177fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f83640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46084/0 (socket says 172.21.15.89:46084) 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a177fe640 1 -- 172.21.15.89:0/338083778 learned_addr learned my addr 172.21.15.89:0/338083778 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/338083778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 0x5590e4faa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a17fff640 1 --2- 172.21.15.89:0/338083778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4faade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a177fe640 1 -- 172.21.15.89:0/338083778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 msgr2=0x5590e4faa8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a177fe640 1 --2- 172.21.15.89:0/338083778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 0x5590e4faa8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a177fe640 1 -- 172.21.15.89:0/338083778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 msgr2=0x5590e4faade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a177fe640 1 --2- 172.21.15.89:0/338083778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4faade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a177fe640 1 -- 172.21.15.89:0/338083778 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590e4f8f090 con 0x5590e4f88060 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/338083778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 0x5590e4faa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f6a177fe640 1 --2- 172.21.15.89:0/338083778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f83640 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800c7e0 tx=0x7f6a0800c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f8e57f1f65614f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:24.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f69fd7fa640 1 -- 172.21.15.89:0/338083778 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a08029070 con 0x5590e4f88060 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f69fd7fa640 1 -- 172.21.15.89:0/338083778 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a08005590 con 0x5590e4f88060 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:23.999+0000 7f69fd7fa640 1 -- 172.21.15.89:0/338083778 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a08014050 con 0x5590e4f88060 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.000+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/338083778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 msgr2=0x5590e4f83640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.000+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/338083778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f83640 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800c7e0 tx=0x7f6a0800c550 comp rx=0 tx=0).stop 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.000+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/338083778 shutdown_connections 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.000+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/338083778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e4faade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.000+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/338083778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 0x5590e4faa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.000+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/338083778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f83640 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.000+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/338083778 >> 172.21.15.89:0/338083778 conn(0x5590e4f845f0 msgr2=0x5590e4f82f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.000+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/338083778 shutdown_connections 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.000+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/338083778 wait complete. 2024-09-15T23:27:24.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.000+0000 7f6a1ff6d740 1 -- start start 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a1ff6d740 1 Processor -- start 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f9b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 0x5590e4f9b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a1ff6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e50068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181b0220 con 0x5590e4f88060 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181afb50 con 0x5590e4f93240 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a1ff6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a181acd90 con 0x5590e4f88980 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a1ff6d740 10 client.0 _ll_register_callbacks cb 0x7f6a0c0b8360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a177fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f9b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a177fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f9b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46096/0 (socket says 172.21.15.89:46096) 2024-09-15T23:27:24.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a177fe640 1 -- 172.21.15.89:0/963255647 learned_addr learned my addr 172.21.15.89:0/963255647 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:24.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 0x5590e4f9b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:24.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a17fff640 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e50068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:24.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a177fe640 1 -- 172.21.15.89:0/963255647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 msgr2=0x5590e4f9b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:24.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a177fe640 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 0x5590e4f9b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a177fe640 1 -- 172.21.15.89:0/963255647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 msgr2=0x5590e50068c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:24.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a177fe640 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e50068c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a177fe640 1 -- 172.21.15.89:0/963255647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590e4f887b0 con 0x5590e4f88060 2024-09-15T23:27:24.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f6a177fe640 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f9b1e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6a08000c00 tx=0x7f6a08005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b2b174fe78e614 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:24.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.001+0000 7f69f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f69f6ffd640 1 -- 172.21.15.89:0/963255647 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a08032060 con 0x5590e4f88060 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f69f6ffd640 1 -- 172.21.15.89:0/963255647 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6a080035f0 con 0x5590e4f88060 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/963255647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a08005590 con 0x5590e4f88060 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f69f6ffd640 1 -- 172.21.15.89:0/963255647 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a08014020 con 0x5590e4f88060 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f6a1ff6d740 10 client.16614 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/963255647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5590e50120d0 con 0x5590e4f88060 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f69f6ffd640 1 -- 172.21.15.89:0/963255647 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a08005590 con 0x5590e4f88060 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f69f6ffd640 1 -- 172.21.15.89:0/963255647 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6a0802c070 con 0x5590e4f88060 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f6a1ff6d740 10 client.16614 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f6a1ff6d740 10 client.16614 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f6a1ff6d740 20 client.16614 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5590e50125a0 0x5590e5014990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:24.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f6a1ff6d740 4 client.16614 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "52838e2a", "release": true, "await": true}] 2024-09-15T23:27:24.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/963255647 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "52838e2a", "release": true, "await": true}) -- 0x5590e4ef50a0 con 0x5590e50125a0 2024-09-15T23:27:24.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.002+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5590e50125a0 0x5590e5014990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:24.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.003+0000 7f6a16ffd640 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5590e50125a0 0x5590e5014990 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dd36657cc7c0447 server_cookie=30374a4f3b029c97 in_seq=0 out_seq=0 2024-09-15T23:27:24.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.003+0000 7f69f6ffd640 10 client.16614 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:24.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f69f6ffd640 1 -- 172.21.15.89:0/963255647 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5590e4ef50a0 con 0x5590e50125a0 2024-09-15T23:27:24.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f69f6ffd640 10 client.16614 handle_command_reply: tid=0 2024-09-15T23:27:24.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 client.16614 shutdown 2024-09-15T23:27:24.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/963255647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 msgr2=0x5590e4f9b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:24.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f9b1e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6a08000c00 tx=0x7f6a08005d60 comp rx=0 tx=0).stop 2024-09-15T23:27:24.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/963255647 shutdown_connections 2024-09-15T23:27:24.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5590e50125a0 0x5590e5014990 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590e4f93240 0x5590e50068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f69f6ffd640 0 client.16614 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:24.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590e4f88980 0x5590e4f9b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 --2- 172.21.15.89:0/963255647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590e4f88060 0x5590e4f9b1e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/963255647 >> 172.21.15.89:0/963255647 conn(0x5590e4f845f0 msgr2=0x5590e4f92160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:24.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/963255647 shutdown_connections 2024-09-15T23:27:24.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 1 -- 172.21.15.89:0/963255647 wait complete. 2024-09-15T23:27:24.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.007+0000 7f6a1ff6d740 20 client.16614 trim_cache size 0 max 16384 2024-09-15T23:27:24.011 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:27:24.011 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:27:24.011 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:27:24.011 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 12, 2024-09-15T23:27:24.011 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "52838e2a": { 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 12, 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 139.6, 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:27:24.012 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:27:24.013 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:27:24.013 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:27:24.013 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:24.013 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:24.013 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:24.013 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:24.013 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:24.013 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:27:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 -- 172.21.15.89:0/2727217385 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a000a3240 msgr2=0x7f6a000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a000a3240 0x7f6a000a5700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6a10005fd0 tx=0x7f6a10004a40 comp rx=0 tx=0).stop 2024-09-15T23:27:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 -- 172.21.15.89:0/2727217385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a18102790 msgr2=0x7f6a181988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a18102790 0x7f6a181988c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c006010 tx=0x7f6a0c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 -- 172.21.15.89:0/2727217385 shutdown_connections 2024-09-15T23:27:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a000a3240 0x7f6a000a5700 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a18106f10 0x7f6a1819cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a18102790 0x7f6a181988c0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 --2- 172.21.15.89:0/2727217385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a18101de0 0x7f6a18198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:24.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 -- 172.21.15.89:0/2727217385 >> 172.21.15.89:0/2727217385 conn(0x7f6a180fd8c0 msgr2=0x7f6a180fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:24.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 -- 172.21.15.89:0/2727217385 shutdown_connections 2024-09-15T23:27:24.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:24.008+0000 7f6a1db94640 1 -- 172.21.15.89:0/2727217385 wait complete. 2024-09-15T23:27:24.015 INFO:tasks.workunit.client.0.smithi089.stdout:1/600: open fd = 3 2024-09-15T23:27:24.015 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.015 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.015 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.015 INFO:tasks.workunit.client.0.smithi089.stdout:2/600: open fd = 3 2024-09-15T23:27:24.015 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.015 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.015 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:3/600: open fd = 3 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:4/600: open fd = 3 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:5/600: open fd = 3 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.016 INFO:tasks.workunit.client.0.smithi089.stdout:6/600: open fd = 3 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:7/600: open fd = 3 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:8/600: open fd = 3 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:9/600: open fd = 3 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.017 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:10/600: open fd = 3 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:11/600: open fd = 3 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:12/600: open fd = 3 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.018 INFO:tasks.workunit.client.0.smithi089.stdout:13/600: open fd = 3 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:14/600: open fd = 3 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:15/600: open fd = 3 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:16/600: open fd = 3 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.019 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:17/600: open fd = 3 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:18/600: open fd = 3 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:19/600: open fd = 3 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.020 INFO:tasks.workunit.client.0.smithi089.stdout:20/600: open fd = 3 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:21/600: open fd = 3 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:22/600: open fd = 3 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:23/600: open fd = 3 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.021 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:24/600: open fd = 3 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:25/600: open fd = 3 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:26/600: open fd = 3 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.022 INFO:tasks.workunit.client.0.smithi089.stdout:27/600: open fd = 3 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:28/600: open fd = 3 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:29/600: open fd = 3 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:30/600: open fd = 3 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.023 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:31/600: open fd = 3 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:32/600: open fd = 3 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:33/600: open fd = 3 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.024 INFO:tasks.workunit.client.0.smithi089.stdout:34/600: open fd = 3 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:35/600: open fd = 3 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:36/600: open fd = 3 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:37/600: open fd = 3 2024-09-15T23:27:24.025 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:38/600: open fd = 3 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:39/600: open fd = 3 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:40/600: open fd = 3 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.026 INFO:tasks.workunit.client.0.smithi089.stdout:41/600: open fd = 3 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:42/600: open fd = 3 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:43/600: open fd = 3 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:44/600: open fd = 3 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.027 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:45/600: open fd = 3 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:46/600: open fd = 3 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:47/600: open fd = 3 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.028 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:48/600: open fd = 3 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:49/600: open fd = 3 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:50/600: open fd = 3 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:51/600: open fd = 3 2024-09-15T23:27:24.029 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:52/600: open fd = 3 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:53/600: open fd = 3 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:54/600: open fd = 3 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.030 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.031 INFO:tasks.workunit.client.0.smithi089.stdout:55/600: open fd = 3 2024-09-15T23:27:24.031 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.031 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.031 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.031 INFO:tasks.workunit.client.0.smithi089.stdout:56/600: open fd = 3 2024-09-15T23:27:24.031 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.031 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.031 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.031 INFO:tasks.workunit.client.0.smithi089.stdout:57/600: open fd = 3 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:58/600: open fd = 3 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:59/600: open fd = 3 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:60/600: open fd = 3 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.032 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:61/600: open fd = 3 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:62/600: open fd = 3 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:63/600: open fd = 3 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:27:24.033 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:27:24.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:24 smithi089 ceph-mon[30933]: pgmap v451: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-09-15T23:27:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:24 smithi120 ceph-mon[35250]: pgmap v451: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-09-15T23:27:24.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:24 smithi174 ceph-mon[35586]: pgmap v451: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-09-15T23:27:24.748 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '52838e2a', total seconds elapsed: 22.8 2024-09-15T23:27:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:25 smithi120 ceph-mon[35250]: pgmap v452: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-09-15T23:27:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:25 smithi174 ceph-mon[35586]: pgmap v452: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-09-15T23:27:25.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:25 smithi089 ceph-mon[30933]: pgmap v452: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-09-15T23:27:26.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:27:26.906 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:27.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.455+0000 7f31f7c7d640 1 -- 172.21.15.89:0/11185688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0102ef0 msgr2=0x7f31f010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:27.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.455+0000 7f31f7c7d640 1 --2- 172.21.15.89:0/11185688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0102ef0 0x7f31f010da40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f31e8009ca0 tx=0x7f31e802a950 comp rx=0 tx=0).stop 2024-09-15T23:27:27.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.455+0000 7f31f7c7d640 1 -- 172.21.15.89:0/11185688 shutdown_connections 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.455+0000 7f31f7c7d640 1 --2- 172.21.15.89:0/11185688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0102ef0 0x7f31f010da40 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.455+0000 7f31f7c7d640 1 --2- 172.21.15.89:0/11185688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f0102570 0x7f31f01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.455+0000 7f31f7c7d640 1 --2- 172.21.15.89:0/11185688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31f0100e20 0x7f31f0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.455+0000 7f31f7c7d640 1 -- 172.21.15.89:0/11185688 >> 172.21.15.89:0/11185688 conn(0x7f31f00fc7d0 msgr2=0x7f31f00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.455+0000 7f31f7c7d640 1 -- 172.21.15.89:0/11185688 shutdown_connections 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.455+0000 7f31f7c7d640 1 -- 172.21.15.89:0/11185688 wait complete. 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f7c7d640 1 Processor -- start 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f7c7d640 1 -- start start 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f7c7d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31f0100e20 0x7f31f00fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f7c7d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f0102570 0x7f31f00fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f7c7d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0102ef0 0x7f31f00fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f7c7d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f31f0110fa0 con 0x7f31f0102ef0 2024-09-15T23:27:27.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f7c7d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f31f0110e20 con 0x7f31f0100e20 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f7c7d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f31f0111120 con 0x7f31f0102570 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f621a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0102ef0 0x7f31f00fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f621a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0102ef0 0x7f31f00fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37380/0 (socket says 172.21.15.89:37380) 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f621a640 1 -- 172.21.15.89:0/1828756668 learned_addr learned my addr 172.21.15.89:0/1828756668 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f5a19640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31f0100e20 0x7f31f00fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f5218640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f0102570 0x7f31f00fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f621a640 1 -- 172.21.15.89:0/1828756668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f0102570 msgr2=0x7f31f00fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f621a640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f0102570 0x7f31f00fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f621a640 1 -- 172.21.15.89:0/1828756668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31f0100e20 msgr2=0x7f31f00fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f621a640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31f0100e20 0x7f31f00fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f621a640 1 -- 172.21.15.89:0/1828756668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31f01a6a50 con 0x7f31f0102ef0 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.456+0000 7f31f5218640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f0102570 0x7f31f00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.457+0000 7f31f621a640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0102ef0 0x7f31f00fd700 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f31e802ae70 tx=0x7f31e802f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:27.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.457+0000 7f31e6ffd640 1 -- 172.21.15.89:0/1828756668 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e8034040 con 0x7f31f0102ef0 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.457+0000 7f31f7c7d640 1 -- 172.21.15.89:0/1828756668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31f01a6ce0 con 0x7f31f0102ef0 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.457+0000 7f31e6ffd640 1 -- 172.21.15.89:0/1828756668 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31e8030920 con 0x7f31f0102ef0 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.457+0000 7f31e6ffd640 1 -- 172.21.15.89:0/1828756668 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e8045480 con 0x7f31f0102ef0 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.457+0000 7f31f7c7d640 1 -- 172.21.15.89:0/1828756668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31f01a7210 con 0x7f31f0102ef0 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.459+0000 7f31e6ffd640 1 -- 172.21.15.89:0/1828756668 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f31e8037020 con 0x7f31f0102ef0 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.459+0000 7f31fa056740 1 Processor -- start 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.459+0000 7f31fa056740 1 -- start start 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31fa056740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a6c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31fa056740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a77140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31fa056740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a77680 0x563f89a79a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31fa056740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563f899d22e0 con 0x563f89a6c4a0 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31fa056740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563f89979ce0 con 0x563f89a77680 2024-09-15T23:27:27.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31fa056740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563f899ce2a0 con 0x563f89a6cdc0 2024-09-15T23:27:27.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f5218640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a6c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31e6ffd640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31dc0a3240 0x7f31dc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f5218640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a6c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37392/0 (socket says 172.21.15.89:37392) 2024-09-15T23:27:27.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f5218640 1 -- 172.21.15.89:0/3633041973 learned_addr learned my addr 172.21.15.89:0/3633041973 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:27.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f621a640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31dc0a3240 0x7f31dc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f5a19640 1 --2- 172.21.15.89:0/3633041973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a77140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f5218640 1 -- 172.21.15.89:0/3633041973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 msgr2=0x563f89a77140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f5218640 1 --2- 172.21.15.89:0/3633041973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a77140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f5218640 1 -- 172.21.15.89:0/3633041973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a77680 msgr2=0x563f89a79a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f5218640 1 --2- 172.21.15.89:0/3633041973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a77680 0x563f89a79a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31e6ffd640 1 -- 172.21.15.89:0/1828756668 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f31e80f0c60 con 0x7f31f0102ef0 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f5218640 1 -- 172.21.15.89:0/3633041973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f8995e440 con 0x563f89a6c4a0 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f5218640 1 --2- 172.21.15.89:0/3633041973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a6c880 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f31e000aa90 tx=0x7f31e000af50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27559098a7ba813a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31f621a640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31dc0a3240 0x7f31dc0a5700 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f31e80b9860 tx=0x7f31e808fd10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31dbfff640 1 -- 172.21.15.89:0/3633041973 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e000f040 con 0x563f89a6c4a0 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31dbfff640 1 -- 172.21.15.89:0/3633041973 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31e0018070 con 0x563f89a6c4a0 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.460+0000 7f31dbfff640 1 -- 172.21.15.89:0/3633041973 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e0012620 con 0x563f89a6c4a0 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.461+0000 7f31fa056740 1 -- 172.21.15.89:0/3633041973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 msgr2=0x563f89a6c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.461+0000 7f31fa056740 1 --2- 172.21.15.89:0/3633041973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a6c880 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f31e000aa90 tx=0x7f31e000af50 comp rx=0 tx=0).stop 2024-09-15T23:27:27.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.461+0000 7f31fa056740 1 -- 172.21.15.89:0/3633041973 shutdown_connections 2024-09-15T23:27:27.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.461+0000 7f31fa056740 1 --2- 172.21.15.89:0/3633041973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a77680 0x563f89a79a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.461+0000 7f31fa056740 1 --2- 172.21.15.89:0/3633041973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a77140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.461+0000 7f31fa056740 1 --2- 172.21.15.89:0/3633041973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a6c880 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.461+0000 7f31fa056740 1 -- 172.21.15.89:0/3633041973 >> 172.21.15.89:0/3633041973 conn(0x563f89a68a30 msgr2=0x563f89a68e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:27.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.461+0000 7f31fa056740 1 -- 172.21.15.89:0/3633041973 shutdown_connections 2024-09-15T23:27:27.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.461+0000 7f31fa056740 1 -- 172.21.15.89:0/3633041973 wait complete. 2024-09-15T23:27:27.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31fa056740 1 -- start start 2024-09-15T23:27:27.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31fa056740 1 Processor -- start 2024-09-15T23:27:27.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31fa056740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a6c4a0 0x563f89a76f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31fa056740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6cdc0 0x563f89a741b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31fa056740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a77680 0x563f89a746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31fa056740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563f89a7a250 con 0x563f89a6cdc0 2024-09-15T23:27:27.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31fa056740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563f8998d050 con 0x563f89a6c4a0 2024-09-15T23:27:27.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31fa056740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563f89a7a3d0 con 0x563f89a77680 2024-09-15T23:27:27.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31fa056740 10 client.0 _ll_register_callbacks cb 0x563f899cd160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:27.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31f5a19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6cdc0 0x563f89a741b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31f5a19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6cdc0 0x563f89a741b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37402/0 (socket says 172.21.15.89:37402) 2024-09-15T23:27:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.462+0000 7f31f5a19640 1 -- 172.21.15.89:0/1147449731 learned_addr learned my addr 172.21.15.89:0/1147449731 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31f5218640 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a6c4a0 0x563f89a76f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31f5218640 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a77680 0x563f89a746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31f5a19640 1 -- 172.21.15.89:0/1147449731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a77680 msgr2=0x563f89a746f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31f5a19640 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a77680 0x563f89a746f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31f5a19640 1 -- 172.21.15.89:0/1147449731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a6c4a0 msgr2=0x563f89a76f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31f5a19640 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a6c4a0 0x563f89a76f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31f5a19640 1 -- 172.21.15.89:0/1147449731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f897433c0 con 0x563f89a6cdc0 2024-09-15T23:27:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31f5218640 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a77680 0x563f89a746f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31f5a19640 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6cdc0 0x563f89a741b0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00bf10 tx=0x7f31ec012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5580221b49c5c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:27.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:27.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31d97fa640 1 -- 172.21.15.89:0/1147449731 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec005220 con 0x563f89a6cdc0 2024-09-15T23:27:27.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31d97fa640 1 -- 172.21.15.89:0/1147449731 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f31ec0053c0 con 0x563f89a6cdc0 2024-09-15T23:27:27.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31fa056740 1 -- 172.21.15.89:0/1147449731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31e0018070 con 0x563f89a6cdc0 2024-09-15T23:27:27.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31d97fa640 1 -- 172.21.15.89:0/1147449731 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f31ec005600 con 0x563f89a6cdc0 2024-09-15T23:27:27.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31fa056740 10 client.16626 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:27.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31fa056740 1 -- 172.21.15.89:0/1147449731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563f89a85c20 con 0x563f89a6cdc0 2024-09-15T23:27:27.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31d97fa640 1 -- 172.21.15.89:0/1147449731 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31e0018070 con 0x563f89a6cdc0 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.463+0000 7f31d97fa640 1 -- 172.21.15.89:0/1147449731 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f31ec010b20 con 0x563f89a6cdc0 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.464+0000 7f31fa056740 10 client.16626 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.464+0000 7f31fa056740 10 client.16626 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.464+0000 7f31fa056740 20 client.16626 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.464+0000 7f31fa056740 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563f89a87130 0x563f89a895d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.464+0000 7f31fa056740 4 client.16626 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.464+0000 7f31fa056740 1 -- 172.21.15.89:0/1147449731 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f899df850 con 0x563f89a87130 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.464+0000 7f31f5218640 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563f89a87130 0x563f89a895d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.464+0000 7f31d97fa640 10 client.16626 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.464+0000 7f31f5218640 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563f89a87130 0x563f89a895d0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e021d1fc183e5ee2 server_cookie=486a65b1e0cccc55 in_seq=0 out_seq=0 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31d97fa640 1 -- 172.21.15.89:0/1147449731 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563f899df850 con 0x563f89a87130 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31d97fa640 10 client.16626 handle_command_reply: tid=0 2024-09-15T23:27:27.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 client.16626 shutdown 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 -- 172.21.15.89:0/1147449731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6cdc0 msgr2=0x563f89a741b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6cdc0 0x563f89a741b0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00bf10 tx=0x7f31ec012820 comp rx=0 tx=0).stop 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 -- 172.21.15.89:0/1147449731 shutdown_connections 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563f89a87130 0x563f89a895d0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a77680 0x563f89a746f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6cdc0 0x563f89a741b0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 --2- 172.21.15.89:0/1147449731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a6c4a0 0x563f89a76f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31d97fa640 0 client.16626 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 -- 172.21.15.89:0/1147449731 >> 172.21.15.89:0/1147449731 conn(0x563f89a68a30 msgr2=0x563f89a6e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 -- 172.21.15.89:0/1147449731 shutdown_connections 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 1 -- 172.21.15.89:0/1147449731 wait complete. 2024-09-15T23:27:27.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.465+0000 7f31fa056740 20 client.16626 trim_cache size 0 max 16384 2024-09-15T23:27:27.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.482+0000 7f31fa056740 1 Processor -- start 2024-09-15T23:27:27.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.482+0000 7f31fa056740 1 -- start start 2024-09-15T23:27:27.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.482+0000 7f31fa056740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 0x563f89a76120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.482+0000 7f31fa056740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a8eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31fa056740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a8f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31fa056740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01b0fa0 con 0x563f89a6c4a0 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31fa056740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01b2ae0 con 0x563f89a68a30 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31fa056740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01b1670 con 0x563f89a6cdc0 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5a19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a8eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5a19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a8eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37410/0 (socket says 172.21.15.89:37410) 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5a19640 1 -- 172.21.15.89:0/3556028907 learned_addr learned my addr 172.21.15.89:0/3556028907 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5218640 1 --2- 172.21.15.89:0/3556028907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a8f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5218640 1 --2- 172.21.15.89:0/3556028907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 0x563f89a76120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5a19640 1 -- 172.21.15.89:0/3556028907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 msgr2=0x563f89a8f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5a19640 1 --2- 172.21.15.89:0/3556028907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a8f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5a19640 1 -- 172.21.15.89:0/3556028907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 msgr2=0x563f89a76120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:27.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5a19640 1 --2- 172.21.15.89:0/3556028907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 0x563f89a76120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5a19640 1 -- 172.21.15.89:0/3556028907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f89a6cbf0 con 0x563f89a6c4a0 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5218640 1 --2- 172.21.15.89:0/3556028907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 0x563f89a76120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31f5a19640 1 --2- 172.21.15.89:0/3556028907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a8eee0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f31ec010fc0 tx=0x7f31ec012da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb19c52a45876b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31db7fe640 1 -- 172.21.15.89:0/3556028907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec02f070 con 0x563f89a6c4a0 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31db7fe640 1 -- 172.21.15.89:0/3556028907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31ec01d7f0 con 0x563f89a6c4a0 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.483+0000 7f31db7fe640 1 -- 172.21.15.89:0/3556028907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec0109f0 con 0x563f89a6c4a0 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 -- 172.21.15.89:0/3556028907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 msgr2=0x563f89a8eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 --2- 172.21.15.89:0/3556028907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a8eee0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f31ec010fc0 tx=0x7f31ec012da0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 -- 172.21.15.89:0/3556028907 shutdown_connections 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 --2- 172.21.15.89:0/3556028907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a8f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 --2- 172.21.15.89:0/3556028907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a8eee0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 --2- 172.21.15.89:0/3556028907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 0x563f89a76120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 -- 172.21.15.89:0/3556028907 >> 172.21.15.89:0/3556028907 conn(0x563f89a74850 msgr2=0x563f89a75a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:27.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 -- 172.21.15.89:0/3556028907 shutdown_connections 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 -- 172.21.15.89:0/3556028907 wait complete. 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 -- start start 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.484+0000 7f31fa056740 1 Processor -- start 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31fa056740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 0x563f89a7fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31fa056740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a7ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31fa056740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a7cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31fa056740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01b2410 con 0x563f89a6c4a0 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31fa056740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01b1d40 con 0x563f89a68a30 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31fa056740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f31f01aef80 con 0x563f89a6cdc0 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31fa056740 10 client.0 _ll_register_callbacks cb 0x7f31ec00bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31f5a19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a7ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31f5a19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a7ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37412/0 (socket says 172.21.15.89:37412) 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31f5a19640 1 -- 172.21.15.89:0/3692275148 learned_addr learned my addr 172.21.15.89:0/3692275148 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31f5218640 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 0x563f89a7fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31f5218640 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a7cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31f5a19640 1 -- 172.21.15.89:0/3692275148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 msgr2=0x563f89a7cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31f5a19640 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a7cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31f5a19640 1 -- 172.21.15.89:0/3692275148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 msgr2=0x563f89a7fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31f5a19640 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 0x563f89a7fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.485+0000 7f31f5a19640 1 -- 172.21.15.89:0/3692275148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f89a7d7e0 con 0x563f89a6c4a0 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31f5a19640 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a7ff80 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0057c0 tx=0x7f31ec000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf65cc586b02f83 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31d97fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31d97fa640 1 -- 172.21.15.89:0/3692275148 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec02f040 con 0x563f89a6c4a0 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31d97fa640 1 -- 172.21.15.89:0/3692275148 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f31ec004430 con 0x563f89a6c4a0 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31fa056740 1 -- 172.21.15.89:0/3692275148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31ec01d7f0 con 0x563f89a6c4a0 2024-09-15T23:27:27.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31d97fa640 1 -- 172.21.15.89:0/3692275148 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f31ec041050 con 0x563f89a6c4a0 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31fa056740 10 client.16632 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31fa056740 1 -- 172.21.15.89:0/3692275148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563f89af1ac0 con 0x563f89a6c4a0 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31d97fa640 1 -- 172.21.15.89:0/3692275148 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31ec01d7f0 con 0x563f89a6c4a0 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31d97fa640 1 -- 172.21.15.89:0/3692275148 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f31ec03c030 con 0x563f89a6c4a0 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31fa056740 10 client.16632 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31fa056740 10 client.16632 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31fa056740 20 client.16632 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31fa056740 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563f89a77680 0x563f89af4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31fa056740 4 client.16632 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.486+0000 7f31fa056740 1 -- 172.21.15.89:0/3692275148 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563f899d9d60 con 0x563f89a77680 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31f5218640 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563f89a77680 0x563f89af4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31d97fa640 10 client.16632 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:27.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31f5218640 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563f89a77680 0x563f89af4150 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce3181c4efbab5f0 server_cookie=48e8dbc03bc416e2 in_seq=0 out_seq=0 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31d97fa640 1 -- 172.21.15.89:0/3692275148 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563f899d9d60 con 0x563f89a77680 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31d97fa640 10 client.16632 handle_command_reply: tid=0 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31fa056740 1 client.16632 shutdown 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31fa056740 1 -- 172.21.15.89:0/3692275148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 msgr2=0x563f89a7ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31fa056740 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a7ff80 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0057c0 tx=0x7f31ec000bc0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31fa056740 1 -- 172.21.15.89:0/3692275148 shutdown_connections 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31fa056740 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563f89a77680 0x563f89af4150 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31fa056740 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563f89a6cdc0 0x563f89a7cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31fa056740 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f89a6c4a0 0x563f89a7ff80 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31d97fa640 0 client.16632 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31fa056740 1 --2- 172.21.15.89:0/3692275148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563f89a68a30 0x563f89a7fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.487+0000 7f31fa056740 1 -- 172.21.15.89:0/3692275148 >> 172.21.15.89:0/3692275148 conn(0x563f89a74850 msgr2=0x563f89a72650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:27.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31fa056740 1 -- 172.21.15.89:0/3692275148 shutdown_connections 2024-09-15T23:27:27.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31fa056740 1 -- 172.21.15.89:0/3692275148 wait complete. 2024-09-15T23:27:27.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31fa056740 20 client.16632 trim_cache size 0 max 16384 2024-09-15T23:27:27.500 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:27:27.500 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:27:27.500 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:27:27.500 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:27:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 -- 172.21.15.89:0/1828756668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31dc0a3240 msgr2=0x7f31dc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31dc0a3240 0x7f31dc0a5700 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f31e80b9860 tx=0x7f31e808fd10 comp rx=0 tx=0).stop 2024-09-15T23:27:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 -- 172.21.15.89:0/1828756668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0102ef0 msgr2=0x7f31f00fd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0102ef0 0x7f31f00fd700 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f31e802ae70 tx=0x7f31e802f5d0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 -- 172.21.15.89:0/1828756668 shutdown_connections 2024-09-15T23:27:27.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31dc0a3240 0x7f31dc0a5700 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31f0102ef0 0x7f31f00fd700 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31f0102570 0x7f31f00fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 --2- 172.21.15.89:0/1828756668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31f0100e20 0x7f31f00fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 -- 172.21.15.89:0/1828756668 >> 172.21.15.89:0/1828756668 conn(0x7f31f00fc7d0 msgr2=0x7f31f006a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 -- 172.21.15.89:0/1828756668 shutdown_connections 2024-09-15T23:27:27.502 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:27.488+0000 7f31f7c7d640 1 -- 172.21.15.89:0/1828756668 wait complete. 2024-09-15T23:27:28.349 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:28 smithi089 ceph-mon[30933]: pgmap v453: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.7 KiB/s wr, 14 op/s 2024-09-15T23:27:28.551 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:28 smithi174 ceph-mon[35586]: pgmap v453: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.7 KiB/s wr, 14 op/s 2024-09-15T23:27:28.551 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:28 smithi120 ceph-mon[35250]: pgmap v453: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 4.7 KiB/s wr, 14 op/s 2024-09-15T23:27:28.552 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:27:28.552 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:27:28.815 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:29.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.322+0000 7f5867ae7640 1 -- 172.21.15.89:0/3992776481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5860101420 msgr2=0x7f586010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.322+0000 7f5867ae7640 1 --2- 172.21.15.89:0/3992776481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5860101420 0x7f586010d950 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f5858009cd0 tx=0x7f585802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:27:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.322+0000 7f5867ae7640 1 -- 172.21.15.89:0/3992776481 shutdown_connections 2024-09-15T23:27:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.322+0000 7f5867ae7640 1 --2- 172.21.15.89:0/3992776481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5860101420 0x7f586010d950 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.322+0000 7f5867ae7640 1 --2- 172.21.15.89:0/3992776481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5860100a80 0x7f5860100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.322+0000 7f5867ae7640 1 --2- 172.21.15.89:0/3992776481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5860106a80 0x7f5860106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.322+0000 7f5867ae7640 1 -- 172.21.15.89:0/3992776481 >> 172.21.15.89:0/3992776481 conn(0x7f58600fc7a0 msgr2=0x7f58600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.322+0000 7f5867ae7640 1 -- 172.21.15.89:0/3992776481 shutdown_connections 2024-09-15T23:27:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.322+0000 7f5867ae7640 1 -- 172.21.15.89:0/3992776481 wait complete. 2024-09-15T23:27:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5867ae7640 1 Processor -- start 2024-09-15T23:27:29.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5867ae7640 1 -- start start 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5867ae7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5860100a80 0x7f5860198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5867ae7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5860101420 0x7f58601988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5867ae7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5860106a80 0x7f586019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5867ae7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5860110e80 con 0x7f5860101420 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5867ae7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5860110d00 con 0x7f5860100a80 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5867ae7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5860111000 con 0x7f5860106a80 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5865082640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5860101420 0x7f58601988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5865082640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5860101420 0x7f58601988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37432/0 (socket says 172.21.15.89:37432) 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5865082640 1 -- 172.21.15.89:0/1863039401 learned_addr learned my addr 172.21.15.89:0/1863039401 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5866084640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5860106a80 0x7f586019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5865883640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5860100a80 0x7f5860198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5865082640 1 -- 172.21.15.89:0/1863039401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5860106a80 msgr2=0x7f586019cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:29.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5865082640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5860106a80 0x7f586019cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5865082640 1 -- 172.21.15.89:0/1863039401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5860100a80 msgr2=0x7f5860198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5865082640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5860100a80 0x7f5860198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.323+0000 7f5865082640 1 -- 172.21.15.89:0/1863039401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586019d330 con 0x7f5860101420 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.324+0000 7f5865883640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5860100a80 0x7f5860198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.324+0000 7f5865082640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5860101420 0x7f58601988c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f585000d950 tx=0x7f585000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.324+0000 7f5856ffd640 1 -- 172.21.15.89:0/1863039401 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850017070 con 0x7f5860101420 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.324+0000 7f5856ffd640 1 -- 172.21.15.89:0/1863039401 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5850009c30 con 0x7f5860101420 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.324+0000 7f5856ffd640 1 -- 172.21.15.89:0/1863039401 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585000f6b0 con 0x7f5860101420 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.324+0000 7f5867ae7640 1 -- 172.21.15.89:0/1863039401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586019d620 con 0x7f5860101420 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.324+0000 7f5867ae7640 1 -- 172.21.15.89:0/1863039401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58601a4f00 con 0x7f5860101420 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.325+0000 7f5867ae7640 1 -- 172.21.15.89:0/1863039401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5860100ee0 con 0x7f5860101420 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.326+0000 7f5856ffd640 1 -- 172.21.15.89:0/1863039401 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f58500109d0 con 0x7f5860101420 2024-09-15T23:27:29.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.326+0000 7f5856ffd640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f584c0a32c0 0x7f584c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:29.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.327+0000 7f5865883640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f584c0a32c0 0x7f584c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:29.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.327+0000 7f5856ffd640 1 -- 172.21.15.89:0/1863039401 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f58500cf9a0 con 0x7f5860101420 2024-09-15T23:27:29.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.327+0000 7f5865883640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f584c0a32c0 0x7f584c0a5780 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f585c00baa0 tx=0x7f585c005d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:29.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.331+0000 7f5856ffd640 1 -- 172.21.15.89:0/1863039401 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5850009de0 con 0x7f5860101420 2024-09-15T23:27:29.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.529+0000 7f5867ae7640 1 -- 172.21.15.89:0/1863039401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58601a51a0 con 0x7f5860101420 2024-09-15T23:27:29.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.530+0000 7f5856ffd640 1 -- 172.21.15.89:0/1863039401 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f585009c2d0 con 0x7f5860101420 2024-09-15T23:27:29.532 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:27:29.532 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:27:29.532 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.532+0000 7f5867ae7640 1 -- 172.21.15.89:0/1863039401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f584c0a32c0 msgr2=0x7f584c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.532+0000 7f5867ae7640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f584c0a32c0 0x7f584c0a5780 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f585c00baa0 tx=0x7f585c005d20 comp rx=0 tx=0).stop 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.532+0000 7f5867ae7640 1 -- 172.21.15.89:0/1863039401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5860101420 msgr2=0x7f58601988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.532+0000 7f5867ae7640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5860101420 0x7f58601988c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f585000d950 tx=0x7f585000de10 comp rx=0 tx=0).stop 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.532+0000 7f5867ae7640 1 -- 172.21.15.89:0/1863039401 shutdown_connections 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.532+0000 7f5867ae7640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f584c0a32c0 0x7f584c0a5780 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.532+0000 7f5867ae7640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5860106a80 0x7f586019cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.532+0000 7f5867ae7640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5860101420 0x7f58601988c0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.532+0000 7f5867ae7640 1 --2- 172.21.15.89:0/1863039401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5860100a80 0x7f5860198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.532+0000 7f5867ae7640 1 -- 172.21.15.89:0/1863039401 >> 172.21.15.89:0/1863039401 conn(0x7f58600fc7a0 msgr2=0x7f5860104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.533+0000 7f5867ae7640 1 -- 172.21.15.89:0/1863039401 shutdown_connections 2024-09-15T23:27:29.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:29.533+0000 7f5867ae7640 1 -- 172.21.15.89:0/1863039401 wait complete. 2024-09-15T23:27:30.290 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:27:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:30 smithi120 ceph-mon[35250]: pgmap v454: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-09-15T23:27:30.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:30 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1863039401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:30.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:30 smithi174 ceph-mon[35586]: pgmap v454: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-09-15T23:27:30.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:30 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1863039401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:30.537 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:30.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:30 smithi089 ceph-mon[30933]: pgmap v454: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-09-15T23:27:30.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:30 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1863039401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:31.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.036+0000 7fbd9ef23640 1 -- 172.21.15.89:0/3568484793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd98106f10 msgr2=0x7fbd9810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.036+0000 7fbd9ef23640 1 --2- 172.21.15.89:0/3568484793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd98106f10 0x7fbd9810ea90 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fbd90009ca0 tx=0x7fbd9002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.036+0000 7fbd9ef23640 1 -- 172.21.15.89:0/3568484793 shutdown_connections 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.036+0000 7fbd9ef23640 1 --2- 172.21.15.89:0/3568484793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd98106f10 0x7fbd9810ea90 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.036+0000 7fbd9ef23640 1 --2- 172.21.15.89:0/3568484793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd98102790 0x7fbd981067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.036+0000 7fbd9ef23640 1 --2- 172.21.15.89:0/3568484793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd98101de0 0x7fbd981021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.036+0000 7fbd9ef23640 1 -- 172.21.15.89:0/3568484793 >> 172.21.15.89:0/3568484793 conn(0x7fbd980fd8c0 msgr2=0x7fbd980ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.036+0000 7fbd9ef23640 1 -- 172.21.15.89:0/3568484793 shutdown_connections 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.036+0000 7fbd9ef23640 1 -- 172.21.15.89:0/3568484793 wait complete. 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ef23640 1 Processor -- start 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ef23640 1 -- start start 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ef23640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd98101de0 0x7fbd98198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ef23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd98102790 0x7fbd981989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ef23640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd98106f10 0x7fbd9819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ef23640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbd98111f90 con 0x7fbd98102790 2024-09-15T23:27:31.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ef23640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbd98111e10 con 0x7fbd98101de0 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ef23640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbd98112110 con 0x7fbd98106f10 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd97fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd98102790 0x7fbd981989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd97fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd98102790 0x7fbd981989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37438/0 (socket says 172.21.15.89:37438) 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd97fff640 1 -- 172.21.15.89:0/1590040198 learned_addr learned my addr 172.21.15.89:0/1590040198 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd98101de0 0x7fbd98198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd98106f10 0x7fbd9819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd97fff640 1 -- 172.21.15.89:0/1590040198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd98106f10 msgr2=0x7fbd9819cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd97fff640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd98106f10 0x7fbd9819cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd97fff640 1 -- 172.21.15.89:0/1590040198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd98101de0 msgr2=0x7fbd98198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd97fff640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd98101de0 0x7fbd98198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd97fff640 1 -- 172.21.15.89:0/1590040198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9819d420 con 0x7fbd98102790 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd98101de0 0x7fbd98198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:31.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd97fff640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd98102790 0x7fbd981989b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbd8400b4a0 tx=0x7fbd8400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd95ffb640 1 -- 172.21.15.89:0/1590040198 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84014070 con 0x7fbd98102790 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd95ffb640 1 -- 172.21.15.89:0/1590040198 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd84010af0 con 0x7fbd98102790 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.037+0000 7fbd9ef23640 1 -- 172.21.15.89:0/1590040198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd9819d710 con 0x7fbd98102790 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.038+0000 7fbd95ffb640 1 -- 172.21.15.89:0/1590040198 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84017d10 con 0x7fbd98102790 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.038+0000 7fbd9ef23640 1 -- 172.21.15.89:0/1590040198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd981a4ff0 con 0x7fbd98102790 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd95ffb640 1 -- 172.21.15.89:0/1590040198 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbd84017070 con 0x7fbd98102790 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbda12fc740 1 Processor -- start 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbda12fc740 1 -- start start 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f320b40 0x556b2f31f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f31f5d0 0x556b2f31fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 0x556b2f330b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbda12fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556b2f2852f0 con 0x556b2f320b40 2024-09-15T23:27:31.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbda12fc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556b2f22ccf0 con 0x556b2f31f5d0 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbda12fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556b2f2812b0 con 0x556b2f32e780 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd9d4c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f320b40 0x556b2f31f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd9d4c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f320b40 0x556b2f31f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37442/0 (socket says 172.21.15.89:37442) 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd9d4c0640 1 -- 172.21.15.89:0/3700722577 learned_addr learned my addr 172.21.15.89:0/3700722577 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd97fff640 1 --2- 172.21.15.89:0/3700722577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 0x556b2f330b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd95ffb640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd880a3240 0x7fbd880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/3700722577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f31f5d0 0x556b2f31fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd9d4c0640 1 -- 172.21.15.89:0/3700722577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 msgr2=0x556b2f330b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/3700722577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 0x556b2f330b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd9d4c0640 1 -- 172.21.15.89:0/3700722577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f31f5d0 msgr2=0x556b2f31fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/3700722577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f31f5d0 0x556b2f31fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.041+0000 7fbd9d4c0640 1 -- 172.21.15.89:0/3700722577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b2f2113a0 con 0x556b2f320b40 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.042+0000 7fbd97fff640 1 --2- 172.21.15.89:0/3700722577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 0x556b2f330b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.042+0000 7fbd95ffb640 1 -- 172.21.15.89:0/1590040198 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbd840cf180 con 0x7fbd98102790 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.042+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd880a3240 0x7fbd880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.042+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/3700722577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f320b40 0x556b2f31f090 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fbd90005cc0 tx=0x7fbd90031bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb864c346dffcdc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.042+0000 7fbd82ffd640 1 -- 172.21.15.89:0/3700722577 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd90041040 con 0x556b2f320b40 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.042+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd880a3240 0x7fbd880a5700 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbd9004c000 tx=0x7fbd9004d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.042+0000 7fbd82ffd640 1 -- 172.21.15.89:0/3700722577 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd90036070 con 0x556b2f320b40 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.042+0000 7fbd82ffd640 1 -- 172.21.15.89:0/3700722577 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9002fad0 con 0x556b2f320b40 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 -- 172.21.15.89:0/3700722577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f320b40 msgr2=0x556b2f31f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 --2- 172.21.15.89:0/3700722577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f320b40 0x556b2f31f090 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fbd90005cc0 tx=0x7fbd90031bb0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 -- 172.21.15.89:0/3700722577 shutdown_connections 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 --2- 172.21.15.89:0/3700722577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 0x556b2f330b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 --2- 172.21.15.89:0/3700722577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f31f5d0 0x556b2f31fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 --2- 172.21.15.89:0/3700722577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f320b40 0x556b2f31f090 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 -- 172.21.15.89:0/3700722577 >> 172.21.15.89:0/3700722577 conn(0x556b2f2e8310 msgr2=0x556b2f31bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 -- 172.21.15.89:0/3700722577 shutdown_connections 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 -- 172.21.15.89:0/3700722577 wait complete. 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 -- start start 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.043+0000 7fbda12fc740 1 Processor -- start 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f31f5d0 0x556b2f32e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f320b40 0x556b2f32b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f32e780 0x556b2f32ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbda12fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556b2f331480 con 0x556b2f32e780 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbda12fc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556b2f240060 con 0x556b2f31f5d0 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbda12fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556b2f331600 con 0x556b2f320b40 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbda12fc740 10 client.0 _ll_register_callbacks cb 0x556b2ef938c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd9d4c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f32e780 0x556b2f32ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd9d4c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f32e780 0x556b2f32ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37448/0 (socket says 172.21.15.89:37448) 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd9d4c0640 1 -- 172.21.15.89:0/4241048606 learned_addr learned my addr 172.21.15.89:0/4241048606 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f31f5d0 0x556b2f32e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd97fff640 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f320b40 0x556b2f32b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd9d4c0640 1 -- 172.21.15.89:0/4241048606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f320b40 msgr2=0x556b2f32b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f320b40 0x556b2f32b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd9d4c0640 1 -- 172.21.15.89:0/4241048606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f31f5d0 msgr2=0x556b2f32e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f31f5d0 0x556b2f32e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd9d4c0640 1 -- 172.21.15.89:0/4241048606 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b2eff63c0 con 0x556b2f32e780 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd77fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f32e780 0x556b2f32ba20 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbd90005f20 tx=0x7fbd9002f6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a688393958ab6d7c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd77fff640 1 -- 172.21.15.89:0/4241048606 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd90059060 con 0x556b2f32e780 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbd77fff640 1 -- 172.21.15.89:0/4241048606 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbd90003fe0 con 0x556b2f32e780 2024-09-15T23:27:31.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.044+0000 7fbda12fc740 1 -- 172.21.15.89:0/4241048606 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd90036070 con 0x556b2f32e780 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbd77fff640 1 -- 172.21.15.89:0/4241048606 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbd90041040 con 0x556b2f32e780 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbda12fc740 10 client.16650 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbda12fc740 1 -- 172.21.15.89:0/4241048606 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556b2f331480 con 0x556b2f32e780 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbd77fff640 1 -- 172.21.15.89:0/4241048606 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd90036070 con 0x556b2f32e780 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbd77fff640 1 -- 172.21.15.89:0/4241048606 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbd90030cd0 con 0x556b2f32e780 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbda12fc740 10 client.16650 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbda12fc740 10 client.16650 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbda12fc740 20 client.16650 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbda12fc740 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556b2f33e0a0 0x556b2f340490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbda12fc740 4 client.16650 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbda12fc740 1 -- 172.21.15.89:0/4241048606 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b2f292860 con 0x556b2f33e0a0 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.045+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556b2f33e0a0 0x556b2f340490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.046+0000 7fbd77fff640 10 client.16650 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:27:31.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.046+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556b2f33e0a0 0x556b2f340490 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=750b62b9a8d1366c server_cookie=9e2b5dcbe98aeec1 in_seq=0 out_seq=0 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.046+0000 7fbd77fff640 1 -- 172.21.15.89:0/4241048606 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556b2f292860 con 0x556b2f33e0a0 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.046+0000 7fbd77fff640 10 client.16650 handle_command_reply: tid=0 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 client.16650 shutdown 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 -- 172.21.15.89:0/4241048606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f32e780 msgr2=0x556b2f32ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f32e780 0x556b2f32ba20 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbd90005f20 tx=0x7fbd9002f6b0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 -- 172.21.15.89:0/4241048606 shutdown_connections 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556b2f33e0a0 0x556b2f340490 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f32e780 0x556b2f32ba20 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f320b40 0x556b2f32b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbd77fff640 0 client.16650 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 --2- 172.21.15.89:0/4241048606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f31f5d0 0x556b2f32e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 -- 172.21.15.89:0/4241048606 >> 172.21.15.89:0/4241048606 conn(0x556b2f2e8310 msgr2=0x556b2f321930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:31.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 -- 172.21.15.89:0/4241048606 shutdown_connections 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 1 -- 172.21.15.89:0/4241048606 wait complete. 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.047+0000 7fbda12fc740 20 client.16650 trim_cache size 0 max 16384 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbda12fc740 1 Processor -- start 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbda12fc740 1 -- start start 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f31be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f320b40 0x556b2f33a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f32e780 0x556b2f345c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbda12fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981acba0 con 0x556b2f31f5d0 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbda12fc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981ae6e0 con 0x556b2f32e780 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbda12fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981ad270 con 0x556b2f320b40 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbd9ccbf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f31be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbd9ccbf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f31be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37456/0 (socket says 172.21.15.89:37456) 2024-09-15T23:27:31.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbd9ccbf640 1 -- 172.21.15.89:0/3908378705 learned_addr learned my addr 172.21.15.89:0/3908378705 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/3908378705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f32e780 0x556b2f345c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbd9ccbf640 1 -- 172.21.15.89:0/3908378705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f320b40 msgr2=0x556b2f33a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/3908378705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f320b40 0x556b2f33a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbd9ccbf640 1 -- 172.21.15.89:0/3908378705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f32e780 msgr2=0x556b2f345c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/3908378705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f32e780 0x556b2f345c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbd9ccbf640 1 -- 172.21.15.89:0/3908378705 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b2f329980 con 0x556b2f31f5d0 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.060+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/3908378705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f31be10 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0058c0 tx=0x7fbd8c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd01fd096712746f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbd827fc640 1 -- 172.21.15.89:0/3908378705 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c01e420 con 0x556b2f31f5d0 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbd827fc640 1 -- 172.21.15.89:0/3908378705 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd8c01eaa0 con 0x556b2f31f5d0 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbd827fc640 1 -- 172.21.15.89:0/3908378705 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c012e40 con 0x556b2f31f5d0 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbda12fc740 1 -- 172.21.15.89:0/3908378705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 msgr2=0x556b2f31be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbda12fc740 1 --2- 172.21.15.89:0/3908378705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f31be10 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0058c0 tx=0x7fbd8c0039c0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbda12fc740 1 -- 172.21.15.89:0/3908378705 shutdown_connections 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbda12fc740 1 --2- 172.21.15.89:0/3908378705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f32e780 0x556b2f345c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbda12fc740 1 --2- 172.21.15.89:0/3908378705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f320b40 0x556b2f33a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbda12fc740 1 --2- 172.21.15.89:0/3908378705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f31be10 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbda12fc740 1 -- 172.21.15.89:0/3908378705 >> 172.21.15.89:0/3908378705 conn(0x556b2f2e8310 msgr2=0x556b2f31b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbda12fc740 1 -- 172.21.15.89:0/3908378705 shutdown_connections 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.061+0000 7fbda12fc740 1 -- 172.21.15.89:0/3908378705 wait complete. 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbda12fc740 1 -- start start 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbda12fc740 1 Processor -- start 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f336df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f320b40 0x556b2f332120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbda12fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 0x556b2f332660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbda12fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981ae010 con 0x556b2f31f5d0 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbda12fc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981ad940 con 0x556b2f320b40 2024-09-15T23:27:31.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbda12fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbd981aab80 con 0x556b2f32e780 2024-09-15T23:27:31.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbda12fc740 10 client.0 _ll_register_callbacks cb 0x7fbd900098b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:31.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbd9ccbf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f336df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbd9ccbf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f336df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37472/0 (socket says 172.21.15.89:37472) 2024-09-15T23:27:31.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbd9ccbf640 1 -- 172.21.15.89:0/2974083943 learned_addr learned my addr 172.21.15.89:0/2974083943 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:31.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbd97fff640 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f320b40 0x556b2f332120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.102 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:27:31.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.062+0000 7fbd9d4c0640 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 0x556b2f332660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd9ccbf640 1 -- 172.21.15.89:0/2974083943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 msgr2=0x556b2f332660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 0x556b2f332660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd9ccbf640 1 -- 172.21.15.89:0/2974083943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f320b40 msgr2=0x556b2f332120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f320b40 0x556b2f332120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd9ccbf640 1 -- 172.21.15.89:0/2974083943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b2f332ca0 con 0x556b2f31f5d0 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd9ccbf640 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f336df0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00bfd0 tx=0x7fbd8c0040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc751818e56398d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd77fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd77fff640 1 -- 172.21.15.89:0/2974083943 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c012cc0 con 0x556b2f31f5d0 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbda12fc740 1 -- 172.21.15.89:0/2974083943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd8c01eaa0 con 0x556b2f31f5d0 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd77fff640 1 -- 172.21.15.89:0/2974083943 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbd8c012450 con 0x556b2f31f5d0 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd77fff640 1 -- 172.21.15.89:0/2974083943 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbd8c012e60 con 0x556b2f31f5d0 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbda12fc740 10 client.16656 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:31.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbda12fc740 1 -- 172.21.15.89:0/2974083943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556b2f3a8a80 con 0x556b2f31f5d0 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.063+0000 7fbd77fff640 1 -- 172.21.15.89:0/2974083943 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd8c01eaa0 con 0x556b2f31f5d0 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbd77fff640 1 -- 172.21.15.89:0/2974083943 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbd8c004700 con 0x556b2f31f5d0 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbda12fc740 10 client.16656 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbda12fc740 10 client.16656 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbda12fc740 20 client.16656 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbda12fc740 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556b2f3a90f0 0x556b2f3ab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbda12fc740 4 client.16656 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbda12fc740 1 -- 172.21.15.89:0/2974083943 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b2f28cd70 con 0x556b2f3a90f0 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbd97fff640 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556b2f3a90f0 0x556b2f3ab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbd77fff640 10 client.16656 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbd97fff640 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556b2f3a90f0 0x556b2f3ab4e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=818792b0d14308ed server_cookie=cca1ecd003004684 in_seq=0 out_seq=0 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbd77fff640 1 -- 172.21.15.89:0/2974083943 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b2f28cd70 con 0x556b2f3a90f0 2024-09-15T23:27:31.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbd77fff640 10 client.16656 handle_command_reply: tid=0 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.064+0000 7fbda12fc740 1 client.16656 shutdown 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 1 -- 172.21.15.89:0/2974083943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 msgr2=0x556b2f336df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f336df0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00bfd0 tx=0x7fbd8c0040f0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 1 -- 172.21.15.89:0/2974083943 shutdown_connections 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556b2f3a90f0 0x556b2f3ab4e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b2f32e780 0x556b2f332660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b2f320b40 0x556b2f332120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 1 --2- 172.21.15.89:0/2974083943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b2f31f5d0 0x556b2f336df0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd77fff640 0 client.16656 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 1 -- 172.21.15.89:0/2974083943 >> 172.21.15.89:0/2974083943 conn(0x556b2f2e8310 msgr2=0x556b2f329d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 1 -- 172.21.15.89:0/2974083943 shutdown_connections 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 1 -- 172.21.15.89:0/2974083943 wait complete. 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbda12fc740 20 client.16656 trim_cache size 0 max 16384 2024-09-15T23:27:31.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd9ef23640 1 -- 172.21.15.89:0/1590040198 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd880a3240 msgr2=0x7fbd880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd9ef23640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd880a3240 0x7fbd880a5700 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbd9004c000 tx=0x7fbd9004d000 comp rx=0 tx=0).stop 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd9ef23640 1 -- 172.21.15.89:0/1590040198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd98102790 msgr2=0x7fbd981989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd9ef23640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd98102790 0x7fbd981989b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fbd8400b4a0 tx=0x7fbd8400b960 comp rx=0 tx=0).stop 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd9ef23640 1 -- 172.21.15.89:0/1590040198 shutdown_connections 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd9ef23640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbd880a3240 0x7fbd880a5700 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd9ef23640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd98106f10 0x7fbd9819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd9ef23640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbd98102790 0x7fbd981989b0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd9ef23640 1 --2- 172.21.15.89:0/1590040198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbd98101de0 0x7fbd98198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.065+0000 7fbd9ef23640 1 -- 172.21.15.89:0/1590040198 >> 172.21.15.89:0/1590040198 conn(0x7fbd980fd8c0 msgr2=0x7fbd980fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.066+0000 7fbd9ef23640 1 -- 172.21.15.89:0/1590040198 shutdown_connections 2024-09-15T23:27:31.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:31.066+0000 7fbd9ef23640 1 -- 172.21.15.89:0/1590040198 wait complete. 2024-09-15T23:27:31.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:31 smithi089 ceph-mon[30933]: pgmap v455: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-09-15T23:27:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:31 smithi120 ceph-mon[35250]: pgmap v455: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-09-15T23:27:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:31 smithi174 ceph-mon[35586]: pgmap v455: 129 pgs: 129 active+clean; 1.5 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-09-15T23:27:31.907 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:27:31.978 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.974+0000 7f62b38d7640 1 -- 172.21.15.89:0/173885787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ac1025c0 msgr2=0x7f62ac1049b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.978 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.974+0000 7f62b38d7640 1 --2- 172.21.15.89:0/173885787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ac1025c0 0x7f62ac1049b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009900 tx=0x7f62a802a410 comp rx=0 tx=0).stop 2024-09-15T23:27:31.978 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.974+0000 7f62b38d7640 1 -- 172.21.15.89:0/173885787 shutdown_connections 2024-09-15T23:27:31.978 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.974+0000 7f62b38d7640 1 --2- 172.21.15.89:0/173885787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62ac107820 0x7f62ac109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.978 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.974+0000 7f62b38d7640 1 --2- 172.21.15.89:0/173885787 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ac104ef0 0x7f62ac1072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.978 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.974+0000 7f62b38d7640 1 --2- 172.21.15.89:0/173885787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ac1025c0 0x7f62ac1049b0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.978 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.974+0000 7f62b38d7640 1 -- 172.21.15.89:0/173885787 >> 172.21.15.89:0/173885787 conn(0x7f62ac0fc1e0 msgr2=0x7f62ac0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:31.978 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.974+0000 7f62b38d7640 1 -- 172.21.15.89:0/173885787 shutdown_connections 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.974+0000 7f62b38d7640 1 -- 172.21.15.89:0/173885787 wait complete. 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b38d7640 1 Processor -- start 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b38d7640 1 -- start start 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b38d7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ac1025c0 0x7f62ac101f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b38d7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62ac104ef0 0x7f62ac100570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b38d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ac107820 0x7f62ac100ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b38d7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac10d820 con 0x7f62ac1025c0 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b38d7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac10d6a0 con 0x7f62ac104ef0 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b38d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac10d9a0 con 0x7f62ac107820 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b1673640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ac1025c0 0x7f62ac101f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b1673640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ac1025c0 0x7f62ac101f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37478/0 (socket says 172.21.15.89:37478) 2024-09-15T23:27:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b1673640 1 -- 172.21.15.89:0/3291657378 learned_addr learned my addr 172.21.15.89:0/3291657378 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.975+0000 7f62b0e72640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62ac104ef0 0x7f62ac100570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62b1673640 1 -- 172.21.15.89:0/3291657378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ac107820 msgr2=0x7f62ac100ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62b1e74640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ac107820 0x7f62ac100ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62b1673640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ac107820 0x7f62ac100ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62b1673640 1 -- 172.21.15.89:0/3291657378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62ac104ef0 msgr2=0x7f62ac100570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62b1673640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62ac104ef0 0x7f62ac100570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62b1673640 1 -- 172.21.15.89:0/3291657378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ac101340 con 0x7f62ac1025c0 2024-09-15T23:27:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62b0e72640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62ac104ef0 0x7f62ac100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62b1673640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ac1025c0 0x7f62ac101f30 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009230 tx=0x7f62a8004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:31.981 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62a27fc640 1 -- 172.21.15.89:0/3291657378 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a803f030 con 0x7f62ac1025c0 2024-09-15T23:27:31.981 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62a27fc640 1 -- 172.21.15.89:0/3291657378 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62a8003ce0 con 0x7f62ac1025c0 2024-09-15T23:27:31.981 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62b38d7640 1 -- 172.21.15.89:0/3291657378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ac1a1f60 con 0x7f62ac1025c0 2024-09-15T23:27:31.981 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62a27fc640 1 -- 172.21.15.89:0/3291657378 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8035070 con 0x7f62ac1025c0 2024-09-15T23:27:31.981 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.976+0000 7f62b38d7640 1 -- 172.21.15.89:0/3291657378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ac1a23c0 con 0x7f62ac1025c0 2024-09-15T23:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.978+0000 7f62a27fc640 1 -- 172.21.15.89:0/3291657378 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f62a8042030 con 0x7f62ac1025c0 2024-09-15T23:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b6670740 1 Processor -- start 2024-09-15T23:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b6670740 1 -- start start 2024-09-15T23:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b6670740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d199a0 0x558ca8d19da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b6670740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d1eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b6670740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1f370 0x558ca8d21830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b6670740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558ca8c7d2f0 con 0x558ca8d1a2e0 2024-09-15T23:27:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b6670740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ca8c25270 con 0x558ca8d199a0 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b6670740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558ca8c792b0 con 0x558ca8d1f370 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b1673640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d1eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b1673640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d1eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37484/0 (socket says 172.21.15.89:37484) 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b1673640 1 -- 172.21.15.89:0/1696916191 learned_addr learned my addr 172.21.15.89:0/1696916191 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b1e74640 1 --2- 172.21.15.89:0/1696916191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d199a0 0x558ca8d19da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62a27fc640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62980a3370 0x7f62980a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b0e72640 1 --2- 172.21.15.89:0/1696916191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1f370 0x558ca8d21830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b1673640 1 -- 172.21.15.89:0/1696916191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1f370 msgr2=0x558ca8d21830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b1673640 1 --2- 172.21.15.89:0/1696916191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1f370 0x558ca8d21830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b1673640 1 -- 172.21.15.89:0/1696916191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d199a0 msgr2=0x558ca8d19da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b1e74640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62980a3370 0x7f62980a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b1673640 1 --2- 172.21.15.89:0/1696916191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d199a0 0x558ca8d19da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.979+0000 7f62b1673640 1 -- 172.21.15.89:0/1696916191 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ca8c24340 con 0x558ca8d1a2e0 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.980+0000 7f62a27fc640 1 -- 172.21.15.89:0/3291657378 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f62a80f06d0 con 0x7f62ac1025c0 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.980+0000 7f62b1673640 1 --2- 172.21.15.89:0/1696916191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d1eda0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f62a80b8d10 tx=0x7f62a804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766f27eb9d90ce4d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.980+0000 7f62b1e74640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62980a3370 0x7f62980a5830 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f62ac06a980 tx=0x7f62a4004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.980+0000 7f62977fe640 1 -- 172.21.15.89:0/1696916191 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a809e880 con 0x558ca8d1a2e0 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.980+0000 7f62977fe640 1 -- 172.21.15.89:0/1696916191 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62a80c0cb0 con 0x558ca8d1a2e0 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.980+0000 7f62977fe640 1 -- 172.21.15.89:0/1696916191 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a80c5570 con 0x558ca8d1a2e0 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 -- 172.21.15.89:0/1696916191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 msgr2=0x558ca8d1eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 --2- 172.21.15.89:0/1696916191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d1eda0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f62a80b8d10 tx=0x7f62a804c000 comp rx=0 tx=0).stop 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 -- 172.21.15.89:0/1696916191 shutdown_connections 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 --2- 172.21.15.89:0/1696916191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1f370 0x558ca8d21830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 --2- 172.21.15.89:0/1696916191 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d1eda0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 --2- 172.21.15.89:0/1696916191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d199a0 0x558ca8d19da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 -- 172.21.15.89:0/1696916191 >> 172.21.15.89:0/1696916191 conn(0x558ca8d15010 msgr2=0x558ca8d15410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 -- 172.21.15.89:0/1696916191 shutdown_connections 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 -- 172.21.15.89:0/1696916191 wait complete. 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 -- start start 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.981+0000 7f62b6670740 1 Processor -- start 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b6670740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d199a0 0x558ca8d23840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b6670740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 0x558ca8d23d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b6670740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 0x558ca8d2adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b6670740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558ca8d22100 con 0x558ca8d199a0 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b6670740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ca8c06290 con 0x558ca8d1f370 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b6670740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558ca8d22280 con 0x558ca8d1a2e0 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b6670740 10 client.0 _ll_register_callbacks cb 0x558ca8c37140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b1673640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d199a0 0x558ca8d23840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b1673640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d199a0 0x558ca8d23840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37498/0 (socket says 172.21.15.89:37498) 2024-09-15T23:27:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b1673640 1 -- 172.21.15.89:0/4089761732 learned_addr learned my addr 172.21.15.89:0/4089761732 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b0e72640 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 0x558ca8d23d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b1e74640 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 0x558ca8d2adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b1673640 1 -- 172.21.15.89:0/4089761732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 msgr2=0x558ca8d23d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b1673640 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 0x558ca8d23d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b1673640 1 -- 172.21.15.89:0/4089761732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 msgr2=0x558ca8d2adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b1673640 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 0x558ca8d2adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b1673640 1 -- 172.21.15.89:0/4089761732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ca8990210 con 0x558ca8d199a0 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b0e72640 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 0x558ca8d23d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b1673640 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d199a0 0x558ca8d23840 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f62a80c0e70 tx=0x7f62a80f0a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4139d419f241a55f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f6294ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f6294ff9640 1 -- 172.21.15.89:0/4089761732 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8049040 con 0x558ca8d199a0 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f6294ff9640 1 -- 172.21.15.89:0/4089761732 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f62a80f6170 con 0x558ca8d199a0 2024-09-15T23:27:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b6670740 1 -- 172.21.15.89:0/4089761732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558ca8c8aa20 con 0x558ca8d199a0 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f6294ff9640 1 -- 172.21.15.89:0/4089761732 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f62a809e880 con 0x558ca8d199a0 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b6670740 10 client.16668 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.982+0000 7f62b6670740 1 -- 172.21.15.89:0/4089761732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ca8a5c620 con 0x558ca8d199a0 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f6294ff9640 1 -- 172.21.15.89:0/4089761732 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558ca8c8aa20 con 0x558ca8d199a0 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f6294ff9640 1 -- 172.21.15.89:0/4089761732 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f62a80b8f70 con 0x558ca8d199a0 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f62b6670740 10 client.16668 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f62b6670740 10 client.16668 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f62b6670740 20 client.16668 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f62b6670740 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558ca8d2f2b0 0x558ca8d316a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f62b6670740 4 client.16668 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f62b6670740 1 -- 172.21.15.89:0/4089761732 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ca8c84d70 con 0x558ca8d2f2b0 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f62b0e72640 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558ca8d2f2b0 0x558ca8d316a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:31.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f6294ff9640 10 client.16668 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.983+0000 7f62b0e72640 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558ca8d2f2b0 0x558ca8d316a0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8f4f7153141e591 server_cookie=fcde3a1ed0004b1 in_seq=0 out_seq=0 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f6294ff9640 1 -- 172.21.15.89:0/4089761732 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558ca8c84d70 con 0x558ca8d2f2b0 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f6294ff9640 10 client.16668 handle_command_reply: tid=0 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f62b6670740 1 client.16668 shutdown 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f62b6670740 1 -- 172.21.15.89:0/4089761732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d199a0 msgr2=0x558ca8d23840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f62b6670740 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d199a0 0x558ca8d23840 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f62a80c0e70 tx=0x7f62a80f0a30 comp rx=0 tx=0).stop 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f62b6670740 1 -- 172.21.15.89:0/4089761732 shutdown_connections 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f62b6670740 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558ca8d2f2b0 0x558ca8d316a0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f62b6670740 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 0x558ca8d2adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f6294ff9640 0 client.16668 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f62b6670740 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 0x558ca8d23d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f62b6670740 1 --2- 172.21.15.89:0/4089761732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d199a0 0x558ca8d23840 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f62b6670740 1 -- 172.21.15.89:0/4089761732 >> 172.21.15.89:0/4089761732 conn(0x558ca8d15010 msgr2=0x558ca8d16820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.984+0000 7f62b6670740 1 -- 172.21.15.89:0/4089761732 shutdown_connections 2024-09-15T23:27:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.985+0000 7f62b6670740 1 -- 172.21.15.89:0/4089761732 wait complete. 2024-09-15T23:27:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.985+0000 7f62b6670740 20 client.16668 trim_cache size 0 max 16384 2024-09-15T23:27:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b6670740 1 Processor -- start 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b6670740 1 -- start start 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b6670740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d199a0 0x558ca8d23e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b6670740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 0x558ca8d2ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b6670740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1f370 0x558ca8d140b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b6670740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1acad0 con 0x558ca8d1f370 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b6670740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1ae610 con 0x558ca8d199a0 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b6670740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1ad1a0 con 0x558ca8d1a2e0 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b1e74640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1f370 0x558ca8d140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b1e74640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1f370 0x558ca8d140b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37500/0 (socket says 172.21.15.89:37500) 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b1e74640 1 -- 172.21.15.89:0/2499530431 learned_addr learned my addr 172.21.15.89:0/2499530431 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b1673640 1 --2- 172.21.15.89:0/2499530431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d199a0 0x558ca8d23e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:31.999+0000 7f62b0e72640 1 --2- 172.21.15.89:0/2499530431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 0x558ca8d2ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.000+0000 7f62b1e74640 1 -- 172.21.15.89:0/2499530431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 msgr2=0x558ca8d2ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.000+0000 7f62b1e74640 1 --2- 172.21.15.89:0/2499530431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 0x558ca8d2ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.000+0000 7f62b1e74640 1 -- 172.21.15.89:0/2499530431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d199a0 msgr2=0x558ca8d23e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.000+0000 7f62b1e74640 1 --2- 172.21.15.89:0/2499530431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d199a0 0x558ca8d23e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.000+0000 7f62b1e74640 1 -- 172.21.15.89:0/2499530431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ca8d23380 con 0x558ca8d1f370 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.000+0000 7f62b1e74640 1 --2- 172.21.15.89:0/2499530431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1f370 0x558ca8d140b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f62a4012610 tx=0x7f62a4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868d96c7d91801ba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.000+0000 7f6296ffd640 1 -- 172.21.15.89:0/2499530431 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a400e980 con 0x558ca8d1f370 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.000+0000 7f6296ffd640 1 -- 172.21.15.89:0/2499530431 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62a4022030 con 0x558ca8d1f370 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.000+0000 7f6296ffd640 1 -- 172.21.15.89:0/2499530431 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a401d430 con 0x558ca8d1f370 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 -- 172.21.15.89:0/2499530431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1f370 msgr2=0x558ca8d140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 --2- 172.21.15.89:0/2499530431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1f370 0x558ca8d140b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f62a4012610 tx=0x7f62a4013000 comp rx=0 tx=0).stop 2024-09-15T23:27:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 -- 172.21.15.89:0/2499530431 shutdown_connections 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 --2- 172.21.15.89:0/2499530431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1f370 0x558ca8d140b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 --2- 172.21.15.89:0/2499530431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d1a2e0 0x558ca8d2ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 --2- 172.21.15.89:0/2499530431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d199a0 0x558ca8d23e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 -- 172.21.15.89:0/2499530431 >> 172.21.15.89:0/2499530431 conn(0x558ca8d15010 msgr2=0x558ca8d139a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 -- 172.21.15.89:0/2499530431 shutdown_connections 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 -- 172.21.15.89:0/2499530431 wait complete. 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 -- start start 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 Processor -- start 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d199a0 0x558ca8d27e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d28380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 0x558ca8d29340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1adf40 con 0x558ca8d1a2e0 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1ad870 con 0x558ca8d1f370 2024-09-15T23:27:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac1aaab0 con 0x558ca8d199a0 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.001+0000 7f62b6670740 10 client.0 _ll_register_callbacks cb 0x7f62a400b4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b0e72640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d28380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b0e72640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d28380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37508/0 (socket says 172.21.15.89:37508) 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b0e72640 1 -- 172.21.15.89:0/2926607938 learned_addr learned my addr 172.21.15.89:0/2926607938 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b1673640 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d199a0 0x558ca8d27e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b1e74640 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 0x558ca8d29340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b0e72640 1 -- 172.21.15.89:0/2926607938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d199a0 msgr2=0x558ca8d27e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b0e72640 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d199a0 0x558ca8d27e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b0e72640 1 -- 172.21.15.89:0/2926607938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 msgr2=0x558ca8d29340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b0e72640 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 0x558ca8d29340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b0e72640 1 -- 172.21.15.89:0/2926607938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ca8d29950 con 0x558ca8d1a2e0 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b1e74640 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 0x558ca8d29340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b0e72640 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d28380 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f62ac101ca0 tx=0x7f629c0128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c560f7d42cdc7a3b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f6294ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f6294ff9640 1 -- 172.21.15.89:0/2926607938 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629c01e070 con 0x558ca8d1a2e0 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f6294ff9640 1 -- 172.21.15.89:0/2926607938 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f629c005500 con 0x558ca8d1a2e0 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b6670740 1 -- 172.21.15.89:0/2926607938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62a4022030 con 0x558ca8d1a2e0 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f6294ff9640 1 -- 172.21.15.89:0/2926607938 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f629c02a050 con 0x558ca8d1a2e0 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b6670740 10 client.16674 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f62b6670740 1 -- 172.21.15.89:0/2926607938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ca8d2ad30 con 0x558ca8d1a2e0 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.002+0000 7f6294ff9640 1 -- 172.21.15.89:0/2926607938 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62a4022030 con 0x558ca8d1a2e0 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.003+0000 7f6294ff9640 1 -- 172.21.15.89:0/2926607938 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f629c011860 con 0x558ca8d1a2e0 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.003+0000 7f62b6670740 10 client.16674 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.003+0000 7f62b6670740 10 client.16674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.003+0000 7f62b6670740 20 client.16674 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.003+0000 7f62b6670740 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558ca8d9a2f0 0x558ca8d9c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.003+0000 7f62b6670740 4 client.16674 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:27:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.003+0000 7f62b6670740 1 -- 172.21.15.89:0/2926607938 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558ca8c01900 con 0x558ca8d9a2f0 2024-09-15T23:27:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.003+0000 7f62b1673640 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558ca8d9a2f0 0x558ca8d9c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.003+0000 7f6294ff9640 10 client.16674 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.003+0000 7f62b1673640 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558ca8d9a2f0 0x558ca8d9c710 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cba4512df317215 server_cookie=38ccc66b34bb1b23 in_seq=0 out_seq=0 2024-09-15T23:27:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.013+0000 7f6294ff9640 1 -- 172.21.15.89:0/2926607938 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558ca8c01900 con 0x558ca8d9a2f0 2024-09-15T23:27:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.013+0000 7f6294ff9640 10 client.16674 handle_command_reply: tid=0 2024-09-15T23:27:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.013+0000 7f62b6670740 1 client.16674 shutdown 2024-09-15T23:27:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 1 -- 172.21.15.89:0/2926607938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 msgr2=0x558ca8d28380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d28380 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f62ac101ca0 tx=0x7f629c0128e0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 1 -- 172.21.15.89:0/2926607938 shutdown_connections 2024-09-15T23:27:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558ca8d9a2f0 0x558ca8d9c710 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ca8d1f370 0x558ca8d29340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ca8d1a2e0 0x558ca8d28380 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f6294ff9640 0 client.16674 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 1 --2- 172.21.15.89:0/2926607938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ca8d199a0 0x558ca8d27e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 1 -- 172.21.15.89:0/2926607938 >> 172.21.15.89:0/2926607938 conn(0x558ca8d15010 msgr2=0x558ca8d15ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 1 -- 172.21.15.89:0/2926607938 shutdown_connections 2024-09-15T23:27:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 1 -- 172.21.15.89:0/2926607938 wait complete. 2024-09-15T23:27:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b6670740 20 client.16674 trim_cache size 0 max 16384 2024-09-15T23:27:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:27:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:27:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:27:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:27:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 -- 172.21.15.89:0/3291657378 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62980a3370 msgr2=0x7f62980a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62980a3370 0x7f62980a5830 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f62ac06a980 tx=0x7f62a4004ab0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 -- 172.21.15.89:0/3291657378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ac1025c0 msgr2=0x7f62ac101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ac1025c0 0x7f62ac101f30 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009230 tx=0x7f62a8004360 comp rx=0 tx=0).stop 2024-09-15T23:27:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 -- 172.21.15.89:0/3291657378 shutdown_connections 2024-09-15T23:27:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62980a3370 0x7f62980a5830 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ac107820 0x7f62ac100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62ac104ef0 0x7f62ac100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 --2- 172.21.15.89:0/3291657378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62ac1025c0 0x7f62ac101f30 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 -- 172.21.15.89:0/3291657378 >> 172.21.15.89:0/3291657378 conn(0x7f62ac0fc1e0 msgr2=0x7f62ac102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 -- 172.21.15.89:0/3291657378 shutdown_connections 2024-09-15T23:27:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:32.014+0000 7f62b38d7640 1 -- 172.21.15.89:0/3291657378 wait complete. 2024-09-15T23:27:32.163 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:32.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/3912232275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74101410 msgr2=0x7f3c7410d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 --2- 172.21.15.89:0/3912232275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74101410 0x7f3c7410d940 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c009cd0 tx=0x7f3c6c02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/3912232275 shutdown_connections 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 --2- 172.21.15.89:0/3912232275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74101410 0x7f3c7410d940 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 --2- 172.21.15.89:0/3912232275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c74100a70 0x7f3c74100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 --2- 172.21.15.89:0/3912232275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c74106a70 0x7f3c74106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/3912232275 >> 172.21.15.89:0/3912232275 conn(0x7f3c740fc790 msgr2=0x7f3c740febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/3912232275 shutdown_connections 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/3912232275 wait complete. 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 Processor -- start 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.688+0000 7f3c7c0cd640 1 -- start start 2024-09-15T23:27:32.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c7c0cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c74100a70 0x7f3c74198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c7c0cd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74101410 0x7f3c74198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c7c0cd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c74106a70 0x7f3c7419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c7c0cd640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c74110e70 con 0x7f3c74101410 2024-09-15T23:27:32.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c7c0cd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c74110cf0 con 0x7f3c74106a70 2024-09-15T23:27:32.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c7c0cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c74110ff0 con 0x7f3c74100a70 2024-09-15T23:27:32.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c79668640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74101410 0x7f3c74198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c79668640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74101410 0x7f3c74198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37532/0 (socket says 172.21.15.89:37532) 2024-09-15T23:27:32.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c79668640 1 -- 172.21.15.89:0/829716544 learned_addr learned my addr 172.21.15.89:0/829716544 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:32.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c79e69640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c74100a70 0x7f3c74198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c79668640 1 -- 172.21.15.89:0/829716544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c74100a70 msgr2=0x7f3c74198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c79668640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c74100a70 0x7f3c74198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c79668640 1 -- 172.21.15.89:0/829716544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c74106a70 msgr2=0x7f3c7419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c79668640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c74106a70 0x7f3c7419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c79668640 1 -- 172.21.15.89:0/829716544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c7419d2f0 con 0x7f3c74101410 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c79668640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74101410 0x7f3c74198880 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000c890 tx=0x7f3c7000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c66ffd640 1 -- 172.21.15.89:0/829716544 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c70004220 con 0x7f3c74101410 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c66ffd640 1 -- 172.21.15.89:0/829716544 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c700043c0 con 0x7f3c74101410 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/829716544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c7419d5e0 con 0x7f3c74101410 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c66ffd640 1 -- 172.21.15.89:0/829716544 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c70004a00 con 0x7f3c74101410 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.689+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/829716544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c741a4ec0 con 0x7f3c74101410 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.691+0000 7f3c66ffd640 1 -- 172.21.15.89:0/829716544 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3c700079b0 con 0x7f3c74101410 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7e4a6740 1 Processor -- start 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7e4a6740 1 -- start start 2024-09-15T23:27:32.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210fb40 0x55b30210e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210e5d0 0x55b30210ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b302119370 0x55b30211b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b3020742f0 con 0x55b30210fb40 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b30201bcf0 con 0x55b30210e5d0 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b3020702b0 con 0x55b302119370 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7a66a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210fb40 0x55b30210e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7a66a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210fb40 0x55b30210e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37536/0 (socket says 172.21.15.89:37536) 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c7a66a640 1 -- 172.21.15.89:0/1255357955 learned_addr learned my addr 172.21.15.89:0/1255357955 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c66ffd640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c600a3240 0x7f3c600a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.692+0000 7f3c79668640 1 --2- 172.21.15.89:0/1255357955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b302119370 0x55b30211b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c79e69640 1 --2- 172.21.15.89:0/1255357955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210e5d0 0x55b30210ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c7a66a640 1 -- 172.21.15.89:0/1255357955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b302119370 msgr2=0x55b30211b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c7a66a640 1 --2- 172.21.15.89:0/1255357955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b302119370 0x55b30211b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c7a66a640 1 -- 172.21.15.89:0/1255357955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210e5d0 msgr2=0x55b30210ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c7a66a640 1 --2- 172.21.15.89:0/1255357955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210e5d0 0x55b30210ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c7a66a640 1 -- 172.21.15.89:0/1255357955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3020003a0 con 0x55b30210fb40 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c66ffd640 1 -- 172.21.15.89:0/829716544 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c700cf9d0 con 0x7f3c74101410 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c79668640 1 --2- 172.21.15.89:0/1255357955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b302119370 0x55b30211b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c7a66a640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c600a3240 0x7f3c600a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c7a66a640 1 --2- 172.21.15.89:0/1255357955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210fb40 0x55b30210e090 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c031500 tx=0x7f3c6c031530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1cf54745e2878b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c5ffff640 1 -- 172.21.15.89:0/1255357955 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c04d070 con 0x55b30210fb40 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c5ffff640 1 -- 172.21.15.89:0/1255357955 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c6c043070 con 0x55b30210fb40 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.693+0000 7f3c5ffff640 1 -- 172.21.15.89:0/1255357955 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c030d30 con 0x55b30210fb40 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/1255357955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210fb40 msgr2=0x55b30210e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7a66a640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c600a3240 0x7f3c600a5700 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c009df0 tx=0x7f3c6c04d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:32.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/1255357955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210fb40 0x55b30210e090 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c031500 tx=0x7f3c6c031530 comp rx=0 tx=0).stop 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/1255357955 shutdown_connections 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/1255357955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b302119370 0x55b30211b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/1255357955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210e5d0 0x55b30210ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/1255357955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210fb40 0x55b30210e090 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/1255357955 >> 172.21.15.89:0/1255357955 conn(0x55b3020d7310 msgr2=0x55b30210ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/1255357955 shutdown_connections 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/1255357955 wait complete. 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 -- start start 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.694+0000 7f3c7e4a6740 1 Processor -- start 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b302118d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210fb40 0x55b302116010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b302119370 0x55b302116600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b30211c070 con 0x55b30210e5d0 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b30202f060 con 0x55b302119370 2024-09-15T23:27:32.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b30211c1f0 con 0x55b30210fb40 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c7e4a6740 10 client.0 _ll_register_callbacks cb 0x55b301d828c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79e69640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b302118d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79e69640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b302118d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37548/0 (socket says 172.21.15.89:37548) 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79e69640 1 -- 172.21.15.89:0/3176984115 learned_addr learned my addr 172.21.15.89:0/3176984115 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79668640 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210fb40 0x55b302116010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79e69640 1 -- 172.21.15.89:0/3176984115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210fb40 msgr2=0x55b302116010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79e69640 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210fb40 0x55b302116010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79e69640 1 -- 172.21.15.89:0/3176984115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b302119370 msgr2=0x55b302116600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79e69640 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b302119370 0x55b302116600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79e69640 1 -- 172.21.15.89:0/3176984115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b301de53c0 con 0x55b30210e5d0 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79668640 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210fb40 0x55b302116010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:32.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c79e69640 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b302118d60 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3c68012750 tx=0x7f3c68012c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4e3dea984e3a27 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/3176984115 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6801f060 con 0x55b30210e5d0 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.695+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/3176984115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c6c043070 con 0x55b30210e5d0 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/3176984115 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c68010dc0 con 0x55b30210e5d0 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/3176984115 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c68026050 con 0x55b30210e5d0 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c7e4a6740 10 client.16686 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/3176984115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b302127b30 con 0x55b30210e5d0 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/3176984115 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c6c043070 con 0x55b30210e5d0 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/3176984115 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3c6800e640 con 0x55b30210e5d0 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c7e4a6740 10 client.16686 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c7e4a6740 10 client.16686 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c7e4a6740 20 client.16686 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:32.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b302128f40 0x55b30212b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c7e4a6740 4 client.16686 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/3176984115 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b302081860 con 0x55b302128f40 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.696+0000 7f3c79668640 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b302128f40 0x55b30212b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.697+0000 7f3c5d7fa640 10 client.16686 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.697+0000 7f3c79668640 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b302128f40 0x55b30212b370 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a0acd03ce085d8fd server_cookie=8023796c3165ac33 in_seq=0 out_seq=0 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.697+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/3176984115 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b302081860 con 0x55b302128f40 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.697+0000 7f3c5d7fa640 10 client.16686 handle_command_reply: tid=0 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 client.16686 shutdown 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/3176984115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 msgr2=0x55b302118d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b302118d60 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f3c68012750 tx=0x7f3c68012c10 comp rx=0 tx=0).stop 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/3176984115 shutdown_connections 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b302128f40 0x55b30212b370 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b302119370 0x55b302116600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210fb40 0x55b302116010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/3176984115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b302118d60 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c5d7fa640 0 client.16686 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:27:32.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/3176984115 >> 172.21.15.89:0/3176984115 conn(0x55b3020d7310 msgr2=0x55b302110930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:32.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/3176984115 shutdown_connections 2024-09-15T23:27:32.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/3176984115 wait complete. 2024-09-15T23:27:32.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.698+0000 7f3c7e4a6740 20 client.16686 trim_cache size 0 max 16384 2024-09-15T23:27:32.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c7e4a6740 1 Processor -- start 2024-09-15T23:27:32.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c7e4a6740 1 -- start start 2024-09-15T23:27:32.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b30210ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 0x55b302125310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b302119370 0x55b302130c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c741aee30 con 0x55b30210e5d0 2024-09-15T23:27:32.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c741b0970 con 0x55b30210fb40 2024-09-15T23:27:32.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c741af500 con 0x55b302119370 2024-09-15T23:27:32.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c79e69640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b30210ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c79e69640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b30210ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37552/0 (socket says 172.21.15.89:37552) 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c79e69640 1 -- 172.21.15.89:0/2545543045 learned_addr learned my addr 172.21.15.89:0/2545543045 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c79e69640 1 -- 172.21.15.89:0/2545543045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b302119370 msgr2=0x55b302130c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c79e69640 1 --2- 172.21.15.89:0/2545543045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b302119370 0x55b302130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c79668640 1 --2- 172.21.15.89:0/2545543045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 0x55b302125310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c79e69640 1 -- 172.21.15.89:0/2545543045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 msgr2=0x55b302125310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c79e69640 1 --2- 172.21.15.89:0/2545543045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 0x55b302125310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c79e69640 1 -- 172.21.15.89:0/2545543045 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b30210e2d0 con 0x55b30210e5d0 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.711+0000 7f3c79668640 1 --2- 172.21.15.89:0/2545543045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 0x55b302125310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c79e69640 1 --2- 172.21.15.89:0/2545543045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b30210ae10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3c68005b70 tx=0x7f3c6800afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201bf96618b5edb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c5f7fe640 1 -- 172.21.15.89:0/2545543045 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68012e60 con 0x55b30210e5d0 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c5f7fe640 1 -- 172.21.15.89:0/2545543045 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c68010bc0 con 0x55b30210e5d0 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c5f7fe640 1 -- 172.21.15.89:0/2545543045 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6802ab10 con 0x55b30210e5d0 2024-09-15T23:27:32.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2545543045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 msgr2=0x55b30210ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/2545543045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b30210ae10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3c68005b70 tx=0x7f3c6800afd0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2545543045 shutdown_connections 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/2545543045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b302119370 0x55b302130c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/2545543045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 0x55b302125310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/2545543045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b30210e5d0 0x55b30210ae10 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2545543045 >> 172.21.15.89:0/2545543045 conn(0x55b3020d7310 msgr2=0x55b30210a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2545543045 shutdown_connections 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.712+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2545543045 wait complete. 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.713+0000 7f3c7e4a6740 1 -- start start 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.713+0000 7f3c7e4a6740 1 Processor -- start 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.713+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210e5d0 0x55b302113180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.713+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 0x55b3021136c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.713+0000 7f3c7e4a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b302119370 0x55b302113c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.713+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c741b02a0 con 0x55b302119370 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.713+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c741afbd0 con 0x55b30210fb40 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.713+0000 7f3c7e4a6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c741ace10 con 0x55b30210e5d0 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.713+0000 7f3c7e4a6740 10 client.0 _ll_register_callbacks cb 0x7f3c6800a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.713+0000 7f3c7a66a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b302119370 0x55b302113c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7a66a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b302119370 0x55b302113c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37562/0 (socket says 172.21.15.89:37562) 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7a66a640 1 -- 172.21.15.89:0/2935379136 learned_addr learned my addr 172.21.15.89:0/2935379136 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c79668640 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 0x55b3021136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c79e69640 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210e5d0 0x55b302113180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7a66a640 1 -- 172.21.15.89:0/2935379136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210e5d0 msgr2=0x55b302113180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7a66a640 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210e5d0 0x55b302113180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7a66a640 1 -- 172.21.15.89:0/2935379136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 msgr2=0x55b3021136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7a66a640 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 0x55b3021136c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7a66a640 1 -- 172.21.15.89:0/2935379136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b302197150 con 0x55b302119370 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7a66a640 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b302119370 0x55b302113c00 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c005f90 tx=0x7f3c6c009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfcf207f6f83754b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/2935379136 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c055070 con 0x55b302119370 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/2935379136 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c6c0309b0 con 0x55b302119370 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2935379136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c68010bc0 con 0x55b302119370 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/2935379136 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c6c004840 con 0x55b302119370 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7e4a6740 10 client.16692 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2935379136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b3021983a0 con 0x55b302119370 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.714+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/2935379136 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c68010bc0 con 0x55b302119370 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.715+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/2935379136 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3c6c030070 con 0x55b302119370 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.715+0000 7f3c7e4a6740 10 client.16692 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.715+0000 7f3c7e4a6740 10 client.16692 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.715+0000 7f3c7e4a6740 20 client.16692 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:32.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.715+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b302198680 0x55b30219aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.715+0000 7f3c7e4a6740 4 client.16692 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.715+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2935379136 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b30207bd70 con 0x55b302198680 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.715+0000 7f3c79e69640 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b302198680 0x55b30219aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c79e69640 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b302198680 0x55b30219aa70 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=42baab68cfb022f server_cookie=e6e7cebb5dcef07 in_seq=0 out_seq=0 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c5d7fa640 10 client.16692 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c5d7fa640 1 -- 172.21.15.89:0/2935379136 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b30207bd70 con 0x55b302198680 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c5d7fa640 10 client.16692 handle_command_reply: tid=0 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 client.16692 shutdown 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2935379136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b302119370 msgr2=0x55b302113c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b302119370 0x55b302113c00 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c005f90 tx=0x7f3c6c009c90 comp rx=0 tx=0).stop 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2935379136 shutdown_connections 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b302198680 0x55b30219aa70 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b302119370 0x55b302113c00 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b30210fb40 0x55b3021136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c5d7fa640 0 client.16692 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:27:32.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 --2- 172.21.15.89:0/2935379136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b30210e5d0 0x55b302113180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2935379136 >> 172.21.15.89:0/2935379136 conn(0x55b3020d7310 msgr2=0x55b3021094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:32.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2935379136 shutdown_connections 2024-09-15T23:27:32.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 1 -- 172.21.15.89:0/2935379136 wait complete. 2024-09-15T23:27:32.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.716+0000 7f3c7e4a6740 20 client.16692 trim_cache size 0 max 16384 2024-09-15T23:27:32.728 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:27:32.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/829716544 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c600a3240 msgr2=0x7f3c600a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c600a3240 0x7f3c600a5700 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c009df0 tx=0x7f3c6c04d000 comp rx=0 tx=0).stop 2024-09-15T23:27:32.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/829716544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74101410 msgr2=0x7f3c74198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:32.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74101410 0x7f3c74198880 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000c890 tx=0x7f3c7000cc90 comp rx=0 tx=0).stop 2024-09-15T23:27:32.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/829716544 shutdown_connections 2024-09-15T23:27:32.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c600a3240 0x7f3c600a5700 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c74106a70 0x7f3c7419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c74101410 0x7f3c74198880 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 --2- 172.21.15.89:0/829716544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c74100a70 0x7f3c74198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:32.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/829716544 >> 172.21.15.89:0/829716544 conn(0x7f3c740fc790 msgr2=0x7f3c7406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:32.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/829716544 shutdown_connections 2024-09-15T23:27:32.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:32.717+0000 7f3c7c0cd640 1 -- 172.21.15.89:0/829716544 wait complete. 2024-09-15T23:27:33.589 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:27:33.836 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:34.235 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:34 smithi089 ceph-mon[30933]: pgmap v456: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:27:34.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.315+0000 7f9ea1078640 1 -- 172.21.15.89:0/54953481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e9c106a80 msgr2=0x7f9e9c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:34.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.315+0000 7f9ea1078640 1 --2- 172.21.15.89:0/54953481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e9c106a80 0x7f9e9c106e60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009930 tx=0x7f9e8c02a420 comp rx=0 tx=0).stop 2024-09-15T23:27:34.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.315+0000 7f9ea1078640 1 -- 172.21.15.89:0/54953481 shutdown_connections 2024-09-15T23:27:34.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.315+0000 7f9ea1078640 1 --2- 172.21.15.89:0/54953481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e9c101420 0x7f9e9c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.315+0000 7f9ea1078640 1 --2- 172.21.15.89:0/54953481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e9c100a80 0x7f9e9c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.315+0000 7f9ea1078640 1 --2- 172.21.15.89:0/54953481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e9c106a80 0x7f9e9c106e60 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.315+0000 7f9ea1078640 1 -- 172.21.15.89:0/54953481 >> 172.21.15.89:0/54953481 conn(0x7f9e9c0fc7a0 msgr2=0x7f9e9c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:34.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.315+0000 7f9ea1078640 1 -- 172.21.15.89:0/54953481 shutdown_connections 2024-09-15T23:27:34.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.315+0000 7f9ea1078640 1 -- 172.21.15.89:0/54953481 wait complete. 2024-09-15T23:27:34.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.315+0000 7f9ea1078640 1 Processor -- start 2024-09-15T23:27:34.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9ea1078640 1 -- start start 2024-09-15T23:27:34.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9ea1078640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e9c100a80 0x7f9e9c1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9ea1078640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e9c101420 0x7f9e9c199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9ea1078640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e9c106a80 0x7f9e9c1a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9ea1078640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c110e40 con 0x7f9e9c106a80 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9ea1078640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c110cc0 con 0x7f9e9c101420 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9ea1078640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c110fc0 con 0x7f9e9c100a80 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e9c106a80 0x7f9e9c1a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e9c106a80 0x7f9e9c1a4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37586/0 (socket says 172.21.15.89:37586) 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9b59e640 1 -- 172.21.15.89:0/1967730126 learned_addr learned my addr 172.21.15.89:0/1967730126 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e9c101420 0x7f9e9c199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e9c100a80 0x7f9e9c1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9b59e640 1 -- 172.21.15.89:0/1967730126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e9c100a80 msgr2=0x7f9e9c1992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9b59e640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e9c100a80 0x7f9e9c1992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9b59e640 1 -- 172.21.15.89:0/1967730126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e9c101420 msgr2=0x7f9e9c199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9b59e640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e9c101420 0x7f9e9c199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9b59e640 1 -- 172.21.15.89:0/1967730126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c1a48b0 con 0x7f9e9c106a80 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.316+0000 7f9e9b59e640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e9c106a80 0x7f9e9c1a4370 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400eaa0 tx=0x7f9e9400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.317+0000 7f9e93fff640 1 -- 172.21.15.89:0/1967730126 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9400cdb0 con 0x7f9e9c106a80 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.317+0000 7f9e93fff640 1 -- 172.21.15.89:0/1967730126 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e94005020 con 0x7f9e9c106a80 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.317+0000 7f9ea1078640 1 -- 172.21.15.89:0/1967730126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9c1a4a80 con 0x7f9e9c106a80 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.317+0000 7f9e93fff640 1 -- 172.21.15.89:0/1967730126 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e940116b0 con 0x7f9e9c106a80 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.317+0000 7f9ea1078640 1 -- 172.21.15.89:0/1967730126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e9c1a4f40 con 0x7f9e9c106a80 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.319+0000 7f9e93fff640 1 -- 172.21.15.89:0/1967730126 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9e94005560 con 0x7f9e9c106a80 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.319+0000 7f9ea3451740 1 Processor -- start 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.319+0000 7f9ea3451740 1 -- start start 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b9c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6b9ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9ea3451740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ffd6afe2f0 con 0x55ffd6b9df50 2024-09-15T23:27:34.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9ea3451740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ffd6aa5cf0 con 0x55ffd6ba7780 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9ea3451740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ffd6afa2b0 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b9c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b9c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37588/0 (socket says 172.21.15.89:37588) 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9a59c640 1 -- 172.21.15.89:0/1569573351 learned_addr learned my addr 172.21.15.89:0/1569573351 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e93fff640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e880a3240 0x7f9e880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/1569573351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6b9ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/1569573351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9b59e640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e880a3240 0x7f9e880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9a59c640 1 -- 172.21.15.89:0/1569573351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 msgr2=0x55ffd6b9ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/1569573351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6b9ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9a59c640 1 -- 172.21.15.89:0/1569573351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6ba7780 msgr2=0x55ffd6ba9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/1569573351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9a59c640 1 -- 172.21.15.89:0/1569573351 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffd6a8a3a0 con 0x55ffd6b9df50 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e93fff640 1 -- 172.21.15.89:0/1967730126 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9e940cfef0 con 0x7f9e9c106a80 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.320+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/1569573351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b9c4a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9e84013550 tx=0x7f9e8400a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f287c96617533465 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.321+0000 7f9e917fa640 1 -- 172.21.15.89:0/1569573351 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84019030 con 0x55ffd6b9df50 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.321+0000 7f9e9b59e640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e880a3240 0x7f9e880a5700 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400efd0 tx=0x7f9e94067000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.321+0000 7f9e917fa640 1 -- 172.21.15.89:0/1569573351 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e84005500 con 0x55ffd6b9df50 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.321+0000 7f9e917fa640 1 -- 172.21.15.89:0/1569573351 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8400f890 con 0x55ffd6b9df50 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.321+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569573351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 msgr2=0x55ffd6b9c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.321+0000 7f9ea3451740 1 --2- 172.21.15.89:0/1569573351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b9c4a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9e84013550 tx=0x7f9e8400a760 comp rx=0 tx=0).stop 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569573351 shutdown_connections 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 --2- 172.21.15.89:0/1569573351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 --2- 172.21.15.89:0/1569573351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6b9ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 --2- 172.21.15.89:0/1569573351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b9c4a0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569573351 >> 172.21.15.89:0/1569573351 conn(0x55ffd6b61310 msgr2=0x55ffd6b99070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:34.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569573351 shutdown_connections 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569573351 wait complete. 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 -- start start 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 Processor -- start 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6ba70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6b9df50 0x55ffd6ba43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ffd6baa480 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ffd6ab9060 con 0x55ffd6b9df50 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ffd6baa600 con 0x55ffd6ba7780 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9ea3451740 10 client.0 _ll_register_callbacks cb 0x55ffd680c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9e9a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6ba70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9e9a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6ba70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37596/0 (socket says 172.21.15.89:37596) 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.322+0000 7f9e9a59c640 1 -- 172.21.15.89:0/696379588 learned_addr learned my addr 172.21.15.89:0/696379588 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:34.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6b9df50 0x55ffd6ba43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e9a59c640 1 -- 172.21.15.89:0/696379588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6ba7780 msgr2=0x55ffd6ba4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e9a59c640 1 -- 172.21.15.89:0/696379588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6b9df50 msgr2=0x55ffd6ba43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6b9df50 0x55ffd6ba43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e9a59c640 1 -- 172.21.15.89:0/696379588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffd686f3c0 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6ba70f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f9e84004f60 tx=0x7f9e84004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3a97f13ab1926d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e827fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e827fc640 1 -- 172.21.15.89:0/696379588 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84019040 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e827fc640 1 -- 172.21.15.89:0/696379588 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9e8400f630 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9ea3451740 1 -- 172.21.15.89:0/696379588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e84005500 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e827fc640 1 -- 172.21.15.89:0/696379588 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e8400f830 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9ea3451740 10 client.16704 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:34.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9ea3451740 1 -- 172.21.15.89:0/696379588 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ffd6bb5e90 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e827fc640 1 -- 172.21.15.89:0/696379588 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e84005500 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9e827fc640 1 -- 172.21.15.89:0/696379588 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9e8401e9b0 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9ea3451740 10 client.16704 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9ea3451740 10 client.16704 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9ea3451740 20 client.16704 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9ea3451740 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ffd6bb7240 0x55ffd6bb9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.323+0000 7f9ea3451740 4 client.16704 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.324+0000 7f9ea3451740 1 -- 172.21.15.89:0/696379588 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffd6b0b860 con 0x55ffd6bb7240 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.324+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ffd6bb7240 0x55ffd6bb9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.324+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ffd6bb7240 0x55ffd6bb9630 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=417afd8e5d4e3ecf server_cookie=235d8c2c17c5b04b in_seq=0 out_seq=0 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.324+0000 7f9e827fc640 10 client.16704 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9e827fc640 1 -- 172.21.15.89:0/696379588 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ffd6b0b860 con 0x55ffd6bb7240 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9e827fc640 10 client.16704 handle_command_reply: tid=0 2024-09-15T23:27:34.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 client.16704 shutdown 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 -- 172.21.15.89:0/696379588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9c9e0 msgr2=0x55ffd6ba70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6ba70f0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f9e84004f60 tx=0x7f9e84004f90 comp rx=0 tx=0).stop 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 -- 172.21.15.89:0/696379588 shutdown_connections 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ffd6bb7240 0x55ffd6bb9630 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6b9df50 0x55ffd6ba43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9e827fc640 0 client.16704 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 --2- 172.21.15.89:0/696379588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6ba70f0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 -- 172.21.15.89:0/696379588 >> 172.21.15.89:0/696379588 conn(0x55ffd6b61310 msgr2=0x55ffd6b9ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 -- 172.21.15.89:0/696379588 shutdown_connections 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 1 -- 172.21.15.89:0/696379588 wait complete. 2024-09-15T23:27:34.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.325+0000 7f9ea3451740 20 client.16704 trim_cache size 0 max 16384 2024-09-15T23:27:34.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.338+0000 7f9ea3451740 1 Processor -- start 2024-09-15T23:27:34.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.338+0000 7f9ea3451740 1 -- start start 2024-09-15T23:27:34.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.338+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6bb3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.338+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b98a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.338+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6ba7780 0x55ffd6bbed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.338+0000 7f9ea3451740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1acd20 con 0x55ffd6b9df50 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.338+0000 7f9ea3451740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1ae860 con 0x55ffd6ba7780 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.338+0000 7f9ea3451740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1ad3f0 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.338+0000 7f9e9ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b98a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b98a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37598/0 (socket says 172.21.15.89:37598) 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9ad9d640 1 -- 172.21.15.89:0/344320217 learned_addr learned my addr 172.21.15.89:0/344320217 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/344320217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6bb3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/344320217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6ba7780 0x55ffd6bbed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9ad9d640 1 -- 172.21.15.89:0/344320217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 msgr2=0x55ffd6bb3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/344320217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6bb3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9ad9d640 1 -- 172.21.15.89:0/344320217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6ba7780 msgr2=0x55ffd6bbed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/344320217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6ba7780 0x55ffd6bbed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9ad9d640 1 -- 172.21.15.89:0/344320217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffd6b9d710 con 0x55ffd6b9df50 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/344320217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6bb3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/344320217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b98a60 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c02a940 tx=0x7f9e8c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b921a40fbc53f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e90ff9640 1 -- 172.21.15.89:0/344320217 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c049070 con 0x55ffd6b9df50 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e90ff9640 1 -- 172.21.15.89:0/344320217 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e8c005520 con 0x55ffd6b9df50 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.339+0000 7f9e90ff9640 1 -- 172.21.15.89:0/344320217 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c035040 con 0x55ffd6b9df50 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 -- 172.21.15.89:0/344320217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 msgr2=0x55ffd6b98a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 --2- 172.21.15.89:0/344320217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b98a60 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c02a940 tx=0x7f9e8c005d50 comp rx=0 tx=0).stop 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 -- 172.21.15.89:0/344320217 shutdown_connections 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 --2- 172.21.15.89:0/344320217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6ba7780 0x55ffd6bbed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 --2- 172.21.15.89:0/344320217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6b98a60 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 --2- 172.21.15.89:0/344320217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6bb3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 -- 172.21.15.89:0/344320217 >> 172.21.15.89:0/344320217 conn(0x55ffd6b61310 msgr2=0x55ffd6ba3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:34.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 -- 172.21.15.89:0/344320217 shutdown_connections 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 -- 172.21.15.89:0/344320217 wait complete. 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 -- start start 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.340+0000 7f9ea3451740 1 Processor -- start 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6ba15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6ba1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9ea3451740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9ea3451740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1ae190 con 0x55ffd6b9df50 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9ea3451740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1adac0 con 0x55ffd6b9c9e0 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9ea3451740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1aad00 con 0x55ffd6ba7780 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9ea3451740 10 client.0 _ll_register_callbacks cb 0x7f9e8c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6ba1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6ba1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37610/0 (socket says 172.21.15.89:37610) 2024-09-15T23:27:34.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9ad9d640 1 -- 172.21.15.89:0/1569196693 learned_addr learned my addr 172.21.15.89:0/1569196693 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6ba15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9ad9d640 1 -- 172.21.15.89:0/1569196693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6ba7780 msgr2=0x55ffd6ba2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9ad9d640 1 -- 172.21.15.89:0/1569196693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6b9c9e0 msgr2=0x55ffd6ba15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6ba15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9ad9d640 1 -- 172.21.15.89:0/1569196693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffd6c20ef0 con 0x55ffd6b9df50 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e9ad9d640 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6ba1af0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009230 tx=0x7f9e8c004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a506e98c3d84dcef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e827fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e827fc640 1 -- 172.21.15.89:0/1569196693 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c049040 con 0x55ffd6b9df50 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9e827fc640 1 -- 172.21.15.89:0/1569196693 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9e8c005040 con 0x55ffd6b9df50 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.341+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569196693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e8c005520 con 0x55ffd6b9df50 2024-09-15T23:27:34.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9e827fc640 1 -- 172.21.15.89:0/1569196693 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e8c035040 con 0x55ffd6b9df50 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9ea3451740 10 client.16710 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569196693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ffd6c221d0 con 0x55ffd6b9df50 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9e827fc640 1 -- 172.21.15.89:0/1569196693 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e8c005520 con 0x55ffd6b9df50 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9e827fc640 1 -- 172.21.15.89:0/1569196693 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9e8c055030 con 0x55ffd6b9df50 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9ea3451740 10 client.16710 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9ea3451740 10 client.16710 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9ea3451740 20 client.16710 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9ea3451740 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ffd6c22830 0x55ffd6c24c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9ea3451740 4 client.16710 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569196693 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ffd6b05d70 con 0x55ffd6c22830 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.342+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ffd6c22830 0x55ffd6c24c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9e827fc640 10 client.16710 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9e9a59c640 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ffd6c22830 0x55ffd6c24c20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5a4ef64a82dcb444 server_cookie=7029c8d1cdd93e00 in_seq=0 out_seq=0 2024-09-15T23:27:34.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9e827fc640 1 -- 172.21.15.89:0/1569196693 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ffd6b05d70 con 0x55ffd6c22830 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9e827fc640 10 client.16710 handle_command_reply: tid=0 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 client.16710 shutdown 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569196693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 msgr2=0x55ffd6ba1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6ba1af0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009230 tx=0x7f9e8c004d30 comp rx=0 tx=0).stop 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569196693 shutdown_connections 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ffd6c22830 0x55ffd6c24c20 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ffd6ba7780 0x55ffd6ba2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ffd6b9df50 0x55ffd6ba1af0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9e827fc640 0 client.16710 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 --2- 172.21.15.89:0/1569196693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ffd6b9c9e0 0x55ffd6ba15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569196693 >> 172.21.15.89:0/1569196693 conn(0x55ffd6b61310 msgr2=0x55ffd6ba2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569196693 shutdown_connections 2024-09-15T23:27:34.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 1 -- 172.21.15.89:0/1569196693 wait complete. 2024-09-15T23:27:34.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.343+0000 7f9ea3451740 20 client.16710 trim_cache size 0 max 16384 2024-09-15T23:27:34.357 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:27:34.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 -- 172.21.15.89:0/1967730126 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e880a3240 msgr2=0x7f9e880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e880a3240 0x7f9e880a5700 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400efd0 tx=0x7f9e94067000 comp rx=0 tx=0).stop 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 -- 172.21.15.89:0/1967730126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e9c106a80 msgr2=0x7f9e9c1a4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e9c106a80 0x7f9e9c1a4370 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400eaa0 tx=0x7f9e9400ef60 comp rx=0 tx=0).stop 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 -- 172.21.15.89:0/1967730126 shutdown_connections 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e880a3240 0x7f9e880a5700 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e9c106a80 0x7f9e9c1a4370 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e9c101420 0x7f9e9c199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 --2- 172.21.15.89:0/1967730126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e9c100a80 0x7f9e9c1992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 -- 172.21.15.89:0/1967730126 >> 172.21.15.89:0/1967730126 conn(0x7f9e9c0fc7a0 msgr2=0x7f9e9c0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 -- 172.21.15.89:0/1967730126 shutdown_connections 2024-09-15T23:27:34.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:34.344+0000 7f9ea1078640 1 -- 172.21.15.89:0/1967730126 wait complete. 2024-09-15T23:27:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:34 smithi120 ceph-mon[35250]: pgmap v456: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:27:34.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:34 smithi174 ceph-mon[35586]: pgmap v456: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:27:35.079 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:27:35.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:35 smithi089 ceph-mon[30933]: pgmap v457: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 KiB/s wr, 13 op/s 2024-09-15T23:27:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:35 smithi120 ceph-mon[35250]: pgmap v457: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 KiB/s wr, 13 op/s 2024-09-15T23:27:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:35 smithi174 ceph-mon[35586]: pgmap v457: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 KiB/s wr, 13 op/s 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 -- 172.21.15.89:0/1285173532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb4106f10 msgr2=0x7f4cb410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 --2- 172.21.15.89:0/1285173532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb4106f10 0x7f4cb410ea90 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f4cac009ca0 tx=0x7f4cac02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 -- 172.21.15.89:0/1285173532 shutdown_connections 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 --2- 172.21.15.89:0/1285173532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb4106f10 0x7f4cb410ea90 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 --2- 172.21.15.89:0/1285173532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb4102790 0x7f4cb41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 --2- 172.21.15.89:0/1285173532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb4101de0 0x7f4cb41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 -- 172.21.15.89:0/1285173532 >> 172.21.15.89:0/1285173532 conn(0x7f4cb40fd8c0 msgr2=0x7f4cb40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 -- 172.21.15.89:0/1285173532 shutdown_connections 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 -- 172.21.15.89:0/1285173532 wait complete. 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 Processor -- start 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.784+0000 7f4cba84f640 1 -- start start 2024-09-15T23:27:35.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cba84f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb4101de0 0x7f4cb41983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cba84f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb4102790 0x7f4cb4198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cba84f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb4106f10 0x7f4cb419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cba84f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb4111f70 con 0x7f4cb4101de0 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cba84f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb4111df0 con 0x7f4cb4106f10 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cba84f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb41120f0 con 0x7f4cb4102790 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb4101de0 0x7f4cb41983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb4101de0 0x7f4cb41983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59410/0 (socket says 172.21.15.89:59410) 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb3fff640 1 -- 172.21.15.89:0/932333450 learned_addr learned my addr 172.21.15.89:0/932333450 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cabfff640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb4102790 0x7f4cb4198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb8dec640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb4106f10 0x7f4cb419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb3fff640 1 -- 172.21.15.89:0/932333450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb4102790 msgr2=0x7f4cb4198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb4102790 0x7f4cb4198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb3fff640 1 -- 172.21.15.89:0/932333450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb4106f10 msgr2=0x7f4cb419cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb4106f10 0x7f4cb419cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb3fff640 1 -- 172.21.15.89:0/932333450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb419d380 con 0x7f4cb4101de0 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb4101de0 0x7f4cb41983d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400d770 tx=0x7f4ca400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb1ffb640 1 -- 172.21.15.89:0/932333450 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4017070 con 0x7f4cb4101de0 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb1ffb640 1 -- 172.21.15.89:0/932333450 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca4005500 con 0x7f4cb4101de0 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cba84f640 1 -- 172.21.15.89:0/932333450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb419d670 con 0x7f4cb4101de0 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.785+0000 7f4cb1ffb640 1 -- 172.21.15.89:0/932333450 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4004e00 con 0x7f4cb4101de0 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.786+0000 7f4cba84f640 1 -- 172.21.15.89:0/932333450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb41a4f50 con 0x7f4cb4101de0 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cb1ffb640 1 -- 172.21.15.89:0/932333450 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4ca4004fa0 con 0x7f4cb4101de0 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cbcc28740 1 Processor -- start 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cbcc28740 1 -- start start 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce30b40 0x560c3ce2f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce2f5d0 0x560c3ce2fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce3c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560c3cd952f0 con 0x560c3ce3a370 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560c3cd3ccf0 con 0x560c3ce30b40 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560c3cd912b0 con 0x560c3ce2f5d0 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cabfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce3c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cabfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce3c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59414/0 (socket says 172.21.15.89:59414) 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cabfff640 1 -- 172.21.15.89:0/3249773699 learned_addr learned my addr 172.21.15.89:0/3249773699 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cb1ffb640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ca00a32c0 0x7f4ca00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.788+0000 7f4cb8dec640 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce30b40 0x560c3ce2f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce2f5d0 0x560c3ce2fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cabfff640 1 -- 172.21.15.89:0/3249773699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce2f5d0 msgr2=0x560c3ce2fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cabfff640 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce2f5d0 0x560c3ce2fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cabfff640 1 -- 172.21.15.89:0/3249773699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce30b40 msgr2=0x560c3ce2f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cabfff640 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce30b40 0x560c3ce2f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cabfff640 1 -- 172.21.15.89:0/3249773699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c3cd213a0 con 0x560c3ce3a370 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cb8dec640 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce30b40 0x560c3ce2f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce2f5d0 0x560c3ce2fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cb8dec640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ca00a32c0 0x7f4ca00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cb1ffb640 1 -- 172.21.15.89:0/932333450 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4ca40cf6f0 con 0x7f4cb4101de0 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4cabfff640 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce3c760 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00bc30 tx=0x7f4c9c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a64f6a30b657350 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4caa7fc640 1 -- 172.21.15.89:0/3249773699 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c01c480 con 0x560c3ce3a370 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4caa7fc640 1 -- 172.21.15.89:0/3249773699 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c9c01cb00 con 0x560c3ce3a370 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.789+0000 7f4caa7fc640 1 -- 172.21.15.89:0/3249773699 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c011ba0 con 0x560c3ce3a370 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3249773699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 msgr2=0x560c3ce3c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce3c760 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00bc30 tx=0x7f4c9c0107a0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cb8dec640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ca00a32c0 0x7f4ca00a5780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4cac005cc0 tx=0x7f4cac0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3249773699 shutdown_connections 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce3c760 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce2f5d0 0x560c3ce2fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/3249773699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce30b40 0x560c3ce2f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3249773699 >> 172.21.15.89:0/3249773699 conn(0x560c3cdf8310 msgr2=0x560c3ce2bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3249773699 shutdown_connections 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3249773699 wait complete. 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 -- start start 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.790+0000 7f4cbcc28740 1 Processor -- start 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce39d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce30b40 0x560c3ce37040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce3a370 0x560c3ce37650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560c3ce3d070 con 0x560c3ce2f5d0 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560c3cd50060 con 0x560c3ce30b40 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560c3ce3d1f0 con 0x560c3ce3a370 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cbcc28740 10 client.0 _ll_register_callbacks cb 0x560c3caa38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:35.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce39d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce39d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59420/0 (socket says 172.21.15.89:59420) 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb3fff640 1 -- 172.21.15.89:0/2921024434 learned_addr learned my addr 172.21.15.89:0/2921024434 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb8dec640 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce3a370 0x560c3ce37650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb3fff640 1 -- 172.21.15.89:0/2921024434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce3a370 msgr2=0x560c3ce37650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce3a370 0x560c3ce37650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb3fff640 1 -- 172.21.15.89:0/2921024434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce30b40 msgr2=0x560c3ce37040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce30b40 0x560c3ce37040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb3fff640 1 -- 172.21.15.89:0/2921024434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c3cb063c0 con 0x560c3ce2f5d0 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb8dec640 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce3a370 0x560c3ce37650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce39d90 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4ca409b8b0 tx=0x7f4ca406b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fffba867c5f1071 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4c937fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4c937fe640 1 -- 172.21.15.89:0/2921024434 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4020070 con 0x560c3ce2f5d0 2024-09-15T23:27:35.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4c937fe640 1 -- 172.21.15.89:0/2921024434 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4ca4099c20 con 0x560c3ce2f5d0 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cbcc28740 1 -- 172.21.15.89:0/2921024434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c9c01cb00 con 0x560c3ce2f5d0 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4c937fe640 1 -- 172.21.15.89:0/2921024434 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4ca409c020 con 0x560c3ce2f5d0 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cbcc28740 10 client.16722 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.791+0000 7f4cbcc28740 1 -- 172.21.15.89:0/2921024434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560c3ce48ab0 con 0x560c3ce2f5d0 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.792+0000 7f4c937fe640 1 -- 172.21.15.89:0/2921024434 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c9c01cb00 con 0x560c3ce2f5d0 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.792+0000 7f4c937fe640 1 -- 172.21.15.89:0/2921024434 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4ca4097f90 con 0x560c3ce2f5d0 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.792+0000 7f4cbcc28740 10 client.16722 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.792+0000 7f4cbcc28740 10 client.16722 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.792+0000 7f4cbcc28740 20 client.16722 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.792+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x560c3ce49fc0 0x560c3ce4c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.792+0000 7f4cbcc28740 4 client.16722 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.792+0000 7f4cbcc28740 1 -- 172.21.15.89:0/2921024434 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c3cda2860 con 0x560c3ce49fc0 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.792+0000 7f4cabfff640 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x560c3ce49fc0 0x560c3ce4c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.793+0000 7f4c937fe640 10 client.16722 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.793+0000 7f4cabfff640 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x560c3ce49fc0 0x560c3ce4c410 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=50a6c530271de933 server_cookie=842ea41213743ea in_seq=0 out_seq=0 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.793+0000 7f4c937fe640 1 -- 172.21.15.89:0/2921024434 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560c3cda2860 con 0x560c3ce49fc0 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.793+0000 7f4c937fe640 10 client.16722 handle_command_reply: tid=0 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.793+0000 7f4cbcc28740 1 client.16722 shutdown 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 1 -- 172.21.15.89:0/2921024434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 msgr2=0x560c3ce39d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce39d90 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4ca409b8b0 tx=0x7f4ca406b590 comp rx=0 tx=0).stop 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 1 -- 172.21.15.89:0/2921024434 shutdown_connections 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x560c3ce49fc0 0x560c3ce4c410 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce3a370 0x560c3ce37650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4c937fe640 0 client.16722 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce30b40 0x560c3ce37040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/2921024434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce39d90 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 1 -- 172.21.15.89:0/2921024434 >> 172.21.15.89:0/2921024434 conn(0x560c3cdf8310 msgr2=0x560c3ce31960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 1 -- 172.21.15.89:0/2921024434 shutdown_connections 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 1 -- 172.21.15.89:0/2921024434 wait complete. 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.794+0000 7f4cbcc28740 20 client.16722 trim_cache size 0 max 16384 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cbcc28740 1 Processor -- start 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cbcc28740 1 -- start start 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce2f5d0 0x560c3ce30300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce30b40 0x560c3ce46310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce2b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb41aee50 con 0x560c3ce3a370 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb41b0990 con 0x560c3ce2f5d0 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb41af520 con 0x560c3ce30b40 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cb8dec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce2b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cb8dec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce2b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59430/0 (socket says 172.21.15.89:59430) 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cb8dec640 1 -- 172.21.15.89:0/1223449960 learned_addr learned my addr 172.21.15.89:0/1223449960 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cabfff640 1 --2- 172.21.15.89:0/1223449960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce30b40 0x560c3ce46310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cb8dec640 1 -- 172.21.15.89:0/1223449960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce30b40 msgr2=0x560c3ce46310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/1223449960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce2f5d0 0x560c3ce30300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cb8dec640 1 --2- 172.21.15.89:0/1223449960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce30b40 0x560c3ce46310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cb8dec640 1 -- 172.21.15.89:0/1223449960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce2f5d0 msgr2=0x560c3ce30300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cb8dec640 1 --2- 172.21.15.89:0/1223449960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce2f5d0 0x560c3ce30300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.807+0000 7f4cb8dec640 1 -- 172.21.15.89:0/1223449960 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c3ce2c960 con 0x560c3ce3a370 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4cb8dec640 1 --2- 172.21.15.89:0/1223449960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce2b700 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4cac031e20 tx=0x7f4cac03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112f864d4976ce2f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4ca9ffb640 1 -- 172.21.15.89:0/1223449960 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac050070 con 0x560c3ce3a370 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4ca9ffb640 1 -- 172.21.15.89:0/1223449960 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cac036040 con 0x560c3ce3a370 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4ca9ffb640 1 -- 172.21.15.89:0/1223449960 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cac0047b0 con 0x560c3ce3a370 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4cbcc28740 1 -- 172.21.15.89:0/1223449960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 msgr2=0x560c3ce2b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/1223449960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce2b700 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f4cac031e20 tx=0x7f4cac03f000 comp rx=0 tx=0).stop 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4cbcc28740 1 -- 172.21.15.89:0/1223449960 shutdown_connections 2024-09-15T23:27:35.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/1223449960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce3a370 0x560c3ce2b700 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/1223449960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce30b40 0x560c3ce46310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/1223449960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce2f5d0 0x560c3ce30300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4cbcc28740 1 -- 172.21.15.89:0/1223449960 >> 172.21.15.89:0/1223449960 conn(0x560c3cdf8310 msgr2=0x560c3ce26a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4cbcc28740 1 -- 172.21.15.89:0/1223449960 shutdown_connections 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.808+0000 7f4cbcc28740 1 -- 172.21.15.89:0/1223449960 wait complete. 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cbcc28740 1 -- start start 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cbcc28740 1 Processor -- start 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce3dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce30b40 0x560c3ce3e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cbcc28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce3a370 0x560c3ceacc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb41b02c0 con 0x560c3ce2f5d0 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb41afbf0 con 0x560c3ce3a370 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cbcc28740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb41ace30 con 0x560c3ce30b40 2024-09-15T23:27:35.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cbcc28740 10 client.0 _ll_register_callbacks cb 0x7f4ca4009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cb3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce3dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cb3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce3dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59434/0 (socket says 172.21.15.89:59434) 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.809+0000 7f4cb3fff640 1 -- 172.21.15.89:0/3086522218 learned_addr learned my addr 172.21.15.89:0/3086522218 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cabfff640 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce30b40 0x560c3ce3e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cb3fff640 1 -- 172.21.15.89:0/3086522218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce30b40 msgr2=0x560c3ce3e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce30b40 0x560c3ce3e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cb3fff640 1 -- 172.21.15.89:0/3086522218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce3a370 msgr2=0x560c3ceacc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce3a370 0x560c3ceacc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cb3fff640 1 -- 172.21.15.89:0/3086522218 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c3cead2e0 con 0x560c3ce2f5d0 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cb3fff640 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce3dbe0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4097d80 tx=0x7f4ca409b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4b7b99a411f694 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4c937fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4c937fe640 1 -- 172.21.15.89:0/3086522218 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4020070 con 0x560c3ce2f5d0 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4c937fe640 1 -- 172.21.15.89:0/3086522218 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4ca4005290 con 0x560c3ce2f5d0 2024-09-15T23:27:35.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3086522218 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cac036040 con 0x560c3ce2f5d0 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4c937fe640 1 -- 172.21.15.89:0/3086522218 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4ca40a4590 con 0x560c3ce2f5d0 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cbcc28740 10 client.16728 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3086522218 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560c3ceb4660 con 0x560c3ce2f5d0 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4c937fe640 1 -- 172.21.15.89:0/3086522218 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cac036040 con 0x560c3ce2f5d0 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4c937fe640 1 -- 172.21.15.89:0/3086522218 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4ca4097700 con 0x560c3ce2f5d0 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cbcc28740 10 client.16728 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cbcc28740 10 client.16728 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cbcc28740 20 client.16728 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x560c3ce49fc0 0x560c3ceb6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cbcc28740 4 client.16728 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.810+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3086522218 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560c3cd9cd70 con 0x560c3ce49fc0 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.811+0000 7f4cabfff640 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x560c3ce49fc0 0x560c3ceb6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.811+0000 7f4c937fe640 10 client.16728 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:27:35.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.811+0000 7f4cabfff640 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x560c3ce49fc0 0x560c3ceb6c30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4ae0c6c02aaf06e7 server_cookie=812d1e056d698c7d in_seq=0 out_seq=0 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4c937fe640 1 -- 172.21.15.89:0/3086522218 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560c3cd9cd70 con 0x560c3ce49fc0 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4c937fe640 10 client.16728 handle_command_reply: tid=0 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 client.16728 shutdown 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3086522218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 msgr2=0x560c3ce3dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce3dbe0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4097d80 tx=0x7f4ca409b9a0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3086522218 shutdown_connections 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x560c3ce49fc0 0x560c3ceb6c30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560c3ce3a370 0x560c3ceacc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560c3ce30b40 0x560c3ce3e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 --2- 172.21.15.89:0/3086522218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560c3ce2f5d0 0x560c3ce3dbe0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3086522218 >> 172.21.15.89:0/3086522218 conn(0x560c3cdf8310 msgr2=0x560c3ce2b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4c937fe640 0 client.16728 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3086522218 shutdown_connections 2024-09-15T23:27:35.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 1 -- 172.21.15.89:0/3086522218 wait complete. 2024-09-15T23:27:35.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.812+0000 7f4cbcc28740 20 client.16728 trim_cache size 0 max 16384 2024-09-15T23:27:35.829 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 -- 172.21.15.89:0/932333450 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ca00a32c0 msgr2=0x7f4ca00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ca00a32c0 0x7f4ca00a5780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4cac005cc0 tx=0x7f4cac0095c0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 -- 172.21.15.89:0/932333450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb4101de0 msgr2=0x7f4cb41983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb4101de0 0x7f4cb41983d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400d770 tx=0x7f4ca400dc30 comp rx=0 tx=0).stop 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 -- 172.21.15.89:0/932333450 shutdown_connections 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ca00a32c0 0x7f4ca00a5780 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4cb4106f10 0x7f4cb419cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cb4102790 0x7f4cb4198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 --2- 172.21.15.89:0/932333450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4cb4101de0 0x7f4cb41983d0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 -- 172.21.15.89:0/932333450 >> 172.21.15.89:0/932333450 conn(0x7f4cb40fd8c0 msgr2=0x7f4cb40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 -- 172.21.15.89:0/932333450 shutdown_connections 2024-09-15T23:27:35.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:35.813+0000 7f4cba84f640 1 -- 172.21.15.89:0/932333450 wait complete. 2024-09-15T23:27:36.648 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:27:36.893 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:37.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 -- 172.21.15.89:0/3909444111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04100a80 msgr2=0x7f4c04100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:37.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 --2- 172.21.15.89:0/3909444111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04100a80 0x7f4c04100ee0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f4c00009930 tx=0x7f4c0002a420 comp rx=0 tx=0).stop 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 -- 172.21.15.89:0/3909444111 shutdown_connections 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 --2- 172.21.15.89:0/3909444111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c04101420 0x7f4c0410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 --2- 172.21.15.89:0/3909444111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04100a80 0x7f4c04100ee0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 --2- 172.21.15.89:0/3909444111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c04106a80 0x7f4c04106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 -- 172.21.15.89:0/3909444111 >> 172.21.15.89:0/3909444111 conn(0x7f4c040fc7a0 msgr2=0x7f4c040febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 -- 172.21.15.89:0/3909444111 shutdown_connections 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 -- 172.21.15.89:0/3909444111 wait complete. 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 Processor -- start 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.412+0000 7f4c0bb56640 1 -- start start 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0bb56640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c04100a80 0x7f4c040fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0bb56640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c04101420 0x7f4c040fd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0bb56640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04106a80 0x7f4c040fd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0bb56640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04110e80 con 0x7f4c04106a80 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0bb56640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04110d00 con 0x7f4c04101420 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0bb56640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c04111000 con 0x7f4c04100a80 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0a0f3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04106a80 0x7f4c040fd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0a0f3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04106a80 0x7f4c040fd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59458/0 (socket says 172.21.15.89:59458) 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0a0f3640 1 -- 172.21.15.89:0/1635417060 learned_addr learned my addr 172.21.15.89:0/1635417060 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c090f1640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c04101420 0x7f4c040fd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c098f2640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c04100a80 0x7f4c040fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0a0f3640 1 -- 172.21.15.89:0/1635417060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c04100a80 msgr2=0x7f4c040fcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0a0f3640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c04100a80 0x7f4c040fcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0a0f3640 1 -- 172.21.15.89:0/1635417060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c04101420 msgr2=0x7f4c040fd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:37.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0a0f3640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c04101420 0x7f4c040fd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0a0f3640 1 -- 172.21.15.89:0/1635417060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c041a6900 con 0x7f4c04106a80 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0a0f3640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04106a80 0x7f4c040fd810 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00cce0 tx=0x7f4bfc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4bf6ffd640 1 -- 172.21.15.89:0/1635417060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc00f450 con 0x7f4c04106a80 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4bf6ffd640 1 -- 172.21.15.89:0/1635417060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bfc00fad0 con 0x7f4c04106a80 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0bb56640 1 -- 172.21.15.89:0/1635417060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c041a6bf0 con 0x7f4c04106a80 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4bf6ffd640 1 -- 172.21.15.89:0/1635417060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc018c10 con 0x7f4c04106a80 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.413+0000 7f4c0bb56640 1 -- 172.21.15.89:0/1635417060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c041a70b0 con 0x7f4c04106a80 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.416+0000 7f4bf6ffd640 1 -- 172.21.15.89:0/1635417060 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4bfc00f5f0 con 0x7f4c04106a80 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.416+0000 7f4c0df2f740 1 Processor -- start 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.416+0000 7f4c0df2f740 1 -- start start 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.416+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74d3b40 0x5567d74d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.416+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 0x5567d74d2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.416+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74dd370 0x5567d74df760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.416+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5567d74382f0 con 0x5567d74d3b40 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.416+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5567d73dfcf0 con 0x5567d74d25d0 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.416+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5567d74342b0 con 0x5567d74dd370 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c090f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74d3b40 0x5567d74d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c090f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74d3b40 0x5567d74d2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59472/0 (socket says 172.21.15.89:59472) 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c090f1640 1 -- 172.21.15.89:0/1063014634 learned_addr learned my addr 172.21.15.89:0/1063014634 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4bf6ffd640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bf00a3240 0x7f4bf00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c090f1640 1 --2- 172.21.15.89:0/1063014634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74dd370 0x5567d74df760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c0a0f3640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bf00a3240 0x7f4bf00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c098f2640 1 --2- 172.21.15.89:0/1063014634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 0x5567d74d2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c090f1640 1 -- 172.21.15.89:0/1063014634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74dd370 msgr2=0x5567d74df760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c090f1640 1 --2- 172.21.15.89:0/1063014634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74dd370 0x5567d74df760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c090f1640 1 -- 172.21.15.89:0/1063014634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 msgr2=0x5567d74d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c090f1640 1 --2- 172.21.15.89:0/1063014634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 0x5567d74d2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4bf6ffd640 1 -- 172.21.15.89:0/1635417060 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4bfc0d0340 con 0x7f4c04106a80 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c090f1640 1 -- 172.21.15.89:0/1063014634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567d73c43a0 con 0x5567d74d3b40 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c098f2640 1 --2- 172.21.15.89:0/1063014634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 0x5567d74d2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c090f1640 1 --2- 172.21.15.89:0/1063014634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74d3b40 0x5567d74d2090 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f4c00005ec0 tx=0x7f4c00005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b38a130f8dc6164 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4c0a0f3640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bf00a3240 0x7f4bf00a5700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0989d0 tx=0x7f4bfc066fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4beffff640 1 -- 172.21.15.89:0/1063014634 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00035040 con 0x5567d74d3b40 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4beffff640 1 -- 172.21.15.89:0/1063014634 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c00005020 con 0x5567d74d3b40 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.417+0000 7f4beffff640 1 -- 172.21.15.89:0/1063014634 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00004bc0 con 0x5567d74d3b40 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.418+0000 7f4c0df2f740 1 -- 172.21.15.89:0/1063014634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74d3b40 msgr2=0x5567d74d2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.418+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/1063014634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74d3b40 0x5567d74d2090 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f4c00005ec0 tx=0x7f4c00005fd0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.418+0000 7f4c0df2f740 1 -- 172.21.15.89:0/1063014634 shutdown_connections 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.418+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/1063014634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74dd370 0x5567d74df760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.418+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/1063014634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 0x5567d74d2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.418+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/1063014634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74d3b40 0x5567d74d2090 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.418+0000 7f4c0df2f740 1 -- 172.21.15.89:0/1063014634 >> 172.21.15.89:0/1063014634 conn(0x5567d749b310 msgr2=0x5567d74cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.418+0000 7f4c0df2f740 1 -- 172.21.15.89:0/1063014634 shutdown_connections 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.418+0000 7f4c0df2f740 1 -- 172.21.15.89:0/1063014634 wait complete. 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.418+0000 7f4c0df2f740 1 -- start start 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c0df2f740 1 Processor -- start 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d25d0 0x5567d74dcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 0x5567d74d9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74da570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5567d74e0010 con 0x5567d74dd370 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5567d73f3060 con 0x5567d74d3b40 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5567d74e0190 con 0x5567d74d25d0 2024-09-15T23:27:37.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c0df2f740 10 client.0 _ll_register_callbacks cb 0x5567d71468c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c090f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74da570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c090f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74da570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59488/0 (socket says 172.21.15.89:59488) 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c098f2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 0x5567d74d9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c090f1640 1 -- 172.21.15.89:0/3147249185 learned_addr learned my addr 172.21.15.89:0/3147249185 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c090f1640 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d25d0 0x5567d74dcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c090f1640 1 -- 172.21.15.89:0/3147249185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d25d0 msgr2=0x5567d74dcd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c090f1640 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d25d0 0x5567d74dcd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c090f1640 1 -- 172.21.15.89:0/3147249185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 msgr2=0x5567d74d9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c090f1640 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 0x5567d74d9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.419+0000 7f4c090f1640 1 -- 172.21.15.89:0/3147249185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567d71a93c0 con 0x5567d74dd370 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c098f2640 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 0x5567d74d9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c090f1640 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74da570 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4c000059b0 tx=0x7f4c000047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe32ec6309b31625 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:37.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4bed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3147249185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00052020 con 0x5567d74dd370 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3147249185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4c000053e0 con 0x5567d74dd370 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3147249185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c00005020 con 0x5567d74dd370 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3147249185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4c00035040 con 0x5567d74dd370 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c0df2f740 10 client.16740 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3147249185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5567d74eba00 con 0x5567d74dd370 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3147249185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c00005020 con 0x5567d74dd370 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3147249185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4c00003ae0 con 0x5567d74dd370 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c0df2f740 10 client.16740 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c0df2f740 10 client.16740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c0df2f740 20 client.16740 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5567d74ecd50 0x5567d74ef140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c0df2f740 4 client.16740 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.420+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3147249185 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567d7445860 con 0x5567d74ecd50 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.421+0000 7f4c098f2640 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5567d74ecd50 0x5567d74ef140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.421+0000 7f4c098f2640 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5567d74ecd50 0x5567d74ef140 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dd101fb126ea570 server_cookie=2d5fae4013d3d057 in_seq=0 out_seq=0 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.421+0000 7f4bed7fa640 10 client.16740 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3147249185 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5567d7445860 con 0x5567d74ecd50 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4bed7fa640 10 client.16740 handle_command_reply: tid=0 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 client.16740 shutdown 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3147249185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 msgr2=0x5567d74da570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74da570 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4c000059b0 tx=0x7f4c000047a0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3147249185 shutdown_connections 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5567d74ecd50 0x5567d74ef140 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74da570 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 0x5567d74d9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3147249185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d25d0 0x5567d74dcd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4bed7fa640 0 client.16740 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:37.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3147249185 >> 172.21.15.89:0/3147249185 conn(0x5567d749b310 msgr2=0x5567d74d4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:37.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3147249185 shutdown_connections 2024-09-15T23:27:37.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3147249185 wait complete. 2024-09-15T23:27:37.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.422+0000 7f4c0df2f740 20 client.16740 trim_cache size 0 max 16384 2024-09-15T23:27:37.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.435+0000 7f4c0df2f740 1 Processor -- start 2024-09-15T23:27:37.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.435+0000 7f4c0df2f740 1 -- start start 2024-09-15T23:27:37.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.435+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 0x5567d74cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.435+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d3b40 0x5567d74f2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.435+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.435+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b0f40 con 0x5567d74dd370 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.435+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b2a80 con 0x5567d74d25d0 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.435+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b1610 con 0x5567d74d3b40 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c090f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c090f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74f26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59498/0 (socket says 172.21.15.89:59498) 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c090f1640 1 -- 172.21.15.89:0/270370131 learned_addr learned my addr 172.21.15.89:0/270370131 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c090f1640 1 --2- 172.21.15.89:0/270370131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 0x5567d74cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c098f2640 1 --2- 172.21.15.89:0/270370131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d3b40 0x5567d74f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c090f1640 1 -- 172.21.15.89:0/270370131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d3b40 msgr2=0x5567d74f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c090f1640 1 --2- 172.21.15.89:0/270370131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d3b40 0x5567d74f2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c090f1640 1 -- 172.21.15.89:0/270370131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 msgr2=0x5567d74cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c090f1640 1 --2- 172.21.15.89:0/270370131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 0x5567d74cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c090f1640 1 -- 172.21.15.89:0/270370131 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567d74d5120 con 0x5567d74dd370 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4c090f1640 1 --2- 172.21.15.89:0/270370131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74f26a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4c00005df0 tx=0x7f4c00000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb134fd3f57262f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4bef7fe640 1 -- 172.21.15.89:0/270370131 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0005e040 con 0x5567d74dd370 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4bef7fe640 1 -- 172.21.15.89:0/270370131 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c0005f040 con 0x5567d74dd370 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.436+0000 7f4bef7fe640 1 -- 172.21.15.89:0/270370131 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00052040 con 0x5567d74dd370 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 -- 172.21.15.89:0/270370131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 msgr2=0x5567d74f26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/270370131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74f26a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4c00005df0 tx=0x7f4c00000bc0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 -- 172.21.15.89:0/270370131 shutdown_connections 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/270370131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74f26a0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/270370131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d3b40 0x5567d74f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/270370131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d25d0 0x5567d74cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 -- 172.21.15.89:0/270370131 >> 172.21.15.89:0/270370131 conn(0x5567d749b310 msgr2=0x5567d74dc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 -- 172.21.15.89:0/270370131 shutdown_connections 2024-09-15T23:27:37.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 -- 172.21.15.89:0/270370131 wait complete. 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 -- start start 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.437+0000 7f4c0df2f740 1 Processor -- start 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d25d0 0x5567d74f24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 0x5567d74f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c0df2f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74efa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b23b0 con 0x5567d74dd370 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041b1ce0 con 0x5567d74d3b40 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c0df2f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c041aef20 con 0x5567d74d25d0 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c0df2f740 10 client.0 _ll_register_callbacks cb 0x7f4c00031970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c090f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74efa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c090f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74efa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59514/0 (socket says 172.21.15.89:59514) 2024-09-15T23:27:37.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c090f1640 1 -- 172.21.15.89:0/3081548190 learned_addr learned my addr 172.21.15.89:0/3081548190 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c098f2640 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 0x5567d74f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c090f1640 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d25d0 0x5567d74f24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c090f1640 1 -- 172.21.15.89:0/3081548190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d25d0 msgr2=0x5567d74f24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c090f1640 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d25d0 0x5567d74f24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c090f1640 1 -- 172.21.15.89:0/3081548190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 msgr2=0x5567d74f2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c090f1640 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 0x5567d74f2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c090f1640 1 -- 172.21.15.89:0/3081548190 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567d74f02b0 con 0x5567d74dd370 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4c090f1640 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74efa80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f4c00005ec0 tx=0x7f4c00005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd30476664b4f51 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.438+0000 7f4bed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3081548190 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00052070 con 0x5567d74dd370 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3081548190 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c0005f040 con 0x5567d74dd370 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3081548190 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4c00049ac0 con 0x5567d74dd370 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3081548190 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4c0005e040 con 0x5567d74dd370 2024-09-15T23:27:37.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4c0df2f740 10 client.16746 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3081548190 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5567d75572d0 con 0x5567d74dd370 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3081548190 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c0005f040 con 0x5567d74dd370 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3081548190 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4c000035f0 con 0x5567d74dd370 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4c0df2f740 10 client.16746 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4c0df2f740 10 client.16746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4c0df2f740 20 client.16746 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5567d75577f0 0x5567d7559be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4c0df2f740 4 client.16746 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3081548190 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5567d743fd70 con 0x5567d75577f0 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.439+0000 7f4c098f2640 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5567d75577f0 0x5567d7559be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4c098f2640 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5567d75577f0 0x5567d7559be0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb85dc0ed450aa97 server_cookie=91ee3fd6198b5717 in_seq=0 out_seq=0 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4bed7fa640 10 client.16746 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4bed7fa640 1 -- 172.21.15.89:0/3081548190 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5567d743fd70 con 0x5567d75577f0 2024-09-15T23:27:37.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4bed7fa640 10 client.16746 handle_command_reply: tid=0 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4c0df2f740 1 client.16746 shutdown 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3081548190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 msgr2=0x5567d74efa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74efa80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f4c00005ec0 tx=0x7f4c00005fd0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3081548190 shutdown_connections 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5567d75577f0 0x5567d7559be0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5567d74dd370 0x5567d74efa80 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567d74d3b40 0x5567d74f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4c0df2f740 1 --2- 172.21.15.89:0/3081548190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5567d74d25d0 0x5567d74f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3081548190 >> 172.21.15.89:0/3081548190 conn(0x5567d749b310 msgr2=0x5567d74cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.440+0000 7f4bed7fa640 0 client.16746 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3081548190 shutdown_connections 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0df2f740 1 -- 172.21.15.89:0/3081548190 wait complete. 2024-09-15T23:27:37.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0df2f740 20 client.16746 trim_cache size 0 max 16384 2024-09-15T23:27:37.453 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:27:37.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0bb56640 1 -- 172.21.15.89:0/1635417060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bf00a3240 msgr2=0x7f4bf00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0bb56640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bf00a3240 0x7f4bf00a5700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0989d0 tx=0x7f4bfc066fb0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0bb56640 1 -- 172.21.15.89:0/1635417060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04106a80 msgr2=0x7f4c040fd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0bb56640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04106a80 0x7f4c040fd810 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00cce0 tx=0x7f4bfc00a620 comp rx=0 tx=0).stop 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0bb56640 1 -- 172.21.15.89:0/1635417060 shutdown_connections 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0bb56640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4bf00a3240 0x7f4bf00a5700 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0bb56640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4c04106a80 0x7f4c040fd810 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0bb56640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c04101420 0x7f4c040fd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0bb56640 1 --2- 172.21.15.89:0/1635417060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c04100a80 0x7f4c040fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.441+0000 7f4c0bb56640 1 -- 172.21.15.89:0/1635417060 >> 172.21.15.89:0/1635417060 conn(0x7f4c040fc7a0 msgr2=0x7f4c0406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.442+0000 7f4c0bb56640 1 -- 172.21.15.89:0/1635417060 shutdown_connections 2024-09-15T23:27:37.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:37.442+0000 7f4c0bb56640 1 -- 172.21.15.89:0/1635417060 wait complete. 2024-09-15T23:27:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:38 smithi120 ceph-mon[35250]: pgmap v458: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-09-15T23:27:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:38 smithi174 ceph-mon[35586]: pgmap v458: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-09-15T23:27:38.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:38 smithi089 ceph-mon[30933]: pgmap v458: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-09-15T23:27:39.323 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:27:39.324 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:27:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:39 smithi120 ceph-mon[35250]: pgmap v459: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:27:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:39 smithi174 ceph-mon[35586]: pgmap v459: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:27:39.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:39 smithi089 ceph-mon[30933]: pgmap v459: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:27:39.585 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.058+0000 7fbb40051640 1 -- 172.21.15.89:0/1313062517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38106f10 msgr2=0x7fbb3810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.058+0000 7fbb40051640 1 --2- 172.21.15.89:0/1313062517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38106f10 0x7fbb3810ea90 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009ca0 tx=0x7fbb3002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.058+0000 7fbb40051640 1 -- 172.21.15.89:0/1313062517 shutdown_connections 2024-09-15T23:27:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.058+0000 7fbb40051640 1 --2- 172.21.15.89:0/1313062517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38106f10 0x7fbb3810ea90 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.058+0000 7fbb40051640 1 --2- 172.21.15.89:0/1313062517 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb38102790 0x7fbb381067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.058+0000 7fbb40051640 1 --2- 172.21.15.89:0/1313062517 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38101de0 0x7fbb381021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.058+0000 7fbb40051640 1 -- 172.21.15.89:0/1313062517 >> 172.21.15.89:0/1313062517 conn(0x7fbb380fd8c0 msgr2=0x7fbb380ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb40051640 1 -- 172.21.15.89:0/1313062517 shutdown_connections 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb40051640 1 -- 172.21.15.89:0/1313062517 wait complete. 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb40051640 1 Processor -- start 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb40051640 1 -- start start 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb40051640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb38101de0 0x7fbb38198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb40051640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38102790 0x7fbb38198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb40051640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38106f10 0x7fbb3819cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb40051640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb38111fb0 con 0x7fbb38102790 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb40051640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb38111e30 con 0x7fbb38106f10 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb40051640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb38112130 con 0x7fbb38101de0 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb3d5ec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38102790 0x7fbb38198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb3d5ec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38102790 0x7fbb38198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59534/0 (socket says 172.21.15.89:59534) 2024-09-15T23:27:40.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.059+0000 7fbb3d5ec640 1 -- 172.21.15.89:0/3573312015 learned_addr learned my addr 172.21.15.89:0/3573312015 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb3e5ee640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38106f10 0x7fbb3819cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb3dded640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb38101de0 0x7fbb38198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb3d5ec640 1 -- 172.21.15.89:0/3573312015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb38101de0 msgr2=0x7fbb38198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb3d5ec640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb38101de0 0x7fbb38198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb3d5ec640 1 -- 172.21.15.89:0/3573312015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38106f10 msgr2=0x7fbb3819cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb3d5ec640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38106f10 0x7fbb3819cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb3d5ec640 1 -- 172.21.15.89:0/3573312015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3819d4c0 con 0x7fbb38102790 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb3dded640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb38101de0 0x7fbb38198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb3d5ec640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38102790 0x7fbb38198a50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fbb3400d770 tx=0x7fbb3400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb2effd640 1 -- 172.21.15.89:0/3573312015 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb34017070 con 0x7fbb38102790 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb40051640 1 -- 172.21.15.89:0/3573312015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3819d7b0 con 0x7fbb38102790 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb2effd640 1 -- 172.21.15.89:0/3573312015 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb34005500 con 0x7fbb38102790 2024-09-15T23:27:40.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb2effd640 1 -- 172.21.15.89:0/3573312015 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb34004e00 con 0x7fbb38102790 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.060+0000 7fbb40051640 1 -- 172.21.15.89:0/3573312015 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb381a5090 con 0x7fbb38102790 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.062+0000 7fbb2effd640 1 -- 172.21.15.89:0/3573312015 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbb34004fa0 con 0x7fbb38102790 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb4242a740 1 Processor -- start 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb4242a740 1 -- start start 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb2effd640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb240a3240 0x7fbb240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868d0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868d0d40 0x5584868d3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb4242a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55848682b280 con 0x5584868c5b60 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb4242a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5584867d2b10 con 0x5584868c6480 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb4242a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558486827240 con 0x5584868d0d40 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb2effd640 1 -- 172.21.15.89:0/3573312015 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb340cf830 con 0x7fbb38102790 2024-09-15T23:27:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb3e5ee640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb240a3240 0x7fbb240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb3dded640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb3dded640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59544/0 (socket says 172.21.15.89:59544) 2024-09-15T23:27:40.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.063+0000 7fbb3dded640 1 -- 172.21.15.89:0/271452037 learned_addr learned my addr 172.21.15.89:0/271452037 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:40.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.064+0000 7fbb3dded640 1 -- 172.21.15.89:0/271452037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868d0d40 msgr2=0x5584868d3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:40.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.064+0000 7fbb3dded640 1 --2- 172.21.15.89:0/271452037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868d0d40 0x5584868d3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.064+0000 7fbb3dded640 1 -- 172.21.15.89:0/271452037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 msgr2=0x5584868d0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:40.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.064+0000 7fbb3dded640 1 --2- 172.21.15.89:0/271452037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868d0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.064+0000 7fbb3dded640 1 -- 172.21.15.89:0/271452037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584867df1f0 con 0x5584868c5b60 2024-09-15T23:27:40.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.064+0000 7fbb3e5ee640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb240a3240 0x7fbb240a5700 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbb30005fe0 tx=0x7fbb30005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:40.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.064+0000 7fbb3dded640 1 --2- 172.21.15.89:0/271452037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c5f40 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800aa10 tx=0x7fbb2800aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e7236e7bf41e56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:40.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.064+0000 7fbb23fff640 1 -- 172.21.15.89:0/271452037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28015070 con 0x5584868c5b60 2024-09-15T23:27:40.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.064+0000 7fbb23fff640 1 -- 172.21.15.89:0/271452037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2800cc60 con 0x5584868c5b60 2024-09-15T23:27:40.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.064+0000 7fbb23fff640 1 -- 172.21.15.89:0/271452037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb280185c0 con 0x5584868c5b60 2024-09-15T23:27:40.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 -- 172.21.15.89:0/271452037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 msgr2=0x5584868c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:40.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 --2- 172.21.15.89:0/271452037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c5f40 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800aa10 tx=0x7fbb2800aed0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 -- 172.21.15.89:0/271452037 shutdown_connections 2024-09-15T23:27:40.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 --2- 172.21.15.89:0/271452037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868d0d40 0x5584868d3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 --2- 172.21.15.89:0/271452037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868d0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 --2- 172.21.15.89:0/271452037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c5f40 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 -- 172.21.15.89:0/271452037 >> 172.21.15.89:0/271452037 conn(0x5584868c20f0 msgr2=0x5584868c24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 -- 172.21.15.89:0/271452037 shutdown_connections 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 -- 172.21.15.89:0/271452037 wait complete. 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 -- start start 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.065+0000 7fbb4242a740 1 Processor -- start 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c5b60 0x5584868cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868ce100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868d0d40 0x5584868ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb4242a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5584868d3910 con 0x5584868d0d40 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb4242a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5584867e5f10 con 0x5584868c6480 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb4242a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5584868d3a90 con 0x5584868c5b60 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb4242a740 10 client.0 _ll_register_callbacks cb 0x5584867aea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3e5ee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868d0d40 0x5584868ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3e5ee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868d0d40 0x5584868ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59552/0 (socket says 172.21.15.89:59552) 2024-09-15T23:27:40.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3e5ee640 1 -- 172.21.15.89:0/3534312533 learned_addr learned my addr 172.21.15.89:0/3534312533 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:40.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3d5ec640 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3dded640 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c5b60 0x5584868cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3e5ee640 1 -- 172.21.15.89:0/3534312533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c5b60 msgr2=0x5584868cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:40.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3e5ee640 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c5b60 0x5584868cdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3e5ee640 1 -- 172.21.15.89:0/3534312533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 msgr2=0x5584868ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:40.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3e5ee640 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3e5ee640 1 -- 172.21.15.89:0/3534312533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55848659c6a0 con 0x5584868d0d40 2024-09-15T23:27:40.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.066+0000 7fbb3d5ec640 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:40.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.067+0000 7fbb3e5ee640 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868d0d40 0x5584868ce640 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009ca0 tx=0x7fbb3002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d8d70854b7933d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.067+0000 7fbb217fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.067+0000 7fbb217fa640 1 -- 172.21.15.89:0/3534312533 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3004c070 con 0x5584868d0d40 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.067+0000 7fbb217fa640 1 -- 172.21.15.89:0/3534312533 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb30004450 con 0x5584868d0d40 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.067+0000 7fbb4242a740 1 -- 172.21.15.89:0/3534312533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb2800cc60 con 0x5584868d0d40 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.067+0000 7fbb217fa640 1 -- 172.21.15.89:0/3534312533 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb30056050 con 0x5584868d0d40 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.067+0000 7fbb4242a740 10 client.16758 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.067+0000 7fbb4242a740 1 -- 172.21.15.89:0/3534312533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5584868df4c0 con 0x5584868d0d40 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.068+0000 7fbb217fa640 1 -- 172.21.15.89:0/3534312533 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2800cc60 con 0x5584868d0d40 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.068+0000 7fbb217fa640 1 -- 172.21.15.89:0/3534312533 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbb30047040 con 0x5584868d0d40 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.068+0000 7fbb4242a740 10 client.16758 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.068+0000 7fbb4242a740 10 client.16758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.068+0000 7fbb4242a740 20 client.16758 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.068+0000 7fbb4242a740 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5584868e09c0 0x5584868e2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.068+0000 7fbb4242a740 4 client.16758 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:40.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.068+0000 7fbb4242a740 1 -- 172.21.15.89:0/3534312533 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584868387f0 con 0x5584868e09c0 2024-09-15T23:27:40.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.068+0000 7fbb3dded640 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5584868e09c0 0x5584868e2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.069+0000 7fbb3dded640 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5584868e09c0 0x5584868e2e30 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1df4b266e03d465 server_cookie=e898d8e8f331af13 in_seq=0 out_seq=0 2024-09-15T23:27:40.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.069+0000 7fbb217fa640 10 client.16758 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:40.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.069+0000 7fbb217fa640 1 -- 172.21.15.89:0/3534312533 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5584868387f0 con 0x5584868e09c0 2024-09-15T23:27:40.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.069+0000 7fbb217fa640 10 client.16758 handle_command_reply: tid=0 2024-09-15T23:27:40.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.069+0000 7fbb4242a740 1 client.16758 shutdown 2024-09-15T23:27:40.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 1 -- 172.21.15.89:0/3534312533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868d0d40 msgr2=0x5584868ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:40.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868d0d40 0x5584868ce640 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009ca0 tx=0x7fbb3002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 1 -- 172.21.15.89:0/3534312533 shutdown_connections 2024-09-15T23:27:40.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5584868e09c0 0x5584868e2e30 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868d0d40 0x5584868ce640 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb217fa640 0 client.16758 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:40.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 1 --2- 172.21.15.89:0/3534312533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c5b60 0x5584868cdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 1 -- 172.21.15.89:0/3534312533 >> 172.21.15.89:0/3534312533 conn(0x5584868c20f0 msgr2=0x5584868c7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:40.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 1 -- 172.21.15.89:0/3534312533 shutdown_connections 2024-09-15T23:27:40.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 1 -- 172.21.15.89:0/3534312533 wait complete. 2024-09-15T23:27:40.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.070+0000 7fbb4242a740 20 client.16758 trim_cache size 0 max 16384 2024-09-15T23:27:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb4242a740 1 Processor -- start 2024-09-15T23:27:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb4242a740 1 -- start start 2024-09-15T23:27:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c6480 0x5584868e8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868d0d40 0x5584868e8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb4242a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381aee30 con 0x5584868c5b60 2024-09-15T23:27:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb4242a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381b0970 con 0x5584868d0d40 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb4242a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381af500 con 0x5584868c6480 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb3dded640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb3dded640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59554/0 (socket says 172.21.15.89:59554) 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb3dded640 1 -- 172.21.15.89:0/2508572594 learned_addr learned my addr 172.21.15.89:0/2508572594 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb3dded640 1 -- 172.21.15.89:0/2508572594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c6480 msgr2=0x5584868e8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb3d5ec640 1 --2- 172.21.15.89:0/2508572594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c6480 0x5584868e8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.087+0000 7fbb3dded640 1 --2- 172.21.15.89:0/2508572594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c6480 0x5584868e8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.088+0000 7fbb3dded640 1 -- 172.21.15.89:0/2508572594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868d0d40 msgr2=0x5584868e8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.088+0000 7fbb3dded640 1 --2- 172.21.15.89:0/2508572594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868d0d40 0x5584868e8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.088+0000 7fbb3dded640 1 -- 172.21.15.89:0/2508572594 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584868c1f20 con 0x5584868c5b60 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.088+0000 7fbb3d5ec640 1 --2- 172.21.15.89:0/2508572594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c6480 0x5584868e8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.088+0000 7fbb3dded640 1 --2- 172.21.15.89:0/2508572594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c85c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fbb28000c00 tx=0x7fbb28011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6eda9b2fe9d9e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.088+0000 7fbb237fe640 1 -- 172.21.15.89:0/2508572594 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28015040 con 0x5584868c5b60 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.088+0000 7fbb237fe640 1 -- 172.21.15.89:0/2508572594 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2800cb20 con 0x5584868c5b60 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.088+0000 7fbb237fe640 1 -- 172.21.15.89:0/2508572594 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2801a6d0 con 0x5584868c5b60 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.088+0000 7fbb4242a740 1 -- 172.21.15.89:0/2508572594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 msgr2=0x5584868c85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.088+0000 7fbb4242a740 1 --2- 172.21.15.89:0/2508572594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c85c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fbb28000c00 tx=0x7fbb28011db0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 -- 172.21.15.89:0/2508572594 shutdown_connections 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 --2- 172.21.15.89:0/2508572594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868d0d40 0x5584868e8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 --2- 172.21.15.89:0/2508572594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868c6480 0x5584868e8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 --2- 172.21.15.89:0/2508572594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868c85c0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 -- 172.21.15.89:0/2508572594 >> 172.21.15.89:0/2508572594 conn(0x5584868c20f0 msgr2=0x5584868ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 -- 172.21.15.89:0/2508572594 shutdown_connections 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 -- 172.21.15.89:0/2508572594 wait complete. 2024-09-15T23:27:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 -- start start 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 Processor -- start 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868d9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868d5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868d0d40 0x5584868d5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381b02a0 con 0x5584868c5b60 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381afbd0 con 0x5584868c6480 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381ace10 con 0x5584868d0d40 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb4242a740 10 client.0 _ll_register_callbacks cb 0x7fbb30009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.089+0000 7fbb3dded640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868d9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3dded640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868d9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59558/0 (socket says 172.21.15.89:59558) 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3dded640 1 -- 172.21.15.89:0/1776702847 learned_addr learned my addr 172.21.15.89:0/1776702847 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3e5ee640 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868d0d40 0x5584868d5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3d5ec640 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868d5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3dded640 1 -- 172.21.15.89:0/1776702847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868d0d40 msgr2=0x5584868d5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3dded640 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868d0d40 0x5584868d5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3dded640 1 -- 172.21.15.89:0/1776702847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 msgr2=0x5584868d5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3dded640 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868d5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3dded640 1 -- 172.21.15.89:0/1776702847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584868d37f0 con 0x5584868c5b60 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3e5ee640 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868d0d40 0x5584868d5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb3dded640 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868d9110 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800a310 tx=0x7fbb28011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1950d259ad2c0af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb217fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb217fa640 1 -- 172.21.15.89:0/1776702847 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb280113f0 con 0x5584868c5b60 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb217fa640 1 -- 172.21.15.89:0/1776702847 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb28011590 con 0x5584868c5b60 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb4242a740 1 -- 172.21.15.89:0/1776702847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb2800cb20 con 0x5584868c5b60 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb217fa640 1 -- 172.21.15.89:0/1776702847 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb28039050 con 0x5584868c5b60 2024-09-15T23:27:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb4242a740 10 client.16764 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.090+0000 7fbb4242a740 1 -- 172.21.15.89:0/1776702847 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55848694c1b0 con 0x5584868c5b60 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb217fa640 1 -- 172.21.15.89:0/1776702847 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2800cb20 con 0x5584868c5b60 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb217fa640 1 -- 172.21.15.89:0/1776702847 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbb2802a070 con 0x5584868c5b60 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb4242a740 10 client.16764 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb4242a740 10 client.16764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb4242a740 20 client.16764 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb4242a740 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55848694c580 0x55848694e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb4242a740 4 client.16764 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb4242a740 1 -- 172.21.15.89:0/1776702847 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558486832d00 con 0x55848694c580 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb3d5ec640 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55848694c580 0x55848694e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb217fa640 10 client.16764 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.091+0000 7fbb3d5ec640 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55848694c580 0x55848694e970 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=520221ffcc052eb8 server_cookie=fc42b1cf0c4827ba in_seq=0 out_seq=0 2024-09-15T23:27:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.092+0000 7fbb217fa640 1 -- 172.21.15.89:0/1776702847 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558486832d00 con 0x55848694c580 2024-09-15T23:27:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.092+0000 7fbb217fa640 10 client.16764 handle_command_reply: tid=0 2024-09-15T23:27:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.092+0000 7fbb4242a740 1 client.16764 shutdown 2024-09-15T23:27:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 1 -- 172.21.15.89:0/1776702847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 msgr2=0x5584868d9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868d9110 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800a310 tx=0x7fbb28011c30 comp rx=0 tx=0).stop 2024-09-15T23:27:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 1 -- 172.21.15.89:0/1776702847 shutdown_connections 2024-09-15T23:27:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55848694c580 0x55848694e970 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5584868d0d40 0x5584868d5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5584868c6480 0x5584868d5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 1 --2- 172.21.15.89:0/1776702847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5584868c5b60 0x5584868d9110 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb217fa640 0 client.16764 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 1 -- 172.21.15.89:0/1776702847 >> 172.21.15.89:0/1776702847 conn(0x5584868c20f0 msgr2=0x5584868cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 1 -- 172.21.15.89:0/1776702847 shutdown_connections 2024-09-15T23:27:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 1 -- 172.21.15.89:0/1776702847 wait complete. 2024-09-15T23:27:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.093+0000 7fbb4242a740 20 client.16764 trim_cache size 0 max 16384 2024-09-15T23:27:40.102 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:27:40.102 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:27:40.103 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "9331ad1c-a5e1-464a-b23a-f730be1120b7", 2024-09-15T23:27:40.103 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:27:40.103 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:27:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 -- 172.21.15.89:0/3573312015 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb240a3240 msgr2=0x7fbb240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb240a3240 0x7fbb240a5700 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbb30005fe0 tx=0x7fbb30005ef0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 -- 172.21.15.89:0/3573312015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38102790 msgr2=0x7fbb38198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38102790 0x7fbb38198a50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fbb3400d770 tx=0x7fbb3400dc30 comp rx=0 tx=0).stop 2024-09-15T23:27:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 -- 172.21.15.89:0/3573312015 shutdown_connections 2024-09-15T23:27:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb240a3240 0x7fbb240a5700 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38106f10 0x7fbb3819cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38102790 0x7fbb38198a50 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 --2- 172.21.15.89:0/3573312015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb38101de0 0x7fbb38198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 -- 172.21.15.89:0/3573312015 >> 172.21.15.89:0/3573312015 conn(0x7fbb380fd8c0 msgr2=0x7fbb380fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 -- 172.21.15.89:0/3573312015 shutdown_connections 2024-09-15T23:27:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:40.094+0000 7fbb40051640 1 -- 172.21.15.89:0/3573312015 wait complete. 2024-09-15T23:27:40.906 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:27:41.157 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:41.444 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:41 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:27:41.444 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:41 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:41.444 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:41 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:27:41.444 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:41 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:41 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:27:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:41 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:41 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:27:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:41 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:41 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:27:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:41 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:41.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:41 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:27:41.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:41 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:41.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.657+0000 7f52cd22b640 1 -- 172.21.15.89:0/3985451301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c8101420 msgr2=0x7f52c810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.657+0000 7f52cd22b640 1 --2- 172.21.15.89:0/3985451301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c8101420 0x7f52c810d950 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f52c0009cd0 tx=0x7f52c002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.657+0000 7f52cd22b640 1 -- 172.21.15.89:0/3985451301 shutdown_connections 2024-09-15T23:27:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.657+0000 7f52cd22b640 1 --2- 172.21.15.89:0/3985451301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c8101420 0x7f52c810d950 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.657+0000 7f52cd22b640 1 --2- 172.21.15.89:0/3985451301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c8100a80 0x7f52c8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.657+0000 7f52cd22b640 1 --2- 172.21.15.89:0/3985451301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52c8106a80 0x7f52c8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.657+0000 7f52cd22b640 1 -- 172.21.15.89:0/3985451301 >> 172.21.15.89:0/3985451301 conn(0x7f52c80fc7a0 msgr2=0x7f52c80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.657+0000 7f52cd22b640 1 -- 172.21.15.89:0/3985451301 shutdown_connections 2024-09-15T23:27:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.657+0000 7f52cd22b640 1 -- 172.21.15.89:0/3985451301 wait complete. 2024-09-15T23:27:41.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.658+0000 7f52cd22b640 1 Processor -- start 2024-09-15T23:27:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.658+0000 7f52cd22b640 1 -- start start 2024-09-15T23:27:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.658+0000 7f52cd22b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52c8100a80 0x7f52c8197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.658+0000 7f52cd22b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c8101420 0x7f52c81977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.658+0000 7f52cd22b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c8106a80 0x7f52c819bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.658+0000 7f52cd22b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f52c8110e80 con 0x7f52c8106a80 2024-09-15T23:27:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.658+0000 7f52cd22b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f52c8110d00 con 0x7f52c8100a80 2024-09-15T23:27:41.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.658+0000 7f52cd22b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52c8111000 con 0x7f52c8101420 2024-09-15T23:27:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.658+0000 7f52c759e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c8106a80 0x7f52c819bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.658+0000 7f52c759e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c8106a80 0x7f52c819bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59572/0 (socket says 172.21.15.89:59572) 2024-09-15T23:27:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c759e640 1 -- 172.21.15.89:0/253195592 learned_addr learned my addr 172.21.15.89:0/253195592 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52c8100a80 0x7f52c8197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52bffff640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c8101420 0x7f52c81977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c759e640 1 -- 172.21.15.89:0/253195592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c8101420 msgr2=0x7f52c81977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:41.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c759e640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c8101420 0x7f52c81977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c759e640 1 -- 172.21.15.89:0/253195592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52c8100a80 msgr2=0x7f52c8197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c759e640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52c8100a80 0x7f52c8197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c759e640 1 -- 172.21.15.89:0/253195592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c819c230 con 0x7f52c8106a80 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52bffff640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c8101420 0x7f52c81977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c759e640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c8106a80 0x7f52c819bb50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f52c0005cc0 tx=0x7f52c0003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c4d99640 1 -- 172.21.15.89:0/253195592 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0004040 con 0x7f52c8106a80 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c4d99640 1 -- 172.21.15.89:0/253195592 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52c00046c0 con 0x7f52c8106a80 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52cd22b640 1 -- 172.21.15.89:0/253195592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c819c4c0 con 0x7f52c8106a80 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52c4d99640 1 -- 172.21.15.89:0/253195592 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c0046480 con 0x7f52c8106a80 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.659+0000 7f52cd22b640 1 -- 172.21.15.89:0/253195592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c81a3e00 con 0x7f52c8106a80 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.661+0000 7f52c4d99640 1 -- 172.21.15.89:0/253195592 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f52c00305a0 con 0x7f52c8106a80 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52cf604740 1 Processor -- start 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52cf604740 1 -- start start 2024-09-15T23:27:41.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52cf604740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb27f9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52cf604740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb2804140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52cf604740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 0x55cbb2806a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52cf604740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cbb275f2e0 con 0x55cbb27f94a0 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52cf604740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cbb2706ce0 con 0x55cbb27f9dc0 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52cf604740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cbb275b2a0 con 0x55cbb2804680 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb27f9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb27f9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59574/0 (socket says 172.21.15.89:59574) 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52bffff640 1 -- 172.21.15.89:0/2873251509 learned_addr learned my addr 172.21.15.89:0/2873251509 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.662+0000 7f52c4d99640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52b40a3240 0x7f52b40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52c759e640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52b40a3240 0x7f52b40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/2873251509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb2804140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52bffff640 1 -- 172.21.15.89:0/2873251509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 msgr2=0x55cbb2806a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:41.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52bffff640 1 --2- 172.21.15.89:0/2873251509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 0x55cbb2806a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52bffff640 1 -- 172.21.15.89:0/2873251509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 msgr2=0x55cbb2804140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52bffff640 1 --2- 172.21.15.89:0/2873251509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb2804140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52bffff640 1 -- 172.21.15.89:0/2873251509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbb26eb440 con 0x55cbb27f94a0 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52c4d99640 1 -- 172.21.15.89:0/253195592 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f52c002e030 con 0x7f52c8106a80 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/2873251509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb2804140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52bffff640 1 --2- 172.21.15.89:0/2873251509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb27f9880 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f52b0005fd0 tx=0x7f52b00106e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488c5e744ce9fa8b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52c759e640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52b40a3240 0x7f52b40a5700 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f52c00ba1d0 tx=0x7f52c0090680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52bd7fa640 1 -- 172.21.15.89:0/2873251509 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0016070 con 0x55cbb27f94a0 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52bd7fa640 1 -- 172.21.15.89:0/2873251509 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b0012900 con 0x55cbb27f94a0 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.663+0000 7f52bd7fa640 1 -- 172.21.15.89:0/2873251509 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0011820 con 0x55cbb27f94a0 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 -- 172.21.15.89:0/2873251509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 msgr2=0x55cbb27f9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 --2- 172.21.15.89:0/2873251509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb27f9880 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f52b0005fd0 tx=0x7f52b00106e0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 -- 172.21.15.89:0/2873251509 shutdown_connections 2024-09-15T23:27:41.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 --2- 172.21.15.89:0/2873251509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 0x55cbb2806a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 --2- 172.21.15.89:0/2873251509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb2804140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 --2- 172.21.15.89:0/2873251509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb27f9880 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 -- 172.21.15.89:0/2873251509 >> 172.21.15.89:0/2873251509 conn(0x55cbb27f5a30 msgr2=0x55cbb27f5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 -- 172.21.15.89:0/2873251509 shutdown_connections 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 -- 172.21.15.89:0/2873251509 wait complete. 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 -- start start 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.664+0000 7f52cf604740 1 Processor -- start 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52cf604740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb2804020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52cf604740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb28012d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52cf604740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 0x55cbb2801870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52cf604740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cbb2807250 con 0x55cbb27f94a0 2024-09-15T23:27:41.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52cf604740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cbb271a050 con 0x55cbb27f9dc0 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52cf604740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cbb28073d0 con 0x55cbb2804680 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52cf604740 10 client.0 _ll_register_callbacks cb 0x55cbb275a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb2804020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb2804020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59582/0 (socket says 172.21.15.89:59582) 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52bffff640 1 -- 172.21.15.89:0/3493603511 learned_addr learned my addr 172.21.15.89:0/3493603511 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52bffff640 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 0x55cbb2801870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb28012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52bffff640 1 -- 172.21.15.89:0/3493603511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 msgr2=0x55cbb2801870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52bffff640 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 0x55cbb2801870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52bffff640 1 -- 172.21.15.89:0/3493603511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 msgr2=0x55cbb28012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52bffff640 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb28012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52bffff640 1 -- 172.21.15.89:0/3493603511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbb24d03c0 con 0x55cbb27f94a0 2024-09-15T23:27:41.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb28012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52bffff640 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb2804020 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009fd0 tx=0x7f52b00095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb64b0d0b198dcd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.665+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3493603511 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0021020 con 0x55cbb27f94a0 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3493603511 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f52b0005250 con 0x55cbb27f94a0 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52cf604740 1 -- 172.21.15.89:0/3493603511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52b0012900 con 0x55cbb27f94a0 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3493603511 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f52b0034050 con 0x55cbb27f94a0 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52cf604740 10 client.16776 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52cf604740 1 -- 172.21.15.89:0/3493603511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cbb2812bd0 con 0x55cbb27f94a0 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3493603511 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b0012900 con 0x55cbb27f94a0 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3493603511 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f52b001d660 con 0x55cbb27f94a0 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52cf604740 10 client.16776 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52cf604740 10 client.16776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52cf604740 20 client.16776 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:41.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52cf604740 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbb2813fe0 0x55cbb28163d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52cf604740 4 client.16776 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52cf604740 1 -- 172.21.15.89:0/3493603511 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbb276c850 con 0x55cbb2813fe0 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.666+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbb2813fe0 0x55cbb28163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.667+0000 7f52ae7fc640 10 client.16776 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.667+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbb2813fe0 0x55cbb28163d0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96b26078d6f1ffb server_cookie=f187047968554181 in_seq=0 out_seq=0 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.667+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3493603511 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cbb276c850 con 0x55cbb2813fe0 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.667+0000 7f52ae7fc640 10 client.16776 handle_command_reply: tid=0 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.667+0000 7f52cf604740 1 client.16776 shutdown 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 1 -- 172.21.15.89:0/3493603511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 msgr2=0x55cbb2804020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb2804020 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f52b0009fd0 tx=0x7f52b00095b0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 1 -- 172.21.15.89:0/3493603511 shutdown_connections 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbb2813fe0 0x55cbb28163d0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 0x55cbb2801870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb28012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52ae7fc640 0 client.16776 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:41.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 1 --2- 172.21.15.89:0/3493603511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb2804020 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 1 -- 172.21.15.89:0/3493603511 >> 172.21.15.89:0/3493603511 conn(0x55cbb27f5a30 msgr2=0x55cbb27fb760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:41.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 1 -- 172.21.15.89:0/3493603511 shutdown_connections 2024-09-15T23:27:41.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 1 -- 172.21.15.89:0/3493603511 wait complete. 2024-09-15T23:27:41.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.668+0000 7f52cf604740 20 client.16776 trim_cache size 0 max 16384 2024-09-15T23:27:41.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52cf604740 1 Processor -- start 2024-09-15T23:27:41.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52cf604740 1 -- start start 2024-09-15T23:27:41.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52cf604740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb27f94a0 0x55cbb28104b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52cf604740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb281ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52cf604740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb2804680 0x55cbb281bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52cf604740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f52c81acae0 con 0x55cbb2804680 2024-09-15T23:27:41.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52cf604740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f52c81ae620 con 0x55cbb27f9dc0 2024-09-15T23:27:41.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52cf604740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52c81ad1b0 con 0x55cbb27f94a0 2024-09-15T23:27:41.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb2804680 0x55cbb281bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb2804680 0x55cbb281bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59588/0 (socket says 172.21.15.89:59588) 2024-09-15T23:27:41.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52bffff640 1 -- 172.21.15.89:0/3137836880 learned_addr learned my addr 172.21.15.89:0/3137836880 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:41.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52bffff640 1 --2- 172.21.15.89:0/3137836880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb27f94a0 0x55cbb28104b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.685+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/3137836880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb281ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.686+0000 7f52bffff640 1 -- 172.21.15.89:0/3137836880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb27f94a0 msgr2=0x55cbb28104b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:41.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.686+0000 7f52bffff640 1 --2- 172.21.15.89:0/3137836880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb27f94a0 0x55cbb28104b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.686+0000 7f52bffff640 1 -- 172.21.15.89:0/3137836880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 msgr2=0x55cbb281ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.686+0000 7f52bffff640 1 --2- 172.21.15.89:0/3137836880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb281ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.686+0000 7f52bffff640 1 -- 172.21.15.89:0/3137836880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbb2803b40 con 0x55cbb2804680 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.686+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/3137836880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb281ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.686+0000 7f52bffff640 1 --2- 172.21.15.89:0/3137836880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb2804680 0x55cbb281bf40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f52b0012700 tx=0x7f52b0014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3dee7e40f1c8031 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.686+0000 7f52bcff9640 1 -- 172.21.15.89:0/3137836880 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b003e020 con 0x55cbb2804680 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.686+0000 7f52bcff9640 1 -- 172.21.15.89:0/3137836880 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b0027040 con 0x55cbb2804680 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.686+0000 7f52bcff9640 1 -- 172.21.15.89:0/3137836880 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0021070 con 0x55cbb2804680 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 -- 172.21.15.89:0/3137836880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb2804680 msgr2=0x55cbb281bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 --2- 172.21.15.89:0/3137836880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb2804680 0x55cbb281bf40 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f52b0012700 tx=0x7f52b0014040 comp rx=0 tx=0).stop 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 -- 172.21.15.89:0/3137836880 shutdown_connections 2024-09-15T23:27:41.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 --2- 172.21.15.89:0/3137836880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb2804680 0x55cbb281bf40 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 --2- 172.21.15.89:0/3137836880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb281ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 --2- 172.21.15.89:0/3137836880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb27f94a0 0x55cbb28104b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 -- 172.21.15.89:0/3137836880 >> 172.21.15.89:0/3137836880 conn(0x55cbb27f5a30 msgr2=0x55cbb27fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 -- 172.21.15.89:0/3137836880 shutdown_connections 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 -- 172.21.15.89:0/3137836880 wait complete. 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 -- start start 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 Processor -- start 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb280cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb2808040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 0x55cbb2808580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f52c81adf50 con 0x55cbb27f94a0 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f52c81ad880 con 0x55cbb27f9dc0 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52c81aaac0 con 0x55cbb2804680 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52cf604740 10 client.0 _ll_register_callbacks cb 0x7f52b0009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb280cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb280cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59604/0 (socket says 172.21.15.89:59604) 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.687+0000 7f52bffff640 1 -- 172.21.15.89:0/3474999325 learned_addr learned my addr 172.21.15.89:0/3474999325 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52bffff640 1 -- 172.21.15.89:0/3474999325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 msgr2=0x55cbb2808580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52bffff640 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 0x55cbb2808580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52bffff640 1 -- 172.21.15.89:0/3474999325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 msgr2=0x55cbb2808040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52bffff640 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb2808040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52bffff640 1 -- 172.21.15.89:0/3474999325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbb2808d60 con 0x55cbb27f94a0 2024-09-15T23:27:41.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52bffff640 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb280cd10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f52b00126d0 tx=0x7f52b0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8680d76acf7c19cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:41.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52ae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:41.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3474999325 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0021050 con 0x55cbb27f94a0 2024-09-15T23:27:41.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3474999325 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f52b00051c0 con 0x55cbb27f94a0 2024-09-15T23:27:41.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52cf604740 1 -- 172.21.15.89:0/3474999325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52b0027040 con 0x55cbb27f94a0 2024-09-15T23:27:41.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3474999325 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f52b0051050 con 0x55cbb27f94a0 2024-09-15T23:27:41.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52cf604740 10 client.16782 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:41.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52cf604740 1 -- 172.21.15.89:0/3474999325 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cbb287e990 con 0x55cbb27f94a0 2024-09-15T23:27:41.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.688+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3474999325 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b0027040 con 0x55cbb27f94a0 2024-09-15T23:27:41.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.689+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3474999325 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f52b0043470 con 0x55cbb27f94a0 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.689+0000 7f52cf604740 10 client.16782 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.689+0000 7f52cf604740 10 client.16782 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.689+0000 7f52cf604740 20 client.16782 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.689+0000 7f52cf604740 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbb287ef90 0x55cbb28813c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.689+0000 7f52cf604740 4 client.16782 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.689+0000 7f52cf604740 1 -- 172.21.15.89:0/3474999325 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cbb2766d60 con 0x55cbb287ef90 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.689+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbb287ef90 0x55cbb28813c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.689+0000 7f52c6d9d640 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbb287ef90 0x55cbb28813c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0367d2ed1d3051f server_cookie=3fc0fc352ed90ca9 in_seq=0 out_seq=0 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.689+0000 7f52ae7fc640 10 client.16782 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52ae7fc640 1 -- 172.21.15.89:0/3474999325 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cbb2766d60 con 0x55cbb287ef90 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52ae7fc640 10 client.16782 handle_command_reply: tid=0 2024-09-15T23:27:41.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 client.16782 shutdown 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 -- 172.21.15.89:0/3474999325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 msgr2=0x55cbb280cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb280cd10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f52b00126d0 tx=0x7f52b0000bc0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 -- 172.21.15.89:0/3474999325 shutdown_connections 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cbb287ef90 0x55cbb28813c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbb2804680 0x55cbb2808580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbb27f9dc0 0x55cbb2808040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 --2- 172.21.15.89:0/3474999325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cbb27f94a0 0x55cbb280cd10 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 -- 172.21.15.89:0/3474999325 >> 172.21.15.89:0/3474999325 conn(0x55cbb27f5a30 msgr2=0x55cbb27f4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52ae7fc640 0 client.16782 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 -- 172.21.15.89:0/3474999325 shutdown_connections 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 1 -- 172.21.15.89:0/3474999325 wait complete. 2024-09-15T23:27:41.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.690+0000 7f52cf604740 20 client.16782 trim_cache size 0 max 16384 2024-09-15T23:27:41.704 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:27:41.704 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:27:41.704 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:27:41.704 INFO:teuthology.orchestra.run.smithi089.stdout: "9331ad1c-a5e1-464a-b23a-f730be1120b7": { 2024-09-15T23:27:41.704 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:27:41.704 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "9331ad1c-a5e1-464a-b23a-f730be1120b7", 2024-09-15T23:27:41.704 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:27:41.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:41.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:27:41.704 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:27:41.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 -- 172.21.15.89:0/253195592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52b40a3240 msgr2=0x7f52b40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:41.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52b40a3240 0x7f52b40a5700 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f52c00ba1d0 tx=0x7f52c0090680 comp rx=0 tx=0).stop 2024-09-15T23:27:41.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 -- 172.21.15.89:0/253195592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c8106a80 msgr2=0x7f52c819bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:41.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c8106a80 0x7f52c819bb50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f52c0005cc0 tx=0x7f52c0003970 comp rx=0 tx=0).stop 2024-09-15T23:27:41.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 -- 172.21.15.89:0/253195592 shutdown_connections 2024-09-15T23:27:41.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f52b40a3240 0x7f52b40a5700 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f52c8106a80 0x7f52c819bb50 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c8101420 0x7f52c81977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 --2- 172.21.15.89:0/253195592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52c8100a80 0x7f52c8197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:41.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 -- 172.21.15.89:0/253195592 >> 172.21.15.89:0/253195592 conn(0x7f52c80fc7a0 msgr2=0x7f52c80fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:41.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 -- 172.21.15.89:0/253195592 shutdown_connections 2024-09-15T23:27:41.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:27:41.691+0000 7f52cd22b640 1 -- 172.21.15.89:0/253195592 wait complete. 2024-09-15T23:27:42.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:42 smithi120 ceph-mon[35250]: pgmap v460: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:27:42.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:42 smithi174 ceph-mon[35586]: pgmap v460: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:27:42.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:42 smithi089 ceph-mon[30933]: pgmap v460: 129 pgs: 129 active+clean; 1.6 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:27:42.629 INFO:tasks.cephfs.filesystem:scrub status for tag:9331ad1c-a5e1-464a-b23a-f730be1120b7 - {'path': '/', 'tag': '9331ad1c-a5e1-464a-b23a-f730be1120b7', 'options': 'recursive,force'} 2024-09-15T23:27:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:43 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:27:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:43 smithi120 ceph-mon[35250]: pgmap v461: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:27:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:43 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:27:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:43 smithi174 ceph-mon[35586]: pgmap v461: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:27:43.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:43 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:27:43.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:43 smithi089 ceph-mon[30933]: pgmap v461: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:27:45.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.101+0000 7f97acb8d640 1 -- 172.21.15.89:0/4161259390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a81025d0 msgr2=0x7f97a81049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:45.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.101+0000 7f97acb8d640 1 --2- 172.21.15.89:0/4161259390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a81025d0 0x7f97a81049c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9798009900 tx=0x7f979802a410 comp rx=0 tx=0).stop 2024-09-15T23:27:45.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.101+0000 7f97acb8d640 1 -- 172.21.15.89:0/4161259390 shutdown_connections 2024-09-15T23:27:45.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.101+0000 7f97acb8d640 1 --2- 172.21.15.89:0/4161259390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97a8107830 0x7f97a8109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.101+0000 7f97acb8d640 1 --2- 172.21.15.89:0/4161259390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a8104f00 0x7f97a81072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.101+0000 7f97acb8d640 1 --2- 172.21.15.89:0/4161259390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a81025d0 0x7f97a81049c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.101+0000 7f97acb8d640 1 -- 172.21.15.89:0/4161259390 >> 172.21.15.89:0/4161259390 conn(0x7f97a80fc1b0 msgr2=0x7f97a80fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:45.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.101+0000 7f97acb8d640 1 -- 172.21.15.89:0/4161259390 shutdown_connections 2024-09-15T23:27:45.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.101+0000 7f97acb8d640 1 -- 172.21.15.89:0/4161259390 wait complete. 2024-09-15T23:27:45.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.102+0000 7f97acb8d640 1 Processor -- start 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.102+0000 7f97acb8d640 1 -- start start 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.102+0000 7f97acb8d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97a81025d0 0x7f97a8197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.102+0000 7f97acb8d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a8104f00 0x7f97a81983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.102+0000 7f97acb8d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a8107830 0x7f97a819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.102+0000 7f97acb8d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97a810d880 con 0x7f97a8107830 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.102+0000 7f97acb8d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f97a810d700 con 0x7f97a81025d0 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.102+0000 7f97acb8d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f97a810da00 con 0x7f97a8104f00 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97a6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a8107830 0x7f97a819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97a6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a8107830 0x7f97a819f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59616/0 (socket says 172.21.15.89:59616) 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97a6d9d640 1 -- 172.21.15.89:0/109078352 learned_addr learned my addr 172.21.15.89:0/109078352 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f979dd9b640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a8104f00 0x7f97a81983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97a659c640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97a81025d0 0x7f97a8197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97a6d9d640 1 -- 172.21.15.89:0/109078352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a8104f00 msgr2=0x7f97a81983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97a6d9d640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a8104f00 0x7f97a81983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97a6d9d640 1 -- 172.21.15.89:0/109078352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97a81025d0 msgr2=0x7f97a8197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97a6d9d640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97a81025d0 0x7f97a8197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97a6d9d640 1 -- 172.21.15.89:0/109078352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a819fa80 con 0x7f97a8107830 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f979dd9b640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a8104f00 0x7f97a81983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97a6d9d640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a8107830 0x7f97a819f410 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f97a000e9d0 tx=0x7f97a000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f979ffff640 1 -- 172.21.15.89:0/109078352 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a000cd80 con 0x7f97a8107830 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f979ffff640 1 -- 172.21.15.89:0/109078352 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97a0005020 con 0x7f97a8107830 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97acb8d640 1 -- 172.21.15.89:0/109078352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a819fd70 con 0x7f97a8107830 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f979ffff640 1 -- 172.21.15.89:0/109078352 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a00116d0 con 0x7f97a8107830 2024-09-15T23:27:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.103+0000 7f97acb8d640 1 -- 172.21.15.89:0/109078352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a81a0290 con 0x7f97a8107830 2024-09-15T23:27:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.105+0000 7f979ffff640 1 -- 172.21.15.89:0/109078352 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f97a00051d0 con 0x7f97a8107830 2024-09-15T23:27:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.106+0000 7f97afa01740 1 Processor -- start 2024-09-15T23:27:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.106+0000 7f97afa01740 1 -- start start 2024-09-15T23:27:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.106+0000 7f97afa01740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923c3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.106+0000 7f979ffff640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97940a32f0 0x7f97940a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.106+0000 7f97afa01740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 0x555d923c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.106+0000 7f97afa01740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c9370 0x555d923cb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.106+0000 7f97afa01740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555d923272f0 con 0x555d923c39a0 2024-09-15T23:27:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.106+0000 7f97afa01740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555d922cf270 con 0x555d923c9370 2024-09-15T23:27:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.106+0000 7f97afa01740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555d923232b0 con 0x555d923c42e0 2024-09-15T23:27:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923c3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923c3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59622/0 (socket says 172.21.15.89:59622) 2024-09-15T23:27:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f97a659c640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97940a32f0 0x7f97940a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979dd9b640 1 -- 172.21.15.89:0/2289692716 learned_addr learned my addr 172.21.15.89:0/2289692716 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979ffff640 1 -- 172.21.15.89:0/109078352 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f97a00d0400 con 0x7f97a8107830 2024-09-15T23:27:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979dd9b640 1 --2- 172.21.15.89:0/2289692716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 0x555d923c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f97a6d9d640 1 --2- 172.21.15.89:0/2289692716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c9370 0x555d923cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979dd9b640 1 -- 172.21.15.89:0/2289692716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 msgr2=0x555d923c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979dd9b640 1 --2- 172.21.15.89:0/2289692716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 0x555d923c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979dd9b640 1 -- 172.21.15.89:0/2289692716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c9370 msgr2=0x555d923cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979dd9b640 1 --2- 172.21.15.89:0/2289692716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c9370 0x555d923cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979dd9b640 1 -- 172.21.15.89:0/2289692716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d922ce340 con 0x555d923c39a0 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979dd9b640 1 --2- 172.21.15.89:0/2289692716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923c3da0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f9790007ba0 tx=0x7f97900073e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceaf6a920240e8b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f97a659c640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97940a32f0 0x7f97940a57b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9798031610 tx=0x7f9798000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979cd99640 1 -- 172.21.15.89:0/2289692716 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979001a070 con 0x555d923c39a0 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979cd99640 1 -- 172.21.15.89:0/2289692716 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9790010040 con 0x555d923c39a0 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.107+0000 7f979cd99640 1 -- 172.21.15.89:0/2289692716 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97900145f0 con 0x555d923c39a0 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 -- 172.21.15.89:0/2289692716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 msgr2=0x555d923c3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 --2- 172.21.15.89:0/2289692716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923c3da0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f9790007ba0 tx=0x7f97900073e0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.115 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 -- 172.21.15.89:0/2289692716 shutdown_connections 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 --2- 172.21.15.89:0/2289692716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c9370 0x555d923cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 --2- 172.21.15.89:0/2289692716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 0x555d923c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 --2- 172.21.15.89:0/2289692716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923c3da0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 -- 172.21.15.89:0/2289692716 >> 172.21.15.89:0/2289692716 conn(0x555d923bf010 msgr2=0x555d923bf410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 -- 172.21.15.89:0/2289692716 shutdown_connections 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 -- 172.21.15.89:0/2289692716 wait complete. 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 -- start start 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.108+0000 7f97afa01740 1 Processor -- start 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97afa01740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 0x555d923cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97afa01740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 0x555d923ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97afa01740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c9370 0x555d923d5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97afa01740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555d923cc340 con 0x555d923c9370 2024-09-15T23:27:45.116 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97afa01740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555d922b0290 con 0x555d923c39a0 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97afa01740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555d923cc4c0 con 0x555d923c42e0 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97afa01740 10 client.0 _ll_register_callbacks cb 0x555d922e1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c9370 0x555d923d5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c9370 0x555d923d5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59630/0 (socket says 172.21.15.89:59630) 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a6d9d640 1 -- 172.21.15.89:0/143282005 learned_addr learned my addr 172.21.15.89:0/143282005 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f979dd9b640 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 0x555d923ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a659c640 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 0x555d923cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a6d9d640 1 -- 172.21.15.89:0/143282005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 msgr2=0x555d923ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a6d9d640 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 0x555d923ce1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a6d9d640 1 -- 172.21.15.89:0/143282005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 msgr2=0x555d923cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:45.117 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a6d9d640 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 0x555d923cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a6d9d640 1 -- 172.21.15.89:0/143282005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d9203a210 con 0x555d923c9370 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a659c640 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 0x555d923cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f979dd9b640 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 0x555d923ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97a6d9d640 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c9370 0x555d923d5210 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f97a00a4d10 tx=0x7f97a009d7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd9555b4380dd2b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f978dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f978dffb640 1 -- 172.21.15.89:0/143282005 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a001f040 con 0x555d923c9370 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f978dffb640 1 -- 172.21.15.89:0/143282005 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f97a00d00b0 con 0x555d923c9370 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97afa01740 1 -- 172.21.15.89:0/143282005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9790010040 con 0x555d923c9370 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f978dffb640 1 -- 172.21.15.89:0/143282005 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f97a009cf60 con 0x555d923c9370 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97afa01740 10 client.16794 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.109+0000 7f97afa01740 1 -- 172.21.15.89:0/143282005 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555d92106620 con 0x555d923c9370 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f978dffb640 1 -- 172.21.15.89:0/143282005 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9790010040 con 0x555d923c9370 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f978dffb640 1 -- 172.21.15.89:0/143282005 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f97a000f040 con 0x555d923c9370 2024-09-15T23:27:45.118 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f97afa01740 10 client.16794 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f97afa01740 10 client.16794 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f97afa01740 20 client.16794 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f97afa01740 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555d923d9960 0x555d923dbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f97afa01740 4 client.16794 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f97afa01740 1 -- 172.21.15.89:0/143282005 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d92334a20 con 0x555d923d9960 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f97a659c640 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555d923d9960 0x555d923dbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f978dffb640 10 client.16794 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.110+0000 7f97a659c640 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555d923d9960 0x555d923dbd90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bd3a32817acaa78 server_cookie=1f2c215a97d396ef in_seq=0 out_seq=0 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f978dffb640 1 -- 172.21.15.89:0/143282005 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555d92334a20 con 0x555d923d9960 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f978dffb640 10 client.16794 handle_command_reply: tid=0 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 client.16794 shutdown 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 -- 172.21.15.89:0/143282005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c9370 msgr2=0x555d923d5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c9370 0x555d923d5210 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f97a00a4d10 tx=0x7f97a009d7c0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 -- 172.21.15.89:0/143282005 shutdown_connections 2024-09-15T23:27:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555d923d9960 0x555d923dbd90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c9370 0x555d923d5210 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f978dffb640 0 client.16794 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c42e0 0x555d923ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 --2- 172.21.15.89:0/143282005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 0x555d923cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 -- 172.21.15.89:0/143282005 >> 172.21.15.89:0/143282005 conn(0x555d923bf010 msgr2=0x555d923c0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 -- 172.21.15.89:0/143282005 shutdown_connections 2024-09-15T23:27:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.111+0000 7f97afa01740 1 -- 172.21.15.89:0/143282005 wait complete. 2024-09-15T23:27:45.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.112+0000 7f97afa01740 20 client.16794 trim_cache size 0 max 16384 2024-09-15T23:27:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97afa01740 1 Processor -- start 2024-09-15T23:27:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97afa01740 1 -- start start 2024-09-15T23:27:45.131 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97afa01740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923be0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97afa01740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c42e0 0x555d923d51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97afa01740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c9370 0x555d923c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97afa01740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97a81aa8c0 con 0x555d923c39a0 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97afa01740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f97a81ac400 con 0x555d923c42e0 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97afa01740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f97a81aaf90 con 0x555d923c9370 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97a659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923be0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97a659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923be0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59642/0 (socket says 172.21.15.89:59642) 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97a659c640 1 -- 172.21.15.89:0/3763094576 learned_addr learned my addr 172.21.15.89:0/3763094576 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97a6d9d640 1 --2- 172.21.15.89:0/3763094576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c9370 0x555d923c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f979dd9b640 1 --2- 172.21.15.89:0/3763094576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c42e0 0x555d923d51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97a659c640 1 -- 172.21.15.89:0/3763094576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c9370 msgr2=0x555d923c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97a659c640 1 --2- 172.21.15.89:0/3763094576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c9370 0x555d923c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97a659c640 1 -- 172.21.15.89:0/3763094576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c42e0 msgr2=0x555d923d51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97a659c640 1 --2- 172.21.15.89:0/3763094576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c42e0 0x555d923d51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97a659c640 1 -- 172.21.15.89:0/3763094576 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d923c8820 con 0x555d923c39a0 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.126+0000 7f97a659c640 1 --2- 172.21.15.89:0/3763094576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923be0b0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9798009900 tx=0x7f979803b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586904d84bd7915e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f978ffff640 1 -- 172.21.15.89:0/3763094576 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798035050 con 0x555d923c39a0 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f978ffff640 1 -- 172.21.15.89:0/3763094576 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9798005520 con 0x555d923c39a0 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f978ffff640 1 -- 172.21.15.89:0/3763094576 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798004e30 con 0x555d923c39a0 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f97afa01740 1 -- 172.21.15.89:0/3763094576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 msgr2=0x555d923be0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f97afa01740 1 --2- 172.21.15.89:0/3763094576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923be0b0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f9798009900 tx=0x7f979803b000 comp rx=0 tx=0).stop 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f97afa01740 1 -- 172.21.15.89:0/3763094576 shutdown_connections 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f97afa01740 1 --2- 172.21.15.89:0/3763094576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c9370 0x555d923c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f97afa01740 1 --2- 172.21.15.89:0/3763094576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c42e0 0x555d923d51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f97afa01740 1 --2- 172.21.15.89:0/3763094576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c39a0 0x555d923be0b0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f97afa01740 1 -- 172.21.15.89:0/3763094576 >> 172.21.15.89:0/3763094576 conn(0x555d923bf010 msgr2=0x555d923bd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f97afa01740 1 -- 172.21.15.89:0/3763094576 shutdown_connections 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f97afa01740 1 -- 172.21.15.89:0/3763094576 wait complete. 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.127+0000 7f97afa01740 1 -- start start 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97afa01740 1 Processor -- start 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97afa01740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 0x555d923d2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97afa01740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c42e0 0x555d923d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97afa01740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c9370 0x555d923d42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97afa01740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97a81abd30 con 0x555d923c42e0 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97afa01740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f97a81ab660 con 0x555d923c39a0 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97afa01740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f97a81a88a0 con 0x555d923c9370 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97afa01740 10 client.0 _ll_register_callbacks cb 0x7f97a0005840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f979dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c42e0 0x555d923d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f979dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c42e0 0x555d923d32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59644/0 (socket says 172.21.15.89:59644) 2024-09-15T23:27:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f979dd9b640 1 -- 172.21.15.89:0/2705023439 learned_addr learned my addr 172.21.15.89:0/2705023439 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97a659c640 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 0x555d923d2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97a6d9d640 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c9370 0x555d923d42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f979dd9b640 1 -- 172.21.15.89:0/2705023439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c9370 msgr2=0x555d923d42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f979dd9b640 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c9370 0x555d923d42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f979dd9b640 1 -- 172.21.15.89:0/2705023439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 msgr2=0x555d923d2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f979dd9b640 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 0x555d923d2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f979dd9b640 1 -- 172.21.15.89:0/2705023439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d923d48e0 con 0x555d923c42e0 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97a659c640 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 0x555d923d2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f979dd9b640 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c42e0 0x555d923d32c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9790007ba0 tx=0x7f9790007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2dc2628fb5455d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f978dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f978dffb640 1 -- 172.21.15.89:0/2705023439 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9790025070 con 0x555d923c42e0 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f978dffb640 1 -- 172.21.15.89:0/2705023439 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f97900059c0 con 0x555d923c42e0 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.128+0000 7f97afa01740 1 -- 172.21.15.89:0/2705023439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9798005520 con 0x555d923c42e0 2024-09-15T23:27:45.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f978dffb640 1 -- 172.21.15.89:0/2705023439 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9790031050 con 0x555d923c42e0 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f97afa01740 10 client.16800 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f97afa01740 1 -- 172.21.15.89:0/2705023439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555d92444b50 con 0x555d923c42e0 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f978dffb640 1 -- 172.21.15.89:0/2705023439 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9798005520 con 0x555d923c42e0 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f978dffb640 1 -- 172.21.15.89:0/2705023439 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9790021460 con 0x555d923c42e0 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f97afa01740 10 client.16800 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f97afa01740 10 client.16800 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f97afa01740 20 client.16800 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f97afa01740 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555d923d9960 0x555d92447360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f97afa01740 4 client.16800 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f97afa01740 1 -- 172.21.15.89:0/2705023439 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d9232ed70 con 0x555d923d9960 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.129+0000 7f97a659c640 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555d923d9960 0x555d92447360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.130+0000 7f978dffb640 10 client.16800 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.130+0000 7f97a659c640 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555d923d9960 0x555d92447360 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b470c615de02860 server_cookie=616066427a330134 in_seq=0 out_seq=0 2024-09-15T23:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f978dffb640 1 -- 172.21.15.89:0/2705023439 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d9232ed70 con 0x555d923d9960 2024-09-15T23:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f978dffb640 10 client.16800 handle_command_reply: tid=0 2024-09-15T23:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 client.16800 shutdown 2024-09-15T23:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 -- 172.21.15.89:0/2705023439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c42e0 msgr2=0x555d923d32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c42e0 0x555d923d32c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9790007ba0 tx=0x7f9790007800 comp rx=0 tx=0).stop 2024-09-15T23:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 -- 172.21.15.89:0/2705023439 shutdown_connections 2024-09-15T23:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555d923d9960 0x555d92447360 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555d923c9370 0x555d923d42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555d923c42e0 0x555d923d32c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 --2- 172.21.15.89:0/2705023439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d923c39a0 0x555d923d2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f978dffb640 0 client.16800 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 -- 172.21.15.89:0/2705023439 >> 172.21.15.89:0/2705023439 conn(0x555d923bf010 msgr2=0x555d923be1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 -- 172.21.15.89:0/2705023439 shutdown_connections 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 1 -- 172.21.15.89:0/2705023439 wait complete. 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.143+0000 7f97afa01740 20 client.16800 trim_cache size 0 max 16384 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:27:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:27:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 -- 172.21.15.89:0/109078352 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97940a32f0 msgr2=0x7f97940a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97940a32f0 0x7f97940a57b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9798031610 tx=0x7f9798000bc0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 -- 172.21.15.89:0/109078352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a8107830 msgr2=0x7f97a819f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a8107830 0x7f97a819f410 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f97a000e9d0 tx=0x7f97a000ee90 comp rx=0 tx=0).stop 2024-09-15T23:27:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 -- 172.21.15.89:0/109078352 shutdown_connections 2024-09-15T23:27:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97940a32f0 0x7f97940a57b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97a8107830 0x7f97a819f410 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97a8104f00 0x7f97a81983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 --2- 172.21.15.89:0/109078352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97a81025d0 0x7f97a8197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 -- 172.21.15.89:0/109078352 >> 172.21.15.89:0/109078352 conn(0x7f97a80fc1b0 msgr2=0x7f97a8102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 -- 172.21.15.89:0/109078352 shutdown_connections 2024-09-15T23:27:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:45.144+0000 7f97acb8d640 1 -- 172.21.15.89:0/109078352 wait complete. 2024-09-15T23:27:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:46 smithi120 ceph-mon[35250]: pgmap v462: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:27:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:46 smithi174 ceph-mon[35586]: pgmap v462: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:27:46.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:46 smithi089 ceph-mon[30933]: pgmap v462: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:27:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:47 smithi120 ceph-mon[35250]: pgmap v463: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 22 op/s 2024-09-15T23:27:47.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:47 smithi174 ceph-mon[35586]: pgmap v463: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 22 op/s 2024-09-15T23:27:47.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:47 smithi089 ceph-mon[30933]: pgmap v463: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 22 op/s 2024-09-15T23:27:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:49 smithi120 ceph-mon[35250]: pgmap v464: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:27:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:49 smithi174 ceph-mon[35586]: pgmap v464: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:27:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:49 smithi089 ceph-mon[30933]: pgmap v464: 129 pgs: 129 active+clean; 1.7 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:27:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:51 smithi120 ceph-mon[35250]: pgmap v465: 129 pgs: 129 active+clean; 1.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:27:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:51 smithi174 ceph-mon[35586]: pgmap v465: 129 pgs: 129 active+clean; 1.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:27:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:51 smithi089 ceph-mon[30933]: pgmap v465: 129 pgs: 129 active+clean; 1.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:27:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:53 smithi120 ceph-mon[35250]: pgmap v466: 129 pgs: 129 active+clean; 1.8 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:27:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:53 smithi174 ceph-mon[35586]: pgmap v466: 129 pgs: 129 active+clean; 1.8 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:27:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:53 smithi089 ceph-mon[30933]: pgmap v466: 129 pgs: 129 active+clean; 1.8 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:27:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:55 smithi120 ceph-mon[35250]: pgmap v467: 129 pgs: 129 active+clean; 1.8 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:27:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:55 smithi174 ceph-mon[35586]: pgmap v467: 129 pgs: 129 active+clean; 1.8 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:27:55.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:55 smithi089 ceph-mon[30933]: pgmap v467: 129 pgs: 129 active+clean; 1.8 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:27:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:57 smithi120 ceph-mon[35250]: pgmap v468: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:27:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:57 smithi174 ceph-mon[35586]: pgmap v468: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:27:57.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:57 smithi089 ceph-mon[30933]: pgmap v468: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:27:58.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.237+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1063747405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33dc101c30 msgr2=0x7f33dc1020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.237+0000 7f33e2bf8640 1 --2- 172.21.15.89:0/1063747405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33dc101c30 0x7f33dc1020b0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009900 tx=0x7f33c802a410 comp rx=0 tx=0).stop 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.237+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1063747405 shutdown_connections 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.237+0000 7f33e2bf8640 1 --2- 172.21.15.89:0/1063747405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33dc1025f0 0x7f33dc108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.237+0000 7f33e2bf8640 1 --2- 172.21.15.89:0/1063747405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33dc101c30 0x7f33dc1020b0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.237+0000 7f33e2bf8640 1 --2- 172.21.15.89:0/1063747405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33dc100a40 0x7f33dc100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.237+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1063747405 >> 172.21.15.89:0/1063747405 conn(0x7f33dc0fc1e0 msgr2=0x7f33dc0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.237+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1063747405 shutdown_connections 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.237+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1063747405 wait complete. 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e2bf8640 1 Processor -- start 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e2bf8640 1 -- start start 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e2bf8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33dc100a40 0x7f33dc10b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e2bf8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33dc101c30 0x7f33dc109a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e2bf8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33dc1025f0 0x7f33dc109f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e2bf8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc10d810 con 0x7f33dc1025f0 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e2bf8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc10d690 con 0x7f33dc100a40 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e2bf8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc10d990 con 0x7f33dc101c30 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e1195640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33dc1025f0 0x7f33dc109f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e1195640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33dc1025f0 0x7f33dc109f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46698/0 (socket says 172.21.15.89:46698) 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e1195640 1 -- 172.21.15.89:0/1956634390 learned_addr learned my addr 172.21.15.89:0/1956634390 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e0994640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33dc100a40 0x7f33dc10b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33dbfff640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33dc101c30 0x7f33dc109a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e1195640 1 -- 172.21.15.89:0/1956634390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33dc101c30 msgr2=0x7f33dc109a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e1195640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33dc101c30 0x7f33dc109a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e1195640 1 -- 172.21.15.89:0/1956634390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33dc100a40 msgr2=0x7f33dc10b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.242 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e1195640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33dc100a40 0x7f33dc10b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e1195640 1 -- 172.21.15.89:0/1956634390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc10a7d0 con 0x7f33dc1025f0 2024-09-15T23:27:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.238+0000 7f33e0994640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33dc100a40 0x7f33dc10b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.239+0000 7f33e1195640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33dc1025f0 0x7f33dc109f40 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f33d400eb10 tx=0x7f33d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.239+0000 7f33d9ffb640 1 -- 172.21.15.89:0/1956634390 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d4004b20 con 0x7f33dc1025f0 2024-09-15T23:27:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.239+0000 7f33d9ffb640 1 -- 172.21.15.89:0/1956634390 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33d4004cc0 con 0x7f33dc1025f0 2024-09-15T23:27:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.239+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1956634390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33dc1a1ee0 con 0x7f33dc1025f0 2024-09-15T23:27:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.239+0000 7f33d9ffb640 1 -- 172.21.15.89:0/1956634390 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d4010760 con 0x7f33dc1025f0 2024-09-15T23:27:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.239+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1956634390 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33dc1a22f0 con 0x7f33dc1025f0 2024-09-15T23:27:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.241+0000 7f33d9ffb640 1 -- 172.21.15.89:0/1956634390 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f33d4005020 con 0x7f33dc1025f0 2024-09-15T23:27:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.242+0000 7f33d9ffb640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33cc0a32f0 0x7f33cc0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.242+0000 7f33e0994640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33cc0a32f0 0x7f33cc0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.242+0000 7f33d9ffb640 1 -- 172.21.15.89:0/1956634390 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f33d40d0290 con 0x7f33dc1025f0 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.242+0000 7f33e5a66740 1 Processor -- start 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.242+0000 7f33e0994640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33cc0a32f0 0x7f33cc0a57b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f33dc101a90 tx=0x7f33d0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.242+0000 7f33e5a66740 1 -- start start 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde74da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde752e0 0x55d8bde79da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde7a370 0x55d8bde7c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e5a66740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d8bddd82f0 con 0x55d8bde7a370 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e5a66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d8bdd80270 con 0x55d8bde752e0 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e5a66740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d8bddd42b0 con 0x55d8bde749a0 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e0994640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde74da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e1195640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde7a370 0x55d8bde7c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e0994640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde74da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:46896/0 (socket says 172.21.15.89:46896) 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e1195640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde7a370 0x55d8bde7c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46710/0 (socket says 172.21.15.89:46710) 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e0994640 1 -- 172.21.15.89:0/2182347445 learned_addr learned my addr 172.21.15.89:0/2182347445 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33dbfff640 1 --2- 172.21.15.89:0/2182347445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde752e0 0x55d8bde79da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e1195640 1 -- 172.21.15.89:0/2182347445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 msgr2=0x55d8bde74da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e1195640 1 --2- 172.21.15.89:0/2182347445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde74da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e1195640 1 -- 172.21.15.89:0/2182347445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde752e0 msgr2=0x55d8bde79da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e1195640 1 --2- 172.21.15.89:0/2182347445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde752e0 0x55d8bde79da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e1195640 1 -- 172.21.15.89:0/2182347445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8bdd7f340 con 0x55d8bde7a370 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e0994640 1 --2- 172.21.15.89:0/2182347445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde74da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33e1195640 1 --2- 172.21.15.89:0/2182347445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde7a370 0x55d8bde7c830 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f33d409c1c0 tx=0x7f33d409c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fad51273c307 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33c6ffd640 1 -- 172.21.15.89:0/2182347445 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d40cde60 con 0x55d8bde7a370 2024-09-15T23:27:58.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33c6ffd640 1 -- 172.21.15.89:0/2182347445 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33d409cbc0 con 0x55d8bde7a370 2024-09-15T23:27:58.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.243+0000 7f33c6ffd640 1 -- 172.21.15.89:0/2182347445 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d40a4ca0 con 0x55d8bde7a370 2024-09-15T23:27:58.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.244+0000 7f33e5a66740 1 -- 172.21.15.89:0/2182347445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde7a370 msgr2=0x55d8bde7c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.244+0000 7f33e5a66740 1 --2- 172.21.15.89:0/2182347445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde7a370 0x55d8bde7c830 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f33d409c1c0 tx=0x7f33d409c350 comp rx=0 tx=0).stop 2024-09-15T23:27:58.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.244+0000 7f33e5a66740 1 -- 172.21.15.89:0/2182347445 shutdown_connections 2024-09-15T23:27:58.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.244+0000 7f33e5a66740 1 --2- 172.21.15.89:0/2182347445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde7a370 0x55d8bde7c830 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.244+0000 7f33e5a66740 1 --2- 172.21.15.89:0/2182347445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde752e0 0x55d8bde79da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.244+0000 7f33e5a66740 1 --2- 172.21.15.89:0/2182347445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde74da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.244+0000 7f33e5a66740 1 -- 172.21.15.89:0/2182347445 >> 172.21.15.89:0/2182347445 conn(0x55d8bde70010 msgr2=0x55d8bde70410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.244+0000 7f33e5a66740 1 -- 172.21.15.89:0/2182347445 shutdown_connections 2024-09-15T23:27:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.244+0000 7f33e5a66740 1 -- 172.21.15.89:0/2182347445 wait complete. 2024-09-15T23:27:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.244+0000 7f33e5a66740 1 -- start start 2024-09-15T23:27:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e5a66740 1 Processor -- start 2024-09-15T23:27:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde749a0 0x55d8bde7eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde752e0 0x55d8bde7f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 0x55d8bde860f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e5a66740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d8bde7d340 con 0x55d8bde749a0 2024-09-15T23:27:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e5a66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d8bdd61290 con 0x55d8bde7a370 2024-09-15T23:27:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e5a66740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d8bde7d4c0 con 0x55d8bde752e0 2024-09-15T23:27:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e5a66740 10 client.0 _ll_register_callbacks cb 0x55d8bdd92140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e0994640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde749a0 0x55d8bde7eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e0994640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde749a0 0x55d8bde7eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46726/0 (socket says 172.21.15.89:46726) 2024-09-15T23:27:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e0994640 1 -- 172.21.15.89:0/3837769372 learned_addr learned my addr 172.21.15.89:0/3837769372 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33dbfff640 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde752e0 0x55d8bde7f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e1195640 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 0x55d8bde860f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e0994640 1 -- 172.21.15.89:0/3837769372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde752e0 msgr2=0x55d8bde7f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e0994640 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde752e0 0x55d8bde7f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e0994640 1 -- 172.21.15.89:0/3837769372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 msgr2=0x55d8bde860f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e0994640 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 0x55d8bde860f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.245+0000 7f33e0994640 1 -- 172.21.15.89:0/3837769372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8bdaeb210 con 0x55d8bde749a0 2024-09-15T23:27:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33e0994640 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde749a0 0x55d8bde7eb70 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f33d000fc00 tx=0x7f33d0013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b165e3106ac13aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33bbfff640 1 -- 172.21.15.89:0/3837769372 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d002c060 con 0x55d8bde749a0 2024-09-15T23:27:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33bbfff640 1 -- 172.21.15.89:0/3837769372 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f33d0027430 con 0x55d8bde749a0 2024-09-15T23:27:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33e5a66740 1 -- 172.21.15.89:0/3837769372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33d409cbc0 con 0x55d8bde749a0 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33bbfff640 1 -- 172.21.15.89:0/3837769372 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f33d002e630 con 0x55d8bde749a0 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33e5a66740 10 client.16812 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33e5a66740 1 -- 172.21.15.89:0/3837769372 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d8bdbb7620 con 0x55d8bde749a0 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33bbfff640 1 -- 172.21.15.89:0/3837769372 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33d409cbc0 con 0x55d8bde749a0 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33bbfff640 1 -- 172.21.15.89:0/3837769372 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f33d0026070 con 0x55d8bde749a0 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33e5a66740 10 client.16812 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33e5a66740 10 client.16812 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33e5a66740 20 client.16812 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d8bde8a840 0x55d8bde8cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33e5a66740 4 client.16812 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.246+0000 7f33e5a66740 1 -- 172.21.15.89:0/3837769372 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8bdde5a20 con 0x55d8bde8a840 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.247+0000 7f33dbfff640 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d8bde8a840 0x55d8bde8cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.247+0000 7f33bbfff640 10 client.16812 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.247+0000 7f33dbfff640 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d8bde8a840 0x55d8bde8cc90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d487781941a12e5d server_cookie=63f83e126d86d5fd in_seq=0 out_seq=0 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33bbfff640 1 -- 172.21.15.89:0/3837769372 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d8bdde5a20 con 0x55d8bde8a840 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33bbfff640 10 client.16812 handle_command_reply: tid=0 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 client.16812 shutdown 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 -- 172.21.15.89:0/3837769372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde749a0 msgr2=0x55d8bde7eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde749a0 0x55d8bde7eb70 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f33d000fc00 tx=0x7f33d0013040 comp rx=0 tx=0).stop 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 -- 172.21.15.89:0/3837769372 shutdown_connections 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d8bde8a840 0x55d8bde8cc90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 0x55d8bde860f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde752e0 0x55d8bde7f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33bbfff640 0 client.16812 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3837769372 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde749a0 0x55d8bde7eb70 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 -- 172.21.15.89:0/3837769372 >> 172.21.15.89:0/3837769372 conn(0x55d8bde70010 msgr2=0x55d8bde71820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 -- 172.21.15.89:0/3837769372 shutdown_connections 2024-09-15T23:27:58.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 1 -- 172.21.15.89:0/3837769372 wait complete. 2024-09-15T23:27:58.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.248+0000 7f33e5a66740 20 client.16812 trim_cache size 0 max 16384 2024-09-15T23:27:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.262+0000 7f33e5a66740 1 Processor -- start 2024-09-15T23:27:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.262+0000 7f33e5a66740 1 -- start start 2024-09-15T23:27:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.262+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde749a0 0x55d8bde7f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.262+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde860b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.265 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.262+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde7a370 0x55d8bde70f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.262+0000 7f33e5a66740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1aca70 con 0x55d8bde752e0 2024-09-15T23:27:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.262+0000 7f33e5a66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1ae5b0 con 0x55d8bde749a0 2024-09-15T23:27:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.262+0000 7f33e5a66740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1ad140 con 0x55d8bde7a370 2024-09-15T23:27:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde860b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde860b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46732/0 (socket says 172.21.15.89:46732) 2024-09-15T23:27:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33dbfff640 1 -- 172.21.15.89:0/3282711774 learned_addr learned my addr 172.21.15.89:0/3282711774 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33e1195640 1 --2- 172.21.15.89:0/3282711774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde7a370 0x55d8bde70f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33e0994640 1 --2- 172.21.15.89:0/3282711774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde749a0 0x55d8bde7f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33dbfff640 1 -- 172.21.15.89:0/3282711774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde7a370 msgr2=0x55d8bde70f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33dbfff640 1 --2- 172.21.15.89:0/3282711774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde7a370 0x55d8bde70f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33dbfff640 1 -- 172.21.15.89:0/3282711774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde749a0 msgr2=0x55d8bde7f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33dbfff640 1 --2- 172.21.15.89:0/3282711774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde749a0 0x55d8bde7f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33dbfff640 1 -- 172.21.15.89:0/3282711774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8bde7e6b0 con 0x55d8bde752e0 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33dbfff640 1 --2- 172.21.15.89:0/3282711774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde860b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f33c802a930 tx=0x7f33c8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5fd4b5b38d5ed0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33c67fc640 1 -- 172.21.15.89:0/3282711774 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c804b070 con 0x55d8bde752e0 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33c67fc640 1 -- 172.21.15.89:0/3282711774 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33c8005520 con 0x55d8bde752e0 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.263+0000 7f33c67fc640 1 -- 172.21.15.89:0/3282711774 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8035040 con 0x55d8bde752e0 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 -- 172.21.15.89:0/3282711774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 msgr2=0x55d8bde860b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3282711774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde860b0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f33c802a930 tx=0x7f33c8005fd0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 -- 172.21.15.89:0/3282711774 shutdown_connections 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3282711774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde7a370 0x55d8bde70f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3282711774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde860b0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3282711774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde749a0 0x55d8bde7f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 -- 172.21.15.89:0/3282711774 >> 172.21.15.89:0/3282711774 conn(0x55d8bde70010 msgr2=0x55d8bde7e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 -- 172.21.15.89:0/3282711774 shutdown_connections 2024-09-15T23:27:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 -- 172.21.15.89:0/3282711774 wait complete. 2024-09-15T23:27:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 -- start start 2024-09-15T23:27:58.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.264+0000 7f33e5a66740 1 Processor -- start 2024-09-15T23:27:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde83550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde83a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e5a66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 0x55d8bde84aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e5a66740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1adee0 con 0x55d8bde752e0 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e5a66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1ad810 con 0x55d8bde7a370 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e5a66740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc1aaa50 con 0x55d8bde749a0 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e5a66740 10 client.0 _ll_register_callbacks cb 0x7f33d000f560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde83a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e0994640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde83550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde83a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46746/0 (socket says 172.21.15.89:46746) 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33dbfff640 1 -- 172.21.15.89:0/3434761018 learned_addr learned my addr 172.21.15.89:0/3434761018 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e1195640 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 0x55d8bde84aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33dbfff640 1 -- 172.21.15.89:0/3434761018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 msgr2=0x55d8bde83550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.271 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33dbfff640 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde83550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33dbfff640 1 -- 172.21.15.89:0/3434761018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 msgr2=0x55d8bde84aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33dbfff640 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 0x55d8bde84aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33dbfff640 1 -- 172.21.15.89:0/3434761018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8bde850b0 con 0x55d8bde752e0 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e0994640 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde83550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33e1195640 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 0x55d8bde84aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33dbfff640 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde83a90 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009850 tx=0x7f33c802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b4c2975bd3d76e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.265+0000 7f33bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33bbfff640 1 -- 172.21.15.89:0/3434761018 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c8053020 con 0x55d8bde752e0 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33bbfff640 1 -- 172.21.15.89:0/3434761018 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f33c8004300 con 0x55d8bde752e0 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33e5a66740 1 -- 172.21.15.89:0/3434761018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33c8005520 con 0x55d8bde752e0 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33bbfff640 1 -- 172.21.15.89:0/3434761018 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f33c805e050 con 0x55d8bde752e0 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33e5a66740 10 client.16818 fetch_fsmap learned FSMap version 29 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33e5a66740 1 -- 172.21.15.89:0/3434761018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d8bde863c0 con 0x55d8bde752e0 2024-09-15T23:27:58.272 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33bbfff640 1 -- 172.21.15.89:0/3434761018 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33c8005520 con 0x55d8bde752e0 2024-09-15T23:27:58.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33bbfff640 1 -- 172.21.15.89:0/3434761018 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f33c804e070 con 0x55d8bde752e0 2024-09-15T23:27:58.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33e5a66740 10 client.16818 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:27:58.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33e5a66740 10 client.16818 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:58.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33e5a66740 20 client.16818 populate_metadata read hostname 'smithi089' 2024-09-15T23:27:58.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d8bdef5c40 0x55d8bdef8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33e5a66740 4 client.16818 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:27:58.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33e5a66740 1 -- 172.21.15.89:0/3434761018 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d8bdddfd70 con 0x55d8bdef5c40 2024-09-15T23:27:58.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.266+0000 7f33e0994640 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d8bdef5c40 0x55d8bdef8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.267+0000 7f33bbfff640 10 client.16818 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:58.273 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.267+0000 7f33e0994640 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d8bdef5c40 0x55d8bdef8060 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14d8a942e53e8d67 server_cookie=635387d4f4a8f0b3 in_seq=0 out_seq=0 2024-09-15T23:27:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33bbfff640 1 -- 172.21.15.89:0/3434761018 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d8bdddfd70 con 0x55d8bdef5c40 2024-09-15T23:27:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33bbfff640 10 client.16818 handle_command_reply: tid=0 2024-09-15T23:27:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33e5a66740 1 client.16818 shutdown 2024-09-15T23:27:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33e5a66740 1 -- 172.21.15.89:0/3434761018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 msgr2=0x55d8bde83a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde83a90 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f33c8009850 tx=0x7f33c802a410 comp rx=0 tx=0).stop 2024-09-15T23:27:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33e5a66740 1 -- 172.21.15.89:0/3434761018 shutdown_connections 2024-09-15T23:27:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d8bdef5c40 0x55d8bdef8060 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d8bde7a370 0x55d8bde84aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33bbfff640 0 client.16818 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:27:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8bde752e0 0x55d8bde83a90 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33e5a66740 1 --2- 172.21.15.89:0/3434761018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d8bde749a0 0x55d8bde83550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.278+0000 7f33e5a66740 1 -- 172.21.15.89:0/3434761018 >> 172.21.15.89:0/3434761018 conn(0x55d8bde70010 msgr2=0x55d8bde6e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e5a66740 1 -- 172.21.15.89:0/3434761018 shutdown_connections 2024-09-15T23:27:58.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e5a66740 1 -- 172.21.15.89:0/3434761018 wait complete. 2024-09-15T23:27:58.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e5a66740 20 client.16818 trim_cache size 0 max 16384 2024-09-15T23:27:58.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:27:58.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:27:58.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:27:58.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:27:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1956634390 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33cc0a32f0 msgr2=0x7f33cc0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33cc0a32f0 0x7f33cc0a57b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f33dc101a90 tx=0x7f33d0005f90 comp rx=0 tx=0).stop 2024-09-15T23:27:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1956634390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33dc1025f0 msgr2=0x7f33dc109f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33dc1025f0 0x7f33dc109f40 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f33d400eb10 tx=0x7f33d400c2d0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1956634390 shutdown_connections 2024-09-15T23:27:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33cc0a32f0 0x7f33cc0a57b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33dc1025f0 0x7f33dc109f40 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33dc101c30 0x7f33dc109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 --2- 172.21.15.89:0/1956634390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f33dc100a40 0x7f33dc10b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1956634390 >> 172.21.15.89:0/1956634390 conn(0x7f33dc0fc1e0 msgr2=0x7f33dc104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1956634390 shutdown_connections 2024-09-15T23:27:58.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:27:58.279+0000 7f33e2bf8640 1 -- 172.21.15.89:0/1956634390 wait complete. 2024-09-15T23:27:58.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:27:58.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:27:58.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:27:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:27:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:27:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:27:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:27:59 smithi120 ceph-mon[35250]: pgmap v469: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:27:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:27:59.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:27:59.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:27:59.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:27:59 smithi174 ceph-mon[35586]: pgmap v469: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:27:59.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:27:59.672 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:27:59.672 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:27:59.672 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:27:59 smithi089 ceph-mon[30933]: pgmap v469: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:28:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:01 smithi120 ceph-mon[35250]: pgmap v470: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:28:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:01 smithi174 ceph-mon[35586]: pgmap v470: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:28:01.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:01 smithi089 ceph-mon[30933]: pgmap v470: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:28:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:03 smithi120 ceph-mon[35250]: pgmap v471: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:28:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:03 smithi174 ceph-mon[35586]: pgmap v471: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:28:03.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:03 smithi089 ceph-mon[30933]: pgmap v471: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:28:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:05 smithi120 ceph-mon[35250]: pgmap v472: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:28:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:05 smithi174 ceph-mon[35586]: pgmap v472: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:28:05.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:05 smithi089 ceph-mon[30933]: pgmap v472: 129 pgs: 129 active+clean; 1.9 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:28:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:07 smithi120 ceph-mon[35250]: pgmap v473: 129 pgs: 129 active+clean; 2.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:28:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:07 smithi174 ceph-mon[35586]: pgmap v473: 129 pgs: 129 active+clean; 2.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:28:07.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:07 smithi089 ceph-mon[30933]: pgmap v473: 129 pgs: 129 active+clean; 2.0 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:28:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:09 smithi120 ceph-mon[35250]: pgmap v474: 129 pgs: 129 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:28:09.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:09 smithi174 ceph-mon[35586]: pgmap v474: 129 pgs: 129 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:28:09.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:09 smithi089 ceph-mon[30933]: pgmap v474: 129 pgs: 129 active+clean; 2.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:28:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.376+0000 7fe750176640 1 -- 172.21.15.89:0/1094073940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe748107860 msgr2=0x7fe748109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.376+0000 7fe750176640 1 --2- 172.21.15.89:0/1094073940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe748107860 0x7fe748109c90 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe740009ca0 tx=0x7fe74002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.376+0000 7fe750176640 1 -- 172.21.15.89:0/1094073940 shutdown_connections 2024-09-15T23:28:11.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.376+0000 7fe750176640 1 --2- 172.21.15.89:0/1094073940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe748107860 0x7fe748109c90 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.376+0000 7fe750176640 1 --2- 172.21.15.89:0/1094073940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe748104f30 0x7fe748107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.376+0000 7fe750176640 1 --2- 172.21.15.89:0/1094073940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe748102600 0x7fe7481049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.376+0000 7fe750176640 1 -- 172.21.15.89:0/1094073940 >> 172.21.15.89:0/1094073940 conn(0x7fe7480fc1e0 msgr2=0x7fe7480fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:11.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.376+0000 7fe750176640 1 -- 172.21.15.89:0/1094073940 shutdown_connections 2024-09-15T23:28:11.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.376+0000 7fe750176640 1 -- 172.21.15.89:0/1094073940 wait complete. 2024-09-15T23:28:11.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe750176640 1 Processor -- start 2024-09-15T23:28:11.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe750176640 1 -- start start 2024-09-15T23:28:11.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe750176640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe748102600 0x7fe748197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe750176640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe748104f30 0x7fe748198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe750176640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe748107860 0x7fe74819f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe750176640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe74810d7e0 con 0x7fe748104f30 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe750176640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe74810d660 con 0x7fe748107860 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe750176640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe74810d960 con 0x7fe748102600 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe74d711640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe748104f30 0x7fe748198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe74d711640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe748104f30 0x7fe748198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45156/0 (socket says 172.21.15.89:45156) 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe74d711640 1 -- 172.21.15.89:0/3446302422 learned_addr learned my addr 172.21.15.89:0/3446302422 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe74df12640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe748102600 0x7fe748197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.377+0000 7fe74e713640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe748107860 0x7fe74819f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe74d711640 1 -- 172.21.15.89:0/3446302422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe748102600 msgr2=0x7fe748197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe74d711640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe748102600 0x7fe748197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe74d711640 1 -- 172.21.15.89:0/3446302422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe748107860 msgr2=0x7fe74819f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe74d711640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe748107860 0x7fe74819f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe74d711640 1 -- 172.21.15.89:0/3446302422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74819fa00 con 0x7fe748104f30 2024-09-15T23:28:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe74df12640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe748102600 0x7fe748197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe74e713640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe748107860 0x7fe74819f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe74d711640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe748104f30 0x7fe748198320 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe73800d770 tx=0x7fe73800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe73effd640 1 -- 172.21.15.89:0/3446302422 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738017070 con 0x7fe748104f30 2024-09-15T23:28:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe73effd640 1 -- 172.21.15.89:0/3446302422 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe738009c30 con 0x7fe748104f30 2024-09-15T23:28:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe750176640 1 -- 172.21.15.89:0/3446302422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74819fcf0 con 0x7fe748104f30 2024-09-15T23:28:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe73effd640 1 -- 172.21.15.89:0/3446302422 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73800f620 con 0x7fe748104f30 2024-09-15T23:28:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.378+0000 7fe750176640 1 -- 172.21.15.89:0/3446302422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7481a01e0 con 0x7fe748104f30 2024-09-15T23:28:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.380+0000 7fe73effd640 1 -- 172.21.15.89:0/3446302422 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe73801a070 con 0x7fe748104f30 2024-09-15T23:28:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe752fed740 1 Processor -- start 2024-09-15T23:28:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe752fed740 1 -- start start 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe752fed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e441b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe752fed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e446f0 0x55e731e491b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe752fed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e4bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe752fed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e731da32f0 con 0x55e731e49780 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe752fed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e731d4b270 con 0x55e731e446f0 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe752fed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e731d9f2b0 con 0x55e731e43db0 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe74d711640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e4bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe74d711640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e4bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45170/0 (socket says 172.21.15.89:45170) 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.381+0000 7fe74d711640 1 -- 172.21.15.89:0/4130771163 learned_addr learned my addr 172.21.15.89:0/4130771163 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe74e713640 1 --2- 172.21.15.89:0/4130771163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe73effd640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7340a32f0 0x7fe7340a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe74df12640 1 --2- 172.21.15.89:0/4130771163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e446f0 0x55e731e491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe74d711640 1 -- 172.21.15.89:0/4130771163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 msgr2=0x55e731e441b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe74d711640 1 --2- 172.21.15.89:0/4130771163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e441b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe74d711640 1 -- 172.21.15.89:0/4130771163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e446f0 msgr2=0x55e731e491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe74d711640 1 --2- 172.21.15.89:0/4130771163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e446f0 0x55e731e491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe74d711640 1 -- 172.21.15.89:0/4130771163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e731d4a340 con 0x55e731e49780 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe74e713640 1 --2- 172.21.15.89:0/4130771163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e441b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe74e713640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7340a32f0 0x7fe7340a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe73effd640 1 -- 172.21.15.89:0/3446302422 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe7380cf920 con 0x7fe748104f30 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe74d711640 1 --2- 172.21.15.89:0/4130771163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e4bc40 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe73808b520 tx=0x7fe738017eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e172adabda08af76 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe733fff640 1 -- 172.21.15.89:0/4130771163 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73809b630 con 0x55e731e49780 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe733fff640 1 -- 172.21.15.89:0/4130771163 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe738086080 con 0x55e731e49780 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.382+0000 7fe733fff640 1 -- 172.21.15.89:0/4130771163 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7380664f0 con 0x55e731e49780 2024-09-15T23:28:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.383+0000 7fe752fed740 1 -- 172.21.15.89:0/4130771163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 msgr2=0x55e731e4bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.383+0000 7fe752fed740 1 --2- 172.21.15.89:0/4130771163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e4bc40 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe73808b520 tx=0x7fe738017eb0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.383+0000 7fe74e713640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7340a32f0 0x7fe7340a57b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe740031ae0 tx=0x7fe7400095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.383+0000 7fe752fed740 1 -- 172.21.15.89:0/4130771163 shutdown_connections 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.383+0000 7fe752fed740 1 --2- 172.21.15.89:0/4130771163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e4bc40 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.383+0000 7fe752fed740 1 --2- 172.21.15.89:0/4130771163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e446f0 0x55e731e491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.383+0000 7fe752fed740 1 --2- 172.21.15.89:0/4130771163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e441b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.383+0000 7fe752fed740 1 -- 172.21.15.89:0/4130771163 >> 172.21.15.89:0/4130771163 conn(0x55e731e3b010 msgr2=0x55e731e3b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.383+0000 7fe752fed740 1 -- 172.21.15.89:0/4130771163 shutdown_connections 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.383+0000 7fe752fed740 1 -- 172.21.15.89:0/4130771163 wait complete. 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe752fed740 1 -- start start 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe752fed740 1 Processor -- start 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe752fed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e4dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe752fed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e4e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe752fed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e49780 0x55e731e554c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe752fed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e731e4c6f0 con 0x55e731e446f0 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe752fed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e731d2c290 con 0x55e731e49780 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe752fed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e731e4c870 con 0x55e731e43db0 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe752fed740 10 client.0 _ll_register_callbacks cb 0x55e731d5d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe74d711640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e4e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe74d711640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e4e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45178/0 (socket says 172.21.15.89:45178) 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe74d711640 1 -- 172.21.15.89:0/426248198 learned_addr learned my addr 172.21.15.89:0/426248198 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe74e713640 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e49780 0x55e731e554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe74df12640 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e4dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe74d711640 1 -- 172.21.15.89:0/426248198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 msgr2=0x55e731e4dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe74d711640 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e4dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe74d711640 1 -- 172.21.15.89:0/426248198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e49780 msgr2=0x55e731e554c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe74d711640 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e49780 0x55e731e554c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.384+0000 7fe74d711640 1 -- 172.21.15.89:0/426248198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e731ab6210 con 0x55e731e446f0 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe74d711640 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e4e4f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe738000c00 tx=0x7fe738009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90394f7b0477731 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe7317fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe7317fa640 1 -- 172.21.15.89:0/426248198 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73809b450 con 0x55e731e446f0 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe7317fa640 1 -- 172.21.15.89:0/426248198 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe738099df0 con 0x55e731e446f0 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe752fed740 1 -- 172.21.15.89:0/426248198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e731db0a20 con 0x55e731e446f0 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe7317fa640 1 -- 172.21.15.89:0/426248198 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe73809fea0 con 0x55e731e446f0 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe752fed740 10 client.16830 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe752fed740 1 -- 172.21.15.89:0/426248198 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e731b82620 con 0x55e731e446f0 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe7317fa640 1 -- 172.21.15.89:0/426248198 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e731db0a20 con 0x55e731e446f0 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe7317fa640 1 -- 172.21.15.89:0/426248198 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe7380cd050 con 0x55e731e446f0 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe752fed740 10 client.16830 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe752fed740 10 client.16830 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe752fed740 20 client.16830 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe752fed740 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e731e59780 0x55e731e5bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe752fed740 4 client.16830 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.385+0000 7fe752fed740 1 -- 172.21.15.89:0/426248198 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e731daad70 con 0x55e731e59780 2024-09-15T23:28:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.386+0000 7fe74df12640 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e731e59780 0x55e731e5bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.386+0000 7fe74df12640 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e731e59780 0x55e731e5bb70 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=473c6bc7029f4c31 server_cookie=72c9d19e74a0087 in_seq=0 out_seq=0 2024-09-15T23:28:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.386+0000 7fe7317fa640 10 client.16830 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe7317fa640 1 -- 172.21.15.89:0/426248198 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e731daad70 con 0x55e731e59780 2024-09-15T23:28:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe7317fa640 10 client.16830 handle_command_reply: tid=0 2024-09-15T23:28:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 client.16830 shutdown 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 -- 172.21.15.89:0/426248198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 msgr2=0x55e731e4e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e4e4f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe738000c00 tx=0x7fe738009e60 comp rx=0 tx=0).stop 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 -- 172.21.15.89:0/426248198 shutdown_connections 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e731e59780 0x55e731e5bb70 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e49780 0x55e731e554c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e4e4f0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe7317fa640 0 client.16830 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 --2- 172.21.15.89:0/426248198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e4dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 -- 172.21.15.89:0/426248198 >> 172.21.15.89:0/426248198 conn(0x55e731e3b010 msgr2=0x55e731e3c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 -- 172.21.15.89:0/426248198 shutdown_connections 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.387+0000 7fe752fed740 1 -- 172.21.15.89:0/426248198 wait complete. 2024-09-15T23:28:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.388+0000 7fe752fed740 20 client.16830 trim_cache size 0 max 16384 2024-09-15T23:28:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe752fed740 1 Processor -- start 2024-09-15T23:28:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe752fed740 1 -- start start 2024-09-15T23:28:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe752fed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e43db0 0x55e731e3bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe752fed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e446f0 0x55e731e55480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe752fed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e3c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe752fed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481a8640 con 0x55e731e49780 2024-09-15T23:28:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe752fed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481aa180 con 0x55e731e43db0 2024-09-15T23:28:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe752fed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481a8d10 con 0x55e731e446f0 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe74e713640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e3c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe74e713640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e3c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45184/0 (socket says 172.21.15.89:45184) 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe74e713640 1 -- 172.21.15.89:0/3164599931 learned_addr learned my addr 172.21.15.89:0/3164599931 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe74d711640 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e446f0 0x55e731e55480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.402+0000 7fe74df12640 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e43db0 0x55e731e3bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe74e713640 1 -- 172.21.15.89:0/3164599931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e446f0 msgr2=0x55e731e55480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe74e713640 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e446f0 0x55e731e55480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe74e713640 1 -- 172.21.15.89:0/3164599931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e43db0 msgr2=0x55e731e3bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe74e713640 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e43db0 0x55e731e3bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe74e713640 1 -- 172.21.15.89:0/3164599931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e731e5bee0 con 0x55e731e49780 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe74df12640 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e43db0 0x55e731e3bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe74d711640 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e446f0 0x55e731e55480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe74e713640 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e3c400 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fe740031580 tx=0x7fe74003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f6d023da0a73 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe7337fe640 1 -- 172.21.15.89:0/3164599931 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7400317f0 con 0x55e731e49780 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe7337fe640 1 -- 172.21.15.89:0/3164599931 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe740036050 con 0x55e731e49780 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe7337fe640 1 -- 172.21.15.89:0/3164599931 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7400309b0 con 0x55e731e49780 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe752fed740 1 -- 172.21.15.89:0/3164599931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 msgr2=0x55e731e3c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.403+0000 7fe752fed740 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e3c400 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fe740031580 tx=0x7fe74003f000 comp rx=0 tx=0).stop 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 -- 172.21.15.89:0/3164599931 shutdown_connections 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e49780 0x55e731e3c400 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e446f0 0x55e731e55480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 --2- 172.21.15.89:0/3164599931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e43db0 0x55e731e3bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 -- 172.21.15.89:0/3164599931 >> 172.21.15.89:0/3164599931 conn(0x55e731e3b010 msgr2=0x55e731e45890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 -- 172.21.15.89:0/3164599931 shutdown_connections 2024-09-15T23:28:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 -- 172.21.15.89:0/3164599931 wait complete. 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 -- start start 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 Processor -- start 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e52a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e52f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e49780 0x55e731e53f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481a9ab0 con 0x55e731e446f0 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481a93e0 con 0x55e731e49780 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe7481a6620 con 0x55e731e43db0 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe752fed740 10 client.0 _ll_register_callbacks cb 0x7fe738097c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.404+0000 7fe74d711640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e52f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe74d711640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e52f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45186/0 (socket says 172.21.15.89:45186) 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe74d711640 1 -- 172.21.15.89:0/365021492 learned_addr learned my addr 172.21.15.89:0/365021492 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe74d711640 1 -- 172.21.15.89:0/365021492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 msgr2=0x55e731e52a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe74d711640 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e52a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe74d711640 1 -- 172.21.15.89:0/365021492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e49780 msgr2=0x55e731e53f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe74d711640 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e49780 0x55e731e53f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe74d711640 1 -- 172.21.15.89:0/365021492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e731e54530 con 0x55e731e446f0 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe74d711640 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e52f60 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fe7380aff20 tx=0x7fe73809b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749f1aada0a63e85 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe7317fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe7317fa640 1 -- 172.21.15.89:0/365021492 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73809b8c0 con 0x55e731e446f0 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe7317fa640 1 -- 172.21.15.89:0/365021492 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe7380668d0 con 0x55e731e446f0 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe752fed740 1 -- 172.21.15.89:0/365021492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe740036050 con 0x55e731e446f0 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe7317fa640 1 -- 172.21.15.89:0/365021492 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe73809b630 con 0x55e731e446f0 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe752fed740 10 client.16836 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe752fed740 1 -- 172.21.15.89:0/365021492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e731e557b0 con 0x55e731e446f0 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe7317fa640 1 -- 172.21.15.89:0/365021492 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe740036050 con 0x55e731e446f0 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.405+0000 7fe7317fa640 1 -- 172.21.15.89:0/365021492 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe73807a1c0 con 0x55e731e446f0 2024-09-15T23:28:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.406+0000 7fe752fed740 10 client.16836 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.406+0000 7fe752fed740 10 client.16836 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.406+0000 7fe752fed740 20 client.16836 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.406+0000 7fe752fed740 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e731ec49b0 0x55e731ec6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.406+0000 7fe752fed740 4 client.16836 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:28:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.406+0000 7fe752fed740 1 -- 172.21.15.89:0/365021492 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e731d27900 con 0x55e731ec49b0 2024-09-15T23:28:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.406+0000 7fe74df12640 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e731ec49b0 0x55e731ec6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.406+0000 7fe7317fa640 10 client.16836 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.406+0000 7fe74df12640 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e731ec49b0 0x55e731ec6dc0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89282f92b5f81014 server_cookie=91bc3abe727ad799 in_seq=0 out_seq=0 2024-09-15T23:28:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.420+0000 7fe7317fa640 1 -- 172.21.15.89:0/365021492 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e731d27900 con 0x55e731ec49b0 2024-09-15T23:28:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.420+0000 7fe7317fa640 10 client.16836 handle_command_reply: tid=0 2024-09-15T23:28:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.420+0000 7fe752fed740 1 client.16836 shutdown 2024-09-15T23:28:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 1 -- 172.21.15.89:0/365021492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 msgr2=0x55e731e52f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e52f60 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fe7380aff20 tx=0x7fe73809b450 comp rx=0 tx=0).stop 2024-09-15T23:28:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 1 -- 172.21.15.89:0/365021492 shutdown_connections 2024-09-15T23:28:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e731ec49b0 0x55e731ec6dc0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e731e49780 0x55e731e53f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e731e446f0 0x55e731e52f60 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe7317fa640 0 client.16836 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:11.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 1 --2- 172.21.15.89:0/365021492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e731e43db0 0x55e731e52a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 1 -- 172.21.15.89:0/365021492 >> 172.21.15.89:0/365021492 conn(0x55e731e3b010 msgr2=0x55e731e3bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:11.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 1 -- 172.21.15.89:0/365021492 shutdown_connections 2024-09-15T23:28:11.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 1 -- 172.21.15.89:0/365021492 wait complete. 2024-09-15T23:28:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe752fed740 20 client.16836 trim_cache size 0 max 16384 2024-09-15T23:28:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:28:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:28:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:28:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:28:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe750176640 1 -- 172.21.15.89:0/3446302422 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7340a32f0 msgr2=0x7fe7340a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe750176640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7340a32f0 0x7fe7340a57b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe740031ae0 tx=0x7fe7400095c0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe750176640 1 -- 172.21.15.89:0/3446302422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe748104f30 msgr2=0x7fe748198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe750176640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe748104f30 0x7fe748198320 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe73800d770 tx=0x7fe73800dc30 comp rx=0 tx=0).stop 2024-09-15T23:28:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe750176640 1 -- 172.21.15.89:0/3446302422 shutdown_connections 2024-09-15T23:28:11.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe750176640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe7340a32f0 0x7fe7340a57b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe750176640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe748107860 0x7fe74819f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe750176640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe748104f30 0x7fe748198320 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe750176640 1 --2- 172.21.15.89:0/3446302422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe748102600 0x7fe748197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:11.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.421+0000 7fe750176640 1 -- 172.21.15.89:0/3446302422 >> 172.21.15.89:0/3446302422 conn(0x7fe7480fc1e0 msgr2=0x7fe748102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:11.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.422+0000 7fe750176640 1 -- 172.21.15.89:0/3446302422 shutdown_connections 2024-09-15T23:28:11.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:11.422+0000 7fe750176640 1 -- 172.21.15.89:0/3446302422 wait complete. 2024-09-15T23:28:11.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:11 smithi120 ceph-mon[35250]: pgmap v475: 129 pgs: 129 active+clean; 2.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:28:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:11 smithi174 ceph-mon[35586]: pgmap v475: 129 pgs: 129 active+clean; 2.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:28:11.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:11 smithi089 ceph-mon[30933]: pgmap v475: 129 pgs: 129 active+clean; 2.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:28:12.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:28:12.890 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:13.285 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:13 smithi089 ceph-mon[30933]: pgmap v476: 129 pgs: 129 active+clean; 2.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:28:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.392+0000 7fef2222a640 1 -- 172.21.15.89:0/882526392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef1c105de0 msgr2=0x7fef1c10d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.392+0000 7fef2222a640 1 --2- 172.21.15.89:0/882526392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef1c105de0 0x7fef1c10d960 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fef14009ca0 tx=0x7fef1402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.392+0000 7fef2222a640 1 -- 172.21.15.89:0/882526392 shutdown_connections 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.392+0000 7fef2222a640 1 --2- 172.21.15.89:0/882526392 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef1c105de0 0x7fef1c10d960 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.392+0000 7fef2222a640 1 --2- 172.21.15.89:0/882526392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef1c101e20 0x7fef1c1056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.392+0000 7fef2222a640 1 --2- 172.21.15.89:0/882526392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef1c101470 0x7fef1c101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.392+0000 7fef2222a640 1 -- 172.21.15.89:0/882526392 >> 172.21.15.89:0/882526392 conn(0x7fef1c077230 msgr2=0x7fef1c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.392+0000 7fef2222a640 1 -- 172.21.15.89:0/882526392 shutdown_connections 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.392+0000 7fef2222a640 1 -- 172.21.15.89:0/882526392 wait complete. 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef2222a640 1 Processor -- start 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef2222a640 1 -- start start 2024-09-15T23:28:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef2222a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef1c101470 0x7fef1c19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef2222a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef1c101e20 0x7fef1c19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef2222a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef1c105de0 0x7fef1c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef2222a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c110e40 con 0x7fef1c101470 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef2222a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c110cc0 con 0x7fef1c105de0 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef2222a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c110fc0 con 0x7fef1c101e20 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef1b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef1c101470 0x7fef1c19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef1b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef1c101470 0x7fef1c19a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45212/0 (socket says 172.21.15.89:45212) 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.393+0000 7fef1b7fe640 1 -- 172.21.15.89:0/1038400567 learned_addr learned my addr 172.21.15.89:0/1038400567 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef13fff640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef1c101e20 0x7fef1c19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef1b7fe640 1 -- 172.21.15.89:0/1038400567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef1c101e20 msgr2=0x7fef1c19ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef1c101e20 0x7fef1c19ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef1b7fe640 1 -- 172.21.15.89:0/1038400567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef1c105de0 msgr2=0x7fef1c19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef1bfff640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef1c105de0 0x7fef1c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef1c105de0 0x7fef1c19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef1b7fe640 1 -- 172.21.15.89:0/1038400567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1c19f5a0 con 0x7fef1c101470 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef13fff640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef1c101e20 0x7fef1c19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef1bfff640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef1c105de0 0x7fef1c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef1c101470 0x7fef1c19a5f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00d950 tx=0x7fef0c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef197fa640 1 -- 172.21.15.89:0/1038400567 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c017070 con 0x7fef1c101470 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef2222a640 1 -- 172.21.15.89:0/1038400567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1c19f890 con 0x7fef1c101470 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef197fa640 1 -- 172.21.15.89:0/1038400567 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef0c009c30 con 0x7fef1c101470 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef197fa640 1 -- 172.21.15.89:0/1038400567 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c00f6b0 con 0x7fef1c101470 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.394+0000 7fef2222a640 1 -- 172.21.15.89:0/1038400567 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef1c1a7170 con 0x7fef1c101470 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.396+0000 7fef197fa640 1 -- 172.21.15.89:0/1038400567 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fef0c0109d0 con 0x7fef1c101470 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef24603740 1 Processor -- start 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef24603740 1 -- start start 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef24603740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce24a0 0x565090ce2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef24603740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce2dc0 0x565090ced140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef24603740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ced680 0x565090cefa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef24603740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x565090c482e0 con 0x565090ced680 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef24603740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565090befce0 con 0x565090ce24a0 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef24603740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565090c442a0 con 0x565090ce2dc0 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef13fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ced680 0x565090cefa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef13fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ced680 0x565090cefa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45228/0 (socket says 172.21.15.89:45228) 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef13fff640 1 -- 172.21.15.89:0/1551177908 learned_addr learned my addr 172.21.15.89:0/1551177908 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef1bfff640 1 --2- 172.21.15.89:0/1551177908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce24a0 0x565090ce2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/1551177908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce2dc0 0x565090ced140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef197fa640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 0x7fef080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef13fff640 1 -- 172.21.15.89:0/1551177908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce2dc0 msgr2=0x565090ced140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef13fff640 1 --2- 172.21.15.89:0/1551177908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce2dc0 0x565090ced140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef13fff640 1 -- 172.21.15.89:0/1551177908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce24a0 msgr2=0x565090ce2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef13fff640 1 --2- 172.21.15.89:0/1551177908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce24a0 0x565090ce2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.397+0000 7fef13fff640 1 -- 172.21.15.89:0/1551177908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565090bd4440 con 0x565090ced680 2024-09-15T23:28:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.398+0000 7fef1bfff640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 0x7fef080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.398+0000 7fef197fa640 1 -- 172.21.15.89:0/1038400567 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fef0c0cfa40 con 0x7fef1c101470 2024-09-15T23:28:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.398+0000 7fef13fff640 1 --2- 172.21.15.89:0/1551177908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ced680 0x565090cefa70 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fef04009ba0 tx=0x7fef0400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56a31ece841ced9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.398+0000 7fef11ffb640 1 -- 172.21.15.89:0/1551177908 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04016070 con 0x565090ced680 2024-09-15T23:28:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.398+0000 7fef11ffb640 1 -- 172.21.15.89:0/1551177908 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef04011950 con 0x565090ced680 2024-09-15T23:28:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.398+0000 7fef11ffb640 1 -- 172.21.15.89:0/1551177908 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04010890 con 0x565090ced680 2024-09-15T23:28:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 -- 172.21.15.89:0/1551177908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ced680 msgr2=0x565090cefa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef1bfff640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 0x7fef080a5700 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fef14005e30 tx=0x7fef140095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 --2- 172.21.15.89:0/1551177908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ced680 0x565090cefa70 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fef04009ba0 tx=0x7fef0400e6c0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 -- 172.21.15.89:0/1551177908 shutdown_connections 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 --2- 172.21.15.89:0/1551177908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ced680 0x565090cefa70 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 --2- 172.21.15.89:0/1551177908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce2dc0 0x565090ced140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 --2- 172.21.15.89:0/1551177908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce24a0 0x565090ce2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 -- 172.21.15.89:0/1551177908 >> 172.21.15.89:0/1551177908 conn(0x565090cdea30 msgr2=0x565090cdee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 -- 172.21.15.89:0/1551177908 shutdown_connections 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 -- 172.21.15.89:0/1551177908 wait complete. 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 -- start start 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.399+0000 7fef24603740 1 Processor -- start 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef24603740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090cecf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef24603740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090cea220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef24603740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ced680 0x565090cea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef24603740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x565090cf0250 con 0x565090ce2dc0 2024-09-15T23:28:13.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef24603740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x565090c03050 con 0x565090ced680 2024-09-15T23:28:13.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef24603740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565090cf03d0 con 0x565090ce24a0 2024-09-15T23:28:13.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef24603740 10 client.0 _ll_register_callbacks cb 0x565090c43160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:13.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef13fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090cea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef13fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090cea220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45234/0 (socket says 172.21.15.89:45234) 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef13fff640 1 -- 172.21.15.89:0/3483385511 learned_addr learned my addr 172.21.15.89:0/3483385511 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090cecf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef1bfff640 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ced680 0x565090cea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef13fff640 1 -- 172.21.15.89:0/3483385511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 msgr2=0x565090cecf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef13fff640 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090cecf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef13fff640 1 -- 172.21.15.89:0/3483385511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ced680 msgr2=0x565090cea760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef13fff640 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ced680 0x565090cea760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef13fff640 1 -- 172.21.15.89:0/3483385511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650909b93c0 con 0x565090ce2dc0 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef1bfff640 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ced680 0x565090cea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef13fff640 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090cea220 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fef04010410 tx=0x7fef04011680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6178ffa1186720 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7feefaffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7feefaffd640 1 -- 172.21.15.89:0/3483385511 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04016050 con 0x565090ce2dc0 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7feefaffd640 1 -- 172.21.15.89:0/3483385511 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fef04005040 con 0x565090ce2dc0 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.400+0000 7fef24603740 1 -- 172.21.15.89:0/3483385511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef04011950 con 0x565090ce2dc0 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7feefaffd640 1 -- 172.21.15.89:0/3483385511 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fef04030050 con 0x565090ce2dc0 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7fef24603740 10 client.16848 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7fef24603740 1 -- 172.21.15.89:0/3483385511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x565090cfbc70 con 0x565090ce2dc0 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7feefaffd640 1 -- 172.21.15.89:0/3483385511 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef04011950 con 0x565090ce2dc0 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7feefaffd640 1 -- 172.21.15.89:0/3483385511 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fef0402b030 con 0x565090ce2dc0 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7fef24603740 10 client.16848 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7fef24603740 10 client.16848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7fef24603740 20 client.16848 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7fef24603740 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565090cfd130 0x565090cff520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7fef24603740 4 client.16848 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7fef24603740 1 -- 172.21.15.89:0/3483385511 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565090c55850 con 0x565090cfd130 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.401+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565090cfd130 0x565090cff520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.402+0000 7feefaffd640 10 client.16848 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.402+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565090cfd130 0x565090cff520 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=228828cd283d58fa server_cookie=eed472653ff6874f in_seq=0 out_seq=0 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.402+0000 7feefaffd640 1 -- 172.21.15.89:0/3483385511 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565090c55850 con 0x565090cfd130 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.402+0000 7feefaffd640 10 client.16848 handle_command_reply: tid=0 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.402+0000 7fef24603740 1 client.16848 shutdown 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 1 -- 172.21.15.89:0/3483385511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 msgr2=0x565090cea220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090cea220 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fef04010410 tx=0x7fef04011680 comp rx=0 tx=0).stop 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 1 -- 172.21.15.89:0/3483385511 shutdown_connections 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565090cfd130 0x565090cff520 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ced680 0x565090cea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7feefaffd640 0 client.16848 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:13.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090cea220 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 1 --2- 172.21.15.89:0/3483385511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090cecf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 1 -- 172.21.15.89:0/3483385511 >> 172.21.15.89:0/3483385511 conn(0x565090cdea30 msgr2=0x565090ce4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 1 -- 172.21.15.89:0/3483385511 shutdown_connections 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 1 -- 172.21.15.89:0/3483385511 wait complete. 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.403+0000 7fef24603740 20 client.16848 trim_cache size 0 max 16384 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.420+0000 7fef24603740 1 Processor -- start 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.420+0000 7fef24603740 1 -- start start 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.420+0000 7fef24603740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090cdea30 0x565090cf94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.420+0000 7fef24603740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090ce57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.420+0000 7fef24603740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090d04b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.420+0000 7fef24603740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1b0fd0 con 0x565090ce2dc0 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.420+0000 7fef24603740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1b2b10 con 0x565090cdea30 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.420+0000 7fef24603740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1b16a0 con 0x565090ce24a0 2024-09-15T23:28:13.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.420+0000 7fef1bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090d04b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef1bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090d04b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45236/0 (socket says 172.21.15.89:45236) 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef1bfff640 1 -- 172.21.15.89:0/647254994 learned_addr learned my addr 172.21.15.89:0/647254994 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/647254994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090cdea30 0x565090cf94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef13fff640 1 --2- 172.21.15.89:0/647254994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090ce57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef1bfff640 1 -- 172.21.15.89:0/647254994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 msgr2=0x565090ce57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef1bfff640 1 --2- 172.21.15.89:0/647254994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090ce57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef1bfff640 1 -- 172.21.15.89:0/647254994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090cdea30 msgr2=0x565090cf94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef1bfff640 1 --2- 172.21.15.89:0/647254994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090cdea30 0x565090cf94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef1bfff640 1 -- 172.21.15.89:0/647254994 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565090ce8770 con 0x565090ce2dc0 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef13fff640 1 --2- 172.21.15.89:0/647254994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090ce57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef1bfff640 1 --2- 172.21.15.89:0/647254994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090d04b80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fef14031f70 tx=0x7fef1403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ec7de5d8c9821d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef117fa640 1 -- 172.21.15.89:0/647254994 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14051070 con 0x565090ce2dc0 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef117fa640 1 -- 172.21.15.89:0/647254994 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef14036040 con 0x565090ce2dc0 2024-09-15T23:28:13.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.421+0000 7fef117fa640 1 -- 172.21.15.89:0/647254994 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1402f900 con 0x565090ce2dc0 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.422+0000 7fef24603740 1 -- 172.21.15.89:0/647254994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 msgr2=0x565090d04b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.422+0000 7fef24603740 1 --2- 172.21.15.89:0/647254994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090d04b80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fef14031f70 tx=0x7fef1403f000 comp rx=0 tx=0).stop 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.422+0000 7fef24603740 1 -- 172.21.15.89:0/647254994 shutdown_connections 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.422+0000 7fef24603740 1 --2- 172.21.15.89:0/647254994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090ce2dc0 0x565090d04b80 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.422+0000 7fef24603740 1 --2- 172.21.15.89:0/647254994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090ce57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.422+0000 7fef24603740 1 --2- 172.21.15.89:0/647254994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090cdea30 0x565090cf94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.422+0000 7fef24603740 1 -- 172.21.15.89:0/647254994 >> 172.21.15.89:0/647254994 conn(0x565090cea8c0 msgr2=0x565090ce50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.422+0000 7fef24603740 1 -- 172.21.15.89:0/647254994 shutdown_connections 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.422+0000 7fef24603740 1 -- 172.21.15.89:0/647254994 wait complete. 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef24603740 1 -- start start 2024-09-15T23:28:13.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef24603740 1 Processor -- start 2024-09-15T23:28:13.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef24603740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090cdea30 0x565090cf5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef24603740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090cf1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef24603740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce2dc0 0x565090cf19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef24603740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1b2440 con 0x565090cdea30 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef24603740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1b1d70 con 0x565090ce2dc0 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef24603740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef1c1aefb0 con 0x565090ce24a0 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef24603740 10 client.0 _ll_register_callbacks cb 0x7fef04009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef1b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090cdea30 0x565090cf5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef1b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090cdea30 0x565090cf5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45252/0 (socket says 172.21.15.89:45252) 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef1b7fe640 1 -- 172.21.15.89:0/1134105404 learned_addr learned my addr 172.21.15.89:0/1134105404 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.423+0000 7fef13fff640 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090cf1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef1bfff640 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce2dc0 0x565090cf19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef1b7fe640 1 -- 172.21.15.89:0/1134105404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 msgr2=0x565090cf1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090cf1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef1b7fe640 1 -- 172.21.15.89:0/1134105404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce2dc0 msgr2=0x565090cf19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce2dc0 0x565090cf19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef1b7fe640 1 -- 172.21.15.89:0/1134105404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565090d66b50 con 0x565090cdea30 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef13fff640 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090cf1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef1bfff640 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce2dc0 0x565090cf19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef1b7fe640 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090cdea30 0x565090cf5f20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009610 tx=0x7fef0c00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1604c27fcce8113d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7feefaffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7feefaffd640 1 -- 172.21.15.89:0/1134105404 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c020040 con 0x565090cdea30 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7feefaffd640 1 -- 172.21.15.89:0/1134105404 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fef0c0982e0 con 0x565090cdea30 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef24603740 1 -- 172.21.15.89:0/1134105404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef14036040 con 0x565090cdea30 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7feefaffd640 1 -- 172.21.15.89:0/1134105404 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fef0c0a48e0 con 0x565090cdea30 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef24603740 10 client.16854 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7fef24603740 1 -- 172.21.15.89:0/1134105404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x565090d67d80 con 0x565090cdea30 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.424+0000 7feefaffd640 1 -- 172.21.15.89:0/1134105404 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef14036040 con 0x565090cdea30 2024-09-15T23:28:13.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.425+0000 7feefaffd640 1 -- 172.21.15.89:0/1134105404 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fef0c09abc0 con 0x565090cdea30 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.425+0000 7fef24603740 10 client.16854 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.425+0000 7fef24603740 10 client.16854 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.425+0000 7fef24603740 20 client.16854 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.425+0000 7fef24603740 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565090ced680 0x565090d6a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.425+0000 7fef24603740 4 client.16854 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.425+0000 7fef24603740 1 -- 172.21.15.89:0/1134105404 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565090c4fd60 con 0x565090ced680 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.425+0000 7fef13fff640 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565090ced680 0x565090d6a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.425+0000 7feefaffd640 10 client.16854 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.425+0000 7fef13fff640 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565090ced680 0x565090d6a4b0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f94d52bb21b3ae8f server_cookie=9d8c1650b9ec5390 in_seq=0 out_seq=0 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7feefaffd640 1 -- 172.21.15.89:0/1134105404 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x565090c4fd60 con 0x565090ced680 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7feefaffd640 10 client.16854 handle_command_reply: tid=0 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 client.16854 shutdown 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 -- 172.21.15.89:0/1134105404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090cdea30 msgr2=0x565090cf5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:13.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090cdea30 0x565090cf5f20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009610 tx=0x7fef0c00bf60 comp rx=0 tx=0).stop 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 -- 172.21.15.89:0/1134105404 shutdown_connections 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x565090ced680 0x565090d6a4b0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x565090ce2dc0 0x565090cf19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565090ce24a0 0x565090cf1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 --2- 172.21.15.89:0/1134105404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565090cdea30 0x565090cf5f20 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 -- 172.21.15.89:0/1134105404 >> 172.21.15.89:0/1134105404 conn(0x565090cea8c0 msgr2=0x565090cec2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7feefaffd640 0 client.16854 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 -- 172.21.15.89:0/1134105404 shutdown_connections 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 1 -- 172.21.15.89:0/1134105404 wait complete. 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.426+0000 7fef24603740 20 client.16854 trim_cache size 0 max 16384 2024-09-15T23:28:13.443 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:28:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:28:13.444 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:28:13.444 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:28:13.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 -- 172.21.15.89:0/1038400567 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 msgr2=0x7fef080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:13.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 0x7fef080a5700 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fef14005e30 tx=0x7fef140095c0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 -- 172.21.15.89:0/1038400567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef1c101470 msgr2=0x7fef1c19a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef1c101470 0x7fef1c19a5f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00d950 tx=0x7fef0c00de10 comp rx=0 tx=0).stop 2024-09-15T23:28:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 -- 172.21.15.89:0/1038400567 shutdown_connections 2024-09-15T23:28:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 0x7fef080a5700 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef1c105de0 0x7fef1c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef1c101e20 0x7fef1c19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 --2- 172.21.15.89:0/1038400567 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef1c101470 0x7fef1c19a5f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 -- 172.21.15.89:0/1038400567 >> 172.21.15.89:0/1038400567 conn(0x7fef1c077230 msgr2=0x7fef1c100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 -- 172.21.15.89:0/1038400567 shutdown_connections 2024-09-15T23:28:13.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:13.427+0000 7fef2222a640 1 -- 172.21.15.89:0/1038400567 wait complete. 2024-09-15T23:28:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:13 smithi120 ceph-mon[35250]: pgmap v476: 129 pgs: 129 active+clean; 2.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:28:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:13 smithi174 ceph-mon[35586]: pgmap v476: 129 pgs: 129 active+clean; 2.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:28:14.389 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:28:14.389 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:28:14.639 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:15.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.129+0000 7fbb55744640 1 -- 172.21.15.89:0/1523957006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb50100a80 msgr2=0x7fbb50100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.129+0000 7fbb55744640 1 --2- 172.21.15.89:0/1523957006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb50100a80 0x7fbb50100ee0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009cd0 tx=0x7fbb3802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.129+0000 7fbb55744640 1 -- 172.21.15.89:0/1523957006 shutdown_connections 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.129+0000 7fbb55744640 1 --2- 172.21.15.89:0/1523957006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101420 0x7fbb5010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.129+0000 7fbb55744640 1 --2- 172.21.15.89:0/1523957006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb50100a80 0x7fbb50100ee0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.129+0000 7fbb55744640 1 --2- 172.21.15.89:0/1523957006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb50106a80 0x7fbb50106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.129+0000 7fbb55744640 1 -- 172.21.15.89:0/1523957006 >> 172.21.15.89:0/1523957006 conn(0x7fbb500fc7a0 msgr2=0x7fbb500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.130+0000 7fbb55744640 1 -- 172.21.15.89:0/1523957006 shutdown_connections 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.130+0000 7fbb55744640 1 -- 172.21.15.89:0/1523957006 wait complete. 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.130+0000 7fbb55744640 1 Processor -- start 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.130+0000 7fbb55744640 1 -- start start 2024-09-15T23:28:15.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.130+0000 7fbb55744640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb50100a80 0x7fbb501973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.130+0000 7fbb55744640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101420 0x7fbb50197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.130+0000 7fbb55744640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb50106a80 0x7fbb5019bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.130+0000 7fbb55744640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb50110ea0 con 0x7fbb50106a80 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.130+0000 7fbb55744640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb50110d20 con 0x7fbb50101420 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.130+0000 7fbb55744640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb50111020 con 0x7fbb50100a80 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb50106a80 0x7fbb5019bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb50106a80 0x7fbb5019bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45266/0 (socket says 172.21.15.89:45266) 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4f7fe640 1 -- 172.21.15.89:0/2616385933 learned_addr learned my addr 172.21.15.89:0/2616385933 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4effd640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb50100a80 0x7fbb501973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4f7fe640 1 -- 172.21.15.89:0/2616385933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb50100a80 msgr2=0x7fbb501973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4e7fc640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101420 0x7fbb50197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4f7fe640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb50100a80 0x7fbb501973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4f7fe640 1 -- 172.21.15.89:0/2616385933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101420 msgr2=0x7fbb50197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4f7fe640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101420 0x7fbb50197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4f7fe640 1 -- 172.21.15.89:0/2616385933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5019c370 con 0x7fbb50106a80 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb4f7fe640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb50106a80 0x7fbb5019bc90 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009fd0 tx=0x7fbb4000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb47fff640 1 -- 172.21.15.89:0/2616385933 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb40011830 con 0x7fbb50106a80 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb55744640 1 -- 172.21.15.89:0/2616385933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5019c660 con 0x7fbb50106a80 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb47fff640 1 -- 172.21.15.89:0/2616385933 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb40016070 con 0x7fbb50106a80 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb47fff640 1 -- 172.21.15.89:0/2616385933 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4000f2e0 con 0x7fbb50106a80 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.131+0000 7fbb55744640 1 -- 172.21.15.89:0/2616385933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb501a3f40 con 0x7fbb50106a80 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.132+0000 7fbb55744640 1 -- 172.21.15.89:0/2616385933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb50100ee0 con 0x7fbb50106a80 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.133+0000 7fbb47fff640 1 -- 172.21.15.89:0/2616385933 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbb400119d0 con 0x7fbb50106a80 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.134+0000 7fbb47fff640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb3c0a3240 0x7fbb3c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.134+0000 7fbb47fff640 1 -- 172.21.15.89:0/2616385933 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb400cf3c0 con 0x7fbb50106a80 2024-09-15T23:28:15.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.134+0000 7fbb4effd640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb3c0a3240 0x7fbb3c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.135+0000 7fbb4effd640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb3c0a3240 0x7fbb3c0a5700 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbb48005fd0 tx=0x7fbb48005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:15.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.138+0000 7fbb47fff640 1 -- 172.21.15.89:0/2616385933 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbb4009bd90 con 0x7fbb50106a80 2024-09-15T23:28:15.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:15 smithi089 ceph-mon[30933]: pgmap v477: 129 pgs: 129 active+clean; 2.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:28:15.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.341+0000 7fbb55744640 1 -- 172.21.15.89:0/2616385933 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb50109010 con 0x7fbb50106a80 2024-09-15T23:28:15.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.342+0000 7fbb47fff640 1 -- 172.21.15.89:0/2616385933 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fbb4009b690 con 0x7fbb50106a80 2024-09-15T23:28:15.355 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:28:15.355 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:28:15.355 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:28:15.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 -- 172.21.15.89:0/2616385933 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb3c0a3240 msgr2=0x7fbb3c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb3c0a3240 0x7fbb3c0a5700 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbb48005fd0 tx=0x7fbb48005850 comp rx=0 tx=0).stop 2024-09-15T23:28:15.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 -- 172.21.15.89:0/2616385933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb50106a80 msgr2=0x7fbb5019bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb50106a80 0x7fbb5019bc90 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fbb40009fd0 tx=0x7fbb4000b8e0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 -- 172.21.15.89:0/2616385933 shutdown_connections 2024-09-15T23:28:15.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb3c0a3240 0x7fbb3c0a5700 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb50106a80 0x7fbb5019bc90 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101420 0x7fbb50197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 --2- 172.21.15.89:0/2616385933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb50100a80 0x7fbb501973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 -- 172.21.15.89:0/2616385933 >> 172.21.15.89:0/2616385933 conn(0x7fbb500fc7a0 msgr2=0x7fbb500fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:15.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 -- 172.21.15.89:0/2616385933 shutdown_connections 2024-09-15T23:28:15.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:15.344+0000 7fbb55744640 1 -- 172.21.15.89:0/2616385933 wait complete. 2024-09-15T23:28:15.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:15 smithi120 ceph-mon[35250]: pgmap v477: 129 pgs: 129 active+clean; 2.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:28:15.496 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:15 smithi174 ceph-mon[35586]: pgmap v477: 129 pgs: 129 active+clean; 2.0 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:28:16.017 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:28:16.263 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:16.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:16 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2616385933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:16.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:16 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2616385933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:16 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2616385933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.765+0000 7f8c0c891640 1 -- 172.21.15.89:0/1530338699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c08101de0 msgr2=0x7f8c081021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.765+0000 7f8c0c891640 1 --2- 172.21.15.89:0/1530338699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c08101de0 0x7f8c081021c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0009900 tx=0x7f8bf002a410 comp rx=0 tx=0).stop 2024-09-15T23:28:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.765+0000 7f8c0c891640 1 -- 172.21.15.89:0/1530338699 shutdown_connections 2024-09-15T23:28:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.765+0000 7f8c0c891640 1 --2- 172.21.15.89:0/1530338699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c08106f10 0x7f8c0810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.765+0000 7f8c0c891640 1 --2- 172.21.15.89:0/1530338699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c08102790 0x7f8c081067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.765+0000 7f8c0c891640 1 --2- 172.21.15.89:0/1530338699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c08101de0 0x7f8c081021c0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.765+0000 7f8c0c891640 1 -- 172.21.15.89:0/1530338699 >> 172.21.15.89:0/1530338699 conn(0x7f8c080fd8c0 msgr2=0x7f8c080ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.765+0000 7f8c0c891640 1 -- 172.21.15.89:0/1530338699 shutdown_connections 2024-09-15T23:28:16.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.765+0000 7f8c0c891640 1 -- 172.21.15.89:0/1530338699 wait complete. 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.765+0000 7f8c0c891640 1 Processor -- start 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0c891640 1 -- start start 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0c891640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c08101de0 0x7f8c08198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0c891640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c08102790 0x7f8c08198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0c891640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c08106f10 0x7f8c0819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0c891640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c08111f70 con 0x7f8c08101de0 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0c891640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c08111df0 con 0x7f8c08106f10 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0c891640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081120f0 con 0x7f8c08102790 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c08101de0 0x7f8c08198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c08101de0 0x7f8c08198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48834/0 (socket says 172.21.15.89:48834) 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0659c640 1 -- 172.21.15.89:0/2987886647 learned_addr learned my addr 172.21.15.89:0/2987886647 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c08106f10 0x7f8c0819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c05d9b640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c08102790 0x7f8c08198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0659c640 1 -- 172.21.15.89:0/2987886647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c08102790 msgr2=0x7f8c08198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:16.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0659c640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c08102790 0x7f8c08198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0659c640 1 -- 172.21.15.89:0/2987886647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c08106f10 msgr2=0x7f8c0819cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:16.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0659c640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c08106f10 0x7f8c0819cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0659c640 1 -- 172.21.15.89:0/2987886647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0819d3e0 con 0x7f8c08101de0 2024-09-15T23:28:16.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c08106f10 0x7f8c0819cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:16.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.766+0000 7f8c0659c640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c08101de0 0x7f8c08198430 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0006010 tx=0x7f8bf00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:16.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.767+0000 7f8bff7fe640 1 -- 172.21.15.89:0/2987886647 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf003f030 con 0x7f8c08101de0 2024-09-15T23:28:16.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.767+0000 7f8bff7fe640 1 -- 172.21.15.89:0/2987886647 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf0003ce0 con 0x7f8c08101de0 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.767+0000 7f8c0c891640 1 -- 172.21.15.89:0/2987886647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c0819d670 con 0x7f8c08101de0 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.767+0000 7f8bff7fe640 1 -- 172.21.15.89:0/2987886647 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0035070 con 0x7f8c08101de0 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.767+0000 7f8c0c891640 1 -- 172.21.15.89:0/2987886647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c081a4fb0 con 0x7f8c08101de0 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.769+0000 7f8bff7fe640 1 -- 172.21.15.89:0/2987886647 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8bf0042030 con 0x7f8c08101de0 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.769+0000 7f8c0ec6a740 1 Processor -- start 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.769+0000 7f8c0ec6a740 1 -- start start 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.769+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe94090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.769+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe94a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.769+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 0x56220fea1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.769+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56220fdfa2f0 con 0x56220fe95b40 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56220fda1cf0 con 0x56220fe945d0 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56220fdf62b0 con 0x56220fe9f370 2024-09-15T23:28:16.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe94090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf40a3240 0x7f8bf40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe94090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48846/0 (socket says 172.21.15.89:48846) 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 -- 172.21.15.89:0/3179567982 learned_addr learned my addr 172.21.15.89:0/3179567982 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c05d9b640 1 --2- 172.21.15.89:0/3179567982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 0x56220fea1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8bff7fe640 1 -- 172.21.15.89:0/2987886647 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8bf00f0350 con 0x7f8c08101de0 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf40a3240 0x7f8bf40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c0659c640 1 --2- 172.21.15.89:0/3179567982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe94a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 -- 172.21.15.89:0/3179567982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 msgr2=0x56220fea1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/3179567982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 0x56220fea1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 -- 172.21.15.89:0/3179567982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 msgr2=0x56220fe94a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/3179567982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe94a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 -- 172.21.15.89:0/3179567982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56220fd863a0 con 0x56220fe95b40 2024-09-15T23:28:16.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/3179567982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe94090 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8c00014930 tx=0x7f8c00014df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7785ed651533110a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8bfcff9640 1 -- 172.21.15.89:0/3179567982 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00021070 con 0x56220fe95b40 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8bfcff9640 1 -- 172.21.15.89:0/3179567982 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c00008df0 con 0x56220fe95b40 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf40a3240 0x7f8bf40a5700 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8c0001e000 tx=0x7f8c0001a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.770+0000 7f8bfcff9640 1 -- 172.21.15.89:0/3179567982 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0000f630 con 0x56220fe95b40 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.771+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3179567982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 msgr2=0x56220fe94090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.771+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/3179567982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe94090 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8c00014930 tx=0x7f8c00014df0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.771+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3179567982 shutdown_connections 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.771+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/3179567982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 0x56220fea1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.771+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/3179567982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe94a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.771+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/3179567982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe94090 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.771+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3179567982 >> 172.21.15.89:0/3179567982 conn(0x56220fe5d310 msgr2=0x56220fe90c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.771+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3179567982 shutdown_connections 2024-09-15T23:28:16.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.771+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3179567982 wait complete. 2024-09-15T23:28:16.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c0ec6a740 1 -- start start 2024-09-15T23:28:16.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c0ec6a740 1 Processor -- start 2024-09-15T23:28:16.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe9ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe95b40 0x56220fe9c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220fe9c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56220fea2070 con 0x56220fe9f370 2024-09-15T23:28:16.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56220fdb5060 con 0x56220fe945d0 2024-09-15T23:28:16.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56220fea21f0 con 0x56220fe95b40 2024-09-15T23:28:16.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c0ec6a740 10 client.0 _ll_register_callbacks cb 0x56220fb088c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:16.802 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:28:16.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c06d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220fe9c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c06d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220fe9c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48860/0 (socket says 172.21.15.89:48860) 2024-09-15T23:28:16.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c06d9d640 1 -- 172.21.15.89:0/2509255333 learned_addr learned my addr 172.21.15.89:0/2509255333 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:16.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c0659c640 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe9ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c06d9d640 1 -- 172.21.15.89:0/2509255333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe95b40 msgr2=0x56220fe9c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:16.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe95b40 0x56220fe9c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c06d9d640 1 -- 172.21.15.89:0/2509255333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 msgr2=0x56220fe9ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe9ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.772+0000 7f8c06d9d640 1 -- 172.21.15.89:0/2509255333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56220fb6b3c0 con 0x56220fe9f370 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220fe9c650 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f8c00005c70 tx=0x7f8c0000f800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9644e57cb1826bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8bedffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8bedffb640 1 -- 172.21.15.89:0/2509255333 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00021050 con 0x56220fe9f370 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8bedffb640 1 -- 172.21.15.89:0/2509255333 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8c0000fd80 con 0x56220fe9f370 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/2509255333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c00008df0 con 0x56220fe9f370 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8bedffb640 1 -- 172.21.15.89:0/2509255333 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c0003a050 con 0x56220fe9f370 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8c0ec6a740 10 client.16872 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/2509255333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56220feadb40 con 0x56220fe9f370 2024-09-15T23:28:16.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8bedffb640 1 -- 172.21.15.89:0/2509255333 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c00008df0 con 0x56220fe9f370 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8bedffb640 1 -- 172.21.15.89:0/2509255333 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8c000254f0 con 0x56220fe9f370 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8c0ec6a740 10 client.16872 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8c0ec6a740 10 client.16872 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8c0ec6a740 20 client.16872 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56220feaee90 0x56220feb1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8c0ec6a740 4 client.16872 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.773+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/2509255333 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56220fe07860 con 0x56220feaee90 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.774+0000 7f8c0659c640 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56220feaee90 0x56220feb1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.774+0000 7f8bedffb640 10 client.16872 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.774+0000 7f8c0659c640 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56220feaee90 0x56220feb1350 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fbb8764c27df2bf4 server_cookie=21c7738153c30c3c in_seq=0 out_seq=0 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8bedffb640 1 -- 172.21.15.89:0/2509255333 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56220fe07860 con 0x56220feaee90 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8bedffb640 10 client.16872 handle_command_reply: tid=0 2024-09-15T23:28:16.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 client.16872 shutdown 2024-09-15T23:28:16.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/2509255333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 msgr2=0x56220fe9c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:16.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220fe9c650 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f8c00005c70 tx=0x7f8c0000f800 comp rx=0 tx=0).stop 2024-09-15T23:28:16.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/2509255333 shutdown_connections 2024-09-15T23:28:16.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56220feaee90 0x56220feb1350 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220fe9c650 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8bedffb640 0 client.16872 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:28:16.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe95b40 0x56220fe9c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/2509255333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe9ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/2509255333 >> 172.21.15.89:0/2509255333 conn(0x56220fe5d310 msgr2=0x56220fe96960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/2509255333 shutdown_connections 2024-09-15T23:28:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/2509255333 wait complete. 2024-09-15T23:28:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.775+0000 7f8c0ec6a740 20 client.16872 trim_cache size 0 max 16384 2024-09-15T23:28:16.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c0ec6a740 1 Processor -- start 2024-09-15T23:28:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c0ec6a740 1 -- start start 2024-09-15T23:28:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe8fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe97120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 0x56220fe97660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081aee70 con 0x56220fe95b40 2024-09-15T23:28:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081b09b0 con 0x56220fe945d0 2024-09-15T23:28:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081af540 con 0x56220fe9f370 2024-09-15T23:28:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c05d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe97120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c05d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe97120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48874/0 (socket says 172.21.15.89:48874) 2024-09-15T23:28:16.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c05d9b640 1 -- 172.21.15.89:0/1728761222 learned_addr learned my addr 172.21.15.89:0/1728761222 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/1728761222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 0x56220fe97660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c05d9b640 1 -- 172.21.15.89:0/1728761222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 msgr2=0x56220fe97660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c05d9b640 1 --2- 172.21.15.89:0/1728761222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 0x56220fe97660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c05d9b640 1 -- 172.21.15.89:0/1728761222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 msgr2=0x56220fe8fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c05d9b640 1 --2- 172.21.15.89:0/1728761222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe8fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c05d9b640 1 -- 172.21.15.89:0/1728761222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56220fe9a590 con 0x56220fe95b40 2024-09-15T23:28:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.788+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/1728761222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 0x56220fe97660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8c05d9b640 1 --2- 172.21.15.89:0/1728761222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe97120 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009ac0 tx=0x7f8bf8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12920aec19fc280 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8beffff640 1 -- 172.21.15.89:0/1728761222 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8011bc0 con 0x56220fe95b40 2024-09-15T23:28:16.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8beffff640 1 -- 172.21.15.89:0/1728761222 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf8011d60 con 0x56220fe95b40 2024-09-15T23:28:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8beffff640 1 -- 172.21.15.89:0/1728761222 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8012470 con 0x56220fe95b40 2024-09-15T23:28:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/1728761222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 msgr2=0x56220fe97120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/1728761222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe97120 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009ac0 tx=0x7f8bf8009f80 comp rx=0 tx=0).stop 2024-09-15T23:28:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/1728761222 shutdown_connections 2024-09-15T23:28:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/1728761222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe9f370 0x56220fe97660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/1728761222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe95b40 0x56220fe97120 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/1728761222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fe8fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/1728761222 >> 172.21.15.89:0/1728761222 conn(0x56220fe5d310 msgr2=0x56220fe8f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/1728761222 shutdown_connections 2024-09-15T23:28:16.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.789+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/1728761222 wait complete. 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c0ec6a740 1 -- start start 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c0ec6a740 1 Processor -- start 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fea3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe95b40 0x56220fea3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c0ec6a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220ff18c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081b02e0 con 0x56220fe9f370 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081afc10 con 0x56220fe945d0 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c0ec6a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c081ace50 con 0x56220fe95b40 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c0ec6a740 10 client.0 _ll_register_callbacks cb 0x7f8c0001f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c06d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220ff18c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c06d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220ff18c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48880/0 (socket says 172.21.15.89:48880) 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c06d9d640 1 -- 172.21.15.89:0/3894482066 learned_addr learned my addr 172.21.15.89:0/3894482066 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c0659c640 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fea3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c05d9b640 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe95b40 0x56220fea3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c06d9d640 1 -- 172.21.15.89:0/3894482066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe95b40 msgr2=0x56220fea3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe95b40 0x56220fea3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c06d9d640 1 -- 172.21.15.89:0/3894482066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 msgr2=0x56220fea3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fea3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c06d9d640 1 -- 172.21.15.89:0/3894482066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56220ff19180 con 0x56220fe9f370 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.790+0000 7f8c05d9b640 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe95b40 0x56220fea3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8c06d9d640 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220ff18c40 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8c00008540 tx=0x7f8c0000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201ef59ba0a235b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8bedffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8bedffb640 1 -- 172.21.15.89:0/3894482066 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00043060 con 0x56220fe9f370 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8bedffb640 1 -- 172.21.15.89:0/3894482066 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8c00025660 con 0x56220fe9f370 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3894482066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bf8011d60 con 0x56220fe9f370 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8bedffb640 1 -- 172.21.15.89:0/3894482066 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c0004a050 con 0x56220fe9f370 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8c0ec6a740 10 client.16878 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:16.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3894482066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56220ff1a330 con 0x56220fe9f370 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8bedffb640 1 -- 172.21.15.89:0/3894482066 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf8011d60 con 0x56220fe9f370 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8bedffb640 1 -- 172.21.15.89:0/3894482066 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8c0000fb80 con 0x56220fe9f370 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8c0ec6a740 10 client.16878 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8c0ec6a740 10 client.16878 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8c0ec6a740 20 client.16878 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56220ff1a930 0x56220ff1cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8c0ec6a740 4 client.16878 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.791+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3894482066 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56220fe01d70 con 0x56220ff1a930 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0659c640 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56220ff1a930 0x56220ff1cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0659c640 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56220ff1a930 0x56220ff1cd20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b1e818d231cd6661 server_cookie=cf20509168952382 in_seq=0 out_seq=0 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8bedffb640 10 client.16878 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8bedffb640 1 -- 172.21.15.89:0/3894482066 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56220fe01d70 con 0x56220ff1a930 2024-09-15T23:28:16.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8bedffb640 10 client.16878 handle_command_reply: tid=0 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 client.16878 shutdown 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3894482066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 msgr2=0x56220ff18c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220ff18c40 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8c00008540 tx=0x7f8c0000ee70 comp rx=0 tx=0).stop 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3894482066 shutdown_connections 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56220ff1a930 0x56220ff1cd20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56220fe9f370 0x56220ff18c40 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56220fe95b40 0x56220fea3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 --2- 172.21.15.89:0/3894482066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56220fe945d0 0x56220fea3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3894482066 >> 172.21.15.89:0/3894482066 conn(0x56220fe5d310 msgr2=0x56220fe8fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8bedffb640 0 client.16878 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3894482066 shutdown_connections 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 1 -- 172.21.15.89:0/3894482066 wait complete. 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.792+0000 7f8c0ec6a740 20 client.16878 trim_cache size 0 max 16384 2024-09-15T23:28:16.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 -- 172.21.15.89:0/2987886647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf40a3240 msgr2=0x7f8bf40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf40a3240 0x7f8bf40a5700 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8c0001e000 tx=0x7f8c0001a000 comp rx=0 tx=0).stop 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 -- 172.21.15.89:0/2987886647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c08101de0 msgr2=0x7f8c08198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c08101de0 0x7f8c08198430 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0006010 tx=0x7f8bf00042f0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 -- 172.21.15.89:0/2987886647 shutdown_connections 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf40a3240 0x7f8bf40a5700 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c08106f10 0x7f8c0819cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c08102790 0x7f8c08198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 --2- 172.21.15.89:0/2987886647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c08101de0 0x7f8c08198430 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 -- 172.21.15.89:0/2987886647 >> 172.21.15.89:0/2987886647 conn(0x7f8c080fd8c0 msgr2=0x7f8c080fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 -- 172.21.15.89:0/2987886647 shutdown_connections 2024-09-15T23:28:16.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:16.793+0000 7f8c0c891640 1 -- 172.21.15.89:0/2987886647 wait complete. 2024-09-15T23:28:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:17 smithi120 ceph-mon[35250]: pgmap v478: 129 pgs: 129 active+clean; 2.1 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:28:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:17 smithi174 ceph-mon[35586]: pgmap v478: 129 pgs: 129 active+clean; 2.1 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:28:17.551 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:17 smithi089 ceph-mon[30933]: pgmap v478: 129 pgs: 129 active+clean; 2.1 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:28:17.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:28:17.822 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:18.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.303+0000 7f546c3f5640 1 -- 172.21.15.89:0/3738723484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5464106f10 msgr2=0x7f546410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:18.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.303+0000 7f546c3f5640 1 --2- 172.21.15.89:0/3738723484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5464106f10 0x7f546410ea90 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f545c009ca0 tx=0x7f545c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.303+0000 7f546c3f5640 1 -- 172.21.15.89:0/3738723484 shutdown_connections 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.303+0000 7f546c3f5640 1 --2- 172.21.15.89:0/3738723484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5464106f10 0x7f546410ea90 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.303+0000 7f546c3f5640 1 --2- 172.21.15.89:0/3738723484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5464102790 0x7f54641067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.303+0000 7f546c3f5640 1 --2- 172.21.15.89:0/3738723484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464101de0 0x7f54641021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.303+0000 7f546c3f5640 1 -- 172.21.15.89:0/3738723484 >> 172.21.15.89:0/3738723484 conn(0x7f54640fd8c0 msgr2=0x7f54640ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.303+0000 7f546c3f5640 1 -- 172.21.15.89:0/3738723484 shutdown_connections 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.303+0000 7f546c3f5640 1 -- 172.21.15.89:0/3738723484 wait complete. 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546c3f5640 1 Processor -- start 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546c3f5640 1 -- start start 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546c3f5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464101de0 0x7f54641983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546c3f5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5464102790 0x7f54641988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546c3f5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5464106f10 0x7f546419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546c3f5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5464111f70 con 0x7f5464106f10 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546c3f5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5464111df0 con 0x7f5464102790 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546c3f5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f54641120f0 con 0x7f5464101de0 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546a992640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5464106f10 0x7f546419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546a992640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5464106f10 0x7f546419cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48902/0 (socket says 172.21.15.89:48902) 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.304+0000 7f546a992640 1 -- 172.21.15.89:0/2989033798 learned_addr learned my addr 172.21.15.89:0/2989033798 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f5469990640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5464102790 0x7f54641988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f546a191640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464101de0 0x7f54641983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f546a992640 1 -- 172.21.15.89:0/2989033798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464101de0 msgr2=0x7f54641983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f546a992640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464101de0 0x7f54641983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f546a992640 1 -- 172.21.15.89:0/2989033798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5464102790 msgr2=0x7f54641988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f546a992640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5464102790 0x7f54641988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f546a992640 1 -- 172.21.15.89:0/2989033798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546419d360 con 0x7f5464106f10 2024-09-15T23:28:18.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f5469990640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5464102790 0x7f54641988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f546a992640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5464106f10 0x7f546419cc80 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f545c031a20 tx=0x7f545c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f545b7fe640 1 -- 172.21.15.89:0/2989033798 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c003c30 con 0x7f5464106f10 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f545b7fe640 1 -- 172.21.15.89:0/2989033798 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f545c003dd0 con 0x7f5464106f10 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f546c3f5640 1 -- 172.21.15.89:0/2989033798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546419d5f0 con 0x7f5464106f10 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f545b7fe640 1 -- 172.21.15.89:0/2989033798 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c02fcf0 con 0x7f5464106f10 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.305+0000 7f546c3f5640 1 -- 172.21.15.89:0/2989033798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54641a4f30 con 0x7f5464106f10 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.307+0000 7f545b7fe640 1 -- 172.21.15.89:0/2989033798 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f545c004050 con 0x7f5464106f10 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546e7ce740 1 Processor -- start 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546e7ce740 1 -- start start 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f545b7fe640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54500a3240 0x7f54500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9ecb40 0x563bda9eb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9eb5d0 0x563bda9eba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9fa780 0x563bda9fcb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546e7ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563bda9512f0 con 0x563bda9ecb40 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546e7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563bda8f8cf0 con 0x563bda9fa780 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546e7ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563bda94d2b0 con 0x563bda9eb5d0 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546a191640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9ecb40 0x563bda9eb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f5469990640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54500a3240 0x7f54500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546a191640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9ecb40 0x563bda9eb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48904/0 (socket says 172.21.15.89:48904) 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f546a191640 1 -- 172.21.15.89:0/539333717 learned_addr learned my addr 172.21.15.89:0/539333717 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f545b7fe640 1 -- 172.21.15.89:0/2989033798 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f545c0f11b0 con 0x7f5464106f10 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.308+0000 7f5469990640 1 --2- 172.21.15.89:0/539333717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9eb5d0 0x563bda9eba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f546a992640 1 --2- 172.21.15.89:0/539333717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9fa780 0x563bda9fcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f546a191640 1 -- 172.21.15.89:0/539333717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9eb5d0 msgr2=0x563bda9eba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f546a191640 1 --2- 172.21.15.89:0/539333717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9eb5d0 0x563bda9eba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f546a191640 1 -- 172.21.15.89:0/539333717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9fa780 msgr2=0x563bda9fcb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f546a191640 1 --2- 172.21.15.89:0/539333717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9fa780 0x563bda9fcb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f546a191640 1 -- 172.21.15.89:0/539333717 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bda8dd3a0 con 0x563bda9ecb40 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f546a992640 1 --2- 172.21.15.89:0/539333717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9fa780 0x563bda9fcb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f5469990640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54500a3240 0x7f54500a5700 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5454005d80 tx=0x7f5454013890 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f546a191640 1 --2- 172.21.15.89:0/539333717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9ecb40 0x563bda9eb090 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f5460009bc0 tx=0x7f546000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a509f1b5cb96d18 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f5458dea640 1 -- 172.21.15.89:0/539333717 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460016070 con 0x563bda9ecb40 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f5458dea640 1 -- 172.21.15.89:0/539333717 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5460005500 con 0x563bda9ecb40 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.309+0000 7f5458dea640 1 -- 172.21.15.89:0/539333717 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460011470 con 0x563bda9ecb40 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 -- 172.21.15.89:0/539333717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9ecb40 msgr2=0x563bda9eb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 --2- 172.21.15.89:0/539333717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9ecb40 0x563bda9eb090 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f5460009bc0 tx=0x7f546000e6c0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 -- 172.21.15.89:0/539333717 shutdown_connections 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 --2- 172.21.15.89:0/539333717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9fa780 0x563bda9fcb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 --2- 172.21.15.89:0/539333717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9eb5d0 0x563bda9eba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 --2- 172.21.15.89:0/539333717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9ecb40 0x563bda9eb090 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 -- 172.21.15.89:0/539333717 >> 172.21.15.89:0/539333717 conn(0x563bda9b4310 msgr2=0x563bda9e7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 -- 172.21.15.89:0/539333717 shutdown_connections 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 -- 172.21.15.89:0/539333717 wait complete. 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 -- start start 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 Processor -- start 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9eb5d0 0x563bda9fa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9ecb40 0x563bda9f7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9fa780 0x563bda9f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.310+0000 7f546e7ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563bda9fd480 con 0x563bda9fa780 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546e7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563bda90c060 con 0x563bda9eb5d0 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546e7ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563bda9fd600 con 0x563bda9ecb40 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546e7ce740 10 client.0 _ll_register_callbacks cb 0x563bda65f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a992640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9fa780 0x563bda9f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a992640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9fa780 0x563bda9f7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48918/0 (socket says 172.21.15.89:48918) 2024-09-15T23:28:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a992640 1 -- 172.21.15.89:0/2537382255 learned_addr learned my addr 172.21.15.89:0/2537382255 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a191640 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9eb5d0 0x563bda9fa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f5469990640 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9ecb40 0x563bda9f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a992640 1 -- 172.21.15.89:0/2537382255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9ecb40 msgr2=0x563bda9f7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a992640 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9ecb40 0x563bda9f7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a992640 1 -- 172.21.15.89:0/2537382255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9eb5d0 msgr2=0x563bda9fa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a992640 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9eb5d0 0x563bda9fa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a992640 1 -- 172.21.15.89:0/2537382255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bda6c23c0 con 0x563bda9fa780 2024-09-15T23:28:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f5469990640 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9ecb40 0x563bda9f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:18.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a191640 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9eb5d0 0x563bda9fa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546a992640 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9fa780 0x563bda9f7a10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f545c005e30 tx=0x7f545c0bce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40a6c6860809c2b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f544dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f544dffb640 1 -- 172.21.15.89:0/2537382255 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c0bd410 con 0x563bda9fa780 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f544dffb640 1 -- 172.21.15.89:0/2537382255 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f545c0bb710 con 0x563bda9fa780 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546e7ce740 1 -- 172.21.15.89:0/2537382255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5460005500 con 0x563bda9fa780 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f544dffb640 1 -- 172.21.15.89:0/2537382255 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f545c0c17c0 con 0x563bda9fa780 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546e7ce740 10 client.16890 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f546e7ce740 1 -- 172.21.15.89:0/2537382255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563bda9fd480 con 0x563bda9fa780 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.311+0000 7f544dffb640 1 -- 172.21.15.89:0/2537382255 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5460005500 con 0x563bda9fa780 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.312+0000 7f544dffb640 1 -- 172.21.15.89:0/2537382255 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f545c044c30 con 0x563bda9fa780 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.312+0000 7f546e7ce740 10 client.16890 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.312+0000 7f546e7ce740 10 client.16890 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:28:18.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.312+0000 7f546e7ce740 20 client.16890 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.312+0000 7f546e7ce740 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x563bdaa0a340 0x563bdaa0c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.312+0000 7f546e7ce740 4 client.16890 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.312+0000 7f546e7ce740 1 -- 172.21.15.89:0/2537382255 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bda95e860 con 0x563bdaa0a340 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.312+0000 7f546a191640 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x563bdaa0a340 0x563bdaa0c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.312+0000 7f544dffb640 10 client.16890 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.312+0000 7f546a191640 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x563bdaa0a340 0x563bdaa0c730 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4a23e8db4e8ab2f3 server_cookie=4f32072dfe23f521 in_seq=0 out_seq=0 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f544dffb640 1 -- 172.21.15.89:0/2537382255 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563bda95e860 con 0x563bdaa0a340 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f544dffb640 10 client.16890 handle_command_reply: tid=0 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 client.16890 shutdown 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 -- 172.21.15.89:0/2537382255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9fa780 msgr2=0x563bda9f7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9fa780 0x563bda9f7a10 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f545c005e30 tx=0x7f545c0bce30 comp rx=0 tx=0).stop 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 -- 172.21.15.89:0/2537382255 shutdown_connections 2024-09-15T23:28:18.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x563bdaa0a340 0x563bdaa0c730 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9fa780 0x563bda9f7a10 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9ecb40 0x563bda9f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f544dffb640 0 client.16890 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:28:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 --2- 172.21.15.89:0/2537382255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9eb5d0 0x563bda9fa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 -- 172.21.15.89:0/2537382255 >> 172.21.15.89:0/2537382255 conn(0x563bda9b4310 msgr2=0x563bda9ed960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 -- 172.21.15.89:0/2537382255 shutdown_connections 2024-09-15T23:28:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.313+0000 7f546e7ce740 1 -- 172.21.15.89:0/2537382255 wait complete. 2024-09-15T23:28:18.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.314+0000 7f546e7ce740 20 client.16890 trim_cache size 0 max 16384 2024-09-15T23:28:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546e7ce740 1 Processor -- start 2024-09-15T23:28:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546e7ce740 1 -- start start 2024-09-15T23:28:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bda9f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9ecb40 0x563bda9ec300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 0x563bdaa06720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546e7ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f54641acb90 con 0x563bda9eb5d0 2024-09-15T23:28:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546e7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f54641ae6d0 con 0x563bda9ecb40 2024-09-15T23:28:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546e7ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f54641ad260 con 0x563bda9fa780 2024-09-15T23:28:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546a191640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bda9f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546a191640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bda9f73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48930/0 (socket says 172.21.15.89:48930) 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546a191640 1 -- 172.21.15.89:0/2119899498 learned_addr learned my addr 172.21.15.89:0/2119899498 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546a992640 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 0x563bdaa06720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f5469990640 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9ecb40 0x563bda9ec300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546a191640 1 -- 172.21.15.89:0/2119899498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 msgr2=0x563bdaa06720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546a191640 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 0x563bdaa06720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546a191640 1 -- 172.21.15.89:0/2119899498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9ecb40 msgr2=0x563bda9ec300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546a191640 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9ecb40 0x563bda9ec300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546a191640 1 -- 172.21.15.89:0/2119899498 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bda9f5970 con 0x563bda9eb5d0 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f546a992640 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 0x563bdaa06720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.327+0000 7f5469990640 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9ecb40 0x563bda9ec300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.328+0000 7f546a191640 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bda9f73f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f546000ec40 tx=0x7f5460004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340e972d8394b459 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.328+0000 7f544ffff640 1 -- 172.21.15.89:0/2119899498 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460016040 con 0x563bda9eb5d0 2024-09-15T23:28:18.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.328+0000 7f544ffff640 1 -- 172.21.15.89:0/2119899498 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5460004620 con 0x563bda9eb5d0 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.328+0000 7f544ffff640 1 -- 172.21.15.89:0/2119899498 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54600116d0 con 0x563bda9eb5d0 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.328+0000 7f546e7ce740 1 -- 172.21.15.89:0/2119899498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 msgr2=0x563bda9f73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.328+0000 7f546e7ce740 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bda9f73f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f546000ec40 tx=0x7f5460004ac0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 -- 172.21.15.89:0/2119899498 shutdown_connections 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 0x563bdaa06720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9ecb40 0x563bda9ec300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 --2- 172.21.15.89:0/2119899498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bda9f73f0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 -- 172.21.15.89:0/2119899498 >> 172.21.15.89:0/2119899498 conn(0x563bda9b4310 msgr2=0x563bda9f6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 -- 172.21.15.89:0/2119899498 shutdown_connections 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 -- 172.21.15.89:0/2119899498 wait complete. 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 -- start start 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 Processor -- start 2024-09-15T23:28:18.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bdaa03150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9ecb40 0x563bda9f4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 0x563bda9f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f54641ae000 con 0x563bda9eb5d0 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f54641ad930 con 0x563bda9ecb40 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f54641aab70 con 0x563bda9fa780 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546e7ce740 10 client.0 _ll_register_callbacks cb 0x7f545c0b9050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546a191640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bdaa03150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546a191640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bdaa03150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48946/0 (socket says 172.21.15.89:48946) 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.329+0000 7f546a191640 1 -- 172.21.15.89:0/3594344888 learned_addr learned my addr 172.21.15.89:0/3594344888 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f546a992640 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 0x563bda9f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f546a191640 1 -- 172.21.15.89:0/3594344888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 msgr2=0x563bda9f4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:18.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f546a191640 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 0x563bda9f4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f546a191640 1 -- 172.21.15.89:0/3594344888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9ecb40 msgr2=0x563bda9f4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f546a191640 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9ecb40 0x563bda9f4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f546a191640 1 -- 172.21.15.89:0/3594344888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bdaa73fe0 con 0x563bda9eb5d0 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f546a191640 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bdaa03150 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f5460029cb0 tx=0x7f54600049b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0b00f108ad7bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f544dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f544dffb640 1 -- 172.21.15.89:0/3594344888 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460016030 con 0x563bda9eb5d0 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f544dffb640 1 -- 172.21.15.89:0/3594344888 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5460029860 con 0x563bda9eb5d0 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f546e7ce740 1 -- 172.21.15.89:0/3594344888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5460004620 con 0x563bda9eb5d0 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f544dffb640 1 -- 172.21.15.89:0/3594344888 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5460029a60 con 0x563bda9eb5d0 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f546e7ce740 10 client.16896 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:18.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f546e7ce740 1 -- 172.21.15.89:0/3594344888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563bdaa75110 con 0x563bda9eb5d0 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f544dffb640 1 -- 172.21.15.89:0/3594344888 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5460004620 con 0x563bda9eb5d0 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.330+0000 7f544dffb640 1 -- 172.21.15.89:0/3594344888 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f546000f040 con 0x563bda9eb5d0 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.331+0000 7f546e7ce740 10 client.16896 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.331+0000 7f546e7ce740 10 client.16896 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.331+0000 7f546e7ce740 20 client.16896 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.331+0000 7f546e7ce740 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x563bdaa75630 0x563bdaa77a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.331+0000 7f546e7ce740 4 client.16896 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.331+0000 7f546e7ce740 1 -- 172.21.15.89:0/3594344888 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563bda958d70 con 0x563bdaa75630 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.331+0000 7f5469990640 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x563bdaa75630 0x563bdaa77a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.331+0000 7f544dffb640 10 client.16896 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.331+0000 7f5469990640 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x563bdaa75630 0x563bdaa77a20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=163fe54320c9e6d7 server_cookie=dd6381856a0e34ab in_seq=0 out_seq=0 2024-09-15T23:28:18.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f544dffb640 1 -- 172.21.15.89:0/3594344888 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563bda958d70 con 0x563bdaa75630 2024-09-15T23:28:18.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f544dffb640 10 client.16896 handle_command_reply: tid=0 2024-09-15T23:28:18.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 client.16896 shutdown 2024-09-15T23:28:18.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 -- 172.21.15.89:0/3594344888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 msgr2=0x563bdaa03150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:18.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bdaa03150 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f5460029cb0 tx=0x7f54600049b0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 -- 172.21.15.89:0/3594344888 shutdown_connections 2024-09-15T23:28:18.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x563bdaa75630 0x563bdaa77a20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563bda9fa780 0x563bda9f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563bda9ecb40 0x563bda9f4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f544dffb640 0 client.16896 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:28:18.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 --2- 172.21.15.89:0/3594344888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563bda9eb5d0 0x563bdaa03150 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 -- 172.21.15.89:0/3594344888 >> 172.21.15.89:0/3594344888 conn(0x563bda9b4310 msgr2=0x563bda9f6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:18.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 -- 172.21.15.89:0/3594344888 shutdown_connections 2024-09-15T23:28:18.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 1 -- 172.21.15.89:0/3594344888 wait complete. 2024-09-15T23:28:18.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546e7ce740 20 client.16896 trim_cache size 0 max 16384 2024-09-15T23:28:18.345 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:28:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546c3f5640 1 -- 172.21.15.89:0/2989033798 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54500a3240 msgr2=0x7f54500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546c3f5640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54500a3240 0x7f54500a5700 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5454005d80 tx=0x7f5454013890 comp rx=0 tx=0).stop 2024-09-15T23:28:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546c3f5640 1 -- 172.21.15.89:0/2989033798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5464106f10 msgr2=0x7f546419cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546c3f5640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5464106f10 0x7f546419cc80 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f545c031a20 tx=0x7f545c031a50 comp rx=0 tx=0).stop 2024-09-15T23:28:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546c3f5640 1 -- 172.21.15.89:0/2989033798 shutdown_connections 2024-09-15T23:28:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546c3f5640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f54500a3240 0x7f54500a5700 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546c3f5640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5464106f10 0x7f546419cc80 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546c3f5640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5464102790 0x7f54641988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546c3f5640 1 --2- 172.21.15.89:0/2989033798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5464101de0 0x7f54641983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:18.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.332+0000 7f546c3f5640 1 -- 172.21.15.89:0/2989033798 >> 172.21.15.89:0/2989033798 conn(0x7f54640fd8c0 msgr2=0x7f54640fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:18.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.333+0000 7f546c3f5640 1 -- 172.21.15.89:0/2989033798 shutdown_connections 2024-09-15T23:28:18.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:18.333+0000 7f546c3f5640 1 -- 172.21.15.89:0/2989033798 wait complete. 2024-09-15T23:28:19.057 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:28:19.305 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:19.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:19 smithi089 ceph-mon[30933]: pgmap v479: 129 pgs: 129 active+clean; 2.1 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:28:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:19 smithi120 ceph-mon[35250]: pgmap v479: 129 pgs: 129 active+clean; 2.1 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:28:19.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:19 smithi174 ceph-mon[35586]: pgmap v479: 129 pgs: 129 active+clean; 2.1 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:28:19.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.788+0000 7fad2756d640 1 -- 172.21.15.89:0/3351023412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad20102790 msgr2=0x7fad201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:19.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.788+0000 7fad2756d640 1 --2- 172.21.15.89:0/3351023412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad20102790 0x7fad201067e0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009900 tx=0x7fad1c02a410 comp rx=0 tx=0).stop 2024-09-15T23:28:19.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.788+0000 7fad2756d640 1 -- 172.21.15.89:0/3351023412 shutdown_connections 2024-09-15T23:28:19.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.788+0000 7fad2756d640 1 --2- 172.21.15.89:0/3351023412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad20106f10 0x7fad2010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.788+0000 7fad2756d640 1 --2- 172.21.15.89:0/3351023412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad20102790 0x7fad201067e0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.788+0000 7fad2756d640 1 --2- 172.21.15.89:0/3351023412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad20101de0 0x7fad201021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.788+0000 7fad2756d640 1 -- 172.21.15.89:0/3351023412 >> 172.21.15.89:0/3351023412 conn(0x7fad200fd8c0 msgr2=0x7fad200ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.788+0000 7fad2756d640 1 -- 172.21.15.89:0/3351023412 shutdown_connections 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.788+0000 7fad2756d640 1 -- 172.21.15.89:0/3351023412 wait complete. 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad2756d640 1 Processor -- start 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad2756d640 1 -- start start 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad2756d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad20101de0 0x7fad20198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad2756d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad20102790 0x7fad201989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad2756d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad20106f10 0x7fad2019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad2756d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fad20111f70 con 0x7fad20102790 2024-09-15T23:28:19.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad2756d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fad20111df0 con 0x7fad20101de0 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad2756d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fad201120f0 con 0x7fad20106f10 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad24b08640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad20102790 0x7fad201989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad24b08640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad20102790 0x7fad201989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48960/0 (socket says 172.21.15.89:48960) 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad24b08640 1 -- 172.21.15.89:0/2177517299 learned_addr learned my addr 172.21.15.89:0/2177517299 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad25309640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad20101de0 0x7fad20198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad25b0a640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad20106f10 0x7fad2019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad24b08640 1 -- 172.21.15.89:0/2177517299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad20106f10 msgr2=0x7fad2019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad24b08640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad20106f10 0x7fad2019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad24b08640 1 -- 172.21.15.89:0/2177517299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad20101de0 msgr2=0x7fad20198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad24b08640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad20101de0 0x7fad20198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.789+0000 7fad24b08640 1 -- 172.21.15.89:0/2177517299 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2019d440 con 0x7fad20102790 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.790+0000 7fad24b08640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad20102790 0x7fad201989d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fad1c005ab0 tx=0x7fad1c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:19.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.790+0000 7fad127fc640 1 -- 172.21.15.89:0/2177517299 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c03f030 con 0x7fad20102790 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.790+0000 7fad127fc640 1 -- 172.21.15.89:0/2177517299 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c003ce0 con 0x7fad20102790 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.790+0000 7fad127fc640 1 -- 172.21.15.89:0/2177517299 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c035070 con 0x7fad20102790 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.790+0000 7fad2756d640 1 -- 172.21.15.89:0/2177517299 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2019d6d0 con 0x7fad20102790 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.790+0000 7fad2756d640 1 -- 172.21.15.89:0/2177517299 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad201a5010 con 0x7fad20102790 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.792+0000 7fad127fc640 1 -- 172.21.15.89:0/2177517299 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fad1c042030 con 0x7fad20102790 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad29946740 1 Processor -- start 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad29946740 1 -- start start 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad29946740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 0x559a8758f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad29946740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad29946740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a8759c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad29946740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559a874f52f0 con 0x559a8758f5d0 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad29946740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559a8749ccf0 con 0x559a8759a370 2024-09-15T23:28:19.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad29946740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559a874f12b0 con 0x559a87590b40 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad25309640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad25309640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48972/0 (socket says 172.21.15.89:48972) 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad25309640 1 -- 172.21.15.89:0/2403559432 learned_addr learned my addr 172.21.15.89:0/2403559432 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad127fc640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fad0c0a3240 0x7fad0c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad24b08640 1 --2- 172.21.15.89:0/2403559432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a8759c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad25b0a640 1 --2- 172.21.15.89:0/2403559432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 0x559a8758f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad127fc640 1 -- 172.21.15.89:0/2177517299 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fad1c0f05f0 con 0x7fad20102790 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad25b0a640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fad0c0a3240 0x7fad0c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad25309640 1 -- 172.21.15.89:0/2403559432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 msgr2=0x559a8758f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad25309640 1 --2- 172.21.15.89:0/2403559432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 0x559a8758f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad25309640 1 -- 172.21.15.89:0/2403559432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 msgr2=0x559a8759c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad25309640 1 --2- 172.21.15.89:0/2403559432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a8759c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.793+0000 7fad25309640 1 -- 172.21.15.89:0/2403559432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a874813a0 con 0x559a8758f5d0 2024-09-15T23:28:19.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.794+0000 7fad25309640 1 --2- 172.21.15.89:0/2403559432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758fa10 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fad1400ac20 tx=0x7fad1400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d0428d6a446e1d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.794+0000 7fad0b7fe640 1 -- 172.21.15.89:0/2403559432 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1401d070 con 0x559a8758f5d0 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.794+0000 7fad25b0a640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fad0c0a3240 0x7fad0c0a5700 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fad18013460 tx=0x7fad18005c20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.794+0000 7fad0b7fe640 1 -- 172.21.15.89:0/2403559432 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad14018b20 con 0x559a8758f5d0 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.794+0000 7fad0b7fe640 1 -- 172.21.15.89:0/2403559432 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14010c00 con 0x559a8758f5d0 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.794+0000 7fad29946740 1 -- 172.21.15.89:0/2403559432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 msgr2=0x559a8758fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.794+0000 7fad29946740 1 --2- 172.21.15.89:0/2403559432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758fa10 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fad1400ac20 tx=0x7fad1400f790 comp rx=0 tx=0).stop 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 -- 172.21.15.89:0/2403559432 shutdown_connections 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 --2- 172.21.15.89:0/2403559432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a8759c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 --2- 172.21.15.89:0/2403559432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758fa10 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 --2- 172.21.15.89:0/2403559432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 0x559a8758f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 -- 172.21.15.89:0/2403559432 >> 172.21.15.89:0/2403559432 conn(0x559a87558310 msgr2=0x559a8758bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 -- 172.21.15.89:0/2403559432 shutdown_connections 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 -- 172.21.15.89:0/2403559432 wait complete. 2024-09-15T23:28:19.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 -- start start 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 Processor -- start 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8758f5d0 0x559a87599d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a87590b40 0x559a87596ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a875975e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559a8759d070 con 0x559a87590b40 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559a874b0060 con 0x559a8759a370 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559a8759d1f0 con 0x559a8758f5d0 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.795+0000 7fad29946740 10 client.0 _ll_register_callbacks cb 0x559a872038c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad24b08640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a87590b40 0x559a87596ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad24b08640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a87590b40 0x559a87596ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48976/0 (socket says 172.21.15.89:48976) 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad24b08640 1 -- 172.21.15.89:0/1497206754 learned_addr learned my addr 172.21.15.89:0/1497206754 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad25309640 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8758f5d0 0x559a87599d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad25b0a640 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a875975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad24b08640 1 -- 172.21.15.89:0/1497206754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8758f5d0 msgr2=0x559a87599d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad24b08640 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8758f5d0 0x559a87599d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad24b08640 1 -- 172.21.15.89:0/1497206754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 msgr2=0x559a875975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad24b08640 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a875975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad24b08640 1 -- 172.21.15.89:0/1497206754 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a872663c0 con 0x559a87590b40 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad25309640 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8758f5d0 0x559a87599d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad25b0a640 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a875975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad24b08640 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a87590b40 0x559a87596ff0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fad1c005940 tx=0x7fad1c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfca2bc824dbb780 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad08ff9640 1 -- 172.21.15.89:0/1497206754 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c049070 con 0x559a87590b40 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad08ff9640 1 -- 172.21.15.89:0/1497206754 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fad1c0b8e90 con 0x559a87590b40 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad29946740 1 -- 172.21.15.89:0/1497206754 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad14018b20 con 0x559a87590b40 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad08ff9640 1 -- 172.21.15.89:0/1497206754 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fad1c0bc820 con 0x559a87590b40 2024-09-15T23:28:19.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad29946740 10 client.16908 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad29946740 1 -- 172.21.15.89:0/1497206754 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559a875a8ae0 con 0x559a87590b40 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.796+0000 7fad08ff9640 1 -- 172.21.15.89:0/1497206754 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad14018b20 con 0x559a87590b40 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.797+0000 7fad08ff9640 1 -- 172.21.15.89:0/1497206754 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fad1c03b460 con 0x559a87590b40 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.797+0000 7fad29946740 10 client.16908 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.797+0000 7fad29946740 10 client.16908 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.797+0000 7fad29946740 20 client.16908 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.797+0000 7fad29946740 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559a875a9ef0 0x559a875ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.797+0000 7fad29946740 4 client.16908 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.797+0000 7fad29946740 1 -- 172.21.15.89:0/1497206754 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a87502860 con 0x559a875a9ef0 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.797+0000 7fad25309640 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559a875a9ef0 0x559a875ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.797+0000 7fad08ff9640 10 client.16908 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.797+0000 7fad25309640 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559a875a9ef0 0x559a875ac2e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=936735dbec678856 server_cookie=3b01cbf279e2cb6c in_seq=0 out_seq=0 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad08ff9640 1 -- 172.21.15.89:0/1497206754 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559a87502860 con 0x559a875a9ef0 2024-09-15T23:28:19.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad08ff9640 10 client.16908 handle_command_reply: tid=0 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad29946740 1 client.16908 shutdown 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad29946740 1 -- 172.21.15.89:0/1497206754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a87590b40 msgr2=0x559a87596ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad29946740 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a87590b40 0x559a87596ff0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fad1c005940 tx=0x7fad1c02a410 comp rx=0 tx=0).stop 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad29946740 1 -- 172.21.15.89:0/1497206754 shutdown_connections 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad29946740 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a875975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad29946740 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a87590b40 0x559a87596ff0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad29946740 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559a875a9ef0 0x559a875ac2e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad29946740 1 --2- 172.21.15.89:0/1497206754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8758f5d0 0x559a87599d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad29946740 1 -- 172.21.15.89:0/1497206754 >> 172.21.15.89:0/1497206754 conn(0x559a87558310 msgr2=0x559a87591960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad08ff9640 0 client.16908 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.798+0000 7fad29946740 1 -- 172.21.15.89:0/1497206754 shutdown_connections 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.799+0000 7fad29946740 1 -- 172.21.15.89:0/1497206754 wait complete. 2024-09-15T23:28:19.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.799+0000 7fad29946740 20 client.16908 trim_cache size 0 max 16384 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.811+0000 7fad29946740 1 Processor -- start 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.811+0000 7fad29946740 1 -- start start 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.811+0000 7fad29946740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.811+0000 7fad29946740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 0x559a875a6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.811+0000 7fad29946740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a875968b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.811+0000 7fad29946740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fad201aee50 con 0x559a8758f5d0 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad29946740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fad201b0990 con 0x559a8759a370 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad29946740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fad201af520 con 0x559a87590b40 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad25309640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad25309640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48986/0 (socket says 172.21.15.89:48986) 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad25309640 1 -- 172.21.15.89:0/510111581 learned_addr learned my addr 172.21.15.89:0/510111581 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad24b08640 1 --2- 172.21.15.89:0/510111581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 0x559a875a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad25309640 1 -- 172.21.15.89:0/510111581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 msgr2=0x559a875a6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad25309640 1 --2- 172.21.15.89:0/510111581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 0x559a875a6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad25309640 1 -- 172.21.15.89:0/510111581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 msgr2=0x559a875968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad25309640 1 --2- 172.21.15.89:0/510111581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a875968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad25309640 1 -- 172.21.15.89:0/510111581 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a87595fc0 con 0x559a8758f5d0 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad24b08640 1 --2- 172.21.15.89:0/510111581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 0x559a875a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad25309640 1 --2- 172.21.15.89:0/510111581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758be10 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fad1400ac20 tx=0x7fad1400a470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4ae995993266bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad0affd640 1 -- 172.21.15.89:0/510111581 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14032070 con 0x559a8758f5d0 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad0affd640 1 -- 172.21.15.89:0/510111581 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1401e040 con 0x559a8758f5d0 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.812+0000 7fad0affd640 1 -- 172.21.15.89:0/510111581 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1401d040 con 0x559a8758f5d0 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 -- 172.21.15.89:0/510111581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 msgr2=0x559a8758be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 --2- 172.21.15.89:0/510111581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758be10 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fad1400ac20 tx=0x7fad1400a470 comp rx=0 tx=0).stop 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 -- 172.21.15.89:0/510111581 shutdown_connections 2024-09-15T23:28:19.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 --2- 172.21.15.89:0/510111581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a8759a370 0x559a875968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 --2- 172.21.15.89:0/510111581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a87590b40 0x559a875a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 --2- 172.21.15.89:0/510111581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a8758be10 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 -- 172.21.15.89:0/510111581 >> 172.21.15.89:0/510111581 conn(0x559a87558310 msgr2=0x559a8758b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 -- 172.21.15.89:0/510111581 shutdown_connections 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 -- 172.21.15.89:0/510111581 wait complete. 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 -- start start 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.813+0000 7fad29946740 1 Processor -- start 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad29946740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a87594160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad29946740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a87590b40 0x559a875946a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad29946740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8759a370 0x559a87594be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad29946740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fad201b02c0 con 0x559a8758f5d0 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad29946740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fad201afbf0 con 0x559a87590b40 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad29946740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fad201ace30 con 0x559a8759a370 2024-09-15T23:28:19.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad29946740 10 client.0 _ll_register_callbacks cb 0x7fad1c0044e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad25309640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a87594160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad25309640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a87594160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48990/0 (socket says 172.21.15.89:48990) 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad25309640 1 -- 172.21.15.89:0/392652271 learned_addr learned my addr 172.21.15.89:0/392652271 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad25b0a640 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8759a370 0x559a87594be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad24b08640 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a87590b40 0x559a875946a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad25309640 1 -- 172.21.15.89:0/392652271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8759a370 msgr2=0x559a87594be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad25309640 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8759a370 0x559a87594be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad25309640 1 -- 172.21.15.89:0/392652271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a87590b40 msgr2=0x559a875946a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad25309640 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a87590b40 0x559a875946a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad25309640 1 -- 172.21.15.89:0/392652271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a87613cf0 con 0x559a8758f5d0 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad25309640 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a87594160 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fad1400a6f0 tx=0x7fad1400a720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec0cea0cd79e7cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad08ff9640 1 -- 172.21.15.89:0/392652271 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14032070 con 0x559a8758f5d0 2024-09-15T23:28:19.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad08ff9640 1 -- 172.21.15.89:0/392652271 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fad14010d20 con 0x559a8758f5d0 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.814+0000 7fad29946740 1 -- 172.21.15.89:0/392652271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad1401e040 con 0x559a8758f5d0 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad08ff9640 1 -- 172.21.15.89:0/392652271 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fad1401d040 con 0x559a8758f5d0 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad29946740 10 client.16914 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad29946740 1 -- 172.21.15.89:0/392652271 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559a87614fd0 con 0x559a8758f5d0 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad08ff9640 1 -- 172.21.15.89:0/392652271 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1401e040 con 0x559a8758f5d0 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad08ff9640 1 -- 172.21.15.89:0/392652271 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fad14018820 con 0x559a8758f5d0 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad29946740 10 client.16914 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad29946740 10 client.16914 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad29946740 20 client.16914 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad29946740 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559a876154d0 0x559a876178f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad29946740 4 client.16914 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad29946740 1 -- 172.21.15.89:0/392652271 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559a874fcd70 con 0x559a876154d0 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.815+0000 7fad24b08640 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559a876154d0 0x559a876178f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:19.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad24b08640 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559a876154d0 0x559a876178f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=50cc7547fb1f184a server_cookie=c134009875594a9a in_seq=0 out_seq=0 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad08ff9640 10 client.16914 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad08ff9640 1 -- 172.21.15.89:0/392652271 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559a874fcd70 con 0x559a876154d0 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad08ff9640 10 client.16914 handle_command_reply: tid=0 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 client.16914 shutdown 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 -- 172.21.15.89:0/392652271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 msgr2=0x559a87594160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a87594160 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fad1400a6f0 tx=0x7fad1400a720 comp rx=0 tx=0).stop 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 -- 172.21.15.89:0/392652271 shutdown_connections 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559a876154d0 0x559a876178f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559a8759a370 0x559a87594be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559a87590b40 0x559a875946a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad08ff9640 0 client.16914 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 --2- 172.21.15.89:0/392652271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a8758f5d0 0x559a87594160 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 -- 172.21.15.89:0/392652271 >> 172.21.15.89:0/392652271 conn(0x559a87558310 msgr2=0x559a87599070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:19.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 -- 172.21.15.89:0/392652271 shutdown_connections 2024-09-15T23:28:19.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 1 -- 172.21.15.89:0/392652271 wait complete. 2024-09-15T23:28:19.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.816+0000 7fad29946740 20 client.16914 trim_cache size 0 max 16384 2024-09-15T23:28:19.834 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 -- 172.21.15.89:0/2177517299 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fad0c0a3240 msgr2=0x7fad0c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fad0c0a3240 0x7fad0c0a5700 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fad18013460 tx=0x7fad18005c20 comp rx=0 tx=0).stop 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 -- 172.21.15.89:0/2177517299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad20102790 msgr2=0x7fad201989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad20102790 0x7fad201989d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fad1c005ab0 tx=0x7fad1c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 -- 172.21.15.89:0/2177517299 shutdown_connections 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fad0c0a3240 0x7fad0c0a5700 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad20106f10 0x7fad2019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fad20102790 0x7fad201989d0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 --2- 172.21.15.89:0/2177517299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad20101de0 0x7fad20198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 -- 172.21.15.89:0/2177517299 >> 172.21.15.89:0/2177517299 conn(0x7fad200fd8c0 msgr2=0x7fad200fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:19.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 -- 172.21.15.89:0/2177517299 shutdown_connections 2024-09-15T23:28:19.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:19.817+0000 7fad2756d640 1 -- 172.21.15.89:0/2177517299 wait complete. 2024-09-15T23:28:20.625 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:28:20.870 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:21.276 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:21 smithi089 ceph-mon[30933]: pgmap v480: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:28:21.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.356+0000 7f5151651640 1 -- 172.21.15.89:0/3653369064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c106fc0 msgr2=0x7f514c10eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.356+0000 7f5151651640 1 --2- 172.21.15.89:0/3653369064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c106fc0 0x7f514c10eab0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5144009cd0 tx=0x7f514402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.356+0000 7f5151651640 1 -- 172.21.15.89:0/3653369064 shutdown_connections 2024-09-15T23:28:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.356+0000 7f5151651640 1 --2- 172.21.15.89:0/3653369064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c106fc0 0x7f514c10eab0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.356+0000 7f5151651640 1 --2- 172.21.15.89:0/3653369064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f514c1007f0 0x7f514c106860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.356+0000 7f5151651640 1 --2- 172.21.15.89:0/3653369064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f514c0ffe40 0x7f514c100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.356+0000 7f5151651640 1 -- 172.21.15.89:0/3653369064 >> 172.21.15.89:0/3653369064 conn(0x7f514c0fba90 msgr2=0x7f514c0fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.356+0000 7f5151651640 1 -- 172.21.15.89:0/3653369064 shutdown_connections 2024-09-15T23:28:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.356+0000 7f5151651640 1 -- 172.21.15.89:0/3653369064 wait complete. 2024-09-15T23:28:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.357+0000 7f5151651640 1 Processor -- start 2024-09-15T23:28:21.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.357+0000 7f5151651640 1 -- start start 2024-09-15T23:28:21.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.357+0000 7f5151651640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f514c0ffe40 0x7f514c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.357+0000 7f5151651640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c1007f0 0x7f514c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.357+0000 7f5151651640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f514c106fc0 0x7f514c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.357+0000 7f5151651640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f514c111fc0 con 0x7f514c1007f0 2024-09-15T23:28:21.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f5151651640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f514c111e40 con 0x7f514c0ffe40 2024-09-15T23:28:21.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f5151651640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f514c112140 con 0x7f514c106fc0 2024-09-15T23:28:21.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f51427fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c1007f0 0x7f514c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f51427fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c1007f0 0x7f514c1989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49002/0 (socket says 172.21.15.89:49002) 2024-09-15T23:28:21.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f51427fc640 1 -- 172.21.15.89:0/3556757744 learned_addr learned my addr 172.21.15.89:0/3556757744 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f514b7fe640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f514c106fc0 0x7f514c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f514affd640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f514c0ffe40 0x7f514c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f51427fc640 1 -- 172.21.15.89:0/3556757744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f514c106fc0 msgr2=0x7f514c19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f51427fc640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f514c106fc0 0x7f514c19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f51427fc640 1 -- 172.21.15.89:0/3556757744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f514c0ffe40 msgr2=0x7f514c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f51427fc640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f514c0ffe40 0x7f514c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f51427fc640 1 -- 172.21.15.89:0/3556757744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514c19d420 con 0x7f514c1007f0 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f514affd640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f514c0ffe40 0x7f514c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f51427fc640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c1007f0 0x7f514c1989b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f513400c970 tx=0x7f513400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f5148ff9640 1 -- 172.21.15.89:0/3556757744 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134007bb0 con 0x7f514c1007f0 2024-09-15T23:28:21.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f5148ff9640 1 -- 172.21.15.89:0/3556757744 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5134007d50 con 0x7f514c1007f0 2024-09-15T23:28:21.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f5151651640 1 -- 172.21.15.89:0/3556757744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514c19d710 con 0x7f514c1007f0 2024-09-15T23:28:21.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f5148ff9640 1 -- 172.21.15.89:0/3556757744 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513400f450 con 0x7f514c1007f0 2024-09-15T23:28:21.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.358+0000 7f5151651640 1 -- 172.21.15.89:0/3556757744 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f514c1a4ff0 con 0x7f514c1007f0 2024-09-15T23:28:21.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f5148ff9640 1 -- 172.21.15.89:0/3556757744 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5134004f40 con 0x7f514c1007f0 2024-09-15T23:28:21.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f5153a2a740 1 Processor -- start 2024-09-15T23:28:21.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f5153a2a740 1 -- start start 2024-09-15T23:28:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1d8b40 0x559fca1d7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 0x559fca1d7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1e4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f5153a2a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559fca13d2f0 con 0x559fca1e2370 2024-09-15T23:28:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f5153a2a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559fca0e4cf0 con 0x559fca1d75d0 2024-09-15T23:28:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f5153a2a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559fca1392b0 con 0x559fca1d8b40 2024-09-15T23:28:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f51427fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1e4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f51427fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1e4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49018/0 (socket says 172.21.15.89:49018) 2024-09-15T23:28:21.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f51427fc640 1 -- 172.21.15.89:0/1884717696 learned_addr learned my addr 172.21.15.89:0/1884717696 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.361+0000 7f514b7fe640 1 --2- 172.21.15.89:0/1884717696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1d8b40 0x559fca1d7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f514affd640 1 --2- 172.21.15.89:0/1884717696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 0x559fca1d7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f5148ff9640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51380a3240 0x7f51380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f51427fc640 1 -- 172.21.15.89:0/1884717696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1d8b40 msgr2=0x559fca1d7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f51427fc640 1 --2- 172.21.15.89:0/1884717696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1d8b40 0x559fca1d7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f51427fc640 1 -- 172.21.15.89:0/1884717696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 msgr2=0x559fca1d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f51427fc640 1 --2- 172.21.15.89:0/1884717696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 0x559fca1d7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f51427fc640 1 -- 172.21.15.89:0/1884717696 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fca0c93a0 con 0x559fca1e2370 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f514affd640 1 --2- 172.21.15.89:0/1884717696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 0x559fca1d7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f514b7fe640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51380a3240 0x7f51380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f5148ff9640 1 -- 172.21.15.89:0/3556757744 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f51340cfa40 con 0x7f514c1007f0 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f51427fc640 1 --2- 172.21.15.89:0/1884717696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1e4760 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f51340b67a0 tx=0x7f51340afdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f52ccccacbde51b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f51417fa640 1 -- 172.21.15.89:0/1884717696 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513401d030 con 0x559fca1e2370 2024-09-15T23:28:21.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f51417fa640 1 -- 172.21.15.89:0/1884717696 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5134005600 con 0x559fca1e2370 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.362+0000 7f51417fa640 1 -- 172.21.15.89:0/1884717696 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513409b5d0 con 0x559fca1e2370 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f5153a2a740 1 -- 172.21.15.89:0/1884717696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 msgr2=0x559fca1e4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f514b7fe640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51380a3240 0x7f51380a5700 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f51440098d0 tx=0x7f51440095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f5153a2a740 1 --2- 172.21.15.89:0/1884717696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1e4760 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f51340b67a0 tx=0x7f51340afdd0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f5153a2a740 1 -- 172.21.15.89:0/1884717696 shutdown_connections 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f5153a2a740 1 --2- 172.21.15.89:0/1884717696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1e4760 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f5153a2a740 1 --2- 172.21.15.89:0/1884717696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 0x559fca1d7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f5153a2a740 1 --2- 172.21.15.89:0/1884717696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1d8b40 0x559fca1d7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f5153a2a740 1 -- 172.21.15.89:0/1884717696 >> 172.21.15.89:0/1884717696 conn(0x559fca1a0310 msgr2=0x559fca1d3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f5153a2a740 1 -- 172.21.15.89:0/1884717696 shutdown_connections 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f5153a2a740 1 -- 172.21.15.89:0/1884717696 wait complete. 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.363+0000 7f5153a2a740 1 -- start start 2024-09-15T23:28:21.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f5153a2a740 1 Processor -- start 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1d75d0 0x559fca1e1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 0x559fca1df060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1df7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f5153a2a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559fca1e5070 con 0x559fca1e2370 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f5153a2a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559fca0f8060 con 0x559fca1d8b40 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f5153a2a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559fca1e51f0 con 0x559fca1d75d0 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f5153a2a740 10 client.0 _ll_register_callbacks cb 0x559fc9e4b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f514b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1df7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f514b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1df7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49028/0 (socket says 172.21.15.89:49028) 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f514b7fe640 1 -- 172.21.15.89:0/905271827 learned_addr learned my addr 172.21.15.89:0/905271827 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f51427fc640 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 0x559fca1df060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f514affd640 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1d75d0 0x559fca1e1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f514b7fe640 1 -- 172.21.15.89:0/905271827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1d75d0 msgr2=0x559fca1e1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f514b7fe640 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1d75d0 0x559fca1e1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f514b7fe640 1 -- 172.21.15.89:0/905271827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 msgr2=0x559fca1df060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f514b7fe640 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 0x559fca1df060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f514b7fe640 1 -- 172.21.15.89:0/905271827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fc9eae3c0 con 0x559fca1e2370 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f51427fc640 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 0x559fca1df060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f514b7fe640 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1df7f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5144035000 tx=0x7f514402a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabcbbd42eaf27ba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.364+0000 7f51327fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f51327fc640 1 -- 172.21.15.89:0/905271827 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51440305a0 con 0x559fca1e2370 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f51327fc640 1 -- 172.21.15.89:0/905271827 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5144030740 con 0x559fca1e2370 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f5153a2a740 1 -- 172.21.15.89:0/905271827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559fca14a860 con 0x559fca1e2370 2024-09-15T23:28:21.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f51327fc640 1 -- 172.21.15.89:0/905271827 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5144030940 con 0x559fca1e2370 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f5153a2a740 10 client.16926 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f5153a2a740 1 -- 172.21.15.89:0/905271827 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559fca1f0b60 con 0x559fca1e2370 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f51327fc640 1 -- 172.21.15.89:0/905271827 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559fca14a860 con 0x559fca1e2370 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f51327fc640 1 -- 172.21.15.89:0/905271827 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5144030070 con 0x559fca1e2370 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f5153a2a740 10 client.16926 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f5153a2a740 10 client.16926 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f5153a2a740 20 client.16926 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f5153a2a740 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559fca1f2020 0x559fca1f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f5153a2a740 4 client.16926 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.365+0000 7f5153a2a740 1 -- 172.21.15.89:0/905271827 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559fca144d70 con 0x559fca1f2020 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.366+0000 7f514affd640 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559fca1f2020 0x559fca1f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.366+0000 7f51327fc640 10 client.16926 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.366+0000 7f514affd640 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559fca1f2020 0x559fca1f44a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=302bad4b464e6f09 server_cookie=191adeab81b67b7a in_seq=0 out_seq=0 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f51327fc640 1 -- 172.21.15.89:0/905271827 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559fca144d70 con 0x559fca1f2020 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f51327fc640 10 client.16926 handle_command_reply: tid=0 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 client.16926 shutdown 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 -- 172.21.15.89:0/905271827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 msgr2=0x559fca1df7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1df7f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5144035000 tx=0x7f514402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 -- 172.21.15.89:0/905271827 shutdown_connections 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559fca1f2020 0x559fca1f44a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1e2370 0x559fca1df7f0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 0x559fca1df060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 --2- 172.21.15.89:0/905271827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1d75d0 0x559fca1e1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 -- 172.21.15.89:0/905271827 >> 172.21.15.89:0/905271827 conn(0x559fca1a0310 msgr2=0x559fca1d9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f51327fc640 0 client.16926 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:28:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 -- 172.21.15.89:0/905271827 shutdown_connections 2024-09-15T23:28:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 1 -- 172.21.15.89:0/905271827 wait complete. 2024-09-15T23:28:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.367+0000 7f5153a2a740 20 client.16926 trim_cache size 0 max 16384 2024-09-15T23:28:21.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f5153a2a740 1 Processor -- start 2024-09-15T23:28:21.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f5153a2a740 1 -- start start 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 0x559fca1da260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d8b40 0x559fca1da7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1e2370 0x559fca1d3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f5153a2a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f514c1aefb0 con 0x559fca1d8b40 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f5153a2a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f514c1b0af0 con 0x559fca1d75d0 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f5153a2a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f514c1af680 con 0x559fca1e2370 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f51427fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d8b40 0x559fca1da7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f51427fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d8b40 0x559fca1da7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49040/0 (socket says 172.21.15.89:49040) 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f51427fc640 1 -- 172.21.15.89:0/935827416 learned_addr learned my addr 172.21.15.89:0/935827416 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f514b7fe640 1 --2- 172.21.15.89:0/935827416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1e2370 0x559fca1d3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f514affd640 1 --2- 172.21.15.89:0/935827416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 0x559fca1da260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f51427fc640 1 -- 172.21.15.89:0/935827416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1e2370 msgr2=0x559fca1d3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f51427fc640 1 --2- 172.21.15.89:0/935827416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1e2370 0x559fca1d3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f51427fc640 1 -- 172.21.15.89:0/935827416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 msgr2=0x559fca1da260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f51427fc640 1 --2- 172.21.15.89:0/935827416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 0x559fca1da260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f51427fc640 1 -- 172.21.15.89:0/935827416 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fca1d4960 con 0x559fca1d8b40 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.380+0000 7f514b7fe640 1 --2- 172.21.15.89:0/935827416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1e2370 0x559fca1d3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.381+0000 7f51427fc640 1 --2- 172.21.15.89:0/935827416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d8b40 0x559fca1da7a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5134007590 tx=0x7f5134007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d6a364a5ab1462 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.381+0000 7f5140ff9640 1 -- 172.21.15.89:0/935827416 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513401d070 con 0x559fca1d8b40 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.381+0000 7f5140ff9640 1 -- 172.21.15.89:0/935827416 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513409c370 con 0x559fca1d8b40 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.381+0000 7f5140ff9640 1 -- 172.21.15.89:0/935827416 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51340a0020 con 0x559fca1d8b40 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.381+0000 7f5153a2a740 1 -- 172.21.15.89:0/935827416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d8b40 msgr2=0x559fca1da7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.381+0000 7f5153a2a740 1 --2- 172.21.15.89:0/935827416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d8b40 0x559fca1da7a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5134007590 tx=0x7f5134007f00 comp rx=0 tx=0).stop 2024-09-15T23:28:21.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 -- 172.21.15.89:0/935827416 shutdown_connections 2024-09-15T23:28:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 --2- 172.21.15.89:0/935827416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1e2370 0x559fca1d3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 --2- 172.21.15.89:0/935827416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d8b40 0x559fca1da7a0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 --2- 172.21.15.89:0/935827416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d75d0 0x559fca1da260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 -- 172.21.15.89:0/935827416 >> 172.21.15.89:0/935827416 conn(0x559fca1a0310 msgr2=0x559fca1e4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 -- 172.21.15.89:0/935827416 shutdown_connections 2024-09-15T23:28:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 -- 172.21.15.89:0/935827416 wait complete. 2024-09-15T23:28:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 -- start start 2024-09-15T23:28:21.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 Processor -- start 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d75d0 0x559fca1eaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 0x559fca1e36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1e2370 0x559fca1e3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f514c1b0420 con 0x559fca1d75d0 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f514c1afd50 con 0x559fca1d8b40 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f514c1acf90 con 0x559fca1e2370 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f5153a2a740 10 client.0 _ll_register_callbacks cb 0x7f5144031550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f514affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d75d0 0x559fca1eaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f514affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d75d0 0x559fca1eaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49042/0 (socket says 172.21.15.89:49042) 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f514affd640 1 -- 172.21.15.89:0/4120611868 learned_addr learned my addr 172.21.15.89:0/4120611868 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.382+0000 7f51427fc640 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 0x559fca1e36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f514affd640 1 -- 172.21.15.89:0/4120611868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1e2370 msgr2=0x559fca1e3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f514affd640 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1e2370 0x559fca1e3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f514affd640 1 -- 172.21.15.89:0/4120611868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 msgr2=0x559fca1e36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f514affd640 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 0x559fca1e36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f514affd640 1 -- 172.21.15.89:0/4120611868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fca1e4440 con 0x559fca1d75d0 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f51427fc640 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 0x559fca1e36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f514affd640 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d75d0 0x559fca1eaad0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f513c000c00 tx=0x7f513c017810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09ab0ded53ea0c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f51327fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f51327fc640 1 -- 172.21.15.89:0/4120611868 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c0170d0 con 0x559fca1d75d0 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f51327fc640 1 -- 172.21.15.89:0/4120611868 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f513c024420 con 0x559fca1d75d0 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f5153a2a740 1 -- 172.21.15.89:0/4120611868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513409c370 con 0x559fca1d75d0 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f51327fc640 1 -- 172.21.15.89:0/4120611868 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f513c02c050 con 0x559fca1d75d0 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f5153a2a740 10 client.16932 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f5153a2a740 1 -- 172.21.15.89:0/4120611868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559fca2611a0 con 0x559fca1d75d0 2024-09-15T23:28:21.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f51327fc640 1 -- 172.21.15.89:0/4120611868 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513409c370 con 0x559fca1d75d0 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f51327fc640 1 -- 172.21.15.89:0/4120611868 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f513c015520 con 0x559fca1d75d0 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.383+0000 7f5153a2a740 10 client.16932 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 10 client.16932 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 20 client.16932 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559fca261830 0x559fca263c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 4 client.16932 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 -- 172.21.15.89:0/4120611868 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559fca0c1170 con 0x559fca261830 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f51427fc640 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559fca261830 0x559fca263c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f51427fc640 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559fca261830 0x559fca263c50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=afdd3ead0d096697 server_cookie=954e7d1952b69a91 in_seq=0 out_seq=0 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f51327fc640 10 client.16932 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f51327fc640 1 -- 172.21.15.89:0/4120611868 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559fca0c1170 con 0x559fca261830 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f51327fc640 10 client.16932 handle_command_reply: tid=0 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 client.16932 shutdown 2024-09-15T23:28:21.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 -- 172.21.15.89:0/4120611868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d75d0 msgr2=0x559fca1eaad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d75d0 0x559fca1eaad0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f513c000c00 tx=0x7f513c017810 comp rx=0 tx=0).stop 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 -- 172.21.15.89:0/4120611868 shutdown_connections 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559fca261830 0x559fca263c50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559fca1e2370 0x559fca1e3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559fca1d8b40 0x559fca1e36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 --2- 172.21.15.89:0/4120611868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fca1d75d0 0x559fca1eaad0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.384+0000 7f5153a2a740 1 -- 172.21.15.89:0/4120611868 >> 172.21.15.89:0/4120611868 conn(0x559fca1a0310 msgr2=0x559fca1d3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f51327fc640 0 client.16932 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5153a2a740 1 -- 172.21.15.89:0/4120611868 shutdown_connections 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5153a2a740 1 -- 172.21.15.89:0/4120611868 wait complete. 2024-09-15T23:28:21.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5153a2a740 20 client.16932 trim_cache size 0 max 16384 2024-09-15T23:28:21.393 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:28:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 -- 172.21.15.89:0/3556757744 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51380a3240 msgr2=0x7f51380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51380a3240 0x7f51380a5700 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f51440098d0 tx=0x7f51440095c0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 -- 172.21.15.89:0/3556757744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c1007f0 msgr2=0x7f514c1989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c1007f0 0x7f514c1989b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f513400c970 tx=0x7f513400ce30 comp rx=0 tx=0).stop 2024-09-15T23:28:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 -- 172.21.15.89:0/3556757744 shutdown_connections 2024-09-15T23:28:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51380a3240 0x7f51380a5700 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f514c106fc0 0x7f514c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c1007f0 0x7f514c1989b0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 --2- 172.21.15.89:0/3556757744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f514c0ffe40 0x7f514c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 -- 172.21.15.89:0/3556757744 >> 172.21.15.89:0/3556757744 conn(0x7f514c0fba90 msgr2=0x7f514c0fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 -- 172.21.15.89:0/3556757744 shutdown_connections 2024-09-15T23:28:21.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:21.385+0000 7f5151651640 1 -- 172.21.15.89:0/3556757744 wait complete. 2024-09-15T23:28:21.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:21 smithi120 ceph-mon[35250]: pgmap v480: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:28:21.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:21 smithi174 ceph-mon[35586]: pgmap v480: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:28:22.109 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:28:22.354 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:22.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.798+0000 7f32868c2640 1 -- 172.21.15.89:0/264904697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f328010c180 msgr2=0x7f328010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.798+0000 7f32868c2640 1 --2- 172.21.15.89:0/264904697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f328010c180 0x7f328010e600 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f3278009ca0 tx=0x7f327802a950 comp rx=0 tx=0).stop 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.798+0000 7f32868c2640 1 -- 172.21.15.89:0/264904697 shutdown_connections 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.798+0000 7f32868c2640 1 --2- 172.21.15.89:0/264904697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f328010c180 0x7f328010e600 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.798+0000 7f32868c2640 1 --2- 172.21.15.89:0/264904697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801036c0 0x7f328010bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.798+0000 7f32868c2640 1 --2- 172.21.15.89:0/264904697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3280102da0 0x7f3280103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.798+0000 7f32868c2640 1 -- 172.21.15.89:0/264904697 >> 172.21.15.89:0/264904697 conn(0x7f32800fc7a0 msgr2=0x7f32800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.799+0000 7f32868c2640 1 -- 172.21.15.89:0/264904697 shutdown_connections 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.799+0000 7f32868c2640 1 -- 172.21.15.89:0/264904697 wait complete. 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f32868c2640 1 Processor -- start 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f32868c2640 1 -- start start 2024-09-15T23:28:22.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f32868c2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3280102da0 0x7f3280198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f32868c2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801036c0 0x7f32801988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f32868c2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f328010c180 0x7f328019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f32868c2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3280111f60 con 0x7f328010c180 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f32868c2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3280111de0 con 0x7f32801036c0 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f32868c2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f32801120e0 con 0x7f3280102da0 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f3284e5f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f328010c180 0x7f328019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f3284e5f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f328010c180 0x7f328019cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49072/0 (socket says 172.21.15.89:49072) 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f3284e5f640 1 -- 172.21.15.89:0/846174534 learned_addr learned my addr 172.21.15.89:0/846174534 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f327ffff640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3280102da0 0x7f3280198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f3284e5f640 1 -- 172.21.15.89:0/846174534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3280102da0 msgr2=0x7f3280198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f3284e5f640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3280102da0 0x7f3280198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f3284e5f640 1 -- 172.21.15.89:0/846174534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801036c0 msgr2=0x7f32801988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:22.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f3284e5f640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801036c0 0x7f32801988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.800+0000 7f3284e5f640 1 -- 172.21.15.89:0/846174534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328019d310 con 0x7f328010c180 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.801+0000 7f3284e5f640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f328010c180 0x7f328019cc30 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f3278006010 tx=0x7f327802f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.801+0000 7f327dffb640 1 -- 172.21.15.89:0/846174534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278034040 con 0x7f328010c180 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.801+0000 7f327dffb640 1 -- 172.21.15.89:0/846174534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3278003e00 con 0x7f328010c180 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.801+0000 7f32868c2640 1 -- 172.21.15.89:0/846174534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328019d5a0 con 0x7f328010c180 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.801+0000 7f327dffb640 1 -- 172.21.15.89:0/846174534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32780456e0 con 0x7f328010c180 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.801+0000 7f32868c2640 1 -- 172.21.15.89:0/846174534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32801a4ee0 con 0x7f328010c180 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.803+0000 7f327dffb640 1 -- 172.21.15.89:0/846174534 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3278037020 con 0x7f328010c180 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3288c9b740 1 Processor -- start 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f327dffb640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f326c0a32c0 0x7f326c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3288c9b740 1 -- start start 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3277fff640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f326c0a32c0 0x7f326c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f327dffb640 1 -- 172.21.15.89:0/846174534 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f32780f0e90 con 0x7f328010c180 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac007b40 0x5625ac006090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac011370 0x5625ac013760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3288c9b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5625abf6c2f0 con 0x5625ac0065d0 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3288c9b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5625abf13cf0 con 0x5625ac011370 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3288c9b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5625abf682b0 con 0x5625ac007b40 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3277fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3277fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac006a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49074/0 (socket says 172.21.15.89:49074) 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3277fff640 1 -- 172.21.15.89:0/3418088615 learned_addr learned my addr 172.21.15.89:0/3418088615 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3277fff640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f326c0a32c0 0x7f326c0a5780 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f328010d4a0 tx=0x7f326800ad50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f327ffff640 1 --2- 172.21.15.89:0/3418088615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac007b40 0x5625ac006090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.804+0000 7f3284e5f640 1 --2- 172.21.15.89:0/3418088615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac011370 0x5625ac013760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.805+0000 7f3277fff640 1 -- 172.21.15.89:0/3418088615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac007b40 msgr2=0x5625ac006090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.805+0000 7f3277fff640 1 --2- 172.21.15.89:0/3418088615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac007b40 0x5625ac006090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.805+0000 7f3277fff640 1 -- 172.21.15.89:0/3418088615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac011370 msgr2=0x5625ac013760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.805+0000 7f3277fff640 1 --2- 172.21.15.89:0/3418088615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac011370 0x5625ac013760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.805+0000 7f3277fff640 1 -- 172.21.15.89:0/3418088615 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625abef83a0 con 0x5625ac0065d0 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.805+0000 7f3277fff640 1 --2- 172.21.15.89:0/3418088615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac006a10 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f326800cda0 tx=0x7f3268017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8fec0a076409e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.805+0000 7f32767fc640 1 -- 172.21.15.89:0/3418088615 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3268018070 con 0x5625ac0065d0 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.805+0000 7f32767fc640 1 -- 172.21.15.89:0/3418088615 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3268005020 con 0x5625ac0065d0 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.805+0000 7f32767fc640 1 -- 172.21.15.89:0/3418088615 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3268004cd0 con 0x5625ac0065d0 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 -- 172.21.15.89:0/3418088615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 msgr2=0x5625ac006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 --2- 172.21.15.89:0/3418088615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac006a10 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f326800cda0 tx=0x7f3268017000 comp rx=0 tx=0).stop 2024-09-15T23:28:22.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 -- 172.21.15.89:0/3418088615 shutdown_connections 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 --2- 172.21.15.89:0/3418088615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac011370 0x5625ac013760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 --2- 172.21.15.89:0/3418088615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac006a10 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 --2- 172.21.15.89:0/3418088615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac007b40 0x5625ac006090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 -- 172.21.15.89:0/3418088615 >> 172.21.15.89:0/3418088615 conn(0x5625abfcf310 msgr2=0x5625ac002c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 -- 172.21.15.89:0/3418088615 shutdown_connections 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 -- 172.21.15.89:0/3418088615 wait complete. 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 -- start start 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.806+0000 7f3288c9b740 1 Processor -- start 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 0x5625ac010d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac00e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac00e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3288c9b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5625ac014070 con 0x5625ac011370 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3288c9b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5625abf27060 con 0x5625ac007b40 2024-09-15T23:28:22.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3288c9b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5625ac0141f0 con 0x5625ac0065d0 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3288c9b740 10 client.0 _ll_register_callbacks cb 0x5625abc7a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3284e5f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac00e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3284e5f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac00e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49080/0 (socket says 172.21.15.89:49080) 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3284e5f640 1 -- 172.21.15.89:0/1968364926 learned_addr learned my addr 172.21.15.89:0/1968364926 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f327ffff640 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 0x5625ac010d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3277fff640 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac00e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3284e5f640 1 -- 172.21.15.89:0/1968364926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 msgr2=0x5625ac010d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3284e5f640 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 0x5625ac010d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3284e5f640 1 -- 172.21.15.89:0/1968364926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 msgr2=0x5625ac00e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3284e5f640 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac00e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3284e5f640 1 -- 172.21.15.89:0/1968364926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625abcdd3c0 con 0x5625ac011370 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3284e5f640 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac00e5d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f32780bd050 tx=0x7f32780bcae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839230bbd7345b33 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:22.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f325f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f325f7fe640 1 -- 172.21.15.89:0/1968364926 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327802c040 con 0x5625ac011370 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f325f7fe640 1 -- 172.21.15.89:0/1968364926 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f32780c1170 con 0x5625ac011370 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.807+0000 7f3288c9b740 1 -- 172.21.15.89:0/1968364926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3268005020 con 0x5625ac011370 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f325f7fe640 1 -- 172.21.15.89:0/1968364926 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f32780bd7c0 con 0x5625ac011370 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f3288c9b740 10 client.16944 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f3288c9b740 1 -- 172.21.15.89:0/1968364926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625ac01faf0 con 0x5625ac011370 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f325f7fe640 1 -- 172.21.15.89:0/1968364926 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3268005020 con 0x5625ac011370 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f325f7fe640 1 -- 172.21.15.89:0/1968364926 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f327804d030 con 0x5625ac011370 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f3288c9b740 10 client.16944 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f3288c9b740 10 client.16944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f3288c9b740 20 client.16944 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:22.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f3288c9b740 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625ac020fa0 0x5625ac023390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f3288c9b740 4 client.16944 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f3288c9b740 1 -- 172.21.15.89:0/1968364926 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625abf79860 con 0x5625ac020fa0 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.808+0000 7f327ffff640 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625ac020fa0 0x5625ac023390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f327ffff640 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625ac020fa0 0x5625ac023390 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b34a4230d358fa43 server_cookie=48325765a147847e in_seq=0 out_seq=0 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f325f7fe640 10 client.16944 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f325f7fe640 1 -- 172.21.15.89:0/1968364926 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5625abf79860 con 0x5625ac020fa0 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f325f7fe640 10 client.16944 handle_command_reply: tid=0 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f3288c9b740 1 client.16944 shutdown 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f3288c9b740 1 -- 172.21.15.89:0/1968364926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 msgr2=0x5625ac00e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f3288c9b740 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac00e5d0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f32780bd050 tx=0x7f32780bcae0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f3288c9b740 1 -- 172.21.15.89:0/1968364926 shutdown_connections 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f3288c9b740 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625ac020fa0 0x5625ac023390 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f3288c9b740 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac00e5d0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f3288c9b740 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac00e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f325f7fe640 0 client.16944 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:22.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f3288c9b740 1 --2- 172.21.15.89:0/1968364926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 0x5625ac010d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.809+0000 7f3288c9b740 1 -- 172.21.15.89:0/1968364926 >> 172.21.15.89:0/1968364926 conn(0x5625abfcf310 msgr2=0x5625ac008960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:22.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.810+0000 7f3288c9b740 1 -- 172.21.15.89:0/1968364926 shutdown_connections 2024-09-15T23:28:22.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.810+0000 7f3288c9b740 1 -- 172.21.15.89:0/1968364926 wait complete. 2024-09-15T23:28:22.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.810+0000 7f3288c9b740 20 client.16944 trim_cache size 0 max 16384 2024-09-15T23:28:22.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.822+0000 7f3288c9b740 1 Processor -- start 2024-09-15T23:28:22.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.822+0000 7f3288c9b740 1 -- start start 2024-09-15T23:28:22.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac00ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac009120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac011370 0x5625ac009660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3288c9b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f32801aed00 con 0x5625ac0065d0 2024-09-15T23:28:22.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3288c9b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f32801b0840 con 0x5625ac007b40 2024-09-15T23:28:22.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3288c9b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f32801af3d0 con 0x5625ac011370 2024-09-15T23:28:22.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f327ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac00ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f327ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac00ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49096/0 (socket says 172.21.15.89:49096) 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f327ffff640 1 -- 172.21.15.89:0/1136358646 learned_addr learned my addr 172.21.15.89:0/1136358646 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3284e5f640 1 --2- 172.21.15.89:0/1136358646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac011370 0x5625ac009660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f327ffff640 1 -- 172.21.15.89:0/1136358646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac011370 msgr2=0x5625ac009660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f327ffff640 1 --2- 172.21.15.89:0/1136358646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac011370 0x5625ac009660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f327ffff640 1 -- 172.21.15.89:0/1136358646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 msgr2=0x5625ac009120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f327ffff640 1 --2- 172.21.15.89:0/1136358646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac009120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f327ffff640 1 -- 172.21.15.89:0/1136358646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625ac00cfb0 con 0x5625ac0065d0 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3284e5f640 1 --2- 172.21.15.89:0/1136358646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac011370 0x5625ac009660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f327ffff640 1 --2- 172.21.15.89:0/1136358646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac00ff70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f327000ec80 tx=0x7f32700118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4b3d0d6203e58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3275ffb640 1 -- 172.21.15.89:0/1136358646 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327001c4d0 con 0x5625ac0065d0 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3275ffb640 1 -- 172.21.15.89:0/1136358646 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f327001cb50 con 0x5625ac0065d0 2024-09-15T23:28:22.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.823+0000 7f3275ffb640 1 -- 172.21.15.89:0/1136358646 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270010dc0 con 0x5625ac0065d0 2024-09-15T23:28:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.824+0000 7f3288c9b740 1 -- 172.21.15.89:0/1136358646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 msgr2=0x5625ac00ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.824+0000 7f3288c9b740 1 --2- 172.21.15.89:0/1136358646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac00ff70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f327000ec80 tx=0x7f32700118d0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.824+0000 7f3288c9b740 1 -- 172.21.15.89:0/1136358646 shutdown_connections 2024-09-15T23:28:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.824+0000 7f3288c9b740 1 --2- 172.21.15.89:0/1136358646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac011370 0x5625ac009660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.824+0000 7f3288c9b740 1 --2- 172.21.15.89:0/1136358646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac009120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.824+0000 7f3288c9b740 1 --2- 172.21.15.89:0/1136358646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac0065d0 0x5625ac00ff70 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.824+0000 7f3288c9b740 1 -- 172.21.15.89:0/1136358646 >> 172.21.15.89:0/1136358646 conn(0x5625abfcf310 msgr2=0x5625ac00f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.824+0000 7f3288c9b740 1 -- 172.21.15.89:0/1136358646 shutdown_connections 2024-09-15T23:28:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.824+0000 7f3288c9b740 1 -- 172.21.15.89:0/1136358646 wait complete. 2024-09-15T23:28:22.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.824+0000 7f3288c9b740 1 -- start start 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3288c9b740 1 Processor -- start 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 0x5625ac0152d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac015810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3288c9b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac015d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3288c9b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f32801b0170 con 0x5625ac011370 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3288c9b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f32801afaa0 con 0x5625ac007b40 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3288c9b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f32801acce0 con 0x5625ac0065d0 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3288c9b740 10 client.0 _ll_register_callbacks cb 0x7f3278004430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3284e5f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac015d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f327ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 0x5625ac0152d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f327ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 0x5625ac0152d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:51676/0 (socket says 172.21.15.89:51676) 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3284e5f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac015d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49112/0 (socket says 172.21.15.89:49112) 2024-09-15T23:28:22.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f327ffff640 1 -- 172.21.15.89:0/3281963109 learned_addr learned my addr 172.21.15.89:0/3281963109 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3277fff640 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac015810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3284e5f640 1 -- 172.21.15.89:0/3281963109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 msgr2=0x5625ac0152d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3284e5f640 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 0x5625ac0152d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3284e5f640 1 -- 172.21.15.89:0/3281963109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 msgr2=0x5625ac015810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3284e5f640 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac015810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3284e5f640 1 -- 172.21.15.89:0/3281963109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625ac08f160 con 0x5625ac011370 2024-09-15T23:28:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3277fff640 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac015810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f3284e5f640 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac015d50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f32780ee910 tx=0x7f32780bcb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cefbb10ea071b6e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.825+0000 7f325f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:22.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f325f7fe640 1 -- 172.21.15.89:0/3281963109 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327802c040 con 0x5625ac011370 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f325f7fe640 1 -- 172.21.15.89:0/3281963109 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3278004d40 con 0x5625ac011370 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f3288c9b740 1 -- 172.21.15.89:0/3281963109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f327001cb50 con 0x5625ac011370 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f325f7fe640 1 -- 172.21.15.89:0/3281963109 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f32780bce90 con 0x5625ac011370 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f3288c9b740 10 client.16950 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f3288c9b740 1 -- 172.21.15.89:0/3281963109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625ac0903b0 con 0x5625ac011370 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f325f7fe640 1 -- 172.21.15.89:0/3281963109 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f327001cb50 con 0x5625ac011370 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f325f7fe640 1 -- 172.21.15.89:0/3281963109 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f327804d070 con 0x5625ac011370 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f3288c9b740 10 client.16950 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f3288c9b740 10 client.16950 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f3288c9b740 20 client.16950 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f3288c9b740 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625ac090930 0x5625ac092d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f3288c9b740 4 client.16950 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f3288c9b740 1 -- 172.21.15.89:0/3281963109 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625abf73d70 con 0x5625ac090930 2024-09-15T23:28:22.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.826+0000 7f327ffff640 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625ac090930 0x5625ac092d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f327ffff640 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625ac090930 0x5625ac092d20 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87f8b62a931ebecd server_cookie=c6ef7bba0d8d4e50 in_seq=0 out_seq=0 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f325f7fe640 10 client.16950 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f325f7fe640 1 -- 172.21.15.89:0/3281963109 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625abf73d70 con 0x5625ac090930 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f325f7fe640 10 client.16950 handle_command_reply: tid=0 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 client.16950 shutdown 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 -- 172.21.15.89:0/3281963109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 msgr2=0x5625ac015d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac015d50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f32780ee910 tx=0x7f32780bcb50 comp rx=0 tx=0).stop 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 -- 172.21.15.89:0/3281963109 shutdown_connections 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625ac090930 0x5625ac092d20 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625ac011370 0x5625ac015d50 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625ac007b40 0x5625ac015810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f325f7fe640 0 client.16950 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 --2- 172.21.15.89:0/3281963109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625ac0065d0 0x5625ac0152d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 -- 172.21.15.89:0/3281963109 >> 172.21.15.89:0/3281963109 conn(0x5625abfcf310 msgr2=0x5625ac010080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 -- 172.21.15.89:0/3281963109 shutdown_connections 2024-09-15T23:28:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 1 -- 172.21.15.89:0/3281963109 wait complete. 2024-09-15T23:28:22.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.827+0000 7f3288c9b740 20 client.16950 trim_cache size 0 max 16384 2024-09-15T23:28:22.842 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:28:22.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 -- 172.21.15.89:0/846174534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f326c0a32c0 msgr2=0x7f326c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:22.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f326c0a32c0 0x7f326c0a5780 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f328010d4a0 tx=0x7f326800ad50 comp rx=0 tx=0).stop 2024-09-15T23:28:22.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 -- 172.21.15.89:0/846174534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f328010c180 msgr2=0x7f328019cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:22.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f328010c180 0x7f328019cc30 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f3278006010 tx=0x7f327802f710 comp rx=0 tx=0).stop 2024-09-15T23:28:22.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 -- 172.21.15.89:0/846174534 shutdown_connections 2024-09-15T23:28:22.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f326c0a32c0 0x7f326c0a5780 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f328010c180 0x7f328019cc30 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801036c0 0x7f32801988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 --2- 172.21.15.89:0/846174534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3280102da0 0x7f3280198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:22.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 -- 172.21.15.89:0/846174534 >> 172.21.15.89:0/846174534 conn(0x7f32800fc7a0 msgr2=0x7f32800fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:22.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 -- 172.21.15.89:0/846174534 shutdown_connections 2024-09-15T23:28:22.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:22.828+0000 7f32868c2640 1 -- 172.21.15.89:0/846174534 wait complete. 2024-09-15T23:28:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:23 smithi120 ceph-mon[35250]: pgmap v481: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:28:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:23 smithi174 ceph-mon[35586]: pgmap v481: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:28:23.497 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:23 smithi089 ceph-mon[30933]: pgmap v481: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:28:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.511+0000 7fda772e6640 1 -- 172.21.15.89:0/325244298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda70102620 msgr2=0x7fda70104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.511+0000 7fda772e6640 1 --2- 172.21.15.89:0/325244298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda70102620 0x7fda70104a10 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009900 tx=0x7fda6c02a410 comp rx=0 tx=0).stop 2024-09-15T23:28:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.511+0000 7fda772e6640 1 -- 172.21.15.89:0/325244298 shutdown_connections 2024-09-15T23:28:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.511+0000 7fda772e6640 1 --2- 172.21.15.89:0/325244298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda70107880 0x7fda70109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.511+0000 7fda772e6640 1 --2- 172.21.15.89:0/325244298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda70104f50 0x7fda70107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.511+0000 7fda772e6640 1 --2- 172.21.15.89:0/325244298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda70102620 0x7fda70104a10 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.511+0000 7fda772e6640 1 -- 172.21.15.89:0/325244298 >> 172.21.15.89:0/325244298 conn(0x7fda700fc1e0 msgr2=0x7fda700fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.511+0000 7fda772e6640 1 -- 172.21.15.89:0/325244298 shutdown_connections 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.511+0000 7fda772e6640 1 -- 172.21.15.89:0/325244298 wait complete. 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda772e6640 1 Processor -- start 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda772e6640 1 -- start start 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda772e6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda70102620 0x7fda70197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda772e6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda70104f50 0x7fda701983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda772e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda70107880 0x7fda7019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda772e6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fda7010d830 con 0x7fda70104f50 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda772e6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda7010d6b0 con 0x7fda70102620 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda772e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fda7010d9b0 con 0x7fda70107880 2024-09-15T23:28:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda74881640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda70104f50 0x7fda701983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda74881640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda70104f50 0x7fda701983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49126/0 (socket says 172.21.15.89:49126) 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda74881640 1 -- 172.21.15.89:0/2300334575 learned_addr learned my addr 172.21.15.89:0/2300334575 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda75883640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda70107880 0x7fda7019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda75082640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda70102620 0x7fda70197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda74881640 1 -- 172.21.15.89:0/2300334575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda70107880 msgr2=0x7fda7019f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda74881640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda70107880 0x7fda7019f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda74881640 1 -- 172.21.15.89:0/2300334575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda70102620 msgr2=0x7fda70197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda74881640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda70102620 0x7fda70197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda74881640 1 -- 172.21.15.89:0/2300334575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda7019faa0 con 0x7fda70104f50 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.512+0000 7fda74881640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda70104f50 0x7fda701983c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fda6000d740 tx=0x7fda6000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.513+0000 7fda667fc640 1 -- 172.21.15.89:0/2300334575 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60017070 con 0x7fda70104f50 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.513+0000 7fda667fc640 1 -- 172.21.15.89:0/2300334575 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda6000de50 con 0x7fda70104f50 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.513+0000 7fda772e6640 1 -- 172.21.15.89:0/2300334575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda7019fd90 con 0x7fda70104f50 2024-09-15T23:28:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.513+0000 7fda667fc640 1 -- 172.21.15.89:0/2300334575 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60004e00 con 0x7fda70104f50 2024-09-15T23:28:24.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.513+0000 7fda772e6640 1 -- 172.21.15.89:0/2300334575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda701a0230 con 0x7fda70104f50 2024-09-15T23:28:24.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.515+0000 7fda667fc640 1 -- 172.21.15.89:0/2300334575 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fda60004fa0 con 0x7fda70104f50 2024-09-15T23:28:24.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.515+0000 7fda7a07e740 1 Processor -- start 2024-09-15T23:28:24.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.515+0000 7fda7a07e740 1 -- start start 2024-09-15T23:28:24.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.515+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abc72da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.515+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc77da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.515+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc78370 0x5602abc7a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.515+0000 7fda7a07e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5602abbd62f0 con 0x5602abc732e0 2024-09-15T23:28:24.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.515+0000 7fda7a07e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5602abb7e270 con 0x5602abc78370 2024-09-15T23:28:24.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.515+0000 7fda7a07e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5602abbd22b0 con 0x5602abc729a0 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75082640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc77da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75082640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc77da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49136/0 (socket says 172.21.15.89:49136) 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75082640 1 -- 172.21.15.89:0/4287090248 learned_addr learned my addr 172.21.15.89:0/4287090248 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75883640 1 --2- 172.21.15.89:0/4287090248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abc72da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda74881640 1 --2- 172.21.15.89:0/4287090248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc78370 0x5602abc7a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75082640 1 -- 172.21.15.89:0/4287090248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 msgr2=0x5602abc72da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75082640 1 --2- 172.21.15.89:0/4287090248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abc72da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75082640 1 -- 172.21.15.89:0/4287090248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc78370 msgr2=0x5602abc7a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75082640 1 --2- 172.21.15.89:0/4287090248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc78370 0x5602abc7a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75082640 1 -- 172.21.15.89:0/4287090248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602abb7d340 con 0x5602abc732e0 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda74881640 1 --2- 172.21.15.89:0/4287090248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc78370 0x5602abc7a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda667fc640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda5c0a32f0 0x7fda5c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75082640 1 --2- 172.21.15.89:0/4287090248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc77da0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fda6c02a930 tx=0x7fda6c032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f916e78cb929e7b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda75883640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda5c0a32f0 0x7fda5c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda667fc640 1 -- 172.21.15.89:0/2300334575 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fda600cf6d0 con 0x7fda70104f50 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda5b7fe640 1 -- 172.21.15.89:0/4287090248 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c03f070 con 0x5602abc732e0 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda5b7fe640 1 -- 172.21.15.89:0/4287090248 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda6c031970 con 0x5602abc732e0 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.516+0000 7fda5b7fe640 1 -- 172.21.15.89:0/4287090248 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c035070 con 0x5602abc732e0 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 -- 172.21.15.89:0/4287090248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 msgr2=0x5602abc77da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda75883640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda5c0a32f0 0x7fda5c0a57b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fda7006a980 tx=0x7fda680096a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 --2- 172.21.15.89:0/4287090248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc77da0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fda6c02a930 tx=0x7fda6c032d60 comp rx=0 tx=0).stop 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 -- 172.21.15.89:0/4287090248 shutdown_connections 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 --2- 172.21.15.89:0/4287090248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc78370 0x5602abc7a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 --2- 172.21.15.89:0/4287090248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc77da0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 --2- 172.21.15.89:0/4287090248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abc72da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 -- 172.21.15.89:0/4287090248 >> 172.21.15.89:0/4287090248 conn(0x5602abc6e010 msgr2=0x5602abc6e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 -- 172.21.15.89:0/4287090248 shutdown_connections 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 -- 172.21.15.89:0/4287090248 wait complete. 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 -- start start 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.517+0000 7fda7a07e740 1 Processor -- start 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc729a0 0x5602abc7cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc7d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc78370 0x5602abc84180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda7a07e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5602abc7b2e0 con 0x5602abc732e0 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda7a07e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5602abb5f290 con 0x5602abc729a0 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda7a07e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5602abc7b460 con 0x5602abc78370 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda7a07e740 10 client.0 _ll_register_callbacks cb 0x5602abb90140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda74881640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc7d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda74881640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc7d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49150/0 (socket says 172.21.15.89:49150) 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda74881640 1 -- 172.21.15.89:0/3978017913 learned_addr learned my addr 172.21.15.89:0/3978017913 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda75883640 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc78370 0x5602abc84180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda75082640 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc729a0 0x5602abc7cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda74881640 1 -- 172.21.15.89:0/3978017913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc78370 msgr2=0x5602abc84180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda74881640 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc78370 0x5602abc84180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda74881640 1 -- 172.21.15.89:0/3978017913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc729a0 msgr2=0x5602abc7cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda74881640 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc729a0 0x5602abc7cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda74881640 1 -- 172.21.15.89:0/3978017913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602ab8e9210 con 0x5602abc732e0 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda75082640 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc729a0 0x5602abc7cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda74881640 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc7d140 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fda60009610 tx=0x7fda6009c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce91b62646820f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.518+0000 7fda58ff9640 1 -- 172.21.15.89:0/3978017913 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60020070 con 0x5602abc732e0 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda58ff9640 1 -- 172.21.15.89:0/3978017913 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fda60009cc0 con 0x5602abc732e0 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda7a07e740 1 -- 172.21.15.89:0/3978017913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda6c031970 con 0x5602abc732e0 2024-09-15T23:28:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda58ff9640 1 -- 172.21.15.89:0/3978017913 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fda6009fcb0 con 0x5602abc732e0 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda7a07e740 10 client.16962 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda7a07e740 1 -- 172.21.15.89:0/3978017913 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5602ab9b5620 con 0x5602abc732e0 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda58ff9640 1 -- 172.21.15.89:0/3978017913 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda6c031970 con 0x5602abc732e0 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda58ff9640 1 -- 172.21.15.89:0/3978017913 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fda600cfc10 con 0x5602abc732e0 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda7a07e740 10 client.16962 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda7a07e740 10 client.16962 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda7a07e740 20 client.16962 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda7a07e740 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602abc887e0 0x5602abc8ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda7a07e740 4 client.16962 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda7a07e740 1 -- 172.21.15.89:0/3978017913 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602abbe3a20 con 0x5602abc887e0 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.519+0000 7fda75082640 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602abc887e0 0x5602abc8ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.520+0000 7fda58ff9640 10 client.16962 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.520+0000 7fda75082640 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602abc887e0 0x5602abc8ac30 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41bee806b1ebaf6a server_cookie=33fe4cbc28c71af9 in_seq=0 out_seq=0 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.520+0000 7fda58ff9640 1 -- 172.21.15.89:0/3978017913 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5602abbe3a20 con 0x5602abc887e0 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.520+0000 7fda58ff9640 10 client.16962 handle_command_reply: tid=0 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.520+0000 7fda7a07e740 1 client.16962 shutdown 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 1 -- 172.21.15.89:0/3978017913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 msgr2=0x5602abc7d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc7d140 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fda60009610 tx=0x7fda6009c000 comp rx=0 tx=0).stop 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 1 -- 172.21.15.89:0/3978017913 shutdown_connections 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602abc887e0 0x5602abc8ac30 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc78370 0x5602abc84180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc7d140 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda58ff9640 0 client.16962 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 1 --2- 172.21.15.89:0/3978017913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc729a0 0x5602abc7cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 1 -- 172.21.15.89:0/3978017913 >> 172.21.15.89:0/3978017913 conn(0x5602abc6e010 msgr2=0x5602abc6f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 1 -- 172.21.15.89:0/3978017913 shutdown_connections 2024-09-15T23:28:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 1 -- 172.21.15.89:0/3978017913 wait complete. 2024-09-15T23:28:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.521+0000 7fda7a07e740 20 client.16962 trim_cache size 0 max 16384 2024-09-15T23:28:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda7a07e740 1 Processor -- start 2024-09-15T23:28:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda7a07e740 1 -- start start 2024-09-15T23:28:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abc7b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc6d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc78370 0x5602abc8f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda7a07e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fda701aa7b0 con 0x5602abc732e0 2024-09-15T23:28:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda7a07e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda701ac2f0 con 0x5602abc78370 2024-09-15T23:28:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda7a07e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fda701aae80 con 0x5602abc729a0 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda74881640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc6d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda74881640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc6d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49154/0 (socket says 172.21.15.89:49154) 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.535+0000 7fda74881640 1 -- 172.21.15.89:0/3780201521 learned_addr learned my addr 172.21.15.89:0/3780201521 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda74881640 1 -- 172.21.15.89:0/3780201521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 msgr2=0x5602abc7b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda75082640 1 --2- 172.21.15.89:0/3780201521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abc7b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda74881640 1 --2- 172.21.15.89:0/3780201521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abc7b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda74881640 1 -- 172.21.15.89:0/3780201521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc78370 msgr2=0x5602abc8f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda74881640 1 --2- 172.21.15.89:0/3780201521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc78370 0x5602abc8f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda74881640 1 -- 172.21.15.89:0/3780201521 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602abc6de30 con 0x5602abc732e0 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda74881640 1 --2- 172.21.15.89:0/3780201521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc6d0b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fda600982d0 tx=0x7fda6009b370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db36ccdd68b001e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda5affd640 1 -- 172.21.15.89:0/3780201521 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60020070 con 0x5602abc732e0 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda5affd640 1 -- 172.21.15.89:0/3780201521 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda60009cc0 con 0x5602abc732e0 2024-09-15T23:28:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda5affd640 1 -- 172.21.15.89:0/3780201521 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda600cfb30 con 0x5602abc732e0 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda7a07e740 1 -- 172.21.15.89:0/3780201521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 msgr2=0x5602abc6d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda7a07e740 1 --2- 172.21.15.89:0/3780201521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc6d0b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fda600982d0 tx=0x7fda6009b370 comp rx=0 tx=0).stop 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda7a07e740 1 -- 172.21.15.89:0/3780201521 shutdown_connections 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda7a07e740 1 --2- 172.21.15.89:0/3780201521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc78370 0x5602abc8f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda7a07e740 1 --2- 172.21.15.89:0/3780201521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc732e0 0x5602abc6d0b0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda7a07e740 1 --2- 172.21.15.89:0/3780201521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abc7b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda7a07e740 1 -- 172.21.15.89:0/3780201521 >> 172.21.15.89:0/3780201521 conn(0x5602abc6e010 msgr2=0x5602abc6c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda7a07e740 1 -- 172.21.15.89:0/3780201521 shutdown_connections 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.536+0000 7fda7a07e740 1 -- 172.21.15.89:0/3780201521 wait complete. 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda7a07e740 1 -- start start 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda7a07e740 1 Processor -- start 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abceead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc732e0 0x5602abcef010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda7a07e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc78370 0x5602abcf6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.542 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda7a07e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fda701abc20 con 0x5602abc78370 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda7a07e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda701ab550 con 0x5602abc732e0 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda7a07e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fda701a8790 con 0x5602abc729a0 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda7a07e740 10 client.0 _ll_register_callbacks cb 0x7fda60097ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda75883640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc78370 0x5602abcf6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda75883640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc78370 0x5602abcf6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49170/0 (socket says 172.21.15.89:49170) 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda75883640 1 -- 172.21.15.89:0/1987928024 learned_addr learned my addr 172.21.15.89:0/1987928024 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.537+0000 7fda75082640 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abceead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.538+0000 7fda74881640 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc732e0 0x5602abcef010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.538+0000 7fda75082640 1 -- 172.21.15.89:0/1987928024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc732e0 msgr2=0x5602abcef010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.538+0000 7fda75082640 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc732e0 0x5602abcef010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.538+0000 7fda75082640 1 -- 172.21.15.89:0/1987928024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc78370 msgr2=0x5602abcf6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.538+0000 7fda75082640 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc78370 0x5602abcf6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.543 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.538+0000 7fda75082640 1 -- 172.21.15.89:0/1987928024 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602abcf6660 con 0x5602abc729a0 2024-09-15T23:28:24.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.539+0000 7fda75082640 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abceead0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda6c031440 tx=0x7fda6c0315b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db54cbb1cac268ec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:24.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.539+0000 7fda58ff9640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:28:24.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.539+0000 7fda58ff9640 1 -- 172.21.15.89:0/1987928024 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c03f040 con 0x5602abc729a0 2024-09-15T23:28:24.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.539+0000 7fda58ff9640 1 -- 172.21.15.89:0/1987928024 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fda6c0040c0 con 0x5602abc729a0 2024-09-15T23:28:24.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.539+0000 7fda7a07e740 1 -- 172.21.15.89:0/1987928024 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda60009cc0 con 0x5602abc729a0 2024-09-15T23:28:24.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.539+0000 7fda58ff9640 1 -- 172.21.15.89:0/1987928024 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fda6c060050 con 0x5602abc729a0 2024-09-15T23:28:24.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.539+0000 7fda7a07e740 10 client.25969 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:24.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.539+0000 7fda7a07e740 1 -- 172.21.15.89:0/1987928024 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5602abcf79e0 con 0x5602abc729a0 2024-09-15T23:28:24.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.540+0000 7fda58ff9640 1 -- 172.21.15.89:0/1987928024 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda60009cc0 con 0x5602abc729a0 2024-09-15T23:28:24.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.540+0000 7fda58ff9640 1 -- 172.21.15.89:0/1987928024 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fda6c03b430 con 0x5602abc729a0 2024-09-15T23:28:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.540+0000 7fda7a07e740 10 client.25969 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.540+0000 7fda7a07e740 10 client.25969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.540+0000 7fda7a07e740 20 client.25969 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.540+0000 7fda7a07e740 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602abc887e0 0x5602abcf9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.540+0000 7fda7a07e740 4 client.25969 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:28:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.540+0000 7fda7a07e740 1 -- 172.21.15.89:0/1987928024 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5602abbddd70 con 0x5602abc887e0 2024-09-15T23:28:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.540+0000 7fda74881640 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602abc887e0 0x5602abcf9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.541+0000 7fda74881640 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602abc887e0 0x5602abcf9f70 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7097ac0428df80e4 server_cookie=10825d89fb8cfa7a in_seq=0 out_seq=0 2024-09-15T23:28:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.541+0000 7fda58ff9640 10 client.25969 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda58ff9640 1 -- 172.21.15.89:0/1987928024 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5602abbddd70 con 0x5602abc887e0 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda58ff9640 10 client.25969 handle_command_reply: tid=0 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda7a07e740 1 client.25969 shutdown 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda7a07e740 1 -- 172.21.15.89:0/1987928024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 msgr2=0x5602abceead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda7a07e740 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abceead0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda6c031440 tx=0x7fda6c0315b0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda7a07e740 1 -- 172.21.15.89:0/1987928024 shutdown_connections 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda7a07e740 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602abc887e0 0x5602abcf9f70 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda7a07e740 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602abc78370 0x5602abcf6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda58ff9640 0 client.25969 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda7a07e740 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602abc732e0 0x5602abcef010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda7a07e740 1 --2- 172.21.15.89:0/1987928024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602abc729a0 0x5602abceead0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.556+0000 7fda7a07e740 1 -- 172.21.15.89:0/1987928024 >> 172.21.15.89:0/1987928024 conn(0x5602abc6e010 msgr2=0x5602abc6d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:24.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda7a07e740 1 -- 172.21.15.89:0/1987928024 shutdown_connections 2024-09-15T23:28:24.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda7a07e740 1 -- 172.21.15.89:0/1987928024 wait complete. 2024-09-15T23:28:24.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda7a07e740 20 client.25969 trim_cache size 0 max 16384 2024-09-15T23:28:24.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:28:24.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:28:24.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:28:24.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:28:24.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 -- 172.21.15.89:0/2300334575 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda5c0a32f0 msgr2=0x7fda5c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:24.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda5c0a32f0 0x7fda5c0a57b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fda7006a980 tx=0x7fda680096a0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 -- 172.21.15.89:0/2300334575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda70104f50 msgr2=0x7fda701983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:24.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda70104f50 0x7fda701983c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fda6000d740 tx=0x7fda6000dc00 comp rx=0 tx=0).stop 2024-09-15T23:28:24.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 -- 172.21.15.89:0/2300334575 shutdown_connections 2024-09-15T23:28:24.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda5c0a32f0 0x7fda5c0a57b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda70107880 0x7fda7019f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda70104f50 0x7fda701983c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 --2- 172.21.15.89:0/2300334575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda70102620 0x7fda70197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:24.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 -- 172.21.15.89:0/2300334575 >> 172.21.15.89:0/2300334575 conn(0x7fda700fc1e0 msgr2=0x7fda70102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:24.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 -- 172.21.15.89:0/2300334575 shutdown_connections 2024-09-15T23:28:24.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:24.557+0000 7fda772e6640 1 -- 172.21.15.89:0/2300334575 wait complete. 2024-09-15T23:28:24.699 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:28:24.700 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:28:24.944 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:25.373 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:25 smithi089 ceph-mon[30933]: pgmap v482: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-09-15T23:28:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:25 smithi120 ceph-mon[35250]: pgmap v482: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-09-15T23:28:25.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.455+0000 7f85608fe640 1 -- 172.21.15.89:0/815626565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f855c101420 msgr2=0x7f855c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:25.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.455+0000 7f85608fe640 1 --2- 172.21.15.89:0/815626565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f855c101420 0x7f855c10d950 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f8554009cd0 tx=0x7f855402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.455+0000 7f85608fe640 1 -- 172.21.15.89:0/815626565 shutdown_connections 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.455+0000 7f85608fe640 1 --2- 172.21.15.89:0/815626565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f855c101420 0x7f855c10d950 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.455+0000 7f85608fe640 1 --2- 172.21.15.89:0/815626565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c100a80 0x7f855c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.455+0000 7f85608fe640 1 --2- 172.21.15.89:0/815626565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f855c106a80 0x7f855c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.455+0000 7f85608fe640 1 -- 172.21.15.89:0/815626565 >> 172.21.15.89:0/815626565 conn(0x7f855c0fc7a0 msgr2=0x7f855c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.455+0000 7f85608fe640 1 -- 172.21.15.89:0/815626565 shutdown_connections 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.455+0000 7f85608fe640 1 -- 172.21.15.89:0/815626565 wait complete. 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f85608fe640 1 Processor -- start 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f85608fe640 1 -- start start 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f85608fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f855c100a80 0x7f855c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f85608fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c101420 0x7f855c198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f85608fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f855c106a80 0x7f855c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f85608fe640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f855c110f20 con 0x7f855c100a80 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f85608fe640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f855c110da0 con 0x7f855c101420 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f85608fe640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1110a0 con 0x7f855c106a80 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f855a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f855c100a80 0x7f855c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f855a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f855c100a80 0x7f855c198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38792/0 (socket says 172.21.15.89:38792) 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.456+0000 7f855a59c640 1 -- 172.21.15.89:0/1989520273 learned_addr learned my addr 172.21.15.89:0/1989520273 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f855ad9d640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f855c106a80 0x7f855c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f8553fff640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c101420 0x7f855c198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f855a59c640 1 -- 172.21.15.89:0/1989520273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f855c106a80 msgr2=0x7f855c19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f855a59c640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f855c106a80 0x7f855c19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f855a59c640 1 -- 172.21.15.89:0/1989520273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c101420 msgr2=0x7f855c198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f855a59c640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c101420 0x7f855c198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.476 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f855a59c640 1 -- 172.21.15.89:0/1989520273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855c19d500 con 0x7f855c100a80 2024-09-15T23:28:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f855a59c640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f855c100a80 0x7f855c198550 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f854c00b4a0 tx=0x7f854c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f85537fe640 1 -- 172.21.15.89:0/1989520273 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c014070 con 0x7f855c100a80 2024-09-15T23:28:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f85608fe640 1 -- 172.21.15.89:0/1989520273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f855c19d7f0 con 0x7f855c100a80 2024-09-15T23:28:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f85537fe640 1 -- 172.21.15.89:0/1989520273 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f854c010af0 con 0x7f855c100a80 2024-09-15T23:28:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f85537fe640 1 -- 172.21.15.89:0/1989520273 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c017d10 con 0x7f855c100a80 2024-09-15T23:28:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.457+0000 7f85608fe640 1 -- 172.21.15.89:0/1989520273 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f855c1a50d0 con 0x7f855c100a80 2024-09-15T23:28:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.460+0000 7f8562cd7740 1 Processor -- start 2024-09-15T23:28:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.460+0000 7f8562cd7740 1 -- start start 2024-09-15T23:28:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.460+0000 7f85537fe640 1 -- 172.21.15.89:0/1989520273 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f854c017070 con 0x7f855c100a80 2024-09-15T23:28:25.477 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.460+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b68f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.460+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b69480 0x55f941b73800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.460+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 0x55f941b76130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.460+0000 7f8562cd7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f941ace280 con 0x55f941b69480 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.460+0000 7f8562cd7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f941a75b10 con 0x55f941b68b60 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.460+0000 7f8562cd7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f941aca240 con 0x55f941b73d40 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b69480 0x55f941b73800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b69480 0x55f941b73800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38794/0 (socket says 172.21.15.89:38794) 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855a59c640 1 -- 172.21.15.89:0/1137544241 learned_addr learned my addr 172.21.15.89:0/1137544241 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855ad9d640 1 --2- 172.21.15.89:0/1137544241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b68f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f8553fff640 1 --2- 172.21.15.89:0/1137544241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 0x55f941b76130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855a59c640 1 -- 172.21.15.89:0/1137544241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 msgr2=0x55f941b76130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:25.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855a59c640 1 --2- 172.21.15.89:0/1137544241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 0x55f941b76130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855a59c640 1 -- 172.21.15.89:0/1137544241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 msgr2=0x55f941b68f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855a59c640 1 --2- 172.21.15.89:0/1137544241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b68f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855a59c640 1 -- 172.21.15.89:0/1137544241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f941a821f0 con 0x55f941b69480 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f8553fff640 1 --2- 172.21.15.89:0/1137544241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 0x55f941b76130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f85537fe640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85480a3240 0x7f85480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855ad9d640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85480a3240 0x7f85480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f855a59c640 1 --2- 172.21.15.89:0/1137544241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b69480 0x55f941b73800 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f854c0d7480 tx=0x7f854c0d7940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1996f45e24af5ab8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f85537fe640 1 -- 172.21.15.89:0/1989520273 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f854c0cf180 con 0x7f855c100a80 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f8550ff9640 1 -- 172.21.15.89:0/1137544241 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c09b180 con 0x55f941b69480 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f8550ff9640 1 -- 172.21.15.89:0/1137544241 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f854c09bab0 con 0x55f941b69480 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.461+0000 7f8550ff9640 1 -- 172.21.15.89:0/1137544241 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c09ad10 con 0x55f941b69480 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.462+0000 7f8562cd7740 1 -- 172.21.15.89:0/1137544241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b69480 msgr2=0x55f941b73800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:25.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.462+0000 7f855ad9d640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85480a3240 0x7f85480a5700 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8554009430 tx=0x7f8554005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.462+0000 7f8562cd7740 1 --2- 172.21.15.89:0/1137544241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b69480 0x55f941b73800 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f854c0d7480 tx=0x7f854c0d7940 comp rx=0 tx=0).stop 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.462+0000 7f8562cd7740 1 -- 172.21.15.89:0/1137544241 shutdown_connections 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.462+0000 7f8562cd7740 1 --2- 172.21.15.89:0/1137544241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 0x55f941b76130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.462+0000 7f8562cd7740 1 --2- 172.21.15.89:0/1137544241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b69480 0x55f941b73800 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.462+0000 7f8562cd7740 1 --2- 172.21.15.89:0/1137544241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b68f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.462+0000 7f8562cd7740 1 -- 172.21.15.89:0/1137544241 >> 172.21.15.89:0/1137544241 conn(0x55f941b650f0 msgr2=0x55f941b654d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.462+0000 7f8562cd7740 1 -- 172.21.15.89:0/1137544241 shutdown_connections 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.462+0000 7f8562cd7740 1 -- 172.21.15.89:0/1137544241 wait complete. 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f8562cd7740 1 -- start start 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f8562cd7740 1 Processor -- start 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b70bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b69480 0x55f941b71130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b71670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f8562cd7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f941b76910 con 0x55f941b73d40 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f8562cd7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f941a88f10 con 0x55f941b68b60 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f8562cd7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f941b76a90 con 0x55f941b69480 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f8562cd7740 10 client.0 _ll_register_callbacks cb 0x55f941a51a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f855ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b71670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f855ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b71670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38796/0 (socket says 172.21.15.89:38796) 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f855ad9d640 1 -- 172.21.15.89:0/2476982845 learned_addr learned my addr 172.21.15.89:0/2476982845 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f855a59c640 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b70bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.463+0000 7f8553fff640 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b69480 0x55f941b71130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f855a59c640 1 -- 172.21.15.89:0/2476982845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b69480 msgr2=0x55f941b71130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f855a59c640 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b69480 0x55f941b71130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f855a59c640 1 -- 172.21.15.89:0/2476982845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 msgr2=0x55f941b71670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f855a59c640 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b71670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f855a59c640 1 -- 172.21.15.89:0/2476982845 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f94183f6a0 con 0x55f941b68b60 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f855ad9d640 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b71670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f855a59c640 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b70bf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f854c00b470 tx=0x7f854c0cf480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46b803801afcd223 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f8541ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f8541ffb640 1 -- 172.21.15.89:0/2476982845 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c01e070 con 0x55f941b68b60 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f8562cd7740 1 -- 172.21.15.89:0/2476982845 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f854c09bab0 con 0x55f941b68b60 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f8541ffb640 1 -- 172.21.15.89:0/2476982845 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f854c0d2490 con 0x55f941b68b60 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f8541ffb640 1 -- 172.21.15.89:0/2476982845 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f854c09aef0 con 0x55f941b68b60 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f8562cd7740 10 client.25931 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f8562cd7740 1 -- 172.21.15.89:0/2476982845 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f941b76910 con 0x55f941b68b60 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.464+0000 7f8541ffb640 1 -- 172.21.15.89:0/2476982845 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f854c09bab0 con 0x55f941b68b60 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.465+0000 7f8541ffb640 1 -- 172.21.15.89:0/2476982845 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f854c066110 con 0x55f941b68b60 2024-09-15T23:28:25.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.465+0000 7f8562cd7740 10 client.25931 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.465+0000 7f8562cd7740 10 client.25931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.465+0000 7f8562cd7740 20 client.25931 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.465+0000 7f8562cd7740 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f941b83660 0x55f941b85a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.465+0000 7f8562cd7740 4 client.25931 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.465+0000 7f8562cd7740 1 -- 172.21.15.89:0/2476982845 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f941adb7f0 con 0x55f941b83660 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.465+0000 7f8553fff640 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f941b83660 0x55f941b85a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.465+0000 7f8541ffb640 10 client.25931 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.465+0000 7f8553fff640 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f941b83660 0x55f941b85a50 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af867df08f553a41 server_cookie=efd54f33fb34492d in_seq=0 out_seq=0 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8541ffb640 1 -- 172.21.15.89:0/2476982845 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f941adb7f0 con 0x55f941b83660 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8541ffb640 10 client.25931 handle_command_reply: tid=0 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 client.25931 shutdown 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 -- 172.21.15.89:0/2476982845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 msgr2=0x55f941b70bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:25.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b70bf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f854c00b470 tx=0x7f854c0cf480 comp rx=0 tx=0).stop 2024-09-15T23:28:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 -- 172.21.15.89:0/2476982845 shutdown_connections 2024-09-15T23:28:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f941b83660 0x55f941b85a50 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b71670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b69480 0x55f941b71130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 --2- 172.21.15.89:0/2476982845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b70bf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8541ffb640 0 client.25931 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 -- 172.21.15.89:0/2476982845 >> 172.21.15.89:0/2476982845 conn(0x55f941b650f0 msgr2=0x55f941b6ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 -- 172.21.15.89:0/2476982845 shutdown_connections 2024-09-15T23:28:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 1 -- 172.21.15.89:0/2476982845 wait complete. 2024-09-15T23:28:25.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.466+0000 7f8562cd7740 20 client.25931 trim_cache size 0 max 16384 2024-09-15T23:28:25.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:25 smithi174 ceph-mon[35586]: pgmap v482: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 14 op/s 2024-09-15T23:28:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.483+0000 7f8562cd7740 1 Processor -- start 2024-09-15T23:28:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8562cd7740 1 -- start start 2024-09-15T23:28:25.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b68b60 0x55f941b6faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b69480 0x55f941b8b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 0x55f941b8b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8562cd7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1aefd0 con 0x55f941b68b60 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8562cd7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1b0b10 con 0x55f941b69480 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8562cd7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1af6a0 con 0x55f941b73d40 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f855a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b68b60 0x55f941b6faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f855a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b68b60 0x55f941b6faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38806/0 (socket says 172.21.15.89:38806) 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f855a59c640 1 -- 172.21.15.89:0/2912280972 learned_addr learned my addr 172.21.15.89:0/2912280972 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f855ad9d640 1 --2- 172.21.15.89:0/2912280972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 0x55f941b8b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8553fff640 1 --2- 172.21.15.89:0/2912280972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b69480 0x55f941b8b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f855a59c640 1 -- 172.21.15.89:0/2912280972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 msgr2=0x55f941b8b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:25.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f855a59c640 1 --2- 172.21.15.89:0/2912280972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 0x55f941b8b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f855a59c640 1 -- 172.21.15.89:0/2912280972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b69480 msgr2=0x55f941b8b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f855a59c640 1 --2- 172.21.15.89:0/2912280972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b69480 0x55f941b8b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f855a59c640 1 -- 172.21.15.89:0/2912280972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f941b64f20 con 0x55f941b68b60 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f855a59c640 1 --2- 172.21.15.89:0/2912280972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b68b60 0x55f941b6faa0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f854c097810 tx=0x7f854c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f477bd1ff07989f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8543fff640 1 -- 172.21.15.89:0/2912280972 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c01e070 con 0x55f941b68b60 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8543fff640 1 -- 172.21.15.89:0/2912280972 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f854c0d2db0 con 0x55f941b68b60 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.484+0000 7f8543fff640 1 -- 172.21.15.89:0/2912280972 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854c017360 con 0x55f941b68b60 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.485+0000 7f8562cd7740 1 -- 172.21.15.89:0/2912280972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b68b60 msgr2=0x55f941b6faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.485+0000 7f8562cd7740 1 --2- 172.21.15.89:0/2912280972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b68b60 0x55f941b6faa0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f854c097810 tx=0x7f854c0047d0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.485+0000 7f8562cd7740 1 -- 172.21.15.89:0/2912280972 shutdown_connections 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.485+0000 7f8562cd7740 1 --2- 172.21.15.89:0/2912280972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b73d40 0x55f941b8b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.485+0000 7f8562cd7740 1 --2- 172.21.15.89:0/2912280972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b69480 0x55f941b8b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.485+0000 7f8562cd7740 1 --2- 172.21.15.89:0/2912280972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b68b60 0x55f941b6faa0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.485+0000 7f8562cd7740 1 -- 172.21.15.89:0/2912280972 >> 172.21.15.89:0/2912280972 conn(0x55f941b650f0 msgr2=0x55f941b6f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.485+0000 7f8562cd7740 1 -- 172.21.15.89:0/2912280972 shutdown_connections 2024-09-15T23:28:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.485+0000 7f8562cd7740 1 -- 172.21.15.89:0/2912280972 wait complete. 2024-09-15T23:28:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.485+0000 7f8562cd7740 1 -- start start 2024-09-15T23:28:25.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f8562cd7740 1 Processor -- start 2024-09-15T23:28:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b7c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b69480 0x55f941b7c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f8562cd7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b77430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f8562cd7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1b0440 con 0x55f941b73d40 2024-09-15T23:28:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f8562cd7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1afd70 con 0x55f941b68b60 2024-09-15T23:28:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f8562cd7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1acfb0 con 0x55f941b69480 2024-09-15T23:28:25.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f8562cd7740 10 client.0 _ll_register_callbacks cb 0x7f8554009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f855ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b77430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f855ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b77430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38814/0 (socket says 172.21.15.89:38814) 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f855ad9d640 1 -- 172.21.15.89:0/1892550043 learned_addr learned my addr 172.21.15.89:0/1892550043 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f855a59c640 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b7c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f8553fff640 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b69480 0x55f941b7c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f855ad9d640 1 -- 172.21.15.89:0/1892550043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b69480 msgr2=0x55f941b7c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f855ad9d640 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b69480 0x55f941b7c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f855ad9d640 1 -- 172.21.15.89:0/1892550043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 msgr2=0x55f941b7c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f855ad9d640 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b7c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f855ad9d640 1 -- 172.21.15.89:0/1892550043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f941b77c60 con 0x55f941b73d40 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f855ad9d640 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b77430 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8554009cd0 tx=0x7f85540095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901d1162884ecd57 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:25.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.486+0000 7f8541ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8541ffb640 1 -- 172.21.15.89:0/1892550043 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8554004720 con 0x55f941b73d40 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8541ffb640 1 -- 172.21.15.89:0/1892550043 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f85540305a0 con 0x55f941b73d40 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8562cd7740 1 -- 172.21.15.89:0/1892550043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f941ad5d00 con 0x55f941b73d40 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8541ffb640 1 -- 172.21.15.89:0/1892550043 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8554060050 con 0x55f941b73d40 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8562cd7740 10 client.16986 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8562cd7740 1 -- 172.21.15.89:0/1892550043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f941bedcf0 con 0x55f941b73d40 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8541ffb640 1 -- 172.21.15.89:0/1892550043 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f941ad5d00 con 0x55f941b73d40 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8541ffb640 1 -- 172.21.15.89:0/1892550043 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8554030070 con 0x55f941b73d40 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8562cd7740 10 client.16986 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8562cd7740 10 client.16986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8562cd7740 20 client.16986 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8562cd7740 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f941bee4c0 0x55f941bf08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:25.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8562cd7740 4 client.16986 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.487+0000 7f8562cd7740 1 -- 172.21.15.89:0/1892550043 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f941a52110 con 0x55f941bee4c0 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.488+0000 7f855a59c640 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f941bee4c0 0x55f941bf08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.488+0000 7f8541ffb640 10 client.16986 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.488+0000 7f855a59c640 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f941bee4c0 0x55f941bf08e0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=379f279085411d9f server_cookie=445d6bfc11447791 in_seq=0 out_seq=0 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8541ffb640 1 -- 172.21.15.89:0/1892550043 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f941a52110 con 0x55f941bee4c0 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8541ffb640 10 client.16986 handle_command_reply: tid=0 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 client.16986 shutdown 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 -- 172.21.15.89:0/1892550043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 msgr2=0x55f941b77430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b77430 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f8554009cd0 tx=0x7f85540095c0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 -- 172.21.15.89:0/1892550043 shutdown_connections 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f941bee4c0 0x55f941bf08e0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f941b73d40 0x55f941b77430 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f941b69480 0x55f941b7c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8541ffb640 0 client.16986 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 --2- 172.21.15.89:0/1892550043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f941b68b60 0x55f941b7c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 -- 172.21.15.89:0/1892550043 >> 172.21.15.89:0/1892550043 conn(0x55f941b650f0 msgr2=0x55f941b6fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 -- 172.21.15.89:0/1892550043 shutdown_connections 2024-09-15T23:28:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 1 -- 172.21.15.89:0/1892550043 wait complete. 2024-09-15T23:28:25.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.489+0000 7f8562cd7740 20 client.16986 trim_cache size 0 max 16384 2024-09-15T23:28:25.497 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:28:25.497 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:28:25.497 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "36c9922c-2cca-4ab9-902d-f1352ecb6542", 2024-09-15T23:28:25.497 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:28:25.498 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:28:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 -- 172.21.15.89:0/1989520273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85480a3240 msgr2=0x7f85480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85480a3240 0x7f85480a5700 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8554009430 tx=0x7f8554005c50 comp rx=0 tx=0).stop 2024-09-15T23:28:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 -- 172.21.15.89:0/1989520273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f855c100a80 msgr2=0x7f855c198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f855c100a80 0x7f855c198550 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f854c00b4a0 tx=0x7f854c00b960 comp rx=0 tx=0).stop 2024-09-15T23:28:25.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 -- 172.21.15.89:0/1989520273 shutdown_connections 2024-09-15T23:28:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f85480a3240 0x7f85480a5700 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f855c106a80 0x7f855c19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f855c101420 0x7f855c198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 --2- 172.21.15.89:0/1989520273 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f855c100a80 0x7f855c198550 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 -- 172.21.15.89:0/1989520273 >> 172.21.15.89:0/1989520273 conn(0x7f855c0fc7a0 msgr2=0x7f855c1048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 -- 172.21.15.89:0/1989520273 shutdown_connections 2024-09-15T23:28:25.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:25.490+0000 7f85608fe640 1 -- 172.21.15.89:0/1989520273 wait complete. 2024-09-15T23:28:26.368 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:26 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:28:26.368 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:26 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:26.368 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:26 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:28:26.368 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:26 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:26.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:28:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:26 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:28:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:26 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:26 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:28:26.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:26 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:26 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:28:26.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:26 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:26.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:26 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:28:26.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:26 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:26.616 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.095+0000 7f998252a640 1 -- 172.21.15.89:0/688192307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c106f20 msgr2=0x7f997c10eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.095+0000 7f998252a640 1 --2- 172.21.15.89:0/688192307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c106f20 0x7f997c10eaa0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9974009ca0 tx=0x7f997402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.095+0000 7f998252a640 1 -- 172.21.15.89:0/688192307 shutdown_connections 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.095+0000 7f998252a640 1 --2- 172.21.15.89:0/688192307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c106f20 0x7f997c10eaa0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.095+0000 7f998252a640 1 --2- 172.21.15.89:0/688192307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997c1027a0 0x7f997c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.095+0000 7f998252a640 1 --2- 172.21.15.89:0/688192307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f997c101df0 0x7f997c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.095+0000 7f998252a640 1 -- 172.21.15.89:0/688192307 >> 172.21.15.89:0/688192307 conn(0x7f997c0fd8b0 msgr2=0x7f997c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.095+0000 7f998252a640 1 -- 172.21.15.89:0/688192307 shutdown_connections 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.095+0000 7f998252a640 1 -- 172.21.15.89:0/688192307 wait complete. 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f998252a640 1 Processor -- start 2024-09-15T23:28:27.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f998252a640 1 -- start start 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f998252a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f997c101df0 0x7f997c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f998252a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c1027a0 0x7f997c198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f998252a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997c106f20 0x7f997c19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f998252a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f997c112020 con 0x7f997c1027a0 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f998252a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f997c111ea0 con 0x7f997c106f20 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f998252a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1121a0 con 0x7f997c101df0 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c1027a0 0x7f997c198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c1027a0 0x7f997c198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38832/0 (socket says 172.21.15.89:38832) 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.096+0000 7f997b7fe640 1 -- 172.21.15.89:0/85918751 learned_addr learned my addr 172.21.15.89:0/85918751 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f997bfff640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f997c101df0 0x7f997c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f997b7fe640 1 -- 172.21.15.89:0/85918751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f997c101df0 msgr2=0x7f997c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f997b7fe640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f997c101df0 0x7f997c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f997b7fe640 1 -- 172.21.15.89:0/85918751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997c106f20 msgr2=0x7f997c19cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:27.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f9980ac7640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997c106f20 0x7f997c19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f997b7fe640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997c106f20 0x7f997c19cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f997b7fe640 1 -- 172.21.15.89:0/85918751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c19d4b0 con 0x7f997c1027a0 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f997bfff640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f997c101df0 0x7f997c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f997b7fe640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c1027a0 0x7f997c198a40 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f997000b4a0 tx=0x7f997000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f99797fa640 1 -- 172.21.15.89:0/85918751 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9970014070 con 0x7f997c1027a0 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f99797fa640 1 -- 172.21.15.89:0/85918751 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9970010af0 con 0x7f997c1027a0 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f998252a640 1 -- 172.21.15.89:0/85918751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f997c19d7a0 con 0x7f997c1027a0 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f99797fa640 1 -- 172.21.15.89:0/85918751 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9970017d10 con 0x7f997c1027a0 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.097+0000 7f998252a640 1 -- 172.21.15.89:0/85918751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f997c1a5080 con 0x7f997c1027a0 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.099+0000 7f99797fa640 1 -- 172.21.15.89:0/85918751 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9970017070 con 0x7f997c1027a0 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f9984903740 1 Processor -- start 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f9984903740 1 -- start start 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f9984903740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf04a0 0x55de9bcf0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f99797fa640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99640a3240 0x7f99640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f9984903740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf0dc0 0x55de9bcfb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f9984903740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 0x55de9bcfda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f9984903740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55de9bc562e0 con 0x55de9bcf0dc0 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f9984903740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55de9bbfdce0 con 0x55de9bcfb680 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f9984903740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55de9bc522a0 con 0x55de9bcf04a0 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf0dc0 0x55de9bcfb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf0dc0 0x55de9bcfb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38838/0 (socket says 172.21.15.89:38838) 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f997b7fe640 1 -- 172.21.15.89:0/699255654 learned_addr learned my addr 172.21.15.89:0/699255654 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f997bfff640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99640a3240 0x7f99640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f99797fa640 1 -- 172.21.15.89:0/85918751 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f99700cf180 con 0x7f997c1027a0 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f9980ac7640 1 --2- 172.21.15.89:0/699255654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf04a0 0x55de9bcf0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f9980ac7640 1 --2- 172.21.15.89:0/699255654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 0x55de9bcfda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f997b7fe640 1 -- 172.21.15.89:0/699255654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf04a0 msgr2=0x55de9bcf0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:27.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f997b7fe640 1 --2- 172.21.15.89:0/699255654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf04a0 0x55de9bcf0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f997b7fe640 1 -- 172.21.15.89:0/699255654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 msgr2=0x55de9bcfda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f997b7fe640 1 --2- 172.21.15.89:0/699255654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 0x55de9bcfda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.100+0000 7f997b7fe640 1 -- 172.21.15.89:0/699255654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de9bbe2440 con 0x55de9bcf0dc0 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f997b7fe640 1 --2- 172.21.15.89:0/699255654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf0dc0 0x55de9bcfb140 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f9970009420 tx=0x7f9970013500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72eac027583493e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f997bfff640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99640a3240 0x7f99640a5700 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f996c006fd0 tx=0x7f996c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f99627fc640 1 -- 172.21.15.89:0/699255654 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997001e070 con 0x55de9bcf0dc0 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f99627fc640 1 -- 172.21.15.89:0/699255654 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99700a4ee0 con 0x55de9bcf0dc0 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f99627fc640 1 -- 172.21.15.89:0/699255654 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997009c910 con 0x55de9bcf0dc0 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f9984903740 1 -- 172.21.15.89:0/699255654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf0dc0 msgr2=0x55de9bcfb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f9984903740 1 --2- 172.21.15.89:0/699255654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf0dc0 0x55de9bcfb140 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f9970009420 tx=0x7f9970013500 comp rx=0 tx=0).stop 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f9984903740 1 -- 172.21.15.89:0/699255654 shutdown_connections 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f9984903740 1 --2- 172.21.15.89:0/699255654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 0x55de9bcfda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f9984903740 1 --2- 172.21.15.89:0/699255654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf0dc0 0x55de9bcfb140 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f9984903740 1 --2- 172.21.15.89:0/699255654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf04a0 0x55de9bcf0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f9984903740 1 -- 172.21.15.89:0/699255654 >> 172.21.15.89:0/699255654 conn(0x55de9bceca30 msgr2=0x55de9bcece10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:27.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f9984903740 1 -- 172.21.15.89:0/699255654 shutdown_connections 2024-09-15T23:28:27.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.101+0000 7f9984903740 1 -- 172.21.15.89:0/699255654 wait complete. 2024-09-15T23:28:27.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f9984903740 1 -- start start 2024-09-15T23:28:27.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f9984903740 1 Processor -- start 2024-09-15T23:28:27.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f9984903740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bcfaf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f9984903740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf0dc0 0x55de9bcf8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f9984903740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 0x55de9bcf8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f9984903740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55de9bcfe250 con 0x55de9bcf04a0 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f9984903740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55de9bc11050 con 0x55de9bcfb680 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f9984903740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55de9bcfe3d0 con 0x55de9bcf0dc0 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f9984903740 10 client.0 _ll_register_callbacks cb 0x55de9bc51160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f997bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bcfaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.102+0000 7f997bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bcfaf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38852/0 (socket says 172.21.15.89:38852) 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f997bfff640 1 -- 172.21.15.89:0/4207655052 learned_addr learned my addr 172.21.15.89:0/4207655052 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f997b7fe640 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf0dc0 0x55de9bcf8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f9980ac7640 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 0x55de9bcf8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f997bfff640 1 -- 172.21.15.89:0/4207655052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf0dc0 msgr2=0x55de9bcf8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f997bfff640 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf0dc0 0x55de9bcf8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f997bfff640 1 -- 172.21.15.89:0/4207655052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 msgr2=0x55de9bcf8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f997bfff640 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 0x55de9bcf8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f997bfff640 1 -- 172.21.15.89:0/4207655052 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de9b9c73c0 con 0x55de9bcf04a0 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f9980ac7640 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 0x55de9bcf8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f997b7fe640 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf0dc0 0x55de9bcf8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f997bfff640 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bcfaf90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f996c00ced0 tx=0x7f996c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda18e181c6cbe5a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f995b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f995b7fe640 1 -- 172.21.15.89:0/4207655052 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c023060 con 0x55de9bcf04a0 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f995b7fe640 1 -- 172.21.15.89:0/4207655052 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f996c0117e0 con 0x55de9bcf04a0 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f9984903740 1 -- 172.21.15.89:0/4207655052 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99700a4ee0 con 0x55de9bcf04a0 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f995b7fe640 1 -- 172.21.15.89:0/4207655052 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f996c02a050 con 0x55de9bcf04a0 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f9984903740 10 client.16998 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.103+0000 7f9984903740 1 -- 172.21.15.89:0/4207655052 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55de9bd09c30 con 0x55de9bcf04a0 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.104+0000 7f995b7fe640 1 -- 172.21.15.89:0/4207655052 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99700a4ee0 con 0x55de9bcf04a0 2024-09-15T23:28:27.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.104+0000 7f995b7fe640 1 -- 172.21.15.89:0/4207655052 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f996c01d9f0 con 0x55de9bcf04a0 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.104+0000 7f9984903740 10 client.16998 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.104+0000 7f9984903740 10 client.16998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.104+0000 7f9984903740 20 client.16998 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.104+0000 7f9984903740 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55de9bd0b0f0 0x55de9bd0d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.104+0000 7f9984903740 4 client.16998 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.104+0000 7f9984903740 1 -- 172.21.15.89:0/4207655052 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de9bc63850 con 0x55de9bd0b0f0 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.104+0000 7f997b7fe640 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55de9bd0b0f0 0x55de9bd0d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.105+0000 7f995b7fe640 10 client.16998 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.105+0000 7f997b7fe640 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55de9bd0b0f0 0x55de9bd0d560 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4db421f7a374ce58 server_cookie=87ad981d0d072619 in_seq=0 out_seq=0 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.105+0000 7f995b7fe640 1 -- 172.21.15.89:0/4207655052 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55de9bc63850 con 0x55de9bd0b0f0 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.105+0000 7f995b7fe640 10 client.16998 handle_command_reply: tid=0 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.105+0000 7f9984903740 1 client.16998 shutdown 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.105+0000 7f9984903740 1 -- 172.21.15.89:0/4207655052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 msgr2=0x55de9bcfaf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:27.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.105+0000 7f9984903740 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bcfaf90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f996c00ced0 tx=0x7f996c0047d0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.106+0000 7f9984903740 1 -- 172.21.15.89:0/4207655052 shutdown_connections 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.106+0000 7f9984903740 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55de9bd0b0f0 0x55de9bd0d560 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.106+0000 7f9984903740 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcfb680 0x55de9bcf8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.106+0000 7f995b7fe640 0 client.16998 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.106+0000 7f9984903740 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf0dc0 0x55de9bcf8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.106+0000 7f9984903740 1 --2- 172.21.15.89:0/4207655052 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bcfaf90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.106+0000 7f9984903740 1 -- 172.21.15.89:0/4207655052 >> 172.21.15.89:0/4207655052 conn(0x55de9bceca30 msgr2=0x55de9bcf2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.106+0000 7f9984903740 1 -- 172.21.15.89:0/4207655052 shutdown_connections 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.106+0000 7f9984903740 1 -- 172.21.15.89:0/4207655052 wait complete. 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.106+0000 7f9984903740 20 client.16998 trim_cache size 0 max 16384 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f9984903740 1 Processor -- start 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f9984903740 1 -- start start 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f9984903740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bceca30 0x55de9bcfa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f9984903740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bd12e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f9984903740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf0dc0 0x55de9bd13370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f9984903740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1aefb0 con 0x55de9bcf04a0 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f9984903740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1b0af0 con 0x55de9bceca30 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f9984903740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1af680 con 0x55de9bcf0dc0 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bd12e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f997b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bd12e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38856/0 (socket says 172.21.15.89:38856) 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f997b7fe640 1 -- 172.21.15.89:0/4166892974 learned_addr learned my addr 172.21.15.89:0/4166892974 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f997b7fe640 1 -- 172.21.15.89:0/4166892974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf0dc0 msgr2=0x55de9bd13370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f997b7fe640 1 --2- 172.21.15.89:0/4166892974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf0dc0 0x55de9bd13370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f997b7fe640 1 -- 172.21.15.89:0/4166892974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bceca30 msgr2=0x55de9bcfa1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f997b7fe640 1 --2- 172.21.15.89:0/4166892974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bceca30 0x55de9bcfa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f997b7fe640 1 -- 172.21.15.89:0/4166892974 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de9bcfb300 con 0x55de9bcf04a0 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f997b7fe640 1 --2- 172.21.15.89:0/4166892974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bd12e30 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f99700cc8e0 tx=0x7f99700ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17db0c35fc9183bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.123+0000 7f9961ffb640 1 -- 172.21.15.89:0/4166892974 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997001e040 con 0x55de9bcf04a0 2024-09-15T23:28:27.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9961ffb640 1 -- 172.21.15.89:0/4166892974 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f997009bab0 con 0x55de9bcf04a0 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9961ffb640 1 -- 172.21.15.89:0/4166892974 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9970010ca0 con 0x55de9bcf04a0 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9984903740 1 -- 172.21.15.89:0/4166892974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 msgr2=0x55de9bd12e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9984903740 1 --2- 172.21.15.89:0/4166892974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bd12e30 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f99700cc8e0 tx=0x7f99700ce040 comp rx=0 tx=0).stop 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9984903740 1 -- 172.21.15.89:0/4166892974 shutdown_connections 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9984903740 1 --2- 172.21.15.89:0/4166892974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf0dc0 0x55de9bd13370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9984903740 1 --2- 172.21.15.89:0/4166892974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bcf04a0 0x55de9bd12e30 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9984903740 1 --2- 172.21.15.89:0/4166892974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bceca30 0x55de9bcfa1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9984903740 1 -- 172.21.15.89:0/4166892974 >> 172.21.15.89:0/4166892974 conn(0x55de9bcf88e0 msgr2=0x55de9bcf9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9984903740 1 -- 172.21.15.89:0/4166892974 shutdown_connections 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.124+0000 7f9984903740 1 -- 172.21.15.89:0/4166892974 wait complete. 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f9984903740 1 -- start start 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f9984903740 1 Processor -- start 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f9984903740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bceca30 0x55de9bd03e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f9984903740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf04a0 0x55de9bd01190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f9984903740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcf0dc0 0x55de9bd01730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f9984903740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1b0420 con 0x55de9bceca30 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f9984903740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1afd50 con 0x55de9bcf0dc0 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f9984903740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1acf90 con 0x55de9bcf04a0 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f9984903740 10 client.0 _ll_register_callbacks cb 0x7f996c00ae40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f997bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bceca30 0x55de9bd03e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f997bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bceca30 0x55de9bd03e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38870/0 (socket says 172.21.15.89:38870) 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f997bfff640 1 -- 172.21.15.89:0/189021457 learned_addr learned my addr 172.21.15.89:0/189021457 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f9980ac7640 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcf0dc0 0x55de9bd01730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f997bfff640 1 -- 172.21.15.89:0/189021457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf04a0 msgr2=0x55de9bd01190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f997bfff640 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf04a0 0x55de9bd01190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f997bfff640 1 -- 172.21.15.89:0/189021457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcf0dc0 msgr2=0x55de9bd01730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f997bfff640 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcf0dc0 0x55de9bd01730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.125+0000 7f997bfff640 1 -- 172.21.15.89:0/189021457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de9bd01f60 con 0x55de9bceca30 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f997bfff640 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bceca30 0x55de9bd03e40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f996c000c00 tx=0x7f996c01d6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd384535bdd6812 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f995b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f995b7fe640 1 -- 172.21.15.89:0/189021457 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c032020 con 0x55de9bceca30 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f995b7fe640 1 -- 172.21.15.89:0/189021457 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f996c0115e0 con 0x55de9bceca30 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f9984903740 1 -- 172.21.15.89:0/189021457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f997009bab0 con 0x55de9bceca30 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f995b7fe640 1 -- 172.21.15.89:0/189021457 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f996c023020 con 0x55de9bceca30 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f9984903740 10 client.17004 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f9984903740 1 -- 172.21.15.89:0/189021457 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55de9bd75f20 con 0x55de9bceca30 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f995b7fe640 1 -- 172.21.15.89:0/189021457 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f997009bab0 con 0x55de9bceca30 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f995b7fe640 1 -- 172.21.15.89:0/189021457 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f996c02e4c0 con 0x55de9bceca30 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f9984903740 10 client.17004 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f9984903740 10 client.17004 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f9984903740 20 client.17004 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:27.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f9984903740 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55de9bcfb680 0x55de9bd78760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:27.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f9984903740 4 client.17004 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:28:27.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f9984903740 1 -- 172.21.15.89:0/189021457 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55de9bc5dd60 con 0x55de9bcfb680 2024-09-15T23:28:27.150 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:28:27.150 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:28:27.150 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:28:27.150 INFO:teuthology.orchestra.run.smithi089.stdout: "36c9922c-2cca-4ab9-902d-f1352ecb6542": { 2024-09-15T23:28:27.150 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:28:27.150 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "36c9922c-2cca-4ab9-902d-f1352ecb6542", 2024-09-15T23:28:27.150 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:28:27.151 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:27.151 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:27.151 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:28:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.126+0000 7f997b7fe640 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55de9bcfb680 0x55de9bd78760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f997b7fe640 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55de9bcfb680 0x55de9bd78760 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6c856b53a4b6e2e server_cookie=7a1c2d08b732651a in_seq=0 out_seq=0 2024-09-15T23:28:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f995b7fe640 10 client.17004 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f995b7fe640 1 -- 172.21.15.89:0/189021457 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55de9bc5dd60 con 0x55de9bcfb680 2024-09-15T23:28:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f995b7fe640 10 client.17004 handle_command_reply: tid=0 2024-09-15T23:28:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 client.17004 shutdown 2024-09-15T23:28:27.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 -- 172.21.15.89:0/189021457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bceca30 msgr2=0x55de9bd03e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bceca30 0x55de9bd03e40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f996c000c00 tx=0x7f996c01d6d0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 -- 172.21.15.89:0/189021457 shutdown_connections 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55de9bcfb680 0x55de9bd78760 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de9bcf0dc0 0x55de9bd01730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de9bcf04a0 0x55de9bd01190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f995b7fe640 0 client.17004 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 --2- 172.21.15.89:0/189021457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de9bceca30 0x55de9bd03e40 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 -- 172.21.15.89:0/189021457 >> 172.21.15.89:0/189021457 conn(0x55de9bcf88e0 msgr2=0x55de9bceb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 -- 172.21.15.89:0/189021457 shutdown_connections 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.127+0000 7f9984903740 1 -- 172.21.15.89:0/189021457 wait complete. 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f9984903740 20 client.17004 trim_cache size 0 max 16384 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 -- 172.21.15.89:0/85918751 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99640a3240 msgr2=0x7f99640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:27.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99640a3240 0x7f99640a5700 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f996c006fd0 tx=0x7f996c008040 comp rx=0 tx=0).stop 2024-09-15T23:28:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 -- 172.21.15.89:0/85918751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c1027a0 msgr2=0x7f997c198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c1027a0 0x7f997c198a40 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f997000b4a0 tx=0x7f997000b960 comp rx=0 tx=0).stop 2024-09-15T23:28:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 -- 172.21.15.89:0/85918751 shutdown_connections 2024-09-15T23:28:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99640a3240 0x7f99640a5700 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f997c106f20 0x7f997c19cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c1027a0 0x7f997c198a40 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 --2- 172.21.15.89:0/85918751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f997c101df0 0x7f997c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 -- 172.21.15.89:0/85918751 >> 172.21.15.89:0/85918751 conn(0x7f997c0fd8b0 msgr2=0x7f997c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 -- 172.21.15.89:0/85918751 shutdown_connections 2024-09-15T23:28:27.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:27.128+0000 7f998252a640 1 -- 172.21.15.89:0/85918751 wait complete. 2024-09-15T23:28:27.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:27 smithi089 ceph-mon[30933]: pgmap v483: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 23 op/s 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:64/600: open fd = 3 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:65/600: open fd = 3 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:66/600: open fd = 3 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.446 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.447 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.447 INFO:tasks.workunit.client.0.smithi089.stdout:67/600: open fd = 3 2024-09-15T23:28:27.447 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.447 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.447 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:68/600: open fd = 3 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:69/600: open fd = 3 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:70/600: open fd = 3 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.448 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:71/600: open fd = 3 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:72/600: open fd = 3 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:73/600: open fd = 3 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:74/600: open fd = 3 2024-09-15T23:28:27.449 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:75/600: open fd = 3 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:76/600: open fd = 3 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:77/600: open fd = 3 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.450 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.451 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.451 INFO:tasks.workunit.client.0.smithi089.stdout:78/600: open fd = 3 2024-09-15T23:28:27.451 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.451 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.451 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.451 INFO:tasks.workunit.client.0.smithi089.stdout:79/600: open fd = 3 2024-09-15T23:28:27.451 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.451 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.452 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.452 INFO:tasks.workunit.client.0.smithi089.stdout:80/600: open fd = 3 2024-09-15T23:28:27.452 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.452 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.452 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.452 INFO:tasks.workunit.client.0.smithi089.stdout:81/600: open fd = 3 2024-09-15T23:28:27.452 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.452 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.452 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:82/600: open fd = 3 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:83/600: open fd = 3 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:84/600: open fd = 3 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.453 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:85/600: open fd = 3 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:86/600: open fd = 3 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:87/600: open fd = 3 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:88/600: open fd = 3 2024-09-15T23:28:27.454 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:89/600: open fd = 3 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:90/600: open fd = 3 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.455 INFO:tasks.workunit.client.0.smithi089.stdout:91/600: open fd = 3 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:92/600: open fd = 3 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:93/600: open fd = 3 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:94/600: open fd = 3 2024-09-15T23:28:27.456 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.457 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.457 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.457 INFO:tasks.workunit.client.0.smithi089.stdout:95/600: open fd = 3 2024-09-15T23:28:27.457 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.457 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.457 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:96/600: open fd = 3 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:97/600: open fd = 3 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:98/600: open fd = 3 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.458 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:99/600: open fd = 3 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:100/600: open fd = 3 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:101/600: open fd = 3 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.459 INFO:tasks.workunit.client.0.smithi089.stdout:102/600: open fd = 3 2024-09-15T23:28:27.460 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.460 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.460 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.460 INFO:tasks.workunit.client.0.smithi089.stdout:103/600: open fd = 3 2024-09-15T23:28:27.460 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.460 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.460 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.460 INFO:tasks.workunit.client.0.smithi089.stdout:104/600: open fd = 3 2024-09-15T23:28:27.460 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.460 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:105/600: open fd = 3 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:106/600: open fd = 3 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:107/600: open fd = 3 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.461 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.462 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.462 INFO:tasks.workunit.client.0.smithi089.stdout:108/600: open fd = 3 2024-09-15T23:28:27.462 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.462 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.462 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.462 INFO:tasks.workunit.client.0.smithi089.stdout:109/600: open fd = 3 2024-09-15T23:28:27.462 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.462 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.462 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.462 INFO:tasks.workunit.client.0.smithi089.stdout:110/600: open fd = 3 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:111/600: open fd = 3 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:112/600: open fd = 3 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.463 INFO:tasks.workunit.client.0.smithi089.stdout:113/600: open fd = 3 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:114/600: open fd = 3 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:115/600: open fd = 3 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.464 INFO:tasks.workunit.client.0.smithi089.stdout:116/600: open fd = 3 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:117/600: open fd = 3 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:118/600: open fd = 3 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:119/600: open fd = 3 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.466 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.466 INFO:tasks.workunit.client.0.smithi089.stdout:120/600: open fd = 3 2024-09-15T23:28:27.466 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.466 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.466 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.466 INFO:tasks.workunit.client.0.smithi089.stdout:121/600: open fd = 3 2024-09-15T23:28:27.466 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.467 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.467 INFO:tasks.workunit.client.0.smithi089.stdout:122/600: open fd = 3 2024-09-15T23:28:27.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.467 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.467 INFO:tasks.workunit.client.0.smithi089.stdout:123/600: open fd = 3 2024-09-15T23:28:27.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.468 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.468 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.469 INFO:tasks.workunit.client.0.smithi089.stdout:124/600: open fd = 3 2024-09-15T23:28:27.469 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.469 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.469 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.469 INFO:tasks.workunit.client.0.smithi089.stdout:125/600: open fd = 3 2024-09-15T23:28:27.469 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.469 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.469 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:28:27.469 INFO:tasks.workunit.client.0.smithi089.stdout:126/600: open fd = 3 2024-09-15T23:28:27.469 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:28:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:27 smithi120 ceph-mon[35250]: pgmap v483: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 23 op/s 2024-09-15T23:28:27.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:27 smithi174 ceph-mon[35586]: pgmap v483: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 23 op/s 2024-09-15T23:28:27.921 INFO:tasks.cephfs.filesystem:scrub status for tag:36c9922c-2cca-4ab9-902d-f1352ecb6542 - {'path': '/', 'tag': '36c9922c-2cca-4ab9-902d-f1352ecb6542', 'options': 'recursive,force'} 2024-09-15T23:28:28.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:28 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:28:28.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:28 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:28:28.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:28 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:28:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:29 smithi120 ceph-mon[35250]: pgmap v484: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:28:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:29 smithi174 ceph-mon[35586]: pgmap v484: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:28:29.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:29 smithi089 ceph-mon[30933]: pgmap v484: 129 pgs: 129 active+clean; 2.2 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:28:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:31 smithi120 ceph-mon[35250]: pgmap v485: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 22 op/s 2024-09-15T23:28:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:31 smithi174 ceph-mon[35586]: pgmap v485: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 22 op/s 2024-09-15T23:28:31.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:31 smithi089 ceph-mon[30933]: pgmap v485: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 22 op/s 2024-09-15T23:28:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:33 smithi120 ceph-mon[35250]: pgmap v486: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:28:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:33 smithi174 ceph-mon[35586]: pgmap v486: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:28:33.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:33 smithi089 ceph-mon[30933]: pgmap v486: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:28:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:35 smithi120 ceph-mon[35250]: pgmap v487: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:28:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:35 smithi174 ceph-mon[35586]: pgmap v487: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:28:35.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:35 smithi089 ceph-mon[30933]: pgmap v487: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:28:37.364 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 13.8 2024-09-15T23:28:37.365 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:28:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:37 smithi120 ceph-mon[35250]: pgmap v488: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 23 op/s 2024-09-15T23:28:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:37 smithi174 ceph-mon[35586]: pgmap v488: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 23 op/s 2024-09-15T23:28:37.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:37 smithi089 ceph-mon[30933]: pgmap v488: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 23 op/s 2024-09-15T23:28:37.625 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.649+0000 7f11e28b7640 1 -- 172.21.15.89:0/4239768620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11dc100a50 msgr2=0x7f11dc100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.649+0000 7f11e28b7640 1 --2- 172.21.15.89:0/4239768620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11dc100a50 0x7f11dc100e50 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009900 tx=0x7f11cc02a410 comp rx=0 tx=0).stop 2024-09-15T23:28:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.649+0000 7f11e28b7640 1 -- 172.21.15.89:0/4239768620 shutdown_connections 2024-09-15T23:28:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.649+0000 7f11e28b7640 1 --2- 172.21.15.89:0/4239768620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11dc102600 0x7f11dc108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.649+0000 7f11e28b7640 1 --2- 172.21.15.89:0/4239768620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11dc101c40 0x7f11dc1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.649+0000 7f11e28b7640 1 --2- 172.21.15.89:0/4239768620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11dc100a50 0x7f11dc100e50 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.649+0000 7f11e28b7640 1 -- 172.21.15.89:0/4239768620 >> 172.21.15.89:0/4239768620 conn(0x7f11dc0fc1b0 msgr2=0x7f11dc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.649+0000 7f11e28b7640 1 -- 172.21.15.89:0/4239768620 shutdown_connections 2024-09-15T23:28:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.649+0000 7f11e28b7640 1 -- 172.21.15.89:0/4239768620 wait complete. 2024-09-15T23:28:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11e28b7640 1 Processor -- start 2024-09-15T23:28:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11e28b7640 1 -- start start 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11e28b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11dc100a50 0x7f11dc197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11e28b7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11dc101c40 0x7f11dc198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11e28b7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11dc102600 0x7f11dc19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11e28b7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc10c6b0 con 0x7f11dc101c40 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11e28b7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc10c530 con 0x7f11dc102600 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11e28b7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc10c830 con 0x7f11dc100a50 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11dc101c40 0x7f11dc198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11dc101c40 0x7f11dc198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52638/0 (socket says 172.21.15.89:52638) 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11db7fe640 1 -- 172.21.15.89:0/2367920317 learned_addr learned my addr 172.21.15.89:0/2367920317 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.650+0000 7f11dbfff640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11dc100a50 0x7f11dc197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11dc102600 0x7f11dc19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11db7fe640 1 -- 172.21.15.89:0/2367920317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11dc100a50 msgr2=0x7f11dc197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11db7fe640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11dc100a50 0x7f11dc197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11db7fe640 1 -- 172.21.15.89:0/2367920317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11dc102600 msgr2=0x7f11dc19f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11db7fe640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11dc102600 0x7f11dc19f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11db7fe640 1 -- 172.21.15.89:0/2367920317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc19fa70 con 0x7f11dc101c40 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11dc102600 0x7f11dc19f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11db7fe640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11dc101c40 0x7f11dc198390 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f11d000d950 tx=0x7f11d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11d97fa640 1 -- 172.21.15.89:0/2367920317 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0017070 con 0x7f11dc101c40 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11d97fa640 1 -- 172.21.15.89:0/2367920317 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d0009c30 con 0x7f11dc101c40 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11d97fa640 1 -- 172.21.15.89:0/2367920317 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d000f6b0 con 0x7f11dc101c40 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11e28b7640 1 -- 172.21.15.89:0/2367920317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11dc102ec0 con 0x7f11dc101c40 2024-09-15T23:28:37.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.651+0000 7f11e28b7640 1 -- 172.21.15.89:0/2367920317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11dc1033c0 con 0x7f11dc101c40 2024-09-15T23:28:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.654+0000 7f11d97fa640 1 -- 172.21.15.89:0/2367920317 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f11d00109d0 con 0x7f11dc101c40 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.654+0000 7f11e564f740 1 Processor -- start 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.654+0000 7f11e564f740 1 -- start start 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.654+0000 7f11e564f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.654+0000 7f11e564f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7a8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.654+0000 7f11e564f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7ab830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.654+0000 7f11e564f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558c9b7072f0 con 0x558c9b7a9370 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11e564f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558c9b6af270 con 0x558c9b7a39a0 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11e564f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558c9b7032b0 con 0x558c9b7a42e0 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7ab830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11d97fa640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11c40a7780 0x7f11c40a9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7ab830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52648/0 (socket says 172.21.15.89:52648) 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11db7fe640 1 -- 172.21.15.89:0/2135087867 learned_addr learned my addr 172.21.15.89:0/2135087867 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11dbfff640 1 --2- 172.21.15.89:0/2135087867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7a8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11c40a7780 0x7f11c40a9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2135087867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11d97fa640 1 -- 172.21.15.89:0/2367920317 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f11d00cf210 con 0x7f11dc101c40 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11db7fe640 1 -- 172.21.15.89:0/2135087867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 msgr2=0x558c9b7a8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11db7fe640 1 --2- 172.21.15.89:0/2135087867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7a8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11db7fe640 1 -- 172.21.15.89:0/2135087867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 msgr2=0x558c9b7a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11db7fe640 1 --2- 172.21.15.89:0/2135087867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7a3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11db7fe640 1 -- 172.21.15.89:0/2135087867 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c9b6ae340 con 0x558c9b7a9370 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11dbfff640 1 --2- 172.21.15.89:0/2135087867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7a8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11c40a7780 0x7f11c40a9c40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f11d4005fd0 tx=0x7f11d400ac90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11db7fe640 1 --2- 172.21.15.89:0/2135087867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7ab830 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f11d00978a0 tx=0x7f11d000dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26539030d6f12a21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.655+0000 7f11c27fc640 1 -- 172.21.15.89:0/2135087867 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0020070 con 0x558c9b7a9370 2024-09-15T23:28:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11c27fc640 1 -- 172.21.15.89:0/2135087867 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d009bb40 con 0x558c9b7a9370 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11c27fc640 1 -- 172.21.15.89:0/2135087867 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d00a40b0 con 0x558c9b7a9370 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11e564f740 1 -- 172.21.15.89:0/2135087867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 msgr2=0x558c9b7ab830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11e564f740 1 --2- 172.21.15.89:0/2135087867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7ab830 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f11d00978a0 tx=0x7f11d000dfa0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11e564f740 1 -- 172.21.15.89:0/2135087867 shutdown_connections 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11e564f740 1 --2- 172.21.15.89:0/2135087867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7ab830 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11e564f740 1 --2- 172.21.15.89:0/2135087867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7a8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11e564f740 1 --2- 172.21.15.89:0/2135087867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11e564f740 1 -- 172.21.15.89:0/2135087867 >> 172.21.15.89:0/2135087867 conn(0x558c9b79f010 msgr2=0x558c9b79f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11e564f740 1 -- 172.21.15.89:0/2135087867 shutdown_connections 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.656+0000 7f11e564f740 1 -- 172.21.15.89:0/2135087867 wait complete. 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e564f740 1 -- start start 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e564f740 1 Processor -- start 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e564f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e564f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7ae130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e564f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e564f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558c9b7ac340 con 0x558c9b7a9370 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e564f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558c9b690290 con 0x558c9b7a39a0 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e564f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558c9b7ac4c0 con 0x558c9b7a42e0 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e564f740 10 client.0 _ll_register_callbacks cb 0x558c9b6c1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e0e54640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e0e54640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52662/0 (socket says 172.21.15.89:52662) 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e0e54640 1 -- 172.21.15.89:0/2417227609 learned_addr learned my addr 172.21.15.89:0/2417227609 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11dbfff640 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11db7fe640 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e0e54640 1 -- 172.21.15.89:0/2417227609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 msgr2=0x558c9b7ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7ae130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e0e54640 1 -- 172.21.15.89:0/2417227609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 msgr2=0x558c9b7adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.657+0000 7f11e0e54640 1 -- 172.21.15.89:0/2417227609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c9b41a210 con 0x558c9b7a9370 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b5170 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f11d400ab20 tx=0x7f11d4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41cede47e1c0b34 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2417227609 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4011bc0 con 0x558c9b7a9370 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2417227609 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f11d4011d60 con 0x558c9b7a9370 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11e564f740 1 -- 172.21.15.89:0/2417227609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11d009bb40 con 0x558c9b7a9370 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2417227609 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f11d402c580 con 0x558c9b7a9370 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11e564f740 10 client.17016 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11e564f740 1 -- 172.21.15.89:0/2417227609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558c9b4e6620 con 0x558c9b7a9370 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2417227609 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d009bb40 con 0x558c9b7a9370 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2417227609 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f11d401b070 con 0x558c9b7a9370 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11e564f740 10 client.17016 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11e564f740 10 client.17016 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11e564f740 20 client.17016 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11e564f740 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c9b7b9860 0x558c9b7bbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11e564f740 4 client.17016 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.658+0000 7f11e564f740 1 -- 172.21.15.89:0/2417227609 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c9b714a20 con 0x558c9b7b9860 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.659+0000 7f11dbfff640 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c9b7b9860 0x558c9b7bbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.659+0000 7f11dbfff640 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c9b7b9860 0x558c9b7bbcd0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=251a06878adde306 server_cookie=fa2c3b54f32c4a8 in_seq=0 out_seq=0 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.659+0000 7f11bb7fe640 10 client.17016 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2417227609 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558c9b714a20 con 0x558c9b7b9860 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11bb7fe640 10 client.17016 handle_command_reply: tid=0 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 client.17016 shutdown 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 -- 172.21.15.89:0/2417227609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 msgr2=0x558c9b7b5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b5170 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f11d400ab20 tx=0x7f11d4005e80 comp rx=0 tx=0).stop 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 -- 172.21.15.89:0/2417227609 shutdown_connections 2024-09-15T23:28:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c9b7b9860 0x558c9b7bbcd0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b5170 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11bb7fe640 0 client.17016 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 --2- 172.21.15.89:0/2417227609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 -- 172.21.15.89:0/2417227609 >> 172.21.15.89:0/2417227609 conn(0x558c9b79f010 msgr2=0x558c9b7a0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 -- 172.21.15.89:0/2417227609 shutdown_connections 2024-09-15T23:28:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 1 -- 172.21.15.89:0/2417227609 wait complete. 2024-09-15T23:28:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.660+0000 7f11e564f740 20 client.17016 trim_cache size 0 max 16384 2024-09-15T23:28:37.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11e564f740 1 Processor -- start 2024-09-15T23:28:37.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11e564f740 1 -- start start 2024-09-15T23:28:37.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11e564f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a39a0 0x558c9b79e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11e564f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a42e0 0x558c9b7b5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11e564f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a9370 0x558c9b7c05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.679 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11e564f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1a87f0 con 0x558c9b7a42e0 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11e564f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1aa330 con 0x558c9b7a9370 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11e564f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1a8ec0 con 0x558c9b7a39a0 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a42e0 0x558c9b7b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11db7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a42e0 0x558c9b7b5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52666/0 (socket says 172.21.15.89:52666) 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.674+0000 7f11db7fe640 1 -- 172.21.15.89:0/1682645499 learned_addr learned my addr 172.21.15.89:0/1682645499 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11dbfff640 1 --2- 172.21.15.89:0/1682645499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a39a0 0x558c9b79e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11e0e54640 1 --2- 172.21.15.89:0/1682645499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a9370 0x558c9b7c05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11db7fe640 1 -- 172.21.15.89:0/1682645499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a39a0 msgr2=0x558c9b79e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11db7fe640 1 --2- 172.21.15.89:0/1682645499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a39a0 0x558c9b79e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11db7fe640 1 -- 172.21.15.89:0/1682645499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a9370 msgr2=0x558c9b7c05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11db7fe640 1 --2- 172.21.15.89:0/1682645499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a9370 0x558c9b7c05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11db7fe640 1 -- 172.21.15.89:0/1682645499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c9b7992f0 con 0x558c9b7a42e0 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11e0e54640 1 --2- 172.21.15.89:0/1682645499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a9370 0x558c9b7c05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11db7fe640 1 --2- 172.21.15.89:0/1682645499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a42e0 0x558c9b7b5130 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f11d00ccc90 tx=0x7f11d0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da022a2a203dc92 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11c1ffb640 1 -- 172.21.15.89:0/1682645499 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0020050 con 0x558c9b7a42e0 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11c1ffb640 1 -- 172.21.15.89:0/1682645499 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d009bb40 con 0x558c9b7a42e0 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.675+0000 7f11c1ffb640 1 -- 172.21.15.89:0/1682645499 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d009a890 con 0x558c9b7a42e0 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 -- 172.21.15.89:0/1682645499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a42e0 msgr2=0x558c9b7b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 --2- 172.21.15.89:0/1682645499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a42e0 0x558c9b7b5130 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f11d00ccc90 tx=0x7f11d0000bc0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 -- 172.21.15.89:0/1682645499 shutdown_connections 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 --2- 172.21.15.89:0/1682645499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a9370 0x558c9b7c05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 --2- 172.21.15.89:0/1682645499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a42e0 0x558c9b7b5130 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 --2- 172.21.15.89:0/1682645499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a39a0 0x558c9b79e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 -- 172.21.15.89:0/1682645499 >> 172.21.15.89:0/1682645499 conn(0x558c9b79f010 msgr2=0x558c9b79d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 -- 172.21.15.89:0/1682645499 shutdown_connections 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 -- 172.21.15.89:0/1682645499 wait complete. 2024-09-15T23:28:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 -- start start 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 Processor -- start 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1a9c60 con 0x558c9b7a9370 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1a9590 con 0x558c9b7a39a0 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc1a67d0 con 0x558c9b7a42e0 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e564f740 10 client.0 _ll_register_callbacks cb 0x7f11d4012270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e0e54640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e0e54640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52674/0 (socket says 172.21.15.89:52674) 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.676+0000 7f11e0e54640 1 -- 172.21.15.89:0/2410937395 learned_addr learned my addr 172.21.15.89:0/2410937395 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11db7fe640 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.682 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11dbfff640 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e0e54640 1 -- 172.21.15.89:0/2410937395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 msgr2=0x558c9b7b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7b35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e0e54640 1 -- 172.21.15.89:0/2410937395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 msgr2=0x558c9b7b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7b30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e0e54640 1 -- 172.21.15.89:0/2410937395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c9b7b4c10 con 0x558c9b7a9370 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e0e54640 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b4600 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f11d400ab20 tx=0x7f11d402c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1131e10d7ecdb46 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2410937395 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d403b060 con 0x558c9b7a9370 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2410937395 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f11d4011a40 con 0x558c9b7a9370 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e564f740 1 -- 172.21.15.89:0/2410937395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11d009bb40 con 0x558c9b7a9370 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2410937395 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f11d4042050 con 0x558c9b7a9370 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e564f740 10 client.17022 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e564f740 1 -- 172.21.15.89:0/2410937395 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558c9b824d60 con 0x558c9b7a9370 2024-09-15T23:28:37.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2410937395 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d009bb40 con 0x558c9b7a9370 2024-09-15T23:28:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2410937395 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f11d401b040 con 0x558c9b7a9370 2024-09-15T23:28:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e564f740 10 client.17022 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e564f740 10 client.17022 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e564f740 20 client.17022 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.677+0000 7f11e564f740 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c9b825740 0x558c9b827b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.678+0000 7f11e564f740 4 client.17022 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:28:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.678+0000 7f11e564f740 1 -- 172.21.15.89:0/2410937395 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c9b7a90f0 con 0x558c9b825740 2024-09-15T23:28:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.678+0000 7f11dbfff640 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c9b825740 0x558c9b827b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.678+0000 7f11bb7fe640 10 client.17022 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:37.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.678+0000 7f11dbfff640 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c9b825740 0x558c9b827b60 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33805b4191d06fc server_cookie=4f60afdcebf244e7 in_seq=0 out_seq=0 2024-09-15T23:28:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11bb7fe640 1 -- 172.21.15.89:0/2410937395 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558c9b7a90f0 con 0x558c9b825740 2024-09-15T23:28:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11bb7fe640 10 client.17022 handle_command_reply: tid=0 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 client.17022 shutdown 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 -- 172.21.15.89:0/2410937395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 msgr2=0x558c9b7b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b4600 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f11d400ab20 tx=0x7f11d402c480 comp rx=0 tx=0).stop 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 -- 172.21.15.89:0/2410937395 shutdown_connections 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558c9b825740 0x558c9b827b60 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558c9b7a9370 0x558c9b7b4600 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558c9b7a42e0 0x558c9b7b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11bb7fe640 0 client.17022 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 --2- 172.21.15.89:0/2410937395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c9b7a39a0 0x558c9b7b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 -- 172.21.15.89:0/2410937395 >> 172.21.15.89:0/2410937395 conn(0x558c9b79f010 msgr2=0x558c9b7a0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 -- 172.21.15.89:0/2410937395 shutdown_connections 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 1 -- 172.21.15.89:0/2410937395 wait complete. 2024-09-15T23:28:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.693+0000 7f11e564f740 20 client.17022 trim_cache size 0 max 16384 2024-09-15T23:28:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:28:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:28:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:28:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:28:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 -- 172.21.15.89:0/2367920317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11c40a7780 msgr2=0x7f11c40a9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11c40a7780 0x7f11c40a9c40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f11d4005fd0 tx=0x7f11d400ac90 comp rx=0 tx=0).stop 2024-09-15T23:28:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 -- 172.21.15.89:0/2367920317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11dc101c40 msgr2=0x7f11dc198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11dc101c40 0x7f11dc198390 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f11d000d950 tx=0x7f11d000de10 comp rx=0 tx=0).stop 2024-09-15T23:28:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 -- 172.21.15.89:0/2367920317 shutdown_connections 2024-09-15T23:28:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f11c40a7780 0x7f11c40a9c40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11dc102600 0x7f11dc19f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11dc101c40 0x7f11dc198390 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 --2- 172.21.15.89:0/2367920317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11dc100a50 0x7f11dc197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 -- 172.21.15.89:0/2367920317 >> 172.21.15.89:0/2367920317 conn(0x7f11dc0fc1b0 msgr2=0x7f11dc06a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 -- 172.21.15.89:0/2367920317 shutdown_connections 2024-09-15T23:28:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:37.694+0000 7f11e28b7640 1 -- 172.21.15.89:0/2367920317 wait complete. 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.106+0000 7f20b5264640 1 -- 172.21.15.89:0/2156345859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b0106f10 msgr2=0x7f20b010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.106+0000 7f20b5264640 1 --2- 172.21.15.89:0/2156345859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b0106f10 0x7f20b010ea90 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009ca0 tx=0x7f20a802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.107+0000 7f20b5264640 1 -- 172.21.15.89:0/2156345859 shutdown_connections 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.107+0000 7f20b5264640 1 --2- 172.21.15.89:0/2156345859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b0106f10 0x7f20b010ea90 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.107+0000 7f20b5264640 1 --2- 172.21.15.89:0/2156345859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b0102790 0x7f20b01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.107+0000 7f20b5264640 1 --2- 172.21.15.89:0/2156345859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20b0101de0 0x7f20b01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.107+0000 7f20b5264640 1 -- 172.21.15.89:0/2156345859 >> 172.21.15.89:0/2156345859 conn(0x7f20b00fd8c0 msgr2=0x7f20b00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.107+0000 7f20b5264640 1 -- 172.21.15.89:0/2156345859 shutdown_connections 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.107+0000 7f20b5264640 1 -- 172.21.15.89:0/2156345859 wait complete. 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.107+0000 7f20b5264640 1 Processor -- start 2024-09-15T23:28:38.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.107+0000 7f20b5264640 1 -- start start 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20b5264640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b0101de0 0x7f20b0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20b5264640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20b0102790 0x7f20b0198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20b5264640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b0106f10 0x7f20b019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20b5264640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f20b0111fe0 con 0x7f20b0106f10 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20b5264640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f20b0111e60 con 0x7f20b0101de0 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20b5264640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f20b0112160 con 0x7f20b0102790 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20af7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b0106f10 0x7f20b019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20af7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b0106f10 0x7f20b019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52706/0 (socket says 172.21.15.89:52706) 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20af7fe640 1 -- 172.21.15.89:0/2667093405 learned_addr learned my addr 172.21.15.89:0/2667093405 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20ae7fc640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20b0102790 0x7f20b0198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20aeffd640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b0101de0 0x7f20b0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20af7fe640 1 -- 172.21.15.89:0/2667093405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20b0102790 msgr2=0x7f20b0198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:38.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20af7fe640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20b0102790 0x7f20b0198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:38.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20af7fe640 1 -- 172.21.15.89:0/2667093405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b0101de0 msgr2=0x7f20b0198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:38.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20af7fe640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b0101de0 0x7f20b0198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:38.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20af7fe640 1 -- 172.21.15.89:0/2667093405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b019d4b0 con 0x7f20b0106f10 2024-09-15T23:28:38.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20af7fe640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b0106f10 0x7f20b019cdd0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f20a8003f40 tx=0x7f20a8003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:38.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f2097fff640 1 -- 172.21.15.89:0/2667093405 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a80318c0 con 0x7f20b0106f10 2024-09-15T23:28:38.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f2097fff640 1 -- 172.21.15.89:0/2667093405 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20a8004520 con 0x7f20b0106f10 2024-09-15T23:28:38.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20b5264640 1 -- 172.21.15.89:0/2667093405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b019d740 con 0x7f20b0106f10 2024-09-15T23:28:38.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f2097fff640 1 -- 172.21.15.89:0/2667093405 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a802fe10 con 0x7f20b0106f10 2024-09-15T23:28:38.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.108+0000 7f20b5264640 1 -- 172.21.15.89:0/2667093405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b01a5080 con 0x7f20b0106f10 2024-09-15T23:28:38.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.109+0000 7f20b5264640 1 -- 172.21.15.89:0/2667093405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20b01021c0 con 0x7f20b0106f10 2024-09-15T23:28:38.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.110+0000 7f2097fff640 1 -- 172.21.15.89:0/2667093405 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f20a80046d0 con 0x7f20b0106f10 2024-09-15T23:28:38.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.111+0000 7f2097fff640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20980a3240 0x7f20980a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:38.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.111+0000 7f20aeffd640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20980a3240 0x7f20980a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:38.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.111+0000 7f2097fff640 1 -- 172.21.15.89:0/2667093405 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f20a80f1380 con 0x7f20b0106f10 2024-09-15T23:28:38.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.111+0000 7f20aeffd640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20980a3240 0x7f20980a5700 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009890 tx=0x7f20a0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:38.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.115+0000 7f2097fff640 1 -- 172.21.15.89:0/2667093405 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f20a80bdcb0 con 0x7f20b0106f10 2024-09-15T23:28:38.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.317+0000 7f20b5264640 1 -- 172.21.15.89:0/2667093405 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20b010a0f0 con 0x7f20b0106f10 2024-09-15T23:28:38.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.318+0000 7f2097fff640 1 -- 172.21.15.89:0/2667093405 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f20a80bd5b0 con 0x7f20b0106f10 2024-09-15T23:28:38.320 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:28:38.321 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:28:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:28:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 -- 172.21.15.89:0/2667093405 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20980a3240 msgr2=0x7f20980a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20980a3240 0x7f20980a5700 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009890 tx=0x7f20a0008040 comp rx=0 tx=0).stop 2024-09-15T23:28:38.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 -- 172.21.15.89:0/2667093405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b0106f10 msgr2=0x7f20b019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b0106f10 0x7f20b019cdd0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f20a8003f40 tx=0x7f20a8003f70 comp rx=0 tx=0).stop 2024-09-15T23:28:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 -- 172.21.15.89:0/2667093405 shutdown_connections 2024-09-15T23:28:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20980a3240 0x7f20980a5700 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f20b0106f10 0x7f20b019cdd0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20b0102790 0x7f20b0198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:38.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 --2- 172.21.15.89:0/2667093405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f20b0101de0 0x7f20b0198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:38.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 -- 172.21.15.89:0/2667093405 >> 172.21.15.89:0/2667093405 conn(0x7f20b00fd8c0 msgr2=0x7f20b00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:38.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.320+0000 7f20b5264640 1 -- 172.21.15.89:0/2667093405 shutdown_connections 2024-09-15T23:28:38.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:38.321+0000 7f20b5264640 1 -- 172.21.15.89:0/2667093405 wait complete. 2024-09-15T23:28:38.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:38 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2667093405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:38.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:38 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2667093405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:38.761 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:38 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2667093405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:39.045 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 133.8 --await-for 45.0' 2024-09-15T23:28:39.045 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 133.8 --await-for 45.0 2024-09-15T23:28:39.292 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:39.688 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:39 smithi089 ceph-mon[30933]: pgmap v489: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:28:39.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:39 smithi120 ceph-mon[35250]: pgmap v489: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:28:39.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:39 smithi174 ceph-mon[35586]: pgmap v489: 129 pgs: 129 active+clean; 2.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:28:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.779+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/1919229844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fec106a90 msgr2=0x7f9fec106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.779+0000 7f9ff3bc9640 1 --2- 172.21.15.89:0/1919229844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fec106a90 0x7f9fec106e70 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009930 tx=0x7f9fe402a420 comp rx=0 tx=0).stop 2024-09-15T23:28:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.779+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/1919229844 shutdown_connections 2024-09-15T23:28:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.779+0000 7f9ff3bc9640 1 --2- 172.21.15.89:0/1919229844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec101430 0x7f9fec10d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.779+0000 7f9ff3bc9640 1 --2- 172.21.15.89:0/1919229844 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fec100a90 0x7f9fec100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.779+0000 7f9ff3bc9640 1 --2- 172.21.15.89:0/1919229844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fec106a90 0x7f9fec106e70 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.779+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/1919229844 >> 172.21.15.89:0/1919229844 conn(0x7f9fec0fc7d0 msgr2=0x7f9fec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.779+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/1919229844 shutdown_connections 2024-09-15T23:28:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.779+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/1919229844 wait complete. 2024-09-15T23:28:39.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff3bc9640 1 Processor -- start 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff3bc9640 1 -- start start 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff3bc9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fec100a90 0x7f9fec197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff3bc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec101430 0x7f9fec1978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff3bc9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fec106a90 0x7f9fec19bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff3bc9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec110e90 con 0x7f9fec106a90 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff3bc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec110d10 con 0x7f9fec101430 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff3bc9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec111010 con 0x7f9fec100a90 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff2166640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fec106a90 0x7f9fec19bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff2166640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fec106a90 0x7f9fec19bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52726/0 (socket says 172.21.15.89:52726) 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff2166640 1 -- 172.21.15.89:0/2904914242 learned_addr learned my addr 172.21.15.89:0/2904914242 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff1965640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fec100a90 0x7f9fec197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff1164640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec101430 0x7f9fec1978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff2166640 1 -- 172.21.15.89:0/2904914242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fec100a90 msgr2=0x7f9fec197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff2166640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fec100a90 0x7f9fec197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff2166640 1 -- 172.21.15.89:0/2904914242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec101430 msgr2=0x7f9fec1978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff2166640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec101430 0x7f9fec1978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.780+0000 7f9ff2166640 1 -- 172.21.15.89:0/2904914242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fec19c320 con 0x7f9fec106a90 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.781+0000 7f9ff2166640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fec106a90 0x7f9fec19bc40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00eaa0 tx=0x7f9fdc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.781+0000 7f9feaffd640 1 -- 172.21.15.89:0/2904914242 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc00cdb0 con 0x7f9fec106a90 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.781+0000 7f9feaffd640 1 -- 172.21.15.89:0/2904914242 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fdc005020 con 0x7f9fec106a90 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.781+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/2904914242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fec19c610 con 0x7f9fec106a90 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.781+0000 7f9feaffd640 1 -- 172.21.15.89:0/2904914242 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc0116b0 con 0x7f9fec106a90 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.781+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/2904914242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fec1a3ef0 con 0x7f9fec106a90 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.783+0000 7f9feaffd640 1 -- 172.21.15.89:0/2904914242 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9fdc005560 con 0x7f9fec106a90 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.783+0000 7f9ff5fa2740 1 Processor -- start 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff5fa2740 1 -- start start 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 0x55d5db6d5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6d5b50 0x55d5db6d7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d5db62e2a0 con 0x55d5db6c8640 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d5db5d5b10 con 0x55d5db6c8f60 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d5db62a260 con 0x55d5db6d5b50 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff1164640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff1164640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6c8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52732/0 (socket says 172.21.15.89:52732) 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff1164640 1 -- 172.21.15.89:0/1839774235 learned_addr learned my addr 172.21.15.89:0/1839774235 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9feaffd640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd40a3240 0x7f9fd40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1839774235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6d5b50 0x55d5db6d7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff1965640 1 --2- 172.21.15.89:0/1839774235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 0x55d5db6d5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff2166640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd40a3240 0x7f9fd40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff1164640 1 -- 172.21.15.89:0/1839774235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6d5b50 msgr2=0x55d5db6d7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:39.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1839774235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6d5b50 0x55d5db6d7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff1164640 1 -- 172.21.15.89:0/1839774235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 msgr2=0x55d5db6d5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1839774235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 0x55d5db6d5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.784+0000 7f9ff1164640 1 -- 172.21.15.89:0/1839774235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5db39fb90 con 0x55d5db6c8640 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1839774235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6c8a20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x55d5db6c5de0 tx=0x7f9fe0004160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c2df176e80ecff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9feaffd640 1 -- 172.21.15.89:0/2904914242 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9fdc0cfe40 con 0x7f9fec106a90 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9fd3fff640 1 -- 172.21.15.89:0/1839774235 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0003e40 con 0x55d5db6c8640 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9ff2166640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd40a3240 0x7f9fd40a5700 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc098af0 tx=0x7f9fdc061ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9fd3fff640 1 -- 172.21.15.89:0/1839774235 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0005500 con 0x55d5db6c8640 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9fd3fff640 1 -- 172.21.15.89:0/1839774235 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe00106c0 con 0x55d5db6c8640 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1839774235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 msgr2=0x55d5db6c8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1839774235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6c8a20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x55d5db6c5de0 tx=0x7f9fe0004160 comp rx=0 tx=0).stop 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1839774235 shutdown_connections 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1839774235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6d5b50 0x55d5db6d7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.785+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1839774235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 0x55d5db6d5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1839774235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6c8a20 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1839774235 >> 172.21.15.89:0/1839774235 conn(0x55d5db6c4cf0 msgr2=0x55d5db6c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1839774235 shutdown_connections 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1839774235 wait complete. 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 -- start start 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 Processor -- start 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 0x55d5db6d2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6d5b50 0x55d5db6d34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d5db6d8720 con 0x55d5db6c8640 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d5db5b66e0 con 0x55d5db6d5b50 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d5db6d88a0 con 0x55d5db6c8f60 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.786+0000 7f9ff5fa2740 10 client.0 _ll_register_callbacks cb 0x55d5db5daa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:39.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1164640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1164640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52734/0 (socket says 172.21.15.89:52734) 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1164640 1 -- 172.21.15.89:0/1680615112 learned_addr learned my addr 172.21.15.89:0/1680615112 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1965640 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 0x55d5db6d2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1164640 1 -- 172.21.15.89:0/1680615112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 msgr2=0x55d5db6d2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 0x55d5db6d2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1164640 1 -- 172.21.15.89:0/1680615112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6d5b50 msgr2=0x55d5db6d34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6d5b50 0x55d5db6d34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1164640 1 -- 172.21.15.89:0/1680615112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5db6e1c20 con 0x55d5db6c8640 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1965640 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 0x55d5db6d2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d2a30 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0004210 tx=0x7f9fe0004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69458b24844f9a47 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9fd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:39.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1680615112 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe00046c0 con 0x55d5db6c8640 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1680615112 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9fe0004860 con 0x55d5db6c8640 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1680615112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fe0005500 con 0x55d5db6c8640 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1680615112 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9fe0004a60 con 0x55d5db6c8640 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff5fa2740 10 client.17040 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.787+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1680615112 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5db6e45d0 con 0x55d5db6c8640 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.788+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1680615112 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0005500 con 0x55d5db6c8640 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.788+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1680615112 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9fe001c020 con 0x55d5db6c8640 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.788+0000 7f9ff5fa2740 10 client.17040 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.788+0000 7f9ff5fa2740 10 client.17040 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.788+0000 7f9ff5fa2740 20 client.17040 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.788+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d5db6e5920 0x55d5db6e7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.788+0000 7f9ff5fa2740 4 client.17040 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.788+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1680615112 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5db63b9d0 con 0x55d5db6e5920 2024-09-15T23:28:39.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.788+0000 7f9ff1965640 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d5db6e5920 0x55d5db6e7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.789+0000 7f9fd17fa640 10 client.17040 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:39.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.789+0000 7f9ff1965640 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d5db6e5920 0x55d5db6e7d10 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72384702d09ccd46 server_cookie=93825355d4537bb6 in_seq=0 out_seq=0 2024-09-15T23:28:39.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.789+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1680615112 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d5db63b9d0 con 0x55d5db6e5920 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.789+0000 7f9fd17fa640 10 client.17040 handle_command_reply: tid=0 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.789+0000 7f9ff5fa2740 1 client.17040 shutdown 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.789+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1680615112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 msgr2=0x55d5db6d2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.789+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d2a30 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0004210 tx=0x7f9fe0004f80 comp rx=0 tx=0).stop 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.790+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1680615112 shutdown_connections 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.790+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d5db6e5920 0x55d5db6e7d10 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.790+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6d5b50 0x55d5db6d34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.790+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 0x55d5db6d2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.790+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1680615112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d2a30 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.790+0000 7f9fd17fa640 0 client.17040 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.790+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1680615112 >> 172.21.15.89:0/1680615112 conn(0x55d5db6c4cf0 msgr2=0x55d5db6caa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:39.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.790+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1680615112 shutdown_connections 2024-09-15T23:28:39.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.790+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1680615112 wait complete. 2024-09-15T23:28:39.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.790+0000 7f9ff5fa2740 20 client.17040 trim_cache size 0 max 16384 2024-09-15T23:28:39.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.806+0000 7f9ff5fa2740 1 Processor -- start 2024-09-15T23:28:39.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.806+0000 7f9ff5fa2740 1 -- start start 2024-09-15T23:28:39.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.806+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.806+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 0x55d5db6ca860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.806+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6d5b50 0x55d5db6cada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.806+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec1acce0 con 0x55d5db6c8640 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.806+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec1ae820 con 0x55d5db6c8f60 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.806+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec1ad3b0 con 0x55d5db6d5b50 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1164640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1164640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52750/0 (socket says 172.21.15.89:52750) 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1164640 1 -- 172.21.15.89:0/661100460 learned_addr learned my addr 172.21.15.89:0/661100460 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1965640 1 --2- 172.21.15.89:0/661100460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 0x55d5db6ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1164640 1 -- 172.21.15.89:0/661100460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6d5b50 msgr2=0x55d5db6cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1164640 1 --2- 172.21.15.89:0/661100460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6d5b50 0x55d5db6cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1164640 1 -- 172.21.15.89:0/661100460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 msgr2=0x55d5db6ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:39.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1164640 1 --2- 172.21.15.89:0/661100460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 0x55d5db6ca860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1164640 1 -- 172.21.15.89:0/661100460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5db6c0260 con 0x55d5db6c8640 2024-09-15T23:28:39.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1965640 1 --2- 172.21.15.89:0/661100460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 0x55d5db6ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:39.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9ff1164640 1 --2- 172.21.15.89:0/661100460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d21e0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000b520 tx=0x7f9fe000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94cac61545fca6e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:39.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9fd37fe640 1 -- 172.21.15.89:0/661100460 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe00124d0 con 0x55d5db6c8640 2024-09-15T23:28:39.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9fd37fe640 1 -- 172.21.15.89:0/661100460 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0004c00 con 0x55d5db6c8640 2024-09-15T23:28:39.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.807+0000 7f9fd37fe640 1 -- 172.21.15.89:0/661100460 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0005570 con 0x55d5db6c8640 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.808+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/661100460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 msgr2=0x55d5db6d21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.808+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/661100460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d21e0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000b520 tx=0x7f9fe000bfd0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.808+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/661100460 shutdown_connections 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.808+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/661100460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6d5b50 0x55d5db6cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.808+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/661100460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8f60 0x55d5db6ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.808+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/661100460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6c8640 0x55d5db6d21e0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.808+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/661100460 >> 172.21.15.89:0/661100460 conn(0x55d5db6c4cf0 msgr2=0x55d5db6ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.808+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/661100460 shutdown_connections 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.808+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/661100460 wait complete. 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.808+0000 7f9ff5fa2740 1 -- start start 2024-09-15T23:28:39.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff5fa2740 1 Processor -- start 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8640 0x55d5db6d9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 0x55d5db6d9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff5fa2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6d5b50 0x55d5db6da3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec1ae150 con 0x55d5db6d5b50 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec1ada80 con 0x55d5db6c8640 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff5fa2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec1aacc0 con 0x55d5db6c8f60 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff5fa2740 10 client.0 _ll_register_callbacks cb 0x7f9fe000b8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff1164640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6d5b50 0x55d5db6da3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff1164640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6d5b50 0x55d5db6da3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:52758/0 (socket says 172.21.15.89:52758) 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff1164640 1 -- 172.21.15.89:0/1749702491 learned_addr learned my addr 172.21.15.89:0/1749702491 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff1965640 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 0x55d5db6d9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8640 0x55d5db6d9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff1164640 1 -- 172.21.15.89:0/1749702491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 msgr2=0x55d5db6d9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 0x55d5db6d9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff1164640 1 -- 172.21.15.89:0/1749702491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8640 msgr2=0x55d5db6d9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8640 0x55d5db6d9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.809+0000 7f9ff1164640 1 -- 172.21.15.89:0/1749702491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5db74f470 con 0x55d5db6d5b50 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9ff1164640 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6d5b50 0x55d5db6da3f0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0004ed0 tx=0x7f9fe0005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5c88fb60dadb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9fd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1749702491 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0045060 con 0x55d5db6d5b50 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1749702491 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9fe0010460 con 0x55d5db6d5b50 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1749702491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fe0004c00 con 0x55d5db6d5b50 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1749702491 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9fe0010660 con 0x55d5db6d5b50 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9ff5fa2740 10 client.17046 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1749702491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5db7506a0 con 0x55d5db6d5b50 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1749702491 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe0004c00 con 0x55d5db6d5b50 2024-09-15T23:28:39.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1749702491 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9fe001c070 con 0x55d5db6d5b50 2024-09-15T23:28:39.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9ff5fa2740 10 client.17046 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:39.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9ff5fa2740 10 client.17046 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:28:39.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9ff5fa2740 20 client.17046 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:39.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d5db750b30 0x55d5db752f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:39.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9ff5fa2740 4 client.17046 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 133.8, "await_for": 45.0}] 2024-09-15T23:28:39.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.810+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1749702491 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 133.8, "await_for": 45.0}) -- 0x55d5db635d20 con 0x55d5db750b30 2024-09-15T23:28:39.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.811+0000 7f9ff1965640 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d5db750b30 0x55d5db752f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:39.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.811+0000 7f9fd17fa640 10 client.17046 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:39.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.811+0000 7f9ff1965640 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d5db750b30 0x55d5db752f20 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59598b2aa61c27a server_cookie=1a7b559844742466 in_seq=0 out_seq=0 2024-09-15T23:28:39.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9fd17fa640 1 -- 172.21.15.89:0/1749702491 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d5db635d20 con 0x55d5db750b30 2024-09-15T23:28:39.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9fd17fa640 10 client.17046 handle_command_reply: tid=0 2024-09-15T23:28:39.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 client.17046 shutdown 2024-09-15T23:28:39.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1749702491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6d5b50 msgr2=0x55d5db6da3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:39.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6d5b50 0x55d5db6da3f0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0004ed0 tx=0x7f9fe0005bb0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1749702491 shutdown_connections 2024-09-15T23:28:39.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55d5db750b30 0x55d5db752f20 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d5db6d5b50 0x55d5db6da3f0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d5db6c8f60 0x55d5db6d9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9fd17fa640 0 client.17046 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:39.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 --2- 172.21.15.89:0/1749702491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d5db6c8640 0x55d5db6d9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1749702491 >> 172.21.15.89:0/1749702491 conn(0x55d5db6c4cf0 msgr2=0x55d5db6ca970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:39.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1749702491 shutdown_connections 2024-09-15T23:28:39.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 1 -- 172.21.15.89:0/1749702491 wait complete. 2024-09-15T23:28:39.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff5fa2740 20 client.17046 trim_cache size 0 max 16384 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 14, 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "735ccfd0": { 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 14, 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:28:39.826 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 133.8, 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:39.827 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:28:39.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.823+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/2904914242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd40a3240 msgr2=0x7f9fd40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:39.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd40a3240 0x7f9fd40a5700 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc098af0 tx=0x7f9fdc061ef0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/2904914242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fec106a90 msgr2=0x7f9fec19bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:39.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fec106a90 0x7f9fec19bc40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00eaa0 tx=0x7f9fdc00ef60 comp rx=0 tx=0).stop 2024-09-15T23:28:39.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/2904914242 shutdown_connections 2024-09-15T23:28:39.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd40a3240 0x7f9fd40a5700 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fec106a90 0x7f9fec19bc40 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec101430 0x7f9fec1978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 --2- 172.21.15.89:0/2904914242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fec100a90 0x7f9fec197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:39.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/2904914242 >> 172.21.15.89:0/2904914242 conn(0x7f9fec0fc7d0 msgr2=0x7f9fec0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:39.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/2904914242 shutdown_connections 2024-09-15T23:28:39.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:39.824+0000 7f9ff3bc9640 1 -- 172.21.15.89:0/2904914242 wait complete. 2024-09-15T23:28:40.635 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '735ccfd0', quiesce took 3.3 seconds. Will release after: 10.5 2024-09-15T23:28:41.475 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:41 smithi120 ceph-mon[35250]: pgmap v490: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:28:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:41 smithi174 ceph-mon[35586]: pgmap v490: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:28:41.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:41 smithi089 ceph-mon[30933]: pgmap v490: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:28:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:43 smithi120 ceph-mon[35250]: pgmap v491: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:28:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:43 smithi174 ceph-mon[35586]: pgmap v491: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:28:43.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:43 smithi089 ceph-mon[30933]: pgmap v491: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:28:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:45 smithi120 ceph-mon[35250]: pgmap v492: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:28:45.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:45 smithi174 ceph-mon[35586]: pgmap v492: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:28:45.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:45 smithi089 ceph-mon[30933]: pgmap v492: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:28:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:47 smithi120 ceph-mon[35250]: pgmap v493: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:28:47.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:47 smithi174 ceph-mon[35586]: pgmap v493: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:28:47.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:47 smithi089 ceph-mon[30933]: pgmap v493: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:28:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:49 smithi120 ceph-mon[35250]: pgmap v494: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 13 op/s 2024-09-15T23:28:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:49 smithi174 ceph-mon[35586]: pgmap v494: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 13 op/s 2024-09-15T23:28:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:49 smithi089 ceph-mon[30933]: pgmap v494: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 13 op/s 2024-09-15T23:28:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 -- 172.21.15.89:0/229982762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38104f00 msgr2=0x7fbb381072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 --2- 172.21.15.89:0/229982762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38104f00 0x7fbb381072f0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009900 tx=0x7fbb2c02a410 comp rx=0 tx=0).stop 2024-09-15T23:28:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 -- 172.21.15.89:0/229982762 shutdown_connections 2024-09-15T23:28:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 --2- 172.21.15.89:0/229982762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38107830 0x7fbb38109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 --2- 172.21.15.89:0/229982762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38104f00 0x7fbb381072f0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 --2- 172.21.15.89:0/229982762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb381025d0 0x7fbb381049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 -- 172.21.15.89:0/229982762 >> 172.21.15.89:0/229982762 conn(0x7fbb380fc1b0 msgr2=0x7fbb380fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 -- 172.21.15.89:0/229982762 shutdown_connections 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 -- 172.21.15.89:0/229982762 wait complete. 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 Processor -- start 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.791+0000 7fbb3e942640 1 -- start start 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb3e942640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb381025d0 0x7fbb38197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb3e942640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38104f00 0x7fbb381983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb3e942640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38107830 0x7fbb3819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb3e942640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3810d800 con 0x7fbb38104f00 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb3e942640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3810d680 con 0x7fbb38107830 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb3e942640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3810d980 con 0x7fbb381025d0 2024-09-15T23:28:50.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38104f00 0x7fbb381983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38104f00 0x7fbb381983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54774/0 (socket says 172.21.15.89:54774) 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb377fe640 1 -- 172.21.15.89:0/1216671871 learned_addr learned my addr 172.21.15.89:0/1216671871 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38107830 0x7fbb3819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb37fff640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb381025d0 0x7fbb38197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb377fe640 1 -- 172.21.15.89:0/1216671871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb381025d0 msgr2=0x7fbb38197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb377fe640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb381025d0 0x7fbb38197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb377fe640 1 -- 172.21.15.89:0/1216671871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38107830 msgr2=0x7fbb3819f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb377fe640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38107830 0x7fbb3819f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb377fe640 1 -- 172.21.15.89:0/1216671871 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3819fa80 con 0x7fbb38104f00 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.792+0000 7fbb377fe640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38104f00 0x7fbb381983a0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c031610 tx=0x7fbb2c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.793+0000 7fbb357fa640 1 -- 172.21.15.89:0/1216671871 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c03f030 con 0x7fbb38104f00 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.793+0000 7fbb3e942640 1 -- 172.21.15.89:0/1216671871 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3819fd10 con 0x7fbb38104f00 2024-09-15T23:28:50.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.793+0000 7fbb357fa640 1 -- 172.21.15.89:0/1216671871 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2c003ce0 con 0x7fbb38104f00 2024-09-15T23:28:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.793+0000 7fbb357fa640 1 -- 172.21.15.89:0/1216671871 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c035070 con 0x7fbb38104f00 2024-09-15T23:28:50.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.793+0000 7fbb3e942640 1 -- 172.21.15.89:0/1216671871 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb381a01f0 con 0x7fbb38104f00 2024-09-15T23:28:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.795+0000 7fbb357fa640 1 -- 172.21.15.89:0/1216671871 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbb2c042030 con 0x7fbb38104f00 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.795+0000 7fbb416dc740 1 Processor -- start 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.795+0000 7fbb416dc740 1 -- start start 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.795+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 0x5565db803da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.795+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db808da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.795+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db80b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.795+0000 7fbb416dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5565db7672f0 con 0x5565db809370 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.795+0000 7fbb416dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565db70f270 con 0x5565db8042e0 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.795+0000 7fbb416dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5565db7632b0 con 0x5565db8039a0 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.795+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db80b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db80b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54778/0 (socket says 172.21.15.89:54778) 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb377fe640 1 -- 172.21.15.89:0/2757988485 learned_addr learned my addr 172.21.15.89:0/2757988485 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb37fff640 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db808da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 0x5565db803da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb377fe640 1 -- 172.21.15.89:0/2757988485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 msgr2=0x5565db803da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb377fe640 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 0x5565db803da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb377fe640 1 -- 172.21.15.89:0/2757988485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 msgr2=0x5565db808da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb377fe640 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db808da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb377fe640 1 -- 172.21.15.89:0/2757988485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565db70e340 con 0x5565db809370 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb37fff640 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db808da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 0x5565db803da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb357fa640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb200a32f0 0x7fbb200a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb377fe640 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db80b830 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0f8480 tx=0x7fbb2c0f8570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39617389f44bc20c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb200a32f0 0x7fbb200a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb357fa640 1 -- 172.21.15.89:0/1216671871 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb2c0f01f0 con 0x7fbb38104f00 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb1e7fc640 1 -- 172.21.15.89:0/2757988485 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c0b8770 con 0x5565db809370 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb1e7fc640 1 -- 172.21.15.89:0/2757988485 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2c0b8030 con 0x5565db809370 2024-09-15T23:28:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.796+0000 7fbb1e7fc640 1 -- 172.21.15.89:0/2757988485 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c0bbe30 con 0x5565db809370 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb200a32f0 0x7fbb200a57b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbb3806b3e0 tx=0x7fbb30009600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb416dc740 1 -- 172.21.15.89:0/2757988485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 msgr2=0x5565db80b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb416dc740 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db80b830 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0f8480 tx=0x7fbb2c0f8570 comp rx=0 tx=0).stop 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb416dc740 1 -- 172.21.15.89:0/2757988485 shutdown_connections 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb416dc740 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db80b830 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb416dc740 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db808da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb416dc740 1 --2- 172.21.15.89:0/2757988485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 0x5565db803da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb416dc740 1 -- 172.21.15.89:0/2757988485 >> 172.21.15.89:0/2757988485 conn(0x5565db7ff010 msgr2=0x5565db7ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb416dc740 1 -- 172.21.15.89:0/2757988485 shutdown_connections 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb416dc740 1 -- 172.21.15.89:0/2757988485 wait complete. 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.797+0000 7fbb416dc740 1 -- start start 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb416dc740 1 Processor -- start 2024-09-15T23:28:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8039a0 0x5565db80d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db80de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 0x5565db814e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb416dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5565db80c100 con 0x5565db8039a0 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb416dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565db6f0290 con 0x5565db8042e0 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb416dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5565db80c280 con 0x5565db809370 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb416dc740 10 client.0 _ll_register_callbacks cb 0x5565db721140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8039a0 0x5565db80d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8039a0 0x5565db80d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54780/0 (socket says 172.21.15.89:54780) 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb37fff640 1 -- 172.21.15.89:0/3665127084 learned_addr learned my addr 172.21.15.89:0/3665127084 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb377fe640 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db80de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 0x5565db814e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb37fff640 1 -- 172.21.15.89:0/3665127084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 msgr2=0x5565db814e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb37fff640 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 0x5565db814e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.798+0000 7fbb37fff640 1 -- 172.21.15.89:0/3665127084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 msgr2=0x5565db80de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb37fff640 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db80de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb37fff640 1 -- 172.21.15.89:0/3665127084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565db47a210 con 0x5565db8039a0 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 0x5565db814e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb37fff640 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8039a0 0x5565db80d8f0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800b9a0 tx=0x7fbb2800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37afe8301caa6a4b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb177fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb177fe640 1 -- 172.21.15.89:0/3665127084 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28005020 con 0x5565db8039a0 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb177fe640 1 -- 172.21.15.89:0/3665127084 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb280051c0 con 0x5565db8039a0 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb416dc740 1 -- 172.21.15.89:0/3665127084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb2c0b8030 con 0x5565db8039a0 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb177fe640 1 -- 172.21.15.89:0/3665127084 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb28023050 con 0x5565db8039a0 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb416dc740 10 client.17058 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb416dc740 1 -- 172.21.15.89:0/3665127084 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565db546620 con 0x5565db8039a0 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb177fe640 1 -- 172.21.15.89:0/3665127084 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2c0b8030 con 0x5565db8039a0 2024-09-15T23:28:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb177fe640 1 -- 172.21.15.89:0/3665127084 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbb28019030 con 0x5565db8039a0 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb416dc740 10 client.17058 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.799+0000 7fbb416dc740 10 client.17058 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.800+0000 7fbb416dc740 20 client.17058 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.800+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565db819610 0x5565db81bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.800+0000 7fbb416dc740 4 client.17058 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.800+0000 7fbb416dc740 1 -- 172.21.15.89:0/3665127084 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565db774a20 con 0x5565db819610 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.800+0000 7fbb377fe640 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565db819610 0x5565db81bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.800+0000 7fbb377fe640 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565db819610 0x5565db81bab0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfcc0cfd1ba4efd4 server_cookie=346fdaf96563b1b5 in_seq=0 out_seq=0 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.800+0000 7fbb177fe640 10 client.17058 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb177fe640 1 -- 172.21.15.89:0/3665127084 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5565db774a20 con 0x5565db819610 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb177fe640 10 client.17058 handle_command_reply: tid=0 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 client.17058 shutdown 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 -- 172.21.15.89:0/3665127084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8039a0 msgr2=0x5565db80d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8039a0 0x5565db80d8f0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800b9a0 tx=0x7fbb2800be60 comp rx=0 tx=0).stop 2024-09-15T23:28:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 -- 172.21.15.89:0/3665127084 shutdown_connections 2024-09-15T23:28:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565db819610 0x5565db81bab0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 0x5565db814e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db80de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb177fe640 0 client.17058 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3665127084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8039a0 0x5565db80d8f0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 -- 172.21.15.89:0/3665127084 >> 172.21.15.89:0/3665127084 conn(0x5565db7ff010 msgr2=0x5565db800820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 -- 172.21.15.89:0/3665127084 shutdown_connections 2024-09-15T23:28:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 1 -- 172.21.15.89:0/3665127084 wait complete. 2024-09-15T23:28:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.801+0000 7fbb416dc740 20 client.17058 trim_cache size 0 max 16384 2024-09-15T23:28:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.815+0000 7fbb416dc740 1 Processor -- start 2024-09-15T23:28:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.815+0000 7fbb416dc740 1 -- start start 2024-09-15T23:28:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8039a0 0x5565db7ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8042e0 0x5565db814d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 0x5565db7ff930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb416dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381aa8b0 con 0x5565db8042e0 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb416dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381ac3f0 con 0x5565db8039a0 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb416dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381aaf80 con 0x5565db809370 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8042e0 0x5565db814d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb377fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8042e0 0x5565db814d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54784/0 (socket says 172.21.15.89:54784) 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb377fe640 1 -- 172.21.15.89:0/96769016 learned_addr learned my addr 172.21.15.89:0/96769016 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/96769016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 0x5565db7ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb377fe640 1 -- 172.21.15.89:0/96769016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 msgr2=0x5565db7ff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb377fe640 1 --2- 172.21.15.89:0/96769016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 0x5565db7ff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb377fe640 1 -- 172.21.15.89:0/96769016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8039a0 msgr2=0x5565db7ff3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb37fff640 1 --2- 172.21.15.89:0/96769016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8039a0 0x5565db7ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb377fe640 1 --2- 172.21.15.89:0/96769016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8039a0 0x5565db7ff3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb377fe640 1 -- 172.21.15.89:0/96769016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565db7fee30 con 0x5565db8042e0 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/96769016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 0x5565db7ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb377fe640 1 --2- 172.21.15.89:0/96769016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8042e0 0x5565db814d90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0ed950 tx=0x7fbb2c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8de9ddbdb4a361 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb1dffb640 1 -- 172.21.15.89:0/96769016 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c049040 con 0x5565db8042e0 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb1dffb640 1 -- 172.21.15.89:0/96769016 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2c0f3aa0 con 0x5565db8042e0 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.816+0000 7fbb1dffb640 1 -- 172.21.15.89:0/96769016 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c0bc010 con 0x5565db8042e0 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.817+0000 7fbb416dc740 1 -- 172.21.15.89:0/96769016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8042e0 msgr2=0x5565db814d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.817+0000 7fbb416dc740 1 --2- 172.21.15.89:0/96769016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8042e0 0x5565db814d90 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0ed950 tx=0x7fbb2c02a410 comp rx=0 tx=0).stop 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.817+0000 7fbb416dc740 1 -- 172.21.15.89:0/96769016 shutdown_connections 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.817+0000 7fbb416dc740 1 --2- 172.21.15.89:0/96769016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db809370 0x5565db7ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.817+0000 7fbb416dc740 1 --2- 172.21.15.89:0/96769016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db8042e0 0x5565db814d90 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.817+0000 7fbb416dc740 1 --2- 172.21.15.89:0/96769016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8039a0 0x5565db7ff3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.817+0000 7fbb416dc740 1 -- 172.21.15.89:0/96769016 >> 172.21.15.89:0/96769016 conn(0x5565db7ff010 msgr2=0x5565db808b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.817+0000 7fbb416dc740 1 -- 172.21.15.89:0/96769016 shutdown_connections 2024-09-15T23:28:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.817+0000 7fbb416dc740 1 -- 172.21.15.89:0/96769016 wait complete. 2024-09-15T23:28:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.817+0000 7fbb416dc740 1 -- start start 2024-09-15T23:28:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb416dc740 1 Processor -- start 2024-09-15T23:28:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 0x5565db812340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db812880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb416dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db813890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb416dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381abd20 con 0x5565db809370 2024-09-15T23:28:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb416dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381ab650 con 0x5565db8042e0 2024-09-15T23:28:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb416dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb381a8890 con 0x5565db8039a0 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb416dc740 10 client.0 _ll_register_callbacks cb 0x7fbb28004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb3cedf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db813890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb3cedf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db813890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54798/0 (socket says 172.21.15.89:54798) 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb3cedf640 1 -- 172.21.15.89:0/3559621266 learned_addr learned my addr 172.21.15.89:0/3559621266 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb37fff640 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 0x5565db812340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb3cedf640 1 -- 172.21.15.89:0/3559621266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 msgr2=0x5565db812340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 0x5565db812340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb377fe640 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db812880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb3cedf640 1 -- 172.21.15.89:0/3559621266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 msgr2=0x5565db812880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db812880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb3cedf640 1 -- 172.21.15.89:0/3559621266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565db813ea0 con 0x5565db809370 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb37fff640 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 0x5565db812340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb3cedf640 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db813890 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fbb300107a0 tx=0x7fbb30014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8e1a9c02b34e20 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.818+0000 7fbb177fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb177fe640 1 -- 172.21.15.89:0/3559621266 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3002c060 con 0x5565db809370 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb177fe640 1 -- 172.21.15.89:0/3559621266 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb3001cb40 con 0x5565db809370 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb416dc740 1 -- 172.21.15.89:0/3559621266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb2c0f3aa0 con 0x5565db809370 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb177fe640 1 -- 172.21.15.89:0/3559621266 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb30033050 con 0x5565db809370 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb416dc740 10 client.17064 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb416dc740 1 -- 172.21.15.89:0/3559621266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565db815220 con 0x5565db809370 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb177fe640 1 -- 172.21.15.89:0/3559621266 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb2c0f3aa0 con 0x5565db809370 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb177fe640 1 -- 172.21.15.89:0/3559621266 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbb30026070 con 0x5565db809370 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb416dc740 10 client.17064 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb416dc740 10 client.17064 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb416dc740 20 client.17064 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565db884b40 0x5565db886f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb416dc740 4 client.17064 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:28:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb416dc740 1 -- 172.21.15.89:0/3559621266 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5565db76ed70 con 0x5565db884b40 2024-09-15T23:28:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.819+0000 7fbb37fff640 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565db884b40 0x5565db886f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.820+0000 7fbb177fe640 10 client.17064 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.820+0000 7fbb37fff640 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565db884b40 0x5565db886f60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0cac1387b7ed86e server_cookie=faa18f451f07cdd0 in_seq=0 out_seq=0 2024-09-15T23:28:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb177fe640 1 -- 172.21.15.89:0/3559621266 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5565db76ed70 con 0x5565db884b40 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb177fe640 10 client.17064 handle_command_reply: tid=0 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 client.17064 shutdown 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 -- 172.21.15.89:0/3559621266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 msgr2=0x5565db813890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db813890 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fbb300107a0 tx=0x7fbb30014040 comp rx=0 tx=0).stop 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 -- 172.21.15.89:0/3559621266 shutdown_connections 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565db884b40 0x5565db886f60 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565db809370 0x5565db813890 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565db8042e0 0x5565db812880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 --2- 172.21.15.89:0/3559621266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565db8039a0 0x5565db812340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 -- 172.21.15.89:0/3559621266 >> 172.21.15.89:0/3559621266 conn(0x5565db7ff010 msgr2=0x5565db7fd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb177fe640 0 client.17064 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 -- 172.21.15.89:0/3559621266 shutdown_connections 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 1 -- 172.21.15.89:0/3559621266 wait complete. 2024-09-15T23:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.833+0000 7fbb416dc740 20 client.17064 trim_cache size 0 max 16384 2024-09-15T23:28:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:28:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:28:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:28:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:28:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 -- 172.21.15.89:0/1216671871 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb200a32f0 msgr2=0x7fbb200a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb200a32f0 0x7fbb200a57b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbb3806b3e0 tx=0x7fbb30009600 comp rx=0 tx=0).stop 2024-09-15T23:28:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 -- 172.21.15.89:0/1216671871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38104f00 msgr2=0x7fbb381983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38104f00 0x7fbb381983a0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c031610 tx=0x7fbb2c0043b0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 -- 172.21.15.89:0/1216671871 shutdown_connections 2024-09-15T23:28:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb200a32f0 0x7fbb200a57b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb38107830 0x7fbb3819f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb38104f00 0x7fbb381983a0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 --2- 172.21.15.89:0/1216671871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb381025d0 0x7fbb38197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 -- 172.21.15.89:0/1216671871 >> 172.21.15.89:0/1216671871 conn(0x7fbb380fc1b0 msgr2=0x7fbb38102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 -- 172.21.15.89:0/1216671871 shutdown_connections 2024-09-15T23:28:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:28:50.834+0000 7fbb3e942640 1 -- 172.21.15.89:0/1216671871 wait complete. 2024-09-15T23:28:51.136 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:28:51.388 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:51 smithi120 ceph-mon[35250]: pgmap v495: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 16 op/s 2024-09-15T23:28:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:51 smithi174 ceph-mon[35586]: pgmap v495: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 16 op/s 2024-09-15T23:28:51.509 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:51 smithi089 ceph-mon[30933]: pgmap v495: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 16 op/s 2024-09-15T23:28:51.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.863+0000 7f3c7da3a640 1 -- 172.21.15.89:0/3826527008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c78103670 msgr2=0x7f3c7810bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:51.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.863+0000 7f3c7da3a640 1 --2- 172.21.15.89:0/3826527008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c78103670 0x7f3c7810bb60 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009900 tx=0x7f3c6002a480 comp rx=0 tx=0).stop 2024-09-15T23:28:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.864+0000 7f3c7da3a640 1 -- 172.21.15.89:0/3826527008 shutdown_connections 2024-09-15T23:28:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.864+0000 7f3c7da3a640 1 --2- 172.21.15.89:0/3826527008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c7810c180 0x7f3c7810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.864+0000 7f3c7da3a640 1 --2- 172.21.15.89:0/3826527008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c78103670 0x7f3c7810bb60 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.864+0000 7f3c7da3a640 1 --2- 172.21.15.89:0/3826527008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c78102d50 0x7f3c78103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.864+0000 7f3c7da3a640 1 -- 172.21.15.89:0/3826527008 >> 172.21.15.89:0/3826527008 conn(0x7f3c780fc7d0 msgr2=0x7f3c780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.864+0000 7f3c7da3a640 1 -- 172.21.15.89:0/3826527008 shutdown_connections 2024-09-15T23:28:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.864+0000 7f3c7da3a640 1 -- 172.21.15.89:0/3826527008 wait complete. 2024-09-15T23:28:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.864+0000 7f3c7da3a640 1 Processor -- start 2024-09-15T23:28:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.864+0000 7f3c7da3a640 1 -- start start 2024-09-15T23:28:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7da3a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c78102d50 0x7f3c781009b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7da3a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c78103670 0x7f3c78100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7da3a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c7810c180 0x7f3c78101430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7da3a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78111f80 con 0x7f3c78102d50 2024-09-15T23:28:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7da3a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78111e00 con 0x7f3c7810c180 2024-09-15T23:28:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7da3a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78112100 con 0x7f3c78103670 2024-09-15T23:28:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c76ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c78102d50 0x7f3c781009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c76ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c78102d50 0x7f3c781009b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54822/0 (socket says 172.21.15.89:54822) 2024-09-15T23:28:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c76ffd640 1 -- 172.21.15.89:0/931485331 learned_addr learned my addr 172.21.15.89:0/931485331 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c767fc640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c78103670 0x7f3c78100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c777fe640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c7810c180 0x7f3c78101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c76ffd640 1 -- 172.21.15.89:0/931485331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c78103670 msgr2=0x7f3c78100ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c76ffd640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c78103670 0x7f3c78100ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c76ffd640 1 -- 172.21.15.89:0/931485331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c7810c180 msgr2=0x7f3c78101430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c76ffd640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c7810c180 0x7f3c78101430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c76ffd640 1 -- 172.21.15.89:0/931485331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c781a69a0 con 0x7f3c78102d50 2024-09-15T23:28:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c76ffd640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c78102d50 0x7f3c781009b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000eab0 tx=0x7f3c7000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7ca38640 1 -- 172.21.15.89:0/931485331 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7000ce40 con 0x7f3c78102d50 2024-09-15T23:28:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7da3a640 1 -- 172.21.15.89:0/931485331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c781a6c90 con 0x7f3c78102d50 2024-09-15T23:28:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7ca38640 1 -- 172.21.15.89:0/931485331 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c70011950 con 0x7f3c78102d50 2024-09-15T23:28:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7ca38640 1 -- 172.21.15.89:0/931485331 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c70010ae0 con 0x7f3c78102d50 2024-09-15T23:28:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.865+0000 7f3c7da3a640 1 -- 172.21.15.89:0/931485331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c781a71c0 con 0x7f3c78102d50 2024-09-15T23:28:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.867+0000 7f3c7da3a640 1 -- 172.21.15.89:0/931485331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c78103130 con 0x7f3c78102d50 2024-09-15T23:28:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.867+0000 7f3c7ca38640 1 -- 172.21.15.89:0/931485331 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3c70011b00 con 0x7f3c78102d50 2024-09-15T23:28:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.868+0000 7f3c7ca38640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c640a3240 0x7f3c640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.868+0000 7f3c767fc640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c640a3240 0x7f3c640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.868+0000 7f3c7ca38640 1 -- 172.21.15.89:0/931485331 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c700d04e0 con 0x7f3c78102d50 2024-09-15T23:28:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.869+0000 7f3c767fc640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c640a3240 0x7f3c640a5700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3c78102440 tx=0x7f3c60005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:51.872+0000 7f3c7ca38640 1 -- 172.21.15.89:0/931485331 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3c70016030 con 0x7f3c78102d50 2024-09-15T23:28:52.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.072+0000 7f3c7da3a640 1 -- 172.21.15.89:0/931485331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c781071f0 con 0x7f3c78102d50 2024-09-15T23:28:52.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.073+0000 7f3c7ca38640 1 -- 172.21.15.89:0/931485331 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f3c7009ce10 con 0x7f3c78102d50 2024-09-15T23:28:52.075 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:28:52.075 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:28:52.075 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:28:52.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 -- 172.21.15.89:0/931485331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c640a3240 msgr2=0x7f3c640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c640a3240 0x7f3c640a5700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3c78102440 tx=0x7f3c60005a20 comp rx=0 tx=0).stop 2024-09-15T23:28:52.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 -- 172.21.15.89:0/931485331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c78102d50 msgr2=0x7f3c781009b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c78102d50 0x7f3c781009b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000eab0 tx=0x7f3c7000ef70 comp rx=0 tx=0).stop 2024-09-15T23:28:52.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 -- 172.21.15.89:0/931485331 shutdown_connections 2024-09-15T23:28:52.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c640a3240 0x7f3c640a5700 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c7810c180 0x7f3c78101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c78103670 0x7f3c78100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 --2- 172.21.15.89:0/931485331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c78102d50 0x7f3c781009b0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 -- 172.21.15.89:0/931485331 >> 172.21.15.89:0/931485331 conn(0x7f3c780fc7d0 msgr2=0x7f3c780fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:52.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 -- 172.21.15.89:0/931485331 shutdown_connections 2024-09-15T23:28:52.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:52.075+0000 7f3c7da3a640 1 -- 172.21.15.89:0/931485331 wait complete. 2024-09-15T23:28:52.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:52 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/931485331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:52.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:52 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/931485331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:52.541 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:52 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/931485331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:52.883 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 735ccfd0 --release --await' 2024-09-15T23:28:52.884 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 735ccfd0 --release --await 2024-09-15T23:28:53.130 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:53 smithi120 ceph-mon[35250]: pgmap v496: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-09-15T23:28:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:53 smithi174 ceph-mon[35586]: pgmap v496: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-09-15T23:28:53.543 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:53 smithi089 ceph-mon[30933]: pgmap v496: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-09-15T23:28:53.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.628+0000 7f333c188640 1 -- 172.21.15.89:0/3586656847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334102de0 msgr2=0x7f33341031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:53.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.628+0000 7f333c188640 1 --2- 172.21.15.89:0/3586656847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334102de0 0x7f33341031c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f332c009900 tx=0x7f332c02a480 comp rx=0 tx=0).stop 2024-09-15T23:28:53.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.628+0000 7f333c188640 1 -- 172.21.15.89:0/3586656847 shutdown_connections 2024-09-15T23:28:53.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.628+0000 7f333c188640 1 --2- 172.21.15.89:0/3586656847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f333410c180 0x7f333410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.628+0000 7f333c188640 1 --2- 172.21.15.89:0/3586656847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3334103700 0x7f333410bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.628+0000 7f333c188640 1 --2- 172.21.15.89:0/3586656847 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334102de0 0x7f33341031c0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.628+0000 7f333c188640 1 -- 172.21.15.89:0/3586656847 >> 172.21.15.89:0/3586656847 conn(0x7f33340fc7a0 msgr2=0x7f33340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:53.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.628+0000 7f333c188640 1 -- 172.21.15.89:0/3586656847 shutdown_connections 2024-09-15T23:28:53.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.628+0000 7f333c188640 1 -- 172.21.15.89:0/3586656847 wait complete. 2024-09-15T23:28:53.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f333c188640 1 Processor -- start 2024-09-15T23:28:53.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f333c188640 1 -- start start 2024-09-15T23:28:53.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f333c188640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334102de0 0x7f33341984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f333c188640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3334103700 0x7f33341989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f333c188640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f333410c180 0x7f333419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f333c188640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3334112020 con 0x7f3334102de0 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f333c188640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3334111ea0 con 0x7f3334103700 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f333c188640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f33341121a0 con 0x7f333410c180 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f3339f24640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334102de0 0x7f33341984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f3339f24640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334102de0 0x7f33341984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54848/0 (socket says 172.21.15.89:54848) 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.629+0000 7f3339f24640 1 -- 172.21.15.89:0/2081440692 learned_addr learned my addr 172.21.15.89:0/2081440692 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f3339f24640 1 -- 172.21.15.89:0/2081440692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f333410c180 msgr2=0x7f333419cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f3339f24640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f333410c180 0x7f333419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f3339f24640 1 -- 172.21.15.89:0/2081440692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3334103700 msgr2=0x7f33341989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:53.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f3339723640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3334103700 0x7f33341989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f3339f24640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3334103700 0x7f33341989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f3339f24640 1 -- 172.21.15.89:0/2081440692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333419d460 con 0x7f3334102de0 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f3339f24640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334102de0 0x7f33341984b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f332c004770 tx=0x7f332c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f3332ffd640 1 -- 172.21.15.89:0/2081440692 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c03f070 con 0x7f3334102de0 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f3332ffd640 1 -- 172.21.15.89:0/2081440692 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f332c031ca0 con 0x7f3334102de0 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f333c188640 1 -- 172.21.15.89:0/2081440692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f333419d6f0 con 0x7f3334102de0 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f3332ffd640 1 -- 172.21.15.89:0/2081440692 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c035030 con 0x7f3334102de0 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.630+0000 7f333c188640 1 -- 172.21.15.89:0/2081440692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33341a5030 con 0x7f3334102de0 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.633+0000 7f3332ffd640 1 -- 172.21.15.89:0/2081440692 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f332c042070 con 0x7f3334102de0 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.633+0000 7f333e561740 1 Processor -- start 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.633+0000 7f333e561740 1 -- start start 2024-09-15T23:28:53.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.633+0000 7f333e561740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2060 0x5612574c2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.633+0000 7f333e561740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574c2980 0x5612574ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.633+0000 7f333e561740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.633+0000 7f333e561740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561257427620 con 0x5612574cd240 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.633+0000 7f333e561740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5612573cee50 con 0x5612574c2980 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.633+0000 7f333e561740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5612574235e0 con 0x5612574c2060 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3339723640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3339723640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574cf630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54856/0 (socket says 172.21.15.89:54856) 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3339723640 1 -- 172.21.15.89:0/931369905 learned_addr learned my addr 172.21.15.89:0/931369905 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f333a725640 1 --2- 172.21.15.89:0/931369905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2060 0x5612574c2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3332ffd640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f331c0a3240 0x7f331c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3339f24640 1 --2- 172.21.15.89:0/931369905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574c2980 0x5612574ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3339723640 1 -- 172.21.15.89:0/931369905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2060 msgr2=0x5612574c2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3339723640 1 --2- 172.21.15.89:0/931369905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2060 0x5612574c2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3339723640 1 -- 172.21.15.89:0/931369905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574c2980 msgr2=0x5612574ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3339723640 1 --2- 172.21.15.89:0/931369905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574c2980 0x5612574ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3339723640 1 -- 172.21.15.89:0/931369905 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612573bfd40 con 0x5612574cd240 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f333a725640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f331c0a3240 0x7f331c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3339723640 1 --2- 172.21.15.89:0/931369905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574cf630 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3328007dc0 tx=0x7f332800c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82c40fc820f5c4e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f3332ffd640 1 -- 172.21.15.89:0/2081440692 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f332c0f05a0 con 0x7f3334102de0 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f331bfff640 1 -- 172.21.15.89:0/931369905 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328015020 con 0x5612574cd240 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f333a725640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f331c0a3240 0x7f331c0a5700 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3324005d00 tx=0x7f33240046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f331bfff640 1 -- 172.21.15.89:0/931369905 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3328011b10 con 0x5612574cd240 2024-09-15T23:28:53.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.634+0000 7f331bfff640 1 -- 172.21.15.89:0/931369905 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328010ac0 con 0x5612574cd240 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.635+0000 7f333e561740 1 -- 172.21.15.89:0/931369905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 msgr2=0x5612574cf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.635+0000 7f333e561740 1 --2- 172.21.15.89:0/931369905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574cf630 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3328007dc0 tx=0x7f332800c560 comp rx=0 tx=0).stop 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.635+0000 7f333e561740 1 -- 172.21.15.89:0/931369905 shutdown_connections 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.635+0000 7f333e561740 1 --2- 172.21.15.89:0/931369905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574cf630 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.635+0000 7f333e561740 1 --2- 172.21.15.89:0/931369905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574c2980 0x5612574ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.635+0000 7f333e561740 1 --2- 172.21.15.89:0/931369905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2060 0x5612574c2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.635+0000 7f333e561740 1 -- 172.21.15.89:0/931369905 >> 172.21.15.89:0/931369905 conn(0x5612574be5f0 msgr2=0x5612574be9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.635+0000 7f333e561740 1 -- 172.21.15.89:0/931369905 shutdown_connections 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.635+0000 7f333e561740 1 -- 172.21.15.89:0/931369905 wait complete. 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333e561740 1 -- start start 2024-09-15T23:28:53.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333e561740 1 Processor -- start 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333e561740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333e561740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333e561740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 0x5612574cab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333e561740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5612574cfe10 con 0x5612574c2060 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333e561740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5612573aff10 con 0x5612574cd240 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333e561740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5612574cff90 con 0x5612574c2980 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333e561740 10 client.0 _ll_register_callbacks cb 0x5612573d3db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f3339f24640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f3339f24640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574ca0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54858/0 (socket says 172.21.15.89:54858) 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f3339f24640 1 -- 172.21.15.89:0/3944805432 learned_addr learned my addr 172.21.15.89:0/3944805432 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333a725640 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 0x5612574cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f3339f24640 1 -- 172.21.15.89:0/3944805432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 msgr2=0x5612574ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f3339f24640 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f3339f24640 1 -- 172.21.15.89:0/3944805432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 msgr2=0x5612574cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f3339f24640 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 0x5612574cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f3339f24640 1 -- 172.21.15.89:0/3944805432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561257136580 con 0x5612574c2060 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.636+0000 7f333a725640 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 0x5612574cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f3339f24640 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574ca0a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f332c0b8c30 tx=0x7f332c0bc5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d6ebac20bad366 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f33197fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f33197fa640 1 -- 172.21.15.89:0/3944805432 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c0ef070 con 0x5612574c2060 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f33197fa640 1 -- 172.21.15.89:0/3944805432 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f332c0b8e40 con 0x5612574c2060 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f333e561740 1 -- 172.21.15.89:0/3944805432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3328011b10 con 0x5612574c2060 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f33197fa640 1 -- 172.21.15.89:0/3944805432 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f332c049070 con 0x5612574c2060 2024-09-15T23:28:53.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f333e561740 10 client.17082 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f333e561740 1 -- 172.21.15.89:0/3944805432 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561257200960 con 0x5612574c2060 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f33197fa640 1 -- 172.21.15.89:0/3944805432 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3328011b10 con 0x5612574c2060 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f33197fa640 1 -- 172.21.15.89:0/3944805432 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f332c03b5f0 con 0x5612574c2060 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f333e561740 10 client.17082 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f333e561740 10 client.17082 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f333e561740 20 client.17082 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f333e561740 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5612574dccd0 0x5612574df160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f333e561740 4 client.17082 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.637+0000 7f333e561740 1 -- 172.21.15.89:0/3944805432 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561257434b90 con 0x5612574dccd0 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.638+0000 7f3339723640 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5612574dccd0 0x5612574df160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.638+0000 7f33197fa640 10 client.17082 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:53.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.638+0000 7f3339723640 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5612574dccd0 0x5612574df160 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8830a9b044eac30 server_cookie=8b36ab75378cc929 in_seq=0 out_seq=0 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f33197fa640 1 -- 172.21.15.89:0/3944805432 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561257434b90 con 0x5612574dccd0 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f33197fa640 10 client.17082 handle_command_reply: tid=0 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 client.17082 shutdown 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 -- 172.21.15.89:0/3944805432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 msgr2=0x5612574ca0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574ca0a0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f332c0b8c30 tx=0x7f332c0bc5a0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 -- 172.21.15.89:0/3944805432 shutdown_connections 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5612574dccd0 0x5612574df160 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 0x5612574cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f33197fa640 0 client.17082 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 --2- 172.21.15.89:0/3944805432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574ca0a0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 -- 172.21.15.89:0/3944805432 >> 172.21.15.89:0/3944805432 conn(0x5612574be5f0 msgr2=0x5612574c42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:53.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 -- 172.21.15.89:0/3944805432 shutdown_connections 2024-09-15T23:28:53.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 1 -- 172.21.15.89:0/3944805432 wait complete. 2024-09-15T23:28:53.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.639+0000 7f333e561740 20 client.17082 trim_cache size 0 max 16384 2024-09-15T23:28:53.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f333e561740 1 Processor -- start 2024-09-15T23:28:53.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f333e561740 1 -- start start 2024-09-15T23:28:53.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f333e561740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574bd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f333e561740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574e4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f333e561740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 0x5612574e4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f333e561740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f33341aeec0 con 0x5612574c2060 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f333e561740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f33341b0a00 con 0x5612574cd240 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f333e561740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f33341af590 con 0x5612574c2980 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f3339f24640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574bd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f3339f24640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574bd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54872/0 (socket says 172.21.15.89:54872) 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.656+0000 7f3339f24640 1 -- 172.21.15.89:0/3459344830 learned_addr learned my addr 172.21.15.89:0/3459344830 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f333a725640 1 --2- 172.21.15.89:0/3459344830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 0x5612574e4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f3339723640 1 --2- 172.21.15.89:0/3459344830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574e4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f3339f24640 1 -- 172.21.15.89:0/3459344830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 msgr2=0x5612574e4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f3339f24640 1 --2- 172.21.15.89:0/3459344830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574e4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f3339f24640 1 -- 172.21.15.89:0/3459344830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 msgr2=0x5612574e4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:53.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f3339f24640 1 --2- 172.21.15.89:0/3459344830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 0x5612574e4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f3339f24640 1 -- 172.21.15.89:0/3459344830 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612574c27b0 con 0x5612574c2060 2024-09-15T23:28:53.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f3339f24640 1 --2- 172.21.15.89:0/3459344830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574bd640 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f332c0edd00 tx=0x7f332c0bace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec6234f0958a72c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:53.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f331b7fe640 1 -- 172.21.15.89:0/3459344830 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c0ef070 con 0x5612574c2060 2024-09-15T23:28:53.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f331b7fe640 1 -- 172.21.15.89:0/3459344830 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f332c0c0b80 con 0x5612574c2060 2024-09-15T23:28:53.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.657+0000 7f331b7fe640 1 -- 172.21.15.89:0/3459344830 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c0bcca0 con 0x5612574c2060 2024-09-15T23:28:53.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 -- 172.21.15.89:0/3459344830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 msgr2=0x5612574bd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 --2- 172.21.15.89:0/3459344830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574bd640 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f332c0edd00 tx=0x7f332c0bace0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 -- 172.21.15.89:0/3459344830 shutdown_connections 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 --2- 172.21.15.89:0/3459344830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574cd240 0x5612574e4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 --2- 172.21.15.89:0/3459344830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574e4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 --2- 172.21.15.89:0/3459344830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574c2060 0x5612574bd640 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 -- 172.21.15.89:0/3459344830 >> 172.21.15.89:0/3459344830 conn(0x5612574be5f0 msgr2=0x5612574bcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 -- 172.21.15.89:0/3459344830 shutdown_connections 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 -- 172.21.15.89:0/3459344830 wait complete. 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 -- start start 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.658+0000 7f333e561740 1 Processor -- start 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333e561740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574c2060 0x5612574d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333e561740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574d26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333e561740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333e561740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f33341b0330 con 0x5612574cd240 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333e561740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f33341afc60 con 0x5612574c2060 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333e561740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f33341acea0 con 0x5612574c2980 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333e561740 10 client.0 _ll_register_callbacks cb 0x7f332c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333a725640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f3339723640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574d26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333a725640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574d2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54884/0 (socket says 172.21.15.89:54884) 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333a725640 1 -- 172.21.15.89:0/1789822745 learned_addr learned my addr 172.21.15.89:0/1789822745 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f3339f24640 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574c2060 0x5612574d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333a725640 1 -- 172.21.15.89:0/1789822745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 msgr2=0x5612574d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333a725640 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574d26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333a725640 1 -- 172.21.15.89:0/1789822745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574c2060 msgr2=0x5612574d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333a725640 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574c2060 0x5612574d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333a725640 1 -- 172.21.15.89:0/1789822745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561257547a50 con 0x5612574cd240 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f333a725640 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574d2c30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f332401f850 tx=0x7f332400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c2b9cd8f60951f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f33197fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.659+0000 7f33197fa640 1 -- 172.21.15.89:0/1789822745 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3324028440 con 0x5612574cd240 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f33197fa640 1 -- 172.21.15.89:0/1789822745 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f33240285e0 con 0x5612574cd240 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f333e561740 1 -- 172.21.15.89:0/1789822745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f332c0c0b80 con 0x5612574cd240 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f33197fa640 1 -- 172.21.15.89:0/1789822745 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3324031050 con 0x5612574cd240 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f333e561740 10 client.17088 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:53.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f333e561740 1 -- 172.21.15.89:0/1789822745 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561257548d30 con 0x5612574cd240 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f33197fa640 1 -- 172.21.15.89:0/1789822745 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f332c0c0b80 con 0x5612574cd240 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f33197fa640 1 -- 172.21.15.89:0/1789822745 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3324026020 con 0x5612574cd240 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f333e561740 10 client.17088 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f333e561740 10 client.17088 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f333e561740 20 client.17088 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f333e561740 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5612575494b0 0x56125754b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f333e561740 4 client.17088 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "735ccfd0", "release": true, "await": true}] 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f333e561740 1 -- 172.21.15.89:0/1789822745 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "735ccfd0", "release": true, "await": true}) -- 0x56125742f0a0 con 0x5612575494b0 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.660+0000 7f3339f24640 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5612575494b0 0x56125754b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.661+0000 7f33197fa640 10 client.17088 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:53.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.661+0000 7f3339f24640 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5612575494b0 0x56125754b8d0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=347fd83198968bf5 server_cookie=a8e093e13c999c74 in_seq=0 out_seq=0 2024-09-15T23:28:53.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.664+0000 7f33197fa640 1 -- 172.21.15.89:0/1789822745 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56125742f0a0 con 0x5612575494b0 2024-09-15T23:28:53.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.664+0000 7f33197fa640 10 client.17088 handle_command_reply: tid=0 2024-09-15T23:28:53.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.664+0000 7f333e561740 1 client.17088 shutdown 2024-09-15T23:28:53.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.664+0000 7f333e561740 1 -- 172.21.15.89:0/1789822745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 msgr2=0x5612574d2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:53.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.664+0000 7f333e561740 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574d2c30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f332401f850 tx=0x7f332400b260 comp rx=0 tx=0).stop 2024-09-15T23:28:53.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.664+0000 7f333e561740 1 -- 172.21.15.89:0/1789822745 shutdown_connections 2024-09-15T23:28:53.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333e561740 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5612575494b0 0x56125754b8d0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333e561740 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612574cd240 0x5612574d2c30 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333e561740 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612574c2980 0x5612574d26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f33197fa640 0 client.17088 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:53.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333e561740 1 --2- 172.21.15.89:0/1789822745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612574c2060 0x5612574d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333e561740 1 -- 172.21.15.89:0/1789822745 >> 172.21.15.89:0/1789822745 conn(0x5612574be5f0 msgr2=0x5612574ca880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:53.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333e561740 1 -- 172.21.15.89:0/1789822745 shutdown_connections 2024-09-15T23:28:53.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333e561740 1 -- 172.21.15.89:0/1789822745 wait complete. 2024-09-15T23:28:53.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333e561740 20 client.17088 trim_cache size 0 max 16384 2024-09-15T23:28:53.670 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:28:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:28:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:28:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 16, 2024-09-15T23:28:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:28:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "735ccfd0": { 2024-09-15T23:28:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 16, 2024-09-15T23:28:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:28:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:28:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:28:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:28:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:28:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:28:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 133.8, 2024-09-15T23:28:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:28:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:28:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:28:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:28:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:28:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:28:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:28:53.672 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:28:53.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 -- 172.21.15.89:0/2081440692 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f331c0a3240 msgr2=0x7f331c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:53.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f331c0a3240 0x7f331c0a5700 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3324005d00 tx=0x7f33240046c0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 -- 172.21.15.89:0/2081440692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334102de0 msgr2=0x7f33341984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:53.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334102de0 0x7f33341984b0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f332c004770 tx=0x7f332c003c10 comp rx=0 tx=0).stop 2024-09-15T23:28:53.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 -- 172.21.15.89:0/2081440692 shutdown_connections 2024-09-15T23:28:53.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f331c0a3240 0x7f331c0a5700 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f333410c180 0x7f333419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3334103700 0x7f33341989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 --2- 172.21.15.89:0/2081440692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3334102de0 0x7f33341984b0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:53.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 -- 172.21.15.89:0/2081440692 >> 172.21.15.89:0/2081440692 conn(0x7f33340fc7a0 msgr2=0x7f33340fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:53.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 -- 172.21.15.89:0/2081440692 shutdown_connections 2024-09-15T23:28:53.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:53.665+0000 7f333c188640 1 -- 172.21.15.89:0/2081440692 wait complete. 2024-09-15T23:28:54.461 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '735ccfd0', total seconds elapsed: 17.1 2024-09-15T23:28:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:55 smithi120 ceph-mon[35250]: pgmap v497: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-09-15T23:28:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:55 smithi174 ceph-mon[35586]: pgmap v497: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-09-15T23:28:55.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:55 smithi089 ceph-mon[30933]: pgmap v497: 129 pgs: 129 active+clean; 2.4 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-09-15T23:28:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:57 smithi120 ceph-mon[35250]: pgmap v498: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.1 KiB/s wr, 16 op/s 2024-09-15T23:28:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:57 smithi174 ceph-mon[35586]: pgmap v498: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.1 KiB/s wr, 16 op/s 2024-09-15T23:28:57.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:57 smithi089 ceph-mon[30933]: pgmap v498: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.1 KiB/s wr, 16 op/s 2024-09-15T23:28:57.922 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:28:58.183 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.649+0000 7f980f8e8640 1 -- 172.21.15.89:0/2991860361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9808106a70 msgr2=0x7f9808106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.649+0000 7f980f8e8640 1 --2- 172.21.15.89:0/2991860361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9808106a70 0x7f9808106e50 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f9804009930 tx=0x7f980402a420 comp rx=0 tx=0).stop 2024-09-15T23:28:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.649+0000 7f980f8e8640 1 -- 172.21.15.89:0/2991860361 shutdown_connections 2024-09-15T23:28:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.649+0000 7f980f8e8640 1 --2- 172.21.15.89:0/2991860361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9808101410 0x7f980810d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.649+0000 7f980f8e8640 1 --2- 172.21.15.89:0/2991860361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9808100a70 0x7f9808100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.649+0000 7f980f8e8640 1 --2- 172.21.15.89:0/2991860361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9808106a70 0x7f9808106e50 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.649+0000 7f980f8e8640 1 -- 172.21.15.89:0/2991860361 >> 172.21.15.89:0/2991860361 conn(0x7f98080fc7d0 msgr2=0x7f98080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.649+0000 7f980f8e8640 1 -- 172.21.15.89:0/2991860361 shutdown_connections 2024-09-15T23:28:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.649+0000 7f980f8e8640 1 -- 172.21.15.89:0/2991860361 wait complete. 2024-09-15T23:28:58.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.649+0000 7f980f8e8640 1 Processor -- start 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980f8e8640 1 -- start start 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980f8e8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9808100a70 0x7f98081973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980f8e8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9808101410 0x7f98081978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980f8e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9808106a70 0x7f980819bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980f8e8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9808110e70 con 0x7f9808100a70 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980f8e8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9808110cf0 con 0x7f9808101410 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980f8e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9808110ff0 con 0x7f9808106a70 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980d684640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9808100a70 0x7f98081973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980d684640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9808100a70 0x7f98081973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42536/0 (socket says 172.21.15.89:42536) 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980d684640 1 -- 172.21.15.89:0/3129912147 learned_addr learned my addr 172.21.15.89:0/3129912147 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980de85640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9808106a70 0x7f980819bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980ce83640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9808101410 0x7f98081978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980d684640 1 -- 172.21.15.89:0/3129912147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9808106a70 msgr2=0x7f980819bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980d684640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9808106a70 0x7f980819bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980d684640 1 -- 172.21.15.89:0/3129912147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9808101410 msgr2=0x7f98081978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980d684640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9808101410 0x7f98081978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.650+0000 7f980d684640 1 -- 172.21.15.89:0/3129912147 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f980819c360 con 0x7f9808100a70 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.651+0000 7f980d684640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9808100a70 0x7f98081973b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f9804009230 tx=0x7f9804004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.651+0000 7f97fe7fc640 1 -- 172.21.15.89:0/3129912147 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980403f030 con 0x7f9808100a70 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.651+0000 7f97fe7fc640 1 -- 172.21.15.89:0/3129912147 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9804004440 con 0x7f9808100a70 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.651+0000 7f980f8e8640 1 -- 172.21.15.89:0/3129912147 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f980819c5f0 con 0x7f9808100a70 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.651+0000 7f97fe7fc640 1 -- 172.21.15.89:0/3129912147 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804035070 con 0x7f9808100a70 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.651+0000 7f980f8e8640 1 -- 172.21.15.89:0/3129912147 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98081a3f30 con 0x7f9808100a70 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.653+0000 7f97fe7fc640 1 -- 172.21.15.89:0/3129912147 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9804042030 con 0x7f9808100a70 2024-09-15T23:28:58.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f9811cc1740 1 Processor -- start 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f9811cc1740 1 -- start start 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f97fe7fc640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97f40a32c0 0x7f97f40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 0x564b4b9c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9d4740 0x564b4b9d6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f9811cc1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b4b92b2e0 con 0x564b4b9c8d00 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f9811cc1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b4b8d2ce0 con 0x564b4b9d4740 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f9811cc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b4b9272a0 con 0x564b4b9c7ad0 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f97fe7fc640 1 -- 172.21.15.89:0/3129912147 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f98040eff40 con 0x7f9808100a70 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f980d684640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97f40a32c0 0x7f97f40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f980de85640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f980ce83640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 0x564b4b9c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f980ce83640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 0x564b4b9c7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:33744/0 (socket says 172.21.15.89:33744) 2024-09-15T23:28:58.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f980ce83640 1 -- 172.21.15.89:0/1156578902 learned_addr learned my addr 172.21.15.89:0/1156578902 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f980de85640 1 --2- 172.21.15.89:0/1156578902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9d4740 0x564b4b9d6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f980ce83640 1 -- 172.21.15.89:0/1156578902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9d4740 msgr2=0x564b4b9d6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.654+0000 7f980ce83640 1 --2- 172.21.15.89:0/1156578902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9d4740 0x564b4b9d6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.655+0000 7f980ce83640 1 -- 172.21.15.89:0/1156578902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 msgr2=0x564b4b9c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.655+0000 7f980ce83640 1 --2- 172.21.15.89:0/1156578902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.655+0000 7f980ce83640 1 -- 172.21.15.89:0/1156578902 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b4b8b7440 con 0x564b4b9c7ad0 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.655+0000 7f980de85640 1 --2- 172.21.15.89:0/1156578902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.655+0000 7f980d684640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97f40a32c0 0x7f97f40a5780 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f98040b8bf0 tx=0x7f980402a420 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.655+0000 7f980ce83640 1 --2- 172.21.15.89:0/1156578902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 0x564b4b9c7f30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f97f8009ca0 tx=0x7f97f8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f188ecb8e2d131d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.655+0000 7f97f37fe640 1 -- 172.21.15.89:0/1156578902 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f801e070 con 0x564b4b9c7ad0 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.655+0000 7f97f37fe640 1 -- 172.21.15.89:0/1156578902 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f8019b10 con 0x564b4b9c7ad0 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.655+0000 7f97f37fe640 1 -- 172.21.15.89:0/1156578902 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f800ea10 con 0x564b4b9c7ad0 2024-09-15T23:28:58.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 -- 172.21.15.89:0/1156578902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 msgr2=0x564b4b9c7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 --2- 172.21.15.89:0/1156578902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 0x564b4b9c7f30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f97f8009ca0 tx=0x7f97f8011810 comp rx=0 tx=0).stop 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 -- 172.21.15.89:0/1156578902 shutdown_connections 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 --2- 172.21.15.89:0/1156578902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9d4740 0x564b4b9d6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 --2- 172.21.15.89:0/1156578902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 0x564b4b9c7f30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 --2- 172.21.15.89:0/1156578902 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 -- 172.21.15.89:0/1156578902 >> 172.21.15.89:0/1156578902 conn(0x564b4b9c3c40 msgr2=0x564b4b9c4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 -- 172.21.15.89:0/1156578902 shutdown_connections 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 -- 172.21.15.89:0/1156578902 wait complete. 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 -- start start 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.656+0000 7f9811cc1740 1 Processor -- start 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 0x564b4b9d4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c8d00 0x564b4b9d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9d4740 0x564b4b9d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f9811cc1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b4b9d7650 con 0x564b4b9d4740 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f9811cc1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b4b8e6050 con 0x564b4b9c8d00 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f9811cc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b4b9d77d0 con 0x564b4b9c7ad0 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f9811cc1740 10 client.0 _ll_register_callbacks cb 0x564b4b926160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980de85640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9d4740 0x564b4b9d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980ce83640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c8d00 0x564b4b9d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980de85640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9d4740 0x564b4b9d1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42550/0 (socket says 172.21.15.89:42550) 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980de85640 1 -- 172.21.15.89:0/3873853948 learned_addr learned my addr 172.21.15.89:0/3873853948 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980de85640 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 0x564b4b9d4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980ce83640 1 -- 172.21.15.89:0/3873853948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 msgr2=0x564b4b9d4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980ce83640 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 0x564b4b9d4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980ce83640 1 -- 172.21.15.89:0/3873853948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9d4740 msgr2=0x564b4b9d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980ce83640 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9d4740 0x564b4b9d1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980ce83640 1 -- 172.21.15.89:0/3873853948 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b4b69c3c0 con 0x564b4b9c8d00 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980de85640 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9d4740 0x564b4b9d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f980ce83640 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c8d00 0x564b4b9d1440 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f97f8011d10 tx=0x7f97f8005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a7173db531643bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.657+0000 7f97f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f97f0ff9640 1 -- 172.21.15.89:0/3873853948 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f8027020 con 0x564b4b9c8d00 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f9811cc1740 1 -- 172.21.15.89:0/3873853948 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97f8019b10 con 0x564b4b9c8d00 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f97f0ff9640 1 -- 172.21.15.89:0/3873853948 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f97f8005460 con 0x564b4b9c8d00 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f97f0ff9640 1 -- 172.21.15.89:0/3873853948 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f97f8032050 con 0x564b4b9c8d00 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f9811cc1740 10 client.26009 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f9811cc1740 1 -- 172.21.15.89:0/3873853948 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b4b9e3160 con 0x564b4b9c8d00 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f97f0ff9640 1 -- 172.21.15.89:0/3873853948 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f8019b10 con 0x564b4b9c8d00 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f97f0ff9640 1 -- 172.21.15.89:0/3873853948 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f97f80046e0 con 0x564b4b9c8d00 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f9811cc1740 10 client.26009 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:58.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f9811cc1740 10 client.26009 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f9811cc1740 20 client.26009 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f9811cc1740 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564b4b9e45c0 0x564b4b9e69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f9811cc1740 4 client.26009 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.658+0000 7f9811cc1740 1 -- 172.21.15.89:0/3873853948 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b4b938850 con 0x564b4b9e45c0 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.659+0000 7f980de85640 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564b4b9e45c0 0x564b4b9e69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.659+0000 7f980de85640 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564b4b9e45c0 0x564b4b9e69b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1987852e3a53a6c8 server_cookie=a208fd2470254bcd in_seq=0 out_seq=0 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.659+0000 7f97f0ff9640 10 client.26009 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.659+0000 7f97f0ff9640 1 -- 172.21.15.89:0/3873853948 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564b4b938850 con 0x564b4b9e45c0 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.659+0000 7f97f0ff9640 10 client.26009 handle_command_reply: tid=0 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.659+0000 7f9811cc1740 1 client.26009 shutdown 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 1 -- 172.21.15.89:0/3873853948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c8d00 msgr2=0x564b4b9d1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c8d00 0x564b4b9d1440 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f97f8011d10 tx=0x7f97f8005db0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 1 -- 172.21.15.89:0/3873853948 shutdown_connections 2024-09-15T23:28:58.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564b4b9e45c0 0x564b4b9e69b0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9d4740 0x564b4b9d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c8d00 0x564b4b9d1440 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f97f0ff9640 0 client.26009 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:58.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 1 --2- 172.21.15.89:0/3873853948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9c7ad0 0x564b4b9d4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 1 -- 172.21.15.89:0/3873853948 >> 172.21.15.89:0/3873853948 conn(0x564b4b9c3c40 msgr2=0x564b4b9c99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:58.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 1 -- 172.21.15.89:0/3873853948 shutdown_connections 2024-09-15T23:28:58.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 1 -- 172.21.15.89:0/3873853948 wait complete. 2024-09-15T23:28:58.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.660+0000 7f9811cc1740 20 client.26009 trim_cache size 0 max 16384 2024-09-15T23:28:58.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.677+0000 7f9811cc1740 1 Processor -- start 2024-09-15T23:28:58.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.677+0000 7f9811cc1740 1 -- start start 2024-09-15T23:28:58.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.677+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.677+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.677+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9d4740 0x564b4b9d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.677+0000 7f9811cc1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98081acbd0 con 0x564b4b9c8d00 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.677+0000 7f9811cc1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98081ae710 con 0x564b4b9c7ad0 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.677+0000 7f9811cc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98081ad2a0 con 0x564b4b9d4740 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.677+0000 7f980ce83640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980de85640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980de85640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:48668/0 (socket says 172.21.15.89:48668) 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980de85640 1 -- 172.21.15.89:0/961595146 learned_addr learned my addr 172.21.15.89:0/961595146 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980ce83640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42566/0 (socket says 172.21.15.89:42566) 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980de85640 1 --2- 172.21.15.89:0/961595146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9d4740 0x564b4b9d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980de85640 1 -- 172.21.15.89:0/961595146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9d4740 msgr2=0x564b4b9d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980de85640 1 --2- 172.21.15.89:0/961595146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9d4740 0x564b4b9d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980de85640 1 -- 172.21.15.89:0/961595146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 msgr2=0x564b4b9c0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980de85640 1 --2- 172.21.15.89:0/961595146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980de85640 1 -- 172.21.15.89:0/961595146 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b4b9d1f40 con 0x564b4b9c7ad0 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980ce83640 1 --2- 172.21.15.89:0/961595146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f980de85640 1 --2- 172.21.15.89:0/961595146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9cef20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9800005fd0 tx=0x7f9800009e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56e00a79c220d4fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f97f2ffd640 1 -- 172.21.15.89:0/961595146 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9800004b20 con 0x564b4b9c7ad0 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f97f2ffd640 1 -- 172.21.15.89:0/961595146 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f980002c070 con 0x564b4b9c7ad0 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.678+0000 7f97f2ffd640 1 -- 172.21.15.89:0/961595146 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9800004410 con 0x564b4b9c7ad0 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.679+0000 7f9811cc1740 1 -- 172.21.15.89:0/961595146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 msgr2=0x564b4b9cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.679+0000 7f9811cc1740 1 --2- 172.21.15.89:0/961595146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9cef20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9800005fd0 tx=0x7f9800009e00 comp rx=0 tx=0).stop 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.679+0000 7f9811cc1740 1 -- 172.21.15.89:0/961595146 shutdown_connections 2024-09-15T23:28:58.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.679+0000 7f9811cc1740 1 --2- 172.21.15.89:0/961595146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9d4740 0x564b4b9d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.679+0000 7f9811cc1740 1 --2- 172.21.15.89:0/961595146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.679+0000 7f9811cc1740 1 --2- 172.21.15.89:0/961595146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9cef20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.679+0000 7f9811cc1740 1 -- 172.21.15.89:0/961595146 >> 172.21.15.89:0/961595146 conn(0x564b4b9c3c40 msgr2=0x564b4b9cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.679+0000 7f9811cc1740 1 -- 172.21.15.89:0/961595146 shutdown_connections 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.679+0000 7f9811cc1740 1 -- 172.21.15.89:0/961595146 wait complete. 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.679+0000 7f9811cc1740 1 -- start start 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f9811cc1740 1 Processor -- start 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9dd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9eb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f9811cc1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9d4740 0x564b4b9ebe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f9811cc1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98081ae040 con 0x564b4b9c8d00 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f9811cc1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98081ad970 con 0x564b4b9c7ad0 2024-09-15T23:28:58.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f9811cc1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98081aabb0 con 0x564b4b9d4740 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f9811cc1740 10 client.0 _ll_register_callbacks cb 0x7f97f801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f980ce83640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9eb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f980de85640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9dd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f980ce83640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9eb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42580/0 (socket says 172.21.15.89:42580) 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f980de85640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9dd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:48680/0 (socket says 172.21.15.89:48680) 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f980ce83640 1 -- 172.21.15.89:0/1675961063 learned_addr learned my addr 172.21.15.89:0/1675961063 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.680+0000 7f980de85640 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9d4740 0x564b4b9ebe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f980ce83640 1 -- 172.21.15.89:0/1675961063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9d4740 msgr2=0x564b4b9ebe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f980ce83640 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9d4740 0x564b4b9ebe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f980ce83640 1 -- 172.21.15.89:0/1675961063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 msgr2=0x564b4b9dd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f980ce83640 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9dd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f980ce83640 1 -- 172.21.15.89:0/1675961063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b4b9ec530 con 0x564b4b9c8d00 2024-09-15T23:28:58.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f980de85640 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9dd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f980ce83640 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9eb950 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f97f800e640 tx=0x7f97f8009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc85f1fa498e949a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f97f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f97f0ff9640 1 -- 172.21.15.89:0/1675961063 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f8027040 con 0x564b4b9c8d00 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f9811cc1740 1 -- 172.21.15.89:0/1675961063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f980002c070 con 0x564b4b9c8d00 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f97f0ff9640 1 -- 172.21.15.89:0/1675961063 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f97f8004d20 con 0x564b4b9c8d00 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f97f0ff9640 1 -- 172.21.15.89:0/1675961063 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f97f801e070 con 0x564b4b9c8d00 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f9811cc1740 10 client.17106 fetch_fsmap learned FSMap version 29 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f9811cc1740 1 -- 172.21.15.89:0/1675961063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b4ba4f000 con 0x564b4b9c8d00 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.681+0000 7f97f0ff9640 1 -- 172.21.15.89:0/1675961063 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f980002c070 con 0x564b4b9c8d00 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.682+0000 7f97f0ff9640 1 -- 172.21.15.89:0/1675961063 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f97f800ec10 con 0x564b4b9c8d00 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.682+0000 7f9811cc1740 10 client.17106 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.682+0000 7f9811cc1740 10 client.17106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:58.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.682+0000 7f9811cc1740 20 client.17106 populate_metadata read hostname 'smithi089' 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.682+0000 7f9811cc1740 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564b4ba4f770 0x564b4ba51b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.682+0000 7f9811cc1740 4 client.17106 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.682+0000 7f9811cc1740 1 -- 172.21.15.89:0/1675961063 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564b4b932d60 con 0x564b4ba4f770 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.682+0000 7f980de85640 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564b4ba4f770 0x564b4ba51b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.682+0000 7f97f0ff9640 10 client.17106 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.682+0000 7f980de85640 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564b4ba4f770 0x564b4ba51b90 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=754716e3c7450a73 server_cookie=15bcb86edaee7509 in_seq=0 out_seq=0 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f97f0ff9640 1 -- 172.21.15.89:0/1675961063 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564b4b932d60 con 0x564b4ba4f770 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f97f0ff9640 10 client.17106 handle_command_reply: tid=0 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 client.17106 shutdown 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 -- 172.21.15.89:0/1675961063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 msgr2=0x564b4b9eb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9eb950 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f97f800e640 tx=0x7f97f8009ba0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 -- 172.21.15.89:0/1675961063 shutdown_connections 2024-09-15T23:28:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564b4ba4f770 0x564b4ba51b90 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b4b9d4740 0x564b4b9ebe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b4b9c8d00 0x564b4b9eb950 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 --2- 172.21.15.89:0/1675961063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b4b9c7ad0 0x564b4b9dd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f97f0ff9640 0 client.17106 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:28:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 -- 172.21.15.89:0/1675961063 >> 172.21.15.89:0/1675961063 conn(0x564b4b9c3c40 msgr2=0x564b4b9d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 -- 172.21.15.89:0/1675961063 shutdown_connections 2024-09-15T23:28:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 1 -- 172.21.15.89:0/1675961063 wait complete. 2024-09-15T23:28:58.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.683+0000 7f9811cc1740 20 client.17106 trim_cache size 0 max 16384 2024-09-15T23:28:58.693 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:28:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:28:58.694 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:28:58.694 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 -- 172.21.15.89:0/3129912147 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97f40a32c0 msgr2=0x7f97f40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97f40a32c0 0x7f97f40a5780 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f98040b8bf0 tx=0x7f980402a420 comp rx=0 tx=0).stop 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 -- 172.21.15.89:0/3129912147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9808100a70 msgr2=0x7f98081973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9808100a70 0x7f98081973b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f9804009230 tx=0x7f9804004040 comp rx=0 tx=0).stop 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 -- 172.21.15.89:0/3129912147 shutdown_connections 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97f40a32c0 0x7f97f40a5780 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9808106a70 0x7f980819bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9808101410 0x7f98081978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 --2- 172.21.15.89:0/3129912147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9808100a70 0x7f98081973b0 secure :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f9804009230 tx=0x7f9804004040 comp rx=0 tx=0).stop 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 -- 172.21.15.89:0/3129912147 >> 172.21.15.89:0/3129912147 conn(0x7f98080fc7d0 msgr2=0x7f98080fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 -- 172.21.15.89:0/3129912147 shutdown_connections 2024-09-15T23:28:58.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:28:58.684+0000 7f980f8e8640 1 -- 172.21.15.89:0/3129912147 wait complete. 2024-09-15T23:28:58.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:58 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:28:58.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:58 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:28:59.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:58 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:28:59.555 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:28:59.555 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:28:59.801 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:59.922 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:28:59.922 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:28:59.922 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:28:59.922 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:28:59 smithi089 ceph-mon[30933]: pgmap v499: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-09-15T23:28:59.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:28:59.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:28:59.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:28:59.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:28:59 smithi120 ceph-mon[35250]: pgmap v499: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-09-15T23:28:59.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:28:59.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:28:59.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:28:59.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:28:59 smithi174 ceph-mon[35586]: pgmap v499: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-09-15T23:29:00.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 -- 172.21.15.89:0/3588192650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c1002f0 msgr2=0x7f9e5c1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:00.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 --2- 172.21.15.89:0/3588192650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c1002f0 0x7f9e5c1006d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009930 tx=0x7f9e5802a420 comp rx=0 tx=0).stop 2024-09-15T23:29:00.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 -- 172.21.15.89:0/3588192650 shutdown_connections 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 --2- 172.21.15.89:0/3588192650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c106f50 0x7f9e5c10eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 --2- 172.21.15.89:0/3588192650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e5c100ca0 0x7f9e5c106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 --2- 172.21.15.89:0/3588192650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c1002f0 0x7f9e5c1006d0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 -- 172.21.15.89:0/3588192650 >> 172.21.15.89:0/3588192650 conn(0x7f9e5c077210 msgr2=0x7f9e5c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 -- 172.21.15.89:0/3588192650 shutdown_connections 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 -- 172.21.15.89:0/3588192650 wait complete. 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 Processor -- start 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 -- start start 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c1002f0 0x7f9e5c101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e5c100ca0 0x7f9e5c1017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c106f50 0x7f9e5c101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c111fa0 con 0x7f9e5c1002f0 2024-09-15T23:29:00.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c111e20 con 0x7f9e5c106f50 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.277+0000 7f9e6326d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c112120 con 0x7f9e5c100ca0 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e61009640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c1002f0 0x7f9e5c101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e61009640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c1002f0 0x7f9e5c101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42600/0 (socket says 172.21.15.89:42600) 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e61009640 1 -- 172.21.15.89:0/1862873688 learned_addr learned my addr 172.21.15.89:0/1862873688 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e60808640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e5c100ca0 0x7f9e5c1017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e6180a640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c106f50 0x7f9e5c101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e61009640 1 -- 172.21.15.89:0/1862873688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e5c100ca0 msgr2=0x7f9e5c1017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e61009640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e5c100ca0 0x7f9e5c1017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e61009640 1 -- 172.21.15.89:0/1862873688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c106f50 msgr2=0x7f9e5c101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e61009640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c106f50 0x7f9e5c101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e61009640 1 -- 172.21.15.89:0/1862873688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5c1a69c0 con 0x7f9e5c1002f0 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e6180a640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c106f50 0x7f9e5c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:00.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e61009640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c1002f0 0x7f9e5c101270 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009230 tx=0x7f9e58004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:00.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e4e7fc640 1 -- 172.21.15.89:0/1862873688 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5803f030 con 0x7f9e5c1002f0 2024-09-15T23:29:00.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e4e7fc640 1 -- 172.21.15.89:0/1862873688 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e58003ad0 con 0x7f9e5c1002f0 2024-09-15T23:29:00.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e4e7fc640 1 -- 172.21.15.89:0/1862873688 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58035070 con 0x7f9e5c1002f0 2024-09-15T23:29:00.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e6326d640 1 -- 172.21.15.89:0/1862873688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e5c1a6c50 con 0x7f9e5c1002f0 2024-09-15T23:29:00.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.278+0000 7f9e6326d640 1 -- 172.21.15.89:0/1862873688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e5c1a70e0 con 0x7f9e5c1002f0 2024-09-15T23:29:00.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.279+0000 7f9e6326d640 1 -- 172.21.15.89:0/1862873688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e5c1006d0 con 0x7f9e5c1002f0 2024-09-15T23:29:00.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.280+0000 7f9e4e7fc640 1 -- 172.21.15.89:0/1862873688 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9e58042030 con 0x7f9e5c1002f0 2024-09-15T23:29:00.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.281+0000 7f9e4e7fc640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e480a3240 0x7f9e480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:00.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.281+0000 7f9e4e7fc640 1 -- 172.21.15.89:0/1862873688 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9e580f0600 con 0x7f9e5c1002f0 2024-09-15T23:29:00.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.281+0000 7f9e60808640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e480a3240 0x7f9e480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:00.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.282+0000 7f9e60808640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e480a3240 0x7f9e480a5700 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c102d00 tx=0x7f9e5000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:00.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.284+0000 7f9e4e7fc640 1 -- 172.21.15.89:0/1862873688 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9e580bcf30 con 0x7f9e5c1002f0 2024-09-15T23:29:00.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.483+0000 7f9e6326d640 1 -- 172.21.15.89:0/1862873688 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e5c1027a0 con 0x7f9e5c1002f0 2024-09-15T23:29:00.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.483+0000 7f9e4e7fc640 1 -- 172.21.15.89:0/1862873688 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f9e580bc830 con 0x7f9e5c1002f0 2024-09-15T23:29:00.485 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:29:00.485 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:29:00.485 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:29:00.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 -- 172.21.15.89:0/1862873688 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e480a3240 msgr2=0x7f9e480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:00.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e480a3240 0x7f9e480a5700 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c102d00 tx=0x7f9e5000b390 comp rx=0 tx=0).stop 2024-09-15T23:29:00.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 -- 172.21.15.89:0/1862873688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c1002f0 msgr2=0x7f9e5c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:00.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c1002f0 0x7f9e5c101270 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009230 tx=0x7f9e58004270 comp rx=0 tx=0).stop 2024-09-15T23:29:00.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 -- 172.21.15.89:0/1862873688 shutdown_connections 2024-09-15T23:29:00.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e480a3240 0x7f9e480a5700 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:00.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e5c106f50 0x7f9e5c101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:00.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e5c100ca0 0x7f9e5c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:00.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 --2- 172.21.15.89:0/1862873688 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9e5c1002f0 0x7f9e5c101270 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:00.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 -- 172.21.15.89:0/1862873688 >> 172.21.15.89:0/1862873688 conn(0x7f9e5c077210 msgr2=0x7f9e5c105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:00.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 -- 172.21.15.89:0/1862873688 shutdown_connections 2024-09-15T23:29:00.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:00.486+0000 7f9e6326d640 1 -- 172.21.15.89:0/1862873688 wait complete. 2024-09-15T23:29:01.267 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:29:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:01 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1862873688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:01 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1862873688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:01.514 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:01.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:01 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1862873688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:02.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.007+0000 7f33a4669640 1 -- 172.21.15.89:0/3403775549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f339c106a80 msgr2=0x7f339c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:02.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.007+0000 7f33a4669640 1 --2- 172.21.15.89:0/3403775549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f339c106a80 0x7f339c106e60 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f3398009930 tx=0x7f339802a420 comp rx=0 tx=0).stop 2024-09-15T23:29:02.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.007+0000 7f33a4669640 1 -- 172.21.15.89:0/3403775549 shutdown_connections 2024-09-15T23:29:02.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.007+0000 7f33a4669640 1 --2- 172.21.15.89:0/3403775549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339c101420 0x7f339c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.007+0000 7f33a4669640 1 --2- 172.21.15.89:0/3403775549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f339c100a80 0x7f339c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.007+0000 7f33a4669640 1 --2- 172.21.15.89:0/3403775549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f339c106a80 0x7f339c106e60 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.007+0000 7f33a4669640 1 -- 172.21.15.89:0/3403775549 >> 172.21.15.89:0/3403775549 conn(0x7f339c0fc7a0 msgr2=0x7f339c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:02.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.007+0000 7f33a4669640 1 -- 172.21.15.89:0/3403775549 shutdown_connections 2024-09-15T23:29:02.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.007+0000 7f33a4669640 1 -- 172.21.15.89:0/3403775549 wait complete. 2024-09-15T23:29:02.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a4669640 1 Processor -- start 2024-09-15T23:29:02.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a4669640 1 -- start start 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a4669640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339c100a80 0x7f339c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a4669640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f339c101420 0x7f339c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a4669640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f339c106a80 0x7f339c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a4669640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f339c110e80 con 0x7f339c106a80 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a4669640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f339c110d00 con 0x7f339c100a80 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a4669640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f339c111000 con 0x7f339c101420 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a2c06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f339c106a80 0x7f339c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a2c06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f339c106a80 0x7f339c19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42622/0 (socket says 172.21.15.89:42622) 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a2c06640 1 -- 172.21.15.89:0/771590496 learned_addr learned my addr 172.21.15.89:0/771590496 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a2405640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339c100a80 0x7f339c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a1c04640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f339c101420 0x7f339c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a2c06640 1 -- 172.21.15.89:0/771590496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f339c101420 msgr2=0x7f339c1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:02.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a2c06640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f339c101420 0x7f339c1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a2c06640 1 -- 172.21.15.89:0/771590496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339c100a80 msgr2=0x7f339c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:02.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a2c06640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339c100a80 0x7f339c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a2c06640 1 -- 172.21.15.89:0/771590496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339c19d440 con 0x7f339c106a80 2024-09-15T23:29:02.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a1c04640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f339c101420 0x7f339c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:02.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.008+0000 7f33a2405640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339c100a80 0x7f339c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:02.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.009+0000 7f33a2c06640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f339c106a80 0x7f339c19cd60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f339400cad0 tx=0x7f339400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.009+0000 7f33937fe640 1 -- 172.21.15.89:0/771590496 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339400ae20 con 0x7f339c106a80 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.009+0000 7f33a4669640 1 -- 172.21.15.89:0/771590496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f339c19d730 con 0x7f339c106a80 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.009+0000 7f33937fe640 1 -- 172.21.15.89:0/771590496 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3394005020 con 0x7f339c106a80 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.009+0000 7f33937fe640 1 -- 172.21.15.89:0/771590496 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339400f6f0 con 0x7f339c106a80 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.009+0000 7f33a4669640 1 -- 172.21.15.89:0/771590496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f339c1a5010 con 0x7f339c106a80 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.011+0000 7f33937fe640 1 -- 172.21.15.89:0/771590496 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f33940051d0 con 0x7f339c106a80 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a6a42740 1 Processor -- start 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a6a42740 1 -- start start 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33937fe640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33880a3240 0x7f33880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b41b40 0x5651e0b40090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b405d0 0x5651e0b40a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b4b370 0x5651e0b4d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a6a42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5651e0aa62f0 con 0x5651e0b41b40 2024-09-15T23:29:02.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a6a42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5651e0a4dcf0 con 0x5651e0b4b370 2024-09-15T23:29:02.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a6a42740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5651e0aa22b0 con 0x5651e0b405d0 2024-09-15T23:29:02.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a2405640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b41b40 0x5651e0b40090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a1c04640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33880a3240 0x7f33880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a2405640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b41b40 0x5651e0b40090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42636/0 (socket says 172.21.15.89:42636) 2024-09-15T23:29:02.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33a2405640 1 -- 172.21.15.89:0/525926032 learned_addr learned my addr 172.21.15.89:0/525926032 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:02.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.012+0000 7f33937fe640 1 -- 172.21.15.89:0/771590496 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f33940cfe40 con 0x7f339c106a80 2024-09-15T23:29:02.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f33a2c06640 1 --2- 172.21.15.89:0/525926032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b4b370 0x5651e0b4d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f33a1c04640 1 --2- 172.21.15.89:0/525926032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b405d0 0x5651e0b40a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f33a2405640 1 -- 172.21.15.89:0/525926032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b405d0 msgr2=0x5651e0b40a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f33a2405640 1 --2- 172.21.15.89:0/525926032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b405d0 0x5651e0b40a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f33a2405640 1 -- 172.21.15.89:0/525926032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b4b370 msgr2=0x5651e0b4d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f33a2405640 1 --2- 172.21.15.89:0/525926032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b4b370 0x5651e0b4d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f33a2405640 1 -- 172.21.15.89:0/525926032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651e0a323a0 con 0x5651e0b41b40 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f33a2c06640 1 --2- 172.21.15.89:0/525926032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b4b370 0x5651e0b4d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f33a1c04640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33880a3240 0x7f33880a5700 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f338c005fd0 tx=0x7f338c014840 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f33a2405640 1 --2- 172.21.15.89:0/525926032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b41b40 0x5651e0b40090 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f339802a940 tx=0x7f33980316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5d39089b259c98 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f3390ff9640 1 -- 172.21.15.89:0/525926032 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339803f070 con 0x5651e0b41b40 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f3390ff9640 1 -- 172.21.15.89:0/525926032 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33980035f0 con 0x5651e0b41b40 2024-09-15T23:29:02.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.013+0000 7f3390ff9640 1 -- 172.21.15.89:0/525926032 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398035070 con 0x5651e0b41b40 2024-09-15T23:29:02.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 -- 172.21.15.89:0/525926032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b41b40 msgr2=0x5651e0b40090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:02.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 --2- 172.21.15.89:0/525926032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b41b40 0x5651e0b40090 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f339802a940 tx=0x7f33980316e0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 -- 172.21.15.89:0/525926032 shutdown_connections 2024-09-15T23:29:02.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 --2- 172.21.15.89:0/525926032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b4b370 0x5651e0b4d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 --2- 172.21.15.89:0/525926032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b405d0 0x5651e0b40a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 --2- 172.21.15.89:0/525926032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b41b40 0x5651e0b40090 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 -- 172.21.15.89:0/525926032 >> 172.21.15.89:0/525926032 conn(0x5651e0b09310 msgr2=0x5651e0b3cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:02.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 -- 172.21.15.89:0/525926032 shutdown_connections 2024-09-15T23:29:02.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 -- 172.21.15.89:0/525926032 wait complete. 2024-09-15T23:29:02.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 -- start start 2024-09-15T23:29:02.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.014+0000 7f33a6a42740 1 Processor -- start 2024-09-15T23:29:02.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b405d0 0x5651e0b4ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 0x5651e0b48020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b4b370 0x5651e0b48610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a6a42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5651e0b4e010 con 0x5651e0b4b370 2024-09-15T23:29:02.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a6a42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5651e0a61060 con 0x5651e0b41b40 2024-09-15T23:29:02.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a6a42740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5651e0b4e190 con 0x5651e0b405d0 2024-09-15T23:29:02.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a6a42740 10 client.0 _ll_register_callbacks cb 0x5651e07b48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:02.051 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:29:02.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a2c06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b4b370 0x5651e0b48610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a2c06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b4b370 0x5651e0b48610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42638/0 (socket says 172.21.15.89:42638) 2024-09-15T23:29:02.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a2c06640 1 -- 172.21.15.89:0/652968462 learned_addr learned my addr 172.21.15.89:0/652968462 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a2405640 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b405d0 0x5651e0b4ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a1c04640 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 0x5651e0b48020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a2c06640 1 -- 172.21.15.89:0/652968462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b405d0 msgr2=0x5651e0b4ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a2c06640 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b405d0 0x5651e0b4ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a2c06640 1 -- 172.21.15.89:0/652968462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 msgr2=0x5651e0b48020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a2c06640 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 0x5651e0b48020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a2c06640 1 -- 172.21.15.89:0/652968462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651e08173c0 con 0x5651e0b4b370 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a2c06640 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b4b370 0x5651e0b48610 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f339409bfa0 tx=0x7f339409c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cb8de24a72a781 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f3385ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f3385ffb640 1 -- 172.21.15.89:0/652968462 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339401d040 con 0x5651e0b4b370 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f3385ffb640 1 -- 172.21.15.89:0/652968462 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3394005560 con 0x5651e0b4b370 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a6a42740 1 -- 172.21.15.89:0/652968462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33980035f0 con 0x5651e0b4b370 2024-09-15T23:29:02.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f3385ffb640 1 -- 172.21.15.89:0/652968462 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f33940bc110 con 0x5651e0b4b370 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a6a42740 10 client.17124 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.015+0000 7f33a6a42740 1 -- 172.21.15.89:0/652968462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5651e0b59b10 con 0x5651e0b4b370 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.016+0000 7f3385ffb640 1 -- 172.21.15.89:0/652968462 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33980035f0 con 0x5651e0b4b370 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.016+0000 7f3385ffb640 1 -- 172.21.15.89:0/652968462 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3394064290 con 0x5651e0b4b370 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.016+0000 7f33a6a42740 10 client.17124 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.016+0000 7f33a6a42740 10 client.17124 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.016+0000 7f33a6a42740 20 client.17124 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.016+0000 7f33a6a42740 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651e0b5af70 0x5651e0b5d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.016+0000 7f33a6a42740 4 client.17124 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.016+0000 7f33a6a42740 1 -- 172.21.15.89:0/652968462 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651e0ab3860 con 0x5651e0b5af70 2024-09-15T23:29:02.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.016+0000 7f33a2405640 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651e0b5af70 0x5651e0b5d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f3385ffb640 10 client.17124 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f33a2405640 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651e0b5af70 0x5651e0b5d3c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=66d7419b60ccf989 server_cookie=f0a061e320773d93 in_seq=0 out_seq=0 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f3385ffb640 1 -- 172.21.15.89:0/652968462 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5651e0ab3860 con 0x5651e0b5af70 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f3385ffb640 10 client.17124 handle_command_reply: tid=0 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f33a6a42740 1 client.17124 shutdown 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f33a6a42740 1 -- 172.21.15.89:0/652968462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b4b370 msgr2=0x5651e0b48610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f33a6a42740 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b4b370 0x5651e0b48610 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f339409bfa0 tx=0x7f339409c7d0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f33a6a42740 1 -- 172.21.15.89:0/652968462 shutdown_connections 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f33a6a42740 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651e0b5af70 0x5651e0b5d3c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f33a6a42740 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b4b370 0x5651e0b48610 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f33a6a42740 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 0x5651e0b48020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f3385ffb640 0 client.17124 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:29:02.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f33a6a42740 1 --2- 172.21.15.89:0/652968462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b405d0 0x5651e0b4ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.017+0000 7f33a6a42740 1 -- 172.21.15.89:0/652968462 >> 172.21.15.89:0/652968462 conn(0x5651e0b09310 msgr2=0x5651e0b42960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.018+0000 7f33a6a42740 1 -- 172.21.15.89:0/652968462 shutdown_connections 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.018+0000 7f33a6a42740 1 -- 172.21.15.89:0/652968462 wait complete. 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.018+0000 7f33a6a42740 20 client.17124 trim_cache size 0 max 16384 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.030+0000 7f33a6a42740 1 Processor -- start 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.030+0000 7f33a6a42740 1 -- start start 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.030+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b572b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.030+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b41b40 0x5651e0b468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.030+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b4b370 0x5651e0b46e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.030+0000 7f33a6a42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f339c1aedf0 con 0x5651e0b405d0 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.030+0000 7f33a6a42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f339c1b0930 con 0x5651e0b4b370 2024-09-15T23:29:02.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.030+0000 7f33a6a42740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f339c1af4c0 con 0x5651e0b41b40 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f33a2405640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f33a2405640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b572b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42654/0 (socket says 172.21.15.89:42654) 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f33a2405640 1 -- 172.21.15.89:0/3964017192 learned_addr learned my addr 172.21.15.89:0/3964017192 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f33a2405640 1 -- 172.21.15.89:0/3964017192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b41b40 msgr2=0x5651e0b468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f33a2405640 1 --2- 172.21.15.89:0/3964017192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b41b40 0x5651e0b468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f33a2405640 1 -- 172.21.15.89:0/3964017192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b4b370 msgr2=0x5651e0b46e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f33a2405640 1 --2- 172.21.15.89:0/3964017192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b4b370 0x5651e0b46e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f33a2405640 1 -- 172.21.15.89:0/3964017192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651e0b4df50 con 0x5651e0b405d0 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f33a2405640 1 --2- 172.21.15.89:0/3964017192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b572b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f339802a9c0 tx=0x7f339802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb95c8f65f260ef4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f3387fff640 1 -- 172.21.15.89:0/3964017192 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339803f040 con 0x5651e0b405d0 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f3387fff640 1 -- 172.21.15.89:0/3964017192 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33980043f0 con 0x5651e0b405d0 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.031+0000 7f3387fff640 1 -- 172.21.15.89:0/3964017192 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398035040 con 0x5651e0b405d0 2024-09-15T23:29:02.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 -- 172.21.15.89:0/3964017192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 msgr2=0x5651e0b572b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 --2- 172.21.15.89:0/3964017192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b572b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f339802a9c0 tx=0x7f339802a420 comp rx=0 tx=0).stop 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 -- 172.21.15.89:0/3964017192 shutdown_connections 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 --2- 172.21.15.89:0/3964017192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b4b370 0x5651e0b46e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 --2- 172.21.15.89:0/3964017192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b41b40 0x5651e0b468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 --2- 172.21.15.89:0/3964017192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b572b0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 -- 172.21.15.89:0/3964017192 >> 172.21.15.89:0/3964017192 conn(0x5651e0b09310 msgr2=0x5651e0b47fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 -- 172.21.15.89:0/3964017192 shutdown_connections 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 -- 172.21.15.89:0/3964017192 wait complete. 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 -- start start 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 Processor -- start 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b53900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 0x5651e0b4ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b4b370 0x5651e0b4f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f339c1b0260 con 0x5651e0b405d0 2024-09-15T23:29:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f339c1afb90 con 0x5651e0b41b40 2024-09-15T23:29:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f339c1acdd0 con 0x5651e0b4b370 2024-09-15T23:29:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a6a42740 10 client.0 _ll_register_callbacks cb 0x7f3394083f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.032+0000 7f33a2405640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b53900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a2405640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b53900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42664/0 (socket says 172.21.15.89:42664) 2024-09-15T23:29:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a2405640 1 -- 172.21.15.89:0/1031966876 learned_addr learned my addr 172.21.15.89:0/1031966876 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a2c06640 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b4b370 0x5651e0b4f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a1c04640 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 0x5651e0b4ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a2405640 1 -- 172.21.15.89:0/1031966876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b4b370 msgr2=0x5651e0b4f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:02.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a2405640 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b4b370 0x5651e0b4f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a2405640 1 -- 172.21.15.89:0/1031966876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 msgr2=0x5651e0b4ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a2405640 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 0x5651e0b4ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a2405640 1 -- 172.21.15.89:0/1031966876 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651e0b4f730 con 0x5651e0b405d0 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a2c06640 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b4b370 0x5651e0b4f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a1c04640 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 0x5651e0b4ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a2405640 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b53900 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f3398031720 tx=0x7f3398004ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1729d9e09c9735ce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f3385ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f3385ffb640 1 -- 172.21.15.89:0/1031966876 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339805b060 con 0x5651e0b405d0 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f3385ffb640 1 -- 172.21.15.89:0/1031966876 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f33980051a0 con 0x5651e0b405d0 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a6a42740 1 -- 172.21.15.89:0/1031966876 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33980043f0 con 0x5651e0b405d0 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f3385ffb640 1 -- 172.21.15.89:0/1031966876 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3398035030 con 0x5651e0b405d0 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a6a42740 10 client.17130 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:02.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f33a6a42740 1 -- 172.21.15.89:0/1031966876 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5651e0bc59f0 con 0x5651e0b405d0 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.033+0000 7f3385ffb640 1 -- 172.21.15.89:0/1031966876 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33980043f0 con 0x5651e0b405d0 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.034+0000 7f3385ffb640 1 -- 172.21.15.89:0/1031966876 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f339803b430 con 0x5651e0b405d0 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.034+0000 7f33a6a42740 10 client.17130 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.034+0000 7f33a6a42740 10 client.17130 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.034+0000 7f33a6a42740 20 client.17130 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.034+0000 7f33a6a42740 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651e0b5af70 0x5651e0bc7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.034+0000 7f33a6a42740 4 client.17130 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.034+0000 7f33a6a42740 1 -- 172.21.15.89:0/1031966876 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5651e0aadd70 con 0x5651e0b5af70 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.034+0000 7f33a1c04640 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651e0b5af70 0x5651e0bc7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.034+0000 7f3385ffb640 10 client.17130 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.034+0000 7f33a1c04640 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651e0b5af70 0x5651e0bc7c80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c32f7bb2b80971cc server_cookie=9cc047bab44bd963 in_seq=0 out_seq=0 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f3385ffb640 1 -- 172.21.15.89:0/1031966876 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5651e0aadd70 con 0x5651e0b5af70 2024-09-15T23:29:02.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f3385ffb640 10 client.17130 handle_command_reply: tid=0 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 client.17130 shutdown 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 -- 172.21.15.89:0/1031966876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 msgr2=0x5651e0b53900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b53900 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f3398031720 tx=0x7f3398004ba0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 -- 172.21.15.89:0/1031966876 shutdown_connections 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5651e0b5af70 0x5651e0bc7c80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5651e0b4b370 0x5651e0b4f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5651e0b41b40 0x5651e0b4ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f3385ffb640 0 client.17130 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 --2- 172.21.15.89:0/1031966876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5651e0b405d0 0x5651e0b53900 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 -- 172.21.15.89:0/1031966876 >> 172.21.15.89:0/1031966876 conn(0x5651e0b09310 msgr2=0x5651e0b46880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 -- 172.21.15.89:0/1031966876 shutdown_connections 2024-09-15T23:29:02.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 1 -- 172.21.15.89:0/1031966876 wait complete. 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a6a42740 20 client.17130 trim_cache size 0 max 16384 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 -- 172.21.15.89:0/771590496 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33880a3240 msgr2=0x7f33880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33880a3240 0x7f33880a5700 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f338c005fd0 tx=0x7f338c014840 comp rx=0 tx=0).stop 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 -- 172.21.15.89:0/771590496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f339c106a80 msgr2=0x7f339c19cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f339c106a80 0x7f339c19cd60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f339400cad0 tx=0x7f339400cf90 comp rx=0 tx=0).stop 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 -- 172.21.15.89:0/771590496 shutdown_connections 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f33880a3240 0x7f33880a5700 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f339c106a80 0x7f339c19cd60 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f339c101420 0x7f339c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 --2- 172.21.15.89:0/771590496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f339c100a80 0x7f339c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 -- 172.21.15.89:0/771590496 >> 172.21.15.89:0/771590496 conn(0x7f339c0fc7a0 msgr2=0x7f339c1048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 -- 172.21.15.89:0/771590496 shutdown_connections 2024-09-15T23:29:02.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:02.035+0000 7f33a4669640 1 -- 172.21.15.89:0/771590496 wait complete. 2024-09-15T23:29:02.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:02 smithi089 ceph-mon[30933]: pgmap v500: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:29:02.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:02 smithi120 ceph-mon[35250]: pgmap v500: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:29:02.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:02 smithi174 ceph-mon[35586]: pgmap v500: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:29:02.915 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:29:03.163 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:03.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:03 smithi089 ceph-mon[30933]: pgmap v501: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.8 KiB/s wr, 15 op/s 2024-09-15T23:29:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:03 smithi120 ceph-mon[35250]: pgmap v501: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.8 KiB/s wr, 15 op/s 2024-09-15T23:29:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:03 smithi174 ceph-mon[35586]: pgmap v501: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.8 KiB/s wr, 15 op/s 2024-09-15T23:29:03.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.639+0000 7fe9c6892640 1 -- 172.21.15.89:0/927978863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9c0106f10 msgr2=0x7fe9c010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.639+0000 7fe9c6892640 1 --2- 172.21.15.89:0/927978863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9c0106f10 0x7fe9c010ea90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009ca0 tx=0x7fe9b802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.639+0000 7fe9c6892640 1 -- 172.21.15.89:0/927978863 shutdown_connections 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.639+0000 7fe9c6892640 1 --2- 172.21.15.89:0/927978863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9c0106f10 0x7fe9c010ea90 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.639+0000 7fe9c6892640 1 --2- 172.21.15.89:0/927978863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0102790 0x7fe9c01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.639+0000 7fe9c6892640 1 --2- 172.21.15.89:0/927978863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c0101de0 0x7fe9c01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.639+0000 7fe9c6892640 1 -- 172.21.15.89:0/927978863 >> 172.21.15.89:0/927978863 conn(0x7fe9c00fd8c0 msgr2=0x7fe9c00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.639+0000 7fe9c6892640 1 -- 172.21.15.89:0/927978863 shutdown_connections 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.639+0000 7fe9c6892640 1 -- 172.21.15.89:0/927978863 wait complete. 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9c6892640 1 Processor -- start 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9c6892640 1 -- start start 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9c6892640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c0101de0 0x7fe9c01985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9c6892640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9c0102790 0x7fe9c0198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9c6892640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0106f10 0x7fe9c019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9c6892640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c0112010 con 0x7fe9c0102790 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9c6892640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c0111e90 con 0x7fe9c0106f10 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9c6892640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c0112190 con 0x7fe9c0101de0 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9b7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9c0102790 0x7fe9c0198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9b7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9c0102790 0x7fe9c0198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42692/0 (socket says 172.21.15.89:42692) 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9b7fff640 1 -- 172.21.15.89:0/501788907 learned_addr learned my addr 172.21.15.89:0/501788907 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.640+0000 7fe9bffff640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c0101de0 0x7fe9c01985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0106f10 0x7fe9c019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9b7fff640 1 -- 172.21.15.89:0/501788907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c0101de0 msgr2=0x7fe9c01985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c0101de0 0x7fe9c01985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9b7fff640 1 -- 172.21.15.89:0/501788907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0106f10 msgr2=0x7fe9c019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0106f10 0x7fe9c019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9b7fff640 1 -- 172.21.15.89:0/501788907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c019d5a0 con 0x7fe9c0102790 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9c0102790 0x7fe9c0198b30 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800b4a0 tx=0x7fe9a800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9bdffb640 1 -- 172.21.15.89:0/501788907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8015020 con 0x7fe9c0102790 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9bdffb640 1 -- 172.21.15.89:0/501788907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a8011d10 con 0x7fe9c0102790 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9c6892640 1 -- 172.21.15.89:0/501788907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c019d890 con 0x7fe9c0102790 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9bdffb640 1 -- 172.21.15.89:0/501788907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a800f2e0 con 0x7fe9c0102790 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.641+0000 7fe9c6892640 1 -- 172.21.15.89:0/501788907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c01a5170 con 0x7fe9c0102790 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9bdffb640 1 -- 172.21.15.89:0/501788907 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe9a8018020 con 0x7fe9c0102790 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c8c6b740 1 Processor -- start 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c8c6b740 1 -- start start 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd68b40 0x55cc7dd67090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 0x55cc7dd67a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd72370 0x55cc7dd74760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cc7dccd2f0 con 0x55cc7dd68b40 2024-09-15T23:29:03.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cc7dc74cf0 con 0x55cc7dd675d0 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cc7dcc92b0 con 0x55cc7dd72370 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c4e2f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd68b40 0x55cc7dd67090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c4e2f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd68b40 0x55cc7dd67090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42696/0 (socket says 172.21.15.89:42696) 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.644+0000 7fe9c4e2f640 1 -- 172.21.15.89:0/1345579203 learned_addr learned my addr 172.21.15.89:0/1345579203 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9bffff640 1 --2- 172.21.15.89:0/1345579203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 0x55cc7dd67a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9bdffb640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9ac0a3240 0x7fe9ac0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/1345579203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd72370 0x55cc7dd74760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9c4e2f640 1 -- 172.21.15.89:0/1345579203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd72370 msgr2=0x55cc7dd74760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/1345579203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd72370 0x55cc7dd74760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9c4e2f640 1 -- 172.21.15.89:0/1345579203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 msgr2=0x55cc7dd67a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/1345579203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 0x55cc7dd67a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9c4e2f640 1 -- 172.21.15.89:0/1345579203 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc7dc593a0 con 0x55cc7dd68b40 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9ac0a3240 0x7fe9ac0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9bdffb640 1 -- 172.21.15.89:0/501788907 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe9a80cf5d0 con 0x7fe9c0102790 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/1345579203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd68b40 0x55cc7dd67090 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8005cc0 tx=0x7fe9b8031bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc25177ffdcb64f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9b67fc640 1 -- 172.21.15.89:0/1345579203 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8041040 con 0x55cc7dd68b40 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9b67fc640 1 -- 172.21.15.89:0/1345579203 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9b8036070 con 0x55cc7dd68b40 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9ac0a3240 0x7fe9ac0a5700 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe9b804c000 tx=0x7fe9b804d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.645+0000 7fe9b67fc640 1 -- 172.21.15.89:0/1345579203 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b802fad0 con 0x55cc7dd68b40 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.646+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1345579203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd68b40 msgr2=0x55cc7dd67090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.646+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1345579203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd68b40 0x55cc7dd67090 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8005cc0 tx=0x7fe9b8031bb0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.646+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1345579203 shutdown_connections 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.646+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1345579203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd72370 0x55cc7dd74760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.646+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1345579203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 0x55cc7dd67a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.646+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1345579203 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd68b40 0x55cc7dd67090 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.646+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1345579203 >> 172.21.15.89:0/1345579203 conn(0x55cc7dd30310 msgr2=0x55cc7dd63c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.646+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1345579203 shutdown_connections 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.646+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1345579203 wait complete. 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.646+0000 7fe9c8c6b740 1 -- start start 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9c8c6b740 1 Processor -- start 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd71d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd68b40 0x55cc7dd6f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd72370 0x55cc7dd6f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cc7dd75070 con 0x55cc7dd675d0 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cc7dc88060 con 0x55cc7dd68b40 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cc7dd751f0 con 0x55cc7dd72370 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9c8c6b740 10 client.0 _ll_register_callbacks cb 0x55cc7d9db8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd71d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd71d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42710/0 (socket says 172.21.15.89:42710) 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9bffff640 1 -- 172.21.15.89:0/1665597682 learned_addr learned my addr 172.21.15.89:0/1665597682 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd72370 0x55cc7dd6f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd68b40 0x55cc7dd6f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9bffff640 1 -- 172.21.15.89:0/1665597682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd72370 msgr2=0x55cc7dd6f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9bffff640 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd72370 0x55cc7dd6f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9bffff640 1 -- 172.21.15.89:0/1665597682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd68b40 msgr2=0x55cc7dd6f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9bffff640 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd68b40 0x55cc7dd6f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9bffff640 1 -- 172.21.15.89:0/1665597682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc7da3e3c0 con 0x55cc7dd675d0 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd68b40 0x55cc7dd6f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe9bffff640 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd71d60 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0011540 tx=0x7fe9b0011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170e038055d81df2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.647+0000 7fe99f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe99f7fe640 1 -- 172.21.15.89:0/1665597682 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b000f6d0 con 0x55cc7dd675d0 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe99f7fe640 1 -- 172.21.15.89:0/1665597682 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe9b000f870 con 0x55cc7dd675d0 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1665597682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9b8036070 con 0x55cc7dd675d0 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe99f7fe640 1 -- 172.21.15.89:0/1665597682 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe9b0026050 con 0x55cc7dd675d0 2024-09-15T23:29:03.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe9c8c6b740 10 client.17142 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1665597682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cc7dd80b70 con 0x55cc7dd675d0 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe99f7fe640 1 -- 172.21.15.89:0/1665597682 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9b8036070 con 0x55cc7dd675d0 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe99f7fe640 1 -- 172.21.15.89:0/1665597682 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe9b0014cd0 con 0x55cc7dd675d0 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe9c8c6b740 10 client.17142 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe9c8c6b740 10 client.17142 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe9c8c6b740 20 client.17142 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cc7dd82030 0x55cc7dd84420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe9c8c6b740 4 client.17142 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1665597682 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc7dcda860 con 0x55cc7dd82030 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.648+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cc7dd82030 0x55cc7dd84420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.649+0000 7fe99f7fe640 10 client.17142 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.649+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cc7dd82030 0x55cc7dd84420 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2c3d0c6c07018be5 server_cookie=35abacc65d0a3f3b in_seq=0 out_seq=0 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.649+0000 7fe99f7fe640 1 -- 172.21.15.89:0/1665597682 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cc7dcda860 con 0x55cc7dd82030 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.649+0000 7fe99f7fe640 10 client.17142 handle_command_reply: tid=0 2024-09-15T23:29:03.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 client.17142 shutdown 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1665597682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 msgr2=0x55cc7dd71d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd71d60 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0011540 tx=0x7fe9b0011a00 comp rx=0 tx=0).stop 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1665597682 shutdown_connections 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cc7dd82030 0x55cc7dd84420 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd72370 0x55cc7dd6f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd68b40 0x55cc7dd6f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe99f7fe640 0 client.17142 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1665597682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd71d60 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1665597682 >> 172.21.15.89:0/1665597682 conn(0x55cc7dd30310 msgr2=0x55cc7dd69930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1665597682 shutdown_connections 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1665597682 wait complete. 2024-09-15T23:29:03.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.650+0000 7fe9c8c6b740 20 client.17142 trim_cache size 0 max 16384 2024-09-15T23:29:03.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c8c6b740 1 Processor -- start 2024-09-15T23:29:03.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c8c6b740 1 -- start start 2024-09-15T23:29:03.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 0x55cc7dd6d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 0x55cc7dd6a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd72370 0x55cc7dd6a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01aef00 con 0x55cc7dd72370 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01b0a40 con 0x55cc7dd675d0 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01af5d0 con 0x55cc7dd68b40 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c4e2f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd72370 0x55cc7dd6a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c4e2f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd72370 0x55cc7dd6a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42712/0 (socket says 172.21.15.89:42712) 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c4e2f640 1 -- 172.21.15.89:0/1758387797 learned_addr learned my addr 172.21.15.89:0/1758387797 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 0x55cc7dd6a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9bffff640 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 0x55cc7dd6d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c4e2f640 1 -- 172.21.15.89:0/1758387797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 msgr2=0x55cc7dd6a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 0x55cc7dd6a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c4e2f640 1 -- 172.21.15.89:0/1758387797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 msgr2=0x55cc7dd6d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 0x55cc7dd6d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9c4e2f640 1 -- 172.21.15.89:0/1758387797 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc7dd6dfc0 con 0x55cc7dd72370 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 0x55cc7dd6a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.663+0000 7fe9bffff640 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 0x55cc7dd6d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd72370 0x55cc7dd6a660 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fe9b802f720 tx=0x7fe9b802f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f187bb0e47b15b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9b5ffb640 1 -- 172.21.15.89:0/1758387797 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8041040 con 0x55cc7dd72370 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9b5ffb640 1 -- 172.21.15.89:0/1758387797 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9b8036040 con 0x55cc7dd72370 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9b5ffb640 1 -- 172.21.15.89:0/1758387797 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b802fac0 con 0x55cc7dd72370 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1758387797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd72370 msgr2=0x55cc7dd6a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd72370 0x55cc7dd6a660 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fe9b802f720 tx=0x7fe9b802f7f0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1758387797 shutdown_connections 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd72370 0x55cc7dd6a660 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 0x55cc7dd6a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/1758387797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd675d0 0x55cc7dd6d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1758387797 >> 172.21.15.89:0/1758387797 conn(0x55cc7dd30310 msgr2=0x55cc7dd71090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1758387797 shutdown_connections 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.664+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/1758387797 wait complete. 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9c8c6b740 1 -- start start 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9c8c6b740 1 Processor -- start 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd76860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 0x55cc7dd76da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9c8c6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd72370 0x55cc7ddf00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01b0370 con 0x55cc7dd675d0 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01afca0 con 0x55cc7dd72370 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9c8c6b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c01acee0 con 0x55cc7dd68b40 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9c8c6b740 10 client.0 _ll_register_callbacks cb 0x7fe9b0009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd76860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd76860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42728/0 (socket says 172.21.15.89:42728) 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9bffff640 1 -- 172.21.15.89:0/2877513115 learned_addr learned my addr 172.21.15.89:0/2877513115 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:03.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.665+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 0x55cc7dd76da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd72370 0x55cc7ddf00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9bffff640 1 -- 172.21.15.89:0/2877513115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 msgr2=0x55cc7dd76da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9bffff640 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 0x55cc7dd76da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9bffff640 1 -- 172.21.15.89:0/2877513115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd72370 msgr2=0x55cc7ddf00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9bffff640 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd72370 0x55cc7ddf00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9bffff640 1 -- 172.21.15.89:0/2877513115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc7dd6f970 con 0x55cc7dd675d0 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9c4e2f640 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd72370 0x55cc7ddf00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9bffff640 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd76860 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000f270 tx=0x7fe9b000f2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6612ae34908ce99f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe99f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe99f7fe640 1 -- 172.21.15.89:0/2877513115 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b001d930 con 0x55cc7dd675d0 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe99f7fe640 1 -- 172.21.15.89:0/2877513115 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe9b001dad0 con 0x55cc7dd675d0 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/2877513115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9b8036040 con 0x55cc7dd675d0 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe99f7fe640 1 -- 172.21.15.89:0/2877513115 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe9b001dcd0 con 0x55cc7dd675d0 2024-09-15T23:29:03.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9c8c6b740 10 client.17148 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/2877513115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cc7ddf1680 con 0x55cc7dd675d0 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.666+0000 7fe99f7fe640 1 -- 172.21.15.89:0/2877513115 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9b8036040 con 0x55cc7dd675d0 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe99f7fe640 1 -- 172.21.15.89:0/2877513115 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe9b0014740 con 0x55cc7dd675d0 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe9c8c6b740 10 client.17148 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe9c8c6b740 10 client.17148 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe9c8c6b740 20 client.17148 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cc7ddf1e20 0x55cc7ddf4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe9c8c6b740 4 client.17148 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/2877513115 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cc7dcd4d70 con 0x55cc7ddf1e20 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cc7ddf1e20 0x55cc7ddf4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe99f7fe640 10 client.17148 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe9b7fff640 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cc7ddf1e20 0x55cc7ddf4240 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4cbbe20852fb0544 server_cookie=51c5c49956c6bb59 in_seq=0 out_seq=0 2024-09-15T23:29:03.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe99f7fe640 1 -- 172.21.15.89:0/2877513115 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cc7dcd4d70 con 0x55cc7ddf1e20 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe99f7fe640 10 client.17148 handle_command_reply: tid=0 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.667+0000 7fe9c8c6b740 1 client.17148 shutdown 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/2877513115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 msgr2=0x55cc7dd76860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd76860 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000f270 tx=0x7fe9b000f2a0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/2877513115 shutdown_connections 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cc7ddf1e20 0x55cc7ddf4240 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cc7dd72370 0x55cc7ddf00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cc7dd68b40 0x55cc7dd76da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 1 --2- 172.21.15.89:0/2877513115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7dd675d0 0x55cc7dd76860 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/2877513115 >> 172.21.15.89:0/2877513115 conn(0x55cc7dd30310 msgr2=0x55cc7dd6d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe99f7fe640 0 client.17148 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/2877513115 shutdown_connections 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 1 -- 172.21.15.89:0/2877513115 wait complete. 2024-09-15T23:29:03.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c8c6b740 20 client.17148 trim_cache size 0 max 16384 2024-09-15T23:29:03.678 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:29:03.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c6892640 1 -- 172.21.15.89:0/501788907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9ac0a3240 msgr2=0x7fe9ac0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c6892640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9ac0a3240 0x7fe9ac0a5700 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe9b804c000 tx=0x7fe9b804d000 comp rx=0 tx=0).stop 2024-09-15T23:29:03.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c6892640 1 -- 172.21.15.89:0/501788907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9c0102790 msgr2=0x7fe9c0198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c6892640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9c0102790 0x7fe9c0198b30 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800b4a0 tx=0x7fe9a800b8a0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.668+0000 7fe9c6892640 1 -- 172.21.15.89:0/501788907 shutdown_connections 2024-09-15T23:29:03.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.669+0000 7fe9c6892640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9ac0a3240 0x7fe9ac0a5700 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.669+0000 7fe9c6892640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9c0106f10 0x7fe9c019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.669+0000 7fe9c6892640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe9c0102790 0x7fe9c0198b30 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.669+0000 7fe9c6892640 1 --2- 172.21.15.89:0/501788907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c0101de0 0x7fe9c01985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.669+0000 7fe9c6892640 1 -- 172.21.15.89:0/501788907 >> 172.21.15.89:0/501788907 conn(0x7fe9c00fd8c0 msgr2=0x7fe9c00fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.669+0000 7fe9c6892640 1 -- 172.21.15.89:0/501788907 shutdown_connections 2024-09-15T23:29:03.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:03.669+0000 7fe9c6892640 1 -- 172.21.15.89:0/501788907 wait complete. 2024-09-15T23:29:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.923+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2594796434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f8102600 msgr2=0x7f62f8108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.923+0000 7f62fdf6d640 1 --2- 172.21.15.89:0/2594796434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f8102600 0x7f62f8108e90 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f62f0009ca0 tx=0x7f62f002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.923+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2594796434 shutdown_connections 2024-09-15T23:29:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.923+0000 7f62fdf6d640 1 --2- 172.21.15.89:0/2594796434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f8102600 0x7f62f8108e90 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.923+0000 7f62fdf6d640 1 --2- 172.21.15.89:0/2594796434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f8101c40 0x7f62f81020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.923+0000 7f62fdf6d640 1 --2- 172.21.15.89:0/2594796434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62f8100a50 0x7f62f8100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.923+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2594796434 >> 172.21.15.89:0/2594796434 conn(0x7f62f80fc1b0 msgr2=0x7f62f80fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.923+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2594796434 shutdown_connections 2024-09-15T23:29:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.923+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2594796434 wait complete. 2024-09-15T23:29:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62fdf6d640 1 Processor -- start 2024-09-15T23:29:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62fdf6d640 1 -- start start 2024-09-15T23:29:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62fdf6d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f8100a50 0x7f62f8196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62fdf6d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f8101c40 0x7f62f8197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62fdf6d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62f8102600 0x7f62f819e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62fdf6d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62f810c730 con 0x7f62f8101c40 2024-09-15T23:29:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62fdf6d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62f810c5b0 con 0x7f62f8102600 2024-09-15T23:29:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62fdf6d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62f810c8b0 con 0x7f62f8100a50 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f8101c40 0x7f62f8197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f8101c40 0x7f62f8197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42750/0 (socket says 172.21.15.89:42750) 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62f6ffd640 1 -- 172.21.15.89:0/2644485947 learned_addr learned my addr 172.21.15.89:0/2644485947 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62f7fff640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62f8102600 0x7f62f819e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.924+0000 7f62f77fe640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f8100a50 0x7f62f8196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62f6ffd640 1 -- 172.21.15.89:0/2644485947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f8100a50 msgr2=0x7f62f8196cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f8100a50 0x7f62f8196cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62f6ffd640 1 -- 172.21.15.89:0/2644485947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62f8102600 msgr2=0x7f62f819e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62f8102600 0x7f62f819e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62f6ffd640 1 -- 172.21.15.89:0/2644485947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f819e900 con 0x7f62f8101c40 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f8101c40 0x7f62f8197200 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00d990 tx=0x7f62ec00de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62f4ff9640 1 -- 172.21.15.89:0/2644485947 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec01b070 con 0x7f62f8101c40 2024-09-15T23:29:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62f4ff9640 1 -- 172.21.15.89:0/2644485947 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62ec00edf0 con 0x7f62f8101c40 2024-09-15T23:29:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2644485947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f806a380 con 0x7f62f8101c40 2024-09-15T23:29:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62f4ff9640 1 -- 172.21.15.89:0/2644485947 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec014870 con 0x7f62f8101c40 2024-09-15T23:29:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.925+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2644485947 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f806a9a0 con 0x7f62f8101c40 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.927+0000 7f6300de6740 1 Processor -- start 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.927+0000 7f6300de6740 1 -- start start 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.927+0000 7f62f4ff9640 1 -- 172.21.15.89:0/2644485947 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f62ec01e070 con 0x7f62f8101c40 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f6300de6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 0x564ed03321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f6300de6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed03326f0 0x564ed03371b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f6300de6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0337780 0x564ed0339c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f6300de6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564ed02912f0 con 0x564ed0337780 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f6300de6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564ed0239270 con 0x564ed03326f0 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f6300de6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564ed028d2b0 con 0x564ed0331db0 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0337780 0x564ed0339c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 0x564ed03321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 0x564ed03321b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:33936/0 (socket says 172.21.15.89:33936) 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0337780 0x564ed0339c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42762/0 (socket says 172.21.15.89:42762) 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f7fff640 1 -- 172.21.15.89:0/1879690864 learned_addr learned my addr 172.21.15.89:0/1879690864 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f77fe640 1 --2- 172.21.15.89:0/1879690864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed03326f0 0x564ed03371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f7fff640 1 -- 172.21.15.89:0/1879690864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed03326f0 msgr2=0x564ed03371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f7fff640 1 --2- 172.21.15.89:0/1879690864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed03326f0 0x564ed03371b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f7fff640 1 -- 172.21.15.89:0/1879690864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0337780 msgr2=0x564ed0339c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f7fff640 1 --2- 172.21.15.89:0/1879690864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0337780 0x564ed0339c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f7fff640 1 -- 172.21.15.89:0/1879690864 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed0238340 con 0x564ed0331db0 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/1879690864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0337780 0x564ed0339c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f4ff9640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62e00a3370 0x7f62e00a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f77fe640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62e00a3370 0x7f62e00a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.928+0000 7f62f7fff640 1 --2- 172.21.15.89:0/1879690864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 0x564ed03321b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62f0035000 tx=0x7f62f0009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62899a69e5cc4c50 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.929+0000 7f62f4ff9640 1 -- 172.21.15.89:0/2644485947 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f62ec0070a0 con 0x7f62f8101c40 2024-09-15T23:29:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.929+0000 7f62ddffb640 1 -- 172.21.15.89:0/1879690864 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f0031450 con 0x564ed0331db0 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.929+0000 7f62ddffb640 1 -- 172.21.15.89:0/1879690864 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62f0036050 con 0x564ed0331db0 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.929+0000 7f62ddffb640 1 -- 172.21.15.89:0/1879690864 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f0004500 con 0x564ed0331db0 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.929+0000 7f62f77fe640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62e00a3370 0x7f62e00a5830 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x564ed0333120 tx=0x7f62e800b4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 -- 172.21.15.89:0/1879690864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 msgr2=0x564ed03321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 --2- 172.21.15.89:0/1879690864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 0x564ed03321b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62f0035000 tx=0x7f62f0009800 comp rx=0 tx=0).stop 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 -- 172.21.15.89:0/1879690864 shutdown_connections 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 --2- 172.21.15.89:0/1879690864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0337780 0x564ed0339c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 --2- 172.21.15.89:0/1879690864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed03326f0 0x564ed03371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 --2- 172.21.15.89:0/1879690864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 0x564ed03321b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 -- 172.21.15.89:0/1879690864 >> 172.21.15.89:0/1879690864 conn(0x564ed0329010 msgr2=0x564ed0329410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 -- 172.21.15.89:0/1879690864 shutdown_connections 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 -- 172.21.15.89:0/1879690864 wait complete. 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 -- start start 2024-09-15T23:29:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 Processor -- start 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed033be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed03326f0 0x564ed033c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0337780 0x564ed03433d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564ed033a570 con 0x564ed0331db0 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564ed021a290 con 0x564ed03326f0 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564ed033a6f0 con 0x564ed0337780 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.930+0000 7f6300de6740 10 client.0 _ll_register_callbacks cb 0x564ed024b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed033be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed033be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42772/0 (socket says 172.21.15.89:42772) 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f77fe640 1 -- 172.21.15.89:0/3447935460 learned_addr learned my addr 172.21.15.89:0/3447935460 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f7fff640 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0337780 0x564ed03433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed03326f0 0x564ed033c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f77fe640 1 -- 172.21.15.89:0/3447935460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0337780 msgr2=0x564ed03433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f77fe640 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0337780 0x564ed03433d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f77fe640 1 -- 172.21.15.89:0/3447935460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed03326f0 msgr2=0x564ed033c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f77fe640 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed03326f0 0x564ed033c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f77fe640 1 -- 172.21.15.89:0/3447935460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ecffa4210 con 0x564ed0331db0 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62f77fe640 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed033be50 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f62e8010680 tx=0x7f62e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217e932df3e7794e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62d6ffd640 1 -- 172.21.15.89:0/3447935460 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e801f8c0 con 0x564ed0331db0 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62d6ffd640 1 -- 172.21.15.89:0/3447935460 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f62e801fa60 con 0x564ed0331db0 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f6300de6740 1 -- 172.21.15.89:0/3447935460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62f0036050 con 0x564ed0331db0 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f62d6ffd640 1 -- 172.21.15.89:0/3447935460 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f62e801fc60 con 0x564ed0331db0 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f6300de6740 10 client.17160 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.931+0000 7f6300de6740 1 -- 172.21.15.89:0/3447935460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ed0070620 con 0x564ed0331db0 2024-09-15T23:29:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f62d6ffd640 1 -- 172.21.15.89:0/3447935460 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62f0036050 con 0x564ed0331db0 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f62d6ffd640 1 -- 172.21.15.89:0/3447935460 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f62e801e430 con 0x564ed0331db0 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f6300de6740 10 client.17160 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f6300de6740 10 client.17160 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f6300de6740 20 client.17160 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f6300de6740 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ed0347bc0 0x564ed0349fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f6300de6740 4 client.17160 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f6300de6740 1 -- 172.21.15.89:0/3447935460 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ed029ea20 con 0x564ed0347bc0 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ed0347bc0 0x564ed0349fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f62d6ffd640 10 client.17160 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.932+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ed0347bc0 0x564ed0349fd0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f08d9a75ff1ff2e server_cookie=b1047bfbf6c08d9c in_seq=0 out_seq=0 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f62d6ffd640 1 -- 172.21.15.89:0/3447935460 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564ed029ea20 con 0x564ed0347bc0 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f62d6ffd640 10 client.17160 handle_command_reply: tid=0 2024-09-15T23:29:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f6300de6740 1 client.17160 shutdown 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f6300de6740 1 -- 172.21.15.89:0/3447935460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 msgr2=0x564ed033be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f6300de6740 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed033be50 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f62e8010680 tx=0x7f62e80046c0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f6300de6740 1 -- 172.21.15.89:0/3447935460 shutdown_connections 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f6300de6740 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ed0347bc0 0x564ed0349fd0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f6300de6740 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0337780 0x564ed03433d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f6300de6740 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed03326f0 0x564ed033c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f62d6ffd640 0 client.17160 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.933+0000 7f6300de6740 1 --2- 172.21.15.89:0/3447935460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed033be50 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.934+0000 7f6300de6740 1 -- 172.21.15.89:0/3447935460 >> 172.21.15.89:0/3447935460 conn(0x564ed0329010 msgr2=0x564ed032a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.934+0000 7f6300de6740 1 -- 172.21.15.89:0/3447935460 shutdown_connections 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.934+0000 7f6300de6740 1 -- 172.21.15.89:0/3447935460 wait complete. 2024-09-15T23:29:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.934+0000 7f6300de6740 20 client.17160 trim_cache size 0 max 16384 2024-09-15T23:29:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.948+0000 7f6300de6740 1 Processor -- start 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.948+0000 7f6300de6740 1 -- start start 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f6300de6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 0x564ed033c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f6300de6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed03326f0 0x564ed0343390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f6300de6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed0337780 0x564ed034e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f6300de6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62f81a86d0 con 0x564ed03326f0 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f6300de6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62f81aa210 con 0x564ed0337780 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f6300de6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62f81a8da0 con 0x564ed0331db0 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed03326f0 0x564ed0343390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed03326f0 0x564ed0343390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42782/0 (socket says 172.21.15.89:42782) 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f6ffd640 1 -- 172.21.15.89:0/249403431 learned_addr learned my addr 172.21.15.89:0/249403431 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f77fe640 1 --2- 172.21.15.89:0/249403431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 0x564ed033c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f6ffd640 1 -- 172.21.15.89:0/249403431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 msgr2=0x564ed033c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f7fff640 1 --2- 172.21.15.89:0/249403431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed0337780 0x564ed034e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/249403431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 0x564ed033c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f6ffd640 1 -- 172.21.15.89:0/249403431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed0337780 msgr2=0x564ed034e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/249403431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed0337780 0x564ed034e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f6ffd640 1 -- 172.21.15.89:0/249403431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed033b990 con 0x564ed03326f0 2024-09-15T23:29:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/249403431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed03326f0 0x564ed0343390 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f62ec099a90 tx=0x7f62ec0085b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75994416ec273485 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62dd7fa640 1 -- 172.21.15.89:0/249403431 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec09d170 con 0x564ed03326f0 2024-09-15T23:29:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62dd7fa640 1 -- 172.21.15.89:0/249403431 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62ec09dd30 con 0x564ed03326f0 2024-09-15T23:29:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.949+0000 7f62dd7fa640 1 -- 172.21.15.89:0/249403431 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec09cf90 con 0x564ed03326f0 2024-09-15T23:29:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.950+0000 7f6300de6740 1 -- 172.21.15.89:0/249403431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed03326f0 msgr2=0x564ed0343390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.950+0000 7f6300de6740 1 --2- 172.21.15.89:0/249403431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed03326f0 0x564ed0343390 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f62ec099a90 tx=0x7f62ec0085b0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.950+0000 7f6300de6740 1 -- 172.21.15.89:0/249403431 shutdown_connections 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.950+0000 7f6300de6740 1 --2- 172.21.15.89:0/249403431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed0337780 0x564ed034e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.950+0000 7f6300de6740 1 --2- 172.21.15.89:0/249403431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed03326f0 0x564ed0343390 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.950+0000 7f6300de6740 1 --2- 172.21.15.89:0/249403431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed0331db0 0x564ed033c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.950+0000 7f6300de6740 1 -- 172.21.15.89:0/249403431 >> 172.21.15.89:0/249403431 conn(0x564ed0329010 msgr2=0x564ed03281c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.950+0000 7f6300de6740 1 -- 172.21.15.89:0/249403431 shutdown_connections 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.950+0000 7f6300de6740 1 -- 172.21.15.89:0/249403431 wait complete. 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.950+0000 7f6300de6740 1 -- start start 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f6300de6740 1 Processor -- start 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f6300de6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed0342f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f6300de6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed03326f0 0x564ed0343440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f6300de6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed0337780 0x564ed0341650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f6300de6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f62f81a9b40 con 0x564ed0331db0 2024-09-15T23:29:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f6300de6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62f81a9470 con 0x564ed0337780 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f6300de6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62f81a66b0 con 0x564ed03326f0 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f6300de6740 10 client.0 _ll_register_callbacks cb 0x7f62e800bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed0342f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed0342f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42786/0 (socket says 172.21.15.89:42786) 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f77fe640 1 -- 172.21.15.89:0/175441730 learned_addr learned my addr 172.21.15.89:0/175441730 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed03326f0 0x564ed0343440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f7fff640 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed0337780 0x564ed0341650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f77fe640 1 -- 172.21.15.89:0/175441730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed03326f0 msgr2=0x564ed0343440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f77fe640 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed03326f0 0x564ed0343440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f77fe640 1 -- 172.21.15.89:0/175441730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed0337780 msgr2=0x564ed0341650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f77fe640 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed0337780 0x564ed0341650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f77fe640 1 -- 172.21.15.89:0/175441730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed03242d0 con 0x564ed0331db0 2024-09-15T23:29:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed03326f0 0x564ed0343440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62f77fe640 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed0342f00 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f62e801e8a0 tx=0x7f62e801e630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a510b2cf43781b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.951+0000 7f62d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f62d6ffd640 1 -- 172.21.15.89:0/175441730 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e801ee60 con 0x564ed0331db0 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f62d6ffd640 1 -- 172.21.15.89:0/175441730 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f62e801f8d0 con 0x564ed0331db0 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f6300de6740 1 -- 172.21.15.89:0/175441730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62ec09dd30 con 0x564ed0331db0 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f62d6ffd640 1 -- 172.21.15.89:0/175441730 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f62e8046050 con 0x564ed0331db0 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f6300de6740 10 client.17166 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f6300de6740 1 -- 172.21.15.89:0/175441730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ed03b33b0 con 0x564ed0331db0 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f62d6ffd640 1 -- 172.21.15.89:0/175441730 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62ec09dd30 con 0x564ed0331db0 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f62d6ffd640 1 -- 172.21.15.89:0/175441730 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f62e8004a70 con 0x564ed0331db0 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f6300de6740 10 client.17166 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f6300de6740 10 client.17166 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f6300de6740 20 client.17166 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f6300de6740 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ed03b3d90 0x564ed03b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.959 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f6300de6740 4 client.17166 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:29:03.959 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f6300de6740 1 -- 172.21.15.89:0/175441730 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564ed0298d70 con 0x564ed03b3d90 2024-09-15T23:29:03.959 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.952+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ed03b3d90 0x564ed03b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.959 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.953+0000 7f62d6ffd640 10 client.17166 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:03.959 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.953+0000 7f62f6ffd640 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ed03b3d90 0x564ed03b61b0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fc20e9c3b931370 server_cookie=f9de7c6d13ff90f2 in_seq=0 out_seq=0 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f62d6ffd640 1 -- 172.21.15.89:0/175441730 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564ed0298d70 con 0x564ed03b3d90 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f62d6ffd640 10 client.17166 handle_command_reply: tid=0 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 client.17166 shutdown 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 -- 172.21.15.89:0/175441730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 msgr2=0x564ed0342f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed0342f00 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f62e801e8a0 tx=0x7f62e801e630 comp rx=0 tx=0).stop 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 -- 172.21.15.89:0/175441730 shutdown_connections 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ed03b3d90 0x564ed03b61b0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ed0337780 0x564ed0341650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ed03326f0 0x564ed0343440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f62d6ffd640 0 client.17166 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 --2- 172.21.15.89:0/175441730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ed0331db0 0x564ed0342f00 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 -- 172.21.15.89:0/175441730 >> 172.21.15.89:0/175441730 conn(0x564ed0329010 msgr2=0x564ed0329ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 -- 172.21.15.89:0/175441730 shutdown_connections 2024-09-15T23:29:03.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.966+0000 7f6300de6740 1 -- 172.21.15.89:0/175441730 wait complete. 2024-09-15T23:29:03.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f6300de6740 20 client.17166 trim_cache size 0 max 16384 2024-09-15T23:29:03.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:29:03.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:29:03.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:29:03.969 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:29:03.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2644485947 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62e00a3370 msgr2=0x7f62e00a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62e00a3370 0x7f62e00a5830 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x564ed0333120 tx=0x7f62e800b4d0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2644485947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f8101c40 msgr2=0x7f62f8197200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f8101c40 0x7f62f8197200 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00d990 tx=0x7f62ec00de50 comp rx=0 tx=0).stop 2024-09-15T23:29:03.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2644485947 shutdown_connections 2024-09-15T23:29:03.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f62e00a3370 0x7f62e00a5830 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62f8102600 0x7f62f819e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.970 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62f8101c40 0x7f62f8197200 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.971 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 --2- 172.21.15.89:0/2644485947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62f8100a50 0x7f62f8196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.971 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2644485947 >> 172.21.15.89:0/2644485947 conn(0x7f62f80fc1b0 msgr2=0x7f62f8104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.971 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2644485947 shutdown_connections 2024-09-15T23:29:03.971 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:03.967+0000 7f62fdf6d640 1 -- 172.21.15.89:0/2644485947 wait complete. 2024-09-15T23:29:04.491 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:29:04.740 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:05.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.200+0000 7fc4f89a3640 1 -- 172.21.15.89:0/4191221370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f4102570 msgr2=0x7fc4f41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:05.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.200+0000 7fc4f89a3640 1 --2- 172.21.15.89:0/4191221370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f4102570 0x7fc4f41029b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc4e8009900 tx=0x7fc4e802a480 comp rx=0 tx=0).stop 2024-09-15T23:29:05.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.200+0000 7fc4f89a3640 1 -- 172.21.15.89:0/4191221370 shutdown_connections 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.200+0000 7fc4f89a3640 1 --2- 172.21.15.89:0/4191221370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4f4102ef0 0x7fc4f410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.200+0000 7fc4f89a3640 1 --2- 172.21.15.89:0/4191221370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f4102570 0x7fc4f41029b0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.200+0000 7fc4f89a3640 1 --2- 172.21.15.89:0/4191221370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f4100e20 0x7fc4f4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.200+0000 7fc4f89a3640 1 -- 172.21.15.89:0/4191221370 >> 172.21.15.89:0/4191221370 conn(0x7fc4f40fc7d0 msgr2=0x7fc4f40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.200+0000 7fc4f89a3640 1 -- 172.21.15.89:0/4191221370 shutdown_connections 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.200+0000 7fc4f89a3640 1 -- 172.21.15.89:0/4191221370 wait complete. 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f89a3640 1 Processor -- start 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f89a3640 1 -- start start 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f89a3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f4100e20 0x7fc4f41973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f89a3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f4102570 0x7fc4f4197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f89a3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4f4102ef0 0x7fc4f419bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f89a3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f4110fd0 con 0x7fc4f4102570 2024-09-15T23:29:05.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f89a3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f4110e50 con 0x7fc4f4100e20 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f89a3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f4111150 con 0x7fc4f4102ef0 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f4102570 0x7fc4f4197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f4102570 0x7fc4f4197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56840/0 (socket says 172.21.15.89:56840) 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.201+0000 7fc4f1d9b640 1 -- 172.21.15.89:0/3392752333 learned_addr learned my addr 172.21.15.89:0/3392752333 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4f4102ef0 0x7fc4f419bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4f259c640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f4100e20 0x7fc4f41973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4f1d9b640 1 -- 172.21.15.89:0/3392752333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4f4102ef0 msgr2=0x7fc4f419bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4f1d9b640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4f4102ef0 0x7fc4f419bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4f1d9b640 1 -- 172.21.15.89:0/3392752333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f4100e20 msgr2=0x7fc4f41973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4f1d9b640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f4100e20 0x7fc4f41973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4f1d9b640 1 -- 172.21.15.89:0/3392752333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f419c380 con 0x7fc4f4102570 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4f1d9b640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f4102570 0x7fc4f4197910 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc4e802aa70 tx=0x7fc4e8003b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4e37fe640 1 -- 172.21.15.89:0/3392752333 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e803f070 con 0x7fc4f4102570 2024-09-15T23:29:05.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4e37fe640 1 -- 172.21.15.89:0/3392752333 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4e8031c20 con 0x7fc4f4102570 2024-09-15T23:29:05.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4f89a3640 1 -- 172.21.15.89:0/3392752333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f419c610 con 0x7fc4f4102570 2024-09-15T23:29:05.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4e37fe640 1 -- 172.21.15.89:0/3392752333 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8035030 con 0x7fc4f4102570 2024-09-15T23:29:05.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.202+0000 7fc4f89a3640 1 -- 172.21.15.89:0/3392752333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f419ca60 con 0x7fc4f4102570 2024-09-15T23:29:05.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.204+0000 7fc4e37fe640 1 -- 172.21.15.89:0/3392752333 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc4e8042070 con 0x7fc4f4102570 2024-09-15T23:29:05.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4fad7c740 1 Processor -- start 2024-09-15T23:29:05.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4fad7c740 1 -- start start 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff970090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff970a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 0x55feff97d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55feff8d62f0 con 0x55feff971b40 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55feff87dcf0 con 0x55feff9705d0 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55feff8d22b0 con 0x55feff97b370 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4f2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff970090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4f2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff970090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56848/0 (socket says 172.21.15.89:56848) 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/3498656894 learned_addr learned my addr 172.21.15.89:0/3498656894 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4f259c640 1 --2- 172.21.15.89:0/3498656894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff970a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4e37fe640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4dc0a3240 0x7fc4dc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.205+0000 7fc4f1d9b640 1 --2- 172.21.15.89:0/3498656894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 0x55feff97d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/3498656894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 msgr2=0x55feff97d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/3498656894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 0x55feff97d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/3498656894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 msgr2=0x55feff970a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/3498656894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff970a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/3498656894 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55feff8623a0 con 0x55feff971b40 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4e37fe640 1 -- 172.21.15.89:0/3392752333 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc4e80f0410 con 0x7fc4f4102570 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4f1d9b640 1 --2- 172.21.15.89:0/3498656894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 0x55feff97d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4dc0a3240 0x7fc4dc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/3498656894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff970090 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec010510 tx=0x7fc4ec009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc385adddecd4024 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4e0ff9640 1 -- 172.21.15.89:0/3498656894 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec01d400 con 0x55feff971b40 2024-09-15T23:29:05.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4e0ff9640 1 -- 172.21.15.89:0/3498656894 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4ec0134c0 con 0x55feff971b40 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4e0ff9640 1 -- 172.21.15.89:0/3498656894 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec0118a0 con 0x55feff971b40 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.206+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4dc0a3240 0x7fc4dc0a5700 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec021000 tx=0x7fc4ec01e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.207+0000 7fc4fad7c740 1 -- 172.21.15.89:0/3498656894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 msgr2=0x55feff970090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.207+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/3498656894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff970090 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec010510 tx=0x7fc4ec009f90 comp rx=0 tx=0).stop 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.207+0000 7fc4fad7c740 1 -- 172.21.15.89:0/3498656894 shutdown_connections 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.207+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/3498656894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 0x55feff97d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.207+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/3498656894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff970a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.207+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/3498656894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff970090 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.207+0000 7fc4fad7c740 1 -- 172.21.15.89:0/3498656894 >> 172.21.15.89:0/3498656894 conn(0x55feff939310 msgr2=0x55feff96cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.207+0000 7fc4fad7c740 1 -- 172.21.15.89:0/3498656894 shutdown_connections 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.207+0000 7fc4fad7c740 1 -- 172.21.15.89:0/3498656894 wait complete. 2024-09-15T23:29:05.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.207+0000 7fc4fad7c740 1 -- start start 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4fad7c740 1 Processor -- start 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff97ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff971b40 0x55feff978010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff978600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55feff97e070 con 0x55feff97b370 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55feff891060 con 0x55feff9705d0 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55feff97e1f0 con 0x55feff971b40 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4fad7c740 10 client.0 _ll_register_callbacks cb 0x55feff5e48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff978600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff978600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56858/0 (socket says 172.21.15.89:56858) 2024-09-15T23:29:05.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/2854189746 learned_addr learned my addr 172.21.15.89:0/2854189746 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f259c640 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff97ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f1d9b640 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff971b40 0x55feff978010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/2854189746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff971b40 msgr2=0x55feff978010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff971b40 0x55feff978010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/2854189746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 msgr2=0x55feff97ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff97ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/2854189746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55feff6473c0 con 0x55feff97b370 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff978600 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0114c0 tx=0x7fc4ec0118b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7592e83a2bec20 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.208+0000 7fc4d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/2854189746 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec003c40 con 0x55feff97b370 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/2854189746 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc4ec003de0 con 0x55feff97b370 2024-09-15T23:29:05.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2854189746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4ec0134c0 con 0x55feff97b370 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/2854189746 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc4ec027480 con 0x55feff97b370 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4fad7c740 10 client.17178 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2854189746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55feff989b60 con 0x55feff97b370 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/2854189746 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4ec0134c0 con 0x55feff97b370 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/2854189746 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc4ec00f040 con 0x55feff97b370 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4fad7c740 10 client.17178 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4fad7c740 10 client.17178 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4fad7c740 20 client.17178 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55feff98b020 0x55feff98d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4fad7c740 4 client.17178 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2854189746 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55feff8e3860 con 0x55feff98b020 2024-09-15T23:29:05.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.209+0000 7fc4f259c640 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55feff98b020 0x55feff98d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.210+0000 7fc4f259c640 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55feff98b020 0x55feff98d410 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60d31f6c861d466d server_cookie=57b53ec799859cbb in_seq=0 out_seq=0 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.210+0000 7fc4d9ffb640 10 client.17178 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/2854189746 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55feff8e3860 con 0x55feff98b020 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4d9ffb640 10 client.17178 handle_command_reply: tid=0 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 client.17178 shutdown 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2854189746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 msgr2=0x55feff978600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff978600 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0114c0 tx=0x7fc4ec0118b0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2854189746 shutdown_connections 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff978600 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55feff98b020 0x55feff98d410 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff971b40 0x55feff978010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/2854189746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff97ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2854189746 >> 172.21.15.89:0/2854189746 conn(0x55feff939310 msgr2=0x55feff972930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:05.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4d9ffb640 0 client.17178 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:29:05.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2854189746 shutdown_connections 2024-09-15T23:29:05.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2854189746 wait complete. 2024-09-15T23:29:05.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.211+0000 7fc4fad7c740 20 client.17178 trim_cache size 0 max 16384 2024-09-15T23:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4fad7c740 1 Processor -- start 2024-09-15T23:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4fad7c740 1 -- start start 2024-09-15T23:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff9705d0 0x55feff979f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff971b40 0x55feff973120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff973660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41aedd0 con 0x55feff97b370 2024-09-15T23:29:05.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41b0910 con 0x55feff971b40 2024-09-15T23:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41af4a0 con 0x55feff9705d0 2024-09-15T23:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff973660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff973660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56872/0 (socket says 172.21.15.89:56872) 2024-09-15T23:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/2883754417 learned_addr learned my addr 172.21.15.89:0/2883754417 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f259c640 1 --2- 172.21.15.89:0/2883754417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff9705d0 0x55feff979f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f1d9b640 1 --2- 172.21.15.89:0/2883754417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff971b40 0x55feff973120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/2883754417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff9705d0 msgr2=0x55feff979f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/2883754417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff9705d0 0x55feff979f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/2883754417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff971b40 msgr2=0x55feff973120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:05.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/2883754417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff971b40 0x55feff973120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f2d9d640 1 -- 172.21.15.89:0/2883754417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55feff9702d0 con 0x55feff97b370 2024-09-15T23:29:05.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.224+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/2883754417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff973660 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0054a0 tx=0x7fc4ec0097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a53de97a727939a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:05.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4dbfff640 1 -- 172.21.15.89:0/2883754417 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec046070 con 0x55feff97b370 2024-09-15T23:29:05.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4dbfff640 1 -- 172.21.15.89:0/2883754417 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4ec005040 con 0x55feff97b370 2024-09-15T23:29:05.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4dbfff640 1 -- 172.21.15.89:0/2883754417 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec027860 con 0x55feff97b370 2024-09-15T23:29:05.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2883754417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 msgr2=0x55feff973660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:05.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/2883754417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff973660 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0054a0 tx=0x7fc4ec0097a0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2883754417 shutdown_connections 2024-09-15T23:29:05.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/2883754417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff97b370 0x55feff973660 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/2883754417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff971b40 0x55feff973120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/2883754417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff9705d0 0x55feff979f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2883754417 >> 172.21.15.89:0/2883754417 conn(0x55feff939310 msgr2=0x55feff979870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2883754417 shutdown_connections 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.225+0000 7fc4fad7c740 1 -- 172.21.15.89:0/2883754417 wait complete. 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4fad7c740 1 -- start start 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4fad7c740 1 Processor -- start 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff983a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff974c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4fad7c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 0x55feff975170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41b0240 con 0x55feff971b40 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41afb70 con 0x55feff9705d0 2024-09-15T23:29:05.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4fad7c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f41acdb0 con 0x55feff97b370 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4fad7c740 10 client.0 _ll_register_callbacks cb 0x7fc4ec013f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff974c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff974c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56878/0 (socket says 172.21.15.89:56878) 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f1d9b640 1 -- 172.21.15.89:0/117459120 learned_addr learned my addr 172.21.15.89:0/117459120 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f259c640 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff983a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 0x55feff975170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f1d9b640 1 -- 172.21.15.89:0/117459120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 msgr2=0x55feff975170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f1d9b640 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 0x55feff975170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f1d9b640 1 -- 172.21.15.89:0/117459120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 msgr2=0x55feff983a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f1d9b640 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff983a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f1d9b640 1 -- 172.21.15.89:0/117459120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55feff975860 con 0x55feff971b40 2024-09-15T23:29:05.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.226+0000 7fc4f2d9d640 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 0x55feff975170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4f1d9b640 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff974c30 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80edb20 tx=0x7fc4e803b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc2eb6198ce002b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/117459120 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8049040 con 0x55feff971b40 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/117459120 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc4e80b8d60 con 0x55feff971b40 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4fad7c740 1 -- 172.21.15.89:0/117459120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4ec005040 con 0x55feff971b40 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/117459120 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc4e80bc6f0 con 0x55feff971b40 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4fad7c740 10 client.17184 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4fad7c740 1 -- 172.21.15.89:0/117459120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55feff9f5910 con 0x55feff971b40 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/117459120 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4ec005040 con 0x55feff971b40 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/117459120 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc4e80c4e20 con 0x55feff971b40 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4fad7c740 10 client.17184 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:05.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4fad7c740 10 client.17184 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4fad7c740 20 client.17184 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55feff9f5f40 0x55feff9f8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4fad7c740 4 client.17184 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4fad7c740 1 -- 172.21.15.89:0/117459120 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55feff8ddd70 con 0x55feff9f5f40 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.227+0000 7fc4f259c640 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55feff9f5f40 0x55feff9f8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4f259c640 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55feff9f5f40 0x55feff9f8330 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=279a318559ea0b5d server_cookie=1321e57460374674 in_seq=0 out_seq=0 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4d9ffb640 10 client.17184 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4d9ffb640 1 -- 172.21.15.89:0/117459120 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55feff8ddd70 con 0x55feff9f5f40 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4d9ffb640 10 client.17184 handle_command_reply: tid=0 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 client.17184 shutdown 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 -- 172.21.15.89:0/117459120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 msgr2=0x55feff974c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:05.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff974c30 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80edb20 tx=0x7fc4e803b920 comp rx=0 tx=0).stop 2024-09-15T23:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 -- 172.21.15.89:0/117459120 shutdown_connections 2024-09-15T23:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55feff9f5f40 0x55feff9f8330 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55feff97b370 0x55feff975170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55feff971b40 0x55feff974c30 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 --2- 172.21.15.89:0/117459120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55feff9705d0 0x55feff983a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 -- 172.21.15.89:0/117459120 >> 172.21.15.89:0/117459120 conn(0x55feff939310 msgr2=0x55feff97a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4d9ffb640 0 client.17184 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 -- 172.21.15.89:0/117459120 shutdown_connections 2024-09-15T23:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.228+0000 7fc4fad7c740 1 -- 172.21.15.89:0/117459120 wait complete. 2024-09-15T23:29:05.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4fad7c740 20 client.17184 trim_cache size 0 max 16384 2024-09-15T23:29:05.238 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:29:05.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 -- 172.21.15.89:0/3392752333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4dc0a3240 msgr2=0x7fc4dc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4dc0a3240 0x7fc4dc0a5700 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec021000 tx=0x7fc4ec01e000 comp rx=0 tx=0).stop 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 -- 172.21.15.89:0/3392752333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f4102570 msgr2=0x7fc4f4197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f4102570 0x7fc4f4197910 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc4e802aa70 tx=0x7fc4e8003b60 comp rx=0 tx=0).stop 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 -- 172.21.15.89:0/3392752333 shutdown_connections 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4dc0a3240 0x7fc4dc0a5700 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4f4102ef0 0x7fc4f419bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4f4102570 0x7fc4f4197910 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 --2- 172.21.15.89:0/3392752333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4f4100e20 0x7fc4f41973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 -- 172.21.15.89:0/3392752333 >> 172.21.15.89:0/3392752333 conn(0x7fc4f40fc7d0 msgr2=0x7fc4f4106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 -- 172.21.15.89:0/3392752333 shutdown_connections 2024-09-15T23:29:05.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:05.229+0000 7fc4f89a3640 1 -- 172.21.15.89:0/3392752333 wait complete. 2024-09-15T23:29:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:05 smithi120 ceph-mon[35250]: pgmap v502: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 12 op/s 2024-09-15T23:29:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:05 smithi174 ceph-mon[35586]: pgmap v502: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 12 op/s 2024-09-15T23:29:05.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:05 smithi089 ceph-mon[30933]: pgmap v502: 129 pgs: 129 active+clean; 2.5 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 12 op/s 2024-09-15T23:29:06.060 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:29:06.306 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:06.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.784+0000 7fb8f0d16640 1 -- 172.21.15.89:0/3797090733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec102db0 msgr2=0x7fb8ec103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:06.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.784+0000 7fb8f0d16640 1 --2- 172.21.15.89:0/3797090733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec102db0 0x7fb8ec103190 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc009cd0 tx=0x7fb8dc02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.785+0000 7fb8f0d16640 1 -- 172.21.15.89:0/3797090733 shutdown_connections 2024-09-15T23:29:06.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.785+0000 7fb8f0d16640 1 --2- 172.21.15.89:0/3797090733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ec1082e0 0x7fb8ec10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.785+0000 7fb8f0d16640 1 --2- 172.21.15.89:0/3797090733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec103760 0x7fb8ec107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.785+0000 7fb8f0d16640 1 --2- 172.21.15.89:0/3797090733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec102db0 0x7fb8ec103190 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.785+0000 7fb8f0d16640 1 -- 172.21.15.89:0/3797090733 >> 172.21.15.89:0/3797090733 conn(0x7fb8ec0fc7d0 msgr2=0x7fb8ec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.785+0000 7fb8f0d16640 1 -- 172.21.15.89:0/3797090733 shutdown_connections 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.785+0000 7fb8f0d16640 1 -- 172.21.15.89:0/3797090733 wait complete. 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.785+0000 7fb8f0d16640 1 Processor -- start 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.785+0000 7fb8f0d16640 1 -- start start 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8f0d16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec102db0 0x7fb8ec198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8f0d16640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ec103760 0x7fb8ec198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8f0d16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec1082e0 0x7fb8ec19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8f0d16640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1120b0 con 0x7fb8ec102db0 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8f0d16640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec111f30 con 0x7fb8ec1082e0 2024-09-15T23:29:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8f0d16640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec112230 con 0x7fb8ec103760 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8ea59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec102db0 0x7fb8ec198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8ea59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec102db0 0x7fb8ec198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56898/0 (socket says 172.21.15.89:56898) 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8ea59c640 1 -- 172.21.15.89:0/350724967 learned_addr learned my addr 172.21.15.89:0/350724967 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ec103760 0x7fb8ec198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec1082e0 0x7fb8ec19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8ea59c640 1 -- 172.21.15.89:0/350724967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ec103760 msgr2=0x7fb8ec198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8ea59c640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ec103760 0x7fb8ec198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8ea59c640 1 -- 172.21.15.89:0/350724967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec1082e0 msgr2=0x7fb8ec19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8ea59c640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec1082e0 0x7fb8ec19ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8ea59c640 1 -- 172.21.15.89:0/350724967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec19d540 con 0x7fb8ec102db0 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8ea59c640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec102db0 0x7fb8ec198590 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc02ae00 tx=0x7fb8dc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:06.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8db7fe640 1 -- 172.21.15.89:0/350724967 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc004240 con 0x7fb8ec102db0 2024-09-15T23:29:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8db7fe640 1 -- 172.21.15.89:0/350724967 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8dc031d50 con 0x7fb8ec102db0 2024-09-15T23:29:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8db7fe640 1 -- 172.21.15.89:0/350724967 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc03a400 con 0x7fb8ec102db0 2024-09-15T23:29:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8f0d16640 1 -- 172.21.15.89:0/350724967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ec19d7d0 con 0x7fb8ec102db0 2024-09-15T23:29:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.786+0000 7fb8f0d16640 1 -- 172.21.15.89:0/350724967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ec1a5110 con 0x7fb8ec102db0 2024-09-15T23:29:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.789+0000 7fb8db7fe640 1 -- 172.21.15.89:0/350724967 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb8dc0306c0 con 0x7fb8ec102db0 2024-09-15T23:29:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.789+0000 7fb8f30ef740 1 Processor -- start 2024-09-15T23:29:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.789+0000 7fb8f30ef740 1 -- start start 2024-09-15T23:29:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.789+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 0x5648de952090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.789+0000 7fb8db7fe640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb8d40a3240 0x7fb8d40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.789+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de952a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.789+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de963b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.789+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5648de8b82f0 con 0x5648de9525d0 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.789+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5648de85fcf0 con 0x5648de961780 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.789+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5648de8b42b0 con 0x5648de953b40 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8ea59c640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb8d40a3240 0x7fb8d40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8e9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de952a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8e9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de952a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56910/0 (socket says 172.21.15.89:56910) 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8e9d9b640 1 -- 172.21.15.89:0/33177216 learned_addr learned my addr 172.21.15.89:0/33177216 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8db7fe640 1 -- 172.21.15.89:0/350724967 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb8dc02e030 con 0x7fb8ec102db0 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/33177216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 0x5648de952090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/33177216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de963b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8e9d9b640 1 -- 172.21.15.89:0/33177216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 msgr2=0x5648de952090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:06.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/33177216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 0x5648de952090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8e9d9b640 1 -- 172.21.15.89:0/33177216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 msgr2=0x5648de963b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/33177216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de963b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8e9d9b640 1 -- 172.21.15.89:0/33177216 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648de8443a0 con 0x5648de9525d0 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8ea59c640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb8d40a3240 0x7fb8d40a5700 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0bcfb0 tx=0x7fb8dc03d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/33177216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de952a10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000bc30 tx=0x7fb8e00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e0b4ab5b6af31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8d8ff9640 1 -- 172.21.15.89:0/33177216 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e001c480 con 0x5648de9525d0 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8d8ff9640 1 -- 172.21.15.89:0/33177216 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8e001cb00 con 0x5648de9525d0 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.790+0000 7fb8d8ff9640 1 -- 172.21.15.89:0/33177216 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e0011af0 con 0x5648de9525d0 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.791+0000 7fb8f30ef740 1 -- 172.21.15.89:0/33177216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 msgr2=0x5648de952a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.791+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/33177216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de952a10 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fb8e000bc30 tx=0x7fb8e00107a0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.791+0000 7fb8f30ef740 1 -- 172.21.15.89:0/33177216 shutdown_connections 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.791+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/33177216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de963b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.791+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/33177216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de952a10 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.791+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/33177216 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 0x5648de952090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.791+0000 7fb8f30ef740 1 -- 172.21.15.89:0/33177216 >> 172.21.15.89:0/33177216 conn(0x5648de91b310 msgr2=0x5648de94ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.791+0000 7fb8f30ef740 1 -- 172.21.15.89:0/33177216 shutdown_connections 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.791+0000 7fb8f30ef740 1 -- 172.21.15.89:0/33177216 wait complete. 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.791+0000 7fb8f30ef740 1 -- start start 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8f30ef740 1 Processor -- start 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de9525d0 0x5648de961170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de953b40 0x5648de95e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de95ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5648de964480 con 0x5648de953b40 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5648de873060 con 0x5648de961780 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5648de964600 con 0x5648de9525d0 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8f30ef740 10 client.0 _ll_register_callbacks cb 0x5648de5c68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8e9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de953b40 0x5648de95e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8e9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de953b40 0x5648de95e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56922/0 (socket says 172.21.15.89:56922) 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8e9d9b640 1 -- 172.21.15.89:0/1017914916 learned_addr learned my addr 172.21.15.89:0/1017914916 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de9525d0 0x5648de961170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de95ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8e9d9b640 1 -- 172.21.15.89:0/1017914916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de9525d0 msgr2=0x5648de961170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de9525d0 0x5648de961170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8e9d9b640 1 -- 172.21.15.89:0/1017914916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 msgr2=0x5648de95ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de95ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8e9d9b640 1 -- 172.21.15.89:0/1017914916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648de6293c0 con 0x5648de953b40 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de953b40 0x5648de95e420 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0000c00 tx=0x7fb8e0011670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d016cf73d2c3b978 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.792+0000 7fb8d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/1017914916 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e002b060 con 0x5648de953b40 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8f30ef740 1 -- 172.21.15.89:0/1017914916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8e001cb00 con 0x5648de953b40 2024-09-15T23:29:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/1017914916 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb8e001c480 con 0x5648de953b40 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/1017914916 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb8e001c680 con 0x5648de953b40 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8f30ef740 10 client.17196 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8f30ef740 1 -- 172.21.15.89:0/1017914916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5648de96ff20 con 0x5648de953b40 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/1017914916 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8e001cb00 con 0x5648de953b40 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/1017914916 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb8e0005310 con 0x5648de953b40 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8f30ef740 10 client.17196 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8f30ef740 10 client.17196 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8f30ef740 20 client.17196 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5648de9712d0 0x5648de9736c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8f30ef740 4 client.17196 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8f30ef740 1 -- 172.21.15.89:0/1017914916 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648de8c5860 con 0x5648de9712d0 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.793+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5648de9712d0 0x5648de9736c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.794+0000 7fb8d1ffb640 10 client.17196 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:29:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.794+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5648de9712d0 0x5648de9736c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1564ff02ddb09777 server_cookie=4c23a73ec0e4a843 in_seq=0 out_seq=0 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/1017914916 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5648de8c5860 con 0x5648de9712d0 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8d1ffb640 10 client.17196 handle_command_reply: tid=0 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 client.17196 shutdown 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 -- 172.21.15.89:0/1017914916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de953b40 msgr2=0x5648de95e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de953b40 0x5648de95e420 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0000c00 tx=0x7fb8e0011670 comp rx=0 tx=0).stop 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 -- 172.21.15.89:0/1017914916 shutdown_connections 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5648de9712d0 0x5648de9736c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de95ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de953b40 0x5648de95e420 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8d1ffb640 0 client.17196 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/1017914916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de9525d0 0x5648de961170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 -- 172.21.15.89:0/1017914916 >> 172.21.15.89:0/1017914916 conn(0x5648de91b310 msgr2=0x5648de954960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 -- 172.21.15.89:0/1017914916 shutdown_connections 2024-09-15T23:29:06.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 1 -- 172.21.15.89:0/1017914916 wait complete. 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.795+0000 7fb8f30ef740 20 client.17196 trim_cache size 0 max 16384 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8f30ef740 1 Processor -- start 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8f30ef740 1 -- start start 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de9525d0 0x5648de96d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de953b40 0x5648de949ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de961780 0x5648de94dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1acc00 con 0x5648de961780 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1ae740 con 0x5648de953b40 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1ad2d0 con 0x5648de9525d0 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8ead9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de961780 0x5648de94dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8ead9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de961780 0x5648de94dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56932/0 (socket says 172.21.15.89:56932) 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8ead9d640 1 -- 172.21.15.89:0/3826972484 learned_addr learned my addr 172.21.15.89:0/3826972484 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/3826972484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de9525d0 0x5648de96d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/3826972484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de953b40 0x5648de949ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8ead9d640 1 -- 172.21.15.89:0/3826972484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de9525d0 msgr2=0x5648de96d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:06.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/3826972484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de9525d0 0x5648de96d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8ead9d640 1 -- 172.21.15.89:0/3826972484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de953b40 msgr2=0x5648de949ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:06.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/3826972484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de953b40 0x5648de949ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8ead9d640 1 -- 172.21.15.89:0/3826972484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648de95c9b0 con 0x5648de961780 2024-09-15T23:29:06.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/3826972484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de961780 0x5648de94dbf0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400e700 tx=0x7fb8e400e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d472cf642ac032 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:06.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.808+0000 7fb8d3fff640 1 -- 172.21.15.89:0/3826972484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e401c040 con 0x5648de961780 2024-09-15T23:29:06.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8d3fff640 1 -- 172.21.15.89:0/3826972484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8e4013dc0 con 0x5648de961780 2024-09-15T23:29:06.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8d3fff640 1 -- 172.21.15.89:0/3826972484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e40047c0 con 0x5648de961780 2024-09-15T23:29:06.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8f30ef740 1 -- 172.21.15.89:0/3826972484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de961780 msgr2=0x5648de94dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:06.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/3826972484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de961780 0x5648de94dbf0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fb8e400e700 tx=0x7fb8e400e2f0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8f30ef740 1 -- 172.21.15.89:0/3826972484 shutdown_connections 2024-09-15T23:29:06.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/3826972484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de961780 0x5648de94dbf0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/3826972484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de953b40 0x5648de949ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/3826972484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de9525d0 0x5648de96d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8f30ef740 1 -- 172.21.15.89:0/3826972484 >> 172.21.15.89:0/3826972484 conn(0x5648de91b310 msgr2=0x5648de94d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:06.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8f30ef740 1 -- 172.21.15.89:0/3826972484 shutdown_connections 2024-09-15T23:29:06.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8f30ef740 1 -- 172.21.15.89:0/3826972484 wait complete. 2024-09-15T23:29:06.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.809+0000 7fb8f30ef740 1 -- start start 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8f30ef740 1 Processor -- start 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de969890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 0x5648de969dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8f30ef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de95aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1ae070 con 0x5648de9525d0 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1ad9a0 con 0x5648de961780 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8f30ef740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1aabe0 con 0x5648de953b40 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8f30ef740 10 client.0 _ll_register_callbacks cb 0x7fb8e000ba80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8ead9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de969890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8ead9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de969890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56934/0 (socket says 172.21.15.89:56934) 2024-09-15T23:29:06.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8ead9d640 1 -- 172.21.15.89:0/485056714 learned_addr learned my addr 172.21.15.89:0/485056714 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 0x5648de969dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de95aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8ead9d640 1 -- 172.21.15.89:0/485056714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 msgr2=0x5648de969dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 0x5648de969dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8ead9d640 1 -- 172.21.15.89:0/485056714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 msgr2=0x5648de95aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de95aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8ead9d640 1 -- 172.21.15.89:0/485056714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648de95b070 con 0x5648de9525d0 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 0x5648de969dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8ead9d640 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de969890 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40057a0 tx=0x7fb8e400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1c559273fb76db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.810+0000 7fb8d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:06.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/485056714 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4005230 con 0x5648de9525d0 2024-09-15T23:29:06.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/485056714 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb8e4004050 con 0x5648de9525d0 2024-09-15T23:29:06.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8f30ef740 1 -- 172.21.15.89:0/485056714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8e4013dc0 con 0x5648de9525d0 2024-09-15T23:29:06.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/485056714 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb8e4004250 con 0x5648de9525d0 2024-09-15T23:29:06.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8f30ef740 10 client.17202 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:06.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8f30ef740 1 -- 172.21.15.89:0/485056714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5648de9db500 con 0x5648de9525d0 2024-09-15T23:29:06.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/485056714 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8e4034070 con 0x5648de9525d0 2024-09-15T23:29:06.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/485056714 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb8e400f040 con 0x5648de9525d0 2024-09-15T23:29:06.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8f30ef740 10 client.17202 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:06.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8f30ef740 10 client.17202 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:29:06.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8f30ef740 20 client.17202 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:06.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5648de9dbb10 0x5648de9ddf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:06.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8f30ef740 4 client.17202 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:06.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8f30ef740 1 -- 172.21.15.89:0/485056714 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5648de8bfd70 con 0x5648de9dbb10 2024-09-15T23:29:06.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.811+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5648de9dbb10 0x5648de9ddf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:06.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8d1ffb640 10 client.17202 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:29:06.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8e9d9b640 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5648de9dbb10 0x5648de9ddf00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c375dec2d11344b4 server_cookie=b73b9ae1f77f3478 in_seq=0 out_seq=0 2024-09-15T23:29:06.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8d1ffb640 1 -- 172.21.15.89:0/485056714 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5648de8bfd70 con 0x5648de9dbb10 2024-09-15T23:29:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8d1ffb640 10 client.17202 handle_command_reply: tid=0 2024-09-15T23:29:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 client.17202 shutdown 2024-09-15T23:29:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 -- 172.21.15.89:0/485056714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 msgr2=0x5648de969890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de969890 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40057a0 tx=0x7fb8e400ec20 comp rx=0 tx=0).stop 2024-09-15T23:29:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 -- 172.21.15.89:0/485056714 shutdown_connections 2024-09-15T23:29:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5648de9dbb10 0x5648de9ddf00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648de961780 0x5648de95aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648de953b40 0x5648de969dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8d1ffb640 0 client.17202 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:29:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 --2- 172.21.15.89:0/485056714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5648de9525d0 0x5648de969890 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 -- 172.21.15.89:0/485056714 >> 172.21.15.89:0/485056714 conn(0x5648de91b310 msgr2=0x5648de95fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:06.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 -- 172.21.15.89:0/485056714 shutdown_connections 2024-09-15T23:29:06.840 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 1 -- 172.21.15.89:0/485056714 wait complete. 2024-09-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.812+0000 7fb8f30ef740 20 client.17202 trim_cache size 0 max 16384 2024-09-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 -- 172.21.15.89:0/350724967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb8d40a3240 msgr2=0x7fb8d40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb8d40a3240 0x7fb8d40a5700 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0bcfb0 tx=0x7fb8dc03d040 comp rx=0 tx=0).stop 2024-09-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 -- 172.21.15.89:0/350724967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec102db0 msgr2=0x7fb8ec198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec102db0 0x7fb8ec198590 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc02ae00 tx=0x7fb8dc003ba0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 -- 172.21.15.89:0/350724967 shutdown_connections 2024-09-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb8d40a3240 0x7fb8d40a5700 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb8ec1082e0 0x7fb8ec19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ec103760 0x7fb8ec198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 --2- 172.21.15.89:0/350724967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb8ec102db0 0x7fb8ec198590 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:06.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 -- 172.21.15.89:0/350724967 >> 172.21.15.89:0/350724967 conn(0x7fb8ec0fc7d0 msgr2=0x7fb8ec105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:06.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 -- 172.21.15.89:0/350724967 shutdown_connections 2024-09-15T23:29:06.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:06.813+0000 7fb8f0d16640 1 -- 172.21.15.89:0/350724967 wait complete. 2024-09-15T23:29:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:07 smithi120 ceph-mon[35250]: pgmap v503: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 20 op/s 2024-09-15T23:29:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:07 smithi174 ceph-mon[35586]: pgmap v503: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 20 op/s 2024-09-15T23:29:07.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:07 smithi089 ceph-mon[30933]: pgmap v503: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 20 op/s 2024-09-15T23:29:07.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:29:07.895 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:08.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.402+0000 7f36f39ef640 1 -- 172.21.15.89:0/544012958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36ec100e20 msgr2=0x7f36ec101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.402+0000 7f36f39ef640 1 --2- 172.21.15.89:0/544012958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36ec100e20 0x7f36ec101200 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009900 tx=0x7f36e802a480 comp rx=0 tx=0).stop 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.402+0000 7f36f39ef640 1 -- 172.21.15.89:0/544012958 shutdown_connections 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.402+0000 7f36f39ef640 1 --2- 172.21.15.89:0/544012958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ec102ef0 0x7f36ec10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.402+0000 7f36f39ef640 1 --2- 172.21.15.89:0/544012958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ec102570 0x7f36ec1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.402+0000 7f36f39ef640 1 --2- 172.21.15.89:0/544012958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36ec100e20 0x7f36ec101200 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.402+0000 7f36f39ef640 1 -- 172.21.15.89:0/544012958 >> 172.21.15.89:0/544012958 conn(0x7f36ec0fc7d0 msgr2=0x7f36ec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f39ef640 1 -- 172.21.15.89:0/544012958 shutdown_connections 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f39ef640 1 -- 172.21.15.89:0/544012958 wait complete. 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f39ef640 1 Processor -- start 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f39ef640 1 -- start start 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f39ef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36ec100e20 0x7f36ec196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f39ef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ec102570 0x7f36ec196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f39ef640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ec102ef0 0x7f36ec19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f39ef640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f36ec10fec0 con 0x7f36ec100e20 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f39ef640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f36ec10fd40 con 0x7f36ec102ef0 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f39ef640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36ec110040 con 0x7f36ec102570 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f178b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36ec100e20 0x7f36ec196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f178b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36ec100e20 0x7f36ec196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56960/0 (socket says 172.21.15.89:56960) 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.403+0000 7f36f178b640 1 -- 172.21.15.89:0/1256264818 learned_addr learned my addr 172.21.15.89:0/1256264818 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ec102ef0 0x7f36ec19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f0f8a640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ec102570 0x7f36ec196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f178b640 1 -- 172.21.15.89:0/1256264818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ec102570 msgr2=0x7f36ec196840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f178b640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ec102570 0x7f36ec196840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f178b640 1 -- 172.21.15.89:0/1256264818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ec102ef0 msgr2=0x7f36ec19abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f178b640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ec102ef0 0x7f36ec19abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f178b640 1 -- 172.21.15.89:0/1256264818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36ec19b2b0 con 0x7f36ec100e20 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f0f8a640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ec102570 0x7f36ec196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f178b640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36ec100e20 0x7f36ec196300 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f36e8006010 tx=0x7f36e8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36e27fc640 1 -- 172.21.15.89:0/1256264818 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e803f070 con 0x7f36ec100e20 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36e27fc640 1 -- 172.21.15.89:0/1256264818 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e8031c30 con 0x7f36ec100e20 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f39ef640 1 -- 172.21.15.89:0/1256264818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36ec19b540 con 0x7f36ec100e20 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36e27fc640 1 -- 172.21.15.89:0/1256264818 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8035030 con 0x7f36ec100e20 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.404+0000 7f36f39ef640 1 -- 172.21.15.89:0/1256264818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36ec1a2e80 con 0x7f36ec100e20 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.406+0000 7f36e27fc640 1 -- 172.21.15.89:0/1256264818 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f36e8042070 con 0x7f36ec100e20 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f5dc8740 1 Processor -- start 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f5dc8740 1 -- start start 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef82b40 0x55a59ef81090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef815d0 0x55a59ef81a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 0x55a59ef8e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a59eee72f0 con 0x55a59ef82b40 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a59ee8ecf0 con 0x55a59ef815d0 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36e27fc640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36d80a5450 0x7f36d80a7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a59eee32b0 con 0x55a59ef8c370 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f1f8c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef82b40 0x55a59ef81090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f1f8c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef82b40 0x55a59ef81090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56966/0 (socket says 172.21.15.89:56966) 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1114658565 learned_addr learned my addr 172.21.15.89:0/1114658565 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f178b640 1 --2- 172.21.15.89:0/1114658565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef815d0 0x55a59ef81a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36d80a5450 0x7f36d80a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36e27fc640 1 -- 172.21.15.89:0/1256264818 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f36e80f0560 con 0x7f36ec100e20 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.407+0000 7f36f0f8a640 1 --2- 172.21.15.89:0/1114658565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 0x55a59ef8e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1114658565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 msgr2=0x55a59ef8e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1114658565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 0x55a59ef8e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1114658565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef815d0 msgr2=0x55a59ef81a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1114658565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef815d0 0x55a59ef81a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1114658565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a59ee733a0 con 0x55a59ef82b40 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36f0f8a640 1 --2- 172.21.15.89:0/1114658565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 0x55a59ef8e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1114658565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef82b40 0x55a59ef81090 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f36e4012d80 tx=0x7f36e4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3457817b688e5930 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36d80a5450 0x7f36d80a7910 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f36e4010c40 tx=0x7f36e400c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36d77fe640 1 -- 172.21.15.89:0/1114658565 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4024070 con 0x55a59ef82b40 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36d77fe640 1 -- 172.21.15.89:0/1114658565 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e40125a0 con 0x55a59ef82b40 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.408+0000 7f36d77fe640 1 -- 172.21.15.89:0/1114658565 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4005220 con 0x55a59ef82b40 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1114658565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef82b40 msgr2=0x55a59ef81090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1114658565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef82b40 0x55a59ef81090 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f36e4012d80 tx=0x7f36e4010740 comp rx=0 tx=0).stop 2024-09-15T23:29:08.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1114658565 shutdown_connections 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1114658565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 0x55a59ef8e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1114658565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef815d0 0x55a59ef81a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1114658565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef82b40 0x55a59ef81090 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1114658565 >> 172.21.15.89:0/1114658565 conn(0x55a59ef4a310 msgr2=0x55a59ef7dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1114658565 shutdown_connections 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1114658565 wait complete. 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 -- start start 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.409+0000 7f36f5dc8740 1 Processor -- start 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef815d0 0x55a59ef8bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef82b40 0x55a59ef89050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef897e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a59ef8f010 con 0x55a59ef8c370 2024-09-15T23:29:08.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a59eea2060 con 0x55a59ef815d0 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a59ef8f190 con 0x55a59ef82b40 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f5dc8740 10 client.0 _ll_register_callbacks cb 0x55a59ebf58c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f1f8c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef897e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f1f8c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef897e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56972/0 (socket says 172.21.15.89:56972) 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1263203926 learned_addr learned my addr 172.21.15.89:0/1263203926 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f178b640 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef815d0 0x55a59ef8bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f0f8a640 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef82b40 0x55a59ef89050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1263203926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef82b40 msgr2=0x55a59ef89050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef82b40 0x55a59ef89050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1263203926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef815d0 msgr2=0x55a59ef8bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef815d0 0x55a59ef8bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1263203926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a59ec583c0 con 0x55a59ef8c370 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f0f8a640 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef82b40 0x55a59ef89050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef897e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f36e4012fd0 tx=0x7f36e4009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0220256511df4a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1263203926 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4024050 con 0x55a59ef8c370 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1263203926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36e40125a0 con 0x55a59ef8c370 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1263203926 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f36e4005730 con 0x55a59ef8c370 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1263203926 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f36e4039050 con 0x55a59ef8c370 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f5dc8740 10 client.17214 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.410+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1263203926 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a59ef9aaf0 con 0x55a59ef8c370 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.411+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1263203926 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e40125a0 con 0x55a59ef8c370 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.411+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1263203926 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f36e400f040 con 0x55a59ef8c370 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.411+0000 7f36f5dc8740 10 client.17214 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.411+0000 7f36f5dc8740 10 client.17214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:08.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.411+0000 7f36f5dc8740 20 client.17214 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.411+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a59ef9bfb0 0x55a59ef9e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.411+0000 7f36f5dc8740 4 client.17214 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.411+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1263203926 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a59eef4860 con 0x55a59ef9bfb0 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.411+0000 7f36f178b640 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a59ef9bfb0 0x55a59ef9e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36d4ff9640 10 client.17214 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36f178b640 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a59ef9bfb0 0x55a59ef9e3a0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e840bd0b4377068c server_cookie=6efacb24ea7461c7 in_seq=0 out_seq=0 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1263203926 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a59eef4860 con 0x55a59ef9bfb0 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36d4ff9640 10 client.17214 handle_command_reply: tid=0 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36f5dc8740 1 client.17214 shutdown 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1263203926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 msgr2=0x55a59ef897e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef897e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f36e4012fd0 tx=0x7f36e4009a70 comp rx=0 tx=0).stop 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1263203926 shutdown_connections 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a59ef9bfb0 0x55a59ef9e3a0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef897e0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef82b40 0x55a59ef89050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36d4ff9640 0 client.17214 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:08.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1263203926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef815d0 0x55a59ef8bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.412+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1263203926 >> 172.21.15.89:0/1263203926 conn(0x55a59ef4a310 msgr2=0x55a59ef83960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.413+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1263203926 shutdown_connections 2024-09-15T23:29:08.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.413+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1263203926 wait complete. 2024-09-15T23:29:08.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.413+0000 7f36f5dc8740 20 client.17214 trim_cache size 0 max 16384 2024-09-15T23:29:08.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f5dc8740 1 Processor -- start 2024-09-15T23:29:08.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f5dc8740 1 -- start start 2024-09-15T23:29:08.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef815d0 0x55a59ef7dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 0x55a59ef84120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 0x55a59ef84660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f36ec1acbc0 con 0x55a59ef815d0 2024-09-15T23:29:08.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f36ec1ae700 con 0x55a59ef82b40 2024-09-15T23:29:08.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36ec1ad290 con 0x55a59ef8c370 2024-09-15T23:29:08.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f178b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef815d0 0x55a59ef7dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f178b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef815d0 0x55a59ef7dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56986/0 (socket says 172.21.15.89:56986) 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f178b640 1 -- 172.21.15.89:0/2655245304 learned_addr learned my addr 172.21.15.89:0/2655245304 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.426+0000 7f36f0f8a640 1 --2- 172.21.15.89:0/2655245304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 0x55a59ef84120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/2655245304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 0x55a59ef84660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36f178b640 1 -- 172.21.15.89:0/2655245304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 msgr2=0x55a59ef84660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36f178b640 1 --2- 172.21.15.89:0/2655245304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 0x55a59ef84660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36f178b640 1 -- 172.21.15.89:0/2655245304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 msgr2=0x55a59ef84120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36f178b640 1 --2- 172.21.15.89:0/2655245304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 0x55a59ef84120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36f178b640 1 -- 172.21.15.89:0/2655245304 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a59ef7e960 con 0x55a59ef815d0 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36f0f8a640 1 --2- 172.21.15.89:0/2655245304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 0x55a59ef84120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36f178b640 1 --2- 172.21.15.89:0/2655245304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef815d0 0x55a59ef7dd70 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f36e8000c00 tx=0x7f36e803b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d15b41f23729f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.440 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36d6ffd640 1 -- 172.21.15.89:0/2655245304 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8049040 con 0x55a59ef815d0 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36d6ffd640 1 -- 172.21.15.89:0/2655245304 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e80bc790 con 0x55a59ef815d0 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.427+0000 7f36d6ffd640 1 -- 172.21.15.89:0/2655245304 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e80b8e00 con 0x55a59ef815d0 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 -- 172.21.15.89:0/2655245304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef815d0 msgr2=0x55a59ef7dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/2655245304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef815d0 0x55a59ef7dd70 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f36e8000c00 tx=0x7f36e803b870 comp rx=0 tx=0).stop 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 -- 172.21.15.89:0/2655245304 shutdown_connections 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/2655245304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef8c370 0x55a59ef84660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/2655245304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 0x55a59ef84120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/2655245304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef815d0 0x55a59ef7dd70 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 -- 172.21.15.89:0/2655245304 >> 172.21.15.89:0/2655245304 conn(0x55a59ef4a310 msgr2=0x55a59ef8b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 -- 172.21.15.89:0/2655245304 shutdown_connections 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 -- 172.21.15.89:0/2655245304 wait complete. 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 -- start start 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.428+0000 7f36f5dc8740 1 Processor -- start 2024-09-15T23:29:08.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef815d0 0x55a59ef94600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 0x55a59ef94b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f5dc8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef913f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f36ec1ae030 con 0x55a59ef8c370 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f36ec1ad960 con 0x55a59ef82b40 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f5dc8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36ec1aaba0 con 0x55a59ef815d0 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f5dc8740 10 client.0 _ll_register_callbacks cb 0x7f36e8005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f1f8c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef913f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f1f8c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef913f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56994/0 (socket says 172.21.15.89:56994) 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1147496673 learned_addr learned my addr 172.21.15.89:0/1147496673 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f178b640 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef815d0 0x55a59ef94600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f0f8a640 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 0x55a59ef94b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1147496673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef815d0 msgr2=0x55a59ef94600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef815d0 0x55a59ef94600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1147496673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 msgr2=0x55a59ef94b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 0x55a59ef94b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f1f8c640 1 -- 172.21.15.89:0/1147496673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a59ef91b00 con 0x55a59ef8c370 2024-09-15T23:29:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f178b640 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef815d0 0x55a59ef94600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f0f8a640 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 0x55a59ef94b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f1f8c640 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef913f0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f36e4010780 tx=0x7f36e4010c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe5188953c8cccc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1147496673 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4024070 con 0x55a59ef8c370 2024-09-15T23:29:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1147496673 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f36e4005730 con 0x55a59ef8c370 2024-09-15T23:29:08.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.429+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1147496673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36e80bc790 con 0x55a59ef8c370 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1147496673 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f36e4009600 con 0x55a59ef8c370 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36f5dc8740 10 client.17220 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1147496673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a59f0073e0 con 0x55a59ef8c370 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1147496673 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e80bc790 con 0x55a59ef8c370 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1147496673 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f36e4020c20 con 0x55a59ef8c370 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36f5dc8740 10 client.17220 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36f5dc8740 10 client.17220 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36f5dc8740 20 client.17220 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a59f007a50 0x55a59f009e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36f5dc8740 4 client.17220 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:08.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1147496673 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a59eeeed70 con 0x55a59f007a50 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.430+0000 7f36f178b640 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a59f007a50 0x55a59f009e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f178b640 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a59f007a50 0x55a59f009e40 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17aed62d50238b99 server_cookie=b02c97403e4c3a66 in_seq=0 out_seq=0 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36d4ff9640 10 client.17220 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36d4ff9640 1 -- 172.21.15.89:0/1147496673 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a59eeeed70 con 0x55a59f007a50 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36d4ff9640 10 client.17220 handle_command_reply: tid=0 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 client.17220 shutdown 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1147496673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 msgr2=0x55a59ef913f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef913f0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f36e4010780 tx=0x7f36e4010c70 comp rx=0 tx=0).stop 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1147496673 shutdown_connections 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a59f007a50 0x55a59f009e40 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a59ef8c370 0x55a59ef913f0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a59ef82b40 0x55a59ef94b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 --2- 172.21.15.89:0/1147496673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a59ef815d0 0x55a59ef94600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1147496673 >> 172.21.15.89:0/1147496673 conn(0x55a59ef4a310 msgr2=0x55a59ef8a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36d4ff9640 0 client.17220 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1147496673 shutdown_connections 2024-09-15T23:29:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 1 -- 172.21.15.89:0/1147496673 wait complete. 2024-09-15T23:29:08.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.431+0000 7f36f5dc8740 20 client.17220 trim_cache size 0 max 16384 2024-09-15T23:29:08.446 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 -- 172.21.15.89:0/1256264818 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36d80a5450 msgr2=0x7f36d80a7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36d80a5450 0x7f36d80a7910 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f36e4010c40 tx=0x7f36e400c000 comp rx=0 tx=0).stop 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 -- 172.21.15.89:0/1256264818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36ec100e20 msgr2=0x7f36ec196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36ec100e20 0x7f36ec196300 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f36e8006010 tx=0x7f36e8003c10 comp rx=0 tx=0).stop 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 -- 172.21.15.89:0/1256264818 shutdown_connections 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f36d80a5450 0x7f36d80a7910 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f36ec102ef0 0x7f36ec19abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36ec102570 0x7f36ec196840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 --2- 172.21.15.89:0/1256264818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36ec100e20 0x7f36ec196300 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 -- 172.21.15.89:0/1256264818 >> 172.21.15.89:0/1256264818 conn(0x7f36ec0fc7d0 msgr2=0x7f36ec0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 -- 172.21.15.89:0/1256264818 shutdown_connections 2024-09-15T23:29:08.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:08.432+0000 7f36f39ef640 1 -- 172.21.15.89:0/1256264818 wait complete. 2024-09-15T23:29:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:09 smithi120 ceph-mon[35250]: pgmap v504: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:29:09.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:09 smithi174 ceph-mon[35586]: pgmap v504: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:29:09.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:09 smithi089 ceph-mon[30933]: pgmap v504: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:29:10.311 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:29:10.311 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:29:10.570 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:11.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.056+0000 7fc1d19c8640 1 -- 172.21.15.89:0/4047238935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc101400 msgr2=0x7fc1cc10d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:11.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.056+0000 7fc1d19c8640 1 --2- 172.21.15.89:0/4047238935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc101400 0x7fc1cc10d930 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009cd0 tx=0x7fc1c402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.056+0000 7fc1d19c8640 1 -- 172.21.15.89:0/4047238935 shutdown_connections 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.056+0000 7fc1d19c8640 1 --2- 172.21.15.89:0/4047238935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc101400 0x7fc1cc10d930 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.056+0000 7fc1d19c8640 1 --2- 172.21.15.89:0/4047238935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc100a60 0x7fc1cc100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.056+0000 7fc1d19c8640 1 --2- 172.21.15.89:0/4047238935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1cc106a60 0x7fc1cc106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.056+0000 7fc1d19c8640 1 -- 172.21.15.89:0/4047238935 >> 172.21.15.89:0/4047238935 conn(0x7fc1cc0fc780 msgr2=0x7fc1cc0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.056+0000 7fc1d19c8640 1 -- 172.21.15.89:0/4047238935 shutdown_connections 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.056+0000 7fc1d19c8640 1 -- 172.21.15.89:0/4047238935 wait complete. 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.057+0000 7fc1d19c8640 1 Processor -- start 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.057+0000 7fc1d19c8640 1 -- start start 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.057+0000 7fc1d19c8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc100a60 0x7fc1cc199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.057+0000 7fc1d19c8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc101400 0x7fc1cc1997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.057+0000 7fc1d19c8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1cc106a60 0x7fc1cc1a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.057+0000 7fc1d19c8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc110e60 con 0x7fc1cc101400 2024-09-15T23:29:11.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.057+0000 7fc1d19c8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc110ce0 con 0x7fc1cc106a60 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.057+0000 7fc1d19c8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc110fe0 con 0x7fc1cc100a60 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1c27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc101400 0x7fc1cc1997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1c27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc101400 0x7fc1cc1997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56998/0 (socket says 172.21.15.89:56998) 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1c27fc640 1 -- 172.21.15.89:0/2559201127 learned_addr learned my addr 172.21.15.89:0/2559201127 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1caffd640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc100a60 0x7fc1cc199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1cc106a60 0x7fc1cc1a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1c27fc640 1 -- 172.21.15.89:0/2559201127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc100a60 msgr2=0x7fc1cc199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc100a60 0x7fc1cc199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1c27fc640 1 -- 172.21.15.89:0/2559201127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1cc106a60 msgr2=0x7fc1cc1a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1cc106a60 0x7fc1cc1a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1c27fc640 1 -- 172.21.15.89:0/2559201127 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1cc1a4840 con 0x7fc1cc101400 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc101400 0x7fc1cc1997c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400d770 tx=0x7fc1b400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:11.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1d09c6640 1 -- 172.21.15.89:0/2559201127 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4017070 con 0x7fc1cc101400 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1d09c6640 1 -- 172.21.15.89:0/2559201127 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1b4005500 con 0x7fc1cc101400 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1d19c8640 1 -- 172.21.15.89:0/2559201127 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1cc1a4a10 con 0x7fc1cc101400 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1d09c6640 1 -- 172.21.15.89:0/2559201127 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4004e00 con 0x7fc1cc101400 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.058+0000 7fc1d19c8640 1 -- 172.21.15.89:0/2559201127 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1cc1a4ed0 con 0x7fc1cc101400 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.061+0000 7fc1d09c6640 1 -- 172.21.15.89:0/2559201127 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc1b4004fa0 con 0x7fc1cc101400 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.061+0000 7fc1d3da1740 1 Processor -- start 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.061+0000 7fc1d3da1740 1 -- start start 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.061+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fcaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.061+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.061+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fd8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c2a3f30280 con 0x55c2a3fcb480 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c2a3ed7b10 con 0x55c2a3fd5d40 2024-09-15T23:29:11.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c2a3f2c240 con 0x55c2a3fcab60 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1caffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1caffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57000/0 (socket says 172.21.15.89:57000) 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1caffd640 1 -- 172.21.15.89:0/1152026069 learned_addr learned my addr 172.21.15.89:0/1152026069 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/1152026069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fcaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/1152026069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fd8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1d09c6640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1b80a3240 0x7fc1b80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1caffd640 1 -- 172.21.15.89:0/1152026069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 msgr2=0x55c2a3fcaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1caffd640 1 --2- 172.21.15.89:0/1152026069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fcaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1caffd640 1 -- 172.21.15.89:0/1152026069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 msgr2=0x55c2a3fd8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1caffd640 1 --2- 172.21.15.89:0/1152026069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fd8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1caffd640 1 -- 172.21.15.89:0/1152026069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2a3ee41f0 con 0x55c2a3fcb480 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/1152026069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fd8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:11.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1b80a3240 0x7fc1b80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1d09c6640 1 -- 172.21.15.89:0/2559201127 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc1b40cf780 con 0x7fc1cc101400 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1caffd640 1 --2- 172.21.15.89:0/1152026069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd5800 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00bc90 tx=0x7fc1bc0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be56824749b2651 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1c1ffb640 1 -- 172.21.15.89:0/1152026069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc01c480 con 0x55c2a3fcb480 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1c1ffb640 1 -- 172.21.15.89:0/1152026069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc01cb00 con 0x55c2a3fcb480 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.062+0000 7fc1c1ffb640 1 -- 172.21.15.89:0/1152026069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc011bb0 con 0x55c2a3fcb480 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1d3da1740 1 -- 172.21.15.89:0/1152026069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 msgr2=0x55c2a3fd5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1b80a3240 0x7fc1b80a5700 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4005e30 tx=0x7fc1c403f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/1152026069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd5800 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00bc90 tx=0x7fc1bc0107a0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1d3da1740 1 -- 172.21.15.89:0/1152026069 shutdown_connections 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/1152026069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fd8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/1152026069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd5800 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/1152026069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fcaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1d3da1740 1 -- 172.21.15.89:0/1152026069 >> 172.21.15.89:0/1152026069 conn(0x55c2a3fc70f0 msgr2=0x55c2a3fc74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:11.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1d3da1740 1 -- 172.21.15.89:0/1152026069 shutdown_connections 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1d3da1740 1 -- 172.21.15.89:0/1152026069 wait complete. 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.063+0000 7fc1d3da1740 1 -- start start 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1d3da1740 1 Processor -- start 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fd2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fd3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c2a3fd8910 con 0x55c2a3fcb480 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c2a3eeaf10 con 0x55c2a3fd5d40 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c2a3fd8a90 con 0x55c2a3fcab60 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1d3da1740 10 client.0 _ll_register_callbacks cb 0x55c2a3eb3a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1c27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1c27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57004/0 (socket says 172.21.15.89:57004) 2024-09-15T23:29:11.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1c27fc640 1 -- 172.21.15.89:0/999327378 learned_addr learned my addr 172.21.15.89:0/999327378 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1caffd640 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fd2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fd3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1c27fc640 1 -- 172.21.15.89:0/999327378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 msgr2=0x55c2a3fd2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fd2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1c27fc640 1 -- 172.21.15.89:0/999327378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 msgr2=0x55c2a3fd3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fd3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1c27fc640 1 -- 172.21.15.89:0/999327378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2a3ca16a0 con 0x55c2a3fcb480 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.064+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fd3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd3110 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400d740 tx=0x7fc1b409b3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9508115cb53a004c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1aaffd640 1 -- 172.21.15.89:0/999327378 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4020070 con 0x55c2a3fcb480 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1aaffd640 1 -- 172.21.15.89:0/999327378 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc1b4005290 con 0x55c2a3fcb480 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1d3da1740 1 -- 172.21.15.89:0/999327378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1bc01cb00 con 0x55c2a3fcb480 2024-09-15T23:29:11.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1aaffd640 1 -- 172.21.15.89:0/999327378 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc1b409b9b0 con 0x55c2a3fcb480 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1d3da1740 10 client.17232 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1d3da1740 1 -- 172.21.15.89:0/999327378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c2a3fe4460 con 0x55c2a3fcb480 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1aaffd640 1 -- 172.21.15.89:0/999327378 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc01cb00 con 0x55c2a3fcb480 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1aaffd640 1 -- 172.21.15.89:0/999327378 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc1b409cce0 con 0x55c2a3fcb480 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1d3da1740 10 client.17232 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1d3da1740 10 client.17232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1d3da1740 20 client.17232 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2a3fe5870 0x55c2a3fe7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1d3da1740 4 client.17232 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.065+0000 7fc1d3da1740 1 -- 172.21.15.89:0/999327378 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2a3f3d7f0 con 0x55c2a3fe5870 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.066+0000 7fc1caffd640 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2a3fe5870 0x55c2a3fe7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.066+0000 7fc1aaffd640 10 client.17232 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:11.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.066+0000 7fc1caffd640 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2a3fe5870 0x55c2a3fe7c60 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e84fae416d37a3dd server_cookie=f8a68a82190860c4 in_seq=0 out_seq=0 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1aaffd640 1 -- 172.21.15.89:0/999327378 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c2a3f3d7f0 con 0x55c2a3fe5870 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1aaffd640 10 client.17232 handle_command_reply: tid=0 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 client.17232 shutdown 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 -- 172.21.15.89:0/999327378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 msgr2=0x55c2a3fd3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd3110 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400d740 tx=0x7fc1b409b3d0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 -- 172.21.15.89:0/999327378 shutdown_connections 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2a3fe5870 0x55c2a3fe7c60 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fd3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1aaffd640 0 client.17232 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3fd3110 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/999327378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fd2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 -- 172.21.15.89:0/999327378 >> 172.21.15.89:0/999327378 conn(0x55c2a3fc70f0 msgr2=0x55c2a3fccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:11.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 -- 172.21.15.89:0/999327378 shutdown_connections 2024-09-15T23:29:11.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 1 -- 172.21.15.89:0/999327378 wait complete. 2024-09-15T23:29:11.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.067+0000 7fc1d3da1740 20 client.17232 trim_cache size 0 max 16384 2024-09-15T23:29:11.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.084+0000 7fc1d3da1740 1 Processor -- start 2024-09-15T23:29:11.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.084+0000 7fc1d3da1740 1 -- start start 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcab60 0x55c2a3fed530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3feda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fcfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1aee80 con 0x55c2a3fcb480 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1b09c0 con 0x55c2a3fcab60 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1af550 con 0x55c2a3fd5d40 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3feda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3feda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57010/0 (socket says 172.21.15.89:57010) 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c27fc640 1 -- 172.21.15.89:0/1356264472 learned_addr learned my addr 172.21.15.89:0/1356264472 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fcfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1caffd640 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcab60 0x55c2a3fed530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c27fc640 1 -- 172.21.15.89:0/1356264472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fd5d40 msgr2=0x55c2a3fcfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:11.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fcfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c27fc640 1 -- 172.21.15.89:0/1356264472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcab60 msgr2=0x55c2a3fed530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcab60 0x55c2a3fed530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c27fc640 1 -- 172.21.15.89:0/1356264472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2a3fcb2b0 con 0x55c2a3fcb480 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fcfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1caffd640 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcab60 0x55c2a3fed530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3feda70 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4097e10 tx=0x7fc1b4009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421c39fb8dd5bf9c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c17fa640 1 -- 172.21.15.89:0/1356264472 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4020070 con 0x55c2a3fcb480 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c17fa640 1 -- 172.21.15.89:0/1356264472 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1b409fd60 con 0x55c2a3fcb480 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.085+0000 7fc1c17fa640 1 -- 172.21.15.89:0/1356264472 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b409b040 con 0x55c2a3fcb480 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.086+0000 7fc1d3da1740 1 -- 172.21.15.89:0/1356264472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 msgr2=0x55c2a3feda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.086+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3feda70 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4097e10 tx=0x7fc1b4009c30 comp rx=0 tx=0).stop 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.086+0000 7fc1d3da1740 1 -- 172.21.15.89:0/1356264472 shutdown_connections 2024-09-15T23:29:11.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.086+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fcfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.086+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fcb480 0x55c2a3feda70 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.086+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/1356264472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcab60 0x55c2a3fed530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.086+0000 7fc1d3da1740 1 -- 172.21.15.89:0/1356264472 >> 172.21.15.89:0/1356264472 conn(0x55c2a3fc70f0 msgr2=0x55c2a3fd4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.086+0000 7fc1d3da1740 1 -- 172.21.15.89:0/1356264472 shutdown_connections 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.086+0000 7fc1d3da1740 1 -- 172.21.15.89:0/1356264472 wait complete. 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.086+0000 7fc1d3da1740 1 -- start start 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1d3da1740 1 Processor -- start 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fde110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcb480 0x55c2a3fda4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1d3da1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fda9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1b02f0 con 0x55c2a3fd5d40 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1afc20 con 0x55c2a3fcb480 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1d3da1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1ace60 con 0x55c2a3fcab60 2024-09-15T23:29:11.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1d3da1740 10 client.0 _ll_register_callbacks cb 0x7fc1b40056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1cb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fda9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1cb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fda9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57020/0 (socket says 172.21.15.89:57020) 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1cb7fe640 1 -- 172.21.15.89:0/3418002192 learned_addr learned my addr 172.21.15.89:0/3418002192 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1c27fc640 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcb480 0x55c2a3fda4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1caffd640 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fde110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1cb7fe640 1 -- 172.21.15.89:0/3418002192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 msgr2=0x55c2a3fde110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fde110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1cb7fe640 1 -- 172.21.15.89:0/3418002192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcb480 msgr2=0x55c2a3fda4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcb480 0x55c2a3fda4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1cb7fe640 1 -- 172.21.15.89:0/3418002192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2a3fe7fd0 con 0x55c2a3fd5d40 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.087+0000 7fc1caffd640 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fde110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1cb7fe640 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fda9e0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4035000 tx=0x7fc1c4005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca2c6f272e359 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:11.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:11.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1aaffd640 1 -- 172.21.15.89:0/3418002192 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4004740 con 0x55c2a3fd5d40 2024-09-15T23:29:11.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1aaffd640 1 -- 172.21.15.89:0/3418002192 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc1c40305a0 con 0x55c2a3fd5d40 2024-09-15T23:29:11.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1d3da1740 1 -- 172.21.15.89:0/3418002192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1b409fd60 con 0x55c2a3fd5d40 2024-09-15T23:29:11.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1aaffd640 1 -- 172.21.15.89:0/3418002192 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc1c40316e0 con 0x55c2a3fd5d40 2024-09-15T23:29:11.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1d3da1740 10 client.17238 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:11.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1d3da1740 1 -- 172.21.15.89:0/3418002192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c2a4051030 con 0x55c2a3fd5d40 2024-09-15T23:29:11.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1aaffd640 1 -- 172.21.15.89:0/3418002192 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1b409fd60 con 0x55c2a3fd5d40 2024-09-15T23:29:11.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1aaffd640 1 -- 172.21.15.89:0/3418002192 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc1c4030070 con 0x55c2a3fd5d40 2024-09-15T23:29:11.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1d3da1740 10 client.17238 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:11.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1d3da1740 10 client.17238 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1d3da1740 20 client.17238 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2a4051800 0x55c2a4053c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1d3da1740 4 client.17238 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:29:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.088+0000 7fc1d3da1740 1 -- 172.21.15.89:0/3418002192 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c2a3f37d00 con 0x55c2a4051800 2024-09-15T23:29:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.089+0000 7fc1caffd640 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2a4051800 0x55c2a4053c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.089+0000 7fc1caffd640 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2a4051800 0x55c2a4053c20 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6166d13044c5b407 server_cookie=d801c31cd75dab71 in_seq=0 out_seq=0 2024-09-15T23:29:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.089+0000 7fc1aaffd640 10 client.17238 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1aaffd640 1 -- 172.21.15.89:0/3418002192 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c2a3f37d00 con 0x55c2a4051800 2024-09-15T23:29:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1aaffd640 10 client.17238 handle_command_reply: tid=0 2024-09-15T23:29:11.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 client.17238 shutdown 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 -- 172.21.15.89:0/3418002192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fd5d40 msgr2=0x55c2a3fda9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fda9e0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4035000 tx=0x7fc1c4005c50 comp rx=0 tx=0).stop 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 -- 172.21.15.89:0/3418002192 shutdown_connections 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2a4051800 0x55c2a4053c20 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2a3fd5d40 0x55c2a3fda9e0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2a3fcb480 0x55c2a3fda4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1aaffd640 0 client.17238 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 --2- 172.21.15.89:0/3418002192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2a3fcab60 0x55c2a3fde110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 -- 172.21.15.89:0/3418002192 >> 172.21.15.89:0/3418002192 conn(0x55c2a3fc70f0 msgr2=0x55c2a3fcc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 -- 172.21.15.89:0/3418002192 shutdown_connections 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.090+0000 7fc1d3da1740 1 -- 172.21.15.89:0/3418002192 wait complete. 2024-09-15T23:29:11.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d3da1740 20 client.17238 trim_cache size 0 max 16384 2024-09-15T23:29:11.107 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:29:11.107 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:29:11.107 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "1312e2db-8b8c-4da4-ad45-65024bf0fb4a", 2024-09-15T23:29:11.107 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:29:11.107 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 -- 172.21.15.89:0/2559201127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1b80a3240 msgr2=0x7fc1b80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1b80a3240 0x7fc1b80a5700 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4005e30 tx=0x7fc1c403f040 comp rx=0 tx=0).stop 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 -- 172.21.15.89:0/2559201127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc101400 msgr2=0x7fc1cc1997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc101400 0x7fc1cc1997c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400d770 tx=0x7fc1b400dc30 comp rx=0 tx=0).stop 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 -- 172.21.15.89:0/2559201127 shutdown_connections 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1b80a3240 0x7fc1b80a5700 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1cc106a60 0x7fc1cc1a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1cc101400 0x7fc1cc1997c0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 --2- 172.21.15.89:0/2559201127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1cc100a60 0x7fc1cc199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 -- 172.21.15.89:0/2559201127 >> 172.21.15.89:0/2559201127 conn(0x7fc1cc0fc780 msgr2=0x7fc1cc0fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:11.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 -- 172.21.15.89:0/2559201127 shutdown_connections 2024-09-15T23:29:11.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:11.091+0000 7fc1d19c8640 1 -- 172.21.15.89:0/2559201127 wait complete. 2024-09-15T23:29:11.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:11 smithi089 ceph-mon[30933]: pgmap v505: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:29:11.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:11 smithi120 ceph-mon[35250]: pgmap v505: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:29:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:11 smithi174 ceph-mon[35586]: pgmap v505: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:29:11.887 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:29:12.134 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:12.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:12 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:29:12.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:12 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:12.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:12 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:29:12.458 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:12 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:12 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:29:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:12 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:12 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:29:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:12 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:12.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:12 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:29:12.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:12 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:12.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:12 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:29:12.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:12 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:12.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.658+0000 7ff4614f1640 1 -- 172.21.15.89:0/2939178424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff45c101420 msgr2=0x7ff45c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:12.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.658+0000 7ff4614f1640 1 --2- 172.21.15.89:0/2939178424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff45c101420 0x7ff45c10d950 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff454009cd0 tx=0x7ff45402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.658+0000 7ff4614f1640 1 -- 172.21.15.89:0/2939178424 shutdown_connections 2024-09-15T23:29:12.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.658+0000 7ff4614f1640 1 --2- 172.21.15.89:0/2939178424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff45c101420 0x7ff45c10d950 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.658+0000 7ff4614f1640 1 --2- 172.21.15.89:0/2939178424 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff45c100a80 0x7ff45c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.658+0000 7ff4614f1640 1 --2- 172.21.15.89:0/2939178424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff45c106a80 0x7ff45c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.658+0000 7ff4614f1640 1 -- 172.21.15.89:0/2939178424 >> 172.21.15.89:0/2939178424 conn(0x7ff45c0fc7a0 msgr2=0x7ff45c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:12.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.658+0000 7ff4614f1640 1 -- 172.21.15.89:0/2939178424 shutdown_connections 2024-09-15T23:29:12.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.658+0000 7ff4614f1640 1 -- 172.21.15.89:0/2939178424 wait complete. 2024-09-15T23:29:12.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff4614f1640 1 Processor -- start 2024-09-15T23:29:12.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff4614f1640 1 -- start start 2024-09-15T23:29:12.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff4614f1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff45c100a80 0x7ff45c199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff4614f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff45c101420 0x7ff45c1996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff4614f1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff45c106a80 0x7ff45c1a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff4614f1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c110e80 con 0x7ff45c101420 2024-09-15T23:29:12.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff4614f1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c110d00 con 0x7ff45c100a80 2024-09-15T23:29:12.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff4614f1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c111000 con 0x7ff45c106a80 2024-09-15T23:29:12.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff45a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff45c101420 0x7ff45c1996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff45a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff45c101420 0x7ff45c1996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57036/0 (socket says 172.21.15.89:57036) 2024-09-15T23:29:12.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff45a7fc640 1 -- 172.21.15.89:0/2629432257 learned_addr learned my addr 172.21.15.89:0/2629432257 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:12.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff45affd640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff45c100a80 0x7ff45c199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff45c106a80 0x7ff45c1a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff45a7fc640 1 -- 172.21.15.89:0/2629432257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff45c106a80 msgr2=0x7ff45c1a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:12.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff45c106a80 0x7ff45c1a41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff45a7fc640 1 -- 172.21.15.89:0/2629432257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff45c100a80 msgr2=0x7ff45c199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.659+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff45c100a80 0x7ff45c199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.660+0000 7ff45a7fc640 1 -- 172.21.15.89:0/2629432257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45c1a4720 con 0x7ff45c101420 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.660+0000 7ff45affd640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff45c100a80 0x7ff45c199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.660+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff45c106a80 0x7ff45c1a41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.660+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff45c101420 0x7ff45c1996a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00c920 tx=0x7ff44c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.660+0000 7ff443fff640 1 -- 172.21.15.89:0/2629432257 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c007c70 con 0x7ff45c101420 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.660+0000 7ff4614f1640 1 -- 172.21.15.89:0/2629432257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45c1a48f0 con 0x7ff45c101420 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.660+0000 7ff443fff640 1 -- 172.21.15.89:0/2629432257 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff44c007e10 con 0x7ff45c101420 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.660+0000 7ff443fff640 1 -- 172.21.15.89:0/2629432257 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c018890 con 0x7ff45c101420 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.660+0000 7ff4614f1640 1 -- 172.21.15.89:0/2629432257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff45c1a4e80 con 0x7ff45c101420 2024-09-15T23:29:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.662+0000 7ff443fff640 1 -- 172.21.15.89:0/2629432257 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff44c018a70 con 0x7ff45c101420 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.662+0000 7ff4638ca740 1 Processor -- start 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.662+0000 7ff4638ca740 1 -- start start 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94acd6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94ace1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 0x55f94ace3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff4638ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f94ac3c2e0 con 0x55f94acd6dc0 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff4638ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f94abe3ce0 con 0x55f94acd64a0 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff4638ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f94ac382a0 con 0x55f94ace1680 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94ace1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94ace1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57048/0 (socket says 172.21.15.89:57048) 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45affd640 1 -- 172.21.15.89:0/3748314131 learned_addr learned my addr 172.21.15.89:0/3748314131 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 0x55f94ace3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff443fff640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff4440a3240 0x7ff4440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94acd6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45affd640 1 -- 172.21.15.89:0/3748314131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 msgr2=0x55f94ace3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45affd640 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 0x55f94ace3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45affd640 1 -- 172.21.15.89:0/3748314131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 msgr2=0x55f94acd6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45affd640 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94acd6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45affd640 1 -- 172.21.15.89:0/3748314131 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f94abc8440 con 0x55f94acd6dc0 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 0x55f94ace3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94acd6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff4440a3240 0x7ff4440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff443fff640 1 -- 172.21.15.89:0/2629432257 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff44c0cff00 con 0x7ff45c101420 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff45affd640 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94ace1140 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff45000cc20 tx=0x7ff45000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb12187035c023a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.663+0000 7ff4417fa640 1 -- 172.21.15.89:0/3748314131 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff450005220 con 0x55f94acd6dc0 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4417fa640 1 -- 172.21.15.89:0/3748314131 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff450016070 con 0x55f94acd6dc0 2024-09-15T23:29:12.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4417fa640 1 -- 172.21.15.89:0/3748314131 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4500106a0 con 0x55f94acd6dc0 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4638ca740 1 -- 172.21.15.89:0/3748314131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 msgr2=0x55f94ace1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4638ca740 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94ace1140 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff45000cc20 tx=0x7ff45000e720 comp rx=0 tx=0).stop 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff4440a3240 0x7ff4440a5700 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff454006010 tx=0x7ff4540095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4638ca740 1 -- 172.21.15.89:0/3748314131 shutdown_connections 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4638ca740 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 0x55f94ace3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4638ca740 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94ace1140 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4638ca740 1 --2- 172.21.15.89:0/3748314131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94acd6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4638ca740 1 -- 172.21.15.89:0/3748314131 >> 172.21.15.89:0/3748314131 conn(0x55f94acd2a30 msgr2=0x55f94acd2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4638ca740 1 -- 172.21.15.89:0/3748314131 shutdown_connections 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.664+0000 7ff4638ca740 1 -- 172.21.15.89:0/3748314131 wait complete. 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff4638ca740 1 -- start start 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff4638ca740 1 Processor -- start 2024-09-15T23:29:12.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94ace0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94acde1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 0x55f94acde720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff4638ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f94ace4250 con 0x55f94acd6dc0 2024-09-15T23:29:12.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff4638ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f94abf7050 con 0x55f94acd64a0 2024-09-15T23:29:12.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff4638ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f94ace43d0 con 0x55f94ace1680 2024-09-15T23:29:12.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff4638ca740 10 client.0 _ll_register_callbacks cb 0x55f94ac37160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:12.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94acde1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94acde1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57060/0 (socket says 172.21.15.89:57060) 2024-09-15T23:29:12.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45a7fc640 1 -- 172.21.15.89:0/2857855008 learned_addr learned my addr 172.21.15.89:0/2857855008 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45affd640 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94ace0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45a7fc640 1 -- 172.21.15.89:0/2857855008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 msgr2=0x55f94acde720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 0x55f94acde720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 0x55f94acde720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45a7fc640 1 -- 172.21.15.89:0/2857855008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 msgr2=0x55f94ace0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94ace0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45a7fc640 1 -- 172.21.15.89:0/2857855008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f94a9ad3c0 con 0x55f94acd6dc0 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45affd640 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94ace0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 0x55f94acde720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94acde1e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff44c098590 tx=0x7ff44c09bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbf47da071109c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.665+0000 7ff43eff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:12.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff43eff5640 1 -- 172.21.15.89:0/2857855008 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c00de20 con 0x55f94acd6dc0 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff43eff5640 1 -- 172.21.15.89:0/2857855008 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff44c018d60 con 0x55f94acd6dc0 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff4638ca740 1 -- 172.21.15.89:0/2857855008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff450016070 con 0x55f94acd6dc0 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff43eff5640 1 -- 172.21.15.89:0/2857855008 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff44c09c130 con 0x55f94acd6dc0 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff4638ca740 10 client.17250 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff4638ca740 1 -- 172.21.15.89:0/2857855008 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f94acefc00 con 0x55f94acd6dc0 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff43eff5640 1 -- 172.21.15.89:0/2857855008 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff450016070 con 0x55f94acd6dc0 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff43eff5640 1 -- 172.21.15.89:0/2857855008 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff44c09dce0 con 0x55f94acd6dc0 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff4638ca740 10 client.17250 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff4638ca740 10 client.17250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff4638ca740 20 client.17250 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f94acf10c0 0x55f94acf34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff4638ca740 4 client.17250 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.666+0000 7ff4638ca740 1 -- 172.21.15.89:0/2857855008 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f94ac49850 con 0x55f94acf10c0 2024-09-15T23:29:12.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.667+0000 7ff45affd640 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f94acf10c0 0x55f94acf34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.667+0000 7ff43eff5640 10 client.17250 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.667+0000 7ff45affd640 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f94acf10c0 0x55f94acf34b0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6863113d5e772799 server_cookie=a91e885d029dbc8c in_seq=0 out_seq=0 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff43eff5640 1 -- 172.21.15.89:0/2857855008 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f94ac49850 con 0x55f94acf10c0 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff43eff5640 10 client.17250 handle_command_reply: tid=0 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 client.17250 shutdown 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 -- 172.21.15.89:0/2857855008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 msgr2=0x55f94acde1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94acde1e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff44c098590 tx=0x7ff44c09bb50 comp rx=0 tx=0).stop 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 -- 172.21.15.89:0/2857855008 shutdown_connections 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f94acf10c0 0x55f94acf34b0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94ace1680 0x55f94acde720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd6dc0 0x55f94acde1e0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2857855008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94ace0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 -- 172.21.15.89:0/2857855008 >> 172.21.15.89:0/2857855008 conn(0x55f94acd2a30 msgr2=0x55f94acd8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:12.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff43eff5640 0 client.17250 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:12.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 -- 172.21.15.89:0/2857855008 shutdown_connections 2024-09-15T23:29:12.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 1 -- 172.21.15.89:0/2857855008 wait complete. 2024-09-15T23:29:12.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.668+0000 7ff4638ca740 20 client.17250 trim_cache size 0 max 16384 2024-09-15T23:29:12.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.685+0000 7ff4638ca740 1 Processor -- start 2024-09-15T23:29:12.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.685+0000 7ff4638ca740 1 -- start start 2024-09-15T23:29:12.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.685+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94aced4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.685+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94acd6dc0 0x55f94acdcab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.685+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94ace1680 0x55f94acd97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.685+0000 7ff4638ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c1aed10 con 0x55f94ace1680 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.685+0000 7ff4638ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c1b0850 con 0x55f94acd64a0 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.685+0000 7ff4638ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c1af3e0 con 0x55f94acd6dc0 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.685+0000 7ff45b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94ace1680 0x55f94acd97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff45b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94ace1680 0x55f94acd97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57076/0 (socket says 172.21.15.89:57076) 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff45b7fe640 1 -- 172.21.15.89:0/1733711026 learned_addr learned my addr 172.21.15.89:0/1733711026 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/1733711026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94acd6dc0 0x55f94acdcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff45affd640 1 --2- 172.21.15.89:0/1733711026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94aced4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff45b7fe640 1 -- 172.21.15.89:0/1733711026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94acd6dc0 msgr2=0x55f94acdcab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:12.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/1733711026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94acd6dc0 0x55f94acdcab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff45b7fe640 1 -- 172.21.15.89:0/1733711026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 msgr2=0x55f94aced4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:12.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/1733711026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94aced4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff45b7fe640 1 -- 172.21.15.89:0/1733711026 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f94acdd190 con 0x55f94ace1680 2024-09-15T23:29:12.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/1733711026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94ace1680 0x55f94acd97f0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff454031c90 tx=0x7ff45403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f88e7e2cc50620 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:12.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff440ff9640 1 -- 172.21.15.89:0/1733711026 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff454004240 con 0x55f94ace1680 2024-09-15T23:29:12.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff440ff9640 1 -- 172.21.15.89:0/1733711026 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff454036040 con 0x55f94ace1680 2024-09-15T23:29:12.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.686+0000 7ff440ff9640 1 -- 172.21.15.89:0/1733711026 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff454030e60 con 0x55f94ace1680 2024-09-15T23:29:12.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 -- 172.21.15.89:0/1733711026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94ace1680 msgr2=0x55f94acd97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:12.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 --2- 172.21.15.89:0/1733711026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94ace1680 0x55f94acd97f0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff454031c90 tx=0x7ff45403f000 comp rx=0 tx=0).stop 2024-09-15T23:29:12.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 -- 172.21.15.89:0/1733711026 shutdown_connections 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 --2- 172.21.15.89:0/1733711026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94ace1680 0x55f94acd97f0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 --2- 172.21.15.89:0/1733711026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94acd6dc0 0x55f94acdcab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 --2- 172.21.15.89:0/1733711026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94acd64a0 0x55f94aced4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 -- 172.21.15.89:0/1733711026 >> 172.21.15.89:0/1733711026 conn(0x55f94acd2a30 msgr2=0x55f94acd90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 -- 172.21.15.89:0/1733711026 shutdown_connections 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 -- 172.21.15.89:0/1733711026 wait complete. 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 -- start start 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.687+0000 7ff4638ca740 1 Processor -- start 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd64a0 0x55f94ace9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94acd6dc0 0x55f94ace5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff4638ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94ace1680 0x55f94ace5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff4638ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c1b0180 con 0x55f94acd64a0 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff4638ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c1afab0 con 0x55f94ace1680 2024-09-15T23:29:12.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff4638ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c1accf0 con 0x55f94acd6dc0 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff4638ca740 10 client.0 _ll_register_callbacks cb 0x7ff44c00c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd64a0 0x55f94ace9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd64a0 0x55f94ace9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:57088/0 (socket says 172.21.15.89:57088) 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45affd640 1 -- 172.21.15.89:0/2372055409 learned_addr learned my addr 172.21.15.89:0/2372055409 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45b7fe640 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94ace1680 0x55f94ace5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94acd6dc0 0x55f94ace5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45affd640 1 -- 172.21.15.89:0/2372055409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94acd6dc0 msgr2=0x55f94ace5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45affd640 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94acd6dc0 0x55f94ace5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45affd640 1 -- 172.21.15.89:0/2372055409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94ace1680 msgr2=0x55f94ace5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45affd640 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94ace1680 0x55f94ace5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45affd640 1 -- 172.21.15.89:0/2372055409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f94acd6bf0 con 0x55f94acd64a0 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff45affd640 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd64a0 0x55f94ace9d00 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff45000cc20 tx=0x7ff450005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c125f41e4181a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:12.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff43eff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff43eff5640 1 -- 172.21.15.89:0/2372055409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4500054d0 con 0x55f94acd64a0 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff43eff5640 1 -- 172.21.15.89:0/2372055409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff450004020 con 0x55f94acd64a0 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.688+0000 7ff4638ca740 1 -- 172.21.15.89:0/2372055409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff454036040 con 0x55f94acd64a0 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff43eff5640 1 -- 172.21.15.89:0/2372055409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff450037050 con 0x55f94acd64a0 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff4638ca740 10 client.17256 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff4638ca740 1 -- 172.21.15.89:0/2372055409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f94ad5bae0 con 0x55f94acd64a0 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff43eff5640 1 -- 172.21.15.89:0/2372055409 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff454036040 con 0x55f94acd64a0 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff43eff5640 1 -- 172.21.15.89:0/2372055409 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff4500035f0 con 0x55f94acd64a0 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff4638ca740 10 client.17256 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff4638ca740 10 client.17256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff4638ca740 20 client.17256 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f94ad5bf30 0x55f94ad5e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:12.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff4638ca740 4 client.17256 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff4638ca740 1 -- 172.21.15.89:0/2372055409 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f94ac43d60 con 0x55f94ad5bf30 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.689+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f94ad5bf30 0x55f94ad5e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff45a7fc640 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f94ad5bf30 0x55f94ad5e320 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23c00368173fdc08 server_cookie=3ba35a87d48c83c9 in_seq=0 out_seq=0 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff43eff5640 10 client.17256 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff43eff5640 1 -- 172.21.15.89:0/2372055409 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f94ac43d60 con 0x55f94ad5bf30 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff43eff5640 10 client.17256 handle_command_reply: tid=0 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 client.17256 shutdown 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 -- 172.21.15.89:0/2372055409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd64a0 msgr2=0x55f94ace9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd64a0 0x55f94ace9d00 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ff45000cc20 tx=0x7ff450005710 comp rx=0 tx=0).stop 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 -- 172.21.15.89:0/2372055409 shutdown_connections 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f94ad5bf30 0x55f94ad5e320 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f94ace1680 0x55f94ace5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f94acd6dc0 0x55f94ace5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 --2- 172.21.15.89:0/2372055409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f94acd64a0 0x55f94ace9d00 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 -- 172.21.15.89:0/2372055409 >> 172.21.15.89:0/2372055409 conn(0x55f94acd2a30 msgr2=0x55f94acdfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:12.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff43eff5640 0 client.17256 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:12.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 -- 172.21.15.89:0/2372055409 shutdown_connections 2024-09-15T23:29:12.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 1 -- 172.21.15.89:0/2372055409 wait complete. 2024-09-15T23:29:12.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.690+0000 7ff4638ca740 20 client.17256 trim_cache size 0 max 16384 2024-09-15T23:29:12.698 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:29:12.698 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:29:12.698 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:29:12.698 INFO:teuthology.orchestra.run.smithi089.stdout: "1312e2db-8b8c-4da4-ad45-65024bf0fb4a": { 2024-09-15T23:29:12.698 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:29:12.698 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "1312e2db-8b8c-4da4-ad45-65024bf0fb4a", 2024-09-15T23:29:12.699 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:29:12.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:12.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:12.699 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:29:12.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 -- 172.21.15.89:0/2629432257 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff4440a3240 msgr2=0x7ff4440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:12.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff4440a3240 0x7ff4440a5700 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff454006010 tx=0x7ff4540095c0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 -- 172.21.15.89:0/2629432257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff45c101420 msgr2=0x7ff45c1996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:12.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff45c101420 0x7ff45c1996a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00c920 tx=0x7ff44c00cde0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 -- 172.21.15.89:0/2629432257 shutdown_connections 2024-09-15T23:29:12.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff4440a3240 0x7ff4440a5700 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff45c106a80 0x7ff45c1a41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff45c101420 0x7ff45c1996a0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 --2- 172.21.15.89:0/2629432257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff45c100a80 0x7ff45c199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:12.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 -- 172.21.15.89:0/2629432257 >> 172.21.15.89:0/2629432257 conn(0x7ff45c0fc7a0 msgr2=0x7ff45c0fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:12.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 -- 172.21.15.89:0/2629432257 shutdown_connections 2024-09-15T23:29:12.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:12.691+0000 7ff4614f1640 1 -- 172.21.15.89:0/2629432257 wait complete. 2024-09-15T23:29:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:13 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:29:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:13 smithi120 ceph-mon[35250]: pgmap v506: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:29:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:13 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:29:13.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:13 smithi174 ceph-mon[35586]: pgmap v506: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:29:13.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:13 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:29:13.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:13 smithi089 ceph-mon[30933]: pgmap v506: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:29:13.630 INFO:tasks.cephfs.filesystem:scrub status for tag:1312e2db-8b8c-4da4-ad45-65024bf0fb4a - {'path': '/', 'tag': '1312e2db-8b8c-4da4-ad45-65024bf0fb4a', 'options': 'recursive,force'} 2024-09-15T23:29:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:15 smithi120 ceph-mon[35250]: pgmap v507: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:29:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:15 smithi174 ceph-mon[35586]: pgmap v507: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:29:15.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:15 smithi089 ceph-mon[30933]: pgmap v507: 129 pgs: 129 active+clean; 2.6 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:29:17.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.064+0000 7f777daba640 1 -- 172.21.15.89:0/2379698598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7778102600 msgr2=0x7f7778108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.064+0000 7f777daba640 1 --2- 172.21.15.89:0/2379698598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7778102600 0x7f7778108e90 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f7770009ca0 tx=0x7f777002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.064+0000 7f777daba640 1 -- 172.21.15.89:0/2379698598 shutdown_connections 2024-09-15T23:29:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.064+0000 7f777daba640 1 --2- 172.21.15.89:0/2379698598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7778102600 0x7f7778108e90 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.064+0000 7f777daba640 1 --2- 172.21.15.89:0/2379698598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7778101c40 0x7f77781020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.064+0000 7f777daba640 1 --2- 172.21.15.89:0/2379698598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7778100a50 0x7f7778100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.064+0000 7f777daba640 1 -- 172.21.15.89:0/2379698598 >> 172.21.15.89:0/2379698598 conn(0x7f77780fc1b0 msgr2=0x7f77780fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.064+0000 7f777daba640 1 -- 172.21.15.89:0/2379698598 shutdown_connections 2024-09-15T23:29:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.064+0000 7f777daba640 1 -- 172.21.15.89:0/2379698598 wait complete. 2024-09-15T23:29:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f777daba640 1 Processor -- start 2024-09-15T23:29:17.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f777daba640 1 -- start start 2024-09-15T23:29:17.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f777daba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7778100a50 0x7f7778197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f777daba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7778101c40 0x7f7778198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f777daba640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7778102600 0x7f777819f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f777daba640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f777810d820 con 0x7f7778102600 2024-09-15T23:29:17.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f777daba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f777810d6a0 con 0x7f7778101c40 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f777daba640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f777810d9a0 con 0x7f7778100a50 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f7777fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7778102600 0x7f777819f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f7777fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7778102600 0x7f777819f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50594/0 (socket says 172.21.15.89:50594) 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f7777fff640 1 -- 172.21.15.89:0/2867377979 learned_addr learned my addr 172.21.15.89:0/2867377979 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.065+0000 7f7776ffd640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7778101c40 0x7f7778198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f77777fe640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7778100a50 0x7f7778197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f7777fff640 1 -- 172.21.15.89:0/2867377979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7778100a50 msgr2=0x7f7778197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f7777fff640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7778100a50 0x7f7778197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f7777fff640 1 -- 172.21.15.89:0/2867377979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7778101c40 msgr2=0x7f7778198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f7777fff640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7778101c40 0x7f7778198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f7777fff640 1 -- 172.21.15.89:0/2867377979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777819f970 con 0x7f7778102600 2024-09-15T23:29:17.069 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f7777fff640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7778102600 0x7f777819f300 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f7770005e30 tx=0x7f7770031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:17.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f7774ff9640 1 -- 172.21.15.89:0/2867377979 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7770003c30 con 0x7f7778102600 2024-09-15T23:29:17.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f777daba640 1 -- 172.21.15.89:0/2867377979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777819fc00 con 0x7f7778102600 2024-09-15T23:29:17.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f7774ff9640 1 -- 172.21.15.89:0/2867377979 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7770003dd0 con 0x7f7778102600 2024-09-15T23:29:17.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f7774ff9640 1 -- 172.21.15.89:0/2867377979 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777002fde0 con 0x7f7778102600 2024-09-15T23:29:17.070 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.066+0000 7f777daba640 1 -- 172.21.15.89:0/2867377979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77781a0150 con 0x7f7778102600 2024-09-15T23:29:17.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f7774ff9640 1 -- 172.21.15.89:0/2867377979 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7770004050 con 0x7f7778102600 2024-09-15T23:29:17.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f778085b740 1 Processor -- start 2024-09-15T23:29:17.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f778085b740 1 -- start start 2024-09-15T23:29:17.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f778085b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010af6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f778085b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010afbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f778085b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010afe830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f778085b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560010a5a2f0 con 0x560010af69a0 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f778085b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560010a02270 con 0x560010afc370 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f778085b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560010a562b0 con 0x560010af72e0 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f7774ff9640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f77600a32f0 0x7f77600a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f7776ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010af6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f7776ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010af6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50598/0 (socket says 172.21.15.89:50598) 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f7776ffd640 1 -- 172.21.15.89:0/168698955 learned_addr learned my addr 172.21.15.89:0/168698955 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.069+0000 7f7777fff640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f77600a32f0 0x7f77600a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f77777fe640 1 --2- 172.21.15.89:0/168698955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010afbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f7774ff9640 1 -- 172.21.15.89:0/2867377979 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f77700f0f40 con 0x7f7778102600 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f7776ffd640 1 --2- 172.21.15.89:0/168698955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010afe830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f7776ffd640 1 -- 172.21.15.89:0/168698955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 msgr2=0x560010afbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f7776ffd640 1 --2- 172.21.15.89:0/168698955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010afbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f7776ffd640 1 -- 172.21.15.89:0/168698955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 msgr2=0x560010afe830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f7776ffd640 1 --2- 172.21.15.89:0/168698955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010afe830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f7776ffd640 1 -- 172.21.15.89:0/168698955 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560010a01340 con 0x560010af69a0 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f77777fe640 1 --2- 172.21.15.89:0/168698955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010afbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f7776ffd640 1 --2- 172.21.15.89:0/168698955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010af6da0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f776c00b8f0 tx=0x7f776c004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d755dce92fa783 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f7777fff640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f77600a32f0 0x7f77600a57b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f77700ee6a0 tx=0x7f777008fff0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f775dffb640 1 -- 172.21.15.89:0/168698955 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c003d70 con 0x560010af69a0 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f775dffb640 1 -- 172.21.15.89:0/168698955 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776c005500 con 0x560010af69a0 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.070+0000 7f775dffb640 1 -- 172.21.15.89:0/168698955 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c0106c0 con 0x560010af69a0 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 -- 172.21.15.89:0/168698955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 msgr2=0x560010af6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 --2- 172.21.15.89:0/168698955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010af6da0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f776c00b8f0 tx=0x7f776c004250 comp rx=0 tx=0).stop 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 -- 172.21.15.89:0/168698955 shutdown_connections 2024-09-15T23:29:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 --2- 172.21.15.89:0/168698955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010afe830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 --2- 172.21.15.89:0/168698955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010afbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 --2- 172.21.15.89:0/168698955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010af6da0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 -- 172.21.15.89:0/168698955 >> 172.21.15.89:0/168698955 conn(0x560010af2010 msgr2=0x560010af2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 -- 172.21.15.89:0/168698955 shutdown_connections 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 -- 172.21.15.89:0/168698955 wait complete. 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 -- start start 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.071+0000 7f778085b740 1 Processor -- start 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f778085b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b00bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f778085b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010b01130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f778085b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b08170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f778085b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560010aff340 con 0x560010af69a0 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f778085b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5600109e3290 con 0x560010afc370 2024-09-15T23:29:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f778085b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560010aff4c0 con 0x560010af72e0 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f778085b740 10 client.0 _ll_register_callbacks cb 0x560010a14140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7776ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b00bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7776ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b00bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50606/0 (socket says 172.21.15.89:50606) 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7776ffd640 1 -- 172.21.15.89:0/1435090109 learned_addr learned my addr 172.21.15.89:0/1435090109 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f77777fe640 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010b01130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7776ffd640 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b08170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7776ffd640 1 -- 172.21.15.89:0/1435090109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 msgr2=0x560010b01130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7776ffd640 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010b01130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7776ffd640 1 -- 172.21.15.89:0/1435090109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 msgr2=0x560010b08170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7776ffd640 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b08170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7776ffd640 1 -- 172.21.15.89:0/1435090109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56001076d210 con 0x560010af69a0 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7776ffd640 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b00bf0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f776c006010 tx=0x7f776c00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6738552aa9df0fe8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7756ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:17.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7756ffd640 1 -- 172.21.15.89:0/1435090109 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c021020 con 0x560010af69a0 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f7756ffd640 1 -- 172.21.15.89:0/1435090109 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f776c005000 con 0x560010af69a0 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.072+0000 7f778085b740 1 -- 172.21.15.89:0/1435090109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f776c005500 con 0x560010af69a0 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f7756ffd640 1 -- 172.21.15.89:0/1435090109 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f776c031050 con 0x560010af69a0 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f778085b740 10 client.17268 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f778085b740 1 -- 172.21.15.89:0/1435090109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560010839620 con 0x560010af69a0 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f7756ffd640 1 -- 172.21.15.89:0/1435090109 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776c005500 con 0x560010af69a0 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f7756ffd640 1 -- 172.21.15.89:0/1435090109 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f776c01c450 con 0x560010af69a0 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f778085b740 10 client.17268 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f778085b740 10 client.17268 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f778085b740 20 client.17268 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f778085b740 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560010b0c6d0 0x560010b0eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f778085b740 4 client.17268 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f778085b740 1 -- 172.21.15.89:0/1435090109 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560010a67a20 con 0x560010b0c6d0 2024-09-15T23:29:17.081 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.073+0000 7f77777fe640 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560010b0c6d0 0x560010b0eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.074+0000 7f77777fe640 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560010b0c6d0 0x560010b0eac0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f48de6e94e4e3e5 server_cookie=d34ae2156559c3f0 in_seq=0 out_seq=0 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.074+0000 7f7756ffd640 10 client.17268 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.074+0000 7f7756ffd640 1 -- 172.21.15.89:0/1435090109 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560010a67a20 con 0x560010b0c6d0 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.074+0000 7f7756ffd640 10 client.17268 handle_command_reply: tid=0 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.074+0000 7f778085b740 1 client.17268 shutdown 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 1 -- 172.21.15.89:0/1435090109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 msgr2=0x560010b00bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b00bf0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f776c006010 tx=0x7f776c00bf90 comp rx=0 tx=0).stop 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 1 -- 172.21.15.89:0/1435090109 shutdown_connections 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560010b0c6d0 0x560010b0eac0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b08170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f7756ffd640 0 client.17268 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010b01130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.082 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 1 --2- 172.21.15.89:0/1435090109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b00bf0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 1 -- 172.21.15.89:0/1435090109 >> 172.21.15.89:0/1435090109 conn(0x560010af2010 msgr2=0x560010af3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 1 -- 172.21.15.89:0/1435090109 shutdown_connections 2024-09-15T23:29:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 1 -- 172.21.15.89:0/1435090109 wait complete. 2024-09-15T23:29:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.075+0000 7f778085b740 20 client.17268 trim_cache size 0 max 16384 2024-09-15T23:29:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.089+0000 7f778085b740 1 Processor -- start 2024-09-15T23:29:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.089+0000 7f778085b740 1 -- start start 2024-09-15T23:29:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f778085b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af69a0 0x560010b08be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f778085b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af72e0 0x560010b010b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f778085b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b08100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f778085b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f77781aa7d0 con 0x560010af72e0 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f778085b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f77781ac310 con 0x560010afc370 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f778085b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f77781aaea0 con 0x560010af69a0 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f77777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af72e0 0x560010b010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f77777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af72e0 0x560010b010b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50616/0 (socket says 172.21.15.89:50616) 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f77777fe640 1 -- 172.21.15.89:0/3985236849 learned_addr learned my addr 172.21.15.89:0/3985236849 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f7776ffd640 1 --2- 172.21.15.89:0/3985236849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af69a0 0x560010b08be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f77777fe640 1 -- 172.21.15.89:0/3985236849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af69a0 msgr2=0x560010b08be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f77777fe640 1 --2- 172.21.15.89:0/3985236849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af69a0 0x560010b08be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f77777fe640 1 -- 172.21.15.89:0/3985236849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 msgr2=0x560010b08100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f77777fe640 1 --2- 172.21.15.89:0/3985236849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b08100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f77777fe640 1 -- 172.21.15.89:0/3985236849 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560010b0f140 con 0x560010af72e0 2024-09-15T23:29:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f77777fe640 1 --2- 172.21.15.89:0/3985236849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af72e0 0x560010b010b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x560010af7d10 tx=0x7f776800b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0ee9dce7fd0686 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f775d7fa640 1 -- 172.21.15.89:0/3985236849 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77680140d0 con 0x560010af72e0 2024-09-15T23:29:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f775d7fa640 1 -- 172.21.15.89:0/3985236849 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7768014270 con 0x560010af72e0 2024-09-15T23:29:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.090+0000 7f775d7fa640 1 -- 172.21.15.89:0/3985236849 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768026810 con 0x560010af72e0 2024-09-15T23:29:17.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.091+0000 7f778085b740 1 -- 172.21.15.89:0/3985236849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af72e0 msgr2=0x560010b010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:17.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.091+0000 7f778085b740 1 --2- 172.21.15.89:0/3985236849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af72e0 0x560010b010b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x560010af7d10 tx=0x7f776800b7f0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.091+0000 7f778085b740 1 -- 172.21.15.89:0/3985236849 shutdown_connections 2024-09-15T23:29:17.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.091+0000 7f778085b740 1 --2- 172.21.15.89:0/3985236849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b08100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.091+0000 7f778085b740 1 --2- 172.21.15.89:0/3985236849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af72e0 0x560010b010b0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.091+0000 7f778085b740 1 --2- 172.21.15.89:0/3985236849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af69a0 0x560010b08be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.091+0000 7f778085b740 1 -- 172.21.15.89:0/3985236849 >> 172.21.15.89:0/3985236849 conn(0x560010af2010 msgr2=0x560010b009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:17.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.091+0000 7f778085b740 1 -- 172.21.15.89:0/3985236849 shutdown_connections 2024-09-15T23:29:17.098 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.091+0000 7f778085b740 1 -- 172.21.15.89:0/3985236849 wait complete. 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.091+0000 7f778085b740 1 -- start start 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f778085b740 1 Processor -- start 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f778085b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b05470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f778085b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010b059b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f778085b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b06920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f778085b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f77781abc40 con 0x560010af69a0 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f778085b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f77781ab570 con 0x560010afc370 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f778085b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f77781a87b0 con 0x560010af72e0 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f778085b740 10 client.0 _ll_register_callbacks cb 0x7f776c00b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7776ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b05470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7776ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b05470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50632/0 (socket says 172.21.15.89:50632) 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7776ffd640 1 -- 172.21.15.89:0/2769701356 learned_addr learned my addr 172.21.15.89:0/2769701356 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:17.099 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f77777fe640 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010b059b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7776ffd640 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b06920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7776ffd640 1 -- 172.21.15.89:0/2769701356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 msgr2=0x560010b059b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7776ffd640 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010b059b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7776ffd640 1 -- 172.21.15.89:0/2769701356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 msgr2=0x560010b06920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7776ffd640 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b06920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7776ffd640 1 -- 172.21.15.89:0/2769701356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560010b00730 con 0x560010af69a0 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f77777fe640 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010b059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7776ffd640 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b05470 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f776c001eb0 tx=0x7f776c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1279140ecc43cdf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7756ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.092+0000 7f7756ffd640 1 -- 172.21.15.89:0/2769701356 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c03d020 con 0x560010af69a0 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f7756ffd640 1 -- 172.21.15.89:0/2769701356 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f776c004b30 con 0x560010af69a0 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f778085b740 1 -- 172.21.15.89:0/2769701356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7768014270 con 0x560010af69a0 2024-09-15T23:29:17.100 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f7756ffd640 1 -- 172.21.15.89:0/2769701356 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f776c045050 con 0x560010af69a0 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f778085b740 10 client.17274 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f778085b740 1 -- 172.21.15.89:0/2769701356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560010b080a0 con 0x560010af69a0 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f7756ffd640 1 -- 172.21.15.89:0/2769701356 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7768014270 con 0x560010af69a0 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f7756ffd640 1 -- 172.21.15.89:0/2769701356 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f776c0054b0 con 0x560010af69a0 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f778085b740 10 client.17274 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f778085b740 10 client.17274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f778085b740 20 client.17274 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f778085b740 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560010b777e0 0x560010b79c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f778085b740 4 client.17274 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f778085b740 1 -- 172.21.15.89:0/2769701356 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560010a61d70 con 0x560010b777e0 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.093+0000 7f77777fe640 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560010b777e0 0x560010b79c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.094+0000 7f7756ffd640 10 client.17274 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.094+0000 7f77777fe640 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560010b777e0 0x560010b79c00 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df68d4c8b933d2c1 server_cookie=dafc5266eb1df7e3 in_seq=0 out_seq=0 2024-09-15T23:29:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f7756ffd640 1 -- 172.21.15.89:0/2769701356 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560010a61d70 con 0x560010b777e0 2024-09-15T23:29:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f7756ffd640 10 client.17274 handle_command_reply: tid=0 2024-09-15T23:29:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 client.17274 shutdown 2024-09-15T23:29:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 -- 172.21.15.89:0/2769701356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 msgr2=0x560010b05470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b05470 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f776c001eb0 tx=0x7f776c00b390 comp rx=0 tx=0).stop 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 -- 172.21.15.89:0/2769701356 shutdown_connections 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560010b777e0 0x560010b79c00 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560010afc370 0x560010b06920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f7756ffd640 0 client.17274 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560010af72e0 0x560010b059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 --2- 172.21.15.89:0/2769701356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560010af69a0 0x560010b05470 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 -- 172.21.15.89:0/2769701356 >> 172.21.15.89:0/2769701356 conn(0x560010af2010 msgr2=0x560010af09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 -- 172.21.15.89:0/2769701356 shutdown_connections 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 1 -- 172.21.15.89:0/2769701356 wait complete. 2024-09-15T23:29:17.111 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.109+0000 7f778085b740 20 client.17274 trim_cache size 0 max 16384 2024-09-15T23:29:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:29:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:29:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:29:17.112 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 -- 172.21.15.89:0/2867377979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f77600a32f0 msgr2=0x7f77600a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f77600a32f0 0x7f77600a57b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f77700ee6a0 tx=0x7f777008fff0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 -- 172.21.15.89:0/2867377979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7778102600 msgr2=0x7f777819f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7778102600 0x7f777819f300 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f7770005e30 tx=0x7f7770031ee0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 -- 172.21.15.89:0/2867377979 shutdown_connections 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f77600a32f0 0x7f77600a57b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7778102600 0x7f777819f300 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7778101c40 0x7f7778198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 --2- 172.21.15.89:0/2867377979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7778100a50 0x7f7778197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 -- 172.21.15.89:0/2867377979 >> 172.21.15.89:0/2867377979 conn(0x7f77780fc1b0 msgr2=0x7f7778104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 -- 172.21.15.89:0/2867377979 shutdown_connections 2024-09-15T23:29:17.114 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:17.110+0000 7f777daba640 1 -- 172.21.15.89:0/2867377979 wait complete. 2024-09-15T23:29:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:17 smithi120 ceph-mon[35250]: pgmap v508: 129 pgs: 129 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 22 op/s 2024-09-15T23:29:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:17 smithi174 ceph-mon[35586]: pgmap v508: 129 pgs: 129 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 22 op/s 2024-09-15T23:29:17.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:17 smithi089 ceph-mon[30933]: pgmap v508: 129 pgs: 129 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 22 op/s 2024-09-15T23:29:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:19 smithi120 ceph-mon[35250]: pgmap v509: 129 pgs: 129 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:29:19.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:19 smithi174 ceph-mon[35586]: pgmap v509: 129 pgs: 129 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:29:19.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:19 smithi089 ceph-mon[30933]: pgmap v509: 129 pgs: 129 active+clean; 2.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:29:20.091 INFO:tasks.quiescer.fs.[cephfs]:Total duration (50.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T23:29:20.091 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.35 2024-09-15T23:29:20.091 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:29:20.353 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:20.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 -- 172.21.15.89:0/3210540700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbed8102d30 msgr2=0x7fbed8103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 --2- 172.21.15.89:0/3210540700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbed8102d30 0x7fbed8103110 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009cd0 tx=0x7fbec802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 -- 172.21.15.89:0/3210540700 shutdown_connections 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 --2- 172.21.15.89:0/3210540700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbed81082a0 0x7fbed810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 --2- 172.21.15.89:0/3210540700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbed81036e0 0x7fbed8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 --2- 172.21.15.89:0/3210540700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbed8102d30 0x7fbed8103110 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 -- 172.21.15.89:0/3210540700 >> 172.21.15.89:0/3210540700 conn(0x7fbed80fc7d0 msgr2=0x7fbed80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 -- 172.21.15.89:0/3210540700 shutdown_connections 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 -- 172.21.15.89:0/3210540700 wait complete. 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 Processor -- start 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.811+0000 7fbedfedf640 1 -- start start 2024-09-15T23:29:20.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbedfedf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbed8102d30 0x7fbed8100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbedfedf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbed81036e0 0x7fbed8100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbedfedf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbed81082a0 0x7fbed8101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbedfedf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbed81120a0 con 0x7fbed8102d30 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbedfedf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbed8111f20 con 0x7fbed81082a0 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbedfedf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbed8112220 con 0x7fbed81036e0 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbeddc7b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbed8102d30 0x7fbed8100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbeddc7b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbed8102d30 0x7fbed8100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50650/0 (socket says 172.21.15.89:50650) 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbeddc7b640 1 -- 172.21.15.89:0/1307734427 learned_addr learned my addr 172.21.15.89:0/1307734427 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbede47c640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbed81082a0 0x7fbed8101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbedd47a640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbed81036e0 0x7fbed8100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbeddc7b640 1 -- 172.21.15.89:0/1307734427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbed81036e0 msgr2=0x7fbed8100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbeddc7b640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbed81036e0 0x7fbed8100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:20.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbeddc7b640 1 -- 172.21.15.89:0/1307734427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbed81082a0 msgr2=0x7fbed8101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbeddc7b640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbed81082a0 0x7fbed8101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbeddc7b640 1 -- 172.21.15.89:0/1307734427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed81a69e0 con 0x7fbed8102d30 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbedd47a640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbed81036e0 0x7fbed8100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbeddc7b640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbed8102d30 0x7fbed8100990 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fbec802ae00 tx=0x7fbec8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbed6ffd640 1 -- 172.21.15.89:0/1307734427 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8004040 con 0x7fbed8102d30 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbedfedf640 1 -- 172.21.15.89:0/1307734427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed81a6c70 con 0x7fbed8102d30 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbed6ffd640 1 -- 172.21.15.89:0/1307734427 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbec80046c0 con 0x7fbed8102d30 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.812+0000 7fbed6ffd640 1 -- 172.21.15.89:0/1307734427 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8046480 con 0x7fbed8102d30 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.813+0000 7fbedfedf640 1 -- 172.21.15.89:0/1307734427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed81a71b0 con 0x7fbed8102d30 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.813+0000 7fbedfedf640 1 -- 172.21.15.89:0/1307734427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed8103110 con 0x7fbed8102d30 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.814+0000 7fbed6ffd640 1 -- 172.21.15.89:0/1307734427 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbec80305a0 con 0x7fbed8102d30 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.815+0000 7fbed6ffd640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbec00a3240 0x7fbec00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:20.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.815+0000 7fbedd47a640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbec00a3240 0x7fbec00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:20.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.815+0000 7fbed6ffd640 1 -- 172.21.15.89:0/1307734427 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbec802e030 con 0x7fbed8102d30 2024-09-15T23:29:20.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.816+0000 7fbedd47a640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbec00a3240 0x7fbec00a5700 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0058c0 tx=0x7fbecc005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:20.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:20.819+0000 7fbed6ffd640 1 -- 172.21.15.89:0/1307734427 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbec804b060 con 0x7fbed8102d30 2024-09-15T23:29:21.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.021+0000 7fbedfedf640 1 -- 172.21.15.89:0/1307734427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbed81a73e0 con 0x7fbed8102d30 2024-09-15T23:29:21.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.022+0000 7fbed6ffd640 1 -- 172.21.15.89:0/1307734427 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fbec8030890 con 0x7fbed8102d30 2024-09-15T23:29:21.024 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:29:21.024 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:29:21.024 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:29:21.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 -- 172.21.15.89:0/1307734427 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbec00a3240 msgr2=0x7fbec00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:21.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbec00a3240 0x7fbec00a5700 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0058c0 tx=0x7fbecc005850 comp rx=0 tx=0).stop 2024-09-15T23:29:21.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 -- 172.21.15.89:0/1307734427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbed8102d30 msgr2=0x7fbed8100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbed8102d30 0x7fbed8100990 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fbec802ae00 tx=0x7fbec8003970 comp rx=0 tx=0).stop 2024-09-15T23:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 -- 172.21.15.89:0/1307734427 shutdown_connections 2024-09-15T23:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbec00a3240 0x7fbec00a5700 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbed81082a0 0x7fbed8101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbed81036e0 0x7fbed8100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 --2- 172.21.15.89:0/1307734427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbed8102d30 0x7fbed8100990 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 -- 172.21.15.89:0/1307734427 >> 172.21.15.89:0/1307734427 conn(0x7fbed80fc7d0 msgr2=0x7fbed8105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 -- 172.21.15.89:0/1307734427 shutdown_connections 2024-09-15T23:29:21.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:21.025+0000 7fbedfedf640 1 -- 172.21.15.89:0/1307734427 wait complete. 2024-09-15T23:29:21.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:21 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1307734427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:21.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:21 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1307734427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:21.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:21 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1307734427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:21.846 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0' 2024-09-15T23:29:21.846 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0 2024-09-15T23:29:22.098 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:22.372 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:22 smithi089 ceph-mon[30933]: pgmap v510: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:29:22.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:22 smithi120 ceph-mon[35250]: pgmap v510: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:29:22.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:22 smithi174 ceph-mon[35586]: pgmap v510: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.589+0000 7fb5235a0640 1 -- 172.21.15.89:0/1195003079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb51c100480 msgr2=0x7fb51c100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.589+0000 7fb5235a0640 1 --2- 172.21.15.89:0/1195003079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb51c100480 0x7fb51c100860 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb518009900 tx=0x7fb51802a410 comp rx=0 tx=0).stop 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.589+0000 7fb5235a0640 1 -- 172.21.15.89:0/1195003079 shutdown_connections 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.589+0000 7fb5235a0640 1 --2- 172.21.15.89:0/1195003079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb51c1055b0 0x7fb51c10d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.589+0000 7fb5235a0640 1 --2- 172.21.15.89:0/1195003079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c100e30 0x7fb51c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.589+0000 7fb5235a0640 1 --2- 172.21.15.89:0/1195003079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb51c100480 0x7fb51c100860 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.589+0000 7fb5235a0640 1 -- 172.21.15.89:0/1195003079 >> 172.21.15.89:0/1195003079 conn(0x7fb51c077210 msgr2=0x7fb51c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.589+0000 7fb5235a0640 1 -- 172.21.15.89:0/1195003079 shutdown_connections 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.589+0000 7fb5235a0640 1 -- 172.21.15.89:0/1195003079 wait complete. 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.589+0000 7fb5235a0640 1 Processor -- start 2024-09-15T23:29:22.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb5235a0640 1 -- start start 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb5235a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb51c100480 0x7fb51c1992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb5235a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c100e30 0x7fb51c199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb5235a0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb51c1055b0 0x7fb51c1a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb5235a0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c1105b0 con 0x7fb51c100480 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb5235a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c110430 con 0x7fb51c100e30 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb5235a0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c110730 con 0x7fb51c1055b0 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb52133c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb51c100480 0x7fb51c1992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb52133c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb51c100480 0x7fb51c1992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50662/0 (socket says 172.21.15.89:50662) 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb52133c640 1 -- 172.21.15.89:0/44326431 learned_addr learned my addr 172.21.15.89:0/44326431 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb520b3b640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c100e30 0x7fb51c199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb521b3d640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb51c1055b0 0x7fb51c1a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb52133c640 1 -- 172.21.15.89:0/44326431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb51c1055b0 msgr2=0x7fb51c1a4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb52133c640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb51c1055b0 0x7fb51c1a4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb52133c640 1 -- 172.21.15.89:0/44326431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c100e30 msgr2=0x7fb51c199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb52133c640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c100e30 0x7fb51c199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb52133c640 1 -- 172.21.15.89:0/44326431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51c1a4880 con 0x7fb51c100480 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb52133c640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb51c100480 0x7fb51c1992c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb518006010 tx=0x7fb5180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.590+0000 7fb5127fc640 1 -- 172.21.15.89:0/44326431 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51803f030 con 0x7fb51c100480 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.591+0000 7fb5235a0640 1 -- 172.21.15.89:0/44326431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51c1a4a50 con 0x7fb51c100480 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.591+0000 7fb5127fc640 1 -- 172.21.15.89:0/44326431 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb518003ce0 con 0x7fb51c100480 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.591+0000 7fb5127fc640 1 -- 172.21.15.89:0/44326431 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518035070 con 0x7fb51c100480 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.591+0000 7fb5235a0640 1 -- 172.21.15.89:0/44326431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51c1a4f00 con 0x7fb51c100480 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.593+0000 7fb5127fc640 1 -- 172.21.15.89:0/44326431 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb518042030 con 0x7fb51c100480 2024-09-15T23:29:22.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.593+0000 7fb525979740 1 Processor -- start 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.593+0000 7fb525979740 1 -- start start 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.593+0000 7fb525979740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deeb2850 0x5573deeb2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.593+0000 7fb525979740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.593+0000 7fb5127fc640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb5080a3240 0x7fb5080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.593+0000 7fb525979740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deebda30 0x5573deebfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb525979740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5573dee17160 con 0x5573deeb3170 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb525979740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5573dedbe720 con 0x5573deebda30 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb525979740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5573dee13120 con 0x5573deeb2850 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb52133c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb52133c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50678/0 (socket says 172.21.15.89:50678) 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb52133c640 1 -- 172.21.15.89:0/3078150885 learned_addr learned my addr 172.21.15.89:0/3078150885 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:22.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb520b3b640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb5080a3240 0x7fb5080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb521b3d640 1 --2- 172.21.15.89:0/3078150885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deeb2850 0x5573deeb2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb5127fc640 1 -- 172.21.15.89:0/44326431 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb5180f03c0 con 0x7fb51c100480 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb521b3d640 1 --2- 172.21.15.89:0/3078150885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deebda30 0x5573deebfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb52133c640 1 -- 172.21.15.89:0/3078150885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deeb2850 msgr2=0x5573deeb2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb52133c640 1 --2- 172.21.15.89:0/3078150885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deeb2850 0x5573deeb2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb52133c640 1 -- 172.21.15.89:0/3078150885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deebda30 msgr2=0x5573deebfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb52133c640 1 --2- 172.21.15.89:0/3078150885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deebda30 0x5573deebfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb52133c640 1 -- 172.21.15.89:0/3078150885 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573deb88ba0 con 0x5573deeb3170 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb520b3b640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb5080a3240 0x7fb5080a5700 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb51c104070 tx=0x7fb50c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb52133c640 1 --2- 172.21.15.89:0/3078150885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebd4f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb518005ab0 tx=0x7fb51808e540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da93d421d44c2886 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb5077fe640 1 -- 172.21.15.89:0/3078150885 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518049070 con 0x5573deeb3170 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb5077fe640 1 -- 172.21.15.89:0/3078150885 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb51803b580 con 0x5573deeb3170 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.594+0000 7fb5077fe640 1 -- 172.21.15.89:0/3078150885 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5180bcdb0 con 0x5573deeb3170 2024-09-15T23:29:22.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.595+0000 7fb525979740 1 -- 172.21.15.89:0/3078150885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 msgr2=0x5573deebd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.595+0000 7fb525979740 1 --2- 172.21.15.89:0/3078150885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebd4f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb518005ab0 tx=0x7fb51808e540 comp rx=0 tx=0).stop 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.595+0000 7fb525979740 1 -- 172.21.15.89:0/3078150885 shutdown_connections 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.595+0000 7fb525979740 1 --2- 172.21.15.89:0/3078150885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deebda30 0x5573deebfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.595+0000 7fb525979740 1 --2- 172.21.15.89:0/3078150885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebd4f0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.595+0000 7fb525979740 1 --2- 172.21.15.89:0/3078150885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deeb2850 0x5573deeb2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.595+0000 7fb525979740 1 -- 172.21.15.89:0/3078150885 >> 172.21.15.89:0/3078150885 conn(0x5573deeaede0 msgr2=0x5573deeaf1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.595+0000 7fb525979740 1 -- 172.21.15.89:0/3078150885 shutdown_connections 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.595+0000 7fb525979740 1 -- 172.21.15.89:0/3078150885 wait complete. 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.595+0000 7fb525979740 1 -- start start 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb525979740 1 Processor -- start 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb525979740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb2850 0x5573deeba8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb525979740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb525979740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deebb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb525979740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5573deec03c0 con 0x5573deeb3170 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb525979740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5573ded9f900 con 0x5573deeb2850 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb525979740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5573deec0540 con 0x5573deebda30 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb525979740 10 client.0 _ll_register_callbacks cb 0x5573ded9d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb520b3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb520b3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50680/0 (socket says 172.21.15.89:50680) 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb520b3b640 1 -- 172.21.15.89:0/1485456839 learned_addr learned my addr 172.21.15.89:0/1485456839 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb52133c640 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb2850 0x5573deeba8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb521b3d640 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deebb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb520b3b640 1 -- 172.21.15.89:0/1485456839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 msgr2=0x5573deebb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb520b3b640 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deebb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb520b3b640 1 -- 172.21.15.89:0/1485456839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb2850 msgr2=0x5573deeba8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb520b3b640 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb2850 0x5573deeba8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb520b3b640 1 -- 172.21.15.89:0/1485456839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573deddfc50 con 0x5573deeb3170 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb52133c640 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb2850 0x5573deeba8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb520b3b640 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebae00 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fb50c005880 tx=0x7fb50c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e0be8fc87de17e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.596+0000 7fb504ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb504ff9640 1 -- 172.21.15.89:0/1485456839 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50c01e8f0 con 0x5573deeb3170 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb525979740 1 -- 172.21.15.89:0/1485456839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5573dee24890 con 0x5573deeb3170 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb504ff9640 1 -- 172.21.15.89:0/1485456839 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb50c01ea90 con 0x5573deeb3170 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb504ff9640 1 -- 172.21.15.89:0/1485456839 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb50c02f050 con 0x5573deeb3170 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb525979740 10 client.17292 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb525979740 1 -- 172.21.15.89:0/1485456839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5573deecbfd0 con 0x5573deeb3170 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb504ff9640 1 -- 172.21.15.89:0/1485456839 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5573dee24890 con 0x5573deeb3170 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb504ff9640 1 -- 172.21.15.89:0/1485456839 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb50c01c3b0 con 0x5573deeb3170 2024-09-15T23:29:22.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb525979740 10 client.17292 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb525979740 10 client.17292 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb525979740 20 client.17292 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb525979740 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5573deecd490 0x5573deecf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb525979740 4 client.17292 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.597+0000 7fb525979740 1 -- 172.21.15.89:0/1485456839 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573dee1ebe0 con 0x5573deecd490 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.598+0000 7fb52133c640 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5573deecd490 0x5573deecf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.598+0000 7fb504ff9640 10 client.17292 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.598+0000 7fb52133c640 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5573deecd490 0x5573deecf900 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6e28ea96c8155fe server_cookie=77ede7cee865b219 in_seq=0 out_seq=0 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb504ff9640 1 -- 172.21.15.89:0/1485456839 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5573dee1ebe0 con 0x5573deecd490 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb504ff9640 10 client.17292 handle_command_reply: tid=0 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 client.17292 shutdown 2024-09-15T23:29:22.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 -- 172.21.15.89:0/1485456839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 msgr2=0x5573deebae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebae00 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fb50c005880 tx=0x7fb50c004730 comp rx=0 tx=0).stop 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 -- 172.21.15.89:0/1485456839 shutdown_connections 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5573deecd490 0x5573deecf900 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deebb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb3170 0x5573deebae00 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 --2- 172.21.15.89:0/1485456839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb2850 0x5573deeba8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb504ff9640 0 client.17292 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 -- 172.21.15.89:0/1485456839 >> 172.21.15.89:0/1485456839 conn(0x5573deeaede0 msgr2=0x5573deeb4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 -- 172.21.15.89:0/1485456839 shutdown_connections 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 1 -- 172.21.15.89:0/1485456839 wait complete. 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.599+0000 7fb525979740 20 client.17292 trim_cache size 0 max 16384 2024-09-15T23:29:22.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb525979740 1 Processor -- start 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb525979740 1 -- start start 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb525979740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deed5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb525979740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 0x5573deed5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb525979740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deeb7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb525979740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c1aee30 con 0x5573deeb2850 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb525979740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c1b0970 con 0x5573deeb3170 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb525979740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c1af500 con 0x5573deebda30 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb52133c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deed5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb52133c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deed5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50682/0 (socket says 172.21.15.89:50682) 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb52133c640 1 -- 172.21.15.89:0/3996329829 learned_addr learned my addr 172.21.15.89:0/3996329829 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb520b3b640 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 0x5573deed5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb52133c640 1 -- 172.21.15.89:0/3996329829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 msgr2=0x5573deeb7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb521b3d640 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deeb7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb52133c640 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deeb7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb52133c640 1 -- 172.21.15.89:0/3996329829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 msgr2=0x5573deed5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb52133c640 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 0x5573deed5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb52133c640 1 -- 172.21.15.89:0/3996329829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573deeaa3f0 con 0x5573deeb2850 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb521b3d640 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deeb7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.616+0000 7fb520b3b640 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 0x5573deed5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.617+0000 7fb52133c640 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deed5210 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fb5180b8390 tx=0x7fb51808e540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28234718a794c30d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.617+0000 7fb506ffd640 1 -- 172.21.15.89:0/3996329829 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518049040 con 0x5573deeb2850 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.617+0000 7fb506ffd640 1 -- 172.21.15.89:0/3996329829 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5180a8040 con 0x5573deeb2850 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.617+0000 7fb506ffd640 1 -- 172.21.15.89:0/3996329829 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5180c0a60 con 0x5573deeb2850 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 -- 172.21.15.89:0/3996329829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 msgr2=0x5573deed5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deed5210 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fb5180b8390 tx=0x7fb51808e540 comp rx=0 tx=0).stop 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 -- 172.21.15.89:0/3996329829 shutdown_connections 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deeb7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 0x5573deed5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 --2- 172.21.15.89:0/3996329829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deed5210 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 -- 172.21.15.89:0/3996329829 >> 172.21.15.89:0/3996329829 conn(0x5573deeaede0 msgr2=0x5573deebc120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 -- 172.21.15.89:0/3996329829 shutdown_connections 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 -- 172.21.15.89:0/3996329829 wait complete. 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 -- start start 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 Processor -- start 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deec5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 0x5573deec5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deec0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c1b02a0 con 0x5573deeb2850 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c1afbd0 con 0x5573deeb3170 2024-09-15T23:29:22.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.618+0000 7fb525979740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c1ace10 con 0x5573deebda30 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb525979740 10 client.0 _ll_register_callbacks cb 0x7fb50c00b910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb52133c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deec5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb52133c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deec5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50684/0 (socket says 172.21.15.89:50684) 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb52133c640 1 -- 172.21.15.89:0/3709796258 learned_addr learned my addr 172.21.15.89:0/3709796258 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb521b3d640 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deec0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb520b3b640 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 0x5573deec5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb52133c640 1 -- 172.21.15.89:0/3709796258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 msgr2=0x5573deec0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb52133c640 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deec0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb52133c640 1 -- 172.21.15.89:0/3709796258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 msgr2=0x5573deec5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb52133c640 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 0x5573deec5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb52133c640 1 -- 172.21.15.89:0/3709796258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573deec0b90 con 0x5573deeb2850 2024-09-15T23:29:22.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb520b3b640 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 0x5573deec5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb52133c640 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deec5780 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb518003fd0 tx=0x7fb5180bc270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637e20b8cd358d21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb504ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb504ff9640 1 -- 172.21.15.89:0/3709796258 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518049050 con 0x5573deeb2850 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb504ff9640 1 -- 172.21.15.89:0/3709796258 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb5180f0150 con 0x5573deeb2850 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb525979740 1 -- 172.21.15.89:0/3709796258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5180a8040 con 0x5573deeb2850 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb504ff9640 1 -- 172.21.15.89:0/3709796258 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb5180bcdb0 con 0x5573deeb2850 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb525979740 10 client.17298 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.619+0000 7fb525979740 1 -- 172.21.15.89:0/3709796258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5573def38a30 con 0x5573deeb2850 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb504ff9640 1 -- 172.21.15.89:0/3709796258 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5180a8040 con 0x5573deeb2850 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb504ff9640 1 -- 172.21.15.89:0/3709796258 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb5180ab050 con 0x5573deeb2850 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb525979740 10 client.17298 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb525979740 10 client.17298 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb525979740 20 client.17298 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb525979740 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5573def38de0 0x5573def3b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb525979740 4 client.17298 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}] 2024-09-15T23:29:22.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb525979740 1 -- 172.21.15.89:0/3709796258 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}) -- 0x5573ded9af40 con 0x5573def38de0 2024-09-15T23:29:22.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb520b3b640 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5573def38de0 0x5573def3b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb504ff9640 10 client.17298 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:22.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.620+0000 7fb520b3b640 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5573def38de0 0x5573def3b1d0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30bc3208e401c6e4 server_cookie=fc84f30606ce26e4 in_seq=0 out_seq=0 2024-09-15T23:29:22.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.632+0000 7fb504ff9640 1 -- 172.21.15.89:0/3709796258 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5573ded9af40 con 0x5573def38de0 2024-09-15T23:29:22.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.632+0000 7fb504ff9640 10 client.17298 handle_command_reply: tid=0 2024-09-15T23:29:22.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.632+0000 7fb525979740 1 client.17298 shutdown 2024-09-15T23:29:22.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 1 -- 172.21.15.89:0/3709796258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 msgr2=0x5573deec5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deec5780 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb518003fd0 tx=0x7fb5180bc270 comp rx=0 tx=0).stop 2024-09-15T23:29:22.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 1 -- 172.21.15.89:0/3709796258 shutdown_connections 2024-09-15T23:29:22.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5573def38de0 0x5573def3b1d0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5573deebda30 0x5573deec0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573deeb3170 0x5573deec5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 1 --2- 172.21.15.89:0/3709796258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5573deeb2850 0x5573deec5780 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb504ff9640 0 client.17298 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:22.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 1 -- 172.21.15.89:0/3709796258 >> 172.21.15.89:0/3709796258 conn(0x5573deeaede0 msgr2=0x5573deeba860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:22.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 1 -- 172.21.15.89:0/3709796258 shutdown_connections 2024-09-15T23:29:22.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 1 -- 172.21.15.89:0/3709796258 wait complete. 2024-09-15T23:29:22.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.633+0000 7fb525979740 20 client.17298 trim_cache size 0 max 16384 2024-09-15T23:29:22.639 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:29:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:29:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:29:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 18, 2024-09-15T23:29:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "5e38a27a": { 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 18, 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 145.3, 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:29:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:29:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:29:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:29:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:22.641 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:22.641 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 -- 172.21.15.89:0/44326431 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb5080a3240 msgr2=0x7fb5080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb5080a3240 0x7fb5080a5700 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb51c104070 tx=0x7fb50c00b390 comp rx=0 tx=0).stop 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 -- 172.21.15.89:0/44326431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb51c100480 msgr2=0x7fb51c1992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb51c100480 0x7fb51c1992c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb518006010 tx=0x7fb5180042f0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 -- 172.21.15.89:0/44326431 shutdown_connections 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb5080a3240 0x7fb5080a5700 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb51c1055b0 0x7fb51c1a4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb51c100e30 0x7fb51c199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 --2- 172.21.15.89:0/44326431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb51c100480 0x7fb51c1992c0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 -- 172.21.15.89:0/44326431 >> 172.21.15.89:0/44326431 conn(0x7fb51c077210 msgr2=0x7fb51c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:22.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 -- 172.21.15.89:0/44326431 shutdown_connections 2024-09-15T23:29:22.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:22.634+0000 7fb5235a0640 1 -- 172.21.15.89:0/44326431 wait complete. 2024-09-15T23:29:23.452 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:23 smithi089 ceph-mon[30933]: pgmap v511: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:29:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:23 smithi120 ceph-mon[35250]: pgmap v511: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:29:23.480 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5e38a27a', quiesce took 3.4 seconds. Will release after: 21.950000000000003 2024-09-15T23:29:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:23 smithi174 ceph-mon[35586]: pgmap v511: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:29:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:25 smithi120 ceph-mon[35250]: pgmap v512: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:29:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:25 smithi174 ceph-mon[35586]: pgmap v512: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:29:25.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:25 smithi089 ceph-mon[30933]: pgmap v512: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:29:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:27 smithi120 ceph-mon[35250]: pgmap v513: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 KiB/s wr, 20 op/s 2024-09-15T23:29:27.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:27 smithi174 ceph-mon[35586]: pgmap v513: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 KiB/s wr, 20 op/s 2024-09-15T23:29:27.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:27 smithi089 ceph-mon[30933]: pgmap v513: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 KiB/s wr, 20 op/s 2024-09-15T23:29:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:29 smithi120 ceph-mon[35250]: pgmap v514: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-09-15T23:29:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:29 smithi174 ceph-mon[35586]: pgmap v514: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-09-15T23:29:29.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:29 smithi089 ceph-mon[30933]: pgmap v514: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-09-15T23:29:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 -- 172.21.15.89:0/515214617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7834102070 msgr2=0x7f7834102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 --2- 172.21.15.89:0/515214617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7834102070 0x7f7834102470 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f7824009900 tx=0x7f782402a410 comp rx=0 tx=0).stop 2024-09-15T23:29:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 -- 172.21.15.89:0/515214617 shutdown_connections 2024-09-15T23:29:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 --2- 172.21.15.89:0/515214617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7834101680 0x7f7834108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 --2- 172.21.15.89:0/515214617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7834100cc0 0x7f7834101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 --2- 172.21.15.89:0/515214617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7834102070 0x7f7834102470 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 -- 172.21.15.89:0/515214617 >> 172.21.15.89:0/515214617 conn(0x7f78340fb980 msgr2=0x7f78340fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 -- 172.21.15.89:0/515214617 shutdown_connections 2024-09-15T23:29:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 -- 172.21.15.89:0/515214617 wait complete. 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 Processor -- start 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.205+0000 7f783aaf1640 1 -- start start 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783aaf1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7834100cc0 0x7f7834197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783aaf1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7834101680 0x7f7834198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783aaf1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7834102070 0x7f783419f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783aaf1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f783410c6f0 con 0x7f7834100cc0 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783aaf1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f783410c570 con 0x7f7834102070 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783aaf1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f783410c870 con 0x7f7834101680 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783888d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7834100cc0 0x7f7834197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783888d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7834100cc0 0x7f7834197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48978/0 (socket says 172.21.15.89:48978) 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783888d640 1 -- 172.21.15.89:0/357910370 learned_addr learned my addr 172.21.15.89:0/357910370 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f7833fff640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7834101680 0x7f7834198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783908e640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7834102070 0x7f783419f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783888d640 1 -- 172.21.15.89:0/357910370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7834101680 msgr2=0x7f7834198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783888d640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7834101680 0x7f7834198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783888d640 1 -- 172.21.15.89:0/357910370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7834102070 msgr2=0x7f783419f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783888d640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7834102070 0x7f783419f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783888d640 1 -- 172.21.15.89:0/357910370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f783419fa70 con 0x7f7834100cc0 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783908e640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7834102070 0x7f783419f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783888d640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7834100cc0 0x7f7834197e50 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7824031610 tx=0x7f78240043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f7831ffb640 1 -- 172.21.15.89:0/357910370 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782403f030 con 0x7f7834100cc0 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f7831ffb640 1 -- 172.21.15.89:0/357910370 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7824003ce0 con 0x7f7834100cc0 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f783aaf1640 1 -- 172.21.15.89:0/357910370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f783419fd00 con 0x7f7834100cc0 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.206+0000 7f7831ffb640 1 -- 172.21.15.89:0/357910370 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824035070 con 0x7f7834100cc0 2024-09-15T23:29:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.207+0000 7f783aaf1640 1 -- 172.21.15.89:0/357910370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78341a01a0 con 0x7f7834100cc0 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.209+0000 7f7831ffb640 1 -- 172.21.15.89:0/357910370 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7824042030 con 0x7f7834100cc0 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.209+0000 7f783d890740 1 Processor -- start 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.209+0000 7f783d890740 1 -- start start 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.209+0000 7f783d890740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 0x5618698e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783d890740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x5618698e9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783d890740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698ea370 0x5618698ec830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783d890740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5618698482f0 con 0x5618698e52e0 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783d890740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618697f0270 con 0x5618698ea370 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783d890740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5618698442b0 con 0x5618698e49a0 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f7831ffb640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f781c0a32f0 0x7f781c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783888d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x5618698e9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783888d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x5618698e9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48988/0 (socket says 172.21.15.89:48988) 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783888d640 1 -- 172.21.15.89:0/1008329217 learned_addr learned my addr 172.21.15.89:0/1008329217 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783908e640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f781c0a32f0 0x7f781c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f7833fff640 1 --2- 172.21.15.89:0/1008329217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698ea370 0x5618698ec830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783908e640 1 --2- 172.21.15.89:0/1008329217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 0x5618698e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f7831ffb640 1 -- 172.21.15.89:0/357910370 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f78240f03e0 con 0x7f7834100cc0 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783888d640 1 -- 172.21.15.89:0/1008329217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 msgr2=0x5618698e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783888d640 1 --2- 172.21.15.89:0/1008329217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 0x5618698e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783888d640 1 -- 172.21.15.89:0/1008329217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698ea370 msgr2=0x5618698ec830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783888d640 1 --2- 172.21.15.89:0/1008329217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698ea370 0x5618698ec830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783888d640 1 -- 172.21.15.89:0/1008329217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618697ef340 con 0x5618698e52e0 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783908e640 1 --2- 172.21.15.89:0/1008329217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 0x5618698e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783888d640 1 --2- 172.21.15.89:0/1008329217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x5618698e9da0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f7824086f10 tx=0x7f782408f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe60176eb3cb3c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:30.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f783908e640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f781c0a32f0 0x7f781c0a57b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f782c009fd0 tx=0x7f782c009550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.210+0000 7f781affd640 1 -- 172.21.15.89:0/1008329217 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78240bccb0 con 0x5618698e52e0 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f781affd640 1 -- 172.21.15.89:0/1008329217 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7824087450 con 0x5618698e52e0 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f781affd640 1 -- 172.21.15.89:0/1008329217 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78240edba0 con 0x5618698e52e0 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f783d890740 1 -- 172.21.15.89:0/1008329217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 msgr2=0x5618698e9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f783d890740 1 --2- 172.21.15.89:0/1008329217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x5618698e9da0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f7824086f10 tx=0x7f782408f430 comp rx=0 tx=0).stop 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f783d890740 1 -- 172.21.15.89:0/1008329217 shutdown_connections 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f783d890740 1 --2- 172.21.15.89:0/1008329217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698ea370 0x5618698ec830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f783d890740 1 --2- 172.21.15.89:0/1008329217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x5618698e9da0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f783d890740 1 --2- 172.21.15.89:0/1008329217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 0x5618698e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f783d890740 1 -- 172.21.15.89:0/1008329217 >> 172.21.15.89:0/1008329217 conn(0x5618698e0010 msgr2=0x5618698e0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f783d890740 1 -- 172.21.15.89:0/1008329217 shutdown_connections 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.211+0000 7f783d890740 1 -- 172.21.15.89:0/1008329217 wait complete. 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783d890740 1 -- start start 2024-09-15T23:29:30.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783d890740 1 Processor -- start 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783d890740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 0x5618698ee910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783d890740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698e52e0 0x5618698eee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783d890740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698ea370 0x5618698f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783d890740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5618698ed100 con 0x5618698ea370 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783d890740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618697d1290 con 0x5618698e52e0 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783d890740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5618698ed280 con 0x5618698e49a0 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783d890740 10 client.0 _ll_register_callbacks cb 0x561869802140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783908e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698ea370 0x5618698f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783908e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698ea370 0x5618698f5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49004/0 (socket says 172.21.15.89:49004) 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783908e640 1 -- 172.21.15.89:0/3887586352 learned_addr learned my addr 172.21.15.89:0/3887586352 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f7833fff640 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698e52e0 0x5618698eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.212+0000 7f783888d640 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 0x5618698ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783908e640 1 -- 172.21.15.89:0/3887586352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 msgr2=0x5618698ee910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783908e640 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 0x5618698ee910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783908e640 1 -- 172.21.15.89:0/3887586352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698e52e0 msgr2=0x5618698eee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783908e640 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698e52e0 0x5618698eee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783908e640 1 -- 172.21.15.89:0/3887586352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56186955b210 con 0x5618698ea370 2024-09-15T23:29:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783888d640 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 0x5618698ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783908e640 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698ea370 0x5618698f5e90 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f782c005f40 tx=0x7f782c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edbf266589ed606 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f7813fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:30.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f7813fff640 1 -- 172.21.15.89:0/3887586352 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c026070 con 0x5618698ea370 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f7813fff640 1 -- 172.21.15.89:0/3887586352 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f782c003b70 con 0x5618698ea370 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783d890740 1 -- 172.21.15.89:0/3887586352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7824087450 con 0x5618698ea370 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f7813fff640 1 -- 172.21.15.89:0/3887586352 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f782c031050 con 0x5618698ea370 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783d890740 10 client.17310 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783d890740 1 -- 172.21.15.89:0/3887586352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561869627620 con 0x5618698ea370 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f7813fff640 1 -- 172.21.15.89:0/3887586352 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7824087450 con 0x5618698ea370 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f7813fff640 1 -- 172.21.15.89:0/3887586352 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f782c01d050 con 0x5618698ea370 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783d890740 10 client.17310 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.213+0000 7f783d890740 10 client.17310 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.214+0000 7f783d890740 20 client.17310 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.214+0000 7f783d890740 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5618698fa600 0x5618698fcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.214+0000 7f783d890740 4 client.17310 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:30.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.214+0000 7f783d890740 1 -- 172.21.15.89:0/3887586352 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561869855a20 con 0x5618698fa600 2024-09-15T23:29:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.214+0000 7f783888d640 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5618698fa600 0x5618698fcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.214+0000 7f7813fff640 10 client.17310 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:30.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.214+0000 7f783888d640 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5618698fa600 0x5618698fcaa0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=877a4f4a3ff01fe6 server_cookie=ac0450166e3d67a4 in_seq=0 out_seq=0 2024-09-15T23:29:30.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f7813fff640 1 -- 172.21.15.89:0/3887586352 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561869855a20 con 0x5618698fa600 2024-09-15T23:29:30.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f7813fff640 10 client.17310 handle_command_reply: tid=0 2024-09-15T23:29:30.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f783d890740 1 client.17310 shutdown 2024-09-15T23:29:30.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f783d890740 1 -- 172.21.15.89:0/3887586352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698ea370 msgr2=0x5618698f5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:30.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f783d890740 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698ea370 0x5618698f5e90 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f782c005f40 tx=0x7f782c004ab0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f783d890740 1 -- 172.21.15.89:0/3887586352 shutdown_connections 2024-09-15T23:29:30.223 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f783d890740 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5618698fa600 0x5618698fcaa0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f783d890740 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698ea370 0x5618698f5e90 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f7813fff640 0 client.17310 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:30.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f783d890740 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698e52e0 0x5618698eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f783d890740 1 --2- 172.21.15.89:0/3887586352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e49a0 0x5618698ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.215+0000 7f783d890740 1 -- 172.21.15.89:0/3887586352 >> 172.21.15.89:0/3887586352 conn(0x5618698e0010 msgr2=0x5618698e1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:30.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.216+0000 7f783d890740 1 -- 172.21.15.89:0/3887586352 shutdown_connections 2024-09-15T23:29:30.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.216+0000 7f783d890740 1 -- 172.21.15.89:0/3887586352 wait complete. 2024-09-15T23:29:30.224 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.216+0000 7f783d890740 20 client.17310 trim_cache size 0 max 16384 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783d890740 1 Processor -- start 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783d890740 1 -- start start 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783d890740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e49a0 0x5618698e0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783d890740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e52e0 0x5618698e1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783d890740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698ea370 0x5618698e1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783d890740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f78341aa8f0 con 0x5618698e49a0 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783d890740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78341ac430 con 0x5618698ea370 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783d890740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f78341aafc0 con 0x5618698e52e0 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783888d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e49a0 0x5618698e0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783888d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e49a0 0x5618698e0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49012/0 (socket says 172.21.15.89:49012) 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783888d640 1 -- 172.21.15.89:0/1567435489 learned_addr learned my addr 172.21.15.89:0/1567435489 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783888d640 1 -- 172.21.15.89:0/1567435489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e52e0 msgr2=0x5618698e1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783888d640 1 --2- 172.21.15.89:0/1567435489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e52e0 0x5618698e1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783888d640 1 -- 172.21.15.89:0/1567435489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698ea370 msgr2=0x5618698e1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783908e640 1 --2- 172.21.15.89:0/1567435489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698ea370 0x5618698e1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783888d640 1 --2- 172.21.15.89:0/1567435489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698ea370 0x5618698e1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783888d640 1 -- 172.21.15.89:0/1567435489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618698dfe30 con 0x5618698e49a0 2024-09-15T23:29:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.230+0000 7f783888d640 1 --2- 172.21.15.89:0/1567435489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e49a0 0x5618698e0ec0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f7824038020 tx=0x7f78240f0660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70811274921649 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f781a7fc640 1 -- 172.21.15.89:0/1567435489 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7824049040 con 0x5618698e49a0 2024-09-15T23:29:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f781a7fc640 1 -- 172.21.15.89:0/1567435489 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78240f0120 con 0x5618698e49a0 2024-09-15T23:29:30.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f781a7fc640 1 -- 172.21.15.89:0/1567435489 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782403b460 con 0x5618698e49a0 2024-09-15T23:29:30.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f783d890740 1 -- 172.21.15.89:0/1567435489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e49a0 msgr2=0x5618698e0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f783d890740 1 --2- 172.21.15.89:0/1567435489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e49a0 0x5618698e0ec0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f7824038020 tx=0x7f78240f0660 comp rx=0 tx=0).stop 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f783d890740 1 -- 172.21.15.89:0/1567435489 shutdown_connections 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f783d890740 1 --2- 172.21.15.89:0/1567435489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698ea370 0x5618698e1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f783d890740 1 --2- 172.21.15.89:0/1567435489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698e52e0 0x5618698e1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f783d890740 1 --2- 172.21.15.89:0/1567435489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e49a0 0x5618698e0ec0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f783d890740 1 -- 172.21.15.89:0/1567435489 >> 172.21.15.89:0/1567435489 conn(0x5618698e0010 msgr2=0x5618698ee910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f783d890740 1 -- 172.21.15.89:0/1567435489 shutdown_connections 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.231+0000 7f783d890740 1 -- 172.21.15.89:0/1567435489 wait complete. 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783d890740 1 -- start start 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783d890740 1 Processor -- start 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783d890740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698e49a0 0x561869960420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783d890740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x561869960960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783d890740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698ea370 0x5618699679a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783d890740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f78341abd60 con 0x5618698e52e0 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783d890740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78341ab690 con 0x5618698e49a0 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783d890740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f78341a88d0 con 0x5618698ea370 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783d890740 10 client.0 _ll_register_callbacks cb 0x7f782c009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f7833fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x561869960960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f7833fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x561869960960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49014/0 (socket says 172.21.15.89:49014) 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f7833fff640 1 -- 172.21.15.89:0/278574248 learned_addr learned my addr 172.21.15.89:0/278574248 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783888d640 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698e49a0 0x561869960420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f7833fff640 1 -- 172.21.15.89:0/278574248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698ea370 msgr2=0x5618699679a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f783908e640 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698ea370 0x5618699679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f7833fff640 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698ea370 0x5618699679a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f7833fff640 1 -- 172.21.15.89:0/278574248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698e49a0 msgr2=0x561869960420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f7833fff640 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698e49a0 0x561869960420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.232+0000 7f7833fff640 1 -- 172.21.15.89:0/278574248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561869967fb0 con 0x5618698e52e0 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f7833fff640 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x561869960960 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f783406a2d0 tx=0x7f7828013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6525efdfa6ecb57 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f7813fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f7813fff640 1 -- 172.21.15.89:0/278574248 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7828020060 con 0x5618698e52e0 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f7813fff640 1 -- 172.21.15.89:0/278574248 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7828011650 con 0x5618698e52e0 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f783d890740 1 -- 172.21.15.89:0/278574248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78240f0120 con 0x5618698e52e0 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f7813fff640 1 -- 172.21.15.89:0/278574248 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7828027050 con 0x5618698e52e0 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f783d890740 10 client.17316 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f783d890740 1 -- 172.21.15.89:0/278574248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561869969390 con 0x5618698e52e0 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f7813fff640 1 -- 172.21.15.89:0/278574248 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78240f0120 con 0x5618698e52e0 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f7813fff640 1 -- 172.21.15.89:0/278574248 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f782801a030 con 0x5618698e52e0 2024-09-15T23:29:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f783d890740 10 client.17316 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f783d890740 10 client.17316 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f783d890740 20 client.17316 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f783d890740 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561869969d70 0x56186996c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f783d890740 4 client.17316 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:29:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.233+0000 7f783d890740 1 -- 172.21.15.89:0/278574248 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56186984fd70 con 0x561869969d70 2024-09-15T23:29:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.234+0000 7f783888d640 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561869969d70 0x56186996c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.234+0000 7f783888d640 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561869969d70 0x56186996c190 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e96ef4a9416fae6 server_cookie=aa6546fa149b2e3d in_seq=0 out_seq=0 2024-09-15T23:29:30.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.234+0000 7f7813fff640 10 client.17316 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.248+0000 7f7813fff640 1 -- 172.21.15.89:0/278574248 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56186984fd70 con 0x561869969d70 2024-09-15T23:29:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.248+0000 7f7813fff640 10 client.17316 handle_command_reply: tid=0 2024-09-15T23:29:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.248+0000 7f783d890740 1 client.17316 shutdown 2024-09-15T23:29:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.248+0000 7f783d890740 1 -- 172.21.15.89:0/278574248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 msgr2=0x561869960960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.248+0000 7f783d890740 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x561869960960 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f783406a2d0 tx=0x7f7828013830 comp rx=0 tx=0).stop 2024-09-15T23:29:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783d890740 1 -- 172.21.15.89:0/278574248 shutdown_connections 2024-09-15T23:29:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783d890740 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561869969d70 0x56186996c190 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783d890740 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618698ea370 0x5618699679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783d890740 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618698e52e0 0x561869960960 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783d890740 1 --2- 172.21.15.89:0/278574248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618698e49a0 0x561869960420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f7813fff640 0 client.17316 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783d890740 1 -- 172.21.15.89:0/278574248 >> 172.21.15.89:0/278574248 conn(0x5618698e0010 msgr2=0x5618698ee8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783d890740 1 -- 172.21.15.89:0/278574248 shutdown_connections 2024-09-15T23:29:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783d890740 1 -- 172.21.15.89:0/278574248 wait complete. 2024-09-15T23:29:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783d890740 20 client.17316 trim_cache size 0 max 16384 2024-09-15T23:29:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:29:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:29:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:29:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:29:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783aaf1640 1 -- 172.21.15.89:0/357910370 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f781c0a32f0 msgr2=0x7f781c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783aaf1640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f781c0a32f0 0x7f781c0a57b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f782c009fd0 tx=0x7f782c009550 comp rx=0 tx=0).stop 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783aaf1640 1 -- 172.21.15.89:0/357910370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7834100cc0 msgr2=0x7f7834197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783aaf1640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7834100cc0 0x7f7834197e50 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7824031610 tx=0x7f78240043b0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783aaf1640 1 -- 172.21.15.89:0/357910370 shutdown_connections 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783aaf1640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f781c0a32f0 0x7f781c0a57b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783aaf1640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7834102070 0x7f783419f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783aaf1640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7834101680 0x7f7834198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783aaf1640 1 --2- 172.21.15.89:0/357910370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7834100cc0 0x7f7834197e50 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.249+0000 7f783aaf1640 1 -- 172.21.15.89:0/357910370 >> 172.21.15.89:0/357910370 conn(0x7f78340fb980 msgr2=0x7f78340fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.250+0000 7f783aaf1640 1 -- 172.21.15.89:0/357910370 shutdown_connections 2024-09-15T23:29:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:30.250+0000 7f783aaf1640 1 -- 172.21.15.89:0/357910370 wait complete. 2024-09-15T23:29:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:31 smithi120 ceph-mon[35250]: pgmap v515: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:29:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:31 smithi174 ceph-mon[35586]: pgmap v515: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:29:31.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:31 smithi089 ceph-mon[30933]: pgmap v515: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:29:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:33 smithi120 ceph-mon[35250]: pgmap v516: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 4.1 KiB/s wr, 13 op/s 2024-09-15T23:29:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:33 smithi174 ceph-mon[35586]: pgmap v516: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 4.1 KiB/s wr, 13 op/s 2024-09-15T23:29:33.547 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:33 smithi089 ceph-mon[30933]: pgmap v516: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 4.1 KiB/s wr, 13 op/s 2024-09-15T23:29:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:35 smithi120 ceph-mon[35250]: pgmap v517: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.1 KiB/s wr, 12 op/s 2024-09-15T23:29:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:35 smithi174 ceph-mon[35586]: pgmap v517: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.1 KiB/s wr, 12 op/s 2024-09-15T23:29:35.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:35 smithi089 ceph-mon[30933]: pgmap v517: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 4.1 KiB/s wr, 12 op/s 2024-09-15T23:29:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:37 smithi120 ceph-mon[35250]: pgmap v518: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.0 KiB/s wr, 15 op/s 2024-09-15T23:29:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:37 smithi174 ceph-mon[35586]: pgmap v518: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.0 KiB/s wr, 15 op/s 2024-09-15T23:29:37.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:37 smithi089 ceph-mon[30933]: pgmap v518: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 4.0 KiB/s wr, 15 op/s 2024-09-15T23:29:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:39 smithi120 ceph-mon[35250]: pgmap v519: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-15T23:29:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:39 smithi174 ceph-mon[35586]: pgmap v519: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-15T23:29:39.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:39 smithi089 ceph-mon[30933]: pgmap v519: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-15T23:29:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:41 smithi120 ceph-mon[35250]: pgmap v520: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.4 KiB/s wr, 14 op/s 2024-09-15T23:29:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:41 smithi174 ceph-mon[35586]: pgmap v520: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.4 KiB/s wr, 14 op/s 2024-09-15T23:29:41.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:41 smithi089 ceph-mon[30933]: pgmap v520: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.4 KiB/s wr, 14 op/s 2024-09-15T23:29:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.345+0000 7f3c5983c640 1 -- 172.21.15.89:0/699300060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54100a50 msgr2=0x7f3c54100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.345+0000 7f3c5983c640 1 --2- 172.21.15.89:0/699300060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54100a50 0x7f3c54100e50 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f3c44009900 tx=0x7f3c4402a410 comp rx=0 tx=0).stop 2024-09-15T23:29:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.345+0000 7f3c5983c640 1 -- 172.21.15.89:0/699300060 shutdown_connections 2024-09-15T23:29:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.345+0000 7f3c5983c640 1 --2- 172.21.15.89:0/699300060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c54102600 0x7f3c54108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.345+0000 7f3c5983c640 1 --2- 172.21.15.89:0/699300060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c54101c40 0x7f3c541020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.345+0000 7f3c5983c640 1 --2- 172.21.15.89:0/699300060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54100a50 0x7f3c54100e50 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.345+0000 7f3c5983c640 1 -- 172.21.15.89:0/699300060 >> 172.21.15.89:0/699300060 conn(0x7f3c540fc1b0 msgr2=0x7f3c540fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.345+0000 7f3c5983c640 1 -- 172.21.15.89:0/699300060 shutdown_connections 2024-09-15T23:29:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.345+0000 7f3c5983c640 1 -- 172.21.15.89:0/699300060 wait complete. 2024-09-15T23:29:43.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c5983c640 1 Processor -- start 2024-09-15T23:29:43.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c5983c640 1 -- start start 2024-09-15T23:29:43.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c5983c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54100a50 0x7f3c5406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c5983c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c54101c40 0x7f3c5419cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.349 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c5983c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c54102600 0x7f3c5419f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c5983c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5410c6f0 con 0x7f3c54100a50 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c5983c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5410c570 con 0x7f3c54102600 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c5983c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5410c870 con 0x7f3c54101c40 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c52ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54100a50 0x7f3c5406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c52ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54100a50 0x7f3c5406ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60706/0 (socket says 172.21.15.89:60706) 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c52ffd640 1 -- 172.21.15.89:0/3465338823 learned_addr learned my addr 172.21.15.89:0/3465338823 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c537fe640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c54102600 0x7f3c5419f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c54101c40 0x7f3c5419cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c52ffd640 1 -- 172.21.15.89:0/3465338823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c54101c40 msgr2=0x7f3c5419cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c54101c40 0x7f3c5419cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c52ffd640 1 -- 172.21.15.89:0/3465338823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c54102600 msgr2=0x7f3c5419f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c54102600 0x7f3c5419f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c52ffd640 1 -- 172.21.15.89:0/3465338823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5419fa40 con 0x7f3c54100a50 2024-09-15T23:29:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.346+0000 7f3c537fe640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c54102600 0x7f3c5419f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.347+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54100a50 0x7f3c5406ae70 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f3c44031610 tx=0x7f3c440043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.347+0000 7f3c5883a640 1 -- 172.21.15.89:0/3465338823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4403f030 con 0x7f3c54100a50 2024-09-15T23:29:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.347+0000 7f3c5883a640 1 -- 172.21.15.89:0/3465338823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c44003ce0 con 0x7f3c54100a50 2024-09-15T23:29:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.347+0000 7f3c5983c640 1 -- 172.21.15.89:0/3465338823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5419fc70 con 0x7f3c54100a50 2024-09-15T23:29:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.347+0000 7f3c5883a640 1 -- 172.21.15.89:0/3465338823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c44035070 con 0x7f3c54100a50 2024-09-15T23:29:43.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.347+0000 7f3c5983c640 1 -- 172.21.15.89:0/3465338823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c541a0160 con 0x7f3c54100a50 2024-09-15T23:29:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.349+0000 7f3c5883a640 1 -- 172.21.15.89:0/3465338823 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3c44042030 con 0x7f3c54100a50 2024-09-15T23:29:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.349+0000 7f3c5c5d6740 1 Processor -- start 2024-09-15T23:29:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c5c5d6740 1 -- start start 2024-09-15T23:29:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce589a0 0x55e8fce58da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce5dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 0x55e8fce60830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e8fcdbc2f0 con 0x55e8fce592e0 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e8fcd64270 con 0x55e8fce5e370 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e8fcdb82b0 con 0x55e8fce589a0 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c52ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce5dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c52ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce5dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60712/0 (socket says 172.21.15.89:60712) 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c52ffd640 1 -- 172.21.15.89:0/232606650 learned_addr learned my addr 172.21.15.89:0/232606650 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c5883a640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c400a32f0 0x7f3c400a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/232606650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 0x55e8fce60830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c537fe640 1 --2- 172.21.15.89:0/232606650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce589a0 0x55e8fce58da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c537fe640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c400a32f0 0x7f3c400a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c5883a640 1 -- 172.21.15.89:0/3465338823 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c440f0350 con 0x7f3c54100a50 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c52ffd640 1 -- 172.21.15.89:0/232606650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce589a0 msgr2=0x55e8fce58da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/232606650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce589a0 0x55e8fce58da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c52ffd640 1 -- 172.21.15.89:0/232606650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 msgr2=0x55e8fce60830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/232606650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 0x55e8fce60830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c52ffd640 1 -- 172.21.15.89:0/232606650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8fcd63340 con 0x55e8fce592e0 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.350+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/232606650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce5dda0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f3c440bc280 tx=0x7f3c4402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a46da3a7a27b313 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c537fe640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c400a32f0 0x7f3c400a57b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009fd0 tx=0x7f3c4c0094a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c49ffb640 1 -- 172.21.15.89:0/232606650 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c44049070 con 0x55e8fce592e0 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c49ffb640 1 -- 172.21.15.89:0/232606650 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c440bcc80 con 0x55e8fce592e0 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c49ffb640 1 -- 172.21.15.89:0/232606650 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4403bd70 con 0x55e8fce592e0 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/232606650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 msgr2=0x55e8fce5dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/232606650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce5dda0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f3c440bc280 tx=0x7f3c4402a410 comp rx=0 tx=0).stop 2024-09-15T23:29:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/232606650 shutdown_connections 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/232606650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 0x55e8fce60830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/232606650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce5dda0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/232606650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce589a0 0x55e8fce58da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.351+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/232606650 >> 172.21.15.89:0/232606650 conn(0x55e8fce54010 msgr2=0x55e8fce54410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/232606650 shutdown_connections 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/232606650 wait complete. 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 1 -- start start 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 1 Processor -- start 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 0x55e8fce627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce62ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 0x55e8fce69d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e8fcdbc2f0 con 0x55e8fce592e0 2024-09-15T23:29:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e8fcd45290 con 0x55e8fce589a0 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e8fce61100 con 0x55e8fce5e370 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c5c5d6740 10 client.0 _ll_register_callbacks cb 0x55e8fcd76140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c4a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce62ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.352+0000 7f3c4a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce62ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60724/0 (socket says 172.21.15.89:60724) 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/4069670800 learned_addr learned my addr 172.21.15.89:0/4069670800 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 0x55e8fce627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c537fe640 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 0x55e8fce69d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/4069670800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 msgr2=0x55e8fce69d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 0x55e8fce69d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/4069670800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 msgr2=0x55e8fce627a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 0x55e8fce627a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/4069670800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8fcacf210 con 0x55e8fce592e0 2024-09-15T23:29:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce62ce0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009dc0 tx=0x7f3c3c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0bf6ee26d1c8d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c32ffd640 1 -- 172.21.15.89:0/4069670800 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c012860 con 0x55e8fce592e0 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c32ffd640 1 -- 172.21.15.89:0/4069670800 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c3c012a00 con 0x55e8fce592e0 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/4069670800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c440bcc80 con 0x55e8fce592e0 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c32ffd640 1 -- 172.21.15.89:0/4069670800 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c3c012c00 con 0x55e8fce592e0 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c5c5d6740 10 client.17328 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.353+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/4069670800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e8fcb9b620 con 0x55e8fce592e0 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c32ffd640 1 -- 172.21.15.89:0/4069670800 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c440bcc80 con 0x55e8fce592e0 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c32ffd640 1 -- 172.21.15.89:0/4069670800 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3c3c00f040 con 0x55e8fce592e0 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c5c5d6740 10 client.17328 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c5c5d6740 10 client.17328 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c5c5d6740 20 client.17328 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e8fce6e4e0 0x55e8fce70910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c5c5d6740 4 client.17328 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/4069670800 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8fcdc9a20 con 0x55e8fce6e4e0 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e8fce6e4e0 0x55e8fce70910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e8fce6e4e0 0x55e8fce70910 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d7eccdae97e38c server_cookie=fbde4d6a8fe3b4c6 in_seq=0 out_seq=0 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.354+0000 7f3c32ffd640 10 client.17328 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c32ffd640 1 -- 172.21.15.89:0/4069670800 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e8fcdc9a20 con 0x55e8fce6e4e0 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c32ffd640 10 client.17328 handle_command_reply: tid=0 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 client.17328 shutdown 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/4069670800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 msgr2=0x55e8fce62ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce62ce0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009dc0 tx=0x7f3c3c0117b0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/4069670800 shutdown_connections 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e8fce6e4e0 0x55e8fce70910 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 0x55e8fce69d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c32ffd640 0 client.17328 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce62ce0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/4069670800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 0x55e8fce627a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/4069670800 >> 172.21.15.89:0/4069670800 conn(0x55e8fce54010 msgr2=0x55e8fce55820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/4069670800 shutdown_connections 2024-09-15T23:29:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.355+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/4069670800 wait complete. 2024-09-15T23:29:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.356+0000 7f3c5c5d6740 20 client.17328 trim_cache size 0 max 16384 2024-09-15T23:29:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c5c5d6740 1 Processor -- start 2024-09-15T23:29:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c5c5d6740 1 -- start start 2024-09-15T23:29:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce589a0 0x55e8fce530b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce75210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 0x55e8fce75750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c541aa850 con 0x55e8fce592e0 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c541ac390 con 0x55e8fce5e370 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c541aaf20 con 0x55e8fce589a0 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c4a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce75210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c4a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce75210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60740/0 (socket says 172.21.15.89:60740) 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/1650763906 learned_addr learned my addr 172.21.15.89:0/1650763906 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c537fe640 1 --2- 172.21.15.89:0/1650763906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 0x55e8fce75750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/1650763906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce589a0 msgr2=0x55e8fce530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/1650763906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce589a0 0x55e8fce530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/1650763906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 msgr2=0x55e8fce75750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/1650763906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 0x55e8fce75750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/1650763906 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8fce60ce0 con 0x55e8fce592e0 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c537fe640 1 --2- 172.21.15.89:0/1650763906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 0x55e8fce75750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.370+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/1650763906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce75210 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009f60 tx=0x7f3c3c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cee73cd22316a4c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c497fa640 1 -- 172.21.15.89:0/1650763906 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c011cb0 con 0x55e8fce592e0 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c497fa640 1 -- 172.21.15.89:0/1650763906 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c3c004e30 con 0x55e8fce592e0 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c497fa640 1 -- 172.21.15.89:0/1650763906 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c003850 con 0x55e8fce592e0 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/1650763906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 msgr2=0x55e8fce75210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/1650763906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce75210 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009f60 tx=0x7f3c3c009f90 comp rx=0 tx=0).stop 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/1650763906 shutdown_connections 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/1650763906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce5e370 0x55e8fce75750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/1650763906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fce75210 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/1650763906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce589a0 0x55e8fce530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/1650763906 >> 172.21.15.89:0/1650763906 conn(0x55e8fce54010 msgr2=0x55e8fce529a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/1650763906 shutdown_connections 2024-09-15T23:29:43.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.371+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/1650763906 wait complete. 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c5c5d6740 1 -- start start 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c5c5d6740 1 Processor -- start 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 0x55e8fced4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fced4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c5c5d6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 0x55e8fcedbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c541abcc0 con 0x55e8fce592e0 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c541ab5f0 con 0x55e8fce589a0 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c5c5d6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c541a8830 con 0x55e8fce5e370 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c5c5d6740 10 client.0 _ll_register_callbacks cb 0x7f3c3c009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c4a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fced4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c4a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fced4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60744/0 (socket says 172.21.15.89:60744) 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/3280130126 learned_addr learned my addr 172.21.15.89:0/3280130126 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 0x55e8fced4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/3280130126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 msgr2=0x55e8fcedbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.372+0000 7f3c537fe640 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 0x55e8fcedbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 0x55e8fcedbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/3280130126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 msgr2=0x55e8fced4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 0x55e8fced4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c4a7fc640 1 -- 172.21.15.89:0/3280130126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8fcedc0e0 con 0x55e8fce592e0 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c537fe640 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 0x55e8fcedbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 0x55e8fced4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c4a7fc640 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fced4a90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009dc0 tx=0x7f3c3c009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b247488c02d9129 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c32ffd640 1 -- 172.21.15.89:0/3280130126 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0053f0 con 0x55e8fce592e0 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c32ffd640 1 -- 172.21.15.89:0/3280130126 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c3c0035f0 con 0x55e8fce592e0 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/3280130126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c3c004e30 con 0x55e8fce592e0 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c32ffd640 1 -- 172.21.15.89:0/3280130126 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c3c043050 con 0x55e8fce592e0 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c5c5d6740 10 client.17334 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/3280130126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e8fcedd360 con 0x55e8fce592e0 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c32ffd640 1 -- 172.21.15.89:0/3280130126 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c3c004e30 con 0x55e8fce592e0 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c32ffd640 1 -- 172.21.15.89:0/3280130126 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3c3c004520 con 0x55e8fce592e0 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c5c5d6740 10 client.17334 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c5c5d6740 10 client.17334 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c5c5d6740 20 client.17334 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e8fceddca0 0x55e8fcee0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c5c5d6740 4 client.17334 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.373+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/3280130126 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e8fcdc3d70 con 0x55e8fceddca0 2024-09-15T23:29:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.374+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e8fceddca0 0x55e8fcee0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:43.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.374+0000 7f3c32ffd640 10 client.17334 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:43.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.374+0000 7f3c52ffd640 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e8fceddca0 0x55e8fcee0090 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12ce76d0967b5eb6 server_cookie=74d709f25041b7b1 in_seq=0 out_seq=0 2024-09-15T23:29:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.385+0000 7f3c32ffd640 1 -- 172.21.15.89:0/3280130126 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e8fcdc3d70 con 0x55e8fceddca0 2024-09-15T23:29:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.385+0000 7f3c32ffd640 10 client.17334 handle_command_reply: tid=0 2024-09-15T23:29:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.385+0000 7f3c5c5d6740 1 client.17334 shutdown 2024-09-15T23:29:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.385+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/3280130126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 msgr2=0x55e8fced4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.385+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fced4a90 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009dc0 tx=0x7f3c3c009390 comp rx=0 tx=0).stop 2024-09-15T23:29:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/3280130126 shutdown_connections 2024-09-15T23:29:43.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e8fceddca0 0x55e8fcee0090 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e8fce5e370 0x55e8fcedbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e8fce592e0 0x55e8fced4a90 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5c5d6740 1 --2- 172.21.15.89:0/3280130126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e8fce589a0 0x55e8fced4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c32ffd640 0 client.17334 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/3280130126 >> 172.21.15.89:0/3280130126 conn(0x55e8fce54010 msgr2=0x55e8fce529a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/3280130126 shutdown_connections 2024-09-15T23:29:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5c5d6740 1 -- 172.21.15.89:0/3280130126 wait complete. 2024-09-15T23:29:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5c5d6740 20 client.17334 trim_cache size 0 max 16384 2024-09-15T23:29:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:29:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:29:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:29:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:29:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 -- 172.21.15.89:0/3465338823 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c400a32f0 msgr2=0x7f3c400a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c400a32f0 0x7f3c400a57b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c009fd0 tx=0x7f3c4c0094a0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.389 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 -- 172.21.15.89:0/3465338823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54100a50 msgr2=0x7f3c5406ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54100a50 0x7f3c5406ae70 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f3c44031610 tx=0x7f3c440043b0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 -- 172.21.15.89:0/3465338823 shutdown_connections 2024-09-15T23:29:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c400a32f0 0x7f3c400a57b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c54102600 0x7f3c5419f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c54101c40 0x7f3c5419cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 --2- 172.21.15.89:0/3465338823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c54100a50 0x7f3c5406ae70 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 -- 172.21.15.89:0/3465338823 >> 172.21.15.89:0/3465338823 conn(0x7f3c540fc1b0 msgr2=0x7f3c54104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 -- 172.21.15.89:0/3465338823 shutdown_connections 2024-09-15T23:29:43.390 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:43.386+0000 7f3c5983c640 1 -- 172.21.15.89:0/3465338823 wait complete. 2024-09-15T23:29:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:43 smithi120 ceph-mon[35250]: pgmap v521: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-15T23:29:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:43 smithi174 ceph-mon[35586]: pgmap v521: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-15T23:29:43.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:43 smithi089 ceph-mon[30933]: pgmap v521: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1023 B/s wr, 11 op/s 2024-09-15T23:29:43.631 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:29:43.883 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.385+0000 7fd063013640 1 -- 172.21.15.89:0/1376578368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102570 msgr2=0x7fd05c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.385+0000 7fd063013640 1 --2- 172.21.15.89:0/1376578368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102570 0x7fd05c1029b0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd050009ca0 tx=0x7fd05002a950 comp rx=0 tx=0).stop 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.385+0000 7fd063013640 1 -- 172.21.15.89:0/1376578368 shutdown_connections 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.385+0000 7fd063013640 1 --2- 172.21.15.89:0/1376578368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd05c102ef0 0x7fd05c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.385+0000 7fd063013640 1 --2- 172.21.15.89:0/1376578368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102570 0x7fd05c1029b0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.385+0000 7fd063013640 1 --2- 172.21.15.89:0/1376578368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd05c100e20 0x7fd05c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.385+0000 7fd063013640 1 -- 172.21.15.89:0/1376578368 >> 172.21.15.89:0/1376578368 conn(0x7fd05c0fc7d0 msgr2=0x7fd05c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.385+0000 7fd063013640 1 -- 172.21.15.89:0/1376578368 shutdown_connections 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.385+0000 7fd063013640 1 -- 172.21.15.89:0/1376578368 wait complete. 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.385+0000 7fd063013640 1 Processor -- start 2024-09-15T23:29:44.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd063013640 1 -- start start 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd063013640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd05c100e20 0x7fd05c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd063013640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102570 0x7fd05c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd063013640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd05c102ef0 0x7fd05c19cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd063013640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c1120a0 con 0x7fd05c102570 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd063013640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c111f20 con 0x7fd05c102ef0 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd063013640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c112220 con 0x7fd05c100e20 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd05bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102570 0x7fd05c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd05bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102570 0x7fd05c198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60762/0 (socket says 172.21.15.89:60762) 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd05bfff640 1 -- 172.21.15.89:0/2078108060 learned_addr learned my addr 172.21.15.89:0/2078108060 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd0615b0640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd05c102ef0 0x7fd05c19cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd060daf640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd05c100e20 0x7fd05c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd05bfff640 1 -- 172.21.15.89:0/2078108060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd05c100e20 msgr2=0x7fd05c1984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd05bfff640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd05c100e20 0x7fd05c1984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd05bfff640 1 -- 172.21.15.89:0/2078108060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd05c102ef0 msgr2=0x7fd05c19cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd05bfff640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd05c102ef0 0x7fd05c19cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd05bfff640 1 -- 172.21.15.89:0/2078108060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05c19d4a0 con 0x7fd05c102570 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd05bfff640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102570 0x7fd05c198a30 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd050005cc0 tx=0x7fd05002f650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd059ffb640 1 -- 172.21.15.89:0/2078108060 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050034040 con 0x7fd05c102570 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd059ffb640 1 -- 172.21.15.89:0/2078108060 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd050030920 con 0x7fd05c102570 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd063013640 1 -- 172.21.15.89:0/2078108060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05c19d730 con 0x7fd05c102570 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.386+0000 7fd059ffb640 1 -- 172.21.15.89:0/2078108060 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050004830 con 0x7fd05c102570 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.387+0000 7fd063013640 1 -- 172.21.15.89:0/2078108060 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd05c1a5070 con 0x7fd05c102570 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd059ffb640 1 -- 172.21.15.89:0/2078108060 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd050037020 con 0x7fd05c102570 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd0653ec740 1 Processor -- start 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd0653ec740 1 -- start start 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95380 0x559018a95760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018aa4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018aa4a90 0x559018aa6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd0653ec740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5590189fb2e0 con 0x559018a95380 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd0653ec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5590189a2ce0 con 0x559018a95ca0 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd0653ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5590189f72a0 con 0x559018aa4a90 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd0615b0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95380 0x559018a95760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.389+0000 7fd0615b0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95380 0x559018a95760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60776/0 (socket says 172.21.15.89:60776) 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd0615b0640 1 -- 172.21.15.89:0/1586992025 learned_addr learned my addr 172.21.15.89:0/1586992025 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd059ffb640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a3240 0x7fd0440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd05bfff640 1 --2- 172.21.15.89:0/1586992025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018aa4a90 0x559018aa6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd060daf640 1 --2- 172.21.15.89:0/1586992025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018aa4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd059ffb640 1 -- 172.21.15.89:0/2078108060 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd0500f0e20 con 0x7fd05c102570 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd0615b0640 1 -- 172.21.15.89:0/1586992025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018aa4a90 msgr2=0x559018aa6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd0615b0640 1 --2- 172.21.15.89:0/1586992025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018aa4a90 0x559018aa6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd0615b0640 1 -- 172.21.15.89:0/1586992025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 msgr2=0x559018aa4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd0615b0640 1 --2- 172.21.15.89:0/1586992025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018aa4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd0615b0640 1 -- 172.21.15.89:0/1586992025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559018987440 con 0x559018a95380 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd0615b0640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a3240 0x7fd0440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd0615b0640 1 --2- 172.21.15.89:0/1586992025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95380 0x559018a95760 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fd05400cc40 tx=0x7fd054010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd893971c21364af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd042ffd640 1 -- 172.21.15.89:0/1586992025 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054022070 con 0x559018a95380 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd042ffd640 1 -- 172.21.15.89:0/1586992025 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd054018070 con 0x559018a95380 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.390+0000 7fd042ffd640 1 -- 172.21.15.89:0/1586992025 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05401ca80 con 0x559018a95380 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0653ec740 1 -- 172.21.15.89:0/1586992025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95380 msgr2=0x559018a95760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0615b0640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a3240 0x7fd0440a5700 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd05401f000 tx=0x7fd054020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0653ec740 1 --2- 172.21.15.89:0/1586992025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95380 0x559018a95760 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fd05400cc40 tx=0x7fd054010740 comp rx=0 tx=0).stop 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0653ec740 1 -- 172.21.15.89:0/1586992025 shutdown_connections 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0653ec740 1 --2- 172.21.15.89:0/1586992025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018aa4a90 0x559018aa6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0653ec740 1 --2- 172.21.15.89:0/1586992025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018aa4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0653ec740 1 --2- 172.21.15.89:0/1586992025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95380 0x559018a95760 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0653ec740 1 -- 172.21.15.89:0/1586992025 >> 172.21.15.89:0/1586992025 conn(0x559018a91a30 msgr2=0x559018a91e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0653ec740 1 -- 172.21.15.89:0/1586992025 shutdown_connections 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0653ec740 1 -- 172.21.15.89:0/1586992025 wait complete. 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.391+0000 7fd0653ec740 1 -- start start 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd0653ec740 1 Processor -- start 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018aa4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95ca0 0x559018aa16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018aa4a90 0x559018aa1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd0653ec740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559018aa7660 con 0x559018a95ca0 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd0653ec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5590189b6050 con 0x559018aa4a90 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd0653ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559018aa77e0 con 0x559018a95380 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd0653ec740 10 client.0 _ll_register_callbacks cb 0x5590189f6160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:44.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd05bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95ca0 0x559018aa16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd05bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95ca0 0x559018aa16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60788/0 (socket says 172.21.15.89:60788) 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd05bfff640 1 -- 172.21.15.89:0/1702092782 learned_addr learned my addr 172.21.15.89:0/1702092782 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd060daf640 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018aa4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd0615b0640 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018aa4a90 0x559018aa1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd05bfff640 1 -- 172.21.15.89:0/1702092782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 msgr2=0x559018aa4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd05bfff640 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018aa4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd05bfff640 1 -- 172.21.15.89:0/1702092782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018aa4a90 msgr2=0x559018aa1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd05bfff640 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018aa4a90 0x559018aa1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd05bfff640 1 -- 172.21.15.89:0/1702092782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55901876c3c0 con 0x559018a95ca0 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd0615b0640 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018aa4a90 0x559018aa1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.392+0000 7fd060daf640 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018aa4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd05bfff640 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95ca0 0x559018aa16d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fd0500b99c0 tx=0x7fd050087e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e086d41be5a249a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd03bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd03bfff640 1 -- 172.21.15.89:0/1702092782 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0500bcff0 con 0x559018a95ca0 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd03bfff640 1 -- 172.21.15.89:0/1702092782 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd0500ee5e0 con 0x559018a95ca0 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd0653ec740 1 -- 172.21.15.89:0/1702092782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd054018070 con 0x559018a95ca0 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd03bfff640 1 -- 172.21.15.89:0/1702092782 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd0500c13a0 con 0x559018a95ca0 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd0653ec740 10 client.17346 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd0653ec740 1 -- 172.21.15.89:0/1702092782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559018ab3090 con 0x559018a95ca0 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd03bfff640 1 -- 172.21.15.89:0/1702092782 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd054018070 con 0x559018a95ca0 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd03bfff640 1 -- 172.21.15.89:0/1702092782 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd05004d030 con 0x559018a95ca0 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd0653ec740 10 client.17346 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd0653ec740 10 client.17346 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd0653ec740 20 client.17346 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd0653ec740 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559018ab4480 0x559018ab6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd0653ec740 4 client.17346 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:44.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.393+0000 7fd0653ec740 1 -- 172.21.15.89:0/1702092782 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559018a08850 con 0x559018ab4480 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.394+0000 7fd060daf640 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559018ab4480 0x559018ab6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.394+0000 7fd03bfff640 10 client.17346 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.394+0000 7fd060daf640 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559018ab4480 0x559018ab6870 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=873a7f348602f8ee server_cookie=db63fd55958dae64 in_seq=0 out_seq=0 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd03bfff640 1 -- 172.21.15.89:0/1702092782 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559018a08850 con 0x559018ab4480 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd03bfff640 10 client.17346 handle_command_reply: tid=0 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 client.17346 shutdown 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 -- 172.21.15.89:0/1702092782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95ca0 msgr2=0x559018aa16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95ca0 0x559018aa16d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fd0500b99c0 tx=0x7fd050087e20 comp rx=0 tx=0).stop 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 -- 172.21.15.89:0/1702092782 shutdown_connections 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559018ab4480 0x559018ab6870 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018aa4a90 0x559018aa1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018a95ca0 0x559018aa16d0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd03bfff640 0 client.17346 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:44.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 --2- 172.21.15.89:0/1702092782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018aa4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 -- 172.21.15.89:0/1702092782 >> 172.21.15.89:0/1702092782 conn(0x559018a91a30 msgr2=0x559018a97790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 -- 172.21.15.89:0/1702092782 shutdown_connections 2024-09-15T23:29:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 1 -- 172.21.15.89:0/1702092782 wait complete. 2024-09-15T23:29:44.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.395+0000 7fd0653ec740 20 client.17346 trim_cache size 0 max 16384 2024-09-15T23:29:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0653ec740 1 Processor -- start 2024-09-15T23:29:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0653ec740 1 -- start start 2024-09-15T23:29:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018ab0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018abc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018abc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0653ec740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c1acbd0 con 0x559018aa4a90 2024-09-15T23:29:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0653ec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c1ae710 con 0x559018a95ca0 2024-09-15T23:29:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0653ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c1ad2a0 con 0x559018a95380 2024-09-15T23:29:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0615b0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018abc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0615b0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018abc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60798/0 (socket says 172.21.15.89:60798) 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0615b0640 1 -- 172.21.15.89:0/3087522742 learned_addr learned my addr 172.21.15.89:0/3087522742 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd060daf640 1 --2- 172.21.15.89:0/3087522742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018ab0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd05bfff640 1 --2- 172.21.15.89:0/3087522742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018abc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.412+0000 7fd0615b0640 1 -- 172.21.15.89:0/3087522742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 msgr2=0x559018ab0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.413+0000 7fd0615b0640 1 --2- 172.21.15.89:0/3087522742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018ab0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.413+0000 7fd0615b0640 1 -- 172.21.15.89:0/3087522742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 msgr2=0x559018abc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.413+0000 7fd0615b0640 1 --2- 172.21.15.89:0/3087522742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018abc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.413+0000 7fd0615b0640 1 -- 172.21.15.89:0/3087522742 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559018aa1fc0 con 0x559018aa4a90 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.413+0000 7fd0615b0640 1 --2- 172.21.15.89:0/3087522742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018abc630 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fd054014010 tx=0x7fd05400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3779977ac1aa76c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.413+0000 7fd0427fc640 1 -- 172.21.15.89:0/3087522742 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054022040 con 0x559018aa4a90 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.413+0000 7fd0427fc640 1 -- 172.21.15.89:0/3087522742 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd054018040 con 0x559018aa4a90 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.413+0000 7fd0427fc640 1 -- 172.21.15.89:0/3087522742 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054011a90 con 0x559018aa4a90 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.413+0000 7fd0653ec740 1 -- 172.21.15.89:0/3087522742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 msgr2=0x559018abc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 --2- 172.21.15.89:0/3087522742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018abc630 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fd054014010 tx=0x7fd05400f000 comp rx=0 tx=0).stop 2024-09-15T23:29:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 -- 172.21.15.89:0/3087522742 shutdown_connections 2024-09-15T23:29:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 --2- 172.21.15.89:0/3087522742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018abc630 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 --2- 172.21.15.89:0/3087522742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018abc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 --2- 172.21.15.89:0/3087522742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018ab0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 -- 172.21.15.89:0/3087522742 >> 172.21.15.89:0/3087522742 conn(0x559018a91a30 msgr2=0x559018a9cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 -- 172.21.15.89:0/3087522742 shutdown_connections 2024-09-15T23:29:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 -- 172.21.15.89:0/3087522742 wait complete. 2024-09-15T23:29:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 -- start start 2024-09-15T23:29:44.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 Processor -- start 2024-09-15T23:29:44.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018aa8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018aa9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018b1e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c1ae040 con 0x559018aa4a90 2024-09-15T23:29:44.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c1ad970 con 0x559018a95ca0 2024-09-15T23:29:44.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c1aabb0 con 0x559018a95380 2024-09-15T23:29:44.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.414+0000 7fd0653ec740 10 client.0 _ll_register_callbacks cb 0x7fd050005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:44.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0615b0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018b1e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0615b0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018b1e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60814/0 (socket says 172.21.15.89:60814) 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0615b0640 1 -- 172.21.15.89:0/351580212 learned_addr learned my addr 172.21.15.89:0/351580212 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd060daf640 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018aa8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd05bfff640 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018aa9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0615b0640 1 -- 172.21.15.89:0/351580212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 msgr2=0x559018aa8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0615b0640 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018aa8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0615b0640 1 -- 172.21.15.89:0/351580212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 msgr2=0x559018aa9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0615b0640 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018aa9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0615b0640 1 -- 172.21.15.89:0/351580212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559018b1e600 con 0x559018aa4a90 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0615b0640 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018b1e0c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fd05400c760 tx=0x7fd054011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0b8283951f5eae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd03bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd03bfff640 1 -- 172.21.15.89:0/351580212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05400c2d0 con 0x559018aa4a90 2024-09-15T23:29:44.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd03bfff640 1 -- 172.21.15.89:0/351580212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd05401c640 con 0x559018aa4a90 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0653ec740 1 -- 172.21.15.89:0/351580212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd054018040 con 0x559018aa4a90 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd03bfff640 1 -- 172.21.15.89:0/351580212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd05401c840 con 0x559018aa4a90 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0653ec740 10 client.17352 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.415+0000 7fd0653ec740 1 -- 172.21.15.89:0/351580212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559018b1f870 con 0x559018aa4a90 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd03bfff640 1 -- 172.21.15.89:0/351580212 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd054018040 con 0x559018aa4a90 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd03bfff640 1 -- 172.21.15.89:0/351580212 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd054038950 con 0x559018aa4a90 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd0653ec740 10 client.17352 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd0653ec740 10 client.17352 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd0653ec740 20 client.17352 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd0653ec740 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559018b1fd30 0x559018b22120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd0653ec740 4 client.17352 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd0653ec740 1 -- 172.21.15.89:0/351580212 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559018a02d60 con 0x559018b1fd30 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd060daf640 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559018b1fd30 0x559018b22120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd060daf640 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559018b1fd30 0x559018b22120 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0f10d32df715a5f server_cookie=7cdb55e85461279 in_seq=0 out_seq=0 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.416+0000 7fd03bfff640 10 client.17352 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd03bfff640 1 -- 172.21.15.89:0/351580212 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559018a02d60 con 0x559018b1fd30 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd03bfff640 10 client.17352 handle_command_reply: tid=0 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 client.17352 shutdown 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 -- 172.21.15.89:0/351580212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 msgr2=0x559018b1e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018b1e0c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fd05400c760 tx=0x7fd054011eb0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 -- 172.21.15.89:0/351580212 shutdown_connections 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559018b1fd30 0x559018b22120 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd03bfff640 0 client.17352 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559018aa4a90 0x559018b1e0c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559018a95ca0 0x559018aa9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 --2- 172.21.15.89:0/351580212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559018a95380 0x559018aa8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 -- 172.21.15.89:0/351580212 >> 172.21.15.89:0/351580212 conn(0x559018a91a30 msgr2=0x559018aa2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 -- 172.21.15.89:0/351580212 shutdown_connections 2024-09-15T23:29:44.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 1 -- 172.21.15.89:0/351580212 wait complete. 2024-09-15T23:29:44.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.417+0000 7fd0653ec740 20 client.17352 trim_cache size 0 max 16384 2024-09-15T23:29:44.424 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:29:44.424 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:29:44.424 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:29:44.424 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 -- 172.21.15.89:0/2078108060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a3240 msgr2=0x7fd0440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a3240 0x7fd0440a5700 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd05401f000 tx=0x7fd054020000 comp rx=0 tx=0).stop 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 -- 172.21.15.89:0/2078108060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102570 msgr2=0x7fd05c198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102570 0x7fd05c198a30 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd050005cc0 tx=0x7fd05002f650 comp rx=0 tx=0).stop 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 -- 172.21.15.89:0/2078108060 shutdown_connections 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0440a3240 0x7fd0440a5700 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd05c102ef0 0x7fd05c19cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd05c102570 0x7fd05c198a30 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 --2- 172.21.15.89:0/2078108060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd05c100e20 0x7fd05c1984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 -- 172.21.15.89:0/2078108060 >> 172.21.15.89:0/2078108060 conn(0x7fd05c0fc7d0 msgr2=0x7fd05c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 -- 172.21.15.89:0/2078108060 shutdown_connections 2024-09-15T23:29:44.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:44.418+0000 7fd063013640 1 -- 172.21.15.89:0/2078108060 wait complete. 2024-09-15T23:29:45.189 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:29:45.189 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:29:45.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:45 smithi089 ceph-mon[30933]: pgmap v522: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:29:45.431 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:29:45.435 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:45 smithi120 ceph-mon[35250]: pgmap v522: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:29:45.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:45 smithi174 ceph-mon[35586]: pgmap v522: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:29:45.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.924+0000 7ffa05db7640 1 -- 172.21.15.89:0/1041559278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00102ef0 msgr2=0x7ffa0010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:45.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.924+0000 7ffa05db7640 1 --2- 172.21.15.89:0/1041559278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00102ef0 0x7ffa0010da40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ff9f8009ca0 tx=0x7ff9f802a950 comp rx=0 tx=0).stop 2024-09-15T23:29:45.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.924+0000 7ffa05db7640 1 -- 172.21.15.89:0/1041559278 shutdown_connections 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.924+0000 7ffa05db7640 1 --2- 172.21.15.89:0/1041559278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00102ef0 0x7ffa0010da40 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.924+0000 7ffa05db7640 1 --2- 172.21.15.89:0/1041559278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa00102570 0x7ffa001029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.924+0000 7ffa05db7640 1 --2- 172.21.15.89:0/1041559278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa00100e20 0x7ffa00101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.924+0000 7ffa05db7640 1 -- 172.21.15.89:0/1041559278 >> 172.21.15.89:0/1041559278 conn(0x7ffa000fc7d0 msgr2=0x7ffa000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.924+0000 7ffa05db7640 1 -- 172.21.15.89:0/1041559278 shutdown_connections 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.924+0000 7ffa05db7640 1 -- 172.21.15.89:0/1041559278 wait complete. 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.925+0000 7ffa05db7640 1 Processor -- start 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.925+0000 7ffa05db7640 1 -- start start 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.925+0000 7ffa05db7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa00100e20 0x7ffa00198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.925+0000 7ffa05db7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa00102570 0x7ffa00198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.925+0000 7ffa05db7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00102ef0 0x7ffa0019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.925+0000 7ffa05db7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffa00110f80 con 0x7ffa00102ef0 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.925+0000 7ffa05db7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa00110e00 con 0x7ffa00100e20 2024-09-15T23:29:45.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.925+0000 7ffa05db7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa00111100 con 0x7ffa00102570 2024-09-15T23:29:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00102ef0 0x7ffa0019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00102ef0 0x7ffa0019ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40376/0 (socket says 172.21.15.89:40376) 2024-09-15T23:29:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fffff640 1 -- 172.21.15.89:0/2850152201 learned_addr learned my addr 172.21.15.89:0/2850152201 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9ff7fe640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa00100e20 0x7ffa00198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9feffd640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa00102570 0x7ffa00198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fffff640 1 -- 172.21.15.89:0/2850152201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa00102570 msgr2=0x7ffa00198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fffff640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa00102570 0x7ffa00198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fffff640 1 -- 172.21.15.89:0/2850152201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa00100e20 msgr2=0x7ffa00198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fffff640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa00100e20 0x7ffa00198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:45.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fffff640 1 -- 172.21.15.89:0/2850152201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0019d4e0 con 0x7ffa00102ef0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9feffd640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa00102570 0x7ffa00198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fffff640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00102ef0 0x7ffa0019ce00 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ff9f802ae70 tx=0x7ff9f802f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fcff9640 1 -- 172.21.15.89:0/2850152201 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8034040 con 0x7ffa00102ef0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fcff9640 1 -- 172.21.15.89:0/2850152201 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9f8030920 con 0x7ffa00102ef0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ffa05db7640 1 -- 172.21.15.89:0/2850152201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa0019d770 con 0x7ffa00102ef0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ff9fcff9640 1 -- 172.21.15.89:0/2850152201 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8045480 con 0x7ffa00102ef0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.926+0000 7ffa05db7640 1 -- 172.21.15.89:0/2850152201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa001a50b0 con 0x7ffa00102ef0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.927+0000 7ffa05db7640 1 -- 172.21.15.89:0/2850152201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa00101200 con 0x7ffa00102ef0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.928+0000 7ff9fcff9640 1 -- 172.21.15.89:0/2850152201 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff9f8037020 con 0x7ffa00102ef0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.929+0000 7ff9fcff9640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9e80a32c0 0x7ff9e80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.929+0000 7ff9ff7fe640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9e80a32c0 0x7ff9e80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.929+0000 7ff9fcff9640 1 -- 172.21.15.89:0/2850152201 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff9f80f0e40 con 0x7ffa00102ef0 2024-09-15T23:29:45.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.930+0000 7ff9ff7fe640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9e80a32c0 0x7ff9e80a5780 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff9f00058c0 tx=0x7ff9f000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:45.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:45.933+0000 7ff9fcff9640 1 -- 172.21.15.89:0/2850152201 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff9f80811b0 con 0x7ffa00102ef0 2024-09-15T23:29:45.974 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:46.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.149+0000 7ffa05db7640 1 -- 172.21.15.89:0/2850152201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa0010a0a0 con 0x7ffa00102ef0 2024-09-15T23:29:46.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.150+0000 7ff9fcff9640 1 -- 172.21.15.89:0/2850152201 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7ff9f80bd710 con 0x7ffa00102ef0 2024-09-15T23:29:46.151 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:29:46.151 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:29:46.151 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:29:46.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 -- 172.21.15.89:0/2850152201 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9e80a32c0 msgr2=0x7ff9e80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:46.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9e80a32c0 0x7ff9e80a5780 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff9f00058c0 tx=0x7ff9f000a4c0 comp rx=0 tx=0).stop 2024-09-15T23:29:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 -- 172.21.15.89:0/2850152201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00102ef0 msgr2=0x7ffa0019ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00102ef0 0x7ffa0019ce00 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ff9f802ae70 tx=0x7ff9f802f5d0 comp rx=0 tx=0).stop 2024-09-15T23:29:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 -- 172.21.15.89:0/2850152201 shutdown_connections 2024-09-15T23:29:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff9e80a32c0 0x7ff9e80a5780 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffa00102ef0 0x7ffa0019ce00 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa00102570 0x7ffa00198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 --2- 172.21.15.89:0/2850152201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa00100e20 0x7ffa00198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 -- 172.21.15.89:0/2850152201 >> 172.21.15.89:0/2850152201 conn(0x7ffa000fc7d0 msgr2=0x7ffa0006a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.152+0000 7ffa05db7640 1 -- 172.21.15.89:0/2850152201 shutdown_connections 2024-09-15T23:29:46.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:46.153+0000 7ffa05db7640 1 -- 172.21.15.89:0/2850152201 wait complete. 2024-09-15T23:29:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:46 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2850152201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:46 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2850152201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:46.554 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:46 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2850152201' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:47.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:29:47.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.173+0000 7f79fce2f640 1 -- 172.21.15.89:0/584320021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f8100e20 msgr2=0x7f79f8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.173+0000 7f79fce2f640 1 --2- 172.21.15.89:0/584320021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f8100e20 0x7f79f8101200 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009900 tx=0x7f79e802a480 comp rx=0 tx=0).stop 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.173+0000 7f79fce2f640 1 -- 172.21.15.89:0/584320021 shutdown_connections 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.173+0000 7f79fce2f640 1 --2- 172.21.15.89:0/584320021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79f8102ef0 0x7f79f810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.173+0000 7f79fce2f640 1 --2- 172.21.15.89:0/584320021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f8102570 0x7f79f81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.173+0000 7f79fce2f640 1 --2- 172.21.15.89:0/584320021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f8100e20 0x7f79f8101200 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.173+0000 7f79fce2f640 1 -- 172.21.15.89:0/584320021 >> 172.21.15.89:0/584320021 conn(0x7f79f80fc7d0 msgr2=0x7f79f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.173+0000 7f79fce2f640 1 -- 172.21.15.89:0/584320021 shutdown_connections 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.173+0000 7f79fce2f640 1 -- 172.21.15.89:0/584320021 wait complete. 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79fce2f640 1 Processor -- start 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79fce2f640 1 -- start start 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79fce2f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79f8100e20 0x7f79f8198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:47.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79fce2f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f8102570 0x7f79f81989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79fce2f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f8102ef0 0x7f79f819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79fce2f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79f8110f50 con 0x7f79f8102570 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79fce2f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f79f8110dd0 con 0x7f79f8100e20 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79fce2f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79f81110d0 con 0x7f79f8102ef0 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79f5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f8102570 0x7f79f81989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79f5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f8102570 0x7f79f81989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40406/0 (socket says 172.21.15.89:40406) 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79f5d9b640 1 -- 172.21.15.89:0/1000507510 learned_addr learned my addr 172.21.15.89:0/1000507510 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79f6d9d640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f8102ef0 0x7f79f819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.174+0000 7f79f659c640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79f8100e20 0x7f79f8198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79f5d9b640 1 -- 172.21.15.89:0/1000507510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f8102ef0 msgr2=0x7f79f819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79f5d9b640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f8102ef0 0x7f79f819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79f5d9b640 1 -- 172.21.15.89:0/1000507510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79f8100e20 msgr2=0x7f79f8198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:47.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79f5d9b640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79f8100e20 0x7f79f8198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79f5d9b640 1 -- 172.21.15.89:0/1000507510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f819d430 con 0x7f79f8102570 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79f5d9b640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f8102570 0x7f79f81989c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00c920 tx=0x7f79ec00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79e77fe640 1 -- 172.21.15.89:0/1000507510 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec007c70 con 0x7f79f8102570 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79e77fe640 1 -- 172.21.15.89:0/1000507510 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79ec007e10 con 0x7f79f8102570 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79fce2f640 1 -- 172.21.15.89:0/1000507510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f819d720 con 0x7f79f8102570 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79e77fe640 1 -- 172.21.15.89:0/1000507510 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec018890 con 0x7f79f8102570 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.175+0000 7f79fce2f640 1 -- 172.21.15.89:0/1000507510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f81a5000 con 0x7f79f8102570 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.176+0000 7f79fce2f640 1 -- 172.21.15.89:0/1000507510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f8101200 con 0x7f79f8102570 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.177+0000 7f79e77fe640 1 -- 172.21.15.89:0/1000507510 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f79ec018a50 con 0x7f79f8102570 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.178+0000 7f79e77fe640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e00a3240 0x7f79e00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.178+0000 7f79f659c640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e00a3240 0x7f79e00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.178+0000 7f79e77fe640 1 -- 172.21.15.89:0/1000507510 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f79ec0d0070 con 0x7f79f8102570 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.179+0000 7f79f659c640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e00a3240 0x7f79e00a5700 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f79e8000c00 tx=0x7f79e80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:47.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.181+0000 7f79e77fe640 1 -- 172.21.15.89:0/1000507510 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f79ec014030 con 0x7f79f8102570 2024-09-15T23:29:47.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:47 smithi089 ceph-mon[30933]: pgmap v523: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-09-15T23:29:47.322 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:47.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.400+0000 7f79fce2f640 1 -- 172.21.15.89:0/1000507510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79f81997e0 con 0x7f79f8102570 2024-09-15T23:29:47.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.401+0000 7f79e77fe640 1 -- 172.21.15.89:0/1000507510 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f79f81997e0 con 0x7f79f8102570 2024-09-15T23:29:47.402 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:29:47.402 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:29:47.403 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:29:47.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 -- 172.21.15.89:0/1000507510 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e00a3240 msgr2=0x7f79e00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:47.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e00a3240 0x7f79e00a5700 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f79e8000c00 tx=0x7f79e80058d0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 -- 172.21.15.89:0/1000507510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f8102570 msgr2=0x7f79f81989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:47.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f8102570 0x7f79f81989c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00c920 tx=0x7f79ec00cde0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 -- 172.21.15.89:0/1000507510 shutdown_connections 2024-09-15T23:29:47.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79e00a3240 0x7f79e00a5700 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f8102ef0 0x7f79f819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f8102570 0x7f79f81989c0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 --2- 172.21.15.89:0/1000507510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79f8100e20 0x7f79f8198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:47.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 -- 172.21.15.89:0/1000507510 >> 172.21.15.89:0/1000507510 conn(0x7f79f80fc7d0 msgr2=0x7f79f806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:47.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 -- 172.21.15.89:0/1000507510 shutdown_connections 2024-09-15T23:29:47.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.403+0000 7f79fce2f640 1 -- 172.21.15.89:0/1000507510 wait complete. 2024-09-15T23:29:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:47 smithi120 ceph-mon[35250]: pgmap v523: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-09-15T23:29:47.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:47 smithi174 ceph-mon[35586]: pgmap v523: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-09-15T23:29:48.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.993+0000 7fe4acef6640 1 -- 172.21.15.89:0/1727067957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4a8103660 msgr2=0x7fe4a810bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:48.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.993+0000 7fe4acef6640 1 --2- 172.21.15.89:0/1727067957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4a8103660 0x7fe4a810bb50 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fe490009ca0 tx=0x7fe49002a950 comp rx=0 tx=0).stop 2024-09-15T23:29:48.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.994+0000 7fe4acef6640 1 -- 172.21.15.89:0/1727067957 shutdown_connections 2024-09-15T23:29:48.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.994+0000 7fe4acef6640 1 --2- 172.21.15.89:0/1727067957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a810c170 0x7fe4a810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.994+0000 7fe4acef6640 1 --2- 172.21.15.89:0/1727067957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4a8103660 0x7fe4a810bb50 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.994+0000 7fe4acef6640 1 --2- 172.21.15.89:0/1727067957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a8102d40 0x7fe4a8103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.994+0000 7fe4acef6640 1 -- 172.21.15.89:0/1727067957 >> 172.21.15.89:0/1727067957 conn(0x7fe4a80fc7a0 msgr2=0x7fe4a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.994+0000 7fe4acef6640 1 -- 172.21.15.89:0/1727067957 shutdown_connections 2024-09-15T23:29:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.994+0000 7fe4acef6640 1 -- 172.21.15.89:0/1727067957 wait complete. 2024-09-15T23:29:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.994+0000 7fe4acef6640 1 Processor -- start 2024-09-15T23:29:48.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.994+0000 7fe4acef6640 1 -- start start 2024-09-15T23:29:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4acef6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a8102d40 0x7fe4a81009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4acef6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a8103660 0x7fe4a8100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4acef6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4a810c170 0x7fe4a8101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4acef6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8112020 con 0x7fe4a810c170 2024-09-15T23:29:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4acef6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8111ea0 con 0x7fe4a8102d40 2024-09-15T23:29:48.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4acef6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81121a0 con 0x7fe4a8103660 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4a810c170 0x7fe4a8101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4a810c170 0x7fe4a8101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40422/0 (socket says 172.21.15.89:40422) 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a6d9d640 1 -- 172.21.15.89:0/3940389454 learned_addr learned my addr 172.21.15.89:0/3940389454 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a659c640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a8102d40 0x7fe4a81009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a8103660 0x7fe4a8100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a6d9d640 1 -- 172.21.15.89:0/3940389454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a8103660 msgr2=0x7fe4a8100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a6d9d640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a8103660 0x7fe4a8100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a6d9d640 1 -- 172.21.15.89:0/3940389454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a8102d40 msgr2=0x7fe4a81009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a6d9d640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a8102d40 0x7fe4a81009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a6d9d640 1 -- 172.21.15.89:0/3940389454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a81a69a0 con 0x7fe4a810c170 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4a6d9d640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4a810c170 0x7fe4a8101420 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000c970 tx=0x7fe4a000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:48.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe49f7fe640 1 -- 172.21.15.89:0/3940389454 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a000ae60 con 0x7fe4a810c170 2024-09-15T23:29:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe49f7fe640 1 -- 172.21.15.89:0/3940389454 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4a0005500 con 0x7fe4a810c170 2024-09-15T23:29:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe49f7fe640 1 -- 172.21.15.89:0/3940389454 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a000f450 con 0x7fe4a810c170 2024-09-15T23:29:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.995+0000 7fe4acef6640 1 -- 172.21.15.89:0/3940389454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a81a6c90 con 0x7fe4a810c170 2024-09-15T23:29:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.996+0000 7fe4acef6640 1 -- 172.21.15.89:0/3940389454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a81a71a0 con 0x7fe4a810c170 2024-09-15T23:29:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.998+0000 7fe49f7fe640 1 -- 172.21.15.89:0/3940389454 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe4a0004ee0 con 0x7fe4a810c170 2024-09-15T23:29:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.998+0000 7fe4af2cf740 1 Processor -- start 2024-09-15T23:29:48.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.998+0000 7fe4af2cf740 1 -- start start 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1bc370 0x55fe7b1be760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fe7b1172f0 con 0x55fe7b1b2b40 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe7b0becf0 con 0x55fe7b1bc370 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fe7b1132b0 con 0x55fe7b1b15d0 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe49f7fe640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4880a3240 0x7fe4880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40434/0 (socket says 172.21.15.89:40434) 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/2874900709 learned_addr learned my addr 172.21.15.89:0/2874900709 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:48.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a6d9d640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4880a3240 0x7fe4880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a659c640 1 --2- 172.21.15.89:0/2874900709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/2874900709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b15d0 msgr2=0x55fe7b1b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/2874900709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/2874900709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1bc370 msgr2=0x55fe7b1be760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/2874900709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1bc370 0x55fe7b1be760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/2874900709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe7b0a33a0 con 0x55fe7b1b2b40 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a659c640 1 --2- 172.21.15.89:0/2874900709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe49f7fe640 1 -- 172.21.15.89:0/3940389454 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe4a00cf890 con 0x7fe4a810c170 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/2874900709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b1090 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fe490009430 tx=0x7fe4900099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd5adb0f7a72f03 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe4a6d9d640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4880a3240 0x7fe4880a5700 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe4a006e910 tx=0x7fe4a006e940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:47.999+0000 7fe49cff9640 1 -- 172.21.15.89:0/2874900709 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490034040 con 0x55fe7b1b2b40 2024-09-15T23:29:48.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe49cff9640 1 -- 172.21.15.89:0/2874900709 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe490041070 con 0x55fe7b1b2b40 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe49cff9640 1 -- 172.21.15.89:0/2874900709 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490004270 con 0x55fe7b1b2b40 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe4af2cf740 1 -- 172.21.15.89:0/2874900709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b2b40 msgr2=0x55fe7b1b1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/2874900709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b1090 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fe490009430 tx=0x7fe4900099a0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe4af2cf740 1 -- 172.21.15.89:0/2874900709 shutdown_connections 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/2874900709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1bc370 0x55fe7b1be760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/2874900709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/2874900709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b1090 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe4af2cf740 1 -- 172.21.15.89:0/2874900709 >> 172.21.15.89:0/2874900709 conn(0x55fe7b17a310 msgr2=0x55fe7b1adc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe4af2cf740 1 -- 172.21.15.89:0/2874900709 shutdown_connections 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.000+0000 7fe4af2cf740 1 -- 172.21.15.89:0/2874900709 wait complete. 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4af2cf740 1 -- start start 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4af2cf740 1 Processor -- start 2024-09-15T23:29:48.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1bbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fe7b1bf010 con 0x55fe7b1bc370 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe7b0d2060 con 0x55fe7b1b15d0 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fe7b1bf190 con 0x55fe7b1b2b40 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4af2cf740 10 client.0 _ll_register_callbacks cb 0x55fe7ae258c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4a5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4a5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40448/0 (socket says 172.21.15.89:40448) 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/3034371438 learned_addr learned my addr 172.21.15.89:0/3034371438 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1bbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4a659c640 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/3034371438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b2b40 msgr2=0x55fe7b1b9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/3034371438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b15d0 msgr2=0x55fe7b1bbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:48.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1bbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.001+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/3034371438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe7ae883c0 con 0x55fe7b1bc370 2024-09-15T23:29:48.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b9650 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe490009620 tx=0x7fe49002fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf97e6381cba478d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:48.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe495ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:48.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe495ffb640 1 -- 172.21.15.89:0/3034371438 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490034040 con 0x55fe7b1bc370 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe495ffb640 1 -- 172.21.15.89:0/3034371438 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe490003970 con 0x55fe7b1bc370 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3034371438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe490041070 con 0x55fe7b1bc370 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe495ffb640 1 -- 172.21.15.89:0/3034371438 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe49005d050 con 0x55fe7b1bc370 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe4af2cf740 10 client.17376 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3034371438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fe7b1bf010 con 0x55fe7b1bc370 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe495ffb640 1 -- 172.21.15.89:0/3034371438 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe490041070 con 0x55fe7b1bc370 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe495ffb640 1 -- 172.21.15.89:0/3034371438 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe4900574e0 con 0x55fe7b1bc370 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe4af2cf740 10 client.17376 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe4af2cf740 10 client.17376 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe4af2cf740 20 client.17376 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55fe7b1cbc00 0x55fe7b1cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe4af2cf740 4 client.17376 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:48.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.002+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3034371438 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe7b124860 con 0x55fe7b1cbc00 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.003+0000 7fe4a659c640 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55fe7b1cbc00 0x55fe7b1cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.003+0000 7fe4a659c640 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55fe7b1cbc00 0x55fe7b1cdff0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ff2e295d6286a56a server_cookie=7ecf72d94fbe49b1 in_seq=0 out_seq=0 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.003+0000 7fe495ffb640 10 client.17376 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe495ffb640 1 -- 172.21.15.89:0/3034371438 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe7b124860 con 0x55fe7b1cbc00 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe495ffb640 10 client.17376 handle_command_reply: tid=0 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 client.17376 shutdown 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3034371438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 msgr2=0x55fe7b1b9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b9650 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe490009620 tx=0x7fe49002fe20 comp rx=0 tx=0).stop 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3034371438 shutdown_connections 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55fe7b1cbc00 0x55fe7b1cdff0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b9650 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/3034371438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1bbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe495ffb640 0 client.17376 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:29:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3034371438 >> 172.21.15.89:0/3034371438 conn(0x55fe7b17a310 msgr2=0x55fe7b1b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3034371438 shutdown_connections 2024-09-15T23:29:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.004+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3034371438 wait complete. 2024-09-15T23:29:48.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.005+0000 7fe4af2cf740 20 client.17376 trim_cache size 0 max 16384 2024-09-15T23:29:48.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.017+0000 7fe4af2cf740 1 Processor -- start 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.017+0000 7fe4af2cf740 1 -- start start 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.017+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1acc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.017+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.017+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.017+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b1000 con 0x55fe7b1bc370 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.017+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b2b40 con 0x55fe7b1b15d0 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.017+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b16d0 con 0x55fe7b1b2b40 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe4a5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe4a5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40454/0 (socket says 172.21.15.89:40454) 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/3249879503 learned_addr learned my addr 172.21.15.89:0/3249879503 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/3249879503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1acc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/3249879503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b2b40 msgr2=0x55fe7b1b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/3249879503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/3249879503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b15d0 msgr2=0x55fe7b1acc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/3249879503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1acc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/3249879503 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe7b1bc100 con 0x55fe7b1bc370 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/3249879503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b4660 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x55fe7b1b7790 tx=0x7fe490005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f619e132e1bc0051 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe497fff640 1 -- 172.21.15.89:0/3249879503 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490034040 con 0x55fe7b1bc370 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe497fff640 1 -- 172.21.15.89:0/3249879503 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe490041070 con 0x55fe7b1bc370 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.018+0000 7fe497fff640 1 -- 172.21.15.89:0/3249879503 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4900570d0 con 0x55fe7b1bc370 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3249879503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 msgr2=0x55fe7b1b4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/3249879503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b4660 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x55fe7b1b7790 tx=0x7fe490005ef0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3249879503 shutdown_connections 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/3249879503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b4660 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/3249879503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/3249879503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1acc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3249879503 >> 172.21.15.89:0/3249879503 conn(0x55fe7b17a310 msgr2=0x55fe7b1ac500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3249879503 shutdown_connections 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 -- 172.21.15.89:0/3249879503 wait complete. 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 -- start start 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 Processor -- start 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.019+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4af2cf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b2470 con 0x55fe7b1b15d0 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81b1da0 con 0x55fe7b1b2b40 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4af2cf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81aefe0 con 0x55fe7b1bc370 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4af2cf740 10 client.0 _ll_register_callbacks cb 0x7fe490031f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40460/0 (socket says 172.21.15.89:40460) 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/1915739221 learned_addr learned my addr 172.21.15.89:0/1915739221 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a659c640 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/1915739221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1bc370 msgr2=0x55fe7b1b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/1915739221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b2b40 msgr2=0x55fe7b1b65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a5d9b640 1 -- 172.21.15.89:0/1915739221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe7b235620 con 0x55fe7b1b15d0 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4a5d9b640 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b6070 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fe490009ca0 tx=0x7fe490057cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3034b363e708bba7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe495ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe495ffb640 1 -- 172.21.15.89:0/1915739221 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490034070 con 0x55fe7b1b15d0 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe495ffb640 1 -- 172.21.15.89:0/1915739221 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe490003db0 con 0x55fe7b1b15d0 2024-09-15T23:29:48.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe4af2cf740 1 -- 172.21.15.89:0/1915739221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe490041070 con 0x55fe7b1b15d0 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.020+0000 7fe495ffb640 1 -- 172.21.15.89:0/1915739221 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe49002fc00 con 0x55fe7b1b15d0 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe4af2cf740 10 client.17382 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe4af2cf740 1 -- 172.21.15.89:0/1915739221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fe7b2368e0 con 0x55fe7b1b15d0 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe495ffb640 1 -- 172.21.15.89:0/1915739221 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe490041070 con 0x55fe7b1b15d0 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe495ffb640 1 -- 172.21.15.89:0/1915739221 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe490004410 con 0x55fe7b1b15d0 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe4af2cf740 10 client.17382 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe4af2cf740 10 client.17382 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe4af2cf740 20 client.17382 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55fe7b236d90 0x55fe7b2391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe4af2cf740 4 client.17382 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe4af2cf740 1 -- 172.21.15.89:0/1915739221 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe7b11ed70 con 0x55fe7b236d90 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.021+0000 7fe4a659c640 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55fe7b236d90 0x55fe7b2391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:48.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4a659c640 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55fe7b236d90 0x55fe7b2391b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6f8066b5efae8d3a server_cookie=601f0df1ad1e4726 in_seq=0 out_seq=0 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe495ffb640 10 client.17382 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe495ffb640 1 -- 172.21.15.89:0/1915739221 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe7b11ed70 con 0x55fe7b236d90 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe495ffb640 10 client.17382 handle_command_reply: tid=0 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 client.17382 shutdown 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 -- 172.21.15.89:0/1915739221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b15d0 msgr2=0x55fe7b1b6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b6070 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fe490009ca0 tx=0x7fe490057cf0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 -- 172.21.15.89:0/1915739221 shutdown_connections 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55fe7b236d90 0x55fe7b2391b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe7b1bc370 0x55fe7b1b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe495ffb640 0 client.17382 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe7b1b2b40 0x55fe7b1b65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 --2- 172.21.15.89:0/1915739221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe7b1b15d0 0x55fe7b1b6070 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 -- 172.21.15.89:0/1915739221 >> 172.21.15.89:0/1915739221 conn(0x55fe7b17a310 msgr2=0x55fe7b1bb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:48.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 -- 172.21.15.89:0/1915739221 shutdown_connections 2024-09-15T23:29:48.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 1 -- 172.21.15.89:0/1915739221 wait complete. 2024-09-15T23:29:48.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.022+0000 7fe4af2cf740 20 client.17382 trim_cache size 0 max 16384 2024-09-15T23:29:48.038 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 -- 172.21.15.89:0/3940389454 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4880a3240 msgr2=0x7fe4880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4880a3240 0x7fe4880a5700 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe4a006e910 tx=0x7fe4a006e940 comp rx=0 tx=0).stop 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 -- 172.21.15.89:0/3940389454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4a810c170 msgr2=0x7fe4a8101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4a810c170 0x7fe4a8101420 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000c970 tx=0x7fe4a000cd70 comp rx=0 tx=0).stop 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 -- 172.21.15.89:0/3940389454 shutdown_connections 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4880a3240 0x7fe4880a5700 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe4a810c170 0x7fe4a8101420 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4a8103660 0x7fe4a8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 --2- 172.21.15.89:0/3940389454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe4a8102d40 0x7fe4a81009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 -- 172.21.15.89:0/3940389454 >> 172.21.15.89:0/3940389454 conn(0x7fe4a80fc7a0 msgr2=0x7fe4a80fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 -- 172.21.15.89:0/3940389454 shutdown_connections 2024-09-15T23:29:48.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:48.023+0000 7fe4acef6640 1 -- 172.21.15.89:0/3940389454 wait complete. 2024-09-15T23:29:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:48 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1000507510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:48.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:48 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1000507510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:48.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:48 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1000507510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:48.880 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:29:49.335 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 5e38a27a --release --await' 2024-09-15T23:29:49.335 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 5e38a27a --release --await 2024-09-15T23:29:49.459 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:49 smithi120 ceph-mon[35250]: pgmap v524: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-09-15T23:29:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:49 smithi174 ceph-mon[35586]: pgmap v524: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-09-15T23:29:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:49 smithi089 ceph-mon[30933]: pgmap v524: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-09-15T23:29:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 -- 172.21.15.89:0/2150827134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18b41082a0 msgr2=0x7f18b410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 --2- 172.21.15.89:0/2150827134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18b41082a0 0x7f18b410ec40 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009cd0 tx=0x7f18ac02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 -- 172.21.15.89:0/2150827134 shutdown_connections 2024-09-15T23:29:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 --2- 172.21.15.89:0/2150827134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18b41082a0 0x7f18b410ec40 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 --2- 172.21.15.89:0/2150827134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b41036e0 0x7f18b4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 --2- 172.21.15.89:0/2150827134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b4102d30 0x7f18b4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 -- 172.21.15.89:0/2150827134 >> 172.21.15.89:0/2150827134 conn(0x7f18b40fc7d0 msgr2=0x7f18b40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 -- 172.21.15.89:0/2150827134 shutdown_connections 2024-09-15T23:29:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 -- 172.21.15.89:0/2150827134 wait complete. 2024-09-15T23:29:49.963 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 Processor -- start 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.942+0000 7f18ba64c640 1 -- start start 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18ba64c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18b4102d30 0x7f18b4196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18ba64c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b41036e0 0x7f18b41968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18ba64c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b41082a0 0x7f18b419ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18ba64c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f18b410fe90 con 0x7f18b4102d30 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18ba64c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f18b410fd10 con 0x7f18b41082a0 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18ba64c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f18b4110010 con 0x7f18b41036e0 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18b4102d30 0x7f18b4196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18b4102d30 0x7f18b4196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40480/0 (socket says 172.21.15.89:40480) 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b3fff640 1 -- 172.21.15.89:0/1294809898 learned_addr learned my addr 172.21.15.89:0/1294809898 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18abfff640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b41036e0 0x7f18b41968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b8be9640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b41082a0 0x7f18b419ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b3fff640 1 -- 172.21.15.89:0/1294809898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b41036e0 msgr2=0x7f18b41968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b3fff640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b41036e0 0x7f18b41968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b3fff640 1 -- 172.21.15.89:0/1294809898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b41082a0 msgr2=0x7f18b419ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b3fff640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b41082a0 0x7f18b419ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b3fff640 1 -- 172.21.15.89:0/1294809898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b419b320 con 0x7f18b4102d30 2024-09-15T23:29:49.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b3fff640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18b4102d30 0x7f18b4196370 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f18a4011b20 tx=0x7f18a400f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b1ffb640 1 -- 172.21.15.89:0/1294809898 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a401e070 con 0x7f18b4102d30 2024-09-15T23:29:49.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.943+0000 7f18b1ffb640 1 -- 172.21.15.89:0/1294809898 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a400fc90 con 0x7f18b4102d30 2024-09-15T23:29:49.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.944+0000 7f18ba64c640 1 -- 172.21.15.89:0/1294809898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b419b610 con 0x7f18b4102d30 2024-09-15T23:29:49.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.944+0000 7f18b1ffb640 1 -- 172.21.15.89:0/1294809898 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4018780 con 0x7f18b4102d30 2024-09-15T23:29:49.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.944+0000 7f18ba64c640 1 -- 172.21.15.89:0/1294809898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b41a2ef0 con 0x7f18b4102d30 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.946+0000 7f18b1ffb640 1 -- 172.21.15.89:0/1294809898 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f18a4021070 con 0x7f18b4102d30 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.946+0000 7f18bca25740 1 Processor -- start 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.946+0000 7f18bca25740 1 -- start start 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.946+0000 7f18bca25740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 0x557af605b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.946+0000 7f18bca25740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.946+0000 7f18bca25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af6066370 0x557af6068760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.946+0000 7f18bca25740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557af5fc12f0 con 0x557af605b5d0 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.946+0000 7f18bca25740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557af5f68cf0 con 0x557af605cb40 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.946+0000 7f18bca25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557af5fbd2b0 con 0x557af6066370 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b1ffb640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18a00a32c0 0x7f18a00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40496/0 (socket says 172.21.15.89:40496) 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b3fff640 1 -- 172.21.15.89:0/1075609342 learned_addr learned my addr 172.21.15.89:0/1075609342 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b8be9640 1 --2- 172.21.15.89:0/1075609342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 0x557af605b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18abfff640 1 --2- 172.21.15.89:0/1075609342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af6066370 0x557af6068760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b8be9640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18a00a32c0 0x7f18a00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b1ffb640 1 -- 172.21.15.89:0/1294809898 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f18a40d3300 con 0x7f18b4102d30 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b3fff640 1 -- 172.21.15.89:0/1075609342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af6066370 msgr2=0x557af6068760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b3fff640 1 --2- 172.21.15.89:0/1075609342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af6066370 0x557af6068760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b3fff640 1 -- 172.21.15.89:0/1075609342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 msgr2=0x557af605b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b3fff640 1 --2- 172.21.15.89:0/1075609342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 0x557af605b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b3fff640 1 -- 172.21.15.89:0/1075609342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557af5f4d3a0 con 0x557af605b5d0 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b3fff640 1 --2- 172.21.15.89:0/1075609342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605ba10 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f18a40d0950 tx=0x7f18a4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190e0c916e7e71e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18b8be9640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18a00a32c0 0x7f18a00a5780 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f18ac004ab0 tx=0x7f18ac004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18aa7fc640 1 -- 172.21.15.89:0/1075609342 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a40bf070 con 0x557af605b5d0 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18aa7fc640 1 -- 172.21.15.89:0/1075609342 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a40d3120 con 0x557af605b5d0 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.947+0000 7f18aa7fc640 1 -- 172.21.15.89:0/1075609342 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4019bd0 con 0x557af605b5d0 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.948+0000 7f18bca25740 1 -- 172.21.15.89:0/1075609342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 msgr2=0x557af605ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.948+0000 7f18bca25740 1 --2- 172.21.15.89:0/1075609342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605ba10 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f18a40d0950 tx=0x7f18a4000bc0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.948+0000 7f18bca25740 1 -- 172.21.15.89:0/1075609342 shutdown_connections 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.948+0000 7f18bca25740 1 --2- 172.21.15.89:0/1075609342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af6066370 0x557af6068760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.948+0000 7f18bca25740 1 --2- 172.21.15.89:0/1075609342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605ba10 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.948+0000 7f18bca25740 1 --2- 172.21.15.89:0/1075609342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 0x557af605b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.948+0000 7f18bca25740 1 -- 172.21.15.89:0/1075609342 >> 172.21.15.89:0/1075609342 conn(0x557af6024310 msgr2=0x557af6057c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.948+0000 7f18bca25740 1 -- 172.21.15.89:0/1075609342 shutdown_connections 2024-09-15T23:29:49.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.948+0000 7f18bca25740 1 -- 172.21.15.89:0/1075609342 wait complete. 2024-09-15T23:29:49.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.948+0000 7f18bca25740 1 -- start start 2024-09-15T23:29:49.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18bca25740 1 Processor -- start 2024-09-15T23:29:49.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18bca25740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af6065d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18bca25740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 0x557af6062ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18bca25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af6066370 0x557af60635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18bca25740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557af6068fb0 con 0x557af605b5d0 2024-09-15T23:29:49.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18bca25740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557af5f7c060 con 0x557af605cb40 2024-09-15T23:29:49.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18bca25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557af6069130 con 0x557af6066370 2024-09-15T23:29:49.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18bca25740 10 client.0 _ll_register_callbacks cb 0x557af5ccf8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:49.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af6065d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af6065d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40506/0 (socket says 172.21.15.89:40506) 2024-09-15T23:29:49.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18b3fff640 1 -- 172.21.15.89:0/4046429896 learned_addr learned my addr 172.21.15.89:0/4046429896 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:49.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18b8be9640 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af6066370 0x557af60635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18abfff640 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 0x557af6062ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18b3fff640 1 -- 172.21.15.89:0/4046429896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af6066370 msgr2=0x557af60635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18b3fff640 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af6066370 0x557af60635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18b3fff640 1 -- 172.21.15.89:0/4046429896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 msgr2=0x557af6062ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18b3fff640 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 0x557af6062ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18b3fff640 1 -- 172.21.15.89:0/4046429896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557af5d323c0 con 0x557af605b5d0 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18abfff640 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 0x557af6062ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18b3fff640 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af6065d40 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f18a40d36b0 tx=0x7f18a409f720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1296904aeb5c6b66 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.949+0000 7f18937fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18937fe640 1 -- 172.21.15.89:0/4046429896 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a40bf040 con 0x557af605b5d0 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18937fe640 1 -- 172.21.15.89:0/4046429896 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f18a40a7d10 con 0x557af605b5d0 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18bca25740 1 -- 172.21.15.89:0/4046429896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18a40d3120 con 0x557af605b5d0 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18937fe640 1 -- 172.21.15.89:0/4046429896 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f18a40a7f30 con 0x557af605b5d0 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18bca25740 10 client.17394 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18bca25740 1 -- 172.21.15.89:0/4046429896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557af6074a20 con 0x557af605b5d0 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18937fe640 1 -- 172.21.15.89:0/4046429896 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a40d3120 con 0x557af605b5d0 2024-09-15T23:29:49.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18937fe640 1 -- 172.21.15.89:0/4046429896 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f18a409e4a0 con 0x557af605b5d0 2024-09-15T23:29:49.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18bca25740 10 client.17394 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:49.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18bca25740 10 client.17394 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:29:49.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18bca25740 20 client.17394 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:49.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18bca25740 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557af6075d70 0x557af6078160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18bca25740 4 client.17394 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:49.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18bca25740 1 -- 172.21.15.89:0/4046429896 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557af5fce860 con 0x557af6075d70 2024-09-15T23:29:49.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.950+0000 7f18abfff640 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557af6075d70 0x557af6078160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.951+0000 7f18937fe640 10 client.17394 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:29:49.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.951+0000 7f18abfff640 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557af6075d70 0x557af6078160 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c2ebb4a375953a31 server_cookie=c9ae4c697d7f725c in_seq=0 out_seq=0 2024-09-15T23:29:49.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.951+0000 7f18937fe640 1 -- 172.21.15.89:0/4046429896 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557af5fce860 con 0x557af6075d70 2024-09-15T23:29:49.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.951+0000 7f18937fe640 10 client.17394 handle_command_reply: tid=0 2024-09-15T23:29:49.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 client.17394 shutdown 2024-09-15T23:29:49.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 -- 172.21.15.89:0/4046429896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 msgr2=0x557af6065d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af6065d40 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f18a40d36b0 tx=0x7f18a409f720 comp rx=0 tx=0).stop 2024-09-15T23:29:49.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 -- 172.21.15.89:0/4046429896 shutdown_connections 2024-09-15T23:29:49.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557af6075d70 0x557af6078160 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af6066370 0x557af60635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605cb40 0x557af6062ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18937fe640 0 client.17394 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:29:49.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 --2- 172.21.15.89:0/4046429896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af6065d40 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 -- 172.21.15.89:0/4046429896 >> 172.21.15.89:0/4046429896 conn(0x557af6024310 msgr2=0x557af605d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 -- 172.21.15.89:0/4046429896 shutdown_connections 2024-09-15T23:29:49.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 1 -- 172.21.15.89:0/4046429896 wait complete. 2024-09-15T23:29:49.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.952+0000 7f18bca25740 20 client.17394 trim_cache size 0 max 16384 2024-09-15T23:29:49.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18bca25740 1 Processor -- start 2024-09-15T23:29:49.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18bca25740 1 -- start start 2024-09-15T23:29:49.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18bca25740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18bca25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af605cb40 0x557af60628d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18bca25740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af6066370 0x557af607b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18bca25740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41acd20 con 0x557af605b5d0 2024-09-15T23:29:49.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18bca25740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41ae860 con 0x557af6066370 2024-09-15T23:29:49.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18bca25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41ad3f0 con 0x557af605cb40 2024-09-15T23:29:49.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18b3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40518/0 (socket says 172.21.15.89:40518) 2024-09-15T23:29:49.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18b3fff640 1 -- 172.21.15.89:0/37489782 learned_addr learned my addr 172.21.15.89:0/37489782 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:49.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18abfff640 1 --2- 172.21.15.89:0/37489782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af605cb40 0x557af60628d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.980 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18b3fff640 1 -- 172.21.15.89:0/37489782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af605cb40 msgr2=0x557af60628d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18b3fff640 1 --2- 172.21.15.89:0/37489782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af605cb40 0x557af60628d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18b3fff640 1 -- 172.21.15.89:0/37489782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af6066370 msgr2=0x557af607b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18b3fff640 1 --2- 172.21.15.89:0/37489782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af6066370 0x557af607b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18b3fff640 1 -- 172.21.15.89:0/37489782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557af6057de0 con 0x557af605b5d0 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.965+0000 7f18b3fff640 1 --2- 172.21.15.89:0/37489782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605c410 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f18a409f570 tx=0x7f18a40a80d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399dbe2bdfea3b1e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18a9ffb640 1 -- 172.21.15.89:0/37489782 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a40bf040 con 0x557af605b5d0 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18a9ffb640 1 -- 172.21.15.89:0/37489782 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a40d3120 con 0x557af605b5d0 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18a9ffb640 1 -- 172.21.15.89:0/37489782 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a409ea30 con 0x557af605b5d0 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18bca25740 1 -- 172.21.15.89:0/37489782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 msgr2=0x557af605c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18bca25740 1 --2- 172.21.15.89:0/37489782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605c410 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f18a409f570 tx=0x7f18a40a80d0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18bca25740 1 -- 172.21.15.89:0/37489782 shutdown_connections 2024-09-15T23:29:49.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18bca25740 1 --2- 172.21.15.89:0/37489782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af6066370 0x557af607b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18bca25740 1 --2- 172.21.15.89:0/37489782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af605cb40 0x557af60628d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18bca25740 1 --2- 172.21.15.89:0/37489782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af605b5d0 0x557af605c410 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18bca25740 1 -- 172.21.15.89:0/37489782 >> 172.21.15.89:0/37489782 conn(0x557af6024310 msgr2=0x557af6065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18bca25740 1 -- 172.21.15.89:0/37489782 shutdown_connections 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.966+0000 7f18bca25740 1 -- 172.21.15.89:0/37489782 wait complete. 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18bca25740 1 -- start start 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18bca25740 1 Processor -- start 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18bca25740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605b5d0 0x557af607b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18bca25740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af605cb40 0x557af6078b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18bca25740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af6066370 0x557af60790c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18bca25740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41ae190 con 0x557af6066370 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18bca25740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41adac0 con 0x557af605b5d0 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18bca25740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f18b41aad00 con 0x557af605cb40 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18bca25740 10 client.0 _ll_register_callbacks cb 0x7f189c00b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18b8be9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af6066370 0x557af60790c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18b8be9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af6066370 0x557af60790c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40526/0 (socket says 172.21.15.89:40526) 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.967+0000 7f18b8be9640 1 -- 172.21.15.89:0/62502403 learned_addr learned my addr 172.21.15.89:0/62502403 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18abfff640 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af605cb40 0x557af6078b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18b3fff640 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605b5d0 0x557af607b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18b8be9640 1 -- 172.21.15.89:0/62502403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af605cb40 msgr2=0x557af6078b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18b8be9640 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af605cb40 0x557af6078b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18b8be9640 1 -- 172.21.15.89:0/62502403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605b5d0 msgr2=0x557af607b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18b8be9640 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605b5d0 0x557af607b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18b8be9640 1 -- 172.21.15.89:0/62502403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557af60798f0 con 0x557af6066370 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18b8be9640 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af6066370 0x557af60790c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009cd0 tx=0x7f18ac0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265dd0fd9910cc6e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18937fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18937fe640 1 -- 172.21.15.89:0/62502403 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac044040 con 0x557af6066370 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18bca25740 1 -- 172.21.15.89:0/62502403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18a40d3120 con 0x557af6066370 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18937fe640 1 -- 172.21.15.89:0/62502403 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f18ac031430 con 0x557af6066370 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18937fe640 1 -- 172.21.15.89:0/62502403 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f18ac05e050 con 0x557af6066370 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18bca25740 10 client.17400 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18bca25740 1 -- 172.21.15.89:0/62502403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557af60e0670 con 0x557af6066370 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.968+0000 7f18937fe640 1 -- 172.21.15.89:0/62502403 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a40d3120 con 0x557af6066370 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.969+0000 7f18937fe640 1 -- 172.21.15.89:0/62502403 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f18ac059030 con 0x557af6066370 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.969+0000 7f18bca25740 10 client.17400 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:49.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.969+0000 7f18bca25740 10 client.17400 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.969+0000 7f18bca25740 20 client.17400 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.969+0000 7f18bca25740 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557af60e0d90 0x557af60e3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.969+0000 7f18bca25740 4 client.17400 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.969+0000 7f18bca25740 1 -- 172.21.15.89:0/62502403 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557af5fc8d70 con 0x557af60e0d90 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.969+0000 7f18b3fff640 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557af60e0d90 0x557af60e3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.969+0000 7f18937fe640 10 client.17400 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.969+0000 7f18b3fff640 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557af60e0d90 0x557af60e3180 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=99905a1d41d98e17 server_cookie=3b7aa88675b24ab0 in_seq=0 out_seq=0 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18937fe640 1 -- 172.21.15.89:0/62502403 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557af5fc8d70 con 0x557af60e0d90 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18937fe640 10 client.17400 handle_command_reply: tid=0 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 client.17400 shutdown 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 -- 172.21.15.89:0/62502403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af6066370 msgr2=0x557af60790c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af6066370 0x557af60790c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009cd0 tx=0x7f18ac0093f0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 -- 172.21.15.89:0/62502403 shutdown_connections 2024-09-15T23:29:49.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557af60e0d90 0x557af60e3180 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557af6066370 0x557af60790c0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557af605cb40 0x557af6078b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18937fe640 0 client.17400 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:29:49.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 --2- 172.21.15.89:0/62502403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557af605b5d0 0x557af607b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 -- 172.21.15.89:0/62502403 >> 172.21.15.89:0/62502403 conn(0x557af6024310 msgr2=0x557af605d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 -- 172.21.15.89:0/62502403 shutdown_connections 2024-09-15T23:29:49.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 1 -- 172.21.15.89:0/62502403 wait complete. 2024-09-15T23:29:49.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18bca25740 20 client.17400 trim_cache size 0 max 16384 2024-09-15T23:29:49.987 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:29:49.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18ba64c640 1 -- 172.21.15.89:0/1294809898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18a00a32c0 msgr2=0x7f18a00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18ba64c640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18a00a32c0 0x7f18a00a5780 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f18ac004ab0 tx=0x7f18ac004a40 comp rx=0 tx=0).stop 2024-09-15T23:29:49.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18ba64c640 1 -- 172.21.15.89:0/1294809898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18b4102d30 msgr2=0x7f18b4196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.970+0000 7f18ba64c640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18b4102d30 0x7f18b4196370 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f18a4011b20 tx=0x7f18a400f050 comp rx=0 tx=0).stop 2024-09-15T23:29:49.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.971+0000 7f18ba64c640 1 -- 172.21.15.89:0/1294809898 shutdown_connections 2024-09-15T23:29:49.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.971+0000 7f18ba64c640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18a00a32c0 0x7f18a00a5780 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.971+0000 7f18ba64c640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18b41082a0 0x7f18b419ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.971+0000 7f18ba64c640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b41036e0 0x7f18b41968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.971+0000 7f18ba64c640 1 --2- 172.21.15.89:0/1294809898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f18b4102d30 0x7f18b4196370 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.971+0000 7f18ba64c640 1 -- 172.21.15.89:0/1294809898 >> 172.21.15.89:0/1294809898 conn(0x7f18b40fc7d0 msgr2=0x7f18b4106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.971+0000 7f18ba64c640 1 -- 172.21.15.89:0/1294809898 shutdown_connections 2024-09-15T23:29:49.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:49.971+0000 7f18ba64c640 1 -- 172.21.15.89:0/1294809898 wait complete. 2024-09-15T23:29:50.897 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:29:50.981 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:51.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.424+0000 7fef26589640 1 -- 172.21.15.89:0/2220619152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef20101de0 msgr2=0x7fef201021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:51.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.424+0000 7fef26589640 1 --2- 172.21.15.89:0/2220619152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef20101de0 0x7fef201021c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fef10009900 tx=0x7fef1002a410 comp rx=0 tx=0).stop 2024-09-15T23:29:51.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.424+0000 7fef26589640 1 -- 172.21.15.89:0/2220619152 shutdown_connections 2024-09-15T23:29:51.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.424+0000 7fef26589640 1 --2- 172.21.15.89:0/2220619152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef20106f10 0x7fef2010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.424+0000 7fef26589640 1 --2- 172.21.15.89:0/2220619152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef20102790 0x7fef201067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.424+0000 7fef26589640 1 --2- 172.21.15.89:0/2220619152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef20101de0 0x7fef201021c0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.424+0000 7fef26589640 1 -- 172.21.15.89:0/2220619152 >> 172.21.15.89:0/2220619152 conn(0x7fef200fd8c0 msgr2=0x7fef200ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:51.441 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.424+0000 7fef26589640 1 -- 172.21.15.89:0/2220619152 shutdown_connections 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.424+0000 7fef26589640 1 -- 172.21.15.89:0/2220619152 wait complete. 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef26589640 1 Processor -- start 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef26589640 1 -- start start 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef26589640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef20101de0 0x7fef20198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef26589640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef20102790 0x7fef20198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef26589640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef20106f10 0x7fef2019cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef26589640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fef20111f30 con 0x7fef20106f10 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef26589640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fef20111db0 con 0x7fef20101de0 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef26589640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef201120b0 con 0x7fef20102790 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef24b26640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef20106f10 0x7fef2019cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef24b26640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef20106f10 0x7fef2019cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40548/0 (socket says 172.21.15.89:40548) 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef24b26640 1 -- 172.21.15.89:0/1186711270 learned_addr learned my addr 172.21.15.89:0/1186711270 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:51.442 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef1ffff640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef20101de0 0x7fef20198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.425+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef20102790 0x7fef20198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef24b26640 1 -- 172.21.15.89:0/1186711270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef20102790 msgr2=0x7fef20198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef24b26640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef20102790 0x7fef20198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef24b26640 1 -- 172.21.15.89:0/1186711270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef20101de0 msgr2=0x7fef20198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef24b26640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef20101de0 0x7fef20198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef24b26640 1 -- 172.21.15.89:0/1186711270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef2019d2c0 con 0x7fef20106f10 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef20102790 0x7fef20198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef24b26640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef20106f10 0x7fef2019cbe0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fef1800cce0 tx=0x7fef1800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef1d7fa640 1 -- 172.21.15.89:0/1186711270 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1800f450 con 0x7fef20106f10 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef26589640 1 -- 172.21.15.89:0/1186711270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef2019d5b0 con 0x7fef20106f10 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef1d7fa640 1 -- 172.21.15.89:0/1186711270 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef1800fad0 con 0x7fef20106f10 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef1d7fa640 1 -- 172.21.15.89:0/1186711270 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef18018c10 con 0x7fef20106f10 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.426+0000 7fef26589640 1 -- 172.21.15.89:0/1186711270 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef201a4e90 con 0x7fef20106f10 2024-09-15T23:29:51.443 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.428+0000 7fef1d7fa640 1 -- 172.21.15.89:0/1186711270 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fef1800f5f0 con 0x7fef20106f10 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef28962740 1 Processor -- start 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef28962740 1 -- start start 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1d7fa640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 0x7fef080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef28962740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9060 0x55faeb0a9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef28962740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9980 0x55faeb0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef28962740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0b6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef28962740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55faeb00e620 con 0x55faeb0a9060 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef28962740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55faeafb5e50 con 0x55faeb0b4240 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef28962740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55faeb00a5e0 con 0x55faeb0a9980 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 0x7fef080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9060 0x55faeb0a9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9060 0x55faeb0a9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40550/0 (socket says 172.21.15.89:40550) 2024-09-15T23:29:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1ffff640 1 -- 172.21.15.89:0/3227258102 learned_addr learned my addr 172.21.15.89:0/3227258102 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1d7fa640 1 -- 172.21.15.89:0/1186711270 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fef180d03f0 con 0x7fef20106f10 2024-09-15T23:29:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef24b26640 1 --2- 172.21.15.89:0/3227258102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/3227258102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9980 0x55faeb0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1ffff640 1 -- 172.21.15.89:0/3227258102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9980 msgr2=0x55faeb0b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1ffff640 1 --2- 172.21.15.89:0/3227258102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9980 0x55faeb0b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1ffff640 1 -- 172.21.15.89:0/3227258102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 msgr2=0x55faeb0b6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1ffff640 1 --2- 172.21.15.89:0/3227258102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0b6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1ffff640 1 -- 172.21.15.89:0/3227258102 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faeafa6d40 con 0x55faeb0a9060 2024-09-15T23:29:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 0x7fef080a5700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fef200fe0c0 tx=0x7fef14014840 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.429+0000 7fef1ffff640 1 --2- 172.21.15.89:0/3227258102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9060 0x55faeb0a9440 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fef10005ab0 tx=0x7fef10031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ccb93c5bf784e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef067fc640 1 -- 172.21.15.89:0/3227258102 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1003f070 con 0x55faeb0a9060 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef067fc640 1 -- 172.21.15.89:0/3227258102 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef10031a20 con 0x55faeb0a9060 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef067fc640 1 -- 172.21.15.89:0/3227258102 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10035070 con 0x55faeb0a9060 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef28962740 1 -- 172.21.15.89:0/3227258102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9060 msgr2=0x55faeb0a9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef28962740 1 --2- 172.21.15.89:0/3227258102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9060 0x55faeb0a9440 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fef10005ab0 tx=0x7fef10031630 comp rx=0 tx=0).stop 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef28962740 1 -- 172.21.15.89:0/3227258102 shutdown_connections 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef28962740 1 --2- 172.21.15.89:0/3227258102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0b6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef28962740 1 --2- 172.21.15.89:0/3227258102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9980 0x55faeb0b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef28962740 1 --2- 172.21.15.89:0/3227258102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9060 0x55faeb0a9440 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef28962740 1 -- 172.21.15.89:0/3227258102 >> 172.21.15.89:0/3227258102 conn(0x55faeb0a55f0 msgr2=0x55faeb0a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef28962740 1 -- 172.21.15.89:0/3227258102 shutdown_connections 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.430+0000 7fef28962740 1 -- 172.21.15.89:0/3227258102 wait complete. 2024-09-15T23:29:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef28962740 1 -- start start 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef28962740 1 Processor -- start 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef28962740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0b1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef28962740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef28962740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef28962740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55faeb0b6e10 con 0x55faeb0a9980 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef28962740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55faeaf96f10 con 0x55faeb0b4240 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef28962740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55faeb0b6f90 con 0x55faeb0a9060 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef28962740 10 client.0 _ll_register_callbacks cb 0x55faeafbadb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef1f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef1f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40556/0 (socket says 172.21.15.89:40556) 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef1f7fe640 1 -- 172.21.15.89:0/1361851619 learned_addr learned my addr 172.21.15.89:0/1361851619 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.431+0000 7fef24b26640 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef1ffff640 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef1f7fe640 1 -- 172.21.15.89:0/1361851619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 msgr2=0x55faeb0b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0b1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef1f7fe640 1 -- 172.21.15.89:0/1361851619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 msgr2=0x55faeb0b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef1f7fe640 1 -- 172.21.15.89:0/1361851619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faead1d580 con 0x55faeb0a9980 2024-09-15T23:29:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b1580 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fef14011ad0 tx=0x7fef14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e948d1d63270eb49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7feeff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7feeff7fe640 1 -- 172.21.15.89:0/1361851619 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14023070 con 0x55faeb0a9980 2024-09-15T23:29:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7feeff7fe640 1 -- 172.21.15.89:0/1361851619 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fef1401f3f0 con 0x55faeb0a9980 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef28962740 1 -- 172.21.15.89:0/1361851619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef10031a20 con 0x55faeb0a9980 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7feeff7fe640 1 -- 172.21.15.89:0/1361851619 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fef1402f050 con 0x55faeb0a9980 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef28962740 10 client.17412 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef28962740 1 -- 172.21.15.89:0/1361851619 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55faeade7960 con 0x55faeb0a9980 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7feeff7fe640 1 -- 172.21.15.89:0/1361851619 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef10031a20 con 0x55faeb0a9980 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7feeff7fe640 1 -- 172.21.15.89:0/1361851619 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fef1401e070 con 0x55faeb0a9980 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef28962740 10 client.17412 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.432+0000 7fef28962740 10 client.17412 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.433+0000 7fef28962740 20 client.17412 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.433+0000 7fef28962740 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55faeb0c3c70 0x55faeb0c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.433+0000 7fef28962740 4 client.17412 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.433+0000 7fef28962740 1 -- 172.21.15.89:0/1361851619 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55faeb01bb90 con 0x55faeb0c3c70 2024-09-15T23:29:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.433+0000 7fef1ffff640 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55faeb0c3c70 0x55faeb0c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.433+0000 7fef1ffff640 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55faeb0c3c70 0x55faeb0c60d0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=621c290545231753 server_cookie=20090c3063daba08 in_seq=0 out_seq=0 2024-09-15T23:29:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.433+0000 7feeff7fe640 10 client.17412 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7feeff7fe640 1 -- 172.21.15.89:0/1361851619 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55faeb01bb90 con 0x55faeb0c3c70 2024-09-15T23:29:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7feeff7fe640 10 client.17412 handle_command_reply: tid=0 2024-09-15T23:29:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 client.17412 shutdown 2024-09-15T23:29:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 -- 172.21.15.89:0/1361851619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 msgr2=0x55faeb0b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b1580 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fef14011ad0 tx=0x7fef14004730 comp rx=0 tx=0).stop 2024-09-15T23:29:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 -- 172.21.15.89:0/1361851619 shutdown_connections 2024-09-15T23:29:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55faeb0c3c70 0x55faeb0c60d0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b1580 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7feeff7fe640 0 client.17412 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 --2- 172.21.15.89:0/1361851619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 -- 172.21.15.89:0/1361851619 >> 172.21.15.89:0/1361851619 conn(0x55faeb0a55f0 msgr2=0x55faeb0a4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 -- 172.21.15.89:0/1361851619 shutdown_connections 2024-09-15T23:29:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 1 -- 172.21.15.89:0/1361851619 wait complete. 2024-09-15T23:29:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.434+0000 7fef28962740 20 client.17412 trim_cache size 0 max 16384 2024-09-15T23:29:51.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.452+0000 7fef28962740 1 Processor -- start 2024-09-15T23:29:51.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.452+0000 7fef28962740 1 -- start start 2024-09-15T23:29:51.455 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:51 smithi089 ceph-mon[30933]: pgmap v525: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:29:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.452+0000 7fef28962740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0abac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.452+0000 7fef28962740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0a4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.452+0000 7fef28962740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef28962740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fef201aec90 con 0x55faeb0a9980 2024-09-15T23:29:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef28962740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fef201b07d0 con 0x55faeb0b4240 2024-09-15T23:29:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0abac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef28962740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef201af360 con 0x55faeb0a9060 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0abac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:46520/0 (socket says 172.21.15.89:46520) 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40562/0 (socket says 172.21.15.89:40562) 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1ffff640 1 -- 172.21.15.89:0/1718565479 learned_addr learned my addr 172.21.15.89:0/1718565479 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef24b26640 1 --2- 172.21.15.89:0/1718565479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1f7fe640 1 -- 172.21.15.89:0/1718565479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 msgr2=0x55faeb0abac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/1718565479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0abac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1f7fe640 1 -- 172.21.15.89:0/1718565479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 msgr2=0x55faeb0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/1718565479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1f7fe640 1 -- 172.21.15.89:0/1718565479 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faeb0c6750 con 0x55faeb0a9980 2024-09-15T23:29:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef24b26640 1 --2- 172.21.15.89:0/1718565479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.453+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/1718565479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0a4640 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fef1401e6e0 tx=0x7fef14004660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a58cac68d6e8d8e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.454+0000 7fef05ffb640 1 -- 172.21.15.89:0/1718565479 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14038070 con 0x55faeb0a9980 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.454+0000 7fef05ffb640 1 -- 172.21.15.89:0/1718565479 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef14012820 con 0x55faeb0a9980 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.454+0000 7fef05ffb640 1 -- 172.21.15.89:0/1718565479 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14023040 con 0x55faeb0a9980 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 -- 172.21.15.89:0/1718565479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 msgr2=0x55faeb0a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 --2- 172.21.15.89:0/1718565479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0a4640 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fef1401e6e0 tx=0x7fef14004660 comp rx=0 tx=0).stop 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 -- 172.21.15.89:0/1718565479 shutdown_connections 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 --2- 172.21.15.89:0/1718565479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 --2- 172.21.15.89:0/1718565479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0a4640 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 --2- 172.21.15.89:0/1718565479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0abac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 -- 172.21.15.89:0/1718565479 >> 172.21.15.89:0/1718565479 conn(0x55faeb0a55f0 msgr2=0x55faeb0a3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 -- 172.21.15.89:0/1718565479 shutdown_connections 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 -- 172.21.15.89:0/1718565479 wait complete. 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 -- start start 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.455+0000 7fef28962740 1 Processor -- start 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef28962740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0b8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef28962740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef28962740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb12d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef28962740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fef201b0100 con 0x55faeb0a9980 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef28962740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fef201afa30 con 0x55faeb0b4240 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef28962740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef201acc70 con 0x55faeb0a9060 2024-09-15T23:29:51.461 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef28962740 10 client.0 _ll_register_callbacks cb 0x7fef14011780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:51.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef1f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef1f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40570/0 (socket says 172.21.15.89:40570) 2024-09-15T23:29:51.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef1f7fe640 1 -- 172.21.15.89:0/2590382061 learned_addr learned my addr 172.21.15.89:0/2590382061 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:51.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef24b26640 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb12d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef1f7fe640 1 -- 172.21.15.89:0/2590382061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 msgr2=0x55faeb0b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef1f7fe640 1 -- 172.21.15.89:0/2590382061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 msgr2=0x55faeb12d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb12d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.456+0000 7fef1f7fe640 1 -- 172.21.15.89:0/2590382061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faeb12deb0 con 0x55faeb0a9980 2024-09-15T23:29:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.457+0000 7fef1f7fe640 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b8b40 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fef14014d40 tx=0x7fef14005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7ab701c720b482 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.457+0000 7feeff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.457+0000 7feeff7fe640 1 -- 172.21.15.89:0/2590382061 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14038070 con 0x55faeb0a9980 2024-09-15T23:29:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.457+0000 7feeff7fe640 1 -- 172.21.15.89:0/2590382061 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fef1401edb0 con 0x55faeb0a9980 2024-09-15T23:29:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.457+0000 7fef28962740 1 -- 172.21.15.89:0/2590382061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef14012820 con 0x55faeb0a9980 2024-09-15T23:29:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.457+0000 7feeff7fe640 1 -- 172.21.15.89:0/2590382061 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fef14045050 con 0x55faeb0a9980 2024-09-15T23:29:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.457+0000 7fef28962740 10 client.17418 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.457+0000 7fef28962740 1 -- 172.21.15.89:0/2590382061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55faeb12f130 con 0x55faeb0a9980 2024-09-15T23:29:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.458+0000 7feeff7fe640 1 -- 172.21.15.89:0/2590382061 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef1401e360 con 0x55faeb0a9980 2024-09-15T23:29:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.458+0000 7feeff7fe640 1 -- 172.21.15.89:0/2590382061 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fef1401e040 con 0x55faeb0a9980 2024-09-15T23:29:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.458+0000 7fef28962740 10 client.17418 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.458+0000 7fef28962740 10 client.17418 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:29:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.458+0000 7fef28962740 20 client.17418 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.458+0000 7fef28962740 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55faeb12f730 0x55faeb131b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.458+0000 7fef28962740 4 client.17418 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5e38a27a", "release": true, "await": true}] 2024-09-15T23:29:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.458+0000 7fef28962740 1 -- 172.21.15.89:0/2590382061 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5e38a27a", "release": true, "await": true}) -- 0x7fef14012820 con 0x55faeb12f730 2024-09-15T23:29:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.458+0000 7fef1ffff640 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55faeb12f730 0x55faeb131b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.459+0000 7feeff7fe640 10 client.17418 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.459+0000 7fef1ffff640 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55faeb12f730 0x55faeb131b50 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e98dad7502c4c4d server_cookie=f0dc47fd588750fa in_seq=0 out_seq=0 2024-09-15T23:29:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.461+0000 7feeff7fe640 1 -- 172.21.15.89:0/2590382061 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7fef14012820 con 0x55faeb12f730 2024-09-15T23:29:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.461+0000 7feeff7fe640 10 client.17418 handle_command_reply: tid=0 2024-09-15T23:29:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.461+0000 7fef28962740 1 client.17418 shutdown 2024-09-15T23:29:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7fef28962740 1 -- 172.21.15.89:0/2590382061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 msgr2=0x55faeb0b8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7fef28962740 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b8b40 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fef14014d40 tx=0x7fef14005df0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7fef28962740 1 -- 172.21.15.89:0/2590382061 shutdown_connections 2024-09-15T23:29:51.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7fef28962740 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55faeb12f730 0x55faeb131b50 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7fef28962740 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55faeb0b4240 0x55faeb12d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7fef28962740 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faeb0a9980 0x55faeb0b8b40 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7feeff7fe640 0 client.17418 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7fef28962740 1 --2- 172.21.15.89:0/2590382061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55faeb0a9060 0x55faeb0b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7fef28962740 1 -- 172.21.15.89:0/2590382061 >> 172.21.15.89:0/2590382061 conn(0x55faeb0a55f0 msgr2=0x55faeb0a4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7fef28962740 1 -- 172.21.15.89:0/2590382061 shutdown_connections 2024-09-15T23:29:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.462+0000 7fef28962740 1 -- 172.21.15.89:0/2590382061 wait complete. 2024-09-15T23:29:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.463+0000 7fef28962740 20 client.17418 trim_cache size 0 max 16384 2024-09-15T23:29:51.469 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:29:51.469 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 20, 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "5e38a27a": { 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 20, 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 145.3, 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:29:51.470 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:29:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:29:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:29:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:29:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:29:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:51.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:51.471 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:29:51.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.463+0000 7fef26589640 1 -- 172.21.15.89:0/1186711270 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 msgr2=0x7fef080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:51.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.463+0000 7fef26589640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 0x7fef080a5700 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fef200fe0c0 tx=0x7fef14014840 comp rx=0 tx=0).stop 2024-09-15T23:29:51.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.463+0000 7fef26589640 1 -- 172.21.15.89:0/1186711270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef20106f10 msgr2=0x7fef2019cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:51.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.463+0000 7fef26589640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef20106f10 0x7fef2019cbe0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fef1800cce0 tx=0x7fef1800a620 comp rx=0 tx=0).stop 2024-09-15T23:29:51.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.463+0000 7fef26589640 1 -- 172.21.15.89:0/1186711270 shutdown_connections 2024-09-15T23:29:51.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.463+0000 7fef26589640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fef080a3240 0x7fef080a5700 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.463+0000 7fef26589640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef20106f10 0x7fef2019cbe0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.463+0000 7fef26589640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef20102790 0x7fef20198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.464+0000 7fef26589640 1 --2- 172.21.15.89:0/1186711270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef20101de0 0x7fef20198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:51.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.464+0000 7fef26589640 1 -- 172.21.15.89:0/1186711270 >> 172.21.15.89:0/1186711270 conn(0x7fef200fd8c0 msgr2=0x7fef200feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:51.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.464+0000 7fef26589640 1 -- 172.21.15.89:0/1186711270 shutdown_connections 2024-09-15T23:29:51.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:51.464+0000 7fef26589640 1 -- 172.21.15.89:0/1186711270 wait complete. 2024-09-15T23:29:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:51 smithi120 ceph-mon[35250]: pgmap v525: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:29:51.479 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:51 smithi174 ceph-mon[35586]: pgmap v525: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:29:52.195 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5e38a27a', total seconds elapsed: 32.1 2024-09-15T23:29:52.195 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.35 2024-09-15T23:29:52.195 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 -- 172.21.15.89:0/1638582792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec074620 msgr2=0x7f30ec074a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 --2- 172.21.15.89:0/1638582792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec074620 0x7f30ec074a00 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009ca0 tx=0x7f30dc02a950 comp rx=0 tx=0).stop 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 -- 172.21.15.89:0/1638582792 shutdown_connections 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 --2- 172.21.15.89:0/1638582792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec10e3d0 0x7f30ec110820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 --2- 172.21.15.89:0/1638582792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec074f40 0x7f30ec10de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 --2- 172.21.15.89:0/1638582792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec074620 0x7f30ec074a00 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 -- 172.21.15.89:0/1638582792 >> 172.21.15.89:0/1638582792 conn(0x7f30ec0fd8f0 msgr2=0x7f30ec0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 -- 172.21.15.89:0/1638582792 shutdown_connections 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 -- 172.21.15.89:0/1638582792 wait complete. 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 Processor -- start 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.623+0000 7f30f32c5640 1 -- start start 2024-09-15T23:29:52.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f32c5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec074620 0x7f30ec19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f32c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec074f40 0x7f30ec19ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f32c5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec10e3d0 0x7f30ec19eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f32c5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec114160 con 0x7f30ec074f40 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f32c5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec113fe0 con 0x7f30ec074620 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f32c5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1142e0 con 0x7f30ec10e3d0 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f0860640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec074f40 0x7f30ec19ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f0860640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec074f40 0x7f30ec19ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40592/0 (socket says 172.21.15.89:40592) 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f0860640 1 -- 172.21.15.89:0/2631915483 learned_addr learned my addr 172.21.15.89:0/2631915483 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f1061640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec074620 0x7f30ec19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f1862640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec10e3d0 0x7f30ec19eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f0860640 1 -- 172.21.15.89:0/2631915483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec10e3d0 msgr2=0x7f30ec19eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f0860640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec10e3d0 0x7f30ec19eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f0860640 1 -- 172.21.15.89:0/2631915483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec074620 msgr2=0x7f30ec19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f0860640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec074620 0x7f30ec19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f0860640 1 -- 172.21.15.89:0/2631915483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec19f5c0 con 0x7f30ec074f40 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f0860640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec074f40 0x7f30ec19ab50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f30e000c920 tx=0x7f30e000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30ea7fc640 1 -- 172.21.15.89:0/2631915483 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e0007c70 con 0x7f30ec074f40 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30ea7fc640 1 -- 172.21.15.89:0/2631915483 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30e0007e10 con 0x7f30ec074f40 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30f32c5640 1 -- 172.21.15.89:0/2631915483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ec19f8b0 con 0x7f30ec074f40 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.624+0000 7f30ea7fc640 1 -- 172.21.15.89:0/2631915483 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e0018890 con 0x7f30ec074f40 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.625+0000 7f30f32c5640 1 -- 172.21.15.89:0/2631915483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ec1a7190 con 0x7f30ec074f40 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.627+0000 7f30ea7fc640 1 -- 172.21.15.89:0/2631915483 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f30e0018a70 con 0x7f30ec074f40 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.627+0000 7f30f569e740 1 Processor -- start 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.627+0000 7f30f569e740 1 -- start start 2024-09-15T23:29:52.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f569e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 0x564b2f7ef090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f569e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7ef5d0 0x564b2f7efa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f569e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7fc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f569e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b2f7552f0 con 0x564b2f7fa370 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f569e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b2f6fccf0 con 0x564b2f7f0b40 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f569e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b2f7512b0 con 0x564b2f7ef5d0 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f0860640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7fc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f0860640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7fc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40594/0 (socket says 172.21.15.89:40594) 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f0860640 1 -- 172.21.15.89:0/1164306571 learned_addr learned my addr 172.21.15.89:0/1164306571 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f1061640 1 --2- 172.21.15.89:0/1164306571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7ef5d0 0x564b2f7efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30ea7fc640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d40a3240 0x7f30d40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f1862640 1 --2- 172.21.15.89:0/1164306571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 0x564b2f7ef090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f1862640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d40a3240 0x7f30d40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f0860640 1 -- 172.21.15.89:0/1164306571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7ef5d0 msgr2=0x564b2f7efa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f0860640 1 --2- 172.21.15.89:0/1164306571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7ef5d0 0x564b2f7efa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f0860640 1 -- 172.21.15.89:0/1164306571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 msgr2=0x564b2f7ef090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f0860640 1 --2- 172.21.15.89:0/1164306571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 0x564b2f7ef090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30ea7fc640 1 -- 172.21.15.89:0/2631915483 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f30e00cfda0 con 0x7f30ec074f40 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f0860640 1 -- 172.21.15.89:0/1164306571 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b2f6e13a0 con 0x564b2f7fa370 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.628+0000 7f30f0860640 1 --2- 172.21.15.89:0/1164306571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7fc760 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f30e009bcd0 tx=0x7f30e000ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769107ab7cdabbf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30d37fe640 1 -- 172.21.15.89:0/1164306571 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e009c6d0 con 0x564b2f7fa370 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30f1862640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d40a3240 0x7f30d40a5700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f30e400bea0 tx=0x7f30e400b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30d37fe640 1 -- 172.21.15.89:0/1164306571 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30e00a4c40 con 0x564b2f7fa370 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30d37fe640 1 -- 172.21.15.89:0/1164306571 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e009a360 con 0x564b2f7fa370 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30f569e740 1 -- 172.21.15.89:0/1164306571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 msgr2=0x564b2f7fc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30f569e740 1 --2- 172.21.15.89:0/1164306571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7fc760 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f30e009bcd0 tx=0x7f30e000ff30 comp rx=0 tx=0).stop 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30f569e740 1 -- 172.21.15.89:0/1164306571 shutdown_connections 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30f569e740 1 --2- 172.21.15.89:0/1164306571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7fc760 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30f569e740 1 --2- 172.21.15.89:0/1164306571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7ef5d0 0x564b2f7efa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30f569e740 1 --2- 172.21.15.89:0/1164306571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 0x564b2f7ef090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30f569e740 1 -- 172.21.15.89:0/1164306571 >> 172.21.15.89:0/1164306571 conn(0x564b2f7b8310 msgr2=0x564b2f7ebc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30f569e740 1 -- 172.21.15.89:0/1164306571 shutdown_connections 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.629+0000 7f30f569e740 1 -- 172.21.15.89:0/1164306571 wait complete. 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f569e740 1 -- start start 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f569e740 1 Processor -- start 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f569e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7ef5d0 0x564b2f7f9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f569e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 0x564b2f7f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f569e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f569e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564b2f7fd010 con 0x564b2f7fa370 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f569e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b2f710060 con 0x564b2f7f0b40 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f569e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564b2f7fd190 con 0x564b2f7ef5d0 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f569e740 10 client.0 _ll_register_callbacks cb 0x564b2f4638c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:52.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f1862640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f1862640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7f75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40602/0 (socket says 172.21.15.89:40602) 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f1862640 1 -- 172.21.15.89:0/912706486 learned_addr learned my addr 172.21.15.89:0/912706486 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f1061640 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7ef5d0 0x564b2f7f9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.630+0000 7f30f0860640 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 0x564b2f7f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30f1862640 1 -- 172.21.15.89:0/912706486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7ef5d0 msgr2=0x564b2f7f9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30f1862640 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7ef5d0 0x564b2f7f9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30f1862640 1 -- 172.21.15.89:0/912706486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 msgr2=0x564b2f7f6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30f1862640 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 0x564b2f7f6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30f1862640 1 -- 172.21.15.89:0/912706486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b2f4c63c0 con 0x564b2f7fa370 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30f0860640 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 0x564b2f7f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30f1862640 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7f75d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f30e4010710 tx=0x7f30e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f73b7730e92788 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:52.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30d0ff9640 1 -- 172.21.15.89:0/912706486 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e4024070 con 0x564b2f7fa370 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30d0ff9640 1 -- 172.21.15.89:0/912706486 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f30e400f9d0 con 0x564b2f7fa370 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30f569e740 1 -- 172.21.15.89:0/912706486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30e00a4c40 con 0x564b2f7fa370 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30d0ff9640 1 -- 172.21.15.89:0/912706486 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f30e400fbd0 con 0x564b2f7fa370 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30f569e740 10 client.17430 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30f569e740 1 -- 172.21.15.89:0/912706486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b2f808ad0 con 0x564b2f7fa370 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.631+0000 7f30d0ff9640 1 -- 172.21.15.89:0/912706486 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30e00a4c40 con 0x564b2f7fa370 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.632+0000 7f30d0ff9640 1 -- 172.21.15.89:0/912706486 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f30e4004b60 con 0x564b2f7fa370 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.632+0000 7f30f569e740 10 client.17430 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.632+0000 7f30f569e740 10 client.17430 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.632+0000 7f30f569e740 20 client.17430 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.632+0000 7f30f569e740 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564b2f809f90 0x564b2f80c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.632+0000 7f30f569e740 4 client.17430 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.632+0000 7f30f569e740 1 -- 172.21.15.89:0/912706486 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b2f762860 con 0x564b2f809f90 2024-09-15T23:29:52.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.632+0000 7f30f1061640 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564b2f809f90 0x564b2f80c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.632+0000 7f30f1061640 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564b2f809f90 0x564b2f80c380 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4ad8bdb78bbd10ea server_cookie=72fdfc8d35a9279a in_seq=0 out_seq=0 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.632+0000 7f30d0ff9640 10 client.17430 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30d0ff9640 1 -- 172.21.15.89:0/912706486 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564b2f762860 con 0x564b2f809f90 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30d0ff9640 10 client.17430 handle_command_reply: tid=0 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 client.17430 shutdown 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 -- 172.21.15.89:0/912706486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 msgr2=0x564b2f7f75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7f75d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f30e4010710 tx=0x7f30e40046c0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 -- 172.21.15.89:0/912706486 shutdown_connections 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7fa370 0x564b2f7f75d0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7f0b40 0x564b2f7f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564b2f809f90 0x564b2f80c380 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 --2- 172.21.15.89:0/912706486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7ef5d0 0x564b2f7f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 -- 172.21.15.89:0/912706486 >> 172.21.15.89:0/912706486 conn(0x564b2f7b8310 msgr2=0x564b2f7f1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30d0ff9640 0 client.17430 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 -- 172.21.15.89:0/912706486 shutdown_connections 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.633+0000 7f30f569e740 1 -- 172.21.15.89:0/912706486 wait complete. 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.634+0000 7f30f569e740 20 client.17430 trim_cache size 0 max 16384 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.646+0000 7f30f569e740 1 Processor -- start 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.646+0000 7f30f569e740 1 -- start start 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.646+0000 7f30f569e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7ef5d0 0x564b2f8062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.646+0000 7f30f569e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7f0b40 0x564b2f7eb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.646+0000 7f30f569e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7fa370 0x564b2f7ebc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.646+0000 7f30f569e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1b1100 con 0x564b2f7ef5d0 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.646+0000 7f30f569e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1b2c40 con 0x564b2f7fa370 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.646+0000 7f30f569e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1b17d0 con 0x564b2f7f0b40 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.646+0000 7f30f1061640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7ef5d0 0x564b2f8062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30f1061640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7ef5d0 0x564b2f8062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40608/0 (socket says 172.21.15.89:40608) 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30f1061640 1 -- 172.21.15.89:0/1911034212 learned_addr learned my addr 172.21.15.89:0/1911034212 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30f1061640 1 -- 172.21.15.89:0/1911034212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7f0b40 msgr2=0x564b2f7eb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30f1862640 1 --2- 172.21.15.89:0/1911034212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7fa370 0x564b2f7ebc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30f1061640 1 --2- 172.21.15.89:0/1911034212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7f0b40 0x564b2f7eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30f1061640 1 -- 172.21.15.89:0/1911034212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7fa370 msgr2=0x564b2f7ebc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30f1061640 1 --2- 172.21.15.89:0/1911034212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7fa370 0x564b2f7ebc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30f1061640 1 -- 172.21.15.89:0/1911034212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b2f7fa0a0 con 0x564b2f7ef5d0 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30f1061640 1 --2- 172.21.15.89:0/1911034212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7ef5d0 0x564b2f8062b0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009ca0 tx=0x7f30dc009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a09b7d040156cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30d2ffd640 1 -- 172.21.15.89:0/1911034212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc04a070 con 0x564b2f7ef5d0 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30d2ffd640 1 -- 172.21.15.89:0/1911034212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30dc003e50 con 0x564b2f7ef5d0 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.647+0000 7f30d2ffd640 1 -- 172.21.15.89:0/1911034212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30dc034040 con 0x564b2f7ef5d0 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 -- 172.21.15.89:0/1911034212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7ef5d0 msgr2=0x564b2f8062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 --2- 172.21.15.89:0/1911034212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7ef5d0 0x564b2f8062b0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009ca0 tx=0x7f30dc009800 comp rx=0 tx=0).stop 2024-09-15T23:29:52.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 -- 172.21.15.89:0/1911034212 shutdown_connections 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 --2- 172.21.15.89:0/1911034212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7fa370 0x564b2f7ebc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 --2- 172.21.15.89:0/1911034212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7f0b40 0x564b2f7eb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 --2- 172.21.15.89:0/1911034212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7ef5d0 0x564b2f8062b0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 -- 172.21.15.89:0/1911034212 >> 172.21.15.89:0/1911034212 conn(0x564b2f7b8310 msgr2=0x564b2f7f6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 -- 172.21.15.89:0/1911034212 shutdown_connections 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 -- 172.21.15.89:0/1911034212 wait complete. 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 -- start start 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.648+0000 7f30f569e740 1 Processor -- start 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f569e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7ef5d0 0x564b2f802b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f569e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7f0b40 0x564b2f7fde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f569e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7fa370 0x564b2f7fe3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f569e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1b2570 con 0x564b2f7f0b40 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f569e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1b1ea0 con 0x564b2f7ef5d0 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f569e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30ec1af0e0 con 0x564b2f7fa370 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f569e740 10 client.0 _ll_register_callbacks cb 0x7f30e4012f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f0860640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7f0b40 0x564b2f7fde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f0860640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7f0b40 0x564b2f7fde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40612/0 (socket says 172.21.15.89:40612) 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f0860640 1 -- 172.21.15.89:0/2613864417 learned_addr learned my addr 172.21.15.89:0/2613864417 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f1862640 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7fa370 0x564b2f7fe3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f1061640 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7ef5d0 0x564b2f802b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f0860640 1 -- 172.21.15.89:0/2613864417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7fa370 msgr2=0x564b2f7fe3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f0860640 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7fa370 0x564b2f7fe3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f0860640 1 -- 172.21.15.89:0/2613864417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7ef5d0 msgr2=0x564b2f802b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f0860640 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7ef5d0 0x564b2f802b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f0860640 1 -- 172.21.15.89:0/2613864417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b2f7feba0 con 0x564b2f7f0b40 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f1061640 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7ef5d0 0x564b2f802b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30f0860640 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7f0b40 0x564b2f7fde60 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f30e0098430 tx=0x7f30e00bc1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e23f0fb1ace60a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:52.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30d0ff9640 1 -- 172.21.15.89:0/2613864417 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e0057060 con 0x564b2f7f0b40 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.649+0000 7f30d0ff9640 1 -- 172.21.15.89:0/2613864417 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f30e000fb30 con 0x564b2f7f0b40 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30f569e740 1 -- 172.21.15.89:0/2613864417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30dc003e50 con 0x564b2f7f0b40 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30d0ff9640 1 -- 172.21.15.89:0/2613864417 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f30e00b4030 con 0x564b2f7f0b40 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30f569e740 10 client.17436 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30f569e740 1 -- 172.21.15.89:0/2613864417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564b2f87d400 con 0x564b2f7f0b40 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30d0ff9640 1 -- 172.21.15.89:0/2613864417 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30dc003e50 con 0x564b2f7f0b40 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30d0ff9640 1 -- 172.21.15.89:0/2613864417 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f30e00d2040 con 0x564b2f7f0b40 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30f569e740 10 client.17436 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30f569e740 10 client.17436 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30f569e740 20 client.17436 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30f569e740 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564b2f87d6f0 0x564b2f87fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30f569e740 4 client.17436 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30f569e740 1 -- 172.21.15.89:0/2613864417 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564b2f75cd70 con 0x564b2f87d6f0 2024-09-15T23:29:52.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.650+0000 7f30f1061640 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564b2f87d6f0 0x564b2f87fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f1061640 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564b2f87d6f0 0x564b2f87fae0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bc7b089af0fa9c54 server_cookie=c958c95d22b82ae7 in_seq=0 out_seq=0 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30d0ff9640 10 client.17436 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30d0ff9640 1 -- 172.21.15.89:0/2613864417 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564b2f75cd70 con 0x564b2f87d6f0 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30d0ff9640 10 client.17436 handle_command_reply: tid=0 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 client.17436 shutdown 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 -- 172.21.15.89:0/2613864417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7f0b40 msgr2=0x564b2f7fde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7f0b40 0x564b2f7fde60 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f30e0098430 tx=0x7f30e00bc1c0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 -- 172.21.15.89:0/2613864417 shutdown_connections 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564b2f87d6f0 0x564b2f87fae0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30d0ff9640 0 client.17436 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564b2f7fa370 0x564b2f7fe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564b2f7f0b40 0x564b2f7fde60 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 --2- 172.21.15.89:0/2613864417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2f7ef5d0 0x564b2f802b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 -- 172.21.15.89:0/2613864417 >> 172.21.15.89:0/2613864417 conn(0x564b2f7b8310 msgr2=0x564b2f7eb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:52.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 -- 172.21.15.89:0/2613864417 shutdown_connections 2024-09-15T23:29:52.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.651+0000 7f30f569e740 1 -- 172.21.15.89:0/2613864417 wait complete. 2024-09-15T23:29:52.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f569e740 20 client.17436 trim_cache size 0 max 16384 2024-09-15T23:29:52.669 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:29:52.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 -- 172.21.15.89:0/2631915483 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d40a3240 msgr2=0x7f30d40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:52.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d40a3240 0x7f30d40a5700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f30e400bea0 tx=0x7f30e400b2e0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 -- 172.21.15.89:0/2631915483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec074f40 msgr2=0x7f30ec19ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:52.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec074f40 0x7f30ec19ab50 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f30e000c920 tx=0x7f30e000cde0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 -- 172.21.15.89:0/2631915483 shutdown_connections 2024-09-15T23:29:52.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30d40a3240 0x7f30d40a5700 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30ec10e3d0 0x7f30ec19eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f30ec074f40 0x7f30ec19ab50 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 --2- 172.21.15.89:0/2631915483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ec074620 0x7f30ec19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:52.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 -- 172.21.15.89:0/2631915483 >> 172.21.15.89:0/2631915483 conn(0x7f30ec0fd8f0 msgr2=0x7f30ec0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:52.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 -- 172.21.15.89:0/2631915483 shutdown_connections 2024-09-15T23:29:52.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:52.652+0000 7f30f32c5640 1 -- 172.21.15.89:0/2631915483 wait complete. 2024-09-15T23:29:53.017 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:53.424 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:53 smithi089 ceph-mon[30933]: pgmap v526: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-09-15T23:29:53.451 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:29:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:53 smithi120 ceph-mon[35250]: pgmap v526: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-09-15T23:29:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:53 smithi174 ceph-mon[35586]: pgmap v526: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-09-15T23:29:53.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.891+0000 7fe961274640 1 -- 172.21.15.89:0/1562154408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe95c1082e0 msgr2=0x7fe95c10ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:53.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.891+0000 7fe961274640 1 --2- 172.21.15.89:0/1562154408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe95c1082e0 0x7fe95c10ec50 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe954009cd0 tx=0x7fe95402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.891+0000 7fe961274640 1 -- 172.21.15.89:0/1562154408 shutdown_connections 2024-09-15T23:29:53.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.891+0000 7fe961274640 1 --2- 172.21.15.89:0/1562154408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe95c1082e0 0x7fe95c10ec50 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.891+0000 7fe961274640 1 --2- 172.21.15.89:0/1562154408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95c103760 0x7fe95c107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.891+0000 7fe961274640 1 --2- 172.21.15.89:0/1562154408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe95c102db0 0x7fe95c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.891+0000 7fe961274640 1 -- 172.21.15.89:0/1562154408 >> 172.21.15.89:0/1562154408 conn(0x7fe95c0fc7d0 msgr2=0x7fe95c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.891+0000 7fe961274640 1 -- 172.21.15.89:0/1562154408 shutdown_connections 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.891+0000 7fe961274640 1 -- 172.21.15.89:0/1562154408 wait complete. 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.891+0000 7fe961274640 1 Processor -- start 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe961274640 1 -- start start 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe961274640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95c102db0 0x7fe95c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe961274640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe95c103760 0x7fe95c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe961274640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe95c1082e0 0x7fe95c19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe961274640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe95c1120b0 con 0x7fe95c1082e0 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe961274640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe95c111f30 con 0x7fe95c102db0 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe961274640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe95c112230 con 0x7fe95c103760 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe95b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe95c1082e0 0x7fe95c19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:53.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe95b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe95c1082e0 0x7fe95c19ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40634/0 (socket says 172.21.15.89:40634) 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe95b7fe640 1 -- 172.21.15.89:0/2782855746 learned_addr learned my addr 172.21.15.89:0/2782855746 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe95affd640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95c102db0 0x7fe95c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.892+0000 7fe95a7fc640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe95c103760 0x7fe95c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe95b7fe640 1 -- 172.21.15.89:0/2782855746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe95c103760 msgr2=0x7fe95c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe95b7fe640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe95c103760 0x7fe95c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe95b7fe640 1 -- 172.21.15.89:0/2782855746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95c102db0 msgr2=0x7fe95c198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe95b7fe640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95c102db0 0x7fe95c198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe95b7fe640 1 -- 172.21.15.89:0/2782855746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95c19d510 con 0x7fe95c1082e0 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe95b7fe640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe95c1082e0 0x7fe95c19ce30 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe954005e30 tx=0x7fe954003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe943fff640 1 -- 172.21.15.89:0/2782855746 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe954004040 con 0x7fe95c1082e0 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe943fff640 1 -- 172.21.15.89:0/2782855746 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9540046c0 con 0x7fe95c1082e0 2024-09-15T23:29:53.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe943fff640 1 -- 172.21.15.89:0/2782855746 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95403a400 con 0x7fe95c1082e0 2024-09-15T23:29:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe961274640 1 -- 172.21.15.89:0/2782855746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe95c19d7a0 con 0x7fe95c1082e0 2024-09-15T23:29:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.893+0000 7fe961274640 1 -- 172.21.15.89:0/2782855746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe95c1a50e0 con 0x7fe95c1082e0 2024-09-15T23:29:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.894+0000 7fe961274640 1 -- 172.21.15.89:0/2782855746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe95c103190 con 0x7fe95c1082e0 2024-09-15T23:29:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.895+0000 7fe943fff640 1 -- 172.21.15.89:0/2782855746 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe954030630 con 0x7fe95c1082e0 2024-09-15T23:29:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.896+0000 7fe943fff640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9440a3240 0x7fe9440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.896+0000 7fe943fff640 1 -- 172.21.15.89:0/2782855746 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe95402e030 con 0x7fe95c1082e0 2024-09-15T23:29:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.896+0000 7fe95affd640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9440a3240 0x7fe9440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.897+0000 7fe95affd640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9440a3240 0x7fe9440a5700 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00bbe0 tx=0x7fe94c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:53.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:53.900+0000 7fe943fff640 1 -- 172.21.15.89:0/2782855746 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe9540bd8c0 con 0x7fe95c1082e0 2024-09-15T23:29:53.973 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.115+0000 7fe961274640 1 -- 172.21.15.89:0/2782855746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe95c10a2c0 con 0x7fe95c1082e0 2024-09-15T23:29:54.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.115+0000 7fe943fff640 1 -- 172.21.15.89:0/2782855746 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fe9540bd1c0 con 0x7fe95c1082e0 2024-09-15T23:29:54.118 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:29:54.118 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:29:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:29:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 -- 172.21.15.89:0/2782855746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9440a3240 msgr2=0x7fe9440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9440a3240 0x7fe9440a5700 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00bbe0 tx=0x7fe94c00b390 comp rx=0 tx=0).stop 2024-09-15T23:29:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 -- 172.21.15.89:0/2782855746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe95c1082e0 msgr2=0x7fe95c19ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe95c1082e0 0x7fe95c19ce30 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fe954005e30 tx=0x7fe954003970 comp rx=0 tx=0).stop 2024-09-15T23:29:54.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 -- 172.21.15.89:0/2782855746 shutdown_connections 2024-09-15T23:29:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe9440a3240 0x7fe9440a5700 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe95c1082e0 0x7fe95c19ce30 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe95c103760 0x7fe95c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 --2- 172.21.15.89:0/2782855746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe95c102db0 0x7fe95c198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 -- 172.21.15.89:0/2782855746 >> 172.21.15.89:0/2782855746 conn(0x7fe95c0fc7d0 msgr2=0x7fe95c105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 -- 172.21.15.89:0/2782855746 shutdown_connections 2024-09-15T23:29:54.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:54.118+0000 7fe961274640 1 -- 172.21.15.89:0/2782855746 wait complete. 2024-09-15T23:29:54.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:54 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2782855746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:54 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2782855746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:54.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:54 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2782855746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:54.922 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0' 2024-09-15T23:29:54.923 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0 2024-09-15T23:29:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.162+0000 7f15a70de640 1 -- 172.21.15.89:0/2023526180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a0100a60 msgr2=0x7f15a0100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.162+0000 7f15a70de640 1 --2- 172.21.15.89:0/2023526180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a0100a60 0x7f15a0100ec0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f158c009930 tx=0x7f158c02a420 comp rx=0 tx=0).stop 2024-09-15T23:29:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.162+0000 7f15a70de640 1 -- 172.21.15.89:0/2023526180 shutdown_connections 2024-09-15T23:29:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.162+0000 7f15a70de640 1 --2- 172.21.15.89:0/2023526180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15a0101400 0x7f15a010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.162+0000 7f15a70de640 1 --2- 172.21.15.89:0/2023526180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a0100a60 0x7f15a0100ec0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.162+0000 7f15a70de640 1 --2- 172.21.15.89:0/2023526180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15a0106a60 0x7f15a0106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.162+0000 7f15a70de640 1 -- 172.21.15.89:0/2023526180 >> 172.21.15.89:0/2023526180 conn(0x7f15a00fc780 msgr2=0x7f15a00feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.162+0000 7f15a70de640 1 -- 172.21.15.89:0/2023526180 shutdown_connections 2024-09-15T23:29:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.162+0000 7f15a70de640 1 -- 172.21.15.89:0/2023526180 wait complete. 2024-09-15T23:29:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a70de640 1 Processor -- start 2024-09-15T23:29:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a70de640 1 -- start start 2024-09-15T23:29:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a70de640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a0100a60 0x7f15a01973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a70de640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15a0101400 0x7f15a0197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a70de640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15a0106a60 0x7f15a019bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a70de640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f15a0110e60 con 0x7f15a0100a60 2024-09-15T23:29:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a70de640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f15a0110ce0 con 0x7f15a0101400 2024-09-15T23:29:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a70de640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f15a0110fe0 con 0x7f15a0106a60 2024-09-15T23:29:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a4e7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a0100a60 0x7f15a01973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a4e7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a0100a60 0x7f15a01973d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36906/0 (socket says 172.21.15.89:36906) 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a4e7a640 1 -- 172.21.15.89:0/3645942253 learned_addr learned my addr 172.21.15.89:0/3645942253 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f159ffff640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15a0101400 0x7f15a0197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a567b640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15a0106a60 0x7f15a019bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a4e7a640 1 -- 172.21.15.89:0/3645942253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15a0106a60 msgr2=0x7f15a019bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15a0106a60 0x7f15a019bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a4e7a640 1 -- 172.21.15.89:0/3645942253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15a0101400 msgr2=0x7f15a0197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15a0101400 0x7f15a0197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.163+0000 7f15a4e7a640 1 -- 172.21.15.89:0/3645942253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a019c380 con 0x7f15a0100a60 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.164+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a0100a60 0x7f15a01973d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f159400d950 tx=0x7f159400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.164+0000 7f159dffb640 1 -- 172.21.15.89:0/3645942253 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1594017070 con 0x7f15a0100a60 2024-09-15T23:29:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.164+0000 7f159dffb640 1 -- 172.21.15.89:0/3645942253 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1594009c30 con 0x7f15a0100a60 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.164+0000 7f15a70de640 1 -- 172.21.15.89:0/3645942253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a019c670 con 0x7f15a0100a60 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.164+0000 7f159dffb640 1 -- 172.21.15.89:0/3645942253 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159400f6b0 con 0x7f15a0100a60 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.164+0000 7f15a70de640 1 -- 172.21.15.89:0/3645942253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a01a3f50 con 0x7f15a0100a60 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.166+0000 7f159dffb640 1 -- 172.21.15.89:0/3645942253 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f15940109d0 con 0x7f15a0100a60 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.166+0000 7f15a94b7740 1 Processor -- start 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.166+0000 7f15a94b7740 1 -- start start 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d472a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d477e0 0x55aa75d47c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 0x55aa75d54970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a94b7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55aa75cab2f0 con 0x55aa75d48d50 2024-09-15T23:29:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a94b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55aa75c52cf0 con 0x55aa75d477e0 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a94b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55aa75ca72b0 con 0x55aa75d52580 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a567b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a567b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d472a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36908/0 (socket says 172.21.15.89:36908) 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a567b640 1 -- 172.21.15.89:0/3611053575 learned_addr learned my addr 172.21.15.89:0/3611053575 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f159dffb640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f15900a3240 0x7f15900a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f159ffff640 1 --2- 172.21.15.89:0/3611053575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 0x55aa75d54970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a567b640 1 -- 172.21.15.89:0/3611053575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 msgr2=0x55aa75d54970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a567b640 1 --2- 172.21.15.89:0/3611053575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 0x55aa75d54970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a567b640 1 -- 172.21.15.89:0/3611053575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d477e0 msgr2=0x55aa75d47c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a567b640 1 --2- 172.21.15.89:0/3611053575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d477e0 0x55aa75d47c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a567b640 1 -- 172.21.15.89:0/3611053575 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa75c373a0 con 0x55aa75d48d50 2024-09-15T23:29:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a567b640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f15900a3240 0x7f15900a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f159dffb640 1 -- 172.21.15.89:0/3645942253 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f15940cf8a0 con 0x7f15a0100a60 2024-09-15T23:29:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.167+0000 7f15a567b640 1 --2- 172.21.15.89:0/3611053575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d472a0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1598008f40 tx=0x7f159800e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e77f2628fbcfd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f158affd640 1 -- 172.21.15.89:0/3611053575 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598022070 con 0x55aa75d48d50 2024-09-15T23:29:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f158affd640 1 -- 172.21.15.89:0/3611053575 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1598011b70 con 0x55aa75d48d50 2024-09-15T23:29:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f158affd640 1 -- 172.21.15.89:0/3611053575 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159801cbb0 con 0x55aa75d48d50 2024-09-15T23:29:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f15a94b7740 1 -- 172.21.15.89:0/3611053575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 msgr2=0x55aa75d472a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:55.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f15a567b640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f15900a3240 0x7f15900a5700 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f159801f000 tx=0x7f1598020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f15a94b7740 1 --2- 172.21.15.89:0/3611053575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d472a0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1598008f40 tx=0x7f159800e7e0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f15a94b7740 1 -- 172.21.15.89:0/3611053575 shutdown_connections 2024-09-15T23:29:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f15a94b7740 1 --2- 172.21.15.89:0/3611053575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 0x55aa75d54970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f15a94b7740 1 --2- 172.21.15.89:0/3611053575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d477e0 0x55aa75d47c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f15a94b7740 1 --2- 172.21.15.89:0/3611053575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d472a0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f15a94b7740 1 -- 172.21.15.89:0/3611053575 >> 172.21.15.89:0/3611053575 conn(0x55aa75d0e310 msgr2=0x55aa75d43e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:55.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f15a94b7740 1 -- 172.21.15.89:0/3611053575 shutdown_connections 2024-09-15T23:29:55.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.168+0000 7f15a94b7740 1 -- 172.21.15.89:0/3611053575 wait complete. 2024-09-15T23:29:55.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a94b7740 1 -- start start 2024-09-15T23:29:55.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a94b7740 1 Processor -- start 2024-09-15T23:29:55.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d477e0 0x55aa75d51ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d4f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d52580 0x55aa75d4f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a94b7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55aa75d55280 con 0x55aa75d48d50 2024-09-15T23:29:55.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a94b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55aa75c66060 con 0x55aa75d52580 2024-09-15T23:29:55.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a94b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55aa75d55400 con 0x55aa75d477e0 2024-09-15T23:29:55.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a94b7740 10 client.0 _ll_register_callbacks cb 0x55aa759b98c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:55.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f159ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d4f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f159ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d4f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36920/0 (socket says 172.21.15.89:36920) 2024-09-15T23:29:55.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f159ffff640 1 -- 172.21.15.89:0/1529361256 learned_addr learned my addr 172.21.15.89:0/1529361256 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:55.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a567b640 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d52580 0x55aa75d4f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.169+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d477e0 0x55aa75d51ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f159ffff640 1 -- 172.21.15.89:0/1529361256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d477e0 msgr2=0x55aa75d51ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:55.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f159ffff640 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d477e0 0x55aa75d51ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f159ffff640 1 -- 172.21.15.89:0/1529361256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d52580 msgr2=0x55aa75d4f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:55.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f159ffff640 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d52580 0x55aa75d4f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f159ffff640 1 -- 172.21.15.89:0/1529361256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa75a1c3c0 con 0x55aa75d48d50 2024-09-15T23:29:55.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d477e0 0x55aa75d51ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f15a567b640 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d52580 0x55aa75d4f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f159ffff640 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d4f2a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f158c02a940 tx=0x7f158c031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ccde50b6fc107b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f157ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f157ffff640 1 -- 172.21.15.89:0/1529361256 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c035040 con 0x55aa75d48d50 2024-09-15T23:29:55.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f15a94b7740 1 -- 172.21.15.89:0/1529361256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1598011b70 con 0x55aa75d48d50 2024-09-15T23:29:55.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f157ffff640 1 -- 172.21.15.89:0/1529361256 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f158c004240 con 0x55aa75d48d50 2024-09-15T23:29:55.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f157ffff640 1 -- 172.21.15.89:0/1529361256 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f158c004440 con 0x55aa75d48d50 2024-09-15T23:29:55.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f15a94b7740 10 client.17454 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:55.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f15a94b7740 1 -- 172.21.15.89:0/1529361256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55aa75d60e00 con 0x55aa75d48d50 2024-09-15T23:29:55.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f157ffff640 1 -- 172.21.15.89:0/1529361256 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1598011b70 con 0x55aa75d48d50 2024-09-15T23:29:55.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.170+0000 7f157ffff640 1 -- 172.21.15.89:0/1529361256 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f158c03c710 con 0x55aa75d48d50 2024-09-15T23:29:55.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.171+0000 7f15a94b7740 10 client.17454 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:55.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.171+0000 7f15a94b7740 10 client.17454 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:29:55.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.171+0000 7f15a94b7740 20 client.17454 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:55.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.171+0000 7f15a94b7740 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55aa75d62260 0x55aa75d64680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.171+0000 7f15a94b7740 4 client.17454 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:55.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.171+0000 7f15a94b7740 1 -- 172.21.15.89:0/1529361256 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa75cb8860 con 0x55aa75d62260 2024-09-15T23:29:55.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.171+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55aa75d62260 0x55aa75d64680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.171+0000 7f157ffff640 10 client.17454 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:29:55.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.171+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55aa75d62260 0x55aa75d64680 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6bad9e43f82e425a server_cookie=47a3e8971567d0bb in_seq=0 out_seq=0 2024-09-15T23:29:55.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f157ffff640 1 -- 172.21.15.89:0/1529361256 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aa75cb8860 con 0x55aa75d62260 2024-09-15T23:29:55.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f157ffff640 10 client.17454 handle_command_reply: tid=0 2024-09-15T23:29:55.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 client.17454 shutdown 2024-09-15T23:29:55.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 -- 172.21.15.89:0/1529361256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 msgr2=0x55aa75d4f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:55.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d4f2a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f158c02a940 tx=0x7f158c031920 comp rx=0 tx=0).stop 2024-09-15T23:29:55.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 -- 172.21.15.89:0/1529361256 shutdown_connections 2024-09-15T23:29:55.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55aa75d62260 0x55aa75d64680 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d52580 0x55aa75d4f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d48d50 0x55aa75d4f2a0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 --2- 172.21.15.89:0/1529361256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d477e0 0x55aa75d51ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f157ffff640 0 client.17454 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:29:55.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 -- 172.21.15.89:0/1529361256 >> 172.21.15.89:0/1529361256 conn(0x55aa75d0e310 msgr2=0x55aa75d49b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:55.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 -- 172.21.15.89:0/1529361256 shutdown_connections 2024-09-15T23:29:55.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 1 -- 172.21.15.89:0/1529361256 wait complete. 2024-09-15T23:29:55.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.172+0000 7f15a94b7740 20 client.17454 trim_cache size 0 max 16384 2024-09-15T23:29:55.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a94b7740 1 Processor -- start 2024-09-15T23:29:55.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a94b7740 1 -- start start 2024-09-15T23:29:55.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d477e0 0x55aa75d43f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d48d50 0x55aa75d5e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d52580 0x55aa75d4e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a94b7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01acd60 con 0x55aa75d52580 2024-09-15T23:29:55.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a94b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01ae8a0 con 0x55aa75d477e0 2024-09-15T23:29:55.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a94b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01ad430 con 0x55aa75d48d50 2024-09-15T23:29:55.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a567b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d52580 0x55aa75d4e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a567b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d52580 0x55aa75d4e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36922/0 (socket says 172.21.15.89:36922) 2024-09-15T23:29:55.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a567b640 1 -- 172.21.15.89:0/78227409 learned_addr learned my addr 172.21.15.89:0/78227409 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:55.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f159ffff640 1 --2- 172.21.15.89:0/78227409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d48d50 0x55aa75d5e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/78227409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d477e0 0x55aa75d43f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a567b640 1 -- 172.21.15.89:0/78227409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d48d50 msgr2=0x55aa75d5e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:55.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a567b640 1 --2- 172.21.15.89:0/78227409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d48d50 0x55aa75d5e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a567b640 1 -- 172.21.15.89:0/78227409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d477e0 msgr2=0x55aa75d43f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:55.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a567b640 1 --2- 172.21.15.89:0/78227409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d477e0 0x55aa75d43f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.186+0000 7f15a567b640 1 -- 172.21.15.89:0/78227409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa75d40310 con 0x55aa75d52580 2024-09-15T23:29:55.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.187+0000 7f15a567b640 1 --2- 172.21.15.89:0/78227409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d52580 0x55aa75d4e150 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f159800ece0 tx=0x7f1598012690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8085aeb7919ca3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:55.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.187+0000 7f158a7fc640 1 -- 172.21.15.89:0/78227409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598022040 con 0x55aa75d52580 2024-09-15T23:29:55.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.187+0000 7f158a7fc640 1 -- 172.21.15.89:0/78227409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1598011970 con 0x55aa75d52580 2024-09-15T23:29:55.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.187+0000 7f158a7fc640 1 -- 172.21.15.89:0/78227409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159801c930 con 0x55aa75d52580 2024-09-15T23:29:55.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.187+0000 7f15a94b7740 1 -- 172.21.15.89:0/78227409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d52580 msgr2=0x55aa75d4e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:55.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.187+0000 7f15a94b7740 1 --2- 172.21.15.89:0/78227409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d52580 0x55aa75d4e150 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f159800ece0 tx=0x7f1598012690 comp rx=0 tx=0).stop 2024-09-15T23:29:55.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.187+0000 7f15a94b7740 1 -- 172.21.15.89:0/78227409 shutdown_connections 2024-09-15T23:29:55.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.187+0000 7f15a94b7740 1 --2- 172.21.15.89:0/78227409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d52580 0x55aa75d4e150 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.187+0000 7f15a94b7740 1 --2- 172.21.15.89:0/78227409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d48d50 0x55aa75d5e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.187+0000 7f15a94b7740 1 --2- 172.21.15.89:0/78227409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d477e0 0x55aa75d43f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 -- 172.21.15.89:0/78227409 >> 172.21.15.89:0/78227409 conn(0x55aa75d0e310 msgr2=0x55aa75d4da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:55.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 -- 172.21.15.89:0/78227409 shutdown_connections 2024-09-15T23:29:55.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 -- 172.21.15.89:0/78227409 wait complete. 2024-09-15T23:29:55.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 -- start start 2024-09-15T23:29:55.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 Processor -- start 2024-09-15T23:29:55.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d477e0 0x55aa75dc74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d48d50 0x55aa75dc7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 0x55aa75d404e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01ae1d0 con 0x55aa75d477e0 2024-09-15T23:29:55.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01adb00 con 0x55aa75d48d50 2024-09-15T23:29:55.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.188+0000 7f15a94b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01aad40 con 0x55aa75d52580 2024-09-15T23:29:55.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a94b7740 10 client.0 _ll_register_callbacks cb 0x7f158c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:55.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a4e7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d477e0 0x55aa75dc74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a4e7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d477e0 0x55aa75dc74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36936/0 (socket says 172.21.15.89:36936) 2024-09-15T23:29:55.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a4e7a640 1 -- 172.21.15.89:0/55564903 learned_addr learned my addr 172.21.15.89:0/55564903 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:55.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a567b640 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 0x55aa75d404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f159ffff640 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d48d50 0x55aa75dc7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a4e7a640 1 -- 172.21.15.89:0/55564903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 msgr2=0x55aa75d404e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:55.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 0x55aa75d404e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a4e7a640 1 -- 172.21.15.89:0/55564903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d48d50 msgr2=0x55aa75dc7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:55.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d48d50 0x55aa75dc7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a4e7a640 1 -- 172.21.15.89:0/55564903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa75d40ce0 con 0x55aa75d477e0 2024-09-15T23:29:55.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a567b640 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 0x55aa75d404e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:55.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f159ffff640 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d48d50 0x55aa75dc7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:55.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a4e7a640 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d477e0 0x55aa75dc74e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f15940984a0 tx=0x7f1594005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ea4ea1e5fafd5b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:55.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f157ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:55.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f157ffff640 1 -- 172.21.15.89:0/55564903 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1594020040 con 0x55aa75d477e0 2024-09-15T23:29:55.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f157ffff640 1 -- 172.21.15.89:0/55564903 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1594099dd0 con 0x55aa75d477e0 2024-09-15T23:29:55.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.189+0000 7f15a94b7740 1 -- 172.21.15.89:0/55564903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1598011970 con 0x55aa75d477e0 2024-09-15T23:29:55.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f157ffff640 1 -- 172.21.15.89:0/55564903 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f159409bad0 con 0x55aa75d477e0 2024-09-15T23:29:55.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f15a94b7740 10 client.17460 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:55.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f15a94b7740 1 -- 172.21.15.89:0/55564903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55aa75dcebc0 con 0x55aa75d477e0 2024-09-15T23:29:55.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f157ffff640 1 -- 172.21.15.89:0/55564903 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1598011970 con 0x55aa75d477e0 2024-09-15T23:29:55.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f157ffff640 1 -- 172.21.15.89:0/55564903 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f15940a42b0 con 0x55aa75d477e0 2024-09-15T23:29:55.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f15a94b7740 10 client.17460 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:55.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f15a94b7740 10 client.17460 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:29:55.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f15a94b7740 20 client.17460 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:55.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f15a94b7740 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55aa75dcf290 0x55aa75dd1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:55.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f15a94b7740 4 client.17460 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:55.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.190+0000 7f15a94b7740 1 -- 172.21.15.89:0/55564903 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aa75cb2d70 con 0x55aa75dcf290 2024-09-15T23:29:55.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f159ffff640 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55aa75dcf290 0x55aa75dd1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:55.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f157ffff640 10 client.17460 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:29:55.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f159ffff640 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55aa75dcf290 0x55aa75dd1680 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8ff682e642402958 server_cookie=b5b7b13306c91954 in_seq=0 out_seq=0 2024-09-15T23:29:55.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f157ffff640 1 -- 172.21.15.89:0/55564903 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aa75cb2d70 con 0x55aa75dcf290 2024-09-15T23:29:55.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f157ffff640 10 client.17460 handle_command_reply: tid=0 2024-09-15T23:29:55.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f15a94b7740 1 client.17460 shutdown 2024-09-15T23:29:55.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f15a94b7740 1 -- 172.21.15.89:0/55564903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d477e0 msgr2=0x55aa75dc74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:55.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f15a94b7740 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d477e0 0x55aa75dc74e0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f15940984a0 tx=0x7f1594005880 comp rx=0 tx=0).stop 2024-09-15T23:29:55.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f15a94b7740 1 -- 172.21.15.89:0/55564903 shutdown_connections 2024-09-15T23:29:55.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f15a94b7740 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55aa75dcf290 0x55aa75dd1680 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f15a94b7740 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aa75d52580 0x55aa75d404e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f15a94b7740 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aa75d48d50 0x55aa75dc7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f15a94b7740 1 --2- 172.21.15.89:0/55564903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aa75d477e0 0x55aa75dc74e0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f15a94b7740 1 -- 172.21.15.89:0/55564903 >> 172.21.15.89:0/55564903 conn(0x55aa75d0e310 msgr2=0x55aa75d49b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:55.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.191+0000 7f157ffff640 0 client.17460 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:29:55.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a94b7740 1 -- 172.21.15.89:0/55564903 shutdown_connections 2024-09-15T23:29:55.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a94b7740 1 -- 172.21.15.89:0/55564903 wait complete. 2024-09-15T23:29:55.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a94b7740 20 client.17460 trim_cache size 0 max 16384 2024-09-15T23:29:55.224 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:29:55.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 -- 172.21.15.89:0/3645942253 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f15900a3240 msgr2=0x7f15900a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:55.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f15900a3240 0x7f15900a5700 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f159801f000 tx=0x7f1598020000 comp rx=0 tx=0).stop 2024-09-15T23:29:55.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 -- 172.21.15.89:0/3645942253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a0100a60 msgr2=0x7f15a01973d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:55.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a0100a60 0x7f15a01973d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f159400d950 tx=0x7f159400de10 comp rx=0 tx=0).stop 2024-09-15T23:29:55.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 -- 172.21.15.89:0/3645942253 shutdown_connections 2024-09-15T23:29:55.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f15900a3240 0x7f15900a5700 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15a0106a60 0x7f15a019bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f15a0101400 0x7f15a0197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 --2- 172.21.15.89:0/3645942253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15a0100a60 0x7f15a01973d0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 -- 172.21.15.89:0/3645942253 >> 172.21.15.89:0/3645942253 conn(0x7f15a00fc780 msgr2=0x7f15a00fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.192+0000 7f15a70de640 1 -- 172.21.15.89:0/3645942253 shutdown_connections 2024-09-15T23:29:55.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:55.193+0000 7f15a70de640 1 -- 172.21.15.89:0/3645942253 wait complete. 2024-09-15T23:29:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:55 smithi120 ceph-mon[35250]: pgmap v527: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-15T23:29:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:55 smithi174 ceph-mon[35586]: pgmap v527: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-15T23:29:55.517 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:55 smithi089 ceph-mon[30933]: pgmap v527: 129 pgs: 129 active+clean; 2.8 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-09-15T23:29:55.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:29:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.477+0000 7ff12f0a5640 1 -- 172.21.15.89:0/537931130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128100a40 msgr2=0x7ff128100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.477+0000 7ff12f0a5640 1 --2- 172.21.15.89:0/537931130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128100a40 0x7ff128100e40 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009900 tx=0x7ff11c02a410 comp rx=0 tx=0).stop 2024-09-15T23:29:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.477+0000 7ff12f0a5640 1 -- 172.21.15.89:0/537931130 shutdown_connections 2024-09-15T23:29:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.477+0000 7ff12f0a5640 1 --2- 172.21.15.89:0/537931130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1281025f0 0x7ff128108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.477+0000 7ff12f0a5640 1 --2- 172.21.15.89:0/537931130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff128101c30 0x7ff1281020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.477+0000 7ff12f0a5640 1 --2- 172.21.15.89:0/537931130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff128100a40 0x7ff128100e40 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.477+0000 7ff12f0a5640 1 -- 172.21.15.89:0/537931130 >> 172.21.15.89:0/537931130 conn(0x7ff1280fc1e0 msgr2=0x7ff1280fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.477+0000 7ff12f0a5640 1 -- 172.21.15.89:0/537931130 shutdown_connections 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.477+0000 7ff12f0a5640 1 -- 172.21.15.89:0/537931130 wait complete. 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12f0a5640 1 Processor -- start 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12f0a5640 1 -- start start 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12f0a5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff128100a40 0x7ff128197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12f0a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff128101c30 0x7ff128198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12f0a5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1281025f0 0x7ff12819f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12f0a5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff12810c6c0 con 0x7ff1281025f0 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12f0a5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff12810c540 con 0x7ff128101c30 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12f0a5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff12810c840 con 0x7ff128100a40 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12d642640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1281025f0 0x7ff12819f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12d642640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1281025f0 0x7ff12819f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36954/0 (socket says 172.21.15.89:36954) 2024-09-15T23:29:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12d642640 1 -- 172.21.15.89:0/100465695 learned_addr learned my addr 172.21.15.89:0/100465695 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12ce41640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff128100a40 0x7ff128197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff127fff640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff128101c30 0x7ff128198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12d642640 1 -- 172.21.15.89:0/100465695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff128100a40 msgr2=0x7ff128197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12d642640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff128100a40 0x7ff128197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12d642640 1 -- 172.21.15.89:0/100465695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff128101c30 msgr2=0x7ff128198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12d642640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff128101c30 0x7ff128198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12d642640 1 -- 172.21.15.89:0/100465695 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12819fa40 con 0x7ff1281025f0 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.478+0000 7ff12d642640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1281025f0 0x7ff12819f3d0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ff12000eaa0 tx=0x7ff12000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.479+0000 7ff125ffb640 1 -- 172.21.15.89:0/100465695 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12000cdb0 con 0x7ff1281025f0 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.479+0000 7ff125ffb640 1 -- 172.21.15.89:0/100465695 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff120005020 con 0x7ff1281025f0 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.479+0000 7ff125ffb640 1 -- 172.21.15.89:0/100465695 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1200116b0 con 0x7ff1281025f0 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.479+0000 7ff12f0a5640 1 -- 172.21.15.89:0/100465695 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff12819fd30 con 0x7ff1281025f0 2024-09-15T23:29:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.479+0000 7ff12f0a5640 1 -- 172.21.15.89:0/100465695 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1281a0220 con 0x7ff1281025f0 2024-09-15T23:29:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.481+0000 7ff131e44740 1 Processor -- start 2024-09-15T23:29:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.481+0000 7ff125ffb640 1 -- 172.21.15.89:0/100465695 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff120005560 con 0x7ff1281025f0 2024-09-15T23:29:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.481+0000 7ff131e44740 1 -- start start 2024-09-15T23:29:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff131e44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa7f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff131e44740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 0x556a6aa841b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff131e44740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa84780 0x556a6aa86c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff131e44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556a6a9de2f0 con 0x556a6aa7edb0 2024-09-15T23:29:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff131e44740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556a6a986270 con 0x556a6aa84780 2024-09-15T23:29:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff131e44740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556a6a9da2b0 con 0x556a6aa7f6f0 2024-09-15T23:29:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff127fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa7f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff127fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa7f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36960/0 (socket says 172.21.15.89:36960) 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff127fff640 1 -- 172.21.15.89:0/3409467769 learned_addr learned my addr 172.21.15.89:0/3409467769 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff12ce41640 1 --2- 172.21.15.89:0/3409467769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 0x556a6aa841b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff127fff640 1 --2- 172.21.15.89:0/3409467769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa84780 0x556a6aa86c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff127fff640 1 -- 172.21.15.89:0/3409467769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 msgr2=0x556a6aa841b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff125ffb640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1180a3370 0x7ff1180a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff127fff640 1 --2- 172.21.15.89:0/3409467769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 0x556a6aa841b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff127fff640 1 -- 172.21.15.89:0/3409467769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa84780 msgr2=0x556a6aa86c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff127fff640 1 --2- 172.21.15.89:0/3409467769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa84780 0x556a6aa86c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff127fff640 1 -- 172.21.15.89:0/3409467769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a6a985340 con 0x556a6aa7edb0 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff12d642640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1180a3370 0x7ff1180a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.482+0000 7ff127fff640 1 --2- 172.21.15.89:0/3409467769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa7f1b0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ff11400bc20 tx=0x7ff114004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff704222484e9da0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff125ffb640 1 -- 172.21.15.89:0/100465695 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff1200cfe40 con 0x7ff1281025f0 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff112ffd640 1 -- 172.21.15.89:0/3409467769 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114005020 con 0x556a6aa7edb0 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff112ffd640 1 -- 172.21.15.89:0/3409467769 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1140056a0 con 0x556a6aa7edb0 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff112ffd640 1 -- 172.21.15.89:0/3409467769 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1140106a0 con 0x556a6aa7edb0 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff131e44740 1 -- 172.21.15.89:0/3409467769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 msgr2=0x556a6aa7f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff131e44740 1 --2- 172.21.15.89:0/3409467769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa7f1b0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ff11400bc20 tx=0x7ff114004250 comp rx=0 tx=0).stop 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff12d642640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1180a3370 0x7ff1180a5830 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff120096bc0 tx=0x7ff120086a60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff131e44740 1 -- 172.21.15.89:0/3409467769 shutdown_connections 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff131e44740 1 --2- 172.21.15.89:0/3409467769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa84780 0x556a6aa86c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff131e44740 1 --2- 172.21.15.89:0/3409467769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 0x556a6aa841b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff131e44740 1 --2- 172.21.15.89:0/3409467769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa7f1b0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff131e44740 1 -- 172.21.15.89:0/3409467769 >> 172.21.15.89:0/3409467769 conn(0x556a6aa76010 msgr2=0x556a6aa76410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff131e44740 1 -- 172.21.15.89:0/3409467769 shutdown_connections 2024-09-15T23:29:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.483+0000 7ff131e44740 1 -- 172.21.15.89:0/3409467769 wait complete. 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff131e44740 1 -- start start 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff131e44740 1 Processor -- start 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff131e44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa88ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff131e44740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa7f6f0 0x556a6aa89400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff131e44740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa84780 0x556a6aa90440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff131e44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556a6aa876f0 con 0x556a6aa7edb0 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff131e44740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556a6a967290 con 0x556a6aa7f6f0 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff131e44740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556a6aa87870 con 0x556a6aa84780 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff131e44740 10 client.0 _ll_register_callbacks cb 0x556a6a998140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff127fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa88ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff127fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa88ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36972/0 (socket says 172.21.15.89:36972) 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff127fff640 1 -- 172.21.15.89:0/1235220120 learned_addr learned my addr 172.21.15.89:0/1235220120 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.484+0000 7ff127fff640 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa84780 0x556a6aa90440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff12ce41640 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa7f6f0 0x556a6aa89400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff127fff640 1 -- 172.21.15.89:0/1235220120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa84780 msgr2=0x556a6aa90440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff127fff640 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa84780 0x556a6aa90440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff127fff640 1 -- 172.21.15.89:0/1235220120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa7f6f0 msgr2=0x556a6aa89400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff127fff640 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa7f6f0 0x556a6aa89400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff127fff640 1 -- 172.21.15.89:0/1235220120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a6a6f1210 con 0x556a6aa7edb0 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff127fff640 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa88ec0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff114006010 tx=0x7ff114004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7fb95f0ccca40 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff107fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff107fff640 1 -- 172.21.15.89:0/1235220120 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114005430 con 0x556a6aa7edb0 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff107fff640 1 -- 172.21.15.89:0/1235220120 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff114012890 con 0x556a6aa7edb0 2024-09-15T23:29:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff131e44740 1 -- 172.21.15.89:0/1235220120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1140056a0 con 0x556a6aa7edb0 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff107fff640 1 -- 172.21.15.89:0/1235220120 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff11402f050 con 0x556a6aa7edb0 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff131e44740 10 client.17472 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff131e44740 1 -- 172.21.15.89:0/1235220120 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556a6a7bd620 con 0x556a6aa7edb0 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.485+0000 7ff107fff640 1 -- 172.21.15.89:0/1235220120 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1140056a0 con 0x556a6aa7edb0 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.486+0000 7ff107fff640 1 -- 172.21.15.89:0/1235220120 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff1140124a0 con 0x556a6aa7edb0 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.486+0000 7ff131e44740 10 client.17472 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.486+0000 7ff131e44740 10 client.17472 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.486+0000 7ff131e44740 20 client.17472 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.486+0000 7ff131e44740 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a6aa948b0 0x556a6aa96cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.486+0000 7ff131e44740 4 client.17472 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.486+0000 7ff131e44740 1 -- 172.21.15.89:0/1235220120 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a6a9eba20 con 0x556a6aa948b0 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.486+0000 7ff12ce41640 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a6aa948b0 0x556a6aa96cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.486+0000 7ff107fff640 10 client.17472 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.486+0000 7ff12ce41640 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a6aa948b0 0x556a6aa96cd0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae261512d3cd9051 server_cookie=fb3ee4b64f3b541b in_seq=0 out_seq=0 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff107fff640 1 -- 172.21.15.89:0/1235220120 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556a6a9eba20 con 0x556a6aa948b0 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff107fff640 10 client.17472 handle_command_reply: tid=0 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 client.17472 shutdown 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 -- 172.21.15.89:0/1235220120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 msgr2=0x556a6aa88ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa88ec0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7ff114006010 tx=0x7ff114004bc0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 -- 172.21.15.89:0/1235220120 shutdown_connections 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a6aa948b0 0x556a6aa96cd0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa84780 0x556a6aa90440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff107fff640 0 client.17472 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa7f6f0 0x556a6aa89400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 --2- 172.21.15.89:0/1235220120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa88ec0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 -- 172.21.15.89:0/1235220120 >> 172.21.15.89:0/1235220120 conn(0x556a6aa76010 msgr2=0x556a6aa77820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 -- 172.21.15.89:0/1235220120 shutdown_connections 2024-09-15T23:29:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.487+0000 7ff131e44740 1 -- 172.21.15.89:0/1235220120 wait complete. 2024-09-15T23:29:56.496 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.488+0000 7ff131e44740 20 client.17472 trim_cache size 0 max 16384 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.504+0000 7ff131e44740 1 Processor -- start 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.504+0000 7ff131e44740 1 -- start start 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.504+0000 7ff131e44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa89490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.504+0000 7ff131e44740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 0x556a6aa90400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.504+0000 7ff131e44740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa84780 0x556a6aa759b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.504+0000 7ff131e44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281a8670 con 0x556a6aa7edb0 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.504+0000 7ff131e44740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281aa1b0 con 0x556a6aa84780 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.504+0000 7ff131e44740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281a8d40 con 0x556a6aa7f6f0 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff127fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa89490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff127fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa89490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36994/0 (socket says 172.21.15.89:36994) 2024-09-15T23:29:56.507 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff127fff640 1 -- 172.21.15.89:0/425919861 learned_addr learned my addr 172.21.15.89:0/425919861 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff12ce41640 1 --2- 172.21.15.89:0/425919861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 0x556a6aa90400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff127fff640 1 -- 172.21.15.89:0/425919861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 msgr2=0x556a6aa90400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff127fff640 1 --2- 172.21.15.89:0/425919861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 0x556a6aa90400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff127fff640 1 -- 172.21.15.89:0/425919861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa84780 msgr2=0x556a6aa759b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff127fff640 1 --2- 172.21.15.89:0/425919861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa84780 0x556a6aa759b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff127fff640 1 -- 172.21.15.89:0/425919861 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a6aa88a00 con 0x556a6aa7edb0 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff12ce41640 1 --2- 172.21.15.89:0/425919861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 0x556a6aa90400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff127fff640 1 --2- 172.21.15.89:0/425919861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa89490 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff11401d530 tx=0x7ff114004c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd5268a0656c850 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.505+0000 7ff1127fc640 1 -- 172.21.15.89:0/425919861 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114004120 con 0x556a6aa7edb0 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.506+0000 7ff1127fc640 1 -- 172.21.15.89:0/425919861 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff11400be50 con 0x556a6aa7edb0 2024-09-15T23:29:56.508 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.506+0000 7ff1127fc640 1 -- 172.21.15.89:0/425919861 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11400b9d0 con 0x556a6aa7edb0 2024-09-15T23:29:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.507+0000 7ff131e44740 1 -- 172.21.15.89:0/425919861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 msgr2=0x556a6aa89490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.507+0000 7ff131e44740 1 --2- 172.21.15.89:0/425919861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa89490 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7ff11401d530 tx=0x7ff114004c30 comp rx=0 tx=0).stop 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.507+0000 7ff131e44740 1 -- 172.21.15.89:0/425919861 shutdown_connections 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.507+0000 7ff131e44740 1 --2- 172.21.15.89:0/425919861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa84780 0x556a6aa759b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.507+0000 7ff131e44740 1 --2- 172.21.15.89:0/425919861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa7f6f0 0x556a6aa90400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.507+0000 7ff131e44740 1 --2- 172.21.15.89:0/425919861 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa89490 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.507+0000 7ff131e44740 1 -- 172.21.15.89:0/425919861 >> 172.21.15.89:0/425919861 conn(0x556a6aa76010 msgr2=0x556a6aa76f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.507+0000 7ff131e44740 1 -- 172.21.15.89:0/425919861 shutdown_connections 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.507+0000 7ff131e44740 1 -- 172.21.15.89:0/425919861 wait complete. 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.507+0000 7ff131e44740 1 -- start start 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff131e44740 1 Processor -- start 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff131e44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa8d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff131e44740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa7f6f0 0x556a6aa8dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff131e44740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa84780 0x556a6aa8ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff131e44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281a9ae0 con 0x556a6aa7edb0 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff131e44740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281a9410 con 0x556a6aa7f6f0 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff131e44740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281a6650 con 0x556a6aa84780 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff131e44740 10 client.0 _ll_register_callbacks cb 0x7ff11c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff127fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa8d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff127fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa8d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37012/0 (socket says 172.21.15.89:37012) 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff127fff640 1 -- 172.21.15.89:0/526571136 learned_addr learned my addr 172.21.15.89:0/526571136 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff12ce41640 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa7f6f0 0x556a6aa8dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.508+0000 7ff127fff640 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa84780 0x556a6aa8ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.509+0000 7ff127fff640 1 -- 172.21.15.89:0/526571136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa84780 msgr2=0x556a6aa8ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.509+0000 7ff127fff640 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa84780 0x556a6aa8ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.509+0000 7ff127fff640 1 -- 172.21.15.89:0/526571136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa7f6f0 msgr2=0x556a6aa8dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.509+0000 7ff127fff640 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa7f6f0 0x556a6aa8dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.509+0000 7ff127fff640 1 -- 172.21.15.89:0/526571136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a6aa712d0 con 0x556a6aa7edb0 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.509+0000 7ff127fff640 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa8d7a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff11402c000 tx=0x7ff114004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5897f3805d88861e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.509+0000 7ff107fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.509+0000 7ff107fff640 1 -- 172.21.15.89:0/526571136 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114004120 con 0x556a6aa7edb0 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.509+0000 7ff107fff640 1 -- 172.21.15.89:0/526571136 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff114005230 con 0x556a6aa7edb0 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.509+0000 7ff131e44740 1 -- 172.21.15.89:0/526571136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff11400be50 con 0x556a6aa7edb0 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff107fff640 1 -- 172.21.15.89:0/526571136 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff114045050 con 0x556a6aa7edb0 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff131e44740 10 client.17487 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff131e44740 1 -- 172.21.15.89:0/526571136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556a6aa90440 con 0x556a6aa7edb0 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff107fff640 1 -- 172.21.15.89:0/526571136 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff114005430 con 0x556a6aa7edb0 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff107fff640 1 -- 172.21.15.89:0/526571136 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff114020a70 con 0x556a6aa7edb0 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff131e44740 10 client.17487 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff131e44740 10 client.17487 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:56.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff131e44740 20 client.17487 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff131e44740 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a6aaffa60 0x556a6ab01e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff131e44740 4 client.17487 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:29:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.510+0000 7ff131e44740 1 -- 172.21.15.89:0/526571136 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7ff11400be50 con 0x556a6aaffa60 2024-09-15T23:29:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.511+0000 7ff12ce41640 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a6aaffa60 0x556a6ab01e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.511+0000 7ff107fff640 10 client.17487 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.511+0000 7ff12ce41640 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a6aaffa60 0x556a6ab01e80 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ce57fdf8ac81d76 server_cookie=ba39e616757cd237 in_seq=0 out_seq=0 2024-09-15T23:29:56.523 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff107fff640 1 -- 172.21.15.89:0/526571136 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7ff11400be50 con 0x556a6aaffa60 2024-09-15T23:29:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff107fff640 10 client.17487 handle_command_reply: tid=0 2024-09-15T23:29:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 client.17487 shutdown 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 -- 172.21.15.89:0/526571136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 msgr2=0x556a6aa8d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa8d7a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7ff11402c000 tx=0x7ff114004f80 comp rx=0 tx=0).stop 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 -- 172.21.15.89:0/526571136 shutdown_connections 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556a6aaffa60 0x556a6ab01e80 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a6aa84780 0x556a6aa8ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff107fff640 0 client.17487 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556a6aa7f6f0 0x556a6aa8dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 --2- 172.21.15.89:0/526571136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556a6aa7edb0 0x556a6aa8d7a0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 -- 172.21.15.89:0/526571136 >> 172.21.15.89:0/526571136 conn(0x556a6aa76010 msgr2=0x556a6aa77740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 -- 172.21.15.89:0/526571136 shutdown_connections 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 1 -- 172.21.15.89:0/526571136 wait complete. 2024-09-15T23:29:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.522+0000 7ff131e44740 20 client.17487 trim_cache size 0 max 16384 2024-09-15T23:29:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:29:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:29:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:29:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:29:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.523+0000 7ff12f0a5640 1 -- 172.21.15.89:0/100465695 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1180a3370 msgr2=0x7ff1180a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.523+0000 7ff12f0a5640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1180a3370 0x7ff1180a5830 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff120096bc0 tx=0x7ff120086a60 comp rx=0 tx=0).stop 2024-09-15T23:29:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.523+0000 7ff12f0a5640 1 -- 172.21.15.89:0/100465695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1281025f0 msgr2=0x7ff12819f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.523+0000 7ff12f0a5640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1281025f0 0x7ff12819f3d0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7ff12000eaa0 tx=0x7ff12000ef60 comp rx=0 tx=0).stop 2024-09-15T23:29:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.523+0000 7ff12f0a5640 1 -- 172.21.15.89:0/100465695 shutdown_connections 2024-09-15T23:29:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.523+0000 7ff12f0a5640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1180a3370 0x7ff1180a5830 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.523+0000 7ff12f0a5640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1281025f0 0x7ff12819f3d0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.523+0000 7ff12f0a5640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff128101c30 0x7ff128198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.523+0000 7ff12f0a5640 1 --2- 172.21.15.89:0/100465695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff128100a40 0x7ff128197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.523+0000 7ff12f0a5640 1 -- 172.21.15.89:0/100465695 >> 172.21.15.89:0/100465695 conn(0x7ff1280fc1e0 msgr2=0x7ff1280fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.500+0000 7f3437af5640 1 -- 172.21.15.89:0/1126708722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3430106a70 msgr2=0x7f3430106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.500+0000 7f3437af5640 1 --2- 172.21.15.89:0/1126708722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3430106a70 0x7f3430106e50 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f342c009930 tx=0x7f342c02a420 comp rx=0 tx=0).stop 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.500+0000 7f3437af5640 1 -- 172.21.15.89:0/1126708722 shutdown_connections 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.500+0000 7f3437af5640 1 --2- 172.21.15.89:0/1126708722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3430101410 0x7f343010d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.500+0000 7f3437af5640 1 --2- 172.21.15.89:0/1126708722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3430100a70 0x7f3430100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.500+0000 7f3437af5640 1 --2- 172.21.15.89:0/1126708722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3430106a70 0x7f3430106e50 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.500+0000 7f3437af5640 1 -- 172.21.15.89:0/1126708722 >> 172.21.15.89:0/1126708722 conn(0x7f34300fc790 msgr2=0x7f34300febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.501+0000 7f3437af5640 1 -- 172.21.15.89:0/1126708722 shutdown_connections 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.501+0000 7f3437af5640 1 -- 172.21.15.89:0/1126708722 wait complete. 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.501+0000 7f3437af5640 1 Processor -- start 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3437af5640 1 -- start start 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3437af5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3430100a70 0x7f3430198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3437af5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3430101410 0x7f3430198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3437af5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3430106a70 0x7f343019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3437af5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3430111f50 con 0x7f3430106a70 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3437af5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3430111dd0 con 0x7f3430101410 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3437af5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f34301120d0 con 0x7f3430100a70 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3436092640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3430106a70 0x7f343019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3436092640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3430106a70 0x7f343019cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36992/0 (socket says 172.21.15.89:36992) 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3436092640 1 -- 172.21.15.89:0/2375796010 learned_addr learned my addr 172.21.15.89:0/2375796010 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3435090640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3430101410 0x7f3430198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3435891640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3430100a70 0x7f3430198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3436092640 1 -- 172.21.15.89:0/2375796010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3430100a70 msgr2=0x7f3430198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3436092640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3430100a70 0x7f3430198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3436092640 1 -- 172.21.15.89:0/2375796010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3430101410 msgr2=0x7f3430198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3436092640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3430101410 0x7f3430198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.502+0000 7f3436092640 1 -- 172.21.15.89:0/2375796010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343019d3c0 con 0x7f3430106a70 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.503+0000 7f3436092640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3430106a70 0x7f343019cce0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f342800cad0 tx=0x7f342800ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.503+0000 7f3426ffd640 1 -- 172.21.15.89:0/2375796010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342800ad60 con 0x7f3430106a70 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.503+0000 7f3426ffd640 1 -- 172.21.15.89:0/2375796010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3428005020 con 0x7f3430106a70 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.503+0000 7f3437af5640 1 -- 172.21.15.89:0/2375796010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343019d6b0 con 0x7f3430106a70 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.503+0000 7f3426ffd640 1 -- 172.21.15.89:0/2375796010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342800f6b0 con 0x7f3430106a70 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.503+0000 7f3437af5640 1 -- 172.21.15.89:0/2375796010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34301a4f90 con 0x7f3430106a70 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.506+0000 7f3426ffd640 1 -- 172.21.15.89:0/2375796010 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3428005560 con 0x7f3430106a70 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.506+0000 7f3439ece740 1 Processor -- start 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.506+0000 7f3439ece740 1 -- start start 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.506+0000 7f3426ffd640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f341c0a32c0 0x7f341c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3439ece740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf35ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3426ffd640 1 -- 172.21.15.89:0/2375796010 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f34280cfc00 con 0x7f3430106a70 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3439ece740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf36010 0x563a2bf44900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3439ece740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf44e40 0x563a2bf47230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3439ece740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563a2be9a160 con 0x563a2bf36010 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3439ece740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563a2be41720 con 0x563a2bf44e40 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3439ece740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563a2be96120 con 0x563a2bf356f0 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435090640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f341c0a32c0 0x7f341c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435891640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf35ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435891640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf35ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:34292/0 (socket says 172.21.15.89:34292) 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435891640 1 -- 172.21.15.89:0/1743255343 learned_addr learned my addr 172.21.15.89:0/1743255343 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435090640 1 --2- 172.21.15.89:0/1743255343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf36010 0x563a2bf44900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435891640 1 -- 172.21.15.89:0/1743255343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf44e40 msgr2=0x563a2bf47230 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435891640 1 --2- 172.21.15.89:0/1743255343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf44e40 0x563a2bf47230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435090640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f341c0a32c0 0x7f341c0a5780 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f34300fed70 tx=0x7f3420013940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435891640 1 -- 172.21.15.89:0/1743255343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf36010 msgr2=0x563a2bf44900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435891640 1 --2- 172.21.15.89:0/1743255343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf36010 0x563a2bf44900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.507+0000 7f3435891640 1 -- 172.21.15.89:0/1743255343 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a2bc0bba0 con 0x563a2bf356f0 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.508+0000 7f3435891640 1 --2- 172.21.15.89:0/1743255343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf35ad0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f342c02a940 tx=0x7f342c0316e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d13e43695887c61 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.508+0000 7f341bfff640 1 -- 172.21.15.89:0/1743255343 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c040070 con 0x563a2bf356f0 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.508+0000 7f341bfff640 1 -- 172.21.15.89:0/1743255343 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f342c005020 con 0x563a2bf356f0 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.508+0000 7f341bfff640 1 -- 172.21.15.89:0/1743255343 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c035040 con 0x563a2bf356f0 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.509+0000 7f3439ece740 1 -- 172.21.15.89:0/1743255343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 msgr2=0x563a2bf35ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.509+0000 7f3439ece740 1 --2- 172.21.15.89:0/1743255343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf35ad0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f342c02a940 tx=0x7f342c0316e0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.509+0000 7f3439ece740 1 -- 172.21.15.89:0/1743255343 shutdown_connections 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.509+0000 7f3439ece740 1 --2- 172.21.15.89:0/1743255343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf44e40 0x563a2bf47230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.509+0000 7f3439ece740 1 --2- 172.21.15.89:0/1743255343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf36010 0x563a2bf44900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.509+0000 7f3439ece740 1 --2- 172.21.15.89:0/1743255343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf35ad0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.509+0000 7f3439ece740 1 -- 172.21.15.89:0/1743255343 >> 172.21.15.89:0/1743255343 conn(0x563a2bf31de0 msgr2=0x563a2bf321c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.509+0000 7f3439ece740 1 -- 172.21.15.89:0/1743255343 shutdown_connections 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.509+0000 7f3439ece740 1 -- 172.21.15.89:0/1743255343 wait complete. 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.509+0000 7f3439ece740 1 -- start start 2024-09-15T23:29:56.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.510+0000 7f3439ece740 1 Processor -- start 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.510+0000 7f3439ece740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf41db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.510+0000 7f3439ece740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 0x563a2bf422f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.510+0000 7f3439ece740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf44e40 0x563a2bf42830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3439ece740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563a2bf47a10 con 0x563a2bf44e40 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3439ece740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563a2be22900 con 0x563a2bf36010 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3439ece740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563a2bf47b90 con 0x563a2bf356f0 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.510+0000 7f3435891640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf41db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3435891640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf41db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:34320/0 (socket says 172.21.15.89:34320) 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3435891640 1 -- 172.21.15.89:0/435090778 learned_addr learned my addr 172.21.15.89:0/435090778 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3439ece740 10 client.0 _ll_register_callbacks cb 0x563a2be20780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3435090640 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 0x563a2bf422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3436092640 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf44e40 0x563a2bf42830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3435891640 1 -- 172.21.15.89:0/435090778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 msgr2=0x563a2bf422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3435891640 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 0x563a2bf422f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3435891640 1 -- 172.21.15.89:0/435090778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf44e40 msgr2=0x563a2bf42830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3435891640 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf44e40 0x563a2bf42830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3435891640 1 -- 172.21.15.89:0/435090778 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a2be62c50 con 0x563a2bf356f0 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3435891640 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf41db0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f342c004800 tx=0x7f342c031b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b2547be8c7eeeb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f34197fa640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f34197fa640 1 -- 172.21.15.89:0/435090778 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c040070 con 0x563a2bf356f0 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f3439ece740 1 -- 172.21.15.89:0/435090778 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f342c005020 con 0x563a2bf356f0 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.511+0000 7f34197fa640 1 -- 172.21.15.89:0/435090778 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f342c0053e0 con 0x563a2bf356f0 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f34197fa640 1 -- 172.21.15.89:0/435090778 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f342c035050 con 0x563a2bf356f0 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f3439ece740 10 client.26350 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f3439ece740 1 -- 172.21.15.89:0/435090778 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563a2bf47a10 con 0x563a2bf356f0 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f34197fa640 1 -- 172.21.15.89:0/435090778 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f342c005020 con 0x563a2bf356f0 2024-09-15T23:29:56.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f34197fa640 1 -- 172.21.15.89:0/435090778 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f342c03c930 con 0x563a2bf356f0 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f3439ece740 10 client.26350 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f3439ece740 10 client.26350 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f3439ece740 20 client.26350 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f3439ece740 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a2bf548e0 0x563a2bf56cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f3439ece740 4 client.26350 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f3439ece740 1 -- 172.21.15.89:0/435090778 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a2bea7890 con 0x563a2bf548e0 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.512+0000 7f3435090640 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a2bf548e0 0x563a2bf56cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.513+0000 7f34197fa640 10 client.26350 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.513+0000 7f3435090640 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a2bf548e0 0x563a2bf56cd0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee80ae47ca64ff98 server_cookie=2d89d82d33783604 in_seq=0 out_seq=0 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.515+0000 7f34197fa640 1 -- 172.21.15.89:0/435090778 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563a2bea7890 con 0x563a2bf548e0 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.515+0000 7f34197fa640 10 client.26350 handle_command_reply: tid=0 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.515+0000 7f3439ece740 1 client.26350 shutdown 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f3439ece740 1 -- 172.21.15.89:0/435090778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 msgr2=0x563a2bf41db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f3439ece740 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf41db0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f342c004800 tx=0x7f342c031b40 comp rx=0 tx=0).stop 2024-09-15T23:29:56.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f3439ece740 1 -- 172.21.15.89:0/435090778 shutdown_connections 2024-09-15T23:29:56.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f3439ece740 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a2bf548e0 0x563a2bf56cd0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f3439ece740 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf44e40 0x563a2bf42830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f3439ece740 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 0x563a2bf422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f3439ece740 1 --2- 172.21.15.89:0/435090778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf356f0 0x563a2bf41db0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f3439ece740 1 -- 172.21.15.89:0/435090778 >> 172.21.15.89:0/435090778 conn(0x563a2bf31de0 msgr2=0x563a2bf37b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f34197fa640 0 client.26350 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:56.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f3439ece740 1 -- 172.21.15.89:0/435090778 shutdown_connections 2024-09-15T23:29:56.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.516+0000 7f3439ece740 1 -- 172.21.15.89:0/435090778 wait complete. 2024-09-15T23:29:56.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.517+0000 7f3439ece740 20 client.26350 trim_cache size 0 max 16384 2024-09-15T23:29:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.524+0000 7ff12f0a5640 1 -- 172.21.15.89:0/100465695 shutdown_connections 2024-09-15T23:29:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:29:56.524+0000 7ff12f0a5640 1 -- 172.21.15.89:0/100465695 wait complete. 2024-09-15T23:29:56.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.538+0000 7f3439ece740 1 Processor -- start 2024-09-15T23:29:56.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.538+0000 7f3439ece740 1 -- start start 2024-09-15T23:29:56.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.538+0000 7f3439ece740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf31de0 0x563a2bf30d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.538+0000 7f3439ece740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf406a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.538+0000 7f3439ece740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf36010 0x563a2bf42680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.538+0000 7f3439ece740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f34301acd10 con 0x563a2bf356f0 2024-09-15T23:29:56.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.538+0000 7f3439ece740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34301ae850 con 0x563a2bf31de0 2024-09-15T23:29:56.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.538+0000 7f3439ece740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f34301ad3e0 con 0x563a2bf36010 2024-09-15T23:29:56.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3435090640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3435090640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf406a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37036/0 (socket says 172.21.15.89:37036) 2024-09-15T23:29:56.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3435090640 1 -- 172.21.15.89:0/3590396135 learned_addr learned my addr 172.21.15.89:0/3590396135 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:56.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3436092640 1 --2- 172.21.15.89:0/3590396135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf36010 0x563a2bf42680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3435891640 1 --2- 172.21.15.89:0/3590396135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf31de0 0x563a2bf30d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3435090640 1 -- 172.21.15.89:0/3590396135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf36010 msgr2=0x563a2bf42680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3435090640 1 --2- 172.21.15.89:0/3590396135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf36010 0x563a2bf42680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3435090640 1 -- 172.21.15.89:0/3590396135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf31de0 msgr2=0x563a2bf30d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3435090640 1 --2- 172.21.15.89:0/3590396135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf31de0 0x563a2bf30d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3435090640 1 -- 172.21.15.89:0/3590396135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a2bf40320 con 0x563a2bf356f0 2024-09-15T23:29:56.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f3435090640 1 --2- 172.21.15.89:0/3590396135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf406a0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f342000a7a0 tx=0x7f342001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d00175c0dfb860d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f341b7fe640 1 -- 172.21.15.89:0/3590396135 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3420030030 con 0x563a2bf356f0 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f341b7fe640 1 -- 172.21.15.89:0/3590396135 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f342001fe50 con 0x563a2bf356f0 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.539+0000 7f341b7fe640 1 -- 172.21.15.89:0/3590396135 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342002c8b0 con 0x563a2bf356f0 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.540+0000 7f3439ece740 1 -- 172.21.15.89:0/3590396135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 msgr2=0x563a2bf406a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.540+0000 7f3439ece740 1 --2- 172.21.15.89:0/3590396135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf406a0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f342000a7a0 tx=0x7f342001d000 comp rx=0 tx=0).stop 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.540+0000 7f3439ece740 1 -- 172.21.15.89:0/3590396135 shutdown_connections 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.540+0000 7f3439ece740 1 --2- 172.21.15.89:0/3590396135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf36010 0x563a2bf42680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.540+0000 7f3439ece740 1 --2- 172.21.15.89:0/3590396135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf406a0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.540+0000 7f3439ece740 1 --2- 172.21.15.89:0/3590396135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf31de0 0x563a2bf30d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.540+0000 7f3439ece740 1 -- 172.21.15.89:0/3590396135 >> 172.21.15.89:0/3590396135 conn(0x563a2bf56fd0 msgr2=0x563a2bf379e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.540+0000 7f3439ece740 1 -- 172.21.15.89:0/3590396135 shutdown_connections 2024-09-15T23:29:56.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.540+0000 7f3439ece740 1 -- 172.21.15.89:0/3590396135 wait complete. 2024-09-15T23:29:56.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3439ece740 1 -- start start 2024-09-15T23:29:56.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3439ece740 1 Processor -- start 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3439ece740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf31de0 0x563a2bf4ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3439ece740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf4d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3439ece740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 0x563a2bf49df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3439ece740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f34301ae180 con 0x563a2bf356f0 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3439ece740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34301adab0 con 0x563a2bf36010 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3439ece740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f34301aacf0 con 0x563a2bf31de0 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3439ece740 10 client.0 _ll_register_callbacks cb 0x7f342c0315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3435090640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf4d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3435090640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf4d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37046/0 (socket says 172.21.15.89:37046) 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3435090640 1 -- 172.21.15.89:0/1896800098 learned_addr learned my addr 172.21.15.89:0/1896800098 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.541+0000 7f3436092640 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 0x563a2bf49df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3435891640 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf31de0 0x563a2bf4ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3435090640 1 -- 172.21.15.89:0/1896800098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf31de0 msgr2=0x563a2bf4ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3435090640 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf31de0 0x563a2bf4ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3435090640 1 -- 172.21.15.89:0/1896800098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 msgr2=0x563a2bf49df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3435090640 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 0x563a2bf49df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3435090640 1 -- 172.21.15.89:0/1896800098 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a2bf4a330 con 0x563a2bf356f0 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3436092640 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 0x563a2bf49df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3435090640 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf4d360 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f342000a5c0 tx=0x7f34200116a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdffe0a85bf100e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f34197fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f34197fa640 1 -- 172.21.15.89:0/1896800098 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3420030050 con 0x563a2bf356f0 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f34197fa640 1 -- 172.21.15.89:0/1896800098 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f342001fb70 con 0x563a2bf356f0 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3439ece740 1 -- 172.21.15.89:0/1896800098 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f342001fe50 con 0x563a2bf356f0 2024-09-15T23:29:56.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f34197fa640 1 -- 172.21.15.89:0/1896800098 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3420005040 con 0x563a2bf356f0 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3439ece740 10 client.17496 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.542+0000 7f3439ece740 1 -- 172.21.15.89:0/1896800098 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563a2bfbfb80 con 0x563a2bf356f0 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f34197fa640 1 -- 172.21.15.89:0/1896800098 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3420031040 con 0x563a2bf356f0 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f34197fa640 1 -- 172.21.15.89:0/1896800098 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f34200046c0 con 0x563a2bf356f0 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f3439ece740 10 client.17496 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f3439ece740 10 client.17496 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f3439ece740 20 client.17496 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f3439ece740 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a2bf44e40 0x563a2bfc2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f3439ece740 4 client.17496 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}] 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f3439ece740 1 -- 172.21.15.89:0/1896800098 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}) -- 0x7f342001fe50 con 0x563a2bf44e40 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f3435891640 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a2bf44e40 0x563a2bfc2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f3435891640 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a2bf44e40 0x563a2bfc2240 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16364d5bb548c908 server_cookie=3b56014a29863f33 in_seq=0 out_seq=0 2024-09-15T23:29:56.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.543+0000 7f34197fa640 10 client.17496 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:56.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.554+0000 7f34197fa640 1 -- 172.21.15.89:0/1896800098 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f342001fe50 con 0x563a2bf44e40 2024-09-15T23:29:56.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.554+0000 7f34197fa640 10 client.17496 handle_command_reply: tid=0 2024-09-15T23:29:56.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.554+0000 7f3439ece740 1 client.17496 shutdown 2024-09-15T23:29:56.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.554+0000 7f3439ece740 1 -- 172.21.15.89:0/1896800098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 msgr2=0x563a2bf4d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.554+0000 7f3439ece740 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf4d360 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f342000a5c0 tx=0x7f34200116a0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3439ece740 1 -- 172.21.15.89:0/1896800098 shutdown_connections 2024-09-15T23:29:56.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3439ece740 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a2bf44e40 0x563a2bfc2240 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f34197fa640 0 client.17496 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:56.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3439ece740 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a2bf36010 0x563a2bf49df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3439ece740 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a2bf356f0 0x563a2bf4d360 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3439ece740 1 --2- 172.21.15.89:0/1896800098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a2bf31de0 0x563a2bf4ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3439ece740 1 -- 172.21.15.89:0/1896800098 >> 172.21.15.89:0/1896800098 conn(0x563a2bf56fd0 msgr2=0x563a2bf43e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3439ece740 1 -- 172.21.15.89:0/1896800098 shutdown_connections 2024-09-15T23:29:56.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3439ece740 1 -- 172.21.15.89:0/1896800098 wait complete. 2024-09-15T23:29:56.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3439ece740 20 client.17496 trim_cache size 0 max 16384 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 22, 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: "73e8cd96": { 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 22, 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:29:56.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 145.3, 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:29:56.559 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:29:56.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3437af5640 1 -- 172.21.15.89:0/2375796010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f341c0a32c0 msgr2=0x7f341c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3437af5640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f341c0a32c0 0x7f341c0a5780 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f34300fed70 tx=0x7f3420013940 comp rx=0 tx=0).stop 2024-09-15T23:29:56.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3437af5640 1 -- 172.21.15.89:0/2375796010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3430106a70 msgr2=0x7f343019cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3437af5640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3430106a70 0x7f343019cce0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f342800cad0 tx=0x7f342800ced0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3437af5640 1 -- 172.21.15.89:0/2375796010 shutdown_connections 2024-09-15T23:29:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3437af5640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f341c0a32c0 0x7f341c0a5780 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3437af5640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3430106a70 0x7f343019cce0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3437af5640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3430101410 0x7f3430198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3437af5640 1 --2- 172.21.15.89:0/2375796010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3430100a70 0x7f3430198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.555+0000 7f3437af5640 1 -- 172.21.15.89:0/2375796010 >> 172.21.15.89:0/2375796010 conn(0x7f34300fc790 msgr2=0x7f34300fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.556+0000 7f3437af5640 1 -- 172.21.15.89:0/2375796010 shutdown_connections 2024-09-15T23:29:56.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:56.556+0000 7f3437af5640 1 -- 172.21.15.89:0/2375796010 wait complete. 2024-09-15T23:29:56.565 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:57.277 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '73e8cd96', quiesce took 5.1 seconds. Will release after: 20.25 2024-09-15T23:29:57.459 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:57 smithi089 ceph-mon[30933]: pgmap v528: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.9 KiB/s wr, 15 op/s 2024-09-15T23:29:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:57 smithi120 ceph-mon[35250]: pgmap v528: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.9 KiB/s wr, 15 op/s 2024-09-15T23:29:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:57 smithi174 ceph-mon[35586]: pgmap v528: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.9 KiB/s wr, 15 op/s 2024-09-15T23:29:57.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.671+0000 7f1c88b8a640 1 -- 172.21.15.89:0/1058092397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c8410c170 msgr2=0x7f1c8410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.671+0000 7f1c88b8a640 1 --2- 172.21.15.89:0/1058092397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c8410c170 0x7f1c8410e610 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c009ca0 tx=0x7f1c7c02a950 comp rx=0 tx=0).stop 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.672+0000 7f1c88b8a640 1 -- 172.21.15.89:0/1058092397 shutdown_connections 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.672+0000 7f1c88b8a640 1 --2- 172.21.15.89:0/1058092397 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c8410c170 0x7f1c8410e610 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.672+0000 7f1c88b8a640 1 --2- 172.21.15.89:0/1058092397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c84103660 0x7f1c8410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.672+0000 7f1c88b8a640 1 --2- 172.21.15.89:0/1058092397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c84102d40 0x7f1c84103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.672+0000 7f1c88b8a640 1 -- 172.21.15.89:0/1058092397 >> 172.21.15.89:0/1058092397 conn(0x7f1c840fc7a0 msgr2=0x7f1c840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.672+0000 7f1c88b8a640 1 -- 172.21.15.89:0/1058092397 shutdown_connections 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.672+0000 7f1c88b8a640 1 -- 172.21.15.89:0/1058092397 wait complete. 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.672+0000 7f1c88b8a640 1 Processor -- start 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.672+0000 7f1c88b8a640 1 -- start start 2024-09-15T23:29:57.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.672+0000 7f1c88b8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c84102d40 0x7f1c841009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c88b8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c84103660 0x7f1c84100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c88b8a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c8410c170 0x7f1c84101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c88b8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1c84111f50 con 0x7f1c84102d40 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c88b8a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c84111dd0 con 0x7f1c84103660 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c88b8a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841120d0 con 0x7f1c8410c170 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c8259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c84102d40 0x7f1c841009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c8259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c84102d40 0x7f1c841009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37074/0 (socket says 172.21.15.89:37074) 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c8259c640 1 -- 172.21.15.89:0/4228020565 learned_addr learned my addr 172.21.15.89:0/4228020565 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c84103660 0x7f1c84100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c82d9d640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c8410c170 0x7f1c84101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c8259c640 1 -- 172.21.15.89:0/4228020565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c8410c170 msgr2=0x7f1c84101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:57.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c8259c640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c8410c170 0x7f1c84101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c8259c640 1 -- 172.21.15.89:0/4228020565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c84103660 msgr2=0x7f1c84100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c8259c640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c84103660 0x7f1c84100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c8259c640 1 -- 172.21.15.89:0/4228020565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c841a6830 con 0x7f1c84102d40 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c84103660 0x7f1c84100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c8259c640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c84102d40 0x7f1c841009a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400c920 tx=0x7f1c7400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c7b7fe640 1 -- 172.21.15.89:0/4228020565 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c74007c70 con 0x7f1c84102d40 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c7b7fe640 1 -- 172.21.15.89:0/4228020565 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c74007e10 con 0x7f1c84102d40 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c88b8a640 1 -- 172.21.15.89:0/4228020565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c841a6b20 con 0x7f1c84102d40 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c7b7fe640 1 -- 172.21.15.89:0/4228020565 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c74018890 con 0x7f1c84102d40 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.673+0000 7f1c88b8a640 1 -- 172.21.15.89:0/4228020565 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c841a70b0 con 0x7f1c84102d40 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.676+0000 7f1c7b7fe640 1 -- 172.21.15.89:0/4228020565 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1c74018a70 con 0x7f1c84102d40 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.676+0000 7f1c8af63740 1 Processor -- start 2024-09-15T23:29:57.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.676+0000 7f1c8af63740 1 -- start start 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.676+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 0x55a92c20d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.676+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c20da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c218370 0x55a92c21a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c8af63740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a92c1732f0 con 0x55a92c218370 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c8af63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a92c11acf0 con 0x55a92c20eb40 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c8af63740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a92c16f2b0 con 0x55a92c20d5d0 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c218370 0x55a92c21a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c8259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c20da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c218370 0x55a92c21a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37086/0 (socket says 172.21.15.89:37086) 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7bfff640 1 -- 172.21.15.89:0/58504735 learned_addr learned my addr 172.21.15.89:0/58504735 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7b7fe640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1c700a32c0 0x7f1c700a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c82d9d640 1 --2- 172.21.15.89:0/58504735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 0x55a92c20d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7bfff640 1 -- 172.21.15.89:0/58504735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 msgr2=0x55a92c20da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/58504735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c20da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c82d9d640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1c700a32c0 0x7f1c700a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7bfff640 1 -- 172.21.15.89:0/58504735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 msgr2=0x55a92c20d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/58504735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 0x55a92c20d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7bfff640 1 -- 172.21.15.89:0/58504735 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a92c0ff3a0 con 0x55a92c218370 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7b7fe640 1 -- 172.21.15.89:0/4228020565 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1c740cfec0 con 0x7f1c84102d40 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/58504735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c218370 0x55a92c21a760 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00bc60 tx=0x7f1c6c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef927633330a1b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c78ff9640 1 -- 172.21.15.89:0/58504735 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c005020 con 0x55a92c218370 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.677+0000 7f1c82d9d640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1c700a32c0 0x7f1c700a5780 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c02ae70 tx=0x7f1c7c0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c78ff9640 1 -- 172.21.15.89:0/58504735 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6c0047a0 con 0x55a92c218370 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c78ff9640 1 -- 172.21.15.89:0/58504735 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c0116a0 con 0x55a92c218370 2024-09-15T23:29:57.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c8af63740 1 -- 172.21.15.89:0/58504735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c218370 msgr2=0x55a92c21a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c8af63740 1 --2- 172.21.15.89:0/58504735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c218370 0x55a92c21a760 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00bc60 tx=0x7f1c6c00f730 comp rx=0 tx=0).stop 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c8af63740 1 -- 172.21.15.89:0/58504735 shutdown_connections 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c8af63740 1 --2- 172.21.15.89:0/58504735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c218370 0x55a92c21a760 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c8af63740 1 --2- 172.21.15.89:0/58504735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c20da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c8af63740 1 --2- 172.21.15.89:0/58504735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 0x55a92c20d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c8af63740 1 -- 172.21.15.89:0/58504735 >> 172.21.15.89:0/58504735 conn(0x55a92c1d6310 msgr2=0x55a92c209c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c8af63740 1 -- 172.21.15.89:0/58504735 shutdown_connections 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.678+0000 7f1c8af63740 1 -- 172.21.15.89:0/58504735 wait complete. 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8af63740 1 -- start start 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8af63740 1 Processor -- start 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c217ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c214f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c218370 0x55a92c215490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8af63740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a92c21b070 con 0x55a92c20eb40 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8af63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a92c12e060 con 0x55a92c218370 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8af63740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a92c21b1f0 con 0x55a92c20d5d0 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8af63740 10 client.0 _ll_register_callbacks cb 0x55a92be818c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c7bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c214f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c7bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c214f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37090/0 (socket says 172.21.15.89:37090) 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c7bfff640 1 -- 172.21.15.89:0/728345146 learned_addr learned my addr 172.21.15.89:0/728345146 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c82d9d640 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c218370 0x55a92c215490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8259c640 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c217ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c7bfff640 1 -- 172.21.15.89:0/728345146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 msgr2=0x55a92c217ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c217ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c7bfff640 1 -- 172.21.15.89:0/728345146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c218370 msgr2=0x55a92c215490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c218370 0x55a92c215490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c7bfff640 1 -- 172.21.15.89:0/728345146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a92bee43c0 con 0x55a92c20eb40 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.679+0000 7f1c8259c640 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c217ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c214f50 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c006010 tx=0x7f1c6c00bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e279810fa4515af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c69ffb640 1 -- 172.21.15.89:0/728345146 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c02b060 con 0x55a92c20eb40 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c69ffb640 1 -- 172.21.15.89:0/728345146 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1c6c004c20 con 0x55a92c20eb40 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c8af63740 1 -- 172.21.15.89:0/728345146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c6c0047a0 con 0x55a92c20eb40 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c69ffb640 1 -- 172.21.15.89:0/728345146 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1c6c032050 con 0x55a92c20eb40 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c8af63740 10 client.17508 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c8af63740 1 -- 172.21.15.89:0/728345146 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a92c226a90 con 0x55a92c20eb40 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c69ffb640 1 -- 172.21.15.89:0/728345146 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c6c0047a0 con 0x55a92c20eb40 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c69ffb640 1 -- 172.21.15.89:0/728345146 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1c6c026b20 con 0x55a92c20eb40 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c8af63740 10 client.17508 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:57.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c8af63740 10 client.17508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c8af63740 20 client.17508 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c8af63740 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a92c227e90 0x55a92c22a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c8af63740 4 client.17508 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.680+0000 7f1c8af63740 1 -- 172.21.15.89:0/728345146 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a92c180860 con 0x55a92c227e90 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.681+0000 7f1c8259c640 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a92c227e90 0x55a92c22a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.681+0000 7f1c69ffb640 10 client.17508 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.681+0000 7f1c8259c640 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a92c227e90 0x55a92c22a280 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32afd80fa2651a45 server_cookie=fe93f906cf38684b in_seq=0 out_seq=0 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c69ffb640 1 -- 172.21.15.89:0/728345146 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a92c180860 con 0x55a92c227e90 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c69ffb640 10 client.17508 handle_command_reply: tid=0 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 client.17508 shutdown 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 -- 172.21.15.89:0/728345146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 msgr2=0x55a92c214f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c214f50 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c006010 tx=0x7f1c6c00bf80 comp rx=0 tx=0).stop 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 -- 172.21.15.89:0/728345146 shutdown_connections 2024-09-15T23:29:57.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a92c227e90 0x55a92c22a280 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c218370 0x55a92c215490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c214f50 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c69ffb640 0 client.17508 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:57.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 --2- 172.21.15.89:0/728345146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c217ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 -- 172.21.15.89:0/728345146 >> 172.21.15.89:0/728345146 conn(0x55a92c1d6310 msgr2=0x55a92c20f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:57.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 -- 172.21.15.89:0/728345146 shutdown_connections 2024-09-15T23:29:57.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 1 -- 172.21.15.89:0/728345146 wait complete. 2024-09-15T23:29:57.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.682+0000 7f1c8af63740 20 client.17508 trim_cache size 0 max 16384 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.695+0000 7f1c8af63740 1 Processor -- start 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.695+0000 7f1c8af63740 1 -- start start 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c1d6310 0x55a92c224340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c216ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 0x55a92c22f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8af63740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b0f10 con 0x55a92c1d6310 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8af63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b2a50 con 0x55a92c20eb40 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8af63740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b15e0 con 0x55a92c20d5d0 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c1d6310 0x55a92c224340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c1d6310 0x55a92c224340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37106/0 (socket says 172.21.15.89:37106) 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8259c640 1 -- 172.21.15.89:0/539981719 learned_addr learned my addr 172.21.15.89:0/539981719 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:57.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c82d9d640 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 0x55a92c22f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c216ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8259c640 1 -- 172.21.15.89:0/539981719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 msgr2=0x55a92c216ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8259c640 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c216ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8259c640 1 -- 172.21.15.89:0/539981719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 msgr2=0x55a92c22f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8259c640 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 0x55a92c22f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8259c640 1 -- 172.21.15.89:0/539981719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a92c2134e0 con 0x55a92c1d6310 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c216ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c82d9d640 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 0x55a92c22f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c8259c640 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c1d6310 0x55a92c224340 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f1c740bc010 tx=0x7f1c740d2310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3c4c906e9fd8c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c6bfff640 1 -- 172.21.15.89:0/539981719 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7409c090 con 0x55a92c1d6310 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c6bfff640 1 -- 172.21.15.89:0/539981719 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c740a0440 con 0x55a92c1d6310 2024-09-15T23:29:57.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.696+0000 7f1c6bfff640 1 -- 172.21.15.89:0/539981719 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c74018d60 con 0x55a92c1d6310 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 -- 172.21.15.89:0/539981719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c1d6310 msgr2=0x55a92c224340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c1d6310 0x55a92c224340 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f1c740bc010 tx=0x7f1c740d2310 comp rx=0 tx=0).stop 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 -- 172.21.15.89:0/539981719 shutdown_connections 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20eb40 0x55a92c22f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c20d5d0 0x55a92c216ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 --2- 172.21.15.89:0/539981719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c1d6310 0x55a92c224340 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 -- 172.21.15.89:0/539981719 >> 172.21.15.89:0/539981719 conn(0x55a92c2155f0 msgr2=0x55a92c2167b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 -- 172.21.15.89:0/539981719 shutdown_connections 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 -- 172.21.15.89:0/539981719 wait complete. 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 -- start start 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.697+0000 7f1c8af63740 1 Processor -- start 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c1d6310 0x55a92c212060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20d5d0 0x55a92c2125a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c8af63740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c212ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c8af63740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b2380 con 0x55a92c20eb40 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c8af63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841b1cb0 con 0x55a92c20d5d0 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c8af63740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1c841aeef0 con 0x55a92c1d6310 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c8af63740 10 client.0 _ll_register_callbacks cb 0x7f1c6c015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c82d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c212ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c8259c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c1d6310 0x55a92c212060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c82d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c212ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37112/0 (socket says 172.21.15.89:37112) 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c82d9d640 1 -- 172.21.15.89:0/4058312690 learned_addr learned my addr 172.21.15.89:0/4058312690 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c7bfff640 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20d5d0 0x55a92c2125a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c82d9d640 1 -- 172.21.15.89:0/4058312690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c1d6310 msgr2=0x55a92c212060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c82d9d640 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c1d6310 0x55a92c212060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c82d9d640 1 -- 172.21.15.89:0/4058312690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20d5d0 msgr2=0x55a92c2125a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c82d9d640 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20d5d0 0x55a92c2125a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c82d9d640 1 -- 172.21.15.89:0/4058312690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a92c2919b0 con 0x55a92c20eb40 2024-09-15T23:29:57.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c82d9d640 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c212ae0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c035000 tx=0x7f1c7c02a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fc5fd713ff1367 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.698+0000 7f1c69ffb640 1 -- 172.21.15.89:0/4058312690 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c034040 con 0x55a92c20eb40 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c69ffb640 1 -- 172.21.15.89:0/4058312690 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1c7c004450 con 0x55a92c20eb40 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c8af63740 1 -- 172.21.15.89:0/4058312690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c740a0440 con 0x55a92c20eb40 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c69ffb640 1 -- 172.21.15.89:0/4058312690 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1c7c004650 con 0x55a92c20eb40 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c8af63740 10 client.17514 fetch_fsmap learned FSMap version 29 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c8af63740 1 -- 172.21.15.89:0/4058312690 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a92c292c00 con 0x55a92c20eb40 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c69ffb640 1 -- 172.21.15.89:0/4058312690 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c740a0440 con 0x55a92c20eb40 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c69ffb640 1 -- 172.21.15.89:0/4058312690 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1c7c039420 con 0x55a92c20eb40 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c8af63740 10 client.17514 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c8af63740 10 client.17514 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c8af63740 20 client.17514 populate_metadata read hostname 'smithi089' 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c8af63740 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a92c218370 0x55a92c2953f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c8af63740 4 client.17514 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:57.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c8af63740 1 -- 172.21.15.89:0/4058312690 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a92c17ad70 con 0x55a92c218370 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.699+0000 7f1c8259c640 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a92c218370 0x55a92c2953f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8259c640 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a92c218370 0x55a92c2953f0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57c9763f77c00d80 server_cookie=2f0e4bb66ae0f9d2 in_seq=0 out_seq=0 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c69ffb640 10 client.17514 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c69ffb640 1 -- 172.21.15.89:0/4058312690 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a92c17ad70 con 0x55a92c218370 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c69ffb640 10 client.17514 handle_command_reply: tid=0 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 client.17514 shutdown 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 -- 172.21.15.89:0/4058312690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 msgr2=0x55a92c212ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c212ae0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c035000 tx=0x7f1c7c02a950 comp rx=0 tx=0).stop 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 -- 172.21.15.89:0/4058312690 shutdown_connections 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a92c218370 0x55a92c2953f0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a92c20eb40 0x55a92c212ae0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a92c20d5d0 0x55a92c2125a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 --2- 172.21.15.89:0/4058312690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a92c1d6310 0x55a92c212060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c69ffb640 0 client.17514 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:29:57.712 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 -- 172.21.15.89:0/4058312690 >> 172.21.15.89:0/4058312690 conn(0x55a92c2155f0 msgr2=0x55a92c216fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:57.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 -- 172.21.15.89:0/4058312690 shutdown_connections 2024-09-15T23:29:57.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 1 -- 172.21.15.89:0/4058312690 wait complete. 2024-09-15T23:29:57.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.700+0000 7f1c8af63740 20 client.17514 trim_cache size 0 max 16384 2024-09-15T23:29:57.713 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:29:57.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 -- 172.21.15.89:0/4228020565 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1c700a32c0 msgr2=0x7f1c700a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1c700a32c0 0x7f1c700a5780 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c02ae70 tx=0x7f1c7c0095c0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 -- 172.21.15.89:0/4228020565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c84102d40 msgr2=0x7f1c841009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c84102d40 0x7f1c841009a0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400c920 tx=0x7f1c7400cde0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 -- 172.21.15.89:0/4228020565 shutdown_connections 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1c700a32c0 0x7f1c700a5780 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c8410c170 0x7f1c84101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c84103660 0x7f1c84100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 --2- 172.21.15.89:0/4228020565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c84102d40 0x7f1c841009a0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 -- 172.21.15.89:0/4228020565 >> 172.21.15.89:0/4228020565 conn(0x7f1c840fc7a0 msgr2=0x7f1c840fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 -- 172.21.15.89:0/4228020565 shutdown_connections 2024-09-15T23:29:57.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:29:57.701+0000 7f1c88b8a640 1 -- 172.21.15.89:0/4228020565 wait complete. 2024-09-15T23:29:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:59 smithi120 ceph-mon[35250]: pgmap v529: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-09-15T23:29:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:29:59 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:29:59.479 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:29:59.480 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:29:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:59 smithi174 ceph-mon[35586]: pgmap v529: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-09-15T23:29:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:29:59 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:29:59.502 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:59 smithi089 ceph-mon[30933]: pgmap v529: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-09-15T23:29:59.502 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:29:59 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:29:59.744 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:00.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/3076292859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bf4101de0 msgr2=0x7f2bf41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 --2- 172.21.15.89:0/3076292859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bf4101de0 0x7f2bf41021c0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009900 tx=0x7f2be402a410 comp rx=0 tx=0).stop 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/3076292859 shutdown_connections 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 --2- 172.21.15.89:0/3076292859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf4106f10 0x7f2bf410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 --2- 172.21.15.89:0/3076292859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf4102790 0x7f2bf41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 --2- 172.21.15.89:0/3076292859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bf4101de0 0x7f2bf41021c0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/3076292859 >> 172.21.15.89:0/3076292859 conn(0x7f2bf40fd8c0 msgr2=0x7f2bf40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/3076292859 shutdown_connections 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/3076292859 wait complete. 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 Processor -- start 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.228+0000 7f2bfa4b1640 1 -- start start 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bfa4b1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bf4101de0 0x7f2bf40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bfa4b1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf4102790 0x7f2bf4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bfa4b1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf4106f10 0x7f2bf4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bfa4b1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf4112010 con 0x7f2bf4101de0 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bfa4b1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf4111e90 con 0x7f2bf4102790 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bfa4b1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf4112190 con 0x7f2bf4106f10 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bf4101de0 0x7f2bf40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bf4101de0 0x7f2bf40744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37122/0 (socket says 172.21.15.89:37122) 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf3fff640 1 -- 172.21.15.89:0/1217638185 learned_addr learned my addr 172.21.15.89:0/1217638185 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf8a4e640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf4106f10 0x7f2bf4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bebfff640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf4102790 0x7f2bf4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf3fff640 1 -- 172.21.15.89:0/1217638185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf4106f10 msgr2=0x7f2bf4074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf4106f10 0x7f2bf4074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf3fff640 1 -- 172.21.15.89:0/1217638185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf4102790 msgr2=0x7f2bf4074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf4102790 0x7f2bf4074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf3fff640 1 -- 172.21.15.89:0/1217638185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf41a6a70 con 0x7f2bf4101de0 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bf4101de0 0x7f2bf40744f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f2be4005ab0 tx=0x7f2be40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf1ffb640 1 -- 172.21.15.89:0/1217638185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be403f030 con 0x7f2bf4101de0 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/1217638185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf41a6d00 con 0x7f2bf4101de0 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf1ffb640 1 -- 172.21.15.89:0/1217638185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be4003ce0 con 0x7f2bf4101de0 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.229+0000 7f2bf1ffb640 1 -- 172.21.15.89:0/1217638185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be4035070 con 0x7f2bf4101de0 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.230+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/1217638185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf41a7180 con 0x7f2bf4101de0 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.232+0000 7f2bf1ffb640 1 -- 172.21.15.89:0/1217638185 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2be4042030 con 0x7f2bf4101de0 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.232+0000 7f2bfc88a740 1 Processor -- start 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.232+0000 7f2bfc88a740 1 -- start start 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.232+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 0x5597c3bb1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.232+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bbc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.232+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 0x5597c3bbf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.232+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5597c3b17280 con 0x5597c3bb2480 2024-09-15T23:30:00.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.232+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5597c3abeb10 con 0x5597c3bbcd40 2024-09-15T23:30:00.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.232+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5597c3b13240 con 0x5597c3bb1b60 2024-09-15T23:30:00.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.232+0000 7f2bf1ffb640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2be00a3240 0x7f2be00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bbc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bbc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37134/0 (socket says 172.21.15.89:37134) 2024-09-15T23:30:00.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf3fff640 1 -- 172.21.15.89:0/266553409 learned_addr learned my addr 172.21.15.89:0/266553409 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:00.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf8a4e640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2be00a3240 0x7f2be00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf8a4e640 1 --2- 172.21.15.89:0/266553409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 0x5597c3bb1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf1ffb640 1 -- 172.21.15.89:0/1217638185 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2be40f0720 con 0x7f2bf4101de0 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bebfff640 1 --2- 172.21.15.89:0/266553409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 0x5597c3bbf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf3fff640 1 -- 172.21.15.89:0/266553409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 msgr2=0x5597c3bb1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/266553409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 0x5597c3bb1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf3fff640 1 -- 172.21.15.89:0/266553409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 msgr2=0x5597c3bbf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/266553409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 0x5597c3bbf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf3fff640 1 -- 172.21.15.89:0/266553409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597c3acb1f0 con 0x5597c3bb2480 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf8a4e640 1 --2- 172.21.15.89:0/266553409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 0x5597c3bb1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/266553409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bbc800 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f2be4003fd0 tx=0x7f2be408f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f075272f782bbf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bf8a4e640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2be00a3240 0x7f2be00a5700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2bec0070b0 tx=0x7f2bec0094c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:00.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bea7fc640 1 -- 172.21.15.89:0/266553409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be4049030 con 0x5597c3bb2480 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bea7fc640 1 -- 172.21.15.89:0/266553409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be40bc8f0 con 0x5597c3bb2480 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.233+0000 7f2bea7fc640 1 -- 172.21.15.89:0/266553409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be40edee0 con 0x5597c3bb2480 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.234+0000 7f2bfc88a740 1 -- 172.21.15.89:0/266553409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 msgr2=0x5597c3bbc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.234+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/266553409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bbc800 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f2be4003fd0 tx=0x7f2be408f770 comp rx=0 tx=0).stop 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.234+0000 7f2bfc88a740 1 -- 172.21.15.89:0/266553409 shutdown_connections 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.234+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/266553409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 0x5597c3bbf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.234+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/266553409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bbc800 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.234+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/266553409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 0x5597c3bb1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.234+0000 7f2bfc88a740 1 -- 172.21.15.89:0/266553409 >> 172.21.15.89:0/266553409 conn(0x5597c3bae0f0 msgr2=0x5597c3bae4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.234+0000 7f2bfc88a740 1 -- 172.21.15.89:0/266553409 shutdown_connections 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.234+0000 7f2bfc88a740 1 -- 172.21.15.89:0/266553409 wait complete. 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.234+0000 7f2bfc88a740 1 -- start start 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bfc88a740 1 Processor -- start 2024-09-15T23:30:00.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 0x5597c3bb9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bba190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 0x5597c3bba6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5597c3bbf6d0 con 0x5597c3bb2480 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5597c3ad1f10 con 0x5597c3bbcd40 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5597c3bbf850 con 0x5597c3bb1b60 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bfc88a740 10 client.0 _ll_register_callbacks cb 0x5597c3a9aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bba190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bba190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37148/0 (socket says 172.21.15.89:37148) 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bebfff640 1 -- 172.21.15.89:0/3627489802 learned_addr learned my addr 172.21.15.89:0/3627489802 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bf8a4e640 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 0x5597c3bba6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 0x5597c3bb9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bebfff640 1 -- 172.21.15.89:0/3627489802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 msgr2=0x5597c3bb9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:00.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bebfff640 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 0x5597c3bb9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bebfff640 1 -- 172.21.15.89:0/3627489802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 msgr2=0x5597c3bba6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bebfff640 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 0x5597c3bba6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.235+0000 7f2bebfff640 1 -- 172.21.15.89:0/3627489802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597c38886a0 con 0x5597c3bb2480 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bf8a4e640 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 0x5597c3bba6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bebfff640 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bba190 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00bfc0 tx=0x7f2bdc00f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f11fb3f8ee9e0b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bd37fe640 1 -- 172.21.15.89:0/3627489802 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc018020 con 0x5597c3bb2480 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bd37fe640 1 -- 172.21.15.89:0/3627489802 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2bdc014870 con 0x5597c3bb2480 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3627489802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2be40bc8f0 con 0x5597c3bb2480 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bd37fe640 1 -- 172.21.15.89:0/3627489802 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2bdc023050 con 0x5597c3bb2480 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bfc88a740 10 client.17526 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3627489802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5597c3bcb250 con 0x5597c3bb2480 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bd37fe640 1 -- 172.21.15.89:0/3627489802 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be40bc8f0 con 0x5597c3bb2480 2024-09-15T23:30:00.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bd37fe640 1 -- 172.21.15.89:0/3627489802 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2bdc013b60 con 0x5597c3bb2480 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bfc88a740 10 client.17526 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bfc88a740 10 client.17526 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bfc88a740 20 client.17526 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5597c3bcc7b0 0x5597c3bcebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.236+0000 7f2bfc88a740 4 client.17526 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.237+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3627489802 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597c3b247f0 con 0x5597c3bcc7b0 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.237+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5597c3bcc7b0 0x5597c3bcebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.237+0000 7f2bd37fe640 10 client.17526 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.237+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5597c3bcc7b0 0x5597c3bcebf0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a12f7dc7e5340b3 server_cookie=51b8374de38e40f8 in_seq=0 out_seq=0 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bd37fe640 1 -- 172.21.15.89:0/3627489802 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5597c3b247f0 con 0x5597c3bcc7b0 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bd37fe640 10 client.17526 handle_command_reply: tid=0 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 client.17526 shutdown 2024-09-15T23:30:00.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3627489802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 msgr2=0x5597c3bba190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bba190 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00bfc0 tx=0x7f2bdc00f670 comp rx=0 tx=0).stop 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3627489802 shutdown_connections 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5597c3bcc7b0 0x5597c3bcebf0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bbcd40 0x5597c3bba6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bd37fe640 0 client.17526 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bba190 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/3627489802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bb1b60 0x5597c3bb9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3627489802 >> 172.21.15.89:0/3627489802 conn(0x5597c3bae0f0 msgr2=0x5597c3bad270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3627489802 shutdown_connections 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3627489802 wait complete. 2024-09-15T23:30:00.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.238+0000 7f2bfc88a740 20 client.17526 trim_cache size 0 max 16384 2024-09-15T23:30:00.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bfc88a740 1 Processor -- start 2024-09-15T23:30:00.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bfc88a740 1 -- start start 2024-09-15T23:30:00.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb1b60 0x5597c3bb45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bbbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 0x5597c3bd44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41b11c0 con 0x5597c3bb2480 2024-09-15T23:30:00.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41b2d00 con 0x5597c3bb1b60 2024-09-15T23:30:00.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41b1890 con 0x5597c3bbcd40 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bf3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb1b60 0x5597c3bb45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bf3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb1b60 0x5597c3bb45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:52642/0 (socket says 172.21.15.89:52642) 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bf3fff640 1 -- 172.21.15.89:0/3386551649 learned_addr learned my addr 172.21.15.89:0/3386551649 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bebfff640 1 --2- 172.21.15.89:0/3386551649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bbbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bf8a4e640 1 --2- 172.21.15.89:0/3386551649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 0x5597c3bd44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bf3fff640 1 -- 172.21.15.89:0/3386551649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 msgr2=0x5597c3bd44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/3386551649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 0x5597c3bd44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bf3fff640 1 -- 172.21.15.89:0/3386551649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 msgr2=0x5597c3bbbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.256+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/3386551649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bbbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.257+0000 7f2bf3fff640 1 -- 172.21.15.89:0/3386551649 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597c3bb81a0 con 0x5597c3bb1b60 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.257+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/3386551649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb1b60 0x5597c3bb45c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2be4000c00 tx=0x7f2be40bc350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80e2f43f649926f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.257+0000 7f2be9ffb640 1 -- 172.21.15.89:0/3386551649 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be4049040 con 0x5597c3bb1b60 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.257+0000 7f2be9ffb640 1 -- 172.21.15.89:0/3386551649 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be40bc8f0 con 0x5597c3bb1b60 2024-09-15T23:30:00.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.257+0000 7f2be9ffb640 1 -- 172.21.15.89:0/3386551649 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be40ede50 con 0x5597c3bb1b60 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3386551649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb1b60 msgr2=0x5597c3bb45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/3386551649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb1b60 0x5597c3bb45c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2be4000c00 tx=0x7f2be40bc350 comp rx=0 tx=0).stop 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3386551649 shutdown_connections 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/3386551649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 0x5597c3bd44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/3386551649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb2480 0x5597c3bbbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/3386551649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb1b60 0x5597c3bb45c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3386551649 >> 172.21.15.89:0/3386551649 conn(0x5597c3bae0f0 msgr2=0x5597c3bbb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3386551649 shutdown_connections 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 -- 172.21.15.89:0/3386551649 wait complete. 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 -- start start 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 Processor -- start 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb1b60 0x5597c3bc0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.258+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb2480 0x5597c3bc0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bfc88a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 0x5597c3bc13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41b2630 con 0x5597c3bb1b60 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41b1f60 con 0x5597c3bb2480 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bfc88a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf41af1a0 con 0x5597c3bbcd40 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bfc88a740 10 client.0 _ll_register_callbacks cb 0x7f2bdc00b2a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb1b60 0x5597c3bc0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb1b60 0x5597c3bc0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37168/0 (socket says 172.21.15.89:37168) 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf3fff640 1 -- 172.21.15.89:0/1253118613 learned_addr learned my addr 172.21.15.89:0/1253118613 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf8a4e640 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 0x5597c3bc13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bebfff640 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb2480 0x5597c3bc0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf3fff640 1 -- 172.21.15.89:0/1253118613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 msgr2=0x5597c3bc13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 0x5597c3bc13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf3fff640 1 -- 172.21.15.89:0/1253118613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb2480 msgr2=0x5597c3bc0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb2480 0x5597c3bc0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf3fff640 1 -- 172.21.15.89:0/1253118613 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597c3c36730 con 0x5597c3bb1b60 2024-09-15T23:30:00.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf8a4e640 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 0x5597c3bc13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bebfff640 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb2480 0x5597c3bc0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bf3fff640 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb1b60 0x5597c3bc0950 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2be40bc3c0 tx=0x7f2be40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9117014db6536945 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bd37fe640 1 -- 172.21.15.89:0/1253118613 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be40d0070 con 0x5597c3bb1b60 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bd37fe640 1 -- 172.21.15.89:0/1253118613 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2be403bd70 con 0x5597c3bb1b60 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bfc88a740 1 -- 172.21.15.89:0/1253118613 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2be40bc8f0 con 0x5597c3bb1b60 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bd37fe640 1 -- 172.21.15.89:0/1253118613 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2be4049070 con 0x5597c3bb1b60 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bfc88a740 10 client.17532 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.259+0000 7f2bfc88a740 1 -- 172.21.15.89:0/1253118613 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5597c3c37a70 con 0x5597c3bb1b60 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bd37fe640 1 -- 172.21.15.89:0/1253118613 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2be40bc8f0 con 0x5597c3bb1b60 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bd37fe640 1 -- 172.21.15.89:0/1253118613 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2be40f0380 con 0x5597c3bb1b60 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bfc88a740 10 client.17532 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:00.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bfc88a740 10 client.17532 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bfc88a740 20 client.17532 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5597c3c37e00 0x5597c3c3a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bfc88a740 4 client.17532 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bfc88a740 1 -- 172.21.15.89:0/1253118613 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5597c3b1ed00 con 0x5597c3c37e00 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bebfff640 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5597c3c37e00 0x5597c3c3a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bd37fe640 10 client.17532 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.260+0000 7f2bebfff640 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5597c3c37e00 0x5597c3c3a1f0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47c943c8bdde81b0 server_cookie=d2400fff0eb784dd in_seq=0 out_seq=0 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.261+0000 7f2bd37fe640 1 -- 172.21.15.89:0/1253118613 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5597c3b1ed00 con 0x5597c3c37e00 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.261+0000 7f2bd37fe640 10 client.17532 handle_command_reply: tid=0 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.261+0000 7f2bfc88a740 1 client.17532 shutdown 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.261+0000 7f2bfc88a740 1 -- 172.21.15.89:0/1253118613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb1b60 msgr2=0x5597c3bc0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.261+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb1b60 0x5597c3bc0950 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2be40bc3c0 tx=0x7f2be40045e0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.261+0000 7f2bfc88a740 1 -- 172.21.15.89:0/1253118613 shutdown_connections 2024-09-15T23:30:00.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5597c3c37e00 0x5597c3c3a1f0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5597c3bbcd40 0x5597c3bc13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5597c3bb2480 0x5597c3bc0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bd37fe640 0 client.17532 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:00.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfc88a740 1 --2- 172.21.15.89:0/1253118613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597c3bb1b60 0x5597c3bc0950 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfc88a740 1 -- 172.21.15.89:0/1253118613 >> 172.21.15.89:0/1253118613 conn(0x5597c3bae0f0 msgr2=0x5597c3bbbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:00.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfc88a740 1 -- 172.21.15.89:0/1253118613 shutdown_connections 2024-09-15T23:30:00.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfc88a740 1 -- 172.21.15.89:0/1253118613 wait complete. 2024-09-15T23:30:00.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfc88a740 20 client.17532 trim_cache size 0 max 16384 2024-09-15T23:30:00.274 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:30:00.275 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:30:00.275 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "7e824a50-522f-4da9-83f2-4409f88766b5", 2024-09-15T23:30:00.275 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:30:00.275 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/1217638185 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2be00a3240 msgr2=0x7f2be00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2be00a3240 0x7f2be00a5700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2bec0070b0 tx=0x7f2bec0094c0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/1217638185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bf4101de0 msgr2=0x7f2bf40744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bf4101de0 0x7f2bf40744f0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f2be4005ab0 tx=0x7f2be40042f0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/1217638185 shutdown_connections 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2be00a3240 0x7f2be00a5700 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bf4106f10 0x7f2bf4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bf4102790 0x7f2bf4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 --2- 172.21.15.89:0/1217638185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bf4101de0 0x7f2bf40744f0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/1217638185 >> 172.21.15.89:0/1217638185 conn(0x7f2bf40fd8c0 msgr2=0x7f2bf40fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/1217638185 shutdown_connections 2024-09-15T23:30:00.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:00.262+0000 7f2bfa4b1640 1 -- 172.21.15.89:0/1217638185 wait complete. 2024-09-15T23:30:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:30:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:30:00.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:30:00.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:00 smithi120 ceph-mon[35250]: overall HEALTH_OK 2024-09-15T23:30:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:30:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:30:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:30:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:00 smithi174 ceph-mon[35586]: overall HEALTH_OK 2024-09-15T23:30:00.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:30:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:30:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:30:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:00 smithi089 ceph-mon[30933]: overall HEALTH_OK 2024-09-15T23:30:01.003 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:30:01.292 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:01.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:01 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:30:01.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:01 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:01.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:01 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:30:01.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:01 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:01.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:01 smithi089 ceph-mon[30933]: pgmap v530: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 16 op/s 2024-09-15T23:30:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:01 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:30:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:01 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:01 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:30:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:01 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:01 smithi120 ceph-mon[35250]: pgmap v530: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 16 op/s 2024-09-15T23:30:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:01 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:30:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:01 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:01 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:30:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:01 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:01 smithi174 ceph-mon[35586]: pgmap v530: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 16 op/s 2024-09-15T23:30:01.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.752+0000 7fedb09af640 1 -- 172.21.15.89:0/2752332418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac101400 msgr2=0x7fedac10d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.752+0000 7fedb09af640 1 --2- 172.21.15.89:0/2752332418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac101400 0x7fedac10d930 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7feda4009cd0 tx=0x7feda402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.752+0000 7fedb09af640 1 -- 172.21.15.89:0/2752332418 shutdown_connections 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.752+0000 7fedb09af640 1 --2- 172.21.15.89:0/2752332418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac101400 0x7fedac10d930 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.752+0000 7fedb09af640 1 --2- 172.21.15.89:0/2752332418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedac100a60 0x7fedac100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.752+0000 7fedb09af640 1 --2- 172.21.15.89:0/2752332418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedac106a60 0x7fedac106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.752+0000 7fedb09af640 1 -- 172.21.15.89:0/2752332418 >> 172.21.15.89:0/2752332418 conn(0x7fedac0fc780 msgr2=0x7fedac0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.752+0000 7fedb09af640 1 -- 172.21.15.89:0/2752332418 shutdown_connections 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.752+0000 7fedb09af640 1 -- 172.21.15.89:0/2752332418 wait complete. 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7fedb09af640 1 Processor -- start 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7fedb09af640 1 -- start start 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7fedb09af640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedac100a60 0x7fedac198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7fedb09af640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac101400 0x7fedac1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7fedb09af640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedac106a60 0x7fedac19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7fedb09af640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fedac110ea0 con 0x7fedac101400 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7fedb09af640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fedac110d20 con 0x7fedac106a60 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7fedb09af640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fedac111020 con 0x7fedac100a60 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7feda9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac101400 0x7fedac1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7feda9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac101400 0x7fedac1989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37194/0 (socket says 172.21.15.89:37194) 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7feda9d9b640 1 -- 172.21.15.89:0/2588596932 learned_addr learned my addr 172.21.15.89:0/2588596932 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7fedaa59c640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedac100a60 0x7fedac198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7feda9d9b640 1 -- 172.21.15.89:0/2588596932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedac100a60 msgr2=0x7fedac198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7feda9d9b640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedac100a60 0x7fedac198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7feda9d9b640 1 -- 172.21.15.89:0/2588596932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedac106a60 msgr2=0x7fedac19cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7fedaad9d640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedac106a60 0x7fedac19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7feda9d9b640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedac106a60 0x7fedac19cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7feda9d9b640 1 -- 172.21.15.89:0/2588596932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedac19d410 con 0x7fedac101400 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7feda9d9b640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac101400 0x7fedac1989a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fed9400c970 tx=0x7fed9400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.753+0000 7feda37fe640 1 -- 172.21.15.89:0/2588596932 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed94007bb0 con 0x7fedac101400 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.754+0000 7fedb09af640 1 -- 172.21.15.89:0/2588596932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedac19d700 con 0x7fedac101400 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.754+0000 7feda37fe640 1 -- 172.21.15.89:0/2588596932 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed94007d50 con 0x7fedac101400 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.754+0000 7feda37fe640 1 -- 172.21.15.89:0/2588596932 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9400f450 con 0x7fedac101400 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.754+0000 7fedb09af640 1 -- 172.21.15.89:0/2588596932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedac1a4fe0 con 0x7fedac101400 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.756+0000 7feda37fe640 1 -- 172.21.15.89:0/2588596932 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fed94005000 con 0x7fedac101400 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.756+0000 7fedb2d88740 1 Processor -- start 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.756+0000 7fedb2d88740 1 -- start start 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.756+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e69910880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.756+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e69910dc0 0x564e6991b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.756+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6991b680 0x564e6991da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.756+0000 7fedb2d88740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564e698762e0 con 0x564e69910dc0 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.756+0000 7fedb2d88740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e6981dce0 con 0x564e699104a0 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.756+0000 7fedb2d88740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e698722a0 con 0x564e6991b680 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e69910dc0 0x564e6991b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e69910dc0 0x564e6991b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37196/0 (socket says 172.21.15.89:37196) 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaa59c640 1 -- 172.21.15.89:0/3060624106 learned_addr learned my addr 172.21.15.89:0/3060624106 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaad9d640 1 --2- 172.21.15.89:0/3060624106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e69910880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7feda9d9b640 1 --2- 172.21.15.89:0/3060624106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6991b680 0x564e6991da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaa59c640 1 -- 172.21.15.89:0/3060624106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6991b680 msgr2=0x564e6991da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaa59c640 1 --2- 172.21.15.89:0/3060624106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6991b680 0x564e6991da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaa59c640 1 -- 172.21.15.89:0/3060624106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 msgr2=0x564e69910880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaa59c640 1 --2- 172.21.15.89:0/3060624106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e69910880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaa59c640 1 -- 172.21.15.89:0/3060624106 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e69802440 con 0x564e69910dc0 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7feda9d9b640 1 --2- 172.21.15.89:0/3060624106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6991b680 0x564e6991da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7feda37fe640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fed980a3240 0x7fed980a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaad9d640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fed980a3240 0x7fed980a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7fedaa59c640 1 --2- 172.21.15.89:0/3060624106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e69910dc0 0x564e6991b140 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fed9c00b820 tx=0x7fed9c00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01cb92e9b141b5f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7feda37fe640 1 -- 172.21.15.89:0/2588596932 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fed940cfbd0 con 0x7fedac101400 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7feda0ff9640 1 -- 172.21.15.89:0/3060624106 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c00d950 con 0x564e69910dc0 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7feda0ff9640 1 -- 172.21.15.89:0/3060624106 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed9c019070 con 0x564e69910dc0 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.757+0000 7feda0ff9640 1 -- 172.21.15.89:0/3060624106 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c0123e0 con 0x564e69910dc0 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.758+0000 7fedb2d88740 1 -- 172.21.15.89:0/3060624106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e69910dc0 msgr2=0x564e6991b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.758+0000 7fedaad9d640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fed980a3240 0x7fed980a5700 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7feda4009430 tx=0x7feda4005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.758+0000 7fedb2d88740 1 --2- 172.21.15.89:0/3060624106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e69910dc0 0x564e6991b140 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fed9c00b820 tx=0x7fed9c00bce0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.758+0000 7fedb2d88740 1 -- 172.21.15.89:0/3060624106 shutdown_connections 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.758+0000 7fedb2d88740 1 --2- 172.21.15.89:0/3060624106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6991b680 0x564e6991da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.758+0000 7fedb2d88740 1 --2- 172.21.15.89:0/3060624106 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e69910dc0 0x564e6991b140 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.758+0000 7fedb2d88740 1 --2- 172.21.15.89:0/3060624106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e69910880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.758+0000 7fedb2d88740 1 -- 172.21.15.89:0/3060624106 >> 172.21.15.89:0/3060624106 conn(0x564e6990ca30 msgr2=0x564e6990ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.758+0000 7fedb2d88740 1 -- 172.21.15.89:0/3060624106 shutdown_connections 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.758+0000 7fedb2d88740 1 -- 172.21.15.89:0/3060624106 wait complete. 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedb2d88740 1 -- start start 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedb2d88740 1 Processor -- start 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e6991aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e69910dc0 0x564e699181a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6991b680 0x564e699186e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedb2d88740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564e6991e250 con 0x564e6991b680 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedb2d88740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e69831050 con 0x564e699104a0 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedb2d88740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e6991e3d0 con 0x564e69910dc0 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedb2d88740 10 client.0 _ll_register_callbacks cb 0x564e69871160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:01.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedaad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6991b680 0x564e699186e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedaad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6991b680 0x564e699186e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37204/0 (socket says 172.21.15.89:37204) 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedaad9d640 1 -- 172.21.15.89:0/3110785287 learned_addr learned my addr 172.21.15.89:0/3110785287 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedaa59c640 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e6991aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7feda9d9b640 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e69910dc0 0x564e699181a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedaad9d640 1 -- 172.21.15.89:0/3110785287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e69910dc0 msgr2=0x564e699181a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedaad9d640 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e69910dc0 0x564e699181a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedaad9d640 1 -- 172.21.15.89:0/3110785287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 msgr2=0x564e6991aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedaad9d640 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e6991aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.759+0000 7fedaad9d640 1 -- 172.21.15.89:0/3110785287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e695e73c0 con 0x564e6991b680 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fedaad9d640 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6991b680 0x564e699186e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7feda4035000 tx=0x7feda40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d916c18ee5a38f81 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fed91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fed91ffb640 1 -- 172.21.15.89:0/3110785287 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda4004300 con 0x564e6991b680 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fed91ffb640 1 -- 172.21.15.89:0/3110785287 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7feda40044a0 con 0x564e6991b680 2024-09-15T23:30:01.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fedb2d88740 1 -- 172.21.15.89:0/3110785287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed9c019070 con 0x564e6991b680 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fed91ffb640 1 -- 172.21.15.89:0/3110785287 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feda40046a0 con 0x564e6991b680 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fedb2d88740 10 client.17544 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fedb2d88740 1 -- 172.21.15.89:0/3110785287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564e69929b40 con 0x564e6991b680 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fed91ffb640 1 -- 172.21.15.89:0/3110785287 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed9c019070 con 0x564e6991b680 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fed91ffb640 1 -- 172.21.15.89:0/3110785287 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7feda4030070 con 0x564e6991b680 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fedb2d88740 10 client.17544 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fedb2d88740 10 client.17544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fedb2d88740 20 client.17544 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fedb2d88740 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e6992afa0 0x564e6992d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fedb2d88740 4 client.17544 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.760+0000 7fedb2d88740 1 -- 172.21.15.89:0/3110785287 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e69883850 con 0x564e6992afa0 2024-09-15T23:30:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.761+0000 7fedaa59c640 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e6992afa0 0x564e6992d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.761+0000 7fed91ffb640 10 client.17544 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.761+0000 7fedaa59c640 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e6992afa0 0x564e6992d410 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c587882373dadfde server_cookie=d4ca02e2e1e26960 in_seq=0 out_seq=0 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fed91ffb640 1 -- 172.21.15.89:0/3110785287 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564e69883850 con 0x564e6992afa0 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fed91ffb640 10 client.17544 handle_command_reply: tid=0 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 client.17544 shutdown 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 -- 172.21.15.89:0/3110785287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6991b680 msgr2=0x564e699186e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6991b680 0x564e699186e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7feda4035000 tx=0x7feda40095c0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 -- 172.21.15.89:0/3110785287 shutdown_connections 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e6992afa0 0x564e6992d410 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6991b680 0x564e699186e0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e69910dc0 0x564e699181a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fed91ffb640 0 client.17544 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 --2- 172.21.15.89:0/3110785287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e6991aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 -- 172.21.15.89:0/3110785287 >> 172.21.15.89:0/3110785287 conn(0x564e6990ca30 msgr2=0x564e69912760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:01.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 -- 172.21.15.89:0/3110785287 shutdown_connections 2024-09-15T23:30:01.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 1 -- 172.21.15.89:0/3110785287 wait complete. 2024-09-15T23:30:01.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.762+0000 7fedb2d88740 20 client.17544 trim_cache size 0 max 16384 2024-09-15T23:30:01.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedb2d88740 1 Processor -- start 2024-09-15T23:30:01.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedb2d88740 1 -- start start 2024-09-15T23:30:01.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6990ca30 0x564e6991a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e69932d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e69910dc0 0x564e69933270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedb2d88740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fedac1aee50 con 0x564e6990ca30 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedb2d88740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fedac1b0990 con 0x564e699104a0 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedb2d88740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fedac1af520 con 0x564e69910dc0 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedaa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6990ca30 0x564e6991a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedaa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6990ca30 0x564e6991a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37220/0 (socket says 172.21.15.89:37220) 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedaa59c640 1 -- 172.21.15.89:0/1565782036 learned_addr learned my addr 172.21.15.89:0/1565782036 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedaad9d640 1 --2- 172.21.15.89:0/1565782036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e69910dc0 0x564e69933270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7feda9d9b640 1 --2- 172.21.15.89:0/1565782036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e69932d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedaa59c640 1 -- 172.21.15.89:0/1565782036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e69910dc0 msgr2=0x564e69933270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedaa59c640 1 --2- 172.21.15.89:0/1565782036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e69910dc0 0x564e69933270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedaa59c640 1 -- 172.21.15.89:0/1565782036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 msgr2=0x564e69932d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.779+0000 7fedaa59c640 1 --2- 172.21.15.89:0/1565782036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e69932d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.780+0000 7fedaa59c640 1 -- 172.21.15.89:0/1565782036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e6990c860 con 0x564e6990ca30 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.780+0000 7fedaa59c640 1 --2- 172.21.15.89:0/1565782036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6990ca30 0x564e6991a110 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fed9c014570 tx=0x7fed9c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6081660e1cd3503 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.780+0000 7fed93fff640 1 -- 172.21.15.89:0/1565782036 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c02b070 con 0x564e6990ca30 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.780+0000 7fed93fff640 1 -- 172.21.15.89:0/1565782036 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed9c019040 con 0x564e6990ca30 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.780+0000 7fed93fff640 1 -- 172.21.15.89:0/1565782036 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c027900 con 0x564e6990ca30 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.780+0000 7fedb2d88740 1 -- 172.21.15.89:0/1565782036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6990ca30 msgr2=0x564e6991a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.780+0000 7fedb2d88740 1 --2- 172.21.15.89:0/1565782036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6990ca30 0x564e6991a110 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fed9c014570 tx=0x7fed9c008040 comp rx=0 tx=0).stop 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.780+0000 7fedb2d88740 1 -- 172.21.15.89:0/1565782036 shutdown_connections 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.780+0000 7fedb2d88740 1 --2- 172.21.15.89:0/1565782036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e69910dc0 0x564e69933270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.780+0000 7fedb2d88740 1 --2- 172.21.15.89:0/1565782036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e699104a0 0x564e69932d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 --2- 172.21.15.89:0/1565782036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6990ca30 0x564e6991a110 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 -- 172.21.15.89:0/1565782036 >> 172.21.15.89:0/1565782036 conn(0x564e69918840 msgr2=0x564e69919a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 -- 172.21.15.89:0/1565782036 shutdown_connections 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 -- 172.21.15.89:0/1565782036 wait complete. 2024-09-15T23:30:01.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 -- start start 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 Processor -- start 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6990ca30 0x564e699213e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e699104a0 0x564e69921920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e69910dc0 0x564e69921e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fedac1b02c0 con 0x564e699104a0 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fedac1afbf0 con 0x564e69910dc0 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fedac1ace30 con 0x564e6990ca30 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedb2d88740 10 client.0 _ll_register_callbacks cb 0x7feda4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7feda9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e699104a0 0x564e69921920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7feda9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e699104a0 0x564e69921920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37236/0 (socket says 172.21.15.89:37236) 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7feda9d9b640 1 -- 172.21.15.89:0/2144320873 learned_addr learned my addr 172.21.15.89:0/2144320873 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.781+0000 7fedaa59c640 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6990ca30 0x564e699213e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fedaad9d640 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e69910dc0 0x564e69921e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7feda9d9b640 1 -- 172.21.15.89:0/2144320873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6990ca30 msgr2=0x564e699213e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7feda9d9b640 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6990ca30 0x564e699213e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7feda9d9b640 1 -- 172.21.15.89:0/2144320873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e69910dc0 msgr2=0x564e69921e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7feda9d9b640 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e69910dc0 0x564e69921e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7feda9d9b640 1 -- 172.21.15.89:0/2144320873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e69994d10 con 0x564e699104a0 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7feda9d9b640 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e699104a0 0x564e69921920 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fed940987d0 tx=0x7fed9409b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb3bf7d86f56187 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fed91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fed91ffb640 1 -- 172.21.15.89:0/2144320873 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9401d040 con 0x564e699104a0 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fed91ffb640 1 -- 172.21.15.89:0/2144320873 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fed9409a100 con 0x564e699104a0 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fedb2d88740 1 -- 172.21.15.89:0/2144320873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed9c019040 con 0x564e699104a0 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fed91ffb640 1 -- 172.21.15.89:0/2144320873 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fed940b9050 con 0x564e699104a0 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fedb2d88740 10 client.17550 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fedb2d88740 1 -- 172.21.15.89:0/2144320873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564e69996090 con 0x564e699104a0 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fed91ffb640 1 -- 172.21.15.89:0/2144320873 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed9c019040 con 0x564e699104a0 2024-09-15T23:30:01.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fed91ffb640 1 -- 172.21.15.89:0/2144320873 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fed9409b250 con 0x564e699104a0 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fedb2d88740 10 client.17550 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fedb2d88740 10 client.17550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fedb2d88740 20 client.17550 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fedb2d88740 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e6991b680 0x564e699988a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fedb2d88740 4 client.17550 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.782+0000 7fedb2d88740 1 -- 172.21.15.89:0/2144320873 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564e6987dd60 con 0x564e6991b680 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.783+0000 7fedaa59c640 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e6991b680 0x564e699988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.783+0000 7fedaa59c640 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e6991b680 0x564e699988a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25214e0dc69221c9 server_cookie=8dd130fb0a8af691 in_seq=0 out_seq=0 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.783+0000 7fed91ffb640 10 client.17550 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.783+0000 7fed91ffb640 1 -- 172.21.15.89:0/2144320873 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564e6987dd60 con 0x564e6991b680 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.783+0000 7fed91ffb640 10 client.17550 handle_command_reply: tid=0 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.783+0000 7fedb2d88740 1 client.17550 shutdown 2024-09-15T23:30:01.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 1 -- 172.21.15.89:0/2144320873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e699104a0 msgr2=0x564e69921920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e699104a0 0x564e69921920 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fed940987d0 tx=0x7fed9409b940 comp rx=0 tx=0).stop 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 1 -- 172.21.15.89:0/2144320873 shutdown_connections 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e6991b680 0x564e699988a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e69910dc0 0x564e69921e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e699104a0 0x564e69921920 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 1 --2- 172.21.15.89:0/2144320873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6990ca30 0x564e699213e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 1 -- 172.21.15.89:0/2144320873 >> 172.21.15.89:0/2144320873 conn(0x564e69918840 msgr2=0x564e6990b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fed91ffb640 0 client.17550 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 1 -- 172.21.15.89:0/2144320873 shutdown_connections 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 1 -- 172.21.15.89:0/2144320873 wait complete. 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb2d88740 20 client.17550 trim_cache size 0 max 16384 2024-09-15T23:30:01.792 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:30:01.793 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:30:01.793 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:30:01.793 INFO:teuthology.orchestra.run.smithi089.stdout: "7e824a50-522f-4da9-83f2-4409f88766b5": { 2024-09-15T23:30:01.793 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:30:01.793 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "7e824a50-522f-4da9-83f2-4409f88766b5", 2024-09-15T23:30:01.793 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:30:01.793 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:30:01.793 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:30:01.793 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:30:01.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb09af640 1 -- 172.21.15.89:0/2588596932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fed980a3240 msgr2=0x7fed980a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:01.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb09af640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fed980a3240 0x7fed980a5700 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7feda4009430 tx=0x7feda4005c50 comp rx=0 tx=0).stop 2024-09-15T23:30:01.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb09af640 1 -- 172.21.15.89:0/2588596932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac101400 msgr2=0x7fedac1989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:01.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb09af640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac101400 0x7fedac1989a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fed9400c970 tx=0x7fed9400ce30 comp rx=0 tx=0).stop 2024-09-15T23:30:01.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb09af640 1 -- 172.21.15.89:0/2588596932 shutdown_connections 2024-09-15T23:30:01.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb09af640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fed980a3240 0x7fed980a5700 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb09af640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fedac106a60 0x7fedac19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb09af640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fedac101400 0x7fedac1989a0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb09af640 1 --2- 172.21.15.89:0/2588596932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedac100a60 0x7fedac198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:01.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.784+0000 7fedb09af640 1 -- 172.21.15.89:0/2588596932 >> 172.21.15.89:0/2588596932 conn(0x7fedac0fc780 msgr2=0x7fedac06aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:01.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.785+0000 7fedb09af640 1 -- 172.21.15.89:0/2588596932 shutdown_connections 2024-09-15T23:30:01.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:01.785+0000 7fedb09af640 1 -- 172.21.15.89:0/2588596932 wait complete. 2024-09-15T23:30:02.631 INFO:tasks.cephfs.filesystem:scrub status for tag:7e824a50-522f-4da9-83f2-4409f88766b5 - {'path': '/', 'tag': '7e824a50-522f-4da9-83f2-4409f88766b5', 'options': 'recursive,force'} 2024-09-15T23:30:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:03 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:30:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:03 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:30:03.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:03 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:30:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:04 smithi120 ceph-mon[35250]: pgmap v531: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 15 op/s 2024-09-15T23:30:04.565 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:04 smithi089 ceph-mon[30933]: pgmap v531: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 15 op/s 2024-09-15T23:30:04.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:04 smithi174 ceph-mon[35586]: pgmap v531: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.7 KiB/s wr, 15 op/s 2024-09-15T23:30:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:05 smithi120 ceph-mon[35250]: pgmap v532: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 15 op/s 2024-09-15T23:30:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:05 smithi174 ceph-mon[35586]: pgmap v532: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 15 op/s 2024-09-15T23:30:05.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:05 smithi089 ceph-mon[30933]: pgmap v532: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 15 op/s 2024-09-15T23:30:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:07 smithi120 ceph-mon[35250]: pgmap v533: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.4 KiB/s wr, 18 op/s 2024-09-15T23:30:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:07 smithi174 ceph-mon[35586]: pgmap v533: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.4 KiB/s wr, 18 op/s 2024-09-15T23:30:07.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:07 smithi089 ceph-mon[30933]: pgmap v533: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.4 KiB/s wr, 18 op/s 2024-09-15T23:30:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:09 smithi120 ceph-mon[35250]: pgmap v534: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 3.8 KiB/s wr, 13 op/s 2024-09-15T23:30:09.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:09 smithi174 ceph-mon[35586]: pgmap v534: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 3.8 KiB/s wr, 13 op/s 2024-09-15T23:30:09.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:09 smithi089 ceph-mon[30933]: pgmap v534: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 3.8 KiB/s wr, 13 op/s 2024-09-15T23:30:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.622+0000 7f072409b640 1 -- 172.21.15.89:0/1942893968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f071c101c40 msgr2=0x7f071c1020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.622+0000 7f072409b640 1 --2- 172.21.15.89:0/1942893968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f071c101c40 0x7f071c1020c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f0710009900 tx=0x7f071002a410 comp rx=0 tx=0).stop 2024-09-15T23:30:09.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.622+0000 7f072409b640 1 -- 172.21.15.89:0/1942893968 shutdown_connections 2024-09-15T23:30:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.622+0000 7f072409b640 1 --2- 172.21.15.89:0/1942893968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f071c102600 0x7f071c108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.622+0000 7f072409b640 1 --2- 172.21.15.89:0/1942893968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f071c101c40 0x7f071c1020c0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.622+0000 7f072409b640 1 --2- 172.21.15.89:0/1942893968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f071c100a50 0x7f071c100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.622+0000 7f072409b640 1 -- 172.21.15.89:0/1942893968 >> 172.21.15.89:0/1942893968 conn(0x7f071c0fc1b0 msgr2=0x7f071c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.623+0000 7f072409b640 1 -- 172.21.15.89:0/1942893968 shutdown_connections 2024-09-15T23:30:09.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.623+0000 7f072409b640 1 -- 172.21.15.89:0/1942893968 wait complete. 2024-09-15T23:30:09.627 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.623+0000 7f072409b640 1 Processor -- start 2024-09-15T23:30:09.627 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f072409b640 1 -- start start 2024-09-15T23:30:09.627 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f072409b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f071c100a50 0x7f071c1049e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.627 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f072409b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f071c101c40 0x7f071c103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f072409b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f071c102600 0x7f071c103560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f072409b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f071c10c750 con 0x7f071c100a50 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f072409b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f071c10c5d0 con 0x7f071c102600 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f072409b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f071c10c8d0 con 0x7f071c101c40 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0721e37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f071c100a50 0x7f071c1049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0721e37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f071c100a50 0x7f071c1049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44746/0 (socket says 172.21.15.89:44746) 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0721e37640 1 -- 172.21.15.89:0/2989703804 learned_addr learned my addr 172.21.15.89:0/2989703804 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0721636640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f071c101c40 0x7f071c103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0722638640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f071c102600 0x7f071c103560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0721e37640 1 -- 172.21.15.89:0/2989703804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f071c101c40 msgr2=0x7f071c103020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:09.628 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0721e37640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f071c101c40 0x7f071c103020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0721e37640 1 -- 172.21.15.89:0/2989703804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f071c102600 msgr2=0x7f071c103560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0721e37640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f071c102600 0x7f071c103560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0721e37640 1 -- 172.21.15.89:0/2989703804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c103df0 con 0x7f071c100a50 2024-09-15T23:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0722638640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f071c102600 0x7f071c103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.624+0000 7f0721e37640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f071c100a50 0x7f071c1049e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f071800b450 tx=0x7f071800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.625+0000 7f070effd640 1 -- 172.21.15.89:0/2989703804 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0718015020 con 0x7f071c100a50 2024-09-15T23:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.625+0000 7f072409b640 1 -- 172.21.15.89:0/2989703804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071c1a1f70 con 0x7f071c100a50 2024-09-15T23:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.625+0000 7f070effd640 1 -- 172.21.15.89:0/2989703804 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0718011d70 con 0x7f071c100a50 2024-09-15T23:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.625+0000 7f070effd640 1 -- 172.21.15.89:0/2989703804 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071800f340 con 0x7f071c100a50 2024-09-15T23:30:09.629 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.625+0000 7f072409b640 1 -- 172.21.15.89:0/2989703804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071c1a23f0 con 0x7f071c100a50 2024-09-15T23:30:09.636 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.627+0000 7f070effd640 1 -- 172.21.15.89:0/2989703804 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0718018020 con 0x7f071c100a50 2024-09-15T23:30:09.636 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.627+0000 7f0726e32740 1 Processor -- start 2024-09-15T23:30:09.636 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0726e32740 1 -- start start 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0726e32740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f699a0 0x556bb7f69da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0726e32740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f6eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0726e32740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f71830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0726e32740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556bb7ecd2f0 con 0x556bb7f6f370 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0726e32740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556bb7e75270 con 0x556bb7f699a0 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0726e32740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556bb7ec92b0 con 0x556bb7f6a2e0 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0721636640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f71830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0721636640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f71830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44756/0 (socket says 172.21.15.89:44756) 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0721636640 1 -- 172.21.15.89:0/223198199 learned_addr learned my addr 172.21.15.89:0/223198199 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0722638640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f699a0 0x556bb7f69da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.637 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f070effd640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07080a32f0 0x7f07080a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0721e37640 1 --2- 172.21.15.89:0/223198199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f6eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0722638640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07080a32f0 0x7f07080a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0721636640 1 -- 172.21.15.89:0/223198199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 msgr2=0x556bb7f6eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0721636640 1 --2- 172.21.15.89:0/223198199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f6eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0721636640 1 -- 172.21.15.89:0/223198199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f699a0 msgr2=0x556bb7f69da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0721636640 1 --2- 172.21.15.89:0/223198199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f699a0 0x556bb7f69da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0721636640 1 -- 172.21.15.89:0/223198199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bb7e74340 con 0x556bb7f6f370 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f0721e37640 1 --2- 172.21.15.89:0/223198199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f6eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.628+0000 7f070effd640 1 -- 172.21.15.89:0/2989703804 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f07180cf760 con 0x7f071c100a50 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.629+0000 7f0721636640 1 --2- 172.21.15.89:0/223198199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f71830 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f0710031610 tx=0x7f0710005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46aab3a54746abc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.629+0000 7f0707fff640 1 -- 172.21.15.89:0/223198199 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0710040070 con 0x556bb7f6f370 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.629+0000 7f0722638640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07080a32f0 0x7f07080a57b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0714005fd0 tx=0x7f0714004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:09.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.629+0000 7f0707fff640 1 -- 172.21.15.89:0/223198199 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0710005500 con 0x556bb7f6f370 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.629+0000 7f0707fff640 1 -- 172.21.15.89:0/223198199 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0710035040 con 0x556bb7f6f370 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 -- 172.21.15.89:0/223198199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 msgr2=0x556bb7f71830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 --2- 172.21.15.89:0/223198199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f71830 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f0710031610 tx=0x7f0710005c00 comp rx=0 tx=0).stop 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 -- 172.21.15.89:0/223198199 shutdown_connections 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 --2- 172.21.15.89:0/223198199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f71830 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 --2- 172.21.15.89:0/223198199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f6eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 --2- 172.21.15.89:0/223198199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f699a0 0x556bb7f69da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 -- 172.21.15.89:0/223198199 >> 172.21.15.89:0/223198199 conn(0x556bb7f65010 msgr2=0x556bb7f65410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 -- 172.21.15.89:0/223198199 shutdown_connections 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 -- 172.21.15.89:0/223198199 wait complete. 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 -- start start 2024-09-15T23:30:09.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.630+0000 7f0726e32740 1 Processor -- start 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0726e32740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f73ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0726e32740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f6a2e0 0x556bb7f73ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0726e32740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6f370 0x556bb7f7b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0726e32740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556bb7f722e0 con 0x556bb7f699a0 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0726e32740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556bb7e56290 con 0x556bb7f6a2e0 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0726e32740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556bb7f72460 con 0x556bb7f6f370 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0726e32740 10 client.0 _ll_register_callbacks cb 0x556bb7e87140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0721e37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f73ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0721e37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f73ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44766/0 (socket says 172.21.15.89:44766) 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0721e37640 1 -- 172.21.15.89:0/2723649752 learned_addr learned my addr 172.21.15.89:0/2723649752 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:09.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0722638640 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6f370 0x556bb7f7b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0721636640 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f6a2e0 0x556bb7f73ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0721e37640 1 -- 172.21.15.89:0/2723649752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6f370 msgr2=0x556bb7f7b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0721e37640 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6f370 0x556bb7f7b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0721e37640 1 -- 172.21.15.89:0/2723649752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f6a2e0 msgr2=0x556bb7f73ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0721e37640 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f6a2e0 0x556bb7f73ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0721e37640 1 -- 172.21.15.89:0/2723649752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bb7be0210 con 0x556bb7f699a0 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0722638640 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6f370 0x556bb7f7b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0721e37640 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f73ab0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f07180cce70 tx=0x7f071809c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2768bcb87cd9a28e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f07057fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f07057fa640 1 -- 172.21.15.89:0/2723649752 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071809fd40 con 0x556bb7f699a0 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f07057fa640 1 -- 172.21.15.89:0/2723649752 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0718099c90 con 0x556bb7f699a0 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0726e32740 1 -- 172.21.15.89:0/2723649752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0710005500 con 0x556bb7f699a0 2024-09-15T23:30:09.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f07057fa640 1 -- 172.21.15.89:0/2723649752 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f07180a4600 con 0x556bb7f699a0 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0726e32740 10 client.17562 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.631+0000 7f0726e32740 1 -- 172.21.15.89:0/2723649752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556bb7cac620 con 0x556bb7f699a0 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.632+0000 7f07057fa640 1 -- 172.21.15.89:0/2723649752 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0710005500 con 0x556bb7f699a0 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.632+0000 7f07057fa640 1 -- 172.21.15.89:0/2723649752 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0718084ce0 con 0x556bb7f699a0 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.632+0000 7f0726e32740 10 client.17562 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.632+0000 7f0726e32740 10 client.17562 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.632+0000 7f0726e32740 20 client.17562 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.632+0000 7f0726e32740 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556bb7f7f720 0x556bb7f81b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.632+0000 7f0726e32740 4 client.17562 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.632+0000 7f0726e32740 1 -- 172.21.15.89:0/2723649752 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bb7edaa20 con 0x556bb7f7f720 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.632+0000 7f0721636640 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556bb7f7f720 0x556bb7f81b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f0721636640 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556bb7f7f720 0x556bb7f81b10 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=992ceac1c31f8abe server_cookie=64536b0cafe60644 in_seq=0 out_seq=0 2024-09-15T23:30:09.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f07057fa640 10 client.17562 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f07057fa640 1 -- 172.21.15.89:0/2723649752 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556bb7edaa20 con 0x556bb7f7f720 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f07057fa640 10 client.17562 handle_command_reply: tid=0 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f0726e32740 1 client.17562 shutdown 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f0726e32740 1 -- 172.21.15.89:0/2723649752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 msgr2=0x556bb7f73ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f0726e32740 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f73ab0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f07180cce70 tx=0x7f071809c090 comp rx=0 tx=0).stop 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f0726e32740 1 -- 172.21.15.89:0/2723649752 shutdown_connections 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f0726e32740 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556bb7f7f720 0x556bb7f81b10 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f0726e32740 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6f370 0x556bb7f7b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f0726e32740 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f6a2e0 0x556bb7f73ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f0726e32740 1 --2- 172.21.15.89:0/2723649752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f73ab0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f07057fa640 0 client.17562 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.633+0000 7f0726e32740 1 -- 172.21.15.89:0/2723649752 >> 172.21.15.89:0/2723649752 conn(0x556bb7f65010 msgr2=0x556bb7f66820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:09.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.634+0000 7f0726e32740 1 -- 172.21.15.89:0/2723649752 shutdown_connections 2024-09-15T23:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.634+0000 7f0726e32740 1 -- 172.21.15.89:0/2723649752 wait complete. 2024-09-15T23:30:09.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.634+0000 7f0726e32740 20 client.17562 trim_cache size 0 max 16384 2024-09-15T23:30:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0726e32740 1 Processor -- start 2024-09-15T23:30:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0726e32740 1 -- start start 2024-09-15T23:30:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0726e32740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f699a0 0x556bb7f65ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0726e32740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f7aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0726e32740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f66400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0726e32740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f071c1acb40 con 0x556bb7f6f370 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0726e32740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f071c1ae680 con 0x556bb7f699a0 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0726e32740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f071c1ad210 con 0x556bb7f6a2e0 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0722638640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f66400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0722638640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f66400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44780/0 (socket says 172.21.15.89:44780) 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0722638640 1 -- 172.21.15.89:0/155268463 learned_addr learned my addr 172.21.15.89:0/155268463 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0721e37640 1 --2- 172.21.15.89:0/155268463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f699a0 0x556bb7f65ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0721636640 1 --2- 172.21.15.89:0/155268463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f7aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0722638640 1 -- 172.21.15.89:0/155268463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 msgr2=0x556bb7f7aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0722638640 1 --2- 172.21.15.89:0/155268463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f7aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0722638640 1 -- 172.21.15.89:0/155268463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f699a0 msgr2=0x556bb7f65ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0722638640 1 --2- 172.21.15.89:0/155268463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f699a0 0x556bb7f65ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0722638640 1 -- 172.21.15.89:0/155268463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bb7f6a7c0 con 0x556bb7f6f370 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.648+0000 7f0721636640 1 --2- 172.21.15.89:0/155268463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f7aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f0722638640 1 --2- 172.21.15.89:0/155268463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f66400 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f0714012910 tx=0x7f0714013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc79d88dd50d1829 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f07077fe640 1 -- 172.21.15.89:0/155268463 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071400edc0 con 0x556bb7f6f370 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f07077fe640 1 -- 172.21.15.89:0/155268463 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0714023030 con 0x556bb7f6f370 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f07077fe640 1 -- 172.21.15.89:0/155268463 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071401e730 con 0x556bb7f6f370 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f0726e32740 1 -- 172.21.15.89:0/155268463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 msgr2=0x556bb7f66400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f0726e32740 1 --2- 172.21.15.89:0/155268463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f66400 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f0714012910 tx=0x7f0714013000 comp rx=0 tx=0).stop 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f0726e32740 1 -- 172.21.15.89:0/155268463 shutdown_connections 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f0726e32740 1 --2- 172.21.15.89:0/155268463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f6f370 0x556bb7f66400 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f0726e32740 1 --2- 172.21.15.89:0/155268463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f7aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f0726e32740 1 --2- 172.21.15.89:0/155268463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f699a0 0x556bb7f65ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f0726e32740 1 -- 172.21.15.89:0/155268463 >> 172.21.15.89:0/155268463 conn(0x556bb7f65010 msgr2=0x556bb7f73860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f0726e32740 1 -- 172.21.15.89:0/155268463 shutdown_connections 2024-09-15T23:30:09.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.649+0000 7f0726e32740 1 -- 172.21.15.89:0/155268463 wait complete. 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0726e32740 1 -- start start 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0726e32740 1 Processor -- start 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0726e32740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f788a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0726e32740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f78de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0726e32740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f6f370 0x556bb7f79da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0726e32740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f071c1adfb0 con 0x556bb7f699a0 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0726e32740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f071c1ad8e0 con 0x556bb7f6f370 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0726e32740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f071c1aab20 con 0x556bb7f6a2e0 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0726e32740 10 client.0 _ll_register_callbacks cb 0x7f07180103e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0721e37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0721e37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f788a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44782/0 (socket says 172.21.15.89:44782) 2024-09-15T23:30:09.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.650+0000 7f0721e37640 1 -- 172.21.15.89:0/2697749016 learned_addr learned my addr 172.21.15.89:0/2697749016 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0721636640 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f78de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0722638640 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f6f370 0x556bb7f79da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0721e37640 1 -- 172.21.15.89:0/2697749016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 msgr2=0x556bb7f78de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0721e37640 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f78de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0721e37640 1 -- 172.21.15.89:0/2697749016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f6f370 msgr2=0x556bb7f79da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0721e37640 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f6f370 0x556bb7f79da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0721e37640 1 -- 172.21.15.89:0/2697749016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bb7f7a3b0 con 0x556bb7f699a0 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0721e37640 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f788a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f071809b690 tx=0x7f0718097c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ecc3f5deb77cc6a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f07057fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f07057fa640 1 -- 172.21.15.89:0/2697749016 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0718099c90 con 0x556bb7f699a0 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f07057fa640 1 -- 172.21.15.89:0/2697749016 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f07180860b0 con 0x556bb7f699a0 2024-09-15T23:30:09.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0726e32740 1 -- 172.21.15.89:0/2697749016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0714023030 con 0x556bb7f699a0 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f07057fa640 1 -- 172.21.15.89:0/2697749016 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f07180cf430 con 0x556bb7f699a0 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0726e32740 10 client.17568 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0726e32740 1 -- 172.21.15.89:0/2697749016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556bb7fea4e0 con 0x556bb7f699a0 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f07057fa640 1 -- 172.21.15.89:0/2697749016 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0714023030 con 0x556bb7f699a0 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f07057fa640 1 -- 172.21.15.89:0/2697749016 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0718084ce0 con 0x556bb7f699a0 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0726e32740 10 client.17568 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0726e32740 10 client.17568 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0726e32740 20 client.17568 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0726e32740 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556bb7feae60 0x556bb7fed280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0726e32740 4 client.17568 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.651+0000 7f0726e32740 1 -- 172.21.15.89:0/2697749016 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556bb7ed4d70 con 0x556bb7feae60 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.652+0000 7f0721636640 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556bb7feae60 0x556bb7fed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.652+0000 7f07057fa640 10 client.17568 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.652+0000 7f0721636640 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556bb7feae60 0x556bb7fed280 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bbbb08b8f99cf84 server_cookie=4a2b5abd9ff885f4 in_seq=0 out_seq=0 2024-09-15T23:30:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.662+0000 7f07057fa640 1 -- 172.21.15.89:0/2697749016 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556bb7ed4d70 con 0x556bb7feae60 2024-09-15T23:30:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.662+0000 7f07057fa640 10 client.17568 handle_command_reply: tid=0 2024-09-15T23:30:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.662+0000 7f0726e32740 1 client.17568 shutdown 2024-09-15T23:30:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.662+0000 7f0726e32740 1 -- 172.21.15.89:0/2697749016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 msgr2=0x556bb7f788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.662+0000 7f0726e32740 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f788a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f071809b690 tx=0x7f0718097c30 comp rx=0 tx=0).stop 2024-09-15T23:30:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f0726e32740 1 -- 172.21.15.89:0/2697749016 shutdown_connections 2024-09-15T23:30:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f0726e32740 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556bb7feae60 0x556bb7fed280 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f0726e32740 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556bb7f6f370 0x556bb7f79da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f07057fa640 0 client.17568 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f0726e32740 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556bb7f6a2e0 0x556bb7f78de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f0726e32740 1 --2- 172.21.15.89:0/2697749016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556bb7f699a0 0x556bb7f788a0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f0726e32740 1 -- 172.21.15.89:0/2697749016 >> 172.21.15.89:0/2697749016 conn(0x556bb7f65010 msgr2=0x556bb7f639a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f0726e32740 1 -- 172.21.15.89:0/2697749016 shutdown_connections 2024-09-15T23:30:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f0726e32740 1 -- 172.21.15.89:0/2697749016 wait complete. 2024-09-15T23:30:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f0726e32740 20 client.17568 trim_cache size 0 max 16384 2024-09-15T23:30:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:30:09.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:30:09.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:30:09.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:30:09.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 -- 172.21.15.89:0/2989703804 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07080a32f0 msgr2=0x7f07080a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:09.666 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07080a32f0 0x7f07080a57b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0714005fd0 tx=0x7f0714004ab0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 -- 172.21.15.89:0/2989703804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f071c100a50 msgr2=0x7f071c1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f071c100a50 0x7f071c1049e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f071800b450 tx=0x7f071800b910 comp rx=0 tx=0).stop 2024-09-15T23:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 -- 172.21.15.89:0/2989703804 shutdown_connections 2024-09-15T23:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07080a32f0 0x7f07080a57b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f071c102600 0x7f071c103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f071c101c40 0x7f071c103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 --2- 172.21.15.89:0/2989703804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f071c100a50 0x7f071c1049e0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 -- 172.21.15.89:0/2989703804 >> 172.21.15.89:0/2989703804 conn(0x7f071c0fc1b0 msgr2=0x7f071c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 -- 172.21.15.89:0/2989703804 shutdown_connections 2024-09-15T23:30:09.667 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:09.663+0000 7f072409b640 1 -- 172.21.15.89:0/2989703804 wait complete. 2024-09-15T23:30:11.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:11 smithi120 ceph-mon[35250]: pgmap v535: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-09-15T23:30:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:11 smithi174 ceph-mon[35586]: pgmap v535: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-09-15T23:30:11.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:11 smithi089 ceph-mon[30933]: pgmap v535: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-09-15T23:30:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:13 smithi120 ceph-mon[35250]: pgmap v536: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-09-15T23:30:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:13 smithi174 ceph-mon[35586]: pgmap v536: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-09-15T23:30:13.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:13 smithi089 ceph-mon[30933]: pgmap v536: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-09-15T23:30:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:15 smithi120 ceph-mon[35250]: pgmap v537: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-15T23:30:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:15 smithi174 ceph-mon[35586]: pgmap v537: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-15T23:30:15.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:15 smithi089 ceph-mon[30933]: pgmap v537: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-09-15T23:30:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:17 smithi120 ceph-mon[35250]: pgmap v538: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-09-15T23:30:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:17 smithi174 ceph-mon[35586]: pgmap v538: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-09-15T23:30:17.528 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:30:17.557 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:17 smithi089 ceph-mon[30933]: pgmap v538: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2024-09-15T23:30:17.791 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.270+0000 7f566f6ad640 1 -- 172.21.15.89:0/446400416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56681036c0 msgr2=0x7f566810bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.270+0000 7f566f6ad640 1 --2- 172.21.15.89:0/446400416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56681036c0 0x7f566810bbb0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f565c009ca0 tx=0x7f565c02a950 comp rx=0 tx=0).stop 2024-09-15T23:30:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.270+0000 7f566f6ad640 1 -- 172.21.15.89:0/446400416 shutdown_connections 2024-09-15T23:30:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.270+0000 7f566f6ad640 1 --2- 172.21.15.89:0/446400416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566810c180 0x7f566810e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.270+0000 7f566f6ad640 1 --2- 172.21.15.89:0/446400416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56681036c0 0x7f566810bbb0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.270+0000 7f566f6ad640 1 --2- 172.21.15.89:0/446400416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5668102da0 0x7f5668103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.270+0000 7f566f6ad640 1 -- 172.21.15.89:0/446400416 >> 172.21.15.89:0/446400416 conn(0x7f56680fc7a0 msgr2=0x7f56680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:18.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.270+0000 7f566f6ad640 1 -- 172.21.15.89:0/446400416 shutdown_connections 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.270+0000 7f566f6ad640 1 -- 172.21.15.89:0/446400416 wait complete. 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566f6ad640 1 Processor -- start 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566f6ad640 1 -- start start 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566f6ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5668102da0 0x7f5668198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566f6ad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56681036c0 0x7f5668198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566f6ad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566810c180 0x7f566819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566f6ad640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5668111f60 con 0x7f56681036c0 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566f6ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5668111de0 con 0x7f5668102da0 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566f6ad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f56681120e0 con 0x7f566810c180 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566cc48640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56681036c0 0x7f5668198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:18.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566cc48640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56681036c0 0x7f5668198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39672/0 (socket says 172.21.15.89:39672) 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566cc48640 1 -- 172.21.15.89:0/771438344 learned_addr learned my addr 172.21.15.89:0/771438344 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566dc4a640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566810c180 0x7f566819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.271+0000 7f566d449640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5668102da0 0x7f5668198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f566cc48640 1 -- 172.21.15.89:0/771438344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566810c180 msgr2=0x7f566819ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f566cc48640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566810c180 0x7f566819ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f566cc48640 1 -- 172.21.15.89:0/771438344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5668102da0 msgr2=0x7f5668198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f566cc48640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5668102da0 0x7f5668198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f566cc48640 1 -- 172.21.15.89:0/771438344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566819d3d0 con 0x7f56681036c0 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f566cc48640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56681036c0 0x7f5668198960 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f565c005cc0 tx=0x7f565c02f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f56667fc640 1 -- 172.21.15.89:0/771438344 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c034040 con 0x7f56681036c0 2024-09-15T23:30:18.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f56667fc640 1 -- 172.21.15.89:0/771438344 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f565c030920 con 0x7f56681036c0 2024-09-15T23:30:18.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f566f6ad640 1 -- 172.21.15.89:0/771438344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566819d660 con 0x7f56681036c0 2024-09-15T23:30:18.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f56667fc640 1 -- 172.21.15.89:0/771438344 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c004780 con 0x7f56681036c0 2024-09-15T23:30:18.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.272+0000 7f566f6ad640 1 -- 172.21.15.89:0/771438344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56681a4fa0 con 0x7f56681036c0 2024-09-15T23:30:18.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.273+0000 7f566f6ad640 1 -- 172.21.15.89:0/771438344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f566806b010 con 0x7f56681036c0 2024-09-15T23:30:18.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.274+0000 7f56667fc640 1 -- 172.21.15.89:0/771438344 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f565c037020 con 0x7f56681036c0 2024-09-15T23:30:18.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.275+0000 7f56667fc640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56500a3240 0x7f56500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:18.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.275+0000 7f566d449640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56500a3240 0x7f56500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:18.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.275+0000 7f56667fc640 1 -- 172.21.15.89:0/771438344 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f565c0f0d70 con 0x7f56681036c0 2024-09-15T23:30:18.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.275+0000 7f566d449640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56500a3240 0x7f56500a5700 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f56600058c0 tx=0x7f566000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:18.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.278+0000 7f56667fc640 1 -- 172.21.15.89:0/771438344 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f565c0811b0 con 0x7f56681036c0 2024-09-15T23:30:18.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.479+0000 7f566f6ad640 1 -- 172.21.15.89:0/771438344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56681071c0 con 0x7f56681036c0 2024-09-15T23:30:18.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.480+0000 7f56667fc640 1 -- 172.21.15.89:0/771438344 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f565c0bd6a0 con 0x7f56681036c0 2024-09-15T23:30:18.482 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:30:18.482 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:30:18.482 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:30:18.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 -- 172.21.15.89:0/771438344 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56500a3240 msgr2=0x7f56500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56500a3240 0x7f56500a5700 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f56600058c0 tx=0x7f566000a2d0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 -- 172.21.15.89:0/771438344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56681036c0 msgr2=0x7f5668198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56681036c0 0x7f5668198960 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f565c005cc0 tx=0x7f565c02f630 comp rx=0 tx=0).stop 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 -- 172.21.15.89:0/771438344 shutdown_connections 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56500a3240 0x7f56500a5700 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566810c180 0x7f566819ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56681036c0 0x7f5668198960 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 --2- 172.21.15.89:0/771438344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5668102da0 0x7f5668198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 -- 172.21.15.89:0/771438344 >> 172.21.15.89:0/771438344 conn(0x7f56680fc7a0 msgr2=0x7f56680fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 -- 172.21.15.89:0/771438344 shutdown_connections 2024-09-15T23:30:18.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:18.483+0000 7f566f6ad640 1 -- 172.21.15.89:0/771438344 wait complete. 2024-09-15T23:30:18.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:18 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/771438344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:18.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:18 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/771438344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:18.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:18 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/771438344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:19.286 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 73e8cd96 --release --await' 2024-09-15T23:30:19.286 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 73e8cd96 --release --await 2024-09-15T23:30:19.574 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:19.812 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:19 smithi089 ceph-mon[30933]: pgmap v539: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-09-15T23:30:19.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:19 smithi120 ceph-mon[35250]: pgmap v539: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-09-15T23:30:19.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:19 smithi174 ceph-mon[35586]: pgmap v539: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-09-15T23:30:20.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.098+0000 7f54776e1640 1 -- 172.21.15.89:0/1774393543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5470100a80 msgr2=0x7f5470100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:20.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.098+0000 7f54776e1640 1 --2- 172.21.15.89:0/1774393543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5470100a80 0x7f5470100ee0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f5460009930 tx=0x7f546002a420 comp rx=0 tx=0).stop 2024-09-15T23:30:20.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.098+0000 7f54776e1640 1 -- 172.21.15.89:0/1774393543 shutdown_connections 2024-09-15T23:30:20.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.098+0000 7f54776e1640 1 --2- 172.21.15.89:0/1774393543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5470101420 0x7f547010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.098+0000 7f54776e1640 1 --2- 172.21.15.89:0/1774393543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5470100a80 0x7f5470100ee0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.098+0000 7f54776e1640 1 --2- 172.21.15.89:0/1774393543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5470106a80 0x7f5470106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.098+0000 7f54776e1640 1 -- 172.21.15.89:0/1774393543 >> 172.21.15.89:0/1774393543 conn(0x7f54700fc7a0 msgr2=0x7f54700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:20.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.098+0000 7f54776e1640 1 -- 172.21.15.89:0/1774393543 shutdown_connections 2024-09-15T23:30:20.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.098+0000 7f54776e1640 1 -- 172.21.15.89:0/1774393543 wait complete. 2024-09-15T23:30:20.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f54776e1640 1 Processor -- start 2024-09-15T23:30:20.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f54776e1640 1 -- start start 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f54776e1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5470100a80 0x7f5470198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f54776e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5470101420 0x7f54701989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f54776e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5470106a80 0x7f547019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f54776e1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5470110e80 con 0x7f5470101420 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f54776e1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5470110d00 con 0x7f5470100a80 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f54776e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5470111000 con 0x7f5470106a80 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f5474c7c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5470101420 0x7f54701989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f5474c7c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5470101420 0x7f54701989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39682/0 (socket says 172.21.15.89:39682) 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f5474c7c640 1 -- 172.21.15.89:0/1057604732 learned_addr learned my addr 172.21.15.89:0/1057604732 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.099+0000 7f547547d640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5470100a80 0x7f5470198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f5474c7c640 1 -- 172.21.15.89:0/1057604732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5470106a80 msgr2=0x7f547019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:20.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f5474c7c640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5470106a80 0x7f547019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f5474c7c640 1 -- 172.21.15.89:0/1057604732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5470100a80 msgr2=0x7f5470198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:20.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f5474c7c640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5470100a80 0x7f5470198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f5474c7c640 1 -- 172.21.15.89:0/1057604732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547019d420 con 0x7f5470101420 2024-09-15T23:30:20.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f5474c7c640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5470101420 0x7f54701989b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f5460009230 tx=0x7f5460004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:20.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f54667fc640 1 -- 172.21.15.89:0/1057604732 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546003f030 con 0x7f5470101420 2024-09-15T23:30:20.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f54776e1640 1 -- 172.21.15.89:0/1057604732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f547019d6b0 con 0x7f5470101420 2024-09-15T23:30:20.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f54667fc640 1 -- 172.21.15.89:0/1057604732 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5460003ad0 con 0x7f5470101420 2024-09-15T23:30:20.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f54667fc640 1 -- 172.21.15.89:0/1057604732 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460035070 con 0x7f5470101420 2024-09-15T23:30:20.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.100+0000 7f54776e1640 1 -- 172.21.15.89:0/1057604732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54701a4ff0 con 0x7f5470101420 2024-09-15T23:30:20.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.102+0000 7f54667fc640 1 -- 172.21.15.89:0/1057604732 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5460042030 con 0x7f5470101420 2024-09-15T23:30:20.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5479aba740 1 Processor -- start 2024-09-15T23:30:20.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5479aba740 1 -- start start 2024-09-15T23:30:20.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5479aba740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217060 0x5628b7217440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5479aba740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7221d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5479aba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b7224630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5479aba740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5628b717c620 con 0x5628b7217980 2024-09-15T23:30:20.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5479aba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5628b7123e50 con 0x5628b7222240 2024-09-15T23:30:20.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5479aba740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5628b71785e0 con 0x5628b7217060 2024-09-15T23:30:20.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f54667fc640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f545c0a3240 0x7f545c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f547547d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7221d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f547547d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7221d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39688/0 (socket says 172.21.15.89:39688) 2024-09-15T23:30:20.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f547547d640 1 -- 172.21.15.89:0/378027868 learned_addr learned my addr 172.21.15.89:0/378027868 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:20.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5474c7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b7224630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5475c7e640 1 --2- 172.21.15.89:0/378027868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217060 0x5628b7217440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.103+0000 7f5475c7e640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f545c0a3240 0x7f545c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f547547d640 1 -- 172.21.15.89:0/378027868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217060 msgr2=0x5628b7217440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f547547d640 1 --2- 172.21.15.89:0/378027868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217060 0x5628b7217440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f547547d640 1 -- 172.21.15.89:0/378027868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 msgr2=0x5628b7224630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f547547d640 1 --2- 172.21.15.89:0/378027868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b7224630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f547547d640 1 -- 172.21.15.89:0/378027868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b7114d40 con 0x5628b7217980 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f54667fc640 1 -- 172.21.15.89:0/1057604732 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f54600f01f0 con 0x7f5470101420 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f547547d640 1 --2- 172.21.15.89:0/378027868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7221d00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f546c00b710 tx=0x7f546c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6809f5c12bdf394 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f5475c7e640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f545c0a3240 0x7f545c0a5700 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5468009270 tx=0x7f5468005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f545b7fe640 1 -- 172.21.15.89:0/378027868 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c003bd0 con 0x5628b7217980 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f545b7fe640 1 -- 172.21.15.89:0/378027868 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f546c015070 con 0x5628b7217980 2024-09-15T23:30:20.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.104+0000 7f545b7fe640 1 -- 172.21.15.89:0/378027868 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c004e40 con 0x5628b7217980 2024-09-15T23:30:20.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 -- 172.21.15.89:0/378027868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 msgr2=0x5628b7221d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:20.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 --2- 172.21.15.89:0/378027868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7221d00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f546c00b710 tx=0x7f546c004610 comp rx=0 tx=0).stop 2024-09-15T23:30:20.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 -- 172.21.15.89:0/378027868 shutdown_connections 2024-09-15T23:30:20.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 --2- 172.21.15.89:0/378027868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b7224630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 --2- 172.21.15.89:0/378027868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7221d00 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 --2- 172.21.15.89:0/378027868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217060 0x5628b7217440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 -- 172.21.15.89:0/378027868 >> 172.21.15.89:0/378027868 conn(0x5628b72135f0 msgr2=0x5628b72139d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 -- 172.21.15.89:0/378027868 shutdown_connections 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 -- 172.21.15.89:0/378027868 wait complete. 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 -- start start 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.105+0000 7f5479aba740 1 Processor -- start 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5479aba740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217060 0x5628b721f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5479aba740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217980 0x5628b721f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5479aba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b721fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5479aba740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5628b7224e10 con 0x5628b7217060 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5479aba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5628b7104f10 con 0x5628b7222240 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5479aba740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5628b7224f90 con 0x5628b7217980 2024-09-15T23:30:20.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5479aba740 10 client.0 _ll_register_callbacks cb 0x5628b7128db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f547547d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217060 0x5628b721f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f547547d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217060 0x5628b721f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39702/0 (socket says 172.21.15.89:39702) 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f547547d640 1 -- 172.21.15.89:0/2525381303 learned_addr learned my addr 172.21.15.89:0/2525381303 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5474c7c640 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217980 0x5628b721f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f547547d640 1 -- 172.21.15.89:0/2525381303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217980 msgr2=0x5628b721f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f547547d640 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217980 0x5628b721f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f547547d640 1 -- 172.21.15.89:0/2525381303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 msgr2=0x5628b721fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f547547d640 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b721fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f547547d640 1 -- 172.21.15.89:0/2525381303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b6e8b580 con 0x5628b7217060 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f547547d640 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217060 0x5628b721f100 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f546c0039c0 tx=0x7f546c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3877e163aafe1021 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5458ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5458ff9640 1 -- 172.21.15.89:0/2525381303 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c028060 con 0x5628b7217060 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5458ff9640 1 -- 172.21.15.89:0/2525381303 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f546c011440 con 0x5628b7217060 2024-09-15T23:30:20.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.106+0000 7f5479aba740 1 -- 172.21.15.89:0/2525381303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f546c015070 con 0x5628b7217060 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5458ff9640 1 -- 172.21.15.89:0/2525381303 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f546c02f050 con 0x5628b7217060 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5479aba740 10 client.17586 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5479aba740 1 -- 172.21.15.89:0/2525381303 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5628b6f55960 con 0x5628b7217060 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5458ff9640 1 -- 172.21.15.89:0/2525381303 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f546c015070 con 0x5628b7217060 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5458ff9640 1 -- 172.21.15.89:0/2525381303 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f546c022070 con 0x5628b7217060 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5479aba740 10 client.17586 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5479aba740 10 client.17586 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5479aba740 20 client.17586 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5479aba740 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5628b7231de0 0x5628b72341d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5479aba740 4 client.17586 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5479aba740 1 -- 172.21.15.89:0/2525381303 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628b7189b90 con 0x5628b7231de0 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.107+0000 7f5474c7c640 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5628b7231de0 0x5628b72341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.108+0000 7f5458ff9640 10 client.17586 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.108+0000 7f5474c7c640 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5628b7231de0 0x5628b72341d0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=155df4f7c1840f42 server_cookie=26a8088628efb3af in_seq=0 out_seq=0 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.108+0000 7f5458ff9640 1 -- 172.21.15.89:0/2525381303 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5628b7189b90 con 0x5628b7231de0 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.108+0000 7f5458ff9640 10 client.17586 handle_command_reply: tid=0 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.108+0000 7f5479aba740 1 client.17586 shutdown 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 1 -- 172.21.15.89:0/2525381303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217060 msgr2=0x5628b721f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217060 0x5628b721f100 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f546c0039c0 tx=0x7f546c004730 comp rx=0 tx=0).stop 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 1 -- 172.21.15.89:0/2525381303 shutdown_connections 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5628b7231de0 0x5628b72341d0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b721fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217980 0x5628b721f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5458ff9640 0 client.17586 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 1 --2- 172.21.15.89:0/2525381303 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217060 0x5628b721f100 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 1 -- 172.21.15.89:0/2525381303 >> 172.21.15.89:0/2525381303 conn(0x5628b72135f0 msgr2=0x5628b72192d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:20.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 1 -- 172.21.15.89:0/2525381303 shutdown_connections 2024-09-15T23:30:20.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 1 -- 172.21.15.89:0/2525381303 wait complete. 2024-09-15T23:30:20.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.109+0000 7f5479aba740 20 client.17586 trim_cache size 0 max 16384 2024-09-15T23:30:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5479aba740 1 Processor -- start 2024-09-15T23:30:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5479aba740 1 -- start start 2024-09-15T23:30:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5479aba740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217060 0x5628b721dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5479aba740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7239800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5479aba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b7239d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5479aba740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f54701aedf0 con 0x5628b7217980 2024-09-15T23:30:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5479aba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f54701b0930 con 0x5628b7222240 2024-09-15T23:30:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5479aba740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f54701af4c0 con 0x5628b7217060 2024-09-15T23:30:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5474c7c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7239800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5474c7c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7239800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39710/0 (socket says 172.21.15.89:39710) 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.126+0000 7f5474c7c640 1 -- 172.21.15.89:0/3307454082 learned_addr learned my addr 172.21.15.89:0/3307454082 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f547547d640 1 --2- 172.21.15.89:0/3307454082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217060 0x5628b721dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f5475c7e640 1 --2- 172.21.15.89:0/3307454082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b7239d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f5474c7c640 1 -- 172.21.15.89:0/3307454082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217060 msgr2=0x5628b721dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f5474c7c640 1 --2- 172.21.15.89:0/3307454082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217060 0x5628b721dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f5474c7c640 1 -- 172.21.15.89:0/3307454082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 msgr2=0x5628b7239d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f5474c7c640 1 --2- 172.21.15.89:0/3307454082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b7239d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f5474c7c640 1 -- 172.21.15.89:0/3307454082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628b7213090 con 0x5628b7217980 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f5474c7c640 1 --2- 172.21.15.89:0/3307454082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7239800 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f5460009900 tx=0x7f546008f2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc14b900f8e4b96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f545affd640 1 -- 172.21.15.89:0/3307454082 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460049040 con 0x5628b7217980 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f545affd640 1 -- 172.21.15.89:0/3307454082 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f546008e040 con 0x5628b7217980 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.127+0000 7f545affd640 1 -- 172.21.15.89:0/3307454082 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54600ba550 con 0x5628b7217980 2024-09-15T23:30:20.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 -- 172.21.15.89:0/3307454082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 msgr2=0x5628b7239800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 --2- 172.21.15.89:0/3307454082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7239800 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f5460009900 tx=0x7f546008f2a0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 -- 172.21.15.89:0/3307454082 shutdown_connections 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 --2- 172.21.15.89:0/3307454082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7222240 0x5628b7239d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 --2- 172.21.15.89:0/3307454082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b7239800 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 --2- 172.21.15.89:0/3307454082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7217060 0x5628b721dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 -- 172.21.15.89:0/3307454082 >> 172.21.15.89:0/3307454082 conn(0x5628b72135f0 msgr2=0x5628b721d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 -- 172.21.15.89:0/3307454082 shutdown_connections 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 -- 172.21.15.89:0/3307454082 wait complete. 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 -- start start 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.128+0000 7f5479aba740 1 Processor -- start 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5479aba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7217060 0x5628b721c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5479aba740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b721c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5479aba740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7222240 0x5628b721cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5479aba740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f54701b0260 con 0x5628b7217980 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5479aba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f54701afb90 con 0x5628b7217060 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5479aba740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f54701acdd0 con 0x5628b7222240 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5479aba740 10 client.0 _ll_register_callbacks cb 0x7f546c0043a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5474c7c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b721c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5474c7c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b721c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39720/0 (socket says 172.21.15.89:39720) 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5474c7c640 1 -- 172.21.15.89:0/2593352758 learned_addr learned my addr 172.21.15.89:0/2593352758 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f547547d640 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7217060 0x5628b721c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5475c7e640 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7222240 0x5628b721cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5474c7c640 1 -- 172.21.15.89:0/2593352758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7222240 msgr2=0x5628b721cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5474c7c640 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7222240 0x5628b721cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5474c7c640 1 -- 172.21.15.89:0/2593352758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7217060 msgr2=0x5628b721c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:20.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5474c7c640 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7217060 0x5628b721c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5474c7c640 1 -- 172.21.15.89:0/2593352758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628b729b7d0 con 0x5628b7217980 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5475c7e640 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7222240 0x5628b721cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f547547d640 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7217060 0x5628b721c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5474c7c640 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b721c660 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5460009a50 tx=0x7f54600045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c804457600ded46 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.129+0000 7f5458ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5458ff9640 1 -- 172.21.15.89:0/2593352758 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460049070 con 0x5628b7217980 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5458ff9640 1 -- 172.21.15.89:0/2593352758 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f546003b460 con 0x5628b7217980 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5479aba740 1 -- 172.21.15.89:0/2593352758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f546008e040 con 0x5628b7217980 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5458ff9640 1 -- 172.21.15.89:0/2593352758 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f54600bc420 con 0x5628b7217980 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5479aba740 10 client.17592 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5479aba740 1 -- 172.21.15.89:0/2593352758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5628b729caf0 con 0x5628b7217980 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5458ff9640 1 -- 172.21.15.89:0/2593352758 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f546009f070 con 0x5628b7217980 2024-09-15T23:30:20.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5458ff9640 1 -- 172.21.15.89:0/2593352758 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5460031bd0 con 0x5628b7217980 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5479aba740 10 client.17592 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5479aba740 10 client.17592 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5479aba740 20 client.17592 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5479aba740 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5628b729d1b0 0x5628b729f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5479aba740 4 client.17592 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "73e8cd96", "release": true, "await": true}] 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f5479aba740 1 -- 172.21.15.89:0/2593352758 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "73e8cd96", "release": true, "await": true}) -- 0x5628b71840a0 con 0x5628b729d1b0 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.130+0000 7f547547d640 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5628b729d1b0 0x5628b729f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.131+0000 7f5458ff9640 10 client.17592 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.131+0000 7f547547d640 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5628b729d1b0 0x5628b729f5d0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f18cc5805bc5a28d server_cookie=3fe06d424ec9618b in_seq=0 out_seq=0 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.134+0000 7f5458ff9640 1 -- 172.21.15.89:0/2593352758 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5628b71840a0 con 0x5628b729d1b0 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.134+0000 7f5458ff9640 10 client.17592 handle_command_reply: tid=0 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.134+0000 7f5479aba740 1 client.17592 shutdown 2024-09-15T23:30:20.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 1 -- 172.21.15.89:0/2593352758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 msgr2=0x5628b721c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b721c660 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f5460009a50 tx=0x7f54600045f0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 1 -- 172.21.15.89:0/2593352758 shutdown_connections 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5628b729d1b0 0x5628b729f5d0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628b7222240 0x5628b721cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5458ff9640 0 client.17592 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628b7217980 0x5628b721c660 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 1 --2- 172.21.15.89:0/2593352758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628b7217060 0x5628b721c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 1 -- 172.21.15.89:0/2593352758 >> 172.21.15.89:0/2593352758 conn(0x5628b72135f0 msgr2=0x5628b721e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 1 -- 172.21.15.89:0/2593352758 shutdown_connections 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 1 -- 172.21.15.89:0/2593352758 wait complete. 2024-09-15T23:30:20.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f5479aba740 20 client.17592 trim_cache size 0 max 16384 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 24, 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "73e8cd96": { 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 24, 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:30:20.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 145.3, 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:30:20.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:30:20.146 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:30:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f54776e1640 1 -- 172.21.15.89:0/1057604732 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f545c0a3240 msgr2=0x7f545c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f54776e1640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f545c0a3240 0x7f545c0a5700 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5468009270 tx=0x7f5468005bd0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f54776e1640 1 -- 172.21.15.89:0/1057604732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5470101420 msgr2=0x7f54701989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:20.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.135+0000 7f54776e1640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5470101420 0x7f54701989b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f5460009230 tx=0x7f5460004270 comp rx=0 tx=0).stop 2024-09-15T23:30:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.136+0000 7f54776e1640 1 -- 172.21.15.89:0/1057604732 shutdown_connections 2024-09-15T23:30:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.136+0000 7f54776e1640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f545c0a3240 0x7f545c0a5700 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.136+0000 7f54776e1640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5470106a80 0x7f547019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.136+0000 7f54776e1640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5470101420 0x7f54701989b0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.136+0000 7f54776e1640 1 --2- 172.21.15.89:0/1057604732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5470100a80 0x7f5470198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.136+0000 7f54776e1640 1 -- 172.21.15.89:0/1057604732 >> 172.21.15.89:0/1057604732 conn(0x7f54700fc7a0 msgr2=0x7f5470104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.136+0000 7f54776e1640 1 -- 172.21.15.89:0/1057604732 shutdown_connections 2024-09-15T23:30:20.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:20.136+0000 7f54776e1640 1 -- 172.21.15.89:0/1057604732 wait complete. 2024-09-15T23:30:20.950 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '73e8cd96', total seconds elapsed: 28.8 2024-09-15T23:30:21.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:21 smithi120 ceph-mon[35250]: pgmap v540: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-15T23:30:21.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:21 smithi174 ceph-mon[35586]: pgmap v540: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-15T23:30:21.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:21 smithi089 ceph-mon[30933]: pgmap v540: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-09-15T23:30:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 -- 172.21.15.89:0/1073299337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff150102d50 msgr2=0x7ff1501031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 --2- 172.21.15.89:0/1073299337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff150102d50 0x7ff1501031d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff138009900 tx=0x7ff13802a410 comp rx=0 tx=0).stop 2024-09-15T23:30:22.762 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 -- 172.21.15.89:0/1073299337 shutdown_connections 2024-09-15T23:30:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 --2- 172.21.15.89:0/1073299337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff150103710 0x7ff150109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 --2- 172.21.15.89:0/1073299337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff150102d50 0x7ff1501031d0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 --2- 172.21.15.89:0/1073299337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff150101b60 0x7ff150101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 -- 172.21.15.89:0/1073299337 >> 172.21.15.89:0/1073299337 conn(0x7ff1500fd2e0 msgr2=0x7ff1500ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 -- 172.21.15.89:0/1073299337 shutdown_connections 2024-09-15T23:30:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 -- 172.21.15.89:0/1073299337 wait complete. 2024-09-15T23:30:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 Processor -- start 2024-09-15T23:30:22.763 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.761+0000 7ff15597f640 1 -- start start 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff15597f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff150101b60 0x7ff150075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff15597f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff150102d50 0x7ff1500745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff15597f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff150103710 0x7ff150074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff15597f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010d7c0 con 0x7ff150103710 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff15597f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010d640 con 0x7ff150102d50 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff15597f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff15010d940 con 0x7ff150101b60 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff14f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff150103710 0x7ff150074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff14f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff150103710 0x7ff150074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39736/0 (socket says 172.21.15.89:39736) 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff14f7fe640 1 -- 172.21.15.89:0/1743689331 learned_addr learned my addr 172.21.15.89:0/1743689331 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff1467fc640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff150102d50 0x7ff1500745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.766 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff14effd640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff150101b60 0x7ff150075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff14f7fe640 1 -- 172.21.15.89:0/1743689331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff150101b60 msgr2=0x7ff150075fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff14f7fe640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff150101b60 0x7ff150075fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff14f7fe640 1 -- 172.21.15.89:0/1743689331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff150102d50 msgr2=0x7ff1500745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff14f7fe640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff150102d50 0x7ff1500745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff14f7fe640 1 -- 172.21.15.89:0/1743689331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1500753b0 con 0x7ff150103710 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.762+0000 7ff14f7fe640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff150103710 0x7ff150074b20 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff14800cad0 tx=0x7ff14800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.763+0000 7ff15497d640 1 -- 172.21.15.89:0/1743689331 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14800ae20 con 0x7ff150103710 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.763+0000 7ff15497d640 1 -- 172.21.15.89:0/1743689331 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff148005020 con 0x7ff150103710 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.763+0000 7ff15597f640 1 -- 172.21.15.89:0/1743689331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1501a1eb0 con 0x7ff150103710 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.763+0000 7ff15497d640 1 -- 172.21.15.89:0/1743689331 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14800f6f0 con 0x7ff150103710 2024-09-15T23:30:22.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.763+0000 7ff15597f640 1 -- 172.21.15.89:0/1743689331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1501a22c0 con 0x7ff150103710 2024-09-15T23:30:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.765+0000 7ff15497d640 1 -- 172.21.15.89:0/1743689331 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff1480051d0 con 0x7ff150103710 2024-09-15T23:30:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.765+0000 7ff1587f5740 1 Processor -- start 2024-09-15T23:30:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff1587f5740 1 -- start start 2024-09-15T23:30:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00c9a0 0x555c8a00cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a011da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a014830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff1587f5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555c89f702f0 con 0x555c8a00d2e0 2024-09-15T23:30:22.776 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff1587f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555c89f18270 con 0x555c8a012370 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff1587f5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555c89f6c2b0 con 0x555c8a00c9a0 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff14effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a011da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff14effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a011da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39752/0 (socket says 172.21.15.89:39752) 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff14effd640 1 -- 172.21.15.89:0/2887032190 learned_addr learned my addr 172.21.15.89:0/2887032190 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff1467fc640 1 --2- 172.21.15.89:0/2887032190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00c9a0 0x555c8a00cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff15497d640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff13c0a32f0 0x7ff13c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff1467fc640 1 --2- 172.21.15.89:0/2887032190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a014830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff14effd640 1 -- 172.21.15.89:0/2887032190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00c9a0 msgr2=0x555c8a00cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff14effd640 1 --2- 172.21.15.89:0/2887032190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00c9a0 0x555c8a00cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff14effd640 1 -- 172.21.15.89:0/2887032190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 msgr2=0x555c8a014830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff14effd640 1 --2- 172.21.15.89:0/2887032190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a014830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff14effd640 1 -- 172.21.15.89:0/2887032190 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c89f17340 con 0x555c8a00d2e0 2024-09-15T23:30:22.777 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff1467fc640 1 --2- 172.21.15.89:0/2887032190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a014830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff14f7fe640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff13c0a32f0 0x7ff13c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.766+0000 7ff14effd640 1 --2- 172.21.15.89:0/2887032190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a011da0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ff14000bd10 tx=0x7ff14000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf952f3d58732ae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff15497d640 1 -- 172.21.15.89:0/1743689331 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff1480cfd80 con 0x7ff150103710 2024-09-15T23:30:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff145ffb640 1 -- 172.21.15.89:0/2887032190 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140015070 con 0x555c8a00d2e0 2024-09-15T23:30:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff145ffb640 1 -- 172.21.15.89:0/2887032190 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff14001a940 con 0x555c8a00d2e0 2024-09-15T23:30:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff145ffb640 1 -- 172.21.15.89:0/2887032190 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140011a00 con 0x555c8a00d2e0 2024-09-15T23:30:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff1587f5740 1 -- 172.21.15.89:0/2887032190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 msgr2=0x555c8a011da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff1587f5740 1 --2- 172.21.15.89:0/2887032190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a011da0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ff14000bd10 tx=0x7ff14000f730 comp rx=0 tx=0).stop 2024-09-15T23:30:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff14f7fe640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff13c0a32f0 0x7ff13c0a57b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff1480a4690 tx=0x7ff14806edd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:22.778 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff1587f5740 1 -- 172.21.15.89:0/2887032190 shutdown_connections 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff1587f5740 1 --2- 172.21.15.89:0/2887032190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a014830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff1587f5740 1 --2- 172.21.15.89:0/2887032190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a011da0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff1587f5740 1 --2- 172.21.15.89:0/2887032190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00c9a0 0x555c8a00cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.767+0000 7ff1587f5740 1 -- 172.21.15.89:0/2887032190 >> 172.21.15.89:0/2887032190 conn(0x555c8a008010 msgr2=0x555c8a008410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 1 -- 172.21.15.89:0/2887032190 shutdown_connections 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 1 -- 172.21.15.89:0/2887032190 wait complete. 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 1 -- start start 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 1 Processor -- start 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a016c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 0x555c8a017190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a01e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555c8a015340 con 0x555c8a00c9a0 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555c89ef9290 con 0x555c8a012370 2024-09-15T23:30:22.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555c8a0154c0 con 0x555c8a00d2e0 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.768+0000 7ff1587f5740 10 client.0 _ll_register_callbacks cb 0x555c89f2a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a016c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a016c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39756/0 (socket says 172.21.15.89:39756) 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1467fc640 1 -- 172.21.15.89:0/1146558434 learned_addr learned my addr 172.21.15.89:0/1146558434 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff14effd640 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 0x555c8a017190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1467fc640 1 -- 172.21.15.89:0/1146558434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 msgr2=0x555c8a017190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1467fc640 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 0x555c8a017190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1467fc640 1 -- 172.21.15.89:0/1146558434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 msgr2=0x555c8a01e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1467fc640 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a01e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1467fc640 1 -- 172.21.15.89:0/1146558434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c89c83210 con 0x555c8a00c9a0 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff14effd640 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 0x555c8a017190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1467fc640 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a016c50 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff138009900 tx=0x7ff1380318e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c962b823bac6c3d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:22.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff12effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff12effd640 1 -- 172.21.15.89:0/1146558434 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13804a060 con 0x555c8a00c9a0 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff12effd640 1 -- 172.21.15.89:0/1146558434 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff138004250 con 0x555c8a00c9a0 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1587f5740 1 -- 172.21.15.89:0/1146558434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff14001a940 con 0x555c8a00c9a0 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff12effd640 1 -- 172.21.15.89:0/1146558434 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff138051050 con 0x555c8a00c9a0 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1587f5740 10 client.17604 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.769+0000 7ff1587f5740 1 -- 172.21.15.89:0/1146558434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555c89d4f620 con 0x555c8a00c9a0 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.770+0000 7ff12effd640 1 -- 172.21.15.89:0/1146558434 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff14001a940 con 0x555c8a00c9a0 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.770+0000 7ff12effd640 1 -- 172.21.15.89:0/1146558434 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff138044030 con 0x555c8a00c9a0 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.770+0000 7ff1587f5740 10 client.17604 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.770+0000 7ff1587f5740 10 client.17604 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.770+0000 7ff1587f5740 20 client.17604 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.770+0000 7ff1587f5740 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555c8a022810 0x555c8a024c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.770+0000 7ff1587f5740 4 client.17604 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:22.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.770+0000 7ff1587f5740 1 -- 172.21.15.89:0/1146558434 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c89f7da20 con 0x555c8a022810 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.770+0000 7ff14effd640 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555c8a022810 0x555c8a024c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff12effd640 10 client.17604 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff14effd640 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555c8a022810 0x555c8a024c60 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c8766d5e8c490f5 server_cookie=4a2a7346294376e8 in_seq=0 out_seq=0 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff12effd640 1 -- 172.21.15.89:0/1146558434 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555c89f7da20 con 0x555c8a022810 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff12effd640 10 client.17604 handle_command_reply: tid=0 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff1587f5740 1 client.17604 shutdown 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff1587f5740 1 -- 172.21.15.89:0/1146558434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 msgr2=0x555c8a016c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff1587f5740 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a016c50 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff138009900 tx=0x7ff1380318e0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff1587f5740 1 -- 172.21.15.89:0/1146558434 shutdown_connections 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff1587f5740 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555c8a022810 0x555c8a024c60 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff1587f5740 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a01e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff1587f5740 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 0x555c8a017190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff1587f5740 1 --2- 172.21.15.89:0/1146558434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a016c50 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.771+0000 7ff12effd640 0 client.17604 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.772+0000 7ff1587f5740 1 -- 172.21.15.89:0/1146558434 >> 172.21.15.89:0/1146558434 conn(0x555c8a008010 msgr2=0x555c8a009820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.772+0000 7ff1587f5740 1 -- 172.21.15.89:0/1146558434 shutdown_connections 2024-09-15T23:30:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.772+0000 7ff1587f5740 1 -- 172.21.15.89:0/1146558434 wait complete. 2024-09-15T23:30:22.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.772+0000 7ff1587f5740 20 client.17604 trim_cache size 0 max 16384 2024-09-15T23:30:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.786+0000 7ff1587f5740 1 Processor -- start 2024-09-15T23:30:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.786+0000 7ff1587f5740 1 -- start start 2024-09-15T23:30:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.786+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00c9a0 0x555c8a017110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.786+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a01e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.786+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a0069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.786+0000 7ff1587f5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff1501ac9f0 con 0x555c8a00d2e0 2024-09-15T23:30:22.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.786+0000 7ff1587f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff1501ae530 con 0x555c8a012370 2024-09-15T23:30:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.786+0000 7ff1587f5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1501ad0c0 con 0x555c8a00c9a0 2024-09-15T23:30:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff14effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a01e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff14effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a01e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39770/0 (socket says 172.21.15.89:39770) 2024-09-15T23:30:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff14effd640 1 -- 172.21.15.89:0/343546125 learned_addr learned my addr 172.21.15.89:0/343546125 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff1467fc640 1 --2- 172.21.15.89:0/343546125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00c9a0 0x555c8a017110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff1467fc640 1 --2- 172.21.15.89:0/343546125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a0069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff14effd640 1 -- 172.21.15.89:0/343546125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00c9a0 msgr2=0x555c8a017110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff14effd640 1 --2- 172.21.15.89:0/343546125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00c9a0 0x555c8a017110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff14effd640 1 -- 172.21.15.89:0/343546125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 msgr2=0x555c8a0069a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:22.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff14effd640 1 --2- 172.21.15.89:0/343546125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a0069a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff14effd640 1 -- 172.21.15.89:0/343546125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c8a006650 con 0x555c8a00d2e0 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff14effd640 1 --2- 172.21.15.89:0/343546125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a01e190 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff14000fc30 tx=0x7ff140005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34218b9f3c25b361 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff1457fa640 1 -- 172.21.15.89:0/343546125 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140015070 con 0x555c8a00d2e0 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff1457fa640 1 -- 172.21.15.89:0/343546125 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff14000b3a0 con 0x555c8a00d2e0 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.787+0000 7ff1457fa640 1 -- 172.21.15.89:0/343546125 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140011870 con 0x555c8a00d2e0 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 -- 172.21.15.89:0/343546125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 msgr2=0x555c8a01e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 --2- 172.21.15.89:0/343546125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a01e190 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff14000fc30 tx=0x7ff140005280 comp rx=0 tx=0).stop 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 -- 172.21.15.89:0/343546125 shutdown_connections 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 --2- 172.21.15.89:0/343546125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a0069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 --2- 172.21.15.89:0/343546125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00d2e0 0x555c8a01e190 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 --2- 172.21.15.89:0/343546125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00c9a0 0x555c8a017110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 -- 172.21.15.89:0/343546125 >> 172.21.15.89:0/343546125 conn(0x555c8a008010 msgr2=0x555c8a016a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 -- 172.21.15.89:0/343546125 shutdown_connections 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 -- 172.21.15.89:0/343546125 wait complete. 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 -- start start 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.788+0000 7ff1587f5740 1 Processor -- start 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a01b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 0x555c8a01bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1587f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a01cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1587f5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff1501ade60 con 0x555c8a00c9a0 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1587f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff1501ad790 con 0x555c8a012370 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1587f5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1501aa9d0 con 0x555c8a00d2e0 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1587f5740 10 client.0 _ll_register_callbacks cb 0x7ff138005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a01b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a01b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39780/0 (socket says 172.21.15.89:39780) 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1467fc640 1 -- 172.21.15.89:0/3632368839 learned_addr learned my addr 172.21.15.89:0/3632368839 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff14effd640 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 0x555c8a01bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1467fc640 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a01cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1467fc640 1 -- 172.21.15.89:0/3632368839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 msgr2=0x555c8a01bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1467fc640 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 0x555c8a01bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1467fc640 1 -- 172.21.15.89:0/3632368839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 msgr2=0x555c8a01cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1467fc640 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a01cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1467fc640 1 -- 172.21.15.89:0/3632368839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c8a01d200 con 0x555c8a00c9a0 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1467fc640 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a01b6a0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff138005ab0 tx=0x7ff1380318e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fdbdc3c17479bd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff12effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:22.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff12effd640 1 -- 172.21.15.89:0/3632368839 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13805b020 con 0x555c8a00c9a0 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff12effd640 1 -- 172.21.15.89:0/3632368839 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff138004450 con 0x555c8a00c9a0 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.789+0000 7ff1587f5740 1 -- 172.21.15.89:0/3632368839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff14000b3a0 con 0x555c8a00c9a0 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff12effd640 1 -- 172.21.15.89:0/3632368839 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff138066050 con 0x555c8a00c9a0 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff1587f5740 10 client.17610 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff1587f5740 1 -- 172.21.15.89:0/3632368839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555c8a08d540 con 0x555c8a00c9a0 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff12effd640 1 -- 172.21.15.89:0/3632368839 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff14000b3a0 con 0x555c8a00c9a0 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff12effd640 1 -- 172.21.15.89:0/3632368839 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff138044040 con 0x555c8a00c9a0 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff1587f5740 10 client.17610 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff1587f5740 10 client.17610 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff1587f5740 20 client.17610 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff1587f5740 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555c8a08dd80 0x555c8a0901a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff1587f5740 4 client.17610 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff1587f5740 1 -- 172.21.15.89:0/3632368839 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555c8a0120f0 con 0x555c8a08dd80 2024-09-15T23:30:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.790+0000 7ff14effd640 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555c8a08dd80 0x555c8a0901a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.791+0000 7ff12effd640 10 client.17610 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:22.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.791+0000 7ff14effd640 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555c8a08dd80 0x555c8a0901a0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c277fa1d9bc2ff4 server_cookie=43f25b53a8cb6cee in_seq=0 out_seq=0 2024-09-15T23:30:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.799+0000 7ff12effd640 1 -- 172.21.15.89:0/3632368839 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555c8a0120f0 con 0x555c8a08dd80 2024-09-15T23:30:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.799+0000 7ff12effd640 10 client.17610 handle_command_reply: tid=0 2024-09-15T23:30:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.799+0000 7ff1587f5740 1 client.17610 shutdown 2024-09-15T23:30:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.799+0000 7ff1587f5740 1 -- 172.21.15.89:0/3632368839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 msgr2=0x555c8a01b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.799+0000 7ff1587f5740 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a01b6a0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7ff138005ab0 tx=0x7ff1380318e0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff1587f5740 1 -- 172.21.15.89:0/3632368839 shutdown_connections 2024-09-15T23:30:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff1587f5740 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555c8a08dd80 0x555c8a0901a0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.801 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff1587f5740 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c8a012370 0x555c8a01cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff1587f5740 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555c8a00d2e0 0x555c8a01bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff12effd640 0 client.17610 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff1587f5740 1 --2- 172.21.15.89:0/3632368839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c8a00c9a0 0x555c8a01b6a0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff1587f5740 1 -- 172.21.15.89:0/3632368839 >> 172.21.15.89:0/3632368839 conn(0x555c8a008010 msgr2=0x555c8a008e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff1587f5740 1 -- 172.21.15.89:0/3632368839 shutdown_connections 2024-09-15T23:30:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff1587f5740 1 -- 172.21.15.89:0/3632368839 wait complete. 2024-09-15T23:30:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff1587f5740 20 client.17610 trim_cache size 0 max 16384 2024-09-15T23:30:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:30:22.802 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:30:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:30:22.803 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:30:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 -- 172.21.15.89:0/1743689331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff13c0a32f0 msgr2=0x7ff13c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff13c0a32f0 0x7ff13c0a57b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff1480a4690 tx=0x7ff14806edd0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 -- 172.21.15.89:0/1743689331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff150103710 msgr2=0x7ff150074b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff150103710 0x7ff150074b20 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff14800cad0 tx=0x7ff14800cf90 comp rx=0 tx=0).stop 2024-09-15T23:30:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 -- 172.21.15.89:0/1743689331 shutdown_connections 2024-09-15T23:30:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff13c0a32f0 0x7ff13c0a57b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff150103710 0x7ff150074b20 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff150102d50 0x7ff1500745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 --2- 172.21.15.89:0/1743689331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff150101b60 0x7ff150075fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:22.804 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 -- 172.21.15.89:0/1743689331 >> 172.21.15.89:0/1743689331 conn(0x7ff1500fd2e0 msgr2=0x7ff1501060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 -- 172.21.15.89:0/1743689331 shutdown_connections 2024-09-15T23:30:22.805 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:22.800+0000 7ff15597f640 1 -- 172.21.15.89:0/1743689331 wait complete. 2024-09-15T23:30:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:23 smithi120 ceph-mon[35250]: pgmap v541: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-09-15T23:30:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:23 smithi174 ceph-mon[35586]: pgmap v541: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-09-15T23:30:23.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:23 smithi089 ceph-mon[30933]: pgmap v541: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-09-15T23:30:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:25 smithi120 ceph-mon[35250]: pgmap v542: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:30:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:25 smithi174 ceph-mon[35586]: pgmap v542: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:30:25.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:25 smithi089 ceph-mon[30933]: pgmap v542: 129 pgs: 129 active+clean; 2.9 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:30:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:27 smithi120 ceph-mon[35250]: pgmap v543: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-09-15T23:30:27.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:27 smithi174 ceph-mon[35586]: pgmap v543: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-09-15T23:30:27.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:27 smithi089 ceph-mon[30933]: pgmap v543: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-09-15T23:30:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:29 smithi120 ceph-mon[35250]: pgmap v544: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 KiB/s wr, 13 op/s 2024-09-15T23:30:29.489 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:29 smithi174 ceph-mon[35586]: pgmap v544: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 KiB/s wr, 13 op/s 2024-09-15T23:30:29.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:29 smithi089 ceph-mon[30933]: pgmap v544: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 KiB/s wr, 13 op/s 2024-09-15T23:30:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:31 smithi120 ceph-mon[35250]: pgmap v545: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:30:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:31 smithi174 ceph-mon[35586]: pgmap v545: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:30:31.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:31 smithi089 ceph-mon[30933]: pgmap v545: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:30:32.633 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:30:32.893 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:33 smithi089 ceph-mon[30933]: pgmap v546: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:30:33.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.330+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3161452388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd010c160 msgr2=0x7fcfd010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:33.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.330+0000 7fcfd4db6640 1 --2- 172.21.15.89:0/3161452388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd010c160 0x7fcfd010e600 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009ca0 tx=0x7fcfc802a950 comp rx=0 tx=0).stop 2024-09-15T23:30:33.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.330+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3161452388 shutdown_connections 2024-09-15T23:30:33.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.330+0000 7fcfd4db6640 1 --2- 172.21.15.89:0/3161452388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd010c160 0x7fcfd010e600 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.330+0000 7fcfd4db6640 1 --2- 172.21.15.89:0/3161452388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd0103650 0x7fcfd010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.330+0000 7fcfd4db6640 1 --2- 172.21.15.89:0/3161452388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd0102d30 0x7fcfd0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.330+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3161452388 >> 172.21.15.89:0/3161452388 conn(0x7fcfd00fc790 msgr2=0x7fcfd00febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.330+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3161452388 shutdown_connections 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.330+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3161452388 wait complete. 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.331+0000 7fcfd4db6640 1 Processor -- start 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.331+0000 7fcfd4db6640 1 -- start start 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.331+0000 7fcfd4db6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd0102d30 0x7fcfd01983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.331+0000 7fcfd4db6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd0103650 0x7fcfd0198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.331+0000 7fcfd4db6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd010c160 0x7fcfd019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.331+0000 7fcfd4db6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0111f60 con 0x7fcfd0103650 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.331+0000 7fcfd4db6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0111de0 con 0x7fcfd010c160 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.331+0000 7fcfd4db6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd01120e0 con 0x7fcfd0102d30 2024-09-15T23:30:33.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfcdd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd0103650 0x7fcfd0198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfcdd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd0103650 0x7fcfd0198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46392/0 (socket says 172.21.15.89:46392) 2024-09-15T23:30:33.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfcdd9b640 1 -- 172.21.15.89:0/3662244034 learned_addr learned my addr 172.21.15.89:0/3662244034 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:33.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfce59c640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd0102d30 0x7fcfd01983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfced9d640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd010c160 0x7fcfd019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfcdd9b640 1 -- 172.21.15.89:0/3662244034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd0102d30 msgr2=0x7fcfd01983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:33.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd0102d30 0x7fcfd01983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfcdd9b640 1 -- 172.21.15.89:0/3662244034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd010c160 msgr2=0x7fcfd019cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:33.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd010c160 0x7fcfd019cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfcdd9b640 1 -- 172.21.15.89:0/3662244034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd019d370 con 0x7fcfd0103650 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfced9d640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd010c160 0x7fcfd019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd0103650 0x7fcfd0198900 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400c970 tx=0x7fcfc400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfbf7fe640 1 -- 172.21.15.89:0/3662244034 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4007bb0 con 0x7fcfd0103650 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3662244034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd019d660 con 0x7fcfd0103650 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfbf7fe640 1 -- 172.21.15.89:0/3662244034 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc4007d50 con 0x7fcfd0103650 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfbf7fe640 1 -- 172.21.15.89:0/3662244034 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4018820 con 0x7fcfd0103650 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.332+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3662244034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd01a4f40 con 0x7fcfd0103650 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.335+0000 7fcfd718f740 1 Processor -- start 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.335+0000 7fcfd718f740 1 -- start start 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.335+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84e380 0x557f8b84e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.335+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84eca0 0x557f8b85d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.335+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 0x557f8b85fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.335+0000 7fcfd718f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557f8b7b42e0 con 0x557f8b85da90 2024-09-15T23:30:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.335+0000 7fcfd718f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f8b75bce0 con 0x557f8b84eca0 2024-09-15T23:30:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.335+0000 7fcfd718f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557f8b7b02a0 con 0x557f8b84e380 2024-09-15T23:30:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfced9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84e380 0x557f8b84e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfced9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84e380 0x557f8b84e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:48028/0 (socket says 172.21.15.89:48028) 2024-09-15T23:30:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfced9d640 1 -- 172.21.15.89:0/737066437 learned_addr learned my addr 172.21.15.89:0/737066437 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfce59c640 1 --2- 172.21.15.89:0/737066437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84eca0 0x557f8b85d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfbf7fe640 1 -- 172.21.15.89:0/3662244034 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcfc40189c0 con 0x7fcfd0103650 2024-09-15T23:30:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfced9d640 1 -- 172.21.15.89:0/737066437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84eca0 msgr2=0x557f8b85d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfced9d640 1 --2- 172.21.15.89:0/737066437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84eca0 0x557f8b85d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfced9d640 1 -- 172.21.15.89:0/737066437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 msgr2=0x557f8b85fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/737066437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 0x557f8b85fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfced9d640 1 --2- 172.21.15.89:0/737066437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 0x557f8b85fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfced9d640 1 -- 172.21.15.89:0/737066437 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f8b740440 con 0x557f8b84e380 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.336+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/737066437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 0x557f8b85fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.337+0000 7fcfced9d640 1 --2- 172.21.15.89:0/737066437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84e380 0x557f8b84e760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009880 tx=0x7fcfc80098b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8aa2a4b74d7a6ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.337+0000 7fcfbcff9640 1 -- 172.21.15.89:0/737066437 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8034050 con 0x557f8b84e380 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.337+0000 7fcfbcff9640 1 -- 172.21.15.89:0/737066437 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc802fbc0 con 0x557f8b84e380 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.337+0000 7fcfbcff9640 1 -- 172.21.15.89:0/737066437 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80044e0 con 0x557f8b84e380 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.337+0000 7fcfbf7fe640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfb80a3240 0x7fcfb80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.337+0000 7fcfce59c640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfb80a3240 0x7fcfb80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.337+0000 7fcfbf7fe640 1 -- 172.21.15.89:0/3662244034 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcfc40cfd30 con 0x7fcfd0103650 2024-09-15T23:30:33.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 -- 172.21.15.89:0/737066437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84e380 msgr2=0x557f8b84e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 --2- 172.21.15.89:0/737066437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84e380 0x557f8b84e760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009880 tx=0x7fcfc80098b0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfce59c640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfb80a3240 0x7fcfb80a5700 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x557f8b851090 tx=0x7fcfc000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 -- 172.21.15.89:0/737066437 shutdown_connections 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 --2- 172.21.15.89:0/737066437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 0x557f8b85fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 --2- 172.21.15.89:0/737066437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84eca0 0x557f8b85d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 --2- 172.21.15.89:0/737066437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84e380 0x557f8b84e760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 -- 172.21.15.89:0/737066437 >> 172.21.15.89:0/737066437 conn(0x557f8b84aa30 msgr2=0x557f8b84ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 -- 172.21.15.89:0/737066437 shutdown_connections 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 -- 172.21.15.89:0/737066437 wait complete. 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 -- start start 2024-09-15T23:30:33.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 Processor -- start 2024-09-15T23:30:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.338+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b85d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84eca0 0x557f8b85a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b85da90 0x557f8b85ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfd718f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557f8b860660 con 0x557f8b84eca0 2024-09-15T23:30:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfd718f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f8b76f050 con 0x557f8b84e380 2024-09-15T23:30:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfd718f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557f8b8607e0 con 0x557f8b85da90 2024-09-15T23:30:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfcdd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84eca0 0x557f8b85a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfd718f740 10 client.0 _ll_register_callbacks cb 0x557f8b7af160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfced9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b85da90 0x557f8b85ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfcdd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84eca0 0x557f8b85a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46406/0 (socket says 172.21.15.89:46406) 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfced9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b85da90 0x557f8b85ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:48034/0 (socket says 172.21.15.89:48034) 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfcdd9b640 1 -- 172.21.15.89:0/3930976448 learned_addr learned my addr 172.21.15.89:0/3930976448 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfce59c640 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b85d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfce59c640 1 -- 172.21.15.89:0/3930976448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b85da90 msgr2=0x557f8b85ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfce59c640 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b85da90 0x557f8b85ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfce59c640 1 -- 172.21.15.89:0/3930976448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84eca0 msgr2=0x557f8b85a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfce59c640 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84eca0 0x557f8b85a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfce59c640 1 -- 172.21.15.89:0/3930976448 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f8b5253c0 con 0x557f8b84e380 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.339+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84eca0 0x557f8b85a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfce59c640 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b85d3b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00039c0 tx=0x7fcfc00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b157c5d7ad9f39e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:30:33.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/3930976448 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc002c060 con 0x557f8b84e380 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfd718f740 1 -- 172.21.15.89:0/3930976448 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfc802fbc0 con 0x557f8b84e380 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/3930976448 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcfc0017030 con 0x557f8b84e380 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/3930976448 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcfc0003e90 con 0x557f8b84e380 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfd718f740 10 client.26381 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfd718f740 1 -- 172.21.15.89:0/3930976448 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557f8b86c090 con 0x557f8b84e380 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/3930976448 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc802fbc0 con 0x557f8b84e380 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/3930976448 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcfc0011600 con 0x557f8b84e380 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfd718f740 10 client.26381 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfd718f740 10 client.26381 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.340+0000 7fcfd718f740 20 client.26381 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.341+0000 7fcfd718f740 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557f8b86d4a0 0x557f8b86f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.341+0000 7fcfd718f740 4 client.26381 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.341+0000 7fcfd718f740 1 -- 172.21.15.89:0/3930976448 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f8b7c1850 con 0x557f8b86d4a0 2024-09-15T23:30:33.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.341+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557f8b86d4a0 0x557f8b86f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.341+0000 7fcfb5ffb640 10 client.26381 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.341+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557f8b86d4a0 0x557f8b86f8d0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1820b0c8b3ea4f81 server_cookie=bb42a233ba464c4e in_seq=0 out_seq=0 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/3930976448 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557f8b7c1850 con 0x557f8b86d4a0 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfb5ffb640 10 client.26381 handle_command_reply: tid=0 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfd718f740 1 client.26381 shutdown 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfd718f740 1 -- 172.21.15.89:0/3930976448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 msgr2=0x557f8b85d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfd718f740 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b85d3b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00039c0 tx=0x7fcfc00046c0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfd718f740 1 -- 172.21.15.89:0/3930976448 shutdown_connections 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfd718f740 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557f8b86d4a0 0x557f8b86f8d0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfd718f740 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b85da90 0x557f8b85ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfd718f740 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84eca0 0x557f8b85a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfb5ffb640 0 client.26381 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfd718f740 1 --2- 172.21.15.89:0/3930976448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b85d3b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.342+0000 7fcfd718f740 1 -- 172.21.15.89:0/3930976448 >> 172.21.15.89:0/3930976448 conn(0x557f8b84aa30 msgr2=0x557f8b850790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.343+0000 7fcfd718f740 1 -- 172.21.15.89:0/3930976448 shutdown_connections 2024-09-15T23:30:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.343+0000 7fcfd718f740 1 -- 172.21.15.89:0/3930976448 wait complete. 2024-09-15T23:30:33.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.343+0000 7fcfd718f740 20 client.26381 trim_cache size 0 max 16384 2024-09-15T23:30:33.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.360+0000 7fcfd718f740 1 Processor -- start 2024-09-15T23:30:33.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.360+0000 7fcfd718f740 1 -- start start 2024-09-15T23:30:33.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.360+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b86a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.360+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 0x557f8b869900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.360+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 0x557f8b858f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.360+0000 7fcfd718f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd01acb00 con 0x557f8b85da90 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.360+0000 7fcfd718f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd01ae640 con 0x557f8b84e380 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.360+0000 7fcfd718f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd01ad1d0 con 0x557f8b84eca0 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfce59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b86a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfced9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 0x557f8b858f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfce59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b86a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:45940/0 (socket says 172.21.15.89:45940) 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfce59c640 1 -- 172.21.15.89:0/473453007 learned_addr learned my addr 172.21.15.89:0/473453007 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/473453007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 0x557f8b869900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfce59c640 1 -- 172.21.15.89:0/473453007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 msgr2=0x557f8b869900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfce59c640 1 --2- 172.21.15.89:0/473453007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 0x557f8b869900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfce59c640 1 -- 172.21.15.89:0/473453007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 msgr2=0x557f8b858f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:33.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfce59c640 1 --2- 172.21.15.89:0/473453007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 0x557f8b858f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfce59c640 1 -- 172.21.15.89:0/473453007 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f8b859660 con 0x557f8b84e380 2024-09-15T23:30:33.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/473453007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 0x557f8b869900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:33.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfce59c640 1 --2- 172.21.15.89:0/473453007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b86a060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0000c00 tx=0x7fcfc0004470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=570930fd5cbb41a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:33.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfb7fff640 1 -- 172.21.15.89:0/473453007 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0003e90 con 0x557f8b84e380 2024-09-15T23:30:33.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfb7fff640 1 -- 172.21.15.89:0/473453007 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc0011e90 con 0x557f8b84e380 2024-09-15T23:30:33.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.361+0000 7fcfb7fff640 1 -- 172.21.15.89:0/473453007 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0004a20 con 0x557f8b84e380 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.362+0000 7fcfd718f740 1 -- 172.21.15.89:0/473453007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 msgr2=0x557f8b86a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.362+0000 7fcfd718f740 1 --2- 172.21.15.89:0/473453007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b86a060 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0000c00 tx=0x7fcfc0004470 comp rx=0 tx=0).stop 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.362+0000 7fcfd718f740 1 -- 172.21.15.89:0/473453007 shutdown_connections 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.362+0000 7fcfd718f740 1 --2- 172.21.15.89:0/473453007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b85da90 0x557f8b858f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.362+0000 7fcfd718f740 1 --2- 172.21.15.89:0/473453007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 0x557f8b869900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.362+0000 7fcfd718f740 1 --2- 172.21.15.89:0/473453007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b84e380 0x557f8b86a060 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.362+0000 7fcfd718f740 1 -- 172.21.15.89:0/473453007 >> 172.21.15.89:0/473453007 conn(0x557f8b84aa30 msgr2=0x557f8b855d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.362+0000 7fcfd718f740 1 -- 172.21.15.89:0/473453007 shutdown_connections 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.362+0000 7fcfd718f740 1 -- 172.21.15.89:0/473453007 wait complete. 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.362+0000 7fcfd718f740 1 -- start start 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfd718f740 1 Processor -- start 2024-09-15T23:30:33.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84e380 0x557f8b866090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 0x557f8b861410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfd718f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b85da90 0x557f8b861950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfd718f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd01adf70 con 0x557f8b84e380 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfd718f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd01ad8a0 con 0x557f8b85da90 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfd718f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd01aaae0 con 0x557f8b84eca0 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfd718f740 10 client.0 _ll_register_callbacks cb 0x7fcfc0004320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfce59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84e380 0x557f8b866090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfce59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84e380 0x557f8b866090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46426/0 (socket says 172.21.15.89:46426) 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfcdd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 0x557f8b861410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfce59c640 1 -- 172.21.15.89:0/1663626236 learned_addr learned my addr 172.21.15.89:0/1663626236 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfced9d640 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b85da90 0x557f8b861950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfcdd9b640 1 -- 172.21.15.89:0/1663626236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b85da90 msgr2=0x557f8b861950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b85da90 0x557f8b861950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfcdd9b640 1 -- 172.21.15.89:0/1663626236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84e380 msgr2=0x557f8b866090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84e380 0x557f8b866090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfcdd9b640 1 -- 172.21.15.89:0/1663626236 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f8b862040 con 0x557f8b84eca0 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.363+0000 7fcfce59c640 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84e380 0x557f8b866090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfcdd9b640 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 0x557f8b861410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4007590 tx=0x7fcfc40bc190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=241725dcc7116f8e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/1663626236 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc40a4bd0 con 0x557f8b84eca0 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/1663626236 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcfc409a000 con 0x557f8b84eca0 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfd718f740 1 -- 172.21.15.89:0/1663626236 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfc0011e90 con 0x557f8b84eca0 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/1663626236 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcfc409a260 con 0x557f8b84eca0 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfd718f740 10 client.26449 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:33.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfd718f740 1 -- 172.21.15.89:0/1663626236 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557f8b8d80a0 con 0x557f8b84eca0 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/1663626236 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc0011e90 con 0x557f8b84eca0 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/1663626236 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcfc40baba0 con 0x557f8b84eca0 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfd718f740 10 client.26449 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfd718f740 10 client.26449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfd718f740 20 client.26449 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfd718f740 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557f8b8d87d0 0x557f8b8dabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfd718f740 4 client.26449 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.364+0000 7fcfd718f740 1 -- 172.21.15.89:0/1663626236 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557f8b7bbd60 con 0x557f8b8d87d0 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.365+0000 7fcfce59c640 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557f8b8d87d0 0x557f8b8dabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.365+0000 7fcfb5ffb640 10 client.26449 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.365+0000 7fcfce59c640 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557f8b8d87d0 0x557f8b8dabf0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0fcd1a9286272f server_cookie=43ffd1439009545c in_seq=0 out_seq=0 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.365+0000 7fcfb5ffb640 1 -- 172.21.15.89:0/1663626236 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557f8b7bbd60 con 0x557f8b8d87d0 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.365+0000 7fcfb5ffb640 10 client.26449 handle_command_reply: tid=0 2024-09-15T23:30:33.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.365+0000 7fcfd718f740 1 client.26449 shutdown 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 1 -- 172.21.15.89:0/1663626236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 msgr2=0x557f8b861410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 0x557f8b861410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4007590 tx=0x7fcfc40bc190 comp rx=0 tx=0).stop 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 1 -- 172.21.15.89:0/1663626236 shutdown_connections 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557f8b8d87d0 0x557f8b8dabf0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f8b85da90 0x557f8b861950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557f8b84eca0 0x557f8b861410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfb5ffb640 0 client.26449 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 1 --2- 172.21.15.89:0/1663626236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557f8b84e380 0x557f8b866090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 1 -- 172.21.15.89:0/1663626236 >> 172.21.15.89:0/1663626236 conn(0x557f8b84aa30 msgr2=0x557f8b855d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 1 -- 172.21.15.89:0/1663626236 shutdown_connections 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 1 -- 172.21.15.89:0/1663626236 wait complete. 2024-09-15T23:30:33.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd718f740 20 client.26449 trim_cache size 0 max 16384 2024-09-15T23:30:33.376 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:30:33.376 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:30:33.376 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:30:33.376 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:30:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3662244034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfb80a3240 msgr2=0x7fcfb80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd4db6640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfb80a3240 0x7fcfb80a5700 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x557f8b851090 tx=0x7fcfc000b390 comp rx=0 tx=0).stop 2024-09-15T23:30:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3662244034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd0103650 msgr2=0x7fcfd0198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd4db6640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd0103650 0x7fcfd0198900 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400c970 tx=0x7fcfc400ce30 comp rx=0 tx=0).stop 2024-09-15T23:30:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3662244034 shutdown_connections 2024-09-15T23:30:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd4db6640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfb80a3240 0x7fcfb80a5700 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd4db6640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfd010c160 0x7fcfd019cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd4db6640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfd0103650 0x7fcfd0198900 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd4db6640 1 --2- 172.21.15.89:0/3662244034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd0102d30 0x7fcfd01983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:33.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.366+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3662244034 >> 172.21.15.89:0/3662244034 conn(0x7fcfd00fc790 msgr2=0x7fcfd00fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:33.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.367+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3662244034 shutdown_connections 2024-09-15T23:30:33.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:33.367+0000 7fcfd4db6640 1 -- 172.21.15.89:0/3662244034 wait complete. 2024-09-15T23:30:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:33 smithi120 ceph-mon[35250]: pgmap v546: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:30:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:33 smithi174 ceph-mon[35586]: pgmap v546: 129 pgs: 129 active+clean; 3.0 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:30:34.158 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:30:34.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:30:34.236 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.236 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.236 INFO:tasks.workunit.client.0.smithi089.stdout:127/600: open fd = 3 2024-09-15T23:30:34.236 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:128/600: open fd = 3 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:129/600: open fd = 3 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:130/600: open fd = 3 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.237 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:131/600: open fd = 3 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:132/600: open fd = 3 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:133/600: open fd = 3 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:134/600: open fd = 3 2024-09-15T23:30:34.238 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:135/600: open fd = 3 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:136/600: open fd = 3 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:137/600: open fd = 3 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.239 INFO:tasks.workunit.client.0.smithi089.stdout:138/600: open fd = 3 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:139/600: open fd = 3 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:140/600: open fd = 3 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:141/600: open fd = 3 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.240 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:142/600: open fd = 3 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:143/600: open fd = 3 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:144/600: open fd = 3 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:145/600: open fd = 3 2024-09-15T23:30:34.241 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:146/600: open fd = 3 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:147/600: open fd = 3 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:148/600: open fd = 3 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.242 INFO:tasks.workunit.client.0.smithi089.stdout:149/600: open fd = 3 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:150/600: open fd = 3 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:151/600: open fd = 3 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:152/600: open fd = 3 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.243 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:153/600: open fd = 3 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:154/600: open fd = 3 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:155/600: open fd = 3 2024-09-15T23:30:34.244 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:156/600: open fd = 3 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:157/600: open fd = 3 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:158/600: open fd = 3 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.245 INFO:tasks.workunit.client.0.smithi089.stdout:159/600: open fd = 3 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:160/600: open fd = 3 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:161/600: open fd = 3 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:162/600: open fd = 3 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.246 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:163/600: open fd = 3 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:164/600: open fd = 3 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:165/600: open fd = 3 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:166/600: open fd = 3 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.247 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:167/600: open fd = 3 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:168/600: open fd = 3 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:169/600: open fd = 3 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.248 INFO:tasks.workunit.client.0.smithi089.stdout:170/600: open fd = 3 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:171/600: open fd = 3 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:172/600: open fd = 3 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:173/600: open fd = 3 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.249 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:174/600: open fd = 3 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:175/600: open fd = 3 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:176/600: open fd = 3 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:177/600: open fd = 3 2024-09-15T23:30:34.250 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:178/600: open fd = 3 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:179/600: open fd = 3 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:180/600: open fd = 3 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:181/600: open fd = 3 2024-09-15T23:30:34.251 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:182/600: open fd = 3 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:183/600: open fd = 3 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:184/600: open fd = 3 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.252 INFO:tasks.workunit.client.0.smithi089.stdout:185/600: open fd = 3 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:186/600: open fd = 3 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:187/600: open fd = 3 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:188/600: open fd = 3 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.253 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:30:34.416 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:34.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.896+0000 7fc06a44f640 1 -- 172.21.15.89:0/3778334564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc064100a80 msgr2=0x7fc064100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:34.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.896+0000 7fc06a44f640 1 --2- 172.21.15.89:0/3778334564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc064100a80 0x7fc064100ee0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fc04c009930 tx=0x7fc04c02a420 comp rx=0 tx=0).stop 2024-09-15T23:30:34.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.896+0000 7fc06a44f640 1 -- 172.21.15.89:0/3778334564 shutdown_connections 2024-09-15T23:30:34.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.896+0000 7fc06a44f640 1 --2- 172.21.15.89:0/3778334564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc064101420 0x7fc06410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:34.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.896+0000 7fc06a44f640 1 --2- 172.21.15.89:0/3778334564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc064100a80 0x7fc064100ee0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.896+0000 7fc06a44f640 1 --2- 172.21.15.89:0/3778334564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc064106a80 0x7fc064106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.896+0000 7fc06a44f640 1 -- 172.21.15.89:0/3778334564 >> 172.21.15.89:0/3778334564 conn(0x7fc0640fc7a0 msgr2=0x7fc0640febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.896+0000 7fc06a44f640 1 -- 172.21.15.89:0/3778334564 shutdown_connections 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.896+0000 7fc06a44f640 1 -- 172.21.15.89:0/3778334564 wait complete. 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.896+0000 7fc06a44f640 1 Processor -- start 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc06a44f640 1 -- start start 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc06a44f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc064100a80 0x7fc064199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc06a44f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc064101420 0x7fc0641996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc06a44f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc064106a80 0x7fc0641a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc06a44f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc064110e80 con 0x7fc064106a80 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc06a44f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc064110d00 con 0x7fc064101420 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc06a44f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc064111000 con 0x7fc064100a80 2024-09-15T23:30:34.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc0689ec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc064106a80 0x7fc0641a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc0689ec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc064106a80 0x7fc0641a41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46432/0 (socket says 172.21.15.89:46432) 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc0689ec640 1 -- 172.21.15.89:0/3838924505 learned_addr learned my addr 172.21.15.89:0/3838924505 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc063fff640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc064100a80 0x7fc064199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc05bfff640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc064101420 0x7fc0641996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc0689ec640 1 -- 172.21.15.89:0/3838924505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc064100a80 msgr2=0x7fc064199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc0689ec640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc064100a80 0x7fc064199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc0689ec640 1 -- 172.21.15.89:0/3838924505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc064101420 msgr2=0x7fc0641996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc0689ec640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc064101420 0x7fc0641996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc0689ec640 1 -- 172.21.15.89:0/3838924505 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0641a4730 con 0x7fc064106a80 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc063fff640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc064100a80 0x7fc064199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc0689ec640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc064106a80 0x7fc0641a41f0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00cce0 tx=0x7fc05c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.897+0000 7fc061ffb640 1 -- 172.21.15.89:0/3838924505 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c00f450 con 0x7fc064106a80 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.898+0000 7fc06a44f640 1 -- 172.21.15.89:0/3838924505 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0641a4900 con 0x7fc064106a80 2024-09-15T23:30:34.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.898+0000 7fc061ffb640 1 -- 172.21.15.89:0/3838924505 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc05c00fad0 con 0x7fc064106a80 2024-09-15T23:30:34.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.898+0000 7fc061ffb640 1 -- 172.21.15.89:0/3838924505 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc05c018c10 con 0x7fc064106a80 2024-09-15T23:30:34.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.898+0000 7fc06a44f640 1 -- 172.21.15.89:0/3838924505 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0641a4dc0 con 0x7fc064106a80 2024-09-15T23:30:34.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.898+0000 7fc06a44f640 1 -- 172.21.15.89:0/3838924505 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc064100ee0 con 0x7fc064106a80 2024-09-15T23:30:34.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.900+0000 7fc061ffb640 1 -- 172.21.15.89:0/3838924505 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc05c00f5f0 con 0x7fc064106a80 2024-09-15T23:30:34.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.900+0000 7fc061ffb640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0500a3240 0x7fc0500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:34.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.900+0000 7fc061ffb640 1 -- 172.21.15.89:0/3838924505 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc05c0d0500 con 0x7fc064106a80 2024-09-15T23:30:34.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.900+0000 7fc063fff640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0500a3240 0x7fc0500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:34.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.901+0000 7fc063fff640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0500a3240 0x7fc0500a5700 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc054005e10 tx=0x7fc054005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:34.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:34.903+0000 7fc061ffb640 1 -- 172.21.15.89:0/3838924505 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc05c014030 con 0x7fc064106a80 2024-09-15T23:30:35.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.103+0000 7fc06a44f640 1 -- 172.21.15.89:0/3838924505 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0641a5080 con 0x7fc064106a80 2024-09-15T23:30:35.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.104+0000 7fc061ffb640 1 -- 172.21.15.89:0/3838924505 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fc05c09cec0 con 0x7fc064106a80 2024-09-15T23:30:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:30:35.106 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:30:35.107 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:30:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 -- 172.21.15.89:0/3838924505 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0500a3240 msgr2=0x7fc0500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0500a3240 0x7fc0500a5700 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc054005e10 tx=0x7fc054005da0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 -- 172.21.15.89:0/3838924505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc064106a80 msgr2=0x7fc0641a41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc064106a80 0x7fc0641a41f0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fc05c00cce0 tx=0x7fc05c00a620 comp rx=0 tx=0).stop 2024-09-15T23:30:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 -- 172.21.15.89:0/3838924505 shutdown_connections 2024-09-15T23:30:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0500a3240 0x7fc0500a5700 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc064106a80 0x7fc0641a41f0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc064101420 0x7fc0641996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 --2- 172.21.15.89:0/3838924505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc064100a80 0x7fc064199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 -- 172.21.15.89:0/3838924505 >> 172.21.15.89:0/3838924505 conn(0x7fc0640fc7a0 msgr2=0x7fc0640fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 -- 172.21.15.89:0/3838924505 shutdown_connections 2024-09-15T23:30:35.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:35.106+0000 7fc06a44f640 1 -- 172.21.15.89:0/3838924505 wait complete. 2024-09-15T23:30:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:35 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3838924505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:35.480 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:35 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3838924505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:35 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3838924505' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:35.801 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:30:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.888+0000 7f5631d49640 1 -- 172.21.15.89:0/2426988093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c100a40 msgr2=0x7f562c100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.888+0000 7f5631d49640 1 --2- 172.21.15.89:0/2426988093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c100a40 0x7f562c100e40 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f561c009900 tx=0x7f561c02a410 comp rx=0 tx=0).stop 2024-09-15T23:30:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.888+0000 7f5631d49640 1 -- 172.21.15.89:0/2426988093 shutdown_connections 2024-09-15T23:30:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.888+0000 7f5631d49640 1 --2- 172.21.15.89:0/2426988093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f562c1025f0 0x7f562c108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.888+0000 7f5631d49640 1 --2- 172.21.15.89:0/2426988093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f562c101c30 0x7f562c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.888+0000 7f5631d49640 1 --2- 172.21.15.89:0/2426988093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c100a40 0x7f562c100e40 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.888+0000 7f5631d49640 1 -- 172.21.15.89:0/2426988093 >> 172.21.15.89:0/2426988093 conn(0x7f562c0fc1e0 msgr2=0x7f562c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:35.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.888+0000 7f5631d49640 1 -- 172.21.15.89:0/2426988093 shutdown_connections 2024-09-15T23:30:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.888+0000 7f5631d49640 1 -- 172.21.15.89:0/2426988093 wait complete. 2024-09-15T23:30:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.889+0000 7f5631d49640 1 Processor -- start 2024-09-15T23:30:35.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.889+0000 7f5631d49640 1 -- start start 2024-09-15T23:30:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.889+0000 7f5631d49640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f562c100a40 0x7f562c1049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.889+0000 7f5631d49640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c101c30 0x7f562c103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.889+0000 7f5631d49640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f562c1025f0 0x7f562c103550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.889+0000 7f5631d49640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f562c10c6a0 con 0x7f562c101c30 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.889+0000 7f5631d49640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f562c10c520 con 0x7f562c100a40 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.889+0000 7f5631d49640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f562c10c820 con 0x7f562c1025f0 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.889+0000 7f562affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c101c30 0x7f562c103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f562affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c101c30 0x7f562c103010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39002/0 (socket says 172.21.15.89:39002) 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f562affd640 1 -- 172.21.15.89:0/3887950803 learned_addr learned my addr 172.21.15.89:0/3887950803 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f562b7fe640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f562c100a40 0x7f562c1049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f562affd640 1 -- 172.21.15.89:0/3887950803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f562c1025f0 msgr2=0x7f562c103550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f562affd640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f562c1025f0 0x7f562c103550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f562affd640 1 -- 172.21.15.89:0/3887950803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f562c100a40 msgr2=0x7f562c1049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f562affd640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f562c100a40 0x7f562c1049d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f562affd640 1 -- 172.21.15.89:0/3887950803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562c103de0 con 0x7f562c101c30 2024-09-15T23:30:35.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f562affd640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c101c30 0x7f562c103010 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f562000d950 tx=0x7f562000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:35.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f5628ff9640 1 -- 172.21.15.89:0/3887950803 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5620017070 con 0x7f562c101c30 2024-09-15T23:30:35.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f5631d49640 1 -- 172.21.15.89:0/3887950803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f562c1a1f30 con 0x7f562c101c30 2024-09-15T23:30:35.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f5628ff9640 1 -- 172.21.15.89:0/3887950803 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5620009c30 con 0x7f562c101c30 2024-09-15T23:30:35.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f5628ff9640 1 -- 172.21.15.89:0/3887950803 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562000f6b0 con 0x7f562c101c30 2024-09-15T23:30:35.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.890+0000 7f5631d49640 1 -- 172.21.15.89:0/3887950803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f562c1a23f0 con 0x7f562c101c30 2024-09-15T23:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.892+0000 7f5634bbb740 1 Processor -- start 2024-09-15T23:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.892+0000 7f5634bbb740 1 -- start start 2024-09-15T23:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.892+0000 7f5628ff9640 1 -- 172.21.15.89:0/3887950803 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f56200109d0 con 0x7f562c101c30 2024-09-15T23:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663caada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cab2e0 0x555663cafda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cb0370 0x555663cb2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f5634bbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555663c0e2f0 con 0x555663caa9a0 2024-09-15T23:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f5634bbb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555663bb6270 con 0x555663cb0370 2024-09-15T23:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f5634bbb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555663c0a2b0 con 0x555663cab2e0 2024-09-15T23:30:35.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663caada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663caada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39014/0 (socket says 172.21.15.89:39014) 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562bfff640 1 -- 172.21.15.89:0/529957631 learned_addr learned my addr 172.21.15.89:0/529957631 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562b7fe640 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cab2e0 0x555663cafda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562affd640 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cb0370 0x555663cb2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562bfff640 1 -- 172.21.15.89:0/529957631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cab2e0 msgr2=0x555663cafda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562bfff640 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cab2e0 0x555663cafda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562bfff640 1 -- 172.21.15.89:0/529957631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cb0370 msgr2=0x555663cb2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562bfff640 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cb0370 0x555663cb2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562bfff640 1 -- 172.21.15.89:0/529957631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555663bb5340 con 0x555663caa9a0 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562b7fe640 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cab2e0 0x555663cafda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562affd640 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cb0370 0x555663cb2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f5628ff9640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56140a32f0 0x7f56140a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562bfff640 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663caada0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f5624009ea0 tx=0x7f5624009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee381b2d9223ea24 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.893+0000 7f562b7fe640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56140a32f0 0x7f56140a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5611ffb640 1 -- 172.21.15.89:0/529957631 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624004d20 con 0x555663caa9a0 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5628ff9640 1 -- 172.21.15.89:0/3887950803 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f56200cf9c0 con 0x7f562c101c30 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5611ffb640 1 -- 172.21.15.89:0/529957631 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562400ee60 con 0x555663caa9a0 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5611ffb640 1 -- 172.21.15.89:0/529957631 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56240107f0 con 0x555663caa9a0 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5634bbb740 1 -- 172.21.15.89:0/529957631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 msgr2=0x555663caada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5634bbb740 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663caada0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f5624009ea0 tx=0x7f5624009f90 comp rx=0 tx=0).stop 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f562b7fe640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56140a32f0 0x7f56140a57b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x555663cabd10 tx=0x7f561c0095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5634bbb740 1 -- 172.21.15.89:0/529957631 shutdown_connections 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5634bbb740 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cb0370 0x555663cb2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5634bbb740 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cab2e0 0x555663cafda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5634bbb740 1 --2- 172.21.15.89:0/529957631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663caada0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5634bbb740 1 -- 172.21.15.89:0/529957631 >> 172.21.15.89:0/529957631 conn(0x555663ca6010 msgr2=0x555663ca6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:35.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5634bbb740 1 -- 172.21.15.89:0/529957631 shutdown_connections 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.894+0000 7f5634bbb740 1 -- 172.21.15.89:0/529957631 wait complete. 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f5634bbb740 1 -- start start 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f5634bbb740 1 Processor -- start 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663cb4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cab2e0 0x555663cb51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cb0370 0x555663cbc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f5634bbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555663cb3390 con 0x555663caa9a0 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f5634bbb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555663b97290 con 0x555663cab2e0 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f5634bbb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555663cb3510 con 0x555663cb0370 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f5634bbb740 10 client.0 _ll_register_callbacks cb 0x555663bc8140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f562b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663cb4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f562b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663cb4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39028/0 (socket says 172.21.15.89:39028) 2024-09-15T23:30:35.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f562b7fe640 1 -- 172.21.15.89:0/3792168813 learned_addr learned my addr 172.21.15.89:0/3792168813 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.895+0000 7f562bfff640 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cb0370 0x555663cbc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f562affd640 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cab2e0 0x555663cb51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f562b7fe640 1 -- 172.21.15.89:0/3792168813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cb0370 msgr2=0x555663cbc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f562b7fe640 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cb0370 0x555663cbc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f562b7fe640 1 -- 172.21.15.89:0/3792168813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cab2e0 msgr2=0x555663cb51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f562b7fe640 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cab2e0 0x555663cb51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f562b7fe640 1 -- 172.21.15.89:0/3792168813 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555663921210 con 0x555663caa9a0 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f562affd640 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cab2e0 0x555663cb51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f562b7fe640 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663cb4c70 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f561c009900 tx=0x7f561c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2c23ece52705dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f560affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f560affd640 1 -- 172.21.15.89:0/3792168813 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561c04b020 con 0x555663caa9a0 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f560affd640 1 -- 172.21.15.89:0/3792168813 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f561c005050 con 0x555663caa9a0 2024-09-15T23:30:35.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f5634bbb740 1 -- 172.21.15.89:0/3792168813 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f562400ee60 con 0x555663caa9a0 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f560affd640 1 -- 172.21.15.89:0/3792168813 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f561c056050 con 0x555663caa9a0 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f5634bbb740 10 client.17643 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f5634bbb740 1 -- 172.21.15.89:0/3792168813 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5556639ed620 con 0x555663caa9a0 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.896+0000 7f560affd640 1 -- 172.21.15.89:0/3792168813 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f562400ee60 con 0x555663caa9a0 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.897+0000 7f560affd640 1 -- 172.21.15.89:0/3792168813 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f561c004cc0 con 0x555663caa9a0 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.897+0000 7f5634bbb740 10 client.17643 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.897+0000 7f5634bbb740 10 client.17643 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.897+0000 7f5634bbb740 20 client.17643 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.897+0000 7f5634bbb740 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555663cc08c0 0x555663cc2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.897+0000 7f5634bbb740 4 client.17643 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.897+0000 7f5634bbb740 1 -- 172.21.15.89:0/3792168813 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555663c1ba20 con 0x555663cc08c0 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.897+0000 7f562affd640 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555663cc08c0 0x555663cc2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.897+0000 7f562affd640 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555663cc08c0 0x555663cc2d10 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cc28a7a3d19ca99 server_cookie=2083066d722a4727 in_seq=0 out_seq=0 2024-09-15T23:30:35.906 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.897+0000 7f560affd640 10 client.17643 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f560affd640 1 -- 172.21.15.89:0/3792168813 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555663c1ba20 con 0x555663cc08c0 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f560affd640 10 client.17643 handle_command_reply: tid=0 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 client.17643 shutdown 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 -- 172.21.15.89:0/3792168813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 msgr2=0x555663cb4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663cb4c70 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f561c009900 tx=0x7f561c02a410 comp rx=0 tx=0).stop 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 -- 172.21.15.89:0/3792168813 shutdown_connections 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555663cc08c0 0x555663cc2d10 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663cb0370 0x555663cbc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663cab2e0 0x555663cb51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f560affd640 0 client.17643 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 --2- 172.21.15.89:0/3792168813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663caa9a0 0x555663cb4c70 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 -- 172.21.15.89:0/3792168813 >> 172.21.15.89:0/3792168813 conn(0x555663ca6010 msgr2=0x555663ca7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 -- 172.21.15.89:0/3792168813 shutdown_connections 2024-09-15T23:30:35.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.898+0000 7f5634bbb740 1 -- 172.21.15.89:0/3792168813 wait complete. 2024-09-15T23:30:35.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.899+0000 7f5634bbb740 20 client.17643 trim_cache size 0 max 16384 2024-09-15T23:30:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f5634bbb740 1 Processor -- start 2024-09-15T23:30:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f5634bbb740 1 -- start start 2024-09-15T23:30:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663ca6010 0x555663cb5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663caa9a0 0x555663caad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f5634bbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f562c1acbd0 con 0x555663cab2e0 2024-09-15T23:30:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f5634bbb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f562c1ae710 con 0x555663ca6010 2024-09-15T23:30:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f5634bbb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f562c1ad2a0 con 0x555663caa9a0 2024-09-15T23:30:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39036/0 (socket says 172.21.15.89:39036) 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562bfff640 1 -- 172.21.15.89:0/3120644144 learned_addr learned my addr 172.21.15.89:0/3120644144 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562b7fe640 1 --2- 172.21.15.89:0/3120644144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663ca6010 0x555663cb5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562affd640 1 --2- 172.21.15.89:0/3120644144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663caa9a0 0x555663caad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562bfff640 1 -- 172.21.15.89:0/3120644144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663caa9a0 msgr2=0x555663caad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562bfff640 1 --2- 172.21.15.89:0/3120644144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663caa9a0 0x555663caad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562bfff640 1 -- 172.21.15.89:0/3120644144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663ca6010 msgr2=0x555663cb5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562bfff640 1 --2- 172.21.15.89:0/3120644144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663ca6010 0x555663cb5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562bfff640 1 -- 172.21.15.89:0/3120644144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555663ca4650 con 0x555663cab2e0 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.913+0000 7f562bfff640 1 --2- 172.21.15.89:0/3120644144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbc180 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f5624009270 tx=0x7f5624005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae8c411a3a1b6b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f56117fa640 1 -- 172.21.15.89:0/3120644144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562401d040 con 0x555663cab2e0 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f56117fa640 1 -- 172.21.15.89:0/3120644144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5624028070 con 0x555663cab2e0 2024-09-15T23:30:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f56117fa640 1 -- 172.21.15.89:0/3120644144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56240237f0 con 0x555663cab2e0 2024-09-15T23:30:35.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f5634bbb740 1 -- 172.21.15.89:0/3120644144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 msgr2=0x555663cbc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f5634bbb740 1 --2- 172.21.15.89:0/3120644144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbc180 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f5624009270 tx=0x7f5624005bd0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f5634bbb740 1 -- 172.21.15.89:0/3120644144 shutdown_connections 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f5634bbb740 1 --2- 172.21.15.89:0/3120644144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbc180 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f5634bbb740 1 --2- 172.21.15.89:0/3120644144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663caa9a0 0x555663caad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f5634bbb740 1 --2- 172.21.15.89:0/3120644144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663ca6010 0x555663cb5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f5634bbb740 1 -- 172.21.15.89:0/3120644144 >> 172.21.15.89:0/3120644144 conn(0x555663cafec0 msgr2=0x555663ca49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f5634bbb740 1 -- 172.21.15.89:0/3120644144 shutdown_connections 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.914+0000 7f5634bbb740 1 -- 172.21.15.89:0/3120644144 wait complete. 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f5634bbb740 1 -- start start 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f5634bbb740 1 Processor -- start 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663ca6010 0x555663cb9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663caa9a0 0x555663cb9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f5634bbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f5634bbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f562c1ae040 con 0x555663cab2e0 2024-09-15T23:30:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f5634bbb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f562c1ad970 con 0x555663caa9a0 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f5634bbb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f562c1aabb0 con 0x555663ca6010 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f5634bbb740 10 client.0 _ll_register_callbacks cb 0x7f561c03f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f562bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f562bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39044/0 (socket says 172.21.15.89:39044) 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f562bfff640 1 -- 172.21.15.89:0/654314984 learned_addr learned my addr 172.21.15.89:0/654314984 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f562b7fe640 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663ca6010 0x555663cb9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f562affd640 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663caa9a0 0x555663cb9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f562bfff640 1 -- 172.21.15.89:0/654314984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663ca6010 msgr2=0x555663cb9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f562bfff640 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663ca6010 0x555663cb9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f562bfff640 1 -- 172.21.15.89:0/654314984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663caa9a0 msgr2=0x555663cb9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.915+0000 7f562bfff640 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663caa9a0 0x555663cb9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f562bfff640 1 -- 172.21.15.89:0/654314984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555663cbb480 con 0x555663cab2e0 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f562b7fe640 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663ca6010 0x555663cb9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f562bfff640 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbae70 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f5624000f80 tx=0x7f5624018b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512742080c248827 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f560affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f560affd640 1 -- 172.21.15.89:0/654314984 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624032020 con 0x555663cab2e0 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f560affd640 1 -- 172.21.15.89:0/654314984 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5624004b20 con 0x555663cab2e0 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f5634bbb740 1 -- 172.21.15.89:0/654314984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5624028070 con 0x555663cab2e0 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f560affd640 1 -- 172.21.15.89:0/654314984 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f562403d050 con 0x555663cab2e0 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f5634bbb740 10 client.17649 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f5634bbb740 1 -- 172.21.15.89:0/654314984 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555663d2b650 con 0x555663cab2e0 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f560affd640 1 -- 172.21.15.89:0/654314984 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5624028070 con 0x555663cab2e0 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f560affd640 1 -- 172.21.15.89:0/654314984 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5624005050 con 0x555663cab2e0 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f5634bbb740 10 client.17649 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f5634bbb740 10 client.17649 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f5634bbb740 20 client.17649 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f5634bbb740 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555663cb0370 0x555663d2de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f5634bbb740 4 client.17649 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:30:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.916+0000 7f5634bbb740 1 -- 172.21.15.89:0/654314984 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555663c15d70 con 0x555663cb0370 2024-09-15T23:30:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.917+0000 7f562b7fe640 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555663cb0370 0x555663d2de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.917+0000 7f560affd640 10 client.17649 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:35.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.917+0000 7f562b7fe640 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555663cb0370 0x555663d2de30 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e273b8ecad1c4eb server_cookie=489547485ea20582 in_seq=0 out_seq=0 2024-09-15T23:30:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.931+0000 7f560affd640 1 -- 172.21.15.89:0/654314984 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555663c15d70 con 0x555663cb0370 2024-09-15T23:30:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.931+0000 7f560affd640 10 client.17649 handle_command_reply: tid=0 2024-09-15T23:30:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.931+0000 7f5634bbb740 1 client.17649 shutdown 2024-09-15T23:30:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 1 -- 172.21.15.89:0/654314984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 msgr2=0x555663cbae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbae70 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f5624000f80 tx=0x7f5624018b20 comp rx=0 tx=0).stop 2024-09-15T23:30:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 1 -- 172.21.15.89:0/654314984 shutdown_connections 2024-09-15T23:30:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555663cb0370 0x555663d2de30 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555663cab2e0 0x555663cbae70 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f560affd640 0 client.17649 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555663caa9a0 0x555663cb9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 1 --2- 172.21.15.89:0/654314984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555663ca6010 0x555663cb9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 1 -- 172.21.15.89:0/654314984 >> 172.21.15.89:0/654314984 conn(0x555663cafec0 msgr2=0x555663ca51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 1 -- 172.21.15.89:0/654314984 shutdown_connections 2024-09-15T23:30:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 1 -- 172.21.15.89:0/654314984 wait complete. 2024-09-15T23:30:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5634bbb740 20 client.17649 trim_cache size 0 max 16384 2024-09-15T23:30:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:30:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:30:35.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:30:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:30:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5631d49640 1 -- 172.21.15.89:0/3887950803 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56140a32f0 msgr2=0x7f56140a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5631d49640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56140a32f0 0x7f56140a57b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x555663cabd10 tx=0x7f561c0095f0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5631d49640 1 -- 172.21.15.89:0/3887950803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c101c30 msgr2=0x7f562c103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:35.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5631d49640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c101c30 0x7f562c103010 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f562000d950 tx=0x7f562000de10 comp rx=0 tx=0).stop 2024-09-15T23:30:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5631d49640 1 -- 172.21.15.89:0/3887950803 shutdown_connections 2024-09-15T23:30:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5631d49640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f56140a32f0 0x7f56140a57b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5631d49640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f562c1025f0 0x7f562c103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5631d49640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f562c101c30 0x7f562c103010 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5631d49640 1 --2- 172.21.15.89:0/3887950803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f562c100a40 0x7f562c1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.932+0000 7f5631d49640 1 -- 172.21.15.89:0/3887950803 >> 172.21.15.89:0/3887950803 conn(0x7f562c0fc1e0 msgr2=0x7f562c0fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.933+0000 7f5631d49640 1 -- 172.21.15.89:0/3887950803 shutdown_connections 2024-09-15T23:30:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:35.933+0000 7f5631d49640 1 -- 172.21.15.89:0/3887950803 wait complete. 2024-09-15T23:30:36.058 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:36.444 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:36 smithi089 ceph-mon[30933]: pgmap v547: 129 pgs: 129 active+clean; 3.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:30:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:36 smithi120 ceph-mon[35250]: pgmap v547: 129 pgs: 129 active+clean; 3.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:30:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:36 smithi174 ceph-mon[35586]: pgmap v547: 129 pgs: 129 active+clean; 3.0 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:30:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.521+0000 7f182116c640 1 -- 172.21.15.89:0/1198903530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f181c102d40 msgr2=0x7f181c103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.521+0000 7f182116c640 1 --2- 172.21.15.89:0/1198903530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f181c102d40 0x7f181c103120 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f180c009900 tx=0x7f180c02a480 comp rx=0 tx=0).stop 2024-09-15T23:30:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.521+0000 7f182116c640 1 -- 172.21.15.89:0/1198903530 shutdown_connections 2024-09-15T23:30:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.521+0000 7f182116c640 1 --2- 172.21.15.89:0/1198903530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f181c10c170 0x7f181c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.521+0000 7f182116c640 1 --2- 172.21.15.89:0/1198903530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f181c103660 0x7f181c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.521+0000 7f182116c640 1 --2- 172.21.15.89:0/1198903530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f181c102d40 0x7f181c103120 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.521+0000 7f182116c640 1 -- 172.21.15.89:0/1198903530 >> 172.21.15.89:0/1198903530 conn(0x7f181c0fc7a0 msgr2=0x7f181c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.521+0000 7f182116c640 1 -- 172.21.15.89:0/1198903530 shutdown_connections 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.521+0000 7f182116c640 1 -- 172.21.15.89:0/1198903530 wait complete. 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f182116c640 1 Processor -- start 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f182116c640 1 -- start start 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f182116c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f181c102d40 0x7f181c1009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f182116c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f181c103660 0x7f181c100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f182116c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f181c10c170 0x7f181c101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f182116c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f181c111f70 con 0x7f181c103660 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f182116c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f181c111df0 con 0x7f181c102d40 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f182116c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f181c1120f0 con 0x7f181c10c170 2024-09-15T23:30:36.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f1813fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f181c103660 0x7f181c100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f1813fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f181c103660 0x7f181c100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39058/0 (socket says 172.21.15.89:39058) 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.522+0000 7f1813fff640 1 -- 172.21.15.89:0/1047839254 learned_addr learned my addr 172.21.15.89:0/1047839254 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f1813fff640 1 -- 172.21.15.89:0/1047839254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f181c10c170 msgr2=0x7f181c101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f1813fff640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f181c10c170 0x7f181c101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f1813fff640 1 -- 172.21.15.89:0/1047839254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f181c102d40 msgr2=0x7f181c1009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f1813fff640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f181c102d40 0x7f181c1009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f1813fff640 1 -- 172.21.15.89:0/1047839254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181c1a6870 con 0x7f181c103660 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f1813fff640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f181c103660 0x7f181c100ee0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f180400c970 tx=0x7f180400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f1818d99640 1 -- 172.21.15.89:0/1047839254 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804007bb0 con 0x7f181c103660 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f1818d99640 1 -- 172.21.15.89:0/1047839254 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1804007d50 con 0x7f181c103660 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f1818d99640 1 -- 172.21.15.89:0/1047839254 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804018820 con 0x7f181c103660 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f182116c640 1 -- 172.21.15.89:0/1047839254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f181c1a6b60 con 0x7f181c103660 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.523+0000 7f182116c640 1 -- 172.21.15.89:0/1047839254 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f181c1a70a0 con 0x7f181c103660 2024-09-15T23:30:36.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.526+0000 7f1818d99640 1 -- 172.21.15.89:0/1047839254 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f18040189c0 con 0x7f181c103660 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.526+0000 7f1823545740 1 Processor -- start 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.526+0000 7f1823545740 1 -- start start 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.526+0000 7f1823545740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be742090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.526+0000 7f1823545740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 0x5641be742a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.526+0000 7f1823545740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.526+0000 7f1823545740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5641be6a82f0 con 0x5641be74d370 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.526+0000 7f1823545740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5641be64fcf0 con 0x5641be743b40 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.526+0000 7f1823545740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5641be6a42b0 con 0x5641be7425d0 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f1813fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f1813fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39060/0 (socket says 172.21.15.89:39060) 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f1813fff640 1 -- 172.21.15.89:0/1984768653 learned_addr learned my addr 172.21.15.89:0/1984768653 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f181ad9d640 1 --2- 172.21.15.89:0/1984768653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 0x5641be742a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f181b59e640 1 --2- 172.21.15.89:0/1984768653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be742090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f1818d99640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18080a3240 0x7f18080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f181b59e640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18080a3240 0x7f18080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f1818d99640 1 -- 172.21.15.89:0/1047839254 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f18040cfe70 con 0x7f181c103660 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f181ad9d640 1 -- 172.21.15.89:0/1984768653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 msgr2=0x5641be742090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f181ad9d640 1 --2- 172.21.15.89:0/1984768653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be742090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f181ad9d640 1 -- 172.21.15.89:0/1984768653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 msgr2=0x5641be74f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f181ad9d640 1 --2- 172.21.15.89:0/1984768653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f181ad9d640 1 -- 172.21.15.89:0/1984768653 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641be6343a0 con 0x5641be7425d0 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f1813fff640 1 --2- 172.21.15.89:0/1984768653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f181ad9d640 1 --2- 172.21.15.89:0/1984768653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 0x5641be742a10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f180c009900 tx=0x7f180c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45b21fef4bd5d62c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.527+0000 7f181b59e640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18080a3240 0x7f18080a5700 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1814005c40 tx=0x7f1814005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.528+0000 7f18117fa640 1 -- 172.21.15.89:0/1984768653 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c040020 con 0x5641be7425d0 2024-09-15T23:30:36.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.528+0000 7f18117fa640 1 -- 172.21.15.89:0/1984768653 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f180c004040 con 0x5641be7425d0 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.528+0000 7f18117fa640 1 -- 172.21.15.89:0/1984768653 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c035040 con 0x5641be7425d0 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.528+0000 7f1823545740 1 -- 172.21.15.89:0/1984768653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 msgr2=0x5641be742a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.528+0000 7f1823545740 1 --2- 172.21.15.89:0/1984768653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 0x5641be742a10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f180c009900 tx=0x7f180c005d50 comp rx=0 tx=0).stop 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.528+0000 7f1823545740 1 -- 172.21.15.89:0/1984768653 shutdown_connections 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.528+0000 7f1823545740 1 --2- 172.21.15.89:0/1984768653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.528+0000 7f1823545740 1 --2- 172.21.15.89:0/1984768653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 0x5641be742a10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.528+0000 7f1823545740 1 --2- 172.21.15.89:0/1984768653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be742090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.528+0000 7f1823545740 1 -- 172.21.15.89:0/1984768653 >> 172.21.15.89:0/1984768653 conn(0x5641be70b310 msgr2=0x5641be73ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 1 -- 172.21.15.89:0/1984768653 shutdown_connections 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 1 -- 172.21.15.89:0/1984768653 wait complete. 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 1 -- start start 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 1 Processor -- start 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be7425d0 0x5641be74cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be743b40 0x5641be749f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5641be750010 con 0x5641be74d370 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5641be663060 con 0x5641be7425d0 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5641be750190 con 0x5641be743b40 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1823545740 10 client.0 _ll_register_callbacks cb 0x5641be3b68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f181b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f181b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39068/0 (socket says 172.21.15.89:39068) 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f181b59e640 1 -- 172.21.15.89:0/1524279422 learned_addr learned my addr 172.21.15.89:0/1524279422 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f1813fff640 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be743b40 0x5641be749f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.529+0000 7f181ad9d640 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be7425d0 0x5641be74cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f181b59e640 1 -- 172.21.15.89:0/1524279422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be743b40 msgr2=0x5641be749f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f181b59e640 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be743b40 0x5641be749f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f181b59e640 1 -- 172.21.15.89:0/1524279422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be7425d0 msgr2=0x5641be74cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:36.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f181b59e640 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be7425d0 0x5641be74cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f181b59e640 1 -- 172.21.15.89:0/1524279422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641be4193c0 con 0x5641be74d370 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f1813fff640 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be743b40 0x5641be749f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f181b59e640 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74a460 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f1814003dd0 tx=0x7f1814004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49f1faaa33733d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f18027fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f18027fc640 1 -- 172.21.15.89:0/1524279422 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181401c050 con 0x5641be74d370 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f18027fc640 1 -- 172.21.15.89:0/1524279422 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1814012930 con 0x5641be74d370 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f1823545740 1 -- 172.21.15.89:0/1524279422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f180c004040 con 0x5641be74d370 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f18027fc640 1 -- 172.21.15.89:0/1524279422 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f181402f050 con 0x5641be74d370 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f1823545740 10 client.17661 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f1823545740 1 -- 172.21.15.89:0/1524279422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5641be75b9f0 con 0x5641be74d370 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f18027fc640 1 -- 172.21.15.89:0/1524279422 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f180c004040 con 0x5641be74d370 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f18027fc640 1 -- 172.21.15.89:0/1524279422 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1814021c60 con 0x5641be74d370 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f1823545740 10 client.17661 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:36.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.530+0000 7f1823545740 10 client.17661 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.531+0000 7f1823545740 20 client.17661 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.531+0000 7f1823545740 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5641be75ceb0 0x5641be75f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.531+0000 7f1823545740 4 client.17661 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.531+0000 7f1823545740 1 -- 172.21.15.89:0/1524279422 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641be6b5860 con 0x5641be75ceb0 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.531+0000 7f181ad9d640 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5641be75ceb0 0x5641be75f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.531+0000 7f18027fc640 10 client.17661 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.531+0000 7f181ad9d640 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5641be75ceb0 0x5641be75f350 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2b639a04029b40f6 server_cookie=31a85701ff2fe2d0 in_seq=0 out_seq=0 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f18027fc640 1 -- 172.21.15.89:0/1524279422 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5641be6b5860 con 0x5641be75ceb0 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f18027fc640 10 client.17661 handle_command_reply: tid=0 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 client.17661 shutdown 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 -- 172.21.15.89:0/1524279422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 msgr2=0x5641be74a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74a460 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f1814003dd0 tx=0x7f1814004a40 comp rx=0 tx=0).stop 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 -- 172.21.15.89:0/1524279422 shutdown_connections 2024-09-15T23:30:36.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5641be75ceb0 0x5641be75f350 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be74a460 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be743b40 0x5641be749f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f18027fc640 0 client.17661 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:30:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 --2- 172.21.15.89:0/1524279422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be7425d0 0x5641be74cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 -- 172.21.15.89:0/1524279422 >> 172.21.15.89:0/1524279422 conn(0x5641be70b310 msgr2=0x5641be744960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 -- 172.21.15.89:0/1524279422 shutdown_connections 2024-09-15T23:30:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 1 -- 172.21.15.89:0/1524279422 wait complete. 2024-09-15T23:30:36.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.532+0000 7f1823545740 20 client.17661 trim_cache size 0 max 16384 2024-09-15T23:30:36.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.545+0000 7f1823545740 1 Processor -- start 2024-09-15T23:30:36.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.545+0000 7f1823545740 1 -- start start 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.545+0000 7f1823545740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be7425d0 0x5641be74be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.545+0000 7f1823545740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be745120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.545+0000 7f1823545740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be74d370 0x5641be745660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.545+0000 7f1823545740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f181c1b0f80 con 0x5641be7425d0 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.545+0000 7f1823545740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f181c1b2ac0 con 0x5641be743b40 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.545+0000 7f1823545740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f181c1b1650 con 0x5641be74d370 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.545+0000 7f181ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be7425d0 0x5641be74be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f181ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be7425d0 0x5641be74be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39070/0 (socket says 172.21.15.89:39070) 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f181ad9d640 1 -- 172.21.15.89:0/3143723821 learned_addr learned my addr 172.21.15.89:0/3143723821 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f181ad9d640 1 -- 172.21.15.89:0/3143723821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be74d370 msgr2=0x5641be745660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f1813fff640 1 --2- 172.21.15.89:0/3143723821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be745120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f181ad9d640 1 --2- 172.21.15.89:0/3143723821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be74d370 0x5641be745660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f181ad9d640 1 -- 172.21.15.89:0/3143723821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 msgr2=0x5641be745120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f181ad9d640 1 --2- 172.21.15.89:0/3143723821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be745120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f181ad9d640 1 -- 172.21.15.89:0/3143723821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641be74aa60 con 0x5641be7425d0 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f181ad9d640 1 --2- 172.21.15.89:0/3143723821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be7425d0 0x5641be74be90 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f180c009900 tx=0x7f180c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821f99c048d20284 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f1810ff9640 1 -- 172.21.15.89:0/3143723821 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c040040 con 0x5641be7425d0 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f1810ff9640 1 -- 172.21.15.89:0/3143723821 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f180c03e070 con 0x5641be7425d0 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.546+0000 7f1810ff9640 1 -- 172.21.15.89:0/3143723821 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c035040 con 0x5641be7425d0 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 -- 172.21.15.89:0/3143723821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be7425d0 msgr2=0x5641be74be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 --2- 172.21.15.89:0/3143723821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be7425d0 0x5641be74be90 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f180c009900 tx=0x7f180c005a20 comp rx=0 tx=0).stop 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 -- 172.21.15.89:0/3143723821 shutdown_connections 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 --2- 172.21.15.89:0/3143723821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be74d370 0x5641be745660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 --2- 172.21.15.89:0/3143723821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be745120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 --2- 172.21.15.89:0/3143723821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be7425d0 0x5641be74be90 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 -- 172.21.15.89:0/3143723821 >> 172.21.15.89:0/3143723821 conn(0x5641be70b310 msgr2=0x5641be74b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 -- 172.21.15.89:0/3143723821 shutdown_connections 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 -- 172.21.15.89:0/3143723821 wait complete. 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 -- start start 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.547+0000 7f1823545740 1 Processor -- start 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f1823545740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 0x5641be746f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f1823545740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be7474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f1823545740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be7479f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f1823545740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f181c1b23f0 con 0x5641be74d370 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f1823545740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f181c1b1d20 con 0x5641be743b40 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f1823545740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f181c1aef60 con 0x5641be7425d0 2024-09-15T23:30:36.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f1823545740 10 client.0 _ll_register_callbacks cb 0x7f1814003a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f181b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be7479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f181b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be7479f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39074/0 (socket says 172.21.15.89:39074) 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f181b59e640 1 -- 172.21.15.89:0/1222392380 learned_addr learned my addr 172.21.15.89:0/1222392380 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f181ad9d640 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 0x5641be746f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f181b59e640 1 -- 172.21.15.89:0/1222392380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 msgr2=0x5641be746f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f181b59e640 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 0x5641be746f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f181b59e640 1 -- 172.21.15.89:0/1222392380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 msgr2=0x5641be7474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f181b59e640 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be7474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f181b59e640 1 -- 172.21.15.89:0/1222392380 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641be7c6c90 con 0x5641be74d370 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f181b59e640 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be7479f0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f1814000f80 tx=0x7f1814015660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c0ab7d3974253c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:36.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f18027fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f18027fc640 1 -- 172.21.15.89:0/1222392380 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1814037020 con 0x5641be74d370 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f1823545740 1 -- 172.21.15.89:0/1222392380 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f180c03e070 con 0x5641be74d370 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f18027fc640 1 -- 172.21.15.89:0/1222392380 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1814015230 con 0x5641be74d370 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.548+0000 7f18027fc640 1 -- 172.21.15.89:0/1222392380 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1814042050 con 0x5641be74d370 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f1823545740 10 client.17667 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f1823545740 1 -- 172.21.15.89:0/1222392380 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5641be7c7f70 con 0x5641be74d370 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f18027fc640 1 -- 172.21.15.89:0/1222392380 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f180c03e070 con 0x5641be74d370 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f18027fc640 1 -- 172.21.15.89:0/1222392380 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f181403e4c0 con 0x5641be74d370 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f1823545740 10 client.17667 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f1823545740 10 client.17667 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f1823545740 20 client.17667 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f1823545740 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5641be7c8680 0x5641be7caa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:36.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f1823545740 4 client.17667 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f1823545740 1 -- 172.21.15.89:0/1222392380 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641be6afd70 con 0x5641be7c8680 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.549+0000 7f181ad9d640 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5641be7c8680 0x5641be7caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f181ad9d640 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5641be7c8680 0x5641be7caa70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4cb5aa1b22905216 server_cookie=6ad24884fbe6bb43 in_seq=0 out_seq=0 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f18027fc640 10 client.17667 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f18027fc640 1 -- 172.21.15.89:0/1222392380 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641be6afd70 con 0x5641be7c8680 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f18027fc640 10 client.17667 handle_command_reply: tid=0 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 client.17667 shutdown 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 -- 172.21.15.89:0/1222392380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 msgr2=0x5641be7479f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be7479f0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f1814000f80 tx=0x7f1814015660 comp rx=0 tx=0).stop 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 -- 172.21.15.89:0/1222392380 shutdown_connections 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5641be7c8680 0x5641be7caa70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641be74d370 0x5641be7479f0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641be743b40 0x5641be7474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 --2- 172.21.15.89:0/1222392380 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5641be7425d0 0x5641be746f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 -- 172.21.15.89:0/1222392380 >> 172.21.15.89:0/1222392380 conn(0x5641be70b310 msgr2=0x5641be74bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:36.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f18027fc640 0 client.17667 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:30:36.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 -- 172.21.15.89:0/1222392380 shutdown_connections 2024-09-15T23:30:36.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 1 -- 172.21.15.89:0/1222392380 wait complete. 2024-09-15T23:30:36.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.550+0000 7f1823545740 20 client.17667 trim_cache size 0 max 16384 2024-09-15T23:30:36.564 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:30:36.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 -- 172.21.15.89:0/1047839254 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18080a3240 msgr2=0x7f18080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:36.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18080a3240 0x7f18080a5700 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1814005c40 tx=0x7f1814005bd0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 -- 172.21.15.89:0/1047839254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f181c103660 msgr2=0x7f181c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:36.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f181c103660 0x7f181c100ee0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f180400c970 tx=0x7f180400ce30 comp rx=0 tx=0).stop 2024-09-15T23:30:36.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 -- 172.21.15.89:0/1047839254 shutdown_connections 2024-09-15T23:30:36.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18080a3240 0x7f18080a5700 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f181c10c170 0x7f181c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f181c103660 0x7f181c100ee0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 --2- 172.21.15.89:0/1047839254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f181c102d40 0x7f181c1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:36.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 -- 172.21.15.89:0/1047839254 >> 172.21.15.89:0/1047839254 conn(0x7f181c0fc7a0 msgr2=0x7f181c0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:36.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 -- 172.21.15.89:0/1047839254 shutdown_connections 2024-09-15T23:30:36.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:36.551+0000 7f182116c640 1 -- 172.21.15.89:0/1047839254 wait complete. 2024-09-15T23:30:37.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:30:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:37 smithi120 ceph-mon[35250]: pgmap v548: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:30:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:37 smithi174 ceph-mon[35586]: pgmap v548: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:30:37.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:37 smithi089 ceph-mon[30933]: pgmap v548: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:30:37.661 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.160+0000 7feaeb278640 1 -- 172.21.15.89:0/1223746732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae4100a80 msgr2=0x7feae4100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.160+0000 7feaeb278640 1 --2- 172.21.15.89:0/1223746732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae4100a80 0x7feae4100ee0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fead4009930 tx=0x7fead402a420 comp rx=0 tx=0).stop 2024-09-15T23:30:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.160+0000 7feaeb278640 1 -- 172.21.15.89:0/1223746732 shutdown_connections 2024-09-15T23:30:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.160+0000 7feaeb278640 1 --2- 172.21.15.89:0/1223746732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae4101420 0x7feae410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.160+0000 7feaeb278640 1 --2- 172.21.15.89:0/1223746732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae4100a80 0x7feae4100ee0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.160+0000 7feaeb278640 1 --2- 172.21.15.89:0/1223746732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae4106a80 0x7feae4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.160+0000 7feaeb278640 1 -- 172.21.15.89:0/1223746732 >> 172.21.15.89:0/1223746732 conn(0x7feae40fc7a0 msgr2=0x7feae40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.160+0000 7feaeb278640 1 -- 172.21.15.89:0/1223746732 shutdown_connections 2024-09-15T23:30:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.160+0000 7feaeb278640 1 -- 172.21.15.89:0/1223746732 wait complete. 2024-09-15T23:30:38.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.160+0000 7feaeb278640 1 Processor -- start 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feaeb278640 1 -- start start 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feaeb278640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae4100a80 0x7feae41973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feaeb278640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae4101420 0x7feae4197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feaeb278640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae4106a80 0x7feae419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feaeb278640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feae4110ec0 con 0x7feae4101420 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feaeb278640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feae4110d40 con 0x7feae4100a80 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feaeb278640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feae4111040 con 0x7feae4106a80 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae8813640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae4101420 0x7feae4197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae8813640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae4101420 0x7feae4197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39104/0 (socket says 172.21.15.89:39104) 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae8813640 1 -- 172.21.15.89:0/4069717298 learned_addr learned my addr 172.21.15.89:0/4069717298 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae9014640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae4100a80 0x7feae41973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae9815640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae4106a80 0x7feae419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae8813640 1 -- 172.21.15.89:0/4069717298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae4106a80 msgr2=0x7feae419bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae8813640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae4106a80 0x7feae419bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae8813640 1 -- 172.21.15.89:0/4069717298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae4100a80 msgr2=0x7feae41973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae8813640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae4100a80 0x7feae41973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae8813640 1 -- 172.21.15.89:0/4069717298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae419c390 con 0x7feae4101420 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae9014640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae4100a80 0x7feae41973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.161+0000 7feae8813640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae4101420 0x7feae4197920 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fead4009a50 tx=0x7fead4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.162+0000 7feada7fc640 1 -- 172.21.15.89:0/4069717298 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead403f030 con 0x7feae4101420 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.162+0000 7feada7fc640 1 -- 172.21.15.89:0/4069717298 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead4003ce0 con 0x7feae4101420 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.162+0000 7feaeb278640 1 -- 172.21.15.89:0/4069717298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae419c620 con 0x7feae4101420 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.162+0000 7feada7fc640 1 -- 172.21.15.89:0/4069717298 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead4035070 con 0x7feae4101420 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.162+0000 7feaeb278640 1 -- 172.21.15.89:0/4069717298 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae41a3f60 con 0x7feae4101420 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.164+0000 7feada7fc640 1 -- 172.21.15.89:0/4069717298 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fead4042030 con 0x7feae4101420 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feaed651740 1 Processor -- start 2024-09-15T23:30:38.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feaed651740 1 -- start start 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feada7fc640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a3240 0x7fead00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feaed651740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f03090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feaed651740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f035d0 0x559d10f03a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feae9815640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a3240 0x7fead00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feaed651740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f10580 0x559d10f12970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feaed651740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559d10e692f0 con 0x559d10f04b40 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feaed651740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559d10e10cf0 con 0x559d10f035d0 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feaed651740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559d10e652b0 con 0x559d10f10580 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feada7fc640 1 -- 172.21.15.89:0/4069717298 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fead40f0370 con 0x7feae4101420 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feae9014640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f03090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feae9014640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f03090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39110/0 (socket says 172.21.15.89:39110) 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feae9014640 1 -- 172.21.15.89:0/771131921 learned_addr learned my addr 172.21.15.89:0/771131921 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:38.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feae8813640 1 --2- 172.21.15.89:0/771131921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f035d0 0x559d10f03a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feae9815640 1 --2- 172.21.15.89:0/771131921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f10580 0x559d10f12970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.165+0000 7feae9815640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a3240 0x7fead00a5700 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7feadc005fd0 tx=0x7feadc00d710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.166+0000 7feae9014640 1 -- 172.21.15.89:0/771131921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f10580 msgr2=0x559d10f12970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.166+0000 7feae9014640 1 --2- 172.21.15.89:0/771131921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f10580 0x559d10f12970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.166+0000 7feae9014640 1 -- 172.21.15.89:0/771131921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f035d0 msgr2=0x559d10f03a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.166+0000 7feae9014640 1 --2- 172.21.15.89:0/771131921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f035d0 0x559d10f03a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.166+0000 7feae9014640 1 -- 172.21.15.89:0/771131921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d10df53a0 con 0x559d10f04b40 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.166+0000 7feae8813640 1 --2- 172.21.15.89:0/771131921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f035d0 0x559d10f03a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.166+0000 7feae9014640 1 --2- 172.21.15.89:0/771131921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f03090 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7feae0007850 tx=0x7feae0007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bdc8ed50ad9ff74 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.166+0000 7feacf7fe640 1 -- 172.21.15.89:0/771131921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae000d950 con 0x559d10f04b40 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.166+0000 7feacf7fe640 1 -- 172.21.15.89:0/771131921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feae0018070 con 0x559d10f04b40 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.166+0000 7feacf7fe640 1 -- 172.21.15.89:0/771131921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae00123e0 con 0x559d10f04b40 2024-09-15T23:30:38.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 -- 172.21.15.89:0/771131921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 msgr2=0x559d10f03090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 --2- 172.21.15.89:0/771131921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f03090 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7feae0007850 tx=0x7feae0007d10 comp rx=0 tx=0).stop 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 -- 172.21.15.89:0/771131921 shutdown_connections 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 --2- 172.21.15.89:0/771131921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f10580 0x559d10f12970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 --2- 172.21.15.89:0/771131921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f035d0 0x559d10f03a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 --2- 172.21.15.89:0/771131921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f03090 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 -- 172.21.15.89:0/771131921 >> 172.21.15.89:0/771131921 conn(0x559d10ecc310 msgr2=0x559d10effc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 -- 172.21.15.89:0/771131921 shutdown_connections 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 -- 172.21.15.89:0/771131921 wait complete. 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 -- start start 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.167+0000 7feaed651740 1 Processor -- start 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaed651740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f0fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaed651740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f04b40 0x559d10f0d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaed651740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f10580 0x559d10f0d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaed651740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559d10f13280 con 0x559d10f10580 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaed651740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559d10e24060 con 0x559d10f04b40 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaed651740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559d10f13400 con 0x559d10f035d0 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaed651740 10 client.0 _ll_register_callbacks cb 0x559d10b778c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae9815640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f10580 0x559d10f0d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae9815640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f10580 0x559d10f0d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39118/0 (socket says 172.21.15.89:39118) 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae9815640 1 -- 172.21.15.89:0/2289274889 learned_addr learned my addr 172.21.15.89:0/2289274889 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae9014640 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f0fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae9815640 1 -- 172.21.15.89:0/2289274889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 msgr2=0x559d10f0fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:38.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae9815640 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f0fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae9815640 1 -- 172.21.15.89:0/2289274889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f04b40 msgr2=0x559d10f0d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae8813640 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f04b40 0x559d10f0d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae9815640 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f04b40 0x559d10f0d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae9815640 1 -- 172.21.15.89:0/2289274889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d10bda3c0 con 0x559d10f10580 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feae9815640 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f10580 0x559d10f0d690 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7feadc011de0 tx=0x7feadc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8465c7acda9b50b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaccff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaccff9640 1 -- 172.21.15.89:0/2289274889 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc021070 con 0x559d10f10580 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaccff9640 1 -- 172.21.15.89:0/2289274889 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7feadc01d410 con 0x559d10f10580 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.168+0000 7feaed651740 1 -- 172.21.15.89:0/2289274889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feae0018070 con 0x559d10f10580 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaccff9640 1 -- 172.21.15.89:0/2289274889 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feadc008e50 con 0x559d10f10580 2024-09-15T23:30:38.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaed651740 10 client.17679 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaed651740 1 -- 172.21.15.89:0/2289274889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559d10f1ecf0 con 0x559d10f10580 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaccff9640 1 -- 172.21.15.89:0/2289274889 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feae0018070 con 0x559d10f10580 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaccff9640 1 -- 172.21.15.89:0/2289274889 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7feadc01d940 con 0x559d10f10580 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaed651740 10 client.17679 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaed651740 10 client.17679 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaed651740 20 client.17679 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaed651740 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x559d10f201b0 0x559d10f22600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaed651740 4 client.17679 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feaed651740 1 -- 172.21.15.89:0/2289274889 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d10e76860 con 0x559d10f201b0 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.169+0000 7feae9014640 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x559d10f201b0 0x559d10f22600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.170+0000 7feaccff9640 10 client.17679 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.170+0000 7feae9014640 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x559d10f201b0 0x559d10f22600 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f91e59af0aae4b41 server_cookie=ca01bce9c82f77b1 in_seq=0 out_seq=0 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.170+0000 7feaccff9640 1 -- 172.21.15.89:0/2289274889 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559d10e76860 con 0x559d10f201b0 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.170+0000 7feaccff9640 10 client.17679 handle_command_reply: tid=0 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.170+0000 7feaed651740 1 client.17679 shutdown 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 1 -- 172.21.15.89:0/2289274889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f10580 msgr2=0x559d10f0d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f10580 0x559d10f0d690 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7feadc011de0 tx=0x7feadc004ab0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 1 -- 172.21.15.89:0/2289274889 shutdown_connections 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x559d10f201b0 0x559d10f22600 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f10580 0x559d10f0d690 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f04b40 0x559d10f0d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 1 --2- 172.21.15.89:0/2289274889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f0fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaccff9640 0 client.17679 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 1 -- 172.21.15.89:0/2289274889 >> 172.21.15.89:0/2289274889 conn(0x559d10ecc310 msgr2=0x559d10f05960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:38.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 1 -- 172.21.15.89:0/2289274889 shutdown_connections 2024-09-15T23:30:38.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 1 -- 172.21.15.89:0/2289274889 wait complete. 2024-09-15T23:30:38.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.171+0000 7feaed651740 20 client.17679 trim_cache size 0 max 16384 2024-09-15T23:30:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feaed651740 1 Processor -- start 2024-09-15T23:30:38.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feaed651740 1 -- start start 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feaed651740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f1c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feaed651740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f27ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feaed651740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 0x559d10f28410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feaed651740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feae41acc30 con 0x559d10f04b40 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feaed651740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feae41ae770 con 0x559d10f10580 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feaed651740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feae41ad300 con 0x559d10f035d0 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feae8813640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f27ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feae8813640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f27ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39126/0 (socket says 172.21.15.89:39126) 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feae8813640 1 -- 172.21.15.89:0/1684025296 learned_addr learned my addr 172.21.15.89:0/1684025296 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feae9815640 1 --2- 172.21.15.89:0/1684025296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 0x559d10f28410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feae9014640 1 --2- 172.21.15.89:0/1684025296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f1c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.197 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feae8813640 1 -- 172.21.15.89:0/1684025296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 msgr2=0x559d10f1c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feae8813640 1 --2- 172.21.15.89:0/1684025296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f1c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feae8813640 1 -- 172.21.15.89:0/1684025296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 msgr2=0x559d10f28410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.184+0000 7feae8813640 1 --2- 172.21.15.89:0/1684025296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 0x559d10f28410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.185+0000 7feae8813640 1 -- 172.21.15.89:0/1684025296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d10f00960 con 0x559d10f04b40 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.185+0000 7feae8813640 1 --2- 172.21.15.89:0/1684025296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f27ed0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fead40bc2a0 tx=0x7fead403b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac862c1c8762f34 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.185+0000 7feaceffd640 1 -- 172.21.15.89:0/1684025296 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead4049040 con 0x559d10f04b40 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.185+0000 7feaceffd640 1 -- 172.21.15.89:0/1684025296 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead40bcca0 con 0x559d10f04b40 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.185+0000 7feaceffd640 1 -- 172.21.15.89:0/1684025296 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead40ba640 con 0x559d10f04b40 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 -- 172.21.15.89:0/1684025296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 msgr2=0x559d10f27ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 --2- 172.21.15.89:0/1684025296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f27ed0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fead40bc2a0 tx=0x7fead403b800 comp rx=0 tx=0).stop 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 -- 172.21.15.89:0/1684025296 shutdown_connections 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 --2- 172.21.15.89:0/1684025296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 0x559d10f28410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 --2- 172.21.15.89:0/1684025296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f27ed0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 --2- 172.21.15.89:0/1684025296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f1c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 -- 172.21.15.89:0/1684025296 >> 172.21.15.89:0/1684025296 conn(0x559d10ecc310 msgr2=0x559d10f0d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 -- 172.21.15.89:0/1684025296 shutdown_connections 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 -- 172.21.15.89:0/1684025296 wait complete. 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 -- start start 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 Processor -- start 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f186a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f18be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 0x559d10f13650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feae41ae0a0 con 0x559d10f04b40 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feae41ad9d0 con 0x559d10f10580 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.186+0000 7feaed651740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feae41aac10 con 0x559d10f035d0 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feaed651740 10 client.0 _ll_register_callbacks cb 0x7feadc015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae8813640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f18be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae8813640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f18be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39130/0 (socket says 172.21.15.89:39130) 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae8813640 1 -- 172.21.15.89:0/1220239523 learned_addr learned my addr 172.21.15.89:0/1220239523 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae9014640 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f186a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae9815640 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 0x559d10f13650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae8813640 1 -- 172.21.15.89:0/1220239523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 msgr2=0x559d10f186a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae8813640 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f186a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae8813640 1 -- 172.21.15.89:0/1220239523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 msgr2=0x559d10f13650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae8813640 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 0x559d10f13650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae8813640 1 -- 172.21.15.89:0/1220239523 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d10f13b90 con 0x559d10f04b40 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae9815640 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 0x559d10f13650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feae8813640 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f18be0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fead40b8a00 tx=0x7fead40bc6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4cdb23b65ac6c4a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feaccff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feaccff9640 1 -- 172.21.15.89:0/1220239523 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead40d9070 con 0x559d10f04b40 2024-09-15T23:30:38.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feaccff9640 1 -- 172.21.15.89:0/1220239523 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fead40ba8a0 con 0x559d10f04b40 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feaed651740 1 -- 172.21.15.89:0/1220239523 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fead40bcca0 con 0x559d10f04b40 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feaccff9640 1 -- 172.21.15.89:0/1220239523 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fead4049030 con 0x559d10f04b40 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feaed651740 10 client.17685 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.187+0000 7feaed651740 1 -- 172.21.15.89:0/1220239523 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559d10f8d980 con 0x559d10f04b40 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feaccff9640 1 -- 172.21.15.89:0/1220239523 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fead40bcca0 con 0x559d10f04b40 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feaccff9640 1 -- 172.21.15.89:0/1220239523 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fead403b460 con 0x559d10f04b40 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feaed651740 10 client.17685 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feaed651740 10 client.17685 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feaed651740 20 client.17685 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feaed651740 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x559d10f8e080 0x559d10f904a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feaed651740 4 client.17685 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feaed651740 1 -- 172.21.15.89:0/1220239523 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559d10e70d70 con 0x559d10f8e080 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feae9014640 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x559d10f8e080 0x559d10f904a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:38.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feae9014640 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x559d10f8e080 0x559d10f904a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=46e0c4da81b605f6 server_cookie=3d8a1121f0ae1c40 in_seq=0 out_seq=0 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.188+0000 7feaccff9640 10 client.17685 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaccff9640 1 -- 172.21.15.89:0/1220239523 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559d10e70d70 con 0x559d10f8e080 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaccff9640 10 client.17685 handle_command_reply: tid=0 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 client.17685 shutdown 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 -- 172.21.15.89:0/1220239523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 msgr2=0x559d10f18be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f18be0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fead40b8a00 tx=0x7fead40bc6e0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 -- 172.21.15.89:0/1220239523 shutdown_connections 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x559d10f8e080 0x559d10f904a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d10f10580 0x559d10f13650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d10f04b40 0x559d10f18be0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaccff9640 0 client.17685 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 --2- 172.21.15.89:0/1220239523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d10f035d0 0x559d10f186a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 -- 172.21.15.89:0/1220239523 >> 172.21.15.89:0/1220239523 conn(0x559d10ecc310 msgr2=0x559d10f0e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:38.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 -- 172.21.15.89:0/1220239523 shutdown_connections 2024-09-15T23:30:38.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 1 -- 172.21.15.89:0/1220239523 wait complete. 2024-09-15T23:30:38.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.189+0000 7feaed651740 20 client.17685 trim_cache size 0 max 16384 2024-09-15T23:30:38.203 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 -- 172.21.15.89:0/4069717298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a3240 msgr2=0x7fead00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a3240 0x7fead00a5700 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7feadc005fd0 tx=0x7feadc00d710 comp rx=0 tx=0).stop 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 -- 172.21.15.89:0/4069717298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae4101420 msgr2=0x7feae4197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae4101420 0x7feae4197920 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fead4009a50 tx=0x7fead4004270 comp rx=0 tx=0).stop 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 -- 172.21.15.89:0/4069717298 shutdown_connections 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fead00a3240 0x7fead00a5700 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feae4106a80 0x7feae419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feae4101420 0x7feae4197920 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 --2- 172.21.15.89:0/4069717298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae4100a80 0x7feae41973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 -- 172.21.15.89:0/4069717298 >> 172.21.15.89:0/4069717298 conn(0x7feae40fc7a0 msgr2=0x7feae40fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 -- 172.21.15.89:0/4069717298 shutdown_connections 2024-09-15T23:30:38.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:38.190+0000 7feaeb278640 1 -- 172.21.15.89:0/4069717298 wait complete. 2024-09-15T23:30:39.000 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:30:39.250 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:39 smithi120 ceph-mon[35250]: pgmap v549: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:30:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:39 smithi174 ceph-mon[35586]: pgmap v549: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:30:39.502 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:39 smithi089 ceph-mon[30933]: pgmap v549: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:30:39.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.723+0000 7f6a05bcf640 1 -- 172.21.15.89:0/1864966993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a00106a80 msgr2=0x7f6a00106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:39.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.723+0000 7f6a05bcf640 1 --2- 172.21.15.89:0/1864966993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a00106a80 0x7f6a00106e60 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009930 tx=0x7f69f002a420 comp rx=0 tx=0).stop 2024-09-15T23:30:39.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.723+0000 7f6a05bcf640 1 -- 172.21.15.89:0/1864966993 shutdown_connections 2024-09-15T23:30:39.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.723+0000 7f6a05bcf640 1 --2- 172.21.15.89:0/1864966993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a00101420 0x7f6a0010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.723+0000 7f6a05bcf640 1 --2- 172.21.15.89:0/1864966993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a00100a80 0x7f6a00100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.723+0000 7f6a05bcf640 1 --2- 172.21.15.89:0/1864966993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a00106a80 0x7f6a00106e60 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.723+0000 7f6a05bcf640 1 -- 172.21.15.89:0/1864966993 >> 172.21.15.89:0/1864966993 conn(0x7f6a000fc7a0 msgr2=0x7f6a000febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.723+0000 7f6a05bcf640 1 -- 172.21.15.89:0/1864966993 shutdown_connections 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.723+0000 7f6a05bcf640 1 -- 172.21.15.89:0/1864966993 wait complete. 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f6a05bcf640 1 Processor -- start 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f6a05bcf640 1 -- start start 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f6a05bcf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a00100a80 0x7f6a00199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f6a05bcf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a00101420 0x7f6a00199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f6a05bcf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a00106a80 0x7f6a001a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f6a05bcf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a00110f20 con 0x7f6a00100a80 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f6a05bcf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6a00110da0 con 0x7f6a00101420 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f6a05bcf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a001110a0 con 0x7f6a00106a80 2024-09-15T23:30:39.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f69ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a00100a80 0x7f6a00199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f69ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a00100a80 0x7f6a00199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39152/0 (socket says 172.21.15.89:39152) 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f69ff7fe640 1 -- 172.21.15.89:0/2303850647 learned_addr learned my addr 172.21.15.89:0/2303850647 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f69feffd640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a00101420 0x7f6a00199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.724+0000 7f69ff7fe640 1 -- 172.21.15.89:0/2303850647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a00106a80 msgr2=0x7f6a001a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a00106a80 0x7f6a001a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f69ff7fe640 1 -- 172.21.15.89:0/2303850647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a00101420 msgr2=0x7f6a00199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a00101420 0x7f6a00199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f69ff7fe640 1 -- 172.21.15.89:0/2303850647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a001a4a10 con 0x7f6a00100a80 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f69feffd640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a00101420 0x7f6a00199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a00100a80 0x7f6a00199450 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009a50 tx=0x7f69f0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f69fcff9640 1 -- 172.21.15.89:0/2303850647 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f003f030 con 0x7f6a00100a80 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f69fcff9640 1 -- 172.21.15.89:0/2303850647 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69f0003ce0 con 0x7f6a00100a80 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f6a05bcf640 1 -- 172.21.15.89:0/2303850647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a001a4be0 con 0x7f6a00100a80 2024-09-15T23:30:39.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f69fcff9640 1 -- 172.21.15.89:0/2303850647 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f0035070 con 0x7f6a00100a80 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.725+0000 7f6a05bcf640 1 -- 172.21.15.89:0/2303850647 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a001a5030 con 0x7f6a00100a80 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.727+0000 7f69fcff9640 1 -- 172.21.15.89:0/2303850647 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f69f0042030 con 0x7f6a00100a80 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.727+0000 7f6a07fa8740 1 Processor -- start 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.727+0000 7f6a07fa8740 1 -- start start 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2bbb40 0x55b13d2ba090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2ba5d0 0x55b13d2baa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fcff9640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69e80a3240 0x7f69e80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 0x55b13d2c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b13d2202f0 con 0x55b13d2bbb40 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b13d1c7cf0 con 0x55b13d2c5370 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b13d21c2b0 con 0x55b13d2ba5d0 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2bbb40 0x55b13d2ba090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2bbb40 0x55b13d2ba090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39156/0 (socket says 172.21.15.89:39156) 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fffff640 1 -- 172.21.15.89:0/1126656996 learned_addr learned my addr 172.21.15.89:0/1126656996 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69feffd640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69e80a3240 0x7f69e80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/1126656996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2ba5d0 0x55b13d2baa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fcff9640 1 -- 172.21.15.89:0/2303850647 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f69f00f0340 con 0x7f6a00100a80 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fffff640 1 --2- 172.21.15.89:0/1126656996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 0x55b13d2c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fffff640 1 -- 172.21.15.89:0/1126656996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2ba5d0 msgr2=0x55b13d2baa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fffff640 1 --2- 172.21.15.89:0/1126656996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2ba5d0 0x55b13d2baa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fffff640 1 -- 172.21.15.89:0/1126656996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 msgr2=0x55b13d2c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fffff640 1 --2- 172.21.15.89:0/1126656996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 0x55b13d2c7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fffff640 1 -- 172.21.15.89:0/1126656996 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b13d1ac3a0 con 0x55b13d2bbb40 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69feffd640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69e80a3240 0x7f69e80a5700 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6a0006b2e0 tx=0x7f69f4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69fffff640 1 --2- 172.21.15.89:0/1126656996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2bbb40 0x55b13d2ba090 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f69f8009fb0 tx=0x7f69f8009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db815cc0a5eb539 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.728+0000 7f69e5ffb640 1 -- 172.21.15.89:0/1126656996 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f8003b70 con 0x55b13d2bbb40 2024-09-15T23:30:39.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f69e5ffb640 1 -- 172.21.15.89:0/1126656996 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69f8003d10 con 0x55b13d2bbb40 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f69e5ffb640 1 -- 172.21.15.89:0/1126656996 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f8015780 con 0x55b13d2bbb40 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f6a07fa8740 1 -- 172.21.15.89:0/1126656996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2bbb40 msgr2=0x55b13d2ba090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/1126656996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2bbb40 0x55b13d2ba090 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f69f8009fb0 tx=0x7f69f8009760 comp rx=0 tx=0).stop 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f6a07fa8740 1 -- 172.21.15.89:0/1126656996 shutdown_connections 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/1126656996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 0x55b13d2c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/1126656996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2ba5d0 0x55b13d2baa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/1126656996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2bbb40 0x55b13d2ba090 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f6a07fa8740 1 -- 172.21.15.89:0/1126656996 >> 172.21.15.89:0/1126656996 conn(0x55b13d283310 msgr2=0x55b13d2b6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f6a07fa8740 1 -- 172.21.15.89:0/1126656996 shutdown_connections 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.729+0000 7f6a07fa8740 1 -- 172.21.15.89:0/1126656996 wait complete. 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f6a07fa8740 1 -- start start 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f6a07fa8740 1 Processor -- start 2024-09-15T23:30:39.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2c1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d2c2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b13d2c8070 con 0x55b13d2c5370 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b13d1db060 con 0x55b13d2ba5d0 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b13d2c81f0 con 0x55b13d2bbb40 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f6a07fa8740 10 client.0 _ll_register_callbacks cb 0x55b13cf2e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f69fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d2c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f69fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d2c2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39162/0 (socket says 172.21.15.89:39162) 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f69fffff640 1 -- 172.21.15.89:0/2001962276 learned_addr learned my addr 172.21.15.89:0/2001962276 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f69feffd640 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2c1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f69fffff640 1 -- 172.21.15.89:0/2001962276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 msgr2=0x55b13d2c1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.730+0000 7f69fffff640 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2c1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69fffff640 1 -- 172.21.15.89:0/2001962276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2ba5d0 msgr2=0x55b13d2c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69fffff640 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69fffff640 1 -- 172.21.15.89:0/2001962276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b13cf913c0 con 0x55b13d2c5370 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69feffd640 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69fffff640 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d2c2530 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f69f80040f0 tx=0x7f69f8004120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b69a6dcdcff93b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69deffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69deffd640 1 -- 172.21.15.89:0/2001962276 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f8024020 con 0x55b13d2c5370 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69deffd640 1 -- 172.21.15.89:0/2001962276 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f69f8015560 con 0x55b13d2c5370 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f6a07fa8740 1 -- 172.21.15.89:0/2001962276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69f8003d10 con 0x55b13d2c5370 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69deffd640 1 -- 172.21.15.89:0/2001962276 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f69f802f050 con 0x55b13d2c5370 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f6a07fa8740 10 client.17697 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f6a07fa8740 1 -- 172.21.15.89:0/2001962276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b13d2d3aa0 con 0x55b13d2c5370 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69deffd640 1 -- 172.21.15.89:0/2001962276 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69f8003d10 con 0x55b13d2c5370 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f69deffd640 1 -- 172.21.15.89:0/2001962276 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f69f8020500 con 0x55b13d2c5370 2024-09-15T23:30:39.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f6a07fa8740 10 client.17697 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:39.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f6a07fa8740 10 client.17697 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:30:39.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f6a07fa8740 20 client.17697 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:39.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b13d2d4f60 0x55b13d2d7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f6a07fa8740 4 client.17697 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:39.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.731+0000 7f6a07fa8740 1 -- 172.21.15.89:0/2001962276 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b13d22d860 con 0x55b13d2d4f60 2024-09-15T23:30:39.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.732+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b13d2d4f60 0x55b13d2d7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.732+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b13d2d4f60 0x55b13d2d7350 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6032eddbde71bede server_cookie=b612b161b22f3448 in_seq=0 out_seq=0 2024-09-15T23:30:39.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.732+0000 7f69deffd640 10 client.17697 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:30:39.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f69deffd640 1 -- 172.21.15.89:0/2001962276 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b13d22d860 con 0x55b13d2d4f60 2024-09-15T23:30:39.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f69deffd640 10 client.17697 handle_command_reply: tid=0 2024-09-15T23:30:39.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 client.17697 shutdown 2024-09-15T23:30:39.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 -- 172.21.15.89:0/2001962276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 msgr2=0x55b13d2c2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:39.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d2c2530 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f69f80040f0 tx=0x7f69f8004120 comp rx=0 tx=0).stop 2024-09-15T23:30:39.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 -- 172.21.15.89:0/2001962276 shutdown_connections 2024-09-15T23:30:39.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d2c2530 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b13d2d4f60 0x55b13d2d7350 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f69deffd640 0 client.17697 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:30:39.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/2001962276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 -- 172.21.15.89:0/2001962276 >> 172.21.15.89:0/2001962276 conn(0x55b13d283310 msgr2=0x55b13d2bc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:39.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 -- 172.21.15.89:0/2001962276 shutdown_connections 2024-09-15T23:30:39.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.733+0000 7f6a07fa8740 1 -- 172.21.15.89:0/2001962276 wait complete. 2024-09-15T23:30:39.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.734+0000 7f6a07fa8740 20 client.17697 trim_cache size 0 max 16384 2024-09-15T23:30:39.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.746+0000 7f6a07fa8740 1 Processor -- start 2024-09-15T23:30:39.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.746+0000 7f6a07fa8740 1 -- start start 2024-09-15T23:30:39.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.746+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.746+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2bd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.746+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 0x55b13d2bd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.746+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a001aef10 con 0x55b13d2ba5d0 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.746+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6a001b0a50 con 0x55b13d2c5370 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.746+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a001af5e0 con 0x55b13d2bbb40 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.746+0000 7f69ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39174/0 (socket says 172.21.15.89:39174) 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69ff7fe640 1 -- 172.21.15.89:0/3683097821 learned_addr learned my addr 172.21.15.89:0/3683097821 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69fffff640 1 --2- 172.21.15.89:0/3683097821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 0x55b13d2bd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69ff7fe640 1 -- 172.21.15.89:0/3683097821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 msgr2=0x55b13d2bd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/3683097821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2bd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69ff7fe640 1 -- 172.21.15.89:0/3683097821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 msgr2=0x55b13d2bd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/3683097821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 0x55b13d2bd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69ff7fe640 1 -- 172.21.15.89:0/3683097821 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b13d2b5310 con 0x55b13d2ba5d0 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69fffff640 1 --2- 172.21.15.89:0/3683097821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 0x55b13d2bd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:39.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/3683097821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c3f20 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f69f00b8250 tx=0x7f69f003b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c92ab4e501536c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:39.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69e57fa640 1 -- 172.21.15.89:0/3683097821 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f0049040 con 0x55b13d2ba5d0 2024-09-15T23:30:39.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69e57fa640 1 -- 172.21.15.89:0/3683097821 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69f00bcc70 con 0x55b13d2ba5d0 2024-09-15T23:30:39.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.747+0000 7f69e57fa640 1 -- 172.21.15.89:0/3683097821 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f00f00c0 con 0x55b13d2ba5d0 2024-09-15T23:30:39.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3683097821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2ba5d0 msgr2=0x55b13d2c3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:39.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/3683097821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c3f20 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f69f00b8250 tx=0x7f69f003b800 comp rx=0 tx=0).stop 2024-09-15T23:30:39.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3683097821 shutdown_connections 2024-09-15T23:30:39.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/3683097821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2c5370 0x55b13d2bd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/3683097821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2bd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/3683097821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c3f20 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3683097821 >> 172.21.15.89:0/3683097821 conn(0x55b13d283310 msgr2=0x55b13d2c3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3683097821 shutdown_connections 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3683097821 wait complete. 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 -- start start 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.748+0000 7f6a07fa8740 1 Processor -- start 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f6a07fa8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d33e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a001b0380 con 0x55b13d2c5370 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6a001afcb0 con 0x55b13d2ba5d0 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f6a07fa8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a001acef0 con 0x55b13d2bbb40 2024-09-15T23:30:39.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f6a07fa8740 10 client.0 _ll_register_callbacks cb 0x7f69f8009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d33e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d33e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39184/0 (socket says 172.21.15.89:39184) 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69fffff640 1 -- 172.21.15.89:0/3533181733 learned_addr learned my addr 172.21.15.89:0/3533181733 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69feffd640 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69fffff640 1 -- 172.21.15.89:0/3533181733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 msgr2=0x55b13d2c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69fffff640 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69fffff640 1 -- 172.21.15.89:0/3533181733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2ba5d0 msgr2=0x55b13d2c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69fffff640 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69fffff640 1 -- 172.21.15.89:0/3533181733 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b13d2b7960 con 0x55b13d2c5370 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69fffff640 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d33e8f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f69f8004c50 tx=0x7f69f8004210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1fe49eaf358173b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69deffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69deffd640 1 -- 172.21.15.89:0/3533181733 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f8024040 con 0x55b13d2c5370 2024-09-15T23:30:39.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f69deffd640 1 -- 172.21.15.89:0/3533181733 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f69f8004550 con 0x55b13d2c5370 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.749+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3533181733 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69f00bcc70 con 0x55b13d2c5370 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f69deffd640 1 -- 172.21.15.89:0/3533181733 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f69f8040050 con 0x55b13d2c5370 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f6a07fa8740 10 client.17703 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3533181733 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b13d33fff0 con 0x55b13d2c5370 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f69deffd640 1 -- 172.21.15.89:0/3533181733 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69f00bcc70 con 0x55b13d2c5370 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f69deffd640 1 -- 172.21.15.89:0/3533181733 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f69f80159d0 con 0x55b13d2c5370 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f6a07fa8740 10 client.17703 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f6a07fa8740 10 client.17703 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f6a07fa8740 20 client.17703 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b13d2d4f60 0x55b13d342740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f6a07fa8740 4 client.17703 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.750+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3533181733 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b13d227d70 con 0x55b13d2d4f60 2024-09-15T23:30:39.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.751+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b13d2d4f60 0x55b13d342740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:39.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f69deffd640 10 client.17703 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:30:39.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f69ff7fe640 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b13d2d4f60 0x55b13d342740 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=536e4fa210a44701 server_cookie=fe8631133deb015a in_seq=0 out_seq=0 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f69deffd640 1 -- 172.21.15.89:0/3533181733 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b13d227d70 con 0x55b13d2d4f60 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f69deffd640 10 client.17703 handle_command_reply: tid=0 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 client.17703 shutdown 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3533181733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 msgr2=0x55b13d33e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d33e8f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f69f8004c50 tx=0x7f69f8004210 comp rx=0 tx=0).stop 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3533181733 shutdown_connections 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b13d2c5370 0x55b13d33e8f0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b13d2d4f60 0x55b13d342740 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f69deffd640 0 client.17703 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b13d2bbb40 0x55b13d2c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 --2- 172.21.15.89:0/3533181733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b13d2ba5d0 0x55b13d2c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3533181733 >> 172.21.15.89:0/3533181733 conn(0x55b13d283310 msgr2=0x55b13d2c4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:39.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3533181733 shutdown_connections 2024-09-15T23:30:39.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.752+0000 7f6a07fa8740 1 -- 172.21.15.89:0/3533181733 wait complete. 2024-09-15T23:30:39.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a07fa8740 20 client.17703 trim_cache size 0 max 16384 2024-09-15T23:30:39.758 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:30:39.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 -- 172.21.15.89:0/2303850647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69e80a3240 msgr2=0x7f69e80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:39.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69e80a3240 0x7f69e80a5700 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6a0006b2e0 tx=0x7f69f4008040 comp rx=0 tx=0).stop 2024-09-15T23:30:39.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 -- 172.21.15.89:0/2303850647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a00100a80 msgr2=0x7f6a00199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:39.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a00100a80 0x7f6a00199450 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009a50 tx=0x7f69f0004270 comp rx=0 tx=0).stop 2024-09-15T23:30:39.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 -- 172.21.15.89:0/2303850647 shutdown_connections 2024-09-15T23:30:39.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f69e80a3240 0x7f69e80a5700 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a00106a80 0x7f6a001a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a00101420 0x7f6a00199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 --2- 172.21.15.89:0/2303850647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a00100a80 0x7f6a00199450 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:39.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 -- 172.21.15.89:0/2303850647 >> 172.21.15.89:0/2303850647 conn(0x7f6a000fc7a0 msgr2=0x7f6a000fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:39.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 -- 172.21.15.89:0/2303850647 shutdown_connections 2024-09-15T23:30:39.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:39.753+0000 7f6a05bcf640 1 -- 172.21.15.89:0/2303850647 wait complete. 2024-09-15T23:30:40.560 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:30:40.806 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.280+0000 7f536245f640 1 -- 172.21.15.89:0/2733702656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f535c106f20 msgr2=0x7f535c10eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.280+0000 7f536245f640 1 --2- 172.21.15.89:0/2733702656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f535c106f20 0x7f535c10eaa0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f5354009ca0 tx=0x7f535402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.280+0000 7f536245f640 1 -- 172.21.15.89:0/2733702656 shutdown_connections 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.280+0000 7f536245f640 1 --2- 172.21.15.89:0/2733702656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f535c106f20 0x7f535c10eaa0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.280+0000 7f536245f640 1 --2- 172.21.15.89:0/2733702656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f535c1027a0 0x7f535c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.280+0000 7f536245f640 1 --2- 172.21.15.89:0/2733702656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f535c101df0 0x7f535c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.280+0000 7f536245f640 1 -- 172.21.15.89:0/2733702656 >> 172.21.15.89:0/2733702656 conn(0x7f535c0fd8b0 msgr2=0x7f535c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.280+0000 7f536245f640 1 -- 172.21.15.89:0/2733702656 shutdown_connections 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.280+0000 7f536245f640 1 -- 172.21.15.89:0/2733702656 wait complete. 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f536245f640 1 Processor -- start 2024-09-15T23:30:41.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f536245f640 1 -- start start 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f536245f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f535c101df0 0x7f535c198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f536245f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f535c1027a0 0x7f535c198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f536245f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f535c106f20 0x7f535c19ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f536245f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f535c112020 con 0x7f535c106f20 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f536245f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f535c111ea0 con 0x7f535c101df0 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f536245f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1121a0 con 0x7f535c1027a0 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f53609fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f535c106f20 0x7f535c19ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f53609fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f535c106f20 0x7f535c19ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39210/0 (socket says 172.21.15.89:39210) 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f53609fc640 1 -- 172.21.15.89:0/1603796931 learned_addr learned my addr 172.21.15.89:0/1603796931 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.281+0000 7f535b7fe640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f535c1027a0 0x7f535c198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f535bfff640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f535c101df0 0x7f535c198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f53609fc640 1 -- 172.21.15.89:0/1603796931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f535c1027a0 msgr2=0x7f535c198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f53609fc640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f535c1027a0 0x7f535c198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f53609fc640 1 -- 172.21.15.89:0/1603796931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f535c101df0 msgr2=0x7f535c198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f53609fc640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f535c101df0 0x7f535c198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f53609fc640 1 -- 172.21.15.89:0/1603796931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535c19d530 con 0x7f535c106f20 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f53609fc640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f535c106f20 0x7f535c19ce50 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f5354031a60 tx=0x7f5354031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f53597fa640 1 -- 172.21.15.89:0/1603796931 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5354003b70 con 0x7f535c106f20 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f53597fa640 1 -- 172.21.15.89:0/1603796931 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5354003d10 con 0x7f535c106f20 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f536245f640 1 -- 172.21.15.89:0/1603796931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f535c19d7c0 con 0x7f535c106f20 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f53597fa640 1 -- 172.21.15.89:0/1603796931 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5354004770 con 0x7f535c106f20 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.282+0000 7f536245f640 1 -- 172.21.15.89:0/1603796931 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f535c1a5100 con 0x7f535c106f20 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.284+0000 7f53597fa640 1 -- 172.21.15.89:0/1603796931 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f535402f790 con 0x7f535c106f20 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f5364838740 1 Processor -- start 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f5364838740 1 -- start start 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f5364838740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f837090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f5364838740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f8375d0 0x56034f837a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f5364838740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f842370 0x56034f844760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f5364838740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56034f79d2f0 con 0x56034f838b40 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f5364838740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56034f744cf0 con 0x56034f8375d0 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f5364838740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56034f7992b0 con 0x56034f842370 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f837090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f837090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39222/0 (socket says 172.21.15.89:39222) 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f535b7fe640 1 -- 172.21.15.89:0/3075802509 learned_addr learned my addr 172.21.15.89:0/3075802509 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f53597fa640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53440a3240 0x7f53440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f535bfff640 1 --2- 172.21.15.89:0/3075802509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f8375d0 0x56034f837a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f535b7fe640 1 --2- 172.21.15.89:0/3075802509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f842370 0x56034f844760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f53609fc640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53440a3240 0x7f53440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f535b7fe640 1 -- 172.21.15.89:0/3075802509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f842370 msgr2=0x56034f844760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f535b7fe640 1 --2- 172.21.15.89:0/3075802509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f842370 0x56034f844760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f535b7fe640 1 -- 172.21.15.89:0/3075802509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f8375d0 msgr2=0x56034f837a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f535b7fe640 1 --2- 172.21.15.89:0/3075802509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f8375d0 0x56034f837a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f535b7fe640 1 -- 172.21.15.89:0/3075802509 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56034f7293a0 con 0x56034f838b40 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.285+0000 7f53597fa640 1 -- 172.21.15.89:0/1603796931 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f53540f1050 con 0x7f535c106f20 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.286+0000 7f535b7fe640 1 --2- 172.21.15.89:0/3075802509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f837090 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f5350006aa0 tx=0x7f53500095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52abd5cdd415930 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.286+0000 7f53427fc640 1 -- 172.21.15.89:0/3075802509 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350017610 con 0x56034f838b40 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.286+0000 7f53609fc640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53440a3240 0x7f53440a5700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f53540bcf80 tx=0x7f53540e8050 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.286+0000 7f53427fc640 1 -- 172.21.15.89:0/3075802509 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f535001c070 con 0x56034f838b40 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.286+0000 7f53427fc640 1 -- 172.21.15.89:0/3075802509 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350015940 con 0x56034f838b40 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 -- 172.21.15.89:0/3075802509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 msgr2=0x56034f837090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 --2- 172.21.15.89:0/3075802509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f837090 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f5350006aa0 tx=0x7f53500095d0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 -- 172.21.15.89:0/3075802509 shutdown_connections 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 --2- 172.21.15.89:0/3075802509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f842370 0x56034f844760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 --2- 172.21.15.89:0/3075802509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f8375d0 0x56034f837a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 --2- 172.21.15.89:0/3075802509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f837090 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 -- 172.21.15.89:0/3075802509 >> 172.21.15.89:0/3075802509 conn(0x56034f800310 msgr2=0x56034f833c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 -- 172.21.15.89:0/3075802509 shutdown_connections 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 -- 172.21.15.89:0/3075802509 wait complete. 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 -- start start 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.287+0000 7f5364838740 1 Processor -- start 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f5364838740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f841ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f5364838740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f5364838740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f842370 0x56034f83f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f5364838740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56034f845010 con 0x56034f838b40 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f5364838740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56034f758060 con 0x56034f842370 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f5364838740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56034f845190 con 0x56034f8375d0 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f5364838740 10 client.0 _ll_register_callbacks cb 0x56034f4ab8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39234/0 (socket says 172.21.15.89:39234) 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535bfff640 1 -- 172.21.15.89:0/3338092875 learned_addr learned my addr 172.21.15.89:0/3338092875 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535b7fe640 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f841ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535bfff640 1 -- 172.21.15.89:0/3338092875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 msgr2=0x56034f841ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535bfff640 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f841ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535bfff640 1 -- 172.21.15.89:0/3338092875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f842370 msgr2=0x56034f83f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535bfff640 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f842370 0x56034f83f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535bfff640 1 -- 172.21.15.89:0/3338092875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56034f50e3c0 con 0x56034f838b40 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535b7fe640 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f841ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f535bfff640 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83ef90 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f534c0104d0 tx=0x7f534c010990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5d538a88bcb52b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f533b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f533b7fe640 1 -- 172.21.15.89:0/3338092875 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c01a020 con 0x56034f838b40 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f533b7fe640 1 -- 172.21.15.89:0/3338092875 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f534c016690 con 0x56034f838b40 2024-09-15T23:30:41.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.288+0000 7f5364838740 1 -- 172.21.15.89:0/3338092875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f535001c070 con 0x56034f838b40 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f533b7fe640 1 -- 172.21.15.89:0/3338092875 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f534c025050 con 0x56034f838b40 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f5364838740 10 client.17715 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f5364838740 1 -- 172.21.15.89:0/3338092875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56034f850a80 con 0x56034f838b40 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f533b7fe640 1 -- 172.21.15.89:0/3338092875 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f535001c070 con 0x56034f838b40 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f533b7fe640 1 -- 172.21.15.89:0/3338092875 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f534c014a60 con 0x56034f838b40 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f5364838740 10 client.17715 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f5364838740 10 client.17715 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f5364838740 20 client.17715 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f5364838740 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56034f851fe0 0x56034f854420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f5364838740 4 client.17715 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f5364838740 1 -- 172.21.15.89:0/3338092875 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56034f7aa860 con 0x56034f851fe0 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.289+0000 7f535b7fe640 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56034f851fe0 0x56034f854420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.290+0000 7f533b7fe640 10 client.17715 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:30:41.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.290+0000 7f535b7fe640 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56034f851fe0 0x56034f854420 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d9d777e8baa8544b server_cookie=8b65b286d5c30a8e in_seq=0 out_seq=0 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f533b7fe640 1 -- 172.21.15.89:0/3338092875 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56034f7aa860 con 0x56034f851fe0 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f533b7fe640 10 client.17715 handle_command_reply: tid=0 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 client.17715 shutdown 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 -- 172.21.15.89:0/3338092875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 msgr2=0x56034f83ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83ef90 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f534c0104d0 tx=0x7f534c010990 comp rx=0 tx=0).stop 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 -- 172.21.15.89:0/3338092875 shutdown_connections 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56034f851fe0 0x56034f854420 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f842370 0x56034f83f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83ef90 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 --2- 172.21.15.89:0/3338092875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f841ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 -- 172.21.15.89:0/3338092875 >> 172.21.15.89:0/3338092875 conn(0x56034f800310 msgr2=0x56034f839930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f533b7fe640 0 client.17715 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:30:41.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 -- 172.21.15.89:0/3338092875 shutdown_connections 2024-09-15T23:30:41.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 1 -- 172.21.15.89:0/3338092875 wait complete. 2024-09-15T23:30:41.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.291+0000 7f5364838740 20 client.17715 trim_cache size 0 max 16384 2024-09-15T23:30:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f5364838740 1 Processor -- start 2024-09-15T23:30:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f5364838740 1 -- start start 2024-09-15T23:30:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f5364838740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f84e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f5364838740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f5364838740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f842370 0x56034f83a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f5364838740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1aef40 con 0x56034f838b40 2024-09-15T23:30:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f5364838740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1b0a80 con 0x56034f842370 2024-09-15T23:30:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f5364838740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1af610 con 0x56034f8375d0 2024-09-15T23:30:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f535bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f535bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39240/0 (socket says 172.21.15.89:39240) 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f535bfff640 1 -- 172.21.15.89:0/812942611 learned_addr learned my addr 172.21.15.89:0/812942611 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f535b7fe640 1 --2- 172.21.15.89:0/812942611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f842370 0x56034f83a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f535bfff640 1 -- 172.21.15.89:0/812942611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 msgr2=0x56034f84e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f535bfff640 1 --2- 172.21.15.89:0/812942611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f84e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f535bfff640 1 -- 172.21.15.89:0/812942611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f842370 msgr2=0x56034f83a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f535bfff640 1 --2- 172.21.15.89:0/812942611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f842370 0x56034f83a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f535bfff640 1 -- 172.21.15.89:0/812942611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56034f83f860 con 0x56034f838b40 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.304+0000 7f535bfff640 1 --2- 172.21.15.89:0/812942611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83e850 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f534c004810 tx=0x7f534c00b310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601485a6c654806d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5341ffb640 1 -- 172.21.15.89:0/812942611 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c01a050 con 0x56034f838b40 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5341ffb640 1 -- 172.21.15.89:0/812942611 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f534c010e90 con 0x56034f838b40 2024-09-15T23:30:41.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5341ffb640 1 -- 172.21.15.89:0/812942611 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534c014820 con 0x56034f838b40 2024-09-15T23:30:41.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5364838740 1 -- 172.21.15.89:0/812942611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 msgr2=0x56034f83e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:41.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5364838740 1 --2- 172.21.15.89:0/812942611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83e850 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f534c004810 tx=0x7f534c00b310 comp rx=0 tx=0).stop 2024-09-15T23:30:41.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5364838740 1 -- 172.21.15.89:0/812942611 shutdown_connections 2024-09-15T23:30:41.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5364838740 1 --2- 172.21.15.89:0/812942611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f842370 0x56034f83a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5364838740 1 --2- 172.21.15.89:0/812942611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f838b40 0x56034f83e850 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5364838740 1 --2- 172.21.15.89:0/812942611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f84e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5364838740 1 -- 172.21.15.89:0/812942611 >> 172.21.15.89:0/812942611 conn(0x56034f800310 msgr2=0x56034f83e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:41.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5364838740 1 -- 172.21.15.89:0/812942611 shutdown_connections 2024-09-15T23:30:41.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.305+0000 7f5364838740 1 -- 172.21.15.89:0/812942611 wait complete. 2024-09-15T23:30:41.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.306+0000 7f5364838740 1 -- start start 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.306+0000 7f5364838740 1 Processor -- start 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.306+0000 7f5364838740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f84a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.306+0000 7f5364838740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f838b40 0x56034f84ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.306+0000 7f5364838740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f842370 0x56034f83b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.306+0000 7f5364838740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1b03b0 con 0x56034f842370 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.306+0000 7f5364838740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1afce0 con 0x56034f838b40 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.306+0000 7f5364838740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f535c1acf20 con 0x56034f8375d0 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.306+0000 7f5364838740 10 client.0 _ll_register_callbacks cb 0x7f534c00b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.306+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f842370 0x56034f83b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f84a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f535bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f838b40 0x56034f84ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f535b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f842370 0x56034f83b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39250/0 (socket says 172.21.15.89:39250) 2024-09-15T23:30:41.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f535b7fe640 1 -- 172.21.15.89:0/2146444093 learned_addr learned my addr 172.21.15.89:0/2146444093 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f535b7fe640 1 -- 172.21.15.89:0/2146444093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 msgr2=0x56034f84a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f535b7fe640 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f84a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f535b7fe640 1 -- 172.21.15.89:0/2146444093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f838b40 msgr2=0x56034f84ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f535b7fe640 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f838b40 0x56034f84ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f535b7fe640 1 -- 172.21.15.89:0/2146444093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56034f83c0f0 con 0x56034f842370 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f535b7fe640 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f842370 0x56034f83b8c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5350009c30 tx=0x7f5350010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314db95a86519988 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f533b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f533b7fe640 1 -- 172.21.15.89:0/2146444093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350015940 con 0x56034f842370 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f533b7fe640 1 -- 172.21.15.89:0/2146444093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5350015ae0 con 0x56034f842370 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f5364838740 1 -- 172.21.15.89:0/2146444093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f534c010e90 con 0x56034f842370 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f533b7fe640 1 -- 172.21.15.89:0/2146444093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5350037050 con 0x56034f842370 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f5364838740 10 client.17721 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:41.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.307+0000 7f5364838740 1 -- 172.21.15.89:0/2146444093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56034f8bc840 con 0x56034f842370 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f533b7fe640 1 -- 172.21.15.89:0/2146444093 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f534c010e90 con 0x56034f842370 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f533b7fe640 1 -- 172.21.15.89:0/2146444093 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f53500292a0 con 0x56034f842370 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f5364838740 10 client.17721 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f5364838740 10 client.17721 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f5364838740 20 client.17721 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f5364838740 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56034f8bcce0 0x56034f8bf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f5364838740 4 client.17721 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f5364838740 1 -- 172.21.15.89:0/2146444093 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56034f7a4d70 con 0x56034f8bcce0 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f535bfff640 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56034f8bcce0 0x56034f8bf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f535bfff640 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56034f8bcce0 0x56034f8bf0d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bcd9f86ea1256bfe server_cookie=3640c83cd531d395 in_seq=0 out_seq=0 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.308+0000 7f533b7fe640 10 client.17721 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:30:41.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f533b7fe640 1 -- 172.21.15.89:0/2146444093 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56034f7a4d70 con 0x56034f8bcce0 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f533b7fe640 10 client.17721 handle_command_reply: tid=0 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 client.17721 shutdown 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 -- 172.21.15.89:0/2146444093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f842370 msgr2=0x56034f83b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f842370 0x56034f83b8c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5350009c30 tx=0x7f5350010f90 comp rx=0 tx=0).stop 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 -- 172.21.15.89:0/2146444093 shutdown_connections 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56034f8bcce0 0x56034f8bf0d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f533b7fe640 0 client.17721 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034f842370 0x56034f83b8c0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56034f838b40 0x56034f84ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 --2- 172.21.15.89:0/2146444093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56034f8375d0 0x56034f84a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 -- 172.21.15.89:0/2146444093 >> 172.21.15.89:0/2146444093 conn(0x56034f800310 msgr2=0x56034f832310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 -- 172.21.15.89:0/2146444093 shutdown_connections 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 1 -- 172.21.15.89:0/2146444093 wait complete. 2024-09-15T23:30:41.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f5364838740 20 client.17721 trim_cache size 0 max 16384 2024-09-15T23:30:41.316 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:30:41.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f536245f640 1 -- 172.21.15.89:0/1603796931 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53440a3240 msgr2=0x7f53440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:41.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f536245f640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53440a3240 0x7f53440a5700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f53540bcf80 tx=0x7f53540e8050 comp rx=0 tx=0).stop 2024-09-15T23:30:41.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.309+0000 7f536245f640 1 -- 172.21.15.89:0/1603796931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f535c106f20 msgr2=0x7f535c19ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:41.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.310+0000 7f536245f640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f535c106f20 0x7f535c19ce50 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f5354031a60 tx=0x7f5354031a90 comp rx=0 tx=0).stop 2024-09-15T23:30:41.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.310+0000 7f536245f640 1 -- 172.21.15.89:0/1603796931 shutdown_connections 2024-09-15T23:30:41.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.310+0000 7f536245f640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f53440a3240 0x7f53440a5700 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.310+0000 7f536245f640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f535c106f20 0x7f535c19ce50 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.310+0000 7f536245f640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f535c1027a0 0x7f535c198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.310+0000 7f536245f640 1 --2- 172.21.15.89:0/1603796931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f535c101df0 0x7f535c198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:41.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.310+0000 7f536245f640 1 -- 172.21.15.89:0/1603796931 >> 172.21.15.89:0/1603796931 conn(0x7f535c0fd8b0 msgr2=0x7f535c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:41.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.310+0000 7f536245f640 1 -- 172.21.15.89:0/1603796931 shutdown_connections 2024-09-15T23:30:41.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:41.310+0000 7f536245f640 1 -- 172.21.15.89:0/1603796931 wait complete. 2024-09-15T23:30:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:41 smithi120 ceph-mon[35250]: pgmap v550: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:30:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:41 smithi174 ceph-mon[35586]: pgmap v550: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:30:41.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:41 smithi089 ceph-mon[30933]: pgmap v550: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:30:42.183 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:30:42.433 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:42.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.908+0000 7fa65d85f640 1 -- 172.21.15.89:0/1690300664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa658106f20 msgr2=0x7fa65810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.908+0000 7fa65d85f640 1 --2- 172.21.15.89:0/1690300664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa658106f20 0x7fa65810eaa0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fa650009ca0 tx=0x7fa65002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.908+0000 7fa65d85f640 1 -- 172.21.15.89:0/1690300664 shutdown_connections 2024-09-15T23:30:42.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.908+0000 7fa65d85f640 1 --2- 172.21.15.89:0/1690300664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa658106f20 0x7fa65810eaa0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.908+0000 7fa65d85f640 1 --2- 172.21.15.89:0/1690300664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6581027a0 0x7fa6581067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.908+0000 7fa65d85f640 1 --2- 172.21.15.89:0/1690300664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658101df0 0x7fa6581021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.908+0000 7fa65d85f640 1 -- 172.21.15.89:0/1690300664 >> 172.21.15.89:0/1690300664 conn(0x7fa6580fd8b0 msgr2=0x7fa6580ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.908+0000 7fa65d85f640 1 -- 172.21.15.89:0/1690300664 shutdown_connections 2024-09-15T23:30:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.908+0000 7fa65d85f640 1 -- 172.21.15.89:0/1690300664 wait complete. 2024-09-15T23:30:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa65d85f640 1 Processor -- start 2024-09-15T23:30:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa65d85f640 1 -- start start 2024-09-15T23:30:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa65d85f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658101df0 0x7fa658198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa65d85f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6581027a0 0x7fa658198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa65d85f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa658106f20 0x7fa65819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa65d85f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa658111fd0 con 0x7fa658106f20 2024-09-15T23:30:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa65d85f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa658111e50 con 0x7fa658101df0 2024-09-15T23:30:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa65d85f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa658112150 con 0x7fa6581027a0 2024-09-15T23:30:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa6577fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa658106f20 0x7fa65819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa6577fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa658106f20 0x7fa65819cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39282/0 (socket says 172.21.15.89:39282) 2024-09-15T23:30:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa6577fe640 1 -- 172.21.15.89:0/3304894335 learned_addr learned my addr 172.21.15.89:0/3304894335 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa656ffd640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658101df0 0x7fa658198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.909+0000 7fa6567fc640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6581027a0 0x7fa658198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa6577fe640 1 -- 172.21.15.89:0/3304894335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6581027a0 msgr2=0x7fa658198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa6577fe640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6581027a0 0x7fa658198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa6577fe640 1 -- 172.21.15.89:0/3304894335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658101df0 msgr2=0x7fa658198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa6577fe640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658101df0 0x7fa658198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa6577fe640 1 -- 172.21.15.89:0/3304894335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65819d3c0 con 0x7fa658106f20 2024-09-15T23:30:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa6567fc640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6581027a0 0x7fa658198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa6577fe640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa658106f20 0x7fa65819cce0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa650003f10 tx=0x7fa650003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa65c85d640 1 -- 172.21.15.89:0/3304894335 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650031710 con 0x7fa658106f20 2024-09-15T23:30:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa65d85f640 1 -- 172.21.15.89:0/3304894335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65819d650 con 0x7fa658106f20 2024-09-15T23:30:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa65c85d640 1 -- 172.21.15.89:0/3304894335 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6500318b0 con 0x7fa658106f20 2024-09-15T23:30:42.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa65c85d640 1 -- 172.21.15.89:0/3304894335 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65002fb50 con 0x7fa658106f20 2024-09-15T23:30:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.910+0000 7fa65d85f640 1 -- 172.21.15.89:0/3304894335 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6581a4f90 con 0x7fa658106f20 2024-09-15T23:30:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.912+0000 7fa65c85d640 1 -- 172.21.15.89:0/3304894335 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa650004040 con 0x7fa658106f20 2024-09-15T23:30:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.912+0000 7fa65fc38740 1 Processor -- start 2024-09-15T23:30:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.912+0000 7fa65fc38740 1 -- start start 2024-09-15T23:30:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c21b40 0x562196c20090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c205d0 0x562196c20a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c2f780 0x562196c31b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa65fc38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562196b862f0 con 0x562196c205d0 2024-09-15T23:30:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa65fc38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562196b2dcf0 con 0x562196c2f780 2024-09-15T23:30:42.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa65fc38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562196b822b0 con 0x562196c21b40 2024-09-15T23:30:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c205d0 0x562196c20a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c205d0 0x562196c20a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39290/0 (socket says 172.21.15.89:39290) 2024-09-15T23:30:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa656ffd640 1 -- 172.21.15.89:0/3692968697 learned_addr learned my addr 172.21.15.89:0/3692968697 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa6567fc640 1 --2- 172.21.15.89:0/3692968697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c21b40 0x562196c20090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa6567fc640 1 --2- 172.21.15.89:0/3692968697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c2f780 0x562196c31b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa656ffd640 1 -- 172.21.15.89:0/3692968697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c21b40 msgr2=0x562196c20090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa65c85d640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa6440a3240 0x7fa6440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa656ffd640 1 --2- 172.21.15.89:0/3692968697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c21b40 0x562196c20090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa656ffd640 1 -- 172.21.15.89:0/3692968697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c2f780 msgr2=0x562196c31b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa656ffd640 1 --2- 172.21.15.89:0/3692968697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c2f780 0x562196c31b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa656ffd640 1 -- 172.21.15.89:0/3692968697 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562196b123a0 con 0x562196c205d0 2024-09-15T23:30:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa6567fc640 1 --2- 172.21.15.89:0/3692968697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c21b40 0x562196c20090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa6577fe640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa6440a3240 0x7fa6440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa656ffd640 1 --2- 172.21.15.89:0/3692968697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c205d0 0x562196c20a10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fa64800ac20 tx=0x7fa64800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3391c30a7ef178bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.913+0000 7fa65c85d640 1 -- 172.21.15.89:0/3304894335 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa6500f1290 con 0x7fa658106f20 2024-09-15T23:30:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa64dffb640 1 -- 172.21.15.89:0/3692968697 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64801d070 con 0x562196c205d0 2024-09-15T23:30:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa64dffb640 1 -- 172.21.15.89:0/3692968697 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa648018b20 con 0x562196c205d0 2024-09-15T23:30:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa64dffb640 1 -- 172.21.15.89:0/3692968697 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648010c00 con 0x562196c205d0 2024-09-15T23:30:42.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa65fc38740 1 -- 172.21.15.89:0/3692968697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c205d0 msgr2=0x562196c20a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa6577fe640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa6440a3240 0x7fa6440a5700 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa6500b9920 tx=0x7fa65008d250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa65fc38740 1 --2- 172.21.15.89:0/3692968697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c205d0 0x562196c20a10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fa64800ac20 tx=0x7fa64800f790 comp rx=0 tx=0).stop 2024-09-15T23:30:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa65fc38740 1 -- 172.21.15.89:0/3692968697 shutdown_connections 2024-09-15T23:30:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa65fc38740 1 --2- 172.21.15.89:0/3692968697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c2f780 0x562196c31b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa65fc38740 1 --2- 172.21.15.89:0/3692968697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c205d0 0x562196c20a10 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa65fc38740 1 --2- 172.21.15.89:0/3692968697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c21b40 0x562196c20090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa65fc38740 1 -- 172.21.15.89:0/3692968697 >> 172.21.15.89:0/3692968697 conn(0x562196be9310 msgr2=0x562196c1cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa65fc38740 1 -- 172.21.15.89:0/3692968697 shutdown_connections 2024-09-15T23:30:42.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.914+0000 7fa65fc38740 1 -- 172.21.15.89:0/3692968697 wait complete. 2024-09-15T23:30:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa65fc38740 1 -- start start 2024-09-15T23:30:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa65fc38740 1 Processor -- start 2024-09-15T23:30:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c205d0 0x562196c2f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c2f780 0x562196c2ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa65fc38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562196c32480 con 0x562196c21b40 2024-09-15T23:30:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa65fc38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562196b41060 con 0x562196c205d0 2024-09-15T23:30:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa65fc38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562196c32600 con 0x562196c2f780 2024-09-15T23:30:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa65fc38740 10 client.0 _ll_register_callbacks cb 0x5621968948c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:42.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39306/0 (socket says 172.21.15.89:39306) 2024-09-15T23:30:42.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa656ffd640 1 -- 172.21.15.89:0/3953943661 learned_addr learned my addr 172.21.15.89:0/3953943661 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:42.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.915+0000 7fa6567fc640 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c205d0 0x562196c2f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa656ffd640 1 -- 172.21.15.89:0/3953943661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c2f780 msgr2=0x562196c2ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa656ffd640 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c2f780 0x562196c2ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa656ffd640 1 -- 172.21.15.89:0/3953943661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c205d0 msgr2=0x562196c2f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa656ffd640 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c205d0 0x562196c2f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa656ffd640 1 -- 172.21.15.89:0/3953943661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621968f73c0 con 0x562196c21b40 2024-09-15T23:30:42.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa656ffd640 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2c440 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fa648005e90 tx=0x7fa6480186f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8987d19c4187c3b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa636ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa636ffd640 1 -- 172.21.15.89:0/3953943661 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64801d050 con 0x562196c21b40 2024-09-15T23:30:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa636ffd640 1 -- 172.21.15.89:0/3953943661 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa648010b00 con 0x562196c21b40 2024-09-15T23:30:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa65fc38740 1 -- 172.21.15.89:0/3953943661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa648018b20 con 0x562196c21b40 2024-09-15T23:30:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa636ffd640 1 -- 172.21.15.89:0/3953943661 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa648032050 con 0x562196c21b40 2024-09-15T23:30:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa65fc38740 10 client.17733 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa65fc38740 1 -- 172.21.15.89:0/3953943661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562196c3df50 con 0x562196c21b40 2024-09-15T23:30:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa636ffd640 1 -- 172.21.15.89:0/3953943661 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa648018b20 con 0x562196c21b40 2024-09-15T23:30:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa636ffd640 1 -- 172.21.15.89:0/3953943661 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa6480046c0 con 0x562196c21b40 2024-09-15T23:30:42.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa65fc38740 10 client.17733 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:42.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa65fc38740 10 client.17733 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:42.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa65fc38740 20 client.17733 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:42.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa65fc38740 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562196c3f350 0x562196c41740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa65fc38740 4 client.17733 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:42.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.916+0000 7fa65fc38740 1 -- 172.21.15.89:0/3953943661 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562196b93860 con 0x562196c3f350 2024-09-15T23:30:42.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.917+0000 7fa6567fc640 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562196c3f350 0x562196c41740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.917+0000 7fa6567fc640 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562196c3f350 0x562196c41740 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f66f0002092b80b server_cookie=ef891906e57398dd in_seq=0 out_seq=0 2024-09-15T23:30:42.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.917+0000 7fa636ffd640 10 client.17733 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:42.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa636ffd640 1 -- 172.21.15.89:0/3953943661 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562196b93860 con 0x562196c3f350 2024-09-15T23:30:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa636ffd640 10 client.17733 handle_command_reply: tid=0 2024-09-15T23:30:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 client.17733 shutdown 2024-09-15T23:30:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 -- 172.21.15.89:0/3953943661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 msgr2=0x562196c2c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2c440 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fa648005e90 tx=0x7fa6480186f0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 -- 172.21.15.89:0/3953943661 shutdown_connections 2024-09-15T23:30:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562196c3f350 0x562196c41740 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c2f780 0x562196c2ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2c440 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa636ffd640 0 client.17733 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:42.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 --2- 172.21.15.89:0/3953943661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c205d0 0x562196c2f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 -- 172.21.15.89:0/3953943661 >> 172.21.15.89:0/3953943661 conn(0x562196be9310 msgr2=0x562196c22960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 -- 172.21.15.89:0/3953943661 shutdown_connections 2024-09-15T23:30:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 1 -- 172.21.15.89:0/3953943661 wait complete. 2024-09-15T23:30:42.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.918+0000 7fa65fc38740 20 client.17733 trim_cache size 0 max 16384 2024-09-15T23:30:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.931+0000 7fa65fc38740 1 Processor -- start 2024-09-15T23:30:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.931+0000 7fa65fc38740 1 -- start start 2024-09-15T23:30:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.931+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c205d0 0x562196c3b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.931+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.931+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c2f780 0x562196c2e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.931+0000 7fa65fc38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581accf0 con 0x562196c21b40 2024-09-15T23:30:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.931+0000 7fa65fc38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581ae830 con 0x562196c205d0 2024-09-15T23:30:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.931+0000 7fa65fc38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581ad3c0 con 0x562196c2f780 2024-09-15T23:30:42.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39314/0 (socket says 172.21.15.89:39314) 2024-09-15T23:30:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa656ffd640 1 -- 172.21.15.89:0/1335041157 learned_addr learned my addr 172.21.15.89:0/1335041157 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa6567fc640 1 --2- 172.21.15.89:0/1335041157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c2f780 0x562196c2e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa656ffd640 1 -- 172.21.15.89:0/1335041157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c2f780 msgr2=0x562196c2e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa656ffd640 1 --2- 172.21.15.89:0/1335041157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c2f780 0x562196c2e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa656ffd640 1 -- 172.21.15.89:0/1335041157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c205d0 msgr2=0x562196c3b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa656ffd640 1 --2- 172.21.15.89:0/1335041157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c205d0 0x562196c3b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa656ffd640 1 -- 172.21.15.89:0/1335041157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562196c2b410 con 0x562196c21b40 2024-09-15T23:30:42.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa6567fc640 1 --2- 172.21.15.89:0/1335041157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c2f780 0x562196c2e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa656ffd640 1 --2- 172.21.15.89:0/1335041157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2bd00 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fa648004c30 tx=0x7fa64800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5b0a7ae8b0b68a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa64d7fa640 1 -- 172.21.15.89:0/1335041157 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64803a030 con 0x562196c21b40 2024-09-15T23:30:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa64d7fa640 1 -- 172.21.15.89:0/1335041157 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa648029070 con 0x562196c21b40 2024-09-15T23:30:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.932+0000 7fa64d7fa640 1 -- 172.21.15.89:0/1335041157 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64801d040 con 0x562196c21b40 2024-09-15T23:30:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.933+0000 7fa65fc38740 1 -- 172.21.15.89:0/1335041157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 msgr2=0x562196c2bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.933+0000 7fa65fc38740 1 --2- 172.21.15.89:0/1335041157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2bd00 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fa648004c30 tx=0x7fa64800f790 comp rx=0 tx=0).stop 2024-09-15T23:30:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.933+0000 7fa65fc38740 1 -- 172.21.15.89:0/1335041157 shutdown_connections 2024-09-15T23:30:42.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.933+0000 7fa65fc38740 1 --2- 172.21.15.89:0/1335041157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c2f780 0x562196c2e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.933+0000 7fa65fc38740 1 --2- 172.21.15.89:0/1335041157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c2bd00 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.933+0000 7fa65fc38740 1 --2- 172.21.15.89:0/1335041157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c205d0 0x562196c3b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.933+0000 7fa65fc38740 1 -- 172.21.15.89:0/1335041157 >> 172.21.15.89:0/1335041157 conn(0x562196be9310 msgr2=0x562196c2dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.933+0000 7fa65fc38740 1 -- 172.21.15.89:0/1335041157 shutdown_connections 2024-09-15T23:30:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.933+0000 7fa65fc38740 1 -- 172.21.15.89:0/1335041157 wait complete. 2024-09-15T23:30:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.933+0000 7fa65fc38740 1 -- start start 2024-09-15T23:30:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa65fc38740 1 Processor -- start 2024-09-15T23:30:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c205d0 0x562196c37eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c29070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa65fc38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c2f780 0x562196c295b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa65fc38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581ae160 con 0x562196c21b40 2024-09-15T23:30:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa65fc38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581ada90 con 0x562196c2f780 2024-09-15T23:30:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa65fc38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa6581aacd0 con 0x562196c205d0 2024-09-15T23:30:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa65fc38740 10 client.0 _ll_register_callbacks cb 0x7fa64801b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c29070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa656ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c29070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39320/0 (socket says 172.21.15.89:39320) 2024-09-15T23:30:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa656ffd640 1 -- 172.21.15.89:0/2720768143 learned_addr learned my addr 172.21.15.89:0/2720768143 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:42.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa6567fc640 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c205d0 0x562196c37eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa6567fc640 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c2f780 0x562196c295b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa656ffd640 1 -- 172.21.15.89:0/2720768143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c205d0 msgr2=0x562196c37eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa656ffd640 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c205d0 0x562196c37eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa656ffd640 1 -- 172.21.15.89:0/2720768143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c2f780 msgr2=0x562196c295b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa656ffd640 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c2f780 0x562196c295b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.934+0000 7fa656ffd640 1 -- 172.21.15.89:0/2720768143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562196c29ca0 con 0x562196c21b40 2024-09-15T23:30:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa656ffd640 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c29070 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fa648004870 tx=0x7fa648004660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84f745fef878dff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa636ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:42.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa636ffd640 1 -- 172.21.15.89:0/2720768143 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648042030 con 0x562196c21b40 2024-09-15T23:30:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa636ffd640 1 -- 172.21.15.89:0/2720768143 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa648018650 con 0x562196c21b40 2024-09-15T23:30:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa65fc38740 1 -- 172.21.15.89:0/2720768143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa648029070 con 0x562196c21b40 2024-09-15T23:30:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa636ffd640 1 -- 172.21.15.89:0/2720768143 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa64804c050 con 0x562196c21b40 2024-09-15T23:30:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa65fc38740 10 client.17739 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa65fc38740 1 -- 172.21.15.89:0/2720768143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562196ca9ba0 con 0x562196c21b40 2024-09-15T23:30:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa636ffd640 1 -- 172.21.15.89:0/2720768143 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa648029070 con 0x562196c21b40 2024-09-15T23:30:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa636ffd640 1 -- 172.21.15.89:0/2720768143 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa648010690 con 0x562196c21b40 2024-09-15T23:30:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa65fc38740 10 client.17739 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa65fc38740 10 client.17739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:42.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa65fc38740 20 client.17739 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa65fc38740 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562196caa2e0 0x562196cac700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa65fc38740 4 client.17739 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:30:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.935+0000 7fa65fc38740 1 -- 172.21.15.89:0/2720768143 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562196b8dd70 con 0x562196caa2e0 2024-09-15T23:30:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa6567fc640 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562196caa2e0 0x562196cac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa6567fc640 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562196caa2e0 0x562196cac700 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84efee13b431200a server_cookie=f4f8add410d988b2 in_seq=0 out_seq=0 2024-09-15T23:30:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa636ffd640 10 client.17739 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa636ffd640 1 -- 172.21.15.89:0/2720768143 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562196b8dd70 con 0x562196caa2e0 2024-09-15T23:30:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa636ffd640 10 client.17739 handle_command_reply: tid=0 2024-09-15T23:30:42.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa65fc38740 1 client.17739 shutdown 2024-09-15T23:30:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa65fc38740 1 -- 172.21.15.89:0/2720768143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 msgr2=0x562196c29070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa65fc38740 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c29070 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fa648004870 tx=0x7fa648004660 comp rx=0 tx=0).stop 2024-09-15T23:30:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa65fc38740 1 -- 172.21.15.89:0/2720768143 shutdown_connections 2024-09-15T23:30:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa65fc38740 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562196caa2e0 0x562196cac700 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa65fc38740 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562196c2f780 0x562196c295b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa65fc38740 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562196c21b40 0x562196c29070 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa65fc38740 1 --2- 172.21.15.89:0/2720768143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562196c205d0 0x562196c37eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.936+0000 7fa65fc38740 1 -- 172.21.15.89:0/2720768143 >> 172.21.15.89:0/2720768143 conn(0x562196be9310 msgr2=0x562196c2dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa636ffd640 0 client.17739 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:42.958 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65fc38740 1 -- 172.21.15.89:0/2720768143 shutdown_connections 2024-09-15T23:30:42.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65fc38740 1 -- 172.21.15.89:0/2720768143 wait complete. 2024-09-15T23:30:42.959 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65fc38740 20 client.17739 trim_cache size 0 max 16384 2024-09-15T23:30:42.959 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:30:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 -- 172.21.15.89:0/3304894335 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa6440a3240 msgr2=0x7fa6440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa6440a3240 0x7fa6440a5700 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa6500b9920 tx=0x7fa65008d250 comp rx=0 tx=0).stop 2024-09-15T23:30:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 -- 172.21.15.89:0/3304894335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa658106f20 msgr2=0x7fa65819cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa658106f20 0x7fa65819cce0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa650003f10 tx=0x7fa650003f40 comp rx=0 tx=0).stop 2024-09-15T23:30:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 -- 172.21.15.89:0/3304894335 shutdown_connections 2024-09-15T23:30:42.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa6440a3240 0x7fa6440a5700 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa658106f20 0x7fa65819cce0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6581027a0 0x7fa658198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 --2- 172.21.15.89:0/3304894335 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa658101df0 0x7fa658198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 -- 172.21.15.89:0/3304894335 >> 172.21.15.89:0/3304894335 conn(0x7fa6580fd8b0 msgr2=0x7fa6580feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 -- 172.21.15.89:0/3304894335 shutdown_connections 2024-09-15T23:30:42.961 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:42.937+0000 7fa65d85f640 1 -- 172.21.15.89:0/3304894335 wait complete. 2024-09-15T23:30:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:43 smithi120 ceph-mon[35250]: pgmap v551: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:30:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:43 smithi174 ceph-mon[35586]: pgmap v551: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:30:43.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:43 smithi089 ceph-mon[30933]: pgmap v551: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:30:44.823 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:30:44.823 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:30:45.070 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:45.449 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:45 smithi089 ceph-mon[30933]: pgmap v552: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:30:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:45 smithi120 ceph-mon[35250]: pgmap v552: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:30:45.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:45 smithi174 ceph-mon[35586]: pgmap v552: 129 pgs: 129 active+clean; 3.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:30:45.575 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:30:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:30:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "92a81101-87b2-4bd7-8904-c9e1fc2231e0", 2024-09-15T23:30:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:30:45.576 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:30:45.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 -- 172.21.15.89:0/280037487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9e81027a0 msgr2=0x7fa9e81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:45.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 --2- 172.21.15.89:0/280037487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9e81027a0 0x7fa9e81067f0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009900 tx=0x7fa9d802a410 comp rx=0 tx=0).stop 2024-09-15T23:30:45.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 -- 172.21.15.89:0/280037487 shutdown_connections 2024-09-15T23:30:45.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 --2- 172.21.15.89:0/280037487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e8106f20 0x7fa9e810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 --2- 172.21.15.89:0/280037487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9e81027a0 0x7fa9e81067f0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 --2- 172.21.15.89:0/280037487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e8101df0 0x7fa9e81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 -- 172.21.15.89:0/280037487 >> 172.21.15.89:0/280037487 conn(0x7fa9e80fd8f0 msgr2=0x7fa9e80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 -- 172.21.15.89:0/280037487 shutdown_connections 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 -- 172.21.15.89:0/280037487 wait complete. 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 Processor -- start 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.533+0000 7fa9f0676640 1 -- start start 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9f0676640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e8101df0 0x7fa9e81984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9f0676640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9e81027a0 0x7fa9e8198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9f0676640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e8106f20 0x7fa9e819cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9f0676640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8111fa0 con 0x7fa9e81027a0 2024-09-15T23:30:45.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9f0676640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8111e20 con 0x7fa9e8101df0 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9f0676640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8112120 con 0x7fa9e8106f20 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9edc11640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9e81027a0 0x7fa9e8198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9edc11640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9e81027a0 0x7fa9e8198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47230/0 (socket says 172.21.15.89:47230) 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9edc11640 1 -- 172.21.15.89:0/2175074355 learned_addr learned my addr 172.21.15.89:0/2175074355 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9ee412640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e8101df0 0x7fa9e81984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9eec13640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e8106f20 0x7fa9e819cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9edc11640 1 -- 172.21.15.89:0/2175074355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e8106f20 msgr2=0x7fa9e819cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9edc11640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e8106f20 0x7fa9e819cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9edc11640 1 -- 172.21.15.89:0/2175074355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e8101df0 msgr2=0x7fa9e81984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9edc11640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e8101df0 0x7fa9e81984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9edc11640 1 -- 172.21.15.89:0/2175074355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e819d470 con 0x7fa9e81027a0 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9eec13640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e8106f20 0x7fa9e819cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.534+0000 7fa9edc11640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9e81027a0 0x7fa9e8198a00 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8006010 tx=0x7fa9d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:45.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.535+0000 7fa9df7fe640 1 -- 172.21.15.89:0/2175074355 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d803f030 con 0x7fa9e81027a0 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.535+0000 7fa9df7fe640 1 -- 172.21.15.89:0/2175074355 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d8003ce0 con 0x7fa9e81027a0 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.535+0000 7fa9f0676640 1 -- 172.21.15.89:0/2175074355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e819d700 con 0x7fa9e81027a0 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.535+0000 7fa9df7fe640 1 -- 172.21.15.89:0/2175074355 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8035070 con 0x7fa9e81027a0 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.535+0000 7fa9f0676640 1 -- 172.21.15.89:0/2175074355 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e81a5040 con 0x7fa9e81027a0 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.537+0000 7fa9df7fe640 1 -- 172.21.15.89:0/2175074355 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa9d8042030 con 0x7fa9e81027a0 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.537+0000 7fa9f2a4f740 1 Processor -- start 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.537+0000 7fa9f2a4f740 1 -- start start 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce75f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce76480 0x55e60ce80800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce80d40 0x55e60ce83130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e60cddb280 con 0x55e60ce75b60 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e60cd82b10 con 0x55e60ce80d40 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e60cdd7240 con 0x55e60ce76480 2024-09-15T23:30:45.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9eec13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce75f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9eec13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce75f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47234/0 (socket says 172.21.15.89:47234) 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9eec13640 1 -- 172.21.15.89:0/667586427 learned_addr learned my addr 172.21.15.89:0/667586427 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9df7fe640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9d40a3240 0x7fa9d40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9edc11640 1 --2- 172.21.15.89:0/667586427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce80d40 0x55e60ce83130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9ee412640 1 --2- 172.21.15.89:0/667586427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce76480 0x55e60ce80800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9eec13640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9d40a3240 0x7fa9d40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9eec13640 1 -- 172.21.15.89:0/667586427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce76480 msgr2=0x55e60ce80800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9eec13640 1 --2- 172.21.15.89:0/667586427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce76480 0x55e60ce80800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9eec13640 1 -- 172.21.15.89:0/667586427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce80d40 msgr2=0x55e60ce83130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9eec13640 1 --2- 172.21.15.89:0/667586427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce80d40 0x55e60ce83130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9eec13640 1 -- 172.21.15.89:0/667586427 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e60cd8f1f0 con 0x55e60ce75b60 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9df7fe640 1 -- 172.21.15.89:0/2175074355 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa9d80f0180 con 0x7fa9e81027a0 2024-09-15T23:30:45.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.538+0000 7fa9eec13640 1 --2- 172.21.15.89:0/667586427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce75f40 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0014a90 tx=0x7fa9e0014f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f0776c1ca11a0c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9dcff9640 1 -- 172.21.15.89:0/667586427 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0014530 con 0x55e60ce75b60 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9dcff9640 1 -- 172.21.15.89:0/667586427 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9e0023070 con 0x55e60ce75b60 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9dcff9640 1 -- 172.21.15.89:0/667586427 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000f360 con 0x55e60ce75b60 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/667586427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 msgr2=0x55e60ce75f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9eec13640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9d40a3240 0x7fa9d40a5700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0022000 tx=0x7fa9e0020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/667586427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce75f40 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0014a90 tx=0x7fa9e0014f50 comp rx=0 tx=0).stop 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/667586427 shutdown_connections 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/667586427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce80d40 0x55e60ce83130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/667586427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce76480 0x55e60ce80800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/667586427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce75f40 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.539+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/667586427 >> 172.21.15.89:0/667586427 conn(0x55e60ce720f0 msgr2=0x55e60ce724d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/667586427 shutdown_connections 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/667586427 wait complete. 2024-09-15T23:30:45.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 1 -- start start 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 1 Processor -- start 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce7dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce7e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce80d40 0x55e60ce7e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e60ce83910 con 0x55e60ce75b60 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e60cd95f10 con 0x55e60ce76480 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e60ce83a90 con 0x55e60ce80d40 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9f2a4f740 10 client.0 _ll_register_callbacks cb 0x55e60cd5ea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9ee412640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce7dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9ee412640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce7dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47242/0 (socket says 172.21.15.89:47242) 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9ee412640 1 -- 172.21.15.89:0/2146517420 learned_addr learned my addr 172.21.15.89:0/2146517420 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9edc11640 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce7e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.540+0000 7fa9eec13640 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce80d40 0x55e60ce7e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9ee412640 1 -- 172.21.15.89:0/2146517420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce80d40 msgr2=0x55e60ce7e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9ee412640 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce80d40 0x55e60ce7e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9ee412640 1 -- 172.21.15.89:0/2146517420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 msgr2=0x55e60ce7e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9ee412640 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce7e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9ee412640 1 -- 172.21.15.89:0/2146517420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e60cb4c6a0 con 0x55e60ce75b60 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9edc11640 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce7e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9ee412640 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce7dc70 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4009ee0 tx=0x7fa9e4011630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77510ce57708ed79 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/2146517420 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4019070 con 0x55e60ce75b60 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/2146517420 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa9e4015660 con 0x55e60ce75b60 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2146517420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9e0023070 con 0x55e60ce75b60 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/2146517420 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa9e4025050 con 0x55e60ce75b60 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9f2a4f740 10 client.17751 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:45.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2146517420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e60ce8f5b0 con 0x55e60ce75b60 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/2146517420 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9e0023070 con 0x55e60ce75b60 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/2146517420 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa9e4014b40 con 0x55e60ce75b60 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9f2a4f740 10 client.17751 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9f2a4f740 10 client.17751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9f2a4f740 20 client.17751 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e60ce909c0 0x55e60ce92e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9f2a4f740 4 client.17751 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.541+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2146517420 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e60cde87f0 con 0x55e60ce909c0 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.542+0000 7fa9edc11640 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e60ce909c0 0x55e60ce92e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.542+0000 7fa9d1ffb640 10 client.17751 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.542+0000 7fa9edc11640 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e60ce909c0 0x55e60ce92e80 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5ac20dff897de79 server_cookie=b2e0817c972aa69 in_seq=0 out_seq=0 2024-09-15T23:30:45.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/2146517420 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e60cde87f0 con 0x55e60ce909c0 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9d1ffb640 10 client.17751 handle_command_reply: tid=0 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 client.17751 shutdown 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2146517420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 msgr2=0x55e60ce7dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce7dc70 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4009ee0 tx=0x7fa9e4011630 comp rx=0 tx=0).stop 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2146517420 shutdown_connections 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e60ce909c0 0x55e60ce92e80 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce80d40 0x55e60ce7e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce7e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/2146517420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce7dc70 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9d1ffb640 0 client.17751 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2146517420 >> 172.21.15.89:0/2146517420 conn(0x55e60ce720f0 msgr2=0x55e60ce71270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2146517420 shutdown_connections 2024-09-15T23:30:45.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2146517420 wait complete. 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.543+0000 7fa9f2a4f740 20 client.17751 trim_cache size 0 max 16384 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9f2a4f740 1 Processor -- start 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9f2a4f740 1 -- start start 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce75b60 0x55e60ce7cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce987a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce80d40 0x55e60ce98ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81aedf0 con 0x55e60ce80d40 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b0930 con 0x55e60ce76480 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81af4c0 con 0x55e60ce75b60 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9eec13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce80d40 0x55e60ce98ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9eec13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce80d40 0x55e60ce98ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47256/0 (socket says 172.21.15.89:47256) 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9eec13640 1 -- 172.21.15.89:0/2582820529 learned_addr learned my addr 172.21.15.89:0/2582820529 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9edc11640 1 --2- 172.21.15.89:0/2582820529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9ee412640 1 --2- 172.21.15.89:0/2582820529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce75b60 0x55e60ce7cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9eec13640 1 -- 172.21.15.89:0/2582820529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce75b60 msgr2=0x55e60ce7cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9eec13640 1 --2- 172.21.15.89:0/2582820529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce75b60 0x55e60ce7cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9eec13640 1 -- 172.21.15.89:0/2582820529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 msgr2=0x55e60ce987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9eec13640 1 --2- 172.21.15.89:0/2582820529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.560+0000 7fa9eec13640 1 -- 172.21.15.89:0/2582820529 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e60ce7c1e0 con 0x55e60ce80d40 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9eec13640 1 --2- 172.21.15.89:0/2582820529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce80d40 0x55e60ce98ce0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009c20 tx=0x7fa9e0009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a2d195e852cbc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9d3fff640 1 -- 172.21.15.89:0/2582820529 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0014040 con 0x55e60ce80d40 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9d3fff640 1 -- 172.21.15.89:0/2582820529 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9e0023040 con 0x55e60ce80d40 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9d3fff640 1 -- 172.21.15.89:0/2582820529 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000f9e0 con 0x55e60ce80d40 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2582820529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce80d40 msgr2=0x55e60ce98ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/2582820529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce80d40 0x55e60ce98ce0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009c20 tx=0x7fa9e0009c50 comp rx=0 tx=0).stop 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2582820529 shutdown_connections 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/2582820529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce80d40 0x55e60ce98ce0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/2582820529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/2582820529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce75b60 0x55e60ce7cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2582820529 >> 172.21.15.89:0/2582820529 conn(0x55e60ce720f0 msgr2=0x55e60ce7fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2582820529 shutdown_connections 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.561+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/2582820529 wait complete. 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9f2a4f740 1 -- start start 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9f2a4f740 1 Processor -- start 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce89240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce85cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9f2a4f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce80d40 0x55e60ce861f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b0260 con 0x55e60ce75b60 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81afb90 con 0x55e60ce76480 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9f2a4f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81acdd0 con 0x55e60ce80d40 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9f2a4f740 10 client.0 _ll_register_callbacks cb 0x7fa9e40094a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:45.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9ee412640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce89240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9ee412640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce89240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47258/0 (socket says 172.21.15.89:47258) 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9ee412640 1 -- 172.21.15.89:0/3024766487 learned_addr learned my addr 172.21.15.89:0/3024766487 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9edc11640 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce85cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9ee412640 1 -- 172.21.15.89:0/3024766487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce80d40 msgr2=0x55e60ce861f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9ee412640 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce80d40 0x55e60ce861f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9ee412640 1 -- 172.21.15.89:0/3024766487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 msgr2=0x55e60ce85cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9ee412640 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce85cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.562+0000 7fa9ee412640 1 -- 172.21.15.89:0/3024766487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e60ce86830 con 0x55e60ce75b60 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9ee412640 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce89240 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4008010 tx=0x7fa9e40059c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83dfc3cb9fe20e23 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/3024766487 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e402d020 con 0x55e60ce75b60 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/3024766487 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa9e40143e0 con 0x55e60ce75b60 2024-09-15T23:30:45.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/3024766487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9e0023040 con 0x55e60ce75b60 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/3024766487 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa9e4038050 con 0x55e60ce75b60 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9f2a4f740 10 client.17757 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/3024766487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e60cefc650 con 0x55e60ce75b60 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/3024766487 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9e0023040 con 0x55e60ce75b60 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/3024766487 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa9e401dac0 con 0x55e60ce75b60 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9f2a4f740 10 client.17757 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9f2a4f740 10 client.17757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9f2a4f740 20 client.17757 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e60cefce50 0x55e60ceff270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9f2a4f740 4 client.17757 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/3024766487 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e60cde2d00 con 0x55e60cefce50 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.563+0000 7fa9edc11640 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e60cefce50 0x55e60ceff270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.564+0000 7fa9d1ffb640 10 client.17757 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:45.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.564+0000 7fa9edc11640 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e60cefce50 0x55e60ceff270 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64f33e3ca72aa2c8 server_cookie=c7b6101c64e29b15 in_seq=0 out_seq=0 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9d1ffb640 1 -- 172.21.15.89:0/3024766487 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e60cde2d00 con 0x55e60cefce50 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9d1ffb640 10 client.17757 handle_command_reply: tid=0 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 client.17757 shutdown 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/3024766487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 msgr2=0x55e60ce89240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce89240 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4008010 tx=0x7fa9e40059c0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/3024766487 shutdown_connections 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e60cefce50 0x55e60ceff270 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e60ce80d40 0x55e60ce861f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e60ce76480 0x55e60ce85cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 --2- 172.21.15.89:0/3024766487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e60ce75b60 0x55e60ce89240 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/3024766487 >> 172.21.15.89:0/3024766487 conn(0x55e60ce720f0 msgr2=0x55e60ce775b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:45.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9d1ffb640 0 client.17757 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:45.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/3024766487 shutdown_connections 2024-09-15T23:30:45.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 1 -- 172.21.15.89:0/3024766487 wait complete. 2024-09-15T23:30:45.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.565+0000 7fa9f2a4f740 20 client.17757 trim_cache size 0 max 16384 2024-09-15T23:30:45.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 -- 172.21.15.89:0/2175074355 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9d40a3240 msgr2=0x7fa9d40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:45.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9d40a3240 0x7fa9d40a5700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0022000 tx=0x7fa9e0020000 comp rx=0 tx=0).stop 2024-09-15T23:30:45.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 -- 172.21.15.89:0/2175074355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9e81027a0 msgr2=0x7fa9e8198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:45.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9e81027a0 0x7fa9e8198a00 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8006010 tx=0x7fa9d80042f0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 -- 172.21.15.89:0/2175074355 shutdown_connections 2024-09-15T23:30:45.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9d40a3240 0x7fa9d40a5700 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e8106f20 0x7fa9e819cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9e81027a0 0x7fa9e8198a00 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 --2- 172.21.15.89:0/2175074355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa9e8101df0 0x7fa9e81984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:45.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 -- 172.21.15.89:0/2175074355 >> 172.21.15.89:0/2175074355 conn(0x7fa9e80fd8f0 msgr2=0x7fa9e80fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:45.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 -- 172.21.15.89:0/2175074355 shutdown_connections 2024-09-15T23:30:45.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:45.566+0000 7fa9f0676640 1 -- 172.21.15.89:0/2175074355 wait complete. 2024-09-15T23:30:46.342 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:30:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:46 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:30:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:46 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:46 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:30:46.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:46 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:46.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:46 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:30:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:46 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:46 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:30:46.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:46 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:46.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:46 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:30:46.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:46 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:46.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:46 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:30:46.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:46 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:47.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.091+0000 7fc585537640 1 -- 172.21.15.89:0/531444758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc580101420 msgr2=0x7fc58010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.091+0000 7fc585537640 1 --2- 172.21.15.89:0/531444758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc580101420 0x7fc58010d950 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fc578009930 tx=0x7fc57802a420 comp rx=0 tx=0).stop 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.091+0000 7fc585537640 1 -- 172.21.15.89:0/531444758 shutdown_connections 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.091+0000 7fc585537640 1 --2- 172.21.15.89:0/531444758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc580101420 0x7fc58010d950 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.091+0000 7fc585537640 1 --2- 172.21.15.89:0/531444758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc580100a80 0x7fc580100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.091+0000 7fc585537640 1 --2- 172.21.15.89:0/531444758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc580106a80 0x7fc580106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.091+0000 7fc585537640 1 -- 172.21.15.89:0/531444758 >> 172.21.15.89:0/531444758 conn(0x7fc5800fc7a0 msgr2=0x7fc5800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.091+0000 7fc585537640 1 -- 172.21.15.89:0/531444758 shutdown_connections 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.091+0000 7fc585537640 1 -- 172.21.15.89:0/531444758 wait complete. 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc585537640 1 Processor -- start 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc585537640 1 -- start start 2024-09-15T23:30:47.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc585537640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc580100a80 0x7fc5801991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc585537640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc580101420 0x7fc580199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc585537640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc580106a80 0x7fc580199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc585537640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc580110e80 con 0x7fc580100a80 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc585537640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc580110d00 con 0x7fc580101420 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc585537640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc580111000 con 0x7fc580106a80 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc57effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc580100a80 0x7fc5801991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc57effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc580100a80 0x7fc5801991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47274/0 (socket says 172.21.15.89:47274) 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc57effd640 1 -- 172.21.15.89:0/3502255349 learned_addr learned my addr 172.21.15.89:0/3502255349 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.092+0000 7fc57e7fc640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc580101420 0x7fc580199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc57effd640 1 -- 172.21.15.89:0/3502255349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc580106a80 msgr2=0x7fc580199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc57effd640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc580106a80 0x7fc580199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc57effd640 1 -- 172.21.15.89:0/3502255349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc580101420 msgr2=0x7fc580199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc57effd640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc580101420 0x7fc580199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc57effd640 1 -- 172.21.15.89:0/3502255349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5801a45f0 con 0x7fc580100a80 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc57e7fc640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc580101420 0x7fc580199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc57effd640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc580100a80 0x7fc5801991f0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc57000cce0 tx=0x7fc57000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc567fff640 1 -- 172.21.15.89:0/3502255349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57000f650 con 0x7fc580100a80 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc585537640 1 -- 172.21.15.89:0/3502255349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5801a48e0 con 0x7fc580100a80 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc567fff640 1 -- 172.21.15.89:0/3502255349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc57000fcd0 con 0x7fc580100a80 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc567fff640 1 -- 172.21.15.89:0/3502255349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570018c70 con 0x7fc580100a80 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.093+0000 7fc585537640 1 -- 172.21.15.89:0/3502255349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5801a4e70 con 0x7fc580100a80 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.095+0000 7fc567fff640 1 -- 172.21.15.89:0/3502255349 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc57000f7f0 con 0x7fc580100a80 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.095+0000 7fc587910740 1 Processor -- start 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc587910740 1 -- start start 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc587910740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77bd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc587910740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bddc0 0x5625f77c8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc587910740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77c8680 0x5625f77caa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc587910740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5625f77232e0 con 0x5625f77bd4a0 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc587910740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5625f76cace0 con 0x5625f77c8680 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc587910740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5625f771f2a0 con 0x5625f77bddc0 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc567fff640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5680a3240 0x7fc5680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77bd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77bd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47286/0 (socket says 172.21.15.89:47286) 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57f7fe640 1 -- 172.21.15.89:0/459823502 learned_addr learned my addr 172.21.15.89:0/459823502 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5680a3240 0x7fc5680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57effd640 1 --2- 172.21.15.89:0/459823502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bddc0 0x5625f77c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc567fff640 1 -- 172.21.15.89:0/3502255349 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc5700d01f0 con 0x7fc580100a80 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57e7fc640 1 --2- 172.21.15.89:0/459823502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77c8680 0x5625f77caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57f7fe640 1 -- 172.21.15.89:0/459823502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bddc0 msgr2=0x5625f77c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:47.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/459823502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bddc0 0x5625f77c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57f7fe640 1 -- 172.21.15.89:0/459823502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77c8680 msgr2=0x5625f77caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/459823502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77c8680 0x5625f77caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.096+0000 7fc57f7fe640 1 -- 172.21.15.89:0/459823502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625f76af440 con 0x5625f77bd4a0 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/459823502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77bd880 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc578005ec0 tx=0x7fc57803e6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addc9baf25724d1d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5680a3240 0x7fc5680a5700 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc57802a940 tx=0x7fc578035000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc5657fa640 1 -- 172.21.15.89:0/459823502 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc578035070 con 0x5625f77bd4a0 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc5657fa640 1 -- 172.21.15.89:0/459823502 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc57804d070 con 0x5625f77bd4a0 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc5657fa640 1 -- 172.21.15.89:0/459823502 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc578003b00 con 0x5625f77bd4a0 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc587910740 1 -- 172.21.15.89:0/459823502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 msgr2=0x5625f77bd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc587910740 1 --2- 172.21.15.89:0/459823502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77bd880 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc578005ec0 tx=0x7fc57803e6a0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc587910740 1 -- 172.21.15.89:0/459823502 shutdown_connections 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc587910740 1 --2- 172.21.15.89:0/459823502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77c8680 0x5625f77caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc587910740 1 --2- 172.21.15.89:0/459823502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bddc0 0x5625f77c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc587910740 1 --2- 172.21.15.89:0/459823502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77bd880 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc587910740 1 -- 172.21.15.89:0/459823502 >> 172.21.15.89:0/459823502 conn(0x5625f77b9a30 msgr2=0x5625f77b9e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc587910740 1 -- 172.21.15.89:0/459823502 shutdown_connections 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.097+0000 7fc587910740 1 -- 172.21.15.89:0/459823502 wait complete. 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc587910740 1 -- start start 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc587910740 1 Processor -- start 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc587910740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77bd4a0 0x5625f77c8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc587910740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bddc0 0x5625f77c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc587910740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77c8680 0x5625f77c5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc587910740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5625f77cb250 con 0x5625f77bddc0 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc587910740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5625f76de050 con 0x5625f77bd4a0 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc587910740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5625f77cb3d0 con 0x5625f77c8680 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc587910740 10 client.0 _ll_register_callbacks cb 0x5625f771e160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:47.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bddc0 0x5625f77c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bddc0 0x5625f77c52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47294/0 (socket says 172.21.15.89:47294) 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57e7fc640 1 -- 172.21.15.89:0/3872093404 learned_addr learned my addr 172.21.15.89:0/3872093404 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57effd640 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77bd4a0 0x5625f77c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77c8680 0x5625f77c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57e7fc640 1 -- 172.21.15.89:0/3872093404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77c8680 msgr2=0x5625f77c5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57e7fc640 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77c8680 0x5625f77c5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57e7fc640 1 -- 172.21.15.89:0/3872093404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77bd4a0 msgr2=0x5625f77c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57e7fc640 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77bd4a0 0x5625f77c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57e7fc640 1 -- 172.21.15.89:0/3872093404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625f74943c0 con 0x5625f77bddc0 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.098+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77c8680 0x5625f77c5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc57e7fc640 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bddc0 0x5625f77c52b0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fc57400f710 tx=0x7fc57400fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca32c87e6f62efe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc562ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:47.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc562ff5640 1 -- 172.21.15.89:0/3872093404 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574016070 con 0x5625f77bddc0 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc562ff5640 1 -- 172.21.15.89:0/3872093404 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc57400bb00 con 0x5625f77bddc0 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc587910740 1 -- 172.21.15.89:0/3872093404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc57804d070 con 0x5625f77bddc0 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc562ff5640 1 -- 172.21.15.89:0/3872093404 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc574025050 con 0x5625f77bddc0 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc587910740 10 client.17769 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc587910740 1 -- 172.21.15.89:0/3872093404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625f77d6ca0 con 0x5625f77bddc0 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc562ff5640 1 -- 172.21.15.89:0/3872093404 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc57804d070 con 0x5625f77bddc0 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc562ff5640 1 -- 172.21.15.89:0/3872093404 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc57401a400 con 0x5625f77bddc0 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc587910740 10 client.17769 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc587910740 10 client.17769 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc587910740 20 client.17769 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc587910740 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625f77d81b0 0x5625f77da5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc587910740 4 client.17769 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.099+0000 7fc587910740 1 -- 172.21.15.89:0/3872093404 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625f7730850 con 0x5625f77d81b0 2024-09-15T23:30:47.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.100+0000 7fc57effd640 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625f77d81b0 0x5625f77da5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.100+0000 7fc562ff5640 10 client.17769 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.100+0000 7fc57effd640 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625f77d81b0 0x5625f77da5f0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46d3fa967282c241 server_cookie=cb03ce73a1c2289e in_seq=0 out_seq=0 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc562ff5640 1 -- 172.21.15.89:0/3872093404 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5625f7730850 con 0x5625f77d81b0 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc562ff5640 10 client.17769 handle_command_reply: tid=0 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 client.17769 shutdown 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 -- 172.21.15.89:0/3872093404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bddc0 msgr2=0x5625f77c52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bddc0 0x5625f77c52b0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fc57400f710 tx=0x7fc57400fb10 comp rx=0 tx=0).stop 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 -- 172.21.15.89:0/3872093404 shutdown_connections 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625f77d81b0 0x5625f77da5f0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77c8680 0x5625f77c5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc562ff5640 0 client.17769 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bddc0 0x5625f77c52b0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 --2- 172.21.15.89:0/3872093404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77bd4a0 0x5625f77c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 -- 172.21.15.89:0/3872093404 >> 172.21.15.89:0/3872093404 conn(0x5625f77b9a30 msgr2=0x5625f77bf710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 -- 172.21.15.89:0/3872093404 shutdown_connections 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 1 -- 172.21.15.89:0/3872093404 wait complete. 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.101+0000 7fc587910740 20 client.17769 trim_cache size 0 max 16384 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc587910740 1 Processor -- start 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc587910740 1 -- start start 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc587910740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77c5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc587910740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bddc0 0x5625f77d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc587910740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77c8680 0x5625f77dfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc587910740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801aed90 con 0x5625f77bd4a0 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc587910740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801b08d0 con 0x5625f77c8680 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc587910740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801af460 con 0x5625f77bddc0 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77c5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47302/0 (socket says 172.21.15.89:47302) 2024-09-15T23:30:47.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57effd640 1 -- 172.21.15.89:0/3837605714 learned_addr learned my addr 172.21.15.89:0/3837605714 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/3837605714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77c8680 0x5625f77dfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57e7fc640 1 --2- 172.21.15.89:0/3837605714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bddc0 0x5625f77d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57effd640 1 -- 172.21.15.89:0/3837605714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bddc0 msgr2=0x5625f77d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57effd640 1 --2- 172.21.15.89:0/3837605714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bddc0 0x5625f77d44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57effd640 1 -- 172.21.15.89:0/3837605714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77c8680 msgr2=0x5625f77dfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57effd640 1 --2- 172.21.15.89:0/3837605714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77c8680 0x5625f77dfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57effd640 1 -- 172.21.15.89:0/3837605714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625f77c4260 con 0x5625f77bd4a0 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.118+0000 7fc57effd640 1 --2- 172.21.15.89:0/3837605714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77c5280 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fc57000af80 tx=0x7fc570013b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b616cdef4dfbddb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc564ff9640 1 -- 172.21.15.89:0/3837605714 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5700b4070 con 0x5625f77bd4a0 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc564ff9640 1 -- 172.21.15.89:0/3837605714 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc57009cb20 con 0x5625f77bd4a0 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc564ff9640 1 -- 172.21.15.89:0/3837605714 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc570018e10 con 0x5625f77bd4a0 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc587910740 1 -- 172.21.15.89:0/3837605714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 msgr2=0x5625f77c5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc587910740 1 --2- 172.21.15.89:0/3837605714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77c5280 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fc57000af80 tx=0x7fc570013b80 comp rx=0 tx=0).stop 2024-09-15T23:30:47.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc587910740 1 -- 172.21.15.89:0/3837605714 shutdown_connections 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc587910740 1 --2- 172.21.15.89:0/3837605714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77c8680 0x5625f77dfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc587910740 1 --2- 172.21.15.89:0/3837605714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bddc0 0x5625f77d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc587910740 1 --2- 172.21.15.89:0/3837605714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77bd4a0 0x5625f77c5280 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc587910740 1 -- 172.21.15.89:0/3837605714 >> 172.21.15.89:0/3837605714 conn(0x5625f77b9a30 msgr2=0x5625f77c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc587910740 1 -- 172.21.15.89:0/3837605714 shutdown_connections 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.119+0000 7fc587910740 1 -- 172.21.15.89:0/3837605714 wait complete. 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc587910740 1 -- start start 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc587910740 1 Processor -- start 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc587910740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bd4a0 0x5625f77cc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc587910740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77bddc0 0x5625f77ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc587910740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77c8680 0x5625f77cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc587910740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801b0200 con 0x5625f77c8680 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc587910740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801afb30 con 0x5625f77bddc0 2024-09-15T23:30:47.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc587910740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc5801acd70 con 0x5625f77bd4a0 2024-09-15T23:30:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc587910740 10 client.0 _ll_register_callbacks cb 0x7fc57400d360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc57f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77c8680 0x5625f77cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc57f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77c8680 0x5625f77cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47318/0 (socket says 172.21.15.89:47318) 2024-09-15T23:30:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc57f7fe640 1 -- 172.21.15.89:0/1541151452 learned_addr learned my addr 172.21.15.89:0/1541151452 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc57e7fc640 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77bddc0 0x5625f77ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc57effd640 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bd4a0 0x5625f77cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc57f7fe640 1 -- 172.21.15.89:0/1541151452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bd4a0 msgr2=0x5625f77cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bd4a0 0x5625f77cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc57f7fe640 1 -- 172.21.15.89:0/1541151452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77bddc0 msgr2=0x5625f77ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77bddc0 0x5625f77ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.120+0000 7fc57f7fe640 1 -- 172.21.15.89:0/1541151452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625f77bdbf0 con 0x5625f77c8680 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc57f7fe640 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77c8680 0x5625f77cd090 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x5625f77be360 tx=0x7fc578005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7ddf0f3988d081 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc562ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc562ff5640 1 -- 172.21.15.89:0/1541151452 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57803e250 con 0x5625f77c8680 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc562ff5640 1 -- 172.21.15.89:0/1541151452 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc5780035f0 con 0x5625f77c8680 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc587910740 1 -- 172.21.15.89:0/1541151452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc57009cb20 con 0x5625f77c8680 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc562ff5640 1 -- 172.21.15.89:0/1541151452 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc578065050 con 0x5625f77c8680 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc587910740 10 client.17775 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc587910740 1 -- 172.21.15.89:0/1541151452 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625f7843250 con 0x5625f77c8680 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc562ff5640 1 -- 172.21.15.89:0/1541151452 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc57009cb20 con 0x5625f77c8680 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc562ff5640 1 -- 172.21.15.89:0/1541151452 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc57803a040 con 0x5625f77c8680 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc587910740 10 client.17775 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:47.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc587910740 10 client.17775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:47.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc587910740 20 client.17775 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:47.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc587910740 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625f77d81b0 0x5625f78455e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc587910740 4 client.17775 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:30:47.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.121+0000 7fc587910740 1 -- 172.21.15.89:0/1541151452 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5625f772ad60 con 0x5625f77d81b0 2024-09-15T23:30:47.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.122+0000 7fc57effd640 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625f77d81b0 0x5625f78455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.122+0000 7fc57effd640 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625f77d81b0 0x5625f78455e0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e5b125920b8e43a server_cookie=83c05ce00523cb70 in_seq=0 out_seq=0 2024-09-15T23:30:47.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.122+0000 7fc562ff5640 10 client.17775 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:47.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.122+0000 7fc562ff5640 1 -- 172.21.15.89:0/1541151452 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5625f772ad60 con 0x5625f77d81b0 2024-09-15T23:30:47.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.122+0000 7fc562ff5640 10 client.17775 handle_command_reply: tid=0 2024-09-15T23:30:47.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.122+0000 7fc587910740 1 client.17775 shutdown 2024-09-15T23:30:47.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 1 -- 172.21.15.89:0/1541151452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77c8680 msgr2=0x5625f77cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77c8680 0x5625f77cd090 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x5625f77be360 tx=0x7fc578005a20 comp rx=0 tx=0).stop 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 1 -- 172.21.15.89:0/1541151452 shutdown_connections 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5625f77d81b0 0x5625f78455e0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625f77c8680 0x5625f77cd090 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625f77bddc0 0x5625f77ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 1 --2- 172.21.15.89:0/1541151452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625f77bd4a0 0x5625f77cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 1 -- 172.21.15.89:0/1541151452 >> 172.21.15.89:0/1541151452 conn(0x5625f77b9a30 msgr2=0x5625f77c7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc562ff5640 0 client.17775 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 1 -- 172.21.15.89:0/1541151452 shutdown_connections 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 1 -- 172.21.15.89:0/1541151452 wait complete. 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc587910740 20 client.17775 trim_cache size 0 max 16384 2024-09-15T23:30:47.139 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:30:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:30:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:30:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "92a81101-87b2-4bd7-8904-c9e1fc2231e0": { 2024-09-15T23:30:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:30:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "92a81101-87b2-4bd7-8904-c9e1fc2231e0", 2024-09-15T23:30:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:30:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:30:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:30:47.140 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 -- 172.21.15.89:0/3502255349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5680a3240 msgr2=0x7fc5680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5680a3240 0x7fc5680a5700 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc57802a940 tx=0x7fc578035000 comp rx=0 tx=0).stop 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 -- 172.21.15.89:0/3502255349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc580100a80 msgr2=0x7fc5801991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc580100a80 0x7fc5801991f0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc57000cce0 tx=0x7fc57000a620 comp rx=0 tx=0).stop 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 -- 172.21.15.89:0/3502255349 shutdown_connections 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5680a3240 0x7fc5680a5700 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc580106a80 0x7fc580199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc580101420 0x7fc580199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 --2- 172.21.15.89:0/3502255349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc580100a80 0x7fc5801991f0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 -- 172.21.15.89:0/3502255349 >> 172.21.15.89:0/3502255349 conn(0x7fc5800fc7a0 msgr2=0x7fc5800fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.123+0000 7fc585537640 1 -- 172.21.15.89:0/3502255349 shutdown_connections 2024-09-15T23:30:47.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:30:47.124+0000 7fc585537640 1 -- 172.21.15.89:0/3502255349 wait complete. 2024-09-15T23:30:47.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:47 smithi089 ceph-mon[30933]: pgmap v553: 129 pgs: 129 active+clean; 3.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:30:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:47 smithi120 ceph-mon[35250]: pgmap v553: 129 pgs: 129 active+clean; 3.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:30:47.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:47 smithi174 ceph-mon[35586]: pgmap v553: 129 pgs: 129 active+clean; 3.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:30:47.943 INFO:tasks.cephfs.filesystem:scrub status for tag:92a81101-87b2-4bd7-8904-c9e1fc2231e0 - {'path': '/', 'tag': '92a81101-87b2-4bd7-8904-c9e1fc2231e0', 'options': 'recursive,force'} 2024-09-15T23:30:48.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:48 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:30:48.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:48 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:30:48.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:48 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.031+0000 7f1624752640 1 -- 172.21.15.89:0/305733524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f161c102610 msgr2=0x7f161c108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.031+0000 7f1624752640 1 --2- 172.21.15.89:0/305733524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f161c102610 0x7f161c108ea0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f1614009ca0 tx=0x7f161402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.031+0000 7f1624752640 1 -- 172.21.15.89:0/305733524 shutdown_connections 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.031+0000 7f1624752640 1 --2- 172.21.15.89:0/305733524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f161c102610 0x7f161c108ea0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.031+0000 7f1624752640 1 --2- 172.21.15.89:0/305733524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c101c50 0x7f161c1020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.031+0000 7f1624752640 1 --2- 172.21.15.89:0/305733524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c100a60 0x7f161c100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.031+0000 7f1624752640 1 -- 172.21.15.89:0/305733524 >> 172.21.15.89:0/305733524 conn(0x7f161c0fc1e0 msgr2=0x7f161c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f1624752640 1 -- 172.21.15.89:0/305733524 shutdown_connections 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f1624752640 1 -- 172.21.15.89:0/305733524 wait complete. 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f1624752640 1 Processor -- start 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f1624752640 1 -- start start 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f1624752640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f161c100a60 0x7f161c196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f1624752640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c101c50 0x7f161c1971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f1624752640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c102610 0x7f161c19e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f1624752640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f161c10c6e0 con 0x7f161c100a60 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f1624752640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f161c10c560 con 0x7f161c101c50 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f1624752640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f161c10c860 con 0x7f161c102610 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f16224ee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f161c100a60 0x7f161c196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.032+0000 7f16224ee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f161c100a60 0x7f161c196ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47334/0 (socket says 172.21.15.89:47334) 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f16224ee640 1 -- 172.21.15.89:0/2902946291 learned_addr learned my addr 172.21.15.89:0/2902946291 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f16224ee640 1 -- 172.21.15.89:0/2902946291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c102610 msgr2=0x7f161c19e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f16224ee640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c102610 0x7f161c19e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f16224ee640 1 -- 172.21.15.89:0/2902946291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c101c50 msgr2=0x7f161c1971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f1621ced640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c101c50 0x7f161c1971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f16224ee640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c101c50 0x7f161c1971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f16224ee640 1 -- 172.21.15.89:0/2902946291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f161c19e8e0 con 0x7f161c100a60 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f16224ee640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f161c100a60 0x7f161c196ca0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f1618009850 tx=0x7f1618009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f16137fe640 1 -- 172.21.15.89:0/2902946291 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161801d070 con 0x7f161c100a60 2024-09-15T23:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f16137fe640 1 -- 172.21.15.89:0/2902946291 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1618007c80 con 0x7f161c100a60 2024-09-15T23:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f1624752640 1 -- 172.21.15.89:0/2902946291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f161c06a360 con 0x7f161c100a60 2024-09-15T23:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f16137fe640 1 -- 172.21.15.89:0/2902946291 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1618016320 con 0x7f161c100a60 2024-09-15T23:30:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.033+0000 7f1624752640 1 -- 172.21.15.89:0/2902946291 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f161c06a860 con 0x7f161c100a60 2024-09-15T23:30:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.036+0000 7f16137fe640 1 -- 172.21.15.89:0/2902946291 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1618020070 con 0x7f161c100a60 2024-09-15T23:30:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.036+0000 7f16275ca740 1 Processor -- start 2024-09-15T23:30:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.036+0000 7f16275ca740 1 -- start start 2024-09-15T23:30:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.036+0000 7f16275ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 0x559442ec9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.036+0000 7f16275ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442eceda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.036+0000 7f16275ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ed1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.036+0000 7f16275ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559442e2d2f0 con 0x559442ecf370 2024-09-15T23:30:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.036+0000 7f16275ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559442dd5270 con 0x559442ec99a0 2024-09-15T23:30:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.036+0000 7f16275ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559442e292b0 con 0x559442eca2e0 2024-09-15T23:30:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f16137fe640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a32f0 0x7f16080a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1621ced640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ed1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1621ced640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ed1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47338/0 (socket says 172.21.15.89:47338) 2024-09-15T23:30:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1621ced640 1 -- 172.21.15.89:0/2384805654 learned_addr learned my addr 172.21.15.89:0/2384805654 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f16224ee640 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442eceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1622cef640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a32f0 0x7f16080a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1622cef640 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 0x559442ec9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1621ced640 1 -- 172.21.15.89:0/2384805654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 msgr2=0x559442eceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f16137fe640 1 -- 172.21.15.89:0/2902946291 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f16180d1290 con 0x7f161c100a60 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1621ced640 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442eceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1621ced640 1 -- 172.21.15.89:0/2384805654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 msgr2=0x559442ec9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1621ced640 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 0x559442ec9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1621ced640 1 -- 172.21.15.89:0/2384805654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559442dd4340 con 0x559442ecf370 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1622cef640 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 0x559442ec9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f16224ee640 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442eceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1621ced640 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ed1830 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f160c00ba70 tx=0x7f160c00bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed602160cef964b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1610ff9640 1 -- 172.21.15.89:0/2384805654 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c017420 con 0x559442ecf370 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1622cef640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a32f0 0x7f16080a57b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1614009430 tx=0x7f1614004a60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1610ff9640 1 -- 172.21.15.89:0/2384805654 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f160c017aa0 con 0x559442ecf370 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.037+0000 7f1610ff9640 1 -- 172.21.15.89:0/2384805654 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c00faa0 con 0x559442ecf370 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.038+0000 7f16275ca740 1 -- 172.21.15.89:0/2384805654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 msgr2=0x559442ed1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.038+0000 7f16275ca740 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ed1830 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f160c00ba70 tx=0x7f160c00bf30 comp rx=0 tx=0).stop 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.038+0000 7f16275ca740 1 -- 172.21.15.89:0/2384805654 shutdown_connections 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.038+0000 7f16275ca740 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ed1830 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.038+0000 7f16275ca740 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442eceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.038+0000 7f16275ca740 1 --2- 172.21.15.89:0/2384805654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 0x559442ec9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.038+0000 7f16275ca740 1 -- 172.21.15.89:0/2384805654 >> 172.21.15.89:0/2384805654 conn(0x559442ec5010 msgr2=0x559442ec5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.038+0000 7f16275ca740 1 -- 172.21.15.89:0/2384805654 shutdown_connections 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.038+0000 7f16275ca740 1 -- 172.21.15.89:0/2384805654 wait complete. 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.038+0000 7f16275ca740 1 -- start start 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16275ca740 1 Processor -- start 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16275ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16275ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442eca2e0 0x559442ed3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16275ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442ecf370 0x559442edb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16275ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559442ed2340 con 0x559442ec99a0 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16275ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559442db6290 con 0x559442eca2e0 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16275ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559442ed24c0 con 0x559442ecf370 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16275ca740 10 client.0 _ll_register_callbacks cb 0x559442de7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16224ee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16224ee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47350/0 (socket says 172.21.15.89:47350) 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16224ee640 1 -- 172.21.15.89:0/2639530345 learned_addr learned my addr 172.21.15.89:0/2639530345 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f1621ced640 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442eca2e0 0x559442ed3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f1622cef640 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442ecf370 0x559442edb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16224ee640 1 -- 172.21.15.89:0/2639530345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442ecf370 msgr2=0x559442edb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16224ee640 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442ecf370 0x559442edb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16224ee640 1 -- 172.21.15.89:0/2639530345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442eca2e0 msgr2=0x559442ed3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16224ee640 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442eca2e0 0x559442ed3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16224ee640 1 -- 172.21.15.89:0/2639530345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559442b40210 con 0x559442ec99a0 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f1622cef640 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442ecf370 0x559442edb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f16224ee640 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed3ab0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f161809d1c0 tx=0x7f161801dec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a0a1ddf545740f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.039+0000 7f1605ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f1605ffb640 1 -- 172.21.15.89:0/2639530345 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16180cee60 con 0x559442ec99a0 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f1605ffb640 1 -- 172.21.15.89:0/2639530345 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f161801cbb0 con 0x559442ec99a0 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f16275ca740 1 -- 172.21.15.89:0/2639530345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f160c017aa0 con 0x559442ec99a0 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f1605ffb640 1 -- 172.21.15.89:0/2639530345 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f16180d10d0 con 0x559442ec99a0 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f16275ca740 10 client.17787 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f16275ca740 1 -- 172.21.15.89:0/2639530345 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559442c0c620 con 0x559442ec99a0 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f1605ffb640 1 -- 172.21.15.89:0/2639530345 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f160c017aa0 con 0x559442ec99a0 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f1605ffb640 1 -- 172.21.15.89:0/2639530345 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f16180b6240 con 0x559442ec99a0 2024-09-15T23:30:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f16275ca740 10 client.17787 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f16275ca740 10 client.17787 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f16275ca740 20 client.17787 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f16275ca740 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559442edf760 0x559442ee1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f16275ca740 4 client.17787 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.040+0000 7f16275ca740 1 -- 172.21.15.89:0/2639530345 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559442e3aa20 con 0x559442edf760 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.041+0000 7f1621ced640 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559442edf760 0x559442ee1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.041+0000 7f1621ced640 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559442edf760 0x559442ee1b50 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7d2400c19906765 server_cookie=6a56b7111cc2a22 in_seq=0 out_seq=0 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.041+0000 7f1605ffb640 10 client.17787 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f1605ffb640 1 -- 172.21.15.89:0/2639530345 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559442e3aa20 con 0x559442edf760 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f1605ffb640 10 client.17787 handle_command_reply: tid=0 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 client.17787 shutdown 2024-09-15T23:30:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 -- 172.21.15.89:0/2639530345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 msgr2=0x559442ed3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed3ab0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f161809d1c0 tx=0x7f161801dec0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 -- 172.21.15.89:0/2639530345 shutdown_connections 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559442edf760 0x559442ee1b50 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442ecf370 0x559442edb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442eca2e0 0x559442ed3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f1605ffb640 0 client.17787 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 --2- 172.21.15.89:0/2639530345 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed3ab0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 -- 172.21.15.89:0/2639530345 >> 172.21.15.89:0/2639530345 conn(0x559442ec5010 msgr2=0x559442ec6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 -- 172.21.15.89:0/2639530345 shutdown_connections 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 1 -- 172.21.15.89:0/2639530345 wait complete. 2024-09-15T23:30:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.042+0000 7f16275ca740 20 client.17787 trim_cache size 0 max 16384 2024-09-15T23:30:49.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.056+0000 7f16275ca740 1 Processor -- start 2024-09-15T23:30:49.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.056+0000 7f16275ca740 1 -- start start 2024-09-15T23:30:49.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.056+0000 7f16275ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 0x559442ed3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.056+0000 7f16275ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442edaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.056+0000 7f16275ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ec7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.056+0000 7f16275ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1a85f0 con 0x559442ecf370 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.056+0000 7f16275ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1aa130 con 0x559442ec99a0 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.056+0000 7f16275ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1a8cc0 con 0x559442eca2e0 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1622cef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ec7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1622cef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ec7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47358/0 (socket says 172.21.15.89:47358) 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1622cef640 1 -- 172.21.15.89:0/2718889463 learned_addr learned my addr 172.21.15.89:0/2718889463 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1621ced640 1 --2- 172.21.15.89:0/2718889463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442edaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f16224ee640 1 --2- 172.21.15.89:0/2718889463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 0x559442ed3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1622cef640 1 -- 172.21.15.89:0/2718889463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 msgr2=0x559442edaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1622cef640 1 --2- 172.21.15.89:0/2718889463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442edaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1622cef640 1 -- 172.21.15.89:0/2718889463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 msgr2=0x559442ed3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1622cef640 1 --2- 172.21.15.89:0/2718889463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 0x559442ed3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1622cef640 1 -- 172.21.15.89:0/2718889463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559442ed35f0 con 0x559442ecf370 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1622cef640 1 --2- 172.21.15.89:0/2718889463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ec7ed0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f1614035000 tx=0x7f161403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e45a8a045879a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1607fff640 1 -- 172.21.15.89:0/2718889463 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614031810 con 0x559442ecf370 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1607fff640 1 -- 172.21.15.89:0/2718889463 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1614036040 con 0x559442ecf370 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.057+0000 7f1607fff640 1 -- 172.21.15.89:0/2718889463 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1614030b90 con 0x559442ecf370 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 -- 172.21.15.89:0/2718889463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 msgr2=0x559442ec7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 --2- 172.21.15.89:0/2718889463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ec7ed0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f1614035000 tx=0x7f161403f000 comp rx=0 tx=0).stop 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 -- 172.21.15.89:0/2718889463 shutdown_connections 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 --2- 172.21.15.89:0/2718889463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ecf370 0x559442ec7ed0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 --2- 172.21.15.89:0/2718889463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442edaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 --2- 172.21.15.89:0/2718889463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ec99a0 0x559442ed3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 -- 172.21.15.89:0/2718889463 >> 172.21.15.89:0/2718889463 conn(0x559442ec5010 msgr2=0x559442ed3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 -- 172.21.15.89:0/2718889463 shutdown_connections 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 -- 172.21.15.89:0/2718889463 wait complete. 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 -- start start 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 Processor -- start 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442ed8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ecf370 0x559442ed9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.058+0000 7f16275ca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1a9a60 con 0x559442ec99a0 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16275ca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1a9390 con 0x559442ecf370 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16275ca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f161c1a65d0 con 0x559442eca2e0 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16275ca740 10 client.0 _ll_register_callbacks cb 0x7f1618005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16224ee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16224ee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47370/0 (socket says 172.21.15.89:47370) 2024-09-15T23:30:49.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16224ee640 1 -- 172.21.15.89:0/2558909795 learned_addr learned my addr 172.21.15.89:0/2558909795 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:30:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f1621ced640 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442ed8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f1622cef640 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ecf370 0x559442ed9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16224ee640 1 -- 172.21.15.89:0/2558909795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 msgr2=0x559442ed8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16224ee640 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442ed8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16224ee640 1 -- 172.21.15.89:0/2558909795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ecf370 msgr2=0x559442ed9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16224ee640 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ecf370 0x559442ed9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16224ee640 1 -- 172.21.15.89:0/2558909795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559442eda430 con 0x559442ec99a0 2024-09-15T23:30:49.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f1622cef640 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ecf370 0x559442ed9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16224ee640 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed8920 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f16180a1a30 tx=0x7f1618018750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3833f7a0da5f0b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f1605ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f1605ffb640 1 -- 172.21.15.89:0/2558909795 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16180bb060 con 0x559442ec99a0 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f1605ffb640 1 -- 172.21.15.89:0/2558909795 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f161809cb60 con 0x559442ec99a0 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16275ca740 1 -- 172.21.15.89:0/2558909795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1614036040 con 0x559442ec99a0 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f1605ffb640 1 -- 172.21.15.89:0/2558909795 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f16180cee60 con 0x559442ec99a0 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16275ca740 10 client.17793 fetch_fsmap learned FSMap version 29 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.059+0000 7f16275ca740 1 -- 172.21.15.89:0/2558909795 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559442f4a660 con 0x559442ec99a0 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f1605ffb640 1 -- 172.21.15.89:0/2558909795 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1614036040 con 0x559442ec99a0 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f1605ffb640 1 -- 172.21.15.89:0/2558909795 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1618068170 con 0x559442ec99a0 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f16275ca740 10 client.17793 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f16275ca740 10 client.17793 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:49.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f16275ca740 20 client.17793 populate_metadata read hostname 'smithi089' 2024-09-15T23:30:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f16275ca740 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559442f4b010 0x559442f4d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f16275ca740 4 client.17793 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:30:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f16275ca740 1 -- 172.21.15.89:0/2558909795 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559442e34d70 con 0x559442f4b010 2024-09-15T23:30:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f1621ced640 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559442f4b010 0x559442f4d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f1605ffb640 10 client.17793 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.060+0000 7f1621ced640 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559442f4b010 0x559442f4d430 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ec8d769cc4a35ee server_cookie=a9906f3e6cafdeb0 in_seq=0 out_seq=0 2024-09-15T23:30:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f1605ffb640 1 -- 172.21.15.89:0/2558909795 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559442e34d70 con 0x559442f4b010 2024-09-15T23:30:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f1605ffb640 10 client.17793 handle_command_reply: tid=0 2024-09-15T23:30:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f16275ca740 1 client.17793 shutdown 2024-09-15T23:30:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f16275ca740 1 -- 172.21.15.89:0/2558909795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 msgr2=0x559442ed8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f16275ca740 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed8920 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f16180a1a30 tx=0x7f1618018750 comp rx=0 tx=0).stop 2024-09-15T23:30:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f16275ca740 1 -- 172.21.15.89:0/2558909795 shutdown_connections 2024-09-15T23:30:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f16275ca740 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559442f4b010 0x559442f4d430 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f16275ca740 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559442ecf370 0x559442ed9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f16275ca740 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559442eca2e0 0x559442ed8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f16275ca740 1 --2- 172.21.15.89:0/2558909795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559442ec99a0 0x559442ed8920 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f1605ffb640 0 client.17793 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:30:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.073+0000 7f16275ca740 1 -- 172.21.15.89:0/2558909795 >> 172.21.15.89:0/2558909795 conn(0x559442ec5010 msgr2=0x559442ec39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f16275ca740 1 -- 172.21.15.89:0/2558909795 shutdown_connections 2024-09-15T23:30:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f16275ca740 1 -- 172.21.15.89:0/2558909795 wait complete. 2024-09-15T23:30:49.076 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f16275ca740 20 client.17793 trim_cache size 0 max 16384 2024-09-15T23:30:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:30:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:30:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:30:49.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 -- 172.21.15.89:0/2902946291 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a32f0 msgr2=0x7f16080a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a32f0 0x7f16080a57b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1614009430 tx=0x7f1614004a60 comp rx=0 tx=0).stop 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 -- 172.21.15.89:0/2902946291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f161c100a60 msgr2=0x7f161c196ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f161c100a60 0x7f161c196ca0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f1618009850 tx=0x7f1618009d10 comp rx=0 tx=0).stop 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 -- 172.21.15.89:0/2902946291 shutdown_connections 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16080a32f0 0x7f16080a57b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f161c102610 0x7f161c19e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f161c101c50 0x7f161c1971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 --2- 172.21.15.89:0/2902946291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f161c100a60 0x7f161c196ca0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 -- 172.21.15.89:0/2902946291 >> 172.21.15.89:0/2902946291 conn(0x7f161c0fc1e0 msgr2=0x7f161c104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 -- 172.21.15.89:0/2902946291 shutdown_connections 2024-09-15T23:30:49.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:30:49.074+0000 7f1624752640 1 -- 172.21.15.89:0/2902946291 wait complete. 2024-09-15T23:30:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:49 smithi174 ceph-mon[35586]: pgmap v554: 129 pgs: 129 active+clean; 3.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:30:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:49 smithi089 ceph-mon[30933]: pgmap v554: 129 pgs: 129 active+clean; 3.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:30:49.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:49 smithi120 ceph-mon[35250]: pgmap v554: 129 pgs: 129 active+clean; 3.2 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:30:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:51 smithi120 ceph-mon[35250]: pgmap v555: 129 pgs: 129 active+clean; 3.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:30:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:51 smithi174 ceph-mon[35586]: pgmap v555: 129 pgs: 129 active+clean; 3.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:30:51.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:51 smithi089 ceph-mon[30933]: pgmap v555: 129 pgs: 129 active+clean; 3.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:30:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:53 smithi120 ceph-mon[35250]: pgmap v556: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:30:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:53 smithi174 ceph-mon[35586]: pgmap v556: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:30:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:53 smithi089 ceph-mon[30933]: pgmap v556: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:30:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:55 smithi120 ceph-mon[35250]: pgmap v557: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:30:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:55 smithi174 ceph-mon[35586]: pgmap v557: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:30:55.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:55 smithi089 ceph-mon[30933]: pgmap v557: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:30:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:57 smithi120 ceph-mon[35250]: pgmap v558: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:30:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:57 smithi174 ceph-mon[35586]: pgmap v558: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:30:57.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:57 smithi089 ceph-mon[30933]: pgmap v558: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:30:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:30:59 smithi120 ceph-mon[35250]: pgmap v559: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:30:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:30:59 smithi174 ceph-mon[35586]: pgmap v559: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:30:59.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:30:59 smithi089 ceph-mon[30933]: pgmap v559: 129 pgs: 129 active+clean; 3.3 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:31:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:31:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:31:00.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:31:00.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:31:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:31:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:31:00.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:31:00.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:31:00.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:31:00.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:31:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:31:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:31:00.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:31:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:01 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi120 to 176.0M 2024-09-15T23:31:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:01 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi089 to 175.9M 2024-09-15T23:31:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:01 smithi120 ceph-mon[35250]: Unable to set osd_memory_target on smithi120 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T23:31:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:01 smithi120 ceph-mon[35250]: Unable to set osd_memory_target on smithi089 to 184518656: error parsing value: Value '184518656' is below minimum 939524096 2024-09-15T23:31:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:01 smithi120 ceph-mon[35250]: pgmap v560: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:31:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:01 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi120 to 176.0M 2024-09-15T23:31:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:01 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi089 to 175.9M 2024-09-15T23:31:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:01 smithi174 ceph-mon[35586]: Unable to set osd_memory_target on smithi120 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T23:31:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:01 smithi174 ceph-mon[35586]: Unable to set osd_memory_target on smithi089 to 184518656: error parsing value: Value '184518656' is below minimum 939524096 2024-09-15T23:31:01.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:01 smithi174 ceph-mon[35586]: pgmap v560: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:31:01.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:01 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi120 to 176.0M 2024-09-15T23:31:01.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:01 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi089 to 175.9M 2024-09-15T23:31:01.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:01 smithi089 ceph-mon[30933]: Unable to set osd_memory_target on smithi120 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T23:31:01.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:01 smithi089 ceph-mon[30933]: Unable to set osd_memory_target on smithi089 to 184518656: error parsing value: Value '184518656' is below minimum 939524096 2024-09-15T23:31:01.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:01 smithi089 ceph-mon[30933]: pgmap v560: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:31:02.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.171+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1220658283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9c1025f0 msgr2=0x7f2e9c108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:02.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.171+0000 7f2ea0cc9640 1 --2- 172.21.15.89:0/1220658283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9c1025f0 0x7f2e9c108e80 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009ca0 tx=0x7f2e9402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.171+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1220658283 shutdown_connections 2024-09-15T23:31:02.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.171+0000 7f2ea0cc9640 1 --2- 172.21.15.89:0/1220658283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9c1025f0 0x7f2e9c108e80 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.171+0000 7f2ea0cc9640 1 --2- 172.21.15.89:0/1220658283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c101c30 0x7f2e9c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.171+0000 7f2ea0cc9640 1 --2- 172.21.15.89:0/1220658283 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e9c100a40 0x7f2e9c100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.171+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1220658283 >> 172.21.15.89:0/1220658283 conn(0x7f2e9c0fc1e0 msgr2=0x7f2e9c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:02.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.171+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1220658283 shutdown_connections 2024-09-15T23:31:02.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.171+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1220658283 wait complete. 2024-09-15T23:31:02.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2ea0cc9640 1 Processor -- start 2024-09-15T23:31:02.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2ea0cc9640 1 -- start start 2024-09-15T23:31:02.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2ea0cc9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e9c100a40 0x7f2e9c06a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2ea0cc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c101c30 0x7f2e9c19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2ea0cc9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9c1025f0 0x7f2e9c19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2ea0cc9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c10c6c0 con 0x7f2e9c1025f0 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2ea0cc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c10c540 con 0x7f2e9c101c30 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2ea0cc9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c10c840 con 0x7f2e9c100a40 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2e9ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9c1025f0 0x7f2e9c19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2e9ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9c1025f0 0x7f2e9c19f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42784/0 (socket says 172.21.15.89:42784) 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2e9ad9d640 1 -- 172.21.15.89:0/1514062752 learned_addr learned my addr 172.21.15.89:0/1514062752 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c101c30 0x7f2e9c19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e9c100a40 0x7f2e9c06a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2e9ad9d640 1 -- 172.21.15.89:0/1514062752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e9c100a40 msgr2=0x7f2e9c06a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2e9ad9d640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e9c100a40 0x7f2e9c06a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.172+0000 7f2e9ad9d640 1 -- 172.21.15.89:0/1514062752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c101c30 msgr2=0x7f2e9c19cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.173+0000 7f2e9ad9d640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c101c30 0x7f2e9c19cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.173+0000 7f2e9ad9d640 1 -- 172.21.15.89:0/1514062752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9c19fa10 con 0x7f2e9c1025f0 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.173+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c101c30 0x7f2e9c19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:31:02.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.173+0000 7f2e9ad9d640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9c1025f0 0x7f2e9c19f100 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f2e94031ae0 tx=0x7f2e94031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:02.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.173+0000 7f2e8b7fe640 1 -- 172.21.15.89:0/1514062752 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94003b70 con 0x7f2e9c1025f0 2024-09-15T23:31:02.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.173+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1514062752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e9c19fc40 con 0x7f2e9c1025f0 2024-09-15T23:31:02.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.173+0000 7f2e8b7fe640 1 -- 172.21.15.89:0/1514062752 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e94003d10 con 0x7f2e9c1025f0 2024-09-15T23:31:02.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.173+0000 7f2e8b7fe640 1 -- 172.21.15.89:0/1514062752 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9402fc40 con 0x7f2e9c1025f0 2024-09-15T23:31:02.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.173+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1514062752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e9c1a0190 con 0x7f2e9c1025f0 2024-09-15T23:31:02.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.175+0000 7f2e8b7fe640 1 -- 172.21.15.89:0/1514062752 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2e94004050 con 0x7f2e9c1025f0 2024-09-15T23:31:02.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2ea3a61740 1 Processor -- start 2024-09-15T23:31:02.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2ea3a61740 1 -- start start 2024-09-15T23:31:02.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402cf780 0x559e402d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559e402292f0 con 0x559e402ca6f0 2024-09-15T23:31:02.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559e401d1270 con 0x559e402cf780 2024-09-15T23:31:02.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559e402252b0 con 0x559e402c9db0 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2e9a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2e9a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42798/0 (socket says 172.21.15.89:42798) 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2e9a59c640 1 -- 172.21.15.89:0/3171973174 learned_addr learned my addr 172.21.15.89:0/3171973174 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2e8b7fe640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2e840a32f0 0x7f2e840a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.176+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/3171973174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402cf780 0x559e402d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/3171973174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e9ad9d640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2e840a32f0 0x7f2e840a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e9a59c640 1 -- 172.21.15.89:0/3171973174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 msgr2=0x559e402ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/3171973174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e9a59c640 1 -- 172.21.15.89:0/3171973174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402cf780 msgr2=0x559e402d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/3171973174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402cf780 0x559e402d1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e9a59c640 1 -- 172.21.15.89:0/3171973174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e401d0340 con 0x559e402ca6f0 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e8b7fe640 1 -- 172.21.15.89:0/1514062752 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2e940f0e30 con 0x7f2e9c1025f0 2024-09-15T23:31:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/3171973174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402cf1b0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009c20 tx=0x7f2e8c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a6402cf2e96532 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e88ff9640 1 -- 172.21.15.89:0/3171973174 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c012450 con 0x559e402ca6f0 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e88ff9640 1 -- 172.21.15.89:0/3171973174 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e8c012ad0 con 0x559e402ca6f0 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.177+0000 7f2e88ff9640 1 -- 172.21.15.89:0/3171973174 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c00fbc0 con 0x559e402ca6f0 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3171973174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 msgr2=0x559e402cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2e9ad9d640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2e840a32f0 0x7f2e840a57b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2e940b9b50 tx=0x7f2e9408df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/3171973174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402cf1b0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009c20 tx=0x7f2e8c0117b0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3171973174 shutdown_connections 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/3171973174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402cf780 0x559e402d1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/3171973174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402cf1b0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/3171973174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3171973174 >> 172.21.15.89:0/3171973174 conn(0x559e402c1010 msgr2=0x559e402c1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3171973174 shutdown_connections 2024-09-15T23:31:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3171973174 wait complete. 2024-09-15T23:31:02.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 -- start start 2024-09-15T23:31:02.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.178+0000 7f2ea3a61740 1 Processor -- start 2024-09-15T23:31:02.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402c9db0 0x559e402d3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 0x559e402d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402cf780 0x559e402db4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559e402d2750 con 0x559e402c9db0 2024-09-15T23:31:02.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559e401b2290 con 0x559e402ca6f0 2024-09-15T23:31:02.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559e402d28d0 con 0x559e402cf780 2024-09-15T23:31:02.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2ea3a61740 10 client.0 _ll_register_callbacks cb 0x559e401e3140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:02.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e99d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402c9db0 0x559e402d3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e99d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402c9db0 0x559e402d3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42812/0 (socket says 172.21.15.89:42812) 2024-09-15T23:31:02.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e99d9b640 1 -- 172.21.15.89:0/3389199446 learned_addr learned my addr 172.21.15.89:0/3389199446 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402cf780 0x559e402db4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 0x559e402d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e99d9b640 1 -- 172.21.15.89:0/3389199446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402cf780 msgr2=0x559e402db4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402cf780 0x559e402db4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e99d9b640 1 -- 172.21.15.89:0/3389199446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 msgr2=0x559e402d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 0x559e402d44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e99d9b640 1 -- 172.21.15.89:0/3389199446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e3ff3c210 con 0x559e402c9db0 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 0x559e402d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402c9db0 0x559e402d3f70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000e500 tx=0x7f2e90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6584fb0667923f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e81ffb640 1 -- 172.21.15.89:0/3389199446 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90013110 con 0x559e402c9db0 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2e81ffb640 1 -- 172.21.15.89:0/3389199446 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2e900132b0 con 0x559e402c9db0 2024-09-15T23:31:02.190 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.179+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3389199446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e8c012ad0 con 0x559e402c9db0 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2e81ffb640 1 -- 172.21.15.89:0/3389199446 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2e90004030 con 0x559e402c9db0 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2ea3a61740 10 client.17805 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3389199446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559e40008620 con 0x559e402c9db0 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2e81ffb640 1 -- 172.21.15.89:0/3389199446 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e8c012ad0 con 0x559e402c9db0 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2e81ffb640 1 -- 172.21.15.89:0/3389199446 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2e900035f0 con 0x559e402c9db0 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2ea3a61740 10 client.17805 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2ea3a61740 10 client.17805 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2ea3a61740 20 client.17805 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e402dfc60 0x559e402e20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2ea3a61740 4 client.17805 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3389199446 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e40236a20 con 0x559e402dfc60 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.180+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e402dfc60 0x559e402e20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.181+0000 7f2e81ffb640 10 client.17805 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:02.191 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.181+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e402dfc60 0x559e402e20b0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c809566419eb8295 server_cookie=bed822444405c640 in_seq=0 out_seq=0 2024-09-15T23:31:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.181+0000 7f2e81ffb640 1 -- 172.21.15.89:0/3389199446 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559e40236a20 con 0x559e402dfc60 2024-09-15T23:31:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.181+0000 7f2e81ffb640 10 client.17805 handle_command_reply: tid=0 2024-09-15T23:31:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 client.17805 shutdown 2024-09-15T23:31:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3389199446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402c9db0 msgr2=0x559e402d3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402c9db0 0x559e402d3f70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000e500 tx=0x7f2e90004730 comp rx=0 tx=0).stop 2024-09-15T23:31:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3389199446 shutdown_connections 2024-09-15T23:31:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e402dfc60 0x559e402e20b0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402cf780 0x559e402db4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 0x559e402d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/3389199446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402c9db0 0x559e402d3f70 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2e81ffb640 0 client.17805 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3389199446 >> 172.21.15.89:0/3389199446 conn(0x559e402c1010 msgr2=0x559e402c2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3389199446 shutdown_connections 2024-09-15T23:31:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 1 -- 172.21.15.89:0/3389199446 wait complete. 2024-09-15T23:31:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.182+0000 7f2ea3a61740 20 client.17805 trim_cache size 0 max 16384 2024-09-15T23:31:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.196+0000 7f2ea3a61740 1 Processor -- start 2024-09-15T23:31:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.196+0000 7f2ea3a61740 1 -- start start 2024-09-15T23:31:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.196+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.196+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402db4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.196+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402cf780 0x559e402c2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.196+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c1a8660 con 0x559e402ca6f0 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.196+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c1aa1a0 con 0x559e402cf780 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.196+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c1a8d30 con 0x559e402c9db0 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e9a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402db4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e9a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402db4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42818/0 (socket says 172.21.15.89:42818) 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e9a59c640 1 -- 172.21.15.89:0/94442657 learned_addr learned my addr 172.21.15.89:0/94442657 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/94442657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e9a59c640 1 -- 172.21.15.89:0/94442657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 msgr2=0x559e402c1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/94442657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402c1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e9a59c640 1 -- 172.21.15.89:0/94442657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402cf780 msgr2=0x559e402c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/94442657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402cf780 0x559e402c2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e9a59c640 1 -- 172.21.15.89:0/94442657 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e402d3ab0 con 0x559e402ca6f0 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/94442657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402db4b0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c01c040 tx=0x7f2e8c005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a923ad42a5fbd768 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e83fff640 1 -- 172.21.15.89:0/94442657 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c012d00 con 0x559e402ca6f0 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e83fff640 1 -- 172.21.15.89:0/94442657 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e8c005590 con 0x559e402ca6f0 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.197+0000 7f2e83fff640 1 -- 172.21.15.89:0/94442657 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c0035f0 con 0x559e402ca6f0 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 -- 172.21.15.89:0/94442657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 msgr2=0x559e402db4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/94442657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402db4b0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c01c040 tx=0x7f2e8c005040 comp rx=0 tx=0).stop 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 -- 172.21.15.89:0/94442657 shutdown_connections 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/94442657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402cf780 0x559e402c2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/94442657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402ca6f0 0x559e402db4b0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/94442657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402c1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 -- 172.21.15.89:0/94442657 >> 172.21.15.89:0/94442657 conn(0x559e402c1010 msgr2=0x559e402d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 -- 172.21.15.89:0/94442657 shutdown_connections 2024-09-15T23:31:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 -- 172.21.15.89:0/94442657 wait complete. 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 -- start start 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 Processor -- start 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402d8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 0x559e402d9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402cf780 0x559e402da420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c1a9ad0 con 0x559e402cf780 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c1a9400 con 0x559e402ca6f0 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c1a6640 con 0x559e402c9db0 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.198+0000 7f2ea3a61740 10 client.0 _ll_register_callbacks cb 0x7f2e9000e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e99d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402cf780 0x559e402da420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e99d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402cf780 0x559e402da420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42826/0 (socket says 172.21.15.89:42826) 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e99d9b640 1 -- 172.21.15.89:0/292235356 learned_addr learned my addr 172.21.15.89:0/292235356 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402d8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e99d9b640 1 -- 172.21.15.89:0/292235356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 msgr2=0x559e402d8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 0x559e402d9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402d8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e99d9b640 1 -- 172.21.15.89:0/292235356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 msgr2=0x559e402d9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 0x559e402d9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e99d9b640 1 -- 172.21.15.89:0/292235356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e402cf4e0 con 0x559e402cf780 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e99d9b640 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402cf780 0x559e402da420 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009fd0 tx=0x7f2e900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f71cb835209806b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e81ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e81ffb640 1 -- 172.21.15.89:0/292235356 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900035f0 con 0x559e402cf780 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2ea3a61740 1 -- 172.21.15.89:0/292235356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e8c005590 con 0x559e402cf780 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e81ffb640 1 -- 172.21.15.89:0/292235356 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2e90003790 con 0x559e402cf780 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e81ffb640 1 -- 172.21.15.89:0/292235356 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2e90013110 con 0x559e402cf780 2024-09-15T23:31:02.205 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2ea3a61740 10 client.17811 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2ea3a61740 1 -- 172.21.15.89:0/292235356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559e4034ab80 con 0x559e402cf780 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.199+0000 7f2e81ffb640 1 -- 172.21.15.89:0/292235356 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e8c005590 con 0x559e402cf780 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.200+0000 7f2e81ffb640 1 -- 172.21.15.89:0/292235356 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2e900041d0 con 0x559e402cf780 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.200+0000 7f2ea3a61740 10 client.17811 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.200+0000 7f2ea3a61740 10 client.17811 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.200+0000 7f2ea3a61740 20 client.17811 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.200+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4034b530 0x559e4034d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.200+0000 7f2ea3a61740 4 client.17811 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.200+0000 7f2ea3a61740 1 -- 172.21.15.89:0/292235356 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559e40230d70 con 0x559e4034b530 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.200+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4034b530 0x559e4034d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.200+0000 7f2e81ffb640 10 client.17811 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:02.206 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.200+0000 7f2e9a59c640 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4034b530 0x559e4034d950 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4dc3619d75a6cc2 server_cookie=fe4539db6ca1e2e7 in_seq=0 out_seq=0 2024-09-15T23:31:02.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2e81ffb640 1 -- 172.21.15.89:0/292235356 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559e40230d70 con 0x559e4034b530 2024-09-15T23:31:02.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2e81ffb640 10 client.17811 handle_command_reply: tid=0 2024-09-15T23:31:02.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 client.17811 shutdown 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 -- 172.21.15.89:0/292235356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402cf780 msgr2=0x559e402da420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402cf780 0x559e402da420 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009fd0 tx=0x7f2e900046c0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 -- 172.21.15.89:0/292235356 shutdown_connections 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4034b530 0x559e4034d950 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e402cf780 0x559e402da420 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e402ca6f0 0x559e402d9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 --2- 172.21.15.89:0/292235356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e402c9db0 0x559e402d8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2e81ffb640 0 client.17811 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 -- 172.21.15.89:0/292235356 >> 172.21.15.89:0/292235356 conn(0x559e402c1010 msgr2=0x559e402bf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 -- 172.21.15.89:0/292235356 shutdown_connections 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 1 -- 172.21.15.89:0/292235356 wait complete. 2024-09-15T23:31:02.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.216+0000 7f2ea3a61740 20 client.17811 trim_cache size 0 max 16384 2024-09-15T23:31:02.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:31:02.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:31:02.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:31:02.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:31:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1514062752 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2e840a32f0 msgr2=0x7f2e840a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2e840a32f0 0x7f2e840a57b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2e940b9b50 tx=0x7f2e9408df90 comp rx=0 tx=0).stop 2024-09-15T23:31:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1514062752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9c1025f0 msgr2=0x7f2e9c19f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9c1025f0 0x7f2e9c19f100 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f2e94031ae0 tx=0x7f2e94031ee0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1514062752 shutdown_connections 2024-09-15T23:31:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2e840a32f0 0x7f2e840a57b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e9c1025f0 0x7f2e9c19f100 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2e9c101c30 0x7f2e9c19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 --2- 172.21.15.89:0/1514062752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e9c100a40 0x7f2e9c06a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1514062752 >> 172.21.15.89:0/1514062752 conn(0x7f2e9c0fc1e0 msgr2=0x7f2e9c104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:02.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1514062752 shutdown_connections 2024-09-15T23:31:02.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:02.217+0000 7f2ea0cc9640 1 -- 172.21.15.89:0/1514062752 wait complete. 2024-09-15T23:31:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:03 smithi120 ceph-mon[35250]: pgmap v561: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:31:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:03 smithi174 ceph-mon[35586]: pgmap v561: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:31:03.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:03 smithi089 ceph-mon[30933]: pgmap v561: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:31:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:05 smithi120 ceph-mon[35250]: pgmap v562: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:31:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:05 smithi174 ceph-mon[35586]: pgmap v562: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:31:05.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:05 smithi089 ceph-mon[30933]: pgmap v562: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:31:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:07 smithi120 ceph-mon[35250]: pgmap v563: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:07 smithi174 ceph-mon[35586]: pgmap v563: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:07.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:07 smithi089 ceph-mon[30933]: pgmap v563: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:09 smithi120 ceph-mon[35250]: pgmap v564: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:31:09.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:09 smithi174 ceph-mon[35586]: pgmap v564: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:31:09.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:09 smithi089 ceph-mon[30933]: pgmap v564: 129 pgs: 129 active+clean; 3.4 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:31:11.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:11 smithi120 ceph-mon[35250]: pgmap v565: 129 pgs: 129 active+clean; 3.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:11 smithi174 ceph-mon[35586]: pgmap v565: 129 pgs: 129 active+clean; 3.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:11.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:11 smithi089 ceph-mon[30933]: pgmap v565: 129 pgs: 129 active+clean; 3.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:13 smithi120 ceph-mon[35250]: pgmap v566: 129 pgs: 129 active+clean; 3.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:13 smithi174 ceph-mon[35586]: pgmap v566: 129 pgs: 129 active+clean; 3.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:13.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:13 smithi089 ceph-mon[30933]: pgmap v566: 129 pgs: 129 active+clean; 3.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:15.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.305+0000 7f0b42ff6640 1 -- 172.21.15.89:0/4218710466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3c104f60 msgr2=0x7f0b3c107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.305+0000 7f0b42ff6640 1 --2- 172.21.15.89:0/4218710466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3c104f60 0x7f0b3c107350 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009900 tx=0x7f0b3002a410 comp rx=0 tx=0).stop 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.305+0000 7f0b42ff6640 1 -- 172.21.15.89:0/4218710466 shutdown_connections 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 --2- 172.21.15.89:0/4218710466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b3c107890 0x7f0b3c109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 --2- 172.21.15.89:0/4218710466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3c104f60 0x7f0b3c107350 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 --2- 172.21.15.89:0/4218710466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b3c102630 0x7f0b3c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 -- 172.21.15.89:0/4218710466 >> 172.21.15.89:0/4218710466 conn(0x7f0b3c0fc1b0 msgr2=0x7f0b3c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 -- 172.21.15.89:0/4218710466 shutdown_connections 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 -- 172.21.15.89:0/4218710466 wait complete. 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 Processor -- start 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 -- start start 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b3c102630 0x7f0b3c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b3c104f60 0x7f0b3c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3c107890 0x7f0b3c19f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c10d810 con 0x7f0b3c107890 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c10d690 con 0x7f0b3c104f60 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b42ff6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c10d990 con 0x7f0b3c102630 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b41593640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3c107890 0x7f0b3c19f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b41593640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3c107890 0x7f0b3c19f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58984/0 (socket says 172.21.15.89:58984) 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.306+0000 7f0b41593640 1 -- 172.21.15.89:0/1904962918 learned_addr learned my addr 172.21.15.89:0/1904962918 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b41593640 1 -- 172.21.15.89:0/1904962918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b3c102630 msgr2=0x7f0b3c197e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b41593640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b3c102630 0x7f0b3c197e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b41593640 1 -- 172.21.15.89:0/1904962918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b3c104f60 msgr2=0x7f0b3c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b41593640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b3c104f60 0x7f0b3c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b41593640 1 -- 172.21.15.89:0/1904962918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3c19fab0 con 0x7f0b3c107890 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b41593640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3c107890 0x7f0b3c19f440 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400e9d0 tx=0x7f0b3400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b39ffb640 1 -- 172.21.15.89:0/1904962918 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3400cd80 con 0x7f0b3c107890 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b39ffb640 1 -- 172.21.15.89:0/1904962918 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b34005020 con 0x7f0b3c107890 2024-09-15T23:31:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b42ff6640 1 -- 172.21.15.89:0/1904962918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b3c19fda0 con 0x7f0b3c107890 2024-09-15T23:31:15.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b39ffb640 1 -- 172.21.15.89:0/1904962918 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b340116d0 con 0x7f0b3c107890 2024-09-15T23:31:15.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.307+0000 7f0b42ff6640 1 -- 172.21.15.89:0/1904962918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b3c1a0270 con 0x7f0b3c107890 2024-09-15T23:31:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.310+0000 7f0b39ffb640 1 -- 172.21.15.89:0/1904962918 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0b340051d0 con 0x7f0b3c107890 2024-09-15T23:31:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.310+0000 7f0b45e69740 1 Processor -- start 2024-09-15T23:31:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.310+0000 7f0b45e69740 1 -- start start 2024-09-15T23:31:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.310+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882229a0 0x55fe88222da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.310+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88227da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.310+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe88228370 0x55fe8822a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.310+0000 7f0b45e69740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fe881862f0 con 0x55fe882232e0 2024-09-15T23:31:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.310+0000 7f0b45e69740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe8812e270 con 0x55fe882229a0 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.310+0000 7f0b45e69740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fe881822b0 con 0x55fe88228370 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.310+0000 7f0b40d92640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88227da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b39ffb640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b240a32f0 0x7f0b240a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b40d92640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88227da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58994/0 (socket says 172.21.15.89:58994) 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b40d92640 1 -- 172.21.15.89:0/1014499265 learned_addr learned my addr 172.21.15.89:0/1014499265 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b41593640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b240a32f0 0x7f0b240a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/1014499265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882229a0 0x55fe88222da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b39ffb640 1 -- 172.21.15.89:0/1904962918 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0b340d03b0 con 0x7f0b3c107890 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/1014499265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe88228370 0x55fe8822a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b40d92640 1 -- 172.21.15.89:0/1014499265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe88228370 msgr2=0x55fe8822a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b40d92640 1 --2- 172.21.15.89:0/1014499265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe88228370 0x55fe8822a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b40d92640 1 -- 172.21.15.89:0/1014499265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882229a0 msgr2=0x55fe88222da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b40d92640 1 --2- 172.21.15.89:0/1014499265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882229a0 0x55fe88222da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b40d92640 1 -- 172.21.15.89:0/1014499265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe8812d340 con 0x55fe882232e0 2024-09-15T23:31:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/1014499265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe88228370 0x55fe8822a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b40d92640 1 --2- 172.21.15.89:0/1014499265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88227da0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00ba80 tx=0x7f0b2c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb13e727e30daa5c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b41593640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b240a32f0 0x7f0b240a57b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009fd0 tx=0x7f0b34066f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b22ffd640 1 -- 172.21.15.89:0/1014499265 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c016070 con 0x55fe882232e0 2024-09-15T23:31:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b22ffd640 1 -- 172.21.15.89:0/1014499265 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b2c005500 con 0x55fe882232e0 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.311+0000 7f0b22ffd640 1 -- 172.21.15.89:0/1014499265 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c010460 con 0x55fe882232e0 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 -- 172.21.15.89:0/1014499265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 msgr2=0x55fe88227da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 --2- 172.21.15.89:0/1014499265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88227da0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00ba80 tx=0x7f0b2c00bf40 comp rx=0 tx=0).stop 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 -- 172.21.15.89:0/1014499265 shutdown_connections 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 --2- 172.21.15.89:0/1014499265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe88228370 0x55fe8822a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 --2- 172.21.15.89:0/1014499265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88227da0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 --2- 172.21.15.89:0/1014499265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882229a0 0x55fe88222da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 -- 172.21.15.89:0/1014499265 >> 172.21.15.89:0/1014499265 conn(0x55fe8821e010 msgr2=0x55fe8821e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 -- 172.21.15.89:0/1014499265 shutdown_connections 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 -- 172.21.15.89:0/1014499265 wait complete. 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 -- start start 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.312+0000 7f0b45e69740 1 Processor -- start 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 0x55fe8822cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882232e0 0x55fe8822d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe88228370 0x55fe882340e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b45e69740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fe8822b340 con 0x55fe88228370 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b45e69740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe8810f290 con 0x55fe882232e0 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b45e69740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fe8822b4c0 con 0x55fe882229a0 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b45e69740 10 client.0 _ll_register_callbacks cb 0x55fe88140140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b3bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe88228370 0x55fe882340e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b3bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe88228370 0x55fe882340e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59000/0 (socket says 172.21.15.89:59000) 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b3bfff640 1 -- 172.21.15.89:0/898667977 learned_addr learned my addr 172.21.15.89:0/898667977 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 0x55fe8822cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b40d92640 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882232e0 0x55fe8822d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b3bfff640 1 -- 172.21.15.89:0/898667977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 msgr2=0x55fe8822cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 0x55fe8822cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b3bfff640 1 -- 172.21.15.89:0/898667977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882232e0 msgr2=0x55fe8822d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882232e0 0x55fe8822d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b3bfff640 1 -- 172.21.15.89:0/898667977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe87e99210 con 0x55fe88228370 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b40d92640 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882232e0 0x55fe8822d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe88228370 0x55fe882340e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0b300057c0 tx=0x7f0b30005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27e05d3f56b9d80 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b1bfff640 1 -- 172.21.15.89:0/898667977 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30044070 con 0x55fe88228370 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b1bfff640 1 -- 172.21.15.89:0/898667977 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0b30004220 con 0x55fe88228370 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.313+0000 7f0b45e69740 1 -- 172.21.15.89:0/898667977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b2c005500 con 0x55fe88228370 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b1bfff640 1 -- 172.21.15.89:0/898667977 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0b30050050 con 0x55fe88228370 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b45e69740 10 client.17823 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b45e69740 1 -- 172.21.15.89:0/898667977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fe87f65620 con 0x55fe88228370 2024-09-15T23:31:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b1bfff640 1 -- 172.21.15.89:0/898667977 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b2c005500 con 0x55fe88228370 2024-09-15T23:31:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b1bfff640 1 -- 172.21.15.89:0/898667977 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0b3003a040 con 0x55fe88228370 2024-09-15T23:31:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b45e69740 10 client.17823 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b45e69740 10 client.17823 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b45e69740 20 client.17823 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b45e69740 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fe88238740 0x55fe8823abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b45e69740 4 client.17823 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b45e69740 1 -- 172.21.15.89:0/898667977 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe88193a20 con 0x55fe88238740 2024-09-15T23:31:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.314+0000 7f0b40d92640 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fe88238740 0x55fe8823abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.315+0000 7f0b1bfff640 10 client.17823 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:15.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.315+0000 7f0b40d92640 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fe88238740 0x55fe8823abb0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdce87318ce47a57 server_cookie=8b8905737368088f in_seq=0 out_seq=0 2024-09-15T23:31:15.323 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.315+0000 7f0b1bfff640 1 -- 172.21.15.89:0/898667977 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe88193a20 con 0x55fe88238740 2024-09-15T23:31:15.323 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.315+0000 7f0b1bfff640 10 client.17823 handle_command_reply: tid=0 2024-09-15T23:31:15.323 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.315+0000 7f0b45e69740 1 client.17823 shutdown 2024-09-15T23:31:15.323 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 1 -- 172.21.15.89:0/898667977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe88228370 msgr2=0x55fe882340e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe88228370 0x55fe882340e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0b300057c0 tx=0x7f0b30005fd0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 1 -- 172.21.15.89:0/898667977 shutdown_connections 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fe88238740 0x55fe8823abb0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe88228370 0x55fe882340e0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b1bfff640 0 client.17823 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882232e0 0x55fe8822d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 1 --2- 172.21.15.89:0/898667977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 0x55fe8822cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 1 -- 172.21.15.89:0/898667977 >> 172.21.15.89:0/898667977 conn(0x55fe8821e010 msgr2=0x55fe8821f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 1 -- 172.21.15.89:0/898667977 shutdown_connections 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 1 -- 172.21.15.89:0/898667977 wait complete. 2024-09-15T23:31:15.324 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.316+0000 7f0b45e69740 20 client.17823 trim_cache size 0 max 16384 2024-09-15T23:31:15.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b45e69740 1 Processor -- start 2024-09-15T23:31:15.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b45e69740 1 -- start start 2024-09-15T23:31:15.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882229a0 0x55fe8821eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe882340a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe88228370 0x55fe8821f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b45e69740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c1aa8f0 con 0x55fe882232e0 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b45e69740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c1ac430 con 0x55fe882229a0 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b45e69740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c1aafc0 con 0x55fe88228370 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b40d92640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe882340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b40d92640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe882340a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59012/0 (socket says 172.21.15.89:59012) 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b40d92640 1 -- 172.21.15.89:0/3850946375 learned_addr learned my addr 172.21.15.89:0/3850946375 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/3850946375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882229a0 0x55fe8821eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b40d92640 1 -- 172.21.15.89:0/3850946375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe88228370 msgr2=0x55fe8821f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b40d92640 1 --2- 172.21.15.89:0/3850946375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe88228370 0x55fe8821f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b40d92640 1 -- 172.21.15.89:0/3850946375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882229a0 msgr2=0x55fe8821eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b40d92640 1 --2- 172.21.15.89:0/3850946375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882229a0 0x55fe8821eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b40d92640 1 -- 172.21.15.89:0/3850946375 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe8822b580 con 0x55fe882232e0 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.330+0000 7f0b40d92640 1 --2- 172.21.15.89:0/3850946375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe882340a0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00ba80 tx=0x7f0b2c0056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3cbbedb3bae374 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b227fc640 1 -- 172.21.15.89:0/3850946375 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c028070 con 0x55fe882232e0 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b227fc640 1 -- 172.21.15.89:0/3850946375 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b2c00dca0 con 0x55fe882232e0 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b227fc640 1 -- 172.21.15.89:0/3850946375 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c016040 con 0x55fe882232e0 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b45e69740 1 -- 172.21.15.89:0/3850946375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 msgr2=0x55fe882340a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b45e69740 1 --2- 172.21.15.89:0/3850946375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe882340a0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00ba80 tx=0x7f0b2c0056b0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b45e69740 1 -- 172.21.15.89:0/3850946375 shutdown_connections 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b45e69740 1 --2- 172.21.15.89:0/3850946375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe88228370 0x55fe8821f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b45e69740 1 --2- 172.21.15.89:0/3850946375 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe882340a0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b45e69740 1 --2- 172.21.15.89:0/3850946375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe882229a0 0x55fe8821eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b45e69740 1 -- 172.21.15.89:0/3850946375 >> 172.21.15.89:0/3850946375 conn(0x55fe8821e010 msgr2=0x55fe8822cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b45e69740 1 -- 172.21.15.89:0/3850946375 shutdown_connections 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b45e69740 1 -- 172.21.15.89:0/3850946375 wait complete. 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.331+0000 7f0b45e69740 1 -- start start 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b45e69740 1 Processor -- start 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 0x55fe882312c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88231800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b45e69740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe88228370 0x55fe88232810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b45e69740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c1abd60 con 0x55fe882232e0 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b45e69740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c1ab690 con 0x55fe88228370 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b45e69740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c1a88d0 con 0x55fe882229a0 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b45e69740 10 client.0 _ll_register_callbacks cb 0x7f0b30009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b40d92640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88231800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b40d92640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88231800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59020/0 (socket says 172.21.15.89:59020) 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b40d92640 1 -- 172.21.15.89:0/795441660 learned_addr learned my addr 172.21.15.89:0/795441660 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe88228370 0x55fe88232810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 0x55fe882312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b40d92640 1 -- 172.21.15.89:0/795441660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 msgr2=0x55fe882312c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b40d92640 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 0x55fe882312c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b40d92640 1 -- 172.21.15.89:0/795441660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe88228370 msgr2=0x55fe88232810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b40d92640 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe88228370 0x55fe88232810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b40d92640 1 -- 172.21.15.89:0/795441660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe88232e20 con 0x55fe882232e0 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.332+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 0x55fe882312c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b40d92640 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88231800 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00ba50 tx=0x7f0b2c005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f6a9fe904e202a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b1bfff640 1 -- 172.21.15.89:0/795441660 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c028040 con 0x55fe882232e0 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b1bfff640 1 -- 172.21.15.89:0/795441660 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0b2c004b20 con 0x55fe882232e0 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b45e69740 1 -- 172.21.15.89:0/795441660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b2c00dca0 con 0x55fe882232e0 2024-09-15T23:31:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b1bfff640 1 -- 172.21.15.89:0/795441660 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0b2c03a050 con 0x55fe882232e0 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b45e69740 10 client.17829 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b45e69740 1 -- 172.21.15.89:0/795441660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fe88234130 con 0x55fe882232e0 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b1bfff640 1 -- 172.21.15.89:0/795441660 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b2c00dca0 con 0x55fe882232e0 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b1bfff640 1 -- 172.21.15.89:0/795441660 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0b2c02b070 con 0x55fe882232e0 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b45e69740 10 client.17829 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b45e69740 10 client.17829 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b45e69740 20 client.17829 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b45e69740 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fe88238740 0x55fe882a5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b45e69740 4 client.17829 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b45e69740 1 -- 172.21.15.89:0/795441660 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fe8818dd70 con 0x55fe88238740 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.333+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fe88238740 0x55fe882a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.334+0000 7f0b3bfff640 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fe88238740 0x55fe882a5790 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a6e802d2f88b26b server_cookie=809ad4d167aac6df in_seq=0 out_seq=0 2024-09-15T23:31:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.334+0000 7f0b1bfff640 10 client.17829 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b1bfff640 1 -- 172.21.15.89:0/795441660 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fe8818dd70 con 0x55fe88238740 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b1bfff640 10 client.17829 handle_command_reply: tid=0 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 client.17829 shutdown 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 -- 172.21.15.89:0/795441660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 msgr2=0x55fe88231800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88231800 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00ba50 tx=0x7f0b2c005b00 comp rx=0 tx=0).stop 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 -- 172.21.15.89:0/795441660 shutdown_connections 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fe88238740 0x55fe882a5790 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe88228370 0x55fe88232810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe882232e0 0x55fe88231800 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b1bfff640 0 client.17829 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 --2- 172.21.15.89:0/795441660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fe882229a0 0x55fe882312c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 -- 172.21.15.89:0/795441660 >> 172.21.15.89:0/795441660 conn(0x55fe8821e010 msgr2=0x55fe8821c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 -- 172.21.15.89:0/795441660 shutdown_connections 2024-09-15T23:31:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 1 -- 172.21.15.89:0/795441660 wait complete. 2024-09-15T23:31:15.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.349+0000 7f0b45e69740 20 client.17829 trim_cache size 0 max 16384 2024-09-15T23:31:15.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:31:15.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:31:15.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:31:15.352 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:31:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 -- 172.21.15.89:0/1904962918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b240a32f0 msgr2=0x7f0b240a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b240a32f0 0x7f0b240a57b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009fd0 tx=0x7f0b34066f10 comp rx=0 tx=0).stop 2024-09-15T23:31:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 -- 172.21.15.89:0/1904962918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3c107890 msgr2=0x7f0b3c19f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3c107890 0x7f0b3c19f440 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400e9d0 tx=0x7f0b3400ee90 comp rx=0 tx=0).stop 2024-09-15T23:31:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 -- 172.21.15.89:0/1904962918 shutdown_connections 2024-09-15T23:31:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0b240a32f0 0x7f0b240a57b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0b3c107890 0x7f0b3c19f440 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0b3c104f60 0x7f0b3c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.353 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 --2- 172.21.15.89:0/1904962918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b3c102630 0x7f0b3c197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.354 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 -- 172.21.15.89:0/1904962918 >> 172.21.15.89:0/1904962918 conn(0x7f0b3c0fc1b0 msgr2=0x7f0b3c103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.354 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 -- 172.21.15.89:0/1904962918 shutdown_connections 2024-09-15T23:31:15.354 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:15.350+0000 7f0b42ff6640 1 -- 172.21.15.89:0/1904962918 wait complete. 2024-09-15T23:31:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:15 smithi120 ceph-mon[35250]: pgmap v567: 129 pgs: 129 active+clean; 3.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:31:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:15 smithi174 ceph-mon[35586]: pgmap v567: 129 pgs: 129 active+clean; 3.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:31:15.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:15 smithi089 ceph-mon[30933]: pgmap v567: 129 pgs: 129 active+clean; 3.5 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:31:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:17 smithi120 ceph-mon[35250]: pgmap v568: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 21 op/s 2024-09-15T23:31:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:17 smithi174 ceph-mon[35586]: pgmap v568: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 21 op/s 2024-09-15T23:31:17.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:17 smithi089 ceph-mon[30933]: pgmap v568: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 21 op/s 2024-09-15T23:31:17.945 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:31:18.206 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:18.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.710+0000 7fd66acd0640 1 -- 172.21.15.89:0/2594214350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd664102790 msgr2=0x7fd6641067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:18.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.710+0000 7fd66acd0640 1 --2- 172.21.15.89:0/2594214350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd664102790 0x7fd6641067e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fd654009ca0 tx=0x7fd65402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.710+0000 7fd66acd0640 1 -- 172.21.15.89:0/2594214350 shutdown_connections 2024-09-15T23:31:18.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.710+0000 7fd66acd0640 1 --2- 172.21.15.89:0/2594214350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd664106f10 0x7fd66410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.710+0000 7fd66acd0640 1 --2- 172.21.15.89:0/2594214350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd664102790 0x7fd6641067e0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.710+0000 7fd66acd0640 1 --2- 172.21.15.89:0/2594214350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd664101de0 0x7fd6641021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.710+0000 7fd66acd0640 1 -- 172.21.15.89:0/2594214350 >> 172.21.15.89:0/2594214350 conn(0x7fd6640fd8c0 msgr2=0x7fd6640ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:18.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.710+0000 7fd66acd0640 1 -- 172.21.15.89:0/2594214350 shutdown_connections 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.710+0000 7fd66acd0640 1 -- 172.21.15.89:0/2594214350 wait complete. 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66acd0640 1 Processor -- start 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66acd0640 1 -- start start 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66acd0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd664101de0 0x7fd664198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66acd0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd664102790 0x7fd6641988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66acd0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd664106f10 0x7fd66419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66acd0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd664111f70 con 0x7fd664106f10 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66acd0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd664111df0 con 0x7fd664101de0 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66acd0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd6641120f0 con 0x7fd664102790 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66926d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd664106f10 0x7fd66419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66926d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd664106f10 0x7fd66419cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59044/0 (socket says 172.21.15.89:59044) 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66926d640 1 -- 172.21.15.89:0/1090734284 learned_addr learned my addr 172.21.15.89:0/1090734284 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:18.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd663fff640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd664102790 0x7fd6641988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd668a6c640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd664101de0 0x7fd664198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66926d640 1 -- 172.21.15.89:0/1090734284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd664102790 msgr2=0x7fd6641988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66926d640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd664102790 0x7fd6641988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66926d640 1 -- 172.21.15.89:0/1090734284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd664101de0 msgr2=0x7fd664198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66926d640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd664101de0 0x7fd664198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.711+0000 7fd66926d640 1 -- 172.21.15.89:0/1090734284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66419d310 con 0x7fd664106f10 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.712+0000 7fd66926d640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd664106f10 0x7fd66419cc30 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00d750 tx=0x7fd64c00db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.712+0000 7fd661ffb640 1 -- 172.21.15.89:0/1090734284 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c017070 con 0x7fd664106f10 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.712+0000 7fd661ffb640 1 -- 172.21.15.89:0/1090734284 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd64c0097c0 con 0x7fd664106f10 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.712+0000 7fd66acd0640 1 -- 172.21.15.89:0/1090734284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66419d600 con 0x7fd664106f10 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.712+0000 7fd661ffb640 1 -- 172.21.15.89:0/1090734284 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c004d00 con 0x7fd664106f10 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.712+0000 7fd66acd0640 1 -- 172.21.15.89:0/1090734284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6641a4ee0 con 0x7fd664106f10 2024-09-15T23:31:18.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.714+0000 7fd661ffb640 1 -- 172.21.15.89:0/1090734284 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd64c009970 con 0x7fd664106f10 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.714+0000 7fd66d0a9740 1 Processor -- start 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.714+0000 7fd66d0a9740 1 -- start start 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d14a0 0x55a1d86d1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86dc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86dc680 0x55a1d86dea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a1d86372e0 con 0x55a1d86d1dc0 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a1d85dece0 con 0x55a1d86dc680 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a1d86332a0 con 0x55a1d86d14a0 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd668a6c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86dc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd668a6c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86dc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59060/0 (socket says 172.21.15.89:59060) 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd668a6c640 1 -- 172.21.15.89:0/571688502 learned_addr learned my addr 172.21.15.89:0/571688502 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:18.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd661ffb640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6500a3240 0x7fd6500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd663fff640 1 --2- 172.21.15.89:0/571688502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d14a0 0x55a1d86d1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd66926d640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6500a3240 0x7fd6500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd663fff640 1 --2- 172.21.15.89:0/571688502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86dc680 0x55a1d86dea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd668a6c640 1 -- 172.21.15.89:0/571688502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d14a0 msgr2=0x55a1d86d1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd668a6c640 1 --2- 172.21.15.89:0/571688502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d14a0 0x55a1d86d1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd668a6c640 1 -- 172.21.15.89:0/571688502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86dc680 msgr2=0x55a1d86dea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd668a6c640 1 --2- 172.21.15.89:0/571688502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86dc680 0x55a1d86dea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd668a6c640 1 -- 172.21.15.89:0/571688502 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1d85c3440 con 0x55a1d86d1dc0 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd663fff640 1 --2- 172.21.15.89:0/571688502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d14a0 0x55a1d86d1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd661ffb640 1 -- 172.21.15.89:0/1090734284 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd64c0cf5b0 con 0x7fd664106f10 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.715+0000 7fd668a6c640 1 --2- 172.21.15.89:0/571688502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86dc140 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00ad00 tx=0x7fd65c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f348337a453a5c98 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd66926d640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6500a3240 0x7fd6500a5700 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd64c097c40 tx=0x7fd64c0c65b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:18.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd65affd640 1 -- 172.21.15.89:0/571688502 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c01d070 con 0x55a1d86d1dc0 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd65affd640 1 -- 172.21.15.89:0/571688502 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c018b20 con 0x55a1d86d1dc0 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd65affd640 1 -- 172.21.15.89:0/571688502 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c010b80 con 0x55a1d86d1dc0 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd66d0a9740 1 -- 172.21.15.89:0/571688502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 msgr2=0x55a1d86dc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/571688502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86dc140 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00ad00 tx=0x7fd65c00f790 comp rx=0 tx=0).stop 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd66d0a9740 1 -- 172.21.15.89:0/571688502 shutdown_connections 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/571688502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86dc680 0x55a1d86dea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/571688502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86dc140 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/571688502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d14a0 0x55a1d86d1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd66d0a9740 1 -- 172.21.15.89:0/571688502 >> 172.21.15.89:0/571688502 conn(0x55a1d86cda30 msgr2=0x55a1d86cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd66d0a9740 1 -- 172.21.15.89:0/571688502 shutdown_connections 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.716+0000 7fd66d0a9740 1 -- 172.21.15.89:0/571688502 wait complete. 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd66d0a9740 1 -- start start 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd66d0a9740 1 Processor -- start 2024-09-15T23:31:18.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d14a0 0x55a1d86dbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d1dc0 0x55a1d86d9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86dc680 0x55a1d86d9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a1d86df250 con 0x55a1d86d14a0 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a1d85f2050 con 0x55a1d86dc680 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a1d86df3d0 con 0x55a1d86d1dc0 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd66d0a9740 10 client.0 _ll_register_callbacks cb 0x55a1d8632160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd663fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d14a0 0x55a1d86dbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd663fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d14a0 0x55a1d86dbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59070/0 (socket says 172.21.15.89:59070) 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd663fff640 1 -- 172.21.15.89:0/3313930212 learned_addr learned my addr 172.21.15.89:0/3313930212 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd668a6c640 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d1dc0 0x55a1d86d9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.717+0000 7fd663fff640 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86dc680 0x55a1d86d9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd663fff640 1 -- 172.21.15.89:0/3313930212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d1dc0 msgr2=0x55a1d86d9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd663fff640 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d1dc0 0x55a1d86d9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd663fff640 1 -- 172.21.15.89:0/3313930212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86dc680 msgr2=0x55a1d86d9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd663fff640 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86dc680 0x55a1d86d9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd663fff640 1 -- 172.21.15.89:0/3313930212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1d83a83c0 con 0x55a1d86d14a0 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd663fff640 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d14a0 0x55a1d86dbf70 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fd654009430 tx=0x7fd654009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6b4b890aab4d5a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd643fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd643fff640 1 -- 172.21.15.89:0/3313930212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65402f820 con 0x55a1d86d14a0 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd643fff640 1 -- 172.21.15.89:0/3313930212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd65402f9c0 con 0x55a1d86d14a0 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd66d0a9740 1 -- 172.21.15.89:0/3313930212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd65c018b20 con 0x55a1d86d14a0 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd643fff640 1 -- 172.21.15.89:0/3313930212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd654054050 con 0x55a1d86d14a0 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd66d0a9740 10 client.17841 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd66d0a9740 1 -- 172.21.15.89:0/3313930212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a1d86eabc0 con 0x55a1d86d14a0 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd643fff640 1 -- 172.21.15.89:0/3313930212 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c018b20 con 0x55a1d86d14a0 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.718+0000 7fd643fff640 1 -- 172.21.15.89:0/3313930212 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd65404b450 con 0x55a1d86d14a0 2024-09-15T23:31:18.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.719+0000 7fd66d0a9740 10 client.17841 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.719+0000 7fd66d0a9740 10 client.17841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.719+0000 7fd66d0a9740 20 client.17841 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.719+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1d86ebf10 0x55a1d86ee300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.719+0000 7fd66d0a9740 4 client.17841 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.719+0000 7fd66d0a9740 1 -- 172.21.15.89:0/3313930212 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1d8644850 con 0x55a1d86ebf10 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.719+0000 7fd668a6c640 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1d86ebf10 0x55a1d86ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.719+0000 7fd643fff640 10 client.17841 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.719+0000 7fd668a6c640 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1d86ebf10 0x55a1d86ee300 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4320e7de3e2aeba0 server_cookie=ff5158bf6df17de2 in_seq=0 out_seq=0 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd643fff640 1 -- 172.21.15.89:0/3313930212 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a1d8644850 con 0x55a1d86ebf10 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd643fff640 10 client.17841 handle_command_reply: tid=0 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 client.17841 shutdown 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 -- 172.21.15.89:0/3313930212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d14a0 msgr2=0x55a1d86dbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:18.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d14a0 0x55a1d86dbf70 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fd654009430 tx=0x7fd654009800 comp rx=0 tx=0).stop 2024-09-15T23:31:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 -- 172.21.15.89:0/3313930212 shutdown_connections 2024-09-15T23:31:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1d86ebf10 0x55a1d86ee300 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86dc680 0x55a1d86d9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86d1dc0 0x55a1d86d9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd643fff640 0 client.17841 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/3313930212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d14a0 0x55a1d86dbf70 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 -- 172.21.15.89:0/3313930212 >> 172.21.15.89:0/3313930212 conn(0x55a1d86cda30 msgr2=0x55a1d86d3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 -- 172.21.15.89:0/3313930212 shutdown_connections 2024-09-15T23:31:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 1 -- 172.21.15.89:0/3313930212 wait complete. 2024-09-15T23:31:18.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.720+0000 7fd66d0a9740 20 client.17841 trim_cache size 0 max 16384 2024-09-15T23:31:18.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.737+0000 7fd66d0a9740 1 Processor -- start 2024-09-15T23:31:18.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.737+0000 7fd66d0a9740 1 -- start start 2024-09-15T23:31:18.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.737+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 0x55a1d86e4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.737+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.737+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86dc680 0x55a1d86f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.737+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd6641aed50 con 0x55a1d86d1dc0 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.737+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd6641b0890 con 0x55a1d86d14a0 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.737+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd6641af420 con 0x55a1d86dc680 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd668a6c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd668a6c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86e4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59086/0 (socket says 172.21.15.89:59086) 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd668a6c640 1 -- 172.21.15.89:0/555030524 learned_addr learned my addr 172.21.15.89:0/555030524 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd663fff640 1 --2- 172.21.15.89:0/555030524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86dc680 0x55a1d86f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd663fff640 1 --2- 172.21.15.89:0/555030524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 0x55a1d86e4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd668a6c640 1 -- 172.21.15.89:0/555030524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86dc680 msgr2=0x55a1d86f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:18.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd668a6c640 1 --2- 172.21.15.89:0/555030524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86dc680 0x55a1d86f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd668a6c640 1 -- 172.21.15.89:0/555030524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 msgr2=0x55a1d86e4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd668a6c640 1 --2- 172.21.15.89:0/555030524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 0x55a1d86e4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd668a6c640 1 -- 172.21.15.89:0/555030524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1d86cd2e0 con 0x55a1d86d1dc0 2024-09-15T23:31:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd663fff640 1 --2- 172.21.15.89:0/555030524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 0x55a1d86e4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd668a6c640 1 --2- 172.21.15.89:0/555030524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86e4f40 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00ac50 tx=0x7fd65c010760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b51cb1fb8961ec2a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd65a7fc640 1 -- 172.21.15.89:0/555030524 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c031070 con 0x55a1d86d1dc0 2024-09-15T23:31:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd65a7fc640 1 -- 172.21.15.89:0/555030524 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c00a730 con 0x55a1d86d1dc0 2024-09-15T23:31:18.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.738+0000 7fd65a7fc640 1 -- 172.21.15.89:0/555030524 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c01d040 con 0x55a1d86d1dc0 2024-09-15T23:31:18.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 -- 172.21.15.89:0/555030524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 msgr2=0x55a1d86e4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:18.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/555030524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86e4f40 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fd65c00ac50 tx=0x7fd65c010760 comp rx=0 tx=0).stop 2024-09-15T23:31:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 -- 172.21.15.89:0/555030524 shutdown_connections 2024-09-15T23:31:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/555030524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86dc680 0x55a1d86f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/555030524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d86e4f40 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/555030524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 0x55a1d86e4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 -- 172.21.15.89:0/555030524 >> 172.21.15.89:0/555030524 conn(0x55a1d86cda30 msgr2=0x55a1d86ccb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 -- 172.21.15.89:0/555030524 shutdown_connections 2024-09-15T23:31:18.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 -- 172.21.15.89:0/555030524 wait complete. 2024-09-15T23:31:18.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 -- start start 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.739+0000 7fd66d0a9740 1 Processor -- start 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 0x55a1d874df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d874e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd66d0a9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86dc680 0x55a1d8752550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd6641b01c0 con 0x55a1d86d1dc0 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd6641afaf0 con 0x55a1d86d14a0 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd66d0a9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd6641acd30 con 0x55a1d86dc680 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd66d0a9740 10 client.0 _ll_register_callbacks cb 0x7fd654031640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd668a6c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d874e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd668a6c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d874e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59096/0 (socket says 172.21.15.89:59096) 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd668a6c640 1 -- 172.21.15.89:0/2918526714 learned_addr learned my addr 172.21.15.89:0/2918526714 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:18.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd663fff640 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 0x55a1d874df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd668a6c640 1 -- 172.21.15.89:0/2918526714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86dc680 msgr2=0x55a1d8752550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd663fff640 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86dc680 0x55a1d8752550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd668a6c640 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86dc680 0x55a1d8752550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd668a6c640 1 -- 172.21.15.89:0/2918526714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 msgr2=0x55a1d874df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd668a6c640 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 0x55a1d874df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd668a6c640 1 -- 172.21.15.89:0/2918526714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1d86ee670 con 0x55a1d86d1dc0 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd668a6c640 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d874e480 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fd65c010ca0 tx=0x7fd65c00fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a39c73265aca82 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.740+0000 7fd643fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd643fff640 1 -- 172.21.15.89:0/2918526714 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c031070 con 0x55a1d86d1dc0 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd643fff640 1 -- 172.21.15.89:0/2918526714 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd65c005020 con 0x55a1d86d1dc0 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd66d0a9740 1 -- 172.21.15.89:0/2918526714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd65c00a730 con 0x55a1d86d1dc0 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd643fff640 1 -- 172.21.15.89:0/2918526714 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd65c03c050 con 0x55a1d86d1dc0 2024-09-15T23:31:18.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd66d0a9740 10 client.17847 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd66d0a9740 1 -- 172.21.15.89:0/2918526714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a1d875af20 con 0x55a1d86d1dc0 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd643fff640 1 -- 172.21.15.89:0/2918526714 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd65c00a730 con 0x55a1d86d1dc0 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd643fff640 1 -- 172.21.15.89:0/2918526714 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd65c0046d0 con 0x55a1d86d1dc0 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd66d0a9740 10 client.17847 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd66d0a9740 10 client.17847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd66d0a9740 20 client.17847 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1d875b410 0x55a1d875d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd66d0a9740 4 client.17847 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.741+0000 7fd66d0a9740 1 -- 172.21.15.89:0/2918526714 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a1d86d9af0 con 0x55a1d875b410 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd663fff640 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1d875b410 0x55a1d875d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd643fff640 10 client.17847 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd663fff640 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1d875b410 0x55a1d875d830 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d727fd1aa81767fe server_cookie=e956fc9df66da128 in_seq=0 out_seq=0 2024-09-15T23:31:18.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd643fff640 1 -- 172.21.15.89:0/2918526714 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a1d86d9af0 con 0x55a1d875b410 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd643fff640 10 client.17847 handle_command_reply: tid=0 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 client.17847 shutdown 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 -- 172.21.15.89:0/2918526714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 msgr2=0x55a1d874e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d874e480 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fd65c010ca0 tx=0x7fd65c00fd10 comp rx=0 tx=0).stop 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 -- 172.21.15.89:0/2918526714 shutdown_connections 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a1d875b410 0x55a1d875d830 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a1d86dc680 0x55a1d8752550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a1d86d1dc0 0x55a1d874e480 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 --2- 172.21.15.89:0/2918526714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a1d86d14a0 0x55a1d874df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 -- 172.21.15.89:0/2918526714 >> 172.21.15.89:0/2918526714 conn(0x55a1d86cda30 msgr2=0x55a1d86daa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd643fff640 0 client.17847 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 -- 172.21.15.89:0/2918526714 shutdown_connections 2024-09-15T23:31:18.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.742+0000 7fd66d0a9740 1 -- 172.21.15.89:0/2918526714 wait complete. 2024-09-15T23:31:18.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66d0a9740 20 client.17847 trim_cache size 0 max 16384 2024-09-15T23:31:18.752 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:31:18.752 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:31:18.752 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:31:18.752 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 -- 172.21.15.89:0/1090734284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6500a3240 msgr2=0x7fd6500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6500a3240 0x7fd6500a5700 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd64c097c40 tx=0x7fd64c0c65b0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 -- 172.21.15.89:0/1090734284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd664106f10 msgr2=0x7fd66419cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd664106f10 0x7fd66419cc30 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00d750 tx=0x7fd64c00db50 comp rx=0 tx=0).stop 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 -- 172.21.15.89:0/1090734284 shutdown_connections 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6500a3240 0x7fd6500a5700 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd664106f10 0x7fd66419cc30 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd664102790 0x7fd6641988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 --2- 172.21.15.89:0/1090734284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd664101de0 0x7fd664198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 -- 172.21.15.89:0/1090734284 >> 172.21.15.89:0/1090734284 conn(0x7fd6640fd8c0 msgr2=0x7fd6640fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:18.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 -- 172.21.15.89:0/1090734284 shutdown_connections 2024-09-15T23:31:18.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:18.743+0000 7fd66acd0640 1 -- 172.21.15.89:0/1090734284 wait complete. 2024-09-15T23:31:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:19 smithi120 ceph-mon[35250]: pgmap v569: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:31:19.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:19 smithi174 ceph-mon[35586]: pgmap v569: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:31:19.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:19 smithi089 ceph-mon[30933]: pgmap v569: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:31:19.795 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:31:19.796 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:31:20.043 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:20.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.533+0000 7fd1f7f38640 1 -- 172.21.15.89:0/3199479915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f0106a90 msgr2=0x7fd1f0106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:20.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.533+0000 7fd1f7f38640 1 --2- 172.21.15.89:0/3199479915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f0106a90 0x7fd1f0106e70 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec009930 tx=0x7fd1ec02a420 comp rx=0 tx=0).stop 2024-09-15T23:31:20.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.533+0000 7fd1f7f38640 1 -- 172.21.15.89:0/3199479915 shutdown_connections 2024-09-15T23:31:20.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.533+0000 7fd1f7f38640 1 --2- 172.21.15.89:0/3199479915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1f0101430 0x7fd1f010d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:20.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.533+0000 7fd1f7f38640 1 --2- 172.21.15.89:0/3199479915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1f0100a90 0x7fd1f0100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:20.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.533+0000 7fd1f7f38640 1 --2- 172.21.15.89:0/3199479915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f0106a90 0x7fd1f0106e70 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:20.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.533+0000 7fd1f7f38640 1 -- 172.21.15.89:0/3199479915 >> 172.21.15.89:0/3199479915 conn(0x7fd1f00fc7d0 msgr2=0x7fd1f00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:20.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f7f38640 1 -- 172.21.15.89:0/3199479915 shutdown_connections 2024-09-15T23:31:20.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f7f38640 1 -- 172.21.15.89:0/3199479915 wait complete. 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f7f38640 1 Processor -- start 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f7f38640 1 -- start start 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f7f38640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1f0100a90 0x7fd1f0199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f7f38640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1f0101430 0x7fd1f0199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f7f38640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f0106a90 0x7fd1f01a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f7f38640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f0110f30 con 0x7fd1f0106a90 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f7f38640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f0110db0 con 0x7fd1f0101430 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f7f38640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f01110b0 con 0x7fd1f0100a90 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f64d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f0106a90 0x7fd1f01a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f64d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f0106a90 0x7fd1f01a44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59118/0 (socket says 172.21.15.89:59118) 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.534+0000 7fd1f64d5640 1 -- 172.21.15.89:0/1639166884 learned_addr learned my addr 172.21.15.89:0/1639166884 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:20.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1f54d3640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1f0101430 0x7fd1f0199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1f5cd4640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1f0100a90 0x7fd1f0199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1f64d5640 1 -- 172.21.15.89:0/1639166884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1f0100a90 msgr2=0x7fd1f0199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1f64d5640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1f0100a90 0x7fd1f0199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1f64d5640 1 -- 172.21.15.89:0/1639166884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1f0101430 msgr2=0x7fd1f0199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1f64d5640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1f0101430 0x7fd1f0199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1f64d5640 1 -- 172.21.15.89:0/1639166884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f01a4a10 con 0x7fd1f0106a90 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1f64d5640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f0106a90 0x7fd1f01a44d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd1e800b970 tx=0x7fd1e800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1e6ffd640 1 -- 172.21.15.89:0/1639166884 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e8019070 con 0x7fd1f0106a90 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1e6ffd640 1 -- 172.21.15.89:0/1639166884 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1e8014440 con 0x7fd1f0106a90 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1f7f38640 1 -- 172.21.15.89:0/1639166884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f01a4be0 con 0x7fd1f0106a90 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1e6ffd640 1 -- 172.21.15.89:0/1639166884 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e80075d0 con 0x7fd1f0106a90 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.535+0000 7fd1f7f38640 1 -- 172.21.15.89:0/1639166884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f01a50f0 con 0x7fd1f0106a90 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.536+0000 7fd1f7f38640 1 -- 172.21.15.89:0/1639166884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1f0100f70 con 0x7fd1f0106a90 2024-09-15T23:31:20.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.537+0000 7fd1e6ffd640 1 -- 172.21.15.89:0/1639166884 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd1e801c070 con 0x7fd1f0106a90 2024-09-15T23:31:20.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.538+0000 7fd1e6ffd640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1dc0a3240 0x7fd1dc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:20.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.538+0000 7fd1f5cd4640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1dc0a3240 0x7fd1dc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:20.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.538+0000 7fd1e6ffd640 1 -- 172.21.15.89:0/1639166884 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd1e80cf2f0 con 0x7fd1f0106a90 2024-09-15T23:31:20.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.539+0000 7fd1f5cd4640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1dc0a3240 0x7fd1dc0a5700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec000c00 tx=0x7fd1ec031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:20.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.543+0000 7fd1e6ffd640 1 -- 172.21.15.89:0/1639166884 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd1e809bc20 con 0x7fd1f0106a90 2024-09-15T23:31:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.743+0000 7fd1f7f38640 1 -- 172.21.15.89:0/1639166884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1f002d7c0 con 0x7fd1f0106a90 2024-09-15T23:31:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.744+0000 7fd1e6ffd640 1 -- 172.21.15.89:0/1639166884 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fd1e809b520 con 0x7fd1f0106a90 2024-09-15T23:31:20.747 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:31:20.747 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:31:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:31:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 -- 172.21.15.89:0/1639166884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1dc0a3240 msgr2=0x7fd1dc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1dc0a3240 0x7fd1dc0a5700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec000c00 tx=0x7fd1ec031490 comp rx=0 tx=0).stop 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 -- 172.21.15.89:0/1639166884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f0106a90 msgr2=0x7fd1f01a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f0106a90 0x7fd1f01a44d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd1e800b970 tx=0x7fd1e800bd70 comp rx=0 tx=0).stop 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 -- 172.21.15.89:0/1639166884 shutdown_connections 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd1dc0a3240 0x7fd1dc0a5700 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd1f0106a90 0x7fd1f01a44d0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1f0101430 0x7fd1f0199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 --2- 172.21.15.89:0/1639166884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1f0100a90 0x7fd1f0199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 -- 172.21.15.89:0/1639166884 >> 172.21.15.89:0/1639166884 conn(0x7fd1f00fc7d0 msgr2=0x7fd1f00fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 -- 172.21.15.89:0/1639166884 shutdown_connections 2024-09-15T23:31:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:20.747+0000 7fd1f7f38640 1 -- 172.21.15.89:0/1639166884 wait complete. 2024-09-15T23:31:21.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:20 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1639166884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:21.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:20 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1639166884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:21.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:20 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1639166884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:21.466 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:31:21.710 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:22.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:21 smithi089 ceph-mon[30933]: pgmap v570: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:22.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:21 smithi120 ceph-mon[35250]: pgmap v570: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:22.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.213+0000 7f324964a640 1 -- 172.21.15.89:0/2088069314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244101420 msgr2=0x7f324410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.213+0000 7f324964a640 1 --2- 172.21.15.89:0/2088069314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244101420 0x7f324410d950 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f323c009cd0 tx=0x7f323c02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.213+0000 7f324964a640 1 -- 172.21.15.89:0/2088069314 shutdown_connections 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.213+0000 7f324964a640 1 --2- 172.21.15.89:0/2088069314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244101420 0x7f324410d950 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.213+0000 7f324964a640 1 --2- 172.21.15.89:0/2088069314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3244100a80 0x7f3244100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.213+0000 7f324964a640 1 --2- 172.21.15.89:0/2088069314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3244106a80 0x7f3244106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.213+0000 7f324964a640 1 -- 172.21.15.89:0/2088069314 >> 172.21.15.89:0/2088069314 conn(0x7f32440fc7a0 msgr2=0x7f32440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.213+0000 7f324964a640 1 -- 172.21.15.89:0/2088069314 shutdown_connections 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.213+0000 7f324964a640 1 -- 172.21.15.89:0/2088069314 wait complete. 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.213+0000 7f324964a640 1 Processor -- start 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f324964a640 1 -- start start 2024-09-15T23:31:22.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f324964a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3244100a80 0x7f3244198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f324964a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3244101420 0x7f3244198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f324964a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244106a80 0x7f324419ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f324964a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3244110e80 con 0x7f3244106a80 2024-09-15T23:31:22.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:21 smithi174 ceph-mon[35586]: pgmap v570: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:22.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f324964a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3244110d00 con 0x7f3244101420 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f324964a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3244111000 con 0x7f3244100a80 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f32437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244106a80 0x7f324419ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f32437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244106a80 0x7f324419ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59144/0 (socket says 172.21.15.89:59144) 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f32437fe640 1 -- 172.21.15.89:0/2047399663 learned_addr learned my addr 172.21.15.89:0/2047399663 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f3242ffd640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3244100a80 0x7f3244198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f323bfff640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3244101420 0x7f3244198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f32437fe640 1 -- 172.21.15.89:0/2047399663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3244100a80 msgr2=0x7f3244198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f32437fe640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3244100a80 0x7f3244198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f32437fe640 1 -- 172.21.15.89:0/2047399663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3244101420 msgr2=0x7f3244198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f32437fe640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3244101420 0x7f3244198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f32437fe640 1 -- 172.21.15.89:0/2047399663 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324419d3b0 con 0x7f3244106a80 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.214+0000 7f32437fe640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244106a80 0x7f324419ccd0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f323c005cc0 tx=0x7f323c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.215+0000 7f3240ff9640 1 -- 172.21.15.89:0/2047399663 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c004040 con 0x7f3244106a80 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.215+0000 7f3240ff9640 1 -- 172.21.15.89:0/2047399663 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f323c0046c0 con 0x7f3244106a80 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.215+0000 7f324964a640 1 -- 172.21.15.89:0/2047399663 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324419d640 con 0x7f3244106a80 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.215+0000 7f3240ff9640 1 -- 172.21.15.89:0/2047399663 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c046480 con 0x7f3244106a80 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.215+0000 7f324964a640 1 -- 172.21.15.89:0/2047399663 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32441a4f80 con 0x7f3244106a80 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.217+0000 7f3240ff9640 1 -- 172.21.15.89:0/2047399663 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f323c0305a0 con 0x7f3244106a80 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.217+0000 7f324ba23740 1 Processor -- start 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.217+0000 7f324ba23740 1 -- start start 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f324ba23740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2db40 0x559098c2c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f324ba23740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2c5d0 0x559098c2ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f324ba23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 0x559098c39760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f324ba23740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559098b922f0 con 0x559098c2db40 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f324ba23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559098b39cf0 con 0x559098c37370 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f324ba23740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559098b8e2b0 con 0x559098c2c5d0 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f323bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2db40 0x559098c2c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f323bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2db40 0x559098c2c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59152/0 (socket says 172.21.15.89:59152) 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f323bfff640 1 -- 172.21.15.89:0/2207163508 learned_addr learned my addr 172.21.15.89:0/2207163508 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f3240ff9640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f32300a32c0 0x7f32300a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f3242ffd640 1 --2- 172.21.15.89:0/2207163508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2c5d0 0x559098c2ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f32437fe640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f32300a32c0 0x7f32300a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f323bfff640 1 -- 172.21.15.89:0/2207163508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2c5d0 msgr2=0x559098c2ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f323bfff640 1 --2- 172.21.15.89:0/2207163508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2c5d0 0x559098c2ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f323bfff640 1 -- 172.21.15.89:0/2207163508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 msgr2=0x559098c39760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f323bfff640 1 --2- 172.21.15.89:0/2207163508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 0x559098c39760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f323bfff640 1 -- 172.21.15.89:0/2207163508 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559098b1e3a0 con 0x559098c2db40 2024-09-15T23:31:22.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f3240ff9640 1 -- 172.21.15.89:0/2047399663 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f323c02e030 con 0x7f3244106a80 2024-09-15T23:31:22.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f323bfff640 1 --2- 172.21.15.89:0/2207163508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2db40 0x559098c2c090 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f322c00ac70 tx=0x7f322c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35227f8c0366c0c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f32397fa640 1 -- 172.21.15.89:0/2207163508 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c005020 con 0x559098c2db40 2024-09-15T23:31:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f32397fa640 1 -- 172.21.15.89:0/2207163508 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f322c0047a0 con 0x559098c2db40 2024-09-15T23:31:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.218+0000 7f32397fa640 1 -- 172.21.15.89:0/2207163508 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c012650 con 0x559098c2db40 2024-09-15T23:31:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.219+0000 7f32437fe640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f32300a32c0 0x7f32300a5780 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f323c0ba0e0 tx=0x7f323c0c1040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.219+0000 7f324ba23740 1 -- 172.21.15.89:0/2207163508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2db40 msgr2=0x559098c2c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.219+0000 7f324ba23740 1 --2- 172.21.15.89:0/2207163508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2db40 0x559098c2c090 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f322c00ac70 tx=0x7f322c0117b0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.219+0000 7f324ba23740 1 -- 172.21.15.89:0/2207163508 shutdown_connections 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.219+0000 7f324ba23740 1 --2- 172.21.15.89:0/2207163508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 0x559098c39760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.219+0000 7f324ba23740 1 --2- 172.21.15.89:0/2207163508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2c5d0 0x559098c2ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.219+0000 7f324ba23740 1 --2- 172.21.15.89:0/2207163508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2db40 0x559098c2c090 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.219+0000 7f324ba23740 1 -- 172.21.15.89:0/2207163508 >> 172.21.15.89:0/2207163508 conn(0x559098bf5310 msgr2=0x559098c28c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.219+0000 7f324ba23740 1 -- 172.21.15.89:0/2207163508 shutdown_connections 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.219+0000 7f324ba23740 1 -- 172.21.15.89:0/2207163508 wait complete. 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f324ba23740 1 -- start start 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f324ba23740 1 Processor -- start 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f324ba23740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c36d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f324ba23740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2db40 0x559098c33fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f324ba23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 0x559098c345d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f324ba23740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559098c3a070 con 0x559098c2c5d0 2024-09-15T23:31:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f324ba23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559098b4d060 con 0x559098c37370 2024-09-15T23:31:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f324ba23740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559098c3a1f0 con 0x559098c2db40 2024-09-15T23:31:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f324ba23740 10 client.0 _ll_register_callbacks cb 0x5590988a08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f323bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c36d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f323bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c36d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59162/0 (socket says 172.21.15.89:59162) 2024-09-15T23:31:22.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f323bfff640 1 -- 172.21.15.89:0/1191031412 learned_addr learned my addr 172.21.15.89:0/1191031412 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f323bfff640 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 0x559098c345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f323bfff640 1 -- 172.21.15.89:0/1191031412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2db40 msgr2=0x559098c33fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f323bfff640 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2db40 0x559098c33fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f323bfff640 1 -- 172.21.15.89:0/1191031412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 msgr2=0x559098c345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f323bfff640 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 0x559098c345d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.220+0000 7f323bfff640 1 -- 172.21.15.89:0/1191031412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590989033c0 con 0x559098c2c5d0 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f323bfff640 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c36d30 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f322c005cb0 tx=0x7f322c00a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a97d9b3e47a8fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f322a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f322a7fc640 1 -- 172.21.15.89:0/1191031412 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c0054d0 con 0x559098c2c5d0 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f322a7fc640 1 -- 172.21.15.89:0/1191031412 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f322c004020 con 0x559098c2c5d0 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f324ba23740 1 -- 172.21.15.89:0/1191031412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f322c0047a0 con 0x559098c2c5d0 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f322a7fc640 1 -- 172.21.15.89:0/1191031412 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f322c031050 con 0x559098c2c5d0 2024-09-15T23:31:22.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f324ba23740 10 client.17865 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f324ba23740 1 -- 172.21.15.89:0/1191031412 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559098c45ad0 con 0x559098c2c5d0 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f322a7fc640 1 -- 172.21.15.89:0/1191031412 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f322c0047a0 con 0x559098c2c5d0 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f322a7fc640 1 -- 172.21.15.89:0/1191031412 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f322c028070 con 0x559098c2c5d0 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f324ba23740 10 client.17865 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f324ba23740 10 client.17865 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f324ba23740 20 client.17865 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f324ba23740 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x559098c46ee0 0x559098c492d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f324ba23740 4 client.17865 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.221+0000 7f324ba23740 1 -- 172.21.15.89:0/1191031412 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559098b9f860 con 0x559098c46ee0 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.222+0000 7f3242ffd640 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x559098c46ee0 0x559098c492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.222+0000 7f322a7fc640 10 client.17865 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:31:22.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.222+0000 7f3242ffd640 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x559098c46ee0 0x559098c492d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1d590520895b7ca6 server_cookie=8e8e0420612c30aa in_seq=0 out_seq=0 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f322a7fc640 1 -- 172.21.15.89:0/1191031412 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559098b9f860 con 0x559098c46ee0 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f322a7fc640 10 client.17865 handle_command_reply: tid=0 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 client.17865 shutdown 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 -- 172.21.15.89:0/1191031412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 msgr2=0x559098c36d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c36d30 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f322c005cb0 tx=0x7f322c00a490 comp rx=0 tx=0).stop 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 -- 172.21.15.89:0/1191031412 shutdown_connections 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x559098c46ee0 0x559098c492d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 0x559098c345d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2db40 0x559098c33fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 --2- 172.21.15.89:0/1191031412 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c36d30 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f322a7fc640 0 client.17865 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:31:22.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 -- 172.21.15.89:0/1191031412 >> 172.21.15.89:0/1191031412 conn(0x559098bf5310 msgr2=0x559098c2e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 -- 172.21.15.89:0/1191031412 shutdown_connections 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 1 -- 172.21.15.89:0/1191031412 wait complete. 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.223+0000 7f324ba23740 20 client.17865 trim_cache size 0 max 16384 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f324ba23740 1 Processor -- start 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f324ba23740 1 -- start start 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f324ba23740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2c5d0 0x559098c32780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f324ba23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c2db40 0x559098c2f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f324ba23740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c37370 0x559098c2f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f324ba23740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f32441aed90 con 0x559098c37370 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f324ba23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f32441b08d0 con 0x559098c2db40 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f324ba23740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f32441af460 con 0x559098c2c5d0 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f323bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c37370 0x559098c2f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f3242ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c2db40 0x559098c2f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f323bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c37370 0x559098c2f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59166/0 (socket says 172.21.15.89:59166) 2024-09-15T23:31:22.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f323bfff640 1 -- 172.21.15.89:0/2261429234 learned_addr learned my addr 172.21.15.89:0/2261429234 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:22.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.237+0000 7f323bfff640 1 --2- 172.21.15.89:0/2261429234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2c5d0 0x559098c32780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.238+0000 7f323bfff640 1 -- 172.21.15.89:0/2261429234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2c5d0 msgr2=0x559098c32780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.238+0000 7f323bfff640 1 --2- 172.21.15.89:0/2261429234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2c5d0 0x559098c32780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.238+0000 7f323bfff640 1 -- 172.21.15.89:0/2261429234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c2db40 msgr2=0x559098c2f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.238+0000 7f323bfff640 1 --2- 172.21.15.89:0/2261429234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c2db40 0x559098c2f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.238+0000 7f323bfff640 1 -- 172.21.15.89:0/2261429234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559098c2d780 con 0x559098c37370 2024-09-15T23:31:22.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.238+0000 7f3242ffd640 1 --2- 172.21.15.89:0/2261429234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c2db40 0x559098c2f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:31:22.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.238+0000 7f323bfff640 1 --2- 172.21.15.89:0/2261429234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c37370 0x559098c2f660 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f322c012780 tx=0x7f322c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2395af91621c9ab6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.238+0000 7f3238ff9640 1 -- 172.21.15.89:0/2261429234 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c0046a0 con 0x559098c37370 2024-09-15T23:31:22.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.238+0000 7f3238ff9640 1 -- 172.21.15.89:0/2261429234 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f322c01f030 con 0x559098c37370 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.238+0000 7f3238ff9640 1 -- 172.21.15.89:0/2261429234 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c0129c0 con 0x559098c37370 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 -- 172.21.15.89:0/2261429234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c37370 msgr2=0x559098c2f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 --2- 172.21.15.89:0/2261429234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c37370 0x559098c2f660 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f322c012780 tx=0x7f322c02c040 comp rx=0 tx=0).stop 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 -- 172.21.15.89:0/2261429234 shutdown_connections 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 --2- 172.21.15.89:0/2261429234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c37370 0x559098c2f660 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 --2- 172.21.15.89:0/2261429234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c2db40 0x559098c2f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 --2- 172.21.15.89:0/2261429234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2c5d0 0x559098c32780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 -- 172.21.15.89:0/2261429234 >> 172.21.15.89:0/2261429234 conn(0x559098bf5310 msgr2=0x559098c27d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 -- 172.21.15.89:0/2261429234 shutdown_connections 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 -- 172.21.15.89:0/2261429234 wait complete. 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 -- start start 2024-09-15T23:31:22.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 Processor -- start 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c3b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2db40 0x559098c3b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 0x559098c3bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f32441b0200 con 0x559098c2c5d0 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f32441afb30 con 0x559098c37370 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.239+0000 7f324ba23740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f32441acd70 con 0x559098c2db40 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f324ba23740 10 client.0 _ll_register_callbacks cb 0x7f323400b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f323bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c3b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f323bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c3b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59168/0 (socket says 172.21.15.89:59168) 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f323bfff640 1 -- 172.21.15.89:0/3529942057 learned_addr learned my addr 172.21.15.89:0/3529942057 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f3242ffd640 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2db40 0x559098c3b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f323bfff640 1 -- 172.21.15.89:0/3529942057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2db40 msgr2=0x559098c3b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f323bfff640 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2db40 0x559098c3b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f323bfff640 1 -- 172.21.15.89:0/3529942057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 msgr2=0x559098c3bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f323bfff640 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 0x559098c3bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f323bfff640 1 -- 172.21.15.89:0/3529942057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559098cb0950 con 0x559098c2c5d0 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f323bfff640 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c3b230 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f322c012940 tx=0x7f322c0040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21aa244dcbc2ccbb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f322a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f322a7fc640 1 -- 172.21.15.89:0/3529942057 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c00a590 con 0x559098c2c5d0 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f322a7fc640 1 -- 172.21.15.89:0/3529942057 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f322c004290 con 0x559098c2c5d0 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f324ba23740 1 -- 172.21.15.89:0/3529942057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f322c01f030 con 0x559098c2c5d0 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f322a7fc640 1 -- 172.21.15.89:0/3529942057 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f322c0052d0 con 0x559098c2c5d0 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f324ba23740 10 client.17871 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f324ba23740 1 -- 172.21.15.89:0/3529942057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559098cb1c30 con 0x559098c2c5d0 2024-09-15T23:31:22.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.240+0000 7f322a7fc640 1 -- 172.21.15.89:0/3529942057 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f322c01f030 con 0x559098c2c5d0 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.241+0000 7f322a7fc640 1 -- 172.21.15.89:0/3529942057 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f322c028040 con 0x559098c2c5d0 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.241+0000 7f324ba23740 10 client.17871 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.241+0000 7f324ba23740 10 client.17871 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.241+0000 7f324ba23740 20 client.17871 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.241+0000 7f324ba23740 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x559098cb21e0 0x559098cb45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.241+0000 7f324ba23740 4 client.17871 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.241+0000 7f324ba23740 1 -- 172.21.15.89:0/3529942057 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559098b99d70 con 0x559098cb21e0 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.241+0000 7f3242ffd640 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x559098cb21e0 0x559098cb45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.241+0000 7f3242ffd640 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x559098cb21e0 0x559098cb45d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7d1d273915818bb4 server_cookie=e29cc5233a359fb3 in_seq=0 out_seq=0 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.241+0000 7f322a7fc640 10 client.17871 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f322a7fc640 1 -- 172.21.15.89:0/3529942057 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559098b99d70 con 0x559098cb21e0 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f322a7fc640 10 client.17871 handle_command_reply: tid=0 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 client.17871 shutdown 2024-09-15T23:31:22.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 -- 172.21.15.89:0/3529942057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 msgr2=0x559098c3b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c3b230 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f322c012940 tx=0x7f322c0040b0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 -- 172.21.15.89:0/3529942057 shutdown_connections 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x559098cb21e0 0x559098cb45d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559098c37370 0x559098c3bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559098c2db40 0x559098c3b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f322a7fc640 0 client.17871 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 --2- 172.21.15.89:0/3529942057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559098c2c5d0 0x559098c3b230 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 -- 172.21.15.89:0/3529942057 >> 172.21.15.89:0/3529942057 conn(0x559098bf5310 msgr2=0x559098c35840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 -- 172.21.15.89:0/3529942057 shutdown_connections 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 1 -- 172.21.15.89:0/3529942057 wait complete. 2024-09-15T23:31:22.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324ba23740 20 client.17871 trim_cache size 0 max 16384 2024-09-15T23:31:22.259 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:31:22.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324964a640 1 -- 172.21.15.89:0/2047399663 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f32300a32c0 msgr2=0x7f32300a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324964a640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f32300a32c0 0x7f32300a5780 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f323c0ba0e0 tx=0x7f323c0c1040 comp rx=0 tx=0).stop 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324964a640 1 -- 172.21.15.89:0/2047399663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244106a80 msgr2=0x7f324419ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324964a640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244106a80 0x7f324419ccd0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f323c005cc0 tx=0x7f323c003970 comp rx=0 tx=0).stop 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324964a640 1 -- 172.21.15.89:0/2047399663 shutdown_connections 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324964a640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f32300a32c0 0x7f32300a5780 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324964a640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3244106a80 0x7f324419ccd0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324964a640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3244101420 0x7f3244198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324964a640 1 --2- 172.21.15.89:0/2047399663 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3244100a80 0x7f3244198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.242+0000 7f324964a640 1 -- 172.21.15.89:0/2047399663 >> 172.21.15.89:0/2047399663 conn(0x7f32440fc7a0 msgr2=0x7f324406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.243+0000 7f324964a640 1 -- 172.21.15.89:0/2047399663 shutdown_connections 2024-09-15T23:31:22.260 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:22.243+0000 7f324964a640 1 -- 172.21.15.89:0/2047399663 wait complete. 2024-09-15T23:31:22.998 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:31:23.246 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:23 smithi120 ceph-mon[35250]: pgmap v571: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:23 smithi174 ceph-mon[35586]: pgmap v571: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:23.534 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:23 smithi089 ceph-mon[30933]: pgmap v571: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:23.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.773+0000 7f817d490640 1 -- 172.21.15.89:0/3567946439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8178100a80 msgr2=0x7f8178100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:23.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.773+0000 7f817d490640 1 --2- 172.21.15.89:0/3567946439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8178100a80 0x7f8178100ee0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f8160009930 tx=0x7f816002a420 comp rx=0 tx=0).stop 2024-09-15T23:31:23.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.773+0000 7f817d490640 1 -- 172.21.15.89:0/3567946439 shutdown_connections 2024-09-15T23:31:23.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.773+0000 7f817d490640 1 --2- 172.21.15.89:0/3567946439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8178101420 0x7f817810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.773+0000 7f817d490640 1 --2- 172.21.15.89:0/3567946439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8178100a80 0x7f8178100ee0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.773+0000 7f817d490640 1 --2- 172.21.15.89:0/3567946439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8178106a80 0x7f8178106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.773+0000 7f817d490640 1 -- 172.21.15.89:0/3567946439 >> 172.21.15.89:0/3567946439 conn(0x7f81780fc7a0 msgr2=0x7f81780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.773+0000 7f817d490640 1 -- 172.21.15.89:0/3567946439 shutdown_connections 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.773+0000 7f817d490640 1 -- 172.21.15.89:0/3567946439 wait complete. 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f817d490640 1 Processor -- start 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f817d490640 1 -- start start 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f817d490640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8178100a80 0x7f8178199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f817d490640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8178101420 0x7f8178199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f817d490640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8178106a80 0x7f81781a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f817d490640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8178110f20 con 0x7f8178106a80 2024-09-15T23:31:23.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f817d490640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8178110da0 con 0x7f8178101420 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f817d490640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f81781110a0 con 0x7f8178100a80 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f81777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8178106a80 0x7f81781a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f81777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8178106a80 0x7f81781a44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59182/0 (socket says 172.21.15.89:59182) 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f81777fe640 1 -- 172.21.15.89:0/1128569903 learned_addr learned my addr 172.21.15.89:0/1128569903 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f816ffff640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8178101420 0x7f8178199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f8176ffd640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8178100a80 0x7f8178199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f81777fe640 1 -- 172.21.15.89:0/1128569903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8178100a80 msgr2=0x7f8178199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f81777fe640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8178100a80 0x7f8178199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f81777fe640 1 -- 172.21.15.89:0/1128569903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8178101420 msgr2=0x7f8178199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f81777fe640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8178101420 0x7f8178199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f81777fe640 1 -- 172.21.15.89:0/1128569903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81781a49f0 con 0x7f8178106a80 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f81777fe640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8178106a80 0x7f81781a44b0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f817000b970 tx=0x7f817000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:23.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f8174ff9640 1 -- 172.21.15.89:0/1128569903 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8170019070 con 0x7f8178106a80 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f8174ff9640 1 -- 172.21.15.89:0/1128569903 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f817000cb40 con 0x7f8178106a80 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f817d490640 1 -- 172.21.15.89:0/1128569903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81781a4bc0 con 0x7f8178106a80 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f8174ff9640 1 -- 172.21.15.89:0/1128569903 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8170007650 con 0x7f8178106a80 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.775+0000 7f817d490640 1 -- 172.21.15.89:0/1128569903 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81781a5080 con 0x7f8178106a80 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f8174ff9640 1 -- 172.21.15.89:0/1128569903 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f817001c070 con 0x7f8178106a80 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f817f869740 1 Processor -- start 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f817f869740 1 -- start start 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f817f869740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b0251b40 0x5592b0250090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f817f869740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b02505d0 0x5592b0250a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f817f869740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 0x5592b025d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f817f869740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5592b01b62f0 con 0x5592b0251b40 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f817f869740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5592b015dcf0 con 0x5592b025b370 2024-09-15T23:31:23.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f817f869740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5592b01b22b0 con 0x5592b02505d0 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b0251b40 0x5592b0250090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b0251b40 0x5592b0250090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59186/0 (socket says 172.21.15.89:59186) 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816ffff640 1 -- 172.21.15.89:0/1788023806 learned_addr learned my addr 172.21.15.89:0/1788023806 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f8174ff9640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81640a3240 0x7f81640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f8176ffd640 1 --2- 172.21.15.89:0/1788023806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b02505d0 0x5592b0250a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816ffff640 1 -- 172.21.15.89:0/1788023806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b02505d0 msgr2=0x5592b0250a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816ffff640 1 --2- 172.21.15.89:0/1788023806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b02505d0 0x5592b0250a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816ffff640 1 -- 172.21.15.89:0/1788023806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 msgr2=0x5592b025d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816ffff640 1 --2- 172.21.15.89:0/1788023806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 0x5592b025d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816ffff640 1 -- 172.21.15.89:0/1788023806 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592b01423a0 con 0x5592b0251b40 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f81777fe640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81640a3240 0x7f81640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f8174ff9640 1 -- 172.21.15.89:0/1128569903 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f81700cf3f0 con 0x7f8178106a80 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816ffff640 1 --2- 172.21.15.89:0/1788023806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b0251b40 0x5592b0250090 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f8160005ab0 tx=0x7f8160009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966e8fe98f9c7bd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816d7fa640 1 -- 172.21.15.89:0/1788023806 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160035050 con 0x5592b0251b40 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f81777fe640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81640a3240 0x7f81640a5700 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8170014d40 tx=0x7f817006a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816d7fa640 1 -- 172.21.15.89:0/1788023806 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8160005020 con 0x5592b0251b40 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.779+0000 7f816d7fa640 1 -- 172.21.15.89:0/1788023806 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160004a30 con 0x5592b0251b40 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.780+0000 7f817f869740 1 -- 172.21.15.89:0/1788023806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b0251b40 msgr2=0x5592b0250090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.780+0000 7f817f869740 1 --2- 172.21.15.89:0/1788023806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b0251b40 0x5592b0250090 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f8160005ab0 tx=0x7f8160009680 comp rx=0 tx=0).stop 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.780+0000 7f817f869740 1 -- 172.21.15.89:0/1788023806 shutdown_connections 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.780+0000 7f817f869740 1 --2- 172.21.15.89:0/1788023806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 0x5592b025d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.780+0000 7f817f869740 1 --2- 172.21.15.89:0/1788023806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b02505d0 0x5592b0250a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.780+0000 7f817f869740 1 --2- 172.21.15.89:0/1788023806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b0251b40 0x5592b0250090 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.780+0000 7f817f869740 1 -- 172.21.15.89:0/1788023806 >> 172.21.15.89:0/1788023806 conn(0x5592b0219310 msgr2=0x5592b024cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.780+0000 7f817f869740 1 -- 172.21.15.89:0/1788023806 shutdown_connections 2024-09-15T23:31:23.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.780+0000 7f817f869740 1 -- 172.21.15.89:0/1788023806 wait complete. 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.780+0000 7f817f869740 1 -- start start 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 Processor -- start 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b025ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b0257fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 0x5592b0258530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5592b025e010 con 0x5592b02505d0 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5592b0171060 con 0x5592b025b370 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5592b025e190 con 0x5592b0251b40 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 10 client.0 _ll_register_callbacks cb 0x5592afec48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f816ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b025ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f816ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b025ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59196/0 (socket says 172.21.15.89:59196) 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f816ffff640 1 -- 172.21.15.89:0/1007097610 learned_addr learned my addr 172.21.15.89:0/1007097610 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:23.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f8176ffd640 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b0257fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f816ffff640 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 0x5592b0258530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f816ffff640 1 -- 172.21.15.89:0/1007097610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 msgr2=0x5592b0257fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f816ffff640 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b0257fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f816ffff640 1 -- 172.21.15.89:0/1007097610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 msgr2=0x5592b0258530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f816ffff640 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 0x5592b0258530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f816ffff640 1 -- 172.21.15.89:0/1007097610 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592aff273c0 con 0x5592b02505d0 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f816ffff640 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b025ad00 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f81600092a0 tx=0x7f8160009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e298607b7c26cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f815e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f815e7fc640 1 -- 172.21.15.89:0/1007097610 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160035040 con 0x5592b02505d0 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f815e7fc640 1 -- 172.21.15.89:0/1007097610 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f816003d680 con 0x5592b02505d0 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 -- 172.21.15.89:0/1007097610 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8160005020 con 0x5592b02505d0 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f815e7fc640 1 -- 172.21.15.89:0/1007097610 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8160059050 con 0x5592b02505d0 2024-09-15T23:31:23.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 10 client.17883 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 -- 172.21.15.89:0/1007097610 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5592b0269a40 con 0x5592b02505d0 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f815e7fc640 1 -- 172.21.15.89:0/1007097610 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8160005020 con 0x5592b02505d0 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f815e7fc640 1 -- 172.21.15.89:0/1007097610 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8160045930 con 0x5592b02505d0 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 10 client.17883 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 10 client.17883 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 20 client.17883 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5592b026ad90 0x5592b026d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 4 client.17883 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.782+0000 7f817f869740 1 -- 172.21.15.89:0/1007097610 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592b01c3860 con 0x5592b026ad90 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.784+0000 7f8176ffd640 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5592b026ad90 0x5592b026d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.784+0000 7f815e7fc640 10 client.17883 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.784+0000 7f8176ffd640 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5592b026ad90 0x5592b026d180 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a4216dc58cd88123 server_cookie=3c10e60282968522 in_seq=0 out_seq=0 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f815e7fc640 1 -- 172.21.15.89:0/1007097610 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5592b01c3860 con 0x5592b026ad90 2024-09-15T23:31:23.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f815e7fc640 10 client.17883 handle_command_reply: tid=0 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 client.17883 shutdown 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 -- 172.21.15.89:0/1007097610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 msgr2=0x5592b025ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b025ad00 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f81600092a0 tx=0x7f8160009230 comp rx=0 tx=0).stop 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 -- 172.21.15.89:0/1007097610 shutdown_connections 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5592b026ad90 0x5592b026d180 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 0x5592b0258530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b0257fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 --2- 172.21.15.89:0/1007097610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b025ad00 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f815e7fc640 0 client.17883 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 -- 172.21.15.89:0/1007097610 >> 172.21.15.89:0/1007097610 conn(0x5592b0219310 msgr2=0x5592b0252960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 -- 172.21.15.89:0/1007097610 shutdown_connections 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 1 -- 172.21.15.89:0/1007097610 wait complete. 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.785+0000 7f817f869740 20 client.17883 trim_cache size 0 max 16384 2024-09-15T23:31:23.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f817f869740 1 Processor -- start 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f817f869740 1 -- start start 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f817f869740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b02505d0 0x5592b0257870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f817f869740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b02701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f817f869740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b025b370 0x5592b02706e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f817f869740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f81781aee20 con 0x5592b025b370 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f817f869740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81781b0960 con 0x5592b02505d0 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f817f869740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f81781af4f0 con 0x5592b0251b40 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f816ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b025b370 0x5592b02706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f816ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b025b370 0x5592b02706e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59206/0 (socket says 172.21.15.89:59206) 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f816ffff640 1 -- 172.21.15.89:0/1089683476 learned_addr learned my addr 172.21.15.89:0/1089683476 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f816ffff640 1 --2- 172.21.15.89:0/1089683476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b02505d0 0x5592b0257870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.798+0000 7f8176ffd640 1 --2- 172.21.15.89:0/1089683476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b02701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.799+0000 7f816ffff640 1 -- 172.21.15.89:0/1089683476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 msgr2=0x5592b02701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.799+0000 7f816ffff640 1 --2- 172.21.15.89:0/1089683476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b02701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.799+0000 7f816ffff640 1 -- 172.21.15.89:0/1089683476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b02505d0 msgr2=0x5592b0257870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.799+0000 7f816ffff640 1 --2- 172.21.15.89:0/1089683476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b02505d0 0x5592b0257870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.799+0000 7f816ffff640 1 -- 172.21.15.89:0/1089683476 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592b0251750 con 0x5592b025b370 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.799+0000 7f816ffff640 1 --2- 172.21.15.89:0/1089683476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b025b370 0x5592b02706e0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f8160031520 tx=0x7f81600058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ea32671980b0a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.799+0000 7f816cff9640 1 -- 172.21.15.89:0/1089683476 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160035040 con 0x5592b025b370 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.799+0000 7f816cff9640 1 -- 172.21.15.89:0/1089683476 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8160050070 con 0x5592b025b370 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.799+0000 7f816cff9640 1 -- 172.21.15.89:0/1089683476 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81600486a0 con 0x5592b025b370 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 -- 172.21.15.89:0/1089683476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b025b370 msgr2=0x5592b02706e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 --2- 172.21.15.89:0/1089683476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b025b370 0x5592b02706e0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f8160031520 tx=0x7f81600058b0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 -- 172.21.15.89:0/1089683476 shutdown_connections 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 --2- 172.21.15.89:0/1089683476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b025b370 0x5592b02706e0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 --2- 172.21.15.89:0/1089683476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b02701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 --2- 172.21.15.89:0/1089683476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b02505d0 0x5592b0257870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 -- 172.21.15.89:0/1089683476 >> 172.21.15.89:0/1089683476 conn(0x5592b0219310 msgr2=0x5592b0256880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 -- 172.21.15.89:0/1089683476 shutdown_connections 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 -- 172.21.15.89:0/1089683476 wait complete. 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 -- start start 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 Processor -- start 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b0270690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b026d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 0x5592b026df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f81781b0290 con 0x5592b02505d0 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81781afbc0 con 0x5592b025b370 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f81781ace00 con 0x5592b0251b40 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.800+0000 7f817f869740 10 client.0 _ll_register_callbacks cb 0x7f8160031640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:23.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f816ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b0270690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f816ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b0270690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59208/0 (socket says 172.21.15.89:59208) 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f816ffff640 1 -- 172.21.15.89:0/371806435 learned_addr learned my addr 172.21.15.89:0/371806435 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f8176ffd640 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b026d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f816ffff640 1 -- 172.21.15.89:0/371806435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 msgr2=0x5592b026d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f816ffff640 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b026d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f816ffff640 1 -- 172.21.15.89:0/371806435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 msgr2=0x5592b026df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f816ffff640 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 0x5592b026df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f816ffff640 1 -- 172.21.15.89:0/371806435 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592b026e4e0 con 0x5592b02505d0 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f8176ffd640 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b026d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f816ffff640 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b0270690 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f8160004650 tx=0x7f816002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad5f094dbf57fe7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f815e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f815e7fc640 1 -- 172.21.15.89:0/371806435 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160035040 con 0x5592b02505d0 2024-09-15T23:31:23.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f815e7fc640 1 -- 172.21.15.89:0/371806435 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8160045640 con 0x5592b02505d0 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f817f869740 1 -- 172.21.15.89:0/371806435 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8160050070 con 0x5592b02505d0 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f815e7fc640 1 -- 172.21.15.89:0/371806435 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f81600047a0 con 0x5592b02505d0 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f817f869740 10 client.17889 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.801+0000 7f817f869740 1 -- 172.21.15.89:0/371806435 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5592b02d5460 con 0x5592b02505d0 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.802+0000 7f815e7fc640 1 -- 172.21.15.89:0/371806435 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8160050070 con 0x5592b02505d0 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.802+0000 7f815e7fc640 1 -- 172.21.15.89:0/371806435 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f81600458f0 con 0x5592b02505d0 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.802+0000 7f817f869740 10 client.17889 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.802+0000 7f817f869740 10 client.17889 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.802+0000 7f817f869740 20 client.17889 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.802+0000 7f817f869740 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5592b02d58d0 0x5592b02d7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.802+0000 7f817f869740 4 client.17889 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.802+0000 7f817f869740 1 -- 172.21.15.89:0/371806435 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5592b01bdd70 con 0x5592b02d58d0 2024-09-15T23:31:23.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.802+0000 7f8176ffd640 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5592b02d58d0 0x5592b02d7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f815e7fc640 10 client.17889 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f8176ffd640 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5592b02d58d0 0x5592b02d7cc0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1d288992ac1e5d0e server_cookie=eec0b0b89e2545de in_seq=0 out_seq=0 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f815e7fc640 1 -- 172.21.15.89:0/371806435 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5592b01bdd70 con 0x5592b02d58d0 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f815e7fc640 10 client.17889 handle_command_reply: tid=0 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 client.17889 shutdown 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 -- 172.21.15.89:0/371806435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 msgr2=0x5592b0270690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b0270690 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f8160004650 tx=0x7f816002a420 comp rx=0 tx=0).stop 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 -- 172.21.15.89:0/371806435 shutdown_connections 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5592b02d58d0 0x5592b02d7cc0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592b025b370 0x5592b026df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5592b0251b40 0x5592b026d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f815e7fc640 0 client.17889 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 --2- 172.21.15.89:0/371806435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592b02505d0 0x5592b0270690 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 -- 172.21.15.89:0/371806435 >> 172.21.15.89:0/371806435 conn(0x5592b0219310 msgr2=0x5592b024bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:23.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 -- 172.21.15.89:0/371806435 shutdown_connections 2024-09-15T23:31:23.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 1 -- 172.21.15.89:0/371806435 wait complete. 2024-09-15T23:31:23.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.803+0000 7f817f869740 20 client.17889 trim_cache size 0 max 16384 2024-09-15T23:31:23.821 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:31:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 -- 172.21.15.89:0/1128569903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81640a3240 msgr2=0x7f81640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81640a3240 0x7f81640a5700 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8170014d40 tx=0x7f817006a300 comp rx=0 tx=0).stop 2024-09-15T23:31:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 -- 172.21.15.89:0/1128569903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8178106a80 msgr2=0x7f81781a44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8178106a80 0x7f81781a44b0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f817000b970 tx=0x7f817000be30 comp rx=0 tx=0).stop 2024-09-15T23:31:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 -- 172.21.15.89:0/1128569903 shutdown_connections 2024-09-15T23:31:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f81640a3240 0x7f81640a5700 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8178106a80 0x7f81781a44b0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8178101420 0x7f8178199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 --2- 172.21.15.89:0/1128569903 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8178100a80 0x7f8178199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:23.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 -- 172.21.15.89:0/1128569903 >> 172.21.15.89:0/1128569903 conn(0x7f81780fc7a0 msgr2=0x7f81780fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:23.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 -- 172.21.15.89:0/1128569903 shutdown_connections 2024-09-15T23:31:23.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:23.804+0000 7f817d490640 1 -- 172.21.15.89:0/1128569903 wait complete. 2024-09-15T23:31:24.497 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:31:24.745 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 -- 172.21.15.89:0/766749687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f215c100e20 msgr2=0x7f215c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 --2- 172.21.15.89:0/766749687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f215c100e20 0x7f215c101200 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f214c009900 tx=0x7f214c02a480 comp rx=0 tx=0).stop 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 -- 172.21.15.89:0/766749687 shutdown_connections 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 --2- 172.21.15.89:0/766749687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c102ef0 0x7f215c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 --2- 172.21.15.89:0/766749687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f215c102570 0x7f215c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 --2- 172.21.15.89:0/766749687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f215c100e20 0x7f215c101200 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 -- 172.21.15.89:0/766749687 >> 172.21.15.89:0/766749687 conn(0x7f215c0fc7d0 msgr2=0x7f215c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 -- 172.21.15.89:0/766749687 shutdown_connections 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 -- 172.21.15.89:0/766749687 wait complete. 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 Processor -- start 2024-09-15T23:31:25.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.201+0000 7f21624dd640 1 -- start start 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f21624dd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f215c100e20 0x7f215c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f21624dd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f215c102570 0x7f215c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f21624dd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c102ef0 0x7f215c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f21624dd640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f215c110fd0 con 0x7f215c102570 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f21624dd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f215c110e50 con 0x7f215c102ef0 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f21624dd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f215c111150 con 0x7f215c100e20 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f215b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f215c102570 0x7f215c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f215b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f215c102570 0x7f215c198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59354/0 (socket says 172.21.15.89:59354) 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f215b7fe640 1 -- 172.21.15.89:0/1000097513 learned_addr learned my addr 172.21.15.89:0/1000097513 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f215bfff640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f215c100e20 0x7f215c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f2160a7a640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c102ef0 0x7f215c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f215b7fe640 1 -- 172.21.15.89:0/1000097513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f215c100e20 msgr2=0x7f215c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f215b7fe640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f215c100e20 0x7f215c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f215b7fe640 1 -- 172.21.15.89:0/1000097513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c102ef0 msgr2=0x7f215c19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f215b7fe640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c102ef0 0x7f215c19ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.202+0000 7f215b7fe640 1 -- 172.21.15.89:0/1000097513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215c19d540 con 0x7f215c102570 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.203+0000 7f215b7fe640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f215c102570 0x7f215c198ad0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f215000d950 tx=0x7f215000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.203+0000 7f21597fa640 1 -- 172.21.15.89:0/1000097513 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150017070 con 0x7f215c102570 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.203+0000 7f21597fa640 1 -- 172.21.15.89:0/1000097513 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2150009c30 con 0x7f215c102570 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.203+0000 7f21624dd640 1 -- 172.21.15.89:0/1000097513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215c19d830 con 0x7f215c102570 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.203+0000 7f21597fa640 1 -- 172.21.15.89:0/1000097513 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215000f6b0 con 0x7f215c102570 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.203+0000 7f21624dd640 1 -- 172.21.15.89:0/1000097513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f215c1a5110 con 0x7f215c102570 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.205+0000 7f21597fa640 1 -- 172.21.15.89:0/1000097513 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f21500109d0 con 0x7f215c102570 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.205+0000 7f21648b6740 1 Processor -- start 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.205+0000 7f21648b6740 1 -- start start 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f21597fa640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 0x7f21440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f21648b6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65e5b40 0x5588d65e4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f21648b6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65e4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f21648b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65ef370 0x5588d65f1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f21648b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5588d654a2f0 con 0x5588d65ef370 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f21648b6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5588d64f1cf0 con 0x5588d65e5b40 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f21648b6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5588d65462b0 con 0x5588d65e45d0 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f21597fa640 1 -- 172.21.15.89:0/1000097513 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f21500cf8e0 con 0x7f215c102570 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 0x7f21440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65ef370 0x5588d65f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65ef370 0x5588d65f1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59368/0 (socket says 172.21.15.89:59368) 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 -- 172.21.15.89:0/3675961141 learned_addr learned my addr 172.21.15.89:0/3675961141 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f215b7fe640 1 --2- 172.21.15.89:0/3675961141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65e4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f215bfff640 1 --2- 172.21.15.89:0/3675961141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65e5b40 0x5588d65e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 -- 172.21.15.89:0/3675961141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 msgr2=0x5588d65e4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:25.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 --2- 172.21.15.89:0/3675961141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65e4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 -- 172.21.15.89:0/3675961141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65e5b40 msgr2=0x5588d65e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 --2- 172.21.15.89:0/3675961141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65e5b40 0x5588d65e4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 -- 172.21.15.89:0/3675961141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588d64d63a0 con 0x5588d65ef370 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f215b7fe640 1 --2- 172.21.15.89:0/3675961141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 0x7f21440a5700 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2154012c00 tx=0x7f2154009e00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.206+0000 7f2160a7a640 1 --2- 172.21.15.89:0/3675961141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65ef370 0x5588d65f1760 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f2154010800 tx=0x7f215401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9165ed9d3037200 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21427fc640 1 -- 172.21.15.89:0/3675961141 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2154024070 con 0x5588d65ef370 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21427fc640 1 -- 172.21.15.89:0/3675961141 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f215400dcf0 con 0x5588d65ef370 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21427fc640 1 -- 172.21.15.89:0/3675961141 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2154020c80 con 0x5588d65ef370 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21648b6740 1 -- 172.21.15.89:0/3675961141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65ef370 msgr2=0x5588d65f1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21648b6740 1 --2- 172.21.15.89:0/3675961141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65ef370 0x5588d65f1760 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f2154010800 tx=0x7f215401c000 comp rx=0 tx=0).stop 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21648b6740 1 -- 172.21.15.89:0/3675961141 shutdown_connections 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21648b6740 1 --2- 172.21.15.89:0/3675961141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65ef370 0x5588d65f1760 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21648b6740 1 --2- 172.21.15.89:0/3675961141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21648b6740 1 --2- 172.21.15.89:0/3675961141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65e5b40 0x5588d65e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21648b6740 1 -- 172.21.15.89:0/3675961141 >> 172.21.15.89:0/3675961141 conn(0x5588d65ad310 msgr2=0x5588d65e0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21648b6740 1 -- 172.21.15.89:0/3675961141 shutdown_connections 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.207+0000 7f21648b6740 1 -- 172.21.15.89:0/3675961141 wait complete. 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f21648b6740 1 -- start start 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f21648b6740 1 Processor -- start 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f21648b6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65eed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f21648b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65ec010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f21648b6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d65ec600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f21648b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5588d65f2010 con 0x5588d65e5b40 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f21648b6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5588d6505060 con 0x5588d65ef370 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f21648b6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5588d65f2190 con 0x5588d65e45d0 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f215b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65ec010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f215b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65ec010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59376/0 (socket says 172.21.15.89:59376) 2024-09-15T23:31:25.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.208+0000 7f215b7fe640 1 -- 172.21.15.89:0/1845807257 learned_addr learned my addr 172.21.15.89:0/1845807257 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f21648b6740 10 client.0 _ll_register_callbacks cb 0x5588d62588c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f2160a7a640 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d65ec600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f215bfff640 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65eed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f215b7fe640 1 -- 172.21.15.89:0/1845807257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 msgr2=0x5588d65eed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f215b7fe640 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65eed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f215b7fe640 1 -- 172.21.15.89:0/1845807257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 msgr2=0x5588d65ec600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f215b7fe640 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d65ec600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f215b7fe640 1 -- 172.21.15.89:0/1845807257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588d62bb3c0 con 0x5588d65e5b40 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f2160a7a640 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d65ec600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f215bfff640 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65eed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f215b7fe640 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65ec010 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f21500ccff0 tx=0x7f2150097e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dccfdd38324f9db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f213b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f213b7fe640 1 -- 172.21.15.89:0/1845807257 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150020040 con 0x5588d65e5b40 2024-09-15T23:31:25.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f213b7fe640 1 -- 172.21.15.89:0/1845807257 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f215004fb70 con 0x5588d65e5b40 2024-09-15T23:31:25.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f21648b6740 1 -- 172.21.15.89:0/1845807257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f215400dcf0 con 0x5588d65e5b40 2024-09-15T23:31:25.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f213b7fe640 1 -- 172.21.15.89:0/1845807257 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f215009fec0 con 0x5588d65e5b40 2024-09-15T23:31:25.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f21648b6740 10 client.17901 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:25.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f21648b6740 1 -- 172.21.15.89:0/1845807257 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588d6601f10 con 0x5588d65e5b40 2024-09-15T23:31:25.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.209+0000 7f213b7fe640 1 -- 172.21.15.89:0/1845807257 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f215400dcf0 con 0x5588d65e5b40 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.210+0000 7f213b7fe640 1 -- 172.21.15.89:0/1845807257 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2150097910 con 0x5588d65e5b40 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.210+0000 7f21648b6740 10 client.17901 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.210+0000 7f21648b6740 10 client.17901 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.210+0000 7f21648b6740 20 client.17901 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.210+0000 7f21648b6740 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5588d66033c0 0x5588d6605810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.210+0000 7f21648b6740 4 client.17901 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.210+0000 7f21648b6740 1 -- 172.21.15.89:0/1845807257 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588d6557860 con 0x5588d66033c0 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.210+0000 7f215bfff640 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5588d66033c0 0x5588d6605810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.211+0000 7f213b7fe640 10 client.17901 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.211+0000 7f215bfff640 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5588d66033c0 0x5588d6605810 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b898f3e32106b4ce server_cookie=bea0f407f769730a in_seq=0 out_seq=0 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.211+0000 7f213b7fe640 1 -- 172.21.15.89:0/1845807257 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5588d6557860 con 0x5588d66033c0 2024-09-15T23:31:25.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.211+0000 7f213b7fe640 10 client.17901 handle_command_reply: tid=0 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.211+0000 7f21648b6740 1 client.17901 shutdown 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.211+0000 7f21648b6740 1 -- 172.21.15.89:0/1845807257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 msgr2=0x5588d65ec010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.211+0000 7f21648b6740 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65ec010 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f21500ccff0 tx=0x7f2150097e60 comp rx=0 tx=0).stop 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.211+0000 7f21648b6740 1 -- 172.21.15.89:0/1845807257 shutdown_connections 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.212+0000 7f21648b6740 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d65ec600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.212+0000 7f21648b6740 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65ec010 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.212+0000 7f21648b6740 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5588d66033c0 0x5588d6605810 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.212+0000 7f21648b6740 1 --2- 172.21.15.89:0/1845807257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65eed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.212+0000 7f21648b6740 1 -- 172.21.15.89:0/1845807257 >> 172.21.15.89:0/1845807257 conn(0x5588d65ad310 msgr2=0x5588d65e6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.212+0000 7f213b7fe640 0 client.17901 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.212+0000 7f21648b6740 1 -- 172.21.15.89:0/1845807257 shutdown_connections 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.212+0000 7f21648b6740 1 -- 172.21.15.89:0/1845807257 wait complete. 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.212+0000 7f21648b6740 20 client.17901 trim_cache size 0 max 16384 2024-09-15T23:31:25.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f21648b6740 1 Processor -- start 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f21648b6740 1 -- start start 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f21648b6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65ff790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f21648b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65edf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f21648b6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d660b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f21648b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f215c1acc00 con 0x5588d65e5b40 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f21648b6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f215c1ae740 con 0x5588d65ef370 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f21648b6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f215c1ad2d0 con 0x5588d65e45d0 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f215b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f215b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65edf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59386/0 (socket says 172.21.15.89:59386) 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.225+0000 7f215b7fe640 1 -- 172.21.15.89:0/1817153957 learned_addr learned my addr 172.21.15.89:0/1817153957 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.226+0000 7f215bfff640 1 --2- 172.21.15.89:0/1817153957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.226+0000 7f215b7fe640 1 -- 172.21.15.89:0/1817153957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 msgr2=0x5588d65ff790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.226+0000 7f215b7fe640 1 --2- 172.21.15.89:0/1817153957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65ff790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.226+0000 7f215b7fe640 1 -- 172.21.15.89:0/1817153957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 msgr2=0x5588d660b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.226+0000 7f215b7fe640 1 --2- 172.21.15.89:0/1817153957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d660b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.226+0000 7f215b7fe640 1 -- 172.21.15.89:0/1817153957 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588d65ec970 con 0x5588d65e5b40 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.226+0000 7f215b7fe640 1 --2- 172.21.15.89:0/1817153957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65edf80 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f215009b8c0 tx=0x7f2150000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81cb45ad896c3d55 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.226+0000 7f2141ffb640 1 -- 172.21.15.89:0/1817153957 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21500b9070 con 0x5588d65e5b40 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.226+0000 7f2141ffb640 1 -- 172.21.15.89:0/1817153957 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21500a4780 con 0x5588d65e5b40 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.226+0000 7f2141ffb640 1 -- 172.21.15.89:0/1817153957 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150014030 con 0x5588d65e5b40 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 -- 172.21.15.89:0/1817153957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 msgr2=0x5588d65edf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 --2- 172.21.15.89:0/1817153957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65edf80 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f215009b8c0 tx=0x7f2150000bc0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 -- 172.21.15.89:0/1817153957 shutdown_connections 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 --2- 172.21.15.89:0/1817153957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d660b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 --2- 172.21.15.89:0/1817153957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65edf80 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 --2- 172.21.15.89:0/1817153957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 -- 172.21.15.89:0/1817153957 >> 172.21.15.89:0/1817153957 conn(0x5588d65ad310 msgr2=0x5588d65ed870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:25.243 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 -- 172.21.15.89:0/1817153957 shutdown_connections 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 -- 172.21.15.89:0/1817153957 wait complete. 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 -- start start 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.227+0000 7f21648b6740 1 Processor -- start 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f21648b6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f21648b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65e8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f21648b6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d65e9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f21648b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f215c1ae070 con 0x5588d65e5b40 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f21648b6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f215c1ad9a0 con 0x5588d65ef370 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f21648b6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f215c1aabe0 con 0x5588d65e45d0 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f21648b6740 10 client.0 _ll_register_callbacks cb 0x7f2150097740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f215b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65e8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f215b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65e8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59390/0 (socket says 172.21.15.89:59390) 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f215b7fe640 1 -- 172.21.15.89:0/3389781572 learned_addr learned my addr 172.21.15.89:0/3389781572 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:25.244 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f215bfff640 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f2160a7a640 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d65e9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f215b7fe640 1 -- 172.21.15.89:0/3389781572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 msgr2=0x5588d65fc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f215b7fe640 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65fc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f215b7fe640 1 -- 172.21.15.89:0/3389781572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 msgr2=0x5588d65e9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f215b7fe640 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d65e9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f215b7fe640 1 -- 172.21.15.89:0/3389781572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588d666d0b0 con 0x5588d65e5b40 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f215b7fe640 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65e8e50 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f215009bd30 tx=0x7f2150097e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab896e42e93ef35 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.228+0000 7f213b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f213b7fe640 1 -- 172.21.15.89:0/3389781572 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215007f060 con 0x5588d65e5b40 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f213b7fe640 1 -- 172.21.15.89:0/3389781572 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f21500860b0 con 0x5588d65e5b40 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f21648b6740 1 -- 172.21.15.89:0/3389781572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21500a4780 con 0x5588d65e5b40 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f213b7fe640 1 -- 172.21.15.89:0/3389781572 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f21500b9050 con 0x5588d65e5b40 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f21648b6740 10 client.17907 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:25.245 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f21648b6740 1 -- 172.21.15.89:0/3389781572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588d666d0b0 con 0x5588d65e5b40 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f213b7fe640 1 -- 172.21.15.89:0/3389781572 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21500a4780 con 0x5588d65e5b40 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f213b7fe640 1 -- 172.21.15.89:0/3389781572 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2150010690 con 0x5588d65e5b40 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f21648b6740 10 client.17907 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f21648b6740 10 client.17907 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f21648b6740 20 client.17907 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f21648b6740 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5588d666e4f0 0x5588d66708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f21648b6740 4 client.17907 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f21648b6740 1 -- 172.21.15.89:0/3389781572 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5588d6605ba0 con 0x5588d666e4f0 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.229+0000 7f215bfff640 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5588d666e4f0 0x5588d66708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f213b7fe640 10 client.17907 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f215bfff640 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5588d666e4f0 0x5588d66708e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a90a07e1fef1a7e9 server_cookie=3f0530bcdf719193 in_seq=0 out_seq=0 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f213b7fe640 1 -- 172.21.15.89:0/3389781572 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5588d6605ba0 con 0x5588d666e4f0 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f213b7fe640 10 client.17907 handle_command_reply: tid=0 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 client.17907 shutdown 2024-09-15T23:31:25.246 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 -- 172.21.15.89:0/3389781572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 msgr2=0x5588d65e8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65e8e50 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f215009bd30 tx=0x7f2150097e60 comp rx=0 tx=0).stop 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 -- 172.21.15.89:0/3389781572 shutdown_connections 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5588d65ef370 0x5588d65e9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5588d65e5b40 0x5588d65e8e50 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5588d666e4f0 0x5588d66708e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f213b7fe640 0 client.17907 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 --2- 172.21.15.89:0/3389781572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5588d65e45d0 0x5588d65fc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 -- 172.21.15.89:0/3389781572 >> 172.21.15.89:0/3389781572 conn(0x5588d65ad310 msgr2=0x5588d65ea4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 -- 172.21.15.89:0/3389781572 shutdown_connections 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 1 -- 172.21.15.89:0/3389781572 wait complete. 2024-09-15T23:31:25.247 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.230+0000 7f21648b6740 20 client.17907 trim_cache size 0 max 16384 2024-09-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 -- 172.21.15.89:0/1000097513 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 msgr2=0x7f21440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 0x7f21440a5700 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2154012c00 tx=0x7f2154009e00 comp rx=0 tx=0).stop 2024-09-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 -- 172.21.15.89:0/1000097513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f215c102570 msgr2=0x7f215c198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f215c102570 0x7f215c198ad0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f215000d950 tx=0x7f215000de10 comp rx=0 tx=0).stop 2024-09-15T23:31:25.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 -- 172.21.15.89:0/1000097513 shutdown_connections 2024-09-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 0x7f21440a5700 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f215c102ef0 0x7f215c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f215c102570 0x7f215c198ad0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 --2- 172.21.15.89:0/1000097513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f215c100e20 0x7f215c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 -- 172.21.15.89:0/1000097513 >> 172.21.15.89:0/1000097513 conn(0x7f215c0fc7d0 msgr2=0x7f215c06a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 -- 172.21.15.89:0/1000097513 shutdown_connections 2024-09-15T23:31:25.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:25.231+0000 7f21624dd640 1 -- 172.21.15.89:0/1000097513 wait complete. 2024-09-15T23:31:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:25 smithi120 ceph-mon[35250]: pgmap v572: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:25 smithi174 ceph-mon[35586]: pgmap v572: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:25.541 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:25 smithi089 ceph-mon[30933]: pgmap v572: 129 pgs: 129 active+clean; 3.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:25.896 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:31:26.142 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.628+0000 7f93af538640 1 -- 172.21.15.89:0/942462321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a8100a80 msgr2=0x7f93a8100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.628+0000 7f93af538640 1 --2- 172.21.15.89:0/942462321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a8100a80 0x7f93a8100ee0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009930 tx=0x7f93a402a420 comp rx=0 tx=0).stop 2024-09-15T23:31:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.628+0000 7f93af538640 1 -- 172.21.15.89:0/942462321 shutdown_connections 2024-09-15T23:31:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.628+0000 7f93af538640 1 --2- 172.21.15.89:0/942462321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a8101420 0x7f93a810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.628+0000 7f93af538640 1 --2- 172.21.15.89:0/942462321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a8100a80 0x7f93a8100ee0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.628+0000 7f93af538640 1 --2- 172.21.15.89:0/942462321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a8106a80 0x7f93a8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.628+0000 7f93af538640 1 -- 172.21.15.89:0/942462321 >> 172.21.15.89:0/942462321 conn(0x7f93a80fc7a0 msgr2=0x7f93a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.628+0000 7f93af538640 1 -- 172.21.15.89:0/942462321 shutdown_connections 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.628+0000 7f93af538640 1 -- 172.21.15.89:0/942462321 wait complete. 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.628+0000 7f93af538640 1 Processor -- start 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93af538640 1 -- start start 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93af538640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a8100a80 0x7f93a8198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93af538640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a8101420 0x7f93a81989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93af538640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a8106a80 0x7f93a819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93af538640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f93a8110e40 con 0x7f93a8106a80 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93af538640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93a8110cc0 con 0x7f93a8101420 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93af538640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93a8110fc0 con 0x7f93a8100a80 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93adad5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a8106a80 0x7f93a819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93adad5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a8106a80 0x7f93a819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59414/0 (socket says 172.21.15.89:59414) 2024-09-15T23:31:26.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93adad5640 1 -- 172.21.15.89:0/2442567928 learned_addr learned my addr 172.21.15.89:0/2442567928 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93ad2d4640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a8100a80 0x7f93a8198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93acad3640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a8101420 0x7f93a81989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93adad5640 1 -- 172.21.15.89:0/2442567928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a8100a80 msgr2=0x7f93a8198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93adad5640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a8100a80 0x7f93a8198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93adad5640 1 -- 172.21.15.89:0/2442567928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a8101420 msgr2=0x7f93a81989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93adad5640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a8101420 0x7f93a81989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93adad5640 1 -- 172.21.15.89:0/2442567928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a819d410 con 0x7f93a8106a80 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.629+0000 7f93adad5640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a8106a80 0x7f93a819cd30 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f93a000cad0 tx=0x7f93a000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.630+0000 7f939e7fc640 1 -- 172.21.15.89:0/2442567928 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a000ae20 con 0x7f93a8106a80 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.630+0000 7f939e7fc640 1 -- 172.21.15.89:0/2442567928 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93a0005020 con 0x7f93a8106a80 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.630+0000 7f93af538640 1 -- 172.21.15.89:0/2442567928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93a819d700 con 0x7f93a8106a80 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.630+0000 7f939e7fc640 1 -- 172.21.15.89:0/2442567928 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a000f6f0 con 0x7f93a8106a80 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.630+0000 7f93af538640 1 -- 172.21.15.89:0/2442567928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93a81a4fe0 con 0x7f93a8106a80 2024-09-15T23:31:26.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.632+0000 7f939e7fc640 1 -- 172.21.15.89:0/2442567928 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f93a00051d0 con 0x7f93a8106a80 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.632+0000 7f93b1911740 1 Processor -- start 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.632+0000 7f93b1911740 1 -- start start 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93b1911740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f1b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93b1911740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 0x558438f1ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93b1911740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f28760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93b1911740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558438e812f0 con 0x558438f26370 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93b1911740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558438e28cf0 con 0x558438f1cb40 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93b1911740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558438e7d2b0 con 0x558438f1b5d0 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f939e7fc640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93940a32c0 0x7f93940a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93acad3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f28760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93ad2d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 0x558438f1ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93ad2d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 0x558438f1ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:44644/0 (socket says 172.21.15.89:44644) 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93acad3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f28760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59430/0 (socket says 172.21.15.89:59430) 2024-09-15T23:31:26.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93ad2d4640 1 -- 172.21.15.89:0/658700713 learned_addr learned my addr 172.21.15.89:0/658700713 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93acad3640 1 --2- 172.21.15.89:0/658700713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f1b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f939e7fc640 1 -- 172.21.15.89:0/2442567928 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f93a00cfc60 con 0x7f93a8106a80 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93adad5640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93940a32c0 0x7f93940a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93ad2d4640 1 -- 172.21.15.89:0/658700713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 msgr2=0x558438f1b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93ad2d4640 1 --2- 172.21.15.89:0/658700713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f1b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93ad2d4640 1 -- 172.21.15.89:0/658700713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 msgr2=0x558438f28760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93ad2d4640 1 --2- 172.21.15.89:0/658700713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f28760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93ad2d4640 1 -- 172.21.15.89:0/658700713 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558438e0d3a0 con 0x558438f1b5d0 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93acad3640 1 --2- 172.21.15.89:0/658700713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f1b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93ad2d4640 1 --2- 172.21.15.89:0/658700713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 0x558438f1ba10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f939800bc90 tx=0x7f93980107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ef5ed242b725697 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.633+0000 7f93adad5640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93940a32c0 0x7f93940a5780 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f93a00cd370 tx=0x7f93a0005b40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93937fe640 1 -- 172.21.15.89:0/658700713 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939801c480 con 0x558438f1b5d0 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93937fe640 1 -- 172.21.15.89:0/658700713 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f939801cb00 con 0x558438f1b5d0 2024-09-15T23:31:26.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93937fe640 1 -- 172.21.15.89:0/658700713 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9398011bb0 con 0x558438f1b5d0 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93b1911740 1 -- 172.21.15.89:0/658700713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 msgr2=0x558438f1ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93b1911740 1 --2- 172.21.15.89:0/658700713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 0x558438f1ba10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f939800bc90 tx=0x7f93980107a0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93b1911740 1 -- 172.21.15.89:0/658700713 shutdown_connections 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93b1911740 1 --2- 172.21.15.89:0/658700713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f28760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93b1911740 1 --2- 172.21.15.89:0/658700713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 0x558438f1ba10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93b1911740 1 --2- 172.21.15.89:0/658700713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f1b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93b1911740 1 -- 172.21.15.89:0/658700713 >> 172.21.15.89:0/658700713 conn(0x558438ee4310 msgr2=0x558438f17c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93b1911740 1 -- 172.21.15.89:0/658700713 shutdown_connections 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.634+0000 7f93b1911740 1 -- 172.21.15.89:0/658700713 wait complete. 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93b1911740 1 -- start start 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93b1911740 1 Processor -- start 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93b1911740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 0x558438f25d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93b1911740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f23020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93b1911740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f23610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93b1911740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558438f29070 con 0x558438f26370 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93b1911740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558438e3c060 con 0x558438f1cb40 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93b1911740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558438f291f0 con 0x558438f1b5d0 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93b1911740 10 client.0 _ll_register_callbacks cb 0x558438b8f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93acad3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f23610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93acad3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f23610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59432/0 (socket says 172.21.15.89:59432) 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.635+0000 7f93acad3640 1 -- 172.21.15.89:0/2024714545 learned_addr learned my addr 172.21.15.89:0/2024714545 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93ad2d4640 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f23020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93acad3640 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 0x558438f25d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93acad3640 1 -- 172.21.15.89:0/2024714545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 msgr2=0x558438f25d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93acad3640 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 0x558438f25d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93acad3640 1 -- 172.21.15.89:0/2024714545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 msgr2=0x558438f23020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:26.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93acad3640 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f23020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93acad3640 1 -- 172.21.15.89:0/2024714545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558438bf23c0 con 0x558438f26370 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93acad3640 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f23610 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f93a402a940 tx=0x7f93a4009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e9bb15324f8df6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f9390ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f9390ff9640 1 -- 172.21.15.89:0/2024714545 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a4035070 con 0x558438f26370 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f9390ff9640 1 -- 172.21.15.89:0/2024714545 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f93a4004250 con 0x558438f26370 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93b1911740 1 -- 172.21.15.89:0/2024714545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f939801cb00 con 0x558438f26370 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f9390ff9640 1 -- 172.21.15.89:0/2024714545 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f93a4004450 con 0x558438f26370 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93b1911740 10 client.17919 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f93b1911740 1 -- 172.21.15.89:0/2024714545 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558438f34b10 con 0x558438f26370 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.636+0000 7f9390ff9640 1 -- 172.21.15.89:0/2024714545 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f939801cb00 con 0x558438f26370 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.637+0000 7f9390ff9640 1 -- 172.21.15.89:0/2024714545 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f93a4003b10 con 0x558438f26370 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.637+0000 7f93b1911740 10 client.17919 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:26.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.637+0000 7f93b1911740 10 client.17919 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.637+0000 7f93b1911740 20 client.17919 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.637+0000 7f93b1911740 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558438f35f20 0x558438f38350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.637+0000 7f93b1911740 4 client.17919 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.637+0000 7f93b1911740 1 -- 172.21.15.89:0/2024714545 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558438e8e860 con 0x558438f35f20 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.637+0000 7f93ad2d4640 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558438f35f20 0x558438f38350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.637+0000 7f93ad2d4640 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558438f35f20 0x558438f38350 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5c4f5d9b4bd906bc server_cookie=10c3d49d8e6860e0 in_seq=0 out_seq=0 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.637+0000 7f9390ff9640 10 client.17919 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f9390ff9640 1 -- 172.21.15.89:0/2024714545 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558438e8e860 con 0x558438f35f20 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f9390ff9640 10 client.17919 handle_command_reply: tid=0 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f93b1911740 1 client.17919 shutdown 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f93b1911740 1 -- 172.21.15.89:0/2024714545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 msgr2=0x558438f23610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f93b1911740 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f23610 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f93a402a940 tx=0x7f93a4009680 comp rx=0 tx=0).stop 2024-09-15T23:31:26.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f93b1911740 1 -- 172.21.15.89:0/2024714545 shutdown_connections 2024-09-15T23:31:26.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f93b1911740 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558438f35f20 0x558438f38350 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f93b1911740 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f26370 0x558438f23610 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f93b1911740 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f23020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f9390ff9640 0 client.17919 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:31:26.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f93b1911740 1 --2- 172.21.15.89:0/2024714545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f1b5d0 0x558438f25d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f93b1911740 1 -- 172.21.15.89:0/2024714545 >> 172.21.15.89:0/2024714545 conn(0x558438ee4310 msgr2=0x558438f1d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:26.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.638+0000 7f93b1911740 1 -- 172.21.15.89:0/2024714545 shutdown_connections 2024-09-15T23:31:26.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.639+0000 7f93b1911740 1 -- 172.21.15.89:0/2024714545 wait complete. 2024-09-15T23:31:26.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.639+0000 7f93b1911740 20 client.17919 trim_cache size 0 max 16384 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.651+0000 7f93b1911740 1 Processor -- start 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.651+0000 7f93b1911740 1 -- start start 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93b1911740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f17e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93b1911740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f32310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93b1911740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f26370 0x558438f1e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93b1911740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f93a81acbf0 con 0x558438f1b5d0 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93b1911740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93a81ae730 con 0x558438f1cb40 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93b1911740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93a81ad2c0 con 0x558438f26370 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93acad3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f17e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93acad3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f17e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59444/0 (socket says 172.21.15.89:59444) 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93acad3640 1 -- 172.21.15.89:0/3948058176 learned_addr learned my addr 172.21.15.89:0/3948058176 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93acad3640 1 -- 172.21.15.89:0/3948058176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f26370 msgr2=0x558438f1e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93acad3640 1 --2- 172.21.15.89:0/3948058176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f26370 0x558438f1e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93acad3640 1 -- 172.21.15.89:0/3948058176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 msgr2=0x558438f32310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93acad3640 1 --2- 172.21.15.89:0/3948058176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f32310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93acad3640 1 -- 172.21.15.89:0/3948058176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558438f1c760 con 0x558438f1b5d0 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f93acad3640 1 --2- 172.21.15.89:0/3948058176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f17e10 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f93a40095d0 tx=0x7f93a4009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3d056b2f6bab45 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f9392ffd640 1 -- 172.21.15.89:0/3948058176 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a4035070 con 0x558438f1b5d0 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f9392ffd640 1 -- 172.21.15.89:0/3948058176 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93a4004250 con 0x558438f1b5d0 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.652+0000 7f9392ffd640 1 -- 172.21.15.89:0/3948058176 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a40053a0 con 0x558438f1b5d0 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 -- 172.21.15.89:0/3948058176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 msgr2=0x558438f17e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 --2- 172.21.15.89:0/3948058176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f17e10 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f93a40095d0 tx=0x7f93a4009600 comp rx=0 tx=0).stop 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 -- 172.21.15.89:0/3948058176 shutdown_connections 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 --2- 172.21.15.89:0/3948058176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f26370 0x558438f1e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 --2- 172.21.15.89:0/3948058176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f32310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 --2- 172.21.15.89:0/3948058176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f17e10 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 -- 172.21.15.89:0/3948058176 >> 172.21.15.89:0/3948058176 conn(0x558438ee4310 msgr2=0x558438f17700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 -- 172.21.15.89:0/3948058176 shutdown_connections 2024-09-15T23:31:26.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 -- 172.21.15.89:0/3948058176 wait complete. 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 -- start start 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.653+0000 7f93b1911740 1 Processor -- start 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93b1911740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f2eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93b1911740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f1fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93b1911740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f26370 0x558438f201e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93b1911740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f93a81ae060 con 0x558438f1b5d0 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93b1911740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93a81ad990 con 0x558438f1cb40 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93b1911740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93a81aabd0 con 0x558438f26370 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93b1911740 10 client.0 _ll_register_callbacks cb 0x7f93a4009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93acad3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f2eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93acad3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f2eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59450/0 (socket says 172.21.15.89:59450) 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93acad3640 1 -- 172.21.15.89:0/1690360341 learned_addr learned my addr 172.21.15.89:0/1690360341 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:26.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93ad2d4640 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f1fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93acad3640 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f26370 0x558438f201e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93acad3640 1 -- 172.21.15.89:0/1690360341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f26370 msgr2=0x558438f201e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93acad3640 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f26370 0x558438f201e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93acad3640 1 -- 172.21.15.89:0/1690360341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 msgr2=0x558438f1fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93acad3640 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f1fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93acad3640 1 -- 172.21.15.89:0/1690360341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558438f209e0 con 0x558438f1b5d0 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f93acad3640 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f2eb50 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f93a404eff0 tx=0x7f93a404dee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f03a1d3d2e497e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.654+0000 7f9390ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f9390ff9640 1 -- 172.21.15.89:0/1690360341 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a4035040 con 0x558438f1b5d0 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f9390ff9640 1 -- 172.21.15.89:0/1690360341 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f93a404dae0 con 0x558438f1b5d0 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f93b1911740 1 -- 172.21.15.89:0/1690360341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93a4004250 con 0x558438f1b5d0 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f9390ff9640 1 -- 172.21.15.89:0/1690360341 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f93a40315b0 con 0x558438f1b5d0 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f93b1911740 10 client.17925 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:26.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f93b1911740 1 -- 172.21.15.89:0/1690360341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558438fa0ab0 con 0x558438f1b5d0 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f9390ff9640 1 -- 172.21.15.89:0/1690360341 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93a4004250 con 0x558438f1b5d0 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f9390ff9640 1 -- 172.21.15.89:0/1690360341 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f93a4063070 con 0x558438f1b5d0 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f93b1911740 10 client.17925 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f93b1911740 10 client.17925 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f93b1911740 20 client.17925 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f93b1911740 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558438fa10f0 0x558438fa3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f93b1911740 4 client.17925 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.655+0000 7f93b1911740 1 -- 172.21.15.89:0/1690360341 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558438e88d70 con 0x558438fa10f0 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93ad2d4640 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558438fa10f0 0x558438fa3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93ad2d4640 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558438fa10f0 0x558438fa3510 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73a09003161ce8b7 server_cookie=1276500161243986 in_seq=0 out_seq=0 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f9390ff9640 10 client.17925 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f9390ff9640 1 -- 172.21.15.89:0/1690360341 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558438e88d70 con 0x558438fa10f0 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f9390ff9640 10 client.17925 handle_command_reply: tid=0 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93b1911740 1 client.17925 shutdown 2024-09-15T23:31:26.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93b1911740 1 -- 172.21.15.89:0/1690360341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 msgr2=0x558438f2eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93b1911740 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f2eb50 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f93a404eff0 tx=0x7f93a404dee0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93b1911740 1 -- 172.21.15.89:0/1690360341 shutdown_connections 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93b1911740 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558438fa10f0 0x558438fa3510 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93b1911740 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558438f26370 0x558438f201e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93b1911740 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558438f1cb40 0x558438f1fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f9390ff9640 0 client.17925 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93b1911740 1 --2- 172.21.15.89:0/1690360341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558438f1b5d0 0x558438f2eb50 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93b1911740 1 -- 172.21.15.89:0/1690360341 >> 172.21.15.89:0/1690360341 conn(0x558438ee4310 msgr2=0x558438f250a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.656+0000 7f93b1911740 1 -- 172.21.15.89:0/1690360341 shutdown_connections 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93b1911740 1 -- 172.21.15.89:0/1690360341 wait complete. 2024-09-15T23:31:26.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93b1911740 20 client.17925 trim_cache size 0 max 16384 2024-09-15T23:31:26.668 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:31:26.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 -- 172.21.15.89:0/2442567928 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93940a32c0 msgr2=0x7f93940a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93940a32c0 0x7f93940a5780 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f93a00cd370 tx=0x7f93a0005b40 comp rx=0 tx=0).stop 2024-09-15T23:31:26.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 -- 172.21.15.89:0/2442567928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a8106a80 msgr2=0x7f93a819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a8106a80 0x7f93a819cd30 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f93a000cad0 tx=0x7f93a000cf90 comp rx=0 tx=0).stop 2024-09-15T23:31:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 -- 172.21.15.89:0/2442567928 shutdown_connections 2024-09-15T23:31:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93940a32c0 0x7f93940a5780 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a8106a80 0x7f93a819cd30 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a8101420 0x7f93a81989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 --2- 172.21.15.89:0/2442567928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a8100a80 0x7f93a8198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 -- 172.21.15.89:0/2442567928 >> 172.21.15.89:0/2442567928 conn(0x7f93a80fc7a0 msgr2=0x7f93a8104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 -- 172.21.15.89:0/2442567928 shutdown_connections 2024-09-15T23:31:26.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:26.657+0000 7f93af538640 1 -- 172.21.15.89:0/2442567928 wait complete. 2024-09-15T23:31:27.367 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:31:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:27 smithi120 ceph-mon[35250]: pgmap v573: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:27.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:27 smithi174 ceph-mon[35586]: pgmap v573: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:27.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:27 smithi089 ceph-mon[30933]: pgmap v573: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:27.615 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:28.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.140+0000 7f8952865640 1 -- 172.21.15.89:0/2601508682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f894c102790 msgr2=0x7f894c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.140+0000 7f8952865640 1 --2- 172.21.15.89:0/2601508682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f894c102790 0x7f894c1067e0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f8934009900 tx=0x7f893402a410 comp rx=0 tx=0).stop 2024-09-15T23:31:28.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 -- 172.21.15.89:0/2601508682 shutdown_connections 2024-09-15T23:31:28.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 --2- 172.21.15.89:0/2601508682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f894c106f10 0x7f894c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 --2- 172.21.15.89:0/2601508682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f894c102790 0x7f894c1067e0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 --2- 172.21.15.89:0/2601508682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f894c101de0 0x7f894c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 -- 172.21.15.89:0/2601508682 >> 172.21.15.89:0/2601508682 conn(0x7f894c0fd8c0 msgr2=0x7f894c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 -- 172.21.15.89:0/2601508682 shutdown_connections 2024-09-15T23:31:28.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 -- 172.21.15.89:0/2601508682 wait complete. 2024-09-15T23:31:28.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 Processor -- start 2024-09-15T23:31:28.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 -- start start 2024-09-15T23:31:28.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f894c101de0 0x7f894c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f894c102790 0x7f894c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f894c106f10 0x7f894c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f894c111f70 con 0x7f894c102790 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f894c111df0 con 0x7f894c101de0 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.141+0000 7f8952865640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f894c1120f0 con 0x7f894c106f10 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8943fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f894c102790 0x7f894c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8943fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f894c102790 0x7f894c1989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59460/0 (socket says 172.21.15.89:59460) 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8943fff640 1 -- 172.21.15.89:0/2968272907 learned_addr learned my addr 172.21.15.89:0/2968272907 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f894bfff640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f894c101de0 0x7f894c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8943fff640 1 -- 172.21.15.89:0/2968272907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f894c106f10 msgr2=0x7f894c19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8950e02640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f894c106f10 0x7f894c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8943fff640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f894c106f10 0x7f894c19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8943fff640 1 -- 172.21.15.89:0/2968272907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f894c101de0 msgr2=0x7f894c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8943fff640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f894c101de0 0x7f894c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8943fff640 1 -- 172.21.15.89:0/2968272907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f894c19d420 con 0x7f894c102790 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8950e02640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f894c106f10 0x7f894c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8943fff640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f894c102790 0x7f894c1989b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f8934005ab0 tx=0x7f89340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8949ffb640 1 -- 172.21.15.89:0/2968272907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893403f030 con 0x7f894c102790 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8952865640 1 -- 172.21.15.89:0/2968272907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f894c19d6b0 con 0x7f894c102790 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8949ffb640 1 -- 172.21.15.89:0/2968272907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8934003ce0 con 0x7f894c102790 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8949ffb640 1 -- 172.21.15.89:0/2968272907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934035070 con 0x7f894c102790 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.142+0000 7f8952865640 1 -- 172.21.15.89:0/2968272907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f894c1a4ff0 con 0x7f894c102790 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.145+0000 7f8949ffb640 1 -- 172.21.15.89:0/2968272907 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8934042030 con 0x7f894c102790 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.145+0000 7f8954c3e740 1 Processor -- start 2024-09-15T23:31:28.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.145+0000 7f8954c3e740 1 -- start start 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.145+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 0x55f83aef7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.145+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef75d0 0x55f83aef7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.145+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af04760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.145+0000 7f8954c3e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f83ae5d2f0 con 0x55f83af02370 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.145+0000 7f8954c3e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f83ae04cf0 con 0x55f83aef75d0 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.145+0000 7f8954c3e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f83ae592b0 con 0x55f83aef8b40 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8943fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af04760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8949ffb640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89380a3240 0x7f89380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8943fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af04760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59476/0 (socket says 172.21.15.89:59476) 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8943fff640 1 -- 172.21.15.89:0/2314390536 learned_addr learned my addr 172.21.15.89:0/2314390536 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8950e02640 1 --2- 172.21.15.89:0/2314390536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 0x55f83aef7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8950e02640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89380a3240 0x7f89380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f894bfff640 1 --2- 172.21.15.89:0/2314390536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef75d0 0x55f83aef7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8949ffb640 1 -- 172.21.15.89:0/2968272907 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f89340f0650 con 0x7f894c102790 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8943fff640 1 -- 172.21.15.89:0/2314390536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 msgr2=0x55f83aef7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8943fff640 1 --2- 172.21.15.89:0/2314390536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 0x55f83aef7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8943fff640 1 -- 172.21.15.89:0/2314390536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef75d0 msgr2=0x55f83aef7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8943fff640 1 --2- 172.21.15.89:0/2314390536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef75d0 0x55f83aef7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8943fff640 1 -- 172.21.15.89:0/2314390536 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f83ade93a0 con 0x55f83af02370 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f894bfff640 1 --2- 172.21.15.89:0/2314390536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef75d0 0x55f83aef7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8943fff640 1 --2- 172.21.15.89:0/2314390536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af04760 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f89340b91f0 tx=0x7f8934009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5f2e3ebc306bad server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f8950e02640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89380a3240 0x7f89380a5700 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8944005fd0 tx=0x7f8944005c80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f89427fc640 1 -- 172.21.15.89:0/2314390536 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934049070 con 0x55f83af02370 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f89427fc640 1 -- 172.21.15.89:0/2314390536 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89340bcf20 con 0x55f83af02370 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.146+0000 7f89427fc640 1 -- 172.21.15.89:0/2314390536 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89340f0300 con 0x55f83af02370 2024-09-15T23:31:28.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.147+0000 7f8954c3e740 1 -- 172.21.15.89:0/2314390536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 msgr2=0x55f83af04760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.147+0000 7f8954c3e740 1 --2- 172.21.15.89:0/2314390536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af04760 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f89340b91f0 tx=0x7f8934009680 comp rx=0 tx=0).stop 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.147+0000 7f8954c3e740 1 -- 172.21.15.89:0/2314390536 shutdown_connections 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.147+0000 7f8954c3e740 1 --2- 172.21.15.89:0/2314390536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af04760 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.147+0000 7f8954c3e740 1 --2- 172.21.15.89:0/2314390536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef75d0 0x55f83aef7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.147+0000 7f8954c3e740 1 --2- 172.21.15.89:0/2314390536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 0x55f83aef7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.147+0000 7f8954c3e740 1 -- 172.21.15.89:0/2314390536 >> 172.21.15.89:0/2314390536 conn(0x55f83aec0310 msgr2=0x55f83aef3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.147+0000 7f8954c3e740 1 -- 172.21.15.89:0/2314390536 shutdown_connections 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.147+0000 7f8954c3e740 1 -- 172.21.15.89:0/2314390536 wait complete. 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f8954c3e740 1 -- start start 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f8954c3e740 1 Processor -- start 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef75d0 0x55f83af01de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 0x55f83aeff090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83af02370 0x55f83aeff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f8954c3e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f83ae5d2f0 con 0x55f83aef75d0 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f8954c3e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f83ae18060 con 0x55f83af02370 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f8954c3e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f83af04e30 con 0x55f83aef8b40 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f8954c3e740 10 client.0 _ll_register_callbacks cb 0x55f83ab6b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f894bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef75d0 0x55f83af01de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f894bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef75d0 0x55f83af01de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59486/0 (socket says 172.21.15.89:59486) 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f894bfff640 1 -- 172.21.15.89:0/2803187322 learned_addr learned my addr 172.21.15.89:0/2803187322 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.148+0000 7f8943fff640 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 0x55f83aeff090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8950e02640 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83af02370 0x55f83aeff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f894bfff640 1 -- 172.21.15.89:0/2803187322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 msgr2=0x55f83aeff090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f894bfff640 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 0x55f83aeff090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f894bfff640 1 -- 172.21.15.89:0/2803187322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83af02370 msgr2=0x55f83aeff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f894bfff640 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83af02370 0x55f83aeff7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f894bfff640 1 -- 172.21.15.89:0/2803187322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f83abce3c0 con 0x55f83aef75d0 2024-09-15T23:31:28.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8943fff640 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 0x55f83aeff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f894bfff640 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef75d0 0x55f83af01de0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f893c012780 tx=0x7f893c00af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1009f6afbc498ae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f892b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f892b7fe640 1 -- 172.21.15.89:0/2803187322 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c010dc0 con 0x55f83aef75d0 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f892b7fe640 1 -- 172.21.15.89:0/2803187322 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f893c005020 con 0x55f83aef75d0 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8954c3e740 1 -- 172.21.15.89:0/2803187322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89340bcf20 con 0x55f83aef75d0 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f892b7fe640 1 -- 172.21.15.89:0/2803187322 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f893c025050 con 0x55f83aef75d0 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8954c3e740 10 client.17937 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8954c3e740 1 -- 172.21.15.89:0/2803187322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f83af10840 con 0x55f83aef75d0 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f892b7fe640 1 -- 172.21.15.89:0/2803187322 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89340bcf20 con 0x55f83aef75d0 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f892b7fe640 1 -- 172.21.15.89:0/2803187322 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f893c01a430 con 0x55f83aef75d0 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8954c3e740 10 client.17937 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8954c3e740 10 client.17937 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8954c3e740 20 client.17937 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:28.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8954c3e740 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f83af11d50 0x55f83af141f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8954c3e740 4 client.17937 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.149+0000 7f8954c3e740 1 -- 172.21.15.89:0/2803187322 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f83ae6a860 con 0x55f83af11d50 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.150+0000 7f8943fff640 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f83af11d50 0x55f83af141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.150+0000 7f8943fff640 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f83af11d50 0x55f83af141f0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2a6e4f12b68d8d1 server_cookie=d5aa93828ccdd6e1 in_seq=0 out_seq=0 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.150+0000 7f892b7fe640 10 client.17937 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f892b7fe640 1 -- 172.21.15.89:0/2803187322 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f83ae6a860 con 0x55f83af11d50 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f892b7fe640 10 client.17937 handle_command_reply: tid=0 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f8954c3e740 1 client.17937 shutdown 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f8954c3e740 1 -- 172.21.15.89:0/2803187322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef75d0 msgr2=0x55f83af01de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f8954c3e740 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef75d0 0x55f83af01de0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f893c012780 tx=0x7f893c00af90 comp rx=0 tx=0).stop 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f8954c3e740 1 -- 172.21.15.89:0/2803187322 shutdown_connections 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f8954c3e740 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f83af11d50 0x55f83af141f0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f8954c3e740 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83af02370 0x55f83aeff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f8954c3e740 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef8b40 0x55f83aeff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f892b7fe640 0 client.17937 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f8954c3e740 1 --2- 172.21.15.89:0/2803187322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef75d0 0x55f83af01de0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.151+0000 7f8954c3e740 1 -- 172.21.15.89:0/2803187322 >> 172.21.15.89:0/2803187322 conn(0x55f83aec0310 msgr2=0x55f83aef9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.152+0000 7f8954c3e740 1 -- 172.21.15.89:0/2803187322 shutdown_connections 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.152+0000 7f8954c3e740 1 -- 172.21.15.89:0/2803187322 wait complete. 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.152+0000 7f8954c3e740 20 client.17937 trim_cache size 0 max 16384 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.164+0000 7f8954c3e740 1 Processor -- start 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.164+0000 7f8954c3e740 1 -- start start 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef75d0 0x55f83af0df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef8b40 0x55f83af19ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83af02370 0x55f83af1a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8954c3e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f894c1aeed0 con 0x55f83aef8b40 2024-09-15T23:31:28.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8954c3e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f894c1b0a10 con 0x55f83af02370 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8954c3e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f894c1af5a0 con 0x55f83aef75d0 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8943fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef8b40 0x55f83af19ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8943fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef8b40 0x55f83af19ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59502/0 (socket says 172.21.15.89:59502) 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8943fff640 1 -- 172.21.15.89:0/1635064389 learned_addr learned my addr 172.21.15.89:0/1635064389 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8950e02640 1 --2- 172.21.15.89:0/1635064389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83af02370 0x55f83af1a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f894bfff640 1 --2- 172.21.15.89:0/1635064389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef75d0 0x55f83af0df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8943fff640 1 -- 172.21.15.89:0/1635064389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef75d0 msgr2=0x55f83af0df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8943fff640 1 --2- 172.21.15.89:0/1635064389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef75d0 0x55f83af0df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8943fff640 1 -- 172.21.15.89:0/1635064389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83af02370 msgr2=0x55f83af1a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8943fff640 1 --2- 172.21.15.89:0/1635064389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83af02370 0x55f83af1a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8943fff640 1 -- 172.21.15.89:0/1635064389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f83af0e490 con 0x55f83aef8b40 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.165+0000 7f8943fff640 1 --2- 172.21.15.89:0/1635064389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef8b40 0x55f83af19ac0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f8934004620 tx=0x7f893402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d641ae478d0b1c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.166+0000 7f8941ffb640 1 -- 172.21.15.89:0/1635064389 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934049040 con 0x55f83aef8b40 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.166+0000 7f8941ffb640 1 -- 172.21.15.89:0/1635064389 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89340bcf20 con 0x55f83aef8b40 2024-09-15T23:31:28.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.166+0000 7f8941ffb640 1 -- 172.21.15.89:0/1635064389 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89340edd80 con 0x55f83aef8b40 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 -- 172.21.15.89:0/1635064389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef8b40 msgr2=0x55f83af19ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 --2- 172.21.15.89:0/1635064389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef8b40 0x55f83af19ac0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f8934004620 tx=0x7f893402a410 comp rx=0 tx=0).stop 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 -- 172.21.15.89:0/1635064389 shutdown_connections 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 --2- 172.21.15.89:0/1635064389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83af02370 0x55f83af1a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 --2- 172.21.15.89:0/1635064389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83aef8b40 0x55f83af19ac0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 --2- 172.21.15.89:0/1635064389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef75d0 0x55f83af0df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 -- 172.21.15.89:0/1635064389 >> 172.21.15.89:0/1635064389 conn(0x55f83aec0310 msgr2=0x55f83aef9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 -- 172.21.15.89:0/1635064389 shutdown_connections 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 -- 172.21.15.89:0/1635064389 wait complete. 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 -- start start 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.167+0000 7f8954c3e740 1 Processor -- start 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef75d0 0x55f83af0a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef8b40 0x55f83af05b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8954c3e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af06080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8954c3e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f894c1b0340 con 0x55f83af02370 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8954c3e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f894c1afc70 con 0x55f83aef8b40 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8954c3e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f894c1aceb0 con 0x55f83aef75d0 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8954c3e740 10 client.0 _ll_register_callbacks cb 0x7f893c00a590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8950e02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af06080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8950e02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af06080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59504/0 (socket says 172.21.15.89:59504) 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8943fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef8b40 0x55f83af05b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8950e02640 1 -- 172.21.15.89:0/1989296166 learned_addr learned my addr 172.21.15.89:0/1989296166 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f894bfff640 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef75d0 0x55f83af0a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8950e02640 1 -- 172.21.15.89:0/1989296166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef75d0 msgr2=0x55f83af0a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8950e02640 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef75d0 0x55f83af0a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8950e02640 1 -- 172.21.15.89:0/1989296166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef8b40 msgr2=0x55f83af05b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8950e02640 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef8b40 0x55f83af05b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8950e02640 1 -- 172.21.15.89:0/1989296166 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f83af06860 con 0x55f83af02370 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8950e02640 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af06080 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f894400fab0 tx=0x7f8944004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905649a9e0a5a8bd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f892b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f892b7fe640 1 -- 172.21.15.89:0/1989296166 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894400d400 con 0x55f83af02370 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f892b7fe640 1 -- 172.21.15.89:0/1989296166 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f894400d5a0 con 0x55f83af02370 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.168+0000 7f8954c3e740 1 -- 172.21.15.89:0/1989296166 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89340bcf20 con 0x55f83af02370 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f892b7fe640 1 -- 172.21.15.89:0/1989296166 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8944033050 con 0x55f83af02370 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f8954c3e740 10 client.17943 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f8954c3e740 1 -- 172.21.15.89:0/1989296166 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f83af7caa0 con 0x55f83af02370 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f892b7fe640 1 -- 172.21.15.89:0/1989296166 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89340bcf20 con 0x55f83af02370 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f892b7fe640 1 -- 172.21.15.89:0/1989296166 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f894401b040 con 0x55f83af02370 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f8954c3e740 10 client.17943 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f8954c3e740 10 client.17943 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:28.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f8954c3e740 20 client.17943 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f8954c3e740 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f83af7d270 0x55f83af7f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f8954c3e740 4 client.17943 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f8954c3e740 1 -- 172.21.15.89:0/1989296166 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f83ae64d70 con 0x55f83af7d270 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.169+0000 7f894bfff640 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f83af7d270 0x55f83af7f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f894bfff640 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f83af7d270 0x55f83af7f690 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d79cb2230276d4c5 server_cookie=e00520e1e1529d9b in_seq=0 out_seq=0 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f892b7fe640 10 client.17943 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f892b7fe640 1 -- 172.21.15.89:0/1989296166 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f83ae64d70 con 0x55f83af7d270 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f892b7fe640 10 client.17943 handle_command_reply: tid=0 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 client.17943 shutdown 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 -- 172.21.15.89:0/1989296166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 msgr2=0x55f83af06080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af06080 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f894400fab0 tx=0x7f8944004a40 comp rx=0 tx=0).stop 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 -- 172.21.15.89:0/1989296166 shutdown_connections 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f83af7d270 0x55f83af7f690 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f83af02370 0x55f83af06080 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f83aef8b40 0x55f83af05b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f892b7fe640 0 client.17943 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:28.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 --2- 172.21.15.89:0/1989296166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f83aef75d0 0x55f83af0a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 -- 172.21.15.89:0/1989296166 >> 172.21.15.89:0/1989296166 conn(0x55f83aec0310 msgr2=0x55f83aef24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 -- 172.21.15.89:0/1989296166 shutdown_connections 2024-09-15T23:31:28.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 1 -- 172.21.15.89:0/1989296166 wait complete. 2024-09-15T23:31:28.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.170+0000 7f8954c3e740 20 client.17943 trim_cache size 0 max 16384 2024-09-15T23:31:28.186 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:31:28.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 -- 172.21.15.89:0/2968272907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89380a3240 msgr2=0x7f89380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89380a3240 0x7f89380a5700 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8944005fd0 tx=0x7f8944005c80 comp rx=0 tx=0).stop 2024-09-15T23:31:28.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 -- 172.21.15.89:0/2968272907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f894c102790 msgr2=0x7f894c1989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f894c102790 0x7f894c1989b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f8934005ab0 tx=0x7f89340042f0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 -- 172.21.15.89:0/2968272907 shutdown_connections 2024-09-15T23:31:28.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f89380a3240 0x7f89380a5700 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f894c106f10 0x7f894c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f894c102790 0x7f894c1989b0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 --2- 172.21.15.89:0/2968272907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f894c101de0 0x7f894c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 -- 172.21.15.89:0/2968272907 >> 172.21.15.89:0/2968272907 conn(0x7f894c0fd8c0 msgr2=0x7f894c0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 -- 172.21.15.89:0/2968272907 shutdown_connections 2024-09-15T23:31:28.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:28.171+0000 7f8952865640 1 -- 172.21.15.89:0/2968272907 wait complete. 2024-09-15T23:31:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.437+0000 7fb485fd6640 1 -- 172.21.15.89:0/4067656851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb480100410 msgr2=0x7fb480100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.437+0000 7fb485fd6640 1 --2- 172.21.15.89:0/4067656851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb480100410 0x7fb480100810 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fb470009900 tx=0x7fb47002a410 comp rx=0 tx=0).stop 2024-09-15T23:31:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.437+0000 7fb485fd6640 1 -- 172.21.15.89:0/4067656851 shutdown_connections 2024-09-15T23:31:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.437+0000 7fb485fd6640 1 --2- 172.21.15.89:0/4067656851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4801016f0 0x7fb480101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.437+0000 7fb485fd6640 1 --2- 172.21.15.89:0/4067656851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb480100d50 0x7fb4801011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.437+0000 7fb485fd6640 1 --2- 172.21.15.89:0/4067656851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb480100410 0x7fb480100810 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.437+0000 7fb485fd6640 1 -- 172.21.15.89:0/4067656851 >> 172.21.15.89:0/4067656851 conn(0x7fb4800fc1e0 msgr2=0x7fb4800fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.437+0000 7fb485fd6640 1 -- 172.21.15.89:0/4067656851 shutdown_connections 2024-09-15T23:31:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.437+0000 7fb485fd6640 1 -- 172.21.15.89:0/4067656851 wait complete. 2024-09-15T23:31:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.437+0000 7fb485fd6640 1 Processor -- start 2024-09-15T23:31:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb485fd6640 1 -- start start 2024-09-15T23:31:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb485fd6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb480100410 0x7fb480199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb485fd6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb480100d50 0x7fb48019a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb485fd6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4801016f0 0x7fb4801a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb485fd6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb48010f950 con 0x7fb480100410 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb485fd6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb48010f7d0 con 0x7fb4801016f0 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb485fd6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb48010fad0 con 0x7fb480100d50 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb47f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb480100410 0x7fb480199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb47f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb480100410 0x7fb480199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59518/0 (socket says 172.21.15.89:59518) 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb47f7fe640 1 -- 172.21.15.89:0/2602478104 learned_addr learned my addr 172.21.15.89:0/2602478104 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb47ffff640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4801016f0 0x7fb4801a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb47effd640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb480100d50 0x7fb48019a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb47f7fe640 1 -- 172.21.15.89:0/2602478104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb480100d50 msgr2=0x7fb48019a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb480100d50 0x7fb48019a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb47f7fe640 1 -- 172.21.15.89:0/2602478104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4801016f0 msgr2=0x7fb4801a1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4801016f0 0x7fb4801a1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.438+0000 7fb47f7fe640 1 -- 172.21.15.89:0/2602478104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4801a1ba0 con 0x7fb480100410 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.439+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb480100410 0x7fb480199f80 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb470009230 tx=0x7fb470031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.439+0000 7fb47cff9640 1 -- 172.21.15.89:0/2602478104 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47003f030 con 0x7fb480100410 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.439+0000 7fb47cff9640 1 -- 172.21.15.89:0/2602478104 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb470003ad0 con 0x7fb480100410 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.439+0000 7fb47cff9640 1 -- 172.21.15.89:0/2602478104 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470035070 con 0x7fb480100410 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.439+0000 7fb485fd6640 1 -- 172.21.15.89:0/2602478104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4801a1e30 con 0x7fb480100410 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.439+0000 7fb485fd6640 1 -- 172.21.15.89:0/2602478104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4801a2290 con 0x7fb480100410 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.441+0000 7fb488e51740 1 Processor -- start 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.441+0000 7fb488e51740 1 -- start start 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.441+0000 7fb47cff9640 1 -- 172.21.15.89:0/2602478104 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb470042030 con 0x7fb480100410 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.441+0000 7fb488e51740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497819a0 0x560149781da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.441+0000 7fb488e51740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 0x560149786da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.441+0000 7fb488e51740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149789830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.441+0000 7fb488e51740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5601496e52f0 con 0x560149787370 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.441+0000 7fb488e51740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56014968d270 con 0x5601497822e0 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.441+0000 7fb488e51740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5601496e12b0 con 0x5601497819a0 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149789830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149789830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59522/0 (socket says 172.21.15.89:59522) 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47effd640 1 -- 172.21.15.89:0/2044319928 learned_addr learned my addr 172.21.15.89:0/2044319928 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47ffff640 1 --2- 172.21.15.89:0/2044319928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497819a0 0x560149781da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/2044319928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 0x560149786da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47effd640 1 -- 172.21.15.89:0/2044319928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497819a0 msgr2=0x560149781da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47effd640 1 --2- 172.21.15.89:0/2044319928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497819a0 0x560149781da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47effd640 1 -- 172.21.15.89:0/2044319928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 msgr2=0x560149786da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47effd640 1 --2- 172.21.15.89:0/2044319928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 0x560149786da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47effd640 1 -- 172.21.15.89:0/2044319928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56014968c340 con 0x560149787370 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/2044319928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 0x560149786da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47cff9640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4680a32f0 0x7fb4680a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47effd640 1 --2- 172.21.15.89:0/2044319928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149789830 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fb474005bc0 tx=0x7fb47400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a10eb42318159b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47ffff640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4680a32f0 0x7fb4680a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb47cff9640 1 -- 172.21.15.89:0/2602478104 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb4700f0320 con 0x7fb480100410 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb465ffb640 1 -- 172.21.15.89:0/2044319928 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb474016070 con 0x560149787370 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb465ffb640 1 -- 172.21.15.89:0/2044319928 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb474005500 con 0x560149787370 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.442+0000 7fb465ffb640 1 -- 172.21.15.89:0/2044319928 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4740116d0 con 0x560149787370 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb488e51740 1 -- 172.21.15.89:0/2044319928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 msgr2=0x560149789830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb47ffff640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4680a32f0 0x7fb4680a57b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb478005fd0 tx=0x7fb478009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb488e51740 1 --2- 172.21.15.89:0/2044319928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149789830 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fb474005bc0 tx=0x7fb47400e6c0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb488e51740 1 -- 172.21.15.89:0/2044319928 shutdown_connections 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb488e51740 1 --2- 172.21.15.89:0/2044319928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149789830 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb488e51740 1 --2- 172.21.15.89:0/2044319928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 0x560149786da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb488e51740 1 --2- 172.21.15.89:0/2044319928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497819a0 0x560149781da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb488e51740 1 -- 172.21.15.89:0/2044319928 >> 172.21.15.89:0/2044319928 conn(0x56014977d010 msgr2=0x56014977d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb488e51740 1 -- 172.21.15.89:0/2044319928 shutdown_connections 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb488e51740 1 -- 172.21.15.89:0/2044319928 wait complete. 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.443+0000 7fb488e51740 1 -- start start 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb488e51740 1 Processor -- start 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb488e51740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb488e51740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497822e0 0x56014978c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb488e51740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560149787370 0x5601497930b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb488e51740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56014978a340 con 0x5601497819a0 2024-09-15T23:31:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb488e51740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56014966e290 con 0x560149787370 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb488e51740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56014978a4c0 con 0x5601497822e0 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb488e51740 10 client.0 _ll_register_callbacks cb 0x56014969f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59526/0 (socket says 172.21.15.89:59526) 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47f7fe640 1 -- 172.21.15.89:0/1238947479 learned_addr learned my addr 172.21.15.89:0/1238947479 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47effd640 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497822e0 0x56014978c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47ffff640 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560149787370 0x5601497930b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47f7fe640 1 -- 172.21.15.89:0/1238947479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497822e0 msgr2=0x56014978c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497822e0 0x56014978c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47f7fe640 1 -- 172.21.15.89:0/1238947479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560149787370 msgr2=0x5601497930b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560149787370 0x5601497930b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47f7fe640 1 -- 172.21.15.89:0/1238947479 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601493f8210 con 0x5601497819a0 2024-09-15T23:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47ffff640 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560149787370 0x5601497930b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978bb30 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb4700bc250 tx=0x7fb4700b88b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f4440fd5714ee1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.444+0000 7fb45effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb45effd640 1 -- 172.21.15.89:0/1238947479 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470049070 con 0x5601497819a0 2024-09-15T23:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb45effd640 1 -- 172.21.15.89:0/1238947479 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb4700f3cd0 con 0x5601497819a0 2024-09-15T23:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb488e51740 1 -- 172.21.15.89:0/1238947479 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb474005500 con 0x5601497819a0 2024-09-15T23:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb45effd640 1 -- 172.21.15.89:0/1238947479 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb4700f0170 con 0x5601497819a0 2024-09-15T23:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb488e51740 10 client.17955 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb488e51740 1 -- 172.21.15.89:0/1238947479 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5601494c4620 con 0x5601497819a0 2024-09-15T23:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb45effd640 1 -- 172.21.15.89:0/1238947479 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb474005500 con 0x5601497819a0 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb45effd640 1 -- 172.21.15.89:0/1238947479 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb4700b8bc0 con 0x5601497819a0 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb488e51740 10 client.17955 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb488e51740 10 client.17955 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb488e51740 20 client.17955 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb488e51740 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5601497977e0 0x560149799c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb488e51740 4 client.17955 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb488e51740 1 -- 172.21.15.89:0/1238947479 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601496f2a20 con 0x5601497977e0 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.445+0000 7fb47effd640 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5601497977e0 0x560149799c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.446+0000 7fb45effd640 10 client.17955 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.446+0000 7fb47effd640 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5601497977e0 0x560149799c50 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8812d346f7cbfb7 server_cookie=a4da6650697f3969 in_seq=0 out_seq=0 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.446+0000 7fb45effd640 1 -- 172.21.15.89:0/1238947479 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5601496f2a20 con 0x5601497977e0 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.446+0000 7fb45effd640 10 client.17955 handle_command_reply: tid=0 2024-09-15T23:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 client.17955 shutdown 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 -- 172.21.15.89:0/1238947479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 msgr2=0x56014978bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978bb30 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb4700bc250 tx=0x7fb4700b88b0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 -- 172.21.15.89:0/1238947479 shutdown_connections 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5601497977e0 0x560149799c50 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560149787370 0x5601497930b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497822e0 0x56014978c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb45effd640 0 client.17955 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 --2- 172.21.15.89:0/1238947479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978bb30 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 -- 172.21.15.89:0/1238947479 >> 172.21.15.89:0/1238947479 conn(0x56014977d010 msgr2=0x56014977e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 -- 172.21.15.89:0/1238947479 shutdown_connections 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 1 -- 172.21.15.89:0/1238947479 wait complete. 2024-09-15T23:31:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.447+0000 7fb488e51740 20 client.17955 trim_cache size 0 max 16384 2024-09-15T23:31:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb488e51740 1 Processor -- start 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb488e51740 1 -- start start 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb488e51740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb488e51740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497822e0 0x560149793040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb488e51740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560149787370 0x56014977c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb488e51740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4801acb10 con 0x5601497819a0 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb488e51740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4801ae650 con 0x560149787370 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb488e51740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4801ad1e0 con 0x5601497822e0 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb47f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb47f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59538/0 (socket says 172.21.15.89:59538) 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb47f7fe640 1 -- 172.21.15.89:0/3213636151 learned_addr learned my addr 172.21.15.89:0/3213636151 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb47effd640 1 --2- 172.21.15.89:0/3213636151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497822e0 0x560149793040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb47f7fe640 1 -- 172.21.15.89:0/3213636151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497822e0 msgr2=0x560149793040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/3213636151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497822e0 0x560149793040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb47f7fe640 1 -- 172.21.15.89:0/3213636151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560149787370 msgr2=0x56014977c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/3213636151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560149787370 0x56014977c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.461+0000 7fb47f7fe640 1 -- 172.21.15.89:0/3213636151 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56014978b670 con 0x5601497819a0 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/3213636151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978c100 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fb4700314e0 tx=0x7fb4700baa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701a16c84134eab1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb4657fa640 1 -- 172.21.15.89:0/3213636151 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4700bc090 con 0x5601497819a0 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb4657fa640 1 -- 172.21.15.89:0/3213636151 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4700f0170 con 0x5601497819a0 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb4657fa640 1 -- 172.21.15.89:0/3213636151 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4700c4d30 con 0x5601497819a0 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb488e51740 1 -- 172.21.15.89:0/3213636151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 msgr2=0x56014978c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb488e51740 1 --2- 172.21.15.89:0/3213636151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978c100 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fb4700314e0 tx=0x7fb4700baa60 comp rx=0 tx=0).stop 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb488e51740 1 -- 172.21.15.89:0/3213636151 shutdown_connections 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb488e51740 1 --2- 172.21.15.89:0/3213636151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560149787370 0x56014977c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb488e51740 1 --2- 172.21.15.89:0/3213636151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497822e0 0x560149793040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb488e51740 1 --2- 172.21.15.89:0/3213636151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601497819a0 0x56014978c100 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb488e51740 1 -- 172.21.15.89:0/3213636151 >> 172.21.15.89:0/3213636151 conn(0x56014977d010 msgr2=0x56014977b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb488e51740 1 -- 172.21.15.89:0/3213636151 shutdown_connections 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.462+0000 7fb488e51740 1 -- 172.21.15.89:0/3213636151 wait complete. 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb488e51740 1 -- start start 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb488e51740 1 Processor -- start 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb488e51740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497819a0 0x5601497908b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb488e51740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 0x560149790df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb488e51740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149791d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb488e51740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4801adf80 con 0x560149787370 2024-09-15T23:31:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb488e51740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4801ad8b0 con 0x5601497822e0 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb488e51740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4801aaaf0 con 0x5601497819a0 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb488e51740 10 client.0 _ll_register_callbacks cb 0x7fb470005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb47ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149791d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb47ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149791d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59554/0 (socket says 172.21.15.89:59554) 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb47ffff640 1 -- 172.21.15.89:0/3683980167 learned_addr learned my addr 172.21.15.89:0/3683980167 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497819a0 0x5601497908b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb47effd640 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 0x560149790df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb47ffff640 1 -- 172.21.15.89:0/3683980167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497819a0 msgr2=0x5601497908b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb47ffff640 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497819a0 0x5601497908b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb47ffff640 1 -- 172.21.15.89:0/3683980167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 msgr2=0x560149790df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.463+0000 7fb47ffff640 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 0x560149790df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb47ffff640 1 -- 172.21.15.89:0/3683980167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560149782f10 con 0x560149787370 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb47ffff640 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149791d40 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb47801ea00 tx=0x7fb47801ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54411c8bae431 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb45effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb45effd640 1 -- 172.21.15.89:0/3683980167 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47802c060 con 0x560149787370 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb45effd640 1 -- 172.21.15.89:0/3683980167 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb478017030 con 0x560149787370 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb488e51740 1 -- 172.21.15.89:0/3683980167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4700f0170 con 0x560149787370 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb45effd640 1 -- 172.21.15.89:0/3683980167 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb478033050 con 0x560149787370 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb488e51740 10 client.17961 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb488e51740 1 -- 172.21.15.89:0/3683980167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560149802530 con 0x560149787370 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb45effd640 1 -- 172.21.15.89:0/3683980167 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4700f0170 con 0x560149787370 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb45effd640 1 -- 172.21.15.89:0/3683980167 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb47801c820 con 0x560149787370 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb488e51740 10 client.17961 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb488e51740 10 client.17961 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb488e51740 20 client.17961 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb488e51740 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560149802e70 0x560149805290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb488e51740 4 client.17961 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:31:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.464+0000 7fb488e51740 1 -- 172.21.15.89:0/3683980167 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5601496ecd70 con 0x560149802e70 2024-09-15T23:31:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.465+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560149802e70 0x560149805290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.465+0000 7fb45effd640 10 client.17961 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:28.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.465+0000 7fb47f7fe640 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560149802e70 0x560149805290 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=660df276a32001d server_cookie=1c7d88ce794b9611 in_seq=0 out_seq=0 2024-09-15T23:31:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb45effd640 1 -- 172.21.15.89:0/3683980167 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5601496ecd70 con 0x560149802e70 2024-09-15T23:31:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb45effd640 10 client.17961 handle_command_reply: tid=0 2024-09-15T23:31:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 client.17961 shutdown 2024-09-15T23:31:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 -- 172.21.15.89:0/3683980167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 msgr2=0x560149791d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149791d40 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb47801ea00 tx=0x7fb47801ee00 comp rx=0 tx=0).stop 2024-09-15T23:31:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 -- 172.21.15.89:0/3683980167 shutdown_connections 2024-09-15T23:31:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560149802e70 0x560149805290 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560149787370 0x560149791d40 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb45effd640 0 client.17961 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601497822e0 0x560149790df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 --2- 172.21.15.89:0/3683980167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601497819a0 0x5601497908b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 -- 172.21.15.89:0/3683980167 >> 172.21.15.89:0/3683980167 conn(0x56014977d010 msgr2=0x56014977c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 -- 172.21.15.89:0/3683980167 shutdown_connections 2024-09-15T23:31:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 1 -- 172.21.15.89:0/3683980167 wait complete. 2024-09-15T23:31:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.481+0000 7fb488e51740 20 client.17961 trim_cache size 0 max 16384 2024-09-15T23:31:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:31:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:31:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:31:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:31:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 -- 172.21.15.89:0/2602478104 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4680a32f0 msgr2=0x7fb4680a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4680a32f0 0x7fb4680a57b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb478005fd0 tx=0x7fb478009500 comp rx=0 tx=0).stop 2024-09-15T23:31:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 -- 172.21.15.89:0/2602478104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb480100410 msgr2=0x7fb480199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb480100410 0x7fb480199f80 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb470009230 tx=0x7fb470031b60 comp rx=0 tx=0).stop 2024-09-15T23:31:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 -- 172.21.15.89:0/2602478104 shutdown_connections 2024-09-15T23:31:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4680a32f0 0x7fb4680a57b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4801016f0 0x7fb4801a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb480100d50 0x7fb48019a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 --2- 172.21.15.89:0/2602478104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb480100410 0x7fb480199f80 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 -- 172.21.15.89:0/2602478104 >> 172.21.15.89:0/2602478104 conn(0x7fb4800fc1e0 msgr2=0x7fb4800fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 -- 172.21.15.89:0/2602478104 shutdown_connections 2024-09-15T23:31:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:28.482+0000 7fb485fd6640 1 -- 172.21.15.89:0/2602478104 wait complete. 2024-09-15T23:31:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:29 smithi120 ceph-mon[35250]: pgmap v574: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:31:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:29 smithi174 ceph-mon[35586]: pgmap v574: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:31:29.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:29 smithi089 ceph-mon[30933]: pgmap v574: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:31:29.908 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:31:29.909 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:31:30.167 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:30.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.634+0000 7f9499207640 1 -- 172.21.15.89:0/3512487948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f949410c170 msgr2=0x7f949410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.634+0000 7f9499207640 1 --2- 172.21.15.89:0/3512487948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f949410c170 0x7f949410e610 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f948c009ca0 tx=0x7f948c02a950 comp rx=0 tx=0).stop 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.634+0000 7f9499207640 1 -- 172.21.15.89:0/3512487948 shutdown_connections 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.634+0000 7f9499207640 1 --2- 172.21.15.89:0/3512487948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f949410c170 0x7f949410e610 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.634+0000 7f9499207640 1 --2- 172.21.15.89:0/3512487948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9494103660 0x7f949410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.634+0000 7f9499207640 1 --2- 172.21.15.89:0/3512487948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9494102d40 0x7f9494103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.634+0000 7f9499207640 1 -- 172.21.15.89:0/3512487948 >> 172.21.15.89:0/3512487948 conn(0x7f94940fc7a0 msgr2=0x7f94940febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.634+0000 7f9499207640 1 -- 172.21.15.89:0/3512487948 shutdown_connections 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.635+0000 7f9499207640 1 -- 172.21.15.89:0/3512487948 wait complete. 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.636+0000 7f9499207640 1 Processor -- start 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.636+0000 7f9499207640 1 -- start start 2024-09-15T23:31:30.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.636+0000 7f9499207640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9494102d40 0x7f9494198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.636+0000 7f9499207640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9494103660 0x7f94941988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.636+0000 7f9499207640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949410c170 0x7f949419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.636+0000 7f9499207640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9494111f50 con 0x7f9494102d40 2024-09-15T23:31:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.636+0000 7f9499207640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9494111dd0 con 0x7f949410c170 2024-09-15T23:31:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.636+0000 7f9499207640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94941120d0 con 0x7f9494103660 2024-09-15T23:31:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.636+0000 7f9492d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9494102d40 0x7f9494198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9492d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9494102d40 0x7f9494198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59576/0 (socket says 172.21.15.89:59576) 2024-09-15T23:31:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9492d9d640 1 -- 172.21.15.89:0/1357832078 learned_addr learned my addr 172.21.15.89:0/1357832078 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:30.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f949359e640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949410c170 0x7f949419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f948a59c640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9494103660 0x7f94941988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9492d9d640 1 -- 172.21.15.89:0/1357832078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9494103660 msgr2=0x7f94941988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9492d9d640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9494103660 0x7f94941988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9492d9d640 1 -- 172.21.15.89:0/1357832078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949410c170 msgr2=0x7f949419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9492d9d640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949410c170 0x7f949419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9492d9d640 1 -- 172.21.15.89:0/1357832078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949419d320 con 0x7f9494102d40 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f949359e640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949410c170 0x7f949419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f948a59c640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9494103660 0x7f94941988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9492d9d640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9494102d40 0x7f9494198370 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f948400c920 tx=0x7f948400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9490d99640 1 -- 172.21.15.89:0/1357832078 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484007c70 con 0x7f9494102d40 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9490d99640 1 -- 172.21.15.89:0/1357832078 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9484007e10 con 0x7f9494102d40 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9499207640 1 -- 172.21.15.89:0/1357832078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949419d610 con 0x7f9494102d40 2024-09-15T23:31:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9490d99640 1 -- 172.21.15.89:0/1357832078 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484018890 con 0x7f9494102d40 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.637+0000 7f9499207640 1 -- 172.21.15.89:0/1357832078 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94941a4ef0 con 0x7f9494102d40 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.640+0000 7f9490d99640 1 -- 172.21.15.89:0/1357832078 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9484018a70 con 0x7f9494102d40 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.640+0000 7f949b5e0740 1 Processor -- start 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.640+0000 7f949b5e0740 1 -- start start 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb4af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 0x55fa1fb55800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb55d40 0x55fa1fb58130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f949b5e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fa1fab0280 con 0x55fa1fb55d40 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f949b5e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fa1fa57b10 con 0x55fa1fb4ab60 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f949b5e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fa1faac240 con 0x55fa1fb4b480 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f9490d99640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94800a3240 0x7f94800a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f948a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb55d40 0x55fa1fb58130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f948a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb55d40 0x55fa1fb58130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59582/0 (socket says 172.21.15.89:59582) 2024-09-15T23:31:30.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f948a59c640 1 -- 172.21.15.89:0/1100607150 learned_addr learned my addr 172.21.15.89:0/1100607150 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:30.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f9490d99640 1 -- 172.21.15.89:0/1357832078 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f94840cff60 con 0x7f9494102d40 2024-09-15T23:31:30.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f949359e640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94800a3240 0x7f94800a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f9492d9d640 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 0x55fa1fb55800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f949359e640 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb4af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f948a59c640 1 -- 172.21.15.89:0/1100607150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 msgr2=0x55fa1fb55800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:30.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f948a59c640 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 0x55fa1fb55800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f948a59c640 1 -- 172.21.15.89:0/1100607150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 msgr2=0x55fa1fb4af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:30.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f948a59c640 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb4af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f948a59c640 1 -- 172.21.15.89:0/1100607150 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa1fa641f0 con 0x55fa1fb55d40 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f9492d9d640 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 0x55fa1fb55800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f949359e640 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb4af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f948a59c640 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb55d40 0x55fa1fb58130 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f947c00bc60 tx=0x7f947c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8dabe5e9eab20a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f949359e640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94800a3240 0x7f94800a5700 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f948c005e30 tx=0x7f948c03d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.641+0000 7f948959a640 1 -- 172.21.15.89:0/1100607150 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c005020 con 0x55fa1fb55d40 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f948959a640 1 -- 172.21.15.89:0/1100607150 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947c0047a0 con 0x55fa1fb55d40 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f948959a640 1 -- 172.21.15.89:0/1100607150 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c0116a0 con 0x55fa1fb55d40 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f949b5e0740 1 -- 172.21.15.89:0/1100607150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb55d40 msgr2=0x55fa1fb58130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f949b5e0740 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb55d40 0x55fa1fb58130 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f947c00bc60 tx=0x7f947c00f730 comp rx=0 tx=0).stop 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f949b5e0740 1 -- 172.21.15.89:0/1100607150 shutdown_connections 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f949b5e0740 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb55d40 0x55fa1fb58130 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f949b5e0740 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 0x55fa1fb55800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f949b5e0740 1 --2- 172.21.15.89:0/1100607150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb4af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f949b5e0740 1 -- 172.21.15.89:0/1100607150 >> 172.21.15.89:0/1100607150 conn(0x55fa1fb470f0 msgr2=0x55fa1fb474d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f949b5e0740 1 -- 172.21.15.89:0/1100607150 shutdown_connections 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.642+0000 7f949b5e0740 1 -- 172.21.15.89:0/1100607150 wait complete. 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f949b5e0740 1 -- start start 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f949b5e0740 1 Processor -- start 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb52b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4b480 0x55fa1fb530b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb55d40 0x55fa1fb535f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f949b5e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fa1fb58910 con 0x55fa1fb4ab60 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f949b5e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fa1fa6af10 con 0x55fa1fb4b480 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f949b5e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fa1fb58a90 con 0x55fa1fb55d40 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f949b5e0740 10 client.0 _ll_register_callbacks cb 0x55fa1fa33a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f9492d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb52b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f9492d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb52b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59598/0 (socket says 172.21.15.89:59598) 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f9492d9d640 1 -- 172.21.15.89:0/3500084910 learned_addr learned my addr 172.21.15.89:0/3500084910 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f948a59c640 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4b480 0x55fa1fb530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f9492d9d640 1 -- 172.21.15.89:0/3500084910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb55d40 msgr2=0x55fa1fb535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f9492d9d640 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb55d40 0x55fa1fb535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f9492d9d640 1 -- 172.21.15.89:0/3500084910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4b480 msgr2=0x55fa1fb530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f9492d9d640 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4b480 0x55fa1fb530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f9492d9d640 1 -- 172.21.15.89:0/3500084910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa1f8216a0 con 0x55fa1fb4ab60 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.643+0000 7f948a59c640 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4b480 0x55fa1fb530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f9492d9d640 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb52b70 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f948400c8f0 tx=0x7f948409bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af49f51703ce5ea4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f947a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f947a7fc640 1 -- 172.21.15.89:0/3500084910 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94840b4020 con 0x55fa1fb4ab60 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f947a7fc640 1 -- 172.21.15.89:0/3500084910 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f948400fb30 con 0x55fa1fb4ab60 2024-09-15T23:31:30.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f949b5e0740 1 -- 172.21.15.89:0/3500084910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f947c0047a0 con 0x55fa1fb4ab60 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f947a7fc640 1 -- 172.21.15.89:0/3500084910 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9484066050 con 0x55fa1fb4ab60 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f949b5e0740 10 client.17973 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f949b5e0740 1 -- 172.21.15.89:0/3500084910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fa1fb64430 con 0x55fa1fb4ab60 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f947a7fc640 1 -- 172.21.15.89:0/3500084910 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947c0047a0 con 0x55fa1fb4ab60 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f947a7fc640 1 -- 172.21.15.89:0/3500084910 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f94840a00d0 con 0x55fa1fb4ab60 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f949b5e0740 10 client.17973 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f949b5e0740 10 client.17973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f949b5e0740 20 client.17973 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f949b5e0740 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fa1fb65990 0x55fa1fb67e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f949b5e0740 4 client.17973 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.644+0000 7f949b5e0740 1 -- 172.21.15.89:0/3500084910 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa1fabd7f0 con 0x55fa1fb65990 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.645+0000 7f948a59c640 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fa1fb65990 0x55fa1fb67e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.645+0000 7f948a59c640 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fa1fb65990 0x55fa1fb67e20 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d02a178336363ae1 server_cookie=7fd41401798c584c in_seq=0 out_seq=0 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.645+0000 7f947a7fc640 10 client.17973 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f947a7fc640 1 -- 172.21.15.89:0/3500084910 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fa1fabd7f0 con 0x55fa1fb65990 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f947a7fc640 10 client.17973 handle_command_reply: tid=0 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 client.17973 shutdown 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 -- 172.21.15.89:0/3500084910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 msgr2=0x55fa1fb52b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb52b70 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f948400c8f0 tx=0x7f948409bb50 comp rx=0 tx=0).stop 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 -- 172.21.15.89:0/3500084910 shutdown_connections 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fa1fb65990 0x55fa1fb67e20 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb55d40 0x55fa1fb535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4b480 0x55fa1fb530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f947a7fc640 0 client.17973 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 --2- 172.21.15.89:0/3500084910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb52b70 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 -- 172.21.15.89:0/3500084910 >> 172.21.15.89:0/3500084910 conn(0x55fa1fb470f0 msgr2=0x55fa1fb46270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:30.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 -- 172.21.15.89:0/3500084910 shutdown_connections 2024-09-15T23:31:30.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 1 -- 172.21.15.89:0/3500084910 wait complete. 2024-09-15T23:31:30.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.646+0000 7f949b5e0740 20 client.17973 trim_cache size 0 max 16384 2024-09-15T23:31:30.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.663+0000 7f949b5e0740 1 Processor -- start 2024-09-15T23:31:30.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.663+0000 7f949b5e0740 1 -- start start 2024-09-15T23:31:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb470f0 0x55fa1fb4ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb5e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4b480 0x55fa1fb5e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949b5e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f94941aedc0 con 0x55fa1fb4b480 2024-09-15T23:31:30.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949b5e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94941b0900 con 0x55fa1fb4ab60 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949b5e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94941af490 con 0x55fa1fb470f0 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4b480 0x55fa1fb5e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4b480 0x55fa1fb5e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59612/0 (socket says 172.21.15.89:59612) 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949359e640 1 -- 172.21.15.89:0/3954132922 learned_addr learned my addr 172.21.15.89:0/3954132922 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f9492d9d640 1 --2- 172.21.15.89:0/3954132922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb470f0 0x55fa1fb4ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949359e640 1 -- 172.21.15.89:0/3954132922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb470f0 msgr2=0x55fa1fb4ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949359e640 1 --2- 172.21.15.89:0/3954132922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb470f0 0x55fa1fb4ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949359e640 1 -- 172.21.15.89:0/3954132922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 msgr2=0x55fa1fb5e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f948a59c640 1 --2- 172.21.15.89:0/3954132922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb5e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949359e640 1 --2- 172.21.15.89:0/3954132922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb5e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949359e640 1 -- 172.21.15.89:0/3954132922 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa1fb4b2b0 con 0x55fa1fb4b480 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f948a59c640 1 --2- 172.21.15.89:0/3954132922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb5e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f949359e640 1 --2- 172.21.15.89:0/3954132922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4b480 0x55fa1fb5e600 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f948c005fe0 tx=0x7f948c02a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37820c7317f72e7a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:30.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f9488d99640 1 -- 172.21.15.89:0/3954132922 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c034040 con 0x55fa1fb4b480 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f9488d99640 1 -- 172.21.15.89:0/3954132922 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f948c04b070 con 0x55fa1fb4b480 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.664+0000 7f9488d99640 1 -- 172.21.15.89:0/3954132922 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c0304a0 con 0x55fa1fb4b480 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.665+0000 7f949b5e0740 1 -- 172.21.15.89:0/3954132922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4b480 msgr2=0x55fa1fb5e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.665+0000 7f949b5e0740 1 --2- 172.21.15.89:0/3954132922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4b480 0x55fa1fb5e600 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f948c005fe0 tx=0x7f948c02a950 comp rx=0 tx=0).stop 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.665+0000 7f949b5e0740 1 -- 172.21.15.89:0/3954132922 shutdown_connections 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.665+0000 7f949b5e0740 1 --2- 172.21.15.89:0/3954132922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4b480 0x55fa1fb5e600 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.665+0000 7f949b5e0740 1 --2- 172.21.15.89:0/3954132922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb4ab60 0x55fa1fb5e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.665+0000 7f949b5e0740 1 --2- 172.21.15.89:0/3954132922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb470f0 0x55fa1fb4ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.665+0000 7f949b5e0740 1 -- 172.21.15.89:0/3954132922 >> 172.21.15.89:0/3954132922 conn(0x55fa1fb4e560 msgr2=0x55fa1fb4c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.665+0000 7f949b5e0740 1 -- 172.21.15.89:0/3954132922 shutdown_connections 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.665+0000 7f949b5e0740 1 -- 172.21.15.89:0/3954132922 wait complete. 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f949b5e0740 1 -- start start 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f949b5e0740 1 Processor -- start 2024-09-15T23:31:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb470f0 0x55fa1fbc8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fbc8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f949b5e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 0x55fa1fbcd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f949b5e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f94941b0230 con 0x55fa1fb4ab60 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f949b5e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94941afb60 con 0x55fa1fb470f0 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f949b5e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94941acda0 con 0x55fa1fb4b480 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f949b5e0740 10 client.0 _ll_register_callbacks cb 0x7f948400cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f948a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fbc8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f948a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fbc8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59626/0 (socket says 172.21.15.89:59626) 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f948a59c640 1 -- 172.21.15.89:0/945135547 learned_addr learned my addr 172.21.15.89:0/945135547 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f9492d9d640 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb470f0 0x55fa1fbc8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f949359e640 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 0x55fa1fbcd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f948a59c640 1 -- 172.21.15.89:0/945135547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 msgr2=0x55fa1fbcd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f948a59c640 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 0x55fa1fbcd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f948a59c640 1 -- 172.21.15.89:0/945135547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb470f0 msgr2=0x55fa1fbc8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f948a59c640 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb470f0 0x55fa1fbc8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.666+0000 7f948a59c640 1 -- 172.21.15.89:0/945135547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa1fbcdd70 con 0x55fa1fb4ab60 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f948a59c640 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fbc8da0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f947c00fc30 tx=0x7f947c00b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b08d27894666f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f947a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f947a7fc640 1 -- 172.21.15.89:0/945135547 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c0051a0 con 0x55fa1fb4ab60 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f947a7fc640 1 -- 172.21.15.89:0/945135547 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f947c005340 con 0x55fa1fb4ab60 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f949b5e0740 1 -- 172.21.15.89:0/945135547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f948c04b070 con 0x55fa1fb4ab60 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f947a7fc640 1 -- 172.21.15.89:0/945135547 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f947c005540 con 0x55fa1fb4ab60 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f949b5e0740 10 client.17979 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f949b5e0740 1 -- 172.21.15.89:0/945135547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fa1fbd5170 con 0x55fa1fb4ab60 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f947a7fc640 1 -- 172.21.15.89:0/945135547 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f948c04b070 con 0x55fa1fb4ab60 2024-09-15T23:31:30.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f947a7fc640 1 -- 172.21.15.89:0/945135547 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f94840bc0b0 con 0x55fa1fb4ab60 2024-09-15T23:31:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f949b5e0740 10 client.17979 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f949b5e0740 10 client.17979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f949b5e0740 20 client.17979 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f949b5e0740 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fa1fb55d40 0x55fa1fbd7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f949b5e0740 4 client.17979 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:31:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.667+0000 7f949b5e0740 1 -- 172.21.15.89:0/945135547 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fa1fab7d00 con 0x55fa1fb55d40 2024-09-15T23:31:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.668+0000 7f9492d9d640 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fa1fb55d40 0x55fa1fbd7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.668+0000 7f9492d9d640 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fa1fb55d40 0x55fa1fbd7820 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=660ed2dd1631bbd0 server_cookie=c7aa2433c7b82eae in_seq=0 out_seq=0 2024-09-15T23:31:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.668+0000 7f947a7fc640 10 client.17979 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f947a7fc640 1 -- 172.21.15.89:0/945135547 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fa1fab7d00 con 0x55fa1fb55d40 2024-09-15T23:31:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f947a7fc640 10 client.17979 handle_command_reply: tid=0 2024-09-15T23:31:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f949b5e0740 1 client.17979 shutdown 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f949b5e0740 1 -- 172.21.15.89:0/945135547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 msgr2=0x55fa1fbc8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f949b5e0740 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fbc8da0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f947c00fc30 tx=0x7f947c00b770 comp rx=0 tx=0).stop 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f949b5e0740 1 -- 172.21.15.89:0/945135547 shutdown_connections 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f949b5e0740 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fa1fb55d40 0x55fa1fbd7820 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f949b5e0740 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fa1fb4b480 0x55fa1fbcd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f949b5e0740 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa1fb4ab60 0x55fa1fbc8da0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f949b5e0740 1 --2- 172.21.15.89:0/945135547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fa1fb470f0 0x55fa1fbc8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f949b5e0740 1 -- 172.21.15.89:0/945135547 >> 172.21.15.89:0/945135547 conn(0x55fa1fb4e560 msgr2=0x55fa1fb46270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.669+0000 7f947a7fc640 0 client.17979 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f949b5e0740 1 -- 172.21.15.89:0/945135547 shutdown_connections 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f949b5e0740 1 -- 172.21.15.89:0/945135547 wait complete. 2024-09-15T23:31:30.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f949b5e0740 20 client.17979 trim_cache size 0 max 16384 2024-09-15T23:31:30.677 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:31:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:31:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "4805e906-af96-4293-8e98-fe7176f792b4", 2024-09-15T23:31:30.677 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:31:30.677 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 -- 172.21.15.89:0/1357832078 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94800a3240 msgr2=0x7f94800a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94800a3240 0x7f94800a5700 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f948c005e30 tx=0x7f948c03d040 comp rx=0 tx=0).stop 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 -- 172.21.15.89:0/1357832078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9494102d40 msgr2=0x7f9494198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9494102d40 0x7f9494198370 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f948400c920 tx=0x7f948400cde0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 -- 172.21.15.89:0/1357832078 shutdown_connections 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94800a3240 0x7f94800a5700 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949410c170 0x7f949419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9494103660 0x7f94941988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 --2- 172.21.15.89:0/1357832078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9494102d40 0x7f9494198370 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 -- 172.21.15.89:0/1357832078 >> 172.21.15.89:0/1357832078 conn(0x7f94940fc7a0 msgr2=0x7f94940fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:30.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 -- 172.21.15.89:0/1357832078 shutdown_connections 2024-09-15T23:31:30.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:30.670+0000 7f9499207640 1 -- 172.21.15.89:0/1357832078 wait complete. 2024-09-15T23:31:31.442 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:31 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:31:31.442 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:31 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:31.442 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:31 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:31:31.442 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:31 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:31.472 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:31:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:31 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:31:31.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:31 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:31.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:31 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:31:31.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:31 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:31 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:31:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:31 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:31.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:31 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:31:31.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:31 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:31.721 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 -- 172.21.15.89:0/98140016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda24100a80 msgr2=0x7fda24100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 --2- 172.21.15.89:0/98140016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda24100a80 0x7fda24100ee0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fda10009930 tx=0x7fda1002a420 comp rx=0 tx=0).stop 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 -- 172.21.15.89:0/98140016 shutdown_connections 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 --2- 172.21.15.89:0/98140016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda24101420 0x7fda2410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 --2- 172.21.15.89:0/98140016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda24100a80 0x7fda24100ee0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 --2- 172.21.15.89:0/98140016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda24106a80 0x7fda24106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 -- 172.21.15.89:0/98140016 >> 172.21.15.89:0/98140016 conn(0x7fda240fc7a0 msgr2=0x7fda240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 -- 172.21.15.89:0/98140016 shutdown_connections 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 -- 172.21.15.89:0/98140016 wait complete. 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 Processor -- start 2024-09-15T23:31:32.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.217+0000 7fda2b1d5640 1 -- start start 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda2b1d5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda24100a80 0x7fda24199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda2b1d5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda24101420 0x7fda241997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda2b1d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda24106a80 0x7fda241a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda2b1d5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fda24110e80 con 0x7fda24100a80 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda2b1d5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda24110d00 con 0x7fda24106a80 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda2b1d5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fda24111000 con 0x7fda24101420 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda28f71640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda24100a80 0x7fda24199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda28f71640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda24100a80 0x7fda24199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59640/0 (socket says 172.21.15.89:59640) 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda28f71640 1 -- 172.21.15.89:0/1405099403 learned_addr learned my addr 172.21.15.89:0/1405099403 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda23fff640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda24101420 0x7fda241997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda28f71640 1 -- 172.21.15.89:0/1405099403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda24101420 msgr2=0x7fda241997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:32.249 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda29772640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda24106a80 0x7fda241a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda28f71640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda24101420 0x7fda241997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda28f71640 1 -- 172.21.15.89:0/1405099403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda24106a80 msgr2=0x7fda241a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda28f71640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda24106a80 0x7fda241a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda28f71640 1 -- 172.21.15.89:0/1405099403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda241a4840 con 0x7fda24100a80 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda29772640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda24106a80 0x7fda241a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda28f71640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda24100a80 0x7fda24199280 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fda1800c890 tx=0x7fda1800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda21ffb640 1 -- 172.21.15.89:0/1405099403 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda18004220 con 0x7fda24100a80 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda21ffb640 1 -- 172.21.15.89:0/1405099403 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda180043c0 con 0x7fda24100a80 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda2b1d5640 1 -- 172.21.15.89:0/1405099403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda241a4a10 con 0x7fda24100a80 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.218+0000 7fda21ffb640 1 -- 172.21.15.89:0/1405099403 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda180049e0 con 0x7fda24100a80 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.219+0000 7fda2b1d5640 1 -- 172.21.15.89:0/1405099403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda241a4f20 con 0x7fda24100a80 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.220+0000 7fda21ffb640 1 -- 172.21.15.89:0/1405099403 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fda180079b0 con 0x7fda24100a80 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda21ffb640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda140a3240 0x7fda140a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.250 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda2d5ae740 1 Processor -- start 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda2d5ae740 1 -- start start 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda23fff640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda140a3240 0x7fda140a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda21ffb640 1 -- 172.21.15.89:0/1405099403 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fda180cfdd0 con 0x7fda24100a80 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 0x558e2d5db780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5dbcc0 0x558e2d5ea550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d5ece80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558e2d5412e0 con 0x558e2d5eaa90 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558e2d4e8ce0 con 0x558e2d5db3a0 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.221+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558e2d53d2a0 con 0x558e2d5dbcc0 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda29772640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d5ece80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda29772640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d5ece80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59656/0 (socket says 172.21.15.89:59656) 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda29772640 1 -- 172.21.15.89:0/496228315 learned_addr learned my addr 172.21.15.89:0/496228315 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:32.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda28f71640 1 --2- 172.21.15.89:0/496228315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 0x558e2d5db780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda23fff640 1 --2- 172.21.15.89:0/496228315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5dbcc0 0x558e2d5ea550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda23fff640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda140a3240 0x7fda140a5700 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fda100096c0 tx=0x7fda10000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda29772640 1 -- 172.21.15.89:0/496228315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5dbcc0 msgr2=0x558e2d5ea550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda29772640 1 --2- 172.21.15.89:0/496228315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5dbcc0 0x558e2d5ea550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda29772640 1 -- 172.21.15.89:0/496228315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 msgr2=0x558e2d5db780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda29772640 1 --2- 172.21.15.89:0/496228315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 0x558e2d5db780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda29772640 1 -- 172.21.15.89:0/496228315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2d4cd440 con 0x558e2d5eaa90 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda29772640 1 --2- 172.21.15.89:0/496228315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d5ece80 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fda1c008e60 tx=0x7fda1c00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e043a9ab7498be server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda0effd640 1 -- 172.21.15.89:0/496228315 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c00f830 con 0x558e2d5eaa90 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda0effd640 1 -- 172.21.15.89:0/496228315 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda1c018480 con 0x558e2d5eaa90 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.222+0000 7fda0effd640 1 -- 172.21.15.89:0/496228315 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c00e610 con 0x558e2d5eaa90 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 -- 172.21.15.89:0/496228315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 msgr2=0x558e2d5ece80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:32.252 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/496228315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d5ece80 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fda1c008e60 tx=0x7fda1c00d710 comp rx=0 tx=0).stop 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 -- 172.21.15.89:0/496228315 shutdown_connections 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/496228315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d5ece80 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/496228315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5dbcc0 0x558e2d5ea550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/496228315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 0x558e2d5db780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 -- 172.21.15.89:0/496228315 >> 172.21.15.89:0/496228315 conn(0x558e2d5d7a30 msgr2=0x558e2d5d7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 -- 172.21.15.89:0/496228315 shutdown_connections 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 -- 172.21.15.89:0/496228315 wait complete. 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 -- start start 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.223+0000 7fda2d5ae740 1 Processor -- start 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5db3a0 0x558e2d5ea400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5eaa90 0x558e2d5e7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558e2d5ed6c0 con 0x558e2d5dbcc0 2024-09-15T23:31:32.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558e2d4fc050 con 0x558e2d5eaa90 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558e2d5ed840 con 0x558e2d5db3a0 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda2d5ae740 10 client.0 _ll_register_callbacks cb 0x558e2d53c160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda23fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda23fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59662/0 (socket says 172.21.15.89:59662) 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda23fff640 1 -- 172.21.15.89:0/2800837314 learned_addr learned my addr 172.21.15.89:0/2800837314 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda29772640 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5eaa90 0x558e2d5e7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda28f71640 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5db3a0 0x558e2d5ea400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda23fff640 1 -- 172.21.15.89:0/2800837314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5db3a0 msgr2=0x558e2d5ea400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda23fff640 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5db3a0 0x558e2d5ea400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda23fff640 1 -- 172.21.15.89:0/2800837314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5eaa90 msgr2=0x558e2d5e7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda23fff640 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5eaa90 0x558e2d5e7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda23fff640 1 -- 172.21.15.89:0/2800837314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e2d2b23c0 con 0x558e2d5dbcc0 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda23fff640 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e76b0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fda10009900 tx=0x7fda10009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda3b0d2d4d089d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:32.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.224+0000 7fda03fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda03fff640 1 -- 172.21.15.89:0/2800837314 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10041070 con 0x558e2d5dbcc0 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda03fff640 1 -- 172.21.15.89:0/2800837314 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fda10005450 con 0x558e2d5dbcc0 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda2d5ae740 1 -- 172.21.15.89:0/2800837314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda1c018480 con 0x558e2d5dbcc0 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda03fff640 1 -- 172.21.15.89:0/2800837314 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fda10056050 con 0x558e2d5dbcc0 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda2d5ae740 10 client.17991 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda2d5ae740 1 -- 172.21.15.89:0/2800837314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558e2d5f9130 con 0x558e2d5dbcc0 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda03fff640 1 -- 172.21.15.89:0/2800837314 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda1c018480 con 0x558e2d5dbcc0 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda03fff640 1 -- 172.21.15.89:0/2800837314 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fda10003820 con 0x558e2d5dbcc0 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda2d5ae740 10 client.17991 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda2d5ae740 10 client.17991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda2d5ae740 20 client.17991 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558e2d5fa640 0x558e2d5fca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda2d5ae740 4 client.17991 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:32.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.225+0000 7fda2d5ae740 1 -- 172.21.15.89:0/2800837314 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e2d54e850 con 0x558e2d5fa640 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.226+0000 7fda28f71640 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558e2d5fa640 0x558e2d5fca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.226+0000 7fda28f71640 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558e2d5fa640 0x558e2d5fca80 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ba8e5944c4f7394 server_cookie=fad047e7150412d4 in_seq=0 out_seq=0 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.226+0000 7fda03fff640 10 client.17991 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda03fff640 1 -- 172.21.15.89:0/2800837314 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558e2d54e850 con 0x558e2d5fa640 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda03fff640 10 client.17991 handle_command_reply: tid=0 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 client.17991 shutdown 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 -- 172.21.15.89:0/2800837314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 msgr2=0x558e2d5e76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e76b0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fda10009900 tx=0x7fda10009260 comp rx=0 tx=0).stop 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 -- 172.21.15.89:0/2800837314 shutdown_connections 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558e2d5fa640 0x558e2d5fca80 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5eaa90 0x558e2d5e7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda03fff640 0 client.17991 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e76b0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/2800837314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5db3a0 0x558e2d5ea400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 -- 172.21.15.89:0/2800837314 >> 172.21.15.89:0/2800837314 conn(0x558e2d5d7a30 msgr2=0x558e2d5dd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 -- 172.21.15.89:0/2800837314 shutdown_connections 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 1 -- 172.21.15.89:0/2800837314 wait complete. 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.227+0000 7fda2d5ae740 20 client.17991 trim_cache size 0 max 16384 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda2d5ae740 1 Processor -- start 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda2d5ae740 1 -- start start 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5db3a0 0x558e2d5f69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d602350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fda241acd60 con 0x558e2d5eaa90 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda241ae8a0 con 0x558e2d5dbcc0 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fda241ad430 con 0x558e2d5db3a0 2024-09-15T23:31:32.257 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda29772640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d602350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda29772640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d602350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59664/0 (socket says 172.21.15.89:59664) 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.244+0000 7fda29772640 1 -- 172.21.15.89:0/1055122477 learned_addr learned my addr 172.21.15.89:0/1055122477 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda29772640 1 -- 172.21.15.89:0/1055122477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5db3a0 msgr2=0x558e2d5f69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda29772640 1 --2- 172.21.15.89:0/1055122477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5db3a0 0x558e2d5f69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda29772640 1 -- 172.21.15.89:0/1055122477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5dbcc0 msgr2=0x558e2d5e5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda23fff640 1 --2- 172.21.15.89:0/1055122477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda29772640 1 --2- 172.21.15.89:0/1055122477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda29772640 1 -- 172.21.15.89:0/1055122477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e2d5e6660 con 0x558e2d5eaa90 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda23fff640 1 --2- 172.21.15.89:0/1055122477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda29772640 1 --2- 172.21.15.89:0/1055122477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d602350 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fda1c000f80 tx=0x7fda1c018b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd0462f0623b91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda0e7fc640 1 -- 172.21.15.89:0/1055122477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c02a070 con 0x558e2d5eaa90 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda0e7fc640 1 -- 172.21.15.89:0/1055122477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda1c018480 con 0x558e2d5eaa90 2024-09-15T23:31:32.258 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.245+0000 7fda0e7fc640 1 -- 172.21.15.89:0/1055122477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c00f920 con 0x558e2d5eaa90 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.246+0000 7fda2d5ae740 1 -- 172.21.15.89:0/1055122477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 msgr2=0x558e2d602350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.246+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/1055122477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d602350 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fda1c000f80 tx=0x7fda1c018b90 comp rx=0 tx=0).stop 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.246+0000 7fda2d5ae740 1 -- 172.21.15.89:0/1055122477 shutdown_connections 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.246+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/1055122477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5eaa90 0x558e2d602350 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.246+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/1055122477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5dbcc0 0x558e2d5e5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.246+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/1055122477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5db3a0 0x558e2d5f69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.246+0000 7fda2d5ae740 1 -- 172.21.15.89:0/1055122477 >> 172.21.15.89:0/1055122477 conn(0x558e2d5d7a30 msgr2=0x558e2d5e2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.246+0000 7fda2d5ae740 1 -- 172.21.15.89:0/1055122477 shutdown_connections 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.246+0000 7fda2d5ae740 1 -- 172.21.15.89:0/1055122477 wait complete. 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda2d5ae740 1 -- start start 2024-09-15T23:31:32.259 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda2d5ae740 1 Processor -- start 2024-09-15T23:31:32.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 0x558e2d5f3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda2d5ae740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5eaa90 0x558e2d5eef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fda241ae1d0 con 0x558e2d5dbcc0 2024-09-15T23:31:32.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda241adb00 con 0x558e2d5db3a0 2024-09-15T23:31:32.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda2d5ae740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fda241aad40 con 0x558e2d5eaa90 2024-09-15T23:31:32.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda2d5ae740 10 client.0 _ll_register_callbacks cb 0x7fda10009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda23fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda23fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5ee7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59672/0 (socket says 172.21.15.89:59672) 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.247+0000 7fda23fff640 1 -- 172.21.15.89:0/882143177 learned_addr learned my addr 172.21.15.89:0/882143177 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda29772640 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5eaa90 0x558e2d5eef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda28f71640 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 0x558e2d5f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda23fff640 1 -- 172.21.15.89:0/882143177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5eaa90 msgr2=0x558e2d5eef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda23fff640 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5eaa90 0x558e2d5eef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda23fff640 1 -- 172.21.15.89:0/882143177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 msgr2=0x558e2d5f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda23fff640 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 0x558e2d5f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda23fff640 1 -- 172.21.15.89:0/882143177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e2d664320 con 0x558e2d5dbcc0 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda29772640 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5eaa90 0x558e2d5eef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda28f71640 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 0x558e2d5f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda23fff640 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5ee7c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fda10003c70 tx=0x7fda10005500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814fa145a08d7ec6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:32.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda03fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda03fff640 1 -- 172.21.15.89:0/882143177 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10041070 con 0x558e2d5dbcc0 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda03fff640 1 -- 172.21.15.89:0/882143177 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fda100046c0 con 0x558e2d5dbcc0 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda2d5ae740 1 -- 172.21.15.89:0/882143177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda1c018480 con 0x558e2d5dbcc0 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda03fff640 1 -- 172.21.15.89:0/882143177 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fda10068050 con 0x558e2d5dbcc0 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda2d5ae740 10 client.17997 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda2d5ae740 1 -- 172.21.15.89:0/882143177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558e2d665460 con 0x558e2d5dbcc0 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.248+0000 7fda03fff640 1 -- 172.21.15.89:0/882143177 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda1c018480 con 0x558e2d5dbcc0 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda03fff640 1 -- 172.21.15.89:0/882143177 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fda1003b460 con 0x558e2d5dbcc0 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda2d5ae740 10 client.17997 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda2d5ae740 10 client.17997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda2d5ae740 20 client.17997 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558e2d665bd0 0x558e2d667ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda2d5ae740 4 client.17997 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:31:32.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda2d5ae740 1 -- 172.21.15.89:0/882143177 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558e2d548d60 con 0x558e2d665bd0 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda28f71640 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558e2d665bd0 0x558e2d667ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda28f71640 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558e2d665bd0 0x558e2d667ff0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15633c0a76691220 server_cookie=92cf524b870cce75 in_seq=0 out_seq=0 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda03fff640 10 client.17997 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda03fff640 1 -- 172.21.15.89:0/882143177 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558e2d548d60 con 0x558e2d665bd0 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda03fff640 10 client.17997 handle_command_reply: tid=0 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.249+0000 7fda2d5ae740 1 client.17997 shutdown 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 1 -- 172.21.15.89:0/882143177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 msgr2=0x558e2d5ee7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5ee7c0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fda10003c70 tx=0x7fda10005500 comp rx=0 tx=0).stop 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 1 -- 172.21.15.89:0/882143177 shutdown_connections 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558e2d665bd0 0x558e2d667ff0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558e2d5eaa90 0x558e2d5eef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558e2d5dbcc0 0x558e2d5ee7c0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 1 --2- 172.21.15.89:0/882143177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e2d5db3a0 0x558e2d5f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda03fff640 0 client.17997 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 1 -- 172.21.15.89:0/882143177 >> 172.21.15.89:0/882143177 conn(0x558e2d5d7a30 msgr2=0x558e2d5e2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 1 -- 172.21.15.89:0/882143177 shutdown_connections 2024-09-15T23:31:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 1 -- 172.21.15.89:0/882143177 wait complete. 2024-09-15T23:31:32.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2d5ae740 20 client.17997 trim_cache size 0 max 16384 2024-09-15T23:31:32.266 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:31:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:31:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:31:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "4805e906-af96-4293-8e98-fe7176f792b4": { 2024-09-15T23:31:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:31:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "4805e906-af96-4293-8e98-fe7176f792b4", 2024-09-15T23:31:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:31:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:31:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:31:32.267 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:31:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2b1d5640 1 -- 172.21.15.89:0/1405099403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda140a3240 msgr2=0x7fda140a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:32.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2b1d5640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda140a3240 0x7fda140a5700 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fda100096c0 tx=0x7fda10000bc0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2b1d5640 1 -- 172.21.15.89:0/1405099403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda24100a80 msgr2=0x7fda24199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2b1d5640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda24100a80 0x7fda24199280 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fda1800c890 tx=0x7fda1800cd50 comp rx=0 tx=0).stop 2024-09-15T23:31:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2b1d5640 1 -- 172.21.15.89:0/1405099403 shutdown_connections 2024-09-15T23:31:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2b1d5640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda140a3240 0x7fda140a5700 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2b1d5640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda24106a80 0x7fda241a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2b1d5640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda24101420 0x7fda241997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.250+0000 7fda2b1d5640 1 --2- 172.21.15.89:0/1405099403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda24100a80 0x7fda24199280 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.251+0000 7fda2b1d5640 1 -- 172.21.15.89:0/1405099403 >> 172.21.15.89:0/1405099403 conn(0x7fda240fc7a0 msgr2=0x7fda240fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.251+0000 7fda2b1d5640 1 -- 172.21.15.89:0/1405099403 shutdown_connections 2024-09-15T23:31:32.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:32.251+0000 7fda2b1d5640 1 -- 172.21.15.89:0/1405099403 wait complete. 2024-09-15T23:31:32.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:32 smithi120 ceph-mon[35250]: pgmap v575: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:32.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:32 smithi174 ceph-mon[35586]: pgmap v575: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:32.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:32 smithi089 ceph-mon[30933]: pgmap v575: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:32.947 INFO:tasks.cephfs.filesystem:scrub status for tag:4805e906-af96-4293-8e98-fe7176f792b4 - {'path': '/', 'tag': '4805e906-af96-4293-8e98-fe7176f792b4', 'options': 'recursive,force'} 2024-09-15T23:31:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:33 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:31:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:33 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:31:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:33 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:31:34.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:34 smithi120 ceph-mon[35250]: pgmap v576: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:31:34.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:34 smithi174 ceph-mon[35586]: pgmap v576: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:31:34.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:34 smithi089 ceph-mon[30933]: pgmap v576: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:31:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:35 smithi120 ceph-mon[35250]: pgmap v577: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:31:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:35 smithi174 ceph-mon[35586]: pgmap v577: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:31:35.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:35 smithi089 ceph-mon[30933]: pgmap v577: 129 pgs: 129 active+clean; 3.7 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:31:36.695 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.695 INFO:tasks.workunit.client.0.smithi089.stdout:189/600: open fd = 3 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:190/600: open fd = 3 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:191/600: open fd = 3 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:192/600: open fd = 3 2024-09-15T23:31:36.696 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:193/600: open fd = 3 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:194/600: open fd = 3 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.697 INFO:tasks.workunit.client.0.smithi089.stdout:195/600: open fd = 3 2024-09-15T23:31:36.698 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.698 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.698 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.698 INFO:tasks.workunit.client.0.smithi089.stdout:196/600: open fd = 3 2024-09-15T23:31:36.698 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.698 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.698 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.699 INFO:tasks.workunit.client.0.smithi089.stdout:197/600: open fd = 3 2024-09-15T23:31:36.699 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.699 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.699 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.699 INFO:tasks.workunit.client.0.smithi089.stdout:198/600: open fd = 3 2024-09-15T23:31:36.699 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.700 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.700 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.700 INFO:tasks.workunit.client.0.smithi089.stdout:199/600: open fd = 3 2024-09-15T23:31:36.700 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.700 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.700 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.700 INFO:tasks.workunit.client.0.smithi089.stdout:200/600: open fd = 3 2024-09-15T23:31:36.701 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.701 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.701 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.701 INFO:tasks.workunit.client.0.smithi089.stdout:201/600: open fd = 3 2024-09-15T23:31:36.702 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.702 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.702 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.702 INFO:tasks.workunit.client.0.smithi089.stdout:202/600: open fd = 3 2024-09-15T23:31:36.702 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.702 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.702 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.703 INFO:tasks.workunit.client.0.smithi089.stdout:203/600: open fd = 3 2024-09-15T23:31:36.703 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.703 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.703 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.703 INFO:tasks.workunit.client.0.smithi089.stdout:204/600: open fd = 3 2024-09-15T23:31:36.704 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.704 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.704 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.704 INFO:tasks.workunit.client.0.smithi089.stdout:205/600: open fd = 3 2024-09-15T23:31:36.704 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.704 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.704 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.705 INFO:tasks.workunit.client.0.smithi089.stdout:206/600: open fd = 3 2024-09-15T23:31:36.705 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.705 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.705 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.705 INFO:tasks.workunit.client.0.smithi089.stdout:207/600: open fd = 3 2024-09-15T23:31:36.706 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.706 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.706 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.706 INFO:tasks.workunit.client.0.smithi089.stdout:208/600: open fd = 3 2024-09-15T23:31:36.706 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.706 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.707 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.707 INFO:tasks.workunit.client.0.smithi089.stdout:209/600: open fd = 3 2024-09-15T23:31:36.707 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.707 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.707 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.707 INFO:tasks.workunit.client.0.smithi089.stdout:210/600: open fd = 3 2024-09-15T23:31:36.707 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.708 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.708 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.708 INFO:tasks.workunit.client.0.smithi089.stdout:211/600: open fd = 3 2024-09-15T23:31:36.708 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.708 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.709 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.709 INFO:tasks.workunit.client.0.smithi089.stdout:212/600: open fd = 3 2024-09-15T23:31:36.709 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.709 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.709 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.709 INFO:tasks.workunit.client.0.smithi089.stdout:213/600: open fd = 3 2024-09-15T23:31:36.710 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.710 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.710 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.710 INFO:tasks.workunit.client.0.smithi089.stdout:214/600: open fd = 3 2024-09-15T23:31:36.710 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.710 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.711 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.711 INFO:tasks.workunit.client.0.smithi089.stdout:215/600: open fd = 3 2024-09-15T23:31:36.711 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.711 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.711 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.711 INFO:tasks.workunit.client.0.smithi089.stdout:216/600: open fd = 3 2024-09-15T23:31:36.712 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.712 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.712 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.712 INFO:tasks.workunit.client.0.smithi089.stdout:217/600: open fd = 3 2024-09-15T23:31:36.712 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.712 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.713 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.713 INFO:tasks.workunit.client.0.smithi089.stdout:218/600: open fd = 3 2024-09-15T23:31:36.713 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.713 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.713 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.713 INFO:tasks.workunit.client.0.smithi089.stdout:219/600: open fd = 3 2024-09-15T23:31:36.714 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.714 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.714 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.714 INFO:tasks.workunit.client.0.smithi089.stdout:220/600: open fd = 3 2024-09-15T23:31:36.714 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.714 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.715 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.715 INFO:tasks.workunit.client.0.smithi089.stdout:221/600: open fd = 3 2024-09-15T23:31:36.715 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.715 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.715 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.715 INFO:tasks.workunit.client.0.smithi089.stdout:222/600: open fd = 3 2024-09-15T23:31:36.716 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.716 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.716 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.716 INFO:tasks.workunit.client.0.smithi089.stdout:223/600: open fd = 3 2024-09-15T23:31:36.716 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.716 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.717 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.717 INFO:tasks.workunit.client.0.smithi089.stdout:224/600: open fd = 3 2024-09-15T23:31:36.717 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.717 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.717 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.717 INFO:tasks.workunit.client.0.smithi089.stdout:225/600: open fd = 3 2024-09-15T23:31:36.718 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.718 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.718 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.718 INFO:tasks.workunit.client.0.smithi089.stdout:226/600: open fd = 3 2024-09-15T23:31:36.718 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.718 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.719 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.719 INFO:tasks.workunit.client.0.smithi089.stdout:227/600: open fd = 3 2024-09-15T23:31:36.719 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.719 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.719 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.719 INFO:tasks.workunit.client.0.smithi089.stdout:228/600: open fd = 3 2024-09-15T23:31:36.720 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.720 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.720 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.720 INFO:tasks.workunit.client.0.smithi089.stdout:229/600: open fd = 3 2024-09-15T23:31:36.720 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.720 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.721 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.721 INFO:tasks.workunit.client.0.smithi089.stdout:230/600: open fd = 3 2024-09-15T23:31:36.721 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.721 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.721 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.721 INFO:tasks.workunit.client.0.smithi089.stdout:231/600: open fd = 3 2024-09-15T23:31:36.722 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.722 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.722 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.722 INFO:tasks.workunit.client.0.smithi089.stdout:232/600: open fd = 3 2024-09-15T23:31:36.722 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.722 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.723 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.723 INFO:tasks.workunit.client.0.smithi089.stdout:233/600: open fd = 3 2024-09-15T23:31:36.723 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.723 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.723 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.723 INFO:tasks.workunit.client.0.smithi089.stdout:234/600: open fd = 3 2024-09-15T23:31:36.723 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.723 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.724 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.724 INFO:tasks.workunit.client.0.smithi089.stdout:235/600: open fd = 3 2024-09-15T23:31:36.724 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.724 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.724 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.724 INFO:tasks.workunit.client.0.smithi089.stdout:236/600: open fd = 3 2024-09-15T23:31:36.725 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.725 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.725 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.725 INFO:tasks.workunit.client.0.smithi089.stdout:237/600: open fd = 3 2024-09-15T23:31:36.725 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.725 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.725 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:238/600: open fd = 3 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:239/600: open fd = 3 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:240/600: open fd = 3 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.726 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:241/600: open fd = 3 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:242/600: open fd = 3 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:243/600: open fd = 3 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:244/600: open fd = 3 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.727 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:245/600: open fd = 3 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:246/600: open fd = 3 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:247/600: open fd = 3 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.728 INFO:tasks.workunit.client.0.smithi089.stdout:248/600: open fd = 3 2024-09-15T23:31:36.729 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.729 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.729 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.729 INFO:tasks.workunit.client.0.smithi089.stdout:249/600: open fd = 3 2024-09-15T23:31:36.729 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.729 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.729 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:31:36.729 INFO:tasks.workunit.client.0.smithi089.stdout:250/600: open fd = 3 2024-09-15T23:31:36.729 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:36.729 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:31:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:37 smithi120 ceph-mon[35250]: pgmap v578: 129 pgs: 129 active+clean; 3.8 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:31:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:37 smithi174 ceph-mon[35586]: pgmap v578: 129 pgs: 129 active+clean; 3.8 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:31:37.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:37 smithi089 ceph-mon[30933]: pgmap v578: 129 pgs: 129 active+clean; 3.8 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:31:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:39 smithi120 ceph-mon[35250]: pgmap v579: 129 pgs: 129 active+clean; 3.8 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:39 smithi174 ceph-mon[35586]: pgmap v579: 129 pgs: 129 active+clean; 3.8 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:39.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:39 smithi089 ceph-mon[30933]: pgmap v579: 129 pgs: 129 active+clean; 3.8 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:31:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:41 smithi120 ceph-mon[35250]: pgmap v580: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:31:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:41 smithi174 ceph-mon[35586]: pgmap v580: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:31:41.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:41 smithi089 ceph-mon[30933]: pgmap v580: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:31:41.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.578+0000 7f316f103640 1 -- 172.21.15.89:0/4292318249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3168101c30 msgr2=0x7f31681020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:41.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.578+0000 7f316f103640 1 --2- 172.21.15.89:0/4292318249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3168101c30 0x7f31681020b0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f3154009900 tx=0x7f315402a3f0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.578+0000 7f316f103640 1 -- 172.21.15.89:0/4292318249 shutdown_connections 2024-09-15T23:31:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.578+0000 7f316f103640 1 --2- 172.21.15.89:0/4292318249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31681025f0 0x7f3168108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.578+0000 7f316f103640 1 --2- 172.21.15.89:0/4292318249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3168101c30 0x7f31681020b0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.578+0000 7f316f103640 1 --2- 172.21.15.89:0/4292318249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3168100a40 0x7f3168100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.578+0000 7f316f103640 1 -- 172.21.15.89:0/4292318249 >> 172.21.15.89:0/4292318249 conn(0x7f31680fc1e0 msgr2=0x7f31680fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.578+0000 7f316f103640 1 -- 172.21.15.89:0/4292318249 shutdown_connections 2024-09-15T23:31:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.578+0000 7f316f103640 1 -- 172.21.15.89:0/4292318249 wait complete. 2024-09-15T23:31:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f316f103640 1 Processor -- start 2024-09-15T23:31:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f316f103640 1 -- start start 2024-09-15T23:31:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f316f103640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3168100a40 0x7f3168196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f316f103640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3168101c30 0x7f31681971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f316f103640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31681025f0 0x7f316819e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f316f103640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f316810c6a0 con 0x7f3168101c30 2024-09-15T23:31:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f316f103640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f316810c520 con 0x7f31681025f0 2024-09-15T23:31:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f316f103640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f316810c820 con 0x7f3168100a40 2024-09-15T23:31:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f3167fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3168101c30 0x7f31681971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f3167fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3168101c30 0x7f31681971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60750/0 (socket says 172.21.15.89:60750) 2024-09-15T23:31:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f3167fff640 1 -- 172.21.15.89:0/754693583 learned_addr learned my addr 172.21.15.89:0/754693583 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:41.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f316d6a0640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31681025f0 0x7f316819e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.579+0000 7f316ce9f640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3168100a40 0x7f3168196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f3167fff640 1 -- 172.21.15.89:0/754693583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3168100a40 msgr2=0x7f3168196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f3167fff640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3168100a40 0x7f3168196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f3167fff640 1 -- 172.21.15.89:0/754693583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31681025f0 msgr2=0x7f316819e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f3167fff640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31681025f0 0x7f316819e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f3167fff640 1 -- 172.21.15.89:0/754693583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316819e890 con 0x7f3168101c30 2024-09-15T23:31:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f3167fff640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3168101c30 0x7f31681971c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f3154009230 tx=0x7f3154004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f3165ffb640 1 -- 172.21.15.89:0/754693583 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315403f030 con 0x7f3168101c30 2024-09-15T23:31:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f3165ffb640 1 -- 172.21.15.89:0/754693583 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3154003ce0 con 0x7f3168101c30 2024-09-15T23:31:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f316f103640 1 -- 172.21.15.89:0/754693583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f316819eb20 con 0x7f3168101c30 2024-09-15T23:31:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f3165ffb640 1 -- 172.21.15.89:0/754693583 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154035070 con 0x7f3168101c30 2024-09-15T23:31:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.580+0000 7f316f103640 1 -- 172.21.15.89:0/754693583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f316819f060 con 0x7f3168101c30 2024-09-15T23:31:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f3165ffb640 1 -- 172.21.15.89:0/754693583 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3154042030 con 0x7f3168101c30 2024-09-15T23:31:41.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f3171e98740 1 Processor -- start 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f3171e98740 1 -- start start 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f3171e98740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a8bb0 0x556b796a8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f3171e98740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f3171e98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 0x556b796b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f3171e98740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556b7960a2f0 con 0x556b796a94f0 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f3171e98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556b795b2270 con 0x556b796ae580 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f3171e98740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556b796062b0 con 0x556b796a8bb0 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f316ce9f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f3165ffb640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31580a3370 0x7f31580a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f316ce9f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60752/0 (socket says 172.21.15.89:60752) 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f316ce9f640 1 -- 172.21.15.89:0/3559462282 learned_addr learned my addr 172.21.15.89:0/3559462282 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:41.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.583+0000 7f316d6a0640 1 --2- 172.21.15.89:0/3559462282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a8bb0 0x556b796a8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f316d6a0640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31580a3370 0x7f31580a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f3165ffb640 1 -- 172.21.15.89:0/754693583 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f31540f0490 con 0x7f3168101c30 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f3167fff640 1 --2- 172.21.15.89:0/3559462282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 0x556b796b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f316ce9f640 1 -- 172.21.15.89:0/3559462282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a8bb0 msgr2=0x556b796a8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f316ce9f640 1 --2- 172.21.15.89:0/3559462282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a8bb0 0x556b796a8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f316ce9f640 1 -- 172.21.15.89:0/3559462282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 msgr2=0x556b796b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f316ce9f640 1 --2- 172.21.15.89:0/3559462282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 0x556b796b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f316ce9f640 1 -- 172.21.15.89:0/3559462282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b795b1340 con 0x556b796a94f0 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f3167fff640 1 --2- 172.21.15.89:0/3559462282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 0x556b796b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f316ce9f640 1 --2- 172.21.15.89:0/3559462282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796adfb0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f315c00cc00 tx=0x7f315c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaec59405a019141 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f316d6a0640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31580a3370 0x7f31580a5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f316000afd0 tx=0x7f3160004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:41.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f3152ffd640 1 -- 172.21.15.89:0/3559462282 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c005220 con 0x556b796a94f0 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f3152ffd640 1 -- 172.21.15.89:0/3559462282 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c016070 con 0x556b796a94f0 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.584+0000 7f3152ffd640 1 -- 172.21.15.89:0/3559462282 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c0107b0 con 0x556b796a94f0 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 -- 172.21.15.89:0/3559462282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 msgr2=0x556b796adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 --2- 172.21.15.89:0/3559462282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796adfb0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f315c00cc00 tx=0x7f315c00e720 comp rx=0 tx=0).stop 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 -- 172.21.15.89:0/3559462282 shutdown_connections 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 --2- 172.21.15.89:0/3559462282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 0x556b796b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 --2- 172.21.15.89:0/3559462282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796adfb0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 --2- 172.21.15.89:0/3559462282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a8bb0 0x556b796a8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 -- 172.21.15.89:0/3559462282 >> 172.21.15.89:0/3559462282 conn(0x556b796a4220 msgr2=0x556b796a4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 -- 172.21.15.89:0/3559462282 shutdown_connections 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 -- 172.21.15.89:0/3559462282 wait complete. 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 -- start start 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.585+0000 7f3171e98740 1 Processor -- start 2024-09-15T23:31:41.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3171e98740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a8bb0 0x556b796b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3171e98740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3171e98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 0x556b796ba2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3171e98740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556b796b1550 con 0x556b796a94f0 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3171e98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556b79593290 con 0x556b796ae580 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3171e98740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556b796b16d0 con 0x556b796a8bb0 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3171e98740 10 client.0 _ll_register_callbacks cb 0x556b795c4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3167fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3167fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796b32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60758/0 (socket says 172.21.15.89:60758) 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3167fff640 1 -- 172.21.15.89:0/830723221 learned_addr learned my addr 172.21.15.89:0/830723221 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f316ce9f640 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a8bb0 0x556b796b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f316d6a0640 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 0x556b796ba2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3167fff640 1 -- 172.21.15.89:0/830723221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a8bb0 msgr2=0x556b796b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:41.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3167fff640 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a8bb0 0x556b796b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3167fff640 1 -- 172.21.15.89:0/830723221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 msgr2=0x556b796ba2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3167fff640 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 0x556b796ba2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3167fff640 1 -- 172.21.15.89:0/830723221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b7931d210 con 0x556b796a94f0 2024-09-15T23:31:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f316d6a0640 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 0x556b796ba2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3167fff640 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796b32b0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f31540b8a90 tx=0x7f31540bc550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0dccea69d98c61 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.586+0000 7f3147fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3147fff640 1 -- 172.21.15.89:0/830723221 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154049040 con 0x556b796a94f0 2024-09-15T23:31:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3147fff640 1 -- 172.21.15.89:0/830723221 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f31540d0be0 con 0x556b796a94f0 2024-09-15T23:31:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3171e98740 1 -- 172.21.15.89:0/830723221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f315c016070 con 0x556b796a94f0 2024-09-15T23:31:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3147fff640 1 -- 172.21.15.89:0/830723221 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f31540bcdc0 con 0x556b796a94f0 2024-09-15T23:31:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3171e98740 10 client.18009 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3171e98740 1 -- 172.21.15.89:0/830723221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556b793e9620 con 0x556b796a94f0 2024-09-15T23:31:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3147fff640 1 -- 172.21.15.89:0/830723221 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c016070 con 0x556b796a94f0 2024-09-15T23:31:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3147fff640 1 -- 172.21.15.89:0/830723221 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f31540c4ea0 con 0x556b796a94f0 2024-09-15T23:31:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3171e98740 10 client.18009 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3171e98740 10 client.18009 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3171e98740 20 client.18009 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3171e98740 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556b796be8f0 0x556b796c0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3171e98740 4 client.18009 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f3171e98740 1 -- 172.21.15.89:0/830723221 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b79617a20 con 0x556b796be8f0 2024-09-15T23:31:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.587+0000 7f316ce9f640 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556b796be8f0 0x556b796c0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.588+0000 7f3147fff640 10 client.18009 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.588+0000 7f316ce9f640 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556b796be8f0 0x556b796c0d20 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70df126bc6ea533e server_cookie=cc746ea92e4bda0f in_seq=0 out_seq=0 2024-09-15T23:31:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3147fff640 1 -- 172.21.15.89:0/830723221 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556b79617a20 con 0x556b796be8f0 2024-09-15T23:31:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3147fff640 10 client.18009 handle_command_reply: tid=0 2024-09-15T23:31:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 client.18009 shutdown 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 -- 172.21.15.89:0/830723221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 msgr2=0x556b796b32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796b32b0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f31540b8a90 tx=0x7f31540bc550 comp rx=0 tx=0).stop 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 -- 172.21.15.89:0/830723221 shutdown_connections 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556b796be8f0 0x556b796c0d20 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796ae580 0x556b796ba2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3147fff640 0 client.18009 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796b32b0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 --2- 172.21.15.89:0/830723221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a8bb0 0x556b796b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 -- 172.21.15.89:0/830723221 >> 172.21.15.89:0/830723221 conn(0x556b796a4220 msgr2=0x556b796a5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 -- 172.21.15.89:0/830723221 shutdown_connections 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 1 -- 172.21.15.89:0/830723221 wait complete. 2024-09-15T23:31:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.589+0000 7f3171e98740 20 client.18009 trim_cache size 0 max 16384 2024-09-15T23:31:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.603+0000 7f3171e98740 1 Processor -- start 2024-09-15T23:31:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.603+0000 7f3171e98740 1 -- start start 2024-09-15T23:31:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.603+0000 7f3171e98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796a8bb0 0x556b796b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.603+0000 7f3171e98740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796ba2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.603+0000 7f3171e98740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796ae580 0x556b796a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.603+0000 7f3171e98740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f31681a85a0 con 0x556b796a94f0 2024-09-15T23:31:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.603+0000 7f3171e98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f31681aa0e0 con 0x556b796a8bb0 2024-09-15T23:31:41.612 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.603+0000 7f3171e98740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f31681a8c70 con 0x556b796ae580 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f3167fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796ba2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f3167fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796ba2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60772/0 (socket says 172.21.15.89:60772) 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f3167fff640 1 -- 172.21.15.89:0/1809196430 learned_addr learned my addr 172.21.15.89:0/1809196430 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f3167fff640 1 -- 172.21.15.89:0/1809196430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796ae580 msgr2=0x556b796a50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f3167fff640 1 --2- 172.21.15.89:0/1809196430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796ae580 0x556b796a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f3167fff640 1 -- 172.21.15.89:0/1809196430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796a8bb0 msgr2=0x556b796b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f3167fff640 1 --2- 172.21.15.89:0/1809196430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796a8bb0 0x556b796b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f3167fff640 1 -- 172.21.15.89:0/1809196430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b796a99d0 con 0x556b796a94f0 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f3167fff640 1 --2- 172.21.15.89:0/1809196430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796ba2b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f31540edf10 tx=0x7f31540bc5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81608c59cdef751 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f31527fc640 1 -- 172.21.15.89:0/1809196430 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154049070 con 0x556b796a94f0 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f31527fc640 1 -- 172.21.15.89:0/1809196430 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31540bc6c0 con 0x556b796a94f0 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.604+0000 7f31527fc640 1 -- 172.21.15.89:0/1809196430 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31540b8ca0 con 0x556b796a94f0 2024-09-15T23:31:41.613 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 -- 172.21.15.89:0/1809196430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 msgr2=0x556b796ba2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 --2- 172.21.15.89:0/1809196430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796ba2b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f31540edf10 tx=0x7f31540bc5c0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 -- 172.21.15.89:0/1809196430 shutdown_connections 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 --2- 172.21.15.89:0/1809196430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796ae580 0x556b796a50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 --2- 172.21.15.89:0/1809196430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796a94f0 0x556b796ba2b0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 --2- 172.21.15.89:0/1809196430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796a8bb0 0x556b796b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 -- 172.21.15.89:0/1809196430 >> 172.21.15.89:0/1809196430 conn(0x556b796a4220 msgr2=0x556b796b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 -- 172.21.15.89:0/1809196430 shutdown_connections 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 -- 172.21.15.89:0/1809196430 wait complete. 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 -- start start 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.605+0000 7f3171e98740 1 Processor -- start 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3171e98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796a8bb0 0x556b796b7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3171e98740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a94f0 0x556b796b83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3171e98740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796ae580 0x556b796b9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3171e98740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f31681a9a10 con 0x556b796ae580 2024-09-15T23:31:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3171e98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f31681a9340 con 0x556b796a8bb0 2024-09-15T23:31:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3171e98740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f31681a6580 con 0x556b796a94f0 2024-09-15T23:31:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3171e98740 10 client.0 _ll_register_callbacks cb 0x7f3154005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316d6a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796ae580 0x556b796b9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316d6a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796ae580 0x556b796b9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:60774/0 (socket says 172.21.15.89:60774) 2024-09-15T23:31:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316d6a0640 1 -- 172.21.15.89:0/1773347702 learned_addr learned my addr 172.21.15.89:0/1773347702 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3167fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a94f0 0x556b796b83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316ce9f640 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796a8bb0 0x556b796b7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316d6a0640 1 -- 172.21.15.89:0/1773347702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a94f0 msgr2=0x556b796b83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316d6a0640 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a94f0 0x556b796b83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316d6a0640 1 -- 172.21.15.89:0/1773347702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796a8bb0 msgr2=0x556b796b7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316d6a0640 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796a8bb0 0x556b796b7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316d6a0640 1 -- 172.21.15.89:0/1773347702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b796b9980 con 0x556b796ae580 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316ce9f640 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796a8bb0 0x556b796b7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f316d6a0640 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796ae580 0x556b796b9370 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3160005f10 tx=0x7f316000ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf75d816c5a7a0f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3147fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3147fff640 1 -- 172.21.15.89:0/1773347702 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160019070 con 0x556b796ae580 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3147fff640 1 -- 172.21.15.89:0/1773347702 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f316001d840 con 0x556b796ae580 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.606+0000 7f3171e98740 1 -- 172.21.15.89:0/1773347702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31540bc6c0 con 0x556b796ae580 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3147fff640 1 -- 172.21.15.89:0/1773347702 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f316001da40 con 0x556b796ae580 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3171e98740 10 client.18015 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:41.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3171e98740 1 -- 172.21.15.89:0/1773347702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556b79729bc0 con 0x556b796ae580 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3147fff640 1 -- 172.21.15.89:0/1773347702 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31540bc6c0 con 0x556b796ae580 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3147fff640 1 -- 172.21.15.89:0/1773347702 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3160028070 con 0x556b796ae580 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3171e98740 10 client.18015 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3171e98740 10 client.18015 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3171e98740 20 client.18015 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3171e98740 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556b7972a570 0x556b7972c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3171e98740 4 client.18015 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f3171e98740 1 -- 172.21.15.89:0/1773347702 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556b79611d70 con 0x556b7972a570 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.607+0000 7f316ce9f640 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556b7972a570 0x556b7972c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.608+0000 7f316ce9f640 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556b7972a570 0x556b7972c990 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffbfa2cdfdb85204 server_cookie=81ed6f0a8e76c95a in_seq=0 out_seq=0 2024-09-15T23:31:41.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.608+0000 7f3147fff640 10 client.18015 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3147fff640 1 -- 172.21.15.89:0/1773347702 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556b79611d70 con 0x556b7972a570 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3147fff640 10 client.18015 handle_command_reply: tid=0 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3171e98740 1 client.18015 shutdown 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3171e98740 1 -- 172.21.15.89:0/1773347702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796ae580 msgr2=0x556b796b9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3171e98740 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796ae580 0x556b796b9370 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3160005f10 tx=0x7f316000ad00 comp rx=0 tx=0).stop 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3171e98740 1 -- 172.21.15.89:0/1773347702 shutdown_connections 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3171e98740 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556b7972a570 0x556b7972c990 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3171e98740 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556b796ae580 0x556b796b9370 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3171e98740 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556b796a94f0 0x556b796b83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3171e98740 1 --2- 172.21.15.89:0/1773347702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556b796a8bb0 0x556b796b7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3147fff640 0 client.18015 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.622+0000 7f3171e98740 1 -- 172.21.15.89:0/1773347702 >> 172.21.15.89:0/1773347702 conn(0x556b796a4220 msgr2=0x556b796a09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f3171e98740 1 -- 172.21.15.89:0/1773347702 shutdown_connections 2024-09-15T23:31:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f3171e98740 1 -- 172.21.15.89:0/1773347702 wait complete. 2024-09-15T23:31:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f3171e98740 20 client.18015 trim_cache size 0 max 16384 2024-09-15T23:31:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:31:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:31:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:31:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:31:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 -- 172.21.15.89:0/754693583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31580a3370 msgr2=0x7f31580a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31580a3370 0x7f31580a5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f316000afd0 tx=0x7f3160004ab0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 -- 172.21.15.89:0/754693583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3168101c30 msgr2=0x7f31681971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3168101c30 0x7f31681971c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f3154009230 tx=0x7f3154004430 comp rx=0 tx=0).stop 2024-09-15T23:31:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 -- 172.21.15.89:0/754693583 shutdown_connections 2024-09-15T23:31:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f31580a3370 0x7f31580a5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f31681025f0 0x7f316819e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3168101c30 0x7f31681971c0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 --2- 172.21.15.89:0/754693583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3168100a40 0x7f3168196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 -- 172.21.15.89:0/754693583 >> 172.21.15.89:0/754693583 conn(0x7f31680fc1e0 msgr2=0x7f3168104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 -- 172.21.15.89:0/754693583 shutdown_connections 2024-09-15T23:31:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:41.623+0000 7f316f103640 1 -- 172.21.15.89:0/754693583 wait complete. 2024-09-15T23:31:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:43 smithi120 ceph-mon[35250]: pgmap v581: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:31:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:43 smithi174 ceph-mon[35586]: pgmap v581: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:31:43.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:43 smithi089 ceph-mon[30933]: pgmap v581: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:31:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:45 smithi120 ceph-mon[35250]: pgmap v582: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:31:45.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:45 smithi174 ceph-mon[35586]: pgmap v582: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:31:45.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:45 smithi089 ceph-mon[30933]: pgmap v582: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:31:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:47 smithi120 ceph-mon[35250]: pgmap v583: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:47.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:47 smithi174 ceph-mon[35586]: pgmap v583: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:47.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:47 smithi089 ceph-mon[30933]: pgmap v583: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:31:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:49 smithi120 ceph-mon[35250]: pgmap v584: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:31:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:49 smithi174 ceph-mon[35586]: pgmap v584: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:31:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:49 smithi089 ceph-mon[30933]: pgmap v584: 129 pgs: 129 active+clean; 3.9 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:31:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:51 smithi174 ceph-mon[35586]: pgmap v585: 129 pgs: 129 active+clean; 4.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:31:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:51 smithi089 ceph-mon[30933]: pgmap v585: 129 pgs: 129 active+clean; 4.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:31:51.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:51 smithi120 ceph-mon[35250]: pgmap v585: 129 pgs: 129 active+clean; 4.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:31:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:53 smithi120 ceph-mon[35250]: pgmap v586: 129 pgs: 129 active+clean; 4.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:31:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:53 smithi174 ceph-mon[35586]: pgmap v586: 129 pgs: 129 active+clean; 4.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:31:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:53 smithi089 ceph-mon[30933]: pgmap v586: 129 pgs: 129 active+clean; 4.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:31:54.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.719+0000 7f786a28d640 1 -- 172.21.15.89:0/2066435786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78641078c0 msgr2=0x7f7864109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.719+0000 7f786a28d640 1 --2- 172.21.15.89:0/2066435786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78641078c0 0x7f7864109cb0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f785c009ca0 tx=0x7f785c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.719+0000 7f786a28d640 1 -- 172.21.15.89:0/2066435786 shutdown_connections 2024-09-15T23:31:54.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.719+0000 7f786a28d640 1 --2- 172.21.15.89:0/2066435786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f78641078c0 0x7f7864109cb0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.719+0000 7f786a28d640 1 --2- 172.21.15.89:0/2066435786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864104f90 0x7f7864107380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.719+0000 7f786a28d640 1 --2- 172.21.15.89:0/2066435786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7864102660 0x7f7864104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.719+0000 7f786a28d640 1 -- 172.21.15.89:0/2066435786 >> 172.21.15.89:0/2066435786 conn(0x7f78640fc1e0 msgr2=0x7f78640fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.719+0000 7f786a28d640 1 -- 172.21.15.89:0/2066435786 shutdown_connections 2024-09-15T23:31:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.719+0000 7f786a28d640 1 -- 172.21.15.89:0/2066435786 wait complete. 2024-09-15T23:31:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.720+0000 7f786a28d640 1 Processor -- start 2024-09-15T23:31:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.720+0000 7f786a28d640 1 -- start start 2024-09-15T23:31:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.720+0000 7f786a28d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864102660 0x7f7864101fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.720+0000 7f786a28d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7864104f90 0x7f7864100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.720+0000 7f786a28d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78641078c0 0x7f7864100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.720+0000 7f786a28d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f786410d850 con 0x7f7864104f90 2024-09-15T23:31:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.720+0000 7f786a28d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f786410d6d0 con 0x7f7864102660 2024-09-15T23:31:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.720+0000 7f786a28d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f786410d9d0 con 0x7f78641078c0 2024-09-15T23:31:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78637fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7864104f90 0x7f7864100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78637fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7864104f90 0x7f7864100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58072/0 (socket says 172.21.15.89:58072) 2024-09-15T23:31:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78637fe640 1 -- 172.21.15.89:0/734965115 learned_addr learned my addr 172.21.15.89:0/734965115 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f7863fff640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864102660 0x7f7864101fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f786882a640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78641078c0 0x7f7864100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78637fe640 1 -- 172.21.15.89:0/734965115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78641078c0 msgr2=0x7f7864100b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78637fe640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78641078c0 0x7f7864100b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78637fe640 1 -- 172.21.15.89:0/734965115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864102660 msgr2=0x7f7864101fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78637fe640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864102660 0x7f7864101fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78637fe640 1 -- 172.21.15.89:0/734965115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78641013e0 con 0x7f7864104f90 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f786882a640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78641078c0 0x7f7864100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78637fe640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7864104f90 0x7f7864100610 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f785800d770 tx=0x7f785800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78617fa640 1 -- 172.21.15.89:0/734965115 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858017070 con 0x7f7864104f90 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78617fa640 1 -- 172.21.15.89:0/734965115 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7858009c30 con 0x7f7864104f90 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f78617fa640 1 -- 172.21.15.89:0/734965115 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785800f620 con 0x7f7864104f90 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f786a28d640 1 -- 172.21.15.89:0/734965115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78641a1f10 con 0x7f7864104f90 2024-09-15T23:31:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.721+0000 7f786a28d640 1 -- 172.21.15.89:0/734965115 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78641a2370 con 0x7f7864104f90 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f78617fa640 1 -- 172.21.15.89:0/734965115 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f785801a070 con 0x7f7864104f90 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f786d02b740 1 Processor -- start 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f786d02b740 1 -- start start 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f786d02b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b349a0 0x556ba8b34da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f786d02b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b352e0 0x556ba8b39da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f78617fa640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f784c0a32f0 0x7f784c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f786d02b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b3c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f786d02b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556ba8a982f0 con 0x556ba8b3a370 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f786d02b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ba8a40270 con 0x556ba8b349a0 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f786d02b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556ba8a942b0 con 0x556ba8b352e0 2024-09-15T23:31:54.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f78637fe640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f784c0a32f0 0x7f784c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f786882a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b3c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.724+0000 7f78617fa640 1 -- 172.21.15.89:0/734965115 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f78580cf7b0 con 0x7f7864104f90 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f786882a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b3c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58086/0 (socket says 172.21.15.89:58086) 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f786882a640 1 -- 172.21.15.89:0/2964031737 learned_addr learned my addr 172.21.15.89:0/2964031737 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f7863fff640 1 --2- 172.21.15.89:0/2964031737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b352e0 0x556ba8b39da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f786882a640 1 --2- 172.21.15.89:0/2964031737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b349a0 0x556ba8b34da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f786882a640 1 -- 172.21.15.89:0/2964031737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b352e0 msgr2=0x556ba8b39da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f786882a640 1 --2- 172.21.15.89:0/2964031737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b352e0 0x556ba8b39da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f786882a640 1 -- 172.21.15.89:0/2964031737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b349a0 msgr2=0x556ba8b34da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f786882a640 1 --2- 172.21.15.89:0/2964031737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b349a0 0x556ba8b34da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f786882a640 1 -- 172.21.15.89:0/2964031737 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ba8a3f340 con 0x556ba8b3a370 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f78637fe640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f784c0a32f0 0x7f784c0a57b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f785809b6e0 tx=0x7f7858086040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f786882a640 1 --2- 172.21.15.89:0/2964031737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b3c830 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f785c009a20 tx=0x7f785c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafb468701bc3499 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f784a7fc640 1 -- 172.21.15.89:0/2964031737 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c031450 con 0x556ba8b3a370 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f784a7fc640 1 -- 172.21.15.89:0/2964031737 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785c036050 con 0x556ba8b3a370 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.725+0000 7f784a7fc640 1 -- 172.21.15.89:0/2964031737 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c0044a0 con 0x556ba8b3a370 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.726+0000 7f786d02b740 1 -- 172.21.15.89:0/2964031737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 msgr2=0x556ba8b3c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.726+0000 7f786d02b740 1 --2- 172.21.15.89:0/2964031737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b3c830 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f785c009a20 tx=0x7f785c005fd0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.726+0000 7f786d02b740 1 -- 172.21.15.89:0/2964031737 shutdown_connections 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.726+0000 7f786d02b740 1 --2- 172.21.15.89:0/2964031737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b3c830 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.726+0000 7f786d02b740 1 --2- 172.21.15.89:0/2964031737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b352e0 0x556ba8b39da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.726+0000 7f786d02b740 1 --2- 172.21.15.89:0/2964031737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b349a0 0x556ba8b34da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.726+0000 7f786d02b740 1 -- 172.21.15.89:0/2964031737 >> 172.21.15.89:0/2964031737 conn(0x556ba8b30010 msgr2=0x556ba8b30410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.726+0000 7f786d02b740 1 -- 172.21.15.89:0/2964031737 shutdown_connections 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.726+0000 7f786d02b740 1 -- 172.21.15.89:0/2964031737 wait complete. 2024-09-15T23:31:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.726+0000 7f786d02b740 1 -- start start 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786d02b740 1 Processor -- start 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786d02b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b3eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786d02b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b352e0 0x556ba8b3eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786d02b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b3a370 0x556ba8b46030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786d02b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556ba8b3d160 con 0x556ba8b349a0 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786d02b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ba8a21290 con 0x556ba8b352e0 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786d02b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556ba8b3d2e0 con 0x556ba8b3a370 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786d02b740 10 client.0 _ll_register_callbacks cb 0x556ba8a52140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786882a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b3eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786882a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b3eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58094/0 (socket says 172.21.15.89:58094) 2024-09-15T23:31:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786882a640 1 -- 172.21.15.89:0/3312501775 learned_addr learned my addr 172.21.15.89:0/3312501775 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f7863fff640 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b352e0 0x556ba8b3eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786882a640 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b3a370 0x556ba8b46030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786882a640 1 -- 172.21.15.89:0/3312501775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b3a370 msgr2=0x556ba8b46030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786882a640 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b3a370 0x556ba8b46030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786882a640 1 -- 172.21.15.89:0/3312501775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b352e0 msgr2=0x556ba8b3eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786882a640 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b352e0 0x556ba8b3eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.727+0000 7f786882a640 1 -- 172.21.15.89:0/3312501775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ba87ab210 con 0x556ba8b349a0 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f786882a640 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b3eab0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f785c02f4b0 tx=0x7f785c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b34a7320ec8b113 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f78437fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f78437fe640 1 -- 172.21.15.89:0/3312501775 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c057060 con 0x556ba8b349a0 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f78437fe640 1 -- 172.21.15.89:0/3312501775 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f785c003d80 con 0x556ba8b349a0 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f786d02b740 1 -- 172.21.15.89:0/3312501775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f785c036050 con 0x556ba8b349a0 2024-09-15T23:31:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f78437fe640 1 -- 172.21.15.89:0/3312501775 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f785c05e050 con 0x556ba8b349a0 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f786d02b740 10 client.18027 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f786d02b740 1 -- 172.21.15.89:0/3312501775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556ba8877620 con 0x556ba8b349a0 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f78437fe640 1 -- 172.21.15.89:0/3312501775 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785c036050 con 0x556ba8b349a0 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f78437fe640 1 -- 172.21.15.89:0/3312501775 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f785c02fc80 con 0x556ba8b349a0 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f786d02b740 10 client.18027 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f786d02b740 10 client.18027 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f786d02b740 20 client.18027 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f786d02b740 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ba8b4a420 0x556ba8b4c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f786d02b740 4 client.18027 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.728+0000 7f786d02b740 1 -- 172.21.15.89:0/3312501775 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ba8aa5a20 con 0x556ba8b4a420 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.729+0000 7f7863fff640 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ba8b4a420 0x556ba8b4c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.729+0000 7f7863fff640 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ba8b4a420 0x556ba8b4c810 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47fb1a9ecd361dbf server_cookie=ae0f0f0c05103d33 in_seq=0 out_seq=0 2024-09-15T23:31:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.729+0000 7f78437fe640 10 client.18027 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f78437fe640 1 -- 172.21.15.89:0/3312501775 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556ba8aa5a20 con 0x556ba8b4a420 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f78437fe640 10 client.18027 handle_command_reply: tid=0 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 client.18027 shutdown 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 -- 172.21.15.89:0/3312501775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 msgr2=0x556ba8b3eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b3eab0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f785c02f4b0 tx=0x7f785c005dc0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 -- 172.21.15.89:0/3312501775 shutdown_connections 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ba8b4a420 0x556ba8b4c810 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b3a370 0x556ba8b46030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b352e0 0x556ba8b3eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f78437fe640 0 client.18027 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 --2- 172.21.15.89:0/3312501775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b3eab0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 -- 172.21.15.89:0/3312501775 >> 172.21.15.89:0/3312501775 conn(0x556ba8b30010 msgr2=0x556ba8b31820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 -- 172.21.15.89:0/3312501775 shutdown_connections 2024-09-15T23:31:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 1 -- 172.21.15.89:0/3312501775 wait complete. 2024-09-15T23:31:54.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.730+0000 7f786d02b740 20 client.18027 trim_cache size 0 max 16384 2024-09-15T23:31:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786d02b740 1 Processor -- start 2024-09-15T23:31:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786d02b740 1 -- start start 2024-09-15T23:31:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786d02b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b349a0 0x556ba8b3f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786d02b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b352e0 0x556ba8b45f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786d02b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b2f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786d02b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f78641ac9e0 con 0x556ba8b3a370 2024-09-15T23:31:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786d02b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78641ae520 con 0x556ba8b352e0 2024-09-15T23:31:54.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786d02b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f78641ad0b0 con 0x556ba8b349a0 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786882a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b2f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786882a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b2f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58106/0 (socket says 172.21.15.89:58106) 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786882a640 1 -- 172.21.15.89:0/1891523584 learned_addr learned my addr 172.21.15.89:0/1891523584 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786882a640 1 --2- 172.21.15.89:0/1891523584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b349a0 0x556ba8b3f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786882a640 1 -- 172.21.15.89:0/1891523584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b349a0 msgr2=0x556ba8b3f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f7863fff640 1 --2- 172.21.15.89:0/1891523584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b352e0 0x556ba8b45f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786882a640 1 --2- 172.21.15.89:0/1891523584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b349a0 0x556ba8b3f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786882a640 1 -- 172.21.15.89:0/1891523584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b352e0 msgr2=0x556ba8b45f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786882a640 1 --2- 172.21.15.89:0/1891523584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b352e0 0x556ba8b45f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786882a640 1 -- 172.21.15.89:0/1891523584 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ba8b4cb80 con 0x556ba8b3a370 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f786882a640 1 --2- 172.21.15.89:0/1891523584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b2f0b0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f785c0093e0 tx=0x7f785c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97015a03d7767798 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.745+0000 7f7849ffb640 1 -- 172.21.15.89:0/1891523584 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c04d070 con 0x556ba8b3a370 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f7849ffb640 1 -- 172.21.15.89:0/1891523584 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785c036040 con 0x556ba8b3a370 2024-09-15T23:31:54.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f7849ffb640 1 -- 172.21.15.89:0/1891523584 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c004670 con 0x556ba8b3a370 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f786d02b740 1 -- 172.21.15.89:0/1891523584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 msgr2=0x556ba8b2f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f786d02b740 1 --2- 172.21.15.89:0/1891523584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b2f0b0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f785c0093e0 tx=0x7f785c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f786d02b740 1 -- 172.21.15.89:0/1891523584 shutdown_connections 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f786d02b740 1 --2- 172.21.15.89:0/1891523584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b3a370 0x556ba8b2f0b0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f786d02b740 1 --2- 172.21.15.89:0/1891523584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b352e0 0x556ba8b45f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f786d02b740 1 --2- 172.21.15.89:0/1891523584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b349a0 0x556ba8b3f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f786d02b740 1 -- 172.21.15.89:0/1891523584 >> 172.21.15.89:0/1891523584 conn(0x556ba8b30010 msgr2=0x556ba8b2e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f786d02b740 1 -- 172.21.15.89:0/1891523584 shutdown_connections 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f786d02b740 1 -- 172.21.15.89:0/1891523584 wait complete. 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.746+0000 7f786d02b740 1 -- start start 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786d02b740 1 Processor -- start 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786d02b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b43370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786d02b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b352e0 0x556ba8b438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786d02b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b3a370 0x556ba8b448c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786d02b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f78641ade50 con 0x556ba8b349a0 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786d02b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f78641ad780 con 0x556ba8b3a370 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786d02b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f78641aa9c0 con 0x556ba8b352e0 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786d02b740 10 client.0 _ll_register_callbacks cb 0x7f7854004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786882a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b43370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786882a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b43370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:58120/0 (socket says 172.21.15.89:58120) 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786882a640 1 -- 172.21.15.89:0/1609202915 learned_addr learned my addr 172.21.15.89:0/1609202915 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f7863fff640 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b352e0 0x556ba8b438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786882a640 1 -- 172.21.15.89:0/1609202915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b352e0 msgr2=0x556ba8b438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786882a640 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b352e0 0x556ba8b438b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786882a640 1 -- 172.21.15.89:0/1609202915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b3a370 msgr2=0x556ba8b448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786882a640 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b3a370 0x556ba8b448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786882a640 1 -- 172.21.15.89:0/1609202915 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ba8b44ed0 con 0x556ba8b349a0 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f786882a640 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b43370 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f785c0318c0 tx=0x7f785c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef61ab102d0116a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.747+0000 7f78437fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f78437fe640 1 -- 172.21.15.89:0/1609202915 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c04d040 con 0x556ba8b349a0 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f78437fe640 1 -- 172.21.15.89:0/1609202915 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f785c004b40 con 0x556ba8b349a0 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f786d02b740 1 -- 172.21.15.89:0/1609202915 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f785c036040 con 0x556ba8b349a0 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f78437fe640 1 -- 172.21.15.89:0/1609202915 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f785c070050 con 0x556ba8b349a0 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f786d02b740 10 client.18033 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f786d02b740 1 -- 172.21.15.89:0/1609202915 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556ba8b461e0 con 0x556ba8b349a0 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f78437fe640 1 -- 172.21.15.89:0/1609202915 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785c036040 con 0x556ba8b349a0 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f78437fe640 1 -- 172.21.15.89:0/1609202915 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f785c02f610 con 0x556ba8b349a0 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f786d02b740 10 client.18033 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f786d02b740 10 client.18033 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:54.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f786d02b740 20 client.18033 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f786d02b740 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ba8bb5750 0x556ba8bb7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f786d02b740 4 client.18033 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:31:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f786d02b740 1 -- 172.21.15.89:0/1609202915 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556ba8a9fd70 con 0x556ba8bb5750 2024-09-15T23:31:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.748+0000 7f7863fff640 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ba8bb5750 0x556ba8bb7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.749+0000 7f78437fe640 10 client.18033 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:54.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.749+0000 7f7863fff640 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ba8bb5750 0x556ba8bb7b70 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae9e1c7362461431 server_cookie=6635b16c05513e4e in_seq=0 out_seq=0 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f78437fe640 1 -- 172.21.15.89:0/1609202915 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556ba8a9fd70 con 0x556ba8bb5750 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f78437fe640 10 client.18033 handle_command_reply: tid=0 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 client.18033 shutdown 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 -- 172.21.15.89:0/1609202915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 msgr2=0x556ba8b43370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b43370 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f785c0318c0 tx=0x7f785c009a10 comp rx=0 tx=0).stop 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 -- 172.21.15.89:0/1609202915 shutdown_connections 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ba8bb5750 0x556ba8bb7b70 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ba8b3a370 0x556ba8b448c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f78437fe640 0 client.18033 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ba8b352e0 0x556ba8b438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 --2- 172.21.15.89:0/1609202915 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ba8b349a0 0x556ba8b43370 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 -- 172.21.15.89:0/1609202915 >> 172.21.15.89:0/1609202915 conn(0x556ba8b30010 msgr2=0x556ba8b2f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 -- 172.21.15.89:0/1609202915 shutdown_connections 2024-09-15T23:31:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 1 -- 172.21.15.89:0/1609202915 wait complete. 2024-09-15T23:31:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.765+0000 7f786d02b740 20 client.18033 trim_cache size 0 max 16384 2024-09-15T23:31:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:31:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:31:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:31:54.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:31:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 -- 172.21.15.89:0/734965115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f784c0a32f0 msgr2=0x7f784c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f784c0a32f0 0x7f784c0a57b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f785809b6e0 tx=0x7f7858086040 comp rx=0 tx=0).stop 2024-09-15T23:31:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 -- 172.21.15.89:0/734965115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7864104f90 msgr2=0x7f7864100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7864104f90 0x7f7864100610 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f785800d770 tx=0x7f785800dc30 comp rx=0 tx=0).stop 2024-09-15T23:31:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 -- 172.21.15.89:0/734965115 shutdown_connections 2024-09-15T23:31:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f784c0a32f0 0x7f784c0a57b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78641078c0 0x7f7864100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7864104f90 0x7f7864100610 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 --2- 172.21.15.89:0/734965115 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7864102660 0x7f7864101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 -- 172.21.15.89:0/734965115 >> 172.21.15.89:0/734965115 conn(0x7f78640fc1e0 msgr2=0x7f7864102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 -- 172.21.15.89:0/734965115 shutdown_connections 2024-09-15T23:31:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:31:54.766+0000 7f786a28d640 1 -- 172.21.15.89:0/734965115 wait complete. 2024-09-15T23:31:55.109 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.7 2024-09-15T23:31:55.109 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:31:55.363 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:55 smithi120 ceph-mon[35250]: pgmap v587: 129 pgs: 129 active+clean; 4.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:31:55.483 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:55 smithi089 ceph-mon[30933]: pgmap v587: 129 pgs: 129 active+clean; 4.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:31:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:55 smithi174 ceph-mon[35586]: pgmap v587: 129 pgs: 129 active+clean; 4.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:31:55.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.827+0000 7fcaab423640 1 -- 172.21.15.89:0/2764015882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaa41082a0 msgr2=0x7fcaa410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.827+0000 7fcaab423640 1 --2- 172.21.15.89:0/2764015882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaa41082a0 0x7fcaa410ec40 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fca9c009930 tx=0x7fca9c02a420 comp rx=0 tx=0).stop 2024-09-15T23:31:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.827+0000 7fcaab423640 1 -- 172.21.15.89:0/2764015882 shutdown_connections 2024-09-15T23:31:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.827+0000 7fcaab423640 1 --2- 172.21.15.89:0/2764015882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaa41082a0 0x7fcaa410ec40 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.827+0000 7fcaab423640 1 --2- 172.21.15.89:0/2764015882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa41036e0 0x7fcaa4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.827+0000 7fcaab423640 1 --2- 172.21.15.89:0/2764015882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcaa4102d30 0x7fcaa4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.827+0000 7fcaab423640 1 -- 172.21.15.89:0/2764015882 >> 172.21.15.89:0/2764015882 conn(0x7fcaa40fc7d0 msgr2=0x7fcaa40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.827+0000 7fcaab423640 1 -- 172.21.15.89:0/2764015882 shutdown_connections 2024-09-15T23:31:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.827+0000 7fcaab423640 1 -- 172.21.15.89:0/2764015882 wait complete. 2024-09-15T23:31:55.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.827+0000 7fcaab423640 1 Processor -- start 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaab423640 1 -- start start 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaab423640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa4102d30 0x7fcaa4198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaab423640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaa41036e0 0x7fcaa4198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaab423640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcaa41082a0 0x7fcaa419ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaab423640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa41120a0 con 0x7fcaa41036e0 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaab423640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa4111f20 con 0x7fcaa4102d30 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaab423640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa4112220 con 0x7fcaa41082a0 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa89be640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaa41036e0 0x7fcaa4198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa89be640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaa41036e0 0x7fcaa4198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54824/0 (socket says 172.21.15.89:54824) 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa89be640 1 -- 172.21.15.89:0/1105408017 learned_addr learned my addr 172.21.15.89:0/1105408017 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa91bf640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa4102d30 0x7fcaa4198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa99c0640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcaa41082a0 0x7fcaa419ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa89be640 1 -- 172.21.15.89:0/1105408017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcaa41082a0 msgr2=0x7fcaa419ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa89be640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcaa41082a0 0x7fcaa419ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa89be640 1 -- 172.21.15.89:0/1105408017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa4102d30 msgr2=0x7fcaa4198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa89be640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa4102d30 0x7fcaa4198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa89be640 1 -- 172.21.15.89:0/1105408017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa419d4e0 con 0x7fcaa41036e0 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa99c0640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcaa41082a0 0x7fcaa419ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa89be640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaa41036e0 0x7fcaa4198a70 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fca9800c970 tx=0x7fca9800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.828+0000 7fcaa27fc640 1 -- 172.21.15.89:0/1105408017 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca98007bb0 con 0x7fcaa41036e0 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.829+0000 7fcaa27fc640 1 -- 172.21.15.89:0/1105408017 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca98007d50 con 0x7fcaa41036e0 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.829+0000 7fcaab423640 1 -- 172.21.15.89:0/1105408017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa419d7d0 con 0x7fcaa41036e0 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.829+0000 7fcaa27fc640 1 -- 172.21.15.89:0/1105408017 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9800f450 con 0x7fcaa41036e0 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.829+0000 7fcaab423640 1 -- 172.21.15.89:0/1105408017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa41a50b0 con 0x7fcaa41036e0 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.830+0000 7fcaab423640 1 -- 172.21.15.89:0/1105408017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaa4103110 con 0x7fcaa41036e0 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.831+0000 7fcaa27fc640 1 -- 172.21.15.89:0/1105408017 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fca98004f60 con 0x7fcaa41036e0 2024-09-15T23:31:55.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.831+0000 7fcaa27fc640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca8c0a32c0 0x7fca8c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.832+0000 7fcaa91bf640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca8c0a32c0 0x7fca8c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.832+0000 7fcaa27fc640 1 -- 172.21.15.89:0/1105408017 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fca980cf920 con 0x7fcaa41036e0 2024-09-15T23:31:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.832+0000 7fcaa91bf640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca8c0a32c0 0x7fca8c0a5780 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fca94005c40 tx=0x7fca9400a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:55.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:55.836+0000 7fcaa27fc640 1 -- 172.21.15.89:0/1105408017 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fca98014030 con 0x7fcaa41036e0 2024-09-15T23:31:56.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.038+0000 7fcaab423640 1 -- 172.21.15.89:0/1105408017 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcaa410a2e0 con 0x7fcaa41036e0 2024-09-15T23:31:56.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.038+0000 7fcaa27fc640 1 -- 172.21.15.89:0/1105408017 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fca9809c250 con 0x7fcaa41036e0 2024-09-15T23:31:56.040 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:31:56.040 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:31:56.040 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:31:56.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.040+0000 7fcaab423640 1 -- 172.21.15.89:0/1105408017 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca8c0a32c0 msgr2=0x7fca8c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:56.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.040+0000 7fcaab423640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca8c0a32c0 0x7fca8c0a5780 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fca94005c40 tx=0x7fca9400a590 comp rx=0 tx=0).stop 2024-09-15T23:31:56.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.040+0000 7fcaab423640 1 -- 172.21.15.89:0/1105408017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaa41036e0 msgr2=0x7fcaa4198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:56.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.040+0000 7fcaab423640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaa41036e0 0x7fcaa4198a70 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fca9800c970 tx=0x7fca9800ce30 comp rx=0 tx=0).stop 2024-09-15T23:31:56.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.041+0000 7fcaab423640 1 -- 172.21.15.89:0/1105408017 shutdown_connections 2024-09-15T23:31:56.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.041+0000 7fcaab423640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca8c0a32c0 0x7fca8c0a5780 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:56.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.041+0000 7fcaab423640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcaa41082a0 0x7fcaa419ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:56.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.041+0000 7fcaab423640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaa41036e0 0x7fcaa4198a70 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:56.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.041+0000 7fcaab423640 1 --2- 172.21.15.89:0/1105408017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcaa4102d30 0x7fcaa4198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:56.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.041+0000 7fcaab423640 1 -- 172.21.15.89:0/1105408017 >> 172.21.15.89:0/1105408017 conn(0x7fcaa40fc7d0 msgr2=0x7fcaa4105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:56.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.041+0000 7fcaab423640 1 -- 172.21.15.89:0/1105408017 shutdown_connections 2024-09-15T23:31:56.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:56.041+0000 7fcaab423640 1 -- 172.21.15.89:0/1105408017 wait complete. 2024-09-15T23:31:56.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:56 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1105408017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:56.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:56 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1105408017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:56.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:56 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1105408017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:56.823 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 143.7 --await-for 45.0' 2024-09-15T23:31:56.823 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 143.7 --await-for 45.0 2024-09-15T23:31:57.080 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:57 smithi174 ceph-mon[35586]: pgmap v588: 129 pgs: 129 active+clean; 4.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 20 op/s 2024-09-15T23:31:57.524 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:57 smithi089 ceph-mon[30933]: pgmap v588: 129 pgs: 129 active+clean; 4.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 20 op/s 2024-09-15T23:31:57.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.614+0000 7f5914c65640 1 -- 172.21.15.89:0/2033579314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5910106a80 msgr2=0x7f5910106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.614+0000 7f5914c65640 1 --2- 172.21.15.89:0/2033579314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5910106a80 0x7f5910106e60 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f5900009930 tx=0x7f590002a420 comp rx=0 tx=0).stop 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.614+0000 7f5914c65640 1 -- 172.21.15.89:0/2033579314 shutdown_connections 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.614+0000 7f5914c65640 1 --2- 172.21.15.89:0/2033579314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5910101420 0x7f591010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.614+0000 7f5914c65640 1 --2- 172.21.15.89:0/2033579314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5910100a80 0x7f5910100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.614+0000 7f5914c65640 1 --2- 172.21.15.89:0/2033579314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5910106a80 0x7f5910106e60 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.614+0000 7f5914c65640 1 -- 172.21.15.89:0/2033579314 >> 172.21.15.89:0/2033579314 conn(0x7f59100fc7a0 msgr2=0x7f59100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.614+0000 7f5914c65640 1 -- 172.21.15.89:0/2033579314 shutdown_connections 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.614+0000 7f5914c65640 1 -- 172.21.15.89:0/2033579314 wait complete. 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f5914c65640 1 Processor -- start 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f5914c65640 1 -- start start 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f5914c65640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5910100a80 0x7f59100fceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f5914c65640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5910101420 0x7f59100fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f5914c65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5910106a80 0x7f59100fd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f5914c65640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5910110f20 con 0x7f5910106a80 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f5914c65640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5910110da0 con 0x7f5910101420 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f5914c65640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59101110a0 con 0x7f5910100a80 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f590ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5910106a80 0x7f59100fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f590ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5910106a80 0x7f59100fd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54852/0 (socket says 172.21.15.89:54852) 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f590ed9d640 1 -- 172.21.15.89:0/446829987 learned_addr learned my addr 172.21.15.89:0/446829987 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f590e59c640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5910100a80 0x7f59100fceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.615+0000 7f590dd9b640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5910101420 0x7f59100fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f590ed9d640 1 -- 172.21.15.89:0/446829987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5910100a80 msgr2=0x7f59100fceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f590ed9d640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5910100a80 0x7f59100fceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f590ed9d640 1 -- 172.21.15.89:0/446829987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5910101420 msgr2=0x7f59100fd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f590ed9d640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5910101420 0x7f59100fd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f590ed9d640 1 -- 172.21.15.89:0/446829987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59101a6a50 con 0x7f5910106a80 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f590dd9b640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5910101420 0x7f59100fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:57.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f590ed9d640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5910106a80 0x7f59100fd930 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f590800b970 tx=0x7f590800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f58ff7fe640 1 -- 172.21.15.89:0/446829987 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5908019070 con 0x7f5910106a80 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f5914c65640 1 -- 172.21.15.89:0/446829987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59101a6d40 con 0x7f5910106a80 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f58ff7fe640 1 -- 172.21.15.89:0/446829987 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f590800cb40 con 0x7f5910106a80 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f58ff7fe640 1 -- 172.21.15.89:0/446829987 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5908007610 con 0x7f5910106a80 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.616+0000 7f5914c65640 1 -- 172.21.15.89:0/446829987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59101a7270 con 0x7f5910106a80 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.618+0000 7f58ff7fe640 1 -- 172.21.15.89:0/446829987 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f590801c070 con 0x7f5910106a80 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.618+0000 7f591703e740 1 Processor -- start 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f591703e740 1 -- start start 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f591703e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91e760 0x5652fc91eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f591703e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc929400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f591703e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc92bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f591703e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5652fc8842a0 con 0x5652fc91f080 2024-09-15T23:31:57.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f591703e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5652fc82bb10 con 0x5652fc91e760 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f591703e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5652fc880260 con 0x5652fc929940 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f590e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc929400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f590e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc929400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54860/0 (socket says 172.21.15.89:54860) 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f590e59c640 1 -- 172.21.15.89:0/3199481139 learned_addr learned my addr 172.21.15.89:0/3199481139 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f590dd9b640 1 --2- 172.21.15.89:0/3199481139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91e760 0x5652fc91eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f58ff7fe640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58f80a3240 0x7f58f80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f590dd9b640 1 --2- 172.21.15.89:0/3199481139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc92bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f590e59c640 1 -- 172.21.15.89:0/3199481139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 msgr2=0x5652fc92bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f590e59c640 1 --2- 172.21.15.89:0/3199481139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc92bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f590e59c640 1 -- 172.21.15.89:0/3199481139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91e760 msgr2=0x5652fc91eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f590e59c640 1 --2- 172.21.15.89:0/3199481139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91e760 0x5652fc91eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.619+0000 7f590e59c640 1 -- 172.21.15.89:0/3199481139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652fc5f5b90 con 0x5652fc91f080 2024-09-15T23:31:57.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.620+0000 7f590ed9d640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58f80a3240 0x7f58f80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.620+0000 7f58ff7fe640 1 -- 172.21.15.89:0/446829987 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f59080cf380 con 0x7f5910106a80 2024-09-15T23:31:57.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.620+0000 7f590e59c640 1 --2- 172.21.15.89:0/3199481139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc929400 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f5900005ab0 tx=0x7f5900031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa50da8bbb5ed90 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.620+0000 7f58fcff9640 1 -- 172.21.15.89:0/3199481139 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590003f070 con 0x5652fc91f080 2024-09-15T23:31:57.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.620+0000 7f58fcff9640 1 -- 172.21.15.89:0/3199481139 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5900031a20 con 0x5652fc91f080 2024-09-15T23:31:57.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.620+0000 7f58fcff9640 1 -- 172.21.15.89:0/3199481139 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900035070 con 0x5652fc91f080 2024-09-15T23:31:57.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 -- 172.21.15.89:0/3199481139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 msgr2=0x5652fc929400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f590ed9d640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58f80a3240 0x7f58f80a5700 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f590809b4e0 tx=0x7f5908065f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 --2- 172.21.15.89:0/3199481139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc929400 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f5900005ab0 tx=0x7f5900031630 comp rx=0 tx=0).stop 2024-09-15T23:31:57.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 -- 172.21.15.89:0/3199481139 shutdown_connections 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 --2- 172.21.15.89:0/3199481139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc92bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 --2- 172.21.15.89:0/3199481139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc929400 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 --2- 172.21.15.89:0/3199481139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91e760 0x5652fc91eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 -- 172.21.15.89:0/3199481139 >> 172.21.15.89:0/3199481139 conn(0x5652fc91acf0 msgr2=0x5652fc91b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 -- 172.21.15.89:0/3199481139 shutdown_connections 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 -- 172.21.15.89:0/3199481139 wait complete. 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 -- start start 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 Processor -- start 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc926750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91f080 0x5652fc926c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc9271d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5652fc92c510 con 0x5652fc91e760 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5652fc80c6e0 con 0x5652fc91f080 2024-09-15T23:31:57.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5652fc92c690 con 0x5652fc929940 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.621+0000 7f591703e740 10 client.0 _ll_register_callbacks cb 0x5652fc830a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc926750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc926750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54872/0 (socket says 172.21.15.89:54872) 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590dd9b640 1 -- 172.21.15.89:0/1196034699 learned_addr learned my addr 172.21.15.89:0/1196034699 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590e59c640 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91f080 0x5652fc926c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590dd9b640 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc9271d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590dd9b640 1 -- 172.21.15.89:0/1196034699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 msgr2=0x5652fc9271d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590dd9b640 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc9271d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590dd9b640 1 -- 172.21.15.89:0/1196034699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91f080 msgr2=0x5652fc926c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590dd9b640 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91f080 0x5652fc926c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590dd9b640 1 -- 172.21.15.89:0/1196034699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652fc9359b0 con 0x5652fc91e760 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f590dd9b640 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc926750 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f59040197f0 tx=0x7f59040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ff99d4499c9741 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f58f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f58f5ffb640 1 -- 172.21.15.89:0/1196034699 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590401f060 con 0x5652fc91e760 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f58f5ffb640 1 -- 172.21.15.89:0/1196034699 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5904014a60 con 0x5652fc91e760 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f591703e740 1 -- 172.21.15.89:0/1196034699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5900031a20 con 0x5652fc91e760 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f58f5ffb640 1 -- 172.21.15.89:0/1196034699 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5904026050 con 0x5652fc91e760 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f591703e740 10 client.18051 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.622+0000 7f591703e740 1 -- 172.21.15.89:0/1196034699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5652fc938340 con 0x5652fc91e760 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f58f5ffb640 1 -- 172.21.15.89:0/1196034699 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5900031a20 con 0x5652fc91e760 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f58f5ffb640 1 -- 172.21.15.89:0/1196034699 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5904015040 con 0x5652fc91e760 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f591703e740 10 client.18051 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f591703e740 10 client.18051 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f591703e740 20 client.18051 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f591703e740 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5652fc939800 0x5652fc93bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f591703e740 4 client.18051 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:57.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f591703e740 1 -- 172.21.15.89:0/1196034699 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652fc8919d0 con 0x5652fc939800 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f590e59c640 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5652fc939800 0x5652fc93bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f58f5ffb640 10 client.18051 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.623+0000 7f590e59c640 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5652fc939800 0x5652fc93bbf0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78ca51d4f25d6ab7 server_cookie=c9f21480c07b0e47 in_seq=0 out_seq=0 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f58f5ffb640 1 -- 172.21.15.89:0/1196034699 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5652fc8919d0 con 0x5652fc939800 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f58f5ffb640 10 client.18051 handle_command_reply: tid=0 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f591703e740 1 client.18051 shutdown 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f591703e740 1 -- 172.21.15.89:0/1196034699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 msgr2=0x5652fc926750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f591703e740 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc926750 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f59040197f0 tx=0x7f59040046c0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f591703e740 1 -- 172.21.15.89:0/1196034699 shutdown_connections 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f591703e740 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5652fc939800 0x5652fc93bbf0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f591703e740 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc9271d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f58f5ffb640 0 client.18051 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f591703e740 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91f080 0x5652fc926c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f591703e740 1 --2- 172.21.15.89:0/1196034699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc926750 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f591703e740 1 -- 172.21.15.89:0/1196034699 >> 172.21.15.89:0/1196034699 conn(0x5652fc91acf0 msgr2=0x5652fc920a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.624+0000 7f591703e740 1 -- 172.21.15.89:0/1196034699 shutdown_connections 2024-09-15T23:31:57.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.625+0000 7f591703e740 1 -- 172.21.15.89:0/1196034699 wait complete. 2024-09-15T23:31:57.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.625+0000 7f591703e740 20 client.18051 trim_cache size 0 max 16384 2024-09-15T23:31:57.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.642+0000 7f591703e740 1 Processor -- start 2024-09-15T23:31:57.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.642+0000 7f591703e740 1 -- start start 2024-09-15T23:31:57.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f591703e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc91e760 0x5652fc941480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f591703e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc9419c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f591703e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc929940 0x5652fc9239b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f591703e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f59101b1130 con 0x5652fc91f080 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f591703e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f59101b2c70 con 0x5652fc929940 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f591703e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59101b1800 con 0x5652fc91e760 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f590e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc9419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f590e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc9419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54874/0 (socket says 172.21.15.89:54874) 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f590e59c640 1 -- 172.21.15.89:0/1252249130 learned_addr learned my addr 172.21.15.89:0/1252249130 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f590dd9b640 1 --2- 172.21.15.89:0/1252249130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc91e760 0x5652fc941480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f590e59c640 1 -- 172.21.15.89:0/1252249130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc91e760 msgr2=0x5652fc941480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f590e59c640 1 --2- 172.21.15.89:0/1252249130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc91e760 0x5652fc941480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f590e59c640 1 -- 172.21.15.89:0/1252249130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc929940 msgr2=0x5652fc9239b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f590e59c640 1 --2- 172.21.15.89:0/1252249130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc929940 0x5652fc9239b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f590e59c640 1 -- 172.21.15.89:0/1252249130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652fc91eeb0 con 0x5652fc91f080 2024-09-15T23:31:57.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f590e59c640 1 --2- 172.21.15.89:0/1252249130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc9419c0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f590002a940 tx=0x7f590002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787b3dabe4309b8c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f58f7fff640 1 -- 172.21.15.89:0/1252249130 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5900035050 con 0x5652fc91f080 2024-09-15T23:31:57.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f58f7fff640 1 -- 172.21.15.89:0/1252249130 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59000319c0 con 0x5652fc91f080 2024-09-15T23:31:57.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.643+0000 7f58f7fff640 1 -- 172.21.15.89:0/1252249130 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590003f040 con 0x5652fc91f080 2024-09-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 -- 172.21.15.89:0/1252249130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 msgr2=0x5652fc9419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 --2- 172.21.15.89:0/1252249130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc9419c0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f590002a940 tx=0x7f590002a420 comp rx=0 tx=0).stop 2024-09-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 -- 172.21.15.89:0/1252249130 shutdown_connections 2024-09-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 --2- 172.21.15.89:0/1252249130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc929940 0x5652fc9239b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 --2- 172.21.15.89:0/1252249130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91f080 0x5652fc9419c0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 --2- 172.21.15.89:0/1252249130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc91e760 0x5652fc941480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 -- 172.21.15.89:0/1252249130 >> 172.21.15.89:0/1252249130 conn(0x5652fc91acf0 msgr2=0x5652fc9287d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 -- 172.21.15.89:0/1252249130 shutdown_connections 2024-09-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 -- 172.21.15.89:0/1252249130 wait complete. 2024-09-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 -- start start 2024-09-15T23:31:57.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.644+0000 7f591703e740 1 Processor -- start 2024-09-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f591703e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc931920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f591703e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91f080 0x5652fc931e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f591703e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc92c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f591703e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f59101b25a0 con 0x5652fc91e760 2024-09-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f591703e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f59101b1ed0 con 0x5652fc91f080 2024-09-15T23:31:57.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f591703e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59101af110 con 0x5652fc929940 2024-09-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f591703e740 10 client.0 _ll_register_callbacks cb 0x7f5904019400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f590dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc931920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f590dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc931920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54880/0 (socket says 172.21.15.89:54880) 2024-09-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f590dd9b640 1 -- 172.21.15.89:0/2026345488 learned_addr learned my addr 172.21.15.89:0/2026345488 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f590dd9b640 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc92c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f590dd9b640 1 -- 172.21.15.89:0/2026345488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 msgr2=0x5652fc92c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f590dd9b640 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc92c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f590dd9b640 1 -- 172.21.15.89:0/2026345488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91f080 msgr2=0x5652fc931e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f590dd9b640 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91f080 0x5652fc931e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f590dd9b640 1 -- 172.21.15.89:0/2026345488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652fc9325a0 con 0x5652fc91e760 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f590dd9b640 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc931920 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f5904017eb0 tx=0x7f5904006ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8060069a3f5db21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f58f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f58f5ffb640 1 -- 172.21.15.89:0/2026345488 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59040163d0 con 0x5652fc91e760 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f58f5ffb640 1 -- 172.21.15.89:0/2026345488 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5904014a60 con 0x5652fc91e760 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.645+0000 7f591703e740 1 -- 172.21.15.89:0/2026345488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59000319c0 con 0x5652fc91e760 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f58f5ffb640 1 -- 172.21.15.89:0/2026345488 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5904014ca0 con 0x5652fc91e760 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f591703e740 10 client.18057 fetch_fsmap learned FSMap version 29 2024-09-15T23:31:57.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f591703e740 1 -- 172.21.15.89:0/2026345488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5652fc9a4ac0 con 0x5652fc91e760 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f58f5ffb640 1 -- 172.21.15.89:0/2026345488 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59000319c0 con 0x5652fc91e760 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f58f5ffb640 1 -- 172.21.15.89:0/2026345488 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5904017070 con 0x5652fc91e760 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f591703e740 10 client.18057 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f591703e740 10 client.18057 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f591703e740 20 client.18057 populate_metadata read hostname 'smithi089' 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f591703e740 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5652fc939800 0x5652fc9a72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f591703e740 4 client.18057 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.7, "await_for": 45.0}] 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f591703e740 1 -- 172.21.15.89:0/2026345488 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.7, "await_for": 45.0}) -- 0x5652fc88bd20 con 0x5652fc939800 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.646+0000 7f590e59c640 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5652fc939800 0x5652fc9a72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.647+0000 7f590e59c640 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5652fc939800 0x5652fc9a72f0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90af580d8cc0bc5b server_cookie=6802ef5891552c04 in_seq=0 out_seq=0 2024-09-15T23:31:57.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.647+0000 7f58f5ffb640 10 client.18057 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f58f5ffb640 1 -- 172.21.15.89:0/2026345488 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5652fc88bd20 con 0x5652fc939800 2024-09-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f58f5ffb640 10 client.18057 handle_command_reply: tid=0 2024-09-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 client.18057 shutdown 2024-09-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 -- 172.21.15.89:0/2026345488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 msgr2=0x5652fc931920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc931920 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f5904017eb0 tx=0x7f5904006ce0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 -- 172.21.15.89:0/2026345488 shutdown_connections 2024-09-15T23:31:57.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5652fc939800 0x5652fc9a72f0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652fc929940 0x5652fc92c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652fc91f080 0x5652fc931e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 --2- 172.21.15.89:0/2026345488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652fc91e760 0x5652fc931920 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 -- 172.21.15.89:0/2026345488 >> 172.21.15.89:0/2026345488 conn(0x5652fc91acf0 msgr2=0x5652fc919650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f58f5ffb640 0 client.18057 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 -- 172.21.15.89:0/2026345488 shutdown_connections 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 1 -- 172.21.15.89:0/2026345488 wait complete. 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.658+0000 7f591703e740 20 client.18057 trim_cache size 0 max 16384 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:31:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 26, 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "6514e407": { 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 26, 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 143.7, 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:31:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:31:57.663 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 -- 172.21.15.89:0/446829987 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58f80a3240 msgr2=0x7f58f80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58f80a3240 0x7f58f80a5700 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f590809b4e0 tx=0x7f5908065f90 comp rx=0 tx=0).stop 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 -- 172.21.15.89:0/446829987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5910106a80 msgr2=0x7f59100fd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5910106a80 0x7f59100fd930 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f590800b970 tx=0x7f590800be30 comp rx=0 tx=0).stop 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 -- 172.21.15.89:0/446829987 shutdown_connections 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58f80a3240 0x7f58f80a5700 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5910106a80 0x7f59100fd930 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5910101420 0x7f59100fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 --2- 172.21.15.89:0/446829987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5910100a80 0x7f59100fceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 -- 172.21.15.89:0/446829987 >> 172.21.15.89:0/446829987 conn(0x7f59100fc7a0 msgr2=0x7f591006aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 -- 172.21.15.89:0/446829987 shutdown_connections 2024-09-15T23:31:57.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:31:57.659+0000 7f5914c65640 1 -- 172.21.15.89:0/446829987 wait complete. 2024-09-15T23:31:57.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:57 smithi120 ceph-mon[35250]: pgmap v588: 129 pgs: 129 active+clean; 4.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 20 op/s 2024-09-15T23:31:58.384 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6514e407', quiesce took 3.3 seconds. Will release after: 20.4 2024-09-15T23:31:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:31:59 smithi120 ceph-mon[35250]: pgmap v589: 129 pgs: 129 active+clean; 4.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:31:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:31:59 smithi174 ceph-mon[35586]: pgmap v589: 129 pgs: 129 active+clean; 4.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:31:59.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:31:59 smithi089 ceph-mon[30933]: pgmap v589: 129 pgs: 129 active+clean; 4.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:32:00.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:32:00.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:32:00.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:32:01.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:01 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi174 to 176.0M 2024-09-15T23:32:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:01 smithi120 ceph-mon[35250]: Unable to set osd_memory_target on smithi174 to 184573849: error parsing value: Value '184573849' is below minimum 939524096 2024-09-15T23:32:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:32:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:32:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:32:01.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:01 smithi120 ceph-mon[35250]: pgmap v590: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-09-15T23:32:01.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:01 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi174 to 176.0M 2024-09-15T23:32:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:01 smithi174 ceph-mon[35586]: Unable to set osd_memory_target on smithi174 to 184573849: error parsing value: Value '184573849' is below minimum 939524096 2024-09-15T23:32:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:32:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:32:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:32:01.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:01 smithi174 ceph-mon[35586]: pgmap v590: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-09-15T23:32:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:32:01.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:01 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi174 to 176.0M 2024-09-15T23:32:01.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:01 smithi089 ceph-mon[30933]: Unable to set osd_memory_target on smithi174 to 184573849: error parsing value: Value '184573849' is below minimum 939524096 2024-09-15T23:32:01.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:32:01.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:32:01.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:32:01.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:01 smithi089 ceph-mon[30933]: pgmap v590: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 19 op/s 2024-09-15T23:32:02.948 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:32:03.211 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:03.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:03 smithi089 ceph-mon[30933]: pgmap v591: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 KiB/s wr, 15 op/s 2024-09-15T23:32:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:03 smithi120 ceph-mon[35250]: pgmap v591: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 KiB/s wr, 15 op/s 2024-09-15T23:32:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:03 smithi174 ceph-mon[35586]: pgmap v591: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 KiB/s wr, 15 op/s 2024-09-15T23:32:03.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.839+0000 7ff8b5521640 1 -- 172.21.15.89:0/2471767071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0102790 msgr2=0x7ff8b01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:03.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.839+0000 7ff8b5521640 1 --2- 172.21.15.89:0/2471767071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0102790 0x7ff8b01067e0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4009900 tx=0x7ff8a402a410 comp rx=0 tx=0).stop 2024-09-15T23:32:03.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.839+0000 7ff8b5521640 1 -- 172.21.15.89:0/2471767071 shutdown_connections 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.839+0000 7ff8b5521640 1 --2- 172.21.15.89:0/2471767071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b0106f10 0x7ff8b010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.839+0000 7ff8b5521640 1 --2- 172.21.15.89:0/2471767071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0102790 0x7ff8b01067e0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.839+0000 7ff8b5521640 1 --2- 172.21.15.89:0/2471767071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8b0101de0 0x7ff8b01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.839+0000 7ff8b5521640 1 -- 172.21.15.89:0/2471767071 >> 172.21.15.89:0/2471767071 conn(0x7ff8b00fd8c0 msgr2=0x7ff8b00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.839+0000 7ff8b5521640 1 -- 172.21.15.89:0/2471767071 shutdown_connections 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.839+0000 7ff8b5521640 1 -- 172.21.15.89:0/2471767071 wait complete. 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8b5521640 1 Processor -- start 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8b5521640 1 -- start start 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8b5521640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8b0101de0 0x7ff8b0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8b5521640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b0102790 0x7ff8b0198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8b5521640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0106f10 0x7ff8b019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8b5521640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b0111f90 con 0x7ff8b0106f10 2024-09-15T23:32:03.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8b5521640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b0111e10 con 0x7ff8b0101de0 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8b5521640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b0112110 con 0x7ff8b0102790 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8af7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0106f10 0x7ff8b019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8af7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0106f10 0x7ff8b019ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54906/0 (socket says 172.21.15.89:54906) 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.840+0000 7ff8af7fe640 1 -- 172.21.15.89:0/3187505282 learned_addr learned my addr 172.21.15.89:0/3187505282 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8b0101de0 0x7ff8b0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b0102790 0x7ff8b0198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff8af7fe640 1 -- 172.21.15.89:0/3187505282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b0102790 msgr2=0x7ff8b0198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff8af7fe640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b0102790 0x7ff8b0198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff8af7fe640 1 -- 172.21.15.89:0/3187505282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8b0101de0 msgr2=0x7ff8b0198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff8af7fe640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8b0101de0 0x7ff8b0198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff8af7fe640 1 -- 172.21.15.89:0/3187505282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b019d3d0 con 0x7ff8b0106f10 2024-09-15T23:32:03.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff8af7fe640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0106f10 0x7ff8b019ccf0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800b970 tx=0x7ff8a800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff897fff640 1 -- 172.21.15.89:0/3187505282 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8019070 con 0x7ff8b0106f10 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff897fff640 1 -- 172.21.15.89:0/3187505282 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8a800cb40 con 0x7ff8b0106f10 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff8b5521640 1 -- 172.21.15.89:0/3187505282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b019d6c0 con 0x7ff8b0106f10 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff897fff640 1 -- 172.21.15.89:0/3187505282 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8007650 con 0x7ff8b0106f10 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.841+0000 7ff8b5521640 1 -- 172.21.15.89:0/3187505282 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b01a4fa0 con 0x7ff8b0106f10 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.843+0000 7ff897fff640 1 -- 172.21.15.89:0/3187505282 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff8a801c070 con 0x7ff8b0106f10 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.843+0000 7ff8b78fa740 1 Processor -- start 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8b78fa740 1 -- start start 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d446ff4a0 0x559d446ff880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d4470a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d4470a680 0x559d4470ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559d446652e0 con 0x559d446ffdc0 2024-09-15T23:32:03.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559d4460cce0 con 0x559d4470a680 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559d446612a0 con 0x559d446ff4a0 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff897fff640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8980a3240 0x7ff8980a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8aeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d4470a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8aeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d4470a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54916/0 (socket says 172.21.15.89:54916) 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8aeffd640 1 -- 172.21.15.89:0/920185169 learned_addr learned my addr 172.21.15.89:0/920185169 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/920185169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d446ff4a0 0x559d446ff880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8af7fe640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8980a3240 0x7ff8980a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/920185169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d4470a680 0x559d4470ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff897fff640 1 -- 172.21.15.89:0/3187505282 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff8a80cf310 con 0x7ff8b0106f10 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8aeffd640 1 -- 172.21.15.89:0/920185169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d446ff4a0 msgr2=0x559d446ff880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/920185169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d446ff4a0 0x559d446ff880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8aeffd640 1 -- 172.21.15.89:0/920185169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d4470a680 msgr2=0x559d4470ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/920185169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d4470a680 0x559d4470ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8aeffd640 1 -- 172.21.15.89:0/920185169 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d445f1440 con 0x559d446ffdc0 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/920185169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d4470a680 0x559d4470ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.844+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/920185169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d4470a140 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000cc60 tx=0x7ff8a000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fd82f9754fdd9b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8af7fe640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8980a3240 0x7ff8980a5700 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8097f10 tx=0x7ff8a806e3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8957fa640 1 -- 172.21.15.89:0/920185169 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0014070 con 0x559d446ffdc0 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8957fa640 1 -- 172.21.15.89:0/920185169 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8a00047a0 con 0x559d446ffdc0 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8957fa640 1 -- 172.21.15.89:0/920185169 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0010720 con 0x559d446ffdc0 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8b78fa740 1 -- 172.21.15.89:0/920185169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 msgr2=0x559d4470a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/920185169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d4470a140 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000cc60 tx=0x7ff8a000e720 comp rx=0 tx=0).stop 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8b78fa740 1 -- 172.21.15.89:0/920185169 shutdown_connections 2024-09-15T23:32:03.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/920185169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d4470a680 0x559d4470ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/920185169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d4470a140 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/920185169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d446ff4a0 0x559d446ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.845+0000 7ff8b78fa740 1 -- 172.21.15.89:0/920185169 >> 172.21.15.89:0/920185169 conn(0x559d446fba30 msgr2=0x559d446fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 1 -- 172.21.15.89:0/920185169 shutdown_connections 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 1 -- 172.21.15.89:0/920185169 wait complete. 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 1 -- start start 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 1 Processor -- start 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ff4a0 0x559d44709f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d44707220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d4470a680 0x559d44707760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559d4470d250 con 0x559d446ffdc0 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559d44620050 con 0x559d446ff4a0 2024-09-15T23:32:03.866 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559d4470d3d0 con 0x559d4470a680 2024-09-15T23:32:03.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8b78fa740 10 client.0 _ll_register_callbacks cb 0x559d44660160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:03.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8aeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d44707220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8aeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d44707220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54918/0 (socket says 172.21.15.89:54918) 2024-09-15T23:32:03.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8aeffd640 1 -- 172.21.15.89:0/1219135423 learned_addr learned my addr 172.21.15.89:0/1219135423 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:03.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ff4a0 0x559d44709f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.846+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d4470a680 0x559d44707760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8aeffd640 1 -- 172.21.15.89:0/1219135423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d4470a680 msgr2=0x559d44707760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:03.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d4470a680 0x559d44707760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8aeffd640 1 -- 172.21.15.89:0/1219135423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ff4a0 msgr2=0x559d44709f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ff4a0 0x559d44709f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8aeffd640 1 -- 172.21.15.89:0/1219135423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d443d63c0 con 0x559d446ffdc0 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ff4a0 0x559d44709f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d44707220 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000cc30 tx=0x7ff8a0010bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e069f203957b77d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff892ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff892ff5640 1 -- 172.21.15.89:0/1219135423 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0027060 con 0x559d446ffdc0 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff892ff5640 1 -- 172.21.15.89:0/1219135423 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff8a0022410 con 0x559d446ffdc0 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1219135423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8a00047a0 con 0x559d446ffdc0 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff892ff5640 1 -- 172.21.15.89:0/1219135423 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff8a002e050 con 0x559d446ffdc0 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8b78fa740 10 client.18069 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1219135423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559d44718b30 con 0x559d446ffdc0 2024-09-15T23:32:03.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff892ff5640 1 -- 172.21.15.89:0/1219135423 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8a00047a0 con 0x559d446ffdc0 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff892ff5640 1 -- 172.21.15.89:0/1219135423 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff8a000f3c0 con 0x559d446ffdc0 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8b78fa740 10 client.18069 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.847+0000 7ff8b78fa740 10 client.18069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.848+0000 7ff8b78fa740 20 client.18069 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.848+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559d44719ff0 0x559d4471c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.848+0000 7ff8b78fa740 4 client.18069 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.848+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1219135423 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d44672850 con 0x559d44719ff0 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.848+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559d44719ff0 0x559d4471c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.848+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559d44719ff0 0x559d4471c3e0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73a80b5a6c64568a server_cookie=ebf27c4749b405ae in_seq=0 out_seq=0 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.848+0000 7ff892ff5640 10 client.18069 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff892ff5640 1 -- 172.21.15.89:0/1219135423 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559d44672850 con 0x559d44719ff0 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff892ff5640 10 client.18069 handle_command_reply: tid=0 2024-09-15T23:32:03.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 client.18069 shutdown 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1219135423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 msgr2=0x559d44707220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d44707220 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000cc30 tx=0x7ff8a0010bb0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1219135423 shutdown_connections 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559d44719ff0 0x559d4471c3e0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d4470a680 0x559d44707760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ffdc0 0x559d44707220 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff892ff5640 0 client.18069 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/1219135423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ff4a0 0x559d44709f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1219135423 >> 172.21.15.89:0/1219135423 conn(0x559d446fba30 msgr2=0x559d44701710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1219135423 shutdown_connections 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1219135423 wait complete. 2024-09-15T23:32:03.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.849+0000 7ff8b78fa740 20 client.18069 trim_cache size 0 max 16384 2024-09-15T23:32:03.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.866+0000 7ff8b78fa740 1 Processor -- start 2024-09-15T23:32:03.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8b78fa740 1 -- start start 2024-09-15T23:32:03.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d446ff4a0 0x559d44705af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 0x559d44712a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4470a680 0x559d44712f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b01aeed0 con 0x559d4470a680 2024-09-15T23:32:03.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b01b0a10 con 0x559d446ffdc0 2024-09-15T23:32:03.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b01af5a0 con 0x559d446ff4a0 2024-09-15T23:32:03.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8ae7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4470a680 0x559d44712f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8ae7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4470a680 0x559d44712f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54930/0 (socket says 172.21.15.89:54930) 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8ae7fc640 1 -- 172.21.15.89:0/1731414080 learned_addr learned my addr 172.21.15.89:0/1731414080 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/1731414080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 0x559d44712a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/1731414080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d446ff4a0 0x559d44705af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8ae7fc640 1 -- 172.21.15.89:0/1731414080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d446ff4a0 msgr2=0x559d44705af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/1731414080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d446ff4a0 0x559d44705af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8ae7fc640 1 -- 172.21.15.89:0/1731414080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 msgr2=0x559d44712a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/1731414080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 0x559d44712a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8ae7fc640 1 -- 172.21.15.89:0/1731414080 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d44700330 con 0x559d4470a680 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.867+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/1731414080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4470a680 0x559d44712f40 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4033000 tx=0x7ff8a4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80947907a4dc221a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff894ff9640 1 -- 172.21.15.89:0/1731414080 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4035040 con 0x559d4470a680 2024-09-15T23:32:03.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff894ff9640 1 -- 172.21.15.89:0/1731414080 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8a40035f0 con 0x559d4470a680 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff894ff9640 1 -- 172.21.15.89:0/1731414080 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4047510 con 0x559d4470a680 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1731414080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4470a680 msgr2=0x559d44712f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/1731414080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4470a680 0x559d44712f40 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4033000 tx=0x7ff8a4009260 comp rx=0 tx=0).stop 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1731414080 shutdown_connections 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/1731414080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4470a680 0x559d44712f40 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/1731414080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 0x559d44712a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/1731414080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d446ff4a0 0x559d44705af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1731414080 >> 172.21.15.89:0/1731414080 conn(0x559d446fba30 msgr2=0x559d447092a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1731414080 shutdown_connections 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.868+0000 7ff8b78fa740 1 -- 172.21.15.89:0/1731414080 wait complete. 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8b78fa740 1 -- start start 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8b78fa740 1 Processor -- start 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ff4a0 0x559d4477ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 0x559d4477bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8b78fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d4470a680 0x559d44780090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b01b0340 con 0x559d446ff4a0 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b01afc70 con 0x559d446ffdc0 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8b78fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b01aceb0 con 0x559d4470a680 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8b78fa740 10 client.0 _ll_register_callbacks cb 0x7ff8a4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8ae7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ff4a0 0x559d4477ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8ae7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ff4a0 0x559d4477ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54946/0 (socket says 172.21.15.89:54946) 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8ae7fc640 1 -- 172.21.15.89:0/987323848 learned_addr learned my addr 172.21.15.89:0/987323848 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.869+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 0x559d4477bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff8ae7fc640 1 -- 172.21.15.89:0/987323848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d4470a680 msgr2=0x559d44780090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d4470a680 0x559d44780090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff8ae7fc640 1 -- 172.21.15.89:0/987323848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 msgr2=0x559d4477bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 0x559d4477bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff8ae7fc640 1 -- 172.21.15.89:0/987323848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d44780730 con 0x559d446ff4a0 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 0x559d4477bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff8ae7fc640 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ff4a0 0x559d4477ba80 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4006010 tx=0x7ff8a4031760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b4bcdd56e5d47 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff892ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff892ff5640 1 -- 172.21.15.89:0/987323848 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4035040 con 0x559d446ff4a0 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff892ff5640 1 -- 172.21.15.89:0/987323848 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff8a40037a0 con 0x559d446ff4a0 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff8b78fa740 1 -- 172.21.15.89:0/987323848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8a40035f0 con 0x559d446ff4a0 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff892ff5640 1 -- 172.21.15.89:0/987323848 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff8a4062050 con 0x559d446ff4a0 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff8b78fa740 10 client.18075 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff8b78fa740 1 -- 172.21.15.89:0/987323848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559d44788cd0 con 0x559d446ff4a0 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff892ff5640 1 -- 172.21.15.89:0/987323848 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8a40035f0 con 0x559d446ff4a0 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.870+0000 7ff892ff5640 1 -- 172.21.15.89:0/987323848 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff8a403a040 con 0x559d446ff4a0 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff8b78fa740 10 client.18075 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff8b78fa740 10 client.18075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:03.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff8b78fa740 20 client.18075 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559d447892c0 0x559d4478b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff8b78fa740 4 client.18075 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff8b78fa740 1 -- 172.21.15.89:0/987323848 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559d4466cd60 con 0x559d447892c0 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559d447892c0 0x559d4478b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff8aeffd640 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559d447892c0 0x559d4478b6e0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a439f27f9fda0d00 server_cookie=a906b359c9b95ef0 in_seq=0 out_seq=0 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff892ff5640 10 client.18075 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff892ff5640 1 -- 172.21.15.89:0/987323848 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559d4466cd60 con 0x559d447892c0 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff892ff5640 10 client.18075 handle_command_reply: tid=0 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.871+0000 7ff8b78fa740 1 client.18075 shutdown 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 1 -- 172.21.15.89:0/987323848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ff4a0 msgr2=0x559d4477ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ff4a0 0x559d4477ba80 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4006010 tx=0x7ff8a4031760 comp rx=0 tx=0).stop 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 1 -- 172.21.15.89:0/987323848 shutdown_connections 2024-09-15T23:32:03.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559d447892c0 0x559d4478b6e0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559d4470a680 0x559d44780090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d446ffdc0 0x559d4477bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 1 --2- 172.21.15.89:0/987323848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d446ff4a0 0x559d4477ba80 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 1 -- 172.21.15.89:0/987323848 >> 172.21.15.89:0/987323848 conn(0x559d446fba30 msgr2=0x559d446fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:03.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff892ff5640 0 client.18075 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:03.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 1 -- 172.21.15.89:0/987323848 shutdown_connections 2024-09-15T23:32:03.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 1 -- 172.21.15.89:0/987323848 wait complete. 2024-09-15T23:32:03.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b78fa740 20 client.18075 trim_cache size 0 max 16384 2024-09-15T23:32:03.881 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:32:03.882 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:32:03.882 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:32:03.882 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:32:03.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b5521640 1 -- 172.21.15.89:0/3187505282 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8980a3240 msgr2=0x7ff8980a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:03.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b5521640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8980a3240 0x7ff8980a5700 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8097f10 tx=0x7ff8a806e3c0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b5521640 1 -- 172.21.15.89:0/3187505282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0106f10 msgr2=0x7ff8b019ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:03.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b5521640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0106f10 0x7ff8b019ccf0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800b970 tx=0x7ff8a800be30 comp rx=0 tx=0).stop 2024-09-15T23:32:03.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b5521640 1 -- 172.21.15.89:0/3187505282 shutdown_connections 2024-09-15T23:32:03.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b5521640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8980a3240 0x7ff8980a5700 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b5521640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0106f10 0x7ff8b019ccf0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b5521640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b0102790 0x7ff8b0198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b5521640 1 --2- 172.21.15.89:0/3187505282 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8b0101de0 0x7ff8b0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:03.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.872+0000 7ff8b5521640 1 -- 172.21.15.89:0/3187505282 >> 172.21.15.89:0/3187505282 conn(0x7ff8b00fd8c0 msgr2=0x7ff8b00fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:03.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.873+0000 7ff8b5521640 1 -- 172.21.15.89:0/3187505282 shutdown_connections 2024-09-15T23:32:03.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:03.873+0000 7ff8b5521640 1 -- 172.21.15.89:0/3187505282 wait complete. 2024-09-15T23:32:04.744 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:32:04.744 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:32:04.992 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:05.387 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:05 smithi089 ceph-mon[30933]: pgmap v592: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 KiB/s wr, 14 op/s 2024-09-15T23:32:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:05 smithi174 ceph-mon[35586]: pgmap v592: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 KiB/s wr, 14 op/s 2024-09-15T23:32:05.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 -- 172.21.15.89:0/2651968355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb0100a80 msgr2=0x7feeb0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:05.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 --2- 172.21.15.89:0/2651968355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb0100a80 0x7feeb0100ee0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7feea4009930 tx=0x7feea402a420 comp rx=0 tx=0).stop 2024-09-15T23:32:05.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 -- 172.21.15.89:0/2651968355 shutdown_connections 2024-09-15T23:32:05.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 --2- 172.21.15.89:0/2651968355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feeb0101420 0x7feeb010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:05.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 --2- 172.21.15.89:0/2651968355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb0100a80 0x7feeb0100ee0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:05.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 --2- 172.21.15.89:0/2651968355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feeb0106a80 0x7feeb0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 -- 172.21.15.89:0/2651968355 >> 172.21.15.89:0/2651968355 conn(0x7feeb00fc7a0 msgr2=0x7feeb00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 -- 172.21.15.89:0/2651968355 shutdown_connections 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 -- 172.21.15.89:0/2651968355 wait complete. 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 Processor -- start 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.465+0000 7feeb8321640 1 -- start start 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb8321640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feeb0100a80 0x7feeb01992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb8321640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb0101420 0x7feeb01997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb8321640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feeb0106a80 0x7feeb01a4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb8321640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feeb0110e80 con 0x7feeb0101420 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb8321640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feeb0110d00 con 0x7feeb0106a80 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb8321640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feeb0111000 con 0x7feeb0100a80 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb58bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb0101420 0x7feeb01997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:05.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb58bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb0101420 0x7feeb01997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53364/0 (socket says 172.21.15.89:53364) 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb58bc640 1 -- 172.21.15.89:0/2963200739 learned_addr learned my addr 172.21.15.89:0/2963200739 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb60bd640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feeb0100a80 0x7feeb01992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb68be640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feeb0106a80 0x7feeb01a4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb58bc640 1 -- 172.21.15.89:0/2963200739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feeb0100a80 msgr2=0x7feeb01992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb58bc640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feeb0100a80 0x7feeb01992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb58bc640 1 -- 172.21.15.89:0/2963200739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feeb0106a80 msgr2=0x7feeb01a4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb58bc640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feeb0106a80 0x7feeb01a4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb58bc640 1 -- 172.21.15.89:0/2963200739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb01a4860 con 0x7feeb0101420 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb58bc640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb0101420 0x7feeb01997e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7feea4009230 tx=0x7feea4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeaf7fe640 1 -- 172.21.15.89:0/2963200739 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea403f030 con 0x7feeb0101420 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeaf7fe640 1 -- 172.21.15.89:0/2963200739 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea4003ad0 con 0x7feeb0101420 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeb8321640 1 -- 172.21.15.89:0/2963200739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb01a4a30 con 0x7feeb0101420 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.466+0000 7feeaf7fe640 1 -- 172.21.15.89:0/2963200739 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4035070 con 0x7feeb0101420 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.467+0000 7feeb8321640 1 -- 172.21.15.89:0/2963200739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb01a4e80 con 0x7feeb0101420 2024-09-15T23:32:05.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.467+0000 7feeb8321640 1 -- 172.21.15.89:0/2963200739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb0100ee0 con 0x7feeb0101420 2024-09-15T23:32:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.469+0000 7feeaf7fe640 1 -- 172.21.15.89:0/2963200739 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7feea4042030 con 0x7feeb0101420 2024-09-15T23:32:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.469+0000 7feeaf7fe640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fee980a3240 0x7fee980a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.469+0000 7feeb60bd640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fee980a3240 0x7fee980a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.469+0000 7feeaf7fe640 1 -- 172.21.15.89:0/2963200739 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7feea40f0630 con 0x7feeb0101420 2024-09-15T23:32:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.470+0000 7feeb60bd640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fee980a3240 0x7fee980a5700 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7feea0005c40 tx=0x7feea000a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:05.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.472+0000 7feeaf7fe640 1 -- 172.21.15.89:0/2963200739 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7feea40bcf60 con 0x7feeb0101420 2024-09-15T23:32:05.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.673+0000 7feeb8321640 1 -- 172.21.15.89:0/2963200739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feeb0109010 con 0x7feeb0101420 2024-09-15T23:32:05.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.674+0000 7feeaf7fe640 1 -- 172.21.15.89:0/2963200739 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7feea40bc860 con 0x7feeb0101420 2024-09-15T23:32:05.677 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:32:05.677 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:32:05.677 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:32:05.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 -- 172.21.15.89:0/2963200739 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fee980a3240 msgr2=0x7fee980a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:05.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fee980a3240 0x7fee980a5700 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7feea0005c40 tx=0x7feea000a590 comp rx=0 tx=0).stop 2024-09-15T23:32:05.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 -- 172.21.15.89:0/2963200739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb0101420 msgr2=0x7feeb01997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:05.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb0101420 0x7feeb01997e0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7feea4009230 tx=0x7feea4004270 comp rx=0 tx=0).stop 2024-09-15T23:32:05.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 -- 172.21.15.89:0/2963200739 shutdown_connections 2024-09-15T23:32:05.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fee980a3240 0x7fee980a5700 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:05.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feeb0106a80 0x7feeb01a4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:05.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb0101420 0x7feeb01997e0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:05.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 --2- 172.21.15.89:0/2963200739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feeb0100a80 0x7feeb01992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:05.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 -- 172.21.15.89:0/2963200739 >> 172.21.15.89:0/2963200739 conn(0x7feeb00fc7a0 msgr2=0x7feeb00fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:05.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 -- 172.21.15.89:0/2963200739 shutdown_connections 2024-09-15T23:32:05.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:05.677+0000 7feeb8321640 1 -- 172.21.15.89:0/2963200739 wait complete. 2024-09-15T23:32:05.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:05 smithi120 ceph-mon[35250]: pgmap v592: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.0 KiB/s wr, 14 op/s 2024-09-15T23:32:06.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:06 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2963200739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:06.365 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:32:06.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:06 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2963200739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:06.614 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:06.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:06 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2963200739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:07.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 -- 172.21.15.89:0/1593527353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51e0100e20 msgr2=0x7f51e0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 --2- 172.21.15.89:0/1593527353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51e0100e20 0x7f51e0101200 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009900 tx=0x7f51d002a480 comp rx=0 tx=0).stop 2024-09-15T23:32:07.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 -- 172.21.15.89:0/1593527353 shutdown_connections 2024-09-15T23:32:07.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 --2- 172.21.15.89:0/1593527353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51e0102ef0 0x7f51e010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 --2- 172.21.15.89:0/1593527353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51e0102570 0x7f51e01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 --2- 172.21.15.89:0/1593527353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51e0100e20 0x7f51e0101200 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 -- 172.21.15.89:0/1593527353 >> 172.21.15.89:0/1593527353 conn(0x7f51e00fc7d0 msgr2=0x7f51e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 -- 172.21.15.89:0/1593527353 shutdown_connections 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 -- 172.21.15.89:0/1593527353 wait complete. 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 Processor -- start 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.160+0000 7f51e5840640 1 -- start start 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51e5840640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51e0100e20 0x7f51e0198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51e5840640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51e0102570 0x7f51e0198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51e5840640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51e0102ef0 0x7f51e019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51e5840640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51e01120c0 con 0x7f51e0102570 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51e5840640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51e0111f40 con 0x7f51e0102ef0 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51e5840640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f51e0112240 con 0x7f51e0100e20 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51e0102570 0x7f51e0198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51e0102570 0x7f51e0198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53388/0 (socket says 172.21.15.89:53388) 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51d67fc640 1 -- 172.21.15.89:0/1303582767 learned_addr learned my addr 172.21.15.89:0/1303582767 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51deffd640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51e0100e20 0x7f51e0198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51df7fe640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51e0102ef0 0x7f51e019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51d67fc640 1 -- 172.21.15.89:0/1303582767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51e0100e20 msgr2=0x7f51e0198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51d67fc640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51e0100e20 0x7f51e0198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51d67fc640 1 -- 172.21.15.89:0/1303582767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51e0102ef0 msgr2=0x7f51e019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51d67fc640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51e0102ef0 0x7f51e019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51d67fc640 1 -- 172.21.15.89:0/1303582767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e019d4e0 con 0x7f51e0102570 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51df7fe640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51e0102ef0 0x7f51e019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51d67fc640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51e0102570 0x7f51e0198a70 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c970 tx=0x7f51c800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51e483e640 1 -- 172.21.15.89:0/1303582767 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8007bb0 con 0x7f51e0102570 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51e483e640 1 -- 172.21.15.89:0/1303582767 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c8007d50 con 0x7f51e0102570 2024-09-15T23:32:07.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51e5840640 1 -- 172.21.15.89:0/1303582767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e019d7d0 con 0x7f51e0102570 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.161+0000 7f51e483e640 1 -- 172.21.15.89:0/1303582767 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c800f450 con 0x7f51e0102570 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.162+0000 7f51e5840640 1 -- 172.21.15.89:0/1303582767 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e01a50b0 con 0x7f51e0102570 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.164+0000 7f51e483e640 1 -- 172.21.15.89:0/1303582767 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f51c8005020 con 0x7f51e0102570 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.164+0000 7f51e7c19740 1 Processor -- start 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.164+0000 7f51e7c19740 1 -- start start 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51e483e640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51cc0a3240 0x7f51cc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 0x5579e4c83090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c83a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c8e370 0x5579e4c90760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51e7c19740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5579e4be92f0 con 0x5579e4c835d0 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51e7c19740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5579e4b90cf0 con 0x5579e4c84b40 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51e7c19740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5579e4be52b0 con 0x5579e4c8e370 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51df7fe640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51cc0a3240 0x7f51cc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c83a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c83a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53396/0 (socket says 172.21.15.89:53396) 2024-09-15T23:32:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d67fc640 1 -- 172.21.15.89:0/2028039179 learned_addr learned my addr 172.21.15.89:0/2028039179 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51e483e640 1 -- 172.21.15.89:0/1303582767 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f51c80cfb90 con 0x7f51e0102570 2024-09-15T23:32:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51df7fe640 1 --2- 172.21.15.89:0/2028039179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c8e370 0x5579e4c90760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51deffd640 1 --2- 172.21.15.89:0/2028039179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 0x5579e4c83090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d67fc640 1 -- 172.21.15.89:0/2028039179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c8e370 msgr2=0x5579e4c90760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d67fc640 1 --2- 172.21.15.89:0/2028039179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c8e370 0x5579e4c90760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d67fc640 1 -- 172.21.15.89:0/2028039179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 msgr2=0x5579e4c83090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d67fc640 1 --2- 172.21.15.89:0/2028039179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 0x5579e4c83090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d67fc640 1 -- 172.21.15.89:0/2028039179 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579e4b753a0 con 0x5579e4c835d0 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51deffd640 1 --2- 172.21.15.89:0/2028039179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 0x5579e4c83090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51df7fe640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51cc0a3240 0x7f51cc0a5700 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f51d8010740 tx=0x7f51d8007a50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d67fc640 1 --2- 172.21.15.89:0/2028039179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c83a10 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f51c8007590 tx=0x7f51c801baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769e679d58d14e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d5ffb640 1 -- 172.21.15.89:0/2028039179 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c809b720 con 0x5579e4c835d0 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d5ffb640 1 -- 172.21.15.89:0/2028039179 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c809c4c0 con 0x5579e4c835d0 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.165+0000 7f51d5ffb640 1 -- 172.21.15.89:0/2028039179 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c80b4fc0 con 0x5579e4c835d0 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.166+0000 7f51e7c19740 1 -- 172.21.15.89:0/2028039179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 msgr2=0x5579e4c83a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.166+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2028039179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c83a10 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f51c8007590 tx=0x7f51c801baf0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.166+0000 7f51e7c19740 1 -- 172.21.15.89:0/2028039179 shutdown_connections 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.166+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2028039179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c8e370 0x5579e4c90760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.166+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2028039179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c83a10 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.166+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2028039179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 0x5579e4c83090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.166+0000 7f51e7c19740 1 -- 172.21.15.89:0/2028039179 >> 172.21.15.89:0/2028039179 conn(0x5579e4c4c310 msgr2=0x5579e4c7fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.166+0000 7f51e7c19740 1 -- 172.21.15.89:0/2028039179 shutdown_connections 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.166+0000 7f51e7c19740 1 -- 172.21.15.89:0/2028039179 wait complete. 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51e7c19740 1 -- start start 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51e7c19740 1 Processor -- start 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c8de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 0x5579e4c8b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c8e370 0x5579e4c8b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51e7c19740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5579e4c91070 con 0x5579e4c835d0 2024-09-15T23:32:07.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51e7c19740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5579e4ba4060 con 0x5579e4c84b40 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51e7c19740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5579e4c911f0 con 0x5579e4c8e370 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51e7c19740 10 client.0 _ll_register_callbacks cb 0x5579e48f78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51deffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c8de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51deffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c8de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53408/0 (socket says 172.21.15.89:53408) 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51deffd640 1 -- 172.21.15.89:0/2111423993 learned_addr learned my addr 172.21.15.89:0/2111423993 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51d67fc640 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 0x5579e4c8b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51df7fe640 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c8e370 0x5579e4c8b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51deffd640 1 -- 172.21.15.89:0/2111423993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c8e370 msgr2=0x5579e4c8b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51deffd640 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c8e370 0x5579e4c8b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.167+0000 7f51deffd640 1 -- 172.21.15.89:0/2111423993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 msgr2=0x5579e4c8b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51deffd640 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 0x5579e4c8b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51deffd640 1 -- 172.21.15.89:0/2111423993 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579e495a3c0 con 0x5579e4c835d0 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51d67fc640 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 0x5579e4c8b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51deffd640 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c8de00 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f51d002a9a0 tx=0x7f51d00044a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a0456e2fdcb82c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51beffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51beffd640 1 -- 172.21.15.89:0/2111423993 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d0035030 con 0x5579e4c835d0 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51beffd640 1 -- 172.21.15.89:0/2111423993 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f51d0003a80 con 0x5579e4c835d0 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51e7c19740 1 -- 172.21.15.89:0/2111423993 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51c809c4c0 con 0x5579e4c835d0 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51beffd640 1 -- 172.21.15.89:0/2111423993 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51d004d050 con 0x5579e4c835d0 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51e7c19740 10 client.18093 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51e7c19740 1 -- 172.21.15.89:0/2111423993 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5579e4c9cbc0 con 0x5579e4c835d0 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51beffd640 1 -- 172.21.15.89:0/2111423993 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c809c4c0 con 0x5579e4c835d0 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51beffd640 1 -- 172.21.15.89:0/2111423993 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f51d003c460 con 0x5579e4c835d0 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51e7c19740 10 client.18093 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51e7c19740 10 client.18093 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:32:07.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51e7c19740 20 client.18093 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5579e4c9e080 0x5579e4ca0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51e7c19740 4 client.18093 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.168+0000 7f51e7c19740 1 -- 172.21.15.89:0/2111423993 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579e4bf6860 con 0x5579e4c9e080 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.169+0000 7f51d67fc640 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5579e4c9e080 0x5579e4ca0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.169+0000 7f51d67fc640 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5579e4c9e080 0x5579e4ca0490 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=366e65ae39f26b13 server_cookie=7688975528ee410c in_seq=0 out_seq=0 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.169+0000 7f51beffd640 10 client.18093 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51beffd640 1 -- 172.21.15.89:0/2111423993 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5579e4bf6860 con 0x5579e4c9e080 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51beffd640 10 client.18093 handle_command_reply: tid=0 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 client.18093 shutdown 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 -- 172.21.15.89:0/2111423993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 msgr2=0x5579e4c8de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c8de00 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f51d002a9a0 tx=0x7f51d00044a0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 -- 172.21.15.89:0/2111423993 shutdown_connections 2024-09-15T23:32:07.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5579e4c9e080 0x5579e4ca0490 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c8e370 0x5579e4c8b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c84b40 0x5579e4c8b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2111423993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c835d0 0x5579e4c8de00 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 -- 172.21.15.89:0/2111423993 >> 172.21.15.89:0/2111423993 conn(0x5579e4c4c310 msgr2=0x5579e4c85990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51beffd640 0 client.18093 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:32:07.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 -- 172.21.15.89:0/2111423993 shutdown_connections 2024-09-15T23:32:07.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 1 -- 172.21.15.89:0/2111423993 wait complete. 2024-09-15T23:32:07.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.170+0000 7f51e7c19740 20 client.18093 trim_cache size 0 max 16384 2024-09-15T23:32:07.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.183+0000 7f51e7c19740 1 Processor -- start 2024-09-15T23:32:07.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.183+0000 7f51e7c19740 1 -- start start 2024-09-15T23:32:07.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.183+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c835d0 0x5579e4c8a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.183+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c84b40 0x5579e4c85880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c8e370 0x5579e4ca5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51e7c19740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51e01aee30 con 0x5579e4c8e370 2024-09-15T23:32:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51e7c19740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51e01b0970 con 0x5579e4c835d0 2024-09-15T23:32:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51e7c19740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f51e01af500 con 0x5579e4c84b40 2024-09-15T23:32:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51df7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c8e370 0x5579e4ca5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51df7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c8e370 0x5579e4ca5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53418/0 (socket says 172.21.15.89:53418) 2024-09-15T23:32:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51df7fe640 1 -- 172.21.15.89:0/2024248312 learned_addr learned my addr 172.21.15.89:0/2024248312 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51d67fc640 1 --2- 172.21.15.89:0/2024248312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c84b40 0x5579e4c85880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51deffd640 1 --2- 172.21.15.89:0/2024248312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c835d0 0x5579e4c8a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51df7fe640 1 -- 172.21.15.89:0/2024248312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c84b40 msgr2=0x5579e4c85880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51df7fe640 1 --2- 172.21.15.89:0/2024248312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c84b40 0x5579e4c85880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51df7fe640 1 -- 172.21.15.89:0/2024248312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c835d0 msgr2=0x5579e4c8a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51df7fe640 1 --2- 172.21.15.89:0/2024248312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c835d0 0x5579e4c8a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51df7fe640 1 -- 172.21.15.89:0/2024248312 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579e4c89600 con 0x5579e4c8e370 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51deffd640 1 --2- 172.21.15.89:0/2024248312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c835d0 0x5579e4c8a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51df7fe640 1 --2- 172.21.15.89:0/2024248312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c8e370 0x5579e4ca5d60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f51d80137c0 tx=0x7f51d8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e6e4a1f3be67d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51d57fa640 1 -- 172.21.15.89:0/2024248312 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8011b60 con 0x5579e4c8e370 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51d57fa640 1 -- 172.21.15.89:0/2024248312 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d8011d00 con 0x5579e4c8e370 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.184+0000 7f51d57fa640 1 -- 172.21.15.89:0/2024248312 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d800d690 con 0x5579e4c8e370 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.185+0000 7f51e7c19740 1 -- 172.21.15.89:0/2024248312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c8e370 msgr2=0x5579e4ca5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.185+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2024248312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c8e370 0x5579e4ca5d60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f51d80137c0 tx=0x7f51d8016000 comp rx=0 tx=0).stop 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.185+0000 7f51e7c19740 1 -- 172.21.15.89:0/2024248312 shutdown_connections 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.185+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2024248312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c8e370 0x5579e4ca5d60 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.185+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2024248312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c84b40 0x5579e4c85880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.185+0000 7f51e7c19740 1 --2- 172.21.15.89:0/2024248312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c835d0 0x5579e4c8a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.185+0000 7f51e7c19740 1 -- 172.21.15.89:0/2024248312 >> 172.21.15.89:0/2024248312 conn(0x5579e4c4c310 msgr2=0x5579e4c85170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.185+0000 7f51e7c19740 1 -- 172.21.15.89:0/2024248312 shutdown_connections 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.185+0000 7f51e7c19740 1 -- 172.21.15.89:0/2024248312 wait complete. 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51e7c19740 1 -- start start 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51e7c19740 1 Processor -- start 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c835d0 0x5579e4c968a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c84b40 0x5579e4c91c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51e7c19740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c8e370 0x5579e4c921b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51e7c19740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f51e01b02a0 con 0x5579e4c84b40 2024-09-15T23:32:07.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51e7c19740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51e01afbd0 con 0x5579e4c8e370 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51e7c19740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f51e01ace10 con 0x5579e4c835d0 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51e7c19740 10 client.0 _ll_register_callbacks cb 0x7f51d0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c84b40 0x5579e4c91c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c84b40 0x5579e4c91c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53420/0 (socket says 172.21.15.89:53420) 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51d67fc640 1 -- 172.21.15.89:0/3082624578 learned_addr learned my addr 172.21.15.89:0/3082624578 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51deffd640 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c835d0 0x5579e4c968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.186+0000 7f51df7fe640 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c8e370 0x5579e4c921b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51d67fc640 1 -- 172.21.15.89:0/3082624578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c835d0 msgr2=0x5579e4c968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51d67fc640 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c835d0 0x5579e4c968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51d67fc640 1 -- 172.21.15.89:0/3082624578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c8e370 msgr2=0x5579e4c921b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51d67fc640 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c8e370 0x5579e4c921b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51d67fc640 1 -- 172.21.15.89:0/3082624578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579e4c926f0 con 0x5579e4c84b40 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51df7fe640 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c8e370 0x5579e4c921b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51deffd640 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c835d0 0x5579e4c968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51d67fc640 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c84b40 0x5579e4c91c70 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f51c809bac0 tx=0x7f51c809bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31feb677cb3bd87 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51beffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51beffd640 1 -- 172.21.15.89:0/3082624578 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c801d040 con 0x5579e4c84b40 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51beffd640 1 -- 172.21.15.89:0/3082624578 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f51c80b90b0 con 0x5579e4c84b40 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51e7c19740 1 -- 172.21.15.89:0/3082624578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d8011d00 con 0x5579e4c84b40 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51beffd640 1 -- 172.21.15.89:0/3082624578 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51c80b51a0 con 0x5579e4c84b40 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51e7c19740 10 client.18099 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51e7c19740 1 -- 172.21.15.89:0/3082624578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5579e4d08a00 con 0x5579e4c84b40 2024-09-15T23:32:07.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51beffd640 1 -- 172.21.15.89:0/3082624578 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d8011d00 con 0x5579e4c84b40 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51beffd640 1 -- 172.21.15.89:0/3082624578 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f51c8005310 con 0x5579e4c84b40 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51e7c19740 10 client.18099 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51e7c19740 10 client.18099 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51e7c19740 20 client.18099 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51e7c19740 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5579e4d08d00 0x5579e4d0b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51e7c19740 4 client.18099 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.187+0000 7f51e7c19740 1 -- 172.21.15.89:0/3082624578 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5579e4bf0d70 con 0x5579e4d08d00 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51deffd640 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5579e4d08d00 0x5579e4d0b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51deffd640 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5579e4d08d00 0x5579e4d0b0f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b3c2500f6525d88f server_cookie=88eff26ede6d3918 in_seq=0 out_seq=0 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51beffd640 10 client.18099 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:32:07.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51beffd640 1 -- 172.21.15.89:0/3082624578 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5579e4bf0d70 con 0x5579e4d08d00 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51beffd640 10 client.18099 handle_command_reply: tid=0 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51e7c19740 1 client.18099 shutdown 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51e7c19740 1 -- 172.21.15.89:0/3082624578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c84b40 msgr2=0x5579e4c91c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51e7c19740 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c84b40 0x5579e4c91c70 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f51c809bac0 tx=0x7f51c809bc90 comp rx=0 tx=0).stop 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51e7c19740 1 -- 172.21.15.89:0/3082624578 shutdown_connections 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51e7c19740 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5579e4d08d00 0x5579e4d0b0f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51e7c19740 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5579e4c8e370 0x5579e4c921b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51e7c19740 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5579e4c84b40 0x5579e4c91c70 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51beffd640 0 client.18099 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51e7c19740 1 --2- 172.21.15.89:0/3082624578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5579e4c835d0 0x5579e4c968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.188+0000 7f51e7c19740 1 -- 172.21.15.89:0/3082624578 >> 172.21.15.89:0/3082624578 conn(0x5579e4c4c310 msgr2=0x5579e4c7ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e7c19740 1 -- 172.21.15.89:0/3082624578 shutdown_connections 2024-09-15T23:32:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e7c19740 1 -- 172.21.15.89:0/3082624578 wait complete. 2024-09-15T23:32:07.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e7c19740 20 client.18099 trim_cache size 0 max 16384 2024-09-15T23:32:07.207 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 -- 172.21.15.89:0/1303582767 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51cc0a3240 msgr2=0x7f51cc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51cc0a3240 0x7f51cc0a5700 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f51d8010740 tx=0x7f51d8007a50 comp rx=0 tx=0).stop 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 -- 172.21.15.89:0/1303582767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51e0102570 msgr2=0x7f51e0198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51e0102570 0x7f51e0198a70 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c970 tx=0x7f51c800ce30 comp rx=0 tx=0).stop 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 -- 172.21.15.89:0/1303582767 shutdown_connections 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f51cc0a3240 0x7f51cc0a5700 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51e0102ef0 0x7f51e019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f51e0102570 0x7f51e0198a70 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 --2- 172.21.15.89:0/1303582767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51e0100e20 0x7f51e0198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 -- 172.21.15.89:0/1303582767 >> 172.21.15.89:0/1303582767 conn(0x7f51e00fc7d0 msgr2=0x7f51e00fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 -- 172.21.15.89:0/1303582767 shutdown_connections 2024-09-15T23:32:07.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:07.189+0000 7f51e5840640 1 -- 172.21.15.89:0/1303582767 wait complete. 2024-09-15T23:32:07.374 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:07 smithi089 ceph-mon[30933]: pgmap v593: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.6 KiB/s wr, 16 op/s 2024-09-15T23:32:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:07 smithi174 ceph-mon[35586]: pgmap v593: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.6 KiB/s wr, 16 op/s 2024-09-15T23:32:07.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:07 smithi120 ceph-mon[35250]: pgmap v593: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.6 KiB/s wr, 16 op/s 2024-09-15T23:32:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.854+0000 7fb56a376640 1 -- 172.21.15.89:0/2010601717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb564102610 msgr2=0x7fb564108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.854+0000 7fb56a376640 1 --2- 172.21.15.89:0/2010601717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb564102610 0x7fb564108ea0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009ca0 tx=0x7fb55c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.854+0000 7fb56a376640 1 -- 172.21.15.89:0/2010601717 shutdown_connections 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.854+0000 7fb56a376640 1 --2- 172.21.15.89:0/2010601717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb564102610 0x7fb564108ea0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.854+0000 7fb56a376640 1 --2- 172.21.15.89:0/2010601717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb564101c50 0x7fb5641020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.854+0000 7fb56a376640 1 --2- 172.21.15.89:0/2010601717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb564100a60 0x7fb564100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.854+0000 7fb56a376640 1 -- 172.21.15.89:0/2010601717 >> 172.21.15.89:0/2010601717 conn(0x7fb5640fc1e0 msgr2=0x7fb5640fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.854+0000 7fb56a376640 1 -- 172.21.15.89:0/2010601717 shutdown_connections 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.854+0000 7fb56a376640 1 -- 172.21.15.89:0/2010601717 wait complete. 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb56a376640 1 Processor -- start 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb56a376640 1 -- start start 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb56a376640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb564100a60 0x7fb56406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb56a376640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb564101c50 0x7fb56419cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb56a376640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb564102610 0x7fb56419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb56a376640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb56410c6e0 con 0x7fb564101c50 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb56a376640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb56410c560 con 0x7fb564100a60 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb56a376640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb56410c860 con 0x7fb564102610 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb5637fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb564101c50 0x7fb56419cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb5637fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb564101c50 0x7fb56419cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53446/0 (socket says 172.21.15.89:53446) 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb5637fe640 1 -- 172.21.15.89:0/2238456672 learned_addr learned my addr 172.21.15.89:0/2238456672 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.855+0000 7fb563fff640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb564100a60 0x7fb56406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb568913640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb564102610 0x7fb56419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb5637fe640 1 -- 172.21.15.89:0/2238456672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb564102610 msgr2=0x7fb56419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb5637fe640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb564102610 0x7fb56419f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb5637fe640 1 -- 172.21.15.89:0/2238456672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb564100a60 msgr2=0x7fb56406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb5637fe640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb564100a60 0x7fb56406ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb5637fe640 1 -- 172.21.15.89:0/2238456672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56419f9f0 con 0x7fb564101c50 2024-09-15T23:32:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb563fff640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb564100a60 0x7fb56406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb5637fe640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb564101c50 0x7fb56419cac0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fb55800d950 tx=0x7fb55800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb5617fa640 1 -- 172.21.15.89:0/2238456672 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558017070 con 0x7fb564101c50 2024-09-15T23:32:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb5617fa640 1 -- 172.21.15.89:0/2238456672 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb558009c30 con 0x7fb564101c50 2024-09-15T23:32:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb56a376640 1 -- 172.21.15.89:0/2238456672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56419fc80 con 0x7fb564101c50 2024-09-15T23:32:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb5617fa640 1 -- 172.21.15.89:0/2238456672 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55800f6b0 con 0x7fb564101c50 2024-09-15T23:32:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.856+0000 7fb56a376640 1 -- 172.21.15.89:0/2238456672 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5641a01d0 con 0x7fb564101c50 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.858+0000 7fb5617fa640 1 -- 172.21.15.89:0/2238456672 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb5580109d0 con 0x7fb564101c50 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb56d1ed740 1 Processor -- start 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb56d1ed740 1 -- start start 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a7ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a7fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 0x556c13a82830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556c139de2f0 con 0x556c13a7a9a0 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556c13986270 con 0x556c13a80370 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556c139da2b0 con 0x556c13a7b2e0 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb568913640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a7ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb568913640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a7ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53458/0 (socket says 172.21.15.89:53458) 2024-09-15T23:32:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb568913640 1 -- 172.21.15.89:0/1662414853 learned_addr learned my addr 172.21.15.89:0/1662414853 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb563fff640 1 --2- 172.21.15.89:0/1662414853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a7fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb5637fe640 1 --2- 172.21.15.89:0/1662414853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 0x556c13a82830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb568913640 1 -- 172.21.15.89:0/1662414853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 msgr2=0x556c13a7fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb568913640 1 --2- 172.21.15.89:0/1662414853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a7fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb568913640 1 -- 172.21.15.89:0/1662414853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 msgr2=0x556c13a82830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb568913640 1 --2- 172.21.15.89:0/1662414853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 0x556c13a82830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb5617fa640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb54c0a32f0 0x7fb54c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb568913640 1 -- 172.21.15.89:0/1662414853 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c13985340 con 0x556c13a7a9a0 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb5637fe640 1 --2- 172.21.15.89:0/1662414853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 0x556c13a82830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.859+0000 7fb5617fa640 1 -- 172.21.15.89:0/2238456672 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb5580cfa20 con 0x7fb564101c50 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.860+0000 7fb568913640 1 --2- 172.21.15.89:0/1662414853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a7ada0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb55c02ade0 tx=0x7fb55c005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fae3f25b2bb958 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.860+0000 7fb568913640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb54c0a32f0 0x7fb54c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.860+0000 7fb54a7fc640 1 -- 172.21.15.89:0/1662414853 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c04c070 con 0x556c13a7a9a0 2024-09-15T23:32:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.860+0000 7fb54a7fc640 1 -- 172.21.15.89:0/1662414853 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb55c036070 con 0x556c13a7a9a0 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.860+0000 7fb54a7fc640 1 -- 172.21.15.89:0/1662414853 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c02fcb0 con 0x556c13a7a9a0 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.860+0000 7fb56d1ed740 1 -- 172.21.15.89:0/1662414853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 msgr2=0x556c13a7ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.860+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/1662414853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a7ada0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb55c02ade0 tx=0x7fb55c005f70 comp rx=0 tx=0).stop 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.860+0000 7fb568913640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb54c0a32f0 0x7fb54c0a57b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009430 tx=0x7fb55c04a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 -- 172.21.15.89:0/1662414853 shutdown_connections 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/1662414853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 0x556c13a82830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/1662414853 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a7fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/1662414853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a7ada0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 -- 172.21.15.89:0/1662414853 >> 172.21.15.89:0/1662414853 conn(0x556c13a76010 msgr2=0x556c13a76410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 -- 172.21.15.89:0/1662414853 shutdown_connections 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 -- 172.21.15.89:0/1662414853 wait complete. 2024-09-15T23:32:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 -- start start 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 Processor -- start 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 0x556c13a84bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a85130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556c13a83340 con 0x556c13a80370 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556c13967290 con 0x556c13a7a9a0 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556c13a834c0 con 0x556c13a7b2e0 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb56d1ed740 10 client.0 _ll_register_callbacks cb 0x556c13998140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb568913640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.861+0000 7fb568913640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53464/0 (socket says 172.21.15.89:53464) 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb568913640 1 -- 172.21.15.89:0/3452106858 learned_addr learned my addr 172.21.15.89:0/3452106858 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb563fff640 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 0x556c13a84bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb5637fe640 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a85130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb568913640 1 -- 172.21.15.89:0/3452106858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 msgr2=0x556c13a85130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb568913640 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a85130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb568913640 1 -- 172.21.15.89:0/3452106858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 msgr2=0x556c13a84bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb568913640 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 0x556c13a84bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb568913640 1 -- 172.21.15.89:0/3452106858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c136f1210 con 0x556c13a80370 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb5637fe640 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a85130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb568913640 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8c170 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009780 tx=0x7fb55c02fdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f771c187bcee5944 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb5437fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb5437fe640 1 -- 172.21.15.89:0/3452106858 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c04c040 con 0x556c13a80370 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb5437fe640 1 -- 172.21.15.89:0/3452106858 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb55c004690 con 0x556c13a80370 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3452106858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb55c036070 con 0x556c13a80370 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb5437fe640 1 -- 172.21.15.89:0/3452106858 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb55c062050 con 0x556c13a80370 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb56d1ed740 10 client.18111 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:07.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3452106858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c137bd620 con 0x556c13a80370 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.862+0000 7fb5437fe640 1 -- 172.21.15.89:0/3452106858 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb55c036070 con 0x556c13a80370 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.863+0000 7fb5437fe640 1 -- 172.21.15.89:0/3452106858 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb55c042050 con 0x556c13a80370 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.863+0000 7fb56d1ed740 10 client.18111 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.863+0000 7fb56d1ed740 10 client.18111 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.863+0000 7fb56d1ed740 20 client.18111 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.863+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556c13a903e0 0x556c13a927d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.863+0000 7fb56d1ed740 4 client.18111 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.863+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3452106858 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c139eba20 con 0x556c13a903e0 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.863+0000 7fb563fff640 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556c13a903e0 0x556c13a927d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.863+0000 7fb5437fe640 10 client.18111 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.863+0000 7fb563fff640 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556c13a903e0 0x556c13a927d0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=430d622a3b1b2984 server_cookie=ebeee9bf74b1ddf in_seq=0 out_seq=0 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb5437fe640 1 -- 172.21.15.89:0/3452106858 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556c139eba20 con 0x556c13a903e0 2024-09-15T23:32:07.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb5437fe640 10 client.18111 handle_command_reply: tid=0 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 client.18111 shutdown 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3452106858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 msgr2=0x556c13a8c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8c170 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009780 tx=0x7fb55c02fdc0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3452106858 shutdown_connections 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556c13a903e0 0x556c13a927d0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8c170 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a85130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb5437fe640 0 client.18111 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/3452106858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 0x556c13a84bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3452106858 >> 172.21.15.89:0/3452106858 conn(0x556c13a76010 msgr2=0x556c13a77820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3452106858 shutdown_connections 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3452106858 wait complete. 2024-09-15T23:32:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.864+0000 7fb56d1ed740 20 client.18111 trim_cache size 0 max 16384 2024-09-15T23:32:07.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb56d1ed740 1 Processor -- start 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb56d1ed740 1 -- start start 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a76ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a8c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 0x556c13a77400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641aa820 con 0x556c13a7a9a0 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641ac360 con 0x556c13a80370 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641aaef0 con 0x556c13a7b2e0 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb563fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a76ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb563fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a76ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53468/0 (socket says 172.21.15.89:53468) 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb563fff640 1 -- 172.21.15.89:0/3009776858 learned_addr learned my addr 172.21.15.89:0/3009776858 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb5637fe640 1 --2- 172.21.15.89:0/3009776858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a8c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb568913640 1 --2- 172.21.15.89:0/3009776858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 0x556c13a77400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb563fff640 1 -- 172.21.15.89:0/3009776858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 msgr2=0x556c13a8c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.879+0000 7fb563fff640 1 --2- 172.21.15.89:0/3009776858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a8c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.880+0000 7fb563fff640 1 -- 172.21.15.89:0/3009776858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 msgr2=0x556c13a77400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.880+0000 7fb563fff640 1 --2- 172.21.15.89:0/3009776858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 0x556c13a77400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.880+0000 7fb563fff640 1 -- 172.21.15.89:0/3009776858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c13a7b6c0 con 0x556c13a7a9a0 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.880+0000 7fb563fff640 1 --2- 172.21.15.89:0/3009776858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a76ec0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x556c13a7bd10 tx=0x7fb554013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76b93e091c87052 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.880+0000 7fb549ffb640 1 -- 172.21.15.89:0/3009776858 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb554004030 con 0x556c13a7a9a0 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.880+0000 7fb549ffb640 1 -- 172.21.15.89:0/3009776858 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5540041d0 con 0x556c13a7a9a0 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.880+0000 7fb549ffb640 1 -- 172.21.15.89:0/3009776858 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5540116a0 con 0x556c13a7a9a0 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3009776858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 msgr2=0x556c13a76ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/3009776858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a76ec0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x556c13a7bd10 tx=0x7fb554013830 comp rx=0 tx=0).stop 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3009776858 shutdown_connections 2024-09-15T23:32:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/3009776858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a80370 0x556c13a77400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/3009776858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a8c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/3009776858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a7a9a0 0x556c13a76ec0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3009776858 >> 172.21.15.89:0/3009776858 conn(0x556c13a76010 msgr2=0x556c13a84bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3009776858 shutdown_connections 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 -- 172.21.15.89:0/3009776858 wait complete. 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 -- start start 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.881+0000 7fb56d1ed740 1 Processor -- start 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 0x556c13a899a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a89ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb56d1ed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641abc90 con 0x556c13a80370 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641ab5c0 con 0x556c13a7a9a0 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb56d1ed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb5641a8800 con 0x556c13a7b2e0 2024-09-15T23:32:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb56d1ed740 10 client.0 _ll_register_callbacks cb 0x7fb55c04a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb568913640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb568913640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53478/0 (socket says 172.21.15.89:53478) 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb568913640 1 -- 172.21.15.89:0/7084804 learned_addr learned my addr 172.21.15.89:0/7084804 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb5637fe640 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a89ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb563fff640 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 0x556c13a899a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb568913640 1 -- 172.21.15.89:0/7084804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 msgr2=0x556c13a89ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb568913640 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a89ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb568913640 1 -- 172.21.15.89:0/7084804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 msgr2=0x556c13a899a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb568913640 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 0x556c13a899a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb568913640 1 -- 172.21.15.89:0/7084804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c13a8b500 con 0x556c13a80370 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb563fff640 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 0x556c13a899a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb568913640 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8aef0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fb55c031ae0 tx=0x7fb55c031e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173ab6c5c28080b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb5437fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb5437fe640 1 -- 172.21.15.89:0/7084804 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c04c070 con 0x556c13a80370 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.882+0000 7fb5437fe640 1 -- 172.21.15.89:0/7084804 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb55c0307c0 con 0x556c13a80370 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb56d1ed740 1 -- 172.21.15.89:0/7084804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5540041d0 con 0x556c13a80370 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb5437fe640 1 -- 172.21.15.89:0/7084804 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb55c009800 con 0x556c13a80370 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb56d1ed740 10 client.18117 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb56d1ed740 1 -- 172.21.15.89:0/7084804 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c13afb110 con 0x556c13a80370 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb5437fe640 1 -- 172.21.15.89:0/7084804 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5540041d0 con 0x556c13a80370 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb5437fe640 1 -- 172.21.15.89:0/7084804 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb55c042030 con 0x556c13a80370 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb56d1ed740 10 client.18117 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb56d1ed740 10 client.18117 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb56d1ed740 20 client.18117 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556c13afb9b0 0x556c13afddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb56d1ed740 4 client.18117 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:32:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb56d1ed740 1 -- 172.21.15.89:0/7084804 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556c139e5d70 con 0x556c13afb9b0 2024-09-15T23:32:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.883+0000 7fb563fff640 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556c13afb9b0 0x556c13afddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.884+0000 7fb5437fe640 10 client.18117 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.884+0000 7fb563fff640 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556c13afb9b0 0x556c13afddd0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5ec9ec5aba905a1 server_cookie=952d447bc923a468 in_seq=0 out_seq=0 2024-09-15T23:32:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb5437fe640 1 -- 172.21.15.89:0/7084804 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556c139e5d70 con 0x556c13afb9b0 2024-09-15T23:32:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb5437fe640 10 client.18117 handle_command_reply: tid=0 2024-09-15T23:32:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 client.18117 shutdown 2024-09-15T23:32:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 -- 172.21.15.89:0/7084804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 msgr2=0x556c13a8aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8aef0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fb55c031ae0 tx=0x7fb55c031e50 comp rx=0 tx=0).stop 2024-09-15T23:32:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 -- 172.21.15.89:0/7084804 shutdown_connections 2024-09-15T23:32:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556c13afb9b0 0x556c13afddd0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556c13a80370 0x556c13a8aef0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb5437fe640 0 client.18117 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556c13a7b2e0 0x556c13a89ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 --2- 172.21.15.89:0/7084804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c13a7a9a0 0x556c13a899a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 -- 172.21.15.89:0/7084804 >> 172.21.15.89:0/7084804 conn(0x556c13a76010 msgr2=0x556c13a749a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 -- 172.21.15.89:0/7084804 shutdown_connections 2024-09-15T23:32:07.902 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 1 -- 172.21.15.89:0/7084804 wait complete. 2024-09-15T23:32:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.899+0000 7fb56d1ed740 20 client.18117 trim_cache size 0 max 16384 2024-09-15T23:32:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:32:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:32:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:32:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:32:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 -- 172.21.15.89:0/2238456672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb54c0a32f0 msgr2=0x7fb54c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb54c0a32f0 0x7fb54c0a57b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009430 tx=0x7fb55c04a000 comp rx=0 tx=0).stop 2024-09-15T23:32:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 -- 172.21.15.89:0/2238456672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb564101c50 msgr2=0x7fb56419cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb564101c50 0x7fb56419cac0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fb55800d950 tx=0x7fb55800de10 comp rx=0 tx=0).stop 2024-09-15T23:32:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 -- 172.21.15.89:0/2238456672 shutdown_connections 2024-09-15T23:32:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb54c0a32f0 0x7fb54c0a57b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb564102610 0x7fb56419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb564101c50 0x7fb56419cac0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 --2- 172.21.15.89:0/2238456672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb564100a60 0x7fb56406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 -- 172.21.15.89:0/2238456672 >> 172.21.15.89:0/2238456672 conn(0x7fb5640fc1e0 msgr2=0x7fb564104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 -- 172.21.15.89:0/2238456672 shutdown_connections 2024-09-15T23:32:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:07.900+0000 7fb56a376640 1 -- 172.21.15.89:0/2238456672 wait complete. 2024-09-15T23:32:07.910 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:32:08.157 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:08.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.638+0000 7f582b125640 1 -- 172.21.15.89:0/804177018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5824101420 msgr2=0x7f582410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:08.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.638+0000 7f582b125640 1 --2- 172.21.15.89:0/804177018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5824101420 0x7f582410d950 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f581c009cd0 tx=0x7f581c02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.638+0000 7f582b125640 1 -- 172.21.15.89:0/804177018 shutdown_connections 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.638+0000 7f582b125640 1 --2- 172.21.15.89:0/804177018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5824101420 0x7f582410d950 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.638+0000 7f582b125640 1 --2- 172.21.15.89:0/804177018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5824100a80 0x7f5824100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.638+0000 7f582b125640 1 --2- 172.21.15.89:0/804177018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5824106a80 0x7f5824106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.638+0000 7f582b125640 1 -- 172.21.15.89:0/804177018 >> 172.21.15.89:0/804177018 conn(0x7f58240fc7a0 msgr2=0x7f58240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.638+0000 7f582b125640 1 -- 172.21.15.89:0/804177018 shutdown_connections 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.638+0000 7f582b125640 1 -- 172.21.15.89:0/804177018 wait complete. 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f582b125640 1 Processor -- start 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f582b125640 1 -- start start 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f582b125640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5824100a80 0x7f5824196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f582b125640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5824101420 0x7f5824197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f582b125640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5824106a80 0x7f5824197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f582b125640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5824110e80 con 0x7f5824101420 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f582b125640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5824110d00 con 0x7f5824100a80 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f582b125640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5824111000 con 0x7f5824106a80 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f5823fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5824101420 0x7f5824197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f5823fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5824101420 0x7f5824197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53492/0 (socket says 172.21.15.89:53492) 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f5823fff640 1 -- 172.21.15.89:0/2693157316 learned_addr learned my addr 172.21.15.89:0/2693157316 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.639+0000 7f5828ec1640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5824100a80 0x7f5824196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f58296c2640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5824106a80 0x7f5824197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f5823fff640 1 -- 172.21.15.89:0/2693157316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5824106a80 msgr2=0x7f5824197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f5823fff640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5824106a80 0x7f5824197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f5823fff640 1 -- 172.21.15.89:0/2693157316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5824100a80 msgr2=0x7f5824196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:08.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f5823fff640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5824100a80 0x7f5824196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f5823fff640 1 -- 172.21.15.89:0/2693157316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58241a6850 con 0x7f5824101420 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f5828ec1640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5824100a80 0x7f5824196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f58296c2640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5824106a80 0x7f5824197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f5823fff640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5824101420 0x7f5824197520 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f581800c970 tx=0x7f581800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f5821ffb640 1 -- 172.21.15.89:0/2693157316 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5818007bb0 con 0x7f5824101420 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f582b125640 1 -- 172.21.15.89:0/2693157316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58241a6b40 con 0x7f5824101420 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f5821ffb640 1 -- 172.21.15.89:0/2693157316 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5818007d50 con 0x7f5824101420 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f5821ffb640 1 -- 172.21.15.89:0/2693157316 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581800f450 con 0x7f5824101420 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.640+0000 7f582b125640 1 -- 172.21.15.89:0/2693157316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58241a7030 con 0x7f5824101420 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.642+0000 7f5821ffb640 1 -- 172.21.15.89:0/2693157316 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5818005020 con 0x7f5824101420 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f582d4fe740 1 Processor -- start 2024-09-15T23:32:08.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f582d4fe740 1 -- start start 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd2b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f5821ffb640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f580c0a3240 0x7f580c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd2b5d0 0x56410dd2ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd36370 0x56410dd38760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f582d4fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56410dc912f0 con 0x56410dd2cb40 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f582d4fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56410dc38cf0 con 0x56410dd36370 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f582d4fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56410dc8d2b0 con 0x56410dd2b5d0 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f5828ec1640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f580c0a3240 0x7f580c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f5821ffb640 1 -- 172.21.15.89:0/2693157316 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f58180cfc00 con 0x7f5824101420 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f58296c2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd2b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f58296c2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd2b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53500/0 (socket says 172.21.15.89:53500) 2024-09-15T23:32:08.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f58296c2640 1 -- 172.21.15.89:0/504267096 learned_addr learned my addr 172.21.15.89:0/504267096 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:08.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f5823fff640 1 --2- 172.21.15.89:0/504267096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd2b5d0 0x56410dd2ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f58296c2640 1 --2- 172.21.15.89:0/504267096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd36370 0x56410dd38760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f58296c2640 1 -- 172.21.15.89:0/504267096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd2b5d0 msgr2=0x56410dd2ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:08.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f58296c2640 1 --2- 172.21.15.89:0/504267096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd2b5d0 0x56410dd2ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f58296c2640 1 -- 172.21.15.89:0/504267096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd36370 msgr2=0x56410dd38760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:08.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f58296c2640 1 --2- 172.21.15.89:0/504267096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd36370 0x56410dd38760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.643+0000 7f58296c2640 1 -- 172.21.15.89:0/504267096 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56410dc1d3a0 con 0x56410dd2cb40 2024-09-15T23:32:08.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f5828ec1640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f580c0a3240 0x7f580c0a5700 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5814005e10 tx=0x7f5814005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f58296c2640 1 --2- 172.21.15.89:0/504267096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd2b090 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f581c009430 tx=0x7f581c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c4b36a12accb46 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f580affd640 1 -- 172.21.15.89:0/504267096 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581c031c00 con 0x56410dd2cb40 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f580affd640 1 -- 172.21.15.89:0/504267096 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f581c036040 con 0x56410dd2cb40 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f580affd640 1 -- 172.21.15.89:0/504267096 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581c0305a0 con 0x56410dd2cb40 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f582d4fe740 1 -- 172.21.15.89:0/504267096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 msgr2=0x56410dd2b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f582d4fe740 1 --2- 172.21.15.89:0/504267096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd2b090 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f581c009430 tx=0x7f581c005ef0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f582d4fe740 1 -- 172.21.15.89:0/504267096 shutdown_connections 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f582d4fe740 1 --2- 172.21.15.89:0/504267096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd36370 0x56410dd38760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f582d4fe740 1 --2- 172.21.15.89:0/504267096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd2b5d0 0x56410dd2ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f582d4fe740 1 --2- 172.21.15.89:0/504267096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd2b090 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.644+0000 7f582d4fe740 1 -- 172.21.15.89:0/504267096 >> 172.21.15.89:0/504267096 conn(0x56410dcf4310 msgr2=0x56410dd27c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 1 -- 172.21.15.89:0/504267096 shutdown_connections 2024-09-15T23:32:08.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 1 -- 172.21.15.89:0/504267096 wait complete. 2024-09-15T23:32:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 1 -- start start 2024-09-15T23:32:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 1 Processor -- start 2024-09-15T23:32:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2b5d0 0x56410dd35d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2cb40 0x56410dd32fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410dd33530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56410dd39010 con 0x56410dd2b5d0 2024-09-15T23:32:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56410dc4c060 con 0x56410dd2cb40 2024-09-15T23:32:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56410dd39190 con 0x56410dd36370 2024-09-15T23:32:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f582d4fe740 10 client.0 _ll_register_callbacks cb 0x56410d99f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:08.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f5828ec1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2b5d0 0x56410dd35d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f5828ec1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2b5d0 0x56410dd35d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53506/0 (socket says 172.21.15.89:53506) 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.645+0000 7f5828ec1640 1 -- 172.21.15.89:0/4224012807 learned_addr learned my addr 172.21.15.89:0/4224012807 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f58296c2640 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410dd33530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5823fff640 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2cb40 0x56410dd32fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5828ec1640 1 -- 172.21.15.89:0/4224012807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 msgr2=0x56410dd33530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5828ec1640 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410dd33530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5828ec1640 1 -- 172.21.15.89:0/4224012807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2cb40 msgr2=0x56410dd32fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5828ec1640 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2cb40 0x56410dd32fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5828ec1640 1 -- 172.21.15.89:0/4224012807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56410da023c0 con 0x56410dd2b5d0 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f58296c2640 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410dd33530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:08.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5823fff640 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2cb40 0x56410dd32fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5828ec1640 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2b5d0 0x56410dd35d00 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5814005880 tx=0x7f58140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8319fb943173450 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5803fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5803fff640 1 -- 172.21.15.89:0/4224012807 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5814021070 con 0x56410dd2b5d0 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5803fff640 1 -- 172.21.15.89:0/4224012807 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f581400fc50 con 0x56410dd2b5d0 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f582d4fe740 1 -- 172.21.15.89:0/4224012807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f581c036040 con 0x56410dd2b5d0 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5803fff640 1 -- 172.21.15.89:0/4224012807 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f581402d050 con 0x56410dd2b5d0 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f582d4fe740 10 client.18129 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f582d4fe740 1 -- 172.21.15.89:0/4224012807 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56410dd44aa0 con 0x56410dd2b5d0 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.646+0000 7f5803fff640 1 -- 172.21.15.89:0/4224012807 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f581c036040 con 0x56410dd2b5d0 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.647+0000 7f5803fff640 1 -- 172.21.15.89:0/4224012807 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5814025440 con 0x56410dd2b5d0 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.647+0000 7f582d4fe740 10 client.18129 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.647+0000 7f582d4fe740 10 client.18129 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.647+0000 7f582d4fe740 20 client.18129 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:08.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.647+0000 7f582d4fe740 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x56410dd45eb0 0x56410dd48310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.647+0000 7f582d4fe740 4 client.18129 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.647+0000 7f582d4fe740 1 -- 172.21.15.89:0/4224012807 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56410dc9e860 con 0x56410dd45eb0 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.647+0000 7f5823fff640 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x56410dd45eb0 0x56410dd48310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.647+0000 7f5823fff640 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x56410dd45eb0 0x56410dd48310 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d3aa054bafde93ed server_cookie=e44e977b363f76e0 in_seq=0 out_seq=0 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.647+0000 7f5803fff640 10 client.18129 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f5803fff640 1 -- 172.21.15.89:0/4224012807 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56410dc9e860 con 0x56410dd45eb0 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f5803fff640 10 client.18129 handle_command_reply: tid=0 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 client.18129 shutdown 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 -- 172.21.15.89:0/4224012807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2b5d0 msgr2=0x56410dd35d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2b5d0 0x56410dd35d00 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5814005880 tx=0x7f58140046c0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 -- 172.21.15.89:0/4224012807 shutdown_connections 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x56410dd45eb0 0x56410dd48310 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410dd33530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2cb40 0x56410dd32fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f5803fff640 0 client.18129 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:32:08.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 --2- 172.21.15.89:0/4224012807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2b5d0 0x56410dd35d00 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 -- 172.21.15.89:0/4224012807 >> 172.21.15.89:0/4224012807 conn(0x56410dcf4310 msgr2=0x56410dd2d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:08.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 -- 172.21.15.89:0/4224012807 shutdown_connections 2024-09-15T23:32:08.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 1 -- 172.21.15.89:0/4224012807 wait complete. 2024-09-15T23:32:08.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.648+0000 7f582d4fe740 20 client.18129 trim_cache size 0 max 16384 2024-09-15T23:32:08.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f582d4fe740 1 Processor -- start 2024-09-15T23:32:08.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f582d4fe740 1 -- start start 2024-09-15T23:32:08.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 0x56410dd422b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd34f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410dd35460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f582d4fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f58241b0fe0 con 0x56410dd2cb40 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f582d4fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58241b2b20 con 0x56410dd2b5d0 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f582d4fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58241b16b0 con 0x56410dd36370 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f5823fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd34f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f5828ec1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 0x56410dd422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f5823fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd34f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53520/0 (socket says 172.21.15.89:53520) 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f5823fff640 1 -- 172.21.15.89:0/4223585839 learned_addr learned my addr 172.21.15.89:0/4223585839 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.662+0000 7f58296c2640 1 --2- 172.21.15.89:0/4223585839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410dd35460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.663+0000 7f5823fff640 1 -- 172.21.15.89:0/4223585839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 msgr2=0x56410dd35460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.663+0000 7f5823fff640 1 --2- 172.21.15.89:0/4223585839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410dd35460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.663+0000 7f5823fff640 1 -- 172.21.15.89:0/4223585839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 msgr2=0x56410dd422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.663+0000 7f5823fff640 1 --2- 172.21.15.89:0/4223585839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 0x56410dd422b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.663+0000 7f5823fff640 1 -- 172.21.15.89:0/4223585839 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56410dd28960 con 0x56410dd2cb40 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.663+0000 7f58296c2640 1 --2- 172.21.15.89:0/4223585839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410dd35460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.663+0000 7f5823fff640 1 --2- 172.21.15.89:0/4223585839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd34f20 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f581809bd60 tx=0x7f581800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81963aca5584acdb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.663+0000 7f580a7fc640 1 -- 172.21.15.89:0/4223585839 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581801d040 con 0x56410dd2cb40 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.663+0000 7f580a7fc640 1 -- 172.21.15.89:0/4223585839 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58180b0070 con 0x56410dd2cb40 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.663+0000 7f580a7fc640 1 -- 172.21.15.89:0/4223585839 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581809a0d0 con 0x56410dd2cb40 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 -- 172.21.15.89:0/4223585839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 msgr2=0x56410dd34f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 --2- 172.21.15.89:0/4223585839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd34f20 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f581809bd60 tx=0x7f581800cf60 comp rx=0 tx=0).stop 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 -- 172.21.15.89:0/4223585839 shutdown_connections 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 --2- 172.21.15.89:0/4223585839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410dd35460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 --2- 172.21.15.89:0/4223585839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd34f20 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 --2- 172.21.15.89:0/4223585839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 0x56410dd422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 -- 172.21.15.89:0/4223585839 >> 172.21.15.89:0/4223585839 conn(0x56410dcf4310 msgr2=0x56410dd34810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 -- 172.21.15.89:0/4223585839 shutdown_connections 2024-09-15T23:32:08.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 -- 172.21.15.89:0/4223585839 wait complete. 2024-09-15T23:32:08.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 -- start start 2024-09-15T23:32:08.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.664+0000 7f582d4fe740 1 Processor -- start 2024-09-15T23:32:08.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 0x56410dd3ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd3b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f582d4fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410ddb13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f582d4fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f58241b2450 con 0x56410dd2cb40 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f582d4fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58241b1d80 con 0x56410dd2b5d0 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f582d4fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58241aefc0 con 0x56410dd36370 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f582d4fe740 10 client.0 _ll_register_callbacks cb 0x7f5814014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5823fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd3b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5823fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd3b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53524/0 (socket says 172.21.15.89:53524) 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5823fff640 1 -- 172.21.15.89:0/3343093706 learned_addr learned my addr 172.21.15.89:0/3343093706 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5828ec1640 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 0x56410dd3ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f58296c2640 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410ddb13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5823fff640 1 -- 172.21.15.89:0/3343093706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 msgr2=0x56410ddb13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5823fff640 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410ddb13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5823fff640 1 -- 172.21.15.89:0/3343093706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 msgr2=0x56410dd3ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:08.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5823fff640 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 0x56410dd3ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5823fff640 1 -- 172.21.15.89:0/3343093706 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56410ddb1be0 con 0x56410dd2cb40 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f58296c2640 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410ddb13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5828ec1640 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 0x56410dd3ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5823fff640 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd3b2d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f581809bad0 tx=0x7f58180a4a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a1bcef46709780 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5803fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5803fff640 1 -- 172.21.15.89:0/3343093706 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581801d040 con 0x56410dd2cb40 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5803fff640 1 -- 172.21.15.89:0/3343093706 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f58180cd3c0 con 0x56410dd2cb40 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f582d4fe740 1 -- 172.21.15.89:0/3343093706 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56410dc98d70 con 0x56410dd2cb40 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f5803fff640 1 -- 172.21.15.89:0/3343093706 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f58180b4040 con 0x56410dd2cb40 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.665+0000 7f582d4fe740 10 client.18135 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.666+0000 7f582d4fe740 1 -- 172.21.15.89:0/3343093706 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56410ddb9f30 con 0x56410dd2cb40 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.666+0000 7f5803fff640 1 -- 172.21.15.89:0/3343093706 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56410dc98d70 con 0x56410dd2cb40 2024-09-15T23:32:08.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.666+0000 7f5803fff640 1 -- 172.21.15.89:0/3343093706 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f58180b0a20 con 0x56410dd2cb40 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.666+0000 7f582d4fe740 10 client.18135 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.666+0000 7f582d4fe740 10 client.18135 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.666+0000 7f582d4fe740 20 client.18135 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.666+0000 7f582d4fe740 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x56410ddba550 0x56410ddbc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.666+0000 7f582d4fe740 4 client.18135 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.666+0000 7f582d4fe740 1 -- 172.21.15.89:0/3343093706 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56410dc15170 con 0x56410ddba550 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.666+0000 7f5828ec1640 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x56410ddba550 0x56410ddbc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f5803fff640 10 client.18135 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f5828ec1640 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x56410ddba550 0x56410ddbc970 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=20b7bc407a71cf5b server_cookie=3dc6d03598049447 in_seq=0 out_seq=0 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f5803fff640 1 -- 172.21.15.89:0/3343093706 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56410dc15170 con 0x56410ddba550 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f5803fff640 10 client.18135 handle_command_reply: tid=0 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 client.18135 shutdown 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 -- 172.21.15.89:0/3343093706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 msgr2=0x56410dd3b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd3b2d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f581809bad0 tx=0x7f58180a4a40 comp rx=0 tx=0).stop 2024-09-15T23:32:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 -- 172.21.15.89:0/3343093706 shutdown_connections 2024-09-15T23:32:08.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x56410ddba550 0x56410ddbc970 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56410dd36370 0x56410ddb13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56410dd2cb40 0x56410dd3b2d0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f5803fff640 0 client.18135 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:32:08.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 --2- 172.21.15.89:0/3343093706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56410dd2b5d0 0x56410dd3ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 -- 172.21.15.89:0/3343093706 >> 172.21.15.89:0/3343093706 conn(0x56410dcf4310 msgr2=0x56410dd34810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:08.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 -- 172.21.15.89:0/3343093706 shutdown_connections 2024-09-15T23:32:08.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 1 -- 172.21.15.89:0/3343093706 wait complete. 2024-09-15T23:32:08.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.667+0000 7f582d4fe740 20 client.18135 trim_cache size 0 max 16384 2024-09-15T23:32:08.684 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:32:08.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 -- 172.21.15.89:0/2693157316 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f580c0a3240 msgr2=0x7f580c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:08.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f580c0a3240 0x7f580c0a5700 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5814005e10 tx=0x7f5814005da0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 -- 172.21.15.89:0/2693157316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5824101420 msgr2=0x7f5824197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:08.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5824101420 0x7f5824197520 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f581800c970 tx=0x7f581800ce30 comp rx=0 tx=0).stop 2024-09-15T23:32:08.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 -- 172.21.15.89:0/2693157316 shutdown_connections 2024-09-15T23:32:08.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f580c0a3240 0x7f580c0a5700 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5824106a80 0x7f5824197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5824101420 0x7f5824197520 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 --2- 172.21.15.89:0/2693157316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5824100a80 0x7f5824196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:08.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 -- 172.21.15.89:0/2693157316 >> 172.21.15.89:0/2693157316 conn(0x7f58240fc7a0 msgr2=0x7f58240fcd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:08.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 -- 172.21.15.89:0/2693157316 shutdown_connections 2024-09-15T23:32:08.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:08.668+0000 7f582b125640 1 -- 172.21.15.89:0/2693157316 wait complete. 2024-09-15T23:32:09.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:09 smithi089 ceph-mon[30933]: pgmap v594: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-09-15T23:32:09.385 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:32:09.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:09 smithi174 ceph-mon[35586]: pgmap v594: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:09 smithi120 ceph-mon[35250]: pgmap v594: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.6 KiB/s wr, 11 op/s 2024-09-15T23:32:10.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.092+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/2676704086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d18106f20 msgr2=0x7f0d1810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:10.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.092+0000 7f0d1d2ad640 1 --2- 172.21.15.89:0/2676704086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d18106f20 0x7f0d1810eaa0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009900 tx=0x7f0d1002a410 comp rx=0 tx=0).stop 2024-09-15T23:32:10.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.092+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/2676704086 shutdown_connections 2024-09-15T23:32:10.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.092+0000 7f0d1d2ad640 1 --2- 172.21.15.89:0/2676704086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d18106f20 0x7f0d1810eaa0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.092+0000 7f0d1d2ad640 1 --2- 172.21.15.89:0/2676704086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d181027a0 0x7f0d181067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.092+0000 7f0d1d2ad640 1 --2- 172.21.15.89:0/2676704086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d18101df0 0x7f0d181021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.092+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/2676704086 >> 172.21.15.89:0/2676704086 conn(0x7f0d180fd8b0 msgr2=0x7f0d180ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.092+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/2676704086 shutdown_connections 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.093+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/2676704086 wait complete. 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.093+0000 7f0d1d2ad640 1 Processor -- start 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.093+0000 7f0d1d2ad640 1 -- start start 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d1d2ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d18101df0 0x7f0d18198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d1d2ad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d181027a0 0x7f0d181988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d1d2ad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d18106f20 0x7f0d1819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d1d2ad640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d18111fa0 con 0x7f0d181027a0 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d1d2ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0d18111e20 con 0x7f0d18101df0 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d1d2ad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0d18112120 con 0x7f0d18106f20 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d167fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d181027a0 0x7f0d181988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d167fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d181027a0 0x7f0d181988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53548/0 (socket says 172.21.15.89:53548) 2024-09-15T23:32:10.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d167fc640 1 -- 172.21.15.89:0/548742341 learned_addr learned my addr 172.21.15.89:0/548742341 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d177fe640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d18106f20 0x7f0d1819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d16ffd640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d18101df0 0x7f0d18198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d167fc640 1 -- 172.21.15.89:0/548742341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d18106f20 msgr2=0x7f0d1819cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d167fc640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d18106f20 0x7f0d1819cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d167fc640 1 -- 172.21.15.89:0/548742341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d18101df0 msgr2=0x7f0d18198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d167fc640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d18101df0 0x7f0d18198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d167fc640 1 -- 172.21.15.89:0/548742341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1819d330 con 0x7f0d181027a0 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d16ffd640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d18101df0 0x7f0d18198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.094+0000 7f0d167fc640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d181027a0 0x7f0d181988c0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000b4a0 tx=0x7f0d0000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.095+0000 7f0d0ffff640 1 -- 172.21.15.89:0/548742341 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00014070 con 0x7f0d181027a0 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.095+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/548742341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d1819d620 con 0x7f0d181027a0 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.095+0000 7f0d0ffff640 1 -- 172.21.15.89:0/548742341 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d00010af0 con 0x7f0d181027a0 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.095+0000 7f0d0ffff640 1 -- 172.21.15.89:0/548742341 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00017d10 con 0x7f0d181027a0 2024-09-15T23:32:10.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.095+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/548742341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d181a4f00 con 0x7f0d181027a0 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.097+0000 7f0d0ffff640 1 -- 172.21.15.89:0/548742341 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0d00017070 con 0x7f0d181027a0 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.097+0000 7f0d1f686740 1 Processor -- start 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.097+0000 7f0d1f686740 1 -- start start 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b392641b40 0x55b392640090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3926405d0 0x55b392640a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d0ffff640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d040a3240 0x7f0d040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b39264d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d1f686740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b3925a62f0 con 0x55b39264b370 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d1f686740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b39254dcf0 con 0x55b392641b40 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d1f686740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b3925a22b0 con 0x55b3926405d0 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d167fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b39264d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d177fe640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d040a3240 0x7f0d040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d16ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3926405d0 0x55b392640a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d16ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3926405d0 0x55b392640a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:53954/0 (socket says 172.21.15.89:53954) 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d16ffd640 1 -- 172.21.15.89:0/3325546465 learned_addr learned my addr 172.21.15.89:0/3325546465 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d0ffff640 1 -- 172.21.15.89:0/548742341 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0d000cf0d0 con 0x7f0d181027a0 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d177fe640 1 --2- 172.21.15.89:0/3325546465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b392641b40 0x55b392640090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d167fc640 1 -- 172.21.15.89:0/3325546465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3926405d0 msgr2=0x55b392640a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d167fc640 1 --2- 172.21.15.89:0/3325546465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3926405d0 0x55b392640a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d167fc640 1 -- 172.21.15.89:0/3325546465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b392641b40 msgr2=0x55b392640090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d167fc640 1 --2- 172.21.15.89:0/3325546465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b392641b40 0x55b392640090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d167fc640 1 -- 172.21.15.89:0/3325546465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3925323a0 con 0x55b39264b370 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d177fe640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d040a3240 0x7f0d040a5700 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0d10031550 tx=0x7f0d10032cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d167fc640 1 --2- 172.21.15.89:0/3325546465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b39264d760 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f0d000cc830 tx=0x7f0d000047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc70031a69fad75a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:10.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.098+0000 7f0d0d7fa640 1 -- 172.21.15.89:0/3325546465 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0009ac60 con 0x55b39264b370 2024-09-15T23:32:10.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d0d7fa640 1 -- 172.21.15.89:0/3325546465 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d0009ba00 con 0x55b39264b370 2024-09-15T23:32:10.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d0d7fa640 1 -- 172.21.15.89:0/3325546465 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00098950 con 0x55b39264b370 2024-09-15T23:32:10.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d1f686740 1 -- 172.21.15.89:0/3325546465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 msgr2=0x55b39264d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:10.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d1f686740 1 --2- 172.21.15.89:0/3325546465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b39264d760 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f0d000cc830 tx=0x7f0d000047d0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d1f686740 1 -- 172.21.15.89:0/3325546465 shutdown_connections 2024-09-15T23:32:10.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d1f686740 1 --2- 172.21.15.89:0/3325546465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b39264d760 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d1f686740 1 --2- 172.21.15.89:0/3325546465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3926405d0 0x55b392640a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d1f686740 1 --2- 172.21.15.89:0/3325546465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b392641b40 0x55b392640090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d1f686740 1 -- 172.21.15.89:0/3325546465 >> 172.21.15.89:0/3325546465 conn(0x55b392609310 msgr2=0x55b39263cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:10.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d1f686740 1 -- 172.21.15.89:0/3325546465 shutdown_connections 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.099+0000 7f0d1f686740 1 -- 172.21.15.89:0/3325546465 wait complete. 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d1f686740 1 -- start start 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d1f686740 1 Processor -- start 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 0x55b39264add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392648080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b392648810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d1f686740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b39264e070 con 0x55b39264b370 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d1f686740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b392561060 con 0x55b3926405d0 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d1f686740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b39264e1f0 con 0x55b392641b40 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d1f686740 10 client.0 _ll_register_callbacks cb 0x55b3922b48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d177fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b392648810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d177fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b392648810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53570/0 (socket says 172.21.15.89:53570) 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d177fe640 1 -- 172.21.15.89:0/4019319593 learned_addr learned my addr 172.21.15.89:0/4019319593 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d16ffd640 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 0x55b39264add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d167fc640 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392648080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d177fe640 1 -- 172.21.15.89:0/4019319593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 msgr2=0x55b392648080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d177fe640 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392648080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d177fe640 1 -- 172.21.15.89:0/4019319593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 msgr2=0x55b39264add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d177fe640 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 0x55b39264add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.100+0000 7f0d177fe640 1 -- 172.21.15.89:0/4019319593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3923173c0 con 0x55b39264b370 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0d177fe640 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b392648810 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f0d10033000 tx=0x7f0d10046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c9b51663f9e88a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0cfe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/4019319593 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1004f030 con 0x55b39264b370 2024-09-15T23:32:10.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0d1f686740 1 -- 172.21.15.89:0/4019319593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d0009ba00 con 0x55b39264b370 2024-09-15T23:32:10.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/4019319593 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0d10005280 con 0x55b39264b370 2024-09-15T23:32:10.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/4019319593 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0d1005c050 con 0x55b39264b370 2024-09-15T23:32:10.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0d1f686740 10 client.18147 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:10.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0d1f686740 1 -- 172.21.15.89:0/4019319593 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b392659b90 con 0x55b39264b370 2024-09-15T23:32:10.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/4019319593 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d0009ba00 con 0x55b39264b370 2024-09-15T23:32:10.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/4019319593 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0d10003680 con 0x55b39264b370 2024-09-15T23:32:10.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0d1f686740 10 client.18147 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0d1f686740 10 client.18147 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0d1f686740 20 client.18147 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0d1f686740 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b39265afa0 0x55b39265d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0d1f686740 4 client.18147 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.101+0000 7f0d1f686740 1 -- 172.21.15.89:0/4019319593 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3925b3860 con 0x55b39265afa0 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.102+0000 7f0d16ffd640 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b39265afa0 0x55b39265d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.102+0000 7f0cfe7fc640 10 client.18147 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.102+0000 7f0d16ffd640 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b39265afa0 0x55b39265d400 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2cadc9d1d8860e6 server_cookie=ef964c1b2a6798b2 in_seq=0 out_seq=0 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/4019319593 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b3925b3860 con 0x55b39265afa0 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0cfe7fc640 10 client.18147 handle_command_reply: tid=0 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 client.18147 shutdown 2024-09-15T23:32:10.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 -- 172.21.15.89:0/4019319593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 msgr2=0x55b392648810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b392648810 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f0d10033000 tx=0x7f0d10046000 comp rx=0 tx=0).stop 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 -- 172.21.15.89:0/4019319593 shutdown_connections 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b392648810 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b39265afa0 0x55b39265d400 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392648080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0cfe7fc640 0 client.18147 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 --2- 172.21.15.89:0/4019319593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 0x55b39264add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 -- 172.21.15.89:0/4019319593 >> 172.21.15.89:0/4019319593 conn(0x55b392609310 msgr2=0x55b392642990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 -- 172.21.15.89:0/4019319593 shutdown_connections 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 1 -- 172.21.15.89:0/4019319593 wait complete. 2024-09-15T23:32:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.103+0000 7f0d1f686740 20 client.18147 trim_cache size 0 max 16384 2024-09-15T23:32:10.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d1f686740 1 Processor -- start 2024-09-15T23:32:10.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d1f686740 1 -- start start 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 0x55b39264d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392643180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b3926436c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d1f686740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d181aee80 con 0x55b39264b370 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d1f686740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0d181b09c0 con 0x55b3926405d0 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d1f686740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0d181af550 con 0x55b392641b40 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d177fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b3926436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d177fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b3926436c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53580/0 (socket says 172.21.15.89:53580) 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d177fe640 1 -- 172.21.15.89:0/1159518524 learned_addr learned my addr 172.21.15.89:0/1159518524 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d16ffd640 1 --2- 172.21.15.89:0/1159518524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 0x55b39264d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d167fc640 1 --2- 172.21.15.89:0/1159518524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392643180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d177fe640 1 -- 172.21.15.89:0/1159518524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 msgr2=0x55b392643180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d177fe640 1 --2- 172.21.15.89:0/1159518524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392643180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d177fe640 1 -- 172.21.15.89:0/1159518524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 msgr2=0x55b39264d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.116+0000 7f0d177fe640 1 --2- 172.21.15.89:0/1159518524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 0x55b39264d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.117+0000 7f0d177fe640 1 -- 172.21.15.89:0/1159518524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b392657310 con 0x55b39264b370 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.117+0000 7f0d16ffd640 1 --2- 172.21.15.89:0/1159518524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 0x55b39264d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.117+0000 7f0d177fe640 1 --2- 172.21.15.89:0/1159518524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b3926436c0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f0d1002a930 tx=0x7f0d10032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a3c716e43a149d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.117+0000 7f0d0cff9640 1 -- 172.21.15.89:0/1159518524 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10058020 con 0x55b39264b370 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.117+0000 7f0d0cff9640 1 -- 172.21.15.89:0/1159518524 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d10003cc0 con 0x55b39264b370 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.117+0000 7f0d0cff9640 1 -- 172.21.15.89:0/1159518524 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1004f040 con 0x55b39264b370 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 -- 172.21.15.89:0/1159518524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 msgr2=0x55b3926436c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 --2- 172.21.15.89:0/1159518524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b3926436c0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f0d1002a930 tx=0x7f0d10032e90 comp rx=0 tx=0).stop 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 -- 172.21.15.89:0/1159518524 shutdown_connections 2024-09-15T23:32:10.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 --2- 172.21.15.89:0/1159518524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b39264b370 0x55b3926436c0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 --2- 172.21.15.89:0/1159518524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392643180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 --2- 172.21.15.89:0/1159518524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3926405d0 0x55b39264d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 -- 172.21.15.89:0/1159518524 >> 172.21.15.89:0/1159518524 conn(0x55b392609310 msgr2=0x55b392637a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 -- 172.21.15.89:0/1159518524 shutdown_connections 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 -- 172.21.15.89:0/1159518524 wait complete. 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 -- start start 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 Processor -- start 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3926405d0 0x55b392653d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392644f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b39264b370 0x55b3926456a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0d181b02f0 con 0x55b3926405d0 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0d181afc20 con 0x55b39264b370 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0d181ace60 con 0x55b392641b40 2024-09-15T23:32:10.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d1f686740 10 client.0 _ll_register_callbacks cb 0x7f0d10032f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d16ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3926405d0 0x55b392653d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d16ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3926405d0 0x55b392653d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53584/0 (socket says 172.21.15.89:53584) 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.118+0000 7f0d16ffd640 1 -- 172.21.15.89:0/2925036253 learned_addr learned my addr 172.21.15.89:0/2925036253 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d167fc640 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392644f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d177fe640 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b39264b370 0x55b3926456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d16ffd640 1 -- 172.21.15.89:0/2925036253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 msgr2=0x55b392644f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d16ffd640 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392644f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d16ffd640 1 -- 172.21.15.89:0/2925036253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b39264b370 msgr2=0x55b3926456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d16ffd640 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b39264b370 0x55b3926456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d16ffd640 1 -- 172.21.15.89:0/2925036253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3926c4cf0 con 0x55b3926405d0 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d177fe640 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b39264b370 0x55b3926456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d16ffd640 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3926405d0 0x55b392653d90 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x55b39265dc40 tx=0x7f0d08004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a80095f0dd4433 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:10.131 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0cfe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/2925036253 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08004d20 con 0x55b3926405d0 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/2925036253 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0d0801e700 con 0x55b3926405d0 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d1f686740 1 -- 172.21.15.89:0/2925036253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d10003cc0 con 0x55b3926405d0 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/2925036253 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0d0802c050 con 0x55b3926405d0 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d1f686740 10 client.18153 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0d1f686740 1 -- 172.21.15.89:0/2925036253 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b3926c5eb0 con 0x55b3926405d0 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.119+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/2925036253 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d10003cc0 con 0x55b3926405d0 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/2925036253 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0d08025060 con 0x55b3926405d0 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0d1f686740 10 client.18153 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0d1f686740 10 client.18153 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0d1f686740 20 client.18153 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:10.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0d1f686740 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b3926c6600 0x55b3926c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0d1f686740 4 client.18153 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0d1f686740 1 -- 172.21.15.89:0/2925036253 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b3925add70 con 0x55b3926c6600 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0d167fc640 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b3926c6600 0x55b3926c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0cfe7fc640 10 client.18153 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0d167fc640 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b3926c6600 0x55b3926c8a20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=babb6f93e449c99 server_cookie=a1d0630a68ed720f in_seq=0 out_seq=0 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0cfe7fc640 1 -- 172.21.15.89:0/2925036253 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b3925add70 con 0x55b3926c6600 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0cfe7fc640 10 client.18153 handle_command_reply: tid=0 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.120+0000 7f0d1f686740 1 client.18153 shutdown 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 1 -- 172.21.15.89:0/2925036253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3926405d0 msgr2=0x55b392653d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3926405d0 0x55b392653d90 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x55b39265dc40 tx=0x7f0d08004410 comp rx=0 tx=0).stop 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 1 -- 172.21.15.89:0/2925036253 shutdown_connections 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b39264b370 0x55b3926456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55b3926c6600 0x55b3926c8a20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b392641b40 0x55b392644f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 1 --2- 172.21.15.89:0/2925036253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3926405d0 0x55b392653d90 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 1 -- 172.21.15.89:0/2925036253 >> 172.21.15.89:0/2925036253 conn(0x55b392609310 msgr2=0x55b3926498e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:10.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0cfe7fc640 0 client.18153 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:32:10.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 1 -- 172.21.15.89:0/2925036253 shutdown_connections 2024-09-15T23:32:10.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 1 -- 172.21.15.89:0/2925036253 wait complete. 2024-09-15T23:32:10.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1f686740 20 client.18153 trim_cache size 0 max 16384 2024-09-15T23:32:10.134 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:32:10.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/548742341 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d040a3240 msgr2=0x7f0d040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:10.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1d2ad640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d040a3240 0x7f0d040a5700 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0d10031550 tx=0x7f0d10032cc0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/548742341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d181027a0 msgr2=0x7f0d181988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:10.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1d2ad640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d181027a0 0x7f0d181988c0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f0d0000b4a0 tx=0x7f0d0000b960 comp rx=0 tx=0).stop 2024-09-15T23:32:10.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/548742341 shutdown_connections 2024-09-15T23:32:10.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1d2ad640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0d040a3240 0x7f0d040a5700 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1d2ad640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d18106f20 0x7f0d1819cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1d2ad640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d181027a0 0x7f0d181988c0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1d2ad640 1 --2- 172.21.15.89:0/548742341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d18101df0 0x7f0d18198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:10.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.121+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/548742341 >> 172.21.15.89:0/548742341 conn(0x7f0d180fd8b0 msgr2=0x7f0d180fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:10.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.122+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/548742341 shutdown_connections 2024-09-15T23:32:10.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:10.122+0000 7f0d1d2ad640 1 -- 172.21.15.89:0/548742341 wait complete. 2024-09-15T23:32:10.779 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:32:11.026 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:11.446 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:11 smithi089 ceph-mon[30933]: pgmap v595: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-15T23:32:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:11 smithi174 ceph-mon[35586]: pgmap v595: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.526+0000 7f7ee0a86640 1 -- 172.21.15.89:0/2908076150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7edc101420 msgr2=0x7f7edc10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.526+0000 7f7ee0a86640 1 --2- 172.21.15.89:0/2908076150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7edc101420 0x7f7edc10d950 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009cd0 tx=0x7f7ed402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.526+0000 7f7ee0a86640 1 -- 172.21.15.89:0/2908076150 shutdown_connections 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.526+0000 7f7ee0a86640 1 --2- 172.21.15.89:0/2908076150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7edc101420 0x7f7edc10d950 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.526+0000 7f7ee0a86640 1 --2- 172.21.15.89:0/2908076150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7edc100a80 0x7f7edc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.526+0000 7f7ee0a86640 1 --2- 172.21.15.89:0/2908076150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc106a80 0x7f7edc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.526+0000 7f7ee0a86640 1 -- 172.21.15.89:0/2908076150 >> 172.21.15.89:0/2908076150 conn(0x7f7edc0fc7a0 msgr2=0x7f7edc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.526+0000 7f7ee0a86640 1 -- 172.21.15.89:0/2908076150 shutdown_connections 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.526+0000 7f7ee0a86640 1 -- 172.21.15.89:0/2908076150 wait complete. 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ee0a86640 1 Processor -- start 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ee0a86640 1 -- start start 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ee0a86640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc100a80 0x7f7edc19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ee0a86640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7edc101420 0x7f7edc19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ee0a86640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7edc106a80 0x7f7edc19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ee0a86640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc111fa0 con 0x7f7edc101420 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ee0a86640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc111e20 con 0x7f7edc106a80 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ee0a86640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc112120 con 0x7f7edc100a80 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ed9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7edc101420 0x7f7edc19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ed9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7edc101420 0x7f7edc19aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53604/0 (socket says 172.21.15.89:53604) 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/4196960326 learned_addr learned my addr 172.21.15.89:0/4196960326 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7edad9d640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7edc106a80 0x7f7edc19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7eda59c640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc100a80 0x7f7edc19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/4196960326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc100a80 msgr2=0x7f7edc19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc100a80 0x7f7edc19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/4196960326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7edc106a80 msgr2=0x7f7edc19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7edc106a80 0x7f7edc19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.527+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/4196960326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7edc19f610 con 0x7f7edc101420 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.528+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7edc101420 0x7f7edc19aba0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7ed000d950 tx=0x7f7ed000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.528+0000 7f7ecb7fe640 1 -- 172.21.15.89:0/4196960326 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0017070 con 0x7f7edc101420 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.528+0000 7f7ecb7fe640 1 -- 172.21.15.89:0/4196960326 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed0009c30 con 0x7f7edc101420 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.528+0000 7f7ee0a86640 1 -- 172.21.15.89:0/4196960326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7edc19f900 con 0x7f7edc101420 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.528+0000 7f7ecb7fe640 1 -- 172.21.15.89:0/4196960326 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed000f6b0 con 0x7f7edc101420 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.528+0000 7f7ee0a86640 1 -- 172.21.15.89:0/4196960326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7edc1a71e0 con 0x7f7edc101420 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.530+0000 7f7ecb7fe640 1 -- 172.21.15.89:0/4196960326 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7ed00109d0 con 0x7f7edc101420 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.530+0000 7f7ee2e5f740 1 Processor -- start 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ee2e5f740 1 -- start start 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ecb7fe640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7ec40a3240 0x7f7ec40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 0x558ceaf43090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf435d0 0x558ceaf43a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 0x558ceaf50760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558ceaea92f0 con 0x558ceaf435d0 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ceae50cf0 con 0x558ceaf4e370 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558ceaea52b0 con 0x558ceaf44b40 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7edad9d640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7ec40a3240 0x7f7ec40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ecb7fe640 1 -- 172.21.15.89:0/4196960326 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7ed00cf990 con 0x7f7edc101420 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ed9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf435d0 0x558ceaf43a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7eda59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 0x558ceaf43090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ed9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf435d0 0x558ceaf43a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53606/0 (socket says 172.21.15.89:53606) 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/4063924459 learned_addr learned my addr 172.21.15.89:0/4063924459 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7edad9d640 1 --2- 172.21.15.89:0/4063924459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 0x558ceaf50760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/4063924459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 msgr2=0x558ceaf43090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/4063924459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 0x558ceaf43090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/4063924459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 msgr2=0x558ceaf50760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/4063924459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 0x558ceaf50760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.531+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/4063924459 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ceae353a0 con 0x558ceaf435d0 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7edad9d640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7ec40a3240 0x7f7ec40a5700 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4004ab0 tx=0x7f7ed40317a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/4063924459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf435d0 0x558ceaf43a10 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0098020 tx=0x7f7ed0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78cad9b3380b8c89 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ec8ff9640 1 -- 172.21.15.89:0/4063924459 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0020070 con 0x558ceaf435d0 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ec8ff9640 1 -- 172.21.15.89:0/4063924459 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed0009de0 con 0x558ceaf435d0 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ec8ff9640 1 -- 172.21.15.89:0/4063924459 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0099ec0 con 0x558ceaf435d0 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/4063924459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf435d0 msgr2=0x558ceaf43a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/4063924459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf435d0 0x558ceaf43a10 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0098020 tx=0x7f7ed0005880 comp rx=0 tx=0).stop 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/4063924459 shutdown_connections 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/4063924459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 0x558ceaf50760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/4063924459 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf435d0 0x558ceaf43a10 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/4063924459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 0x558ceaf43090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.532+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/4063924459 >> 172.21.15.89:0/4063924459 conn(0x558ceaf0c310 msgr2=0x558ceaf3fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/4063924459 shutdown_connections 2024-09-15T23:32:11.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/4063924459 wait complete. 2024-09-15T23:32:11.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 1 -- start start 2024-09-15T23:32:11.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 1 Processor -- start 2024-09-15T23:32:11.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 0x558ceaf4ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 0x558ceaf4b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf4e370 0x558ceaf4b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558ceaf51010 con 0x558ceaf4e370 2024-09-15T23:32:11.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558ceae64060 con 0x558ceaf435d0 2024-09-15T23:32:11.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558ceaf51190 con 0x558ceaf44b40 2024-09-15T23:32:11.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ee2e5f740 10 client.0 _ll_register_callbacks cb 0x558ceabb78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:11.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7edad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf4e370 0x558ceaf4b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7edad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf4e370 0x558ceaf4b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53610/0 (socket says 172.21.15.89:53610) 2024-09-15T23:32:11.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7edad9d640 1 -- 172.21.15.89:0/1996588234 learned_addr learned my addr 172.21.15.89:0/1996588234 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:11.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7eda59c640 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 0x558ceaf4ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.533+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 0x558ceaf4b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7edad9d640 1 -- 172.21.15.89:0/1996588234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 msgr2=0x558ceaf4b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:11.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7edad9d640 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 0x558ceaf4b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7edad9d640 1 -- 172.21.15.89:0/1996588234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 msgr2=0x558ceaf4ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:11.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7edad9d640 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 0x558ceaf4ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7edad9d640 1 -- 172.21.15.89:0/1996588234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ceac1a3c0 con 0x558ceaf4e370 2024-09-15T23:32:11.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 0x558ceaf4b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:11.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7edad9d640 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf4e370 0x558ceaf4b7f0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009cd0 tx=0x7f7ed402a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c141ec569c83b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:11.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ec1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:11.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1996588234 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed404e070 con 0x558ceaf4e370 2024-09-15T23:32:11.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1996588234 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7ed40309b0 con 0x558ceaf4e370 2024-09-15T23:32:11.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1996588234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ed0009de0 con 0x558ceaf4e370 2024-09-15T23:32:11.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1996588234 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7ed405a050 con 0x558ceaf4e370 2024-09-15T23:32:11.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ee2e5f740 10 client.18165 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:11.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1996588234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ceaf5cb00 con 0x558ceaf4e370 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1996588234 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed0009de0 con 0x558ceaf4e370 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1996588234 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7ed4049070 con 0x558ceaf4e370 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ee2e5f740 10 client.18165 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ee2e5f740 10 client.18165 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ee2e5f740 20 client.18165 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.534+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558ceaf5dfe0 0x558ceaf60460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.535+0000 7f7ee2e5f740 4 client.18165 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.535+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1996588234 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ceaeb6860 con 0x558ceaf5dfe0 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.535+0000 7f7eda59c640 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558ceaf5dfe0 0x558ceaf60460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.535+0000 7f7ec1ffb640 10 client.18165 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.535+0000 7f7eda59c640 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558ceaf5dfe0 0x558ceaf60460 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3159ef49c972bf50 server_cookie=957e1542bb55d341 in_seq=0 out_seq=0 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1996588234 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558ceaeb6860 con 0x558ceaf5dfe0 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ec1ffb640 10 client.18165 handle_command_reply: tid=0 2024-09-15T23:32:11.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 client.18165 shutdown 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1996588234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf4e370 msgr2=0x558ceaf4b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf4e370 0x558ceaf4b7f0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009cd0 tx=0x7f7ed402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1996588234 shutdown_connections 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558ceaf5dfe0 0x558ceaf60460 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf4e370 0x558ceaf4b7f0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf44b40 0x558ceaf4b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ec1ffb640 0 client.18165 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1996588234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 0x558ceaf4ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1996588234 >> 172.21.15.89:0/1996588234 conn(0x558ceaf0c310 msgr2=0x558ceaf45990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1996588234 shutdown_connections 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1996588234 wait complete. 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.536+0000 7f7ee2e5f740 20 client.18165 trim_cache size 0 max 16384 2024-09-15T23:32:11.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ee2e5f740 1 Processor -- start 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ee2e5f740 1 -- start start 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf435d0 0x558ceaf4ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf46180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 0x558ceaf466c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc1b1080 con 0x558ceaf44b40 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc1b2bc0 con 0x558ceaf4e370 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc1b1750 con 0x558ceaf435d0 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ed9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf46180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ed9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf46180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53626/0 (socket says 172.21.15.89:53626) 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.549+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/663036574 learned_addr learned my addr 172.21.15.89:0/663036574 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7edad9d640 1 --2- 172.21.15.89:0/663036574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 0x558ceaf466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/663036574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf435d0 msgr2=0x558ceaf4ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:11.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/663036574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf435d0 0x558ceaf4ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/663036574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 msgr2=0x558ceaf466c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/663036574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 0x558ceaf466c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/663036574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ceaf60ae0 con 0x558ceaf44b40 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7edad9d640 1 --2- 172.21.15.89:0/663036574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 0x558ceaf466c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/663036574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf46180 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00cd0f0 tx=0x7f7ed0078f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b895e9c311f725e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ec3fff640 1 -- 172.21.15.89:0/663036574 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0020070 con 0x558ceaf44b40 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ec3fff640 1 -- 172.21.15.89:0/663036574 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed009ff70 con 0x558ceaf44b40 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ec3fff640 1 -- 172.21.15.89:0/663036574 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed007de20 con 0x558ceaf44b40 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/663036574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 msgr2=0x558ceaf46180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/663036574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf46180 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00cd0f0 tx=0x7f7ed0078f90 comp rx=0 tx=0).stop 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/663036574 shutdown_connections 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/663036574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf4e370 0x558ceaf466c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/663036574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf46180 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/663036574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf435d0 0x558ceaf4ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.550+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/663036574 >> 172.21.15.89:0/663036574 conn(0x558ceaf0c310 msgr2=0x558ceaf3aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/663036574 shutdown_connections 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/663036574 wait complete. 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 1 -- start start 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 1 Processor -- start 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 0x558ceaf56810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf56d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf4e370 0x558ceaf51d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc1b24f0 con 0x558ceaf44b40 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc1b1e20 con 0x558ceaf435d0 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7edc1af060 con 0x558ceaf4e370 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ee2e5f740 10 client.0 _ll_register_callbacks cb 0x7f7ed4031a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ed9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf56d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ed9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf56d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53630/0 (socket says 172.21.15.89:53630) 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/1793414580 learned_addr learned my addr 172.21.15.89:0/1793414580 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7eda59c640 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 0x558ceaf56810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.551+0000 7f7edad9d640 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf4e370 0x558ceaf51d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/1793414580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf4e370 msgr2=0x558ceaf51d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf4e370 0x558ceaf51d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/1793414580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 msgr2=0x558ceaf56810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 0x558ceaf56810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ed9d9b640 1 -- 172.21.15.89:0/1793414580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ceaf525c0 con 0x558ceaf44b40 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7eda59c640 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 0x558ceaf56810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ed9d9b640 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf56d50 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0010fd0 tx=0x7f7ed0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8546c71e3e5c3cec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ec1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1793414580 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0020030 con 0x558ceaf44b40 2024-09-15T23:32:11.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1793414580 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7ed007de20 con 0x558ceaf44b40 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1793414580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558ceaeb0d70 con 0x558ceaf44b40 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1793414580 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7ed009bbc0 con 0x558ceaf44b40 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ee2e5f740 10 client.18171 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1793414580 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558ceafc8960 con 0x558ceaf44b40 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.552+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1793414580 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558ceaeb0d70 con 0x558ceaf44b40 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.553+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1793414580 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7ed007f040 con 0x558ceaf44b40 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.553+0000 7f7ee2e5f740 10 client.18171 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.553+0000 7f7ee2e5f740 10 client.18171 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.553+0000 7f7ee2e5f740 20 client.18171 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.553+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558ceafc8c90 0x558ceafcb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.553+0000 7f7ee2e5f740 4 client.18171 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.553+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1793414580 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ceae2d170 con 0x558ceafc8c90 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.553+0000 7f7eda59c640 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558ceafc8c90 0x558ceafcb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:11.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.553+0000 7f7eda59c640 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558ceafc8c90 0x558ceafcb080 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=55e97f41519eac28 server_cookie=4485fad75e17d408 in_seq=0 out_seq=0 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.553+0000 7f7ec1ffb640 10 client.18171 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ec1ffb640 1 -- 172.21.15.89:0/1793414580 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ceae2d170 con 0x558ceafc8c90 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ec1ffb640 10 client.18171 handle_command_reply: tid=0 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 client.18171 shutdown 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1793414580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 msgr2=0x558ceaf56d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf56d50 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0010fd0 tx=0x7f7ed0005880 comp rx=0 tx=0).stop 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1793414580 shutdown_connections 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x558ceafc8c90 0x558ceafcb080 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558ceaf4e370 0x558ceaf51d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ceaf44b40 0x558ceaf56d50 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 --2- 172.21.15.89:0/1793414580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558ceaf435d0 0x558ceaf56810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1793414580 >> 172.21.15.89:0/1793414580 conn(0x558ceaf0c310 msgr2=0x558ceaf4c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ec1ffb640 0 client.18171 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:32:11.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1793414580 shutdown_connections 2024-09-15T23:32:11.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 1 -- 172.21.15.89:0/1793414580 wait complete. 2024-09-15T23:32:11.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.554+0000 7f7ee2e5f740 20 client.18171 trim_cache size 0 max 16384 2024-09-15T23:32:11.574 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 -- 172.21.15.89:0/4196960326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7ec40a3240 msgr2=0x7f7ec40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7ec40a3240 0x7f7ec40a5700 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4004ab0 tx=0x7f7ed40317a0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 -- 172.21.15.89:0/4196960326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7edc101420 msgr2=0x7f7edc19aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7edc101420 0x7f7edc19aba0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7ed000d950 tx=0x7f7ed000de10 comp rx=0 tx=0).stop 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 -- 172.21.15.89:0/4196960326 shutdown_connections 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7ec40a3240 0x7f7ec40a5700 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7edc106a80 0x7f7edc19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7edc101420 0x7f7edc19aba0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 --2- 172.21.15.89:0/4196960326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7edc100a80 0x7f7edc19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 -- 172.21.15.89:0/4196960326 >> 172.21.15.89:0/4196960326 conn(0x7f7edc0fc7a0 msgr2=0x7f7edc0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:11.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 -- 172.21.15.89:0/4196960326 shutdown_connections 2024-09-15T23:32:11.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:11.555+0000 7f7ee0a86640 1 -- 172.21.15.89:0/4196960326 wait complete. 2024-09-15T23:32:11.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:11 smithi120 ceph-mon[35250]: pgmap v595: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-15T23:32:12.276 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:32:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:13.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.061+0000 7f0f79f29640 1 -- 172.21.15.89:0/3311574874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f7410c160 msgr2=0x7f0f7410e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:13.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.061+0000 7f0f79f29640 1 --2- 172.21.15.89:0/3311574874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f7410c160 0x7f0f7410e600 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009ca0 tx=0x7f0f6c02a950 comp rx=0 tx=0).stop 2024-09-15T23:32:13.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.061+0000 7f0f79f29640 1 -- 172.21.15.89:0/3311574874 shutdown_connections 2024-09-15T23:32:13.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.061+0000 7f0f79f29640 1 --2- 172.21.15.89:0/3311574874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f7410c160 0x7f0f7410e600 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.061+0000 7f0f79f29640 1 --2- 172.21.15.89:0/3311574874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f74103650 0x7f0f7410bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.061+0000 7f0f79f29640 1 --2- 172.21.15.89:0/3311574874 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f74102d30 0x7f0f74103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.061+0000 7f0f79f29640 1 -- 172.21.15.89:0/3311574874 >> 172.21.15.89:0/3311574874 conn(0x7f0f740fc790 msgr2=0x7f0f740febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:13.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.062+0000 7f0f79f29640 1 -- 172.21.15.89:0/3311574874 shutdown_connections 2024-09-15T23:32:13.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.062+0000 7f0f79f29640 1 -- 172.21.15.89:0/3311574874 wait complete. 2024-09-15T23:32:13.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.062+0000 7f0f79f29640 1 Processor -- start 2024-09-15T23:32:13.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.062+0000 7f0f79f29640 1 -- start start 2024-09-15T23:32:13.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f79f29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f74102d30 0x7f0f74198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f79f29640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f74103650 0x7f0f741988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f79f29640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f7410c160 0x7f0f7419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f79f29640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0f74111f60 con 0x7f0f74102d30 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f79f29640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0f74111de0 con 0x7f0f74103650 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f79f29640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0f741120e0 con 0x7f0f7410c160 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f74102d30 0x7f0f74198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f74102d30 0x7f0f74198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53642/0 (socket says 172.21.15.89:53642) 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f737fe640 1 -- 172.21.15.89:0/3381175854 learned_addr learned my addr 172.21.15.89:0/3381175854 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f73fff640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f7410c160 0x7f0f7419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f6affd640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f74103650 0x7f0f741988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f737fe640 1 -- 172.21.15.89:0/3381175854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f7410c160 msgr2=0x7f0f7419cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f737fe640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f7410c160 0x7f0f7419cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f737fe640 1 -- 172.21.15.89:0/3381175854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f74103650 msgr2=0x7f0f741988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f737fe640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f74103650 0x7f0f741988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f737fe640 1 -- 172.21.15.89:0/3381175854 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7419d330 con 0x7f0f74102d30 2024-09-15T23:32:13.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f737fe640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f74102d30 0x7f0f74198380 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f0f6400d770 tx=0x7f0f6400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:13.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.063+0000 7f0f717fa640 1 -- 172.21.15.89:0/3381175854 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f64017070 con 0x7f0f74102d30 2024-09-15T23:32:13.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.064+0000 7f0f717fa640 1 -- 172.21.15.89:0/3381175854 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f64005500 con 0x7f0f74102d30 2024-09-15T23:32:13.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.064+0000 7f0f79f29640 1 -- 172.21.15.89:0/3381175854 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7419d620 con 0x7f0f74102d30 2024-09-15T23:32:13.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.064+0000 7f0f717fa640 1 -- 172.21.15.89:0/3381175854 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f64004e00 con 0x7f0f74102d30 2024-09-15T23:32:13.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.064+0000 7f0f79f29640 1 -- 172.21.15.89:0/3381175854 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f741a4f00 con 0x7f0f74102d30 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.066+0000 7f0f717fa640 1 -- 172.21.15.89:0/3381175854 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0f64004fa0 con 0x7f0f74102d30 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.066+0000 7f0f7c302740 1 Processor -- start 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.066+0000 7f0f7c302740 1 -- start start 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.066+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564808892b40 0x564808891090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.066+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x564808891a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.066+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480889c370 0x56480889e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f7c302740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5648087f72f0 con 0x56480889c370 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f7c302740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56480879ecf0 con 0x564808892b40 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f7c302740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5648087f32b0 con 0x5648088915d0 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f717fa640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f600a3240 0x7f0f600a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f6affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480889c370 0x56480889e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f6affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480889c370 0x56480889e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53652/0 (socket says 172.21.15.89:53652) 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f6affd640 1 -- 172.21.15.89:0/1828785948 learned_addr learned my addr 172.21.15.89:0/1828785948 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f737fe640 1 --2- 172.21.15.89:0/1828785948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x564808891a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f73fff640 1 --2- 172.21.15.89:0/1828785948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564808892b40 0x564808891090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f73fff640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f600a3240 0x7f0f600a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f717fa640 1 -- 172.21.15.89:0/3381175854 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0f640cf6d0 con 0x7f0f74102d30 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f6affd640 1 -- 172.21.15.89:0/1828785948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 msgr2=0x564808891a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f6affd640 1 --2- 172.21.15.89:0/1828785948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x564808891a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f6affd640 1 -- 172.21.15.89:0/1828785948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564808892b40 msgr2=0x564808891090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f6affd640 1 --2- 172.21.15.89:0/1828785948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564808892b40 0x564808891090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f6affd640 1 -- 172.21.15.89:0/1828785948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648087833a0 con 0x56480889c370 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f73fff640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f600a3240 0x7f0f600a5700 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c005e30 tx=0x7f0f6c005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f6affd640 1 --2- 172.21.15.89:0/1828785948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480889c370 0x56480889e760 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00ac00 tx=0x7f0f5c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bda7f2bbc3d597a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.067+0000 7f0f69ffb640 1 -- 172.21.15.89:0/1828785948 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c01d070 con 0x56480889c370 2024-09-15T23:32:13.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f69ffb640 1 -- 172.21.15.89:0/1828785948 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f5c018b20 con 0x56480889c370 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f69ffb640 1 -- 172.21.15.89:0/1828785948 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c010b00 con 0x56480889c370 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f7c302740 1 -- 172.21.15.89:0/1828785948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480889c370 msgr2=0x56480889e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f7c302740 1 --2- 172.21.15.89:0/1828785948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480889c370 0x56480889e760 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00ac00 tx=0x7f0f5c00f790 comp rx=0 tx=0).stop 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f7c302740 1 -- 172.21.15.89:0/1828785948 shutdown_connections 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f7c302740 1 --2- 172.21.15.89:0/1828785948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480889c370 0x56480889e760 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f7c302740 1 --2- 172.21.15.89:0/1828785948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x564808891a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f7c302740 1 --2- 172.21.15.89:0/1828785948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564808892b40 0x564808891090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f7c302740 1 -- 172.21.15.89:0/1828785948 >> 172.21.15.89:0/1828785948 conn(0x56480885a310 msgr2=0x56480888dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f7c302740 1 -- 172.21.15.89:0/1828785948 shutdown_connections 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.068+0000 7f0f7c302740 1 -- 172.21.15.89:0/1828785948 wait complete. 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f7c302740 1 -- start start 2024-09-15T23:32:13.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f7c302740 1 Processor -- start 2024-09-15T23:32:13.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x56480889bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x564808899000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480889c370 0x5648088995d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f7c302740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56480889f070 con 0x564808892b40 2024-09-15T23:32:13.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f7c302740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5648087b2060 con 0x56480889c370 2024-09-15T23:32:13.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f7c302740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56480889f1f0 con 0x5648088915d0 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f7c302740 10 client.0 _ll_register_callbacks cb 0x5648085058c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f6affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x564808899000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f6affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x564808899000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53656/0 (socket says 172.21.15.89:53656) 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.069+0000 7f0f6affd640 1 -- 172.21.15.89:0/2427106314 learned_addr learned my addr 172.21.15.89:0/2427106314 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f737fe640 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x56480889bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f73fff640 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480889c370 0x5648088995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f6affd640 1 -- 172.21.15.89:0/2427106314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 msgr2=0x56480889bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f6affd640 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x56480889bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f6affd640 1 -- 172.21.15.89:0/2427106314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480889c370 msgr2=0x5648088995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f6affd640 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480889c370 0x5648088995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f6affd640 1 -- 172.21.15.89:0/2427106314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648085683c0 con 0x564808892b40 2024-09-15T23:32:13.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f73fff640 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480889c370 0x5648088995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f6affd640 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x564808899000 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00a800 tx=0x7f0f5c0186f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18acf641a403500b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f52ffd640 1 -- 172.21.15.89:0/2427106314 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c01d050 con 0x564808892b40 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f52ffd640 1 -- 172.21.15.89:0/2427106314 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0f5c010840 con 0x564808892b40 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f7c302740 1 -- 172.21.15.89:0/2427106314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f5c018b20 con 0x564808892b40 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f52ffd640 1 -- 172.21.15.89:0/2427106314 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0f5c031050 con 0x564808892b40 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f7c302740 10 client.18183 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f7c302740 1 -- 172.21.15.89:0/2427106314 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5648088aaaf0 con 0x564808892b40 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.070+0000 7f0f52ffd640 1 -- 172.21.15.89:0/2427106314 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f5c018b20 con 0x564808892b40 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.071+0000 7f0f52ffd640 1 -- 172.21.15.89:0/2427106314 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0f5c0046c0 con 0x564808892b40 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.071+0000 7f0f7c302740 10 client.18183 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.071+0000 7f0f7c302740 10 client.18183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.071+0000 7f0f7c302740 20 client.18183 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.071+0000 7f0f7c302740 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5648088abe40 0x5648088ae230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.071+0000 7f0f7c302740 4 client.18183 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.071+0000 7f0f7c302740 1 -- 172.21.15.89:0/2427106314 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564808804860 con 0x5648088abe40 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.071+0000 7f0f737fe640 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5648088abe40 0x5648088ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.072+0000 7f0f737fe640 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5648088abe40 0x5648088ae230 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b54363d7e1b9126 server_cookie=e6d56ace6f5c4a1b in_seq=0 out_seq=0 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.072+0000 7f0f52ffd640 10 client.18183 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.072+0000 7f0f52ffd640 1 -- 172.21.15.89:0/2427106314 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564808804860 con 0x5648088abe40 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.072+0000 7f0f52ffd640 10 client.18183 handle_command_reply: tid=0 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.072+0000 7f0f7c302740 1 client.18183 shutdown 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.072+0000 7f0f7c302740 1 -- 172.21.15.89:0/2427106314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 msgr2=0x564808899000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.072+0000 7f0f7c302740 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x564808899000 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00a800 tx=0x7f0f5c0186f0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.073+0000 7f0f7c302740 1 -- 172.21.15.89:0/2427106314 shutdown_connections 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.073+0000 7f0f7c302740 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5648088abe40 0x5648088ae230 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.073+0000 7f0f7c302740 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480889c370 0x5648088995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.073+0000 7f0f7c302740 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x564808899000 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.073+0000 7f0f52ffd640 0 client.18183 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:13.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.073+0000 7f0f7c302740 1 --2- 172.21.15.89:0/2427106314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x56480889bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.073+0000 7f0f7c302740 1 -- 172.21.15.89:0/2427106314 >> 172.21.15.89:0/2427106314 conn(0x56480885a310 msgr2=0x564808893960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:13.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.073+0000 7f0f7c302740 1 -- 172.21.15.89:0/2427106314 shutdown_connections 2024-09-15T23:32:13.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.073+0000 7f0f7c302740 1 -- 172.21.15.89:0/2427106314 wait complete. 2024-09-15T23:32:13.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.073+0000 7f0f7c302740 20 client.18183 trim_cache size 0 max 16384 2024-09-15T23:32:13.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.085+0000 7f0f7c302740 1 Processor -- start 2024-09-15T23:32:13.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f7c302740 1 -- start start 2024-09-15T23:32:13.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480885a310 0x5648088a8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648088915d0 0x564808890cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564808892b40 0x5648088b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f7c302740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0f741aedb0 con 0x56480885a310 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f7c302740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0f741b08f0 con 0x5648088915d0 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f7c302740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0f741af480 con 0x564808892b40 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480885a310 0x5648088a8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480885a310 0x5648088a8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53664/0 (socket says 172.21.15.89:53664) 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f737fe640 1 -- 172.21.15.89:0/3476366964 learned_addr learned my addr 172.21.15.89:0/3476366964 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f737fe640 1 -- 172.21.15.89:0/3476366964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564808892b40 msgr2=0x5648088b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f737fe640 1 --2- 172.21.15.89:0/3476366964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564808892b40 0x5648088b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f737fe640 1 -- 172.21.15.89:0/3476366964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648088915d0 msgr2=0x564808890cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f737fe640 1 --2- 172.21.15.89:0/3476366964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648088915d0 0x564808890cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f737fe640 1 -- 172.21.15.89:0/3476366964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56480889f1b0 con 0x56480885a310 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f737fe640 1 --2- 172.21.15.89:0/3476366964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480885a310 0x5648088a8310 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009610 tx=0x7f0f640d1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13867a16755923af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f697fa640 1 -- 172.21.15.89:0/3476366964 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f64020040 con 0x56480885a310 2024-09-15T23:32:13.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f697fa640 1 -- 172.21.15.89:0/3476366964 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f6409fcb0 con 0x56480885a310 2024-09-15T23:32:13.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.086+0000 7f0f697fa640 1 -- 172.21.15.89:0/3476366964 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f64099d90 con 0x56480885a310 2024-09-15T23:32:13.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.087+0000 7f0f7c302740 1 -- 172.21.15.89:0/3476366964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480885a310 msgr2=0x5648088a8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:13.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.087+0000 7f0f7c302740 1 --2- 172.21.15.89:0/3476366964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480885a310 0x5648088a8310 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009610 tx=0x7f0f640d1040 comp rx=0 tx=0).stop 2024-09-15T23:32:13.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.087+0000 7f0f7c302740 1 -- 172.21.15.89:0/3476366964 shutdown_connections 2024-09-15T23:32:13.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.087+0000 7f0f7c302740 1 --2- 172.21.15.89:0/3476366964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564808892b40 0x5648088b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.087+0000 7f0f7c302740 1 --2- 172.21.15.89:0/3476366964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648088915d0 0x564808890cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.087+0000 7f0f7c302740 1 --2- 172.21.15.89:0/3476366964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56480885a310 0x5648088a8310 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.087+0000 7f0f7c302740 1 -- 172.21.15.89:0/3476366964 >> 172.21.15.89:0/3476366964 conn(0x5648088949f0 msgr2=0x56480888cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:13.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.087+0000 7f0f7c302740 1 -- 172.21.15.89:0/3476366964 shutdown_connections 2024-09-15T23:32:13.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.087+0000 7f0f7c302740 1 -- 172.21.15.89:0/3476366964 wait complete. 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.087+0000 7f0f7c302740 1 -- start start 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f7c302740 1 Processor -- start 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480885a310 0x5648088a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x5648088a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f7c302740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x5648089157d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f7c302740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0f741b0220 con 0x564808892b40 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f7c302740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0f741afb50 con 0x56480885a310 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f7c302740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0f741acd90 con 0x5648088915d0 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f7c302740 10 client.0 _ll_register_callbacks cb 0x7f0f5c01b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f73fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x5648089157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f73fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x5648089157d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:53678/0 (socket says 172.21.15.89:53678) 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f73fff640 1 -- 172.21.15.89:0/3413921193 learned_addr learned my addr 172.21.15.89:0/3413921193 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:13.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f737fe640 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480885a310 0x5648088a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f6affd640 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x5648088a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f73fff640 1 -- 172.21.15.89:0/3413921193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 msgr2=0x5648088a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f73fff640 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x5648088a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f73fff640 1 -- 172.21.15.89:0/3413921193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480885a310 msgr2=0x5648088a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f73fff640 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480885a310 0x5648088a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.088+0000 7f0f73fff640 1 -- 172.21.15.89:0/3413921193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564808915d10 con 0x564808892b40 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f73fff640 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x5648089157d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009ca0 tx=0x7f0f6c02fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd024536cc16fc8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f52ffd640 1 -- 172.21.15.89:0/3413921193 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c034040 con 0x564808892b40 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f52ffd640 1 -- 172.21.15.89:0/3413921193 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0f6c004250 con 0x564808892b40 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f7c302740 1 -- 172.21.15.89:0/3413921193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f6409fcb0 con 0x564808892b40 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f52ffd640 1 -- 172.21.15.89:0/3413921193 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0f6c004450 con 0x564808892b40 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f7c302740 10 client.18189 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:13.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f7c302740 1 -- 172.21.15.89:0/3413921193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564808916f90 con 0x564808892b40 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f52ffd640 1 -- 172.21.15.89:0/3413921193 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f6409fcb0 con 0x564808892b40 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f52ffd640 1 -- 172.21.15.89:0/3413921193 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0f6c04c430 con 0x564808892b40 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f7c302740 10 client.18189 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f7c302740 10 client.18189 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f7c302740 20 client.18189 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f7c302740 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56480889c370 0x564808919750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f7c302740 4 client.18189 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.089+0000 7f0f7c302740 1 -- 172.21.15.89:0/3413921193 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5648087fed70 con 0x56480889c370 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.090+0000 7f0f737fe640 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56480889c370 0x564808919750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.090+0000 7f0f52ffd640 10 client.18189 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.090+0000 7f0f737fe640 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56480889c370 0x564808919750 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c36d1abb5260b80 server_cookie=5aa54aa648fc2b35 in_seq=0 out_seq=0 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.090+0000 7f0f52ffd640 1 -- 172.21.15.89:0/3413921193 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5648087fed70 con 0x56480889c370 2024-09-15T23:32:13.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.090+0000 7f0f52ffd640 10 client.18189 handle_command_reply: tid=0 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.090+0000 7f0f7c302740 1 client.18189 shutdown 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.090+0000 7f0f7c302740 1 -- 172.21.15.89:0/3413921193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 msgr2=0x5648089157d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.090+0000 7f0f7c302740 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x5648089157d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009ca0 tx=0x7f0f6c02fe10 comp rx=0 tx=0).stop 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f7c302740 1 -- 172.21.15.89:0/3413921193 shutdown_connections 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f7c302740 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56480889c370 0x564808919750 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f7c302740 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564808892b40 0x5648089157d0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f7c302740 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5648088915d0 0x5648088a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f7c302740 1 --2- 172.21.15.89:0/3413921193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56480885a310 0x5648088a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f7c302740 1 -- 172.21.15.89:0/3413921193 >> 172.21.15.89:0/3413921193 conn(0x5648088949f0 msgr2=0x5648088977a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f52ffd640 0 client.18189 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f7c302740 1 -- 172.21.15.89:0/3413921193 shutdown_connections 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f7c302740 1 -- 172.21.15.89:0/3413921193 wait complete. 2024-09-15T23:32:13.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f7c302740 20 client.18189 trim_cache size 0 max 16384 2024-09-15T23:32:13.097 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:32:13.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 -- 172.21.15.89:0/3381175854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f600a3240 msgr2=0x7f0f600a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:13.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f600a3240 0x7f0f600a5700 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c005e30 tx=0x7f0f6c005c50 comp rx=0 tx=0).stop 2024-09-15T23:32:13.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 -- 172.21.15.89:0/3381175854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f74102d30 msgr2=0x7f0f74198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:13.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f74102d30 0x7f0f74198380 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f0f6400d770 tx=0x7f0f6400dc30 comp rx=0 tx=0).stop 2024-09-15T23:32:13.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 -- 172.21.15.89:0/3381175854 shutdown_connections 2024-09-15T23:32:13.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f600a3240 0x7f0f600a5700 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f7410c160 0x7f0f7419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f74103650 0x7f0f741988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 --2- 172.21.15.89:0/3381175854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f74102d30 0x7f0f74198380 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:13.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 -- 172.21.15.89:0/3381175854 >> 172.21.15.89:0/3381175854 conn(0x7f0f740fc790 msgr2=0x7f0f740fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:13.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 -- 172.21.15.89:0/3381175854 shutdown_connections 2024-09-15T23:32:13.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:13.091+0000 7f0f79f29640 1 -- 172.21.15.89:0/3381175854 wait complete. 2024-09-15T23:32:13.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:13 smithi089 ceph-mon[30933]: pgmap v596: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-09-15T23:32:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:13 smithi174 ceph-mon[35586]: pgmap v596: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-09-15T23:32:13.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:13 smithi120 ceph-mon[35250]: pgmap v596: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.7 KiB/s wr, 12 op/s 2024-09-15T23:32:14.755 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:32:14.756 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:32:15.016 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:15.432 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:15 smithi089 ceph-mon[30933]: pgmap v597: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-09-15T23:32:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:15 smithi174 ceph-mon[35586]: pgmap v597: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-09-15T23:32:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 -- 172.21.15.89:0/3898611935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca54102ef0 msgr2=0x7fca5410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 --2- 172.21.15.89:0/3898611935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca54102ef0 0x7fca5410da40 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009ca0 tx=0x7fca4c02a950 comp rx=0 tx=0).stop 2024-09-15T23:32:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 -- 172.21.15.89:0/3898611935 shutdown_connections 2024-09-15T23:32:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 --2- 172.21.15.89:0/3898611935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca54102ef0 0x7fca5410da40 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 --2- 172.21.15.89:0/3898611935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca54102570 0x7fca541029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 --2- 172.21.15.89:0/3898611935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca54100e20 0x7fca54101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 -- 172.21.15.89:0/3898611935 >> 172.21.15.89:0/3898611935 conn(0x7fca540fc7d0 msgr2=0x7fca540febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 -- 172.21.15.89:0/3898611935 shutdown_connections 2024-09-15T23:32:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 -- 172.21.15.89:0/3898611935 wait complete. 2024-09-15T23:32:15.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 Processor -- start 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.518+0000 7fca59424640 1 -- start start 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca59424640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca54100e20 0x7fca541992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca59424640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca54102570 0x7fca54199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca59424640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca54102ef0 0x7fca541a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca59424640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fca54110f80 con 0x7fca54102ef0 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca59424640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fca54110e00 con 0x7fca54100e20 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca59424640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fca54111100 con 0x7fca54102570 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca537fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca54102ef0 0x7fca541a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca537fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca54102ef0 0x7fca541a4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33396/0 (socket says 172.21.15.89:33396) 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca527fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca54102570 0x7fca54199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca537fe640 1 -- 172.21.15.89:0/4278347862 learned_addr learned my addr 172.21.15.89:0/4278347862 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca52ffd640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca54100e20 0x7fca541992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca537fe640 1 -- 172.21.15.89:0/4278347862 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca54102570 msgr2=0x7fca54199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:15.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca537fe640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca54102570 0x7fca54199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca537fe640 1 -- 172.21.15.89:0/4278347862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca54100e20 msgr2=0x7fca541992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca537fe640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca54100e20 0x7fca541992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca537fe640 1 -- 172.21.15.89:0/4278347862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca541a4880 con 0x7fca54102ef0 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.519+0000 7fca537fe640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca54102ef0 0x7fca541a4340 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fca4c004af0 tx=0x7fca4c02f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.520+0000 7fca3bfff640 1 -- 172.21.15.89:0/4278347862 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c034040 con 0x7fca54102ef0 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.520+0000 7fca59424640 1 -- 172.21.15.89:0/4278347862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca541a4a50 con 0x7fca54102ef0 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.520+0000 7fca3bfff640 1 -- 172.21.15.89:0/4278347862 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca4c030920 con 0x7fca54102ef0 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.520+0000 7fca3bfff640 1 -- 172.21.15.89:0/4278347862 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c004820 con 0x7fca54102ef0 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.520+0000 7fca59424640 1 -- 172.21.15.89:0/4278347862 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca541a4f00 con 0x7fca54102ef0 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.522+0000 7fca3bfff640 1 -- 172.21.15.89:0/4278347862 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fca4c037020 con 0x7fca54102ef0 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.522+0000 7fca5b7fd740 1 Processor -- start 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.522+0000 7fca5b7fd740 1 -- start start 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.522+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c88313e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.522+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88314340 0x557c88322c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.522+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88323150 0x557c88325540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.522+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557c88279280 con 0x557c88313a20 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.522+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557c88220b10 con 0x557c88323150 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.522+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557c88275240 con 0x557c88314340 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca527fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c88313e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca527fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c88313e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33408/0 (socket says 172.21.15.89:33408) 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca527fc640 1 -- 172.21.15.89:0/220743892 learned_addr learned my addr 172.21.15.89:0/220743892 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca52ffd640 1 --2- 172.21.15.89:0/220743892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88314340 0x557c88322c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca3bfff640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca3c0a3240 0x7fca3c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca527fc640 1 -- 172.21.15.89:0/220743892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88314340 msgr2=0x557c88322c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca527fc640 1 --2- 172.21.15.89:0/220743892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88314340 0x557c88322c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca527fc640 1 -- 172.21.15.89:0/220743892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88323150 msgr2=0x557c88325540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca527fc640 1 --2- 172.21.15.89:0/220743892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88323150 0x557c88325540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca527fc640 1 -- 172.21.15.89:0/220743892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c8822d1f0 con 0x557c88313a20 2024-09-15T23:32:15.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca527fc640 1 --2- 172.21.15.89:0/220743892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c88313e00 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fca4800bcc0 tx=0x7fca480127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a75b91783c03cbd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca3bfff640 1 -- 172.21.15.89:0/4278347862 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fca4c0f0c30 con 0x7fca54102ef0 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca537fe640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca3c0a3240 0x7fca3c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca397fa640 1 -- 172.21.15.89:0/220743892 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4800be70 con 0x557c88313a20 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca397fa640 1 -- 172.21.15.89:0/220743892 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca48016070 con 0x557c88313a20 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.523+0000 7fca397fa640 1 -- 172.21.15.89:0/220743892 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca48010430 con 0x557c88313a20 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.524+0000 7fca5b7fd740 1 -- 172.21.15.89:0/220743892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 msgr2=0x557c88313e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.524+0000 7fca537fe640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca3c0a3240 0x7fca3c0a5700 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fca4c032000 tx=0x7fca4c0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.524+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/220743892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c88313e00 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fca4800bcc0 tx=0x7fca480127c0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.524+0000 7fca5b7fd740 1 -- 172.21.15.89:0/220743892 shutdown_connections 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.524+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/220743892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88323150 0x557c88325540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.524+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/220743892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88314340 0x557c88322c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.545 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.524+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/220743892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c88313e00 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.524+0000 7fca5b7fd740 1 -- 172.21.15.89:0/220743892 >> 172.21.15.89:0/220743892 conn(0x557c883100f0 msgr2=0x557c883104d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.524+0000 7fca5b7fd740 1 -- 172.21.15.89:0/220743892 shutdown_connections 2024-09-15T23:32:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.524+0000 7fca5b7fd740 1 -- 172.21.15.89:0/220743892 wait complete. 2024-09-15T23:32:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca5b7fd740 1 -- start start 2024-09-15T23:32:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca5b7fd740 1 Processor -- start 2024-09-15T23:32:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88313a20 0x557c8831fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88314340 0x557c88320530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c88320a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557c88325d20 con 0x557c88323150 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557c88233f10 con 0x557c88314340 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557c88325ea0 con 0x557c88313a20 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca5b7fd740 10 client.0 _ll_register_callbacks cb 0x557c881fca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca527fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c88320a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca527fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c88320a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33422/0 (socket says 172.21.15.89:33422) 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca527fc640 1 -- 172.21.15.89:0/3944493065 learned_addr learned my addr 172.21.15.89:0/3944493065 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca52ffd640 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88314340 0x557c88320530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca527fc640 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88313a20 0x557c8831fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca527fc640 1 -- 172.21.15.89:0/3944493065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88313a20 msgr2=0x557c8831fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca527fc640 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88313a20 0x557c8831fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca527fc640 1 -- 172.21.15.89:0/3944493065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88314340 msgr2=0x557c88320530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca527fc640 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88314340 0x557c88320530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca527fc640 1 -- 172.21.15.89:0/3944493065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c87fea6a0 con 0x557c88323150 2024-09-15T23:32:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca527fc640 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c88320a70 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fca480122c0 tx=0x7fca48005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31843447f6f8aae7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.525+0000 7fca36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca36ff5640 1 -- 172.21.15.89:0/3944493065 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca480048c0 con 0x557c88323150 2024-09-15T23:32:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca36ff5640 1 -- 172.21.15.89:0/3944493065 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fca48004a60 con 0x557c88323150 2024-09-15T23:32:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca5b7fd740 1 -- 172.21.15.89:0/3944493065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca48016070 con 0x557c88323150 2024-09-15T23:32:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca36ff5640 1 -- 172.21.15.89:0/3944493065 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fca4802e050 con 0x557c88323150 2024-09-15T23:32:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca5b7fd740 10 client.18201 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca5b7fd740 1 -- 172.21.15.89:0/3944493065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557c88331940 con 0x557c88323150 2024-09-15T23:32:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca36ff5640 1 -- 172.21.15.89:0/3944493065 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca48016070 con 0x557c88323150 2024-09-15T23:32:15.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca36ff5640 1 -- 172.21.15.89:0/3944493065 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fca48026020 con 0x557c88323150 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca5b7fd740 10 client.18201 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca5b7fd740 10 client.18201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca5b7fd740 20 client.18201 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557c88332d40 0x557c88335130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca5b7fd740 4 client.18201 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca5b7fd740 1 -- 172.21.15.89:0/3944493065 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c882867f0 con 0x557c88332d40 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.526+0000 7fca52ffd640 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557c88332d40 0x557c88335130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.527+0000 7fca36ff5640 10 client.18201 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.527+0000 7fca52ffd640 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557c88332d40 0x557c88335130 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=514f119af6dec1ce server_cookie=3737098a9fcb5d78 in_seq=0 out_seq=0 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca36ff5640 1 -- 172.21.15.89:0/3944493065 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557c882867f0 con 0x557c88332d40 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca36ff5640 10 client.18201 handle_command_reply: tid=0 2024-09-15T23:32:15.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 client.18201 shutdown 2024-09-15T23:32:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 -- 172.21.15.89:0/3944493065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 msgr2=0x557c88320a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c88320a70 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fca480122c0 tx=0x7fca48005340 comp rx=0 tx=0).stop 2024-09-15T23:32:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 -- 172.21.15.89:0/3944493065 shutdown_connections 2024-09-15T23:32:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557c88332d40 0x557c88335130 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca36ff5640 0 client.18201 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c88320a70 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88314340 0x557c88320530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/3944493065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88313a20 0x557c8831fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 -- 172.21.15.89:0/3944493065 >> 172.21.15.89:0/3944493065 conn(0x557c883100f0 msgr2=0x557c88315e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:15.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 -- 172.21.15.89:0/3944493065 shutdown_connections 2024-09-15T23:32:15.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 1 -- 172.21.15.89:0/3944493065 wait complete. 2024-09-15T23:32:15.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.528+0000 7fca5b7fd740 20 client.18201 trim_cache size 0 max 16384 2024-09-15T23:32:15.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.545+0000 7fca5b7fd740 1 Processor -- start 2024-09-15T23:32:15.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.545+0000 7fca5b7fd740 1 -- start start 2024-09-15T23:32:15.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.545+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88313a20 0x557c8833a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.545+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88314340 0x557c8833ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.545+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c8831ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.545+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fca541acb60 con 0x557c88323150 2024-09-15T23:32:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.545+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fca541ae6a0 con 0x557c88313a20 2024-09-15T23:32:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.545+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fca541ad230 con 0x557c88314340 2024-09-15T23:32:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca527fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c8831ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca527fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c8831ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33432/0 (socket says 172.21.15.89:33432) 2024-09-15T23:32:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca527fc640 1 -- 172.21.15.89:0/18432231 learned_addr learned my addr 172.21.15.89:0/18432231 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca52ffd640 1 --2- 172.21.15.89:0/18432231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88314340 0x557c8833ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca527fc640 1 --2- 172.21.15.89:0/18432231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88313a20 0x557c8833a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca527fc640 1 -- 172.21.15.89:0/18432231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88314340 msgr2=0x557c8833ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca527fc640 1 --2- 172.21.15.89:0/18432231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88314340 0x557c8833ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca527fc640 1 -- 172.21.15.89:0/18432231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88313a20 msgr2=0x557c8833a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca527fc640 1 --2- 172.21.15.89:0/18432231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88313a20 0x557c8833a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca527fc640 1 -- 172.21.15.89:0/18432231 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c8830fb90 con 0x557c88323150 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca527fc640 1 --2- 172.21.15.89:0/18432231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c8831ffb0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fca4800bcc0 tx=0x7fca4802a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45c9546a32993c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca38ff9640 1 -- 172.21.15.89:0/18432231 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4801a7c0 con 0x557c88323150 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca38ff9640 1 -- 172.21.15.89:0/18432231 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca48016040 con 0x557c88323150 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.546+0000 7fca38ff9640 1 -- 172.21.15.89:0/18432231 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca48010a00 con 0x557c88323150 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.547+0000 7fca5b7fd740 1 -- 172.21.15.89:0/18432231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 msgr2=0x557c8831ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.547+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/18432231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c8831ffb0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fca4800bcc0 tx=0x7fca4802a040 comp rx=0 tx=0).stop 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.547+0000 7fca5b7fd740 1 -- 172.21.15.89:0/18432231 shutdown_connections 2024-09-15T23:32:15.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.547+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/18432231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88323150 0x557c8831ffb0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.547+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/18432231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88314340 0x557c8833ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.547+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/18432231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88313a20 0x557c8833a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.547+0000 7fca5b7fd740 1 -- 172.21.15.89:0/18432231 >> 172.21.15.89:0/18432231 conn(0x557c883100f0 msgr2=0x557c88321f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.547+0000 7fca5b7fd740 1 -- 172.21.15.89:0/18432231 shutdown_connections 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.547+0000 7fca5b7fd740 1 -- 172.21.15.89:0/18432231 wait complete. 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.547+0000 7fca5b7fd740 1 -- start start 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca5b7fd740 1 Processor -- start 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c8832b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88314340 0x557c8832b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca5b7fd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88323150 0x557c8831c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fca541adfd0 con 0x557c88313a20 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fca541ad900 con 0x557c88314340 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca5b7fd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fca541aab40 con 0x557c88323150 2024-09-15T23:32:15.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca5b7fd740 10 client.0 _ll_register_callbacks cb 0x7fca4800bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca527fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c8832b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca527fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c8832b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33444/0 (socket says 172.21.15.89:33444) 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca527fc640 1 -- 172.21.15.89:0/2446590599 learned_addr learned my addr 172.21.15.89:0/2446590599 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca52ffd640 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88314340 0x557c8832b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca527fc640 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88323150 0x557c8831c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca527fc640 1 -- 172.21.15.89:0/2446590599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88323150 msgr2=0x557c8831c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca527fc640 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88323150 0x557c8831c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca527fc640 1 -- 172.21.15.89:0/2446590599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88314340 msgr2=0x557c8832b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca527fc640 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88314340 0x557c8832b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.548+0000 7fca527fc640 1 -- 172.21.15.89:0/2446590599 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c8831d020 con 0x557c88313a20 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca527fc640 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c8832b340 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fca4800b560 tx=0x7fca4800b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2d9afe99800ee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:15.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca36ff5640 1 -- 172.21.15.89:0/2446590599 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4801abd0 con 0x557c88313a20 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca36ff5640 1 -- 172.21.15.89:0/2446590599 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fca4801ad70 con 0x557c88313a20 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca5b7fd740 1 -- 172.21.15.89:0/2446590599 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca48016040 con 0x557c88313a20 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca36ff5640 1 -- 172.21.15.89:0/2446590599 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fca48010430 con 0x557c88313a20 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca5b7fd740 10 client.18207 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca5b7fd740 1 -- 172.21.15.89:0/2446590599 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557c8839d400 con 0x557c88313a20 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca36ff5640 1 -- 172.21.15.89:0/2446590599 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca48016040 con 0x557c88313a20 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca36ff5640 1 -- 172.21.15.89:0/2446590599 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fca48026070 con 0x557c88313a20 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca5b7fd740 10 client.18207 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca5b7fd740 10 client.18207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca5b7fd740 20 client.18207 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557c88332d40 0x557c8839fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:15.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca5b7fd740 4 client.18207 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.549+0000 7fca5b7fd740 1 -- 172.21.15.89:0/2446590599 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557c88280d00 con 0x557c88332d40 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.550+0000 7fca52ffd640 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557c88332d40 0x557c8839fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.550+0000 7fca36ff5640 10 client.18207 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.550+0000 7fca52ffd640 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557c88332d40 0x557c8839fad0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42411b2e77bbf180 server_cookie=d481d3a0dcd5fa02 in_seq=0 out_seq=0 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.551+0000 7fca36ff5640 1 -- 172.21.15.89:0/2446590599 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557c88280d00 con 0x557c88332d40 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.551+0000 7fca36ff5640 10 client.18207 handle_command_reply: tid=0 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.551+0000 7fca5b7fd740 1 client.18207 shutdown 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 1 -- 172.21.15.89:0/2446590599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 msgr2=0x557c8832b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c8832b340 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fca4800b560 tx=0x7fca4800b7d0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 1 -- 172.21.15.89:0/2446590599 shutdown_connections 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557c88332d40 0x557c8839fad0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c88323150 0x557c8831c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c88314340 0x557c8832b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 1 --2- 172.21.15.89:0/2446590599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c88313a20 0x557c8832b340 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 1 -- 172.21.15.89:0/2446590599 >> 172.21.15.89:0/2446590599 conn(0x557c883100f0 msgr2=0x557c8830ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:15.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca36ff5640 0 client.18207 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:15.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 1 -- 172.21.15.89:0/2446590599 shutdown_connections 2024-09-15T23:32:15.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 1 -- 172.21.15.89:0/2446590599 wait complete. 2024-09-15T23:32:15.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca5b7fd740 20 client.18207 trim_cache size 0 max 16384 2024-09-15T23:32:15.564 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:32:15.564 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:32:15.564 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "11721a9e-5391-42a2-9eef-7de1d2d90ea6", 2024-09-15T23:32:15.564 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:32:15.565 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:32:15.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca59424640 1 -- 172.21.15.89:0/4278347862 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca3c0a3240 msgr2=0x7fca3c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:15.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca59424640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca3c0a3240 0x7fca3c0a5700 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fca4c032000 tx=0x7fca4c0095c0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca59424640 1 -- 172.21.15.89:0/4278347862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca54102ef0 msgr2=0x7fca541a4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.552+0000 7fca59424640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca54102ef0 0x7fca541a4340 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fca4c004af0 tx=0x7fca4c02f5b0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.553+0000 7fca59424640 1 -- 172.21.15.89:0/4278347862 shutdown_connections 2024-09-15T23:32:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.553+0000 7fca59424640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fca3c0a3240 0x7fca3c0a5700 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.553+0000 7fca59424640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fca54102ef0 0x7fca541a4340 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.553+0000 7fca59424640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca54102570 0x7fca54199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.553+0000 7fca59424640 1 --2- 172.21.15.89:0/4278347862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fca54100e20 0x7fca541992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.553+0000 7fca59424640 1 -- 172.21.15.89:0/4278347862 >> 172.21.15.89:0/4278347862 conn(0x7fca540fc7d0 msgr2=0x7fca540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.553+0000 7fca59424640 1 -- 172.21.15.89:0/4278347862 shutdown_connections 2024-09-15T23:32:15.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:15.553+0000 7fca59424640 1 -- 172.21.15.89:0/4278347862 wait complete. 2024-09-15T23:32:15.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:15 smithi120 ceph-mon[35250]: pgmap v597: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-09-15T23:32:16.292 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:32:16.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:16 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:32:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:16 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:32:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:16 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:32:16.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:16 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:32:16.542 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:16.553 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:16 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:32:16.553 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:16 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:32:16.553 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:16 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:32:16.553 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:16 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:32:16.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:16 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:32:16.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:16 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:32:16.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:16 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:32:16.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:16 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:32:17.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.058+0000 7f6038c24640 1 -- 172.21.15.89:0/1149989252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6034102790 msgr2=0x7f60341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:17.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.058+0000 7f6038c24640 1 --2- 172.21.15.89:0/1149989252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6034102790 0x7f60341067e0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f6028009900 tx=0x7f602802a410 comp rx=0 tx=0).stop 2024-09-15T23:32:17.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.058+0000 7f6038c24640 1 -- 172.21.15.89:0/1149989252 shutdown_connections 2024-09-15T23:32:17.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.058+0000 7f6038c24640 1 --2- 172.21.15.89:0/1149989252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6034106f10 0x7f603410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.058+0000 7f6038c24640 1 --2- 172.21.15.89:0/1149989252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6034102790 0x7f60341067e0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.058+0000 7f6038c24640 1 --2- 172.21.15.89:0/1149989252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6034101de0 0x7f60341021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.058+0000 7f6038c24640 1 -- 172.21.15.89:0/1149989252 >> 172.21.15.89:0/1149989252 conn(0x7f60340fd8c0 msgr2=0x7f60340ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.058+0000 7f6038c24640 1 -- 172.21.15.89:0/1149989252 shutdown_connections 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.058+0000 7f6038c24640 1 -- 172.21.15.89:0/1149989252 wait complete. 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.059+0000 7f6038c24640 1 Processor -- start 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.059+0000 7f6038c24640 1 -- start start 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.059+0000 7f6038c24640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6034101de0 0x7f6034198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6038c24640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6034102790 0x7f6034198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6038c24640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6034106f10 0x7f603419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6038c24640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6034111f90 con 0x7f6034106f10 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6038c24640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6034111e10 con 0x7f6034101de0 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6038c24640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6034112110 con 0x7f6034102790 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6032d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6034106f10 0x7f603419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6032d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6034106f10 0x7f603419ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33468/0 (socket says 172.21.15.89:33468) 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6032d9d640 1 -- 172.21.15.89:0/1647280196 learned_addr learned my addr 172.21.15.89:0/1647280196 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6031d9b640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6034102790 0x7f6034198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f603259c640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6034101de0 0x7f6034198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6032d9d640 1 -- 172.21.15.89:0/1647280196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6034102790 msgr2=0x7f6034198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6032d9d640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6034102790 0x7f6034198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6032d9d640 1 -- 172.21.15.89:0/1647280196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6034101de0 msgr2=0x7f6034198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:17.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6032d9d640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6034101de0 0x7f6034198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6032d9d640 1 -- 172.21.15.89:0/1647280196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603419d3d0 con 0x7f6034106f10 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f603259c640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6034101de0 0x7f6034198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f6032d9d640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6034106f10 0x7f603419ccf0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f602c00b970 tx=0x7f602c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.060+0000 7f60237fe640 1 -- 172.21.15.89:0/1647280196 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602c019070 con 0x7f6034106f10 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.061+0000 7f6038c24640 1 -- 172.21.15.89:0/1647280196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f603419d6c0 con 0x7f6034106f10 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.061+0000 7f60237fe640 1 -- 172.21.15.89:0/1647280196 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f602c00cb40 con 0x7f6034106f10 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.061+0000 7f60237fe640 1 -- 172.21.15.89:0/1647280196 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602c007650 con 0x7f6034106f10 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.061+0000 7f6038c24640 1 -- 172.21.15.89:0/1647280196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60341a4fa0 con 0x7f6034106f10 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.063+0000 7f60237fe640 1 -- 172.21.15.89:0/1647280196 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f602c01c070 con 0x7f6034106f10 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.063+0000 7f603affd740 1 Processor -- start 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.063+0000 7f603affd740 1 -- start start 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.063+0000 7f603affd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd522754a0 0x55fd52275880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.063+0000 7f603affd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd52280140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.063+0000 7f603affd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd52280680 0x55fd52282a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.063+0000 7f603affd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fd521db2e0 con 0x55fd52275dc0 2024-09-15T23:32:17.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.063+0000 7f603affd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fd52182ce0 con 0x55fd52280680 2024-09-15T23:32:17.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.063+0000 7f603affd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fd521d72a0 con 0x55fd522754a0 2024-09-15T23:32:17.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f60237fe640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f601c0a3240 0x7f601c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f603259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd52280140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f603259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd52280140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33478/0 (socket says 172.21.15.89:33478) 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f603259c640 1 -- 172.21.15.89:0/4066597241 learned_addr learned my addr 172.21.15.89:0/4066597241 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f6031d9b640 1 --2- 172.21.15.89:0/4066597241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd52280680 0x55fd52282a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f6032d9d640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f601c0a3240 0x7f601c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f60237fe640 1 -- 172.21.15.89:0/1647280196 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f602c0cf2e0 con 0x7f6034106f10 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f6031d9b640 1 --2- 172.21.15.89:0/4066597241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd522754a0 0x55fd52275880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f603259c640 1 -- 172.21.15.89:0/4066597241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd522754a0 msgr2=0x55fd52275880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f603259c640 1 --2- 172.21.15.89:0/4066597241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd522754a0 0x55fd52275880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f603259c640 1 -- 172.21.15.89:0/4066597241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd52280680 msgr2=0x55fd52282a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f603259c640 1 --2- 172.21.15.89:0/4066597241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd52280680 0x55fd52282a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f603259c640 1 -- 172.21.15.89:0/4066597241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd52167440 con 0x55fd52275dc0 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f6031d9b640 1 --2- 172.21.15.89:0/4066597241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd52280680 0x55fd52282a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:17.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f603259c640 1 --2- 172.21.15.89:0/4066597241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd52280140 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f6024008c60 tx=0x7f602400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ddc0a92bb3874d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f6032d9d640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f601c0a3240 0x7f601c0a5700 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f602c09b2c0 tx=0x7f602c066000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f6020ff9640 1 -- 172.21.15.89:0/4066597241 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024014070 con 0x55fd52275dc0 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f6020ff9640 1 -- 172.21.15.89:0/4066597241 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60240047a0 con 0x55fd52275dc0 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.064+0000 7f6020ff9640 1 -- 172.21.15.89:0/4066597241 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60240106e0 con 0x55fd52275dc0 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.065+0000 7f603affd740 1 -- 172.21.15.89:0/4066597241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 msgr2=0x55fd52280140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.065+0000 7f603affd740 1 --2- 172.21.15.89:0/4066597241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd52280140 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f6024008c60 tx=0x7f602400e720 comp rx=0 tx=0).stop 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.065+0000 7f603affd740 1 -- 172.21.15.89:0/4066597241 shutdown_connections 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.065+0000 7f603affd740 1 --2- 172.21.15.89:0/4066597241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd52280680 0x55fd52282a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.065+0000 7f603affd740 1 --2- 172.21.15.89:0/4066597241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd52280140 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.065+0000 7f603affd740 1 --2- 172.21.15.89:0/4066597241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd522754a0 0x55fd52275880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.065+0000 7f603affd740 1 -- 172.21.15.89:0/4066597241 >> 172.21.15.89:0/4066597241 conn(0x55fd52271a30 msgr2=0x55fd52271e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.065+0000 7f603affd740 1 -- 172.21.15.89:0/4066597241 shutdown_connections 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.065+0000 7f603affd740 1 -- 172.21.15.89:0/4066597241 wait complete. 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.065+0000 7f603affd740 1 -- start start 2024-09-15T23:32:17.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603affd740 1 Processor -- start 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603affd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd5227ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603affd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd5227d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603affd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52280680 0x55fd5227d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603affd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fd52283250 con 0x55fd52275dc0 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603affd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fd52196050 con 0x55fd522754a0 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603affd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fd522833d0 con 0x55fd52280680 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603affd740 10 client.0 _ll_register_callbacks cb 0x55fd521d6160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd5227d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd5227d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33488/0 (socket says 172.21.15.89:33488) 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603259c640 1 -- 172.21.15.89:0/406214475 learned_addr learned my addr 172.21.15.89:0/406214475 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603259c640 1 -- 172.21.15.89:0/406214475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52280680 msgr2=0x55fd5227d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:17.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603259c640 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52280680 0x55fd5227d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603259c640 1 -- 172.21.15.89:0/406214475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 msgr2=0x55fd5227ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f6031d9b640 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd5227ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603259c640 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd5227ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603259c640 1 -- 172.21.15.89:0/406214475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd51f4c3c0 con 0x55fd52275dc0 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f603259c640 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd5227d220 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f6024000c00 tx=0x7f6024004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b9cead1e78ebed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.066+0000 7f6019ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f6019ffb640 1 -- 172.21.15.89:0/406214475 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024027060 con 0x55fd52275dc0 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f6019ffb640 1 -- 172.21.15.89:0/406214475 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6024005340 con 0x55fd52275dc0 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f603affd740 1 -- 172.21.15.89:0/406214475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60240047a0 con 0x55fd52275dc0 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f6019ffb640 1 -- 172.21.15.89:0/406214475 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f602402e050 con 0x55fd52275dc0 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f603affd740 10 client.18219 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f603affd740 1 -- 172.21.15.89:0/406214475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fd5228ebc0 con 0x55fd52275dc0 2024-09-15T23:32:17.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f6019ffb640 1 -- 172.21.15.89:0/406214475 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60240047a0 con 0x55fd52275dc0 2024-09-15T23:32:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f6019ffb640 1 -- 172.21.15.89:0/406214475 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6024022530 con 0x55fd52275dc0 2024-09-15T23:32:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f603affd740 10 client.18219 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f603affd740 10 client.18219 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f603affd740 20 client.18219 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f603affd740 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fd52290080 0x55fd52292470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f603affd740 4 client.18219 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.067+0000 7f603affd740 1 -- 172.21.15.89:0/406214475 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd521e8850 con 0x55fd52290080 2024-09-15T23:32:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.068+0000 7f6031d9b640 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fd52290080 0x55fd52292470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.068+0000 7f6031d9b640 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fd52290080 0x55fd52292470 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9e894ea4172ddd server_cookie=887c3f02882ad5c4 in_seq=0 out_seq=0 2024-09-15T23:32:17.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.068+0000 7f6019ffb640 10 client.18219 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f6019ffb640 1 -- 172.21.15.89:0/406214475 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fd521e8850 con 0x55fd52290080 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f6019ffb640 10 client.18219 handle_command_reply: tid=0 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 client.18219 shutdown 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 -- 172.21.15.89:0/406214475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 msgr2=0x55fd5227d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd5227d220 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f6024000c00 tx=0x7f6024004730 comp rx=0 tx=0).stop 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 -- 172.21.15.89:0/406214475 shutdown_connections 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fd52290080 0x55fd52292470 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52280680 0x55fd5227d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd5227d220 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 --2- 172.21.15.89:0/406214475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd5227ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 -- 172.21.15.89:0/406214475 >> 172.21.15.89:0/406214475 conn(0x55fd52271a30 msgr2=0x55fd52277710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f6019ffb640 0 client.18219 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 -- 172.21.15.89:0/406214475 shutdown_connections 2024-09-15T23:32:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 1 -- 172.21.15.89:0/406214475 wait complete. 2024-09-15T23:32:17.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.069+0000 7f603affd740 20 client.18219 trim_cache size 0 max 16384 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603affd740 1 Processor -- start 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603affd740 1 -- start start 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603affd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd5227f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603affd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd522950c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603affd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52280680 0x55fd52295600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603affd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f60341aee00 con 0x55fd52275dc0 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603affd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f60341b0940 con 0x55fd522754a0 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603affd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f60341af4d0 con 0x55fd52280680 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd522950c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd522950c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33496/0 (socket says 172.21.15.89:33496) 2024-09-15T23:32:17.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.091+0000 7f603259c640 1 -- 172.21.15.89:0/3713062228 learned_addr learned my addr 172.21.15.89:0/3713062228 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.092+0000 7f603259c640 1 -- 172.21.15.89:0/3713062228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52280680 msgr2=0x55fd52295600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.092+0000 7f603259c640 1 --2- 172.21.15.89:0/3713062228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52280680 0x55fd52295600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.092+0000 7f603259c640 1 -- 172.21.15.89:0/3713062228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 msgr2=0x55fd5227f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.092+0000 7f603259c640 1 --2- 172.21.15.89:0/3713062228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd5227f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.092+0000 7f603259c640 1 -- 172.21.15.89:0/3713062228 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd52276330 con 0x55fd52275dc0 2024-09-15T23:32:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.092+0000 7f603259c640 1 --2- 172.21.15.89:0/3713062228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd522950c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f6024008b80 tx=0x7f6024004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30b1fb1127214b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.092+0000 7f601bfff640 1 -- 172.21.15.89:0/3713062228 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024037070 con 0x55fd52275dc0 2024-09-15T23:32:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.092+0000 7f601bfff640 1 -- 172.21.15.89:0/3713062228 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60240047a0 con 0x55fd52275dc0 2024-09-15T23:32:17.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.092+0000 7f601bfff640 1 -- 172.21.15.89:0/3713062228 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602401c030 con 0x55fd52275dc0 2024-09-15T23:32:17.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 -- 172.21.15.89:0/3713062228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 msgr2=0x55fd522950c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 --2- 172.21.15.89:0/3713062228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd522950c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f6024008b80 tx=0x7f6024004600 comp rx=0 tx=0).stop 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 -- 172.21.15.89:0/3713062228 shutdown_connections 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 --2- 172.21.15.89:0/3713062228 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52280680 0x55fd52295600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 --2- 172.21.15.89:0/3713062228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52275dc0 0x55fd522950c0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 --2- 172.21.15.89:0/3713062228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd5227f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 -- 172.21.15.89:0/3713062228 >> 172.21.15.89:0/3713062228 conn(0x55fd52271a30 msgr2=0x55fd5227ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 -- 172.21.15.89:0/3713062228 shutdown_connections 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 -- 172.21.15.89:0/3713062228 wait complete. 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 -- start start 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 Processor -- start 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd522947a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52275dc0 0x55fd52294ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52280680 0x55fd522f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f60341b0270 con 0x55fd52280680 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f60341afba0 con 0x55fd522754a0 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f60341acde0 con 0x55fd52275dc0 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.093+0000 7f603affd740 10 client.0 _ll_register_callbacks cb 0x7f6024008ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6031d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52280680 0x55fd522f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6031d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52280680 0x55fd522f6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33510/0 (socket says 172.21.15.89:33510) 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6031d9b640 1 -- 172.21.15.89:0/1303387900 learned_addr learned my addr 172.21.15.89:0/1303387900 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f603259c640 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52275dc0 0x55fd52294ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6031d9b640 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd522947a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6031d9b640 1 -- 172.21.15.89:0/1303387900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52275dc0 msgr2=0x55fd52294ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6031d9b640 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52275dc0 0x55fd52294ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6031d9b640 1 -- 172.21.15.89:0/1303387900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 msgr2=0x55fd522947a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6031d9b640 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd522947a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6031d9b640 1 -- 172.21.15.89:0/1303387900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd522f7500 con 0x55fd52280680 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6031d9b640 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52280680 0x55fd522f6e10 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f602802a930 tx=0x7f6028000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540e0fdf7e3151ec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6019ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6019ffb640 1 -- 172.21.15.89:0/1303387900 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6028046040 con 0x55fd52280680 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6019ffb640 1 -- 172.21.15.89:0/1303387900 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6028004030 con 0x55fd52280680 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f603affd740 1 -- 172.21.15.89:0/1303387900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60240047a0 con 0x55fd52280680 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f6019ffb640 1 -- 172.21.15.89:0/1303387900 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6028055050 con 0x55fd52280680 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f603affd740 10 client.18225 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.094+0000 7f603affd740 1 -- 172.21.15.89:0/1303387900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fd522fe890 con 0x55fd52280680 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.095+0000 7f6019ffb640 1 -- 172.21.15.89:0/1303387900 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60240047a0 con 0x55fd52280680 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.095+0000 7f6019ffb640 1 -- 172.21.15.89:0/1303387900 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f602803d450 con 0x55fd52280680 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.095+0000 7f603affd740 10 client.18225 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.095+0000 7f603affd740 10 client.18225 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:17.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.095+0000 7f603affd740 20 client.18225 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.095+0000 7f603affd740 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fd522fef90 0x55fd523013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.095+0000 7f603affd740 4 client.18225 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.095+0000 7f603affd740 1 -- 172.21.15.89:0/1303387900 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fd5227daf0 con 0x55fd522fef90 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.095+0000 7f603259c640 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fd522fef90 0x55fd523013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f6019ffb640 10 client.18225 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603259c640 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fd522fef90 0x55fd523013b0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aaaa30226beda3c server_cookie=20147d4ac209fb3d in_seq=0 out_seq=0 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f6019ffb640 1 -- 172.21.15.89:0/1303387900 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fd5227daf0 con 0x55fd522fef90 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f6019ffb640 10 client.18225 handle_command_reply: tid=0 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 client.18225 shutdown 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 -- 172.21.15.89:0/1303387900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52280680 msgr2=0x55fd522f6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52280680 0x55fd522f6e10 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f602802a930 tx=0x7f6028000bc0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 -- 172.21.15.89:0/1303387900 shutdown_connections 2024-09-15T23:32:17.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fd522fef90 0x55fd523013b0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fd52280680 0x55fd522f6e10 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fd52275dc0 0x55fd52294ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 --2- 172.21.15.89:0/1303387900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fd522754a0 0x55fd522947a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 -- 172.21.15.89:0/1303387900 >> 172.21.15.89:0/1303387900 conn(0x55fd52271a30 msgr2=0x55fd52270350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f6019ffb640 0 client.18225 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 -- 172.21.15.89:0/1303387900 shutdown_connections 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 1 -- 172.21.15.89:0/1303387900 wait complete. 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.096+0000 7f603affd740 20 client.18225 trim_cache size 0 max 16384 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:32:17.102 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:32:17.103 INFO:teuthology.orchestra.run.smithi089.stdout: "11721a9e-5391-42a2-9eef-7de1d2d90ea6": { 2024-09-15T23:32:17.103 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:32:17.103 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "11721a9e-5391-42a2-9eef-7de1d2d90ea6", 2024-09-15T23:32:17.103 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:32:17.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:32:17.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:32:17.103 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 -- 172.21.15.89:0/1647280196 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f601c0a3240 msgr2=0x7f601c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f601c0a3240 0x7f601c0a5700 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f602c09b2c0 tx=0x7f602c066000 comp rx=0 tx=0).stop 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 -- 172.21.15.89:0/1647280196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6034106f10 msgr2=0x7f603419ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6034106f10 0x7f603419ccf0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f602c00b970 tx=0x7f602c00be30 comp rx=0 tx=0).stop 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 -- 172.21.15.89:0/1647280196 shutdown_connections 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f601c0a3240 0x7f601c0a5700 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6034106f10 0x7f603419ccf0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6034102790 0x7f6034198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 --2- 172.21.15.89:0/1647280196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6034101de0 0x7f6034198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 -- 172.21.15.89:0/1647280196 >> 172.21.15.89:0/1647280196 conn(0x7f60340fd8c0 msgr2=0x7f60340fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 -- 172.21.15.89:0/1647280196 shutdown_connections 2024-09-15T23:32:17.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:17.097+0000 7f6038c24640 1 -- 172.21.15.89:0/1647280196 wait complete. 2024-09-15T23:32:17.464 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:17 smithi089 ceph-mon[30933]: pgmap v598: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-09-15T23:32:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:17 smithi174 ceph-mon[35586]: pgmap v598: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-09-15T23:32:17.625 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:17 smithi120 ceph-mon[35250]: pgmap v598: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.2 KiB/s wr, 15 op/s 2024-09-15T23:32:17.869 INFO:tasks.cephfs.filesystem:scrub status for tag:11721a9e-5391-42a2-9eef-7de1d2d90ea6 - {'path': '/', 'tag': '11721a9e-5391-42a2-9eef-7de1d2d90ea6', 'options': 'recursive,force'} 2024-09-15T23:32:18.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:18 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:32:18.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:18 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:32:18.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:18 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:32:18.785 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:32:19.048 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:19.462 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:19 smithi089 ceph-mon[30933]: pgmap v599: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-15T23:32:19.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.539+0000 7f93a863e640 1 -- 172.21.15.89:0/3057908332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a010c170 msgr2=0x7f93a010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:19.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.539+0000 7f93a863e640 1 --2- 172.21.15.89:0/3057908332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a010c170 0x7f93a010e600 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9390009ca0 tx=0x7f939002a950 comp rx=0 tx=0).stop 2024-09-15T23:32:19.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.539+0000 7f93a863e640 1 -- 172.21.15.89:0/3057908332 shutdown_connections 2024-09-15T23:32:19.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.539+0000 7f93a863e640 1 --2- 172.21.15.89:0/3057908332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a010c170 0x7f93a010e600 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:19.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.539+0000 7f93a863e640 1 --2- 172.21.15.89:0/3057908332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a01036f0 0x7f93a010bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:19.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.539+0000 7f93a863e640 1 --2- 172.21.15.89:0/3057908332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a0102dd0 0x7f93a01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:19.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.539+0000 7f93a863e640 1 -- 172.21.15.89:0/3057908332 >> 172.21.15.89:0/3057908332 conn(0x7f93a00fc790 msgr2=0x7f93a00febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.539+0000 7f93a863e640 1 -- 172.21.15.89:0/3057908332 shutdown_connections 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.539+0000 7f93a863e640 1 -- 172.21.15.89:0/3057908332 wait complete. 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a863e640 1 Processor -- start 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a863e640 1 -- start start 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a863e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a0102dd0 0x7f93a0198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a863e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a01036f0 0x7f93a0198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a863e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a010c170 0x7f93a019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a863e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f93a0111f60 con 0x7f93a0102dd0 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a863e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93a0111de0 con 0x7f93a01036f0 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a863e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93a01120e0 con 0x7f93a010c170 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a63da640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a0102dd0 0x7f93a0198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a63da640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a0102dd0 0x7f93a0198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33530/0 (socket says 172.21.15.89:33530) 2024-09-15T23:32:19.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.540+0000 7f93a63da640 1 -- 172.21.15.89:0/4005619919 learned_addr learned my addr 172.21.15.89:0/4005619919 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f93a5bd9640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a01036f0 0x7f93a0198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f93a63da640 1 -- 172.21.15.89:0/4005619919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a010c170 msgr2=0x7f93a019cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f93a63da640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a010c170 0x7f93a019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f93a63da640 1 -- 172.21.15.89:0/4005619919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a01036f0 msgr2=0x7f93a0198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f93a63da640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a01036f0 0x7f93a0198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f93a63da640 1 -- 172.21.15.89:0/4005619919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93a019d2e0 con 0x7f93a0102dd0 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f93a5bd9640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a01036f0 0x7f93a0198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f93a63da640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a0102dd0 0x7f93a0198330 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f939800d950 tx=0x7f939800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f939f7fe640 1 -- 172.21.15.89:0/4005619919 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9398017070 con 0x7f93a0102dd0 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f93a863e640 1 -- 172.21.15.89:0/4005619919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93a019d5d0 con 0x7f93a0102dd0 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f939f7fe640 1 -- 172.21.15.89:0/4005619919 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9398009c30 con 0x7f93a0102dd0 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f939f7fe640 1 -- 172.21.15.89:0/4005619919 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939800f6b0 con 0x7f93a0102dd0 2024-09-15T23:32:19.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.541+0000 7f93a863e640 1 -- 172.21.15.89:0/4005619919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93a01a4eb0 con 0x7f93a0102dd0 2024-09-15T23:32:19.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.542+0000 7f93a863e640 1 -- 172.21.15.89:0/4005619919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93a01031b0 con 0x7f93a0102dd0 2024-09-15T23:32:19.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.543+0000 7f939f7fe640 1 -- 172.21.15.89:0/4005619919 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f93980109d0 con 0x7f93a0102dd0 2024-09-15T23:32:19.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.544+0000 7f939f7fe640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93880a3240 0x7f93880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:19.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.544+0000 7f93a5bd9640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93880a3240 0x7f93880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:19.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.544+0000 7f939f7fe640 1 -- 172.21.15.89:0/4005619919 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f93980cf9e0 con 0x7f93a0102dd0 2024-09-15T23:32:19.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.544+0000 7f93a5bd9640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93880a3240 0x7f93880a5700 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f93a00fed20 tx=0x7f9394005d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:19.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.547+0000 7f939f7fe640 1 -- 172.21.15.89:0/4005619919 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9398009de0 con 0x7f93a0102dd0 2024-09-15T23:32:19.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:19 smithi120 ceph-mon[35250]: pgmap v599: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-15T23:32:19.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:19 smithi174 ceph-mon[35586]: pgmap v599: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-09-15T23:32:19.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.749+0000 7f93a863e640 1 -- 172.21.15.89:0/4005619919 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93a01a51a0 con 0x7f93a0102dd0 2024-09-15T23:32:19.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.750+0000 7f939f7fe640 1 -- 172.21.15.89:0/4005619919 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f939809c310 con 0x7f93a0102dd0 2024-09-15T23:32:19.751 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:32:19.751 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:32:19.752 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:32:19.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 -- 172.21.15.89:0/4005619919 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93880a3240 msgr2=0x7f93880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:19.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93880a3240 0x7f93880a5700 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f93a00fed20 tx=0x7f9394005d20 comp rx=0 tx=0).stop 2024-09-15T23:32:19.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 -- 172.21.15.89:0/4005619919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a0102dd0 msgr2=0x7f93a0198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:19.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a0102dd0 0x7f93a0198330 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f939800d950 tx=0x7f939800de10 comp rx=0 tx=0).stop 2024-09-15T23:32:19.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 -- 172.21.15.89:0/4005619919 shutdown_connections 2024-09-15T23:32:19.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f93880a3240 0x7f93880a5700 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:19.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93a010c170 0x7f93a019cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:19.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93a01036f0 0x7f93a0198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:19.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 --2- 172.21.15.89:0/4005619919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f93a0102dd0 0x7f93a0198330 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:19.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 -- 172.21.15.89:0/4005619919 >> 172.21.15.89:0/4005619919 conn(0x7f93a00fc790 msgr2=0x7f93a00fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:19.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 -- 172.21.15.89:0/4005619919 shutdown_connections 2024-09-15T23:32:19.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:19.752+0000 7f93a863e640 1 -- 172.21.15.89:0/4005619919 wait complete. 2024-09-15T23:32:20.403 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 6514e407 --release --await' 2024-09-15T23:32:20.403 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 6514e407 --release --await 2024-09-15T23:32:20.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:20 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/4005619919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:20.647 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:20.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:20 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/4005619919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:20.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:20 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/4005619919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 -- 172.21.15.89:0/4092043194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9850102d50 msgr2=0x7f98501031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 --2- 172.21.15.89:0/4092043194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9850102d50 0x7f98501031d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f9838009900 tx=0x7f983802a410 comp rx=0 tx=0).stop 2024-09-15T23:32:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 -- 172.21.15.89:0/4092043194 shutdown_connections 2024-09-15T23:32:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 --2- 172.21.15.89:0/4092043194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9850103710 0x7f9850109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 --2- 172.21.15.89:0/4092043194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9850102d50 0x7f98501031d0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 --2- 172.21.15.89:0/4092043194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9850101b60 0x7f9850101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 -- 172.21.15.89:0/4092043194 >> 172.21.15.89:0/4092043194 conn(0x7f98500fd2e0 msgr2=0x7f98500ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:20.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 -- 172.21.15.89:0/4092043194 shutdown_connections 2024-09-15T23:32:20.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 -- 172.21.15.89:0/4092043194 wait complete. 2024-09-15T23:32:20.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 Processor -- start 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.990+0000 7f985643c640 1 -- start start 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f985643c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9850101b60 0x7f9850197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f985643c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9850102d50 0x7f9850198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f985643c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9850103710 0x7f985019f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f985643c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f985010d830 con 0x7f9850101b60 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f985643c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f985010d6b0 con 0x7f9850102d50 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f985643c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f985010d9b0 con 0x7f9850103710 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9850101b60 0x7f9850197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9850101b60 0x7f9850197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33548/0 (socket says 172.21.15.89:33548) 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984ffff640 1 -- 172.21.15.89:0/3708069292 learned_addr learned my addr 172.21.15.89:0/3708069292 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984f7fe640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9850102d50 0x7f9850198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f98549d9640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9850103710 0x7f985019f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:20.994 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984ffff640 1 -- 172.21.15.89:0/3708069292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9850103710 msgr2=0x7f985019f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984ffff640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9850103710 0x7f985019f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984ffff640 1 -- 172.21.15.89:0/3708069292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9850102d50 msgr2=0x7f9850198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984ffff640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9850102d50 0x7f9850198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984ffff640 1 -- 172.21.15.89:0/3708069292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985019f9e0 con 0x7f9850101b60 2024-09-15T23:32:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984ffff640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9850101b60 0x7f9850197dc0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f984000d770 tx=0x7f984000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984d7fa640 1 -- 172.21.15.89:0/3708069292 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840017070 con 0x7f9850101b60 2024-09-15T23:32:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984d7fa640 1 -- 172.21.15.89:0/3708069292 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9840009c30 con 0x7f9850101b60 2024-09-15T23:32:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f984d7fa640 1 -- 172.21.15.89:0/3708069292 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840003dd0 con 0x7f9850101b60 2024-09-15T23:32:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.991+0000 7f985643c640 1 -- 172.21.15.89:0/3708069292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f985019fcd0 con 0x7f9850101b60 2024-09-15T23:32:20.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.992+0000 7f985643c640 1 -- 172.21.15.89:0/3708069292 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98501a01a0 con 0x7f9850101b60 2024-09-15T23:32:21.001 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.994+0000 7f984d7fa640 1 -- 172.21.15.89:0/3708069292 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9840003f70 con 0x7f9850101b60 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.994+0000 7f98591d4740 1 Processor -- start 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.994+0000 7f98591d4740 1 -- start start 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.994+0000 7f98591d4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 0x5605ca111da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.994+0000 7f98591d4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1122e0 0x5605ca116da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.994+0000 7f98591d4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca117370 0x5605ca119830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.994+0000 7f98591d4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5605ca0752f0 con 0x5605ca1122e0 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.994+0000 7f98591d4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5605ca01d270 con 0x5605ca1119a0 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.994+0000 7f98591d4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5605ca0712b0 con 0x5605ca117370 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1122e0 0x5605ca116da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1122e0 0x5605ca116da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33552/0 (socket says 172.21.15.89:33552) 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984ffff640 1 -- 172.21.15.89:0/118914598 learned_addr learned my addr 172.21.15.89:0/118914598 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:21.002 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f98549d9640 1 --2- 172.21.15.89:0/118914598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 0x5605ca111da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984d7fa640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f983c0a32f0 0x7f983c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984f7fe640 1 --2- 172.21.15.89:0/118914598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca117370 0x5605ca119830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984ffff640 1 -- 172.21.15.89:0/118914598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca117370 msgr2=0x5605ca119830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984ffff640 1 --2- 172.21.15.89:0/118914598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca117370 0x5605ca119830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984ffff640 1 -- 172.21.15.89:0/118914598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 msgr2=0x5605ca111da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984ffff640 1 --2- 172.21.15.89:0/118914598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 0x5605ca111da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984ffff640 1 -- 172.21.15.89:0/118914598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ca01c340 con 0x5605ca1122e0 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f98549d9640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f983c0a32f0 0x7f983c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984d7fa640 1 -- 172.21.15.89:0/3708069292 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f98400cf7c0 con 0x7f9850101b60 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f984ffff640 1 --2- 172.21.15.89:0/118914598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1122e0 0x5605ca116da0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f984000ea60 tx=0x7f9840090ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d3a77827e3db0f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f98467fc640 1 -- 172.21.15.89:0/118914598 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984009b530 con 0x5605ca1122e0 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f98467fc640 1 -- 172.21.15.89:0/118914598 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98400713c0 con 0x5605ca1122e0 2024-09-15T23:32:21.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.995+0000 7f98467fc640 1 -- 172.21.15.89:0/118914598 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984007a3c0 con 0x5605ca1122e0 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98591d4740 1 -- 172.21.15.89:0/118914598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1122e0 msgr2=0x5605ca116da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98591d4740 1 --2- 172.21.15.89:0/118914598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1122e0 0x5605ca116da0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f984000ea60 tx=0x7f9840090ed0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98549d9640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f983c0a32f0 0x7f983c0a57b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9848005fd0 tx=0x7f9848004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98591d4740 1 -- 172.21.15.89:0/118914598 shutdown_connections 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98591d4740 1 --2- 172.21.15.89:0/118914598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca117370 0x5605ca119830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98591d4740 1 --2- 172.21.15.89:0/118914598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1122e0 0x5605ca116da0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98591d4740 1 --2- 172.21.15.89:0/118914598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 0x5605ca111da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98591d4740 1 -- 172.21.15.89:0/118914598 >> 172.21.15.89:0/118914598 conn(0x5605ca10d010 msgr2=0x5605ca10d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98591d4740 1 -- 172.21.15.89:0/118914598 shutdown_connections 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98591d4740 1 -- 172.21.15.89:0/118914598 wait complete. 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.996+0000 7f98591d4740 1 -- start start 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98591d4740 1 Processor -- start 2024-09-15T23:32:21.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98591d4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 0x5605ca11bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98591d4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1122e0 0x5605ca11c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98591d4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca123040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98591d4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5605ca11a280 con 0x5605ca117370 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98591d4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5605c9ffe290 con 0x5605ca1119a0 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98591d4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5605ca11a400 con 0x5605ca1122e0 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98591d4740 10 client.0 _ll_register_callbacks cb 0x5605ca02f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98549d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca123040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98549d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca123040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33562/0 (socket says 172.21.15.89:33562) 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98549d9640 1 -- 172.21.15.89:0/2818816389 learned_addr learned my addr 172.21.15.89:0/2818816389 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f984f7fe640 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1122e0 0x5605ca11c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f984ffff640 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 0x5605ca11bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98549d9640 1 -- 172.21.15.89:0/2818816389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1122e0 msgr2=0x5605ca11c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98549d9640 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1122e0 0x5605ca11c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98549d9640 1 -- 172.21.15.89:0/2818816389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 msgr2=0x5605ca11bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98549d9640 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 0x5605ca11bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98549d9640 1 -- 172.21.15.89:0/2818816389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605c9d88210 con 0x5605ca117370 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f984ffff640 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 0x5605ca11bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f98549d9640 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca123040 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9848013970 tx=0x7f9848004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf22da3915c1525f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.997+0000 7f982f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f982f7fe640 1 -- 172.21.15.89:0/2818816389 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984801ed70 con 0x5605ca117370 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f982f7fe640 1 -- 172.21.15.89:0/2818816389 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9848028470 con 0x5605ca117370 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f98591d4740 1 -- 172.21.15.89:0/2818816389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98400713c0 con 0x5605ca117370 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f982f7fe640 1 -- 172.21.15.89:0/2818816389 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9848031050 con 0x5605ca117370 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f98591d4740 10 client.18243 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f98591d4740 1 -- 172.21.15.89:0/2818816389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5605c9e54620 con 0x5605ca117370 2024-09-15T23:32:21.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f982f7fe640 1 -- 172.21.15.89:0/2818816389 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98400713c0 con 0x5605ca117370 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f982f7fe640 1 -- 172.21.15.89:0/2818816389 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f984800f040 con 0x5605ca117370 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f98591d4740 10 client.18243 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f98591d4740 10 client.18243 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f98591d4740 20 client.18243 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f98591d4740 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5605ca1277c0 0x5605ca129c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f98591d4740 4 client.18243 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f98591d4740 1 -- 172.21.15.89:0/2818816389 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605ca082a20 con 0x5605ca1277c0 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.998+0000 7f984ffff640 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5605ca1277c0 0x5605ca129c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.999+0000 7f982f7fe640 10 client.18243 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.999+0000 7f984ffff640 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5605ca1277c0 0x5605ca129c50 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a67100d0f017e755 server_cookie=6ec2d592379d2c0f in_seq=0 out_seq=0 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.999+0000 7f982f7fe640 1 -- 172.21.15.89:0/2818816389 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5605ca082a20 con 0x5605ca1277c0 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:20.999+0000 7f982f7fe640 10 client.18243 handle_command_reply: tid=0 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 client.18243 shutdown 2024-09-15T23:32:21.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 -- 172.21.15.89:0/2818816389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 msgr2=0x5605ca123040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca123040 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9848013970 tx=0x7f9848004ab0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 -- 172.21.15.89:0/2818816389 shutdown_connections 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5605ca1277c0 0x5605ca129c50 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca123040 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1122e0 0x5605ca11c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f982f7fe640 0 client.18243 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 --2- 172.21.15.89:0/2818816389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1119a0 0x5605ca11bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 -- 172.21.15.89:0/2818816389 >> 172.21.15.89:0/2818816389 conn(0x5605ca10d010 msgr2=0x5605ca10e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 -- 172.21.15.89:0/2818816389 shutdown_connections 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 1 -- 172.21.15.89:0/2818816389 wait complete. 2024-09-15T23:32:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.000+0000 7f98591d4740 20 client.18243 trim_cache size 0 max 16384 2024-09-15T23:32:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98591d4740 1 Processor -- start 2024-09-15T23:32:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98591d4740 1 -- start start 2024-09-15T23:32:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98591d4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1119a0 0x5605ca10c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98591d4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1122e0 0x5605ca123000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98591d4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca10e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98591d4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98501aa8b0 con 0x5605ca117370 2024-09-15T23:32:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98591d4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98501ac3f0 con 0x5605ca1122e0 2024-09-15T23:32:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98591d4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98501aaf80 con 0x5605ca1119a0 2024-09-15T23:32:21.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98549d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca10e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98549d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca10e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33566/0 (socket says 172.21.15.89:33566) 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98549d9640 1 -- 172.21.15.89:0/1662458454 learned_addr learned my addr 172.21.15.89:0/1662458454 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f984ffff640 1 --2- 172.21.15.89:0/1662458454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1119a0 0x5605ca10c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98549d9640 1 -- 172.21.15.89:0/1662458454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1119a0 msgr2=0x5605ca10c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98549d9640 1 --2- 172.21.15.89:0/1662458454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1119a0 0x5605ca10c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98549d9640 1 -- 172.21.15.89:0/1662458454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1122e0 msgr2=0x5605ca123000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f984f7fe640 1 --2- 172.21.15.89:0/1662458454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1122e0 0x5605ca123000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98549d9640 1 --2- 172.21.15.89:0/1662458454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1122e0 0x5605ca123000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.014+0000 7f98549d9640 1 -- 172.21.15.89:0/1662458454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ca10ce30 con 0x5605ca117370 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f98549d9640 1 --2- 172.21.15.89:0/1662458454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca10e400 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f984801eff0 tx=0x7f984801e7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4205a06b76b86a3f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f9845ffb640 1 -- 172.21.15.89:0/1662458454 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984803a070 con 0x5605ca117370 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f9845ffb640 1 -- 172.21.15.89:0/1662458454 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9848023070 con 0x5605ca117370 2024-09-15T23:32:21.017 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f9845ffb640 1 -- 172.21.15.89:0/1662458454 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848028950 con 0x5605ca117370 2024-09-15T23:32:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f98591d4740 1 -- 172.21.15.89:0/1662458454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 msgr2=0x5605ca10e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f98591d4740 1 --2- 172.21.15.89:0/1662458454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca10e400 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f984801eff0 tx=0x7f984801e7b0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f98591d4740 1 -- 172.21.15.89:0/1662458454 shutdown_connections 2024-09-15T23:32:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f98591d4740 1 --2- 172.21.15.89:0/1662458454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca117370 0x5605ca10e400 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f98591d4740 1 --2- 172.21.15.89:0/1662458454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca1122e0 0x5605ca123000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f98591d4740 1 --2- 172.21.15.89:0/1662458454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1119a0 0x5605ca10c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f98591d4740 1 -- 172.21.15.89:0/1662458454 >> 172.21.15.89:0/1662458454 conn(0x5605ca10d010 msgr2=0x5605ca10b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f98591d4740 1 -- 172.21.15.89:0/1662458454 shutdown_connections 2024-09-15T23:32:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.015+0000 7f98591d4740 1 -- 172.21.15.89:0/1662458454 wait complete. 2024-09-15T23:32:21.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98591d4740 1 -- start start 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98591d4740 1 Processor -- start 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98591d4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1119a0 0x5605ca120900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98591d4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1122e0 0x5605ca120e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98591d4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca117370 0x5605ca121e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98591d4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98501abd20 con 0x5605ca1119a0 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98591d4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98501ab650 con 0x5605ca117370 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98591d4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98501a8890 con 0x5605ca1122e0 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98591d4740 10 client.0 _ll_register_callbacks cb 0x7f9848009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f984ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1119a0 0x5605ca120900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f984ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1119a0 0x5605ca120900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33574/0 (socket says 172.21.15.89:33574) 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f984ffff640 1 -- 172.21.15.89:0/2987889524 learned_addr learned my addr 172.21.15.89:0/2987889524 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98549d9640 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca117370 0x5605ca121e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f984ffff640 1 -- 172.21.15.89:0/2987889524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1122e0 msgr2=0x5605ca120e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f984ffff640 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1122e0 0x5605ca120e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f984ffff640 1 -- 172.21.15.89:0/2987889524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca117370 msgr2=0x5605ca121e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f984ffff640 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca117370 0x5605ca121e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f984ffff640 1 -- 172.21.15.89:0/2987889524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605ca122460 con 0x5605ca1119a0 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.016+0000 7f98549d9640 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca117370 0x5605ca121e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f984ffff640 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1119a0 0x5605ca120900 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f984000d740 tx=0x7f9840004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86daa59ede21662 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f982f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f982f7fe640 1 -- 172.21.15.89:0/2987889524 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984009b350 con 0x5605ca1119a0 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f98591d4740 1 -- 172.21.15.89:0/2987889524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9848023070 con 0x5605ca1119a0 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f982f7fe640 1 -- 172.21.15.89:0/2987889524 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9840099a90 con 0x5605ca1119a0 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f982f7fe640 1 -- 172.21.15.89:0/2987889524 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f98400b4050 con 0x5605ca1119a0 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f98591d4740 10 client.18249 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f98591d4740 1 -- 172.21.15.89:0/2987889524 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5605ca1927e0 con 0x5605ca1119a0 2024-09-15T23:32:21.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f982f7fe640 1 -- 172.21.15.89:0/2987889524 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9848023070 con 0x5605ca1119a0 2024-09-15T23:32:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f982f7fe640 1 -- 172.21.15.89:0/2987889524 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9840086070 con 0x5605ca1119a0 2024-09-15T23:32:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f98591d4740 10 client.18249 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f98591d4740 10 client.18249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f98591d4740 20 client.18249 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f98591d4740 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5605ca192d50 0x5605ca195140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f98591d4740 4 client.18249 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:32:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.017+0000 7f98591d4740 1 -- 172.21.15.89:0/2987889524 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5605ca07cd70 con 0x5605ca192d50 2024-09-15T23:32:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.018+0000 7f984f7fe640 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5605ca192d50 0x5605ca195140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.018+0000 7f982f7fe640 10 client.18249 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:21.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.018+0000 7f984f7fe640 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5605ca192d50 0x5605ca195140 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d15bcdee7ef0489 server_cookie=3b7f612de18d61a0 in_seq=0 out_seq=0 2024-09-15T23:32:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.029+0000 7f982f7fe640 1 -- 172.21.15.89:0/2987889524 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5605ca07cd70 con 0x5605ca192d50 2024-09-15T23:32:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.029+0000 7f982f7fe640 10 client.18249 handle_command_reply: tid=0 2024-09-15T23:32:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 client.18249 shutdown 2024-09-15T23:32:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 -- 172.21.15.89:0/2987889524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1119a0 msgr2=0x5605ca120900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1119a0 0x5605ca120900 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f984000d740 tx=0x7f9840004590 comp rx=0 tx=0).stop 2024-09-15T23:32:21.031 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 -- 172.21.15.89:0/2987889524 shutdown_connections 2024-09-15T23:32:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5605ca192d50 0x5605ca195140 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5605ca117370 0x5605ca121e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5605ca1122e0 0x5605ca120e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f982f7fe640 0 client.18249 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 --2- 172.21.15.89:0/2987889524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5605ca1119a0 0x5605ca120900 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 -- 172.21.15.89:0/2987889524 >> 172.21.15.89:0/2987889524 conn(0x5605ca10d010 msgr2=0x5605ca10b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 -- 172.21.15.89:0/2987889524 shutdown_connections 2024-09-15T23:32:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 1 -- 172.21.15.89:0/2987889524 wait complete. 2024-09-15T23:32:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.030+0000 7f98591d4740 20 client.18249 trim_cache size 0 max 16384 2024-09-15T23:32:21.032 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:32:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:32:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:32:21.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 -- 172.21.15.89:0/3708069292 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f983c0a32f0 msgr2=0x7f983c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f983c0a32f0 0x7f983c0a57b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9848005fd0 tx=0x7f9848004a40 comp rx=0 tx=0).stop 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 -- 172.21.15.89:0/3708069292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9850101b60 msgr2=0x7f9850197dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9850101b60 0x7f9850197dc0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f984000d770 tx=0x7f984000dc30 comp rx=0 tx=0).stop 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 -- 172.21.15.89:0/3708069292 shutdown_connections 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f983c0a32f0 0x7f983c0a57b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9850103710 0x7f985019f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9850102d50 0x7f9850198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 --2- 172.21.15.89:0/3708069292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9850101b60 0x7f9850197dc0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 -- 172.21.15.89:0/3708069292 >> 172.21.15.89:0/3708069292 conn(0x7f98500fd2e0 msgr2=0x7f9850106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 -- 172.21.15.89:0/3708069292 shutdown_connections 2024-09-15T23:32:21.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:21.031+0000 7f985643c640 1 -- 172.21.15.89:0/3708069292 wait complete. 2024-09-15T23:32:21.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.118+0000 7f82f3b2e640 1 -- 172.21.15.89:0/1637667486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ec101240 msgr2=0x7f82ec101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.118+0000 7f82f3b2e640 1 --2- 172.21.15.89:0/1637667486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ec101240 0x7f82ec101680 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009900 tx=0x7f82dc02a410 comp rx=0 tx=0).stop 2024-09-15T23:32:21.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.118+0000 7f82f3b2e640 1 -- 172.21.15.89:0/1637667486 shutdown_connections 2024-09-15T23:32:21.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.118+0000 7f82f3b2e640 1 --2- 172.21.15.89:0/1637667486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82ec101bc0 0x7f82ec10d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.118+0000 7f82f3b2e640 1 --2- 172.21.15.89:0/1637667486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ec101240 0x7f82ec101680 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.118+0000 7f82f3b2e640 1 --2- 172.21.15.89:0/1637667486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82ec105a90 0x7f82ec105e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.118+0000 7f82f3b2e640 1 -- 172.21.15.89:0/1637667486 >> 172.21.15.89:0/1637667486 conn(0x7f82ec077210 msgr2=0x7f82ec0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.118+0000 7f82f3b2e640 1 -- 172.21.15.89:0/1637667486 shutdown_connections 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.118+0000 7f82f3b2e640 1 -- 172.21.15.89:0/1637667486 wait complete. 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f3b2e640 1 Processor -- start 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f3b2e640 1 -- start start 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f3b2e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ec101240 0x7f82ec198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f3b2e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82ec101bc0 0x7f82ec198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f3b2e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82ec105a90 0x7f82ec19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f3b2e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec112090 con 0x7f82ec101240 2024-09-15T23:32:21.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f3b2e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec111f10 con 0x7f82ec101bc0 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f3b2e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec112210 con 0x7f82ec105a90 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f18ca640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ec101240 0x7f82ec198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f18ca640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ec101240 0x7f82ec198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33592/0 (socket says 172.21.15.89:33592) 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f10c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82ec101bc0 0x7f82ec198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f18ca640 1 -- 172.21.15.89:0/2449299814 learned_addr learned my addr 172.21.15.89:0/2449299814 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f20cb640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82ec105a90 0x7f82ec19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f18ca640 1 -- 172.21.15.89:0/2449299814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82ec105a90 msgr2=0x7f82ec19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f18ca640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82ec105a90 0x7f82ec19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f18ca640 1 -- 172.21.15.89:0/2449299814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82ec101bc0 msgr2=0x7f82ec198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f18ca640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82ec101bc0 0x7f82ec198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.119+0000 7f82f18ca640 1 -- 172.21.15.89:0/2449299814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ec19d400 con 0x7f82ec101240 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.120+0000 7f82f18ca640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ec101240 0x7f82ec198450 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f82e800d770 tx=0x7f82e800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.120+0000 7f82e2ffd640 1 -- 172.21.15.89:0/2449299814 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8017070 con 0x7f82ec101240 2024-09-15T23:32:21.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.120+0000 7f82f3b2e640 1 -- 172.21.15.89:0/2449299814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82ec19d6f0 con 0x7f82ec101240 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.120+0000 7f82e2ffd640 1 -- 172.21.15.89:0/2449299814 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e8005500 con 0x7f82ec101240 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.120+0000 7f82e2ffd640 1 -- 172.21.15.89:0/2449299814 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8004e00 con 0x7f82ec101240 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.120+0000 7f82f3b2e640 1 -- 172.21.15.89:0/2449299814 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82ec1a4fd0 con 0x7f82ec101240 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.122+0000 7f82e2ffd640 1 -- 172.21.15.89:0/2449299814 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f82e8004fa0 con 0x7f82ec101240 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.122+0000 7f82f5f07740 1 Processor -- start 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.122+0000 7f82f5f07740 1 -- start start 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eec7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eec7860 0x556d4eed6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eed6650 0x556d4eed8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f5f07740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556d4ee2c620 con 0x556d4eec6f40 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f5f07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556d4edd3e50 con 0x556d4eed6650 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f5f07740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556d4ee285e0 con 0x556d4eec7860 2024-09-15T23:32:21.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82e2ffd640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82d80a3240 0x7f82d80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f20cb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eec7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f20cb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eec7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33604/0 (socket says 172.21.15.89:33604) 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f20cb640 1 -- 172.21.15.89:0/337492135 learned_addr learned my addr 172.21.15.89:0/337492135 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f18ca640 1 --2- 172.21.15.89:0/337492135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eec7860 0x556d4eed6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f20cb640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82d80a3240 0x7f82d80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82e2ffd640 1 -- 172.21.15.89:0/2449299814 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f82e80cf780 con 0x7f82ec101240 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f20cb640 1 -- 172.21.15.89:0/337492135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eec7860 msgr2=0x556d4eed6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f20cb640 1 --2- 172.21.15.89:0/337492135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eec7860 0x556d4eed6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f20cb640 1 -- 172.21.15.89:0/337492135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eed6650 msgr2=0x556d4eed8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f20cb640 1 --2- 172.21.15.89:0/337492135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eed6650 0x556d4eed8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f20cb640 1 -- 172.21.15.89:0/337492135 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d4edc4d40 con 0x556d4eec6f40 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.123+0000 7f82f20cb640 1 --2- 172.21.15.89:0/337492135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eec7320 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f82e4012910 tx=0x7f82e4012dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9fa2133618a967 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82d7fff640 1 -- 172.21.15.89:0/337492135 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4005020 con 0x556d4eec6f40 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82d7fff640 1 -- 172.21.15.89:0/337492135 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e40056a0 con 0x556d4eec6f40 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82d7fff640 1 -- 172.21.15.89:0/337492135 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e401c790 con 0x556d4eec6f40 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82f20cb640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82d80a3240 0x7f82d80a5700 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f82e401f000 tx=0x7f82e401e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82f5f07740 1 -- 172.21.15.89:0/337492135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 msgr2=0x556d4eec7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82f5f07740 1 --2- 172.21.15.89:0/337492135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eec7320 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f82e4012910 tx=0x7f82e4012dd0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82f5f07740 1 -- 172.21.15.89:0/337492135 shutdown_connections 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82f5f07740 1 --2- 172.21.15.89:0/337492135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eed6650 0x556d4eed8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82f5f07740 1 --2- 172.21.15.89:0/337492135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eec7860 0x556d4eed6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82f5f07740 1 --2- 172.21.15.89:0/337492135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eec7320 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82f5f07740 1 -- 172.21.15.89:0/337492135 >> 172.21.15.89:0/337492135 conn(0x556d4eec35f0 msgr2=0x556d4eec39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82f5f07740 1 -- 172.21.15.89:0/337492135 shutdown_connections 2024-09-15T23:32:21.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.124+0000 7f82f5f07740 1 -- 172.21.15.89:0/337492135 wait complete. 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f5f07740 1 -- start start 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f5f07740 1 Processor -- start 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eed3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eec7860 0x556d4eed3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 0x556d4eed4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f5f07740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556d4eed9220 con 0x556d4eec6f40 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f5f07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556d4edb4f10 con 0x556d4eec7860 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f5f07740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556d4eed93a0 con 0x556d4eed6650 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f5f07740 10 client.0 _ll_register_callbacks cb 0x556d4edd8db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f18ca640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eed3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f18ca640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eed3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33616/0 (socket says 172.21.15.89:33616) 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f18ca640 1 -- 172.21.15.89:0/4036233385 learned_addr learned my addr 172.21.15.89:0/4036233385 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:21.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.125+0000 7f82f10c9640 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eec7860 0x556d4eed3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f20cb640 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 0x556d4eed4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f18ca640 1 -- 172.21.15.89:0/4036233385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 msgr2=0x556d4eed4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f18ca640 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 0x556d4eed4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f18ca640 1 -- 172.21.15.89:0/4036233385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eec7860 msgr2=0x556d4eed3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f18ca640 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eec7860 0x556d4eed3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f18ca640 1 -- 172.21.15.89:0/4036233385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d4eb3b580 con 0x556d4eec6f40 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f18ca640 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eed3580 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f82e8097e10 tx=0x7f82e8018cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8b3c785434fb91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82d57fa640 1 -- 172.21.15.89:0/4036233385 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8020040 con 0x556d4eec6f40 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82d57fa640 1 -- 172.21.15.89:0/4036233385 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f82e80860b0 con 0x556d4eec6f40 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f5f07740 1 -- 172.21.15.89:0/4036233385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82e40056a0 con 0x556d4eec6f40 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82d57fa640 1 -- 172.21.15.89:0/4036233385 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82e809c0b0 con 0x556d4eec6f40 2024-09-15T23:32:21.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f5f07740 10 client.18261 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f5f07740 1 -- 172.21.15.89:0/4036233385 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556d4ec05960 con 0x556d4eec6f40 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82d57fa640 1 -- 172.21.15.89:0/4036233385 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e40056a0 con 0x556d4eec6f40 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82d57fa640 1 -- 172.21.15.89:0/4036233385 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f82e809cce0 con 0x556d4eec6f40 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f5f07740 10 client.18261 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f5f07740 10 client.18261 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f5f07740 20 client.18261 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f5f07740 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d4eee61d0 0x556d4eee85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f5f07740 4 client.18261 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.126+0000 7f82f5f07740 1 -- 172.21.15.89:0/4036233385 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d4ee39b90 con 0x556d4eee61d0 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.127+0000 7f82f10c9640 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d4eee61d0 0x556d4eee85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.127+0000 7f82f10c9640 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d4eee61d0 0x556d4eee85e0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6c8522217f290a1 server_cookie=49d2de2d3ea69254 in_seq=0 out_seq=0 2024-09-15T23:32:21.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.127+0000 7f82d57fa640 10 client.18261 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82d57fa640 1 -- 172.21.15.89:0/4036233385 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556d4ee39b90 con 0x556d4eee61d0 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82d57fa640 10 client.18261 handle_command_reply: tid=0 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 client.18261 shutdown 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 -- 172.21.15.89:0/4036233385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 msgr2=0x556d4eed3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eed3580 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f82e8097e10 tx=0x7f82e8018cf0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 -- 172.21.15.89:0/4036233385 shutdown_connections 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d4eee61d0 0x556d4eee85e0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 0x556d4eed4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eec7860 0x556d4eed3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 --2- 172.21.15.89:0/4036233385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eed3580 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82d57fa640 0 client.18261 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 -- 172.21.15.89:0/4036233385 >> 172.21.15.89:0/4036233385 conn(0x556d4eec35f0 msgr2=0x556d4eec9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 -- 172.21.15.89:0/4036233385 shutdown_connections 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 1 -- 172.21.15.89:0/4036233385 wait complete. 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.128+0000 7f82f5f07740 20 client.18261 trim_cache size 0 max 16384 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f5f07740 1 Processor -- start 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f5f07740 1 -- start start 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eec6f40 0x556d4eeedeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec7860 0x556d4eeee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 0x556d4eede620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f5f07740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1acaa0 con 0x556d4eec7860 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f5f07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1ae5e0 con 0x556d4eec6f40 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f5f07740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1ad170 con 0x556d4eed6650 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f10c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec7860 0x556d4eeee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f10c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec7860 0x556d4eeee3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33624/0 (socket says 172.21.15.89:33624) 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f10c9640 1 -- 172.21.15.89:0/2132869511 learned_addr learned my addr 172.21.15.89:0/2132869511 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f18ca640 1 --2- 172.21.15.89:0/2132869511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eec6f40 0x556d4eeedeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f20cb640 1 --2- 172.21.15.89:0/2132869511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 0x556d4eede620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f10c9640 1 -- 172.21.15.89:0/2132869511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 msgr2=0x556d4eede620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f10c9640 1 --2- 172.21.15.89:0/2132869511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 0x556d4eede620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f10c9640 1 -- 172.21.15.89:0/2132869511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eec6f40 msgr2=0x556d4eeedeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f10c9640 1 --2- 172.21.15.89:0/2132869511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eec6f40 0x556d4eeedeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f10c9640 1 -- 172.21.15.89:0/2132869511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d4eed2530 con 0x556d4eec7860 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.145+0000 7f82f20cb640 1 --2- 172.21.15.89:0/2132869511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 0x556d4eede620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.146+0000 7f82f10c9640 1 --2- 172.21.15.89:0/2132869511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec7860 0x556d4eeee3f0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009230 tx=0x7f82dc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266b6f1414e78c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.146+0000 7f82d77fe640 1 -- 172.21.15.89:0/2132869511 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc048030 con 0x556d4eec7860 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.146+0000 7f82d77fe640 1 -- 172.21.15.89:0/2132869511 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82dc005500 con 0x556d4eec7860 2024-09-15T23:32:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.146+0000 7f82d77fe640 1 -- 172.21.15.89:0/2132869511 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc035040 con 0x556d4eec7860 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 -- 172.21.15.89:0/2132869511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec7860 msgr2=0x556d4eeee3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 --2- 172.21.15.89:0/2132869511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec7860 0x556d4eeee3f0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009230 tx=0x7f82dc000bc0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 -- 172.21.15.89:0/2132869511 shutdown_connections 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 --2- 172.21.15.89:0/2132869511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eed6650 0x556d4eede620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 --2- 172.21.15.89:0/2132869511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec7860 0x556d4eeee3f0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 --2- 172.21.15.89:0/2132869511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eec6f40 0x556d4eeedeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 -- 172.21.15.89:0/2132869511 >> 172.21.15.89:0/2132869511 conn(0x556d4eec35f0 msgr2=0x556d4eed4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 -- 172.21.15.89:0/2132869511 shutdown_connections 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 -- 172.21.15.89:0/2132869511 wait complete. 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 -- start start 2024-09-15T23:32:21.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.147+0000 7f82f5f07740 1 Processor -- start 2024-09-15T23:32:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eedece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eec7860 0x556d4eedb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f5f07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eed6650 0x556d4eedb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f5f07740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1adf10 con 0x556d4eec6f40 2024-09-15T23:32:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f5f07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1ad840 con 0x556d4eed6650 2024-09-15T23:32:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f5f07740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82ec1aaa80 con 0x556d4eec7860 2024-09-15T23:32:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f5f07740 10 client.0 _ll_register_callbacks cb 0x7f82e809b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:21.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f18ca640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eedece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f18ca640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eedece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33634/0 (socket says 172.21.15.89:33634) 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f18ca640 1 -- 172.21.15.89:0/1128472671 learned_addr learned my addr 172.21.15.89:0/1128472671 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f10c9640 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eec7860 0x556d4eedb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f20cb640 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eed6650 0x556d4eedb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f18ca640 1 -- 172.21.15.89:0/1128472671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eec7860 msgr2=0x556d4eedb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f18ca640 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eec7860 0x556d4eedb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f18ca640 1 -- 172.21.15.89:0/1128472671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eed6650 msgr2=0x556d4eedb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f18ca640 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eed6650 0x556d4eedb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f18ca640 1 -- 172.21.15.89:0/1128472671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d4eedbd60 con 0x556d4eec6f40 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f18ca640 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eedece0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f82e800dfd0 tx=0x7f82e8005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c9789c72f85964 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:21.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82d57fa640 1 -- 172.21.15.89:0/1128472671 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8014020 con 0x556d4eec6f40 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82d57fa640 1 -- 172.21.15.89:0/1128472671 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f82e8097700 con 0x556d4eec6f40 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.148+0000 7f82f5f07740 1 -- 172.21.15.89:0/1128472671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82dc005500 con 0x556d4eec6f40 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82d57fa640 1 -- 172.21.15.89:0/1128472671 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82e8020040 con 0x556d4eec6f40 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82f5f07740 10 client.18267 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82f5f07740 1 -- 172.21.15.89:0/1128472671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556d4ef51db0 con 0x556d4eec6f40 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82d57fa640 1 -- 172.21.15.89:0/1128472671 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82dc005500 con 0x556d4eec6f40 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82d57fa640 1 -- 172.21.15.89:0/1128472671 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f82e809cce0 con 0x556d4eec6f40 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82f5f07740 10 client.18267 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82f5f07740 10 client.18267 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82f5f07740 20 client.18267 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82f5f07740 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d4ef524d0 0x556d4ef548f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82f5f07740 4 client.18267 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6514e407", "release": true, "await": true}] 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82f5f07740 1 -- 172.21.15.89:0/1128472671 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6514e407", "release": true, "await": true}) -- 0x556d4ee340a0 con 0x556d4ef524d0 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.149+0000 7f82f10c9640 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d4ef524d0 0x556d4ef548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.150+0000 7f82f10c9640 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d4ef524d0 0x556d4ef548f0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c382c99c710c5a6 server_cookie=a43c1f71bd7ae0f6 in_seq=0 out_seq=0 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.150+0000 7f82d57fa640 10 client.18267 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82d57fa640 1 -- 172.21.15.89:0/1128472671 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x556d4ee340a0 con 0x556d4ef524d0 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82d57fa640 10 client.18267 handle_command_reply: tid=0 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 client.18267 shutdown 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 -- 172.21.15.89:0/1128472671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 msgr2=0x556d4eedece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eedece0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f82e800dfd0 tx=0x7f82e8005bb0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 -- 172.21.15.89:0/1128472671 shutdown_connections 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d4ef524d0 0x556d4ef548f0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d4eed6650 0x556d4eedb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d4eec7860 0x556d4eedb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82d57fa640 0 client.18267 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 --2- 172.21.15.89:0/1128472671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d4eec6f40 0x556d4eedece0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 -- 172.21.15.89:0/1128472671 >> 172.21.15.89:0/1128472671 conn(0x556d4eec35f0 msgr2=0x556d4eec1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 -- 172.21.15.89:0/1128472671 shutdown_connections 2024-09-15T23:32:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 1 -- 172.21.15.89:0/1128472671 wait complete. 2024-09-15T23:32:21.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.153+0000 7f82f5f07740 20 client.18267 trim_cache size 0 max 16384 2024-09-15T23:32:21.167 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:32:21.167 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:32:21.167 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 28, 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "6514e407": { 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 28, 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 143.7, 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:32:21.168 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:32:21.169 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:32:21.169 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:32:21.169 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:32:21.169 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:32:21.169 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:32:21.169 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:32:21.169 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:32:21.169 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:32:21.169 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:32:21.169 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 -- 172.21.15.89:0/2449299814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82d80a3240 msgr2=0x7f82d80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82d80a3240 0x7f82d80a5700 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f82e401f000 tx=0x7f82e401e000 comp rx=0 tx=0).stop 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 -- 172.21.15.89:0/2449299814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ec101240 msgr2=0x7f82ec198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ec101240 0x7f82ec198450 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f82e800d770 tx=0x7f82e800dc30 comp rx=0 tx=0).stop 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 -- 172.21.15.89:0/2449299814 shutdown_connections 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f82d80a3240 0x7f82d80a5700 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82ec105a90 0x7f82ec19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82ec101bc0 0x7f82ec198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 --2- 172.21.15.89:0/2449299814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82ec101240 0x7f82ec198450 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 -- 172.21.15.89:0/2449299814 >> 172.21.15.89:0/2449299814 conn(0x7f82ec077210 msgr2=0x7f82ec103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:21.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 -- 172.21.15.89:0/2449299814 shutdown_connections 2024-09-15T23:32:21.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:21.154+0000 7f82f3b2e640 1 -- 172.21.15.89:0/2449299814 wait complete. 2024-09-15T23:32:21.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:21 smithi089 ceph-mon[30933]: pgmap v600: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-09-15T23:32:21.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:21 smithi120 ceph-mon[35250]: pgmap v600: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-09-15T23:32:21.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:21 smithi174 ceph-mon[35586]: pgmap v600: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-09-15T23:32:21.881 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6514e407', total seconds elapsed: 26.8 2024-09-15T23:32:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:23 smithi174 ceph-mon[35586]: pgmap v601: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-09-15T23:32:23.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:23 smithi089 ceph-mon[30933]: pgmap v601: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-09-15T23:32:23.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:23 smithi120 ceph-mon[35250]: pgmap v601: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-09-15T23:32:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:25 smithi174 ceph-mon[35586]: pgmap v602: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-09-15T23:32:25.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:25 smithi089 ceph-mon[30933]: pgmap v602: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-09-15T23:32:25.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:25 smithi120 ceph-mon[35250]: pgmap v602: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-09-15T23:32:27.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:27 smithi174 ceph-mon[35586]: pgmap v603: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-09-15T23:32:27.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:27 smithi089 ceph-mon[30933]: pgmap v603: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-09-15T23:32:27.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:27 smithi120 ceph-mon[35250]: pgmap v603: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-09-15T23:32:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:29 smithi174 ceph-mon[35586]: pgmap v604: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-09-15T23:32:29.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:29 smithi089 ceph-mon[30933]: pgmap v604: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-09-15T23:32:29.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:29 smithi120 ceph-mon[35250]: pgmap v604: 129 pgs: 129 active+clean; 4.1 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-09-15T23:32:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:31 smithi174 ceph-mon[35586]: pgmap v605: 129 pgs: 129 active+clean; 4.2 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:32:31.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:31 smithi089 ceph-mon[30933]: pgmap v605: 129 pgs: 129 active+clean; 4.2 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:32:31.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:31 smithi120 ceph-mon[35250]: pgmap v605: 129 pgs: 129 active+clean; 4.2 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:32:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:33 smithi174 ceph-mon[35586]: pgmap v606: 129 pgs: 129 active+clean; 4.2 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:32:33.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:33 smithi089 ceph-mon[30933]: pgmap v606: 129 pgs: 129 active+clean; 4.2 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:32:33.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:33 smithi120 ceph-mon[35250]: pgmap v606: 129 pgs: 129 active+clean; 4.2 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.128+0000 7f7ab940a640 1 -- 172.21.15.89:0/3582758316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4104f70 msgr2=0x7f7ab4107360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.128+0000 7f7ab940a640 1 --2- 172.21.15.89:0/3582758316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4104f70 0x7f7ab4107360 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009900 tx=0x7f7aa802a410 comp rx=0 tx=0).stop 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.128+0000 7f7ab940a640 1 -- 172.21.15.89:0/3582758316 shutdown_connections 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.128+0000 7f7ab940a640 1 --2- 172.21.15.89:0/3582758316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ab41078a0 0x7f7ab4109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.128+0000 7f7ab940a640 1 --2- 172.21.15.89:0/3582758316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4104f70 0x7f7ab4107360 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.128+0000 7f7ab940a640 1 --2- 172.21.15.89:0/3582758316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ab4102640 0x7f7ab4104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.128+0000 7f7ab940a640 1 -- 172.21.15.89:0/3582758316 >> 172.21.15.89:0/3582758316 conn(0x7f7ab40fc1e0 msgr2=0x7f7ab40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.129+0000 7f7ab940a640 1 -- 172.21.15.89:0/3582758316 shutdown_connections 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.129+0000 7f7ab940a640 1 -- 172.21.15.89:0/3582758316 wait complete. 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.129+0000 7f7ab940a640 1 Processor -- start 2024-09-15T23:32:34.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.129+0000 7f7ab940a640 1 -- start start 2024-09-15T23:32:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab940a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ab4102640 0x7f7ab4197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab940a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4104f70 0x7f7ab41983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab940a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ab41078a0 0x7f7ab419f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab940a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab410d820 con 0x7f7ab4104f70 2024-09-15T23:32:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab940a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab410d6a0 con 0x7f7ab41078a0 2024-09-15T23:32:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab940a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab410d9a0 con 0x7f7ab4102640 2024-09-15T23:32:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4104f70 0x7f7ab41983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4104f70 0x7f7ab41983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36928/0 (socket says 172.21.15.89:36928) 2024-09-15T23:32:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab27fc640 1 -- 172.21.15.89:0/2998866703 learned_addr learned my addr 172.21.15.89:0/2998866703 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:34.144 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ab4102640 0x7f7ab4197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab37fe640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ab41078a0 0x7f7ab419f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab27fc640 1 -- 172.21.15.89:0/2998866703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ab4102640 msgr2=0x7f7ab4197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ab4102640 0x7f7ab4197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab27fc640 1 -- 172.21.15.89:0/2998866703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ab41078a0 msgr2=0x7f7ab419f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ab41078a0 0x7f7ab419f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab27fc640 1 -- 172.21.15.89:0/2998866703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab419fac0 con 0x7f7ab4104f70 2024-09-15T23:32:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab37fe640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ab41078a0 0x7f7ab419f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4104f70 0x7f7ab41983e0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8031610 tx=0x7f7aa80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7a9bfff640 1 -- 172.21.15.89:0/2998866703 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa803f030 con 0x7f7ab4104f70 2024-09-15T23:32:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7a9bfff640 1 -- 172.21.15.89:0/2998866703 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa8003ce0 con 0x7f7ab4104f70 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7ab940a640 1 -- 172.21.15.89:0/2998866703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab419fd50 con 0x7f7ab4104f70 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.130+0000 7f7a9bfff640 1 -- 172.21.15.89:0/2998866703 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8035070 con 0x7f7ab4104f70 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.131+0000 7f7ab940a640 1 -- 172.21.15.89:0/2998866703 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab41a01e0 con 0x7f7ab4104f70 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.133+0000 7f7a9bfff640 1 -- 172.21.15.89:0/2998866703 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7aa8042030 con 0x7f7ab4104f70 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.133+0000 7f7abc277740 1 Processor -- start 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.133+0000 7f7abc277740 1 -- start start 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7abc277740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bcdb0 0x559acf0bd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7abc277740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bd6f0 0x559acf0c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7abc277740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 0x559acf0c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7abc277740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559acf01c2f0 con 0x559acf0bd6f0 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7abc277740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559acefc4270 con 0x559acf0c2780 2024-09-15T23:32:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7abc277740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559acf0182b0 con 0x559acf0bcdb0 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7a9bfff640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7a9c0a32f0 0x7f7a9c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bd6f0 0x559acf0c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bd6f0 0x559acf0c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36932/0 (socket says 172.21.15.89:36932) 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/1122593071 learned_addr learned my addr 172.21.15.89:0/1122593071 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab37fe640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7a9c0a32f0 0x7f7a9c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/1122593071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 0x559acf0c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab37fe640 1 --2- 172.21.15.89:0/1122593071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bcdb0 0x559acf0bd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7a9bfff640 1 -- 172.21.15.89:0/2998866703 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7aa80f02d0 con 0x7f7ab4104f70 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/1122593071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bcdb0 msgr2=0x559acf0bd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/1122593071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bcdb0 0x559acf0bd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/1122593071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 msgr2=0x559acf0c4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/1122593071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 0x559acf0c4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/1122593071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559acefc3340 con 0x559acf0bd6f0 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/1122593071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 0x559acf0c4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab37fe640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7a9c0a32f0 0x7f7a9c0a57b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7aac005c40 tx=0x7f7aac0099f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/1122593071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bd6f0 0x559acf0c21b0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f7aa400b760 tx=0x7f7aa400bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6ca3f0994f2a48 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7a997fa640 1 -- 172.21.15.89:0/1122593071 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4003dd0 con 0x559acf0bd6f0 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7a997fa640 1 -- 172.21.15.89:0/1122593071 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa4015070 con 0x559acf0bd6f0 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.134+0000 7f7a997fa640 1 -- 172.21.15.89:0/1122593071 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4004e00 con 0x559acf0bd6f0 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.135+0000 7f7abc277740 1 -- 172.21.15.89:0/1122593071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bd6f0 msgr2=0x559acf0c21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.135+0000 7f7abc277740 1 --2- 172.21.15.89:0/1122593071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bd6f0 0x559acf0c21b0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f7aa400b760 tx=0x7f7aa400bc20 comp rx=0 tx=0).stop 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.135+0000 7f7abc277740 1 -- 172.21.15.89:0/1122593071 shutdown_connections 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.135+0000 7f7abc277740 1 --2- 172.21.15.89:0/1122593071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 0x559acf0c4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.135+0000 7f7abc277740 1 --2- 172.21.15.89:0/1122593071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bd6f0 0x559acf0c21b0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.135+0000 7f7abc277740 1 --2- 172.21.15.89:0/1122593071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bcdb0 0x559acf0bd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.135+0000 7f7abc277740 1 -- 172.21.15.89:0/1122593071 >> 172.21.15.89:0/1122593071 conn(0x559acf0b4010 msgr2=0x559acf0b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.135+0000 7f7abc277740 1 -- 172.21.15.89:0/1122593071 shutdown_connections 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.135+0000 7f7abc277740 1 -- 172.21.15.89:0/1122593071 wait complete. 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7abc277740 1 -- start start 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7abc277740 1 Processor -- start 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7abc277740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7abc277740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bd6f0 0x559acf0c7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7abc277740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 0x559acf0ce560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7abc277740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559acf0c5750 con 0x559acf0bcdb0 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7abc277740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559acefa5290 con 0x559acf0c2780 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7abc277740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559acf0c58d0 con 0x559acf0bd6f0 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7abc277740 10 client.0 _ll_register_callbacks cb 0x559acefd6140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36940/0 (socket says 172.21.15.89:36940) 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/1707696765 learned_addr learned my addr 172.21.15.89:0/1707696765 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab37fe640 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 0x559acf0ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/1707696765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bd6f0 msgr2=0x559acf0c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bd6f0 0x559acf0c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bd6f0 0x559acf0c7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/1707696765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 msgr2=0x559acf0ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 0x559acf0ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/1707696765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aced2f210 con 0x559acf0bcdb0 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.136+0000 7f7ab37fe640 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 0x559acf0ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bd6f0 0x559acf0c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c6fe0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f7aa40039c0 tx=0x7f7aa4003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bf339b6eae32c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7a96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7a96ff5640 1 -- 172.21.15.89:0/1707696765 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4028060 con 0x559acf0bcdb0 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7abc277740 1 -- 172.21.15.89:0/1707696765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7aa4015070 con 0x559acf0bcdb0 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7a96ff5640 1 -- 172.21.15.89:0/1707696765 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7aa4011850 con 0x559acf0bcdb0 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7a96ff5640 1 -- 172.21.15.89:0/1707696765 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7aa402f050 con 0x559acf0bcdb0 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7abc277740 10 client.18279 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7abc277740 1 -- 172.21.15.89:0/1707696765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559acedfb620 con 0x559acf0bcdb0 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7a96ff5640 1 -- 172.21.15.89:0/1707696765 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa4015070 con 0x559acf0bcdb0 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7a96ff5640 1 -- 172.21.15.89:0/1707696765 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7aa4022030 con 0x559acf0bcdb0 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7abc277740 10 client.18279 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7abc277740 10 client.18279 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7abc277740 20 client.18279 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7abc277740 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559acf0d2970 0x559acf0d4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.137+0000 7f7abc277740 4 client.18279 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.138+0000 7f7abc277740 1 -- 172.21.15.89:0/1707696765 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559acf029a20 con 0x559acf0d2970 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.138+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559acf0d2970 0x559acf0d4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.138+0000 7f7a96ff5640 10 client.18279 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.138+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559acf0d2970 0x559acf0d4d60 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b67f9af28d2fa9fc server_cookie=f4023353a8ea5d10 in_seq=0 out_seq=0 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7a96ff5640 1 -- 172.21.15.89:0/1707696765 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559acf029a20 con 0x559acf0d2970 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7a96ff5640 10 client.18279 handle_command_reply: tid=0 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 client.18279 shutdown 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 -- 172.21.15.89:0/1707696765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 msgr2=0x559acf0c6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c6fe0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f7aa40039c0 tx=0x7f7aa4003e00 comp rx=0 tx=0).stop 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 -- 172.21.15.89:0/1707696765 shutdown_connections 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559acf0d2970 0x559acf0d4d60 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0c2780 0x559acf0ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bd6f0 0x559acf0c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 --2- 172.21.15.89:0/1707696765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c6fe0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7a96ff5640 0 client.18279 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:34.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 -- 172.21.15.89:0/1707696765 >> 172.21.15.89:0/1707696765 conn(0x559acf0b4010 msgr2=0x559acf0b5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:34.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 -- 172.21.15.89:0/1707696765 shutdown_connections 2024-09-15T23:32:34.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.139+0000 7f7abc277740 1 -- 172.21.15.89:0/1707696765 wait complete. 2024-09-15T23:32:34.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.140+0000 7f7abc277740 20 client.18279 trim_cache size 0 max 16384 2024-09-15T23:32:34.167 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7abc277740 1 Processor -- start 2024-09-15T23:32:34.167 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7abc277740 1 -- start start 2024-09-15T23:32:34.167 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7abc277740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.167 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7abc277740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 0x559acf0ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.167 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7abc277740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0c2780 0x559acf0b6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7abc277740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41a8730 con 0x559acf0bcdb0 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7abc277740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41aa270 con 0x559acf0bd6f0 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7abc277740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41a8e00 con 0x559acf0c2780 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36954/0 (socket says 172.21.15.89:36954) 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/2137092209 learned_addr learned my addr 172.21.15.89:0/2137092209 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/2137092209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0c2780 msgr2=0x559acf0b6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/2137092209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0c2780 0x559acf0b6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/2137092209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 msgr2=0x559acf0ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/2137092209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 0x559acf0ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/2137092209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 0x559acf0ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab2ffd640 1 -- 172.21.15.89:0/2137092209 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559acf0c2560 con 0x559acf0bcdb0 2024-09-15T23:32:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/2137092209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 0x559acf0ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/2137092209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c74a0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f7aa400b6b0 tx=0x7f7aa400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b59be64f45ee08f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7a98ff9640 1 -- 172.21.15.89:0/2137092209 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4028070 con 0x559acf0bcdb0 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7a98ff9640 1 -- 172.21.15.89:0/2137092209 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa4015030 con 0x559acf0bcdb0 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.154+0000 7f7a98ff9640 1 -- 172.21.15.89:0/2137092209 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40119d0 con 0x559acf0bcdb0 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 -- 172.21.15.89:0/2137092209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 msgr2=0x559acf0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 --2- 172.21.15.89:0/2137092209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c74a0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f7aa400b6b0 tx=0x7f7aa400bf90 comp rx=0 tx=0).stop 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 -- 172.21.15.89:0/2137092209 shutdown_connections 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 --2- 172.21.15.89:0/2137092209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0c2780 0x559acf0b6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 --2- 172.21.15.89:0/2137092209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 0x559acf0ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 --2- 172.21.15.89:0/2137092209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0bcdb0 0x559acf0c74a0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 -- 172.21.15.89:0/2137092209 >> 172.21.15.89:0/2137092209 conn(0x559acf0b4010 msgr2=0x559acf0c6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 -- 172.21.15.89:0/2137092209 shutdown_connections 2024-09-15T23:32:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 -- 172.21.15.89:0/2137092209 wait complete. 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 -- start start 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.155+0000 7f7abc277740 1 Processor -- start 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7abc277740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bcdb0 0x559acf0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7abc277740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 0x559acf0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7abc277740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0c2780 0x559acf0cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7abc277740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41a9ba0 con 0x559acf0c2780 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7abc277740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41a94d0 con 0x559acf0bd6f0 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7abc277740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41a6710 con 0x559acf0bcdb0 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7abc277740 10 client.0 _ll_register_callbacks cb 0x7f7aa4004590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0c2780 0x559acf0cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0c2780 0x559acf0cd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36960/0 (socket says 172.21.15.89:36960) 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab37fe640 1 -- 172.21.15.89:0/4026798829 learned_addr learned my addr 172.21.15.89:0/4026798829 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bcdb0 0x559acf0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab37fe640 1 -- 172.21.15.89:0/4026798829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bcdb0 msgr2=0x559acf0cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab37fe640 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bcdb0 0x559acf0cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab37fe640 1 -- 172.21.15.89:0/4026798829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 msgr2=0x559acf0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab27fc640 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 0x559acf0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab37fe640 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 0x559acf0cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab37fe640 1 -- 172.21.15.89:0/4026798829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559acf0cd950 con 0x559acf0c2780 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7ab37fe640 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0c2780 0x559acf0cd340 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009e90 tx=0x7f7aac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509c45ecc4f5d6cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7a96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7a96ff5640 1 -- 172.21.15.89:0/4026798829 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac024030 con 0x559acf0c2780 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7a96ff5640 1 -- 172.21.15.89:0/4026798829 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7aac004b20 con 0x559acf0c2780 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.156+0000 7f7abc277740 1 -- 172.21.15.89:0/4026798829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7aa4015030 con 0x559acf0c2780 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7a96ff5640 1 -- 172.21.15.89:0/4026798829 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7aac030050 con 0x559acf0c2780 2024-09-15T23:32:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7abc277740 10 client.18285 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7abc277740 1 -- 172.21.15.89:0/4026798829 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559acf13d6b0 con 0x559acf0c2780 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7a96ff5640 1 -- 172.21.15.89:0/4026798829 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa4015030 con 0x559acf0c2780 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7a96ff5640 1 -- 172.21.15.89:0/4026798829 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7aac00f050 con 0x559acf0c2780 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7abc277740 10 client.18285 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7abc277740 10 client.18285 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7abc277740 20 client.18285 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7abc277740 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559acf13e0c0 0x559acf1404e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7abc277740 4 client.18285 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7abc277740 1 -- 172.21.15.89:0/4026798829 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559acf023d70 con 0x559acf13e0c0 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.157+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559acf13e0c0 0x559acf1404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.158+0000 7f7a96ff5640 10 client.18285 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.158+0000 7f7ab2ffd640 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559acf13e0c0 0x559acf1404e0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53a7e054165330b0 server_cookie=5d5c1d981fc6fdf2 in_seq=0 out_seq=0 2024-09-15T23:32:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.172+0000 7f7a96ff5640 1 -- 172.21.15.89:0/4026798829 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559acf023d70 con 0x559acf13e0c0 2024-09-15T23:32:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.172+0000 7f7a96ff5640 10 client.18285 handle_command_reply: tid=0 2024-09-15T23:32:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.172+0000 7f7abc277740 1 client.18285 shutdown 2024-09-15T23:32:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.172+0000 7f7abc277740 1 -- 172.21.15.89:0/4026798829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0c2780 msgr2=0x559acf0cd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.172+0000 7f7abc277740 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0c2780 0x559acf0cd340 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009e90 tx=0x7f7aac004a40 comp rx=0 tx=0).stop 2024-09-15T23:32:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7abc277740 1 -- 172.21.15.89:0/4026798829 shutdown_connections 2024-09-15T23:32:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7abc277740 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559acf13e0c0 0x559acf1404e0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7abc277740 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559acf0c2780 0x559acf0cd340 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7a96ff5640 0 client.18285 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7abc277740 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559acf0bd6f0 0x559acf0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7abc277740 1 --2- 172.21.15.89:0/4026798829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559acf0bcdb0 0x559acf0cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7abc277740 1 -- 172.21.15.89:0/4026798829 >> 172.21.15.89:0/4026798829 conn(0x559acf0b4010 msgr2=0x559acf0b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7abc277740 1 -- 172.21.15.89:0/4026798829 shutdown_connections 2024-09-15T23:32:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7abc277740 1 -- 172.21.15.89:0/4026798829 wait complete. 2024-09-15T23:32:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7abc277740 20 client.18285 trim_cache size 0 max 16384 2024-09-15T23:32:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:32:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:32:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:32:34.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:32:34.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 -- 172.21.15.89:0/2998866703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7a9c0a32f0 msgr2=0x7f7a9c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7a9c0a32f0 0x7f7a9c0a57b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7aac005c40 tx=0x7f7aac0099f0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 -- 172.21.15.89:0/2998866703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4104f70 msgr2=0x7f7ab41983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4104f70 0x7f7ab41983e0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8031610 tx=0x7f7aa80043b0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 -- 172.21.15.89:0/2998866703 shutdown_connections 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f7a9c0a32f0 0x7f7a9c0a57b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ab41078a0 0x7f7ab419f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ab4104f70 0x7f7ab41983e0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 --2- 172.21.15.89:0/2998866703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ab4102640 0x7f7ab4197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 -- 172.21.15.89:0/2998866703 >> 172.21.15.89:0/2998866703 conn(0x7f7ab40fc1e0 msgr2=0x7f7ab4102fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 -- 172.21.15.89:0/2998866703 shutdown_connections 2024-09-15T23:32:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:34.173+0000 7f7ab940a640 1 -- 172.21.15.89:0/2998866703 wait complete. 2024-09-15T23:32:35.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:35 smithi174 ceph-mon[35586]: pgmap v607: 129 pgs: 129 active+clean; 4.2 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T23:32:35.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:35 smithi089 ceph-mon[30933]: pgmap v607: 129 pgs: 129 active+clean; 4.2 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T23:32:35.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:35 smithi120 ceph-mon[35250]: pgmap v607: 129 pgs: 129 active+clean; 4.2 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T23:32:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:37 smithi174 ceph-mon[35586]: pgmap v608: 129 pgs: 129 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 19 op/s 2024-09-15T23:32:37.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:37 smithi089 ceph-mon[30933]: pgmap v608: 129 pgs: 129 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 19 op/s 2024-09-15T23:32:37.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:37 smithi120 ceph-mon[35250]: pgmap v608: 129 pgs: 129 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 19 op/s 2024-09-15T23:32:39.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:39 smithi174 ceph-mon[35586]: pgmap v609: 129 pgs: 129 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:32:39.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:39 smithi089 ceph-mon[30933]: pgmap v609: 129 pgs: 129 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:32:39.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:39 smithi120 ceph-mon[35250]: pgmap v609: 129 pgs: 129 active+clean; 4.3 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:32:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:41 smithi174 ceph-mon[35586]: pgmap v610: 129 pgs: 129 active+clean; 4.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:32:41.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:41 smithi089 ceph-mon[30933]: pgmap v610: 129 pgs: 129 active+clean; 4.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:32:41.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:41 smithi120 ceph-mon[35250]: pgmap v610: 129 pgs: 129 active+clean; 4.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:32:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:43 smithi174 ceph-mon[35586]: pgmap v611: 129 pgs: 129 active+clean; 4.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:32:43.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:43 smithi089 ceph-mon[30933]: pgmap v611: 129 pgs: 129 active+clean; 4.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:32:43.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:43 smithi120 ceph-mon[35250]: pgmap v611: 129 pgs: 129 active+clean; 4.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:32:45.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:45 smithi174 ceph-mon[35586]: pgmap v612: 129 pgs: 129 active+clean; 4.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:32:45.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:45 smithi089 ceph-mon[30933]: pgmap v612: 129 pgs: 129 active+clean; 4.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:32:45.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:45 smithi120 ceph-mon[35250]: pgmap v612: 129 pgs: 129 active+clean; 4.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:32:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.271+0000 7f39fd5f6640 1 -- 172.21.15.89:0/1428026735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f8101c40 msgr2=0x7f39f81020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.271+0000 7f39fd5f6640 1 --2- 172.21.15.89:0/1428026735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f8101c40 0x7f39f81020c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009900 tx=0x7f39e002a410 comp rx=0 tx=0).stop 2024-09-15T23:32:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.271+0000 7f39fd5f6640 1 -- 172.21.15.89:0/1428026735 shutdown_connections 2024-09-15T23:32:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.271+0000 7f39fd5f6640 1 --2- 172.21.15.89:0/1428026735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39f8102600 0x7f39f8108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.271+0000 7f39fd5f6640 1 --2- 172.21.15.89:0/1428026735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f8101c40 0x7f39f81020c0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.271+0000 7f39fd5f6640 1 --2- 172.21.15.89:0/1428026735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39f8100a50 0x7f39f8100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.272+0000 7f39fd5f6640 1 -- 172.21.15.89:0/1428026735 >> 172.21.15.89:0/1428026735 conn(0x7f39f80fc1b0 msgr2=0x7f39f80fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:47.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.272+0000 7f39fd5f6640 1 -- 172.21.15.89:0/1428026735 shutdown_connections 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.272+0000 7f39fd5f6640 1 -- 172.21.15.89:0/1428026735 wait complete. 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.272+0000 7f39fd5f6640 1 Processor -- start 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.272+0000 7f39fd5f6640 1 -- start start 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39fd5f6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39f8100a50 0x7f39f8197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39fd5f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f8101c40 0x7f39f8198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39fd5f6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39f8102600 0x7f39f819f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39fd5f6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f39f810c6b0 con 0x7f39f8101c40 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39fd5f6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39f810c530 con 0x7f39f8100a50 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39fd5f6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f39f810c830 con 0x7f39f8102600 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39ee7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f8101c40 0x7f39f8198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39ee7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f8101c40 0x7f39f8198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46996/0 (socket says 172.21.15.89:46996) 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39ee7fc640 1 -- 172.21.15.89:0/4157486189 learned_addr learned my addr 172.21.15.89:0/4157486189 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:47.277 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39f6ffd640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39f8100a50 0x7f39f8197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39f77fe640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39f8102600 0x7f39f819f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39ee7fc640 1 -- 172.21.15.89:0/4157486189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39f8102600 msgr2=0x7f39f819f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39ee7fc640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39f8102600 0x7f39f819f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39ee7fc640 1 -- 172.21.15.89:0/4157486189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39f8100a50 msgr2=0x7f39f8197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39ee7fc640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39f8100a50 0x7f39f8197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39ee7fc640 1 -- 172.21.15.89:0/4157486189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f819f9d0 con 0x7f39f8101c40 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39ee7fc640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f8101c40 0x7f39f8198300 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f39e0006010 tx=0x7f39e0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.273+0000 7f39f4ff9640 1 -- 172.21.15.89:0/4157486189 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e003f030 con 0x7f39f8101c40 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.274+0000 7f39f4ff9640 1 -- 172.21.15.89:0/4157486189 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39e0003ce0 con 0x7f39f8101c40 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.274+0000 7f39fd5f6640 1 -- 172.21.15.89:0/4157486189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f819fc60 con 0x7f39f8101c40 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.274+0000 7f39f4ff9640 1 -- 172.21.15.89:0/4157486189 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0035070 con 0x7f39f8101c40 2024-09-15T23:32:47.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.274+0000 7f39fd5f6640 1 -- 172.21.15.89:0/4157486189 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f81a0150 con 0x7f39f8101c40 2024-09-15T23:32:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.276+0000 7f39f4ff9640 1 -- 172.21.15.89:0/4157486189 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f39e0042030 con 0x7f39f8101c40 2024-09-15T23:32:47.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.276+0000 7f3a0046d740 1 Processor -- start 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.276+0000 7f3a0046d740 1 -- start start 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.276+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197d5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 0x55cf197d8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f3a0046d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cf197342f0 con 0x55cf197d09a0 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f3a0046d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cf196dc270 con 0x55cf197d6370 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f3a0046d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cf197302b0 con 0x55cf197d12e0 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f4ff9640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f39e40a32f0 0x7f39e40a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f6ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197d5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197d0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47000/0 (socket says 172.21.15.89:47000) 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 -- 172.21.15.89:0/973661659 learned_addr learned my addr 172.21.15.89:0/973661659 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:47.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f39e40a32f0 0x7f39e40a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f4ff9640 1 -- 172.21.15.89:0/4157486189 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f39e00f0730 con 0x7f39f8101c40 2024-09-15T23:32:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39ee7fc640 1 --2- 172.21.15.89:0/973661659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 0x55cf197d8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 -- 172.21.15.89:0/973661659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 msgr2=0x55cf197d5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 --2- 172.21.15.89:0/973661659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197d5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 -- 172.21.15.89:0/973661659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 msgr2=0x55cf197d8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 --2- 172.21.15.89:0/973661659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 0x55cf197d8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 -- 172.21.15.89:0/973661659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf196db340 con 0x55cf197d09a0 2024-09-15T23:32:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f6ffd640 1 --2- 172.21.15.89:0/973661659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197d5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:32:47.287 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 --2- 172.21.15.89:0/973661659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197d0da0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f39f0014e70 tx=0x7f39f0012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8450a71c84d63701 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39f77fe640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f39e40a32f0 0x7f39e40a57b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f39f0005c40 tx=0x7f39f001e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.277+0000 7f39ed7fa640 1 -- 172.21.15.89:0/973661659 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f0023070 con 0x55cf197d09a0 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f39ed7fa640 1 -- 172.21.15.89:0/973661659 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39f001dc80 con 0x55cf197d09a0 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f39ed7fa640 1 -- 172.21.15.89:0/973661659 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f0010640 con 0x55cf197d09a0 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f3a0046d740 1 -- 172.21.15.89:0/973661659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 msgr2=0x55cf197d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f3a0046d740 1 --2- 172.21.15.89:0/973661659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197d0da0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f39f0014e70 tx=0x7f39f0012760 comp rx=0 tx=0).stop 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f3a0046d740 1 -- 172.21.15.89:0/973661659 shutdown_connections 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f3a0046d740 1 --2- 172.21.15.89:0/973661659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 0x55cf197d8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f3a0046d740 1 --2- 172.21.15.89:0/973661659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197d5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f3a0046d740 1 --2- 172.21.15.89:0/973661659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197d0da0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f3a0046d740 1 -- 172.21.15.89:0/973661659 >> 172.21.15.89:0/973661659 conn(0x55cf197cc010 msgr2=0x55cf197cc410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f3a0046d740 1 -- 172.21.15.89:0/973661659 shutdown_connections 2024-09-15T23:32:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.278+0000 7f3a0046d740 1 -- 172.21.15.89:0/973661659 wait complete. 2024-09-15T23:32:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f3a0046d740 1 -- start start 2024-09-15T23:32:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f3a0046d740 1 Processor -- start 2024-09-15T23:32:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d09a0 0x55cf197dac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197db160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197e2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f3a0046d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cf197d92e0 con 0x55cf197d6370 2024-09-15T23:32:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f3a0046d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cf196bd290 con 0x55cf197d09a0 2024-09-15T23:32:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f3a0046d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cf197d9460 con 0x55cf197d12e0 2024-09-15T23:32:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f3a0046d740 10 client.0 _ll_register_callbacks cb 0x55cf196ee140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f39f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197e2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f39f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197e2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47006/0 (socket says 172.21.15.89:47006) 2024-09-15T23:32:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f39f77fe640 1 -- 172.21.15.89:0/2199052557 learned_addr learned my addr 172.21.15.89:0/2199052557 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f39f6ffd640 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d09a0 0x55cf197dac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f39ee7fc640 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197db160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f39f77fe640 1 -- 172.21.15.89:0/2199052557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 msgr2=0x55cf197db160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f39f77fe640 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197db160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f39f77fe640 1 -- 172.21.15.89:0/2199052557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d09a0 msgr2=0x55cf197dac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:47.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f39f77fe640 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d09a0 0x55cf197dac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.279+0000 7f39f77fe640 1 -- 172.21.15.89:0/2199052557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf19447210 con 0x55cf197d6370 2024-09-15T23:32:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f39f77fe640 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197e2130 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009560 tx=0x7f39f001df20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950c8b7a81026557 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f39de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f39de7fc640 1 -- 172.21.15.89:0/2199052557 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f002b030 con 0x55cf197d6370 2024-09-15T23:32:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f39de7fc640 1 -- 172.21.15.89:0/2199052557 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f39f0014040 con 0x55cf197d6370 2024-09-15T23:32:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f3a0046d740 1 -- 172.21.15.89:0/2199052557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39f001dc80 con 0x55cf197d6370 2024-09-15T23:32:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f39de7fc640 1 -- 172.21.15.89:0/2199052557 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f39f0038050 con 0x55cf197d6370 2024-09-15T23:32:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f3a0046d740 10 client.18297 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:47.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f3a0046d740 1 -- 172.21.15.89:0/2199052557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cf19513620 con 0x55cf197d6370 2024-09-15T23:32:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f39de7fc640 1 -- 172.21.15.89:0/2199052557 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39f001dc80 con 0x55cf197d6370 2024-09-15T23:32:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f39de7fc640 1 -- 172.21.15.89:0/2199052557 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f39f000f040 con 0x55cf197d6370 2024-09-15T23:32:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f3a0046d740 10 client.18297 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f3a0046d740 10 client.18297 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f3a0046d740 20 client.18297 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f3a0046d740 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cf197e6620 0x55cf197e8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f3a0046d740 4 client.18297 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.280+0000 7f3a0046d740 1 -- 172.21.15.89:0/2199052557 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf19741a20 con 0x55cf197e6620 2024-09-15T23:32:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.281+0000 7f39f6ffd640 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cf197e6620 0x55cf197e8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.281+0000 7f39f6ffd640 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cf197e6620 0x55cf197e8a70 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e966157bb48429be server_cookie=558d8c8529d44d72 in_seq=0 out_seq=0 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.281+0000 7f39de7fc640 10 client.18297 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f39de7fc640 1 -- 172.21.15.89:0/2199052557 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cf19741a20 con 0x55cf197e6620 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f39de7fc640 10 client.18297 handle_command_reply: tid=0 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 client.18297 shutdown 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 -- 172.21.15.89:0/2199052557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 msgr2=0x55cf197e2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197e2130 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009560 tx=0x7f39f001df20 comp rx=0 tx=0).stop 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 -- 172.21.15.89:0/2199052557 shutdown_connections 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cf197e6620 0x55cf197e8a70 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197e2130 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197db160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 --2- 172.21.15.89:0/2199052557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d09a0 0x55cf197dac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f39de7fc640 0 client.18297 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 -- 172.21.15.89:0/2199052557 >> 172.21.15.89:0/2199052557 conn(0x55cf197cc010 msgr2=0x55cf197cd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 -- 172.21.15.89:0/2199052557 shutdown_connections 2024-09-15T23:32:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 1 -- 172.21.15.89:0/2199052557 wait complete. 2024-09-15T23:32:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.282+0000 7f3a0046d740 20 client.18297 trim_cache size 0 max 16384 2024-09-15T23:32:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.296+0000 7f3a0046d740 1 Processor -- start 2024-09-15T23:32:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.296+0000 7f3a0046d740 1 -- start start 2024-09-15T23:32:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d09a0 0x55cf197ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d12e0 0x55cf197e2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197cd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f3a0046d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f39f81aa840 con 0x55cf197d6370 2024-09-15T23:32:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f3a0046d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39f81ac380 con 0x55cf197d12e0 2024-09-15T23:32:47.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f3a0046d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f39f81aaf10 con 0x55cf197d09a0 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47016/0 (socket says 172.21.15.89:47016) 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f77fe640 1 -- 172.21.15.89:0/643694156 learned_addr learned my addr 172.21.15.89:0/643694156 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f6ffd640 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d09a0 0x55cf197ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39ee7fc640 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d12e0 0x55cf197e2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f77fe640 1 -- 172.21.15.89:0/643694156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d09a0 msgr2=0x55cf197ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f77fe640 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d09a0 0x55cf197ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f77fe640 1 -- 172.21.15.89:0/643694156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d12e0 msgr2=0x55cf197e2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f77fe640 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d12e0 0x55cf197e2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f77fe640 1 -- 172.21.15.89:0/643694156 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf197e90f0 con 0x55cf197d6370 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f6ffd640 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d09a0 0x55cf197ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:47.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39ee7fc640 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d12e0 0x55cf197e2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39f77fe640 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197cd400 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f39f00127a0 tx=0x7f39f0009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0fc93a48893317f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39ecff9640 1 -- 172.21.15.89:0/643694156 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f0034020 con 0x55cf197d6370 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39ecff9640 1 -- 172.21.15.89:0/643694156 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39f00050a0 con 0x55cf197d6370 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.297+0000 7f39ecff9640 1 -- 172.21.15.89:0/643694156 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f002b070 con 0x55cf197d6370 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.298+0000 7f3a0046d740 1 -- 172.21.15.89:0/643694156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 msgr2=0x55cf197cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.298+0000 7f3a0046d740 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197cd400 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f39f00127a0 tx=0x7f39f0009be0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.298+0000 7f3a0046d740 1 -- 172.21.15.89:0/643694156 shutdown_connections 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.298+0000 7f3a0046d740 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d6370 0x55cf197cd400 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.298+0000 7f3a0046d740 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d12e0 0x55cf197e2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.298+0000 7f3a0046d740 1 --2- 172.21.15.89:0/643694156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d09a0 0x55cf197ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.298+0000 7f3a0046d740 1 -- 172.21.15.89:0/643694156 >> 172.21.15.89:0/643694156 conn(0x55cf197cc010 msgr2=0x55cf197dac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.298+0000 7f3a0046d740 1 -- 172.21.15.89:0/643694156 shutdown_connections 2024-09-15T23:32:47.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.298+0000 7f3a0046d740 1 -- 172.21.15.89:0/643694156 wait complete. 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f3a0046d740 1 -- start start 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f3a0046d740 1 Processor -- start 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197df410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197df950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f3a0046d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 0x55cf197e0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f3a0046d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f39f81abcb0 con 0x55cf197d09a0 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f3a0046d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39f81ab5e0 con 0x55cf197d6370 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f3a0046d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f39f81a8820 con 0x55cf197d12e0 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f3a0046d740 10 client.0 _ll_register_callbacks cb 0x7f39e800a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197df410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197df410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47032/0 (socket says 172.21.15.89:47032) 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f6ffd640 1 -- 172.21.15.89:0/824278654 learned_addr learned my addr 172.21.15.89:0/824278654 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39ee7fc640 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197df950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f77fe640 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 0x55cf197e0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f6ffd640 1 -- 172.21.15.89:0/824278654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 msgr2=0x55cf197df950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f6ffd640 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197df950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f6ffd640 1 -- 172.21.15.89:0/824278654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 msgr2=0x55cf197e0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f6ffd640 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 0x55cf197e0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f6ffd640 1 -- 172.21.15.89:0/824278654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf197e0f70 con 0x55cf197d09a0 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f77fe640 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 0x55cf197e0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39f6ffd640 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197df410 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x55cf197d1d10 tx=0x7f39e8010af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96317b2f3c594fd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.299+0000 7f39de7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f39de7fc640 1 -- 172.21.15.89:0/824278654 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e802b060 con 0x55cf197d09a0 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f39de7fc640 1 -- 172.21.15.89:0/824278654 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f39e8004230 con 0x55cf197d09a0 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f3a0046d740 1 -- 172.21.15.89:0/824278654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39f00050a0 con 0x55cf197d09a0 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f39de7fc640 1 -- 172.21.15.89:0/824278654 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f39e8032050 con 0x55cf197d09a0 2024-09-15T23:32:47.307 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f3a0046d740 10 client.18303 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f3a0046d740 1 -- 172.21.15.89:0/824278654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cf197e2350 con 0x55cf197d09a0 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f39de7fc640 1 -- 172.21.15.89:0/824278654 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39f00050a0 con 0x55cf197d09a0 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f39de7fc640 1 -- 172.21.15.89:0/824278654 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f39e80038c0 con 0x55cf197d09a0 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f3a0046d740 10 client.18303 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f3a0046d740 10 client.18303 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f3a0046d740 20 client.18303 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f3a0046d740 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cf19851a40 0x55cf19853e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f3a0046d740 4 client.18303 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f3a0046d740 1 -- 172.21.15.89:0/824278654 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf1973bd70 con 0x55cf19851a40 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.300+0000 7f39ee7fc640 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cf19851a40 0x55cf19853e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.301+0000 7f39ee7fc640 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cf19851a40 0x55cf19853e60 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e964034da49d5e70 server_cookie=64e89624b011c3cd in_seq=0 out_seq=0 2024-09-15T23:32:47.308 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.301+0000 7f39de7fc640 10 client.18303 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f39de7fc640 1 -- 172.21.15.89:0/824278654 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cf1973bd70 con 0x55cf19851a40 2024-09-15T23:32:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f39de7fc640 10 client.18303 handle_command_reply: tid=0 2024-09-15T23:32:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 client.18303 shutdown 2024-09-15T23:32:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 -- 172.21.15.89:0/824278654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 msgr2=0x55cf197df410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197df410 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x55cf197d1d10 tx=0x7f39e8010af0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 -- 172.21.15.89:0/824278654 shutdown_connections 2024-09-15T23:32:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cf19851a40 0x55cf19853e60 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cf197d6370 0x55cf197e0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cf197d12e0 0x55cf197df950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f39de7fc640 0 client.18303 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:47.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 --2- 172.21.15.89:0/824278654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cf197d09a0 0x55cf197df410 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 -- 172.21.15.89:0/824278654 >> 172.21.15.89:0/824278654 conn(0x55cf197cc010 msgr2=0x55cf197ca9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:47.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 -- 172.21.15.89:0/824278654 shutdown_connections 2024-09-15T23:32:47.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 1 -- 172.21.15.89:0/824278654 wait complete. 2024-09-15T23:32:47.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.316+0000 7f3a0046d740 20 client.18303 trim_cache size 0 max 16384 2024-09-15T23:32:47.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:32:47.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:32:47.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:32:47.320 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:32:47.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 -- 172.21.15.89:0/4157486189 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f39e40a32f0 msgr2=0x7f39e40a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:47.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f39e40a32f0 0x7f39e40a57b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f39f0005c40 tx=0x7f39f001e000 comp rx=0 tx=0).stop 2024-09-15T23:32:47.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 -- 172.21.15.89:0/4157486189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f8101c40 msgr2=0x7f39f8198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:47.321 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f8101c40 0x7f39f8198300 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f39e0006010 tx=0x7f39e0004360 comp rx=0 tx=0).stop 2024-09-15T23:32:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 -- 172.21.15.89:0/4157486189 shutdown_connections 2024-09-15T23:32:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f39e40a32f0 0x7f39e40a57b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39f8102600 0x7f39f819f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f39f8101c40 0x7f39f8198300 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 --2- 172.21.15.89:0/4157486189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39f8100a50 0x7f39f8197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 -- 172.21.15.89:0/4157486189 >> 172.21.15.89:0/4157486189 conn(0x7f39f80fc1b0 msgr2=0x7f39f806a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 -- 172.21.15.89:0/4157486189 shutdown_connections 2024-09-15T23:32:47.322 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:32:47.317+0000 7f39fd5f6640 1 -- 172.21.15.89:0/4157486189 wait complete. 2024-09-15T23:32:47.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:47 smithi174 ceph-mon[35586]: pgmap v613: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:32:47.525 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:47 smithi120 ceph-mon[35250]: pgmap v613: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:32:47.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:47 smithi089 ceph-mon[30933]: pgmap v613: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:32:47.871 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:32:48.132 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:48.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.639+0000 7f2365801640 1 -- 172.21.15.89:0/618791000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2360101290 msgr2=0x7f236010d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:48.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.639+0000 7f2365801640 1 --2- 172.21.15.89:0/618791000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2360101290 0x7f236010d7c0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2358009cd0 tx=0x7f235802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.639+0000 7f2365801640 1 -- 172.21.15.89:0/618791000 shutdown_connections 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.639+0000 7f2365801640 1 --2- 172.21.15.89:0/618791000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2360101290 0x7f236010d7c0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.639+0000 7f2365801640 1 --2- 172.21.15.89:0/618791000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23601008f0 0x7f2360100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.639+0000 7f2365801640 1 --2- 172.21.15.89:0/618791000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23601068f0 0x7f2360106cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.639+0000 7f2365801640 1 -- 172.21.15.89:0/618791000 >> 172.21.15.89:0/618791000 conn(0x7f23600fc630 msgr2=0x7f23600fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.639+0000 7f2365801640 1 -- 172.21.15.89:0/618791000 shutdown_connections 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.639+0000 7f2365801640 1 -- 172.21.15.89:0/618791000 wait complete. 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f2365801640 1 Processor -- start 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f2365801640 1 -- start start 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f2365801640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23601008f0 0x7f2360113090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f2365801640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2360101290 0x7f2360111450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f2365801640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23601068f0 0x7f2360111b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f2365801640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2360110cf0 con 0x7f23601068f0 2024-09-15T23:32:48.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f2365801640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2360110b70 con 0x7f2360101290 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f2365801640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2360110e70 con 0x7f23601008f0 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f235f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23601068f0 0x7f2360111b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f235f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23601068f0 0x7f2360111b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47064/0 (socket says 172.21.15.89:47064) 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f235f7fe640 1 -- 172.21.15.89:0/3039067667 learned_addr learned my addr 172.21.15.89:0/3039067667 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f235effd640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23601008f0 0x7f2360113090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.640+0000 7f235e7fc640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2360101290 0x7f2360111450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f235f7fe640 1 -- 172.21.15.89:0/3039067667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23601008f0 msgr2=0x7f2360113090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f235f7fe640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23601008f0 0x7f2360113090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f235f7fe640 1 -- 172.21.15.89:0/3039067667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2360101290 msgr2=0x7f2360111450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f235f7fe640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2360101290 0x7f2360111450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f235f7fe640 1 -- 172.21.15.89:0/3039067667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23601a67f0 con 0x7f23601068f0 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f235f7fe640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23601068f0 0x7f2360111b90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f2358005cc0 tx=0x7f2358003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:48.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f2347fff640 1 -- 172.21.15.89:0/3039067667 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358004040 con 0x7f23601068f0 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f2365801640 1 -- 172.21.15.89:0/3039067667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23601a6a20 con 0x7f23601068f0 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f2347fff640 1 -- 172.21.15.89:0/3039067667 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23580046c0 con 0x7f23601068f0 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f2347fff640 1 -- 172.21.15.89:0/3039067667 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358046480 con 0x7f23601068f0 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.641+0000 7f2365801640 1 -- 172.21.15.89:0/3039067667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23601a6f30 con 0x7f23601068f0 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.643+0000 7f2347fff640 1 -- 172.21.15.89:0/3039067667 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f23580305a0 con 0x7f23601068f0 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f2367bda740 1 Processor -- start 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f2347fff640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23480a3240 0x7f23480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f2367bda740 1 -- start start 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f235e7fc640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23480a3240 0x7f23480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f2367bda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c52ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f2347fff640 1 -- 172.21.15.89:0/3039067667 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f235802e030 con 0x7f23601068f0 2024-09-15T23:32:48.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f2367bda740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c537310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f2367bda740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c539c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f2367bda740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562b8c4922e0 con 0x562b8c537850 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f2367bda740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562b8c439ce0 con 0x562b8c52cf90 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.644+0000 7f2367bda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562b8c48e2a0 con 0x562b8c52c670 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c539c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c539c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47078/0 (socket says 172.21.15.89:47078) 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235f7fe640 1 -- 172.21.15.89:0/3467477210 learned_addr learned my addr 172.21.15.89:0/3467477210 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235effd640 1 --2- 172.21.15.89:0/3467477210 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c52ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235e7fc640 1 --2- 172.21.15.89:0/3467477210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c537310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235f7fe640 1 -- 172.21.15.89:0/3467477210 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 msgr2=0x562b8c52ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235f7fe640 1 --2- 172.21.15.89:0/3467477210 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c52ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235f7fe640 1 -- 172.21.15.89:0/3467477210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 msgr2=0x562b8c537310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235f7fe640 1 --2- 172.21.15.89:0/3467477210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c537310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235f7fe640 1 -- 172.21.15.89:0/3467477210 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b8c41e440 con 0x562b8c537850 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235e7fc640 1 --2- 172.21.15.89:0/3467477210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c537310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235e7fc640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23480a3240 0x7f23480a5700 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2360112c30 tx=0x7f235000af40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f235f7fe640 1 --2- 172.21.15.89:0/3467477210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c539c40 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f2358005e30 tx=0x7f23580b9a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ededef71886925e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f23457fa640 1 -- 172.21.15.89:0/3467477210 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23580bdda0 con 0x562b8c537850 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f23457fa640 1 -- 172.21.15.89:0/3467477210 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2358036040 con 0x562b8c537850 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.645+0000 7f23457fa640 1 -- 172.21.15.89:0/3467477210 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23580b9d10 con 0x562b8c537850 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.646+0000 7f2367bda740 1 -- 172.21.15.89:0/3467477210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 msgr2=0x562b8c539c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.646+0000 7f2367bda740 1 --2- 172.21.15.89:0/3467477210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c539c40 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f2358005e30 tx=0x7f23580b9a40 comp rx=0 tx=0).stop 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.646+0000 7f2367bda740 1 -- 172.21.15.89:0/3467477210 shutdown_connections 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.646+0000 7f2367bda740 1 --2- 172.21.15.89:0/3467477210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c539c40 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.646+0000 7f2367bda740 1 --2- 172.21.15.89:0/3467477210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c537310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.646+0000 7f2367bda740 1 --2- 172.21.15.89:0/3467477210 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c52ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.646+0000 7f2367bda740 1 -- 172.21.15.89:0/3467477210 >> 172.21.15.89:0/3467477210 conn(0x562b8c528c00 msgr2=0x562b8c528fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:48.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.646+0000 7f2367bda740 1 -- 172.21.15.89:0/3467477210 shutdown_connections 2024-09-15T23:32:48.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.646+0000 7f2367bda740 1 -- 172.21.15.89:0/3467477210 wait complete. 2024-09-15T23:32:48.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.646+0000 7f2367bda740 1 -- start start 2024-09-15T23:32:48.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2367bda740 1 Processor -- start 2024-09-15T23:32:48.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2367bda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c537290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2367bda740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c534540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2367bda740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c534cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2367bda740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562b8c53a1e0 con 0x562b8c537850 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2367bda740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562b8c44d050 con 0x562b8c52cf90 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2367bda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562b8c53a360 con 0x562b8c52c670 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2367bda740 10 client.0 _ll_register_callbacks cb 0x562b8c48d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c534cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c534cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47094/0 (socket says 172.21.15.89:47094) 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235f7fe640 1 -- 172.21.15.89:0/331413311 learned_addr learned my addr 172.21.15.89:0/331413311 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235effd640 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c537290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235e7fc640 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c534540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235f7fe640 1 -- 172.21.15.89:0/331413311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 msgr2=0x562b8c537290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:48.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235f7fe640 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c537290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235f7fe640 1 -- 172.21.15.89:0/331413311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 msgr2=0x562b8c534540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235f7fe640 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c534540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235f7fe640 1 -- 172.21.15.89:0/331413311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b8c2033c0 con 0x562b8c537850 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f235f7fe640 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c534cd0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f2358009ca0 tx=0x7f2358005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441e5aa3b9292a3b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2342ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2342ff5640 1 -- 172.21.15.89:0/331413311 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23580da030 con 0x562b8c537850 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2342ff5640 1 -- 172.21.15.89:0/331413311 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f23580b9d10 con 0x562b8c537850 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.647+0000 7f2367bda740 1 -- 172.21.15.89:0/331413311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2358036040 con 0x562b8c537850 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2342ff5640 1 -- 172.21.15.89:0/331413311 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f23580a5050 con 0x562b8c537850 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2367bda740 10 client.18315 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2367bda740 1 -- 172.21.15.89:0/331413311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562b8c53a1e0 con 0x562b8c537850 2024-09-15T23:32:48.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2342ff5640 1 -- 172.21.15.89:0/331413311 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2358036040 con 0x562b8c537850 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2342ff5640 1 -- 172.21.15.89:0/331413311 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f235804e070 con 0x562b8c537850 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2367bda740 10 client.18315 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2367bda740 10 client.18315 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2367bda740 20 client.18315 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2367bda740 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562b8c546ee0 0x562b8c549300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2367bda740 4 client.18315 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f2367bda740 1 -- 172.21.15.89:0/331413311 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b8c49f850 con 0x562b8c546ee0 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.648+0000 7f235effd640 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562b8c546ee0 0x562b8c549300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.649+0000 7f2342ff5640 10 client.18315 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.649+0000 7f235effd640 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562b8c546ee0 0x562b8c549300 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84dac98d5c63bd93 server_cookie=d029b87085744808 in_seq=0 out_seq=0 2024-09-15T23:32:48.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.649+0000 7f2342ff5640 1 -- 172.21.15.89:0/331413311 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562b8c49f850 con 0x562b8c546ee0 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.649+0000 7f2342ff5640 10 client.18315 handle_command_reply: tid=0 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.649+0000 7f2367bda740 1 client.18315 shutdown 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 1 -- 172.21.15.89:0/331413311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 msgr2=0x562b8c534cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c534cd0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f2358009ca0 tx=0x7f2358005ef0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 1 -- 172.21.15.89:0/331413311 shutdown_connections 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562b8c546ee0 0x562b8c549300 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c534cd0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c534540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 1 --2- 172.21.15.89:0/331413311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c537290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 1 -- 172.21.15.89:0/331413311 >> 172.21.15.89:0/331413311 conn(0x562b8c528c00 msgr2=0x562b8c52e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2342ff5640 0 client.18315 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 1 -- 172.21.15.89:0/331413311 shutdown_connections 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 1 -- 172.21.15.89:0/331413311 wait complete. 2024-09-15T23:32:48.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.650+0000 7f2367bda740 20 client.18315 trim_cache size 0 max 16384 2024-09-15T23:32:48.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f2367bda740 1 Processor -- start 2024-09-15T23:32:48.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f2367bda740 1 -- start start 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f2367bda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c534510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f2367bda740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c54ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f2367bda740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c54f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f2367bda740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f23601b0e20 con 0x562b8c537850 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f2367bda740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f23601b2960 con 0x562b8c52cf90 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f2367bda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f23601b14f0 con 0x562b8c52c670 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f235f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c54f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f235f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c54f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47100/0 (socket says 172.21.15.89:47100) 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f235f7fe640 1 -- 172.21.15.89:0/223866192 learned_addr learned my addr 172.21.15.89:0/223866192 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f235effd640 1 --2- 172.21.15.89:0/223866192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c534510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f235f7fe640 1 -- 172.21.15.89:0/223866192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 msgr2=0x562b8c534510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f235f7fe640 1 --2- 172.21.15.89:0/223866192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c534510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f235f7fe640 1 -- 172.21.15.89:0/223866192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 msgr2=0x562b8c54ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f235f7fe640 1 --2- 172.21.15.89:0/223866192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c54ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f235f7fe640 1 -- 172.21.15.89:0/223866192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b8c52c4a0 con 0x562b8c537850 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.667+0000 7f235f7fe640 1 --2- 172.21.15.89:0/223866192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c54f150 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f23580c5de0 tx=0x7f23580098b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31dea732d9ec2432 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2344ff9640 1 -- 172.21.15.89:0/223866192 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23580da030 con 0x562b8c537850 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2344ff9640 1 -- 172.21.15.89:0/223866192 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2358036040 con 0x562b8c537850 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2344ff9640 1 -- 172.21.15.89:0/223866192 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23580bb9a0 con 0x562b8c537850 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2367bda740 1 -- 172.21.15.89:0/223866192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 msgr2=0x562b8c54f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2367bda740 1 --2- 172.21.15.89:0/223866192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c54f150 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f23580c5de0 tx=0x7f23580098b0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2367bda740 1 -- 172.21.15.89:0/223866192 shutdown_connections 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2367bda740 1 --2- 172.21.15.89:0/223866192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c537850 0x562b8c54f150 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2367bda740 1 --2- 172.21.15.89:0/223866192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c52cf90 0x562b8c54ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2367bda740 1 --2- 172.21.15.89:0/223866192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52c670 0x562b8c534510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2367bda740 1 -- 172.21.15.89:0/223866192 >> 172.21.15.89:0/223866192 conn(0x562b8c528c00 msgr2=0x562b8c533e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2367bda740 1 -- 172.21.15.89:0/223866192 shutdown_connections 2024-09-15T23:32:48.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.668+0000 7f2367bda740 1 -- 172.21.15.89:0/223866192 wait complete. 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f2367bda740 1 -- start start 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f2367bda740 1 Processor -- start 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f2367bda740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c52c670 0x562b8c53f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f2367bda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52cf90 0x562b8c53fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f2367bda740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c537850 0x562b8c530e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f2367bda740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f23601b2290 con 0x562b8c52c670 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f2367bda740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f23601b1bc0 con 0x562b8c537850 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f2367bda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f23601aee00 con 0x562b8c52cf90 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f2367bda740 10 client.0 _ll_register_callbacks cb 0x7f235400b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f235effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c52c670 0x562b8c53f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f235effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c52c670 0x562b8c53f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47104/0 (socket says 172.21.15.89:47104) 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f235effd640 1 -- 172.21.15.89:0/3873568240 learned_addr learned my addr 172.21.15.89:0/3873568240 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f235e7fc640 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52cf90 0x562b8c53fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f235f7fe640 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c537850 0x562b8c530e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f235effd640 1 -- 172.21.15.89:0/3873568240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52cf90 msgr2=0x562b8c53fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:48.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f235effd640 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52cf90 0x562b8c53fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f235effd640 1 -- 172.21.15.89:0/3873568240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c537850 msgr2=0x562b8c530e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f235effd640 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c537850 0x562b8c530e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.669+0000 7f235effd640 1 -- 172.21.15.89:0/3873568240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b8c5316a0 con 0x562b8c52c670 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f235effd640 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c52c670 0x562b8c53f790 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f2354005fd0 tx=0x7f235400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=759d4b14fce533f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2342ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2342ff5640 1 -- 172.21.15.89:0/3873568240 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2354005220 con 0x562b8c52c670 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2342ff5640 1 -- 172.21.15.89:0/3873568240 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f23540053c0 con 0x562b8c52c670 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2367bda740 1 -- 172.21.15.89:0/3873568240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2358036040 con 0x562b8c52c670 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2342ff5640 1 -- 172.21.15.89:0/3873568240 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f23540055c0 con 0x562b8c52c670 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2367bda740 10 client.18321 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:48.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2367bda740 1 -- 172.21.15.89:0/3873568240 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562b8c5b17b0 con 0x562b8c52c670 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2342ff5640 1 -- 172.21.15.89:0/3873568240 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2358036040 con 0x562b8c52c670 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2342ff5640 1 -- 172.21.15.89:0/3873568240 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f235401d810 con 0x562b8c52c670 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2367bda740 10 client.18321 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2367bda740 10 client.18321 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2367bda740 20 client.18321 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2367bda740 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562b8c5b1f00 0x562b8c5b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2367bda740 4 client.18321 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.670+0000 7f2367bda740 1 -- 172.21.15.89:0/3873568240 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562b8c499d60 con 0x562b8c5b1f00 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f235e7fc640 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562b8c5b1f00 0x562b8c5b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f235e7fc640 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562b8c5b1f00 0x562b8c5b4320 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec076b0a05eed322 server_cookie=4494ccaef7c20488 in_seq=0 out_seq=0 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2342ff5640 10 client.18321 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2342ff5640 1 -- 172.21.15.89:0/3873568240 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562b8c499d60 con 0x562b8c5b1f00 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2342ff5640 10 client.18321 handle_command_reply: tid=0 2024-09-15T23:32:48.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 client.18321 shutdown 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 -- 172.21.15.89:0/3873568240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c52c670 msgr2=0x562b8c53f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c52c670 0x562b8c53f790 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f2354005fd0 tx=0x7f235400bdd0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 -- 172.21.15.89:0/3873568240 shutdown_connections 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562b8c5b1f00 0x562b8c5b4320 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562b8c537850 0x562b8c530e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562b8c52cf90 0x562b8c53fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 --2- 172.21.15.89:0/3873568240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b8c52c670 0x562b8c53f790 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 -- 172.21.15.89:0/3873568240 >> 172.21.15.89:0/3873568240 conn(0x562b8c528c00 msgr2=0x562b8c52f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2342ff5640 0 client.18321 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 -- 172.21.15.89:0/3873568240 shutdown_connections 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 1 -- 172.21.15.89:0/3873568240 wait complete. 2024-09-15T23:32:48.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.671+0000 7f2367bda740 20 client.18321 trim_cache size 0 max 16384 2024-09-15T23:32:48.686 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:32:48.686 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:32:48.686 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:32:48.686 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:32:48.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 -- 172.21.15.89:0/3039067667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23480a3240 msgr2=0x7f23480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:48.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23480a3240 0x7f23480a5700 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2360112c30 tx=0x7f235000af40 comp rx=0 tx=0).stop 2024-09-15T23:32:48.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 -- 172.21.15.89:0/3039067667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23601068f0 msgr2=0x7f2360111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:48.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23601068f0 0x7f2360111b90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f2358005cc0 tx=0x7f2358003970 comp rx=0 tx=0).stop 2024-09-15T23:32:48.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 -- 172.21.15.89:0/3039067667 shutdown_connections 2024-09-15T23:32:48.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23480a3240 0x7f23480a5700 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f23601068f0 0x7f2360111b90 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2360101290 0x7f2360111450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 --2- 172.21.15.89:0/3039067667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23601008f0 0x7f2360113090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:48.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 -- 172.21.15.89:0/3039067667 >> 172.21.15.89:0/3039067667 conn(0x7f23600fc630 msgr2=0x7f23600fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:48.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 -- 172.21.15.89:0/3039067667 shutdown_connections 2024-09-15T23:32:48.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:48.672+0000 7f2365801640 1 -- 172.21.15.89:0/3039067667 wait complete. 2024-09-15T23:32:49.448 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:32:49.448 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:32:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:49 smithi174 ceph-mon[35586]: pgmap v614: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:32:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:49 smithi089 ceph-mon[30933]: pgmap v614: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:32:49.695 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:49.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:49 smithi120 ceph-mon[35250]: pgmap v614: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:32:50.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 -- 172.21.15.89:0/1314288432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6160102570 msgr2=0x7f61601029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:50.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 --2- 172.21.15.89:0/1314288432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6160102570 0x7f61601029b0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f614c009900 tx=0x7f614c02a480 comp rx=0 tx=0).stop 2024-09-15T23:32:50.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 -- 172.21.15.89:0/1314288432 shutdown_connections 2024-09-15T23:32:50.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 --2- 172.21.15.89:0/1314288432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6160102ef0 0x7f616010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:50.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 --2- 172.21.15.89:0/1314288432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6160102570 0x7f61601029b0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:50.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 --2- 172.21.15.89:0/1314288432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6160100e20 0x7f6160101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:50.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 -- 172.21.15.89:0/1314288432 >> 172.21.15.89:0/1314288432 conn(0x7f61600fc7d0 msgr2=0x7f61600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:50.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 -- 172.21.15.89:0/1314288432 shutdown_connections 2024-09-15T23:32:50.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 -- 172.21.15.89:0/1314288432 wait complete. 2024-09-15T23:32:50.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 Processor -- start 2024-09-15T23:32:50.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.178+0000 7f6166c7a640 1 -- start start 2024-09-15T23:32:50.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f6166c7a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6160100e20 0x7f61601993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:50.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f6166c7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6160102570 0x7f6160199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:50.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f6166c7a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6160102ef0 0x7f61601a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:50.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f6166c7a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6160110fc0 con 0x7f6160102570 2024-09-15T23:32:50.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f6166c7a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6160110e40 con 0x7f6160100e20 2024-09-15T23:32:50.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f6166c7a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6160111140 con 0x7f6160102ef0 2024-09-15T23:32:50.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6160102570 0x7f6160199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:50.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6160102570 0x7f6160199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47134/0 (socket says 172.21.15.89:47134) 2024-09-15T23:32:50.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615ffff640 1 -- 172.21.15.89:0/1159835543 learned_addr learned my addr 172.21.15.89:0/1159835543 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f6164a16640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6160100e20 0x7f61601993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f6165217640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6160102ef0 0x7f61601a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615ffff640 1 -- 172.21.15.89:0/1159835543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6160102ef0 msgr2=0x7f61601a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615ffff640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6160102ef0 0x7f61601a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615ffff640 1 -- 172.21.15.89:0/1159835543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6160100e20 msgr2=0x7f61601993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615ffff640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6160100e20 0x7f61601993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615ffff640 1 -- 172.21.15.89:0/1159835543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61601a49b0 con 0x7f6160102570 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615ffff640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6160102570 0x7f6160199930 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f614c006010 tx=0x7f614c003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615dffb640 1 -- 172.21.15.89:0/1159835543 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c03f070 con 0x7f6160102570 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f6166c7a640 1 -- 172.21.15.89:0/1159835543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61601a4b80 con 0x7f6160102570 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615dffb640 1 -- 172.21.15.89:0/1159835543 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f614c031e30 con 0x7f6160102570 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.179+0000 7f615dffb640 1 -- 172.21.15.89:0/1159835543 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c035030 con 0x7f6160102570 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.180+0000 7f6166c7a640 1 -- 172.21.15.89:0/1159835543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61601a4fd0 con 0x7f6160102570 2024-09-15T23:32:50.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.181+0000 7f6166c7a640 1 -- 172.21.15.89:0/1159835543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6160101200 con 0x7f6160102570 2024-09-15T23:32:50.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.182+0000 7f615dffb640 1 -- 172.21.15.89:0/1159835543 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f614c042070 con 0x7f6160102570 2024-09-15T23:32:50.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.183+0000 7f615dffb640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61500a3240 0x7f61500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:50.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.183+0000 7f6164a16640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61500a3240 0x7f61500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:50.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.183+0000 7f615dffb640 1 -- 172.21.15.89:0/1159835543 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f614c0f0e30 con 0x7f6160102570 2024-09-15T23:32:50.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.183+0000 7f6164a16640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61500a3240 0x7f61500a5700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6154009990 tx=0x7f6154009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:50.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.186+0000 7f615dffb640 1 -- 172.21.15.89:0/1159835543 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f614c0bd760 con 0x7f6160102570 2024-09-15T23:32:50.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.390+0000 7f6166c7a640 1 -- 172.21.15.89:0/1159835543 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6160109090 con 0x7f6160102570 2024-09-15T23:32:50.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.391+0000 7f615dffb640 1 -- 172.21.15.89:0/1159835543 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f614c0bd060 con 0x7f6160102570 2024-09-15T23:32:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:32:50.393 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:32:50.393 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:32:50.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 -- 172.21.15.89:0/1159835543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61500a3240 msgr2=0x7f61500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:50.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61500a3240 0x7f61500a5700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6154009990 tx=0x7f6154009110 comp rx=0 tx=0).stop 2024-09-15T23:32:50.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 -- 172.21.15.89:0/1159835543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6160102570 msgr2=0x7f6160199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:50.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6160102570 0x7f6160199930 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f614c006010 tx=0x7f614c003c80 comp rx=0 tx=0).stop 2024-09-15T23:32:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 -- 172.21.15.89:0/1159835543 shutdown_connections 2024-09-15T23:32:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61500a3240 0x7f61500a5700 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6160102ef0 0x7f61601a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6160102570 0x7f6160199930 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 --2- 172.21.15.89:0/1159835543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6160100e20 0x7f61601993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 -- 172.21.15.89:0/1159835543 >> 172.21.15.89:0/1159835543 conn(0x7f61600fc7d0 msgr2=0x7f61600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 -- 172.21.15.89:0/1159835543 shutdown_connections 2024-09-15T23:32:50.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:50.393+0000 7f6166c7a640 1 -- 172.21.15.89:0/1159835543 wait complete. 2024-09-15T23:32:50.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:50 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1159835543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:50.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:50 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1159835543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:50.763 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:50 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1159835543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:51.109 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:32:51.356 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:51.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:51 smithi120 ceph-mon[35250]: pgmap v615: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:32:51.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:51 smithi174 ceph-mon[35586]: pgmap v615: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:32:51.785 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:51 smithi089 ceph-mon[30933]: pgmap v615: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:32:51.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.864+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/4253065976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d98102d40 msgr2=0x7f3d98103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:51.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.864+0000 7f3d9d5e6640 1 --2- 172.21.15.89:0/4253065976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d98102d40 0x7f3d98103120 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009900 tx=0x7f3d8802a480 comp rx=0 tx=0).stop 2024-09-15T23:32:51.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.864+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/4253065976 shutdown_connections 2024-09-15T23:32:51.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.864+0000 7f3d9d5e6640 1 --2- 172.21.15.89:0/4253065976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d9810c170 0x7f3d9810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.864+0000 7f3d9d5e6640 1 --2- 172.21.15.89:0/4253065976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98103660 0x7f3d9810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.864+0000 7f3d9d5e6640 1 --2- 172.21.15.89:0/4253065976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d98102d40 0x7f3d98103120 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.864+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/4253065976 >> 172.21.15.89:0/4253065976 conn(0x7f3d980fc7a0 msgr2=0x7f3d980febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:51.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.864+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/4253065976 shutdown_connections 2024-09-15T23:32:51.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.864+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/4253065976 wait complete. 2024-09-15T23:32:51.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d9d5e6640 1 Processor -- start 2024-09-15T23:32:51.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d9d5e6640 1 -- start start 2024-09-15T23:32:51.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d9d5e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d98102d40 0x7f3d98198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d9d5e6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98103660 0x7f3d98198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d9d5e6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9810c170 0x7f3d9819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d9d5e6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3d98111f70 con 0x7f3d9810c170 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d9d5e6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d98111df0 con 0x7f3d98103660 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d9d5e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981120f0 con 0x7f3d98102d40 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d977fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9810c170 0x7f3d9819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d96ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d98102d40 0x7f3d98198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d977fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9810c170 0x7f3d9819cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47158/0 (socket says 172.21.15.89:47158) 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d977fe640 1 -- 172.21.15.89:0/2340310866 learned_addr learned my addr 172.21.15.89:0/2340310866 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.865+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98103660 0x7f3d98198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d977fe640 1 -- 172.21.15.89:0/2340310866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d98102d40 msgr2=0x7f3d98198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:51.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d977fe640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d98102d40 0x7f3d98198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d977fe640 1 -- 172.21.15.89:0/2340310866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98103660 msgr2=0x7f3d98198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d977fe640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98103660 0x7f3d98198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d977fe640 1 -- 172.21.15.89:0/2340310866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9819d300 con 0x7f3d9810c170 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d977fe640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9810c170 0x7f3d9819cc20 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f3d9000cce0 tx=0x7f3d9000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d94ff9640 1 -- 172.21.15.89:0/2340310866 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9000f450 con 0x7f3d9810c170 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d94ff9640 1 -- 172.21.15.89:0/2340310866 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d9000fad0 con 0x7f3d9810c170 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/2340310866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9819d5f0 con 0x7f3d9810c170 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d94ff9640 1 -- 172.21.15.89:0/2340310866 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90018c10 con 0x7f3d9810c170 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.866+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/2340310866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d981a4ed0 con 0x7f3d9810c170 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.869+0000 7f3d9f9bf740 1 Processor -- start 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.869+0000 7f3d94ff9640 1 -- 172.21.15.89:0/2340310866 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3d9000f5f0 con 0x7f3d9810c170 2024-09-15T23:32:51.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.869+0000 7f3d9f9bf740 1 -- start start 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.869+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a3b40 0x55e0d99a2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.869+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.869+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99b1780 0x55e0d99b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.869+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e0d99082f0 con 0x55e0d99a25d0 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.869+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e0d98afcf0 con 0x55e0d99a3b40 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.869+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e0d99042b0 con 0x55e0d99b1780 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99a2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47170/0 (socket says 172.21.15.89:47170) 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d96ffd640 1 -- 172.21.15.89:0/150517125 learned_addr learned my addr 172.21.15.89:0/150517125 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/150517125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a3b40 0x55e0d99a2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d96ffd640 1 -- 172.21.15.89:0/150517125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99b1780 msgr2=0x55e0d99b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d94ff9640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3d840a3240 0x7f3d840a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/150517125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99b1780 0x55e0d99b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d96ffd640 1 -- 172.21.15.89:0/150517125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a3b40 msgr2=0x55e0d99a2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/150517125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a3b40 0x55e0d99a2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d96ffd640 1 -- 172.21.15.89:0/150517125 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0d98943a0 con 0x55e0d99a25d0 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/150517125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a3b40 0x55e0d99a2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d977fe640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3d840a3240 0x7f3d840a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d94ff9640 1 -- 172.21.15.89:0/2340310866 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3d900d0330 con 0x7f3d9810c170 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/150517125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99a2a10 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009900 tx=0x7f3d880316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fcf1ddf4e28b572 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d8d7fa640 1 -- 172.21.15.89:0/150517125 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d88040070 con 0x55e0d99a25d0 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d8d7fa640 1 -- 172.21.15.89:0/150517125 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d88005020 con 0x55e0d99a25d0 2024-09-15T23:32:51.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.870+0000 7f3d8d7fa640 1 -- 172.21.15.89:0/150517125 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d88035040 con 0x55e0d99a25d0 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/150517125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 msgr2=0x55e0d99a2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d977fe640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3d840a3240 0x7f3d840a5700 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3d900989c0 tx=0x7f3d90091a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/150517125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99a2a10 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009900 tx=0x7f3d880316e0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/150517125 shutdown_connections 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/150517125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99b1780 0x55e0d99b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/150517125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99a2a10 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/150517125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a3b40 0x55e0d99a2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/150517125 >> 172.21.15.89:0/150517125 conn(0x55e0d996b310 msgr2=0x55e0d999ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/150517125 shutdown_connections 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/150517125 wait complete. 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.871+0000 7f3d9f9bf740 1 -- start start 2024-09-15T23:32:51.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d9f9bf740 1 Processor -- start 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a25d0 0x55e0d99b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99ae480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99b1780 0x55e0d99aec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e0d99b4420 con 0x55e0d99a3b40 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e0d98c3060 con 0x55e0d99b1780 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e0d99b45a0 con 0x55e0d99a25d0 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d9f9bf740 10 client.0 _ll_register_callbacks cb 0x55e0d96168c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99ae480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47172/0 (socket says 172.21.15.89:47172) 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d96ffd640 1 -- 172.21.15.89:0/2524242194 learned_addr learned my addr 172.21.15.89:0/2524242194 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:51.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a25d0 0x55e0d99b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99b1780 0x55e0d99aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d96ffd640 1 -- 172.21.15.89:0/2524242194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a25d0 msgr2=0x55e0d99b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a25d0 0x55e0d99b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d96ffd640 1 -- 172.21.15.89:0/2524242194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99b1780 msgr2=0x55e0d99aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99b1780 0x55e0d99aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d96ffd640 1 -- 172.21.15.89:0/2524242194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0d96793c0 con 0x55e0d99a3b40 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99ae480 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f3d88031ba0 tx=0x7f3d8802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd3bd956f3a1600 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d7e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.872+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/2524242194 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d88040070 con 0x55e0d99a3b40 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/2524242194 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3d88004be0 con 0x55e0d99a3b40 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/2524242194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d88005020 con 0x55e0d99a3b40 2024-09-15T23:32:51.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/2524242194 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3d88056050 con 0x55e0d99a3b40 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d9f9bf740 10 client.18339 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/2524242194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e0d99bfea0 con 0x55e0d99a3b40 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/2524242194 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d88005020 con 0x55e0d99a3b40 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/2524242194 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3d88044400 con 0x55e0d99a3b40 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d9f9bf740 10 client.18339 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d9f9bf740 10 client.18339 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d9f9bf740 20 client.18339 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55e0d99c11f0 0x55e0d99c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d9f9bf740 4 client.18339 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/2524242194 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0d9915860 con 0x55e0d99c11f0 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.873+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55e0d99c11f0 0x55e0d99c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.874+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55e0d99c11f0 0x55e0d99c35e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6dcb7fd8df40b898 server_cookie=f19b66d4dc8e8289 in_seq=0 out_seq=0 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.874+0000 7f3d7e7fc640 10 client.18339 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:32:51.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/2524242194 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e0d9915860 con 0x55e0d99c11f0 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d7e7fc640 10 client.18339 handle_command_reply: tid=0 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 client.18339 shutdown 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/2524242194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 msgr2=0x55e0d99ae480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99ae480 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f3d88031ba0 tx=0x7f3d8802a480 comp rx=0 tx=0).stop 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/2524242194 shutdown_connections 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55e0d99c11f0 0x55e0d99c35e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99b1780 0x55e0d99aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99ae480 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/2524242194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a25d0 0x55e0d99b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/2524242194 >> 172.21.15.89:0/2524242194 conn(0x55e0d996b310 msgr2=0x55e0d99a4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d7e7fc640 0 client.18339 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/2524242194 shutdown_connections 2024-09-15T23:32:51.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/2524242194 wait complete. 2024-09-15T23:32:51.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.875+0000 7f3d9f9bf740 20 client.18339 trim_cache size 0 max 16384 2024-09-15T23:32:51.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d9f9bf740 1 Processor -- start 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d9f9bf740 1 -- start start 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99ae450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a3b40 0x55e0d99ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99b1780 0x55e0d99abc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981acaf0 con 0x55e0d99a25d0 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981ae630 con 0x55e0d99b1780 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981ad1c0 con 0x55e0d99a3b40 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99ae450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47176/0 (socket says 172.21.15.89:47176) 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8ffff640 1 -- 172.21.15.89:0/61958359 learned_addr learned my addr 172.21.15.89:0/61958359 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/61958359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a3b40 0x55e0d99ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/61958359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99b1780 0x55e0d99abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8ffff640 1 -- 172.21.15.89:0/61958359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a3b40 msgr2=0x55e0d99ab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/61958359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a3b40 0x55e0d99ab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8ffff640 1 -- 172.21.15.89:0/61958359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99b1780 msgr2=0x55e0d99abc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/61958359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99b1780 0x55e0d99abc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8ffff640 1 -- 172.21.15.89:0/61958359 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0d999e510 con 0x55e0d99a25d0 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/61958359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a3b40 0x55e0d99ab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/61958359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99ae450 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f3d80005890 tx=0x7f3d80005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4a43ce822e1c2a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8cff9640 1 -- 172.21.15.89:0/61958359 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8001a040 con 0x55e0d99a25d0 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8cff9640 1 -- 172.21.15.89:0/61958359 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d80012e70 con 0x55e0d99a25d0 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.889+0000 7f3d8cff9640 1 -- 172.21.15.89:0/61958359 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8001b5d0 con 0x55e0d99a25d0 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/61958359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 msgr2=0x55e0d99ae450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/61958359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99ae450 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f3d80005890 tx=0x7f3d80005740 comp rx=0 tx=0).stop 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/61958359 shutdown_connections 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/61958359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99b1780 0x55e0d99abc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/61958359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99a3b40 0x55e0d99ab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/61958359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a25d0 0x55e0d99ae450 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/61958359 >> 172.21.15.89:0/61958359 conn(0x55e0d996b310 msgr2=0x55e0d99add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/61958359 shutdown_connections 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/61958359 wait complete. 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 -- start start 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.890+0000 7f3d9f9bf740 1 Processor -- start 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a25d0 0x55e0d99c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99c3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d9f9bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99b1780 0x55e0d99c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981adf60 con 0x55e0d99a3b40 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981ad890 con 0x55e0d99a25d0 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d9f9bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981aaad0 con 0x55e0d99b1780 2024-09-15T23:32:51.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d9f9bf740 10 client.0 _ll_register_callbacks cb 0x7f3d880315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99c3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47192/0 (socket says 172.21.15.89:47192) 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d96ffd640 1 -- 172.21.15.89:0/3353960852 learned_addr learned my addr 172.21.15.89:0/3353960852 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a25d0 0x55e0d99c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99b1780 0x55e0d99c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d96ffd640 1 -- 172.21.15.89:0/3353960852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99b1780 msgr2=0x55e0d99c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99b1780 0x55e0d99c3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d96ffd640 1 -- 172.21.15.89:0/3353960852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a25d0 msgr2=0x55e0d99c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a25d0 0x55e0d99c31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d96ffd640 1 -- 172.21.15.89:0/3353960852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0d9a29d20 con 0x55e0d99a3b40 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d96ffd640 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99c3710 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f3d8802a9a0 tx=0x7f3d88005620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c48f7b98410372 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d7e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/3353960852 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8805e020 con 0x55e0d99a3b40 2024-09-15T23:32:51.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/3353960852 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3d880047a0 con 0x55e0d99a3b40 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.891+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/3353960852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d80012e70 con 0x55e0d99a3b40 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/3353960852 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3d88040050 con 0x55e0d99a3b40 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d9f9bf740 10 client.18345 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/3353960852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e0d9a2afe0 con 0x55e0d99a3b40 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/3353960852 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d80012e70 con 0x55e0d99a3b40 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/3353960852 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3d8803c5d0 con 0x55e0d99a3b40 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d9f9bf740 10 client.18345 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d9f9bf740 10 client.18345 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d9f9bf740 20 client.18345 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55e0d9a2b750 0x55e0d9a2db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d9f9bf740 4 client.18345 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/3353960852 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e0d990fd70 con 0x55e0d9a2b750 2024-09-15T23:32:51.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.892+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55e0d9a2b750 0x55e0d9a2db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d8ffff640 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55e0d9a2b750 0x55e0d9a2db70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=266d629495112724 server_cookie=e295bcc41ee49736 in_seq=0 out_seq=0 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d7e7fc640 10 client.18345 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d7e7fc640 1 -- 172.21.15.89:0/3353960852 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e0d990fd70 con 0x55e0d9a2b750 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d7e7fc640 10 client.18345 handle_command_reply: tid=0 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 client.18345 shutdown 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/3353960852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 msgr2=0x55e0d99c3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99c3710 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f3d8802a9a0 tx=0x7f3d88005620 comp rx=0 tx=0).stop 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/3353960852 shutdown_connections 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55e0d9a2b750 0x55e0d9a2db70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e0d99b1780 0x55e0d99c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e0d99a3b40 0x55e0d99c3710 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d7e7fc640 0 client.18345 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 --2- 172.21.15.89:0/3353960852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0d99a25d0 0x55e0d99c31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/3353960852 >> 172.21.15.89:0/3353960852 conn(0x55e0d996b310 msgr2=0x55e0d999d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:51.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/3353960852 shutdown_connections 2024-09-15T23:32:51.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 1 -- 172.21.15.89:0/3353960852 wait complete. 2024-09-15T23:32:51.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.893+0000 7f3d9f9bf740 20 client.18345 trim_cache size 0 max 16384 2024-09-15T23:32:51.907 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:32:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/2340310866 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3d840a3240 msgr2=0x7f3d840a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3d840a3240 0x7f3d840a5700 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3d900989c0 tx=0x7f3d90091a40 comp rx=0 tx=0).stop 2024-09-15T23:32:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/2340310866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9810c170 msgr2=0x7f3d9819cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9810c170 0x7f3d9819cc20 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f3d9000cce0 tx=0x7f3d9000a620 comp rx=0 tx=0).stop 2024-09-15T23:32:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/2340310866 shutdown_connections 2024-09-15T23:32:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3d840a3240 0x7f3d840a5700 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9810c170 0x7f3d9819cc20 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98103660 0x7f3d98198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 --2- 172.21.15.89:0/2340310866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d98102d40 0x7f3d98198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:51.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/2340310866 >> 172.21.15.89:0/2340310866 conn(0x7f3d980fc7a0 msgr2=0x7f3d980fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:51.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/2340310866 shutdown_connections 2024-09-15T23:32:51.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:51.894+0000 7f3d9d5e6640 1 -- 172.21.15.89:0/2340310866 wait complete. 2024-09-15T23:32:52.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:32:52.831 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:53.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.267+0000 7f86f8605640 1 -- 172.21.15.89:0/1161177180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f0100480 msgr2=0x7f86f0100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:53.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.267+0000 7f86f8605640 1 --2- 172.21.15.89:0/1161177180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f0100480 0x7f86f0100860 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f86e4009900 tx=0x7f86e402a410 comp rx=0 tx=0).stop 2024-09-15T23:32:53.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.267+0000 7f86f8605640 1 -- 172.21.15.89:0/1161177180 shutdown_connections 2024-09-15T23:32:53.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.267+0000 7f86f8605640 1 --2- 172.21.15.89:0/1161177180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f01055b0 0x7f86f010d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.267+0000 7f86f8605640 1 --2- 172.21.15.89:0/1161177180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86f0100e30 0x7f86f0104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.267+0000 7f86f8605640 1 --2- 172.21.15.89:0/1161177180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f0100480 0x7f86f0100860 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.267+0000 7f86f8605640 1 -- 172.21.15.89:0/1161177180 >> 172.21.15.89:0/1161177180 conn(0x7f86f0077210 msgr2=0x7f86f00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:53.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.267+0000 7f86f8605640 1 -- 172.21.15.89:0/1161177180 shutdown_connections 2024-09-15T23:32:53.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.267+0000 7f86f8605640 1 -- 172.21.15.89:0/1161177180 wait complete. 2024-09-15T23:32:53.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f8605640 1 Processor -- start 2024-09-15T23:32:53.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f8605640 1 -- start start 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f8605640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f0100480 0x7f86f0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f8605640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86f0100e30 0x7f86f0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f8605640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f01055b0 0x7f86f019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f8605640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f86f0111f90 con 0x7f86f0100480 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f8605640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f86f0111e10 con 0x7f86f01055b0 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f8605640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f86f0112110 con 0x7f86f0100e30 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f63a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f0100480 0x7f86f0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f63a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f0100480 0x7f86f0198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47204/0 (socket says 172.21.15.89:47204) 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f63a1640 1 -- 172.21.15.89:0/3047219592 learned_addr learned my addr 172.21.15.89:0/3047219592 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f5ba0640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86f0100e30 0x7f86f0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f63a1640 1 -- 172.21.15.89:0/3047219592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86f0100e30 msgr2=0x7f86f0198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f63a1640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86f0100e30 0x7f86f0198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f01055b0 0x7f86f019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f63a1640 1 -- 172.21.15.89:0/3047219592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f01055b0 msgr2=0x7f86f019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:53.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f63a1640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f01055b0 0x7f86f019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f63a1640 1 -- 172.21.15.89:0/3047219592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f019d3e0 con 0x7f86f0100480 2024-09-15T23:32:53.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.268+0000 7f86f63a1640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f0100480 0x7f86f0198430 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f86e4006010 tx=0x7f86e40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:53.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.269+0000 7f86e37fe640 1 -- 172.21.15.89:0/3047219592 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e403f030 con 0x7f86f0100480 2024-09-15T23:32:53.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.269+0000 7f86e37fe640 1 -- 172.21.15.89:0/3047219592 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86e4003ce0 con 0x7f86f0100480 2024-09-15T23:32:53.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.269+0000 7f86e37fe640 1 -- 172.21.15.89:0/3047219592 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e4035070 con 0x7f86f0100480 2024-09-15T23:32:53.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.269+0000 7f86f8605640 1 -- 172.21.15.89:0/3047219592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f019d670 con 0x7f86f0100480 2024-09-15T23:32:53.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.269+0000 7f86f8605640 1 -- 172.21.15.89:0/3047219592 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f01a4fb0 con 0x7f86f0100480 2024-09-15T23:32:53.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.271+0000 7f86e37fe640 1 -- 172.21.15.89:0/3047219592 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f86e4042030 con 0x7f86f0100480 2024-09-15T23:32:53.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.271+0000 7f86fa9de740 1 Processor -- start 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.271+0000 7f86fa9de740 1 -- start start 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd335d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 0x562fd335da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd336c780 0x562fd336eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86fa9de740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562fd32c32f0 con 0x562fd335eb40 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86fa9de740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562fd326acf0 con 0x562fd335d5d0 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86fa9de740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562fd32bf2b0 con 0x562fd336c780 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86e37fe640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86dc0a3240 0x7f86dc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd335d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd335d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47208/0 (socket says 172.21.15.89:47208) 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f5ba0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd336c780 0x562fd336eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 -- 172.21.15.89:0/2721146221 learned_addr learned my addr 172.21.15.89:0/2721146221 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:53.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86dc0a3240 0x7f86dc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86e37fe640 1 -- 172.21.15.89:0/3047219592 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f86e40f02c0 con 0x7f86f0100480 2024-09-15T23:32:53.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f63a1640 1 --2- 172.21.15.89:0/2721146221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 0x562fd335da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 -- 172.21.15.89:0/2721146221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd336c780 msgr2=0x562fd336eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:53.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/2721146221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd336c780 0x562fd336eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 -- 172.21.15.89:0/2721146221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 msgr2=0x562fd335da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:53.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/2721146221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 0x562fd335da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 -- 172.21.15.89:0/2721146221 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fd324f3a0 con 0x562fd335eb40 2024-09-15T23:32:53.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/2721146221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd335d090 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f86e8014cf0 tx=0x7f86e8011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d437feb3ffb0a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:53.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.272+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86dc0a3240 0x7f86dc0a5700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009fd0 tx=0x7f86e8016000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:53.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86e0ff9640 1 -- 172.21.15.89:0/2721146221 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e8009ab0 con 0x562fd335eb40 2024-09-15T23:32:53.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86e0ff9640 1 -- 172.21.15.89:0/2721146221 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86e8014640 con 0x562fd335eb40 2024-09-15T23:32:53.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86e0ff9640 1 -- 172.21.15.89:0/2721146221 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e8005020 con 0x562fd335eb40 2024-09-15T23:32:53.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86fa9de740 1 -- 172.21.15.89:0/2721146221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 msgr2=0x562fd335d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:53.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86fa9de740 1 --2- 172.21.15.89:0/2721146221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd335d090 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f86e8014cf0 tx=0x7f86e8011750 comp rx=0 tx=0).stop 2024-09-15T23:32:53.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86fa9de740 1 -- 172.21.15.89:0/2721146221 shutdown_connections 2024-09-15T23:32:53.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86fa9de740 1 --2- 172.21.15.89:0/2721146221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd336c780 0x562fd336eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86fa9de740 1 --2- 172.21.15.89:0/2721146221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 0x562fd335da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86fa9de740 1 --2- 172.21.15.89:0/2721146221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd335d090 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86fa9de740 1 -- 172.21.15.89:0/2721146221 >> 172.21.15.89:0/2721146221 conn(0x562fd3326310 msgr2=0x562fd3359c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.273+0000 7f86fa9de740 1 -- 172.21.15.89:0/2721146221 shutdown_connections 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86fa9de740 1 -- 172.21.15.89:0/2721146221 wait complete. 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86fa9de740 1 -- start start 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86fa9de740 1 Processor -- start 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335d5d0 0x562fd336c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335eb40 0x562fd3369490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd336c780 0x562fd3369c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86fa9de740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562fd336f420 con 0x562fd335d5d0 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86fa9de740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562fd327e060 con 0x562fd335eb40 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86fa9de740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562fd336f5a0 con 0x562fd336c780 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86fa9de740 10 client.0 _ll_register_callbacks cb 0x562fd2fd18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:53.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86f63a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335d5d0 0x562fd336c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86f63a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335d5d0 0x562fd336c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47220/0 (socket says 172.21.15.89:47220) 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86f63a1640 1 -- 172.21.15.89:0/29333583 learned_addr learned my addr 172.21.15.89:0/29333583 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86f5ba0640 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335eb40 0x562fd3369490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd336c780 0x562fd3369c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.274+0000 7f86f63a1640 1 -- 172.21.15.89:0/29333583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd336c780 msgr2=0x562fd3369c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86f63a1640 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd336c780 0x562fd3369c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86f63a1640 1 -- 172.21.15.89:0/29333583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335eb40 msgr2=0x562fd3369490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86f63a1640 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335eb40 0x562fd3369490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86f63a1640 1 -- 172.21.15.89:0/29333583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fd30343c0 con 0x562fd335d5d0 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86f63a1640 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335d5d0 0x562fd336c1e0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f86e40ed9d0 tx=0x7f86e40b8910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb709ffcb08ae39 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86d9ffb640 1 -- 172.21.15.89:0/29333583 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e40da060 con 0x562fd335d5d0 2024-09-15T23:32:53.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86d9ffb640 1 -- 172.21.15.89:0/29333583 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f86e40bc810 con 0x562fd335d5d0 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86fa9de740 1 -- 172.21.15.89:0/29333583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86e8014640 con 0x562fd335d5d0 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86d9ffb640 1 -- 172.21.15.89:0/29333583 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86e4049040 con 0x562fd335d5d0 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86fa9de740 10 client.18357 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86fa9de740 1 -- 172.21.15.89:0/29333583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562fd337af40 con 0x562fd335d5d0 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86d9ffb640 1 -- 172.21.15.89:0/29333583 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86e8014640 con 0x562fd335d5d0 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86d9ffb640 1 -- 172.21.15.89:0/29333583 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f86e40c4d90 con 0x562fd335d5d0 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86fa9de740 10 client.18357 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86fa9de740 10 client.18357 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86fa9de740 20 client.18357 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86fa9de740 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x562fd337c330 0x562fd337e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86fa9de740 4 client.18357 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.275+0000 7f86fa9de740 1 -- 172.21.15.89:0/29333583 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fd32d0860 con 0x562fd337c330 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.276+0000 7f86f5ba0640 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x562fd337c330 0x562fd337e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.276+0000 7f86d9ffb640 10 client.18357 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.276+0000 7f86f5ba0640 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x562fd337c330 0x562fd337e780 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6ce44a024bde2284 server_cookie=53930df468a8478 in_seq=0 out_seq=0 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86d9ffb640 1 -- 172.21.15.89:0/29333583 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562fd32d0860 con 0x562fd337c330 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86d9ffb640 10 client.18357 handle_command_reply: tid=0 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 client.18357 shutdown 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 -- 172.21.15.89:0/29333583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335d5d0 msgr2=0x562fd336c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335d5d0 0x562fd336c1e0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f86e40ed9d0 tx=0x7f86e40b8910 comp rx=0 tx=0).stop 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 -- 172.21.15.89:0/29333583 shutdown_connections 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x562fd337c330 0x562fd337e780 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd336c780 0x562fd3369c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335eb40 0x562fd3369490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86d9ffb640 0 client.18357 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 --2- 172.21.15.89:0/29333583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335d5d0 0x562fd336c1e0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 -- 172.21.15.89:0/29333583 >> 172.21.15.89:0/29333583 conn(0x562fd3326310 msgr2=0x562fd335f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:53.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 -- 172.21.15.89:0/29333583 shutdown_connections 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 1 -- 172.21.15.89:0/29333583 wait complete. 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.277+0000 7f86fa9de740 20 client.18357 trim_cache size 0 max 16384 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86fa9de740 1 Processor -- start 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86fa9de740 1 -- start start 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd335d5d0 0x562fd3359e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd3368d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd336c780 0x562fd3384050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86fa9de740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f86f01acb70 con 0x562fd335eb40 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86fa9de740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f86f01ae6b0 con 0x562fd336c780 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86fa9de740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f86f01ad240 con 0x562fd335d5d0 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86f5ba0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd3368d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86f5ba0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd3368d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47222/0 (socket says 172.21.15.89:47222) 2024-09-15T23:32:53.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86f5ba0640 1 -- 172.21.15.89:0/3604002474 learned_addr learned my addr 172.21.15.89:0/3604002474 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/3604002474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd336c780 0x562fd3384050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86f63a1640 1 --2- 172.21.15.89:0/3604002474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd335d5d0 0x562fd3359e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86f5ba0640 1 -- 172.21.15.89:0/3604002474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd335d5d0 msgr2=0x562fd3359e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86f5ba0640 1 --2- 172.21.15.89:0/3604002474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd335d5d0 0x562fd3359e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86f5ba0640 1 -- 172.21.15.89:0/3604002474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd336c780 msgr2=0x562fd3384050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86f5ba0640 1 --2- 172.21.15.89:0/3604002474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd336c780 0x562fd3384050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.290+0000 7f86f5ba0640 1 -- 172.21.15.89:0/3604002474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fd3360120 con 0x562fd335eb40 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86f5ba0640 1 --2- 172.21.15.89:0/3604002474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd3368d50 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0058c0 tx=0x7f86ec00c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b546d84fa740b50 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86dbfff640 1 -- 172.21.15.89:0/3604002474 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec021070 con 0x562fd335eb40 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86dbfff640 1 -- 172.21.15.89:0/3604002474 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86ec0047a0 con 0x562fd335eb40 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86dbfff640 1 -- 172.21.15.89:0/3604002474 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec01c730 con 0x562fd335eb40 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86fa9de740 1 -- 172.21.15.89:0/3604002474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 msgr2=0x562fd3368d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:53.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86fa9de740 1 --2- 172.21.15.89:0/3604002474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd3368d50 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0058c0 tx=0x7f86ec00c430 comp rx=0 tx=0).stop 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86fa9de740 1 -- 172.21.15.89:0/3604002474 shutdown_connections 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86fa9de740 1 --2- 172.21.15.89:0/3604002474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd336c780 0x562fd3384050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86fa9de740 1 --2- 172.21.15.89:0/3604002474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd335eb40 0x562fd3368d50 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86fa9de740 1 --2- 172.21.15.89:0/3604002474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd335d5d0 0x562fd3359e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86fa9de740 1 -- 172.21.15.89:0/3604002474 >> 172.21.15.89:0/3604002474 conn(0x562fd3326310 msgr2=0x562fd3359700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86fa9de740 1 -- 172.21.15.89:0/3604002474 shutdown_connections 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.291+0000 7f86fa9de740 1 -- 172.21.15.89:0/3604002474 wait complete. 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86fa9de740 1 -- start start 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86fa9de740 1 Processor -- start 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 0x562fd3374df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd335eb40 0x562fd3370120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86fa9de740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd336c780 0x562fd3370660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86fa9de740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f86f01adfe0 con 0x562fd336c780 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86fa9de740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f86f01ad910 con 0x562fd335d5d0 2024-09-15T23:32:53.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86fa9de740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f86f01aab50 con 0x562fd335eb40 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86fa9de740 10 client.0 _ll_register_callbacks cb 0x7f86e4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86f6ba2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd336c780 0x562fd3370660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86f6ba2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd336c780 0x562fd3370660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47236/0 (socket says 172.21.15.89:47236) 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86f63a1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 0x562fd3374df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86f6ba2640 1 -- 172.21.15.89:0/2677500475 learned_addr learned my addr 172.21.15.89:0/2677500475 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86f5ba0640 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd335eb40 0x562fd3370120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86f6ba2640 1 -- 172.21.15.89:0/2677500475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd335eb40 msgr2=0x562fd3370120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd335eb40 0x562fd3370120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86f6ba2640 1 -- 172.21.15.89:0/2677500475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 msgr2=0x562fd3374df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 0x562fd3374df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.292+0000 7f86f6ba2640 1 -- 172.21.15.89:0/2677500475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fd3370cd0 con 0x562fd336c780 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86f63a1640 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 0x562fd3374df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:32:53.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86f6ba2640 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd336c780 0x562fd3370660 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f86e8014cf0 tx=0x7f86e800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7954a81d64bcf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86d9ffb640 1 -- 172.21.15.89:0/2677500475 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e800ce50 con 0x562fd336c780 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86d9ffb640 1 -- 172.21.15.89:0/2677500475 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f86e8014250 con 0x562fd336c780 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86fa9de740 1 -- 172.21.15.89:0/2677500475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86ec0047a0 con 0x562fd336c780 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86d9ffb640 1 -- 172.21.15.89:0/2677500475 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86e8009ab0 con 0x562fd336c780 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86fa9de740 10 client.18363 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86fa9de740 1 -- 172.21.15.89:0/2677500475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562fd33e6e90 con 0x562fd336c780 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86d9ffb640 1 -- 172.21.15.89:0/2677500475 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86ec0047a0 con 0x562fd336c780 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86d9ffb640 1 -- 172.21.15.89:0/2677500475 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f86e801d070 con 0x562fd336c780 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86fa9de740 10 client.18363 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86fa9de740 10 client.18363 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86fa9de740 20 client.18363 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:53.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86fa9de740 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x562fd33e7470 0x562fd33e9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86fa9de740 4 client.18363 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.293+0000 7f86fa9de740 1 -- 172.21.15.89:0/2677500475 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562fd32cad70 con 0x562fd33e7470 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86f63a1640 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x562fd33e7470 0x562fd33e9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86f63a1640 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x562fd33e7470 0x562fd33e9890 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=32314d26bcbbf90f server_cookie=92f99de9b7f8c5f2 in_seq=0 out_seq=0 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86d9ffb640 10 client.18363 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86d9ffb640 1 -- 172.21.15.89:0/2677500475 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562fd32cad70 con 0x562fd33e7470 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86d9ffb640 10 client.18363 handle_command_reply: tid=0 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86fa9de740 1 client.18363 shutdown 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86fa9de740 1 -- 172.21.15.89:0/2677500475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd336c780 msgr2=0x562fd3370660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86fa9de740 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd336c780 0x562fd3370660 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f86e8014cf0 tx=0x7f86e800cd70 comp rx=0 tx=0).stop 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86fa9de740 1 -- 172.21.15.89:0/2677500475 shutdown_connections 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86fa9de740 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x562fd33e7470 0x562fd33e9890 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86fa9de740 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fd336c780 0x562fd3370660 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86fa9de740 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fd335eb40 0x562fd3370120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86fa9de740 1 --2- 172.21.15.89:0/2677500475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fd335d5d0 0x562fd3374df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86d9ffb640 0 client.18363 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:32:53.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.294+0000 7f86fa9de740 1 -- 172.21.15.89:0/2677500475 >> 172.21.15.89:0/2677500475 conn(0x562fd3326310 msgr2=0x562fd335f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:53.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86fa9de740 1 -- 172.21.15.89:0/2677500475 shutdown_connections 2024-09-15T23:32:53.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86fa9de740 1 -- 172.21.15.89:0/2677500475 wait complete. 2024-09-15T23:32:53.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86fa9de740 20 client.18363 trim_cache size 0 max 16384 2024-09-15T23:32:53.313 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:32:53.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 -- 172.21.15.89:0/3047219592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86dc0a3240 msgr2=0x7f86dc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:53.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86dc0a3240 0x7f86dc0a5700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009fd0 tx=0x7f86e8016000 comp rx=0 tx=0).stop 2024-09-15T23:32:53.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 -- 172.21.15.89:0/3047219592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f0100480 msgr2=0x7f86f0198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:53.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f0100480 0x7f86f0198430 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f86e4006010 tx=0x7f86e40042f0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 -- 172.21.15.89:0/3047219592 shutdown_connections 2024-09-15T23:32:53.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f86dc0a3240 0x7f86dc0a5700 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f86f01055b0 0x7f86f019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86f0100e30 0x7f86f0198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 --2- 172.21.15.89:0/3047219592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86f0100480 0x7f86f0198430 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:53.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 -- 172.21.15.89:0/3047219592 >> 172.21.15.89:0/3047219592 conn(0x7f86f0077210 msgr2=0x7f86f0103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:53.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 -- 172.21.15.89:0/3047219592 shutdown_connections 2024-09-15T23:32:53.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:53.295+0000 7f86f8605640 1 -- 172.21.15.89:0/3047219592 wait complete. 2024-09-15T23:32:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:53 smithi089 ceph-mon[30933]: pgmap v616: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:32:53.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:53 smithi120 ceph-mon[35250]: pgmap v616: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:32:53.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:53 smithi174 ceph-mon[35586]: pgmap v616: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:32:54.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:32:54.316 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:54.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.818+0000 7f7543476640 1 -- 172.21.15.89:0/187445156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f753c102790 msgr2=0x7f753c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:54.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.818+0000 7f7543476640 1 --2- 172.21.15.89:0/187445156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f753c102790 0x7f753c1067e0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f752c009900 tx=0x7f752c02a410 comp rx=0 tx=0).stop 2024-09-15T23:32:54.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.818+0000 7f7543476640 1 -- 172.21.15.89:0/187445156 shutdown_connections 2024-09-15T23:32:54.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.818+0000 7f7543476640 1 --2- 172.21.15.89:0/187445156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f753c106f10 0x7f753c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.818+0000 7f7543476640 1 --2- 172.21.15.89:0/187445156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f753c102790 0x7f753c1067e0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.818+0000 7f7543476640 1 --2- 172.21.15.89:0/187445156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f753c101de0 0x7f753c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.818+0000 7f7543476640 1 -- 172.21.15.89:0/187445156 >> 172.21.15.89:0/187445156 conn(0x7f753c0fd8c0 msgr2=0x7f753c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:54.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.818+0000 7f7543476640 1 -- 172.21.15.89:0/187445156 shutdown_connections 2024-09-15T23:32:54.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.818+0000 7f7543476640 1 -- 172.21.15.89:0/187445156 wait complete. 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7543476640 1 Processor -- start 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7543476640 1 -- start start 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7543476640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f753c101de0 0x7f753c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7543476640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f753c102790 0x7f753c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7543476640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f753c106f10 0x7f753c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7543476640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f753c111f70 con 0x7f753c101de0 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7543476640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f753c111df0 con 0x7f753c102790 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7543476640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1120f0 con 0x7f753c106f10 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7541212640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f753c101de0 0x7f753c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7541212640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f753c101de0 0x7f753c0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47264/0 (socket says 172.21.15.89:47264) 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7541212640 1 -- 172.21.15.89:0/3635325531 learned_addr learned my addr 172.21.15.89:0/3635325531 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7541a13640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f753c106f10 0x7f753c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.819+0000 7f7540a11640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f753c102790 0x7f753c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f7541212640 1 -- 172.21.15.89:0/3635325531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f753c106f10 msgr2=0x7f753c074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f7541212640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f753c106f10 0x7f753c074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f7541212640 1 -- 172.21.15.89:0/3635325531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f753c102790 msgr2=0x7f753c074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f7541212640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f753c102790 0x7f753c074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f7541212640 1 -- 172.21.15.89:0/3635325531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753c1a6900 con 0x7f753c101de0 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f7541a13640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f753c106f10 0x7f753c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f7541212640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f753c101de0 0x7f753c0744f0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f753800d950 tx=0x7f753800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f75327fc640 1 -- 172.21.15.89:0/3635325531 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7538017070 con 0x7f753c101de0 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f7543476640 1 -- 172.21.15.89:0/3635325531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753c1a6bf0 con 0x7f753c101de0 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f75327fc640 1 -- 172.21.15.89:0/3635325531 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7538009c30 con 0x7f753c101de0 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f75327fc640 1 -- 172.21.15.89:0/3635325531 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753800f6b0 con 0x7f753c101de0 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.820+0000 7f7543476640 1 -- 172.21.15.89:0/3635325531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f753c1a7180 con 0x7f753c101de0 2024-09-15T23:32:54.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.822+0000 7f75327fc640 1 -- 172.21.15.89:0/3635325531 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f75380109d0 con 0x7f753c101de0 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.822+0000 7f754584f740 1 Processor -- start 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f754584f740 1 -- start start 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f754584f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21cce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f75327fc640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75280a32c0 0x7f75280a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f754584f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cce5d0 0x561b21ccea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f754584f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cdb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f754584f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561b21c342f0 con 0x561b21cd9370 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f754584f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561b21bdbcf0 con 0x561b21cce5d0 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f754584f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561b21c302b0 con 0x561b21ccfb40 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541212640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75280a32c0 0x7f75280a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cdb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cdb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47280/0 (socket says 172.21.15.89:47280) 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541a13640 1 -- 172.21.15.89:0/716318553 learned_addr learned my addr 172.21.15.89:0/716318553 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:54.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541a13640 1 --2- 172.21.15.89:0/716318553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21cce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f75327fc640 1 -- 172.21.15.89:0/3635325531 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f75380cfcb0 con 0x7f753c101de0 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7540a11640 1 --2- 172.21.15.89:0/716318553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cce5d0 0x561b21ccea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541a13640 1 -- 172.21.15.89:0/716318553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 msgr2=0x561b21cce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541a13640 1 --2- 172.21.15.89:0/716318553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21cce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541a13640 1 -- 172.21.15.89:0/716318553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cce5d0 msgr2=0x561b21ccea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541a13640 1 --2- 172.21.15.89:0/716318553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cce5d0 0x561b21ccea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541a13640 1 -- 172.21.15.89:0/716318553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b21bc03a0 con 0x561b21cd9370 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541212640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75280a32c0 0x7f75280a5780 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7538098340 tx=0x7f75380913c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.823+0000 7f7541a13640 1 --2- 172.21.15.89:0/716318553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cdb760 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7534012a10 tx=0x7f7534009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ae71f5eabf0dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f75277fe640 1 -- 172.21.15.89:0/716318553 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7534004b20 con 0x561b21cd9370 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f75277fe640 1 -- 172.21.15.89:0/716318553 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7534010d10 con 0x561b21cd9370 2024-09-15T23:32:54.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f75277fe640 1 -- 172.21.15.89:0/716318553 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7534005510 con 0x561b21cd9370 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f754584f740 1 -- 172.21.15.89:0/716318553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 msgr2=0x561b21cdb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f754584f740 1 --2- 172.21.15.89:0/716318553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cdb760 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7534012a10 tx=0x7f7534009750 comp rx=0 tx=0).stop 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f754584f740 1 -- 172.21.15.89:0/716318553 shutdown_connections 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f754584f740 1 --2- 172.21.15.89:0/716318553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cdb760 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f754584f740 1 --2- 172.21.15.89:0/716318553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cce5d0 0x561b21ccea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f754584f740 1 --2- 172.21.15.89:0/716318553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21cce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f754584f740 1 -- 172.21.15.89:0/716318553 >> 172.21.15.89:0/716318553 conn(0x561b21c97310 msgr2=0x561b21ccac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f754584f740 1 -- 172.21.15.89:0/716318553 shutdown_connections 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.824+0000 7f754584f740 1 -- 172.21.15.89:0/716318553 wait complete. 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f754584f740 1 -- start start 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f754584f740 1 Processor -- start 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f754584f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21cd8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f754584f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21cd6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f754584f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cd9370 0x561b21cd65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f754584f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561b21cdc070 con 0x561b21cce5d0 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f754584f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561b21bef060 con 0x561b21cd9370 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f754584f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561b21cdc1f0 con 0x561b21ccfb40 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f754584f740 10 client.0 _ll_register_callbacks cb 0x561b219428c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f7541a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21cd8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f7541a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21cd8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47290/0 (socket says 172.21.15.89:47290) 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.825+0000 7f7541a13640 1 -- 172.21.15.89:0/549365782 learned_addr learned my addr 172.21.15.89:0/549365782 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7540a11640 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21cd6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7541a13640 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cd9370 0x561b21cd65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7541a13640 1 -- 172.21.15.89:0/549365782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 msgr2=0x561b21cd6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7541a13640 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21cd6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7541a13640 1 -- 172.21.15.89:0/549365782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cd9370 msgr2=0x561b21cd65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:54.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7541a13640 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cd9370 0x561b21cd65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7541a13640 1 -- 172.21.15.89:0/549365782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b219a53c0 con 0x561b21cce5d0 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7540a11640 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21cd6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7541a13640 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21cd8d50 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f75340124a0 tx=0x7f7534005130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc3f192f3447cea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7524ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7524ff9640 1 -- 172.21.15.89:0/549365782 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7534004b00 con 0x561b21cce5d0 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7524ff9640 1 -- 172.21.15.89:0/549365782 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7534004ca0 con 0x561b21cce5d0 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f754584f740 1 -- 172.21.15.89:0/549365782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7534010d10 con 0x561b21cce5d0 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7524ff9640 1 -- 172.21.15.89:0/549365782 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7534030050 con 0x561b21cce5d0 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f754584f740 10 client.18375 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f754584f740 1 -- 172.21.15.89:0/549365782 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b21ce7af0 con 0x561b21cce5d0 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7524ff9640 1 -- 172.21.15.89:0/549365782 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7534010d10 con 0x561b21cce5d0 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f7524ff9640 1 -- 172.21.15.89:0/549365782 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f753400c030 con 0x561b21cce5d0 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f754584f740 10 client.18375 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:54.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.826+0000 7f754584f740 10 client.18375 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.827+0000 7f754584f740 20 client.18375 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.827+0000 7f754584f740 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561b21ce8ef0 0x561b21ceb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.827+0000 7f754584f740 4 client.18375 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.827+0000 7f754584f740 1 -- 172.21.15.89:0/549365782 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b21c41860 con 0x561b21ce8ef0 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.827+0000 7f7540a11640 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561b21ce8ef0 0x561b21ceb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.827+0000 7f7524ff9640 10 client.18375 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.827+0000 7f7540a11640 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561b21ce8ef0 0x561b21ceb2e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=90402669063534dc server_cookie=7577db95ca04fc19 in_seq=0 out_seq=0 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f7524ff9640 1 -- 172.21.15.89:0/549365782 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561b21c41860 con 0x561b21ce8ef0 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f7524ff9640 10 client.18375 handle_command_reply: tid=0 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 client.18375 shutdown 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 -- 172.21.15.89:0/549365782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 msgr2=0x561b21cd8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21cd8d50 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f75340124a0 tx=0x7f7534005130 comp rx=0 tx=0).stop 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 -- 172.21.15.89:0/549365782 shutdown_connections 2024-09-15T23:32:54.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cd9370 0x561b21cd65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561b21ce8ef0 0x561b21ceb2e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21cd6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 --2- 172.21.15.89:0/549365782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21cd8d50 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f7524ff9640 0 client.18375 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 -- 172.21.15.89:0/549365782 >> 172.21.15.89:0/549365782 conn(0x561b21c97310 msgr2=0x561b21cd0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 -- 172.21.15.89:0/549365782 shutdown_connections 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 1 -- 172.21.15.89:0/549365782 wait complete. 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.828+0000 7f754584f740 20 client.18375 trim_cache size 0 max 16384 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f754584f740 1 Processor -- start 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f754584f740 1 -- start start 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f754584f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21ce5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f754584f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21ccfb40 0x561b21cd58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f754584f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21cd9370 0x561b21cd4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f754584f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1b1190 con 0x561b21cce5d0 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f754584f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1b2cd0 con 0x561b21ccfb40 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f754584f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1b1860 con 0x561b21cd9370 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f7541a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21ce5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f7541a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21ce5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47296/0 (socket says 172.21.15.89:47296) 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f7541a13640 1 -- 172.21.15.89:0/591729122 learned_addr learned my addr 172.21.15.89:0/591729122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.841+0000 7f7540a11640 1 --2- 172.21.15.89:0/591729122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21ccfb40 0x561b21cd58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f7541a13640 1 -- 172.21.15.89:0/591729122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21cd9370 msgr2=0x561b21cd4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f7541a13640 1 --2- 172.21.15.89:0/591729122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21cd9370 0x561b21cd4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f7541a13640 1 -- 172.21.15.89:0/591729122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21ccfb40 msgr2=0x561b21cd58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f7541a13640 1 --2- 172.21.15.89:0/591729122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21ccfb40 0x561b21cd58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f7541a13640 1 -- 172.21.15.89:0/591729122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b21cd1b30 con 0x561b21cce5d0 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f7541a13640 1 --2- 172.21.15.89:0/591729122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21ce5310 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f7534012fd0 tx=0x7f7534011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efe052a321301b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:54.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f7526ffd640 1 -- 172.21.15.89:0/591729122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753402d020 con 0x561b21cce5d0 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f7526ffd640 1 -- 172.21.15.89:0/591729122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7534016040 con 0x561b21cce5d0 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f7526ffd640 1 -- 172.21.15.89:0/591729122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75340124f0 con 0x561b21cce5d0 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f754584f740 1 -- 172.21.15.89:0/591729122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 msgr2=0x561b21ce5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.842+0000 7f754584f740 1 --2- 172.21.15.89:0/591729122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21ce5310 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f7534012fd0 tx=0x7f7534011a70 comp rx=0 tx=0).stop 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 -- 172.21.15.89:0/591729122 shutdown_connections 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 --2- 172.21.15.89:0/591729122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21cd9370 0x561b21cd4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 --2- 172.21.15.89:0/591729122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21ccfb40 0x561b21cd58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 --2- 172.21.15.89:0/591729122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cce5d0 0x561b21ce5310 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 -- 172.21.15.89:0/591729122 >> 172.21.15.89:0/591729122 conn(0x561b21c97310 msgr2=0x561b21cd48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 -- 172.21.15.89:0/591729122 shutdown_connections 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 -- 172.21.15.89:0/591729122 wait complete. 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 -- start start 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 Processor -- start 2024-09-15T23:32:54.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cce5d0 0x561b21ce1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21ce1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cf00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1b2600 con 0x561b21cd9370 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1b1f30 con 0x561b21cce5d0 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f753c1af170 con 0x561b21ccfb40 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f754584f740 10 client.0 _ll_register_callbacks cb 0x7f752c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.843+0000 7f7541a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cf00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7541a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cf00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47302/0 (socket says 172.21.15.89:47302) 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7541a13640 1 -- 172.21.15.89:0/649655925 learned_addr learned my addr 172.21.15.89:0/649655925 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7540a11640 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21ce1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7541a13640 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cce5d0 0x561b21ce1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7541a13640 1 -- 172.21.15.89:0/649655925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 msgr2=0x561b21ce1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:54.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7541a13640 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21ce1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7541a13640 1 -- 172.21.15.89:0/649655925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cce5d0 msgr2=0x561b21ce1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7541a13640 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cce5d0 0x561b21ce1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7541a13640 1 -- 172.21.15.89:0/649655925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b21cf08d0 con 0x561b21cd9370 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7541a13640 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cf00d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f7534011c50 tx=0x7f7534005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74a42893b870592 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7524ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7524ff9640 1 -- 172.21.15.89:0/649655925 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753402d040 con 0x561b21cd9370 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f754584f740 1 -- 172.21.15.89:0/649655925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7534016040 con 0x561b21cd9370 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7524ff9640 1 -- 172.21.15.89:0/649655925 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7534004b00 con 0x561b21cd9370 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7524ff9640 1 -- 172.21.15.89:0/649655925 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7534004d00 con 0x561b21cd9370 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f754584f740 10 client.18381 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f754584f740 1 -- 172.21.15.89:0/649655925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b21ce11c0 con 0x561b21cd9370 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.844+0000 7f7524ff9640 1 -- 172.21.15.89:0/649655925 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7534016040 con 0x561b21cd9370 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.845+0000 7f7524ff9640 1 -- 172.21.15.89:0/649655925 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7534033070 con 0x561b21cd9370 2024-09-15T23:32:54.861 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.845+0000 7f754584f740 10 client.18381 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.845+0000 7f754584f740 10 client.18381 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.845+0000 7f754584f740 20 client.18381 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.845+0000 7f754584f740 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561b21d53c00 0x561b21d56020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.845+0000 7f754584f740 4 client.18381 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.845+0000 7f754584f740 1 -- 172.21.15.89:0/649655925 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b21c3bd70 con 0x561b21d53c00 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.845+0000 7f7540a11640 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561b21d53c00 0x561b21d56020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.845+0000 7f7540a11640 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561b21d53c00 0x561b21d56020 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=76f72bb8ded410c0 server_cookie=dce98eecdfaa0a99 in_seq=0 out_seq=0 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.845+0000 7f7524ff9640 10 client.18381 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f7524ff9640 1 -- 172.21.15.89:0/649655925 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b21c3bd70 con 0x561b21d53c00 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f7524ff9640 10 client.18381 handle_command_reply: tid=0 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 client.18381 shutdown 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 -- 172.21.15.89:0/649655925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 msgr2=0x561b21cf00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cf00d0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f7534011c50 tx=0x7f7534005f00 comp rx=0 tx=0).stop 2024-09-15T23:32:54.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 -- 172.21.15.89:0/649655925 shutdown_connections 2024-09-15T23:32:54.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b21cd9370 0x561b21cf00d0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561b21d53c00 0x561b21d56020 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b21ccfb40 0x561b21ce1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 --2- 172.21.15.89:0/649655925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b21cce5d0 0x561b21ce1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f7524ff9640 0 client.18381 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:32:54.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 -- 172.21.15.89:0/649655925 >> 172.21.15.89:0/649655925 conn(0x561b21c97310 msgr2=0x561b21cd7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:54.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 -- 172.21.15.89:0/649655925 shutdown_connections 2024-09-15T23:32:54.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 1 -- 172.21.15.89:0/649655925 wait complete. 2024-09-15T23:32:54.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.846+0000 7f754584f740 20 client.18381 trim_cache size 0 max 16384 2024-09-15T23:32:54.863 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:32:54.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 -- 172.21.15.89:0/3635325531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75280a32c0 msgr2=0x7f75280a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:54.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75280a32c0 0x7f75280a5780 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7538098340 tx=0x7f75380913c0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 -- 172.21.15.89:0/3635325531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f753c101de0 msgr2=0x7f753c0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:54.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f753c101de0 0x7f753c0744f0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f753800d950 tx=0x7f753800de10 comp rx=0 tx=0).stop 2024-09-15T23:32:54.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 -- 172.21.15.89:0/3635325531 shutdown_connections 2024-09-15T23:32:54.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f75280a32c0 0x7f75280a5780 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f753c106f10 0x7f753c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f753c102790 0x7f753c074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 --2- 172.21.15.89:0/3635325531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f753c101de0 0x7f753c0744f0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:54.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 -- 172.21.15.89:0/3635325531 >> 172.21.15.89:0/3635325531 conn(0x7f753c0fd8c0 msgr2=0x7f753c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:54.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 -- 172.21.15.89:0/3635325531 shutdown_connections 2024-09-15T23:32:54.865 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:54.847+0000 7f7543476640 1 -- 172.21.15.89:0/3635325531 wait complete. 2024-09-15T23:32:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:55 smithi174 ceph-mon[35586]: pgmap v617: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:32:55.545 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:55 smithi089 ceph-mon[30933]: pgmap v617: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:32:55.573 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:32:55.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:55 smithi120 ceph-mon[35250]: pgmap v617: 129 pgs: 129 active+clean; 4.4 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:32:55.820 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:56.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.320+0000 7f004d8d9640 1 -- 172.21.15.89:0/2996045470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0048106f10 msgr2=0x7f004810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:56.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.320+0000 7f004d8d9640 1 --2- 172.21.15.89:0/2996045470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0048106f10 0x7f004810ea90 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f0040009ca0 tx=0x7f004002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.320+0000 7f004d8d9640 1 -- 172.21.15.89:0/2996045470 shutdown_connections 2024-09-15T23:32:56.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.320+0000 7f004d8d9640 1 --2- 172.21.15.89:0/2996045470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0048106f10 0x7f004810ea90 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.320+0000 7f004d8d9640 1 --2- 172.21.15.89:0/2996045470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048102790 0x7f00481067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.320+0000 7f004d8d9640 1 --2- 172.21.15.89:0/2996045470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0048101de0 0x7f00481021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.320+0000 7f004d8d9640 1 -- 172.21.15.89:0/2996045470 >> 172.21.15.89:0/2996045470 conn(0x7f00480fd8c0 msgr2=0x7f00480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.320+0000 7f004d8d9640 1 -- 172.21.15.89:0/2996045470 shutdown_connections 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.320+0000 7f004d8d9640 1 -- 172.21.15.89:0/2996045470 wait complete. 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f004d8d9640 1 Processor -- start 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f004d8d9640 1 -- start start 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f004d8d9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0048101de0 0x7f0048198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f004d8d9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048102790 0x7f00481987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f004d8d9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0048106f10 0x7f004819cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f004d8d9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0048111f70 con 0x7f0048101de0 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f004d8d9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0048111df0 con 0x7f0048102790 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f004d8d9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f00481120f0 con 0x7f0048106f10 2024-09-15T23:32:56.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f0046ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0048101de0 0x7f0048198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f0046ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0048101de0 0x7f0048198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44814/0 (socket says 172.21.15.89:44814) 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f0046ffd640 1 -- 172.21.15.89:0/818368442 learned_addr learned my addr 172.21.15.89:0/818368442 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f00477fe640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0048106f10 0x7f004819cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f00467fc640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048102790 0x7f00481987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f0046ffd640 1 -- 172.21.15.89:0/818368442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0048106f10 msgr2=0x7f004819cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f0046ffd640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0048106f10 0x7f004819cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f0046ffd640 1 -- 172.21.15.89:0/818368442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048102790 msgr2=0x7f00481987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f0046ffd640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048102790 0x7f00481987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.321+0000 7f0046ffd640 1 -- 172.21.15.89:0/818368442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004819d230 con 0x7f0048101de0 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.322+0000 7f0046ffd640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0048101de0 0x7f0048198280 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f003800c920 tx=0x7f003800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.322+0000 7f004c8d7640 1 -- 172.21.15.89:0/818368442 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038007c70 con 0x7f0048101de0 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.322+0000 7f004d8d9640 1 -- 172.21.15.89:0/818368442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f004819d520 con 0x7f0048101de0 2024-09-15T23:32:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.322+0000 7f004c8d7640 1 -- 172.21.15.89:0/818368442 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0038007e10 con 0x7f0048101de0 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.322+0000 7f004c8d7640 1 -- 172.21.15.89:0/818368442 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038018890 con 0x7f0048101de0 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.322+0000 7f004d8d9640 1 -- 172.21.15.89:0/818368442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00481a4e00 con 0x7f0048101de0 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.324+0000 7f004c8d7640 1 -- 172.21.15.89:0/818368442 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0038018a70 con 0x7f0048101de0 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.324+0000 7f004fcb2740 1 Processor -- start 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.324+0000 7f004fcb2740 1 -- start start 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c95104b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 0x55c95104ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c951058760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f004fcb2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c950fb12f0 con 0x55c951056370 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f004fcb2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c950f58cf0 con 0x55c95104cb40 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f004fcb2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c950fad2b0 con 0x55c95104b5d0 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f004c8d7640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00300a3240 0x7f00300a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c951058760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c951058760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44822/0 (socket says 172.21.15.89:44822) 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00467fc640 1 -- 172.21.15.89:0/2593544133 learned_addr learned my addr 172.21.15.89:0/2593544133 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f0046ffd640 1 --2- 172.21.15.89:0/2593544133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 0x55c95104ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00477fe640 1 --2- 172.21.15.89:0/2593544133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c95104b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f004c8d7640 1 -- 172.21.15.89:0/818368442 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f00380cff60 con 0x7f0048101de0 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00477fe640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00300a3240 0x7f00300a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00467fc640 1 -- 172.21.15.89:0/2593544133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 msgr2=0x55c95104ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00467fc640 1 --2- 172.21.15.89:0/2593544133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 0x55c95104ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00467fc640 1 -- 172.21.15.89:0/2593544133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 msgr2=0x55c95104b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00467fc640 1 --2- 172.21.15.89:0/2593544133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c95104b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00467fc640 1 -- 172.21.15.89:0/2593544133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c950f3d3a0 con 0x55c951056370 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.325+0000 7f00467fc640 1 --2- 172.21.15.89:0/2593544133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c951058760 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f003c00cbb0 tx=0x7f003c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131ca721afcec970 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f002dffb640 1 -- 172.21.15.89:0/2593544133 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c005220 con 0x55c951056370 2024-09-15T23:32:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f00477fe640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00300a3240 0x7f00300a5700 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0040009a60 tx=0x7f004002a8c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f002dffb640 1 -- 172.21.15.89:0/2593544133 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f003c004c80 con 0x55c951056370 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f002dffb640 1 -- 172.21.15.89:0/2593544133 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c010740 con 0x55c951056370 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f004fcb2740 1 -- 172.21.15.89:0/2593544133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 msgr2=0x55c951058760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f004fcb2740 1 --2- 172.21.15.89:0/2593544133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c951058760 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f003c00cbb0 tx=0x7f003c00e720 comp rx=0 tx=0).stop 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f004fcb2740 1 -- 172.21.15.89:0/2593544133 shutdown_connections 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f004fcb2740 1 --2- 172.21.15.89:0/2593544133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c951058760 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f004fcb2740 1 --2- 172.21.15.89:0/2593544133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 0x55c95104ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f004fcb2740 1 --2- 172.21.15.89:0/2593544133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c95104b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f004fcb2740 1 -- 172.21.15.89:0/2593544133 >> 172.21.15.89:0/2593544133 conn(0x55c951014310 msgr2=0x55c951047c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f004fcb2740 1 -- 172.21.15.89:0/2593544133 shutdown_connections 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.326+0000 7f004fcb2740 1 -- 172.21.15.89:0/2593544133 wait complete. 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f004fcb2740 1 -- start start 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f004fcb2740 1 Processor -- start 2024-09-15T23:32:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 0x55c951055cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c951052f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c9510534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f004fcb2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c951059070 con 0x55c951056370 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f004fcb2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c950f6c060 con 0x55c95104cb40 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f004fcb2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c9510591f0 con 0x55c95104b5d0 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f004fcb2740 10 client.0 _ll_register_callbacks cb 0x55c950cbf8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f00477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c9510534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f00477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c9510534e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44830/0 (socket says 172.21.15.89:44830) 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f00477fe640 1 -- 172.21.15.89:0/1998973489 learned_addr learned my addr 172.21.15.89:0/1998973489 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f0046ffd640 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 0x55c951055cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.327+0000 7f00467fc640 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c951052f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f00477fe640 1 -- 172.21.15.89:0/1998973489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 msgr2=0x55c951055cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f00477fe640 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 0x55c951055cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f00477fe640 1 -- 172.21.15.89:0/1998973489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 msgr2=0x55c951052f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f00477fe640 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c951052f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f00477fe640 1 -- 172.21.15.89:0/1998973489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c950d223c0 con 0x55c951056370 2024-09-15T23:32:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f0046ffd640 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 0x55c951055cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f00477fe640 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c9510534e0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f0040035000 tx=0x7f00400095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e0e51c9c407bf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f0026ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f0026ffd640 1 -- 172.21.15.89:0/1998973489 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004004d070 con 0x55c951056370 2024-09-15T23:32:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f0026ffd640 1 -- 172.21.15.89:0/1998973489 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0040004450 con 0x55c951056370 2024-09-15T23:32:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f004fcb2740 1 -- 172.21.15.89:0/1998973489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f003c004c80 con 0x55c951056370 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f0026ffd640 1 -- 172.21.15.89:0/1998973489 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0040043070 con 0x55c951056370 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f004fcb2740 10 client.18393 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f004fcb2740 1 -- 172.21.15.89:0/1998973489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c951064a80 con 0x55c951056370 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.328+0000 7f0026ffd640 1 -- 172.21.15.89:0/1998973489 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f003c004c80 con 0x55c951056370 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.329+0000 7f0026ffd640 1 -- 172.21.15.89:0/1998973489 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f00400305c0 con 0x55c951056370 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.329+0000 7f004fcb2740 10 client.18393 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.329+0000 7f004fcb2740 10 client.18393 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.329+0000 7f004fcb2740 20 client.18393 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.329+0000 7f004fcb2740 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c951065f40 0x55c951068390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.329+0000 7f004fcb2740 4 client.18393 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.329+0000 7f004fcb2740 1 -- 172.21.15.89:0/1998973489 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c950fbe860 con 0x55c951065f40 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.329+0000 7f0046ffd640 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c951065f40 0x55c951068390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.329+0000 7f0026ffd640 10 client.18393 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:32:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.329+0000 7f0046ffd640 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c951065f40 0x55c951068390 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a5fbbd784cbbe422 server_cookie=ff013e21592af1e4 in_seq=0 out_seq=0 2024-09-15T23:32:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f0026ffd640 1 -- 172.21.15.89:0/1998973489 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c950fbe860 con 0x55c951065f40 2024-09-15T23:32:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f0026ffd640 10 client.18393 handle_command_reply: tid=0 2024-09-15T23:32:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f004fcb2740 1 client.18393 shutdown 2024-09-15T23:32:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f004fcb2740 1 -- 172.21.15.89:0/1998973489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 msgr2=0x55c9510534e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f004fcb2740 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c9510534e0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f0040035000 tx=0x7f00400095c0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f004fcb2740 1 -- 172.21.15.89:0/1998973489 shutdown_connections 2024-09-15T23:32:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f004fcb2740 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c951065f40 0x55c951068390 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f004fcb2740 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c951056370 0x55c9510534e0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f004fcb2740 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c951052f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f004fcb2740 1 --2- 172.21.15.89:0/1998973489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c95104b5d0 0x55c951055cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.330+0000 7f0026ffd640 0 client.18393 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:32:56.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.331+0000 7f004fcb2740 1 -- 172.21.15.89:0/1998973489 >> 172.21.15.89:0/1998973489 conn(0x55c951014310 msgr2=0x55c95104d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:56.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.331+0000 7f004fcb2740 1 -- 172.21.15.89:0/1998973489 shutdown_connections 2024-09-15T23:32:56.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.331+0000 7f004fcb2740 1 -- 172.21.15.89:0/1998973489 wait complete. 2024-09-15T23:32:56.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.331+0000 7f004fcb2740 20 client.18393 trim_cache size 0 max 16384 2024-09-15T23:32:56.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.343+0000 7f004fcb2740 1 Processor -- start 2024-09-15T23:32:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.343+0000 7f004fcb2740 1 -- start start 2024-09-15T23:32:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104b5d0 0x55c951062310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104cb40 0x55c95104ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 0x55c951057760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f004fcb2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f00481aed50 con 0x55c95104cb40 2024-09-15T23:32:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f004fcb2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f00481b0890 con 0x55c95104b5d0 2024-09-15T23:32:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f004fcb2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f00481af420 con 0x55c951056370 2024-09-15T23:32:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f00467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104cb40 0x55c95104ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f00467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104cb40 0x55c95104ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44838/0 (socket says 172.21.15.89:44838) 2024-09-15T23:32:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f00467fc640 1 -- 172.21.15.89:0/1415452628 learned_addr learned my addr 172.21.15.89:0/1415452628 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:56.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f0046ffd640 1 --2- 172.21.15.89:0/1415452628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104b5d0 0x55c951062310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f00477fe640 1 --2- 172.21.15.89:0/1415452628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 0x55c951057760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f00467fc640 1 -- 172.21.15.89:0/1415452628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 msgr2=0x55c951057760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:56.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f00467fc640 1 --2- 172.21.15.89:0/1415452628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 0x55c951057760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f00467fc640 1 -- 172.21.15.89:0/1415452628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104b5d0 msgr2=0x55c951062310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:56.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f00467fc640 1 --2- 172.21.15.89:0/1415452628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104b5d0 0x55c951062310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f00467fc640 1 -- 172.21.15.89:0/1415452628 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c951048960 con 0x55c95104cb40 2024-09-15T23:32:56.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f00467fc640 1 --2- 172.21.15.89:0/1415452628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104cb40 0x55c95104ac40 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f003c00cbb0 tx=0x7f003c00c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827fc3dc1a2a82ff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:56.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f002d7fa640 1 -- 172.21.15.89:0/1415452628 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c02a070 con 0x55c95104cb40 2024-09-15T23:32:56.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.344+0000 7f002d7fa640 1 -- 172.21.15.89:0/1415452628 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f003c00c7b0 con 0x55c95104cb40 2024-09-15T23:32:56.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f002d7fa640 1 -- 172.21.15.89:0/1415452628 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c0256a0 con 0x55c95104cb40 2024-09-15T23:32:56.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f004fcb2740 1 -- 172.21.15.89:0/1415452628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104cb40 msgr2=0x55c95104ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:56.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f004fcb2740 1 --2- 172.21.15.89:0/1415452628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104cb40 0x55c95104ac40 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f003c00cbb0 tx=0x7f003c00c390 comp rx=0 tx=0).stop 2024-09-15T23:32:56.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f004fcb2740 1 -- 172.21.15.89:0/1415452628 shutdown_connections 2024-09-15T23:32:56.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f004fcb2740 1 --2- 172.21.15.89:0/1415452628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 0x55c951057760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f004fcb2740 1 --2- 172.21.15.89:0/1415452628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104cb40 0x55c95104ac40 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f004fcb2740 1 --2- 172.21.15.89:0/1415452628 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104b5d0 0x55c951062310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f004fcb2740 1 -- 172.21.15.89:0/1415452628 >> 172.21.15.89:0/1415452628 conn(0x55c951014310 msgr2=0x55c951051850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:56.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f004fcb2740 1 -- 172.21.15.89:0/1415452628 shutdown_connections 2024-09-15T23:32:56.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f004fcb2740 1 -- 172.21.15.89:0/1415452628 wait complete. 2024-09-15T23:32:56.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.345+0000 7f004fcb2740 1 -- start start 2024-09-15T23:32:56.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f004fcb2740 1 Processor -- start 2024-09-15T23:32:56.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104b5d0 0x55c95106d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c95106dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f004fcb2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 0x55c95106e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f004fcb2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f00481b01c0 con 0x55c95104b5d0 2024-09-15T23:32:56.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f004fcb2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f00481afaf0 con 0x55c95104cb40 2024-09-15T23:32:56.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f004fcb2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f00481acd30 con 0x55c951056370 2024-09-15T23:32:56.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f004fcb2740 10 client.0 _ll_register_callbacks cb 0x7f0040009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:56.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f0046ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104b5d0 0x55c95106d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f0046ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104b5d0 0x55c95106d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44848/0 (socket says 172.21.15.89:44848) 2024-09-15T23:32:56.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f0046ffd640 1 -- 172.21.15.89:0/3120877855 learned_addr learned my addr 172.21.15.89:0/3120877855 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:56.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f00467fc640 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c95106dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f00477fe640 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 0x55c95106e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f0046ffd640 1 -- 172.21.15.89:0/3120877855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 msgr2=0x55c95106e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:56.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f0046ffd640 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 0x55c95106e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f0046ffd640 1 -- 172.21.15.89:0/3120877855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 msgr2=0x55c95106dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:56.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f0046ffd640 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c95106dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f0046ffd640 1 -- 172.21.15.89:0/3120877855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c95105cb90 con 0x55c95104b5d0 2024-09-15T23:32:56.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.346+0000 7f00477fe640 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 0x55c95106e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:56.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f0046ffd640 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104b5d0 0x55c95106d960 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f0038098590 tx=0x7f0038015aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314795d2e05cfdd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:56.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f0026ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:56.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f0026ffd640 1 -- 172.21.15.89:0/3120877855 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003809c130 con 0x55c95104b5d0 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f0026ffd640 1 -- 172.21.15.89:0/3120877855 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0038018d60 con 0x55c95104b5d0 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f004fcb2740 1 -- 172.21.15.89:0/3120877855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f003c00c7b0 con 0x55c95104b5d0 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f0026ffd640 1 -- 172.21.15.89:0/3120877855 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f00380a04e0 con 0x55c95104b5d0 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f004fcb2740 10 client.18399 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f004fcb2740 1 -- 172.21.15.89:0/3120877855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c9510d1030 con 0x55c95104b5d0 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f0026ffd640 1 -- 172.21.15.89:0/3120877855 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f003c00c7b0 con 0x55c95104b5d0 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f0026ffd640 1 -- 172.21.15.89:0/3120877855 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f00380bc0b0 con 0x55c95104b5d0 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f004fcb2740 10 client.18399 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f004fcb2740 10 client.18399 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f004fcb2740 20 client.18399 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f004fcb2740 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c9510d1700 0x55c9510d3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f004fcb2740 4 client.18399 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.347+0000 7f004fcb2740 1 -- 172.21.15.89:0/3120877855 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c950fb8d70 con 0x55c9510d1700 2024-09-15T23:32:56.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.348+0000 7f00467fc640 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c9510d1700 0x55c9510d3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.348+0000 7f0026ffd640 10 client.18399 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.348+0000 7f00467fc640 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c9510d1700 0x55c9510d3b20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d2c0d9d79b4ee32 server_cookie=8c5fbdca11cd95ec in_seq=0 out_seq=0 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.348+0000 7f0026ffd640 1 -- 172.21.15.89:0/3120877855 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c950fb8d70 con 0x55c9510d1700 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.348+0000 7f0026ffd640 10 client.18399 handle_command_reply: tid=0 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.348+0000 7f004fcb2740 1 client.18399 shutdown 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.348+0000 7f004fcb2740 1 -- 172.21.15.89:0/3120877855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104b5d0 msgr2=0x55c95106d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.348+0000 7f004fcb2740 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104b5d0 0x55c95106d960 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f0038098590 tx=0x7f0038015aa0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004fcb2740 1 -- 172.21.15.89:0/3120877855 shutdown_connections 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004fcb2740 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c9510d1700 0x55c9510d3b20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004fcb2740 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c951056370 0x55c95106e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004fcb2740 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c95104cb40 0x55c95106dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f0026ffd640 0 client.18399 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:32:56.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004fcb2740 1 --2- 172.21.15.89:0/3120877855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c95104b5d0 0x55c95106d960 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004fcb2740 1 -- 172.21.15.89:0/3120877855 >> 172.21.15.89:0/3120877855 conn(0x55c951014310 msgr2=0x55c951051d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:56.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004fcb2740 1 -- 172.21.15.89:0/3120877855 shutdown_connections 2024-09-15T23:32:56.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004fcb2740 1 -- 172.21.15.89:0/3120877855 wait complete. 2024-09-15T23:32:56.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004fcb2740 20 client.18399 trim_cache size 0 max 16384 2024-09-15T23:32:56.367 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:32:56.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004d8d9640 1 -- 172.21.15.89:0/818368442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00300a3240 msgr2=0x7f00300a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:56.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004d8d9640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00300a3240 0x7f00300a5700 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0040009a60 tx=0x7f004002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004d8d9640 1 -- 172.21.15.89:0/818368442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0048101de0 msgr2=0x7f0048198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:56.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004d8d9640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0048101de0 0x7f0048198280 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f003800c920 tx=0x7f003800cde0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004d8d9640 1 -- 172.21.15.89:0/818368442 shutdown_connections 2024-09-15T23:32:56.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004d8d9640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00300a3240 0x7f00300a5700 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004d8d9640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0048106f10 0x7f004819cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004d8d9640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0048102790 0x7f00481987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004d8d9640 1 --2- 172.21.15.89:0/818368442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0048101de0 0x7f0048198280 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:56.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.349+0000 7f004d8d9640 1 -- 172.21.15.89:0/818368442 >> 172.21.15.89:0/818368442 conn(0x7f00480fd8c0 msgr2=0x7f00480fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:56.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.350+0000 7f004d8d9640 1 -- 172.21.15.89:0/818368442 shutdown_connections 2024-09-15T23:32:56.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:56.350+0000 7f004d8d9640 1 -- 172.21.15.89:0/818368442 wait complete. 2024-09-15T23:32:57.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:32:57.294 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:57 smithi174 ceph-mon[35586]: pgmap v618: 129 pgs: 129 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:32:57.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:57 smithi089 ceph-mon[30933]: pgmap v618: 129 pgs: 129 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:32:57.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:57 smithi120 ceph-mon[35250]: pgmap v618: 129 pgs: 129 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:32:57.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.806+0000 7f1f214c5640 1 -- 172.21.15.89:0/1076706885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f1c106f10 msgr2=0x7f1f1c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:57.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.806+0000 7f1f214c5640 1 --2- 172.21.15.89:0/1076706885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f1c106f10 0x7f1f1c10ea90 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009ca0 tx=0x7f1f1402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.806+0000 7f1f214c5640 1 -- 172.21.15.89:0/1076706885 shutdown_connections 2024-09-15T23:32:57.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.806+0000 7f1f214c5640 1 --2- 172.21.15.89:0/1076706885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f1c106f10 0x7f1f1c10ea90 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.806+0000 7f1f214c5640 1 --2- 172.21.15.89:0/1076706885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f1c102790 0x7f1f1c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.806+0000 7f1f214c5640 1 --2- 172.21.15.89:0/1076706885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f1c101de0 0x7f1f1c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.806+0000 7f1f214c5640 1 -- 172.21.15.89:0/1076706885 >> 172.21.15.89:0/1076706885 conn(0x7f1f1c0fd8c0 msgr2=0x7f1f1c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:57.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.806+0000 7f1f214c5640 1 -- 172.21.15.89:0/1076706885 shutdown_connections 2024-09-15T23:32:57.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.806+0000 7f1f214c5640 1 -- 172.21.15.89:0/1076706885 wait complete. 2024-09-15T23:32:57.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.806+0000 7f1f214c5640 1 Processor -- start 2024-09-15T23:32:57.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f214c5640 1 -- start start 2024-09-15T23:32:57.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f214c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f1c101de0 0x7f1f1c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f214c5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f1c102790 0x7f1f1c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f214c5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f1c106f10 0x7f1f1c19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f214c5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c111f90 con 0x7f1f1c101de0 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f214c5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c111e10 con 0x7f1f1c102790 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f214c5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c112110 con 0x7f1f1c106f10 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f1c101de0 0x7f1f1c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f1c101de0 0x7f1f1c198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44870/0 (socket says 172.21.15.89:44870) 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1affd640 1 -- 172.21.15.89:0/464723317 learned_addr learned my addr 172.21.15.89:0/464723317 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1a7fc640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f1c102790 0x7f1f1c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1b7fe640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f1c106f10 0x7f1f1c19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1affd640 1 -- 172.21.15.89:0/464723317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f1c106f10 msgr2=0x7f1f1c19cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:57.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1affd640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f1c106f10 0x7f1f1c19cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1affd640 1 -- 172.21.15.89:0/464723317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f1c102790 msgr2=0x7f1f1c198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1affd640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f1c102790 0x7f1f1c198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1affd640 1 -- 172.21.15.89:0/464723317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f1c19d3e0 con 0x7f1f1c101de0 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1a7fc640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f1c102790 0x7f1f1c198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.807+0000 7f1f1affd640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f1c101de0 0x7f1f1c198430 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400b4a0 tx=0x7f1f0400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.808+0000 7f1f13fff640 1 -- 172.21.15.89:0/464723317 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04014070 con 0x7f1f1c101de0 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.808+0000 7f1f13fff640 1 -- 172.21.15.89:0/464723317 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f04010af0 con 0x7f1f1c101de0 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.808+0000 7f1f214c5640 1 -- 172.21.15.89:0/464723317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f1c19d6d0 con 0x7f1f1c101de0 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.808+0000 7f1f13fff640 1 -- 172.21.15.89:0/464723317 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04017d10 con 0x7f1f1c101de0 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.808+0000 7f1f214c5640 1 -- 172.21.15.89:0/464723317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f1c1a4fb0 con 0x7f1f1c101de0 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.810+0000 7f1f13fff640 1 -- 172.21.15.89:0/464723317 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1f04017070 con 0x7f1f1c101de0 2024-09-15T23:32:57.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.810+0000 7f1f2389e740 1 Processor -- start 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.810+0000 7f1f2389e740 1 -- start start 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 0x55acfe4ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 0x55acfe4b8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f2389e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55acfe4112f0 con 0x55acfe4ab5d0 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f2389e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55acfe3b8cf0 con 0x55acfe4b6370 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f2389e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55acfe40d2b0 con 0x55acfe4acb40 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f13fff640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f080a3240 0x7f1f080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 0x55acfe4ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4aba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44876/0 (socket says 172.21.15.89:44876) 2024-09-15T23:32:57.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1affd640 1 -- 172.21.15.89:0/960448883 learned_addr learned my addr 172.21.15.89:0/960448883 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1a7fc640 1 --2- 172.21.15.89:0/960448883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 0x55acfe4b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1b7fe640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f080a3240 0x7f1f080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f13fff640 1 -- 172.21.15.89:0/464723317 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1f040cf1f0 con 0x7f1f1c101de0 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1affd640 1 -- 172.21.15.89:0/960448883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 msgr2=0x55acfe4ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1affd640 1 --2- 172.21.15.89:0/960448883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 0x55acfe4ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1affd640 1 -- 172.21.15.89:0/960448883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 msgr2=0x55acfe4b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1affd640 1 --2- 172.21.15.89:0/960448883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 0x55acfe4b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1affd640 1 -- 172.21.15.89:0/960448883 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acfe39d3a0 con 0x55acfe4ab5d0 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1a7fc640 1 --2- 172.21.15.89:0/960448883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 0x55acfe4b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.811+0000 7f1f1affd640 1 --2- 172.21.15.89:0/960448883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4aba10 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1f0409f990 tx=0x7f1f040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d37a709acb5d4c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f1b7fe640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f080a3240 0x7f1f080a5700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009430 tx=0x7f1f1402a8c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f117fa640 1 -- 172.21.15.89:0/960448883 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0401e070 con 0x55acfe4ab5d0 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f117fa640 1 -- 172.21.15.89:0/960448883 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f0409b420 con 0x55acfe4ab5d0 2024-09-15T23:32:57.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f117fa640 1 -- 172.21.15.89:0/960448883 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0409f350 con 0x55acfe4ab5d0 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f2389e740 1 -- 172.21.15.89:0/960448883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 msgr2=0x55acfe4aba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f2389e740 1 --2- 172.21.15.89:0/960448883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4aba10 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1f0409f990 tx=0x7f1f040046c0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f2389e740 1 -- 172.21.15.89:0/960448883 shutdown_connections 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f2389e740 1 --2- 172.21.15.89:0/960448883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 0x55acfe4b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f2389e740 1 --2- 172.21.15.89:0/960448883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4aba10 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f2389e740 1 --2- 172.21.15.89:0/960448883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 0x55acfe4ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.812+0000 7f1f2389e740 1 -- 172.21.15.89:0/960448883 >> 172.21.15.89:0/960448883 conn(0x55acfe474310 msgr2=0x55acfe4a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 1 -- 172.21.15.89:0/960448883 shutdown_connections 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 1 -- 172.21.15.89:0/960448883 wait complete. 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 1 -- start start 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 1 Processor -- start 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4ab5d0 0x55acfe4b5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4acb40 0x55acfe4b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 0x55acfe4b3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55acfe4b9010 con 0x55acfe4acb40 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55acfe3cc060 con 0x55acfe4b6370 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55acfe4b9190 con 0x55acfe4ab5d0 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f2389e740 10 client.0 _ll_register_callbacks cb 0x55acfe11f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f1a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4acb40 0x55acfe4b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f1a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4acb40 0x55acfe4b30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44884/0 (socket says 172.21.15.89:44884) 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f1a7fc640 1 -- 172.21.15.89:0/814790149 learned_addr learned my addr 172.21.15.89:0/814790149 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.813+0000 7f1f1affd640 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4ab5d0 0x55acfe4b5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f1b7fe640 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 0x55acfe4b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f1a7fc640 1 -- 172.21.15.89:0/814790149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4ab5d0 msgr2=0x55acfe4b5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f1a7fc640 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4ab5d0 0x55acfe4b5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f1a7fc640 1 -- 172.21.15.89:0/814790149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 msgr2=0x55acfe4b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:57.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f1a7fc640 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 0x55acfe4b3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f1a7fc640 1 -- 172.21.15.89:0/814790149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acfe1823c0 con 0x55acfe4acb40 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f1b7fe640 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 0x55acfe4b3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f1a7fc640 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4acb40 0x55acfe4b30b0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00bf40 tx=0x7f1f0c011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472cacf2ef6ba6d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f027fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f027fc640 1 -- 172.21.15.89:0/814790149 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c012710 con 0x55acfe4acb40 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f027fc640 1 -- 172.21.15.89:0/814790149 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1f0c0128b0 con 0x55acfe4acb40 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f2389e740 1 -- 172.21.15.89:0/814790149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f0409b420 con 0x55acfe4acb40 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f027fc640 1 -- 172.21.15.89:0/814790149 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1f0c012ab0 con 0x55acfe4acb40 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f2389e740 10 client.18411 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f2389e740 1 -- 172.21.15.89:0/814790149 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55acfe4c4bc0 con 0x55acfe4acb40 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f027fc640 1 -- 172.21.15.89:0/814790149 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f0409b420 con 0x55acfe4acb40 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.814+0000 7f1f027fc640 1 -- 172.21.15.89:0/814790149 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1f0c01a3d0 con 0x55acfe4acb40 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.815+0000 7f1f2389e740 10 client.18411 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:57.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.815+0000 7f1f2389e740 10 client.18411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.815+0000 7f1f2389e740 20 client.18411 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.815+0000 7f1f2389e740 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55acfe4c6120 0x55acfe4c8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.815+0000 7f1f2389e740 4 client.18411 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.815+0000 7f1f2389e740 1 -- 172.21.15.89:0/814790149 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acfe41e860 con 0x55acfe4c6120 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.815+0000 7f1f1affd640 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55acfe4c6120 0x55acfe4c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.815+0000 7f1f027fc640 10 client.18411 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.815+0000 7f1f1affd640 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55acfe4c6120 0x55acfe4c8560 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42d358a57775dd3 server_cookie=c84d28e2623d16ec in_seq=0 out_seq=0 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f027fc640 1 -- 172.21.15.89:0/814790149 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55acfe41e860 con 0x55acfe4c6120 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f027fc640 10 client.18411 handle_command_reply: tid=0 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 client.18411 shutdown 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 -- 172.21.15.89:0/814790149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4acb40 msgr2=0x55acfe4b30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4acb40 0x55acfe4b30b0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00bf40 tx=0x7f1f0c011870 comp rx=0 tx=0).stop 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 -- 172.21.15.89:0/814790149 shutdown_connections 2024-09-15T23:32:57.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55acfe4c6120 0x55acfe4c8560 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4b6370 0x55acfe4b3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f027fc640 0 client.18411 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4acb40 0x55acfe4b30b0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 --2- 172.21.15.89:0/814790149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4ab5d0 0x55acfe4b5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 -- 172.21.15.89:0/814790149 >> 172.21.15.89:0/814790149 conn(0x55acfe474310 msgr2=0x55acfe4ad990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 -- 172.21.15.89:0/814790149 shutdown_connections 2024-09-15T23:32:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.816+0000 7f1f2389e740 1 -- 172.21.15.89:0/814790149 wait complete. 2024-09-15T23:32:57.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.817+0000 7f1f2389e740 20 client.18411 trim_cache size 0 max 16384 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.829+0000 7f1f2389e740 1 Processor -- start 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.829+0000 7f1f2389e740 1 -- start start 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4ac300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4acb40 0x55acfe4cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4b6370 0x55acfe4ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f2389e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1aef20 con 0x55acfe4ab5d0 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f2389e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1b0a60 con 0x55acfe4acb40 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f2389e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1af5f0 con 0x55acfe4b6370 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4ac300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4ac300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44886/0 (socket says 172.21.15.89:44886) 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1affd640 1 -- 172.21.15.89:0/1195668024 learned_addr learned my addr 172.21.15.89:0/1195668024 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:57.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1b7fe640 1 --2- 172.21.15.89:0/1195668024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4b6370 0x55acfe4ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1a7fc640 1 --2- 172.21.15.89:0/1195668024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4acb40 0x55acfe4cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1affd640 1 -- 172.21.15.89:0/1195668024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4b6370 msgr2=0x55acfe4ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1affd640 1 --2- 172.21.15.89:0/1195668024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4b6370 0x55acfe4ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1affd640 1 -- 172.21.15.89:0/1195668024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4acb40 msgr2=0x55acfe4cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1affd640 1 --2- 172.21.15.89:0/1195668024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4acb40 0x55acfe4cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1affd640 1 -- 172.21.15.89:0/1195668024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acfe4c8be0 con 0x55acfe4ab5d0 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f1affd640 1 --2- 172.21.15.89:0/1195668024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4ac300 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f1f0409b3b0 tx=0x7f1f04009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa775713d81123d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f10ff9640 1 -- 172.21.15.89:0/1195668024 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0401e040 con 0x55acfe4ab5d0 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f10ff9640 1 -- 172.21.15.89:0/1195668024 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f04010ca0 con 0x55acfe4ab5d0 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.830+0000 7f1f10ff9640 1 -- 172.21.15.89:0/1195668024 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04017430 con 0x55acfe4ab5d0 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.831+0000 7f1f2389e740 1 -- 172.21.15.89:0/1195668024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 msgr2=0x55acfe4ac300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.831+0000 7f1f2389e740 1 --2- 172.21.15.89:0/1195668024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4ac300 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f1f0409b3b0 tx=0x7f1f04009390 comp rx=0 tx=0).stop 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.831+0000 7f1f2389e740 1 -- 172.21.15.89:0/1195668024 shutdown_connections 2024-09-15T23:32:57.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.831+0000 7f1f2389e740 1 --2- 172.21.15.89:0/1195668024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4b6370 0x55acfe4ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.831+0000 7f1f2389e740 1 --2- 172.21.15.89:0/1195668024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4acb40 0x55acfe4cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.831+0000 7f1f2389e740 1 --2- 172.21.15.89:0/1195668024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4ab5d0 0x55acfe4ac300 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.831+0000 7f1f2389e740 1 -- 172.21.15.89:0/1195668024 >> 172.21.15.89:0/1195668024 conn(0x55acfe474310 msgr2=0x55acfe4a2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.831+0000 7f1f2389e740 1 -- 172.21.15.89:0/1195668024 shutdown_connections 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.831+0000 7f1f2389e740 1 -- 172.21.15.89:0/1195668024 wait complete. 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.831+0000 7f1f2389e740 1 -- start start 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f2389e740 1 Processor -- start 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4ab5d0 0x55acfe4be9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 0x55acfe4afc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f2389e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4b6370 0x55acfe4b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f2389e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1b0390 con 0x55acfe4b6370 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f2389e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1afcc0 con 0x55acfe4ab5d0 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f2389e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1acf00 con 0x55acfe4acb40 2024-09-15T23:32:57.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f2389e740 10 client.0 _ll_register_callbacks cb 0x7f1f0c00b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4b6370 0x55acfe4b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 0x55acfe4afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4b6370 0x55acfe4b01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44894/0 (socket says 172.21.15.89:44894) 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1b7fe640 1 -- 172.21.15.89:0/3725437704 learned_addr learned my addr 172.21.15.89:0/3725437704 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1a7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 0x55acfe4afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:48956/0 (socket says 172.21.15.89:48956) 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1affd640 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4ab5d0 0x55acfe4be9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1b7fe640 1 -- 172.21.15.89:0/3725437704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 msgr2=0x55acfe4afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1b7fe640 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 0x55acfe4afc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1b7fe640 1 -- 172.21.15.89:0/3725437704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4ab5d0 msgr2=0x55acfe4be9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1b7fe640 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4ab5d0 0x55acfe4be9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.832+0000 7f1f1b7fe640 1 -- 172.21.15.89:0/3725437704 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acfe4b0840 con 0x55acfe4b6370 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f1b7fe640 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4b6370 0x55acfe4b01d0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009ca0 tx=0x7f1f140095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c38b9378f390a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:57.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f027fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f027fc640 1 -- 172.21.15.89:0/3725437704 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f140046a0 con 0x55acfe4b6370 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f027fc640 1 -- 172.21.15.89:0/3725437704 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1f1402f610 con 0x55acfe4b6370 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f2389e740 1 -- 172.21.15.89:0/3725437704 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f04010ca0 con 0x55acfe4b6370 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f027fc640 1 -- 172.21.15.89:0/3725437704 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1f14004840 con 0x55acfe4b6370 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f2389e740 10 client.18417 fetch_fsmap learned FSMap version 29 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f2389e740 1 -- 172.21.15.89:0/3725437704 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55acfe530c70 con 0x55acfe4b6370 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f027fc640 1 -- 172.21.15.89:0/3725437704 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f04010ca0 con 0x55acfe4b6370 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f027fc640 1 -- 172.21.15.89:0/3725437704 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1f140305c0 con 0x55acfe4b6370 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f2389e740 10 client.18417 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f2389e740 10 client.18417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f2389e740 20 client.18417 populate_metadata read hostname 'smithi089' 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.833+0000 7f1f2389e740 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55acfe5313f0 0x55acfe533810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.834+0000 7f1f2389e740 4 client.18417 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:32:57.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.834+0000 7f1f2389e740 1 -- 172.21.15.89:0/3725437704 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55acfe418d70 con 0x55acfe5313f0 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.834+0000 7f1f1affd640 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55acfe5313f0 0x55acfe533810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.834+0000 7f1f027fc640 10 client.18417 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.834+0000 7f1f1affd640 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55acfe5313f0 0x55acfe533810 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d49b6c22fd4bb6b server_cookie=45b901b3d35aa680 in_seq=0 out_seq=0 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.834+0000 7f1f027fc640 1 -- 172.21.15.89:0/3725437704 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55acfe418d70 con 0x55acfe5313f0 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.834+0000 7f1f027fc640 10 client.18417 handle_command_reply: tid=0 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.834+0000 7f1f2389e740 1 client.18417 shutdown 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 1 -- 172.21.15.89:0/3725437704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4b6370 msgr2=0x55acfe4b01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4b6370 0x55acfe4b01d0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009ca0 tx=0x7f1f140095c0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 1 -- 172.21.15.89:0/3725437704 shutdown_connections 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55acfe5313f0 0x55acfe533810 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55acfe4b6370 0x55acfe4b01d0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55acfe4acb40 0x55acfe4afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f027fc640 0 client.18417 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:32:57.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 1 --2- 172.21.15.89:0/3725437704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55acfe4ab5d0 0x55acfe4be9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 1 -- 172.21.15.89:0/3725437704 >> 172.21.15.89:0/3725437704 conn(0x55acfe474310 msgr2=0x55acfe4b4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 1 -- 172.21.15.89:0/3725437704 shutdown_connections 2024-09-15T23:32:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 1 -- 172.21.15.89:0/3725437704 wait complete. 2024-09-15T23:32:57.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.835+0000 7f1f2389e740 20 client.18417 trim_cache size 0 max 16384 2024-09-15T23:32:57.850 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 -- 172.21.15.89:0/464723317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f080a3240 msgr2=0x7f1f080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f080a3240 0x7f1f080a5700 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009430 tx=0x7f1f1402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 -- 172.21.15.89:0/464723317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f1c101de0 msgr2=0x7f1f1c198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f1c101de0 0x7f1f1c198430 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400b4a0 tx=0x7f1f0400b960 comp rx=0 tx=0).stop 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 -- 172.21.15.89:0/464723317 shutdown_connections 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1f080a3240 0x7f1f080a5700 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f1c106f10 0x7f1f1c19cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f1c102790 0x7f1f1c198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 --2- 172.21.15.89:0/464723317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f1c101de0 0x7f1f1c198430 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 -- 172.21.15.89:0/464723317 >> 172.21.15.89:0/464723317 conn(0x7f1f1c0fd8c0 msgr2=0x7f1f1c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 -- 172.21.15.89:0/464723317 shutdown_connections 2024-09-15T23:32:57.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:32:57.836+0000 7f1f214c5640 1 -- 172.21.15.89:0/464723317 wait complete. 2024-09-15T23:32:59.534 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:32:59.534 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:32:59.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:32:59 smithi089 ceph-mon[30933]: pgmap v619: 129 pgs: 129 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:32:59.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:32:59 smithi120 ceph-mon[35250]: pgmap v619: 129 pgs: 129 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:32:59.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:32:59 smithi174 ceph-mon[35586]: pgmap v619: 129 pgs: 129 active+clean; 4.5 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:32:59.793 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:00.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.267+0000 7fcd5582d640 1 -- 172.21.15.89:0/3981116350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd50106a80 msgr2=0x7fcd50106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.267+0000 7fcd5582d640 1 --2- 172.21.15.89:0/3981116350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd50106a80 0x7fcd50106e60 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009cd0 tx=0x7fcd4802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.267+0000 7fcd5582d640 1 -- 172.21.15.89:0/3981116350 shutdown_connections 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.267+0000 7fcd5582d640 1 --2- 172.21.15.89:0/3981116350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd50101420 0x7fcd5010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.267+0000 7fcd5582d640 1 --2- 172.21.15.89:0/3981116350 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd50100a80 0x7fcd50100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.267+0000 7fcd5582d640 1 --2- 172.21.15.89:0/3981116350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd50106a80 0x7fcd50106e60 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.267+0000 7fcd5582d640 1 -- 172.21.15.89:0/3981116350 >> 172.21.15.89:0/3981116350 conn(0x7fcd500fc7a0 msgr2=0x7fcd500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.267+0000 7fcd5582d640 1 -- 172.21.15.89:0/3981116350 shutdown_connections 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.267+0000 7fcd5582d640 1 -- 172.21.15.89:0/3981116350 wait complete. 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd5582d640 1 Processor -- start 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd5582d640 1 -- start start 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd5582d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd50100a80 0x7fcd500fce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd5582d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd50101420 0x7fcd500fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd5582d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd50106a80 0x7fcd500fd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd5582d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcd50110ec0 con 0x7fcd50106a80 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd5582d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcd50110d40 con 0x7fcd50100a80 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd5582d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcd50111040 con 0x7fcd50101420 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd4f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd50106a80 0x7fcd500fd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd4f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd50106a80 0x7fcd500fd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44912/0 (socket says 172.21.15.89:44912) 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd4f7fe640 1 -- 172.21.15.89:0/2251480734 learned_addr learned my addr 172.21.15.89:0/2251480734 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd4effd640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd50100a80 0x7fcd500fce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd50101420 0x7fcd500fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd4f7fe640 1 -- 172.21.15.89:0/2251480734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd50101420 msgr2=0x7fcd500fd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd4f7fe640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd50101420 0x7fcd500fd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd4f7fe640 1 -- 172.21.15.89:0/2251480734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd50100a80 msgr2=0x7fcd500fce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd4f7fe640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd50100a80 0x7fcd500fce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.268+0000 7fcd4f7fe640 1 -- 172.21.15.89:0/2251480734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd501a6970 con 0x7fcd50106a80 2024-09-15T23:33:00.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.269+0000 7fcd4f7fe640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd50106a80 0x7fcd500fd8e0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009fd0 tx=0x7fcd4000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.269+0000 7fcd5482b640 1 -- 172.21.15.89:0/2251480734 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40011830 con 0x7fcd50106a80 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.269+0000 7fcd5582d640 1 -- 172.21.15.89:0/2251480734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd501a6c60 con 0x7fcd50106a80 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.269+0000 7fcd5482b640 1 -- 172.21.15.89:0/2251480734 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd40016070 con 0x7fcd50106a80 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.269+0000 7fcd5482b640 1 -- 172.21.15.89:0/2251480734 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4000f2e0 con 0x7fcd50106a80 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.269+0000 7fcd5582d640 1 -- 172.21.15.89:0/2251480734 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd501a71a0 con 0x7fcd50106a80 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.271+0000 7fcd5482b640 1 -- 172.21.15.89:0/2251480734 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcd400119d0 con 0x7fcd50106a80 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.271+0000 7fcd57c06740 1 Processor -- start 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.271+0000 7fcd57c06740 1 -- start start 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.271+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2137b60 0x5565f2137f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.271+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f2142800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.271+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2142d40 0x5565f2145130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd57c06740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5565f209d280 con 0x5565f2138480 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd57c06740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565f2044b10 con 0x5565f2142d40 2024-09-15T23:33:00.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd57c06740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5565f2099240 con 0x5565f2137b60 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd5482b640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcd3c0a3240 0x7fcd3c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f2142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f2142800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44922/0 (socket says 172.21.15.89:44922) 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4effd640 1 -- 172.21.15.89:0/571085964 learned_addr learned my addr 172.21.15.89:0/571085964 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/571085964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2142d40 0x5565f2145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4f7fe640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcd3c0a3240 0x7fcd3c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/571085964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2137b60 0x5565f2137f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd5482b640 1 -- 172.21.15.89:0/2251480734 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcd400cf370 con 0x7fcd50106a80 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4effd640 1 -- 172.21.15.89:0/571085964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2137b60 msgr2=0x5565f2137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4effd640 1 --2- 172.21.15.89:0/571085964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2137b60 0x5565f2137f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4effd640 1 -- 172.21.15.89:0/571085964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2142d40 msgr2=0x5565f2145130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4effd640 1 --2- 172.21.15.89:0/571085964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2142d40 0x5565f2145130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4effd640 1 -- 172.21.15.89:0/571085964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565f20511f0 con 0x5565f2138480 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4effd640 1 --2- 172.21.15.89:0/571085964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f2142800 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009cd0 tx=0x7fcd480314e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f70e6933cdbf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd4f7fe640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcd3c0a3240 0x7fcd3c0a5700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fcd400a3ce0 tx=0x7fcd40085ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd45ffb640 1 -- 172.21.15.89:0/571085964 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48048070 con 0x5565f2138480 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd45ffb640 1 -- 172.21.15.89:0/571085964 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd48036070 con 0x5565f2138480 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.272+0000 7fcd45ffb640 1 -- 172.21.15.89:0/571085964 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48004490 con 0x5565f2138480 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.273+0000 7fcd57c06740 1 -- 172.21.15.89:0/571085964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 msgr2=0x5565f2142800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.273+0000 7fcd57c06740 1 --2- 172.21.15.89:0/571085964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f2142800 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009cd0 tx=0x7fcd480314e0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.273+0000 7fcd57c06740 1 -- 172.21.15.89:0/571085964 shutdown_connections 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.273+0000 7fcd57c06740 1 --2- 172.21.15.89:0/571085964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2142d40 0x5565f2145130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.273+0000 7fcd57c06740 1 --2- 172.21.15.89:0/571085964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f2142800 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.273+0000 7fcd57c06740 1 --2- 172.21.15.89:0/571085964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2137b60 0x5565f2137f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.273+0000 7fcd57c06740 1 -- 172.21.15.89:0/571085964 >> 172.21.15.89:0/571085964 conn(0x5565f21340f0 msgr2=0x5565f21344d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.273+0000 7fcd57c06740 1 -- 172.21.15.89:0/571085964 shutdown_connections 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.273+0000 7fcd57c06740 1 -- 172.21.15.89:0/571085964 wait complete. 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd57c06740 1 -- start start 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd57c06740 1 Processor -- start 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2137b60 0x5565f213fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2138480 0x5565f21400a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2142d40 0x5565f21405e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd57c06740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5565f2145910 con 0x5565f2137b60 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd57c06740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565f2057f10 con 0x5565f2142d40 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd57c06740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5565f2145a90 con 0x5565f2138480 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd57c06740 10 client.0 _ll_register_callbacks cb 0x5565f2020a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd4e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2137b60 0x5565f213fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd4e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2137b60 0x5565f213fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44938/0 (socket says 172.21.15.89:44938) 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd4e7fc640 1 -- 172.21.15.89:0/586752514 learned_addr learned my addr 172.21.15.89:0/586752514 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:00.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd4effd640 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2138480 0x5565f21400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd4e7fc640 1 -- 172.21.15.89:0/586752514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2138480 msgr2=0x5565f21400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2138480 0x5565f21400a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd4e7fc640 1 -- 172.21.15.89:0/586752514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2142d40 msgr2=0x5565f21405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.274+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2142d40 0x5565f21405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd4e7fc640 1 -- 172.21.15.89:0/586752514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565f1e0e6a0 con 0x5565f2137b60 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2137b60 0x5565f213fb60 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fcd3800ce20 tx=0x7fcd3800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac984c0bd303474 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd2effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd2effd640 1 -- 172.21.15.89:0/586752514 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38011d90 con 0x5565f2137b60 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd2effd640 1 -- 172.21.15.89:0/586752514 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcd38005020 con 0x5565f2137b60 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd57c06740 1 -- 172.21.15.89:0/586752514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd48036070 con 0x5565f2137b60 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd2effd640 1 -- 172.21.15.89:0/586752514 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcd38024050 con 0x5565f2137b60 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd57c06740 10 client.18429 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:00.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd57c06740 1 -- 172.21.15.89:0/586752514 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565f2151520 con 0x5565f2137b60 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd2effd640 1 -- 172.21.15.89:0/586752514 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd48036070 con 0x5565f2137b60 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd2effd640 1 -- 172.21.15.89:0/586752514 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcd3800f480 con 0x5565f2137b60 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd57c06740 10 client.18429 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd57c06740 10 client.18429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd57c06740 20 client.18429 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd57c06740 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565f2152930 0x5565f2154d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd57c06740 4 client.18429 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.275+0000 7fcd57c06740 1 -- 172.21.15.89:0/586752514 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565f20aa7f0 con 0x5565f2152930 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.276+0000 7fcd4effd640 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565f2152930 0x5565f2154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.276+0000 7fcd2effd640 10 client.18429 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.276+0000 7fcd4effd640 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565f2152930 0x5565f2154d20 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c8282d0c6ac8b60 server_cookie=a465b2c40fe2ae7a in_seq=0 out_seq=0 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd2effd640 1 -- 172.21.15.89:0/586752514 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5565f20aa7f0 con 0x5565f2152930 2024-09-15T23:33:00.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd2effd640 10 client.18429 handle_command_reply: tid=0 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 client.18429 shutdown 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 -- 172.21.15.89:0/586752514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2137b60 msgr2=0x5565f213fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2137b60 0x5565f213fb60 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fcd3800ce20 tx=0x7fcd3800cf10 comp rx=0 tx=0).stop 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 -- 172.21.15.89:0/586752514 shutdown_connections 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565f2152930 0x5565f2154d20 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2142d40 0x5565f21405e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2138480 0x5565f21400a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 --2- 172.21.15.89:0/586752514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2137b60 0x5565f213fb60 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd2effd640 0 client.18429 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 -- 172.21.15.89:0/586752514 >> 172.21.15.89:0/586752514 conn(0x5565f21340f0 msgr2=0x5565f2133270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 -- 172.21.15.89:0/586752514 shutdown_connections 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 1 -- 172.21.15.89:0/586752514 wait complete. 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.277+0000 7fcd57c06740 20 client.18429 trim_cache size 0 max 16384 2024-09-15T23:33:00.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.294+0000 7fcd57c06740 1 Processor -- start 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.294+0000 7fcd57c06740 1 -- start start 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2137b60 0x5565f2139cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2138480 0x5565f215a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2142d40 0x5565f215aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd57c06740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501b1060 con 0x5565f2142d40 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd57c06740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501b2ba0 con 0x5565f2138480 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd57c06740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501b1730 con 0x5565f2137b60 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd4e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2142d40 0x5565f215aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd4e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2142d40 0x5565f215aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44942/0 (socket says 172.21.15.89:44942) 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd4e7fc640 1 -- 172.21.15.89:0/3889610941 learned_addr learned my addr 172.21.15.89:0/3889610941 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/3889610941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2137b60 0x5565f2139cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd4e7fc640 1 -- 172.21.15.89:0/3889610941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2137b60 msgr2=0x5565f2139cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/3889610941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2137b60 0x5565f2139cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd4e7fc640 1 -- 172.21.15.89:0/3889610941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2138480 msgr2=0x5565f215a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/3889610941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2138480 0x5565f215a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd4e7fc640 1 -- 172.21.15.89:0/3889610941 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565f2133f20 con 0x5565f2142d40 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/3889610941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2142d40 0x5565f215aae0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fcd38011890 tx=0x7fcd3800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155e470b6b9bc365 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd457fa640 1 -- 172.21.15.89:0/3889610941 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3800f680 con 0x5565f2142d40 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd457fa640 1 -- 172.21.15.89:0/3889610941 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd38004730 con 0x5565f2142d40 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.295+0000 7fcd457fa640 1 -- 172.21.15.89:0/3889610941 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38028b70 con 0x5565f2142d40 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 -- 172.21.15.89:0/3889610941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2142d40 msgr2=0x5565f215aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 --2- 172.21.15.89:0/3889610941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2142d40 0x5565f215aae0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fcd38011890 tx=0x7fcd3800cd70 comp rx=0 tx=0).stop 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 -- 172.21.15.89:0/3889610941 shutdown_connections 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 --2- 172.21.15.89:0/3889610941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2142d40 0x5565f215aae0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 --2- 172.21.15.89:0/3889610941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2138480 0x5565f215a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 --2- 172.21.15.89:0/3889610941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2137b60 0x5565f2139cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 -- 172.21.15.89:0/3889610941 >> 172.21.15.89:0/3889610941 conn(0x5565f21340f0 msgr2=0x5565f21395b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 -- 172.21.15.89:0/3889610941 shutdown_connections 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 -- 172.21.15.89:0/3889610941 wait complete. 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 -- start start 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.296+0000 7fcd57c06740 1 Processor -- start 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2137b60 0x5565f214ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f214b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd57c06740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2142d40 0x5565f2145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd57c06740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501b24d0 con 0x5565f2138480 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd57c06740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501b1e00 con 0x5565f2137b60 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd57c06740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcd501af040 con 0x5565f2142d40 2024-09-15T23:33:00.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd57c06740 10 client.0 _ll_register_callbacks cb 0x7fcd3800c7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd4effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f214b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd4effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f214b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44948/0 (socket says 172.21.15.89:44948) 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd4effd640 1 -- 172.21.15.89:0/2095885708 learned_addr learned my addr 172.21.15.89:0/2095885708 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2142d40 0x5565f2145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd4effd640 1 -- 172.21.15.89:0/2095885708 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2142d40 msgr2=0x5565f2145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd4effd640 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2142d40 0x5565f2145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd4effd640 1 -- 172.21.15.89:0/2095885708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2137b60 msgr2=0x5565f214ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd4effd640 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2137b60 0x5565f214ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd4effd640 1 -- 172.21.15.89:0/2095885708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565f212f830 con 0x5565f2138480 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd4effd640 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f214b260 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fcd48046040 tx=0x7fcd48005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379687c818ff5421 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd2effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd2effd640 1 -- 172.21.15.89:0/2095885708 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4805d060 con 0x5565f2138480 2024-09-15T23:33:00.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd57c06740 1 -- 172.21.15.89:0/2095885708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd38004730 con 0x5565f2138480 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd2effd640 1 -- 172.21.15.89:0/2095885708 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcd48031a40 con 0x5565f2138480 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd2effd640 1 -- 172.21.15.89:0/2095885708 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcd48064050 con 0x5565f2138480 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd57c06740 10 client.18435 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.297+0000 7fcd57c06740 1 -- 172.21.15.89:0/2095885708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565f21bdc10 con 0x5565f2138480 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd2effd640 1 -- 172.21.15.89:0/2095885708 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd38004730 con 0x5565f2138480 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd2effd640 1 -- 172.21.15.89:0/2095885708 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcd48042030 con 0x5565f2138480 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd57c06740 10 client.18435 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd57c06740 10 client.18435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd57c06740 20 client.18435 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd57c06740 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565f21bdf50 0x5565f21c0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd57c06740 4 client.18435 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd57c06740 1 -- 172.21.15.89:0/2095885708 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5565f20a4d00 con 0x5565f21bdf50 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565f21bdf50 0x5565f21c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd2effd640 10 client.18435 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:00.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.298+0000 7fcd4e7fc640 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565f21bdf50 0x5565f21c0340 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e5fa8151aabb2a7 server_cookie=77831900a1b23e7a in_seq=0 out_seq=0 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.299+0000 7fcd2effd640 1 -- 172.21.15.89:0/2095885708 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5565f20a4d00 con 0x5565f21bdf50 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.299+0000 7fcd2effd640 10 client.18435 handle_command_reply: tid=0 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.299+0000 7fcd57c06740 1 client.18435 shutdown 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 1 -- 172.21.15.89:0/2095885708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 msgr2=0x5565f214b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f214b260 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fcd48046040 tx=0x7fcd48005bd0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 1 -- 172.21.15.89:0/2095885708 shutdown_connections 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5565f21bdf50 0x5565f21c0340 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5565f2142d40 0x5565f2145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f2138480 0x5565f214b260 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 1 --2- 172.21.15.89:0/2095885708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565f2137b60 0x5565f214ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 1 -- 172.21.15.89:0/2095885708 >> 172.21.15.89:0/2095885708 conn(0x5565f21340f0 msgr2=0x5565f2133270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd2effd640 0 client.18435 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 1 -- 172.21.15.89:0/2095885708 shutdown_connections 2024-09-15T23:33:00.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 1 -- 172.21.15.89:0/2095885708 wait complete. 2024-09-15T23:33:00.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd57c06740 20 client.18435 trim_cache size 0 max 16384 2024-09-15T23:33:00.335 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:33:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:33:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "dcb8a062-6269-46cd-9166-1e1f1e8b866b", 2024-09-15T23:33:00.335 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:33:00.335 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:33:00.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd5582d640 1 -- 172.21.15.89:0/2251480734 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcd3c0a3240 msgr2=0x7fcd3c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd5582d640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcd3c0a3240 0x7fcd3c0a5700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fcd400a3ce0 tx=0x7fcd40085ee0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd5582d640 1 -- 172.21.15.89:0/2251480734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd50106a80 msgr2=0x7fcd500fd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd5582d640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd50106a80 0x7fcd500fd8e0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009fd0 tx=0x7fcd4000b8e0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd5582d640 1 -- 172.21.15.89:0/2251480734 shutdown_connections 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd5582d640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcd3c0a3240 0x7fcd3c0a5700 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd5582d640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd50106a80 0x7fcd500fd8e0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd5582d640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd50101420 0x7fcd500fd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd5582d640 1 --2- 172.21.15.89:0/2251480734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd50100a80 0x7fcd500fce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.300+0000 7fcd5582d640 1 -- 172.21.15.89:0/2251480734 >> 172.21.15.89:0/2251480734 conn(0x7fcd500fc7a0 msgr2=0x7fcd5006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.301+0000 7fcd5582d640 1 -- 172.21.15.89:0/2251480734 shutdown_connections 2024-09-15T23:33:00.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:00.301+0000 7fcd5582d640 1 -- 172.21.15.89:0/2251480734 wait complete. 2024-09-15T23:33:00.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.404+0000 7f5e13cba640 1 -- 172.21.15.89:0/2173933295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e0c0ffe10 msgr2=0x7f5e0c100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.404+0000 7f5e13cba640 1 --2- 172.21.15.89:0/2173933295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e0c0ffe10 0x7f5e0c100210 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009900 tx=0x7f5e0802a410 comp rx=0 tx=0).stop 2024-09-15T23:33:00.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.404+0000 7f5e13cba640 1 -- 172.21.15.89:0/2173933295 shutdown_connections 2024-09-15T23:33:00.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.404+0000 7f5e13cba640 1 --2- 172.21.15.89:0/2173933295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e0c1037a0 0x7f5e0c10a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.404+0000 7f5e13cba640 1 --2- 172.21.15.89:0/2173933295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c102de0 0x7f5e0c103260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.404+0000 7f5e13cba640 1 --2- 172.21.15.89:0/2173933295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e0c0ffe10 0x7f5e0c100210 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.404+0000 7f5e13cba640 1 -- 172.21.15.89:0/2173933295 >> 172.21.15.89:0/2173933295 conn(0x7f5e0c0fb970 msgr2=0x7f5e0c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.404+0000 7f5e13cba640 1 -- 172.21.15.89:0/2173933295 shutdown_connections 2024-09-15T23:33:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.404+0000 7f5e13cba640 1 -- 172.21.15.89:0/2173933295 wait complete. 2024-09-15T23:33:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e13cba640 1 Processor -- start 2024-09-15T23:33:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e13cba640 1 -- start start 2024-09-15T23:33:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e13cba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0ffe10 0x7f5e0c197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e13cba640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e0c102de0 0x7f5e0c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e13cba640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e0c1037a0 0x7f5e0c19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e13cba640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c10d860 con 0x7f5e0c102de0 2024-09-15T23:33:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e13cba640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c10d6e0 con 0x7f5e0c0ffe10 2024-09-15T23:33:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e13cba640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c10d9e0 con 0x7f5e0c1037a0 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e11255640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e0c102de0 0x7f5e0c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e11255640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e0c102de0 0x7f5e0c198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44972/0 (socket says 172.21.15.89:44972) 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e11255640 1 -- 172.21.15.89:0/3085382069 learned_addr learned my addr 172.21.15.89:0/3085382069 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e11a56640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0ffe10 0x7f5e0c197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e11255640 1 -- 172.21.15.89:0/3085382069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e0c1037a0 msgr2=0x7f5e0c19f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e11255640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e0c1037a0 0x7f5e0c19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e11255640 1 -- 172.21.15.89:0/3085382069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0ffe10 msgr2=0x7f5e0c197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e11255640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0ffe10 0x7f5e0c197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e11255640 1 -- 172.21.15.89:0/3085382069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0c19fa50 con 0x7f5e0c102de0 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.405+0000 7f5e11a56640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0ffe10 0x7f5e0c197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.406+0000 7f5e11255640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e0c102de0 0x7f5e0c198370 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00b4a0 tx=0x7f5dfc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.406+0000 7f5e02ffd640 1 -- 172.21.15.89:0/3085382069 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc014070 con 0x7f5e0c102de0 2024-09-15T23:33:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.406+0000 7f5e02ffd640 1 -- 172.21.15.89:0/3085382069 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dfc010af0 con 0x7f5e0c102de0 2024-09-15T23:33:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.406+0000 7f5e13cba640 1 -- 172.21.15.89:0/3085382069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e0c19fd40 con 0x7f5e0c102de0 2024-09-15T23:33:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.406+0000 7f5e02ffd640 1 -- 172.21.15.89:0/3085382069 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc017d10 con 0x7f5e0c102de0 2024-09-15T23:33:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.406+0000 7f5e13cba640 1 -- 172.21.15.89:0/3085382069 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e0c1a02b0 con 0x7f5e0c102de0 2024-09-15T23:33:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.408+0000 7f5e02ffd640 1 -- 172.21.15.89:0/3085382069 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5dfc017070 con 0x7f5e0c102de0 2024-09-15T23:33:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.408+0000 7f5e16a5c740 1 Processor -- start 2024-09-15T23:33:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e16a5c740 1 -- start start 2024-09-15T23:33:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e9822ddb0 0x561e9822e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e982331b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e98233780 0x561e98235c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561e9818d2f0 con 0x561e98233780 2024-09-15T23:33:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561e98135270 con 0x561e9822ddb0 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561e981892b0 con 0x561e9822e6f0 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e02ffd640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5df80a32f0 0x7f5df80a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11255640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e98233780 0x561e98235c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11255640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e98233780 0x561e98235c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44988/0 (socket says 172.21.15.89:44988) 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11255640 1 -- 172.21.15.89:0/101309389 learned_addr learned my addr 172.21.15.89:0/101309389 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e12257640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5df80a32f0 0x7f5df80a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e12257640 1 --2- 172.21.15.89:0/101309389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e9822ddb0 0x561e9822e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e02ffd640 1 -- 172.21.15.89:0/3085382069 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5dfc0cf120 con 0x7f5e0c102de0 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11a56640 1 --2- 172.21.15.89:0/101309389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e982331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11255640 1 -- 172.21.15.89:0/101309389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 msgr2=0x561e982331b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11255640 1 --2- 172.21.15.89:0/101309389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e982331b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11255640 1 -- 172.21.15.89:0/101309389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e9822ddb0 msgr2=0x561e9822e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11255640 1 --2- 172.21.15.89:0/101309389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e9822ddb0 0x561e9822e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11255640 1 -- 172.21.15.89:0/101309389 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e98134340 con 0x561e98233780 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11a56640 1 --2- 172.21.15.89:0/101309389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e982331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e12257640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5df80a32f0 0x7f5df80a57b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5e04005c40 tx=0x7f5e04009f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.409+0000 7f5e11255640 1 --2- 172.21.15.89:0/101309389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e98233780 0x561e98235c40 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0176e0 tx=0x7f5dfc017730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f272b18dc606c7e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.410+0000 7f5df7fff640 1 -- 172.21.15.89:0/101309389 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc01e070 con 0x561e98233780 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.410+0000 7f5df7fff640 1 -- 172.21.15.89:0/101309389 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dfc09b350 con 0x561e98233780 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.410+0000 7f5df7fff640 1 -- 172.21.15.89:0/101309389 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc0979c0 con 0x561e98233780 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.410+0000 7f5e16a5c740 1 -- 172.21.15.89:0/101309389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e98233780 msgr2=0x561e98235c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.410+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/101309389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e98233780 0x561e98235c40 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0176e0 tx=0x7f5dfc017730 comp rx=0 tx=0).stop 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.410+0000 7f5e16a5c740 1 -- 172.21.15.89:0/101309389 shutdown_connections 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.410+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/101309389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e98233780 0x561e98235c40 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.410+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/101309389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e982331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.410+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/101309389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e9822ddb0 0x561e9822e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.410+0000 7f5e16a5c740 1 -- 172.21.15.89:0/101309389 >> 172.21.15.89:0/101309389 conn(0x561e98225010 msgr2=0x561e98225410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 1 -- 172.21.15.89:0/101309389 shutdown_connections 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 1 -- 172.21.15.89:0/101309389 wait complete. 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 1 -- start start 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 1 Processor -- start 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98237f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e98238450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e9823f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561e98236690 con 0x561e9822ddb0 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561e98116290 con 0x561e98233780 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561e98236810 con 0x561e9822e6f0 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e16a5c740 10 client.0 _ll_register_callbacks cb 0x561e98147140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e11a56640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98237f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.421 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e11a56640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98237f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45004/0 (socket says 172.21.15.89:45004) 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.411+0000 7f5e11a56640 1 -- 172.21.15.89:0/2111180490 learned_addr learned my addr 172.21.15.89:0/2111180490 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e12257640 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e9823f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e11255640 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e98238450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e11a56640 1 -- 172.21.15.89:0/2111180490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 msgr2=0x561e98238450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e11a56640 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e98238450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e11a56640 1 -- 172.21.15.89:0/2111180490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 msgr2=0x561e9823f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e11a56640 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e9823f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e11a56640 1 -- 172.21.15.89:0/2111180490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e97ea0210 con 0x561e9822ddb0 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e11255640 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e98238450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e11a56640 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98237f10 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5e08031440 tx=0x7f5e080316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef531f88b734ecc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5df57fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5df57fa640 1 -- 172.21.15.89:0/2111180490 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0803f070 con 0x561e9822ddb0 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5df57fa640 1 -- 172.21.15.89:0/2111180490 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5e08004240 con 0x561e9822ddb0 2024-09-15T23:33:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e16a5c740 1 -- 172.21.15.89:0/2111180490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dfc09b350 con 0x561e9822ddb0 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5df57fa640 1 -- 172.21.15.89:0/2111180490 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e0804b050 con 0x561e9822ddb0 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e16a5c740 10 client.18447 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5e16a5c740 1 -- 172.21.15.89:0/2111180490 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561e97f6c620 con 0x561e9822ddb0 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5df57fa640 1 -- 172.21.15.89:0/2111180490 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dfc09b350 con 0x561e9822ddb0 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.412+0000 7f5df57fa640 1 -- 172.21.15.89:0/2111180490 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5e08004a50 con 0x561e9822ddb0 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.413+0000 7f5e16a5c740 10 client.18447 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.413+0000 7f5e16a5c740 10 client.18447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.413+0000 7f5e16a5c740 20 client.18447 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.413+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e98243b10 0x561e98245f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.413+0000 7f5e16a5c740 4 client.18447 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.413+0000 7f5e16a5c740 1 -- 172.21.15.89:0/2111180490 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e9819aa20 con 0x561e98243b10 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.413+0000 7f5e11255640 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e98243b10 0x561e98245f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.413+0000 7f5df57fa640 10 client.18447 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.413+0000 7f5e11255640 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e98243b10 0x561e98245f80 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2a0c95f498b170 server_cookie=be610918afb73cb3 in_seq=0 out_seq=0 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5df57fa640 1 -- 172.21.15.89:0/2111180490 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561e9819aa20 con 0x561e98243b10 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5df57fa640 10 client.18447 handle_command_reply: tid=0 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 client.18447 shutdown 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 -- 172.21.15.89:0/2111180490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 msgr2=0x561e98237f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98237f10 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f5e08031440 tx=0x7f5e080316b0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 -- 172.21.15.89:0/2111180490 shutdown_connections 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e98243b10 0x561e98245f80 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5df57fa640 0 client.18447 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e9823f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e98238450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/2111180490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98237f10 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 -- 172.21.15.89:0/2111180490 >> 172.21.15.89:0/2111180490 conn(0x561e98225010 msgr2=0x561e98226820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 -- 172.21.15.89:0/2111180490 shutdown_connections 2024-09-15T23:33:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 1 -- 172.21.15.89:0/2111180490 wait complete. 2024-09-15T23:33:00.425 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.414+0000 7f5e16a5c740 20 client.18447 trim_cache size 0 max 16384 2024-09-15T23:33:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e16a5c740 1 Processor -- start 2024-09-15T23:33:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e16a5c740 1 -- start start 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98238330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e9823f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e98226400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1a87d0 con 0x561e9822ddb0 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1aa310 con 0x561e98233780 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1a8ea0 con 0x561e9822e6f0 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11a56640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98238330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11a56640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98238330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45008/0 (socket says 172.21.15.89:45008) 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11a56640 1 -- 172.21.15.89:0/148350049 learned_addr learned my addr 172.21.15.89:0/148350049 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e12257640 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e98226400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11255640 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e9823f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11a56640 1 -- 172.21.15.89:0/148350049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 msgr2=0x561e9823f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11a56640 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e9823f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11a56640 1 -- 172.21.15.89:0/148350049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 msgr2=0x561e98226400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11a56640 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e98226400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11a56640 1 -- 172.21.15.89:0/148350049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e98223650 con 0x561e9822ddb0 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e12257640 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e98226400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11255640 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e9823f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5e11a56640 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98238330 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f5e08033000 tx=0x7f5e08000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3ea8987a648d45 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5df77fe640 1 -- 172.21.15.89:0/148350049 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0803f070 con 0x561e9822ddb0 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5df77fe640 1 -- 172.21.15.89:0/148350049 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e08031de0 con 0x561e9822ddb0 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.429+0000 7f5df77fe640 1 -- 172.21.15.89:0/148350049 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08035070 con 0x561e9822ddb0 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 -- 172.21.15.89:0/148350049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 msgr2=0x561e98238330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98238330 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f5e08033000 tx=0x7f5e08000bc0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 -- 172.21.15.89:0/148350049 shutdown_connections 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e98226400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822e6f0 0x561e9823f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/148350049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822ddb0 0x561e98238330 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 -- 172.21.15.89:0/148350049 >> 172.21.15.89:0/148350049 conn(0x561e98225010 msgr2=0x561e982241c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 -- 172.21.15.89:0/148350049 shutdown_connections 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 -- 172.21.15.89:0/148350049 wait complete. 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 -- start start 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.430+0000 7f5e16a5c740 1 Processor -- start 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822ddb0 0x561e9823d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822e6f0 0x561e9823d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e16a5c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e9823e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1a9c40 con 0x561e9822e6f0 2024-09-15T23:33:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1a9570 con 0x561e98233780 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e16a5c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c1a67b0 con 0x561e9822ddb0 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e16a5c740 10 client.0 _ll_register_callbacks cb 0x7f5e08009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e11255640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822e6f0 0x561e9823d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e11255640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822e6f0 0x561e9823d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45024/0 (socket says 172.21.15.89:45024) 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e11255640 1 -- 172.21.15.89:0/1824750900 learned_addr learned my addr 172.21.15.89:0/1824750900 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e11a56640 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822ddb0 0x561e9823d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e12257640 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e9823e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e11255640 1 -- 172.21.15.89:0/1824750900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822ddb0 msgr2=0x561e9823d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e11255640 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822ddb0 0x561e9823d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e11255640 1 -- 172.21.15.89:0/1824750900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 msgr2=0x561e9823e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e11255640 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e9823e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e11255640 1 -- 172.21.15.89:0/1824750900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e9823ee60 con 0x561e9822e6f0 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e12257640 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e9823e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e11255640 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822e6f0 0x561e9823d890 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc097d20 tx=0x7f5dfc0cf320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8352bca8cc771474 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5df57fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5df57fa640 1 -- 172.21.15.89:0/1824750900 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc01e040 con 0x561e9822e6f0 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5df57fa640 1 -- 172.21.15.89:0/1824750900 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5dfc010ca0 con 0x561e9822e6f0 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.431+0000 7f5e16a5c740 1 -- 172.21.15.89:0/1824750900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e08031de0 con 0x561e9822e6f0 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5df57fa640 1 -- 172.21.15.89:0/1824750900 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5dfc09b350 con 0x561e9822e6f0 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5e16a5c740 10 client.18453 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5e16a5c740 1 -- 172.21.15.89:0/1824750900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561e982af030 con 0x561e9822e6f0 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5df57fa640 1 -- 172.21.15.89:0/1824750900 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e08031de0 con 0x561e9822e6f0 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5df57fa640 1 -- 172.21.15.89:0/1824750900 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5dfc09a8a0 con 0x561e9822e6f0 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5e16a5c740 10 client.18453 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5e16a5c740 10 client.18453 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5e16a5c740 20 client.18453 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e982af4e0 0x561e982b1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5e16a5c740 4 client.18453 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:33:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5e16a5c740 1 -- 172.21.15.89:0/1824750900 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561e98194d70 con 0x561e982af4e0 2024-09-15T23:33:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.432+0000 7f5e11a56640 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e982af4e0 0x561e982b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.433+0000 7f5df57fa640 10 client.18453 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.433+0000 7f5e11a56640 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e982af4e0 0x561e982b1900 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f5e1343550ed67e server_cookie=a2c4daa2c8cb0e9f in_seq=0 out_seq=0 2024-09-15T23:33:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.448+0000 7f5df57fa640 1 -- 172.21.15.89:0/1824750900 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561e98194d70 con 0x561e982af4e0 2024-09-15T23:33:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.448+0000 7f5df57fa640 10 client.18453 handle_command_reply: tid=0 2024-09-15T23:33:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.448+0000 7f5e16a5c740 1 client.18453 shutdown 2024-09-15T23:33:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 1 -- 172.21.15.89:0/1824750900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822e6f0 msgr2=0x561e9823d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822e6f0 0x561e9823d890 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc097d20 tx=0x7f5dfc0cf320 comp rx=0 tx=0).stop 2024-09-15T23:33:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 1 -- 172.21.15.89:0/1824750900 shutdown_connections 2024-09-15T23:33:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e982af4e0 0x561e982b1900 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e98233780 0x561e9823e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e9822e6f0 0x561e9823d890 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 1 --2- 172.21.15.89:0/1824750900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e9822ddb0 0x561e9823d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5df57fa640 0 client.18453 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 1 -- 172.21.15.89:0/1824750900 >> 172.21.15.89:0/1824750900 conn(0x561e98225010 msgr2=0x561e982241c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 1 -- 172.21.15.89:0/1824750900 shutdown_connections 2024-09-15T23:33:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 1 -- 172.21.15.89:0/1824750900 wait complete. 2024-09-15T23:33:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e16a5c740 20 client.18453 trim_cache size 0 max 16384 2024-09-15T23:33:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:33:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:33:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:33:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:33:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e13cba640 1 -- 172.21.15.89:0/3085382069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5df80a32f0 msgr2=0x7f5df80a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e13cba640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5df80a32f0 0x7f5df80a57b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5e04005c40 tx=0x7f5e04009f90 comp rx=0 tx=0).stop 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e13cba640 1 -- 172.21.15.89:0/3085382069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e0c102de0 msgr2=0x7f5e0c198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.449+0000 7f5e13cba640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e0c102de0 0x7f5e0c198370 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00b4a0 tx=0x7f5dfc00b960 comp rx=0 tx=0).stop 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.450+0000 7f5e13cba640 1 -- 172.21.15.89:0/3085382069 shutdown_connections 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.450+0000 7f5e13cba640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5df80a32f0 0x7f5df80a57b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.450+0000 7f5e13cba640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e0c1037a0 0x7f5e0c19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.450+0000 7f5e13cba640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e0c102de0 0x7f5e0c198370 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.450+0000 7f5e13cba640 1 --2- 172.21.15.89:0/3085382069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e0c0ffe10 0x7f5e0c197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.450+0000 7f5e13cba640 1 -- 172.21.15.89:0/3085382069 >> 172.21.15.89:0/3085382069 conn(0x7f5e0c0fb970 msgr2=0x7f5e0c106160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.450+0000 7f5e13cba640 1 -- 172.21.15.89:0/3085382069 shutdown_connections 2024-09-15T23:33:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:00.450+0000 7f5e13cba640 1 -- 172.21.15.89:0/3085382069 wait complete. 2024-09-15T23:33:01.042 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:00 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:33:01.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:33:01.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:00 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:33:01.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:00 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:33:01.366 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:01.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.833+0000 7f8b5eff3640 1 -- 172.21.15.89:0/2986556498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b58100ca0 msgr2=0x7f8b58106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:01.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.833+0000 7f8b5eff3640 1 --2- 172.21.15.89:0/2986556498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b58100ca0 0x7f8b58106840 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009930 tx=0x7f8b4402a420 comp rx=0 tx=0).stop 2024-09-15T23:33:01.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.833+0000 7f8b5eff3640 1 -- 172.21.15.89:0/2986556498 shutdown_connections 2024-09-15T23:33:01.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.833+0000 7f8b5eff3640 1 --2- 172.21.15.89:0/2986556498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b58106f50 0x7f8b5810eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.833+0000 7f8b5eff3640 1 --2- 172.21.15.89:0/2986556498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b58100ca0 0x7f8b58106840 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.833+0000 7f8b5eff3640 1 --2- 172.21.15.89:0/2986556498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b581002f0 0x7f8b581006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.833+0000 7f8b5eff3640 1 -- 172.21.15.89:0/2986556498 >> 172.21.15.89:0/2986556498 conn(0x7f8b58077210 msgr2=0x7f8b580fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:01.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.833+0000 7f8b5eff3640 1 -- 172.21.15.89:0/2986556498 shutdown_connections 2024-09-15T23:33:01.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.833+0000 7f8b5eff3640 1 -- 172.21.15.89:0/2986556498 wait complete. 2024-09-15T23:33:01.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.833+0000 7f8b5eff3640 1 Processor -- start 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b5eff3640 1 -- start start 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b5eff3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b581002f0 0x7f8b58101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b5eff3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b58100ca0 0x7f8b581017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b5eff3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b58106f50 0x7f8b58101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b5eff3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8b58111fa0 con 0x7f8b58100ca0 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b5eff3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b58111e20 con 0x7f8b581002f0 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b5eff3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b58112120 con 0x7f8b58106f50 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b57fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b58100ca0 0x7f8b581017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b57fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b58100ca0 0x7f8b581017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45046/0 (socket says 172.21.15.89:45046) 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b57fff640 1 -- 172.21.15.89:0/1317055082 learned_addr learned my addr 172.21.15.89:0/1317055082 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b5d590640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b58106f50 0x7f8b58101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b581002f0 0x7f8b58101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b57fff640 1 -- 172.21.15.89:0/1317055082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b58106f50 msgr2=0x7f8b58101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:01.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b57fff640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b58106f50 0x7f8b58101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b57fff640 1 -- 172.21.15.89:0/1317055082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b581002f0 msgr2=0x7f8b58101270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b57fff640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b581002f0 0x7f8b58101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b57fff640 1 -- 172.21.15.89:0/1317055082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b581a6990 con 0x7f8b58100ca0 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b57fff640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b58100ca0 0x7f8b581017b0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f8b44005940 tx=0x7f8b440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.834+0000 7f8b55ffb640 1 -- 172.21.15.89:0/1317055082 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4403f030 con 0x7f8b58100ca0 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.835+0000 7f8b5eff3640 1 -- 172.21.15.89:0/1317055082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b581a6c20 con 0x7f8b58100ca0 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.835+0000 7f8b55ffb640 1 -- 172.21.15.89:0/1317055082 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b44003ce0 con 0x7f8b58100ca0 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.835+0000 7f8b55ffb640 1 -- 172.21.15.89:0/1317055082 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b44035070 con 0x7f8b58100ca0 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.835+0000 7f8b5eff3640 1 -- 172.21.15.89:0/1317055082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b581a7100 con 0x7f8b58100ca0 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.837+0000 7f8b55ffb640 1 -- 172.21.15.89:0/1317055082 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8b44042030 con 0x7f8b58100ca0 2024-09-15T23:33:01.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.837+0000 7f8b613cc740 1 Processor -- start 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.837+0000 7f8b613cc740 1 -- start start 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.837+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e65880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.837+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e70140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.837+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e72a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.837+0000 7f8b613cc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560fe2dcb2e0 con 0x560fe2e654a0 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.837+0000 7f8b613cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560fe2d72ce0 con 0x560fe2e70680 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.837+0000 7f8b613cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560fe2dc72a0 con 0x560fe2e65dc0 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5cd8f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e70140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5d590640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e65880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b55ffb640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b480a3240 0x7f8b480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5d590640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e65880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45060/0 (socket says 172.21.15.89:45060) 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5d590640 1 -- 172.21.15.89:0/758427730 learned_addr learned my addr 172.21.15.89:0/758427730 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:01.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5d590640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b480a3240 0x7f8b480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b55ffb640 1 -- 172.21.15.89:0/1317055082 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8b440f0220 con 0x7f8b58100ca0 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b57fff640 1 --2- 172.21.15.89:0/758427730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e72a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/758427730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 msgr2=0x560fe2e72a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/758427730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e72a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/758427730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 msgr2=0x560fe2e65880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/758427730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e65880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/758427730 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fe2d57440 con 0x560fe2e65dc0 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5d590640 1 --2- 172.21.15.89:0/758427730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e65880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b57fff640 1 --2- 172.21.15.89:0/758427730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e72a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/758427730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e70140 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009ab0 tx=0x7f8b4c009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8193d5009abd241a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b5d590640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b480a3240 0x7f8b480a5700 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8b50013af0 tx=0x7f8b50004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b42ffd640 1 -- 172.21.15.89:0/758427730 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c016070 con 0x560fe2e65dc0 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b42ffd640 1 -- 172.21.15.89:0/758427730 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b4c005500 con 0x560fe2e65dc0 2024-09-15T23:33:01.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.838+0000 7f8b42ffd640 1 -- 172.21.15.89:0/758427730 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c011470 con 0x560fe2e65dc0 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.839+0000 7f8b613cc740 1 -- 172.21.15.89:0/758427730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 msgr2=0x560fe2e70140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.839+0000 7f8b613cc740 1 --2- 172.21.15.89:0/758427730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e70140 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009ab0 tx=0x7f8b4c009f70 comp rx=0 tx=0).stop 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.839+0000 7f8b613cc740 1 -- 172.21.15.89:0/758427730 shutdown_connections 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.839+0000 7f8b613cc740 1 --2- 172.21.15.89:0/758427730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e72a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.839+0000 7f8b613cc740 1 --2- 172.21.15.89:0/758427730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e70140 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.839+0000 7f8b613cc740 1 --2- 172.21.15.89:0/758427730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e65880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.839+0000 7f8b613cc740 1 -- 172.21.15.89:0/758427730 >> 172.21.15.89:0/758427730 conn(0x560fe2e61a30 msgr2=0x560fe2e61e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.839+0000 7f8b613cc740 1 -- 172.21.15.89:0/758427730 shutdown_connections 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.839+0000 7f8b613cc740 1 -- 172.21.15.89:0/758427730 wait complete. 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b613cc740 1 -- start start 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b613cc740 1 Processor -- start 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e6fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e6d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e6d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b613cc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560fe2e73250 con 0x560fe2e654a0 2024-09-15T23:33:01.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b613cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560fe2d86050 con 0x560fe2e70680 2024-09-15T23:33:01.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b613cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560fe2e733d0 con 0x560fe2e65dc0 2024-09-15T23:33:01.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b613cc740 10 client.0 _ll_register_callbacks cb 0x560fe2dc6160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:01.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b5cd8f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e6fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b5cd8f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e6fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45066/0 (socket says 172.21.15.89:45066) 2024-09-15T23:33:01.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/3195133048 learned_addr learned my addr 172.21.15.89:0/3195133048 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:01.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b5d590640 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e6d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.840+0000 7f8b57fff640 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e6d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/3195133048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 msgr2=0x560fe2e6d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e6d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/3195133048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 msgr2=0x560fe2e6d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e6d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/3195133048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fe2b3c3c0 con 0x560fe2e654a0 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e6fe90 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c004b10 tx=0x7f8b4c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695b1a1e12b49afc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b37fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b37fff640 1 -- 172.21.15.89:0/3195133048 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c028060 con 0x560fe2e654a0 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b37fff640 1 -- 172.21.15.89:0/3195133048 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8b4c004e60 con 0x560fe2e654a0 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b613cc740 1 -- 172.21.15.89:0/3195133048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b4c005500 con 0x560fe2e654a0 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b37fff640 1 -- 172.21.15.89:0/3195133048 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8b4c02f050 con 0x560fe2e654a0 2024-09-15T23:33:01.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b613cc740 10 client.18465 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b613cc740 1 -- 172.21.15.89:0/3195133048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560fe2e7ea80 con 0x560fe2e654a0 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b37fff640 1 -- 172.21.15.89:0/3195133048 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b4c005500 con 0x560fe2e654a0 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b37fff640 1 -- 172.21.15.89:0/3195133048 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8b4c00f040 con 0x560fe2e654a0 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b613cc740 10 client.18465 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b613cc740 10 client.18465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b613cc740 20 client.18465 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b613cc740 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560fe2e7ff40 0x560fe2e82330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.841+0000 7f8b613cc740 4 client.18465 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.842+0000 7f8b613cc740 1 -- 172.21.15.89:0/3195133048 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fe2dd8850 con 0x560fe2e7ff40 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.842+0000 7f8b57fff640 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560fe2e7ff40 0x560fe2e82330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.842+0000 7f8b57fff640 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560fe2e7ff40 0x560fe2e82330 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6257e984e9e9faeb server_cookie=73c7ea59c4b78b06 in_seq=0 out_seq=0 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.842+0000 7f8b37fff640 10 client.18465 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b37fff640 1 -- 172.21.15.89:0/3195133048 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560fe2dd8850 con 0x560fe2e7ff40 2024-09-15T23:33:01.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b37fff640 10 client.18465 handle_command_reply: tid=0 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 client.18465 shutdown 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 -- 172.21.15.89:0/3195133048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 msgr2=0x560fe2e6fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e6fe90 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c004b10 tx=0x7f8b4c00ef90 comp rx=0 tx=0).stop 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 -- 172.21.15.89:0/3195133048 shutdown_connections 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560fe2e7ff40 0x560fe2e82330 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e6d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e6d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 --2- 172.21.15.89:0/3195133048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e6fe90 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b37fff640 0 client.18465 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 -- 172.21.15.89:0/3195133048 >> 172.21.15.89:0/3195133048 conn(0x560fe2e61a30 msgr2=0x560fe2e67760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 -- 172.21.15.89:0/3195133048 shutdown_connections 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 1 -- 172.21.15.89:0/3195133048 wait complete. 2024-09-15T23:33:01.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.843+0000 7f8b613cc740 20 client.18465 trim_cache size 0 max 16384 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b613cc740 1 Processor -- start 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b613cc740 1 -- start start 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e654a0 0x560fe2e6c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e65dc0 0x560fe2e87960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e70680 0x560fe2e87ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b613cc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8b581b10f0 con 0x560fe2e70680 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b613cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b581b2c30 con 0x560fe2e65dc0 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b613cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b581b17c0 con 0x560fe2e654a0 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5d590640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e70680 0x560fe2e87ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5d590640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e70680 0x560fe2e87ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45068/0 (socket says 172.21.15.89:45068) 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5d590640 1 -- 172.21.15.89:0/1457771863 learned_addr learned my addr 172.21.15.89:0/1457771863 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e654a0 0x560fe2e6c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b57fff640 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e65dc0 0x560fe2e87960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5d590640 1 -- 172.21.15.89:0/1457771863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e654a0 msgr2=0x560fe2e6c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5d590640 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e654a0 0x560fe2e6c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5d590640 1 -- 172.21.15.89:0/1457771863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e65dc0 msgr2=0x560fe2e87960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5d590640 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e65dc0 0x560fe2e87960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5d590640 1 -- 172.21.15.89:0/1457771863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fe2e61860 con 0x560fe2e70680 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b57fff640 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e65dc0 0x560fe2e87960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e654a0 0x560fe2e6c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.860+0000 7f8b5d590640 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e70680 0x560fe2e87ea0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f8b50013920 tx=0x7f8b5000a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245b8296d6476b98 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b427fc640 1 -- 172.21.15.89:0/1457771863 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b50019070 con 0x560fe2e70680 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b427fc640 1 -- 172.21.15.89:0/1457771863 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b50021dd0 con 0x560fe2e70680 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b427fc640 1 -- 172.21.15.89:0/1457771863 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5000f410 con 0x560fe2e70680 2024-09-15T23:33:01.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b613cc740 1 -- 172.21.15.89:0/1457771863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e70680 msgr2=0x560fe2e87ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b613cc740 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e70680 0x560fe2e87ea0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f8b50013920 tx=0x7f8b5000a970 comp rx=0 tx=0).stop 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b613cc740 1 -- 172.21.15.89:0/1457771863 shutdown_connections 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b613cc740 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e70680 0x560fe2e87ea0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b613cc740 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e65dc0 0x560fe2e87960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b613cc740 1 --2- 172.21.15.89:0/1457771863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e654a0 0x560fe2e6c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b613cc740 1 -- 172.21.15.89:0/1457771863 >> 172.21.15.89:0/1457771863 conn(0x560fe2e61a30 msgr2=0x560fe2e60b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b613cc740 1 -- 172.21.15.89:0/1457771863 shutdown_connections 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.861+0000 7f8b613cc740 1 -- 172.21.15.89:0/1457771863 wait complete. 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b613cc740 1 -- start start 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b613cc740 1 Processor -- start 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e78cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e73ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b613cc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e74530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b613cc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8b581b2560 con 0x560fe2e654a0 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b613cc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b581b1e90 con 0x560fe2e70680 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b613cc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b581af0d0 con 0x560fe2e65dc0 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b613cc740 10 client.0 _ll_register_callbacks cb 0x7f8b5000a880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b5cd8f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e78cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b5cd8f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e78cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45084/0 (socket says 172.21.15.89:45084) 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/2066207557 learned_addr learned my addr 172.21.15.89:0/2066207557 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b5d590640 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e74530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.862+0000 7f8b57fff640 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e73ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/2066207557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 msgr2=0x560fe2e73ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e73ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/2066207557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 msgr2=0x560fe2e74530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:01.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e74530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b5cd8f640 1 -- 172.21.15.89:0/2066207557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fe2e74d60 con 0x560fe2e654a0 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b5d590640 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e74530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b5cd8f640 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e78cc0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c000c00 tx=0x7f8b4c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179b10542c2e4fcb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b37fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b37fff640 1 -- 172.21.15.89:0/2066207557 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c038060 con 0x560fe2e654a0 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b613cc740 1 -- 172.21.15.89:0/2066207557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b50021dd0 con 0x560fe2e654a0 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b37fff640 1 -- 172.21.15.89:0/2066207557 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8b4c011680 con 0x560fe2e654a0 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b37fff640 1 -- 172.21.15.89:0/2066207557 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8b4c03f050 con 0x560fe2e654a0 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b613cc740 10 client.18471 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b613cc740 1 -- 172.21.15.89:0/2066207557 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560fe2eea7e0 con 0x560fe2e654a0 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.863+0000 7f8b37fff640 1 -- 172.21.15.89:0/2066207557 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b50021dd0 con 0x560fe2e654a0 2024-09-15T23:33:01.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b37fff640 1 -- 172.21.15.89:0/2066207557 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8b4c005340 con 0x560fe2e654a0 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b613cc740 10 client.18471 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b613cc740 10 client.18471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b613cc740 20 client.18471 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b613cc740 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560fe2eeaed0 0x560fe2eed2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b613cc740 4 client.18471 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b613cc740 1 -- 172.21.15.89:0/2066207557 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560fe2dd2d60 con 0x560fe2eeaed0 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b57fff640 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560fe2eeaed0 0x560fe2eed2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b57fff640 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560fe2eeaed0 0x560fe2eed2f0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb7f18a11c5b453c server_cookie=58762586075bcc2 in_seq=0 out_seq=0 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b37fff640 10 client.18471 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b37fff640 1 -- 172.21.15.89:0/2066207557 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560fe2dd2d60 con 0x560fe2eeaed0 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b37fff640 10 client.18471 handle_command_reply: tid=0 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.864+0000 7f8b613cc740 1 client.18471 shutdown 2024-09-15T23:33:01.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 1 -- 172.21.15.89:0/2066207557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 msgr2=0x560fe2e78cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e78cc0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c000c00 tx=0x7f8b4c00ee30 comp rx=0 tx=0).stop 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 1 -- 172.21.15.89:0/2066207557 shutdown_connections 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560fe2eeaed0 0x560fe2eed2f0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560fe2e70680 0x560fe2e74530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560fe2e65dc0 0x560fe2e73ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 1 --2- 172.21.15.89:0/2066207557 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560fe2e654a0 0x560fe2e78cc0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 1 -- 172.21.15.89:0/2066207557 >> 172.21.15.89:0/2066207557 conn(0x560fe2e61a30 msgr2=0x560fe2e66ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b37fff640 0 client.18471 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 1 -- 172.21.15.89:0/2066207557 shutdown_connections 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 1 -- 172.21.15.89:0/2066207557 wait complete. 2024-09-15T23:33:01.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b613cc740 20 client.18471 trim_cache size 0 max 16384 2024-09-15T23:33:01.884 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:33:01.884 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:33:01.884 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:33:01.884 INFO:teuthology.orchestra.run.smithi089.stdout: "dcb8a062-6269-46cd-9166-1e1f1e8b866b": { 2024-09-15T23:33:01.884 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:33:01.885 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "dcb8a062-6269-46cd-9166-1e1f1e8b866b", 2024-09-15T23:33:01.885 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:33:01.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:01.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:01.885 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:33:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b5eff3640 1 -- 172.21.15.89:0/1317055082 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b480a3240 msgr2=0x7f8b480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b5eff3640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b480a3240 0x7f8b480a5700 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8b50013af0 tx=0x7f8b50004ab0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b5eff3640 1 -- 172.21.15.89:0/1317055082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b58100ca0 msgr2=0x7f8b581017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b5eff3640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b58100ca0 0x7f8b581017b0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f8b44005940 tx=0x7f8b440042f0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b5eff3640 1 -- 172.21.15.89:0/1317055082 shutdown_connections 2024-09-15T23:33:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b5eff3640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b480a3240 0x7f8b480a5700 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b5eff3640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b58106f50 0x7f8b58101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.865+0000 7f8b5eff3640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b58100ca0 0x7f8b581017b0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.866+0000 7f8b5eff3640 1 --2- 172.21.15.89:0/1317055082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b581002f0 0x7f8b58101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:01.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.866+0000 7f8b5eff3640 1 -- 172.21.15.89:0/1317055082 >> 172.21.15.89:0/1317055082 conn(0x7f8b58077210 msgr2=0x7f8b580fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:01.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.866+0000 7f8b5eff3640 1 -- 172.21.15.89:0/1317055082 shutdown_connections 2024-09-15T23:33:01.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:01.866+0000 7f8b5eff3640 1 -- 172.21.15.89:0/1317055082 wait complete. 2024-09-15T23:33:02.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:01 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:33:02.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:01 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:02.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:01 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:33:02.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:01 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:02.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:01 smithi089 ceph-mon[30933]: pgmap v620: 129 pgs: 129 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:33:02.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:33:02.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:33:02.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:01 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:33:02.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:01 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:33:02.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:01 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:02.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:01 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:33:02.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:01 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:02.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:01 smithi120 ceph-mon[35250]: pgmap v620: 129 pgs: 129 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:33:02.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:33:02.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:33:02.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:01 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:33:02.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:01 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:33:02.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:01 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:02.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:01 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:33:02.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:01 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:02.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:01 smithi174 ceph-mon[35586]: pgmap v620: 129 pgs: 129 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:33:02.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:33:02.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:33:02.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:01 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:33:02.461 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.461 INFO:tasks.workunit.client.0.smithi089.stdout:251/600: open fd = 3 2024-09-15T23:33:02.461 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.461 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.461 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:252/600: open fd = 3 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:253/600: open fd = 3 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:254/600: open fd = 3 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.462 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:255/600: open fd = 3 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:256/600: open fd = 3 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:257/600: open fd = 3 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.463 INFO:tasks.workunit.client.0.smithi089.stdout:258/600: open fd = 3 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:259/600: open fd = 3 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:260/600: open fd = 3 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:261/600: open fd = 3 2024-09-15T23:33:02.464 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:262/600: open fd = 3 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:263/600: open fd = 3 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:264/600: open fd = 3 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.465 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:265/600: open fd = 3 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:266/600: open fd = 3 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:267/600: open fd = 3 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.466 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:268/600: open fd = 3 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:269/600: open fd = 3 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:270/600: open fd = 3 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:271/600: open fd = 3 2024-09-15T23:33:02.467 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:272/600: open fd = 3 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:273/600: open fd = 3 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:274/600: open fd = 3 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.468 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.469 INFO:tasks.workunit.client.0.smithi089.stdout:275/600: open fd = 3 2024-09-15T23:33:02.469 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.469 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.469 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.469 INFO:tasks.workunit.client.0.smithi089.stdout:276/600: open fd = 3 2024-09-15T23:33:02.469 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.469 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.469 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.469 INFO:tasks.workunit.client.0.smithi089.stdout:277/600: open fd = 3 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:278/600: open fd = 3 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:279/600: open fd = 3 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.470 INFO:tasks.workunit.client.0.smithi089.stdout:280/600: open fd = 3 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:281/600: open fd = 3 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:282/600: open fd = 3 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.471 INFO:tasks.workunit.client.0.smithi089.stdout:283/600: open fd = 3 2024-09-15T23:33:02.472 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.472 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.472 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.472 INFO:tasks.workunit.client.0.smithi089.stdout:284/600: open fd = 3 2024-09-15T23:33:02.472 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.473 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.473 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.473 INFO:tasks.workunit.client.0.smithi089.stdout:285/600: open fd = 3 2024-09-15T23:33:02.473 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.473 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.473 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.473 INFO:tasks.workunit.client.0.smithi089.stdout:286/600: open fd = 3 2024-09-15T23:33:02.473 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.473 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:287/600: open fd = 3 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:288/600: open fd = 3 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:289/600: open fd = 3 2024-09-15T23:33:02.474 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:290/600: open fd = 3 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:291/600: open fd = 3 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:292/600: open fd = 3 2024-09-15T23:33:02.475 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:293/600: open fd = 3 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:294/600: open fd = 3 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:295/600: open fd = 3 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.476 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:296/600: open fd = 3 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:297/600: open fd = 3 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:298/600: open fd = 3 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.477 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:299/600: open fd = 3 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:300/600: open fd = 3 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:301/600: open fd = 3 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:302/600: open fd = 3 2024-09-15T23:33:02.478 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:303/600: open fd = 3 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:304/600: open fd = 3 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:305/600: open fd = 3 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.479 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:306/600: open fd = 3 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:307/600: open fd = 3 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:308/600: open fd = 3 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:309/600: open fd = 3 2024-09-15T23:33:02.480 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:310/600: open fd = 3 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:311/600: open fd = 3 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:312/600: open fd = 3 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.481 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:33:02.578 INFO:tasks.cephfs.filesystem:scrub status for tag:dcb8a062-6269-46cd-9166-1e1f1e8b866b - {'path': '/', 'tag': 'dcb8a062-6269-46cd-9166-1e1f1e8b866b', 'options': 'recursive,force'} 2024-09-15T23:33:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:03 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:33:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:03 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:33:03.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:03 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:33:04.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:04 smithi120 ceph-mon[35250]: pgmap v621: 129 pgs: 129 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:33:04.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:04 smithi174 ceph-mon[35586]: pgmap v621: 129 pgs: 129 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:33:04.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:04 smithi089 ceph-mon[30933]: pgmap v621: 129 pgs: 129 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:33:05.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:05 smithi089 ceph-mon[30933]: pgmap v622: 129 pgs: 129 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:33:05.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:05 smithi120 ceph-mon[35250]: pgmap v622: 129 pgs: 129 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:33:05.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:05 smithi174 ceph-mon[35586]: pgmap v622: 129 pgs: 129 active+clean; 4.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:33:05.897 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T23:33:05.897 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.35 2024-09-15T23:33:05.897 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:33:06.158 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.643+0000 7f6a86faa640 1 -- 172.21.15.89:0/659853404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a80106a80 msgr2=0x7f6a80106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.643+0000 7f6a86faa640 1 --2- 172.21.15.89:0/659853404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a80106a80 0x7f6a80106e60 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009cd0 tx=0x7f6a7002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:33:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.643+0000 7f6a86faa640 1 -- 172.21.15.89:0/659853404 shutdown_connections 2024-09-15T23:33:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.643+0000 7f6a86faa640 1 --2- 172.21.15.89:0/659853404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a80101420 0x7f6a8010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.643+0000 7f6a86faa640 1 --2- 172.21.15.89:0/659853404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a80100a80 0x7f6a80100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.643+0000 7f6a86faa640 1 --2- 172.21.15.89:0/659853404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a80106a80 0x7f6a80106e60 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:06.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.643+0000 7f6a86faa640 1 -- 172.21.15.89:0/659853404 >> 172.21.15.89:0/659853404 conn(0x7f6a800fc7a0 msgr2=0x7f6a800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:06.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.643+0000 7f6a86faa640 1 -- 172.21.15.89:0/659853404 shutdown_connections 2024-09-15T23:33:06.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.643+0000 7f6a86faa640 1 -- 172.21.15.89:0/659853404 wait complete. 2024-09-15T23:33:06.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.644+0000 7f6a86faa640 1 Processor -- start 2024-09-15T23:33:06.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.644+0000 7f6a86faa640 1 -- start start 2024-09-15T23:33:06.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.644+0000 7f6a86faa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a80100a80 0x7f6a80198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:06.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.644+0000 7f6a86faa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a80101420 0x7f6a80198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:06.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.644+0000 7f6a86faa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a80106a80 0x7f6a8019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:06.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.644+0000 7f6a86faa640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6a80110e40 con 0x7f6a80100a80 2024-09-15T23:33:06.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.644+0000 7f6a86faa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6a80110cc0 con 0x7f6a80106a80 2024-09-15T23:33:06.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.644+0000 7f6a86faa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6a80110fc0 con 0x7f6a80101420 2024-09-15T23:33:06.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a84d46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a80100a80 0x7f6a80198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:06.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a84d46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a80100a80 0x7f6a80198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35358/0 (socket says 172.21.15.89:35358) 2024-09-15T23:33:06.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a84d46640 1 -- 172.21.15.89:0/2427647650 learned_addr learned my addr 172.21.15.89:0/2427647650 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:06.666 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a7ffff640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a80101420 0x7f6a80198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:06.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a84d46640 1 -- 172.21.15.89:0/2427647650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a80101420 msgr2=0x7f6a80198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:06.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a84d46640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a80101420 0x7f6a80198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:06.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a84d46640 1 -- 172.21.15.89:0/2427647650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a80106a80 msgr2=0x7f6a8019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:06.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a84d46640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a80106a80 0x7f6a8019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:06.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a84d46640 1 -- 172.21.15.89:0/2427647650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8019d360 con 0x7f6a80100a80 2024-09-15T23:33:06.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a7ffff640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a80101420 0x7f6a80198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:06.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a84d46640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a80100a80 0x7f6a80198450 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f6a7002ae00 tx=0x7f6a70003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:06.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a7dffb640 1 -- 172.21.15.89:0/2427647650 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a70004040 con 0x7f6a80100a80 2024-09-15T23:33:06.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a86faa640 1 -- 172.21.15.89:0/2427647650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a8019d5f0 con 0x7f6a80100a80 2024-09-15T23:33:06.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a7dffb640 1 -- 172.21.15.89:0/2427647650 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a700046c0 con 0x7f6a80100a80 2024-09-15T23:33:06.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a7dffb640 1 -- 172.21.15.89:0/2427647650 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a70046480 con 0x7f6a80100a80 2024-09-15T23:33:06.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.645+0000 7f6a86faa640 1 -- 172.21.15.89:0/2427647650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a801a4f30 con 0x7f6a80100a80 2024-09-15T23:33:06.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.646+0000 7f6a86faa640 1 -- 172.21.15.89:0/2427647650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a80100ee0 con 0x7f6a80100a80 2024-09-15T23:33:06.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.647+0000 7f6a7dffb640 1 -- 172.21.15.89:0/2427647650 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6a700305a0 con 0x7f6a80100a80 2024-09-15T23:33:06.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.648+0000 7f6a7dffb640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a680a3240 0x7f6a680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:06.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.648+0000 7f6a7ffff640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a680a3240 0x7f6a680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:06.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.648+0000 7f6a7dffb640 1 -- 172.21.15.89:0/2427647650 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6a7002e030 con 0x7f6a80100a80 2024-09-15T23:33:06.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.648+0000 7f6a7ffff640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a680a3240 0x7f6a680a5700 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6a8006b870 tx=0x7f6a74005d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:06.670 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.652+0000 7f6a7dffb640 1 -- 172.21.15.89:0/2427647650 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6a7004b060 con 0x7f6a80100a80 2024-09-15T23:33:06.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.854+0000 7f6a86faa640 1 -- 172.21.15.89:0/2427647650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a801a5160 con 0x7f6a80100a80 2024-09-15T23:33:06.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.854+0000 7f6a7dffb640 1 -- 172.21.15.89:0/2427647650 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f6a70030890 con 0x7f6a80100a80 2024-09-15T23:33:06.856 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:33:06.856 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:33:06.856 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 -- 172.21.15.89:0/2427647650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a680a3240 msgr2=0x7f6a680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a680a3240 0x7f6a680a5700 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6a8006b870 tx=0x7f6a74005d90 comp rx=0 tx=0).stop 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 -- 172.21.15.89:0/2427647650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a80100a80 msgr2=0x7f6a80198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a80100a80 0x7f6a80198450 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f6a7002ae00 tx=0x7f6a70003970 comp rx=0 tx=0).stop 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 -- 172.21.15.89:0/2427647650 shutdown_connections 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6a680a3240 0x7f6a680a5700 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a80106a80 0x7f6a8019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a80101420 0x7f6a80198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 --2- 172.21.15.89:0/2427647650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a80100a80 0x7f6a80198450 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 -- 172.21.15.89:0/2427647650 >> 172.21.15.89:0/2427647650 conn(0x7f6a800fc7a0 msgr2=0x7f6a801047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:06.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 -- 172.21.15.89:0/2427647650 shutdown_connections 2024-09-15T23:33:06.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:06.858+0000 7f6a86faa640 1 -- 172.21.15.89:0/2427647650 wait complete. 2024-09-15T23:33:07.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:06 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2427647650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:07.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:06 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2427647650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:07.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:06 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2427647650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:07.564 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0' 2024-09-15T23:33:07.564 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0 2024-09-15T23:33:07.808 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:08.208 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:07 smithi089 ceph-mon[30933]: pgmap v623: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 22 op/s 2024-09-15T23:33:08.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:07 smithi120 ceph-mon[35250]: pgmap v623: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 22 op/s 2024-09-15T23:33:08.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:07 smithi174 ceph-mon[35586]: pgmap v623: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 22 op/s 2024-09-15T23:33:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.295+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1591502149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf68106a70 msgr2=0x7fdf68106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.295+0000 7fdf6dd89640 1 --2- 172.21.15.89:0/1591502149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf68106a70 0x7fdf68106e50 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009cd0 tx=0x7fdf6002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.295+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1591502149 shutdown_connections 2024-09-15T23:33:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.295+0000 7fdf6dd89640 1 --2- 172.21.15.89:0/1591502149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf68101410 0x7fdf6810d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.295+0000 7fdf6dd89640 1 --2- 172.21.15.89:0/1591502149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf68100a70 0x7fdf68100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.295+0000 7fdf6dd89640 1 --2- 172.21.15.89:0/1591502149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf68106a70 0x7fdf68106e50 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.295+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1591502149 >> 172.21.15.89:0/1591502149 conn(0x7fdf680fc7d0 msgr2=0x7fdf680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.295+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1591502149 shutdown_connections 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.295+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1591502149 wait complete. 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf6dd89640 1 Processor -- start 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf6dd89640 1 -- start start 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf6dd89640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf68100a70 0x7fdf681983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf6dd89640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf68101410 0x7fdf68198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf6dd89640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf68106a70 0x7fdf6819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf6dd89640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdf68110e30 con 0x7fdf68101410 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf6dd89640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf68110cb0 con 0x7fdf68106a70 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf6dd89640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdf68110fb0 con 0x7fdf68100a70 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf66ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf68101410 0x7fdf68198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf66ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf68101410 0x7fdf68198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35368/0 (socket says 172.21.15.89:35368) 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf66ffd640 1 -- 172.21.15.89:0/1135782321 learned_addr learned my addr 172.21.15.89:0/1135782321 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf677fe640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf68100a70 0x7fdf681983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf66ffd640 1 -- 172.21.15.89:0/1135782321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf68100a70 msgr2=0x7fdf681983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf66ffd640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf68100a70 0x7fdf681983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf66ffd640 1 -- 172.21.15.89:0/1135782321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf68106a70 msgr2=0x7fdf6819ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf67fff640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf68106a70 0x7fdf6819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf66ffd640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf68106a70 0x7fdf6819ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.296+0000 7fdf66ffd640 1 -- 172.21.15.89:0/1135782321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6819d3a0 con 0x7fdf68101410 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.297+0000 7fdf66ffd640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf68101410 0x7fdf68198930 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000c920 tx=0x7fdf5000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.297+0000 7fdf64ff9640 1 -- 172.21.15.89:0/1135782321 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50007c70 con 0x7fdf68101410 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.297+0000 7fdf64ff9640 1 -- 172.21.15.89:0/1135782321 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf50007e10 con 0x7fdf68101410 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.297+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1135782321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6819d690 con 0x7fdf68101410 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.297+0000 7fdf64ff9640 1 -- 172.21.15.89:0/1135782321 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50018890 con 0x7fdf68101410 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.297+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1135782321 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf681a4f70 con 0x7fdf68101410 2024-09-15T23:33:08.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.299+0000 7fdf64ff9640 1 -- 172.21.15.89:0/1135782321 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fdf50018a70 con 0x7fdf68101410 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.299+0000 7fdf70162740 1 Processor -- start 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.299+0000 7fdf70162740 1 -- start start 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.299+0000 7fdf70162740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b8850 0x55842a8b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.299+0000 7fdf70162740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b9170 0x55842a8c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf70162740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 0x55842a8c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf70162740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55842a81d160 con 0x55842a8b9170 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf70162740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55842a7c4720 con 0x55842a8c3a30 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf70162740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55842a819120 con 0x55842a8b8850 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b9170 0x55842a8c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b9170 0x55842a8c34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35370/0 (socket says 172.21.15.89:35370) 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf677fe640 1 -- 172.21.15.89:0/46367141 learned_addr learned my addr 172.21.15.89:0/46367141 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf66ffd640 1 --2- 172.21.15.89:0/46367141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 0x55842a8c5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.319 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf67fff640 1 --2- 172.21.15.89:0/46367141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b8850 0x55842a8b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf64ff9640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf540a3240 0x7fdf540a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf677fe640 1 -- 172.21.15.89:0/46367141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b8850 msgr2=0x55842a8b8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf677fe640 1 --2- 172.21.15.89:0/46367141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b8850 0x55842a8b8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf677fe640 1 -- 172.21.15.89:0/46367141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 msgr2=0x55842a8c5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf677fe640 1 --2- 172.21.15.89:0/46367141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 0x55842a8c5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf677fe640 1 -- 172.21.15.89:0/46367141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55842a58eba0 con 0x55842a8b9170 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf67fff640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf540a3240 0x7fdf540a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf64ff9640 1 -- 172.21.15.89:0/1135782321 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdf500cff00 con 0x7fdf68101410 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf677fe640 1 --2- 172.21.15.89:0/46367141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b9170 0x55842a8c34f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fdf60000f80 tx=0x7fdf60009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2761aa405c96a3f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf5dffb640 1 -- 172.21.15.89:0/46367141 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60031ac0 con 0x55842a8b9170 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf5dffb640 1 -- 172.21.15.89:0/46367141 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf60036040 con 0x55842a8b9170 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.300+0000 7fdf5dffb640 1 -- 172.21.15.89:0/46367141 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60004700 con 0x55842a8b9170 2024-09-15T23:33:08.320 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.301+0000 7fdf70162740 1 -- 172.21.15.89:0/46367141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b9170 msgr2=0x55842a8c34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.301+0000 7fdf70162740 1 --2- 172.21.15.89:0/46367141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b9170 0x55842a8c34f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fdf60000f80 tx=0x7fdf60009a20 comp rx=0 tx=0).stop 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.301+0000 7fdf67fff640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf540a3240 0x7fdf540a5700 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800ba90 tx=0x7fdf580046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.301+0000 7fdf70162740 1 -- 172.21.15.89:0/46367141 shutdown_connections 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.301+0000 7fdf70162740 1 --2- 172.21.15.89:0/46367141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 0x55842a8c5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.301+0000 7fdf70162740 1 --2- 172.21.15.89:0/46367141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b9170 0x55842a8c34f0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.301+0000 7fdf70162740 1 --2- 172.21.15.89:0/46367141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b8850 0x55842a8b8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.301+0000 7fdf70162740 1 -- 172.21.15.89:0/46367141 >> 172.21.15.89:0/46367141 conn(0x55842a8b4de0 msgr2=0x55842a8b51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.301+0000 7fdf70162740 1 -- 172.21.15.89:0/46367141 shutdown_connections 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.301+0000 7fdf70162740 1 -- 172.21.15.89:0/46367141 wait complete. 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf70162740 1 -- start start 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf70162740 1 Processor -- start 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf70162740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 0x55842a8c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf70162740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8c0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.321 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf70162740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf70162740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55842a8c6600 con 0x55842a8c3a30 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf70162740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55842a7a5900 con 0x55842a8b8850 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf70162740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55842a8c6780 con 0x55842a8b9170 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf70162740 10 client.0 _ll_register_callbacks cb 0x55842a7a3780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf67fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf67fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35376/0 (socket says 172.21.15.89:35376) 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf67fff640 1 -- 172.21.15.89:0/1491821122 learned_addr learned my addr 172.21.15.89:0/1491821122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf66ffd640 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8c0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.302+0000 7fdf677fe640 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 0x55842a8c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf67fff640 1 -- 172.21.15.89:0/1491821122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 msgr2=0x55842a8c0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf67fff640 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8c0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf67fff640 1 -- 172.21.15.89:0/1491821122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 msgr2=0x55842a8c08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf67fff640 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 0x55842a8c08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf67fff640 1 -- 172.21.15.89:0/1491821122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55842a7e5c50 con 0x55842a8c3a30 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf67fff640 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c1350 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fdf58011720 tx=0x7fdf58005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c11631f62d6217 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf46ffd640 1 -- 172.21.15.89:0/1491821122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5801c8c0 con 0x55842a8c3a30 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf46ffd640 1 -- 172.21.15.89:0/1491821122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdf5801ca60 con 0x55842a8c3a30 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf70162740 1 -- 172.21.15.89:0/1491821122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf60036040 con 0x55842a8c3a30 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf46ffd640 1 -- 172.21.15.89:0/1491821122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdf5802e050 con 0x55842a8c3a30 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf70162740 10 client.18489 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf70162740 1 -- 172.21.15.89:0/1491821122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55842a8d22c0 con 0x55842a8c3a30 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf46ffd640 1 -- 172.21.15.89:0/1491821122 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf60036040 con 0x55842a8c3a30 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf46ffd640 1 -- 172.21.15.89:0/1491821122 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fdf58023020 con 0x55842a8c3a30 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf70162740 10 client.18489 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:08.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf70162740 10 client.18489 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf70162740 20 client.18489 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.303+0000 7fdf70162740 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55842a8d3780 0x55842a8d5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.304+0000 7fdf70162740 4 client.18489 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.304+0000 7fdf70162740 1 -- 172.21.15.89:0/1491821122 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55842a82a890 con 0x55842a8d3780 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.304+0000 7fdf677fe640 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55842a8d3780 0x55842a8d5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.304+0000 7fdf677fe640 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55842a8d3780 0x55842a8d5b90 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36657bf027c2a01e server_cookie=a746cbc794c0b6f1 in_seq=0 out_seq=0 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.304+0000 7fdf46ffd640 10 client.18489 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf46ffd640 1 -- 172.21.15.89:0/1491821122 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55842a82a890 con 0x55842a8d3780 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf46ffd640 10 client.18489 handle_command_reply: tid=0 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 client.18489 shutdown 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 -- 172.21.15.89:0/1491821122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 msgr2=0x55842a8c1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c1350 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fdf58011720 tx=0x7fdf58005d90 comp rx=0 tx=0).stop 2024-09-15T23:33:08.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 -- 172.21.15.89:0/1491821122 shutdown_connections 2024-09-15T23:33:08.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55842a8d3780 0x55842a8d5b90 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf46ffd640 0 client.18489 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:08.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c1350 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8c0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 --2- 172.21.15.89:0/1491821122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 0x55842a8c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 -- 172.21.15.89:0/1491821122 >> 172.21.15.89:0/1491821122 conn(0x55842a8b4de0 msgr2=0x55842a8baac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:08.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 -- 172.21.15.89:0/1491821122 shutdown_connections 2024-09-15T23:33:08.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 1 -- 172.21.15.89:0/1491821122 wait complete. 2024-09-15T23:33:08.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.305+0000 7fdf70162740 20 client.18489 trim_cache size 0 max 16384 2024-09-15T23:33:08.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf70162740 1 Processor -- start 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf70162740 1 -- start start 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf70162740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b8850 0x55842a8db460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf70162740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8db9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf70162740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 0x55842a8bdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf70162740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdf681aeef0 con 0x55842a8b8850 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf70162740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf681b0a30 con 0x55842a8c3a30 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf70162740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdf681af5c0 con 0x55842a8b9170 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b8850 0x55842a8db460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b8850 0x55842a8db460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35382/0 (socket says 172.21.15.89:35382) 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf677fe640 1 -- 172.21.15.89:0/455451528 learned_addr learned my addr 172.21.15.89:0/455451528 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf66ffd640 1 --2- 172.21.15.89:0/455451528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8db9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.323+0000 7fdf677fe640 1 -- 172.21.15.89:0/455451528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 msgr2=0x55842a8db9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.323+0000 7fdf677fe640 1 --2- 172.21.15.89:0/455451528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8db9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.322+0000 7fdf67fff640 1 --2- 172.21.15.89:0/455451528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 0x55842a8bdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.323+0000 7fdf677fe640 1 -- 172.21.15.89:0/455451528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 msgr2=0x55842a8bdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.323+0000 7fdf677fe640 1 --2- 172.21.15.89:0/455451528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 0x55842a8bdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.323+0000 7fdf677fe640 1 -- 172.21.15.89:0/455451528 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55842a8b4c10 con 0x55842a8b8850 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.323+0000 7fdf67fff640 1 --2- 172.21.15.89:0/455451528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 0x55842a8bdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.323+0000 7fdf677fe640 1 --2- 172.21.15.89:0/455451528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b8850 0x55842a8db460 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fdf60000f80 tx=0x7fdf60004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c866987dedb3ec79 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.323+0000 7fdf5d7fa640 1 -- 172.21.15.89:0/455451528 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60031ad0 con 0x55842a8b8850 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.323+0000 7fdf5d7fa640 1 -- 172.21.15.89:0/455451528 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf60036050 con 0x55842a8b8850 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.323+0000 7fdf5d7fa640 1 -- 172.21.15.89:0/455451528 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60003bd0 con 0x55842a8b8850 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 -- 172.21.15.89:0/455451528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b8850 msgr2=0x55842a8db460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 --2- 172.21.15.89:0/455451528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b8850 0x55842a8db460 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fdf60000f80 tx=0x7fdf60004320 comp rx=0 tx=0).stop 2024-09-15T23:33:08.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 -- 172.21.15.89:0/455451528 shutdown_connections 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 --2- 172.21.15.89:0/455451528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8c3a30 0x55842a8bdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 --2- 172.21.15.89:0/455451528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8db9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 --2- 172.21.15.89:0/455451528 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8b8850 0x55842a8db460 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 -- 172.21.15.89:0/455451528 >> 172.21.15.89:0/455451528 conn(0x55842a8b4de0 msgr2=0x55842a8bf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 -- 172.21.15.89:0/455451528 shutdown_connections 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 -- 172.21.15.89:0/455451528 wait complete. 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 -- start start 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 Processor -- start 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 0x55842a8cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdf681b0360 con 0x55842a8c3a30 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf681afc90 con 0x55842a8b8850 2024-09-15T23:33:08.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.324+0000 7fdf70162740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdf681aced0 con 0x55842a8b9170 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf70162740 10 client.0 _ll_register_callbacks cb 0x7fdf5800fe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf67fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf67fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35384/0 (socket says 172.21.15.89:35384) 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf67fff640 1 -- 172.21.15.89:0/3727582818 learned_addr learned my addr 172.21.15.89:0/3727582818 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf66ffd640 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf677fe640 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 0x55842a8cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf67fff640 1 -- 172.21.15.89:0/3727582818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 msgr2=0x55842a8cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf67fff640 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf67fff640 1 -- 172.21.15.89:0/3727582818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 msgr2=0x55842a8cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf67fff640 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 0x55842a8cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf67fff640 1 -- 172.21.15.89:0/3727582818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55842a8b03f0 con 0x55842a8c3a30 2024-09-15T23:33:08.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf677fe640 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 0x55842a8cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf67fff640 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c7ee0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fdf5801f040 tx=0x7fdf5801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c689e399f367e18 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf46ffd640 1 -- 172.21.15.89:0/3727582818 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf58036060 con 0x55842a8c3a30 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf46ffd640 1 -- 172.21.15.89:0/3727582818 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdf58005220 con 0x55842a8c3a30 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf70162740 1 -- 172.21.15.89:0/3727582818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf60036050 con 0x55842a8c3a30 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf46ffd640 1 -- 172.21.15.89:0/3727582818 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdf58005420 con 0x55842a8c3a30 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf70162740 10 client.18495 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.325+0000 7fdf70162740 1 -- 172.21.15.89:0/3727582818 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55842a93ec30 con 0x55842a8c3a30 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf46ffd640 1 -- 172.21.15.89:0/3727582818 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf60036050 con 0x55842a8c3a30 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf46ffd640 1 -- 172.21.15.89:0/3727582818 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fdf58023070 con 0x55842a8c3a30 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf70162740 10 client.18495 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf70162740 10 client.18495 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf70162740 20 client.18495 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:08.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf70162740 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55842a93f410 0x55842a941850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:08.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf70162740 4 client.18495 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}] 2024-09-15T23:33:08.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf70162740 1 -- 172.21.15.89:0/3727582818 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}) -- 0x55842a824be0 con 0x55842a93f410 2024-09-15T23:33:08.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf677fe640 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55842a93f410 0x55842a941850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:08.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf46ffd640 10 client.18495 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:08.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.326+0000 7fdf677fe640 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55842a93f410 0x55842a941850 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca80ceb37304fa98 server_cookie=d8cd1b6c2055e1bd in_seq=0 out_seq=0 2024-09-15T23:33:08.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf46ffd640 1 -- 172.21.15.89:0/3727582818 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55842a824be0 con 0x55842a93f410 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf46ffd640 10 client.18495 handle_command_reply: tid=0 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 client.18495 shutdown 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 -- 172.21.15.89:0/3727582818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 msgr2=0x55842a8c7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c7ee0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fdf5801f040 tx=0x7fdf5801c8d0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 -- 172.21.15.89:0/3727582818 shutdown_connections 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55842a93f410 0x55842a941850 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55842a8c3a30 0x55842a8c7ee0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55842a8b9170 0x55842a8cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 --2- 172.21.15.89:0/3727582818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55842a8b8850 0x55842a8cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 -- 172.21.15.89:0/3727582818 >> 172.21.15.89:0/3727582818 conn(0x55842a8b4de0 msgr2=0x55842a8c0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf46ffd640 0 client.18495 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:08.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 -- 172.21.15.89:0/3727582818 shutdown_connections 2024-09-15T23:33:08.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 1 -- 172.21.15.89:0/3727582818 wait complete. 2024-09-15T23:33:08.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.342+0000 7fdf70162740 20 client.18495 trim_cache size 0 max 16384 2024-09-15T23:33:08.345 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:33:08.345 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:33:08.345 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:33:08.345 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 30, 2024-09-15T23:33:08.345 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:33:08.345 INFO:teuthology.orchestra.run.smithi089.stdout: "34a03dc2": { 2024-09-15T23:33:08.345 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 30, 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 142.3, 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:33:08.346 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:08.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:08.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:08.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:08.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:08.347 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:33:08.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1135782321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf540a3240 msgr2=0x7fdf540a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:08.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf540a3240 0x7fdf540a5700 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800ba90 tx=0x7fdf580046c0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1135782321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf68101410 msgr2=0x7fdf68198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:08.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf68101410 0x7fdf68198930 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000c920 tx=0x7fdf5000cde0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1135782321 shutdown_connections 2024-09-15T23:33:08.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf540a3240 0x7fdf540a5700 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf68106a70 0x7fdf6819ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf68101410 0x7fdf68198930 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 --2- 172.21.15.89:0/1135782321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf68100a70 0x7fdf681983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:08.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1135782321 >> 172.21.15.89:0/1135782321 conn(0x7fdf680fc7d0 msgr2=0x7fdf6806aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:08.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1135782321 shutdown_connections 2024-09-15T23:33:08.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:08.343+0000 7fdf6dd89640 1 -- 172.21.15.89:0/1135782321 wait complete. 2024-09-15T23:33:09.101 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '34a03dc2', quiesce took 3.2 seconds. Will release after: 19.150000000000002 2024-09-15T23:33:09.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:09 smithi089 ceph-mon[30933]: pgmap v624: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:33:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:09 smithi120 ceph-mon[35250]: pgmap v624: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:33:09.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:09 smithi174 ceph-mon[35586]: pgmap v624: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:33:12.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:12 smithi174 ceph-mon[35586]: pgmap v625: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 20 op/s 2024-09-15T23:33:12.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:12 smithi089 ceph-mon[30933]: pgmap v625: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 20 op/s 2024-09-15T23:33:12.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:12 smithi120 ceph-mon[35250]: pgmap v625: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 20 op/s 2024-09-15T23:33:13.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 -- 172.21.15.89:0/640451755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97ec107840 msgr2=0x7f97ec109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:13.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 --2- 172.21.15.89:0/640451755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97ec107840 0x7f97ec109cb0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009ca0 tx=0x7f97e402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 -- 172.21.15.89:0/640451755 shutdown_connections 2024-09-15T23:33:13.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 --2- 172.21.15.89:0/640451755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97ec107840 0x7f97ec109cb0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 --2- 172.21.15.89:0/640451755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97ec104f10 0x7f97ec107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.550 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 --2- 172.21.15.89:0/640451755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec1025e0 0x7f97ec1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 -- 172.21.15.89:0/640451755 >> 172.21.15.89:0/640451755 conn(0x7f97ec0fc1e0 msgr2=0x7f97ec0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 -- 172.21.15.89:0/640451755 shutdown_connections 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 -- 172.21.15.89:0/640451755 wait complete. 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 Processor -- start 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.546+0000 7f97f2427640 1 -- start start 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97f2427640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97ec1025e0 0x7f97ec197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97f2427640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec104f10 0x7f97ec198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97f2427640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97ec107840 0x7f97ec19f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97f2427640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec10d800 con 0x7f97ec1025e0 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97f2427640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec10d680 con 0x7f97ec104f10 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97f2427640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec10d980 con 0x7f97ec107840 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97ec1025e0 0x7f97ec197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97ec1025e0 0x7f97ec197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35394/0 (socket says 172.21.15.89:35394) 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97ebfff640 1 -- 172.21.15.89:0/377825791 learned_addr learned my addr 172.21.15.89:0/377825791 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97f09c4640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97ec107840 0x7f97ec19f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97eb7fe640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec104f10 0x7f97ec198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97ebfff640 1 -- 172.21.15.89:0/377825791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97ec107840 msgr2=0x7f97ec19f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97ebfff640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97ec107840 0x7f97ec19f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97ebfff640 1 -- 172.21.15.89:0/377825791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec104f10 msgr2=0x7f97ec198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97ebfff640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec104f10 0x7f97ec198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97ebfff640 1 -- 172.21.15.89:0/377825791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec19fa20 con 0x7f97ec1025e0 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97eb7fe640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec104f10 0x7f97ec198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.547+0000 7f97ebfff640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97ec1025e0 0x7f97ec197e00 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00d950 tx=0x7f97dc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.548+0000 7f97e97fa640 1 -- 172.21.15.89:0/377825791 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc017070 con 0x7f97ec1025e0 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.548+0000 7f97f2427640 1 -- 172.21.15.89:0/377825791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ec19fd10 con 0x7f97ec1025e0 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.548+0000 7f97e97fa640 1 -- 172.21.15.89:0/377825791 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97dc009c30 con 0x7f97ec1025e0 2024-09-15T23:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.548+0000 7f97e97fa640 1 -- 172.21.15.89:0/377825791 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc00f6b0 con 0x7f97ec1025e0 2024-09-15T23:33:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.548+0000 7f97f2427640 1 -- 172.21.15.89:0/377825791 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ec1a01b0 con 0x7f97ec1025e0 2024-09-15T23:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.550+0000 7f97f51c3740 1 Processor -- start 2024-09-15T23:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.550+0000 7f97e97fa640 1 -- 172.21.15.89:0/377825791 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f97dc0109d0 con 0x7f97ec1025e0 2024-09-15T23:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.550+0000 7f97f51c3740 1 -- start start 2024-09-15T23:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.550+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87a9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.550+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.550+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87af370 0x5569c87b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.550+0000 7f97f51c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5569c870d2f0 con 0x5569c87aa2e0 2024-09-15T23:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.550+0000 7f97f51c3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5569c86b5270 con 0x5569c87a99a0 2024-09-15T23:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.550+0000 7f97f51c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5569c87092b0 con 0x5569c87af370 2024-09-15T23:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35402/0 (socket says 172.21.15.89:35402) 2024-09-15T23:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97ebfff640 1 -- 172.21.15.89:0/2781884775 learned_addr learned my addr 172.21.15.89:0/2781884775 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97eb7fe640 1 --2- 172.21.15.89:0/2781884775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87af370 0x5569c87b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97f09c4640 1 --2- 172.21.15.89:0/2781884775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87a9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97e97fa640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97d40a32f0 0x7f97d40a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97ebfff640 1 -- 172.21.15.89:0/2781884775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87af370 msgr2=0x5569c87b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97ebfff640 1 --2- 172.21.15.89:0/2781884775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87af370 0x5569c87b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97ebfff640 1 -- 172.21.15.89:0/2781884775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 msgr2=0x5569c87a9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97ebfff640 1 --2- 172.21.15.89:0/2781884775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87a9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97ebfff640 1 -- 172.21.15.89:0/2781884775 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569c86b4340 con 0x5569c87aa2e0 2024-09-15T23:33:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97eb7fe640 1 --2- 172.21.15.89:0/2781884775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87af370 0x5569c87b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97f09c4640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97d40a32f0 0x7f97d40a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97e97fa640 1 -- 172.21.15.89:0/377825791 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f97dc0cf970 con 0x7f97ec1025e0 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97ebfff640 1 --2- 172.21.15.89:0/2781884775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87aeda0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f97dc06e450 tx=0x7f97dc06b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9b66e696971271 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97d27fc640 1 -- 172.21.15.89:0/2781884775 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc09b6e0 con 0x5569c87aa2e0 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97d27fc640 1 -- 172.21.15.89:0/2781884775 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97dc06ef20 con 0x5569c87aa2e0 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97d27fc640 1 -- 172.21.15.89:0/2781884775 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97dc06ea90 con 0x5569c87aa2e0 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.551+0000 7f97f09c4640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97d40a32f0 0x7f97d40a57b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f97e4006010 tx=0x7f97e403f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.552+0000 7f97f51c3740 1 -- 172.21.15.89:0/2781884775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 msgr2=0x5569c87aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.552+0000 7f97f51c3740 1 --2- 172.21.15.89:0/2781884775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87aeda0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f97dc06e450 tx=0x7f97dc06b810 comp rx=0 tx=0).stop 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.552+0000 7f97f51c3740 1 -- 172.21.15.89:0/2781884775 shutdown_connections 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.552+0000 7f97f51c3740 1 --2- 172.21.15.89:0/2781884775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87af370 0x5569c87b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.552+0000 7f97f51c3740 1 --2- 172.21.15.89:0/2781884775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87aeda0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.552+0000 7f97f51c3740 1 --2- 172.21.15.89:0/2781884775 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87a9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.552+0000 7f97f51c3740 1 -- 172.21.15.89:0/2781884775 >> 172.21.15.89:0/2781884775 conn(0x5569c87a5010 msgr2=0x5569c87a5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.552+0000 7f97f51c3740 1 -- 172.21.15.89:0/2781884775 shutdown_connections 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.552+0000 7f97f51c3740 1 -- 172.21.15.89:0/2781884775 wait complete. 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97f51c3740 1 -- start start 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97f51c3740 1 Processor -- start 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87a99a0 0x5569c87b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87b4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87af370 0x5569c87bb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97f51c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5569c87b22e0 con 0x5569c87aa2e0 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97f51c3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5569c8696290 con 0x5569c87af370 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97f51c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5569c87b2460 con 0x5569c87a99a0 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97f51c3740 10 client.0 _ll_register_callbacks cb 0x5569c86c7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87b4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35408/0 (socket says 172.21.15.89:35408) 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97eb7fe640 1 -- 172.21.15.89:0/359229560 learned_addr learned my addr 172.21.15.89:0/359229560 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97f09c4640 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87af370 0x5569c87bb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.553+0000 7f97ebfff640 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87a99a0 0x5569c87b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97eb7fe640 1 -- 172.21.15.89:0/359229560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87a99a0 msgr2=0x5569c87b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97eb7fe640 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87a99a0 0x5569c87b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97eb7fe640 1 -- 172.21.15.89:0/359229560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87af370 msgr2=0x5569c87bb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97eb7fe640 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87af370 0x5569c87bb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97eb7fe640 1 -- 172.21.15.89:0/359229560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569c8420210 con 0x5569c87aa2e0 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97eb7fe640 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87b4080 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f97e000bf00 tx=0x7f97e00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e58a93987b62 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97cb7fe640 1 -- 172.21.15.89:0/359229560 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0012650 con 0x5569c87aa2e0 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97cb7fe640 1 -- 172.21.15.89:0/359229560 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f97e00127f0 con 0x5569c87aa2e0 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97f51c3740 1 -- 172.21.15.89:0/359229560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5569c871aa20 con 0x5569c87aa2e0 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97cb7fe640 1 -- 172.21.15.89:0/359229560 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f97e00129f0 con 0x5569c87aa2e0 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97f51c3740 10 client.18507 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97f51c3740 1 -- 172.21.15.89:0/359229560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5569c84ec620 con 0x5569c87aa2e0 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97cb7fe640 1 -- 172.21.15.89:0/359229560 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5569c871aa20 con 0x5569c87aa2e0 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.554+0000 7f97cb7fe640 1 -- 172.21.15.89:0/359229560 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f97e001a3d0 con 0x5569c87aa2e0 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.555+0000 7f97f51c3740 10 client.18507 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.555+0000 7f97f51c3740 10 client.18507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.555+0000 7f97f51c3740 20 client.18507 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.555+0000 7f97f51c3740 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5569c87bf8d0 0x5569c87c1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.555+0000 7f97f51c3740 4 client.18507 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.555+0000 7f97f51c3740 1 -- 172.21.15.89:0/359229560 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569c8714d70 con 0x5569c87bf8d0 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.555+0000 7f97ebfff640 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5569c87bf8d0 0x5569c87c1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.555+0000 7f97cb7fe640 10 client.18507 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.555+0000 7f97ebfff640 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5569c87bf8d0 0x5569c87c1d30 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=625040937c3fd23a server_cookie=d100be074b782b8b in_seq=0 out_seq=0 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97cb7fe640 1 -- 172.21.15.89:0/359229560 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5569c8714d70 con 0x5569c87bf8d0 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97cb7fe640 10 client.18507 handle_command_reply: tid=0 2024-09-15T23:33:13.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97f51c3740 1 client.18507 shutdown 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97f51c3740 1 -- 172.21.15.89:0/359229560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 msgr2=0x5569c87b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97f51c3740 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87b4080 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f97e000bf00 tx=0x7f97e00117b0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97f51c3740 1 -- 172.21.15.89:0/359229560 shutdown_connections 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97f51c3740 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5569c87bf8d0 0x5569c87c1d30 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97f51c3740 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87af370 0x5569c87bb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97cb7fe640 0 client.18507 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97f51c3740 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87aa2e0 0x5569c87b4080 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97f51c3740 1 --2- 172.21.15.89:0/359229560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87a99a0 0x5569c87b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.556+0000 7f97f51c3740 1 -- 172.21.15.89:0/359229560 >> 172.21.15.89:0/359229560 conn(0x5569c87a5010 msgr2=0x5569c87a6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.557+0000 7f97f51c3740 1 -- 172.21.15.89:0/359229560 shutdown_connections 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.557+0000 7f97f51c3740 1 -- 172.21.15.89:0/359229560 wait complete. 2024-09-15T23:33:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.557+0000 7f97f51c3740 20 client.18507 trim_cache size 0 max 16384 2024-09-15T23:33:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f51c3740 1 Processor -- start 2024-09-15T23:33:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f51c3740 1 -- start start 2024-09-15T23:33:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87a6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87aa2e0 0x5569c87bb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87a6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f51c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec1aa8c0 con 0x5569c87af370 2024-09-15T23:33:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f51c3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec1ac400 con 0x5569c87a99a0 2024-09-15T23:33:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f51c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec1aaf90 con 0x5569c87aa2e0 2024-09-15T23:33:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f09c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f09c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87a6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35414/0 (socket says 172.21.15.89:35414) 2024-09-15T23:33:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f09c4640 1 -- 172.21.15.89:0/3442224809 learned_addr learned my addr 172.21.15.89:0/3442224809 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97ebfff640 1 --2- 172.21.15.89:0/3442224809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f09c4640 1 -- 172.21.15.89:0/3442224809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87aa2e0 msgr2=0x5569c87bb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97eb7fe640 1 --2- 172.21.15.89:0/3442224809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87aa2e0 0x5569c87bb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f09c4640 1 --2- 172.21.15.89:0/3442224809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87aa2e0 0x5569c87bb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f09c4640 1 -- 172.21.15.89:0/3442224809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 msgr2=0x5569c87a6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f09c4640 1 --2- 172.21.15.89:0/3442224809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87a6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.571+0000 7f97f09c4640 1 -- 172.21.15.89:0/3442224809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569c87be180 con 0x5569c87af370 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97f09c4640 1 --2- 172.21.15.89:0/3442224809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87a6940 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009ca0 tx=0x7f97e4043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f47a5973488923 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97d1ffb640 1 -- 172.21.15.89:0/3442224809 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e402fd60 con 0x5569c87af370 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97d1ffb640 1 -- 172.21.15.89:0/3442224809 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97e4036040 con 0x5569c87af370 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97d1ffb640 1 -- 172.21.15.89:0/3442224809 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e40528e0 con 0x5569c87af370 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97f51c3740 1 -- 172.21.15.89:0/3442224809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 msgr2=0x5569c87a6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97f51c3740 1 --2- 172.21.15.89:0/3442224809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87a6940 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009ca0 tx=0x7f97e4043000 comp rx=0 tx=0).stop 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97f51c3740 1 -- 172.21.15.89:0/3442224809 shutdown_connections 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97f51c3740 1 --2- 172.21.15.89:0/3442224809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87a6940 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97f51c3740 1 --2- 172.21.15.89:0/3442224809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87aa2e0 0x5569c87bb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97f51c3740 1 --2- 172.21.15.89:0/3442224809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87a6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97f51c3740 1 -- 172.21.15.89:0/3442224809 >> 172.21.15.89:0/3442224809 conn(0x5569c87a5010 msgr2=0x5569c87a41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97f51c3740 1 -- 172.21.15.89:0/3442224809 shutdown_connections 2024-09-15T23:33:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.572+0000 7f97f51c3740 1 -- 172.21.15.89:0/3442224809 wait complete. 2024-09-15T23:33:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f51c3740 1 -- start start 2024-09-15T23:33:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f51c3740 1 Processor -- start 2024-09-15T23:33:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87aa2e0 0x5569c87b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f51c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87b9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f51c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec1abd30 con 0x5569c87af370 2024-09-15T23:33:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f51c3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec1ab660 con 0x5569c87a99a0 2024-09-15T23:33:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f51c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec1a88a0 con 0x5569c87aa2e0 2024-09-15T23:33:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f51c3740 10 client.0 _ll_register_callbacks cb 0x7f97e000b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f09c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87b9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f09c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87b9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35418/0 (socket says 172.21.15.89:35418) 2024-09-15T23:33:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f09c4640 1 -- 172.21.15.89:0/678880771 learned_addr learned my addr 172.21.15.89:0/678880771 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97ebfff640 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f09c4640 1 -- 172.21.15.89:0/678880771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87aa2e0 msgr2=0x5569c87b8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f09c4640 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87aa2e0 0x5569c87b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f09c4640 1 -- 172.21.15.89:0/678880771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 msgr2=0x5569c87b88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.573+0000 7f97f09c4640 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87b88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f09c4640 1 -- 172.21.15.89:0/678880771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569c87ba3a0 con 0x5569c87af370 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f09c4640 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87b9d90 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f97e4005de0 tx=0x7f97e402fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a28979e35530056 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97cb7fe640 1 -- 172.21.15.89:0/678880771 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e405a060 con 0x5569c87af370 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97cb7fe640 1 -- 172.21.15.89:0/678880771 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f97e40309d0 con 0x5569c87af370 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f51c3740 1 -- 172.21.15.89:0/678880771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97e4036040 con 0x5569c87af370 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97cb7fe640 1 -- 172.21.15.89:0/678880771 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f97e4061050 con 0x5569c87af370 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f51c3740 10 client.18513 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f51c3740 1 -- 172.21.15.89:0/678880771 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5569c882a6d0 con 0x5569c87af370 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97cb7fe640 1 -- 172.21.15.89:0/678880771 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97e4036040 con 0x5569c87af370 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97cb7fe640 1 -- 172.21.15.89:0/678880771 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f97e4046590 con 0x5569c87af370 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f51c3740 10 client.18513 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f51c3740 10 client.18513 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f51c3740 20 client.18513 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:13.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f51c3740 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5569c882b060 0x5569c882d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f51c3740 4 client.18513 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:33:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.574+0000 7f97f51c3740 1 -- 172.21.15.89:0/678880771 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5569c8691900 con 0x5569c882b060 2024-09-15T23:33:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.575+0000 7f97ebfff640 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5569c882b060 0x5569c882d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.575+0000 7f97ebfff640 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5569c882b060 0x5569c882d480 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6eecb0e92f2ed1e server_cookie=d22d766784be987f in_seq=0 out_seq=0 2024-09-15T23:33:13.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.575+0000 7f97cb7fe640 10 client.18513 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.589+0000 7f97cb7fe640 1 -- 172.21.15.89:0/678880771 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5569c8691900 con 0x5569c882b060 2024-09-15T23:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.589+0000 7f97cb7fe640 10 client.18513 handle_command_reply: tid=0 2024-09-15T23:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 client.18513 shutdown 2024-09-15T23:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 -- 172.21.15.89:0/678880771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 msgr2=0x5569c87b9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87b9d90 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f97e4005de0 tx=0x7f97e402fe10 comp rx=0 tx=0).stop 2024-09-15T23:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 -- 172.21.15.89:0/678880771 shutdown_connections 2024-09-15T23:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5569c882b060 0x5569c882d480 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569c87af370 0x5569c87b9d90 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97cb7fe640 0 client.18513 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569c87aa2e0 0x5569c87b8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 --2- 172.21.15.89:0/678880771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c87a99a0 0x5569c87b88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 -- 172.21.15.89:0/678880771 >> 172.21.15.89:0/678880771 conn(0x5569c87a5010 msgr2=0x5569c87a41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 -- 172.21.15.89:0/678880771 shutdown_connections 2024-09-15T23:33:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 1 -- 172.21.15.89:0/678880771 wait complete. 2024-09-15T23:33:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f51c3740 20 client.18513 trim_cache size 0 max 16384 2024-09-15T23:33:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:33:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:33:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:33:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:33:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f2427640 1 -- 172.21.15.89:0/377825791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97d40a32f0 msgr2=0x7f97d40a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f2427640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97d40a32f0 0x7f97d40a57b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f97e4006010 tx=0x7f97e403f040 comp rx=0 tx=0).stop 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f2427640 1 -- 172.21.15.89:0/377825791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97ec1025e0 msgr2=0x7f97ec197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.590+0000 7f97f2427640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97ec1025e0 0x7f97ec197e00 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00d950 tx=0x7f97dc00de10 comp rx=0 tx=0).stop 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.591+0000 7f97f2427640 1 -- 172.21.15.89:0/377825791 shutdown_connections 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.591+0000 7f97f2427640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f97d40a32f0 0x7f97d40a57b0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.591+0000 7f97f2427640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f97ec107840 0x7f97ec19f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.591+0000 7f97f2427640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97ec104f10 0x7f97ec198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.591+0000 7f97f2427640 1 --2- 172.21.15.89:0/377825791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97ec1025e0 0x7f97ec197e00 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.591+0000 7f97f2427640 1 -- 172.21.15.89:0/377825791 >> 172.21.15.89:0/377825791 conn(0x7f97ec0fc1e0 msgr2=0x7f97ec102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.591+0000 7f97f2427640 1 -- 172.21.15.89:0/377825791 shutdown_connections 2024-09-15T23:33:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:13.591+0000 7f97f2427640 1 -- 172.21.15.89:0/377825791 wait complete. 2024-09-15T23:33:14.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:14 smithi089 ceph-mon[30933]: pgmap v626: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.2 KiB/s wr, 16 op/s 2024-09-15T23:33:14.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:14 smithi120 ceph-mon[35250]: pgmap v626: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.2 KiB/s wr, 16 op/s 2024-09-15T23:33:14.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:14 smithi174 ceph-mon[35586]: pgmap v626: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.2 KiB/s wr, 16 op/s 2024-09-15T23:33:16.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:16 smithi089 ceph-mon[30933]: pgmap v627: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.9 KiB/s wr, 14 op/s 2024-09-15T23:33:16.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:16 smithi120 ceph-mon[35250]: pgmap v627: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.9 KiB/s wr, 14 op/s 2024-09-15T23:33:16.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:16 smithi174 ceph-mon[35586]: pgmap v627: 129 pgs: 129 active+clean; 4.6 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.9 KiB/s wr, 14 op/s 2024-09-15T23:33:18.525 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:18 smithi120 ceph-mon[35250]: pgmap v628: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-15T23:33:18.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:18 smithi174 ceph-mon[35586]: pgmap v628: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-15T23:33:18.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:18 smithi089 ceph-mon[30933]: pgmap v628: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-15T23:33:20.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:20 smithi089 ceph-mon[30933]: pgmap v629: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.4 KiB/s wr, 13 op/s 2024-09-15T23:33:20.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:20 smithi120 ceph-mon[35250]: pgmap v629: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.4 KiB/s wr, 13 op/s 2024-09-15T23:33:20.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:20 smithi174 ceph-mon[35586]: pgmap v629: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.4 KiB/s wr, 13 op/s 2024-09-15T23:33:22.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:22 smithi089 ceph-mon[30933]: pgmap v630: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.4 KiB/s wr, 15 op/s 2024-09-15T23:33:22.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:22 smithi120 ceph-mon[35250]: pgmap v630: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.4 KiB/s wr, 15 op/s 2024-09-15T23:33:22.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:22 smithi174 ceph-mon[35586]: pgmap v630: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.4 KiB/s wr, 15 op/s 2024-09-15T23:33:24.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:24 smithi089 ceph-mon[30933]: pgmap v631: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-15T23:33:24.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:24 smithi120 ceph-mon[35250]: pgmap v631: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-15T23:33:24.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:24 smithi174 ceph-mon[35586]: pgmap v631: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-15T23:33:26.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:26 smithi089 ceph-mon[30933]: pgmap v632: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-09-15T23:33:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.688+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/1816194223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b8100a40 msgr2=0x7fc5b8100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.688+0000 7fc5bfbc0640 1 --2- 172.21.15.89:0/1816194223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b8100a40 0x7fc5b8100e40 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009ca0 tx=0x7fc5b402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.688+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/1816194223 shutdown_connections 2024-09-15T23:33:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.688+0000 7fc5bfbc0640 1 --2- 172.21.15.89:0/1816194223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b81025f0 0x7fc5b8108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.688+0000 7fc5bfbc0640 1 --2- 172.21.15.89:0/1816194223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5b8101c30 0x7fc5b81020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.688+0000 7fc5bfbc0640 1 --2- 172.21.15.89:0/1816194223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b8100a40 0x7fc5b8100e40 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.691 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.688+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/1816194223 >> 172.21.15.89:0/1816194223 conn(0x7fc5b80fc1e0 msgr2=0x7fc5b80fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.688+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/1816194223 shutdown_connections 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.688+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/1816194223 wait complete. 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bfbc0640 1 Processor -- start 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bfbc0640 1 -- start start 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bfbc0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b8100a40 0x7fc5b81049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bfbc0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b8101c30 0x7fc5b8103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bfbc0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5b81025f0 0x7fc5b8103550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bfbc0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b810c6c0 con 0x7fc5b8100a40 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bfbc0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b810c540 con 0x7fc5b8101c30 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bfbc0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b810c840 con 0x7fc5b81025f0 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bd95c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b8100a40 0x7fc5b81049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bd95c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b8100a40 0x7fc5b81049d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54846/0 (socket says 172.21.15.89:54846) 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bd95c640 1 -- 172.21.15.89:0/3708735362 learned_addr learned my addr 172.21.15.89:0/3708735362 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bd15b640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b8101c30 0x7fc5b8103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5be15d640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5b81025f0 0x7fc5b8103550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bd95c640 1 -- 172.21.15.89:0/3708735362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5b81025f0 msgr2=0x7fc5b8103550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5b81025f0 0x7fc5b8103550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bd95c640 1 -- 172.21.15.89:0/3708735362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b8101c30 msgr2=0x7fc5b8103010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b8101c30 0x7fc5b8103010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5bd95c640 1 -- 172.21.15.89:0/3708735362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b8103de0 con 0x7fc5b8100a40 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.689+0000 7fc5be15d640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5b81025f0 0x7fc5b8103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.690+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b8100a40 0x7fc5b81049d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4004af0 tx=0x7fc5b4031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.690+0000 7fc5aeffd640 1 -- 172.21.15.89:0/3708735362 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b402fb60 con 0x7fc5b8100a40 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.690+0000 7fc5aeffd640 1 -- 172.21.15.89:0/3708735362 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b402fd00 con 0x7fc5b8100a40 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.690+0000 7fc5aeffd640 1 -- 172.21.15.89:0/3708735362 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4046920 con 0x7fc5b8100a40 2024-09-15T23:33:26.693 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.690+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/3708735362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b81a1f40 con 0x7fc5b8100a40 2024-09-15T23:33:26.694 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.690+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/3708735362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b81a2350 con 0x7fc5b8100a40 2024-09-15T23:33:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.692+0000 7fc5aeffd640 1 -- 172.21.15.89:0/3708735362 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc5b4030c30 con 0x7fc5b8100a40 2024-09-15T23:33:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5c2a2e740 1 Processor -- start 2024-09-15T23:33:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5c2a2e740 1 -- start start 2024-09-15T23:33:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefd7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefdcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefdf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e5fef3b2f0 con 0x55e5fefd79a0 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e5feee3270 con 0x55e5fefdd370 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e5fef372b0 con 0x55e5fefd82e0 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5aeffd640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5a40a32f0 0x7fc5a40a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5be15d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefd7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5be15d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefd7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54860/0 (socket says 172.21.15.89:54860) 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.693+0000 7fc5be15d640 1 -- 172.21.15.89:0/1088211538 learned_addr learned my addr 172.21.15.89:0/1088211538 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5be15d640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5a40a32f0 0x7fc5a40a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5aeffd640 1 -- 172.21.15.89:0/3708735362 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc5b40f11a0 con 0x7fc5b8100a40 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/1088211538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefdcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5bd15b640 1 --2- 172.21.15.89:0/1088211538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefdf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5be15d640 1 -- 172.21.15.89:0/1088211538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 msgr2=0x55e5fefdcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5be15d640 1 --2- 172.21.15.89:0/1088211538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefdcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5be15d640 1 -- 172.21.15.89:0/1088211538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 msgr2=0x55e5fefdf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5be15d640 1 --2- 172.21.15.89:0/1088211538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefdf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5be15d640 1 -- 172.21.15.89:0/1088211538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5feee2340 con 0x55e5fefd79a0 2024-09-15T23:33:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5be15d640 1 --2- 172.21.15.89:0/1088211538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefd7da0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0014cf0 tx=0x7fc5b00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4eaff9d0cf41d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5be15d640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5a40a32f0 0x7fc5a40a57b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0011cb0 tx=0x7fc5b0016000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5a3fff640 1 -- 172.21.15.89:0/1088211538 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0009b40 con 0x55e5fefd79a0 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5a3fff640 1 -- 172.21.15.89:0/1088211538 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b000ca80 con 0x55e5fefd79a0 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.694+0000 7fc5a3fff640 1 -- 172.21.15.89:0/1088211538 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00047a0 con 0x55e5fefd79a0 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.695+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1088211538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 msgr2=0x55e5fefd7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.695+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/1088211538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefd7da0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0014cf0 tx=0x7fc5b00117b0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.695+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1088211538 shutdown_connections 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.695+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/1088211538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefdf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.695+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/1088211538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefdcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.695+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/1088211538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefd7da0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.695+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1088211538 >> 172.21.15.89:0/1088211538 conn(0x55e5fefd3010 msgr2=0x55e5fefd3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.695+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1088211538 shutdown_connections 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.695+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1088211538 wait complete. 2024-09-15T23:33:26.702 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.695+0000 7fc5c2a2e740 1 -- start start 2024-09-15T23:33:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5c2a2e740 1 Processor -- start 2024-09-15T23:33:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd79a0 0x55e5fefe1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd82e0 0x55e5fefe20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefe9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e5fefe02e0 con 0x55e5fefd82e0 2024-09-15T23:33:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e5feec4290 con 0x55e5fefdd370 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e5fefe0460 con 0x55e5fefd79a0 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5c2a2e740 10 client.0 _ll_register_callbacks cb 0x55e5feef5140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5bd15b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd82e0 0x55e5fefe20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5bd15b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd82e0 0x55e5fefe20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54872/0 (socket says 172.21.15.89:54872) 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5bd15b640 1 -- 172.21.15.89:0/18932288 learned_addr learned my addr 172.21.15.89:0/18932288 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd79a0 0x55e5fefe1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5bd15b640 1 -- 172.21.15.89:0/18932288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd79a0 msgr2=0x55e5fefe1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5bd15b640 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd79a0 0x55e5fefe1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5bd15b640 1 -- 172.21.15.89:0/18932288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 msgr2=0x55e5fefe9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5be15d640 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefe9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5bd15b640 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefe9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.705 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.696+0000 7fc5bd15b640 1 -- 172.21.15.89:0/18932288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5fec4e210 con 0x55e5fefd82e0 2024-09-15T23:33:26.705 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5bd15b640 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd82e0 0x55e5fefe20d0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8012700 tx=0x7fc5a8012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f40d55af5ed0814 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5a17fa640 1 -- 172.21.15.89:0/18932288 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a801f060 con 0x55e5fefd82e0 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/18932288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e5fef48a20 con 0x55e5fefd82e0 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5a17fa640 1 -- 172.21.15.89:0/18932288 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc5a801a430 con 0x55e5fefd82e0 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5a17fa640 1 -- 172.21.15.89:0/18932288 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc5a8026050 con 0x55e5fefd82e0 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5c2a2e740 10 client.18525 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/18932288 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e5fed1a620 con 0x55e5fefd82e0 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5a17fa640 1 -- 172.21.15.89:0/18932288 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e5fef48a20 con 0x55e5fefd82e0 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5a17fa640 1 -- 172.21.15.89:0/18932288 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc5a800d340 con 0x55e5fefd82e0 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5c2a2e740 10 client.18525 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5c2a2e740 10 client.18525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5c2a2e740 20 client.18525 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e5fefed8b0 0x55e5fefefcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5c2a2e740 4 client.18525 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.697+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/18932288 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5fef42d70 con 0x55e5fefed8b0 2024-09-15T23:33:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.698+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e5fefed8b0 0x55e5fefefcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.698+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e5fefed8b0 0x55e5fefefcc0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65459431e0f3daee server_cookie=2a79799a7057e507 in_seq=0 out_seq=0 2024-09-15T23:33:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.698+0000 7fc5a17fa640 10 client.18525 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:26.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5a17fa640 1 -- 172.21.15.89:0/18932288 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e5fef42d70 con 0x55e5fefed8b0 2024-09-15T23:33:26.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5a17fa640 10 client.18525 handle_command_reply: tid=0 2024-09-15T23:33:26.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 client.18525 shutdown 2024-09-15T23:33:26.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/18932288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd82e0 msgr2=0x55e5fefe20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd82e0 0x55e5fefe20d0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8012700 tx=0x7fc5a8012bc0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/18932288 shutdown_connections 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e5fefed8b0 0x55e5fefefcc0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefe9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd82e0 0x55e5fefe20d0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/18932288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd79a0 0x55e5fefe1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5a17fa640 0 client.18525 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/18932288 >> 172.21.15.89:0/18932288 conn(0x55e5fefd3010 msgr2=0x55e5fefd4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/18932288 shutdown_connections 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/18932288 wait complete. 2024-09-15T23:33:26.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.699+0000 7fc5c2a2e740 20 client.18525 trim_cache size 0 max 16384 2024-09-15T23:33:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5c2a2e740 1 Processor -- start 2024-09-15T23:33:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5c2a2e740 1 -- start start 2024-09-15T23:33:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefd79a0 0x55e5fefd4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefe90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefdd370 0x55e5feff45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b81aca10 con 0x55e5fefdd370 2024-09-15T23:33:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b81ae550 con 0x55e5fefd79a0 2024-09-15T23:33:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b81ad0e0 con 0x55e5fefd82e0 2024-09-15T23:33:26.720 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5be15d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefdd370 0x55e5feff45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5be15d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefdd370 0x55e5feff45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54876/0 (socket says 172.21.15.89:54876) 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5be15d640 1 -- 172.21.15.89:0/521590129 learned_addr learned my addr 172.21.15.89:0/521590129 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/521590129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefd79a0 0x55e5fefd4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5bd15b640 1 --2- 172.21.15.89:0/521590129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefe90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5be15d640 1 -- 172.21.15.89:0/521590129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 msgr2=0x55e5fefe90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5be15d640 1 --2- 172.21.15.89:0/521590129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefe90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5be15d640 1 -- 172.21.15.89:0/521590129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefd79a0 msgr2=0x55e5fefd4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5be15d640 1 --2- 172.21.15.89:0/521590129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefd79a0 0x55e5fefd4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5be15d640 1 -- 172.21.15.89:0/521590129 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5fefd2e30 con 0x55e5fefdd370 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.714+0000 7fc5be15d640 1 --2- 172.21.15.89:0/521590129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefdd370 0x55e5feff45c0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0004810 tx=0x7fc5b0004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f60877d573dfc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5a37fe640 1 -- 172.21.15.89:0/521590129 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0014840 con 0x55e5fefdd370 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5a37fe640 1 -- 172.21.15.89:0/521590129 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b00053b0 con 0x55e5fefdd370 2024-09-15T23:33:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5a37fe640 1 -- 172.21.15.89:0/521590129 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0004c30 con 0x55e5fefdd370 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/521590129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefdd370 msgr2=0x55e5feff45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/521590129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefdd370 0x55e5feff45c0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0004810 tx=0x7fc5b0004840 comp rx=0 tx=0).stop 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/521590129 shutdown_connections 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/521590129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefdd370 0x55e5feff45c0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/521590129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefe90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/521590129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefd79a0 0x55e5fefd4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/521590129 >> 172.21.15.89:0/521590129 conn(0x55e5fefd3010 msgr2=0x55e5fefd33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/521590129 shutdown_connections 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.715+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/521590129 wait complete. 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5c2a2e740 1 -- start start 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5c2a2e740 1 Processor -- start 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefe8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.722 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefe8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5c2a2e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefe70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b81ade80 con 0x55e5fefd79a0 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b81ad7b0 con 0x55e5fefdd370 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5c2a2e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b81aa9f0 con 0x55e5fefd82e0 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5c2a2e740 10 client.0 _ll_register_callbacks cb 0x7fc5a800a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5bd95c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefe8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5bd95c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefe8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54890/0 (socket says 172.21.15.89:54890) 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5bd95c640 1 -- 172.21.15.89:0/1404292728 learned_addr learned my addr 172.21.15.89:0/1404292728 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5be15d640 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefe70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5bd15b640 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefe8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5bd95c640 1 -- 172.21.15.89:0/1404292728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 msgr2=0x55e5fefe8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefe8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.723 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5bd95c640 1 -- 172.21.15.89:0/1404292728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 msgr2=0x55e5fefe70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefe70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.716+0000 7fc5bd95c640 1 -- 172.21.15.89:0/1404292728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5fefe78d0 con 0x55e5fefd79a0 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5bd95c640 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefe8a90 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4031ae0 tx=0x7fc5b403d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df4b56461904386 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5a17fa640 1 -- 172.21.15.89:0/1404292728 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b40bdad0 con 0x55e5fefd79a0 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5a17fa640 1 -- 172.21.15.89:0/1404292728 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc5b40b9a40 con 0x55e5fefd79a0 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1404292728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5b00053b0 con 0x55e5fefd79a0 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5a17fa640 1 -- 172.21.15.89:0/1404292728 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc5b40bd3d0 con 0x55e5fefd79a0 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5c2a2e740 10 client.18531 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1404292728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e5ff059140 con 0x55e5fefd79a0 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5a17fa640 1 -- 172.21.15.89:0/1404292728 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b00053b0 con 0x55e5fefd79a0 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5a17fa640 1 -- 172.21.15.89:0/1404292728 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc5b404e030 con 0x55e5fefd79a0 2024-09-15T23:33:26.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5c2a2e740 10 client.18531 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:26.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5c2a2e740 10 client.18531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:26.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5c2a2e740 20 client.18531 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:26.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e5ff0599d0 0x55e5ff05bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:26.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5c2a2e740 4 client.18531 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:33:26.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.717+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1404292728 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e5feebf900 con 0x55e5ff0599d0 2024-09-15T23:33:26.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.718+0000 7fc5bd15b640 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e5ff0599d0 0x55e5ff05bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:26.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.718+0000 7fc5a17fa640 10 client.18531 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:26.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.718+0000 7fc5bd15b640 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e5ff0599d0 0x55e5ff05bdf0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=234d68cf5102e058 server_cookie=fced1d03f26744bb in_seq=0 out_seq=0 2024-09-15T23:33:26.725 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:26 smithi120 ceph-mon[35250]: pgmap v632: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5a17fa640 1 -- 172.21.15.89:0/1404292728 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e5feebf900 con 0x55e5ff0599d0 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5a17fa640 10 client.18531 handle_command_reply: tid=0 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5c2a2e740 1 client.18531 shutdown 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1404292728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 msgr2=0x55e5fefe8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefe8a90 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4031ae0 tx=0x7fc5b403d040 comp rx=0 tx=0).stop 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1404292728 shutdown_connections 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e5ff0599d0 0x55e5ff05bdf0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e5fefdd370 0x55e5fefe70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e5fefd82e0 0x55e5fefe8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5c2a2e740 1 --2- 172.21.15.89:0/1404292728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5fefd79a0 0x55e5fefe8a90 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5a17fa640 0 client.18531 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:26.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.730+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1404292728 >> 172.21.15.89:0/1404292728 conn(0x55e5fefd3010 msgr2=0x55e5fefd3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:26.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1404292728 shutdown_connections 2024-09-15T23:33:26.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5c2a2e740 1 -- 172.21.15.89:0/1404292728 wait complete. 2024-09-15T23:33:26.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5c2a2e740 20 client.18531 trim_cache size 0 max 16384 2024-09-15T23:33:26.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:33:26.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:33:26.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:33:26.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:33:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/3708735362 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5a40a32f0 msgr2=0x7fc5a40a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5a40a32f0 0x7fc5a40a57b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0011cb0 tx=0x7fc5b0016000 comp rx=0 tx=0).stop 2024-09-15T23:33:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/3708735362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b8100a40 msgr2=0x7fc5b81049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b8100a40 0x7fc5b81049d0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4004af0 tx=0x7fc5b4031ee0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/3708735362 shutdown_connections 2024-09-15T23:33:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc5a40a32f0 0x7fc5a40a57b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5b81025f0 0x7fc5b8103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5b8101c30 0x7fc5b8103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 --2- 172.21.15.89:0/3708735362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5b8100a40 0x7fc5b81049d0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/3708735362 >> 172.21.15.89:0/3708735362 conn(0x7fc5b80fc1e0 msgr2=0x7fc5b80fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/3708735362 shutdown_connections 2024-09-15T23:33:26.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:26.731+0000 7fc5bfbc0640 1 -- 172.21.15.89:0/3708735362 wait complete. 2024-09-15T23:33:26.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:26 smithi174 ceph-mon[35586]: pgmap v632: 129 pgs: 129 active+clean; 677 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-09-15T23:33:28.252 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:33:28.514 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:28.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:28 smithi089 ceph-mon[30933]: pgmap v633: 129 pgs: 129 active+clean; 690 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2024-09-15T23:33:28.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:28 smithi120 ceph-mon[35250]: pgmap v633: 129 pgs: 129 active+clean; 690 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2024-09-15T23:33:28.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:28 smithi174 ceph-mon[35586]: pgmap v633: 129 pgs: 129 active+clean; 690 KiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 3.5 KiB/s wr, 15 op/s 2024-09-15T23:33:29.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.007+0000 7f876b8f0640 1 -- 172.21.15.89:0/527315904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764102ef0 msgr2=0x7f876410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:29.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.007+0000 7f876b8f0640 1 --2- 172.21.15.89:0/527315904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764102ef0 0x7f876410da40 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f875c009ca0 tx=0x7f875c02a950 comp rx=0 tx=0).stop 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.007+0000 7f876b8f0640 1 -- 172.21.15.89:0/527315904 shutdown_connections 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.007+0000 7f876b8f0640 1 --2- 172.21.15.89:0/527315904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764102ef0 0x7f876410da40 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.007+0000 7f876b8f0640 1 --2- 172.21.15.89:0/527315904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8764102570 0x7f87641029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.007+0000 7f876b8f0640 1 --2- 172.21.15.89:0/527315904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8764100e20 0x7f8764101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.007+0000 7f876b8f0640 1 -- 172.21.15.89:0/527315904 >> 172.21.15.89:0/527315904 conn(0x7f87640fc7d0 msgr2=0x7f87640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.007+0000 7f876b8f0640 1 -- 172.21.15.89:0/527315904 shutdown_connections 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.007+0000 7f876b8f0640 1 -- 172.21.15.89:0/527315904 wait complete. 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f876b8f0640 1 Processor -- start 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f876b8f0640 1 -- start start 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f876b8f0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8764100e20 0x7f876419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f876b8f0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8764102570 0x7f876419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f876b8f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764102ef0 0x7f876419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f876b8f0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8764110f80 con 0x7f8764102ef0 2024-09-15T23:33:29.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f876b8f0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8764110e00 con 0x7f8764102570 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f876b8f0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8764111100 con 0x7f8764100e20 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f8769e8d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764102ef0 0x7f876419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f8769e8d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764102ef0 0x7f876419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54904/0 (socket says 172.21.15.89:54904) 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f8769e8d640 1 -- 172.21.15.89:0/3484669097 learned_addr learned my addr 172.21.15.89:0/3484669097 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f8768e8b640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8764102570 0x7f876419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.008+0000 7f876968c640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8764100e20 0x7f876419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f8769e8d640 1 -- 172.21.15.89:0/3484669097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8764100e20 msgr2=0x7f876419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f8769e8d640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8764100e20 0x7f876419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f8769e8d640 1 -- 172.21.15.89:0/3484669097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8764102570 msgr2=0x7f876419ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f8769e8d640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8764102570 0x7f876419ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f8769e8d640 1 -- 172.21.15.89:0/3484669097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876419f5f0 con 0x7f8764102ef0 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f8769e8d640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764102ef0 0x7f876419ef10 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f875c004af0 tx=0x7f875c02f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f875a7fc640 1 -- 172.21.15.89:0/3484669097 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c034040 con 0x7f8764102ef0 2024-09-15T23:33:29.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f876b8f0640 1 -- 172.21.15.89:0/3484669097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876419f880 con 0x7f8764102ef0 2024-09-15T23:33:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f875a7fc640 1 -- 172.21.15.89:0/3484669097 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f875c030920 con 0x7f8764102ef0 2024-09-15T23:33:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f875a7fc640 1 -- 172.21.15.89:0/3484669097 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c004630 con 0x7f8764102ef0 2024-09-15T23:33:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.009+0000 7f876b8f0640 1 -- 172.21.15.89:0/3484669097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87641a71c0 con 0x7f8764102ef0 2024-09-15T23:33:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.010+0000 7f876b8f0640 1 -- 172.21.15.89:0/3484669097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8764101200 con 0x7f8764102ef0 2024-09-15T23:33:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.011+0000 7f875a7fc640 1 -- 172.21.15.89:0/3484669097 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f875c037020 con 0x7f8764102ef0 2024-09-15T23:33:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.012+0000 7f875a7fc640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87500a3240 0x7f87500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.012+0000 7f876968c640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87500a3240 0x7f87500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.013+0000 7f875a7fc640 1 -- 172.21.15.89:0/3484669097 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f875c0f0a40 con 0x7f8764102ef0 2024-09-15T23:33:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.013+0000 7f876968c640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87500a3240 0x7f87500a5700 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8760009bd0 tx=0x7f8760005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:29.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.017+0000 7f875a7fc640 1 -- 172.21.15.89:0/3484669097 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f875c0bd370 con 0x7f8764102ef0 2024-09-15T23:33:29.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.222+0000 7f876b8f0640 1 -- 172.21.15.89:0/3484669097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87641a7390 con 0x7f8764102ef0 2024-09-15T23:33:29.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.223+0000 7f875a7fc640 1 -- 172.21.15.89:0/3484669097 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f875c0bcc70 con 0x7f8764102ef0 2024-09-15T23:33:29.225 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:33:29.225 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:33:29.225 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.225+0000 7f876b8f0640 1 -- 172.21.15.89:0/3484669097 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87500a3240 msgr2=0x7f87500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.225+0000 7f876b8f0640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87500a3240 0x7f87500a5700 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8760009bd0 tx=0x7f8760005f90 comp rx=0 tx=0).stop 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.225+0000 7f876b8f0640 1 -- 172.21.15.89:0/3484669097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764102ef0 msgr2=0x7f876419ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.225+0000 7f876b8f0640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764102ef0 0x7f876419ef10 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f875c004af0 tx=0x7f875c02f3a0 comp rx=0 tx=0).stop 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.225+0000 7f876b8f0640 1 -- 172.21.15.89:0/3484669097 shutdown_connections 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.225+0000 7f876b8f0640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87500a3240 0x7f87500a5700 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.225+0000 7f876b8f0640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8764102ef0 0x7f876419ef10 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.225+0000 7f876b8f0640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8764102570 0x7f876419ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.225+0000 7f876b8f0640 1 --2- 172.21.15.89:0/3484669097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8764100e20 0x7f876419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.225+0000 7f876b8f0640 1 -- 172.21.15.89:0/3484669097 >> 172.21.15.89:0/3484669097 conn(0x7f87640fc7d0 msgr2=0x7f876406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:29.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.226+0000 7f876b8f0640 1 -- 172.21.15.89:0/3484669097 shutdown_connections 2024-09-15T23:33:29.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:29.226+0000 7f876b8f0640 1 -- 172.21.15.89:0/3484669097 wait complete. 2024-09-15T23:33:29.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:29 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3484669097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:29.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:29 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3484669097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:29.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:29 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3484669097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:30.259 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 34a03dc2 --release --await' 2024-09-15T23:33:30.259 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 34a03dc2 --release --await 2024-09-15T23:33:30.507 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:30.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:30 smithi089 ceph-mon[30933]: pgmap v634: 129 pgs: 129 active+clean; 690 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-09-15T23:33:30.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:30 smithi120 ceph-mon[35250]: pgmap v634: 129 pgs: 129 active+clean; 690 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-09-15T23:33:30.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:30 smithi174 ceph-mon[35586]: pgmap v634: 129 pgs: 129 active+clean; 690 KiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-09-15T23:33:31.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.996+0000 7faafca97640 1 -- 172.21.15.89:0/4092889317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf8102d40 msgr2=0x7faaf8103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:31.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.996+0000 7faafca97640 1 --2- 172.21.15.89:0/4092889317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf8102d40 0x7faaf8103120 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7faae8009ca0 tx=0x7faae802a950 comp rx=0 tx=0).stop 2024-09-15T23:33:31.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.996+0000 7faafca97640 1 -- 172.21.15.89:0/4092889317 shutdown_connections 2024-09-15T23:33:31.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.996+0000 7faafca97640 1 --2- 172.21.15.89:0/4092889317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faaf810c170 0x7faaf810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.996+0000 7faafca97640 1 --2- 172.21.15.89:0/4092889317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaf8103660 0x7faaf810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.996+0000 7faafca97640 1 --2- 172.21.15.89:0/4092889317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf8102d40 0x7faaf8103120 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.996+0000 7faafca97640 1 -- 172.21.15.89:0/4092889317 >> 172.21.15.89:0/4092889317 conn(0x7faaf80fc7a0 msgr2=0x7faaf80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:31.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.996+0000 7faafca97640 1 -- 172.21.15.89:0/4092889317 shutdown_connections 2024-09-15T23:33:31.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.996+0000 7faafca97640 1 -- 172.21.15.89:0/4092889317 wait complete. 2024-09-15T23:33:31.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faafca97640 1 Processor -- start 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faafca97640 1 -- start start 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faafca97640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faaf8102d40 0x7faaf8198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faafca97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaf8103660 0x7faaf8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faafca97640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf810c170 0x7faaf819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faafca97640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8111f70 con 0x7faaf810c170 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faafca97640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8111df0 con 0x7faaf8103660 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faafca97640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81120f0 con 0x7faaf8102d40 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faaf6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf810c170 0x7faaf819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faaf6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf810c170 0x7faaf819cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54920/0 (socket says 172.21.15.89:54920) 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faaf6d9d640 1 -- 172.21.15.89:0/3750182172 learned_addr learned my addr 172.21.15.89:0/3750182172 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.997+0000 7faaf659c640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faaf8102d40 0x7faaf8198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faaedd9b640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaf8103660 0x7faaf8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faaf6d9d640 1 -- 172.21.15.89:0/3750182172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faaf8102d40 msgr2=0x7faaf8198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:31.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faaf6d9d640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faaf8102d40 0x7faaf8198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faaf6d9d640 1 -- 172.21.15.89:0/3750182172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaf8103660 msgr2=0x7faaf8198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faaf6d9d640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaf8103660 0x7faaf8198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faaf6d9d640 1 -- 172.21.15.89:0/3750182172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf819d3c0 con 0x7faaf810c170 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faaf6d9d640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf810c170 0x7faaf819cce0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7faaf000d750 tx=0x7faaf000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faaeffff640 1 -- 172.21.15.89:0/3750182172 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf0017070 con 0x7faaf810c170 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faafca97640 1 -- 172.21.15.89:0/3750182172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf819d6b0 con 0x7faaf810c170 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faaeffff640 1 -- 172.21.15.89:0/3750182172 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf0005500 con 0x7faaf810c170 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faaeffff640 1 -- 172.21.15.89:0/3750182172 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf0004d90 con 0x7faaf810c170 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:30.998+0000 7faafca97640 1 -- 172.21.15.89:0/3750182172 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf81a4f90 con 0x7faaf810c170 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.000+0000 7faaeffff640 1 -- 172.21.15.89:0/3750182172 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7faaf0004f30 con 0x7faaf810c170 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faafee70740 1 Processor -- start 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faaeffff640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faae40a3240 0x7faae40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faafee70740 1 -- start start 2024-09-15T23:33:31.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faaedd9b640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faae40a3240 0x7faae40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faaeffff640 1 -- 172.21.15.89:0/3750182172 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7faaf00cf640 con 0x7faaf810c170 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faafee70740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faafee70740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x562212895d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faafee70740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562212896240 0x562212898630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faafee70740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5622127f0620 con 0x56221288b060 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faafee70740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562212797e50 con 0x56221288b980 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faafee70740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5622127ec5e0 con 0x562212896240 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faaf659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faaf659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54924/0 (socket says 172.21.15.89:54924) 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faaf659c640 1 -- 172.21.15.89:0/1737017066 learned_addr learned my addr 172.21.15.89:0/1737017066 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.001+0000 7faaedd9b640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faae40a3240 0x7faae40a5700 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7faae00058c0 tx=0x7faae0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaf6d9d640 1 --2- 172.21.15.89:0/1737017066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562212896240 0x562212898630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaedd9b640 1 --2- 172.21.15.89:0/1737017066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x562212895d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaf659c640 1 -- 172.21.15.89:0/1737017066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562212896240 msgr2=0x562212898630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaf659c640 1 --2- 172.21.15.89:0/1737017066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562212896240 0x562212898630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaf659c640 1 -- 172.21.15.89:0/1737017066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 msgr2=0x562212895d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaf659c640 1 --2- 172.21.15.89:0/1737017066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x562212895d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaf659c640 1 -- 172.21.15.89:0/1737017066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562212788d40 con 0x56221288b060 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaf6d9d640 1 --2- 172.21.15.89:0/1737017066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562212896240 0x562212898630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaf659c640 1 --2- 172.21.15.89:0/1737017066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288b440 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7faae8004ad0 tx=0x7faae80098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ccd9ed56bd267 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaecd99640 1 -- 172.21.15.89:0/1737017066 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8048070 con 0x56221288b060 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaecd99640 1 -- 172.21.15.89:0/1737017066 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae802f940 con 0x56221288b060 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.002+0000 7faaecd99640 1 -- 172.21.15.89:0/1737017066 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8034040 con 0x56221288b060 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 -- 172.21.15.89:0/1737017066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 msgr2=0x56221288b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 --2- 172.21.15.89:0/1737017066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288b440 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7faae8004ad0 tx=0x7faae80098d0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 -- 172.21.15.89:0/1737017066 shutdown_connections 2024-09-15T23:33:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 --2- 172.21.15.89:0/1737017066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562212896240 0x562212898630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 --2- 172.21.15.89:0/1737017066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x562212895d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 --2- 172.21.15.89:0/1737017066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288b440 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 -- 172.21.15.89:0/1737017066 >> 172.21.15.89:0/1737017066 conn(0x5622128875f0 msgr2=0x5622128879d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 -- 172.21.15.89:0/1737017066 shutdown_connections 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 -- 172.21.15.89:0/1737017066 wait complete. 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 -- start start 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.003+0000 7faafee70740 1 Processor -- start 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faafee70740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56221288b060 0x5622128931b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faafee70740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x5622128936f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faafee70740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562212896240 0x562212893c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faafee70740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562212898e10 con 0x562212896240 2024-09-15T23:33:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faafee70740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562212778f10 con 0x56221288b980 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faafee70740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562212898f90 con 0x56221288b060 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faafee70740 10 client.0 _ll_register_callbacks cb 0x56221279cdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562212896240 0x562212893c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562212896240 0x562212893c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54938/0 (socket says 172.21.15.89:54938) 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf6d9d640 1 -- 172.21.15.89:0/2130969297 learned_addr learned my addr 172.21.15.89:0/2130969297 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf659c640 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56221288b060 0x5622128931b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaedd9b640 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x5622128936f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf6d9d640 1 -- 172.21.15.89:0/2130969297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56221288b060 msgr2=0x5622128931b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf6d9d640 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56221288b060 0x5622128931b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf6d9d640 1 -- 172.21.15.89:0/2130969297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 msgr2=0x5622128936f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf6d9d640 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x5622128936f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf6d9d640 1 -- 172.21.15.89:0/2130969297 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622124ff580 con 0x562212896240 2024-09-15T23:33:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf659c640 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56221288b060 0x5622128931b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaedd9b640 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x5622128936f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaf6d9d640 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562212896240 0x562212893c30 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7faaf000d720 tx=0x7faaf009b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e7daaa4fedf013 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaddffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaddffb640 1 -- 172.21.15.89:0/2130969297 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf00b4060 con 0x562212896240 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaddffb640 1 -- 172.21.15.89:0/2130969297 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7faaf00cf310 con 0x562212896240 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faafee70740 1 -- 172.21.15.89:0/2130969297 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faae802f940 con 0x562212896240 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.004+0000 7faaddffb640 1 -- 172.21.15.89:0/2130969297 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faaf0020040 con 0x562212896240 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faafee70740 10 client.18549 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faafee70740 1 -- 172.21.15.89:0/2130969297 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5622125c9960 con 0x562212896240 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faaddffb640 1 -- 172.21.15.89:0/2130969297 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae802f940 con 0x562212896240 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faaddffb640 1 -- 172.21.15.89:0/2130969297 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7faaf0097ee0 con 0x562212896240 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faafee70740 10 client.18549 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faafee70740 10 client.18549 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faafee70740 20 client.18549 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faafee70740 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622128a5dd0 0x5622128a8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faafee70740 4 client.18549 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faafee70740 1 -- 172.21.15.89:0/2130969297 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622127fdb90 con 0x5622128a5dd0 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.005+0000 7faaf659c640 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622128a5dd0 0x5622128a8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.006+0000 7faaf659c640 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622128a5dd0 0x5622128a8260 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b23eeff4bafe7fd9 server_cookie=4567455ee85ab69a in_seq=0 out_seq=0 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.006+0000 7faaddffb640 10 client.18549 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.006+0000 7faaddffb640 1 -- 172.21.15.89:0/2130969297 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5622127fdb90 con 0x5622128a5dd0 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.006+0000 7faaddffb640 10 client.18549 handle_command_reply: tid=0 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.006+0000 7faafee70740 1 client.18549 shutdown 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.006+0000 7faafee70740 1 -- 172.21.15.89:0/2130969297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562212896240 msgr2=0x562212893c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.006+0000 7faafee70740 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562212896240 0x562212893c30 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7faaf000d720 tx=0x7faaf009b470 comp rx=0 tx=0).stop 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.007+0000 7faafee70740 1 -- 172.21.15.89:0/2130969297 shutdown_connections 2024-09-15T23:33:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.007+0000 7faafee70740 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622128a5dd0 0x5622128a8260 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.007+0000 7faafee70740 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562212896240 0x562212893c30 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.007+0000 7faaddffb640 0 client.18549 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.007+0000 7faafee70740 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x5622128936f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.007+0000 7faafee70740 1 --2- 172.21.15.89:0/2130969297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56221288b060 0x5622128931b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.007+0000 7faafee70740 1 -- 172.21.15.89:0/2130969297 >> 172.21.15.89:0/2130969297 conn(0x5622128875f0 msgr2=0x56221288d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.007+0000 7faafee70740 1 -- 172.21.15.89:0/2130969297 shutdown_connections 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.007+0000 7faafee70740 1 -- 172.21.15.89:0/2130969297 wait complete. 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.007+0000 7faafee70740 20 client.18549 trim_cache size 0 max 16384 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.023+0000 7faafee70740 1 Processor -- start 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.023+0000 7faafee70740 1 -- start start 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.023+0000 7faafee70740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622128875f0 0x56221288dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.023+0000 7faafee70740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.023+0000 7faafee70740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x562212886640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.023+0000 7faafee70740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81aee60 con 0x56221288b060 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.023+0000 7faafee70740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81b09a0 con 0x56221288b980 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.023+0000 7faafee70740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81af530 con 0x5622128875f0 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaedd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaedd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54954/0 (socket says 172.21.15.89:54954) 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaedd9b640 1 -- 172.21.15.89:0/3584008352 learned_addr learned my addr 172.21.15.89:0/3584008352 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaf659c640 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622128875f0 0x56221288dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaf6d9d640 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x562212886640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaedd9b640 1 -- 172.21.15.89:0/3584008352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622128875f0 msgr2=0x56221288dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaedd9b640 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622128875f0 0x56221288dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaedd9b640 1 -- 172.21.15.89:0/3584008352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 msgr2=0x562212886640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaedd9b640 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x562212886640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaedd9b640 1 -- 172.21.15.89:0/3584008352 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622128950f0 con 0x56221288b060 2024-09-15T23:33:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaf659c640 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622128875f0 0x56221288dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaf6d9d640 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x562212886640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faaedd9b640 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288e4c0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7faae0005fd0 tx=0x7faae0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f185feb9649f16 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faadffff640 1 -- 172.21.15.89:0/3584008352 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae0020640 con 0x56221288b060 2024-09-15T23:33:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faadffff640 1 -- 172.21.15.89:0/3584008352 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae0011e20 con 0x56221288b060 2024-09-15T23:33:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.024+0000 7faadffff640 1 -- 172.21.15.89:0/3584008352 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae001fb80 con 0x56221288b060 2024-09-15T23:33:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 -- 172.21.15.89:0/3584008352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 msgr2=0x56221288e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288e4c0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7faae0005fd0 tx=0x7faae0012000 comp rx=0 tx=0).stop 2024-09-15T23:33:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 -- 172.21.15.89:0/3584008352 shutdown_connections 2024-09-15T23:33:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x562212886640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56221288b060 0x56221288e4c0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 --2- 172.21.15.89:0/3584008352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622128875f0 0x56221288dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 -- 172.21.15.89:0/3584008352 >> 172.21.15.89:0/3584008352 conn(0x56221288eba0 msgr2=0x562212885f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 -- 172.21.15.89:0/3584008352 shutdown_connections 2024-09-15T23:33:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 -- 172.21.15.89:0/3584008352 wait complete. 2024-09-15T23:33:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 -- start start 2024-09-15T23:33:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.025+0000 7faafee70740 1 Processor -- start 2024-09-15T23:33:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faafee70740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622128875f0 0x562212908ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faafee70740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56221288b060 0x5622129091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faafee70740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x56221290cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faafee70740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81b02d0 con 0x5622128875f0 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faafee70740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81afc00 con 0x56221288b980 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faafee70740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faaf81ace40 con 0x56221288b060 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faaf659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622128875f0 0x562212908ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faaf659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622128875f0 0x562212908ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54968/0 (socket says 172.21.15.89:54968) 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faaf659c640 1 -- 172.21.15.89:0/635298879 learned_addr learned my addr 172.21.15.89:0/635298879 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faafee70740 10 client.0 _ll_register_callbacks cb 0x7faaf009b640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faaf659c640 1 -- 172.21.15.89:0/635298879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56221288b060 msgr2=0x5622129091e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faaf659c640 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56221288b060 0x5622129091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faaf659c640 1 -- 172.21.15.89:0/635298879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 msgr2=0x56221290cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faaf659c640 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x56221290cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.026+0000 7faaf659c640 1 -- 172.21.15.89:0/635298879 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56221290d280 con 0x5622128875f0 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faaf659c640 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622128875f0 0x562212908ca0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7faae802ff70 tx=0x7faae80095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc62d49cc7613eef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faaddffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faaddffb640 1 -- 172.21.15.89:0/635298879 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8048040 con 0x5622128875f0 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faaddffb640 1 -- 172.21.15.89:0/635298879 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7faae802f3a0 con 0x5622128875f0 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faafee70740 1 -- 172.21.15.89:0/635298879 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faae0011e20 con 0x5622128875f0 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faaddffb640 1 -- 172.21.15.89:0/635298879 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faae805f050 con 0x5622128875f0 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faafee70740 10 client.18555 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faafee70740 1 -- 172.21.15.89:0/635298879 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562212915880 con 0x5622128875f0 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faaddffb640 1 -- 172.21.15.89:0/635298879 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faae0011e20 con 0x5622128875f0 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faaddffb640 1 -- 172.21.15.89:0/635298879 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7faae80042a0 con 0x5622128875f0 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faafee70740 10 client.18555 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faafee70740 10 client.18555 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faafee70740 20 client.18555 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:31.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faafee70740 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562212896240 0x562212917bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faafee70740 4 client.18555 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "34a03dc2", "release": true, "await": true}] 2024-09-15T23:33:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.027+0000 7faafee70740 1 -- 172.21.15.89:0/635298879 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "34a03dc2", "release": true, "await": true}) -- 0x5622127f80a0 con 0x562212896240 2024-09-15T23:33:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.028+0000 7faaedd9b640 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562212896240 0x562212917bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.028+0000 7faaedd9b640 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562212896240 0x562212917bd0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=669b83c03d6106fe server_cookie=339330313a35fb03 in_seq=0 out_seq=0 2024-09-15T23:33:31.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.028+0000 7faaddffb640 10 client.18555 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faaddffb640 1 -- 172.21.15.89:0/635298879 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5622127f80a0 con 0x562212896240 2024-09-15T23:33:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faaddffb640 10 client.18555 handle_command_reply: tid=0 2024-09-15T23:33:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 client.18555 shutdown 2024-09-15T23:33:31.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 -- 172.21.15.89:0/635298879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622128875f0 msgr2=0x562212908ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622128875f0 0x562212908ca0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7faae802ff70 tx=0x7faae80095e0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 -- 172.21.15.89:0/635298879 shutdown_connections 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562212896240 0x562212917bd0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56221288b980 0x56221290cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56221288b060 0x5622129091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 --2- 172.21.15.89:0/635298879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622128875f0 0x562212908ca0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 -- 172.21.15.89:0/635298879 >> 172.21.15.89:0/635298879 conn(0x56221288eba0 msgr2=0x562212893990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faaddffb640 0 client.18555 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 -- 172.21.15.89:0/635298879 shutdown_connections 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 1 -- 172.21.15.89:0/635298879 wait complete. 2024-09-15T23:33:31.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.031+0000 7faafee70740 20 client.18555 trim_cache size 0 max 16384 2024-09-15T23:33:31.049 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:33:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:33:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:33:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 32, 2024-09-15T23:33:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:33:31.049 INFO:teuthology.orchestra.run.smithi089.stdout: "34a03dc2": { 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 32, 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 142.3, 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:33:31.050 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:33:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:33:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:33:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:31.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:31.051 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:33:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 -- 172.21.15.89:0/3750182172 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faae40a3240 msgr2=0x7faae40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faae40a3240 0x7faae40a5700 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7faae00058c0 tx=0x7faae0005850 comp rx=0 tx=0).stop 2024-09-15T23:33:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 -- 172.21.15.89:0/3750182172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf810c170 msgr2=0x7faaf819cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf810c170 0x7faaf819cce0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7faaf000d750 tx=0x7faaf000dc10 comp rx=0 tx=0).stop 2024-09-15T23:33:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 -- 172.21.15.89:0/3750182172 shutdown_connections 2024-09-15T23:33:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faae40a3240 0x7faae40a5700 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faaf810c170 0x7faaf819cce0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faaf8103660 0x7faaf8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 --2- 172.21.15.89:0/3750182172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faaf8102d40 0x7faaf8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:31.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 -- 172.21.15.89:0/3750182172 >> 172.21.15.89:0/3750182172 conn(0x7faaf80fc7a0 msgr2=0x7faaf80fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:31.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 -- 172.21.15.89:0/3750182172 shutdown_connections 2024-09-15T23:33:31.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:31.032+0000 7faafca97640 1 -- 172.21.15.89:0/3750182172 wait complete. 2024-09-15T23:33:31.753 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '34a03dc2', total seconds elapsed: 25.9 2024-09-15T23:33:31.753 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.35 2024-09-15T23:33:31.754 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:33:31.999 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:32.380 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:32 smithi089 ceph-mon[30933]: pgmap v635: 129 pgs: 129 active+clean; 690 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-09-15T23:33:32.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.456+0000 7f9359a1a640 1 -- 172.21.15.89:0/4157072868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9354100aa0 msgr2=0x7f935410ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:32.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.456+0000 7f9359a1a640 1 --2- 172.21.15.89:0/4157072868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9354100aa0 0x7f935410ea20 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f934c009ca0 tx=0x7f934c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.456+0000 7f9359a1a640 1 -- 172.21.15.89:0/4157072868 shutdown_connections 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.456+0000 7f9359a1a640 1 --2- 172.21.15.89:0/4157072868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9354100aa0 0x7f935410ea20 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.456+0000 7f9359a1a640 1 --2- 172.21.15.89:0/4157072868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354100100 0x7f9354100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.456+0000 7f9359a1a640 1 --2- 172.21.15.89:0/4157072868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9354106b80 0x7f9354106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.456+0000 7f9359a1a640 1 -- 172.21.15.89:0/4157072868 >> 172.21.15.89:0/4157072868 conn(0x7f9354077230 msgr2=0x7f93540fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.456+0000 7f9359a1a640 1 -- 172.21.15.89:0/4157072868 shutdown_connections 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.456+0000 7f9359a1a640 1 -- 172.21.15.89:0/4157072868 wait complete. 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f9359a1a640 1 Processor -- start 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f9359a1a640 1 -- start start 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f9359a1a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354100100 0x7f9354198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f9359a1a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9354100aa0 0x7f9354198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:32.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f9359a1a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9354106b80 0x7f935419ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f9359a1a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9354112090 con 0x7f9354106b80 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f9359a1a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9354111f10 con 0x7f9354100100 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f9359a1a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9354112210 con 0x7f9354100aa0 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f93537fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9354106b80 0x7f935419ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f93537fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9354106b80 0x7f935419ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54984/0 (socket says 172.21.15.89:54984) 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.457+0000 7f93537fe640 1 -- 172.21.15.89:0/1125568296 learned_addr learned my addr 172.21.15.89:0/1125568296 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f93527fc640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9354100aa0 0x7f9354198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f9352ffd640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354100100 0x7f9354198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f93537fe640 1 -- 172.21.15.89:0/1125568296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9354100aa0 msgr2=0x7f9354198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f93537fe640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9354100aa0 0x7f9354198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f93537fe640 1 -- 172.21.15.89:0/1125568296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354100100 msgr2=0x7f9354198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f93537fe640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354100100 0x7f9354198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f93537fe640 1 -- 172.21.15.89:0/1125568296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935419d4e0 con 0x7f9354106b80 2024-09-15T23:33:32.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f93537fe640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9354106b80 0x7f935419ce00 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f934c031f80 tx=0x7f934c031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f9358a18640 1 -- 172.21.15.89:0/1125568296 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c003b70 con 0x7f9354106b80 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f9359a1a640 1 -- 172.21.15.89:0/1125568296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f935419d770 con 0x7f9354106b80 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f9358a18640 1 -- 172.21.15.89:0/1125568296 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f934c003d10 con 0x7f9354106b80 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f9358a18640 1 -- 172.21.15.89:0/1125568296 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c004760 con 0x7f9354106b80 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.458+0000 7f9359a1a640 1 -- 172.21.15.89:0/1125568296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93541a50b0 con 0x7f9354106b80 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.459+0000 7f9359a1a640 1 -- 172.21.15.89:0/1125568296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9354100560 con 0x7f9354106b80 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.460+0000 7f9358a18640 1 -- 172.21.15.89:0/1125568296 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f934c02f710 con 0x7f9354106b80 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.461+0000 7f9358a18640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f933c0a3240 0x7f933c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.461+0000 7f9352ffd640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f933c0a3240 0x7f933c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.461+0000 7f9358a18640 1 -- 172.21.15.89:0/1125568296 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f934c0f11b0 con 0x7f9354106b80 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.462+0000 7f9352ffd640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f933c0a3240 0x7f933c0a5700 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9344005fd0 tx=0x7f9344005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:32.475 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.464+0000 7f9358a18640 1 -- 172.21.15.89:0/1125568296 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f934c0bdae0 con 0x7f9354106b80 2024-09-15T23:33:32.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:33:32.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.665+0000 7f9359a1a640 1 -- 172.21.15.89:0/1125568296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f935410a0b0 con 0x7f9354106b80 2024-09-15T23:33:32.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.666+0000 7f9358a18640 1 -- 172.21.15.89:0/1125568296 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f934c0bd3e0 con 0x7f9354106b80 2024-09-15T23:33:32.669 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:33:32.669 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:33:32.669 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:33:32.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 -- 172.21.15.89:0/1125568296 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f933c0a3240 msgr2=0x7f933c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:32.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f933c0a3240 0x7f933c0a5700 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9344005fd0 tx=0x7f9344005e20 comp rx=0 tx=0).stop 2024-09-15T23:33:32.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 -- 172.21.15.89:0/1125568296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9354106b80 msgr2=0x7f935419ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:32.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9354106b80 0x7f935419ce00 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f934c031f80 tx=0x7f934c031a60 comp rx=0 tx=0).stop 2024-09-15T23:33:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 -- 172.21.15.89:0/1125568296 shutdown_connections 2024-09-15T23:33:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f933c0a3240 0x7f933c0a5700 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9354106b80 0x7f935419ce00 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9354100aa0 0x7f9354198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 --2- 172.21.15.89:0/1125568296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9354100100 0x7f9354198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 -- 172.21.15.89:0/1125568296 >> 172.21.15.89:0/1125568296 conn(0x7f9354077230 msgr2=0x7f9354104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 -- 172.21.15.89:0/1125568296 shutdown_connections 2024-09-15T23:33:32.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:32.669+0000 7f9359a1a640 1 -- 172.21.15.89:0/1125568296 wait complete. 2024-09-15T23:33:32.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:32 smithi120 ceph-mon[35250]: pgmap v635: 129 pgs: 129 active+clean; 690 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-09-15T23:33:32.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:32 smithi174 ceph-mon[35586]: pgmap v635: 129 pgs: 129 active+clean; 690 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-09-15T23:33:33.312 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0' 2024-09-15T23:33:33.313 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0 2024-09-15T23:33:33.372 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:33.493 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:33 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1125568296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:33.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:33 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1125568296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:33.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:33 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1125568296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:33.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.861+0000 7f5adb769640 1 -- 172.21.15.89:0/1275724566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad4102790 msgr2=0x7f5ad41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.861+0000 7f5adb769640 1 --2- 172.21.15.89:0/1275724566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad4102790 0x7f5ad41067e0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4009900 tx=0x7f5ac402a410 comp rx=0 tx=0).stop 2024-09-15T23:33:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.861+0000 7f5adb769640 1 -- 172.21.15.89:0/1275724566 shutdown_connections 2024-09-15T23:33:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.861+0000 7f5adb769640 1 --2- 172.21.15.89:0/1275724566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad4106f10 0x7f5ad410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.861+0000 7f5adb769640 1 --2- 172.21.15.89:0/1275724566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad4102790 0x7f5ad41067e0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.861+0000 7f5adb769640 1 --2- 172.21.15.89:0/1275724566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ad4101de0 0x7f5ad41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.861+0000 7f5adb769640 1 -- 172.21.15.89:0/1275724566 >> 172.21.15.89:0/1275724566 conn(0x7f5ad40fd8c0 msgr2=0x7f5ad40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:33.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.861+0000 7f5adb769640 1 -- 172.21.15.89:0/1275724566 shutdown_connections 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.861+0000 7f5adb769640 1 -- 172.21.15.89:0/1275724566 wait complete. 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5adb769640 1 Processor -- start 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5adb769640 1 -- start start 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5adb769640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ad4101de0 0x7f5ad40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5adb769640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad4102790 0x7f5ad4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5adb769640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad4106f10 0x7f5ad4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5adb769640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4111f70 con 0x7f5ad4102790 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5adb769640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4111df0 con 0x7f5ad4106f10 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5adb769640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41120f0 con 0x7f5ad4101de0 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad8d04640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad4102790 0x7f5ad4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad8d04640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad4102790 0x7f5ad4074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54998/0 (socket says 172.21.15.89:54998) 2024-09-15T23:33:33.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad8d04640 1 -- 172.21.15.89:0/2936981962 learned_addr learned my addr 172.21.15.89:0/2936981962 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad9505640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ad4101de0 0x7f5ad40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad4106f10 0x7f5ad4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad8d04640 1 -- 172.21.15.89:0/2936981962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ad4101de0 msgr2=0x7f5ad40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad8d04640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ad4101de0 0x7f5ad40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad8d04640 1 -- 172.21.15.89:0/2936981962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad4106f10 msgr2=0x7f5ad4074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad8d04640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad4106f10 0x7f5ad4074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad8d04640 1 -- 172.21.15.89:0/2936981962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad41a6910 con 0x7f5ad4102790 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.862+0000 7f5ad8d04640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad4102790 0x7f5ad4074a30 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4006010 tx=0x7f5ac40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.863+0000 7f5aca7fc640 1 -- 172.21.15.89:0/2936981962 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac403f030 con 0x7f5ad4102790 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.863+0000 7f5aca7fc640 1 -- 172.21.15.89:0/2936981962 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac4003ce0 con 0x7f5ad4102790 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.863+0000 7f5adb769640 1 -- 172.21.15.89:0/2936981962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad41a6ba0 con 0x7f5ad4102790 2024-09-15T23:33:33.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.863+0000 7f5aca7fc640 1 -- 172.21.15.89:0/2936981962 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4035070 con 0x7f5ad4102790 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.863+0000 7f5adb769640 1 -- 172.21.15.89:0/2936981962 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad41a7080 con 0x7f5ad4102790 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.865+0000 7f5aca7fc640 1 -- 172.21.15.89:0/2936981962 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5ac4042030 con 0x7f5ad4102790 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.865+0000 7f5addb42740 1 Processor -- start 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.865+0000 7f5addb42740 1 -- start start 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5addb42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5614156404a0 0x561415640880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5addb42740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561415640dc0 0x56141564b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5addb42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56141564b680 0x56141564da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5addb42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5614155a62e0 con 0x5614156404a0 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5addb42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56141554dce0 con 0x56141564b680 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5addb42740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5614155a22a0 con 0x561415640dc0 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5aca7fc640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5ac00a3240 0x7f5ac00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5614156404a0 0x561415640880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5614156404a0 0x561415640880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55008/0 (socket says 172.21.15.89:55008) 2024-09-15T23:33:33.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 -- 172.21.15.89:0/680054095 learned_addr learned my addr 172.21.15.89:0/680054095 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:33.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5ac00a3240 0x7f5ac00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9505640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561415640dc0 0x56141564b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5aca7fc640 1 -- 172.21.15.89:0/2936981962 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5ac40f0390 con 0x7f5ad4102790 2024-09-15T23:33:33.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad8d04640 1 --2- 172.21.15.89:0/680054095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56141564b680 0x56141564da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 -- 172.21.15.89:0/680054095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561415640dc0 msgr2=0x56141564b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:33.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/680054095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561415640dc0 0x56141564b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 -- 172.21.15.89:0/680054095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56141564b680 msgr2=0x56141564da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:33.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/680054095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56141564b680 0x56141564da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 -- 172.21.15.89:0/680054095 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561415532440 con 0x5614156404a0 2024-09-15T23:33:33.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad8d04640 1 --2- 172.21.15.89:0/680054095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56141564b680 0x56141564da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:33.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/680054095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5614156404a0 0x561415640880 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f5acc012c50 tx=0x7f5acc010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9479d1b141bbaa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:33.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.866+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5ac00a3240 0x7f5ac00a5700 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5acc010c40 tx=0x7f5acc017000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:33.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5abf7fe640 1 -- 172.21.15.89:0/680054095 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc0127f0 con 0x5614156404a0 2024-09-15T23:33:33.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5abf7fe640 1 -- 172.21.15.89:0/680054095 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5acc009c00 con 0x5614156404a0 2024-09-15T23:33:33.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5abf7fe640 1 -- 172.21.15.89:0/680054095 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc0053c0 con 0x5614156404a0 2024-09-15T23:33:33.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5addb42740 1 -- 172.21.15.89:0/680054095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5614156404a0 msgr2=0x561415640880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:33.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5addb42740 1 --2- 172.21.15.89:0/680054095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5614156404a0 0x561415640880 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f5acc012c50 tx=0x7f5acc010740 comp rx=0 tx=0).stop 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5addb42740 1 -- 172.21.15.89:0/680054095 shutdown_connections 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5addb42740 1 --2- 172.21.15.89:0/680054095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56141564b680 0x56141564da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5addb42740 1 --2- 172.21.15.89:0/680054095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561415640dc0 0x56141564b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5addb42740 1 --2- 172.21.15.89:0/680054095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5614156404a0 0x561415640880 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5addb42740 1 -- 172.21.15.89:0/680054095 >> 172.21.15.89:0/680054095 conn(0x56141563ca30 msgr2=0x56141563ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5addb42740 1 -- 172.21.15.89:0/680054095 shutdown_connections 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.867+0000 7f5addb42740 1 -- 172.21.15.89:0/680054095 wait complete. 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5addb42740 1 -- start start 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5addb42740 1 Processor -- start 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5addb42740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5614156404a0 0x56141564b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5addb42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561415640dc0 0x5614156482b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5addb42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415648830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5addb42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56141564e250 con 0x56141564b680 2024-09-15T23:33:33.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5addb42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561415561050 con 0x561415640dc0 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5addb42740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56141564e3d0 con 0x5614156404a0 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5addb42740 10 client.0 _ll_register_callbacks cb 0x5614155a1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5ad9d06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415648830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5ad9d06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415648830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55014/0 (socket says 172.21.15.89:55014) 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.868+0000 7f5ad9d06640 1 -- 172.21.15.89:0/2113236073 learned_addr learned my addr 172.21.15.89:0/2113236073 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5ad8d04640 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561415640dc0 0x5614156482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5ad9505640 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5614156404a0 0x56141564b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5ad9d06640 1 -- 172.21.15.89:0/2113236073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5614156404a0 msgr2=0x56141564b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5614156404a0 0x56141564b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5ad9d06640 1 -- 172.21.15.89:0/2113236073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561415640dc0 msgr2=0x5614156482b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561415640dc0 0x5614156482b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5ad9d06640 1 -- 172.21.15.89:0/2113236073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614153173c0 con 0x56141564b680 2024-09-15T23:33:33.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415648830 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f5acc012fd0 tx=0x7f5acc005350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d9429045e0afa1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5abcff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5abcff9640 1 -- 172.21.15.89:0/2113236073 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc005440 con 0x56141564b680 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5abcff9640 1 -- 172.21.15.89:0/2113236073 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5acc005840 con 0x56141564b680 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5addb42740 1 -- 172.21.15.89:0/2113236073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5614155b3850 con 0x56141564b680 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5abcff9640 1 -- 172.21.15.89:0/2113236073 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5acc038050 con 0x56141564b680 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5addb42740 10 client.18573 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5addb42740 1 -- 172.21.15.89:0/2113236073 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561415659c10 con 0x56141564b680 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5abcff9640 1 -- 172.21.15.89:0/2113236073 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5614155b3850 con 0x56141564b680 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.869+0000 7f5abcff9640 1 -- 172.21.15.89:0/2113236073 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5acc02f590 con 0x56141564b680 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.870+0000 7f5addb42740 10 client.18573 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.870+0000 7f5addb42740 10 client.18573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.870+0000 7f5addb42740 20 client.18573 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.870+0000 7f5addb42740 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56141565b0d0 0x56141565d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.870+0000 7f5addb42740 4 client.18573 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.870+0000 7f5addb42740 1 -- 172.21.15.89:0/2113236073 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614155add60 con 0x56141565b0d0 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.870+0000 7f5ad9505640 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56141565b0d0 0x56141565d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.870+0000 7f5ad9505640 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56141565b0d0 0x56141565d4c0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe9d8a543757113c server_cookie=a737af03d4aa1a39 in_seq=0 out_seq=0 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.870+0000 7f5abcff9640 10 client.18573 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5abcff9640 1 -- 172.21.15.89:0/2113236073 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5614155add60 con 0x56141565b0d0 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5abcff9640 10 client.18573 handle_command_reply: tid=0 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 client.18573 shutdown 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 -- 172.21.15.89:0/2113236073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 msgr2=0x561415648830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415648830 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f5acc012fd0 tx=0x7f5acc005350 comp rx=0 tx=0).stop 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 -- 172.21.15.89:0/2113236073 shutdown_connections 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56141565b0d0 0x56141565d4c0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415648830 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561415640dc0 0x5614156482b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5abcff9640 0 client.18573 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:33.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 --2- 172.21.15.89:0/2113236073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5614156404a0 0x56141564b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 -- 172.21.15.89:0/2113236073 >> 172.21.15.89:0/2113236073 conn(0x56141563ca30 msgr2=0x561415642710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:33.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 -- 172.21.15.89:0/2113236073 shutdown_connections 2024-09-15T23:33:33.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 1 -- 172.21.15.89:0/2113236073 wait complete. 2024-09-15T23:33:33.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.871+0000 7f5addb42740 20 client.18573 trim_cache size 0 max 16384 2024-09-15T23:33:33.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.888+0000 7f5addb42740 1 Processor -- start 2024-09-15T23:33:33.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.888+0000 7f5addb42740 1 -- start start 2024-09-15T23:33:33.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.888+0000 7f5addb42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614156404a0 0x56141564a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.888+0000 7f5addb42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561415640dc0 0x561415662a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.888+0000 7f5addb42740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56141564b680 0x561415662fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.888+0000 7f5addb42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41b1010 con 0x561415640dc0 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.888+0000 7f5addb42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41b2b50 con 0x5614156404a0 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.888+0000 7f5addb42740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41b16e0 con 0x56141564b680 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.888+0000 7f5ad8d04640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561415640dc0 0x561415662a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.888+0000 7f5ad8d04640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561415640dc0 0x561415662a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55026/0 (socket says 172.21.15.89:55026) 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5ad8d04640 1 -- 172.21.15.89:0/504352102 learned_addr learned my addr 172.21.15.89:0/504352102 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/504352102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56141564b680 0x561415662fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5ad8d04640 1 -- 172.21.15.89:0/504352102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56141564b680 msgr2=0x561415662fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5ad9505640 1 --2- 172.21.15.89:0/504352102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614156404a0 0x56141564a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5ad8d04640 1 --2- 172.21.15.89:0/504352102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56141564b680 0x561415662fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5ad8d04640 1 -- 172.21.15.89:0/504352102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614156404a0 msgr2=0x56141564a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5ad8d04640 1 --2- 172.21.15.89:0/504352102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614156404a0 0x56141564a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5ad8d04640 1 -- 172.21.15.89:0/504352102 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56141565d830 con 0x561415640dc0 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/504352102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56141564b680 0x561415662fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5ad8d04640 1 --2- 172.21.15.89:0/504352102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561415640dc0 0x561415662a90 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40bc380 tx=0x7f5ac402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3328cefaabf1753 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5abeffd640 1 -- 172.21.15.89:0/504352102 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40da070 con 0x561415640dc0 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5abeffd640 1 -- 172.21.15.89:0/504352102 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac40bcd80 con 0x561415640dc0 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.889+0000 7f5abeffd640 1 -- 172.21.15.89:0/504352102 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4049040 con 0x561415640dc0 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 -- 172.21.15.89:0/504352102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561415640dc0 msgr2=0x561415662a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 --2- 172.21.15.89:0/504352102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561415640dc0 0x561415662a90 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40bc380 tx=0x7f5ac402a410 comp rx=0 tx=0).stop 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 -- 172.21.15.89:0/504352102 shutdown_connections 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 --2- 172.21.15.89:0/504352102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56141564b680 0x561415662fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 --2- 172.21.15.89:0/504352102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561415640dc0 0x561415662a90 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 --2- 172.21.15.89:0/504352102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614156404a0 0x56141564a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 -- 172.21.15.89:0/504352102 >> 172.21.15.89:0/504352102 conn(0x56141563ca30 msgr2=0x56141563bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 -- 172.21.15.89:0/504352102 shutdown_connections 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 -- 172.21.15.89:0/504352102 wait complete. 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 -- start start 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.890+0000 7f5addb42740 1 Processor -- start 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5addb42740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5614156404a0 0x561415653d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5addb42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561415640dc0 0x5614156510e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5addb42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415651620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5addb42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41b2480 con 0x56141564b680 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5addb42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41b1db0 con 0x561415640dc0 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5addb42740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41aeff0 con 0x5614156404a0 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5addb42740 10 client.0 _ll_register_callbacks cb 0x7f5ad0009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad9d06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415651620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad9d06640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415651620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55030/0 (socket says 172.21.15.89:55030) 2024-09-15T23:33:33.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad9d06640 1 -- 172.21.15.89:0/4290286489 learned_addr learned my addr 172.21.15.89:0/4290286489 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:33.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad9505640 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5614156404a0 0x561415653d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad8d04640 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561415640dc0 0x5614156510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad9d06640 1 -- 172.21.15.89:0/4290286489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5614156404a0 msgr2=0x561415653d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:33.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5614156404a0 0x561415653d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad9d06640 1 -- 172.21.15.89:0/4290286489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561415640dc0 msgr2=0x5614156510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561415640dc0 0x5614156510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad9d06640 1 -- 172.21.15.89:0/4290286489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561415651e20 con 0x56141564b680 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5ad9d06640 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415651620 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0097d0 tx=0x7f5acc0099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0633b7e8424544f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5abcff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5abcff9640 1 -- 172.21.15.89:0/4290286489 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc005bd0 con 0x56141564b680 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5abcff9640 1 -- 172.21.15.89:0/4290286489 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5acc01b040 con 0x56141564b680 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.891+0000 7f5addb42740 1 -- 172.21.15.89:0/4290286489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ac40bcd80 con 0x56141564b680 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5abcff9640 1 -- 172.21.15.89:0/4290286489 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5acc044050 con 0x56141564b680 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5addb42740 10 client.18579 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5addb42740 1 -- 172.21.15.89:0/4290286489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614156c59c0 con 0x56141564b680 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5abcff9640 1 -- 172.21.15.89:0/4290286489 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac40bcd80 con 0x56141564b680 2024-09-15T23:33:33.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5abcff9640 1 -- 172.21.15.89:0/4290286489 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5acc02fb30 con 0x56141564b680 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5addb42740 10 client.18579 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5addb42740 10 client.18579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5addb42740 20 client.18579 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5addb42740 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56141565b0d0 0x5614156c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5addb42740 4 client.18579 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5addb42740 1 -- 172.21.15.89:0/4290286489 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56141552a210 con 0x56141565b0d0 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.892+0000 7f5ad9505640 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56141565b0d0 0x5614156c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5ad9505640 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56141565b0d0 0x5614156c80b0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60c0d325a3cae4d4 server_cookie=d4829e05c4a22a35 in_seq=0 out_seq=0 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5abcff9640 10 client.18579 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5abcff9640 1 -- 172.21.15.89:0/4290286489 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56141552a210 con 0x56141565b0d0 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5abcff9640 10 client.18579 handle_command_reply: tid=0 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 client.18579 shutdown 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 -- 172.21.15.89:0/4290286489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 msgr2=0x561415651620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:33.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415651620 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0097d0 tx=0x7f5acc0099a0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 -- 172.21.15.89:0/4290286489 shutdown_connections 2024-09-15T23:33:33.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56141565b0d0 0x5614156c80b0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5abcff9640 0 client.18579 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:33.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56141564b680 0x561415651620 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561415640dc0 0x5614156510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 --2- 172.21.15.89:0/4290286489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5614156404a0 0x561415653d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 -- 172.21.15.89:0/4290286489 >> 172.21.15.89:0/4290286489 conn(0x56141563ca30 msgr2=0x56141563b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:33.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 -- 172.21.15.89:0/4290286489 shutdown_connections 2024-09-15T23:33:33.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 1 -- 172.21.15.89:0/4290286489 wait complete. 2024-09-15T23:33:33.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.893+0000 7f5addb42740 20 client.18579 trim_cache size 0 max 16384 2024-09-15T23:33:33.917 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:33:33.917 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:33:33.917 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:33:33.917 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:33:33.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 -- 172.21.15.89:0/2936981962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5ac00a3240 msgr2=0x7f5ac00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5ac00a3240 0x7f5ac00a5700 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5acc010c40 tx=0x7f5acc017000 comp rx=0 tx=0).stop 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 -- 172.21.15.89:0/2936981962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad4102790 msgr2=0x7f5ad4074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad4102790 0x7f5ad4074a30 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4006010 tx=0x7f5ac40042f0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 -- 172.21.15.89:0/2936981962 shutdown_connections 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5ac00a3240 0x7f5ac00a5700 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ad4106f10 0x7f5ad4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ad4102790 0x7f5ad4074a30 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 --2- 172.21.15.89:0/2936981962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ad4101de0 0x7f5ad40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 -- 172.21.15.89:0/2936981962 >> 172.21.15.89:0/2936981962 conn(0x7f5ad40fd8c0 msgr2=0x7f5ad40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 -- 172.21.15.89:0/2936981962 shutdown_connections 2024-09-15T23:33:33.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:33.894+0000 7f5adb769640 1 -- 172.21.15.89:0/2936981962 wait complete. 2024-09-15T23:33:34.165 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:34.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:34 smithi089 ceph-mon[30933]: pgmap v636: 129 pgs: 129 active+clean; 701 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-09-15T23:33:34.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:34 smithi120 ceph-mon[35250]: pgmap v636: 129 pgs: 129 active+clean; 701 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-09-15T23:33:34.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:34 smithi174 ceph-mon[35586]: pgmap v636: 129 pgs: 129 active+clean; 701 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.1 KiB/s wr, 12 op/s 2024-09-15T23:33:34.749 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:33:34.750 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:33:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.313+0000 7f1f00b24640 1 -- 172.21.15.89:0/2518921400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1efc101420 msgr2=0x7f1efc10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.313+0000 7f1f00b24640 1 --2- 172.21.15.89:0/2518921400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1efc101420 0x7f1efc10d950 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009cd0 tx=0x7f1ef402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.313+0000 7f1f00b24640 1 -- 172.21.15.89:0/2518921400 shutdown_connections 2024-09-15T23:33:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.313+0000 7f1f00b24640 1 --2- 172.21.15.89:0/2518921400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1efc101420 0x7f1efc10d950 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.313+0000 7f1f00b24640 1 --2- 172.21.15.89:0/2518921400 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1efc100a80 0x7f1efc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.313+0000 7f1f00b24640 1 --2- 172.21.15.89:0/2518921400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1efc106a80 0x7f1efc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.313+0000 7f1f00b24640 1 -- 172.21.15.89:0/2518921400 >> 172.21.15.89:0/2518921400 conn(0x7f1efc0fc7a0 msgr2=0x7f1efc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:35.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.313+0000 7f1f00b24640 1 -- 172.21.15.89:0/2518921400 shutdown_connections 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.313+0000 7f1f00b24640 1 -- 172.21.15.89:0/2518921400 wait complete. 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1f00b24640 1 Processor -- start 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1f00b24640 1 -- start start 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1f00b24640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1efc100a80 0x7f1efc1974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1f00b24640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1efc101420 0x7f1efc1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1f00b24640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1efc106a80 0x7f1efc19bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1f00b24640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc110f20 con 0x7f1efc100a80 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1f00b24640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc110da0 con 0x7f1efc101420 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1f00b24640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1110a0 con 0x7f1efc106a80 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1efa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1efc100a80 0x7f1efc1974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1efa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1efc100a80 0x7f1efc1974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43932/0 (socket says 172.21.15.89:43932) 2024-09-15T23:33:35.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.314+0000 7f1efa59c640 1 -- 172.21.15.89:0/3788479428 learned_addr learned my addr 172.21.15.89:0/3788479428 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1efc101420 0x7f1efc1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1efad9d640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1efc106a80 0x7f1efc19bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1efa59c640 1 -- 172.21.15.89:0/3788479428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1efc106a80 msgr2=0x7f1efc19bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1efa59c640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1efc106a80 0x7f1efc19bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1efa59c640 1 -- 172.21.15.89:0/3788479428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1efc101420 msgr2=0x7f1efc1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1efa59c640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1efc101420 0x7f1efc1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1efa59c640 1 -- 172.21.15.89:0/3788479428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1efc19c450 con 0x7f1efc100a80 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1efad9d640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1efc106a80 0x7f1efc19bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1efa59c640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1efc100a80 0x7f1efc1974a0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f1eec00b4a0 tx=0x7f1eec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1eeb7fe640 1 -- 172.21.15.89:0/3788479428 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec014070 con 0x7f1efc100a80 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1eeb7fe640 1 -- 172.21.15.89:0/3788479428 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eec010af0 con 0x7f1efc100a80 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1f00b24640 1 -- 172.21.15.89:0/3788479428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1efc19c740 con 0x7f1efc100a80 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1eeb7fe640 1 -- 172.21.15.89:0/3788479428 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec017d10 con 0x7f1efc100a80 2024-09-15T23:33:35.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.315+0000 7f1f00b24640 1 -- 172.21.15.89:0/3788479428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1efc1a4020 con 0x7f1efc100a80 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.317+0000 7f1eeb7fe640 1 -- 172.21.15.89:0/3788479428 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1eec017070 con 0x7f1efc100a80 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.318+0000 7f1f02efd740 1 Processor -- start 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.318+0000 7f1f02efd740 1 -- start start 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.318+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d2ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.318+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d2f170 0x563e43d394f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.318+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d39a30 0x563e43d3be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.318+0000 7f1f02efd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563e43c93160 con 0x563e43d39a30 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.318+0000 7f1f02efd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563e43c3a720 con 0x563e43d2f170 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.318+0000 7f1f02efd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563e43c8f120 con 0x563e43d2e850 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ef9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d39a30 0x563e43d3be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ef9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d39a30 0x563e43d3be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43938/0 (socket says 172.21.15.89:43938) 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ef9d9b640 1 -- 172.21.15.89:0/4091556525 learned_addr learned my addr 172.21.15.89:0/4091556525 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:35.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1efa59c640 1 --2- 172.21.15.89:0/4091556525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d2f170 0x563e43d394f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1efad9d640 1 --2- 172.21.15.89:0/4091556525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d2ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1eeb7fe640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1ee40a32c0 0x7f1ee40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ef9d9b640 1 -- 172.21.15.89:0/4091556525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 msgr2=0x563e43d2ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/4091556525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d2ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ef9d9b640 1 -- 172.21.15.89:0/4091556525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d2f170 msgr2=0x563e43d394f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/4091556525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d2f170 0x563e43d394f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ef9d9b640 1 -- 172.21.15.89:0/4091556525 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e43a04ba0 con 0x563e43d39a30 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1efad9d640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1ee40a32c0 0x7f1ee40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1eeb7fe640 1 -- 172.21.15.89:0/3788479428 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1eec0cf100 con 0x7f1efc100a80 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/4091556525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d39a30 0x563e43d3be20 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000b7f0 tx=0x7f1ef000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4faf5653c25446 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ee8ff9640 1 -- 172.21.15.89:0/4091556525 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef000d950 con 0x563e43d39a30 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ee8ff9640 1 -- 172.21.15.89:0/4091556525 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef0018070 con 0x563e43d39a30 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.319+0000 7f1ee8ff9640 1 -- 172.21.15.89:0/4091556525 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef00123e0 con 0x563e43d39a30 2024-09-15T23:33:35.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.320+0000 7f1f02efd740 1 -- 172.21.15.89:0/4091556525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d39a30 msgr2=0x563e43d3be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.320+0000 7f1f02efd740 1 --2- 172.21.15.89:0/4091556525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d39a30 0x563e43d3be20 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000b7f0 tx=0x7f1ef000bcb0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.320+0000 7f1efad9d640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1ee40a32c0 0x7f1ee40a5780 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009430 tx=0x7f1ef40095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.320+0000 7f1f02efd740 1 -- 172.21.15.89:0/4091556525 shutdown_connections 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.320+0000 7f1f02efd740 1 --2- 172.21.15.89:0/4091556525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d39a30 0x563e43d3be20 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.320+0000 7f1f02efd740 1 --2- 172.21.15.89:0/4091556525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d2f170 0x563e43d394f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.320+0000 7f1f02efd740 1 --2- 172.21.15.89:0/4091556525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d2ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.320+0000 7f1f02efd740 1 -- 172.21.15.89:0/4091556525 >> 172.21.15.89:0/4091556525 conn(0x563e43d2ade0 msgr2=0x563e43d2b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.320+0000 7f1f02efd740 1 -- 172.21.15.89:0/4091556525 shutdown_connections 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.320+0000 7f1f02efd740 1 -- 172.21.15.89:0/4091556525 wait complete. 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1f02efd740 1 -- start start 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1f02efd740 1 Processor -- start 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2e850 0x563e43d36870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d2f170 0x563e43d36db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d39a30 0x563e43d372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1f02efd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563e43d3c600 con 0x563e43d2e850 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1f02efd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563e43c1b900 con 0x563e43d2f170 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1f02efd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563e43d3c780 con 0x563e43d39a30 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1f02efd740 10 client.0 _ll_register_callbacks cb 0x563e43c19780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1efa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2e850 0x563e43d36870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1efa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2e850 0x563e43d36870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43948/0 (socket says 172.21.15.89:43948) 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1efa59c640 1 -- 172.21.15.89:0/86236469 learned_addr learned my addr 172.21.15.89:0/86236469 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d2f170 0x563e43d36db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1efa59c640 1 -- 172.21.15.89:0/86236469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d39a30 msgr2=0x563e43d372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1efa59c640 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d39a30 0x563e43d372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1efa59c640 1 -- 172.21.15.89:0/86236469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d2f170 msgr2=0x563e43d36db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1efa59c640 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d2f170 0x563e43d36db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.321+0000 7f1efa59c640 1 -- 172.21.15.89:0/86236469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e43c5bc50 con 0x563e43d2e850 2024-09-15T23:33:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1efa59c640 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2e850 0x563e43d36870 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f1eec097ca0 tx=0x7f1eec097660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3af315476e94f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1ee1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/86236469 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec01e070 con 0x563e43d2e850 2024-09-15T23:33:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/86236469 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1eec010ca0 con 0x563e43d2e850 2024-09-15T23:33:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1f02efd740 1 -- 172.21.15.89:0/86236469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ef0018070 con 0x563e43d2e850 2024-09-15T23:33:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/86236469 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1eec09b9d0 con 0x563e43d2e850 2024-09-15T23:33:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1f02efd740 10 client.18591 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1f02efd740 1 -- 172.21.15.89:0/86236469 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e43d48200 con 0x563e43d2e850 2024-09-15T23:33:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/86236469 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef0018070 con 0x563e43d2e850 2024-09-15T23:33:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/86236469 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1eec097960 con 0x563e43d2e850 2024-09-15T23:33:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1f02efd740 10 client.18591 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1f02efd740 10 client.18591 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:33:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1f02efd740 20 client.18591 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1f02efd740 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563e43d49610 0x563e43d4ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1f02efd740 4 client.18591 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.322+0000 7f1f02efd740 1 -- 172.21.15.89:0/86236469 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e43ca0890 con 0x563e43d49610 2024-09-15T23:33:35.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.323+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563e43d49610 0x563e43d4ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.323+0000 7f1ee1ffb640 10 client.18591 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.323+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563e43d49610 0x563e43d4ba60 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6333c6722b8560f4 server_cookie=5f57438f36cd5e60 in_seq=0 out_seq=0 2024-09-15T23:33:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/86236469 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563e43ca0890 con 0x563e43d49610 2024-09-15T23:33:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1ee1ffb640 10 client.18591 handle_command_reply: tid=0 2024-09-15T23:33:35.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1f02efd740 1 client.18591 shutdown 2024-09-15T23:33:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1f02efd740 1 -- 172.21.15.89:0/86236469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2e850 msgr2=0x563e43d36870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1f02efd740 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2e850 0x563e43d36870 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f1eec097ca0 tx=0x7f1eec097660 comp rx=0 tx=0).stop 2024-09-15T23:33:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1f02efd740 1 -- 172.21.15.89:0/86236469 shutdown_connections 2024-09-15T23:33:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1f02efd740 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563e43d49610 0x563e43d4ba60 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1f02efd740 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d39a30 0x563e43d372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1f02efd740 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d2f170 0x563e43d36db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1f02efd740 1 --2- 172.21.15.89:0/86236469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2e850 0x563e43d36870 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1f02efd740 1 -- 172.21.15.89:0/86236469 >> 172.21.15.89:0/86236469 conn(0x563e43d2ade0 msgr2=0x563e43d30ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:35.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1ee1ffb640 0 client.18591 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:35.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.324+0000 7f1f02efd740 1 -- 172.21.15.89:0/86236469 shutdown_connections 2024-09-15T23:33:35.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.325+0000 7f1f02efd740 1 -- 172.21.15.89:0/86236469 wait complete. 2024-09-15T23:33:35.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.325+0000 7f1f02efd740 20 client.18591 trim_cache size 0 max 16384 2024-09-15T23:33:35.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1f02efd740 1 Processor -- start 2024-09-15T23:33:35.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1f02efd740 1 -- start start 2024-09-15T23:33:35.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d36150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d41bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d39a30 0x563e43d42100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1f02efd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1adf10 con 0x563e43d2f170 2024-09-15T23:33:35.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1f02efd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1afa50 con 0x563e43d39a30 2024-09-15T23:33:35.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1f02efd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1ae5e0 con 0x563e43d2e850 2024-09-15T23:33:35.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1ef9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d41bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1ef9d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d41bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43950/0 (socket says 172.21.15.89:43950) 2024-09-15T23:33:35.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1ef9d9b640 1 -- 172.21.15.89:0/3065180916 learned_addr learned my addr 172.21.15.89:0/3065180916 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:35.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1efa59c640 1 --2- 172.21.15.89:0/3065180916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d36150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1efad9d640 1 --2- 172.21.15.89:0/3065180916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d39a30 0x563e43d42100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1ef9d9b640 1 -- 172.21.15.89:0/3065180916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 msgr2=0x563e43d36150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:35.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/3065180916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d36150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1ef9d9b640 1 -- 172.21.15.89:0/3065180916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d39a30 msgr2=0x563e43d42100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:35.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.344+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/3065180916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d39a30 0x563e43d42100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.345+0000 7f1ef9d9b640 1 -- 172.21.15.89:0/3065180916 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e43d4c090 con 0x563e43d2f170 2024-09-15T23:33:35.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.345+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/3065180916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d41bc0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000b7f0 tx=0x7f1ef000f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e5e3694fd30d02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:35.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.345+0000 7f1ee3fff640 1 -- 172.21.15.89:0/3065180916 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0029070 con 0x563e43d2f170 2024-09-15T23:33:35.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.345+0000 7f1ee3fff640 1 -- 172.21.15.89:0/3065180916 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef0021030 con 0x563e43d2f170 2024-09-15T23:33:35.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.345+0000 7f1ee3fff640 1 -- 172.21.15.89:0/3065180916 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0024990 con 0x563e43d2f170 2024-09-15T23:33:35.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 -- 172.21.15.89:0/3065180916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 msgr2=0x563e43d41bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:35.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 --2- 172.21.15.89:0/3065180916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d41bc0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f1ef000b7f0 tx=0x7f1ef000f040 comp rx=0 tx=0).stop 2024-09-15T23:33:35.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 -- 172.21.15.89:0/3065180916 shutdown_connections 2024-09-15T23:33:35.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 --2- 172.21.15.89:0/3065180916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d39a30 0x563e43d42100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 --2- 172.21.15.89:0/3065180916 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d41bc0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 --2- 172.21.15.89:0/3065180916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d36150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 -- 172.21.15.89:0/3065180916 >> 172.21.15.89:0/3065180916 conn(0x563e43d2ade0 msgr2=0x563e43d29ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:35.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 -- 172.21.15.89:0/3065180916 shutdown_connections 2024-09-15T23:33:35.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 -- 172.21.15.89:0/3065180916 wait complete. 2024-09-15T23:33:35.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 -- start start 2024-09-15T23:33:35.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 Processor -- start 2024-09-15T23:33:35.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.346+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d41d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d3c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1f02efd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d39a30 0x563e43d3ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1efa59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d41d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1efa59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d41d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:35422/0 (socket says 172.21.15.89:35422) 2024-09-15T23:33:35.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1efa59c640 1 -- 172.21.15.89:0/2762023133 learned_addr learned my addr 172.21.15.89:0/2762023133 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:35.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1af380 con 0x563e43d2f170 2024-09-15T23:33:35.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1aecb0 con 0x563e43d39a30 2024-09-15T23:33:35.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1abef0 con 0x563e43d2e850 2024-09-15T23:33:35.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d3c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1efad9d640 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d39a30 0x563e43d3ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1f02efd740 10 client.0 _ll_register_callbacks cb 0x7f1eec00b3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:35.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1efa59c640 1 -- 172.21.15.89:0/2762023133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d39a30 msgr2=0x563e43d3ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:35.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1efa59c640 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d39a30 0x563e43d3ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1efa59c640 1 -- 172.21.15.89:0/2762023133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 msgr2=0x563e43d3c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:35.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1efa59c640 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d3c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1efa59c640 1 -- 172.21.15.89:0/2762023133 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e43d263f0 con 0x563e43d2e850 2024-09-15T23:33:35.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d3c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:35.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ef0021030 con 0x563e43d2e850 2024-09-15T23:33:35.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1efa59c640 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d41d70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1eec09afd0 tx=0x7f1eec0047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=779b4d3c0b6bca46 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:35.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.347+0000 7f1ee1ffb640 10 client.27121 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:33:35.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/2762023133 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec01e070 con 0x563e43d2e850 2024-09-15T23:33:35.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/2762023133 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1eec010ca0 con 0x563e43d2e850 2024-09-15T23:33:35.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/2762023133 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1eec09f680 con 0x563e43d2e850 2024-09-15T23:33:35.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1f02efd740 10 client.27121 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:35.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=29}) -- 0x563e43d2ac10 con 0x563e43d2e850 2024-09-15T23:33:35.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/2762023133 <== mon.1 v2:172.21.15.174:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1eec09f180 con 0x563e43d2e850 2024-09-15T23:33:35.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1f02efd740 10 client.27121 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:35.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1f02efd740 10 client.27121 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:33:35.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1f02efd740 20 client.27121 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:35.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1f02efd740 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563e43db52e0 0x563e43db7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:35.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1f02efd740 4 client.27121 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}] 2024-09-15T23:33:35.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.348+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}) -- 0x563e43c9abe0 con 0x563e43db52e0 2024-09-15T23:33:35.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.349+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563e43db52e0 0x563e43db7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:35.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.349+0000 7f1ef9d9b640 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563e43db52e0 0x563e43db7710 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fef516381d3b2d7e server_cookie=d76aca7dea806fe0 in_seq=0 out_seq=0 2024-09-15T23:33:35.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.349+0000 7f1ee1ffb640 10 client.27121 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:35.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1ee1ffb640 1 -- 172.21.15.89:0/2762023133 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x563e43c9abe0 con 0x563e43db52e0 2024-09-15T23:33:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1ee1ffb640 10 client.27121 handle_command_reply: tid=0 2024-09-15T23:33:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1f02efd740 1 client.27121 shutdown 2024-09-15T23:33:35.364 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:35.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 msgr2=0x563e43d41d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:35.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1f02efd740 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d41d70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1eec09afd0 tx=0x7f1eec0047d0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 shutdown_connections 2024-09-15T23:33:35.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1f02efd740 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563e43db52e0 0x563e43db7710 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1f02efd740 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e43d39a30 0x563e43d3ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1f02efd740 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563e43d2f170 0x563e43d3c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1ee1ffb640 0 client.27121 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:35.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1f02efd740 1 --2- 172.21.15.89:0/2762023133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563e43d2e850 0x563e43d41d70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 >> 172.21.15.89:0/2762023133 conn(0x563e43d2ade0 msgr2=0x563e43d36830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:35.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.361+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 shutdown_connections 2024-09-15T23:33:35.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f02efd740 1 -- 172.21.15.89:0/2762023133 wait complete. 2024-09-15T23:33:35.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f02efd740 20 client.27121 trim_cache size 0 max 16384 2024-09-15T23:33:35.366 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:33:35.366 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:33:35.366 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:33:35.366 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 34, 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "ca53bdfb": { 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 34, 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 142.3, 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:33:35.367 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:33:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:33:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:33:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:35.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:35.368 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:33:35.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 -- 172.21.15.89:0/3788479428 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1ee40a32c0 msgr2=0x7f1ee40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:35.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1ee40a32c0 0x7f1ee40a5780 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009430 tx=0x7f1ef40095c0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 -- 172.21.15.89:0/3788479428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1efc100a80 msgr2=0x7f1efc1974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:35.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1efc100a80 0x7f1efc1974a0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f1eec00b4a0 tx=0x7f1eec00b960 comp rx=0 tx=0).stop 2024-09-15T23:33:35.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 -- 172.21.15.89:0/3788479428 shutdown_connections 2024-09-15T23:33:35.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1ee40a32c0 0x7f1ee40a5780 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1efc106a80 0x7f1efc19bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1efc101420 0x7f1efc1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 --2- 172.21.15.89:0/3788479428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1efc100a80 0x7f1efc1974a0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:35.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 -- 172.21.15.89:0/3788479428 >> 172.21.15.89:0/3788479428 conn(0x7f1efc0fc7a0 msgr2=0x7f1efc0fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:35.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 -- 172.21.15.89:0/3788479428 shutdown_connections 2024-09-15T23:33:35.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:35.362+0000 7f1f00b24640 1 -- 172.21.15.89:0/3788479428 wait complete. 2024-09-15T23:33:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.130+0000 7f3c5759e640 1 -- 172.21.15.89:0/2572012259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58101080 msgr2=0x7f3c581014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.130+0000 7f3c5759e640 1 --2- 172.21.15.89:0/2572012259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58101080 0x7f3c581014c0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f3c48009900 tx=0x7f3c4802a410 comp rx=0 tx=0).stop 2024-09-15T23:33:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.130+0000 7f3c5759e640 1 -- 172.21.15.89:0/2572012259 shutdown_connections 2024-09-15T23:33:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.130+0000 7f3c5759e640 1 --2- 172.21.15.89:0/2572012259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c58101a00 0x7f3c5810d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.130+0000 7f3c5759e640 1 --2- 172.21.15.89:0/2572012259 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58101080 0x7f3c581014c0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.130+0000 7f3c5759e640 1 --2- 172.21.15.89:0/2572012259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c581059e0 0x7f3c58105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.130+0000 7f3c5759e640 1 -- 172.21.15.89:0/2572012259 >> 172.21.15.89:0/2572012259 conn(0x7f3c58077010 msgr2=0x7f3c580fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:36.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.130+0000 7f3c5759e640 1 -- 172.21.15.89:0/2572012259 shutdown_connections 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.130+0000 7f3c5759e640 1 -- 172.21.15.89:0/2572012259 wait complete. 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c5759e640 1 Processor -- start 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c5759e640 1 -- start start 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c5759e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c58101080 0x7f3c58198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c5759e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58101a00 0x7f3c581989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c5759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c581059e0 0x7f3c5819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c5759e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c58111fa0 con 0x7f3c58101a00 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c5759e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c58111e20 con 0x7f3c58101080 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c5759e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c58112120 con 0x7f3c581059e0 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c55d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58101a00 0x7f3c581989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c55d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58101a00 0x7f3c581989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43966/0 (socket says 172.21.15.89:43966) 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c55d9b640 1 -- 172.21.15.89:0/204477023 learned_addr learned my addr 172.21.15.89:0/204477023 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c5659c640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c58101080 0x7f3c58198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:36.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c56d9d640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c581059e0 0x7f3c5819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c55d9b640 1 -- 172.21.15.89:0/204477023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c581059e0 msgr2=0x7f3c5819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c55d9b640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c581059e0 0x7f3c5819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c55d9b640 1 -- 172.21.15.89:0/204477023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c58101080 msgr2=0x7f3c58198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c55d9b640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c58101080 0x7f3c58198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c55d9b640 1 -- 172.21.15.89:0/204477023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5819d430 con 0x7f3c58101a00 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c56d9d640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c581059e0 0x7f3c5819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.131+0000 7f3c55d9b640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58101a00 0x7f3c581989c0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f3c48006010 tx=0x7f3c48003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.132+0000 7f3c4f7fe640 1 -- 172.21.15.89:0/204477023 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4803f070 con 0x7f3c58101a00 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.132+0000 7f3c4f7fe640 1 -- 172.21.15.89:0/204477023 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c48031c30 con 0x7f3c58101a00 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.132+0000 7f3c5759e640 1 -- 172.21.15.89:0/204477023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5819d6c0 con 0x7f3c58101a00 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.132+0000 7f3c4f7fe640 1 -- 172.21.15.89:0/204477023 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c48035030 con 0x7f3c58101a00 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.132+0000 7f3c5759e640 1 -- 172.21.15.89:0/204477023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c581a5000 con 0x7f3c58101a00 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.133+0000 7f3c5759e640 1 -- 172.21.15.89:0/204477023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c58102090 con 0x7f3c58101a00 2024-09-15T23:33:36.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.134+0000 7f3c4f7fe640 1 -- 172.21.15.89:0/204477023 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3c48042070 con 0x7f3c58101a00 2024-09-15T23:33:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.134+0000 7f3c4f7fe640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3240 0x7f3c440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.134+0000 7f3c5659c640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3240 0x7f3c440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.134+0000 7f3c4f7fe640 1 -- 172.21.15.89:0/204477023 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c480f08e0 con 0x7f3c58101a00 2024-09-15T23:33:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.135+0000 7f3c5659c640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3240 0x7f3c440a5700 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3c40005d00 tx=0x7f3c4000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:36.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.138+0000 7f3c4f7fe640 1 -- 172.21.15.89:0/204477023 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3c480bd210 con 0x7f3c58101a00 2024-09-15T23:33:36.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:36 smithi089 ceph-mon[30933]: pgmap v637: 129 pgs: 129 active+clean; 701 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-09-15T23:33:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.345+0000 7f3c5759e640 1 -- 172.21.15.89:0/204477023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c581997e0 con 0x7f3c58101a00 2024-09-15T23:33:36.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.346+0000 7f3c4f7fe640 1 -- 172.21.15.89:0/204477023 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f3c480bcb10 con 0x7f3c58101a00 2024-09-15T23:33:36.364 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:33:36.364 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:33:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:33:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 -- 172.21.15.89:0/204477023 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3240 msgr2=0x7f3c440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3240 0x7f3c440a5700 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3c40005d00 tx=0x7f3c4000b410 comp rx=0 tx=0).stop 2024-09-15T23:33:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 -- 172.21.15.89:0/204477023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58101a00 msgr2=0x7f3c581989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58101a00 0x7f3c581989c0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f3c48006010 tx=0x7f3c48003c10 comp rx=0 tx=0).stop 2024-09-15T23:33:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 -- 172.21.15.89:0/204477023 shutdown_connections 2024-09-15T23:33:36.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c440a3240 0x7f3c440a5700 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c581059e0 0x7f3c5819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c58101a00 0x7f3c581989c0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 --2- 172.21.15.89:0/204477023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c58101080 0x7f3c58198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 -- 172.21.15.89:0/204477023 >> 172.21.15.89:0/204477023 conn(0x7f3c58077010 msgr2=0x7f3c581039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 -- 172.21.15.89:0/204477023 shutdown_connections 2024-09-15T23:33:36.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:36.348+0000 7f3c5759e640 1 -- 172.21.15.89:0/204477023 wait complete. 2024-09-15T23:33:36.614 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ca53bdfb', quiesce took 4.9 seconds. Will release after: 17.450000000000003 2024-09-15T23:33:36.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:36 smithi120 ceph-mon[35250]: pgmap v637: 129 pgs: 129 active+clean; 701 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-09-15T23:33:36.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:36 smithi174 ceph-mon[35586]: pgmap v637: 129 pgs: 129 active+clean; 701 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-09-15T23:33:37.254 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:33:37.500 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:37.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:37 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/204477023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:37.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:37 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/204477023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:37.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:37 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/204477023' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:37.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 -- 172.21.15.89:0/872536834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa34c100e20 msgr2=0x7fa34c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:37.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 --2- 172.21.15.89:0/872536834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa34c100e20 0x7fa34c101200 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fa348009900 tx=0x7fa34802a480 comp rx=0 tx=0).stop 2024-09-15T23:33:37.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 -- 172.21.15.89:0/872536834 shutdown_connections 2024-09-15T23:33:37.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 --2- 172.21.15.89:0/872536834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c102ef0 0x7fa34c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:37.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 --2- 172.21.15.89:0/872536834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa34c102570 0x7fa34c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 --2- 172.21.15.89:0/872536834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa34c100e20 0x7fa34c101200 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 -- 172.21.15.89:0/872536834 >> 172.21.15.89:0/872536834 conn(0x7fa34c0fc7d0 msgr2=0x7fa34c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 -- 172.21.15.89:0/872536834 shutdown_connections 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 -- 172.21.15.89:0/872536834 wait complete. 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 Processor -- start 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.993+0000 7fa353476640 1 -- start start 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa353476640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c100e20 0x7fa34c197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa353476640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa34c102570 0x7fa34c197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa353476640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa34c102ef0 0x7fa34c19bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa353476640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c110fc0 con 0x7fa34c102570 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa353476640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c110e40 con 0x7fa34c100e20 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa353476640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c111140 con 0x7fa34c102ef0 2024-09-15T23:33:37.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa350a11640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa34c102570 0x7fa34c197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa350a11640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa34c102570 0x7fa34c197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43980/0 (socket says 172.21.15.89:43980) 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa350a11640 1 -- 172.21.15.89:0/2130816553 learned_addr learned my addr 172.21.15.89:0/2130816553 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa351a13640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa34c102ef0 0x7fa34c19bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa351212640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c100e20 0x7fa34c197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa350a11640 1 -- 172.21.15.89:0/2130816553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa34c102ef0 msgr2=0x7fa34c19bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa350a11640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa34c102ef0 0x7fa34c19bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa350a11640 1 -- 172.21.15.89:0/2130816553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c100e20 msgr2=0x7fa34c197430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa350a11640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c100e20 0x7fa34c197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa350a11640 1 -- 172.21.15.89:0/2130816553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa34c19c3e0 con 0x7fa34c102570 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa350a11640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa34c102570 0x7fa34c197970 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fa34000d770 tx=0x7fa34000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa33e7fc640 1 -- 172.21.15.89:0/2130816553 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa340017070 con 0x7fa34c102570 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa33e7fc640 1 -- 172.21.15.89:0/2130816553 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa340005500 con 0x7fa34c102570 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa353476640 1 -- 172.21.15.89:0/2130816553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa34c19c6d0 con 0x7fa34c102570 2024-09-15T23:33:37.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa33e7fc640 1 -- 172.21.15.89:0/2130816553 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa340004e00 con 0x7fa34c102570 2024-09-15T23:33:37.999 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.994+0000 7fa353476640 1 -- 172.21.15.89:0/2130816553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa34c1a3fb0 con 0x7fa34c102570 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.997+0000 7fa33e7fc640 1 -- 172.21.15.89:0/2130816553 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa340004fa0 con 0x7fa34c102570 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.997+0000 7fa35584f740 1 Processor -- start 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.997+0000 7fa35584f740 1 -- start start 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.997+0000 7fa35584f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a3d50 0x563d1c0a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.997+0000 7fa35584f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.997+0000 7fa35584f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0ad580 0x563d1c0af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.997+0000 7fa35584f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563d1c0062f0 con 0x563d1c0a27e0 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.997+0000 7fa35584f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563d1bfadcf0 con 0x563d1c0a3d50 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.997+0000 7fa35584f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563d1c0022b0 con 0x563d1c0ad580 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa33e7fc640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3380a32c0 0x7fa3380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351212640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351212640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0a2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43988/0 (socket says 172.21.15.89:43988) 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351212640 1 -- 172.21.15.89:0/596702685 learned_addr learned my addr 172.21.15.89:0/596702685 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351a13640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3380a32c0 0x7fa3380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa350a11640 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0ad580 0x563d1c0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa33e7fc640 1 -- 172.21.15.89:0/2130816553 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa3400cf780 con 0x7fa34c102570 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351a13640 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a3d50 0x563d1c0a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351212640 1 -- 172.21.15.89:0/596702685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0ad580 msgr2=0x563d1c0af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351212640 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0ad580 0x563d1c0af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351212640 1 -- 172.21.15.89:0/596702685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a3d50 msgr2=0x563d1c0a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351212640 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a3d50 0x563d1c0a22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351212640 1 -- 172.21.15.89:0/596702685 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d1bf923a0 con 0x563d1c0a27e0 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351a13640 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a3d50 0x563d1c0a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa350a11640 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0ad580 0x563d1c0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:38.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351a13640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3380a32c0 0x7fa3380a5780 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa344009fd0 tx=0x7fa344009540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.998+0000 7fa351212640 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0a2c20 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fa348005ab0 tx=0x7fa3480040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5615c4c305fc0844 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa3377fe640 1 -- 172.21.15.89:0/596702685 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34803f070 con 0x563d1c0a27e0 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa3377fe640 1 -- 172.21.15.89:0/596702685 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa348005520 con 0x563d1c0a27e0 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa3377fe640 1 -- 172.21.15.89:0/596702685 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa348035030 con 0x563d1c0a27e0 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa35584f740 1 -- 172.21.15.89:0/596702685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 msgr2=0x563d1c0a2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa35584f740 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0a2c20 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fa348005ab0 tx=0x7fa3480040c0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa35584f740 1 -- 172.21.15.89:0/596702685 shutdown_connections 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa35584f740 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0ad580 0x563d1c0af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa35584f740 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0a2c20 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa35584f740 1 --2- 172.21.15.89:0/596702685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a3d50 0x563d1c0a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa35584f740 1 -- 172.21.15.89:0/596702685 >> 172.21.15.89:0/596702685 conn(0x563d1c069310 msgr2=0x563d1c09ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa35584f740 1 -- 172.21.15.89:0/596702685 shutdown_connections 2024-09-15T23:33:38.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:37.999+0000 7fa35584f740 1 -- 172.21.15.89:0/596702685 wait complete. 2024-09-15T23:33:38.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa35584f740 1 -- start start 2024-09-15T23:33:38.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa35584f740 1 Processor -- start 2024-09-15T23:33:38.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa35584f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0acf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa35584f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa35584f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0ad580 0x563d1c0aa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa35584f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563d1c0b0220 con 0x563d1c0a27e0 2024-09-15T23:33:38.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa35584f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563d1bfc1060 con 0x563d1c0ad580 2024-09-15T23:33:38.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa35584f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563d1c0b03a0 con 0x563d1c0a3d50 2024-09-15T23:33:38.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa35584f740 10 client.0 _ll_register_callbacks cb 0x563d1bd148c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa351212640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa351212640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0acf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43998/0 (socket says 172.21.15.89:43998) 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.000+0000 7fa351212640 1 -- 172.21.15.89:0/2619901249 learned_addr learned my addr 172.21.15.89:0/2619901249 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa350a11640 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa351a13640 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0ad580 0x563d1c0aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa351212640 1 -- 172.21.15.89:0/2619901249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 msgr2=0x563d1c0aa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa351212640 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0aa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa351212640 1 -- 172.21.15.89:0/2619901249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0ad580 msgr2=0x563d1c0aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa351212640 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0ad580 0x563d1c0aa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa351212640 1 -- 172.21.15.89:0/2619901249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d1bd773c0 con 0x563d1c0a27e0 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa351212640 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0acf10 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa348006010 tx=0x7fa348004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c608db15c6aeb056 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:38.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa334ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa334ff9640 1 -- 172.21.15.89:0/2619901249 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34804c020 con 0x563d1c0a27e0 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa334ff9640 1 -- 172.21.15.89:0/2619901249 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa348004fb0 con 0x563d1c0a27e0 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa35584f740 1 -- 172.21.15.89:0/2619901249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa348005520 con 0x563d1c0a27e0 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa334ff9640 1 -- 172.21.15.89:0/2619901249 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa348057050 con 0x563d1c0a27e0 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa35584f740 10 client.18615 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.001+0000 7fa35584f740 1 -- 172.21.15.89:0/2619901249 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563d1c0bbc50 con 0x563d1c0a27e0 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.002+0000 7fa334ff9640 1 -- 172.21.15.89:0/2619901249 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa348005520 con 0x563d1c0a27e0 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.002+0000 7fa334ff9640 1 -- 172.21.15.89:0/2619901249 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa348031540 con 0x563d1c0a27e0 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.002+0000 7fa35584f740 10 client.18615 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.002+0000 7fa35584f740 10 client.18615 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.002+0000 7fa35584f740 20 client.18615 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.002+0000 7fa35584f740 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x563d1c0bcfa0 0x563d1c0bf390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.002+0000 7fa35584f740 4 client.18615 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:38.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.002+0000 7fa35584f740 1 -- 172.21.15.89:0/2619901249 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d1c013860 con 0x563d1c0bcfa0 2024-09-15T23:33:38.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.002+0000 7fa350a11640 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x563d1c0bcfa0 0x563d1c0bf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa350a11640 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x563d1c0bcfa0 0x563d1c0bf390 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e5097d010fb9625c server_cookie=1b045a72898b19f6 in_seq=0 out_seq=0 2024-09-15T23:33:38.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa334ff9640 10 client.18615 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:33:38.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa334ff9640 1 -- 172.21.15.89:0/2619901249 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563d1c013860 con 0x563d1c0bcfa0 2024-09-15T23:33:38.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa334ff9640 10 client.18615 handle_command_reply: tid=0 2024-09-15T23:33:38.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa35584f740 1 client.18615 shutdown 2024-09-15T23:33:38.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa35584f740 1 -- 172.21.15.89:0/2619901249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 msgr2=0x563d1c0acf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:38.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa35584f740 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0acf10 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fa348006010 tx=0x7fa348004740 comp rx=0 tx=0).stop 2024-09-15T23:33:38.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa35584f740 1 -- 172.21.15.89:0/2619901249 shutdown_connections 2024-09-15T23:33:38.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa35584f740 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x563d1c0bcfa0 0x563d1c0bf390 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa35584f740 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0ad580 0x563d1c0aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa35584f740 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0aa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa334ff9640 0 client.18615 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:33:38.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa35584f740 1 --2- 172.21.15.89:0/2619901249 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0a27e0 0x563d1c0acf10 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.003+0000 7fa35584f740 1 -- 172.21.15.89:0/2619901249 >> 172.21.15.89:0/2619901249 conn(0x563d1c069310 msgr2=0x563d1c0a4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:38.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.004+0000 7fa35584f740 1 -- 172.21.15.89:0/2619901249 shutdown_connections 2024-09-15T23:33:38.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.004+0000 7fa35584f740 1 -- 172.21.15.89:0/2619901249 wait complete. 2024-09-15T23:33:38.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.004+0000 7fa35584f740 20 client.18615 trim_cache size 0 max 16384 2024-09-15T23:33:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa35584f740 1 Processor -- start 2024-09-15T23:33:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa35584f740 1 -- start start 2024-09-15T23:33:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa35584f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a27e0 0x563d1c09bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa35584f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0ae970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa35584f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c0aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa35584f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c1acc40 con 0x563d1c0ad580 2024-09-15T23:33:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa35584f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c1ae780 con 0x563d1c0a27e0 2024-09-15T23:33:38.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa35584f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c1ad310 con 0x563d1c0a3d50 2024-09-15T23:33:38.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa351a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c0aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa351a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c0aeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44006/0 (socket says 172.21.15.89:44006) 2024-09-15T23:33:38.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa351a13640 1 -- 172.21.15.89:0/2368672340 learned_addr learned my addr 172.21.15.89:0/2368672340 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:38.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa351212640 1 --2- 172.21.15.89:0/2368672340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a27e0 0x563d1c09bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa351a13640 1 -- 172.21.15.89:0/2368672340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 msgr2=0x563d1c0ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa351a13640 1 --2- 172.21.15.89:0/2368672340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa351a13640 1 -- 172.21.15.89:0/2368672340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a27e0 msgr2=0x563d1c09bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa351a13640 1 --2- 172.21.15.89:0/2368672340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a27e0 0x563d1c09bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.017+0000 7fa351a13640 1 -- 172.21.15.89:0/2368672340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d1c0afdb0 con 0x563d1c0ad580 2024-09-15T23:33:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa351a13640 1 --2- 172.21.15.89:0/2368672340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c0aeeb0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fa34400a450 tx=0x7fa3440093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dbc88ad9272f9a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa336ffd640 1 -- 172.21.15.89:0/2368672340 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa344013040 con 0x563d1c0ad580 2024-09-15T23:33:38.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa336ffd640 1 -- 172.21.15.89:0/2368672340 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa344028070 con 0x563d1c0ad580 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa336ffd640 1 -- 172.21.15.89:0/2368672340 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa344023950 con 0x563d1c0ad580 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa35584f740 1 -- 172.21.15.89:0/2368672340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 msgr2=0x563d1c0aeeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa35584f740 1 --2- 172.21.15.89:0/2368672340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c0aeeb0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fa34400a450 tx=0x7fa3440093d0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa35584f740 1 -- 172.21.15.89:0/2368672340 shutdown_connections 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa35584f740 1 --2- 172.21.15.89:0/2368672340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c0aeeb0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa35584f740 1 --2- 172.21.15.89:0/2368672340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa35584f740 1 --2- 172.21.15.89:0/2368672340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a27e0 0x563d1c09bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa35584f740 1 -- 172.21.15.89:0/2368672340 >> 172.21.15.89:0/2368672340 conn(0x563d1c069310 msgr2=0x563d1c09b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa35584f740 1 -- 172.21.15.89:0/2368672340 shutdown_connections 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.018+0000 7fa35584f740 1 -- 172.21.15.89:0/2368672340 wait complete. 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.019+0000 7fa35584f740 1 -- start start 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.019+0000 7fa35584f740 1 Processor -- start 2024-09-15T23:33:38.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.019+0000 7fa35584f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a27e0 0x563d1c0be4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.019+0000 7fa35584f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.019+0000 7fa35584f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c1268d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.019+0000 7fa35584f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c1ae0b0 con 0x563d1c0ad580 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.019+0000 7fa35584f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c1ad9e0 con 0x563d1c0a27e0 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.019+0000 7fa35584f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c1aac20 con 0x563d1c0a3d50 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.019+0000 7fa35584f740 10 client.0 _ll_register_callbacks cb 0x7fa3400056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.019+0000 7fa351a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c1268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa351a13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c1268d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44022/0 (socket says 172.21.15.89:44022) 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa351a13640 1 -- 172.21.15.89:0/180625011 learned_addr learned my addr 172.21.15.89:0/180625011 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa351212640 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a27e0 0x563d1c0be4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa350a11640 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa351a13640 1 -- 172.21.15.89:0/180625011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 msgr2=0x563d1c0be9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:38.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa351a13640 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0be9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa351a13640 1 -- 172.21.15.89:0/180625011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a27e0 msgr2=0x563d1c0be4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa351a13640 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a27e0 0x563d1c0be4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa351a13640 1 -- 172.21.15.89:0/180625011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d1c0bf2f0 con 0x563d1c0ad580 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa351a13640 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c1268d0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fa344009440 tx=0x7fa344024d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864a2046bff6d10 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa334ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa334ff9640 1 -- 172.21.15.89:0/180625011 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa344030070 con 0x563d1c0ad580 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa334ff9640 1 -- 172.21.15.89:0/180625011 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa34402c820 con 0x563d1c0ad580 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa35584f740 1 -- 172.21.15.89:0/180625011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa344028070 con 0x563d1c0ad580 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa334ff9640 1 -- 172.21.15.89:0/180625011 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa34403c050 con 0x563d1c0ad580 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa35584f740 10 client.18621 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.020+0000 7fa35584f740 1 -- 172.21.15.89:0/180625011 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563d1c127f40 con 0x563d1c0ad580 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa334ff9640 1 -- 172.21.15.89:0/180625011 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa344028070 con 0x563d1c0ad580 2024-09-15T23:33:38.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa334ff9640 1 -- 172.21.15.89:0/180625011 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa3440185d0 con 0x563d1c0ad580 2024-09-15T23:33:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa35584f740 10 client.18621 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa35584f740 10 client.18621 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:33:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa35584f740 20 client.18621 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa35584f740 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x563d1c1285b0 0x563d1c12a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa35584f740 4 client.18621 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:33:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa35584f740 1 -- 172.21.15.89:0/180625011 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563d1c00dd70 con 0x563d1c1285b0 2024-09-15T23:33:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa351212640 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x563d1c1285b0 0x563d1c12a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa351212640 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x563d1c1285b0 0x563d1c12a9d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=357b48878bc107d3 server_cookie=ff45f1d194fa34b0 in_seq=0 out_seq=0 2024-09-15T23:33:38.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa334ff9640 10 client.18621 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:33:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa334ff9640 1 -- 172.21.15.89:0/180625011 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563d1c00dd70 con 0x563d1c1285b0 2024-09-15T23:33:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.021+0000 7fa334ff9640 10 client.18621 handle_command_reply: tid=0 2024-09-15T23:33:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 client.18621 shutdown 2024-09-15T23:33:38.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 -- 172.21.15.89:0/180625011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 msgr2=0x563d1c1268d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c1268d0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fa344009440 tx=0x7fa344024d30 comp rx=0 tx=0).stop 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 -- 172.21.15.89:0/180625011 shutdown_connections 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x563d1c1285b0 0x563d1c12a9d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563d1c0ad580 0x563d1c1268d0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa334ff9640 0 client.18621 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563d1c0a3d50 0x563d1c0be9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 --2- 172.21.15.89:0/180625011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563d1c0a27e0 0x563d1c0be4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 -- 172.21.15.89:0/180625011 >> 172.21.15.89:0/180625011 conn(0x563d1c069310 msgr2=0x563d1c09bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 -- 172.21.15.89:0/180625011 shutdown_connections 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 1 -- 172.21.15.89:0/180625011 wait complete. 2024-09-15T23:33:38.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa35584f740 20 client.18621 trim_cache size 0 max 16384 2024-09-15T23:33:38.029 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:33:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.022+0000 7fa353476640 1 -- 172.21.15.89:0/2130816553 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3380a32c0 msgr2=0x7fa3380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3380a32c0 0x7fa3380a5780 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa344009fd0 tx=0x7fa344009540 comp rx=0 tx=0).stop 2024-09-15T23:33:38.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 -- 172.21.15.89:0/2130816553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa34c102570 msgr2=0x7fa34c197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa34c102570 0x7fa34c197970 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fa34000d770 tx=0x7fa34000dc30 comp rx=0 tx=0).stop 2024-09-15T23:33:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 -- 172.21.15.89:0/2130816553 shutdown_connections 2024-09-15T23:33:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3380a32c0 0x7fa3380a5780 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa34c102ef0 0x7fa34c19bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa34c102570 0x7fa34c197970 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 --2- 172.21.15.89:0/2130816553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa34c100e20 0x7fa34c197430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 -- 172.21.15.89:0/2130816553 >> 172.21.15.89:0/2130816553 conn(0x7fa34c0fc7d0 msgr2=0x7fa34c0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 -- 172.21.15.89:0/2130816553 shutdown_connections 2024-09-15T23:33:38.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:38.023+0000 7fa353476640 1 -- 172.21.15.89:0/2130816553 wait complete. 2024-09-15T23:33:38.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:38 smithi089 ceph-mon[30933]: pgmap v638: 129 pgs: 129 active+clean; 748 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-09-15T23:33:38.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:38 smithi120 ceph-mon[35250]: pgmap v638: 129 pgs: 129 active+clean; 748 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-09-15T23:33:38.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:38 smithi174 ceph-mon[35586]: pgmap v638: 129 pgs: 129 active+clean; 748 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 KiB/s wr, 16 op/s 2024-09-15T23:33:38.938 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:33:39.192 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.673+0000 7f49f2aad640 1 -- 172.21.15.89:0/3275019626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49ec102570 msgr2=0x7f49ec1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.673+0000 7f49f2aad640 1 --2- 172.21.15.89:0/3275019626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49ec102570 0x7f49ec1029b0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009900 tx=0x7f49d802a480 comp rx=0 tx=0).stop 2024-09-15T23:33:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.673+0000 7f49f2aad640 1 -- 172.21.15.89:0/3275019626 shutdown_connections 2024-09-15T23:33:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.673+0000 7f49f2aad640 1 --2- 172.21.15.89:0/3275019626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ec102ef0 0x7f49ec10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.673+0000 7f49f2aad640 1 --2- 172.21.15.89:0/3275019626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49ec102570 0x7f49ec1029b0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.673+0000 7f49f2aad640 1 --2- 172.21.15.89:0/3275019626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec100e20 0x7f49ec101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.673+0000 7f49f2aad640 1 -- 172.21.15.89:0/3275019626 >> 172.21.15.89:0/3275019626 conn(0x7f49ec0fc7d0 msgr2=0x7f49ec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.673+0000 7f49f2aad640 1 -- 172.21.15.89:0/3275019626 shutdown_connections 2024-09-15T23:33:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.673+0000 7f49f2aad640 1 -- 172.21.15.89:0/3275019626 wait complete. 2024-09-15T23:33:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f2aad640 1 Processor -- start 2024-09-15T23:33:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f2aad640 1 -- start start 2024-09-15T23:33:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f2aad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ec100e20 0x7f49ec199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f2aad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec102570 0x7f49ec199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f2aad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49ec102ef0 0x7f49ec1a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f2aad640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec110f80 con 0x7f49ec102ef0 2024-09-15T23:33:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f2aad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec110e00 con 0x7f49ec102570 2024-09-15T23:33:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f2aad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec111100 con 0x7f49ec100e20 2024-09-15T23:33:39.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f104a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49ec102ef0 0x7f49ec1a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f104a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49ec102ef0 0x7f49ec1a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44030/0 (socket says 172.21.15.89:44030) 2024-09-15T23:33:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49f104a640 1 -- 172.21.15.89:0/2978836152 learned_addr learned my addr 172.21.15.89:0/2978836152 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.674+0000 7f49ebfff640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec102570 0x7f49ec199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49f0849640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ec100e20 0x7f49ec199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49f104a640 1 -- 172.21.15.89:0/2978836152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ec100e20 msgr2=0x7f49ec199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49f104a640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ec100e20 0x7f49ec199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49f104a640 1 -- 172.21.15.89:0/2978836152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec102570 msgr2=0x7f49ec199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49f104a640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec102570 0x7f49ec199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49f104a640 1 -- 172.21.15.89:0/2978836152 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec1a4910 con 0x7f49ec102ef0 2024-09-15T23:33:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49f104a640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49ec102ef0 0x7f49ec1a43d0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f49e400eaa0 tx=0x7f49e400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49e9ffb640 1 -- 172.21.15.89:0/2978836152 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e400cdb0 con 0x7f49ec102ef0 2024-09-15T23:33:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49e9ffb640 1 -- 172.21.15.89:0/2978836152 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49e4005020 con 0x7f49ec102ef0 2024-09-15T23:33:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49f2aad640 1 -- 172.21.15.89:0/2978836152 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec1a4ae0 con 0x7f49ec102ef0 2024-09-15T23:33:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49e9ffb640 1 -- 172.21.15.89:0/2978836152 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e40116b0 con 0x7f49ec102ef0 2024-09-15T23:33:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.675+0000 7f49f2aad640 1 -- 172.21.15.89:0/2978836152 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec1a4fa0 con 0x7f49ec102ef0 2024-09-15T23:33:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.677+0000 7f49e9ffb640 1 -- 172.21.15.89:0/2978836152 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f49e4005560 con 0x7f49ec102ef0 2024-09-15T23:33:39.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.678+0000 7f49e9ffb640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49dc0a3240 0x7f49dc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.678+0000 7f49e9ffb640 1 -- 172.21.15.89:0/2978836152 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f49e40cfee0 con 0x7f49ec102ef0 2024-09-15T23:33:39.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.678+0000 7f49f0849640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49dc0a3240 0x7f49dc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.678+0000 7f49f4e86740 1 Processor -- start 2024-09-15T23:33:39.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.678+0000 7f49f0849640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49dc0a3240 0x7f49dc0a5700 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f49e0005fd0 tx=0x7f49e0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49f4e86740 1 -- start start 2024-09-15T23:33:39.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f38b40 0x558170f37090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f37a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 0x558170f44760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49f4e86740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558170e9d2f0 con 0x558170f375d0 2024-09-15T23:33:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49f4e86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558170e44cf0 con 0x558170f38b40 2024-09-15T23:33:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49f4e86740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558170e992b0 con 0x558170f42370 2024-09-15T23:33:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f37a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f37a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44042/0 (socket says 172.21.15.89:44042) 2024-09-15T23:33:39.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49ebfff640 1 -- 172.21.15.89:0/415956426 learned_addr learned my addr 172.21.15.89:0/415956426 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49f104a640 1 --2- 172.21.15.89:0/415956426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 0x558170f44760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49f0849640 1 --2- 172.21.15.89:0/415956426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f38b40 0x558170f37090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49ebfff640 1 -- 172.21.15.89:0/415956426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 msgr2=0x558170f44760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49ebfff640 1 --2- 172.21.15.89:0/415956426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 0x558170f44760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49ebfff640 1 -- 172.21.15.89:0/415956426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f38b40 msgr2=0x558170f37090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49ebfff640 1 --2- 172.21.15.89:0/415956426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f38b40 0x558170f37090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49ebfff640 1 -- 172.21.15.89:0/415956426 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558170e293a0 con 0x558170f375d0 2024-09-15T23:33:39.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49f104a640 1 --2- 172.21.15.89:0/415956426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 0x558170f44760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:39.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.679+0000 7f49ebfff640 1 --2- 172.21.15.89:0/415956426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f37a10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f49d802a9a0 tx=0x7f49d80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97f4841a0c82914 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49d6ffd640 1 -- 172.21.15.89:0/415956426 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8035030 con 0x558170f375d0 2024-09-15T23:33:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49d6ffd640 1 -- 172.21.15.89:0/415956426 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d8040070 con 0x558170f375d0 2024-09-15T23:33:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49d6ffd640 1 -- 172.21.15.89:0/415956426 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d80319d0 con 0x558170f375d0 2024-09-15T23:33:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49f4e86740 1 -- 172.21.15.89:0/415956426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 msgr2=0x558170f37a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49f4e86740 1 --2- 172.21.15.89:0/415956426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f37a10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f49d802a9a0 tx=0x7f49d80040c0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49f4e86740 1 -- 172.21.15.89:0/415956426 shutdown_connections 2024-09-15T23:33:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49f4e86740 1 --2- 172.21.15.89:0/415956426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 0x558170f44760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49f4e86740 1 --2- 172.21.15.89:0/415956426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f37a10 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49f4e86740 1 --2- 172.21.15.89:0/415956426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f38b40 0x558170f37090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49f4e86740 1 -- 172.21.15.89:0/415956426 >> 172.21.15.89:0/415956426 conn(0x558170f00310 msgr2=0x558170f33c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49f4e86740 1 -- 172.21.15.89:0/415956426 shutdown_connections 2024-09-15T23:33:39.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.680+0000 7f49f4e86740 1 -- 172.21.15.89:0/415956426 wait complete. 2024-09-15T23:33:39.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f4e86740 1 -- start start 2024-09-15T23:33:39.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f4e86740 1 Processor -- start 2024-09-15T23:33:39.702 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f41d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f38b40 0x558170f3f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f42370 0x558170f3f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f4e86740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558170f45010 con 0x558170f375d0 2024-09-15T23:33:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f4e86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558170e58060 con 0x558170f42370 2024-09-15T23:33:39.703 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f4e86740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558170f45190 con 0x558170f38b40 2024-09-15T23:33:39.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f4e86740 10 client.0 _ll_register_callbacks cb 0x558170bab8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:39.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f0849640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f41d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f0849640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f41d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44044/0 (socket says 172.21.15.89:44044) 2024-09-15T23:33:39.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.681+0000 7f49f0849640 1 -- 172.21.15.89:0/3764952371 learned_addr learned my addr 172.21.15.89:0/3764952371 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:39.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49f104a640 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f42370 0x558170f3f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49f0849640 1 -- 172.21.15.89:0/3764952371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f38b40 msgr2=0x558170f3f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49f0849640 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f38b40 0x558170f3f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49f0849640 1 -- 172.21.15.89:0/3764952371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f42370 msgr2=0x558170f3f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49f0849640 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f42370 0x558170f3f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49f0849640 1 -- 172.21.15.89:0/3764952371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558170c0e3c0 con 0x558170f375d0 2024-09-15T23:33:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49f0849640 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f41d70 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f49e0011490 tx=0x7f49e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd592ce79dc5269a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49cbfff640 1 -- 172.21.15.89:0/3764952371 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e001f840 con 0x558170f375d0 2024-09-15T23:33:39.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49cbfff640 1 -- 172.21.15.89:0/3764952371 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f49e001f9e0 con 0x558170f375d0 2024-09-15T23:33:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49f4e86740 1 -- 172.21.15.89:0/3764952371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49d8040070 con 0x558170f375d0 2024-09-15T23:33:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49cbfff640 1 -- 172.21.15.89:0/3764952371 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f49e0030050 con 0x558170f375d0 2024-09-15T23:33:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49f4e86740 10 client.18633 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:39.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49f4e86740 1 -- 172.21.15.89:0/3764952371 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558170f50b10 con 0x558170f375d0 2024-09-15T23:33:39.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.682+0000 7f49cbfff640 1 -- 172.21.15.89:0/3764952371 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d8040070 con 0x558170f375d0 2024-09-15T23:33:39.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.683+0000 7f49cbfff640 1 -- 172.21.15.89:0/3764952371 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f49e001e3c0 con 0x558170f375d0 2024-09-15T23:33:39.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.683+0000 7f49f4e86740 10 client.18633 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:39.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.683+0000 7f49f4e86740 10 client.18633 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:33:39.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.683+0000 7f49f4e86740 20 client.18633 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:39.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.683+0000 7f49f4e86740 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x558170f52020 0x558170f544b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.683+0000 7f49f4e86740 4 client.18633 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:39.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.683+0000 7f49f4e86740 1 -- 172.21.15.89:0/3764952371 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558170eaa860 con 0x558170f52020 2024-09-15T23:33:39.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.683+0000 7f49ebfff640 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x558170f52020 0x558170f544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.683+0000 7f49ebfff640 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x558170f52020 0x558170f544b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aca37da890924a7b server_cookie=af870e175942da88 in_seq=0 out_seq=0 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.683+0000 7f49cbfff640 10 client.18633 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49cbfff640 1 -- 172.21.15.89:0/3764952371 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558170eaa860 con 0x558170f52020 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49cbfff640 10 client.18633 handle_command_reply: tid=0 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 client.18633 shutdown 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 -- 172.21.15.89:0/3764952371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 msgr2=0x558170f41d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f41d70 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f49e0011490 tx=0x7f49e00046c0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 -- 172.21.15.89:0/3764952371 shutdown_connections 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x558170f52020 0x558170f544b0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f42370 0x558170f3f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f38b40 0x558170f3f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49cbfff640 0 client.18633 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:33:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 --2- 172.21.15.89:0/3764952371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f375d0 0x558170f41d70 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 -- 172.21.15.89:0/3764952371 >> 172.21.15.89:0/3764952371 conn(0x558170f00310 msgr2=0x558170f39930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 -- 172.21.15.89:0/3764952371 shutdown_connections 2024-09-15T23:33:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.684+0000 7f49f4e86740 1 -- 172.21.15.89:0/3764952371 wait complete. 2024-09-15T23:33:39.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.685+0000 7f49f4e86740 20 client.18633 trim_cache size 0 max 16384 2024-09-15T23:33:39.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.697+0000 7f49f4e86740 1 Processor -- start 2024-09-15T23:33:39.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.697+0000 7f49f4e86740 1 -- start start 2024-09-15T23:33:39.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 0x558170f38300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f38b40 0x558170f4e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 0x558170f33700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49f4e86740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec1aee90 con 0x558170f38b40 2024-09-15T23:33:39.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49f4e86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec1b09d0 con 0x558170f375d0 2024-09-15T23:33:39.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49f4e86740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec1af560 con 0x558170f42370 2024-09-15T23:33:39.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f38b40 0x558170f4e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49ebfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f38b40 0x558170f4e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44054/0 (socket says 172.21.15.89:44054) 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49ebfff640 1 -- 172.21.15.89:0/628911130 learned_addr learned my addr 172.21.15.89:0/628911130 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49f0849640 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 0x558170f38300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49f104a640 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 0x558170f33700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49ebfff640 1 -- 172.21.15.89:0/628911130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 msgr2=0x558170f33700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49ebfff640 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 0x558170f33700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49ebfff640 1 -- 172.21.15.89:0/628911130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 msgr2=0x558170f38300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49ebfff640 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 0x558170f38300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49ebfff640 1 -- 172.21.15.89:0/628911130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558170f420a0 con 0x558170f38b40 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49f104a640 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 0x558170f33700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49f0849640 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 0x558170f38300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:39.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49ebfff640 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f38b40 0x558170f4e2b0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f49d80314e0 tx=0x7f49d803e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e82b4fb946760ec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49d67fc640 1 -- 172.21.15.89:0/628911130 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8053070 con 0x558170f38b40 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49d67fc640 1 -- 172.21.15.89:0/628911130 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d8040070 con 0x558170f38b40 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.698+0000 7f49d67fc640 1 -- 172.21.15.89:0/628911130 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8035040 con 0x558170f38b40 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.699+0000 7f49f4e86740 1 -- 172.21.15.89:0/628911130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f38b40 msgr2=0x558170f4e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.699+0000 7f49f4e86740 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f38b40 0x558170f4e2b0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f49d80314e0 tx=0x7f49d803e040 comp rx=0 tx=0).stop 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.699+0000 7f49f4e86740 1 -- 172.21.15.89:0/628911130 shutdown_connections 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.699+0000 7f49f4e86740 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f42370 0x558170f33700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.699+0000 7f49f4e86740 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f38b40 0x558170f4e2b0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.699+0000 7f49f4e86740 1 --2- 172.21.15.89:0/628911130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 0x558170f38300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.699+0000 7f49f4e86740 1 -- 172.21.15.89:0/628911130 >> 172.21.15.89:0/628911130 conn(0x558170f00310 msgr2=0x558170f2ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.699+0000 7f49f4e86740 1 -- 172.21.15.89:0/628911130 shutdown_connections 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.699+0000 7f49f4e86740 1 -- 172.21.15.89:0/628911130 wait complete. 2024-09-15T23:33:39.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.699+0000 7f49f4e86740 1 -- start start 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f4e86740 1 Processor -- start 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 0x558170f4a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f38b40 0x558170f45c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f4e86740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f42370 0x558170f46180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f4e86740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec1b0300 con 0x558170f42370 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f4e86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec1afc30 con 0x558170f375d0 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f4e86740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec1ace70 con 0x558170f38b40 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f4e86740 10 client.0 _ll_register_callbacks cb 0x7f49e0015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f104a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f42370 0x558170f46180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f104a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f42370 0x558170f46180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44066/0 (socket says 172.21.15.89:44066) 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f104a640 1 -- 172.21.15.89:0/462234105 learned_addr learned my addr 172.21.15.89:0/462234105 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f0849640 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 0x558170f4a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49ebfff640 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f38b40 0x558170f45c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.723 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f104a640 1 -- 172.21.15.89:0/462234105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f38b40 msgr2=0x558170f45c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f104a640 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f38b40 0x558170f45c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f104a640 1 -- 172.21.15.89:0/462234105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 msgr2=0x558170f4a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f104a640 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 0x558170f4a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.700+0000 7f49f104a640 1 -- 172.21.15.89:0/462234105 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558170f46740 con 0x558170f42370 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49f104a640 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f42370 0x558170f46180 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f49e40cd960 tx=0x7f49e409c110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a520d5778a1ba3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49cbfff640 1 -- 172.21.15.89:0/462234105 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e401f040 con 0x558170f42370 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49cbfff640 1 -- 172.21.15.89:0/462234105 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f49e40a01b0 con 0x558170f42370 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49f4e86740 1 -- 172.21.15.89:0/462234105 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49d8040070 con 0x558170f42370 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49cbfff640 1 -- 172.21.15.89:0/462234105 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f49e40051d0 con 0x558170f42370 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49f4e86740 10 client.18639 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49f4e86740 1 -- 172.21.15.89:0/462234105 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558170fbcab0 con 0x558170f42370 2024-09-15T23:33:39.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49cbfff640 1 -- 172.21.15.89:0/462234105 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d8040070 con 0x558170f42370 2024-09-15T23:33:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49cbfff640 1 -- 172.21.15.89:0/462234105 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f49e400f040 con 0x558170f42370 2024-09-15T23:33:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49f4e86740 10 client.18639 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49f4e86740 10 client.18639 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:33:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49f4e86740 20 client.18639 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49f4e86740 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x558170f52020 0x558170fbf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49f4e86740 4 client.18639 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:33:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.701+0000 7f49f4e86740 1 -- 172.21.15.89:0/462234105 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558170ea4d70 con 0x558170f52020 2024-09-15T23:33:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f0849640 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x558170f52020 0x558170fbf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f0849640 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x558170f52020 0x558170fbf1e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c7f24779c69f3334 server_cookie=f13f19fb068cbd26 in_seq=0 out_seq=0 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49cbfff640 10 client.18639 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49cbfff640 1 -- 172.21.15.89:0/462234105 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558170ea4d70 con 0x558170f52020 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49cbfff640 10 client.18639 handle_command_reply: tid=0 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f4e86740 1 client.18639 shutdown 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f4e86740 1 -- 172.21.15.89:0/462234105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f42370 msgr2=0x558170f46180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f4e86740 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f42370 0x558170f46180 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f49e40cd960 tx=0x7f49e409c110 comp rx=0 tx=0).stop 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f4e86740 1 -- 172.21.15.89:0/462234105 shutdown_connections 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f4e86740 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x558170f52020 0x558170fbf1e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f4e86740 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558170f42370 0x558170f46180 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f4e86740 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558170f38b40 0x558170f45c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f4e86740 1 --2- 172.21.15.89:0/462234105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558170f375d0 0x558170f4a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49f4e86740 1 -- 172.21.15.89:0/462234105 >> 172.21.15.89:0/462234105 conn(0x558170f00310 msgr2=0x558170f33700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.702+0000 7f49cbfff640 0 client.18639 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:33:39.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f4e86740 1 -- 172.21.15.89:0/462234105 shutdown_connections 2024-09-15T23:33:39.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f4e86740 1 -- 172.21.15.89:0/462234105 wait complete. 2024-09-15T23:33:39.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f4e86740 20 client.18639 trim_cache size 0 max 16384 2024-09-15T23:33:39.727 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:33:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 -- 172.21.15.89:0/2978836152 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49dc0a3240 msgr2=0x7f49dc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49dc0a3240 0x7f49dc0a5700 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f49e0005fd0 tx=0x7f49e0005850 comp rx=0 tx=0).stop 2024-09-15T23:33:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 -- 172.21.15.89:0/2978836152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49ec102ef0 msgr2=0x7f49ec1a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49ec102ef0 0x7f49ec1a43d0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f49e400eaa0 tx=0x7f49e400ef60 comp rx=0 tx=0).stop 2024-09-15T23:33:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 -- 172.21.15.89:0/2978836152 shutdown_connections 2024-09-15T23:33:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f49dc0a3240 0x7f49dc0a5700 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f49ec102ef0 0x7f49ec1a43d0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49ec102570 0x7f49ec199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 --2- 172.21.15.89:0/2978836152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ec100e20 0x7f49ec199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 -- 172.21.15.89:0/2978836152 >> 172.21.15.89:0/2978836152 conn(0x7f49ec0fc7d0 msgr2=0x7f49ec1068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 -- 172.21.15.89:0/2978836152 shutdown_connections 2024-09-15T23:33:39.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:39.703+0000 7f49f2aad640 1 -- 172.21.15.89:0/2978836152 wait complete. 2024-09-15T23:33:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.817+0000 7fd9737d8640 1 -- 172.21.15.89:0/1556453637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c101b60 msgr2=0x7fd96c101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.817+0000 7fd9737d8640 1 --2- 172.21.15.89:0/1556453637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c101b60 0x7fd96c101f60 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fd968009900 tx=0x7fd96802a410 comp rx=0 tx=0).stop 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.817+0000 7fd9737d8640 1 -- 172.21.15.89:0/1556453637 shutdown_connections 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 --2- 172.21.15.89:0/1556453637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c103710 0x7fd96c109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 --2- 172.21.15.89:0/1556453637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c102d50 0x7fd96c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 --2- 172.21.15.89:0/1556453637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c101b60 0x7fd96c101f60 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 -- 172.21.15.89:0/1556453637 >> 172.21.15.89:0/1556453637 conn(0x7fd96c0fd2e0 msgr2=0x7fd96c0ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 -- 172.21.15.89:0/1556453637 shutdown_connections 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 -- 172.21.15.89:0/1556453637 wait complete. 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 Processor -- start 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 -- start start 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c101b60 0x7fd96c197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c102d50 0x7fd96c198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c103710 0x7fd96c19f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c10d7c0 con 0x7fd96c103710 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c10d640 con 0x7fd96c102d50 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd9737d8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c10d940 con 0x7fd96c101b60 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd971d75640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c103710 0x7fd96c19f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd971d75640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c103710 0x7fd96c19f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44078/0 (socket says 172.21.15.89:44078) 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.818+0000 7fd971d75640 1 -- 172.21.15.89:0/2767122401 learned_addr learned my addr 172.21.15.89:0/2767122401 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd971574640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c101b60 0x7fd96c197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd970d73640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c102d50 0x7fd96c198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd971d75640 1 -- 172.21.15.89:0/2767122401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c101b60 msgr2=0x7fd96c197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd971d75640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c101b60 0x7fd96c197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd971d75640 1 -- 172.21.15.89:0/2767122401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c102d50 msgr2=0x7fd96c198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd971d75640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c102d50 0x7fd96c198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd971d75640 1 -- 172.21.15.89:0/2767122401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96c19fa00 con 0x7fd96c103710 2024-09-15T23:33:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd971d75640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c103710 0x7fd96c19f390 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd96400eaa0 tx=0x7fd96400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd9627fc640 1 -- 172.21.15.89:0/2767122401 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96400cdb0 con 0x7fd96c103710 2024-09-15T23:33:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd9627fc640 1 -- 172.21.15.89:0/2767122401 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd964005020 con 0x7fd96c103710 2024-09-15T23:33:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd9627fc640 1 -- 172.21.15.89:0/2767122401 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9640116b0 con 0x7fd96c103710 2024-09-15T23:33:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd9737d8640 1 -- 172.21.15.89:0/2767122401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96c19fcf0 con 0x7fd96c103710 2024-09-15T23:33:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.819+0000 7fd9737d8640 1 -- 172.21.15.89:0/2767122401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96c1a0190 con 0x7fd96c103710 2024-09-15T23:33:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.821+0000 7fd9627fc640 1 -- 172.21.15.89:0/2767122401 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd964005560 con 0x7fd96c103710 2024-09-15T23:33:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd976649740 1 Processor -- start 2024-09-15T23:33:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd976649740 1 -- start start 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd976649740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4adb0 0x55f078e4b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd976649740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4b6f0 0x55f078e501b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd976649740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e50780 0x55f078e52c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd976649740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f078daa2f0 con 0x55f078e4adb0 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd976649740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f078d52270 con 0x55f078e4b6f0 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd976649740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f078da62b0 con 0x55f078e50780 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd9627fc640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9580a3370 0x7fd9580a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd970d73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4adb0 0x55f078e4b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd970d73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4adb0 0x55f078e4b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44092/0 (socket says 172.21.15.89:44092) 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd970d73640 1 -- 172.21.15.89:0/3210665707 learned_addr learned my addr 172.21.15.89:0/3210665707 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd971574640 1 --2- 172.21.15.89:0/3210665707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4b6f0 0x55f078e501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd971d75640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9580a3370 0x7fd9580a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd9627fc640 1 -- 172.21.15.89:0/2767122401 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd9640cfe40 con 0x7fd96c103710 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.822+0000 7fd970d73640 1 --2- 172.21.15.89:0/3210665707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e50780 0x55f078e52c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd970d73640 1 -- 172.21.15.89:0/3210665707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e50780 msgr2=0x55f078e52c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd970d73640 1 --2- 172.21.15.89:0/3210665707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e50780 0x55f078e52c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd970d73640 1 -- 172.21.15.89:0/3210665707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4b6f0 msgr2=0x55f078e501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd970d73640 1 --2- 172.21.15.89:0/3210665707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4b6f0 0x55f078e501b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd970d73640 1 -- 172.21.15.89:0/3210665707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f078d51340 con 0x55f078e4adb0 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd971574640 1 --2- 172.21.15.89:0/3210665707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4b6f0 0x55f078e501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd970d73640 1 --2- 172.21.15.89:0/3210665707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4adb0 0x55f078e4b1b0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00bfa0 tx=0x7fd95c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2cdf922f640c3c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd971d75640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9580a3370 0x7fd9580a5830 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd96409be20 tx=0x7fd96401c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd9577fe640 1 -- 172.21.15.89:0/3210665707 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c004030 con 0x55f078e4adb0 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd9577fe640 1 -- 172.21.15.89:0/3210665707 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c0041d0 con 0x55f078e4adb0 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.823+0000 7fd9577fe640 1 -- 172.21.15.89:0/3210665707 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c0106c0 con 0x55f078e4adb0 2024-09-15T23:33:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 -- 172.21.15.89:0/3210665707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4adb0 msgr2=0x55f078e4b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 --2- 172.21.15.89:0/3210665707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4adb0 0x55f078e4b1b0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00bfa0 tx=0x7fd95c00b840 comp rx=0 tx=0).stop 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 -- 172.21.15.89:0/3210665707 shutdown_connections 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 --2- 172.21.15.89:0/3210665707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e50780 0x55f078e52c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 --2- 172.21.15.89:0/3210665707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4b6f0 0x55f078e501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 --2- 172.21.15.89:0/3210665707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4adb0 0x55f078e4b1b0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 -- 172.21.15.89:0/3210665707 >> 172.21.15.89:0/3210665707 conn(0x55f078e42010 msgr2=0x55f078e42410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 -- 172.21.15.89:0/3210665707 shutdown_connections 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 -- 172.21.15.89:0/3210665707 wait complete. 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 -- start start 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 Processor -- start 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 0x55f078e55010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e55550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e50780 0x55f078e5c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f078e53750 con 0x55f078e4b6f0 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f078d33290 con 0x55f078e4adb0 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.824+0000 7fd976649740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f078e538d0 con 0x55f078e50780 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd976649740 10 client.0 _ll_register_callbacks cb 0x55f078d64140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd971574640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e55550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd971574640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e55550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44104/0 (socket says 172.21.15.89:44104) 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd971574640 1 -- 172.21.15.89:0/3621990177 learned_addr learned my addr 172.21.15.89:0/3621990177 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd970d73640 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 0x55f078e55010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd971574640 1 -- 172.21.15.89:0/3621990177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e50780 msgr2=0x55f078e5c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd971574640 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e50780 0x55f078e5c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd971574640 1 -- 172.21.15.89:0/3621990177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 msgr2=0x55f078e55010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd971574640 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 0x55f078e55010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd971574640 1 -- 172.21.15.89:0/3621990177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f078abd210 con 0x55f078e4b6f0 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd970d73640 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 0x55f078e55010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd971574640 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e55550 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fd9680314e0 tx=0x7fd9680318e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c161a37edd0a7d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd954ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd954ff9640 1 -- 172.21.15.89:0/3621990177 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968035040 con 0x55f078e4b6f0 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd954ff9640 1 -- 172.21.15.89:0/3621990177 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd968004450 con 0x55f078e4b6f0 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd976649740 1 -- 172.21.15.89:0/3621990177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd95c0041d0 con 0x55f078e4b6f0 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd954ff9640 1 -- 172.21.15.89:0/3621990177 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd968005030 con 0x55f078e4b6f0 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd976649740 10 client.18651 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd976649740 1 -- 172.21.15.89:0/3621990177 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f078b89620 con 0x55f078e4b6f0 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.825+0000 7fd954ff9640 1 -- 172.21.15.89:0/3621990177 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c0041d0 con 0x55f078e4b6f0 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.826+0000 7fd954ff9640 1 -- 172.21.15.89:0/3621990177 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd9680035f0 con 0x55f078e4b6f0 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.826+0000 7fd976649740 10 client.18651 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.826+0000 7fd976649740 10 client.18651 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.826+0000 7fd976649740 20 client.18651 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.826+0000 7fd976649740 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f078e60ca0 0x55f078e63120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.826+0000 7fd976649740 4 client.18651 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.826+0000 7fd976649740 1 -- 172.21.15.89:0/3621990177 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f078db7a20 con 0x55f078e60ca0 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.826+0000 7fd970d73640 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f078e60ca0 0x55f078e63120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.826+0000 7fd954ff9640 10 client.18651 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.826+0000 7fd970d73640 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f078e60ca0 0x55f078e63120 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=477049489bf69855 server_cookie=5fffe27db0df5b7e in_seq=0 out_seq=0 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd954ff9640 1 -- 172.21.15.89:0/3621990177 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f078db7a20 con 0x55f078e60ca0 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd954ff9640 10 client.18651 handle_command_reply: tid=0 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd976649740 1 client.18651 shutdown 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd976649740 1 -- 172.21.15.89:0/3621990177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 msgr2=0x55f078e55550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd976649740 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e55550 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fd9680314e0 tx=0x7fd9680318e0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd976649740 1 -- 172.21.15.89:0/3621990177 shutdown_connections 2024-09-15T23:33:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd976649740 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f078e60ca0 0x55f078e63120 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd976649740 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e50780 0x55f078e5c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd976649740 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e55550 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd954ff9640 0 client.18651 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd976649740 1 --2- 172.21.15.89:0/3621990177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 0x55f078e55010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.827+0000 7fd976649740 1 -- 172.21.15.89:0/3621990177 >> 172.21.15.89:0/3621990177 conn(0x55f078e42010 msgr2=0x55f078e43820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.828+0000 7fd976649740 1 -- 172.21.15.89:0/3621990177 shutdown_connections 2024-09-15T23:33:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.828+0000 7fd976649740 1 -- 172.21.15.89:0/3621990177 wait complete. 2024-09-15T23:33:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.828+0000 7fd976649740 20 client.18651 trim_cache size 0 max 16384 2024-09-15T23:33:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd976649740 1 Processor -- start 2024-09-15T23:33:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd976649740 1 -- start start 2024-09-15T23:33:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd976649740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 0x55f078e410b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd976649740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4b6f0 0x55f078e5c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd976649740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e50780 0x55f078e43400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd976649740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1a8670 con 0x55f078e50780 2024-09-15T23:33:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd976649740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1aa1b0 con 0x55f078e4adb0 2024-09-15T23:33:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd976649740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1a8d40 con 0x55f078e4b6f0 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd970d73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e50780 0x55f078e43400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd970d73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e50780 0x55f078e43400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44112/0 (socket says 172.21.15.89:44112) 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd970d73640 1 -- 172.21.15.89:0/789569357 learned_addr learned my addr 172.21.15.89:0/789569357 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd971574640 1 --2- 172.21.15.89:0/789569357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4b6f0 0x55f078e5c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd970d73640 1 --2- 172.21.15.89:0/789569357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 0x55f078e410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd970d73640 1 -- 172.21.15.89:0/789569357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4b6f0 msgr2=0x55f078e5c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd970d73640 1 --2- 172.21.15.89:0/789569357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4b6f0 0x55f078e5c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd970d73640 1 -- 172.21.15.89:0/789569357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 msgr2=0x55f078e410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd970d73640 1 --2- 172.21.15.89:0/789569357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 0x55f078e410b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd970d73640 1 -- 172.21.15.89:0/789569357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f078e637a0 con 0x55f078e50780 2024-09-15T23:33:39.848 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.842+0000 7fd970d73640 1 --2- 172.21.15.89:0/789569357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e50780 0x55f078e43400 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00b810 tx=0x7fd95c012530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8616999253b467 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd956ffd640 1 -- 172.21.15.89:0/789569357 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c005040 con 0x55f078e50780 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd956ffd640 1 -- 172.21.15.89:0/789569357 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c0293c0 con 0x55f078e50780 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd956ffd640 1 -- 172.21.15.89:0/789569357 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c004760 con 0x55f078e50780 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd976649740 1 -- 172.21.15.89:0/789569357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e50780 msgr2=0x55f078e43400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd976649740 1 --2- 172.21.15.89:0/789569357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e50780 0x55f078e43400 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fd95c00b810 tx=0x7fd95c012530 comp rx=0 tx=0).stop 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd976649740 1 -- 172.21.15.89:0/789569357 shutdown_connections 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd976649740 1 --2- 172.21.15.89:0/789569357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e50780 0x55f078e43400 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd976649740 1 --2- 172.21.15.89:0/789569357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4b6f0 0x55f078e5c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd976649740 1 --2- 172.21.15.89:0/789569357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e4adb0 0x55f078e410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd976649740 1 -- 172.21.15.89:0/789569357 >> 172.21.15.89:0/789569357 conn(0x55f078e42010 msgr2=0x55f078e409a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd976649740 1 -- 172.21.15.89:0/789569357 shutdown_connections 2024-09-15T23:33:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd976649740 1 -- 172.21.15.89:0/789569357 wait complete. 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.843+0000 7fd976649740 1 -- start start 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd976649740 1 Processor -- start 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd976649740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4adb0 0x55f078e596a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd976649740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e59be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd976649740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e50780 0x55f078e5abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd976649740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1a9ae0 con 0x55f078e4b6f0 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd976649740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1a9410 con 0x55f078e50780 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd976649740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c1a6650 con 0x55f078e4adb0 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd976649740 10 client.0 _ll_register_callbacks cb 0x7fd968009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd971574640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e59be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd971574640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e59be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44120/0 (socket says 172.21.15.89:44120) 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd971574640 1 -- 172.21.15.89:0/3512649513 learned_addr learned my addr 172.21.15.89:0/3512649513 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd970d73640 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4adb0 0x55f078e596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd970d73640 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e50780 0x55f078e5abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd971574640 1 -- 172.21.15.89:0/3512649513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4adb0 msgr2=0x55f078e596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd971574640 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4adb0 0x55f078e596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd971574640 1 -- 172.21.15.89:0/3512649513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e50780 msgr2=0x55f078e5abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd971574640 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e50780 0x55f078e5abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd971574640 1 -- 172.21.15.89:0/3512649513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f078e5b200 con 0x55f078e4b6f0 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.844+0000 7fd970d73640 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4adb0 0x55f078e596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd971574640 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e59be0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fd96802a450 tx=0x7fd968032f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8309c3dfe0104f0f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd954ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd954ff9640 1 -- 172.21.15.89:0/3512649513 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968004450 con 0x55f078e4b6f0 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd954ff9640 1 -- 172.21.15.89:0/3512649513 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd9680051d0 con 0x55f078e4b6f0 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd976649740 1 -- 172.21.15.89:0/3512649513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd95c0293c0 con 0x55f078e4b6f0 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd954ff9640 1 -- 172.21.15.89:0/3512649513 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd968062050 con 0x55f078e4b6f0 2024-09-15T23:33:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd976649740 10 client.18657 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd976649740 1 -- 172.21.15.89:0/3512649513 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f078e5c540 con 0x55f078e4b6f0 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd954ff9640 1 -- 172.21.15.89:0/3512649513 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95c0293c0 con 0x55f078e4b6f0 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd954ff9640 1 -- 172.21.15.89:0/3512649513 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd96803d450 con 0x55f078e4b6f0 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd976649740 10 client.18657 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd976649740 10 client.18657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd976649740 20 client.18657 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd976649740 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f078ecbe40 0x55f078ece260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd976649740 4 client.18657 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.845+0000 7fd976649740 1 -- 172.21.15.89:0/3512649513 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f078db1d70 con 0x55f078ecbe40 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.846+0000 7fd970d73640 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f078ecbe40 0x55f078ece260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.846+0000 7fd970d73640 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f078ecbe40 0x55f078ece260 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=991686c4c77c2c85 server_cookie=787fbee6b39f279f in_seq=0 out_seq=0 2024-09-15T23:33:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.846+0000 7fd954ff9640 10 client.18657 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd954ff9640 1 -- 172.21.15.89:0/3512649513 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f078db1d70 con 0x55f078ecbe40 2024-09-15T23:33:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd954ff9640 10 client.18657 handle_command_reply: tid=0 2024-09-15T23:33:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 client.18657 shutdown 2024-09-15T23:33:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 -- 172.21.15.89:0/3512649513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 msgr2=0x55f078e59be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e59be0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fd96802a450 tx=0x7fd968032f70 comp rx=0 tx=0).stop 2024-09-15T23:33:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 -- 172.21.15.89:0/3512649513 shutdown_connections 2024-09-15T23:33:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f078ecbe40 0x55f078ece260 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f078e50780 0x55f078e5abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd954ff9640 0 client.18657 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f078e4b6f0 0x55f078e59be0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 --2- 172.21.15.89:0/3512649513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f078e4adb0 0x55f078e596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 -- 172.21.15.89:0/3512649513 >> 172.21.15.89:0/3512649513 conn(0x55f078e42010 msgr2=0x55f078e411c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.859 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 -- 172.21.15.89:0/3512649513 shutdown_connections 2024-09-15T23:33:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 1 -- 172.21.15.89:0/3512649513 wait complete. 2024-09-15T23:33:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.857+0000 7fd976649740 20 client.18657 trim_cache size 0 max 16384 2024-09-15T23:33:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:33:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:33:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:33:39.860 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:33:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 -- 172.21.15.89:0/2767122401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9580a3370 msgr2=0x7fd9580a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9580a3370 0x7fd9580a5830 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd96409be20 tx=0x7fd96401c040 comp rx=0 tx=0).stop 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 -- 172.21.15.89:0/2767122401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c103710 msgr2=0x7fd96c19f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c103710 0x7fd96c19f390 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fd96400eaa0 tx=0x7fd96400ef60 comp rx=0 tx=0).stop 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 -- 172.21.15.89:0/2767122401 shutdown_connections 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9580a3370 0x7fd9580a5830 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96c103710 0x7fd96c19f390 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd96c102d50 0x7fd96c198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 --2- 172.21.15.89:0/2767122401 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd96c101b60 0x7fd96c197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 -- 172.21.15.89:0/2767122401 >> 172.21.15.89:0/2767122401 conn(0x7fd96c0fd2e0 msgr2=0x7fd96c1060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 -- 172.21.15.89:0/2767122401 shutdown_connections 2024-09-15T23:33:39.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:39.858+0000 7fd9737d8640 1 -- 172.21.15.89:0/2767122401 wait complete. 2024-09-15T23:33:40.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:40 smithi089 ceph-mon[30933]: pgmap v639: 129 pgs: 129 active+clean; 748 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-09-15T23:33:40.620 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:33:40.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:40 smithi120 ceph-mon[35250]: pgmap v639: 129 pgs: 129 active+clean; 748 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-09-15T23:33:40.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:40 smithi174 ceph-mon[35586]: pgmap v639: 129 pgs: 129 active+clean; 748 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-09-15T23:33:40.866 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:41.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.376+0000 7f07f199a640 1 -- 172.21.15.89:0/1345886131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07ec103660 msgr2=0x7f07ec10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:41.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.376+0000 7f07f199a640 1 --2- 172.21.15.89:0/1345886131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07ec103660 0x7f07ec10bb50 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f07e0009900 tx=0x7f07e002a480 comp rx=0 tx=0).stop 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.376+0000 7f07f199a640 1 -- 172.21.15.89:0/1345886131 shutdown_connections 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.376+0000 7f07f199a640 1 --2- 172.21.15.89:0/1345886131 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ec10c170 0x7f07ec10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.376+0000 7f07f199a640 1 --2- 172.21.15.89:0/1345886131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07ec103660 0x7f07ec10bb50 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.376+0000 7f07f199a640 1 --2- 172.21.15.89:0/1345886131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07ec102d40 0x7f07ec103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.376+0000 7f07f199a640 1 -- 172.21.15.89:0/1345886131 >> 172.21.15.89:0/1345886131 conn(0x7f07ec0fc7a0 msgr2=0x7f07ec0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.376+0000 7f07f199a640 1 -- 172.21.15.89:0/1345886131 shutdown_connections 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.376+0000 7f07f199a640 1 -- 172.21.15.89:0/1345886131 wait complete. 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07f199a640 1 Processor -- start 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07f199a640 1 -- start start 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07f199a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ec102d40 0x7f07ec198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07f199a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07ec103660 0x7f07ec198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07f199a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07ec10c170 0x7f07ec19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07f199a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec112030 con 0x7f07ec10c170 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07f199a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec111eb0 con 0x7f07ec102d40 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07f199a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec1121b0 con 0x7f07ec103660 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07ec10c170 0x7f07ec19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07ec10c170 0x7f07ec19cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44124/0 (socket says 172.21.15.89:44124) 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.377+0000 7f07eb7fe640 1 -- 172.21.15.89:0/206219583 learned_addr learned my addr 172.21.15.89:0/206219583 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07eaffd640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ec102d40 0x7f07ec198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07ea7fc640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07ec103660 0x7f07ec198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07eb7fe640 1 -- 172.21.15.89:0/206219583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07ec103660 msgr2=0x7f07ec198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07ec103660 0x7f07ec198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07eb7fe640 1 -- 172.21.15.89:0/206219583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ec102d40 msgr2=0x7f07ec198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ec102d40 0x7f07ec198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07eb7fe640 1 -- 172.21.15.89:0/206219583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ec19d640 con 0x7f07ec10c170 2024-09-15T23:33:41.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07ec10c170 0x7f07ec19cf60 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f07e400b970 tx=0x7f07e400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07f0998640 1 -- 172.21.15.89:0/206219583 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4019070 con 0x7f07ec10c170 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07f199a640 1 -- 172.21.15.89:0/206219583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ec19d930 con 0x7f07ec10c170 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07f0998640 1 -- 172.21.15.89:0/206219583 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07e4014440 con 0x7f07ec10c170 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07f0998640 1 -- 172.21.15.89:0/206219583 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e40075d0 con 0x7f07ec10c170 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.378+0000 7f07f199a640 1 -- 172.21.15.89:0/206219583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ec1a5210 con 0x7f07ec10c170 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.380+0000 7f07f0998640 1 -- 172.21.15.89:0/206219583 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f07e401c070 con 0x7f07ec10c170 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07f0998640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07d40a3240 0x7f07d40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07f3d73740 1 Processor -- start 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07f3d73740 1 -- start start 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07eaffd640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07d40a3240 0x7f07d40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07f0998640 1 -- 172.21.15.89:0/206219583 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f07e40cf370 con 0x7f07ec10c170 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa5b40 0x56026bfa4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa45d0 0x56026bfa4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07f3d73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56026bf0a2f0 con 0x56026bfb3780 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07f3d73740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56026beb1cf0 con 0x56026bfa45d0 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07f3d73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56026bf062b0 con 0x56026bfa5b40 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44128/0 (socket says 172.21.15.89:44128) 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07eb7fe640 1 -- 172.21.15.89:0/259632477 learned_addr learned my addr 172.21.15.89:0/259632477 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.381+0000 7f07eaffd640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07d40a3240 0x7f07d40a5700 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009990 tx=0x7f07dc009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07ea7fc640 1 --2- 172.21.15.89:0/259632477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa45d0 0x56026bfa4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07eb7fe640 1 -- 172.21.15.89:0/259632477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa5b40 msgr2=0x56026bfa4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07eaffd640 1 --2- 172.21.15.89:0/259632477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa5b40 0x56026bfa4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/259632477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa5b40 0x56026bfa4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07eb7fe640 1 -- 172.21.15.89:0/259632477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa45d0 msgr2=0x56026bfa4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/259632477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa45d0 0x56026bfa4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07eb7fe640 1 -- 172.21.15.89:0/259632477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56026be963a0 con 0x56026bfb3780 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/259632477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb5b70 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f07e409b2a0 tx=0x7f07e409bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b866b085ddcba0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07d1ffb640 1 -- 172.21.15.89:0/259632477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e409f950 con 0x56026bfb3780 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07d1ffb640 1 -- 172.21.15.89:0/259632477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07e40a4210 con 0x56026bfb3780 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.382+0000 7f07d1ffb640 1 -- 172.21.15.89:0/259632477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e40145f0 con 0x56026bfb3780 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 -- 172.21.15.89:0/259632477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 msgr2=0x56026bfb5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 --2- 172.21.15.89:0/259632477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb5b70 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f07e409b2a0 tx=0x7f07e409bca0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 -- 172.21.15.89:0/259632477 shutdown_connections 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 --2- 172.21.15.89:0/259632477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb5b70 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 --2- 172.21.15.89:0/259632477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa45d0 0x56026bfa4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 --2- 172.21.15.89:0/259632477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa5b40 0x56026bfa4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 -- 172.21.15.89:0/259632477 >> 172.21.15.89:0/259632477 conn(0x56026bf6d310 msgr2=0x56026bfa0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:41.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 -- 172.21.15.89:0/259632477 shutdown_connections 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 -- 172.21.15.89:0/259632477 wait complete. 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 -- start start 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.383+0000 7f07f3d73740 1 Processor -- start 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa45d0 0x56026bfb31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfa5b40 0x56026bfb0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfb3780 0x56026bfb0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07f3d73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56026bfb6480 con 0x56026bfa5b40 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07f3d73740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56026bec5060 con 0x56026bfb3780 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07f3d73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56026bfb6600 con 0x56026bfa45d0 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07f3d73740 10 client.0 _ll_register_callbacks cb 0x56026bc188c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07ea7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfa5b40 0x56026bfb0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07ea7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfa5b40 0x56026bfb0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44142/0 (socket says 172.21.15.89:44142) 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07ea7fc640 1 -- 172.21.15.89:0/2547122176 learned_addr learned my addr 172.21.15.89:0/2547122176 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:41.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfb3780 0x56026bfb0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07eaffd640 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa45d0 0x56026bfb31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07ea7fc640 1 -- 172.21.15.89:0/2547122176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa45d0 msgr2=0x56026bfb31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07ea7fc640 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa45d0 0x56026bfb31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07ea7fc640 1 -- 172.21.15.89:0/2547122176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfb3780 msgr2=0x56026bfb0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07ea7fc640 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfb3780 0x56026bfb0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07ea7fc640 1 -- 172.21.15.89:0/2547122176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56026bc7b3c0 con 0x56026bfa5b40 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfb3780 0x56026bfb0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07ea7fc640 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfa5b40 0x56026bfb0460 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f07e0031550 tx=0x7f07e0031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c2e4eec57e3b20 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07caffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07caffd640 1 -- 172.21.15.89:0/2547122176 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e0035040 con 0x56026bfa5b40 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07caffd640 1 -- 172.21.15.89:0/2547122176 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f07e0005020 con 0x56026bfa5b40 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.384+0000 7f07f3d73740 1 -- 172.21.15.89:0/2547122176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07e40a4210 con 0x56026bfa5b40 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07caffd640 1 -- 172.21.15.89:0/2547122176 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f07e004e050 con 0x56026bfa5b40 2024-09-15T23:33:41.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07f3d73740 10 client.18669 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07f3d73740 1 -- 172.21.15.89:0/2547122176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56026bfc1f60 con 0x56026bfa5b40 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07caffd640 1 -- 172.21.15.89:0/2547122176 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07e40a4210 con 0x56026bfa5b40 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07caffd640 1 -- 172.21.15.89:0/2547122176 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f07e00035f0 con 0x56026bfa5b40 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07f3d73740 10 client.18669 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07f3d73740 10 client.18669 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07f3d73740 20 client.18669 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07f3d73740 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56026bfc3420 0x56026bfc5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07f3d73740 4 client.18669 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07f3d73740 1 -- 172.21.15.89:0/2547122176 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56026bf17860 con 0x56026bfc3420 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.385+0000 7f07eaffd640 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56026bfc3420 0x56026bfc5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.386+0000 7f07eaffd640 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56026bfc3420 0x56026bfc5860 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=673cffad0e792126 server_cookie=83eeee8f67267535 in_seq=0 out_seq=0 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.386+0000 7f07caffd640 10 client.18669 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:33:41.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.386+0000 7f07caffd640 1 -- 172.21.15.89:0/2547122176 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56026bf17860 con 0x56026bfc3420 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.386+0000 7f07caffd640 10 client.18669 handle_command_reply: tid=0 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.386+0000 7f07f3d73740 1 client.18669 shutdown 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 1 -- 172.21.15.89:0/2547122176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfa5b40 msgr2=0x56026bfb0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfa5b40 0x56026bfb0460 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f07e0031550 tx=0x7f07e0031b60 comp rx=0 tx=0).stop 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 1 -- 172.21.15.89:0/2547122176 shutdown_connections 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfb3780 0x56026bfb0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfa5b40 0x56026bfb0460 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56026bfc3420 0x56026bfc5860 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 1 --2- 172.21.15.89:0/2547122176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa45d0 0x56026bfb31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07caffd640 0 client.18669 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 1 -- 172.21.15.89:0/2547122176 >> 172.21.15.89:0/2547122176 conn(0x56026bf6d310 msgr2=0x56026bfa6990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 1 -- 172.21.15.89:0/2547122176 shutdown_connections 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 1 -- 172.21.15.89:0/2547122176 wait complete. 2024-09-15T23:33:41.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.387+0000 7f07f3d73740 20 client.18669 trim_cache size 0 max 16384 2024-09-15T23:33:41.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07f3d73740 1 Processor -- start 2024-09-15T23:33:41.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07f3d73740 1 -- start start 2024-09-15T23:33:41.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa45d0 0x56026bfaf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa5b40 0x56026bfcb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfcb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07f3d73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec1acd50 con 0x56026bfb3780 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07f3d73740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec1ae890 con 0x56026bfa45d0 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07f3d73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec1ad420 con 0x56026bfa5b40 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfcb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfcb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44158/0 (socket says 172.21.15.89:44158) 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.400+0000 7f07eb7fe640 1 -- 172.21.15.89:0/2576503998 learned_addr learned my addr 172.21.15.89:0/2576503998 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07eaffd640 1 --2- 172.21.15.89:0/2576503998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa45d0 0x56026bfaf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07ea7fc640 1 --2- 172.21.15.89:0/2576503998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa5b40 0x56026bfcb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07eb7fe640 1 -- 172.21.15.89:0/2576503998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa5b40 msgr2=0x56026bfcb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/2576503998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa5b40 0x56026bfcb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07eb7fe640 1 -- 172.21.15.89:0/2576503998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa45d0 msgr2=0x56026bfaf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/2576503998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa45d0 0x56026bfaf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07eb7fe640 1 -- 172.21.15.89:0/2576503998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56026bfa1960 con 0x56026bfb3780 2024-09-15T23:33:41.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/2576503998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfcb670 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f07e400bfd0 tx=0x7f07e409b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf09027174f5843 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07d17fa640 1 -- 172.21.15.89:0/2576503998 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e40a4210 con 0x56026bfb3780 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07d17fa640 1 -- 172.21.15.89:0/2576503998 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07e409f950 con 0x56026bfb3780 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.401+0000 7f07d17fa640 1 -- 172.21.15.89:0/2576503998 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e40145f0 con 0x56026bfb3780 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 -- 172.21.15.89:0/2576503998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 msgr2=0x56026bfcb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 --2- 172.21.15.89:0/2576503998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfcb670 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f07e400bfd0 tx=0x7f07e409b460 comp rx=0 tx=0).stop 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 -- 172.21.15.89:0/2576503998 shutdown_connections 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 --2- 172.21.15.89:0/2576503998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfcb670 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 --2- 172.21.15.89:0/2576503998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa5b40 0x56026bfcb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 --2- 172.21.15.89:0/2576503998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa45d0 0x56026bfaf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 -- 172.21.15.89:0/2576503998 >> 172.21.15.89:0/2576503998 conn(0x56026bf6d310 msgr2=0x56026bfaed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 -- 172.21.15.89:0/2576503998 shutdown_connections 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 -- 172.21.15.89:0/2576503998 wait complete. 2024-09-15T23:33:41.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 -- start start 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.402+0000 7f07f3d73740 1 Processor -- start 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa45d0 0x56026bfbbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa5b40 0x56026bfb72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07f3d73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07f3d73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec1ae1c0 con 0x56026bfb3780 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07f3d73740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec1adaf0 con 0x56026bfa5b40 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07f3d73740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07ec1aad30 con 0x56026bfa45d0 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07f3d73740 10 client.0 _ll_register_callbacks cb 0x7f07e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44170/0 (socket says 172.21.15.89:44170) 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07eb7fe640 1 -- 172.21.15.89:0/1723704224 learned_addr learned my addr 172.21.15.89:0/1723704224 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07eaffd640 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa45d0 0x56026bfbbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07ea7fc640 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa5b40 0x56026bfb72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07eb7fe640 1 -- 172.21.15.89:0/1723704224 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa45d0 msgr2=0x56026bfbbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa45d0 0x56026bfbbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07eb7fe640 1 -- 172.21.15.89:0/1723704224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa5b40 msgr2=0x56026bfb72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa5b40 0x56026bfb72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07eb7fe640 1 -- 172.21.15.89:0/1723704224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56026bfb8010 con 0x56026bfb3780 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07eb7fe640 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb7810 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f07e4097f70 tx=0x7f07e40a4210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd48e651052aca5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07caffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.403+0000 7f07caffd640 1 -- 172.21.15.89:0/1723704224 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e40998a0 con 0x56026bfb3780 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07caffd640 1 -- 172.21.15.89:0/1723704224 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f07e40145f0 con 0x56026bfb3780 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07f3d73740 1 -- 172.21.15.89:0/1723704224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07e409f950 con 0x56026bfb3780 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07caffd640 1 -- 172.21.15.89:0/1723704224 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f07e40a3d80 con 0x56026bfb3780 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07f3d73740 10 client.18675 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:41.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07f3d73740 1 -- 172.21.15.89:0/1723704224 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56026c02e0d0 con 0x56026bfb3780 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07caffd640 1 -- 172.21.15.89:0/1723704224 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07e409f950 con 0x56026bfb3780 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07caffd640 1 -- 172.21.15.89:0/1723704224 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f07e4084370 con 0x56026bfb3780 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07f3d73740 10 client.18675 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07f3d73740 10 client.18675 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07f3d73740 20 client.18675 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07f3d73740 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56026c02e2f0 0x56026c0306e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07f3d73740 4 client.18675 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07f3d73740 1 -- 172.21.15.89:0/1723704224 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56026bf11d70 con 0x56026c02e2f0 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.404+0000 7f07eaffd640 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56026c02e2f0 0x56026c0306e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07caffd640 10 client.18675 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07eaffd640 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56026c02e2f0 0x56026c0306e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=269fc3236b0a3dae server_cookie=37fffc5a2e21fe67 in_seq=0 out_seq=0 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07caffd640 1 -- 172.21.15.89:0/1723704224 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56026bf11d70 con 0x56026c02e2f0 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07caffd640 10 client.18675 handle_command_reply: tid=0 2024-09-15T23:33:41.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 client.18675 shutdown 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 -- 172.21.15.89:0/1723704224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 msgr2=0x56026bfb7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb7810 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f07e4097f70 tx=0x7f07e40a4210 comp rx=0 tx=0).stop 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 -- 172.21.15.89:0/1723704224 shutdown_connections 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56026bfb3780 0x56026bfb7810 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56026bfa5b40 0x56026bfb72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56026c02e2f0 0x56026c0306e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 --2- 172.21.15.89:0/1723704224 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56026bfa45d0 0x56026bfbbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07caffd640 0 client.18675 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 -- 172.21.15.89:0/1723704224 >> 172.21.15.89:0/1723704224 conn(0x56026bf6d310 msgr2=0x56026bfaf550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 -- 172.21.15.89:0/1723704224 shutdown_connections 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 1 -- 172.21.15.89:0/1723704224 wait complete. 2024-09-15T23:33:41.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.405+0000 7f07f3d73740 20 client.18675 trim_cache size 0 max 16384 2024-09-15T23:33:41.417 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:33:41.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 -- 172.21.15.89:0/206219583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07d40a3240 msgr2=0x7f07d40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:41.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07d40a3240 0x7f07d40a5700 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009990 tx=0x7f07dc009110 comp rx=0 tx=0).stop 2024-09-15T23:33:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 -- 172.21.15.89:0/206219583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07ec10c170 msgr2=0x7f07ec19cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07ec10c170 0x7f07ec19cf60 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f07e400b970 tx=0x7f07e400bd70 comp rx=0 tx=0).stop 2024-09-15T23:33:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 -- 172.21.15.89:0/206219583 shutdown_connections 2024-09-15T23:33:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07d40a3240 0x7f07d40a5700 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f07ec10c170 0x7f07ec19cf60 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07ec103660 0x7f07ec198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 --2- 172.21.15.89:0/206219583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ec102d40 0x7f07ec198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 -- 172.21.15.89:0/206219583 >> 172.21.15.89:0/206219583 conn(0x7f07ec0fc7a0 msgr2=0x7f07ec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 -- 172.21.15.89:0/206219583 shutdown_connections 2024-09-15T23:33:41.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:41.406+0000 7f07f199a640 1 -- 172.21.15.89:0/206219583 wait complete. 2024-09-15T23:33:42.113 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:33:42.359 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:42.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:42 smithi089 ceph-mon[30933]: pgmap v640: 129 pgs: 129 active+clean; 748 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-09-15T23:33:42.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:42 smithi120 ceph-mon[35250]: pgmap v640: 129 pgs: 129 active+clean; 748 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-09-15T23:33:42.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:42 smithi174 ceph-mon[35586]: pgmap v640: 129 pgs: 129 active+clean; 748 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-09-15T23:33:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 -- 172.21.15.89:0/38458126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade4102790 msgr2=0x7fade41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 --2- 172.21.15.89:0/38458126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade4102790 0x7fade41067e0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fadd8009900 tx=0x7fadd802a410 comp rx=0 tx=0).stop 2024-09-15T23:33:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 -- 172.21.15.89:0/38458126 shutdown_connections 2024-09-15T23:33:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 --2- 172.21.15.89:0/38458126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fade4106f10 0x7fade410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 --2- 172.21.15.89:0/38458126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade4102790 0x7fade41067e0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 --2- 172.21.15.89:0/38458126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade4101de0 0x7fade41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 -- 172.21.15.89:0/38458126 >> 172.21.15.89:0/38458126 conn(0x7fade40fd880 msgr2=0x7fade40ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 -- 172.21.15.89:0/38458126 shutdown_connections 2024-09-15T23:33:42.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 -- 172.21.15.89:0/38458126 wait complete. 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 Processor -- start 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.844+0000 7fadeb65e640 1 -- start start 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fadeb65e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade4101de0 0x7fade4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fadeb65e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fade4102790 0x7fade4198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fadeb65e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade4106f10 0x7fade419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fadeb65e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fade4111f70 con 0x7fade4101de0 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fadeb65e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fade4111df0 con 0x7fade4106f10 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fadeb65e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fade41120f0 con 0x7fade4102790 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade93fa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade4101de0 0x7fade4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade93fa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade4101de0 0x7fade4198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44184/0 (socket says 172.21.15.89:44184) 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade93fa640 1 -- 172.21.15.89:0/4051870620 learned_addr learned my addr 172.21.15.89:0/4051870620 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade8bf9640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fade4102790 0x7fade4198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade9bfb640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade4106f10 0x7fade419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade93fa640 1 -- 172.21.15.89:0/4051870620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fade4102790 msgr2=0x7fade4198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade93fa640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fade4102790 0x7fade4198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade93fa640 1 -- 172.21.15.89:0/4051870620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade4106f10 msgr2=0x7fade419cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade93fa640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade4106f10 0x7fade419cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade93fa640 1 -- 172.21.15.89:0/4051870620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fade419d3e0 con 0x7fade4101de0 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade8bf9640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fade4102790 0x7fade4198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade93fa640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade4101de0 0x7fade4198430 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fadd400eab0 tx=0x7fadd400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade27fc640 1 -- 172.21.15.89:0/4051870620 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd400ce40 con 0x7fade4101de0 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fadeb65e640 1 -- 172.21.15.89:0/4051870620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fade419d6d0 con 0x7fade4101de0 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade27fc640 1 -- 172.21.15.89:0/4051870620 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadd4011950 con 0x7fade4101de0 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.845+0000 7fade27fc640 1 -- 172.21.15.89:0/4051870620 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd4010ae0 con 0x7fade4101de0 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.846+0000 7fadeb65e640 1 -- 172.21.15.89:0/4051870620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fade41a4fb0 con 0x7fade4101de0 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.848+0000 7fade27fc640 1 -- 172.21.15.89:0/4051870620 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fadd4011b00 con 0x7fade4101de0 2024-09-15T23:33:42.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fadeda37740 1 Processor -- start 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fadeda37740 1 -- start start 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fadeda37740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e46090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fadeda37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e465d0 0x5628d5e46a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fadeda37740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e51370 0x5628d5e53760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fadeda37740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5628d5dac2f0 con 0x5628d5e47b40 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fadeda37740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5628d5d53cf0 con 0x5628d5e51370 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fadeda37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5628d5da82b0 con 0x5628d5e465d0 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade9bfb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e46090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade9bfb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e46090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44190/0 (socket says 172.21.15.89:44190) 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade9bfb640 1 -- 172.21.15.89:0/4191559184 learned_addr learned my addr 172.21.15.89:0/4191559184 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade27fc640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fadcc0a3240 0x7fadcc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade93fa640 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e465d0 0x5628d5e46a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade8bf9640 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e51370 0x5628d5e53760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade9bfb640 1 -- 172.21.15.89:0/4191559184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e465d0 msgr2=0x5628d5e46a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade9bfb640 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e465d0 0x5628d5e46a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade9bfb640 1 -- 172.21.15.89:0/4191559184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e51370 msgr2=0x5628d5e53760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade9bfb640 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e51370 0x5628d5e53760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade9bfb640 1 -- 172.21.15.89:0/4191559184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628d5d383a0 con 0x5628d5e47b40 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade93fa640 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e465d0 0x5628d5e46a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade8bf9640 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e51370 0x5628d5e53760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.849+0000 7fade27fc640 1 -- 172.21.15.89:0/4051870620 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fadd40d04f0 con 0x7fade4101de0 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.850+0000 7fade9bfb640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fadcc0a3240 0x7fadcc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.850+0000 7fade9bfb640 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e46090 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7faddc00cc40 tx=0x7faddc00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b9abc8d4975251 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.850+0000 7fadcb7fe640 1 -- 172.21.15.89:0/4191559184 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc022070 con 0x5628d5e47b40 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.850+0000 7fadcb7fe640 1 -- 172.21.15.89:0/4191559184 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faddc018070 con 0x5628d5e47b40 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.850+0000 7fade9bfb640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fadcc0a3240 0x7fadcc0a5700 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7faddc01f000 tx=0x7faddc020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:42.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.850+0000 7fadcb7fe640 1 -- 172.21.15.89:0/4191559184 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc01c9c0 con 0x5628d5e47b40 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 -- 172.21.15.89:0/4191559184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 msgr2=0x5628d5e46090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e46090 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7faddc00cc40 tx=0x7faddc00e720 comp rx=0 tx=0).stop 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 -- 172.21.15.89:0/4191559184 shutdown_connections 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e51370 0x5628d5e53760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e465d0 0x5628d5e46a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 --2- 172.21.15.89:0/4191559184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e46090 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 -- 172.21.15.89:0/4191559184 >> 172.21.15.89:0/4191559184 conn(0x5628d5e0f310 msgr2=0x5628d5e42c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 -- 172.21.15.89:0/4191559184 shutdown_connections 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 -- 172.21.15.89:0/4191559184 wait complete. 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 -- start start 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.851+0000 7fadeda37740 1 Processor -- start 2024-09-15T23:33:42.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadeda37740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 0x5628d5e50c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadeda37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e47b40 0x5628d5e4df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadeda37740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e51370 0x5628d5e4e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadeda37740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5628d5e54070 con 0x5628d5e51370 2024-09-15T23:33:42.882 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:33:42.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadeda37740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5628d5d67060 con 0x5628d5e465d0 2024-09-15T23:33:42.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadeda37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5628d5e541f0 con 0x5628d5e47b40 2024-09-15T23:33:42.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadeda37740 10 client.0 _ll_register_callbacks cb 0x5628d5aba8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:42.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade9bfb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e51370 0x5628d5e4e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade9bfb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e51370 0x5628d5e4e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44200/0 (socket says 172.21.15.89:44200) 2024-09-15T23:33:42.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade9bfb640 1 -- 172.21.15.89:0/937662893 learned_addr learned my addr 172.21.15.89:0/937662893 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:42.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade8bf9640 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e47b40 0x5628d5e4df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade93fa640 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 0x5628d5e50c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade9bfb640 1 -- 172.21.15.89:0/937662893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e47b40 msgr2=0x5628d5e4df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:42.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade9bfb640 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e47b40 0x5628d5e4df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade9bfb640 1 -- 172.21.15.89:0/937662893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 msgr2=0x5628d5e50c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade9bfb640 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 0x5628d5e50c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade9bfb640 1 -- 172.21.15.89:0/937662893 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628d5b1d3c0 con 0x5628d5e51370 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fade9bfb640 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e51370 0x5628d5e4e480 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7faddc000c00 tx=0x7faddc011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c979743c6399ee24 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadc8ff9640 1 -- 172.21.15.89:0/937662893 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc034060 con 0x5628d5e51370 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadc8ff9640 1 -- 172.21.15.89:0/937662893 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7faddc026670 con 0x5628d5e51370 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.852+0000 7fadeda37740 1 -- 172.21.15.89:0/937662893 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faddc018070 con 0x5628d5e51370 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadc8ff9640 1 -- 172.21.15.89:0/937662893 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faddc03b050 con 0x5628d5e51370 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadeda37740 10 client.18687 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadeda37740 1 -- 172.21.15.89:0/937662893 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5628d5e5f9a0 con 0x5628d5e51370 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadc8ff9640 1 -- 172.21.15.89:0/937662893 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faddc018070 con 0x5628d5e51370 2024-09-15T23:33:42.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadc8ff9640 1 -- 172.21.15.89:0/937662893 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7faddc02e030 con 0x5628d5e51370 2024-09-15T23:33:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadeda37740 10 client.18687 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadeda37740 10 client.18687 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:33:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadeda37740 20 client.18687 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadeda37740 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5628d5e60da0 0x5628d5e63190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadeda37740 4 client.18687 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fadeda37740 1 -- 172.21.15.89:0/937662893 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628d5db9860 con 0x5628d5e60da0 2024-09-15T23:33:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.853+0000 7fade93fa640 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5628d5e60da0 0x5628d5e63190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.854+0000 7fade93fa640 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5628d5e60da0 0x5628d5e63190 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a0af7ff441378a8b server_cookie=f8a5b1f4a9f9a102 in_seq=0 out_seq=0 2024-09-15T23:33:42.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.854+0000 7fadc8ff9640 10 client.18687 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.854+0000 7fadc8ff9640 1 -- 172.21.15.89:0/937662893 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5628d5db9860 con 0x5628d5e60da0 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.854+0000 7fadc8ff9640 10 client.18687 handle_command_reply: tid=0 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 client.18687 shutdown 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 -- 172.21.15.89:0/937662893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e51370 msgr2=0x5628d5e4e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e51370 0x5628d5e4e480 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7faddc000c00 tx=0x7faddc011ae0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 -- 172.21.15.89:0/937662893 shutdown_connections 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5628d5e60da0 0x5628d5e63190 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e51370 0x5628d5e4e480 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e47b40 0x5628d5e4df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 --2- 172.21.15.89:0/937662893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 0x5628d5e50c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadc8ff9640 0 client.18687 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 -- 172.21.15.89:0/937662893 >> 172.21.15.89:0/937662893 conn(0x5628d5e0f310 msgr2=0x5628d5e48960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:42.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 -- 172.21.15.89:0/937662893 shutdown_connections 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 1 -- 172.21.15.89:0/937662893 wait complete. 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.855+0000 7fadeda37740 20 client.18687 trim_cache size 0 max 16384 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fadeda37740 1 Processor -- start 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fadeda37740 1 -- start start 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fadeda37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e0f310 0x5628d5e5d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fadeda37740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 0x5628d5e4d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fadeda37740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e49120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fadeda37740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fade41aef00 con 0x5628d5e47b40 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fadeda37740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fade41b0a40 con 0x5628d5e465d0 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fadeda37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fade41af5d0 con 0x5628d5e0f310 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade9bfb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e49120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade9bfb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e49120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44212/0 (socket says 172.21.15.89:44212) 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade9bfb640 1 -- 172.21.15.89:0/1999111869 learned_addr learned my addr 172.21.15.89:0/1999111869 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:42.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade9bfb640 1 -- 172.21.15.89:0/1999111869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e0f310 msgr2=0x5628d5e5d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade93fa640 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e0f310 0x5628d5e5d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade8bf9640 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 0x5628d5e4d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade9bfb640 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e0f310 0x5628d5e5d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade9bfb640 1 -- 172.21.15.89:0/1999111869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 msgr2=0x5628d5e4d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade9bfb640 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 0x5628d5e4d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade9bfb640 1 -- 172.21.15.89:0/1999111869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628d5e4c490 con 0x5628d5e47b40 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade93fa640 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e0f310 0x5628d5e5d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.869+0000 7fade8bf9640 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 0x5628d5e4d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fade9bfb640 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e49120 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7faddc00c7f0 tx=0x7faddc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82012248291ac4a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadcaffd640 1 -- 172.21.15.89:0/1999111869 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc034040 con 0x5628d5e47b40 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadcaffd640 1 -- 172.21.15.89:0/1999111869 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faddc018040 con 0x5628d5e47b40 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadcaffd640 1 -- 172.21.15.89:0/1999111869 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc022070 con 0x5628d5e47b40 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadeda37740 1 -- 172.21.15.89:0/1999111869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 msgr2=0x5628d5e49120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:42.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadeda37740 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e49120 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7faddc00c7f0 tx=0x7faddc005850 comp rx=0 tx=0).stop 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadeda37740 1 -- 172.21.15.89:0/1999111869 shutdown_connections 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadeda37740 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e47b40 0x5628d5e49120 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadeda37740 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e465d0 0x5628d5e4d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadeda37740 1 --2- 172.21.15.89:0/1999111869 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e0f310 0x5628d5e5d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadeda37740 1 -- 172.21.15.89:0/1999111869 >> 172.21.15.89:0/1999111869 conn(0x5628d5e4e5e0 msgr2=0x5628d5e4d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.870+0000 7fadeda37740 1 -- 172.21.15.89:0/1999111869 shutdown_connections 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fadeda37740 1 -- 172.21.15.89:0/1999111869 wait complete. 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fadeda37740 1 -- start start 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fadeda37740 1 Processor -- start 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fadeda37740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e0f310 0x5628d5e59b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fadeda37740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e465d0 0x5628d5e54e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fadeda37740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e47b40 0x5628d5e55380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fadeda37740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fade41b0370 con 0x5628d5e0f310 2024-09-15T23:33:42.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fadeda37740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fade41afca0 con 0x5628d5e47b40 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fadeda37740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fade41acee0 con 0x5628d5e465d0 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fadeda37740 10 client.0 _ll_register_callbacks cb 0x7faddc020070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fade93fa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e0f310 0x5628d5e59b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fade93fa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e0f310 0x5628d5e59b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44228/0 (socket says 172.21.15.89:44228) 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.871+0000 7fade93fa640 1 -- 172.21.15.89:0/646791051 learned_addr learned my addr 172.21.15.89:0/646791051 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fade9bfb640 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e47b40 0x5628d5e55380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fade93fa640 1 -- 172.21.15.89:0/646791051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e465d0 msgr2=0x5628d5e54e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fade93fa640 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e465d0 0x5628d5e54e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fade93fa640 1 -- 172.21.15.89:0/646791051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e47b40 msgr2=0x5628d5e55380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fade93fa640 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e47b40 0x5628d5e55380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fade93fa640 1 -- 172.21.15.89:0/646791051 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628d5e55b60 con 0x5628d5e0f310 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fade9bfb640 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e47b40 0x5628d5e55380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fade93fa640 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e0f310 0x5628d5e59b10 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fadd400c700 tx=0x7fadd406c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f14289c8c9cd863 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadc8ff9640 1 -- 172.21.15.89:0/646791051 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd409ce20 con 0x5628d5e0f310 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadc8ff9640 1 -- 172.21.15.89:0/646791051 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fadd4098d90 con 0x5628d5e0f310 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadeda37740 1 -- 172.21.15.89:0/646791051 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faddc018040 con 0x5628d5e0f310 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadc8ff9640 1 -- 172.21.15.89:0/646791051 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fadd409c720 con 0x5628d5e0f310 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadeda37740 10 client.18693 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadeda37740 1 -- 172.21.15.89:0/646791051 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5628d5ecb7c0 con 0x5628d5e0f310 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadc8ff9640 1 -- 172.21.15.89:0/646791051 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faddc018040 con 0x5628d5e0f310 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadc8ff9640 1 -- 172.21.15.89:0/646791051 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fadd4085060 con 0x5628d5e0f310 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadeda37740 10 client.18693 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.872+0000 7fadeda37740 10 client.18693 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadeda37740 20 client.18693 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadeda37740 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5628d5e51370 0x5628d5ecdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadeda37740 4 client.18693 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:33:42.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadeda37740 1 -- 172.21.15.89:0/646791051 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628d5db3d70 con 0x5628d5e51370 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fade8bf9640 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5628d5e51370 0x5628d5ecdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fade8bf9640 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5628d5e51370 0x5628d5ecdee0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=78f2c50531008866 server_cookie=fdfabd74b971797a in_seq=0 out_seq=0 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadc8ff9640 10 client.18693 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadc8ff9640 1 -- 172.21.15.89:0/646791051 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628d5db3d70 con 0x5628d5e51370 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadc8ff9640 10 client.18693 handle_command_reply: tid=0 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadeda37740 1 client.18693 shutdown 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadeda37740 1 -- 172.21.15.89:0/646791051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e0f310 msgr2=0x5628d5e59b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadeda37740 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e0f310 0x5628d5e59b10 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fadd400c700 tx=0x7fadd406c390 comp rx=0 tx=0).stop 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadeda37740 1 -- 172.21.15.89:0/646791051 shutdown_connections 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.873+0000 7fadeda37740 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5628d5e51370 0x5628d5ecdee0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeda37740 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628d5e47b40 0x5628d5e55380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeda37740 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5628d5e465d0 0x5628d5e54e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadc8ff9640 0 client.18693 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:33:42.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeda37740 1 --2- 172.21.15.89:0/646791051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5628d5e0f310 0x5628d5e59b10 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeda37740 1 -- 172.21.15.89:0/646791051 >> 172.21.15.89:0/646791051 conn(0x5628d5e4e5e0 msgr2=0x5628d5e41310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeda37740 1 -- 172.21.15.89:0/646791051 shutdown_connections 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeda37740 1 -- 172.21.15.89:0/646791051 wait complete. 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeda37740 20 client.18693 trim_cache size 0 max 16384 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeb65e640 1 -- 172.21.15.89:0/4051870620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fadcc0a3240 msgr2=0x7fadcc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeb65e640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fadcc0a3240 0x7fadcc0a5700 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7faddc01f000 tx=0x7faddc020000 comp rx=0 tx=0).stop 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeb65e640 1 -- 172.21.15.89:0/4051870620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade4101de0 msgr2=0x7fade4198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeb65e640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade4101de0 0x7fade4198430 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fadd400eab0 tx=0x7fadd400ef70 comp rx=0 tx=0).stop 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeb65e640 1 -- 172.21.15.89:0/4051870620 shutdown_connections 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeb65e640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fadcc0a3240 0x7fadcc0a5700 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeb65e640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fade4106f10 0x7fade419cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeb65e640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fade4102790 0x7fade4198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeb65e640 1 --2- 172.21.15.89:0/4051870620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fade4101de0 0x7fade4198430 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:42.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.874+0000 7fadeb65e640 1 -- 172.21.15.89:0/4051870620 >> 172.21.15.89:0/4051870620 conn(0x7fade40fd880 msgr2=0x7fade40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:42.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.875+0000 7fadeb65e640 1 -- 172.21.15.89:0/4051870620 shutdown_connections 2024-09-15T23:33:42.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:42.875+0000 7fadeb65e640 1 -- 172.21.15.89:0/4051870620 wait complete. 2024-09-15T23:33:43.583 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:33:43.831 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.333+0000 7f72e4336640 1 -- 172.21.15.89:0/3743097026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72dc100a80 msgr2=0x7f72dc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.333+0000 7f72e4336640 1 --2- 172.21.15.89:0/3743097026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72dc100a80 0x7f72dc100ee0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009930 tx=0x7f72d002a420 comp rx=0 tx=0).stop 2024-09-15T23:33:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.333+0000 7f72e4336640 1 -- 172.21.15.89:0/3743097026 shutdown_connections 2024-09-15T23:33:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.333+0000 7f72e4336640 1 --2- 172.21.15.89:0/3743097026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72dc101420 0x7f72dc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.333+0000 7f72e4336640 1 --2- 172.21.15.89:0/3743097026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72dc100a80 0x7f72dc100ee0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.333+0000 7f72e4336640 1 --2- 172.21.15.89:0/3743097026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc106a80 0x7f72dc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.333+0000 7f72e4336640 1 -- 172.21.15.89:0/3743097026 >> 172.21.15.89:0/3743097026 conn(0x7f72dc0fc7a0 msgr2=0x7f72dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.333+0000 7f72e4336640 1 -- 172.21.15.89:0/3743097026 shutdown_connections 2024-09-15T23:33:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.333+0000 7f72e4336640 1 -- 172.21.15.89:0/3743097026 wait complete. 2024-09-15T23:33:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e4336640 1 Processor -- start 2024-09-15T23:33:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e4336640 1 -- start start 2024-09-15T23:33:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e4336640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72dc100a80 0x7f72dc1973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e4336640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72dc101420 0x7f72dc1978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e4336640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc106a80 0x7f72dc19bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e4336640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc110ea0 con 0x7f72dc100a80 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e4336640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc110d20 con 0x7f72dc101420 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e4336640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc111020 con 0x7f72dc106a80 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e20d2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72dc100a80 0x7f72dc1973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e20d2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72dc100a80 0x7f72dc1973a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44246/0 (socket says 172.21.15.89:44246) 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e20d2640 1 -- 172.21.15.89:0/8419927 learned_addr learned my addr 172.21.15.89:0/8419927 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e18d1640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72dc101420 0x7f72dc1978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e20d2640 1 -- 172.21.15.89:0/8419927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc106a80 msgr2=0x7f72dc19bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e20d2640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc106a80 0x7f72dc19bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e20d2640 1 -- 172.21.15.89:0/8419927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72dc101420 msgr2=0x7f72dc1978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e20d2640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72dc101420 0x7f72dc1978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e20d2640 1 -- 172.21.15.89:0/8419927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72dc19c350 con 0x7f72dc100a80 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.334+0000 7f72e20d2640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72dc100a80 0x7f72dc1973a0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f72d800b4a0 tx=0x7f72d800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:44.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.335+0000 7f72cf7fe640 1 -- 172.21.15.89:0/8419927 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d8014070 con 0x7f72dc100a80 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.335+0000 7f72cf7fe640 1 -- 172.21.15.89:0/8419927 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d8010af0 con 0x7f72dc100a80 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.335+0000 7f72cf7fe640 1 -- 172.21.15.89:0/8419927 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d8017d10 con 0x7f72dc100a80 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.335+0000 7f72e4336640 1 -- 172.21.15.89:0/8419927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72dc19c640 con 0x7f72dc100a80 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.335+0000 7f72e4336640 1 -- 172.21.15.89:0/8419927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72dc1a3f20 con 0x7f72dc100a80 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.337+0000 7f72cf7fe640 1 -- 172.21.15.89:0/8419927 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f72d8017070 con 0x7f72dc100a80 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.337+0000 7f72e670f740 1 Processor -- start 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.337+0000 7f72e670f740 1 -- start start 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e670f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0d4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e670f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d45d0 0x56271e0d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e670f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e670f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56271e03a2f0 con 0x56271e0e1580 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e670f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56271dfe1cf0 con 0x56271e0d5b40 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e670f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56271e0362b0 con 0x56271e0d45d0 2024-09-15T23:33:44.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72cf7fe640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72c80a32c0 0x7f72c80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e18d1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e18d1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44252/0 (socket says 172.21.15.89:44252) 2024-09-15T23:33:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e28d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0d4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e18d1640 1 -- 172.21.15.89:0/468148381 learned_addr learned my addr 172.21.15.89:0/468148381 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e28d3640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72c80a32c0 0x7f72c80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72cf7fe640 1 -- 172.21.15.89:0/8419927 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f72d80cf110 con 0x7f72dc100a80 2024-09-15T23:33:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e28d3640 1 -- 172.21.15.89:0/468148381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d45d0 msgr2=0x56271e0d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e28d3640 1 --2- 172.21.15.89:0/468148381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d45d0 0x56271e0d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e28d3640 1 -- 172.21.15.89:0/468148381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 msgr2=0x56271e0e3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:44.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e28d3640 1 --2- 172.21.15.89:0/468148381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e28d3640 1 -- 172.21.15.89:0/468148381 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56271dfc63a0 con 0x56271e0d5b40 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e18d1640 1 --2- 172.21.15.89:0/468148381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e28d3640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72c80a32c0 0x7f72c80a5780 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f72d4005fd0 tx=0x7f72d401a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.338+0000 7f72e28d3640 1 --2- 172.21.15.89:0/468148381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0d4090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f72d4013c40 tx=0x7f72d4011750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e768ccfb8533b2b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72ccdea640 1 -- 172.21.15.89:0/468148381 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4020070 con 0x56271e0d5b40 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72ccdea640 1 -- 172.21.15.89:0/468148381 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d4009a00 con 0x56271e0d5b40 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72ccdea640 1 -- 172.21.15.89:0/468148381 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d400ed50 con 0x56271e0d5b40 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72e670f740 1 -- 172.21.15.89:0/468148381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 msgr2=0x56271e0d4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72e670f740 1 --2- 172.21.15.89:0/468148381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0d4090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f72d4013c40 tx=0x7f72d4011750 comp rx=0 tx=0).stop 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72e670f740 1 -- 172.21.15.89:0/468148381 shutdown_connections 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72e670f740 1 --2- 172.21.15.89:0/468148381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72e670f740 1 --2- 172.21.15.89:0/468148381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d45d0 0x56271e0d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72e670f740 1 --2- 172.21.15.89:0/468148381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0d4090 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72e670f740 1 -- 172.21.15.89:0/468148381 >> 172.21.15.89:0/468148381 conn(0x56271e09d310 msgr2=0x56271e0d0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72e670f740 1 -- 172.21.15.89:0/468148381 shutdown_connections 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.339+0000 7f72e670f740 1 -- 172.21.15.89:0/468148381 wait complete. 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e670f740 1 -- start start 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e670f740 1 Processor -- start 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e670f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d45d0 0x56271e0e0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e670f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0de250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e670f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0de860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e670f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56271e0e4220 con 0x56271e0e1580 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e670f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56271dff5060 con 0x56271e0d5b40 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e670f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56271e0e43a0 con 0x56271e0d45d0 2024-09-15T23:33:44.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e670f740 10 client.0 _ll_register_callbacks cb 0x56271dd488c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e28d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0de860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e28d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0de860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44260/0 (socket says 172.21.15.89:44260) 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e28d3640 1 -- 172.21.15.89:0/1964902422 learned_addr learned my addr 172.21.15.89:0/1964902422 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e18d1640 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0de250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e20d2640 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d45d0 0x56271e0e0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.340+0000 7f72e28d3640 1 -- 172.21.15.89:0/1964902422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d45d0 msgr2=0x56271e0e0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72e28d3640 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d45d0 0x56271e0e0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72e28d3640 1 -- 172.21.15.89:0/1964902422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 msgr2=0x56271e0de250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72e28d3640 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0de250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72e28d3640 1 -- 172.21.15.89:0/1964902422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56271ddab3c0 con 0x56271e0e1580 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72e28d3640 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0de860 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009590 tx=0x7f72d4004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e524540b5385a95b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:44.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1964902422 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4020040 con 0x56271e0e1580 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72e670f740 1 -- 172.21.15.89:0/1964902422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72d4009a00 con 0x56271e0e1580 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1964902422 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f72d4003b80 con 0x56271e0e1580 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1964902422 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f72d4037050 con 0x56271e0e1580 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72e670f740 10 client.18705 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72e670f740 1 -- 172.21.15.89:0/1964902422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56271e0efd70 con 0x56271e0e1580 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1964902422 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d4009a00 con 0x56271e0e1580 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1964902422 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f72d401b070 con 0x56271e0e1580 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.341+0000 7f72e670f740 10 client.18705 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.342+0000 7f72e670f740 10 client.18705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.342+0000 7f72e670f740 20 client.18705 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.342+0000 7f72e670f740 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56271e0f1180 0x56271e0f3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.342+0000 7f72e670f740 4 client.18705 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:44.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.342+0000 7f72e670f740 1 -- 172.21.15.89:0/1964902422 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56271e047860 con 0x56271e0f1180 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.342+0000 7f72e20d2640 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56271e0f1180 0x56271e0f3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.342+0000 7f72c5ffb640 10 client.18705 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.342+0000 7f72e20d2640 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56271e0f1180 0x56271e0f3570 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bb6b5b5a1ce1db2 server_cookie=9c84ee73447473fb in_seq=0 out_seq=0 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1964902422 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56271e047860 con 0x56271e0f1180 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72c5ffb640 10 client.18705 handle_command_reply: tid=0 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 client.18705 shutdown 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 -- 172.21.15.89:0/1964902422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 msgr2=0x56271e0de860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0de860 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009590 tx=0x7f72d4004d50 comp rx=0 tx=0).stop 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 -- 172.21.15.89:0/1964902422 shutdown_connections 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56271e0f1180 0x56271e0f3570 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0de860 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72c5ffb640 0 client.18705 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0de250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 --2- 172.21.15.89:0/1964902422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d45d0 0x56271e0e0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 -- 172.21.15.89:0/1964902422 >> 172.21.15.89:0/1964902422 conn(0x56271e09d310 msgr2=0x56271e0d6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 -- 172.21.15.89:0/1964902422 shutdown_connections 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 1 -- 172.21.15.89:0/1964902422 wait complete. 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.343+0000 7f72e670f740 20 client.18705 trim_cache size 0 max 16384 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e670f740 1 Processor -- start 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e670f740 1 -- start start 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e670f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d45d0 0x56271e0ed4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e670f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d5b40 0x56271e0dfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e670f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e670f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1acc50 con 0x56271e0e1580 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e670f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1ae790 con 0x56271e0d45d0 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e670f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1ad320 con 0x56271e0d5b40 2024-09-15T23:33:44.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e28d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e28d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44270/0 (socket says 172.21.15.89:44270) 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e28d3640 1 -- 172.21.15.89:0/27041013 learned_addr learned my addr 172.21.15.89:0/27041013 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e20d2640 1 --2- 172.21.15.89:0/27041013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d45d0 0x56271e0ed4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e28d3640 1 -- 172.21.15.89:0/27041013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d5b40 msgr2=0x56271e0dfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e28d3640 1 --2- 172.21.15.89:0/27041013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d5b40 0x56271e0dfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e28d3640 1 -- 172.21.15.89:0/27041013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d45d0 msgr2=0x56271e0ed4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e28d3640 1 --2- 172.21.15.89:0/27041013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d45d0 0x56271e0ed4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.356+0000 7f72e28d3640 1 -- 172.21.15.89:0/27041013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56271e0e0960 con 0x56271e0e1580 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72e28d3640 1 --2- 172.21.15.89:0/27041013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e9a40 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009d00 tx=0x7f72d4011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c86f1a26ca8b172 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72c7fff640 1 -- 172.21.15.89:0/27041013 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4020040 con 0x56271e0e1580 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72c7fff640 1 -- 172.21.15.89:0/27041013 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d4011c50 con 0x56271e0e1580 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72c7fff640 1 -- 172.21.15.89:0/27041013 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d4013040 con 0x56271e0e1580 2024-09-15T23:33:44.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72e670f740 1 -- 172.21.15.89:0/27041013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 msgr2=0x56271e0e9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:44.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72e670f740 1 --2- 172.21.15.89:0/27041013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e9a40 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009d00 tx=0x7f72d4011750 comp rx=0 tx=0).stop 2024-09-15T23:33:44.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72e670f740 1 -- 172.21.15.89:0/27041013 shutdown_connections 2024-09-15T23:33:44.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72e670f740 1 --2- 172.21.15.89:0/27041013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0e1580 0x56271e0e9a40 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72e670f740 1 --2- 172.21.15.89:0/27041013 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0d5b40 0x56271e0dfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72e670f740 1 --2- 172.21.15.89:0/27041013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d45d0 0x56271e0ed4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72e670f740 1 -- 172.21.15.89:0/27041013 >> 172.21.15.89:0/27041013 conn(0x56271e09d310 msgr2=0x56271e0dd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:44.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72e670f740 1 -- 172.21.15.89:0/27041013 shutdown_connections 2024-09-15T23:33:44.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.357+0000 7f72e670f740 1 -- 172.21.15.89:0/27041013 wait complete. 2024-09-15T23:33:44.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e670f740 1 -- start start 2024-09-15T23:33:44.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e670f740 1 Processor -- start 2024-09-15T23:33:44.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e670f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0d45d0 0x56271e0d64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e670f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0d6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e670f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0e1580 0x56271e0d6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e670f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1ae0c0 con 0x56271e0d45d0 2024-09-15T23:33:44.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e670f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1ad9f0 con 0x56271e0d5b40 2024-09-15T23:33:44.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e670f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f72dc1aac30 con 0x56271e0e1580 2024-09-15T23:33:44.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e670f740 10 client.0 _ll_register_callbacks cb 0x7f72d0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:44.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e20d2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0d45d0 0x56271e0d64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e20d2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0d45d0 0x56271e0d64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44282/0 (socket says 172.21.15.89:44282) 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e20d2640 1 -- 172.21.15.89:0/1111271033 learned_addr learned my addr 172.21.15.89:0/1111271033 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e18d1640 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0d6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.358+0000 7f72e28d3640 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0e1580 0x56271e0d6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72e20d2640 1 -- 172.21.15.89:0/1111271033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0e1580 msgr2=0x56271e0d6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72e20d2640 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0e1580 0x56271e0d6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72e20d2640 1 -- 172.21.15.89:0/1111271033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 msgr2=0x56271e0d6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72e20d2640 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0d6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72e20d2640 1 -- 172.21.15.89:0/1111271033 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56271e15ccf0 con 0x56271e0d45d0 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72e20d2640 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0d45d0 0x56271e0d64d0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f72d80cc820 tx=0x7f72d80bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7626424dbb3b196 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:44.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1111271033 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d801e040 con 0x56271e0d45d0 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1111271033 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f72d8017360 con 0x56271e0d45d0 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72e670f740 1 -- 172.21.15.89:0/1111271033 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72d4011c50 con 0x56271e0d45d0 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1111271033 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f72d80a3fb0 con 0x56271e0d45d0 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72e670f740 10 client.18711 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72e670f740 1 -- 172.21.15.89:0/1111271033 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56271e15df30 con 0x56271e0d45d0 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1111271033 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d4011c50 con 0x56271e0d45d0 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.359+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1111271033 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f72d80ccce0 con 0x56271e0d45d0 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e670f740 10 client.18711 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e670f740 10 client.18711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e670f740 20 client.18711 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e670f740 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56271e15e5f0 0x56271e160a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e670f740 4 client.18711 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e670f740 1 -- 172.21.15.89:0/1111271033 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56271e041d70 con 0x56271e15e5f0 2024-09-15T23:33:44.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e18d1640 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56271e15e5f0 0x56271e160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72c5ffb640 10 client.18711 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e18d1640 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56271e15e5f0 0x56271e160a10 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3b8ebff143b80d8 server_cookie=4a3b631f58e51859 in_seq=0 out_seq=0 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72c5ffb640 1 -- 172.21.15.89:0/1111271033 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56271e041d70 con 0x56271e15e5f0 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72c5ffb640 10 client.18711 handle_command_reply: tid=0 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e670f740 1 client.18711 shutdown 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e670f740 1 -- 172.21.15.89:0/1111271033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0d45d0 msgr2=0x56271e0d64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.360+0000 7f72e670f740 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0d45d0 0x56271e0d64d0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f72d80cc820 tx=0x7f72d80bb040 comp rx=0 tx=0).stop 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e670f740 1 -- 172.21.15.89:0/1111271033 shutdown_connections 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e670f740 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56271e15e5f0 0x56271e160a10 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e670f740 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56271e0e1580 0x56271e0d6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e670f740 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56271e0d5b40 0x56271e0d6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72c5ffb640 0 client.18711 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e670f740 1 --2- 172.21.15.89:0/1111271033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56271e0d45d0 0x56271e0d64d0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e670f740 1 -- 172.21.15.89:0/1111271033 >> 172.21.15.89:0/1111271033 conn(0x56271e09d310 msgr2=0x56271e0ec4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:44.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e670f740 1 -- 172.21.15.89:0/1111271033 shutdown_connections 2024-09-15T23:33:44.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e670f740 1 -- 172.21.15.89:0/1111271033 wait complete. 2024-09-15T23:33:44.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e670f740 20 client.18711 trim_cache size 0 max 16384 2024-09-15T23:33:44.381 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:33:44.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 -- 172.21.15.89:0/8419927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72c80a32c0 msgr2=0x7f72c80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:44.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72c80a32c0 0x7f72c80a5780 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f72d4005fd0 tx=0x7f72d401a000 comp rx=0 tx=0).stop 2024-09-15T23:33:44.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 -- 172.21.15.89:0/8419927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72dc100a80 msgr2=0x7f72dc1973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:44.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72dc100a80 0x7f72dc1973a0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f72d800b4a0 tx=0x7f72d800b960 comp rx=0 tx=0).stop 2024-09-15T23:33:44.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 -- 172.21.15.89:0/8419927 shutdown_connections 2024-09-15T23:33:44.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72c80a32c0 0x7f72c80a5780 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72dc106a80 0x7f72dc19bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72dc101420 0x7f72dc1978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 --2- 172.21.15.89:0/8419927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72dc100a80 0x7f72dc1973a0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:44.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 -- 172.21.15.89:0/8419927 >> 172.21.15.89:0/8419927 conn(0x7f72dc0fc7a0 msgr2=0x7f72dc0fcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:44.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 -- 172.21.15.89:0/8419927 shutdown_connections 2024-09-15T23:33:44.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:44.361+0000 7f72e4336640 1 -- 172.21.15.89:0/8419927 wait complete. 2024-09-15T23:33:44.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:44 smithi089 ceph-mon[30933]: pgmap v641: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 KiB/s wr, 14 op/s 2024-09-15T23:33:44.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:44 smithi120 ceph-mon[35250]: pgmap v641: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 KiB/s wr, 14 op/s 2024-09-15T23:33:44.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:44 smithi174 ceph-mon[35586]: pgmap v641: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 KiB/s wr, 14 op/s 2024-09-15T23:33:46.043 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:33:46.043 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:33:46.292 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:46.414 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:46 smithi089 ceph-mon[30933]: pgmap v642: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.9 KiB/s wr, 13 op/s 2024-09-15T23:33:46.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:46 smithi120 ceph-mon[35250]: pgmap v642: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.9 KiB/s wr, 13 op/s 2024-09-15T23:33:46.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:46 smithi174 ceph-mon[35586]: pgmap v642: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.9 KiB/s wr, 13 op/s 2024-09-15T23:33:46.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.803+0000 7fd638b27640 1 -- 172.21.15.89:0/3738696640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd634102ef0 msgr2=0x7fd63410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.803+0000 7fd638b27640 1 --2- 172.21.15.89:0/3738696640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd634102ef0 0x7fd63410da40 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009ca0 tx=0x7fd62c02a950 comp rx=0 tx=0).stop 2024-09-15T23:33:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.804+0000 7fd638b27640 1 -- 172.21.15.89:0/3738696640 shutdown_connections 2024-09-15T23:33:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.804+0000 7fd638b27640 1 --2- 172.21.15.89:0/3738696640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd634102ef0 0x7fd63410da40 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.804+0000 7fd638b27640 1 --2- 172.21.15.89:0/3738696640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd634102570 0x7fd6341029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.804+0000 7fd638b27640 1 --2- 172.21.15.89:0/3738696640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd634100e20 0x7fd634101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.804+0000 7fd638b27640 1 -- 172.21.15.89:0/3738696640 >> 172.21.15.89:0/3738696640 conn(0x7fd6340fc7d0 msgr2=0x7fd6340febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.804+0000 7fd638b27640 1 -- 172.21.15.89:0/3738696640 shutdown_connections 2024-09-15T23:33:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.804+0000 7fd638b27640 1 -- 172.21.15.89:0/3738696640 wait complete. 2024-09-15T23:33:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.804+0000 7fd638b27640 1 Processor -- start 2024-09-15T23:33:46.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.804+0000 7fd638b27640 1 -- start start 2024-09-15T23:33:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd638b27640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd634100e20 0x7fd6341992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd638b27640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd634102570 0x7fd6341997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd638b27640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd634102ef0 0x7fd6341a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd638b27640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd634110fa0 con 0x7fd634102570 2024-09-15T23:33:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd638b27640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd634110e20 con 0x7fd634102ef0 2024-09-15T23:33:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd638b27640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd634111120 con 0x7fd634100e20 2024-09-15T23:33:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd629d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd634102570 0x7fd6341997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd629d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd634102570 0x7fd6341997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49030/0 (socket says 172.21.15.89:49030) 2024-09-15T23:33:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd629d9b640 1 -- 172.21.15.89:0/406785555 learned_addr learned my addr 172.21.15.89:0/406785555 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:46.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd63259c640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd634100e20 0x7fd6341992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd632d9d640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd634102ef0 0x7fd6341a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd629d9b640 1 -- 172.21.15.89:0/406785555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd634100e20 msgr2=0x7fd6341992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd629d9b640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd634100e20 0x7fd6341992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd629d9b640 1 -- 172.21.15.89:0/406785555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd634102ef0 msgr2=0x7fd6341a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:46.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd629d9b640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd634102ef0 0x7fd6341a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd629d9b640 1 -- 172.21.15.89:0/406785555 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6341a4870 con 0x7fd634102570 2024-09-15T23:33:46.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd629d9b640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd634102570 0x7fd6341997f0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd61c00b4a0 tx=0x7fd61c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd62bfff640 1 -- 172.21.15.89:0/406785555 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c014070 con 0x7fd634102570 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd62bfff640 1 -- 172.21.15.89:0/406785555 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd61c010af0 con 0x7fd634102570 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.805+0000 7fd638b27640 1 -- 172.21.15.89:0/406785555 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6341a4a40 con 0x7fd634102570 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.806+0000 7fd62bfff640 1 -- 172.21.15.89:0/406785555 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c017d10 con 0x7fd634102570 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.806+0000 7fd638b27640 1 -- 172.21.15.89:0/406785555 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6341a4f50 con 0x7fd634102570 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.808+0000 7fd63af00740 1 Processor -- start 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63af00740 1 -- start start 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd62bfff640 1 -- 172.21.15.89:0/406785555 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd61c017070 con 0x7fd634102570 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63af00740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63af00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63af00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8dc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63af00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563a5e834280 con 0x563a5e8cf480 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63af00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563a5e7dbb10 con 0x563a5e8d9d40 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63af00740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563a5e830240 con 0x563a5e8ceb60 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49044/0 (socket says 172.21.15.89:49044) 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63259c640 1 -- 172.21.15.89:0/2892748723 learned_addr learned my addr 172.21.15.89:0/2892748723 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd632d9d640 1 --2- 172.21.15.89:0/2892748723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63259c640 1 -- 172.21.15.89:0/2892748723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 msgr2=0x563a5e8cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd629d9b640 1 --2- 172.21.15.89:0/2892748723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8dc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63259c640 1 --2- 172.21.15.89:0/2892748723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63259c640 1 -- 172.21.15.89:0/2892748723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 msgr2=0x563a5e8dc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63259c640 1 --2- 172.21.15.89:0/2892748723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8dc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd63259c640 1 -- 172.21.15.89:0/2892748723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a5e7e81f0 con 0x563a5e8cf480 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd629d9b640 1 --2- 172.21.15.89:0/2892748723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8dc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.809+0000 7fd62bfff640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6200a3240 0x7fd6200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.810+0000 7fd632d9d640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6200a3240 0x7fd6200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.810+0000 7fd63259c640 1 --2- 172.21.15.89:0/2892748723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d9800 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fd62400cc60 tx=0x7fd62400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91ed0d5c7c8aaf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.810+0000 7fd62bfff640 1 -- 172.21.15.89:0/406785555 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd61c0cf120 con 0x7fd634102570 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.810+0000 7fd628d99640 1 -- 172.21.15.89:0/2892748723 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624014070 con 0x563a5e8cf480 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.810+0000 7fd628d99640 1 -- 172.21.15.89:0/2892748723 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6240047a0 con 0x563a5e8cf480 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.810+0000 7fd628d99640 1 -- 172.21.15.89:0/2892748723 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624010760 con 0x563a5e8cf480 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 -- 172.21.15.89:0/2892748723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 msgr2=0x563a5e8d9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 --2- 172.21.15.89:0/2892748723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d9800 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fd62400cc60 tx=0x7fd62400e720 comp rx=0 tx=0).stop 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd632d9d640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6200a3240 0x7fd6200a5700 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd62c005cc0 tx=0x7fd62c005c30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:46.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 -- 172.21.15.89:0/2892748723 shutdown_connections 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 --2- 172.21.15.89:0/2892748723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8dc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 --2- 172.21.15.89:0/2892748723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d9800 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 --2- 172.21.15.89:0/2892748723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 -- 172.21.15.89:0/2892748723 >> 172.21.15.89:0/2892748723 conn(0x563a5e8cb0f0 msgr2=0x563a5e8cb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 -- 172.21.15.89:0/2892748723 shutdown_connections 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 -- 172.21.15.89:0/2892748723 wait complete. 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 -- start start 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 Processor -- start 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.811+0000 7fd63af00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8d7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd63af00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563a5e8dc910 con 0x563a5e8cf480 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd63af00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563a5e7eef10 con 0x563a5e8d9d40 2024-09-15T23:33:46.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd63af00740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563a5e8dca90 con 0x563a5e8ceb60 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd63af00740 10 client.0 _ll_register_callbacks cb 0x563a5e7b7a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd629d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd629d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49048/0 (socket says 172.21.15.89:49048) 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd629d9b640 1 -- 172.21.15.89:0/1823781925 learned_addr learned my addr 172.21.15.89:0/1823781925 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd63259c640 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd632d9d640 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8d7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd629d9b640 1 -- 172.21.15.89:0/1823781925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 msgr2=0x563a5e8d6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd629d9b640 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd629d9b640 1 -- 172.21.15.89:0/1823781925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 msgr2=0x563a5e8d7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd629d9b640 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8d7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd629d9b640 1 -- 172.21.15.89:0/1823781925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a5e5a56a0 con 0x563a5e8cf480 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd629d9b640 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d70f0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fd61c00b470 tx=0x7fd61c09adb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87ca5551ca490c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:46.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd619ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:46.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd619ffb640 1 -- 172.21.15.89:0/1823781925 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c01e070 con 0x563a5e8cf480 2024-09-15T23:33:46.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd619ffb640 1 -- 172.21.15.89:0/1823781925 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd61c09f490 con 0x563a5e8cf480 2024-09-15T23:33:46.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd63af00740 1 -- 172.21.15.89:0/1823781925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6240047a0 con 0x563a5e8cf480 2024-09-15T23:33:46.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd619ffb640 1 -- 172.21.15.89:0/1823781925 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd61c09f700 con 0x563a5e8cf480 2024-09-15T23:33:46.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd63af00740 10 client.18723 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:46.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.812+0000 7fd63af00740 1 -- 172.21.15.89:0/1823781925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563a5e8e84a0 con 0x563a5e8cf480 2024-09-15T23:33:46.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd619ffb640 1 -- 172.21.15.89:0/1823781925 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6240047a0 con 0x563a5e8cf480 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd619ffb640 1 -- 172.21.15.89:0/1823781925 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd61c09a7a0 con 0x563a5e8cf480 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd63af00740 10 client.18723 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd63af00740 10 client.18723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd63af00740 20 client.18723 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd63af00740 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a5e8e9960 0x563a5e8ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd63af00740 4 client.18723 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd63af00740 1 -- 172.21.15.89:0/1823781925 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a5e8417f0 con 0x563a5e8e9960 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd63259c640 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a5e8e9960 0x563a5e8ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd619ffb640 10 client.18723 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.813+0000 7fd63259c640 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a5e8e9960 0x563a5e8ebdb0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=772841750b238d4a server_cookie=77fb18e0ac52175b in_seq=0 out_seq=0 2024-09-15T23:33:46.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd619ffb640 1 -- 172.21.15.89:0/1823781925 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563a5e8417f0 con 0x563a5e8e9960 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd619ffb640 10 client.18723 handle_command_reply: tid=0 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 client.18723 shutdown 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 -- 172.21.15.89:0/1823781925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 msgr2=0x563a5e8d70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d70f0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fd61c00b470 tx=0x7fd61c09adb0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 -- 172.21.15.89:0/1823781925 shutdown_connections 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a5e8e9960 0x563a5e8ebdb0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8d7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd619ffb640 0 client.18723 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8d70f0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 --2- 172.21.15.89:0/1823781925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 -- 172.21.15.89:0/1823781925 >> 172.21.15.89:0/1823781925 conn(0x563a5e8cb0f0 msgr2=0x563a5e8d0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 -- 172.21.15.89:0/1823781925 shutdown_connections 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.814+0000 7fd63af00740 1 -- 172.21.15.89:0/1823781925 wait complete. 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.815+0000 7fd63af00740 20 client.18723 trim_cache size 0 max 16384 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.831+0000 7fd63af00740 1 Processor -- start 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.831+0000 7fd63af00740 1 -- start start 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.831+0000 7fd63af00740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.831+0000 7fd63af00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8f1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.831+0000 7fd63af00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8f1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.831+0000 7fd63af00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341aee10 con 0x563a5e8cf480 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.831+0000 7fd63af00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341b0950 con 0x563a5e8d9d40 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.831+0000 7fd63af00740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341af4e0 con 0x563a5e8ceb60 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.831+0000 7fd629d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8f1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd629d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8f1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49054/0 (socket says 172.21.15.89:49054) 2024-09-15T23:33:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd629d9b640 1 -- 172.21.15.89:0/3787872683 learned_addr learned my addr 172.21.15.89:0/3787872683 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd629d9b640 1 -- 172.21.15.89:0/3787872683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 msgr2=0x563a5e8d8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd629d9b640 1 --2- 172.21.15.89:0/3787872683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd629d9b640 1 -- 172.21.15.89:0/3787872683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 msgr2=0x563a5e8f1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd629d9b640 1 --2- 172.21.15.89:0/3787872683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8f1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd629d9b640 1 -- 172.21.15.89:0/3787872683 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a5e8cf2b0 con 0x563a5e8cf480 2024-09-15T23:33:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd629d9b640 1 --2- 172.21.15.89:0/3787872683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8f1650 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd61c09ac00 tx=0x7fd61c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beea341b8860351e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd61bfff640 1 -- 172.21.15.89:0/3787872683 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c09ba50 con 0x563a5e8cf480 2024-09-15T23:33:46.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd61bfff640 1 -- 172.21.15.89:0/3787872683 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd61c0a3fc0 con 0x563a5e8cf480 2024-09-15T23:33:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.832+0000 7fd61bfff640 1 -- 172.21.15.89:0/3787872683 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c0970a0 con 0x563a5e8cf480 2024-09-15T23:33:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 -- 172.21.15.89:0/3787872683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 msgr2=0x563a5e8f1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 --2- 172.21.15.89:0/3787872683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8f1650 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fd61c09ac00 tx=0x7fd61c000bc0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 -- 172.21.15.89:0/3787872683 shutdown_connections 2024-09-15T23:33:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 --2- 172.21.15.89:0/3787872683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8d9d40 0x563a5e8f1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 --2- 172.21.15.89:0/3787872683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8cf480 0x563a5e8f1650 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 --2- 172.21.15.89:0/3787872683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 -- 172.21.15.89:0/3787872683 >> 172.21.15.89:0/3787872683 conn(0x563a5e8cb0f0 msgr2=0x563a5e8d8300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 -- 172.21.15.89:0/3787872683 shutdown_connections 2024-09-15T23:33:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 -- 172.21.15.89:0/3787872683 wait complete. 2024-09-15T23:33:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 -- start start 2024-09-15T23:33:46.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 Processor -- start 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8cf480 0x563a5e8d41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8d9d40 0x563a5e8d46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341b0280 con 0x563a5e8d9d40 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341afbb0 con 0x563a5e8cf480 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd6341acdf0 con 0x563a5e8ceb60 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63af00740 10 client.0 _ll_register_callbacks cb 0x7fd61c017630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd632d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8d9d40 0x563a5e8d46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd632d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8d9d40 0x563a5e8d46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49068/0 (socket says 172.21.15.89:49068) 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd632d9d640 1 -- 172.21.15.89:0/1729076510 learned_addr learned my addr 172.21.15.89:0/1729076510 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.833+0000 7fd63259c640 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd629d9b640 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8cf480 0x563a5e8d41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd632d9d640 1 -- 172.21.15.89:0/1729076510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 msgr2=0x563a5e8d3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd632d9d640 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd632d9d640 1 -- 172.21.15.89:0/1729076510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8cf480 msgr2=0x563a5e8d41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd632d9d640 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8cf480 0x563a5e8d41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd632d9d640 1 -- 172.21.15.89:0/1729076510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a5e9536a0 con 0x563a5e8d9d40 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd632d9d640 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8d9d40 0x563a5e8d46f0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd62c02f4f0 tx=0x7fd62c02fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486bf325aca1ca39 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd619ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd619ffb640 1 -- 172.21.15.89:0/1729076510 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c034040 con 0x563a5e8d9d40 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd619ffb640 1 -- 172.21.15.89:0/1729076510 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd62c003f70 con 0x563a5e8d9d40 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd63af00740 1 -- 172.21.15.89:0/1729076510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd61c0a3fc0 con 0x563a5e8d9d40 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd619ffb640 1 -- 172.21.15.89:0/1729076510 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd62c004170 con 0x563a5e8d9d40 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd63af00740 10 client.18729 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd63af00740 1 -- 172.21.15.89:0/1729076510 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563a5e9549c0 con 0x563a5e8d9d40 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.834+0000 7fd619ffb640 1 -- 172.21.15.89:0/1729076510 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd61c0a3fc0 con 0x563a5e8d9d40 2024-09-15T23:33:46.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.835+0000 7fd619ffb640 1 -- 172.21.15.89:0/1729076510 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd62c051020 con 0x563a5e8d9d40 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.835+0000 7fd63af00740 10 client.18729 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.835+0000 7fd63af00740 10 client.18729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.835+0000 7fd63af00740 20 client.18729 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.835+0000 7fd63af00740 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a5e955140 0x563a5e957560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.835+0000 7fd63af00740 4 client.18729 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.835+0000 7fd63af00740 1 -- 172.21.15.89:0/1729076510 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563a5e83bd00 con 0x563a5e955140 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.835+0000 7fd63259c640 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a5e955140 0x563a5e957560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.835+0000 7fd619ffb640 10 client.18729 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.835+0000 7fd63259c640 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a5e955140 0x563a5e957560 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54171839a013c9f9 server_cookie=ddd9f07e2713b41f in_seq=0 out_seq=0 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd619ffb640 1 -- 172.21.15.89:0/1729076510 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563a5e83bd00 con 0x563a5e955140 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd619ffb640 10 client.18729 handle_command_reply: tid=0 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 client.18729 shutdown 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 -- 172.21.15.89:0/1729076510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8d9d40 msgr2=0x563a5e8d46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:46.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8d9d40 0x563a5e8d46f0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fd62c02f4f0 tx=0x7fd62c02fd40 comp rx=0 tx=0).stop 2024-09-15T23:33:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 -- 172.21.15.89:0/1729076510 shutdown_connections 2024-09-15T23:33:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a5e955140 0x563a5e957560 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a5e8d9d40 0x563a5e8d46f0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a5e8cf480 0x563a5e8d41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 --2- 172.21.15.89:0/1729076510 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a5e8ceb60 0x563a5e8d3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 -- 172.21.15.89:0/1729076510 >> 172.21.15.89:0/1729076510 conn(0x563a5e8cb0f0 msgr2=0x563a5e8ca270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd619ffb640 0 client.18729 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 -- 172.21.15.89:0/1729076510 shutdown_connections 2024-09-15T23:33:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 1 -- 172.21.15.89:0/1729076510 wait complete. 2024-09-15T23:33:46.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.837+0000 7fd63af00740 20 client.18729 trim_cache size 0 max 16384 2024-09-15T23:33:46.861 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:33:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:33:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "d68d0a61-489e-4f51-bdff-c25c9ca4e0b0", 2024-09-15T23:33:46.861 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:33:46.861 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 -- 172.21.15.89:0/406785555 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6200a3240 msgr2=0x7fd6200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6200a3240 0x7fd6200a5700 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd62c005cc0 tx=0x7fd62c005c30 comp rx=0 tx=0).stop 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 -- 172.21.15.89:0/406785555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd634102570 msgr2=0x7fd6341997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd634102570 0x7fd6341997f0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd61c00b4a0 tx=0x7fd61c00b960 comp rx=0 tx=0).stop 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 -- 172.21.15.89:0/406785555 shutdown_connections 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd6200a3240 0x7fd6200a5700 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd634102ef0 0x7fd6341a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd634102570 0x7fd6341997f0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 --2- 172.21.15.89:0/406785555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd634100e20 0x7fd6341992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 -- 172.21.15.89:0/406785555 >> 172.21.15.89:0/406785555 conn(0x7fd6340fc7d0 msgr2=0x7fd634106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:46.862 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 -- 172.21.15.89:0/406785555 shutdown_connections 2024-09-15T23:33:46.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:46.838+0000 7fd638b27640 1 -- 172.21.15.89:0/406785555 wait complete. 2024-09-15T23:33:47.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:47 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:33:47.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:47 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:47.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:47 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:33:47.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:47 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:47.586 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:33:47.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:47 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:33:47.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:47 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:47.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:47 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:33:47.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:47 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:47.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:47 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:33:47.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:47 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:47.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:47 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:33:47.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:47 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:33:47.831 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:48.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.311+0000 7f87fdfdb640 1 -- 172.21.15.89:0/3153625904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f81082e0 msgr2=0x7f87f810ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:48.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.311+0000 7f87fdfdb640 1 --2- 172.21.15.89:0/3153625904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f81082e0 0x7f87f810ec50 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009cd0 tx=0x7f87f002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.311+0000 7f87fdfdb640 1 -- 172.21.15.89:0/3153625904 shutdown_connections 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.311+0000 7f87fdfdb640 1 --2- 172.21.15.89:0/3153625904 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f81082e0 0x7f87f810ec50 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.311+0000 7f87fdfdb640 1 --2- 172.21.15.89:0/3153625904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f8103760 0x7f87f8107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.311+0000 7f87fdfdb640 1 --2- 172.21.15.89:0/3153625904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87f8102db0 0x7f87f8103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.311+0000 7f87fdfdb640 1 -- 172.21.15.89:0/3153625904 >> 172.21.15.89:0/3153625904 conn(0x7f87f80fc7d0 msgr2=0x7f87f80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.311+0000 7f87fdfdb640 1 -- 172.21.15.89:0/3153625904 shutdown_connections 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.311+0000 7f87fdfdb640 1 -- 172.21.15.89:0/3153625904 wait complete. 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87fdfdb640 1 Processor -- start 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87fdfdb640 1 -- start start 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87fdfdb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87f8102db0 0x7f87f8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87fdfdb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f8103760 0x7f87f8198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87fdfdb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f81082e0 0x7f87f819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87fdfdb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f87f81120b0 con 0x7f87f81082e0 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87fdfdb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f87f8111f30 con 0x7f87f8102db0 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87fdfdb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f87f8112230 con 0x7f87f8103760 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87f7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f81082e0 0x7f87f819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87f7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f81082e0 0x7f87f819ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49080/0 (socket says 172.21.15.89:49080) 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87f7fff640 1 -- 172.21.15.89:0/4167069812 learned_addr learned my addr 172.21.15.89:0/4167069812 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.312+0000 7f87f77fe640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87f8102db0 0x7f87f8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87effff640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f8103760 0x7f87f8198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87f7fff640 1 -- 172.21.15.89:0/4167069812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f8103760 msgr2=0x7f87f8198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87f7fff640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f8103760 0x7f87f8198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87f7fff640 1 -- 172.21.15.89:0/4167069812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87f8102db0 msgr2=0x7f87f8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87f7fff640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87f8102db0 0x7f87f8198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87f7fff640 1 -- 172.21.15.89:0/4167069812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f819d510 con 0x7f87f81082e0 2024-09-15T23:33:48.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87f7fff640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f81082e0 0x7f87f819ce30 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f87f0005cc0 tx=0x7f87f00093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87f57fa640 1 -- 172.21.15.89:0/4167069812 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f0004040 con 0x7f87f81082e0 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87f57fa640 1 -- 172.21.15.89:0/4167069812 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87f00046c0 con 0x7f87f81082e0 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87fdfdb640 1 -- 172.21.15.89:0/4167069812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f819d7a0 con 0x7f87f81082e0 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87f57fa640 1 -- 172.21.15.89:0/4167069812 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f0030e00 con 0x7f87f81082e0 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.313+0000 7f87fdfdb640 1 -- 172.21.15.89:0/4167069812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f81a50e0 con 0x7f87f81082e0 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.315+0000 7f87f57fa640 1 -- 172.21.15.89:0/4167069812 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f87f00305a0 con 0x7f87f81082e0 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f88003b4740 1 Processor -- start 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f88003b4740 1 -- start start 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f88003b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dbf84a0 0x55965dbf8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f88003b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc03140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f88003b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc05a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f88003b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55965db5e2e0 con 0x55965dc03680 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f88003b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55965db05ce0 con 0x55965dbf8dc0 2024-09-15T23:33:48.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f88003b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55965db5a2a0 con 0x55965dbf84a0 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f87f57fa640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87e40a3240 0x7f87e40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f87effff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc05a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f87effff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc05a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49086/0 (socket says 172.21.15.89:49086) 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f87effff640 1 -- 172.21.15.89:0/3795192992 learned_addr learned my addr 172.21.15.89:0/3795192992 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f87f7fff640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87e40a3240 0x7f87e40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f87f77fe640 1 --2- 172.21.15.89:0/3795192992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc03140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f87f57fa640 1 -- 172.21.15.89:0/4167069812 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f87f002e030 con 0x7f87f81082e0 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.316+0000 7f87effff640 1 --2- 172.21.15.89:0/3795192992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dbf84a0 0x55965dbf8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.317+0000 7f87effff640 1 -- 172.21.15.89:0/3795192992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dbf84a0 msgr2=0x55965dbf8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.317+0000 7f87effff640 1 --2- 172.21.15.89:0/3795192992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dbf84a0 0x55965dbf8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.317+0000 7f87effff640 1 -- 172.21.15.89:0/3795192992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 msgr2=0x55965dc03140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.317+0000 7f87effff640 1 --2- 172.21.15.89:0/3795192992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc03140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.317+0000 7f87effff640 1 -- 172.21.15.89:0/3795192992 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55965daea440 con 0x55965dc03680 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.317+0000 7f87effff640 1 --2- 172.21.15.89:0/3795192992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc05a70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f87e0013500 tx=0x7f87e000f620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656729c81a351c12 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.317+0000 7f87f7fff640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87e40a3240 0x7f87e40a5700 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f87f00b9b40 tx=0x7f87f0088070 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.317+0000 7f87edffb640 1 -- 172.21.15.89:0/3795192992 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e0019030 con 0x55965dc03680 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.317+0000 7f87edffb640 1 -- 172.21.15.89:0/3795192992 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87e0005500 con 0x55965dc03680 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.317+0000 7f87edffb640 1 -- 172.21.15.89:0/3795192992 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e00116a0 con 0x55965dc03680 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 -- 172.21.15.89:0/3795192992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 msgr2=0x55965dc05a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 --2- 172.21.15.89:0/3795192992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc05a70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f87e0013500 tx=0x7f87e000f620 comp rx=0 tx=0).stop 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 -- 172.21.15.89:0/3795192992 shutdown_connections 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 --2- 172.21.15.89:0/3795192992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc05a70 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 --2- 172.21.15.89:0/3795192992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc03140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 --2- 172.21.15.89:0/3795192992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dbf84a0 0x55965dbf8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 -- 172.21.15.89:0/3795192992 >> 172.21.15.89:0/3795192992 conn(0x55965dbf4a30 msgr2=0x55965dbf4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 -- 172.21.15.89:0/3795192992 shutdown_connections 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 -- 172.21.15.89:0/3795192992 wait complete. 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 -- start start 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 Processor -- start 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf84a0 0x55965dc02f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc00200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dc03680 0x55965dc00740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55965dc06250 con 0x55965dbf84a0 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55965db19050 con 0x55965dbf8dc0 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.318+0000 7f88003b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55965dc063d0 con 0x55965dc03680 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f88003b4740 10 client.0 _ll_register_callbacks cb 0x55965db59160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87effff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf84a0 0x55965dc02f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87effff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf84a0 0x55965dc02f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49096/0 (socket says 172.21.15.89:49096) 2024-09-15T23:33:48.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87effff640 1 -- 172.21.15.89:0/479436268 learned_addr learned my addr 172.21.15.89:0/479436268 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87f77fe640 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc00200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87effff640 1 -- 172.21.15.89:0/479436268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dc03680 msgr2=0x55965dc00740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87effff640 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dc03680 0x55965dc00740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87effff640 1 -- 172.21.15.89:0/479436268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 msgr2=0x55965dc00200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87effff640 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc00200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87effff640 1 -- 172.21.15.89:0/479436268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55965d8cf3c0 con 0x55965dbf84a0 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87f77fe640 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc00200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87effff640 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf84a0 0x55965dc02f50 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f87e0016040 tx=0x7f87e000f750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaced7f5b84fc9dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87d6ffd640 1 -- 172.21.15.89:0/479436268 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e0019040 con 0x55965dbf84a0 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87d6ffd640 1 -- 172.21.15.89:0/479436268 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f87e001e460 con 0x55965dbf84a0 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f88003b4740 1 -- 172.21.15.89:0/479436268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87e0005500 con 0x55965dbf84a0 2024-09-15T23:33:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f87d6ffd640 1 -- 172.21.15.89:0/479436268 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f87e0005360 con 0x55965dbf84a0 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f88003b4740 10 client.18741 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.319+0000 7f88003b4740 1 -- 172.21.15.89:0/479436268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55965dc11ba0 con 0x55965dbf84a0 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.320+0000 7f87d6ffd640 1 -- 172.21.15.89:0/479436268 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87e0005500 con 0x55965dbf84a0 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.320+0000 7f87d6ffd640 1 -- 172.21.15.89:0/479436268 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f87e0025070 con 0x55965dbf84a0 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.320+0000 7f88003b4740 10 client.18741 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.320+0000 7f88003b4740 10 client.18741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.320+0000 7f88003b4740 20 client.18741 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.320+0000 7f88003b4740 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55965dc12fb0 0x55965dc153a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.320+0000 7f88003b4740 4 client.18741 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.320+0000 7f88003b4740 1 -- 172.21.15.89:0/479436268 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55965db6b850 con 0x55965dc12fb0 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.320+0000 7f87f77fe640 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55965dc12fb0 0x55965dc153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f87d6ffd640 10 client.18741 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f87f77fe640 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55965dc12fb0 0x55965dc153a0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c029fb8e07dfb6a6 server_cookie=a0f7870131a53b97 in_seq=0 out_seq=0 2024-09-15T23:33:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f87d6ffd640 1 -- 172.21.15.89:0/479436268 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55965db6b850 con 0x55965dc12fb0 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f87d6ffd640 10 client.18741 handle_command_reply: tid=0 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f88003b4740 1 client.18741 shutdown 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f88003b4740 1 -- 172.21.15.89:0/479436268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf84a0 msgr2=0x55965dc02f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f88003b4740 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf84a0 0x55965dc02f50 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f87e0016040 tx=0x7f87e000f750 comp rx=0 tx=0).stop 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f88003b4740 1 -- 172.21.15.89:0/479436268 shutdown_connections 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f88003b4740 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55965dc12fb0 0x55965dc153a0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f88003b4740 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dc03680 0x55965dc00740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f88003b4740 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc00200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f88003b4740 1 --2- 172.21.15.89:0/479436268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf84a0 0x55965dc02f50 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f87d6ffd640 0 client.18741 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.321+0000 7f88003b4740 1 -- 172.21.15.89:0/479436268 >> 172.21.15.89:0/479436268 conn(0x55965dbf4a30 msgr2=0x55965dbfa710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.322+0000 7f88003b4740 1 -- 172.21.15.89:0/479436268 shutdown_connections 2024-09-15T23:33:48.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.322+0000 7f88003b4740 1 -- 172.21.15.89:0/479436268 wait complete. 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.322+0000 7f88003b4740 20 client.18741 trim_cache size 0 max 16384 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.338+0000 7f88003b4740 1 Processor -- start 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.338+0000 7f88003b4740 1 -- start start 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f88003b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf84a0 0x55965dc0f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f88003b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf8dc0 0x55965dbfb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f88003b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dc03680 0x55965dc1ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f88003b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f87f81aee30 con 0x55965dbf8dc0 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f88003b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f87f81b0970 con 0x55965dbf84a0 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f88003b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f87f81af500 con 0x55965dc03680 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf8dc0 0x55965dbfb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87f77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf8dc0 0x55965dbfb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49100/0 (socket says 172.21.15.89:49100) 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87f77fe640 1 -- 172.21.15.89:0/2048004486 learned_addr learned my addr 172.21.15.89:0/2048004486 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87f77fe640 1 -- 172.21.15.89:0/2048004486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dc03680 msgr2=0x55965dc1ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:48.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87effff640 1 --2- 172.21.15.89:0/2048004486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf84a0 0x55965dc0f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87f77fe640 1 --2- 172.21.15.89:0/2048004486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dc03680 0x55965dc1ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87f77fe640 1 -- 172.21.15.89:0/2048004486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf84a0 msgr2=0x55965dc0f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:48.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87f77fe640 1 --2- 172.21.15.89:0/2048004486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf84a0 0x55965dc0f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87f77fe640 1 -- 172.21.15.89:0/2048004486 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55965dbf4860 con 0x55965dbf8dc0 2024-09-15T23:33:48.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87f77fe640 1 --2- 172.21.15.89:0/2048004486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf8dc0 0x55965dbfb7a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f87e8005d00 tx=0x7f87e8012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48d3245918d72b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:48.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.339+0000 7f87ed7fa640 1 -- 172.21.15.89:0/2048004486 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e8009e60 con 0x55965dbf8dc0 2024-09-15T23:33:48.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f87ed7fa640 1 -- 172.21.15.89:0/2048004486 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87e80035f0 con 0x55965dbf8dc0 2024-09-15T23:33:48.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f87ed7fa640 1 -- 172.21.15.89:0/2048004486 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e801d690 con 0x55965dbf8dc0 2024-09-15T23:33:48.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f88003b4740 1 -- 172.21.15.89:0/2048004486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf8dc0 msgr2=0x55965dbfb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:48.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f88003b4740 1 --2- 172.21.15.89:0/2048004486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf8dc0 0x55965dbfb7a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f87e8005d00 tx=0x7f87e8012820 comp rx=0 tx=0).stop 2024-09-15T23:33:48.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f88003b4740 1 -- 172.21.15.89:0/2048004486 shutdown_connections 2024-09-15T23:33:48.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f88003b4740 1 --2- 172.21.15.89:0/2048004486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dc03680 0x55965dc1ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f88003b4740 1 --2- 172.21.15.89:0/2048004486 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dbf8dc0 0x55965dbfb7a0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f88003b4740 1 --2- 172.21.15.89:0/2048004486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf84a0 0x55965dc0f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f88003b4740 1 -- 172.21.15.89:0/2048004486 >> 172.21.15.89:0/2048004486 conn(0x55965dbf4a30 msgr2=0x55965dbfb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:48.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f88003b4740 1 -- 172.21.15.89:0/2048004486 shutdown_connections 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.340+0000 7f88003b4740 1 -- 172.21.15.89:0/2048004486 wait complete. 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f88003b4740 1 -- start start 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f88003b4740 1 Processor -- start 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f88003b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dbf84a0 0x55965dc06680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f88003b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc06bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f88003b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc7bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f88003b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f87f81b02a0 con 0x55965dc03680 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f88003b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f87f81afbd0 con 0x55965dbf8dc0 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f88003b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f87f81ace10 con 0x55965dbf84a0 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f88003b4740 10 client.0 _ll_register_callbacks cb 0x7f87e000ff50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f87effff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc7bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f87effff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc7bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49102/0 (socket says 172.21.15.89:49102) 2024-09-15T23:33:48.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f87effff640 1 -- 172.21.15.89:0/477688483 learned_addr learned my addr 172.21.15.89:0/477688483 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f87f77fe640 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc06bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.341+0000 7f87effff640 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dbf84a0 0x55965dc06680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87effff640 1 -- 172.21.15.89:0/477688483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dbf84a0 msgr2=0x55965dc06680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87effff640 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dbf84a0 0x55965dc06680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87effff640 1 -- 172.21.15.89:0/477688483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 msgr2=0x55965dc06bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87effff640 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc06bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87effff640 1 -- 172.21.15.89:0/477688483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55965dc074a0 con 0x55965dc03680 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87f77fe640 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc06bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87effff640 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc7bb40 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f87e000f720 tx=0x7f87e000f620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c296cd936c1d71 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87d6ffd640 1 -- 172.21.15.89:0/477688483 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e0019040 con 0x55965dc03680 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87d6ffd640 1 -- 172.21.15.89:0/477688483 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f87e00114a0 con 0x55965dc03680 2024-09-15T23:33:48.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f88003b4740 1 -- 172.21.15.89:0/477688483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87e80035f0 con 0x55965dc03680 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87d6ffd640 1 -- 172.21.15.89:0/477688483 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f87e003c050 con 0x55965dc03680 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f88003b4740 10 client.18747 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f88003b4740 1 -- 172.21.15.89:0/477688483 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55965dc7d1a0 con 0x55965dc03680 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87d6ffd640 1 -- 172.21.15.89:0/477688483 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87e80035f0 con 0x55965dc03680 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f87d6ffd640 1 -- 172.21.15.89:0/477688483 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f87e0025070 con 0x55965dc03680 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f88003b4740 10 client.18747 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f88003b4740 10 client.18747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f88003b4740 20 client.18747 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f88003b4740 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55965dc7d7b0 0x55965dc7fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f88003b4740 4 client.18747 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.342+0000 7f88003b4740 1 -- 172.21.15.89:0/477688483 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55965dc00ad0 con 0x55965dc7d7b0 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.343+0000 7f87f77fe640 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55965dc7d7b0 0x55965dc7fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:48.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.343+0000 7f87d6ffd640 10 client.18747 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.343+0000 7f87f77fe640 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55965dc7d7b0 0x55965dc7fbd0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49eb24c431ca3f13 server_cookie=9ebc6fbd4fa8b3a5 in_seq=0 out_seq=0 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.343+0000 7f87d6ffd640 1 -- 172.21.15.89:0/477688483 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55965dc00ad0 con 0x55965dc7d7b0 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.343+0000 7f87d6ffd640 10 client.18747 handle_command_reply: tid=0 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.343+0000 7f88003b4740 1 client.18747 shutdown 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 1 -- 172.21.15.89:0/477688483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 msgr2=0x55965dc7bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc7bb40 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f87e000f720 tx=0x7f87e000f620 comp rx=0 tx=0).stop 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 1 -- 172.21.15.89:0/477688483 shutdown_connections 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55965dc7d7b0 0x55965dc7fbd0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55965dc03680 0x55965dc7bb40 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55965dbf8dc0 0x55965dc06bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 1 --2- 172.21.15.89:0/477688483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55965dbf84a0 0x55965dc06680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 1 -- 172.21.15.89:0/477688483 >> 172.21.15.89:0/477688483 conn(0x55965dbf4a30 msgr2=0x55965dbfb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f87d6ffd640 0 client.18747 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:48.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 1 -- 172.21.15.89:0/477688483 shutdown_connections 2024-09-15T23:33:48.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 1 -- 172.21.15.89:0/477688483 wait complete. 2024-09-15T23:33:48.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f88003b4740 20 client.18747 trim_cache size 0 max 16384 2024-09-15T23:33:48.361 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:33:48.361 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:33:48.361 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:33:48.361 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:33:48.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f87fdfdb640 1 -- 172.21.15.89:0/4167069812 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87e40a3240 msgr2=0x7f87e40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:48.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f87fdfdb640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87e40a3240 0x7f87e40a5700 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f87f00b9b40 tx=0x7f87f0088070 comp rx=0 tx=0).stop 2024-09-15T23:33:48.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f87fdfdb640 1 -- 172.21.15.89:0/4167069812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f81082e0 msgr2=0x7f87f819ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:48.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.344+0000 7f87fdfdb640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f81082e0 0x7f87f819ce30 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f87f0005cc0 tx=0x7f87f00093d0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.345+0000 7f87fdfdb640 1 -- 172.21.15.89:0/4167069812 shutdown_connections 2024-09-15T23:33:48.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.345+0000 7f87fdfdb640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f87e40a3240 0x7f87e40a5700 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.345+0000 7f87fdfdb640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87f81082e0 0x7f87f819ce30 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.345+0000 7f87fdfdb640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f8103760 0x7f87f8198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.345+0000 7f87fdfdb640 1 --2- 172.21.15.89:0/4167069812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87f8102db0 0x7f87f8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:48.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.345+0000 7f87fdfdb640 1 -- 172.21.15.89:0/4167069812 >> 172.21.15.89:0/4167069812 conn(0x7f87f80fc7d0 msgr2=0x7f87f8105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:48.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.345+0000 7f87fdfdb640 1 -- 172.21.15.89:0/4167069812 shutdown_connections 2024-09-15T23:33:48.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:48.345+0000 7f87fdfdb640 1 -- 172.21.15.89:0/4167069812 wait complete. 2024-09-15T23:33:48.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:48 smithi089 ceph-mon[30933]: pgmap v643: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.9 KiB/s wr, 16 op/s 2024-09-15T23:33:48.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:48 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:33:48.624 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:48 smithi120 ceph-mon[35250]: pgmap v643: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.9 KiB/s wr, 16 op/s 2024-09-15T23:33:48.624 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:48 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:33:48.637 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:48 smithi174 ceph-mon[35586]: pgmap v643: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.9 KiB/s wr, 16 op/s 2024-09-15T23:33:48.637 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:48 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:33:49.059 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:33:49.059 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:33:49.309 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:49.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.785+0000 7fa321c58640 1 -- 172.21.15.89:0/1240943720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa31c101df0 msgr2=0x7fa31c1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:49.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.785+0000 7fa321c58640 1 --2- 172.21.15.89:0/1240943720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa31c101df0 0x7fa31c1021d0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009900 tx=0x7fa30c02a410 comp rx=0 tx=0).stop 2024-09-15T23:33:49.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.785+0000 7fa321c58640 1 -- 172.21.15.89:0/1240943720 shutdown_connections 2024-09-15T23:33:49.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.785+0000 7fa321c58640 1 --2- 172.21.15.89:0/1240943720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa31c106f20 0x7fa31c10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.785+0000 7fa321c58640 1 --2- 172.21.15.89:0/1240943720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa31c1027a0 0x7fa31c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.785+0000 7fa321c58640 1 --2- 172.21.15.89:0/1240943720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa31c101df0 0x7fa31c1021d0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.785+0000 7fa321c58640 1 -- 172.21.15.89:0/1240943720 >> 172.21.15.89:0/1240943720 conn(0x7fa31c0fd8f0 msgr2=0x7fa31c0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.785+0000 7fa321c58640 1 -- 172.21.15.89:0/1240943720 shutdown_connections 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.785+0000 7fa321c58640 1 -- 172.21.15.89:0/1240943720 wait complete. 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.785+0000 7fa321c58640 1 Processor -- start 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa321c58640 1 -- start start 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa321c58640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa31c101df0 0x7fa31c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa321c58640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa31c1027a0 0x7fa31c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa321c58640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa31c106f20 0x7fa31c19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa321c58640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c111f80 con 0x7fa31c101df0 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa321c58640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c111e00 con 0x7fa31c106f20 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa321c58640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c112100 con 0x7fa31c1027a0 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa31b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa31c101df0 0x7fa31c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa31b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa31c101df0 0x7fa31c198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49120/0 (socket says 172.21.15.89:49120) 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa31b7fe640 1 -- 172.21.15.89:0/4005685057 learned_addr learned my addr 172.21.15.89:0/4005685057 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa31affd640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa31c1027a0 0x7fa31c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa31bfff640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa31c106f20 0x7fa31c19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa31b7fe640 1 -- 172.21.15.89:0/4005685057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa31c1027a0 msgr2=0x7fa31c198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa31b7fe640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa31c1027a0 0x7fa31c198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa31b7fe640 1 -- 172.21.15.89:0/4005685057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa31c106f20 msgr2=0x7fa31c19cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa31b7fe640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa31c106f20 0x7fa31c19cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.786+0000 7fa31b7fe640 1 -- 172.21.15.89:0/4005685057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31c19d3f0 con 0x7fa31c101df0 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.787+0000 7fa31b7fe640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa31c101df0 0x7fa31c198440 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fa30c005ab0 tx=0x7fa30c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.787+0000 7fa318ff9640 1 -- 172.21.15.89:0/4005685057 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c03f030 con 0x7fa31c101df0 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.787+0000 7fa321c58640 1 -- 172.21.15.89:0/4005685057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31c19d680 con 0x7fa31c101df0 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.787+0000 7fa318ff9640 1 -- 172.21.15.89:0/4005685057 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa30c003ce0 con 0x7fa31c101df0 2024-09-15T23:33:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.787+0000 7fa318ff9640 1 -- 172.21.15.89:0/4005685057 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c035070 con 0x7fa31c101df0 2024-09-15T23:33:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.787+0000 7fa321c58640 1 -- 172.21.15.89:0/4005685057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa31c1a4fc0 con 0x7fa31c101df0 2024-09-15T23:33:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.788+0000 7fa321c58640 1 -- 172.21.15.89:0/4005685057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa31c1021d0 con 0x7fa31c101df0 2024-09-15T23:33:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.789+0000 7fa318ff9640 1 -- 172.21.15.89:0/4005685057 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa30c042030 con 0x7fa31c101df0 2024-09-15T23:33:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.790+0000 7fa318ff9640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3040a3240 0x7fa3040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.790+0000 7fa31affd640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3040a3240 0x7fa3040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.790+0000 7fa318ff9640 1 -- 172.21.15.89:0/4005685057 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa30c0f0560 con 0x7fa31c101df0 2024-09-15T23:33:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.790+0000 7fa31affd640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3040a3240 0x7fa3040a5700 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa31c0fdda0 tx=0x7fa31000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.794+0000 7fa318ff9640 1 -- 172.21.15.89:0/4005685057 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa30c0bc840 con 0x7fa31c101df0 2024-09-15T23:33:49.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.993+0000 7fa321c58640 1 -- 172.21.15.89:0/4005685057 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa31c10a100 con 0x7fa31c101df0 2024-09-15T23:33:49.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.994+0000 7fa318ff9640 1 -- 172.21.15.89:0/4005685057 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fa30c0c0b40 con 0x7fa31c101df0 2024-09-15T23:33:49.995 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:33:49.995 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:33:49.996 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:33:49.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 -- 172.21.15.89:0/4005685057 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3040a3240 msgr2=0x7fa3040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:49.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3040a3240 0x7fa3040a5700 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa31c0fdda0 tx=0x7fa31000a2d0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 -- 172.21.15.89:0/4005685057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa31c101df0 msgr2=0x7fa31c198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:49.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa31c101df0 0x7fa31c198440 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fa30c005ab0 tx=0x7fa30c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 -- 172.21.15.89:0/4005685057 shutdown_connections 2024-09-15T23:33:49.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa3040a3240 0x7fa3040a5700 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa31c106f20 0x7fa31c19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa31c1027a0 0x7fa31c198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 --2- 172.21.15.89:0/4005685057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa31c101df0 0x7fa31c198440 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:49.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 -- 172.21.15.89:0/4005685057 >> 172.21.15.89:0/4005685057 conn(0x7fa31c0fd8f0 msgr2=0x7fa31c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:49.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 -- 172.21.15.89:0/4005685057 shutdown_connections 2024-09-15T23:33:49.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:49.996+0000 7fa321c58640 1 -- 172.21.15.89:0/4005685057 wait complete. 2024-09-15T23:33:50.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:50 smithi089 ceph-mon[30933]: pgmap v644: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-15T23:33:50.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:50 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/4005685057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:50.683 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:33:50.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:50 smithi120 ceph-mon[35250]: pgmap v644: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-15T23:33:50.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:50 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/4005685057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:50.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:50 smithi174 ceph-mon[35586]: pgmap v644: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-15T23:33:50.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:50 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/4005685057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:50.927 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:51.444 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.432+0000 7f3c897ad640 1 -- 172.21.15.89:0/3986946935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c84106f10 msgr2=0x7f3c8410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.432+0000 7f3c897ad640 1 --2- 172.21.15.89:0/3986946935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c84106f10 0x7f3c8410ea90 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009ca0 tx=0x7f3c7c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.432+0000 7f3c897ad640 1 -- 172.21.15.89:0/3986946935 shutdown_connections 2024-09-15T23:33:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.432+0000 7f3c897ad640 1 --2- 172.21.15.89:0/3986946935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c84106f10 0x7f3c8410ea90 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.432+0000 7f3c897ad640 1 --2- 172.21.15.89:0/3986946935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c84102790 0x7f3c841067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.432+0000 7f3c897ad640 1 --2- 172.21.15.89:0/3986946935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c84101de0 0x7f3c841021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.432+0000 7f3c897ad640 1 -- 172.21.15.89:0/3986946935 >> 172.21.15.89:0/3986946935 conn(0x7f3c840fd8c0 msgr2=0x7f3c840ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.432+0000 7f3c897ad640 1 -- 172.21.15.89:0/3986946935 shutdown_connections 2024-09-15T23:33:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.432+0000 7f3c897ad640 1 -- 172.21.15.89:0/3986946935 wait complete. 2024-09-15T23:33:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c897ad640 1 Processor -- start 2024-09-15T23:33:51.445 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c897ad640 1 -- start start 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c897ad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c84101de0 0x7f3c84198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c897ad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c84102790 0x7f3c84198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c897ad640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c84106f10 0x7f3c8419cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c897ad640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c84111f70 con 0x7f3c84102790 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c897ad640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c84111df0 con 0x7f3c84106f10 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c897ad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841120f0 con 0x7f3c84101de0 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c827fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c84102790 0x7f3c84198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c827fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c84102790 0x7f3c84198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49146/0 (socket says 172.21.15.89:49146) 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c827fc640 1 -- 172.21.15.89:0/1598994010 learned_addr learned my addr 172.21.15.89:0/1598994010 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c837fe640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c84106f10 0x7f3c8419cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c84101de0 0x7f3c84198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.446 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c827fc640 1 -- 172.21.15.89:0/1598994010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c84101de0 msgr2=0x7f3c84198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c827fc640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c84101de0 0x7f3c84198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c827fc640 1 -- 172.21.15.89:0/1598994010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c84106f10 msgr2=0x7f3c8419cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c827fc640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c84106f10 0x7f3c8419cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c827fc640 1 -- 172.21.15.89:0/1598994010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8419d2e0 con 0x7f3c84102790 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.433+0000 7f3c837fe640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c84106f10 0x7f3c8419cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.434+0000 7f3c827fc640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c84102790 0x7f3c84198870 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800d950 tx=0x7f3c7800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.434+0000 7f3c6bfff640 1 -- 172.21.15.89:0/1598994010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78017070 con 0x7f3c84102790 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.434+0000 7f3c6bfff640 1 -- 172.21.15.89:0/1598994010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c78009c30 con 0x7f3c84102790 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.434+0000 7f3c897ad640 1 -- 172.21.15.89:0/1598994010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c8419d5d0 con 0x7f3c84102790 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.434+0000 7f3c6bfff640 1 -- 172.21.15.89:0/1598994010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7800f6b0 con 0x7f3c84102790 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.434+0000 7f3c897ad640 1 -- 172.21.15.89:0/1598994010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c841a4eb0 con 0x7f3c84102790 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.436+0000 7f3c6bfff640 1 -- 172.21.15.89:0/1598994010 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3c780109d0 con 0x7f3c84102790 2024-09-15T23:33:51.447 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.436+0000 7f3c8bb86740 1 Processor -- start 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.436+0000 7f3c8bb86740 1 -- start start 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 0x556cf5e8b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5e8ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e98760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556cf5df12f0 con 0x556cf5e8b5d0 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556cf5d98cf0 con 0x556cf5e96370 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556cf5ded2b0 con 0x556cf5e8cb40 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c82ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5e8ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c6bfff640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c6c0a3240 0x7f3c6c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c82ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5e8ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49152/0 (socket says 172.21.15.89:49152) 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c82ffd640 1 -- 172.21.15.89:0/3414913972 learned_addr learned my addr 172.21.15.89:0/3414913972 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c837fe640 1 --2- 172.21.15.89:0/3414913972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 0x556cf5e8b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.448 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c6bfff640 1 -- 172.21.15.89:0/1598994010 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c780cf940 con 0x7f3c84102790 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c827fc640 1 --2- 172.21.15.89:0/3414913972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e98760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c837fe640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c6c0a3240 0x7f3c6c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c82ffd640 1 -- 172.21.15.89:0/3414913972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 msgr2=0x556cf5e8b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/3414913972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 0x556cf5e8b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c82ffd640 1 -- 172.21.15.89:0/3414913972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 msgr2=0x556cf5e98760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/3414913972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e98760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c82ffd640 1 -- 172.21.15.89:0/3414913972 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cf5d7d3a0 con 0x556cf5e8b5d0 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c837fe640 1 --2- 172.21.15.89:0/3414913972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 0x556cf5e8b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.437+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/3414913972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5e8ba10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400cc60 tx=0x7f3c7400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a95a69983beb11 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c837fe640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c6c0a3240 0x7f3c6c0a5700 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009a60 tx=0x7f3c7c0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c697fa640 1 -- 172.21.15.89:0/3414913972 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74014070 con 0x556cf5e8b5d0 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c697fa640 1 -- 172.21.15.89:0/3414913972 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c740047a0 con 0x556cf5e8b5d0 2024-09-15T23:33:51.449 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c697fa640 1 -- 172.21.15.89:0/3414913972 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74010730 con 0x556cf5e8b5d0 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c8bb86740 1 -- 172.21.15.89:0/3414913972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 msgr2=0x556cf5e8ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/3414913972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5e8ba10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400cc60 tx=0x7f3c7400e720 comp rx=0 tx=0).stop 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c8bb86740 1 -- 172.21.15.89:0/3414913972 shutdown_connections 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/3414913972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e98760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/3414913972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5e8ba10 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/3414913972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 0x556cf5e8b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c8bb86740 1 -- 172.21.15.89:0/3414913972 >> 172.21.15.89:0/3414913972 conn(0x556cf5e54310 msgr2=0x556cf5e87c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c8bb86740 1 -- 172.21.15.89:0/3414913972 shutdown_connections 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.438+0000 7f3c8bb86740 1 -- 172.21.15.89:0/3414913972 wait complete. 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c8bb86740 1 -- start start 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c8bb86740 1 Processor -- start 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 0x556cf5e95d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8cb40 0x556cf5e92ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.450 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e935e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556cf5e99070 con 0x556cf5e8cb40 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556cf5dac060 con 0x556cf5e96370 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556cf5e991f0 con 0x556cf5e8b5d0 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c8bb86740 10 client.0 _ll_register_callbacks cb 0x556cf5aff8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c827fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8cb40 0x556cf5e92ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c827fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8cb40 0x556cf5e92ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49164/0 (socket says 172.21.15.89:49164) 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c827fc640 1 -- 172.21.15.89:0/1224430825 learned_addr learned my addr 172.21.15.89:0/1224430825 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c837fe640 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e935e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.439+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 0x556cf5e95d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c827fc640 1 -- 172.21.15.89:0/1224430825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 msgr2=0x556cf5e95d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c827fc640 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 0x556cf5e95d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c827fc640 1 -- 172.21.15.89:0/1224430825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 msgr2=0x556cf5e935e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:51.451 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c827fc640 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e935e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c827fc640 1 -- 172.21.15.89:0/1224430825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cf5b623c0 con 0x556cf5e8cb40 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 0x556cf5e95d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c827fc640 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8cb40 0x556cf5e92ff0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f3c780cd3c0 tx=0x7f3c78005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f8028a0100b886 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c66ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c66ff5640 1 -- 172.21.15.89:0/1224430825 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78020070 con 0x556cf5e8cb40 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c66ff5640 1 -- 172.21.15.89:0/1224430825 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c780860b0 con 0x556cf5e8cb40 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c8bb86740 1 -- 172.21.15.89:0/1224430825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c740047a0 con 0x556cf5e8cb40 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c66ff5640 1 -- 172.21.15.89:0/1224430825 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c7809ff20 con 0x556cf5e8cb40 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c8bb86740 10 client.18765 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c8bb86740 1 -- 172.21.15.89:0/1224430825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556cf5ea4b40 con 0x556cf5e8cb40 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c66ff5640 1 -- 172.21.15.89:0/1224430825 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c740047a0 con 0x556cf5e8cb40 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c66ff5640 1 -- 172.21.15.89:0/1224430825 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3c7809afc0 con 0x556cf5e8cb40 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c8bb86740 10 client.18765 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:51.452 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c8bb86740 10 client.18765 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c8bb86740 20 client.18765 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556cf5ea6000 0x556cf5ea84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c8bb86740 4 client.18765 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.440+0000 7f3c8bb86740 1 -- 172.21.15.89:0/1224430825 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cf5dfe860 con 0x556cf5ea6000 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.441+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556cf5ea6000 0x556cf5ea84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.441+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556cf5ea6000 0x556cf5ea84a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d0c5e50d1d6d4365 server_cookie=d2b60dbe91abdd31 in_seq=0 out_seq=0 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.441+0000 7f3c66ff5640 10 client.18765 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c66ff5640 1 -- 172.21.15.89:0/1224430825 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556cf5dfe860 con 0x556cf5ea6000 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c66ff5640 10 client.18765 handle_command_reply: tid=0 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 client.18765 shutdown 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 -- 172.21.15.89:0/1224430825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8cb40 msgr2=0x556cf5e92ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8cb40 0x556cf5e92ff0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f3c780cd3c0 tx=0x7f3c78005f90 comp rx=0 tx=0).stop 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 -- 172.21.15.89:0/1224430825 shutdown_connections 2024-09-15T23:33:51.453 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556cf5ea6000 0x556cf5ea84a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e935e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8cb40 0x556cf5e92ff0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c66ff5640 0 client.18765 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:33:51.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/1224430825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 0x556cf5e95d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 -- 172.21.15.89:0/1224430825 >> 172.21.15.89:0/1224430825 conn(0x556cf5e54310 msgr2=0x556cf5e8d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:51.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 -- 172.21.15.89:0/1224430825 shutdown_connections 2024-09-15T23:33:51.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 1 -- 172.21.15.89:0/1224430825 wait complete. 2024-09-15T23:33:51.454 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.442+0000 7f3c8bb86740 20 client.18765 trim_cache size 0 max 16384 2024-09-15T23:33:51.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.455+0000 7f3c8bb86740 1 Processor -- start 2024-09-15T23:33:51.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.455+0000 7f3c8bb86740 1 -- start start 2024-09-15T23:33:51.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5ea2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 0x556cf5e8e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.457 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e8e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841aee50 con 0x556cf5e8b5d0 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841b0990 con 0x556cf5e96370 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841af520 con 0x556cf5e8cb40 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c82ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5ea2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c82ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5ea2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49172/0 (socket says 172.21.15.89:49172) 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c82ffd640 1 -- 172.21.15.89:0/2687852245 learned_addr learned my addr 172.21.15.89:0/2687852245 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c837fe640 1 --2- 172.21.15.89:0/2687852245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e8e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c827fc640 1 --2- 172.21.15.89:0/2687852245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 0x556cf5e8e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c82ffd640 1 -- 172.21.15.89:0/2687852245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 msgr2=0x556cf5e8e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/2687852245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 0x556cf5e8e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c82ffd640 1 -- 172.21.15.89:0/2687852245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 msgr2=0x556cf5e8e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/2687852245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e8e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.458 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c82ffd640 1 -- 172.21.15.89:0/2687852245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cf5e91580 con 0x556cf5e8b5d0 2024-09-15T23:33:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c837fe640 1 --2- 172.21.15.89:0/2687852245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e8e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/2687852245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5ea2310 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400f010 tx=0x7f3c74000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23cec24927302c21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c68ff9640 1 -- 172.21.15.89:0/2687852245 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74014040 con 0x556cf5e8b5d0 2024-09-15T23:33:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c68ff9640 1 -- 172.21.15.89:0/2687852245 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c74010b80 con 0x556cf5e8b5d0 2024-09-15T23:33:51.459 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.456+0000 7f3c68ff9640 1 -- 172.21.15.89:0/2687852245 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74024800 con 0x556cf5e8b5d0 2024-09-15T23:33:51.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.457+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2687852245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 msgr2=0x556cf5ea2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:51.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.457+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/2687852245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5ea2310 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400f010 tx=0x7f3c74000bc0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.457+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2687852245 shutdown_connections 2024-09-15T23:33:51.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.457+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/2687852245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e96370 0x556cf5e8e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.457+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/2687852245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8cb40 0x556cf5e8e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.457+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/2687852245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e8b5d0 0x556cf5ea2310 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.457+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2687852245 >> 172.21.15.89:0/2687852245 conn(0x556cf5e54310 msgr2=0x556cf5e92120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:51.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.457+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2687852245 shutdown_connections 2024-09-15T23:33:51.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.457+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2687852245 wait complete. 2024-09-15T23:33:51.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.457+0000 7f3c8bb86740 1 -- start start 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c8bb86740 1 Processor -- start 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 0x556cf5e9ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e8cb40 0x556cf5e8fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c8bb86740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e96370 0x556cf5e90630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841b02c0 con 0x556cf5e96370 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841afbf0 con 0x556cf5e8cb40 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c8bb86740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c841ace30 con 0x556cf5e8b5d0 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c8bb86740 10 client.0 _ll_register_callbacks cb 0x7f3c7809b980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e96370 0x556cf5e90630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e96370 0x556cf5e90630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49182/0 (socket says 172.21.15.89:49182) 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c837fe640 1 -- 172.21.15.89:0/2281135425 learned_addr learned my addr 172.21.15.89:0/2281135425 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c827fc640 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e8cb40 0x556cf5e8fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 0x556cf5e9ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c837fe640 1 -- 172.21.15.89:0/2281135425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 msgr2=0x556cf5e9ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c837fe640 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 0x556cf5e9ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c837fe640 1 -- 172.21.15.89:0/2281135425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e8cb40 msgr2=0x556cf5e8fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c837fe640 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e8cb40 0x556cf5e8fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c837fe640 1 -- 172.21.15.89:0/2281135425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cf5f0fcf0 con 0x556cf5e96370 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c837fe640 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e96370 0x556cf5e90630 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c035000 tx=0x7f3c7c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41272b91e09f55fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.458+0000 7f3c66ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c66ff5640 1 -- 172.21.15.89:0/2281135425 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c043070 con 0x556cf5e96370 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c66ff5640 1 -- 172.21.15.89:0/2281135425 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3c7c004450 con 0x556cf5e96370 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2281135425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c74010b80 con 0x556cf5e96370 2024-09-15T23:33:51.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c66ff5640 1 -- 172.21.15.89:0/2281135425 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c7c004690 con 0x556cf5e96370 2024-09-15T23:33:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c8bb86740 10 client.18771 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2281135425 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556cf5f10e90 con 0x556cf5e96370 2024-09-15T23:33:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c66ff5640 1 -- 172.21.15.89:0/2281135425 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c74010b80 con 0x556cf5e96370 2024-09-15T23:33:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c66ff5640 1 -- 172.21.15.89:0/2281135425 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3c7c052070 con 0x556cf5e96370 2024-09-15T23:33:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c8bb86740 10 client.18771 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c8bb86740 10 client.18771 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:33:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c8bb86740 20 client.18771 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556cf5f11690 0x556cf5f13ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c8bb86740 4 client.18771 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:33:51.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2281135425 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556cf5df8d70 con 0x556cf5f11690 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.459+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556cf5f11690 0x556cf5f13ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c66ff5640 10 client.18771 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c82ffd640 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556cf5f11690 0x556cf5f13ab0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f88df32dfc7ca663 server_cookie=f3886bd534a13828 in_seq=0 out_seq=0 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c66ff5640 1 -- 172.21.15.89:0/2281135425 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556cf5df8d70 con 0x556cf5f11690 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c66ff5640 10 client.18771 handle_command_reply: tid=0 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 client.18771 shutdown 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2281135425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e96370 msgr2=0x556cf5e90630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e96370 0x556cf5e90630 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c035000 tx=0x7f3c7c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2281135425 shutdown_connections 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x556cf5f11690 0x556cf5f13ab0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cf5e96370 0x556cf5e90630 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556cf5e8cb40 0x556cf5e8fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.467 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 --2- 172.21.15.89:0/2281135425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556cf5e8b5d0 0x556cf5e9ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2281135425 >> 172.21.15.89:0/2281135425 conn(0x556cf5e54310 msgr2=0x556cf5e94850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:51.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c66ff5640 0 client.18771 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:33:51.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2281135425 shutdown_connections 2024-09-15T23:33:51.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 1 -- 172.21.15.89:0/2281135425 wait complete. 2024-09-15T23:33:51.468 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.460+0000 7f3c8bb86740 20 client.18771 trim_cache size 0 max 16384 2024-09-15T23:33:51.468 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:33:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.461+0000 7f3c897ad640 1 -- 172.21.15.89:0/1598994010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c6c0a3240 msgr2=0x7f3c6c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.461+0000 7f3c897ad640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c6c0a3240 0x7f3c6c0a5700 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009a60 tx=0x7f3c7c0095c0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.461+0000 7f3c897ad640 1 -- 172.21.15.89:0/1598994010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c84102790 msgr2=0x7f3c84198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.461+0000 7f3c897ad640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c84102790 0x7f3c84198870 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800d950 tx=0x7f3c7800de10 comp rx=0 tx=0).stop 2024-09-15T23:33:51.469 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.461+0000 7f3c897ad640 1 -- 172.21.15.89:0/1598994010 shutdown_connections 2024-09-15T23:33:51.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.461+0000 7f3c897ad640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3c6c0a3240 0x7f3c6c0a5700 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.461+0000 7f3c897ad640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c84106f10 0x7f3c8419cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.461+0000 7f3c897ad640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c84102790 0x7f3c84198870 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.461+0000 7f3c897ad640 1 --2- 172.21.15.89:0/1598994010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c84101de0 0x7f3c84198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:51.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.461+0000 7f3c897ad640 1 -- 172.21.15.89:0/1598994010 >> 172.21.15.89:0/1598994010 conn(0x7f3c840fd8c0 msgr2=0x7f3c840febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:51.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.462+0000 7f3c897ad640 1 -- 172.21.15.89:0/1598994010 shutdown_connections 2024-09-15T23:33:51.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:51.462+0000 7f3c897ad640 1 -- 172.21.15.89:0/1598994010 wait complete. 2024-09-15T23:33:52.233 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:33:52.482 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:52.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:52 smithi089 ceph-mon[30933]: pgmap v645: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-09-15T23:33:52.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:52 smithi120 ceph-mon[35250]: pgmap v645: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-09-15T23:33:52.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:52 smithi174 ceph-mon[35586]: pgmap v645: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.946+0000 7fb35f18d640 1 -- 172.21.15.89:0/2435203426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb358100a40 msgr2=0x7fb358100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.946+0000 7fb35f18d640 1 --2- 172.21.15.89:0/2435203426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb358100a40 0x7fb358100e40 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fb34c009900 tx=0x7fb34c02a410 comp rx=0 tx=0).stop 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 -- 172.21.15.89:0/2435203426 shutdown_connections 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 --2- 172.21.15.89:0/2435203426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3581025f0 0x7fb358108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 --2- 172.21.15.89:0/2435203426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb358101c30 0x7fb3581020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 --2- 172.21.15.89:0/2435203426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb358100a40 0x7fb358100e40 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 -- 172.21.15.89:0/2435203426 >> 172.21.15.89:0/2435203426 conn(0x7fb3580fc1e0 msgr2=0x7fb3580fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 -- 172.21.15.89:0/2435203426 shutdown_connections 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 -- 172.21.15.89:0/2435203426 wait complete. 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 Processor -- start 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 -- start start 2024-09-15T23:33:52.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb358100a40 0x7fb358196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb358101c30 0x7fb358197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3581025f0 0x7fb35819e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb35810c6c0 con 0x7fb358100a40 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb35810c540 con 0x7fb358101c30 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35f18d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb35810c840 con 0x7fb3581025f0 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.947+0000 7fb35cf29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb358100a40 0x7fb358196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35cf29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb358100a40 0x7fb358196d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49196/0 (socket says 172.21.15.89:49196) 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35cf29640 1 -- 172.21.15.89:0/51413988 learned_addr learned my addr 172.21.15.89:0/51413988 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb357fff640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb358101c30 0x7fb358197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35d72a640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3581025f0 0x7fb35819e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35cf29640 1 -- 172.21.15.89:0/51413988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3581025f0 msgr2=0x7fb35819e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35cf29640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3581025f0 0x7fb35819e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.951 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35cf29640 1 -- 172.21.15.89:0/51413988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb358101c30 msgr2=0x7fb358197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:52.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35cf29640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb358101c30 0x7fb358197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35cf29640 1 -- 172.21.15.89:0/51413988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35819e940 con 0x7fb358100a40 2024-09-15T23:33:52.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35cf29640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb358100a40 0x7fb358196d30 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fb34c031610 tx=0x7fb34c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:52.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb355ffb640 1 -- 172.21.15.89:0/51413988 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c046070 con 0x7fb358100a40 2024-09-15T23:33:52.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35f18d640 1 -- 172.21.15.89:0/51413988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3580698a0 con 0x7fb358100a40 2024-09-15T23:33:52.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb355ffb640 1 -- 172.21.15.89:0/51413988 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb34c003ce0 con 0x7fb358100a40 2024-09-15T23:33:52.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb35f18d640 1 -- 172.21.15.89:0/51413988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb358069de0 con 0x7fb358100a40 2024-09-15T23:33:52.952 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.948+0000 7fb355ffb640 1 -- 172.21.15.89:0/51413988 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c035050 con 0x7fb358100a40 2024-09-15T23:33:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.951+0000 7fb355ffb640 1 -- 172.21.15.89:0/51413988 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb34c049070 con 0x7fb358100a40 2024-09-15T23:33:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.951+0000 7fb355ffb640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3480a32f0 0x7fb3480a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb357fff640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3480a32f0 0x7fb3480a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb355ffb640 1 -- 172.21.15.89:0/51413988 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb34c0f18f0 con 0x7fb358100a40 2024-09-15T23:33:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb362001740 1 Processor -- start 2024-09-15T23:33:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb357fff640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3480a32f0 0x7fb3480a57b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb344005fd0 tx=0x7fb34400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb362001740 1 -- start start 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb362001740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc39a0 0x555b62fc3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb362001740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 0x555b62fc8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb362001740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc9370 0x555b62fcb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb362001740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555b62f272f0 con 0x555b62fc9370 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb362001740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555b62ecf270 con 0x555b62fc42e0 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb362001740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555b62f232b0 con 0x555b62fc39a0 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb35d72a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc9370 0x555b62fcb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb35d72a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc9370 0x555b62fcb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49200/0 (socket says 172.21.15.89:49200) 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb35d72a640 1 -- 172.21.15.89:0/1518512450 learned_addr learned my addr 172.21.15.89:0/1518512450 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.952+0000 7fb357fff640 1 --2- 172.21.15.89:0/1518512450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 0x555b62fc8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.953+0000 7fb35cf29640 1 --2- 172.21.15.89:0/1518512450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc39a0 0x555b62fc3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.953+0000 7fb35d72a640 1 -- 172.21.15.89:0/1518512450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc39a0 msgr2=0x555b62fc3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.953+0000 7fb35d72a640 1 --2- 172.21.15.89:0/1518512450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc39a0 0x555b62fc3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.953+0000 7fb35d72a640 1 -- 172.21.15.89:0/1518512450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 msgr2=0x555b62fc8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.953+0000 7fb35d72a640 1 --2- 172.21.15.89:0/1518512450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 0x555b62fc8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.953+0000 7fb35d72a640 1 -- 172.21.15.89:0/1518512450 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b62ece340 con 0x555b62fc9370 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.953+0000 7fb35d72a640 1 --2- 172.21.15.89:0/1518512450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc9370 0x555b62fcb830 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fb350008dc0 tx=0x7fb35000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49da1ea6f6bda3a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.953+0000 7fb342ffd640 1 -- 172.21.15.89:0/1518512450 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb350014070 con 0x555b62fc9370 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.953+0000 7fb342ffd640 1 -- 172.21.15.89:0/1518512450 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb350003b70 con 0x555b62fc9370 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.953+0000 7fb342ffd640 1 -- 172.21.15.89:0/1518512450 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb350010790 con 0x555b62fc9370 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 -- 172.21.15.89:0/1518512450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc9370 msgr2=0x555b62fcb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 --2- 172.21.15.89:0/1518512450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc9370 0x555b62fcb830 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fb350008dc0 tx=0x7fb35000e720 comp rx=0 tx=0).stop 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 -- 172.21.15.89:0/1518512450 shutdown_connections 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 --2- 172.21.15.89:0/1518512450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc9370 0x555b62fcb830 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 --2- 172.21.15.89:0/1518512450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 0x555b62fc8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 --2- 172.21.15.89:0/1518512450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc39a0 0x555b62fc3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 -- 172.21.15.89:0/1518512450 >> 172.21.15.89:0/1518512450 conn(0x555b62fbf010 msgr2=0x555b62fbf410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 -- 172.21.15.89:0/1518512450 shutdown_connections 2024-09-15T23:33:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 -- 172.21.15.89:0/1518512450 wait complete. 2024-09-15T23:33:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 -- start start 2024-09-15T23:33:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.954+0000 7fb362001740 1 Processor -- start 2024-09-15T23:33:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb362001740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc39a0 0x555b62fcdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb362001740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 0x555b62fce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb362001740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc9370 0x555b62fd5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb362001740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555b62fcc340 con 0x555b62fc39a0 2024-09-15T23:33:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb362001740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555b62eb0290 con 0x555b62fc42e0 2024-09-15T23:33:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb362001740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555b62fcc4c0 con 0x555b62fc9370 2024-09-15T23:33:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb362001740 10 client.0 _ll_register_callbacks cb 0x555b62ee1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb35cf29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc39a0 0x555b62fcdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb35cf29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc39a0 0x555b62fcdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49212/0 (socket says 172.21.15.89:49212) 2024-09-15T23:33:52.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb35cf29640 1 -- 172.21.15.89:0/3137355016 learned_addr learned my addr 172.21.15.89:0/3137355016 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb357fff640 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 0x555b62fce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb35cf29640 1 -- 172.21.15.89:0/3137355016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc9370 msgr2=0x555b62fd5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb35cf29640 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc9370 0x555b62fd5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb35cf29640 1 -- 172.21.15.89:0/3137355016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 msgr2=0x555b62fce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb35cf29640 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 0x555b62fce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb35cf29640 1 -- 172.21.15.89:0/3137355016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b62c3a210 con 0x555b62fc39a0 2024-09-15T23:33:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb357fff640 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 0x555b62fce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb35cf29640 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc39a0 0x555b62fcdbf0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0bd8d0 tx=0x7fb34c0bd710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855271e43a1eec34 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb337fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb337fff640 1 -- 172.21.15.89:0/3137355016 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0bdbd0 con 0x555b62fc39a0 2024-09-15T23:33:52.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb337fff640 1 -- 172.21.15.89:0/3137355016 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb34c038470 con 0x555b62fc39a0 2024-09-15T23:33:52.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.955+0000 7fb362001740 1 -- 172.21.15.89:0/3137355016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb350003b70 con 0x555b62fc39a0 2024-09-15T23:33:52.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb337fff640 1 -- 172.21.15.89:0/3137355016 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb34c0c1ed0 con 0x555b62fc39a0 2024-09-15T23:33:52.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb362001740 10 client.18783 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:52.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb362001740 1 -- 172.21.15.89:0/3137355016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555b62d06620 con 0x555b62fc39a0 2024-09-15T23:33:52.965 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb337fff640 1 -- 172.21.15.89:0/3137355016 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb350003b70 con 0x555b62fc39a0 2024-09-15T23:33:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb337fff640 1 -- 172.21.15.89:0/3137355016 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb34c0a80c0 con 0x555b62fc39a0 2024-09-15T23:33:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb362001740 10 client.18783 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb362001740 10 client.18783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb362001740 20 client.18783 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb362001740 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555b62fd9800 0x555b62fdbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb362001740 4 client.18783 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb362001740 1 -- 172.21.15.89:0/3137355016 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b62f34a20 con 0x555b62fd9800 2024-09-15T23:33:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.956+0000 7fb357fff640 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555b62fd9800 0x555b62fdbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.957+0000 7fb337fff640 10 client.18783 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:52.966 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.957+0000 7fb357fff640 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555b62fd9800 0x555b62fdbcc0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e0364910972fe01 server_cookie=e3b88225bdf28b0f in_seq=0 out_seq=0 2024-09-15T23:33:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb337fff640 1 -- 172.21.15.89:0/3137355016 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555b62f34a20 con 0x555b62fd9800 2024-09-15T23:33:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb337fff640 10 client.18783 handle_command_reply: tid=0 2024-09-15T23:33:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 client.18783 shutdown 2024-09-15T23:33:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 -- 172.21.15.89:0/3137355016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc39a0 msgr2=0x555b62fcdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc39a0 0x555b62fcdbf0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0bd8d0 tx=0x7fb34c0bd710 comp rx=0 tx=0).stop 2024-09-15T23:33:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 -- 172.21.15.89:0/3137355016 shutdown_connections 2024-09-15T23:33:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555b62fd9800 0x555b62fdbcc0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.967 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc9370 0x555b62fd5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc42e0 0x555b62fce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb337fff640 0 client.18783 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 --2- 172.21.15.89:0/3137355016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fc39a0 0x555b62fcdbf0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 -- 172.21.15.89:0/3137355016 >> 172.21.15.89:0/3137355016 conn(0x555b62fbf010 msgr2=0x555b62fc0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 -- 172.21.15.89:0/3137355016 shutdown_connections 2024-09-15T23:33:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 1 -- 172.21.15.89:0/3137355016 wait complete. 2024-09-15T23:33:52.968 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.958+0000 7fb362001740 20 client.18783 trim_cache size 0 max 16384 2024-09-15T23:33:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.972+0000 7fb362001740 1 Processor -- start 2024-09-15T23:33:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.972+0000 7fb362001740 1 -- start start 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.972+0000 7fb362001740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fe05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.972+0000 7fb362001740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 0x555b62fd5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.972+0000 7fb362001740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc42e0 0x555b62fe0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.972+0000 7fb362001740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3581a8640 con 0x555b62fbf010 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.972+0000 7fb362001740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb3581aa180 con 0x555b62fc39a0 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.972+0000 7fb362001740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb3581a8d10 con 0x555b62fc42e0 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb35cf29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fe05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb35cf29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fe05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49226/0 (socket says 172.21.15.89:49226) 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb35cf29640 1 -- 172.21.15.89:0/892593786 learned_addr learned my addr 172.21.15.89:0/892593786 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb35d72a640 1 --2- 172.21.15.89:0/892593786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc42e0 0x555b62fe0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.975 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb35cf29640 1 -- 172.21.15.89:0/892593786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc42e0 msgr2=0x555b62fe0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb35cf29640 1 --2- 172.21.15.89:0/892593786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc42e0 0x555b62fe0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb35cf29640 1 -- 172.21.15.89:0/892593786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 msgr2=0x555b62fd5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb357fff640 1 --2- 172.21.15.89:0/892593786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 0x555b62fd5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb35cf29640 1 --2- 172.21.15.89:0/892593786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 0x555b62fd5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb35cf29640 1 -- 172.21.15.89:0/892593786 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b62fdc340 con 0x555b62fbf010 2024-09-15T23:33:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb357fff640 1 --2- 172.21.15.89:0/892593786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 0x555b62fd5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb35cf29640 1 --2- 172.21.15.89:0/892593786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fe05c0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0314e0 tx=0x7fb34c0bd840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1659b9bc6029240 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb3427fc640 1 -- 172.21.15.89:0/892593786 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0bd530 con 0x555b62fbf010 2024-09-15T23:33:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb3427fc640 1 -- 172.21.15.89:0/892593786 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb34c0bee50 con 0x555b62fbf010 2024-09-15T23:33:52.976 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.973+0000 7fb3427fc640 1 -- 172.21.15.89:0/892593786 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c038cb0 con 0x555b62fbf010 2024-09-15T23:33:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.974+0000 7fb362001740 1 -- 172.21.15.89:0/892593786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 msgr2=0x555b62fe05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.974+0000 7fb362001740 1 --2- 172.21.15.89:0/892593786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fe05c0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0314e0 tx=0x7fb34c0bd840 comp rx=0 tx=0).stop 2024-09-15T23:33:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.974+0000 7fb362001740 1 -- 172.21.15.89:0/892593786 shutdown_connections 2024-09-15T23:33:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.974+0000 7fb362001740 1 --2- 172.21.15.89:0/892593786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc42e0 0x555b62fe0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.974+0000 7fb362001740 1 --2- 172.21.15.89:0/892593786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 0x555b62fd5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.974+0000 7fb362001740 1 --2- 172.21.15.89:0/892593786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fe05c0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.974+0000 7fb362001740 1 -- 172.21.15.89:0/892593786 >> 172.21.15.89:0/892593786 conn(0x555b62fc8ec0 msgr2=0x555b62fbe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.974+0000 7fb362001740 1 -- 172.21.15.89:0/892593786 shutdown_connections 2024-09-15T23:33:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.974+0000 7fb362001740 1 -- 172.21.15.89:0/892593786 wait complete. 2024-09-15T23:33:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb362001740 1 -- start start 2024-09-15T23:33:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb362001740 1 Processor -- start 2024-09-15T23:33:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb362001740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fd4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.985 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb35cf29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fd4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb35cf29640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fd4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49258/0 (socket says 172.21.15.89:49258) 2024-09-15T23:33:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb362001740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 0x555b62fd5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb35cf29640 1 -- 172.21.15.89:0/4080621067 learned_addr learned my addr 172.21.15.89:0/4080621067 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb362001740 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc42e0 0x555b62fd31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3581a9ab0 con 0x555b62fbf010 2024-09-15T23:33:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb3581a93e0 con 0x555b62fc39a0 2024-09-15T23:33:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.975+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb3581a6620 con 0x555b62fc42e0 2024-09-15T23:33:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb357fff640 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 0x555b62fd5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb35d72a640 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc42e0 0x555b62fd31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb362001740 10 client.0 _ll_register_callbacks cb 0x7fb34c004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb35cf29640 1 -- 172.21.15.89:0/4080621067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc42e0 msgr2=0x555b62fd31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:52.988 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb35cf29640 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc42e0 0x555b62fd31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb35cf29640 1 -- 172.21.15.89:0/4080621067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 msgr2=0x555b62fd5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb35cf29640 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 0x555b62fd5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb35cf29640 1 -- 172.21.15.89:0/4080621067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b62fd39d0 con 0x555b62fbf010 2024-09-15T23:33:52.989 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb357fff640 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 0x555b62fd5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb35cf29640 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fd4ae0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0ba470 tx=0x7fb34c005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4e15f42fcee339 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb337fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb337fff640 1 -- 172.21.15.89:0/4080621067 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c0bdbd0 con 0x555b62fbf010 2024-09-15T23:33:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb337fff640 1 -- 172.21.15.89:0/4080621067 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb34c038470 con 0x555b62fbf010 2024-09-15T23:33:52.990 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb34c0bee50 con 0x555b62fbf010 2024-09-15T23:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb337fff640 1 -- 172.21.15.89:0/4080621067 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb34c0c6790 con 0x555b62fbf010 2024-09-15T23:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb362001740 10 client.18795 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.976+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555b62fd39d0 con 0x555b62fbf010 2024-09-15T23:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.977+0000 7fb337fff640 1 -- 172.21.15.89:0/4080621067 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb34c0bee50 con 0x555b62fbf010 2024-09-15T23:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.977+0000 7fb337fff640 1 -- 172.21.15.89:0/4080621067 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb34c0bd100 con 0x555b62fbf010 2024-09-15T23:33:52.991 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.977+0000 7fb362001740 10 client.18795 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.977+0000 7fb362001740 10 client.18795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.977+0000 7fb362001740 20 client.18795 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.977+0000 7fb362001740 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555b62fc9370 0x555b63047600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.977+0000 7fb362001740 4 client.18795 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.977+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555b62f2ed70 con 0x555b62fc9370 2024-09-15T23:33:52.992 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.977+0000 7fb357fff640 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555b62fc9370 0x555b63047600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.977+0000 7fb337fff640 10 client.18795 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:52.993 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.978+0000 7fb357fff640 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555b62fc9370 0x555b63047600 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95879c109e11f331 server_cookie=17a9616003ca9fac in_seq=0 out_seq=0 2024-09-15T23:33:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb337fff640 1 -- 172.21.15.89:0/4080621067 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555b62f2ed70 con 0x555b62fc9370 2024-09-15T23:33:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb337fff640 10 client.18795 handle_command_reply: tid=0 2024-09-15T23:33:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 client.18795 shutdown 2024-09-15T23:33:52.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 msgr2=0x555b62fd4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fd4ae0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fb34c0ba470 tx=0x7fb34c005e50 comp rx=0 tx=0).stop 2024-09-15T23:33:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 shutdown_connections 2024-09-15T23:33:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555b62fc9370 0x555b63047600 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555b62fc42e0 0x555b62fd31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb337fff640 0 client.18795 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b62fc39a0 0x555b62fd5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 --2- 172.21.15.89:0/4080621067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555b62fbf010 0x555b62fd4ae0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 >> 172.21.15.89:0/4080621067 conn(0x555b62fc8ec0 msgr2=0x555b62fbfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:52.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 shutdown_connections 2024-09-15T23:33:52.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.973+0000 7f3457d5e640 1 -- 172.21.15.89:0/1291772198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3450102ef0 msgr2=0x7f345010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:52.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.973+0000 7f3457d5e640 1 --2- 172.21.15.89:0/1291772198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3450102ef0 0x7f345010da40 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f3448009ca0 tx=0x7f344802a950 comp rx=0 tx=0).stop 2024-09-15T23:33:52.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.973+0000 7f3457d5e640 1 -- 172.21.15.89:0/1291772198 shutdown_connections 2024-09-15T23:33:52.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.973+0000 7f3457d5e640 1 --2- 172.21.15.89:0/1291772198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3450102ef0 0x7f345010da40 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.973+0000 7f3457d5e640 1 --2- 172.21.15.89:0/1291772198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3450102570 0x7f34501029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.973+0000 7f3457d5e640 1 --2- 172.21.15.89:0/1291772198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3450100e20 0x7f3450101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:52.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.973+0000 7f3457d5e640 1 -- 172.21.15.89:0/1291772198 >> 172.21.15.89:0/1291772198 conn(0x7f34500fc7d0 msgr2=0x7f34500febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:52.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.973+0000 7f3457d5e640 1 -- 172.21.15.89:0/1291772198 shutdown_connections 2024-09-15T23:33:52.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.973+0000 7f3457d5e640 1 -- 172.21.15.89:0/1291772198 wait complete. 2024-09-15T23:33:52.998 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 1 -- 172.21.15.89:0/4080621067 wait complete. 2024-09-15T23:33:53.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f3457d5e640 1 Processor -- start 2024-09-15T23:33:53.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f3457d5e640 1 -- start start 2024-09-15T23:33:53.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f3457d5e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3450100e20 0x7f345019a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f3457d5e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3450102570 0x7f345019ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.000 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f3457d5e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3450102ef0 0x7f345019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f3457d5e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3450110f80 con 0x7f3450102570 2024-09-15T23:33:53.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f3457d5e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3450110e00 con 0x7f3450100e20 2024-09-15T23:33:53.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f3457d5e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3450111100 con 0x7f3450102ef0 2024-09-15T23:33:53.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f34552f9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3450102570 0x7f345019ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f34552f9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3450102570 0x7f345019ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49242/0 (socket says 172.21.15.89:49242) 2024-09-15T23:33:53.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f34552f9640 1 -- 172.21.15.89:0/3782874980 learned_addr learned my addr 172.21.15.89:0/3782874980 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:53.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f34562fb640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3450102ef0 0x7f345019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f3455afa640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3450100e20 0x7f345019a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f34552f9640 1 -- 172.21.15.89:0/3782874980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3450102ef0 msgr2=0x7f345019ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:53.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f34552f9640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3450102ef0 0x7f345019ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f34552f9640 1 -- 172.21.15.89:0/3782874980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3450100e20 msgr2=0x7f345019a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:53.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f34552f9640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3450100e20 0x7f345019a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.974+0000 7f34552f9640 1 -- 172.21.15.89:0/3782874980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f345019f600 con 0x7f3450102570 2024-09-15T23:33:53.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.975+0000 7f3455afa640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3450100e20 0x7f345019a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:53.003 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.975+0000 7f34552f9640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3450102570 0x7f345019ab90 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f344000c890 tx=0x7f344000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.975+0000 7f3446ffd640 1 -- 172.21.15.89:0/3782874980 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3440004220 con 0x7f3450102570 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.975+0000 7f3446ffd640 1 -- 172.21.15.89:0/3782874980 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34400043c0 con 0x7f3450102570 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.975+0000 7f3457d5e640 1 -- 172.21.15.89:0/3782874980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f345019f8f0 con 0x7f3450102570 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.975+0000 7f3446ffd640 1 -- 172.21.15.89:0/3782874980 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34400049e0 con 0x7f3450102570 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.975+0000 7f3457d5e640 1 -- 172.21.15.89:0/3782874980 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34501a71d0 con 0x7f3450102570 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f345a137740 1 Processor -- start 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f345a137740 1 -- start start 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f3446ffd640 1 -- 172.21.15.89:0/3782874980 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f34400079b0 con 0x7f3450102570 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f345a137740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c9b40 0x55a0185c8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f345a137740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 0x55a0185c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f345a137740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185d5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f345a137740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a01852e2f0 con 0x55a0185d3370 2024-09-15T23:33:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f345a137740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a0184d5cf0 con 0x55a0185c85d0 2024-09-15T23:33:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f345a137740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a01852a2b0 con 0x55a0185c9b40 2024-09-15T23:33:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f34552f9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185d5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f34552f9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185d5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49264/0 (socket says 172.21.15.89:49264) 2024-09-15T23:33:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f34562fb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c9b40 0x55a0185c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f34552f9640 1 -- 172.21.15.89:0/362577870 learned_addr learned my addr 172.21.15.89:0/362577870 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.978+0000 7f3455afa640 1 --2- 172.21.15.89:0/362577870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 0x55a0185c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f34552f9640 1 -- 172.21.15.89:0/362577870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c9b40 msgr2=0x55a0185c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f34552f9640 1 --2- 172.21.15.89:0/362577870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c9b40 0x55a0185c8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f34552f9640 1 -- 172.21.15.89:0/362577870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 msgr2=0x55a0185c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f34552f9640 1 --2- 172.21.15.89:0/362577870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 0x55a0185c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f34552f9640 1 -- 172.21.15.89:0/362577870 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0184ba3a0 con 0x55a0185d3370 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f3455afa640 1 --2- 172.21.15.89:0/362577870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 0x55a0185c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f34552f9640 1 --2- 172.21.15.89:0/362577870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185d5760 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f34400d3d90 tx=0x7f34400d3e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7620d7e2f7780a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f3446ffd640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f343c0a32c0 0x7f343c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f343bfff640 1 -- 172.21.15.89:0/362577870 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34400df070 con 0x55a0185d3370 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f343bfff640 1 -- 172.21.15.89:0/362577870 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344009bed0 con 0x55a0185d3370 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f343bfff640 1 -- 172.21.15.89:0/362577870 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34400c6e50 con 0x55a0185d3370 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f34562fb640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f343c0a32c0 0x7f343c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.979+0000 7f3446ffd640 1 -- 172.21.15.89:0/3782874980 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f34400cfbf0 con 0x7f3450102570 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.980+0000 7f345a137740 1 -- 172.21.15.89:0/362577870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 msgr2=0x55a0185d5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.980+0000 7f345a137740 1 --2- 172.21.15.89:0/362577870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185d5760 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f34400d3d90 tx=0x7f34400d3e80 comp rx=0 tx=0).stop 2024-09-15T23:33:53.006 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.980+0000 7f34562fb640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f343c0a32c0 0x7f343c0a5780 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3448005fe0 tx=0x7f3448005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.980+0000 7f345a137740 1 -- 172.21.15.89:0/362577870 shutdown_connections 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.980+0000 7f345a137740 1 --2- 172.21.15.89:0/362577870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185d5760 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.980+0000 7f345a137740 1 --2- 172.21.15.89:0/362577870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 0x55a0185c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.980+0000 7f345a137740 1 --2- 172.21.15.89:0/362577870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c9b40 0x55a0185c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.980+0000 7f345a137740 1 -- 172.21.15.89:0/362577870 >> 172.21.15.89:0/362577870 conn(0x55a018591310 msgr2=0x55a0185c4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.980+0000 7f345a137740 1 -- 172.21.15.89:0/362577870 shutdown_connections 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.980+0000 7f345a137740 1 -- 172.21.15.89:0/362577870 wait complete. 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.981+0000 7f345a137740 1 -- start start 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.981+0000 7f345a137740 1 Processor -- start 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f345a137740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 0x55a0185d2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f345a137740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f345a137740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185d3370 0x55a0185d0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f345a137740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a0185d6070 con 0x55a0185c9b40 2024-09-15T23:33:53.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f345a137740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a0184e9060 con 0x55a0185d3370 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f345a137740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a0185d61f0 con 0x55a0185c85d0 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f345a137740 10 client.0 _ll_register_callbacks cb 0x55a01823c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f3455afa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 0x55a0185d2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f3455afa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 0x55a0185d2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:44354/0 (socket says 172.21.15.89:44354) 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f3455afa640 1 -- 172.21.15.89:0/3278205837 learned_addr learned my addr 172.21.15.89:0/3278205837 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f34562fb640 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185d3370 0x55a0185d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f34552f9640 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f34562fb640 1 -- 172.21.15.89:0/3278205837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 msgr2=0x55a0185d2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f34562fb640 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 0x55a0185d2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f34562fb640 1 -- 172.21.15.89:0/3278205837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 msgr2=0x55a0185cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:53.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f34562fb640 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f34562fb640 1 -- 172.21.15.89:0/3278205837 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a01829f3c0 con 0x55a0185d3370 2024-09-15T23:33:53.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.982+0000 7f34552f9640 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:53.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.983+0000 7f34562fb640 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185d3370 0x55a0185d0490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3448005cc0 tx=0x7f34480095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7a825c4a1f28a1e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:53.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.983+0000 7f34397fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:33:53.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.983+0000 7f34397fa640 1 -- 172.21.15.89:0/3278205837 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3448056060 con 0x55a0185d3370 2024-09-15T23:33:53.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.983+0000 7f345a137740 1 -- 172.21.15.89:0/3278205837 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f344009bed0 con 0x55a0185d3370 2024-09-15T23:33:53.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.983+0000 7f34397fa640 1 -- 172.21.15.89:0/3278205837 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3448004250 con 0x55a0185d3370 2024-09-15T23:33:53.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.983+0000 7f34397fa640 1 -- 172.21.15.89:0/3278205837 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f344805d050 con 0x55a0185d3370 2024-09-15T23:33:53.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.983+0000 7f345a137740 10 client.27146 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.983+0000 7f345a137740 1 -- 172.21.15.89:0/3278205837 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a0185ea2a0 con 0x55a0185d3370 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.983+0000 7f34397fa640 1 -- 172.21.15.89:0/3278205837 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344009bed0 con 0x55a0185d3370 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.984+0000 7f34397fa640 1 -- 172.21.15.89:0/3278205837 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3448030be0 con 0x55a0185d3370 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.984+0000 7f345a137740 10 client.27146 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.984+0000 7f345a137740 10 client.27146 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.984+0000 7f345a137740 20 client.27146 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.984+0000 7f345a137740 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55a0185eb760 0x55a0185edb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.984+0000 7f345a137740 4 client.27146 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.984+0000 7f345a137740 1 -- 172.21.15.89:0/3278205837 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a01853b860 con 0x55a0185eb760 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.984+0000 7f3455afa640 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55a0185eb760 0x55a0185edb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.984+0000 7f34397fa640 10 client.27146 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:33:53.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.984+0000 7f3455afa640 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55a0185eb760 0x55a0185edb50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3fdcba48e921f8e8 server_cookie=209a61ea6cbbb061 in_seq=0 out_seq=0 2024-09-15T23:33:53.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.985+0000 7f34397fa640 1 -- 172.21.15.89:0/3278205837 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a01853b860 con 0x55a0185eb760 2024-09-15T23:33:53.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.985+0000 7f34397fa640 10 client.27146 handle_command_reply: tid=0 2024-09-15T23:33:53.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.985+0000 7f345a137740 1 client.27146 shutdown 2024-09-15T23:33:53.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.985+0000 7f345a137740 1 -- 172.21.15.89:0/3278205837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185d3370 msgr2=0x55a0185d0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:53.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.985+0000 7f345a137740 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185d3370 0x55a0185d0490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3448005cc0 tx=0x7f34480095c0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.985+0000 7f345a137740 1 -- 172.21.15.89:0/3278205837 shutdown_connections 2024-09-15T23:33:53.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.985+0000 7f345a137740 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55a0185eb760 0x55a0185edb50 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.986+0000 7f345a137740 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185d3370 0x55a0185d0490 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.986+0000 7f34397fa640 0 client.27146 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:33:53.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.986+0000 7f345a137740 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.986+0000 7f345a137740 1 --2- 172.21.15.89:0/3278205837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 0x55a0185d2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.986+0000 7f345a137740 1 -- 172.21.15.89:0/3278205837 >> 172.21.15.89:0/3278205837 conn(0x55a018591310 msgr2=0x55a0185ca960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:53.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.986+0000 7f345a137740 1 -- 172.21.15.89:0/3278205837 shutdown_connections 2024-09-15T23:33:53.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.986+0000 7f345a137740 1 -- 172.21.15.89:0/3278205837 wait complete. 2024-09-15T23:33:53.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:52.986+0000 7f345a137740 20 client.27146 trim_cache size 0 max 16384 2024-09-15T23:33:53.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.987+0000 7fb362001740 20 client.18795 trim_cache size 0 max 16384 2024-09-15T23:33:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:33:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:33:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:33:53.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:33:53.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 -- 172.21.15.89:0/51413988 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3480a32f0 msgr2=0x7fb3480a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:53.015 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3480a32f0 0x7fb3480a57b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb344005fd0 tx=0x7fb34400b390 comp rx=0 tx=0).stop 2024-09-15T23:33:53.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 -- 172.21.15.89:0/51413988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb358100a40 msgr2=0x7fb358196d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:53.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb358100a40 0x7fb358196d30 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fb34c031610 tx=0x7fb34c0043b0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 -- 172.21.15.89:0/51413988 shutdown_connections 2024-09-15T23:33:53.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3480a32f0 0x7fb3480a57b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3581025f0 0x7fb35819e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb358101c30 0x7fb358197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 --2- 172.21.15.89:0/51413988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb358100a40 0x7fb358196d30 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 -- 172.21.15.89:0/51413988 >> 172.21.15.89:0/51413988 conn(0x7fb3580fc1e0 msgr2=0x7fb358105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:53.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 -- 172.21.15.89:0/51413988 shutdown_connections 2024-09-15T23:33:53.016 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:33:52.988+0000 7fb35f18d640 1 -- 172.21.15.89:0/51413988 wait complete. 2024-09-15T23:33:53.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.005+0000 7f345a137740 1 Processor -- start 2024-09-15T23:33:53.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.005+0000 7f345a137740 1 -- start start 2024-09-15T23:33:53.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f345a137740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 0x55a0185d1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f345a137740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f345a137740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185d3370 0x55a0185cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f345a137740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f34501acbc0 con 0x55a0185c9b40 2024-09-15T23:33:53.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f345a137740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34501ae700 con 0x55a0185c85d0 2024-09-15T23:33:53.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f345a137740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f34501ad290 con 0x55a0185d3370 2024-09-15T23:33:53.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f34552f9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f34552f9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49282/0 (socket says 172.21.15.89:49282) 2024-09-15T23:33:53.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f34552f9640 1 -- 172.21.15.89:0/2966541302 learned_addr learned my addr 172.21.15.89:0/2966541302 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:53.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f34562fb640 1 --2- 172.21.15.89:0/2966541302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185d3370 0x55a0185cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f3455afa640 1 --2- 172.21.15.89:0/2966541302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 0x55a0185d1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f34552f9640 1 -- 172.21.15.89:0/2966541302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185d3370 msgr2=0x55a0185cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:53.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f34552f9640 1 --2- 172.21.15.89:0/2966541302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185d3370 0x55a0185cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f34552f9640 1 -- 172.21.15.89:0/2966541302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 msgr2=0x55a0185d1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:53.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f34552f9640 1 --2- 172.21.15.89:0/2966541302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 0x55a0185d1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f34552f9640 1 -- 172.21.15.89:0/2966541302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0185edec0 con 0x55a0185c9b40 2024-09-15T23:33:53.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f3455afa640 1 --2- 172.21.15.89:0/2966541302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 0x55a0185d1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:33:53.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f34552f9640 1 --2- 172.21.15.89:0/2966541302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cb120 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f34400cd670 tx=0x7f34400c6bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ce7e324e46c6b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:53.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.006+0000 7f343b7fe640 1 -- 172.21.15.89:0/2966541302 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34400ca970 con 0x55a0185c9b40 2024-09-15T23:33:53.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f343b7fe640 1 -- 172.21.15.89:0/2966541302 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34400d34a0 con 0x55a0185c9b40 2024-09-15T23:33:53.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f343b7fe640 1 -- 172.21.15.89:0/2966541302 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34400c8da0 con 0x55a0185c9b40 2024-09-15T23:33:53.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f345a137740 1 -- 172.21.15.89:0/2966541302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 msgr2=0x55a0185cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:53.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f345a137740 1 --2- 172.21.15.89:0/2966541302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cb120 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f34400cd670 tx=0x7f34400c6bf0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f345a137740 1 -- 172.21.15.89:0/2966541302 shutdown_connections 2024-09-15T23:33:53.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f345a137740 1 --2- 172.21.15.89:0/2966541302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185d3370 0x55a0185cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f345a137740 1 --2- 172.21.15.89:0/2966541302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185c9b40 0x55a0185cb120 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f345a137740 1 --2- 172.21.15.89:0/2966541302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c85d0 0x55a0185d1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f345a137740 1 -- 172.21.15.89:0/2966541302 >> 172.21.15.89:0/2966541302 conn(0x55a018591310 msgr2=0x55a0185d17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:53.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f345a137740 1 -- 172.21.15.89:0/2966541302 shutdown_connections 2024-09-15T23:33:53.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.007+0000 7f345a137740 1 -- 172.21.15.89:0/2966541302 wait complete. 2024-09-15T23:33:53.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f345a137740 1 -- start start 2024-09-15T23:33:53.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f345a137740 1 Processor -- start 2024-09-15T23:33:53.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f345a137740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 0x55a0185dbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f345a137740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c9b40 0x55a0185ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f345a137740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185cd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f345a137740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f34501ae030 con 0x55a0185d3370 2024-09-15T23:33:53.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f345a137740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34501ad960 con 0x55a0185c9b40 2024-09-15T23:33:53.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f345a137740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f34501aabc0 con 0x55a0185c85d0 2024-09-15T23:33:53.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f345a137740 10 client.0 _ll_register_callbacks cb 0x7f344802f500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:53.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f34562fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f34562fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185cd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49284/0 (socket says 172.21.15.89:49284) 2024-09-15T23:33:53.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f34562fb640 1 -- 172.21.15.89:0/872455889 learned_addr learned my addr 172.21.15.89:0/872455889 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:53.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f3455afa640 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 0x55a0185dbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.008+0000 7f34552f9640 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c9b40 0x55a0185ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34562fb640 1 -- 172.21.15.89:0/872455889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 msgr2=0x55a0185dbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:53.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34562fb640 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 0x55a0185dbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34562fb640 1 -- 172.21.15.89:0/872455889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c9b40 msgr2=0x55a0185ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:53.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34562fb640 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c9b40 0x55a0185ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34562fb640 1 -- 172.21.15.89:0/872455889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0185cd950 con 0x55a0185d3370 2024-09-15T23:33:53.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34552f9640 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c9b40 0x55a0185ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:53.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34562fb640 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185cd120 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f344802a990 tx=0x7f344802f930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8502a65acea68b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:53.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34397fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:53.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34397fa640 1 -- 172.21.15.89:0/872455889 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3448066060 con 0x55a0185d3370 2024-09-15T23:33:53.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f345a137740 1 -- 172.21.15.89:0/872455889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34400d34a0 con 0x55a0185d3370 2024-09-15T23:33:53.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34397fa640 1 -- 172.21.15.89:0/872455889 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f344803aa10 con 0x55a0185d3370 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34397fa640 1 -- 172.21.15.89:0/872455889 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f344806d050 con 0x55a0185d3370 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f345a137740 10 client.18807 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f345a137740 1 -- 172.21.15.89:0/872455889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a018656310 con 0x55a0185d3370 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34397fa640 1 -- 172.21.15.89:0/872455889 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34400d34a0 con 0x55a0185d3370 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.009+0000 7f34397fa640 1 -- 172.21.15.89:0/872455889 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3448030560 con 0x55a0185d3370 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f345a137740 10 client.18807 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f345a137740 10 client.18807 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f345a137740 20 client.18807 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f345a137740 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55a0186569d0 0x55a018658df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f345a137740 4 client.18807 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:33:53.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f345a137740 1 -- 172.21.15.89:0/872455889 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a018535d70 con 0x55a0186569d0 2024-09-15T23:33:53.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f3455afa640 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55a0186569d0 0x55a018658df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:53.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f3455afa640 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55a0186569d0 0x55a018658df0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=decc4209ecd9a9ea server_cookie=a5be642d564eb335 in_seq=0 out_seq=0 2024-09-15T23:33:53.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f34397fa640 10 client.18807 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:33:53.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f34397fa640 1 -- 172.21.15.89:0/872455889 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a018535d70 con 0x55a0186569d0 2024-09-15T23:33:53.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f34397fa640 10 client.18807 handle_command_reply: tid=0 2024-09-15T23:33:53.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f345a137740 1 client.18807 shutdown 2024-09-15T23:33:53.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f345a137740 1 -- 172.21.15.89:0/872455889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 msgr2=0x55a0185cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:53.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.010+0000 7f345a137740 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185cd120 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f344802a990 tx=0x7f344802f930 comp rx=0 tx=0).stop 2024-09-15T23:33:53.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f345a137740 1 -- 172.21.15.89:0/872455889 shutdown_connections 2024-09-15T23:33:53.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f345a137740 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55a0186569d0 0x55a018658df0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f345a137740 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a0185d3370 0x55a0185cd120 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f345a137740 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0185c9b40 0x55a0185ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f34397fa640 0 client.18807 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:33:53.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f345a137740 1 --2- 172.21.15.89:0/872455889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a0185c85d0 0x55a0185dbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f345a137740 1 -- 172.21.15.89:0/872455889 >> 172.21.15.89:0/872455889 conn(0x55a018591310 msgr2=0x55a0185c34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:53.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f345a137740 1 -- 172.21.15.89:0/872455889 shutdown_connections 2024-09-15T23:33:53.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f345a137740 1 -- 172.21.15.89:0/872455889 wait complete. 2024-09-15T23:33:53.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f345a137740 20 client.18807 trim_cache size 0 max 16384 2024-09-15T23:33:53.033 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:33:53.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 -- 172.21.15.89:0/3782874980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f343c0a32c0 msgr2=0x7f343c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:53.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f343c0a32c0 0x7f343c0a5780 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3448005fe0 tx=0x7f3448005c50 comp rx=0 tx=0).stop 2024-09-15T23:33:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 -- 172.21.15.89:0/3782874980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3450102570 msgr2=0x7f345019ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3450102570 0x7f345019ab90 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f344000c890 tx=0x7f344000cd50 comp rx=0 tx=0).stop 2024-09-15T23:33:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 -- 172.21.15.89:0/3782874980 shutdown_connections 2024-09-15T23:33:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f343c0a32c0 0x7f343c0a5780 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3450102ef0 0x7f345019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3450102570 0x7f345019ab90 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 --2- 172.21.15.89:0/3782874980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3450100e20 0x7f345019a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 -- 172.21.15.89:0/3782874980 >> 172.21.15.89:0/3782874980 conn(0x7f34500fc7d0 msgr2=0x7f345006a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 -- 172.21.15.89:0/3782874980 shutdown_connections 2024-09-15T23:33:53.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:53.011+0000 7f3457d5e640 1 -- 172.21.15.89:0/3782874980 wait complete. 2024-09-15T23:33:53.873 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:33:54.065 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:33:54.118 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:54.525 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:54 smithi089 ceph-mon[30933]: pgmap v646: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-09-15T23:33:54.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.604+0000 7f948dccd640 1 -- 172.21.15.89:0/1008016934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94881036e0 msgr2=0x7f9488107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:54.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.604+0000 7f948dccd640 1 --2- 172.21.15.89:0/1008016934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94881036e0 0x7f9488107b90 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f9470009930 tx=0x7f947002a420 comp rx=0 tx=0).stop 2024-09-15T23:33:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.604+0000 7f948dccd640 1 -- 172.21.15.89:0/1008016934 shutdown_connections 2024-09-15T23:33:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.604+0000 7f948dccd640 1 --2- 172.21.15.89:0/1008016934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94881082a0 0x7f948810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.604+0000 7f948dccd640 1 --2- 172.21.15.89:0/1008016934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94881036e0 0x7f9488107b90 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.604+0000 7f948dccd640 1 --2- 172.21.15.89:0/1008016934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9488102d30 0x7f9488103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.604+0000 7f948dccd640 1 -- 172.21.15.89:0/1008016934 >> 172.21.15.89:0/1008016934 conn(0x7f94880fc7d0 msgr2=0x7f94880febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.604+0000 7f948dccd640 1 -- 172.21.15.89:0/1008016934 shutdown_connections 2024-09-15T23:33:54.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.604+0000 7f948dccd640 1 -- 172.21.15.89:0/1008016934 wait complete. 2024-09-15T23:33:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f948dccd640 1 Processor -- start 2024-09-15T23:33:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f948dccd640 1 -- start start 2024-09-15T23:33:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f948dccd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9488102d30 0x7f94881985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f948dccd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94881036e0 0x7f9488198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f948dccd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94881082a0 0x7f948819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f948dccd640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9488112090 con 0x7f9488102d30 2024-09-15T23:33:54.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f948dccd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9488111f10 con 0x7f94881082a0 2024-09-15T23:33:54.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f948dccd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9488112210 con 0x7f94881036e0 2024-09-15T23:33:54.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f94877fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9488102d30 0x7f94881985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f94877fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9488102d30 0x7f94881985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49304/0 (socket says 172.21.15.89:49304) 2024-09-15T23:33:54.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f94877fe640 1 -- 172.21.15.89:0/843255944 learned_addr learned my addr 172.21.15.89:0/843255944 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:54.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f9487fff640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94881082a0 0x7f948819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f947effd640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94881036e0 0x7f9488198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f94877fe640 1 -- 172.21.15.89:0/843255944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94881036e0 msgr2=0x7f9488198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:54.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f94877fe640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94881036e0 0x7f9488198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f94877fe640 1 -- 172.21.15.89:0/843255944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94881082a0 msgr2=0x7f948819cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:54.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f94877fe640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94881082a0 0x7f948819cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.605+0000 7f94877fe640 1 -- 172.21.15.89:0/843255944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948819d580 con 0x7f9488102d30 2024-09-15T23:33:54.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.606+0000 7f94877fe640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9488102d30 0x7f94881985d0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f947800d950 tx=0x7f947800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:54.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.606+0000 7f94857fa640 1 -- 172.21.15.89:0/843255944 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478017070 con 0x7f9488102d30 2024-09-15T23:33:54.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.606+0000 7f94857fa640 1 -- 172.21.15.89:0/843255944 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9478009c30 con 0x7f9488102d30 2024-09-15T23:33:54.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.606+0000 7f948dccd640 1 -- 172.21.15.89:0/843255944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f948819d870 con 0x7f9488102d30 2024-09-15T23:33:54.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.606+0000 7f94857fa640 1 -- 172.21.15.89:0/843255944 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947800f6b0 con 0x7f9488102d30 2024-09-15T23:33:54.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.606+0000 7f948dccd640 1 -- 172.21.15.89:0/843255944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94881a5150 con 0x7f9488102d30 2024-09-15T23:33:54.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.608+0000 7f94857fa640 1 -- 172.21.15.89:0/843255944 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f94780109d0 con 0x7f9488102d30 2024-09-15T23:33:54.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.608+0000 7f94900a6740 1 Processor -- start 2024-09-15T23:33:54.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94900a6740 1 -- start start 2024-09-15T23:33:54.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94900a6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f8715b40 0x5650f8714090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94857fa640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94740a3240 0x7f94740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94900a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f87145d0 0x5650f8714a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94900a6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8723780 0x5650f8725b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94900a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5650f867a2f0 con 0x5650f87145d0 2024-09-15T23:33:54.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94900a6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5650f8621cf0 con 0x5650f8723780 2024-09-15T23:33:54.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94900a6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5650f86762b0 con 0x5650f8715b40 2024-09-15T23:33:54.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94877fe640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94740a3240 0x7f94740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f9487fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f8715b40 0x5650f8714090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f947effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f87145d0 0x5650f8714a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f9487fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f8715b40 0x5650f8714090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:44388/0 (socket says 172.21.15.89:44388) 2024-09-15T23:33:54.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f9487fff640 1 -- 172.21.15.89:0/2694490155 learned_addr learned my addr 172.21.15.89:0/2694490155 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:54.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94857fa640 1 -- 172.21.15.89:0/843255944 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f94780cf990 con 0x7f9488102d30 2024-09-15T23:33:54.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f9487fff640 1 --2- 172.21.15.89:0/2694490155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8723780 0x5650f8725b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f9487fff640 1 -- 172.21.15.89:0/2694490155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8723780 msgr2=0x5650f8725b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:54.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f9487fff640 1 --2- 172.21.15.89:0/2694490155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8723780 0x5650f8725b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f9487fff640 1 -- 172.21.15.89:0/2694490155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f87145d0 msgr2=0x5650f8714a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:54.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f9487fff640 1 --2- 172.21.15.89:0/2694490155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f87145d0 0x5650f8714a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f9487fff640 1 -- 172.21.15.89:0/2694490155 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650f86063a0 con 0x5650f8715b40 2024-09-15T23:33:54.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.609+0000 7f94877fe640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94740a3240 0x7f94740a5700 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f947809b8c0 tx=0x7f947806a900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:54.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.610+0000 7f9487fff640 1 --2- 172.21.15.89:0/2694490155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f8715b40 0x5650f8714090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9480005c20 tx=0x7f9480009aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f649ed7305e44b02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:54.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.610+0000 7f947dffb640 1 -- 172.21.15.89:0/2694490155 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480003b70 con 0x5650f8715b40 2024-09-15T23:33:54.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.610+0000 7f947dffb640 1 -- 172.21.15.89:0/2694490155 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9480010dd0 con 0x5650f8715b40 2024-09-15T23:33:54.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.610+0000 7f947dffb640 1 -- 172.21.15.89:0/2694490155 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948001d3a0 con 0x5650f8715b40 2024-09-15T23:33:54.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.610+0000 7f94900a6740 1 -- 172.21.15.89:0/2694490155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f8715b40 msgr2=0x5650f8714090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:54.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.610+0000 7f94900a6740 1 --2- 172.21.15.89:0/2694490155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f8715b40 0x5650f8714090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9480005c20 tx=0x7f9480009aa0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 -- 172.21.15.89:0/2694490155 shutdown_connections 2024-09-15T23:33:54.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 --2- 172.21.15.89:0/2694490155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8723780 0x5650f8725b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 --2- 172.21.15.89:0/2694490155 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f87145d0 0x5650f8714a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 --2- 172.21.15.89:0/2694490155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f8715b40 0x5650f8714090 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 -- 172.21.15.89:0/2694490155 >> 172.21.15.89:0/2694490155 conn(0x5650f86dd310 msgr2=0x5650f8710c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:54.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 -- 172.21.15.89:0/2694490155 shutdown_connections 2024-09-15T23:33:54.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 -- 172.21.15.89:0/2694490155 wait complete. 2024-09-15T23:33:54.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 -- start start 2024-09-15T23:33:54.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 Processor -- start 2024-09-15T23:33:54.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f8723170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 0x5650f8720420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f8720a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5650f8726480 con 0x5650f8723780 2024-09-15T23:33:54.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5650f8635060 con 0x5650f8715b40 2024-09-15T23:33:54.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5650f8726600 con 0x5650f87145d0 2024-09-15T23:33:54.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.611+0000 7f94900a6740 10 client.0 _ll_register_callbacks cb 0x5650f83888c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:54.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9487fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f8720a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9487fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f8720a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49310/0 (socket says 172.21.15.89:49310) 2024-09-15T23:33:54.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9487fff640 1 -- 172.21.15.89:0/829014997 learned_addr learned my addr 172.21.15.89:0/829014997 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:54.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9487fff640 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f8723170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f947effd640 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 0x5650f8720420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9487fff640 1 -- 172.21.15.89:0/829014997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 msgr2=0x5650f8723170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:54.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9487fff640 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f8723170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9487fff640 1 -- 172.21.15.89:0/829014997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 msgr2=0x5650f8720420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:54.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9487fff640 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 0x5650f8720420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9487fff640 1 -- 172.21.15.89:0/829014997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650f83eb3c0 con 0x5650f8723780 2024-09-15T23:33:54.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9487fff640 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f8720a10 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f9480004370 tx=0x7f9480004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a15cdfbf2bce8a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:54.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9466ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:54.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9466ffd640 1 -- 172.21.15.89:0/829014997 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480027040 con 0x5650f8723780 2024-09-15T23:33:54.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9466ffd640 1 -- 172.21.15.89:0/829014997 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9480010af0 con 0x5650f8723780 2024-09-15T23:33:54.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f94900a6740 1 -- 172.21.15.89:0/829014997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9480010dd0 con 0x5650f8723780 2024-09-15T23:33:54.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f9466ffd640 1 -- 172.21.15.89:0/829014997 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9480036050 con 0x5650f8723780 2024-09-15T23:33:54.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f94900a6740 10 client.18819 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:54.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.612+0000 7f94900a6740 1 -- 172.21.15.89:0/829014997 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5650f8731f20 con 0x5650f8723780 2024-09-15T23:33:54.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.613+0000 7f9466ffd640 1 -- 172.21.15.89:0/829014997 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9480010dd0 con 0x5650f8723780 2024-09-15T23:33:54.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.613+0000 7f9466ffd640 1 -- 172.21.15.89:0/829014997 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f948000f040 con 0x5650f8723780 2024-09-15T23:33:54.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.613+0000 7f94900a6740 10 client.18819 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.613+0000 7f94900a6740 10 client.18819 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.613+0000 7f94900a6740 20 client.18819 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.613+0000 7f94900a6740 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5650f8733270 0x5650f8735660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.613+0000 7f94900a6740 4 client.18819 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.613+0000 7f94900a6740 1 -- 172.21.15.89:0/829014997 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650f8687860 con 0x5650f8733270 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.613+0000 7f947effd640 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5650f8733270 0x5650f8735660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.614+0000 7f947effd640 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5650f8733270 0x5650f8735660 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c42c841d7c807b8 server_cookie=d5433c678d5bd7a9 in_seq=0 out_seq=0 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.614+0000 7f9466ffd640 10 client.18819 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.614+0000 7f9466ffd640 1 -- 172.21.15.89:0/829014997 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5650f8687860 con 0x5650f8733270 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.614+0000 7f9466ffd640 10 client.18819 handle_command_reply: tid=0 2024-09-15T23:33:54.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.614+0000 7f94900a6740 1 client.18819 shutdown 2024-09-15T23:33:54.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 1 -- 172.21.15.89:0/829014997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 msgr2=0x5650f8720a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:54.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f8720a10 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f9480004370 tx=0x7f9480004470 comp rx=0 tx=0).stop 2024-09-15T23:33:54.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 1 -- 172.21.15.89:0/829014997 shutdown_connections 2024-09-15T23:33:54.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f8720a10 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 0x5650f8720420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5650f8733270 0x5650f8735660 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 1 --2- 172.21.15.89:0/829014997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f8723170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 1 -- 172.21.15.89:0/829014997 >> 172.21.15.89:0/829014997 conn(0x5650f86dd310 msgr2=0x5650f8716960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:54.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f9466ffd640 0 client.18819 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:33:54.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 1 -- 172.21.15.89:0/829014997 shutdown_connections 2024-09-15T23:33:54.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 1 -- 172.21.15.89:0/829014997 wait complete. 2024-09-15T23:33:54.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.615+0000 7f94900a6740 20 client.18819 trim_cache size 0 max 16384 2024-09-15T23:33:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.628+0000 7f94900a6740 1 Processor -- start 2024-09-15T23:33:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.628+0000 7f94900a6740 1 -- start start 2024-09-15T23:33:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.628+0000 7f94900a6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f872f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.628+0000 7f94900a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8715b40 0x5650f871fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.628+0000 7f94900a6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8723780 0x5650f8722390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.628+0000 7f94900a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f94881acd10 con 0x5650f8715b40 2024-09-15T23:33:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.628+0000 7f94900a6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94881ae850 con 0x5650f8723780 2024-09-15T23:33:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.628+0000 7f94900a6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94881ad3e0 con 0x5650f87145d0 2024-09-15T23:33:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8715b40 0x5650f871fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8715b40 0x5650f871fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49326/0 (socket says 172.21.15.89:49326) 2024-09-15T23:33:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947effd640 1 -- 172.21.15.89:0/3486358643 learned_addr learned my addr 172.21.15.89:0/3486358643 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f9487fff640 1 --2- 172.21.15.89:0/3486358643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f872f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f9487fff640 1 --2- 172.21.15.89:0/3486358643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8723780 0x5650f8722390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947effd640 1 -- 172.21.15.89:0/3486358643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 msgr2=0x5650f872f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947effd640 1 --2- 172.21.15.89:0/3486358643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f872f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947effd640 1 -- 172.21.15.89:0/3486358643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8723780 msgr2=0x5650f8722390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947effd640 1 --2- 172.21.15.89:0/3486358643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8723780 0x5650f8722390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947effd640 1 -- 172.21.15.89:0/3486358643 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650f871e970 con 0x5650f8715b40 2024-09-15T23:33:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f9487fff640 1 --2- 172.21.15.89:0/3486358643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f872f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947effd640 1 --2- 172.21.15.89:0/3486358643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8715b40 0x5650f871fce0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f947002a9e0 tx=0x7f9470005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b074342c784c9c8d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947d7fa640 1 -- 172.21.15.89:0/3486358643 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470035040 con 0x5650f8715b40 2024-09-15T23:33:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947d7fa640 1 -- 172.21.15.89:0/3486358643 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947004a070 con 0x5650f8715b40 2024-09-15T23:33:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.629+0000 7f947d7fa640 1 -- 172.21.15.89:0/3486358643 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470031810 con 0x5650f8715b40 2024-09-15T23:33:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.630+0000 7f94900a6740 1 -- 172.21.15.89:0/3486358643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8715b40 msgr2=0x5650f871fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.630+0000 7f94900a6740 1 --2- 172.21.15.89:0/3486358643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8715b40 0x5650f871fce0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f947002a9e0 tx=0x7f9470005b50 comp rx=0 tx=0).stop 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.630+0000 7f94900a6740 1 -- 172.21.15.89:0/3486358643 shutdown_connections 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.630+0000 7f94900a6740 1 --2- 172.21.15.89:0/3486358643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8723780 0x5650f8722390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.630+0000 7f94900a6740 1 --2- 172.21.15.89:0/3486358643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8715b40 0x5650f871fce0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.630+0000 7f94900a6740 1 --2- 172.21.15.89:0/3486358643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f872f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.630+0000 7f94900a6740 1 -- 172.21.15.89:0/3486358643 >> 172.21.15.89:0/3486358643 conn(0x5650f86dd310 msgr2=0x5650f8721c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.630+0000 7f94900a6740 1 -- 172.21.15.89:0/3486358643 shutdown_connections 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.630+0000 7f94900a6740 1 -- 172.21.15.89:0/3486358643 wait complete. 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.630+0000 7f94900a6740 1 -- start start 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f94900a6740 1 Processor -- start 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f94900a6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f8727630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f94900a6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 0x5650f8727b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f94900a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f87280b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f94900a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f94881ae180 con 0x5650f8723780 2024-09-15T23:33:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f94900a6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94881adab0 con 0x5650f8715b40 2024-09-15T23:33:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f94900a6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94881aacf0 con 0x5650f87145d0 2024-09-15T23:33:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f94900a6740 10 client.0 _ll_register_callbacks cb 0x7f9470009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f9487fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f87280b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f9487fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f87280b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49330/0 (socket says 172.21.15.89:49330) 2024-09-15T23:33:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f9487fff640 1 -- 172.21.15.89:0/3573820730 learned_addr learned my addr 172.21.15.89:0/3573820730 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f947effd640 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 0x5650f8727b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.631+0000 7f9487fff640 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f8727630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9487fff640 1 -- 172.21.15.89:0/3573820730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 msgr2=0x5650f8727630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9487fff640 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f8727630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9487fff640 1 -- 172.21.15.89:0/3573820730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 msgr2=0x5650f8727b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9487fff640 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 0x5650f8727b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9487fff640 1 -- 172.21.15.89:0/3573820730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650f879cd70 con 0x5650f8723780 2024-09-15T23:33:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f947effd640 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 0x5650f8727b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9487fff640 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f87280b0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f9480009940 tx=0x7f9480004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b246adf7b376b419 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9466ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9466ffd640 1 -- 172.21.15.89:0/3573820730 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480012310 con 0x5650f8723780 2024-09-15T23:33:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f94900a6740 1 -- 172.21.15.89:0/3573820730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f947004a070 con 0x5650f8723780 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9466ffd640 1 -- 172.21.15.89:0/3573820730 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9480003f00 con 0x5650f8723780 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9466ffd640 1 -- 172.21.15.89:0/3573820730 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9480004100 con 0x5650f8723780 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f94900a6740 10 client.18825 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f94900a6740 1 -- 172.21.15.89:0/3573820730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5650f879e050 con 0x5650f8723780 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.632+0000 7f9466ffd640 1 -- 172.21.15.89:0/3573820730 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947004a070 con 0x5650f8723780 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f9466ffd640 1 -- 172.21.15.89:0/3573820730 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f94800107d0 con 0x5650f8723780 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f94900a6740 10 client.18825 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f94900a6740 10 client.18825 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f94900a6740 20 client.18825 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f94900a6740 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5650f879e690 0x5650f87a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f94900a6740 4 client.18825 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:33:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f94900a6740 1 -- 172.21.15.89:0/3573820730 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5650f8681d70 con 0x5650f879e690 2024-09-15T23:33:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f947effd640 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5650f879e690 0x5650f87a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f947effd640 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5650f879e690 0x5650f87a0ab0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fcfff1531a7d524a server_cookie=f54b115fd19af37f in_seq=0 out_seq=0 2024-09-15T23:33:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f9466ffd640 10 client.18825 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:33:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f9466ffd640 1 -- 172.21.15.89:0/3573820730 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5650f8681d70 con 0x5650f879e690 2024-09-15T23:33:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f9466ffd640 10 client.18825 handle_command_reply: tid=0 2024-09-15T23:33:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.633+0000 7f94900a6740 1 client.18825 shutdown 2024-09-15T23:33:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 1 -- 172.21.15.89:0/3573820730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 msgr2=0x5650f87280b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f87280b0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f9480009940 tx=0x7f9480004470 comp rx=0 tx=0).stop 2024-09-15T23:33:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 1 -- 172.21.15.89:0/3573820730 shutdown_connections 2024-09-15T23:33:54.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650f8723780 0x5650f87280b0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650f8715b40 0x5650f8727b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x5650f879e690 0x5650f87a0ab0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 1 --2- 172.21.15.89:0/3573820730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650f87145d0 0x5650f8727630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 1 -- 172.21.15.89:0/3573820730 >> 172.21.15.89:0/3573820730 conn(0x5650f86dd310 msgr2=0x5650f8721c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:54.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f9466ffd640 0 client.18825 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:33:54.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 1 -- 172.21.15.89:0/3573820730 shutdown_connections 2024-09-15T23:33:54.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 1 -- 172.21.15.89:0/3573820730 wait complete. 2024-09-15T23:33:54.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.634+0000 7f94900a6740 20 client.18825 trim_cache size 0 max 16384 2024-09-15T23:33:54.649 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:33:54.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 -- 172.21.15.89:0/843255944 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94740a3240 msgr2=0x7f94740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:54.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94740a3240 0x7f94740a5700 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f947809b8c0 tx=0x7f947806a900 comp rx=0 tx=0).stop 2024-09-15T23:33:54.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 -- 172.21.15.89:0/843255944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9488102d30 msgr2=0x7f94881985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:54.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9488102d30 0x7f94881985d0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f947800d950 tx=0x7f947800de10 comp rx=0 tx=0).stop 2024-09-15T23:33:54.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 -- 172.21.15.89:0/843255944 shutdown_connections 2024-09-15T23:33:54.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94740a3240 0x7f94740a5700 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94881082a0 0x7f948819cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94881036e0 0x7f9488198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 --2- 172.21.15.89:0/843255944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9488102d30 0x7f94881985d0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:54.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 -- 172.21.15.89:0/843255944 >> 172.21.15.89:0/843255944 conn(0x7f94880fc7d0 msgr2=0x7f9488105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:54.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 -- 172.21.15.89:0/843255944 shutdown_connections 2024-09-15T23:33:54.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:54.635+0000 7f948dccd640 1 -- 172.21.15.89:0/843255944 wait complete. 2024-09-15T23:33:54.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:54 smithi120 ceph-mon[35250]: pgmap v646: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-09-15T23:33:54.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:54 smithi174 ceph-mon[35586]: pgmap v646: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-09-15T23:33:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:55.375 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.888+0000 7fb4d095a640 1 -- 172.21.15.89:0/2565444269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc103760 msgr2=0x7fb4cc107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.888+0000 7fb4d095a640 1 --2- 172.21.15.89:0/2565444269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc103760 0x7fb4cc107ba0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009930 tx=0x7fb4b402a420 comp rx=0 tx=0).stop 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.889+0000 7fb4d095a640 1 -- 172.21.15.89:0/2565444269 shutdown_connections 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.889+0000 7fb4d095a640 1 --2- 172.21.15.89:0/2565444269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc1082e0 0x7fb4cc10ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.889+0000 7fb4d095a640 1 --2- 172.21.15.89:0/2565444269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc103760 0x7fb4cc107ba0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.889+0000 7fb4d095a640 1 --2- 172.21.15.89:0/2565444269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4cc102db0 0x7fb4cc103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.889+0000 7fb4d095a640 1 -- 172.21.15.89:0/2565444269 >> 172.21.15.89:0/2565444269 conn(0x7fb4cc0fc7d0 msgr2=0x7fb4cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.889+0000 7fb4d095a640 1 -- 172.21.15.89:0/2565444269 shutdown_connections 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.889+0000 7fb4d095a640 1 -- 172.21.15.89:0/2565444269 wait complete. 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.889+0000 7fb4d095a640 1 Processor -- start 2024-09-15T23:33:55.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.889+0000 7fb4d095a640 1 -- start start 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4d095a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc102db0 0x7fb4cc1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4d095a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc103760 0x7fb4cc198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4d095a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4cc1082e0 0x7fb4cc19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4d095a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc112050 con 0x7fb4cc102db0 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4d095a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc111ed0 con 0x7fb4cc103760 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4d095a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4cc1121d0 con 0x7fb4cc1082e0 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4ca59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc102db0 0x7fb4cc1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4ca59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc102db0 0x7fb4cc1985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37156/0 (socket says 172.21.15.89:37156) 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4ca59c640 1 -- 172.21.15.89:0/4021532470 learned_addr learned my addr 172.21.15.89:0/4021532470 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4cad9d640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4cc1082e0 0x7fb4cc19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4ca59c640 1 -- 172.21.15.89:0/4021532470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4cc1082e0 msgr2=0x7fb4cc19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4ca59c640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4cc1082e0 0x7fb4cc19ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:55.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4ca59c640 1 -- 172.21.15.89:0/4021532470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc103760 msgr2=0x7fb4cc198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4c9d9b640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc103760 0x7fb4cc198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4ca59c640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc103760 0x7fb4cc198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4ca59c640 1 -- 172.21.15.89:0/4021532470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4cc19d560 con 0x7fb4cc102db0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4ca59c640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc102db0 0x7fb4cc1985b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00cce0 tx=0x7fb4bc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4c37fe640 1 -- 172.21.15.89:0/4021532470 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc004b20 con 0x7fb4cc102db0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.890+0000 7fb4c37fe640 1 -- 172.21.15.89:0/4021532470 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4bc004cc0 con 0x7fb4cc102db0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.891+0000 7fb4d095a640 1 -- 172.21.15.89:0/4021532470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4cc19d850 con 0x7fb4cc102db0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.891+0000 7fb4c37fe640 1 -- 172.21.15.89:0/4021532470 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc00f7d0 con 0x7fb4cc102db0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.891+0000 7fb4d095a640 1 -- 172.21.15.89:0/4021532470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4cc1a5130 con 0x7fb4cc102db0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.891+0000 7fb4d095a640 1 -- 172.21.15.89:0/4021532470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4cc103190 con 0x7fb4cc102db0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.893+0000 7fb4c37fe640 1 -- 172.21.15.89:0/4021532470 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb4bc00f970 con 0x7fb4cc102db0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.893+0000 7fb4c37fe640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 0x7fb4b80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.893+0000 7fb4c9d9b640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 0x7fb4b80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:55.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.894+0000 7fb4c37fe640 1 -- 172.21.15.89:0/4021532470 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb4bc0cfe80 con 0x7fb4cc102db0 2024-09-15T23:33:55.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.894+0000 7fb4c9d9b640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 0x7fb4b80a5700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009230 tx=0x7fb4b4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:55.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:55.897+0000 7fb4c37fe640 1 -- 172.21.15.89:0/4021532470 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb4bc014030 con 0x7fb4cc102db0 2024-09-15T23:33:55.989 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:56.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.110+0000 7fb4d095a640 1 -- 172.21.15.89:0/4021532470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4cc199910 con 0x7fb4cc102db0 2024-09-15T23:33:56.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.110+0000 7fb4c37fe640 1 -- 172.21.15.89:0/4021532470 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fb4bc09c7b0 con 0x7fb4cc102db0 2024-09-15T23:33:56.112 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:33:56.112 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:33:56.112 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:33:56.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.112+0000 7fb4d095a640 1 -- 172.21.15.89:0/4021532470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 msgr2=0x7fb4b80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:56.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.112+0000 7fb4d095a640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 0x7fb4b80a5700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009230 tx=0x7fb4b4000bc0 comp rx=0 tx=0).stop 2024-09-15T23:33:56.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.112+0000 7fb4d095a640 1 -- 172.21.15.89:0/4021532470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc102db0 msgr2=0x7fb4cc1985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:56.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.112+0000 7fb4d095a640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc102db0 0x7fb4cc1985b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00cce0 tx=0x7fb4bc00a620 comp rx=0 tx=0).stop 2024-09-15T23:33:56.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.112+0000 7fb4d095a640 1 -- 172.21.15.89:0/4021532470 shutdown_connections 2024-09-15T23:33:56.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.112+0000 7fb4d095a640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 0x7fb4b80a5700 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:56.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.113+0000 7fb4d095a640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4cc1082e0 0x7fb4cc19ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:56.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.113+0000 7fb4d095a640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4cc103760 0x7fb4cc198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:56.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.113+0000 7fb4d095a640 1 --2- 172.21.15.89:0/4021532470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc102db0 0x7fb4cc1985b0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:56.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.113+0000 7fb4d095a640 1 -- 172.21.15.89:0/4021532470 >> 172.21.15.89:0/4021532470 conn(0x7fb4cc0fc7d0 msgr2=0x7fb4cc105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:56.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.113+0000 7fb4d095a640 1 -- 172.21.15.89:0/4021532470 shutdown_connections 2024-09-15T23:33:56.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:56.113+0000 7fb4d095a640 1 -- 172.21.15.89:0/4021532470 wait complete. 2024-09-15T23:33:56.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:56 smithi089 ceph-mon[30933]: pgmap v647: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-15T23:33:56.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:56 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/4021532470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:56.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:56 smithi120 ceph-mon[35250]: pgmap v647: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-15T23:33:56.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:56 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/4021532470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:56.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:56 smithi174 ceph-mon[35586]: pgmap v647: 129 pgs: 129 active+clean; 761 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 170 B/s wr, 10 op/s 2024-09-15T23:33:56.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:56 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/4021532470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:33:56.949 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id ca53bdfb --release --await' 2024-09-15T23:33:56.949 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id ca53bdfb --release --await 2024-09-15T23:33:57.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.149+0000 7f90b6d80640 1 -- 172.21.15.89:0/3138500198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b0102d90 msgr2=0x7f90b0103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:57.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.149+0000 7f90b6d80640 1 --2- 172.21.15.89:0/3138500198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b0102d90 0x7f90b0103170 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f90a4009930 tx=0x7f90a402a420 comp rx=0 tx=0).stop 2024-09-15T23:33:57.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.149+0000 7f90b6d80640 1 -- 172.21.15.89:0/3138500198 shutdown_connections 2024-09-15T23:33:57.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.149+0000 7f90b6d80640 1 --2- 172.21.15.89:0/3138500198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b0108290 0x7f90b010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.149+0000 7f90b6d80640 1 --2- 172.21.15.89:0/3138500198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90b0103740 0x7f90b0107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.149+0000 7f90b6d80640 1 --2- 172.21.15.89:0/3138500198 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b0102d90 0x7f90b0103170 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.149+0000 7f90b6d80640 1 -- 172.21.15.89:0/3138500198 >> 172.21.15.89:0/3138500198 conn(0x7f90b00fc7d0 msgr2=0x7f90b00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:57.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.149+0000 7f90b6d80640 1 -- 172.21.15.89:0/3138500198 shutdown_connections 2024-09-15T23:33:57.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.149+0000 7f90b6d80640 1 -- 172.21.15.89:0/3138500198 wait complete. 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b6d80640 1 Processor -- start 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b6d80640 1 -- start start 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b6d80640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b0102d90 0x7f90b01985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b6d80640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b0103740 0x7f90b0198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b6d80640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90b0108290 0x7f90b019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b6d80640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90b0112090 con 0x7f90b0102d90 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b6d80640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f90b0111f10 con 0x7f90b0103740 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b6d80640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f90b0112210 con 0x7f90b0108290 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b4b1c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b0102d90 0x7f90b01985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b4b1c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b0102d90 0x7f90b01985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37162/0 (socket says 172.21.15.89:37162) 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b4b1c640 1 -- 172.21.15.89:0/2598260842 learned_addr learned my addr 172.21.15.89:0/2598260842 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90affff640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b0103740 0x7f90b0198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.150+0000 7f90b531d640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90b0108290 0x7f90b019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90b4b1c640 1 -- 172.21.15.89:0/2598260842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90b0108290 msgr2=0x7f90b019ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90b0108290 0x7f90b019ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90b4b1c640 1 -- 172.21.15.89:0/2598260842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b0103740 msgr2=0x7f90b0198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b0103740 0x7f90b0198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90b4b1c640 1 -- 172.21.15.89:0/2598260842 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b019d550 con 0x7f90b0102d90 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90b531d640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90b0108290 0x7f90b019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90affff640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b0103740 0x7f90b0198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b0102d90 0x7f90b01985a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f90a4009230 tx=0x7f90a4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90adffb640 1 -- 172.21.15.89:0/2598260842 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a403f030 con 0x7f90b0102d90 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90b6d80640 1 -- 172.21.15.89:0/2598260842 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b019d7e0 con 0x7f90b0102d90 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90adffb640 1 -- 172.21.15.89:0/2598260842 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90a4003ad0 con 0x7f90b0102d90 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90adffb640 1 -- 172.21.15.89:0/2598260842 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a4035070 con 0x7f90b0102d90 2024-09-15T23:33:57.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.151+0000 7f90b6d80640 1 -- 172.21.15.89:0/2598260842 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b01a5120 con 0x7f90b0102d90 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.153+0000 7f90adffb640 1 -- 172.21.15.89:0/2598260842 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f90a4042030 con 0x7f90b0102d90 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b9159740 1 Processor -- start 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b9159740 1 -- start start 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b9159740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df128f4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b9159740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df128f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b9159740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12901760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b9159740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55df1285a2f0 con 0x55df128ff370 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b9159740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55df12801cf0 con 0x55df128f45d0 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b9159740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55df128562b0 con 0x55df128f5b40 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90adffb640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90a00a3240 0x7f90a00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90affff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12901760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b531d640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90a00a3240 0x7f90a00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90affff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12901760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37174/0 (socket says 172.21.15.89:37174) 2024-09-15T23:33:57.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90affff640 1 -- 172.21.15.89:0/2374018138 learned_addr learned my addr 172.21.15.89:0/2374018138 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90adffb640 1 -- 172.21.15.89:0/2598260842 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f90a40f02a0 con 0x7f90b0102d90 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b531d640 1 --2- 172.21.15.89:0/2374018138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df128f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.154+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/2374018138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df128f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f90affff640 1 -- 172.21.15.89:0/2374018138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 msgr2=0x55df128f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f90affff640 1 --2- 172.21.15.89:0/2374018138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df128f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f90affff640 1 -- 172.21.15.89:0/2374018138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 msgr2=0x55df128f4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f90affff640 1 --2- 172.21.15.89:0/2374018138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df128f4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f90affff640 1 -- 172.21.15.89:0/2374018138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df127e63a0 con 0x55df128ff370 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/2374018138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df128f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f90b531d640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90a00a3240 0x7f90a00a5700 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f90a8005c40 tx=0x7f90a8009d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f90affff640 1 --2- 172.21.15.89:0/2374018138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12901760 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f909c00bce0 tx=0x7f909c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434e7fd1201a310d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f909affd640 1 -- 172.21.15.89:0/2374018138 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c015070 con 0x55df128ff370 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f909affd640 1 -- 172.21.15.89:0/2374018138 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f909c01a940 con 0x55df128ff370 2024-09-15T23:33:57.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.155+0000 7f909affd640 1 -- 172.21.15.89:0/2374018138 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c011970 con 0x55df128ff370 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 -- 172.21.15.89:0/2374018138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 msgr2=0x55df12901760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 --2- 172.21.15.89:0/2374018138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12901760 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f909c00bce0 tx=0x7f909c00f730 comp rx=0 tx=0).stop 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 -- 172.21.15.89:0/2374018138 shutdown_connections 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 --2- 172.21.15.89:0/2374018138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12901760 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 --2- 172.21.15.89:0/2374018138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df128f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 --2- 172.21.15.89:0/2374018138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df128f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 -- 172.21.15.89:0/2374018138 >> 172.21.15.89:0/2374018138 conn(0x55df128bd310 msgr2=0x55df128f0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 -- 172.21.15.89:0/2374018138 shutdown_connections 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 -- 172.21.15.89:0/2374018138 wait complete. 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 -- start start 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.156+0000 7f90b9159740 1 Processor -- start 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b9159740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f45d0 0x55df128fed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b9159740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df128fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b9159740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128ff370 0x55df128fc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b9159740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55df12902070 con 0x55df128f45d0 2024-09-15T23:33:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b9159740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55df12815060 con 0x55df128ff370 2024-09-15T23:33:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b9159740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55df129021f0 con 0x55df128f5b40 2024-09-15T23:33:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b9159740 10 client.0 _ll_register_callbacks cb 0x55df125688c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b4b1c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f45d0 0x55df128fed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b4b1c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f45d0 0x55df128fed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37182/0 (socket says 172.21.15.89:37182) 2024-09-15T23:33:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b4b1c640 1 -- 172.21.15.89:0/1082953750 learned_addr learned my addr 172.21.15.89:0/1082953750 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:57.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90affff640 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df128fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b531d640 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128ff370 0x55df128fc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b4b1c640 1 -- 172.21.15.89:0/1082953750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 msgr2=0x55df128fbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df128fbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b4b1c640 1 -- 172.21.15.89:0/1082953750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128ff370 msgr2=0x55df128fc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128ff370 0x55df128fc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b4b1c640 1 -- 172.21.15.89:0/1082953750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df125cb3c0 con 0x55df128f45d0 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b531d640 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128ff370 0x55df128fc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90affff640 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df128fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f45d0 0x55df128fed30 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f90a40b8930 tx=0x7f90a4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce53a047ef3490f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f908ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f908ffff640 1 -- 172.21.15.89:0/1082953750 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a40d5060 con 0x55df128f45d0 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f908ffff640 1 -- 172.21.15.89:0/1082953750 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f90a4003cd0 con 0x55df128f45d0 2024-09-15T23:33:57.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.157+0000 7f90b9159740 1 -- 172.21.15.89:0/1082953750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f909c01a940 con 0x55df128f45d0 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f908ffff640 1 -- 172.21.15.89:0/1082953750 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f90a4049070 con 0x55df128f45d0 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f90b9159740 10 client.18843 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f90b9159740 1 -- 172.21.15.89:0/1082953750 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55df1290da40 con 0x55df128f45d0 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f908ffff640 1 -- 172.21.15.89:0/1082953750 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f909c01a940 con 0x55df128f45d0 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f908ffff640 1 -- 172.21.15.89:0/1082953750 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f90a40b8b40 con 0x55df128f45d0 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f90b9159740 10 client.18843 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f90b9159740 10 client.18843 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f90b9159740 20 client.18843 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f90b9159740 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55df1290eef0 0x55df12911340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f90b9159740 4 client.18843 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f90b9159740 1 -- 172.21.15.89:0/1082953750 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df12867860 con 0x55df1290eef0 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.158+0000 7f90affff640 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55df1290eef0 0x55df12911340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.159+0000 7f90affff640 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55df1290eef0 0x55df12911340 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=327b969855ca48aa server_cookie=6352ec6324f0abbe in_seq=0 out_seq=0 2024-09-15T23:33:57.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.159+0000 7f908ffff640 10 client.18843 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.159+0000 7f908ffff640 1 -- 172.21.15.89:0/1082953750 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55df12867860 con 0x55df1290eef0 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.159+0000 7f908ffff640 10 client.18843 handle_command_reply: tid=0 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.159+0000 7f90b9159740 1 client.18843 shutdown 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.159+0000 7f90b9159740 1 -- 172.21.15.89:0/1082953750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f45d0 msgr2=0x55df128fed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.159+0000 7f90b9159740 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f45d0 0x55df128fed30 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f90a40b8930 tx=0x7f90a4009260 comp rx=0 tx=0).stop 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.160+0000 7f90b9159740 1 -- 172.21.15.89:0/1082953750 shutdown_connections 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.160+0000 7f90b9159740 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55df1290eef0 0x55df12911340 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.160+0000 7f90b9159740 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128ff370 0x55df128fc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.160+0000 7f90b9159740 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df128fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.160+0000 7f908ffff640 0 client.18843 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.160+0000 7f90b9159740 1 --2- 172.21.15.89:0/1082953750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f45d0 0x55df128fed30 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.160+0000 7f90b9159740 1 -- 172.21.15.89:0/1082953750 >> 172.21.15.89:0/1082953750 conn(0x55df128bd310 msgr2=0x55df128f6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:57.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.160+0000 7f90b9159740 1 -- 172.21.15.89:0/1082953750 shutdown_connections 2024-09-15T23:33:57.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.160+0000 7f90b9159740 1 -- 172.21.15.89:0/1082953750 wait complete. 2024-09-15T23:33:57.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.160+0000 7f90b9159740 20 client.18843 trim_cache size 0 max 16384 2024-09-15T23:33:57.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90b9159740 1 Processor -- start 2024-09-15T23:33:57.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90b9159740 1 -- start start 2024-09-15T23:33:57.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90b9159740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df1290b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90b9159740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f5b40 0x55df12900760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90b9159740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128ff370 0x55df12900ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90b9159740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90b01aedd0 con 0x55df128f5b40 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90b9159740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f90b01b0910 con 0x55df128f45d0 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90b9159740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f90b01af4a0 con 0x55df128ff370 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90affff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f5b40 0x55df12900760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90affff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f5b40 0x55df12900760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37194/0 (socket says 172.21.15.89:37194) 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90affff640 1 -- 172.21.15.89:0/3511170660 learned_addr learned my addr 172.21.15.89:0/3511170660 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/3511170660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df1290b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.173+0000 7f90b531d640 1 --2- 172.21.15.89:0/3511170660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128ff370 0x55df12900ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.174+0000 7f90affff640 1 -- 172.21.15.89:0/3511170660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128ff370 msgr2=0x55df12900ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.174+0000 7f90affff640 1 --2- 172.21.15.89:0/3511170660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128ff370 0x55df12900ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.174+0000 7f90affff640 1 -- 172.21.15.89:0/3511170660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 msgr2=0x55df1290b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.174+0000 7f90affff640 1 --2- 172.21.15.89:0/3511170660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df1290b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.174+0000 7f90affff640 1 -- 172.21.15.89:0/3511170660 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df128fc940 con 0x55df128f5b40 2024-09-15T23:33:57.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.174+0000 7f90affff640 1 --2- 172.21.15.89:0/3511170660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f5b40 0x55df12900760 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x55df128f5070 tx=0x7f909c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb731bebb692f5c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:57.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.174+0000 7f909a7fc640 1 -- 172.21.15.89:0/3511170660 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c015040 con 0x55df128f5b40 2024-09-15T23:33:57.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.174+0000 7f909a7fc640 1 -- 172.21.15.89:0/3511170660 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f909c004680 con 0x55df128f5b40 2024-09-15T23:33:57.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.174+0000 7f909a7fc640 1 -- 172.21.15.89:0/3511170660 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c0118c0 con 0x55df128f5b40 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 -- 172.21.15.89:0/3511170660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f5b40 msgr2=0x55df12900760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 --2- 172.21.15.89:0/3511170660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f5b40 0x55df12900760 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x55df128f5070 tx=0x7f909c00b4a0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 -- 172.21.15.89:0/3511170660 shutdown_connections 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 --2- 172.21.15.89:0/3511170660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128ff370 0x55df12900ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 --2- 172.21.15.89:0/3511170660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128f5b40 0x55df12900760 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 --2- 172.21.15.89:0/3511170660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df1290b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 -- 172.21.15.89:0/3511170660 >> 172.21.15.89:0/3511170660 conn(0x55df128bd310 msgr2=0x55df128f0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 -- 172.21.15.89:0/3511170660 shutdown_connections 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 -- 172.21.15.89:0/3511170660 wait complete. 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 -- start start 2024-09-15T23:33:57.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.175+0000 7f90b9159740 1 Processor -- start 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b9159740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df12907490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b9159740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df129079d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b9159740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12902440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b9159740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90b01b0240 con 0x55df128ff370 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b9159740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f90b01afb70 con 0x55df128f45d0 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b9159740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f90b01acdb0 con 0x55df128f5b40 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b9159740 10 client.0 _ll_register_callbacks cb 0x7f90a4005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b531d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12902440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b531d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12902440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37200/0 (socket says 172.21.15.89:37200) 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b531d640 1 -- 172.21.15.89:0/185944386 learned_addr learned my addr 172.21.15.89:0/185944386 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90affff640 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df129079d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.176+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df12907490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f90b531d640 1 -- 172.21.15.89:0/185944386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 msgr2=0x55df129079d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f90b531d640 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df129079d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f90b531d640 1 -- 172.21.15.89:0/185944386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 msgr2=0x55df12907490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f90b531d640 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df12907490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f90b531d640 1 -- 172.21.15.89:0/185944386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df12908090 con 0x55df128ff370 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f90b531d640 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12902440 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f90a8010d20 tx=0x7f90a8005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6611c5f6999da5f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f908ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f908ffff640 1 -- 172.21.15.89:0/185944386 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a802f060 con 0x55df128ff370 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f908ffff640 1 -- 172.21.15.89:0/185944386 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f90a801b040 con 0x55df128ff370 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f90b9159740 1 -- 172.21.15.89:0/185944386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f909c004680 con 0x55df128ff370 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f908ffff640 1 -- 172.21.15.89:0/185944386 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f90a8036050 con 0x55df128ff370 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f90b9159740 10 client.18849 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f90b9159740 1 -- 172.21.15.89:0/185944386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55df1297e5b0 con 0x55df128ff370 2024-09-15T23:33:57.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.177+0000 7f908ffff640 1 -- 172.21.15.89:0/185944386 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f909c004680 con 0x55df128ff370 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.178+0000 7f908ffff640 1 -- 172.21.15.89:0/185944386 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f90a8029070 con 0x55df128ff370 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.178+0000 7f90b9159740 10 client.18849 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.178+0000 7f90b9159740 10 client.18849 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.178+0000 7f90b9159740 20 client.18849 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.178+0000 7f90b9159740 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55df1297ed30 0x55df12981150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.178+0000 7f90b9159740 4 client.18849 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.178+0000 7f90b9159740 1 -- 172.21.15.89:0/185944386 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55df12861d70 con 0x55df1297ed30 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.178+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55df1297ed30 0x55df12981150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.178+0000 7f90b4b1c640 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55df1297ed30 0x55df12981150 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb9355681d4df74e server_cookie=3fb9cc027f1b04e3 in_seq=0 out_seq=0 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.178+0000 7f908ffff640 10 client.18849 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f908ffff640 1 -- 172.21.15.89:0/185944386 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55df12861d70 con 0x55df1297ed30 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f908ffff640 10 client.18849 handle_command_reply: tid=0 2024-09-15T23:33:57.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 client.18849 shutdown 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 -- 172.21.15.89:0/185944386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 msgr2=0x55df12902440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12902440 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f90a8010d20 tx=0x7f90a8005bd0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 -- 172.21.15.89:0/185944386 shutdown_connections 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55df1297ed30 0x55df12981150 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55df128ff370 0x55df12902440 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55df128f5b40 0x55df129079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 --2- 172.21.15.89:0/185944386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55df128f45d0 0x55df12907490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 -- 172.21.15.89:0/185944386 >> 172.21.15.89:0/185944386 conn(0x55df128bd310 msgr2=0x55df128ef4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f908ffff640 0 client.18849 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 -- 172.21.15.89:0/185944386 shutdown_connections 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 1 -- 172.21.15.89:0/185944386 wait complete. 2024-09-15T23:33:57.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.179+0000 7f90b9159740 20 client.18849 trim_cache size 0 max 16384 2024-09-15T23:33:57.185 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:33:57.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 -- 172.21.15.89:0/2598260842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90a00a3240 msgr2=0x7f90a00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:57.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90a00a3240 0x7f90a00a5700 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f90a8005c40 tx=0x7f90a8009d50 comp rx=0 tx=0).stop 2024-09-15T23:33:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 -- 172.21.15.89:0/2598260842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b0102d90 msgr2=0x7f90b01985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b0102d90 0x7f90b01985a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f90a4009230 tx=0x7f90a4004270 comp rx=0 tx=0).stop 2024-09-15T23:33:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 -- 172.21.15.89:0/2598260842 shutdown_connections 2024-09-15T23:33:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90a00a3240 0x7f90a00a5700 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90b0108290 0x7f90b019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90b0103740 0x7f90b0198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 --2- 172.21.15.89:0/2598260842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90b0102d90 0x7f90b01985a0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 -- 172.21.15.89:0/2598260842 >> 172.21.15.89:0/2598260842 conn(0x7f90b00fc7d0 msgr2=0x7f90b0105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 -- 172.21.15.89:0/2598260842 shutdown_connections 2024-09-15T23:33:57.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:57.180+0000 7f90b6d80640 1 -- 172.21.15.89:0/2598260842 wait complete. 2024-09-15T23:33:57.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:33:57.899 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:58.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.385+0000 7f0cbe22f640 1 -- 172.21.15.89:0/688088792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb8106f10 msgr2=0x7f0cb810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:58.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.385+0000 7f0cbe22f640 1 --2- 172.21.15.89:0/688088792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb8106f10 0x7f0cb810ea90 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0009ca0 tx=0x7f0cb002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.385+0000 7f0cbe22f640 1 -- 172.21.15.89:0/688088792 shutdown_connections 2024-09-15T23:33:58.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.385+0000 7f0cbe22f640 1 --2- 172.21.15.89:0/688088792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb8106f10 0x7f0cb810ea90 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.385+0000 7f0cbe22f640 1 --2- 172.21.15.89:0/688088792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cb8102790 0x7f0cb81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.385+0000 7f0cbe22f640 1 --2- 172.21.15.89:0/688088792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb8101de0 0x7f0cb81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.385+0000 7f0cbe22f640 1 -- 172.21.15.89:0/688088792 >> 172.21.15.89:0/688088792 conn(0x7f0cb80fd8c0 msgr2=0x7f0cb80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:58.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.385+0000 7f0cbe22f640 1 -- 172.21.15.89:0/688088792 shutdown_connections 2024-09-15T23:33:58.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.385+0000 7f0cbe22f640 1 -- 172.21.15.89:0/688088792 wait complete. 2024-09-15T23:33:58.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cbe22f640 1 Processor -- start 2024-09-15T23:33:58.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cbe22f640 1 -- start start 2024-09-15T23:33:58.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cbe22f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb8101de0 0x7f0cb8198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cbe22f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cb8102790 0x7f0cb81988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cbe22f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb8106f10 0x7f0cb819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cbe22f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8111f70 con 0x7f0cb8106f10 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cbe22f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb8111df0 con 0x7f0cb8101de0 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cbe22f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81120f0 con 0x7f0cb8102790 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cb7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb8106f10 0x7f0cb819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cb7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb8106f10 0x7f0cb819cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37212/0 (socket says 172.21.15.89:37212) 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.386+0000 7f0cb7fff640 1 -- 172.21.15.89:0/28371671 learned_addr learned my addr 172.21.15.89:0/28371671 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb77fe640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb8101de0 0x7f0cb8198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cb8102790 0x7f0cb81988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb7fff640 1 -- 172.21.15.89:0/28371671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cb8102790 msgr2=0x7f0cb81988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb7fff640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cb8102790 0x7f0cb81988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb7fff640 1 -- 172.21.15.89:0/28371671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb8101de0 msgr2=0x7f0cb8198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb7fff640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb8101de0 0x7f0cb8198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb7fff640 1 -- 172.21.15.89:0/28371671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb819d310 con 0x7f0cb8106f10 2024-09-15T23:33:58.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb77fe640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb8101de0 0x7f0cb8198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:58.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb7fff640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb8106f10 0x7f0cb819cc30 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0031a60 tx=0x7f0cb0031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:58.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb4ff9640 1 -- 172.21.15.89:0/28371671 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0003c30 con 0x7f0cb8106f10 2024-09-15T23:33:58.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cbe22f640 1 -- 172.21.15.89:0/28371671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb819d5a0 con 0x7f0cb8106f10 2024-09-15T23:33:58.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb4ff9640 1 -- 172.21.15.89:0/28371671 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cb0003dd0 con 0x7f0cb8106f10 2024-09-15T23:33:58.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cb4ff9640 1 -- 172.21.15.89:0/28371671 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00047e0 con 0x7f0cb8106f10 2024-09-15T23:33:58.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.387+0000 7f0cbe22f640 1 -- 172.21.15.89:0/28371671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb81a4ee0 con 0x7f0cb8106f10 2024-09-15T23:33:58.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.389+0000 7f0cb4ff9640 1 -- 172.21.15.89:0/28371671 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0cb002f790 con 0x7f0cb8106f10 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cc0608740 1 Processor -- start 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cc0608740 1 -- start start 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cb4ff9640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ca00a3240 0x7f0ca00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae84c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae856d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae857240 0x55fcae859630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cc0608740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fcae7b1620 con 0x55fcae84c980 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cc0608740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fcae758e50 con 0x55fcae857240 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cc0608740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fcae7ad5e0 con 0x55fcae84c060 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ca00a3240 0x7f0ca00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cb4ff9640 1 -- 172.21.15.89:0/28371671 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0cb00f10e0 con 0x7f0cb8106f10 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cb6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae856d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cb6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae856d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37222/0 (socket says 172.21.15.89:37222) 2024-09-15T23:33:58.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/1924475721 learned_addr learned my addr 172.21.15.89:0/1924475721 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:58.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cb7fff640 1 --2- 172.21.15.89:0/1924475721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae857240 0x55fcae859630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cb77fe640 1 --2- 172.21.15.89:0/1924475721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae84c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/1924475721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 msgr2=0x55fcae84c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:58.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.390+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/1924475721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae84c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.391+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/1924475721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae857240 msgr2=0x55fcae859630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:58.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.391+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/1924475721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae857240 0x55fcae859630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.391+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/1924475721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcae749d40 con 0x55fcae84c980 2024-09-15T23:33:58.405 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.391+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ca00a3240 0x7f0ca00a5700 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0cac005d50 tx=0x7f0cac00aa80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.391+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/1924475721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae856d00 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00ad30 tx=0x7f0cac01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10d1abbc27b65ae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.391+0000 7f0c9dffb640 1 -- 172.21.15.89:0/1924475721 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac023070 con 0x55fcae84c980 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.391+0000 7f0c9dffb640 1 -- 172.21.15.89:0/1924475721 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cac00fe50 con 0x55fcae84c980 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.391+0000 7f0c9dffb640 1 -- 172.21.15.89:0/1924475721 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac003850 con 0x55fcae84c980 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 -- 172.21.15.89:0/1924475721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 msgr2=0x55fcae856d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 --2- 172.21.15.89:0/1924475721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae856d00 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00ad30 tx=0x7f0cac01b000 comp rx=0 tx=0).stop 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 -- 172.21.15.89:0/1924475721 shutdown_connections 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 --2- 172.21.15.89:0/1924475721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae857240 0x55fcae859630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 --2- 172.21.15.89:0/1924475721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae856d00 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 --2- 172.21.15.89:0/1924475721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae84c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 -- 172.21.15.89:0/1924475721 >> 172.21.15.89:0/1924475721 conn(0x55fcae8485f0 msgr2=0x55fcae8489d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 -- 172.21.15.89:0/1924475721 shutdown_connections 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 -- 172.21.15.89:0/1924475721 wait complete. 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 -- start start 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 Processor -- start 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae854110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae854650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.392+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae857240 0x55fcae854b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cc0608740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fcae85e220 con 0x55fcae84c980 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cc0608740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fcae739f10 con 0x55fcae857240 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cc0608740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fcae85e3a0 con 0x55fcae84c060 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cc0608740 10 client.0 _ll_register_callbacks cb 0x55fcae75ddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae854650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.407 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae854650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37228/0 (socket says 172.21.15.89:37228) 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/3054314404 learned_addr learned my addr 172.21.15.89:0/3054314404 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb77fe640 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae854110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb7fff640 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae857240 0x55fcae854b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/3054314404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 msgr2=0x55fcae854110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae854110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/3054314404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae857240 msgr2=0x55fcae854b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae857240 0x55fcae854b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/3054314404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcae4c0580 con 0x55fcae84c980 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae854650 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0cac000c00 tx=0x7f0cac00ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d22be8d93957e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0c96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:58.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0c96ffd640 1 -- 172.21.15.89:0/3054314404 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac02d070 con 0x55fcae84c980 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cc0608740 1 -- 172.21.15.89:0/3054314404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cac00fe50 con 0x55fcae84c980 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0c96ffd640 1 -- 172.21.15.89:0/3054314404 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0cac004220 con 0x55fcae84c980 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0c96ffd640 1 -- 172.21.15.89:0/3054314404 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0cac038050 con 0x55fcae84c980 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cc0608740 10 client.18861 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.393+0000 7f0cc0608740 1 -- 172.21.15.89:0/3054314404 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcae58a960 con 0x55fcae84c980 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0c96ffd640 1 -- 172.21.15.89:0/3054314404 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cac00fe50 con 0x55fcae84c980 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0c96ffd640 1 -- 172.21.15.89:0/3054314404 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0cac01e050 con 0x55fcae84c980 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0cc0608740 10 client.18861 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0cc0608740 10 client.18861 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0cc0608740 20 client.18861 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:58.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0cc0608740 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fcae86b080 0x55fcae86d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0cc0608740 4 client.18861 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:58.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0cc0608740 1 -- 172.21.15.89:0/3054314404 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcae7beb90 con 0x55fcae86b080 2024-09-15T23:33:58.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0cb77fe640 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fcae86b080 0x55fcae86d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0cb77fe640 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fcae86b080 0x55fcae86d470 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c6c3b161986bbd server_cookie=79dcddee3f6af7b0 in_seq=0 out_seq=0 2024-09-15T23:33:58.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.394+0000 7f0c96ffd640 10 client.18861 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:58.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0c96ffd640 1 -- 172.21.15.89:0/3054314404 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fcae7beb90 con 0x55fcae86b080 2024-09-15T23:33:58.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0c96ffd640 10 client.18861 handle_command_reply: tid=0 2024-09-15T23:33:58.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0cc0608740 1 client.18861 shutdown 2024-09-15T23:33:58.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0cc0608740 1 -- 172.21.15.89:0/3054314404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 msgr2=0x55fcae854650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:58.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0cc0608740 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae854650 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0cac000c00 tx=0x7f0cac00ac90 comp rx=0 tx=0).stop 2024-09-15T23:33:58.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0cc0608740 1 -- 172.21.15.89:0/3054314404 shutdown_connections 2024-09-15T23:33:58.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0cc0608740 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fcae86b080 0x55fcae86d470 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0cc0608740 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae857240 0x55fcae854b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0cc0608740 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae854650 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0cc0608740 1 --2- 172.21.15.89:0/3054314404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae854110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0c96ffd640 0 client.18861 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:58.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0cc0608740 1 -- 172.21.15.89:0/3054314404 >> 172.21.15.89:0/3054314404 conn(0x55fcae8485f0 msgr2=0x55fcae847750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:58.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.395+0000 7f0cc0608740 1 -- 172.21.15.89:0/3054314404 shutdown_connections 2024-09-15T23:33:58.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.396+0000 7f0cc0608740 1 -- 172.21.15.89:0/3054314404 wait complete. 2024-09-15T23:33:58.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.396+0000 7f0cc0608740 20 client.18861 trim_cache size 0 max 16384 2024-09-15T23:33:58.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cc0608740 1 Processor -- start 2024-09-15T23:33:58.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cc0608740 1 -- start start 2024-09-15T23:33:58.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae872c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae84c980 0x55fcae873140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae857240 0x55fcae851370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cc0608740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81acb70 con 0x55fcae857240 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cc0608740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81ae6b0 con 0x55fcae84c980 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cc0608740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81ad240 con 0x55fcae84c060 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cb7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae857240 0x55fcae851370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cb7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae857240 0x55fcae851370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37234/0 (socket says 172.21.15.89:37234) 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cb7fff640 1 -- 172.21.15.89:0/16519174 learned_addr learned my addr 172.21.15.89:0/16519174 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/16519174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae84c980 0x55fcae873140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cb77fe640 1 --2- 172.21.15.89:0/16519174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae872c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cb7fff640 1 -- 172.21.15.89:0/16519174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 msgr2=0x55fcae872c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:58.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.413+0000 7f0cb7fff640 1 --2- 172.21.15.89:0/16519174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae872c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.414+0000 7f0cb7fff640 1 -- 172.21.15.89:0/16519174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae84c980 msgr2=0x55fcae873140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:58.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.414+0000 7f0cb7fff640 1 --2- 172.21.15.89:0/16519174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae84c980 0x55fcae873140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.414+0000 7f0cb7fff640 1 -- 172.21.15.89:0/16519174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcae84cfa0 con 0x55fcae857240 2024-09-15T23:33:58.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.414+0000 7f0cb7fff640 1 --2- 172.21.15.89:0/16519174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae857240 0x55fcae851370 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00bd130 tx=0x7f0cb003d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b38126cefcbaa8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:58.418 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:33:58 smithi089 ceph-mon[30933]: pgmap v648: 129 pgs: 129 active+clean; 765 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 853 B/s wr, 14 op/s 2024-09-15T23:33:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.414+0000 7f0c9d7fa640 1 -- 172.21.15.89:0/16519174 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00b96d0 con 0x55fcae857240 2024-09-15T23:33:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.414+0000 7f0c9d7fa640 1 -- 172.21.15.89:0/16519174 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cb009d070 con 0x55fcae857240 2024-09-15T23:33:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.414+0000 7f0c9d7fa640 1 -- 172.21.15.89:0/16519174 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00c60a0 con 0x55fcae857240 2024-09-15T23:33:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.415+0000 7f0cc0608740 1 -- 172.21.15.89:0/16519174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae857240 msgr2=0x55fcae851370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.415+0000 7f0cc0608740 1 --2- 172.21.15.89:0/16519174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae857240 0x55fcae851370 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00bd130 tx=0x7f0cb003d040 comp rx=0 tx=0).stop 2024-09-15T23:33:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.415+0000 7f0cc0608740 1 -- 172.21.15.89:0/16519174 shutdown_connections 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.415+0000 7f0cc0608740 1 --2- 172.21.15.89:0/16519174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae857240 0x55fcae851370 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.415+0000 7f0cc0608740 1 --2- 172.21.15.89:0/16519174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae84c980 0x55fcae873140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.415+0000 7f0cc0608740 1 --2- 172.21.15.89:0/16519174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae84c060 0x55fcae872c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.415+0000 7f0cc0608740 1 -- 172.21.15.89:0/16519174 >> 172.21.15.89:0/16519174 conn(0x55fcae8485f0 msgr2=0x55fcae855970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.415+0000 7f0cc0608740 1 -- 172.21.15.89:0/16519174 shutdown_connections 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.415+0000 7f0cc0608740 1 -- 172.21.15.89:0/16519174 wait complete. 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.415+0000 7f0cc0608740 1 -- start start 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cc0608740 1 Processor -- start 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae84c060 0x55fcae85fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae8602c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cc0608740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae857240 0x55fcae8633f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cc0608740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81adfe0 con 0x55fcae84c980 2024-09-15T23:33:58.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cc0608740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81ad910 con 0x55fcae84c060 2024-09-15T23:33:58.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cc0608740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb81aab50 con 0x55fcae857240 2024-09-15T23:33:58.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cc0608740 10 client.0 _ll_register_callbacks cb 0x7f0ca8008390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:58.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cb6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae8602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cb7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae857240 0x55fcae8633f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cb6ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae8602c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37246/0 (socket says 172.21.15.89:37246) 2024-09-15T23:33:58.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/2751904035 learned_addr learned my addr 172.21.15.89:0/2751904035 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:58.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.416+0000 7f0cb77fe640 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae84c060 0x55fcae85fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/2751904035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae857240 msgr2=0x55fcae8633f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae857240 0x55fcae8633f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/2751904035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae84c060 msgr2=0x55fcae85fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae84c060 0x55fcae85fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0cb6ffd640 1 -- 172.21.15.89:0/2751904035 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcae863a60 con 0x55fcae84c980 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0cb7fff640 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae857240 0x55fcae8633f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0cb6ffd640 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae8602c0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f0cac015010 tx=0x7f0cac004c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a887da9e3f20e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0c96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0c96ffd640 1 -- 172.21.15.89:0/2751904035 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac02d040 con 0x55fcae84c980 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0c96ffd640 1 -- 172.21.15.89:0/2751904035 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0cac011040 con 0x55fcae84c980 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0cc0608740 1 -- 172.21.15.89:0/2751904035 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cb009d070 con 0x55fcae84c980 2024-09-15T23:33:58.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0c96ffd640 1 -- 172.21.15.89:0/2751904035 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0cac023040 con 0x55fcae84c980 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0cc0608740 10 client.18867 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0cc0608740 1 -- 172.21.15.89:0/2751904035 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcae8d66d0 con 0x55fcae84c980 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.417+0000 7f0c96ffd640 1 -- 172.21.15.89:0/2751904035 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cb009d070 con 0x55fcae84c980 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.418+0000 7f0c96ffd640 1 -- 172.21.15.89:0/2751904035 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0cac010430 con 0x55fcae84c980 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.418+0000 7f0cc0608740 10 client.18867 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.418+0000 7f0cc0608740 10 client.18867 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.418+0000 7f0cc0608740 20 client.18867 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.418+0000 7f0cc0608740 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fcae8d6de0 0x55fcae8d91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.418+0000 7f0cc0608740 4 client.18867 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ca53bdfb", "release": true, "await": true}] 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.418+0000 7f0cc0608740 1 -- 172.21.15.89:0/2751904035 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ca53bdfb", "release": true, "await": true}) -- 0x55fcae7b90a0 con 0x55fcae8d6de0 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.418+0000 7f0cb77fe640 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fcae8d6de0 0x55fcae8d91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.418+0000 7f0c96ffd640 10 client.18867 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:58.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.418+0000 7f0cb77fe640 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fcae8d6de0 0x55fcae8d91d0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=875ad3a71a91f9a8 server_cookie=d4b1920c591407c6 in_seq=0 out_seq=0 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0c96ffd640 1 -- 172.21.15.89:0/2751904035 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55fcae7b90a0 con 0x55fcae8d6de0 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0c96ffd640 10 client.18867 handle_command_reply: tid=0 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 client.18867 shutdown 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 -- 172.21.15.89:0/2751904035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 msgr2=0x55fcae8602c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae8602c0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f0cac015010 tx=0x7f0cac004c10 comp rx=0 tx=0).stop 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 -- 172.21.15.89:0/2751904035 shutdown_connections 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55fcae8d6de0 0x55fcae8d91d0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcae857240 0x55fcae8633f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcae84c980 0x55fcae8602c0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 --2- 172.21.15.89:0/2751904035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcae84c060 0x55fcae85fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 -- 172.21.15.89:0/2751904035 >> 172.21.15.89:0/2751904035 conn(0x55fcae8485f0 msgr2=0x55fcae846f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:58.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0c96ffd640 0 client.18867 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:58.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 -- 172.21.15.89:0/2751904035 shutdown_connections 2024-09-15T23:33:58.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 1 -- 172.21.15.89:0/2751904035 wait complete. 2024-09-15T23:33:58.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.422+0000 7f0cc0608740 20 client.18867 trim_cache size 0 max 16384 2024-09-15T23:33:58.428 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 36, 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "ca53bdfb": { 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 36, 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 142.3, 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:33:58.430 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:33:58.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 -- 172.21.15.89:0/28371671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ca00a3240 msgr2=0x7f0ca00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:58.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ca00a3240 0x7f0ca00a5700 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0cac005d50 tx=0x7f0cac00aa80 comp rx=0 tx=0).stop 2024-09-15T23:33:58.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 -- 172.21.15.89:0/28371671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb8106f10 msgr2=0x7f0cb819cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:58.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb8106f10 0x7f0cb819cc30 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0031a60 tx=0x7f0cb0031a90 comp rx=0 tx=0).stop 2024-09-15T23:33:58.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 -- 172.21.15.89:0/28371671 shutdown_connections 2024-09-15T23:33:58.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ca00a3240 0x7f0ca00a5700 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb8106f10 0x7f0cb819cc30 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cb8102790 0x7f0cb81988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 --2- 172.21.15.89:0/28371671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cb8101de0 0x7f0cb8198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:58.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 -- 172.21.15.89:0/28371671 >> 172.21.15.89:0/28371671 conn(0x7f0cb80fd8c0 msgr2=0x7f0cb80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:58.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 -- 172.21.15.89:0/28371671 shutdown_connections 2024-09-15T23:33:58.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:58.423+0000 7f0cbe22f640 1 -- 172.21.15.89:0/28371671 wait complete. 2024-09-15T23:33:58.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:33:58 smithi120 ceph-mon[35250]: pgmap v648: 129 pgs: 129 active+clean; 765 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 853 B/s wr, 14 op/s 2024-09-15T23:33:58.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:33:58 smithi174 ceph-mon[35586]: pgmap v648: 129 pgs: 129 active+clean; 765 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 853 B/s wr, 14 op/s 2024-09-15T23:33:59.276 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ca53bdfb', total seconds elapsed: 27.5 2024-09-15T23:33:59.386 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:33:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.840+0000 7fd905a4a640 1 -- 172.21.15.89:0/2004009420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd900102ef0 msgr2=0x7fd90010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.840+0000 7fd905a4a640 1 --2- 172.21.15.89:0/2004009420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd900102ef0 0x7fd90010da40 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009900 tx=0x7fd8f802a480 comp rx=0 tx=0).stop 2024-09-15T23:33:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.840+0000 7fd905a4a640 1 -- 172.21.15.89:0/2004009420 shutdown_connections 2024-09-15T23:33:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.840+0000 7fd905a4a640 1 --2- 172.21.15.89:0/2004009420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd900102ef0 0x7fd90010da40 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.840+0000 7fd905a4a640 1 --2- 172.21.15.89:0/2004009420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd900102570 0x7fd9001029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.840+0000 7fd905a4a640 1 --2- 172.21.15.89:0/2004009420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd900100e20 0x7fd900101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.840+0000 7fd905a4a640 1 -- 172.21.15.89:0/2004009420 >> 172.21.15.89:0/2004009420 conn(0x7fd9000fc7d0 msgr2=0x7fd9000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.840+0000 7fd905a4a640 1 -- 172.21.15.89:0/2004009420 shutdown_connections 2024-09-15T23:33:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.840+0000 7fd905a4a640 1 -- 172.21.15.89:0/2004009420 wait complete. 2024-09-15T23:33:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.841+0000 7fd905a4a640 1 Processor -- start 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.841+0000 7fd905a4a640 1 -- start start 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.841+0000 7fd905a4a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd900100e20 0x7fd9001984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.841+0000 7fd905a4a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd900102570 0x7fd900198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.841+0000 7fd905a4a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd900102ef0 0x7fd90019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.841+0000 7fd905a4a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd900110f80 con 0x7fd900102ef0 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.841+0000 7fd905a4a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd900110e00 con 0x7fd900102570 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.841+0000 7fd905a4a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd900111100 con 0x7fd900100e20 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd900102ef0 0x7fd90019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8ff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd900102ef0 0x7fd90019cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37262/0 (socket says 172.21.15.89:37262) 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8ff7fe640 1 -- 172.21.15.89:0/3299875268 learned_addr learned my addr 172.21.15.89:0/3299875268 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd900102570 0x7fd900198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8feffd640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd900100e20 0x7fd9001984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8ff7fe640 1 -- 172.21.15.89:0/3299875268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd900100e20 msgr2=0x7fd9001984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8ff7fe640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd900100e20 0x7fd9001984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8ff7fe640 1 -- 172.21.15.89:0/3299875268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd900102570 msgr2=0x7fd900198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8ff7fe640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd900102570 0x7fd900198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8ff7fe640 1 -- 172.21.15.89:0/3299875268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90019d4a0 con 0x7fd900102ef0 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd8ff7fe640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd900102ef0 0x7fd90019cdc0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8004770 tx=0x7fd8f8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd904a48640 1 -- 172.21.15.89:0/3299875268 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f803f070 con 0x7fd900102ef0 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd904a48640 1 -- 172.21.15.89:0/3299875268 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8f8031c30 con 0x7fd900102ef0 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd904a48640 1 -- 172.21.15.89:0/3299875268 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8035030 con 0x7fd900102ef0 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd905a4a640 1 -- 172.21.15.89:0/3299875268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd90019d730 con 0x7fd900102ef0 2024-09-15T23:33:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.842+0000 7fd905a4a640 1 -- 172.21.15.89:0/3299875268 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9001a5070 con 0x7fd900102ef0 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.845+0000 7fd904a48640 1 -- 172.21.15.89:0/3299875268 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd8f8042070 con 0x7fd900102ef0 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.845+0000 7fd907e23740 1 Processor -- start 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.845+0000 7fd907e23740 1 -- start start 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd907e23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b49090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd907e23740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b495d0 0x55b5a7b49a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd907e23740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b54370 0x55b5a7b56760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd907e23740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b5a7aaf2f0 con 0x55b5a7b54370 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd907e23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b5a7a56cf0 con 0x55b5a7b4ab40 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd907e23740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b5a7aab2b0 con 0x55b5a7b495d0 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd904a48640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8e80a32c0 0x7fd8e80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8fe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b54370 0x55b5a7b56760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8fe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b54370 0x55b5a7b56760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37266/0 (socket says 172.21.15.89:37266) 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/87676183 learned_addr learned my addr 172.21.15.89:0/87676183 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8feffd640 1 --2- 172.21.15.89:0/87676183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b495d0 0x55b5a7b49a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8ff7fe640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8e80a32c0 0x7fd8e80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/87676183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b49090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/87676183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b495d0 msgr2=0x55b5a7b49a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/87676183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b495d0 0x55b5a7b49a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/87676183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b4ab40 msgr2=0x55b5a7b49090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/87676183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b49090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/87676183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a7a3b3a0 con 0x55b5a7b54370 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8feffd640 1 --2- 172.21.15.89:0/87676183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b495d0 0x55b5a7b49a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd904a48640 1 -- 172.21.15.89:0/3299875268 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd8f80f0430 con 0x7fd900102ef0 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/87676183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b54370 0x55b5a7b56760 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400bf90 tx=0x7fd8f400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7288e1cb271de368 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:59.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.846+0000 7fd8ff7fe640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8e80a32c0 0x7fd8e80a5780 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80edb90 tx=0x7fd8f80b8310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd8e5ffb640 1 -- 172.21.15.89:0/87676183 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4004300 con 0x55b5a7b54370 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd8e5ffb640 1 -- 172.21.15.89:0/87676183 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8f40044a0 con 0x55b5a7b54370 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd8e5ffb640 1 -- 172.21.15.89:0/87676183 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4003850 con 0x55b5a7b54370 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd907e23740 1 -- 172.21.15.89:0/87676183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b54370 msgr2=0x55b5a7b56760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd907e23740 1 --2- 172.21.15.89:0/87676183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b54370 0x55b5a7b56760 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400bf90 tx=0x7fd8f400b790 comp rx=0 tx=0).stop 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd907e23740 1 -- 172.21.15.89:0/87676183 shutdown_connections 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd907e23740 1 --2- 172.21.15.89:0/87676183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b54370 0x55b5a7b56760 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd907e23740 1 --2- 172.21.15.89:0/87676183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b495d0 0x55b5a7b49a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd907e23740 1 --2- 172.21.15.89:0/87676183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b49090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd907e23740 1 -- 172.21.15.89:0/87676183 >> 172.21.15.89:0/87676183 conn(0x55b5a7b12310 msgr2=0x55b5a7b45c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd907e23740 1 -- 172.21.15.89:0/87676183 shutdown_connections 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.847+0000 7fd907e23740 1 -- 172.21.15.89:0/87676183 wait complete. 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd907e23740 1 -- start start 2024-09-15T23:33:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd907e23740 1 Processor -- start 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd907e23740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b53d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd907e23740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b51020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd907e23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b54370 0x55b5a7b51610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd907e23740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b5a7b57070 con 0x55b5a7b495d0 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd907e23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b5a7a6a060 con 0x55b5a7b54370 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd907e23740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b5a7b571f0 con 0x55b5a7b4ab40 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd907e23740 10 client.0 _ll_register_callbacks cb 0x55b5a77bd8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd8fe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b53d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd8fe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b53d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37282/0 (socket says 172.21.15.89:37282) 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.848+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/2447954433 learned_addr learned my addr 172.21.15.89:0/2447954433 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8feffd640 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b51020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b54370 0x55b5a7b51610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/2447954433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 msgr2=0x55b5a7b51020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b51020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/2447954433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b54370 msgr2=0x55b5a7b51610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b54370 0x55b5a7b51610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/2447954433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a78203c0 con 0x55b5a7b495d0 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8feffd640 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b51020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b53d70 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4004cc0 tx=0x7fd8f400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4754acedf2f91690 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8deffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8deffd640 1 -- 172.21.15.89:0/2447954433 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4022020 con 0x55b5a7b495d0 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8deffd640 1 -- 172.21.15.89:0/2447954433 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd8f4004fa0 con 0x55b5a7b495d0 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd907e23740 1 -- 172.21.15.89:0/2447954433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8f40044a0 con 0x55b5a7b495d0 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8deffd640 1 -- 172.21.15.89:0/2447954433 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd8f40051a0 con 0x55b5a7b495d0 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd907e23740 10 client.18879 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd907e23740 1 -- 172.21.15.89:0/2447954433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5a7b62b60 con 0x55b5a7b495d0 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.849+0000 7fd8deffd640 1 -- 172.21.15.89:0/2447954433 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8f40044a0 con 0x55b5a7b495d0 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.850+0000 7fd8deffd640 1 -- 172.21.15.89:0/2447954433 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd8f401c020 con 0x55b5a7b495d0 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.850+0000 7fd907e23740 10 client.18879 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.850+0000 7fd907e23740 10 client.18879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:59.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.850+0000 7fd907e23740 20 client.18879 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.850+0000 7fd907e23740 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5a7b63f10 0x55b5a7b66300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.850+0000 7fd907e23740 4 client.18879 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:33:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.850+0000 7fd907e23740 1 -- 172.21.15.89:0/2447954433 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5a7abc860 con 0x55b5a7b63f10 2024-09-15T23:33:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.850+0000 7fd8feffd640 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5a7b63f10 0x55b5a7b66300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.850+0000 7fd8feffd640 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5a7b63f10 0x55b5a7b66300 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ddcff5bf88ea759 server_cookie=33e175785e4956d9 in_seq=0 out_seq=0 2024-09-15T23:33:59.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.850+0000 7fd8deffd640 10 client.18879 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:59.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd8deffd640 1 -- 172.21.15.89:0/2447954433 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5a7abc860 con 0x55b5a7b63f10 2024-09-15T23:33:59.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd8deffd640 10 client.18879 handle_command_reply: tid=0 2024-09-15T23:33:59.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd907e23740 1 client.18879 shutdown 2024-09-15T23:33:59.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd907e23740 1 -- 172.21.15.89:0/2447954433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 msgr2=0x55b5a7b53d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:59.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd907e23740 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b53d70 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4004cc0 tx=0x7fd8f400b390 comp rx=0 tx=0).stop 2024-09-15T23:33:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd907e23740 1 -- 172.21.15.89:0/2447954433 shutdown_connections 2024-09-15T23:33:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd907e23740 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5a7b63f10 0x55b5a7b66300 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd8deffd640 0 client.18879 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd907e23740 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b54370 0x55b5a7b51610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd907e23740 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b51020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd907e23740 1 --2- 172.21.15.89:0/2447954433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b53d70 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd907e23740 1 -- 172.21.15.89:0/2447954433 >> 172.21.15.89:0/2447954433 conn(0x55b5a7b12310 msgr2=0x55b5a7b4b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.851+0000 7fd907e23740 1 -- 172.21.15.89:0/2447954433 shutdown_connections 2024-09-15T23:33:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.852+0000 7fd907e23740 1 -- 172.21.15.89:0/2447954433 wait complete. 2024-09-15T23:33:59.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.852+0000 7fd907e23740 20 client.18879 trim_cache size 0 max 16384 2024-09-15T23:33:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.864+0000 7fd907e23740 1 Processor -- start 2024-09-15T23:33:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.864+0000 7fd907e23740 1 -- start start 2024-09-15T23:33:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.864+0000 7fd907e23740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.864+0000 7fd907e23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b4c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.864+0000 7fd907e23740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b54370 0x55b5a7b6ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd907e23740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9001aed90 con 0x55b5a7b495d0 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd907e23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9001b08d0 con 0x55b5a7b4ab40 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd907e23740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd9001af460 con 0x55b5a7b54370 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8fe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8fe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37284/0 (socket says 172.21.15.89:37284) 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/1619345138 learned_addr learned my addr 172.21.15.89:0/1619345138 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/1619345138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b54370 msgr2=0x55b5a7b6ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/1619345138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b54370 0x55b5a7b6ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/1619345138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b4ab40 msgr2=0x55b5a7b4c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/1619345138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b4c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8fe7fc640 1 -- 172.21.15.89:0/1619345138 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a7b56f50 con 0x55b5a7b495d0 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/1619345138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4c120 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fd8f401e510 tx=0x7fd8f400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a9d3370c2cea2a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8e57fa640 1 -- 172.21.15.89:0/1619345138 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4022030 con 0x55b5a7b495d0 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8e57fa640 1 -- 172.21.15.89:0/1619345138 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8f400b670 con 0x55b5a7b495d0 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.865+0000 7fd8e57fa640 1 -- 172.21.15.89:0/1619345138 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4004950 con 0x55b5a7b495d0 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 -- 172.21.15.89:0/1619345138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 msgr2=0x55b5a7b4c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 --2- 172.21.15.89:0/1619345138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4c120 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fd8f401e510 tx=0x7fd8f400ba50 comp rx=0 tx=0).stop 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 -- 172.21.15.89:0/1619345138 shutdown_connections 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 --2- 172.21.15.89:0/1619345138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b54370 0x55b5a7b6ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 --2- 172.21.15.89:0/1619345138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b4c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 --2- 172.21.15.89:0/1619345138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4c120 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 -- 172.21.15.89:0/1619345138 >> 172.21.15.89:0/1619345138 conn(0x55b5a7b12310 msgr2=0x55b5a7b44d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 -- 172.21.15.89:0/1619345138 shutdown_connections 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 -- 172.21.15.89:0/1619345138 wait complete. 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 -- start start 2024-09-15T23:33:59.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.866+0000 7fd907e23740 1 Processor -- start 2024-09-15T23:33:59.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd907e23740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd907e23740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b4e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd907e23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b54370 0x55b5a7b4eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd907e23740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9001b0200 con 0x55b5a7b495d0 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd907e23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9001afb30 con 0x55b5a7b54370 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd907e23740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd9001acd70 con 0x55b5a7b4ab40 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd907e23740 10 client.0 _ll_register_callbacks cb 0x7fd8f0009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8fe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8feffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b4e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8feffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b4e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:55010/0 (socket says 172.21.15.89:55010) 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8fe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37296/0 (socket says 172.21.15.89:37296) 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8feffd640 1 -- 172.21.15.89:0/3026600569 learned_addr learned my addr 172.21.15.89:0/3026600569 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8feffd640 1 -- 172.21.15.89:0/3026600569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b54370 msgr2=0x55b5a7b4eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:59.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8feffd640 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b54370 0x55b5a7b4eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8feffd640 1 -- 172.21.15.89:0/3026600569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 msgr2=0x55b5a7b4e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8feffd640 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8feffd640 1 -- 172.21.15.89:0/3026600569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a7bcda40 con 0x55b5a7b4ab40 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8feffd640 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b4e5d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009fd0 tx=0x7fd8f0014950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbd65a97fc54c5cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.867+0000 7fd8deffd640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.868+0000 7fd8deffd640 1 -- 172.21.15.89:0/3026600569 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f00116a0 con 0x55b5a7b4ab40 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.868+0000 7fd907e23740 1 -- 172.21.15.89:0/3026600569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8f400b670 con 0x55b5a7b4ab40 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.868+0000 7fd8deffd640 1 -- 172.21.15.89:0/3026600569 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd8f0011840 con 0x55b5a7b4ab40 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.868+0000 7fd8deffd640 1 -- 172.21.15.89:0/3026600569 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd8f002e050 con 0x55b5a7b4ab40 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.868+0000 7fd907e23740 10 client.27325 fetch_fsmap learned FSMap version 29 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.868+0000 7fd907e23740 1 -- 172.21.15.89:0/3026600569 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5a7bced60 con 0x55b5a7b4ab40 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.868+0000 7fd8deffd640 1 -- 172.21.15.89:0/3026600569 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8f400b670 con 0x55b5a7b4ab40 2024-09-15T23:33:59.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.868+0000 7fd8deffd640 1 -- 172.21.15.89:0/3026600569 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd8f000f040 con 0x55b5a7b4ab40 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.869+0000 7fd907e23740 10 client.27325 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.869+0000 7fd907e23740 10 client.27325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.869+0000 7fd907e23740 20 client.27325 populate_metadata read hostname 'smithi089' 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.869+0000 7fd907e23740 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5a7b63f10 0x55b5a7bd14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.869+0000 7fd907e23740 4 client.27325 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.869+0000 7fd907e23740 1 -- 172.21.15.89:0/3026600569 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5a7ab6d70 con 0x55b5a7b63f10 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.869+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5a7b63f10 0x55b5a7bd14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.869+0000 7fd8deffd640 10 client.27325 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.869+0000 7fd8fe7fc640 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5a7b63f10 0x55b5a7bd14a0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10f2908206109e9d server_cookie=215cef4ab350f08e in_seq=0 out_seq=0 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd8deffd640 1 -- 172.21.15.89:0/3026600569 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5a7ab6d70 con 0x55b5a7b63f10 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd8deffd640 10 client.27325 handle_command_reply: tid=0 2024-09-15T23:33:59.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 client.27325 shutdown 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 -- 172.21.15.89:0/3026600569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 msgr2=0x55b5a7b4e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b4e5d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009fd0 tx=0x7fd8f0014950 comp rx=0 tx=0).stop 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 -- 172.21.15.89:0/3026600569 shutdown_connections 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5a7b63f10 0x55b5a7bd14a0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5a7b54370 0x55b5a7b4eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5a7b4ab40 0x55b5a7b4e5d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd8deffd640 0 client.27325 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 --2- 172.21.15.89:0/3026600569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5a7b495d0 0x55b5a7b4e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 -- 172.21.15.89:0/3026600569 >> 172.21.15.89:0/3026600569 conn(0x55b5a7b12310 msgr2=0x55b5a7b52880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 -- 172.21.15.89:0/3026600569 shutdown_connections 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 1 -- 172.21.15.89:0/3026600569 wait complete. 2024-09-15T23:33:59.876 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.870+0000 7fd907e23740 20 client.27325 trim_cache size 0 max 16384 2024-09-15T23:33:59.877 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:33:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 -- 172.21.15.89:0/3299875268 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8e80a32c0 msgr2=0x7fd8e80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:59.877 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8e80a32c0 0x7fd8e80a5780 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80edb90 tx=0x7fd8f80b8310 comp rx=0 tx=0).stop 2024-09-15T23:33:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 -- 172.21.15.89:0/3299875268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd900102ef0 msgr2=0x7fd90019cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:33:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd900102ef0 0x7fd90019cdc0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8004770 tx=0x7fd8f8003c10 comp rx=0 tx=0).stop 2024-09-15T23:33:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 -- 172.21.15.89:0/3299875268 shutdown_connections 2024-09-15T23:33:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd8e80a32c0 0x7fd8e80a5780 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd900102ef0 0x7fd90019cdc0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd900102570 0x7fd900198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 --2- 172.21.15.89:0/3299875268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd900100e20 0x7fd9001984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:33:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 -- 172.21.15.89:0/3299875268 >> 172.21.15.89:0/3299875268 conn(0x7fd9000fc7d0 msgr2=0x7fd90006a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:33:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 -- 172.21.15.89:0/3299875268 shutdown_connections 2024-09-15T23:33:59.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:33:59.871+0000 7fd905a4a640 1 -- 172.21.15.89:0/3299875268 wait complete. 2024-09-15T23:34:00.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:00 smithi089 ceph-mon[30933]: pgmap v649: 129 pgs: 129 active+clean; 765 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 11 op/s 2024-09-15T23:34:00.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:00 smithi120 ceph-mon[35250]: pgmap v649: 129 pgs: 129 active+clean; 765 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 11 op/s 2024-09-15T23:34:00.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:00 smithi174 ceph-mon[35586]: pgmap v649: 129 pgs: 129 active+clean; 765 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 11 op/s 2024-09-15T23:34:01.646 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:34:01.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:34:02.562 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:02.685 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:02 smithi089 ceph-mon[30933]: pgmap v650: 129 pgs: 129 active+clean; 765 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 853 B/s wr, 13 op/s 2024-09-15T23:34:02.685 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:02 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:34:02.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:02 smithi120 ceph-mon[35250]: pgmap v650: 129 pgs: 129 active+clean; 765 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 853 B/s wr, 13 op/s 2024-09-15T23:34:02.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:02 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:34:02.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:02 smithi174 ceph-mon[35586]: pgmap v650: 129 pgs: 129 active+clean; 765 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 853 B/s wr, 13 op/s 2024-09-15T23:34:02.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:02 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:34:03.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.044+0000 7fd497eee640 1 -- 172.21.15.89:0/1347583295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4901036e0 msgr2=0x7fd490107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.044+0000 7fd497eee640 1 --2- 172.21.15.89:0/1347583295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4901036e0 0x7fd490107b90 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fd484009930 tx=0x7fd48402a420 comp rx=0 tx=0).stop 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 -- 172.21.15.89:0/1347583295 shutdown_connections 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 --2- 172.21.15.89:0/1347583295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4901082a0 0x7fd49010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 --2- 172.21.15.89:0/1347583295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4901036e0 0x7fd490107b90 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 --2- 172.21.15.89:0/1347583295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd490102d30 0x7fd490103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 -- 172.21.15.89:0/1347583295 >> 172.21.15.89:0/1347583295 conn(0x7fd4900fc7d0 msgr2=0x7fd4900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 -- 172.21.15.89:0/1347583295 shutdown_connections 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 -- 172.21.15.89:0/1347583295 wait complete. 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 Processor -- start 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 -- start start 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd490102d30 0x7fd4901985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4901036e0 0x7fd490198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4901082a0 0x7fd49019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd490112060 con 0x7fd4901036e0 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd490111ee0 con 0x7fd4901082a0 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.045+0000 7fd497eee640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd4901121e0 con 0x7fd490102d30 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd495489640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4901036e0 0x7fd490198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd495489640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4901036e0 0x7fd490198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37318/0 (socket says 172.21.15.89:37318) 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd495489640 1 -- 172.21.15.89:0/2201370242 learned_addr learned my addr 172.21.15.89:0/2201370242 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd49648b640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4901082a0 0x7fd49019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd495c8a640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd490102d30 0x7fd4901985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd495489640 1 -- 172.21.15.89:0/2201370242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd490102d30 msgr2=0x7fd4901985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd495489640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd490102d30 0x7fd4901985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd495489640 1 -- 172.21.15.89:0/2201370242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4901082a0 msgr2=0x7fd49019ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:03.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd495489640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4901082a0 0x7fd49019ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd495489640 1 -- 172.21.15.89:0/2201370242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49019d550 con 0x7fd4901036e0 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd495489640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4901036e0 0x7fd490198ae0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd484005940 tx=0x7fd4840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd48effd640 1 -- 172.21.15.89:0/2201370242 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48403f030 con 0x7fd4901036e0 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd497eee640 1 -- 172.21.15.89:0/2201370242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd49019d7e0 con 0x7fd4901036e0 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd48effd640 1 -- 172.21.15.89:0/2201370242 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd484003ce0 con 0x7fd4901036e0 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd48effd640 1 -- 172.21.15.89:0/2201370242 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd484035070 con 0x7fd4901036e0 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.046+0000 7fd497eee640 1 -- 172.21.15.89:0/2201370242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4901a5120 con 0x7fd4901036e0 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.048+0000 7fd48effd640 1 -- 172.21.15.89:0/2201370242 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd484042030 con 0x7fd4901036e0 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49a2c7740 1 Processor -- start 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49a2c7740 1 -- start start 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54676f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54681800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54684130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee545dc280 con 0x55ee54681d40 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ee54583b10 con 0x55ee54676b60 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ee545d8240 con 0x55ee54677480 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd495489640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54684130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd495c8a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54681800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd48effd640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4780a3240 0x7fd4780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49648b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54676f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49648b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54676f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:43754/0 (socket says 172.21.15.89:43754) 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd495c8a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54681800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:55042/0 (socket says 172.21.15.89:55042) 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.049+0000 7fd49648b640 1 -- 172.21.15.89:0/2939820016 learned_addr learned my addr 172.21.15.89:0/2939820016 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd49648b640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4780a3240 0x7fd4780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd48effd640 1 -- 172.21.15.89:0/2201370242 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd4840f00d0 con 0x7fd4901036e0 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd495489640 1 -- 172.21.15.89:0/2939820016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 msgr2=0x55ee54681800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:03.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd495489640 1 --2- 172.21.15.89:0/2939820016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54681800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd495489640 1 -- 172.21.15.89:0/2939820016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 msgr2=0x55ee54676f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd495489640 1 --2- 172.21.15.89:0/2939820016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54676f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd495489640 1 -- 172.21.15.89:0/2939820016 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee545901f0 con 0x55ee54681d40 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd495c8a640 1 --2- 172.21.15.89:0/2939820016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54681800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd495489640 1 --2- 172.21.15.89:0/2939820016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54684130 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fd4840ed830 tx=0x7fd48402a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48dd5a5d553b0738 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd49648b640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4780a3240 0x7fd4780a5700 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd48800bd20 tx=0x7fd488004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd477fff640 1 -- 172.21.15.89:0/2939820016 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd484049070 con 0x55ee54681d40 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd477fff640 1 -- 172.21.15.89:0/2939820016 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4840bc3b0 con 0x55ee54681d40 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.050+0000 7fd477fff640 1 -- 172.21.15.89:0/2939820016 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48403bbe0 con 0x55ee54681d40 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.051+0000 7fd49a2c7740 1 -- 172.21.15.89:0/2939820016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 msgr2=0x55ee54684130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.051+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/2939820016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54684130 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fd4840ed830 tx=0x7fd48402a420 comp rx=0 tx=0).stop 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.051+0000 7fd49a2c7740 1 -- 172.21.15.89:0/2939820016 shutdown_connections 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.051+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/2939820016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54684130 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.051+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/2939820016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54681800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.051+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/2939820016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54676f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.051+0000 7fd49a2c7740 1 -- 172.21.15.89:0/2939820016 >> 172.21.15.89:0/2939820016 conn(0x55ee546730f0 msgr2=0x55ee546734d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.051+0000 7fd49a2c7740 1 -- 172.21.15.89:0/2939820016 shutdown_connections 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.051+0000 7fd49a2c7740 1 -- 172.21.15.89:0/2939820016 wait complete. 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.051+0000 7fd49a2c7740 1 -- start start 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd49a2c7740 1 Processor -- start 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54676b60 0x55ee5467ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54677480 0x55ee5467f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54681d40 0x55ee5467f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ee546846d0 con 0x55ee54676b60 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ee54596f10 con 0x55ee54677480 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ee54684850 con 0x55ee54681d40 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd49a2c7740 10 client.0 _ll_register_callbacks cb 0x55ee5455fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd495c8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54676b60 0x55ee5467ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd495c8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54676b60 0x55ee5467ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37332/0 (socket says 172.21.15.89:37332) 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd495c8a640 1 -- 172.21.15.89:0/634372229 learned_addr learned my addr 172.21.15.89:0/634372229 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd495489640 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54677480 0x55ee5467f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd49648b640 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54681d40 0x55ee5467f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd495c8a640 1 -- 172.21.15.89:0/634372229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54681d40 msgr2=0x55ee5467f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd495c8a640 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54681d40 0x55ee5467f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd495c8a640 1 -- 172.21.15.89:0/634372229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54677480 msgr2=0x55ee5467f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd495c8a640 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54677480 0x55ee5467f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd495c8a640 1 -- 172.21.15.89:0/634372229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5434d6a0 con 0x55ee54676b60 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.052+0000 7fd495489640 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54677480 0x55ee5467f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd495c8a640 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54676b60 0x55ee5467ec60 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fd480003cc0 tx=0x7fd480004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6121f55a745fa5e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd4757fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd4757fa640 1 -- 172.21.15.89:0/634372229 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480011650 con 0x55ee54676b60 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd4757fa640 1 -- 172.21.15.89:0/634372229 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd4800117f0 con 0x55ee54676b60 2024-09-15T23:34:03.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd49a2c7740 1 -- 172.21.15.89:0/634372229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4840bc3b0 con 0x55ee54676b60 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd4757fa640 1 -- 172.21.15.89:0/634372229 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd4800119f0 con 0x55ee54676b60 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd49a2c7740 10 client.18897 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd49a2c7740 1 -- 172.21.15.89:0/634372229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ee54690260 con 0x55ee54676b60 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd4757fa640 1 -- 172.21.15.89:0/634372229 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4840bc3b0 con 0x55ee54676b60 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd4757fa640 1 -- 172.21.15.89:0/634372229 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd48001bc30 con 0x55ee54676b60 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd49a2c7740 10 client.18897 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd49a2c7740 10 client.18897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd49a2c7740 20 client.18897 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ee54691720 0x55ee54693be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd49a2c7740 4 client.18897 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.053+0000 7fd49a2c7740 1 -- 172.21.15.89:0/634372229 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee545e97f0 con 0x55ee54691720 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.054+0000 7fd495489640 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ee54691720 0x55ee54693be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.054+0000 7fd4757fa640 10 client.18897 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:03.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.054+0000 7fd495489640 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ee54691720 0x55ee54693be0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe11d2e01013ad25 server_cookie=cd7a5158fcff71fc in_seq=0 out_seq=0 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd4757fa640 1 -- 172.21.15.89:0/634372229 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ee545e97f0 con 0x55ee54691720 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd4757fa640 10 client.18897 handle_command_reply: tid=0 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 client.18897 shutdown 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 -- 172.21.15.89:0/634372229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54676b60 msgr2=0x55ee5467ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54676b60 0x55ee5467ec60 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fd480003cc0 tx=0x7fd480004410 comp rx=0 tx=0).stop 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 -- 172.21.15.89:0/634372229 shutdown_connections 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ee54691720 0x55ee54693be0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54681d40 0x55ee5467f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54677480 0x55ee5467f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd4757fa640 0 client.18897 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/634372229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54676b60 0x55ee5467ec60 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 -- 172.21.15.89:0/634372229 >> 172.21.15.89:0/634372229 conn(0x55ee546730f0 msgr2=0x55ee54678dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 -- 172.21.15.89:0/634372229 shutdown_connections 2024-09-15T23:34:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 1 -- 172.21.15.89:0/634372229 wait complete. 2024-09-15T23:34:03.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.055+0000 7fd49a2c7740 20 client.18897 trim_cache size 0 max 16384 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49a2c7740 1 Processor -- start 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49a2c7740 1 -- start start 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54680ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54699500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54699a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd4901acd30 con 0x55ee54681d40 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd4901ae870 con 0x55ee54676b60 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd4901ad400 con 0x55ee54677480 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49648b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54699a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49648b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54699a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37334/0 (socket says 172.21.15.89:37334) 2024-09-15T23:34:03.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd495489640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54699500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd49648b640 1 -- 172.21.15.89:0/324370038 learned_addr learned my addr 172.21.15.89:0/324370038 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.072+0000 7fd495c8a640 1 --2- 172.21.15.89:0/324370038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54680ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd495489640 1 -- 172.21.15.89:0/324370038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 msgr2=0x55ee54680ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd495489640 1 --2- 172.21.15.89:0/324370038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54680ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd495489640 1 -- 172.21.15.89:0/324370038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 msgr2=0x55ee54699a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd495489640 1 --2- 172.21.15.89:0/324370038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54699a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd495489640 1 -- 172.21.15.89:0/324370038 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5467dc10 con 0x55ee54677480 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd495c8a640 1 --2- 172.21.15.89:0/324370038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54680ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd49648b640 1 --2- 172.21.15.89:0/324370038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54699a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd495489640 1 --2- 172.21.15.89:0/324370038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54699500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd4840bbe80 tx=0x7fd484009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c8a20ea2ec644af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd4777fe640 1 -- 172.21.15.89:0/324370038 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd484049040 con 0x55ee54677480 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd4777fe640 1 -- 172.21.15.89:0/324370038 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4840c4f70 con 0x55ee54677480 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.073+0000 7fd4777fe640 1 -- 172.21.15.89:0/324370038 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48403bbe0 con 0x55ee54677480 2024-09-15T23:34:03.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 -- 172.21.15.89:0/324370038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 msgr2=0x55ee54699500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/324370038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54699500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd4840bbe80 tx=0x7fd484009880 comp rx=0 tx=0).stop 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 -- 172.21.15.89:0/324370038 shutdown_connections 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/324370038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee54699a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/324370038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee54699500 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/324370038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54680ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 -- 172.21.15.89:0/324370038 >> 172.21.15.89:0/324370038 conn(0x55ee546730f0 msgr2=0x55ee546803b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 -- 172.21.15.89:0/324370038 shutdown_connections 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 -- 172.21.15.89:0/324370038 wait complete. 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 -- start start 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.074+0000 7fd49a2c7740 1 Processor -- start 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54685f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee546864b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd49a2c7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee546fb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd4901ae1a0 con 0x55ee54681d40 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd4901adad0 con 0x55ee54676b60 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd49a2c7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd4901aad10 con 0x55ee54677480 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd49a2c7740 10 client.0 _ll_register_callbacks cb 0x7fd480009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd49648b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee546fb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd495c8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54685f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd495489640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee546864b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd495c8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54685f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:43780/0 (socket says 172.21.15.89:43780) 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd495c8a640 1 -- 172.21.15.89:0/3879620789 learned_addr learned my addr 172.21.15.89:0/3879620789 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd495c8a640 1 -- 172.21.15.89:0/3879620789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 msgr2=0x55ee546864b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd495c8a640 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee546864b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd495c8a640 1 -- 172.21.15.89:0/3879620789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 msgr2=0x55ee546fb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd495c8a640 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee546fb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd495c8a640 1 -- 172.21.15.89:0/3879620789 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee546fbb10 con 0x55ee54676b60 2024-09-15T23:34:03.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.075+0000 7fd49648b640 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee546fb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.076+0000 7fd495c8a640 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54685f70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd480006010 tx=0x7fd480009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3b80de0025057e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.076+0000 7fd4757fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.076+0000 7fd4757fa640 1 -- 172.21.15.89:0/3879620789 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480011da0 con 0x55ee54676b60 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.076+0000 7fd49a2c7740 1 -- 172.21.15.89:0/3879620789 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4840c4f70 con 0x55ee54676b60 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.076+0000 7fd4757fa640 1 -- 172.21.15.89:0/3879620789 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd480002030 con 0x55ee54676b60 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.076+0000 7fd4757fa640 1 -- 172.21.15.89:0/3879620789 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd480004910 con 0x55ee54676b60 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.076+0000 7fd49a2c7740 10 client.27230 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.076+0000 7fd49a2c7740 1 -- 172.21.15.89:0/3879620789 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ee546fce70 con 0x55ee54676b60 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd4757fa640 1 -- 172.21.15.89:0/3879620789 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4840c4f70 con 0x55ee54676b60 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd4757fa640 1 -- 172.21.15.89:0/3879620789 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd480005170 con 0x55ee54676b60 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd49a2c7740 10 client.27230 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd49a2c7740 10 client.27230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd49a2c7740 20 client.27230 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:03.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ee546fd5e0 0x55ee546ff9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd49a2c7740 4 client.27230 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd49a2c7740 1 -- 172.21.15.89:0/3879620789 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ee545e3d00 con 0x55ee546fd5e0 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd495489640 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ee546fd5e0 0x55ee546ff9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd4757fa640 10 client.27230 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.077+0000 7fd495489640 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ee546fd5e0 0x55ee546ff9d0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4f7931b5fda0d35 server_cookie=a0ceb809c3f0c162 in_seq=0 out_seq=0 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.078+0000 7fd4757fa640 1 -- 172.21.15.89:0/3879620789 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ee545e3d00 con 0x55ee546fd5e0 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.078+0000 7fd4757fa640 10 client.27230 handle_command_reply: tid=0 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.078+0000 7fd49a2c7740 1 client.27230 shutdown 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 1 -- 172.21.15.89:0/3879620789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 msgr2=0x55ee54685f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54685f70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd480006010 tx=0x7fd480009f30 comp rx=0 tx=0).stop 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 1 -- 172.21.15.89:0/3879620789 shutdown_connections 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ee546fd5e0 0x55ee546ff9d0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee54681d40 0x55ee546fb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ee54677480 0x55ee546864b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 1 --2- 172.21.15.89:0/3879620789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ee54676b60 0x55ee54685f70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 1 -- 172.21.15.89:0/3879620789 >> 172.21.15.89:0/3879620789 conn(0x55ee546730f0 msgr2=0x55ee54678dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:03.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd4757fa640 0 client.27230 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:03.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 1 -- 172.21.15.89:0/3879620789 shutdown_connections 2024-09-15T23:34:03.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 1 -- 172.21.15.89:0/3879620789 wait complete. 2024-09-15T23:34:03.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.079+0000 7fd49a2c7740 20 client.27230 trim_cache size 0 max 16384 2024-09-15T23:34:03.089 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:34:03.089 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:34:03.089 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "769d7fcc-b51f-447a-a6c5-083798aaf7c9", 2024-09-15T23:34:03.089 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:34:03.090 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:34:03.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 -- 172.21.15.89:0/2201370242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4780a3240 msgr2=0x7fd4780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:03.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4780a3240 0x7fd4780a5700 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd48800bd20 tx=0x7fd488004730 comp rx=0 tx=0).stop 2024-09-15T23:34:03.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 -- 172.21.15.89:0/2201370242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4901036e0 msgr2=0x7fd490198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:03.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4901036e0 0x7fd490198ae0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd484005940 tx=0x7fd4840042f0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 -- 172.21.15.89:0/2201370242 shutdown_connections 2024-09-15T23:34:03.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd4780a3240 0x7fd4780a5700 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd4901082a0 0x7fd49019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4901036e0 0x7fd490198ae0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 --2- 172.21.15.89:0/2201370242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd490102d30 0x7fd4901985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:03.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 -- 172.21.15.89:0/2201370242 >> 172.21.15.89:0/2201370242 conn(0x7fd4900fc7d0 msgr2=0x7fd490105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:03.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 -- 172.21.15.89:0/2201370242 shutdown_connections 2024-09-15T23:34:03.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:03.080+0000 7fd497eee640 1 -- 172.21.15.89:0/2201370242 wait complete. 2024-09-15T23:34:03.920 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:34:04.188 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:03 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:03 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:34:04.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:03 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:04.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:03 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:34:04.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:03 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:04.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:03 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:03 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:34:04.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:03 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:04.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:03 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:34:04.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:03 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:04.309 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.309 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.309 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.309 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:03 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:04.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:03 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:34:04.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:03 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:04.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:03 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:34:04.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:03 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:04.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.676+0000 7ff092df5640 1 -- 172.21.15.89:0/2503066562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff08c1082a0 msgr2=0x7ff08c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:04.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.676+0000 7ff092df5640 1 --2- 172.21.15.89:0/2503066562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff08c1082a0 0x7ff08c10ec40 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7ff084009cd0 tx=0x7ff08402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.676+0000 7ff092df5640 1 -- 172.21.15.89:0/2503066562 shutdown_connections 2024-09-15T23:34:04.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.676+0000 7ff092df5640 1 --2- 172.21.15.89:0/2503066562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff08c1082a0 0x7ff08c10ec40 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.676+0000 7ff092df5640 1 --2- 172.21.15.89:0/2503066562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c1036e0 0x7ff08c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.676+0000 7ff092df5640 1 --2- 172.21.15.89:0/2503066562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff08c102d30 0x7ff08c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.676+0000 7ff092df5640 1 -- 172.21.15.89:0/2503066562 >> 172.21.15.89:0/2503066562 conn(0x7ff08c0fc7d0 msgr2=0x7ff08c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:04.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.676+0000 7ff092df5640 1 -- 172.21.15.89:0/2503066562 shutdown_connections 2024-09-15T23:34:04.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.676+0000 7ff092df5640 1 -- 172.21.15.89:0/2503066562 wait complete. 2024-09-15T23:34:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.677+0000 7ff092df5640 1 Processor -- start 2024-09-15T23:34:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.677+0000 7ff092df5640 1 -- start start 2024-09-15T23:34:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.677+0000 7ff092df5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff08c102d30 0x7ff08c198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff092df5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff08c1036e0 0x7ff08c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff092df5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c1082a0 0x7ff08c19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff092df5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c112150 con 0x7ff08c1036e0 2024-09-15T23:34:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff092df5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c111fd0 con 0x7ff08c1082a0 2024-09-15T23:34:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff092df5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c1122d0 con 0x7ff08c102d30 2024-09-15T23:34:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff08bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff08c1036e0 0x7ff08c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff08bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff08c1036e0 0x7ff08c198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37364/0 (socket says 172.21.15.89:37364) 2024-09-15T23:34:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff08bfff640 1 -- 172.21.15.89:0/1597653895 learned_addr learned my addr 172.21.15.89:0/1597653895 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff090b91640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff08c102d30 0x7ff08c198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff091392640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c1082a0 0x7ff08c19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff08bfff640 1 -- 172.21.15.89:0/1597653895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff08c102d30 msgr2=0x7ff08c198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff08bfff640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff08c102d30 0x7ff08c198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff08bfff640 1 -- 172.21.15.89:0/1597653895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c1082a0 msgr2=0x7ff08c19cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff08bfff640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c1082a0 0x7ff08c19cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff08bfff640 1 -- 172.21.15.89:0/1597653895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08c19d640 con 0x7ff08c1036e0 2024-09-15T23:34:04.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff08bfff640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff08c1036e0 0x7ff08c198bd0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7ff07800d740 tx=0x7ff07800db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:04.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.678+0000 7ff089ffb640 1 -- 172.21.15.89:0/1597653895 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff078017070 con 0x7ff08c1036e0 2024-09-15T23:34:04.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.679+0000 7ff092df5640 1 -- 172.21.15.89:0/1597653895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff08c19d930 con 0x7ff08c1036e0 2024-09-15T23:34:04.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.679+0000 7ff089ffb640 1 -- 172.21.15.89:0/1597653895 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff078010460 con 0x7ff08c1036e0 2024-09-15T23:34:04.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.679+0000 7ff089ffb640 1 -- 172.21.15.89:0/1597653895 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07800f620 con 0x7ff08c1036e0 2024-09-15T23:34:04.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.679+0000 7ff092df5640 1 -- 172.21.15.89:0/1597653895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff08c1a5210 con 0x7ff08c1036e0 2024-09-15T23:34:04.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.681+0000 7ff089ffb640 1 -- 172.21.15.89:0/1597653895 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff07801a070 con 0x7ff08c1036e0 2024-09-15T23:34:04.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.681+0000 7ff0951ce740 1 Processor -- start 2024-09-15T23:34:04.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.681+0000 7ff0951ce740 1 -- start start 2024-09-15T23:34:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdee780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff089ffb640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff07c0a32c0 0x7ff07c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdfd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 0x55704bdffe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff0951ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55704bd542e0 con 0x55704bdee3a0 2024-09-15T23:34:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff0951ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55704bcfbce0 con 0x55704bdfda90 2024-09-15T23:34:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff0951ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55704bd502a0 con 0x55704bdeecc0 2024-09-15T23:34:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff091392640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff091392640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdee780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37380/0 (socket says 172.21.15.89:37380) 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff091392640 1 -- 172.21.15.89:0/3107092139 learned_addr learned my addr 172.21.15.89:0/3107092139 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff090b91640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff07c0a32c0 0x7ff07c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff089ffb640 1 -- 172.21.15.89:0/1597653895 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff0780cf6a0 con 0x7ff08c1036e0 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff08bfff640 1 --2- 172.21.15.89:0/3107092139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdfd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff091392640 1 --2- 172.21.15.89:0/3107092139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 0x55704bdffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff091392640 1 -- 172.21.15.89:0/3107092139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 msgr2=0x55704bdfd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff091392640 1 --2- 172.21.15.89:0/3107092139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdfd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff091392640 1 -- 172.21.15.89:0/3107092139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 msgr2=0x55704bdffe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff091392640 1 --2- 172.21.15.89:0/3107092139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 0x55704bdffe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff091392640 1 -- 172.21.15.89:0/3107092139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55704bce0440 con 0x55704bdee3a0 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff08bfff640 1 --2- 172.21.15.89:0/3107092139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdfd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.682+0000 7ff090b91640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff07c0a32c0 0x7ff07c0a5780 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff080009990 tx=0x7ff080009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.683+0000 7ff091392640 1 --2- 172.21.15.89:0/3107092139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdee780 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7ff084009430 tx=0x7ff084005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d4fcdb40f67895 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.683+0000 7ff076ffd640 1 -- 172.21.15.89:0/3107092139 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff084044020 con 0x55704bdee3a0 2024-09-15T23:34:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.683+0000 7ff076ffd640 1 -- 172.21.15.89:0/3107092139 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff084036040 con 0x55704bdee3a0 2024-09-15T23:34:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.683+0000 7ff076ffd640 1 -- 172.21.15.89:0/3107092139 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff084004670 con 0x55704bdee3a0 2024-09-15T23:34:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.683+0000 7ff0951ce740 1 -- 172.21.15.89:0/3107092139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 msgr2=0x55704bdee780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.683+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3107092139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdee780 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7ff084009430 tx=0x7ff084005f10 comp rx=0 tx=0).stop 2024-09-15T23:34:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 -- 172.21.15.89:0/3107092139 shutdown_connections 2024-09-15T23:34:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3107092139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 0x55704bdffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3107092139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdfd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3107092139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdee780 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 -- 172.21.15.89:0/3107092139 >> 172.21.15.89:0/3107092139 conn(0x55704bdeaa30 msgr2=0x55704bdeae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 -- 172.21.15.89:0/3107092139 shutdown_connections 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 -- 172.21.15.89:0/3107092139 wait complete. 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 -- start start 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 Processor -- start 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdfd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdfa720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 0x55704bdfacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55704be00660 con 0x55704bdee3a0 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.684+0000 7ff0951ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55704bd0f050 con 0x55704bdfda90 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff0951ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55704be007e0 con 0x55704bdeecc0 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff0951ce740 10 client.0 _ll_register_callbacks cb 0x55704bd4f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff090b91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdfd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff090b91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdfd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37386/0 (socket says 172.21.15.89:37386) 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff090b91640 1 -- 172.21.15.89:0/3246853544 learned_addr learned my addr 172.21.15.89:0/3246853544 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff091392640 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 0x55704bdfacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff08bfff640 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdfa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff090b91640 1 -- 172.21.15.89:0/3246853544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 msgr2=0x55704bdfa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff090b91640 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdfa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff090b91640 1 -- 172.21.15.89:0/3246853544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 msgr2=0x55704bdfacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff090b91640 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 0x55704bdfacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff090b91640 1 -- 172.21.15.89:0/3246853544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55704bac53c0 con 0x55704bdee3a0 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff091392640 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 0x55704bdfacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff090b91640 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdfd470 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7ff080015720 tx=0x7ff0800047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5f5fa37f84cf4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff06bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff06bfff640 1 -- 172.21.15.89:0/3246853544 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff080023020 con 0x55704bdee3a0 2024-09-15T23:34:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff06bfff640 1 -- 172.21.15.89:0/3246853544 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff080013610 con 0x55704bdee3a0 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff0951ce740 1 -- 172.21.15.89:0/3246853544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff084036040 con 0x55704bdee3a0 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff06bfff640 1 -- 172.21.15.89:0/3246853544 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff08002e050 con 0x55704bdee3a0 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff0951ce740 10 client.18915 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.685+0000 7ff0951ce740 1 -- 172.21.15.89:0/3246853544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55704be0c190 con 0x55704bdee3a0 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.686+0000 7ff06bfff640 1 -- 172.21.15.89:0/3246853544 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff084036040 con 0x55704bdee3a0 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.686+0000 7ff06bfff640 1 -- 172.21.15.89:0/3246853544 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff08001e030 con 0x55704bdee3a0 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.686+0000 7ff0951ce740 10 client.18915 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.686+0000 7ff0951ce740 10 client.18915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.686+0000 7ff0951ce740 20 client.18915 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.686+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55704be0d7e0 0x55704be0fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.686+0000 7ff0951ce740 4 client.18915 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.686+0000 7ff0951ce740 1 -- 172.21.15.89:0/3246853544 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55704bd61850 con 0x55704be0d7e0 2024-09-15T23:34:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.686+0000 7ff08bfff640 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55704be0d7e0 0x55704be0fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff08bfff640 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55704be0d7e0 0x55704be0fbd0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=111729f6cb2bd995 server_cookie=e0b602d618165b6 in_seq=0 out_seq=0 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff06bfff640 10 client.18915 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff06bfff640 1 -- 172.21.15.89:0/3246853544 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55704bd61850 con 0x55704be0d7e0 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff06bfff640 10 client.18915 handle_command_reply: tid=0 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff0951ce740 1 client.18915 shutdown 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff0951ce740 1 -- 172.21.15.89:0/3246853544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 msgr2=0x55704bdfd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdfd470 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7ff080015720 tx=0x7ff0800047d0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff0951ce740 1 -- 172.21.15.89:0/3246853544 shutdown_connections 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55704be0d7e0 0x55704be0fbd0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdfda90 0x55704bdfacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.687+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdfa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.688+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3246853544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704bdfd470 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.688+0000 7ff0951ce740 1 -- 172.21.15.89:0/3246853544 >> 172.21.15.89:0/3246853544 conn(0x55704bdeaa30 msgr2=0x55704bdf0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.688+0000 7ff06bfff640 0 client.18915 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:04.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.688+0000 7ff0951ce740 1 -- 172.21.15.89:0/3246853544 shutdown_connections 2024-09-15T23:34:04.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.688+0000 7ff0951ce740 1 -- 172.21.15.89:0/3246853544 wait complete. 2024-09-15T23:34:04.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.688+0000 7ff0951ce740 20 client.18915 trim_cache size 0 max 16384 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff0951ce740 1 Processor -- start 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff0951ce740 1 -- start start 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdeaa30 0x55704be09900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdee3a0 0x55704bdf9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdf05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff0951ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c1acd40 con 0x55704bdeaa30 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff0951ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c1ae880 con 0x55704bdee3a0 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff0951ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c1ad410 con 0x55704bdeecc0 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff090b91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdeaa30 0x55704be09900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff090b91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdeaa30 0x55704be09900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37396/0 (socket says 172.21.15.89:37396) 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff090b91640 1 -- 172.21.15.89:0/4176787499 learned_addr learned my addr 172.21.15.89:0/4176787499 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff08bfff640 1 --2- 172.21.15.89:0/4176787499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdee3a0 0x55704bdf9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff091392640 1 --2- 172.21.15.89:0/4176787499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdf05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff090b91640 1 -- 172.21.15.89:0/4176787499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 msgr2=0x55704bdf05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff090b91640 1 --2- 172.21.15.89:0/4176787499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdf05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff090b91640 1 -- 172.21.15.89:0/4176787499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdee3a0 msgr2=0x55704bdf9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff090b91640 1 --2- 172.21.15.89:0/4176787499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdee3a0 0x55704bdf9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.708+0000 7ff090b91640 1 -- 172.21.15.89:0/4176787499 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55704bdf96d0 con 0x55704bdeaa30 2024-09-15T23:34:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.709+0000 7ff090b91640 1 --2- 172.21.15.89:0/4176787499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdeaa30 0x55704be09900 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7ff0800134d0 tx=0x7ff080006da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c6ea900989d471 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.709+0000 7ff0767fc640 1 -- 172.21.15.89:0/4176787499 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff080023070 con 0x55704bdeaa30 2024-09-15T23:34:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.709+0000 7ff0767fc640 1 -- 172.21.15.89:0/4176787499 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff080037070 con 0x55704bdeaa30 2024-09-15T23:34:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.709+0000 7ff0767fc640 1 -- 172.21.15.89:0/4176787499 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08001b040 con 0x55704bdeaa30 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 -- 172.21.15.89:0/4176787499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdeaa30 msgr2=0x55704be09900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 --2- 172.21.15.89:0/4176787499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdeaa30 0x55704be09900 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7ff0800134d0 tx=0x7ff080006da0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 -- 172.21.15.89:0/4176787499 shutdown_connections 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 --2- 172.21.15.89:0/4176787499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeecc0 0x55704bdf05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 --2- 172.21.15.89:0/4176787499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdee3a0 0x55704bdf9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 --2- 172.21.15.89:0/4176787499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdeaa30 0x55704be09900 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 -- 172.21.15.89:0/4176787499 >> 172.21.15.89:0/4176787499 conn(0x55704bdf6110 msgr2=0x55704bdefed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 -- 172.21.15.89:0/4176787499 shutdown_connections 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 -- 172.21.15.89:0/4176787499 wait complete. 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 -- start start 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 Processor -- start 2024-09-15T23:34:04.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeaa30 0x55704be01a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704be01f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdeecc0 0x55704be02480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c1ae1b0 con 0x55704bdee3a0 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c1adae0 con 0x55704bdeecc0 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.710+0000 7ff0951ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff08c1aad20 con 0x55704bdeaa30 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff0951ce740 10 client.0 _ll_register_callbacks cb 0x7ff080009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff08bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704be01f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff08bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704be01f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37402/0 (socket says 172.21.15.89:37402) 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff08bfff640 1 -- 172.21.15.89:0/3410208751 learned_addr learned my addr 172.21.15.89:0/3410208751 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff090b91640 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeaa30 0x55704be01a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff091392640 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdeecc0 0x55704be02480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff08bfff640 1 -- 172.21.15.89:0/3410208751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeaa30 msgr2=0x55704be01a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:04.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff08bfff640 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeaa30 0x55704be01a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff08bfff640 1 -- 172.21.15.89:0/3410208751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdeecc0 msgr2=0x55704be02480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff08bfff640 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdeecc0 0x55704be02480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff08bfff640 1 -- 172.21.15.89:0/3410208751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55704bdfb060 con 0x55704bdee3a0 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff08bfff640 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704be01f40 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7ff0780982a0 tx=0x7ff07809c090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef4c1fe120666 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff06bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff06bfff640 1 -- 172.21.15.89:0/3410208751 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0780aa040 con 0x55704bdee3a0 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff06bfff640 1 -- 172.21.15.89:0/3410208751 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff078099bd0 con 0x55704bdee3a0 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff0951ce740 1 -- 172.21.15.89:0/3410208751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff080037070 con 0x55704bdee3a0 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff06bfff640 1 -- 172.21.15.89:0/3410208751 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff078021040 con 0x55704bdee3a0 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff0951ce740 10 client.18921 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff0951ce740 1 -- 172.21.15.89:0/3410208751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55704be78610 con 0x55704bdee3a0 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.711+0000 7ff06bfff640 1 -- 172.21.15.89:0/3410208751 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff080037070 con 0x55704bdee3a0 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.712+0000 7ff06bfff640 1 -- 172.21.15.89:0/3410208751 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff0780cf350 con 0x55704bdee3a0 2024-09-15T23:34:04.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.712+0000 7ff0951ce740 10 client.18921 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.712+0000 7ff0951ce740 10 client.18921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.712+0000 7ff0951ce740 20 client.18921 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.712+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55704bdfda90 0x55704be7ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.712+0000 7ff0951ce740 4 client.18921 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.712+0000 7ff0951ce740 1 -- 172.21.15.89:0/3410208751 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55704bd5bd60 con 0x55704bdfda90 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.712+0000 7ff090b91640 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55704bdfda90 0x55704be7ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.712+0000 7ff06bfff640 10 client.18921 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.712+0000 7ff090b91640 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55704bdfda90 0x55704be7ad50 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3658a6f1130e4fb5 server_cookie=57c21e5833a56fb3 in_seq=0 out_seq=0 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff06bfff640 1 -- 172.21.15.89:0/3410208751 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55704bd5bd60 con 0x55704bdfda90 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff06bfff640 10 client.18921 handle_command_reply: tid=0 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 client.18921 shutdown 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 -- 172.21.15.89:0/3410208751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 msgr2=0x55704be01f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:04.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704be01f40 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7ff0780982a0 tx=0x7ff07809c090 comp rx=0 tx=0).stop 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 -- 172.21.15.89:0/3410208751 shutdown_connections 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55704bdfda90 0x55704be7ad50 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55704bdeecc0 0x55704be02480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55704bdee3a0 0x55704be01f40 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 --2- 172.21.15.89:0/3410208751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55704bdeaa30 0x55704be01a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff06bfff640 0 client.18921 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 -- 172.21.15.89:0/3410208751 >> 172.21.15.89:0/3410208751 conn(0x55704bdf6110 msgr2=0x55704bde9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 -- 172.21.15.89:0/3410208751 shutdown_connections 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 1 -- 172.21.15.89:0/3410208751 wait complete. 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.713+0000 7ff0951ce740 20 client.18921 trim_cache size 0 max 16384 2024-09-15T23:34:04.719 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:34:04.720 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:34:04.720 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:34:04.720 INFO:teuthology.orchestra.run.smithi089.stdout: "769d7fcc-b51f-447a-a6c5-083798aaf7c9": { 2024-09-15T23:34:04.720 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:34:04.720 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "769d7fcc-b51f-447a-a6c5-083798aaf7c9", 2024-09-15T23:34:04.720 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:34:04.720 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:34:04.720 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:34:04.720 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:34:04.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 -- 172.21.15.89:0/1597653895 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff07c0a32c0 msgr2=0x7ff07c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:04.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff07c0a32c0 0x7ff07c0a5780 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff080009990 tx=0x7ff080009110 comp rx=0 tx=0).stop 2024-09-15T23:34:04.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 -- 172.21.15.89:0/1597653895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff08c1036e0 msgr2=0x7ff08c198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:04.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff08c1036e0 0x7ff08c198bd0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7ff07800d740 tx=0x7ff07800db40 comp rx=0 tx=0).stop 2024-09-15T23:34:04.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 -- 172.21.15.89:0/1597653895 shutdown_connections 2024-09-15T23:34:04.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff07c0a32c0 0x7ff07c0a5780 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff08c1082a0 0x7ff08c19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff08c1036e0 0x7ff08c198bd0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 --2- 172.21.15.89:0/1597653895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff08c102d30 0x7ff08c198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:04.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 -- 172.21.15.89:0/1597653895 >> 172.21.15.89:0/1597653895 conn(0x7ff08c0fc7d0 msgr2=0x7ff08c0fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:04.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 -- 172.21.15.89:0/1597653895 shutdown_connections 2024-09-15T23:34:04.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:04.714+0000 7ff092df5640 1 -- 172.21.15.89:0/1597653895 wait complete. 2024-09-15T23:34:05.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:05 smithi089 ceph-mon[30933]: pgmap v651: 129 pgs: 129 active+clean; 809 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-09-15T23:34:05.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:05.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:05.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:34:05.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:34:05.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:05 smithi120 ceph-mon[35250]: pgmap v651: 129 pgs: 129 active+clean; 809 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-09-15T23:34:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:34:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:34:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:05 smithi174 ceph-mon[35586]: pgmap v651: 129 pgs: 129 active+clean; 809 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-09-15T23:34:05.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:05.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:05.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:34:05.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:34:05.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:34:05.829 INFO:tasks.cephfs.filesystem:scrub status for tag:769d7fcc-b51f-447a-a6c5-083798aaf7c9 - {'path': '/', 'tag': '769d7fcc-b51f-447a-a6c5-083798aaf7c9', 'options': 'recursive,force'} 2024-09-15T23:34:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.087+0000 7fbb79277640 1 -- 172.21.15.89:0/1916164517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb74104f00 msgr2=0x7fbb741072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.087+0000 7fbb79277640 1 --2- 172.21.15.89:0/1916164517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb74104f00 0x7fbb741072f0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009900 tx=0x7fbb6802a410 comp rx=0 tx=0).stop 2024-09-15T23:34:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.087+0000 7fbb79277640 1 -- 172.21.15.89:0/1916164517 shutdown_connections 2024-09-15T23:34:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.087+0000 7fbb79277640 1 --2- 172.21.15.89:0/1916164517 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb74107830 0x7fbb74109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.087+0000 7fbb79277640 1 --2- 172.21.15.89:0/1916164517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb74104f00 0x7fbb741072f0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.087+0000 7fbb79277640 1 --2- 172.21.15.89:0/1916164517 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb741025d0 0x7fbb741049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.087+0000 7fbb79277640 1 -- 172.21.15.89:0/1916164517 >> 172.21.15.89:0/1916164517 conn(0x7fbb740fc1b0 msgr2=0x7fbb740fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.087+0000 7fbb79277640 1 -- 172.21.15.89:0/1916164517 shutdown_connections 2024-09-15T23:34:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.087+0000 7fbb79277640 1 -- 172.21.15.89:0/1916164517 wait complete. 2024-09-15T23:34:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.087+0000 7fbb79277640 1 Processor -- start 2024-09-15T23:34:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb79277640 1 -- start start 2024-09-15T23:34:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb79277640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb741025d0 0x7fbb74101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb79277640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb74104f00 0x7fbb74100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb79277640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb74107830 0x7fbb74100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb79277640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7410d810 con 0x7fbb74107830 2024-09-15T23:34:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb79277640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7410d690 con 0x7fbb741025d0 2024-09-15T23:34:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb79277640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7410d990 con 0x7fbb74104f00 2024-09-15T23:34:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb74107830 0x7fbb74100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb74107830 0x7fbb74100ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56402/0 (socket says 172.21.15.89:56402) 2024-09-15T23:34:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb737fe640 1 -- 172.21.15.89:0/4203136533 learned_addr learned my addr 172.21.15.89:0/4203136533 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb727fc640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb74104f00 0x7fbb74100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb741025d0 0x7fbb74101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb737fe640 1 -- 172.21.15.89:0/4203136533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb74104f00 msgr2=0x7fbb74100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb737fe640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb74104f00 0x7fbb74100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb737fe640 1 -- 172.21.15.89:0/4203136533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb741025d0 msgr2=0x7fbb74101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb737fe640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb741025d0 0x7fbb74101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb737fe640 1 -- 172.21.15.89:0/4203136533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb74101350 con 0x7fbb74107830 2024-09-15T23:34:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb737fe640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb74107830 0x7fbb74100ac0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00d720 tx=0x7fbb6c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.088+0000 7fbb5bfff640 1 -- 172.21.15.89:0/4203136533 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c017070 con 0x7fbb74107830 2024-09-15T23:34:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.089+0000 7fbb5bfff640 1 -- 172.21.15.89:0/4203136533 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb6c00de30 con 0x7fbb74107830 2024-09-15T23:34:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.089+0000 7fbb5bfff640 1 -- 172.21.15.89:0/4203136533 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c004d90 con 0x7fbb74107830 2024-09-15T23:34:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.089+0000 7fbb79277640 1 -- 172.21.15.89:0/4203136533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb741a1f60 con 0x7fbb74107830 2024-09-15T23:34:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.089+0000 7fbb79277640 1 -- 172.21.15.89:0/4203136533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb741a23c0 con 0x7fbb74107830 2024-09-15T23:34:06.101 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.091+0000 7fbb5bfff640 1 -- 172.21.15.89:0/4203136533 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbb6c004f30 con 0x7fbb74107830 2024-09-15T23:34:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.091+0000 7fbb7c010740 1 Processor -- start 2024-09-15T23:34:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.091+0000 7fbb7c010740 1 -- start start 2024-09-15T23:34:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8539a0 0x55b95e853da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8542e0 0x55b95e858da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e859370 0x55b95e85b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb7c010740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b95e7b72f0 con 0x55b95e8542e0 2024-09-15T23:34:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb7c010740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b95e75f270 con 0x55b95e859370 2024-09-15T23:34:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb7c010740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b95e7b32b0 con 0x55b95e8539a0 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb72ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8542e0 0x55b95e858da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb72ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8542e0 0x55b95e858da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56406/0 (socket says 172.21.15.89:56406) 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb72ffd640 1 -- 172.21.15.89:0/856102850 learned_addr learned my addr 172.21.15.89:0/856102850 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb727fc640 1 --2- 172.21.15.89:0/856102850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8539a0 0x55b95e853da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb727fc640 1 --2- 172.21.15.89:0/856102850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e859370 0x55b95e85b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb5bfff640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb5c0a32f0 0x7fbb5c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb72ffd640 1 -- 172.21.15.89:0/856102850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8539a0 msgr2=0x55b95e853da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/856102850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8539a0 0x55b95e853da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb72ffd640 1 -- 172.21.15.89:0/856102850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e859370 msgr2=0x55b95e85b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/856102850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e859370 0x55b95e85b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb72ffd640 1 -- 172.21.15.89:0/856102850 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b95e75e340 con 0x55b95e8542e0 2024-09-15T23:34:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb737fe640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb5c0a32f0 0x7fbb5c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb5bfff640 1 -- 172.21.15.89:0/4203136533 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb6c0cf680 con 0x7fbb74107830 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/856102850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8542e0 0x55b95e858da0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009f60 tx=0x7fbb64009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835f1e1ba5e918bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.092+0000 7fbb597fa640 1 -- 172.21.15.89:0/856102850 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64016070 con 0x55b95e8542e0 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb597fa640 1 -- 172.21.15.89:0/856102850 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb6401bb40 con 0x55b95e8542e0 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb597fa640 1 -- 172.21.15.89:0/856102850 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64012cf0 con 0x55b95e8542e0 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb7c010740 1 -- 172.21.15.89:0/856102850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8542e0 msgr2=0x55b95e858da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb737fe640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb5c0a32f0 0x7fbb5c0a57b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c09b5b0 tx=0x7fbb6c090d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb7c010740 1 --2- 172.21.15.89:0/856102850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8542e0 0x55b95e858da0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009f60 tx=0x7fbb64009f90 comp rx=0 tx=0).stop 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb7c010740 1 -- 172.21.15.89:0/856102850 shutdown_connections 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb7c010740 1 --2- 172.21.15.89:0/856102850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e859370 0x55b95e85b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb7c010740 1 --2- 172.21.15.89:0/856102850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8542e0 0x55b95e858da0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb7c010740 1 --2- 172.21.15.89:0/856102850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8539a0 0x55b95e853da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb7c010740 1 -- 172.21.15.89:0/856102850 >> 172.21.15.89:0/856102850 conn(0x55b95e84f010 msgr2=0x55b95e84f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb7c010740 1 -- 172.21.15.89:0/856102850 shutdown_connections 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.093+0000 7fbb7c010740 1 -- 172.21.15.89:0/856102850 wait complete. 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb7c010740 1 -- start start 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb7c010740 1 Processor -- start 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e85dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 0x55b95e85e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e859370 0x55b95e865170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb7c010740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b95e85c340 con 0x55b95e8539a0 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb7c010740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b95e740290 con 0x55b95e8542e0 2024-09-15T23:34:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb7c010740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b95e85c4c0 con 0x55b95e859370 2024-09-15T23:34:06.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb7c010740 10 client.0 _ll_register_callbacks cb 0x55b95e771140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:06.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb727fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e85dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb727fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e85dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56420/0 (socket says 172.21.15.89:56420) 2024-09-15T23:34:06.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb727fc640 1 -- 172.21.15.89:0/2160128686 learned_addr learned my addr 172.21.15.89:0/2160128686 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:06.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 0x55b95e85e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb727fc640 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e859370 0x55b95e865170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb727fc640 1 -- 172.21.15.89:0/2160128686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e859370 msgr2=0x55b95e865170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:06.106 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb727fc640 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e859370 0x55b95e865170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb727fc640 1 -- 172.21.15.89:0/2160128686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 msgr2=0x55b95e85e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb727fc640 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 0x55b95e85e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.094+0000 7fbb727fc640 1 -- 172.21.15.89:0/2160128686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b95e4ca210 con 0x55b95e8539a0 2024-09-15T23:34:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb727fc640 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e85dbf0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009900 tx=0x7fbb68040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc25cba5ba259e2b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb56ff5640 1 -- 172.21.15.89:0/2160128686 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68035040 con 0x55b95e8539a0 2024-09-15T23:34:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb56ff5640 1 -- 172.21.15.89:0/2160128686 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb68004480 con 0x55b95e8539a0 2024-09-15T23:34:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb7c010740 1 -- 172.21.15.89:0/2160128686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6401bb40 con 0x55b95e8539a0 2024-09-15T23:34:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb56ff5640 1 -- 172.21.15.89:0/2160128686 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb68052050 con 0x55b95e8539a0 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb7c010740 10 client.18933 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb7c010740 1 -- 172.21.15.89:0/2160128686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b95e596620 con 0x55b95e8539a0 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb56ff5640 1 -- 172.21.15.89:0/2160128686 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb6401bb40 con 0x55b95e8539a0 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb56ff5640 1 -- 172.21.15.89:0/2160128686 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbb6803d790 con 0x55b95e8539a0 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb7c010740 10 client.18933 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb7c010740 10 client.18933 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb7c010740 20 client.18933 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb7c010740 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b95e869810 0x55b95e86bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb7c010740 4 client.18933 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.095+0000 7fbb7c010740 1 -- 172.21.15.89:0/2160128686 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b95e7c4a20 con 0x55b95e869810 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.096+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b95e869810 0x55b95e86bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.096+0000 7fbb56ff5640 10 client.18933 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.096+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b95e869810 0x55b95e86bc60 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e558f465d0ca533a server_cookie=f4f18d7c693d68e0 in_seq=0 out_seq=0 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb56ff5640 1 -- 172.21.15.89:0/2160128686 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b95e7c4a20 con 0x55b95e869810 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb56ff5640 10 client.18933 handle_command_reply: tid=0 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 client.18933 shutdown 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 -- 172.21.15.89:0/2160128686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 msgr2=0x55b95e85dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e85dbf0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009900 tx=0x7fbb68040040 comp rx=0 tx=0).stop 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 -- 172.21.15.89:0/2160128686 shutdown_connections 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b95e869810 0x55b95e86bc60 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e859370 0x55b95e865170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 0x55b95e85e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb56ff5640 0 client.18933 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 --2- 172.21.15.89:0/2160128686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e85dbf0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 -- 172.21.15.89:0/2160128686 >> 172.21.15.89:0/2160128686 conn(0x55b95e84f010 msgr2=0x55b95e850820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 -- 172.21.15.89:0/2160128686 shutdown_connections 2024-09-15T23:34:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 1 -- 172.21.15.89:0/2160128686 wait complete. 2024-09-15T23:34:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.097+0000 7fbb7c010740 20 client.18933 trim_cache size 0 max 16384 2024-09-15T23:34:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.111+0000 7fbb7c010740 1 Processor -- start 2024-09-15T23:34:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.111+0000 7fbb7c010740 1 -- start start 2024-09-15T23:34:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.111+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e84e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.111+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8542e0 0x55b95e865130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.111+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e859370 0x55b95e870560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.111+0000 7fbb7c010740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741acb90 con 0x55b95e8539a0 2024-09-15T23:34:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.111+0000 7fbb7c010740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741ae6d0 con 0x55b95e859370 2024-09-15T23:34:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.111+0000 7fbb7c010740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741ad260 con 0x55b95e8542e0 2024-09-15T23:34:06.119 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb727fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e84e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb727fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e84e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56424/0 (socket says 172.21.15.89:56424) 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb727fc640 1 -- 172.21.15.89:0/3602317710 learned_addr learned my addr 172.21.15.89:0/3602317710 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/3602317710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8542e0 0x55b95e865130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb727fc640 1 -- 172.21.15.89:0/3602317710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8542e0 msgr2=0x55b95e865130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb727fc640 1 --2- 172.21.15.89:0/3602317710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8542e0 0x55b95e865130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb727fc640 1 -- 172.21.15.89:0/3602317710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e859370 msgr2=0x55b95e870560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb727fc640 1 --2- 172.21.15.89:0/3602317710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e859370 0x55b95e870560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb727fc640 1 -- 172.21.15.89:0/3602317710 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b95e85c580 con 0x55b95e8539a0 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb727fc640 1 --2- 172.21.15.89:0/3602317710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e84e0b0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009230 tx=0x7fbb68005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742438b61b7f45fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb58ff9640 1 -- 172.21.15.89:0/3602317710 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68035050 con 0x55b95e8539a0 2024-09-15T23:34:06.120 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb58ff9640 1 -- 172.21.15.89:0/3602317710 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb6803d930 con 0x55b95e8539a0 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.112+0000 7fbb58ff9640 1 -- 172.21.15.89:0/3602317710 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6804c870 con 0x55b95e8539a0 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 -- 172.21.15.89:0/3602317710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 msgr2=0x55b95e84e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 --2- 172.21.15.89:0/3602317710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e84e0b0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009230 tx=0x7fbb68005a60 comp rx=0 tx=0).stop 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 -- 172.21.15.89:0/3602317710 shutdown_connections 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 --2- 172.21.15.89:0/3602317710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e859370 0x55b95e870560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 --2- 172.21.15.89:0/3602317710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8542e0 0x55b95e865130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 --2- 172.21.15.89:0/3602317710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e8539a0 0x55b95e84e0b0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 -- 172.21.15.89:0/3602317710 >> 172.21.15.89:0/3602317710 conn(0x55b95e84f010 msgr2=0x55b95e84d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 -- 172.21.15.89:0/3602317710 shutdown_connections 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 -- 172.21.15.89:0/3602317710 wait complete. 2024-09-15T23:34:06.121 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 -- start start 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 Processor -- start 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8539a0 0x55b95e864920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 0x55b95e864e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e859370 0x55b95e862f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741ae000 con 0x55b95e859370 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741ad930 con 0x55b95e8542e0 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.113+0000 7fbb7c010740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741aab70 con 0x55b95e8539a0 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb7c010740 10 client.0 _ll_register_callbacks cb 0x7fbb68009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb727fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e859370 0x55b95e862f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb727fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e859370 0x55b95e862f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56434/0 (socket says 172.21.15.89:56434) 2024-09-15T23:34:06.122 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb727fc640 1 -- 172.21.15.89:0/1244132702 learned_addr learned my addr 172.21.15.89:0/1244132702 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 0x55b95e864e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb727fc640 1 -- 172.21.15.89:0/1244132702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8539a0 msgr2=0x55b95e864920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb727fc640 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8539a0 0x55b95e864920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb727fc640 1 -- 172.21.15.89:0/1244132702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 msgr2=0x55b95e864e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb727fc640 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 0x55b95e864e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb727fc640 1 -- 172.21.15.89:0/1244132702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b95e8635a0 con 0x55b95e859370 2024-09-15T23:34:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 0x55b95e864e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb727fc640 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e859370 0x55b95e862f30 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fbb68031610 tx=0x7fbb6804c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d16b89b690bcf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:06.123 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb56ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb56ff5640 1 -- 172.21.15.89:0/1244132702 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68035070 con 0x55b95e859370 2024-09-15T23:34:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb56ff5640 1 -- 172.21.15.89:0/1244132702 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb68004480 con 0x55b95e859370 2024-09-15T23:34:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb7c010740 1 -- 172.21.15.89:0/1244132702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6803d930 con 0x55b95e859370 2024-09-15T23:34:06.124 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb56ff5640 1 -- 172.21.15.89:0/1244132702 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb6803d450 con 0x55b95e859370 2024-09-15T23:34:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb7c010740 10 client.18939 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.114+0000 7fbb7c010740 1 -- 172.21.15.89:0/1244132702 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b95e8d4eb0 con 0x55b95e859370 2024-09-15T23:34:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb56ff5640 1 -- 172.21.15.89:0/1244132702 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb6803d930 con 0x55b95e859370 2024-09-15T23:34:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb56ff5640 1 -- 172.21.15.89:0/1244132702 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbb680036f0 con 0x55b95e859370 2024-09-15T23:34:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb7c010740 10 client.18939 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb7c010740 10 client.18939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:06.125 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb7c010740 20 client.18939 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb7c010740 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b95e8d5750 0x55b95e8d7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb7c010740 4 client.18939 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:34:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb7c010740 1 -- 172.21.15.89:0/1244132702 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b95e7bed70 con 0x55b95e8d5750 2024-09-15T23:34:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b95e8d5750 0x55b95e8d7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:06.126 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb72ffd640 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b95e8d5750 0x55b95e8d7b70 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f267e4debc2a103f server_cookie=8cd6f16cb3aeaadf in_seq=0 out_seq=0 2024-09-15T23:34:06.127 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.115+0000 7fbb56ff5640 10 client.18939 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb56ff5640 1 -- 172.21.15.89:0/1244132702 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b95e7bed70 con 0x55b95e8d5750 2024-09-15T23:34:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb56ff5640 10 client.18939 handle_command_reply: tid=0 2024-09-15T23:34:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb7c010740 1 client.18939 shutdown 2024-09-15T23:34:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb7c010740 1 -- 172.21.15.89:0/1244132702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e859370 msgr2=0x55b95e862f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:06.129 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb7c010740 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e859370 0x55b95e862f30 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fbb68031610 tx=0x7fbb6804c210 comp rx=0 tx=0).stop 2024-09-15T23:34:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb7c010740 1 -- 172.21.15.89:0/1244132702 shutdown_connections 2024-09-15T23:34:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb7c010740 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b95e8d5750 0x55b95e8d7b70 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb7c010740 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b95e859370 0x55b95e862f30 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb56ff5640 0 client.18939 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:06.130 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb7c010740 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b95e8542e0 0x55b95e864e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb7c010740 1 --2- 172.21.15.89:0/1244132702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b95e8539a0 0x55b95e864920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.124+0000 7fbb7c010740 1 -- 172.21.15.89:0/1244132702 >> 172.21.15.89:0/1244132702 conn(0x55b95e84f010 msgr2=0x55b95e84e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb7c010740 1 -- 172.21.15.89:0/1244132702 shutdown_connections 2024-09-15T23:34:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb7c010740 1 -- 172.21.15.89:0/1244132702 wait complete. 2024-09-15T23:34:06.131 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb7c010740 20 client.18939 trim_cache size 0 max 16384 2024-09-15T23:34:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:34:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:34:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:34:06.132 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:34:06.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 -- 172.21.15.89:0/4203136533 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb5c0a32f0 msgr2=0x7fbb5c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:06.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb5c0a32f0 0x7fbb5c0a57b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c09b5b0 tx=0x7fbb6c090d90 comp rx=0 tx=0).stop 2024-09-15T23:34:06.134 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 -- 172.21.15.89:0/4203136533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb74107830 msgr2=0x7fbb74100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb74107830 0x7fbb74100ac0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00d720 tx=0x7fbb6c00dbe0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 -- 172.21.15.89:0/4203136533 shutdown_connections 2024-09-15T23:34:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb5c0a32f0 0x7fbb5c0a57b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb74107830 0x7fbb74100ac0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb74104f00 0x7fbb74100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 --2- 172.21.15.89:0/4203136533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb741025d0 0x7fbb74101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 -- 172.21.15.89:0/4203136533 >> 172.21.15.89:0/4203136533 conn(0x7fbb740fc1b0 msgr2=0x7fbb74102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 -- 172.21.15.89:0/4203136533 shutdown_connections 2024-09-15T23:34:06.135 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:06.125+0000 7fbb79277640 1 -- 172.21.15.89:0/4203136533 wait complete. 2024-09-15T23:34:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:07 smithi120 ceph-mon[35250]: pgmap v652: 129 pgs: 129 active+clean; 809 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2024-09-15T23:34:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:07 smithi174 ceph-mon[35586]: pgmap v652: 129 pgs: 129 active+clean; 809 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2024-09-15T23:34:07.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:07 smithi089 ceph-mon[30933]: pgmap v652: 129 pgs: 129 active+clean; 809 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2024-09-15T23:34:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:09 smithi120 ceph-mon[35250]: pgmap v653: 129 pgs: 129 active+clean; 867 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-15T23:34:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:09 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:34:09.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:09 smithi174 ceph-mon[35586]: pgmap v653: 129 pgs: 129 active+clean; 867 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-15T23:34:09.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:09 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:34:09.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:09 smithi089 ceph-mon[30933]: pgmap v653: 129 pgs: 129 active+clean; 867 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-15T23:34:09.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:09 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:34:11.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:11 smithi120 ceph-mon[35250]: pgmap v654: 129 pgs: 129 active+clean; 867 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:34:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:11 smithi174 ceph-mon[35586]: pgmap v654: 129 pgs: 129 active+clean; 867 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:34:11.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:11 smithi089 ceph-mon[30933]: pgmap v654: 129 pgs: 129 active+clean; 867 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:34:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:13 smithi120 ceph-mon[35250]: pgmap v655: 129 pgs: 129 active+clean; 867 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-15T23:34:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:13 smithi174 ceph-mon[35586]: pgmap v655: 129 pgs: 129 active+clean; 867 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-15T23:34:13.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:13 smithi089 ceph-mon[30933]: pgmap v655: 129 pgs: 129 active+clean; 867 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-09-15T23:34:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:15 smithi120 ceph-mon[35250]: pgmap v656: 129 pgs: 129 active+clean; 923 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-09-15T23:34:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:15 smithi174 ceph-mon[35586]: pgmap v656: 129 pgs: 129 active+clean; 923 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-09-15T23:34:15.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:15 smithi089 ceph-mon[30933]: pgmap v656: 129 pgs: 129 active+clean; 923 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-09-15T23:34:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:17 smithi120 ceph-mon[35250]: pgmap v657: 129 pgs: 129 active+clean; 923 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:34:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:17 smithi174 ceph-mon[35586]: pgmap v657: 129 pgs: 129 active+clean; 923 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:34:17.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:17 smithi089 ceph-mon[30933]: pgmap v657: 129 pgs: 129 active+clean; 923 KiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:34:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.221+0000 7fb4105b7640 1 -- 172.21.15.89:0/2029933493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb408101fe0 msgr2=0x7fb4081023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.221+0000 7fb4105b7640 1 --2- 172.21.15.89:0/2029933493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb408101fe0 0x7fb4081023e0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fb404009900 tx=0x7fb40402a410 comp rx=0 tx=0).stop 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.221+0000 7fb4105b7640 1 -- 172.21.15.89:0/2029933493 shutdown_connections 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.221+0000 7fb4105b7640 1 --2- 172.21.15.89:0/2029933493 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4081016d0 0x7fb408108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.221+0000 7fb4105b7640 1 --2- 172.21.15.89:0/2029933493 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408100d10 0x7fb408101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.221+0000 7fb4105b7640 1 --2- 172.21.15.89:0/2029933493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb408101fe0 0x7fb4081023e0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.221+0000 7fb4105b7640 1 -- 172.21.15.89:0/2029933493 >> 172.21.15.89:0/2029933493 conn(0x7fb4080fb970 msgr2=0x7fb4080fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.221+0000 7fb4105b7640 1 -- 172.21.15.89:0/2029933493 shutdown_connections 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.221+0000 7fb4105b7640 1 -- 172.21.15.89:0/2029933493 wait complete. 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb4105b7640 1 Processor -- start 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb4105b7640 1 -- start start 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb4105b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408100d10 0x7fb408197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb4105b7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4081016d0 0x7fb408198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb4105b7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb408101fe0 0x7fb40819f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb4105b7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb40810d8d0 con 0x7fb4081016d0 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb4105b7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb40810d750 con 0x7fb408101fe0 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb4105b7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb40810da50 con 0x7fb408100d10 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40db52640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4081016d0 0x7fb408198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40db52640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4081016d0 0x7fb408198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33920/0 (socket says 172.21.15.89:33920) 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40db52640 1 -- 172.21.15.89:0/1028907302 learned_addr learned my addr 172.21.15.89:0/1028907302 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40e353640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408100d10 0x7fb408197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40eb54640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb408101fe0 0x7fb40819f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40db52640 1 -- 172.21.15.89:0/1028907302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408100d10 msgr2=0x7fb408197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40db52640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408100d10 0x7fb408197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40db52640 1 -- 172.21.15.89:0/1028907302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb408101fe0 msgr2=0x7fb40819f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:19.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40db52640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb408101fe0 0x7fb40819f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40db52640 1 -- 172.21.15.89:0/1028907302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40819fb70 con 0x7fb4081016d0 2024-09-15T23:34:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40e353640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408100d10 0x7fb408197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.222+0000 7fb40eb54640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb408101fe0 0x7fb40819f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.223+0000 7fb40db52640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4081016d0 0x7fb408198490 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00b4a0 tx=0x7fb3fc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.223+0000 7fb3fb7fe640 1 -- 172.21.15.89:0/1028907302 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc014070 con 0x7fb4081016d0 2024-09-15T23:34:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.223+0000 7fb4105b7640 1 -- 172.21.15.89:0/1028907302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40819fe60 con 0x7fb4081016d0 2024-09-15T23:34:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.223+0000 7fb3fb7fe640 1 -- 172.21.15.89:0/1028907302 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3fc010af0 con 0x7fb4081016d0 2024-09-15T23:34:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.223+0000 7fb3fb7fe640 1 -- 172.21.15.89:0/1028907302 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc017d10 con 0x7fb4081016d0 2024-09-15T23:34:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.223+0000 7fb4105b7640 1 -- 172.21.15.89:0/1028907302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4081a03a0 con 0x7fb4081016d0 2024-09-15T23:34:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.226+0000 7fb3fb7fe640 1 -- 172.21.15.89:0/1028907302 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb3fc017070 con 0x7fb4081016d0 2024-09-15T23:34:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.226+0000 7fb41342b740 1 Processor -- start 2024-09-15T23:34:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.226+0000 7fb41342b740 1 -- start start 2024-09-15T23:34:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb41342b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b803fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb41342b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b80402e0 0x55b0b8044da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb41342b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b8047830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb41342b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b0b7fa32f0 con 0x55b0b8045370 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb41342b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b0b7f4b270 con 0x55b0b80402e0 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb41342b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b0b7f9f2b0 con 0x55b0b803f9a0 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40db52640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b8047830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40db52640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b8047830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33924/0 (socket says 172.21.15.89:33924) 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40db52640 1 -- 172.21.15.89:0/4051854181 learned_addr learned my addr 172.21.15.89:0/4051854181 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40e353640 1 --2- 172.21.15.89:0/4051854181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b80402e0 0x55b0b8044da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40eb54640 1 --2- 172.21.15.89:0/4051854181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b803fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40db52640 1 -- 172.21.15.89:0/4051854181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 msgr2=0x55b0b803fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40db52640 1 --2- 172.21.15.89:0/4051854181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b803fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40db52640 1 -- 172.21.15.89:0/4051854181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b80402e0 msgr2=0x55b0b8044da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40db52640 1 --2- 172.21.15.89:0/4051854181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b80402e0 0x55b0b8044da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40db52640 1 -- 172.21.15.89:0/4051854181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0b7f4a340 con 0x55b0b8045370 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb3fb7fe640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3f40a32f0 0x7fb3f40a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40eb54640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3f40a32f0 0x7fb3f40a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.227+0000 7fb40db52640 1 --2- 172.21.15.89:0/4051854181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b8047830 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0d7480 tx=0x7fb3fc0d7940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77cf57eb3beeafb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.228+0000 7fb3fb7fe640 1 -- 172.21.15.89:0/1028907302 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb3fc0cf110 con 0x7fb4081016d0 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.228+0000 7fb3f8bfe640 1 -- 172.21.15.89:0/4051854181 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc01e030 con 0x55b0b8045370 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.228+0000 7fb3f8bfe640 1 -- 172.21.15.89:0/4051854181 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3fc09ba40 con 0x55b0b8045370 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.228+0000 7fb3f8bfe640 1 -- 172.21.15.89:0/4051854181 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc09b110 con 0x55b0b8045370 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb40eb54640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3f40a32f0 0x7fb3f40a57b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb40000c610 tx=0x7fb40000e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 -- 172.21.15.89:0/4051854181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 msgr2=0x55b0b8047830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 --2- 172.21.15.89:0/4051854181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b8047830 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0d7480 tx=0x7fb3fc0d7940 comp rx=0 tx=0).stop 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 -- 172.21.15.89:0/4051854181 shutdown_connections 2024-09-15T23:34:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 --2- 172.21.15.89:0/4051854181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b8047830 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 --2- 172.21.15.89:0/4051854181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b80402e0 0x55b0b8044da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 --2- 172.21.15.89:0/4051854181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b803fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 -- 172.21.15.89:0/4051854181 >> 172.21.15.89:0/4051854181 conn(0x55b0b803b010 msgr2=0x55b0b803b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 -- 172.21.15.89:0/4051854181 shutdown_connections 2024-09-15T23:34:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 -- 172.21.15.89:0/4051854181 wait complete. 2024-09-15T23:34:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 -- start start 2024-09-15T23:34:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 Processor -- start 2024-09-15T23:34:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b8049b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 0x55b0b8051080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b0b8048340 con 0x55b0b80402e0 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b0b7f2c290 con 0x55b0b8045370 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b0b80484c0 con 0x55b0b803f9a0 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb41342b740 10 client.0 _ll_register_callbacks cb 0x55b0b7f5d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.229+0000 7fb40db52640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40db52640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33930/0 (socket says 172.21.15.89:33930) 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40db52640 1 -- 172.21.15.89:0/1786947611 learned_addr learned my addr 172.21.15.89:0/1786947611 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40e353640 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b8049b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40eb54640 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 0x55b0b8051080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40db52640 1 -- 172.21.15.89:0/1786947611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 msgr2=0x55b0b8049b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40db52640 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b8049b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40db52640 1 -- 172.21.15.89:0/1786947611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 msgr2=0x55b0b8051080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40db52640 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 0x55b0b8051080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40db52640 1 -- 172.21.15.89:0/1786947611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0b7cb6210 con 0x55b0b80402e0 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40e353640 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b8049b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40eb54640 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 0x55b0b8051080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb40db52640 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804a040 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0ccb90 tx=0x7fb3fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70c4d7f753a126e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb3f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/1786947611 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc01e030 con 0x55b0b80402e0 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/1786947611 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb3fc010ca0 con 0x55b0b80402e0 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb41342b740 1 -- 172.21.15.89:0/1786947611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3fc09ba40 con 0x55b0b80402e0 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/1786947611 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb3fc09aca0 con 0x55b0b80402e0 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb41342b740 10 client.18951 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb41342b740 1 -- 172.21.15.89:0/1786947611 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b0b7d82620 con 0x55b0b80402e0 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.230+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/1786947611 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3fc09ba40 con 0x55b0b80402e0 2024-09-15T23:34:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.231+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/1786947611 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb3fc0ccce0 con 0x55b0b80402e0 2024-09-15T23:34:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.231+0000 7fb41342b740 10 client.18951 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.231+0000 7fb41342b740 10 client.18951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.231+0000 7fb41342b740 20 client.18951 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.231+0000 7fb41342b740 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b0b8055380 0x55b0b8057770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.231+0000 7fb41342b740 4 client.18951 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.231+0000 7fb41342b740 1 -- 172.21.15.89:0/1786947611 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0b7fb0a20 con 0x55b0b8055380 2024-09-15T23:34:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.231+0000 7fb40e353640 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b0b8055380 0x55b0b8057770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb3f1ffb640 10 client.18951 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb40e353640 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b0b8055380 0x55b0b8057770 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2b77712eaf996f1 server_cookie=cde5367acf26143e in_seq=0 out_seq=0 2024-09-15T23:34:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/1786947611 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b0b7fb0a20 con 0x55b0b8055380 2024-09-15T23:34:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb3f1ffb640 10 client.18951 handle_command_reply: tid=0 2024-09-15T23:34:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb41342b740 1 client.18951 shutdown 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb41342b740 1 -- 172.21.15.89:0/1786947611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 msgr2=0x55b0b804a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb41342b740 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804a040 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0ccb90 tx=0x7fb3fc0046c0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb41342b740 1 -- 172.21.15.89:0/1786947611 shutdown_connections 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb41342b740 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b0b8055380 0x55b0b8057770 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb41342b740 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 0x55b0b8051080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb41342b740 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804a040 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb3f1ffb640 0 client.18951 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb41342b740 1 --2- 172.21.15.89:0/1786947611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b8049b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.232+0000 7fb41342b740 1 -- 172.21.15.89:0/1786947611 >> 172.21.15.89:0/1786947611 conn(0x55b0b803b010 msgr2=0x55b0b803c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.233+0000 7fb41342b740 1 -- 172.21.15.89:0/1786947611 shutdown_connections 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.233+0000 7fb41342b740 1 -- 172.21.15.89:0/1786947611 wait complete. 2024-09-15T23:34:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.233+0000 7fb41342b740 20 client.18951 trim_cache size 0 max 16384 2024-09-15T23:34:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb41342b740 1 Processor -- start 2024-09-15T23:34:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb41342b740 1 -- start start 2024-09-15T23:34:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb41342b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b803c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb41342b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b80402e0 0x55b0b8051040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb41342b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b803c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb41342b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081aaac0 con 0x55b0b8045370 2024-09-15T23:34:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb41342b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081ac600 con 0x55b0b80402e0 2024-09-15T23:34:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb41342b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081ab190 con 0x55b0b803f9a0 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb40eb54640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b803c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb40eb54640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b803c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33936/0 (socket says 172.21.15.89:33936) 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb40eb54640 1 -- 172.21.15.89:0/190017809 learned_addr learned my addr 172.21.15.89:0/190017809 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb40db52640 1 --2- 172.21.15.89:0/190017809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b80402e0 0x55b0b8051040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb40e353640 1 --2- 172.21.15.89:0/190017809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b803c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb40eb54640 1 -- 172.21.15.89:0/190017809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 msgr2=0x55b0b803c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb40eb54640 1 --2- 172.21.15.89:0/190017809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b803c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb40eb54640 1 -- 172.21.15.89:0/190017809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b80402e0 msgr2=0x55b0b8051040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb40eb54640 1 --2- 172.21.15.89:0/190017809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b80402e0 0x55b0b8051040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.247+0000 7fb40eb54640 1 -- 172.21.15.89:0/190017809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0b803ae30 con 0x55b0b8045370 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb40eb54640 1 --2- 172.21.15.89:0/190017809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b803c940 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fb400009b20 tx=0x7fb400014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab5e7749e05c165d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb3f3fff640 1 -- 172.21.15.89:0/190017809 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40001e050 con 0x55b0b8045370 2024-09-15T23:34:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb3f3fff640 1 -- 172.21.15.89:0/190017809 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb400028070 con 0x55b0b8045370 2024-09-15T23:34:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb3f3fff640 1 -- 172.21.15.89:0/190017809 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4000163c0 con 0x55b0b8045370 2024-09-15T23:34:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb41342b740 1 -- 172.21.15.89:0/190017809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 msgr2=0x55b0b803c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb41342b740 1 --2- 172.21.15.89:0/190017809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b803c940 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fb400009b20 tx=0x7fb400014000 comp rx=0 tx=0).stop 2024-09-15T23:34:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb41342b740 1 -- 172.21.15.89:0/190017809 shutdown_connections 2024-09-15T23:34:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb41342b740 1 --2- 172.21.15.89:0/190017809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b8045370 0x55b0b803c940 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb41342b740 1 --2- 172.21.15.89:0/190017809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b80402e0 0x55b0b8051040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb41342b740 1 --2- 172.21.15.89:0/190017809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b803c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb41342b740 1 -- 172.21.15.89:0/190017809 >> 172.21.15.89:0/190017809 conn(0x55b0b803b010 msgr2=0x55b0b803a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.248+0000 7fb41342b740 1 -- 172.21.15.89:0/190017809 shutdown_connections 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb41342b740 1 -- 172.21.15.89:0/190017809 wait complete. 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb41342b740 1 -- start start 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb41342b740 1 Processor -- start 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb41342b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b804e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb41342b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb41342b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 0x55b0b804f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb41342b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081abf30 con 0x55b0b80402e0 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb41342b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081ab860 con 0x55b0b8045370 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb41342b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081a8aa0 con 0x55b0b803f9a0 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb41342b740 10 client.0 _ll_register_callbacks cb 0x7fb404009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40db52640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40db52640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33942/0 (socket says 172.21.15.89:33942) 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40db52640 1 -- 172.21.15.89:0/2702838719 learned_addr learned my addr 172.21.15.89:0/2702838719 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40eb54640 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 0x55b0b804f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40e353640 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b804e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40db52640 1 -- 172.21.15.89:0/2702838719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 msgr2=0x55b0b804e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40db52640 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b804e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40db52640 1 -- 172.21.15.89:0/2702838719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 msgr2=0x55b0b804f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40db52640 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 0x55b0b804f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40db52640 1 -- 172.21.15.89:0/2702838719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0b804ff40 con 0x55b0b80402e0 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.249+0000 7fb40eb54640 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 0x55b0b804f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb40db52640 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804e920 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0a4000 tx=0x7fb3fc017590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97222d202d358b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb3f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/2702838719 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc0d2e60 con 0x55b0b80402e0 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/2702838719 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb3fc0d29c0 con 0x55b0b80402e0 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb41342b740 1 -- 172.21.15.89:0/2702838719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb400028070 con 0x55b0b80402e0 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/2702838719 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb3fc01e070 con 0x55b0b80402e0 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb41342b740 10 client.18957 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb41342b740 1 -- 172.21.15.89:0/2702838719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b0b8051250 con 0x55b0b80402e0 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/2702838719 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb400028070 con 0x55b0b80402e0 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/2702838719 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb3fc0ccce0 con 0x55b0b80402e0 2024-09-15T23:34:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb41342b740 10 client.18957 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb41342b740 10 client.18957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb41342b740 20 client.18957 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb41342b740 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b0b80c0240 0x55b0b80c2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb41342b740 4 client.18957 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:34:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb41342b740 1 -- 172.21.15.89:0/2702838719 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b0b7faad70 con 0x55b0b80c0240 2024-09-15T23:34:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.250+0000 7fb40e353640 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b0b80c0240 0x55b0b80c2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.251+0000 7fb40e353640 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b0b80c0240 0x55b0b80c2660 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d8fa7694e322371 server_cookie=d8fc8f17763dd25b in_seq=0 out_seq=0 2024-09-15T23:34:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.251+0000 7fb3f1ffb640 10 client.18957 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb3f1ffb640 1 -- 172.21.15.89:0/2702838719 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b0b7faad70 con 0x55b0b80c0240 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb3f1ffb640 10 client.18957 handle_command_reply: tid=0 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 client.18957 shutdown 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 -- 172.21.15.89:0/2702838719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 msgr2=0x55b0b804e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804e920 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0a4000 tx=0x7fb3fc017590 comp rx=0 tx=0).stop 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 -- 172.21.15.89:0/2702838719 shutdown_connections 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b0b80c0240 0x55b0b80c2660 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b0b8045370 0x55b0b804f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b0b80402e0 0x55b0b804e920 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb3f1ffb640 0 client.18957 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 --2- 172.21.15.89:0/2702838719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b0b803f9a0 0x55b0b804e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 -- 172.21.15.89:0/2702838719 >> 172.21.15.89:0/2702838719 conn(0x55b0b803b010 msgr2=0x55b0b803b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 -- 172.21.15.89:0/2702838719 shutdown_connections 2024-09-15T23:34:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.264+0000 7fb41342b740 1 -- 172.21.15.89:0/2702838719 wait complete. 2024-09-15T23:34:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb41342b740 20 client.18957 trim_cache size 0 max 16384 2024-09-15T23:34:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:34:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:34:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:34:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:34:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 -- 172.21.15.89:0/1028907302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3f40a32f0 msgr2=0x7fb3f40a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3f40a32f0 0x7fb3f40a57b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb40000c610 tx=0x7fb40000e040 comp rx=0 tx=0).stop 2024-09-15T23:34:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 -- 172.21.15.89:0/1028907302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4081016d0 msgr2=0x7fb408198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4081016d0 0x7fb408198490 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00b4a0 tx=0x7fb3fc00b960 comp rx=0 tx=0).stop 2024-09-15T23:34:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 -- 172.21.15.89:0/1028907302 shutdown_connections 2024-09-15T23:34:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3f40a32f0 0x7fb3f40a57b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb408101fe0 0x7fb40819f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4081016d0 0x7fb408198490 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 --2- 172.21.15.89:0/1028907302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb408100d10 0x7fb408197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 -- 172.21.15.89:0/1028907302 >> 172.21.15.89:0/1028907302 conn(0x7fb4080fb970 msgr2=0x7fb4081050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 -- 172.21.15.89:0/1028907302 shutdown_connections 2024-09-15T23:34:19.270 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:19.265+0000 7fb4105b7640 1 -- 172.21.15.89:0/1028907302 wait complete. 2024-09-15T23:34:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:19 smithi120 ceph-mon[35250]: pgmap v658: 129 pgs: 129 active+clean; 981 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:34:19.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:19 smithi174 ceph-mon[35586]: pgmap v658: 129 pgs: 129 active+clean; 981 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:34:19.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:19 smithi089 ceph-mon[30933]: pgmap v658: 129 pgs: 129 active+clean; 981 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:34:21.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:21 smithi120 ceph-mon[35250]: pgmap v659: 129 pgs: 129 active+clean; 981 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:34:21.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:21 smithi174 ceph-mon[35586]: pgmap v659: 129 pgs: 129 active+clean; 981 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:34:21.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:21 smithi089 ceph-mon[30933]: pgmap v659: 129 pgs: 129 active+clean; 981 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:34:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:23 smithi120 ceph-mon[35250]: pgmap v660: 129 pgs: 129 active+clean; 981 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:34:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:23 smithi174 ceph-mon[35586]: pgmap v660: 129 pgs: 129 active+clean; 981 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:34:23.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:23 smithi089 ceph-mon[30933]: pgmap v660: 129 pgs: 129 active+clean; 981 KiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:34:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:25 smithi120 ceph-mon[35250]: pgmap v661: 129 pgs: 129 active+clean; 1.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:34:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:25 smithi174 ceph-mon[35586]: pgmap v661: 129 pgs: 129 active+clean; 1.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:34:25.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:25 smithi089 ceph-mon[30933]: pgmap v661: 129 pgs: 129 active+clean; 1.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:34:27.481 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:27 smithi120 ceph-mon[35250]: pgmap v662: 129 pgs: 129 active+clean; 1.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:34:27.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:27 smithi174 ceph-mon[35586]: pgmap v662: 129 pgs: 129 active+clean; 1.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:34:27.573 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:27 smithi089 ceph-mon[30933]: pgmap v662: 129 pgs: 129 active+clean; 1.0 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:34:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:29 smithi120 ceph-mon[35250]: pgmap v663: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:34:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:29 smithi174 ceph-mon[35586]: pgmap v663: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:34:29.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:29 smithi089 ceph-mon[30933]: pgmap v663: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:34:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:31 smithi120 ceph-mon[35250]: pgmap v664: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:31 smithi174 ceph-mon[35586]: pgmap v664: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:31.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:31 smithi089 ceph-mon[30933]: pgmap v664: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.362+0000 7ff6c6166640 1 -- 172.21.15.89:0/2027166057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c0101b60 msgr2=0x7ff6c0101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.362+0000 7ff6c6166640 1 --2- 172.21.15.89:0/2027166057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c0101b60 0x7ff6c0101f60 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009ca0 tx=0x7ff6b002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.362+0000 7ff6c6166640 1 -- 172.21.15.89:0/2027166057 shutdown_connections 2024-09-15T23:34:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.362+0000 7ff6c6166640 1 --2- 172.21.15.89:0/2027166057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c0103710 0x7ff6c0109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.362+0000 7ff6c6166640 1 --2- 172.21.15.89:0/2027166057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6c0102d50 0x7ff6c01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.362+0000 7ff6c6166640 1 --2- 172.21.15.89:0/2027166057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c0101b60 0x7ff6c0101f60 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.362+0000 7ff6c6166640 1 -- 172.21.15.89:0/2027166057 >> 172.21.15.89:0/2027166057 conn(0x7ff6c00fd2e0 msgr2=0x7ff6c00ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.362+0000 7ff6c6166640 1 -- 172.21.15.89:0/2027166057 shutdown_connections 2024-09-15T23:34:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.362+0000 7ff6c6166640 1 -- 172.21.15.89:0/2027166057 wait complete. 2024-09-15T23:34:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6c6166640 1 Processor -- start 2024-09-15T23:34:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6c6166640 1 -- start start 2024-09-15T23:34:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6c6166640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c0101b60 0x7ff6c0197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6c6166640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c0102d50 0x7ff6c0198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6c6166640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6c0103710 0x7ff6c019f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6c6166640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c010d7e0 con 0x7ff6c0101b60 2024-09-15T23:34:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6c6166640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c010d660 con 0x7ff6c0103710 2024-09-15T23:34:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6c6166640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c010d960 con 0x7ff6c0102d50 2024-09-15T23:34:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6bf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c0101b60 0x7ff6c0197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6bf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c0101b60 0x7ff6c0197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59884/0 (socket says 172.21.15.89:59884) 2024-09-15T23:34:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/1904205475 learned_addr learned my addr 172.21.15.89:0/1904205475 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6bffff640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6c0103710 0x7ff6c019f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6beffd640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c0102d50 0x7ff6c0198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/1904205475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c0102d50 msgr2=0x7ff6c0198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c0102d50 0x7ff6c0198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/1904205475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6c0103710 msgr2=0x7ff6c019f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6c0103710 0x7ff6c019f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/1904205475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c019f9f0 con 0x7ff6c0101b60 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.363+0000 7ff6beffd640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c0102d50 0x7ff6c0198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.364+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c0101b60 0x7ff6c0197dd0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0004af0 tx=0x7ff6b0031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.364+0000 7ff6bcff9640 1 -- 172.21.15.89:0/1904205475 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b002fb60 con 0x7ff6c0101b60 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.364+0000 7ff6bcff9640 1 -- 172.21.15.89:0/1904205475 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b002fd00 con 0x7ff6c0101b60 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.364+0000 7ff6c6166640 1 -- 172.21.15.89:0/1904205475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c019fc80 con 0x7ff6c0101b60 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.364+0000 7ff6bcff9640 1 -- 172.21.15.89:0/1904205475 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0046920 con 0x7ff6c0101b60 2024-09-15T23:34:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.364+0000 7ff6c6166640 1 -- 172.21.15.89:0/1904205475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c01a01d0 con 0x7ff6c0101b60 2024-09-15T23:34:32.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.366+0000 7ff6bcff9640 1 -- 172.21.15.89:0/1904205475 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff6b0030c30 con 0x7ff6c0101b60 2024-09-15T23:34:32.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6c8fdc740 1 Processor -- start 2024-09-15T23:34:32.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6c8fdc740 1 -- start start 2024-09-15T23:34:32.376 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bbdb0 0x5576f34bc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 0x5576f34c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5576f341b2f0 con 0x5576f34bc6f0 2024-09-15T23:34:32.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5576f33c3270 con 0x5576f34c1780 2024-09-15T23:34:32.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5576f34172b0 con 0x5576f34bbdb0 2024-09-15T23:34:32.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bcff9640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6a80a32f0 0x7ff6a80a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59886/0 (socket says 172.21.15.89:59886) 2024-09-15T23:34:32.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/3631891707 learned_addr learned my addr 172.21.15.89:0/3631891707 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:32.377 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6beffd640 1 --2- 172.21.15.89:0/3631891707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 0x5576f34c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bffff640 1 --2- 172.21.15.89:0/3631891707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bbdb0 0x5576f34bc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bffff640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6a80a32f0 0x7ff6a80a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bcff9640 1 -- 172.21.15.89:0/1904205475 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff6b00f1230 con 0x7ff6c0101b60 2024-09-15T23:34:32.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/3631891707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bbdb0 msgr2=0x5576f34bc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:32.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/3631891707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bbdb0 0x5576f34bc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/3631891707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 msgr2=0x5576f34c3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:32.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/3631891707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 0x5576f34c3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.367+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/3631891707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576f33c2340 con 0x5576f34bc6f0 2024-09-15T23:34:32.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.368+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/3631891707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c11b0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00ee940 tx=0x7ff6b002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eaa9ef98cacf7e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:32.378 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.368+0000 7ff6bffff640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6a80a32f0 0x7ff6a80a57b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8012910 tx=0x7ff6b800b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:32.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.368+0000 7ff6a5ffb640 1 -- 172.21.15.89:0/3631891707 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b00bdb60 con 0x5576f34bc6f0 2024-09-15T23:34:32.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.368+0000 7ff6a5ffb640 1 -- 172.21.15.89:0/3631891707 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b0036070 con 0x5576f34bc6f0 2024-09-15T23:34:32.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.368+0000 7ff6a5ffb640 1 -- 172.21.15.89:0/3631891707 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b00b9ad0 con 0x5576f34bc6f0 2024-09-15T23:34:32.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/3631891707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 msgr2=0x5576f34c11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:32.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/3631891707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c11b0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00ee940 tx=0x7ff6b002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/3631891707 shutdown_connections 2024-09-15T23:34:32.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/3631891707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 0x5576f34c3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.379 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/3631891707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c11b0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/3631891707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bbdb0 0x5576f34bc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/3631891707 >> 172.21.15.89:0/3631891707 conn(0x5576f34b3010 msgr2=0x5576f34b3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:32.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/3631891707 shutdown_connections 2024-09-15T23:34:32.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/3631891707 wait complete. 2024-09-15T23:34:32.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 -- start start 2024-09-15T23:34:32.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.369+0000 7ff6c8fdc740 1 Processor -- start 2024-09-15T23:34:32.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 0x5576f34c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34c1780 0x5576f34cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.380 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5576f33c3270 con 0x5576f34bc6f0 2024-09-15T23:34:32.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5576f33a4290 con 0x5576f34bbdb0 2024-09-15T23:34:32.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5576f341b2f0 con 0x5576f34c1780 2024-09-15T23:34:32.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6c8fdc740 10 client.0 _ll_register_callbacks cb 0x5576f33d5140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:32.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6beffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6beffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59890/0 (socket says 172.21.15.89:59890) 2024-09-15T23:34:32.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6beffd640 1 -- 172.21.15.89:0/4153219572 learned_addr learned my addr 172.21.15.89:0/4153219572 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:32.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 0x5576f34c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6beffd640 1 -- 172.21.15.89:0/4153219572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34c1780 msgr2=0x5576f34cd0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:34:32.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6beffd640 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34c1780 0x5576f34cd0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.381 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6beffd640 1 -- 172.21.15.89:0/4153219572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 msgr2=0x5576f34c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6beffd640 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 0x5576f34c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6beffd640 1 -- 172.21.15.89:0/4153219572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576f312e210 con 0x5576f34bc6f0 2024-09-15T23:34:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 0x5576f34c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6beffd640 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c6060 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009f30 tx=0x7ff6b4011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc2f48cfd7f89ff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff69effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff69effd640 1 -- 172.21.15.89:0/4153219572 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4012920 con 0x5576f34bc6f0 2024-09-15T23:34:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff69effd640 1 -- 172.21.15.89:0/4153219572 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff6b4012ac0 con 0x5576f34bc6f0 2024-09-15T23:34:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.370+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/4153219572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6b0036070 con 0x5576f34bc6f0 2024-09-15T23:34:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff69effd640 1 -- 172.21.15.89:0/4153219572 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff6b4025050 con 0x5576f34bc6f0 2024-09-15T23:34:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff6c8fdc740 10 client.18969 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/4153219572 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5576f31fa620 con 0x5576f34bc6f0 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff69effd640 1 -- 172.21.15.89:0/4153219572 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b0036070 con 0x5576f34bc6f0 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff69effd640 1 -- 172.21.15.89:0/4153219572 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff6b400f040 con 0x5576f34bc6f0 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff6c8fdc740 10 client.18969 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff6c8fdc740 10 client.18969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff6c8fdc740 20 client.18969 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5576f34d18b0 0x5576f34d3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff6c8fdc740 4 client.18969 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/4153219572 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576f3428a20 con 0x5576f34d18b0 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.371+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5576f34d18b0 0x5576f34d3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.372+0000 7ff69effd640 10 client.18969 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.372+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5576f34d18b0 0x5576f34d3d00 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82ab169252038dc7 server_cookie=3f90be95715e5c37 in_seq=0 out_seq=0 2024-09-15T23:34:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff69effd640 1 -- 172.21.15.89:0/4153219572 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5576f3428a20 con 0x5576f34d18b0 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff69effd640 10 client.18969 handle_command_reply: tid=0 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 client.18969 shutdown 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/4153219572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 msgr2=0x5576f34c6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c6060 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009f30 tx=0x7ff6b4011870 comp rx=0 tx=0).stop 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/4153219572 shutdown_connections 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5576f34d18b0 0x5576f34d3d00 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34c1780 0x5576f34cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bc6f0 0x5576f34c6060 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/4153219572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 0x5576f34c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff69effd640 0 client.18969 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/4153219572 >> 172.21.15.89:0/4153219572 conn(0x5576f34b3010 msgr2=0x5576f34b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/4153219572 shutdown_connections 2024-09-15T23:34:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/4153219572 wait complete. 2024-09-15T23:34:32.388 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.373+0000 7ff6c8fdc740 20 client.18969 trim_cache size 0 max 16384 2024-09-15T23:34:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6c8fdc740 1 Processor -- start 2024-09-15T23:34:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6c8fdc740 1 -- start start 2024-09-15T23:34:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bbdb0 0x5576f34b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bc6f0 0x5576f34cd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 0x5576f34b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c01a8580 con 0x5576f34bbdb0 2024-09-15T23:34:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c01aa0c0 con 0x5576f34c1780 2024-09-15T23:34:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c01a8c50 con 0x5576f34bc6f0 2024-09-15T23:34:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6bf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bbdb0 0x5576f34b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6bf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bbdb0 0x5576f34b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59894/0 (socket says 172.21.15.89:59894) 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.387+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/3199578878 learned_addr learned my addr 172.21.15.89:0/3199578878 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6bffff640 1 --2- 172.21.15.89:0/3199578878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 0x5576f34b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/3199578878 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bc6f0 msgr2=0x5576f34cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/3199578878 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bc6f0 0x5576f34cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/3199578878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 msgr2=0x5576f34b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/3199578878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 0x5576f34b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6bf7fe640 1 -- 172.21.15.89:0/3199578878 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576f34b2e30 con 0x5576f34bbdb0 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6bffff640 1 --2- 172.21.15.89:0/3199578878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 0x5576f34b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/3199578878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bbdb0 0x5576f34b20b0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0005fa0 tx=0x7ff6b003d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aefbed467f406a8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6a57fa640 1 -- 172.21.15.89:0/3199578878 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b00bdb60 con 0x5576f34bbdb0 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6a57fa640 1 -- 172.21.15.89:0/3199578878 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b0036040 con 0x5576f34bbdb0 2024-09-15T23:34:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.388+0000 7ff6a57fa640 1 -- 172.21.15.89:0/3199578878 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b00b9ad0 con 0x5576f34bbdb0 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/3199578878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bbdb0 msgr2=0x5576f34b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/3199578878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bbdb0 0x5576f34b20b0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0005fa0 tx=0x7ff6b003d040 comp rx=0 tx=0).stop 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/3199578878 shutdown_connections 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/3199578878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34c1780 0x5576f34b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/3199578878 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bc6f0 0x5576f34cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/3199578878 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34bbdb0 0x5576f34b20b0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/3199578878 >> 172.21.15.89:0/3199578878 conn(0x5576f34b3010 msgr2=0x5576f34b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/3199578878 shutdown_connections 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/3199578878 wait complete. 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 -- start start 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 Processor -- start 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 0x5576f34ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bc6f0 0x5576f34cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34c1780 0x5576f34cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c01a99f0 con 0x5576f34c1780 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c01a9320 con 0x5576f34bbdb0 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c01a6560 con 0x5576f34bc6f0 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.389+0000 7ff6c8fdc740 10 client.0 _ll_register_callbacks cb 0x7ff6b40097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34c1780 0x5576f34cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6bffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34c1780 0x5576f34cbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59908/0 (socket says 172.21.15.89:59908) 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6bffff640 1 -- 172.21.15.89:0/1477370753 learned_addr learned my addr 172.21.15.89:0/1477370753 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 0x5576f34ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6beffd640 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bc6f0 0x5576f34cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6bffff640 1 -- 172.21.15.89:0/1477370753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bc6f0 msgr2=0x5576f34cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6bffff640 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bc6f0 0x5576f34cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6bffff640 1 -- 172.21.15.89:0/1477370753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 msgr2=0x5576f34ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6bffff640 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 0x5576f34ca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6bffff640 1 -- 172.21.15.89:0/1477370753 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576f34cc330 con 0x5576f34c1780 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6beffd640 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bc6f0 0x5576f34cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6bffff640 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34c1780 0x5576f34cbd20 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8012730 tx=0x7ff6b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9fe0b6803610f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff69effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff69effd640 1 -- 172.21.15.89:0/1477370753 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8009bc0 con 0x5576f34c1780 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff69effd640 1 -- 172.21.15.89:0/1477370753 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff6b8009d60 con 0x5576f34c1780 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/1477370753 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6b0036040 con 0x5576f34c1780 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff69effd640 1 -- 172.21.15.89:0/1477370753 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff6b8005020 con 0x5576f34c1780 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6c8fdc740 10 client.18975 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.390+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/1477370753 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5576f353c640 con 0x5576f34c1780 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.391+0000 7ff69effd640 1 -- 172.21.15.89:0/1477370753 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b0036040 con 0x5576f34c1780 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.391+0000 7ff69effd640 1 -- 172.21.15.89:0/1477370753 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff6b80035f0 con 0x5576f34c1780 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.391+0000 7ff6c8fdc740 10 client.18975 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.391+0000 7ff6c8fdc740 10 client.18975 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.391+0000 7ff6c8fdc740 20 client.18975 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.391+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5576f353d020 0x5576f353f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.391+0000 7ff6c8fdc740 4 client.18975 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.391+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/1477370753 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5576f34c1500 con 0x5576f353d020 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.391+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5576f353d020 0x5576f353f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.392+0000 7ff69effd640 10 client.18975 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:32.398 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.392+0000 7ff6bf7fe640 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5576f353d020 0x5576f353f440 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb73bc90c8f636eb server_cookie=bf5bf144dd0cde6c in_seq=0 out_seq=0 2024-09-15T23:34:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff69effd640 1 -- 172.21.15.89:0/1477370753 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5576f34c1500 con 0x5576f353d020 2024-09-15T23:34:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff69effd640 10 client.18975 handle_command_reply: tid=0 2024-09-15T23:34:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 client.18975 shutdown 2024-09-15T23:34:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/1477370753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34c1780 msgr2=0x5576f34cbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34c1780 0x5576f34cbd20 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8012730 tx=0x7ff6b80046c0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/1477370753 shutdown_connections 2024-09-15T23:34:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5576f353d020 0x5576f353f440 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5576f34c1780 0x5576f34cbd20 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5576f34bc6f0 0x5576f34cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff69effd640 0 client.18975 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:32.406 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 --2- 172.21.15.89:0/1477370753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576f34bbdb0 0x5576f34ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/1477370753 >> 172.21.15.89:0/1477370753 conn(0x5576f34b3010 msgr2=0x5576f34b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/1477370753 shutdown_connections 2024-09-15T23:34:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 1 -- 172.21.15.89:0/1477370753 wait complete. 2024-09-15T23:34:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.404+0000 7ff6c8fdc740 20 client.18975 trim_cache size 0 max 16384 2024-09-15T23:34:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:34:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:34:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:34:32.407 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:34:32.408 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 -- 172.21.15.89:0/1904205475 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6a80a32f0 msgr2=0x7ff6a80a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6a80a32f0 0x7ff6a80a57b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8012910 tx=0x7ff6b800b2e0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 -- 172.21.15.89:0/1904205475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c0101b60 msgr2=0x7ff6c0197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c0101b60 0x7ff6c0197dd0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0004af0 tx=0x7ff6b0031ee0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 -- 172.21.15.89:0/1904205475 shutdown_connections 2024-09-15T23:34:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff6a80a32f0 0x7ff6a80a57b0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff6c0103710 0x7ff6c019f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6c0102d50 0x7ff6c0198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 --2- 172.21.15.89:0/1904205475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff6c0101b60 0x7ff6c0197dd0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:32.409 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 -- 172.21.15.89:0/1904205475 >> 172.21.15.89:0/1904205475 conn(0x7ff6c00fd2e0 msgr2=0x7ff6c0106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:32.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 -- 172.21.15.89:0/1904205475 shutdown_connections 2024-09-15T23:34:32.410 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:32.405+0000 7ff6c6166640 1 -- 172.21.15.89:0/1904205475 wait complete. 2024-09-15T23:34:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:33 smithi120 ceph-mon[35250]: pgmap v665: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:34:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:33 smithi174 ceph-mon[35586]: pgmap v665: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:34:33.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:33 smithi089 ceph-mon[30933]: pgmap v665: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:34:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:35 smithi120 ceph-mon[35250]: pgmap v666: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-15T23:34:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:35 smithi174 ceph-mon[35586]: pgmap v666: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-15T23:34:35.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:35 smithi089 ceph-mon[30933]: pgmap v666: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-15T23:34:35.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:34:36.091 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:36.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.791+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/69492217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c4103660 msgr2=0x7fb3c410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:36.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.791+0000 7fb3cc4a0640 1 --2- 172.21.15.89:0/69492217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c4103660 0x7fb3c410bb50 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009900 tx=0x7fb3b402a480 comp rx=0 tx=0).stop 2024-09-15T23:34:36.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/69492217 shutdown_connections 2024-09-15T23:34:36.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 --2- 172.21.15.89:0/69492217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c410c170 0x7fb3c410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 --2- 172.21.15.89:0/69492217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c4103660 0x7fb3c410bb50 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 --2- 172.21.15.89:0/69492217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3c4102d40 0x7fb3c4103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/69492217 >> 172.21.15.89:0/69492217 conn(0x7fb3c40fc7a0 msgr2=0x7fb3c40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:36.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/69492217 shutdown_connections 2024-09-15T23:34:36.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/69492217 wait complete. 2024-09-15T23:34:36.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 Processor -- start 2024-09-15T23:34:36.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 -- start start 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c4102d40 0x7fb3c41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3c4103660 0x7fb3c4198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c410c170 0x7fb3c419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c4111f70 con 0x7fb3c4102d40 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c4111df0 con 0x7fb3c410c170 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.792+0000 7fb3cc4a0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41120f0 con 0x7fb3c4103660 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3ca23c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c4102d40 0x7fb3c41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3ca23c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c4102d40 0x7fb3c41984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37742/0 (socket says 172.21.15.89:37742) 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3ca23c640 1 -- 172.21.15.89:0/559906641 learned_addr learned my addr 172.21.15.89:0/559906641 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3c4103660 0x7fb3c4198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3caa3d640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c410c170 0x7fb3c419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3ca23c640 1 -- 172.21.15.89:0/559906641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3c4103660 msgr2=0x7fb3c4198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3c4103660 0x7fb3c4198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3ca23c640 1 -- 172.21.15.89:0/559906641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c410c170 msgr2=0x7fb3c419cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c410c170 0x7fb3c419cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3ca23c640 1 -- 172.21.15.89:0/559906641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c419d480 con 0x7fb3c4102d40 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3c4103660 0x7fb3c4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c4102d40 0x7fb3c41984d0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000d770 tx=0x7fb3c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3bb7fe640 1 -- 172.21.15.89:0/559906641 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0017070 con 0x7fb3c4102d40 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/559906641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c419d770 con 0x7fb3c4102d40 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3bb7fe640 1 -- 172.21.15.89:0/559906641 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3c0005500 con 0x7fb3c4102d40 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3bb7fe640 1 -- 172.21.15.89:0/559906641 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0004e00 con 0x7fb3c4102d40 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.793+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/559906641 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3c41a5050 con 0x7fb3c4102d40 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.796+0000 7fb3bb7fe640 1 -- 172.21.15.89:0/559906641 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb3c0004fa0 con 0x7fb3c4102d40 2024-09-15T23:34:36.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.796+0000 7fb3ce879740 1 Processor -- start 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.796+0000 7fb3ce879740 1 -- start start 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde064a0 0x5591bde06880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde11140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde11680 0x5591bde13a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ce879740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5591bdd6c2e0 con 0x5591bde06dc0 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ce879740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5591bdd13ce0 con 0x5591bde064a0 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ce879740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5591bdd682a0 con 0x5591bde11680 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ca23c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde11140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ca23c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde11140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37750/0 (socket says 172.21.15.89:37750) 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ca23c640 1 -- 172.21.15.89:0/1195962111 learned_addr learned my addr 172.21.15.89:0/1195962111 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/1195962111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde11680 0x5591bde13a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3caa3d640 1 --2- 172.21.15.89:0/1195962111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde064a0 0x5591bde06880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3bb7fe640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3b00a3240 0x7fb3b00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ca23c640 1 -- 172.21.15.89:0/1195962111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde11680 msgr2=0x5591bde13a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/1195962111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde11680 0x5591bde13a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ca23c640 1 -- 172.21.15.89:0/1195962111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde064a0 msgr2=0x5591bde06880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/1195962111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde064a0 0x5591bde06880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ca23c640 1 -- 172.21.15.89:0/1195962111 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591bdcf8440 con 0x5591bde06dc0 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/1195962111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde11680 0x5591bde13a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3caa3d640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3b00a3240 0x7fb3b00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3bb7fe640 1 -- 172.21.15.89:0/559906641 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb3c00cf7e0 con 0x7fb3c4102d40 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/1195962111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde11140 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00c79c0 tx=0x7fb3c00c67e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ede89cacd01670 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.797+0000 7fb3b8ff9640 1 -- 172.21.15.89:0/1195962111 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0020070 con 0x5591bde06dc0 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3b8ff9640 1 -- 172.21.15.89:0/1195962111 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3c00046c0 con 0x5591bde06dc0 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3b8ff9640 1 -- 172.21.15.89:0/1195962111 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c009c110 con 0x5591bde06dc0 2024-09-15T23:34:36.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3ce879740 1 -- 172.21.15.89:0/1195962111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 msgr2=0x5591bde11140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3caa3d640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3b00a3240 0x7fb3b00a5700 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc005fd0 tx=0x7fb3bc004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3ce879740 1 --2- 172.21.15.89:0/1195962111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde11140 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00c79c0 tx=0x7fb3c00c67e0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3ce879740 1 -- 172.21.15.89:0/1195962111 shutdown_connections 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3ce879740 1 --2- 172.21.15.89:0/1195962111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde11680 0x5591bde13a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3ce879740 1 --2- 172.21.15.89:0/1195962111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde11140 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3ce879740 1 --2- 172.21.15.89:0/1195962111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde064a0 0x5591bde06880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3ce879740 1 -- 172.21.15.89:0/1195962111 >> 172.21.15.89:0/1195962111 conn(0x5591bde02a30 msgr2=0x5591bde02e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3ce879740 1 -- 172.21.15.89:0/1195962111 shutdown_connections 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.798+0000 7fb3ce879740 1 -- 172.21.15.89:0/1195962111 wait complete. 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ce879740 1 -- start start 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ce879740 1 Processor -- start 2024-09-15T23:34:36.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde10f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 0x5591bde0e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde0e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ce879740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5591bde141f0 con 0x5591bde064a0 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ce879740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5591bdd27050 con 0x5591bde11680 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ce879740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5591bde14370 con 0x5591bde06dc0 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ce879740 10 client.0 _ll_register_callbacks cb 0x5591bdd67160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ca23c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde10f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ca23c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde10f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37766/0 (socket says 172.21.15.89:37766) 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ca23c640 1 -- 172.21.15.89:0/4039690034 learned_addr learned my addr 172.21.15.89:0/4039690034 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3caa3d640 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde0e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 0x5591bde0e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ca23c640 1 -- 172.21.15.89:0/4039690034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 msgr2=0x5591bde0e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:36.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 0x5591bde0e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ca23c640 1 -- 172.21.15.89:0/4039690034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 msgr2=0x5591bde0e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde0e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3ca23c640 1 -- 172.21.15.89:0/4039690034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591bdadd3c0 con 0x5591bde064a0 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 0x5591bde0e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.799+0000 7fb3caa3d640 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde0e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde10f40 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00ccec0 tx=0x7fb3c00afc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2972d8adc04c418 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3adffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3adffb640 1 -- 172.21.15.89:0/4039690034 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0020040 con 0x5591bde064a0 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3adffb640 1 -- 172.21.15.89:0/4039690034 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb3c0099d10 con 0x5591bde064a0 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3ce879740 1 -- 172.21.15.89:0/4039690034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5591bdd79850 con 0x5591bde064a0 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3adffb640 1 -- 172.21.15.89:0/4039690034 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb3c009b550 con 0x5591bde064a0 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3ce879740 10 client.18987 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:36.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3ce879740 1 -- 172.21.15.89:0/4039690034 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5591bde1fb40 con 0x5591bde064a0 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3adffb640 1 -- 172.21.15.89:0/4039690034 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5591bdd79850 con 0x5591bde064a0 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3adffb640 1 -- 172.21.15.89:0/4039690034 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb3c0098080 con 0x5591bde064a0 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3ce879740 10 client.18987 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3ce879740 10 client.18987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3ce879740 20 client.18987 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3ce879740 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5591bde20e90 0x5591bde23280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3ce879740 4 client.18987 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.800+0000 7fb3ce879740 1 -- 172.21.15.89:0/4039690034 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591bdd73d60 con 0x5591bde20e90 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.801+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5591bde20e90 0x5591bde23280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.801+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5591bde20e90 0x5591bde23280 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=273c7745c883b622 server_cookie=66d31fb0c2402c3c in_seq=0 out_seq=0 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.801+0000 7fb3adffb640 10 client.18987 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:36.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3adffb640 1 -- 172.21.15.89:0/4039690034 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5591bdd73d60 con 0x5591bde20e90 2024-09-15T23:34:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3adffb640 10 client.18987 handle_command_reply: tid=0 2024-09-15T23:34:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 client.18987 shutdown 2024-09-15T23:34:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 -- 172.21.15.89:0/4039690034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 msgr2=0x5591bde10f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde10f40 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00ccec0 tx=0x7fb3c00afc30 comp rx=0 tx=0).stop 2024-09-15T23:34:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 -- 172.21.15.89:0/4039690034 shutdown_connections 2024-09-15T23:34:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5591bde20e90 0x5591bde23280 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde0e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 0x5591bde0e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3adffb640 0 client.18987 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:36.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 --2- 172.21.15.89:0/4039690034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde10f40 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 -- 172.21.15.89:0/4039690034 >> 172.21.15.89:0/4039690034 conn(0x5591bde02a30 msgr2=0x5591bde08760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:36.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 -- 172.21.15.89:0/4039690034 shutdown_connections 2024-09-15T23:34:36.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 1 -- 172.21.15.89:0/4039690034 wait complete. 2024-09-15T23:34:36.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.802+0000 7fb3ce879740 20 client.18987 trim_cache size 0 max 16384 2024-09-15T23:34:36.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.819+0000 7fb3ce879740 1 Processor -- start 2024-09-15T23:34:36.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.819+0000 7fb3ce879740 1 -- start start 2024-09-15T23:34:36.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde064a0 0x5591bde262a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde267e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde0b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3ce879740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41aeea0 con 0x5591bde06dc0 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3ce879740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41b09e0 con 0x5591bde11680 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3ce879740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41af570 con 0x5591bde064a0 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3c9a3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde267e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3c9a3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde267e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37768/0 (socket says 172.21.15.89:37768) 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3c9a3b640 1 -- 172.21.15.89:0/3621261507 learned_addr learned my addr 172.21.15.89:0/3621261507 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/3621261507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde064a0 0x5591bde262a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3caa3d640 1 --2- 172.21.15.89:0/3621261507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde0b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3c9a3b640 1 -- 172.21.15.89:0/3621261507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde064a0 msgr2=0x5591bde262a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/3621261507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde064a0 0x5591bde262a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3c9a3b640 1 -- 172.21.15.89:0/3621261507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 msgr2=0x5591bde0b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/3621261507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde0b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3c9a3b640 1 -- 172.21.15.89:0/3621261507 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591bde02860 con 0x5591bde06dc0 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3caa3d640 1 --2- 172.21.15.89:0/3621261507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde0b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/3621261507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde267e0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009850 tx=0x7fb3b4009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0d8c1008976030 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3affff640 1 -- 172.21.15.89:0/3621261507 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b4035040 con 0x5591bde06dc0 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3affff640 1 -- 172.21.15.89:0/3621261507 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3b4047070 con 0x5591bde06dc0 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.820+0000 7fb3affff640 1 -- 172.21.15.89:0/3621261507 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b4004b80 con 0x5591bde06dc0 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.821+0000 7fb3ce879740 1 -- 172.21.15.89:0/3621261507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 msgr2=0x5591bde267e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.821+0000 7fb3ce879740 1 --2- 172.21.15.89:0/3621261507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde267e0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009850 tx=0x7fb3b4009610 comp rx=0 tx=0).stop 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.821+0000 7fb3ce879740 1 -- 172.21.15.89:0/3621261507 shutdown_connections 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.821+0000 7fb3ce879740 1 --2- 172.21.15.89:0/3621261507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde0b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.821+0000 7fb3ce879740 1 --2- 172.21.15.89:0/3621261507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde06dc0 0x5591bde267e0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.821+0000 7fb3ce879740 1 --2- 172.21.15.89:0/3621261507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde064a0 0x5591bde262a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.821+0000 7fb3ce879740 1 -- 172.21.15.89:0/3621261507 >> 172.21.15.89:0/3621261507 conn(0x5591bde02a30 msgr2=0x5591bde0d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.821+0000 7fb3ce879740 1 -- 172.21.15.89:0/3621261507 shutdown_connections 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.821+0000 7fb3ce879740 1 -- 172.21.15.89:0/3621261507 wait complete. 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ce879740 1 -- start start 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ce879740 1 Processor -- start 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde24300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 0x5591bde24840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ce879740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde24da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ce879740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41b0310 con 0x5591bde064a0 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ce879740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41afc40 con 0x5591bde11680 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ce879740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41ace80 con 0x5591bde06dc0 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ce879740 10 client.0 _ll_register_callbacks cb 0x7fb3c0097b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ca23c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde24300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ca23c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde24300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37778/0 (socket says 172.21.15.89:37778) 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ca23c640 1 -- 172.21.15.89:0/85706367 learned_addr learned my addr 172.21.15.89:0/85706367 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3caa3d640 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde24da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 0x5591bde24840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ca23c640 1 -- 172.21.15.89:0/85706367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 msgr2=0x5591bde24840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 0x5591bde24840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ca23c640 1 -- 172.21.15.89:0/85706367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 msgr2=0x5591bde24da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde24da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3ca23c640 1 -- 172.21.15.89:0/85706367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591bde13e40 con 0x5591bde064a0 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3caa3d640 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde24da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.822+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 0x5591bde24840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3ca23c640 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde24300 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00ccef0 tx=0x7fb3c00ba530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e56cd71b04c971 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3adffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3adffb640 1 -- 172.21.15.89:0/85706367 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0020070 con 0x5591bde064a0 2024-09-15T23:34:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3adffb640 1 -- 172.21.15.89:0/85706367 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb3c00a43f0 con 0x5591bde064a0 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3ce879740 1 -- 172.21.15.89:0/85706367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3b4047070 con 0x5591bde064a0 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3adffb640 1 -- 172.21.15.89:0/85706367 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb3c009b370 con 0x5591bde064a0 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3ce879740 10 client.18993 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3ce879740 1 -- 172.21.15.89:0/85706367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5591bde8bb50 con 0x5591bde064a0 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3adffb640 1 -- 172.21.15.89:0/85706367 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3b4047070 con 0x5591bde064a0 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3adffb640 1 -- 172.21.15.89:0/85706367 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb3c009bc70 con 0x5591bde064a0 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3ce879740 10 client.18993 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3ce879740 10 client.18993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3ce879740 20 client.18993 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3ce879740 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5591bde8bf30 0x5591bde8e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3ce879740 4 client.18993 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.823+0000 7fb3ce879740 1 -- 172.21.15.89:0/85706367 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5591bdcf0210 con 0x5591bde8bf30 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5591bde8bf30 0x5591bde8e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:36.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3c9a3b640 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5591bde8bf30 0x5591bde8e390 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=204bd45d23979e4f server_cookie=98e9072e720b4755 in_seq=0 out_seq=0 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3adffb640 10 client.18993 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3adffb640 1 -- 172.21.15.89:0/85706367 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5591bdcf0210 con 0x5591bde8bf30 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3adffb640 10 client.18993 handle_command_reply: tid=0 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 client.18993 shutdown 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 -- 172.21.15.89:0/85706367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 msgr2=0x5591bde24300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde24300 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fb3c00ccef0 tx=0x7fb3c00ba530 comp rx=0 tx=0).stop 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 -- 172.21.15.89:0/85706367 shutdown_connections 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5591bde8bf30 0x5591bde8e390 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591bde11680 0x5591bde24da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3adffb640 0 client.18993 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5591bde06dc0 0x5591bde24840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 --2- 172.21.15.89:0/85706367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5591bde064a0 0x5591bde24300 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 -- 172.21.15.89:0/85706367 >> 172.21.15.89:0/85706367 conn(0x5591bde02a30 msgr2=0x5591bde07ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:36.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 -- 172.21.15.89:0/85706367 shutdown_connections 2024-09-15T23:34:36.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.824+0000 7fb3ce879740 1 -- 172.21.15.89:0/85706367 wait complete. 2024-09-15T23:34:36.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3ce879740 20 client.18993 trim_cache size 0 max 16384 2024-09-15T23:34:36.833 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:34:36.833 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:34:36.833 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:34:36.833 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:34:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/559906641 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3b00a3240 msgr2=0x7fb3b00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3b00a3240 0x7fb3b00a5700 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc005fd0 tx=0x7fb3bc004a40 comp rx=0 tx=0).stop 2024-09-15T23:34:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/559906641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c4102d40 msgr2=0x7fb3c41984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c4102d40 0x7fb3c41984d0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000d770 tx=0x7fb3c000dc30 comp rx=0 tx=0).stop 2024-09-15T23:34:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/559906641 shutdown_connections 2024-09-15T23:34:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb3b00a3240 0x7fb3b00a5700 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb3c410c170 0x7fb3c419cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3c4103660 0x7fb3c4198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 --2- 172.21.15.89:0/559906641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3c4102d40 0x7fb3c41984d0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:36.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/559906641 >> 172.21.15.89:0/559906641 conn(0x7fb3c40fc7a0 msgr2=0x7fb3c40fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:36.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/559906641 shutdown_connections 2024-09-15T23:34:36.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:36.825+0000 7fb3cc4a0640 1 -- 172.21.15.89:0/559906641 wait complete. 2024-09-15T23:34:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:37 smithi120 ceph-mon[35250]: pgmap v667: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:34:37.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:37 smithi174 ceph-mon[35586]: pgmap v667: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:34:37.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:37 smithi089 ceph-mon[30933]: pgmap v667: 129 pgs: 129 active+clean; 1.1 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:34:37.717 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:34:37.717 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:34:37.966 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:38.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.458+0000 7faf2be97640 1 -- 172.21.15.89:0/1691308229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf2410c170 msgr2=0x7faf2410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.458+0000 7faf2be97640 1 --2- 172.21.15.89:0/1691308229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf2410c170 0x7faf2410e610 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7faf1c009ca0 tx=0x7faf1c02a950 comp rx=0 tx=0).stop 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.458+0000 7faf2be97640 1 -- 172.21.15.89:0/1691308229 shutdown_connections 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.458+0000 7faf2be97640 1 --2- 172.21.15.89:0/1691308229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf2410c170 0x7faf2410e610 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.458+0000 7faf2be97640 1 --2- 172.21.15.89:0/1691308229 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf24103660 0x7faf2410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.458+0000 7faf2be97640 1 --2- 172.21.15.89:0/1691308229 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf24102d40 0x7faf24103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.458+0000 7faf2be97640 1 -- 172.21.15.89:0/1691308229 >> 172.21.15.89:0/1691308229 conn(0x7faf240fc7a0 msgr2=0x7faf240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.459+0000 7faf2be97640 1 -- 172.21.15.89:0/1691308229 shutdown_connections 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.459+0000 7faf2be97640 1 -- 172.21.15.89:0/1691308229 wait complete. 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.459+0000 7faf2be97640 1 Processor -- start 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.459+0000 7faf2be97640 1 -- start start 2024-09-15T23:34:38.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2be97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf24102d40 0x7faf24198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2be97640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf24103660 0x7faf24198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2be97640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf2410c170 0x7faf2419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2be97640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faf24112000 con 0x7faf2410c170 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2be97640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faf24111e80 con 0x7faf24102d40 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2be97640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf24112180 con 0x7faf24103660 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2a434640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf2410c170 0x7faf2419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2a434640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf2410c170 0x7faf2419cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37804/0 (socket says 172.21.15.89:37804) 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2a434640 1 -- 172.21.15.89:0/3851394667 learned_addr learned my addr 172.21.15.89:0/3851394667 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf29432640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf24103660 0x7faf24198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf29c33640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf24102d40 0x7faf24198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2a434640 1 -- 172.21.15.89:0/3851394667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf24103660 msgr2=0x7faf24198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2a434640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf24103660 0x7faf24198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:38.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2a434640 1 -- 172.21.15.89:0/3851394667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf24102d40 msgr2=0x7faf24198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2a434640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf24102d40 0x7faf24198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2a434640 1 -- 172.21.15.89:0/3851394667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2419d3f0 con 0x7faf2410c170 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf29432640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf24103660 0x7faf24198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2a434640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf2410c170 0x7faf2419cd10 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7faf1c009dc0 tx=0x7faf1c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf1affd640 1 -- 172.21.15.89:0/3851394667 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c034040 con 0x7faf2410c170 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf1affd640 1 -- 172.21.15.89:0/3851394667 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf1c02fe20 con 0x7faf2410c170 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2be97640 1 -- 172.21.15.89:0/3851394667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2419d680 con 0x7faf2410c170 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf1affd640 1 -- 172.21.15.89:0/3851394667 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c0047f0 con 0x7faf2410c170 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.460+0000 7faf2be97640 1 -- 172.21.15.89:0/3851394667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf241a4fc0 con 0x7faf2410c170 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.461+0000 7faf2be97640 1 -- 172.21.15.89:0/3851394667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf24103120 con 0x7faf2410c170 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.463+0000 7faf1affd640 1 -- 172.21.15.89:0/3851394667 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7faf1c037020 con 0x7faf2410c170 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.463+0000 7faf1affd640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faf100a3240 0x7faf100a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.463+0000 7faf29c33640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faf100a3240 0x7faf100a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.463+0000 7faf1affd640 1 -- 172.21.15.89:0/3851394667 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7faf1c0f0d20 con 0x7faf2410c170 2024-09-15T23:34:38.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.464+0000 7faf29c33640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faf100a3240 0x7faf100a5700 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faf2000a810 tx=0x7faf20008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:38.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.468+0000 7faf1affd640 1 -- 172.21.15.89:0/3851394667 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7faf1c0bd650 con 0x7faf2410c170 2024-09-15T23:34:38.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.672+0000 7faf2be97640 1 -- 172.21.15.89:0/3851394667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf241a5190 con 0x7faf2410c170 2024-09-15T23:34:38.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.673+0000 7faf1affd640 1 -- 172.21.15.89:0/3851394667 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7faf1c0bcf50 con 0x7faf2410c170 2024-09-15T23:34:38.674 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:34:38.674 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:34:38.674 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:34:38.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 -- 172.21.15.89:0/3851394667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faf100a3240 msgr2=0x7faf100a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faf100a3240 0x7faf100a5700 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faf2000a810 tx=0x7faf20008040 comp rx=0 tx=0).stop 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 -- 172.21.15.89:0/3851394667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf2410c170 msgr2=0x7faf2419cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf2410c170 0x7faf2419cd10 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7faf1c009dc0 tx=0x7faf1c003f10 comp rx=0 tx=0).stop 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 -- 172.21.15.89:0/3851394667 shutdown_connections 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faf100a3240 0x7faf100a5700 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf2410c170 0x7faf2419cd10 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf24103660 0x7faf24198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 --2- 172.21.15.89:0/3851394667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf24102d40 0x7faf24198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 -- 172.21.15.89:0/3851394667 >> 172.21.15.89:0/3851394667 conn(0x7faf240fc7a0 msgr2=0x7faf240fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 -- 172.21.15.89:0/3851394667 shutdown_connections 2024-09-15T23:34:38.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:38.675+0000 7faf2be97640 1 -- 172.21.15.89:0/3851394667 wait complete. 2024-09-15T23:34:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:39 smithi089 ceph-mon[30933]: pgmap v668: 129 pgs: 129 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:34:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:39 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3851394667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:34:39.346 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:34:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:39 smithi120 ceph-mon[35250]: pgmap v668: 129 pgs: 129 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:34:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:39 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3851394667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:34:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:39 smithi174 ceph-mon[35586]: pgmap v668: 129 pgs: 129 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:34:39.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:39 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3851394667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:34:39.593 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:40.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.060+0000 7f9f28a01640 1 -- 172.21.15.89:0/3814176460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f24103550 msgr2=0x7f9f2410ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:40.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.060+0000 7f9f28a01640 1 --2- 172.21.15.89:0/3814176460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f24103550 0x7f9f2410ba40 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009900 tx=0x7f9f1802a480 comp rx=0 tx=0).stop 2024-09-15T23:34:40.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.061+0000 7f9f28a01640 1 -- 172.21.15.89:0/3814176460 shutdown_connections 2024-09-15T23:34:40.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.061+0000 7f9f28a01640 1 --2- 172.21.15.89:0/3814176460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f2410bfd0 0x7f9f2410e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.061+0000 7f9f28a01640 1 --2- 172.21.15.89:0/3814176460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f24103550 0x7f9f2410ba40 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.061+0000 7f9f28a01640 1 --2- 172.21.15.89:0/3814176460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f24102c30 0x7f9f24103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.061+0000 7f9f28a01640 1 -- 172.21.15.89:0/3814176460 >> 172.21.15.89:0/3814176460 conn(0x7f9f240fc610 msgr2=0x7f9f240fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.061+0000 7f9f28a01640 1 -- 172.21.15.89:0/3814176460 shutdown_connections 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.061+0000 7f9f28a01640 1 -- 172.21.15.89:0/3814176460 wait complete. 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.061+0000 7f9f28a01640 1 Processor -- start 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f28a01640 1 -- start start 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f28a01640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f24102c30 0x7f9f2419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f28a01640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f24103550 0x7f9f2419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f28a01640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2410bfd0 0x7f9f2419edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f28a01640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9f24111dc0 con 0x7f9f2410bfd0 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f28a01640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9f24111c40 con 0x7f9f24103550 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f28a01640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f24111f40 con 0x7f9f24102c30 2024-09-15T23:34:40.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f2359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2410bfd0 0x7f9f2419edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f2359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2410bfd0 0x7f9f2419edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37820/0 (socket says 172.21.15.89:37820) 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f2359e640 1 -- 172.21.15.89:0/1503657964 learned_addr learned my addr 172.21.15.89:0/1503657964 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f24102c30 0x7f9f2419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f2259c640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f24103550 0x7f9f2419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f2359e640 1 -- 172.21.15.89:0/1503657964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f24102c30 msgr2=0x7f9f2419a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f2359e640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f24102c30 0x7f9f2419a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f2359e640 1 -- 172.21.15.89:0/1503657964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f24103550 msgr2=0x7f9f2419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f2359e640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f24103550 0x7f9f2419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f2359e640 1 -- 172.21.15.89:0/1503657964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2419f4d0 con 0x7f9f2410bfd0 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.062+0000 7f9f2359e640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2410bfd0 0x7f9f2419edf0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00c970 tx=0x7f9f1c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.063+0000 7f9f0bfff640 1 -- 172.21.15.89:0/1503657964 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c007bb0 con 0x7f9f2410bfd0 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.063+0000 7f9f28a01640 1 -- 172.21.15.89:0/1503657964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f2419f7c0 con 0x7f9f2410bfd0 2024-09-15T23:34:40.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.063+0000 7f9f0bfff640 1 -- 172.21.15.89:0/1503657964 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f1c007d50 con 0x7f9f2410bfd0 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.063+0000 7f9f0bfff640 1 -- 172.21.15.89:0/1503657964 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c00f450 con 0x7f9f2410bfd0 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.063+0000 7f9f28a01640 1 -- 172.21.15.89:0/1503657964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f241a70a0 con 0x7f9f2410bfd0 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f0bfff640 1 -- 172.21.15.89:0/1503657964 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9f1c004f60 con 0x7f9f2410bfd0 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2b5db740 1 Processor -- start 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2b5db740 1 -- start start 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc57c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc57cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc589940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5608bc4e22f0 con 0x5608bc587550 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5608bc489cf0 con 0x5608bc57c7b0 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5608bc4de2b0 con 0x5608bc57dd20 2024-09-15T23:34:40.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc589940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc589940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37824/0 (socket says 172.21.15.89:37824) 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.065+0000 7f9f2259c640 1 -- 172.21.15.89:0/438687456 learned_addr learned my addr 172.21.15.89:0/438687456 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f0bfff640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f0c0a3240 0x7f9f0c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/438687456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc57cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f2359e640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f0c0a3240 0x7f9f0c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f2259c640 1 -- 172.21.15.89:0/438687456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 msgr2=0x5608bc57c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f2259c640 1 --2- 172.21.15.89:0/438687456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc57c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f2259c640 1 -- 172.21.15.89:0/438687456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 msgr2=0x5608bc57cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f2259c640 1 --2- 172.21.15.89:0/438687456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc57cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f2259c640 1 -- 172.21.15.89:0/438687456 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608bc46e3a0 con 0x5608bc587550 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/438687456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc57cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f0bfff640 1 -- 172.21.15.89:0/1503657964 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9f1c0cfa60 con 0x7f9f2410bfd0 2024-09-15T23:34:40.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f2259c640 1 --2- 172.21.15.89:0/438687456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc589940 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009900 tx=0x7f9f18032f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e77be4a34fd925a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f2359e640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f0c0a3240 0x7f9f0c0a5700 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c098660 tx=0x7f9f1c086040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f097fa640 1 -- 172.21.15.89:0/438687456 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f18041020 con 0x5608bc587550 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f097fa640 1 -- 172.21.15.89:0/438687456 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f180317b0 con 0x5608bc587550 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.066+0000 7f9f097fa640 1 -- 172.21.15.89:0/438687456 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f18035050 con 0x5608bc587550 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 -- 172.21.15.89:0/438687456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 msgr2=0x5608bc589940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/438687456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc589940 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009900 tx=0x7f9f18032f70 comp rx=0 tx=0).stop 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 -- 172.21.15.89:0/438687456 shutdown_connections 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/438687456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc589940 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/438687456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc57cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/438687456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc57c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 -- 172.21.15.89:0/438687456 >> 172.21.15.89:0/438687456 conn(0x5608bc545310 msgr2=0x5608bc578e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:40.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 -- 172.21.15.89:0/438687456 shutdown_connections 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 -- 172.21.15.89:0/438687456 wait complete. 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 -- start start 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.067+0000 7f9f2b5db740 1 Processor -- start 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc586f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc5841e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc5847b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5608bc58a1f0 con 0x5608bc587550 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5608bc49d060 con 0x5608bc57c7b0 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5608bc58a370 con 0x5608bc57dd20 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2b5db740 10 client.0 _ll_register_callbacks cb 0x5608bc1f08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc5847b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc5847b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37832/0 (socket says 172.21.15.89:37832) 2024-09-15T23:34:40.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2259c640 1 -- 172.21.15.89:0/2195992774 learned_addr learned my addr 172.21.15.89:0/2195992774 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc5841e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2259c640 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc586f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2259c640 1 -- 172.21.15.89:0/2195992774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 msgr2=0x5608bc5841e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2259c640 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc5841e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2259c640 1 -- 172.21.15.89:0/2195992774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 msgr2=0x5608bc586f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2259c640 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc586f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2259c640 1 -- 172.21.15.89:0/2195992774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608bc2533c0 con 0x5608bc587550 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc5841e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2259c640 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc5847b0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f9f1802a9a0 tx=0x7f9f180054b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b987373c525cb87c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f06ff5640 1 -- 172.21.15.89:0/2195992774 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1804f020 con 0x5608bc587550 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f06ff5640 1 -- 172.21.15.89:0/2195992774 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9f18005590 con 0x5608bc587550 2024-09-15T23:34:40.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.068+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2195992774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f180317b0 con 0x5608bc587550 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f06ff5640 1 -- 172.21.15.89:0/2195992774 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f18057050 con 0x5608bc587550 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f2b5db740 10 client.19011 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2195992774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5608bc595be0 con 0x5608bc587550 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f06ff5640 1 -- 172.21.15.89:0/2195992774 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f180317b0 con 0x5608bc587550 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f06ff5640 1 -- 172.21.15.89:0/2195992774 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9f18046570 con 0x5608bc587550 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f2b5db740 10 client.19011 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f2b5db740 10 client.19011 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f2b5db740 20 client.19011 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5608bc596f30 0x5608bc599320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f2b5db740 4 client.19011 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2195992774 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608bc4ef860 con 0x5608bc596f30 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.069+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5608bc596f30 0x5608bc599320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.070+0000 7f9f06ff5640 10 client.19011 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:34:40.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.070+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5608bc596f30 0x5608bc599320 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6618e255bf04de2f server_cookie=88ab576aaf451c04 in_seq=0 out_seq=0 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.070+0000 7f9f06ff5640 1 -- 172.21.15.89:0/2195992774 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5608bc4ef860 con 0x5608bc596f30 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.070+0000 7f9f06ff5640 10 client.19011 handle_command_reply: tid=0 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.070+0000 7f9f2b5db740 1 client.19011 shutdown 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2195992774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 msgr2=0x5608bc5847b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc5847b0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f9f1802a9a0 tx=0x7f9f180054b0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2195992774 shutdown_connections 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5608bc596f30 0x5608bc599320 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc5847b0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc5841e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f06ff5640 0 client.19011 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/2195992774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc586f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2195992774 >> 172.21.15.89:0/2195992774 conn(0x5608bc545310 msgr2=0x5608bc57eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:40.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2195992774 shutdown_connections 2024-09-15T23:34:40.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2195992774 wait complete. 2024-09-15T23:34:40.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.071+0000 7f9f2b5db740 20 client.19011 trim_cache size 0 max 16384 2024-09-15T23:34:40.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.083+0000 7f9f2b5db740 1 Processor -- start 2024-09-15T23:34:40.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.083+0000 7f9f2b5db740 1 -- start start 2024-09-15T23:34:40.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc578f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc593490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.100 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc59e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9f241b0e30 con 0x5608bc587550 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9f241b2970 con 0x5608bc57c7b0 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f241b1500 con 0x5608bc57dd20 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc59e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc59e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37844/0 (socket says 172.21.15.89:37844) 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2259c640 1 -- 172.21.15.89:0/2104923893 learned_addr learned my addr 172.21.15.89:0/2104923893 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/2104923893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc593490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2259c640 1 --2- 172.21.15.89:0/2104923893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc578f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2259c640 1 -- 172.21.15.89:0/2104923893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 msgr2=0x5608bc593490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:40.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2259c640 1 --2- 172.21.15.89:0/2104923893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc593490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2259c640 1 -- 172.21.15.89:0/2104923893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 msgr2=0x5608bc578f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2259c640 1 --2- 172.21.15.89:0/2104923893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc578f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2259c640 1 -- 172.21.15.89:0/2104923893 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608bc58a340 con 0x5608bc587550 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f2259c640 1 --2- 172.21.15.89:0/2104923893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc59e8f0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f9f18000c00 tx=0x7f9f1802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabf69fcce949b6a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f08ff9640 1 -- 172.21.15.89:0/2104923893 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1804f070 con 0x5608bc587550 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f08ff9640 1 -- 172.21.15.89:0/2104923893 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f18005040 con 0x5608bc587550 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.084+0000 7f9f08ff9640 1 -- 172.21.15.89:0/2104923893 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f18035070 con 0x5608bc587550 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2104923893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 msgr2=0x5608bc59e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/2104923893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc59e8f0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f9f18000c00 tx=0x7f9f1802a480 comp rx=0 tx=0).stop 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2104923893 shutdown_connections 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/2104923893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc587550 0x5608bc59e8f0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/2104923893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc57dd20 0x5608bc593490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/2104923893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc578f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2104923893 >> 172.21.15.89:0/2104923893 conn(0x5608bc545310 msgr2=0x5608bc578870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2104923893 shutdown_connections 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 -- 172.21.15.89:0/2104923893 wait complete. 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 -- start start 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.085+0000 7f9f2b5db740 1 Processor -- start 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc58ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc57dd20 0x5608bc5810b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2b5db740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc587550 0x5608bc5817f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9f241b22a0 con 0x5608bc57dd20 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9f241b1bd0 con 0x5608bc57c7b0 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2b5db740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f241aee10 con 0x5608bc587550 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2b5db740 10 client.0 _ll_register_callbacks cb 0x7f9f18032ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc57dd20 0x5608bc5810b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc57dd20 0x5608bc5810b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37852/0 (socket says 172.21.15.89:37852) 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f22d9d640 1 -- 172.21.15.89:0/3839301627 learned_addr learned my addr 172.21.15.89:0/3839301627 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2259c640 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc587550 0x5608bc5817f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2259c640 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc58ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f22d9d640 1 -- 172.21.15.89:0/3839301627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc587550 msgr2=0x5608bc5817f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc587550 0x5608bc5817f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f22d9d640 1 -- 172.21.15.89:0/3839301627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 msgr2=0x5608bc58ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc58ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f22d9d640 1 -- 172.21.15.89:0/3839301627 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608bc6008c0 con 0x5608bc57dd20 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2259c640 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc58ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f22d9d640 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc57dd20 0x5608bc5810b0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x5608bc579ca0 tx=0x7f9f14013920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e230dea666a4cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f06ff5640 1 -- 172.21.15.89:0/3839301627 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f14024670 con 0x5608bc57dd20 2024-09-15T23:34:40.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f06ff5640 1 -- 172.21.15.89:0/3839301627 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9f14024810 con 0x5608bc57dd20 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.086+0000 7f9f2b5db740 1 -- 172.21.15.89:0/3839301627 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f18005040 con 0x5608bc57dd20 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f06ff5640 1 -- 172.21.15.89:0/3839301627 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f1402d050 con 0x5608bc57dd20 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f2b5db740 10 client.19017 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f2b5db740 1 -- 172.21.15.89:0/3839301627 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5608bc601ab0 con 0x5608bc57dd20 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f06ff5640 1 -- 172.21.15.89:0/3839301627 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f18005040 con 0x5608bc57dd20 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f06ff5640 1 -- 172.21.15.89:0/3839301627 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9f1401b290 con 0x5608bc57dd20 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f2b5db740 10 client.19017 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f2b5db740 10 client.19017 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f2b5db740 20 client.19017 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5608bc602210 0x5608bc604630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f2b5db740 4 client.19017 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f2b5db740 1 -- 172.21.15.89:0/3839301627 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5608bc4e9d70 con 0x5608bc602210 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f2259c640 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5608bc602210 0x5608bc604630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:40.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f06ff5640 10 client.19017 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.087+0000 7f9f2259c640 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5608bc602210 0x5608bc604630 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=596e2125b4cd6951 server_cookie=1ee1a35563ce588b in_seq=0 out_seq=0 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f06ff5640 1 -- 172.21.15.89:0/3839301627 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5608bc4e9d70 con 0x5608bc602210 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f06ff5640 10 client.19017 handle_command_reply: tid=0 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 client.19017 shutdown 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 -- 172.21.15.89:0/3839301627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc57dd20 msgr2=0x5608bc5810b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc57dd20 0x5608bc5810b0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x5608bc579ca0 tx=0x7f9f14013920 comp rx=0 tx=0).stop 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 -- 172.21.15.89:0/3839301627 shutdown_connections 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5608bc602210 0x5608bc604630 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608bc587550 0x5608bc5817f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608bc57dd20 0x5608bc5810b0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f06ff5640 0 client.19017 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 --2- 172.21.15.89:0/3839301627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608bc57c7b0 0x5608bc58ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 -- 172.21.15.89:0/3839301627 >> 172.21.15.89:0/3839301627 conn(0x5608bc545310 msgr2=0x5608bc577780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:40.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 -- 172.21.15.89:0/3839301627 shutdown_connections 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 1 -- 172.21.15.89:0/3839301627 wait complete. 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f2b5db740 20 client.19017 trim_cache size 0 max 16384 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f28a01640 1 -- 172.21.15.89:0/1503657964 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f0c0a3240 msgr2=0x7f9f0c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f28a01640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f0c0a3240 0x7f9f0c0a5700 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c098660 tx=0x7f9f1c086040 comp rx=0 tx=0).stop 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f28a01640 1 -- 172.21.15.89:0/1503657964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2410bfd0 msgr2=0x7f9f2419edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.088+0000 7f9f28a01640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2410bfd0 0x7f9f2419edf0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c00c970 tx=0x7f9f1c00ce30 comp rx=0 tx=0).stop 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.089+0000 7f9f28a01640 1 -- 172.21.15.89:0/1503657964 shutdown_connections 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.089+0000 7f9f28a01640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f0c0a3240 0x7f9f0c0a5700 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.089+0000 7f9f28a01640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f2410bfd0 0x7f9f2419edf0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.089+0000 7f9f28a01640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f24103550 0x7f9f2419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.089+0000 7f9f28a01640 1 --2- 172.21.15.89:0/1503657964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f24102c30 0x7f9f2419a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.089+0000 7f9f28a01640 1 -- 172.21.15.89:0/1503657964 >> 172.21.15.89:0/1503657964 conn(0x7f9f240fc610 msgr2=0x7f9f24101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:40.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.089+0000 7f9f28a01640 1 -- 172.21.15.89:0/1503657964 shutdown_connections 2024-09-15T23:34:40.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:40.089+0000 7f9f28a01640 1 -- 172.21.15.89:0/1503657964 wait complete. 2024-09-15T23:34:40.828 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:34:41.074 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:41.366 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:41 smithi089 ceph-mon[30933]: pgmap v669: 129 pgs: 129 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:41 smithi120 ceph-mon[35250]: pgmap v669: 129 pgs: 129 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:41 smithi174 ceph-mon[35586]: pgmap v669: 129 pgs: 129 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:41.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.572+0000 7fe283699640 1 -- 172.21.15.89:0/3237741971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe27c105a70 msgr2=0x7fe27c105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:41.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.572+0000 7fe283699640 1 --2- 172.21.15.89:0/3237741971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe27c105a70 0x7fe27c105e50 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fe278009900 tx=0x7fe27802a410 comp rx=0 tx=0).stop 2024-09-15T23:34:41.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.572+0000 7fe283699640 1 -- 172.21.15.89:0/3237741971 shutdown_connections 2024-09-15T23:34:41.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.572+0000 7fe283699640 1 --2- 172.21.15.89:0/3237741971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe27c101ba0 0x7fe27c10d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.572+0000 7fe283699640 1 --2- 172.21.15.89:0/3237741971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe27c101220 0x7fe27c101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.572+0000 7fe283699640 1 --2- 172.21.15.89:0/3237741971 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe27c105a70 0x7fe27c105e50 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.572+0000 7fe283699640 1 -- 172.21.15.89:0/3237741971 >> 172.21.15.89:0/3237741971 conn(0x7fe27c0fbee0 msgr2=0x7fe27c0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.572+0000 7fe283699640 1 -- 172.21.15.89:0/3237741971 shutdown_connections 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.572+0000 7fe283699640 1 -- 172.21.15.89:0/3237741971 wait complete. 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe283699640 1 Processor -- start 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe283699640 1 -- start start 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe283699640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe27c101220 0x7fe27c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe283699640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe27c101ba0 0x7fe27c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe283699640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe27c105a70 0x7fe27c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe283699640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c112070 con 0x7fe27c101ba0 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe283699640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c111ef0 con 0x7fe27c105a70 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe283699640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c1121f0 con 0x7fe27c101220 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe280c34640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe27c101ba0 0x7fe27c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe280c34640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe27c101ba0 0x7fe27c1989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37858/0 (socket says 172.21.15.89:37858) 2024-09-15T23:34:41.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe280c34640 1 -- 172.21.15.89:0/1964992564 learned_addr learned my addr 172.21.15.89:0/1964992564 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe281435640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe27c101220 0x7fe27c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe280c34640 1 -- 172.21.15.89:0/1964992564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe27c101220 msgr2=0x7fe27c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe281c36640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe27c105a70 0x7fe27c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe280c34640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe27c101220 0x7fe27c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe280c34640 1 -- 172.21.15.89:0/1964992564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe27c105a70 msgr2=0x7fe27c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe280c34640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe27c105a70 0x7fe27c19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe280c34640 1 -- 172.21.15.89:0/1964992564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27c19d440 con 0x7fe27c101ba0 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.573+0000 7fe281435640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe27c101220 0x7fe27c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.574+0000 7fe280c34640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe27c101ba0 0x7fe27c1989d0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fe26c00d770 tx=0x7fe26c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.574+0000 7fe2727fc640 1 -- 172.21.15.89:0/1964992564 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c017070 con 0x7fe27c101ba0 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.574+0000 7fe283699640 1 -- 172.21.15.89:0/1964992564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe27c19d730 con 0x7fe27c101ba0 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.574+0000 7fe2727fc640 1 -- 172.21.15.89:0/1964992564 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe26c005500 con 0x7fe27c101ba0 2024-09-15T23:34:41.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.574+0000 7fe2727fc640 1 -- 172.21.15.89:0/1964992564 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c004e00 con 0x7fe27c101ba0 2024-09-15T23:34:41.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.574+0000 7fe283699640 1 -- 172.21.15.89:0/1964992564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe27c1a5010 con 0x7fe27c101ba0 2024-09-15T23:34:41.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.576+0000 7fe2727fc640 1 -- 172.21.15.89:0/1964992564 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe26c004fa0 con 0x7fe27c101ba0 2024-09-15T23:34:41.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.576+0000 7fe285a72740 1 Processor -- start 2024-09-15T23:34:41.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.576+0000 7fe285a72740 1 -- start start 2024-09-15T23:34:41.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe285a72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de88090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe285a72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 0x55d15de88a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe285a72740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de97780 0x55d15de99b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe285a72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d15ddee2f0 con 0x55d15de89b40 2024-09-15T23:34:41.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe285a72740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d15dd95cf0 con 0x55d15de885d0 2024-09-15T23:34:41.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe285a72740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d15ddea2b0 con 0x55d15de97780 2024-09-15T23:34:41.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe2727fc640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe2680a3240 0x7fe2680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de88090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de88090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37866/0 (socket says 172.21.15.89:37866) 2024-09-15T23:34:41.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 -- 172.21.15.89:0/3341980783 learned_addr learned my addr 172.21.15.89:0/3341980783 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:41.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe2680a3240 0x7fe2680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe2727fc640 1 -- 172.21.15.89:0/1964992564 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe26c0cf830 con 0x7fe27c101ba0 2024-09-15T23:34:41.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe280c34640 1 --2- 172.21.15.89:0/3341980783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de97780 0x55d15de99b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281435640 1 --2- 172.21.15.89:0/3341980783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 0x55d15de88a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 -- 172.21.15.89:0/3341980783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de97780 msgr2=0x55d15de99b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 --2- 172.21.15.89:0/3341980783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de97780 0x55d15de99b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 -- 172.21.15.89:0/3341980783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 msgr2=0x55d15de88a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 --2- 172.21.15.89:0/3341980783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 0x55d15de88a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 -- 172.21.15.89:0/3341980783 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d15dd7a3a0 con 0x55d15de89b40 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 --2- 172.21.15.89:0/3341980783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de88090 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fe274012e00 tx=0x7fe27400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c213f2ca4f04e3f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.577+0000 7fe281c36640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe2680a3240 0x7fe2680a5700 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe27400fc30 tx=0x7fe27401a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe2677fe640 1 -- 172.21.15.89:0/3341980783 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe274020070 con 0x55d15de89b40 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe2677fe640 1 -- 172.21.15.89:0/3341980783 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe27400ab30 con 0x55d15de89b40 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe2677fe640 1 -- 172.21.15.89:0/3341980783 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27400d5d0 con 0x55d15de89b40 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe285a72740 1 -- 172.21.15.89:0/3341980783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 msgr2=0x55d15de88090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe285a72740 1 --2- 172.21.15.89:0/3341980783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de88090 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fe274012e00 tx=0x7fe27400f730 comp rx=0 tx=0).stop 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe285a72740 1 -- 172.21.15.89:0/3341980783 shutdown_connections 2024-09-15T23:34:41.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe285a72740 1 --2- 172.21.15.89:0/3341980783 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de97780 0x55d15de99b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe285a72740 1 --2- 172.21.15.89:0/3341980783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 0x55d15de88a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe285a72740 1 --2- 172.21.15.89:0/3341980783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de88090 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe285a72740 1 -- 172.21.15.89:0/3341980783 >> 172.21.15.89:0/3341980783 conn(0x55d15de51310 msgr2=0x55d15de84c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe285a72740 1 -- 172.21.15.89:0/3341980783 shutdown_connections 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.578+0000 7fe285a72740 1 -- 172.21.15.89:0/3341980783 wait complete. 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe285a72740 1 -- start start 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe285a72740 1 Processor -- start 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe285a72740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de885d0 0x55d15de971b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe285a72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de89b40 0x55d15de94460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe285a72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de97780 0x55d15de94bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe285a72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d15de9a420 con 0x55d15de97780 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe285a72740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d15dda9060 con 0x55d15de89b40 2024-09-15T23:34:41.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe285a72740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d15de9a5a0 con 0x55d15de885d0 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe285a72740 10 client.0 _ll_register_callbacks cb 0x55d15dafc8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe281c36640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de97780 0x55d15de94bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe281c36640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de97780 0x55d15de94bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37868/0 (socket says 172.21.15.89:37868) 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe281c36640 1 -- 172.21.15.89:0/1567340755 learned_addr learned my addr 172.21.15.89:0/1567340755 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe281435640 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de885d0 0x55d15de971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.579+0000 7fe280c34640 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de89b40 0x55d15de94460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe281c36640 1 -- 172.21.15.89:0/1567340755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de885d0 msgr2=0x55d15de971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe281c36640 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de885d0 0x55d15de971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe281c36640 1 -- 172.21.15.89:0/1567340755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de89b40 msgr2=0x55d15de94460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe281c36640 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de89b40 0x55d15de94460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe281c36640 1 -- 172.21.15.89:0/1567340755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15db5f3c0 con 0x55d15de97780 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe281c36640 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de97780 0x55d15de94bf0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fe27401cdd0 tx=0x7fe274019ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43fde56e93fe24b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe264ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:41.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe264ff9640 1 -- 172.21.15.89:0/1567340755 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe274020070 con 0x55d15de97780 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe264ff9640 1 -- 172.21.15.89:0/1567340755 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe274004dd0 con 0x55d15de97780 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe285a72740 1 -- 172.21.15.89:0/1567340755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe27400ab30 con 0x55d15de97780 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe264ff9640 1 -- 172.21.15.89:0/1567340755 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe274036050 con 0x55d15de97780 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe285a72740 10 client.19029 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe285a72740 1 -- 172.21.15.89:0/1567340755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d15dea5f50 con 0x55d15de97780 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe264ff9640 1 -- 172.21.15.89:0/1567340755 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe27400ab30 con 0x55d15de97780 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe264ff9640 1 -- 172.21.15.89:0/1567340755 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe27401b070 con 0x55d15de97780 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe285a72740 10 client.19029 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe285a72740 10 client.19029 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe285a72740 20 client.19029 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.580+0000 7fe285a72740 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55d15dea7410 0x55d15dea9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.581+0000 7fe285a72740 4 client.19029 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.581+0000 7fe285a72740 1 -- 172.21.15.89:0/1567340755 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d15ddfb860 con 0x55d15dea7410 2024-09-15T23:34:41.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.581+0000 7fe281435640 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55d15dea7410 0x55d15dea9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.581+0000 7fe264ff9640 10 client.19029 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.581+0000 7fe281435640 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55d15dea7410 0x55d15dea9800 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1d5e5bfe465a32d8 server_cookie=65f77aaf9d52f611 in_seq=0 out_seq=0 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe264ff9640 1 -- 172.21.15.89:0/1567340755 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d15ddfb860 con 0x55d15dea7410 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe264ff9640 10 client.19029 handle_command_reply: tid=0 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 client.19029 shutdown 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 -- 172.21.15.89:0/1567340755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de97780 msgr2=0x55d15de94bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de97780 0x55d15de94bf0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fe27401cdd0 tx=0x7fe274019ce0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 -- 172.21.15.89:0/1567340755 shutdown_connections 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55d15dea7410 0x55d15dea9800 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de97780 0x55d15de94bf0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de89b40 0x55d15de94460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe264ff9640 0 client.19029 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:34:41.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 --2- 172.21.15.89:0/1567340755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de885d0 0x55d15de971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 -- 172.21.15.89:0/1567340755 >> 172.21.15.89:0/1567340755 conn(0x55d15de51310 msgr2=0x55d15de8a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 -- 172.21.15.89:0/1567340755 shutdown_connections 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 1 -- 172.21.15.89:0/1567340755 wait complete. 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.582+0000 7fe285a72740 20 client.19029 trim_cache size 0 max 16384 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe285a72740 1 Processor -- start 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe285a72740 1 -- start start 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe285a72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 0x55d15de93d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe285a72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15deaef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe285a72740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de97780 0x55d15deaf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe285a72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c1acb00 con 0x55d15de89b40 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe285a72740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c1ae640 con 0x55d15de885d0 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe285a72740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c1ad1d0 con 0x55d15de97780 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe280c34640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15deaef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe280c34640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15deaef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37870/0 (socket says 172.21.15.89:37870) 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe280c34640 1 -- 172.21.15.89:0/1942110488 learned_addr learned my addr 172.21.15.89:0/1942110488 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe281c36640 1 --2- 172.21.15.89:0/1942110488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de97780 0x55d15deaf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.595+0000 7fe281435640 1 --2- 172.21.15.89:0/1942110488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 0x55d15de93d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe280c34640 1 -- 172.21.15.89:0/1942110488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de97780 msgr2=0x55d15deaf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe280c34640 1 --2- 172.21.15.89:0/1942110488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de97780 0x55d15deaf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe280c34640 1 -- 172.21.15.89:0/1942110488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 msgr2=0x55d15de93d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe280c34640 1 --2- 172.21.15.89:0/1942110488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 0x55d15de93d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe280c34640 1 -- 172.21.15.89:0/1942110488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d15de9a360 con 0x55d15de89b40 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe281435640 1 --2- 172.21.15.89:0/1942110488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 0x55d15de93d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe280c34640 1 --2- 172.21.15.89:0/1942110488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15deaef70 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fe26c098430 tx=0x7fe26c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f84d337b47b2895 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe266ffd640 1 -- 172.21.15.89:0/1942110488 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c020040 con 0x55d15de89b40 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe266ffd640 1 -- 172.21.15.89:0/1942110488 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe26c09fe10 con 0x55d15de89b40 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe266ffd640 1 -- 172.21.15.89:0/1942110488 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c009de0 con 0x55d15de89b40 2024-09-15T23:34:41.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe285a72740 1 -- 172.21.15.89:0/1942110488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 msgr2=0x55d15deaef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.596+0000 7fe285a72740 1 --2- 172.21.15.89:0/1942110488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15deaef70 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fe26c098430 tx=0x7fe26c086040 comp rx=0 tx=0).stop 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 -- 172.21.15.89:0/1942110488 shutdown_connections 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 --2- 172.21.15.89:0/1942110488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de97780 0x55d15deaf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 --2- 172.21.15.89:0/1942110488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15deaef70 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 --2- 172.21.15.89:0/1942110488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de885d0 0x55d15de93d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 -- 172.21.15.89:0/1942110488 >> 172.21.15.89:0/1942110488 conn(0x55d15de51310 msgr2=0x55d15dea9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 -- 172.21.15.89:0/1942110488 shutdown_connections 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 -- 172.21.15.89:0/1942110488 wait complete. 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 -- start start 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 Processor -- start 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de885d0 0x55d15de9ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de9b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de97780 0x55d15df0ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c1adf70 con 0x55d15de89b40 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c1ad8a0 con 0x55d15de97780 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c1aaae0 con 0x55d15de885d0 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe285a72740 10 client.0 _ll_register_callbacks cb 0x7fe274012f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.597+0000 7fe280c34640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de9b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe280c34640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de9b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37886/0 (socket says 172.21.15.89:37886) 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe280c34640 1 -- 172.21.15.89:0/1625166774 learned_addr learned my addr 172.21.15.89:0/1625166774 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe281c36640 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de97780 0x55d15df0ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe281435640 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de885d0 0x55d15de9ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe280c34640 1 -- 172.21.15.89:0/1625166774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de885d0 msgr2=0x55d15de9ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe280c34640 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de885d0 0x55d15de9ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe280c34640 1 -- 172.21.15.89:0/1625166774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de97780 msgr2=0x55d15df0ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe280c34640 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de97780 0x55d15df0ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe280c34640 1 -- 172.21.15.89:0/1625166774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d15de85960 con 0x55d15de89b40 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe280c34640 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de9b2d0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0cd2b0 tx=0x7fe26c009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5cca1dc2433b40 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe264ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe264ff9640 1 -- 172.21.15.89:0/1625166774 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c0b8070 con 0x55d15de89b40 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe264ff9640 1 -- 172.21.15.89:0/1625166774 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe26c0980d0 con 0x55d15de89b40 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe285a72740 1 -- 172.21.15.89:0/1625166774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe26c09fe10 con 0x55d15de89b40 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe264ff9640 1 -- 172.21.15.89:0/1625166774 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe26c020070 con 0x55d15de89b40 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe285a72740 10 client.19035 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe285a72740 1 -- 172.21.15.89:0/1625166774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d15df11570 con 0x55d15de89b40 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.598+0000 7fe264ff9640 1 -- 172.21.15.89:0/1625166774 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe26c09fe10 con 0x55d15de89b40 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.599+0000 7fe264ff9640 1 -- 172.21.15.89:0/1625166774 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe26c09aeb0 con 0x55d15de89b40 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.599+0000 7fe285a72740 10 client.19035 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.599+0000 7fe285a72740 10 client.19035 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:34:41.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.599+0000 7fe285a72740 20 client.19035 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.599+0000 7fe285a72740 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55d15df11ce0 0x55d15df14100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.599+0000 7fe285a72740 4 client.19035 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.599+0000 7fe285a72740 1 -- 172.21.15.89:0/1625166774 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d15ddf5d70 con 0x55d15df11ce0 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.599+0000 7fe281435640 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55d15df11ce0 0x55d15df14100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.599+0000 7fe264ff9640 10 client.19035 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.599+0000 7fe281435640 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55d15df11ce0 0x55d15df14100 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1e773bf5d2269803 server_cookie=778086674fef933e in_seq=0 out_seq=0 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe264ff9640 1 -- 172.21.15.89:0/1625166774 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d15ddf5d70 con 0x55d15df11ce0 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe264ff9640 10 client.19035 handle_command_reply: tid=0 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 client.19035 shutdown 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 -- 172.21.15.89:0/1625166774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 msgr2=0x55d15de9b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de9b2d0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0cd2b0 tx=0x7fe26c009c30 comp rx=0 tx=0).stop 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 -- 172.21.15.89:0/1625166774 shutdown_connections 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55d15df11ce0 0x55d15df14100 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d15de97780 0x55d15df0ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe264ff9640 0 client.19035 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:34:41.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d15de89b40 0x55d15de9b2d0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 --2- 172.21.15.89:0/1625166774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d15de885d0 0x55d15de9ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 -- 172.21.15.89:0/1625166774 >> 172.21.15.89:0/1625166774 conn(0x55d15de51310 msgr2=0x55d15de92d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:41.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 -- 172.21.15.89:0/1625166774 shutdown_connections 2024-09-15T23:34:41.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 1 -- 172.21.15.89:0/1625166774 wait complete. 2024-09-15T23:34:41.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.600+0000 7fe285a72740 20 client.19035 trim_cache size 0 max 16384 2024-09-15T23:34:41.626 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 -- 172.21.15.89:0/1964992564 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe2680a3240 msgr2=0x7fe2680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe2680a3240 0x7fe2680a5700 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe27400fc30 tx=0x7fe27401a000 comp rx=0 tx=0).stop 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 -- 172.21.15.89:0/1964992564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe27c101ba0 msgr2=0x7fe27c1989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe27c101ba0 0x7fe27c1989d0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fe26c00d770 tx=0x7fe26c00dc30 comp rx=0 tx=0).stop 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 -- 172.21.15.89:0/1964992564 shutdown_connections 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe2680a3240 0x7fe2680a5700 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe27c105a70 0x7fe27c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe27c101ba0 0x7fe27c1989d0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 --2- 172.21.15.89:0/1964992564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe27c101220 0x7fe27c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 -- 172.21.15.89:0/1964992564 >> 172.21.15.89:0/1964992564 conn(0x7fe27c0fbee0 msgr2=0x7fe27c0fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 -- 172.21.15.89:0/1964992564 shutdown_connections 2024-09-15T23:34:41.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:41.601+0000 7fe283699640 1 -- 172.21.15.89:0/1964992564 wait complete. 2024-09-15T23:34:42.328 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:34:42.575 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.042+0000 7f44376ca640 1 -- 172.21.15.89:0/2664046079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430106f10 msgr2=0x7f443010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.042+0000 7f44376ca640 1 --2- 172.21.15.89:0/2664046079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430106f10 0x7f443010ea90 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f4428009ca0 tx=0x7f442802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.042+0000 7f44376ca640 1 -- 172.21.15.89:0/2664046079 shutdown_connections 2024-09-15T23:34:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.042+0000 7f44376ca640 1 --2- 172.21.15.89:0/2664046079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430106f10 0x7f443010ea90 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.042+0000 7f44376ca640 1 --2- 172.21.15.89:0/2664046079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4430102790 0x7f44301067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.042+0000 7f44376ca640 1 --2- 172.21.15.89:0/2664046079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4430101de0 0x7f44301021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.042+0000 7f44376ca640 1 -- 172.21.15.89:0/2664046079 >> 172.21.15.89:0/2664046079 conn(0x7f44300fd8c0 msgr2=0x7f44300ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.042+0000 7f44376ca640 1 -- 172.21.15.89:0/2664046079 shutdown_connections 2024-09-15T23:34:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.042+0000 7f44376ca640 1 -- 172.21.15.89:0/2664046079 wait complete. 2024-09-15T23:34:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f44376ca640 1 Processor -- start 2024-09-15T23:34:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f44376ca640 1 -- start start 2024-09-15T23:34:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f44376ca640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4430101de0 0x7f4430198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f44376ca640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4430102790 0x7f4430198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f44376ca640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430106f10 0x7f443019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f44376ca640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4430111f70 con 0x7f4430106f10 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f44376ca640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4430111df0 con 0x7f4430101de0 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f44376ca640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f44301120f0 con 0x7f4430102790 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f4435c67640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430106f10 0x7f443019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f4435c67640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430106f10 0x7f443019cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37902/0 (socket says 172.21.15.89:37902) 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f4435c67640 1 -- 172.21.15.89:0/760207716 learned_addr learned my addr 172.21.15.89:0/760207716 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f4434c65640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4430102790 0x7f4430198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.043+0000 7f4435466640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4430101de0 0x7f4430198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f4435c67640 1 -- 172.21.15.89:0/760207716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4430102790 msgr2=0x7f4430198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f4435c67640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4430102790 0x7f4430198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f4435c67640 1 -- 172.21.15.89:0/760207716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4430101de0 msgr2=0x7f4430198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f4435c67640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4430101de0 0x7f4430198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f4435c67640 1 -- 172.21.15.89:0/760207716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443019d2e0 con 0x7f4430106f10 2024-09-15T23:34:43.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f4435c67640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430106f10 0x7f443019cc00 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f4428031a20 tx=0x7f4428031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f44227fc640 1 -- 172.21.15.89:0/760207716 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428003b70 con 0x7f4430106f10 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f44376ca640 1 -- 172.21.15.89:0/760207716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443019d570 con 0x7f4430106f10 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f44227fc640 1 -- 172.21.15.89:0/760207716 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4428003d10 con 0x7f4430106f10 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f44227fc640 1 -- 172.21.15.89:0/760207716 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442802fc80 con 0x7f4430106f10 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.044+0000 7f44376ca640 1 -- 172.21.15.89:0/760207716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44301a4eb0 con 0x7f4430106f10 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.046+0000 7f44227fc640 1 -- 172.21.15.89:0/760207716 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f44280046a0 con 0x7f4430106f10 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4439aa3740 1 Processor -- start 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4439aa3740 1 -- start start 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cbe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21cbe5d0 0x559f21cbea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f44227fc640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f441c0a32c0 0x7f441c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21ccd780 0x559f21ccfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4439aa3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559f21c242f0 con 0x559f21cbfb40 2024-09-15T23:34:43.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4439aa3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559f21bcbcf0 con 0x559f21cbe5d0 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4439aa3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559f21c202b0 con 0x559f21ccd780 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4434c65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cbe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4434c65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cbe090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37904/0 (socket says 172.21.15.89:37904) 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4434c65640 1 -- 172.21.15.89:0/653660130 learned_addr learned my addr 172.21.15.89:0/653660130 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4435c67640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f441c0a32c0 0x7f441c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4435466640 1 --2- 172.21.15.89:0/653660130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21cbe5d0 0x559f21cbea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f44227fc640 1 -- 172.21.15.89:0/760207716 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f44280f1000 con 0x7f4430106f10 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4434c65640 1 --2- 172.21.15.89:0/653660130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21ccd780 0x559f21ccfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4434c65640 1 -- 172.21.15.89:0/653660130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21ccd780 msgr2=0x559f21ccfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4434c65640 1 --2- 172.21.15.89:0/653660130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21ccd780 0x559f21ccfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4434c65640 1 -- 172.21.15.89:0/653660130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21cbe5d0 msgr2=0x559f21cbea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4434c65640 1 --2- 172.21.15.89:0/653660130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21cbe5d0 0x559f21cbea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4434c65640 1 -- 172.21.15.89:0/653660130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f21bb03a0 con 0x559f21cbfb40 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.047+0000 7f4435466640 1 --2- 172.21.15.89:0/653660130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21cbe5d0 0x559f21cbea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.048+0000 7f4434c65640 1 --2- 172.21.15.89:0/653660130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cbe090 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f442c004650 tx=0x7f442c0041d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c25a35925b443d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.048+0000 7f4435c67640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f441c0a32c0 0x7f441c0a5780 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4428005e30 tx=0x7f442803d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.048+0000 7f441b7fe640 1 -- 172.21.15.89:0/653660130 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c005020 con 0x559f21cbfb40 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.048+0000 7f441b7fe640 1 -- 172.21.15.89:0/653660130 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f442c0056a0 con 0x559f21cbfb40 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.048+0000 7f441b7fe640 1 -- 172.21.15.89:0/653660130 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c010660 con 0x559f21cbfb40 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.048+0000 7f4439aa3740 1 -- 172.21.15.89:0/653660130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 msgr2=0x559f21cbe090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.048+0000 7f4439aa3740 1 --2- 172.21.15.89:0/653660130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cbe090 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f442c004650 tx=0x7f442c0041d0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.048+0000 7f4439aa3740 1 -- 172.21.15.89:0/653660130 shutdown_connections 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 --2- 172.21.15.89:0/653660130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21ccd780 0x559f21ccfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 --2- 172.21.15.89:0/653660130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21cbe5d0 0x559f21cbea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 --2- 172.21.15.89:0/653660130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cbe090 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 -- 172.21.15.89:0/653660130 >> 172.21.15.89:0/653660130 conn(0x559f21c87310 msgr2=0x559f21cbac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 -- 172.21.15.89:0/653660130 shutdown_connections 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 -- 172.21.15.89:0/653660130 wait complete. 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 -- start start 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 Processor -- start 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbe5d0 0x559f21ccd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21cca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559f21cd0420 con 0x559f21cbfb40 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559f21bdf060 con 0x559f21ccd780 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559f21cd05a0 con 0x559f21cbe5d0 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4439aa3740 10 client.0 _ll_register_callbacks cb 0x559f219328c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.049+0000 7f4435466640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4435466640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cca3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37910/0 (socket says 172.21.15.89:37910) 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4435466640 1 -- 172.21.15.89:0/3866796582 learned_addr learned my addr 172.21.15.89:0/3866796582 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4434c65640 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbe5d0 0x559f21ccd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4434c65640 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21cca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4435466640 1 -- 172.21.15.89:0/3866796582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbe5d0 msgr2=0x559f21ccd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4435466640 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbe5d0 0x559f21ccd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4435466640 1 -- 172.21.15.89:0/3866796582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 msgr2=0x559f21cca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4435466640 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21cca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4435466640 1 -- 172.21.15.89:0/3866796582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f219953c0 con 0x559f21cbfb40 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4435466640 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cca3a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f4424010a50 tx=0x7f4424010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8308eb5772cc13 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4418ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4418ff9640 1 -- 172.21.15.89:0/3866796582 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442401e060 con 0x559f21cbfb40 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4418ff9640 1 -- 172.21.15.89:0/3866796582 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f442400fbd0 con 0x559f21cbfb40 2024-09-15T23:34:43.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4439aa3740 1 -- 172.21.15.89:0/3866796582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f442c0056a0 con 0x559f21cbfb40 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4418ff9640 1 -- 172.21.15.89:0/3866796582 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4424025050 con 0x559f21cbfb40 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4439aa3740 10 client.19047 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.050+0000 7f4439aa3740 1 -- 172.21.15.89:0/3866796582 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559f21cdbe90 con 0x559f21cbfb40 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4418ff9640 1 -- 172.21.15.89:0/3866796582 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f442c0056a0 con 0x559f21cbfb40 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4418ff9640 1 -- 172.21.15.89:0/3866796582 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4424018400 con 0x559f21cbfb40 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4439aa3740 10 client.19047 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4439aa3740 10 client.19047 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4439aa3740 20 client.19047 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4439aa3740 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f21cdd3a0 0x559f21cdf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4439aa3740 4 client.19047 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4439aa3740 1 -- 172.21.15.89:0/3866796582 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f21c31860 con 0x559f21cdd3a0 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4434c65640 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f21cdd3a0 0x559f21cdf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4418ff9640 10 client.19047 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.051+0000 7f4434c65640 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f21cdd3a0 0x559f21cdf7c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec0ebe50c7a9f90a server_cookie=d538baa029e93d73 in_seq=0 out_seq=0 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4418ff9640 1 -- 172.21.15.89:0/3866796582 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559f21c31860 con 0x559f21cdd3a0 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4418ff9640 10 client.19047 handle_command_reply: tid=0 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 client.19047 shutdown 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 -- 172.21.15.89:0/3866796582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 msgr2=0x559f21cca3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cca3a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f4424010a50 tx=0x7f4424010f10 comp rx=0 tx=0).stop 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 -- 172.21.15.89:0/3866796582 shutdown_connections 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21cca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbfb40 0x559f21cca3a0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f21cdd3a0 0x559f21cdf7c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 --2- 172.21.15.89:0/3866796582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbe5d0 0x559f21ccd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4418ff9640 0 client.19047 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:34:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 -- 172.21.15.89:0/3866796582 >> 172.21.15.89:0/3866796582 conn(0x559f21c87310 msgr2=0x559f21cc0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 -- 172.21.15.89:0/3866796582 shutdown_connections 2024-09-15T23:34:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.052+0000 7f4439aa3740 1 -- 172.21.15.89:0/3866796582 wait complete. 2024-09-15T23:34:43.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.053+0000 7f4439aa3740 20 client.19047 trim_cache size 0 max 16384 2024-09-15T23:34:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.065+0000 7f4439aa3740 1 Processor -- start 2024-09-15T23:34:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.065+0000 7f4439aa3740 1 -- start start 2024-09-15T23:34:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.065+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.065+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbfb40 0x559f21ce5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21ce55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4439aa3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f44301acb40 con 0x559f21cbe5d0 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4439aa3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f44301ae680 con 0x559f21ccd780 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4439aa3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f44301ad210 con 0x559f21cbfb40 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4434c65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4434c65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37926/0 (socket says 172.21.15.89:37926) 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4434c65640 1 -- 172.21.15.89:0/4113560700 learned_addr learned my addr 172.21.15.89:0/4113560700 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4435466640 1 --2- 172.21.15.89:0/4113560700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbfb40 0x559f21ce5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4434c65640 1 -- 172.21.15.89:0/4113560700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbfb40 msgr2=0x559f21ce5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4434c65640 1 --2- 172.21.15.89:0/4113560700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbfb40 0x559f21ce5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4434c65640 1 -- 172.21.15.89:0/4113560700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 msgr2=0x559f21ce55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4434c65640 1 --2- 172.21.15.89:0/4113560700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21ce55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4434c65640 1 -- 172.21.15.89:0/4113560700 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f21cbf530 con 0x559f21cbe5d0 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f4434c65640 1 --2- 172.21.15.89:0/4113560700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd96c0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f442c00b620 tx=0x7f442c004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4dccf794634712 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f441affd640 1 -- 172.21.15.89:0/4113560700 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c003cf0 con 0x559f21cbe5d0 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f441affd640 1 -- 172.21.15.89:0/4113560700 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f442c034020 con 0x559f21cbe5d0 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.066+0000 7f441affd640 1 -- 172.21.15.89:0/4113560700 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c010460 con 0x559f21cbe5d0 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.067+0000 7f4439aa3740 1 -- 172.21.15.89:0/4113560700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 msgr2=0x559f21cd96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.067+0000 7f4439aa3740 1 --2- 172.21.15.89:0/4113560700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd96c0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f442c00b620 tx=0x7f442c004be0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.067+0000 7f4439aa3740 1 -- 172.21.15.89:0/4113560700 shutdown_connections 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.067+0000 7f4439aa3740 1 --2- 172.21.15.89:0/4113560700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21ce55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.067+0000 7f4439aa3740 1 --2- 172.21.15.89:0/4113560700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbfb40 0x559f21ce5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.067+0000 7f4439aa3740 1 --2- 172.21.15.89:0/4113560700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd96c0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.067+0000 7f4439aa3740 1 -- 172.21.15.89:0/4113560700 >> 172.21.15.89:0/4113560700 conn(0x559f21c87310 msgr2=0x559f21cc9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.067+0000 7f4439aa3740 1 -- 172.21.15.89:0/4113560700 shutdown_connections 2024-09-15T23:34:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.067+0000 7f4439aa3740 1 -- 172.21.15.89:0/4113560700 wait complete. 2024-09-15T23:34:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.067+0000 7f4439aa3740 1 -- start start 2024-09-15T23:34:43.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4439aa3740 1 Processor -- start 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbfb40 0x559f21cd5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4439aa3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21cd2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4439aa3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f44301adfb0 con 0x559f21cbe5d0 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4439aa3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f44301ad8e0 con 0x559f21ccd780 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4439aa3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f44301aab20 con 0x559f21cbfb40 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4439aa3740 10 client.0 _ll_register_callbacks cb 0x7f442400a800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4434c65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4434c65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37928/0 (socket says 172.21.15.89:37928) 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4434c65640 1 -- 172.21.15.89:0/160546837 learned_addr learned my addr 172.21.15.89:0/160546837 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4435466640 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbfb40 0x559f21cd5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4434c65640 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21cd2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4434c65640 1 -- 172.21.15.89:0/160546837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbfb40 msgr2=0x559f21cd5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4434c65640 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbfb40 0x559f21cd5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4434c65640 1 -- 172.21.15.89:0/160546837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 msgr2=0x559f21cd2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4434c65640 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21cd2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.068+0000 7f4434c65640 1 -- 172.21.15.89:0/160546837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f21cc1120 con 0x559f21cbe5d0 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4434c65640 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd5830 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f442c004d80 tx=0x7f442c00b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc6123c1697b230 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4418ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4418ff9640 1 -- 172.21.15.89:0/160546837 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442c003cf0 con 0x559f21cbe5d0 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4418ff9640 1 -- 172.21.15.89:0/160546837 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f442c030e80 con 0x559f21cbe5d0 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4439aa3740 1 -- 172.21.15.89:0/160546837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f442c034020 con 0x559f21cbe5d0 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4418ff9640 1 -- 172.21.15.89:0/160546837 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f442c004680 con 0x559f21cbe5d0 2024-09-15T23:34:43.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4439aa3740 10 client.19053 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4439aa3740 1 -- 172.21.15.89:0/160546837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559f21ccac70 con 0x559f21cbe5d0 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4418ff9640 1 -- 172.21.15.89:0/160546837 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f442c034020 con 0x559f21cbe5d0 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4418ff9640 1 -- 172.21.15.89:0/160546837 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f442c02e070 con 0x559f21cbe5d0 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4439aa3740 10 client.19053 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4439aa3740 10 client.19053 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4439aa3740 20 client.19053 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4439aa3740 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f21d487d0 0x559f21d4abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4439aa3740 4 client.19053 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.069+0000 7f4439aa3740 1 -- 172.21.15.89:0/160546837 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f21c2bd70 con 0x559f21d487d0 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4435466640 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f21d487d0 0x559f21d4abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4435466640 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f21d487d0 0x559f21d4abc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=25336e601259b135 server_cookie=78c94e21ed2f8b9 in_seq=0 out_seq=0 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4418ff9640 10 client.19053 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4418ff9640 1 -- 172.21.15.89:0/160546837 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f21c2bd70 con 0x559f21d487d0 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4418ff9640 10 client.19053 handle_command_reply: tid=0 2024-09-15T23:34:43.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4439aa3740 1 client.19053 shutdown 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4439aa3740 1 -- 172.21.15.89:0/160546837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 msgr2=0x559f21cd5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4439aa3740 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd5830 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f442c004d80 tx=0x7f442c00b950 comp rx=0 tx=0).stop 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4439aa3740 1 -- 172.21.15.89:0/160546837 shutdown_connections 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4439aa3740 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f21ccd780 0x559f21cd2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4439aa3740 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f21d487d0 0x559f21d4abc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4439aa3740 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f21cbfb40 0x559f21cd5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4418ff9640 0 client.19053 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4439aa3740 1 --2- 172.21.15.89:0/160546837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f21cbe5d0 0x559f21cd5830 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.070+0000 7f4439aa3740 1 -- 172.21.15.89:0/160546837 >> 172.21.15.89:0/160546837 conn(0x559f21c87310 msgr2=0x559f21ccbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f4439aa3740 1 -- 172.21.15.89:0/160546837 shutdown_connections 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f4439aa3740 1 -- 172.21.15.89:0/160546837 wait complete. 2024-09-15T23:34:43.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f4439aa3740 20 client.19053 trim_cache size 0 max 16384 2024-09-15T23:34:43.078 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:34:43.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 -- 172.21.15.89:0/760207716 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f441c0a32c0 msgr2=0x7f441c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:43.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f441c0a32c0 0x7f441c0a5780 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4428005e30 tx=0x7f442803d040 comp rx=0 tx=0).stop 2024-09-15T23:34:43.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 -- 172.21.15.89:0/760207716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430106f10 msgr2=0x7f443019cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:43.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430106f10 0x7f443019cc00 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f4428031a20 tx=0x7f4428031a50 comp rx=0 tx=0).stop 2024-09-15T23:34:43.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 -- 172.21.15.89:0/760207716 shutdown_connections 2024-09-15T23:34:43.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f441c0a32c0 0x7f441c0a5780 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4430106f10 0x7f443019cc00 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4430102790 0x7f4430198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 --2- 172.21.15.89:0/760207716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4430101de0 0x7f4430198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:43.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 -- 172.21.15.89:0/760207716 >> 172.21.15.89:0/760207716 conn(0x7f44300fd8c0 msgr2=0x7f44300febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:43.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 -- 172.21.15.89:0/760207716 shutdown_connections 2024-09-15T23:34:43.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:43.071+0000 7f44376ca640 1 -- 172.21.15.89:0/760207716 wait complete. 2024-09-15T23:34:43.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:43 smithi089 ceph-mon[30933]: pgmap v670: 129 pgs: 129 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:34:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:43 smithi120 ceph-mon[35250]: pgmap v670: 129 pgs: 129 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:34:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:43 smithi174 ceph-mon[35586]: pgmap v670: 129 pgs: 129 active+clean; 1.2 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:34:43.790 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:34:44.037 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:44.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.504+0000 7f4679bdf640 1 -- 172.21.15.89:0/892755566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4674101de0 msgr2=0x7f46741021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:44.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.504+0000 7f4679bdf640 1 --2- 172.21.15.89:0/892755566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4674101de0 0x7f46741021c0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f4664009900 tx=0x7f466402a410 comp rx=0 tx=0).stop 2024-09-15T23:34:44.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.504+0000 7f4679bdf640 1 -- 172.21.15.89:0/892755566 shutdown_connections 2024-09-15T23:34:44.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.504+0000 7f4679bdf640 1 --2- 172.21.15.89:0/892755566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4674106f10 0x7f467410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.504+0000 7f4679bdf640 1 --2- 172.21.15.89:0/892755566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4674102790 0x7f46741067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.504+0000 7f4679bdf640 1 --2- 172.21.15.89:0/892755566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4674101de0 0x7f46741021c0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.504+0000 7f4679bdf640 1 -- 172.21.15.89:0/892755566 >> 172.21.15.89:0/892755566 conn(0x7f46740fd8c0 msgr2=0x7f46740ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:44.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.504+0000 7f4679bdf640 1 -- 172.21.15.89:0/892755566 shutdown_connections 2024-09-15T23:34:44.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.504+0000 7f4679bdf640 1 -- 172.21.15.89:0/892755566 wait complete. 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4679bdf640 1 Processor -- start 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4679bdf640 1 -- start start 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4679bdf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4674101de0 0x7f4674198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4679bdf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4674102790 0x7f46741989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4679bdf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4674106f10 0x7f467419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4679bdf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4674111f90 con 0x7f4674102790 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4679bdf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4674111e10 con 0x7f4674106f10 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4679bdf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4674112110 con 0x7f4674101de0 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4672ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4674102790 0x7f46741989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4672ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4674102790 0x7f46741989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37950/0 (socket says 172.21.15.89:37950) 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4672ffd640 1 -- 172.21.15.89:0/3291632942 learned_addr learned my addr 172.21.15.89:0/3291632942 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f46737fe640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4674101de0 0x7f4674198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4673fff640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4674106f10 0x7f467419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4672ffd640 1 -- 172.21.15.89:0/3291632942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4674101de0 msgr2=0x7f4674198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4672ffd640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4674101de0 0x7f4674198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4672ffd640 1 -- 172.21.15.89:0/3291632942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4674106f10 msgr2=0x7f467419cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4672ffd640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4674106f10 0x7f467419cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4672ffd640 1 -- 172.21.15.89:0/3291632942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467419d430 con 0x7f4674102790 2024-09-15T23:34:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f46737fe640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4674101de0 0x7f4674198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.505+0000 7f4673fff640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4674106f10 0x7f467419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.506+0000 7f4672ffd640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4674102790 0x7f46741989c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f466800c890 tx=0x7f466800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.506+0000 7f4670ff9640 1 -- 172.21.15.89:0/3291632942 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668004220 con 0x7f4674102790 2024-09-15T23:34:44.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.506+0000 7f4679bdf640 1 -- 172.21.15.89:0/3291632942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f467419d720 con 0x7f4674102790 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.506+0000 7f4670ff9640 1 -- 172.21.15.89:0/3291632942 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46680043c0 con 0x7f4674102790 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.506+0000 7f4670ff9640 1 -- 172.21.15.89:0/3291632942 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668004a00 con 0x7f4674102790 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.506+0000 7f4679bdf640 1 -- 172.21.15.89:0/3291632942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46741a5000 con 0x7f4674102790 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.508+0000 7f4670ff9640 1 -- 172.21.15.89:0/3291632942 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f46680079b0 con 0x7f4674102790 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.508+0000 7f467bfb8740 1 Processor -- start 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.508+0000 7f467bfb8740 1 -- start start 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.508+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800de090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.508+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.508+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 0x5612800eb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.508+0000 7f467bfb8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5612800442f0 con 0x5612800de5d0 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.508+0000 7f467bfb8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56127ffebcf0 con 0x5612800dfb40 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.508+0000 7f467bfb8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5612800402b0 con 0x5612800e9370 2024-09-15T23:34:44.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f46737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f4670ff9640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f465c0a3240 0x7f465c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f46737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37960/0 (socket says 172.21.15.89:37960) 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f46737fe640 1 -- 172.21.15.89:0/1185318899 learned_addr learned my addr 172.21.15.89:0/1185318899 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f4672ffd640 1 --2- 172.21.15.89:0/1185318899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 0x5612800eb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f4673fff640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f465c0a3240 0x7f465c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f4673fff640 1 --2- 172.21.15.89:0/1185318899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800de090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f4670ff9640 1 -- 172.21.15.89:0/3291632942 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f46680cfab0 con 0x7f4674102790 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f46737fe640 1 -- 172.21.15.89:0/1185318899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 msgr2=0x5612800eb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f46737fe640 1 --2- 172.21.15.89:0/1185318899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 0x5612800eb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f46737fe640 1 -- 172.21.15.89:0/1185318899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 msgr2=0x5612800de090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f46737fe640 1 --2- 172.21.15.89:0/1185318899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800de090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f46737fe640 1 -- 172.21.15.89:0/1185318899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56127ffd03a0 con 0x5612800de5d0 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f4673fff640 1 --2- 172.21.15.89:0/1185318899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800de090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f46737fe640 1 --2- 172.21.15.89:0/1185318899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dea10 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4664009900 tx=0x7f46640316e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5f72cf755a1d82 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f4673fff640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f465c0a3240 0x7f465c0a5700 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f466c00e510 tx=0x7f466c0094d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f4659ffb640 1 -- 172.21.15.89:0/1185318899 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466403f070 con 0x5612800de5d0 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f4659ffb640 1 -- 172.21.15.89:0/1185318899 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46640035f0 con 0x5612800de5d0 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.509+0000 7f4659ffb640 1 -- 172.21.15.89:0/1185318899 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4664035070 con 0x5612800de5d0 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.510+0000 7f467bfb8740 1 -- 172.21.15.89:0/1185318899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 msgr2=0x5612800dea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.510+0000 7f467bfb8740 1 --2- 172.21.15.89:0/1185318899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dea10 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4664009900 tx=0x7f46640316e0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.510+0000 7f467bfb8740 1 -- 172.21.15.89:0/1185318899 shutdown_connections 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.510+0000 7f467bfb8740 1 --2- 172.21.15.89:0/1185318899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 0x5612800eb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.510+0000 7f467bfb8740 1 --2- 172.21.15.89:0/1185318899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dea10 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.510+0000 7f467bfb8740 1 --2- 172.21.15.89:0/1185318899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800de090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.510+0000 7f467bfb8740 1 -- 172.21.15.89:0/1185318899 >> 172.21.15.89:0/1185318899 conn(0x5612800a7310 msgr2=0x5612800dac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:44.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.510+0000 7f467bfb8740 1 -- 172.21.15.89:0/1185318899 shutdown_connections 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.510+0000 7f467bfb8740 1 -- 172.21.15.89:0/1185318899 wait complete. 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.510+0000 7f467bfb8740 1 -- start start 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f467bfb8740 1 Processor -- start 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800de5d0 0x5612800e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800e5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f467bfb8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5612800ec010 con 0x5612800e9370 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f467bfb8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56127ffff060 con 0x5612800dfb40 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f467bfb8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5612800ec190 con 0x5612800de5d0 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f467bfb8740 10 client.0 _ll_register_callbacks cb 0x56127fd528c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f4673fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f4673fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37976/0 (socket says 172.21.15.89:37976) 2024-09-15T23:34:44.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f4673fff640 1 -- 172.21.15.89:0/3595665707 learned_addr learned my addr 172.21.15.89:0/3595665707 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f4672ffd640 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800e5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f46737fe640 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800de5d0 0x5612800e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f4673fff640 1 -- 172.21.15.89:0/3595665707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800de5d0 msgr2=0x5612800e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f4673fff640 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800de5d0 0x5612800e8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f4673fff640 1 -- 172.21.15.89:0/3595665707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 msgr2=0x5612800e5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f4673fff640 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800e5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.511+0000 7f4673fff640 1 -- 172.21.15.89:0/3595665707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56127fdb53c0 con 0x5612800e9370 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f4673fff640 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e65e0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f466c00dc90 tx=0x7f466c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd670293b63c3e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f4652ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f4652ffd640 1 -- 172.21.15.89:0/3595665707 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c023600 con 0x5612800e9370 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f4652ffd640 1 -- 172.21.15.89:0/3595665707 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f466c0237a0 con 0x5612800e9370 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f467bfb8740 1 -- 172.21.15.89:0/3595665707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46640035f0 con 0x5612800e9370 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f4652ffd640 1 -- 172.21.15.89:0/3595665707 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f466c02c050 con 0x5612800e9370 2024-09-15T23:34:44.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f467bfb8740 10 client.19065 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f467bfb8740 1 -- 172.21.15.89:0/3595665707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5612800f7a50 con 0x5612800e9370 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f4652ffd640 1 -- 172.21.15.89:0/3595665707 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46640035f0 con 0x5612800e9370 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f4652ffd640 1 -- 172.21.15.89:0/3595665707 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f466c01b960 con 0x5612800e9370 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f467bfb8740 10 client.19065 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f467bfb8740 10 client.19065 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f467bfb8740 20 client.19065 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f467bfb8740 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5612800f8f10 0x5612800fb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f467bfb8740 4 client.19065 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.512+0000 7f467bfb8740 1 -- 172.21.15.89:0/3595665707 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561280051860 con 0x5612800f8f10 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.513+0000 7f46737fe640 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5612800f8f10 0x5612800fb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.513+0000 7f46737fe640 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5612800f8f10 0x5612800fb360 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3344fa281df37f73 server_cookie=b03793a460a09cff in_seq=0 out_seq=0 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.513+0000 7f4652ffd640 10 client.19065 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:34:44.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f4652ffd640 1 -- 172.21.15.89:0/3595665707 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561280051860 con 0x5612800f8f10 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f4652ffd640 10 client.19065 handle_command_reply: tid=0 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 client.19065 shutdown 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 -- 172.21.15.89:0/3595665707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 msgr2=0x5612800e65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e65e0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f466c00dc90 tx=0x7f466c004a40 comp rx=0 tx=0).stop 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 -- 172.21.15.89:0/3595665707 shutdown_connections 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5612800f8f10 0x5612800fb360 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e65e0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800e5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 --2- 172.21.15.89:0/3595665707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800de5d0 0x5612800e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f4652ffd640 0 client.19065 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 -- 172.21.15.89:0/3595665707 >> 172.21.15.89:0/3595665707 conn(0x5612800a7310 msgr2=0x5612800e0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 -- 172.21.15.89:0/3595665707 shutdown_connections 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 1 -- 172.21.15.89:0/3595665707 wait complete. 2024-09-15T23:34:44.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.514+0000 7f467bfb8740 20 client.19065 trim_cache size 0 max 16384 2024-09-15T23:34:44.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f467bfb8740 1 Processor -- start 2024-09-15T23:34:44.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f467bfb8740 1 -- start start 2024-09-15T23:34:44.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 0x5612800e1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f467bfb8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f46741aee60 con 0x5612800de5d0 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f467bfb8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f46741b09a0 con 0x5612800dfb40 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f467bfb8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f46741af530 con 0x5612800e9370 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f46737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f46737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37978/0 (socket says 172.21.15.89:37978) 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.527+0000 7f46737fe640 1 -- 172.21.15.89:0/3718815471 learned_addr learned my addr 172.21.15.89:0/3718815471 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f46737fe640 1 -- 172.21.15.89:0/3718815471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 msgr2=0x5612800e1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f4673fff640 1 --2- 172.21.15.89:0/3718815471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 0x5612800e1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f4672ffd640 1 --2- 172.21.15.89:0/3718815471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f46737fe640 1 --2- 172.21.15.89:0/3718815471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 0x5612800e1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f46737fe640 1 -- 172.21.15.89:0/3718815471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 msgr2=0x5612800f52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f46737fe640 1 --2- 172.21.15.89:0/3718815471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800f52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f46737fe640 1 -- 172.21.15.89:0/3718815471 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612800df300 con 0x5612800de5d0 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f4673fff640 1 --2- 172.21.15.89:0/3718815471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 0x5612800e1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:44.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f46737fe640 1 --2- 172.21.15.89:0/3718815471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dae10 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f4664009900 tx=0x7f4664000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab6bb247a9892c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:44.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f46597fa640 1 -- 172.21.15.89:0/3718815471 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4664035040 con 0x5612800de5d0 2024-09-15T23:34:44.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f46597fa640 1 -- 172.21.15.89:0/3718815471 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4664004450 con 0x5612800de5d0 2024-09-15T23:34:44.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.528+0000 7f46597fa640 1 -- 172.21.15.89:0/3718815471 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466403f040 con 0x5612800de5d0 2024-09-15T23:34:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 -- 172.21.15.89:0/3718815471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 msgr2=0x5612800dae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 --2- 172.21.15.89:0/3718815471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dae10 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f4664009900 tx=0x7f4664000bc0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 -- 172.21.15.89:0/3718815471 shutdown_connections 2024-09-15T23:34:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 --2- 172.21.15.89:0/3718815471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800e9370 0x5612800e1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 --2- 172.21.15.89:0/3718815471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800f52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 --2- 172.21.15.89:0/3718815471 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800de5d0 0x5612800dae10 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 -- 172.21.15.89:0/3718815471 >> 172.21.15.89:0/3718815471 conn(0x5612800a7310 msgr2=0x5612800da700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 -- 172.21.15.89:0/3718815471 shutdown_connections 2024-09-15T23:34:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 -- 172.21.15.89:0/3718815471 wait complete. 2024-09-15T23:34:44.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 -- start start 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 Processor -- start 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800de5d0 0x5612800f1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800f1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f46741b02d0 con 0x5612800e9370 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.529+0000 7f467bfb8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f46741afc00 con 0x5612800dfb40 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f467bfb8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f46741ace40 con 0x5612800de5d0 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f467bfb8740 10 client.0 _ll_register_callbacks cb 0x7f466c009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4673fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4673fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37980/0 (socket says 172.21.15.89:37980) 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4673fff640 1 -- 172.21.15.89:0/2370793317 learned_addr learned my addr 172.21.15.89:0/2370793317 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4672ffd640 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800f1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f46737fe640 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800de5d0 0x5612800f1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4673fff640 1 -- 172.21.15.89:0/2370793317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800de5d0 msgr2=0x5612800f1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4673fff640 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800de5d0 0x5612800f1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4673fff640 1 -- 172.21.15.89:0/2370793317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 msgr2=0x5612800f1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4673fff640 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800f1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4673fff640 1 -- 172.21.15.89:0/2370793317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612800f2020 con 0x5612800e9370 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4673fff640 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e22c0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f466c019630 tx=0x7f466c019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d58b3001f421df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4652ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4652ffd640 1 -- 172.21.15.89:0/2370793317 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c00cc40 con 0x5612800e9370 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4652ffd640 1 -- 172.21.15.89:0/2370793317 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f466c00d0c0 con 0x5612800e9370 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f467bfb8740 1 -- 172.21.15.89:0/2370793317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4664004450 con 0x5612800e9370 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f4652ffd640 1 -- 172.21.15.89:0/2370793317 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f466c00d2c0 con 0x5612800e9370 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f467bfb8740 10 client.19071 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.530+0000 7f467bfb8740 1 -- 172.21.15.89:0/2370793317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561280163130 con 0x5612800e9370 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.531+0000 7f4652ffd640 1 -- 172.21.15.89:0/2370793317 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4664004450 con 0x5612800e9370 2024-09-15T23:34:44.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.531+0000 7f4652ffd640 1 -- 172.21.15.89:0/2370793317 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f466c019930 con 0x5612800e9370 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.531+0000 7f467bfb8740 10 client.19071 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.531+0000 7f467bfb8740 10 client.19071 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.531+0000 7f467bfb8740 20 client.19071 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.531+0000 7f467bfb8740 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5612801638b0 0x561280165cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.531+0000 7f467bfb8740 4 client.19071 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.531+0000 7f467bfb8740 1 -- 172.21.15.89:0/2370793317 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56128004bd70 con 0x5612801638b0 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.531+0000 7f46737fe640 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5612801638b0 0x561280165cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f4652ffd640 10 client.19071 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f46737fe640 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5612801638b0 0x561280165cd0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=11ac5514cb5aaada server_cookie=38eb6bfcec506bd1 in_seq=0 out_seq=0 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f4652ffd640 1 -- 172.21.15.89:0/2370793317 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56128004bd70 con 0x5612801638b0 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f4652ffd640 10 client.19071 handle_command_reply: tid=0 2024-09-15T23:34:44.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 client.19071 shutdown 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 -- 172.21.15.89:0/2370793317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 msgr2=0x5612800e22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e22c0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f466c019630 tx=0x7f466c019660 comp rx=0 tx=0).stop 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 -- 172.21.15.89:0/2370793317 shutdown_connections 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5612801638b0 0x561280165cd0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612800e9370 0x5612800e22c0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612800dfb40 0x5612800f1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 --2- 172.21.15.89:0/2370793317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5612800de5d0 0x5612800f1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 -- 172.21.15.89:0/2370793317 >> 172.21.15.89:0/2370793317 conn(0x5612800a7310 msgr2=0x5612800e7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f4652ffd640 0 client.19071 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 -- 172.21.15.89:0/2370793317 shutdown_connections 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 1 -- 172.21.15.89:0/2370793317 wait complete. 2024-09-15T23:34:44.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.532+0000 7f467bfb8740 20 client.19071 trim_cache size 0 max 16384 2024-09-15T23:34:44.539 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:34:44.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 -- 172.21.15.89:0/3291632942 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f465c0a3240 msgr2=0x7f465c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:44.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f465c0a3240 0x7f465c0a5700 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f466c00e510 tx=0x7f466c0094d0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 -- 172.21.15.89:0/3291632942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4674102790 msgr2=0x7f46741989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:44.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4674102790 0x7f46741989c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f466800c890 tx=0x7f466800cd50 comp rx=0 tx=0).stop 2024-09-15T23:34:44.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 -- 172.21.15.89:0/3291632942 shutdown_connections 2024-09-15T23:34:44.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f465c0a3240 0x7f465c0a5700 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4674106f10 0x7f467419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4674102790 0x7f46741989c0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 --2- 172.21.15.89:0/3291632942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4674101de0 0x7f4674198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:44.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 -- 172.21.15.89:0/3291632942 >> 172.21.15.89:0/3291632942 conn(0x7f46740fd8c0 msgr2=0x7f46740fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:44.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 -- 172.21.15.89:0/3291632942 shutdown_connections 2024-09-15T23:34:44.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:44.533+0000 7f4679bdf640 1 -- 172.21.15.89:0/3291632942 wait complete. 2024-09-15T23:34:45.406 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:45 smithi089 ceph-mon[30933]: pgmap v671: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:34:45.440 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:34:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:45 smithi120 ceph-mon[35250]: pgmap v671: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:34:45.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:45 smithi174 ceph-mon[35586]: pgmap v671: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.500+0000 7fecb54af640 1 -- 172.21.15.89:0/42799153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecb0101660 msgr2=0x7fecb0108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.500+0000 7fecb54af640 1 --2- 172.21.15.89:0/42799153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecb0101660 0x7fecb0108f00 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7feca8009ca0 tx=0x7feca802a8b0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.500+0000 7fecb54af640 1 -- 172.21.15.89:0/42799153 shutdown_connections 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.500+0000 7fecb54af640 1 --2- 172.21.15.89:0/42799153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecb0101660 0x7fecb0108f00 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.500+0000 7fecb54af640 1 --2- 172.21.15.89:0/42799153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecb0100ca0 0x7fecb0101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.500+0000 7fecb54af640 1 --2- 172.21.15.89:0/42799153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecb01020e0 0x7fecb01024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 -- 172.21.15.89:0/42799153 >> 172.21.15.89:0/42799153 conn(0x7fecb00fb980 msgr2=0x7fecb00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 -- 172.21.15.89:0/42799153 shutdown_connections 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 -- 172.21.15.89:0/42799153 wait complete. 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 Processor -- start 2024-09-15T23:34:45.504 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 -- start start 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecb0100ca0 0x7fecb0100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecb0101660 0x7fecb019ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecb01020e0 0x7fecb019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fecb010c710 con 0x7fecb01020e0 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fecb010c590 con 0x7fecb0101660 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.501+0000 7fecb54af640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fecb010c890 con 0x7fecb0100ca0 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecaf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecb01020e0 0x7fecb019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecaf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecb01020e0 0x7fecb019f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59464/0 (socket says 172.21.15.89:59464) 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecaf7fe640 1 -- 172.21.15.89:0/3203974720 learned_addr learned my addr 172.21.15.89:0/3203974720 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecaeffd640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecb0100ca0 0x7fecb0100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecae7fc640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecb0101660 0x7fecb019ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.505 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecaf7fe640 1 -- 172.21.15.89:0/3203974720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecb0100ca0 msgr2=0x7fecb0100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecaf7fe640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecb0100ca0 0x7fecb0100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecaf7fe640 1 -- 172.21.15.89:0/3203974720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecb0101660 msgr2=0x7fecb019ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecaf7fe640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecb0101660 0x7fecb019ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecaf7fe640 1 -- 172.21.15.89:0/3203974720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb019f990 con 0x7fecb01020e0 2024-09-15T23:34:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecaf7fe640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecb01020e0 0x7fecb019f080 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7feca8009dc0 tx=0x7feca8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fec97fff640 1 -- 172.21.15.89:0/3203974720 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8004450 con 0x7fecb01020e0 2024-09-15T23:34:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fec97fff640 1 -- 172.21.15.89:0/3203974720 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feca8040070 con 0x7fecb01020e0 2024-09-15T23:34:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecb54af640 1 -- 172.21.15.89:0/3203974720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb019fbc0 con 0x7fecb01020e0 2024-09-15T23:34:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fec97fff640 1 -- 172.21.15.89:0/3203974720 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8045e10 con 0x7fecb01020e0 2024-09-15T23:34:45.506 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.502+0000 7fecb54af640 1 -- 172.21.15.89:0/3203974720 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb01a0110 con 0x7fecb01020e0 2024-09-15T23:34:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fec97fff640 1 -- 172.21.15.89:0/3203974720 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7feca80045f0 con 0x7fecb01020e0 2024-09-15T23:34:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecb8240740 1 Processor -- start 2024-09-15T23:34:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecb8240740 1 -- start start 2024-09-15T23:34:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecb8240740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b79a0 0x5632c05b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecb8240740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05b82e0 0x5632c05bcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecb8240740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05bd370 0x5632c05bf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecb8240740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5632c051b2f0 con 0x5632c05b79a0 2024-09-15T23:34:45.513 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecb8240740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5632c04c3270 con 0x5632c05bd370 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecb8240740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5632c05172b0 con 0x5632c05b82e0 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fec97fff640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fec980a32f0 0x7fec980a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecae7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b79a0 0x5632c05b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecae7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b79a0 0x5632c05b7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59470/0 (socket says 172.21.15.89:59470) 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.505+0000 7fecae7fc640 1 -- 172.21.15.89:0/2099544817 learned_addr learned my addr 172.21.15.89:0/2099544817 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fec97fff640 1 -- 172.21.15.89:0/3203974720 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7feca80f11d0 con 0x7fecb01020e0 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fecaf7fe640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fec980a32f0 0x7fec980a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fecaeffd640 1 --2- 172.21.15.89:0/2099544817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05b82e0 0x5632c05bcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fecae7fc640 1 --2- 172.21.15.89:0/2099544817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05bd370 0x5632c05bf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fecae7fc640 1 -- 172.21.15.89:0/2099544817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05b82e0 msgr2=0x5632c05bcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fecae7fc640 1 --2- 172.21.15.89:0/2099544817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05b82e0 0x5632c05bcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fecae7fc640 1 -- 172.21.15.89:0/2099544817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05bd370 msgr2=0x5632c05bf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fecae7fc640 1 --2- 172.21.15.89:0/2099544817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05bd370 0x5632c05bf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.514 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fecae7fc640 1 -- 172.21.15.89:0/2099544817 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632c04c2340 con 0x5632c05b79a0 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fecae7fc640 1 --2- 172.21.15.89:0/2099544817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b79a0 0x5632c05b7da0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7feca400bed0 tx=0x7feca400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b690a612cc1017 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fecaf7fe640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fec980a32f0 0x7fec980a57b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7feca80c5b40 tx=0x7feca80dd040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fec957fa640 1 -- 172.21.15.89:0/2099544817 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca4011440 con 0x5632c05b79a0 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fec957fa640 1 -- 172.21.15.89:0/2099544817 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feca4017030 con 0x5632c05b79a0 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.506+0000 7fec957fa640 1 -- 172.21.15.89:0/2099544817 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca401b950 con 0x5632c05b79a0 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 -- 172.21.15.89:0/2099544817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b79a0 msgr2=0x5632c05b7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 --2- 172.21.15.89:0/2099544817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b79a0 0x5632c05b7da0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7feca400bed0 tx=0x7feca400b8e0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 -- 172.21.15.89:0/2099544817 shutdown_connections 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 --2- 172.21.15.89:0/2099544817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05bd370 0x5632c05bf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 --2- 172.21.15.89:0/2099544817 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05b82e0 0x5632c05bcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 --2- 172.21.15.89:0/2099544817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b79a0 0x5632c05b7da0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 -- 172.21.15.89:0/2099544817 >> 172.21.15.89:0/2099544817 conn(0x5632c05b3010 msgr2=0x5632c05b3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 -- 172.21.15.89:0/2099544817 shutdown_connections 2024-09-15T23:34:45.515 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 -- 172.21.15.89:0/2099544817 wait complete. 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 -- start start 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.507+0000 7fecb8240740 1 Processor -- start 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecb8240740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05b79a0 0x5632c05c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecb8240740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b82e0 0x5632c05c2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecb8240740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05bd370 0x5632c05c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecb8240740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5632c05c0340 con 0x5632c05bd370 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecb8240740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5632c04a4290 con 0x5632c05b82e0 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecb8240740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5632c05c04c0 con 0x5632c05b79a0 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecb8240740 10 client.0 _ll_register_callbacks cb 0x5632c04d5140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecae7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05bd370 0x5632c05c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecae7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05bd370 0x5632c05c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59482/0 (socket says 172.21.15.89:59482) 2024-09-15T23:34:45.516 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecae7fc640 1 -- 172.21.15.89:0/716663755 learned_addr learned my addr 172.21.15.89:0/716663755 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecae7fc640 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05b79a0 0x5632c05c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecaeffd640 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b82e0 0x5632c05c2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecae7fc640 1 -- 172.21.15.89:0/716663755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05b79a0 msgr2=0x5632c05c1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecae7fc640 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05b79a0 0x5632c05c1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecae7fc640 1 -- 172.21.15.89:0/716663755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b82e0 msgr2=0x5632c05c2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecae7fc640 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b82e0 0x5632c05c2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecae7fc640 1 -- 172.21.15.89:0/716663755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632c022e210 con 0x5632c05bd370 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecae7fc640 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05bd370 0x5632c05c9170 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7feca400bb10 tx=0x7feca4015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f5313f1b16e631 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fec92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fec92ff5640 1 -- 172.21.15.89:0/716663755 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca4004020 con 0x5632c05bd370 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fec92ff5640 1 -- 172.21.15.89:0/716663755 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7feca40041c0 con 0x5632c05bd370 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.508+0000 7fecb8240740 1 -- 172.21.15.89:0/716663755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feca4017030 con 0x5632c05bd370 2024-09-15T23:34:45.517 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fec92ff5640 1 -- 172.21.15.89:0/716663755 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feca4031050 con 0x5632c05bd370 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fecb8240740 10 client.19083 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fecb8240740 1 -- 172.21.15.89:0/716663755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5632c02fa620 con 0x5632c05bd370 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fec92ff5640 1 -- 172.21.15.89:0/716663755 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feca4017030 con 0x5632c05bd370 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fec92ff5640 1 -- 172.21.15.89:0/716663755 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7feca401bc90 con 0x5632c05bd370 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fecb8240740 10 client.19083 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fecb8240740 10 client.19083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fecb8240740 20 client.19083 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fecb8240740 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632c05cd5e0 0x5632c05cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fecb8240740 4 client.19083 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fecb8240740 1 -- 172.21.15.89:0/716663755 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632c0528a20 con 0x5632c05cd5e0 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.509+0000 7fecaeffd640 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632c05cd5e0 0x5632c05cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.510+0000 7fec92ff5640 10 client.19083 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:45.518 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.510+0000 7fecaeffd640 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632c05cd5e0 0x5632c05cf9d0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fab13f512aa6d558 server_cookie=f483959d039c6354 in_seq=0 out_seq=0 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.510+0000 7fec92ff5640 1 -- 172.21.15.89:0/716663755 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5632c0528a20 con 0x5632c05cd5e0 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.510+0000 7fec92ff5640 10 client.19083 handle_command_reply: tid=0 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.510+0000 7fecb8240740 1 client.19083 shutdown 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.510+0000 7fecb8240740 1 -- 172.21.15.89:0/716663755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05bd370 msgr2=0x5632c05c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.510+0000 7fecb8240740 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05bd370 0x5632c05c9170 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7feca400bb10 tx=0x7feca4015040 comp rx=0 tx=0).stop 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.510+0000 7fecb8240740 1 -- 172.21.15.89:0/716663755 shutdown_connections 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.510+0000 7fecb8240740 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632c05cd5e0 0x5632c05cf9d0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.511+0000 7fecb8240740 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05bd370 0x5632c05c9170 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.511+0000 7fecb8240740 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b82e0 0x5632c05c2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.511+0000 7fec92ff5640 0 client.19083 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.511+0000 7fecb8240740 1 --2- 172.21.15.89:0/716663755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05b79a0 0x5632c05c1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.511+0000 7fecb8240740 1 -- 172.21.15.89:0/716663755 >> 172.21.15.89:0/716663755 conn(0x5632c05b3010 msgr2=0x5632c05b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.511+0000 7fecb8240740 1 -- 172.21.15.89:0/716663755 shutdown_connections 2024-09-15T23:34:45.519 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.511+0000 7fecb8240740 1 -- 172.21.15.89:0/716663755 wait complete. 2024-09-15T23:34:45.520 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.511+0000 7fecb8240740 20 client.19083 trim_cache size 0 max 16384 2024-09-15T23:34:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecb8240740 1 Processor -- start 2024-09-15T23:34:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecb8240740 1 -- start start 2024-09-15T23:34:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecb8240740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 0x5632c05b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecb8240740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecb8240740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05bd370 0x5632c05b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecb8240740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fecb01aa710 con 0x5632c05b82e0 2024-09-15T23:34:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecb8240740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fecb01ac250 con 0x5632c05b79a0 2024-09-15T23:34:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecb8240740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fecb01aade0 con 0x5632c05bd370 2024-09-15T23:34:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecaeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecaeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59494/0 (socket says 172.21.15.89:59494) 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecaeffd640 1 -- 172.21.15.89:0/3702800287 learned_addr learned my addr 172.21.15.89:0/3702800287 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecae7fc640 1 --2- 172.21.15.89:0/3702800287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 0x5632c05b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecaeffd640 1 -- 172.21.15.89:0/3702800287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05bd370 msgr2=0x5632c05b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecaeffd640 1 --2- 172.21.15.89:0/3702800287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05bd370 0x5632c05b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecaeffd640 1 -- 172.21.15.89:0/3702800287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 msgr2=0x5632c05b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecaeffd640 1 --2- 172.21.15.89:0/3702800287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 0x5632c05b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecaeffd640 1 -- 172.21.15.89:0/3702800287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632c05b3c90 con 0x5632c05b82e0 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecae7fc640 1 --2- 172.21.15.89:0/3702800287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 0x5632c05b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.525+0000 7fecaeffd640 1 --2- 172.21.15.89:0/3702800287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c9130 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7feca0005e10 tx=0x7feca0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157c49fccfcb7a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fec94ff9640 1 -- 172.21.15.89:0/3702800287 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca001d620 con 0x5632c05b82e0 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fec94ff9640 1 -- 172.21.15.89:0/3702800287 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feca001dca0 con 0x5632c05b82e0 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fec94ff9640 1 -- 172.21.15.89:0/3702800287 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca000fe20 con 0x5632c05b82e0 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fecb8240740 1 -- 172.21.15.89:0/3702800287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 msgr2=0x5632c05c9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fecb8240740 1 --2- 172.21.15.89:0/3702800287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c9130 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7feca0005e10 tx=0x7feca0011810 comp rx=0 tx=0).stop 2024-09-15T23:34:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fecb8240740 1 -- 172.21.15.89:0/3702800287 shutdown_connections 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fecb8240740 1 --2- 172.21.15.89:0/3702800287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05bd370 0x5632c05b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fecb8240740 1 --2- 172.21.15.89:0/3702800287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c9130 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fecb8240740 1 --2- 172.21.15.89:0/3702800287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 0x5632c05b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fecb8240740 1 -- 172.21.15.89:0/3702800287 >> 172.21.15.89:0/3702800287 conn(0x5632c05b3010 msgr2=0x5632c05c1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fecb8240740 1 -- 172.21.15.89:0/3702800287 shutdown_connections 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.526+0000 7fecb8240740 1 -- 172.21.15.89:0/3702800287 wait complete. 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecb8240740 1 -- start start 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecb8240740 1 Processor -- start 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecb8240740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 0x5632c05c6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecb8240740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecb8240740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05bd370 0x5632c05c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecb8240740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fecb01abb80 con 0x5632c05b82e0 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecb8240740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fecb01ab4b0 con 0x5632c05b79a0 2024-09-15T23:34:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecb8240740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fecb01a86f0 con 0x5632c05bd370 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecb8240740 10 client.0 _ll_register_callbacks cb 0x7feca400bd00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecaeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecaeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59504/0 (socket says 172.21.15.89:59504) 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecaeffd640 1 -- 172.21.15.89:0/1225865192 learned_addr learned my addr 172.21.15.89:0/1225865192 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecae7fc640 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 0x5632c05c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecae7fc640 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05bd370 0x5632c05c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecaeffd640 1 -- 172.21.15.89:0/1225865192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05bd370 msgr2=0x5632c05c7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecaeffd640 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05bd370 0x5632c05c7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecaeffd640 1 -- 172.21.15.89:0/1225865192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 msgr2=0x5632c05c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecaeffd640 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 0x5632c05c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.527+0000 7fecaeffd640 1 -- 172.21.15.89:0/1225865192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632c05c81c0 con 0x5632c05b82e0 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fecaeffd640 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c6c60 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7feca000ac00 tx=0x7feca000fd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660d2268b8691ca1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fec92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fec92ff5640 1 -- 172.21.15.89:0/1225865192 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca001d420 con 0x5632c05b82e0 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fec92ff5640 1 -- 172.21.15.89:0/1225865192 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7feca001d5c0 con 0x5632c05b82e0 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fecb8240740 1 -- 172.21.15.89:0/1225865192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feca001dca0 con 0x5632c05b82e0 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fec92ff5640 1 -- 172.21.15.89:0/1225865192 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feca0039050 con 0x5632c05b82e0 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fecb8240740 10 client.19089 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fecb8240740 1 -- 172.21.15.89:0/1225865192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5632c05c94e0 con 0x5632c05b82e0 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fec92ff5640 1 -- 172.21.15.89:0/1225865192 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feca001dca0 con 0x5632c05b82e0 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fec92ff5640 1 -- 172.21.15.89:0/1225865192 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7feca002e020 con 0x5632c05b82e0 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fecb8240740 10 client.19089 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fecb8240740 10 client.19089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fecb8240740 20 client.19089 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fecb8240740 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632c0638960 0x5632c063ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fecb8240740 4 client.19089 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.528+0000 7fecb8240740 1 -- 172.21.15.89:0/1225865192 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632c0522d70 con 0x5632c0638960 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.529+0000 7fecae7fc640 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632c0638960 0x5632c063ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.529+0000 7fec92ff5640 10 client.19089 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.529+0000 7fecae7fc640 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632c0638960 0x5632c063ad80 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f4460b9874592ae server_cookie=8bfe13447b4ffa71 in_seq=0 out_seq=0 2024-09-15T23:34:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fec92ff5640 1 -- 172.21.15.89:0/1225865192 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632c0522d70 con 0x5632c0638960 2024-09-15T23:34:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fec92ff5640 10 client.19089 handle_command_reply: tid=0 2024-09-15T23:34:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 client.19089 shutdown 2024-09-15T23:34:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 -- 172.21.15.89:0/1225865192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 msgr2=0x5632c05c6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c6c60 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7feca000ac00 tx=0x7feca000fd90 comp rx=0 tx=0).stop 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 -- 172.21.15.89:0/1225865192 shutdown_connections 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632c0638960 0x5632c063ad80 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632c05bd370 0x5632c05c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fec92ff5640 0 client.19089 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632c05b82e0 0x5632c05c6c60 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 --2- 172.21.15.89:0/1225865192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632c05b79a0 0x5632c05c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 -- 172.21.15.89:0/1225865192 >> 172.21.15.89:0/1225865192 conn(0x5632c05b3010 msgr2=0x5632c05b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 -- 172.21.15.89:0/1225865192 shutdown_connections 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 1 -- 172.21.15.89:0/1225865192 wait complete. 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.543+0000 7fecb8240740 20 client.19089 trim_cache size 0 max 16384 2024-09-15T23:34:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:34:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:34:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:34:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 -- 172.21.15.89:0/3203974720 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fec980a32f0 msgr2=0x7fec980a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fec980a32f0 0x7fec980a57b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7feca80c5b40 tx=0x7feca80dd040 comp rx=0 tx=0).stop 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 -- 172.21.15.89:0/3203974720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecb01020e0 msgr2=0x7fecb019f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecb01020e0 0x7fecb019f080 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7feca8009dc0 tx=0x7feca8003c20 comp rx=0 tx=0).stop 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 -- 172.21.15.89:0/3203974720 shutdown_connections 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fec980a32f0 0x7fec980a57b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecb01020e0 0x7fecb019f080 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecb0101660 0x7fecb019ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 --2- 172.21.15.89:0/3203974720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecb0100ca0 0x7fecb0100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 -- 172.21.15.89:0/3203974720 >> 172.21.15.89:0/3203974720 conn(0x7fecb00fb980 msgr2=0x7fecb0104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 -- 172.21.15.89:0/3203974720 shutdown_connections 2024-09-15T23:34:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:45.544+0000 7fecb54af640 1 -- 172.21.15.89:0/3203974720 wait complete. 2024-09-15T23:34:45.696 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:46.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.141+0000 7f1b293f5640 1 -- 172.21.15.89:0/3012948772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b24102790 msgr2=0x7f1b241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:46.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.141+0000 7f1b293f5640 1 --2- 172.21.15.89:0/3012948772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b24102790 0x7f1b241067e0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c009900 tx=0x7f1b0c02a410 comp rx=0 tx=0).stop 2024-09-15T23:34:46.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.141+0000 7f1b293f5640 1 -- 172.21.15.89:0/3012948772 shutdown_connections 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.141+0000 7f1b293f5640 1 --2- 172.21.15.89:0/3012948772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b24106f10 0x7f1b2410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.141+0000 7f1b293f5640 1 --2- 172.21.15.89:0/3012948772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b24102790 0x7f1b241067e0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.141+0000 7f1b293f5640 1 --2- 172.21.15.89:0/3012948772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b24101de0 0x7f1b241021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.141+0000 7f1b293f5640 1 -- 172.21.15.89:0/3012948772 >> 172.21.15.89:0/3012948772 conn(0x7f1b240fd8c0 msgr2=0x7f1b240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.141+0000 7f1b293f5640 1 -- 172.21.15.89:0/3012948772 shutdown_connections 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.141+0000 7f1b293f5640 1 -- 172.21.15.89:0/3012948772 wait complete. 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b293f5640 1 Processor -- start 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b293f5640 1 -- start start 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b293f5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b24101de0 0x7f1b241985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b293f5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b24102790 0x7f1b24198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b293f5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b24106f10 0x7f1b2419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b293f5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24112010 con 0x7f1b24106f10 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b293f5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24111e90 con 0x7f1b24102790 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b293f5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24112190 con 0x7f1b24101de0 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b237fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b24106f10 0x7f1b2419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b22ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b24101de0 0x7f1b241985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b237fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b24106f10 0x7f1b2419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59520/0 (socket says 172.21.15.89:59520) 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b237fe640 1 -- 172.21.15.89:0/267417561 learned_addr learned my addr 172.21.15.89:0/267417561 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b227fc640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b24102790 0x7f1b24198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b237fe640 1 -- 172.21.15.89:0/267417561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b24101de0 msgr2=0x7f1b241985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b237fe640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b24101de0 0x7f1b241985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b237fe640 1 -- 172.21.15.89:0/267417561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b24102790 msgr2=0x7f1b24198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b237fe640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b24102790 0x7f1b24198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.142+0000 7f1b237fe640 1 -- 172.21.15.89:0/267417561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2419d560 con 0x7f1b24106f10 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.143+0000 7f1b237fe640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b24106f10 0x7f1b2419ce80 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00b970 tx=0x7f1b1c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:46.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.143+0000 7f1b1bfff640 1 -- 172.21.15.89:0/267417561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c019070 con 0x7f1b24106f10 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.143+0000 7f1b1bfff640 1 -- 172.21.15.89:0/267417561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b1c014440 con 0x7f1b24106f10 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.143+0000 7f1b293f5640 1 -- 172.21.15.89:0/267417561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b2419d850 con 0x7f1b24106f10 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.143+0000 7f1b1bfff640 1 -- 172.21.15.89:0/267417561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c0075d0 con 0x7f1b24106f10 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.143+0000 7f1b293f5640 1 -- 172.21.15.89:0/267417561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b241a5130 con 0x7f1b24106f10 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.145+0000 7f1b1bfff640 1 -- 172.21.15.89:0/267417561 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1b1c01c070 con 0x7f1b24106f10 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b1bfff640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b100a3240 0x7f1b100a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b100a3240 0x7f1b100a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b1bfff640 1 -- 172.21.15.89:0/267417561 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1b1c0cf240 con 0x7f1b24106f10 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b2b7ce740 1 Processor -- start 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b2b7ce740 1 -- start start 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b100a3240 0x7f1b100a5700 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1b14009990 tx=0x7f1b14009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 0x5602da0b7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0b7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0c6780 0x5602da0c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5602da01d2f0 con 0x5602da0c6780 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5602d9fc4cf0 con 0x5602da0b75d0 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.146+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5602da0192b0 con 0x5602da0b8b40 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b237fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0c6780 0x5602da0c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b237fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0c6780 0x5602da0c8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59536/0 (socket says 172.21.15.89:59536) 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b237fe640 1 -- 172.21.15.89:0/3246416803 learned_addr learned my addr 172.21.15.89:0/3246416803 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b227fc640 1 --2- 172.21.15.89:0/3246416803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0b7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/3246416803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 0x5602da0b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b237fe640 1 -- 172.21.15.89:0/3246416803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 msgr2=0x5602da0b7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b237fe640 1 --2- 172.21.15.89:0/3246416803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 0x5602da0b7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b237fe640 1 -- 172.21.15.89:0/3246416803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 msgr2=0x5602da0b7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:46.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b237fe640 1 --2- 172.21.15.89:0/3246416803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0b7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b237fe640 1 -- 172.21.15.89:0/3246416803 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d9fa93a0 con 0x5602da0c6780 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/3246416803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 0x5602da0b7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b237fe640 1 --2- 172.21.15.89:0/3246416803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0c6780 0x5602da0c8b70 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c09b4b0 tx=0x7f1b1c09af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cb7882d97dece0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b197fa640 1 -- 172.21.15.89:0/3246416803 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c0a40e0 con 0x5602da0c6780 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b197fa640 1 -- 172.21.15.89:0/3246416803 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b1c099820 con 0x5602da0c6780 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.147+0000 7f1b197fa640 1 -- 172.21.15.89:0/3246416803 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c0a3eb0 con 0x5602da0c6780 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3246416803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0c6780 msgr2=0x5602da0c8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/3246416803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0c6780 0x5602da0c8b70 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c09b4b0 tx=0x7f1b1c09af70 comp rx=0 tx=0).stop 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3246416803 shutdown_connections 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/3246416803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0c6780 0x5602da0c8b70 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/3246416803 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0b7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/3246416803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 0x5602da0b7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3246416803 >> 172.21.15.89:0/3246416803 conn(0x5602da080310 msgr2=0x5602da0b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3246416803 shutdown_connections 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3246416803 wait complete. 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 -- start start 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.148+0000 7f1b2b7ce740 1 Processor -- start 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0c3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0c6780 0x5602da0c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5602da0c9480 con 0x5602da0b8b40 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5602d9fd8060 con 0x5602da0b75d0 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5602da0c9600 con 0x5602da0c6780 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b2b7ce740 10 client.0 _ll_register_callbacks cb 0x5602d9d2b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b227fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0c3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b227fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0c3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59540/0 (socket says 172.21.15.89:59540) 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b227fc640 1 -- 172.21.15.89:0/211583343 learned_addr learned my addr 172.21.15.89:0/211583343 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b237fe640 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0c6780 0x5602da0c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b227fc640 1 -- 172.21.15.89:0/211583343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0c6780 msgr2=0x5602da0c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b227fc640 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0c6780 0x5602da0c3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b227fc640 1 -- 172.21.15.89:0/211583343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 msgr2=0x5602da0c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b227fc640 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b227fc640 1 -- 172.21.15.89:0/211583343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d9d8e3c0 con 0x5602da0b8b40 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b227fc640 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0c3480 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c009900 tx=0x7f1b0c031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f029552c928c2cd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b0a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/211583343 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c035040 con 0x5602da0b8b40 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/211583343 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1b0c004240 con 0x5602da0b8b40 2024-09-15T23:34:46.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.149+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/211583343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b1c099820 con 0x5602da0b8b40 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/211583343 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1b0c04e050 con 0x5602da0b8b40 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b2b7ce740 10 client.19101 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/211583343 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5602da0d4f00 con 0x5602da0b8b40 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/211583343 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b1c099820 con 0x5602da0b8b40 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/211583343 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1b0c004be0 con 0x5602da0b8b40 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b2b7ce740 10 client.19101 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b2b7ce740 10 client.19101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b2b7ce740 20 client.19101 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602da0d63c0 0x5602da0d87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b2b7ce740 4 client.19101 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/211583343 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602da02a860 con 0x5602da0d63c0 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.150+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602da0d63c0 0x5602da0d87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.151+0000 7f1b0a7fc640 10 client.19101 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.151+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602da0d63c0 0x5602da0d87b0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ca55533a2660522 server_cookie=119228c85f1eb722 in_seq=0 out_seq=0 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.151+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/211583343 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5602da02a860 con 0x5602da0d63c0 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.151+0000 7f1b0a7fc640 10 client.19101 handle_command_reply: tid=0 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.151+0000 7f1b2b7ce740 1 client.19101 shutdown 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/211583343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 msgr2=0x5602da0c3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0c3480 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c009900 tx=0x7f1b0c031920 comp rx=0 tx=0).stop 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/211583343 shutdown_connections 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602da0d63c0 0x5602da0d87b0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0c6780 0x5602da0c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0c3480 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/211583343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/211583343 >> 172.21.15.89:0/211583343 conn(0x5602da080310 msgr2=0x5602da0b9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b0a7fc640 0 client.19101 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:46.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/211583343 shutdown_connections 2024-09-15T23:34:46.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/211583343 wait complete. 2024-09-15T23:34:46.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.152+0000 7f1b2b7ce740 20 client.19101 trim_cache size 0 max 16384 2024-09-15T23:34:46.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b2b7ce740 1 Processor -- start 2024-09-15T23:34:46.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b2b7ce740 1 -- start start 2024-09-15T23:34:46.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0b8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0be720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0c6780 0x5602da0bec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241ace10 con 0x5602da0b8b40 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241ae950 con 0x5602da0b75d0 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241ad4e0 con 0x5602da0c6780 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b227fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0be720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b227fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0be720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59544/0 (socket says 172.21.15.89:59544) 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.165+0000 7f1b227fc640 1 -- 172.21.15.89:0/2701898074 learned_addr learned my addr 172.21.15.89:0/2701898074 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b237fe640 1 --2- 172.21.15.89:0/2701898074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0c6780 0x5602da0bec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/2701898074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b227fc640 1 -- 172.21.15.89:0/2701898074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0c6780 msgr2=0x5602da0bec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b227fc640 1 --2- 172.21.15.89:0/2701898074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0c6780 0x5602da0bec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b227fc640 1 -- 172.21.15.89:0/2701898074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 msgr2=0x5602da0b8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b227fc640 1 --2- 172.21.15.89:0/2701898074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0b8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b227fc640 1 -- 172.21.15.89:0/2701898074 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602da0c3f80 con 0x5602da0b8b40 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b227fc640 1 --2- 172.21.15.89:0/2701898074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0be720 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c005960 tx=0x7f1b0c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e990a1d686bcf21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b18ff9640 1 -- 172.21.15.89:0/2701898074 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c057070 con 0x5602da0b8b40 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b18ff9640 1 -- 172.21.15.89:0/2701898074 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b0c0038e0 con 0x5602da0b8b40 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.166+0000 7f1b18ff9640 1 -- 172.21.15.89:0/2701898074 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c035030 con 0x5602da0b8b40 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/2701898074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 msgr2=0x5602da0be720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/2701898074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0be720 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c005960 tx=0x7f1b0c02a410 comp rx=0 tx=0).stop 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/2701898074 shutdown_connections 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/2701898074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0c6780 0x5602da0bec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/2701898074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b8b40 0x5602da0be720 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/2701898074 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0b75d0 0x5602da0b8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/2701898074 >> 172.21.15.89:0/2701898074 conn(0x5602da080310 msgr2=0x5602da0aea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/2701898074 shutdown_connections 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/2701898074 wait complete. 2024-09-15T23:34:46.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 -- start start 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.167+0000 7f1b2b7ce740 1 Processor -- start 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b75d0 0x5602da0ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 0x5602da0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b2b7ce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0c6780 0x5602da0bfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241ae280 con 0x5602da0b75d0 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241adbb0 con 0x5602da0c6780 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b2b7ce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241aadf0 con 0x5602da0b8b40 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b2b7ce740 10 client.0 _ll_register_callbacks cb 0x7f1b0c005d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b22ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b75d0 0x5602da0ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b22ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b75d0 0x5602da0ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59554/0 (socket says 172.21.15.89:59554) 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b22ffd640 1 -- 172.21.15.89:0/3550272748 learned_addr learned my addr 172.21.15.89:0/3550272748 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b227fc640 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 0x5602da0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b22ffd640 1 -- 172.21.15.89:0/3550272748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 msgr2=0x5602da0cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:46.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 0x5602da0cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b22ffd640 1 -- 172.21.15.89:0/3550272748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0c6780 msgr2=0x5602da0bfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b237fe640 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0c6780 0x5602da0bfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0c6780 0x5602da0bfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b22ffd640 1 -- 172.21.15.89:0/3550272748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602da0c0670 con 0x5602da0b75d0 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b22ffd640 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b75d0 0x5602da0ceb90 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f1b14009fd0 tx=0x7f1b140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c4cd302be304905 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b0a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.168+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/3550272748 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1402c070 con 0x5602da0b75d0 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/3550272748 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1b1400f040 con 0x5602da0b75d0 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3550272748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b0c0038e0 con 0x5602da0b75d0 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/3550272748 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1b14038050 con 0x5602da0b75d0 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b2b7ce740 10 client.19107 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3550272748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5602da140c70 con 0x5602da0b75d0 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/3550272748 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b0c0038e0 con 0x5602da0b75d0 2024-09-15T23:34:46.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/3550272748 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1b14013500 con 0x5602da0b75d0 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b2b7ce740 10 client.19107 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b2b7ce740 10 client.19107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b2b7ce740 20 client.19107 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602da1413a0 0x5602da143800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b2b7ce740 4 client.19107 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3550272748 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5602da024d70 con 0x5602da1413a0 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.169+0000 7f1b227fc640 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602da1413a0 0x5602da143800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b0a7fc640 10 client.19107 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b227fc640 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602da1413a0 0x5602da143800 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92a76ac190084268 server_cookie=6d4173a5131a9502 in_seq=0 out_seq=0 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b0a7fc640 1 -- 172.21.15.89:0/3550272748 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5602da024d70 con 0x5602da1413a0 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b0a7fc640 10 client.19107 handle_command_reply: tid=0 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 client.19107 shutdown 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3550272748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b75d0 msgr2=0x5602da0ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b75d0 0x5602da0ceb90 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f1b14009fd0 tx=0x7f1b140046c0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3550272748 shutdown_connections 2024-09-15T23:34:46.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5602da1413a0 0x5602da143800 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602da0c6780 0x5602da0bfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5602da0b8b40 0x5602da0cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 --2- 172.21.15.89:0/3550272748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602da0b75d0 0x5602da0ceb90 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b0a7fc640 0 client.19107 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:46.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3550272748 >> 172.21.15.89:0/3550272748 conn(0x5602da080310 msgr2=0x5602da0be720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:46.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3550272748 shutdown_connections 2024-09-15T23:34:46.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 1 -- 172.21.15.89:0/3550272748 wait complete. 2024-09-15T23:34:46.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.170+0000 7f1b2b7ce740 20 client.19107 trim_cache size 0 max 16384 2024-09-15T23:34:46.182 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:34:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 -- 172.21.15.89:0/267417561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b100a3240 msgr2=0x7f1b100a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b100a3240 0x7f1b100a5700 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1b14009990 tx=0x7f1b14009110 comp rx=0 tx=0).stop 2024-09-15T23:34:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 -- 172.21.15.89:0/267417561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b24106f10 msgr2=0x7f1b2419ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b24106f10 0x7f1b2419ce80 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00b970 tx=0x7f1b1c00bd70 comp rx=0 tx=0).stop 2024-09-15T23:34:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 -- 172.21.15.89:0/267417561 shutdown_connections 2024-09-15T23:34:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b100a3240 0x7f1b100a5700 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b24106f10 0x7f1b2419ce80 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b24102790 0x7f1b24198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 --2- 172.21.15.89:0/267417561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b24101de0 0x7f1b241985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:46.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 -- 172.21.15.89:0/267417561 >> 172.21.15.89:0/267417561 conn(0x7f1b240fd8c0 msgr2=0x7f1b240fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:46.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 -- 172.21.15.89:0/267417561 shutdown_connections 2024-09-15T23:34:46.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:46.171+0000 7f1b293f5640 1 -- 172.21.15.89:0/267417561 wait complete. 2024-09-15T23:34:47.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:47 smithi089 ceph-mon[30933]: pgmap v672: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:47 smithi120 ceph-mon[35250]: pgmap v672: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:47.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:47 smithi174 ceph-mon[35586]: pgmap v672: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:47.983 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:34:47.983 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:34:48.243 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:48.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.746+0000 7fd75055f640 1 -- 172.21.15.89:0/2173448974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd748100e20 msgr2=0x7fd748101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:48.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.746+0000 7fd75055f640 1 --2- 172.21.15.89:0/2173448974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd748100e20 0x7fd748101200 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fd744009900 tx=0x7fd74402a480 comp rx=0 tx=0).stop 2024-09-15T23:34:48.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.746+0000 7fd75055f640 1 -- 172.21.15.89:0/2173448974 shutdown_connections 2024-09-15T23:34:48.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.746+0000 7fd75055f640 1 --2- 172.21.15.89:0/2173448974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd748102ef0 0x7fd74810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.746+0000 7fd75055f640 1 --2- 172.21.15.89:0/2173448974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd748102570 0x7fd7481029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.746+0000 7fd75055f640 1 --2- 172.21.15.89:0/2173448974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd748100e20 0x7fd748101200 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.746+0000 7fd75055f640 1 -- 172.21.15.89:0/2173448974 >> 172.21.15.89:0/2173448974 conn(0x7fd7480fc7d0 msgr2=0x7fd7480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.746+0000 7fd75055f640 1 -- 172.21.15.89:0/2173448974 shutdown_connections 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.746+0000 7fd75055f640 1 -- 172.21.15.89:0/2173448974 wait complete. 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd75055f640 1 Processor -- start 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd75055f640 1 -- start start 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd75055f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd748100e20 0x7fd748199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd75055f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd748102570 0x7fd748199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd75055f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd748102ef0 0x7fd7481a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd75055f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd748111020 con 0x7fd748102570 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd75055f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd748110ea0 con 0x7fd748102ef0 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd75055f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd7481111a0 con 0x7fd748100e20 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74dafa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd748102570 0x7fd748199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74dafa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd748102570 0x7fd748199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59570/0 (socket says 172.21.15.89:59570) 2024-09-15T23:34:48.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74dafa640 1 -- 172.21.15.89:0/2162340863 learned_addr learned my addr 172.21.15.89:0/2162340863 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74e2fb640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd748100e20 0x7fd748199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74eafc640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd748102ef0 0x7fd7481a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74dafa640 1 -- 172.21.15.89:0/2162340863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd748100e20 msgr2=0x7fd748199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74dafa640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd748100e20 0x7fd748199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74dafa640 1 -- 172.21.15.89:0/2162340863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd748102ef0 msgr2=0x7fd7481a44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74dafa640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd748102ef0 0x7fd7481a44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74dafa640 1 -- 172.21.15.89:0/2162340863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7481a49f0 con 0x7fd748102570 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74eafc640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd748102ef0 0x7fd7481a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.747+0000 7fd74dafa640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd748102570 0x7fd748199970 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fd73800b4a0 tx=0x7fd73800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.748+0000 7fd73f7fe640 1 -- 172.21.15.89:0/2162340863 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738014070 con 0x7fd748102570 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.748+0000 7fd75055f640 1 -- 172.21.15.89:0/2162340863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7481a4bc0 con 0x7fd748102570 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.748+0000 7fd73f7fe640 1 -- 172.21.15.89:0/2162340863 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd738010af0 con 0x7fd748102570 2024-09-15T23:34:48.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.748+0000 7fd73f7fe640 1 -- 172.21.15.89:0/2162340863 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738017d10 con 0x7fd748102570 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.748+0000 7fd75055f640 1 -- 172.21.15.89:0/2162340863 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7481a5100 con 0x7fd748102570 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.750+0000 7fd73f7fe640 1 -- 172.21.15.89:0/2162340863 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd738017070 con 0x7fd748102570 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.750+0000 7fd752938740 1 Processor -- start 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.750+0000 7fd752938740 1 -- start start 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd752938740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 0x560e499bbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd752938740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bc480 0x560e499c6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd752938740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499c6d40 0x560e499c9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd752938740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560e49921280 con 0x560e499c6d40 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd752938740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560e498c8b10 con 0x560e499bc480 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd752938740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560e4991d240 con 0x560e499bbb60 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd73f7fe640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd7340a3240 0x7fd7340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74dafa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499c6d40 0x560e499c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74dafa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499c6d40 0x560e499c9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59586/0 (socket says 172.21.15.89:59586) 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74dafa640 1 -- 172.21.15.89:0/2079298558 learned_addr learned my addr 172.21.15.89:0/2079298558 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74eafc640 1 --2- 172.21.15.89:0/2079298558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 0x560e499bbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74e2fb640 1 --2- 172.21.15.89:0/2079298558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bc480 0x560e499c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74eafc640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd7340a3240 0x7fd7340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd73f7fe640 1 -- 172.21.15.89:0/2162340863 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd7380cf180 con 0x7fd748102570 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74dafa640 1 -- 172.21.15.89:0/2079298558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 msgr2=0x560e499bbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74dafa640 1 --2- 172.21.15.89:0/2079298558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 0x560e499bbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74dafa640 1 -- 172.21.15.89:0/2079298558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bc480 msgr2=0x560e499c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74dafa640 1 --2- 172.21.15.89:0/2079298558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bc480 0x560e499c6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74dafa640 1 -- 172.21.15.89:0/2079298558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e498d51f0 con 0x560e499c6d40 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74eafc640 1 --2- 172.21.15.89:0/2079298558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 0x560e499bbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:34:48.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.751+0000 7fd74dafa640 1 --2- 172.21.15.89:0/2079298558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499c6d40 0x560e499c9130 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fd738097d80 tx=0x7fd7380047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c116be5074d56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd74eafc640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd7340a3240 0x7fd7340a5700 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd740009fd0 tx=0x7fd740009a10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd73cff9640 1 -- 172.21.15.89:0/2079298558 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73809b3b0 con 0x560e499c6d40 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd73cff9640 1 -- 172.21.15.89:0/2079298558 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd73809f760 con 0x560e499c6d40 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd73cff9640 1 -- 172.21.15.89:0/2079298558 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738010ca0 con 0x560e499c6d40 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd752938740 1 -- 172.21.15.89:0/2079298558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499c6d40 msgr2=0x560e499c9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd752938740 1 --2- 172.21.15.89:0/2079298558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499c6d40 0x560e499c9130 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fd738097d80 tx=0x7fd7380047d0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd752938740 1 -- 172.21.15.89:0/2079298558 shutdown_connections 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd752938740 1 --2- 172.21.15.89:0/2079298558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499c6d40 0x560e499c9130 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd752938740 1 --2- 172.21.15.89:0/2079298558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bc480 0x560e499c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd752938740 1 --2- 172.21.15.89:0/2079298558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 0x560e499bbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.752+0000 7fd752938740 1 -- 172.21.15.89:0/2079298558 >> 172.21.15.89:0/2079298558 conn(0x560e499b80f0 msgr2=0x560e499b84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:48.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 1 -- 172.21.15.89:0/2079298558 shutdown_connections 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 1 -- 172.21.15.89:0/2079298558 wait complete. 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 1 -- start start 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 1 Processor -- start 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 0x560e499c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499c4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499c6d40 0x560e499c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560e499c9910 con 0x560e499bc480 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560e498dbf10 con 0x560e499bbb60 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560e499c9a90 con 0x560e499c6d40 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd752938740 10 client.0 _ll_register_callbacks cb 0x560e498a4a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd74dafa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd74dafa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499c4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59588/0 (socket says 172.21.15.89:59588) 2024-09-15T23:34:48.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd74dafa640 1 -- 172.21.15.89:0/2317631731 learned_addr learned my addr 172.21.15.89:0/2317631731 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.753+0000 7fd74e2fb640 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 0x560e499c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd74eafc640 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499c6d40 0x560e499c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd74dafa640 1 -- 172.21.15.89:0/2317631731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499c6d40 msgr2=0x560e499c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd74dafa640 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499c6d40 0x560e499c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd74dafa640 1 -- 172.21.15.89:0/2317631731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 msgr2=0x560e499c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd74dafa640 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 0x560e499c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd74dafa640 1 -- 172.21.15.89:0/2317631731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e496926a0 con 0x560e499bc480 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd74dafa640 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499c4170 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fd7380176e0 tx=0x7fd7380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b551b9aabba4f39c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd731ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd731ffb640 1 -- 172.21.15.89:0/2317631731 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73801e050 con 0x560e499bc480 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd731ffb640 1 -- 172.21.15.89:0/2317631731 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd738010ca0 con 0x560e499bc480 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd752938740 1 -- 172.21.15.89:0/2317631731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd73809f760 con 0x560e499bc480 2024-09-15T23:34:48.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd731ffb640 1 -- 172.21.15.89:0/2317631731 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd73809b3b0 con 0x560e499bc480 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd752938740 10 client.19119 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd752938740 1 -- 172.21.15.89:0/2317631731 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560e499d5520 con 0x560e499bc480 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd731ffb640 1 -- 172.21.15.89:0/2317631731 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd73809f760 con 0x560e499bc480 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd731ffb640 1 -- 172.21.15.89:0/2317631731 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd7380b41f0 con 0x560e499bc480 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd752938740 10 client.19119 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd752938740 10 client.19119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.754+0000 7fd752938740 20 client.19119 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.755+0000 7fd752938740 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560e499d6870 0x560e499d8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.755+0000 7fd752938740 4 client.19119 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.755+0000 7fd752938740 1 -- 172.21.15.89:0/2317631731 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e4992e7f0 con 0x560e499d6870 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.755+0000 7fd74e2fb640 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560e499d6870 0x560e499d8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.755+0000 7fd74e2fb640 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560e499d6870 0x560e499d8c60 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a057162ae986d971 server_cookie=a579f0d13b7d4657 in_seq=0 out_seq=0 2024-09-15T23:34:48.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.755+0000 7fd731ffb640 10 client.19119 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd731ffb640 1 -- 172.21.15.89:0/2317631731 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560e4992e7f0 con 0x560e499d6870 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd731ffb640 10 client.19119 handle_command_reply: tid=0 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 client.19119 shutdown 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 -- 172.21.15.89:0/2317631731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 msgr2=0x560e499c4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499c4170 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fd7380176e0 tx=0x7fd7380046c0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 -- 172.21.15.89:0/2317631731 shutdown_connections 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560e499d6870 0x560e499d8c60 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499c6d40 0x560e499c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499c4170 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd731ffb640 0 client.19119 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:48.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 --2- 172.21.15.89:0/2317631731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 0x560e499c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 -- 172.21.15.89:0/2317631731 >> 172.21.15.89:0/2317631731 conn(0x560e499b80f0 msgr2=0x560e499bddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 -- 172.21.15.89:0/2317631731 shutdown_connections 2024-09-15T23:34:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 1 -- 172.21.15.89:0/2317631731 wait complete. 2024-09-15T23:34:48.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.756+0000 7fd752938740 20 client.19119 trim_cache size 0 max 16384 2024-09-15T23:34:48.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.773+0000 7fd752938740 1 Processor -- start 2024-09-15T23:34:48.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.773+0000 7fd752938740 1 -- start start 2024-09-15T23:34:48.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.773+0000 7fd752938740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 0x560e499dbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.773+0000 7fd752938740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499dc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.773+0000 7fd752938740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499c6d40 0x560e499c0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.773+0000 7fd752938740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd7481acd50 con 0x560e499bc480 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.773+0000 7fd752938740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7481ae890 con 0x560e499bbb60 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.773+0000 7fd752938740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd7481ad420 con 0x560e499c6d40 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.773+0000 7fd74dafa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499dc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.773+0000 7fd74dafa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499dc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59592/0 (socket says 172.21.15.89:59592) 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd74dafa640 1 -- 172.21.15.89:0/640069636 learned_addr learned my addr 172.21.15.89:0/640069636 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd74e2fb640 1 --2- 172.21.15.89:0/640069636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 0x560e499dbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd74dafa640 1 -- 172.21.15.89:0/640069636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499c6d40 msgr2=0x560e499c0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd74dafa640 1 --2- 172.21.15.89:0/640069636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499c6d40 0x560e499c0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd74dafa640 1 -- 172.21.15.89:0/640069636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 msgr2=0x560e499dbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:48.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd74dafa640 1 --2- 172.21.15.89:0/640069636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 0x560e499dbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd74dafa640 1 -- 172.21.15.89:0/640069636 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e499b8d80 con 0x560e499bc480 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd74e2fb640 1 --2- 172.21.15.89:0/640069636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 0x560e499dbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd74dafa640 1 --2- 172.21.15.89:0/640069636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499dc1c0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fd738097810 tx=0x7fd7380998c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653fbdbf6287464f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd733fff640 1 -- 172.21.15.89:0/640069636 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73801e070 con 0x560e499bc480 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd733fff640 1 -- 172.21.15.89:0/640069636 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd73809f760 con 0x560e499bc480 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.774+0000 7fd733fff640 1 -- 172.21.15.89:0/640069636 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738097270 con 0x560e499bc480 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 -- 172.21.15.89:0/640069636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 msgr2=0x560e499dc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 --2- 172.21.15.89:0/640069636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499dc1c0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fd738097810 tx=0x7fd7380998c0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 -- 172.21.15.89:0/640069636 shutdown_connections 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 --2- 172.21.15.89:0/640069636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499c6d40 0x560e499c0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 --2- 172.21.15.89:0/640069636 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499dc1c0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 --2- 172.21.15.89:0/640069636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499bbb60 0x560e499dbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 -- 172.21.15.89:0/640069636 >> 172.21.15.89:0/640069636 conn(0x560e499b80f0 msgr2=0x560e499d2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 -- 172.21.15.89:0/640069636 shutdown_connections 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 -- 172.21.15.89:0/640069636 wait complete. 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 -- start start 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 Processor -- start 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 0x560e499da300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499da840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499c6d40 0x560e49a3a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd7481ae1c0 con 0x560e499bc480 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7481adaf0 con 0x560e499c6d40 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd7481aad30 con 0x560e499bbb60 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.775+0000 7fd752938740 10 client.0 _ll_register_callbacks cb 0x7fd744009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74dafa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499da840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74dafa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499da840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59602/0 (socket says 172.21.15.89:59602) 2024-09-15T23:34:48.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74dafa640 1 -- 172.21.15.89:0/1711885173 learned_addr learned my addr 172.21.15.89:0/1711885173 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74eafc640 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499c6d40 0x560e49a3a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74e2fb640 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 0x560e499da300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74dafa640 1 -- 172.21.15.89:0/1711885173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 msgr2=0x560e499da300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74dafa640 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 0x560e499da300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74dafa640 1 -- 172.21.15.89:0/1711885173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499c6d40 msgr2=0x560e49a3a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74dafa640 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499c6d40 0x560e49a3a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74dafa640 1 -- 172.21.15.89:0/1711885173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e49a3a830 con 0x560e499bc480 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd74dafa640 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499da840 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fd73809b340 tx=0x7fd7380976c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41763b346cb166e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd731ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd731ffb640 1 -- 172.21.15.89:0/1711885173 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7380b9060 con 0x560e499bc480 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd731ffb640 1 -- 172.21.15.89:0/1711885173 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd738097270 con 0x560e499bc480 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd752938740 1 -- 172.21.15.89:0/1711885173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd73809f760 con 0x560e499bc480 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd731ffb640 1 -- 172.21.15.89:0/1711885173 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd738064070 con 0x560e499bc480 2024-09-15T23:34:48.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd752938740 10 client.19125 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.776+0000 7fd752938740 1 -- 172.21.15.89:0/1711885173 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560e49a3a830 con 0x560e499bc480 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd731ffb640 1 -- 172.21.15.89:0/1711885173 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd73809f760 con 0x560e499bc480 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd731ffb640 1 -- 172.21.15.89:0/1711885173 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd73809a370 con 0x560e499bc480 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd752938740 10 client.19125 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd752938740 10 client.19125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd752938740 20 client.19125 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd752938740 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560e499d6870 0x560e49a43ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd752938740 4 client.19125 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd752938740 1 -- 172.21.15.89:0/1711885173 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560e49928d00 con 0x560e499d6870 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd74e2fb640 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560e499d6870 0x560e49a43ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd731ffb640 10 client.19125 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:48.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.777+0000 7fd74e2fb640 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560e499d6870 0x560e49a43ec0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b678c1768a0f03e server_cookie=b77828f305b2f8a0 in_seq=0 out_seq=0 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.778+0000 7fd731ffb640 1 -- 172.21.15.89:0/1711885173 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560e49928d00 con 0x560e499d6870 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.778+0000 7fd731ffb640 10 client.19125 handle_command_reply: tid=0 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.778+0000 7fd752938740 1 client.19125 shutdown 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.778+0000 7fd752938740 1 -- 172.21.15.89:0/1711885173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 msgr2=0x560e499da840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.778+0000 7fd752938740 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499da840 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fd73809b340 tx=0x7fd7380976c0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd752938740 1 -- 172.21.15.89:0/1711885173 shutdown_connections 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd752938740 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560e499d6870 0x560e49a43ec0 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd752938740 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e499c6d40 0x560e49a3a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd752938740 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e499bc480 0x560e499da840 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd752938740 1 --2- 172.21.15.89:0/1711885173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e499bbb60 0x560e499da300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd752938740 1 -- 172.21.15.89:0/1711885173 >> 172.21.15.89:0/1711885173 conn(0x560e499b80f0 msgr2=0x560e499c3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd731ffb640 0 client.19125 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd752938740 1 -- 172.21.15.89:0/1711885173 shutdown_connections 2024-09-15T23:34:48.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd752938740 1 -- 172.21.15.89:0/1711885173 wait complete. 2024-09-15T23:34:48.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd752938740 20 client.19125 trim_cache size 0 max 16384 2024-09-15T23:34:48.794 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:34:48.794 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:34:48.794 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "4ca9439b-b222-4317-be65-1d7d36d9c8f9", 2024-09-15T23:34:48.794 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:34:48.794 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd75055f640 1 -- 172.21.15.89:0/2162340863 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd7340a3240 msgr2=0x7fd7340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd75055f640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd7340a3240 0x7fd7340a5700 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd740009fd0 tx=0x7fd740009a10 comp rx=0 tx=0).stop 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd75055f640 1 -- 172.21.15.89:0/2162340863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd748102570 msgr2=0x7fd748199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd75055f640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd748102570 0x7fd748199970 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fd73800b4a0 tx=0x7fd73800b960 comp rx=0 tx=0).stop 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd75055f640 1 -- 172.21.15.89:0/2162340863 shutdown_connections 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd75055f640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd7340a3240 0x7fd7340a5700 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd75055f640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd748102ef0 0x7fd7481a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd75055f640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd748102570 0x7fd748199970 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd75055f640 1 --2- 172.21.15.89:0/2162340863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd748100e20 0x7fd748199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.779+0000 7fd75055f640 1 -- 172.21.15.89:0/2162340863 >> 172.21.15.89:0/2162340863 conn(0x7fd7480fc7d0 msgr2=0x7fd7480febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.780+0000 7fd75055f640 1 -- 172.21.15.89:0/2162340863 shutdown_connections 2024-09-15T23:34:48.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:48.780+0000 7fd75055f640 1 -- 172.21.15.89:0/2162340863 wait complete. 2024-09-15T23:34:49.413 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:49 smithi089 ceph-mon[30933]: pgmap v673: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:34:49.413 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:49 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:34:49.413 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:49 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:49.413 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:49 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:34:49.413 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:49 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:49 smithi120 ceph-mon[35250]: pgmap v673: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:34:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:49 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:34:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:49 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:49 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:34:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:49 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:49 smithi174 ceph-mon[35586]: pgmap v673: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:34:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:49 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:34:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:49 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:49 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:34:49.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:49 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:34:49.610 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:34:49.789 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:313/600: open fd = 3 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:314/600: open fd = 3 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:315/600: open fd = 3 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.790 INFO:tasks.workunit.client.0.smithi089.stdout:316/600: open fd = 3 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:317/600: open fd = 3 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:318/600: open fd = 3 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:319/600: open fd = 3 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.791 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:320/600: open fd = 3 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:321/600: open fd = 3 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:322/600: open fd = 3 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:323/600: open fd = 3 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.792 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:324/600: open fd = 3 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:325/600: open fd = 3 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:326/600: open fd = 3 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.793 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:327/600: open fd = 3 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:328/600: open fd = 3 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:329/600: open fd = 3 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:330/600: open fd = 3 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.794 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:331/600: open fd = 3 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:332/600: open fd = 3 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:333/600: open fd = 3 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.795 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.796 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:334/600: open fd = 3 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:335/600: open fd = 3 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:336/600: open fd = 3 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:337/600: open fd = 3 2024-09-15T23:34:49.797 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:338/600: open fd = 3 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:339/600: open fd = 3 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:340/600: open fd = 3 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.798 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:341/600: open fd = 3 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:342/600: open fd = 3 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:343/600: open fd = 3 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:344/600: open fd = 3 2024-09-15T23:34:49.799 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:345/600: open fd = 3 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:346/600: open fd = 3 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:347/600: open fd = 3 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.800 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:348/600: open fd = 3 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:349/600: open fd = 3 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:350/600: open fd = 3 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:351/600: open fd = 3 2024-09-15T23:34:49.801 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:352/600: open fd = 3 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:353/600: open fd = 3 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:354/600: open fd = 3 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.802 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:355/600: open fd = 3 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:356/600: open fd = 3 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:357/600: open fd = 3 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:358/600: open fd = 3 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.803 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:359/600: open fd = 3 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:360/600: open fd = 3 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:361/600: open fd = 3 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:362/600: open fd = 3 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.804 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:363/600: open fd = 3 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:364/600: open fd = 3 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:365/600: open fd = 3 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:366/600: open fd = 3 2024-09-15T23:34:49.805 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:367/600: open fd = 3 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:368/600: open fd = 3 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:369/600: open fd = 3 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.806 INFO:tasks.workunit.client.0.smithi089.stdout:370/600: open fd = 3 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:371/600: open fd = 3 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:372/600: open fd = 3 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:373/600: open fd = 3 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:374/600: open fd = 3 2024-09-15T23:34:49.807 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.808 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:34:49.808 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:34:49.853 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:34:50.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.298+0000 7f3ef89cb640 1 -- 172.21.15.89:0/1659925871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ef4108110 msgr2=0x7f3ef410ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:50.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.298+0000 7f3ef89cb640 1 --2- 172.21.15.89:0/1659925871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ef4108110 0x7f3ef410ead0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009ca0 tx=0x7f3eec02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.298+0000 7f3ef89cb640 1 -- 172.21.15.89:0/1659925871 shutdown_connections 2024-09-15T23:34:50.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.298+0000 7f3ef89cb640 1 --2- 172.21.15.89:0/1659925871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ef4108110 0x7f3ef410ead0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.298+0000 7f3ef89cb640 1 --2- 172.21.15.89:0/1659925871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef4101290 0x7f3ef41079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.298+0000 7f3ef89cb640 1 --2- 172.21.15.89:0/1659925871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ef41008e0 0x7f3ef4100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.298+0000 7f3ef89cb640 1 -- 172.21.15.89:0/1659925871 >> 172.21.15.89:0/1659925871 conn(0x7f3ef4077210 msgr2=0x7f3ef4077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.298+0000 7f3ef89cb640 1 -- 172.21.15.89:0/1659925871 shutdown_connections 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.298+0000 7f3ef89cb640 1 -- 172.21.15.89:0/1659925871 wait complete. 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.299+0000 7f3ef89cb640 1 Processor -- start 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.299+0000 7f3ef89cb640 1 -- start start 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.299+0000 7f3ef89cb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ef41008e0 0x7f3ef4101860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.299+0000 7f3ef89cb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ef4101290 0x7f3ef4101da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.299+0000 7f3ef89cb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef4108110 0x7f3ef41022e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.299+0000 7f3ef89cb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef4111fb0 con 0x7f3ef41008e0 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.299+0000 7f3ef89cb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef4111e30 con 0x7f3ef4108110 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.299+0000 7f3ef89cb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef4112130 con 0x7f3ef4101290 2024-09-15T23:34:50.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ef41008e0 0x7f3ef4101860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ef41008e0 0x7f3ef4101860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59616/0 (socket says 172.21.15.89:59616) 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef259c640 1 -- 172.21.15.89:0/4153473037 learned_addr learned my addr 172.21.15.89:0/4153473037 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef1d9b640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ef4101290 0x7f3ef4101da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef4108110 0x7f3ef41022e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef259c640 1 -- 172.21.15.89:0/4153473037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ef4101290 msgr2=0x7f3ef4101da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef259c640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ef4101290 0x7f3ef4101da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef259c640 1 -- 172.21.15.89:0/4153473037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef4108110 msgr2=0x7f3ef41022e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef259c640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef4108110 0x7f3ef41022e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef259c640 1 -- 172.21.15.89:0/4153473037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef41a6890 con 0x7f3ef41008e0 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef4108110 0x7f3ef41022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef1d9b640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ef4101290 0x7f3ef4101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef259c640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ef41008e0 0x7f3ef4101860 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400d770 tx=0x7f3ee400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:50.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ee37fe640 1 -- 172.21.15.89:0/4153473037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4017070 con 0x7f3ef41008e0 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef89cb640 1 -- 172.21.15.89:0/4153473037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef41a6b80 con 0x7f3ef41008e0 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ee37fe640 1 -- 172.21.15.89:0/4153473037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ee4005500 con 0x7f3ef41008e0 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ee37fe640 1 -- 172.21.15.89:0/4153473037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4004e00 con 0x7f3ef41008e0 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.300+0000 7f3ef89cb640 1 -- 172.21.15.89:0/4153473037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef41a70c0 con 0x7f3ef41008e0 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.302+0000 7f3ee37fe640 1 -- 172.21.15.89:0/4153473037 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3ee4010460 con 0x7f3ef41008e0 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.303+0000 7f3efada4740 1 Processor -- start 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.303+0000 7f3efada4740 1 -- start start 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.303+0000 7f3efada4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2e8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.303+0000 7f3efada4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a2f3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.303+0000 7f3efada4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2f3680 0x56415a2f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.303+0000 7f3efada4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56415a24e2e0 con 0x56415a2e84a0 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.303+0000 7f3efada4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56415a1f5ce0 con 0x56415a2f3680 2024-09-15T23:34:50.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.303+0000 7f3efada4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56415a24a2a0 con 0x56415a2e8dc0 2024-09-15T23:34:50.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.303+0000 7f3ef2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2e8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ee37fe640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3edc0a3240 0x7f3edc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2e8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59618/0 (socket says 172.21.15.89:59618) 2024-09-15T23:34:50.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/1845166320 learned_addr learned my addr 172.21.15.89:0/1845166320 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:50.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef259c640 1 --2- 172.21.15.89:0/1845166320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a2f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3edc0a3240 0x7f3edc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef1d9b640 1 --2- 172.21.15.89:0/1845166320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2f3680 0x56415a2f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ee37fe640 1 -- 172.21.15.89:0/4153473037 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3ee40cf8b0 con 0x7f3ef41008e0 2024-09-15T23:34:50.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/1845166320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 msgr2=0x56415a2f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/1845166320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a2f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/1845166320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2f3680 msgr2=0x56415a2f5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/1845166320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2f3680 0x56415a2f5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/1845166320 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56415a1da440 con 0x56415a2e84a0 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/1845166320 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2e8880 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f3eec003d00 tx=0x7f3eec003d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f90b3bf2bd4fa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3edc0a3240 0x7f3edc0a5700 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009bf0 tx=0x7f3eec047000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.304+0000 7f3ee0ff9640 1 -- 172.21.15.89:0/1845166320 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec04e070 con 0x56415a2e84a0 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3ee0ff9640 1 -- 172.21.15.89:0/1845166320 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eec036070 con 0x56415a2e84a0 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3ee0ff9640 1 -- 172.21.15.89:0/1845166320 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec02fe00 con 0x56415a2e84a0 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3efada4740 1 -- 172.21.15.89:0/1845166320 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 msgr2=0x56415a2e8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3efada4740 1 --2- 172.21.15.89:0/1845166320 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2e8880 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f3eec003d00 tx=0x7f3eec003d30 comp rx=0 tx=0).stop 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3efada4740 1 -- 172.21.15.89:0/1845166320 shutdown_connections 2024-09-15T23:34:50.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3efada4740 1 --2- 172.21.15.89:0/1845166320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2f3680 0x56415a2f5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3efada4740 1 --2- 172.21.15.89:0/1845166320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a2f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3efada4740 1 --2- 172.21.15.89:0/1845166320 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2e8880 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3efada4740 1 -- 172.21.15.89:0/1845166320 >> 172.21.15.89:0/1845166320 conn(0x56415a2e4a30 msgr2=0x56415a2e4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3efada4740 1 -- 172.21.15.89:0/1845166320 shutdown_connections 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.305+0000 7f3efada4740 1 -- 172.21.15.89:0/1845166320 wait complete. 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3efada4740 1 -- start start 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3efada4740 1 Processor -- start 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3efada4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3efada4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e8dc0 0x56415a2f0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3efada4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2f3680 0x56415a2f0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3efada4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56415a2f6250 con 0x56415a2e84a0 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3efada4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56415a209050 con 0x56415a2e8dc0 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3efada4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56415a2f63d0 con 0x56415a2f3680 2024-09-15T23:34:50.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3efada4740 10 client.0 _ll_register_callbacks cb 0x56415a249160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2f2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59622/0 (socket says 172.21.15.89:59622) 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef259c640 1 -- 172.21.15.89:0/463455193 learned_addr learned my addr 172.21.15.89:0/463455193 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2f3680 0x56415a2f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef1d9b640 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e8dc0 0x56415a2f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef259c640 1 -- 172.21.15.89:0/463455193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2f3680 msgr2=0x56415a2f0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef259c640 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2f3680 0x56415a2f0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef259c640 1 -- 172.21.15.89:0/463455193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e8dc0 msgr2=0x56415a2f0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef259c640 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e8dc0 0x56415a2f0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef259c640 1 -- 172.21.15.89:0/463455193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564159fbf3c0 con 0x56415a2e84a0 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ef259c640 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2f2f90 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40ccf60 tx=0x7f3ee401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cce52f6a81a28aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.306+0000 7f3ed9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:50.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/463455193 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4020040 con 0x56415a2e84a0 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/463455193 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3ee40980f0 con 0x56415a2e84a0 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3efada4740 1 -- 172.21.15.89:0/463455193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3eec036070 con 0x56415a2e84a0 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/463455193 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3ee4015110 con 0x56415a2e84a0 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3efada4740 10 client.19137 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3efada4740 1 -- 172.21.15.89:0/463455193 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56415a301c20 con 0x56415a2e84a0 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/463455193 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eec036070 con 0x56415a2e84a0 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/463455193 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3ee40a4260 con 0x56415a2e84a0 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3efada4740 10 client.19137 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3efada4740 10 client.19137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3efada4740 20 client.19137 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3efada4740 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56415a3030e0 0x56415a305530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3efada4740 4 client.19137 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3efada4740 1 -- 172.21.15.89:0/463455193 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56415a25b850 con 0x56415a3030e0 2024-09-15T23:34:50.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.307+0000 7f3ef1d9b640 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56415a3030e0 0x56415a305530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.308+0000 7f3ed9ffb640 10 client.19137 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.308+0000 7f3ef1d9b640 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56415a3030e0 0x56415a305530 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2ba064adb00f733 server_cookie=13902bc36e1206bb in_seq=0 out_seq=0 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.308+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/463455193 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56415a25b850 con 0x56415a3030e0 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.308+0000 7f3ed9ffb640 10 client.19137 handle_command_reply: tid=0 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.308+0000 7f3efada4740 1 client.19137 shutdown 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 1 -- 172.21.15.89:0/463455193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 msgr2=0x56415a2f2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2f2f90 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f3ee40ccf60 tx=0x7f3ee401e000 comp rx=0 tx=0).stop 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 1 -- 172.21.15.89:0/463455193 shutdown_connections 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56415a3030e0 0x56415a305530 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2f3680 0x56415a2f0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e8dc0 0x56415a2f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 1 --2- 172.21.15.89:0/463455193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2e84a0 0x56415a2f2f90 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3ed9ffb640 0 client.19137 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:50.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 1 -- 172.21.15.89:0/463455193 >> 172.21.15.89:0/463455193 conn(0x56415a2e4a30 msgr2=0x56415a2ea710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 1 -- 172.21.15.89:0/463455193 shutdown_connections 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 1 -- 172.21.15.89:0/463455193 wait complete. 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.309+0000 7f3efada4740 20 client.19137 trim_cache size 0 max 16384 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3efada4740 1 Processor -- start 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3efada4740 1 -- start start 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3efada4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 0x56415a2e3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3efada4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a30ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3efada4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a30b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3efada4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef41b0f00 con 0x56415a2f3680 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3efada4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef41b2a40 con 0x56415a2e84a0 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3efada4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef41b15d0 con 0x56415a2e8dc0 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a30b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a30b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59634/0 (socket says 172.21.15.89:59634) 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/3498848753 learned_addr learned my addr 172.21.15.89:0/3498848753 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef1d9b640 1 --2- 172.21.15.89:0/3498848753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a30ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef259c640 1 --2- 172.21.15.89:0/3498848753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 0x56415a2e3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/3498848753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 msgr2=0x56415a30ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/3498848753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a30ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/3498848753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 msgr2=0x56415a2e3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/3498848753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 0x56415a2e3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/3498848753 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56415a2e9a20 con 0x56415a2f3680 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.326+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/3498848753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a30b340 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f3eec02a900 tx=0x7f3eec005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83e93a147a621f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3edbfff640 1 -- 172.21.15.89:0/3498848753 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec057020 con 0x56415a2f3680 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3edbfff640 1 -- 172.21.15.89:0/3498848753 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eec036050 con 0x56415a2f3680 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3edbfff640 1 -- 172.21.15.89:0/3498848753 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec04e070 con 0x56415a2f3680 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3efada4740 1 -- 172.21.15.89:0/3498848753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 msgr2=0x56415a30b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:50.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3efada4740 1 --2- 172.21.15.89:0/3498848753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a30b340 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f3eec02a900 tx=0x7f3eec005c50 comp rx=0 tx=0).stop 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3efada4740 1 -- 172.21.15.89:0/3498848753 shutdown_connections 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3efada4740 1 --2- 172.21.15.89:0/3498848753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a30b340 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3efada4740 1 --2- 172.21.15.89:0/3498848753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a30ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3efada4740 1 --2- 172.21.15.89:0/3498848753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 0x56415a2e3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3efada4740 1 -- 172.21.15.89:0/3498848753 >> 172.21.15.89:0/3498848753 conn(0x56415a2e4a30 msgr2=0x56415a2e3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3efada4740 1 -- 172.21.15.89:0/3498848753 shutdown_connections 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.327+0000 7f3efada4740 1 -- 172.21.15.89:0/3498848753 wait complete. 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3efada4740 1 -- start start 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3efada4740 1 Processor -- start 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3efada4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 0x56415a2fbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3efada4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a2f92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3efada4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a2f99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3efada4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef41b2370 con 0x56415a2f3680 2024-09-15T23:34:50.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3efada4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef41b1ca0 con 0x56415a2e84a0 2024-09-15T23:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3efada4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef41aeee0 con 0x56415a2e8dc0 2024-09-15T23:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3efada4740 10 client.0 _ll_register_callbacks cb 0x7f3ee400d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a2f99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a2f99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:59646/0 (socket says 172.21.15.89:59646) 2024-09-15T23:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/2122090520 learned_addr learned my addr 172.21.15.89:0/2122090520 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef1d9b640 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a2f92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef259c640 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 0x56415a2fbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/2122090520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 msgr2=0x56415a2f92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a2f92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/2122090520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 msgr2=0x56415a2fbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:50.384 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:34:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:34:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:34:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "4ca9439b-b222-4317-be65-1d7d36d9c8f9": { 2024-09-15T23:34:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:34:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "4ca9439b-b222-4317-be65-1d7d36d9c8f9", 2024-09-15T23:34:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:34:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:34:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:34:50.384 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 0x56415a2fbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef2d9d640 1 -- 172.21.15.89:0/2122090520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56415a36cde0 con 0x56415a2f3680 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.328+0000 7f3ef259c640 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 0x56415a2fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3ef2d9d640 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a2f99e0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f3eec02c010 tx=0x7f3eec009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae48bfad9b5aaa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3ed9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/2122090520 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec057030 con 0x56415a2f3680 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/2122090520 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3eec02f610 con 0x56415a2f3680 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3efada4740 1 -- 172.21.15.89:0/2122090520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3eec036050 con 0x56415a2f3680 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/2122090520 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3eec04e040 con 0x56415a2f3680 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3efada4740 10 client.19143 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3efada4740 1 -- 172.21.15.89:0/2122090520 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56415a36df90 con 0x56415a2f3680 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/2122090520 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eec036050 con 0x56415a2f3680 2024-09-15T23:34:50.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/2122090520 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3eec048070 con 0x56415a2f3680 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3efada4740 10 client.19143 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3efada4740 10 client.19143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3efada4740 20 client.19143 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3efada4740 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56415a36e1b0 0x56415a3705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3efada4740 4 client.19143 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.329+0000 7f3efada4740 1 -- 172.21.15.89:0/2122090520 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56415a255d60 con 0x56415a36e1b0 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3ef259c640 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56415a36e1b0 0x56415a3705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3ef259c640 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56415a36e1b0 0x56415a3705a0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=294c4d3b7b94735c server_cookie=ca80fc7a0a7ec979 in_seq=0 out_seq=0 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3ed9ffb640 10 client.19143 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3ed9ffb640 1 -- 172.21.15.89:0/2122090520 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56415a255d60 con 0x56415a36e1b0 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3ed9ffb640 10 client.19143 handle_command_reply: tid=0 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 client.19143 shutdown 2024-09-15T23:34:50.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 -- 172.21.15.89:0/2122090520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 msgr2=0x56415a2f99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a2f99e0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f3eec02c010 tx=0x7f3eec009560 comp rx=0 tx=0).stop 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 -- 172.21.15.89:0/2122090520 shutdown_connections 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56415a36e1b0 0x56415a3705a0 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56415a2f3680 0x56415a2f99e0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56415a2e8dc0 0x56415a2f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 --2- 172.21.15.89:0/2122090520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56415a2e84a0 0x56415a2fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3ed9ffb640 0 client.19143 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 -- 172.21.15.89:0/2122090520 >> 172.21.15.89:0/2122090520 conn(0x56415a2e4a30 msgr2=0x56415a2e3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 -- 172.21.15.89:0/2122090520 shutdown_connections 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.330+0000 7f3efada4740 1 -- 172.21.15.89:0/2122090520 wait complete. 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3efada4740 20 client.19143 trim_cache size 0 max 16384 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 -- 172.21.15.89:0/4153473037 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3edc0a3240 msgr2=0x7f3edc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3edc0a3240 0x7f3edc0a5700 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009bf0 tx=0x7f3eec047000 comp rx=0 tx=0).stop 2024-09-15T23:34:50.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 -- 172.21.15.89:0/4153473037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ef41008e0 msgr2=0x7f3ef4101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:50.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ef41008e0 0x7f3ef4101860 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400d770 tx=0x7f3ee400dc30 comp rx=0 tx=0).stop 2024-09-15T23:34:50.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 -- 172.21.15.89:0/4153473037 shutdown_connections 2024-09-15T23:34:50.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3edc0a3240 0x7f3edc0a5700 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ef4108110 0x7f3ef41022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ef4101290 0x7f3ef4101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 --2- 172.21.15.89:0/4153473037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ef41008e0 0x7f3ef4101860 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:50.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 -- 172.21.15.89:0/4153473037 >> 172.21.15.89:0/4153473037 conn(0x7f3ef4077210 msgr2=0x7f3ef40ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:50.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 -- 172.21.15.89:0/4153473037 shutdown_connections 2024-09-15T23:34:50.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:34:50.331+0000 7f3ef89cb640 1 -- 172.21.15.89:0/4153473037 wait complete. 2024-09-15T23:34:51.143 INFO:tasks.cephfs.filesystem:scrub status for tag:4ca9439b-b222-4317-be65-1d7d36d9c8f9 - {'path': '/', 'tag': '4ca9439b-b222-4317-be65-1d7d36d9c8f9', 'options': 'recursive,force'} 2024-09-15T23:34:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:51 smithi120 ceph-mon[35250]: pgmap v674: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:51 smithi174 ceph-mon[35586]: pgmap v674: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:51 smithi089 ceph-mon[30933]: pgmap v674: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:34:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:53 smithi120 ceph-mon[35250]: pgmap v675: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-15T23:34:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:53 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:34:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:53 smithi174 ceph-mon[35586]: pgmap v675: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-15T23:34:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:53 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:34:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:53 smithi089 ceph-mon[30933]: pgmap v675: 129 pgs: 129 active+clean; 1.3 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-15T23:34:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:53 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:34:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:55 smithi120 ceph-mon[35250]: pgmap v676: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:34:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:55 smithi174 ceph-mon[35586]: pgmap v676: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:34:55.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:55 smithi089 ceph-mon[30933]: pgmap v676: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:34:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:57 smithi120 ceph-mon[35250]: pgmap v677: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:34:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:57 smithi174 ceph-mon[35586]: pgmap v677: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:34:57.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:57 smithi089 ceph-mon[30933]: pgmap v677: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:34:58.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.639+0000 7f950de4c640 1 -- 172.21.15.89:0/4075393402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508100a60 msgr2=0x7f9508100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:58.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.639+0000 7f950de4c640 1 --2- 172.21.15.89:0/4075393402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508100a60 0x7f9508100e60 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f94f8009900 tx=0x7f94f802a410 comp rx=0 tx=0).stop 2024-09-15T23:34:58.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.639+0000 7f950de4c640 1 -- 172.21.15.89:0/4075393402 shutdown_connections 2024-09-15T23:34:58.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.639+0000 7f950de4c640 1 --2- 172.21.15.89:0/4075393402 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9508102610 0x7f9508108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.639+0000 7f950de4c640 1 --2- 172.21.15.89:0/4075393402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9508101c50 0x7f95081020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.639+0000 7f950de4c640 1 --2- 172.21.15.89:0/4075393402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508100a60 0x7f9508100e60 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.639+0000 7f950de4c640 1 -- 172.21.15.89:0/4075393402 >> 172.21.15.89:0/4075393402 conn(0x7f95080fc1e0 msgr2=0x7f95080fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:58.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.639+0000 7f950de4c640 1 -- 172.21.15.89:0/4075393402 shutdown_connections 2024-09-15T23:34:58.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.639+0000 7f950de4c640 1 -- 172.21.15.89:0/4075393402 wait complete. 2024-09-15T23:34:58.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.639+0000 7f950de4c640 1 Processor -- start 2024-09-15T23:34:58.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f950de4c640 1 -- start start 2024-09-15T23:34:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f950de4c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9508100a60 0x7f9508196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f950de4c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9508101c50 0x7f9508197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f950de4c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508102610 0x7f950819e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f950de4c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f950810c6c0 con 0x7f9508102610 2024-09-15T23:34:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f950de4c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f950810c540 con 0x7f9508100a60 2024-09-15T23:34:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f950de4c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f950810c840 con 0x7f9508101c50 2024-09-15T23:34:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9507fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508102610 0x7f950819e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9507fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508102610 0x7f950819e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49416/0 (socket says 172.21.15.89:49416) 2024-09-15T23:34:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9507fff640 1 -- 172.21.15.89:0/1121971534 learned_addr learned my addr 172.21.15.89:0/1121971534 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:58.643 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9506ffd640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9508101c50 0x7f9508197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f95077fe640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9508100a60 0x7f9508196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9507fff640 1 -- 172.21.15.89:0/1121971534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9508101c50 msgr2=0x7f9508197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9507fff640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9508101c50 0x7f9508197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9507fff640 1 -- 172.21.15.89:0/1121971534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9508100a60 msgr2=0x7f9508196d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9507fff640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9508100a60 0x7f9508196d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9507fff640 1 -- 172.21.15.89:0/1121971534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950819e940 con 0x7f9508102610 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f95077fe640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9508100a60 0x7f9508196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9507fff640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508102610 0x7f950819e2a0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f950000ebb0 tx=0x7f9500007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9504ff9640 1 -- 172.21.15.89:0/1121971534 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500007ca0 con 0x7f9508102610 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f9504ff9640 1 -- 172.21.15.89:0/1121971534 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9500007e40 con 0x7f9508102610 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.640+0000 7f950de4c640 1 -- 172.21.15.89:0/1121971534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950806a300 con 0x7f9508102610 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.641+0000 7f9504ff9640 1 -- 172.21.15.89:0/1121971534 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000058a0 con 0x7f9508102610 2024-09-15T23:34:58.644 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.641+0000 7f950de4c640 1 -- 172.21.15.89:0/1121971534 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f950806a850 con 0x7f9508102610 2024-09-15T23:34:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.643+0000 7f9504ff9640 1 -- 172.21.15.89:0/1121971534 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9500004a40 con 0x7f9508102610 2024-09-15T23:34:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.643+0000 7f9510be7740 1 Processor -- start 2024-09-15T23:34:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.643+0000 7f9510be7740 1 -- start start 2024-09-15T23:34:58.651 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9510be7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404a99a0 0x55a8404a9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9510be7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9510be7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404af370 0x55a8404b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9510be7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a84040d2f0 con 0x55a8404aa2e0 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9510be7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a8403b5270 con 0x55a8404af370 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9510be7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a8404092b0 con 0x55a8404a99a0 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f95077fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f95077fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49418/0 (socket says 172.21.15.89:49418) 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f95077fe640 1 -- 172.21.15.89:0/3809033143 learned_addr learned my addr 172.21.15.89:0/3809033143 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9506ffd640 1 --2- 172.21.15.89:0/3809033143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404a99a0 0x55a8404a9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9506ffd640 1 --2- 172.21.15.89:0/3809033143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404af370 0x55a8404b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9504ff9640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94f00a3370 0x7f94f00a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f95077fe640 1 -- 172.21.15.89:0/3809033143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404a99a0 msgr2=0x55a8404a9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f95077fe640 1 --2- 172.21.15.89:0/3809033143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404a99a0 0x55a8404a9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f95077fe640 1 -- 172.21.15.89:0/3809033143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404af370 msgr2=0x55a8404b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f95077fe640 1 --2- 172.21.15.89:0/3809033143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404af370 0x55a8404b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f95077fe640 1 -- 172.21.15.89:0/3809033143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8403b4340 con 0x55a8404aa2e0 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9507fff640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94f00a3370 0x7f94f00a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f9504ff9640 1 -- 172.21.15.89:0/1121971534 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f95000d13b0 con 0x7f9508102610 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.644+0000 7f95077fe640 1 --2- 172.21.15.89:0/3809033143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404aeda0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f94f8006010 tx=0x7f94f8032d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f594a5fd8dec81d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f94edffb640 1 -- 172.21.15.89:0/3809033143 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f803f070 con 0x55a8404aa2e0 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f94edffb640 1 -- 172.21.15.89:0/3809033143 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f80318b0 con 0x55a8404aa2e0 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f94edffb640 1 -- 172.21.15.89:0/3809033143 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8035070 con 0x55a8404aa2e0 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f9510be7740 1 -- 172.21.15.89:0/3809033143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 msgr2=0x55a8404aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f9510be7740 1 --2- 172.21.15.89:0/3809033143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404aeda0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f94f8006010 tx=0x7f94f8032d80 comp rx=0 tx=0).stop 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f9507fff640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94f00a3370 0x7f94f00a5830 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f950009d2e0 tx=0x7f9500005340 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f9510be7740 1 -- 172.21.15.89:0/3809033143 shutdown_connections 2024-09-15T23:34:58.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f9510be7740 1 --2- 172.21.15.89:0/3809033143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404af370 0x55a8404b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f9510be7740 1 --2- 172.21.15.89:0/3809033143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404aeda0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f9510be7740 1 --2- 172.21.15.89:0/3809033143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404a99a0 0x55a8404a9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f9510be7740 1 -- 172.21.15.89:0/3809033143 >> 172.21.15.89:0/3809033143 conn(0x55a8404a5010 msgr2=0x55a8404a5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f9510be7740 1 -- 172.21.15.89:0/3809033143 shutdown_connections 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.645+0000 7f9510be7740 1 -- 172.21.15.89:0/3809033143 wait complete. 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.646+0000 7f9510be7740 1 -- start start 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.646+0000 7f9510be7740 1 Processor -- start 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.646+0000 7f9510be7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404a99a0 0x55a8404b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.646+0000 7f9510be7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404aa2e0 0x55a8404b4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.646+0000 7f9510be7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404bb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.646+0000 7f9510be7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a8404b22e0 con 0x55a8404af370 2024-09-15T23:34:58.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.646+0000 7f9510be7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a840396290 con 0x55a8404aa2e0 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.646+0000 7f9510be7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a8404b2460 con 0x55a8404a99a0 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.646+0000 7f9510be7740 10 client.0 _ll_register_callbacks cb 0x55a8403c7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9506ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404bb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9506ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404bb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49424/0 (socket says 172.21.15.89:49424) 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9506ffd640 1 -- 172.21.15.89:0/431238790 learned_addr learned my addr 172.21.15.89:0/431238790 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9506ffd640 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404a99a0 0x55a8404b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9506ffd640 1 -- 172.21.15.89:0/431238790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404a99a0 msgr2=0x55a8404b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f95077fe640 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404aa2e0 0x55a8404b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9506ffd640 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404a99a0 0x55a8404b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9506ffd640 1 -- 172.21.15.89:0/431238790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404aa2e0 msgr2=0x55a8404b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9506ffd640 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404aa2e0 0x55a8404b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9506ffd640 1 -- 172.21.15.89:0/431238790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a840120210 con 0x55a8404af370 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f95077fe640 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404aa2e0 0x55a8404b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:58.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9506ffd640 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404bb170 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f94fc012ec0 tx=0x7f94fc00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b388360c99307cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f94e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f94e6ffd640 1 -- 172.21.15.89:0/431238790 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc004230 con 0x55a8404af370 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9510be7740 1 -- 172.21.15.89:0/431238790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94f80318b0 con 0x55a8404af370 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f94e6ffd640 1 -- 172.21.15.89:0/431238790 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f94fc0043d0 con 0x55a8404af370 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f94e6ffd640 1 -- 172.21.15.89:0/431238790 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94fc027050 con 0x55a8404af370 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9510be7740 10 client.19155 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.647+0000 7f9510be7740 1 -- 172.21.15.89:0/431238790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8401ec620 con 0x55a8404af370 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.648+0000 7f94e6ffd640 1 -- 172.21.15.89:0/431238790 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f80318b0 con 0x55a8404af370 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.648+0000 7f94e6ffd640 1 -- 172.21.15.89:0/431238790 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f94fc0117a0 con 0x55a8404af370 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.648+0000 7f9510be7740 10 client.19155 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.648+0000 7f9510be7740 10 client.19155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.648+0000 7f9510be7740 20 client.19155 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.648+0000 7f9510be7740 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a8404bf890 0x55a8404c1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.648+0000 7f9510be7740 4 client.19155 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.648+0000 7f9510be7740 1 -- 172.21.15.89:0/431238790 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a84041aa20 con 0x55a8404bf890 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.648+0000 7f95077fe640 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a8404bf890 0x55a8404c1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.649+0000 7f94e6ffd640 10 client.19155 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.649+0000 7f95077fe640 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a8404bf890 0x55a8404c1d00 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d91cdb388689ff5 server_cookie=c6312f4e757033fa in_seq=0 out_seq=0 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.649+0000 7f94e6ffd640 1 -- 172.21.15.89:0/431238790 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a84041aa20 con 0x55a8404bf890 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.649+0000 7f94e6ffd640 10 client.19155 handle_command_reply: tid=0 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.649+0000 7f9510be7740 1 client.19155 shutdown 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.649+0000 7f9510be7740 1 -- 172.21.15.89:0/431238790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 msgr2=0x55a8404bb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.649+0000 7f9510be7740 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404bb170 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f94fc012ec0 tx=0x7f94fc00cb30 comp rx=0 tx=0).stop 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.650+0000 7f9510be7740 1 -- 172.21.15.89:0/431238790 shutdown_connections 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.650+0000 7f9510be7740 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a8404bf890 0x55a8404c1d00 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.650+0000 7f9510be7740 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404bb170 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.650+0000 7f9510be7740 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404aa2e0 0x55a8404b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.650+0000 7f94e6ffd640 0 client.19155 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.650+0000 7f9510be7740 1 --2- 172.21.15.89:0/431238790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404a99a0 0x55a8404b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.650+0000 7f9510be7740 1 -- 172.21.15.89:0/431238790 >> 172.21.15.89:0/431238790 conn(0x55a8404a5010 msgr2=0x55a8404a6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.650+0000 7f9510be7740 1 -- 172.21.15.89:0/431238790 shutdown_connections 2024-09-15T23:34:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.650+0000 7f9510be7740 1 -- 172.21.15.89:0/431238790 wait complete. 2024-09-15T23:34:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.650+0000 7f9510be7740 20 client.19155 trim_cache size 0 max 16384 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9510be7740 1 Processor -- start 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9510be7740 1 -- start start 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9510be7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 0x55a8404a5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9510be7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404aa2e0 0x55a8404bb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9510be7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404a6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9510be7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f95081a8590 con 0x55a8404af370 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9510be7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f95081aa0d0 con 0x55a8404a99a0 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9510be7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f95081a8c60 con 0x55a8404aa2e0 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9506ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9506ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404a6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49432/0 (socket says 172.21.15.89:49432) 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.664+0000 7f9506ffd640 1 -- 172.21.15.89:0/2277402925 learned_addr learned my addr 172.21.15.89:0/2277402925 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f9506ffd640 1 --2- 172.21.15.89:0/2277402925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 0x55a8404a5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f95077fe640 1 --2- 172.21.15.89:0/2277402925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404aa2e0 0x55a8404bb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f9506ffd640 1 -- 172.21.15.89:0/2277402925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404aa2e0 msgr2=0x55a8404bb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f9506ffd640 1 --2- 172.21.15.89:0/2277402925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404aa2e0 0x55a8404bb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f9506ffd640 1 -- 172.21.15.89:0/2277402925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 msgr2=0x55a8404a5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f9506ffd640 1 --2- 172.21.15.89:0/2277402925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 0x55a8404a5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f9506ffd640 1 -- 172.21.15.89:0/2277402925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8404aa7c0 con 0x55a8404af370 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f9506ffd640 1 --2- 172.21.15.89:0/2277402925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404a6400 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0058c0 tx=0x7f94fc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9861e83226215df3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f94ed7fa640 1 -- 172.21.15.89:0/2277402925 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc004030 con 0x55a8404af370 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f94ed7fa640 1 -- 172.21.15.89:0/2277402925 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94fc011920 con 0x55a8404af370 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.665+0000 7f94ed7fa640 1 -- 172.21.15.89:0/2277402925 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94fc0038e0 con 0x55a8404af370 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 -- 172.21.15.89:0/2277402925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 msgr2=0x55a8404a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 --2- 172.21.15.89:0/2277402925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404a6400 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0058c0 tx=0x7f94fc005850 comp rx=0 tx=0).stop 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 -- 172.21.15.89:0/2277402925 shutdown_connections 2024-09-15T23:34:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 --2- 172.21.15.89:0/2277402925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404af370 0x55a8404a6400 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 --2- 172.21.15.89:0/2277402925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404aa2e0 0x55a8404bb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 --2- 172.21.15.89:0/2277402925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 0x55a8404a5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 -- 172.21.15.89:0/2277402925 >> 172.21.15.89:0/2277402925 conn(0x55a8404a5010 msgr2=0x55a8404b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 -- 172.21.15.89:0/2277402925 shutdown_connections 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 -- 172.21.15.89:0/2277402925 wait complete. 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 -- start start 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 Processor -- start 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 0x55a8404b8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.666+0000 7f9510be7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9510be7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404af370 0x55a8404b9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9510be7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f95081a9a00 con 0x55a8404aa2e0 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9510be7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f95081a9330 con 0x55a8404a99a0 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9510be7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f95081a6570 con 0x55a8404af370 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9510be7740 10 client.0 _ll_register_callbacks cb 0x7f94fc00cdd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:34:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f95077fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f95077fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404b8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49438/0 (socket says 172.21.15.89:49438) 2024-09-15T23:34:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f95077fe640 1 -- 172.21.15.89:0/3437858357 learned_addr learned my addr 172.21.15.89:0/3437858357 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:34:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9506ffd640 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404af370 0x55a8404b9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f95077fe640 1 -- 172.21.15.89:0/3437858357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404af370 msgr2=0x55a8404b9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f95077fe640 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404af370 0x55a8404b9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f95077fe640 1 -- 172.21.15.89:0/3437858357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 msgr2=0x55a8404b8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9506ffd640 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 0x55a8404b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f95077fe640 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 0x55a8404b8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f95077fe640 1 -- 172.21.15.89:0/3437858357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8404ba4d0 con 0x55a8404aa2e0 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9506ffd640 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 0x55a8404b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f95077fe640 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404b8eb0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f94f80314e0 tx=0x7f94f802aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48449ba3ff7b3a22 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f94e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f94e6ffd640 1 -- 172.21.15.89:0/3437858357 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8051060 con 0x55a8404aa2e0 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f94e6ffd640 1 -- 172.21.15.89:0/3437858357 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f94f8004230 con 0x55a8404aa2e0 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9510be7740 1 -- 172.21.15.89:0/3437858357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94fc011920 con 0x55a8404aa2e0 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f94e6ffd640 1 -- 172.21.15.89:0/3437858357 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94f8058050 con 0x55a8404aa2e0 2024-09-15T23:34:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9510be7740 10 client.19161 fetch_fsmap learned FSMap version 29 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.667+0000 7f9510be7740 1 -- 172.21.15.89:0/3437858357 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a84052a830 con 0x55a8404aa2e0 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f94e6ffd640 1 -- 172.21.15.89:0/3437858357 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94fc011920 con 0x55a8404aa2e0 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f94e6ffd640 1 -- 172.21.15.89:0/3437858357 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f94f8004aa0 con 0x55a8404aa2e0 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f9510be7740 10 client.19161 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f9510be7740 10 client.19161 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f9510be7740 20 client.19161 populate_metadata read hostname 'smithi089' 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f9510be7740 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a84052adf0 0x55a84052d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f9510be7740 4 client.19161 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f9510be7740 1 -- 172.21.15.89:0/3437858357 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a840414d70 con 0x55a84052adf0 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f9506ffd640 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a84052adf0 0x55a84052d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f94e6ffd640 10 client.19161 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.668+0000 7f9506ffd640 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a84052adf0 0x55a84052d1e0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a3bb8b964cb1127 server_cookie=85ef013684b7922e in_seq=0 out_seq=0 2024-09-15T23:34:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f94e6ffd640 1 -- 172.21.15.89:0/3437858357 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a840414d70 con 0x55a84052adf0 2024-09-15T23:34:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f94e6ffd640 10 client.19161 handle_command_reply: tid=0 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 client.19161 shutdown 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 -- 172.21.15.89:0/3437858357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 msgr2=0x55a8404b8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404b8eb0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f94f80314e0 tx=0x7f94f802aa90 comp rx=0 tx=0).stop 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 -- 172.21.15.89:0/3437858357 shutdown_connections 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a84052adf0 0x55a84052d1e0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a8404af370 0x55a8404b9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a8404aa2e0 0x55a8404b8eb0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f94e6ffd640 0 client.19161 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 --2- 172.21.15.89:0/3437858357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a8404a99a0 0x55a8404b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 -- 172.21.15.89:0/3437858357 >> 172.21.15.89:0/3437858357 conn(0x55a8404a5010 msgr2=0x55a8404a39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 -- 172.21.15.89:0/3437858357 shutdown_connections 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 1 -- 172.21.15.89:0/3437858357 wait complete. 2024-09-15T23:34:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.682+0000 7f9510be7740 20 client.19161 trim_cache size 0 max 16384 2024-09-15T23:34:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:34:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:34:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:34:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:34:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 -- 172.21.15.89:0/1121971534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94f00a3370 msgr2=0x7f94f00a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94f00a3370 0x7f94f00a5830 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f950009d2e0 tx=0x7f9500005340 comp rx=0 tx=0).stop 2024-09-15T23:34:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 -- 172.21.15.89:0/1121971534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508102610 msgr2=0x7f950819e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:34:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508102610 0x7f950819e2a0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f950000ebb0 tx=0x7f9500007590 comp rx=0 tx=0).stop 2024-09-15T23:34:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 -- 172.21.15.89:0/1121971534 shutdown_connections 2024-09-15T23:34:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94f00a3370 0x7f94f00a5830 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9508102610 0x7f950819e2a0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9508101c50 0x7f9508197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 --2- 172.21.15.89:0/1121971534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9508100a60 0x7f9508196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:34:58.686 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 -- 172.21.15.89:0/1121971534 >> 172.21.15.89:0/1121971534 conn(0x7f95080fc1e0 msgr2=0x7f9508105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:34:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 -- 172.21.15.89:0/1121971534 shutdown_connections 2024-09-15T23:34:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:34:58.683+0000 7f950de4c640 1 -- 172.21.15.89:0/1121971534 wait complete. 2024-09-15T23:34:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:34:59 smithi120 ceph-mon[35250]: pgmap v678: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:34:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:34:59 smithi174 ceph-mon[35586]: pgmap v678: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:34:59.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:34:59 smithi089 ceph-mon[30933]: pgmap v678: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:35:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:01 smithi120 ceph-mon[35250]: pgmap v679: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:35:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:01 smithi174 ceph-mon[35586]: pgmap v679: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:35:01.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:01 smithi089 ceph-mon[30933]: pgmap v679: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:35:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:03 smithi120 ceph-mon[35250]: pgmap v680: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 18 op/s 2024-09-15T23:35:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:03 smithi174 ceph-mon[35586]: pgmap v680: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 18 op/s 2024-09-15T23:35:03.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:03 smithi089 ceph-mon[30933]: pgmap v680: 129 pgs: 129 active+clean; 1.4 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 18 op/s 2024-09-15T23:35:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:05 smithi120 ceph-mon[35250]: pgmap v681: 129 pgs: 129 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:35:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:35:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:35:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:05 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:35:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:05 smithi174 ceph-mon[35586]: pgmap v681: 129 pgs: 129 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:35:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:35:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:35:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:05 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:35:05.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:05 smithi089 ceph-mon[30933]: pgmap v681: 129 pgs: 129 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:35:05.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:35:05.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:35:05.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:05 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:35:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:35:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:35:06.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:35:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:35:06.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:35:06.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:35:06.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:35:06.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:35:06.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:35:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:07 smithi120 ceph-mon[35250]: pgmap v682: 129 pgs: 129 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:35:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:07 smithi174 ceph-mon[35586]: pgmap v682: 129 pgs: 129 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:35:07.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:07 smithi089 ceph-mon[30933]: pgmap v682: 129 pgs: 129 active+clean; 1.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:35:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:09 smithi120 ceph-mon[35250]: pgmap v683: 129 pgs: 129 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:35:09.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:09 smithi174 ceph-mon[35586]: pgmap v683: 129 pgs: 129 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:35:09.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:09 smithi089 ceph-mon[30933]: pgmap v683: 129 pgs: 129 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:35:11.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:11 smithi120 ceph-mon[35250]: pgmap v684: 129 pgs: 129 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:35:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:11 smithi174 ceph-mon[35586]: pgmap v684: 129 pgs: 129 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:35:11.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:11 smithi089 ceph-mon[30933]: pgmap v684: 129 pgs: 129 active+clean; 1.6 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.779+0000 7ff1e667b640 1 -- 172.21.15.89:0/290755676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e0102610 msgr2=0x7ff1e0108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.779+0000 7ff1e667b640 1 --2- 172.21.15.89:0/290755676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e0102610 0x7ff1e0108ea0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009ca0 tx=0x7ff1d802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.779+0000 7ff1e667b640 1 -- 172.21.15.89:0/290755676 shutdown_connections 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.779+0000 7ff1e667b640 1 --2- 172.21.15.89:0/290755676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e0102610 0x7ff1e0108ea0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.779+0000 7ff1e667b640 1 --2- 172.21.15.89:0/290755676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e0101c50 0x7ff1e01020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.779+0000 7ff1e667b640 1 --2- 172.21.15.89:0/290755676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e0100a60 0x7ff1e0100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.779+0000 7ff1e667b640 1 -- 172.21.15.89:0/290755676 >> 172.21.15.89:0/290755676 conn(0x7ff1e00fc1e0 msgr2=0x7ff1e00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.779+0000 7ff1e667b640 1 -- 172.21.15.89:0/290755676 shutdown_connections 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.779+0000 7ff1e667b640 1 -- 172.21.15.89:0/290755676 wait complete. 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1e667b640 1 Processor -- start 2024-09-15T23:35:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1e667b640 1 -- start start 2024-09-15T23:35:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1e667b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e0100a60 0x7ff1e0197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1e667b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e0101c50 0x7ff1e01982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1e667b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e0102610 0x7ff1e019f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1e667b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e010d830 con 0x7ff1e0100a60 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1e667b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e010d6b0 con 0x7ff1e0101c50 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1e667b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e010d9b0 con 0x7ff1e0102610 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1dffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e0100a60 0x7ff1e0197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1dffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e0100a60 0x7ff1e0197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49000/0 (socket says 172.21.15.89:49000) 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.780+0000 7ff1dffff640 1 -- 172.21.15.89:0/3576720302 learned_addr learned my addr 172.21.15.89:0/3576720302 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1e4c18640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e0102610 0x7ff1e019f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1df7fe640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e0101c50 0x7ff1e01982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1dffff640 1 -- 172.21.15.89:0/3576720302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e0102610 msgr2=0x7ff1e019f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1dffff640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e0102610 0x7ff1e019f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1dffff640 1 -- 172.21.15.89:0/3576720302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e0101c50 msgr2=0x7ff1e01982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1dffff640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e0101c50 0x7ff1e01982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1dffff640 1 -- 172.21.15.89:0/3576720302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e019f9d0 con 0x7ff1e0100a60 2024-09-15T23:35:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1e4c18640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e0102610 0x7ff1e019f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:11.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1df7fe640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e0101c50 0x7ff1e01982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:11.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1dffff640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e0100a60 0x7ff1e0197db0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000d950 tx=0x7ff1d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:11.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1dd7fa640 1 -- 172.21.15.89:0/3576720302 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0017020 con 0x7ff1e0100a60 2024-09-15T23:35:11.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1dd7fa640 1 -- 172.21.15.89:0/3576720302 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d0009c30 con 0x7ff1e0100a60 2024-09-15T23:35:11.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1e667b640 1 -- 172.21.15.89:0/3576720302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e019fcc0 con 0x7ff1e0100a60 2024-09-15T23:35:11.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1dd7fa640 1 -- 172.21.15.89:0/3576720302 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d000f720 con 0x7ff1e0100a60 2024-09-15T23:35:11.785 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.781+0000 7ff1e667b640 1 -- 172.21.15.89:0/3576720302 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e01a01c0 con 0x7ff1e0100a60 2024-09-15T23:35:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.784+0000 7ff1dd7fa640 1 -- 172.21.15.89:0/3576720302 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff1d001a020 con 0x7ff1e0100a60 2024-09-15T23:35:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.784+0000 7ff1e941c740 1 Processor -- start 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.784+0000 7ff1e941c740 1 -- start start 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.784+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.784+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b12da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.784+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b15830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.784+0000 7ff1e941c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e042a712f0 con 0x55e042b0d9a0 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e941c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e042a19270 con 0x55e042b0e2e0 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e941c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e042a6d2b0 con 0x55e042b13370 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e4c18640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e4c18640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49004/0 (socket says 172.21.15.89:49004) 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e4c18640 1 -- 172.21.15.89:0/1022374896 learned_addr learned my addr 172.21.15.89:0/1022374896 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1df7fe640 1 --2- 172.21.15.89:0/1022374896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b15830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1dffff640 1 --2- 172.21.15.89:0/1022374896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b12da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e4c18640 1 -- 172.21.15.89:0/1022374896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 msgr2=0x55e042b15830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1dd7fa640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1c80a32f0 0x7ff1c80a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e4c18640 1 --2- 172.21.15.89:0/1022374896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b15830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e4c18640 1 -- 172.21.15.89:0/1022374896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 msgr2=0x55e042b12da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e4c18640 1 --2- 172.21.15.89:0/1022374896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b12da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e4c18640 1 -- 172.21.15.89:0/1022374896 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e042a18340 con 0x55e042b0d9a0 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1dffff640 1 --2- 172.21.15.89:0/1022374896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b12da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1dd7fa640 1 -- 172.21.15.89:0/3576720302 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff1d00cfd10 con 0x7ff1e0100a60 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e4c18640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1c80a32f0 0x7ff1c80a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.785+0000 7ff1e4c18640 1 --2- 172.21.15.89:0/1022374896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dda0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8035000 tx=0x7ff1d8005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e53c764c340c35e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1c67fc640 1 -- 172.21.15.89:0/1022374896 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8031490 con 0x55e042b0d9a0 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1c67fc640 1 -- 172.21.15.89:0/1022374896 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d8036050 con 0x55e042b0d9a0 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1c67fc640 1 -- 172.21.15.89:0/1022374896 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8004d40 con 0x55e042b0d9a0 2024-09-15T23:35:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1e941c740 1 -- 172.21.15.89:0/1022374896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 msgr2=0x55e042b0dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1e4c18640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1c80a32f0 0x7ff1c80a57b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8006010 tx=0x7ff1d804a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1e941c740 1 --2- 172.21.15.89:0/1022374896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dda0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8035000 tx=0x7ff1d8005f70 comp rx=0 tx=0).stop 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1e941c740 1 -- 172.21.15.89:0/1022374896 shutdown_connections 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1e941c740 1 --2- 172.21.15.89:0/1022374896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b15830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1e941c740 1 --2- 172.21.15.89:0/1022374896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b12da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1e941c740 1 --2- 172.21.15.89:0/1022374896 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dda0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1e941c740 1 -- 172.21.15.89:0/1022374896 >> 172.21.15.89:0/1022374896 conn(0x55e042b09010 msgr2=0x55e042b09410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1e941c740 1 -- 172.21.15.89:0/1022374896 shutdown_connections 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.786+0000 7ff1e941c740 1 -- 172.21.15.89:0/1022374896 wait complete. 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.787+0000 7ff1e941c740 1 -- start start 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.787+0000 7ff1e941c740 1 Processor -- start 2024-09-15T23:35:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.787+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b0d9a0 0x55e042b179e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.787+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0e2e0 0x55e042b17f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.787+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b13370 0x55e042b1ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.787+0000 7ff1e941c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e042b161c0 con 0x55e042b0e2e0 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.787+0000 7ff1e941c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e0429fa290 con 0x55e042b13370 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.787+0000 7ff1e941c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e042b16340 con 0x55e042b0d9a0 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.787+0000 7ff1e941c740 10 client.0 _ll_register_callbacks cb 0x55e042a2b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.787+0000 7ff1df7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0e2e0 0x55e042b17f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1df7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0e2e0 0x55e042b17f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49012/0 (socket says 172.21.15.89:49012) 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1df7fe640 1 -- 172.21.15.89:0/300949157 learned_addr learned my addr 172.21.15.89:0/300949157 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1dffff640 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b0d9a0 0x55e042b179e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1e4c18640 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b13370 0x55e042b1ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1df7fe640 1 -- 172.21.15.89:0/300949157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b0d9a0 msgr2=0x55e042b179e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1df7fe640 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b0d9a0 0x55e042b179e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1df7fe640 1 -- 172.21.15.89:0/300949157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b13370 msgr2=0x55e042b1ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1df7fe640 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b13370 0x55e042b1ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1df7fe640 1 -- 172.21.15.89:0/300949157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e042784210 con 0x55e042b0e2e0 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1e4c18640 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b13370 0x55e042b1ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1df7fe640 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0e2e0 0x55e042b17f20 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009f10 tx=0x7ff1d40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e9f4144f88777c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/300949157 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4012860 con 0x55e042b0e2e0 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/300949157 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff1d4012a00 con 0x55e042b0e2e0 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1e941c740 1 -- 172.21.15.89:0/300949157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d8036050 con 0x55e042b0e2e0 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/300949157 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff1d4025050 con 0x55e042b0e2e0 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1e941c740 10 client.19173 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.788+0000 7ff1e941c740 1 -- 172.21.15.89:0/300949157 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e042850620 con 0x55e042b0e2e0 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.789+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/300949157 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d8036050 con 0x55e042b0e2e0 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.789+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/300949157 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff1d400f040 con 0x55e042b0e2e0 2024-09-15T23:35:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.789+0000 7ff1e941c740 10 client.19173 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.789+0000 7ff1e941c740 10 client.19173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.789+0000 7ff1e941c740 20 client.19173 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.789+0000 7ff1e941c740 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e042b23720 0x55e042b25b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.789+0000 7ff1e941c740 4 client.19173 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.789+0000 7ff1e941c740 1 -- 172.21.15.89:0/300949157 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e042a7ea20 con 0x55e042b23720 2024-09-15T23:35:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.789+0000 7ff1dffff640 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e042b23720 0x55e042b25b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.790+0000 7ff1bf7fe640 10 client.19173 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.790+0000 7ff1dffff640 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e042b23720 0x55e042b25b70 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a5338f37233d003 server_cookie=c8b413972a174d02 in_seq=0 out_seq=0 2024-09-15T23:35:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.790+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/300949157 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e042a7ea20 con 0x55e042b23720 2024-09-15T23:35:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.790+0000 7ff1bf7fe640 10 client.19173 handle_command_reply: tid=0 2024-09-15T23:35:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.790+0000 7ff1e941c740 1 client.19173 shutdown 2024-09-15T23:35:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.790+0000 7ff1e941c740 1 -- 172.21.15.89:0/300949157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0e2e0 msgr2=0x55e042b17f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.790+0000 7ff1e941c740 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0e2e0 0x55e042b17f20 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009f10 tx=0x7ff1d40117b0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.791+0000 7ff1e941c740 1 -- 172.21.15.89:0/300949157 shutdown_connections 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.791+0000 7ff1e941c740 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e042b23720 0x55e042b25b70 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.791+0000 7ff1e941c740 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b13370 0x55e042b1ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.791+0000 7ff1e941c740 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0e2e0 0x55e042b17f20 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.791+0000 7ff1bf7fe640 0 client.19173 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.791+0000 7ff1e941c740 1 --2- 172.21.15.89:0/300949157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b0d9a0 0x55e042b179e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.791+0000 7ff1e941c740 1 -- 172.21.15.89:0/300949157 >> 172.21.15.89:0/300949157 conn(0x55e042b09010 msgr2=0x55e042b0a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.791+0000 7ff1e941c740 1 -- 172.21.15.89:0/300949157 shutdown_connections 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.791+0000 7ff1e941c740 1 -- 172.21.15.89:0/300949157 wait complete. 2024-09-15T23:35:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.791+0000 7ff1e941c740 20 client.19173 trim_cache size 0 max 16384 2024-09-15T23:35:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1e941c740 1 Processor -- start 2024-09-15T23:35:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1e941c740 1 -- start start 2024-09-15T23:35:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b1ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1e941c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01aa8e0 con 0x55e042b0d9a0 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1e941c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01ac420 con 0x55e042b0e2e0 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1e941c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01aafb0 con 0x55e042b13370 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1dffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1dffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49028/0 (socket says 172.21.15.89:49028) 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1dffff640 1 -- 172.21.15.89:0/2492872318 learned_addr learned my addr 172.21.15.89:0/2492872318 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1dffff640 1 -- 172.21.15.89:0/2492872318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 msgr2=0x55e042b080b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1dffff640 1 --2- 172.21.15.89:0/2492872318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1dffff640 1 -- 172.21.15.89:0/2492872318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 msgr2=0x55e042b1ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1dffff640 1 --2- 172.21.15.89:0/2492872318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b1ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.805+0000 7ff1dffff640 1 -- 172.21.15.89:0/2492872318 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e042b0e7c0 con 0x55e042b0d9a0 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1dffff640 1 --2- 172.21.15.89:0/2492872318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dd80 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009610 tx=0x7ff1d00bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5005080eb495185e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1c5ffb640 1 -- 172.21.15.89:0/2492872318 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d009c640 con 0x55e042b0d9a0 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1c5ffb640 1 -- 172.21.15.89:0/2492872318 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d009bf40 con 0x55e042b0d9a0 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1c5ffb640 1 -- 172.21.15.89:0/2492872318 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00985b0 con 0x55e042b0d9a0 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1e941c740 1 -- 172.21.15.89:0/2492872318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 msgr2=0x55e042b0dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1e941c740 1 --2- 172.21.15.89:0/2492872318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dd80 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009610 tx=0x7ff1d00bc040 comp rx=0 tx=0).stop 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1e941c740 1 -- 172.21.15.89:0/2492872318 shutdown_connections 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1e941c740 1 --2- 172.21.15.89:0/2492872318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1e941c740 1 --2- 172.21.15.89:0/2492872318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b1ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1e941c740 1 --2- 172.21.15.89:0/2492872318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b0dd80 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1e941c740 1 -- 172.21.15.89:0/2492872318 >> 172.21.15.89:0/2492872318 conn(0x55e042b09010 msgr2=0x55e042b079a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1e941c740 1 -- 172.21.15.89:0/2492872318 shutdown_connections 2024-09-15T23:35:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.806+0000 7ff1e941c740 1 -- 172.21.15.89:0/2492872318 wait complete. 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1e941c740 1 -- start start 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1e941c740 1 Processor -- start 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b1c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b1c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1e941c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b1da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1e941c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01abd50 con 0x55e042b0d9a0 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1e941c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01ab680 con 0x55e042b0e2e0 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1e941c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e01a88c0 con 0x55e042b13370 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1e941c740 10 client.0 _ll_register_callbacks cb 0x7ff1d40096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1dffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b1c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1dffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b1c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49042/0 (socket says 172.21.15.89:49042) 2024-09-15T23:35:11.814 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1dffff640 1 -- 172.21.15.89:0/1025243286 learned_addr learned my addr 172.21.15.89:0/1025243286 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1df7fe640 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b1c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.807+0000 7ff1e4c18640 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b1da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1dffff640 1 -- 172.21.15.89:0/1025243286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 msgr2=0x55e042b1da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1dffff640 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b1da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1dffff640 1 -- 172.21.15.89:0/1025243286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 msgr2=0x55e042b1c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1dffff640 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b1c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1dffff640 1 -- 172.21.15.89:0/1025243286 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e042b1e010 con 0x55e042b0d9a0 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1dffff640 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b1c4b0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00a4680 tx=0x7ff1d0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb4bca6c8c962f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/1025243286 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00a02f0 con 0x55e042b0d9a0 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/1025243286 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff1d00985b0 con 0x55e042b0d9a0 2024-09-15T23:35:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1e941c740 1 -- 172.21.15.89:0/1025243286 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d009bf40 con 0x55e042b0d9a0 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/1025243286 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff1d0023bb0 con 0x55e042b0d9a0 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1e941c740 10 client.19179 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1e941c740 1 -- 172.21.15.89:0/1025243286 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e042b8e450 con 0x55e042b0d9a0 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/1025243286 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1d009bf40 con 0x55e042b0d9a0 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/1025243286 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff1d009b390 con 0x55e042b0d9a0 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1e941c740 10 client.19179 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1e941c740 10 client.19179 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1e941c740 20 client.19179 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1e941c740 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e042b8ece0 0x55e042b910d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1e941c740 4 client.19179 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.808+0000 7ff1e941c740 1 -- 172.21.15.89:0/1025243286 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e042a78d70 con 0x55e042b8ece0 2024-09-15T23:35:11.816 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.809+0000 7ff1df7fe640 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e042b8ece0 0x55e042b910d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.809+0000 7ff1bf7fe640 10 client.19179 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.809+0000 7ff1df7fe640 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e042b8ece0 0x55e042b910d0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf21ec2b1bab5852 server_cookie=c140184a28f83a1b in_seq=0 out_seq=0 2024-09-15T23:35:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1bf7fe640 1 -- 172.21.15.89:0/1025243286 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e042a78d70 con 0x55e042b8ece0 2024-09-15T23:35:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1bf7fe640 10 client.19179 handle_command_reply: tid=0 2024-09-15T23:35:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 client.19179 shutdown 2024-09-15T23:35:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 -- 172.21.15.89:0/1025243286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 msgr2=0x55e042b1c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:11.825 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b1c4b0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00a4680 tx=0x7ff1d0000bc0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 -- 172.21.15.89:0/1025243286 shutdown_connections 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e042b8ece0 0x55e042b910d0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e042b13370 0x55e042b1da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1bf7fe640 0 client.19179 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e042b0e2e0 0x55e042b1c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 --2- 172.21.15.89:0/1025243286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e042b0d9a0 0x55e042b1c4b0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 -- 172.21.15.89:0/1025243286 >> 172.21.15.89:0/1025243286 conn(0x55e042b09010 msgr2=0x55e042b081c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 -- 172.21.15.89:0/1025243286 shutdown_connections 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 1 -- 172.21.15.89:0/1025243286 wait complete. 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.824+0000 7ff1e941c740 20 client.19179 trim_cache size 0 max 16384 2024-09-15T23:35:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:35:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:35:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:35:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:35:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 -- 172.21.15.89:0/3576720302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1c80a32f0 msgr2=0x7ff1c80a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:11.827 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1c80a32f0 0x7ff1c80a57b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8006010 tx=0x7ff1d804a000 comp rx=0 tx=0).stop 2024-09-15T23:35:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 -- 172.21.15.89:0/3576720302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e0100a60 msgr2=0x7ff1e0197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e0100a60 0x7ff1e0197db0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000d950 tx=0x7ff1d000de10 comp rx=0 tx=0).stop 2024-09-15T23:35:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 -- 172.21.15.89:0/3576720302 shutdown_connections 2024-09-15T23:35:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff1c80a32f0 0x7ff1c80a57b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1e0102610 0x7ff1e019f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1e0101c50 0x7ff1e01982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 --2- 172.21.15.89:0/3576720302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e0100a60 0x7ff1e0197db0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 -- 172.21.15.89:0/3576720302 >> 172.21.15.89:0/3576720302 conn(0x7ff1e00fc1e0 msgr2=0x7ff1e0104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 -- 172.21.15.89:0/3576720302 shutdown_connections 2024-09-15T23:35:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:11.825+0000 7ff1e667b640 1 -- 172.21.15.89:0/3576720302 wait complete. 2024-09-15T23:35:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:13 smithi120 ceph-mon[35250]: pgmap v685: 129 pgs: 129 active+clean; 1.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:35:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:13 smithi174 ceph-mon[35586]: pgmap v685: 129 pgs: 129 active+clean; 1.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:35:13.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:13 smithi089 ceph-mon[30933]: pgmap v685: 129 pgs: 129 active+clean; 1.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:35:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:15 smithi120 ceph-mon[35250]: pgmap v686: 129 pgs: 129 active+clean; 1.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-09-15T23:35:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:15 smithi174 ceph-mon[35586]: pgmap v686: 129 pgs: 129 active+clean; 1.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-09-15T23:35:15.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:15 smithi089 ceph-mon[30933]: pgmap v686: 129 pgs: 129 active+clean; 1.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-09-15T23:35:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:17 smithi120 ceph-mon[35250]: pgmap v687: 129 pgs: 129 active+clean; 1.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:35:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:17 smithi174 ceph-mon[35586]: pgmap v687: 129 pgs: 129 active+clean; 1.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:35:17.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:17 smithi089 ceph-mon[30933]: pgmap v687: 129 pgs: 129 active+clean; 1.6 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:35:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:19 smithi120 ceph-mon[35250]: pgmap v688: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:35:19.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:19 smithi174 ceph-mon[35586]: pgmap v688: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:35:19.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:19 smithi089 ceph-mon[30933]: pgmap v688: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:35:21.145 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:35:21.410 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:21.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:21 smithi120 ceph-mon[35250]: pgmap v689: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:35:21.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:21 smithi174 ceph-mon[35586]: pgmap v689: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:35:21.530 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:21 smithi089 ceph-mon[30933]: pgmap v689: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 15 op/s 2024-09-15T23:35:21.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.869+0000 7fefc6dc9640 1 -- 172.21.15.89:0/441536659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc0103660 msgr2=0x7fefc010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:21.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.869+0000 7fefc6dc9640 1 --2- 172.21.15.89:0/441536659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc0103660 0x7fefc010bb50 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fefac009900 tx=0x7fefac02a480 comp rx=0 tx=0).stop 2024-09-15T23:35:21.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.869+0000 7fefc6dc9640 1 -- 172.21.15.89:0/441536659 shutdown_connections 2024-09-15T23:35:21.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.869+0000 7fefc6dc9640 1 --2- 172.21.15.89:0/441536659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc010c170 0x7fefc010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.869+0000 7fefc6dc9640 1 --2- 172.21.15.89:0/441536659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc0103660 0x7fefc010bb50 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.869+0000 7fefc6dc9640 1 --2- 172.21.15.89:0/441536659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc0102d40 0x7fefc0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.869+0000 7fefc6dc9640 1 -- 172.21.15.89:0/441536659 >> 172.21.15.89:0/441536659 conn(0x7fefc00fc7a0 msgr2=0x7fefc00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.869+0000 7fefc6dc9640 1 -- 172.21.15.89:0/441536659 shutdown_connections 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.869+0000 7fefc6dc9640 1 -- 172.21.15.89:0/441536659 wait complete. 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc6dc9640 1 Processor -- start 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc6dc9640 1 -- start start 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc6dc9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc0102d40 0x7fefc01983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc6dc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc0103660 0x7fefc0198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc6dc9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc010c170 0x7fefc019ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc6dc9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fefc0111f70 con 0x7fefc0102d40 2024-09-15T23:35:21.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc6dc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fefc0111df0 con 0x7fefc0103660 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc6dc9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01120f0 con 0x7fefc010c170 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc4b65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc0102d40 0x7fefc01983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc4b65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc0102d40 0x7fefc01983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40186/0 (socket says 172.21.15.89:40186) 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc4b65640 1 -- 172.21.15.89:0/2567050989 learned_addr learned my addr 172.21.15.89:0/2567050989 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc5366640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc010c170 0x7fefc019ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefbffff640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc0103660 0x7fefc0198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc4b65640 1 -- 172.21.15.89:0/2567050989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc010c170 msgr2=0x7fefc019ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc4b65640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc010c170 0x7fefc019ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc4b65640 1 -- 172.21.15.89:0/2567050989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc0103660 msgr2=0x7fefc0198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc4b65640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc0103660 0x7fefc0198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc4b65640 1 -- 172.21.15.89:0/2567050989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc019d390 con 0x7fefc0102d40 2024-09-15T23:35:21.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefbffff640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc0103660 0x7fefc0198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefc4b65640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc0102d40 0x7fefc01983e0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fefb400d950 tx=0x7fefb400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.870+0000 7fefbdffb640 1 -- 172.21.15.89:0/2567050989 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4017070 con 0x7fefc0102d40 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.871+0000 7fefc6dc9640 1 -- 172.21.15.89:0/2567050989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc019d680 con 0x7fefc0102d40 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.871+0000 7fefbdffb640 1 -- 172.21.15.89:0/2567050989 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb4009c30 con 0x7fefc0102d40 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.871+0000 7fefbdffb640 1 -- 172.21.15.89:0/2567050989 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb400f6b0 con 0x7fefc0102d40 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.871+0000 7fefc6dc9640 1 -- 172.21.15.89:0/2567050989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc01a4f60 con 0x7fefc0102d40 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.873+0000 7fefbdffb640 1 -- 172.21.15.89:0/2567050989 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fefb40109d0 con 0x7fefc0102d40 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.873+0000 7fefc91a2740 1 Processor -- start 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.873+0000 7fefc91a2740 1 -- start start 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 0x5568b0b0d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefbdffb640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fefb00a3240 0x7fefb00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b0ddc0 0x5568b0b18140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b18680 0x5568b0b1aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc91a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5568b0a732e0 con 0x5568b0b18680 2024-09-15T23:35:21.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc91a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5568b0a1ace0 con 0x5568b0b0d4a0 2024-09-15T23:35:21.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc91a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5568b0a6f2a0 con 0x5568b0b0ddc0 2024-09-15T23:35:21.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc5366640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b18680 0x5568b0b1aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc4b65640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fefb00a3240 0x7fefb00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc5366640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b18680 0x5568b0b1aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40194/0 (socket says 172.21.15.89:40194) 2024-09-15T23:35:21.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc5366640 1 -- 172.21.15.89:0/2313547202 learned_addr learned my addr 172.21.15.89:0/2313547202 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:21.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefbffff640 1 --2- 172.21.15.89:0/2313547202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b0ddc0 0x5568b0b18140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefbdffb640 1 -- 172.21.15.89:0/2567050989 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fefb40cfa20 con 0x7fefc0102d40 2024-09-15T23:35:21.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc5366640 1 --2- 172.21.15.89:0/2313547202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 0x5568b0b0d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc5366640 1 -- 172.21.15.89:0/2313547202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b0ddc0 msgr2=0x5568b0b18140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:21.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc5366640 1 --2- 172.21.15.89:0/2313547202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b0ddc0 0x5568b0b18140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc5366640 1 -- 172.21.15.89:0/2313547202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 msgr2=0x5568b0b0d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:21.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc5366640 1 --2- 172.21.15.89:0/2313547202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 0x5568b0b0d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc5366640 1 -- 172.21.15.89:0/2313547202 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568b09ff440 con 0x5568b0b18680 2024-09-15T23:35:21.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc4b65640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fefb00a3240 0x7fefb00a5700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fefb40980b0 tx=0x7fefb4091130 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:21.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.874+0000 7fefc5366640 1 --2- 172.21.15.89:0/2313547202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b18680 0x5568b0b1aa70 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fefb8007f40 tx=0x7fefb8007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14783aab7100800 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefaaffd640 1 -- 172.21.15.89:0/2313547202 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb8018030 con 0x5568b0b18680 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefaaffd640 1 -- 172.21.15.89:0/2313547202 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb8012e20 con 0x5568b0b18680 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefaaffd640 1 -- 172.21.15.89:0/2313547202 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb80054e0 con 0x5568b0b18680 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefc91a2740 1 -- 172.21.15.89:0/2313547202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b18680 msgr2=0x5568b0b1aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefc91a2740 1 --2- 172.21.15.89:0/2313547202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b18680 0x5568b0b1aa70 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fefb8007f40 tx=0x7fefb8007800 comp rx=0 tx=0).stop 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefc91a2740 1 -- 172.21.15.89:0/2313547202 shutdown_connections 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefc91a2740 1 --2- 172.21.15.89:0/2313547202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b18680 0x5568b0b1aa70 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefc91a2740 1 --2- 172.21.15.89:0/2313547202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b0ddc0 0x5568b0b18140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefc91a2740 1 --2- 172.21.15.89:0/2313547202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 0x5568b0b0d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefc91a2740 1 -- 172.21.15.89:0/2313547202 >> 172.21.15.89:0/2313547202 conn(0x5568b0b09a30 msgr2=0x5568b0b09e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefc91a2740 1 -- 172.21.15.89:0/2313547202 shutdown_connections 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.875+0000 7fefc91a2740 1 -- 172.21.15.89:0/2313547202 wait complete. 2024-09-15T23:35:21.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefc91a2740 1 -- start start 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefc91a2740 1 Processor -- start 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b0d4a0 0x5568b0b18010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b152c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b18680 0x5568b0b15840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefc91a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5568b0b1b250 con 0x5568b0b0ddc0 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefc91a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5568b0a2e050 con 0x5568b0b18680 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefc91a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5568b0b1b3d0 con 0x5568b0b0d4a0 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefc91a2740 10 client.0 _ll_register_callbacks cb 0x5568b0a6e160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefbffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefbffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b152c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40206/0 (socket says 172.21.15.89:40206) 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.876+0000 7fefbffff640 1 -- 172.21.15.89:0/3632233163 learned_addr learned my addr 172.21.15.89:0/3632233163 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:21.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefc5366640 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b18680 0x5568b0b15840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefc5366640 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b0d4a0 0x5568b0b18010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefbffff640 1 -- 172.21.15.89:0/3632233163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b0d4a0 msgr2=0x5568b0b18010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefbffff640 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b0d4a0 0x5568b0b18010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefbffff640 1 -- 172.21.15.89:0/3632233163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b18680 msgr2=0x5568b0b15840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefbffff640 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b18680 0x5568b0b15840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefbffff640 1 -- 172.21.15.89:0/3632233163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568b07e43c0 con 0x5568b0b0ddc0 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefc5366640 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b18680 0x5568b0b15840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefbffff640 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b152c0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fefac009230 tx=0x7fefac0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93b1f564c24244a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fef9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fef9ffff640 1 -- 172.21.15.89:0/3632233163 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac040070 con 0x5568b0b0ddc0 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fef9ffff640 1 -- 172.21.15.89:0/3632233163 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fefac004250 con 0x5568b0b0ddc0 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefc91a2740 1 -- 172.21.15.89:0/3632233163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefb8012e20 con 0x5568b0b0ddc0 2024-09-15T23:35:21.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fef9ffff640 1 -- 172.21.15.89:0/3632233163 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fefac04d050 con 0x5568b0b0ddc0 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefc91a2740 10 client.19191 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefc91a2740 1 -- 172.21.15.89:0/3632233163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5568b0b26c40 con 0x5568b0b0ddc0 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fef9ffff640 1 -- 172.21.15.89:0/3632233163 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb8012e20 con 0x5568b0b0ddc0 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fef9ffff640 1 -- 172.21.15.89:0/3632233163 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fefac004990 con 0x5568b0b0ddc0 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.877+0000 7fefc91a2740 10 client.19191 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.878+0000 7fefc91a2740 10 client.19191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.878+0000 7fefc91a2740 20 client.19191 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.878+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5568b0b28100 0x5568b0b2a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.878+0000 7fefc91a2740 4 client.19191 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.878+0000 7fefc91a2740 1 -- 172.21.15.89:0/3632233163 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568b0a80850 con 0x5568b0b28100 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.878+0000 7fefc5366640 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5568b0b28100 0x5568b0b2a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.878+0000 7fef9ffff640 10 client.19191 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.878+0000 7fefc5366640 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5568b0b28100 0x5568b0b2a540 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2292da8924836bd server_cookie=72280fc87fa9e6cc in_seq=0 out_seq=0 2024-09-15T23:35:21.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fef9ffff640 1 -- 172.21.15.89:0/3632233163 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5568b0a80850 con 0x5568b0b28100 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fef9ffff640 10 client.19191 handle_command_reply: tid=0 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fefc91a2740 1 client.19191 shutdown 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fefc91a2740 1 -- 172.21.15.89:0/3632233163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 msgr2=0x5568b0b152c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b152c0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fefac009230 tx=0x7fefac0317e0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fefc91a2740 1 -- 172.21.15.89:0/3632233163 shutdown_connections 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5568b0b28100 0x5568b0b2a540 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b18680 0x5568b0b15840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b152c0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3632233163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b0d4a0 0x5568b0b18010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fefc91a2740 1 -- 172.21.15.89:0/3632233163 >> 172.21.15.89:0/3632233163 conn(0x5568b0b09a30 msgr2=0x5568b0b0f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.879+0000 7fef9ffff640 0 client.19191 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.880+0000 7fefc91a2740 1 -- 172.21.15.89:0/3632233163 shutdown_connections 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.880+0000 7fefc91a2740 1 -- 172.21.15.89:0/3632233163 wait complete. 2024-09-15T23:35:21.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.880+0000 7fefc91a2740 20 client.19191 trim_cache size 0 max 16384 2024-09-15T23:35:21.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.896+0000 7fefc91a2740 1 Processor -- start 2024-09-15T23:35:21.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.896+0000 7fefc91a2740 1 -- start start 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0d4a0 0x5568b0b244b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0ddc0 0x5568b0b2fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b18680 0x5568b0b30350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc91a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01aef30 con 0x5568b0b0d4a0 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc91a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01b0a70 con 0x5568b0b0ddc0 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc91a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01af600 con 0x5568b0b18680 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc5366640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0d4a0 0x5568b0b244b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc5366640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0d4a0 0x5568b0b244b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40208/0 (socket says 172.21.15.89:40208) 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc5366640 1 -- 172.21.15.89:0/749119133 learned_addr learned my addr 172.21.15.89:0/749119133 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc5366640 1 --2- 172.21.15.89:0/749119133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b18680 0x5568b0b30350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefbffff640 1 --2- 172.21.15.89:0/749119133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0ddc0 0x5568b0b2fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc5366640 1 -- 172.21.15.89:0/749119133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b18680 msgr2=0x5568b0b30350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:21.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc5366640 1 --2- 172.21.15.89:0/749119133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b18680 0x5568b0b30350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc5366640 1 -- 172.21.15.89:0/749119133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0ddc0 msgr2=0x5568b0b2fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:21.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc5366640 1 --2- 172.21.15.89:0/749119133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0ddc0 0x5568b0b2fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc5366640 1 -- 172.21.15.89:0/749119133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568b0b15bb0 con 0x5568b0b0d4a0 2024-09-15T23:35:21.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefc5366640 1 --2- 172.21.15.89:0/749119133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0d4a0 0x5568b0b244b0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fefb8007720 tx=0x7fefb80050c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec816be217a19a0f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:21.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefaa7fc640 1 -- 172.21.15.89:0/749119133 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb8018040 con 0x5568b0b0d4a0 2024-09-15T23:35:21.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefaa7fc640 1 -- 172.21.15.89:0/749119133 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb8010e80 con 0x5568b0b0d4a0 2024-09-15T23:35:21.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.897+0000 7fefaa7fc640 1 -- 172.21.15.89:0/749119133 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb8005390 con 0x5568b0b0d4a0 2024-09-15T23:35:21.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.898+0000 7fefc91a2740 1 -- 172.21.15.89:0/749119133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0d4a0 msgr2=0x5568b0b244b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:21.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.898+0000 7fefc91a2740 1 --2- 172.21.15.89:0/749119133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0d4a0 0x5568b0b244b0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fefb8007720 tx=0x7fefb80050c0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.898+0000 7fefc91a2740 1 -- 172.21.15.89:0/749119133 shutdown_connections 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.898+0000 7fefc91a2740 1 --2- 172.21.15.89:0/749119133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b18680 0x5568b0b30350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.898+0000 7fefc91a2740 1 --2- 172.21.15.89:0/749119133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0ddc0 0x5568b0b2fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.898+0000 7fefc91a2740 1 --2- 172.21.15.89:0/749119133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0d4a0 0x5568b0b244b0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.898+0000 7fefc91a2740 1 -- 172.21.15.89:0/749119133 >> 172.21.15.89:0/749119133 conn(0x5568b0b09a30 msgr2=0x5568b0b143b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.898+0000 7fefc91a2740 1 -- 172.21.15.89:0/749119133 shutdown_connections 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.898+0000 7fefc91a2740 1 -- 172.21.15.89:0/749119133 wait complete. 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.898+0000 7fefc91a2740 1 -- start start 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc91a2740 1 Processor -- start 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 0x5568b0b20d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b1c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc91a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b18680 0x5568b0b1c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc91a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01b03a0 con 0x5568b0b0ddc0 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc91a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01afcd0 con 0x5568b0b0d4a0 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc91a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fefc01acf10 con 0x5568b0b18680 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc91a2740 10 client.0 _ll_register_callbacks cb 0x7fefac009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefbffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b1c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefbffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b1c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40210/0 (socket says 172.21.15.89:40210) 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefbffff640 1 -- 172.21.15.89:0/3550514176 learned_addr learned my addr 172.21.15.89:0/3550514176 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc5366640 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b18680 0x5568b0b1c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc5366640 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 0x5568b0b20d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefbffff640 1 -- 172.21.15.89:0/3550514176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b18680 msgr2=0x5568b0b1c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefbffff640 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b18680 0x5568b0b1c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefbffff640 1 -- 172.21.15.89:0/3550514176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 msgr2=0x5568b0b20d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefbffff640 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 0x5568b0b20d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefbffff640 1 -- 172.21.15.89:0/3550514176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568b0b1ce50 con 0x5568b0b0ddc0 2024-09-15T23:35:21.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefc5366640 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 0x5568b0b20d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fefbffff640 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b1c0e0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fefac031550 tx=0x7fefac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=119f1f7f851dad15 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.899+0000 7fef9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fef9ffff640 1 -- 172.21.15.89:0/3550514176 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac055020 con 0x5568b0b0ddc0 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fefc91a2740 1 -- 172.21.15.89:0/3550514176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5568b0a7ad60 con 0x5568b0b0ddc0 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fef9ffff640 1 -- 172.21.15.89:0/3550514176 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fefac004450 con 0x5568b0b0ddc0 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fef9ffff640 1 -- 172.21.15.89:0/3550514176 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fefac060050 con 0x5568b0b0ddc0 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fefc91a2740 10 client.19197 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fefc91a2740 1 -- 172.21.15.89:0/3550514176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5568b0b92de0 con 0x5568b0b0ddc0 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fef9ffff640 1 -- 172.21.15.89:0/3550514176 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5568b0a7ad60 con 0x5568b0b0ddc0 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fef9ffff640 1 -- 172.21.15.89:0/3550514176 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fefac004f80 con 0x5568b0b0ddc0 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fefc91a2740 10 client.19197 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fefc91a2740 10 client.19197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fefc91a2740 20 client.19197 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:21.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5568b0b934f0 0x5568b0b958e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fefc91a2740 4 client.19197 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.900+0000 7fefc91a2740 1 -- 172.21.15.89:0/3550514176 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5568b09f7210 con 0x5568b0b934f0 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc5366640 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5568b0b934f0 0x5568b0b958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fef9ffff640 10 client.19197 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc5366640 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5568b0b934f0 0x5568b0b958e0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1364f290c34ea676 server_cookie=389f33f0d59a140b in_seq=0 out_seq=0 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fef9ffff640 1 -- 172.21.15.89:0/3550514176 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5568b09f7210 con 0x5568b0b934f0 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fef9ffff640 10 client.19197 handle_command_reply: tid=0 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc91a2740 1 client.19197 shutdown 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc91a2740 1 -- 172.21.15.89:0/3550514176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 msgr2=0x5568b0b1c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b1c0e0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fefac031550 tx=0x7fefac000bc0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc91a2740 1 -- 172.21.15.89:0/3550514176 shutdown_connections 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5568b0b934f0 0x5568b0b958e0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568b0b18680 0x5568b0b1c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5568b0b0ddc0 0x5568b0b1c0e0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc91a2740 1 --2- 172.21.15.89:0/3550514176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568b0b0d4a0 0x5568b0b20d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fefc91a2740 1 -- 172.21.15.89:0/3550514176 >> 172.21.15.89:0/3550514176 conn(0x5568b0b09a30 msgr2=0x5568b0b0eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:21.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.901+0000 7fef9ffff640 0 client.19197 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:21.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc91a2740 1 -- 172.21.15.89:0/3550514176 shutdown_connections 2024-09-15T23:35:21.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc91a2740 1 -- 172.21.15.89:0/3550514176 wait complete. 2024-09-15T23:35:21.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc91a2740 20 client.19197 trim_cache size 0 max 16384 2024-09-15T23:35:21.909 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:35:21.909 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:35:21.910 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:35:21.910 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:35:21.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 -- 172.21.15.89:0/2567050989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fefb00a3240 msgr2=0x7fefb00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fefb00a3240 0x7fefb00a5700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fefb40980b0 tx=0x7fefb4091130 comp rx=0 tx=0).stop 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 -- 172.21.15.89:0/2567050989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc0102d40 msgr2=0x7fefc01983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc0102d40 0x7fefc01983e0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fefb400d950 tx=0x7fefb400de10 comp rx=0 tx=0).stop 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 -- 172.21.15.89:0/2567050989 shutdown_connections 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fefb00a3240 0x7fefb00a5700 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fefc010c170 0x7fefc019ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc0103660 0x7fefc0198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 --2- 172.21.15.89:0/2567050989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc0102d40 0x7fefc01983e0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 -- 172.21.15.89:0/2567050989 >> 172.21.15.89:0/2567050989 conn(0x7fefc00fc7a0 msgr2=0x7fefc00fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 -- 172.21.15.89:0/2567050989 shutdown_connections 2024-09-15T23:35:21.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:21.902+0000 7fefc6dc9640 1 -- 172.21.15.89:0/2567050989 wait complete. 2024-09-15T23:35:22.291 INFO:tasks.quiescer.fs.[cephfs]:Total duration (43.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T23:35:22.291 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.85 2024-09-15T23:35:22.291 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:35:22.817 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:35:22.817 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:35:22.876 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:23.300 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:23 smithi089 ceph-mon[30933]: pgmap v690: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:35:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.379+0000 7fe461da9640 1 -- 172.21.15.89:0/407397378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe45c100a80 msgr2=0x7fe45c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.379+0000 7fe461da9640 1 --2- 172.21.15.89:0/407397378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe45c100a80 0x7fe45c100ee0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe448009930 tx=0x7fe44802a420 comp rx=0 tx=0).stop 2024-09-15T23:35:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.379+0000 7fe461da9640 1 -- 172.21.15.89:0/407397378 shutdown_connections 2024-09-15T23:35:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.379+0000 7fe461da9640 1 --2- 172.21.15.89:0/407397378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe45c101420 0x7fe45c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.379+0000 7fe461da9640 1 --2- 172.21.15.89:0/407397378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe45c100a80 0x7fe45c100ee0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:23.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.379+0000 7fe461da9640 1 --2- 172.21.15.89:0/407397378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe45c106a80 0x7fe45c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.379+0000 7fe461da9640 1 -- 172.21.15.89:0/407397378 >> 172.21.15.89:0/407397378 conn(0x7fe45c0fc7a0 msgr2=0x7fe45c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.379+0000 7fe461da9640 1 -- 172.21.15.89:0/407397378 shutdown_connections 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.379+0000 7fe461da9640 1 -- 172.21.15.89:0/407397378 wait complete. 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.380+0000 7fe461da9640 1 Processor -- start 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.380+0000 7fe461da9640 1 -- start start 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe461da9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe45c100a80 0x7fe45c1992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe461da9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe45c101420 0x7fe45c199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe461da9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe45c106a80 0x7fe45c1a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe461da9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c110e80 con 0x7fe45c106a80 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe461da9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c110d00 con 0x7fe45c101420 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe461da9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c111000 con 0x7fe45c100a80 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe45c106a80 0x7fe45c1a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:23.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe45c106a80 0x7fe45c1a4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40226/0 (socket says 172.21.15.89:40226) 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45bfff640 1 -- 172.21.15.89:0/568339810 learned_addr learned my addr 172.21.15.89:0/568339810 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45b7fe640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe45c100a80 0x7fe45c1992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45affd640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe45c101420 0x7fe45c199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45bfff640 1 -- 172.21.15.89:0/568339810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe45c100a80 msgr2=0x7fe45c1992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45bfff640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe45c100a80 0x7fe45c1992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45bfff640 1 -- 172.21.15.89:0/568339810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe45c101420 msgr2=0x7fe45c199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45bfff640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe45c101420 0x7fe45c199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45bfff640 1 -- 172.21.15.89:0/568339810 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45c1a4890 con 0x7fe45c106a80 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe45bfff640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe45c106a80 0x7fe45c1a4350 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fe45400eaa0 tx=0x7fe45400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe458ff9640 1 -- 172.21.15.89:0/568339810 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45400cdb0 con 0x7fe45c106a80 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe461da9640 1 -- 172.21.15.89:0/568339810 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe45c1a4a60 con 0x7fe45c106a80 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe458ff9640 1 -- 172.21.15.89:0/568339810 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe454005020 con 0x7fe45c106a80 2024-09-15T23:35:23.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe458ff9640 1 -- 172.21.15.89:0/568339810 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4540116b0 con 0x7fe45c106a80 2024-09-15T23:35:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.381+0000 7fe461da9640 1 -- 172.21.15.89:0/568339810 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe45c1a4f20 con 0x7fe45c106a80 2024-09-15T23:35:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.382+0000 7fe461da9640 1 -- 172.21.15.89:0/568339810 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe45c100ee0 con 0x7fe45c106a80 2024-09-15T23:35:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.385+0000 7fe458ff9640 1 -- 172.21.15.89:0/568339810 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe454005560 con 0x7fe45c106a80 2024-09-15T23:35:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.385+0000 7fe458ff9640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4400a3240 0x7fe4400a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.385+0000 7fe45b7fe640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4400a3240 0x7fe4400a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.386+0000 7fe458ff9640 1 -- 172.21.15.89:0/568339810 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe4540d0290 con 0x7fe45c106a80 2024-09-15T23:35:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.386+0000 7fe45b7fe640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4400a3240 0x7fe4400a5700 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe44c005fd0 tx=0x7fe44c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:23.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.388+0000 7fe458ff9640 1 -- 172.21.15.89:0/568339810 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe454016030 con 0x7fe45c106a80 2024-09-15T23:35:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:23 smithi120 ceph-mon[35250]: pgmap v690: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:35:23.482 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:23 smithi174 ceph-mon[35586]: pgmap v690: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:35:23.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.600+0000 7fe461da9640 1 -- 172.21.15.89:0/568339810 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe45c109010 con 0x7fe45c106a80 2024-09-15T23:35:23.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.601+0000 7fe458ff9640 1 -- 172.21.15.89:0/568339810 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fe45409c570 con 0x7fe45c106a80 2024-09-15T23:35:23.602 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:35:23.602 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:35:23.603 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:35:23.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 -- 172.21.15.89:0/568339810 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4400a3240 msgr2=0x7fe4400a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4400a3240 0x7fe4400a5700 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe44c005fd0 tx=0x7fe44c00b390 comp rx=0 tx=0).stop 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 -- 172.21.15.89:0/568339810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe45c106a80 msgr2=0x7fe45c1a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe45c106a80 0x7fe45c1a4350 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fe45400eaa0 tx=0x7fe45400ef60 comp rx=0 tx=0).stop 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 -- 172.21.15.89:0/568339810 shutdown_connections 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe4400a3240 0x7fe4400a5700 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe45c106a80 0x7fe45c1a4350 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe45c101420 0x7fe45c199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 --2- 172.21.15.89:0/568339810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe45c100a80 0x7fe45c1992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 -- 172.21.15.89:0/568339810 >> 172.21.15.89:0/568339810 conn(0x7fe45c0fc7a0 msgr2=0x7fe45c0fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 -- 172.21.15.89:0/568339810 shutdown_connections 2024-09-15T23:35:23.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:23.603+0000 7fe461da9640 1 -- 172.21.15.89:0/568339810 wait complete. 2024-09-15T23:35:24.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:24 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/568339810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:24.357 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0' 2024-09-15T23:35:24.357 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0 2024-09-15T23:35:24.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:24 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/568339810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:24.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:24 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/568339810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.574+0000 7f231ee85640 1 -- 172.21.15.89:0/2207522919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318102ef0 msgr2=0x7f231810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.574+0000 7f231ee85640 1 --2- 172.21.15.89:0/2207522919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318102ef0 0x7f231810da40 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f2310009ca0 tx=0x7f231002a950 comp rx=0 tx=0).stop 2024-09-15T23:35:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.574+0000 7f231ee85640 1 -- 172.21.15.89:0/2207522919 shutdown_connections 2024-09-15T23:35:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.574+0000 7f231ee85640 1 --2- 172.21.15.89:0/2207522919 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318102ef0 0x7f231810da40 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.574+0000 7f231ee85640 1 --2- 172.21.15.89:0/2207522919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2318102570 0x7f23181029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.574+0000 7f231ee85640 1 --2- 172.21.15.89:0/2207522919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2318100e20 0x7f2318101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.574+0000 7f231ee85640 1 -- 172.21.15.89:0/2207522919 >> 172.21.15.89:0/2207522919 conn(0x7f23180fc7d0 msgr2=0x7f23180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.574+0000 7f231ee85640 1 -- 172.21.15.89:0/2207522919 shutdown_connections 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f231ee85640 1 -- 172.21.15.89:0/2207522919 wait complete. 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f231ee85640 1 Processor -- start 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f231ee85640 1 -- start start 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f231ee85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2318100e20 0x7f2318199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f231ee85640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318102570 0x7f2318199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f231ee85640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2318102ef0 0x7f23181a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f231ee85640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2318111020 con 0x7f2318102570 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f231ee85640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2318110ea0 con 0x7f2318102ef0 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f231ee85640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f23181111a0 con 0x7f2318100e20 2024-09-15T23:35:24.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f230ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318102570 0x7f2318199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f230ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318102570 0x7f2318199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40252/0 (socket says 172.21.15.89:40252) 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.575+0000 7f230ffff640 1 -- 172.21.15.89:0/1333450940 learned_addr learned my addr 172.21.15.89:0/1333450940 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f231cc21640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2318100e20 0x7f2318199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f231d422640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2318102ef0 0x7f23181a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f230ffff640 1 -- 172.21.15.89:0/1333450940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2318100e20 msgr2=0x7f2318199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f230ffff640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2318100e20 0x7f2318199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f230ffff640 1 -- 172.21.15.89:0/1333450940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2318102ef0 msgr2=0x7f23181a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f230ffff640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2318102ef0 0x7f23181a44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f230ffff640 1 -- 172.21.15.89:0/1333450940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23181a4a10 con 0x7f2318102570 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f231d422640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2318102ef0 0x7f23181a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f230ffff640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318102570 0x7f2318199990 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f230800b4a0 tx=0x7f230800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f230dffb640 1 -- 172.21.15.89:0/1333450940 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308014070 con 0x7f2318102570 2024-09-15T23:35:24.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f230dffb640 1 -- 172.21.15.89:0/1333450940 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2308010af0 con 0x7f2318102570 2024-09-15T23:35:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f230dffb640 1 -- 172.21.15.89:0/1333450940 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308017d10 con 0x7f2318102570 2024-09-15T23:35:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f231ee85640 1 -- 172.21.15.89:0/1333450940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23181a4be0 con 0x7f2318102570 2024-09-15T23:35:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.576+0000 7f231ee85640 1 -- 172.21.15.89:0/1333450940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23181a50a0 con 0x7f2318102570 2024-09-15T23:35:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.578+0000 7f231ee85640 1 -- 172.21.15.89:0/1333450940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2318101280 con 0x7f2318102570 2024-09-15T23:35:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.579+0000 7f230dffb640 1 -- 172.21.15.89:0/1333450940 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2308017070 con 0x7f2318102570 2024-09-15T23:35:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.579+0000 7f230dffb640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23000a3240 0x7f23000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.579+0000 7f231cc21640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23000a3240 0x7f23000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.579+0000 7f230dffb640 1 -- 172.21.15.89:0/1333450940 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f23080cf120 con 0x7f2318102570 2024-09-15T23:35:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.580+0000 7f231cc21640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23000a3240 0x7f23000a5700 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2314009890 tx=0x7f2314008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:24.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.583+0000 7f230dffb640 1 -- 172.21.15.89:0/1333450940 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f230809ba50 con 0x7f2318102570 2024-09-15T23:35:24.662 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:24.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.792+0000 7f231ee85640 1 -- 172.21.15.89:0/1333450940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2318109090 con 0x7f2318102570 2024-09-15T23:35:24.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.793+0000 7f230dffb640 1 -- 172.21.15.89:0/1333450940 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f230809b350 con 0x7f2318102570 2024-09-15T23:35:24.794 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:35:24.794 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:35:24.795 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:35:24.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 -- 172.21.15.89:0/1333450940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23000a3240 msgr2=0x7f23000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23000a3240 0x7f23000a5700 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2314009890 tx=0x7f2314008040 comp rx=0 tx=0).stop 2024-09-15T23:35:24.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 -- 172.21.15.89:0/1333450940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318102570 msgr2=0x7f2318199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318102570 0x7f2318199990 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f230800b4a0 tx=0x7f230800b960 comp rx=0 tx=0).stop 2024-09-15T23:35:24.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 -- 172.21.15.89:0/1333450940 shutdown_connections 2024-09-15T23:35:24.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f23000a3240 0x7f23000a5700 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2318102ef0 0x7f23181a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2318102570 0x7f2318199990 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 --2- 172.21.15.89:0/1333450940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2318100e20 0x7f2318199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 -- 172.21.15.89:0/1333450940 >> 172.21.15.89:0/1333450940 conn(0x7f23180fc7d0 msgr2=0x7f23180febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:24.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 -- 172.21.15.89:0/1333450940 shutdown_connections 2024-09-15T23:35:24.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:24.795+0000 7f231ee85640 1 -- 172.21.15.89:0/1333450940 wait complete. 2024-09-15T23:35:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 -- 172.21.15.89:0/259097277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2288101b60 msgr2=0x7f2288101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 --2- 172.21.15.89:0/259097277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2288101b60 0x7f2288101f60 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f2278009900 tx=0x7f227802a410 comp rx=0 tx=0).stop 2024-09-15T23:35:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 -- 172.21.15.89:0/259097277 shutdown_connections 2024-09-15T23:35:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 --2- 172.21.15.89:0/259097277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2288103710 0x7f2288109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 --2- 172.21.15.89:0/259097277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2288102d50 0x7f22881031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 --2- 172.21.15.89:0/259097277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2288101b60 0x7f2288101f60 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 -- 172.21.15.89:0/259097277 >> 172.21.15.89:0/259097277 conn(0x7f22880fd2e0 msgr2=0x7f22880ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 -- 172.21.15.89:0/259097277 shutdown_connections 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 -- 172.21.15.89:0/259097277 wait complete. 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 Processor -- start 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.913+0000 7f228e923640 1 -- start start 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f228e923640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2288101b60 0x7f2288197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f228e923640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2288102d50 0x7f22881982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f228e923640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2288103710 0x7f228819f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f228e923640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f228810d7c0 con 0x7f2288101b60 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f228e923640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f228810d640 con 0x7f2288103710 2024-09-15T23:35:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f228e923640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f228810d940 con 0x7f2288102d50 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f2287fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2288101b60 0x7f2288197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f2287fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2288101b60 0x7f2288197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40276/0 (socket says 172.21.15.89:40276) 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f2287fff640 1 -- 172.21.15.89:0/3531690589 learned_addr learned my addr 172.21.15.89:0/3531690589 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f228cec0640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2288103710 0x7f228819f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f22877fe640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2288102d50 0x7f22881982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f2287fff640 1 -- 172.21.15.89:0/3531690589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2288102d50 msgr2=0x7f22881982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f2287fff640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2288102d50 0x7f22881982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f2287fff640 1 -- 172.21.15.89:0/3531690589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2288103710 msgr2=0x7f228819f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f2287fff640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2288103710 0x7f228819f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f2287fff640 1 -- 172.21.15.89:0/3531690589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228819f9b0 con 0x7f2288101b60 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f22877fe640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2288102d50 0x7f22881982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f2287fff640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2288101b60 0x7f2288197da0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f2278031610 tx=0x7f22780043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.914+0000 7f22857fa640 1 -- 172.21.15.89:0/3531690589 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227803f030 con 0x7f2288101b60 2024-09-15T23:35:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.915+0000 7f228e923640 1 -- 172.21.15.89:0/3531690589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228819fc40 con 0x7f2288101b60 2024-09-15T23:35:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.915+0000 7f22857fa640 1 -- 172.21.15.89:0/3531690589 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2278003ce0 con 0x7f2288101b60 2024-09-15T23:35:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.915+0000 7f22857fa640 1 -- 172.21.15.89:0/3531690589 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278035070 con 0x7f2288101b60 2024-09-15T23:35:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.915+0000 7f228e923640 1 -- 172.21.15.89:0/3531690589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22881a0180 con 0x7f2288101b60 2024-09-15T23:35:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.917+0000 7f22857fa640 1 -- 172.21.15.89:0/3531690589 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2278042030 con 0x7f2288101b60 2024-09-15T23:35:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.917+0000 7f22916bc740 1 Processor -- start 2024-09-15T23:35:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.917+0000 7f22916bc740 1 -- start start 2024-09-15T23:35:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f22916bc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f22916bc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 0x563a268b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f22916bc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b6780 0x563a268b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f22916bc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563a268102f0 con 0x563a268b0db0 2024-09-15T23:35:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f22916bc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563a267b8270 con 0x563a268b16f0 2024-09-15T23:35:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f22916bc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563a2680c2b0 con 0x563a268b6780 2024-09-15T23:35:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f228cec0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f22857fa640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f22700a32f0 0x7f22700a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f228cec0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40288/0 (socket says 172.21.15.89:40288) 2024-09-15T23:35:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f228cec0640 1 -- 172.21.15.89:0/3094215845 learned_addr learned my addr 172.21.15.89:0/3094215845 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f2287fff640 1 --2- 172.21.15.89:0/3094215845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 0x563a268b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f22877fe640 1 --2- 172.21.15.89:0/3094215845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b6780 0x563a268b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f22857fa640 1 -- 172.21.15.89:0/3531690589 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f22780f03e0 con 0x7f2288101b60 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f228cec0640 1 -- 172.21.15.89:0/3094215845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b6780 msgr2=0x563a268b8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f228cec0640 1 --2- 172.21.15.89:0/3094215845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b6780 0x563a268b8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f228cec0640 1 -- 172.21.15.89:0/3094215845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 msgr2=0x563a268b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f228cec0640 1 --2- 172.21.15.89:0/3094215845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 0x563a268b61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f228cec0640 1 -- 172.21.15.89:0/3094215845 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a267b7340 con 0x563a268b0db0 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f2287fff640 1 --2- 172.21.15.89:0/3094215845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 0x563a268b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.918+0000 7f228cec0640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f22700a32f0 0x7f22700a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.919+0000 7f228cec0640 1 --2- 172.21.15.89:0/3094215845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268b11b0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f2280007fd0 tx=0x7f228000e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2543f5c88dee47 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.919+0000 7f226e7fc640 1 -- 172.21.15.89:0/3094215845 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2280022070 con 0x563a268b0db0 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.919+0000 7f226e7fc640 1 -- 172.21.15.89:0/3094215845 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2280011c30 con 0x563a268b0db0 2024-09-15T23:35:24.926 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.919+0000 7f226e7fc640 1 -- 172.21.15.89:0/3094215845 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228001a400 con 0x563a268b0db0 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f22916bc740 1 -- 172.21.15.89:0/3094215845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 msgr2=0x563a268b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f228cec0640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f22700a32f0 0x7f22700a57b0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f228001f000 tx=0x7f2280020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f22916bc740 1 --2- 172.21.15.89:0/3094215845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268b11b0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f2280007fd0 tx=0x7f228000e7e0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f22916bc740 1 -- 172.21.15.89:0/3094215845 shutdown_connections 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f22916bc740 1 --2- 172.21.15.89:0/3094215845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b6780 0x563a268b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f22916bc740 1 --2- 172.21.15.89:0/3094215845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 0x563a268b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f22916bc740 1 --2- 172.21.15.89:0/3094215845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268b11b0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f22916bc740 1 -- 172.21.15.89:0/3094215845 >> 172.21.15.89:0/3094215845 conn(0x563a268a8010 msgr2=0x563a268a8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f22916bc740 1 -- 172.21.15.89:0/3094215845 shutdown_connections 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f22916bc740 1 -- 172.21.15.89:0/3094215845 wait complete. 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.920+0000 7f22916bc740 1 -- start start 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f22916bc740 1 Processor -- start 2024-09-15T23:35:24.927 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f22916bc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b0db0 0x563a268bafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f22916bc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b16f0 0x563a268bb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f22916bc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b6780 0x563a268c2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f22916bc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563a268b9750 con 0x563a268b6780 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f22916bc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563a26799290 con 0x563a268b0db0 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f22916bc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563a268b98d0 con 0x563a268b16f0 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f22916bc740 10 client.0 _ll_register_callbacks cb 0x563a267ca140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f228cec0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b6780 0x563a268c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f2287fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b0db0 0x563a268bafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f228cec0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b6780 0x563a268c2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40298/0 (socket says 172.21.15.89:40298) 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f228cec0640 1 -- 172.21.15.89:0/3081506174 learned_addr learned my addr 172.21.15.89:0/3081506174 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f22877fe640 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b16f0 0x563a268bb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.928 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.921+0000 7f228cec0640 1 -- 172.21.15.89:0/3081506174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b16f0 msgr2=0x563a268bb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f228cec0640 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b16f0 0x563a268bb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f228cec0640 1 -- 172.21.15.89:0/3081506174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b0db0 msgr2=0x563a268bafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f228cec0640 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b0db0 0x563a268bafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f228cec0640 1 -- 172.21.15.89:0/3081506174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a26523210 con 0x563a268b6780 2024-09-15T23:35:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f228cec0640 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b6780 0x563a268c2530 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f228000eec0 tx=0x7f2280007920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156b7b47312790cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f22677fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f22677fe640 1 -- 172.21.15.89:0/3081506174 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2280034060 con 0x563a268b6780 2024-09-15T23:35:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f22677fe640 1 -- 172.21.15.89:0/3081506174 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2280011a30 con 0x563a268b6780 2024-09-15T23:35:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f22916bc740 1 -- 172.21.15.89:0/3081506174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2280011c30 con 0x563a268b6780 2024-09-15T23:35:24.929 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f22677fe640 1 -- 172.21.15.89:0/3081506174 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f228003b050 con 0x563a268b6780 2024-09-15T23:35:24.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f22916bc740 10 client.19221 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:24.932 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f22916bc740 1 -- 172.21.15.89:0/3081506174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563a265ef620 con 0x563a268b6780 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.922+0000 7f22677fe640 1 -- 172.21.15.89:0/3081506174 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2280011c30 con 0x563a268b6780 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.923+0000 7f22677fe640 1 -- 172.21.15.89:0/3081506174 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2280019040 con 0x563a268b6780 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.923+0000 7f22916bc740 10 client.19221 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.923+0000 7f22916bc740 10 client.19221 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.923+0000 7f22916bc740 20 client.19221 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.923+0000 7f22916bc740 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a268c6a80 0x563a268c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.923+0000 7f22916bc740 4 client.19221 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.923+0000 7f22916bc740 1 -- 172.21.15.89:0/3081506174 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a2681da20 con 0x563a268c6a80 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.923+0000 7f2287fff640 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a268c6a80 0x563a268c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.923+0000 7f2287fff640 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a268c6a80 0x563a268c8f00 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d82645ad90bb5adb server_cookie=e676f0947b8146f6 in_seq=0 out_seq=0 2024-09-15T23:35:24.933 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.923+0000 7f22677fe640 10 client.19221 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:24.934 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22677fe640 1 -- 172.21.15.89:0/3081506174 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563a2681da20 con 0x563a268c6a80 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22677fe640 10 client.19221 handle_command_reply: tid=0 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 client.19221 shutdown 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 -- 172.21.15.89:0/3081506174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b6780 msgr2=0x563a268c2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b6780 0x563a268c2530 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f228000eec0 tx=0x7f2280007920 comp rx=0 tx=0).stop 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 -- 172.21.15.89:0/3081506174 shutdown_connections 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a268c6a80 0x563a268c8f00 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b6780 0x563a268c2530 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b16f0 0x563a268bb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 --2- 172.21.15.89:0/3081506174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b0db0 0x563a268bafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22677fe640 0 client.19221 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 -- 172.21.15.89:0/3081506174 >> 172.21.15.89:0/3081506174 conn(0x563a268a8010 msgr2=0x563a268a9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:24.935 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 -- 172.21.15.89:0/3081506174 shutdown_connections 2024-09-15T23:35:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.924+0000 7f22916bc740 1 -- 172.21.15.89:0/3081506174 wait complete. 2024-09-15T23:35:24.936 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.925+0000 7f22916bc740 20 client.19221 trim_cache size 0 max 16384 2024-09-15T23:35:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f22916bc740 1 Processor -- start 2024-09-15T23:35:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f22916bc740 1 -- start start 2024-09-15T23:35:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f22916bc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268a70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f22916bc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 0x563a268c24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.944 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f22916bc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b6780 0x563a268cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f22916bc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f22881a8580 con 0x563a268b0db0 2024-09-15T23:35:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f22916bc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f22881aa0c0 con 0x563a268b16f0 2024-09-15T23:35:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f22916bc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f22881a8c50 con 0x563a268b6780 2024-09-15T23:35:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f2287fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268a70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f2287fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268a70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40306/0 (socket says 172.21.15.89:40306) 2024-09-15T23:35:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f2287fff640 1 -- 172.21.15.89:0/2399113519 learned_addr learned my addr 172.21.15.89:0/2399113519 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f228cec0640 1 --2- 172.21.15.89:0/2399113519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b6780 0x563a268cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f22877fe640 1 --2- 172.21.15.89:0/2399113519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 0x563a268c24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.945 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f2287fff640 1 -- 172.21.15.89:0/2399113519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b6780 msgr2=0x563a268cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f2287fff640 1 --2- 172.21.15.89:0/2399113519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b6780 0x563a268cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f2287fff640 1 -- 172.21.15.89:0/2399113519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 msgr2=0x563a268c24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f2287fff640 1 --2- 172.21.15.89:0/2399113519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 0x563a268c24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f2287fff640 1 -- 172.21.15.89:0/2399113519 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a268b9a70 con 0x563a268b0db0 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f2287fff640 1 --2- 172.21.15.89:0/2399113519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268a70b0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f22780ede60 tx=0x7f22780ed9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc366ef2e29636c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f226dffb640 1 -- 172.21.15.89:0/2399113519 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2278049040 con 0x563a268b0db0 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.939+0000 7f226dffb640 1 -- 172.21.15.89:0/2399113519 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22780bcd10 con 0x563a268b0db0 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f226dffb640 1 -- 172.21.15.89:0/2399113519 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22780f0150 con 0x563a268b0db0 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f22916bc740 1 -- 172.21.15.89:0/2399113519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 msgr2=0x563a268a70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f22916bc740 1 --2- 172.21.15.89:0/2399113519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268a70b0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f22780ede60 tx=0x7f22780ed9c0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f22916bc740 1 -- 172.21.15.89:0/2399113519 shutdown_connections 2024-09-15T23:35:24.946 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f22916bc740 1 --2- 172.21.15.89:0/2399113519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b6780 0x563a268cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f22916bc740 1 --2- 172.21.15.89:0/2399113519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b16f0 0x563a268c24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f22916bc740 1 --2- 172.21.15.89:0/2399113519 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b0db0 0x563a268a70b0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f22916bc740 1 -- 172.21.15.89:0/2399113519 >> 172.21.15.89:0/2399113519 conn(0x563a268a8010 msgr2=0x563a268a69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f22916bc740 1 -- 172.21.15.89:0/2399113519 shutdown_connections 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f22916bc740 1 -- 172.21.15.89:0/2399113519 wait complete. 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.940+0000 7f22916bc740 1 -- start start 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22916bc740 1 Processor -- start 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22916bc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b0db0 0x563a268bfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22916bc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b16f0 0x563a268bff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22916bc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b6780 0x563a268c0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22916bc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f22881a99f0 con 0x563a268b16f0 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22916bc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f22881a9320 con 0x563a268b6780 2024-09-15T23:35:24.947 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22916bc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f22881a6560 con 0x563a268b0db0 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22916bc740 10 client.0 _ll_register_callbacks cb 0x7f2280020070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22877fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b16f0 0x563a268bff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22877fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b16f0 0x563a268bff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40310/0 (socket says 172.21.15.89:40310) 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22877fe640 1 -- 172.21.15.89:0/514620855 learned_addr learned my addr 172.21.15.89:0/514620855 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f228cec0640 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b6780 0x563a268c0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f2287fff640 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b0db0 0x563a268bfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22877fe640 1 -- 172.21.15.89:0/514620855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b0db0 msgr2=0x563a268bfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22877fe640 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b0db0 0x563a268bfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22877fe640 1 -- 172.21.15.89:0/514620855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b6780 msgr2=0x563a268c0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22877fe640 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b6780 0x563a268c0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.941+0000 7f22877fe640 1 -- 172.21.15.89:0/514620855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a268c1590 con 0x563a268b16f0 2024-09-15T23:35:24.948 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22877fe640 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b16f0 0x563a268bff70 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f227c005fd0 tx=0x7f227c013b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d74aad2f7241c54 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22677fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22677fe640 1 -- 172.21.15.89:0/514620855 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c011ad0 con 0x563a268b16f0 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22916bc740 1 -- 172.21.15.89:0/514620855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22780bcd10 con 0x563a268b16f0 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22677fe640 1 -- 172.21.15.89:0/514620855 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f227c011c70 con 0x563a268b16f0 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22677fe640 1 -- 172.21.15.89:0/514620855 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f227c026050 con 0x563a268b16f0 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22916bc740 10 client.19227 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22916bc740 1 -- 172.21.15.89:0/514620855 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563a268c2910 con 0x563a268b16f0 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22677fe640 1 -- 172.21.15.89:0/514620855 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22780bcd10 con 0x563a268b16f0 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22677fe640 1 -- 172.21.15.89:0/514620855 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f227c00f040 con 0x563a268b16f0 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22916bc740 10 client.19227 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22916bc740 10 client.19227 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:24.949 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22916bc740 20 client.19227 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22916bc740 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a26931fd0 0x563a269343f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22916bc740 4 client.19227 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:35:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.942+0000 7f22916bc740 1 -- 172.21.15.89:0/514620855 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563a26817d70 con 0x563a26931fd0 2024-09-15T23:35:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.943+0000 7f2287fff640 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a26931fd0 0x563a269343f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.943+0000 7f22677fe640 10 client.19227 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:24.950 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.943+0000 7f2287fff640 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a26931fd0 0x563a269343f0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb13e27d60a91193 server_cookie=30b3543fd7f3cfc1 in_seq=0 out_seq=0 2024-09-15T23:35:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22677fe640 1 -- 172.21.15.89:0/514620855 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563a26817d70 con 0x563a26931fd0 2024-09-15T23:35:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22677fe640 10 client.19227 handle_command_reply: tid=0 2024-09-15T23:35:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22916bc740 1 client.19227 shutdown 2024-09-15T23:35:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22916bc740 1 -- 172.21.15.89:0/514620855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b16f0 msgr2=0x563a268bff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22916bc740 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b16f0 0x563a268bff70 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f227c005fd0 tx=0x7f227c013b70 comp rx=0 tx=0).stop 2024-09-15T23:35:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22916bc740 1 -- 172.21.15.89:0/514620855 shutdown_connections 2024-09-15T23:35:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22916bc740 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563a26931fd0 0x563a269343f0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.960 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22916bc740 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563a268b6780 0x563a268c0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22916bc740 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563a268b16f0 0x563a268bff70 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22677fe640 0 client.19227 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22916bc740 1 --2- 172.21.15.89:0/514620855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563a268b0db0 0x563a268bfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.958+0000 7f22916bc740 1 -- 172.21.15.89:0/514620855 >> 172.21.15.89:0/514620855 conn(0x563a268a8010 msgr2=0x563a268a69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f22916bc740 1 -- 172.21.15.89:0/514620855 shutdown_connections 2024-09-15T23:35:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f22916bc740 1 -- 172.21.15.89:0/514620855 wait complete. 2024-09-15T23:35:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f22916bc740 20 client.19227 trim_cache size 0 max 16384 2024-09-15T23:35:24.961 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:35:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:35:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:35:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:35:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 -- 172.21.15.89:0/3531690589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f22700a32f0 msgr2=0x7f22700a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f22700a32f0 0x7f22700a57b0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f228001f000 tx=0x7f2280020000 comp rx=0 tx=0).stop 2024-09-15T23:35:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 -- 172.21.15.89:0/3531690589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2288101b60 msgr2=0x7f2288197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2288101b60 0x7f2288197da0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f2278031610 tx=0x7f22780043b0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 -- 172.21.15.89:0/3531690589 shutdown_connections 2024-09-15T23:35:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f22700a32f0 0x7f22700a57b0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2288103710 0x7f228819f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2288102d50 0x7f22881982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 --2- 172.21.15.89:0/3531690589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2288101b60 0x7f2288197da0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 -- 172.21.15.89:0/3531690589 >> 172.21.15.89:0/3531690589 conn(0x7f22880fd2e0 msgr2=0x7f22881060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 -- 172.21.15.89:0/3531690589 shutdown_connections 2024-09-15T23:35:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:24.959+0000 7f228e923640 1 -- 172.21.15.89:0/3531690589 wait complete. 2024-09-15T23:35:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:25 smithi120 ceph-mon[35250]: pgmap v691: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:35:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:25 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1333450940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:25 smithi174 ceph-mon[35586]: pgmap v691: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:35:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:25 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1333450940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:25.503 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:25 smithi089 ceph-mon[30933]: pgmap v691: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:35:25.504 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:25 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1333450940' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:25.531 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:35:25.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.778+0000 7f30222b7640 1 -- 172.21.15.89:0/4001418691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f301c101bc0 msgr2=0x7f301c10d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:25.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.778+0000 7f30222b7640 1 --2- 172.21.15.89:0/4001418691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f301c101bc0 0x7f301c10d910 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f3014009ca0 tx=0x7f301402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.778+0000 7f30222b7640 1 -- 172.21.15.89:0/4001418691 shutdown_connections 2024-09-15T23:35:25.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.778+0000 7f30222b7640 1 --2- 172.21.15.89:0/4001418691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f301c101bc0 0x7f301c10d910 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.778+0000 7f30222b7640 1 --2- 172.21.15.89:0/4001418691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f301c101240 0x7f301c101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.778+0000 7f30222b7640 1 --2- 172.21.15.89:0/4001418691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f301c105a90 0x7f301c105e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.778+0000 7f30222b7640 1 -- 172.21.15.89:0/4001418691 >> 172.21.15.89:0/4001418691 conn(0x7f301c077210 msgr2=0x7f301c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:25.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.778+0000 7f30222b7640 1 -- 172.21.15.89:0/4001418691 shutdown_connections 2024-09-15T23:35:25.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.778+0000 7f30222b7640 1 -- 172.21.15.89:0/4001418691 wait complete. 2024-09-15T23:35:25.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.779+0000 7f30222b7640 1 Processor -- start 2024-09-15T23:35:25.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.779+0000 7f30222b7640 1 -- start start 2024-09-15T23:35:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.779+0000 7f30222b7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f301c101240 0x7f301c1972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.779+0000 7f30222b7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f301c101bc0 0x7f301c1977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.779+0000 7f30222b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f301c105a90 0x7f301c19bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.779+0000 7f30222b7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f301c110f70 con 0x7f301c101bc0 2024-09-15T23:35:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.779+0000 7f30222b7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f301c110df0 con 0x7f301c101240 2024-09-15T23:35:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.779+0000 7f30222b7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1110f0 con 0x7f301c105a90 2024-09-15T23:35:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.779+0000 7f301b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f301c101bc0 0x7f301c1977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.779+0000 7f301b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f301c101bc0 0x7f301c1977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44504/0 (socket says 172.21.15.89:44504) 2024-09-15T23:35:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f301b7fe640 1 -- 172.21.15.89:0/2492365127 learned_addr learned my addr 172.21.15.89:0/2492365127 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:25.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f301bfff640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f301c101240 0x7f301c1972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f3020854640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f301c105a90 0x7f301c19bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f301b7fe640 1 -- 172.21.15.89:0/2492365127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f301c105a90 msgr2=0x7f301c19bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f301b7fe640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f301c105a90 0x7f301c19bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f301b7fe640 1 -- 172.21.15.89:0/2492365127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f301c101240 msgr2=0x7f301c1972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f301b7fe640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f301c101240 0x7f301c1972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f301b7fe640 1 -- 172.21.15.89:0/2492365127 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301c19c260 con 0x7f301c101bc0 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f301b7fe640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f301c101bc0 0x7f301c1977f0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f3010007b50 tx=0x7f301000e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f30197fa640 1 -- 172.21.15.89:0/2492365127 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010007c40 con 0x7f301c101bc0 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f30197fa640 1 -- 172.21.15.89:0/2492365127 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3010007de0 con 0x7f301c101bc0 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f30222b7640 1 -- 172.21.15.89:0/2492365127 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f301c19c550 con 0x7f301c101bc0 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f30197fa640 1 -- 172.21.15.89:0/2492365127 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301000ae60 con 0x7f301c101bc0 2024-09-15T23:35:25.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.780+0000 7f30222b7640 1 -- 172.21.15.89:0/2492365127 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f301c100130 con 0x7f301c101bc0 2024-09-15T23:35:25.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f3024690740 1 Processor -- start 2024-09-15T23:35:25.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f30197fa640 1 -- 172.21.15.89:0/2492365127 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f301000a440 con 0x7f301c101bc0 2024-09-15T23:35:25.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f3024690740 1 -- start start 2024-09-15T23:35:25.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f3024690740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7c850 0x556ed6c7cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f3024690740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c874f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f3024690740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c89e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f3024690740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556ed6be1160 con 0x556ed6c87a30 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f3024690740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ed6b88720 con 0x556ed6c7d170 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f3024690740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556ed6bdd120 con 0x556ed6c7c850 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f301b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c89e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f301bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c874f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f301bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c874f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:39774/0 (socket says 172.21.15.89:39774) 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f301b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c89e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44506/0 (socket says 172.21.15.89:44506) 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f301bfff640 1 -- 172.21.15.89:0/1154214911 learned_addr learned my addr 172.21.15.89:0/1154214911 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.783+0000 7f3020854640 1 --2- 172.21.15.89:0/1154214911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7c850 0x556ed6c7cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f301bfff640 1 -- 172.21.15.89:0/1154214911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7c850 msgr2=0x556ed6c7cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f30197fa640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30040a3240 0x7f30040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f301bfff640 1 --2- 172.21.15.89:0/1154214911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7c850 0x556ed6c7cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f301bfff640 1 -- 172.21.15.89:0/1154214911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 msgr2=0x556ed6c89e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f301bfff640 1 --2- 172.21.15.89:0/1154214911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c89e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f301bfff640 1 -- 172.21.15.89:0/1154214911 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed6952ba0 con 0x556ed6c7d170 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f3020854640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30040a3240 0x7f30040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f30197fa640 1 -- 172.21.15.89:0/2492365127 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f30100d11b0 con 0x7f301c101bc0 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f301bfff640 1 --2- 172.21.15.89:0/1154214911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c874f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f300c00b760 tx=0x7f300c00bc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1fee3c2c23d1114 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f30027fc640 1 -- 172.21.15.89:0/1154214911 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c003dd0 con 0x556ed6c7d170 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f30027fc640 1 -- 172.21.15.89:0/1154214911 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f300c015070 con 0x556ed6c7d170 2024-09-15T23:35:25.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.784+0000 7f30027fc640 1 -- 172.21.15.89:0/1154214911 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c011440 con 0x556ed6c7d170 2024-09-15T23:35:25.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3024690740 1 -- 172.21.15.89:0/1154214911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 msgr2=0x556ed6c874f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:25.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3020854640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30040a3240 0x7f30040a5700 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3014005e30 tx=0x7f30140095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:25.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3024690740 1 --2- 172.21.15.89:0/1154214911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c874f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f300c00b760 tx=0x7f300c00bc20 comp rx=0 tx=0).stop 2024-09-15T23:35:25.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3024690740 1 -- 172.21.15.89:0/1154214911 shutdown_connections 2024-09-15T23:35:25.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3024690740 1 --2- 172.21.15.89:0/1154214911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c89e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3024690740 1 --2- 172.21.15.89:0/1154214911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c874f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3024690740 1 --2- 172.21.15.89:0/1154214911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7c850 0x556ed6c7cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3024690740 1 -- 172.21.15.89:0/1154214911 >> 172.21.15.89:0/1154214911 conn(0x556ed6c78de0 msgr2=0x556ed6c791c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3024690740 1 -- 172.21.15.89:0/1154214911 shutdown_connections 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3024690740 1 -- 172.21.15.89:0/1154214911 wait complete. 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.785+0000 7f3024690740 1 -- start start 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f3024690740 1 Processor -- start 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f3024690740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7c850 0x556ed6c848c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f3024690740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c84e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f3024690740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c85340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f3024690740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556ed6c8a600 con 0x556ed6c87a30 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f3024690740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ed6b69900 con 0x556ed6c7d170 2024-09-15T23:35:25.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f3024690740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556ed6c8a780 con 0x556ed6c7c850 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f3024690740 10 client.0 _ll_register_callbacks cb 0x556ed6b67780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f301b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c84e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f301b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c84e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:39782/0 (socket says 172.21.15.89:39782) 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f301b7fe640 1 -- 172.21.15.89:0/2999622470 learned_addr learned my addr 172.21.15.89:0/2999622470 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f3020854640 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c85340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f301bfff640 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7c850 0x556ed6c848c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f301b7fe640 1 -- 172.21.15.89:0/2999622470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7c850 msgr2=0x556ed6c848c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f301b7fe640 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7c850 0x556ed6c848c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f301b7fe640 1 -- 172.21.15.89:0/2999622470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 msgr2=0x556ed6c85340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f301b7fe640 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c85340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f301b7fe640 1 -- 172.21.15.89:0/2999622470 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed6ba9c50 con 0x556ed6c7d170 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f301b7fe640 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c84e00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3010099870 tx=0x7f30100d47e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=503f03ba7120dc52 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:25.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.786+0000 7f2ffb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2999622470 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301009d1e0 con 0x556ed6c7d170 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f3024690740 1 -- 172.21.15.89:0/2999622470 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f300c015070 con 0x556ed6c7d170 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2999622470 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f301000ead0 con 0x556ed6c7d170 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2999622470 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f301009cd70 con 0x556ed6c7d170 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f3024690740 10 client.27437 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f3024690740 1 -- 172.21.15.89:0/2999622470 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556ed6c962f0 con 0x556ed6c7d170 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2999622470 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f300c015070 con 0x556ed6c7d170 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2999622470 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3010014070 con 0x556ed6c7d170 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f3024690740 10 client.27437 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f3024690740 10 client.27437 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f3024690740 20 client.27437 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f3024690740 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ed6c97700 0x556ed6c99af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f3024690740 4 client.27437 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.787+0000 7f3024690740 1 -- 172.21.15.89:0/2999622470 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ed6bee890 con 0x556ed6c97700 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.788+0000 7f301bfff640 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ed6c97700 0x556ed6c99af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.788+0000 7f2ffb7fe640 10 client.27437 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.788+0000 7f301bfff640 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ed6c97700 0x556ed6c99af0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d386ab930ede0e0 server_cookie=2aae6f3b995848c7 in_seq=0 out_seq=0 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2999622470 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556ed6bee890 con 0x556ed6c97700 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f2ffb7fe640 10 client.27437 handle_command_reply: tid=0 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 client.27437 shutdown 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 -- 172.21.15.89:0/2999622470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 msgr2=0x556ed6c84e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c84e00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3010099870 tx=0x7f30100d47e0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 -- 172.21.15.89:0/2999622470 shutdown_connections 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ed6c97700 0x556ed6c99af0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c85340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7d170 0x556ed6c84e00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f2ffb7fe640 0 client.27437 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:25.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 --2- 172.21.15.89:0/2999622470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7c850 0x556ed6c848c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 -- 172.21.15.89:0/2999622470 >> 172.21.15.89:0/2999622470 conn(0x556ed6c78de0 msgr2=0x556ed6c7eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:25.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 -- 172.21.15.89:0/2999622470 shutdown_connections 2024-09-15T23:35:25.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 1 -- 172.21.15.89:0/2999622470 wait complete. 2024-09-15T23:35:25.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.789+0000 7f3024690740 20 client.27437 trim_cache size 0 max 16384 2024-09-15T23:35:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f3024690740 1 Processor -- start 2024-09-15T23:35:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f3024690740 1 -- start start 2024-09-15T23:35:25.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f3024690740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c7c850 0x556ed6c73d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f3024690740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7d170 0x556ed6c77db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f3024690740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c87a30 0x556ed6c9f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f3024690740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1acbc0 con 0x556ed6c7c850 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f3024690740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1ae700 con 0x556ed6c87a30 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f3024690740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1ad290 con 0x556ed6c7d170 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f301bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c7c850 0x556ed6c73d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f301bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c7c850 0x556ed6c73d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44532/0 (socket says 172.21.15.89:44532) 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.806+0000 7f301bfff640 1 -- 172.21.15.89:0/2902878568 learned_addr learned my addr 172.21.15.89:0/2902878568 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f301b7fe640 1 --2- 172.21.15.89:0/2902878568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7d170 0x556ed6c77db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f3020854640 1 --2- 172.21.15.89:0/2902878568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c87a30 0x556ed6c9f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f301bfff640 1 -- 172.21.15.89:0/2902878568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7d170 msgr2=0x556ed6c77db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f301bfff640 1 --2- 172.21.15.89:0/2902878568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7d170 0x556ed6c77db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f301bfff640 1 -- 172.21.15.89:0/2902878568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c87a30 msgr2=0x556ed6c9f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:25.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f301bfff640 1 --2- 172.21.15.89:0/2902878568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c87a30 0x556ed6c9f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f301bfff640 1 -- 172.21.15.89:0/2902878568 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ed6c83870 con 0x556ed6c7c850 2024-09-15T23:35:25.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f301bfff640 1 --2- 172.21.15.89:0/2902878568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c7c850 0x556ed6c73d30 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f300c00b760 tx=0x7f300c0049e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c4ecfa083088b2d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:25.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f3001ffb640 1 -- 172.21.15.89:0/2902878568 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c003bd0 con 0x556ed6c7c850 2024-09-15T23:35:25.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f3001ffb640 1 -- 172.21.15.89:0/2902878568 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f300c015070 con 0x556ed6c7c850 2024-09-15T23:35:25.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.807+0000 7f3001ffb640 1 -- 172.21.15.89:0/2902878568 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c011d50 con 0x556ed6c7c850 2024-09-15T23:35:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.808+0000 7f3024690740 1 -- 172.21.15.89:0/2902878568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c7c850 msgr2=0x556ed6c73d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.808+0000 7f3024690740 1 --2- 172.21.15.89:0/2902878568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c7c850 0x556ed6c73d30 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f300c00b760 tx=0x7f300c0049e0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.808+0000 7f3024690740 1 -- 172.21.15.89:0/2902878568 shutdown_connections 2024-09-15T23:35:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.808+0000 7f3024690740 1 --2- 172.21.15.89:0/2902878568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c87a30 0x556ed6c9f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.808+0000 7f3024690740 1 --2- 172.21.15.89:0/2902878568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7d170 0x556ed6c77db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.808+0000 7f3024690740 1 --2- 172.21.15.89:0/2902878568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c7c850 0x556ed6c73d30 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.808+0000 7f3024690740 1 -- 172.21.15.89:0/2902878568 >> 172.21.15.89:0/2902878568 conn(0x556ed6c78de0 msgr2=0x556ed6c776a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.808+0000 7f3024690740 1 -- 172.21.15.89:0/2902878568 shutdown_connections 2024-09-15T23:35:25.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.808+0000 7f3024690740 1 -- 172.21.15.89:0/2902878568 wait complete. 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.808+0000 7f3024690740 1 -- start start 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3024690740 1 Processor -- start 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3024690740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7c850 0x556ed6c8fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3024690740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7d170 0x556ed6c8c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3024690740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c8c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3024690740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1ae030 con 0x556ed6c87a30 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3024690740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1ad960 con 0x556ed6c7c850 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3024690740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f301c1aaba0 con 0x556ed6c7d170 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3024690740 10 client.0 _ll_register_callbacks cb 0x7f30100d48e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3020854640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c8c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3020854640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c8c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44540/0 (socket says 172.21.15.89:44540) 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f3020854640 1 -- 172.21.15.89:0/2433238006 learned_addr learned my addr 172.21.15.89:0/2433238006 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f301b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7d170 0x556ed6c8c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.809+0000 7f301bfff640 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7c850 0x556ed6c8fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f3020854640 1 -- 172.21.15.89:0/2433238006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7d170 msgr2=0x556ed6c8c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f3020854640 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7d170 0x556ed6c8c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f3020854640 1 -- 172.21.15.89:0/2433238006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7c850 msgr2=0x556ed6c8fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f3020854640 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7c850 0x556ed6c8fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f3020854640 1 -- 172.21.15.89:0/2433238006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ed6c99e60 con 0x556ed6c87a30 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f3020854640 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c8c690 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f3014031950 tx=0x7f301402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2f862f8b69c7ae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f2ffb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2433238006 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014059060 con 0x556ed6c87a30 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2433238006 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3014003d80 con 0x556ed6c87a30 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f3024690740 1 -- 172.21.15.89:0/2433238006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f300c015070 con 0x556ed6c87a30 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2433238006 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3014060050 con 0x556ed6c87a30 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f3024690740 10 client.19245 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.810+0000 7f3024690740 1 -- 172.21.15.89:0/2433238006 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556ed6d072b0 con 0x556ed6c87a30 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.811+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2433238006 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f300c015070 con 0x556ed6c87a30 2024-09-15T23:35:25.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.811+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2433238006 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3014030920 con 0x556ed6c87a30 2024-09-15T23:35:25.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.811+0000 7f3024690740 10 client.19245 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:25.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.811+0000 7f3024690740 10 client.19245 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:35:25.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.811+0000 7f3024690740 20 client.19245 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:25.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.811+0000 7f3024690740 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ed6d07aa0 0x556ed6d09ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:25.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.811+0000 7f3024690740 4 client.19245 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}] 2024-09-15T23:35:25.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.811+0000 7f3024690740 1 -- 172.21.15.89:0/2433238006 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}) -- 0x556ed6c74410 con 0x556ed6d07aa0 2024-09-15T23:35:25.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.811+0000 7f301bfff640 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ed6d07aa0 0x556ed6d09ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:25.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.812+0000 7f301bfff640 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ed6d07aa0 0x556ed6d09ec0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7267c6d4acbff485 server_cookie=916a7b4405ddce5c in_seq=0 out_seq=0 2024-09-15T23:35:25.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.812+0000 7f2ffb7fe640 10 client.19245 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:25.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.823+0000 7f2ffb7fe640 1 -- 172.21.15.89:0/2433238006 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x556ed6c74410 con 0x556ed6d07aa0 2024-09-15T23:35:25.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.823+0000 7f2ffb7fe640 10 client.19245 handle_command_reply: tid=0 2024-09-15T23:35:25.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.823+0000 7f3024690740 1 client.19245 shutdown 2024-09-15T23:35:25.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.823+0000 7f3024690740 1 -- 172.21.15.89:0/2433238006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 msgr2=0x556ed6c8c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:25.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f3024690740 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c8c690 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f3014031950 tx=0x7f301402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f3024690740 1 -- 172.21.15.89:0/2433238006 shutdown_connections 2024-09-15T23:35:25.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f3024690740 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556ed6d07aa0 0x556ed6d09ec0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f3024690740 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ed6c87a30 0x556ed6c8c690 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f3024690740 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ed6c7d170 0x556ed6c8c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f3024690740 1 --2- 172.21.15.89:0/2433238006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ed6c7c850 0x556ed6c8fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f3024690740 1 -- 172.21.15.89:0/2433238006 >> 172.21.15.89:0/2433238006 conn(0x556ed6c78de0 msgr2=0x556ed6c850a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:25.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f2ffb7fe640 0 client.19245 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:25.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f3024690740 1 -- 172.21.15.89:0/2433238006 shutdown_connections 2024-09-15T23:35:25.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f3024690740 1 -- 172.21.15.89:0/2433238006 wait complete. 2024-09-15T23:35:25.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.824+0000 7f3024690740 20 client.19245 trim_cache size 0 max 16384 2024-09-15T23:35:25.826 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:35:25.826 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:35:25.826 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:35:25.826 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 38, 2024-09-15T23:35:25.826 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: "ac2b42c4": { 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 38, 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 141.8, 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:35:25.827 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:35:25.828 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:35:25.828 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:35:25.828 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:35:25.828 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:35:25.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:25.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:25.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:25.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:25.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:25.828 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:35:25.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 -- 172.21.15.89:0/2492365127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30040a3240 msgr2=0x7f30040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:25.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30040a3240 0x7f30040a5700 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3014005e30 tx=0x7f30140095c0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 -- 172.21.15.89:0/2492365127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f301c101bc0 msgr2=0x7f301c1977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:25.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f301c101bc0 0x7f301c1977f0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f3010007b50 tx=0x7f301000e420 comp rx=0 tx=0).stop 2024-09-15T23:35:25.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 -- 172.21.15.89:0/2492365127 shutdown_connections 2024-09-15T23:35:25.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f30040a3240 0x7f30040a5700 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f301c105a90 0x7f301c19bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f301c101bc0 0x7f301c1977f0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 --2- 172.21.15.89:0/2492365127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f301c101240 0x7f301c1972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:25.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 -- 172.21.15.89:0/2492365127 >> 172.21.15.89:0/2492365127 conn(0x7f301c077210 msgr2=0x7f301c1037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:25.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 -- 172.21.15.89:0/2492365127 shutdown_connections 2024-09-15T23:35:25.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:25.825+0000 7f30222b7640 1 -- 172.21.15.89:0/2492365127 wait complete. 2024-09-15T23:35:26.095 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:26.561 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'ac2b42c4', quiesce took 4.3 seconds. Will release after: 17.55 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.037+0000 7f16c1de8640 1 -- 172.21.15.89:0/3028379032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc100eb0 msgr2=0x7f16bc1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.037+0000 7f16c1de8640 1 --2- 172.21.15.89:0/3028379032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc100eb0 0x7f16bc1067a0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009900 tx=0x7f16b002a410 comp rx=0 tx=0).stop 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.037+0000 7f16c1de8640 1 -- 172.21.15.89:0/3028379032 shutdown_connections 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.037+0000 7f16c1de8640 1 --2- 172.21.15.89:0/3028379032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16bc106ed0 0x7f16bc10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.037+0000 7f16c1de8640 1 --2- 172.21.15.89:0/3028379032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc100eb0 0x7f16bc1067a0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.037+0000 7f16c1de8640 1 --2- 172.21.15.89:0/3028379032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16bc100500 0x7f16bc1008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.037+0000 7f16c1de8640 1 -- 172.21.15.89:0/3028379032 >> 172.21.15.89:0/3028379032 conn(0x7f16bc0fc5c0 msgr2=0x7f16bc0fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.037+0000 7f16c1de8640 1 -- 172.21.15.89:0/3028379032 shutdown_connections 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.037+0000 7f16c1de8640 1 -- 172.21.15.89:0/3028379032 wait complete. 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16c1de8640 1 Processor -- start 2024-09-15T23:35:27.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16c1de8640 1 -- start start 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16c1de8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16bc100500 0x7f16bc198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16c1de8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16bc100eb0 0x7f16bc198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16c1de8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc106ed0 0x7f16bc19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16c1de8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc111fa0 con 0x7f16bc106ed0 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16c1de8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc111e20 con 0x7f16bc100500 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16c1de8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc112120 con 0x7f16bc100eb0 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16bbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc106ed0 0x7f16bc19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16bbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc106ed0 0x7f16bc19cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44558/0 (socket says 172.21.15.89:44558) 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16bbfff640 1 -- 172.21.15.89:0/4061716848 learned_addr learned my addr 172.21.15.89:0/4061716848 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16bc100500 0x7f16bc198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16baffd640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16bc100eb0 0x7f16bc198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16bbfff640 1 -- 172.21.15.89:0/4061716848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16bc100eb0 msgr2=0x7f16bc198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16bbfff640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16bc100eb0 0x7f16bc198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16bbfff640 1 -- 172.21.15.89:0/4061716848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16bc100500 msgr2=0x7f16bc198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16bbfff640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16bc100500 0x7f16bc198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16bbfff640 1 -- 172.21.15.89:0/4061716848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16bc19d3f0 con 0x7f16bc106ed0 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16baffd640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16bc100eb0 0x7f16bc198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.038+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16bc100500 0x7f16bc198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.039+0000 7f16bbfff640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc106ed0 0x7f16bc19cd10 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f16b400cce0 tx=0x7f16b400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.039+0000 7f16b8ff9640 1 -- 172.21.15.89:0/4061716848 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b400f450 con 0x7f16bc106ed0 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.039+0000 7f16b8ff9640 1 -- 172.21.15.89:0/4061716848 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b400fad0 con 0x7f16bc106ed0 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.039+0000 7f16c1de8640 1 -- 172.21.15.89:0/4061716848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16bc19d6e0 con 0x7f16bc106ed0 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.039+0000 7f16b8ff9640 1 -- 172.21.15.89:0/4061716848 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b4018c10 con 0x7f16bc106ed0 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.039+0000 7f16c1de8640 1 -- 172.21.15.89:0/4061716848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16bc1a4fc0 con 0x7f16bc106ed0 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.041+0000 7f16b8ff9640 1 -- 172.21.15.89:0/4061716848 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f16b400f5f0 con 0x7f16bc106ed0 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.041+0000 7f16c41c1740 1 Processor -- start 2024-09-15T23:35:27.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.041+0000 7f16c41c1740 1 -- start start 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe9315b40 0x562fe9314090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe93145d0 0x562fe9314a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe931f370 0x562fe9321760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16c41c1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562fe927a2f0 con 0x562fe931f370 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16c41c1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562fe9221cf0 con 0x562fe9315b40 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16c41c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562fe92762b0 con 0x562fe93145d0 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16baffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe931f370 0x562fe9321760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16baffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe931f370 0x562fe9321760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44570/0 (socket says 172.21.15.89:44570) 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16baffd640 1 -- 172.21.15.89:0/1019711063 learned_addr learned my addr 172.21.15.89:0/1019711063 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/1019711063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe93145d0 0x562fe9314a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16baffd640 1 --2- 172.21.15.89:0/1019711063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe9315b40 0x562fe9314090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16b8ff9640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16a40a3240 0x7f16a40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16bbfff640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16a40a3240 0x7f16a40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16baffd640 1 -- 172.21.15.89:0/1019711063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe93145d0 msgr2=0x562fe9314a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16baffd640 1 --2- 172.21.15.89:0/1019711063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe93145d0 0x562fe9314a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16baffd640 1 -- 172.21.15.89:0/1019711063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe9315b40 msgr2=0x562fe9314090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16baffd640 1 --2- 172.21.15.89:0/1019711063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe9315b40 0x562fe9314090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16baffd640 1 -- 172.21.15.89:0/1019711063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fe92063a0 con 0x562fe931f370 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16b8ff9640 1 -- 172.21.15.89:0/4061716848 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f16b40d0490 con 0x7f16bc106ed0 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.042+0000 7f16baffd640 1 --2- 172.21.15.89:0/1019711063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe931f370 0x562fe9321760 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f16b0005ec0 tx=0x7f16b0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53bd730a26d6052 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16a1ffb640 1 -- 172.21.15.89:0/1019711063 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0035040 con 0x562fe931f370 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16a1ffb640 1 -- 172.21.15.89:0/1019711063 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b0005050 con 0x562fe931f370 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16a1ffb640 1 -- 172.21.15.89:0/1019711063 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0004a40 con 0x562fe931f370 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16c41c1740 1 -- 172.21.15.89:0/1019711063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe931f370 msgr2=0x562fe9321760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16bbfff640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16a40a3240 0x7f16a40a5700 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f16b40cdbf0 tx=0x7f16b40cd9e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:27.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16c41c1740 1 --2- 172.21.15.89:0/1019711063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe931f370 0x562fe9321760 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f16b0005ec0 tx=0x7f16b0000bc0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16c41c1740 1 -- 172.21.15.89:0/1019711063 shutdown_connections 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16c41c1740 1 --2- 172.21.15.89:0/1019711063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe931f370 0x562fe9321760 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16c41c1740 1 --2- 172.21.15.89:0/1019711063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe93145d0 0x562fe9314a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16c41c1740 1 --2- 172.21.15.89:0/1019711063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe9315b40 0x562fe9314090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16c41c1740 1 -- 172.21.15.89:0/1019711063 >> 172.21.15.89:0/1019711063 conn(0x562fe92dd310 msgr2=0x562fe9310c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16c41c1740 1 -- 172.21.15.89:0/1019711063 shutdown_connections 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.043+0000 7f16c41c1740 1 -- 172.21.15.89:0/1019711063 wait complete. 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16c41c1740 1 -- start start 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16c41c1740 1 Processor -- start 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe93145d0 0x562fe931ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe9315b40 0x562fe931c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe931f370 0x562fe931c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16c41c1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562fe9322010 con 0x562fe93145d0 2024-09-15T23:35:27.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16c41c1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562fe9235060 con 0x562fe931f370 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16c41c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562fe9322190 con 0x562fe9315b40 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16c41c1740 10 client.0 _ll_register_callbacks cb 0x562fe8f888c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16baffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe93145d0 0x562fe931ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16baffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe93145d0 0x562fe931ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44576/0 (socket says 172.21.15.89:44576) 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.044+0000 7f16baffd640 1 -- 172.21.15.89:0/2995384794 learned_addr learned my addr 172.21.15.89:0/2995384794 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe9315b40 0x562fe931c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f16baffd640 1 -- 172.21.15.89:0/2995384794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe9315b40 msgr2=0x562fe931c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f16baffd640 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe9315b40 0x562fe931c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f16baffd640 1 -- 172.21.15.89:0/2995384794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe931f370 msgr2=0x562fe931c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f16baffd640 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe931f370 0x562fe931c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f16baffd640 1 -- 172.21.15.89:0/2995384794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fe8feb3c0 con 0x562fe93145d0 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f16baffd640 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe93145d0 0x562fe931ed70 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f16b0001eb0 tx=0x7f16b0005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db1099e0d1afe8e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:27.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f169affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f169affd640 1 -- 172.21.15.89:0/2995384794 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0035040 con 0x562fe93145d0 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f169affd640 1 -- 172.21.15.89:0/2995384794 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f16b0004c00 con 0x562fe93145d0 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f16c41c1740 1 -- 172.21.15.89:0/2995384794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16b0005050 con 0x562fe93145d0 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f169affd640 1 -- 172.21.15.89:0/2995384794 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f16b002aa90 con 0x562fe93145d0 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f16c41c1740 10 client.19257 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f16c41c1740 1 -- 172.21.15.89:0/2995384794 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562fe932dab0 con 0x562fe93145d0 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.045+0000 7f169affd640 1 -- 172.21.15.89:0/2995384794 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b0005050 con 0x562fe93145d0 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.046+0000 7f169affd640 1 -- 172.21.15.89:0/2995384794 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f16b003a040 con 0x562fe93145d0 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.046+0000 7f16c41c1740 10 client.19257 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.046+0000 7f16c41c1740 10 client.19257 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.046+0000 7f16c41c1740 20 client.19257 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.046+0000 7f16c41c1740 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x562fe932ee00 0x562fe93311f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.046+0000 7f16c41c1740 4 client.19257 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:27.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.046+0000 7f16c41c1740 1 -- 172.21.15.89:0/2995384794 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fe9287860 con 0x562fe932ee00 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.046+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x562fe932ee00 0x562fe93311f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.046+0000 7f169affd640 10 client.19257 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.046+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x562fe932ee00 0x562fe93311f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c7e02bcffd3b77a3 server_cookie=a0d0c6acd2066582 in_seq=0 out_seq=0 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f169affd640 1 -- 172.21.15.89:0/2995384794 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562fe9287860 con 0x562fe932ee00 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f169affd640 10 client.19257 handle_command_reply: tid=0 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 client.19257 shutdown 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 -- 172.21.15.89:0/2995384794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe93145d0 msgr2=0x562fe931ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe93145d0 0x562fe931ed70 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f16b0001eb0 tx=0x7f16b0005570 comp rx=0 tx=0).stop 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 -- 172.21.15.89:0/2995384794 shutdown_connections 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x562fe932ee00 0x562fe93311f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe931f370 0x562fe931c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe9315b40 0x562fe931c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f169affd640 0 client.19257 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:35:27.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 --2- 172.21.15.89:0/2995384794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe93145d0 0x562fe931ed70 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 -- 172.21.15.89:0/2995384794 >> 172.21.15.89:0/2995384794 conn(0x562fe92dd310 msgr2=0x562fe9316960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:27.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 -- 172.21.15.89:0/2995384794 shutdown_connections 2024-09-15T23:35:27.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 1 -- 172.21.15.89:0/2995384794 wait complete. 2024-09-15T23:35:27.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.047+0000 7f16c41c1740 20 client.19257 trim_cache size 0 max 16384 2024-09-15T23:35:27.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16c41c1740 1 Processor -- start 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16c41c1740 1 -- start start 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 0x562fe9317120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9317660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe931f370 0x562fe9334210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16c41c1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1aee10 con 0x562fe9315b40 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16c41c1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1b0950 con 0x562fe93145d0 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16c41c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1af4e0 con 0x562fe931f370 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16bb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9317660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16bb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9317660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44588/0 (socket says 172.21.15.89:44588) 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16bb7fe640 1 -- 172.21.15.89:0/2221342910 learned_addr learned my addr 172.21.15.89:0/2221342910 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16baffd640 1 --2- 172.21.15.89:0/2221342910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 0x562fe9317120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.063 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16baffd640 1 --2- 172.21.15.89:0/2221342910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe931f370 0x562fe9334210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16bb7fe640 1 -- 172.21.15.89:0/2221342910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe931f370 msgr2=0x562fe9334210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:27.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/2221342910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe931f370 0x562fe9334210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16bb7fe640 1 -- 172.21.15.89:0/2221342910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 msgr2=0x562fe9317120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:27.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/2221342910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 0x562fe9317120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16bb7fe640 1 -- 172.21.15.89:0/2221342910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fe931a590 con 0x562fe9315b40 2024-09-15T23:35:27.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16baffd640 1 --2- 172.21.15.89:0/2221342910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe931f370 0x562fe9334210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:27.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.061+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/2221342910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9317660 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x562fe931d660 tx=0x7f16ac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b375e1e932d2aa9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:27.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16a17fa640 1 -- 172.21.15.89:0/2221342910 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac020070 con 0x562fe9315b40 2024-09-15T23:35:27.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16a17fa640 1 -- 172.21.15.89:0/2221342910 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16ac01c250 con 0x562fe9315b40 2024-09-15T23:35:27.064 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16a17fa640 1 -- 172.21.15.89:0/2221342910 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac014610 con 0x562fe9315b40 2024-09-15T23:35:27.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16c41c1740 1 -- 172.21.15.89:0/2221342910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 msgr2=0x562fe9317660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:27.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16c41c1740 1 --2- 172.21.15.89:0/2221342910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9317660 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x562fe931d660 tx=0x7f16ac008040 comp rx=0 tx=0).stop 2024-09-15T23:35:27.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16c41c1740 1 -- 172.21.15.89:0/2221342910 shutdown_connections 2024-09-15T23:35:27.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16c41c1740 1 --2- 172.21.15.89:0/2221342910 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe931f370 0x562fe9334210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16c41c1740 1 --2- 172.21.15.89:0/2221342910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9317660 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16c41c1740 1 --2- 172.21.15.89:0/2221342910 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 0x562fe9317120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16c41c1740 1 -- 172.21.15.89:0/2221342910 >> 172.21.15.89:0/2221342910 conn(0x562fe92dd310 msgr2=0x562fe930fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:27.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16c41c1740 1 -- 172.21.15.89:0/2221342910 shutdown_connections 2024-09-15T23:35:27.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.062+0000 7f16c41c1740 1 -- 172.21.15.89:0/2221342910 wait complete. 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16c41c1740 1 -- start start 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16c41c1740 1 Processor -- start 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 0x562fe93347e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9331b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16c41c1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe931f370 0x562fe93320c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16c41c1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1b0280 con 0x562fe9315b40 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16c41c1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1afbb0 con 0x562fe93145d0 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16c41c1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f16bc1acdf0 con 0x562fe931f370 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16c41c1740 10 client.0 _ll_register_callbacks cb 0x7f16b0032de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16bb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9331b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16bb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9331b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44596/0 (socket says 172.21.15.89:44596) 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.063+0000 7f16bb7fe640 1 -- 172.21.15.89:0/4186276136 learned_addr learned my addr 172.21.15.89:0/4186276136 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16baffd640 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 0x562fe93347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16bb7fe640 1 -- 172.21.15.89:0/4186276136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe931f370 msgr2=0x562fe93320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe931f370 0x562fe93320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16bb7fe640 1 -- 172.21.15.89:0/4186276136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 msgr2=0x562fe93347e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 0x562fe93347e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16bb7fe640 1 -- 172.21.15.89:0/4186276136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fe9332730 con 0x562fe9315b40 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16baffd640 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 0x562fe93347e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16bb7fe640 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9331b80 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0077b0 tx=0x7f16ac015dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=522ae6e5b48f89b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f169affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f169affd640 1 -- 172.21.15.89:0/4186276136 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac029030 con 0x562fe9315b40 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f169affd640 1 -- 172.21.15.89:0/4186276136 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f16ac01c6d0 con 0x562fe9315b40 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16c41c1740 1 -- 172.21.15.89:0/4186276136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16ac01c250 con 0x562fe9315b40 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f169affd640 1 -- 172.21.15.89:0/4186276136 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f16ac036050 con 0x562fe9315b40 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16c41c1740 10 client.19263 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:27.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.064+0000 7f16c41c1740 1 -- 172.21.15.89:0/4186276136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562fe9399550 con 0x562fe9315b40 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f169affd640 1 -- 172.21.15.89:0/4186276136 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16ac01c250 con 0x562fe9315b40 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f169affd640 1 -- 172.21.15.89:0/4186276136 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f16ac02d3c0 con 0x562fe9315b40 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f16c41c1740 10 client.19263 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f16c41c1740 10 client.19263 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f16c41c1740 20 client.19263 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f16c41c1740 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x562fe9399c70 0x562fe939c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f16c41c1740 4 client.19263 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f16c41c1740 1 -- 172.21.15.89:0/4186276136 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562fe9281d70 con 0x562fe9399c70 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f16baffd640 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x562fe9399c70 0x562fe939c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f169affd640 10 client.19263 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.065+0000 7f16baffd640 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x562fe9399c70 0x562fe939c090 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb2fec83c1e5096b server_cookie=8825da42b9a375eb in_seq=0 out_seq=0 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f169affd640 1 -- 172.21.15.89:0/4186276136 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562fe9281d70 con 0x562fe9399c70 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f169affd640 10 client.19263 handle_command_reply: tid=0 2024-09-15T23:35:27.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 client.19263 shutdown 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 -- 172.21.15.89:0/4186276136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 msgr2=0x562fe9331b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9331b80 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0077b0 tx=0x7f16ac015dc0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 -- 172.21.15.89:0/4186276136 shutdown_connections 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x562fe9399c70 0x562fe939c090 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562fe931f370 0x562fe93320c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f169affd640 0 client.19263 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562fe9315b40 0x562fe9331b80 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 --2- 172.21.15.89:0/4186276136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe93145d0 0x562fe93347e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 -- 172.21.15.89:0/4186276136 >> 172.21.15.89:0/4186276136 conn(0x562fe92dd310 msgr2=0x562fe930f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 -- 172.21.15.89:0/4186276136 shutdown_connections 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 1 -- 172.21.15.89:0/4186276136 wait complete. 2024-09-15T23:35:27.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.066+0000 7f16c41c1740 20 client.19263 trim_cache size 0 max 16384 2024-09-15T23:35:27.073 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:35:27.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 -- 172.21.15.89:0/4061716848 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16a40a3240 msgr2=0x7f16a40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:27.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16a40a3240 0x7f16a40a5700 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f16b40cdbf0 tx=0x7f16b40cd9e0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 -- 172.21.15.89:0/4061716848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc106ed0 msgr2=0x7f16bc19cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:27.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc106ed0 0x7f16bc19cd10 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f16b400cce0 tx=0x7f16b400a620 comp rx=0 tx=0).stop 2024-09-15T23:35:27.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 -- 172.21.15.89:0/4061716848 shutdown_connections 2024-09-15T23:35:27.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f16a40a3240 0x7f16a40a5700 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc106ed0 0x7f16bc19cd10 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16bc100eb0 0x7f16bc198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 --2- 172.21.15.89:0/4061716848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16bc100500 0x7f16bc198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:27.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 -- 172.21.15.89:0/4061716848 >> 172.21.15.89:0/4061716848 conn(0x7f16bc0fc5c0 msgr2=0x7f16bc1056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:27.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 -- 172.21.15.89:0/4061716848 shutdown_connections 2024-09-15T23:35:27.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:27.067+0000 7f16c1de8640 1 -- 172.21.15.89:0/4061716848 wait complete. 2024-09-15T23:35:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:27 smithi120 ceph-mon[35250]: pgmap v692: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:35:27.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:27 smithi174 ceph-mon[35586]: pgmap v692: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:35:27.537 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:27 smithi089 ceph-mon[30933]: pgmap v692: 129 pgs: 129 active+clean; 1.7 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:35:27.879 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:35:28.129 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:28.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.616+0000 7f5817126640 1 -- 172.21.15.89:0/4266941943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5810100a80 msgr2=0x7f5810100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.616+0000 7f5817126640 1 --2- 172.21.15.89:0/4266941943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5810100a80 0x7f5810100ee0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009930 tx=0x7f57fc02a420 comp rx=0 tx=0).stop 2024-09-15T23:35:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.616+0000 7f5817126640 1 -- 172.21.15.89:0/4266941943 shutdown_connections 2024-09-15T23:35:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.616+0000 7f5817126640 1 --2- 172.21.15.89:0/4266941943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5810101420 0x7f581010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.616+0000 7f5817126640 1 --2- 172.21.15.89:0/4266941943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5810100a80 0x7f5810100ee0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.616+0000 7f5817126640 1 --2- 172.21.15.89:0/4266941943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5810106a80 0x7f5810106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.616+0000 7f5817126640 1 -- 172.21.15.89:0/4266941943 >> 172.21.15.89:0/4266941943 conn(0x7f58100fc7a0 msgr2=0x7f58100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.616+0000 7f5817126640 1 -- 172.21.15.89:0/4266941943 shutdown_connections 2024-09-15T23:35:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.616+0000 7f5817126640 1 -- 172.21.15.89:0/4266941943 wait complete. 2024-09-15T23:35:28.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f5817126640 1 Processor -- start 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f5817126640 1 -- start start 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f5817126640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5810100a80 0x7f5810198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f5817126640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5810101420 0x7f5810198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f5817126640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5810106a80 0x7f581019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f5817126640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5810110e80 con 0x7f5810106a80 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f5817126640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5810110d00 con 0x7f5810100a80 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f5817126640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5810111000 con 0x7f5810101420 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f58156c3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5810106a80 0x7f581019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f58156c3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5810106a80 0x7f581019cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44614/0 (socket says 172.21.15.89:44614) 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.617+0000 7f58156c3640 1 -- 172.21.15.89:0/2231567680 learned_addr learned my addr 172.21.15.89:0/2231567680 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f580ffff640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5810101420 0x7f5810198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f5814ec2640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5810100a80 0x7f5810198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f58156c3640 1 -- 172.21.15.89:0/2231567680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5810101420 msgr2=0x7f5810198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f58156c3640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5810101420 0x7f5810198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f58156c3640 1 -- 172.21.15.89:0/2231567680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5810100a80 msgr2=0x7f5810198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f58156c3640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5810100a80 0x7f5810198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f58156c3640 1 -- 172.21.15.89:0/2231567680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581019d400 con 0x7f5810106a80 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f580ffff640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5810101420 0x7f5810198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f58156c3640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5810106a80 0x7f581019cd20 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f580800eaa0 tx=0x7f580800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f580dffb640 1 -- 172.21.15.89:0/2231567680 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580800cdb0 con 0x7f5810106a80 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f580dffb640 1 -- 172.21.15.89:0/2231567680 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5808005020 con 0x7f5810106a80 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f5817126640 1 -- 172.21.15.89:0/2231567680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f581019d6f0 con 0x7f5810106a80 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f580dffb640 1 -- 172.21.15.89:0/2231567680 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58080116b0 con 0x7f5810106a80 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.618+0000 7f5817126640 1 -- 172.21.15.89:0/2231567680 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58101a4fd0 con 0x7f5810106a80 2024-09-15T23:35:28.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.620+0000 7f580dffb640 1 -- 172.21.15.89:0/2231567680 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5808005560 con 0x7f5810106a80 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f58194ff740 1 Processor -- start 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f58194ff740 1 -- start start 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f58194ff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4f9f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f58194ff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4f9fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f58194ff740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fb0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f58194ff740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cfc4f052f0 con 0x55cfc4fae780 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f58194ff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cfc4eaccf0 con 0x55cfc4fa0b40 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f58194ff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cfc4f012b0 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580dffb640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58000a3240 0x7f58000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fb0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fb0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44620/0 (socket says 172.21.15.89:44620) 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580ffff640 1 -- 172.21.15.89:0/1327697238 learned_addr learned my addr 172.21.15.89:0/1327697238 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:28.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f5814ec2640 1 --2- 172.21.15.89:0/1327697238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4f9fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f58156c3640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58000a3240 0x7f58000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580ffff640 1 --2- 172.21.15.89:0/1327697238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4f9f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580dffb640 1 -- 172.21.15.89:0/2231567680 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f58080d0030 con 0x7f5810106a80 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580ffff640 1 -- 172.21.15.89:0/1327697238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 msgr2=0x55cfc4f9fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580ffff640 1 --2- 172.21.15.89:0/1327697238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4f9fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580ffff640 1 -- 172.21.15.89:0/1327697238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 msgr2=0x55cfc4f9f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580ffff640 1 --2- 172.21.15.89:0/1327697238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4f9f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f580ffff640 1 -- 172.21.15.89:0/1327697238 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfc4e913a0 con 0x55cfc4fae780 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.621+0000 7f5814ec2640 1 --2- 172.21.15.89:0/1327697238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4f9fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f580ffff640 1 --2- 172.21.15.89:0/1327697238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fb0b70 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f57fc032f60 tx=0x7f57fc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a462f05a25a16c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f58156c3640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58000a3240 0x7f58000a5700 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f580809c1f0 tx=0x7f5808066bf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f57faffd640 1 -- 172.21.15.89:0/1327697238 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc045070 con 0x55cfc4fae780 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f57faffd640 1 -- 172.21.15.89:0/1327697238 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57fc031a00 con 0x55cfc4fae780 2024-09-15T23:35:28.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f57faffd640 1 -- 172.21.15.89:0/1327697238 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc035070 con 0x55cfc4fae780 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f58194ff740 1 -- 172.21.15.89:0/1327697238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 msgr2=0x55cfc4fb0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f58194ff740 1 --2- 172.21.15.89:0/1327697238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fb0b70 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f57fc032f60 tx=0x7f57fc000bc0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f58194ff740 1 -- 172.21.15.89:0/1327697238 shutdown_connections 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f58194ff740 1 --2- 172.21.15.89:0/1327697238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fb0b70 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f58194ff740 1 --2- 172.21.15.89:0/1327697238 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4f9fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f58194ff740 1 --2- 172.21.15.89:0/1327697238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4f9f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.622+0000 7f58194ff740 1 -- 172.21.15.89:0/1327697238 >> 172.21.15.89:0/1327697238 conn(0x55cfc4f68310 msgr2=0x55cfc4f9bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.623+0000 7f58194ff740 1 -- 172.21.15.89:0/1327697238 shutdown_connections 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.623+0000 7f58194ff740 1 -- 172.21.15.89:0/1327697238 wait complete. 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.623+0000 7f58194ff740 1 -- start start 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.623+0000 7f58194ff740 1 Processor -- start 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.623+0000 7f58194ff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.623+0000 7f58194ff740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.623+0000 7f58194ff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fae780 0x55cfc4fabbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.623+0000 7f58194ff740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cfc4fb1420 con 0x55cfc4fa0b40 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.623+0000 7f58194ff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cfc4ec0060 con 0x55cfc4fae780 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.623+0000 7f58194ff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cfc4fb15a0 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f58194ff740 10 client.0 _ll_register_callbacks cb 0x55cfc4c138c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f5814ec2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f5814ec2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fab4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44636/0 (socket says 172.21.15.89:44636) 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f5814ec2640 1 -- 172.21.15.89:0/3610845445 learned_addr learned my addr 172.21.15.89:0/3610845445 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f580ffff640 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f580ffff640 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fae780 0x55cfc4fabbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f5814ec2640 1 -- 172.21.15.89:0/3610845445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 msgr2=0x55cfc4fae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f5814ec2640 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fae1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f5814ec2640 1 -- 172.21.15.89:0/3610845445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fae780 msgr2=0x55cfc4fabbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f5814ec2640 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fae780 0x55cfc4fabbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f5814ec2640 1 -- 172.21.15.89:0/3610845445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfc4c763c0 con 0x55cfc4fa0b40 2024-09-15T23:35:28.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f580ffff640 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f5814ec2640 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fab4a0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f580400f6f0 tx=0x7f580400fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e47dd846723334a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f57effff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f57effff640 1 -- 172.21.15.89:0/3610845445 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5804016070 con 0x55cfc4fa0b40 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f57effff640 1 -- 172.21.15.89:0/3610845445 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f580400bb00 con 0x55cfc4fa0b40 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f58194ff740 1 -- 172.21.15.89:0/3610845445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57fc031a00 con 0x55cfc4fa0b40 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f57effff640 1 -- 172.21.15.89:0/3610845445 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5804025050 con 0x55cfc4fa0b40 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f58194ff740 10 client.19275 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.624+0000 7f58194ff740 1 -- 172.21.15.89:0/3610845445 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cfc4fbcfb0 con 0x55cfc4fa0b40 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.625+0000 7f57effff640 1 -- 172.21.15.89:0/3610845445 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57fc031a00 con 0x55cfc4fa0b40 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.625+0000 7f57effff640 1 -- 172.21.15.89:0/3610845445 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f580401a400 con 0x55cfc4fa0b40 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.625+0000 7f58194ff740 10 client.19275 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.625+0000 7f58194ff740 10 client.19275 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.625+0000 7f58194ff740 20 client.19275 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:28.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.625+0000 7f58194ff740 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cfc4fbe510 0x55cfc4fc0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.625+0000 7f58194ff740 4 client.19275 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.625+0000 7f58194ff740 1 -- 172.21.15.89:0/3610845445 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfc4f12860 con 0x55cfc4fbe510 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.625+0000 7f580ffff640 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cfc4fbe510 0x55cfc4fc0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f580ffff640 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cfc4fbe510 0x55cfc4fc0930 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eaad072d60b41e82 server_cookie=d65307c2f0fc73b3 in_seq=0 out_seq=0 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f57effff640 10 client.19275 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f57effff640 1 -- 172.21.15.89:0/3610845445 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cfc4f12860 con 0x55cfc4fbe510 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f57effff640 10 client.19275 handle_command_reply: tid=0 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f58194ff740 1 client.19275 shutdown 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f58194ff740 1 -- 172.21.15.89:0/3610845445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fa0b40 msgr2=0x55cfc4fab4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f58194ff740 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fab4a0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f580400f6f0 tx=0x7f580400fbb0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f58194ff740 1 -- 172.21.15.89:0/3610845445 shutdown_connections 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f58194ff740 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cfc4fbe510 0x55cfc4fc0930 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f58194ff740 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fae780 0x55cfc4fabbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f57effff640 0 client.19275 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:35:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f58194ff740 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fab4a0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f58194ff740 1 --2- 172.21.15.89:0/3610845445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.626+0000 7f58194ff740 1 -- 172.21.15.89:0/3610845445 >> 172.21.15.89:0/3610845445 conn(0x55cfc4f68310 msgr2=0x55cfc4fa1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.627+0000 7f58194ff740 1 -- 172.21.15.89:0/3610845445 shutdown_connections 2024-09-15T23:35:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.627+0000 7f58194ff740 1 -- 172.21.15.89:0/3610845445 wait complete. 2024-09-15T23:35:28.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.627+0000 7f58194ff740 20 client.19275 trim_cache size 0 max 16384 2024-09-15T23:35:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.640+0000 7f58194ff740 1 Processor -- start 2024-09-15T23:35:28.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.640+0000 7f58194ff740 1 -- start start 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.640+0000 7f58194ff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fab470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.640+0000 7f58194ff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fba6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.640+0000 7f58194ff740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fc61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.640+0000 7f58194ff740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f58101acc70 con 0x55cfc4fae780 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.640+0000 7f58194ff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58101ae7b0 con 0x55cfc4fa0b40 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.640+0000 7f58194ff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58101ad340 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f580ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fc61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f580ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fc61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44644/0 (socket says 172.21.15.89:44644) 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f580ffff640 1 -- 172.21.15.89:0/4146123107 learned_addr learned my addr 172.21.15.89:0/4146123107 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f580ffff640 1 --2- 172.21.15.89:0/4146123107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fab470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f5814ec2640 1 --2- 172.21.15.89:0/4146123107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fba6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f580ffff640 1 -- 172.21.15.89:0/4146123107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 msgr2=0x55cfc4fab470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:28.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f580ffff640 1 --2- 172.21.15.89:0/4146123107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fab470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f580ffff640 1 -- 172.21.15.89:0/4146123107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 msgr2=0x55cfc4fba6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f580ffff640 1 --2- 172.21.15.89:0/4146123107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fba6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f580ffff640 1 -- 172.21.15.89:0/4146123107 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfc4fa2120 con 0x55cfc4fae780 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f580ffff640 1 --2- 172.21.15.89:0/4146123107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fc61e0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f57fc031bf0 tx=0x7f57fc041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9edd1c4d11684567 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f57fa7fc640 1 -- 172.21.15.89:0/4146123107 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc045040 con 0x55cfc4fae780 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f57fa7fc640 1 -- 172.21.15.89:0/4146123107 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57fc031980 con 0x55cfc4fae780 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.641+0000 7f57fa7fc640 1 -- 172.21.15.89:0/4146123107 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc035040 con 0x55cfc4fae780 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 -- 172.21.15.89:0/4146123107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 msgr2=0x55cfc4fc61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 --2- 172.21.15.89:0/4146123107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fc61e0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f57fc031bf0 tx=0x7f57fc041040 comp rx=0 tx=0).stop 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 -- 172.21.15.89:0/4146123107 shutdown_connections 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 --2- 172.21.15.89:0/4146123107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4fae780 0x55cfc4fc61e0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 --2- 172.21.15.89:0/4146123107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fba6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 --2- 172.21.15.89:0/4146123107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fab470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 -- 172.21.15.89:0/4146123107 >> 172.21.15.89:0/4146123107 conn(0x55cfc4f68310 msgr2=0x55cfc4faad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 -- 172.21.15.89:0/4146123107 shutdown_connections 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 -- 172.21.15.89:0/4146123107 wait complete. 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 -- start start 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.642+0000 7f58194ff740 1 Processor -- start 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f58194ff740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fb6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f58194ff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fb6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f58194ff740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4fae780 0x55cfc4fa7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f58194ff740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f58101ae0e0 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f58194ff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58101ada10 con 0x55cfc4fa0b40 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f58194ff740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58101aac50 con 0x55cfc4fae780 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f58194ff740 10 client.0 _ll_register_callbacks cb 0x7f5804005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f580ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fb6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f580ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fb6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44648/0 (socket says 172.21.15.89:44648) 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f580ffff640 1 -- 172.21.15.89:0/3922830025 learned_addr learned my addr 172.21.15.89:0/3922830025 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f5814ec2640 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fb6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f580ffff640 1 -- 172.21.15.89:0/3922830025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4fae780 msgr2=0x55cfc4fa7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f580ffff640 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4fae780 0x55cfc4fa7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f580ffff640 1 -- 172.21.15.89:0/3922830025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 msgr2=0x55cfc4fb6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f580ffff640 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fb6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f580ffff640 1 -- 172.21.15.89:0/3922830025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfc4fa84c0 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f5814ec2640 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fb6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f580ffff640 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fb6a10 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009850 tx=0x7f57fc032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbaec459ef0241d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.643+0000 7f57effff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f57effff640 1 -- 172.21.15.89:0/3922830025 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc060060 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f58194ff740 1 -- 172.21.15.89:0/3922830025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57fc031980 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f57effff640 1 -- 172.21.15.89:0/3922830025 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f57fc0035f0 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f57effff640 1 -- 172.21.15.89:0/3922830025 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f57fc067050 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f58194ff740 10 client.19281 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f58194ff740 1 -- 172.21.15.89:0/3922830025 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cfc5028c50 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f57effff640 1 -- 172.21.15.89:0/3922830025 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57fc031980 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f57effff640 1 -- 172.21.15.89:0/3922830025 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f57fc056070 con 0x55cfc4f9f5d0 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f58194ff740 10 client.19281 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f58194ff740 10 client.19281 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f58194ff740 20 client.19281 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f58194ff740 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cfc5028f40 0x55cfc502b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f58194ff740 4 client.19281 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.644+0000 7f58194ff740 1 -- 172.21.15.89:0/3922830025 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cfc4f0cd70 con 0x55cfc5028f40 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f5814ec2640 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cfc5028f40 0x55cfc502b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f5814ec2640 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cfc5028f40 0x55cfc502b330 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f2b028374b83d19e server_cookie=e785ca77e34bae4f in_seq=0 out_seq=0 2024-09-15T23:35:28.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f57effff640 10 client.19281 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f57effff640 1 -- 172.21.15.89:0/3922830025 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cfc4f0cd70 con 0x55cfc5028f40 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f57effff640 10 client.19281 handle_command_reply: tid=0 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f58194ff740 1 client.19281 shutdown 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f58194ff740 1 -- 172.21.15.89:0/3922830025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4f9f5d0 msgr2=0x55cfc4fb6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f58194ff740 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fb6a10 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009850 tx=0x7f57fc032fc0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f58194ff740 1 -- 172.21.15.89:0/3922830025 shutdown_connections 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f58194ff740 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55cfc5028f40 0x55cfc502b330 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f58194ff740 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cfc4fae780 0x55cfc4fa7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f58194ff740 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cfc4fa0b40 0x55cfc4fb6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f58194ff740 1 --2- 172.21.15.89:0/3922830025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfc4f9f5d0 0x55cfc4fb6a10 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f58194ff740 1 -- 172.21.15.89:0/3922830025 >> 172.21.15.89:0/3922830025 conn(0x55cfc4f68310 msgr2=0x55cfc4facd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.645+0000 7f57effff640 0 client.19281 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f58194ff740 1 -- 172.21.15.89:0/3922830025 shutdown_connections 2024-09-15T23:35:28.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f58194ff740 1 -- 172.21.15.89:0/3922830025 wait complete. 2024-09-15T23:35:28.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f58194ff740 20 client.19281 trim_cache size 0 max 16384 2024-09-15T23:35:28.657 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:35:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f5817126640 1 -- 172.21.15.89:0/2231567680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58000a3240 msgr2=0x7f58000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f5817126640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58000a3240 0x7f58000a5700 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f580809c1f0 tx=0x7f5808066bf0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f5817126640 1 -- 172.21.15.89:0/2231567680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5810106a80 msgr2=0x7f581019cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f5817126640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5810106a80 0x7f581019cd20 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f580800eaa0 tx=0x7f580800ef60 comp rx=0 tx=0).stop 2024-09-15T23:35:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f5817126640 1 -- 172.21.15.89:0/2231567680 shutdown_connections 2024-09-15T23:35:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f5817126640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58000a3240 0x7f58000a5700 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f5817126640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5810106a80 0x7f581019cd20 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f5817126640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5810101420 0x7f5810198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f5817126640 1 --2- 172.21.15.89:0/2231567680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5810100a80 0x7f5810198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:28.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.646+0000 7f5817126640 1 -- 172.21.15.89:0/2231567680 >> 172.21.15.89:0/2231567680 conn(0x7f58100fc7a0 msgr2=0x7f5810104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.647+0000 7f5817126640 1 -- 172.21.15.89:0/2231567680 shutdown_connections 2024-09-15T23:35:28.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:28.647+0000 7f5817126640 1 -- 172.21.15.89:0/2231567680 wait complete. 2024-09-15T23:35:29.351 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:35:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:29 smithi120 ceph-mon[35250]: pgmap v693: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-09-15T23:35:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:29 smithi174 ceph-mon[35586]: pgmap v693: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-09-15T23:35:29.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:29 smithi089 ceph-mon[30933]: pgmap v693: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 KiB/s wr, 19 op/s 2024-09-15T23:35:29.597 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:30.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.092+0000 7f729200f640 1 -- 172.21.15.89:0/278250605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f728c10c180 msgr2=0x7f728c10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:30.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.092+0000 7f729200f640 1 --2- 172.21.15.89:0/278250605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f728c10c180 0x7f728c10e610 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f7284009ca0 tx=0x7f728402a950 comp rx=0 tx=0).stop 2024-09-15T23:35:30.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.092+0000 7f729200f640 1 -- 172.21.15.89:0/278250605 shutdown_connections 2024-09-15T23:35:30.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.092+0000 7f729200f640 1 --2- 172.21.15.89:0/278250605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f728c10c180 0x7f728c10e610 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.092+0000 7f729200f640 1 --2- 172.21.15.89:0/278250605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f728c103700 0x7f728c10bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.092+0000 7f729200f640 1 --2- 172.21.15.89:0/278250605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f728c102de0 0x7f728c1031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.092+0000 7f729200f640 1 -- 172.21.15.89:0/278250605 >> 172.21.15.89:0/278250605 conn(0x7f728c0fc7a0 msgr2=0x7f728c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:30.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.092+0000 7f729200f640 1 -- 172.21.15.89:0/278250605 shutdown_connections 2024-09-15T23:35:30.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.092+0000 7f729200f640 1 -- 172.21.15.89:0/278250605 wait complete. 2024-09-15T23:35:30.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f729200f640 1 Processor -- start 2024-09-15T23:35:30.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f729200f640 1 -- start start 2024-09-15T23:35:30.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f729200f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f728c102de0 0x7f728c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f729200f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f728c103700 0x7f728c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f729200f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f728c10c180 0x7f728c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f729200f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f728c112030 con 0x7f728c10c180 2024-09-15T23:35:30.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f729200f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f728c111eb0 con 0x7f728c103700 2024-09-15T23:35:30.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f729200f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1121b0 con 0x7f728c102de0 2024-09-15T23:35:30.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f728bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f728c10c180 0x7f728c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f728bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f728c10c180 0x7f728c19ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44668/0 (socket says 172.21.15.89:44668) 2024-09-15T23:35:30.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f728bfff640 1 -- 172.21.15.89:0/1139835540 learned_addr learned my addr 172.21.15.89:0/1139835540 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:30.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f728affd640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f728c103700 0x7f728c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f728b7fe640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f728c102de0 0x7f728c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f728bfff640 1 -- 172.21.15.89:0/1139835540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f728c102de0 msgr2=0x7f728c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:30.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f728bfff640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f728c102de0 0x7f728c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f728bfff640 1 -- 172.21.15.89:0/1139835540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f728c103700 msgr2=0x7f728c198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:30.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f728bfff640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f728c103700 0x7f728c198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.093+0000 7f728bfff640 1 -- 172.21.15.89:0/1139835540 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728c19d4f0 con 0x7f728c10c180 2024-09-15T23:35:30.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.094+0000 7f728bfff640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f728c10c180 0x7f728c19ce10 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f7284006010 tx=0x7f728402f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.094+0000 7f7288ff9640 1 -- 172.21.15.89:0/1139835540 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284034040 con 0x7f728c10c180 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.094+0000 7f729200f640 1 -- 172.21.15.89:0/1139835540 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728c19d780 con 0x7f728c10c180 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.094+0000 7f7288ff9640 1 -- 172.21.15.89:0/1139835540 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7284003e00 con 0x7f728c10c180 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.094+0000 7f7288ff9640 1 -- 172.21.15.89:0/1139835540 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284045480 con 0x7f728c10c180 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.094+0000 7f729200f640 1 -- 172.21.15.89:0/1139835540 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f728c1a50c0 con 0x7f728c10c180 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f7288ff9640 1 -- 172.21.15.89:0/1139835540 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7284037020 con 0x7f728c10c180 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f72943e8740 1 Processor -- start 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f72943e8740 1 -- start start 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f72943e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76ee1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f72943e8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 0x559f76ee1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f72943e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ef0780 0x559f76ef2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f72943e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559f76e472f0 con 0x559f76ee2b40 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f72943e8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559f76deecf0 con 0x559f76ee15d0 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f72943e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559f76e432b0 con 0x559f76ef0780 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76ee1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76ee1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44680/0 (socket says 172.21.15.89:44680) 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728affd640 1 -- 172.21.15.89:0/2572287518 learned_addr learned my addr 172.21.15.89:0/2572287518 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728b7fe640 1 --2- 172.21.15.89:0/2572287518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 0x559f76ee1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728affd640 1 --2- 172.21.15.89:0/2572287518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ef0780 0x559f76ef2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f7288ff9640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72740a3240 0x7f72740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728affd640 1 -- 172.21.15.89:0/2572287518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ef0780 msgr2=0x559f76ef2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728affd640 1 --2- 172.21.15.89:0/2572287518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ef0780 0x559f76ef2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728affd640 1 -- 172.21.15.89:0/2572287518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 msgr2=0x559f76ee1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:30.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728affd640 1 --2- 172.21.15.89:0/2572287518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 0x559f76ee1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728affd640 1 -- 172.21.15.89:0/2572287518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f76dd33a0 con 0x559f76ee2b40 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.097+0000 7f728b7fe640 1 --2- 172.21.15.89:0/2572287518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 0x559f76ee1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.098+0000 7f728bfff640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72740a3240 0x7f72740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.098+0000 7f728affd640 1 --2- 172.21.15.89:0/2572287518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76ee1090 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f7280006aa0 tx=0x7f72800095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73d5bfe1dd2feb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.098+0000 7f7288ff9640 1 -- 172.21.15.89:0/1139835540 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f72840f0e70 con 0x7f728c10c180 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.098+0000 7f7271ffb640 1 -- 172.21.15.89:0/2572287518 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280017610 con 0x559f76ee2b40 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.098+0000 7f7271ffb640 1 -- 172.21.15.89:0/2572287518 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f728001c070 con 0x559f76ee2b40 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.098+0000 7f7271ffb640 1 -- 172.21.15.89:0/2572287518 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72800159b0 con 0x559f76ee2b40 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 -- 172.21.15.89:0/2572287518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 msgr2=0x559f76ee1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f728bfff640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72740a3240 0x7f72740a5700 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7284008000 tx=0x7f72840dd040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 --2- 172.21.15.89:0/2572287518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76ee1090 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f7280006aa0 tx=0x7f72800095d0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 -- 172.21.15.89:0/2572287518 shutdown_connections 2024-09-15T23:35:30.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 --2- 172.21.15.89:0/2572287518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ef0780 0x559f76ef2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 --2- 172.21.15.89:0/2572287518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 0x559f76ee1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 --2- 172.21.15.89:0/2572287518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76ee1090 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 -- 172.21.15.89:0/2572287518 >> 172.21.15.89:0/2572287518 conn(0x559f76eaa310 msgr2=0x559f76eddc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 -- 172.21.15.89:0/2572287518 shutdown_connections 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 -- 172.21.15.89:0/2572287518 wait complete. 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 -- start start 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 Processor -- start 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee15d0 0x559f76ef0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76eed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ef0780 0x559f76eed9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559f76ef3420 con 0x559f76ee2b40 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.099+0000 7f72943e8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559f76e02060 con 0x559f76ef0780 2024-09-15T23:35:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f72943e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559f76ef35a0 con 0x559f76ee15d0 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f72943e8740 10 client.0 _ll_register_callbacks cb 0x559f76b558c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f728b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76eed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f728b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76eed3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44692/0 (socket says 172.21.15.89:44692) 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f728b7fe640 1 -- 172.21.15.89:0/1036316553 learned_addr learned my addr 172.21.15.89:0/1036316553 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f728affd640 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee15d0 0x559f76ef0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f728b7fe640 1 -- 172.21.15.89:0/1036316553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee15d0 msgr2=0x559f76ef0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f728b7fe640 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee15d0 0x559f76ef0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f728b7fe640 1 -- 172.21.15.89:0/1036316553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ef0780 msgr2=0x559f76eed9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f728b7fe640 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ef0780 0x559f76eed9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f728b7fe640 1 -- 172.21.15.89:0/1036316553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f76bb83c0 con 0x559f76ee2b40 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f728b7fe640 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76eed3f0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f727c00bfd0 tx=0x7f727c00f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac08b4ceaf0cacd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:30.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f726affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f726affd640 1 -- 172.21.15.89:0/1036316553 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c018020 con 0x559f76ee2b40 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f726affd640 1 -- 172.21.15.89:0/1036316553 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f727c014870 con 0x559f76ee2b40 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f72943e8740 1 -- 172.21.15.89:0/1036316553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f728001c070 con 0x559f76ee2b40 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f726affd640 1 -- 172.21.15.89:0/1036316553 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f727c023050 con 0x559f76ee2b40 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f72943e8740 10 client.19293 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.100+0000 7f72943e8740 1 -- 172.21.15.89:0/1036316553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559f76efeee0 con 0x559f76ee2b40 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.101+0000 7f726affd640 1 -- 172.21.15.89:0/1036316553 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f728001c070 con 0x559f76ee2b40 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.101+0000 7f726affd640 1 -- 172.21.15.89:0/1036316553 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f727c013b10 con 0x559f76ee2b40 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.101+0000 7f72943e8740 10 client.19293 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.101+0000 7f72943e8740 10 client.19293 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.101+0000 7f72943e8740 20 client.19293 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.101+0000 7f72943e8740 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f76f00440 0x559f76f02860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.101+0000 7f72943e8740 4 client.19293 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.101+0000 7f72943e8740 1 -- 172.21.15.89:0/1036316553 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f76e54860 con 0x559f76f00440 2024-09-15T23:35:30.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.101+0000 7f728affd640 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f76f00440 0x559f76f02860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f726affd640 10 client.19293 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f728affd640 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f76f00440 0x559f76f02860 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=452e11779242809a server_cookie=7dd026d3620b3cef in_seq=0 out_seq=0 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f726affd640 1 -- 172.21.15.89:0/1036316553 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559f76e54860 con 0x559f76f00440 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f726affd640 10 client.19293 handle_command_reply: tid=0 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f72943e8740 1 client.19293 shutdown 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f72943e8740 1 -- 172.21.15.89:0/1036316553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 msgr2=0x559f76eed3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f72943e8740 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76eed3f0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f727c00bfd0 tx=0x7f727c00f670 comp rx=0 tx=0).stop 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f72943e8740 1 -- 172.21.15.89:0/1036316553 shutdown_connections 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f72943e8740 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ef0780 0x559f76eed9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f72943e8740 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee2b40 0x559f76eed3f0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f72943e8740 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f76f00440 0x559f76f02860 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f72943e8740 1 --2- 172.21.15.89:0/1036316553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee15d0 0x559f76ef0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.102+0000 7f726affd640 0 client.19293 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:35:30.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.103+0000 7f72943e8740 1 -- 172.21.15.89:0/1036316553 >> 172.21.15.89:0/1036316553 conn(0x559f76eaa310 msgr2=0x559f76ee3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:30.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.103+0000 7f72943e8740 1 -- 172.21.15.89:0/1036316553 shutdown_connections 2024-09-15T23:35:30.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.103+0000 7f72943e8740 1 -- 172.21.15.89:0/1036316553 wait complete. 2024-09-15T23:35:30.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.103+0000 7f72943e8740 20 client.19293 trim_cache size 0 max 16384 2024-09-15T23:35:30.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f72943e8740 1 Processor -- start 2024-09-15T23:35:30.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f72943e8740 1 -- start start 2024-09-15T23:35:30.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f72943e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee15d0 0x559f76efc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f72943e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee2b40 0x559f76eec3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f72943e8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ef0780 0x559f76f08130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f72943e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1acc90 con 0x559f76ee15d0 2024-09-15T23:35:30.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f72943e8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1ae7d0 con 0x559f76ef0780 2024-09-15T23:35:30.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f72943e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1ad360 con 0x559f76ee2b40 2024-09-15T23:35:30.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f728affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee15d0 0x559f76efc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f728affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee15d0 0x559f76efc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44708/0 (socket says 172.21.15.89:44708) 2024-09-15T23:35:30.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f728affd640 1 -- 172.21.15.89:0/3562235704 learned_addr learned my addr 172.21.15.89:0/3562235704 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:30.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f728affd640 1 -- 172.21.15.89:0/3562235704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee2b40 msgr2=0x559f76eec3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:30.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f728affd640 1 --2- 172.21.15.89:0/3562235704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee2b40 0x559f76eec3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f728affd640 1 -- 172.21.15.89:0/3562235704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ef0780 msgr2=0x559f76f08130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:30.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f728affd640 1 --2- 172.21.15.89:0/3562235704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ef0780 0x559f76f08130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f728affd640 1 -- 172.21.15.89:0/3562235704 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f76ee4120 con 0x559f76ee15d0 2024-09-15T23:35:30.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f728affd640 1 --2- 172.21.15.89:0/3562235704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee15d0 0x559f76efc6c0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f7280006a30 tx=0x7f72800154f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48dde6239cea021a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:30.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.116+0000 7f72717fa640 1 -- 172.21.15.89:0/3562235704 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728002f070 con 0x559f76ee15d0 2024-09-15T23:35:30.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72717fa640 1 -- 172.21.15.89:0/3562235704 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7280027070 con 0x559f76ee15d0 2024-09-15T23:35:30.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72717fa640 1 -- 172.21.15.89:0/3562235704 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728002a8d0 con 0x559f76ee15d0 2024-09-15T23:35:30.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72943e8740 1 -- 172.21.15.89:0/3562235704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee15d0 msgr2=0x559f76efc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:30.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72943e8740 1 --2- 172.21.15.89:0/3562235704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee15d0 0x559f76efc6c0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f7280006a30 tx=0x7f72800154f0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72943e8740 1 -- 172.21.15.89:0/3562235704 shutdown_connections 2024-09-15T23:35:30.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72943e8740 1 --2- 172.21.15.89:0/3562235704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ef0780 0x559f76f08130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72943e8740 1 --2- 172.21.15.89:0/3562235704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee2b40 0x559f76eec3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72943e8740 1 --2- 172.21.15.89:0/3562235704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ee15d0 0x559f76efc6c0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72943e8740 1 -- 172.21.15.89:0/3562235704 >> 172.21.15.89:0/3562235704 conn(0x559f76eaa310 msgr2=0x559f76eebcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:30.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72943e8740 1 -- 172.21.15.89:0/3562235704 shutdown_connections 2024-09-15T23:35:30.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72943e8740 1 -- 172.21.15.89:0/3562235704 wait complete. 2024-09-15T23:35:30.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.117+0000 7f72943e8740 1 -- start start 2024-09-15T23:35:30.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f72943e8740 1 Processor -- start 2024-09-15T23:35:30.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f72943e8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 0x559f76ef47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f72943e8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee2b40 0x559f76ef4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f72943e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ef0780 0x559f76ef5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f72943e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1ae100 con 0x559f76ef0780 2024-09-15T23:35:30.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f72943e8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1ada30 con 0x559f76ee15d0 2024-09-15T23:35:30.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f72943e8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f728c1aac70 con 0x559f76ee2b40 2024-09-15T23:35:30.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f72943e8740 10 client.0 _ll_register_callbacks cb 0x7f727c00b2e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:30.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ef0780 0x559f76ef5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ef0780 0x559f76ef5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44724/0 (socket says 172.21.15.89:44724) 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728affd640 1 -- 172.21.15.89:0/495870061 learned_addr learned my addr 172.21.15.89:0/495870061 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728affd640 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 0x559f76ef47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728b7fe640 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee2b40 0x559f76ef4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728affd640 1 -- 172.21.15.89:0/495870061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee2b40 msgr2=0x559f76ef4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728affd640 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee2b40 0x559f76ef4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728affd640 1 -- 172.21.15.89:0/495870061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 msgr2=0x559f76ef47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728affd640 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 0x559f76ef47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728affd640 1 -- 172.21.15.89:0/495870061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f76f6a350 con 0x559f76ef0780 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.118+0000 7f728b7fe640 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee2b40 0x559f76ef4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f728affd640 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ef0780 0x559f76ef5230 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f7280009c70 tx=0x7f7280015590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c251bcc405f91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f726affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f726affd640 1 -- 172.21.15.89:0/495870061 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728003d060 con 0x559f76ef0780 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f726affd640 1 -- 172.21.15.89:0/495870061 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f728002a650 con 0x559f76ef0780 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f72943e8740 1 -- 172.21.15.89:0/495870061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7280027070 con 0x559f76ef0780 2024-09-15T23:35:30.126 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f726affd640 1 -- 172.21.15.89:0/495870061 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7280044050 con 0x559f76ef0780 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f72943e8740 10 client.19299 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f72943e8740 1 -- 172.21.15.89:0/495870061 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559f76f6b610 con 0x559f76ef0780 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f726affd640 1 -- 172.21.15.89:0/495870061 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7280027070 con 0x559f76ef0780 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f726affd640 1 -- 172.21.15.89:0/495870061 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7280035050 con 0x559f76ef0780 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f72943e8740 10 client.19299 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f72943e8740 10 client.19299 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f72943e8740 20 client.19299 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f72943e8740 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f76f6b8a0 0x559f76f6dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f72943e8740 4 client.19299 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.119+0000 7f72943e8740 1 -- 172.21.15.89:0/495870061 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f76e4ed70 con 0x559f76f6b8a0 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f728b7fe640 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f76f6b8a0 0x559f76f6dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f726affd640 10 client.19299 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f728b7fe640 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f76f6b8a0 0x559f76f6dc90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=615598cc6ff03b0d server_cookie=21935a029ba7f1f6 in_seq=0 out_seq=0 2024-09-15T23:35:30.127 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f726affd640 1 -- 172.21.15.89:0/495870061 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f76e4ed70 con 0x559f76f6b8a0 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f726affd640 10 client.19299 handle_command_reply: tid=0 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f72943e8740 1 client.19299 shutdown 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f72943e8740 1 -- 172.21.15.89:0/495870061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ef0780 msgr2=0x559f76ef5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f72943e8740 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ef0780 0x559f76ef5230 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f7280009c70 tx=0x7f7280015590 comp rx=0 tx=0).stop 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f72943e8740 1 -- 172.21.15.89:0/495870061 shutdown_connections 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f72943e8740 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559f76ef0780 0x559f76ef5230 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f72943e8740 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x559f76f6b8a0 0x559f76f6dc90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.120+0000 7f72943e8740 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559f76ee2b40 0x559f76ef4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f72943e8740 1 --2- 172.21.15.89:0/495870061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559f76ee15d0 0x559f76ef47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f72943e8740 1 -- 172.21.15.89:0/495870061 >> 172.21.15.89:0/495870061 conn(0x559f76eaa310 msgr2=0x559f76eec4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f726affd640 0 client.19299 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f72943e8740 1 -- 172.21.15.89:0/495870061 shutdown_connections 2024-09-15T23:35:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f72943e8740 1 -- 172.21.15.89:0/495870061 wait complete. 2024-09-15T23:35:30.129 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f72943e8740 20 client.19299 trim_cache size 0 max 16384 2024-09-15T23:35:30.129 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 -- 172.21.15.89:0/1139835540 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72740a3240 msgr2=0x7f72740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72740a3240 0x7f72740a5700 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7284008000 tx=0x7f72840dd040 comp rx=0 tx=0).stop 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 -- 172.21.15.89:0/1139835540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f728c10c180 msgr2=0x7f728c19ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f728c10c180 0x7f728c19ce10 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f7284006010 tx=0x7f728402f710 comp rx=0 tx=0).stop 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 -- 172.21.15.89:0/1139835540 shutdown_connections 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f72740a3240 0x7f72740a5700 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f728c10c180 0x7f728c19ce10 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f728c103700 0x7f728c198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 --2- 172.21.15.89:0/1139835540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f728c102de0 0x7f728c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 -- 172.21.15.89:0/1139835540 >> 172.21.15.89:0/1139835540 conn(0x7f728c0fc7a0 msgr2=0x7f728c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 -- 172.21.15.89:0/1139835540 shutdown_connections 2024-09-15T23:35:30.130 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:30.121+0000 7f729200f640 1 -- 172.21.15.89:0/1139835540 wait complete. 2024-09-15T23:35:30.863 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:35:31.108 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:31 smithi120 ceph-mon[35250]: pgmap v694: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:35:31.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:31 smithi174 ceph-mon[35586]: pgmap v694: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:35:31.514 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:31 smithi089 ceph-mon[30933]: pgmap v694: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:35:31.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.595+0000 7f59d7c46640 1 -- 172.21.15.89:0/1949017029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59d0101de0 msgr2=0x7f59d01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:31.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.595+0000 7f59d7c46640 1 --2- 172.21.15.89:0/1949017029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59d0101de0 0x7f59d01021c0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009900 tx=0x7f59cc02a410 comp rx=0 tx=0).stop 2024-09-15T23:35:31.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.595+0000 7f59d7c46640 1 -- 172.21.15.89:0/1949017029 shutdown_connections 2024-09-15T23:35:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.595+0000 7f59d7c46640 1 --2- 172.21.15.89:0/1949017029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59d0106f10 0x7f59d010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.595+0000 7f59d7c46640 1 --2- 172.21.15.89:0/1949017029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0102790 0x7f59d01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.595+0000 7f59d7c46640 1 --2- 172.21.15.89:0/1949017029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59d0101de0 0x7f59d01021c0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.595+0000 7f59d7c46640 1 -- 172.21.15.89:0/1949017029 >> 172.21.15.89:0/1949017029 conn(0x7f59d00fd8c0 msgr2=0x7f59d00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.595+0000 7f59d7c46640 1 -- 172.21.15.89:0/1949017029 shutdown_connections 2024-09-15T23:35:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.595+0000 7f59d7c46640 1 -- 172.21.15.89:0/1949017029 wait complete. 2024-09-15T23:35:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d7c46640 1 Processor -- start 2024-09-15T23:35:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d7c46640 1 -- start start 2024-09-15T23:35:31.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d7c46640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59d0101de0 0x7f59d00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d7c46640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59d0102790 0x7f59d0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d7c46640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0106f10 0x7f59d0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d7c46640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f59d0111f70 con 0x7f59d0102790 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d7c46640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f59d0111df0 con 0x7f59d0106f10 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d7c46640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59d01120f0 con 0x7f59d0101de0 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d51e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59d0102790 0x7f59d0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d51e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59d0102790 0x7f59d0074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44740/0 (socket says 172.21.15.89:44740) 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d51e1640 1 -- 172.21.15.89:0/1251289141 learned_addr learned my addr 172.21.15.89:0/1251289141 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d59e2640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59d0101de0 0x7f59d00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d61e3640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0106f10 0x7f59d0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d51e1640 1 -- 172.21.15.89:0/1251289141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59d0101de0 msgr2=0x7f59d00744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d51e1640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59d0101de0 0x7f59d00744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d51e1640 1 -- 172.21.15.89:0/1251289141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0106f10 msgr2=0x7f59d0074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d51e1640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0106f10 0x7f59d0074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d51e1640 1 -- 172.21.15.89:0/1251289141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d01a68f0 con 0x7f59d0102790 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.596+0000 7f59d51e1640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59d0102790 0x7f59d0074a30 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f59c000d950 tx=0x7f59c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.597+0000 7f59c6ffd640 1 -- 172.21.15.89:0/1251289141 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c0017070 con 0x7f59d0102790 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.597+0000 7f59d7c46640 1 -- 172.21.15.89:0/1251289141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d01a6be0 con 0x7f59d0102790 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.597+0000 7f59c6ffd640 1 -- 172.21.15.89:0/1251289141 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59c0009c30 con 0x7f59d0102790 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.597+0000 7f59c6ffd640 1 -- 172.21.15.89:0/1251289141 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c0020060 con 0x7f59d0102790 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.597+0000 7f59d7c46640 1 -- 172.21.15.89:0/1251289141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d01a7170 con 0x7f59d0102790 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.599+0000 7f59c6ffd640 1 -- 172.21.15.89:0/1251289141 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f59c00109d0 con 0x7f59d0102790 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.599+0000 7f59da01f740 1 Processor -- start 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59da01f740 1 -- start start 2024-09-15T23:35:31.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59da01f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560aaf5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59da01f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 0x55560aaf5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59da01f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560ab02760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59da01f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55560aa5b2f0 con 0x55560aaf6b40 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59da01f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55560aa02cf0 con 0x55560ab00370 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59da01f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55560aa572b0 con 0x55560aaf55d0 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59d61e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560aaf5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59d61e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560aaf5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44748/0 (socket says 172.21.15.89:44748) 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59d61e3640 1 -- 172.21.15.89:0/1629276692 learned_addr learned my addr 172.21.15.89:0/1629276692 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59c6ffd640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59bc0a3240 0x7f59bc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59d59e2640 1 --2- 172.21.15.89:0/1629276692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 0x55560aaf5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59d51e1640 1 --2- 172.21.15.89:0/1629276692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560ab02760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59d61e3640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59bc0a3240 0x7f59bc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.600+0000 7f59c6ffd640 1 -- 172.21.15.89:0/1251289141 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f59c00cfdd0 con 0x7f59d0102790 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.601+0000 7f59d59e2640 1 -- 172.21.15.89:0/1629276692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 msgr2=0x55560ab02760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.601+0000 7f59d59e2640 1 --2- 172.21.15.89:0/1629276692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560ab02760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.601+0000 7f59d59e2640 1 -- 172.21.15.89:0/1629276692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 msgr2=0x55560aaf5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.601+0000 7f59d59e2640 1 --2- 172.21.15.89:0/1629276692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560aaf5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.601+0000 7f59d59e2640 1 -- 172.21.15.89:0/1629276692 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55560a9e73a0 con 0x55560aaf55d0 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.601+0000 7f59d61e3640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59bc0a3240 0x7f59bc0a5700 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009fd0 tx=0x7f59c8004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.601+0000 7f59d59e2640 1 --2- 172.21.15.89:0/1629276692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 0x55560aaf5a10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009230 tx=0x7f59cc005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f54fc97ebd30097a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.601+0000 7f59bbfff640 1 -- 172.21.15.89:0/1629276692 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc03f070 con 0x55560aaf55d0 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.601+0000 7f59bbfff640 1 -- 172.21.15.89:0/1629276692 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59cc004030 con 0x55560aaf55d0 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.601+0000 7f59bbfff640 1 -- 172.21.15.89:0/1629276692 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc035070 con 0x55560aaf55d0 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 -- 172.21.15.89:0/1629276692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 msgr2=0x55560aaf5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:31.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 --2- 172.21.15.89:0/1629276692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 0x55560aaf5a10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009230 tx=0x7f59cc005e80 comp rx=0 tx=0).stop 2024-09-15T23:35:31.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 -- 172.21.15.89:0/1629276692 shutdown_connections 2024-09-15T23:35:31.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 --2- 172.21.15.89:0/1629276692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560ab02760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 --2- 172.21.15.89:0/1629276692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 0x55560aaf5a10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 --2- 172.21.15.89:0/1629276692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560aaf5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 -- 172.21.15.89:0/1629276692 >> 172.21.15.89:0/1629276692 conn(0x55560aabe310 msgr2=0x55560aaf1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:31.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 -- 172.21.15.89:0/1629276692 shutdown_connections 2024-09-15T23:35:31.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 -- 172.21.15.89:0/1629276692 wait complete. 2024-09-15T23:35:31.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 -- start start 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.602+0000 7f59da01f740 1 Processor -- start 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59da01f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf55d0 0x55560aaffd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59da01f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 0x55560aafcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59da01f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560aafd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59da01f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55560ab03010 con 0x55560aaf55d0 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59da01f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55560aa16060 con 0x55560ab00370 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59da01f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55560ab03190 con 0x55560aaf6b40 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59da01f740 10 client.0 _ll_register_callbacks cb 0x55560a7698c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59d59e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf55d0 0x55560aaffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59d59e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf55d0 0x55560aaffd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44764/0 (socket says 172.21.15.89:44764) 2024-09-15T23:35:31.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59d59e2640 1 -- 172.21.15.89:0/2630760693 learned_addr learned my addr 172.21.15.89:0/2630760693 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59d51e1640 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 0x55560aafcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59d59e2640 1 -- 172.21.15.89:0/2630760693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 msgr2=0x55560aafcfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59d59e2640 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 0x55560aafcfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59d59e2640 1 -- 172.21.15.89:0/2630760693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 msgr2=0x55560aafd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59d59e2640 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560aafd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59d59e2640 1 -- 172.21.15.89:0/2630760693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55560a7cc3c0 con 0x55560aaf55d0 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59d59e2640 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf55d0 0x55560aaffd30 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f59cc005c10 tx=0x7f59cc009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c1db9d936d67bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59b97fa640 1 -- 172.21.15.89:0/2630760693 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59cc035070 con 0x55560aaf55d0 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59b97fa640 1 -- 172.21.15.89:0/2630760693 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f59cc0317b0 con 0x55560aaf55d0 2024-09-15T23:35:31.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.603+0000 7f59da01f740 1 -- 172.21.15.89:0/2630760693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59cc004030 con 0x55560aaf55d0 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59b97fa640 1 -- 172.21.15.89:0/2630760693 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f59cc056050 con 0x55560aaf55d0 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59da01f740 10 client.19311 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59da01f740 1 -- 172.21.15.89:0/2630760693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55560ab0ea70 con 0x55560aaf55d0 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59b97fa640 1 -- 172.21.15.89:0/2630760693 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59cc004030 con 0x55560aaf55d0 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59b97fa640 1 -- 172.21.15.89:0/2630760693 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f59cc004730 con 0x55560aaf55d0 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59da01f740 10 client.19311 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59da01f740 10 client.19311 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59da01f740 20 client.19311 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59da01f740 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55560ab0fdc0 0x55560ab121b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59da01f740 4 client.19311 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59da01f740 1 -- 172.21.15.89:0/2630760693 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55560aa68860 con 0x55560ab0fdc0 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.604+0000 7f59d51e1640 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55560ab0fdc0 0x55560ab121b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.605+0000 7f59d51e1640 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55560ab0fdc0 0x55560ab121b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=18f82a8eb583d59a server_cookie=89b170b999fbdae7 in_seq=0 out_seq=0 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.605+0000 7f59b97fa640 10 client.19311 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.605+0000 7f59b97fa640 1 -- 172.21.15.89:0/2630760693 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55560aa68860 con 0x55560ab0fdc0 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.605+0000 7f59b97fa640 10 client.19311 handle_command_reply: tid=0 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.605+0000 7f59da01f740 1 client.19311 shutdown 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 1 -- 172.21.15.89:0/2630760693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf55d0 msgr2=0x55560aaffd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf55d0 0x55560aaffd30 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f59cc005c10 tx=0x7f59cc009610 comp rx=0 tx=0).stop 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 1 -- 172.21.15.89:0/2630760693 shutdown_connections 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55560ab0fdc0 0x55560ab121b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560aafd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 0x55560aafcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59b97fa640 0 client.19311 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 1 --2- 172.21.15.89:0/2630760693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf55d0 0x55560aaffd30 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 1 -- 172.21.15.89:0/2630760693 >> 172.21.15.89:0/2630760693 conn(0x55560aabe310 msgr2=0x55560aaf7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:31.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 1 -- 172.21.15.89:0/2630760693 shutdown_connections 2024-09-15T23:35:31.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 1 -- 172.21.15.89:0/2630760693 wait complete. 2024-09-15T23:35:31.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.606+0000 7f59da01f740 20 client.19311 trim_cache size 0 max 16384 2024-09-15T23:35:31.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.619+0000 7f59da01f740 1 Processor -- start 2024-09-15T23:35:31.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.619+0000 7f59da01f740 1 -- start start 2024-09-15T23:35:31.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.619+0000 7f59da01f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560aaf55d0 0x55560aaf6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.619+0000 7f59da01f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 0x55560aaf1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.619+0000 7f59da01f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560ab00370 0x55560ab151d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.619+0000 7f59da01f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f59d01b0fe0 con 0x55560ab00370 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.619+0000 7f59da01f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f59d01b2b20 con 0x55560aaf55d0 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.619+0000 7f59da01f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59d01b16b0 con 0x55560aaf6b40 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d61e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560ab00370 0x55560ab151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d61e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560ab00370 0x55560ab151d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44766/0 (socket says 172.21.15.89:44766) 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d61e3640 1 -- 172.21.15.89:0/3689836602 learned_addr learned my addr 172.21.15.89:0/3689836602 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d51e1640 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 0x55560aaf1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d59e2640 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560aaf55d0 0x55560aaf6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d61e3640 1 -- 172.21.15.89:0/3689836602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 msgr2=0x55560aaf1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d61e3640 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 0x55560aaf1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d61e3640 1 -- 172.21.15.89:0/3689836602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560aaf55d0 msgr2=0x55560aaf6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:31.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d61e3640 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560aaf55d0 0x55560aaf6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d61e3640 1 -- 172.21.15.89:0/3689836602 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55560aaf2960 con 0x55560ab00370 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d51e1640 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 0x55560aaf1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d59e2640 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560aaf55d0 0x55560aaf6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59d61e3640 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560ab00370 0x55560ab151d0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f59c8005bf0 tx=0x7f59c8009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a81657ce8566c6cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59bb7fe640 1 -- 172.21.15.89:0/3689836602 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c8025070 con 0x55560ab00370 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59bb7fe640 1 -- 172.21.15.89:0/3689836602 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59c8005020 con 0x55560ab00370 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.620+0000 7f59bb7fe640 1 -- 172.21.15.89:0/3689836602 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c8021620 con 0x55560ab00370 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 -- 172.21.15.89:0/3689836602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560ab00370 msgr2=0x55560ab151d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560ab00370 0x55560ab151d0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f59c8005bf0 tx=0x7f59c8009530 comp rx=0 tx=0).stop 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 -- 172.21.15.89:0/3689836602 shutdown_connections 2024-09-15T23:35:31.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560ab00370 0x55560ab151d0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf6b40 0x55560aaf1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 --2- 172.21.15.89:0/3689836602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560aaf55d0 0x55560aaf6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 -- 172.21.15.89:0/3689836602 >> 172.21.15.89:0/3689836602 conn(0x55560aabe310 msgr2=0x55560aafc130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 -- 172.21.15.89:0/3689836602 shutdown_connections 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 -- 172.21.15.89:0/3689836602 wait complete. 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 -- start start 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.621+0000 7f59da01f740 1 Processor -- start 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59da01f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 0x55560ab12ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59da01f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560ab13430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59da01f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560ab13970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59da01f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f59d01b2450 con 0x55560aaf6b40 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59da01f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f59d01b1d80 con 0x55560ab00370 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59da01f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59d01aefc0 con 0x55560aaf55d0 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59da01f740 10 client.0 _ll_register_callbacks cb 0x7f59c8009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59d51e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560ab13430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59d51e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560ab13430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44770/0 (socket says 172.21.15.89:44770) 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59d51e1640 1 -- 172.21.15.89:0/3175731671 learned_addr learned my addr 172.21.15.89:0/3175731671 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59d59e2640 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 0x55560ab12ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59d61e3640 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560ab13970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59d51e1640 1 -- 172.21.15.89:0/3175731671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 msgr2=0x55560ab12ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59d51e1640 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 0x55560ab12ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59d51e1640 1 -- 172.21.15.89:0/3175731671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 msgr2=0x55560ab13970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59d51e1640 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560ab13970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.622+0000 7f59d51e1640 1 -- 172.21.15.89:0/3175731671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55560ab79700 con 0x55560aaf6b40 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59d51e1640 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560ab13430 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f59c0098970 tx=0x7f59c000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf412b6a4232861 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59b97fa640 1 -- 172.21.15.89:0/3175731671 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c00b4070 con 0x55560aaf6b40 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59b97fa640 1 -- 172.21.15.89:0/3175731671 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f59c00cd590 con 0x55560aaf6b40 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59da01f740 1 -- 172.21.15.89:0/3175731671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59c8005020 con 0x55560aaf6b40 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59b97fa640 1 -- 172.21.15.89:0/3175731671 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f59c00a0350 con 0x55560aaf6b40 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59da01f740 10 client.19317 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59da01f740 1 -- 172.21.15.89:0/3175731671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55560ab7a980 con 0x55560aaf6b40 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59b97fa640 1 -- 172.21.15.89:0/3175731671 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59c8005020 con 0x55560aaf6b40 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59b97fa640 1 -- 172.21.15.89:0/3175731671 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f59c00b9020 con 0x55560aaf6b40 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59da01f740 10 client.19317 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59da01f740 10 client.19317 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59da01f740 20 client.19317 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:31.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59da01f740 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55560ab7b100 0x55560ab7d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59da01f740 4 client.19317 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.623+0000 7f59da01f740 1 -- 172.21.15.89:0/3175731671 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55560aa62d70 con 0x55560ab7b100 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59d59e2640 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55560ab7b100 0x55560ab7d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59d59e2640 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55560ab7b100 0x55560ab7d520 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b5233d016482074 server_cookie=d53b1a2ce17e0338 in_seq=0 out_seq=0 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59b97fa640 10 client.19317 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59b97fa640 1 -- 172.21.15.89:0/3175731671 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55560aa62d70 con 0x55560ab7b100 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59b97fa640 10 client.19317 handle_command_reply: tid=0 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 client.19317 shutdown 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 -- 172.21.15.89:0/3175731671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 msgr2=0x55560ab13430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560ab13430 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f59c0098970 tx=0x7f59c000bf60 comp rx=0 tx=0).stop 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 -- 172.21.15.89:0/3175731671 shutdown_connections 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55560ab7b100 0x55560ab7d520 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55560ab00370 0x55560ab13970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55560aaf6b40 0x55560ab13430 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59b97fa640 0 client.19317 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:35:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 --2- 172.21.15.89:0/3175731671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55560aaf55d0 0x55560ab12ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 -- 172.21.15.89:0/3175731671 >> 172.21.15.89:0/3175731671 conn(0x55560aabe310 msgr2=0x55560aaf04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 -- 172.21.15.89:0/3175731671 shutdown_connections 2024-09-15T23:35:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 1 -- 172.21.15.89:0/3175731671 wait complete. 2024-09-15T23:35:31.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.624+0000 7f59da01f740 20 client.19317 trim_cache size 0 max 16384 2024-09-15T23:35:31.637 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:35:31.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 -- 172.21.15.89:0/1251289141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59bc0a3240 msgr2=0x7f59bc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:31.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59bc0a3240 0x7f59bc0a5700 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009fd0 tx=0x7f59c8004ab0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 -- 172.21.15.89:0/1251289141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59d0102790 msgr2=0x7f59d0074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:31.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59d0102790 0x7f59d0074a30 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f59c000d950 tx=0x7f59c000de10 comp rx=0 tx=0).stop 2024-09-15T23:35:31.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 -- 172.21.15.89:0/1251289141 shutdown_connections 2024-09-15T23:35:31.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59bc0a3240 0x7f59bc0a5700 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0106f10 0x7f59d0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59d0102790 0x7f59d0074a30 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 --2- 172.21.15.89:0/1251289141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59d0101de0 0x7f59d00744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 -- 172.21.15.89:0/1251289141 >> 172.21.15.89:0/1251289141 conn(0x7f59d00fd8c0 msgr2=0x7f59d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 -- 172.21.15.89:0/1251289141 shutdown_connections 2024-09-15T23:35:31.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:31.625+0000 7f59d7c46640 1 -- 172.21.15.89:0/1251289141 wait complete. 2024-09-15T23:35:32.343 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:35:32.594 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:33.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.080+0000 7f17189be640 1 -- 172.21.15.89:0/1048588044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1714102ef0 msgr2=0x7f171410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:33.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.080+0000 7f17189be640 1 --2- 172.21.15.89:0/1048588044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1714102ef0 0x7f171410da40 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f170c009ca0 tx=0x7f170c02a950 comp rx=0 tx=0).stop 2024-09-15T23:35:33.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.080+0000 7f17189be640 1 -- 172.21.15.89:0/1048588044 shutdown_connections 2024-09-15T23:35:33.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.080+0000 7f17189be640 1 --2- 172.21.15.89:0/1048588044 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1714102ef0 0x7f171410da40 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.080+0000 7f17189be640 1 --2- 172.21.15.89:0/1048588044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1714102570 0x7f17141029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.080+0000 7f17189be640 1 --2- 172.21.15.89:0/1048588044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1714100e20 0x7f1714101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.080+0000 7f17189be640 1 -- 172.21.15.89:0/1048588044 >> 172.21.15.89:0/1048588044 conn(0x7f17140fc7d0 msgr2=0x7f17140febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.080+0000 7f17189be640 1 -- 172.21.15.89:0/1048588044 shutdown_connections 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.080+0000 7f17189be640 1 -- 172.21.15.89:0/1048588044 wait complete. 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.081+0000 7f17189be640 1 Processor -- start 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.081+0000 7f17189be640 1 -- start start 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.081+0000 7f17189be640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1714100e20 0x7f171419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.081+0000 7f17189be640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1714102570 0x7f171419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.081+0000 7f17189be640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1714102ef0 0x7f171419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.081+0000 7f17189be640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1714110fa0 con 0x7f1714100e20 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.081+0000 7f17189be640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1714110e20 con 0x7f1714102ef0 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.081+0000 7f17189be640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1714111120 con 0x7f1714102570 2024-09-15T23:35:33.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f171259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1714100e20 0x7f171419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f171259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1714100e20 0x7f171419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44782/0 (socket says 172.21.15.89:44782) 2024-09-15T23:35:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f171259c640 1 -- 172.21.15.89:0/554061884 learned_addr learned my addr 172.21.15.89:0/554061884 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f1712d9d640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1714102ef0 0x7f171419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f1709d9b640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1714102570 0x7f171419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f171259c640 1 -- 172.21.15.89:0/554061884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1714102570 msgr2=0x7f171419aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f171259c640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1714102570 0x7f171419aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f171259c640 1 -- 172.21.15.89:0/554061884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1714102ef0 msgr2=0x7f171419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:33.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f171259c640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1714102ef0 0x7f171419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f171259c640 1 -- 172.21.15.89:0/554061884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171419f610 con 0x7f1714100e20 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f1712d9d640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1714102ef0 0x7f171419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f171259c640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1714100e20 0x7f171419a660 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f170400c920 tx=0x7f170400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f170bfff640 1 -- 172.21.15.89:0/554061884 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704007c70 con 0x7f1714100e20 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f17189be640 1 -- 172.21.15.89:0/554061884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171419f900 con 0x7f1714100e20 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f170bfff640 1 -- 172.21.15.89:0/554061884 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1704007e10 con 0x7f1714100e20 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f170bfff640 1 -- 172.21.15.89:0/554061884 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1704018890 con 0x7f1714100e20 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.082+0000 7f17189be640 1 -- 172.21.15.89:0/554061884 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17141a71e0 con 0x7f1714100e20 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.085+0000 7f170bfff640 1 -- 172.21.15.89:0/554061884 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1704018a70 con 0x7f1714100e20 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.085+0000 7f171ad97740 1 Processor -- start 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.085+0000 7f171ad97740 1 -- start start 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.085+0000 7f171ad97740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80abd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.085+0000 7f171ad97740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80abda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.085+0000 7f171ad97740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80aca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.085+0000 7f171ad97740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564e80a232f0 con 0x564e80abeb40 2024-09-15T23:35:33.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.085+0000 7f171ad97740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e809cacf0 con 0x564e80ac8370 2024-09-15T23:35:33.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.085+0000 7f171ad97740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e80a1f2b0 con 0x564e80abd5d0 2024-09-15T23:35:33.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1712d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80abd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1712d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80abd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44794/0 (socket says 172.21.15.89:44794) 2024-09-15T23:35:33.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1712d9d640 1 -- 172.21.15.89:0/3414632770 learned_addr learned my addr 172.21.15.89:0/3414632770 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:33.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f171259c640 1 --2- 172.21.15.89:0/3414632770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80abda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1709d9b640 1 --2- 172.21.15.89:0/3414632770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80aca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1712d9d640 1 -- 172.21.15.89:0/3414632770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 msgr2=0x564e80abda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1712d9d640 1 --2- 172.21.15.89:0/3414632770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80abda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1712d9d640 1 -- 172.21.15.89:0/3414632770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 msgr2=0x564e80aca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1712d9d640 1 --2- 172.21.15.89:0/3414632770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80aca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1712d9d640 1 -- 172.21.15.89:0/3414632770 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e809af3a0 con 0x564e80abeb40 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1709d9b640 1 --2- 172.21.15.89:0/3414632770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80aca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f170bfff640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17000a3240 0x7f17000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f171259c640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17000a3240 0x7f17000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1712d9d640 1 --2- 172.21.15.89:0/3414632770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80abd090 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f170c009a20 tx=0x7f170c02f930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2928a73eb3560115 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:33.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f170bfff640 1 -- 172.21.15.89:0/554061884 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f17040cff80 con 0x7f1714100e20 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1708d99640 1 -- 172.21.15.89:0/3414632770 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c034040 con 0x564e80abeb40 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1708d99640 1 -- 172.21.15.89:0/3414632770 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f170c003970 con 0x564e80abeb40 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.086+0000 7f1708d99640 1 -- 172.21.15.89:0/3414632770 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170c0046b0 con 0x564e80abeb40 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.087+0000 7f171ad97740 1 -- 172.21.15.89:0/3414632770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 msgr2=0x564e80abd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.087+0000 7f171259c640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17000a3240 0x7f17000a5700 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f170409c0e0 tx=0x7f1704000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.087+0000 7f171ad97740 1 --2- 172.21.15.89:0/3414632770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80abd090 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f170c009a20 tx=0x7f170c02f930 comp rx=0 tx=0).stop 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.087+0000 7f171ad97740 1 -- 172.21.15.89:0/3414632770 shutdown_connections 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.087+0000 7f171ad97740 1 --2- 172.21.15.89:0/3414632770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80aca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.087+0000 7f171ad97740 1 --2- 172.21.15.89:0/3414632770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80abda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.087+0000 7f171ad97740 1 --2- 172.21.15.89:0/3414632770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80abd090 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.087+0000 7f171ad97740 1 -- 172.21.15.89:0/3414632770 >> 172.21.15.89:0/3414632770 conn(0x564e80a86310 msgr2=0x564e80ab9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.087+0000 7f171ad97740 1 -- 172.21.15.89:0/3414632770 shutdown_connections 2024-09-15T23:35:33.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.087+0000 7f171ad97740 1 -- 172.21.15.89:0/3414632770 wait complete. 2024-09-15T23:35:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f171ad97740 1 -- start start 2024-09-15T23:35:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f171ad97740 1 Processor -- start 2024-09-15T23:35:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f171ad97740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80abd5d0 0x564e80ac7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f171ad97740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f171ad97740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80ac8370 0x564e80ac55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f171ad97740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564e80acb010 con 0x564e80abeb40 2024-09-15T23:35:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f171ad97740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e809de060 con 0x564e80abd5d0 2024-09-15T23:35:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f171ad97740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e80acb190 con 0x564e80ac8370 2024-09-15T23:35:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f171ad97740 10 client.0 _ll_register_callbacks cb 0x564e807318c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:33.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1709d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1709d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44810/0 (socket says 172.21.15.89:44810) 2024-09-15T23:35:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1709d9b640 1 -- 172.21.15.89:0/1808950346 learned_addr learned my addr 172.21.15.89:0/1808950346 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1712d9d640 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80ac8370 0x564e80ac55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1712d9d640 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80abd5d0 0x564e80ac7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1709d9b640 1 -- 172.21.15.89:0/1808950346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80ac8370 msgr2=0x564e80ac55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1709d9b640 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80ac8370 0x564e80ac55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1709d9b640 1 -- 172.21.15.89:0/1808950346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80abd5d0 msgr2=0x564e80ac7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1709d9b640 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80abd5d0 0x564e80ac7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1709d9b640 1 -- 172.21.15.89:0/1808950346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e807943c0 con 0x564e80abeb40 2024-09-15T23:35:33.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.088+0000 7f1712d9d640 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80ac8370 0x564e80ac55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f1709d9b640 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac4fe0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009fd0 tx=0x7f16fc00e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0537a044fa9ae2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f16f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f16f9ffb640 1 -- 172.21.15.89:0/1808950346 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc017020 con 0x564e80abeb40 2024-09-15T23:35:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f16f9ffb640 1 -- 172.21.15.89:0/1808950346 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f16fc013610 con 0x564e80abeb40 2024-09-15T23:35:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f171ad97740 1 -- 172.21.15.89:0/1808950346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f170c003970 con 0x564e80abeb40 2024-09-15T23:35:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f16f9ffb640 1 -- 172.21.15.89:0/1808950346 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f16fc022050 con 0x564e80abeb40 2024-09-15T23:35:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f171ad97740 10 client.19329 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f171ad97740 1 -- 172.21.15.89:0/1808950346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564e80ad6a70 con 0x564e80abeb40 2024-09-15T23:35:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f16f9ffb640 1 -- 172.21.15.89:0/1808950346 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f170c003970 con 0x564e80abeb40 2024-09-15T23:35:33.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f16f9ffb640 1 -- 172.21.15.89:0/1808950346 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f16fc012050 con 0x564e80abeb40 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f171ad97740 10 client.19329 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f171ad97740 10 client.19329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f171ad97740 20 client.19329 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f171ad97740 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e80ad7fd0 0x564e80ada410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f171ad97740 4 client.19329 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.089+0000 7f171ad97740 1 -- 172.21.15.89:0/1808950346 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e80a30860 con 0x564e80ad7fd0 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.090+0000 7f1712d9d640 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e80ad7fd0 0x564e80ada410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.090+0000 7f16f9ffb640 10 client.19329 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.090+0000 7f1712d9d640 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e80ad7fd0 0x564e80ada410 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31527bf577472b5b server_cookie=ac1390a6c0afc153 in_seq=0 out_seq=0 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f16f9ffb640 1 -- 172.21.15.89:0/1808950346 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564e80a30860 con 0x564e80ad7fd0 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f16f9ffb640 10 client.19329 handle_command_reply: tid=0 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 client.19329 shutdown 2024-09-15T23:35:33.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 -- 172.21.15.89:0/1808950346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 msgr2=0x564e80ac4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac4fe0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009fd0 tx=0x7f16fc00e680 comp rx=0 tx=0).stop 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 -- 172.21.15.89:0/1808950346 shutdown_connections 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e80ad7fd0 0x564e80ada410 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f16f9ffb640 0 client.19329 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80ac8370 0x564e80ac55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac4fe0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 --2- 172.21.15.89:0/1808950346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80abd5d0 0x564e80ac7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 -- 172.21.15.89:0/1808950346 >> 172.21.15.89:0/1808950346 conn(0x564e80a86310 msgr2=0x564e80abf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 -- 172.21.15.89:0/1808950346 shutdown_connections 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 1 -- 172.21.15.89:0/1808950346 wait complete. 2024-09-15T23:35:33.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.091+0000 7f171ad97740 20 client.19329 trim_cache size 0 max 16384 2024-09-15T23:35:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f171ad97740 1 Processor -- start 2024-09-15T23:35:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f171ad97740 1 -- start start 2024-09-15T23:35:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f171ad97740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80ad4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f171ad97740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f171ad97740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80ac0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f171ad97740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f17141b1000 con 0x564e80abeb40 2024-09-15T23:35:33.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f171ad97740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f17141b2b40 con 0x564e80ac8370 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f171ad97740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f17141b16d0 con 0x564e80abd5d0 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f1709d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f1709d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44812/0 (socket says 172.21.15.89:44812) 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.104+0000 7f1709d9b640 1 -- 172.21.15.89:0/1053372747 learned_addr learned my addr 172.21.15.89:0/1053372747 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.105+0000 7f1712d9d640 1 --2- 172.21.15.89:0/1053372747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80ad4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.105+0000 7f1709d9b640 1 -- 172.21.15.89:0/1053372747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 msgr2=0x564e80ad4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.105+0000 7f1709d9b640 1 --2- 172.21.15.89:0/1053372747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80ad4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.105+0000 7f1709d9b640 1 -- 172.21.15.89:0/1053372747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 msgr2=0x564e80ac0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.105+0000 7f1709d9b640 1 --2- 172.21.15.89:0/1053372747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80ac0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.105+0000 7f1709d9b640 1 -- 172.21.15.89:0/1053372747 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e80ac3530 con 0x564e80abeb40 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.105+0000 7f1709d9b640 1 --2- 172.21.15.89:0/1053372747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac6f50 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f16fc01bff0 tx=0x7f16fc009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f6939a86d2261f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.105+0000 7f16fbfff640 1 -- 172.21.15.89:0/1053372747 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc017070 con 0x564e80abeb40 2024-09-15T23:35:33.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.105+0000 7f16fbfff640 1 -- 172.21.15.89:0/1053372747 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16fc01bc00 con 0x564e80abeb40 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.105+0000 7f16fbfff640 1 -- 172.21.15.89:0/1053372747 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc00f040 con 0x564e80abeb40 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 -- 172.21.15.89:0/1053372747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 msgr2=0x564e80ac6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 --2- 172.21.15.89:0/1053372747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac6f50 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f16fc01bff0 tx=0x7f16fc009bf0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 -- 172.21.15.89:0/1053372747 shutdown_connections 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 --2- 172.21.15.89:0/1053372747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80ac0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 --2- 172.21.15.89:0/1053372747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80ac6f50 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 --2- 172.21.15.89:0/1053372747 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80ad4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 -- 172.21.15.89:0/1053372747 >> 172.21.15.89:0/1053372747 conn(0x564e80a86310 msgr2=0x564e80ac6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 -- 172.21.15.89:0/1053372747 shutdown_connections 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 -- 172.21.15.89:0/1053372747 wait complete. 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 -- start start 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 Processor -- start 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80acc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80acce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80b41cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f17141b2470 con 0x564e80abeb40 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f17141b1da0 con 0x564e80ac8370 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f17141aefe0 con 0x564e80abd5d0 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f171ad97740 10 client.0 _ll_register_callbacks cb 0x7f16fc009120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f1709d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80acce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f1709d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80acce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44814/0 (socket says 172.21.15.89:44814) 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.106+0000 7f1709d9b640 1 -- 172.21.15.89:0/843852202 learned_addr learned my addr 172.21.15.89:0/843852202 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f1712d9d640 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80acc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f1712d9d640 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80b41cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f1709d9b640 1 -- 172.21.15.89:0/843852202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 msgr2=0x564e80acc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f1709d9b640 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80acc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f1709d9b640 1 -- 172.21.15.89:0/843852202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 msgr2=0x564e80b41cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:33.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f1709d9b640 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80b41cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f1709d9b640 1 -- 172.21.15.89:0/843852202 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e80b421f0 con 0x564e80abeb40 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f1712d9d640 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80acc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f1709d9b640 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80acce10 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f16fc01b990 tx=0x7f16fc0093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34244d55a611d3ff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f16f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f16f9ffb640 1 -- 172.21.15.89:0/843852202 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc017040 con 0x564e80abeb40 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f16f9ffb640 1 -- 172.21.15.89:0/843852202 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f16fc01b5a0 con 0x564e80abeb40 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f171ad97740 1 -- 172.21.15.89:0/843852202 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16fc01bc00 con 0x564e80abeb40 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f16f9ffb640 1 -- 172.21.15.89:0/843852202 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f16fc03b050 con 0x564e80abeb40 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f171ad97740 10 client.19335 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f171ad97740 1 -- 172.21.15.89:0/843852202 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564e80b434c0 con 0x564e80abeb40 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.107+0000 7f16f9ffb640 1 -- 172.21.15.89:0/843852202 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16fc01bc00 con 0x564e80abeb40 2024-09-15T23:35:33.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.108+0000 7f16f9ffb640 1 -- 172.21.15.89:0/843852202 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f16fc036030 con 0x564e80abeb40 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.108+0000 7f171ad97740 10 client.19335 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.108+0000 7f171ad97740 10 client.19335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.108+0000 7f171ad97740 20 client.19335 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.108+0000 7f171ad97740 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e80ad7fd0 0x564e80b45be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.108+0000 7f171ad97740 4 client.19335 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.108+0000 7f171ad97740 1 -- 172.21.15.89:0/843852202 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e80a2ad70 con 0x564e80ad7fd0 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.108+0000 7f1712d9d640 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e80ad7fd0 0x564e80b45be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.108+0000 7f1712d9d640 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e80ad7fd0 0x564e80b45be0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecaed56be1c3ba22 server_cookie=9daf4759462fd9a9 in_seq=0 out_seq=0 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.108+0000 7f16f9ffb640 10 client.19335 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f16f9ffb640 1 -- 172.21.15.89:0/843852202 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e80a2ad70 con 0x564e80ad7fd0 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f16f9ffb640 10 client.19335 handle_command_reply: tid=0 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 client.19335 shutdown 2024-09-15T23:35:33.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 -- 172.21.15.89:0/843852202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 msgr2=0x564e80acce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80acce10 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f16fc01b990 tx=0x7f16fc0093e0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 -- 172.21.15.89:0/843852202 shutdown_connections 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564e80ad7fd0 0x564e80b45be0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e80ac8370 0x564e80b41cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e80abeb40 0x564e80acce10 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 --2- 172.21.15.89:0/843852202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e80abd5d0 0x564e80acc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 -- 172.21.15.89:0/843852202 >> 172.21.15.89:0/843852202 conn(0x564e80a86310 msgr2=0x564e80ac7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f16f9ffb640 0 client.19335 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 -- 172.21.15.89:0/843852202 shutdown_connections 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 1 -- 172.21.15.89:0/843852202 wait complete. 2024-09-15T23:35:33.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f171ad97740 20 client.19335 trim_cache size 0 max 16384 2024-09-15T23:35:33.124 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:35:33.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f17189be640 1 -- 172.21.15.89:0/554061884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17000a3240 msgr2=0x7f17000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:33.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f17189be640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17000a3240 0x7f17000a5700 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f170409c0e0 tx=0x7f1704000bc0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f17189be640 1 -- 172.21.15.89:0/554061884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1714100e20 msgr2=0x7f171419a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:33.124 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.109+0000 7f17189be640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1714100e20 0x7f171419a660 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f170400c920 tx=0x7f170400cde0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.110+0000 7f17189be640 1 -- 172.21.15.89:0/554061884 shutdown_connections 2024-09-15T23:35:33.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.110+0000 7f17189be640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f17000a3240 0x7f17000a5700 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.110+0000 7f17189be640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1714102ef0 0x7f171419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.110+0000 7f17189be640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1714102570 0x7f171419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.110+0000 7f17189be640 1 --2- 172.21.15.89:0/554061884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1714100e20 0x7f171419a660 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:33.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.110+0000 7f17189be640 1 -- 172.21.15.89:0/554061884 >> 172.21.15.89:0/554061884 conn(0x7f17140fc7d0 msgr2=0x7f171406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:33.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.110+0000 7f17189be640 1 -- 172.21.15.89:0/554061884 shutdown_connections 2024-09-15T23:35:33.125 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:33.110+0000 7f17189be640 1 -- 172.21.15.89:0/554061884 wait complete. 2024-09-15T23:35:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:33 smithi089 ceph-mon[30933]: pgmap v695: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:35:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:33 smithi120 ceph-mon[35250]: pgmap v695: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:35:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:33 smithi174 ceph-mon[35586]: pgmap v695: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 17 op/s 2024-09-15T23:35:34.901 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:35:34.902 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:35:35.159 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:35 smithi120 ceph-mon[35250]: pgmap v696: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-15T23:35:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:35 smithi174 ceph-mon[35586]: pgmap v696: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-15T23:35:35.536 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:35 smithi089 ceph-mon[30933]: pgmap v696: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-15T23:35:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.620+0000 7f678e100640 1 -- 172.21.15.89:0/2053981850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788101de0 msgr2=0x7f67881021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.620+0000 7f678e100640 1 --2- 172.21.15.89:0/2053981850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788101de0 0x7f67881021c0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6778009900 tx=0x7f677802a410 comp rx=0 tx=0).stop 2024-09-15T23:35:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.620+0000 7f678e100640 1 -- 172.21.15.89:0/2053981850 shutdown_connections 2024-09-15T23:35:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.620+0000 7f678e100640 1 --2- 172.21.15.89:0/2053981850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6788106f10 0x7f678810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.620+0000 7f678e100640 1 --2- 172.21.15.89:0/2053981850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6788102790 0x7f67881067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.620+0000 7f678e100640 1 --2- 172.21.15.89:0/2053981850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788101de0 0x7f67881021c0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.620+0000 7f678e100640 1 -- 172.21.15.89:0/2053981850 >> 172.21.15.89:0/2053981850 conn(0x7f67880fd8c0 msgr2=0x7f67880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.620+0000 7f678e100640 1 -- 172.21.15.89:0/2053981850 shutdown_connections 2024-09-15T23:35:35.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.620+0000 7f678e100640 1 -- 172.21.15.89:0/2053981850 wait complete. 2024-09-15T23:35:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f678e100640 1 Processor -- start 2024-09-15T23:35:35.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f678e100640 1 -- start start 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f678e100640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6788101de0 0x7f6788198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f678e100640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6788102790 0x7f6788198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f678e100640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788106f10 0x7f678819cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f678e100640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6788111fb0 con 0x7f6788106f10 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f678e100640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6788111e30 con 0x7f6788101de0 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f678e100640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6788112130 con 0x7f6788102790 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f6787fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788106f10 0x7f678819cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f6787fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788106f10 0x7f678819cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40340/0 (socket says 172.21.15.89:40340) 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f6787fff640 1 -- 172.21.15.89:0/399744605 learned_addr learned my addr 172.21.15.89:0/399744605 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f67877fe640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6788101de0 0x7f6788198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.621+0000 7f6786ffd640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6788102790 0x7f6788198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f6787fff640 1 -- 172.21.15.89:0/399744605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6788102790 msgr2=0x7f6788198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:35.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f6787fff640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6788102790 0x7f6788198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f6787fff640 1 -- 172.21.15.89:0/399744605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6788101de0 msgr2=0x7f6788198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f6787fff640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6788101de0 0x7f6788198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f6787fff640 1 -- 172.21.15.89:0/399744605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f678819d4c0 con 0x7f6788106f10 2024-09-15T23:35:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f6787fff640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788106f10 0x7f678819cde0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f678000eaa0 tx=0x7f678000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f6784ff9640 1 -- 172.21.15.89:0/399744605 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678000cdb0 con 0x7f6788106f10 2024-09-15T23:35:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f6784ff9640 1 -- 172.21.15.89:0/399744605 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6780005020 con 0x7f6788106f10 2024-09-15T23:35:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f678e100640 1 -- 172.21.15.89:0/399744605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f678819d7b0 con 0x7f6788106f10 2024-09-15T23:35:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f6784ff9640 1 -- 172.21.15.89:0/399744605 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67800116b0 con 0x7f6788106f10 2024-09-15T23:35:35.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.622+0000 7f678e100640 1 -- 172.21.15.89:0/399744605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67881a5090 con 0x7f6788106f10 2024-09-15T23:35:35.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f6784ff9640 1 -- 172.21.15.89:0/399744605 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6780005560 con 0x7f6788106f10 2024-09-15T23:35:35.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f67904d9740 1 Processor -- start 2024-09-15T23:35:35.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f67904d9740 1 -- start start 2024-09-15T23:35:35.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f67904d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a32af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f67904d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a335800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f67904d9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a338130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f67904d9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55895a290280 con 0x55895a335d40 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f67904d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55895a237b10 con 0x55895a32ab60 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f67904d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55895a28c240 con 0x55895a32b480 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f6786ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a338130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f6786ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a338130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40350/0 (socket says 172.21.15.89:40350) 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.625+0000 7f6786ffd640 1 -- 172.21.15.89:0/1204990037 learned_addr learned my addr 172.21.15.89:0/1204990037 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6786ffd640 1 --2- 172.21.15.89:0/1204990037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a32af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f67877fe640 1 --2- 172.21.15.89:0/1204990037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a335800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6784ff9640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67700a32c0 0x7f67700a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6786ffd640 1 -- 172.21.15.89:0/1204990037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 msgr2=0x55895a335800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6786ffd640 1 --2- 172.21.15.89:0/1204990037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a335800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6787fff640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67700a32c0 0x7f67700a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6786ffd640 1 -- 172.21.15.89:0/1204990037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 msgr2=0x55895a32af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6786ffd640 1 --2- 172.21.15.89:0/1204990037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a32af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6786ffd640 1 -- 172.21.15.89:0/1204990037 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895a2441f0 con 0x55895a335d40 2024-09-15T23:35:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6784ff9640 1 -- 172.21.15.89:0/399744605 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f67800cfeb0 con 0x7f6788106f10 2024-09-15T23:35:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6786ffd640 1 --2- 172.21.15.89:0/1204990037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a338130 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f677c016530 tx=0x7f677c00a2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f20f9933f5811e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f676dffb640 1 -- 172.21.15.89:0/1204990037 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c018430 con 0x55895a335d40 2024-09-15T23:35:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f676dffb640 1 -- 172.21.15.89:0/1204990037 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f677c018ab0 con 0x55895a335d40 2024-09-15T23:35:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f6787fff640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67700a32c0 0x7f67700a5780 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f678009bde0 tx=0x7f678006deb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:35.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.626+0000 7f676dffb640 1 -- 172.21.15.89:0/1204990037 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c0179d0 con 0x55895a335d40 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.627+0000 7f67904d9740 1 -- 172.21.15.89:0/1204990037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 msgr2=0x55895a338130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.627+0000 7f67904d9740 1 --2- 172.21.15.89:0/1204990037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a338130 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f677c016530 tx=0x7f677c00a2b0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.627+0000 7f67904d9740 1 -- 172.21.15.89:0/1204990037 shutdown_connections 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.627+0000 7f67904d9740 1 --2- 172.21.15.89:0/1204990037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a338130 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.627+0000 7f67904d9740 1 --2- 172.21.15.89:0/1204990037 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a335800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.627+0000 7f67904d9740 1 --2- 172.21.15.89:0/1204990037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a32af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.627+0000 7f67904d9740 1 -- 172.21.15.89:0/1204990037 >> 172.21.15.89:0/1204990037 conn(0x55895a3270f0 msgr2=0x55895a3274d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.627+0000 7f67904d9740 1 -- 172.21.15.89:0/1204990037 shutdown_connections 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.627+0000 7f67904d9740 1 -- 172.21.15.89:0/1204990037 wait complete. 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.627+0000 7f67904d9740 1 -- start start 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f67904d9740 1 Processor -- start 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f67904d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a332c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f67904d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a333180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f67904d9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a3336c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f67904d9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55895a3388b0 con 0x55895a335d40 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f67904d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55895a24af10 con 0x55895a32ab60 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f67904d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55895a338a30 con 0x55895a32b480 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f67904d9740 10 client.0 _ll_register_callbacks cb 0x55895a213a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f6786ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a3336c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f6786ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a3336c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40358/0 (socket says 172.21.15.89:40358) 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f6786ffd640 1 -- 172.21.15.89:0/845216686 learned_addr learned my addr 172.21.15.89:0/845216686 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f6786ffd640 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a332c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f67877fe640 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a333180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f6786ffd640 1 -- 172.21.15.89:0/845216686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 msgr2=0x55895a333180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f6786ffd640 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a333180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f6786ffd640 1 -- 172.21.15.89:0/845216686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 msgr2=0x55895a332c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:35.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f6786ffd640 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a332c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.628+0000 7f6786ffd640 1 -- 172.21.15.89:0/845216686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895a0016a0 con 0x55895a335d40 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f6786ffd640 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a3336c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f677c01a040 tx=0x7f677c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0fdf8685f8cfd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f6766ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f6766ffd640 1 -- 172.21.15.89:0/845216686 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c02b060 con 0x55895a335d40 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f6766ffd640 1 -- 172.21.15.89:0/845216686 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f677c018640 con 0x55895a335d40 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f67904d9740 1 -- 172.21.15.89:0/845216686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f677c018ab0 con 0x55895a335d40 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f6766ffd640 1 -- 172.21.15.89:0/845216686 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f677c032050 con 0x55895a335d40 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f67904d9740 10 client.19347 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f67904d9740 1 -- 172.21.15.89:0/845216686 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55895a3444e0 con 0x55895a335d40 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f6766ffd640 1 -- 172.21.15.89:0/845216686 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f677c018ab0 con 0x55895a335d40 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f6766ffd640 1 -- 172.21.15.89:0/845216686 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f677c012050 con 0x55895a335d40 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f67904d9740 10 client.19347 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f67904d9740 10 client.19347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:35.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f67904d9740 20 client.19347 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f67904d9740 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55895a3458e0 0x55895a347cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f67904d9740 4 client.19347 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.629+0000 7f67904d9740 1 -- 172.21.15.89:0/845216686 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55895a29d7f0 con 0x55895a3458e0 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.630+0000 7f67877fe640 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55895a3458e0 0x55895a347cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.630+0000 7f67877fe640 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55895a3458e0 0x55895a347cd0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bec2388887154377 server_cookie=8fd515499d7f22d3 in_seq=0 out_seq=0 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.630+0000 7f6766ffd640 10 client.19347 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f6766ffd640 1 -- 172.21.15.89:0/845216686 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55895a29d7f0 con 0x55895a3458e0 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f6766ffd640 10 client.19347 handle_command_reply: tid=0 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 client.19347 shutdown 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 -- 172.21.15.89:0/845216686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 msgr2=0x55895a3336c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a3336c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f677c01a040 tx=0x7f677c0047d0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 -- 172.21.15.89:0/845216686 shutdown_connections 2024-09-15T23:35:35.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55895a3458e0 0x55895a347cd0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f6766ffd640 0 client.19347 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:35.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a3336c0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a333180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 --2- 172.21.15.89:0/845216686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a332c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 -- 172.21.15.89:0/845216686 >> 172.21.15.89:0/845216686 conn(0x55895a3270f0 msgr2=0x55895a32cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:35.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 -- 172.21.15.89:0/845216686 shutdown_connections 2024-09-15T23:35:35.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 1 -- 172.21.15.89:0/845216686 wait complete. 2024-09-15T23:35:35.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.631+0000 7f67904d9740 20 client.19347 trim_cache size 0 max 16384 2024-09-15T23:35:35.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.648+0000 7f67904d9740 1 Processor -- start 2024-09-15T23:35:35.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.648+0000 7f67904d9740 1 -- start start 2024-09-15T23:35:35.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.648+0000 7f67904d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a334e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.648+0000 7f67904d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a34d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.648+0000 7f67904d9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a34d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.648+0000 7f67904d9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f67881aef90 con 0x55895a335d40 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.648+0000 7f67904d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f67881b0ad0 con 0x55895a32ab60 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.648+0000 7f67904d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f67881af660 con 0x55895a32b480 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f6786ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a34d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f6786ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a34d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40360/0 (socket says 172.21.15.89:40360) 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f6786ffd640 1 -- 172.21.15.89:0/3899865311 learned_addr learned my addr 172.21.15.89:0/3899865311 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f6786ffd640 1 --2- 172.21.15.89:0/3899865311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a334e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f6786ffd640 1 -- 172.21.15.89:0/3899865311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 msgr2=0x55895a34d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f6786ffd640 1 --2- 172.21.15.89:0/3899865311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a34d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f6786ffd640 1 -- 172.21.15.89:0/3899865311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 msgr2=0x55895a334e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f6786ffd640 1 --2- 172.21.15.89:0/3899865311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a334e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f6786ffd640 1 -- 172.21.15.89:0/3899865311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895a32b2b0 con 0x55895a335d40 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f6786ffd640 1 --2- 172.21.15.89:0/3899865311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a34d7e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f677c024450 tx=0x7f677c00aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40469ec637aa7fc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f676d7fa640 1 -- 172.21.15.89:0/3899865311 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c03b070 con 0x55895a335d40 2024-09-15T23:35:35.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f676d7fa640 1 -- 172.21.15.89:0/3899865311 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f677c00a6c0 con 0x55895a335d40 2024-09-15T23:35:35.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.649+0000 7f676d7fa640 1 -- 172.21.15.89:0/3899865311 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c02b070 con 0x55895a335d40 2024-09-15T23:35:35.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 -- 172.21.15.89:0/3899865311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 msgr2=0x55895a34d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:35.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 --2- 172.21.15.89:0/3899865311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a34d7e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f677c024450 tx=0x7f677c00aae0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 -- 172.21.15.89:0/3899865311 shutdown_connections 2024-09-15T23:35:35.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 --2- 172.21.15.89:0/3899865311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a34d7e0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 --2- 172.21.15.89:0/3899865311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a34d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 --2- 172.21.15.89:0/3899865311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a334e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 -- 172.21.15.89:0/3899865311 >> 172.21.15.89:0/3899865311 conn(0x55895a3270f0 msgr2=0x55895a326270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 -- 172.21.15.89:0/3899865311 shutdown_connections 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 -- 172.21.15.89:0/3899865311 wait complete. 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 -- start start 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.650+0000 7f67904d9740 1 Processor -- start 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67904d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a339c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67904d9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a33a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67904d9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a33a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67904d9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f67881b0400 con 0x55895a335d40 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67904d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f67881afd30 con 0x55895a32ab60 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67904d9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f67881acf70 con 0x55895a32b480 2024-09-15T23:35:35.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67904d9740 10 client.0 _ll_register_callbacks cb 0x7f677c00ac60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6786ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a33a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6786ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a33a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40364/0 (socket says 172.21.15.89:40364) 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6786ffd640 1 -- 172.21.15.89:0/1124701438 learned_addr learned my addr 172.21.15.89:0/1124701438 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67877fe640 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a33a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6786ffd640 1 -- 172.21.15.89:0/1124701438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 msgr2=0x55895a33a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6786ffd640 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a33a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6786ffd640 1 -- 172.21.15.89:0/1124701438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 msgr2=0x55895a339c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6786ffd640 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a339c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6786ffd640 1 -- 172.21.15.89:0/1124701438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895a3af4c0 con 0x55895a335d40 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6786ffd640 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a33a700 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f677c000c00 tx=0x7f677c00af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1254027e49904be server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6766ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6766ffd640 1 -- 172.21.15.89:0/1124701438 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f677c03b070 con 0x55895a335d40 2024-09-15T23:35:35.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6766ffd640 1 -- 172.21.15.89:0/1124701438 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f677c0246c0 con 0x55895a335d40 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67904d9740 1 -- 172.21.15.89:0/1124701438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f677c00a6c0 con 0x55895a335d40 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f6766ffd640 1 -- 172.21.15.89:0/1124701438 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f677c048050 con 0x55895a335d40 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67904d9740 10 client.19353 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.651+0000 7f67904d9740 1 -- 172.21.15.89:0/1124701438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55895a3b07e0 con 0x55895a335d40 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f6766ffd640 1 -- 172.21.15.89:0/1124701438 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f677c00a6c0 con 0x55895a335d40 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f6766ffd640 1 -- 172.21.15.89:0/1124701438 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f677c012050 con 0x55895a335d40 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f67904d9740 10 client.19353 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f67904d9740 10 client.19353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f67904d9740 20 client.19353 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f67904d9740 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55895a3458e0 0x55895a3b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f67904d9740 4 client.19353 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f67904d9740 1 -- 172.21.15.89:0/1124701438 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55895a297d00 con 0x55895a3458e0 2024-09-15T23:35:35.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f67877fe640 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55895a3458e0 0x55895a3b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:35.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f6766ffd640 10 client.19353 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:35.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.652+0000 7f67877fe640 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55895a3458e0 0x55895a3b2e00 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8f1134a129263c server_cookie=9ca7ff19401c361b in_seq=0 out_seq=0 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f6766ffd640 1 -- 172.21.15.89:0/1124701438 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55895a297d00 con 0x55895a3458e0 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f6766ffd640 10 client.19353 handle_command_reply: tid=0 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 client.19353 shutdown 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 -- 172.21.15.89:0/1124701438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 msgr2=0x55895a33a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a33a700 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f677c000c00 tx=0x7f677c00af40 comp rx=0 tx=0).stop 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 -- 172.21.15.89:0/1124701438 shutdown_connections 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55895a3458e0 0x55895a3b2e00 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55895a335d40 0x55895a33a700 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f6766ffd640 0 client.19353 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55895a32b480 0x55895a33a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 --2- 172.21.15.89:0/1124701438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55895a32ab60 0x55895a339c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 -- 172.21.15.89:0/1124701438 >> 172.21.15.89:0/1124701438 conn(0x55895a3270f0 msgr2=0x55895a325a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:35.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 -- 172.21.15.89:0/1124701438 shutdown_connections 2024-09-15T23:35:35.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 1 -- 172.21.15.89:0/1124701438 wait complete. 2024-09-15T23:35:35.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.654+0000 7f67904d9740 20 client.19353 trim_cache size 0 max 16384 2024-09-15T23:35:35.660 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:35:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:35:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "eac31eaf-4243-4059-8f75-403bcada2720", 2024-09-15T23:35:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:35:35.660 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:35:35.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 -- 172.21.15.89:0/399744605 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67700a32c0 msgr2=0x7f67700a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:35.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67700a32c0 0x7f67700a5780 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f678009bde0 tx=0x7f678006deb0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 -- 172.21.15.89:0/399744605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788106f10 msgr2=0x7f678819cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:35.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788106f10 0x7f678819cde0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f678000eaa0 tx=0x7f678000ef60 comp rx=0 tx=0).stop 2024-09-15T23:35:35.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 -- 172.21.15.89:0/399744605 shutdown_connections 2024-09-15T23:35:35.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f67700a32c0 0x7f67700a5780 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6788106f10 0x7f678819cde0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6788102790 0x7f6788198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 --2- 172.21.15.89:0/399744605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6788101de0 0x7f6788198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 -- 172.21.15.89:0/399744605 >> 172.21.15.89:0/399744605 conn(0x7f67880fd8c0 msgr2=0x7f67880febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 -- 172.21.15.89:0/399744605 shutdown_connections 2024-09-15T23:35:35.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:35.655+0000 7f678e100640 1 -- 172.21.15.89:0/399744605 wait complete. 2024-09-15T23:35:36.468 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:36 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:35:36.468 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:36 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:35:36.468 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:36 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:35:36.468 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:36 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:35:36.469 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:35:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:36 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:35:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:36 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:35:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:36 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:35:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:36 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:35:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:36 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:35:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:36 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:35:36.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:36 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:35:36.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:36 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:35:36.717 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:37.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.187+0000 7f989d60b640 1 -- 172.21.15.89:0/1124450845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9898101430 msgr2=0x7f989810d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:37.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.187+0000 7f989d60b640 1 --2- 172.21.15.89:0/1124450845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9898101430 0x7f989810d960 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f9890009930 tx=0x7f989002a420 comp rx=0 tx=0).stop 2024-09-15T23:35:37.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.188+0000 7f989d60b640 1 -- 172.21.15.89:0/1124450845 shutdown_connections 2024-09-15T23:35:37.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.188+0000 7f989d60b640 1 --2- 172.21.15.89:0/1124450845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9898101430 0x7f989810d960 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.188+0000 7f989d60b640 1 --2- 172.21.15.89:0/1124450845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9898100a90 0x7f9898100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.188+0000 7f989d60b640 1 --2- 172.21.15.89:0/1124450845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9898106a90 0x7f9898106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.188+0000 7f989d60b640 1 -- 172.21.15.89:0/1124450845 >> 172.21.15.89:0/1124450845 conn(0x7f98980fc7d0 msgr2=0x7f98980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:37.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.188+0000 7f989d60b640 1 -- 172.21.15.89:0/1124450845 shutdown_connections 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.188+0000 7f989d60b640 1 -- 172.21.15.89:0/1124450845 wait complete. 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.188+0000 7f989d60b640 1 Processor -- start 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.188+0000 7f989d60b640 1 -- start start 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f989d60b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9898100a90 0x7f9898196220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f989d60b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9898101430 0x7f9898196760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f989d60b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9898106a90 0x7f989819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f989d60b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f989810fd80 con 0x7f9898106a90 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f989d60b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f989810fc00 con 0x7f9898100a90 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f989d60b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f989810ff00 con 0x7f9898101430 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f98977fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9898106a90 0x7f989819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f98977fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9898106a90 0x7f989819aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40378/0 (socket says 172.21.15.89:40378) 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f98977fe640 1 -- 172.21.15.89:0/3946404875 learned_addr learned my addr 172.21.15.89:0/3946404875 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f9896ffd640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9898100a90 0x7f9898196220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f98967fc640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9898101430 0x7f9898196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f98977fe640 1 -- 172.21.15.89:0/3946404875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9898101430 msgr2=0x7f9898196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f98977fe640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9898101430 0x7f9898196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f98977fe640 1 -- 172.21.15.89:0/3946404875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9898100a90 msgr2=0x7f9898196220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f98977fe640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9898100a90 0x7f9898196220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f98977fe640 1 -- 172.21.15.89:0/3946404875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f989819b1d0 con 0x7f9898106a90 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f98977fe640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9898106a90 0x7f989819aaf0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f9890005ab0 tx=0x7f98900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f988ffff640 1 -- 172.21.15.89:0/3946404875 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f989003f030 con 0x7f9898106a90 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.189+0000 7f989d60b640 1 -- 172.21.15.89:0/3946404875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f989819b460 con 0x7f9898106a90 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.190+0000 7f988ffff640 1 -- 172.21.15.89:0/3946404875 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9890003ce0 con 0x7f9898106a90 2024-09-15T23:35:37.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.190+0000 7f988ffff640 1 -- 172.21.15.89:0/3946404875 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9890035070 con 0x7f9898106a90 2024-09-15T23:35:37.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.190+0000 7f989d60b640 1 -- 172.21.15.89:0/3946404875 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98981a2da0 con 0x7f9898106a90 2024-09-15T23:35:37.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.192+0000 7f988ffff640 1 -- 172.21.15.89:0/3946404875 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9890042030 con 0x7f9898106a90 2024-09-15T23:35:37.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.192+0000 7f989f9e4740 1 Processor -- start 2024-09-15T23:35:37.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.192+0000 7f989f9e4740 1 -- start start 2024-09-15T23:35:37.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.192+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c5f490 0x5621c4c5f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5fdb0 0x5621c4c6a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c6ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f989f9e4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5621c4bc52e0 con 0x5621c4c6a670 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f989f9e4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5621c4b6cce0 con 0x5621c4c5f490 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f989f9e4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5621c4bc12a0 con 0x5621c4c5fdb0 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f988ffff640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98840a5450 0x7f98840a7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98967fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c6ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98967fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c6ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40382/0 (socket says 172.21.15.89:40382) 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98967fc640 1 -- 172.21.15.89:0/2144315671 learned_addr learned my addr 172.21.15.89:0/2144315671 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f9896ffd640 1 --2- 172.21.15.89:0/2144315671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5fdb0 0x5621c4c6a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98967fc640 1 --2- 172.21.15.89:0/2144315671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c5f490 0x5621c4c5f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98977fe640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98840a5450 0x7f98840a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f988ffff640 1 -- 172.21.15.89:0/3946404875 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f98900f00f0 con 0x7f9898106a90 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98967fc640 1 -- 172.21.15.89:0/2144315671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5fdb0 msgr2=0x5621c4c6a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98967fc640 1 --2- 172.21.15.89:0/2144315671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5fdb0 0x5621c4c6a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98967fc640 1 -- 172.21.15.89:0/2144315671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c5f490 msgr2=0x5621c4c5f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98967fc640 1 --2- 172.21.15.89:0/2144315671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c5f490 0x5621c4c5f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98967fc640 1 -- 172.21.15.89:0/2144315671 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621c4b51440 con 0x5621c4c6a670 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98967fc640 1 --2- 172.21.15.89:0/2144315671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c6ca60 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f988000be10 tx=0x7f9880004120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c62c1189d96e5c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f98977fe640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98840a5450 0x7f98840a7910 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9890006010 tx=0x7f989008e1a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.193+0000 7f988d7fa640 1 -- 172.21.15.89:0/2144315671 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9880003e80 con 0x5621c4c6a670 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f988d7fa640 1 -- 172.21.15.89:0/2144315671 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9880005500 con 0x5621c4c6a670 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f988d7fa640 1 -- 172.21.15.89:0/2144315671 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9880012450 con 0x5621c4c6a670 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f989f9e4740 1 -- 172.21.15.89:0/2144315671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 msgr2=0x5621c4c6ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f989f9e4740 1 --2- 172.21.15.89:0/2144315671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c6ca60 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f988000be10 tx=0x7f9880004120 comp rx=0 tx=0).stop 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f989f9e4740 1 -- 172.21.15.89:0/2144315671 shutdown_connections 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f989f9e4740 1 --2- 172.21.15.89:0/2144315671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c6ca60 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f989f9e4740 1 --2- 172.21.15.89:0/2144315671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5fdb0 0x5621c4c6a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f989f9e4740 1 --2- 172.21.15.89:0/2144315671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c5f490 0x5621c4c5f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f989f9e4740 1 -- 172.21.15.89:0/2144315671 >> 172.21.15.89:0/2144315671 conn(0x5621c4c5ba20 msgr2=0x5621c4c5be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f989f9e4740 1 -- 172.21.15.89:0/2144315671 shutdown_connections 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.194+0000 7f989f9e4740 1 -- 172.21.15.89:0/2144315671 wait complete. 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f989f9e4740 1 -- start start 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f989f9e4740 1 Processor -- start 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5f490 0x5621c4c6a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c5fdb0 0x5621c4c672c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c67860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f989f9e4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5621c4c6d240 con 0x5621c4c6a670 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f989f9e4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5621c4b80050 con 0x5621c4c5fdb0 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f989f9e4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5621c4c6d3c0 con 0x5621c4c5f490 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f989f9e4740 10 client.0 _ll_register_callbacks cb 0x5621c4bc0160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f98967fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c67860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f98967fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c67860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40384/0 (socket says 172.21.15.89:40384) 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f98967fc640 1 -- 172.21.15.89:0/444524616 learned_addr learned my addr 172.21.15.89:0/444524616 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f98967fc640 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5f490 0x5621c4c6a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.195+0000 7f9896ffd640 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c5fdb0 0x5621c4c672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f98967fc640 1 -- 172.21.15.89:0/444524616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5f490 msgr2=0x5621c4c6a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f98967fc640 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5f490 0x5621c4c6a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f98967fc640 1 -- 172.21.15.89:0/444524616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c5fdb0 msgr2=0x5621c4c672c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f98967fc640 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c5fdb0 0x5621c4c672c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f98967fc640 1 -- 172.21.15.89:0/444524616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621c49363c0 con 0x5621c4c6a670 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f98967fc640 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c67860 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f9880005fd0 tx=0x7f9880014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88e154dadfc2f1c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f987e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f987e7fc640 1 -- 172.21.15.89:0/444524616 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9880003e50 con 0x5621c4c6a670 2024-09-15T23:35:37.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f987e7fc640 1 -- 172.21.15.89:0/444524616 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9880005010 con 0x5621c4c6a670 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f989f9e4740 1 -- 172.21.15.89:0/444524616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9880005500 con 0x5621c4c6a670 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f987e7fc640 1 -- 172.21.15.89:0/444524616 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9880031050 con 0x5621c4c6a670 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f989f9e4740 10 client.19365 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f989f9e4740 1 -- 172.21.15.89:0/444524616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5621c4c78c50 con 0x5621c4c6a670 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f987e7fc640 1 -- 172.21.15.89:0/444524616 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9880005500 con 0x5621c4c6a670 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f987e7fc640 1 -- 172.21.15.89:0/444524616 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9880010420 con 0x5621c4c6a670 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f989f9e4740 10 client.19365 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f989f9e4740 10 client.19365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f989f9e4740 20 client.19365 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f989f9e4740 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621c4c79fa0 0x5621c4c7c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f989f9e4740 4 client.19365 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.196+0000 7f989f9e4740 1 -- 172.21.15.89:0/444524616 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621c4bd2850 con 0x5621c4c79fa0 2024-09-15T23:35:37.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.197+0000 7f9896ffd640 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621c4c79fa0 0x5621c4c7c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.197+0000 7f9896ffd640 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621c4c79fa0 0x5621c4c7c390 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=698b2e513af08aeb server_cookie=dde8869ea21dbb13 in_seq=0 out_seq=0 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.197+0000 7f987e7fc640 10 client.19365 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f987e7fc640 1 -- 172.21.15.89:0/444524616 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5621c4bd2850 con 0x5621c4c79fa0 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f987e7fc640 10 client.19365 handle_command_reply: tid=0 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 client.19365 shutdown 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 -- 172.21.15.89:0/444524616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 msgr2=0x5621c4c67860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c67860 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f9880005fd0 tx=0x7f9880014040 comp rx=0 tx=0).stop 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 -- 172.21.15.89:0/444524616 shutdown_connections 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621c4c79fa0 0x5621c4c7c390 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c6a670 0x5621c4c67860 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c5fdb0 0x5621c4c672c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f987e7fc640 0 client.19365 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:37.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 --2- 172.21.15.89:0/444524616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5f490 0x5621c4c6a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 -- 172.21.15.89:0/444524616 >> 172.21.15.89:0/444524616 conn(0x5621c4c5ba20 msgr2=0x5621c4c61700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:37.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 -- 172.21.15.89:0/444524616 shutdown_connections 2024-09-15T23:35:37.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 1 -- 172.21.15.89:0/444524616 wait complete. 2024-09-15T23:35:37.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.198+0000 7f989f9e4740 20 client.19365 trim_cache size 0 max 16384 2024-09-15T23:35:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f989f9e4740 1 Processor -- start 2024-09-15T23:35:37.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f989f9e4740 1 -- start start 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5f490 0x5621c4c764a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5fdb0 0x5621c4c65b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c6a670 0x5621c4c615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f989f9e4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98981accc0 con 0x5621c4c5f490 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f989f9e4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98981ae800 con 0x5621c4c6a670 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f989f9e4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98981ad390 con 0x5621c4c5fdb0 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f98967fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5f490 0x5621c4c764a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f98967fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5f490 0x5621c4c764a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40394/0 (socket says 172.21.15.89:40394) 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f98967fc640 1 -- 172.21.15.89:0/3936595707 learned_addr learned my addr 172.21.15.89:0/3936595707 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f9896ffd640 1 --2- 172.21.15.89:0/3936595707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5fdb0 0x5621c4c65b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f98967fc640 1 -- 172.21.15.89:0/3936595707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5fdb0 msgr2=0x5621c4c65b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f98967fc640 1 --2- 172.21.15.89:0/3936595707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5fdb0 0x5621c4c65b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f98967fc640 1 -- 172.21.15.89:0/3936595707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c6a670 msgr2=0x5621c4c615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f98967fc640 1 --2- 172.21.15.89:0/3936595707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c6a670 0x5621c4c615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.216+0000 7f98967fc640 1 -- 172.21.15.89:0/3936595707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621c4c6cdd0 con 0x5621c4c5f490 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.217+0000 7f98967fc640 1 --2- 172.21.15.89:0/3936595707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5f490 0x5621c4c764a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f9880004160 tx=0x7f988000b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f01bb8fd97c4fad server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.217+0000 7f988cff9640 1 -- 172.21.15.89:0/3936595707 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9880024020 con 0x5621c4c5f490 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.217+0000 7f988cff9640 1 -- 172.21.15.89:0/3936595707 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9880016070 con 0x5621c4c5f490 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.217+0000 7f988cff9640 1 -- 172.21.15.89:0/3936595707 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9880004990 con 0x5621c4c5f490 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 -- 172.21.15.89:0/3936595707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5f490 msgr2=0x5621c4c764a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 --2- 172.21.15.89:0/3936595707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5f490 0x5621c4c764a0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f9880004160 tx=0x7f988000b6f0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 -- 172.21.15.89:0/3936595707 shutdown_connections 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 --2- 172.21.15.89:0/3936595707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c6a670 0x5621c4c615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 --2- 172.21.15.89:0/3936595707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5fdb0 0x5621c4c65b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 --2- 172.21.15.89:0/3936595707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5f490 0x5621c4c764a0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 -- 172.21.15.89:0/3936595707 >> 172.21.15.89:0/3936595707 conn(0x5621c4c5ba20 msgr2=0x5621c4c60ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 -- 172.21.15.89:0/3936595707 shutdown_connections 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 -- 172.21.15.89:0/3936595707 wait complete. 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 -- start start 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 Processor -- start 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5f490 0x5621c4c643a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5fdb0 0x5621c4c648e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c6a670 0x5621c4c64e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f98981ae130 con 0x5621c4c5fdb0 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98981ada60 con 0x5621c4c6a670 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98981aaca0 con 0x5621c4c5f490 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f989f9e4740 10 client.0 _ll_register_callbacks cb 0x7f988000b900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f9896ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5fdb0 0x5621c4c648e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.218+0000 7f9896ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5fdb0 0x5621c4c648e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40408/0 (socket says 172.21.15.89:40408) 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f9896ffd640 1 -- 172.21.15.89:0/2649540197 learned_addr learned my addr 172.21.15.89:0/2649540197 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f98967fc640 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5f490 0x5621c4c643a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f98967fc640 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c6a670 0x5621c4c64e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f9896ffd640 1 -- 172.21.15.89:0/2649540197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5f490 msgr2=0x5621c4c643a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f9896ffd640 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5f490 0x5621c4c643a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f9896ffd640 1 -- 172.21.15.89:0/2649540197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c6a670 msgr2=0x5621c4c64e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f9896ffd640 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c6a670 0x5621c4c64e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f9896ffd640 1 -- 172.21.15.89:0/2649540197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621c4ce3c10 con 0x5621c4c5fdb0 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f98967fc640 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c6a670 0x5621c4c64e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f9896ffd640 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5fdb0 0x5621c4c648e0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x5621c4c5c880 tx=0x7f9888011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8940654620727090 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f987e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f987e7fc640 1 -- 172.21.15.89:0/2649540197 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f988802a060 con 0x5621c4c5fdb0 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f987e7fc640 1 -- 172.21.15.89:0/2649540197 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9888003d80 con 0x5621c4c5fdb0 2024-09-15T23:35:37.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f989f9e4740 1 -- 172.21.15.89:0/2649540197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9880016070 con 0x5621c4c5fdb0 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f987e7fc640 1 -- 172.21.15.89:0/2649540197 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9888031050 con 0x5621c4c5fdb0 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f989f9e4740 10 client.19371 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f989f9e4740 1 -- 172.21.15.89:0/2649540197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5621c4ce4f60 con 0x5621c4c5fdb0 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.219+0000 7f987e7fc640 1 -- 172.21.15.89:0/2649540197 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9880016070 con 0x5621c4c5fdb0 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.220+0000 7f987e7fc640 1 -- 172.21.15.89:0/2649540197 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9888024030 con 0x5621c4c5fdb0 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.220+0000 7f989f9e4740 10 client.19371 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.220+0000 7f989f9e4740 10 client.19371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.220+0000 7f989f9e4740 20 client.19371 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.220+0000 7f989f9e4740 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621c4ce56b0 0x5621c4ce7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.220+0000 7f989f9e4740 4 client.19371 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.220+0000 7f989f9e4740 1 -- 172.21.15.89:0/2649540197 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5621c4bccd60 con 0x5621c4ce56b0 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.220+0000 7f98967fc640 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621c4ce56b0 0x5621c4ce7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:37.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f987e7fc640 10 client.19371 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f98967fc640 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621c4ce56b0 0x5621c4ce7ad0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fa5b832d32ab00a server_cookie=3b792417ae7d773e in_seq=0 out_seq=0 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f987e7fc640 1 -- 172.21.15.89:0/2649540197 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5621c4bccd60 con 0x5621c4ce56b0 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f987e7fc640 10 client.19371 handle_command_reply: tid=0 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 client.19371 shutdown 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 -- 172.21.15.89:0/2649540197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5fdb0 msgr2=0x5621c4c648e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5fdb0 0x5621c4c648e0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x5621c4c5c880 tx=0x7f9888011c50 comp rx=0 tx=0).stop 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 -- 172.21.15.89:0/2649540197 shutdown_connections 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5621c4ce56b0 0x5621c4ce7ad0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c4c6a670 0x5621c4c64e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5621c4c5fdb0 0x5621c4c648e0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 --2- 172.21.15.89:0/2649540197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5621c4c5f490 0x5621c4c643a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 -- 172.21.15.89:0/2649540197 >> 172.21.15.89:0/2649540197 conn(0x5621c4c5ba20 msgr2=0x5621c4c69340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:37.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f987e7fc640 0 client.19371 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:37.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 -- 172.21.15.89:0/2649540197 shutdown_connections 2024-09-15T23:35:37.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 1 -- 172.21.15.89:0/2649540197 wait complete. 2024-09-15T23:35:37.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.221+0000 7f989f9e4740 20 client.19371 trim_cache size 0 max 16384 2024-09-15T23:35:37.236 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:35:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:35:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:35:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "eac31eaf-4243-4059-8f75-403bcada2720": { 2024-09-15T23:35:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:35:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "eac31eaf-4243-4059-8f75-403bcada2720", 2024-09-15T23:35:37.236 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:35:37.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:37.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:37.237 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:35:37.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 -- 172.21.15.89:0/3946404875 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98840a5450 msgr2=0x7f98840a7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:37.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98840a5450 0x7f98840a7910 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9890006010 tx=0x7f989008e1a0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 -- 172.21.15.89:0/3946404875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9898106a90 msgr2=0x7f989819aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:37.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9898106a90 0x7f989819aaf0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f9890005ab0 tx=0x7f98900042f0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 -- 172.21.15.89:0/3946404875 shutdown_connections 2024-09-15T23:35:37.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f98840a5450 0x7f98840a7910 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9898106a90 0x7f989819aaf0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9898101430 0x7f9898196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 --2- 172.21.15.89:0/3946404875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9898100a90 0x7f9898196220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:37.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 -- 172.21.15.89:0/3946404875 >> 172.21.15.89:0/3946404875 conn(0x7f98980fc7d0 msgr2=0x7f98980fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:37.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 -- 172.21.15.89:0/3946404875 shutdown_connections 2024-09-15T23:35:37.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:37.222+0000 7f989d60b640 1 -- 172.21.15.89:0/3946404875 wait complete. 2024-09-15T23:35:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:37 smithi120 ceph-mon[35250]: pgmap v697: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2024-09-15T23:35:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:37 smithi174 ceph-mon[35586]: pgmap v697: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2024-09-15T23:35:37.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:37 smithi089 ceph-mon[30933]: pgmap v697: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2024-09-15T23:35:37.974 INFO:tasks.cephfs.filesystem:scrub status for tag:eac31eaf-4243-4059-8f75-403bcada2720 - {'path': '/', 'tag': 'eac31eaf-4243-4059-8f75-403bcada2720', 'options': 'recursive,force'} 2024-09-15T23:35:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.045+0000 7fd9cf32c640 1 -- 172.21.15.89:0/1586401144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8102600 msgr2=0x7fd9c8108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.045+0000 7fd9cf32c640 1 --2- 172.21.15.89:0/1586401144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8102600 0x7fd9c8108e90 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0009ca0 tx=0x7fd9c002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.045+0000 7fd9cf32c640 1 -- 172.21.15.89:0/1586401144 shutdown_connections 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.045+0000 7fd9cf32c640 1 --2- 172.21.15.89:0/1586401144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8102600 0x7fd9c8108e90 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.045+0000 7fd9cf32c640 1 --2- 172.21.15.89:0/1586401144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9c8101c40 0x7fd9c81020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.045+0000 7fd9cf32c640 1 --2- 172.21.15.89:0/1586401144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c8100a50 0x7fd9c8100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.045+0000 7fd9cf32c640 1 -- 172.21.15.89:0/1586401144 >> 172.21.15.89:0/1586401144 conn(0x7fd9c80fc1b0 msgr2=0x7fd9c80fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.045+0000 7fd9cf32c640 1 -- 172.21.15.89:0/1586401144 shutdown_connections 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cf32c640 1 -- 172.21.15.89:0/1586401144 wait complete. 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cf32c640 1 Processor -- start 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cf32c640 1 -- start start 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cf32c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9c8100a50 0x7fd9c8197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cf32c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c8101c40 0x7fd9c81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cf32c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8102600 0x7fd9c819f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cf32c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c810d820 con 0x7fd9c8102600 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cf32c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c810d6a0 con 0x7fd9c8101c40 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cf32c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c810d9a0 con 0x7fd9c8100a50 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cd8c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8102600 0x7fd9c819f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cd8c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8102600 0x7fd9c819f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40436/0 (socket says 172.21.15.89:40436) 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cd8c9640 1 -- 172.21.15.89:0/2215083561 learned_addr learned my addr 172.21.15.89:0/2215083561 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9c8100a50 0x7fd9c8197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.046+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c8101c40 0x7fd9c81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9cd8c9640 1 -- 172.21.15.89:0/2215083561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9c8100a50 msgr2=0x7fd9c8197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9cd8c9640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9c8100a50 0x7fd9c8197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9cd8c9640 1 -- 172.21.15.89:0/2215083561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c8101c40 msgr2=0x7fd9c81982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9cd8c9640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c8101c40 0x7fd9c81982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9cd8c9640 1 -- 172.21.15.89:0/2215083561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c819f990 con 0x7fd9c8102600 2024-09-15T23:35:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9cd8c9640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8102600 0x7fd9c819f320 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fd9c00098d0 tx=0x7fd9c002f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9be7fc640 1 -- 172.21.15.89:0/2215083561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c002fc30 con 0x7fd9c8102600 2024-09-15T23:35:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9be7fc640 1 -- 172.21.15.89:0/2215083561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9c0003e50 con 0x7fd9c8102600 2024-09-15T23:35:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9be7fc640 1 -- 172.21.15.89:0/2215083561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c00305c0 con 0x7fd9c8102600 2024-09-15T23:35:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9cf32c640 1 -- 172.21.15.89:0/2215083561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c819fc20 con 0x7fd9c8102600 2024-09-15T23:35:38.055 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.047+0000 7fd9cf32c640 1 -- 172.21.15.89:0/2215083561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c81a01c0 con 0x7fd9c8102600 2024-09-15T23:35:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.049+0000 7fd9be7fc640 1 -- 172.21.15.89:0/2215083561 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd9c0003970 con 0x7fd9c8102600 2024-09-15T23:35:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9d20be740 1 Processor -- start 2024-09-15T23:35:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9d20be740 1 -- start start 2024-09-15T23:35:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0619a0 0x55708f061da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0622e0 0x55708f066da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f067370 0x55708f069830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9d20be740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55708efc52f0 con 0x55708f0619a0 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9d20be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55708ef6d270 con 0x55708f0622e0 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9d20be740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55708efc12b0 con 0x55708f067370 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9cc8c7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0619a0 0x55708f061da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9cc8c7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0619a0 0x55708f061da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40450/0 (socket says 172.21.15.89:40450) 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9cc8c7640 1 -- 172.21.15.89:0/2818004383 learned_addr learned my addr 172.21.15.89:0/2818004383 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9be7fc640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9b40a3370 0x7fd9b40a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/2818004383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0622e0 0x55708f066da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9cd8c9640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9b40a3370 0x7fd9b40a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9cc8c7640 1 -- 172.21.15.89:0/2818004383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f067370 msgr2=0x55708f069830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/2818004383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f067370 0x55708f069830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9cc8c7640 1 -- 172.21.15.89:0/2818004383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0622e0 msgr2=0x55708f066da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/2818004383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0622e0 0x55708f066da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9cc8c7640 1 -- 172.21.15.89:0/2818004383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55708ef6c340 con 0x55708f0619a0 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.050+0000 7fd9be7fc640 1 -- 172.21.15.89:0/2215083561 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd9c00f11d0 con 0x7fd9c8102600 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/2818004383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0619a0 0x55708f061da0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800ac90 tx=0x7fd9b80117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d5b15d1f899bf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9b37fe640 1 -- 172.21.15.89:0/2818004383 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8005020 con 0x55708f0619a0 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9cd8c9640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9b40a3370 0x7fd9b40a5830 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd9c003ffb0 tx=0x7fd9c0049000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9b37fe640 1 -- 172.21.15.89:0/2818004383 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9b80047a0 con 0x55708f0619a0 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9b37fe640 1 -- 172.21.15.89:0/2818004383 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b80126b0 con 0x55708f0619a0 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9d20be740 1 -- 172.21.15.89:0/2818004383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0619a0 msgr2=0x55708f061da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9d20be740 1 --2- 172.21.15.89:0/2818004383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0619a0 0x55708f061da0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800ac90 tx=0x7fd9b80117b0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9d20be740 1 -- 172.21.15.89:0/2818004383 shutdown_connections 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9d20be740 1 --2- 172.21.15.89:0/2818004383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f067370 0x55708f069830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9d20be740 1 --2- 172.21.15.89:0/2818004383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0622e0 0x55708f066da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.051+0000 7fd9d20be740 1 --2- 172.21.15.89:0/2818004383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0619a0 0x55708f061da0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 -- 172.21.15.89:0/2818004383 >> 172.21.15.89:0/2818004383 conn(0x55708f05d010 msgr2=0x55708f05d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 -- 172.21.15.89:0/2818004383 shutdown_connections 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 -- 172.21.15.89:0/2818004383 wait complete. 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 -- start start 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 Processor -- start 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 0x55708f06bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0622e0 0x55708f06c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f067370 0x55708f073140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55708f06a340 con 0x55708f067370 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55708ef4e290 con 0x55708f0622e0 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55708f06a4c0 con 0x55708f0619a0 2024-09-15T23:35:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9d20be740 10 client.0 _ll_register_callbacks cb 0x55708ef7f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9cc8c7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f067370 0x55708f073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9cc8c7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f067370 0x55708f073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40454/0 (socket says 172.21.15.89:40454) 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9cc8c7640 1 -- 172.21.15.89:0/1067116512 learned_addr learned my addr 172.21.15.89:0/1067116512 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.052+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0622e0 0x55708f06c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 0x55708f06bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9cc8c7640 1 -- 172.21.15.89:0/1067116512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 msgr2=0x55708f06bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 0x55708f06bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9cc8c7640 1 -- 172.21.15.89:0/1067116512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0622e0 msgr2=0x55708f06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0622e0 0x55708f06c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9cc8c7640 1 -- 172.21.15.89:0/1067116512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55708ecd8210 con 0x55708f067370 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f067370 0x55708f073140 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8005c90 tx=0x7fd9b800af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e007ad82b021593 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/1067116512 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b80110d0 con 0x55708f067370 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/1067116512 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd9b8012450 con 0x55708f067370 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9d20be740 1 -- 172.21.15.89:0/1067116512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9b80047a0 con 0x55708f067370 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/1067116512 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd9b8029770 con 0x55708f067370 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9d20be740 10 client.19383 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9d20be740 1 -- 172.21.15.89:0/1067116512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55708eda4620 con 0x55708f067370 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/1067116512 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9b80047a0 con 0x55708f067370 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/1067116512 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd9b800f540 con 0x55708f067370 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9d20be740 10 client.19383 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9d20be740 10 client.19383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9d20be740 20 client.19383 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9d20be740 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55708f077660 0x55708f079a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9d20be740 4 client.19383 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.053+0000 7fd9d20be740 1 -- 172.21.15.89:0/1067116512 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55708efd2a20 con 0x55708f077660 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.054+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55708f077660 0x55708f079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.054+0000 7fd9b0ff9640 10 client.19383 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.054+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55708f077660 0x55708f079a90 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3d441d1990d490 server_cookie=2d8595e20450ab52 in_seq=0 out_seq=0 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/1067116512 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55708efd2a20 con 0x55708f077660 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9b0ff9640 10 client.19383 handle_command_reply: tid=0 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 client.19383 shutdown 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 -- 172.21.15.89:0/1067116512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f067370 msgr2=0x55708f073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f067370 0x55708f073140 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8005c90 tx=0x7fd9b800af90 comp rx=0 tx=0).stop 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 -- 172.21.15.89:0/1067116512 shutdown_connections 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55708f077660 0x55708f079a90 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f067370 0x55708f073140 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9b0ff9640 0 client.19383 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0622e0 0x55708f06c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 --2- 172.21.15.89:0/1067116512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 0x55708f06bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 -- 172.21.15.89:0/1067116512 >> 172.21.15.89:0/1067116512 conn(0x55708f05d010 msgr2=0x55708f05e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 -- 172.21.15.89:0/1067116512 shutdown_connections 2024-09-15T23:35:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 1 -- 172.21.15.89:0/1067116512 wait complete. 2024-09-15T23:35:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.055+0000 7fd9d20be740 20 client.19383 trim_cache size 0 max 16384 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9d20be740 1 Processor -- start 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9d20be740 1 -- start start 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 0x55708f05c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f067370 0x55708f07e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9d20be740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c81aa890 con 0x55708f0622e0 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9d20be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c81ac3d0 con 0x55708f067370 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9d20be740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c81aaf60 con 0x55708f0619a0 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9cd0c8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9cd0c8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40462/0 (socket says 172.21.15.89:40462) 2024-09-15T23:35:38.072 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9cd0c8640 1 -- 172.21.15.89:0/2391855956 learned_addr learned my addr 172.21.15.89:0/2391855956 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.069+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/2391855956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f067370 0x55708f07e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/2391855956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 0x55708f05c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9cd0c8640 1 -- 172.21.15.89:0/2391855956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 msgr2=0x55708f05c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/2391855956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 0x55708f05c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9cd0c8640 1 -- 172.21.15.89:0/2391855956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f067370 msgr2=0x55708f07e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/2391855956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f067370 0x55708f07e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9cd0c8640 1 -- 172.21.15.89:0/2391855956 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55708f07a110 con 0x55708f0622e0 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/2391855956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 0x55708f05c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/2391855956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f0730b0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x55708f05de90 tx=0x7fd9c4004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9055701571407f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9b2ffd640 1 -- 172.21.15.89:0/2391855956 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4003d20 con 0x55708f0622e0 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9b2ffd640 1 -- 172.21.15.89:0/2391855956 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9c4005500 con 0x55708f0622e0 2024-09-15T23:35:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9b2ffd640 1 -- 172.21.15.89:0/2391855956 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c401d690 con 0x55708f0622e0 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9d20be740 1 -- 172.21.15.89:0/2391855956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 msgr2=0x55708f0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9d20be740 1 --2- 172.21.15.89:0/2391855956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f0730b0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x55708f05de90 tx=0x7fd9c4004410 comp rx=0 tx=0).stop 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9d20be740 1 -- 172.21.15.89:0/2391855956 shutdown_connections 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9d20be740 1 --2- 172.21.15.89:0/2391855956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f067370 0x55708f07e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9d20be740 1 --2- 172.21.15.89:0/2391855956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f0730b0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9d20be740 1 --2- 172.21.15.89:0/2391855956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f0619a0 0x55708f05c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.070+0000 7fd9d20be740 1 -- 172.21.15.89:0/2391855956 >> 172.21.15.89:0/2391855956 conn(0x55708f05d010 msgr2=0x55708f05b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 1 -- 172.21.15.89:0/2391855956 shutdown_connections 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 1 -- 172.21.15.89:0/2391855956 wait complete. 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 1 -- start start 2024-09-15T23:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 1 Processor -- start 2024-09-15T23:35:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0619a0 0x55708f070790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f067370 0x55708f071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c81abd00 con 0x55708f0622e0 2024-09-15T23:35:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c81ab630 con 0x55708f0619a0 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c81a8870 con 0x55708f067370 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9d20be740 10 client.0 _ll_register_callbacks cb 0x7fd9b800ab90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9cd0c8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9cd0c8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f070cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40472/0 (socket says 172.21.15.89:40472) 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9cd0c8640 1 -- 172.21.15.89:0/4221589241 learned_addr learned my addr 172.21.15.89:0/4221589241 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.071+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0619a0 0x55708f070790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f067370 0x55708f071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9cd0c8640 1 -- 172.21.15.89:0/4221589241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f067370 msgr2=0x55708f071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f067370 0x55708f071c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9cd0c8640 1 -- 172.21.15.89:0/4221589241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0619a0 msgr2=0x55708f070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0619a0 0x55708f070790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9cd0c8640 1 -- 172.21.15.89:0/4221589241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55708f0722a0 con 0x55708f0622e0 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9cd0c8640 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f070cd0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400a540 tx=0x7fd9c4005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766f078d8cf75921 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9b0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/4221589241 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4003b20 con 0x55708f0622e0 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9d20be740 1 -- 172.21.15.89:0/4221589241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9c4005500 con 0x55708f0622e0 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/4221589241 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd9c4003cc0 con 0x55708f0622e0 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/4221589241 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd9c401d430 con 0x55708f0622e0 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9d20be740 10 client.19389 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9d20be740 1 -- 172.21.15.89:0/4221589241 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55708f0e2370 con 0x55708f0622e0 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/4221589241 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9c4005500 con 0x55708f0622e0 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/4221589241 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd9c40115f0 con 0x55708f0622e0 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9d20be740 10 client.19389 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9d20be740 10 client.19389 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9d20be740 20 client.19389 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9d20be740 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55708f0e2bc0 0x55708f0e4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9d20be740 4 client.19389 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:35:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.072+0000 7fd9d20be740 1 -- 172.21.15.89:0/4221589241 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55708efccd70 con 0x55708f0e2bc0 2024-09-15T23:35:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.073+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55708f0e2bc0 0x55708f0e4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.073+0000 7fd9cc8c7640 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55708f0e2bc0 0x55708f0e4fe0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b7535b1ea5f4ec server_cookie=be9b1444f435fba6 in_seq=0 out_seq=0 2024-09-15T23:35:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.073+0000 7fd9b0ff9640 10 client.19389 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9b0ff9640 1 -- 172.21.15.89:0/4221589241 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55708efccd70 con 0x55708f0e2bc0 2024-09-15T23:35:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9b0ff9640 10 client.19389 handle_command_reply: tid=0 2024-09-15T23:35:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9d20be740 1 client.19389 shutdown 2024-09-15T23:35:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9d20be740 1 -- 172.21.15.89:0/4221589241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 msgr2=0x55708f070cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9d20be740 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f070cd0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400a540 tx=0x7fd9c4005250 comp rx=0 tx=0).stop 2024-09-15T23:35:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9d20be740 1 -- 172.21.15.89:0/4221589241 shutdown_connections 2024-09-15T23:35:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9d20be740 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55708f0e2bc0 0x55708f0e4fe0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9d20be740 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55708f067370 0x55708f071c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9b0ff9640 0 client.19389 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9d20be740 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55708f0622e0 0x55708f070cd0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9d20be740 1 --2- 172.21.15.89:0/4221589241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55708f0619a0 0x55708f070790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.087+0000 7fd9d20be740 1 -- 172.21.15.89:0/4221589241 >> 172.21.15.89:0/4221589241 conn(0x55708f05d010 msgr2=0x55708f05c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9d20be740 1 -- 172.21.15.89:0/4221589241 shutdown_connections 2024-09-15T23:35:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9d20be740 1 -- 172.21.15.89:0/4221589241 wait complete. 2024-09-15T23:35:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9d20be740 20 client.19389 trim_cache size 0 max 16384 2024-09-15T23:35:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:35:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:35:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:35:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:35:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 -- 172.21.15.89:0/2215083561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9b40a3370 msgr2=0x7fd9b40a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9b40a3370 0x7fd9b40a5830 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd9c003ffb0 tx=0x7fd9c0049000 comp rx=0 tx=0).stop 2024-09-15T23:35:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 -- 172.21.15.89:0/2215083561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8102600 msgr2=0x7fd9c819f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8102600 0x7fd9c819f320 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fd9c00098d0 tx=0x7fd9c002f840 comp rx=0 tx=0).stop 2024-09-15T23:35:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 -- 172.21.15.89:0/2215083561 shutdown_connections 2024-09-15T23:35:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9b40a3370 0x7fd9b40a5830 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd9c8102600 0x7fd9c819f320 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9c8101c40 0x7fd9c81982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 --2- 172.21.15.89:0/2215083561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd9c8100a50 0x7fd9c8197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 -- 172.21.15.89:0/2215083561 >> 172.21.15.89:0/2215083561 conn(0x7fd9c80fc1b0 msgr2=0x7fd9c8104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 -- 172.21.15.89:0/2215083561 shutdown_connections 2024-09-15T23:35:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:38.088+0000 7fd9cf32c640 1 -- 172.21.15.89:0/2215083561 wait complete. 2024-09-15T23:35:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:38 smithi120 ceph-mon[35250]: pgmap v698: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-09-15T23:35:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:38 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:35:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:38 smithi174 ceph-mon[35586]: pgmap v698: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-09-15T23:35:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:38 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:35:38.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:38 smithi089 ceph-mon[30933]: pgmap v698: 129 pgs: 129 active+clean; 1.8 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-09-15T23:35:38.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:38 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:35:40.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:40 smithi089 ceph-mon[30933]: pgmap v699: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-09-15T23:35:40.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:40 smithi120 ceph-mon[35250]: pgmap v699: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-09-15T23:35:40.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:40 smithi174 ceph-mon[35586]: pgmap v699: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-09-15T23:35:42.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:42 smithi089 ceph-mon[30933]: pgmap v700: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-09-15T23:35:42.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:42 smithi120 ceph-mon[35250]: pgmap v700: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-09-15T23:35:42.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:42 smithi174 ceph-mon[35586]: pgmap v700: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-09-15T23:35:44.112 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:35:44.372 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:44.492 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:44 smithi089 ceph-mon[30933]: pgmap v701: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.9 KiB/s wr, 14 op/s 2024-09-15T23:35:44.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:44 smithi120 ceph-mon[35250]: pgmap v701: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.9 KiB/s wr, 14 op/s 2024-09-15T23:35:44.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:44 smithi174 ceph-mon[35586]: pgmap v701: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.9 KiB/s wr, 14 op/s 2024-09-15T23:35:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.853+0000 7f714b7c5640 1 -- 172.21.15.89:0/66104495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7144100480 msgr2=0x7f7144100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:44.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.853+0000 7f714b7c5640 1 --2- 172.21.15.89:0/66104495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7144100480 0x7f7144100860 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f7134009ca0 tx=0x7f713402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.853+0000 7f714b7c5640 1 -- 172.21.15.89:0/66104495 shutdown_connections 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.853+0000 7f714b7c5640 1 --2- 172.21.15.89:0/66104495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f71441055b0 0x7f714410d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.853+0000 7f714b7c5640 1 --2- 172.21.15.89:0/66104495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7144100e30 0x7f7144104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.853+0000 7f714b7c5640 1 --2- 172.21.15.89:0/66104495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7144100480 0x7f7144100860 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.853+0000 7f714b7c5640 1 -- 172.21.15.89:0/66104495 >> 172.21.15.89:0/66104495 conn(0x7f7144077210 msgr2=0x7f71440fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.853+0000 7f714b7c5640 1 -- 172.21.15.89:0/66104495 shutdown_connections 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.853+0000 7f714b7c5640 1 -- 172.21.15.89:0/66104495 wait complete. 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f714b7c5640 1 Processor -- start 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f714b7c5640 1 -- start start 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f714b7c5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7144100480 0x7f7144198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f714b7c5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7144100e30 0x7f7144198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f714b7c5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441055b0 0x7f714419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:44.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f714b7c5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f714410fd90 con 0x7f71441055b0 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f714b7c5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f714410fc10 con 0x7f7144100480 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f714b7c5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f714410ff10 con 0x7f7144100e30 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f7149d62640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441055b0 0x7f714419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f7149d62640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441055b0 0x7f714419cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40496/0 (socket says 172.21.15.89:40496) 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f7149d62640 1 -- 172.21.15.89:0/3670872109 learned_addr learned my addr 172.21.15.89:0/3670872109 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.854+0000 7f7149561640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7144100480 0x7f7144198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f7148d60640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7144100e30 0x7f7144198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f7149d62640 1 -- 172.21.15.89:0/3670872109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7144100e30 msgr2=0x7f7144198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f7149d62640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7144100e30 0x7f7144198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f7149d62640 1 -- 172.21.15.89:0/3670872109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7144100480 msgr2=0x7f7144198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f7149d62640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7144100480 0x7f7144198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f7149d62640 1 -- 172.21.15.89:0/3670872109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714419d400 con 0x7f71441055b0 2024-09-15T23:35:44.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f7149d62640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441055b0 0x7f714419cd20 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f713c00c970 tx=0x7f713c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f71427fc640 1 -- 172.21.15.89:0/3670872109 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c007bb0 con 0x7f71441055b0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f71427fc640 1 -- 172.21.15.89:0/3670872109 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f713c007d50 con 0x7f71441055b0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f71427fc640 1 -- 172.21.15.89:0/3670872109 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c018800 con 0x7f71441055b0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f714b7c5640 1 -- 172.21.15.89:0/3670872109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f714419d6f0 con 0x7f71441055b0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.855+0000 7f714b7c5640 1 -- 172.21.15.89:0/3670872109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71441a4fd0 con 0x7f71441055b0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.856+0000 7f714b7c5640 1 -- 172.21.15.89:0/3670872109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7144100860 con 0x7f71441055b0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.857+0000 7f71427fc640 1 -- 172.21.15.89:0/3670872109 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f713c00fa50 con 0x7f71441055b0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.858+0000 7f71427fc640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f712c0a3240 0x7f712c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.858+0000 7f7149561640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f712c0a3240 0x7f712c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.858+0000 7f71427fc640 1 -- 172.21.15.89:0/3670872109 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f713c0cff70 con 0x7f71441055b0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.858+0000 7f7149561640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f712c0a3240 0x7f712c0a5700 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7134000f80 tx=0x7f71340093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:44.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:44.861+0000 7f71427fc640 1 -- 172.21.15.89:0/3670872109 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f713c014030 con 0x7f71441055b0 2024-09-15T23:35:45.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.060+0000 7f714b7c5640 1 -- 172.21.15.89:0/3670872109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71441997b0 con 0x7f71441055b0 2024-09-15T23:35:45.062 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.061+0000 7f71427fc640 1 -- 172.21.15.89:0/3670872109 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f713c09c8a0 con 0x7f71441055b0 2024-09-15T23:35:45.063 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:35:45.063 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:35:45.063 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:35:45.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 -- 172.21.15.89:0/3670872109 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f712c0a3240 msgr2=0x7f712c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:45.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f712c0a3240 0x7f712c0a5700 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7134000f80 tx=0x7f71340093f0 comp rx=0 tx=0).stop 2024-09-15T23:35:45.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 -- 172.21.15.89:0/3670872109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441055b0 msgr2=0x7f714419cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:45.065 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441055b0 0x7f714419cd20 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f713c00c970 tx=0x7f713c00ce30 comp rx=0 tx=0).stop 2024-09-15T23:35:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 -- 172.21.15.89:0/3670872109 shutdown_connections 2024-09-15T23:35:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f712c0a3240 0x7f712c0a5700 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71441055b0 0x7f714419cd20 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7144100e30 0x7f7144198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 --2- 172.21.15.89:0/3670872109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7144100480 0x7f7144198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 -- 172.21.15.89:0/3670872109 >> 172.21.15.89:0/3670872109 conn(0x7f7144077210 msgr2=0x7f71440fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 -- 172.21.15.89:0/3670872109 shutdown_connections 2024-09-15T23:35:45.066 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:45.063+0000 7f714b7c5640 1 -- 172.21.15.89:0/3670872109 wait complete. 2024-09-15T23:35:45.525 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:45 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3670872109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:45.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:45 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3670872109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:45.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:45 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3670872109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:45.811 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id ac2b42c4 --release --await' 2024-09-15T23:35:45.811 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id ac2b42c4 --release --await 2024-09-15T23:35:46.057 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:46.470 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:46 smithi089 ceph-mon[30933]: pgmap v702: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.9 KiB/s wr, 13 op/s 2024-09-15T23:35:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.556+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/3958934338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b9410c160 msgr2=0x7f5b9410e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.556+0000 7f5b9ad4e640 1 --2- 172.21.15.89:0/3958934338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b9410c160 0x7f5b9410e600 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009ca0 tx=0x7f5b8c02a950 comp rx=0 tx=0).stop 2024-09-15T23:35:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.556+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/3958934338 shutdown_connections 2024-09-15T23:35:46.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.556+0000 7f5b9ad4e640 1 --2- 172.21.15.89:0/3958934338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b9410c160 0x7f5b9410e600 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.556+0000 7f5b9ad4e640 1 --2- 172.21.15.89:0/3958934338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b941036a0 0x7f5b9410bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.556+0000 7f5b9ad4e640 1 --2- 172.21.15.89:0/3958934338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b94102d80 0x7f5b94103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.556+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/3958934338 >> 172.21.15.89:0/3958934338 conn(0x7f5b940fc7a0 msgr2=0x7f5b940febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.556+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/3958934338 shutdown_connections 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.556+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/3958934338 wait complete. 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.556+0000 7f5b9ad4e640 1 Processor -- start 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b9ad4e640 1 -- start start 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b9ad4e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b94102d80 0x7f5b94198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b9ad4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b941036a0 0x7f5b941988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b9ad4e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b9410c160 0x7f5b9419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b9ad4e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5b94111f20 con 0x7f5b9410c160 2024-09-15T23:35:46.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b9ad4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5b94111da0 con 0x7f5b941036a0 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b9ad4e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941120a0 con 0x7f5b94102d80 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b992eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b9410c160 0x7f5b9419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b992eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b9410c160 0x7f5b9419cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45948/0 (socket says 172.21.15.89:45948) 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b992eb640 1 -- 172.21.15.89:0/2519435907 learned_addr learned my addr 172.21.15.89:0/2519435907 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b98aea640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b94102d80 0x7f5b94198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b93fff640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b941036a0 0x7f5b941988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b992eb640 1 -- 172.21.15.89:0/2519435907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b94102d80 msgr2=0x7f5b94198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b992eb640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b94102d80 0x7f5b94198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b992eb640 1 -- 172.21.15.89:0/2519435907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b941036a0 msgr2=0x7f5b941988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b992eb640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b941036a0 0x7f5b941988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.557+0000 7f5b992eb640 1 -- 172.21.15.89:0/2519435907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9419d330 con 0x7f5b9410c160 2024-09-15T23:35:46.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.558+0000 7f5b992eb640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b9410c160 0x7f5b9419cc50 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c006010 tx=0x7f5b8c02f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.558+0000 7f5b91ffb640 1 -- 172.21.15.89:0/2519435907 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c034040 con 0x7f5b9410c160 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.558+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/2519435907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9419d5c0 con 0x7f5b9410c160 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.558+0000 7f5b91ffb640 1 -- 172.21.15.89:0/2519435907 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b8c003e00 con 0x7f5b9410c160 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.558+0000 7f5b91ffb640 1 -- 172.21.15.89:0/2519435907 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c0456e0 con 0x7f5b9410c160 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.558+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/2519435907 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b941a4f00 con 0x7f5b9410c160 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.560+0000 7f5b91ffb640 1 -- 172.21.15.89:0/2519435907 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5b8c037020 con 0x7f5b9410c160 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b9d127740 1 Processor -- start 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b9d127740 1 -- start start 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173e1980 0x5582173ebd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173ec240 0x5582173ee630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b9d127740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558217346620 con 0x5582173e1060 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b9d127740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5582172ede50 con 0x5582173e1980 2024-09-15T23:35:46.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b9d127740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5582173425e0 con 0x5582173ec240 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b93fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b93fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45950/0 (socket says 172.21.15.89:45950) 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b93fff640 1 -- 172.21.15.89:0/2975406539 learned_addr learned my addr 172.21.15.89:0/2975406539 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b91ffb640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5b840a3240 0x7f5b840a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b98aea640 1 --2- 172.21.15.89:0/2975406539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173e1980 0x5582173ebd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b93fff640 1 --2- 172.21.15.89:0/2975406539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173ec240 0x5582173ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b992eb640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5b840a3240 0x7f5b840a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b93fff640 1 -- 172.21.15.89:0/2975406539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173ec240 msgr2=0x5582173ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b93fff640 1 --2- 172.21.15.89:0/2975406539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173ec240 0x5582173ee630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b93fff640 1 -- 172.21.15.89:0/2975406539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173e1980 msgr2=0x5582173ebd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b93fff640 1 --2- 172.21.15.89:0/2975406539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173e1980 0x5582173ebd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.561+0000 7f5b93fff640 1 -- 172.21.15.89:0/2975406539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582172ded40 con 0x5582173e1060 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.562+0000 7f5b91ffb640 1 -- 172.21.15.89:0/2519435907 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5b8c0f0db0 con 0x7f5b9410c160 2024-09-15T23:35:46.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.562+0000 7f5b93fff640 1 --2- 172.21.15.89:0/2975406539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e1440 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f5b800113e0 tx=0x7f5b8000a6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951f0831fc2c58f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.562+0000 7f5b7effd640 1 -- 172.21.15.89:0/2975406539 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80005020 con 0x5582173e1060 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.562+0000 7f5b7effd640 1 -- 172.21.15.89:0/2975406539 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b800047a0 con 0x5582173e1060 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.562+0000 7f5b7effd640 1 -- 172.21.15.89:0/2975406539 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8000f690 con 0x5582173e1060 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.562+0000 7f5b992eb640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5b840a3240 0x7f5b840a5700 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c0b99b0 tx=0x7f5b8c08fe60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 -- 172.21.15.89:0/2975406539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 msgr2=0x5582173e1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 --2- 172.21.15.89:0/2975406539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e1440 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f5b800113e0 tx=0x7f5b8000a6e0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 -- 172.21.15.89:0/2975406539 shutdown_connections 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 --2- 172.21.15.89:0/2975406539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173ec240 0x5582173ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 --2- 172.21.15.89:0/2975406539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173e1980 0x5582173ebd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 --2- 172.21.15.89:0/2975406539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e1440 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 -- 172.21.15.89:0/2975406539 >> 172.21.15.89:0/2975406539 conn(0x5582173dd5f0 msgr2=0x5582173dd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 -- 172.21.15.89:0/2975406539 shutdown_connections 2024-09-15T23:35:46.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 -- 172.21.15.89:0/2975406539 wait complete. 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 -- start start 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.563+0000 7f5b9d127740 1 Processor -- start 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x5582173e9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173ec240 0x5582173e9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b9d127740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5582173eee10 con 0x5582173e1060 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b9d127740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5582172cef10 con 0x5582173ec240 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b9d127740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5582173eef90 con 0x5582173e1980 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b9d127740 10 client.0 _ll_register_callbacks cb 0x5582172f2db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b93fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b93fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45952/0 (socket says 172.21.15.89:45952) 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b93fff640 1 -- 172.21.15.89:0/2637590674 learned_addr learned my addr 172.21.15.89:0/2637590674 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b98aea640 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x5582173e9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b93fff640 1 -- 172.21.15.89:0/2637590674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 msgr2=0x5582173e9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b93fff640 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x5582173e9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b93fff640 1 -- 172.21.15.89:0/2637590674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173ec240 msgr2=0x5582173e9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b93fff640 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173ec240 0x5582173e9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b93fff640 1 -- 172.21.15.89:0/2637590674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558217055580 con 0x5582173e1060 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b93fff640 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e90f0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f5b80004f10 tx=0x7f5b80005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1923beed8f091497 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b73fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b73fff640 1 -- 172.21.15.89:0/2637590674 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b800054f0 con 0x5582173e1060 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b9d127740 1 -- 172.21.15.89:0/2637590674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b800047a0 con 0x5582173e1060 2024-09-15T23:35:46.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b73fff640 1 -- 172.21.15.89:0/2637590674 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5b8000f450 con 0x5582173e1060 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b73fff640 1 -- 172.21.15.89:0/2637590674 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5b8000f650 con 0x5582173e1060 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b9d127740 10 client.19407 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.564+0000 7f5b9d127740 1 -- 172.21.15.89:0/2637590674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55821711f960 con 0x5582173e1060 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.565+0000 7f5b73fff640 1 -- 172.21.15.89:0/2637590674 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b800047a0 con 0x5582173e1060 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.565+0000 7f5b73fff640 1 -- 172.21.15.89:0/2637590674 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5b800129a0 con 0x5582173e1060 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.565+0000 7f5b9d127740 10 client.19407 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.565+0000 7f5b9d127740 10 client.19407 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.565+0000 7f5b9d127740 20 client.19407 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.565+0000 7f5b9d127740 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5582173fbc70 0x5582173fe060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.565+0000 7f5b9d127740 4 client.19407 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.565+0000 7f5b9d127740 1 -- 172.21.15.89:0/2637590674 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558217353b90 con 0x5582173fbc70 2024-09-15T23:35:46.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.565+0000 7f5b98aea640 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5582173fbc70 0x5582173fe060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.566+0000 7f5b73fff640 10 client.19407 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.566+0000 7f5b98aea640 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5582173fbc70 0x5582173fe060 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=899ffdc7a71ecae1 server_cookie=f34a102bb9b2fc69 in_seq=0 out_seq=0 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.566+0000 7f5b73fff640 1 -- 172.21.15.89:0/2637590674 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558217353b90 con 0x5582173fbc70 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.566+0000 7f5b73fff640 10 client.19407 handle_command_reply: tid=0 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.566+0000 7f5b9d127740 1 client.19407 shutdown 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.566+0000 7f5b9d127740 1 -- 172.21.15.89:0/2637590674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 msgr2=0x5582173e90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b9d127740 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e90f0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f5b80004f10 tx=0x7f5b80005730 comp rx=0 tx=0).stop 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b9d127740 1 -- 172.21.15.89:0/2637590674 shutdown_connections 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b9d127740 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5582173fbc70 0x5582173fe060 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b9d127740 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173ec240 0x5582173e9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b9d127740 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x5582173e9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b9d127740 1 --2- 172.21.15.89:0/2637590674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173e90f0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b73fff640 0 client.19407 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:46.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b9d127740 1 -- 172.21.15.89:0/2637590674 >> 172.21.15.89:0/2637590674 conn(0x5582173dd5f0 msgr2=0x5582173e3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:46.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b9d127740 1 -- 172.21.15.89:0/2637590674 shutdown_connections 2024-09-15T23:35:46.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b9d127740 1 -- 172.21.15.89:0/2637590674 wait complete. 2024-09-15T23:35:46.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.567+0000 7f5b9d127740 20 client.19407 trim_cache size 0 max 16384 2024-09-15T23:35:46.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.583+0000 7f5b9d127740 1 Processor -- start 2024-09-15T23:35:46.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.583+0000 7f5b9d127740 1 -- start start 2024-09-15T23:35:46.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173e1060 0x5582173dc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x558217403690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173ec240 0x558217403bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b9d127740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941aee10 con 0x5582173ec240 2024-09-15T23:35:46.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b9d127740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941b0950 con 0x5582173e1060 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b9d127740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941af4e0 con 0x5582173e1980 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b93fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173ec240 0x558217403bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b93fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173ec240 0x558217403bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45964/0 (socket says 172.21.15.89:45964) 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b93fff640 1 -- 172.21.15.89:0/1052979164 learned_addr learned my addr 172.21.15.89:0/1052979164 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b98aea640 1 --2- 172.21.15.89:0/1052979164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x558217403690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b93fff640 1 --2- 172.21.15.89:0/1052979164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173e1060 0x5582173dc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b93fff640 1 -- 172.21.15.89:0/1052979164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 msgr2=0x558217403690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b93fff640 1 --2- 172.21.15.89:0/1052979164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x558217403690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b93fff640 1 -- 172.21.15.89:0/1052979164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173e1060 msgr2=0x5582173dc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b93fff640 1 --2- 172.21.15.89:0/1052979164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173e1060 0x5582173dc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b93fff640 1 -- 172.21.15.89:0/1052979164 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582173de280 con 0x5582173ec240 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b93fff640 1 --2- 172.21.15.89:0/1052979164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173ec240 0x558217403bd0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f5b80005940 tx=0x7f5b80004fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9900544567c113 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b7e7fc640 1 -- 172.21.15.89:0/1052979164 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8000f450 con 0x5582173ec240 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b7e7fc640 1 -- 172.21.15.89:0/1052979164 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b80023030 con 0x5582173ec240 2024-09-15T23:35:46.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.584+0000 7f5b7e7fc640 1 -- 172.21.15.89:0/1052979164 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8000f7f0 con 0x5582173ec240 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.585+0000 7f5b9d127740 1 -- 172.21.15.89:0/1052979164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173ec240 msgr2=0x558217403bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.585+0000 7f5b9d127740 1 --2- 172.21.15.89:0/1052979164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173ec240 0x558217403bd0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f5b80005940 tx=0x7f5b80004fb0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.585+0000 7f5b9d127740 1 -- 172.21.15.89:0/1052979164 shutdown_connections 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.585+0000 7f5b9d127740 1 --2- 172.21.15.89:0/1052979164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173ec240 0x558217403bd0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.585+0000 7f5b9d127740 1 --2- 172.21.15.89:0/1052979164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x558217403690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.585+0000 7f5b9d127740 1 --2- 172.21.15.89:0/1052979164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173e1060 0x5582173dc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.585+0000 7f5b9d127740 1 -- 172.21.15.89:0/1052979164 >> 172.21.15.89:0/1052979164 conn(0x5582173dd5f0 msgr2=0x5582173dbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.585+0000 7f5b9d127740 1 -- 172.21.15.89:0/1052979164 shutdown_connections 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.585+0000 7f5b9d127740 1 -- 172.21.15.89:0/1052979164 wait complete. 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.585+0000 7f5b9d127740 1 -- start start 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b9d127740 1 Processor -- start 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173f4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x5582173f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b9d127740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173ec240 0x5582173f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b9d127740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941b0280 con 0x5582173e1060 2024-09-15T23:35:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b9d127740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941afbb0 con 0x5582173ec240 2024-09-15T23:35:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b9d127740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5b941acdf0 con 0x5582173e1980 2024-09-15T23:35:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b9d127740 10 client.0 _ll_register_callbacks cb 0x7f5b80004df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b93fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173f4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b93fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173f4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45980/0 (socket says 172.21.15.89:45980) 2024-09-15T23:35:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b93fff640 1 -- 172.21.15.89:0/3640387079 learned_addr learned my addr 172.21.15.89:0/3640387079 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b98aea640 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x5582173f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b93fff640 1 -- 172.21.15.89:0/3640387079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 msgr2=0x5582173f0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:46.596 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b93fff640 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x5582173f0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b93fff640 1 -- 172.21.15.89:0/3640387079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173ec240 msgr2=0x5582173f11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b93fff640 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173ec240 0x5582173f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.586+0000 7f5b93fff640 1 -- 172.21.15.89:0/3640387079 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582173f19a0 con 0x5582173e1060 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b93fff640 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173f4940 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f5b80014010 tx=0x7f5b8000a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf501ec08d20807 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b73fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b73fff640 1 -- 172.21.15.89:0/3640387079 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8000f450 con 0x5582173e1060 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b73fff640 1 -- 172.21.15.89:0/3640387079 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5b80004660 con 0x5582173e1060 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b9d127740 1 -- 172.21.15.89:0/3640387079 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b80023030 con 0x5582173e1060 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b73fff640 1 -- 172.21.15.89:0/3640387079 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5b8002a320 con 0x5582173e1060 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b9d127740 10 client.19413 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b9d127740 1 -- 172.21.15.89:0/3640387079 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55821746bad0 con 0x5582173e1060 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b73fff640 1 -- 172.21.15.89:0/3640387079 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b80023030 con 0x5582173e1060 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b73fff640 1 -- 172.21.15.89:0/3640387079 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5b8003b030 con 0x5582173e1060 2024-09-15T23:35:46.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b9d127740 10 client.19413 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b9d127740 10 client.19413 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b9d127740 20 client.19413 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b9d127740 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55821746c080 0x55821746e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b9d127740 4 client.19413 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "ac2b42c4", "release": true, "await": true}] 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.587+0000 7f5b9d127740 1 -- 172.21.15.89:0/3640387079 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "ac2b42c4", "release": true, "await": true}) -- 0x55821734e0a0 con 0x55821746c080 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.588+0000 7f5b98aea640 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55821746c080 0x55821746e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.588+0000 7f5b73fff640 10 client.19413 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.588+0000 7f5b98aea640 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55821746c080 0x55821746e470 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54e8c33ea732c942 server_cookie=57e02484997bbc89 in_seq=0 out_seq=0 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b73fff640 1 -- 172.21.15.89:0/3640387079 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55821734e0a0 con 0x55821746c080 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b73fff640 10 client.19413 handle_command_reply: tid=0 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 client.19413 shutdown 2024-09-15T23:35:46.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 -- 172.21.15.89:0/3640387079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 msgr2=0x5582173f4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173f4940 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f5b80014010 tx=0x7f5b8000a2d0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 -- 172.21.15.89:0/3640387079 shutdown_connections 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55821746c080 0x55821746e470 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5582173ec240 0x5582173f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5582173e1980 0x5582173f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 --2- 172.21.15.89:0/3640387079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582173e1060 0x5582173f4940 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 -- 172.21.15.89:0/3640387079 >> 172.21.15.89:0/3640387079 conn(0x5582173dd5f0 msgr2=0x5582173dc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b73fff640 0 client.19413 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 -- 172.21.15.89:0/3640387079 shutdown_connections 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 1 -- 172.21.15.89:0/3640387079 wait complete. 2024-09-15T23:35:46.599 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.591+0000 7f5b9d127740 20 client.19413 trim_cache size 0 max 16384 2024-09-15T23:35:46.600 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:35:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:35:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:35:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 40, 2024-09-15T23:35:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:35:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "ac2b42c4": { 2024-09-15T23:35:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 40, 2024-09-15T23:35:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 141.8, 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:35:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:46.602 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:35:46.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/2519435907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5b840a3240 msgr2=0x7f5b840a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5b840a3240 0x7f5b840a5700 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c0b99b0 tx=0x7f5b8c08fe60 comp rx=0 tx=0).stop 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/2519435907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b9410c160 msgr2=0x7f5b9419cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b9410c160 0x7f5b9419cc50 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c006010 tx=0x7f5b8c02f710 comp rx=0 tx=0).stop 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/2519435907 shutdown_connections 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5b840a3240 0x7f5b840a5700 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b9410c160 0x7f5b9419cc50 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b941036a0 0x7f5b941988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 --2- 172.21.15.89:0/2519435907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5b94102d80 0x7f5b94198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/2519435907 >> 172.21.15.89:0/2519435907 conn(0x7f5b940fc7a0 msgr2=0x7f5b940fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/2519435907 shutdown_connections 2024-09-15T23:35:46.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:46.592+0000 7f5b9ad4e640 1 -- 172.21.15.89:0/2519435907 wait complete. 2024-09-15T23:35:46.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:46 smithi120 ceph-mon[35250]: pgmap v702: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.9 KiB/s wr, 13 op/s 2024-09-15T23:35:46.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:46 smithi174 ceph-mon[35586]: pgmap v702: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.9 KiB/s wr, 13 op/s 2024-09-15T23:35:47.389 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'ac2b42c4', total seconds elapsed: 25.1 2024-09-15T23:35:47.389 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.85 2024-09-15T23:35:47.390 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:35:47.638 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:48.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.134+0000 7f43eb414640 1 -- 172.21.15.89:0/3435225708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43e4102ef0 msgr2=0x7f43e410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.134+0000 7f43eb414640 1 --2- 172.21.15.89:0/3435225708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43e4102ef0 0x7f43e410da40 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f43dc009ca0 tx=0x7f43dc02a950 comp rx=0 tx=0).stop 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.134+0000 7f43eb414640 1 -- 172.21.15.89:0/3435225708 shutdown_connections 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.134+0000 7f43eb414640 1 --2- 172.21.15.89:0/3435225708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43e4102ef0 0x7f43e410da40 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.134+0000 7f43eb414640 1 --2- 172.21.15.89:0/3435225708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e4102570 0x7f43e41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.134+0000 7f43eb414640 1 --2- 172.21.15.89:0/3435225708 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4100e20 0x7f43e4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.134+0000 7f43eb414640 1 -- 172.21.15.89:0/3435225708 >> 172.21.15.89:0/3435225708 conn(0x7f43e40fc7d0 msgr2=0x7f43e40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.134+0000 7f43eb414640 1 -- 172.21.15.89:0/3435225708 shutdown_connections 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.134+0000 7f43eb414640 1 -- 172.21.15.89:0/3435225708 wait complete. 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43eb414640 1 Processor -- start 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43eb414640 1 -- start start 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43eb414640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43e4100e20 0x7f43e41985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43eb414640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4102570 0x7f43e4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:48.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43eb414640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e4102ef0 0x7f43e419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43eb414640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f43e4112120 con 0x7f43e4100e20 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43eb414640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f43e4111fa0 con 0x7f43e4102ef0 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43eb414640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43e41122a0 con 0x7f43e4102570 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43e91b0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43e4100e20 0x7f43e41985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43e91b0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43e4100e20 0x7f43e41985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46000/0 (socket says 172.21.15.89:46000) 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43e91b0640 1 -- 172.21.15.89:0/1228191652 learned_addr learned my addr 172.21.15.89:0/1228191652 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43e99b1640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e4102ef0 0x7f43e419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.135+0000 7f43e89af640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4102570 0x7f43e4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43e91b0640 1 -- 172.21.15.89:0/1228191652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4102570 msgr2=0x7f43e4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43e91b0640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4102570 0x7f43e4198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43e91b0640 1 -- 172.21.15.89:0/1228191652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e4102ef0 msgr2=0x7f43e419ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43e91b0640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e4102ef0 0x7f43e419ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:48.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43e91b0640 1 -- 172.21.15.89:0/1228191652 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e419d590 con 0x7f43e4100e20 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43e89af640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4102570 0x7f43e4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43e91b0640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43e4100e20 0x7f43e41985e0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f43e000b4a0 tx=0x7f43e000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43da7fc640 1 -- 172.21.15.89:0/1228191652 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0014070 con 0x7f43e4100e20 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43da7fc640 1 -- 172.21.15.89:0/1228191652 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43e0010af0 con 0x7f43e4100e20 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43eb414640 1 -- 172.21.15.89:0/1228191652 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e419d880 con 0x7f43e4100e20 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43da7fc640 1 -- 172.21.15.89:0/1228191652 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0017d10 con 0x7f43e4100e20 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.136+0000 7f43eb414640 1 -- 172.21.15.89:0/1228191652 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e41a5160 con 0x7f43e4100e20 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.137+0000 7f43eb414640 1 -- 172.21.15.89:0/1228191652 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e4101280 con 0x7f43e4100e20 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.138+0000 7f43da7fc640 1 -- 172.21.15.89:0/1228191652 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f43e0017070 con 0x7f43e4100e20 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.139+0000 7f43da7fc640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43d00a32c0 0x7f43d00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.139+0000 7f43e89af640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43d00a32c0 0x7f43d00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.139+0000 7f43da7fc640 1 -- 172.21.15.89:0/1228191652 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f43e00cf420 con 0x7f43e4100e20 2024-09-15T23:35:48.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.139+0000 7f43e89af640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43d00a32c0 0x7f43d00a5780 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f43e40ffe50 tx=0x7f43d4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:48.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.142+0000 7f43da7fc640 1 -- 172.21.15.89:0/1228191652 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f43e009bd90 con 0x7f43e4100e20 2024-09-15T23:35:48.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:48 smithi089 ceph-mon[30933]: pgmap v703: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.0 KiB/s wr, 16 op/s 2024-09-15T23:35:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.342+0000 7f43eb414640 1 -- 172.21.15.89:0/1228191652 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43e4109090 con 0x7f43e4100e20 2024-09-15T23:35:48.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.343+0000 7f43da7fc640 1 -- 172.21.15.89:0/1228191652 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f43e009b690 con 0x7f43e4100e20 2024-09-15T23:35:48.345 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:35:48.345 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:35:48.345 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:35:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 -- 172.21.15.89:0/1228191652 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43d00a32c0 msgr2=0x7f43d00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43d00a32c0 0x7f43d00a5780 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f43e40ffe50 tx=0x7f43d4008040 comp rx=0 tx=0).stop 2024-09-15T23:35:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 -- 172.21.15.89:0/1228191652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43e4100e20 msgr2=0x7f43e41985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43e4100e20 0x7f43e41985e0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f43e000b4a0 tx=0x7f43e000b960 comp rx=0 tx=0).stop 2024-09-15T23:35:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 -- 172.21.15.89:0/1228191652 shutdown_connections 2024-09-15T23:35:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43d00a32c0 0x7f43d00a5780 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43e4102ef0 0x7f43e419ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:48.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43e4102570 0x7f43e4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 --2- 172.21.15.89:0/1228191652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43e4100e20 0x7f43e41985e0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 -- 172.21.15.89:0/1228191652 >> 172.21.15.89:0/1228191652 conn(0x7f43e40fc7d0 msgr2=0x7f43e40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 -- 172.21.15.89:0/1228191652 shutdown_connections 2024-09-15T23:35:48.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:48.346+0000 7f43eb414640 1 -- 172.21.15.89:0/1228191652 wait complete. 2024-09-15T23:35:48.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:48 smithi120 ceph-mon[35250]: pgmap v703: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.0 KiB/s wr, 16 op/s 2024-09-15T23:35:48.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:48 smithi174 ceph-mon[35586]: pgmap v703: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.0 KiB/s wr, 16 op/s 2024-09-15T23:35:49.065 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0' 2024-09-15T23:35:49.066 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0 2024-09-15T23:35:49.311 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:35:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:49 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1228191652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:49.624 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:49 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1228191652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:49.637 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:49 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1228191652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:35:49.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 -- 172.21.15.89:0/2249881080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba9010cf30 msgr2=0x7fba90119410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 --2- 172.21.15.89:0/2249881080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba9010cf30 0x7fba90119410 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fba88009ca0 tx=0x7fba8802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 -- 172.21.15.89:0/2249881080 shutdown_connections 2024-09-15T23:35:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 --2- 172.21.15.89:0/2249881080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba9010cf30 0x7fba90119410 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 --2- 172.21.15.89:0/2249881080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba9010c590 0x7fba9010c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 --2- 172.21.15.89:0/2249881080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba90112640 0x7fba90112a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 -- 172.21.15.89:0/2249881080 >> 172.21.15.89:0/2249881080 conn(0x7fba9006cdc0 msgr2=0x7fba9006d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 -- 172.21.15.89:0/2249881080 shutdown_connections 2024-09-15T23:35:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 -- 172.21.15.89:0/2249881080 wait complete. 2024-09-15T23:35:49.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 Processor -- start 2024-09-15T23:35:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.814+0000 7fba96462640 1 -- start start 2024-09-15T23:35:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba96462640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba9010c590 0x7fba901a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba96462640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba9010cf30 0x7fba901a3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba96462640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba90112640 0x7fba901a7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba96462640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fba9011c9a0 con 0x7fba9010c590 2024-09-15T23:35:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba96462640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fba9011c820 con 0x7fba9010cf30 2024-09-15T23:35:49.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba96462640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fba9011cb20 con 0x7fba90112640 2024-09-15T23:35:49.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba9010c590 0x7fba901a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba9010c590 0x7fba901a2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46016/0 (socket says 172.21.15.89:46016) 2024-09-15T23:35:49.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8ffff640 1 -- 172.21.15.89:0/187031583 learned_addr learned my addr 172.21.15.89:0/187031583 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:49.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba949ff640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba90112640 0x7fba901a7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8ffff640 1 -- 172.21.15.89:0/187031583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba90112640 msgr2=0x7fba901a7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:49.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8ffff640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba90112640 0x7fba901a7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8ffff640 1 -- 172.21.15.89:0/187031583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba9010cf30 msgr2=0x7fba901a3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:49.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba9010cf30 0x7fba901a3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8ffff640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba9010cf30 0x7fba901a3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8ffff640 1 -- 172.21.15.89:0/187031583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba901a7cf0 con 0x7fba9010c590 2024-09-15T23:35:49.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba9010cf30 0x7fba901a3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:35:49.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8ffff640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba9010c590 0x7fba901a2d40 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fba8000b4a0 tx=0x7fba8000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:49.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8d7fa640 1 -- 172.21.15.89:0/187031583 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80014070 con 0x7fba9010c590 2024-09-15T23:35:49.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8d7fa640 1 -- 172.21.15.89:0/187031583 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba80010af0 con 0x7fba9010c590 2024-09-15T23:35:49.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba8d7fa640 1 -- 172.21.15.89:0/187031583 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80017d10 con 0x7fba9010c590 2024-09-15T23:35:49.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.815+0000 7fba96462640 1 -- 172.21.15.89:0/187031583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba901a7fe0 con 0x7fba9010c590 2024-09-15T23:35:49.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.816+0000 7fba96462640 1 -- 172.21.15.89:0/187031583 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba901af8c0 con 0x7fba9010c590 2024-09-15T23:35:49.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.818+0000 7fba8d7fa640 1 -- 172.21.15.89:0/187031583 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fba80017070 con 0x7fba9010c590 2024-09-15T23:35:49.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.818+0000 7fba9883b740 1 Processor -- start 2024-09-15T23:35:49.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.818+0000 7fba9883b740 1 -- start start 2024-09-15T23:35:49.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.818+0000 7fba9883b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x561621556c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.818+0000 7fba9883b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621557170 0x5616215614f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.818+0000 7fba9883b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621563e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.818+0000 7fba9883b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5616214bb160 con 0x561621561a30 2024-09-15T23:35:49.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.818+0000 7fba9883b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561621462720 con 0x561621557170 2024-09-15T23:35:49.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.818+0000 7fba9883b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5616214b7120 con 0x561621556850 2024-09-15T23:35:49.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.818+0000 7fba8d7fa640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba780a3240 0x7fba780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621563e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621563e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46028/0 (socket says 172.21.15.89:46028) 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba949ff640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba780a3240 0x7fba780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8f7fe640 1 -- 172.21.15.89:0/3157310144 learned_addr learned my addr 172.21.15.89:0/3157310144 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8ffff640 1 --2- 172.21.15.89:0/3157310144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621557170 0x5616215614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8d7fa640 1 -- 172.21.15.89:0/187031583 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fba800cf1f0 con 0x7fba9010c590 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba949ff640 1 --2- 172.21.15.89:0/3157310144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x561621556c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8f7fe640 1 -- 172.21.15.89:0/3157310144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 msgr2=0x561621556c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/3157310144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x561621556c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8f7fe640 1 -- 172.21.15.89:0/3157310144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621557170 msgr2=0x5616215614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:49.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/3157310144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621557170 0x5616215614f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8f7fe640 1 -- 172.21.15.89:0/3157310144 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56162122cba0 con 0x561621561a30 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba949ff640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba780a3240 0x7fba780a5700 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fba88009430 tx=0x7fba88031d40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/3157310144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621563e20 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fba8400bbe0 tx=0x7fba8400d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7180b8c1adc2291a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba767fc640 1 -- 172.21.15.89:0/3157310144 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84017620 con 0x561621561a30 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba767fc640 1 -- 172.21.15.89:0/3157310144 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba84017ca0 con 0x561621561a30 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.819+0000 7fba767fc640 1 -- 172.21.15.89:0/3157310144 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8400fd50 con 0x561621561a30 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.820+0000 7fba9883b740 1 -- 172.21.15.89:0/3157310144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 msgr2=0x561621563e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.820+0000 7fba9883b740 1 --2- 172.21.15.89:0/3157310144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621563e20 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fba8400bbe0 tx=0x7fba8400d710 comp rx=0 tx=0).stop 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.820+0000 7fba9883b740 1 -- 172.21.15.89:0/3157310144 shutdown_connections 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.820+0000 7fba9883b740 1 --2- 172.21.15.89:0/3157310144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621563e20 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.820+0000 7fba9883b740 1 --2- 172.21.15.89:0/3157310144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621557170 0x5616215614f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.820+0000 7fba9883b740 1 --2- 172.21.15.89:0/3157310144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x561621556c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.820+0000 7fba9883b740 1 -- 172.21.15.89:0/3157310144 >> 172.21.15.89:0/3157310144 conn(0x561621552de0 msgr2=0x5616215531c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.820+0000 7fba9883b740 1 -- 172.21.15.89:0/3157310144 shutdown_connections 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.820+0000 7fba9883b740 1 -- 172.21.15.89:0/3157310144 wait complete. 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba9883b740 1 -- start start 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba9883b740 1 Processor -- start 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba9883b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x56162155e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba9883b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x56162155ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba9883b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621561a30 0x56162155f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba9883b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561621564600 con 0x561621557170 2024-09-15T23:35:49.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba9883b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561621443900 con 0x561621561a30 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba9883b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561621564780 con 0x561621556850 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba9883b740 10 client.0 _ll_register_callbacks cb 0x561621441780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba8f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x56162155ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba8f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x56162155ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46034/0 (socket says 172.21.15.89:46034) 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba8f7fe640 1 -- 172.21.15.89:0/628540381 learned_addr learned my addr 172.21.15.89:0/628540381 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba949ff640 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621561a30 0x56162155f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.821+0000 7fba8ffff640 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x56162155e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba8f7fe640 1 -- 172.21.15.89:0/628540381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 msgr2=0x56162155e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x56162155e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba8f7fe640 1 -- 172.21.15.89:0/628540381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621561a30 msgr2=0x56162155f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621561a30 0x56162155f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba8f7fe640 1 -- 172.21.15.89:0/628540381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561621483c50 con 0x561621557170 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba8ffff640 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x56162155e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x56162155ee10 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fba8400f650 tx=0x7fba8400ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67597d912b21312b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba6f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba6f7fe640 1 -- 172.21.15.89:0/628540381 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84005c70 con 0x561621557170 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba6f7fe640 1 -- 172.21.15.89:0/628540381 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fba84005030 con 0x561621557170 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba9883b740 1 -- 172.21.15.89:0/628540381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba84017ca0 con 0x561621557170 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba6f7fe640 1 -- 172.21.15.89:0/628540381 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fba84030050 con 0x561621557170 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba9883b740 10 client.19431 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba9883b740 1 -- 172.21.15.89:0/628540381 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561621570260 con 0x561621557170 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.822+0000 7fba6f7fe640 1 -- 172.21.15.89:0/628540381 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba84017ca0 con 0x561621557170 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.823+0000 7fba6f7fe640 1 -- 172.21.15.89:0/628540381 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fba84004770 con 0x561621557170 2024-09-15T23:35:49.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.823+0000 7fba9883b740 10 client.19431 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:49.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.823+0000 7fba9883b740 10 client.19431 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:35:49.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.823+0000 7fba9883b740 20 client.19431 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:49.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.823+0000 7fba9883b740 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561621571610 0x561621573a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.823+0000 7fba9883b740 4 client.19431 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:49.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.823+0000 7fba9883b740 1 -- 172.21.15.89:0/628540381 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616214c8890 con 0x561621571610 2024-09-15T23:35:49.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.823+0000 7fba8ffff640 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561621571610 0x561621573a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.823+0000 7fba8ffff640 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561621571610 0x561621573a00 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e928a2f13140a4f server_cookie=807e66fa533e019c in_seq=0 out_seq=0 2024-09-15T23:35:49.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.823+0000 7fba6f7fe640 10 client.19431 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:49.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba6f7fe640 1 -- 172.21.15.89:0/628540381 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5616214c8890 con 0x561621571610 2024-09-15T23:35:49.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba6f7fe640 10 client.19431 handle_command_reply: tid=0 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 client.19431 shutdown 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 -- 172.21.15.89:0/628540381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 msgr2=0x56162155ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x56162155ee10 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fba8400f650 tx=0x7fba8400ff70 comp rx=0 tx=0).stop 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 -- 172.21.15.89:0/628540381 shutdown_connections 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561621571610 0x561621573a00 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621561a30 0x56162155f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x56162155ee10 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 --2- 172.21.15.89:0/628540381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x56162155e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 -- 172.21.15.89:0/628540381 >> 172.21.15.89:0/628540381 conn(0x561621552de0 msgr2=0x561621558ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba6f7fe640 0 client.19431 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 -- 172.21.15.89:0/628540381 shutdown_connections 2024-09-15T23:35:49.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 1 -- 172.21.15.89:0/628540381 wait complete. 2024-09-15T23:35:49.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.824+0000 7fba9883b740 20 client.19431 trim_cache size 0 max 16384 2024-09-15T23:35:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba9883b740 1 Processor -- start 2024-09-15T23:35:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba9883b740 1 -- start start 2024-09-15T23:35:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba9883b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621556850 0x561621559bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba9883b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x5616215589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba9883b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621561a30 0x561621579000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba9883b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fba901b9870 con 0x561621557170 2024-09-15T23:35:49.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba9883b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fba901bb3b0 con 0x561621556850 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba9883b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fba901b9f40 con 0x561621561a30 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba8f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x5616215589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba8f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x5616215589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46050/0 (socket says 172.21.15.89:46050) 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba8f7fe640 1 -- 172.21.15.89:0/3932639506 learned_addr learned my addr 172.21.15.89:0/3932639506 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba8ffff640 1 --2- 172.21.15.89:0/3932639506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621556850 0x561621559bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba8f7fe640 1 -- 172.21.15.89:0/3932639506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621561a30 msgr2=0x561621579000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/3932639506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621561a30 0x561621579000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba8f7fe640 1 -- 172.21.15.89:0/3932639506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621556850 msgr2=0x561621559bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/3932639506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621556850 0x561621559bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.841+0000 7fba8f7fe640 1 -- 172.21.15.89:0/3932639506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56162155d880 con 0x561621557170 2024-09-15T23:35:49.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.842+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/3932639506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x5616215589b0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fba84004d00 tx=0x7fba8400bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cefd28716109e31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:49.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.842+0000 7fba75ffb640 1 -- 172.21.15.89:0/3932639506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8400b370 con 0x561621557170 2024-09-15T23:35:49.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.842+0000 7fba75ffb640 1 -- 172.21.15.89:0/3932639506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba84027070 con 0x561621557170 2024-09-15T23:35:49.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.842+0000 7fba75ffb640 1 -- 172.21.15.89:0/3932639506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84004d80 con 0x561621557170 2024-09-15T23:35:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.842+0000 7fba9883b740 1 -- 172.21.15.89:0/3932639506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 msgr2=0x5616215589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.842+0000 7fba9883b740 1 --2- 172.21.15.89:0/3932639506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x5616215589b0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fba84004d00 tx=0x7fba8400bf60 comp rx=0 tx=0).stop 2024-09-15T23:35:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 -- 172.21.15.89:0/3932639506 shutdown_connections 2024-09-15T23:35:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 --2- 172.21.15.89:0/3932639506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621561a30 0x561621579000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 --2- 172.21.15.89:0/3932639506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621557170 0x5616215589b0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 --2- 172.21.15.89:0/3932639506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621556850 0x561621559bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 -- 172.21.15.89:0/3932639506 >> 172.21.15.89:0/3932639506 conn(0x561621552de0 msgr2=0x5616215582a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 -- 172.21.15.89:0/3932639506 shutdown_connections 2024-09-15T23:35:49.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 -- 172.21.15.89:0/3932639506 wait complete. 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 -- start start 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 Processor -- start 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x56162156a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621557170 0x5616215655a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621565d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.843+0000 7fba9883b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fba901bace0 con 0x561621561a30 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba9883b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fba901ba610 con 0x561621557170 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba9883b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fba901b7850 con 0x561621556850 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba9883b740 10 client.0 _ll_register_callbacks cb 0x7fba80017ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba949ff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621565d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba949ff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621565d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46058/0 (socket says 172.21.15.89:46058) 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba949ff640 1 -- 172.21.15.89:0/276085793 learned_addr learned my addr 172.21.15.89:0/276085793 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba8ffff640 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x56162156a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba8f7fe640 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621557170 0x5616215655a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba949ff640 1 -- 172.21.15.89:0/276085793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 msgr2=0x56162156a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba949ff640 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x56162156a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba949ff640 1 -- 172.21.15.89:0/276085793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621557170 msgr2=0x5616215655a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba949ff640 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621557170 0x5616215655a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba949ff640 1 -- 172.21.15.89:0/276085793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616215e37e0 con 0x561621561a30 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba949ff640 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621565d30 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fba88009ca0 tx=0x7fba8802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0766ee29cfcb72b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba6f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba6f7fe640 1 -- 172.21.15.89:0/276085793 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba88003e60 con 0x561621561a30 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba6f7fe640 1 -- 172.21.15.89:0/276085793 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fba88004710 con 0x561621561a30 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba9883b740 1 -- 172.21.15.89:0/276085793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba84027070 con 0x561621561a30 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba6f7fe640 1 -- 172.21.15.89:0/276085793 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fba8805e050 con 0x561621561a30 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.844+0000 7fba9883b740 10 client.19437 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:49.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.845+0000 7fba9883b740 1 -- 172.21.15.89:0/276085793 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5616215e4910 con 0x561621561a30 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.845+0000 7fba6f7fe640 1 -- 172.21.15.89:0/276085793 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba84027070 con 0x561621561a30 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.845+0000 7fba6f7fe640 1 -- 172.21.15.89:0/276085793 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fba8802f7d0 con 0x561621561a30 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.845+0000 7fba9883b740 10 client.19437 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.845+0000 7fba9883b740 10 client.19437 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.845+0000 7fba9883b740 20 client.19437 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.845+0000 7fba9883b740 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5616215e5160 0x5616215e7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.845+0000 7fba9883b740 4 client.19437 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}] 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.845+0000 7fba9883b740 1 -- 172.21.15.89:0/276085793 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}) -- 0x5616214c2be0 con 0x5616215e5160 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.845+0000 7fba8ffff640 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5616215e5160 0x5616215e7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.846+0000 7fba6f7fe640 10 client.19437 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:49.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.846+0000 7fba8ffff640 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5616215e5160 0x5616215e7580 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=687244722a0fd255 server_cookie=a8a54e5656e16109 in_seq=0 out_seq=0 2024-09-15T23:35:49.858 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba6f7fe640 1 -- 172.21.15.89:0/276085793 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5616214c2be0 con 0x5616215e5160 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba6f7fe640 10 client.19437 handle_command_reply: tid=0 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 client.19437 shutdown 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 -- 172.21.15.89:0/276085793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 msgr2=0x561621565d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621565d30 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fba88009ca0 tx=0x7fba8802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 -- 172.21.15.89:0/276085793 shutdown_connections 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5616215e5160 0x5616215e7580 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561621561a30 0x561621565d30 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561621557170 0x5616215655a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 --2- 172.21.15.89:0/276085793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561621556850 0x56162156a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 -- 172.21.15.89:0/276085793 >> 172.21.15.89:0/276085793 conn(0x561621552de0 msgr2=0x561621558ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba6f7fe640 0 client.19437 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:49.859 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 -- 172.21.15.89:0/276085793 shutdown_connections 2024-09-15T23:35:49.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 1 -- 172.21.15.89:0/276085793 wait complete. 2024-09-15T23:35:49.860 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.857+0000 7fba9883b740 20 client.19437 trim_cache size 0 max 16384 2024-09-15T23:35:49.860 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:35:49.860 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:35:49.860 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 42, 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "dddc5eba": { 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 42, 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 141.8, 2024-09-15T23:35:49.861 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:35:49.862 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:35:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 -- 172.21.15.89:0/187031583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba780a3240 msgr2=0x7fba780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba780a3240 0x7fba780a5700 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fba88009430 tx=0x7fba88031d40 comp rx=0 tx=0).stop 2024-09-15T23:35:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 -- 172.21.15.89:0/187031583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba9010c590 msgr2=0x7fba901a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba9010c590 0x7fba901a2d40 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fba8000b4a0 tx=0x7fba8000b960 comp rx=0 tx=0).stop 2024-09-15T23:35:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 -- 172.21.15.89:0/187031583 shutdown_connections 2024-09-15T23:35:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fba780a3240 0x7fba780a5700 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba90112640 0x7fba901a7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba9010cf30 0x7fba901a3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.863 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 --2- 172.21.15.89:0/187031583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba9010c590 0x7fba901a2d40 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 -- 172.21.15.89:0/187031583 >> 172.21.15.89:0/187031583 conn(0x7fba9006cdc0 msgr2=0x7fba9010ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 -- 172.21.15.89:0/187031583 shutdown_connections 2024-09-15T23:35:49.864 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:35:49.858+0000 7fba96462640 1 -- 172.21.15.89:0/187031583 wait complete. 2024-09-15T23:35:50.506 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:50 smithi089 ceph-mon[30933]: pgmap v704: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 12 op/s 2024-09-15T23:35:50.535 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'dddc5eba', quiesce took 3.1 seconds. Will release after: 18.75 2024-09-15T23:35:50.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:50 smithi120 ceph-mon[35250]: pgmap v704: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 12 op/s 2024-09-15T23:35:50.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:50 smithi174 ceph-mon[35586]: pgmap v704: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 12 op/s 2024-09-15T23:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.181+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3573858091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd0102600 msgr2=0x7fbbd0108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.181+0000 7fbbd82d0640 1 --2- 172.21.15.89:0/3573858091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd0102600 0x7fbbd0108e90 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009ca0 tx=0x7fbbc802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3573858091 shutdown_connections 2024-09-15T23:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 --2- 172.21.15.89:0/3573858091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd0102600 0x7fbbd0108e90 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 --2- 172.21.15.89:0/3573858091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0101c40 0x7fbbd01020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 --2- 172.21.15.89:0/3573858091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd0100a50 0x7fbbd0100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3573858091 >> 172.21.15.89:0/3573858091 conn(0x7fbbd00fc1b0 msgr2=0x7fbbd00fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3573858091 shutdown_connections 2024-09-15T23:35:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3573858091 wait complete. 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 Processor -- start 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 -- start start 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd0100a50 0x7fbbd006aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd0101c40 0x7fbbd019ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0102600 0x7fbbd019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd010c720 con 0x7fbbd0100a50 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd010c5a0 con 0x7fbbd0102600 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.182+0000 7fbbd82d0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd010c8a0 con 0x7fbbd0101c40 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd606c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd0100a50 0x7fbbd006aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd606c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd0100a50 0x7fbbd006aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46080/0 (socket says 172.21.15.89:46080) 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd606c640 1 -- 172.21.15.89:0/3406891650 learned_addr learned my addr 172.21.15.89:0/3406891650 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd586b640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd0101c40 0x7fbbd019ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd686d640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0102600 0x7fbbd019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd606c640 1 -- 172.21.15.89:0/3406891650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd0101c40 msgr2=0x7fbbd019ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd606c640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd0101c40 0x7fbbd019ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd606c640 1 -- 172.21.15.89:0/3406891650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0102600 msgr2=0x7fbbd019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd606c640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0102600 0x7fbbd019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd606c640 1 -- 172.21.15.89:0/3406891650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd019f9a0 con 0x7fbbd0100a50 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd686d640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0102600 0x7fbbd019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd606c640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd0100a50 0x7fbbd006aef0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc00c970 tx=0x7fbbcc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbc77fe640 1 -- 172.21.15.89:0/3406891650 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc004220 con 0x7fbbd0100a50 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3406891650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd019fc90 con 0x7fbbd0100a50 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbc77fe640 1 -- 172.21.15.89:0/3406891650 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbcc0043c0 con 0x7fbbd0100a50 2024-09-15T23:35:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbc77fe640 1 -- 172.21.15.89:0/3406891650 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc004a00 con 0x7fbbd0100a50 2024-09-15T23:35:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.183+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3406891650 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbd01a0190 con 0x7fbbd0100a50 2024-09-15T23:35:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.186+0000 7fbbc77fe640 1 -- 172.21.15.89:0/3406891650 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbbcc0079b0 con 0x7fbbd0100a50 2024-09-15T23:35:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.186+0000 7fbbdb06c740 1 Processor -- start 2024-09-15T23:35:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.186+0000 7fbbdb06c740 1 -- start start 2024-09-15T23:35:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50145da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501462e0 0x55cb5014ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb5014d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.195 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cb500a92f0 con 0x55cb501459a0 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cb50051270 con 0x55cb501462e0 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cb500a52b0 con 0x55cb5014b370 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd686d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbc77fe640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32f0 0x7fbbbc0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd686d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50145da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46090/0 (socket says 172.21.15.89:46090) 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd686d640 1 -- 172.21.15.89:0/2597071021 learned_addr learned my addr 172.21.15.89:0/2597071021 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd606c640 1 --2- 172.21.15.89:0/2597071021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501462e0 0x55cb5014ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd586b640 1 --2- 172.21.15.89:0/2597071021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb5014d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd686d640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32f0 0x7fbbbc0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbc77fe640 1 -- 172.21.15.89:0/3406891650 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbbcc0cfac0 con 0x7fbbd0100a50 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd686d640 1 -- 172.21.15.89:0/2597071021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 msgr2=0x55cb5014d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:51.196 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd686d640 1 --2- 172.21.15.89:0/2597071021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb5014d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd686d640 1 -- 172.21.15.89:0/2597071021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501462e0 msgr2=0x55cb5014ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd686d640 1 --2- 172.21.15.89:0/2597071021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501462e0 0x55cb5014ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd686d640 1 -- 172.21.15.89:0/2597071021 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb50050340 con 0x55cb501459a0 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.187+0000 7fbbd686d640 1 --2- 172.21.15.89:0/2597071021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50145da0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009ca0 tx=0x7fbbc8031910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107877b16812c05b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.188+0000 7fbbc4bfe640 1 -- 172.21.15.89:0/2597071021 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8003e50 con 0x55cb501459a0 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.188+0000 7fbbd686d640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32f0 0x7fbbbc0a57b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbbc804d000 tx=0x7fbbc804b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.188+0000 7fbbc4bfe640 1 -- 172.21.15.89:0/2597071021 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbc804e070 con 0x55cb501459a0 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.188+0000 7fbbc4bfe640 1 -- 172.21.15.89:0/2597071021 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8030830 con 0x55cb501459a0 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.188+0000 7fbbdb06c740 1 -- 172.21.15.89:0/2597071021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 msgr2=0x55cb50145da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.188+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/2597071021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50145da0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009ca0 tx=0x7fbbc8031910 comp rx=0 tx=0).stop 2024-09-15T23:35:51.197 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 -- 172.21.15.89:0/2597071021 shutdown_connections 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/2597071021 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb5014d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/2597071021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501462e0 0x55cb5014ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/2597071021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50145da0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 -- 172.21.15.89:0/2597071021 >> 172.21.15.89:0/2597071021 conn(0x55cb50141010 msgr2=0x55cb50141410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 -- 172.21.15.89:0/2597071021 shutdown_connections 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 -- 172.21.15.89:0/2597071021 wait complete. 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 -- start start 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 Processor -- start 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 0x55cb5014fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501462e0 0x55cb50150160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb501571a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55cb5014e340 con 0x55cb501462e0 2024-09-15T23:35:51.198 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cb50032290 con 0x55cb501459a0 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cb5014e4c0 con 0x55cb5014b370 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbdb06c740 10 client.0 _ll_register_callbacks cb 0x55cb50063140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbd586b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501462e0 0x55cb50150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbd586b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501462e0 0x55cb50150160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46098/0 (socket says 172.21.15.89:46098) 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.189+0000 7fbbd586b640 1 -- 172.21.15.89:0/1338032654 learned_addr learned my addr 172.21.15.89:0/1338032654 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbd686d640 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb501571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbd606c640 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 0x55cb5014fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbd586b640 1 -- 172.21.15.89:0/1338032654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 msgr2=0x55cb501571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbd586b640 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb501571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbd586b640 1 -- 172.21.15.89:0/1338032654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 msgr2=0x55cb5014fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbd586b640 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 0x55cb5014fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbd586b640 1 -- 172.21.15.89:0/1338032654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb4fdbc210 con 0x55cb501462e0 2024-09-15T23:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbd686d640 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb501571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbd606c640 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 0x55cb5014fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbd586b640 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501462e0 0x55cb50150160 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000bad0 tx=0x7fbbc000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed6ec6145165851c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/1338032654 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0011440 con 0x55cb501462e0 2024-09-15T23:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/1338032654 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbbc00115e0 con 0x55cb501462e0 2024-09-15T23:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbdb06c740 1 -- 172.21.15.89:0/1338032654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbc804e070 con 0x55cb501462e0 2024-09-15T23:35:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/1338032654 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbbc0005020 con 0x55cb501462e0 2024-09-15T23:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbdb06c740 10 client.19449 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbdb06c740 1 -- 172.21.15.89:0/1338032654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cb4fe88620 con 0x55cb501462e0 2024-09-15T23:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.190+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/1338032654 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbc804e070 con 0x55cb501462e0 2024-09-15T23:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.191+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/1338032654 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbbc00047a0 con 0x55cb501462e0 2024-09-15T23:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.191+0000 7fbbdb06c740 10 client.19449 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.191+0000 7fbbdb06c740 10 client.19449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.191+0000 7fbbdb06c740 20 client.19449 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.191+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cb5015b940 0x55cb5015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.191+0000 7fbbdb06c740 4 client.19449 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:35:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.191+0000 7fbbdb06c740 1 -- 172.21.15.89:0/1338032654 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb500b6a20 con 0x55cb5015b940 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.191+0000 7fbbd606c640 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cb5015b940 0x55cb5015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.192+0000 7fbbb9ffb640 10 client.19449 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.192+0000 7fbbd606c640 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cb5015b940 0x55cb5015ddb0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8beba14f85aaaa8c server_cookie=c687dd19e96d93a6 in_seq=0 out_seq=0 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.192+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/1338032654 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cb500b6a20 con 0x55cb5015b940 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.192+0000 7fbbb9ffb640 10 client.19449 handle_command_reply: tid=0 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.192+0000 7fbbdb06c740 1 client.19449 shutdown 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 1 -- 172.21.15.89:0/1338032654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501462e0 msgr2=0x55cb50150160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501462e0 0x55cb50150160 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000bad0 tx=0x7fbbc000bf90 comp rx=0 tx=0).stop 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 1 -- 172.21.15.89:0/1338032654 shutdown_connections 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cb5015b940 0x55cb5015ddb0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb501571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501462e0 0x55cb50150160 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/1338032654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 0x55cb5014fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 1 -- 172.21.15.89:0/1338032654 >> 172.21.15.89:0/1338032654 conn(0x55cb50141010 msgr2=0x55cb50142820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbb9ffb640 0 client.19449 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 1 -- 172.21.15.89:0/1338032654 shutdown_connections 2024-09-15T23:35:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 1 -- 172.21.15.89:0/1338032654 wait complete. 2024-09-15T23:35:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.193+0000 7fbbdb06c740 20 client.19449 trim_cache size 0 max 16384 2024-09-15T23:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.207+0000 7fbbdb06c740 1 Processor -- start 2024-09-15T23:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.207+0000 7fbbdb06c740 1 -- start start 2024-09-15T23:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.207+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50142400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.207+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501462e0 0x55cb50157160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.207+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb50142940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.207+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01aa800 con 0x55cb501459a0 2024-09-15T23:35:51.215 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.207+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01ac340 con 0x55cb501462e0 2024-09-15T23:35:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.207+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01aaed0 con 0x55cb5014b370 2024-09-15T23:35:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbd606c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbd606c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50142400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46100/0 (socket says 172.21.15.89:46100) 2024-09-15T23:35:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbd606c640 1 -- 172.21.15.89:0/690387798 learned_addr learned my addr 172.21.15.89:0/690387798 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbd586b640 1 --2- 172.21.15.89:0/690387798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501462e0 0x55cb50157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbd606c640 1 -- 172.21.15.89:0/690387798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 msgr2=0x55cb50142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbd606c640 1 --2- 172.21.15.89:0/690387798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb50142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbd606c640 1 -- 172.21.15.89:0/690387798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501462e0 msgr2=0x55cb50157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:51.216 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbd606c640 1 --2- 172.21.15.89:0/690387798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501462e0 0x55cb50157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbd606c640 1 -- 172.21.15.89:0/690387798 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb50140e30 con 0x55cb501459a0 2024-09-15T23:35:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbd606c640 1 --2- 172.21.15.89:0/690387798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50142400 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc0cd220 tx=0x7fbbcc06eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb1aa33d5768d5d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbbbfff640 1 -- 172.21.15.89:0/690387798 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc01d040 con 0x55cb501459a0 2024-09-15T23:35:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbbbfff640 1 -- 172.21.15.89:0/690387798 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbcc0a00a0 con 0x55cb501459a0 2024-09-15T23:35:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.208+0000 7fbbbbfff640 1 -- 172.21.15.89:0/690387798 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc09b140 con 0x55cb501459a0 2024-09-15T23:35:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 -- 172.21.15.89:0/690387798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 msgr2=0x55cb50142400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/690387798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50142400 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc0cd220 tx=0x7fbbcc06eb70 comp rx=0 tx=0).stop 2024-09-15T23:35:51.217 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 -- 172.21.15.89:0/690387798 shutdown_connections 2024-09-15T23:35:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/690387798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb5014b370 0x55cb50142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/690387798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501462e0 0x55cb50157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/690387798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb501459a0 0x55cb50142400 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 -- 172.21.15.89:0/690387798 >> 172.21.15.89:0/690387798 conn(0x55cb50141010 msgr2=0x55cb501401c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 -- 172.21.15.89:0/690387798 shutdown_connections 2024-09-15T23:35:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 -- 172.21.15.89:0/690387798 wait complete. 2024-09-15T23:35:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 -- start start 2024-09-15T23:35:51.218 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 Processor -- start 2024-09-15T23:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 0x55cb50154c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb501462e0 0x55cb50155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb5014b370 0x55cb50156140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01abc70 con 0x55cb5014b370 2024-09-15T23:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01ab5a0 con 0x55cb501459a0 2024-09-15T23:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01a87e0 con 0x55cb501462e0 2024-09-15T23:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbdb06c740 10 client.0 _ll_register_callbacks cb 0x7fbbc0004060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbd686d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb5014b370 0x55cb50156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.219 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbd686d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb5014b370 0x55cb50156140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46106/0 (socket says 172.21.15.89:46106) 2024-09-15T23:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.209+0000 7fbbd686d640 1 -- 172.21.15.89:0/3847655464 learned_addr learned my addr 172.21.15.89:0/3847655464 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbd606c640 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 0x55cb50154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbd586b640 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb501462e0 0x55cb50155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbd686d640 1 -- 172.21.15.89:0/3847655464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb501462e0 msgr2=0x55cb50155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbd686d640 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb501462e0 0x55cb50155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbd686d640 1 -- 172.21.15.89:0/3847655464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 msgr2=0x55cb50154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbd686d640 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 0x55cb50154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.220 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbd686d640 1 -- 172.21.15.89:0/3847655464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb50156750 con 0x55cb5014b370 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbd686d640 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb5014b370 0x55cb50156140 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009430 tx=0x7fbbc8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9187e2b41a4c04da server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/3847655464 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc80305c0 con 0x55cb5014b370 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/3847655464 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbbc8030760 con 0x55cb5014b370 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbdb06c740 1 -- 172.21.15.89:0/3847655464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbcc0a00a0 con 0x55cb5014b370 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/3847655464 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbbc8004b40 con 0x55cb5014b370 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbdb06c740 10 client.19455 fetch_fsmap learned FSMap version 29 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbdb06c740 1 -- 172.21.15.89:0/3847655464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cb501c6ac0 con 0x55cb5014b370 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.210+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/3847655464 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbcc0a00a0 con 0x55cb5014b370 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.211+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/3847655464 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbbc802f610 con 0x55cb5014b370 2024-09-15T23:35:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.211+0000 7fbbdb06c740 10 client.19455 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.211+0000 7fbbdb06c740 10 client.19455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.211+0000 7fbbdb06c740 20 client.19455 populate_metadata read hostname 'smithi089' 2024-09-15T23:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.211+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cb501c7070 0x55cb501c9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.211+0000 7fbbdb06c740 4 client.19455 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.211+0000 7fbbdb06c740 1 -- 172.21.15.89:0/3847655464 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cb500b0d70 con 0x55cb501c7070 2024-09-15T23:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.211+0000 7fbbd606c640 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cb501c7070 0x55cb501c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.211+0000 7fbbb9ffb640 10 client.19455 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.211+0000 7fbbd606c640 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cb501c7070 0x55cb501c9460 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0973d3d5011a324 server_cookie=58e70b71eda606be in_seq=0 out_seq=0 2024-09-15T23:35:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbb9ffb640 1 -- 172.21.15.89:0/3847655464 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cb500b0d70 con 0x55cb501c7070 2024-09-15T23:35:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbb9ffb640 10 client.19455 handle_command_reply: tid=0 2024-09-15T23:35:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 client.19455 shutdown 2024-09-15T23:35:51.226 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 -- 172.21.15.89:0/3847655464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb5014b370 msgr2=0x55cb50156140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb5014b370 0x55cb50156140 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8009430 tx=0x7fbbc8009580 comp rx=0 tx=0).stop 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 -- 172.21.15.89:0/3847655464 shutdown_connections 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55cb501c7070 0x55cb501c9460 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cb5014b370 0x55cb50156140 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cb501462e0 0x55cb50155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbb9ffb640 0 client.19455 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 --2- 172.21.15.89:0/3847655464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb501459a0 0x55cb50154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 -- 172.21.15.89:0/3847655464 >> 172.21.15.89:0/3847655464 conn(0x55cb50141010 msgr2=0x55cb501413f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 -- 172.21.15.89:0/3847655464 shutdown_connections 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 1 -- 172.21.15.89:0/3847655464 wait complete. 2024-09-15T23:35:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbdb06c740 20 client.19455 trim_cache size 0 max 16384 2024-09-15T23:35:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:35:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:35:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:35:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:35:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3406891650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32f0 msgr2=0x7fbbbc0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbd82d0640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32f0 0x7fbbbc0a57b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbbc804d000 tx=0x7fbbc804b000 comp rx=0 tx=0).stop 2024-09-15T23:35:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3406891650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd0100a50 msgr2=0x7fbbd006aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:35:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.225+0000 7fbbd82d0640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd0100a50 0x7fbbd006aef0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc00c970 tx=0x7fbbcc00ce30 comp rx=0 tx=0).stop 2024-09-15T23:35:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.226+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3406891650 shutdown_connections 2024-09-15T23:35:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.226+0000 7fbbd82d0640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbbbc0a32f0 0x7fbbbc0a57b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.226+0000 7fbbd82d0640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0102600 0x7fbbd019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.226+0000 7fbbd82d0640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbd0101c40 0x7fbbd019ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.226+0000 7fbbd82d0640 1 --2- 172.21.15.89:0/3406891650 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbd0100a50 0x7fbbd006aef0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:35:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.226+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3406891650 >> 172.21.15.89:0/3406891650 conn(0x7fbbd00fc1b0 msgr2=0x7fbbd0104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:35:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.226+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3406891650 shutdown_connections 2024-09-15T23:35:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:35:51.226+0000 7fbbd82d0640 1 -- 172.21.15.89:0/3406891650 wait complete. 2024-09-15T23:35:52.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:52 smithi089 ceph-mon[30933]: pgmap v705: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.9 KiB/s wr, 14 op/s 2024-09-15T23:35:52.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:52 smithi120 ceph-mon[35250]: pgmap v705: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.9 KiB/s wr, 14 op/s 2024-09-15T23:35:52.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:52 smithi174 ceph-mon[35586]: pgmap v705: 129 pgs: 129 active+clean; 1.8 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.9 KiB/s wr, 14 op/s 2024-09-15T23:35:54.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:54 smithi089 ceph-mon[30933]: pgmap v706: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 15 op/s 2024-09-15T23:35:54.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:54 smithi120 ceph-mon[35250]: pgmap v706: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 15 op/s 2024-09-15T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:54 smithi174 ceph-mon[35586]: pgmap v706: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 15 op/s 2024-09-15T23:35:56.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:56 smithi089 ceph-mon[30933]: pgmap v707: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-09-15T23:35:56.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:56 smithi120 ceph-mon[35250]: pgmap v707: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-09-15T23:35:56.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:56 smithi174 ceph-mon[35586]: pgmap v707: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.4 KiB/s wr, 12 op/s 2024-09-15T23:35:58.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:35:58 smithi120 ceph-mon[35250]: pgmap v708: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-09-15T23:35:58.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:35:58 smithi174 ceph-mon[35586]: pgmap v708: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-09-15T23:35:58.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:35:58 smithi089 ceph-mon[30933]: pgmap v708: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-09-15T23:36:00.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:00 smithi120 ceph-mon[35250]: pgmap v709: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-09-15T23:36:00.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:00 smithi174 ceph-mon[35586]: pgmap v709: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-09-15T23:36:00.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:00 smithi089 ceph-mon[30933]: pgmap v709: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2024-09-15T23:36:02.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:02 smithi120 ceph-mon[35250]: pgmap v710: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 15 op/s 2024-09-15T23:36:02.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:02 smithi174 ceph-mon[35586]: pgmap v710: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 15 op/s 2024-09-15T23:36:02.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:02 smithi089 ceph-mon[30933]: pgmap v710: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 KiB/s wr, 15 op/s 2024-09-15T23:36:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.324+0000 7fb1c13c4640 1 -- 172.21.15.89:0/3898310294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1bc107880 msgr2=0x7fb1bc109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:04.325 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.324+0000 7fb1c13c4640 1 --2- 172.21.15.89:0/3898310294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1bc107880 0x7fb1bc109c90 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009ca0 tx=0x7fb1b402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.324+0000 7fb1c13c4640 1 -- 172.21.15.89:0/3898310294 shutdown_connections 2024-09-15T23:36:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.324+0000 7fb1c13c4640 1 --2- 172.21.15.89:0/3898310294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1bc107880 0x7fb1bc109c90 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.324+0000 7fb1c13c4640 1 --2- 172.21.15.89:0/3898310294 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1bc104f50 0x7fb1bc107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.324+0000 7fb1c13c4640 1 --2- 172.21.15.89:0/3898310294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1bc102620 0x7fb1bc104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.324+0000 7fb1c13c4640 1 -- 172.21.15.89:0/3898310294 >> 172.21.15.89:0/3898310294 conn(0x7fb1bc0fc1e0 msgr2=0x7fb1bc0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.324+0000 7fb1c13c4640 1 -- 172.21.15.89:0/3898310294 shutdown_connections 2024-09-15T23:36:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.324+0000 7fb1c13c4640 1 -- 172.21.15.89:0/3898310294 wait complete. 2024-09-15T23:36:04.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1c13c4640 1 Processor -- start 2024-09-15T23:36:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1c13c4640 1 -- start start 2024-09-15T23:36:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1c13c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1bc102620 0x7fb1bc197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1c13c4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1bc104f50 0x7fb1bc198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1c13c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1bc107880 0x7fb1bc19f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1c13c4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc10d800 con 0x7fb1bc102620 2024-09-15T23:36:04.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1c13c4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc10d680 con 0x7fb1bc104f50 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1c13c4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc10d980 con 0x7fb1bc107880 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1baffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1bc102620 0x7fb1bc197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1baffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1bc102620 0x7fb1bc197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56668/0 (socket says 172.21.15.89:56668) 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1baffd640 1 -- 172.21.15.89:0/2583152382 learned_addr learned my addr 172.21.15.89:0/2583152382 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1bc107880 0x7fb1bc19f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.325+0000 7fb1ba7fc640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1bc104f50 0x7fb1bc198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1baffd640 1 -- 172.21.15.89:0/2583152382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1bc107880 msgr2=0x7fb1bc19f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1baffd640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1bc107880 0x7fb1bc19f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1baffd640 1 -- 172.21.15.89:0/2583152382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1bc104f50 msgr2=0x7fb1bc198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1baffd640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1bc104f50 0x7fb1bc198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1baffd640 1 -- 172.21.15.89:0/2583152382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1bc19fa40 con 0x7fb1bc102620 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1baffd640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1bc102620 0x7fb1bc197e20 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac00d950 tx=0x7fb1ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1a3fff640 1 -- 172.21.15.89:0/2583152382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac017070 con 0x7fb1bc102620 2024-09-15T23:36:04.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1a3fff640 1 -- 172.21.15.89:0/2583152382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1ac009c30 con 0x7fb1bc102620 2024-09-15T23:36:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1c13c4640 1 -- 172.21.15.89:0/2583152382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1bc19fd30 con 0x7fb1bc102620 2024-09-15T23:36:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1a3fff640 1 -- 172.21.15.89:0/2583152382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac00f6b0 con 0x7fb1bc102620 2024-09-15T23:36:04.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.326+0000 7fb1c13c4640 1 -- 172.21.15.89:0/2583152382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1bc1a01d0 con 0x7fb1bc102620 2024-09-15T23:36:04.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.328+0000 7fb1a3fff640 1 -- 172.21.15.89:0/2583152382 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb1ac0109d0 con 0x7fb1bc102620 2024-09-15T23:36:04.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1c423c740 1 Processor -- start 2024-09-15T23:36:04.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1c423c740 1 -- start start 2024-09-15T23:36:04.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1a3fff640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb1a40a32f0 0x7fb1a40a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fd9a0 0x55b5b10fdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fe2e0 0x55b5b1102da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b1103370 0x55b5b1105830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1c423c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b5b10612f0 con 0x55b5b10fe2e0 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1c423c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b5b1009270 con 0x55b5b10fd9a0 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1c423c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b5b105d2b0 con 0x55b5b1103370 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb1a40a32f0 0x7fb1a40a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1a3fff640 1 -- 172.21.15.89:0/2583152382 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb1ac0cf940 con 0x7fb1bc102620 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1ba7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fe2e0 0x55b5b1102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1ba7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fe2e0 0x55b5b1102da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56674/0 (socket says 172.21.15.89:56674) 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1ba7fc640 1 -- 172.21.15.89:0/1950694326 learned_addr learned my addr 172.21.15.89:0/1950694326 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/1950694326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b1103370 0x55b5b1105830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1baffd640 1 --2- 172.21.15.89:0/1950694326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fd9a0 0x55b5b10fdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1ba7fc640 1 -- 172.21.15.89:0/1950694326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b1103370 msgr2=0x55b5b1105830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1ba7fc640 1 --2- 172.21.15.89:0/1950694326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b1103370 0x55b5b1105830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1ba7fc640 1 -- 172.21.15.89:0/1950694326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fd9a0 msgr2=0x55b5b10fdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1ba7fc640 1 --2- 172.21.15.89:0/1950694326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fd9a0 0x55b5b10fdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.329+0000 7fb1ba7fc640 1 -- 172.21.15.89:0/1950694326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5b1008340 con 0x55b5b10fe2e0 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.330+0000 7fb1ba7fc640 1 --2- 172.21.15.89:0/1950694326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fe2e0 0x55b5b1102da0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fb1b000b6c0 tx=0x7fb1b000bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78b49c415060c3d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.330+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb1a40a32f0 0x7fb1a40a57b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4006010 tx=0x7fb1b4009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.330+0000 7fb1a17fa640 1 -- 172.21.15.89:0/1950694326 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0003bd0 con 0x55b5b10fe2e0 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.330+0000 7fb1a17fa640 1 -- 172.21.15.89:0/1950694326 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b0015070 con 0x55b5b10fe2e0 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.330+0000 7fb1a17fa640 1 -- 172.21.15.89:0/1950694326 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0004d40 con 0x55b5b10fe2e0 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.330+0000 7fb1c423c740 1 -- 172.21.15.89:0/1950694326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fe2e0 msgr2=0x55b5b1102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 --2- 172.21.15.89:0/1950694326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fe2e0 0x55b5b1102da0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fb1b000b6c0 tx=0x7fb1b000bb80 comp rx=0 tx=0).stop 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 -- 172.21.15.89:0/1950694326 shutdown_connections 2024-09-15T23:36:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 --2- 172.21.15.89:0/1950694326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b1103370 0x55b5b1105830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 --2- 172.21.15.89:0/1950694326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fe2e0 0x55b5b1102da0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 --2- 172.21.15.89:0/1950694326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fd9a0 0x55b5b10fdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 -- 172.21.15.89:0/1950694326 >> 172.21.15.89:0/1950694326 conn(0x55b5b10f9010 msgr2=0x55b5b10f9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 -- 172.21.15.89:0/1950694326 shutdown_connections 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 -- 172.21.15.89:0/1950694326 wait complete. 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 -- start start 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 Processor -- start 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fd9a0 0x55b5b1107bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fe2e0 0x55b5b1108100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b5b1106340 con 0x55b5b1103370 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b5b0fea290 con 0x55b5b10fe2e0 2024-09-15T23:36:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b5b11064c0 con 0x55b5b10fd9a0 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1c423c740 10 client.0 _ll_register_callbacks cb 0x55b5b101b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1bb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1bb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56688/0 (socket says 172.21.15.89:56688) 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.331+0000 7fb1bb7fe640 1 -- 172.21.15.89:0/359963899 learned_addr learned my addr 172.21.15.89:0/359963899 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1baffd640 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fd9a0 0x55b5b1107bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1ba7fc640 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fe2e0 0x55b5b1108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1bb7fe640 1 -- 172.21.15.89:0/359963899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fd9a0 msgr2=0x55b5b1107bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fd9a0 0x55b5b1107bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1bb7fe640 1 -- 172.21.15.89:0/359963899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fe2e0 msgr2=0x55b5b1108100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fe2e0 0x55b5b1108100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1bb7fe640 1 -- 172.21.15.89:0/359963899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5b0d74210 con 0x55b5b1103370 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1baffd640 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fd9a0 0x55b5b1107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:04.340 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f140 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4035000 tx=0x7fb1b402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e866dca0d0a5a86a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb19eff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb19eff5640 1 -- 172.21.15.89:0/359963899 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b402f610 con 0x55b5b1103370 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb19eff5640 1 -- 172.21.15.89:0/359963899 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb1b402f7b0 con 0x55b5b1103370 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1c423c740 1 -- 172.21.15.89:0/359963899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b0015070 con 0x55b5b1103370 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb19eff5640 1 -- 172.21.15.89:0/359963899 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb1b4004b40 con 0x55b5b1103370 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1c423c740 10 client.19467 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb1c423c740 1 -- 172.21.15.89:0/359963899 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5b0e40620 con 0x55b5b1103370 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.332+0000 7fb19eff5640 1 -- 172.21.15.89:0/359963899 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b0015070 con 0x55b5b1103370 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.333+0000 7fb19eff5640 1 -- 172.21.15.89:0/359963899 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb1b404a070 con 0x55b5b1103370 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.333+0000 7fb1c423c740 10 client.19467 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.333+0000 7fb1c423c740 10 client.19467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.333+0000 7fb1c423c740 20 client.19467 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.333+0000 7fb1c423c740 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5b1113830 0x55b5b1115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.333+0000 7fb1c423c740 4 client.19467 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.333+0000 7fb1c423c740 1 -- 172.21.15.89:0/359963899 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5b106ea20 con 0x55b5b1113830 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.333+0000 7fb1baffd640 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5b1113830 0x55b5b1115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.333+0000 7fb19eff5640 10 client.19467 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.333+0000 7fb1baffd640 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5b1113830 0x55b5b1115ca0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=628dd8d892f9256b server_cookie=caa38044bde981a5 in_seq=0 out_seq=0 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb19eff5640 1 -- 172.21.15.89:0/359963899 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5b106ea20 con 0x55b5b1113830 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb19eff5640 10 client.19467 handle_command_reply: tid=0 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 client.19467 shutdown 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 -- 172.21.15.89:0/359963899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 msgr2=0x55b5b110f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f140 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4035000 tx=0x7fb1b402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 -- 172.21.15.89:0/359963899 shutdown_connections 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5b1113830 0x55b5b1115ca0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f140 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fe2e0 0x55b5b1108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 --2- 172.21.15.89:0/359963899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fd9a0 0x55b5b1107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb19eff5640 0 client.19467 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 -- 172.21.15.89:0/359963899 >> 172.21.15.89:0/359963899 conn(0x55b5b10f9010 msgr2=0x55b5b10fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 -- 172.21.15.89:0/359963899 shutdown_connections 2024-09-15T23:36:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 1 -- 172.21.15.89:0/359963899 wait complete. 2024-09-15T23:36:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.334+0000 7fb1c423c740 20 client.19467 trim_cache size 0 max 16384 2024-09-15T23:36:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.348+0000 7fb1c423c740 1 Processor -- start 2024-09-15T23:36:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.348+0000 7fb1c423c740 1 -- start start 2024-09-15T23:36:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.348+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fd9a0 0x55b5b1107bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fe2e0 0x55b5b1108100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1c423c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc1aa860 con 0x55b5b1103370 2024-09-15T23:36:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1c423c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc1ac3a0 con 0x55b5b10fd9a0 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1c423c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc1aaf30 con 0x55b5b10fe2e0 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1bb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1bb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56700/0 (socket says 172.21.15.89:56700) 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1bb7fe640 1 -- 172.21.15.89:0/2323636362 learned_addr learned my addr 172.21.15.89:0/2323636362 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1baffd640 1 --2- 172.21.15.89:0/2323636362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fd9a0 0x55b5b1107bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1ba7fc640 1 --2- 172.21.15.89:0/2323636362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fe2e0 0x55b5b1108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1bb7fe640 1 -- 172.21.15.89:0/2323636362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fe2e0 msgr2=0x55b5b1108100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/2323636362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fe2e0 0x55b5b1108100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1bb7fe640 1 -- 172.21.15.89:0/2323636362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fd9a0 msgr2=0x55b5b1107bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/2323636362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fd9a0 0x55b5b1107bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1bb7fe640 1 -- 172.21.15.89:0/2323636362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5b10f8e30 con 0x55b5b1103370 2024-09-15T23:36:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1ba7fc640 1 --2- 172.21.15.89:0/2323636362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fe2e0 0x55b5b1108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/2323636362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f060 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009ca0 tx=0x7fb1b4009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af08affc0a939393 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1a0ff9640 1 -- 172.21.15.89:0/2323636362 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4031760 con 0x55b5b1103370 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1a0ff9640 1 -- 172.21.15.89:0/2323636362 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b404f070 con 0x55b5b1103370 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.349+0000 7fb1a0ff9640 1 -- 172.21.15.89:0/2323636362 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b402fe00 con 0x55b5b1103370 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 -- 172.21.15.89:0/2323636362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 msgr2=0x55b5b110f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 --2- 172.21.15.89:0/2323636362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f060 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4009ca0 tx=0x7fb1b4009890 comp rx=0 tx=0).stop 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 -- 172.21.15.89:0/2323636362 shutdown_connections 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 --2- 172.21.15.89:0/2323636362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b1103370 0x55b5b110f060 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 --2- 172.21.15.89:0/2323636362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b10fe2e0 0x55b5b1108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 --2- 172.21.15.89:0/2323636362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fd9a0 0x55b5b1107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 -- 172.21.15.89:0/2323636362 >> 172.21.15.89:0/2323636362 conn(0x55b5b10f9010 msgr2=0x55b5b10ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 -- 172.21.15.89:0/2323636362 shutdown_connections 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 -- 172.21.15.89:0/2323636362 wait complete. 2024-09-15T23:36:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 -- start start 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 Processor -- start 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fd9a0 0x55b5b110c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fe2e0 0x55b5b110c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b1103370 0x55b5b117c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc1abcd0 con 0x55b5b10fd9a0 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc1ab600 con 0x55b5b10fe2e0 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb1bc1a8840 con 0x55b5b1103370 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.350+0000 7fb1c423c740 10 client.0 _ll_register_callbacks cb 0x7fb1b402f450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1baffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fd9a0 0x55b5b110c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1baffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fd9a0 0x55b5b110c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:56714/0 (socket says 172.21.15.89:56714) 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1baffd640 1 -- 172.21.15.89:0/574814186 learned_addr learned my addr 172.21.15.89:0/574814186 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1ba7fc640 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fe2e0 0x55b5b110c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1baffd640 1 -- 172.21.15.89:0/574814186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b1103370 msgr2=0x55b5b117c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1bb7fe640 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b1103370 0x55b5b117c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1baffd640 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b1103370 0x55b5b117c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1baffd640 1 -- 172.21.15.89:0/574814186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fe2e0 msgr2=0x55b5b110c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1baffd640 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fe2e0 0x55b5b110c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1baffd640 1 -- 172.21.15.89:0/574814186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5b117cab0 con 0x55b5b10fd9a0 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1baffd640 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fd9a0 0x55b5b110c0f0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac09bb50 tx=0x7fb1ac009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6832379ed0fbe270 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb19eff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb19eff5640 1 -- 172.21.15.89:0/574814186 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac082e80 con 0x55b5b10fd9a0 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb19eff5640 1 -- 172.21.15.89:0/574814186 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb1ac0cd100 con 0x55b5b10fd9a0 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1c423c740 1 -- 172.21.15.89:0/574814186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b404f070 con 0x55b5b10fd9a0 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb19eff5640 1 -- 172.21.15.89:0/574814186 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb1ac020040 con 0x55b5b10fd9a0 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1c423c740 10 client.19473 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb1c423c740 1 -- 172.21.15.89:0/574814186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5b117dd70 con 0x55b5b10fd9a0 2024-09-15T23:36:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.351+0000 7fb19eff5640 1 -- 172.21.15.89:0/574814186 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1b404f070 con 0x55b5b10fd9a0 2024-09-15T23:36:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.352+0000 7fb19eff5640 1 -- 172.21.15.89:0/574814186 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb1ac07bdb0 con 0x55b5b10fd9a0 2024-09-15T23:36:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.352+0000 7fb1c423c740 10 client.19473 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.352+0000 7fb1c423c740 10 client.19473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.352+0000 7fb1c423c740 20 client.19473 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.352+0000 7fb1c423c740 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5b117e650 0x55b5b1180a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.352+0000 7fb1c423c740 4 client.19473 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:36:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.352+0000 7fb1c423c740 1 -- 172.21.15.89:0/574814186 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b5b1068d70 con 0x55b5b117e650 2024-09-15T23:36:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.352+0000 7fb1ba7fc640 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5b117e650 0x55b5b1180a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.352+0000 7fb19eff5640 10 client.19473 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.352+0000 7fb1ba7fc640 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5b117e650 0x55b5b1180a70 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5aeb8c04d0be6c4e server_cookie=87c2c110e719d8b7 in_seq=0 out_seq=0 2024-09-15T23:36:04.365 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb19eff5640 1 -- 172.21.15.89:0/574814186 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b5b1068d70 con 0x55b5b117e650 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb19eff5640 10 client.19473 handle_command_reply: tid=0 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 client.19473 shutdown 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 -- 172.21.15.89:0/574814186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fd9a0 msgr2=0x55b5b110c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fd9a0 0x55b5b110c0f0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac09bb50 tx=0x7fb1ac009f00 comp rx=0 tx=0).stop 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 -- 172.21.15.89:0/574814186 shutdown_connections 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b5b117e650 0x55b5b1180a70 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b5b1103370 0x55b5b117c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb19eff5640 0 client.19473 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b5b10fe2e0 0x55b5b110c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 --2- 172.21.15.89:0/574814186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b5b10fd9a0 0x55b5b110c0f0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 -- 172.21.15.89:0/574814186 >> 172.21.15.89:0/574814186 conn(0x55b5b10f9010 msgr2=0x55b5b10ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 -- 172.21.15.89:0/574814186 shutdown_connections 2024-09-15T23:36:04.366 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 1 -- 172.21.15.89:0/574814186 wait complete. 2024-09-15T23:36:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.364+0000 7fb1c423c740 20 client.19473 trim_cache size 0 max 16384 2024-09-15T23:36:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:36:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:36:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:36:04.367 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:36:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 -- 172.21.15.89:0/2583152382 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb1a40a32f0 msgr2=0x7fb1a40a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb1a40a32f0 0x7fb1a40a57b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4006010 tx=0x7fb1b4009360 comp rx=0 tx=0).stop 2024-09-15T23:36:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 -- 172.21.15.89:0/2583152382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1bc102620 msgr2=0x7fb1bc197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1bc102620 0x7fb1bc197e20 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac00d950 tx=0x7fb1ac00de10 comp rx=0 tx=0).stop 2024-09-15T23:36:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 -- 172.21.15.89:0/2583152382 shutdown_connections 2024-09-15T23:36:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb1a40a32f0 0x7fb1a40a57b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1bc107880 0x7fb1bc19f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1bc104f50 0x7fb1bc198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.368 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 --2- 172.21.15.89:0/2583152382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1bc102620 0x7fb1bc197e20 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 -- 172.21.15.89:0/2583152382 >> 172.21.15.89:0/2583152382 conn(0x7fb1bc0fc1e0 msgr2=0x7fb1bc102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 -- 172.21.15.89:0/2583152382 shutdown_connections 2024-09-15T23:36:04.369 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:04.365+0000 7fb1c13c4640 1 -- 172.21.15.89:0/2583152382 wait complete. 2024-09-15T23:36:04.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:04 smithi120 ceph-mon[35250]: pgmap v711: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2024-09-15T23:36:04.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:04 smithi174 ceph-mon[35586]: pgmap v711: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2024-09-15T23:36:04.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:04 smithi089 ceph-mon[30933]: pgmap v711: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2024-09-15T23:36:06.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:06 smithi120 ceph-mon[35250]: pgmap v712: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:36:06.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:36:06.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:06 smithi174 ceph-mon[35586]: pgmap v712: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:36:06.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:36:06.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:06 smithi089 ceph-mon[30933]: pgmap v712: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:36:06.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:36:06.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:06.811 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:07.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:36:07.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:36:07.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:07.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:36:07.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:36:07.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:07.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:36:07.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:36:07.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:36:07.975 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:36:08.233 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:08.644 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:08 smithi089 ceph-mon[30933]: pgmap v713: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:36:08.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:08 smithi120 ceph-mon[35250]: pgmap v713: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:36:08.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:08 smithi174 ceph-mon[35586]: pgmap v713: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:36:08.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.725+0000 7f73a797b640 1 -- 172.21.15.89:0/3654955156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a0100b30 msgr2=0x7f73a0100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:08.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.725+0000 7f73a797b640 1 --2- 172.21.15.89:0/3654955156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a0100b30 0x7f73a0100f10 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f739c009900 tx=0x7f739c02a470 comp rx=0 tx=0).stop 2024-09-15T23:36:08.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.725+0000 7f73a797b640 1 -- 172.21.15.89:0/3654955156 shutdown_connections 2024-09-15T23:36:08.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.725+0000 7f73a797b640 1 --2- 172.21.15.89:0/3654955156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73a0105f40 0x7f73a01063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.725+0000 7f73a797b640 1 --2- 172.21.15.89:0/3654955156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73a0101450 0x7f73a0105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.725+0000 7f73a797b640 1 --2- 172.21.15.89:0/3654955156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a0100b30 0x7f73a0100f10 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.725+0000 7f73a797b640 1 -- 172.21.15.89:0/3654955156 >> 172.21.15.89:0/3654955156 conn(0x7f73a00fc7a0 msgr2=0x7f73a00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:08.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.725+0000 7f73a797b640 1 -- 172.21.15.89:0/3654955156 shutdown_connections 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.725+0000 7f73a797b640 1 -- 172.21.15.89:0/3654955156 wait complete. 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a797b640 1 Processor -- start 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a797b640 1 -- start start 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a797b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73a0100b30 0x7f73a01a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a797b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a0101450 0x7f73a01a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a797b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73a0105f40 0x7f73a019e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a797b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f73a0118550 con 0x7f73a0101450 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a797b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73a01183d0 con 0x7f73a0100b30 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a797b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73a01186d0 con 0x7f73a0105f40 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a4f16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a0101450 0x7f73a01a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a4f16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a0101450 0x7f73a01a4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37342/0 (socket says 172.21.15.89:37342) 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.726+0000 7f73a4f16640 1 -- 172.21.15.89:0/721501749 learned_addr learned my addr 172.21.15.89:0/721501749 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:08.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73a5717640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73a0100b30 0x7f73a01a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73a5f18640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73a0105f40 0x7f73a019e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73a4f16640 1 -- 172.21.15.89:0/721501749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73a0105f40 msgr2=0x7f73a019e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73a4f16640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73a0105f40 0x7f73a019e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73a4f16640 1 -- 172.21.15.89:0/721501749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73a0100b30 msgr2=0x7f73a01a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73a4f16640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73a0100b30 0x7f73a01a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73a4f16640 1 -- 172.21.15.89:0/721501749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a019efd0 con 0x7f73a0101450 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73a4f16640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a0101450 0x7f73a01a4c20 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f739000c940 tx=0x7f739000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73967fc640 1 -- 172.21.15.89:0/721501749 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7390004220 con 0x7f73a0101450 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73a797b640 1 -- 172.21.15.89:0/721501749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a019f2c0 con 0x7f73a0101450 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73967fc640 1 -- 172.21.15.89:0/721501749 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73900043c0 con 0x7f73a0101450 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73967fc640 1 -- 172.21.15.89:0/721501749 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7390004a00 con 0x7f73a0101450 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.727+0000 7f73a797b640 1 -- 172.21.15.89:0/721501749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a01ab5b0 con 0x7f73a0101450 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.729+0000 7f73967fc640 1 -- 172.21.15.89:0/721501749 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f73900079b0 con 0x7f73a0101450 2024-09-15T23:36:08.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a9d54740 1 Processor -- start 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a9d54740 1 -- start start 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc42880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc42dc0 0x55f38fc4d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc4d680 0x55f38fc4fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a9d54740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f38fba82e0 con 0x55f38fc424a0 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a9d54740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f38fb4fce0 con 0x55f38fc4d680 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a9d54740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f38fba42a0 con 0x55f38fc42dc0 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73967fc640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f738c0a3240 0x7f738c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a5f18640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc42880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a5717640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc42dc0 0x55f38fc4d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a5717640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc42dc0 0x55f38fc4d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:40904/0 (socket says 172.21.15.89:40904) 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a5717640 1 -- 172.21.15.89:0/2195446148 learned_addr learned my addr 172.21.15.89:0/2195446148 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:08.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a5f18640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc42880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37356/0 (socket says 172.21.15.89:37356) 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a4f16640 1 --2- 172.21.15.89:0/2195446148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc4d680 0x55f38fc4fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.730+0000 7f73a5f18640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f738c0a3240 0x7f738c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f73967fc640 1 -- 172.21.15.89:0/721501749 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f73900cfd00 con 0x7f73a0101450 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f73a5f18640 1 -- 172.21.15.89:0/2195446148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc42dc0 msgr2=0x55f38fc4d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f73a5f18640 1 --2- 172.21.15.89:0/2195446148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc42dc0 0x55f38fc4d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f73a5f18640 1 -- 172.21.15.89:0/2195446148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc4d680 msgr2=0x55f38fc4fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f73a5f18640 1 --2- 172.21.15.89:0/2195446148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc4d680 0x55f38fc4fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f73a5f18640 1 -- 172.21.15.89:0/2195446148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f38fb34440 con 0x55f38fc424a0 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f73a5f18640 1 --2- 172.21.15.89:0/2195446148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc42880 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f7398014dd0 tx=0x7f7398011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6759b41fc71be1dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f73a5f18640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f738c0a3240 0x7f738c0a5700 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7398011c50 tx=0x7f739801c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f738b7fe640 1 -- 172.21.15.89:0/2195446148 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398024070 con 0x55f38fc424a0 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f738b7fe640 1 -- 172.21.15.89:0/2195446148 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7398009bf0 con 0x55f38fc424a0 2024-09-15T23:36:08.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.731+0000 7f738b7fe640 1 -- 172.21.15.89:0/2195446148 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398003d90 con 0x55f38fc424a0 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 -- 172.21.15.89:0/2195446148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 msgr2=0x55f38fc42880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 --2- 172.21.15.89:0/2195446148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc42880 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f7398014dd0 tx=0x7f7398011750 comp rx=0 tx=0).stop 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 -- 172.21.15.89:0/2195446148 shutdown_connections 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 --2- 172.21.15.89:0/2195446148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc4d680 0x55f38fc4fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 --2- 172.21.15.89:0/2195446148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc42dc0 0x55f38fc4d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 --2- 172.21.15.89:0/2195446148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc42880 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 -- 172.21.15.89:0/2195446148 >> 172.21.15.89:0/2195446148 conn(0x55f38fc3ea30 msgr2=0x55f38fc3ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 -- 172.21.15.89:0/2195446148 shutdown_connections 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 -- 172.21.15.89:0/2195446148 wait complete. 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 -- start start 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.732+0000 7f73a9d54740 1 Processor -- start 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc4cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 0x55f38fc4a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc4a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a9d54740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f38fc50250 con 0x55f38fc424a0 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a9d54740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f38fb63050 con 0x55f38fc42dc0 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a9d54740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f38fc503d0 con 0x55f38fc4d680 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a9d54740 10 client.0 _ll_register_callbacks cb 0x55f38fba3160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a5717640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc4cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a5717640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc4cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37362/0 (socket says 172.21.15.89:37362) 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a5717640 1 -- 172.21.15.89:0/1475975758 learned_addr learned my addr 172.21.15.89:0/1475975758 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a4f16640 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 0x55f38fc4a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a5f18640 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc4a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a5717640 1 -- 172.21.15.89:0/1475975758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 msgr2=0x55f38fc4a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a5717640 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc4a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a5717640 1 -- 172.21.15.89:0/1475975758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 msgr2=0x55f38fc4a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:08.759 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a5717640 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 0x55f38fc4a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a5717640 1 -- 172.21.15.89:0/1475975758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f38f9193c0 con 0x55f38fc424a0 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f73a5717640 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc4cff0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f739c005940 tx=0x7f739c0319b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b05fe4b01101d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.733+0000 7f7388ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f7388ff9640 1 -- 172.21.15.89:0/1475975758 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c035040 con 0x55f38fc424a0 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f7388ff9640 1 -- 172.21.15.89:0/1475975758 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f739c005030 con 0x55f38fc424a0 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f73a9d54740 1 -- 172.21.15.89:0/1475975758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f38fbb5850 con 0x55f38fc424a0 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f7388ff9640 1 -- 172.21.15.89:0/1475975758 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f739c04f050 con 0x55f38fc424a0 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f73a9d54740 10 client.19485 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f73a9d54740 1 -- 172.21.15.89:0/1475975758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f38fc5bc20 con 0x55f38fc424a0 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f7388ff9640 1 -- 172.21.15.89:0/1475975758 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f38fbb5850 con 0x55f38fc424a0 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f7388ff9640 1 -- 172.21.15.89:0/1475975758 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f739c0035f0 con 0x55f38fc424a0 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f73a9d54740 10 client.19485 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:08.760 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f73a9d54740 10 client.19485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f73a9d54740 20 client.19485 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f73a9d54740 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f38fc5d0e0 0x55f38fc5f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f73a9d54740 4 client.19485 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f73a9d54740 1 -- 172.21.15.89:0/1475975758 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f38fbafd60 con 0x55f38fc5d0e0 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.734+0000 7f73a4f16640 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f38fc5d0e0 0x55f38fc5f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.735+0000 7f7388ff9640 10 client.19485 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.735+0000 7f73a4f16640 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f38fc5d0e0 0x55f38fc5f4d0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eb09d859faab3c server_cookie=ca0bd0edf3578e89 in_seq=0 out_seq=0 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f7388ff9640 1 -- 172.21.15.89:0/1475975758 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f38fbafd60 con 0x55f38fc5d0e0 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f7388ff9640 10 client.19485 handle_command_reply: tid=0 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 client.19485 shutdown 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 -- 172.21.15.89:0/1475975758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 msgr2=0x55f38fc4cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc4cff0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f739c005940 tx=0x7f739c0319b0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.761 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 -- 172.21.15.89:0/1475975758 shutdown_connections 2024-09-15T23:36:08.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f38fc5d0e0 0x55f38fc5f4d0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc4a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f7388ff9640 0 client.19485 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:08.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 0x55f38fc4a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 --2- 172.21.15.89:0/1475975758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc4cff0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 -- 172.21.15.89:0/1475975758 >> 172.21.15.89:0/1475975758 conn(0x55f38fc3ea30 msgr2=0x55f38fc44710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:08.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 -- 172.21.15.89:0/1475975758 shutdown_connections 2024-09-15T23:36:08.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 1 -- 172.21.15.89:0/1475975758 wait complete. 2024-09-15T23:36:08.762 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.736+0000 7f73a9d54740 20 client.19485 trim_cache size 0 max 16384 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a9d54740 1 Processor -- start 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a9d54740 1 -- start start 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc424a0 0x55f38fc48b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc42dc0 0x55f38fc64d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc652c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a9d54740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f73a01b5420 con 0x55f38fc42dc0 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a9d54740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73a01b6f60 con 0x55f38fc424a0 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a9d54740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73a01b5af0 con 0x55f38fc4d680 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a4f16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc42dc0 0x55f38fc64d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a4f16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc42dc0 0x55f38fc64d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37374/0 (socket says 172.21.15.89:37374) 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.753+0000 7f73a4f16640 1 -- 172.21.15.89:0/1951702780 learned_addr learned my addr 172.21.15.89:0/1951702780 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:08.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.754+0000 7f73a5f18640 1 --2- 172.21.15.89:0/1951702780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.754+0000 7f73a5717640 1 --2- 172.21.15.89:0/1951702780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc424a0 0x55f38fc48b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.754+0000 7f73a4f16640 1 -- 172.21.15.89:0/1951702780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 msgr2=0x55f38fc652c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.754+0000 7f73a4f16640 1 --2- 172.21.15.89:0/1951702780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc652c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.754+0000 7f73a4f16640 1 -- 172.21.15.89:0/1951702780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc424a0 msgr2=0x55f38fc48b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.754+0000 7f73a4f16640 1 --2- 172.21.15.89:0/1951702780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc424a0 0x55f38fc48b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.754+0000 7f73a4f16640 1 -- 172.21.15.89:0/1951702780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f38fc49250 con 0x55f38fc42dc0 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.754+0000 7f73a5717640 1 --2- 172.21.15.89:0/1951702780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc424a0 0x55f38fc48b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f73a4f16640 1 --2- 172.21.15.89:0/1951702780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc42dc0 0x55f38fc64d80 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f7390005430 tx=0x7f7390066970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec44c116fab25a23 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f738affd640 1 -- 172.21.15.89:0/1951702780 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739001d040 con 0x55f38fc42dc0 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f738affd640 1 -- 172.21.15.89:0/1951702780 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f739009bf30 con 0x55f38fc42dc0 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f738affd640 1 -- 172.21.15.89:0/1951702780 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7390014d80 con 0x55f38fc42dc0 2024-09-15T23:36:08.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f73a9d54740 1 -- 172.21.15.89:0/1951702780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc42dc0 msgr2=0x55f38fc64d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f73a9d54740 1 --2- 172.21.15.89:0/1951702780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc42dc0 0x55f38fc64d80 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f7390005430 tx=0x7f7390066970 comp rx=0 tx=0).stop 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f73a9d54740 1 -- 172.21.15.89:0/1951702780 shutdown_connections 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f73a9d54740 1 --2- 172.21.15.89:0/1951702780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc652c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f73a9d54740 1 --2- 172.21.15.89:0/1951702780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc42dc0 0x55f38fc64d80 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f73a9d54740 1 --2- 172.21.15.89:0/1951702780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc424a0 0x55f38fc48b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.755+0000 7f73a9d54740 1 -- 172.21.15.89:0/1951702780 >> 172.21.15.89:0/1951702780 conn(0x55f38fc3ea30 msgr2=0x55f38fc3db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 1 -- 172.21.15.89:0/1951702780 shutdown_connections 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 1 -- 172.21.15.89:0/1951702780 wait complete. 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 1 -- start start 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 1 Processor -- start 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc55d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 0x55f38fc530b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc535f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f73a01b6890 con 0x55f38fc424a0 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73a01b61c0 con 0x55f38fc42dc0 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73a01b3400 con 0x55f38fc4d680 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.756+0000 7f73a9d54740 10 client.0 _ll_register_callbacks cb 0x7f739c031640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a5717640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc55d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a5717640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc55d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37384/0 (socket says 172.21.15.89:37384) 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a5717640 1 -- 172.21.15.89:0/48486530 learned_addr learned my addr 172.21.15.89:0/48486530 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a4f16640 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 0x55f38fc530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a5f18640 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a5717640 1 -- 172.21.15.89:0/48486530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 msgr2=0x55f38fc535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a5717640 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a5717640 1 -- 172.21.15.89:0/48486530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 msgr2=0x55f38fc530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a5717640 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 0x55f38fc530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a5717640 1 -- 172.21.15.89:0/48486530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f38fc53e20 con 0x55f38fc424a0 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a4f16640 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 0x55f38fc530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a5717640 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc55d60 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f739c02a4b0 tx=0x7f739c009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2071fe45b611b52b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f7388ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f7388ff9640 1 -- 172.21.15.89:0/48486530 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c035070 con 0x55f38fc424a0 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f7388ff9640 1 -- 172.21.15.89:0/48486530 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f739c005040 con 0x55f38fc424a0 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a9d54740 1 -- 172.21.15.89:0/48486530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f739009bf30 con 0x55f38fc424a0 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f7388ff9640 1 -- 172.21.15.89:0/48486530 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f739c0037f0 con 0x55f38fc424a0 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a9d54740 10 client.19491 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.757+0000 7f73a9d54740 1 -- 172.21.15.89:0/48486530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f38fcc7cd0 con 0x55f38fc424a0 2024-09-15T23:36:08.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.758+0000 7f7388ff9640 1 -- 172.21.15.89:0/48486530 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f739009bf30 con 0x55f38fc424a0 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.758+0000 7f7388ff9640 1 -- 172.21.15.89:0/48486530 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f739c004b80 con 0x55f38fc424a0 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.758+0000 7f73a9d54740 10 client.19491 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.758+0000 7f73a9d54740 10 client.19491 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.758+0000 7f73a9d54740 20 client.19491 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.758+0000 7f73a9d54740 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f38fcc83f0 0x55f38fcca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.758+0000 7f73a9d54740 4 client.19491 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.758+0000 7f73a9d54740 1 -- 172.21.15.89:0/48486530 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f38fb2c210 con 0x55f38fcc83f0 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.758+0000 7f73a4f16640 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f38fcc83f0 0x55f38fcca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f7388ff9640 10 client.19491 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a4f16640 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f38fcc83f0 0x55f38fcca810 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65ecf15535fdf25d server_cookie=4ac72c6b094eeb3b in_seq=0 out_seq=0 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f7388ff9640 1 -- 172.21.15.89:0/48486530 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f38fb2c210 con 0x55f38fcc83f0 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f7388ff9640 10 client.19491 handle_command_reply: tid=0 2024-09-15T23:36:08.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 client.19491 shutdown 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 -- 172.21.15.89:0/48486530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 msgr2=0x55f38fc55d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc55d60 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f739c02a4b0 tx=0x7f739c009680 comp rx=0 tx=0).stop 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 -- 172.21.15.89:0/48486530 shutdown_connections 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f38fcc83f0 0x55f38fcca810 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f38fc4d680 0x55f38fc535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f38fc42dc0 0x55f38fc530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f7388ff9640 0 client.19491 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 --2- 172.21.15.89:0/48486530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38fc424a0 0x55f38fc55d60 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 -- 172.21.15.89:0/48486530 >> 172.21.15.89:0/48486530 conn(0x55f38fc3ea30 msgr2=0x55f38fc3db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 -- 172.21.15.89:0/48486530 shutdown_connections 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 1 -- 172.21.15.89:0/48486530 wait complete. 2024-09-15T23:36:08.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.759+0000 7f73a9d54740 20 client.19491 trim_cache size 0 max 16384 2024-09-15T23:36:08.774 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:36:08.774 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:36:08.774 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:36:08.774 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 -- 172.21.15.89:0/721501749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f738c0a3240 msgr2=0x7f738c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f738c0a3240 0x7f738c0a5700 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f7398011c50 tx=0x7f739801c000 comp rx=0 tx=0).stop 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 -- 172.21.15.89:0/721501749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a0101450 msgr2=0x7f73a01a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a0101450 0x7f73a01a4c20 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f739000c940 tx=0x7f739000ce00 comp rx=0 tx=0).stop 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 -- 172.21.15.89:0/721501749 shutdown_connections 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f738c0a3240 0x7f738c0a5700 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73a0105f40 0x7f73a019e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73a0101450 0x7f73a01a4c20 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 --2- 172.21.15.89:0/721501749 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73a0100b30 0x7f73a01a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 -- 172.21.15.89:0/721501749 >> 172.21.15.89:0/721501749 conn(0x7f73a00fc7a0 msgr2=0x7f73a00fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:08.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 -- 172.21.15.89:0/721501749 shutdown_connections 2024-09-15T23:36:08.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:08.760+0000 7f73a797b640 1 -- 172.21.15.89:0/721501749 wait complete. 2024-09-15T23:36:09.286 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:36:09.454 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:36:09.454 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:36:09.544 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.025+0000 7f59f2a12640 1 -- 172.21.15.89:0/2198008967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59ec101de0 msgr2=0x7f59ec1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.025+0000 7f59f2a12640 1 --2- 172.21.15.89:0/2198008967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59ec101de0 0x7f59ec1021c0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009900 tx=0x7f59dc02a410 comp rx=0 tx=0).stop 2024-09-15T23:36:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.025+0000 7f59f2a12640 1 -- 172.21.15.89:0/2198008967 shutdown_connections 2024-09-15T23:36:10.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.025+0000 7f59f2a12640 1 --2- 172.21.15.89:0/2198008967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59ec106f10 0x7f59ec10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.025+0000 7f59f2a12640 1 --2- 172.21.15.89:0/2198008967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59ec102790 0x7f59ec1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.025+0000 7f59f2a12640 1 --2- 172.21.15.89:0/2198008967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59ec101de0 0x7f59ec1021c0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.025+0000 7f59f2a12640 1 -- 172.21.15.89:0/2198008967 >> 172.21.15.89:0/2198008967 conn(0x7f59ec0fd8c0 msgr2=0x7f59ec0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.025+0000 7f59f2a12640 1 -- 172.21.15.89:0/2198008967 shutdown_connections 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.025+0000 7f59f2a12640 1 -- 172.21.15.89:0/2198008967 wait complete. 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59f2a12640 1 Processor -- start 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59f2a12640 1 -- start start 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59f2a12640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59ec101de0 0x7f59ec198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59f2a12640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59ec102790 0x7f59ec198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59f2a12640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59ec106f10 0x7f59ec19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:10.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59f2a12640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec111fb0 con 0x7f59ec102790 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59f2a12640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec111e30 con 0x7f59ec101de0 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59f2a12640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec112130 con 0x7f59ec106f10 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59ec102790 0x7f59ec198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59eb7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59ec102790 0x7f59ec198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37404/0 (socket says 172.21.15.89:37404) 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.026+0000 7f59eb7fe640 1 -- 172.21.15.89:0/3344628709 learned_addr learned my addr 172.21.15.89:0/3344628709 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59ebfff640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59ec101de0 0x7f59ec198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59f0faf640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59ec106f10 0x7f59ec19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59eb7fe640 1 -- 172.21.15.89:0/3344628709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59ec106f10 msgr2=0x7f59ec19cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59eb7fe640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59ec106f10 0x7f59ec19cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59eb7fe640 1 -- 172.21.15.89:0/3344628709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59ec101de0 msgr2=0x7f59ec198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59eb7fe640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59ec101de0 0x7f59ec198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59eb7fe640 1 -- 172.21.15.89:0/3344628709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ec19d4c0 con 0x7f59ec102790 2024-09-15T23:36:10.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59f0faf640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59ec106f10 0x7f59ec19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59eb7fe640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59ec102790 0x7f59ec198a50 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f59e000b4a0 tx=0x7f59e000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59e97fa640 1 -- 172.21.15.89:0/3344628709 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0014070 con 0x7f59ec102790 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59f2a12640 1 -- 172.21.15.89:0/3344628709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59ec19d7b0 con 0x7f59ec102790 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59f2a12640 1 -- 172.21.15.89:0/3344628709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59ec1a5090 con 0x7f59ec102790 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59e97fa640 1 -- 172.21.15.89:0/3344628709 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59e0010af0 con 0x7f59ec102790 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.027+0000 7f59e97fa640 1 -- 172.21.15.89:0/3344628709 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e001e060 con 0x7f59ec102790 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.028+0000 7f59f2a12640 1 -- 172.21.15.89:0/3344628709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d8005290 con 0x7f59ec102790 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.029+0000 7f59e97fa640 1 -- 172.21.15.89:0/3344628709 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f59e0017070 con 0x7f59ec102790 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.030+0000 7f59e97fa640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59d40a3240 0x7f59d40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.030+0000 7f59ebfff640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59d40a3240 0x7f59d40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.030+0000 7f59e97fa640 1 -- 172.21.15.89:0/3344628709 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f59e00cf300 con 0x7f59ec102790 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.030+0000 7f59ebfff640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59d40a3240 0x7f59d40a5700 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009230 tx=0x7f59dc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:10.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.033+0000 7f59e97fa640 1 -- 172.21.15.89:0/3344628709 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f59e009bc30 con 0x7f59ec102790 2024-09-15T23:36:10.086 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:10.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.247+0000 7f59f2a12640 1 -- 172.21.15.89:0/3344628709 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59d8005580 con 0x7f59ec102790 2024-09-15T23:36:10.248 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.247+0000 7f59e97fa640 1 -- 172.21.15.89:0/3344628709 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f59e009b530 con 0x7f59ec102790 2024-09-15T23:36:10.249 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:36:10.249 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:36:10.249 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 -- 172.21.15.89:0/3344628709 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59d40a3240 msgr2=0x7f59d40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59d40a3240 0x7f59d40a5700 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009230 tx=0x7f59dc0058d0 comp rx=0 tx=0).stop 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 -- 172.21.15.89:0/3344628709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59ec102790 msgr2=0x7f59ec198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59ec102790 0x7f59ec198a50 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f59e000b4a0 tx=0x7f59e000b960 comp rx=0 tx=0).stop 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 -- 172.21.15.89:0/3344628709 shutdown_connections 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f59d40a3240 0x7f59d40a5700 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59ec106f10 0x7f59ec19cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f59ec102790 0x7f59ec198a50 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 --2- 172.21.15.89:0/3344628709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59ec101de0 0x7f59ec198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 -- 172.21.15.89:0/3344628709 >> 172.21.15.89:0/3344628709 conn(0x7f59ec0fd8c0 msgr2=0x7f59ec0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 -- 172.21.15.89:0/3344628709 shutdown_connections 2024-09-15T23:36:10.251 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:10.250+0000 7f59f2a12640 1 -- 172.21.15.89:0/3344628709 wait complete. 2024-09-15T23:36:10.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:10 smithi089 ceph-mon[30933]: pgmap v714: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:36:10.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:10 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3344628709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:36:10.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:10 smithi120 ceph-mon[35250]: pgmap v714: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:36:10.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:10 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3344628709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:36:10.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:10 smithi174 ceph-mon[35586]: pgmap v714: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-09-15T23:36:10.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:10 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3344628709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:36:11.029 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id dddc5eba --release --await' 2024-09-15T23:36:11.029 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id dddc5eba --release --await 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.179+0000 7fa4c470b640 1 -- 172.21.15.89:0/2623929313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4bc100e20 msgr2=0x7fa4bc101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.179+0000 7fa4c470b640 1 --2- 172.21.15.89:0/2623929313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4bc100e20 0x7fa4bc101200 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009900 tx=0x7fa4b802a480 comp rx=0 tx=0).stop 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.179+0000 7fa4c470b640 1 -- 172.21.15.89:0/2623929313 shutdown_connections 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.179+0000 7fa4c470b640 1 --2- 172.21.15.89:0/2623929313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4bc102ef0 0x7fa4bc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.179+0000 7fa4c470b640 1 --2- 172.21.15.89:0/2623929313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc102570 0x7fa4bc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.179+0000 7fa4c470b640 1 --2- 172.21.15.89:0/2623929313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4bc100e20 0x7fa4bc101200 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.179+0000 7fa4c470b640 1 -- 172.21.15.89:0/2623929313 >> 172.21.15.89:0/2623929313 conn(0x7fa4bc0fc7d0 msgr2=0x7fa4bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.179+0000 7fa4c470b640 1 -- 172.21.15.89:0/2623929313 shutdown_connections 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.180+0000 7fa4c470b640 1 -- 172.21.15.89:0/2623929313 wait complete. 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.180+0000 7fa4c470b640 1 Processor -- start 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.180+0000 7fa4c470b640 1 -- start start 2024-09-15T23:36:11.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.180+0000 7fa4c470b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc100e20 0x7fa4bc103750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.180+0000 7fa4c470b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4bc102570 0x7fa4bc103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.180+0000 7fa4c470b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4bc102ef0 0x7fa4bc1041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.180+0000 7fa4c470b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc110f80 con 0x7fa4bc102570 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.180+0000 7fa4c470b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc110e00 con 0x7fa4bc102ef0 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.180+0000 7fa4c470b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa4bc111100 con 0x7fa4bc100e20 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.180+0000 7fa4c1ca6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4bc102570 0x7fa4bc103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c1ca6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4bc102570 0x7fa4bc103c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37408/0 (socket says 172.21.15.89:37408) 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c1ca6640 1 -- 172.21.15.89:0/1405954646 learned_addr learned my addr 172.21.15.89:0/1405954646 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c24a7640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc100e20 0x7fa4bc103750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c1ca6640 1 -- 172.21.15.89:0/1405954646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc100e20 msgr2=0x7fa4bc103750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c1ca6640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc100e20 0x7fa4bc103750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c2ca8640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4bc102ef0 0x7fa4bc1041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c1ca6640 1 -- 172.21.15.89:0/1405954646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4bc102ef0 msgr2=0x7fa4bc1041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:11.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c1ca6640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4bc102ef0 0x7fa4bc1041d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c1ca6640 1 -- 172.21.15.89:0/1405954646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4bc1a6a30 con 0x7fa4bc102570 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c2ca8640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4bc102ef0 0x7fa4bc1041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c1ca6640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4bc102570 0x7fa4bc103c90 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00d770 tx=0x7fa4ac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4b37fe640 1 -- 172.21.15.89:0/1405954646 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac017070 con 0x7fa4bc102570 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4b37fe640 1 -- 172.21.15.89:0/1405954646 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4ac005500 con 0x7fa4bc102570 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4b37fe640 1 -- 172.21.15.89:0/1405954646 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac004e00 con 0x7fa4bc102570 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c470b640 1 -- 172.21.15.89:0/1405954646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4bc1a6d20 con 0x7fa4bc102570 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.181+0000 7fa4c470b640 1 -- 172.21.15.89:0/1405954646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4bc1a71b0 con 0x7fa4bc102570 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.182+0000 7fa4c470b640 1 -- 172.21.15.89:0/1405954646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4bc101200 con 0x7fa4bc102570 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.184+0000 7fa4b37fe640 1 -- 172.21.15.89:0/1405954646 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa4ac004fa0 con 0x7fa4bc102570 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.184+0000 7fa4b37fe640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa4a80a32c0 0x7fa4a80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.184+0000 7fa4c24a7640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa4a80a32c0 0x7fa4a80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.184+0000 7fa4b37fe640 1 -- 172.21.15.89:0/1405954646 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa4ac0cfb00 con 0x7fa4bc102570 2024-09-15T23:36:11.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.185+0000 7fa4c24a7640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa4a80a32c0 0x7fa4a80a5780 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009230 tx=0x7fa4b80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:11.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.192+0000 7fa4b37fe640 1 -- 172.21.15.89:0/1405954646 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa4ac09c4b0 con 0x7fa4bc102570 2024-09-15T23:36:11.287 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:11.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.410+0000 7fa4c470b640 1 -- 172.21.15.89:0/1405954646 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4bc1a7470 con 0x7fa4bc102570 2024-09-15T23:36:11.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.411+0000 7fa4b37fe640 1 -- 172.21.15.89:0/1405954646 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fa4ac09bdb0 con 0x7fa4bc102570 2024-09-15T23:36:11.412 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:36:11.413 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:36:11.413 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:36:11.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 -- 172.21.15.89:0/1405954646 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa4a80a32c0 msgr2=0x7fa4a80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:11.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa4a80a32c0 0x7fa4a80a5780 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009230 tx=0x7fa4b80058d0 comp rx=0 tx=0).stop 2024-09-15T23:36:11.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 -- 172.21.15.89:0/1405954646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4bc102570 msgr2=0x7fa4bc103c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:11.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4bc102570 0x7fa4bc103c90 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00d770 tx=0x7fa4ac00dc30 comp rx=0 tx=0).stop 2024-09-15T23:36:11.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 -- 172.21.15.89:0/1405954646 shutdown_connections 2024-09-15T23:36:11.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa4a80a32c0 0x7fa4a80a5780 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:11.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa4bc102ef0 0x7fa4bc1041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:11.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4bc102570 0x7fa4bc103c90 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:11.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 --2- 172.21.15.89:0/1405954646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4bc100e20 0x7fa4bc103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:11.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 -- 172.21.15.89:0/1405954646 >> 172.21.15.89:0/1405954646 conn(0x7fa4bc0fc7d0 msgr2=0x7fa4bc06a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:11.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 -- 172.21.15.89:0/1405954646 shutdown_connections 2024-09-15T23:36:11.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:11.414+0000 7fa4c470b640 1 -- 172.21.15.89:0/1405954646 wait complete. 2024-09-15T23:36:12.254 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:36:12.499 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:12 smithi089 ceph-mon[30933]: pgmap v715: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-15T23:36:12.499 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:12 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1405954646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:36:12.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 -- 172.21.15.89:0/2288249087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4104106f20 msgr2=0x7f410410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:12.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 --2- 172.21.15.89:0/2288249087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4104106f20 0x7f410410eaa0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f40fc009900 tx=0x7f40fc02a410 comp rx=0 tx=0).stop 2024-09-15T23:36:12.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 -- 172.21.15.89:0/2288249087 shutdown_connections 2024-09-15T23:36:12.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 --2- 172.21.15.89:0/2288249087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4104106f20 0x7f410410eaa0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 --2- 172.21.15.89:0/2288249087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41041027a0 0x7f41041067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 --2- 172.21.15.89:0/2288249087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4104101df0 0x7f41041021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 -- 172.21.15.89:0/2288249087 >> 172.21.15.89:0/2288249087 conn(0x7f41040fd8f0 msgr2=0x7f41040ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 -- 172.21.15.89:0/2288249087 shutdown_connections 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 -- 172.21.15.89:0/2288249087 wait complete. 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 Processor -- start 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 -- start start 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4104101df0 0x7f41040744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41041027a0 0x7f4104074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.498+0000 7f410afb9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4104106f20 0x7f4104074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f410afb9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4104111f80 con 0x7f4104106f20 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f410afb9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4104111e00 con 0x7f41041027a0 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f410afb9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4104112100 con 0x7f4104101df0 2024-09-15T23:36:12.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4109556640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4104106f20 0x7f4104074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4109556640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4104106f20 0x7f4104074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37434/0 (socket says 172.21.15.89:37434) 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4109556640 1 -- 172.21.15.89:0/2863598746 learned_addr learned my addr 172.21.15.89:0/2863598746 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4108d55640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4104101df0 0x7f41040744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4103fff640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41041027a0 0x7f4104074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4109556640 1 -- 172.21.15.89:0/2863598746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4104101df0 msgr2=0x7f41040744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4109556640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4104101df0 0x7f41040744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4109556640 1 -- 172.21.15.89:0/2863598746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41041027a0 msgr2=0x7f4104074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4109556640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41041027a0 0x7f4104074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4109556640 1 -- 172.21.15.89:0/2863598746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41041a68c0 con 0x7f4104106f20 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4109556640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4104106f20 0x7f4104074f70 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f40fc005ab0 tx=0x7f40fc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4101ffb640 1 -- 172.21.15.89:0/2863598746 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc03f030 con 0x7f4104106f20 2024-09-15T23:36:12.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f410afb9640 1 -- 172.21.15.89:0/2863598746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41041a6b50 con 0x7f4104106f20 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4101ffb640 1 -- 172.21.15.89:0/2863598746 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40fc003ce0 con 0x7f4104106f20 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.499+0000 7f4101ffb640 1 -- 172.21.15.89:0/2863598746 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc035070 con 0x7f4104106f20 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.500+0000 7f410afb9640 1 -- 172.21.15.89:0/2863598746 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41041a7030 con 0x7f4104106f20 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f4101ffb640 1 -- 172.21.15.89:0/2863598746 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f40fc042030 con 0x7f4104106f20 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f410d392740 1 Processor -- start 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f410d392740 1 -- start start 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f410d392740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68060 0x564c93f68440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f410d392740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f72d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f410d392740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 0x564c93f75630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f410d392740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564c93ecd620 con 0x564c93f68980 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f410d392740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564c93e74e50 con 0x564c93f73240 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f410d392740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564c93ec95e0 con 0x564c93f68060 2024-09-15T23:36:12.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f4108d55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f72d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f4108d55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f72d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37436/0 (socket says 172.21.15.89:37436) 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.502+0000 7f4108d55640 1 -- 172.21.15.89:0/151807308 learned_addr learned my addr 172.21.15.89:0/151807308 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4103fff640 1 --2- 172.21.15.89:0/151807308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 0x564c93f75630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4101ffb640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f40f00a3240 0x7f40f00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4103fff640 1 --2- 172.21.15.89:0/151807308 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68060 0x564c93f68440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4108d55640 1 -- 172.21.15.89:0/151807308 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68060 msgr2=0x564c93f68440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4108d55640 1 --2- 172.21.15.89:0/151807308 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68060 0x564c93f68440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4108d55640 1 -- 172.21.15.89:0/151807308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 msgr2=0x564c93f75630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4108d55640 1 --2- 172.21.15.89:0/151807308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 0x564c93f75630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4108d55640 1 -- 172.21.15.89:0/151807308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c93e65d40 con 0x564c93f68980 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4103fff640 1 --2- 172.21.15.89:0/151807308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 0x564c93f75630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4109556640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f40f00a3240 0x7f40f00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4101ffb640 1 -- 172.21.15.89:0/2863598746 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f40fc0f0600 con 0x7f4104106f20 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4108d55640 1 --2- 172.21.15.89:0/151807308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f72d00 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009f60 tx=0x7f40f4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8eec85ec591a569 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f40faffd640 1 -- 172.21.15.89:0/151807308 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4016070 con 0x564c93f68980 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f4109556640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f40f00a3240 0x7f40f00a5700 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0b8c30 tx=0x7f40fc02a410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f40faffd640 1 -- 172.21.15.89:0/151807308 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40f401bb40 con 0x564c93f68980 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.503+0000 7f40faffd640 1 -- 172.21.15.89:0/151807308 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4012cf0 con 0x564c93f68980 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.504+0000 7f410d392740 1 -- 172.21.15.89:0/151807308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 msgr2=0x564c93f72d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.504+0000 7f410d392740 1 --2- 172.21.15.89:0/151807308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f72d00 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009f60 tx=0x7f40f4009f90 comp rx=0 tx=0).stop 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.504+0000 7f410d392740 1 -- 172.21.15.89:0/151807308 shutdown_connections 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.504+0000 7f410d392740 1 --2- 172.21.15.89:0/151807308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 0x564c93f75630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.504+0000 7f410d392740 1 --2- 172.21.15.89:0/151807308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f72d00 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.504+0000 7f410d392740 1 --2- 172.21.15.89:0/151807308 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68060 0x564c93f68440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.504+0000 7f410d392740 1 -- 172.21.15.89:0/151807308 >> 172.21.15.89:0/151807308 conn(0x564c93f645f0 msgr2=0x564c93f649d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:12.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.504+0000 7f410d392740 1 -- 172.21.15.89:0/151807308 shutdown_connections 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.504+0000 7f410d392740 1 -- 172.21.15.89:0/151807308 wait complete. 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.504+0000 7f410d392740 1 -- start start 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f410d392740 1 Processor -- start 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f410d392740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 0x564c93f700e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f410d392740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f70620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f410d392740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f73240 0x564c93f70b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f410d392740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564c93f75e10 con 0x564c93f68980 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f410d392740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564c93e55f10 con 0x564c93f68060 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f410d392740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564c93f75f90 con 0x564c93f73240 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f410d392740 10 client.0 _ll_register_callbacks cb 0x564c93e79db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4108d55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f70620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4108d55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f70620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37448/0 (socket says 172.21.15.89:37448) 2024-09-15T23:36:12.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4108d55640 1 -- 172.21.15.89:0/107441740 learned_addr learned my addr 172.21.15.89:0/107441740 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4103fff640 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f73240 0x564c93f70b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4103fff640 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 0x564c93f700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4108d55640 1 -- 172.21.15.89:0/107441740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f73240 msgr2=0x564c93f70b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4108d55640 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f73240 0x564c93f70b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4108d55640 1 -- 172.21.15.89:0/107441740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 msgr2=0x564c93f700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4108d55640 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 0x564c93f700e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4108d55640 1 -- 172.21.15.89:0/107441740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c93bdc580 con 0x564c93f68980 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.505+0000 7f4103fff640 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 0x564c93f700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f4108d55640 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f70620 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f40f4006010 tx=0x7f40f4009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d185472d43f6430f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f40e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f40e3fff640 1 -- 172.21.15.89:0/107441740 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4016040 con 0x564c93f68980 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f40e3fff640 1 -- 172.21.15.89:0/107441740 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f40f4013e00 con 0x564c93f68980 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f410d392740 1 -- 172.21.15.89:0/107441740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40f401bb40 con 0x564c93f68980 2024-09-15T23:36:12.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f40e3fff640 1 -- 172.21.15.89:0/107441740 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f40f4003970 con 0x564c93f68980 2024-09-15T23:36:12.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f410d392740 10 client.19515 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:12.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f410d392740 1 -- 172.21.15.89:0/107441740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564c93ca6960 con 0x564c93f68980 2024-09-15T23:36:12.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f40e3fff640 1 -- 172.21.15.89:0/107441740 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40f401bb40 con 0x564c93f68980 2024-09-15T23:36:12.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f40e3fff640 1 -- 172.21.15.89:0/107441740 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f40f400f040 con 0x564c93f68980 2024-09-15T23:36:12.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f410d392740 10 client.19515 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:12.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f410d392740 10 client.19515 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:36:12.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f410d392740 20 client.19515 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:12.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f410d392740 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564c93f82c80 0x564c93f85070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f410d392740 4 client.19515 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.506+0000 7f410d392740 1 -- 172.21.15.89:0/107441740 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c93edab90 con 0x564c93f82c80 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.507+0000 7f4103fff640 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564c93f82c80 0x564c93f85070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.507+0000 7f40e3fff640 10 client.19515 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.507+0000 7f4103fff640 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564c93f82c80 0x564c93f85070 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26739a1e45489169 server_cookie=f19937273d887bd in_seq=0 out_seq=0 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f40e3fff640 1 -- 172.21.15.89:0/107441740 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564c93edab90 con 0x564c93f82c80 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f40e3fff640 10 client.19515 handle_command_reply: tid=0 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 client.19515 shutdown 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 -- 172.21.15.89:0/107441740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 msgr2=0x564c93f70620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f70620 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f40f4006010 tx=0x7f40f4009cd0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 -- 172.21.15.89:0/107441740 shutdown_connections 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564c93f82c80 0x564c93f85070 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f73240 0x564c93f70b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.531 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f70620 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 --2- 172.21.15.89:0/107441740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 0x564c93f700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f40e3fff640 0 client.19515 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:12.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 -- 172.21.15.89:0/107441740 >> 172.21.15.89:0/107441740 conn(0x564c93f645f0 msgr2=0x564c93f6a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:12.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 -- 172.21.15.89:0/107441740 shutdown_connections 2024-09-15T23:36:12.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 1 -- 172.21.15.89:0/107441740 wait complete. 2024-09-15T23:36:12.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.508+0000 7f410d392740 20 client.19515 trim_cache size 0 max 16384 2024-09-15T23:36:12.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.529+0000 7f410d392740 1 Processor -- start 2024-09-15T23:36:12.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.529+0000 7f410d392740 1 -- start start 2024-09-15T23:36:12.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f410d392740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 0x564c93f6e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f410d392740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f63640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f410d392740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f73240 0x564c93f8a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f410d392740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f41041b10c0 con 0x564c93f68980 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f410d392740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41041b2c00 con 0x564c93f68060 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f410d392740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f41041b1790 con 0x564c93f73240 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4108d55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f63640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4108d55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f63640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37452/0 (socket says 172.21.15.89:37452) 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4108d55640 1 -- 172.21.15.89:0/3848008043 learned_addr learned my addr 172.21.15.89:0/3848008043 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4103fff640 1 --2- 172.21.15.89:0/3848008043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 0x564c93f6e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4103fff640 1 --2- 172.21.15.89:0/3848008043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f73240 0x564c93f8a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4108d55640 1 -- 172.21.15.89:0/3848008043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f73240 msgr2=0x564c93f8a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4108d55640 1 --2- 172.21.15.89:0/3848008043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f73240 0x564c93f8a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4108d55640 1 -- 172.21.15.89:0/3848008043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 msgr2=0x564c93f6e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:12.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4108d55640 1 --2- 172.21.15.89:0/3848008043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 0x564c93f6e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4108d55640 1 -- 172.21.15.89:0/3848008043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c93f64090 con 0x564c93f68980 2024-09-15T23:36:12.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f4108d55640 1 --2- 172.21.15.89:0/3848008043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f63640 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f40f4004770 tx=0x7f40f4010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47265f3c33d48aec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:12.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.530+0000 7f40fa7fc640 1 -- 172.21.15.89:0/3848008043 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4003d10 con 0x564c93f68980 2024-09-15T23:36:12.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.531+0000 7f40fa7fc640 1 -- 172.21.15.89:0/3848008043 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40f4003eb0 con 0x564c93f68980 2024-09-15T23:36:12.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.531+0000 7f40fa7fc640 1 -- 172.21.15.89:0/3848008043 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4013040 con 0x564c93f68980 2024-09-15T23:36:12.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.531+0000 7f410d392740 1 -- 172.21.15.89:0/3848008043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 msgr2=0x564c93f63640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:12.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.531+0000 7f410d392740 1 --2- 172.21.15.89:0/3848008043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f63640 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f40f4004770 tx=0x7f40f4010dc0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.531+0000 7f410d392740 1 -- 172.21.15.89:0/3848008043 shutdown_connections 2024-09-15T23:36:12.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.531+0000 7f410d392740 1 --2- 172.21.15.89:0/3848008043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f73240 0x564c93f8a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.531+0000 7f410d392740 1 --2- 172.21.15.89:0/3848008043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68980 0x564c93f63640 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.531+0000 7f410d392740 1 --2- 172.21.15.89:0/3848008043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f68060 0x564c93f6e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.531+0000 7f410d392740 1 -- 172.21.15.89:0/3848008043 >> 172.21.15.89:0/3848008043 conn(0x564c93f645f0 msgr2=0x564c93f62f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.531+0000 7f410d392740 1 -- 172.21.15.89:0/3848008043 shutdown_connections 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f410d392740 1 -- 172.21.15.89:0/3848008043 wait complete. 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f410d392740 1 -- start start 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f410d392740 1 Processor -- start 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f410d392740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68060 0x564c93f77210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f410d392740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68980 0x564c93f77750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f410d392740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 0x564c93f77c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f410d392740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f41041b2530 con 0x564c93f68060 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f410d392740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41041b1e60 con 0x564c93f73240 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f410d392740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f41041af080 con 0x564c93f68980 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f410d392740 10 client.0 _ll_register_callbacks cb 0x7f40f4005a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:12.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f4103fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68060 0x564c93f77210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f4103fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68060 0x564c93f77210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37454/0 (socket says 172.21.15.89:37454) 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.532+0000 7f4103fff640 1 -- 172.21.15.89:0/1875350897 learned_addr learned my addr 172.21.15.89:0/1875350897 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f4108d55640 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68980 0x564c93f77750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f4103fff640 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 0x564c93f77c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f4103fff640 1 -- 172.21.15.89:0/1875350897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68980 msgr2=0x564c93f77750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f4103fff640 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68980 0x564c93f77750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f4103fff640 1 -- 172.21.15.89:0/1875350897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 msgr2=0x564c93f77c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f4103fff640 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 0x564c93f77c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f4103fff640 1 -- 172.21.15.89:0/1875350897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c93feca20 con 0x564c93f68060 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f4108d55640 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68980 0x564c93f77750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f4103fff640 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68060 0x564c93f77210 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x564c93e74e90 tx=0x7f40ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f37b1e4e639428 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f40e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:12.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f40e3fff640 1 -- 172.21.15.89:0/1875350897 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ec026030 con 0x564c93f68060 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f40e3fff640 1 -- 172.21.15.89:0/1875350897 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f40ec005020 con 0x564c93f68060 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f410d392740 1 -- 172.21.15.89:0/1875350897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40f4003eb0 con 0x564c93f68060 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f40e3fff640 1 -- 172.21.15.89:0/1875350897 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f40ec033050 con 0x564c93f68060 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f410d392740 10 client.19521 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f410d392740 1 -- 172.21.15.89:0/1875350897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564c93fedda0 con 0x564c93f68060 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.533+0000 7f40e3fff640 1 -- 172.21.15.89:0/1875350897 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40f4003eb0 con 0x564c93f68060 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.534+0000 7f40e3fff640 1 -- 172.21.15.89:0/1875350897 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f40ec00f040 con 0x564c93f68060 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.534+0000 7f410d392740 10 client.19521 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.534+0000 7f410d392740 10 client.19521 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.534+0000 7f410d392740 20 client.19521 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.534+0000 7f410d392740 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564c93fee4f0 0x564c93ff0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.534+0000 7f410d392740 4 client.19521 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "dddc5eba", "release": true, "await": true}] 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.534+0000 7f410d392740 1 -- 172.21.15.89:0/1875350897 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "dddc5eba", "release": true, "await": true}) -- 0x564c93ed50a0 con 0x564c93fee4f0 2024-09-15T23:36:12.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.534+0000 7f4108d55640 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564c93fee4f0 0x564c93ff0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:12.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.534+0000 7f4108d55640 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564c93fee4f0 0x564c93ff0910 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64c3a3aa3d4efcbe server_cookie=72892b3d4ce98a in_seq=0 out_seq=0 2024-09-15T23:36:12.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.534+0000 7f40e3fff640 10 client.19521 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f40e3fff640 1 -- 172.21.15.89:0/1875350897 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x564c93ed50a0 con 0x564c93fee4f0 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f40e3fff640 10 client.19521 handle_command_reply: tid=0 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f410d392740 1 client.19521 shutdown 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f410d392740 1 -- 172.21.15.89:0/1875350897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68060 msgr2=0x564c93f77210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f410d392740 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68060 0x564c93f77210 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x564c93e74e90 tx=0x7f40ec004730 comp rx=0 tx=0).stop 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f410d392740 1 -- 172.21.15.89:0/1875350897 shutdown_connections 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f410d392740 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564c93fee4f0 0x564c93ff0910 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f40e3fff640 0 client.19521 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f410d392740 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c93f73240 0x564c93f77c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f410d392740 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564c93f68980 0x564c93f77750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f410d392740 1 --2- 172.21.15.89:0/1875350897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564c93f68060 0x564c93f77210 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.537+0000 7f410d392740 1 -- 172.21.15.89:0/1875350897 >> 172.21.15.89:0/1875350897 conn(0x564c93f645f0 msgr2=0x564c93f70870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:12.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410d392740 1 -- 172.21.15.89:0/1875350897 shutdown_connections 2024-09-15T23:36:12.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410d392740 1 -- 172.21.15.89:0/1875350897 wait complete. 2024-09-15T23:36:12.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410d392740 20 client.19521 trim_cache size 0 max 16384 2024-09-15T23:36:12.544 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:36:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:36:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:36:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 44, 2024-09-15T23:36:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:36:12.544 INFO:teuthology.orchestra.run.smithi089.stdout: "dddc5eba": { 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 44, 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 141.8, 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:36:12.545 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:36:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:36:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:36:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:36:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:36:12.546 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:36:12.546 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:36:12.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410afb9640 1 -- 172.21.15.89:0/2863598746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f40f00a3240 msgr2=0x7f40f00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:12.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410afb9640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f40f00a3240 0x7f40f00a5700 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0b8c30 tx=0x7f40fc02a410 comp rx=0 tx=0).stop 2024-09-15T23:36:12.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410afb9640 1 -- 172.21.15.89:0/2863598746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4104106f20 msgr2=0x7f4104074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:12.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410afb9640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4104106f20 0x7f4104074f70 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f40fc005ab0 tx=0x7f40fc0042f0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410afb9640 1 -- 172.21.15.89:0/2863598746 shutdown_connections 2024-09-15T23:36:12.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410afb9640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f40f00a3240 0x7f40f00a5700 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410afb9640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4104106f20 0x7f4104074f70 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410afb9640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41041027a0 0x7f4104074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410afb9640 1 --2- 172.21.15.89:0/2863598746 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4104101df0 0x7f41040744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:12.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.538+0000 7f410afb9640 1 -- 172.21.15.89:0/2863598746 >> 172.21.15.89:0/2863598746 conn(0x7f41040fd8f0 msgr2=0x7f41040fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:12.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.539+0000 7f410afb9640 1 -- 172.21.15.89:0/2863598746 shutdown_connections 2024-09-15T23:36:12.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:12.539+0000 7f410afb9640 1 -- 172.21.15.89:0/2863598746 wait complete. 2024-09-15T23:36:12.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:12 smithi120 ceph-mon[35250]: pgmap v715: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-15T23:36:12.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:12 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1405954646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:36:12.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:12 smithi174 ceph-mon[35586]: pgmap v715: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-15T23:36:12.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:12 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1405954646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:36:13.271 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:13.286 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'dddc5eba', total seconds elapsed: 25.9 2024-09-15T23:36:13.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.758+0000 7f8b3a68f640 1 -- 172.21.15.89:0/3848169080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b34102d40 msgr2=0x7f8b34103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.758+0000 7f8b3a68f640 1 --2- 172.21.15.89:0/3848169080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b34102d40 0x7f8b34103120 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009900 tx=0x7f8b2402a480 comp rx=0 tx=0).stop 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.758+0000 7f8b3a68f640 1 -- 172.21.15.89:0/3848169080 shutdown_connections 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.758+0000 7f8b3a68f640 1 --2- 172.21.15.89:0/3848169080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b3410c170 0x7f8b3410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.758+0000 7f8b3a68f640 1 --2- 172.21.15.89:0/3848169080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b34103660 0x7f8b3410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.758+0000 7f8b3a68f640 1 --2- 172.21.15.89:0/3848169080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b34102d40 0x7f8b34103120 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.758+0000 7f8b3a68f640 1 -- 172.21.15.89:0/3848169080 >> 172.21.15.89:0/3848169080 conn(0x7f8b340fc780 msgr2=0x7f8b340feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.758+0000 7f8b3a68f640 1 -- 172.21.15.89:0/3848169080 shutdown_connections 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.758+0000 7f8b3a68f640 1 -- 172.21.15.89:0/3848169080 wait complete. 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.759+0000 7f8b3a68f640 1 Processor -- start 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.759+0000 7f8b3a68f640 1 -- start start 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.759+0000 7f8b3a68f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b34102d40 0x7f8b341985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.759+0000 7f8b3a68f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b34103660 0x7f8b34198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.759+0000 7f8b3a68f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3410c170 0x7f8b3419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.759+0000 7f8b3a68f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8b34112030 con 0x7f8b34103660 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.759+0000 7f8b3a68f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b34111eb0 con 0x7f8b3410c170 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.759+0000 7f8b3a68f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b341121b0 con 0x7f8b34102d40 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b337fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b34103660 0x7f8b34198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b337fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b34103660 0x7f8b34198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37482/0 (socket says 172.21.15.89:37482) 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b337fe640 1 -- 172.21.15.89:0/92672403 learned_addr learned my addr 172.21.15.89:0/92672403 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b33fff640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b34102d40 0x7f8b341985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b337fe640 1 -- 172.21.15.89:0/92672403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b34102d40 msgr2=0x7f8b341985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b337fe640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b34102d40 0x7f8b341985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b337fe640 1 -- 172.21.15.89:0/92672403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3410c170 msgr2=0x7f8b3419ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3410c170 0x7f8b3419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b337fe640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3410c170 0x7f8b3419ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b337fe640 1 -- 172.21.15.89:0/92672403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3419d550 con 0x7f8b34103660 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3410c170 0x7f8b3419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b337fe640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b34103660 0x7f8b34198ae0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800b3f0 tx=0x7f8b2800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b317fa640 1 -- 172.21.15.89:0/92672403 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28015020 con 0x7f8b34103660 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b3a68f640 1 -- 172.21.15.89:0/92672403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b3419d840 con 0x7f8b34103660 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b317fa640 1 -- 172.21.15.89:0/92672403 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b28011d10 con 0x7f8b34103660 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b317fa640 1 -- 172.21.15.89:0/92672403 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2800f2e0 con 0x7f8b34103660 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.760+0000 7f8b3a68f640 1 -- 172.21.15.89:0/92672403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b341a5120 con 0x7f8b34103660 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.762+0000 7f8b317fa640 1 -- 172.21.15.89:0/92672403 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8b28018020 con 0x7f8b34103660 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b3ca68740 1 Processor -- start 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b3ca68740 1 -- start start 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b01b40 0x560e74b00090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b00a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b0b370 0x560e74b0d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560e74a662f0 con 0x560e74b005d0 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560e74a0dcf0 con 0x560e74b01b40 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560e74a622b0 con 0x560e74b0b370 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b00a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b317fa640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b1c0a3240 0x7f8b1c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b00a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37494/0 (socket says 172.21.15.89:37494) 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b33fff640 1 -- 172.21.15.89:0/2269064935 learned_addr learned my addr 172.21.15.89:0/2269064935 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b337fe640 1 --2- 172.21.15.89:0/2269064935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b0b370 0x560e74b0d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/2269064935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b01b40 0x560e74b00090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.763+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b1c0a3240 0x7f8b1c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b317fa640 1 -- 172.21.15.89:0/92672403 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8b280cf610 con 0x7f8b34103660 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b33fff640 1 -- 172.21.15.89:0/2269064935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b0b370 msgr2=0x560e74b0d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:13.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b33fff640 1 --2- 172.21.15.89:0/2269064935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b0b370 0x560e74b0d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b33fff640 1 -- 172.21.15.89:0/2269064935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b01b40 msgr2=0x560e74b00090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b33fff640 1 --2- 172.21.15.89:0/2269064935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b01b40 0x560e74b00090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b33fff640 1 -- 172.21.15.89:0/2269064935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e749f23a0 con 0x560e74b005d0 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/2269064935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b01b40 0x560e74b00090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b33fff640 1 --2- 172.21.15.89:0/2269064935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b00a10 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f8b24006010 tx=0x7f8b24032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cecec97998f91db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b1a7fc640 1 -- 172.21.15.89:0/2269064935 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2403f070 con 0x560e74b005d0 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b1c0a3240 0x7f8b1c0a5700 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00e6b0 tx=0x7f8b2c008220 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b1a7fc640 1 -- 172.21.15.89:0/2269064935 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b240317b0 con 0x560e74b005d0 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.764+0000 7f8b1a7fc640 1 -- 172.21.15.89:0/2269064935 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24035070 con 0x560e74b005d0 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 -- 172.21.15.89:0/2269064935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 msgr2=0x560e74b00a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/2269064935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b00a10 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f8b24006010 tx=0x7f8b24032ce0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 -- 172.21.15.89:0/2269064935 shutdown_connections 2024-09-15T23:36:13.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/2269064935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b0b370 0x560e74b0d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/2269064935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b00a10 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/2269064935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b01b40 0x560e74b00090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 -- 172.21.15.89:0/2269064935 >> 172.21.15.89:0/2269064935 conn(0x560e74ac9310 msgr2=0x560e74afcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 -- 172.21.15.89:0/2269064935 shutdown_connections 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 -- 172.21.15.89:0/2269064935 wait complete. 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 -- start start 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 Processor -- start 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.765+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b0ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b07f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 0x560e74b084f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560e74b0e010 con 0x560e74b005d0 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560e74a21060 con 0x560e74b0b370 2024-09-15T23:36:13.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560e74b0e190 con 0x560e74b01b40 2024-09-15T23:36:13.808 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:36:13.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b3ca68740 10 client.0 _ll_register_callbacks cb 0x560e747748c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:13.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b0ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b0ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37506/0 (socket says 172.21.15.89:37506) 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b33fff640 1 -- 172.21.15.89:0/428213954 learned_addr learned my addr 172.21.15.89:0/428213954 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b337fe640 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b07f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 0x560e74b084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b33fff640 1 -- 172.21.15.89:0/428213954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 msgr2=0x560e74b07f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b33fff640 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b07f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b33fff640 1 -- 172.21.15.89:0/428213954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 msgr2=0x560e74b084f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b33fff640 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 0x560e74b084f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b33fff640 1 -- 172.21.15.89:0/428213954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e747d73c0 con 0x560e74b005d0 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b33fff640 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b0ace0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f8b24031b90 tx=0x7f8b240315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d288c9fadae93c00 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b137fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b137fe640 1 -- 172.21.15.89:0/428213954 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2403f070 con 0x560e74b005d0 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b137fe640 1 -- 172.21.15.89:0/428213954 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8b24004250 con 0x560e74b005d0 2024-09-15T23:36:13.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b3ca68740 1 -- 172.21.15.89:0/428213954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b240317b0 con 0x560e74b005d0 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b137fe640 1 -- 172.21.15.89:0/428213954 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8b24054050 con 0x560e74b005d0 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b3ca68740 10 client.19533 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.766+0000 7f8b3ca68740 1 -- 172.21.15.89:0/428213954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560e74b19a20 con 0x560e74b005d0 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.767+0000 7f8b137fe640 1 -- 172.21.15.89:0/428213954 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b240317b0 con 0x560e74b005d0 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.767+0000 7f8b137fe640 1 -- 172.21.15.89:0/428213954 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8b2404d660 con 0x560e74b005d0 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.767+0000 7f8b3ca68740 10 client.19533 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.767+0000 7f8b3ca68740 10 client.19533 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.767+0000 7f8b3ca68740 20 client.19533 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.767+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x560e74b1ad70 0x560e74b1d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.767+0000 7f8b3ca68740 4 client.19533 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.767+0000 7f8b3ca68740 1 -- 172.21.15.89:0/428213954 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e74a73860 con 0x560e74b1ad70 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.767+0000 7f8b337fe640 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x560e74b1ad70 0x560e74b1d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.768+0000 7f8b337fe640 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x560e74b1ad70 0x560e74b1d160 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c5cd6bbdcb8678a9 server_cookie=c5a6cefc7e2665b8 in_seq=0 out_seq=0 2024-09-15T23:36:13.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.768+0000 7f8b137fe640 10 client.19533 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.768+0000 7f8b137fe640 1 -- 172.21.15.89:0/428213954 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560e74a73860 con 0x560e74b1ad70 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.768+0000 7f8b137fe640 10 client.19533 handle_command_reply: tid=0 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.768+0000 7f8b3ca68740 1 client.19533 shutdown 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.768+0000 7f8b3ca68740 1 -- 172.21.15.89:0/428213954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 msgr2=0x560e74b0ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.768+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b0ace0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f8b24031b90 tx=0x7f8b240315b0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.768+0000 7f8b3ca68740 1 -- 172.21.15.89:0/428213954 shutdown_connections 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.768+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x560e74b1ad70 0x560e74b1d160 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.768+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 0x560e74b084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.769+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b07f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.769+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/428213954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b0ace0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.769+0000 7f8b3ca68740 1 -- 172.21.15.89:0/428213954 >> 172.21.15.89:0/428213954 conn(0x560e74ac9310 msgr2=0x560e74b02960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.769+0000 7f8b137fe640 0 client.19533 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:36:13.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.769+0000 7f8b3ca68740 1 -- 172.21.15.89:0/428213954 shutdown_connections 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.769+0000 7f8b3ca68740 1 -- 172.21.15.89:0/428213954 wait complete. 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.769+0000 7f8b3ca68740 20 client.19533 trim_cache size 0 max 16384 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.781+0000 7f8b3ca68740 1 Processor -- start 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.781+0000 7f8b3ca68740 1 -- start start 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b1fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b202e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 0x560e74b05210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8b341aefa0 con 0x560e74b005d0 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b341b0ae0 con 0x560e74b0b370 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b341af670 con 0x560e74b01b40 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b1fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b33fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b1fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37516/0 (socket says 172.21.15.89:37516) 2024-09-15T23:36:13.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b33fff640 1 -- 172.21.15.89:0/1938216802 learned_addr learned my addr 172.21.15.89:0/1938216802 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b337fe640 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 0x560e74b05210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b33fff640 1 -- 172.21.15.89:0/1938216802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 msgr2=0x560e74b202e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b33fff640 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b202e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b33fff640 1 -- 172.21.15.89:0/1938216802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 msgr2=0x560e74b05210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b33fff640 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 0x560e74b05210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b33fff640 1 -- 172.21.15.89:0/1938216802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e74b002d0 con 0x560e74b005d0 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b337fe640 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 0x560e74b05210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b33fff640 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b1fda0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f8b2404dba0 tx=0x7f8b2402ab00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fb0272facff1b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b19ffb640 1 -- 172.21.15.89:0/1938216802 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2403f040 con 0x560e74b005d0 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b19ffb640 1 -- 172.21.15.89:0/1938216802 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b24048030 con 0x560e74b005d0 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.782+0000 7f8b19ffb640 1 -- 172.21.15.89:0/1938216802 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24035040 con 0x560e74b005d0 2024-09-15T23:36:13.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.783+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1938216802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 msgr2=0x560e74b1fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.783+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b1fda0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f8b2404dba0 tx=0x7f8b2402ab00 comp rx=0 tx=0).stop 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.783+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1938216802 shutdown_connections 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.783+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b0b370 0x560e74b05210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.783+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.783+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/1938216802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b005d0 0x560e74b1fda0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.783+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1938216802 >> 172.21.15.89:0/1938216802 conn(0x560e74ac9310 msgr2=0x560e74b07080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.783+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1938216802 shutdown_connections 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.783+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1938216802 wait complete. 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.783+0000 7f8b3ca68740 1 -- start start 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b3ca68740 1 Processor -- start 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b005d0 0x560e74b204b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b1d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b3ca68740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b0b370 0x560e74b1dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8b341b0410 con 0x560e74b0b370 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b341afd40 con 0x560e74b005d0 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b3ca68740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b341acf80 con 0x560e74b01b40 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b3ca68740 10 client.0 _ll_register_callbacks cb 0x7f8b2809b610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b38c2c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b0b370 0x560e74b1dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b38c2c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b0b370 0x560e74b1dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:37524/0 (socket says 172.21.15.89:37524) 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b38c2c640 1 -- 172.21.15.89:0/1819731220 learned_addr learned my addr 172.21.15.89:0/1819731220 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b337fe640 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b1d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b33fff640 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b005d0 0x560e74b204b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b38c2c640 1 -- 172.21.15.89:0/1819731220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 msgr2=0x560e74b1d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b1d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b38c2c640 1 -- 172.21.15.89:0/1819731220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b005d0 msgr2=0x560e74b204b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b005d0 0x560e74b204b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b38c2c640 1 -- 172.21.15.89:0/1819731220 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e74b1e570 con 0x560e74b0b370 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b33fff640 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b005d0 0x560e74b204b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b38c2c640 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b0b370 0x560e74b1dd90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00abc0 tx=0x7f8b2c004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d8d5db5420491 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.784+0000 7f8b137fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b137fe640 1 -- 172.21.15.89:0/1819731220 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c014070 con 0x560e74b0b370 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b137fe640 1 -- 172.21.15.89:0/1819731220 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8b2c00ce00 con 0x560e74b0b370 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1819731220 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b24048030 con 0x560e74b0b370 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b137fe640 1 -- 172.21.15.89:0/1819731220 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8b2c025440 con 0x560e74b0b370 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b3ca68740 10 client.19539 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1819731220 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560e74b85630 con 0x560e74b0b370 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b137fe640 1 -- 172.21.15.89:0/1819731220 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b24048030 con 0x560e74b0b370 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b137fe640 1 -- 172.21.15.89:0/1819731220 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8b2c023070 con 0x560e74b0b370 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b3ca68740 10 client.19539 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:13.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b3ca68740 10 client.19539 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b3ca68740 20 client.19539 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x560e74b85dd0 0x560e74b881f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b3ca68740 4 client.19539 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1819731220 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e74a6dd70 con 0x560e74b85dd0 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.785+0000 7f8b33fff640 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x560e74b85dd0 0x560e74b881f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b137fe640 10 client.19539 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b33fff640 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x560e74b85dd0 0x560e74b881f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a5e577cee0a6048c server_cookie=bce5b4e525fa89cb in_seq=0 out_seq=0 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b137fe640 1 -- 172.21.15.89:0/1819731220 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e74a6dd70 con 0x560e74b85dd0 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b137fe640 10 client.19539 handle_command_reply: tid=0 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 client.19539 shutdown 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1819731220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b0b370 msgr2=0x560e74b1dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b0b370 0x560e74b1dd90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00abc0 tx=0x7f8b2c004b50 comp rx=0 tx=0).stop 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1819731220 shutdown_connections 2024-09-15T23:36:13.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x560e74b85dd0 0x560e74b881f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560e74b0b370 0x560e74b1dd90 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560e74b01b40 0x560e74b1d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 --2- 172.21.15.89:0/1819731220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e74b005d0 0x560e74b204b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1819731220 >> 172.21.15.89:0/1819731220 conn(0x560e74ac9310 msgr2=0x560e74afb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b137fe640 0 client.19539 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1819731220 shutdown_connections 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 1 -- 172.21.15.89:0/1819731220 wait complete. 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.786+0000 7f8b3ca68740 20 client.19539 trim_cache size 0 max 16384 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 -- 172.21.15.89:0/92672403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b1c0a3240 msgr2=0x7f8b1c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b1c0a3240 0x7f8b1c0a5700 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00e6b0 tx=0x7f8b2c008220 comp rx=0 tx=0).stop 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 -- 172.21.15.89:0/92672403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b34103660 msgr2=0x7f8b34198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b34103660 0x7f8b34198ae0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800b3f0 tx=0x7f8b2800b8b0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 -- 172.21.15.89:0/92672403 shutdown_connections 2024-09-15T23:36:13.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8b1c0a3240 0x7f8b1c0a5700 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b3410c170 0x7f8b3419ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b34103660 0x7f8b34198ae0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 --2- 172.21.15.89:0/92672403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b34102d40 0x7f8b341985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:13.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 -- 172.21.15.89:0/92672403 >> 172.21.15.89:0/92672403 conn(0x7f8b340fc780 msgr2=0x7f8b340fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:13.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 -- 172.21.15.89:0/92672403 shutdown_connections 2024-09-15T23:36:13.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:13.787+0000 7f8b3a68f640 1 -- 172.21.15.89:0/92672403 wait complete. 2024-09-15T23:36:14.480 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:36:14.728 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:14.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:14 smithi089 ceph-mon[30933]: pgmap v716: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-15T23:36:14.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:14 smithi120 ceph-mon[35250]: pgmap v716: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-15T23:36:14.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:14 smithi174 ceph-mon[35586]: pgmap v716: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-09-15T23:36:15.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.187+0000 7fc99a28d640 1 -- 172.21.15.89:0/2232911161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc994100a80 msgr2=0x7fc994100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.187+0000 7fc99a28d640 1 --2- 172.21.15.89:0/2232911161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc994100a80 0x7fc994100ee0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fc988009930 tx=0x7fc98802a420 comp rx=0 tx=0).stop 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.187+0000 7fc99a28d640 1 -- 172.21.15.89:0/2232911161 shutdown_connections 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.187+0000 7fc99a28d640 1 --2- 172.21.15.89:0/2232911161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc994101420 0x7fc99410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.187+0000 7fc99a28d640 1 --2- 172.21.15.89:0/2232911161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc994100a80 0x7fc994100ee0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.187+0000 7fc99a28d640 1 --2- 172.21.15.89:0/2232911161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc994106a80 0x7fc994106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.187+0000 7fc99a28d640 1 -- 172.21.15.89:0/2232911161 >> 172.21.15.89:0/2232911161 conn(0x7fc9940fc7a0 msgr2=0x7fc9940febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.187+0000 7fc99a28d640 1 -- 172.21.15.89:0/2232911161 shutdown_connections 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.187+0000 7fc99a28d640 1 -- 172.21.15.89:0/2232911161 wait complete. 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.188+0000 7fc99a28d640 1 Processor -- start 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.188+0000 7fc99a28d640 1 -- start start 2024-09-15T23:36:15.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.188+0000 7fc99a28d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc994100a80 0x7fc9941972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.188+0000 7fc99a28d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc994101420 0x7fc994197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.188+0000 7fc99a28d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc994106a80 0x7fc99419bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.188+0000 7fc99a28d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc994110e80 con 0x7fc994101420 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.188+0000 7fc99a28d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc994110d00 con 0x7fc994100a80 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.188+0000 7fc99a28d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc994111000 con 0x7fc994106a80 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.188+0000 7fc9937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc994101420 0x7fc994197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc994101420 0x7fc994197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42104/0 (socket says 172.21.15.89:42104) 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9937fe640 1 -- 172.21.15.89:0/1421239484 learned_addr learned my addr 172.21.15.89:0/1421239484 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc993fff640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc994100a80 0x7fc9941972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc99882a640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc994106a80 0x7fc99419bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9937fe640 1 -- 172.21.15.89:0/1421239484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc994106a80 msgr2=0x7fc99419bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9937fe640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc994106a80 0x7fc99419bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9937fe640 1 -- 172.21.15.89:0/1421239484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc994100a80 msgr2=0x7fc9941972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9937fe640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc994100a80 0x7fc9941972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9937fe640 1 -- 172.21.15.89:0/1421239484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99419c270 con 0x7fc994101420 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9937fe640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc994101420 0x7fc994197800 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fc988009230 tx=0x7fc988004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9917fa640 1 -- 172.21.15.89:0/1421239484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98803f030 con 0x7fc994101420 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9917fa640 1 -- 172.21.15.89:0/1421239484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc988003ad0 con 0x7fc994101420 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc99a28d640 1 -- 172.21.15.89:0/1421239484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99419c500 con 0x7fc994101420 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc9917fa640 1 -- 172.21.15.89:0/1421239484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc988035070 con 0x7fc994101420 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.189+0000 7fc99a28d640 1 -- 172.21.15.89:0/1421239484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9941a3e40 con 0x7fc994101420 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.192+0000 7fc9917fa640 1 -- 172.21.15.89:0/1421239484 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc988042030 con 0x7fc994101420 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.192+0000 7fc99c666740 1 Processor -- start 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.192+0000 7fc99c666740 1 -- start start 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.192+0000 7fc99c666740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753fd50 0x55c45753e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.192+0000 7fc99c666740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753e7e0 0x55c45753ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.192+0000 7fc99c666740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.192+0000 7fc99c666740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c4574a22f0 con 0x55c45753e7e0 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.192+0000 7fc99c666740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c457449cf0 con 0x55c457549580 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.192+0000 7fc99c666740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c45749e2b0 con 0x55c45753fd50 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc993fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753e7e0 0x55c45753ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc993fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753e7e0 0x55c45753ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42116/0 (socket says 172.21.15.89:42116) 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc993fff640 1 -- 172.21.15.89:0/2166213201 learned_addr learned my addr 172.21.15.89:0/2166213201 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc99882a640 1 --2- 172.21.15.89:0/2166213201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753fd50 0x55c45753e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc993fff640 1 -- 172.21.15.89:0/2166213201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753fd50 msgr2=0x55c45753e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc993fff640 1 --2- 172.21.15.89:0/2166213201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753fd50 0x55c45753e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc9917fa640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc97c0a3240 0x7fc97c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc993fff640 1 -- 172.21.15.89:0/2166213201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 msgr2=0x55c45754b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc9937fe640 1 --2- 172.21.15.89:0/2166213201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc993fff640 1 --2- 172.21.15.89:0/2166213201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc993fff640 1 -- 172.21.15.89:0/2166213201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c45742e3a0 con 0x55c45753e7e0 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc9937fe640 1 --2- 172.21.15.89:0/2166213201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc99882a640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc97c0a3240 0x7fc97c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc9917fa640 1 -- 172.21.15.89:0/1421239484 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc9880f0340 con 0x7fc994101420 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc993fff640 1 --2- 172.21.15.89:0/2166213201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753e7e0 0x55c45753ec20 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fc98400ac90 tx=0x7fc98400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3601225230e72cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc97a7fc640 1 -- 172.21.15.89:0/2166213201 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98401d070 con 0x55c45753e7e0 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc97a7fc640 1 -- 172.21.15.89:0/2166213201 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc984018b20 con 0x55c45753e7e0 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.193+0000 7fc97a7fc640 1 -- 172.21.15.89:0/2166213201 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984010b40 con 0x55c45753e7e0 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99c666740 1 -- 172.21.15.89:0/2166213201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753e7e0 msgr2=0x55c45753ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99882a640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc97c0a3240 0x7fc97c0a5700 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc98c005fd0 tx=0x7fc98c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99c666740 1 --2- 172.21.15.89:0/2166213201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753e7e0 0x55c45753ec20 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fc98400ac90 tx=0x7fc98400f790 comp rx=0 tx=0).stop 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99c666740 1 -- 172.21.15.89:0/2166213201 shutdown_connections 2024-09-15T23:36:15.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99c666740 1 --2- 172.21.15.89:0/2166213201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99c666740 1 --2- 172.21.15.89:0/2166213201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753e7e0 0x55c45753ec20 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99c666740 1 --2- 172.21.15.89:0/2166213201 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753fd50 0x55c45753e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99c666740 1 -- 172.21.15.89:0/2166213201 >> 172.21.15.89:0/2166213201 conn(0x55c457505310 msgr2=0x55c45753ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99c666740 1 -- 172.21.15.89:0/2166213201 shutdown_connections 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99c666740 1 -- 172.21.15.89:0/2166213201 wait complete. 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.194+0000 7fc99c666740 1 -- start start 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc99c666740 1 Processor -- start 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc99c666740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c45753e7e0 0x55c457548f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc99c666740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c4575461c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc99c666740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c457549580 0x55c457546740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc99c666740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c45754c280 con 0x55c45753fd50 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc99c666740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c45745d060 con 0x55c45753e7e0 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc99c666740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c45754c400 con 0x55c457549580 2024-09-15T23:36:15.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc99c666740 10 client.0 _ll_register_callbacks cb 0x55c4571b08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc9937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c4575461c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc9937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c4575461c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42128/0 (socket says 172.21.15.89:42128) 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc9937fe640 1 -- 172.21.15.89:0/309753764 learned_addr learned my addr 172.21.15.89:0/309753764 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc99882a640 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c457549580 0x55c457546740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc993fff640 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c45753e7e0 0x55c457548f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc9937fe640 1 -- 172.21.15.89:0/309753764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c457549580 msgr2=0x55c457546740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc9937fe640 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c457549580 0x55c457546740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc9937fe640 1 -- 172.21.15.89:0/309753764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c45753e7e0 msgr2=0x55c457548f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc9937fe640 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c45753e7e0 0x55c457548f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc9937fe640 1 -- 172.21.15.89:0/309753764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4572133c0 con 0x55c45753fd50 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc993fff640 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c45753e7e0 0x55c457548f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc9937fe640 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c4575461c0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fc9880bc500 tx=0x7fc988009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a43de4992616f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:15.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.195+0000 7fc9737fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc9737fe640 1 -- 172.21.15.89:0/309753764 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc988049030 con 0x55c45753fd50 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc9737fe640 1 -- 172.21.15.89:0/309753764 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc9880ba870 con 0x55c45753fd50 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc99c666740 1 -- 172.21.15.89:0/309753764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc984018b20 con 0x55c45753fd50 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc9737fe640 1 -- 172.21.15.89:0/309753764 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc9880bbed0 con 0x55c45753fd50 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc99c666740 10 client.19551 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc99c666740 1 -- 172.21.15.89:0/309753764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c457557c20 con 0x55c45753fd50 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc9737fe640 1 -- 172.21.15.89:0/309753764 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc984018b20 con 0x55c45753fd50 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc9737fe640 1 -- 172.21.15.89:0/309753764 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc988003c80 con 0x55c45753fd50 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc99c666740 10 client.19551 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc99c666740 10 client.19551 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc99c666740 20 client.19551 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc99c666740 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c457558f70 0x55c45755b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc99c666740 4 client.19551 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:15.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc99c666740 1 -- 172.21.15.89:0/309753764 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4574af860 con 0x55c457558f70 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.196+0000 7fc993fff640 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c457558f70 0x55c45755b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.197+0000 7fc9737fe640 10 client.19551 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.197+0000 7fc993fff640 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c457558f70 0x55c45755b380 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=94bbdec8215121eb server_cookie=231c082b7e494663 in_seq=0 out_seq=0 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.197+0000 7fc9737fe640 1 -- 172.21.15.89:0/309753764 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c4574af860 con 0x55c457558f70 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.197+0000 7fc9737fe640 10 client.19551 handle_command_reply: tid=0 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 client.19551 shutdown 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 -- 172.21.15.89:0/309753764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 msgr2=0x55c4575461c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c4575461c0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fc9880bc500 tx=0x7fc988009620 comp rx=0 tx=0).stop 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 -- 172.21.15.89:0/309753764 shutdown_connections 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c457558f70 0x55c45755b380 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c457549580 0x55c457546740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c4575461c0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 --2- 172.21.15.89:0/309753764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c45753e7e0 0x55c457548f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc9737fe640 0 client.19551 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:36:15.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 -- 172.21.15.89:0/309753764 >> 172.21.15.89:0/309753764 conn(0x55c457505310 msgr2=0x55c457540b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:15.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 -- 172.21.15.89:0/309753764 shutdown_connections 2024-09-15T23:36:15.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 1 -- 172.21.15.89:0/309753764 wait complete. 2024-09-15T23:36:15.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.198+0000 7fc99c666740 20 client.19551 trim_cache size 0 max 16384 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc99c666740 1 Processor -- start 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc99c666740 1 -- start start 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc99c666740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753e7e0 0x55c457548130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc99c666740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c45754a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc99c666740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc99c666740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc9941acb40 con 0x55c45753fd50 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc99c666740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc9941ae680 con 0x55c457549580 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc99c666740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc9941ad210 con 0x55c45753e7e0 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc9937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c45754a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc9937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c45754a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42138/0 (socket says 172.21.15.89:42138) 2024-09-15T23:36:15.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc9937fe640 1 -- 172.21.15.89:0/723027126 learned_addr learned my addr 172.21.15.89:0/723027126 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc993fff640 1 --2- 172.21.15.89:0/723027126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753e7e0 0x55c457548130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc9937fe640 1 -- 172.21.15.89:0/723027126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753e7e0 msgr2=0x55c457548130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc9937fe640 1 --2- 172.21.15.89:0/723027126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753e7e0 0x55c457548130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc9937fe640 1 -- 172.21.15.89:0/723027126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 msgr2=0x55c45754aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc9937fe640 1 --2- 172.21.15.89:0/723027126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc9937fe640 1 -- 172.21.15.89:0/723027126 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c457544710 con 0x55c45753fd50 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc9937fe640 1 --2- 172.21.15.89:0/723027126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c45754a970 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fc988000c00 tx=0x7fc98804bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52f7c56ac5e43c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.211+0000 7fc979ffb640 1 -- 172.21.15.89:0/723027126 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc988049070 con 0x55c45753fd50 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc979ffb640 1 -- 172.21.15.89:0/723027126 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9880bcc70 con 0x55c45753fd50 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc979ffb640 1 -- 172.21.15.89:0/723027126 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98803bd50 con 0x55c45753fd50 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc99c666740 1 -- 172.21.15.89:0/723027126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 msgr2=0x55c45754a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc99c666740 1 --2- 172.21.15.89:0/723027126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c45754a970 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fc988000c00 tx=0x7fc98804bfb0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc99c666740 1 -- 172.21.15.89:0/723027126 shutdown_connections 2024-09-15T23:36:15.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc99c666740 1 --2- 172.21.15.89:0/723027126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc99c666740 1 --2- 172.21.15.89:0/723027126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c45754a970 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc99c666740 1 --2- 172.21.15.89:0/723027126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753e7e0 0x55c457548130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc99c666740 1 -- 172.21.15.89:0/723027126 >> 172.21.15.89:0/723027126 conn(0x55c457505310 msgr2=0x55c457547a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc99c666740 1 -- 172.21.15.89:0/723027126 shutdown_connections 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc99c666740 1 -- 172.21.15.89:0/723027126 wait complete. 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.212+0000 7fc99c666740 1 -- start start 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc99c666740 1 Processor -- start 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc99c666740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753e7e0 0x55c457551880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc99c666740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c457551dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc99c666740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc99c666740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc9941adfb0 con 0x55c45753fd50 2024-09-15T23:36:15.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc99c666740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc9941ad8e0 con 0x55c457549580 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc99c666740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc9941aab20 con 0x55c45753e7e0 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc99c666740 10 client.0 _ll_register_callbacks cb 0x7fc988009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc9937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c457551dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc9937fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c457551dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42146/0 (socket says 172.21.15.89:42146) 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc9937fe640 1 -- 172.21.15.89:0/1543938263 learned_addr learned my addr 172.21.15.89:0/1543938263 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc993fff640 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753e7e0 0x55c457551880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc99882a640 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc9937fe640 1 -- 172.21.15.89:0/1543938263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753e7e0 msgr2=0x55c457551880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc9937fe640 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753e7e0 0x55c457551880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc9937fe640 1 -- 172.21.15.89:0/1543938263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 msgr2=0x55c45754c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc9937fe640 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc9937fe640 1 -- 172.21.15.89:0/1543938263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c45754ce80 con 0x55c45753fd50 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.213+0000 7fc99882a640 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc9937fe640 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c457551dc0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fc9880f06f0 tx=0x7fc9880b8880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ada71c2cb5d71a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc9737fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc9737fe640 1 -- 172.21.15.89:0/1543938263 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc988086060 con 0x55c45753fd50 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc9737fe640 1 -- 172.21.15.89:0/1543938263 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc988031bd0 con 0x55c45753fd50 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc99c666740 1 -- 172.21.15.89:0/1543938263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9880bcc70 con 0x55c45753fd50 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc9737fe640 1 -- 172.21.15.89:0/1543938263 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc9880da070 con 0x55c45753fd50 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc99c666740 10 client.19557 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc99c666740 1 -- 172.21.15.89:0/1543938263 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c45754ce80 con 0x55c45753fd50 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc9737fe640 1 -- 172.21.15.89:0/1543938263 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9880bcc70 con 0x55c45753fd50 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc9737fe640 1 -- 172.21.15.89:0/1543938263 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc9880872c0 con 0x55c45753fd50 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc99c666740 10 client.19557 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:15.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc99c666740 10 client.19557 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc99c666740 20 client.19557 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc99c666740 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c4575c49e0 0x55c4575c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc99c666740 4 client.19557 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.214+0000 7fc99c666740 1 -- 172.21.15.89:0/1543938263 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4574a9d70 con 0x55c4575c49e0 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc993fff640 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c4575c49e0 0x55c4575c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc993fff640 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c4575c49e0 0x55c4575c6e00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=49d569bd0ccd99f7 server_cookie=3a97bb5594b9a69d in_seq=0 out_seq=0 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc9737fe640 10 client.19557 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc9737fe640 1 -- 172.21.15.89:0/1543938263 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4574a9d70 con 0x55c4575c49e0 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc9737fe640 10 client.19557 handle_command_reply: tid=0 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc99c666740 1 client.19557 shutdown 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc99c666740 1 -- 172.21.15.89:0/1543938263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 msgr2=0x55c457551dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:15.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc99c666740 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c457551dc0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fc9880f06f0 tx=0x7fc9880b8880 comp rx=0 tx=0).stop 2024-09-15T23:36:15.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc99c666740 1 -- 172.21.15.89:0/1543938263 shutdown_connections 2024-09-15T23:36:15.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc99c666740 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c4575c49e0 0x55c4575c6e00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc99c666740 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c457549580 0x55c45754c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc99c666740 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c45753fd50 0x55c457551dc0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc9737fe640 0 client.19557 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:36:15.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc99c666740 1 --2- 172.21.15.89:0/1543938263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c45753e7e0 0x55c457551880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.215+0000 7fc99c666740 1 -- 172.21.15.89:0/1543938263 >> 172.21.15.89:0/1543938263 conn(0x55c457505310 msgr2=0x55c457548240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:15.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99c666740 1 -- 172.21.15.89:0/1543938263 shutdown_connections 2024-09-15T23:36:15.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99c666740 1 -- 172.21.15.89:0/1543938263 wait complete. 2024-09-15T23:36:15.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99c666740 20 client.19557 trim_cache size 0 max 16384 2024-09-15T23:36:15.229 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:36:15.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 -- 172.21.15.89:0/1421239484 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc97c0a3240 msgr2=0x7fc97c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:15.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc97c0a3240 0x7fc97c0a5700 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc98c005fd0 tx=0x7fc98c009500 comp rx=0 tx=0).stop 2024-09-15T23:36:15.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 -- 172.21.15.89:0/1421239484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc994101420 msgr2=0x7fc994197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:15.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc994101420 0x7fc994197800 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fc988009230 tx=0x7fc988004270 comp rx=0 tx=0).stop 2024-09-15T23:36:15.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 -- 172.21.15.89:0/1421239484 shutdown_connections 2024-09-15T23:36:15.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc97c0a3240 0x7fc97c0a5700 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc994106a80 0x7fc99419bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc994101420 0x7fc994197800 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 --2- 172.21.15.89:0/1421239484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc994100a80 0x7fc9941972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:15.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 -- 172.21.15.89:0/1421239484 >> 172.21.15.89:0/1421239484 conn(0x7fc9940fc7a0 msgr2=0x7fc9940fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:15.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 -- 172.21.15.89:0/1421239484 shutdown_connections 2024-09-15T23:36:15.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:15.216+0000 7fc99a28d640 1 -- 172.21.15.89:0/1421239484 wait complete. 2024-09-15T23:36:15.950 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:36:16.199 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:16.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.708+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/2835969049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaa4102dd0 msgr2=0x7fdaa41031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:16.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.708+0000 7fdaa8e4c640 1 --2- 172.21.15.89:0/2835969049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaa4102dd0 0x7fdaa41031b0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fda94009cd0 tx=0x7fda9402a8d0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.709+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/2835969049 shutdown_connections 2024-09-15T23:36:16.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.709+0000 7fdaa8e4c640 1 --2- 172.21.15.89:0/2835969049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa4108300 0x7fdaa410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.709+0000 7fdaa8e4c640 1 --2- 172.21.15.89:0/2835969049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa4103780 0x7fdaa4107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.709+0000 7fdaa8e4c640 1 --2- 172.21.15.89:0/2835969049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaa4102dd0 0x7fdaa41031b0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.709+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/2835969049 >> 172.21.15.89:0/2835969049 conn(0x7fdaa40fc7d0 msgr2=0x7fdaa40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:16.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.709+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/2835969049 shutdown_connections 2024-09-15T23:36:16.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.709+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/2835969049 wait complete. 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.709+0000 7fdaa8e4c640 1 Processor -- start 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.709+0000 7fdaa8e4c640 1 -- start start 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa8e4c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa4102dd0 0x7fdaa4198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa8e4c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa4103780 0x7fdaa4198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa8e4c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaa4108300 0x7fdaa419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa8e4c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa4112110 con 0x7fdaa4108300 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa8e4c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa4111f90 con 0x7fdaa4102dd0 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa8e4c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa4112290 con 0x7fdaa4103780 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaa4108300 0x7fdaa419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaa4108300 0x7fdaa419cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42158/0 (socket says 172.21.15.89:42158) 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa2d9d640 1 -- 172.21.15.89:0/1440332623 learned_addr learned my addr 172.21.15.89:0/1440332623 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:16.729 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa4103780 0x7fdaa4198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa259c640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa4102dd0 0x7fdaa4198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa2d9d640 1 -- 172.21.15.89:0/1440332623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa4103780 msgr2=0x7fdaa4198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa2d9d640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa4103780 0x7fdaa4198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa2d9d640 1 -- 172.21.15.89:0/1440332623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa4102dd0 msgr2=0x7fdaa4198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa2d9d640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa4102dd0 0x7fdaa4198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa2d9d640 1 -- 172.21.15.89:0/1440332623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa419d5f0 con 0x7fdaa4108300 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa4103780 0x7fdaa4198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa2d9d640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaa4108300 0x7fdaa419cf10 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009fd0 tx=0x7fda9c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fda9b7fe640 1 -- 172.21.15.89:0/1440332623 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c011830 con 0x7fdaa4108300 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/1440332623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa419d8e0 con 0x7fdaa4108300 2024-09-15T23:36:16.730 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fda9b7fe640 1 -- 172.21.15.89:0/1440332623 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda9c016070 con 0x7fdaa4108300 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fda9b7fe640 1 -- 172.21.15.89:0/1440332623 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c00f2e0 con 0x7fdaa4108300 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.710+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/1440332623 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa41a51c0 con 0x7fdaa4108300 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.713+0000 7fda9b7fe640 1 -- 172.21.15.89:0/1440332623 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fda9c0119d0 con 0x7fdaa4108300 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.713+0000 7fdaab225740 1 Processor -- start 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.713+0000 7fdaab225740 1 -- start start 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.713+0000 7fdaab225740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2b8c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.713+0000 7fdaab225740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b8ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaab225740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 0x55bbc2b9db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaab225740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bbc2af22f0 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaab225740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bbc2a99cf0 con 0x55bbc2b8db40 2024-09-15T23:36:16.731 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaab225740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bbc2aee2b0 con 0x55bbc2b9b780 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fda9b7fe640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda900a3240 0x7fda900a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b8ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b8ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42174/0 (socket says 172.21.15.89:42174) 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa259c640 1 -- 172.21.15.89:0/1522500113 learned_addr learned my addr 172.21.15.89:0/1522500113 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa2d9d640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda900a3240 0x7fda900a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/1522500113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 0x55bbc2b9db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fda9b7fe640 1 -- 172.21.15.89:0/1440332623 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fda9c0cf390 con 0x7fdaa4108300 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/1522500113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2b8c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa259c640 1 -- 172.21.15.89:0/1522500113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 msgr2=0x55bbc2b9db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa259c640 1 --2- 172.21.15.89:0/1522500113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 0x55bbc2b9db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa259c640 1 -- 172.21.15.89:0/1522500113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 msgr2=0x55bbc2b8c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:16.732 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa259c640 1 --2- 172.21.15.89:0/1522500113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2b8c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa259c640 1 -- 172.21.15.89:0/1522500113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbc2a7e3a0 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/1522500113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2b8c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa2d9d640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda900a3240 0x7fda900a5700 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0a3da0 tx=0x7fda9c06e4e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fdaa259c640 1 --2- 172.21.15.89:0/1522500113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b8ca10 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fda94035000 tx=0x7fda9402f450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e473a3973640ae5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.714+0000 7fda98ff9640 1 -- 172.21.15.89:0/1522500113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94031620 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fda98ff9640 1 -- 172.21.15.89:0/1522500113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda94040070 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fda98ff9640 1 -- 172.21.15.89:0/1522500113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94004860 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fdaab225740 1 -- 172.21.15.89:0/1522500113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 msgr2=0x55bbc2b8ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fdaab225740 1 --2- 172.21.15.89:0/1522500113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b8ca10 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fda94035000 tx=0x7fda9402f450 comp rx=0 tx=0).stop 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fdaab225740 1 -- 172.21.15.89:0/1522500113 shutdown_connections 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fdaab225740 1 --2- 172.21.15.89:0/1522500113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 0x55bbc2b9db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.733 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fdaab225740 1 --2- 172.21.15.89:0/1522500113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b8ca10 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fdaab225740 1 --2- 172.21.15.89:0/1522500113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2b8c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fdaab225740 1 -- 172.21.15.89:0/1522500113 >> 172.21.15.89:0/1522500113 conn(0x55bbc2b55310 msgr2=0x55bbc2b88c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fdaab225740 1 -- 172.21.15.89:0/1522500113 shutdown_connections 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.715+0000 7fdaab225740 1 -- 172.21.15.89:0/1522500113 wait complete. 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaab225740 1 -- start start 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaab225740 1 Processor -- start 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaab225740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b9b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaab225740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2b98410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaab225740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 0x55bbc2b989e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaab225740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bbc2b9e420 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaab225740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bbc2aad060 con 0x55bbc2b8db40 2024-09-15T23:36:16.734 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaab225740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bbc2b9e5a0 con 0x55bbc2b9b780 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaab225740 10 client.0 _ll_register_callbacks cb 0x55bbc28008c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaa1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b9b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaa1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b9b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42186/0 (socket says 172.21.15.89:42186) 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaa1d9b640 1 -- 172.21.15.89:0/2487910349 learned_addr learned my addr 172.21.15.89:0/2487910349 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaa259c640 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2b98410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaa1d9b640 1 -- 172.21.15.89:0/2487910349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 msgr2=0x55bbc2b989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 0x55bbc2b989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaa1d9b640 1 -- 172.21.15.89:0/2487910349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 msgr2=0x55bbc2b98410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2b98410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.716+0000 7fdaa1d9b640 1 -- 172.21.15.89:0/2487910349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbc28633c0 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.735 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b9b160 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fdaa406a2d0 tx=0x7fda8c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5078c10f0db3d47a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fda89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fda89ffb640 1 -- 172.21.15.89:0/2487910349 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c0130d0 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fda89ffb640 1 -- 172.21.15.89:0/2487910349 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fda8c005020 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fdaab225740 1 -- 172.21.15.89:0/2487910349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda94040070 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fda89ffb640 1 -- 172.21.15.89:0/2487910349 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fda8c024050 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fdaab225740 10 client.19569 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fdaab225740 1 -- 172.21.15.89:0/2487910349 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bbc2ba9f00 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fda89ffb640 1 -- 172.21.15.89:0/2487910349 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda94040070 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fda89ffb640 1 -- 172.21.15.89:0/2487910349 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fda8c011440 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fdaab225740 10 client.19569 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fdaab225740 10 client.19569 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:36:16.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fdaab225740 20 client.19569 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fdaab225740 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bbc2bab310 0x55bbc2bad750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fdaab225740 4 client.19569 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.717+0000 7fdaab225740 1 -- 172.21.15.89:0/2487910349 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbc2aff860 con 0x55bbc2bab310 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.718+0000 7fdaa259c640 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bbc2bab310 0x55bbc2bad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.718+0000 7fdaa259c640 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bbc2bab310 0x55bbc2bad750 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=984aeec5dddabefe server_cookie=dcbb089d837c88ee in_seq=0 out_seq=0 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.718+0000 7fda89ffb640 10 client.19569 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fda89ffb640 1 -- 172.21.15.89:0/2487910349 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bbc2aff860 con 0x55bbc2bab310 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fda89ffb640 10 client.19569 handle_command_reply: tid=0 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 client.19569 shutdown 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 -- 172.21.15.89:0/2487910349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 msgr2=0x55bbc2b9b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:16.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b9b160 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fdaa406a2d0 tx=0x7fda8c010800 comp rx=0 tx=0).stop 2024-09-15T23:36:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 -- 172.21.15.89:0/2487910349 shutdown_connections 2024-09-15T23:36:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bbc2bab310 0x55bbc2bad750 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 0x55bbc2b989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2b98410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fda89ffb640 0 client.19569 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:36:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 --2- 172.21.15.89:0/2487910349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2b9b160 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 -- 172.21.15.89:0/2487910349 >> 172.21.15.89:0/2487910349 conn(0x55bbc2b55310 msgr2=0x55bbc2b8e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 -- 172.21.15.89:0/2487910349 shutdown_connections 2024-09-15T23:36:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 1 -- 172.21.15.89:0/2487910349 wait complete. 2024-09-15T23:36:16.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.719+0000 7fdaab225740 20 client.19569 trim_cache size 0 max 16384 2024-09-15T23:36:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.732+0000 7fdaab225740 1 Processor -- start 2024-09-15T23:36:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaab225740 1 -- start start 2024-09-15T23:36:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaab225740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2ba76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaab225740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8db40 0x55bbc2b97430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaab225740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b9b780 0x55bbc2bb2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaab225740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa41acd60 con 0x55bbc2b8db40 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaab225740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa41ae8a0 con 0x55bbc2b9b780 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaab225740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa41ad430 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8db40 0x55bbc2b97430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8db40 0x55bbc2b97430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42202/0 (socket says 172.21.15.89:42202) 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa259c640 1 -- 172.21.15.89:0/579880835 learned_addr learned my addr 172.21.15.89:0/579880835 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/579880835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2ba76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/579880835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b9b780 0x55bbc2bb2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa259c640 1 -- 172.21.15.89:0/579880835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b8c5d0 msgr2=0x55bbc2ba76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa259c640 1 --2- 172.21.15.89:0/579880835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2ba76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa259c640 1 -- 172.21.15.89:0/579880835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b9b780 msgr2=0x55bbc2bb2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:16.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa259c640 1 --2- 172.21.15.89:0/579880835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b9b780 0x55bbc2bb2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa259c640 1 -- 172.21.15.89:0/579880835 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbc2b88510 con 0x55bbc2b8db40 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/579880835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b9b780 0x55bbc2bb2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.733+0000 7fdaa259c640 1 --2- 172.21.15.89:0/579880835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8db40 0x55bbc2b97430 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fda94004490 tx=0x7fda94047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf3b072bbbefc57 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fda8bfff640 1 -- 172.21.15.89:0/579880835 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94031420 con 0x55bbc2b8db40 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fda8bfff640 1 -- 172.21.15.89:0/579880835 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda94040040 con 0x55bbc2b8db40 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fda8bfff640 1 -- 172.21.15.89:0/579880835 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9402f890 con 0x55bbc2b8db40 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fdaab225740 1 -- 172.21.15.89:0/579880835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8db40 msgr2=0x55bbc2b97430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fdaab225740 1 --2- 172.21.15.89:0/579880835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8db40 0x55bbc2b97430 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fda94004490 tx=0x7fda94047040 comp rx=0 tx=0).stop 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fdaab225740 1 -- 172.21.15.89:0/579880835 shutdown_connections 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fdaab225740 1 --2- 172.21.15.89:0/579880835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b9b780 0x55bbc2bb2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fdaab225740 1 --2- 172.21.15.89:0/579880835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8db40 0x55bbc2b97430 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fdaab225740 1 --2- 172.21.15.89:0/579880835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2ba76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fdaab225740 1 -- 172.21.15.89:0/579880835 >> 172.21.15.89:0/579880835 conn(0x55bbc2b55310 msgr2=0x55bbc2b96d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fdaab225740 1 -- 172.21.15.89:0/579880835 shutdown_connections 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.734+0000 7fdaab225740 1 -- 172.21.15.89:0/579880835 wait complete. 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaab225740 1 -- start start 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaab225740 1 Processor -- start 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaab225740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2ba3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaab225740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2ba4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaab225740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 0x55bbc2b94d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaab225740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa41ae1d0 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaab225740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa41adb00 con 0x55bbc2b8db40 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaab225740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa41aad40 con 0x55bbc2b9b780 2024-09-15T23:36:16.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaab225740 10 client.0 _ll_register_callbacks cb 0x7fda8c013a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaa1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2ba3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaa1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2ba3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42210/0 (socket says 172.21.15.89:42210) 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaa1d9b640 1 -- 172.21.15.89:0/3454971948 learned_addr learned my addr 172.21.15.89:0/3454971948 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaa259c640 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2ba4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaa1d9b640 1 -- 172.21.15.89:0/3454971948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 msgr2=0x55bbc2b94d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 0x55bbc2b94d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaa1d9b640 1 -- 172.21.15.89:0/3454971948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 msgr2=0x55bbc2ba4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2ba4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.735+0000 7fdaa1d9b640 1 -- 172.21.15.89:0/3454971948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbc2b955b0 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fdaa1d9b640 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2ba3ac0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0058c0 tx=0x7fda8c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c408b50520d4fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:16.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fda89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fda89ffb640 1 -- 172.21.15.89:0/3454971948 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c02c030 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fda89ffb640 1 -- 172.21.15.89:0/3454971948 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fda8c0130d0 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fdaab225740 1 -- 172.21.15.89:0/3454971948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda94040040 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fda89ffb640 1 -- 172.21.15.89:0/3454971948 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fda8c037050 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fdaab225740 10 client.19575 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fdaab225740 1 -- 172.21.15.89:0/3454971948 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bbc2c15a50 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fda89ffb640 1 -- 172.21.15.89:0/3454971948 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda94040040 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fda89ffb640 1 -- 172.21.15.89:0/3454971948 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fda8c005020 con 0x55bbc2b8c5d0 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fdaab225740 10 client.19575 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fdaab225740 10 client.19575 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fdaab225740 20 client.19575 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:16.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fdaab225740 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bbc2c160f0 0x55bbc2c184e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fdaab225740 4 client.19575 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.736+0000 7fdaab225740 1 -- 172.21.15.89:0/3454971948 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bbc2af9d70 con 0x55bbc2c160f0 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.737+0000 7fdaa259c640 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bbc2c160f0 0x55bbc2c184e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.737+0000 7fda89ffb640 10 client.19575 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.737+0000 7fdaa259c640 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bbc2c160f0 0x55bbc2c184e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5a9c6724b9ff233d server_cookie=c74a03aafb48ed57 in_seq=0 out_seq=0 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.737+0000 7fda89ffb640 1 -- 172.21.15.89:0/3454971948 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bbc2af9d70 con 0x55bbc2c160f0 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.737+0000 7fda89ffb640 10 client.19575 handle_command_reply: tid=0 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.737+0000 7fdaab225740 1 client.19575 shutdown 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.737+0000 7fdaab225740 1 -- 172.21.15.89:0/3454971948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 msgr2=0x55bbc2ba3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.737+0000 7fdaab225740 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2ba3ac0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0058c0 tx=0x7fda8c00bd30 comp rx=0 tx=0).stop 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaab225740 1 -- 172.21.15.89:0/3454971948 shutdown_connections 2024-09-15T23:36:16.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaab225740 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55bbc2c160f0 0x55bbc2c184e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaab225740 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbc2b9b780 0x55bbc2b94d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaab225740 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bbc2b8db40 0x55bbc2ba4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaab225740 1 --2- 172.21.15.89:0/3454971948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bbc2b8c5d0 0x55bbc2ba3ac0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaab225740 1 -- 172.21.15.89:0/3454971948 >> 172.21.15.89:0/3454971948 conn(0x55bbc2b55310 msgr2=0x55bbc2b97540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fda89ffb640 0 client.19575 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:36:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaab225740 1 -- 172.21.15.89:0/3454971948 shutdown_connections 2024-09-15T23:36:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaab225740 1 -- 172.21.15.89:0/3454971948 wait complete. 2024-09-15T23:36:16.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaab225740 20 client.19575 trim_cache size 0 max 16384 2024-09-15T23:36:16.750 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:36:16.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/1440332623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda900a3240 msgr2=0x7fda900a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:16.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda900a3240 0x7fda900a5700 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fda9c0a3da0 tx=0x7fda9c06e4e0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/1440332623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaa4108300 msgr2=0x7fdaa419cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaa4108300 0x7fdaa419cf10 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fda9c009fd0 tx=0x7fda9c00b8e0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/1440332623 shutdown_connections 2024-09-15T23:36:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fda900a3240 0x7fda900a5700 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaa4108300 0x7fdaa419cf10 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa4103780 0x7fdaa4198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 --2- 172.21.15.89:0/1440332623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdaa4102dd0 0x7fdaa4198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/1440332623 >> 172.21.15.89:0/1440332623 conn(0x7fdaa40fc7d0 msgr2=0x7fdaa4105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/1440332623 shutdown_connections 2024-09-15T23:36:16.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:16.738+0000 7fdaa8e4c640 1 -- 172.21.15.89:0/1440332623 wait complete. 2024-09-15T23:36:16.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:16 smithi120 ceph-mon[35250]: pgmap v717: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-15T23:36:16.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:16 smithi174 ceph-mon[35586]: pgmap v717: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-15T23:36:17.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:16 smithi089 ceph-mon[30933]: pgmap v717: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-09-15T23:36:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.457+0000 7f0881152640 1 -- 172.21.15.89:0/1134631465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f087c100a40 msgr2=0x7f087c100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.457+0000 7f0881152640 1 --2- 172.21.15.89:0/1134631465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f087c100a40 0x7f087c100e40 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f086c009ca0 tx=0x7f086c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.457+0000 7f0881152640 1 -- 172.21.15.89:0/1134631465 shutdown_connections 2024-09-15T23:36:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.457+0000 7f0881152640 1 --2- 172.21.15.89:0/1134631465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f087c1025f0 0x7f087c108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.457+0000 7f0881152640 1 --2- 172.21.15.89:0/1134631465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c101c30 0x7f087c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.457+0000 7f0881152640 1 --2- 172.21.15.89:0/1134631465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f087c100a40 0x7f087c100e40 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.457+0000 7f0881152640 1 -- 172.21.15.89:0/1134631465 >> 172.21.15.89:0/1134631465 conn(0x7f087c0fc1e0 msgr2=0x7f087c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.457+0000 7f0881152640 1 -- 172.21.15.89:0/1134631465 shutdown_connections 2024-09-15T23:36:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.457+0000 7f0881152640 1 -- 172.21.15.89:0/1134631465 wait complete. 2024-09-15T23:36:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f0881152640 1 Processor -- start 2024-09-15T23:36:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f0881152640 1 -- start start 2024-09-15T23:36:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f0881152640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c100a40 0x7f087c197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f0881152640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f087c101c30 0x7f087c1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f0881152640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f087c1025f0 0x7f087c19f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f0881152640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f087c10c720 con 0x7f087c1025f0 2024-09-15T23:36:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f0881152640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f087c10c5a0 con 0x7f087c100a40 2024-09-15T23:36:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f0881152640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f087c10c8a0 con 0x7f087c101c30 2024-09-15T23:36:17.461 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f087b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f087c1025f0 0x7f087c19f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f087b59e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f087c1025f0 0x7f087c19f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42220/0 (socket says 172.21.15.89:42220) 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f087b59e640 1 -- 172.21.15.89:0/2995117596 learned_addr learned my addr 172.21.15.89:0/2995117596 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f087ad9d640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c100a40 0x7f087c197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f087a59c640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f087c101c30 0x7f087c1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f087b59e640 1 -- 172.21.15.89:0/2995117596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f087c101c30 msgr2=0x7f087c1983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f087b59e640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f087c101c30 0x7f087c1983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f087b59e640 1 -- 172.21.15.89:0/2995117596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c100a40 msgr2=0x7f087c197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f087b59e640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c100a40 0x7f087c197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.458+0000 7f087b59e640 1 -- 172.21.15.89:0/2995117596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087c19fa90 con 0x7f087c1025f0 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.459+0000 7f087b59e640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f087c1025f0 0x7f087c19f420 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f0874009fd0 tx=0x7f087400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.459+0000 7f0873fff640 1 -- 172.21.15.89:0/2995117596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874011830 con 0x7f087c1025f0 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.459+0000 7f0881152640 1 -- 172.21.15.89:0/2995117596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f087c19fd80 con 0x7f087c1025f0 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.459+0000 7f0873fff640 1 -- 172.21.15.89:0/2995117596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0874016070 con 0x7f087c1025f0 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.459+0000 7f0873fff640 1 -- 172.21.15.89:0/2995117596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087400f2e0 con 0x7f087c1025f0 2024-09-15T23:36:17.462 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.459+0000 7f0881152640 1 -- 172.21.15.89:0/2995117596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f087c1a0220 con 0x7f087c1025f0 2024-09-15T23:36:17.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.461+0000 7f0873fff640 1 -- 172.21.15.89:0/2995117596 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f08740119d0 con 0x7f087c1025f0 2024-09-15T23:36:17.468 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.461+0000 7f0883fcc740 1 Processor -- start 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.461+0000 7f0883fcc740 1 -- start start 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad39da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad3a2e0 0x56499ad3eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad41830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f0883fcc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56499ac9d2f0 con 0x56499ad399a0 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f0883fcc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56499ac45270 con 0x56499ad3f370 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f0883fcc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56499ac992b0 con 0x56499ad3a2e0 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad39da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad39da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42230/0 (socket says 172.21.15.89:42230) 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087a59c640 1 -- 172.21.15.89:0/3408362168 learned_addr learned my addr 172.21.15.89:0/3408362168 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f0873fff640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08680a32f0 0x7f08680a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087ad9d640 1 --2- 172.21.15.89:0/3408362168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad3a2e0 0x56499ad3eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087a59c640 1 -- 172.21.15.89:0/3408362168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad3a2e0 msgr2=0x56499ad3eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087a59c640 1 --2- 172.21.15.89:0/3408362168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad3a2e0 0x56499ad3eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087a59c640 1 -- 172.21.15.89:0/3408362168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 msgr2=0x56499ad41830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087a59c640 1 --2- 172.21.15.89:0/3408362168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad41830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087a59c640 1 -- 172.21.15.89:0/3408362168 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56499ac44340 con 0x56499ad399a0 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087b59e640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08680a32f0 0x7f08680a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f0873fff640 1 -- 172.21.15.89:0/2995117596 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f08740cf370 con 0x7f087c1025f0 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087a59c640 1 --2- 172.21.15.89:0/3408362168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad39da0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f0864005d00 tx=0x7f0864011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7254dd8079e2862a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f08717fa640 1 -- 172.21.15.89:0/3408362168 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864011e60 con 0x56499ad399a0 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f08717fa640 1 -- 172.21.15.89:0/3408362168 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08640128d0 con 0x56499ad399a0 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f08717fa640 1 -- 172.21.15.89:0/3408362168 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086400f900 con 0x56499ad399a0 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.462+0000 7f087b59e640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08680a32f0 0x7f08680a57b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f087409b2a0 tx=0x7f087406a2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.463+0000 7f0883fcc740 1 -- 172.21.15.89:0/3408362168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 msgr2=0x56499ad39da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:17.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.463+0000 7f0883fcc740 1 --2- 172.21.15.89:0/3408362168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad39da0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f0864005d00 tx=0x7f0864011750 comp rx=0 tx=0).stop 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.463+0000 7f0883fcc740 1 -- 172.21.15.89:0/3408362168 shutdown_connections 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.463+0000 7f0883fcc740 1 --2- 172.21.15.89:0/3408362168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad41830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.463+0000 7f0883fcc740 1 --2- 172.21.15.89:0/3408362168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad3a2e0 0x56499ad3eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.463+0000 7f0883fcc740 1 --2- 172.21.15.89:0/3408362168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad39da0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.463+0000 7f0883fcc740 1 -- 172.21.15.89:0/3408362168 >> 172.21.15.89:0/3408362168 conn(0x56499ad35010 msgr2=0x56499ad35410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.463+0000 7f0883fcc740 1 -- 172.21.15.89:0/3408362168 shutdown_connections 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.463+0000 7f0883fcc740 1 -- 172.21.15.89:0/3408362168 wait complete. 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f0883fcc740 1 -- start start 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f0883fcc740 1 Processor -- start 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 0x56499ad43c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad44170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad4b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f0883fcc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56499ad42340 con 0x56499ad3a2e0 2024-09-15T23:36:17.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f0883fcc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56499ac26290 con 0x56499ad3f370 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f0883fcc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56499ad424c0 con 0x56499ad399a0 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f0883fcc740 10 client.0 _ll_register_callbacks cb 0x56499ac57140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad44170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad44170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42244/0 (socket says 172.21.15.89:42244) 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087ad9d640 1 -- 172.21.15.89:0/3431230752 learned_addr learned my addr 172.21.15.89:0/3431230752 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087a59c640 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 0x56499ad43c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087a59c640 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad4b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087ad9d640 1 -- 172.21.15.89:0/3431230752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 msgr2=0x56499ad43c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087ad9d640 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 0x56499ad43c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087ad9d640 1 -- 172.21.15.89:0/3431230752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 msgr2=0x56499ad4b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087ad9d640 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad4b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087ad9d640 1 -- 172.21.15.89:0/3431230752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56499a9b0210 con 0x56499ad3a2e0 2024-09-15T23:36:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087a59c640 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad4b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f087ad9d640 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad44170 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f087c101a90 tx=0x7f086c031a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6086fce67a7077bd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.464+0000 7f08627fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f08627fc640 1 -- 172.21.15.89:0/3431230752 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c02f810 con 0x56499ad3a2e0 2024-09-15T23:36:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f0883fcc740 1 -- 172.21.15.89:0/3431230752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08640128d0 con 0x56499ad3a2e0 2024-09-15T23:36:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f08627fc640 1 -- 172.21.15.89:0/3431230752 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f086c02f9b0 con 0x56499ad3a2e0 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f08627fc640 1 -- 172.21.15.89:0/3431230752 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f086c053050 con 0x56499ad3a2e0 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f0883fcc740 10 client.19587 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f0883fcc740 1 -- 172.21.15.89:0/3431230752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56499aa7c620 con 0x56499ad3a2e0 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f08627fc640 1 -- 172.21.15.89:0/3431230752 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08640128d0 con 0x56499ad3a2e0 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f08627fc640 1 -- 172.21.15.89:0/3431230752 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f086c04e030 con 0x56499ad3a2e0 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f0883fcc740 10 client.19587 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f0883fcc740 10 client.19587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f0883fcc740 20 client.19587 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f0883fcc740 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56499ad4f8a0 0x56499ad51d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f0883fcc740 4 client.19587 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.465+0000 7f0883fcc740 1 -- 172.21.15.89:0/3431230752 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56499acaaa20 con 0x56499ad4f8a0 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.466+0000 7f087a59c640 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56499ad4f8a0 0x56499ad51d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.466+0000 7f08627fc640 10 client.19587 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.466+0000 7f087a59c640 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56499ad4f8a0 0x56499ad51d00 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dcc7de5018955e2 server_cookie=33ae8b1941a2fa2f in_seq=0 out_seq=0 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f08627fc640 1 -- 172.21.15.89:0/3431230752 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56499acaaa20 con 0x56499ad4f8a0 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f08627fc640 10 client.19587 handle_command_reply: tid=0 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 client.19587 shutdown 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 -- 172.21.15.89:0/3431230752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 msgr2=0x56499ad44170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad44170 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f087c101a90 tx=0x7f086c031a00 comp rx=0 tx=0).stop 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 -- 172.21.15.89:0/3431230752 shutdown_connections 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56499ad4f8a0 0x56499ad51d00 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad4b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f08627fc640 0 client.19587 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad44170 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 --2- 172.21.15.89:0/3431230752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 0x56499ad43c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 -- 172.21.15.89:0/3431230752 >> 172.21.15.89:0/3431230752 conn(0x56499ad35010 msgr2=0x56499ad36820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 -- 172.21.15.89:0/3431230752 shutdown_connections 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 1 -- 172.21.15.89:0/3431230752 wait complete. 2024-09-15T23:36:17.475 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.467+0000 7f0883fcc740 20 client.19587 trim_cache size 0 max 16384 2024-09-15T23:36:17.483 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.481+0000 7f0883fcc740 1 Processor -- start 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.481+0000 7f0883fcc740 1 -- start start 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad36b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad3a2e0 0x56499ad4b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad56600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f0883fcc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1aa920 con 0x56499ad399a0 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f0883fcc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1ac460 con 0x56499ad3f370 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f0883fcc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1aaff0 con 0x56499ad3a2e0 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f087a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad36b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f087a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad36b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42252/0 (socket says 172.21.15.89:42252) 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f087a59c640 1 -- 172.21.15.89:0/1242979880 learned_addr learned my addr 172.21.15.89:0/1242979880 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f087ad9d640 1 --2- 172.21.15.89:0/1242979880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad3a2e0 0x56499ad4b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f087a59c640 1 -- 172.21.15.89:0/1242979880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad3a2e0 msgr2=0x56499ad4b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:17.484 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f087a59c640 1 --2- 172.21.15.89:0/1242979880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad3a2e0 0x56499ad4b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f087a59c640 1 -- 172.21.15.89:0/1242979880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 msgr2=0x56499ad56600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f087a59c640 1 --2- 172.21.15.89:0/1242979880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad56600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f087a59c640 1 -- 172.21.15.89:0/1242979880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56499ad34e30 con 0x56499ad399a0 2024-09-15T23:36:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f087a59c640 1 --2- 172.21.15.89:0/1242979880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad36b10 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f086400bd80 tx=0x7f0864011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e4426c8dfb1c1b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f0870ff9640 1 -- 172.21.15.89:0/1242979880 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864005040 con 0x56499ad399a0 2024-09-15T23:36:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.482+0000 7f0870ff9640 1 -- 172.21.15.89:0/1242979880 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f086402c020 con 0x56499ad399a0 2024-09-15T23:36:17.485 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0870ff9640 1 -- 172.21.15.89:0/1242979880 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864012570 con 0x56499ad399a0 2024-09-15T23:36:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0883fcc740 1 -- 172.21.15.89:0/1242979880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 msgr2=0x56499ad36b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0883fcc740 1 --2- 172.21.15.89:0/1242979880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad36b10 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f086400bd80 tx=0x7f0864011750 comp rx=0 tx=0).stop 2024-09-15T23:36:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0883fcc740 1 -- 172.21.15.89:0/1242979880 shutdown_connections 2024-09-15T23:36:17.488 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0883fcc740 1 --2- 172.21.15.89:0/1242979880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad56600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0883fcc740 1 --2- 172.21.15.89:0/1242979880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad3a2e0 0x56499ad4b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0883fcc740 1 --2- 172.21.15.89:0/1242979880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad399a0 0x56499ad36b10 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0883fcc740 1 -- 172.21.15.89:0/1242979880 >> 172.21.15.89:0/1242979880 conn(0x56499ad35010 msgr2=0x56499ad353f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0883fcc740 1 -- 172.21.15.89:0/1242979880 shutdown_connections 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0883fcc740 1 -- 172.21.15.89:0/1242979880 wait complete. 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.483+0000 7f0883fcc740 1 -- start start 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f0883fcc740 1 Processor -- start 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 0x56499ad48f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad494b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f0883fcc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad4a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f0883fcc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1abd90 con 0x56499ad3a2e0 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f0883fcc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1ab6c0 con 0x56499ad3f370 2024-09-15T23:36:17.489 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f0883fcc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1a8900 con 0x56499ad399a0 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f0883fcc740 10 client.0 _ll_register_callbacks cb 0x7f086c0318c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad494b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad494b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42262/0 (socket says 172.21.15.89:42262) 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087ad9d640 1 -- 172.21.15.89:0/2476672533 learned_addr learned my addr 172.21.15.89:0/2476672533 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087a59c640 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 0x56499ad48f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087ad9d640 1 -- 172.21.15.89:0/2476672533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 msgr2=0x56499ad48f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087ad9d640 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 0x56499ad48f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087ad9d640 1 -- 172.21.15.89:0/2476672533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 msgr2=0x56499ad4a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087ad9d640 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad4a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087ad9d640 1 -- 172.21.15.89:0/2476672533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56499ad4aa80 con 0x56499ad3a2e0 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087a59c640 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 0x56499ad48f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f087ad9d640 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad494b0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f086c030dc0 tx=0x7f086c009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d9c1dcab273bd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f08627fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f08627fc640 1 -- 172.21.15.89:0/2476672533 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c05b070 con 0x56499ad3a2e0 2024-09-15T23:36:17.490 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.484+0000 7f08627fc640 1 -- 172.21.15.89:0/2476672533 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f086c0309d0 con 0x56499ad3a2e0 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f0883fcc740 1 -- 172.21.15.89:0/2476672533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f086402c020 con 0x56499ad3a2e0 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f08627fc640 1 -- 172.21.15.89:0/2476672533 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f086c0098c0 con 0x56499ad3a2e0 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f0883fcc740 10 client.19593 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f0883fcc740 1 -- 172.21.15.89:0/2476672533 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56499adbad30 con 0x56499ad3a2e0 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f08627fc640 1 -- 172.21.15.89:0/2476672533 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f086402c020 con 0x56499ad3a2e0 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f08627fc640 1 -- 172.21.15.89:0/2476672533 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f086c04e070 con 0x56499ad3a2e0 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f0883fcc740 10 client.19593 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f0883fcc740 10 client.19593 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f0883fcc740 20 client.19593 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f0883fcc740 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56499adbb710 0x56499adbdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f0883fcc740 4 client.19593 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f0883fcc740 1 -- 172.21.15.89:0/2476672533 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56499aca4d70 con 0x56499adbb710 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.485+0000 7f087a59c640 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56499adbb710 0x56499adbdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:17.491 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.486+0000 7f087a59c640 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56499adbb710 0x56499adbdb30 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66eeface0b6d9b60 server_cookie=db684918c9905dc6 in_seq=0 out_seq=0 2024-09-15T23:36:17.492 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.486+0000 7f08627fc640 10 client.19593 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.497+0000 7f08627fc640 1 -- 172.21.15.89:0/2476672533 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56499aca4d70 con 0x56499adbb710 2024-09-15T23:36:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.497+0000 7f08627fc640 10 client.19593 handle_command_reply: tid=0 2024-09-15T23:36:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.497+0000 7f0883fcc740 1 client.19593 shutdown 2024-09-15T23:36:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.497+0000 7f0883fcc740 1 -- 172.21.15.89:0/2476672533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 msgr2=0x56499ad494b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.497+0000 7f0883fcc740 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad494b0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f086c030dc0 tx=0x7f086c009540 comp rx=0 tx=0).stop 2024-09-15T23:36:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0883fcc740 1 -- 172.21.15.89:0/2476672533 shutdown_connections 2024-09-15T23:36:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0883fcc740 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56499adbb710 0x56499adbdb30 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0883fcc740 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56499ad3f370 0x56499ad4a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0883fcc740 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56499ad3a2e0 0x56499ad494b0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.500 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f08627fc640 0 client.19593 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:17.500 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0883fcc740 1 --2- 172.21.15.89:0/2476672533 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56499ad399a0 0x56499ad48f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.500 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0883fcc740 1 -- 172.21.15.89:0/2476672533 >> 172.21.15.89:0/2476672533 conn(0x56499ad35010 msgr2=0x56499ad35c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:17.500 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0883fcc740 1 -- 172.21.15.89:0/2476672533 shutdown_connections 2024-09-15T23:36:17.500 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0883fcc740 1 -- 172.21.15.89:0/2476672533 wait complete. 2024-09-15T23:36:17.500 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0883fcc740 20 client.19593 trim_cache size 0 max 16384 2024-09-15T23:36:17.500 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:36:17.500 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:36:17.500 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:36:17.500 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:36:17.501 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 -- 172.21.15.89:0/2995117596 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08680a32f0 msgr2=0x7f08680a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:17.501 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08680a32f0 0x7f08680a57b0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f087409b2a0 tx=0x7f087406a2e0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.501 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 -- 172.21.15.89:0/2995117596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f087c1025f0 msgr2=0x7f087c19f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f087c1025f0 0x7f087c19f420 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f0874009fd0 tx=0x7f087400b8e0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 -- 172.21.15.89:0/2995117596 shutdown_connections 2024-09-15T23:36:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f08680a32f0 0x7f08680a57b0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f087c1025f0 0x7f087c19f420 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f087c101c30 0x7f087c1983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 --2- 172.21.15.89:0/2995117596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c100a40 0x7f087c197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 -- 172.21.15.89:0/2995117596 >> 172.21.15.89:0/2995117596 conn(0x7f087c0fc1e0 msgr2=0x7f087c0fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 -- 172.21.15.89:0/2995117596 shutdown_connections 2024-09-15T23:36:17.502 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:17.498+0000 7f0881152640 1 -- 172.21.15.89:0/2995117596 wait complete. 2024-09-15T23:36:17.544 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:36:17.789 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:18.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.260+0000 7fcf1f062640 1 -- 172.21.15.89:0/2723871616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf18112640 msgr2=0x7fcf18112a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:18.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.260+0000 7fcf1f062640 1 --2- 172.21.15.89:0/2723871616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf18112640 0x7fcf18112a20 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009900 tx=0x7fcf1402a410 comp rx=0 tx=0).stop 2024-09-15T23:36:18.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.260+0000 7fcf1f062640 1 -- 172.21.15.89:0/2723871616 shutdown_connections 2024-09-15T23:36:18.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.260+0000 7fcf1f062640 1 --2- 172.21.15.89:0/2723871616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1810cf30 0x7fcf18119410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.260+0000 7fcf1f062640 1 --2- 172.21.15.89:0/2723871616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf1810c590 0x7fcf1810c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.260+0000 7fcf1f062640 1 --2- 172.21.15.89:0/2723871616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf18112640 0x7fcf18112a20 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.260+0000 7fcf1f062640 1 -- 172.21.15.89:0/2723871616 >> 172.21.15.89:0/2723871616 conn(0x7fcf1806cdc0 msgr2=0x7fcf1806d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.260+0000 7fcf1f062640 1 -- 172.21.15.89:0/2723871616 shutdown_connections 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.260+0000 7fcf1f062640 1 -- 172.21.15.89:0/2723871616 wait complete. 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf1f062640 1 Processor -- start 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf1f062640 1 -- start start 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf1f062640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1810c590 0x7fcf181a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf1f062640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1810cf30 0x7fcf181a3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf1f062640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf18112640 0x7fcf181a7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf1f062640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1811c9a0 con 0x7fcf1810cf30 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf1f062640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1811c820 con 0x7fcf1810c590 2024-09-15T23:36:18.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf1f062640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1811cb20 con 0x7fcf18112640 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1810cf30 0x7fcf181a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1810cf30 0x7fcf181a3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42282/0 (socket says 172.21.15.89:42282) 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf0ffff640 1 -- 172.21.15.89:0/994284317 learned_addr learned my addr 172.21.15.89:0/994284317 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1810c590 0x7fcf181a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf0ffff640 1 -- 172.21.15.89:0/994284317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf18112640 msgr2=0x7fcf181a7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf0ffff640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf18112640 0x7fcf181a7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf0ffff640 1 -- 172.21.15.89:0/994284317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1810c590 msgr2=0x7fcf181a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf0ffff640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1810c590 0x7fcf181a2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.261+0000 7fcf0ffff640 1 -- 172.21.15.89:0/994284317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf181a7df0 con 0x7fcf1810cf30 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.262+0000 7fcf0ffff640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1810cf30 0x7fcf181a3380 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fcf0800b4a0 tx=0x7fcf0800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.262+0000 7fcf0dffb640 1 -- 172.21.15.89:0/994284317 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08014070 con 0x7fcf1810cf30 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.262+0000 7fcf0dffb640 1 -- 172.21.15.89:0/994284317 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf08010af0 con 0x7fcf1810cf30 2024-09-15T23:36:18.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.262+0000 7fcf1f062640 1 -- 172.21.15.89:0/994284317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf181a80e0 con 0x7fcf1810cf30 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.262+0000 7fcf0dffb640 1 -- 172.21.15.89:0/994284317 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08017d10 con 0x7fcf1810cf30 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.262+0000 7fcf1f062640 1 -- 172.21.15.89:0/994284317 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf181af9c0 con 0x7fcf1810cf30 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.264+0000 7fcf0dffb640 1 -- 172.21.15.89:0/994284317 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcf08017070 con 0x7fcf1810cf30 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf2143b740 1 Processor -- start 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf2143b740 1 -- start start 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14cb40 0x56190c14b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 0x56190c14ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c158760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf2143b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56190c0b12f0 con 0x56190c156370 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf2143b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56190c058cf0 con 0x56190c14cb40 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf2143b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56190c0ad2b0 con 0x56190c14b5d0 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf0dffb640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf000a3240 0x7fcf000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf1d5ff640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf000a3240 0x7fcf000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42294/0 (socket says 172.21.15.89:42294) 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf0ffff640 1 -- 172.21.15.89:0/568003898 learned_addr learned my addr 172.21.15.89:0/568003898 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/568003898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 0x56190c14ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf1d5ff640 1 --2- 172.21.15.89:0/568003898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14cb40 0x56190c14b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf0dffb640 1 -- 172.21.15.89:0/994284317 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcf080cf250 con 0x7fcf1810cf30 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf0ffff640 1 -- 172.21.15.89:0/568003898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 msgr2=0x56190c14ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf0ffff640 1 --2- 172.21.15.89:0/568003898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 0x56190c14ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf0ffff640 1 -- 172.21.15.89:0/568003898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14cb40 msgr2=0x56190c14b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.265+0000 7fcf0ffff640 1 --2- 172.21.15.89:0/568003898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14cb40 0x56190c14b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.266+0000 7fcf0ffff640 1 -- 172.21.15.89:0/568003898 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56190c03d3a0 con 0x56190c156370 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.266+0000 7fcf1d5ff640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf000a3240 0x7fcf000a5700 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009fd0 tx=0x7fcf10009870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.266+0000 7fcf0ffff640 1 --2- 172.21.15.89:0/568003898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c158760 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fcf08097100 tx=0x7fcf080047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0874bdc42f08581 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.266+0000 7fcefeffd640 1 -- 172.21.15.89:0/568003898 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0801e070 con 0x56190c156370 2024-09-15T23:36:18.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.266+0000 7fcefeffd640 1 -- 172.21.15.89:0/568003898 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf0809f7d0 con 0x56190c156370 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.266+0000 7fcefeffd640 1 -- 172.21.15.89:0/568003898 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08097a90 con 0x56190c156370 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 -- 172.21.15.89:0/568003898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 msgr2=0x56190c158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 --2- 172.21.15.89:0/568003898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c158760 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fcf08097100 tx=0x7fcf080047d0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 -- 172.21.15.89:0/568003898 shutdown_connections 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 --2- 172.21.15.89:0/568003898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c158760 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 --2- 172.21.15.89:0/568003898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 0x56190c14ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 --2- 172.21.15.89:0/568003898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14cb40 0x56190c14b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 -- 172.21.15.89:0/568003898 >> 172.21.15.89:0/568003898 conn(0x56190c114310 msgr2=0x56190c147c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 -- 172.21.15.89:0/568003898 shutdown_connections 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 -- 172.21.15.89:0/568003898 wait complete. 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 -- start start 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 Processor -- start 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14b5d0 0x56190c155d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14cb40 0x56190c153030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c153640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56190c159070 con 0x56190c156370 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56190c06c060 con 0x56190c14b5d0 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56190c1591f0 con 0x56190c14cb40 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.267+0000 7fcf2143b740 10 client.0 _ll_register_callbacks cb 0x56190bdbf8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1d5ff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1d5ff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c153640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42296/0 (socket says 172.21.15.89:42296) 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1d5ff640 1 -- 172.21.15.89:0/1766719461 learned_addr learned my addr 172.21.15.89:0/1766719461 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf0ffff640 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14cb40 0x56190c153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14b5d0 0x56190c155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1d5ff640 1 -- 172.21.15.89:0/1766719461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14cb40 msgr2=0x56190c153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1d5ff640 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14cb40 0x56190c153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1d5ff640 1 -- 172.21.15.89:0/1766719461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14b5d0 msgr2=0x56190c155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1d5ff640 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14b5d0 0x56190c155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1d5ff640 1 -- 172.21.15.89:0/1766719461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56190be223c0 con 0x56190c156370 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14b5d0 0x56190c155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf1d5ff640 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c153640 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fcf100108b0 tx=0x7fcf10005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c5dd83578f6a2b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcef7fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcef7fff640 1 -- 172.21.15.89:0/1766719461 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf10023070 con 0x56190c156370 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcef7fff640 1 -- 172.21.15.89:0/1766719461 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcf10004d20 con 0x56190c156370 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf2143b740 1 -- 172.21.15.89:0/1766719461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf0809f7d0 con 0x56190c156370 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcef7fff640 1 -- 172.21.15.89:0/1766719461 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf1002e050 con 0x56190c156370 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf2143b740 10 client.19605 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.268+0000 7fcf2143b740 1 -- 172.21.15.89:0/1766719461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56190c164b40 con 0x56190c156370 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcef7fff640 1 -- 172.21.15.89:0/1766719461 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf0809f7d0 con 0x56190c156370 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcef7fff640 1 -- 172.21.15.89:0/1766719461 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcf1000f040 con 0x56190c156370 2024-09-15T23:36:18.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcf2143b740 10 client.19605 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcf2143b740 10 client.19605 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcf2143b740 20 client.19605 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56190c165f50 0x56190c1683d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcf2143b740 4 client.19605 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcf2143b740 1 -- 172.21.15.89:0/1766719461 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56190c0be860 con 0x56190c165f50 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56190c165f50 0x56190c1683d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcef7fff640 10 client.19605 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.269+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56190c165f50 0x56190c1683d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26687f0d93394d19 server_cookie=f8f1a78b9e9a1f1 in_seq=0 out_seq=0 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcef7fff640 1 -- 172.21.15.89:0/1766719461 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56190c0be860 con 0x56190c165f50 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcef7fff640 10 client.19605 handle_command_reply: tid=0 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcf2143b740 1 client.19605 shutdown 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcf2143b740 1 -- 172.21.15.89:0/1766719461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 msgr2=0x56190c153640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:18.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c153640 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fcf100108b0 tx=0x7fcf10005f00 comp rx=0 tx=0).stop 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcf2143b740 1 -- 172.21.15.89:0/1766719461 shutdown_connections 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56190c165f50 0x56190c1683d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c156370 0x56190c153640 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcef7fff640 0 client.19605 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14cb40 0x56190c153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1766719461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c14b5d0 0x56190c155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.270+0000 7fcf2143b740 1 -- 172.21.15.89:0/1766719461 >> 172.21.15.89:0/1766719461 conn(0x56190c114310 msgr2=0x56190c14d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.271+0000 7fcf2143b740 1 -- 172.21.15.89:0/1766719461 shutdown_connections 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.271+0000 7fcf2143b740 1 -- 172.21.15.89:0/1766719461 wait complete. 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.271+0000 7fcf2143b740 20 client.19605 trim_cache size 0 max 16384 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.283+0000 7fcf2143b740 1 Processor -- start 2024-09-15T23:36:18.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.283+0000 7fcf2143b740 1 -- start start 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.283+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14b5d0 0x56190c1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.283+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14cb40 0x56190c154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.283+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c156370 0x56190c14e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.283+0000 7fcf2143b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf181b9860 con 0x56190c14b5d0 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.283+0000 7fcf2143b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf181bb3a0 con 0x56190c156370 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcf2143b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf181b9f30 con 0x56190c14cb40 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcf1cdfe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14b5d0 0x56190c1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcf1cdfe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14b5d0 0x56190c1528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42306/0 (socket says 172.21.15.89:42306) 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcf1cdfe640 1 -- 172.21.15.89:0/2197967796 learned_addr learned my addr 172.21.15.89:0/2197967796 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcf1cdfe640 1 -- 172.21.15.89:0/2197967796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14cb40 msgr2=0x56190c154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/2197967796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14cb40 0x56190c154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcf1cdfe640 1 -- 172.21.15.89:0/2197967796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c156370 msgr2=0x56190c14e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/2197967796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c156370 0x56190c14e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcf1cdfe640 1 -- 172.21.15.89:0/2197967796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56190c151fe0 con 0x56190c14b5d0 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/2197967796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14b5d0 0x56190c1528f0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009230 tx=0x7fcf14004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e931022c7c4b27b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcefe7fc640 1 -- 172.21.15.89:0/2197967796 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1404a070 con 0x56190c14b5d0 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcefe7fc640 1 -- 172.21.15.89:0/2197967796 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf14005520 con 0x56190c14b5d0 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.284+0000 7fcefe7fc640 1 -- 172.21.15.89:0/2197967796 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14035040 con 0x56190c14b5d0 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 -- 172.21.15.89:0/2197967796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14b5d0 msgr2=0x56190c1528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 --2- 172.21.15.89:0/2197967796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14b5d0 0x56190c1528f0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009230 tx=0x7fcf14004610 comp rx=0 tx=0).stop 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 -- 172.21.15.89:0/2197967796 shutdown_connections 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 --2- 172.21.15.89:0/2197967796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c156370 0x56190c14e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 --2- 172.21.15.89:0/2197967796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14cb40 0x56190c154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 --2- 172.21.15.89:0/2197967796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14b5d0 0x56190c1528f0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 -- 172.21.15.89:0/2197967796 >> 172.21.15.89:0/2197967796 conn(0x56190c114310 msgr2=0x56190c154890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:18.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 -- 172.21.15.89:0/2197967796 shutdown_connections 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 -- 172.21.15.89:0/2197967796 wait complete. 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 -- start start 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.285+0000 7fcf2143b740 1 Processor -- start 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 0x56190c15a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14cb40 0x56190c15a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf2143b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c156370 0x56190c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf2143b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf181bacd0 con 0x56190c14cb40 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf2143b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf181ba600 con 0x56190c156370 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf2143b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf181b7840 con 0x56190c14b5d0 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf2143b740 10 client.0 _ll_register_callbacks cb 0x7fcf10009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14cb40 0x56190c15a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14cb40 0x56190c15a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42318/0 (socket says 172.21.15.89:42318) 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf0ffff640 1 -- 172.21.15.89:0/1297779707 learned_addr learned my addr 172.21.15.89:0/1297779707 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:18.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 0x56190c15a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf0ffff640 1 -- 172.21.15.89:0/1297779707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 msgr2=0x56190c15a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf1d5ff640 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c156370 0x56190c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf0ffff640 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 0x56190c15a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf0ffff640 1 -- 172.21.15.89:0/1297779707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c156370 msgr2=0x56190c15ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf0ffff640 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c156370 0x56190c15ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf0ffff640 1 -- 172.21.15.89:0/1297779707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56190c1cfec0 con 0x56190c14cb40 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 0x56190c15a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf0ffff640 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14cb40 0x56190c15a900 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fcf0809b120 tx=0x7fcf08009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35adc3dd6fb2fae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcef7fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcef7fff640 1 -- 172.21.15.89:0/1297779707 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0801e040 con 0x56190c14cb40 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcef7fff640 1 -- 172.21.15.89:0/1297779707 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcf08010ca0 con 0x56190c14cb40 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcf2143b740 1 -- 172.21.15.89:0/1297779707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf14005520 con 0x56190c14cb40 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.286+0000 7fcef7fff640 1 -- 172.21.15.89:0/1297779707 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf0809f7d0 con 0x56190c14cb40 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcf2143b740 10 client.19611 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcf2143b740 1 -- 172.21.15.89:0/1297779707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56190c1d1210 con 0x56190c14cb40 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcef7fff640 1 -- 172.21.15.89:0/1297779707 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf14005520 con 0x56190c14cb40 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcef7fff640 1 -- 172.21.15.89:0/1297779707 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcf0809a870 con 0x56190c14cb40 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcf2143b740 10 client.19611 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcf2143b740 10 client.19611 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcf2143b740 20 client.19611 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56190c1d15a0 0x56190c1d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcf2143b740 4 client.19611 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcf2143b740 1 -- 172.21.15.89:0/1297779707 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56190c0b8d70 con 0x56190c1d15a0 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.287+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56190c1d15a0 0x56190c1d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcef7fff640 10 client.19611 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf1cdfe640 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56190c1d15a0 0x56190c1d3990 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a343779e37b92254 server_cookie=2e5764051d722b47 in_seq=0 out_seq=0 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcef7fff640 1 -- 172.21.15.89:0/1297779707 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56190c0b8d70 con 0x56190c1d15a0 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcef7fff640 10 client.19611 handle_command_reply: tid=0 2024-09-15T23:36:18.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 client.19611 shutdown 2024-09-15T23:36:18.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 -- 172.21.15.89:0/1297779707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14cb40 msgr2=0x56190c15a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:18.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14cb40 0x56190c15a900 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fcf0809b120 tx=0x7fcf08009390 comp rx=0 tx=0).stop 2024-09-15T23:36:18.313 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:36:18.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 -- 172.21.15.89:0/1297779707 shutdown_connections 2024-09-15T23:36:18.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x56190c1d15a0 0x56190c1d3990 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56190c156370 0x56190c15ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56190c14cb40 0x56190c15a900 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcef7fff640 0 client.19611 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 --2- 172.21.15.89:0/1297779707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56190c14b5d0 0x56190c15a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 -- 172.21.15.89:0/1297779707 >> 172.21.15.89:0/1297779707 conn(0x56190c114310 msgr2=0x56190c161300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 -- 172.21.15.89:0/1297779707 shutdown_connections 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 1 -- 172.21.15.89:0/1297779707 wait complete. 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.288+0000 7fcf2143b740 20 client.19611 trim_cache size 0 max 16384 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 -- 172.21.15.89:0/994284317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf000a3240 msgr2=0x7fcf000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf000a3240 0x7fcf000a5700 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009fd0 tx=0x7fcf10009870 comp rx=0 tx=0).stop 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 -- 172.21.15.89:0/994284317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1810cf30 msgr2=0x7fcf181a3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1810cf30 0x7fcf181a3380 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fcf0800b4a0 tx=0x7fcf0800b960 comp rx=0 tx=0).stop 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 -- 172.21.15.89:0/994284317 shutdown_connections 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf000a3240 0x7fcf000a5700 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf18112640 0x7fcf181a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1810cf30 0x7fcf181a3380 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 --2- 172.21.15.89:0/994284317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1810c590 0x7fcf181a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:18.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 -- 172.21.15.89:0/994284317 >> 172.21.15.89:0/994284317 conn(0x7fcf1806cdc0 msgr2=0x7fcf1810ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:18.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 -- 172.21.15.89:0/994284317 shutdown_connections 2024-09-15T23:36:18.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:18.289+0000 7fcf1f062640 1 -- 172.21.15.89:0/994284317 wait complete. 2024-09-15T23:36:18.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:18 smithi120 ceph-mon[35250]: pgmap v718: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-09-15T23:36:18.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:18 smithi174 ceph-mon[35586]: pgmap v718: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-09-15T23:36:19.054 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:18 smithi089 ceph-mon[30933]: pgmap v718: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-09-15T23:36:19.168 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:36:19.413 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:19.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 -- 172.21.15.89:0/1643028475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222010c180 msgr2=0x7f222010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 --2- 172.21.15.89:0/1643028475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222010c180 0x7f222010e610 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f2218009ca0 tx=0x7f221802a950 comp rx=0 tx=0).stop 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 -- 172.21.15.89:0/1643028475 shutdown_connections 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 --2- 172.21.15.89:0/1643028475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f222010c180 0x7f222010e610 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 --2- 172.21.15.89:0/1643028475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2220103700 0x7f222010bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 --2- 172.21.15.89:0/1643028475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2220102de0 0x7f22201031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 -- 172.21.15.89:0/1643028475 >> 172.21.15.89:0/1643028475 conn(0x7f22200fc7a0 msgr2=0x7f22200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 -- 172.21.15.89:0/1643028475 shutdown_connections 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 -- 172.21.15.89:0/1643028475 wait complete. 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 Processor -- start 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.898+0000 7f2224bff640 1 -- start start 2024-09-15T23:36:19.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f2224bff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2220102de0 0x7f22201983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f2224bff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2220103700 0x7f2220198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f2224bff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f222010c180 0x7f222019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f2224bff640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2220111fb0 con 0x7f2220103700 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f2224bff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2220111e30 con 0x7f2220102de0 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f2224bff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2220112130 con 0x7f222010c180 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2220103700 0x7f2220198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2220103700 0x7f2220198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42340/0 (socket says 172.21.15.89:42340) 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221dd9b640 1 -- 172.21.15.89:0/3046583333 learned_addr learned my addr 172.21.15.89:0/3046583333 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221ed9d640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f222010c180 0x7f222019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221e59c640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2220102de0 0x7f22201983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221dd9b640 1 -- 172.21.15.89:0/3046583333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f222010c180 msgr2=0x7f222019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:19.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221dd9b640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f222010c180 0x7f222019ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221dd9b640 1 -- 172.21.15.89:0/3046583333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2220102de0 msgr2=0x7f22201983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221dd9b640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2220102de0 0x7f22201983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221dd9b640 1 -- 172.21.15.89:0/3046583333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222019d3a0 con 0x7f2220103700 2024-09-15T23:36:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f221dd9b640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2220103700 0x7f2220198930 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f221000c920 tx=0x7f221000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f22177fe640 1 -- 172.21.15.89:0/3046583333 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2210007c70 con 0x7f2220103700 2024-09-15T23:36:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f2224bff640 1 -- 172.21.15.89:0/3046583333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f222019d690 con 0x7f2220103700 2024-09-15T23:36:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f22177fe640 1 -- 172.21.15.89:0/3046583333 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2210007e10 con 0x7f2220103700 2024-09-15T23:36:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f22177fe640 1 -- 172.21.15.89:0/3046583333 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2210018890 con 0x7f2220103700 2024-09-15T23:36:19.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.899+0000 7f2224bff640 1 -- 172.21.15.89:0/3046583333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22201a4f70 con 0x7f2220103700 2024-09-15T23:36:19.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.902+0000 7f22177fe640 1 -- 172.21.15.89:0/3046583333 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2210018a70 con 0x7f2220103700 2024-09-15T23:36:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.902+0000 7f2226fd8740 1 Processor -- start 2024-09-15T23:36:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.902+0000 7f2226fd8740 1 -- start start 2024-09-15T23:36:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.902+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cee7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.902+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cee7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.902+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cef4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.902+0000 7f2226fd8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56322ce4d2f0 con 0x56322cee8b40 2024-09-15T23:36:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.902+0000 7f2226fd8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56322cdf4cf0 con 0x56322cef2370 2024-09-15T23:36:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.902+0000 7f2226fd8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56322ce492b0 con 0x56322cee75d0 2024-09-15T23:36:19.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f22177fe640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f220c0a3240 0x7f220c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cee7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cee7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42356/0 (socket says 172.21.15.89:42356) 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221ed9d640 1 -- 172.21.15.89:0/2045760659 learned_addr learned my addr 172.21.15.89:0/2045760659 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221e59c640 1 --2- 172.21.15.89:0/2045760659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cee7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221ed9d640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f220c0a3240 0x7f220c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f22177fe640 1 -- 172.21.15.89:0/3046583333 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f22100cff00 con 0x7f2220103700 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221dd9b640 1 --2- 172.21.15.89:0/2045760659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cef4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221e59c640 1 -- 172.21.15.89:0/2045760659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 msgr2=0x56322cef4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221e59c640 1 --2- 172.21.15.89:0/2045760659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cef4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221e59c640 1 -- 172.21.15.89:0/2045760659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 msgr2=0x56322cee7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:19.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221e59c640 1 --2- 172.21.15.89:0/2045760659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cee7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221e59c640 1 -- 172.21.15.89:0/2045760659 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56322cdd93a0 con 0x56322cee75d0 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221ed9d640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f220c0a3240 0x7f220c0a5700 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2218009430 tx=0x7f221802a950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f221e59c640 1 --2- 172.21.15.89:0/2045760659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cee7a10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f220800b760 tx=0x7f220800bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31b297f1c161783b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.903+0000 7f2214ff9640 1 -- 172.21.15.89:0/2045760659 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220800d950 con 0x56322cee75d0 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2214ff9640 1 -- 172.21.15.89:0/2045760659 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2208019070 con 0x56322cee75d0 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2214ff9640 1 -- 172.21.15.89:0/2045760659 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22080123e0 con 0x56322cee75d0 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2226fd8740 1 -- 172.21.15.89:0/2045760659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 msgr2=0x56322cee7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2226fd8740 1 --2- 172.21.15.89:0/2045760659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cee7a10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f220800b760 tx=0x7f220800bc20 comp rx=0 tx=0).stop 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2226fd8740 1 -- 172.21.15.89:0/2045760659 shutdown_connections 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2226fd8740 1 --2- 172.21.15.89:0/2045760659 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cef4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2226fd8740 1 --2- 172.21.15.89:0/2045760659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cee7a10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2226fd8740 1 --2- 172.21.15.89:0/2045760659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cee7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2226fd8740 1 -- 172.21.15.89:0/2045760659 >> 172.21.15.89:0/2045760659 conn(0x56322ceb0310 msgr2=0x56322cee3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2226fd8740 1 -- 172.21.15.89:0/2045760659 shutdown_connections 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.904+0000 7f2226fd8740 1 -- 172.21.15.89:0/2045760659 wait complete. 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f2226fd8740 1 -- start start 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f2226fd8740 1 Processor -- start 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cef1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceeefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322ceef530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f2226fd8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56322cef5070 con 0x56322cee8b40 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f2226fd8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56322ce08060 con 0x56322cef2370 2024-09-15T23:36:19.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f2226fd8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56322cef51f0 con 0x56322cee75d0 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f2226fd8740 10 client.0 _ll_register_callbacks cb 0x56322cb5b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f221e59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cef1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f221e59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cef1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:51598/0 (socket says 172.21.15.89:51598) 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f221e59c640 1 -- 172.21.15.89:0/1263539114 learned_addr learned my addr 172.21.15.89:0/1263539114 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f221dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceeefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.905+0000 7f221ed9d640 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322ceef530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f221e59c640 1 -- 172.21.15.89:0/1263539114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 msgr2=0x56322ceef530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f221e59c640 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322ceef530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f221e59c640 1 -- 172.21.15.89:0/1263539114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 msgr2=0x56322ceeefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f221e59c640 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceeefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f221e59c640 1 -- 172.21.15.89:0/1263539114 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56322cbbe3c0 con 0x56322cee75d0 2024-09-15T23:36:19.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f221dd9b640 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceeefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f221e59c640 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cef1d00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2208006a70 tx=0x7f2208014de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6a6398ffb84afe1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f2205ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f2205ffb640 1 -- 172.21.15.89:0/1263539114 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2208023070 con 0x56322cee75d0 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f2205ffb640 1 -- 172.21.15.89:0/1263539114 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f220800d950 con 0x56322cee75d0 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f2226fd8740 1 -- 172.21.15.89:0/1263539114 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2208019070 con 0x56322cee75d0 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f2205ffb640 1 -- 172.21.15.89:0/1263539114 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f220802f050 con 0x56322cee75d0 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f2226fd8740 10 client.27853 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f2226fd8740 1 -- 172.21.15.89:0/1263539114 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56322cf00aa0 con 0x56322cee75d0 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.906+0000 7f2205ffb640 1 -- 172.21.15.89:0/1263539114 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2208019070 con 0x56322cee75d0 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.907+0000 7f2205ffb640 1 -- 172.21.15.89:0/1263539114 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2208013070 con 0x56322cee75d0 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.907+0000 7f2226fd8740 10 client.27853 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:19.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.907+0000 7f2226fd8740 10 client.27853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.907+0000 7f2226fd8740 20 client.27853 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.907+0000 7f2226fd8740 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56322cf01e50 0x56322cf04240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.907+0000 7f2226fd8740 4 client.27853 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.907+0000 7f2226fd8740 1 -- 172.21.15.89:0/1263539114 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56322ce5a860 con 0x56322cf01e50 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.907+0000 7f221dd9b640 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56322cf01e50 0x56322cf04240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2205ffb640 10 client.27853 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f221dd9b640 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56322cf01e50 0x56322cf04240 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28d2c5b128176556 server_cookie=36c130a4afbd3a29 in_seq=0 out_seq=0 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2205ffb640 1 -- 172.21.15.89:0/1263539114 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56322ce5a860 con 0x56322cf01e50 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2205ffb640 10 client.27853 handle_command_reply: tid=0 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 client.27853 shutdown 2024-09-15T23:36:19.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 -- 172.21.15.89:0/1263539114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 msgr2=0x56322cef1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cef1d00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2208006a70 tx=0x7f2208014de0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 -- 172.21.15.89:0/1263539114 shutdown_connections 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56322cf01e50 0x56322cf04240 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322ceef530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceeefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2205ffb640 0 client.27853 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 --2- 172.21.15.89:0/1263539114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cef1d00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 -- 172.21.15.89:0/1263539114 >> 172.21.15.89:0/1263539114 conn(0x56322ceb0310 msgr2=0x56322cee9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 -- 172.21.15.89:0/1263539114 shutdown_connections 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.908+0000 7f2226fd8740 1 -- 172.21.15.89:0/1263539114 wait complete. 2024-09-15T23:36:19.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.909+0000 7f2226fd8740 20 client.27853 trim_cache size 0 max 16384 2024-09-15T23:36:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.921+0000 7f2226fd8740 1 Processor -- start 2024-09-15T23:36:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.921+0000 7f2226fd8740 1 -- start start 2024-09-15T23:36:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.921+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cefe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.921+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceedf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.921+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cf09840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.921+0000 7f2226fd8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f22201aedd0 con 0x56322cee8b40 2024-09-15T23:36:19.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.921+0000 7f2226fd8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f22201b0910 con 0x56322cef2370 2024-09-15T23:36:19.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.921+0000 7f2226fd8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f22201af4a0 con 0x56322cee75d0 2024-09-15T23:36:19.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f221dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceedf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f221dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceedf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42372/0 (socket says 172.21.15.89:42372) 2024-09-15T23:36:19.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f221dd9b640 1 -- 172.21.15.89:0/3866174032 learned_addr learned my addr 172.21.15.89:0/3866174032 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:19.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f221dd9b640 1 -- 172.21.15.89:0/3866174032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 msgr2=0x56322cefe310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:19.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f221dd9b640 1 --2- 172.21.15.89:0/3866174032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cefe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f221dd9b640 1 -- 172.21.15.89:0/3866174032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 msgr2=0x56322cf09840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:19.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f221dd9b640 1 --2- 172.21.15.89:0/3866174032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cf09840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f221dd9b640 1 -- 172.21.15.89:0/3866174032 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56322cee4960 con 0x56322cee8b40 2024-09-15T23:36:19.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f221dd9b640 1 --2- 172.21.15.89:0/3866174032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceedf90 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f2210098540 tx=0x7f22100bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842051f2bb491aab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:19.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f2207fff640 1 -- 172.21.15.89:0/3866174032 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22100a04e0 con 0x56322cee8b40 2024-09-15T23:36:19.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f2207fff640 1 -- 172.21.15.89:0/3866174032 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22100a4da0 con 0x56322cee8b40 2024-09-15T23:36:19.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.922+0000 7f2207fff640 1 -- 172.21.15.89:0/3866174032 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221000fc50 con 0x56322cee8b40 2024-09-15T23:36:19.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 -- 172.21.15.89:0/3866174032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 msgr2=0x56322ceedf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 --2- 172.21.15.89:0/3866174032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceedf90 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f2210098540 tx=0x7f22100bc040 comp rx=0 tx=0).stop 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 -- 172.21.15.89:0/3866174032 shutdown_connections 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 --2- 172.21.15.89:0/3866174032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cf09840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 --2- 172.21.15.89:0/3866174032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322ceedf90 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 --2- 172.21.15.89:0/3866174032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cefe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 -- 172.21.15.89:0/3866174032 >> 172.21.15.89:0/3866174032 conn(0x56322ceb0310 msgr2=0x56322ceed880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 -- 172.21.15.89:0/3866174032 shutdown_connections 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 -- 172.21.15.89:0/3866174032 wait complete. 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 -- start start 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 Processor -- start 2024-09-15T23:36:19.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cefa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cef5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cef61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f22201b0240 con 0x56322cee8b40 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f22201afb70 con 0x56322cef2370 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f22201acdb0 con 0x56322cee75d0 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.923+0000 7f2226fd8740 10 client.0 _ll_register_callbacks cb 0x7f2218009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cef5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cef5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42376/0 (socket says 172.21.15.89:42376) 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221dd9b640 1 -- 172.21.15.89:0/2338629909 learned_addr learned my addr 172.21.15.89:0/2338629909 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221e59c640 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cefa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221ed9d640 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cef61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221dd9b640 1 -- 172.21.15.89:0/2338629909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 msgr2=0x56322cefa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221dd9b640 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cefa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221dd9b640 1 -- 172.21.15.89:0/2338629909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 msgr2=0x56322cef61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221dd9b640 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cef61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221dd9b640 1 -- 172.21.15.89:0/2338629909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56322cef67a0 con 0x56322cee8b40 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221ed9d640 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cef61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f221dd9b640 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cef5ca0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f221009c0c0 tx=0x7f221009c240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbda975da8b1261 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f2205ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f2205ffb640 1 -- 172.21.15.89:0/2338629909 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221000fb30 con 0x56322cee8b40 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f2205ffb640 1 -- 172.21.15.89:0/2338629909 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f22100a0160 con 0x56322cee8b40 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f2226fd8740 1 -- 172.21.15.89:0/2338629909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22100a4da0 con 0x56322cee8b40 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f2205ffb640 1 -- 172.21.15.89:0/2338629909 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2210083050 con 0x56322cee8b40 2024-09-15T23:36:19.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f2226fd8740 10 client.19629 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f2226fd8740 1 -- 172.21.15.89:0/2338629909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56322cf6c5a0 con 0x56322cee8b40 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.924+0000 7f2205ffb640 1 -- 172.21.15.89:0/2338629909 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22100a4da0 con 0x56322cee8b40 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.925+0000 7f2205ffb640 1 -- 172.21.15.89:0/2338629909 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f22100a4910 con 0x56322cee8b40 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.925+0000 7f2226fd8740 10 client.19629 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.925+0000 7f2226fd8740 10 client.19629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.925+0000 7f2226fd8740 20 client.19629 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.925+0000 7f2226fd8740 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56322cf6cc60 0x56322cf6f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.925+0000 7f2226fd8740 4 client.19629 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.925+0000 7f2226fd8740 1 -- 172.21.15.89:0/2338629909 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56322ce54d70 con 0x56322cf6cc60 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.925+0000 7f221e59c640 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56322cf6cc60 0x56322cf6f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:19.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2205ffb640 10 client.19629 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.925+0000 7f221e59c640 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56322cf6cc60 0x56322cf6f080 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e00a8ecd2a23e634 server_cookie=a67b3b51aefc9204 in_seq=0 out_seq=0 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2205ffb640 1 -- 172.21.15.89:0/2338629909 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56322ce54d70 con 0x56322cf6cc60 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2205ffb640 10 client.19629 handle_command_reply: tid=0 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 client.19629 shutdown 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 -- 172.21.15.89:0/2338629909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 msgr2=0x56322cef5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cef5ca0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f221009c0c0 tx=0x7f221009c240 comp rx=0 tx=0).stop 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 -- 172.21.15.89:0/2338629909 shutdown_connections 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56322cf6cc60 0x56322cf6f080 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56322cef2370 0x56322cef61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56322cee8b40 0x56322cef5ca0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 --2- 172.21.15.89:0/2338629909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56322cee75d0 0x56322cefa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 -- 172.21.15.89:0/2338629909 >> 172.21.15.89:0/2338629909 conn(0x56322ceb0310 msgr2=0x56322ceee0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:19.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2205ffb640 0 client.19629 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:19.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 -- 172.21.15.89:0/2338629909 shutdown_connections 2024-09-15T23:36:19.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 1 -- 172.21.15.89:0/2338629909 wait complete. 2024-09-15T23:36:19.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.926+0000 7f2226fd8740 20 client.19629 trim_cache size 0 max 16384 2024-09-15T23:36:19.935 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:36:19.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 -- 172.21.15.89:0/3046583333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f220c0a3240 msgr2=0x7f220c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:19.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f220c0a3240 0x7f220c0a5700 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2218009430 tx=0x7f221802a950 comp rx=0 tx=0).stop 2024-09-15T23:36:19.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 -- 172.21.15.89:0/3046583333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2220103700 msgr2=0x7f2220198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:19.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2220103700 0x7f2220198930 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f221000c920 tx=0x7f221000cde0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 -- 172.21.15.89:0/3046583333 shutdown_connections 2024-09-15T23:36:19.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f220c0a3240 0x7f220c0a5700 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f222010c180 0x7f222019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2220103700 0x7f2220198930 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 --2- 172.21.15.89:0/3046583333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2220102de0 0x7f22201983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:19.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 -- 172.21.15.89:0/3046583333 >> 172.21.15.89:0/3046583333 conn(0x7f22200fc7a0 msgr2=0x7f22200fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:19.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 -- 172.21.15.89:0/3046583333 shutdown_connections 2024-09-15T23:36:19.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:19.927+0000 7f2224bff640 1 -- 172.21.15.89:0/3046583333 wait complete. 2024-09-15T23:36:20.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:20 smithi120 ceph-mon[35250]: pgmap v719: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-09-15T23:36:20.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:20 smithi174 ceph-mon[35586]: pgmap v719: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-09-15T23:36:21.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:20 smithi089 ceph-mon[30933]: pgmap v719: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-09-15T23:36:21.838 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:36:21.838 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:36:22.095 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:22.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.550+0000 7f9fe6d21640 1 -- 172.21.15.89:0/556459111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0108300 msgr2=0x7f9fe010ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:22.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.550+0000 7f9fe6d21640 1 --2- 172.21.15.89:0/556459111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0108300 0x7f9fe010ec70 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009cd0 tx=0x7f9fd802a8d0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.550+0000 7f9fe6d21640 1 -- 172.21.15.89:0/556459111 shutdown_connections 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.550+0000 7f9fe6d21640 1 --2- 172.21.15.89:0/556459111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0108300 0x7f9fe010ec70 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.550+0000 7f9fe6d21640 1 --2- 172.21.15.89:0/556459111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe0103780 0x7f9fe0107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.550+0000 7f9fe6d21640 1 --2- 172.21.15.89:0/556459111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe0102dd0 0x7f9fe01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.550+0000 7f9fe6d21640 1 -- 172.21.15.89:0/556459111 >> 172.21.15.89:0/556459111 conn(0x7f9fe00fc7d0 msgr2=0x7f9fe00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.550+0000 7f9fe6d21640 1 -- 172.21.15.89:0/556459111 shutdown_connections 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.550+0000 7f9fe6d21640 1 -- 172.21.15.89:0/556459111 wait complete. 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fe6d21640 1 Processor -- start 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fe6d21640 1 -- start start 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fe6d21640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe0102dd0 0x7f9fe0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fe6d21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0103780 0x7f9fe0198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fe6d21640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe0108300 0x7f9fe019ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fe6d21640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01120d0 con 0x7f9fe0103780 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fe6d21640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe0111f50 con 0x7f9fe0108300 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fe6d21640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe0112250 con 0x7f9fe0102dd0 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0103780 0x7f9fe0198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0103780 0x7f9fe0198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42384/0 (socket says 172.21.15.89:42384) 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fdffff640 1 -- 172.21.15.89:0/166408837 learned_addr learned my addr 172.21.15.89:0/166408837 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fe52be640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe0108300 0x7f9fe019ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe0102dd0 0x7f9fe0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fdffff640 1 -- 172.21.15.89:0/166408837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe0102dd0 msgr2=0x7f9fe0198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fdffff640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe0102dd0 0x7f9fe0198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fdffff640 1 -- 172.21.15.89:0/166408837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe0108300 msgr2=0x7f9fe019ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fdffff640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe0108300 0x7f9fe019ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.551+0000 7f9fdffff640 1 -- 172.21.15.89:0/166408837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe019d520 con 0x7f9fe0103780 2024-09-15T23:36:22.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.552+0000 7f9fdffff640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0103780 0x7f9fe0198ab0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00d740 tx=0x7f9fcc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.552+0000 7f9fddffb640 1 -- 172.21.15.89:0/166408837 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc017070 con 0x7f9fe0103780 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.552+0000 7f9fddffb640 1 -- 172.21.15.89:0/166408837 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fcc00de50 con 0x7f9fe0103780 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.552+0000 7f9fe6d21640 1 -- 172.21.15.89:0/166408837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe019d810 con 0x7f9fe0103780 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.552+0000 7f9fddffb640 1 -- 172.21.15.89:0/166408837 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc00f6b0 con 0x7f9fe0103780 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.552+0000 7f9fe6d21640 1 -- 172.21.15.89:0/166408837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe01a50f0 con 0x7f9fe0103780 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.554+0000 7f9fddffb640 1 -- 172.21.15.89:0/166408837 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9fcc01a070 con 0x7f9fe0103780 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.554+0000 7f9fe90fa740 1 Processor -- start 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe90fa740 1 -- start start 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fddffb640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd00a32c0 0x7f9fd00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f12e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 0x564467f21c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f24540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564467e78280 con 0x564467f12a40 2024-09-15T23:36:22.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564467e1fb10 con 0x564467f22150 2024-09-15T23:36:22.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564467e74240 con 0x564467f13360 2024-09-15T23:36:22.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe52be640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd00a32c0 0x7f9fd00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fddffb640 1 -- 172.21.15.89:0/166408837 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9fcc0cf640 con 0x7f9fe0103780 2024-09-15T23:36:22.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe4abd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f12e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe4abd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f12e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42396/0 (socket says 172.21.15.89:42396) 2024-09-15T23:36:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe4abd640 1 -- 172.21.15.89:0/3790242322 learned_addr learned my addr 172.21.15.89:0/3790242322 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fdffff640 1 --2- 172.21.15.89:0/3790242322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 0x564467f21c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe52be640 1 --2- 172.21.15.89:0/3790242322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f24540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe4abd640 1 -- 172.21.15.89:0/3790242322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 msgr2=0x564467f21c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/3790242322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 0x564467f21c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe4abd640 1 -- 172.21.15.89:0/3790242322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 msgr2=0x564467f24540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/3790242322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f24540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe4abd640 1 -- 172.21.15.89:0/3790242322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564467e2c1f0 con 0x564467f12a40 2024-09-15T23:36:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fdffff640 1 --2- 172.21.15.89:0/3790242322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 0x564467f21c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:22.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe52be640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd00a32c0 0x7f9fd00a5780 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8006010 tx=0x7f9fd8009a10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.555+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/3790242322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f12e20 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400dcb0 tx=0x7f9fd400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7d7cb57d17af56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fcaffd640 1 -- 172.21.15.89:0/3790242322 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4011440 con 0x564467f12a40 2024-09-15T23:36:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fcaffd640 1 -- 172.21.15.89:0/3790242322 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd4011ac0 con 0x564467f12a40 2024-09-15T23:36:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fcaffd640 1 -- 172.21.15.89:0/3790242322 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd401ba20 con 0x564467f12a40 2024-09-15T23:36:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fe90fa740 1 -- 172.21.15.89:0/3790242322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 msgr2=0x564467f12e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:22.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/3790242322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f12e20 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400dcb0 tx=0x7f9fd400f730 comp rx=0 tx=0).stop 2024-09-15T23:36:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fe90fa740 1 -- 172.21.15.89:0/3790242322 shutdown_connections 2024-09-15T23:36:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/3790242322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f24540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/3790242322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 0x564467f21c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/3790242322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f12e20 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fe90fa740 1 -- 172.21.15.89:0/3790242322 >> 172.21.15.89:0/3790242322 conn(0x564467f0f0f0 msgr2=0x564467f0f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fe90fa740 1 -- 172.21.15.89:0/3790242322 shutdown_connections 2024-09-15T23:36:22.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.556+0000 7f9fe90fa740 1 -- 172.21.15.89:0/3790242322 wait complete. 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.557+0000 7f9fe90fa740 1 -- start start 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.557+0000 7f9fe90fa740 1 Processor -- start 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.557+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f1f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.557+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 0x564467f1f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.557+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f1fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.557+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564467f24d20 con 0x564467f12a40 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.557+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564467e32f10 con 0x564467f22150 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.557+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564467f24ea0 con 0x564467f13360 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.557+0000 7f9fe90fa740 10 client.0 _ll_register_callbacks cb 0x564467dfba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.557+0000 7f9fe4abd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f1f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe4abd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f1f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42406/0 (socket says 172.21.15.89:42406) 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe4abd640 1 -- 172.21.15.89:0/4009820765 learned_addr learned my addr 172.21.15.89:0/4009820765 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:22.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fdffff640 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 0x564467f1f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe52be640 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f1fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe4abd640 1 -- 172.21.15.89:0/4009820765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 msgr2=0x564467f1f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 0x564467f1f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe4abd640 1 -- 172.21.15.89:0/4009820765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 msgr2=0x564467f1fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f1fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe4abd640 1 -- 172.21.15.89:0/4009820765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564467be96a0 con 0x564467f12a40 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fdffff640 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 0x564467f1f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f1f0c0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400dfd0 tx=0x7f9fd401b550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5fa95d1151cad64 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fbffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fbffff640 1 -- 172.21.15.89:0/4009820765 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4029060 con 0x564467f12a40 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fbffff640 1 -- 172.21.15.89:0/4009820765 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9fd4004250 con 0x564467f12a40 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe90fa740 1 -- 172.21.15.89:0/4009820765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fd4011ac0 con 0x564467f12a40 2024-09-15T23:36:22.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fbffff640 1 -- 172.21.15.89:0/4009820765 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9fd4030050 con 0x564467f12a40 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe90fa740 10 client.19641 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.558+0000 7f9fe90fa740 1 -- 172.21.15.89:0/4009820765 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564467f30950 con 0x564467f12a40 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fbffff640 1 -- 172.21.15.89:0/4009820765 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd4011ac0 con 0x564467f12a40 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fbffff640 1 -- 172.21.15.89:0/4009820765 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9fd401bb30 con 0x564467f12a40 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fe90fa740 10 client.19641 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fe90fa740 10 client.19641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fe90fa740 20 client.19641 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564467f31db0 0x564467f341a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fe90fa740 4 client.19641 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fe90fa740 1 -- 172.21.15.89:0/4009820765 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564467e857f0 con 0x564467f31db0 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fdffff640 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564467f31db0 0x564467f341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fdffff640 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564467f31db0 0x564467f341a0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c90df99844cdfe2 server_cookie=e23c8ef262aae226 in_seq=0 out_seq=0 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.559+0000 7f9fbffff640 10 client.19641 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:22.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fbffff640 1 -- 172.21.15.89:0/4009820765 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564467e857f0 con 0x564467f31db0 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fbffff640 10 client.19641 handle_command_reply: tid=0 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 client.19641 shutdown 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 -- 172.21.15.89:0/4009820765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 msgr2=0x564467f1f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f1f0c0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400dfd0 tx=0x7f9fd401b550 comp rx=0 tx=0).stop 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 -- 172.21.15.89:0/4009820765 shutdown_connections 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564467f31db0 0x564467f341a0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f1fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fbffff640 0 client.19641 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f13360 0x564467f1f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/4009820765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f1f0c0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 -- 172.21.15.89:0/4009820765 >> 172.21.15.89:0/4009820765 conn(0x564467f0f0f0 msgr2=0x564467f14e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 -- 172.21.15.89:0/4009820765 shutdown_connections 2024-09-15T23:36:22.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 1 -- 172.21.15.89:0/4009820765 wait complete. 2024-09-15T23:36:22.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.560+0000 7f9fe90fa740 20 client.19641 trim_cache size 0 max 16384 2024-09-15T23:36:22.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.577+0000 7f9fe90fa740 1 Processor -- start 2024-09-15T23:36:22.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.577+0000 7f9fe90fa740 1 -- start start 2024-09-15T23:36:22.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f12a40 0x564467f39820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f13360 0x564467f39d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f1c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01acda0 con 0x564467f13360 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01ae8e0 con 0x564467f22150 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01ad470 con 0x564467f12a40 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f13360 0x564467f39d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f13360 0x564467f39d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42420/0 (socket says 172.21.15.89:42420) 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fdffff640 1 -- 172.21.15.89:0/2275770735 learned_addr learned my addr 172.21.15.89:0/2275770735 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/2275770735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f12a40 0x564467f39820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fe52be640 1 --2- 172.21.15.89:0/2275770735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f1c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fdffff640 1 -- 172.21.15.89:0/2275770735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f12a40 msgr2=0x564467f39820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fdffff640 1 --2- 172.21.15.89:0/2275770735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f12a40 0x564467f39820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fdffff640 1 -- 172.21.15.89:0/2275770735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 msgr2=0x564467f1c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fdffff640 1 --2- 172.21.15.89:0/2275770735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f1c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fdffff640 1 -- 172.21.15.89:0/2275770735 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564467f13190 con 0x564467f13360 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fe52be640 1 --2- 172.21.15.89:0/2275770735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f1c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fdffff640 1 --2- 172.21.15.89:0/2275770735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f13360 0x564467f39d60 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc005d80 tx=0x7f9fcc010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf9558ff6ddd82f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fca7fc640 1 -- 172.21.15.89:0/2275770735 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc021040 con 0x564467f13360 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fca7fc640 1 -- 172.21.15.89:0/2275770735 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fcc09fc20 con 0x564467f13360 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.578+0000 7f9fca7fc640 1 -- 172.21.15.89:0/2275770735 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc098070 con 0x564467f13360 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.579+0000 7f9fe90fa740 1 -- 172.21.15.89:0/2275770735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f13360 msgr2=0x564467f39d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.579+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/2275770735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f13360 0x564467f39d60 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc005d80 tx=0x7f9fcc010d40 comp rx=0 tx=0).stop 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.579+0000 7f9fe90fa740 1 -- 172.21.15.89:0/2275770735 shutdown_connections 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.579+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/2275770735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f22150 0x564467f1c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.579+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/2275770735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f13360 0x564467f39d60 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.579+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/2275770735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f12a40 0x564467f39820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.579+0000 7f9fe90fa740 1 -- 172.21.15.89:0/2275770735 >> 172.21.15.89:0/2275770735 conn(0x564467f0f0f0 msgr2=0x564467f0e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:22.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.579+0000 7f9fe90fa740 1 -- 172.21.15.89:0/2275770735 shutdown_connections 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.579+0000 7f9fe90fa740 1 -- 172.21.15.89:0/2275770735 wait complete. 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.579+0000 7f9fe90fa740 1 -- start start 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe90fa740 1 Processor -- start 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f26720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f13360 0x564467f26c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe90fa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f22150 0x564467f957c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01ae210 con 0x564467f12a40 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01adb40 con 0x564467f13360 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe90fa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01aad80 con 0x564467f22150 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe90fa740 10 client.0 _ll_register_callbacks cb 0x7f9fd4015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe4abd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f26720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe4abd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f26720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42426/0 (socket says 172.21.15.89:42426) 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe4abd640 1 -- 172.21.15.89:0/1307757927 learned_addr learned my addr 172.21.15.89:0/1307757927 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:22.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fdffff640 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f13360 0x564467f26c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe4abd640 1 -- 172.21.15.89:0/1307757927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f22150 msgr2=0x564467f957c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f22150 0x564467f957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe4abd640 1 -- 172.21.15.89:0/1307757927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f13360 msgr2=0x564467f26c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f13360 0x564467f26c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.580+0000 7f9fe4abd640 1 -- 172.21.15.89:0/1307757927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564467f95e00 con 0x564467f12a40 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fe4abd640 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f26720 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400dfd0 tx=0x7f9fd4003a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e33ba94736eaf87 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fbffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fbffff640 1 -- 172.21.15.89:0/1307757927 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4029050 con 0x564467f12a40 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fbffff640 1 -- 172.21.15.89:0/1307757927 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9fd4004460 con 0x564467f12a40 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fe90fa740 1 -- 172.21.15.89:0/1307757927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fcc09fc20 con 0x564467f12a40 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fbffff640 1 -- 172.21.15.89:0/1307757927 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9fd4041050 con 0x564467f12a40 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fe90fa740 10 client.19647 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:22.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fe90fa740 1 -- 172.21.15.89:0/1307757927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564467f9d1d0 con 0x564467f12a40 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fbffff640 1 -- 172.21.15.89:0/1307757927 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fcc09fc20 con 0x564467f12a40 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fbffff640 1 -- 172.21.15.89:0/1307757927 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9fd403c030 con 0x564467f12a40 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fe90fa740 10 client.19647 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fe90fa740 10 client.19647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fe90fa740 20 client.19647 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564467f9d910 0x564467f9fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fe90fa740 4 client.19647 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.581+0000 7f9fe90fa740 1 -- 172.21.15.89:0/1307757927 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564467e7fd00 con 0x564467f9d910 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.582+0000 7f9fdffff640 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564467f9d910 0x564467f9fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.582+0000 7f9fdffff640 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564467f9d910 0x564467f9fd50 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b7e68eee23e6901 server_cookie=2342091a7bbfd8a0 in_seq=0 out_seq=0 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.582+0000 7f9fbffff640 10 client.19647 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fbffff640 1 -- 172.21.15.89:0/1307757927 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564467e7fd00 con 0x564467f9d910 2024-09-15T23:36:22.594 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fbffff640 10 client.19647 handle_command_reply: tid=0 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 client.19647 shutdown 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 -- 172.21.15.89:0/1307757927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 msgr2=0x564467f26720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f26720 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400dfd0 tx=0x7f9fd4003a90 comp rx=0 tx=0).stop 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 -- 172.21.15.89:0/1307757927 shutdown_connections 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564467f9d910 0x564467f9fd50 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564467f22150 0x564467f957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564467f13360 0x564467f26c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fbffff640 0 client.19647 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 --2- 172.21.15.89:0/1307757927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564467f12a40 0x564467f26720 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 -- 172.21.15.89:0/1307757927 >> 172.21.15.89:0/1307757927 conn(0x564467f0f0f0 msgr2=0x564467f14550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 -- 172.21.15.89:0/1307757927 shutdown_connections 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.583+0000 7f9fe90fa740 1 -- 172.21.15.89:0/1307757927 wait complete. 2024-09-15T23:36:22.595 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe90fa740 20 client.19647 trim_cache size 0 max 16384 2024-09-15T23:36:22.596 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:36:22.596 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:36:22.596 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "728bfc76-1e68-4c3a-b51d-630d47e2b1b2", 2024-09-15T23:36:22.596 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:36:22.596 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:36:22.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 -- 172.21.15.89:0/166408837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd00a32c0 msgr2=0x7f9fd00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:22.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd00a32c0 0x7f9fd00a5780 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8006010 tx=0x7f9fd8009a10 comp rx=0 tx=0).stop 2024-09-15T23:36:22.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 -- 172.21.15.89:0/166408837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0103780 msgr2=0x7f9fe0198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:22.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0103780 0x7f9fe0198ab0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00d740 tx=0x7f9fcc00dc00 comp rx=0 tx=0).stop 2024-09-15T23:36:22.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 -- 172.21.15.89:0/166408837 shutdown_connections 2024-09-15T23:36:22.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9fd00a32c0 0x7f9fd00a5780 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe0108300 0x7f9fe019ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9fe0103780 0x7f9fe0198ab0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 --2- 172.21.15.89:0/166408837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe0102dd0 0x7f9fe0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:22.597 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 -- 172.21.15.89:0/166408837 >> 172.21.15.89:0/166408837 conn(0x7f9fe00fc7d0 msgr2=0x7f9fe0105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:22.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 -- 172.21.15.89:0/166408837 shutdown_connections 2024-09-15T23:36:22.598 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:22.584+0000 7f9fe6d21640 1 -- 172.21.15.89:0/166408837 wait complete. 2024-09-15T23:36:22.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:22 smithi089 ceph-mon[30933]: pgmap v720: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-09-15T23:36:22.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:22 smithi120 ceph-mon[35250]: pgmap v720: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-09-15T23:36:22.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:22 smithi174 ceph-mon[35586]: pgmap v720: 129 pgs: 129 active+clean; 1.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.2 KiB/s wr, 14 op/s 2024-09-15T23:36:23.403 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:36:23.654 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:23.948 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:23 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:36:23.948 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:23 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:23.948 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:23 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:36:23.949 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:23 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:23.949 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:23 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:36:23.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:23 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:36:23.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:23 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:23.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:23 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:36:23.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:23 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:23.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:23 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:36:23.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:23 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:36:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:23 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:23 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:36:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:23 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:23.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:23 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:36:24.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.163+0000 7f354a888640 1 -- 172.21.15.89:0/385403949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3544101df0 msgr2=0x7f35441021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.163+0000 7f354a888640 1 --2- 172.21.15.89:0/385403949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3544101df0 0x7f35441021d0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f3534009900 tx=0x7f353402a410 comp rx=0 tx=0).stop 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.163+0000 7f354a888640 1 -- 172.21.15.89:0/385403949 shutdown_connections 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.163+0000 7f354a888640 1 --2- 172.21.15.89:0/385403949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3544106f20 0x7f354410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.163+0000 7f354a888640 1 --2- 172.21.15.89:0/385403949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35441027a0 0x7f35441067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.163+0000 7f354a888640 1 --2- 172.21.15.89:0/385403949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3544101df0 0x7f35441021d0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.163+0000 7f354a888640 1 -- 172.21.15.89:0/385403949 >> 172.21.15.89:0/385403949 conn(0x7f35440fd8b0 msgr2=0x7f35440ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.163+0000 7f354a888640 1 -- 172.21.15.89:0/385403949 shutdown_connections 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.163+0000 7f354a888640 1 -- 172.21.15.89:0/385403949 wait complete. 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f354a888640 1 Processor -- start 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f354a888640 1 -- start start 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f354a888640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3544101df0 0x7f35441983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f354a888640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35441027a0 0x7f3544198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f354a888640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3544106f20 0x7f354419cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f354a888640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3544111f80 con 0x7f3544101df0 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f354a888640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3544111e00 con 0x7f3544106f20 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f354a888640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3544112100 con 0x7f35441027a0 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f3543fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3544101df0 0x7f35441983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f3543fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3544101df0 0x7f35441983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42448/0 (socket says 172.21.15.89:42448) 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.164+0000 7f3543fff640 1 -- 172.21.15.89:0/1584261470 learned_addr learned my addr 172.21.15.89:0/1584261470 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f35437fe640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35441027a0 0x7f3544198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f3548e25640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3544106f20 0x7f354419cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f3543fff640 1 -- 172.21.15.89:0/1584261470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35441027a0 msgr2=0x7f3544198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:24.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f3543fff640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35441027a0 0x7f3544198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f3543fff640 1 -- 172.21.15.89:0/1584261470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3544106f20 msgr2=0x7f354419cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:24.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f3543fff640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3544106f20 0x7f354419cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f3543fff640 1 -- 172.21.15.89:0/1584261470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354419d370 con 0x7f3544101df0 2024-09-15T23:36:24.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f3543fff640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3544101df0 0x7f35441983c0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f3534006010 tx=0x7f35340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:24.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f35417fa640 1 -- 172.21.15.89:0/1584261470 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353403f030 con 0x7f3544101df0 2024-09-15T23:36:24.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f354a888640 1 -- 172.21.15.89:0/1584261470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354419d600 con 0x7f3544101df0 2024-09-15T23:36:24.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f35417fa640 1 -- 172.21.15.89:0/1584261470 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3534003ce0 con 0x7f3544101df0 2024-09-15T23:36:24.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f35417fa640 1 -- 172.21.15.89:0/1584261470 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534035070 con 0x7f3544101df0 2024-09-15T23:36:24.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.165+0000 7f354a888640 1 -- 172.21.15.89:0/1584261470 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35441a4f40 con 0x7f3544101df0 2024-09-15T23:36:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.167+0000 7f35417fa640 1 -- 172.21.15.89:0/1584261470 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3534042030 con 0x7f3544101df0 2024-09-15T23:36:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f354cc61740 1 Processor -- start 2024-09-15T23:36:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f354cc61740 1 -- start start 2024-09-15T23:36:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f354cc61740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 0x5555d2734740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f354cc61740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2743550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f354cc61740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2743a90 0x5555d2745e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f354cc61740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5555d269a2e0 con 0x5555d2743a90 2024-09-15T23:36:24.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f354cc61740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5555d2641ce0 con 0x5555d2734360 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f354cc61740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5555d26962a0 con 0x5555d2734c80 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f35417fa640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f352c0a3240 0x7f352c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3548e25640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f352c0a3240 0x7f352c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3543fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2743550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f35437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2743a90 0x5555d2745e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3543fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2743550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:51704/0 (socket says 172.21.15.89:51704) 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f35437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2743a90 0x5555d2745e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42460/0 (socket says 172.21.15.89:42460) 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3543fff640 1 -- 172.21.15.89:0/3279633855 learned_addr learned my addr 172.21.15.89:0/3279633855 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3548e25640 1 --2- 172.21.15.89:0/3279633855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 0x5555d2734740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f35417fa640 1 -- 172.21.15.89:0/1584261470 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f35340f03d0 con 0x7f3544101df0 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3548e25640 1 -- 172.21.15.89:0/3279633855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 msgr2=0x5555d2743550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3548e25640 1 --2- 172.21.15.89:0/3279633855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2743550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3548e25640 1 -- 172.21.15.89:0/3279633855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2743a90 msgr2=0x5555d2745e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3548e25640 1 --2- 172.21.15.89:0/3279633855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2743a90 0x5555d2745e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3548e25640 1 -- 172.21.15.89:0/3279633855 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555d2626440 con 0x5555d2734360 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f35437fe640 1 --2- 172.21.15.89:0/3279633855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2743a90 0x5555d2745e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.168+0000 7f3543fff640 1 --2- 172.21.15.89:0/3279633855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2743550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.169+0000 7f3548e25640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f352c0a3240 0x7f352c0a5700 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f353c005c40 tx=0x7f353c009c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.169+0000 7f3548e25640 1 --2- 172.21.15.89:0/3279633855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 0x5555d2734740 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f353c005f10 tx=0x7f353c01c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4580c17ffa3dbce4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.169+0000 7f352a7fc640 1 -- 172.21.15.89:0/3279633855 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c004b20 con 0x5555d2734360 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.169+0000 7f352a7fc640 1 -- 172.21.15.89:0/3279633855 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f353c004cc0 con 0x5555d2734360 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.169+0000 7f352a7fc640 1 -- 172.21.15.89:0/3279633855 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c0051e0 con 0x5555d2734360 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 -- 172.21.15.89:0/3279633855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 msgr2=0x5555d2734740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 --2- 172.21.15.89:0/3279633855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 0x5555d2734740 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f353c005f10 tx=0x7f353c01c000 comp rx=0 tx=0).stop 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 -- 172.21.15.89:0/3279633855 shutdown_connections 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 --2- 172.21.15.89:0/3279633855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2743a90 0x5555d2745e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 --2- 172.21.15.89:0/3279633855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2743550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 --2- 172.21.15.89:0/3279633855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 0x5555d2734740 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 -- 172.21.15.89:0/3279633855 >> 172.21.15.89:0/3279633855 conn(0x5555d2730a30 msgr2=0x5555d2730e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 -- 172.21.15.89:0/3279633855 shutdown_connections 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 -- 172.21.15.89:0/3279633855 wait complete. 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 -- start start 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 Processor -- start 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.170+0000 7f354cc61740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734360 0x5555d2743480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f354cc61740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2740730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f354cc61740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2743a90 0x5555d2740d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f354cc61740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5555d2746600 con 0x5555d2734360 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f354cc61740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5555d2655050 con 0x5555d2743a90 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f354cc61740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5555d2746780 con 0x5555d2734c80 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f354cc61740 10 client.0 _ll_register_callbacks cb 0x5555d2695160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:24.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f3543fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734360 0x5555d2743480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f3543fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734360 0x5555d2743480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42470/0 (socket says 172.21.15.89:42470) 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f3543fff640 1 -- 172.21.15.89:0/1287048655 learned_addr learned my addr 172.21.15.89:0/1287048655 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f3548e25640 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2743a90 0x5555d2740d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f35437fe640 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2740730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f3543fff640 1 -- 172.21.15.89:0/1287048655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 msgr2=0x5555d2740730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f3543fff640 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2740730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f3543fff640 1 -- 172.21.15.89:0/1287048655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2743a90 msgr2=0x5555d2740d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f3543fff640 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2743a90 0x5555d2740d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f3543fff640 1 -- 172.21.15.89:0/1287048655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555d240b3c0 con 0x5555d2734360 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f35437fe640 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2740730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f3543fff640 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734360 0x5555d2743480 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f35340c4ea0 tx=0x7f353408b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7ab2a882e49235 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f35237fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f35237fe640 1 -- 172.21.15.89:0/1287048655 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534049040 con 0x5555d2734360 2024-09-15T23:36:24.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f35237fe640 1 -- 172.21.15.89:0/1287048655 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f35340baa60 con 0x5555d2734360 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f354cc61740 1 -- 172.21.15.89:0/1287048655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f353c004cc0 con 0x5555d2734360 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f35237fe640 1 -- 172.21.15.89:0/1287048655 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f35340bc2a0 con 0x5555d2734360 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.171+0000 7f354cc61740 10 client.19659 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.172+0000 7f354cc61740 1 -- 172.21.15.89:0/1287048655 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5555d27520f0 con 0x5555d2734360 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.172+0000 7f35237fe640 1 -- 172.21.15.89:0/1287048655 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f353c004cc0 con 0x5555d2734360 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.172+0000 7f35237fe640 1 -- 172.21.15.89:0/1287048655 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f35340b8dd0 con 0x5555d2734360 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.172+0000 7f354cc61740 10 client.19659 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.172+0000 7f354cc61740 10 client.19659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.172+0000 7f354cc61740 20 client.19659 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.172+0000 7f354cc61740 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555d2753440 0x5555d2755830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.172+0000 7f354cc61740 4 client.19659 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.172+0000 7f354cc61740 1 -- 172.21.15.89:0/1287048655 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555d26a7850 con 0x5555d2753440 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.172+0000 7f35437fe640 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555d2753440 0x5555d2755830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.173+0000 7f35437fe640 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555d2753440 0x5555d2755830 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5387f72838c8877d server_cookie=2e64d128802ce218 in_seq=0 out_seq=0 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.173+0000 7f35237fe640 10 client.19659 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.173+0000 7f35237fe640 1 -- 172.21.15.89:0/1287048655 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5555d26a7850 con 0x5555d2753440 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.173+0000 7f35237fe640 10 client.19659 handle_command_reply: tid=0 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.173+0000 7f354cc61740 1 client.19659 shutdown 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.173+0000 7f354cc61740 1 -- 172.21.15.89:0/1287048655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734360 msgr2=0x5555d2743480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.173+0000 7f354cc61740 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734360 0x5555d2743480 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f35340c4ea0 tx=0x7f353408b560 comp rx=0 tx=0).stop 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.173+0000 7f354cc61740 1 -- 172.21.15.89:0/1287048655 shutdown_connections 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.173+0000 7f354cc61740 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555d2753440 0x5555d2755830 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.173+0000 7f354cc61740 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2743a90 0x5555d2740d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.174+0000 7f354cc61740 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734c80 0x5555d2740730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.174+0000 7f354cc61740 1 --2- 172.21.15.89:0/1287048655 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734360 0x5555d2743480 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.174+0000 7f35237fe640 0 client.19659 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.174+0000 7f354cc61740 1 -- 172.21.15.89:0/1287048655 >> 172.21.15.89:0/1287048655 conn(0x5555d2730a30 msgr2=0x5555d2736790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.174+0000 7f354cc61740 1 -- 172.21.15.89:0/1287048655 shutdown_connections 2024-09-15T23:36:24.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.174+0000 7f354cc61740 1 -- 172.21.15.89:0/1287048655 wait complete. 2024-09-15T23:36:24.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.174+0000 7f354cc61740 20 client.19659 trim_cache size 0 max 16384 2024-09-15T23:36:24.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.190+0000 7f354cc61740 1 Processor -- start 2024-09-15T23:36:24.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.190+0000 7f354cc61740 1 -- start start 2024-09-15T23:36:24.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f354cc61740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734360 0x5555d274f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f354cc61740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d275b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f354cc61740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2743a90 0x5555d275b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f354cc61740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f35441acba0 con 0x5555d2734c80 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f354cc61740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f35441ae6e0 con 0x5555d2743a90 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f354cc61740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f35441ad270 con 0x5555d2734360 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f35437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d275b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f35437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d275b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42480/0 (socket says 172.21.15.89:42480) 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f35437fe640 1 -- 172.21.15.89:0/3954615338 learned_addr learned my addr 172.21.15.89:0/3954615338 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f3543fff640 1 --2- 172.21.15.89:0/3954615338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734360 0x5555d274f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f35437fe640 1 -- 172.21.15.89:0/3954615338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734360 msgr2=0x5555d274f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f35437fe640 1 --2- 172.21.15.89:0/3954615338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734360 0x5555d274f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f35437fe640 1 -- 172.21.15.89:0/3954615338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2743a90 msgr2=0x5555d275b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f3548e25640 1 --2- 172.21.15.89:0/3954615338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2743a90 0x5555d275b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f35437fe640 1 --2- 172.21.15.89:0/3954615338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2743a90 0x5555d275b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f35437fe640 1 -- 172.21.15.89:0/3954615338 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555d2741050 con 0x5555d2734c80 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f35437fe640 1 --2- 172.21.15.89:0/3954615338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d275b0e0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x5555d2743a60 tx=0x7f353800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dfc8166c1373850 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:24.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f3529ffb640 1 -- 172.21.15.89:0/3954615338 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3538014070 con 0x5555d2734c80 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f3529ffb640 1 -- 172.21.15.89:0/3954615338 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3538003de0 con 0x5555d2734c80 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.191+0000 7f3529ffb640 1 -- 172.21.15.89:0/3954615338 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353801c3b0 con 0x5555d2734c80 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 -- 172.21.15.89:0/3954615338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 msgr2=0x5555d275b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 --2- 172.21.15.89:0/3954615338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d275b0e0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x5555d2743a60 tx=0x7f353800cd80 comp rx=0 tx=0).stop 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 -- 172.21.15.89:0/3954615338 shutdown_connections 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 --2- 172.21.15.89:0/3954615338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2743a90 0x5555d275b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 --2- 172.21.15.89:0/3954615338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d275b0e0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 --2- 172.21.15.89:0/3954615338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2734360 0x5555d274f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 -- 172.21.15.89:0/3954615338 >> 172.21.15.89:0/3954615338 conn(0x5555d2730a30 msgr2=0x5555d273bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 -- 172.21.15.89:0/3954615338 shutdown_connections 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 -- 172.21.15.89:0/3954615338 wait complete. 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 -- start start 2024-09-15T23:36:24.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.192+0000 7f354cc61740 1 Processor -- start 2024-09-15T23:36:24.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f354cc61740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 0x5555d2747bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f354cc61740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d2748130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f354cc61740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2743a90 0x5555d27bd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f354cc61740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f35441ae010 con 0x5555d2734c80 2024-09-15T23:36:24.198 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f354cc61740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f35441ad940 con 0x5555d2734360 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f354cc61740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f35441aab80 con 0x5555d2743a90 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f354cc61740 10 client.0 _ll_register_callbacks cb 0x7f35340edaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f35437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d2748130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f35437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d2748130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:42482/0 (socket says 172.21.15.89:42482) 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f35437fe640 1 -- 172.21.15.89:0/2776332137 learned_addr learned my addr 172.21.15.89:0/2776332137 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f3543fff640 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 0x5555d2747bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f3548e25640 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2743a90 0x5555d27bd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f35437fe640 1 -- 172.21.15.89:0/2776332137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2743a90 msgr2=0x5555d27bd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f35437fe640 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2743a90 0x5555d27bd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f35437fe640 1 -- 172.21.15.89:0/2776332137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 msgr2=0x5555d2747bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f35437fe640 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 0x5555d2747bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f35437fe640 1 -- 172.21.15.89:0/2776332137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555d27bd5f0 con 0x5555d2734c80 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f35437fe640 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d2748130 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f353800d010 tx=0x7f3538004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730c35a200fb2515 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:24.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.193+0000 7f35237fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f35237fe640 1 -- 172.21.15.89:0/2776332137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353802b070 con 0x5555d2734c80 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f35237fe640 1 -- 172.21.15.89:0/2776332137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3538005220 con 0x5555d2734c80 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f354cc61740 1 -- 172.21.15.89:0/2776332137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3538003de0 con 0x5555d2734c80 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f35237fe640 1 -- 172.21.15.89:0/2776332137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3538037050 con 0x5555d2734c80 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f354cc61740 10 client.19665 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f354cc61740 1 -- 172.21.15.89:0/2776332137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5555d27be7e0 con 0x5555d2734c80 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f35237fe640 1 -- 172.21.15.89:0/2776332137 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3538003de0 con 0x5555d2734c80 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f35237fe640 1 -- 172.21.15.89:0/2776332137 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3538030440 con 0x5555d2734c80 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f354cc61740 10 client.19665 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f354cc61740 10 client.19665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f354cc61740 20 client.19665 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f354cc61740 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555d27bee80 0x5555d27c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f354cc61740 4 client.19665 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:36:24.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f354cc61740 1 -- 172.21.15.89:0/2776332137 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5555d26a1d60 con 0x5555d27bee80 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.194+0000 7f3543fff640 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555d27bee80 0x5555d27c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f3543fff640 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555d27bee80 0x5555d27c12a0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ee01eecfc9d8f03 server_cookie=6213bd8573b1844a in_seq=0 out_seq=0 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f35237fe640 10 client.19665 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f35237fe640 1 -- 172.21.15.89:0/2776332137 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5555d26a1d60 con 0x5555d27bee80 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f35237fe640 10 client.19665 handle_command_reply: tid=0 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 client.19665 shutdown 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 -- 172.21.15.89:0/2776332137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 msgr2=0x5555d2748130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d2748130 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f353800d010 tx=0x7f3538004f60 comp rx=0 tx=0).stop 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 -- 172.21.15.89:0/2776332137 shutdown_connections 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5555d27bee80 0x5555d27c12a0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5555d2743a90 0x5555d27bd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5555d2734c80 0x5555d2748130 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 --2- 172.21.15.89:0/2776332137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555d2734360 0x5555d2747bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f35237fe640 0 client.19665 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:24.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 -- 172.21.15.89:0/2776332137 >> 172.21.15.89:0/2776332137 conn(0x5555d2730a30 msgr2=0x5555d272f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:24.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 -- 172.21.15.89:0/2776332137 shutdown_connections 2024-09-15T23:36:24.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 1 -- 172.21.15.89:0/2776332137 wait complete. 2024-09-15T23:36:24.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.195+0000 7f354cc61740 20 client.19665 trim_cache size 0 max 16384 2024-09-15T23:36:24.202 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:36:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:36:24.202 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:36:24.202 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:36:24.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 -- 172.21.15.89:0/1584261470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f352c0a3240 msgr2=0x7f352c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:24.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f352c0a3240 0x7f352c0a5700 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f353c005c40 tx=0x7f353c009c90 comp rx=0 tx=0).stop 2024-09-15T23:36:24.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 -- 172.21.15.89:0/1584261470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3544101df0 msgr2=0x7f35441983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:24.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3544101df0 0x7f35441983c0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f3534006010 tx=0x7f35340042f0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 -- 172.21.15.89:0/1584261470 shutdown_connections 2024-09-15T23:36:24.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f352c0a3240 0x7f352c0a5700 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3544106f20 0x7f354419cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35441027a0 0x7f3544198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 --2- 172.21.15.89:0/1584261470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3544101df0 0x7f35441983c0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:24.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 -- 172.21.15.89:0/1584261470 >> 172.21.15.89:0/1584261470 conn(0x7f35440fd8b0 msgr2=0x7f35440feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:24.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 -- 172.21.15.89:0/1584261470 shutdown_connections 2024-09-15T23:36:24.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:24.196+0000 7f354a888640 1 -- 172.21.15.89:0/1584261470 wait complete. 2024-09-15T23:36:24.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:24 smithi089 ceph-mon[30933]: pgmap v721: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:24.935 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:36:24.936 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:36:24.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:24 smithi120 ceph-mon[35250]: pgmap v721: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:24.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:24 smithi174 ceph-mon[35586]: pgmap v721: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:25.180 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.657+0000 7fc538af4640 1 -- 172.21.15.89:0/138296429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc534106f20 msgr2=0x7fc53410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.657+0000 7fc538af4640 1 --2- 172.21.15.89:0/138296429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc534106f20 0x7fc53410eaa0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fc524009900 tx=0x7fc52402a410 comp rx=0 tx=0).stop 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.657+0000 7fc538af4640 1 -- 172.21.15.89:0/138296429 shutdown_connections 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.657+0000 7fc538af4640 1 --2- 172.21.15.89:0/138296429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc534106f20 0x7fc53410eaa0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.657+0000 7fc538af4640 1 --2- 172.21.15.89:0/138296429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5341027a0 0x7fc5341067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.657+0000 7fc538af4640 1 --2- 172.21.15.89:0/138296429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc534101df0 0x7fc5341021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.657+0000 7fc538af4640 1 -- 172.21.15.89:0/138296429 >> 172.21.15.89:0/138296429 conn(0x7fc5340fd8f0 msgr2=0x7fc5340ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.657+0000 7fc538af4640 1 -- 172.21.15.89:0/138296429 shutdown_connections 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.657+0000 7fc538af4640 1 -- 172.21.15.89:0/138296429 wait complete. 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.658+0000 7fc538af4640 1 Processor -- start 2024-09-15T23:36:25.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.658+0000 7fc538af4640 1 -- start start 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.658+0000 7fc538af4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc534101df0 0x7fc5340744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.658+0000 7fc538af4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5341027a0 0x7fc534074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.658+0000 7fc538af4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc534106f20 0x7fc534074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.658+0000 7fc538af4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc534111f80 con 0x7fc5341027a0 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.658+0000 7fc538af4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc534111e00 con 0x7fc534101df0 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.658+0000 7fc538af4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc534112100 con 0x7fc534106f20 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.658+0000 7fc531d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5341027a0 0x7fc534074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.658+0000 7fc531d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5341027a0 0x7fc534074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47708/0 (socket says 172.21.15.89:47708) 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc531d9b640 1 -- 172.21.15.89:0/32085408 learned_addr learned my addr 172.21.15.89:0/32085408 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc53259c640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc534101df0 0x7fc5340744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc532d9d640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc534106f20 0x7fc534074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc531d9b640 1 -- 172.21.15.89:0/32085408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc534106f20 msgr2=0x7fc534074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:25.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc531d9b640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc534106f20 0x7fc534074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc531d9b640 1 -- 172.21.15.89:0/32085408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc534101df0 msgr2=0x7fc5340744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc531d9b640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc534101df0 0x7fc5340744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc531d9b640 1 -- 172.21.15.89:0/32085408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5341a6880 con 0x7fc5341027a0 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc531d9b640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5341027a0 0x7fc534074a30 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fc52800db80 tx=0x7fc52800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc5237fe640 1 -- 172.21.15.89:0/32085408 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc528017070 con 0x7fc5341027a0 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc5237fe640 1 -- 172.21.15.89:0/32085408 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc528003b70 con 0x7fc5341027a0 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc5237fe640 1 -- 172.21.15.89:0/32085408 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5280106c0 con 0x7fc5341027a0 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc538af4640 1 -- 172.21.15.89:0/32085408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5341a6b70 con 0x7fc5341027a0 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.659+0000 7fc538af4640 1 -- 172.21.15.89:0/32085408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5341a7060 con 0x7fc5341027a0 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.660+0000 7fc538af4640 1 -- 172.21.15.89:0/32085408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5341021d0 con 0x7fc5341027a0 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.661+0000 7fc5237fe640 1 -- 172.21.15.89:0/32085408 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc528004020 con 0x7fc5341027a0 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.662+0000 7fc5237fe640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc51c0a32c0 0x7fc51c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.662+0000 7fc53259c640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc51c0a32c0 0x7fc51c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:25.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.662+0000 7fc5237fe640 1 -- 172.21.15.89:0/32085408 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc5280cfa00 con 0x7fc5341027a0 2024-09-15T23:36:25.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.663+0000 7fc53259c640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc51c0a32c0 0x7fc51c0a5780 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00bbe0 tx=0x7fc52c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:25.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.666+0000 7fc5237fe640 1 -- 172.21.15.89:0/32085408 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc528020070 con 0x7fc5341027a0 2024-09-15T23:36:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.866+0000 7fc538af4640 1 -- 172.21.15.89:0/32085408 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc53402d7c0 con 0x7fc5341027a0 2024-09-15T23:36:25.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.866+0000 7fc5237fe640 1 -- 172.21.15.89:0/32085408 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fc52809c330 con 0x7fc5341027a0 2024-09-15T23:36:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:36:25.868 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:36:25.868 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:36:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 -- 172.21.15.89:0/32085408 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc51c0a32c0 msgr2=0x7fc51c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc51c0a32c0 0x7fc51c0a5780 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00bbe0 tx=0x7fc52c00b390 comp rx=0 tx=0).stop 2024-09-15T23:36:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 -- 172.21.15.89:0/32085408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5341027a0 msgr2=0x7fc534074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5341027a0 0x7fc534074a30 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fc52800db80 tx=0x7fc52800b630 comp rx=0 tx=0).stop 2024-09-15T23:36:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 -- 172.21.15.89:0/32085408 shutdown_connections 2024-09-15T23:36:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc51c0a32c0 0x7fc51c0a5780 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc534106f20 0x7fc534074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:25.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5341027a0 0x7fc534074a30 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 --2- 172.21.15.89:0/32085408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc534101df0 0x7fc5340744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 -- 172.21.15.89:0/32085408 >> 172.21.15.89:0/32085408 conn(0x7fc5340fd8f0 msgr2=0x7fc5340febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 -- 172.21.15.89:0/32085408 shutdown_connections 2024-09-15T23:36:25.871 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:25.869+0000 7fc538af4640 1 -- 172.21.15.89:0/32085408 wait complete. 2024-09-15T23:36:26.662 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:36:26.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:26 smithi089 ceph-mon[30933]: pgmap v722: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:26.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:26 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/32085408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:36:26.909 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:26.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:26 smithi120 ceph-mon[35250]: pgmap v722: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:26.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:26 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/32085408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:36:26.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:26 smithi174 ceph-mon[35586]: pgmap v722: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:26.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:26 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/32085408' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:36:27.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.393+0000 7fd2565d3640 1 -- 172.21.15.89:0/825643270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd250101de0 msgr2=0x7fd2501021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:27.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.393+0000 7fd2565d3640 1 --2- 172.21.15.89:0/825643270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd250101de0 0x7fd2501021c0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fd240009900 tx=0x7fd24002a410 comp rx=0 tx=0).stop 2024-09-15T23:36:27.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.393+0000 7fd2565d3640 1 -- 172.21.15.89:0/825643270 shutdown_connections 2024-09-15T23:36:27.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.393+0000 7fd2565d3640 1 --2- 172.21.15.89:0/825643270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd250106f10 0x7fd25010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.393+0000 7fd2565d3640 1 --2- 172.21.15.89:0/825643270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd250102790 0x7fd2501067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.393+0000 7fd2565d3640 1 --2- 172.21.15.89:0/825643270 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd250101de0 0x7fd2501021c0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.393+0000 7fd2565d3640 1 -- 172.21.15.89:0/825643270 >> 172.21.15.89:0/825643270 conn(0x7fd2500fd8c0 msgr2=0x7fd2500ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:27.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.393+0000 7fd2565d3640 1 -- 172.21.15.89:0/825643270 shutdown_connections 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.393+0000 7fd2565d3640 1 -- 172.21.15.89:0/825643270 wait complete. 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd2565d3640 1 Processor -- start 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd2565d3640 1 -- start start 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd2565d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd250101de0 0x7fd250198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd2565d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd250102790 0x7fd2501989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd2565d3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd250106f10 0x7fd25019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd2565d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd250111f90 con 0x7fd250101de0 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd2565d3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd250111e10 con 0x7fd250102790 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd2565d3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd250112110 con 0x7fd250106f10 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd24ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd250101de0 0x7fd250198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd24ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd250101de0 0x7fd250198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47722/0 (socket says 172.21.15.89:47722) 2024-09-15T23:36:27.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.394+0000 7fd24ffff640 1 -- 172.21.15.89:0/3677417531 learned_addr learned my addr 172.21.15.89:0/3677417531 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd247fff640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd250102790 0x7fd2501989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd24ffff640 1 -- 172.21.15.89:0/3677417531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd250106f10 msgr2=0x7fd25019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd24ffff640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd250106f10 0x7fd25019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd24ffff640 1 -- 172.21.15.89:0/3677417531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd250102790 msgr2=0x7fd2501989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd24ffff640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd250102790 0x7fd2501989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd24ffff640 1 -- 172.21.15.89:0/3677417531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25019d440 con 0x7fd250101de0 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd24ffff640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd250101de0 0x7fd250198490 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fd240005ab0 tx=0x7fd2400042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd24dffb640 1 -- 172.21.15.89:0/3677417531 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24003f030 con 0x7fd250101de0 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd24dffb640 1 -- 172.21.15.89:0/3677417531 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd240003ce0 con 0x7fd250101de0 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd2565d3640 1 -- 172.21.15.89:0/3677417531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25019d6d0 con 0x7fd250101de0 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd24dffb640 1 -- 172.21.15.89:0/3677417531 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240035070 con 0x7fd250101de0 2024-09-15T23:36:27.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.395+0000 7fd2565d3640 1 -- 172.21.15.89:0/3677417531 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2501a5010 con 0x7fd250101de0 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.397+0000 7fd24dffb640 1 -- 172.21.15.89:0/3677417531 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd240042030 con 0x7fd250101de0 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.397+0000 7fd2589ac740 1 Processor -- start 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.397+0000 7fd2589ac740 1 -- start start 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7cc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 0x55de3f7cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7d9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd2589ac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55de3f7322f0 con 0x55de3f7d7370 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd2589ac740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55de3f6d9cf0 con 0x55de3f7cdb40 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd2589ac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55de3f72e2b0 con 0x55de3f7cc5d0 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd247fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7d9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd24dffb640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd23c0a32c0 0x7fd23c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd247fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7d9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47726/0 (socket says 172.21.15.89:47726) 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd247fff640 1 -- 172.21.15.89:0/2699053716 learned_addr learned my addr 172.21.15.89:0/2699053716 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd24ffff640 1 --2- 172.21.15.89:0/2699053716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 0x55de3f7cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd254b70640 1 --2- 172.21.15.89:0/2699053716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7cc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd254b70640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd23c0a32c0 0x7fd23c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd24dffb640 1 -- 172.21.15.89:0/3677417531 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd2400f0630 con 0x7fd250101de0 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd247fff640 1 -- 172.21.15.89:0/2699053716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 msgr2=0x55de3f7cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd247fff640 1 --2- 172.21.15.89:0/2699053716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 0x55de3f7cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd247fff640 1 -- 172.21.15.89:0/2699053716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 msgr2=0x55de3f7cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd247fff640 1 --2- 172.21.15.89:0/2699053716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7cc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd247fff640 1 -- 172.21.15.89:0/2699053716 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de3f6be3a0 con 0x55de3f7d7370 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.398+0000 7fd247fff640 1 --2- 172.21.15.89:0/2699053716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7d9760 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fd23800cc20 tx=0x7fd23800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af25f8de8f568caa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:27.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd254b70640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd23c0a32c0 0x7fd23c0a5780 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd248008fd0 tx=0x7fd248006ce0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2467fc640 1 -- 172.21.15.89:0/2699053716 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2380139d0 con 0x55de3f7d7370 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2467fc640 1 -- 172.21.15.89:0/2699053716 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd238013b70 con 0x55de3f7d7370 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2467fc640 1 -- 172.21.15.89:0/2699053716 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23801c380 con 0x55de3f7d7370 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2589ac740 1 -- 172.21.15.89:0/2699053716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 msgr2=0x55de3f7d9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2589ac740 1 --2- 172.21.15.89:0/2699053716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7d9760 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fd23800cc20 tx=0x7fd23800e720 comp rx=0 tx=0).stop 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2589ac740 1 -- 172.21.15.89:0/2699053716 shutdown_connections 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2589ac740 1 --2- 172.21.15.89:0/2699053716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7d9760 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2589ac740 1 --2- 172.21.15.89:0/2699053716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 0x55de3f7cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2589ac740 1 --2- 172.21.15.89:0/2699053716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7cc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2589ac740 1 -- 172.21.15.89:0/2699053716 >> 172.21.15.89:0/2699053716 conn(0x55de3f795310 msgr2=0x55de3f7c8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2589ac740 1 -- 172.21.15.89:0/2699053716 shutdown_connections 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.399+0000 7fd2589ac740 1 -- 172.21.15.89:0/2699053716 wait complete. 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.400+0000 7fd2589ac740 1 -- start start 2024-09-15T23:36:27.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.400+0000 7fd2589ac740 1 Processor -- start 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.400+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7d6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.400+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cdb40 0x55de3f7d4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.400+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7d7370 0x55de3f7d4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.400+0000 7fd2589ac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55de3f7da070 con 0x55de3f7cc5d0 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.400+0000 7fd2589ac740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55de3f6ed060 con 0x55de3f7d7370 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.400+0000 7fd2589ac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55de3f7da1f0 con 0x55de3f7cdb40 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.400+0000 7fd2589ac740 10 client.0 _ll_register_callbacks cb 0x55de3f4408c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.400+0000 7fd24ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7d6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd24ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7d6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47742/0 (socket says 172.21.15.89:47742) 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd24ffff640 1 -- 172.21.15.89:0/1511326266 learned_addr learned my addr 172.21.15.89:0/1511326266 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd254b70640 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7d7370 0x55de3f7d4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd24ffff640 1 -- 172.21.15.89:0/1511326266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cdb40 msgr2=0x55de3f7d4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd24ffff640 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cdb40 0x55de3f7d4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd24ffff640 1 -- 172.21.15.89:0/1511326266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7d7370 msgr2=0x55de3f7d4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd24ffff640 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7d7370 0x55de3f7d4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd24ffff640 1 -- 172.21.15.89:0/1511326266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de3f4a33c0 con 0x55de3f7cc5d0 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd24ffff640 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7d6d80 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fd240006010 tx=0x7fd24002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a522b56a7ee0a6dd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd22f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd22f7fe640 1 -- 172.21.15.89:0/1511326266 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240049070 con 0x55de3f7cc5d0 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd22f7fe640 1 -- 172.21.15.89:0/1511326266 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd24003bd70 con 0x55de3f7cc5d0 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd2589ac740 1 -- 172.21.15.89:0/1511326266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd238013b70 con 0x55de3f7cc5d0 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd22f7fe640 1 -- 172.21.15.89:0/1511326266 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd240035210 con 0x55de3f7cc5d0 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd2589ac740 10 client.19683 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd2589ac740 1 -- 172.21.15.89:0/1511326266 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55de3f7e5b40 con 0x55de3f7cc5d0 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.401+0000 7fd22f7fe640 1 -- 172.21.15.89:0/1511326266 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd238013b70 con 0x55de3f7cc5d0 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.402+0000 7fd22f7fe640 1 -- 172.21.15.89:0/1511326266 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd2400b8ed0 con 0x55de3f7cc5d0 2024-09-15T23:36:27.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.402+0000 7fd2589ac740 10 client.19683 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.402+0000 7fd2589ac740 10 client.19683 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.402+0000 7fd2589ac740 20 client.19683 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.402+0000 7fd2589ac740 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55de3f7e6fa0 0x55de3f7e9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.402+0000 7fd2589ac740 4 client.19683 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.402+0000 7fd2589ac740 1 -- 172.21.15.89:0/1511326266 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de3f73f860 con 0x55de3f7e6fa0 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.402+0000 7fd247fff640 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55de3f7e6fa0 0x55de3f7e9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.402+0000 7fd247fff640 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55de3f7e6fa0 0x55de3f7e9410 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=18a975765817f2b4 server_cookie=25e9131a3a267db3 in_seq=0 out_seq=0 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.402+0000 7fd22f7fe640 10 client.19683 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd22f7fe640 1 -- 172.21.15.89:0/1511326266 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55de3f73f860 con 0x55de3f7e6fa0 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd22f7fe640 10 client.19683 handle_command_reply: tid=0 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd2589ac740 1 client.19683 shutdown 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd2589ac740 1 -- 172.21.15.89:0/1511326266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 msgr2=0x55de3f7d6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd2589ac740 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7d6d80 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fd240006010 tx=0x7fd24002a410 comp rx=0 tx=0).stop 2024-09-15T23:36:27.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd2589ac740 1 -- 172.21.15.89:0/1511326266 shutdown_connections 2024-09-15T23:36:27.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd2589ac740 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55de3f7e6fa0 0x55de3f7e9410 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd2589ac740 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7d7370 0x55de3f7d4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd2589ac740 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cdb40 0x55de3f7d4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd2589ac740 1 --2- 172.21.15.89:0/1511326266 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7d6d80 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd22f7fe640 0 client.19683 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:36:27.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.403+0000 7fd2589ac740 1 -- 172.21.15.89:0/1511326266 >> 172.21.15.89:0/1511326266 conn(0x55de3f795310 msgr2=0x55de3f7ce960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:27.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.404+0000 7fd2589ac740 1 -- 172.21.15.89:0/1511326266 shutdown_connections 2024-09-15T23:36:27.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.404+0000 7fd2589ac740 1 -- 172.21.15.89:0/1511326266 wait complete. 2024-09-15T23:36:27.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.404+0000 7fd2589ac740 20 client.19683 trim_cache size 0 max 16384 2024-09-15T23:36:27.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.416+0000 7fd2589ac740 1 Processor -- start 2024-09-15T23:36:27.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd2589ac740 1 -- start start 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7cf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7d7370 0x55de3f7eed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd2589ac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd2501acd60 con 0x55de3f7cc5d0 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd2589ac740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd2501ae8a0 con 0x55de3f7cdb40 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd2589ac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd2501ad430 con 0x55de3f7d7370 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd24ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd24ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7cf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47744/0 (socket says 172.21.15.89:47744) 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd24ffff640 1 -- 172.21.15.89:0/139056322 learned_addr learned my addr 172.21.15.89:0/139056322 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd247fff640 1 --2- 172.21.15.89:0/139056322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd254b70640 1 --2- 172.21.15.89:0/139056322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7d7370 0x55de3f7eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd24ffff640 1 -- 172.21.15.89:0/139056322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7d7370 msgr2=0x55de3f7eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd24ffff640 1 --2- 172.21.15.89:0/139056322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7d7370 0x55de3f7eed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd24ffff640 1 -- 172.21.15.89:0/139056322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 msgr2=0x55de3f7cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd24ffff640 1 --2- 172.21.15.89:0/139056322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd24ffff640 1 -- 172.21.15.89:0/139056322 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de3f7d4fd0 con 0x55de3f7cc5d0 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd254b70640 1 --2- 172.21.15.89:0/139056322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7d7370 0x55de3f7eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd24ffff640 1 --2- 172.21.15.89:0/139056322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7cf120 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd2400edd90 tx=0x7fd2400f0930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ea2aea27a8e018 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.417+0000 7fd245ffb640 1 -- 172.21.15.89:0/139056322 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240049070 con 0x55de3f7cc5d0 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd245ffb640 1 -- 172.21.15.89:0/139056322 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2400da070 con 0x55de3f7cc5d0 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd245ffb640 1 -- 172.21.15.89:0/139056322 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2400c0c10 con 0x55de3f7cc5d0 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd2589ac740 1 -- 172.21.15.89:0/139056322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 msgr2=0x55de3f7cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd2589ac740 1 --2- 172.21.15.89:0/139056322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7cf120 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd2400edd90 tx=0x7fd2400f0930 comp rx=0 tx=0).stop 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd2589ac740 1 -- 172.21.15.89:0/139056322 shutdown_connections 2024-09-15T23:36:27.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd2589ac740 1 --2- 172.21.15.89:0/139056322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7d7370 0x55de3f7eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd2589ac740 1 --2- 172.21.15.89:0/139056322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd2589ac740 1 --2- 172.21.15.89:0/139056322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7cc5d0 0x55de3f7cf120 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd2589ac740 1 -- 172.21.15.89:0/139056322 >> 172.21.15.89:0/139056322 conn(0x55de3f795310 msgr2=0x55de3f7c7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd2589ac740 1 -- 172.21.15.89:0/139056322 shutdown_connections 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.418+0000 7fd2589ac740 1 -- 172.21.15.89:0/139056322 wait complete. 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd2589ac740 1 -- start start 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd2589ac740 1 Processor -- start 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 0x55de3f7df480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7df9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd2589ac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7dc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd2589ac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd2501ae1d0 con 0x55de3f7d7370 2024-09-15T23:36:27.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd2589ac740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd2501adb00 con 0x55de3f7cdb40 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd2589ac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd2501aad40 con 0x55de3f7cc5d0 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd2589ac740 10 client.0 _ll_register_callbacks cb 0x7fd2400b8440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd254b70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7dc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd254b70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7dc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47752/0 (socket says 172.21.15.89:47752) 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd254b70640 1 -- 172.21.15.89:0/2271813774 learned_addr learned my addr 172.21.15.89:0/2271813774 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd247fff640 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7df9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd24ffff640 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 0x55de3f7df480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd254b70640 1 -- 172.21.15.89:0/2271813774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 msgr2=0x55de3f7df480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd254b70640 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 0x55de3f7df480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd254b70640 1 -- 172.21.15.89:0/2271813774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 msgr2=0x55de3f7df9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:27.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.419+0000 7fd254b70640 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7df9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd254b70640 1 -- 172.21.15.89:0/2271813774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de3f7dc9c0 con 0x55de3f7d7370 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd24ffff640 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 0x55de3f7df480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd254b70640 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7dc450 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fd2480042f0 tx=0x7fd248004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262a0f198de31451 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd22f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd22f7fe640 1 -- 172.21.15.89:0/2271813774 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd248014810 con 0x55de3f7d7370 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd22f7fe640 1 -- 172.21.15.89:0/2271813774 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd248016070 con 0x55de3f7d7370 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd2589ac740 1 -- 172.21.15.89:0/2271813774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2400da070 con 0x55de3f7d7370 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd22f7fe640 1 -- 172.21.15.89:0/2271813774 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd2480149b0 con 0x55de3f7d7370 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd2589ac740 10 client.19689 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd2589ac740 1 -- 172.21.15.89:0/2271813774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55de3f852540 con 0x55de3f7d7370 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd22f7fe640 1 -- 172.21.15.89:0/2271813774 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2400da070 con 0x55de3f7d7370 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd22f7fe640 1 -- 172.21.15.89:0/2271813774 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd248006690 con 0x55de3f7d7370 2024-09-15T23:36:27.433 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd2589ac740 10 client.19689 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd2589ac740 10 client.19689 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd2589ac740 20 client.19689 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd2589ac740 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55de3f852d40 0x55de3f855160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd2589ac740 4 client.19689 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.420+0000 7fd2589ac740 1 -- 172.21.15.89:0/2271813774 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55de3f739d70 con 0x55de3f852d40 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.421+0000 7fd24ffff640 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55de3f852d40 0x55de3f855160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.421+0000 7fd22f7fe640 10 client.19689 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.421+0000 7fd24ffff640 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55de3f852d40 0x55de3f855160 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f31a4ecc3bf1381d server_cookie=e0598a9b2bd31a11 in_seq=0 out_seq=0 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.421+0000 7fd22f7fe640 1 -- 172.21.15.89:0/2271813774 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55de3f739d70 con 0x55de3f852d40 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.421+0000 7fd22f7fe640 10 client.19689 handle_command_reply: tid=0 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.421+0000 7fd2589ac740 1 client.19689 shutdown 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 1 -- 172.21.15.89:0/2271813774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 msgr2=0x55de3f7dc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:27.434 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7dc450 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fd2480042f0 tx=0x7fd248004b50 comp rx=0 tx=0).stop 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 1 -- 172.21.15.89:0/2271813774 shutdown_connections 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55de3f852d40 0x55de3f855160 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55de3f7d7370 0x55de3f7dc450 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55de3f7cdb40 0x55de3f7df9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd22f7fe640 0 client.19689 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 1 --2- 172.21.15.89:0/2271813774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55de3f7cc5d0 0x55de3f7df480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 1 -- 172.21.15.89:0/2271813774 >> 172.21.15.89:0/2271813774 conn(0x55de3f795310 msgr2=0x55de3f7d5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 1 -- 172.21.15.89:0/2271813774 shutdown_connections 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 1 -- 172.21.15.89:0/2271813774 wait complete. 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2589ac740 20 client.19689 trim_cache size 0 max 16384 2024-09-15T23:36:27.435 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:36:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 -- 172.21.15.89:0/3677417531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd23c0a32c0 msgr2=0x7fd23c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd23c0a32c0 0x7fd23c0a5780 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd248008fd0 tx=0x7fd248006ce0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 -- 172.21.15.89:0/3677417531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd250101de0 msgr2=0x7fd250198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd250101de0 0x7fd250198490 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fd240005ab0 tx=0x7fd2400042f0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 -- 172.21.15.89:0/3677417531 shutdown_connections 2024-09-15T23:36:27.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd23c0a32c0 0x7fd23c0a5780 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd250106f10 0x7fd25019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd250102790 0x7fd2501989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 --2- 172.21.15.89:0/3677417531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd250101de0 0x7fd250198490 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 -- 172.21.15.89:0/3677417531 >> 172.21.15.89:0/3677417531 conn(0x7fd2500fd8c0 msgr2=0x7fd2500fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 -- 172.21.15.89:0/3677417531 shutdown_connections 2024-09-15T23:36:27.437 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:27.422+0000 7fd2565d3640 1 -- 172.21.15.89:0/3677417531 wait complete. 2024-09-15T23:36:28.168 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:36:28.415 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:28.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:28 smithi089 ceph-mon[30933]: pgmap v723: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:36:28.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.937+0000 7f58a22b5640 1 -- 172.21.15.89:0/1370583637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c100a80 msgr2=0x7f589c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:28.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.937+0000 7f58a22b5640 1 --2- 172.21.15.89:0/1370583637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c100a80 0x7f589c100ee0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f5890009930 tx=0x7f589002a420 comp rx=0 tx=0).stop 2024-09-15T23:36:28.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.937+0000 7f58a22b5640 1 -- 172.21.15.89:0/1370583637 shutdown_connections 2024-09-15T23:36:28.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.937+0000 7f58a22b5640 1 --2- 172.21.15.89:0/1370583637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f589c101420 0x7f589c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.937+0000 7f58a22b5640 1 --2- 172.21.15.89:0/1370583637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c100a80 0x7f589c100ee0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.937+0000 7f58a22b5640 1 --2- 172.21.15.89:0/1370583637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f589c106a80 0x7f589c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.937+0000 7f58a22b5640 1 -- 172.21.15.89:0/1370583637 >> 172.21.15.89:0/1370583637 conn(0x7f589c0fc7a0 msgr2=0x7f589c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:28.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.937+0000 7f58a22b5640 1 -- 172.21.15.89:0/1370583637 shutdown_connections 2024-09-15T23:36:28.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.937+0000 7f58a22b5640 1 -- 172.21.15.89:0/1370583637 wait complete. 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a22b5640 1 Processor -- start 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a22b5640 1 -- start start 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a22b5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f589c100a80 0x7f589c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a22b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f589c101420 0x7f589c198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a22b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c106a80 0x7f589c19cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a22b5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f589c110e80 con 0x7f589c106a80 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a22b5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f589c110d00 con 0x7f589c100a80 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a22b5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f589c111000 con 0x7f589c101420 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a0852640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c106a80 0x7f589c19cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a0852640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c106a80 0x7f589c19cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47780/0 (socket says 172.21.15.89:47780) 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a0852640 1 -- 172.21.15.89:0/2880609245 learned_addr learned my addr 172.21.15.89:0/2880609245 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:28.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f589bfff640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f589c100a80 0x7f589c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f589b7fe640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f589c101420 0x7f589c198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a0852640 1 -- 172.21.15.89:0/2880609245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f589c101420 msgr2=0x7f589c198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a0852640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f589c101420 0x7f589c198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a0852640 1 -- 172.21.15.89:0/2880609245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f589c100a80 msgr2=0x7f589c198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.938+0000 7f58a0852640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f589c100a80 0x7f589c198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.939+0000 7f58a0852640 1 -- 172.21.15.89:0/2880609245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589c19d2f0 con 0x7f589c106a80 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.939+0000 7f58a0852640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c106a80 0x7f589c19cc10 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f589400cce0 tx=0x7f589400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.939+0000 7f58997fa640 1 -- 172.21.15.89:0/2880609245 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589400f450 con 0x7f589c106a80 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.939+0000 7f58a22b5640 1 -- 172.21.15.89:0/2880609245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f589c19d5e0 con 0x7f589c106a80 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.939+0000 7f58997fa640 1 -- 172.21.15.89:0/2880609245 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f589400fad0 con 0x7f589c106a80 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.939+0000 7f58997fa640 1 -- 172.21.15.89:0/2880609245 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5894018b60 con 0x7f589c106a80 2024-09-15T23:36:28.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.939+0000 7f58a22b5640 1 -- 172.21.15.89:0/2880609245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f589c1a4ec0 con 0x7f589c106a80 2024-09-15T23:36:28.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.941+0000 7f58997fa640 1 -- 172.21.15.89:0/2880609245 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f589400fc80 con 0x7f589c106a80 2024-09-15T23:36:28.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f58997fa640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58840a3110 0x7f58840a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f58a468e740 1 Processor -- start 2024-09-15T23:36:28.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f58997fa640 1 -- 172.21.15.89:0/2880609245 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f58940d01b0 con 0x7f589c106a80 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f589bfff640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58840a3110 0x7f58840a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f58a468e740 1 -- start start 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f589bfff640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58840a3110 0x7f58840a55d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f588c00ab60 tx=0x7f588c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f58a468e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 0x55c037e72090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f58a468e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e725d0 0x55c037e72a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f58a468e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e83b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f58a468e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c037dd82f0 con 0x55c037e81780 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f58a468e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c037d7fcf0 con 0x55c037e73b40 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.942+0000 7f58a468e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c037dd42b0 con 0x55c037e725d0 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58a0852640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e83b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58a0852640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e83b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47788/0 (socket says 172.21.15.89:47788) 2024-09-15T23:36:28.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58a0852640 1 -- 172.21.15.89:0/1368812344 learned_addr learned my addr 172.21.15.89:0/1368812344 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f589bfff640 1 --2- 172.21.15.89:0/1368812344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 0x55c037e72090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f589b7fe640 1 --2- 172.21.15.89:0/1368812344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e725d0 0x55c037e72a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58a0852640 1 -- 172.21.15.89:0/1368812344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e725d0 msgr2=0x55c037e72a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58a0852640 1 --2- 172.21.15.89:0/1368812344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e725d0 0x55c037e72a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58a0852640 1 -- 172.21.15.89:0/1368812344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 msgr2=0x55c037e72090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58a0852640 1 --2- 172.21.15.89:0/1368812344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 0x55c037e72090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58a0852640 1 -- 172.21.15.89:0/1368812344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c037d643a0 con 0x55c037e81780 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58a0852640 1 --2- 172.21.15.89:0/1368812344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e83b70 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f5894098840 tx=0x7f589409cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8932b6871c12304c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58827fc640 1 -- 172.21.15.89:0/1368812344 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58940a5050 con 0x55c037e81780 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58827fc640 1 -- 172.21.15.89:0/1368812344 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f589409a6e0 con 0x55c037e81780 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.943+0000 7f58827fc640 1 -- 172.21.15.89:0/1368812344 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5894016dd0 con 0x55c037e81780 2024-09-15T23:36:28.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.944+0000 7f58a468e740 1 -- 172.21.15.89:0/1368812344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 msgr2=0x55c037e83b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.944+0000 7f58a468e740 1 --2- 172.21.15.89:0/1368812344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e83b70 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f5894098840 tx=0x7f589409cae0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.944+0000 7f58a468e740 1 -- 172.21.15.89:0/1368812344 shutdown_connections 2024-09-15T23:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.944+0000 7f58a468e740 1 --2- 172.21.15.89:0/1368812344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e83b70 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.944+0000 7f58a468e740 1 --2- 172.21.15.89:0/1368812344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e725d0 0x55c037e72a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.944+0000 7f58a468e740 1 --2- 172.21.15.89:0/1368812344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 0x55c037e72090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.944+0000 7f58a468e740 1 -- 172.21.15.89:0/1368812344 >> 172.21.15.89:0/1368812344 conn(0x55c037e3b310 msgr2=0x55c037e6ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.944+0000 7f58a468e740 1 -- 172.21.15.89:0/1368812344 shutdown_connections 2024-09-15T23:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.944+0000 7f58a468e740 1 -- 172.21.15.89:0/1368812344 wait complete. 2024-09-15T23:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.944+0000 7f58a468e740 1 -- start start 2024-09-15T23:36:28.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a468e740 1 Processor -- start 2024-09-15T23:36:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a468e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e725d0 0x55c037e811e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a468e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 0x55c037e7e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a468e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e7ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a468e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c037e84480 con 0x55c037e81780 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a468e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c037d93060 con 0x55c037e725d0 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a468e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c037e84600 con 0x55c037e73b40 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a468e740 10 client.0 _ll_register_callbacks cb 0x55c037ae68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a0852640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e7ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f589b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 0x55c037e7e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a0852640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e7ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47796/0 (socket says 172.21.15.89:47796) 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a0852640 1 -- 172.21.15.89:0/885215866 learned_addr learned my addr 172.21.15.89:0/885215866 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f589b7fe640 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 0x55c037e7e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:41338/0 (socket says 172.21.15.89:41338) 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f589bfff640 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e725d0 0x55c037e811e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a0852640 1 -- 172.21.15.89:0/885215866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 msgr2=0x55c037e7e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:28.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a0852640 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 0x55c037e7e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a0852640 1 -- 172.21.15.89:0/885215866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e725d0 msgr2=0x55c037e811e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a0852640 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e725d0 0x55c037e811e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a0852640 1 -- 172.21.15.89:0/885215866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c037b493c0 con 0x55c037e81780 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.945+0000 7f58a0852640 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e7ec20 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f589400ccb0 tx=0x7f58940987b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19204ef6cbe6d01 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f587b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f587b7fe640 1 -- 172.21.15.89:0/885215866 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58940ba070 con 0x55c037e81780 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f58a468e740 1 -- 172.21.15.89:0/885215866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f589409a6e0 con 0x55c037e81780 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f587b7fe640 1 -- 172.21.15.89:0/885215866 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5894016b70 con 0x55c037e81780 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f587b7fe640 1 -- 172.21.15.89:0/885215866 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5894016d90 con 0x55c037e81780 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f58a468e740 10 client.19701 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f58a468e740 1 -- 172.21.15.89:0/885215866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c037e8ffa0 con 0x55c037e81780 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f587b7fe640 1 -- 172.21.15.89:0/885215866 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f589409a6e0 con 0x55c037e81780 2024-09-15T23:36:28.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f587b7fe640 1 -- 172.21.15.89:0/885215866 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f58940a7030 con 0x55c037e81780 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f58a468e740 10 client.19701 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f58a468e740 10 client.19701 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f58a468e740 20 client.19701 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f58a468e740 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c037e912f0 0x55c037e936e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f58a468e740 4 client.19701 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.946+0000 7f58a468e740 1 -- 172.21.15.89:0/885215866 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c037de5860 con 0x55c037e912f0 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.947+0000 7f589bfff640 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c037e912f0 0x55c037e936e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.947+0000 7f587b7fe640 10 client.19701 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.947+0000 7f589bfff640 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c037e912f0 0x55c037e936e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5430ad0fbea29395 server_cookie=63974addb50caaed in_seq=0 out_seq=0 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f587b7fe640 1 -- 172.21.15.89:0/885215866 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c037de5860 con 0x55c037e912f0 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f587b7fe640 10 client.19701 handle_command_reply: tid=0 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 client.19701 shutdown 2024-09-15T23:36:28.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 -- 172.21.15.89:0/885215866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 msgr2=0x55c037e7ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e7ec20 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f589400ccb0 tx=0x7f58940987b0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 -- 172.21.15.89:0/885215866 shutdown_connections 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c037e912f0 0x55c037e936e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e7ec20 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 0x55c037e7e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 --2- 172.21.15.89:0/885215866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e725d0 0x55c037e811e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 -- 172.21.15.89:0/885215866 >> 172.21.15.89:0/885215866 conn(0x55c037e3b310 msgr2=0x55c037e74990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f587b7fe640 0 client.19701 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 -- 172.21.15.89:0/885215866 shutdown_connections 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 1 -- 172.21.15.89:0/885215866 wait complete. 2024-09-15T23:36:28.957 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.948+0000 7f58a468e740 20 client.19701 trim_cache size 0 max 16384 2024-09-15T23:36:28.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a468e740 1 Processor -- start 2024-09-15T23:36:28.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a468e740 1 -- start start 2024-09-15T23:36:28.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a468e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e725d0 0x55c037e7dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a468e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 0x55c037e7d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a468e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e96700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a468e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f589c1ac8f0 con 0x55c037e81780 2024-09-15T23:36:28.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a468e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f589c1ae430 con 0x55c037e725d0 2024-09-15T23:36:28.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a468e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f589c1acfc0 con 0x55c037e73b40 2024-09-15T23:36:28.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a0852640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e96700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a0852640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e96700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47804/0 (socket says 172.21.15.89:47804) 2024-09-15T23:36:28.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a0852640 1 -- 172.21.15.89:0/230754480 learned_addr learned my addr 172.21.15.89:0/230754480 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:28.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f589bfff640 1 --2- 172.21.15.89:0/230754480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e725d0 0x55c037e7dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f589b7fe640 1 --2- 172.21.15.89:0/230754480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 0x55c037e7d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a0852640 1 -- 172.21.15.89:0/230754480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 msgr2=0x55c037e7d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a0852640 1 --2- 172.21.15.89:0/230754480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 0x55c037e7d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a0852640 1 -- 172.21.15.89:0/230754480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e725d0 msgr2=0x55c037e7dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a0852640 1 --2- 172.21.15.89:0/230754480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e725d0 0x55c037e7dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.961+0000 7f58a0852640 1 -- 172.21.15.89:0/230754480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c037e6f960 con 0x55c037e81780 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f58a0852640 1 --2- 172.21.15.89:0/230754480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e96700 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f5894098840 tx=0x7f589409cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275d63edab2216d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f5881ffb640 1 -- 172.21.15.89:0/230754480 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58940ba070 con 0x55c037e81780 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f5881ffb640 1 -- 172.21.15.89:0/230754480 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f589409c3e0 con 0x55c037e81780 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f5881ffb640 1 -- 172.21.15.89:0/230754480 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58940a4e40 con 0x55c037e81780 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f58a468e740 1 -- 172.21.15.89:0/230754480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 msgr2=0x55c037e96700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:28.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f58a468e740 1 --2- 172.21.15.89:0/230754480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e96700 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f5894098840 tx=0x7f589409cae0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f58a468e740 1 -- 172.21.15.89:0/230754480 shutdown_connections 2024-09-15T23:36:28.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f58a468e740 1 --2- 172.21.15.89:0/230754480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e81780 0x55c037e96700 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f58a468e740 1 --2- 172.21.15.89:0/230754480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e73b40 0x55c037e7d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f58a468e740 1 --2- 172.21.15.89:0/230754480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e725d0 0x55c037e7dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f58a468e740 1 -- 172.21.15.89:0/230754480 >> 172.21.15.89:0/230754480 conn(0x55c037e3b310 msgr2=0x55c037e7cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:28.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f58a468e740 1 -- 172.21.15.89:0/230754480 shutdown_connections 2024-09-15T23:36:28.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.962+0000 7f58a468e740 1 -- 172.21.15.89:0/230754480 wait complete. 2024-09-15T23:36:28.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f58a468e740 1 -- start start 2024-09-15T23:36:28.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f58a468e740 1 Processor -- start 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f58a468e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e725d0 0x55c037e96e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f58a468e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 0x55c037e941d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f58a468e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e81780 0x55c037e94710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f58a468e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f589c1add60 con 0x55c037e725d0 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f58a468e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f589c1ad690 con 0x55c037e73b40 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f58a468e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f589c1aa8d0 con 0x55c037e81780 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f58a468e740 10 client.0 _ll_register_callbacks cb 0x7f588c00a7b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f589bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e725d0 0x55c037e96e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f589bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e725d0 0x55c037e96e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47814/0 (socket says 172.21.15.89:47814) 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.963+0000 7f589bfff640 1 -- 172.21.15.89:0/3071590407 learned_addr learned my addr 172.21.15.89:0/3071590407 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f589b7fe640 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 0x55c037e941d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f589bfff640 1 -- 172.21.15.89:0/3071590407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e81780 msgr2=0x55c037e94710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:28.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f58a0852640 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e81780 0x55c037e94710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f589bfff640 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e81780 0x55c037e94710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f589bfff640 1 -- 172.21.15.89:0/3071590407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 msgr2=0x55c037e941d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f589bfff640 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 0x55c037e941d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f589bfff640 1 -- 172.21.15.89:0/3071590407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c037e94f10 con 0x55c037e725d0 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f589b7fe640 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 0x55c037e941d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f58a0852640 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e81780 0x55c037e94710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f589bfff640 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e725d0 0x55c037e96e30 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x55c037e845a0 tx=0x7f588c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9243b215ebb51def server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f587b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f587b7fe640 1 -- 172.21.15.89:0/3071590407 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c016070 con 0x55c037e725d0 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f587b7fe640 1 -- 172.21.15.89:0/3071590407 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f588c01c830 con 0x55c037e725d0 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f58a468e740 1 -- 172.21.15.89:0/3071590407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c037ddfd70 con 0x55c037e725d0 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f587b7fe640 1 -- 172.21.15.89:0/3071590407 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f588c03e050 con 0x55c037e725d0 2024-09-15T23:36:28.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f58a468e740 10 client.19707 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f58a468e740 1 -- 172.21.15.89:0/3071590407 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c037efbc00 con 0x55c037e725d0 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.964+0000 7f587b7fe640 1 -- 172.21.15.89:0/3071590407 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c037ddfd70 con 0x55c037e725d0 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f587b7fe640 1 -- 172.21.15.89:0/3071590407 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f588c030c50 con 0x55c037e725d0 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f58a468e740 10 client.19707 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f58a468e740 10 client.19707 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f58a468e740 20 client.19707 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f58a468e740 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c037efc3a0 0x55c037efe7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f58a468e740 4 client.19707 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f58a468e740 1 -- 172.21.15.89:0/3071590407 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c037d5c170 con 0x55c037efc3a0 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f589b7fe640 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c037efc3a0 0x55c037efe7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f589b7fe640 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c037efc3a0 0x55c037efe7c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1aa8c4541141e4e server_cookie=e2ed58ca8d0f04f1 in_seq=0 out_seq=0 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f587b7fe640 10 client.19707 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:36:28.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f587b7fe640 1 -- 172.21.15.89:0/3071590407 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c037d5c170 con 0x55c037efc3a0 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f587b7fe640 10 client.19707 handle_command_reply: tid=0 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.965+0000 7f58a468e740 1 client.19707 shutdown 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 1 -- 172.21.15.89:0/3071590407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e725d0 msgr2=0x55c037e96e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e725d0 0x55c037e96e30 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x55c037e845a0 tx=0x7f588c004730 comp rx=0 tx=0).stop 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 1 -- 172.21.15.89:0/3071590407 shutdown_connections 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55c037efc3a0 0x55c037efe7c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c037e81780 0x55c037e94710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c037e73b40 0x55c037e941d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f587b7fe640 0 client.19707 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 1 --2- 172.21.15.89:0/3071590407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c037e725d0 0x55c037e96e30 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 1 -- 172.21.15.89:0/3071590407 >> 172.21.15.89:0/3071590407 conn(0x55c037e3b310 msgr2=0x55c037e74060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 1 -- 172.21.15.89:0/3071590407 shutdown_connections 2024-09-15T23:36:28.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 1 -- 172.21.15.89:0/3071590407 wait complete. 2024-09-15T23:36:28.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a468e740 20 client.19707 trim_cache size 0 max 16384 2024-09-15T23:36:28.975 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:36:28.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a22b5640 1 -- 172.21.15.89:0/2880609245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58840a3110 msgr2=0x7f58840a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:28.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a22b5640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58840a3110 0x7f58840a55d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f588c00ab60 tx=0x7f588c00a2d0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a22b5640 1 -- 172.21.15.89:0/2880609245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c106a80 msgr2=0x7f589c19cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:28.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.966+0000 7f58a22b5640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c106a80 0x7f589c19cc10 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f589400cce0 tx=0x7f589400a620 comp rx=0 tx=0).stop 2024-09-15T23:36:28.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.967+0000 7f58a22b5640 1 -- 172.21.15.89:0/2880609245 shutdown_connections 2024-09-15T23:36:28.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.967+0000 7f58a22b5640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58840a3110 0x7f58840a55d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.967+0000 7f58a22b5640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f589c106a80 0x7f589c19cc10 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.967+0000 7f58a22b5640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f589c101420 0x7f589c198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.967+0000 7f58a22b5640 1 --2- 172.21.15.89:0/2880609245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f589c100a80 0x7f589c198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:28.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.967+0000 7f58a22b5640 1 -- 172.21.15.89:0/2880609245 >> 172.21.15.89:0/2880609245 conn(0x7f589c0fc7a0 msgr2=0x7f589c104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:28.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.967+0000 7f58a22b5640 1 -- 172.21.15.89:0/2880609245 shutdown_connections 2024-09-15T23:36:28.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:28.967+0000 7f58a22b5640 1 -- 172.21.15.89:0/2880609245 wait complete. 2024-09-15T23:36:28.977 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:28 smithi120 ceph-mon[35250]: pgmap v723: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:36:28.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:28 smithi174 ceph-mon[35586]: pgmap v723: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:36:29.844 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:36:30.088 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:30.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f0096d52640 1 -- 172.21.15.89:0/1223302439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0090102d30 msgr2=0x7f00901031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f0096d52640 1 --2- 172.21.15.89:0/1223302439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0090102d30 0x7f00901031b0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f007c009900 tx=0x7f007c02a410 comp rx=0 tx=0).stop 2024-09-15T23:36:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f0096d52640 1 -- 172.21.15.89:0/1223302439 shutdown_connections 2024-09-15T23:36:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f0096d52640 1 --2- 172.21.15.89:0/1223302439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00901036f0 0x7f0090109f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f0096d52640 1 --2- 172.21.15.89:0/1223302439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0090102d30 0x7f00901031b0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f0096d52640 1 --2- 172.21.15.89:0/1223302439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0090101b40 0x7f0090101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f0096d52640 1 -- 172.21.15.89:0/1223302439 >> 172.21.15.89:0/1223302439 conn(0x7f00900fd2e0 msgr2=0x7f00900ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f0096d52640 1 -- 172.21.15.89:0/1223302439 shutdown_connections 2024-09-15T23:36:30.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f0096d52640 1 -- 172.21.15.89:0/1223302439 wait complete. 2024-09-15T23:36:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0096d52640 1 Processor -- start 2024-09-15T23:36:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0096d52640 1 -- start start 2024-09-15T23:36:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0096d52640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0090101b40 0x7f0090197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0096d52640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0090102d30 0x7f0090198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0096d52640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00901036f0 0x7f009019f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0096d52640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f009010d810 con 0x7f0090101b40 2024-09-15T23:36:30.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0096d52640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f009010d690 con 0x7f0090102d30 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0096d52640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f009010d990 con 0x7f00901036f0 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0094aee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0090101b40 0x7f0090197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0094aee640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0090101b40 0x7f0090197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47844/0 (socket says 172.21.15.89:47844) 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f0094aee640 1 -- 172.21.15.89:0/2686301122 learned_addr learned my addr 172.21.15.89:0/2686301122 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f00952ef640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00901036f0 0x7f009019f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f008ffff640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0090102d30 0x7f0090198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f0094aee640 1 -- 172.21.15.89:0/2686301122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00901036f0 msgr2=0x7f009019f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f0094aee640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00901036f0 0x7f009019f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f0094aee640 1 -- 172.21.15.89:0/2686301122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0090102d30 msgr2=0x7f0090198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f0094aee640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0090102d30 0x7f0090198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f0094aee640 1 -- 172.21.15.89:0/2686301122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009019f9d0 con 0x7f0090101b40 2024-09-15T23:36:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f0094aee640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0090101b40 0x7f0090197dc0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f008400d770 tx=0x7f008400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f008dffb640 1 -- 172.21.15.89:0/2686301122 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084017070 con 0x7f0090101b40 2024-09-15T23:36:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f008dffb640 1 -- 172.21.15.89:0/2686301122 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0084009c30 con 0x7f0090101b40 2024-09-15T23:36:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f0096d52640 1 -- 172.21.15.89:0/2686301122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009019fcc0 con 0x7f0090101b40 2024-09-15T23:36:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f008dffb640 1 -- 172.21.15.89:0/2686301122 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084003dd0 con 0x7f0090101b40 2024-09-15T23:36:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.590+0000 7f0096d52640 1 -- 172.21.15.89:0/2686301122 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00901a01c0 con 0x7f0090101b40 2024-09-15T23:36:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f0099bca740 1 Processor -- start 2024-09-15T23:36:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f0099bca740 1 -- start start 2024-09-15T23:36:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f008dffb640 1 -- 172.21.15.89:0/2686301122 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0084003f70 con 0x7f0090101b40 2024-09-15T23:36:30.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f0099bca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9ac0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f0099bca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac12f0 0x5632b9ac5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f0099bca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac6380 0x5632b9ac8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f0099bca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5632b9a132f0 con 0x5632b9ac6380 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f0099bca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5632b99bb270 con 0x5632b9ac09b0 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f0099bca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5632b9a0f2b0 con 0x5632b9ac12f0 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f008ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac6380 0x5632b9ac8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f008ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac6380 0x5632b9ac8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47862/0 (socket says 172.21.15.89:47862) 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f008ffff640 1 -- 172.21.15.89:0/2211349310 learned_addr learned my addr 172.21.15.89:0/2211349310 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f00952ef640 1 --2- 172.21.15.89:0/2211349310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9ac0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f0094aee640 1 --2- 172.21.15.89:0/2211349310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac12f0 0x5632b9ac5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f008ffff640 1 -- 172.21.15.89:0/2211349310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac12f0 msgr2=0x5632b9ac5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f008ffff640 1 --2- 172.21.15.89:0/2211349310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac12f0 0x5632b9ac5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f008ffff640 1 -- 172.21.15.89:0/2211349310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 msgr2=0x5632b9ac0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f008ffff640 1 --2- 172.21.15.89:0/2211349310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9ac0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f008ffff640 1 -- 172.21.15.89:0/2211349310 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632b99ba340 con 0x5632b9ac6380 2024-09-15T23:36:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f008ffff640 1 --2- 172.21.15.89:0/2211349310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac6380 0x5632b9ac8840 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f007c000c00 tx=0x7f007c004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e95a825ac4ed07b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f007affd640 1 -- 172.21.15.89:0/2211349310 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c035040 con 0x5632b9ac6380 2024-09-15T23:36:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f007affd640 1 -- 172.21.15.89:0/2211349310 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007c005520 con 0x5632b9ac6380 2024-09-15T23:36:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f007affd640 1 -- 172.21.15.89:0/2211349310 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c031910 con 0x5632b9ac6380 2024-09-15T23:36:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f008dffb640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00800a32f0 0x7f00800a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f00952ef640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00800a32f0 0x7f00800a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f008dffb640 1 -- 172.21.15.89:0/2686301122 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f00840cf6f0 con 0x7f0090101b40 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f0099bca740 1 -- 172.21.15.89:0/2211349310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac6380 msgr2=0x5632b9ac8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f00952ef640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00800a32f0 0x7f00800a57b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f0088005fd0 tx=0x7f0088009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f0099bca740 1 --2- 172.21.15.89:0/2211349310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac6380 0x5632b9ac8840 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f007c000c00 tx=0x7f007c004170 comp rx=0 tx=0).stop 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f0099bca740 1 -- 172.21.15.89:0/2211349310 shutdown_connections 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f0099bca740 1 --2- 172.21.15.89:0/2211349310 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac6380 0x5632b9ac8840 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f0099bca740 1 --2- 172.21.15.89:0/2211349310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac12f0 0x5632b9ac5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f0099bca740 1 --2- 172.21.15.89:0/2211349310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9ac0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f0099bca740 1 -- 172.21.15.89:0/2211349310 >> 172.21.15.89:0/2211349310 conn(0x5632b9aab010 msgr2=0x5632b9aab410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f0099bca740 1 -- 172.21.15.89:0/2211349310 shutdown_connections 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f0099bca740 1 -- 172.21.15.89:0/2211349310 wait complete. 2024-09-15T23:36:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f0099bca740 1 -- start start 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f0099bca740 1 Processor -- start 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f0099bca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9acab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f0099bca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f0099bca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac6380 0x5632b9ad20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f0099bca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5632b9ac92f0 con 0x5632b9ac12f0 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f008ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f0099bca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5632b999c290 con 0x5632b9ac09b0 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f0094aee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9acab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f008ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47878/0 (socket says 172.21.15.89:47878) 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f008ffff640 1 -- 172.21.15.89:0/1243558726 learned_addr learned my addr 172.21.15.89:0/1243558726 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f00952ef640 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac6380 0x5632b9ad20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f0099bca740 1 -- 172.21.15.89:0/1243558726 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5632b9ac9470 con 0x5632b9ac6380 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f0099bca740 10 client.0 _ll_register_callbacks cb 0x5632b99cd140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f008ffff640 1 -- 172.21.15.89:0/1243558726 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac6380 msgr2=0x5632b9ad20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f008ffff640 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac6380 0x5632b9ad20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f008ffff640 1 -- 172.21.15.89:0/1243558726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 msgr2=0x5632b9acab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f008ffff640 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9acab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f008ffff640 1 -- 172.21.15.89:0/1243558726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632b9726210 con 0x5632b9ac12f0 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f0094aee640 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9acab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f008ffff640 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acb090 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f007c0314e0 tx=0x7f007c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8baab362b95d00e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f006ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f006ffff640 1 -- 172.21.15.89:0/1243558726 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c052060 con 0x5632b9ac12f0 2024-09-15T23:36:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f006ffff640 1 -- 172.21.15.89:0/1243558726 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f007c004980 con 0x5632b9ac12f0 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f0099bca740 1 -- 172.21.15.89:0/1243558726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f007c005520 con 0x5632b9ac12f0 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f006ffff640 1 -- 172.21.15.89:0/1243558726 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f007c059050 con 0x5632b9ac12f0 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f0099bca740 10 client.19731 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f0099bca740 1 -- 172.21.15.89:0/1243558726 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5632b97f2620 con 0x5632b9ac12f0 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f006ffff640 1 -- 172.21.15.89:0/1243558726 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007c005520 con 0x5632b9ac12f0 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f006ffff640 1 -- 172.21.15.89:0/1243558726 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f007c0035f0 con 0x5632b9ac12f0 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f0099bca740 10 client.19731 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f0099bca740 10 client.19731 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f0099bca740 20 client.19731 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f0099bca740 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632b9ad63b0 0x5632b9ad87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f0099bca740 4 client.19731 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f0099bca740 1 -- 172.21.15.89:0/1243558726 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632b9a20a20 con 0x5632b9ad63b0 2024-09-15T23:36:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f0094aee640 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632b9ad63b0 0x5632b9ad87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f006ffff640 10 client.19731 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:30.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f0094aee640 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632b9ad63b0 0x5632b9ad87a0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bb6a44df8e0cd38 server_cookie=3c0abca9d9a64807 in_seq=0 out_seq=0 2024-09-15T23:36:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f006ffff640 1 -- 172.21.15.89:0/1243558726 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5632b9a20a20 con 0x5632b9ad63b0 2024-09-15T23:36:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f006ffff640 10 client.19731 handle_command_reply: tid=0 2024-09-15T23:36:30.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 client.19731 shutdown 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 -- 172.21.15.89:0/1243558726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 msgr2=0x5632b9acb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acb090 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f007c0314e0 tx=0x7f007c005d50 comp rx=0 tx=0).stop 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 -- 172.21.15.89:0/1243558726 shutdown_connections 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632b9ad63b0 0x5632b9ad87a0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac6380 0x5632b9ad20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acb090 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 --2- 172.21.15.89:0/1243558726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9acab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 -- 172.21.15.89:0/1243558726 >> 172.21.15.89:0/1243558726 conn(0x5632b9aab010 msgr2=0x5632b9aac820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f006ffff640 0 client.19731 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 -- 172.21.15.89:0/1243558726 shutdown_connections 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 1 -- 172.21.15.89:0/1243558726 wait complete. 2024-09-15T23:36:30.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.600+0000 7f0099bca740 20 client.19731 trim_cache size 0 max 16384 2024-09-15T23:36:30.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.586+0000 7f9bac91d640 1 -- 172.21.15.89:0/2189674190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ba8101e10 msgr2=0x7f9ba81021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.611 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.586+0000 7f9bac91d640 1 --2- 172.21.15.89:0/2189674190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ba8101e10 0x7f9ba81021f0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f9b98009900 tx=0x7f9b9802a410 comp rx=0 tx=0).stop 2024-09-15T23:36:30.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.586+0000 7f9bac91d640 1 -- 172.21.15.89:0/2189674190 shutdown_connections 2024-09-15T23:36:30.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.586+0000 7f9bac91d640 1 --2- 172.21.15.89:0/2189674190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ba8106f40 0x7f9ba810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.586+0000 7f9bac91d640 1 --2- 172.21.15.89:0/2189674190 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ba81027c0 0x7f9ba8106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.586+0000 7f9bac91d640 1 --2- 172.21.15.89:0/2189674190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ba8101e10 0x7f9ba81021f0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.586+0000 7f9bac91d640 1 -- 172.21.15.89:0/2189674190 >> 172.21.15.89:0/2189674190 conn(0x7f9ba80fd8d0 msgr2=0x7f9ba80ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.586+0000 7f9bac91d640 1 -- 172.21.15.89:0/2189674190 shutdown_connections 2024-09-15T23:36:30.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.586+0000 7f9bac91d640 1 -- 172.21.15.89:0/2189674190 wait complete. 2024-09-15T23:36:30.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9bac91d640 1 Processor -- start 2024-09-15T23:36:30.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9bac91d640 1 -- start start 2024-09-15T23:36:30.612 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9bac91d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ba8101e10 0x7f9ba8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9bac91d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ba81027c0 0x7f9ba8198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9bac91d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ba8106f40 0x7f9ba819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9bac91d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8112010 con 0x7f9ba81027c0 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9bac91d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8111e90 con 0x7f9ba8101e10 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9bac91d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8112190 con 0x7f9ba8106f40 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9ba5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ba81027c0 0x7f9ba8198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9ba5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ba81027c0 0x7f9ba8198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47830/0 (socket says 172.21.15.89:47830) 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/3275857163 learned_addr learned my addr 172.21.15.89:0/3275857163 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.587+0000 7f9ba659c640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ba8101e10 0x7f9ba8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9ba6d9d640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ba8106f40 0x7f9ba819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/3275857163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ba8106f40 msgr2=0x7f9ba819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.613 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ba8106f40 0x7f9ba819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/3275857163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ba8101e10 msgr2=0x7f9ba8198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ba8101e10 0x7f9ba8198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/3275857163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba819d400 con 0x7f9ba81027c0 2024-09-15T23:36:30.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ba81027c0 0x7f9ba8198990 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00b4a0 tx=0x7f9b9c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9b977fe640 1 -- 172.21.15.89:0/3275857163 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c014070 con 0x7f9ba81027c0 2024-09-15T23:36:30.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9b977fe640 1 -- 172.21.15.89:0/3275857163 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9c010af0 con 0x7f9ba81027c0 2024-09-15T23:36:30.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9bac91d640 1 -- 172.21.15.89:0/3275857163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba819d6f0 con 0x7f9ba81027c0 2024-09-15T23:36:30.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.588+0000 7f9bac91d640 1 -- 172.21.15.89:0/3275857163 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba81a4fd0 con 0x7f9ba81027c0 2024-09-15T23:36:30.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.589+0000 7f9b977fe640 1 -- 172.21.15.89:0/3275857163 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c017d10 con 0x7f9ba81027c0 2024-09-15T23:36:30.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.591+0000 7f9b977fe640 1 -- 172.21.15.89:0/3275857163 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9b9c017070 con 0x7f9ba81027c0 2024-09-15T23:36:30.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.592+0000 7f9baecf6740 1 Processor -- start 2024-09-15T23:36:30.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.592+0000 7f9baecf6740 1 -- start start 2024-09-15T23:36:30.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.592+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e50090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.592+0000 7f9b977fe640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9b900a3340 0x7f9b900a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.592+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e50a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.592+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e5d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.592+0000 7f9baecf6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c250db62f0 con 0x55c250e505d0 2024-09-15T23:36:30.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.592+0000 7f9baecf6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c250d5dcf0 con 0x55c250e5b370 2024-09-15T23:36:30.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.592+0000 7f9baecf6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c250db22b0 con 0x55c250e51b40 2024-09-15T23:36:30.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e50a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e50a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47850/0 (socket says 172.21.15.89:47850) 2024-09-15T23:36:30.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/1676448938 learned_addr learned my addr 172.21.15.89:0/1676448938 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:30.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9b977fe640 1 -- 172.21.15.89:0/3275857163 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9b9c0cf0b0 con 0x7f9ba81027c0 2024-09-15T23:36:30.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba6d9d640 1 --2- 172.21.15.89:0/1676448938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e50090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba6d9d640 1 --2- 172.21.15.89:0/1676448938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e5d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/1676448938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 msgr2=0x55c250e50090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/1676448938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e50090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/1676448938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 msgr2=0x55c250e5d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/1676448938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e5d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/1676448938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c250d423a0 con 0x55c250e505d0 2024-09-15T23:36:30.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba659c640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9b900a3340 0x7f9b900a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba6d9d640 1 --2- 172.21.15.89:0/1676448938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e5d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:30.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.593+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/1676448938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e50a10 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00b470 tx=0x7f9b9c01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfdc8a9150c32de server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f9b94ff9640 1 -- 172.21.15.89:0/1676448938 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c01e070 con 0x55c250e505d0 2024-09-15T23:36:30.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f9b94ff9640 1 -- 172.21.15.89:0/1676448938 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9c0a4db0 con 0x55c250e505d0 2024-09-15T23:36:30.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.594+0000 7f9b94ff9640 1 -- 172.21.15.89:0/1676448938 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0a3ef0 con 0x55c250e505d0 2024-09-15T23:36:30.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 -- 172.21.15.89:0/1676448938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 msgr2=0x55c250e50a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9ba659c640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9b900a3340 0x7f9b900a5800 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9b98000c00 tx=0x7f9b98005df0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 --2- 172.21.15.89:0/1676448938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e50a10 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00b470 tx=0x7f9b9c01c310 comp rx=0 tx=0).stop 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 -- 172.21.15.89:0/1676448938 shutdown_connections 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 --2- 172.21.15.89:0/1676448938 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e5d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 --2- 172.21.15.89:0/1676448938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e50a10 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 --2- 172.21.15.89:0/1676448938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e50090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 -- 172.21.15.89:0/1676448938 >> 172.21.15.89:0/1676448938 conn(0x55c250e19310 msgr2=0x55c250e4cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 -- 172.21.15.89:0/1676448938 shutdown_connections 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 -- 172.21.15.89:0/1676448938 wait complete. 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 -- start start 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.595+0000 7f9baecf6740 1 Processor -- start 2024-09-15T23:36:30.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e505d0 0x55c250e5ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e51b40 0x55c250e58050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e5b370 0x55c250e587e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9baecf6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c250e5e070 con 0x55c250e51b40 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9baecf6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c250d71060 con 0x55c250e505d0 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9baecf6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c250e5e1f0 con 0x55c250e5b370 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e51b40 0x55c250e58050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba5d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e51b40 0x55c250e58050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47876/0 (socket says 172.21.15.89:47876) 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/1937209868 learned_addr learned my addr 172.21.15.89:0/1937209868 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9baecf6740 10 client.0 _ll_register_callbacks cb 0x55c250ac48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba6d9d640 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e5b370 0x55c250e587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba659c640 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e505d0 0x55c250e5ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.621 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/1937209868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e5b370 msgr2=0x55c250e587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e5b370 0x55c250e587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/1937209868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e505d0 msgr2=0x55c250e5ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e505d0 0x55c250e5ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba5d9b640 1 -- 172.21.15.89:0/1937209868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c250b273c0 con 0x55c250e51b40 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9b8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e51b40 0x55c250e58050 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c09b210 tx=0x7f9b9c09b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d11d558e7899b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.596+0000 7f9b8dffb640 1 -- 172.21.15.89:0/1937209868 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c01e070 con 0x55c250e51b40 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9b8dffb640 1 -- 172.21.15.89:0/1937209868 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9b9c017360 con 0x55c250e51b40 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9baecf6740 1 -- 172.21.15.89:0/1937209868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b9c0a4db0 con 0x55c250e51b40 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9b8dffb640 1 -- 172.21.15.89:0/1937209868 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9b9c0a3ef0 con 0x55c250e51b40 2024-09-15T23:36:30.622 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9baecf6740 10 client.19728 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9baecf6740 1 -- 172.21.15.89:0/1937209868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c250e69b30 con 0x55c250e51b40 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9b8dffb640 1 -- 172.21.15.89:0/1937209868 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9c0a4db0 con 0x55c250e51b40 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9b8dffb640 1 -- 172.21.15.89:0/1937209868 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9b9c0bb0e0 con 0x55c250e51b40 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9baecf6740 10 client.19728 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9baecf6740 10 client.19728 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9baecf6740 20 client.19728 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9baecf6740 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55c250e6ae80 0x55c250e6d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9baecf6740 4 client.19728 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.597+0000 7f9baecf6740 1 -- 172.21.15.89:0/1937209868 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c250dc3860 con 0x55c250e6ae80 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f9ba659c640 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55c250e6ae80 0x55c250e6d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f9ba659c640 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55c250e6ae80 0x55c250e6d270 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=73a5eeac571b023c server_cookie=619cdbe07bdd1cc4 in_seq=0 out_seq=0 2024-09-15T23:36:30.623 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.598+0000 7f9b8dffb640 10 client.19728 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9b8dffb640 1 -- 172.21.15.89:0/1937209868 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c250dc3860 con 0x55c250e6ae80 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9b8dffb640 10 client.19728 handle_command_reply: tid=0 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 client.19728 shutdown 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 -- 172.21.15.89:0/1937209868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e51b40 msgr2=0x55c250e58050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e51b40 0x55c250e58050 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c09b210 tx=0x7f9b9c09b980 comp rx=0 tx=0).stop 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 -- 172.21.15.89:0/1937209868 shutdown_connections 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55c250e6ae80 0x55c250e6d270 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e5b370 0x55c250e587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e51b40 0x55c250e58050 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 --2- 172.21.15.89:0/1937209868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e505d0 0x55c250e5ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.624 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 -- 172.21.15.89:0/1937209868 >> 172.21.15.89:0/1937209868 conn(0x55c250e19310 msgr2=0x55c250e52990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9b8dffb640 0 client.19728 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:36:30.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 -- 172.21.15.89:0/1937209868 shutdown_connections 2024-09-15T23:36:30.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 1 -- 172.21.15.89:0/1937209868 wait complete. 2024-09-15T23:36:30.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.599+0000 7f9baecf6740 20 client.19728 trim_cache size 0 max 16384 2024-09-15T23:36:30.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9baecf6740 1 Processor -- start 2024-09-15T23:36:30.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9baecf6740 1 -- start start 2024-09-15T23:36:30.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e57910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e6feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9baecf6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81b1080 con 0x55c250e505d0 2024-09-15T23:36:30.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9baecf6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81b2bc0 con 0x55c250e5b370 2024-09-15T23:36:30.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9baecf6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81b1750 con 0x55c250e51b40 2024-09-15T23:36:30.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9ba659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e57910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9ba659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e57910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47892/0 (socket says 172.21.15.89:47892) 2024-09-15T23:36:30.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9ba659c640 1 -- 172.21.15.89:0/2279137472 learned_addr learned my addr 172.21.15.89:0/2279137472 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/2279137472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e6feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.613+0000 7f9ba6d9d640 1 --2- 172.21.15.89:0/2279137472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.614+0000 7f9ba659c640 1 -- 172.21.15.89:0/2279137472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 msgr2=0x55c250e6feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.614+0000 7f9ba659c640 1 --2- 172.21.15.89:0/2279137472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e6feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.614+0000 7f9ba659c640 1 -- 172.21.15.89:0/2279137472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 msgr2=0x55c250e703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.614+0000 7f9ba659c640 1 --2- 172.21.15.89:0/2279137472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e703f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.614+0000 7f9ba659c640 1 -- 172.21.15.89:0/2279137472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c250e53910 con 0x55c250e505d0 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.614+0000 7f9ba659c640 1 --2- 172.21.15.89:0/2279137472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e57910 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f9b98009230 tx=0x7f9b98040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1464dd0b860641 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.614+0000 7f9b8ffff640 1 -- 172.21.15.89:0/2279137472 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98035040 con 0x55c250e505d0 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.614+0000 7f9b8ffff640 1 -- 172.21.15.89:0/2279137472 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b98056070 con 0x55c250e505d0 2024-09-15T23:36:30.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.614+0000 7f9b8ffff640 1 -- 172.21.15.89:0/2279137472 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98050310 con 0x55c250e505d0 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 -- 172.21.15.89:0/2279137472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 msgr2=0x55c250e57910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 --2- 172.21.15.89:0/2279137472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e57910 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f9b98009230 tx=0x7f9b98040000 comp rx=0 tx=0).stop 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 -- 172.21.15.89:0/2279137472 shutdown_connections 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 --2- 172.21.15.89:0/2279137472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e703f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 --2- 172.21.15.89:0/2279137472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e6feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 --2- 172.21.15.89:0/2279137472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e57910 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 -- 172.21.15.89:0/2279137472 >> 172.21.15.89:0/2279137472 conn(0x55c250e19310 msgr2=0x55c250e56920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 -- 172.21.15.89:0/2279137472 shutdown_connections 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 -- 172.21.15.89:0/2279137472 wait complete. 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 -- start start 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 Processor -- start 2024-09-15T23:36:30.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e70390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e6d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e6dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81b24f0 con 0x55c250e505d0 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81b1e20 con 0x55c250e5b370 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba81af060 con 0x55c250e51b40 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.615+0000 7f9baecf6740 10 client.0 _ll_register_callbacks cb 0x7f9ba0007b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e70390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e70390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47894/0 (socket says 172.21.15.89:47894) 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba659c640 1 -- 172.21.15.89:0/3319311741 learned_addr learned my addr 172.21.15.89:0/3319311741 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e6d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba659c640 1 -- 172.21.15.89:0/3319311741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 msgr2=0x55c250e6d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba659c640 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e6d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba659c640 1 -- 172.21.15.89:0/3319311741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 msgr2=0x55c250e6dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba6d9d640 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e6dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba659c640 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e6dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba659c640 1 -- 172.21.15.89:0/3319311741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c250e6e230 con 0x55c250e505d0 2024-09-15T23:36:30.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9ba659c640 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e70390 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f9b9802a930 tx=0x7f9b980318f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b2102583be927e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9b8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:30.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9b8dffb640 1 -- 172.21.15.89:0/3319311741 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98035070 con 0x55c250e505d0 2024-09-15T23:36:30.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9baecf6740 1 -- 172.21.15.89:0/3319311741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b98056070 con 0x55c250e505d0 2024-09-15T23:36:30.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9b8dffb640 1 -- 172.21.15.89:0/3319311741 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9b98031cf0 con 0x55c250e505d0 2024-09-15T23:36:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f0099bca740 1 Processor -- start 2024-09-15T23:36:30.634 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f0099bca740 1 -- start start 2024-09-15T23:36:30.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9b8dffb640 1 -- 172.21.15.89:0/3319311741 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9b980035f0 con 0x55c250e505d0 2024-09-15T23:36:30.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9baecf6740 10 client.19737 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:30.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.616+0000 7f9baecf6740 1 -- 172.21.15.89:0/3319311741 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c250ed5530 con 0x55c250e505d0 2024-09-15T23:36:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f0099bca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac09b0 0x5632b9aab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.635 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f0099bca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9ad2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f9b8dffb640 1 -- 172.21.15.89:0/3319311741 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b98056070 con 0x55c250e505d0 2024-09-15T23:36:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f008ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9ad2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f008ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9ad2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47896/0 (socket says 172.21.15.89:47896) 2024-09-15T23:36:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f008ffff640 1 -- 172.21.15.89:0/2302252202 learned_addr learned my addr 172.21.15.89:0/2302252202 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f0099bca740 1 --2- 172.21.15.89:0/2302252202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac6380 0x5632b9aab930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f9b8dffb640 1 -- 172.21.15.89:0/3319311741 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9b9803c860 con 0x55c250e505d0 2024-09-15T23:36:30.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f9baecf6740 10 client.19737 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:30.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f9baecf6740 10 client.19737 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:36:30.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f9baecf6740 20 client.19737 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:30.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f9baecf6740 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55c250ed5c80 0x55c250ed80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f9baecf6740 4 client.19737 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:36:30.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f9baecf6740 1 -- 172.21.15.89:0/3319311741 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c250dbdd70 con 0x55c250ed5c80 2024-09-15T23:36:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f0094aee640 1 --2- 172.21.15.89:0/2302252202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac09b0 0x5632b9aab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f0099bca740 1 -- 172.21.15.89:0/2302252202 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0090075ff0 con 0x5632b9ac12f0 2024-09-15T23:36:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f0099bca740 1 -- 172.21.15.89:0/2302252202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f00901a18a0 con 0x5632b9ac6380 2024-09-15T23:36:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f0099bca740 1 -- 172.21.15.89:0/2302252202 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f00901a0430 con 0x5632b9ac09b0 2024-09-15T23:36:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f00952ef640 1 --2- 172.21.15.89:0/2302252202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac6380 0x5632b9aab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55c250ed5c80 0x55c250ed80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f008ffff640 1 -- 172.21.15.89:0/2302252202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac09b0 msgr2=0x5632b9aab3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.617+0000 7f008ffff640 1 --2- 172.21.15.89:0/2302252202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac09b0 0x5632b9aab3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f008ffff640 1 -- 172.21.15.89:0/2302252202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac6380 msgr2=0x5632b9aab930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f008ffff640 1 --2- 172.21.15.89:0/2302252202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac6380 0x5632b9aab930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f008ffff640 1 -- 172.21.15.89:0/2302252202 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632b9ac8bb0 con 0x5632b9ac12f0 2024-09-15T23:36:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9ba5d9b640 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55c250ed5c80 0x55c250ed80a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ae1aa509f62b995 server_cookie=c94b2d0e27cc8f03 in_seq=0 out_seq=0 2024-09-15T23:36:30.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9b8dffb640 10 client.19737 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:36:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f008ffff640 1 --2- 172.21.15.89:0/2302252202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9ad2090 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f007c004650 tx=0x7f007c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7fe2206daec040 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9b8dffb640 1 -- 172.21.15.89:0/3319311741 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c250dbdd70 con 0x55c250ed5c80 2024-09-15T23:36:30.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9b8dffb640 10 client.19737 handle_command_reply: tid=0 2024-09-15T23:36:30.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 client.19737 shutdown 2024-09-15T23:36:30.641 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f007a7fc640 1 -- 172.21.15.89:0/2302252202 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c035070 con 0x5632b9ac12f0 2024-09-15T23:36:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f007a7fc640 1 -- 172.21.15.89:0/2302252202 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007c0035f0 con 0x5632b9ac12f0 2024-09-15T23:36:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f007a7fc640 1 -- 172.21.15.89:0/2302252202 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c005040 con 0x5632b9ac12f0 2024-09-15T23:36:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 -- 172.21.15.89:0/3319311741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 msgr2=0x55c250e70390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e70390 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f9b9802a930 tx=0x7f9b980318f0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 -- 172.21.15.89:0/3319311741 shutdown_connections 2024-09-15T23:36:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c250e5b370 0x55c250e6dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55c250ed5c80 0x55c250ed80a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c250e51b40 0x55c250e6d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 --2- 172.21.15.89:0/3319311741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c250e505d0 0x55c250e70390 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9b8dffb640 0 client.19737 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:36:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 -- 172.21.15.89:0/3319311741 >> 172.21.15.89:0/3319311741 conn(0x55c250e19310 msgr2=0x55c250e598b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 -- 172.21.15.89:0/3319311741 shutdown_connections 2024-09-15T23:36:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 1 -- 172.21.15.89:0/3319311741 wait complete. 2024-09-15T23:36:30.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.618+0000 7f9baecf6740 20 client.19737 trim_cache size 0 max 16384 2024-09-15T23:36:30.644 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:36:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 -- 172.21.15.89:0/3275857163 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9b900a3340 msgr2=0x7f9b900a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9b900a3340 0x7f9b900a5800 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9b98000c00 tx=0x7f9b98005df0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 -- 172.21.15.89:0/3275857163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ba81027c0 msgr2=0x7f9ba8198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ba81027c0 0x7f9ba8198990 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c00b4a0 tx=0x7f9b9c00b960 comp rx=0 tx=0).stop 2024-09-15T23:36:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 -- 172.21.15.89:0/3275857163 shutdown_connections 2024-09-15T23:36:30.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9b900a3340 0x7f9b900a5800 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f0099bca740 1 -- 172.21.15.89:0/2302252202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 msgr2=0x5632b9ad2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.646 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f0099bca740 1 --2- 172.21.15.89:0/2302252202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9ad2090 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f007c004650 tx=0x7f007c000bc0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ba8106f40 0x7f9ba819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ba81027c0 0x7f9ba8198990 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 --2- 172.21.15.89:0/3275857163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ba8101e10 0x7f9ba8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f0099bca740 1 -- 172.21.15.89:0/2302252202 shutdown_connections 2024-09-15T23:36:30.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 -- 172.21.15.89:0/3275857163 >> 172.21.15.89:0/3275857163 conn(0x7f9ba80fd8d0 msgr2=0x7f9ba80feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f0099bca740 1 --2- 172.21.15.89:0/2302252202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac6380 0x5632b9aab930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f0099bca740 1 --2- 172.21.15.89:0/2302252202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9ad2090 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.647 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f0099bca740 1 --2- 172.21.15.89:0/2302252202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac09b0 0x5632b9aab3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 -- 172.21.15.89:0/3275857163 shutdown_connections 2024-09-15T23:36:30.648 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f0099bca740 1 -- 172.21.15.89:0/2302252202 >> 172.21.15.89:0/2302252202 conn(0x5632b9aab010 msgr2=0x5632b9ad83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f9bac91d640 1 -- 172.21.15.89:0/3275857163 wait complete. 2024-09-15T23:36:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f0099bca740 1 -- 172.21.15.89:0/2302252202 shutdown_connections 2024-09-15T23:36:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.619+0000 7f0099bca740 1 -- 172.21.15.89:0/2302252202 wait complete. 2024-09-15T23:36:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f0099bca740 1 -- start start 2024-09-15T23:36:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f0099bca740 1 Processor -- start 2024-09-15T23:36:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f0099bca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9acee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.652 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f0099bca740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f0099bca740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac6380 0x5632b9b3efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f0099bca740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f00901a11d0 con 0x5632b9ac12f0 2024-09-15T23:36:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f0099bca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f00901a0b00 con 0x5632b9ac09b0 2024-09-15T23:36:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f0099bca740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0090073fd0 con 0x5632b9ac6380 2024-09-15T23:36:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f0099bca740 10 client.0 _ll_register_callbacks cb 0x7f0088009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f008ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f008ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47902/0 (socket says 172.21.15.89:47902) 2024-09-15T23:36:30.653 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f008ffff640 1 -- 172.21.15.89:0/1239522133 learned_addr learned my addr 172.21.15.89:0/1239522133 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f0094aee640 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9acee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.620+0000 7f00952ef640 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac6380 0x5632b9b3efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f008ffff640 1 -- 172.21.15.89:0/1239522133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac6380 msgr2=0x5632b9b3efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f008ffff640 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac6380 0x5632b9b3efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f008ffff640 1 -- 172.21.15.89:0/1239522133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 msgr2=0x5632b9acee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f008ffff640 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9acee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f008ffff640 1 -- 172.21.15.89:0/1239522133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632b9b3f4f0 con 0x5632b9ac12f0 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f0094aee640 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9acee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f008ffff640 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acf370 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f007c031610 tx=0x7f007c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee81680329a560a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f006ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:30.654 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f006ffff640 1 -- 172.21.15.89:0/1239522133 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c035070 con 0x5632b9ac12f0 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f006ffff640 1 -- 172.21.15.89:0/1239522133 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f007c04a040 con 0x5632b9ac12f0 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f0099bca740 1 -- 172.21.15.89:0/1239522133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f007c0035f0 con 0x5632b9ac12f0 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f006ffff640 1 -- 172.21.15.89:0/1239522133 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f007c0318c0 con 0x5632b9ac12f0 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f0099bca740 10 client.19743 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f0099bca740 1 -- 172.21.15.89:0/1239522133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5632b9b406a0 con 0x5632b9ac12f0 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.621+0000 7f006ffff640 1 -- 172.21.15.89:0/1239522133 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007c0035f0 con 0x5632b9ac12f0 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.622+0000 7f006ffff640 1 -- 172.21.15.89:0/1239522133 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f007c05e4d0 con 0x5632b9ac12f0 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.622+0000 7f0099bca740 10 client.19743 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.622+0000 7f0099bca740 10 client.19743 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.622+0000 7f0099bca740 20 client.19743 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.622+0000 7f0099bca740 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632b9b40f40 0x5632b9b43360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:30.655 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.622+0000 7f0099bca740 4 client.19743 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.622+0000 7f0099bca740 1 -- 172.21.15.89:0/1239522133 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5632b9a1ad70 con 0x5632b9b40f40 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.622+0000 7f0094aee640 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632b9b40f40 0x5632b9b43360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.622+0000 7f0094aee640 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632b9b40f40 0x5632b9b43360 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f34a5ee7621641a server_cookie=877806dd14376962 in_seq=0 out_seq=0 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.622+0000 7f006ffff640 10 client.19743 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f006ffff640 1 -- 172.21.15.89:0/1239522133 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5632b9a1ad70 con 0x5632b9b40f40 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f006ffff640 10 client.19743 handle_command_reply: tid=0 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 client.19743 shutdown 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 -- 172.21.15.89:0/1239522133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 msgr2=0x5632b9acf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acf370 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f007c031610 tx=0x7f007c005fd0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 -- 172.21.15.89:0/1239522133 shutdown_connections 2024-09-15T23:36:30.656 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5632b9b40f40 0x5632b9b43360 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5632b9ac6380 0x5632b9b3efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b9ac12f0 0x5632b9acf370 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 --2- 172.21.15.89:0/1239522133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5632b9ac09b0 0x5632b9acee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f006ffff640 0 client.19743 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 -- 172.21.15.89:0/1239522133 >> 172.21.15.89:0/1239522133 conn(0x5632b9aab010 msgr2=0x5632b9aa99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 -- 172.21.15.89:0/1239522133 shutdown_connections 2024-09-15T23:36:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.635+0000 7f0099bca740 1 -- 172.21.15.89:0/1239522133 wait complete. 2024-09-15T23:36:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0099bca740 20 client.19743 trim_cache size 0 max 16384 2024-09-15T23:36:30.657 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:36:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:36:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:36:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:36:30.658 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 -- 172.21.15.89:0/2686301122 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00800a32f0 msgr2=0x7f00800a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00800a32f0 0x7f00800a57b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f0088005fd0 tx=0x7f0088009500 comp rx=0 tx=0).stop 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 -- 172.21.15.89:0/2686301122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0090101b40 msgr2=0x7f0090197dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0090101b40 0x7f0090197dc0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f008400d770 tx=0x7f008400dc30 comp rx=0 tx=0).stop 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 -- 172.21.15.89:0/2686301122 shutdown_connections 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f00800a32f0 0x7f00800a57b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00901036f0 0x7f009019f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0090102d30 0x7f0090198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 --2- 172.21.15.89:0/2686301122 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0090101b40 0x7f0090197dc0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 -- 172.21.15.89:0/2686301122 >> 172.21.15.89:0/2686301122 conn(0x7f00900fd2e0 msgr2=0x7f0090106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 -- 172.21.15.89:0/2686301122 shutdown_connections 2024-09-15T23:36:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:30.636+0000 7f0096d52640 1 -- 172.21.15.89:0/2686301122 wait complete. 2024-09-15T23:36:30.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:30 smithi089 ceph-mon[30933]: pgmap v724: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:36:30.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:30 smithi120 ceph-mon[35250]: pgmap v724: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:36:30.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:30 smithi174 ceph-mon[35586]: pgmap v724: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:36:31.447 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:36:31.696 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.193+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1392563589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d60100a80 msgr2=0x7f2d60100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.193+0000 7f2d67fbc640 1 --2- 172.21.15.89:0/1392563589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d60100a80 0x7f2d60100ee0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f2d50009930 tx=0x7f2d5002a420 comp rx=0 tx=0).stop 2024-09-15T23:36:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.193+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1392563589 shutdown_connections 2024-09-15T23:36:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.193+0000 7f2d67fbc640 1 --2- 172.21.15.89:0/1392563589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d60101420 0x7f2d6010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.193+0000 7f2d67fbc640 1 --2- 172.21.15.89:0/1392563589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d60100a80 0x7f2d60100ee0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.193+0000 7f2d67fbc640 1 --2- 172.21.15.89:0/1392563589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d60106a80 0x7f2d60106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.193+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1392563589 >> 172.21.15.89:0/1392563589 conn(0x7f2d600fc7a0 msgr2=0x7f2d600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.193+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1392563589 shutdown_connections 2024-09-15T23:36:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.193+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1392563589 wait complete. 2024-09-15T23:36:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d67fbc640 1 Processor -- start 2024-09-15T23:36:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d67fbc640 1 -- start start 2024-09-15T23:36:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d67fbc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d60100a80 0x7f2d60102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d67fbc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d60101420 0x7f2d60102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d67fbc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d60106a80 0x7f2d601030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d67fbc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d60110e40 con 0x7f2d60106a80 2024-09-15T23:36:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d67fbc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d60110cc0 con 0x7f2d60100a80 2024-09-15T23:36:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d67fbc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d60110fc0 con 0x7f2d60101420 2024-09-15T23:36:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d66559640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d60106a80 0x7f2d601030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d66559640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d60106a80 0x7f2d601030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47912/0 (socket says 172.21.15.89:47912) 2024-09-15T23:36:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d66559640 1 -- 172.21.15.89:0/1787149183 learned_addr learned my addr 172.21.15.89:0/1787149183 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.194+0000 7f2d65d58640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d60100a80 0x7f2d60102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d65557640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d60101420 0x7f2d60102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d66559640 1 -- 172.21.15.89:0/1787149183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d60101420 msgr2=0x7f2d60102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d66559640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d60101420 0x7f2d60102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d66559640 1 -- 172.21.15.89:0/1787149183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d60100a80 msgr2=0x7f2d60102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d66559640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d60100a80 0x7f2d60102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d66559640 1 -- 172.21.15.89:0/1787149183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d601a6910 con 0x7f2d60106a80 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d66559640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d60106a80 0x7f2d601030d0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800eaa0 tx=0x7f2d5800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d56ffd640 1 -- 172.21.15.89:0/1787149183 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5800cdb0 con 0x7f2d60106a80 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d56ffd640 1 -- 172.21.15.89:0/1787149183 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d58005020 con 0x7f2d60106a80 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1787149183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d601a6c00 con 0x7f2d60106a80 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d56ffd640 1 -- 172.21.15.89:0/1787149183 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d580116b0 con 0x7f2d60106a80 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.195+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1787149183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d601a70c0 con 0x7f2d60106a80 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.197+0000 7f2d56ffd640 1 -- 172.21.15.89:0/1787149183 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2d58005560 con 0x7f2d60106a80 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.197+0000 7f2d6a395740 1 Processor -- start 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d6a395740 1 -- start start 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a8311090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 0x5625a8311a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a831c370 0x5625a831e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d6a395740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5625a82772f0 con 0x5625a831c370 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d6a395740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5625a821ecf0 con 0x5625a8312b40 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d6a395740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5625a82732b0 con 0x5625a83115d0 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d56ffd640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d4c0a3240 0x7f2d4c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d65557640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a831c370 0x5625a831e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d65557640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a831c370 0x5625a831e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47922/0 (socket says 172.21.15.89:47922) 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d65557640 1 -- 172.21.15.89:0/2526725670 learned_addr learned my addr 172.21.15.89:0/2526725670 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d66559640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d4c0a3240 0x7f2d4c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d65d58640 1 --2- 172.21.15.89:0/2526725670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 0x5625a8311a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d65557640 1 --2- 172.21.15.89:0/2526725670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a8311090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.198+0000 7f2d56ffd640 1 -- 172.21.15.89:0/1787149183 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2d580cfea0 con 0x7f2d60106a80 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d65557640 1 -- 172.21.15.89:0/2526725670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 msgr2=0x5625a8311a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d65557640 1 --2- 172.21.15.89:0/2526725670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 0x5625a8311a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d65557640 1 -- 172.21.15.89:0/2526725670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 msgr2=0x5625a8311090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d65557640 1 --2- 172.21.15.89:0/2526725670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a8311090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d65557640 1 -- 172.21.15.89:0/2526725670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625a82033a0 con 0x5625a831c370 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d65d58640 1 --2- 172.21.15.89:0/2526725670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 0x5625a8311a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d65557640 1 --2- 172.21.15.89:0/2526725670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a831c370 0x5625a831e760 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f2d5002a940 tx=0x7f2d50005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3234f837a199a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d66559640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d4c0a3240 0x7f2d4c0a5700 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2d5809be80 tx=0x7f2d5806bdf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d4bfff640 1 -- 172.21.15.89:0/2526725670 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50035070 con 0x5625a831c370 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d4bfff640 1 -- 172.21.15.89:0/2526725670 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d50031730 con 0x5625a831c370 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.199+0000 7f2d4bfff640 1 -- 172.21.15.89:0/2526725670 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d500047a0 con 0x5625a831c370 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 -- 172.21.15.89:0/2526725670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a831c370 msgr2=0x5625a831e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 --2- 172.21.15.89:0/2526725670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a831c370 0x5625a831e760 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f2d5002a940 tx=0x7f2d50005c00 comp rx=0 tx=0).stop 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 -- 172.21.15.89:0/2526725670 shutdown_connections 2024-09-15T23:36:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 --2- 172.21.15.89:0/2526725670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a831c370 0x5625a831e760 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 --2- 172.21.15.89:0/2526725670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 0x5625a8311a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 --2- 172.21.15.89:0/2526725670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a8311090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 -- 172.21.15.89:0/2526725670 >> 172.21.15.89:0/2526725670 conn(0x5625a82da310 msgr2=0x5625a830dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:32.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 -- 172.21.15.89:0/2526725670 shutdown_connections 2024-09-15T23:36:32.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 -- 172.21.15.89:0/2526725670 wait complete. 2024-09-15T23:36:32.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 -- start start 2024-09-15T23:36:32.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.200+0000 7f2d6a395740 1 Processor -- start 2024-09-15T23:36:32.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a831bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a8319030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a831c370 0x5625a8319640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d6a395740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5625a831f010 con 0x5625a83115d0 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d6a395740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5625a8232060 con 0x5625a8312b40 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d6a395740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5625a831f190 con 0x5625a831c370 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d6a395740 10 client.0 _ll_register_callbacks cb 0x5625a7f858c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d65557640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a831bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d65557640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a831bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47938/0 (socket says 172.21.15.89:47938) 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d65557640 1 -- 172.21.15.89:0/3385808550 learned_addr learned my addr 172.21.15.89:0/3385808550 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d65557640 1 -- 172.21.15.89:0/3385808550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a831c370 msgr2=0x5625a8319640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d65557640 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a831c370 0x5625a8319640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d65557640 1 -- 172.21.15.89:0/3385808550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 msgr2=0x5625a8319030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d65557640 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a8319030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d65557640 1 -- 172.21.15.89:0/3385808550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625a7fe83c0 con 0x5625a83115d0 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d65557640 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a831bd80 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f2d50006010 tx=0x7f2d5002aae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9efcc299cca77d24 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d497fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d497fa640 1 -- 172.21.15.89:0/3385808550 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50035040 con 0x5625a83115d0 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d497fa640 1 -- 172.21.15.89:0/3385808550 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2d50004040 con 0x5625a83115d0 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d6a395740 1 -- 172.21.15.89:0/3385808550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d50031730 con 0x5625a83115d0 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d497fa640 1 -- 172.21.15.89:0/3385808550 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2d5005b050 con 0x5625a83115d0 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d6a395740 10 client.19755 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.201+0000 7f2d6a395740 1 -- 172.21.15.89:0/3385808550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625a832ab10 con 0x5625a83115d0 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d497fa640 1 -- 172.21.15.89:0/3385808550 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d50031730 con 0x5625a83115d0 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d497fa640 1 -- 172.21.15.89:0/3385808550 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2d500035f0 con 0x5625a83115d0 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d6a395740 10 client.19755 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d6a395740 10 client.19755 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d6a395740 20 client.19755 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:32.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d6a395740 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5625a832be60 0x5625a832e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d6a395740 4 client.19755 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d6a395740 1 -- 172.21.15.89:0/3385808550 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625a8284860 con 0x5625a832be60 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d65d58640 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5625a832be60 0x5625a832e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d65d58640 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5625a832be60 0x5625a832e250 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3eec485b9954c270 server_cookie=e295db98aeb43e23 in_seq=0 out_seq=0 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.202+0000 7f2d497fa640 10 client.19755 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d497fa640 1 -- 172.21.15.89:0/3385808550 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5625a8284860 con 0x5625a832be60 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d497fa640 10 client.19755 handle_command_reply: tid=0 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d6a395740 1 client.19755 shutdown 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d6a395740 1 -- 172.21.15.89:0/3385808550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 msgr2=0x5625a831bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d6a395740 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a831bd80 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f2d50006010 tx=0x7f2d5002aae0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d6a395740 1 -- 172.21.15.89:0/3385808550 shutdown_connections 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d6a395740 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5625a832be60 0x5625a832e250 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d6a395740 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a831c370 0x5625a8319640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d6a395740 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a8319030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d497fa640 0 client.19755 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:36:32.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d6a395740 1 --2- 172.21.15.89:0/3385808550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a831bd80 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d6a395740 1 -- 172.21.15.89:0/3385808550 >> 172.21.15.89:0/3385808550 conn(0x5625a82da310 msgr2=0x5625a8313960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:32.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.203+0000 7f2d6a395740 1 -- 172.21.15.89:0/3385808550 shutdown_connections 2024-09-15T23:36:32.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.204+0000 7f2d6a395740 1 -- 172.21.15.89:0/3385808550 wait complete. 2024-09-15T23:36:32.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.204+0000 7f2d6a395740 20 client.19755 trim_cache size 0 max 16384 2024-09-15T23:36:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d6a395740 1 Processor -- start 2024-09-15T23:36:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d6a395740 1 -- start start 2024-09-15T23:36:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a8328460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a830de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a831c370 0x5625a83188f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d6a395740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d601b1020 con 0x5625a83115d0 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d6a395740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d601b2b60 con 0x5625a8312b40 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d6a395740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d601b16f0 con 0x5625a831c370 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d65557640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a8328460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d65557640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a8328460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47942/0 (socket says 172.21.15.89:47942) 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d65557640 1 -- 172.21.15.89:0/2992301569 learned_addr learned my addr 172.21.15.89:0/2992301569 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d65d58640 1 --2- 172.21.15.89:0/2992301569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a830de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d65557640 1 -- 172.21.15.89:0/2992301569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a831c370 msgr2=0x5625a83188f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d65557640 1 --2- 172.21.15.89:0/2992301569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a831c370 0x5625a83188f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d65557640 1 -- 172.21.15.89:0/2992301569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 msgr2=0x5625a830de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d65557640 1 --2- 172.21.15.89:0/2992301569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a830de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d65557640 1 -- 172.21.15.89:0/2992301569 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625a831ead0 con 0x5625a83115d0 2024-09-15T23:36:32.231 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d65557640 1 --2- 172.21.15.89:0/2992301569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a8328460 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f2d500039f0 tx=0x7f2d50005430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb2f833bebfc3cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d4b7fe640 1 -- 172.21.15.89:0/2992301569 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50035070 con 0x5625a83115d0 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d4b7fe640 1 -- 172.21.15.89:0/2992301569 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d50058030 con 0x5625a83115d0 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.217+0000 7f2d4b7fe640 1 -- 172.21.15.89:0/2992301569 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50003850 con 0x5625a83115d0 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 -- 172.21.15.89:0/2992301569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 msgr2=0x5625a8328460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 --2- 172.21.15.89:0/2992301569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a8328460 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f2d500039f0 tx=0x7f2d50005430 comp rx=0 tx=0).stop 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 -- 172.21.15.89:0/2992301569 shutdown_connections 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 --2- 172.21.15.89:0/2992301569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a831c370 0x5625a83188f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 --2- 172.21.15.89:0/2992301569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a8312b40 0x5625a830de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 --2- 172.21.15.89:0/2992301569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a83115d0 0x5625a8328460 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 -- 172.21.15.89:0/2992301569 >> 172.21.15.89:0/2992301569 conn(0x5625a82da310 msgr2=0x5625a830d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 -- 172.21.15.89:0/2992301569 shutdown_connections 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 -- 172.21.15.89:0/2992301569 wait complete. 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 -- start start 2024-09-15T23:36:32.232 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.218+0000 7f2d6a395740 1 Processor -- start 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 0x5625a8331340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a8312b40 0x5625a8331880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d6a395740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a831c370 0x5625a832e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d6a395740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d601b2490 con 0x5625a8312b40 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d6a395740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d601b1dc0 con 0x5625a831c370 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d6a395740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d601af000 con 0x5625a83115d0 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d6a395740 10 client.0 _ll_register_callbacks cb 0x7f2d50032ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65d58640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a8312b40 0x5625a8331880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65d58640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a8312b40 0x5625a8331880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47954/0 (socket says 172.21.15.89:47954) 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65d58640 1 -- 172.21.15.89:0/1273232812 learned_addr learned my addr 172.21.15.89:0/1273232812 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65557640 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 0x5625a8331340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65557640 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a831c370 0x5625a832e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65d58640 1 -- 172.21.15.89:0/1273232812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 msgr2=0x5625a8331340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65d58640 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 0x5625a8331340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65d58640 1 -- 172.21.15.89:0/1273232812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a831c370 msgr2=0x5625a832e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65d58640 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a831c370 0x5625a832e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65d58640 1 -- 172.21.15.89:0/1273232812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625a832efb0 con 0x5625a8312b40 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65557640 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a831c370 0x5625a832e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d65d58640 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a8312b40 0x5625a8331880 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c000c00 tx=0x7f2d5c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2cc91bdbdbc22f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d497fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d497fa640 1 -- 172.21.15.89:0/1273232812 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c005220 con 0x5625a8312b40 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d497fa640 1 -- 172.21.15.89:0/1273232812 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2d5c0053c0 con 0x5625a8312b40 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.219+0000 7f2d6a395740 1 -- 172.21.15.89:0/1273232812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d50058030 con 0x5625a8312b40 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d497fa640 1 -- 172.21.15.89:0/1273232812 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2d5c02b050 con 0x5625a8312b40 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d6a395740 10 client.19761 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d6a395740 1 -- 172.21.15.89:0/1273232812 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625a83962b0 con 0x5625a8312b40 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d497fa640 1 -- 172.21.15.89:0/1273232812 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d50058030 con 0x5625a8312b40 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d497fa640 1 -- 172.21.15.89:0/1273232812 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2d5c01a810 con 0x5625a8312b40 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d6a395740 10 client.19761 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d6a395740 10 client.19761 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d6a395740 20 client.19761 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d6a395740 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5625a8396a60 0x5625a8398e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d6a395740 4 client.19761 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d6a395740 1 -- 172.21.15.89:0/1273232812 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625a827ed70 con 0x5625a8396a60 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.220+0000 7f2d65557640 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5625a8396a60 0x5625a8398e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d65557640 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5625a8396a60 0x5625a8398e80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=79682c6e66db2838 server_cookie=69bcd9dc3414424c in_seq=0 out_seq=0 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d497fa640 10 client.19761 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d497fa640 1 -- 172.21.15.89:0/1273232812 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625a827ed70 con 0x5625a8396a60 2024-09-15T23:36:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d497fa640 10 client.19761 handle_command_reply: tid=0 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 client.19761 shutdown 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 -- 172.21.15.89:0/1273232812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a8312b40 msgr2=0x5625a8331880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a8312b40 0x5625a8331880 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c000c00 tx=0x7f2d5c011810 comp rx=0 tx=0).stop 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 -- 172.21.15.89:0/1273232812 shutdown_connections 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5625a8396a60 0x5625a8398e80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625a831c370 0x5625a832e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d497fa640 0 client.19761 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5625a8312b40 0x5625a8331880 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 --2- 172.21.15.89:0/1273232812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5625a83115d0 0x5625a8331340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 -- 172.21.15.89:0/1273232812 >> 172.21.15.89:0/1273232812 conn(0x5625a82da310 msgr2=0x5625a830c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 -- 172.21.15.89:0/1273232812 shutdown_connections 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 1 -- 172.21.15.89:0/1273232812 wait complete. 2024-09-15T23:36:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.221+0000 7f2d6a395740 20 client.19761 trim_cache size 0 max 16384 2024-09-15T23:36:32.237 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1787149183 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d4c0a3240 msgr2=0x7f2d4c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d4c0a3240 0x7f2d4c0a5700 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2d5809be80 tx=0x7f2d5806bdf0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1787149183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d60106a80 msgr2=0x7f2d601030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d60106a80 0x7f2d601030d0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800eaa0 tx=0x7f2d5800ef60 comp rx=0 tx=0).stop 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1787149183 shutdown_connections 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d4c0a3240 0x7f2d4c0a5700 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d60106a80 0x7f2d601030d0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d60101420 0x7f2d60102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 --2- 172.21.15.89:0/1787149183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d60100a80 0x7f2d60102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1787149183 >> 172.21.15.89:0/1787149183 conn(0x7f2d600fc7a0 msgr2=0x7f2d60104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1787149183 shutdown_connections 2024-09-15T23:36:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:32.222+0000 7f2d67fbc640 1 -- 172.21.15.89:0/1787149183 wait complete. 2024-09-15T23:36:32.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:32 smithi089 ceph-mon[30933]: pgmap v725: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:36:32.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:32 smithi120 ceph-mon[35250]: pgmap v725: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:36:32.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:32 smithi174 ceph-mon[35586]: pgmap v725: 129 pgs: 129 active+clean; 2.0 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:36:33.024 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:36:33.272 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:33.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.722+0000 7f26e1cf6640 1 -- 172.21.15.89:0/787794134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26dc102d40 msgr2=0x7f26dc103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.722+0000 7f26e1cf6640 1 --2- 172.21.15.89:0/787794134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26dc102d40 0x7f26dc103120 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f26cc009900 tx=0x7f26cc02a480 comp rx=0 tx=0).stop 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.722+0000 7f26e1cf6640 1 -- 172.21.15.89:0/787794134 shutdown_connections 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.722+0000 7f26e1cf6640 1 --2- 172.21.15.89:0/787794134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc10c170 0x7f26dc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.722+0000 7f26e1cf6640 1 --2- 172.21.15.89:0/787794134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26dc103660 0x7f26dc10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.722+0000 7f26e1cf6640 1 --2- 172.21.15.89:0/787794134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26dc102d40 0x7f26dc103120 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.722+0000 7f26e1cf6640 1 -- 172.21.15.89:0/787794134 >> 172.21.15.89:0/787794134 conn(0x7f26dc0fc7a0 msgr2=0x7f26dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.722+0000 7f26e1cf6640 1 -- 172.21.15.89:0/787794134 shutdown_connections 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.722+0000 7f26e1cf6640 1 -- 172.21.15.89:0/787794134 wait complete. 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.723+0000 7f26e1cf6640 1 Processor -- start 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.723+0000 7f26e1cf6640 1 -- start start 2024-09-15T23:36:33.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.723+0000 7f26e1cf6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26dc102d40 0x7f26dc198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.723+0000 7f26e1cf6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26dc103660 0x7f26dc198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.723+0000 7f26e1cf6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc10c170 0x7f26dc19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.723+0000 7f26e1cf6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc111fb0 con 0x7f26dc103660 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.723+0000 7f26e1cf6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc111e30 con 0x7f26dc10c170 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.723+0000 7f26e1cf6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc112130 con 0x7f26dc102d40 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26daffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26dc103660 0x7f26dc198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26daffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26dc103660 0x7f26dc198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47974/0 (socket says 172.21.15.89:47974) 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26daffd640 1 -- 172.21.15.89:0/381417179 learned_addr learned my addr 172.21.15.89:0/381417179 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26db7fe640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26dc102d40 0x7f26dc198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26dbfff640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc10c170 0x7f26dc19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26daffd640 1 -- 172.21.15.89:0/381417179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26dc102d40 msgr2=0x7f26dc198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26daffd640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26dc102d40 0x7f26dc198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26daffd640 1 -- 172.21.15.89:0/381417179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc10c170 msgr2=0x7f26dc19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26daffd640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc10c170 0x7f26dc19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26daffd640 1 -- 172.21.15.89:0/381417179 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc19d400 con 0x7f26dc103660 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26daffd640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26dc103660 0x7f26dc198990 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b3f0 tx=0x7f26d000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26d8ff9640 1 -- 172.21.15.89:0/381417179 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d0015020 con 0x7f26dc103660 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26e1cf6640 1 -- 172.21.15.89:0/381417179 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26dc19d6f0 con 0x7f26dc103660 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26d8ff9640 1 -- 172.21.15.89:0/381417179 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d0011d10 con 0x7f26dc103660 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26d8ff9640 1 -- 172.21.15.89:0/381417179 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d000f2e0 con 0x7f26dc103660 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.724+0000 7f26e1cf6640 1 -- 172.21.15.89:0/381417179 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26dc1a4fd0 con 0x7f26dc103660 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.727+0000 7f26d8ff9640 1 -- 172.21.15.89:0/381417179 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f26d0018020 con 0x7f26dc103660 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.727+0000 7f26e40cf740 1 Processor -- start 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.727+0000 7f26e40cf740 1 -- start start 2024-09-15T23:36:33.770 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.727+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d29b40 0x564fb1d28090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.727+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d285d0 0x564fb1d28a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d35760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26e40cf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564fb1c8e2f0 con 0x564fb1d33370 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26e40cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564fb1c35cf0 con 0x564fb1d285d0 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26e40cf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564fb1c8a2b0 con 0x564fb1d29b40 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26daffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d35760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26daffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d35760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47982/0 (socket says 172.21.15.89:47982) 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26db7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d285d0 0x564fb1d28a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26daffd640 1 -- 172.21.15.89:0/477103252 learned_addr learned my addr 172.21.15.89:0/477103252 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26dbfff640 1 --2- 172.21.15.89:0/477103252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d29b40 0x564fb1d28090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26daffd640 1 -- 172.21.15.89:0/477103252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d29b40 msgr2=0x564fb1d28090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26daffd640 1 --2- 172.21.15.89:0/477103252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d29b40 0x564fb1d28090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26d8ff9640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f26c40a3240 0x7f26c40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26daffd640 1 -- 172.21.15.89:0/477103252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d285d0 msgr2=0x564fb1d28a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:33.771 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26daffd640 1 --2- 172.21.15.89:0/477103252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d285d0 0x564fb1d28a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26daffd640 1 -- 172.21.15.89:0/477103252 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fb1c1a3a0 con 0x564fb1d33370 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26dbfff640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f26c40a3240 0x7f26c40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26d8ff9640 1 -- 172.21.15.89:0/381417179 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f26d00cf4a0 con 0x7f26dc103660 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26daffd640 1 --2- 172.21.15.89:0/477103252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d35760 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f26d00ad600 tx=0x7f26d00aaf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc2025eb74ecfce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.728+0000 7f26c1ffb640 1 -- 172.21.15.89:0/477103252 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d009b4a0 con 0x564fb1d33370 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26c1ffb640 1 -- 172.21.15.89:0/477103252 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d00a5630 con 0x564fb1d33370 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26c1ffb640 1 -- 172.21.15.89:0/477103252 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d0009200 con 0x564fb1d33370 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26e40cf740 1 -- 172.21.15.89:0/477103252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 msgr2=0x564fb1d35760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26dbfff640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f26c40a3240 0x7f26c40a5700 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f26d4007fa0 tx=0x7f26d4007520 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26e40cf740 1 --2- 172.21.15.89:0/477103252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d35760 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f26d00ad600 tx=0x7f26d00aaf10 comp rx=0 tx=0).stop 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26e40cf740 1 -- 172.21.15.89:0/477103252 shutdown_connections 2024-09-15T23:36:33.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26e40cf740 1 --2- 172.21.15.89:0/477103252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d35760 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26e40cf740 1 --2- 172.21.15.89:0/477103252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d285d0 0x564fb1d28a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26e40cf740 1 --2- 172.21.15.89:0/477103252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d29b40 0x564fb1d28090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26e40cf740 1 -- 172.21.15.89:0/477103252 >> 172.21.15.89:0/477103252 conn(0x564fb1cf1310 msgr2=0x564fb1d24c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26e40cf740 1 -- 172.21.15.89:0/477103252 shutdown_connections 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.729+0000 7f26e40cf740 1 -- 172.21.15.89:0/477103252 wait complete. 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26e40cf740 1 -- start start 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26e40cf740 1 Processor -- start 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d32d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d29b40 0x564fb1d2fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d33370 0x564fb1d305e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26e40cf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564fb1d36070 con 0x564fb1d29b40 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26e40cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564fb1c49060 con 0x564fb1d33370 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26e40cf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564fb1d361f0 con 0x564fb1d285d0 2024-09-15T23:36:33.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26e40cf740 10 client.0 _ll_register_callbacks cb 0x564fb199c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:33.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26daffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d29b40 0x564fb1d2fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.774 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:36:33.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26daffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d29b40 0x564fb1d2fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47992/0 (socket says 172.21.15.89:47992) 2024-09-15T23:36:33.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26daffd640 1 -- 172.21.15.89:0/1438237498 learned_addr learned my addr 172.21.15.89:0/1438237498 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26db7fe640 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d32d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.730+0000 7f26dbfff640 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d33370 0x564fb1d305e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26daffd640 1 -- 172.21.15.89:0/1438237498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 msgr2=0x564fb1d32d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26daffd640 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d32d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26daffd640 1 -- 172.21.15.89:0/1438237498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d33370 msgr2=0x564fb1d305e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26daffd640 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d33370 0x564fb1d305e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26daffd640 1 -- 172.21.15.89:0/1438237498 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fb19ff3c0 con 0x564fb1d29b40 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26daffd640 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d29b40 0x564fb1d2fff0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f26d00ccc00 tx=0x7f26d00d2b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc006d371019c73 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26baffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26baffd640 1 -- 172.21.15.89:0/1438237498 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d0010ed0 con 0x564fb1d29b40 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26baffd640 1 -- 172.21.15.89:0/1438237498 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f26d0010510 con 0x564fb1d29b40 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26e40cf740 1 -- 172.21.15.89:0/1438237498 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564fb1c9b860 con 0x564fb1d29b40 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26baffd640 1 -- 172.21.15.89:0/1438237498 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f26d009b030 con 0x564fb1d29b40 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26e40cf740 10 client.19773 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:33.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26e40cf740 1 -- 172.21.15.89:0/1438237498 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564fb1d36070 con 0x564fb1d29b40 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26baffd640 1 -- 172.21.15.89:0/1438237498 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564fb1c9b860 con 0x564fb1d29b40 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26baffd640 1 -- 172.21.15.89:0/1438237498 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f26d00cf170 con 0x564fb1d29b40 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26e40cf740 10 client.19773 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26e40cf740 10 client.19773 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26e40cf740 20 client.19773 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564fb1d42c60 0x564fb1d45050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26e40cf740 4 client.19773 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.731+0000 7f26e40cf740 1 -- 172.21.15.89:0/1438237498 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fb1c95d70 con 0x564fb1d42c60 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.732+0000 7f26db7fe640 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564fb1d42c60 0x564fb1d45050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.732+0000 7f26baffd640 10 client.19773 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.732+0000 7f26db7fe640 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564fb1d42c60 0x564fb1d45050 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30fe82a5e66d6b8a server_cookie=7a3493fabaedfd27 in_seq=0 out_seq=0 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26baffd640 1 -- 172.21.15.89:0/1438237498 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564fb1c95d70 con 0x564fb1d42c60 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26baffd640 10 client.19773 handle_command_reply: tid=0 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 client.19773 shutdown 2024-09-15T23:36:33.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 -- 172.21.15.89:0/1438237498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d29b40 msgr2=0x564fb1d2fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d29b40 0x564fb1d2fff0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f26d00ccc00 tx=0x7f26d00d2b60 comp rx=0 tx=0).stop 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 -- 172.21.15.89:0/1438237498 shutdown_connections 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564fb1d42c60 0x564fb1d45050 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d33370 0x564fb1d305e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d29b40 0x564fb1d2fff0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26baffd640 0 client.19773 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1438237498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d32d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 -- 172.21.15.89:0/1438237498 >> 172.21.15.89:0/1438237498 conn(0x564fb1cf1310 msgr2=0x564fb1d2a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 -- 172.21.15.89:0/1438237498 shutdown_connections 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 1 -- 172.21.15.89:0/1438237498 wait complete. 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.733+0000 7f26e40cf740 20 client.19773 trim_cache size 0 max 16384 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26e40cf740 1 Processor -- start 2024-09-15T23:36:33.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26e40cf740 1 -- start start 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d2eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 0x564fb1d48070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26e40cf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1aee40 con 0x564fb1d33370 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26e40cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1b0980 con 0x564fb1d29b40 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26e40cf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1af510 con 0x564fb1d285d0 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48008/0 (socket says 172.21.15.89:48008) 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26dbfff640 1 -- 172.21.15.89:0/3774438552 learned_addr learned my addr 172.21.15.89:0/3774438552 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26daffd640 1 --2- 172.21.15.89:0/3774438552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 0x564fb1d48070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26db7fe640 1 --2- 172.21.15.89:0/3774438552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d2eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26dbfff640 1 -- 172.21.15.89:0/3774438552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 msgr2=0x564fb1d2eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26dbfff640 1 --2- 172.21.15.89:0/3774438552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d2eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26dbfff640 1 -- 172.21.15.89:0/3774438552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 msgr2=0x564fb1d48070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26dbfff640 1 --2- 172.21.15.89:0/3774438552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 0x564fb1d48070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26dbfff640 1 -- 172.21.15.89:0/3774438552 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fb1d30950 con 0x564fb1d33370 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26db7fe640 1 --2- 172.21.15.89:0/3774438552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d2eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.746+0000 7f26dbfff640 1 --2- 172.21.15.89:0/3774438552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d485b0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f26d401d8a0 tx=0x7f26d4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953998951bbcbeae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26c17fa640 1 -- 172.21.15.89:0/3774438552 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d4003a30 con 0x564fb1d33370 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26c17fa640 1 -- 172.21.15.89:0/3774438552 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d400fe80 con 0x564fb1d33370 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26c17fa640 1 -- 172.21.15.89:0/3774438552 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d40047c0 con 0x564fb1d33370 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26e40cf740 1 -- 172.21.15.89:0/3774438552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 msgr2=0x564fb1d485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26e40cf740 1 --2- 172.21.15.89:0/3774438552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d485b0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f26d401d8a0 tx=0x7f26d4018000 comp rx=0 tx=0).stop 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26e40cf740 1 -- 172.21.15.89:0/3774438552 shutdown_connections 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26e40cf740 1 --2- 172.21.15.89:0/3774438552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d485b0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26e40cf740 1 --2- 172.21.15.89:0/3774438552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 0x564fb1d48070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26e40cf740 1 --2- 172.21.15.89:0/3774438552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d2eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26e40cf740 1 -- 172.21.15.89:0/3774438552 >> 172.21.15.89:0/3774438552 conn(0x564fb1cf1310 msgr2=0x564fb1d2e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26e40cf740 1 -- 172.21.15.89:0/3774438552 shutdown_connections 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.747+0000 7f26e40cf740 1 -- 172.21.15.89:0/3774438552 wait complete. 2024-09-15T23:36:33.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26e40cf740 1 -- start start 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26e40cf740 1 Processor -- start 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d45da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 0x564fb1d462e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26e40cf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d46820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26e40cf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1b02b0 con 0x564fb1d33370 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26e40cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1afbe0 con 0x564fb1d29b40 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26e40cf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1ace20 con 0x564fb1d285d0 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26e40cf740 10 client.0 _ll_register_callbacks cb 0x7f26cc0314d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d46820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26dbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d46820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:48012/0 (socket says 172.21.15.89:48012) 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26daffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 0x564fb1d462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26dbfff640 1 -- 172.21.15.89:0/1932901552 learned_addr learned my addr 172.21.15.89:0/1932901552 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:33.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.748+0000 7f26db7fe640 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d45da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26daffd640 1 -- 172.21.15.89:0/1932901552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 msgr2=0x564fb1d45da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26daffd640 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d45da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26daffd640 1 -- 172.21.15.89:0/1932901552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 msgr2=0x564fb1d46820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26daffd640 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d46820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26daffd640 1 -- 172.21.15.89:0/1932901552 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fb1dac5a0 con 0x564fb1d29b40 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26dbfff640 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d46820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26daffd640 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 0x564fb1d462e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f26d00ccf20 tx=0x7f26d00047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47366add6e6b8c2c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26baffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26baffd640 1 -- 172.21.15.89:0/1932901552 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d009b210 con 0x564fb1d29b40 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26baffd640 1 -- 172.21.15.89:0/1932901552 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f26d0097d40 con 0x564fb1d29b40 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26e40cf740 1 -- 172.21.15.89:0/1932901552 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564fb1c12170 con 0x564fb1d29b40 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26baffd640 1 -- 172.21.15.89:0/1932901552 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f26d009b4a0 con 0x564fb1d29b40 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26e40cf740 10 client.27791 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:33.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26e40cf740 1 -- 172.21.15.89:0/1932901552 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564fb1dac5a0 con 0x564fb1d29b40 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.749+0000 7f26baffd640 1 -- 172.21.15.89:0/1932901552 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564fb1c12170 con 0x564fb1d29b40 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26baffd640 1 -- 172.21.15.89:0/1932901552 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f26d00a3eb0 con 0x564fb1d29b40 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26e40cf740 10 client.27791 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26e40cf740 10 client.27791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26e40cf740 20 client.27791 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564fb1dadad0 0x564fb1dafef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26e40cf740 4 client.27791 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26e40cf740 1 -- 172.21.15.89:0/1932901552 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564fb1c5f9b0 con 0x564fb1dadad0 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26db7fe640 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564fb1dadad0 0x564fb1dafef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26db7fe640 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564fb1dadad0 0x564fb1dafef0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=385f4d6e102c68eb server_cookie=608d0f0e977c70f9 in_seq=0 out_seq=0 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26baffd640 10 client.27791 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26baffd640 1 -- 172.21.15.89:0/1932901552 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564fb1c5f9b0 con 0x564fb1dadad0 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26baffd640 10 client.27791 handle_command_reply: tid=0 2024-09-15T23:36:33.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.750+0000 7f26e40cf740 1 client.27791 shutdown 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 1 -- 172.21.15.89:0/1932901552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 msgr2=0x564fb1d462e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 0x564fb1d462e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f26d00ccf20 tx=0x7f26d00047d0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 1 -- 172.21.15.89:0/1932901552 shutdown_connections 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564fb1dadad0 0x564fb1dafef0 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564fb1d33370 0x564fb1d46820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26baffd640 0 client.27791 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fb1d29b40 0x564fb1d462e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 1 --2- 172.21.15.89:0/1932901552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564fb1d285d0 0x564fb1d45da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 1 -- 172.21.15.89:0/1932901552 >> 172.21.15.89:0/1932901552 conn(0x564fb1cf1310 msgr2=0x564fb1d2a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 1 -- 172.21.15.89:0/1932901552 shutdown_connections 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 1 -- 172.21.15.89:0/1932901552 wait complete. 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e40cf740 20 client.27791 trim_cache size 0 max 16384 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 -- 172.21.15.89:0/381417179 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f26c40a3240 msgr2=0x7f26c40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:33.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f26c40a3240 0x7f26c40a5700 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f26d4007fa0 tx=0x7f26d4007520 comp rx=0 tx=0).stop 2024-09-15T23:36:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 -- 172.21.15.89:0/381417179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26dc103660 msgr2=0x7f26dc198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26dc103660 0x7f26dc198990 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b3f0 tx=0x7f26d000b8b0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 -- 172.21.15.89:0/381417179 shutdown_connections 2024-09-15T23:36:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f26c40a3240 0x7f26c40a5700 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26dc10c170 0x7f26dc19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26dc103660 0x7f26dc198990 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 --2- 172.21.15.89:0/381417179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26dc102d40 0x7f26dc198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 -- 172.21.15.89:0/381417179 >> 172.21.15.89:0/381417179 conn(0x7f26dc0fc7a0 msgr2=0x7f26dc0fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 -- 172.21.15.89:0/381417179 shutdown_connections 2024-09-15T23:36:33.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:33.751+0000 7f26e1cf6640 1 -- 172.21.15.89:0/381417179 wait complete. 2024-09-15T23:36:34.734 INFO:tasks.workunit.client.0.smithi089.stdout:375/600: open fd = 3 2024-09-15T23:36:34.734 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.734 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.734 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.734 INFO:tasks.workunit.client.0.smithi089.stdout:376/600: open fd = 3 2024-09-15T23:36:34.734 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.734 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:377/600: open fd = 3 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:378/600: open fd = 3 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:379/600: open fd = 3 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.735 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:380/600: open fd = 3 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:381/600: open fd = 3 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:382/600: open fd = 3 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:383/600: open fd = 3 2024-09-15T23:36:34.736 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:384/600: open fd = 3 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:385/600: open fd = 3 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:386/600: open fd = 3 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.737 INFO:tasks.workunit.client.0.smithi089.stdout:387/600: open fd = 3 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:388/600: open fd = 3 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:389/600: open fd = 3 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:390/600: open fd = 3 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.738 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:391/600: open fd = 3 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:392/600: open fd = 3 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:393/600: open fd = 3 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.739 INFO:tasks.workunit.client.0.smithi089.stdout:394/600: open fd = 3 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:395/600: open fd = 3 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:396/600: open fd = 3 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:397/600: open fd = 3 2024-09-15T23:36:34.740 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.741 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.741 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.741 INFO:tasks.workunit.client.0.smithi089.stdout:398/600: open fd = 3 2024-09-15T23:36:34.741 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.741 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.741 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.741 INFO:tasks.workunit.client.0.smithi089.stdout:399/600: open fd = 3 2024-09-15T23:36:34.741 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.741 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.741 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:400/600: open fd = 3 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:401/600: open fd = 3 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:402/600: open fd = 3 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:403/600: open fd = 3 2024-09-15T23:36:34.742 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:404/600: open fd = 3 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:405/600: open fd = 3 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:406/600: open fd = 3 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.743 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:407/600: open fd = 3 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:408/600: open fd = 3 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:409/600: open fd = 3 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:410/600: open fd = 3 2024-09-15T23:36:34.744 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:411/600: open fd = 3 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:412/600: open fd = 3 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:413/600: open fd = 3 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.745 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:414/600: open fd = 3 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:415/600: open fd = 3 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:416/600: open fd = 3 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.746 INFO:tasks.workunit.client.0.smithi089.stdout:417/600: open fd = 3 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:418/600: open fd = 3 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:419/600: open fd = 3 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:420/600: open fd = 3 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.747 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:421/600: open fd = 3 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:422/600: open fd = 3 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:423/600: open fd = 3 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.748 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:424/600: open fd = 3 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:425/600: open fd = 3 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:426/600: open fd = 3 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:427/600: open fd = 3 2024-09-15T23:36:34.749 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:428/600: open fd = 3 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:429/600: open fd = 3 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:430/600: open fd = 3 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.750 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:431/600: open fd = 3 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:432/600: open fd = 3 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:433/600: open fd = 3 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:434/600: open fd = 3 2024-09-15T23:36:34.751 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.752 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.752 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.752 INFO:tasks.workunit.client.0.smithi089.stdout:435/600: open fd = 3 2024-09-15T23:36:34.752 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.752 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.752 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.752 INFO:tasks.workunit.client.0.smithi089.stdout:436/600: open fd = 3 2024-09-15T23:36:34.752 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.752 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:36:34.752 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:36:34.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:34 smithi120 ceph-mon[35250]: pgmap v726: 129 pgs: 129 active+clean; 2.1 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:36:34.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:34 smithi174 ceph-mon[35586]: pgmap v726: 129 pgs: 129 active+clean; 2.1 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:36:35.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:34 smithi089 ceph-mon[30933]: pgmap v726: 129 pgs: 129 active+clean; 2.1 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:36:35.465 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:36:35.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:36:35.728 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.186+0000 7fbb821bb640 1 -- 172.21.15.89:0/1548369010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb7c1008e0 msgr2=0x7fbb7c100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.186+0000 7fbb821bb640 1 --2- 172.21.15.89:0/1548369010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb7c1008e0 0x7fbb7c100cc0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009ca0 tx=0x7fbb6c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.186+0000 7fbb821bb640 1 -- 172.21.15.89:0/1548369010 shutdown_connections 2024-09-15T23:36:36.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.186+0000 7fbb821bb640 1 --2- 172.21.15.89:0/1548369010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb7c108110 0x7fbb7c10ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.186+0000 7fbb821bb640 1 --2- 172.21.15.89:0/1548369010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb7c101290 0x7fbb7c1079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.186+0000 7fbb821bb640 1 --2- 172.21.15.89:0/1548369010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb7c1008e0 0x7fbb7c100cc0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.186+0000 7fbb821bb640 1 -- 172.21.15.89:0/1548369010 >> 172.21.15.89:0/1548369010 conn(0x7fbb7c077210 msgr2=0x7fbb7c077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.186+0000 7fbb821bb640 1 -- 172.21.15.89:0/1548369010 shutdown_connections 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.186+0000 7fbb821bb640 1 -- 172.21.15.89:0/1548369010 wait complete. 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.186+0000 7fbb821bb640 1 Processor -- start 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb821bb640 1 -- start start 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb821bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb7c1008e0 0x7fbb7c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb821bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb7c101290 0x7fbb7c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb821bb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb7c108110 0x7fbb7c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb821bb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c111ff0 con 0x7fbb7c1008e0 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb821bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c111e70 con 0x7fbb7c101290 2024-09-15T23:36:36.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb821bb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c112170 con 0x7fbb7c108110 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb7b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb7c1008e0 0x7fbb7c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb7b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb7c1008e0 0x7fbb7c198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50422/0 (socket says 172.21.15.89:50422) 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb7b7fe640 1 -- 172.21.15.89:0/3030182460 learned_addr learned my addr 172.21.15.89:0/3030182460 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb7bfff640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb7c108110 0x7fbb7c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb7affd640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb7c101290 0x7fbb7c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb7b7fe640 1 -- 172.21.15.89:0/3030182460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb7c108110 msgr2=0x7fbb7c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb7c108110 0x7fbb7c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb7b7fe640 1 -- 172.21.15.89:0/3030182460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb7c101290 msgr2=0x7fbb7c198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb7c101290 0x7fbb7c198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.187+0000 7fbb7b7fe640 1 -- 172.21.15.89:0/3030182460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7c19d400 con 0x7fbb7c1008e0 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.188+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb7c1008e0 0x7fbb7c198450 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0097b0 tx=0x7fbb6c031b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.188+0000 7fbb78ff9640 1 -- 172.21.15.89:0/3030182460 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c02fba0 con 0x7fbb7c1008e0 2024-09-15T23:36:36.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.188+0000 7fbb821bb640 1 -- 172.21.15.89:0/3030182460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7c19d690 con 0x7fbb7c1008e0 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.188+0000 7fbb78ff9640 1 -- 172.21.15.89:0/3030182460 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb6c02fd40 con 0x7fbb7c1008e0 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.188+0000 7fbb78ff9640 1 -- 172.21.15.89:0/3030182460 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c03b860 con 0x7fbb7c1008e0 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.188+0000 7fbb821bb640 1 -- 172.21.15.89:0/3030182460 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb7c1a4fd0 con 0x7fbb7c1008e0 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.190+0000 7fbb78ff9640 1 -- 172.21.15.89:0/3030182460 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbb6c030c30 con 0x7fbb7c1008e0 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.190+0000 7fbb84594740 1 Processor -- start 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.190+0000 7fbb84594740 1 -- start start 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb84594740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded1f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb84594740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded204c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb84594740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded207540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb84594740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564ded15b280 con 0x564ded1f5a20 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb84594740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564ded102b10 con 0x564ded205150 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb84594740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564ded157240 con 0x564ded1f6340 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded1f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded1f5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50430/0 (socket says 172.21.15.89:50430) 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7bfff640 1 -- 172.21.15.89:0/3195780758 learned_addr learned my addr 172.21.15.89:0/3195780758 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb78ff9640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb640a3240 0x7fbb640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded204c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7affd640 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded207540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7bfff640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb640a3240 0x7fbb640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb78ff9640 1 -- 172.21.15.89:0/3030182460 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb6c0f1380 con 0x7fbb7c1008e0 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7bfff640 1 -- 172.21.15.89:0/3195780758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 msgr2=0x564ded204c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7bfff640 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded204c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7bfff640 1 -- 172.21.15.89:0/3195780758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 msgr2=0x564ded207540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7bfff640 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded207540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7bfff640 1 -- 172.21.15.89:0/3195780758 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ded10f1f0 con 0x564ded1f5a20 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded204c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:36.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7affd640 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded207540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.191+0000 7fbb7bfff640 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded1f5e00 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fbb74013820 tx=0x7fbb74013ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c729d0ddf96c1a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb61ffb640 1 -- 172.21.15.89:0/3195780758 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74021020 con 0x564ded1f5a20 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb7bfff640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb640a3240 0x7fbb640a5700 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbb74022000 tx=0x7fbb74020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb61ffb640 1 -- 172.21.15.89:0/3195780758 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb7400acc0 con 0x564ded1f5a20 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb61ffb640 1 -- 172.21.15.89:0/3195780758 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7400eb60 con 0x564ded1f5a20 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb84594740 1 -- 172.21.15.89:0/3195780758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 msgr2=0x564ded1f5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb84594740 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded1f5e00 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fbb74013820 tx=0x7fbb74013ce0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb84594740 1 -- 172.21.15.89:0/3195780758 shutdown_connections 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb84594740 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded207540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb84594740 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded204c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb84594740 1 --2- 172.21.15.89:0/3195780758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded1f5e00 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb84594740 1 -- 172.21.15.89:0/3195780758 >> 172.21.15.89:0/3195780758 conn(0x564ded1f20f0 msgr2=0x564ded1f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb84594740 1 -- 172.21.15.89:0/3195780758 shutdown_connections 2024-09-15T23:36:36.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.192+0000 7fbb84594740 1 -- 172.21.15.89:0/3195780758 wait complete. 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb84594740 1 -- start start 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb84594740 1 Processor -- start 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb84594740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f5a20 0x564ded201fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb84594740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f6340 0x564ded202520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb84594740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded202a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb84594740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564ded207d20 con 0x564ded1f6340 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb84594740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564ded115f10 con 0x564ded205150 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb84594740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564ded207ea0 con 0x564ded1f5a20 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb84594740 10 client.0 _ll_register_callbacks cb 0x564ded0dea40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f6340 0x564ded202520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb7affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f6340 0x564ded202520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50432/0 (socket says 172.21.15.89:50432) 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb7affd640 1 -- 172.21.15.89:0/464557413 learned_addr learned my addr 172.21.15.89:0/464557413 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:36.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.193+0000 7fbb7bfff640 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded202a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f5a20 0x564ded201fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb7affd640 1 -- 172.21.15.89:0/464557413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f5a20 msgr2=0x564ded201fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb7affd640 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f5a20 0x564ded201fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb7affd640 1 -- 172.21.15.89:0/464557413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 msgr2=0x564ded202a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb7affd640 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded202a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb7affd640 1 -- 172.21.15.89:0/464557413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dececc6a0 con 0x564ded1f6340 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb7affd640 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f6340 0x564ded202520 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fbb700104a0 tx=0x7fbb70010960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa77c9c632e4b9ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb5affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb5affd640 1 -- 172.21.15.89:0/464557413 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70018070 con 0x564ded1f6340 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb5affd640 1 -- 172.21.15.89:0/464557413 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb70014800 con 0x564ded1f6340 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb84594740 1 -- 172.21.15.89:0/464557413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb7400acc0 con 0x564ded1f6340 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb5affd640 1 -- 172.21.15.89:0/464557413 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb70024050 con 0x564ded1f6340 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb84594740 10 client.19791 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:36.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb84594740 1 -- 172.21.15.89:0/464557413 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ded2138e0 con 0x564ded1f6340 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb5affd640 1 -- 172.21.15.89:0/464557413 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb7400acc0 con 0x564ded1f6340 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb5affd640 1 -- 172.21.15.89:0/464557413 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbb7001cc20 con 0x564ded1f6340 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb84594740 10 client.19791 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb84594740 10 client.19791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.194+0000 7fbb84594740 20 client.19791 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.195+0000 7fbb84594740 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ded214d90 0x564ded2171b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.195+0000 7fbb84594740 4 client.19791 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.195+0000 7fbb84594740 1 -- 172.21.15.89:0/464557413 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ded1687f0 con 0x564ded214d90 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.195+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ded214d90 0x564ded2171b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.195+0000 7fbb5affd640 10 client.19791 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.195+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ded214d90 0x564ded2171b0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b1b014be77a0a7d server_cookie=4bcc88c0931b5d7e in_seq=0 out_seq=0 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb5affd640 1 -- 172.21.15.89:0/464557413 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564ded1687f0 con 0x564ded214d90 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb5affd640 10 client.19791 handle_command_reply: tid=0 2024-09-15T23:36:36.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 client.19791 shutdown 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 -- 172.21.15.89:0/464557413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f6340 msgr2=0x564ded202520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f6340 0x564ded202520 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fbb700104a0 tx=0x7fbb70010960 comp rx=0 tx=0).stop 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 -- 172.21.15.89:0/464557413 shutdown_connections 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ded214d90 0x564ded2171b0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded202a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f6340 0x564ded202520 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 --2- 172.21.15.89:0/464557413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f5a20 0x564ded201fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb5affd640 0 client.19791 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 -- 172.21.15.89:0/464557413 >> 172.21.15.89:0/464557413 conn(0x564ded1f20f0 msgr2=0x564ded1f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 -- 172.21.15.89:0/464557413 shutdown_connections 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.196+0000 7fbb84594740 1 -- 172.21.15.89:0/464557413 wait complete. 2024-09-15T23:36:36.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.197+0000 7fbb84594740 20 client.19791 trim_cache size 0 max 16384 2024-09-15T23:36:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb84594740 1 Processor -- start 2024-09-15T23:36:36.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb84594740 1 -- start start 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb84594740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded200e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb84594740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded2028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb84594740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded21ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb84594740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1acb60 con 0x564ded1f5a20 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb84594740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1ae6a0 con 0x564ded205150 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb84594740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1ad230 con 0x564ded1f6340 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb7b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded200e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb7b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded200e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50436/0 (socket says 172.21.15.89:50436) 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb7b7fe640 1 -- 172.21.15.89:0/1466547802 learned_addr learned my addr 172.21.15.89:0/1466547802 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb7affd640 1 --2- 172.21.15.89:0/1466547802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded2028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb7b7fe640 1 -- 172.21.15.89:0/1466547802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 msgr2=0x564ded2028b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/1466547802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded2028b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb7b7fe640 1 -- 172.21.15.89:0/1466547802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 msgr2=0x564ded21ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:36.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/1466547802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded21ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb7b7fe640 1 -- 172.21.15.89:0/1466547802 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ded207c00 con 0x564ded1f5a20 2024-09-15T23:36:36.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/1466547802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded200e90 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0b9f80 tx=0x7fbb6c033040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5015e44e5a60a56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:36.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb617fa640 1 -- 172.21.15.89:0/1466547802 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0bd5b0 con 0x564ded1f5a20 2024-09-15T23:36:36.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb617fa640 1 -- 172.21.15.89:0/1466547802 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb6c036040 con 0x564ded1f5a20 2024-09-15T23:36:36.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.214+0000 7fbb617fa640 1 -- 172.21.15.89:0/1466547802 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0bb8b0 con 0x564ded1f5a20 2024-09-15T23:36:36.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.215+0000 7fbb84594740 1 -- 172.21.15.89:0/1466547802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 msgr2=0x564ded200e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:36.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.215+0000 7fbb84594740 1 --2- 172.21.15.89:0/1466547802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded200e90 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0b9f80 tx=0x7fbb6c033040 comp rx=0 tx=0).stop 2024-09-15T23:36:36.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.215+0000 7fbb84594740 1 -- 172.21.15.89:0/1466547802 shutdown_connections 2024-09-15T23:36:36.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.215+0000 7fbb84594740 1 --2- 172.21.15.89:0/1466547802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded205150 0x564ded21ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.215+0000 7fbb84594740 1 --2- 172.21.15.89:0/1466547802 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded2028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.215+0000 7fbb84594740 1 --2- 172.21.15.89:0/1466547802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded1f5a20 0x564ded200e90 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.215+0000 7fbb84594740 1 -- 172.21.15.89:0/1466547802 >> 172.21.15.89:0/1466547802 conn(0x564ded1f20f0 msgr2=0x564ded200780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:36.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.215+0000 7fbb84594740 1 -- 172.21.15.89:0/1466547802 shutdown_connections 2024-09-15T23:36:36.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.215+0000 7fbb84594740 1 -- 172.21.15.89:0/1466547802 wait complete. 2024-09-15T23:36:36.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.215+0000 7fbb84594740 1 -- start start 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb84594740 1 Processor -- start 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb84594740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded1f5a20 0x564ded208fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb84594740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded209500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb84594740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded205150 0x564ded209a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb84594740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1adfd0 con 0x564ded205150 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb84594740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1ad900 con 0x564ded1f5a20 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb84594740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c1aab40 con 0x564ded1f6340 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb84594740 10 client.0 _ll_register_callbacks cb 0x7fbb700093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb7bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded205150 0x564ded209a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb7bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded205150 0x564ded209a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50450/0 (socket says 172.21.15.89:50450) 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb7bfff640 1 -- 172.21.15.89:0/381847998 learned_addr learned my addr 172.21.15.89:0/381847998 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded1f5a20 0x564ded208fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb7affd640 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded209500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb7bfff640 1 -- 172.21.15.89:0/381847998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 msgr2=0x564ded209500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb7bfff640 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded209500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb7bfff640 1 -- 172.21.15.89:0/381847998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded1f5a20 msgr2=0x564ded208fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb7bfff640 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded1f5a20 0x564ded208fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.216+0000 7fbb7bfff640 1 -- 172.21.15.89:0/381847998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ded27eb90 con 0x564ded205150 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb7bfff640 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded205150 0x564ded209a40 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400e7f0 tx=0x7fbb74013460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca63489f4ef8595 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb5affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb5affd640 1 -- 172.21.15.89:0/381847998 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74021040 con 0x564ded205150 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb5affd640 1 -- 172.21.15.89:0/381847998 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb7400e890 con 0x564ded205150 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb84594740 1 -- 172.21.15.89:0/381847998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb6c036040 con 0x564ded205150 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb5affd640 1 -- 172.21.15.89:0/381847998 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb7403a050 con 0x564ded205150 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb84594740 10 client.19797 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb84594740 1 -- 172.21.15.89:0/381847998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ded27fe70 con 0x564ded205150 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb5affd640 1 -- 172.21.15.89:0/381847998 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb6c036040 con 0x564ded205150 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb5affd640 1 -- 172.21.15.89:0/381847998 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fbb740052e0 con 0x564ded205150 2024-09-15T23:36:36.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb84594740 10 client.19797 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb84594740 10 client.19797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb84594740 20 client.19797 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb84594740 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ded280300 0x564ded282720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb84594740 4 client.19797 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.217+0000 7fbb84594740 1 -- 172.21.15.89:0/381847998 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564ded162d00 con 0x564ded280300 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.218+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ded280300 0x564ded282720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.218+0000 7fbb5affd640 10 client.19797 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.218+0000 7fbb7b7fe640 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ded280300 0x564ded282720 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca573b58a098d11f server_cookie=f33d15dd11dc3485 in_seq=0 out_seq=0 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.219+0000 7fbb5affd640 1 -- 172.21.15.89:0/381847998 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564ded162d00 con 0x564ded280300 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.219+0000 7fbb5affd640 10 client.19797 handle_command_reply: tid=0 2024-09-15T23:36:36.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.219+0000 7fbb84594740 1 client.19797 shutdown 2024-09-15T23:36:36.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.219+0000 7fbb84594740 1 -- 172.21.15.89:0/381847998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded205150 msgr2=0x564ded209a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:36.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.219+0000 7fbb84594740 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded205150 0x564ded209a40 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400e7f0 tx=0x7fbb74013460 comp rx=0 tx=0).stop 2024-09-15T23:36:36.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.219+0000 7fbb84594740 1 -- 172.21.15.89:0/381847998 shutdown_connections 2024-09-15T23:36:36.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.219+0000 7fbb84594740 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564ded280300 0x564ded282720 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.219+0000 7fbb84594740 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ded205150 0x564ded209a40 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb84594740 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ded1f6340 0x564ded209500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb84594740 1 --2- 172.21.15.89:0/381847998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ded1f5a20 0x564ded208fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb84594740 1 -- 172.21.15.89:0/381847998 >> 172.21.15.89:0/381847998 conn(0x564ded1f20f0 msgr2=0x564ded1f7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb5affd640 0 client.19797 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb84594740 1 -- 172.21.15.89:0/381847998 shutdown_connections 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb84594740 1 -- 172.21.15.89:0/381847998 wait complete. 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb84594740 20 client.19797 trim_cache size 0 max 16384 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "6fb20ded-9bd2-45ba-b7e1-5dfdd55d6f90", 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:36:36.225 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:36:36.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 -- 172.21.15.89:0/3030182460 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb640a3240 msgr2=0x7fbb640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:36.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb640a3240 0x7fbb640a5700 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbb74022000 tx=0x7fbb74020000 comp rx=0 tx=0).stop 2024-09-15T23:36:36.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 -- 172.21.15.89:0/3030182460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb7c1008e0 msgr2=0x7fbb7c198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:36.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb7c1008e0 0x7fbb7c198450 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0097b0 tx=0x7fbb6c031b10 comp rx=0 tx=0).stop 2024-09-15T23:36:36.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 -- 172.21.15.89:0/3030182460 shutdown_connections 2024-09-15T23:36:36.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb640a3240 0x7fbb640a5700 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb7c108110 0x7fbb7c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb7c101290 0x7fbb7c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 --2- 172.21.15.89:0/3030182460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb7c1008e0 0x7fbb7c198450 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:36.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 -- 172.21.15.89:0/3030182460 >> 172.21.15.89:0/3030182460 conn(0x7fbb7c077210 msgr2=0x7fbb7c105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:36.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.220+0000 7fbb821bb640 1 -- 172.21.15.89:0/3030182460 shutdown_connections 2024-09-15T23:36:36.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:36.221+0000 7fbb821bb640 1 -- 172.21.15.89:0/3030182460 wait complete. 2024-09-15T23:36:36.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:36 smithi120 ceph-mon[35250]: pgmap v727: 129 pgs: 129 active+clean; 2.1 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:36:36.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:36 smithi174 ceph-mon[35586]: pgmap v727: 129 pgs: 129 active+clean; 2.1 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:36:37.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:36 smithi089 ceph-mon[30933]: pgmap v727: 129 pgs: 129 active+clean; 2.1 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:36:37.128 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:36:37.375 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:36:37.771 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:37 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:36:37.771 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:37 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:37.771 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:37 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:36:37.771 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:37 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:37.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.852+0000 7fa74c1f8640 1 -- 172.21.15.89:0/1652769710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa744101de0 msgr2=0x7fa7441021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:37.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.852+0000 7fa74c1f8640 1 --2- 172.21.15.89:0/1652769710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa744101de0 0x7fa7441021c0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fa740009900 tx=0x7fa74002a410 comp rx=0 tx=0).stop 2024-09-15T23:36:37.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.852+0000 7fa74c1f8640 1 -- 172.21.15.89:0/1652769710 shutdown_connections 2024-09-15T23:36:37.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.852+0000 7fa74c1f8640 1 --2- 172.21.15.89:0/1652769710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa744106f10 0x7fa74410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.852+0000 7fa74c1f8640 1 --2- 172.21.15.89:0/1652769710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa744102790 0x7fa7441067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.852+0000 7fa74c1f8640 1 --2- 172.21.15.89:0/1652769710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa744101de0 0x7fa7441021c0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.852+0000 7fa74c1f8640 1 -- 172.21.15.89:0/1652769710 >> 172.21.15.89:0/1652769710 conn(0x7fa7440fd8c0 msgr2=0x7fa7440ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.852+0000 7fa74c1f8640 1 -- 172.21.15.89:0/1652769710 shutdown_connections 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.852+0000 7fa74c1f8640 1 -- 172.21.15.89:0/1652769710 wait complete. 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa74c1f8640 1 Processor -- start 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa74c1f8640 1 -- start start 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa74c1f8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa744101de0 0x7fa7441985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa74c1f8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa744102790 0x7fa744198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa74c1f8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa744106f10 0x7fa74419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa74c1f8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa744112010 con 0x7fa744102790 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa74c1f8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa744111e90 con 0x7fa744101de0 2024-09-15T23:36:37.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa74c1f8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa744112190 con 0x7fa744106f10 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa749793640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa744102790 0x7fa744198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa749793640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa744102790 0x7fa744198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50468/0 (socket says 172.21.15.89:50468) 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa749793640 1 -- 172.21.15.89:0/2087194480 learned_addr learned my addr 172.21.15.89:0/2087194480 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa749f94640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa744101de0 0x7fa7441985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa749793640 1 -- 172.21.15.89:0/2087194480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa744106f10 msgr2=0x7fa74419ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.853+0000 7fa74a795640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa744106f10 0x7fa74419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa749793640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa744106f10 0x7fa74419ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa749793640 1 -- 172.21.15.89:0/2087194480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa744101de0 msgr2=0x7fa7441985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa749793640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa744101de0 0x7fa7441985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa749793640 1 -- 172.21.15.89:0/2087194480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74419d560 con 0x7fa744102790 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa74a795640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa744106f10 0x7fa74419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa749793640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa744102790 0x7fa744198af0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fa73400b4a0 tx=0x7fa73400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:37.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa73affd640 1 -- 172.21.15.89:0/2087194480 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa734014070 con 0x7fa744102790 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa74c1f8640 1 -- 172.21.15.89:0/2087194480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa74419d850 con 0x7fa744102790 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa73affd640 1 -- 172.21.15.89:0/2087194480 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa734010af0 con 0x7fa744102790 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa73affd640 1 -- 172.21.15.89:0/2087194480 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa734017d10 con 0x7fa744102790 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.854+0000 7fa74c1f8640 1 -- 172.21.15.89:0/2087194480 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7441a5130 con 0x7fa744102790 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.856+0000 7fa73affd640 1 -- 172.21.15.89:0/2087194480 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa734017070 con 0x7fa744102790 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.856+0000 7fa74e5d1740 1 Processor -- start 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa74e5d1740 1 -- start start 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a94a0 0x55a2f35a9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35b4680 0x55a2f35b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a2f350f2e0 con 0x55a2f35a9dc0 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a2f34b6ce0 con 0x55a2f35a94a0 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a2f350b2a0 con 0x55a2f35b4680 2024-09-15T23:36:37.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa749f94640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa749f94640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50476/0 (socket says 172.21.15.89:50476) 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa749f94640 1 -- 172.21.15.89:0/3894212248 learned_addr learned my addr 172.21.15.89:0/3894212248 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa73affd640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7300a3240 0x7fa7300a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa749793640 1 --2- 172.21.15.89:0/3894212248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35b4680 0x55a2f35b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa74a795640 1 --2- 172.21.15.89:0/3894212248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a94a0 0x55a2f35a9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa749f94640 1 -- 172.21.15.89:0/3894212248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35b4680 msgr2=0x55a2f35b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa749f94640 1 --2- 172.21.15.89:0/3894212248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35b4680 0x55a2f35b6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa749f94640 1 -- 172.21.15.89:0/3894212248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a94a0 msgr2=0x55a2f35a9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa749f94640 1 --2- 172.21.15.89:0/3894212248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a94a0 0x55a2f35a9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa749f94640 1 -- 172.21.15.89:0/3894212248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2f349b440 con 0x55a2f35a9dc0 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa74a795640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7300a3240 0x7fa7300a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa73affd640 1 -- 172.21.15.89:0/2087194480 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa7340cf1f0 con 0x7fa744102790 2024-09-15T23:36:37.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.857+0000 7fa749f94640 1 --2- 172.21.15.89:0/3894212248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b4140 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fa740005ab0 tx=0x7fa740031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4666ce3b75415e83 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa72ffff640 1 -- 172.21.15.89:0/3894212248 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74003f070 con 0x55a2f35a9dc0 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa72ffff640 1 -- 172.21.15.89:0/3894212248 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa740031a20 con 0x55a2f35a9dc0 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa72ffff640 1 -- 172.21.15.89:0/3894212248 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa740035070 con 0x55a2f35a9dc0 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa74e5d1740 1 -- 172.21.15.89:0/3894212248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 msgr2=0x55a2f35b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa74a795640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7300a3240 0x7fa7300a5700 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa73c007fb0 tx=0x7fa73c004b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/3894212248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b4140 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fa740005ab0 tx=0x7fa740031630 comp rx=0 tx=0).stop 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa74e5d1740 1 -- 172.21.15.89:0/3894212248 shutdown_connections 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/3894212248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35b4680 0x55a2f35b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/3894212248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b4140 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/3894212248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a94a0 0x55a2f35a9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.858+0000 7fa74e5d1740 1 -- 172.21.15.89:0/3894212248 >> 172.21.15.89:0/3894212248 conn(0x55a2f35a5a30 msgr2=0x55a2f35a5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 1 -- 172.21.15.89:0/3894212248 shutdown_connections 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 1 -- 172.21.15.89:0/3894212248 wait complete. 2024-09-15T23:36:37.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 1 -- start start 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 1 Processor -- start 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a94a0 0x55a2f35b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35b4680 0x55a2f35b1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55a2f35b7250 con 0x55a2f35a9dc0 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a2f34ca050 con 0x55a2f35b4680 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a2f35b73d0 con 0x55a2f35a94a0 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa74e5d1740 10 client.0 _ll_register_callbacks cb 0x55a2f350a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.859+0000 7fa749793640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa749793640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50484/0 (socket says 172.21.15.89:50484) 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa749793640 1 -- 172.21.15.89:0/1428702600 learned_addr learned my addr 172.21.15.89:0/1428702600 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa74a795640 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35b4680 0x55a2f35b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa749f94640 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a94a0 0x55a2f35b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa749793640 1 -- 172.21.15.89:0/1428702600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a94a0 msgr2=0x55a2f35b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa749793640 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a94a0 0x55a2f35b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa749793640 1 -- 172.21.15.89:0/1428702600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35b4680 msgr2=0x55a2f35b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa749793640 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35b4680 0x55a2f35b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa749793640 1 -- 172.21.15.89:0/1428702600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2f32803c0 con 0x55a2f35a9dc0 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa749793640 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b1220 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fa73400be60 tx=0x7fa73409ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a93e060fa92349b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa72d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa72d7fa640 1 -- 172.21.15.89:0/1428702600 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73401e070 con 0x55a2f35a9dc0 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa72d7fa640 1 -- 172.21.15.89:0/1428702600 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa734017360 con 0x55a2f35a9dc0 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa74e5d1740 1 -- 172.21.15.89:0/1428702600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa740031a20 con 0x55a2f35a9dc0 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa72d7fa640 1 -- 172.21.15.89:0/1428702600 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa73409b420 con 0x55a2f35a9dc0 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa74e5d1740 10 client.19809 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa74e5d1740 1 -- 172.21.15.89:0/1428702600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a2f35c2c70 con 0x55a2f35a9dc0 2024-09-15T23:36:37.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.860+0000 7fa72d7fa640 1 -- 172.21.15.89:0/1428702600 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa740031a20 con 0x55a2f35a9dc0 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.861+0000 7fa72d7fa640 1 -- 172.21.15.89:0/1428702600 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa734097a90 con 0x55a2f35a9dc0 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.861+0000 7fa74e5d1740 10 client.19809 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.861+0000 7fa74e5d1740 10 client.19809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.861+0000 7fa74e5d1740 20 client.19809 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.861+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a2f35c4180 0x55a2f35c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.861+0000 7fa74e5d1740 4 client.19809 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.861+0000 7fa74e5d1740 1 -- 172.21.15.89:0/1428702600 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2f351c850 con 0x55a2f35c4180 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.861+0000 7fa749f94640 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a2f35c4180 0x55a2f35c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.861+0000 7fa72d7fa640 10 client.19809 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.861+0000 7fa749f94640 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a2f35c4180 0x55a2f35c65d0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2dec4d73321afd0 server_cookie=c630152f64c0885d in_seq=0 out_seq=0 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa72d7fa640 1 -- 172.21.15.89:0/1428702600 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a2f351c850 con 0x55a2f35c4180 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa72d7fa640 10 client.19809 handle_command_reply: tid=0 2024-09-15T23:36:37.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 client.19809 shutdown 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 -- 172.21.15.89:0/1428702600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 msgr2=0x55a2f35b1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b1220 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fa73400be60 tx=0x7fa73409ad80 comp rx=0 tx=0).stop 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 -- 172.21.15.89:0/1428702600 shutdown_connections 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a2f35c4180 0x55a2f35c65d0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35b4680 0x55a2f35b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a9dc0 0x55a2f35b1220 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/1428702600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a94a0 0x55a2f35b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa72d7fa640 0 client.19809 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 -- 172.21.15.89:0/1428702600 >> 172.21.15.89:0/1428702600 conn(0x55a2f35a5a30 msgr2=0x55a2f35ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 -- 172.21.15.89:0/1428702600 shutdown_connections 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 1 -- 172.21.15.89:0/1428702600 wait complete. 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.862+0000 7fa74e5d1740 20 client.19809 trim_cache size 0 max 16384 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa74e5d1740 1 Processor -- start 2024-09-15T23:36:37.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa74e5d1740 1 -- start start 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a5a30 0x55a2f35afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a94a0 0x55a2f35cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a9dc0 0x55a2f35cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441aee90 con 0x55a2f35a94a0 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441b09d0 con 0x55a2f35a5a30 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441af560 con 0x55a2f35a9dc0 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa749793640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a94a0 0x55a2f35cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa749793640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a94a0 0x55a2f35cbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50490/0 (socket says 172.21.15.89:50490) 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.879+0000 7fa749793640 1 -- 172.21.15.89:0/2420355197 learned_addr learned my addr 172.21.15.89:0/2420355197 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa749793640 1 -- 172.21.15.89:0/2420355197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a9dc0 msgr2=0x55a2f35cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa749793640 1 --2- 172.21.15.89:0/2420355197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a9dc0 0x55a2f35cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa749793640 1 -- 172.21.15.89:0/2420355197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a5a30 msgr2=0x55a2f35afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:37.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa749793640 1 --2- 172.21.15.89:0/2420355197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a5a30 0x55a2f35afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa749793640 1 -- 172.21.15.89:0/2420355197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2f35a5860 con 0x55a2f35a94a0 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa749793640 1 --2- 172.21.15.89:0/2420355197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a94a0 0x55a2f35cbea0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fa734097880 tx=0x7fa73409aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4716e3ad46ccc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa72f7fe640 1 -- 172.21.15.89:0/2420355197 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73401e070 con 0x55a2f35a94a0 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa72f7fe640 1 -- 172.21.15.89:0/2420355197 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa73409c750 con 0x55a2f35a94a0 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa72f7fe640 1 -- 172.21.15.89:0/2420355197 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa734010ca0 con 0x55a2f35a94a0 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa74e5d1740 1 -- 172.21.15.89:0/2420355197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a94a0 msgr2=0x55a2f35cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/2420355197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a94a0 0x55a2f35cbea0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fa734097880 tx=0x7fa73409aeb0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa74e5d1740 1 -- 172.21.15.89:0/2420355197 shutdown_connections 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/2420355197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a9dc0 0x55a2f35cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/2420355197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a94a0 0x55a2f35cbea0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/2420355197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a5a30 0x55a2f35afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa74e5d1740 1 -- 172.21.15.89:0/2420355197 >> 172.21.15.89:0/2420355197 conn(0x55a2f35b18c0 msgr2=0x55a2f35a4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:37.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa74e5d1740 1 -- 172.21.15.89:0/2420355197 shutdown_connections 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.880+0000 7fa74e5d1740 1 -- 172.21.15.89:0/2420355197 wait complete. 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa74e5d1740 1 -- start start 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa74e5d1740 1 Processor -- start 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a5a30 0x55a2f35ba7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a94a0 0x55a2f35bacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa74e5d1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a9dc0 0x55a2f362de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441b0300 con 0x55a2f35a5a30 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441afc30 con 0x55a2f35a9dc0 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa74e5d1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7441ace70 con 0x55a2f35a94a0 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa74e5d1740 10 client.0 _ll_register_callbacks cb 0x7fa73400b3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa749f94640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a5a30 0x55a2f35ba7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa749f94640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a5a30 0x55a2f35ba7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50498/0 (socket says 172.21.15.89:50498) 2024-09-15T23:36:37.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa749f94640 1 -- 172.21.15.89:0/309129368 learned_addr learned my addr 172.21.15.89:0/309129368 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa749793640 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a94a0 0x55a2f35bacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.881+0000 7fa74a795640 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a9dc0 0x55a2f362de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa749f94640 1 -- 172.21.15.89:0/309129368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a94a0 msgr2=0x55a2f35bacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa749f94640 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a94a0 0x55a2f35bacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa749f94640 1 -- 172.21.15.89:0/309129368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a9dc0 msgr2=0x55a2f362de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa749f94640 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a9dc0 0x55a2f362de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa749f94640 1 -- 172.21.15.89:0/309129368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2f362e3c0 con 0x55a2f35a5a30 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa749f94640 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a5a30 0x55a2f35ba7b0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fa7400096c0 tx=0x7fa74002aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f8985157cf2f5b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa72d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa72d7fa640 1 -- 172.21.15.89:0/309129368 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74003f040 con 0x55a2f35a5a30 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa72d7fa640 1 -- 172.21.15.89:0/309129368 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa7400035f0 con 0x55a2f35a5a30 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa74e5d1740 1 -- 172.21.15.89:0/309129368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a2f3516d60 con 0x55a2f35a5a30 2024-09-15T23:36:37.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa72d7fa640 1 -- 172.21.15.89:0/309129368 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa74005f050 con 0x55a2f35a5a30 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa74e5d1740 10 client.19815 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa74e5d1740 1 -- 172.21.15.89:0/309129368 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a2f362f590 con 0x55a2f35a5a30 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa72d7fa640 1 -- 172.21.15.89:0/309129368 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a2f3516d60 con 0x55a2f35a5a30 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa72d7fa640 1 -- 172.21.15.89:0/309129368 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa74004c070 con 0x55a2f35a5a30 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa74e5d1740 10 client.19815 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.882+0000 7fa74e5d1740 10 client.19815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.883+0000 7fa74e5d1740 20 client.19815 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.883+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a2f35b4680 0x55a2f3631830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.883+0000 7fa74e5d1740 4 client.19815 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.883+0000 7fa74e5d1740 1 -- 172.21.15.89:0/309129368 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a2f3493210 con 0x55a2f35b4680 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.883+0000 7fa749793640 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a2f35b4680 0x55a2f3631830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.883+0000 7fa749793640 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a2f35b4680 0x55a2f3631830 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a3e843813bc435a server_cookie=aed5d5a9f629885 in_seq=0 out_seq=0 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.883+0000 7fa72d7fa640 10 client.19815 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.883+0000 7fa72d7fa640 1 -- 172.21.15.89:0/309129368 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a2f3493210 con 0x55a2f35b4680 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.883+0000 7fa72d7fa640 10 client.19815 handle_command_reply: tid=0 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.883+0000 7fa74e5d1740 1 client.19815 shutdown 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 1 -- 172.21.15.89:0/309129368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a5a30 msgr2=0x55a2f35ba7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a5a30 0x55a2f35ba7b0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fa7400096c0 tx=0x7fa74002aa90 comp rx=0 tx=0).stop 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 1 -- 172.21.15.89:0/309129368 shutdown_connections 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55a2f35b4680 0x55a2f3631830 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a2f35a9dc0 0x55a2f362de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a2f35a94a0 0x55a2f35bacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 1 --2- 172.21.15.89:0/309129368 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a2f35a5a30 0x55a2f35ba7b0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa72d7fa640 0 client.19815 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:37.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 1 -- 172.21.15.89:0/309129368 >> 172.21.15.89:0/309129368 conn(0x55a2f35b18c0 msgr2=0x55a2f35b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 1 -- 172.21.15.89:0/309129368 shutdown_connections 2024-09-15T23:36:37.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 1 -- 172.21.15.89:0/309129368 wait complete. 2024-09-15T23:36:37.905 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:36:37.905 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:36:37.905 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:36:37.905 INFO:teuthology.orchestra.run.smithi089.stdout: "6fb20ded-9bd2-45ba-b7e1-5dfdd55d6f90": { 2024-09-15T23:36:37.905 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:36:37.905 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "6fb20ded-9bd2-45ba-b7e1-5dfdd55d6f90", 2024-09-15T23:36:37.906 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:36:37.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:36:37.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:36:37.906 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:36:37.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74e5d1740 20 client.19815 trim_cache size 0 max 16384 2024-09-15T23:36:37.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74c1f8640 1 -- 172.21.15.89:0/2087194480 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7300a3240 msgr2=0x7fa7300a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:37.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74c1f8640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7300a3240 0x7fa7300a5700 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa73c007fb0 tx=0x7fa73c004b50 comp rx=0 tx=0).stop 2024-09-15T23:36:37.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74c1f8640 1 -- 172.21.15.89:0/2087194480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa744102790 msgr2=0x7fa744198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:37.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74c1f8640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa744102790 0x7fa744198af0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fa73400b4a0 tx=0x7fa73400b960 comp rx=0 tx=0).stop 2024-09-15T23:36:37.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74c1f8640 1 -- 172.21.15.89:0/2087194480 shutdown_connections 2024-09-15T23:36:37.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74c1f8640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7300a3240 0x7fa7300a5700 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74c1f8640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa744106f10 0x7fa74419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74c1f8640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa744102790 0x7fa744198af0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74c1f8640 1 --2- 172.21.15.89:0/2087194480 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa744101de0 0x7fa7441985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:37.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.884+0000 7fa74c1f8640 1 -- 172.21.15.89:0/2087194480 >> 172.21.15.89:0/2087194480 conn(0x7fa7440fd8c0 msgr2=0x7fa7440fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:37.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.885+0000 7fa74c1f8640 1 -- 172.21.15.89:0/2087194480 shutdown_connections 2024-09-15T23:36:37.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:36:37.885+0000 7fa74c1f8640 1 -- 172.21.15.89:0/2087194480 wait complete. 2024-09-15T23:36:37.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:37 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:36:37.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:37 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:37.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:37 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:36:37.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:37 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:37.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:37 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:36:37.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:37 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:37.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:37 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:36:37.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:37 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:36:38.613 INFO:tasks.cephfs.filesystem:scrub status for tag:6fb20ded-9bd2-45ba-b7e1-5dfdd55d6f90 - {'path': '/', 'tag': '6fb20ded-9bd2-45ba-b7e1-5dfdd55d6f90', 'options': 'recursive,force'} 2024-09-15T23:36:38.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:38 smithi120 ceph-mon[35250]: pgmap v728: 129 pgs: 129 active+clean; 2.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:36:38.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:38 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:36:38.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:38 smithi174 ceph-mon[35586]: pgmap v728: 129 pgs: 129 active+clean; 2.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:36:38.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:38 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:36:39.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:38 smithi089 ceph-mon[30933]: pgmap v728: 129 pgs: 129 active+clean; 2.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:36:39.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:38 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:36:40.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:40 smithi120 ceph-mon[35250]: pgmap v729: 129 pgs: 129 active+clean; 2.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:40.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:40 smithi174 ceph-mon[35586]: pgmap v729: 129 pgs: 129 active+clean; 2.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:41.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:40 smithi089 ceph-mon[30933]: pgmap v729: 129 pgs: 129 active+clean; 2.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:42.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:42 smithi174 ceph-mon[35586]: pgmap v730: 129 pgs: 129 active+clean; 2.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:36:43.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:42 smithi089 ceph-mon[30933]: pgmap v730: 129 pgs: 129 active+clean; 2.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:36:43.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:42 smithi120 ceph-mon[35250]: pgmap v730: 129 pgs: 129 active+clean; 2.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:36:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.736+0000 7f2d49b59640 1 -- 172.21.15.89:0/984808317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d44101b60 msgr2=0x7f2d44101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.736+0000 7f2d49b59640 1 --2- 172.21.15.89:0/984808317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d44101b60 0x7f2d44101f60 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f2d34009900 tx=0x7f2d3402a410 comp rx=0 tx=0).stop 2024-09-15T23:36:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.736+0000 7f2d49b59640 1 -- 172.21.15.89:0/984808317 shutdown_connections 2024-09-15T23:36:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.736+0000 7f2d49b59640 1 --2- 172.21.15.89:0/984808317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d44103710 0x7f2d44109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.736+0000 7f2d49b59640 1 --2- 172.21.15.89:0/984808317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44102d50 0x7f2d441031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.736+0000 7f2d49b59640 1 --2- 172.21.15.89:0/984808317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d44101b60 0x7f2d44101f60 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.736+0000 7f2d49b59640 1 -- 172.21.15.89:0/984808317 >> 172.21.15.89:0/984808317 conn(0x7f2d440fd2e0 msgr2=0x7f2d440ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.736+0000 7f2d49b59640 1 -- 172.21.15.89:0/984808317 shutdown_connections 2024-09-15T23:36:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.736+0000 7f2d49b59640 1 -- 172.21.15.89:0/984808317 wait complete. 2024-09-15T23:36:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d49b59640 1 Processor -- start 2024-09-15T23:36:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d49b59640 1 -- start start 2024-09-15T23:36:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d49b59640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d44101b60 0x7f2d44197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d49b59640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d44102d50 0x7f2d44198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d49b59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44103710 0x7f2d4419f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d49b59640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4410d7e0 con 0x7f2d44101b60 2024-09-15T23:36:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d49b59640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4410d660 con 0x7f2d44102d50 2024-09-15T23:36:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d49b59640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4410d960 con 0x7f2d44103710 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d44101b60 0x7f2d44197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d44101b60 0x7f2d44197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50516/0 (socket says 172.21.15.89:50516) 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d437fe640 1 -- 172.21.15.89:0/334841609 learned_addr learned my addr 172.21.15.89:0/334841609 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d44102d50 0x7f2d44198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d43fff640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44103710 0x7f2d4419f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d437fe640 1 -- 172.21.15.89:0/334841609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44103710 msgr2=0x7f2d4419f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d437fe640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44103710 0x7f2d4419f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d437fe640 1 -- 172.21.15.89:0/334841609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d44102d50 msgr2=0x7f2d44198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d437fe640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d44102d50 0x7f2d44198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.737+0000 7f2d437fe640 1 -- 172.21.15.89:0/334841609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4419fa00 con 0x7f2d44101b60 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.738+0000 7f2d437fe640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d44101b60 0x7f2d44197de0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f2d34031610 tx=0x7f2d34004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.738+0000 7f2d40ff9640 1 -- 172.21.15.89:0/334841609 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3403f030 con 0x7f2d44101b60 2024-09-15T23:36:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.738+0000 7f2d40ff9640 1 -- 172.21.15.89:0/334841609 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d34003ad0 con 0x7f2d44101b60 2024-09-15T23:36:43.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.738+0000 7f2d49b59640 1 -- 172.21.15.89:0/334841609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d4419fc90 con 0x7f2d44101b60 2024-09-15T23:36:43.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.738+0000 7f2d40ff9640 1 -- 172.21.15.89:0/334841609 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34035070 con 0x7f2d44101b60 2024-09-15T23:36:43.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.738+0000 7f2d49b59640 1 -- 172.21.15.89:0/334841609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d441a0180 con 0x7f2d44101b60 2024-09-15T23:36:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d40ff9640 1 -- 172.21.15.89:0/334841609 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2d34042030 con 0x7f2d44101b60 2024-09-15T23:36:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d4c9d3740 1 Processor -- start 2024-09-15T23:36:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d4c9d3740 1 -- start start 2024-09-15T23:36:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1949a0 0x55aebc194da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc199da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc19c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55aebc0f82f0 con 0x55aebc1952e0 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55aebc0a0270 con 0x55aebc1949a0 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55aebc0f42b0 con 0x55aebc19a370 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50522/0 (socket says 172.21.15.89:50522) 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.741+0000 7f2d437fe640 1 -- 172.21.15.89:0/4059778489 learned_addr learned my addr 172.21.15.89:0/4059778489 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d43fff640 1 --2- 172.21.15.89:0/4059778489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1949a0 0x55aebc194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d40ff9640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d2c0a32f0 0x7f2d2c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/4059778489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d437fe640 1 -- 172.21.15.89:0/4059778489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 msgr2=0x55aebc19c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d437fe640 1 --2- 172.21.15.89:0/4059778489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc19c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d437fe640 1 -- 172.21.15.89:0/4059778489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1949a0 msgr2=0x55aebc194da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d437fe640 1 --2- 172.21.15.89:0/4059778489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1949a0 0x55aebc194da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d437fe640 1 -- 172.21.15.89:0/4059778489 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aebc09f340 con 0x55aebc1952e0 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/4059778489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d43fff640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d2c0a32f0 0x7f2d2c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d40ff9640 1 -- 172.21.15.89:0/334841609 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2d340f0390 con 0x7f2d44101b60 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d437fe640 1 --2- 172.21.15.89:0/4059778489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc199da0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f2d34001940 tx=0x7f2d3404b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192278bd67c03ab6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d29ffb640 1 -- 172.21.15.89:0/4059778489 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d340bbfc0 con 0x55aebc1952e0 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d43fff640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d2c0a32f0 0x7f2d2c0a57b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c005fd0 tx=0x7f2d3c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d29ffb640 1 -- 172.21.15.89:0/4059778489 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d34001970 con 0x55aebc1952e0 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.742+0000 7f2d29ffb640 1 -- 172.21.15.89:0/4059778489 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3409b030 con 0x55aebc1952e0 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.743+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/4059778489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 msgr2=0x55aebc199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.743+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/4059778489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc199da0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f2d34001940 tx=0x7f2d3404b000 comp rx=0 tx=0).stop 2024-09-15T23:36:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.743+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/4059778489 shutdown_connections 2024-09-15T23:36:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.743+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/4059778489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc19c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.743+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/4059778489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc199da0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.743+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/4059778489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1949a0 0x55aebc194da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.743+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/4059778489 >> 172.21.15.89:0/4059778489 conn(0x55aebc190010 msgr2=0x55aebc190410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.743+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/4059778489 shutdown_connections 2024-09-15T23:36:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.743+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/4059778489 wait complete. 2024-09-15T23:36:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.743+0000 7f2d4c9d3740 1 -- start start 2024-09-15T23:36:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d4c9d3740 1 Processor -- start 2024-09-15T23:36:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1949a0 0x55aebc19e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc1a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55aebc19d100 con 0x55aebc1952e0 2024-09-15T23:36:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55aebc081290 con 0x55aebc1949a0 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55aebc19d280 con 0x55aebc19a370 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d4c9d3740 10 client.0 _ll_register_callbacks cb 0x55aebc0b2140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d42ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d42ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc19eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50538/0 (socket says 172.21.15.89:50538) 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d42ffd640 1 -- 172.21.15.89:0/1953461973 learned_addr learned my addr 172.21.15.89:0/1953461973 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d437fe640 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1949a0 0x55aebc19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d43fff640 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc1a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d42ffd640 1 -- 172.21.15.89:0/1953461973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 msgr2=0x55aebc1a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc1a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d42ffd640 1 -- 172.21.15.89:0/1953461973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1949a0 msgr2=0x55aebc19e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1949a0 0x55aebc19e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.744+0000 7f2d42ffd640 1 -- 172.21.15.89:0/1953461973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aebbe0b210 con 0x55aebc1952e0 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc19eda0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f2d38012700 tx=0x7f2d38012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17bb4eaeb568a595 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1953461973 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d38004050 con 0x55aebc1952e0 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1953461973 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2d380041f0 con 0x55aebc1952e0 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1953461973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55aebc105a20 con 0x55aebc1952e0 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1953461973 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2d38025050 con 0x55aebc1952e0 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d4c9d3740 10 client.19827 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1953461973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55aebbed7620 con 0x55aebc1952e0 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1953461973 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55aebc105a20 con 0x55aebc1952e0 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1953461973 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2d38003a80 con 0x55aebc1952e0 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d4c9d3740 10 client.19827 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d4c9d3740 10 client.19827 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d4c9d3740 20 client.19827 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55aebc1aa4e0 0x55aebc1ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d4c9d3740 4 client.19827 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:43.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.745+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1953461973 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aebc0ffd70 con 0x55aebc1aa4e0 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.746+0000 7f2d437fe640 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55aebc1aa4e0 0x55aebc1ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.746+0000 7f2d22ffd640 10 client.19827 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.746+0000 7f2d437fe640 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55aebc1aa4e0 0x55aebc1ac8d0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fb5036c4b13e50a server_cookie=223bfa43166746c8 in_seq=0 out_seq=0 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1953461973 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aebc0ffd70 con 0x55aebc1aa4e0 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d22ffd640 10 client.19827 handle_command_reply: tid=0 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 client.19827 shutdown 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1953461973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 msgr2=0x55aebc19eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc19eda0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f2d38012700 tx=0x7f2d38012bc0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1953461973 shutdown_connections 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55aebc1aa4e0 0x55aebc1ac8d0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc1a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.755 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1952e0 0x55aebc19eda0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d22ffd640 0 client.19827 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:43.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1953461973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1949a0 0x55aebc19e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1953461973 >> 172.21.15.89:0/1953461973 conn(0x55aebc190010 msgr2=0x55aebc191820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:43.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1953461973 shutdown_connections 2024-09-15T23:36:43.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1953461973 wait complete. 2024-09-15T23:36:43.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.747+0000 7f2d4c9d3740 20 client.19827 trim_cache size 0 max 16384 2024-09-15T23:36:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.761+0000 7f2d4c9d3740 1 Processor -- start 2024-09-15T23:36:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.761+0000 7f2d4c9d3740 1 -- start start 2024-09-15T23:36:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.761+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc18f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.761+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1952e0 0x55aebc191400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.761+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc191940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.761+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d441aa830 con 0x55aebc1949a0 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.761+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d441ac370 con 0x55aebc1952e0 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.761+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d441aaf00 con 0x55aebc19a370 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc18f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc18f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50554/0 (socket says 172.21.15.89:50554) 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d437fe640 1 -- 172.21.15.89:0/462080598 learned_addr learned my addr 172.21.15.89:0/462080598 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d43fff640 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc191940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1952e0 0x55aebc191400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d437fe640 1 -- 172.21.15.89:0/462080598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 msgr2=0x55aebc191940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d437fe640 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc191940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d437fe640 1 -- 172.21.15.89:0/462080598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1952e0 msgr2=0x55aebc191400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d437fe640 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1952e0 0x55aebc191400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d437fe640 1 -- 172.21.15.89:0/462080598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aebc1a8ea0 con 0x55aebc1949a0 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d43fff640 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc191940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1952e0 0x55aebc191400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d437fe640 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc18f0b0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f2d340bc360 tx=0x7f2d340b8860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d4ca8e4b2d42e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d297fa640 1 -- 172.21.15.89:0/462080598 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d340c52d0 con 0x55aebc1949a0 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d297fa640 1 -- 172.21.15.89:0/462080598 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d340b84f0 con 0x55aebc1949a0 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.762+0000 7f2d297fa640 1 -- 172.21.15.89:0/462080598 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34003c80 con 0x55aebc1949a0 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/462080598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 msgr2=0x55aebc18f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc18f0b0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f2d340bc360 tx=0x7f2d340b8860 comp rx=0 tx=0).stop 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/462080598 shutdown_connections 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc19a370 0x55aebc191940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc1952e0 0x55aebc191400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/462080598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc18f0b0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/462080598 >> 172.21.15.89:0/462080598 conn(0x55aebc190010 msgr2=0x55aebc18e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/462080598 shutdown_connections 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/462080598 wait complete. 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 -- start start 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.763+0000 7f2d4c9d3740 1 Processor -- start 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc210710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc1952e0 0x55aebc210c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d4c9d3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc19a370 0x55aebc217c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2d441abca0 con 0x55aebc1949a0 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d441ab5d0 con 0x55aebc19a370 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d4c9d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d441a8810 con 0x55aebc1952e0 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d4c9d3740 10 client.0 _ll_register_callbacks cb 0x7f2d38009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc210710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d437fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc210710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50566/0 (socket says 172.21.15.89:50566) 2024-09-15T23:36:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d437fe640 1 -- 172.21.15.89:0/1284692789 learned_addr learned my addr 172.21.15.89:0/1284692789 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc1952e0 0x55aebc210c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d43fff640 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc19a370 0x55aebc217c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d437fe640 1 -- 172.21.15.89:0/1284692789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc1952e0 msgr2=0x55aebc210c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d437fe640 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc1952e0 0x55aebc210c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d437fe640 1 -- 172.21.15.89:0/1284692789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc19a370 msgr2=0x55aebc217c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d437fe640 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc19a370 0x55aebc217c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d437fe640 1 -- 172.21.15.89:0/1284692789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aebc2182a0 con 0x55aebc1949a0 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc1952e0 0x55aebc210c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d437fe640 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc210710 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f2d340c4da0 tx=0x7f2d3403b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bcb41fbabd016 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1284692789 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d340bcd60 con 0x55aebc1949a0 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1284692789 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2d34003c80 con 0x55aebc1949a0 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1284692789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d340b84f0 con 0x55aebc1949a0 2024-09-15T23:36:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1284692789 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2d340c0a10 con 0x55aebc1949a0 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d4c9d3740 10 client.19833 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.764+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1284692789 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55aebc219520 con 0x55aebc1949a0 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1284692789 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d340b84f0 con 0x55aebc1949a0 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1284692789 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2d3403b460 con 0x55aebc1949a0 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d4c9d3740 10 client.19833 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d4c9d3740 10 client.19833 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d4c9d3740 20 client.19833 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55aebc219a50 0x55aebc21be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d4c9d3740 4 client.19833 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1284692789 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aebc07c900 con 0x55aebc219a50 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55aebc219a50 0x55aebc21be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d22ffd640 10 client.19833 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.765+0000 7f2d42ffd640 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55aebc219a50 0x55aebc21be70 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1806a59cd8ab90ae server_cookie=1176ef50e8d7f3fb in_seq=0 out_seq=0 2024-09-15T23:36:43.779 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d22ffd640 1 -- 172.21.15.89:0/1284692789 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aebc07c900 con 0x55aebc219a50 2024-09-15T23:36:43.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d22ffd640 10 client.19833 handle_command_reply: tid=0 2024-09-15T23:36:43.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 client.19833 shutdown 2024-09-15T23:36:43.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1284692789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 msgr2=0x55aebc210710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:43.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc210710 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f2d340c4da0 tx=0x7f2d3403b800 comp rx=0 tx=0).stop 2024-09-15T23:36:43.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1284692789 shutdown_connections 2024-09-15T23:36:43.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55aebc219a50 0x55aebc21be70 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55aebc19a370 0x55aebc217c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.780 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55aebc1952e0 0x55aebc210c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 --2- 172.21.15.89:0/1284692789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aebc1949a0 0x55aebc210710 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d22ffd640 0 client.19833 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1284692789 >> 172.21.15.89:0/1284692789 conn(0x55aebc190010 msgr2=0x55aebc18f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1284692789 shutdown_connections 2024-09-15T23:36:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 1 -- 172.21.15.89:0/1284692789 wait complete. 2024-09-15T23:36:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.778+0000 7f2d4c9d3740 20 client.19833 trim_cache size 0 max 16384 2024-09-15T23:36:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:36:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:36:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:36:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:36:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 -- 172.21.15.89:0/334841609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d2c0a32f0 msgr2=0x7f2d2c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d2c0a32f0 0x7f2d2c0a57b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c005fd0 tx=0x7f2d3c004a40 comp rx=0 tx=0).stop 2024-09-15T23:36:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 -- 172.21.15.89:0/334841609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d44101b60 msgr2=0x7f2d44197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d44101b60 0x7f2d44197de0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f2d34031610 tx=0x7f2d34004270 comp rx=0 tx=0).stop 2024-09-15T23:36:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 -- 172.21.15.89:0/334841609 shutdown_connections 2024-09-15T23:36:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2d2c0a32f0 0x7f2d2c0a57b0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d44103710 0x7f2d4419f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d44102d50 0x7f2d44198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.783 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 --2- 172.21.15.89:0/334841609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d44101b60 0x7f2d44197de0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:43.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 -- 172.21.15.89:0/334841609 >> 172.21.15.89:0/334841609 conn(0x7f2d440fd2e0 msgr2=0x7f2d44106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:43.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 -- 172.21.15.89:0/334841609 shutdown_connections 2024-09-15T23:36:43.784 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:43.779+0000 7f2d49b59640 1 -- 172.21.15.89:0/334841609 wait complete. 2024-09-15T23:36:44.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:44 smithi174 ceph-mon[35586]: pgmap v731: 129 pgs: 129 active+clean; 2.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 19 op/s 2024-09-15T23:36:45.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:44 smithi089 ceph-mon[30933]: pgmap v731: 129 pgs: 129 active+clean; 2.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 19 op/s 2024-09-15T23:36:45.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:44 smithi120 ceph-mon[35250]: pgmap v731: 129 pgs: 129 active+clean; 2.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 19 op/s 2024-09-15T23:36:46.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:46 smithi174 ceph-mon[35586]: pgmap v732: 129 pgs: 129 active+clean; 2.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:36:47.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:46 smithi089 ceph-mon[30933]: pgmap v732: 129 pgs: 129 active+clean; 2.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:36:47.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:46 smithi120 ceph-mon[35250]: pgmap v732: 129 pgs: 129 active+clean; 2.2 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:36:48.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:48 smithi174 ceph-mon[35586]: pgmap v733: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:36:49.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:48 smithi089 ceph-mon[30933]: pgmap v733: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:36:49.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:48 smithi120 ceph-mon[35250]: pgmap v733: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:36:51.025 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:50 smithi120 ceph-mon[35250]: pgmap v734: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:51.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:50 smithi089 ceph-mon[30933]: pgmap v734: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:51.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:50 smithi174 ceph-mon[35586]: pgmap v734: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:53.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:52 smithi089 ceph-mon[30933]: pgmap v735: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:36:53.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:52 smithi120 ceph-mon[35250]: pgmap v735: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:36:53.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:52 smithi174 ceph-mon[35586]: pgmap v735: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:36:55.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:54 smithi089 ceph-mon[30933]: pgmap v736: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:36:55.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:54 smithi120 ceph-mon[35250]: pgmap v736: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:36:55.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:54 smithi174 ceph-mon[35586]: pgmap v736: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:36:56.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.875+0000 7f8d98c04640 1 -- 172.21.15.89:0/4086451437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d94107820 msgr2=0x7f8d94109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:56.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.875+0000 7f8d98c04640 1 --2- 172.21.15.89:0/4086451437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d94107820 0x7f8d94109c90 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009ca0 tx=0x7f8d8c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.876+0000 7f8d98c04640 1 -- 172.21.15.89:0/4086451437 shutdown_connections 2024-09-15T23:36:56.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.876+0000 7f8d98c04640 1 --2- 172.21.15.89:0/4086451437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d94107820 0x7f8d94109c90 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.876+0000 7f8d98c04640 1 --2- 172.21.15.89:0/4086451437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d94104ef0 0x7f8d941072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.876+0000 7f8d98c04640 1 --2- 172.21.15.89:0/4086451437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d941025c0 0x7f8d941049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.876+0000 7f8d98c04640 1 -- 172.21.15.89:0/4086451437 >> 172.21.15.89:0/4086451437 conn(0x7f8d940fc1e0 msgr2=0x7f8d940fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.876+0000 7f8d98c04640 1 -- 172.21.15.89:0/4086451437 shutdown_connections 2024-09-15T23:36:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.876+0000 7f8d98c04640 1 -- 172.21.15.89:0/4086451437 wait complete. 2024-09-15T23:36:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.876+0000 7f8d98c04640 1 Processor -- start 2024-09-15T23:36:56.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d98c04640 1 -- start start 2024-09-15T23:36:56.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d98c04640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d941025c0 0x7f8d94197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d98c04640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d94104ef0 0x7f8d94198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d98c04640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94107820 0x7f8d9419f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d98c04640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9410d800 con 0x7f8d94104ef0 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d98c04640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9410d680 con 0x7f8d941025c0 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d98c04640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9410d980 con 0x7f8d94107820 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d89d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d94104ef0 0x7f8d94198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d89d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d94104ef0 0x7f8d94198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43222/0 (socket says 172.21.15.89:43222) 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d89d9b640 1 -- 172.21.15.89:0/3578395461 learned_addr learned my addr 172.21.15.89:0/3578395461 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d92d9d640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94107820 0x7f8d9419f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d9259c640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d941025c0 0x7f8d94197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d89d9b640 1 -- 172.21.15.89:0/3578395461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94107820 msgr2=0x7f8d9419f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d89d9b640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94107820 0x7f8d9419f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d89d9b640 1 -- 172.21.15.89:0/3578395461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d941025c0 msgr2=0x7f8d94197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d89d9b640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d941025c0 0x7f8d94197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d89d9b640 1 -- 172.21.15.89:0/3578395461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9419fa70 con 0x7f8d94104ef0 2024-09-15T23:36:56.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d9259c640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d941025c0 0x7f8d94197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:36:56.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.877+0000 7f8d89d9b640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d94104ef0 0x7f8d94198390 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00d770 tx=0x7f8d7c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:56.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.878+0000 7f8d8bfff640 1 -- 172.21.15.89:0/3578395461 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c017070 con 0x7f8d94104ef0 2024-09-15T23:36:56.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.878+0000 7f8d8bfff640 1 -- 172.21.15.89:0/3578395461 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d7c009c30 con 0x7f8d94104ef0 2024-09-15T23:36:56.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.878+0000 7f8d98c04640 1 -- 172.21.15.89:0/3578395461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9419fd60 con 0x7f8d94104ef0 2024-09-15T23:36:56.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.878+0000 7f8d8bfff640 1 -- 172.21.15.89:0/3578395461 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c00f620 con 0x7f8d94104ef0 2024-09-15T23:36:56.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.878+0000 7f8d98c04640 1 -- 172.21.15.89:0/3578395461 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d941a0250 con 0x7f8d94104ef0 2024-09-15T23:36:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.880+0000 7f8d8bfff640 1 -- 172.21.15.89:0/3578395461 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8d7c01a070 con 0x7f8d94104ef0 2024-09-15T23:36:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.880+0000 7f8d9ba78740 1 Processor -- start 2024-09-15T23:36:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d9ba78740 1 -- start start 2024-09-15T23:36:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8cd370 0x55c0ac8cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac82b2f0 con 0x55c0ac8c82e0 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac7d3270 con 0x55c0ac8cd370 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac8272b0 con 0x55c0ac8c79a0 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d9259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d8bfff640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d800a32f0 0x7f8d800a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d9259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43228/0 (socket says 172.21.15.89:43228) 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d9259c640 1 -- 172.21.15.89:0/749208600 learned_addr learned my addr 172.21.15.89:0/749208600 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d89d9b640 1 --2- 172.21.15.89:0/749208600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8cd370 0x55c0ac8cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d92d9d640 1 --2- 172.21.15.89:0/749208600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.890 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.881+0000 7f8d92d9d640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d800a32f0 0x7f8d800a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d8bfff640 1 -- 172.21.15.89:0/3578395461 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8d7c0cfb80 con 0x7f8d94104ef0 2024-09-15T23:36:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d9259c640 1 -- 172.21.15.89:0/749208600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 msgr2=0x55c0ac8c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d9259c640 1 --2- 172.21.15.89:0/749208600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8c7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d9259c640 1 -- 172.21.15.89:0/749208600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8cd370 msgr2=0x55c0ac8cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d9259c640 1 --2- 172.21.15.89:0/749208600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8cd370 0x55c0ac8cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d9259c640 1 -- 172.21.15.89:0/749208600 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0ac7d2340 con 0x55c0ac8c82e0 2024-09-15T23:36:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d9259c640 1 --2- 172.21.15.89:0/749208600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8ccda0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400cca0 tx=0x7f8d8400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4485326c774f65bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d92d9d640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d800a32f0 0x7f8d800a57b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c006010 tx=0x7f8d8c005d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:56.891 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d88d99640 1 -- 172.21.15.89:0/749208600 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84005220 con 0x55c0ac8c82e0 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d88d99640 1 -- 172.21.15.89:0/749208600 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d84004c80 con 0x55c0ac8c82e0 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.882+0000 7f8d88d99640 1 -- 172.21.15.89:0/749208600 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d840108b0 con 0x55c0ac8c82e0 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 -- 172.21.15.89:0/749208600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c82e0 msgr2=0x55c0ac8ccda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/749208600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8ccda0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400cca0 tx=0x7f8d8400e720 comp rx=0 tx=0).stop 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 -- 172.21.15.89:0/749208600 shutdown_connections 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/749208600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8cd370 0x55c0ac8cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/749208600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8ccda0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/749208600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 -- 172.21.15.89:0/749208600 >> 172.21.15.89:0/749208600 conn(0x55c0ac8c3010 msgr2=0x55c0ac8c3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 -- 172.21.15.89:0/749208600 shutdown_connections 2024-09-15T23:36:56.892 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 -- 172.21.15.89:0/749208600 wait complete. 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 -- start start 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.883+0000 7f8d9ba78740 1 Processor -- start 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8cd370 0x55c0ac8d90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac8d0340 con 0x55c0ac8cd370 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac7b4290 con 0x55c0ac8c82e0 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac8d04c0 con 0x55c0ac8c79a0 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d9ba78740 10 client.0 _ll_register_callbacks cb 0x55c0ac7e5140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d92d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8cd370 0x55c0ac8d90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d92d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8cd370 0x55c0ac8d90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43240/0 (socket says 172.21.15.89:43240) 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d92d9d640 1 -- 172.21.15.89:0/2181446675 learned_addr learned my addr 172.21.15.89:0/2181446675 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:56.893 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d89d9b640 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d9259c640 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d92d9d640 1 -- 172.21.15.89:0/2181446675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 msgr2=0x55c0ac8d1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d92d9d640 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d92d9d640 1 -- 172.21.15.89:0/2181446675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8c82e0 msgr2=0x55c0ac8d2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d92d9d640 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d92d9d640 1 -- 172.21.15.89:0/2181446675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0ac53e210 con 0x55c0ac8cd370 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d9259c640 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d92d9d640 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8cd370 0x55c0ac8d90c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c02f540 tx=0x7f8d8c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65055da6f27e8ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d79ffb640 1 -- 172.21.15.89:0/2181446675 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c050030 con 0x55c0ac8cd370 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d79ffb640 1 -- 172.21.15.89:0/2181446675 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8d8c0307d0 con 0x55c0ac8cd370 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.884+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2181446675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d84004c80 con 0x55c0ac8cd370 2024-09-15T23:36:56.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d79ffb640 1 -- 172.21.15.89:0/2181446675 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8d8c05d050 con 0x55c0ac8cd370 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d9ba78740 10 client.19845 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2181446675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c0ac60a620 con 0x55c0ac8cd370 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d79ffb640 1 -- 172.21.15.89:0/2181446675 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d84004c80 con 0x55c0ac8cd370 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d79ffb640 1 -- 172.21.15.89:0/2181446675 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8d8c047c30 con 0x55c0ac8cd370 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d9ba78740 10 client.19845 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d9ba78740 10 client.19845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d9ba78740 20 client.19845 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0ac8dd810 0x55c0ac8dfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d9ba78740 4 client.19845 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2181446675 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0ac838a20 con 0x55c0ac8dd810 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.885+0000 7f8d9259c640 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0ac8dd810 0x55c0ac8dfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.886+0000 7f8d79ffb640 10 client.19845 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.886+0000 7f8d9259c640 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0ac8dd810 0x55c0ac8dfcb0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38db6728b5399e5 server_cookie=271061312c78d688 in_seq=0 out_seq=0 2024-09-15T23:36:56.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.886+0000 7f8d79ffb640 1 -- 172.21.15.89:0/2181446675 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c0ac838a20 con 0x55c0ac8dd810 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.886+0000 7f8d79ffb640 10 client.19845 handle_command_reply: tid=0 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 client.19845 shutdown 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2181446675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8cd370 msgr2=0x55c0ac8d90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8cd370 0x55c0ac8d90c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c02f540 tx=0x7f8d8c0095c0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2181446675 shutdown_connections 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0ac8dd810 0x55c0ac8dfcb0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8cd370 0x55c0ac8d90c0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d79ffb640 0 client.19845 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/2181446675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2181446675 >> 172.21.15.89:0/2181446675 conn(0x55c0ac8c3010 msgr2=0x55c0ac8c4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:56.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2181446675 shutdown_connections 2024-09-15T23:36:56.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2181446675 wait complete. 2024-09-15T23:36:56.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.887+0000 7f8d9ba78740 20 client.19845 trim_cache size 0 max 16384 2024-09-15T23:36:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.901+0000 7f8d9ba78740 1 Processor -- start 2024-09-15T23:36:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.901+0000 7f8d9ba78740 1 -- start start 2024-09-15T23:36:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.901+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.901+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.901+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8cd370 0x55c0ac8c4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.901+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8d941aa910 con 0x55c0ac8c79a0 2024-09-15T23:36:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.901+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8d941ac450 con 0x55c0ac8c82e0 2024-09-15T23:36:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.901+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d941aafe0 con 0x55c0ac8cd370 2024-09-15T23:36:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.901+0000 7f8d9259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d9259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43254/0 (socket says 172.21.15.89:43254) 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d9259c640 1 -- 172.21.15.89:0/2964771049 learned_addr learned my addr 172.21.15.89:0/2964771049 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d92d9d640 1 --2- 172.21.15.89:0/2964771049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8cd370 0x55c0ac8c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d9259c640 1 -- 172.21.15.89:0/2964771049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8cd370 msgr2=0x55c0ac8c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d9259c640 1 --2- 172.21.15.89:0/2964771049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8cd370 0x55c0ac8c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d9259c640 1 -- 172.21.15.89:0/2964771049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8c82e0 msgr2=0x55c0ac8d8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d9259c640 1 --2- 172.21.15.89:0/2964771049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d9259c640 1 -- 172.21.15.89:0/2964771049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0ac8c87c0 con 0x55c0ac8c79a0 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d9259c640 1 --2- 172.21.15.89:0/2964771049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8c3ec0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400cca0 tx=0x7f8d8400c590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8da875fc286c4f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d7bfff640 1 -- 172.21.15.89:0/2964771049 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84004af0 con 0x55c0ac8c79a0 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d7bfff640 1 -- 172.21.15.89:0/2964771049 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d84004f50 con 0x55c0ac8c79a0 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.902+0000 7f8d7bfff640 1 -- 172.21.15.89:0/2964771049 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84004c90 con 0x55c0ac8c79a0 2024-09-15T23:36:56.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2964771049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 msgr2=0x55c0ac8c3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/2964771049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8c3ec0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400cca0 tx=0x7f8d8400c590 comp rx=0 tx=0).stop 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2964771049 shutdown_connections 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/2964771049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8cd370 0x55c0ac8c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/2964771049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/2964771049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8c3ec0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2964771049 >> 172.21.15.89:0/2964771049 conn(0x55c0ac8c3010 msgr2=0x55c0ac8d1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2964771049 shutdown_connections 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 -- 172.21.15.89:0/2964771049 wait complete. 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 -- start start 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 Processor -- start 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8cd370 0x55c0ac8d7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8d941abd80 con 0x55c0ac8c79a0 2024-09-15T23:36:56.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8d941ab6b0 con 0x55c0ac8cd370 2024-09-15T23:36:56.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d941a88f0 con 0x55c0ac8c82e0 2024-09-15T23:36:56.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9ba78740 10 client.0 _ll_register_callbacks cb 0x7f8d8c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:36:56.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43258/0 (socket says 172.21.15.89:43258) 2024-09-15T23:36:56.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.903+0000 7f8d9259c640 1 -- 172.21.15.89:0/4243096934 learned_addr learned my addr 172.21.15.89:0/4243096934 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d89d9b640 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d92d9d640 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8cd370 0x55c0ac8d7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d9259c640 1 -- 172.21.15.89:0/4243096934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c82e0 msgr2=0x55c0ac8d6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d9259c640 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d9259c640 1 -- 172.21.15.89:0/4243096934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8cd370 msgr2=0x55c0ac8d7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d9259c640 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8cd370 0x55c0ac8d7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d9259c640 1 -- 172.21.15.89:0/4243096934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0ac8d7f40 con 0x55c0ac8c79a0 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d9259c640 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d63e0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400ec20 tx=0x7f8d8400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5d0d3070a9661b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d79ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d79ffb640 1 -- 172.21.15.89:0/4243096934 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84036030 con 0x55c0ac8c79a0 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d79ffb640 1 -- 172.21.15.89:0/4243096934 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8d84004020 con 0x55c0ac8c79a0 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d9ba78740 1 -- 172.21.15.89:0/4243096934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d84004f50 con 0x55c0ac8c79a0 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d79ffb640 1 -- 172.21.15.89:0/4243096934 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8d84042050 con 0x55c0ac8c79a0 2024-09-15T23:36:56.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d9ba78740 10 client.19851 fetch_fsmap learned FSMap version 29 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d9ba78740 1 -- 172.21.15.89:0/4243096934 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c0ac8d9250 con 0x55c0ac8c79a0 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d79ffb640 1 -- 172.21.15.89:0/4243096934 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d84004f50 con 0x55c0ac8c79a0 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.904+0000 7f8d79ffb640 1 -- 172.21.15.89:0/4243096934 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8d84003950 con 0x55c0ac8c79a0 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.905+0000 7f8d9ba78740 10 client.19851 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.905+0000 7f8d9ba78740 10 client.19851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.905+0000 7f8d9ba78740 20 client.19851 populate_metadata read hostname 'smithi089' 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.905+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0ac9489d0 0x55c0ac94adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.905+0000 7f8d9ba78740 4 client.19851 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.905+0000 7f8d9ba78740 1 -- 172.21.15.89:0/4243096934 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c0ac832d70 con 0x55c0ac9489d0 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.905+0000 7f8d89d9b640 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0ac9489d0 0x55c0ac94adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.905+0000 7f8d79ffb640 10 client.19851 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:56.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.905+0000 7f8d89d9b640 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0ac9489d0 0x55c0ac94adc0 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2056b7ae390fc6c4 server_cookie=b672526d3d3ca47d in_seq=0 out_seq=0 2024-09-15T23:36:56.920 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d79ffb640 1 -- 172.21.15.89:0/4243096934 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c0ac832d70 con 0x55c0ac9489d0 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d79ffb640 10 client.19851 handle_command_reply: tid=0 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 client.19851 shutdown 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 -- 172.21.15.89:0/4243096934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 msgr2=0x55c0ac8d63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d63e0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400ec20 tx=0x7f8d8400eda0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 -- 172.21.15.89:0/4243096934 shutdown_connections 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c0ac9489d0 0x55c0ac94adc0 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0ac8cd370 0x55c0ac8d7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac8c82e0 0x55c0ac8d6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 --2- 172.21.15.89:0/4243096934 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c0ac8c79a0 0x55c0ac8d63e0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d79ffb640 0 client.19851 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 -- 172.21.15.89:0/4243096934 >> 172.21.15.89:0/4243096934 conn(0x55c0ac8c3010 msgr2=0x55c0ac8c19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 -- 172.21.15.89:0/4243096934 shutdown_connections 2024-09-15T23:36:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 1 -- 172.21.15.89:0/4243096934 wait complete. 2024-09-15T23:36:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.919+0000 7f8d9ba78740 20 client.19851 trim_cache size 0 max 16384 2024-09-15T23:36:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:36:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:36:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:36:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:36:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 -- 172.21.15.89:0/3578395461 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d800a32f0 msgr2=0x7f8d800a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d800a32f0 0x7f8d800a57b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c006010 tx=0x7f8d8c005d80 comp rx=0 tx=0).stop 2024-09-15T23:36:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 -- 172.21.15.89:0/3578395461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d94104ef0 msgr2=0x7f8d94198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:36:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d94104ef0 0x7f8d94198390 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00d770 tx=0x7f8d7c00dc30 comp rx=0 tx=0).stop 2024-09-15T23:36:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 -- 172.21.15.89:0/3578395461 shutdown_connections 2024-09-15T23:36:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8d800a32f0 0x7f8d800a57b0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d94107820 0x7f8d9419f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d94104ef0 0x7f8d94198390 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 --2- 172.21.15.89:0/3578395461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d941025c0 0x7f8d94197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:36:56.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 -- 172.21.15.89:0/3578395461 >> 172.21.15.89:0/3578395461 conn(0x7f8d940fc1e0 msgr2=0x7f8d94102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:36:56.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 -- 172.21.15.89:0/3578395461 shutdown_connections 2024-09-15T23:36:56.924 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:36:56.920+0000 7f8d98c04640 1 -- 172.21.15.89:0/3578395461 wait complete. 2024-09-15T23:36:57.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:56 smithi089 ceph-mon[30933]: pgmap v737: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:57.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:56 smithi120 ceph-mon[35250]: pgmap v737: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:57.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:56 smithi174 ceph-mon[35586]: pgmap v737: 129 pgs: 129 active+clean; 2.3 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:36:59.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:36:58 smithi089 ceph-mon[30933]: pgmap v738: 129 pgs: 129 active+clean; 2.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:36:59.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:36:58 smithi120 ceph-mon[35250]: pgmap v738: 129 pgs: 129 active+clean; 2.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:36:59.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:36:58 smithi174 ceph-mon[35586]: pgmap v738: 129 pgs: 129 active+clean; 2.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 21 op/s 2024-09-15T23:37:01.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:00 smithi089 ceph-mon[30933]: pgmap v739: 129 pgs: 129 active+clean; 2.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:37:01.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:00 smithi120 ceph-mon[35250]: pgmap v739: 129 pgs: 129 active+clean; 2.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:37:01.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:00 smithi174 ceph-mon[35586]: pgmap v739: 129 pgs: 129 active+clean; 2.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:37:03.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:02 smithi089 ceph-mon[30933]: pgmap v740: 129 pgs: 129 active+clean; 2.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:37:03.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:02 smithi120 ceph-mon[35250]: pgmap v740: 129 pgs: 129 active+clean; 2.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:37:03.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:02 smithi174 ceph-mon[35586]: pgmap v740: 129 pgs: 129 active+clean; 2.4 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:37:05.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:04 smithi089 ceph-mon[30933]: pgmap v741: 129 pgs: 129 active+clean; 2.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:37:05.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:04 smithi120 ceph-mon[35250]: pgmap v741: 129 pgs: 129 active+clean; 2.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:37:05.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:04 smithi174 ceph-mon[35586]: pgmap v741: 129 pgs: 129 active+clean; 2.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 19 op/s 2024-09-15T23:37:07.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:06 smithi089 ceph-mon[30933]: pgmap v742: 129 pgs: 129 active+clean; 2.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:37:07.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:06 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:37:07.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:06 smithi120 ceph-mon[35250]: pgmap v742: 129 pgs: 129 active+clean; 2.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:37:07.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:06 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:37:07.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:06 smithi174 ceph-mon[35586]: pgmap v742: 129 pgs: 129 active+clean; 2.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 16 op/s 2024-09-15T23:37:07.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:06 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:37:08.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:37:08.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:37:08.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:37:08.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:37:08.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:37:08.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:37:08.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:37:08.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:37:08.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:37:08.614 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:37:08.872 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:08.994 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:08 smithi089 ceph-mon[30933]: pgmap v743: 129 pgs: 129 active+clean; 2.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:37:09.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:08 smithi120 ceph-mon[35250]: pgmap v743: 129 pgs: 129 active+clean; 2.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:37:09.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:08 smithi174 ceph-mon[35586]: pgmap v743: 129 pgs: 129 active+clean; 2.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:37:09.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.359+0000 7fb45940c640 1 -- 172.21.15.89:0/2972326675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb454100a80 msgr2=0x7fb454100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.359+0000 7fb45940c640 1 --2- 172.21.15.89:0/2972326675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb454100a80 0x7fb454100ee0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fb448009930 tx=0x7fb44802a420 comp rx=0 tx=0).stop 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.359+0000 7fb45940c640 1 -- 172.21.15.89:0/2972326675 shutdown_connections 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.359+0000 7fb45940c640 1 --2- 172.21.15.89:0/2972326675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454101420 0x7fb45410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.359+0000 7fb45940c640 1 --2- 172.21.15.89:0/2972326675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb454100a80 0x7fb454100ee0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.359+0000 7fb45940c640 1 --2- 172.21.15.89:0/2972326675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb454106a80 0x7fb454106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.359+0000 7fb45940c640 1 -- 172.21.15.89:0/2972326675 >> 172.21.15.89:0/2972326675 conn(0x7fb4540fc7a0 msgr2=0x7fb4540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.359+0000 7fb45940c640 1 -- 172.21.15.89:0/2972326675 shutdown_connections 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.359+0000 7fb45940c640 1 -- 172.21.15.89:0/2972326675 wait complete. 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.360+0000 7fb45940c640 1 Processor -- start 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.360+0000 7fb45940c640 1 -- start start 2024-09-15T23:37:09.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.360+0000 7fb45940c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb454100a80 0x7fb4541983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.360+0000 7fb45940c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb454101420 0x7fb4541988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.360+0000 7fb45940c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454106a80 0x7fb45419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.360+0000 7fb45940c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb454111fa0 con 0x7fb454100a80 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.360+0000 7fb45940c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb454111e20 con 0x7fb454101420 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.360+0000 7fb45940c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb454112120 con 0x7fb454106a80 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb452ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb454100a80 0x7fb4541983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb452ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb454100a80 0x7fb4541983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49250/0 (socket says 172.21.15.89:49250) 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb452ffd640 1 -- 172.21.15.89:0/904211245 learned_addr learned my addr 172.21.15.89:0/904211245 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb4527fc640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb454101420 0x7fb4541988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb4537fe640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454106a80 0x7fb45419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb452ffd640 1 -- 172.21.15.89:0/904211245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454106a80 msgr2=0x7fb45419cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb452ffd640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454106a80 0x7fb45419cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb452ffd640 1 -- 172.21.15.89:0/904211245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb454101420 msgr2=0x7fb4541988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb452ffd640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb454101420 0x7fb4541988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb452ffd640 1 -- 172.21.15.89:0/904211245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45419d360 con 0x7fb454100a80 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb452ffd640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb454100a80 0x7fb4541983b0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fb44400c920 tx=0x7fb44400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb43bfff640 1 -- 172.21.15.89:0/904211245 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444007c70 con 0x7fb454100a80 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb43bfff640 1 -- 172.21.15.89:0/904211245 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb444007e10 con 0x7fb454100a80 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb45940c640 1 -- 172.21.15.89:0/904211245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45419d650 con 0x7fb454100a80 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb43bfff640 1 -- 172.21.15.89:0/904211245 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444018890 con 0x7fb454100a80 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.361+0000 7fb45940c640 1 -- 172.21.15.89:0/904211245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4541a4f30 con 0x7fb454100a80 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.363+0000 7fb43bfff640 1 -- 172.21.15.89:0/904211245 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb444018a70 con 0x7fb454100a80 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.364+0000 7fb45b7e5740 1 Processor -- start 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.364+0000 7fb45b7e5740 1 -- start start 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.364+0000 7fb43bfff640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb43c0a3240 0x7fb43c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.364+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca38554a0 0x563ca3855880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.364+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca3860140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.364+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3860680 0x563ca3862a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.364+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563ca37bb2e0 con 0x563ca38554a0 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.364+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563ca3762ce0 con 0x563ca3860680 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.364+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563ca37b72a0 con 0x563ca3855dc0 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb4537fe640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb43c0a3240 0x7fb43c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb452ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca38554a0 0x563ca3855880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb452ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca38554a0 0x563ca3855880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49258/0 (socket says 172.21.15.89:49258) 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb43bfff640 1 -- 172.21.15.89:0/904211245 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb4440cfe60 con 0x7fb454100a80 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb452ffd640 1 -- 172.21.15.89:0/38608100 learned_addr learned my addr 172.21.15.89:0/38608100 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb4527fc640 1 --2- 172.21.15.89:0/38608100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca3860140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb452ffd640 1 -- 172.21.15.89:0/38608100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 msgr2=0x563ca3860140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb4537fe640 1 --2- 172.21.15.89:0/38608100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3860680 0x563ca3862a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb452ffd640 1 --2- 172.21.15.89:0/38608100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca3860140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb452ffd640 1 -- 172.21.15.89:0/38608100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3860680 msgr2=0x563ca3862a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb452ffd640 1 --2- 172.21.15.89:0/38608100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3860680 0x563ca3862a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb452ffd640 1 -- 172.21.15.89:0/38608100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ca3747440 con 0x563ca38554a0 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb4537fe640 1 --2- 172.21.15.89:0/38608100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3860680 0x563ca3862a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb4537fe640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb43c0a3240 0x7fb43c0a5700 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb44c005fd0 tx=0x7fb44c010740 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb452ffd640 1 --2- 172.21.15.89:0/38608100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca38554a0 0x563ca3855880 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fb444012040 tx=0x7fb44401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f807456d6d1411 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb4397fa640 1 -- 172.21.15.89:0/38608100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44409b990 con 0x563ca38554a0 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb4397fa640 1 -- 172.21.15.89:0/38608100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb44409c730 con 0x563ca38554a0 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.365+0000 7fb4397fa640 1 -- 172.21.15.89:0/38608100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4440aeb70 con 0x563ca38554a0 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 -- 172.21.15.89:0/38608100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca38554a0 msgr2=0x563ca3855880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/38608100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca38554a0 0x563ca3855880 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fb444012040 tx=0x7fb44401c000 comp rx=0 tx=0).stop 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 -- 172.21.15.89:0/38608100 shutdown_connections 2024-09-15T23:37:09.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/38608100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3860680 0x563ca3862a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/38608100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca3860140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/38608100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca38554a0 0x563ca3855880 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 -- 172.21.15.89:0/38608100 >> 172.21.15.89:0/38608100 conn(0x563ca3851a30 msgr2=0x563ca3851e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 -- 172.21.15.89:0/38608100 shutdown_connections 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 -- 172.21.15.89:0/38608100 wait complete. 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 -- start start 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.366+0000 7fb45b7e5740 1 Processor -- start 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 0x563ca38600e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca385d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca385db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563ca3863250 con 0x563ca3860680 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563ca3776050 con 0x563ca38554a0 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563ca38633d0 con 0x563ca3855dc0 2024-09-15T23:37:09.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb45b7e5740 10 client.0 _ll_register_callbacks cb 0x563ca37b6160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb4537fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca385db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb4537fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca385db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49272/0 (socket says 172.21.15.89:49272) 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb4537fe640 1 -- 172.21.15.89:0/84589743 learned_addr learned my addr 172.21.15.89:0/84589743 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb4527fc640 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca385d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb452ffd640 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 0x563ca38600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb4537fe640 1 -- 172.21.15.89:0/84589743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 msgr2=0x563ca385d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb4537fe640 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca385d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb4537fe640 1 -- 172.21.15.89:0/84589743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 msgr2=0x563ca38600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb4537fe640 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 0x563ca38600e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb4537fe640 1 -- 172.21.15.89:0/84589743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ca352c3c0 con 0x563ca3860680 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb4537fe640 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca385db20 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fb44c012910 tx=0x7fb44c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47fc32045c7fd197 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb436ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb436ff5640 1 -- 172.21.15.89:0/84589743 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c011d70 con 0x563ca3860680 2024-09-15T23:37:09.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb45b7e5740 1 -- 172.21.15.89:0/84589743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb44409c730 con 0x563ca3860680 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.367+0000 7fb436ff5640 1 -- 172.21.15.89:0/84589743 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb44c01f400 con 0x563ca3860680 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb436ff5640 1 -- 172.21.15.89:0/84589743 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb44c02e050 con 0x563ca3860680 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb45b7e5740 10 client.19863 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb45b7e5740 1 -- 172.21.15.89:0/84589743 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563ca386ed50 con 0x563ca3860680 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb436ff5640 1 -- 172.21.15.89:0/84589743 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb44409c730 con 0x563ca3860680 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb436ff5640 1 -- 172.21.15.89:0/84589743 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb44c00f040 con 0x563ca3860680 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb45b7e5740 10 client.19863 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb45b7e5740 10 client.19863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb45b7e5740 20 client.19863 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563ca3870160 0x563ca3872570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb45b7e5740 4 client.19863 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb45b7e5740 1 -- 172.21.15.89:0/84589743 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ca37c8850 con 0x563ca3870160 2024-09-15T23:37:09.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.368+0000 7fb452ffd640 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563ca3870160 0x563ca3872570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.369+0000 7fb452ffd640 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563ca3870160 0x563ca3872570 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1a92ed7ede2d5dd server_cookie=3f4a2fdac1efaa86 in_seq=0 out_seq=0 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.369+0000 7fb436ff5640 10 client.19863 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.369+0000 7fb436ff5640 1 -- 172.21.15.89:0/84589743 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563ca37c8850 con 0x563ca3870160 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.369+0000 7fb436ff5640 10 client.19863 handle_command_reply: tid=0 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.369+0000 7fb45b7e5740 1 client.19863 shutdown 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 1 -- 172.21.15.89:0/84589743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 msgr2=0x563ca385db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca385db20 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fb44c012910 tx=0x7fb44c004ab0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 1 -- 172.21.15.89:0/84589743 shutdown_connections 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563ca3870160 0x563ca3872570 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca385db20 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca385d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/84589743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 0x563ca38600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb436ff5640 0 client.19863 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:09.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 1 -- 172.21.15.89:0/84589743 >> 172.21.15.89:0/84589743 conn(0x563ca3851a30 msgr2=0x563ca3857760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:09.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 1 -- 172.21.15.89:0/84589743 shutdown_connections 2024-09-15T23:37:09.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 1 -- 172.21.15.89:0/84589743 wait complete. 2024-09-15T23:37:09.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.370+0000 7fb45b7e5740 20 client.19863 trim_cache size 0 max 16384 2024-09-15T23:37:09.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb45b7e5740 1 Processor -- start 2024-09-15T23:37:09.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb45b7e5740 1 -- start start 2024-09-15T23:37:09.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 0x563ca385f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca3877e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3878350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4541aee10 con 0x563ca3860680 2024-09-15T23:37:09.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4541b0950 con 0x563ca38554a0 2024-09-15T23:37:09.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4541af4e0 con 0x563ca3855dc0 2024-09-15T23:37:09.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb4537fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3878350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb4537fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3878350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49286/0 (socket says 172.21.15.89:49286) 2024-09-15T23:37:09.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.387+0000 7fb4537fe640 1 -- 172.21.15.89:0/1643987422 learned_addr learned my addr 172.21.15.89:0/1643987422 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:09.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb4527fc640 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca3877e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb4537fe640 1 -- 172.21.15.89:0/1643987422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 msgr2=0x563ca3877e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:09.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb452ffd640 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 0x563ca385f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb4537fe640 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca3877e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb4537fe640 1 -- 172.21.15.89:0/1643987422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 msgr2=0x563ca385f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:09.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb4537fe640 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 0x563ca385f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb4537fe640 1 -- 172.21.15.89:0/1643987422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ca3855bf0 con 0x563ca3860680 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb4527fc640 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca3877e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb452ffd640 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 0x563ca385f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb4537fe640 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3878350 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fb44c027690 tx=0x7fb44c009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e387367675f81d47 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb438ff9640 1 -- 172.21.15.89:0/1643987422 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c036020 con 0x563ca3860680 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb438ff9640 1 -- 172.21.15.89:0/1643987422 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb44c00f040 con 0x563ca3860680 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.388+0000 7fb438ff9640 1 -- 172.21.15.89:0/1643987422 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c027e30 con 0x563ca3860680 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 -- 172.21.15.89:0/1643987422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 msgr2=0x563ca3878350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3878350 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fb44c027690 tx=0x7fb44c009e20 comp rx=0 tx=0).stop 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 -- 172.21.15.89:0/1643987422 shutdown_connections 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3878350 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca3855dc0 0x563ca3877e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/1643987422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca38554a0 0x563ca385f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 -- 172.21.15.89:0/1643987422 >> 172.21.15.89:0/1643987422 conn(0x563ca3851a30 msgr2=0x563ca385ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 -- 172.21.15.89:0/1643987422 shutdown_connections 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 -- 172.21.15.89:0/1643987422 wait complete. 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 -- start start 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 Processor -- start 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca38554a0 0x563ca38663b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3855dc0 0x563ca38668f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.389+0000 7fb45b7e5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3866e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4541b0280 con 0x563ca3860680 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4541afbb0 con 0x563ca3855dc0 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb45b7e5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4541acdf0 con 0x563ca38554a0 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb45b7e5740 10 client.0 _ll_register_callbacks cb 0x7fb44c009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4537fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3866e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4537fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3866e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49296/0 (socket says 172.21.15.89:49296) 2024-09-15T23:37:09.398 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4537fe640 1 -- 172.21.15.89:0/2376013596 learned_addr learned my addr 172.21.15.89:0/2376013596 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb452ffd640 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca38554a0 0x563ca38663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4537fe640 1 -- 172.21.15.89:0/2376013596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca38554a0 msgr2=0x563ca38663b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4537fe640 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca38554a0 0x563ca38663b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4527fc640 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3855dc0 0x563ca38668f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4537fe640 1 -- 172.21.15.89:0/2376013596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3855dc0 msgr2=0x563ca38668f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4537fe640 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3855dc0 0x563ca38668f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4537fe640 1 -- 172.21.15.89:0/2376013596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ca38de230 con 0x563ca3860680 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4527fc640 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3855dc0 0x563ca38668f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb4537fe640 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3866e30 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009880 tx=0x7fb44c011180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca094d292957a9c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb436ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb436ff5640 1 -- 172.21.15.89:0/2376013596 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c036030 con 0x563ca3860680 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb436ff5640 1 -- 172.21.15.89:0/2376013596 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb44c027a10 con 0x563ca3860680 2024-09-15T23:37:09.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb45b7e5740 1 -- 172.21.15.89:0/2376013596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb44c00f040 con 0x563ca3860680 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb436ff5640 1 -- 172.21.15.89:0/2376013596 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb44c04a050 con 0x563ca3860680 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb45b7e5740 10 client.19869 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.390+0000 7fb45b7e5740 1 -- 172.21.15.89:0/2376013596 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563ca38df470 con 0x563ca3860680 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.391+0000 7fb436ff5640 1 -- 172.21.15.89:0/2376013596 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb44c00f040 con 0x563ca3860680 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.391+0000 7fb436ff5640 1 -- 172.21.15.89:0/2376013596 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb44c0052f0 con 0x563ca3860680 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.391+0000 7fb45b7e5740 10 client.19869 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.391+0000 7fb45b7e5740 10 client.19869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.391+0000 7fb45b7e5740 20 client.19869 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.391+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563ca38dfad0 0x563ca38e1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.391+0000 7fb45b7e5740 4 client.19869 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.391+0000 7fb45b7e5740 1 -- 172.21.15.89:0/2376013596 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ca37c2d60 con 0x563ca38dfad0 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.391+0000 7fb452ffd640 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563ca38dfad0 0x563ca38e1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb452ffd640 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563ca38dfad0 0x563ca38e1ec0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49665b69185e793d server_cookie=e420d6e1bf773dd2 in_seq=0 out_seq=0 2024-09-15T23:37:09.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb436ff5640 10 client.19869 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb436ff5640 1 -- 172.21.15.89:0/2376013596 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563ca37c2d60 con 0x563ca38dfad0 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb436ff5640 10 client.19869 handle_command_reply: tid=0 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 client.19869 shutdown 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 -- 172.21.15.89:0/2376013596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 msgr2=0x563ca3866e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3866e30 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009880 tx=0x7fb44c011180 comp rx=0 tx=0).stop 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 -- 172.21.15.89:0/2376013596 shutdown_connections 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563ca38dfad0 0x563ca38e1ec0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563ca3860680 0x563ca3866e30 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563ca3855dc0 0x563ca38668f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 --2- 172.21.15.89:0/2376013596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563ca38554a0 0x563ca38663b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 -- 172.21.15.89:0/2376013596 >> 172.21.15.89:0/2376013596 conn(0x563ca3851a30 msgr2=0x563ca385f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb436ff5640 0 client.19869 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:09.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 -- 172.21.15.89:0/2376013596 shutdown_connections 2024-09-15T23:37:09.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 1 -- 172.21.15.89:0/2376013596 wait complete. 2024-09-15T23:37:09.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.392+0000 7fb45b7e5740 20 client.19869 trim_cache size 0 max 16384 2024-09-15T23:37:09.402 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:37:09.402 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:37:09.402 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:37:09.402 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:37:09.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 -- 172.21.15.89:0/904211245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb43c0a3240 msgr2=0x7fb43c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:09.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb43c0a3240 0x7fb43c0a5700 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fb44c005fd0 tx=0x7fb44c010740 comp rx=0 tx=0).stop 2024-09-15T23:37:09.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 -- 172.21.15.89:0/904211245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb454100a80 msgr2=0x7fb4541983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:09.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb454100a80 0x7fb4541983b0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fb44400c920 tx=0x7fb44400cde0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 -- 172.21.15.89:0/904211245 shutdown_connections 2024-09-15T23:37:09.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb43c0a3240 0x7fb43c0a5700 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454106a80 0x7fb45419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb454101420 0x7fb4541988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 --2- 172.21.15.89:0/904211245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb454100a80 0x7fb4541983b0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:09.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 -- 172.21.15.89:0/904211245 >> 172.21.15.89:0/904211245 conn(0x7fb4540fc7a0 msgr2=0x7fb4540fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:09.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 -- 172.21.15.89:0/904211245 shutdown_connections 2024-09-15T23:37:09.404 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:09.393+0000 7fb45940c640 1 -- 172.21.15.89:0/904211245 wait complete. 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 -- 172.21.15.89:0/2975947431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c10102620 msgr2=0x7f9c10104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 --2- 172.21.15.89:0/2975947431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c10102620 0x7f9c10104a10 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009900 tx=0x7f9c0002a410 comp rx=0 tx=0).stop 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 -- 172.21.15.89:0/2975947431 shutdown_connections 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 --2- 172.21.15.89:0/2975947431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c10107880 0x7f9c10109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 --2- 172.21.15.89:0/2975947431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c10104f50 0x7f9c10107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 --2- 172.21.15.89:0/2975947431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c10102620 0x7f9c10104a10 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 -- 172.21.15.89:0/2975947431 >> 172.21.15.89:0/2975947431 conn(0x7f9c100fc1e0 msgr2=0x7f9c100fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 -- 172.21.15.89:0/2975947431 shutdown_connections 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 -- 172.21.15.89:0/2975947431 wait complete. 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 Processor -- start 2024-09-15T23:37:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 -- start start 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c10102620 0x7f9c1019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c10104f50 0x7f9c1019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c10107880 0x7f9c101a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1010d8a0 con 0x7f9c10102620 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1010d720 con 0x7f9c10107880 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.009+0000 7f9c15c2e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1010da20 con 0x7f9c10104f50 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c10102620 0x7f9c1019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c10102620 0x7f9c1019a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49308/0 (socket says 172.21.15.89:49308) 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/363827648 learned_addr learned my addr 172.21.15.89:0/363827648 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c07fff640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c10104f50 0x7f9c1019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/363827648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c10104f50 msgr2=0x7f9c1019a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c10104f50 0x7f9c1019a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/363827648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c10107880 msgr2=0x7f9c101a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c10107880 0x7f9c101a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/363827648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c101a1ce0 con 0x7f9c10102620 2024-09-15T23:37:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c10102620 0x7f9c1019a0c0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009230 tx=0x7f9c00004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0d7fa640 1 -- 172.21.15.89:0/363827648 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0003f030 con 0x7f9c10102620 2024-09-15T23:37:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0d7fa640 1 -- 172.21.15.89:0/363827648 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c00003ce0 con 0x7f9c10102620 2024-09-15T23:37:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c15c2e640 1 -- 172.21.15.89:0/363827648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c101a1f70 con 0x7f9c10102620 2024-09-15T23:37:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c0d7fa640 1 -- 172.21.15.89:0/363827648 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00035070 con 0x7f9c10102620 2024-09-15T23:37:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.010+0000 7f9c15c2e640 1 -- 172.21.15.89:0/363827648 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c101a2450 con 0x7f9c10102620 2024-09-15T23:37:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.012+0000 7f9c0d7fa640 1 -- 172.21.15.89:0/363827648 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9c00042030 con 0x7f9c10102620 2024-09-15T23:37:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.012+0000 7f9c189c9740 1 Processor -- start 2024-09-15T23:37:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c189c9740 1 -- start start 2024-09-15T23:37:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae1726eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae17273da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae17276830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c189c9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ae171d22f0 con 0x55ae1726e9a0 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c189c9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ae1717a270 con 0x55ae17274370 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c189c9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ae171ce2b0 con 0x55ae1726f2e0 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae1726eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae1726eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49314/0 (socket says 172.21.15.89:49314) 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0ffff640 1 -- 172.21.15.89:0/66872492 learned_addr learned my addr 172.21.15.89:0/66872492 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/66872492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae17273da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c07fff640 1 --2- 172.21.15.89:0/66872492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae17276830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0d7fa640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9bfc0a32f0 0x7f9bfc0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0ffff640 1 -- 172.21.15.89:0/66872492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 msgr2=0x55ae17273da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0ffff640 1 --2- 172.21.15.89:0/66872492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae17273da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0ffff640 1 -- 172.21.15.89:0/66872492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 msgr2=0x55ae17276830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0ffff640 1 --2- 172.21.15.89:0/66872492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae17276830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0ffff640 1 -- 172.21.15.89:0/66872492 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae17179340 con 0x55ae1726e9a0 2024-09-15T23:37:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0d7fa640 1 -- 172.21.15.89:0/363827648 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9c000f0530 con 0x7f9c10102620 2024-09-15T23:37:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0ffff640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9bfc0a32f0 0x7f9bfc0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.013+0000 7f9c0ffff640 1 --2- 172.21.15.89:0/66872492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae1726eda0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009e90 tx=0x7f9c0800b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f997eae0e33008b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c05ffb640 1 -- 172.21.15.89:0/66872492 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08015e60 con 0x55ae1726e9a0 2024-09-15T23:37:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c05ffb640 1 -- 172.21.15.89:0/66872492 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c08020b20 con 0x55ae1726e9a0 2024-09-15T23:37:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c05ffb640 1 -- 172.21.15.89:0/66872492 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08014aa0 con 0x55ae1726e9a0 2024-09-15T23:37:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c0ffff640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9bfc0a32f0 0x7f9bfc0a57b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9c0801d000 tx=0x7f9c0801e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c189c9740 1 -- 172.21.15.89:0/66872492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 msgr2=0x55ae1726eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c189c9740 1 --2- 172.21.15.89:0/66872492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae1726eda0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009e90 tx=0x7f9c0800b420 comp rx=0 tx=0).stop 2024-09-15T23:37:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c189c9740 1 -- 172.21.15.89:0/66872492 shutdown_connections 2024-09-15T23:37:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c189c9740 1 --2- 172.21.15.89:0/66872492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae17276830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c189c9740 1 --2- 172.21.15.89:0/66872492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae17273da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c189c9740 1 --2- 172.21.15.89:0/66872492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae1726eda0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c189c9740 1 -- 172.21.15.89:0/66872492 >> 172.21.15.89:0/66872492 conn(0x55ae1726a010 msgr2=0x55ae1726a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c189c9740 1 -- 172.21.15.89:0/66872492 shutdown_connections 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.014+0000 7f9c189c9740 1 -- 172.21.15.89:0/66872492 wait complete. 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c189c9740 1 -- start start 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c189c9740 1 Processor -- start 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17278c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae17279160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae17280130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c189c9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ae17277340 con 0x55ae1726e9a0 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c189c9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ae1715b290 con 0x55ae17274370 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c189c9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ae172774c0 con 0x55ae1726f2e0 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c189c9740 10 client.0 _ll_register_callbacks cb 0x55ae1718c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c0f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17278c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c0f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17278c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49322/0 (socket says 172.21.15.89:49322) 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/2759882143 learned_addr learned my addr 172.21.15.89:0/2759882143 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.015+0000 7f9c0ffff640 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae17280130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c07fff640 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae17279160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/2759882143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 msgr2=0x55ae17279160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae17279160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/2759882143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 msgr2=0x55ae17280130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae17280130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/2759882143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae16ee5210 con 0x55ae1726e9a0 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c0ffff640 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae17280130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17278c20 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f9c000314e0 tx=0x7f9c000bc5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58da6c6e74a1452e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9beeffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9beeffd640 1 -- 172.21.15.89:0/2759882143 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00049040 con 0x55ae1726e9a0 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9beeffd640 1 -- 172.21.15.89:0/2759882143 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9c000b8dd0 con 0x55ae1726e9a0 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c189c9740 1 -- 172.21.15.89:0/2759882143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c08020b20 con 0x55ae1726e9a0 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9beeffd640 1 -- 172.21.15.89:0/2759882143 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9c000bce60 con 0x55ae1726e9a0 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c189c9740 10 client.19881 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c189c9740 1 -- 172.21.15.89:0/2759882143 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ae16fb1620 con 0x55ae1726e9a0 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9beeffd640 1 -- 172.21.15.89:0/2759882143 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c08020b20 con 0x55ae1726e9a0 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9beeffd640 1 -- 172.21.15.89:0/2759882143 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9c000bc9c0 con 0x55ae1726e9a0 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c189c9740 10 client.19881 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c189c9740 10 client.19881 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c189c9740 20 client.19881 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c189c9740 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ae172847f0 0x55ae17286c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c189c9740 4 client.19881 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.016+0000 7f9c189c9740 1 -- 172.21.15.89:0/2759882143 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae171dfa20 con 0x55ae172847f0 2024-09-15T23:37:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.017+0000 7f9c07fff640 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ae172847f0 0x55ae17286c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.017+0000 7f9beeffd640 10 client.19881 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.017+0000 7f9c07fff640 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ae172847f0 0x55ae17286c60 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5651ae6c3fdf5001 server_cookie=dbba59f8dfebdeda in_seq=0 out_seq=0 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9beeffd640 1 -- 172.21.15.89:0/2759882143 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ae171dfa20 con 0x55ae172847f0 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9beeffd640 10 client.19881 handle_command_reply: tid=0 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 client.19881 shutdown 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 -- 172.21.15.89:0/2759882143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 msgr2=0x55ae17278c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17278c20 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f9c000314e0 tx=0x7f9c000bc5f0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 -- 172.21.15.89:0/2759882143 shutdown_connections 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ae172847f0 0x55ae17286c60 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae17280130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9beeffd640 0 client.19881 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae17279160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 --2- 172.21.15.89:0/2759882143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17278c20 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 -- 172.21.15.89:0/2759882143 >> 172.21.15.89:0/2759882143 conn(0x55ae1726a010 msgr2=0x55ae1726b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 -- 172.21.15.89:0/2759882143 shutdown_connections 2024-09-15T23:37:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.018+0000 7f9c189c9740 1 -- 172.21.15.89:0/2759882143 wait complete. 2024-09-15T23:37:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.019+0000 7f9c189c9740 20 client.19881 trim_cache size 0 max 16384 2024-09-15T23:37:10.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.033+0000 7f9c189c9740 1 Processor -- start 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.033+0000 7f9c189c9740 1 -- start start 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17279040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae172800c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae1728b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c189c9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9c101acb50 con 0x55ae1726e9a0 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c189c9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9c101ae690 con 0x55ae17274370 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c189c9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9c101ad220 con 0x55ae1726f2e0 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c0f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17279040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c0f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17279040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49324/0 (socket says 172.21.15.89:49324) 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/795907109 learned_addr learned my addr 172.21.15.89:0/795907109 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/795907109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 msgr2=0x55ae172800c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c07fff640 1 --2- 172.21.15.89:0/795907109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae172800c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/795907109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae172800c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/795907109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 msgr2=0x55ae1728b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/795907109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae1728b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c0f7fe640 1 -- 172.21.15.89:0/795907109 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae17278760 con 0x55ae1726e9a0 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c07fff640 1 --2- 172.21.15.89:0/795907109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae172800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/795907109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17279040 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f9c000edfb0 tx=0x7f9c00009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57140eb5ef54c287 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c057fa640 1 -- 172.21.15.89:0/795907109 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00049040 con 0x55ae1726e9a0 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c057fa640 1 -- 172.21.15.89:0/795907109 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c000bc760 con 0x55ae1726e9a0 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.034+0000 7f9c057fa640 1 -- 172.21.15.89:0/795907109 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c000b8dd0 con 0x55ae1726e9a0 2024-09-15T23:37:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 -- 172.21.15.89:0/795907109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 msgr2=0x55ae17279040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 --2- 172.21.15.89:0/795907109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17279040 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f9c000edfb0 tx=0x7f9c00009260 comp rx=0 tx=0).stop 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 -- 172.21.15.89:0/795907109 shutdown_connections 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 --2- 172.21.15.89:0/795907109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae17274370 0x55ae1728b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 --2- 172.21.15.89:0/795907109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae172800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 --2- 172.21.15.89:0/795907109 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae1726e9a0 0x55ae17279040 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 -- 172.21.15.89:0/795907109 >> 172.21.15.89:0/795907109 conn(0x55ae1726a010 msgr2=0x55ae172691c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 -- 172.21.15.89:0/795907109 shutdown_connections 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 -- 172.21.15.89:0/795907109 wait complete. 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 -- start start 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.035+0000 7f9c189c9740 1 Processor -- start 2024-09-15T23:37:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae1726e9a0 0x55ae1727dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae1727e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c189c9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae17274370 0x55ae1727f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c189c9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9c101adfc0 con 0x55ae17274370 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c189c9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9c101ad8f0 con 0x55ae1726e9a0 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c189c9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9c101aab30 con 0x55ae1726f2e0 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c189c9740 10 client.0 _ll_register_callbacks cb 0x7f9c00005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae17274370 0x55ae1727f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae17274370 0x55ae1727f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49338/0 (socket says 172.21.15.89:49338) 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0ffff640 1 -- 172.21.15.89:0/4197875102 learned_addr learned my addr 172.21.15.89:0/4197875102 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c07fff640 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae1727e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae1726e9a0 0x55ae1727dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0ffff640 1 -- 172.21.15.89:0/4197875102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 msgr2=0x55ae1727e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0ffff640 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae1727e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0ffff640 1 -- 172.21.15.89:0/4197875102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae1726e9a0 msgr2=0x55ae1727dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:10.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0ffff640 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae1726e9a0 0x55ae1727dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.043 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0ffff640 1 -- 172.21.15.89:0/4197875102 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae17287290 con 0x55ae17274370 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae1726e9a0 0x55ae1727dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c0ffff640 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae17274370 0x55ae1727f280 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f9c08014610 tx=0x7f9c08014640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5ab073188394b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9beeffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9beeffd640 1 -- 172.21.15.89:0/4197875102 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0802e070 con 0x55ae17274370 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9c189c9740 1 -- 172.21.15.89:0/4197875102 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c000bc760 con 0x55ae17274370 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.036+0000 7f9beeffd640 1 -- 172.21.15.89:0/4197875102 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9c08014a70 con 0x55ae17274370 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9beeffd640 1 -- 172.21.15.89:0/4197875102 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9c08015e60 con 0x55ae17274370 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9c189c9740 10 client.19887 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9c189c9740 1 -- 172.21.15.89:0/4197875102 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ae172ef880 con 0x55ae17274370 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9beeffd640 1 -- 172.21.15.89:0/4197875102 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c000bc760 con 0x55ae17274370 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9beeffd640 1 -- 172.21.15.89:0/4197875102 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9c08020cb0 con 0x55ae17274370 2024-09-15T23:37:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9c189c9740 10 client.19887 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9c189c9740 10 client.19887 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9c189c9740 20 client.19887 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9c189c9740 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ae172f00a0 0x55ae172f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9c189c9740 4 client.19887 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:37:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9c189c9740 1 -- 172.21.15.89:0/4197875102 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ae171d9d70 con 0x55ae172f00a0 2024-09-15T23:37:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.037+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ae172f00a0 0x55ae172f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.038+0000 7f9c0f7fe640 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ae172f00a0 0x55ae172f24c0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91381941af2631ef server_cookie=300600eff437c9ef in_seq=0 out_seq=0 2024-09-15T23:37:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.038+0000 7f9beeffd640 10 client.19887 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9beeffd640 1 -- 172.21.15.89:0/4197875102 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ae171d9d70 con 0x55ae172f00a0 2024-09-15T23:37:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9beeffd640 10 client.19887 handle_command_reply: tid=0 2024-09-15T23:37:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 client.19887 shutdown 2024-09-15T23:37:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 -- 172.21.15.89:0/4197875102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae17274370 msgr2=0x55ae1727f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae17274370 0x55ae1727f280 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f9c08014610 tx=0x7f9c08014640 comp rx=0 tx=0).stop 2024-09-15T23:37:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 -- 172.21.15.89:0/4197875102 shutdown_connections 2024-09-15T23:37:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55ae172f00a0 0x55ae172f24c0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae17274370 0x55ae1727f280 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ae1726f2e0 0x55ae1727e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9beeffd640 0 client.19887 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:10.055 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 --2- 172.21.15.89:0/4197875102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae1726e9a0 0x55ae1727dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 -- 172.21.15.89:0/4197875102 >> 172.21.15.89:0/4197875102 conn(0x55ae1726a010 msgr2=0x55ae172691c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 -- 172.21.15.89:0/4197875102 shutdown_connections 2024-09-15T23:37:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 1 -- 172.21.15.89:0/4197875102 wait complete. 2024-09-15T23:37:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.052+0000 7f9c189c9740 20 client.19887 trim_cache size 0 max 16384 2024-09-15T23:37:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:37:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:37:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:37:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:37:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 -- 172.21.15.89:0/363827648 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9bfc0a32f0 msgr2=0x7f9bfc0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9bfc0a32f0 0x7f9bfc0a57b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9c0801d000 tx=0x7f9c0801e000 comp rx=0 tx=0).stop 2024-09-15T23:37:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 -- 172.21.15.89:0/363827648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c10102620 msgr2=0x7f9c1019a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c10102620 0x7f9c1019a0c0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009230 tx=0x7f9c00004430 comp rx=0 tx=0).stop 2024-09-15T23:37:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 -- 172.21.15.89:0/363827648 shutdown_connections 2024-09-15T23:37:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9bfc0a32f0 0x7f9bfc0a57b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c10107880 0x7f9c101a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c10104f50 0x7f9c1019a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 --2- 172.21.15.89:0/363827648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c10102620 0x7f9c1019a0c0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 -- 172.21.15.89:0/363827648 >> 172.21.15.89:0/363827648 conn(0x7f9c100fc1e0 msgr2=0x7f9c10101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 -- 172.21.15.89:0/363827648 shutdown_connections 2024-09-15T23:37:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:10.053+0000 7f9c15c2e640 1 -- 172.21.15.89:0/363827648 wait complete. 2024-09-15T23:37:10.271 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:37:10.272 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:37:10.558 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:10.941 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:10 smithi089 ceph-mon[30933]: pgmap v744: 129 pgs: 129 active+clean; 2.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:37:11.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.020+0000 7f38ddbbf640 1 -- 172.21.15.89:0/1982334574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8101de0 msgr2=0x7f38d81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:11.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.020+0000 7f38ddbbf640 1 --2- 172.21.15.89:0/1982334574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8101de0 0x7f38d81021c0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009900 tx=0x7f38c802a410 comp rx=0 tx=0).stop 2024-09-15T23:37:11.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.020+0000 7f38ddbbf640 1 -- 172.21.15.89:0/1982334574 shutdown_connections 2024-09-15T23:37:11.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.020+0000 7f38ddbbf640 1 --2- 172.21.15.89:0/1982334574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8106f10 0x7f38d810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:11.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.020+0000 7f38ddbbf640 1 --2- 172.21.15.89:0/1982334574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8102790 0x7f38d81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:11.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.020+0000 7f38ddbbf640 1 --2- 172.21.15.89:0/1982334574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8101de0 0x7f38d81021c0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.020+0000 7f38ddbbf640 1 -- 172.21.15.89:0/1982334574 >> 172.21.15.89:0/1982334574 conn(0x7f38d80fd8e0 msgr2=0x7f38d80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.020+0000 7f38ddbbf640 1 -- 172.21.15.89:0/1982334574 shutdown_connections 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.020+0000 7f38ddbbf640 1 -- 172.21.15.89:0/1982334574 wait complete. 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38ddbbf640 1 Processor -- start 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38ddbbf640 1 -- start start 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38ddbbf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8101de0 0x7f38d81983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38ddbbf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d81988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38ddbbf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8106f10 0x7f38d819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38ddbbf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f38d8111f70 con 0x7f38d8102790 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38ddbbf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f38d8111df0 con 0x7f38d8101de0 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38ddbbf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f38d81120f0 con 0x7f38d8106f10 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d81988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:11.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d81988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49370/0 (socket says 172.21.15.89:49370) 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38cffff640 1 -- 172.21.15.89:0/2569220838 learned_addr learned my addr 172.21.15.89:0/2569220838 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38d77fe640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8101de0 0x7f38d81983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38d7fff640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8106f10 0x7f38d819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38cffff640 1 -- 172.21.15.89:0/2569220838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8106f10 msgr2=0x7f38d819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38cffff640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8106f10 0x7f38d819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38cffff640 1 -- 172.21.15.89:0/2569220838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8101de0 msgr2=0x7f38d81983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38cffff640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8101de0 0x7f38d81983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38cffff640 1 -- 172.21.15.89:0/2569220838 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d819d360 con 0x7f38d8102790 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.021+0000 7f38cffff640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d81988f0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f38c000d950 tx=0x7f38c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.022+0000 7f38d57fa640 1 -- 172.21.15.89:0/2569220838 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c0017070 con 0x7f38d8102790 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.022+0000 7f38d57fa640 1 -- 172.21.15.89:0/2569220838 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c0009c30 con 0x7f38d8102790 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.022+0000 7f38ddbbf640 1 -- 172.21.15.89:0/2569220838 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d819d650 con 0x7f38d8102790 2024-09-15T23:37:11.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.022+0000 7f38d57fa640 1 -- 172.21.15.89:0/2569220838 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c000f6b0 con 0x7f38d8102790 2024-09-15T23:37:11.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.022+0000 7f38ddbbf640 1 -- 172.21.15.89:0/2569220838 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d81a4f30 con 0x7f38d8102790 2024-09-15T23:37:11.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.023+0000 7f38ddbbf640 1 -- 172.21.15.89:0/2569220838 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d81021c0 con 0x7f38d8102790 2024-09-15T23:37:11.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.024+0000 7f38d57fa640 1 -- 172.21.15.89:0/2569220838 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f38c00109d0 con 0x7f38d8102790 2024-09-15T23:37:11.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.024+0000 7f38d57fa640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a3240 0x7f38c40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:11.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.025+0000 7f38d77fe640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a3240 0x7f38c40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:11.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.025+0000 7f38d57fa640 1 -- 172.21.15.89:0/2569220838 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f38c00cf990 con 0x7f38d8102790 2024-09-15T23:37:11.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.025+0000 7f38d77fe640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a3240 0x7f38c40a5700 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009230 tx=0x7f38c80058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:11.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.029+0000 7f38d57fa640 1 -- 172.21.15.89:0/2569220838 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f38c0009de0 con 0x7f38d8102790 2024-09-15T23:37:11.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:10 smithi120 ceph-mon[35250]: pgmap v744: 129 pgs: 129 active+clean; 2.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:37:11.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.231+0000 7f38ddbbf640 1 -- 172.21.15.89:0/2569220838 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38d8109b60 con 0x7f38d8102790 2024-09-15T23:37:11.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.232+0000 7f38d57fa640 1 -- 172.21.15.89:0/2569220838 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f38c009c2c0 con 0x7f38d8102790 2024-09-15T23:37:11.235 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:37:11.235 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:37:11.235 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:37:11.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:10 smithi174 ceph-mon[35586]: pgmap v744: 129 pgs: 129 active+clean; 2.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.234+0000 7f38ddbbf640 1 -- 172.21.15.89:0/2569220838 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a3240 msgr2=0x7f38c40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.234+0000 7f38ddbbf640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a3240 0x7f38c40a5700 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009230 tx=0x7f38c80058d0 comp rx=0 tx=0).stop 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.234+0000 7f38ddbbf640 1 -- 172.21.15.89:0/2569220838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 msgr2=0x7f38d81988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.234+0000 7f38ddbbf640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d81988f0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f38c000d950 tx=0x7f38c000de10 comp rx=0 tx=0).stop 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.234+0000 7f38ddbbf640 1 -- 172.21.15.89:0/2569220838 shutdown_connections 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.234+0000 7f38ddbbf640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f38c40a3240 0x7f38c40a5700 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.234+0000 7f38ddbbf640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38d8106f10 0x7f38d819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.234+0000 7f38ddbbf640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f38d8102790 0x7f38d81988f0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.234+0000 7f38ddbbf640 1 --2- 172.21.15.89:0/2569220838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f38d8101de0 0x7f38d81983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.234+0000 7f38ddbbf640 1 -- 172.21.15.89:0/2569220838 >> 172.21.15.89:0/2569220838 conn(0x7f38d80fd8e0 msgr2=0x7f38d80fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.235+0000 7f38ddbbf640 1 -- 172.21.15.89:0/2569220838 shutdown_connections 2024-09-15T23:37:11.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:11.235+0000 7f38ddbbf640 1 -- 172.21.15.89:0/2569220838 wait complete. 2024-09-15T23:37:11.998 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:11 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2569220838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:12.027 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:37:12.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:11 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2569220838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:12.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:11 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2569220838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:12.274 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.747+0000 7f9f8d995640 1 -- 172.21.15.89:0/2643893771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f88106f20 msgr2=0x7f9f8810eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.747+0000 7f9f8d995640 1 --2- 172.21.15.89:0/2643893771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f88106f20 0x7f9f8810eaa0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009ca0 tx=0x7f9f8002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.747+0000 7f9f8d995640 1 -- 172.21.15.89:0/2643893771 shutdown_connections 2024-09-15T23:37:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.747+0000 7f9f8d995640 1 --2- 172.21.15.89:0/2643893771 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f88106f20 0x7f9f8810eaa0 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.747+0000 7f9f8d995640 1 --2- 172.21.15.89:0/2643893771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f881027a0 0x7f9f881067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.747+0000 7f9f8d995640 1 --2- 172.21.15.89:0/2643893771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f88101df0 0x7f9f881021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.747+0000 7f9f8d995640 1 -- 172.21.15.89:0/2643893771 >> 172.21.15.89:0/2643893771 conn(0x7f9f880fd8b0 msgr2=0x7f9f880ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.747+0000 7f9f8d995640 1 -- 172.21.15.89:0/2643893771 shutdown_connections 2024-09-15T23:37:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.747+0000 7f9f8d995640 1 -- 172.21.15.89:0/2643893771 wait complete. 2024-09-15T23:37:12.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.748+0000 7f9f8d995640 1 Processor -- start 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.748+0000 7f9f8d995640 1 -- start start 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.748+0000 7f9f8d995640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f88101df0 0x7f9f88198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.748+0000 7f9f8d995640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f881027a0 0x7f9f88198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.748+0000 7f9f8d995640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f88106f20 0x7f9f8819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.748+0000 7f9f8d995640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9f88111fa0 con 0x7f9f88106f20 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.748+0000 7f9f8d995640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9f88111e20 con 0x7f9f88101df0 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.748+0000 7f9f8d995640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f88112120 con 0x7f9f881027a0 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.748+0000 7f9f877fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f88106f20 0x7f9f8819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f877fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f88106f20 0x7f9f8819ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49400/0 (socket says 172.21.15.89:49400) 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f877fe640 1 -- 172.21.15.89:0/1993232141 learned_addr learned my addr 172.21.15.89:0/1993232141 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f88101df0 0x7f9f88198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f881027a0 0x7f9f88198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f877fe640 1 -- 172.21.15.89:0/1993232141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f881027a0 msgr2=0x7f9f88198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f877fe640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f881027a0 0x7f9f88198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f877fe640 1 -- 172.21.15.89:0/1993232141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f88101df0 msgr2=0x7f9f88198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f877fe640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f88101df0 0x7f9f88198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f877fe640 1 -- 172.21.15.89:0/1993232141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8819d3b0 con 0x7f9f88106f20 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f881027a0 0x7f9f88198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f877fe640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f88106f20 0x7f9f8819ccd0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f9f80031a20 tx=0x7f9f80031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f8c993640 1 -- 172.21.15.89:0/1993232141 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80003c30 con 0x7f9f88106f20 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f8c993640 1 -- 172.21.15.89:0/1993232141 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f80003dd0 con 0x7f9f88106f20 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f8c993640 1 -- 172.21.15.89:0/1993232141 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8002fcf0 con 0x7f9f88106f20 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f8d995640 1 -- 172.21.15.89:0/1993232141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8819d640 con 0x7f9f88106f20 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.749+0000 7f9f8d995640 1 -- 172.21.15.89:0/1993232141 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f881a4f80 con 0x7f9f88106f20 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.752+0000 7f9f8fd6e740 1 Processor -- start 2024-09-15T23:37:12.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.752+0000 7f9f8c993640 1 -- 172.21.15.89:0/1993232141 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9f80004050 con 0x7f9f88106f20 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.752+0000 7f9f8fd6e740 1 -- start start 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d0ab40 0x5580c2d09090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d09a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 0x5580c2d1ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5580c2c6f2f0 con 0x5580c2d095d0 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580c2c16cf0 con 0x5580c2d18780 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5580c2c6b2b0 con 0x5580c2d0ab40 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f86ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d09a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f86ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d09a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49402/0 (socket says 172.21.15.89:49402) 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f86ffd640 1 -- 172.21.15.89:0/1804308890 learned_addr learned my addr 172.21.15.89:0/1804308890 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/1804308890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d0ab40 0x5580c2d09090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f86ffd640 1 -- 172.21.15.89:0/1804308890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d0ab40 msgr2=0x5580c2d09090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:12.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/1804308890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d0ab40 0x5580c2d09090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f86ffd640 1 -- 172.21.15.89:0/1804308890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 msgr2=0x5580c2d1ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/1804308890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 0x5580c2d1ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f8c993640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f740a3240 0x7f9f740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/1804308890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 0x5580c2d1ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f86ffd640 1 -- 172.21.15.89:0/1804308890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580c2bfb3a0 con 0x5580c2d095d0 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/1804308890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 0x5580c2d1ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/1804308890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d09a10 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800bd10 tx=0x7f9f780107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb077c9365087f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f740a3240 0x7f9f740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f8c993640 1 -- 172.21.15.89:0/1993232141 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9f800f10b0 con 0x7f9f88106f20 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f7dffb640 1 -- 172.21.15.89:0/1804308890 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f78004030 con 0x5580c2d095d0 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f7dffb640 1 -- 172.21.15.89:0/1804308890 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f780041d0 con 0x5580c2d095d0 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.753+0000 7f9f7dffb640 1 -- 172.21.15.89:0/1804308890 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f78011840 con 0x5580c2d095d0 2024-09-15T23:37:12.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.754+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1804308890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 msgr2=0x5580c2d09a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.754+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f740a3240 0x7f9f740a5700 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x5580c2d0a070 tx=0x7f9f70008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.754+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/1804308890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d09a10 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800bd10 tx=0x7f9f780107a0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.754+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1804308890 shutdown_connections 2024-09-15T23:37:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.754+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/1804308890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 0x5580c2d1ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.754+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/1804308890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d09a10 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.754+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/1804308890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d0ab40 0x5580c2d09090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.754+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1804308890 >> 172.21.15.89:0/1804308890 conn(0x5580c2cd2310 msgr2=0x5580c2d05c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.754+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1804308890 shutdown_connections 2024-09-15T23:37:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.754+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1804308890 wait complete. 2024-09-15T23:37:12.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f8fd6e740 1 -- start start 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f8fd6e740 1 Processor -- start 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 0x5580c2d18190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d15440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 0x5580c2d15a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5580c2d1b480 con 0x5580c2d0ab40 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580c2c2a060 con 0x5580c2d095d0 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5580c2d1b600 con 0x5580c2d18780 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f8fd6e740 10 client.0 _ll_register_callbacks cb 0x5580c297d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f7e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d15440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f7e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d15440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49404/0 (socket says 172.21.15.89:49404) 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f7e7fc640 1 -- 172.21.15.89:0/1313743626 learned_addr learned my addr 172.21.15.89:0/1313743626 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f877fe640 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 0x5580c2d15a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 0x5580c2d18190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f7e7fc640 1 -- 172.21.15.89:0/1313743626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 msgr2=0x5580c2d15a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 0x5580c2d15a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f7e7fc640 1 -- 172.21.15.89:0/1313743626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 msgr2=0x5580c2d18190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 0x5580c2d18190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.755+0000 7f9f7e7fc640 1 -- 172.21.15.89:0/1313743626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580c29e03c0 con 0x5580c2d0ab40 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 0x5580c2d18190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f877fe640 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 0x5580c2d15a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d15440 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f9f70016700 tx=0x7f9f700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b471ccbee4adb27e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f66ffd640 1 -- 172.21.15.89:0/1313743626 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f70026060 con 0x5580c2d0ab40 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f66ffd640 1 -- 172.21.15.89:0/1313743626 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9f70014800 con 0x5580c2d0ab40 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1313743626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f780041d0 con 0x5580c2d0ab40 2024-09-15T23:37:12.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f66ffd640 1 -- 172.21.15.89:0/1313743626 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f7002d050 con 0x5580c2d0ab40 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f8fd6e740 10 client.19905 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1313743626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5580c2d26f20 con 0x5580c2d0ab40 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f66ffd640 1 -- 172.21.15.89:0/1313743626 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f780041d0 con 0x5580c2d0ab40 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f66ffd640 1 -- 172.21.15.89:0/1313743626 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9f70021c60 con 0x5580c2d0ab40 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f8fd6e740 10 client.19905 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f8fd6e740 10 client.19905 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f8fd6e740 20 client.19905 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5580c2d283e0 0x5580c2d2a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f8fd6e740 4 client.19905 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.756+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1313743626 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580c2c7c860 con 0x5580c2d283e0 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.757+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5580c2d283e0 0x5580c2d2a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.757+0000 7f9f66ffd640 10 client.19905 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:37:12.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.757+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5580c2d283e0 0x5580c2d2a820 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=90623079fb32d0e9 server_cookie=74aeefe73f59a33 in_seq=0 out_seq=0 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f66ffd640 1 -- 172.21.15.89:0/1313743626 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5580c2c7c860 con 0x5580c2d283e0 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f66ffd640 10 client.19905 handle_command_reply: tid=0 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f8fd6e740 1 client.19905 shutdown 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1313743626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 msgr2=0x5580c2d15440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d15440 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f9f70016700 tx=0x7f9f700046c0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1313743626 shutdown_connections 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5580c2d283e0 0x5580c2d2a820 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 0x5580c2d15a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d15440 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f66ffd640 0 client.19905 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/1313743626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 0x5580c2d18190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.758+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1313743626 >> 172.21.15.89:0/1313743626 conn(0x5580c2cd2310 msgr2=0x5580c2d0b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:12.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.759+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1313743626 shutdown_connections 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.759+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/1313743626 wait complete. 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.759+0000 7f9f8fd6e740 20 client.19905 trim_cache size 0 max 16384 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.771+0000 7f9f8fd6e740 1 Processor -- start 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.771+0000 7f9f8fd6e740 1 -- start start 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.771+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d0a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.771+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d0ab40 0x5580c2d24720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.771+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 0x5580c2d05700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.771+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9f881acc20 con 0x5580c2d095d0 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9f881ae760 con 0x5580c2d18780 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f881ad2f0 con 0x5580c2d0ab40 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f86ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d0a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f86ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d0a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49412/0 (socket says 172.21.15.89:49412) 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f86ffd640 1 -- 172.21.15.89:0/2682954422 learned_addr learned my addr 172.21.15.89:0/2682954422 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:12.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f877fe640 1 --2- 172.21.15.89:0/2682954422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 0x5580c2d05700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f86ffd640 1 -- 172.21.15.89:0/2682954422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d0ab40 msgr2=0x5580c2d24720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/2682954422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d0ab40 0x5580c2d24720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f86ffd640 1 -- 172.21.15.89:0/2682954422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 msgr2=0x5580c2d05700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/2682954422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 0x5580c2d05700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f86ffd640 1 -- 172.21.15.89:0/2682954422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580c2d04310 con 0x5580c2d095d0 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/2682954422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d0a300 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800b3d0 tx=0x7f9f7800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68ef703cdbdf024 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f7d7fa640 1 -- 172.21.15.89:0/2682954422 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f780037f0 con 0x5580c2d095d0 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f7d7fa640 1 -- 172.21.15.89:0/2682954422 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f78005520 con 0x5580c2d095d0 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.772+0000 7f9f7d7fa640 1 -- 172.21.15.89:0/2682954422 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f780278e0 con 0x5580c2d095d0 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2682954422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 msgr2=0x5580c2d0a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/2682954422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d0a300 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800b3d0 tx=0x7f9f7800b900 comp rx=0 tx=0).stop 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2682954422 shutdown_connections 2024-09-15T23:37:12.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/2682954422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d18780 0x5580c2d05700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/2682954422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d0ab40 0x5580c2d24720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/2682954422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d095d0 0x5580c2d0a300 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2682954422 >> 172.21.15.89:0/2682954422 conn(0x5580c2cd2310 msgr2=0x5580c2d00a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2682954422 shutdown_connections 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2682954422 wait complete. 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 -- start start 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.773+0000 7f9f8fd6e740 1 Processor -- start 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 0x5580c2d20d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d11f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f8fd6e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 0x5580c2d12450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9f881ae090 con 0x5580c2d0ab40 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9f881ad9c0 con 0x5580c2d095d0 2024-09-15T23:37:12.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f8fd6e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f881aac00 con 0x5580c2d18780 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f8fd6e740 10 client.0 _ll_register_callbacks cb 0x7f9f7000a8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f7e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d11f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f7e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d11f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49428/0 (socket says 172.21.15.89:49428) 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f7e7fc640 1 -- 172.21.15.89:0/2583601455 learned_addr learned my addr 172.21.15.89:0/2583601455 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 0x5580c2d20d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f877fe640 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 0x5580c2d12450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f7e7fc640 1 -- 172.21.15.89:0/2583601455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 msgr2=0x5580c2d12450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 0x5580c2d12450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f7e7fc640 1 -- 172.21.15.89:0/2583601455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 msgr2=0x5580c2d20d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 0x5580c2d20d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f7e7fc640 1 -- 172.21.15.89:0/2583601455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580c2d163e0 con 0x5580c2d0ab40 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f7e7fc640 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d11f10 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f9f70013420 tx=0x7f9f700138a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695a94e14e5fe059 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:12.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f66ffd640 1 -- 172.21.15.89:0/2583601455 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f70013a40 con 0x5580c2d0ab40 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f66ffd640 1 -- 172.21.15.89:0/2583601455 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9f70013be0 con 0x5580c2d0ab40 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.774+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2583601455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f78005520 con 0x5580c2d0ab40 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f66ffd640 1 -- 172.21.15.89:0/2583601455 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f70026020 con 0x5580c2d0ab40 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f8fd6e740 10 client.19911 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2583601455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5580c2d92cc0 con 0x5580c2d0ab40 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f66ffd640 1 -- 172.21.15.89:0/2583601455 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f78005520 con 0x5580c2d0ab40 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f66ffd640 1 -- 172.21.15.89:0/2583601455 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9f70014c80 con 0x5580c2d0ab40 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f8fd6e740 10 client.19911 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f8fd6e740 10 client.19911 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f8fd6e740 20 client.19911 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5580c2d933f0 0x5580c2d95810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f8fd6e740 4 client.19911 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2583601455 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5580c2c76d70 con 0x5580c2d933f0 2024-09-15T23:37:12.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.775+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5580c2d933f0 0x5580c2d95810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f66ffd640 10 client.19911 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f86ffd640 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5580c2d933f0 0x5580c2d95810 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d6c047d2f85928c3 server_cookie=44e3945edeb68c13 in_seq=0 out_seq=0 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f66ffd640 1 -- 172.21.15.89:0/2583601455 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5580c2c76d70 con 0x5580c2d933f0 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f66ffd640 10 client.19911 handle_command_reply: tid=0 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 client.19911 shutdown 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2583601455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 msgr2=0x5580c2d11f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d11f10 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f9f70013420 tx=0x7f9f700138a0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2583601455 shutdown_connections 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5580c2d933f0 0x5580c2d95810 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580c2d18780 0x5580c2d12450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f66ffd640 0 client.19911 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580c2d0ab40 0x5580c2d11f10 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 --2- 172.21.15.89:0/2583601455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580c2d095d0 0x5580c2d20d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2583601455 >> 172.21.15.89:0/2583601455 conn(0x5580c2cd2310 msgr2=0x5580c2d05700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:12.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2583601455 shutdown_connections 2024-09-15T23:37:12.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.776+0000 7f9f8fd6e740 1 -- 172.21.15.89:0/2583601455 wait complete. 2024-09-15T23:37:12.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8fd6e740 20 client.19911 trim_cache size 0 max 16384 2024-09-15T23:37:12.797 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 -- 172.21.15.89:0/1993232141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f740a3240 msgr2=0x7f9f740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f740a3240 0x7f9f740a5700 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x5580c2d0a070 tx=0x7f9f70008040 comp rx=0 tx=0).stop 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 -- 172.21.15.89:0/1993232141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f88106f20 msgr2=0x7f9f8819ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f88106f20 0x7f9f8819ccd0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f9f80031a20 tx=0x7f9f80031a50 comp rx=0 tx=0).stop 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 -- 172.21.15.89:0/1993232141 shutdown_connections 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9f740a3240 0x7f9f740a5700 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9f88106f20 0x7f9f8819ccd0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f881027a0 0x7f9f88198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 --2- 172.21.15.89:0/1993232141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f88101df0 0x7f9f88198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 -- 172.21.15.89:0/1993232141 >> 172.21.15.89:0/1993232141 conn(0x7f9f880fd8b0 msgr2=0x7f9f880ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 -- 172.21.15.89:0/1993232141 shutdown_connections 2024-09-15T23:37:12.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:12.777+0000 7f9f8d995640 1 -- 172.21.15.89:0/1993232141 wait complete. 2024-09-15T23:37:13.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:12 smithi089 ceph-mon[30933]: pgmap v745: 129 pgs: 129 active+clean; 2.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:37:13.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:12 smithi120 ceph-mon[35250]: pgmap v745: 129 pgs: 129 active+clean; 2.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:37:13.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:12 smithi174 ceph-mon[35586]: pgmap v745: 129 pgs: 129 active+clean; 2.5 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:37:13.595 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:37:13.842 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:14.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.340+0000 7fe35479e640 1 -- 172.21.15.89:0/4292033683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c106a80 msgr2=0x7fe34c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:14.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.340+0000 7fe35479e640 1 --2- 172.21.15.89:0/4292033683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c106a80 0x7fe34c106e60 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fe340009930 tx=0x7fe34002a420 comp rx=0 tx=0).stop 2024-09-15T23:37:14.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.340+0000 7fe35479e640 1 -- 172.21.15.89:0/4292033683 shutdown_connections 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.340+0000 7fe35479e640 1 --2- 172.21.15.89:0/4292033683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe34c101420 0x7fe34c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.340+0000 7fe35479e640 1 --2- 172.21.15.89:0/4292033683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe34c100a80 0x7fe34c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.340+0000 7fe35479e640 1 --2- 172.21.15.89:0/4292033683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c106a80 0x7fe34c106e60 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.340+0000 7fe35479e640 1 -- 172.21.15.89:0/4292033683 >> 172.21.15.89:0/4292033683 conn(0x7fe34c0fc7a0 msgr2=0x7fe34c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.340+0000 7fe35479e640 1 -- 172.21.15.89:0/4292033683 shutdown_connections 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.340+0000 7fe35479e640 1 -- 172.21.15.89:0/4292033683 wait complete. 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe35479e640 1 Processor -- start 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe35479e640 1 -- start start 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe35479e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe34c100a80 0x7fe34c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe35479e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c101420 0x7fe34c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe35479e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe34c106a80 0x7fe34c19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe35479e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c110e80 con 0x7fe34c101420 2024-09-15T23:37:14.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe35479e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c110d00 con 0x7fe34c106a80 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe35479e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c111000 con 0x7fe34c100a80 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe351d39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c101420 0x7fe34c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe351d39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c101420 0x7fe34c1988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49456/0 (socket says 172.21.15.89:49456) 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe351d39640 1 -- 172.21.15.89:0/3920156142 learned_addr learned my addr 172.21.15.89:0/3920156142 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe35253a640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe34c100a80 0x7fe34c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe352d3b640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe34c106a80 0x7fe34c19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe351d39640 1 -- 172.21.15.89:0/3920156142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe34c100a80 msgr2=0x7fe34c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe351d39640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe34c100a80 0x7fe34c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe351d39640 1 -- 172.21.15.89:0/3920156142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe34c106a80 msgr2=0x7fe34c19cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe351d39640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe34c106a80 0x7fe34c19cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe351d39640 1 -- 172.21.15.89:0/3920156142 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34c19d350 con 0x7fe34c101420 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.341+0000 7fe351d39640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c101420 0x7fe34c1988e0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fe34800d950 tx=0x7fe34800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.342+0000 7fe33f7fe640 1 -- 172.21.15.89:0/3920156142 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348017030 con 0x7fe34c101420 2024-09-15T23:37:14.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.342+0000 7fe33f7fe640 1 -- 172.21.15.89:0/3920156142 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe348004020 con 0x7fe34c101420 2024-09-15T23:37:14.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.342+0000 7fe35479e640 1 -- 172.21.15.89:0/3920156142 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe34c19d640 con 0x7fe34c101420 2024-09-15T23:37:14.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.342+0000 7fe33f7fe640 1 -- 172.21.15.89:0/3920156142 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3480047a0 con 0x7fe34c101420 2024-09-15T23:37:14.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.342+0000 7fe35479e640 1 -- 172.21.15.89:0/3920156142 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe34c1a4f20 con 0x7fe34c101420 2024-09-15T23:37:14.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.344+0000 7fe33f7fe640 1 -- 172.21.15.89:0/3920156142 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe3480041d0 con 0x7fe34c101420 2024-09-15T23:37:14.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.344+0000 7fe356b77740 1 Processor -- start 2024-09-15T23:37:14.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe356b77740 1 -- start start 2024-09-15T23:37:14.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe33f7fe640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3380a32c0 0x7fe3380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe352d3b640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3380a32c0 0x7fe3380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe356b77740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0fb40 0x5618ebb0e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe33f7fe640 1 -- 172.21.15.89:0/3920156142 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe3480cf5e0 con 0x7fe34c101420 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe356b77740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb0ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe356b77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb19370 0x5618ebb1b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe356b77740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5618eba742f0 con 0x5618ebb0fb40 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe356b77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618eba1bcf0 con 0x5618ebb19370 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe356b77740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5618eba702b0 con 0x5618ebb0e5d0 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe35253a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0fb40 0x5618ebb0e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe35253a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0fb40 0x5618ebb0e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49472/0 (socket says 172.21.15.89:49472) 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe35253a640 1 -- 172.21.15.89:0/1315435606 learned_addr learned my addr 172.21.15.89:0/1315435606 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe352d3b640 1 --2- 172.21.15.89:0/1315435606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb19370 0x5618ebb1b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe351d39640 1 --2- 172.21.15.89:0/1315435606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb0ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe352d3b640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3380a32c0 0x7fe3380a5780 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe344007de0 tx=0x7fe34400f790 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe35253a640 1 -- 172.21.15.89:0/1315435606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 msgr2=0x5618ebb0ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.345+0000 7fe35253a640 1 --2- 172.21.15.89:0/1315435606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb0ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.346+0000 7fe35253a640 1 -- 172.21.15.89:0/1315435606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb19370 msgr2=0x5618ebb1b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.346+0000 7fe35253a640 1 --2- 172.21.15.89:0/1315435606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb19370 0x5618ebb1b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.346+0000 7fe35253a640 1 -- 172.21.15.89:0/1315435606 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618eba003a0 con 0x5618ebb0fb40 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.346+0000 7fe351d39640 1 --2- 172.21.15.89:0/1315435606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb0ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.346+0000 7fe35253a640 1 --2- 172.21.15.89:0/1315435606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0fb40 0x5618ebb0e090 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fe340009930 tx=0x7fe340031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8170bfc01cb8fdd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.346+0000 7fe33cff9640 1 -- 172.21.15.89:0/1315435606 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe340040070 con 0x5618ebb0fb40 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.346+0000 7fe33cff9640 1 -- 172.21.15.89:0/1315435606 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe340005020 con 0x5618ebb0fb40 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.346+0000 7fe33cff9640 1 -- 172.21.15.89:0/1315435606 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe340035040 con 0x5618ebb0fb40 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 -- 172.21.15.89:0/1315435606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0fb40 msgr2=0x5618ebb0e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 --2- 172.21.15.89:0/1315435606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0fb40 0x5618ebb0e090 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fe340009930 tx=0x7fe340031630 comp rx=0 tx=0).stop 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 -- 172.21.15.89:0/1315435606 shutdown_connections 2024-09-15T23:37:14.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 --2- 172.21.15.89:0/1315435606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb19370 0x5618ebb1b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 --2- 172.21.15.89:0/1315435606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb0ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 --2- 172.21.15.89:0/1315435606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0fb40 0x5618ebb0e090 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 -- 172.21.15.89:0/1315435606 >> 172.21.15.89:0/1315435606 conn(0x5618ebad7310 msgr2=0x5618ebb0ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 -- 172.21.15.89:0/1315435606 shutdown_connections 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 -- 172.21.15.89:0/1315435606 wait complete. 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 -- start start 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 Processor -- start 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb18c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb15eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb163f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5618ebb1c010 con 0x5618ebb19370 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618eba2f060 con 0x5618ebb0fb40 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5618ebb1c190 con 0x5618ebb0e5d0 2024-09-15T23:37:14.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.347+0000 7fe356b77740 10 client.0 _ll_register_callbacks cb 0x5618eb7828c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe352d3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb163f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe352d3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb163f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49476/0 (socket says 172.21.15.89:49476) 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe352d3b640 1 -- 172.21.15.89:0/578651192 learned_addr learned my addr 172.21.15.89:0/578651192 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe35253a640 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb18c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe351d39640 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb15eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe352d3b640 1 -- 172.21.15.89:0/578651192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 msgr2=0x5618ebb18c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe352d3b640 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb18c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe352d3b640 1 -- 172.21.15.89:0/578651192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 msgr2=0x5618ebb15eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe352d3b640 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb15eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe352d3b640 1 -- 172.21.15.89:0/578651192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618eb7e53c0 con 0x5618ebb19370 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe351d39640 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb15eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe352d3b640 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb163f0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fe344012e90 tx=0x7fe344004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7e396f53e20e34 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe335ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:14.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe335ffb640 1 -- 172.21.15.89:0/578651192 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344027640 con 0x5618ebb19370 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe335ffb640 1 -- 172.21.15.89:0/578651192 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe344010430 con 0x5618ebb19370 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe356b77740 1 -- 172.21.15.89:0/578651192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe340005020 con 0x5618ebb19370 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe335ffb640 1 -- 172.21.15.89:0/578651192 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe3440277e0 con 0x5618ebb19370 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe356b77740 10 client.19923 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.348+0000 7fe356b77740 1 -- 172.21.15.89:0/578651192 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618ebb279c0 con 0x5618ebb19370 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.349+0000 7fe335ffb640 1 -- 172.21.15.89:0/578651192 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe340005020 con 0x5618ebb19370 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.349+0000 7fe335ffb640 1 -- 172.21.15.89:0/578651192 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe344025070 con 0x5618ebb19370 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.349+0000 7fe356b77740 10 client.19923 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.349+0000 7fe356b77740 10 client.19923 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.349+0000 7fe356b77740 20 client.19923 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.349+0000 7fe356b77740 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618ebb28e80 0x5618ebb2b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.349+0000 7fe356b77740 4 client.19923 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.349+0000 7fe356b77740 1 -- 172.21.15.89:0/578651192 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618eba81860 con 0x5618ebb28e80 2024-09-15T23:37:14.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.349+0000 7fe35253a640 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618ebb28e80 0x5618ebb2b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe335ffb640 10 client.19923 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe35253a640 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618ebb28e80 0x5618ebb2b320 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bbb7ce5dd1be8eea server_cookie=3b03acb0706efc5a in_seq=0 out_seq=0 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe335ffb640 1 -- 172.21.15.89:0/578651192 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5618eba81860 con 0x5618ebb28e80 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe335ffb640 10 client.19923 handle_command_reply: tid=0 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe356b77740 1 client.19923 shutdown 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe356b77740 1 -- 172.21.15.89:0/578651192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 msgr2=0x5618ebb163f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe356b77740 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb163f0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fe344012e90 tx=0x7fe344004ab0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe356b77740 1 -- 172.21.15.89:0/578651192 shutdown_connections 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe356b77740 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618ebb28e80 0x5618ebb2b320 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe356b77740 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb163f0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe356b77740 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb15eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe356b77740 1 --2- 172.21.15.89:0/578651192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb18c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.350+0000 7fe356b77740 1 -- 172.21.15.89:0/578651192 >> 172.21.15.89:0/578651192 conn(0x5618ebad7310 msgr2=0x5618ebb10960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.351+0000 7fe335ffb640 0 client.19923 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:37:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.351+0000 7fe356b77740 1 -- 172.21.15.89:0/578651192 shutdown_connections 2024-09-15T23:37:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.351+0000 7fe356b77740 1 -- 172.21.15.89:0/578651192 wait complete. 2024-09-15T23:37:14.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.351+0000 7fe356b77740 20 client.19923 trim_cache size 0 max 16384 2024-09-15T23:37:14.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.363+0000 7fe356b77740 1 Processor -- start 2024-09-15T23:37:14.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.363+0000 7fe356b77740 1 -- start start 2024-09-15T23:37:14.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.363+0000 7fe356b77740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0e5d0 0x5618ebb252b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe356b77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb30c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe356b77740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb19370 0x5618ebb31180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe356b77740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1aedc0 con 0x5618ebb0e5d0 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe356b77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1b0900 con 0x5618ebb0fb40 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe356b77740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1af490 con 0x5618ebb19370 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe35253a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0e5d0 0x5618ebb252b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe35253a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0e5d0 0x5618ebb252b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49492/0 (socket says 172.21.15.89:49492) 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe35253a640 1 -- 172.21.15.89:0/3321793954 learned_addr learned my addr 172.21.15.89:0/3321793954 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe35253a640 1 -- 172.21.15.89:0/3321793954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb19370 msgr2=0x5618ebb31180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe352d3b640 1 --2- 172.21.15.89:0/3321793954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb19370 0x5618ebb31180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe35253a640 1 --2- 172.21.15.89:0/3321793954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb19370 0x5618ebb31180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe35253a640 1 -- 172.21.15.89:0/3321793954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 msgr2=0x5618ebb30c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe35253a640 1 --2- 172.21.15.89:0/3321793954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb30c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe35253a640 1 -- 172.21.15.89:0/3321793954 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618ebb0b960 con 0x5618ebb0e5d0 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe352d3b640 1 --2- 172.21.15.89:0/3321793954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb19370 0x5618ebb31180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe35253a640 1 --2- 172.21.15.89:0/3321793954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0e5d0 0x5618ebb252b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fe340009930 tx=0x7fe3400058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bef328d5a2b962 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:14.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe337fff640 1 -- 172.21.15.89:0/3321793954 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe340056070 con 0x5618ebb0e5d0 2024-09-15T23:37:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe337fff640 1 -- 172.21.15.89:0/3321793954 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe340048070 con 0x5618ebb0e5d0 2024-09-15T23:37:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.364+0000 7fe337fff640 1 -- 172.21.15.89:0/3321793954 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe340040040 con 0x5618ebb0e5d0 2024-09-15T23:37:14.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 -- 172.21.15.89:0/3321793954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0e5d0 msgr2=0x5618ebb252b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:14.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 --2- 172.21.15.89:0/3321793954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0e5d0 0x5618ebb252b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fe340009930 tx=0x7fe3400058d0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 -- 172.21.15.89:0/3321793954 shutdown_connections 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 --2- 172.21.15.89:0/3321793954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb19370 0x5618ebb31180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 --2- 172.21.15.89:0/3321793954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb30c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 --2- 172.21.15.89:0/3321793954 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb0e5d0 0x5618ebb252b0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 -- 172.21.15.89:0/3321793954 >> 172.21.15.89:0/3321793954 conn(0x5618ebad7310 msgr2=0x5618ebb14780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 -- 172.21.15.89:0/3321793954 shutdown_connections 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 -- 172.21.15.89:0/3321793954 wait complete. 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 -- start start 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.365+0000 7fe356b77740 1 Processor -- start 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe356b77740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb214f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe356b77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb21a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe356b77740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb1c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe356b77740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1b0230 con 0x5618ebb19370 2024-09-15T23:37:14.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe356b77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1afb60 con 0x5618ebb0fb40 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe356b77740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c1acda0 con 0x5618ebb0e5d0 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe356b77740 10 client.0 _ll_register_callbacks cb 0x7fe344012b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe352d3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb1c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe352d3b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb1c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:49502/0 (socket says 172.21.15.89:49502) 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe352d3b640 1 -- 172.21.15.89:0/3258234944 learned_addr learned my addr 172.21.15.89:0/3258234944 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe35253a640 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb214f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe352d3b640 1 -- 172.21.15.89:0/3258234944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 msgr2=0x5618ebb214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe351d39640 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb21a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe352d3b640 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb214f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe352d3b640 1 -- 172.21.15.89:0/3258234944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 msgr2=0x5618ebb21a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe352d3b640 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb21a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe352d3b640 1 -- 172.21.15.89:0/3258234944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618ebb1d1b0 con 0x5618ebb19370 2024-09-15T23:37:14.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe352d3b640 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb1c980 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fe344010e60 tx=0x7fe344027670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1671183e5c19563d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe335ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe335ffb640 1 -- 172.21.15.89:0/3258234944 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34403a060 con 0x5618ebb19370 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe356b77740 1 -- 172.21.15.89:0/3258234944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe340048070 con 0x5618ebb19370 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.366+0000 7fe335ffb640 1 -- 172.21.15.89:0/3258234944 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe344027d90 con 0x5618ebb19370 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe335ffb640 1 -- 172.21.15.89:0/3258234944 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe344041050 con 0x5618ebb19370 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe356b77740 10 client.19929 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe356b77740 1 -- 172.21.15.89:0/3258234944 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618ebb93660 con 0x5618ebb19370 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe335ffb640 1 -- 172.21.15.89:0/3258234944 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe340048070 con 0x5618ebb19370 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe335ffb640 1 -- 172.21.15.89:0/3258234944 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe344025040 con 0x5618ebb19370 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe356b77740 10 client.19929 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe356b77740 10 client.19929 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe356b77740 20 client.19929 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe356b77740 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618ebb93db0 0x5618ebb96210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:14.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe356b77740 4 client.19929 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe356b77740 1 -- 172.21.15.89:0/3258234944 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5618eba7bd70 con 0x5618ebb93db0 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.367+0000 7fe35253a640 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618ebb93db0 0x5618ebb96210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe35253a640 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618ebb93db0 0x5618ebb96210 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1e163a91926604b8 server_cookie=ad9e94e7560c40b3 in_seq=0 out_seq=0 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe335ffb640 10 client.19929 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe335ffb640 1 -- 172.21.15.89:0/3258234944 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5618eba7bd70 con 0x5618ebb93db0 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe335ffb640 10 client.19929 handle_command_reply: tid=0 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 client.19929 shutdown 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 -- 172.21.15.89:0/3258234944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 msgr2=0x5618ebb1c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb1c980 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fe344010e60 tx=0x7fe344027670 comp rx=0 tx=0).stop 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 -- 172.21.15.89:0/3258234944 shutdown_connections 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618ebb93db0 0x5618ebb96210 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618ebb19370 0x5618ebb1c980 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ebb0fb40 0x5618ebb21a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 --2- 172.21.15.89:0/3258234944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618ebb0e5d0 0x5618ebb214f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 -- 172.21.15.89:0/3258234944 >> 172.21.15.89:0/3258234944 conn(0x5618ebad7310 msgr2=0x5618ebb17f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe335ffb640 0 client.19929 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:37:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 -- 172.21.15.89:0/3258234944 shutdown_connections 2024-09-15T23:37:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 1 -- 172.21.15.89:0/3258234944 wait complete. 2024-09-15T23:37:14.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.368+0000 7fe356b77740 20 client.19929 trim_cache size 0 max 16384 2024-09-15T23:37:14.376 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:37:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 -- 172.21.15.89:0/3920156142 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3380a32c0 msgr2=0x7fe3380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3380a32c0 0x7fe3380a5780 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe344007de0 tx=0x7fe34400f790 comp rx=0 tx=0).stop 2024-09-15T23:37:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 -- 172.21.15.89:0/3920156142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c101420 msgr2=0x7fe34c1988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c101420 0x7fe34c1988e0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fe34800d950 tx=0x7fe34800de10 comp rx=0 tx=0).stop 2024-09-15T23:37:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 -- 172.21.15.89:0/3920156142 shutdown_connections 2024-09-15T23:37:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3380a32c0 0x7fe3380a5780 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe34c106a80 0x7fe34c19cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe34c101420 0x7fe34c1988e0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 --2- 172.21.15.89:0/3920156142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe34c100a80 0x7fe34c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:14.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 -- 172.21.15.89:0/3920156142 >> 172.21.15.89:0/3920156142 conn(0x7fe34c0fc7a0 msgr2=0x7fe34c06a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:14.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 -- 172.21.15.89:0/3920156142 shutdown_connections 2024-09-15T23:37:14.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:14.369+0000 7fe35479e640 1 -- 172.21.15.89:0/3920156142 wait complete. 2024-09-15T23:37:15.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:14 smithi089 ceph-mon[30933]: pgmap v746: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:37:15.129 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:37:15.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:14 smithi120 ceph-mon[35250]: pgmap v746: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:37:15.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:14 smithi174 ceph-mon[35586]: pgmap v746: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:37:15.373 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.868+0000 7fcf22912640 1 -- 172.21.15.89:0/487205951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1c101de0 msgr2=0x7fcf1c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.868+0000 7fcf22912640 1 --2- 172.21.15.89:0/487205951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1c101de0 0x7fcf1c1021c0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009900 tx=0x7fcf0c02a410 comp rx=0 tx=0).stop 2024-09-15T23:37:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.868+0000 7fcf22912640 1 -- 172.21.15.89:0/487205951 shutdown_connections 2024-09-15T23:37:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.868+0000 7fcf22912640 1 --2- 172.21.15.89:0/487205951 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf1c106f10 0x7fcf1c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.868+0000 7fcf22912640 1 --2- 172.21.15.89:0/487205951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1c102790 0x7fcf1c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.868+0000 7fcf22912640 1 --2- 172.21.15.89:0/487205951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1c101de0 0x7fcf1c1021c0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.868+0000 7fcf22912640 1 -- 172.21.15.89:0/487205951 >> 172.21.15.89:0/487205951 conn(0x7fcf1c0fd8c0 msgr2=0x7fcf1c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.868+0000 7fcf22912640 1 -- 172.21.15.89:0/487205951 shutdown_connections 2024-09-15T23:37:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.868+0000 7fcf22912640 1 -- 172.21.15.89:0/487205951 wait complete. 2024-09-15T23:37:15.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf22912640 1 Processor -- start 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf22912640 1 -- start start 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf22912640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1c101de0 0x7fcf1c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf22912640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1c102790 0x7fcf1c198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf22912640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf1c106f10 0x7fcf1c19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf22912640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c111f90 con 0x7fcf1c101de0 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf22912640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c111e10 con 0x7fcf1c102790 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf22912640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c112110 con 0x7fcf1c106f10 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1c101de0 0x7fcf1c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1c101de0 0x7fcf1c1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50736/0 (socket says 172.21.15.89:50736) 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf1bfff640 1 -- 172.21.15.89:0/1627413331 learned_addr learned my addr 172.21.15.89:0/1627413331 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf1b7fe640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1c102790 0x7fcf1c198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.869+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf1c106f10 0x7fcf1c19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.892 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf1bfff640 1 -- 172.21.15.89:0/1627413331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf1c106f10 msgr2=0x7fcf1c19cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf1c106f10 0x7fcf1c19cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf1bfff640 1 -- 172.21.15.89:0/1627413331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1c102790 msgr2=0x7fcf1c198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1c102790 0x7fcf1c198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf1bfff640 1 -- 172.21.15.89:0/1627413331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c19d380 con 0x7fcf1c101de0 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf1c106f10 0x7fcf1c19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1c101de0 0x7fcf1c1983d0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c005ab0 tx=0x7fcf0c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf197fa640 1 -- 172.21.15.89:0/1627413331 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c03f030 con 0x7fcf1c101de0 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf197fa640 1 -- 172.21.15.89:0/1627413331 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf0c003ce0 con 0x7fcf1c101de0 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf22912640 1 -- 172.21.15.89:0/1627413331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1c19d610 con 0x7fcf1c101de0 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf197fa640 1 -- 172.21.15.89:0/1627413331 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c035070 con 0x7fcf1c101de0 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.870+0000 7fcf22912640 1 -- 172.21.15.89:0/1627413331 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf1c1a4f50 con 0x7fcf1c101de0 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.872+0000 7fcf197fa640 1 -- 172.21.15.89:0/1627413331 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcf0c042030 con 0x7fcf1c101de0 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf24ceb740 1 Processor -- start 2024-09-15T23:37:15.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf24ceb740 1 -- start start 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf197fa640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf040a32c0 0x7fcf040a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e41090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e415d0 0x55ad95e41a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e4e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ad95da72f0 con 0x55ad95e4c370 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ad95d4ecf0 con 0x55ad95e42b40 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ad95da32b0 con 0x55ad95e415d0 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf197fa640 1 -- 172.21.15.89:0/1627413331 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcf0c0f0720 con 0x7fcf1c101de0 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf040a32c0 0x7fcf040a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf20eaf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e4e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf20eaf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e4e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50742/0 (socket says 172.21.15.89:50742) 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf20eaf640 1 -- 172.21.15.89:0/3711566474 learned_addr learned my addr 172.21.15.89:0/3711566474 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:15.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e41090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.873+0000 7fcf1b7fe640 1 --2- 172.21.15.89:0/3711566474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e415d0 0x55ad95e41a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf1bfff640 1 -- 172.21.15.89:0/3711566474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e415d0 msgr2=0x55ad95e41a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/3711566474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e415d0 0x55ad95e41a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf1bfff640 1 -- 172.21.15.89:0/3711566474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 msgr2=0x55ad95e4e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/3711566474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e4e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf1bfff640 1 -- 172.21.15.89:0/3711566474 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad95d333a0 con 0x55ad95e42b40 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/3711566474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e4e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf040a32c0 0x7fcf040a5780 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400ec60 tx=0x7fcf14009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/3711566474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e41090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0bc880 tx=0x7fcf0c02a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22e3250051985915 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf027fc640 1 -- 172.21.15.89:0/3711566474 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c049070 con 0x55ad95e42b40 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf027fc640 1 -- 172.21.15.89:0/3711566474 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf0c0bcff0 con 0x55ad95e42b40 2024-09-15T23:37:15.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.874+0000 7fcf027fc640 1 -- 172.21.15.89:0/3711566474 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0c0ca0 con 0x55ad95e42b40 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.875+0000 7fcf24ceb740 1 -- 172.21.15.89:0/3711566474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 msgr2=0x55ad95e41090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.875+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/3711566474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e41090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0bc880 tx=0x7fcf0c02a410 comp rx=0 tx=0).stop 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.875+0000 7fcf24ceb740 1 -- 172.21.15.89:0/3711566474 shutdown_connections 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.875+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/3711566474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e4e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.875+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/3711566474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e415d0 0x55ad95e41a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.875+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/3711566474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e41090 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.875+0000 7fcf24ceb740 1 -- 172.21.15.89:0/3711566474 >> 172.21.15.89:0/3711566474 conn(0x55ad95e0a310 msgr2=0x55ad95e3dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.875+0000 7fcf24ceb740 1 -- 172.21.15.89:0/3711566474 shutdown_connections 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.875+0000 7fcf24ceb740 1 -- 172.21.15.89:0/3711566474 wait complete. 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.875+0000 7fcf24ceb740 1 -- start start 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf24ceb740 1 Processor -- start 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e415d0 0x55ad95e4bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e42b40 0x55ad95e49050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 0x55ad95e497e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55ad95e4ee30 con 0x55ad95e42b40 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ad95d62060 con 0x55ad95e415d0 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55ad95e4efb0 con 0x55ad95e4c370 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf24ceb740 10 client.0 _ll_register_callbacks cb 0x55ad95ab58c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf1b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e42b40 0x55ad95e49050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf1b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e42b40 0x55ad95e49050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50752/0 (socket says 172.21.15.89:50752) 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf1b7fe640 1 -- 172.21.15.89:0/1489359049 learned_addr learned my addr 172.21.15.89:0/1489359049 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e415d0 0x55ad95e4bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 0x55ad95e497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf1b7fe640 1 -- 172.21.15.89:0/1489359049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 msgr2=0x55ad95e497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf1b7fe640 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 0x55ad95e497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf1b7fe640 1 -- 172.21.15.89:0/1489359049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e415d0 msgr2=0x55ad95e4bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:15.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf1b7fe640 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e415d0 0x55ad95e4bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf1b7fe640 1 -- 172.21.15.89:0/1489359049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad95b183c0 con 0x55ad95e42b40 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 0x55ad95e497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcf1b7fe640 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e42b40 0x55ad95e49050 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fcf10004650 tx=0x7fcf100039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1604677b9e3a746f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcefb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.876+0000 7fcefb7fe640 1 -- 172.21.15.89:0/1489359049 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf10011910 con 0x55ad95e42b40 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcefb7fe640 1 -- 172.21.15.89:0/1489359049 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcf10011ab0 con 0x55ad95e42b40 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1489359049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf0c0bcff0 con 0x55ad95e42b40 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcefb7fe640 1 -- 172.21.15.89:0/1489359049 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf10011cb0 con 0x55ad95e42b40 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcf24ceb740 10 client.19941 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1489359049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ad95e5a8b0 con 0x55ad95e42b40 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcefb7fe640 1 -- 172.21.15.89:0/1489359049 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf0c0bcff0 con 0x55ad95e42b40 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcefb7fe640 1 -- 172.21.15.89:0/1489359049 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcf10018070 con 0x55ad95e42b40 2024-09-15T23:37:15.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcf24ceb740 10 client.19941 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcf24ceb740 10 client.19941 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcf24ceb740 20 client.19941 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ad95e5be10 0x55ad95e5e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcf24ceb740 4 client.19941 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1489359049 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad95db4860 con 0x55ad95e5be10 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.877+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ad95e5be10 0x55ad95e5e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.878+0000 7fcefb7fe640 10 client.19941 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.878+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ad95e5be10 0x55ad95e5e230 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=30a151261b289c2a server_cookie=13b034365ba7511a in_seq=0 out_seq=0 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcefb7fe640 1 -- 172.21.15.89:0/1489359049 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ad95db4860 con 0x55ad95e5be10 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcefb7fe640 10 client.19941 handle_command_reply: tid=0 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 client.19941 shutdown 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1489359049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e42b40 msgr2=0x55ad95e49050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:15.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e42b40 0x55ad95e49050 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fcf10004650 tx=0x7fcf100039c0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1489359049 shutdown_connections 2024-09-15T23:37:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ad95e5be10 0x55ad95e5e230 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 0x55ad95e497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e42b40 0x55ad95e49050 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcefb7fe640 0 client.19941 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:37:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/1489359049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e415d0 0x55ad95e4bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1489359049 >> 172.21.15.89:0/1489359049 conn(0x55ad95e0a310 msgr2=0x55ad95e439c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1489359049 shutdown_connections 2024-09-15T23:37:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1489359049 wait complete. 2024-09-15T23:37:15.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.879+0000 7fcf24ceb740 20 client.19941 trim_cache size 0 max 16384 2024-09-15T23:37:15.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.892+0000 7fcf24ceb740 1 Processor -- start 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.892+0000 7fcf24ceb740 1 -- start start 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.892+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e415d0 0x55ad95e38c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.892+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e48fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.892+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e48090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.892+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c1aedc0 con 0x55ad95e4c370 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.892+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c1b0900 con 0x55ad95e42b40 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.892+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c1af490 con 0x55ad95e415d0 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf20eaf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e48090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf20eaf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e48090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50762/0 (socket says 172.21.15.89:50762) 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf20eaf640 1 -- 172.21.15.89:0/1706167222 learned_addr learned my addr 172.21.15.89:0/1706167222 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf1b7fe640 1 --2- 172.21.15.89:0/1706167222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e48fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/1706167222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e415d0 0x55ad95e38c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf20eaf640 1 -- 172.21.15.89:0/1706167222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e415d0 msgr2=0x55ad95e38c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/1706167222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e415d0 0x55ad95e38c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf20eaf640 1 -- 172.21.15.89:0/1706167222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 msgr2=0x55ad95e48fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/1706167222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e48fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf20eaf640 1 -- 172.21.15.89:0/1706167222 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad95e3e960 con 0x55ad95e4c370 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/1706167222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e48090 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009350 tx=0x7fcf1401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc33942c0bd9f9d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf01ffb640 1 -- 172.21.15.89:0/1706167222 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1401ae00 con 0x55ad95e4c370 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf01ffb640 1 -- 172.21.15.89:0/1706167222 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf1401c8e0 con 0x55ad95e4c370 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.893+0000 7fcf01ffb640 1 -- 172.21.15.89:0/1706167222 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14026770 con 0x55ad95e4c370 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.894+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1706167222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 msgr2=0x55ad95e48090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.894+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/1706167222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e48090 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009350 tx=0x7fcf1401b000 comp rx=0 tx=0).stop 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.894+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1706167222 shutdown_connections 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.894+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/1706167222 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e4c370 0x55ad95e48090 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.894+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/1706167222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e48fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.894+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/1706167222 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e415d0 0x55ad95e38c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.894+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1706167222 >> 172.21.15.89:0/1706167222 conn(0x55ad95e0a310 msgr2=0x55ad95e47980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.894+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1706167222 shutdown_connections 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.894+0000 7fcf24ceb740 1 -- 172.21.15.89:0/1706167222 wait complete. 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf24ceb740 1 -- start start 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf24ceb740 1 Processor -- start 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e415d0 0x55ad95e46de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e54200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf24ceb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 0x55ad95e54990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c1b0230 con 0x55ad95e415d0 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c1afb60 con 0x55ad95e42b40 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf24ceb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c1acda0 con 0x55ad95e4c370 2024-09-15T23:37:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf24ceb740 10 client.0 _ll_register_callbacks cb 0x7fcf10004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e415d0 0x55ad95e46de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e415d0 0x55ad95e46de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50764/0 (socket says 172.21.15.89:50764) 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf1bfff640 1 -- 172.21.15.89:0/4266414654 learned_addr learned my addr 172.21.15.89:0/4266414654 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf1b7fe640 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e54200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.895+0000 7fcf20eaf640 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 0x55ad95e54990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.896+0000 7fcf1bfff640 1 -- 172.21.15.89:0/4266414654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 msgr2=0x55ad95e54990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.896+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 0x55ad95e54990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.896+0000 7fcf1bfff640 1 -- 172.21.15.89:0/4266414654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 msgr2=0x55ad95e54200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.896+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e54200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.896+0000 7fcf1bfff640 1 -- 172.21.15.89:0/4266414654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad95e4d4b0 con 0x55ad95e415d0 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.896+0000 7fcf1b7fe640 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e54200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.896+0000 7fcf1bfff640 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e415d0 0x55ad95e46de0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0bc5f0 tx=0x7fcf0c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b280e63f8f8af6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:15.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.896+0000 7fcefb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcefb7fe640 1 -- 172.21.15.89:0/4266414654 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c049040 con 0x55ad95e415d0 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcefb7fe640 1 -- 172.21.15.89:0/4266414654 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcf0c0edee0 con 0x55ad95e415d0 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcf24ceb740 1 -- 172.21.15.89:0/4266414654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf1401c8e0 con 0x55ad95e415d0 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcefb7fe640 1 -- 172.21.15.89:0/4266414654 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf0c0bcff0 con 0x55ad95e415d0 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcf24ceb740 10 client.19947 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcf24ceb740 1 -- 172.21.15.89:0/4266414654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ad95ec6eb0 con 0x55ad95e415d0 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcefb7fe640 1 -- 172.21.15.89:0/4266414654 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf1401c8e0 con 0x55ad95e415d0 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcefb7fe640 1 -- 172.21.15.89:0/4266414654 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcf0c099380 con 0x55ad95e415d0 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcf24ceb740 10 client.19947 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcf24ceb740 10 client.19947 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcf24ceb740 20 client.19947 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:15.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ad95ec7490 0x55ad95ec9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcf24ceb740 4 client.19947 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcf24ceb740 1 -- 172.21.15.89:0/4266414654 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad95daed70 con 0x55ad95ec7490 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.897+0000 7fcf1b7fe640 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ad95ec7490 0x55ad95ec9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcf1b7fe640 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ad95ec7490 0x55ad95ec9880 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=34f55ee1de7bccf3 server_cookie=55a29e4678013251 in_seq=0 out_seq=0 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcefb7fe640 10 client.19947 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcefb7fe640 1 -- 172.21.15.89:0/4266414654 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad95daed70 con 0x55ad95ec7490 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcefb7fe640 10 client.19947 handle_command_reply: tid=0 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcf24ceb740 1 client.19947 shutdown 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcf24ceb740 1 -- 172.21.15.89:0/4266414654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e415d0 msgr2=0x55ad95e46de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e415d0 0x55ad95e46de0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0bc5f0 tx=0x7fcf0c0045e0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcf24ceb740 1 -- 172.21.15.89:0/4266414654 shutdown_connections 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55ad95ec7490 0x55ad95ec9880 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55ad95e4c370 0x55ad95e54990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ad95e42b40 0x55ad95e54200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcf24ceb740 1 --2- 172.21.15.89:0/4266414654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad95e415d0 0x55ad95e46de0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcf24ceb740 1 -- 172.21.15.89:0/4266414654 >> 172.21.15.89:0/4266414654 conn(0x55ad95e0a310 msgr2=0x55ad95e481a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.898+0000 7fcefb7fe640 0 client.19947 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:37:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf24ceb740 1 -- 172.21.15.89:0/4266414654 shutdown_connections 2024-09-15T23:37:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf24ceb740 1 -- 172.21.15.89:0/4266414654 wait complete. 2024-09-15T23:37:15.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf24ceb740 20 client.19947 trim_cache size 0 max 16384 2024-09-15T23:37:15.912 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:37:15.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 -- 172.21.15.89:0/1627413331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf040a32c0 msgr2=0x7fcf040a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:15.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf040a32c0 0x7fcf040a5780 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400ec60 tx=0x7fcf14009e30 comp rx=0 tx=0).stop 2024-09-15T23:37:15.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 -- 172.21.15.89:0/1627413331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1c101de0 msgr2=0x7fcf1c1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:15.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1c101de0 0x7fcf1c1983d0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c005ab0 tx=0x7fcf0c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 -- 172.21.15.89:0/1627413331 shutdown_connections 2024-09-15T23:37:15.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf040a32c0 0x7fcf040a5780 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf1c106f10 0x7fcf1c19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf1c102790 0x7fcf1c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 --2- 172.21.15.89:0/1627413331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf1c101de0 0x7fcf1c1983d0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 -- 172.21.15.89:0/1627413331 >> 172.21.15.89:0/1627413331 conn(0x7fcf1c0fd8c0 msgr2=0x7fcf1c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 -- 172.21.15.89:0/1627413331 shutdown_connections 2024-09-15T23:37:15.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:15.899+0000 7fcf22912640 1 -- 172.21.15.89:0/1627413331 wait complete. 2024-09-15T23:37:16.798 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:37:17.053 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:17.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:16 smithi089 ceph-mon[30933]: pgmap v747: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:37:17.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:16 smithi120 ceph-mon[35250]: pgmap v747: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:37:17.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:16 smithi174 ceph-mon[35586]: pgmap v747: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:37:17.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.538+0000 7f204fbb2640 1 -- 172.21.15.89:0/1853369357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2048102790 msgr2=0x7f20481067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:17.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.538+0000 7f204fbb2640 1 --2- 172.21.15.89:0/1853369357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2048102790 0x7f20481067e0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f2038009900 tx=0x7f203802a410 comp rx=0 tx=0).stop 2024-09-15T23:37:17.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.538+0000 7f204fbb2640 1 -- 172.21.15.89:0/1853369357 shutdown_connections 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.538+0000 7f204fbb2640 1 --2- 172.21.15.89:0/1853369357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2048106f10 0x7f204810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.538+0000 7f204fbb2640 1 --2- 172.21.15.89:0/1853369357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2048102790 0x7f20481067e0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.538+0000 7f204fbb2640 1 --2- 172.21.15.89:0/1853369357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2048101de0 0x7f20481021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.538+0000 7f204fbb2640 1 -- 172.21.15.89:0/1853369357 >> 172.21.15.89:0/1853369357 conn(0x7f20480fd8c0 msgr2=0x7f20480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.538+0000 7f204fbb2640 1 -- 172.21.15.89:0/1853369357 shutdown_connections 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.538+0000 7f204fbb2640 1 -- 172.21.15.89:0/1853369357 wait complete. 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204fbb2640 1 Processor -- start 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204fbb2640 1 -- start start 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204fbb2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2048101de0 0x7f20480744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204fbb2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2048102790 0x7f2048074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204fbb2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2048106f10 0x7f2048074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204fbb2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2048112010 con 0x7f2048106f10 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204fbb2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2048111e90 con 0x7f2048101de0 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204fbb2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2048112190 con 0x7f2048102790 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204e14f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2048106f10 0x7f2048074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204e14f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2048106f10 0x7f2048074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50784/0 (socket says 172.21.15.89:50784) 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204e14f640 1 -- 172.21.15.89:0/1307290617 learned_addr learned my addr 172.21.15.89:0/1307290617 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204d94e640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2048101de0 0x7f20480744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.539+0000 7f204d14d640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2048102790 0x7f2048074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f204e14f640 1 -- 172.21.15.89:0/1307290617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2048102790 msgr2=0x7f2048074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f204e14f640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2048102790 0x7f2048074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f204e14f640 1 -- 172.21.15.89:0/1307290617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2048101de0 msgr2=0x7f20480744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f204e14f640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2048101de0 0x7f20480744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f204e14f640 1 -- 172.21.15.89:0/1307290617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20481a6a70 con 0x7f2048106f10 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f204e14f640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2048106f10 0x7f2048074f70 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f204000b970 tx=0x7f204000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:17.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f203effd640 1 -- 172.21.15.89:0/1307290617 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2040019070 con 0x7f2048106f10 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f203effd640 1 -- 172.21.15.89:0/1307290617 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2040014440 con 0x7f2048106f10 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f204fbb2640 1 -- 172.21.15.89:0/1307290617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20481a6d60 con 0x7f2048106f10 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f203effd640 1 -- 172.21.15.89:0/1307290617 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20400075d0 con 0x7f2048106f10 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.540+0000 7f204fbb2640 1 -- 172.21.15.89:0/1307290617 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20481a72a0 con 0x7f2048106f10 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.542+0000 7f203effd640 1 -- 172.21.15.89:0/1307290617 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f204001c070 con 0x7f2048106f10 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.542+0000 7f2051f8b740 1 Processor -- start 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f2051f8b740 1 -- start start 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a21b40 0x559da9a20090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a205d0 0x559da9a20a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a2b370 0x559da9a2d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f2051f8b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559da99862f0 con 0x559da9a205d0 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f2051f8b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559da992dcf0 con 0x559da9a21b40 2024-09-15T23:37:17.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f2051f8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559da99822b0 con 0x559da9a2b370 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d94e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a205d0 0x559da9a20a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d94e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a205d0 0x559da9a20a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50786/0 (socket says 172.21.15.89:50786) 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f203effd640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20340a32c0 0x7f20340a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d94e640 1 -- 172.21.15.89:0/4113256987 learned_addr learned my addr 172.21.15.89:0/4113256987 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d14d640 1 --2- 172.21.15.89:0/4113256987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a21b40 0x559da9a20090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204e14f640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20340a32c0 0x7f20340a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d14d640 1 --2- 172.21.15.89:0/4113256987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a2b370 0x559da9a2d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d94e640 1 -- 172.21.15.89:0/4113256987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a2b370 msgr2=0x559da9a2d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d94e640 1 --2- 172.21.15.89:0/4113256987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a2b370 0x559da9a2d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d94e640 1 -- 172.21.15.89:0/4113256987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a21b40 msgr2=0x559da9a20090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f203effd640 1 -- 172.21.15.89:0/1307290617 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f20400cf330 con 0x7f2048106f10 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d94e640 1 --2- 172.21.15.89:0/4113256987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a21b40 0x559da9a20090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d94e640 1 -- 172.21.15.89:0/4113256987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559da99123a0 con 0x559da9a205d0 2024-09-15T23:37:17.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.543+0000 7f204d14d640 1 --2- 172.21.15.89:0/4113256987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a21b40 0x559da9a20090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.544+0000 7f204d94e640 1 --2- 172.21.15.89:0/4113256987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a205d0 0x559da9a20a10 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f2044009a70 tx=0x7f2044009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b460456b675cd9c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.544+0000 7f204e14f640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20340a32c0 0x7f20340a5780 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f204000bea0 tx=0x7f20400c62d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.544+0000 7f2033fff640 1 -- 172.21.15.89:0/4113256987 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044017070 con 0x559da9a205d0 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.544+0000 7f2033fff640 1 -- 172.21.15.89:0/4113256987 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2044013470 con 0x559da9a205d0 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.544+0000 7f2033fff640 1 -- 172.21.15.89:0/4113256987 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044012610 con 0x559da9a205d0 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.544+0000 7f2051f8b740 1 -- 172.21.15.89:0/4113256987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a205d0 msgr2=0x559da9a20a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.544+0000 7f2051f8b740 1 --2- 172.21.15.89:0/4113256987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a205d0 0x559da9a20a10 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f2044009a70 tx=0x7f2044009f30 comp rx=0 tx=0).stop 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 -- 172.21.15.89:0/4113256987 shutdown_connections 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 --2- 172.21.15.89:0/4113256987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a2b370 0x559da9a2d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 --2- 172.21.15.89:0/4113256987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a205d0 0x559da9a20a10 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 --2- 172.21.15.89:0/4113256987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a21b40 0x559da9a20090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 -- 172.21.15.89:0/4113256987 >> 172.21.15.89:0/4113256987 conn(0x559da99e9310 msgr2=0x559da9a1cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 -- 172.21.15.89:0/4113256987 shutdown_connections 2024-09-15T23:37:17.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 -- 172.21.15.89:0/4113256987 wait complete. 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 -- start start 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 Processor -- start 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a205d0 0x559da9a2acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 0x559da9a27f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a284d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559da9a2e070 con 0x559da9a2b370 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559da9941060 con 0x559da9a205d0 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.545+0000 7f2051f8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559da9a2e1f0 con 0x559da9a21b40 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f2051f8b740 10 client.0 _ll_register_callbacks cb 0x559da96948c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d14d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a284d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d14d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a284d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50802/0 (socket says 172.21.15.89:50802) 2024-09-15T23:37:17.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d14d640 1 -- 172.21.15.89:0/3482033188 learned_addr learned my addr 172.21.15.89:0/3482033188 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:17.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d94e640 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 0x559da9a27f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d14d640 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a205d0 0x559da9a2acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d14d640 1 -- 172.21.15.89:0/3482033188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 msgr2=0x559da9a27f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:17.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d14d640 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 0x559da9a27f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d14d640 1 -- 172.21.15.89:0/3482033188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a205d0 msgr2=0x559da9a2acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d14d640 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a205d0 0x559da9a2acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d14d640 1 -- 172.21.15.89:0/3482033188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559da96f73c0 con 0x559da9a2b370 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d94e640 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 0x559da9a27f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f204d14d640 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a284d0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f2038005940 tx=0x7f20380092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89cd1b71ca80e74 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f20317fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f20317fa640 1 -- 172.21.15.89:0/3482033188 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038035040 con 0x559da9a2b370 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f20317fa640 1 -- 172.21.15.89:0/3482033188 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f20380035f0 con 0x559da9a2b370 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f2051f8b740 1 -- 172.21.15.89:0/3482033188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2044013470 con 0x559da9a2b370 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f20317fa640 1 -- 172.21.15.89:0/3482033188 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f20380044d0 con 0x559da9a2b370 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f2051f8b740 10 client.19959 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:17.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.546+0000 7f2051f8b740 1 -- 172.21.15.89:0/3482033188 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559da9a39a80 con 0x559da9a2b370 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f20317fa640 1 -- 172.21.15.89:0/3482033188 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2044013470 con 0x559da9a2b370 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f20317fa640 1 -- 172.21.15.89:0/3482033188 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2038048020 con 0x559da9a2b370 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f2051f8b740 10 client.19959 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f2051f8b740 10 client.19959 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f2051f8b740 20 client.19959 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f2051f8b740 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559da9a3af40 0x559da9a3d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f2051f8b740 4 client.19959 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f2051f8b740 1 -- 172.21.15.89:0/3482033188 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559da9993860 con 0x559da9a3af40 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f204d94e640 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559da9a3af40 0x559da9a3d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f20317fa640 10 client.19959 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.547+0000 7f204d94e640 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559da9a3af40 0x559da9a3d330 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=32f6ab74f50b2665 server_cookie=c16909c4285f7589 in_seq=0 out_seq=0 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f20317fa640 1 -- 172.21.15.89:0/3482033188 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559da9993860 con 0x559da9a3af40 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f20317fa640 10 client.19959 handle_command_reply: tid=0 2024-09-15T23:37:17.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f2051f8b740 1 client.19959 shutdown 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f2051f8b740 1 -- 172.21.15.89:0/3482033188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 msgr2=0x559da9a284d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f2051f8b740 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a284d0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f2038005940 tx=0x7f20380092d0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f2051f8b740 1 -- 172.21.15.89:0/3482033188 shutdown_connections 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f2051f8b740 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559da9a3af40 0x559da9a3d330 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f2051f8b740 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a284d0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f2051f8b740 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 0x559da9a27f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f2051f8b740 1 --2- 172.21.15.89:0/3482033188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a205d0 0x559da9a2acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f20317fa640 0 client.19959 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.548+0000 7f2051f8b740 1 -- 172.21.15.89:0/3482033188 >> 172.21.15.89:0/3482033188 conn(0x559da99e9310 msgr2=0x559da9a22930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.549+0000 7f2051f8b740 1 -- 172.21.15.89:0/3482033188 shutdown_connections 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.549+0000 7f2051f8b740 1 -- 172.21.15.89:0/3482033188 wait complete. 2024-09-15T23:37:17.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.549+0000 7f2051f8b740 20 client.19959 trim_cache size 0 max 16384 2024-09-15T23:37:17.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f2051f8b740 1 Processor -- start 2024-09-15T23:37:17.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f2051f8b740 1 -- start start 2024-09-15T23:37:17.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a205d0 0x559da9a1ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 0x559da9a37310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a42c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f2051f8b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f20481b1100 con 0x559da9a2b370 2024-09-15T23:37:17.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f2051f8b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f20481b2c40 con 0x559da9a205d0 2024-09-15T23:37:17.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f2051f8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f20481b17d0 con 0x559da9a21b40 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f204d14d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a42c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f204d14d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a42c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50806/0 (socket says 172.21.15.89:50806) 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f204d14d640 1 -- 172.21.15.89:0/3486013334 learned_addr learned my addr 172.21.15.89:0/3486013334 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f204d94e640 1 --2- 172.21.15.89:0/3486013334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 0x559da9a37310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f204d14d640 1 --2- 172.21.15.89:0/3486013334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a205d0 0x559da9a1ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f204d14d640 1 -- 172.21.15.89:0/3486013334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 msgr2=0x559da9a37310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f204d14d640 1 --2- 172.21.15.89:0/3486013334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 0x559da9a37310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f204d14d640 1 -- 172.21.15.89:0/3486013334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a205d0 msgr2=0x559da9a1ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f204d14d640 1 --2- 172.21.15.89:0/3486013334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a205d0 0x559da9a1ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.562+0000 7f204d14d640 1 -- 172.21.15.89:0/3486013334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559da9a202d0 con 0x559da9a2b370 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f204d14d640 1 --2- 172.21.15.89:0/3486013334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a42c00 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f2038005910 tx=0x7f2038005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb48ddd84cf80d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f20337fe640 1 -- 172.21.15.89:0/3486013334 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038000e60 con 0x559da9a2b370 2024-09-15T23:37:17.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f20337fe640 1 -- 172.21.15.89:0/3486013334 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20380035f0 con 0x559da9a2b370 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f20337fe640 1 -- 172.21.15.89:0/3486013334 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f203803c6a0 con 0x559da9a2b370 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f2051f8b740 1 -- 172.21.15.89:0/3486013334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 msgr2=0x559da9a42c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f2051f8b740 1 --2- 172.21.15.89:0/3486013334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a42c00 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f2038005910 tx=0x7f2038005a20 comp rx=0 tx=0).stop 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f2051f8b740 1 -- 172.21.15.89:0/3486013334 shutdown_connections 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f2051f8b740 1 --2- 172.21.15.89:0/3486013334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a2b370 0x559da9a42c00 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f2051f8b740 1 --2- 172.21.15.89:0/3486013334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a21b40 0x559da9a37310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f2051f8b740 1 --2- 172.21.15.89:0/3486013334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a205d0 0x559da9a1ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f2051f8b740 1 -- 172.21.15.89:0/3486013334 >> 172.21.15.89:0/3486013334 conn(0x559da99e9310 msgr2=0x559da9a1c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.563+0000 7f2051f8b740 1 -- 172.21.15.89:0/3486013334 shutdown_connections 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.564+0000 7f2051f8b740 1 -- 172.21.15.89:0/3486013334 wait complete. 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.564+0000 7f2051f8b740 1 -- start start 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.564+0000 7f2051f8b740 1 Processor -- start 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.564+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a205d0 0x559da9a33c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.564+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a21b40 0x559da9a2efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.564+0000 7f2051f8b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a2b370 0x559da9a2f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.564+0000 7f2051f8b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f20481b2570 con 0x559da9a21b40 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.564+0000 7f2051f8b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f20481b1ea0 con 0x559da9a2b370 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.564+0000 7f2051f8b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f20481af0e0 con 0x559da9a205d0 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.564+0000 7f2051f8b740 10 client.0 _ll_register_callbacks cb 0x7f2038009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d94e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a21b40 0x559da9a2efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d94e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a21b40 0x559da9a2efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50820/0 (socket says 172.21.15.89:50820) 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d94e640 1 -- 172.21.15.89:0/170020893 learned_addr learned my addr 172.21.15.89:0/170020893 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d14d640 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a205d0 0x559da9a33c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d14d640 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a2b370 0x559da9a2f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d94e640 1 -- 172.21.15.89:0/170020893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a205d0 msgr2=0x559da9a33c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d94e640 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a205d0 0x559da9a33c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d94e640 1 -- 172.21.15.89:0/170020893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a2b370 msgr2=0x559da9a2f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d94e640 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a2b370 0x559da9a2f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d94e640 1 -- 172.21.15.89:0/170020893 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559da9a23b70 con 0x559da9a21b40 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f204d94e640 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a21b40 0x559da9a2efb0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f2044009830 tx=0x7f2044006a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df34ad727a14fc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f20317fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f20317fa640 1 -- 172.21.15.89:0/170020893 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044017040 con 0x559da9a21b40 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f20317fa640 1 -- 172.21.15.89:0/170020893 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2044013470 con 0x559da9a21b40 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f2051f8b740 1 -- 172.21.15.89:0/170020893 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20380035f0 con 0x559da9a21b40 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f20317fa640 1 -- 172.21.15.89:0/170020893 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2044009520 con 0x559da9a21b40 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f2051f8b740 10 client.19965 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.565+0000 7f2051f8b740 1 -- 172.21.15.89:0/170020893 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559da9aa5a10 con 0x559da9a21b40 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f20317fa640 1 -- 172.21.15.89:0/170020893 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20380035f0 con 0x559da9a21b40 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f20317fa640 1 -- 172.21.15.89:0/170020893 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2044006140 con 0x559da9a21b40 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f2051f8b740 10 client.19965 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f2051f8b740 10 client.19965 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:37:17.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f2051f8b740 20 client.19965 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f2051f8b740 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559da9aa5eb0 0x559da9aa82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f2051f8b740 4 client.19965 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f2051f8b740 1 -- 172.21.15.89:0/170020893 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559da998dd70 con 0x559da9aa5eb0 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f204d14d640 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559da9aa5eb0 0x559da9aa82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f204d14d640 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559da9aa5eb0 0x559da9aa82a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=79b1822802ebdca2 server_cookie=e41411f7b3029421 in_seq=0 out_seq=0 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.566+0000 7f20317fa640 10 client.19965 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f20317fa640 1 -- 172.21.15.89:0/170020893 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559da998dd70 con 0x559da9aa5eb0 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f20317fa640 10 client.19965 handle_command_reply: tid=0 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 client.19965 shutdown 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 -- 172.21.15.89:0/170020893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a21b40 msgr2=0x559da9a2efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a21b40 0x559da9a2efb0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f2044009830 tx=0x7f2044006a50 comp rx=0 tx=0).stop 2024-09-15T23:37:17.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 -- 172.21.15.89:0/170020893 shutdown_connections 2024-09-15T23:37:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559da9aa5eb0 0x559da9aa82a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559da9a2b370 0x559da9a2f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559da9a21b40 0x559da9a2efb0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 --2- 172.21.15.89:0/170020893 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559da9a205d0 0x559da9a33c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f20317fa640 0 client.19965 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:37:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 -- 172.21.15.89:0/170020893 >> 172.21.15.89:0/170020893 conn(0x559da99e9310 msgr2=0x559da9a22930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 -- 172.21.15.89:0/170020893 shutdown_connections 2024-09-15T23:37:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 1 -- 172.21.15.89:0/170020893 wait complete. 2024-09-15T23:37:17.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.567+0000 7f2051f8b740 20 client.19965 trim_cache size 0 max 16384 2024-09-15T23:37:17.579 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:37:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 -- 172.21.15.89:0/1307290617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20340a32c0 msgr2=0x7f20340a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20340a32c0 0x7f20340a5780 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f204000bea0 tx=0x7f20400c62d0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 -- 172.21.15.89:0/1307290617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2048106f10 msgr2=0x7f2048074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2048106f10 0x7f2048074f70 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f204000b970 tx=0x7f204000bd70 comp rx=0 tx=0).stop 2024-09-15T23:37:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 -- 172.21.15.89:0/1307290617 shutdown_connections 2024-09-15T23:37:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f20340a32c0 0x7f20340a5780 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2048106f10 0x7f2048074f70 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2048102790 0x7f2048074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 --2- 172.21.15.89:0/1307290617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2048101de0 0x7f20480744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 -- 172.21.15.89:0/1307290617 >> 172.21.15.89:0/1307290617 conn(0x7f20480fd8c0 msgr2=0x7f20480fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 -- 172.21.15.89:0/1307290617 shutdown_connections 2024-09-15T23:37:17.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:17.568+0000 7f204fbb2640 1 -- 172.21.15.89:0/1307290617 wait complete. 2024-09-15T23:37:18.341 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:37:18.586 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:19.002 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:18 smithi089 ceph-mon[30933]: pgmap v748: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:37:19.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.082+0000 7f6624e12640 1 -- 172.21.15.89:0/2449983358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6620103660 msgr2=0x7f662010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:19.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.082+0000 7f6624e12640 1 --2- 172.21.15.89:0/2449983358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6620103660 0x7f662010bb50 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f6608009900 tx=0x7f660802a480 comp rx=0 tx=0).stop 2024-09-15T23:37:19.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.082+0000 7f6624e12640 1 -- 172.21.15.89:0/2449983358 shutdown_connections 2024-09-15T23:37:19.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.082+0000 7f6624e12640 1 --2- 172.21.15.89:0/2449983358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f662010c170 0x7f662010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.082+0000 7f6624e12640 1 --2- 172.21.15.89:0/2449983358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6620103660 0x7f662010bb50 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.082+0000 7f6624e12640 1 --2- 172.21.15.89:0/2449983358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6620102d40 0x7f6620103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.082+0000 7f6624e12640 1 -- 172.21.15.89:0/2449983358 >> 172.21.15.89:0/2449983358 conn(0x7f66200fc7a0 msgr2=0x7f66200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:19.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.082+0000 7f6624e12640 1 -- 172.21.15.89:0/2449983358 shutdown_connections 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.082+0000 7f6624e12640 1 -- 172.21.15.89:0/2449983358 wait complete. 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f6624e12640 1 Processor -- start 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f6624e12640 1 -- start start 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f6624e12640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6620102d40 0x7f6620198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f6624e12640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6620103660 0x7f6620198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f6624e12640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f662010c170 0x7f662019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f6624e12640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6620111f70 con 0x7f6620102d40 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f6624e12640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6620111df0 con 0x7f6620103660 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f6624e12640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f66201120f0 con 0x7f662010c170 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f661e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6620102d40 0x7f6620198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f661e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6620102d40 0x7f6620198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50838/0 (socket says 172.21.15.89:50838) 2024-09-15T23:37:19.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f661e59c640 1 -- 172.21.15.89:0/702750003 learned_addr learned my addr 172.21.15.89:0/702750003 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f6615d9b640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6620103660 0x7f6620198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.083+0000 7f661ed9d640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f662010c170 0x7f662019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f661e59c640 1 -- 172.21.15.89:0/702750003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f662010c170 msgr2=0x7f662019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f661e59c640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f662010c170 0x7f662019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f661e59c640 1 -- 172.21.15.89:0/702750003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6620103660 msgr2=0x7f6620198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f661e59c640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6620103660 0x7f6620198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f661e59c640 1 -- 172.21.15.89:0/702750003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662019d3c0 con 0x7f6620102d40 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f661e59c640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6620102d40 0x7f6620198410 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f661000c920 tx=0x7f661000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f6617fff640 1 -- 172.21.15.89:0/702750003 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610007c70 con 0x7f6620102d40 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f6617fff640 1 -- 172.21.15.89:0/702750003 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6610007e10 con 0x7f6620102d40 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f6624e12640 1 -- 172.21.15.89:0/702750003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f662019d6b0 con 0x7f6620102d40 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f6617fff640 1 -- 172.21.15.89:0/702750003 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610018890 con 0x7f6620102d40 2024-09-15T23:37:19.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.084+0000 7f6624e12640 1 -- 172.21.15.89:0/702750003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66201a4f90 con 0x7f6620102d40 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.086+0000 7f6617fff640 1 -- 172.21.15.89:0/702750003 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6610018a70 con 0x7f6620102d40 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.086+0000 7f66271eb740 1 Processor -- start 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f66271eb740 1 -- start start 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f66271eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bce7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f66271eb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bce7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f66271eb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 0x564f5bcf4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f66271eb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564f5bc4d2f0 con 0x564f5bce75d0 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f66271eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564f5bbf4cf0 con 0x564f5bce8b40 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f66271eb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564f5bc492b0 con 0x564f5bcf2370 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bce7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bce7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50840/0 (socket says 172.21.15.89:50840) 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f6615d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 0x564f5bcf4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.105 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661e59c640 1 -- 172.21.15.89:0/3077700668 learned_addr learned my addr 172.21.15.89:0/3077700668 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f6617fff640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f660c0a3240 0x7f660c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661ed9d640 1 --2- 172.21.15.89:0/3077700668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bce7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661ed9d640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f660c0a3240 0x7f660c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661e59c640 1 -- 172.21.15.89:0/3077700668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 msgr2=0x564f5bcf4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661e59c640 1 --2- 172.21.15.89:0/3077700668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 0x564f5bcf4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661e59c640 1 -- 172.21.15.89:0/3077700668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 msgr2=0x564f5bce7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661e59c640 1 --2- 172.21.15.89:0/3077700668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bce7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f6617fff640 1 -- 172.21.15.89:0/702750003 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f66100cfe50 con 0x7f6620102d40 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661e59c640 1 -- 172.21.15.89:0/3077700668 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f5bbd93a0 con 0x564f5bce75d0 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f6615d9b640 1 --2- 172.21.15.89:0/3077700668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 0x564f5bcf4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.087+0000 7f661ed9d640 1 --2- 172.21.15.89:0/3077700668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bce7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.088+0000 7f661e59c640 1 --2- 172.21.15.89:0/3077700668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bce7a10 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f661009bd80 tx=0x7f661000ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4bec569de5f586 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:19.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.088+0000 7f6614d99640 1 -- 172.21.15.89:0/3077700668 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661009c780 con 0x564f5bce75d0 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.088+0000 7f661ed9d640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f660c0a3240 0x7f660c0a5700 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6618013540 tx=0x7f6618005c20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.088+0000 7f6614d99640 1 -- 172.21.15.89:0/3077700668 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f661009c080 con 0x564f5bce75d0 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.088+0000 7f6614d99640 1 -- 172.21.15.89:0/3077700668 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661000fb30 con 0x564f5bce75d0 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 -- 172.21.15.89:0/3077700668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 msgr2=0x564f5bce7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 --2- 172.21.15.89:0/3077700668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bce7a10 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f661009bd80 tx=0x7f661000ff30 comp rx=0 tx=0).stop 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 -- 172.21.15.89:0/3077700668 shutdown_connections 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 --2- 172.21.15.89:0/3077700668 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 0x564f5bcf4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 --2- 172.21.15.89:0/3077700668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bce7a10 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 --2- 172.21.15.89:0/3077700668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bce7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 -- 172.21.15.89:0/3077700668 >> 172.21.15.89:0/3077700668 conn(0x564f5bcb0310 msgr2=0x564f5bce3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 -- 172.21.15.89:0/3077700668 shutdown_connections 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 -- 172.21.15.89:0/3077700668 wait complete. 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 -- start start 2024-09-15T23:37:19.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.089+0000 7f66271eb740 1 Processor -- start 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f66271eb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bcf1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f66271eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bcef000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f66271eb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 0x564f5bcef5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f66271eb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564f5bcf5010 con 0x564f5bce75d0 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f66271eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564f5bc08060 con 0x564f5bce8b40 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f66271eb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564f5bcf5190 con 0x564f5bcf2370 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f66271eb740 10 client.0 _ll_register_callbacks cb 0x564f5b95b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bcf1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bcf1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50846/0 (socket says 172.21.15.89:50846) 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661e59c640 1 -- 172.21.15.89:0/661329976 learned_addr learned my addr 172.21.15.89:0/661329976 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f6615d9b640 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bcef000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661ed9d640 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 0x564f5bcef5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661e59c640 1 -- 172.21.15.89:0/661329976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 msgr2=0x564f5bcef5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:19.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661e59c640 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 0x564f5bcef5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661e59c640 1 -- 172.21.15.89:0/661329976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 msgr2=0x564f5bcef000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661e59c640 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bcef000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661e59c640 1 -- 172.21.15.89:0/661329976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f5b9be3c0 con 0x564f5bce75d0 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661ed9d640 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 0x564f5bcef5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f661e59c640 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bcf1d50 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f6610098a50 tx=0x7f66100d34a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58002c4386ec5ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f6605ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f6605ffb640 1 -- 172.21.15.89:0/661329976 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610061170 con 0x564f5bce75d0 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f6605ffb640 1 -- 172.21.15.89:0/661329976 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f66100a00d0 con 0x564f5bce75d0 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f66271eb740 1 -- 172.21.15.89:0/661329976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f661009c080 con 0x564f5bce75d0 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.090+0000 7f6605ffb640 1 -- 172.21.15.89:0/661329976 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f66100a4cf0 con 0x564f5bce75d0 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f66271eb740 10 client.19977 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:19.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f66271eb740 1 -- 172.21.15.89:0/661329976 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564f5bd00a90 con 0x564f5bce75d0 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f6605ffb640 1 -- 172.21.15.89:0/661329976 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f661009c080 con 0x564f5bce75d0 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f6605ffb640 1 -- 172.21.15.89:0/661329976 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f66100a4860 con 0x564f5bce75d0 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f66271eb740 10 client.19977 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f66271eb740 10 client.19977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f66271eb740 20 client.19977 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f66271eb740 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564f5bd01de0 0x564f5bd041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f66271eb740 4 client.19977 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f66271eb740 1 -- 172.21.15.89:0/661329976 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f5bc5a860 con 0x564f5bd01de0 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.091+0000 7f6615d9b640 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564f5bd01de0 0x564f5bd041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f6615d9b640 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564f5bd01de0 0x564f5bd041d0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9959e3c753c22aa server_cookie=ee4922654dc1a866 in_seq=0 out_seq=0 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f6605ffb640 10 client.19977 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:19.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f6605ffb640 1 -- 172.21.15.89:0/661329976 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564f5bc5a860 con 0x564f5bd01de0 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f6605ffb640 10 client.19977 handle_command_reply: tid=0 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f66271eb740 1 client.19977 shutdown 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f66271eb740 1 -- 172.21.15.89:0/661329976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 msgr2=0x564f5bcf1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f66271eb740 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bcf1d50 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f6610098a50 tx=0x7f66100d34a0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f66271eb740 1 -- 172.21.15.89:0/661329976 shutdown_connections 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f66271eb740 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564f5bd01de0 0x564f5bd041d0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f66271eb740 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bcf2370 0x564f5bcef5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f66271eb740 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bcef000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f6605ffb640 0 client.19977 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f66271eb740 1 --2- 172.21.15.89:0/661329976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bce75d0 0x564f5bcf1d50 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.092+0000 7f66271eb740 1 -- 172.21.15.89:0/661329976 >> 172.21.15.89:0/661329976 conn(0x564f5bcb0310 msgr2=0x564f5bce9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:19.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.093+0000 7f66271eb740 1 -- 172.21.15.89:0/661329976 shutdown_connections 2024-09-15T23:37:19.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.093+0000 7f66271eb740 1 -- 172.21.15.89:0/661329976 wait complete. 2024-09-15T23:37:19.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.093+0000 7f66271eb740 20 client.19977 trim_cache size 0 max 16384 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.105+0000 7f66271eb740 1 Processor -- start 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f66271eb740 1 -- start start 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f66271eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce75d0 0x564f5bce3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f66271eb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce8b40 0x564f5bcedfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f66271eb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd071f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f66271eb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f66201aee70 con 0x564f5bcf2370 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f66271eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f66201b09b0 con 0x564f5bce75d0 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f66271eb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f66201af540 con 0x564f5bce8b40 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f661ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd071f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f661ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd071f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50854/0 (socket says 172.21.15.89:50854) 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f661ed9d640 1 -- 172.21.15.89:0/1797676333 learned_addr learned my addr 172.21.15.89:0/1797676333 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:19.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f661e59c640 1 --2- 172.21.15.89:0/1797676333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce75d0 0x564f5bce3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f6615d9b640 1 --2- 172.21.15.89:0/1797676333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce8b40 0x564f5bcedfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f661ed9d640 1 -- 172.21.15.89:0/1797676333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce8b40 msgr2=0x564f5bcedfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f661ed9d640 1 --2- 172.21.15.89:0/1797676333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce8b40 0x564f5bcedfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f661ed9d640 1 -- 172.21.15.89:0/1797676333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce75d0 msgr2=0x564f5bce3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f661ed9d640 1 --2- 172.21.15.89:0/1797676333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce75d0 0x564f5bce3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f661ed9d640 1 -- 172.21.15.89:0/1797676333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f5bcf2080 con 0x564f5bcf2370 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f6615d9b640 1 --2- 172.21.15.89:0/1797676333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce8b40 0x564f5bcedfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.106+0000 7f661ed9d640 1 --2- 172.21.15.89:0/1797676333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd071f0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f6618005fd0 tx=0x7f661800b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d82a75c9b68fd89d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f6607fff640 1 -- 172.21.15.89:0/1797676333 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618017040 con 0x564f5bcf2370 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f6607fff640 1 -- 172.21.15.89:0/1797676333 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6618004d20 con 0x564f5bcf2370 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f6607fff640 1 -- 172.21.15.89:0/1797676333 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618005420 con 0x564f5bcf2370 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f66271eb740 1 -- 172.21.15.89:0/1797676333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 msgr2=0x564f5bd071f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:19.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f66271eb740 1 --2- 172.21.15.89:0/1797676333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd071f0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f6618005fd0 tx=0x7f661800b540 comp rx=0 tx=0).stop 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f66271eb740 1 -- 172.21.15.89:0/1797676333 shutdown_connections 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f66271eb740 1 --2- 172.21.15.89:0/1797676333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd071f0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f66271eb740 1 --2- 172.21.15.89:0/1797676333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce8b40 0x564f5bcedfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f66271eb740 1 --2- 172.21.15.89:0/1797676333 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce75d0 0x564f5bce3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f66271eb740 1 -- 172.21.15.89:0/1797676333 >> 172.21.15.89:0/1797676333 conn(0x564f5bcb0310 msgr2=0x564f5bced8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f66271eb740 1 -- 172.21.15.89:0/1797676333 shutdown_connections 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.107+0000 7f66271eb740 1 -- 172.21.15.89:0/1797676333 wait complete. 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f66271eb740 1 -- start start 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f66271eb740 1 Processor -- start 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f66271eb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce75d0 0x564f5bd07640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f66271eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bd049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f66271eb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd04f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f66271eb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f66201b02e0 con 0x564f5bcf2370 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f66271eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f66201afc10 con 0x564f5bce8b40 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f66271eb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f66201ace50 con 0x564f5bce75d0 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f66271eb740 10 client.0 _ll_register_callbacks cb 0x7f6608009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f661ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd04f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f661ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd04f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50858/0 (socket says 172.21.15.89:50858) 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f661ed9d640 1 -- 172.21.15.89:0/504459909 learned_addr learned my addr 172.21.15.89:0/504459909 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f6615d9b640 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bd049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.108+0000 7f661e59c640 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce75d0 0x564f5bd07640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f661ed9d640 1 -- 172.21.15.89:0/504459909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce75d0 msgr2=0x564f5bd07640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f661ed9d640 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce75d0 0x564f5bd07640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f661ed9d640 1 -- 172.21.15.89:0/504459909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 msgr2=0x564f5bd049e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f661ed9d640 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bd049e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f661ed9d640 1 -- 172.21.15.89:0/504459909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f5bd05460 con 0x564f5bcf2370 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f661ed9d640 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd04f20 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f6618005100 tx=0x7f6618005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fad2cf046107e5a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f6605ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f6605ffb640 1 -- 172.21.15.89:0/504459909 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618017050 con 0x564f5bcf2370 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f6605ffb640 1 -- 172.21.15.89:0/504459909 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6618003b70 con 0x564f5bcf2370 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f66271eb740 1 -- 172.21.15.89:0/504459909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6618004d20 con 0x564f5bcf2370 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f6605ffb640 1 -- 172.21.15.89:0/504459909 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f661800e500 con 0x564f5bcf2370 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f66271eb740 10 client.19983 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f66271eb740 1 -- 172.21.15.89:0/504459909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564f5bd6c410 con 0x564f5bcf2370 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f6605ffb640 1 -- 172.21.15.89:0/504459909 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6618004d20 con 0x564f5bcf2370 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f6605ffb640 1 -- 172.21.15.89:0/504459909 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6618030070 con 0x564f5bcf2370 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f66271eb740 10 client.19983 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.109+0000 7f66271eb740 10 client.19983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f66271eb740 20 client.19983 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:19.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f66271eb740 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564f5bd6cad0 0x564f5bd6eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f66271eb740 4 client.19983 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f66271eb740 1 -- 172.21.15.89:0/504459909 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564f5bc54d70 con 0x564f5bd6cad0 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f661e59c640 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564f5bd6cad0 0x564f5bd6eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f661e59c640 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564f5bd6cad0 0x564f5bd6eef0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=484fdaf4c421b8f2 server_cookie=a2c5f3d25900ab8 in_seq=0 out_seq=0 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f6605ffb640 10 client.19983 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f6605ffb640 1 -- 172.21.15.89:0/504459909 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564f5bc54d70 con 0x564f5bd6cad0 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f6605ffb640 10 client.19983 handle_command_reply: tid=0 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f66271eb740 1 client.19983 shutdown 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f66271eb740 1 -- 172.21.15.89:0/504459909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 msgr2=0x564f5bd04f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f66271eb740 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd04f20 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f6618005100 tx=0x7f6618005880 comp rx=0 tx=0).stop 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f66271eb740 1 -- 172.21.15.89:0/504459909 shutdown_connections 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.110+0000 7f66271eb740 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x564f5bd6cad0 0x564f5bd6eef0 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f66271eb740 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564f5bcf2370 0x564f5bd04f20 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f66271eb740 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564f5bce8b40 0x564f5bd049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6605ffb640 0 client.19983 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:19.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f66271eb740 1 --2- 172.21.15.89:0/504459909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564f5bce75d0 0x564f5bd07640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f66271eb740 1 -- 172.21.15.89:0/504459909 >> 172.21.15.89:0/504459909 conn(0x564f5bcb0310 msgr2=0x564f5bce9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:19.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f66271eb740 1 -- 172.21.15.89:0/504459909 shutdown_connections 2024-09-15T23:37:19.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f66271eb740 1 -- 172.21.15.89:0/504459909 wait complete. 2024-09-15T23:37:19.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f66271eb740 20 client.19983 trim_cache size 0 max 16384 2024-09-15T23:37:19.122 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 -- 172.21.15.89:0/702750003 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f660c0a3240 msgr2=0x7f660c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f660c0a3240 0x7f660c0a5700 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6618013540 tx=0x7f6618005c20 comp rx=0 tx=0).stop 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 -- 172.21.15.89:0/702750003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6620102d40 msgr2=0x7f6620198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6620102d40 0x7f6620198410 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f661000c920 tx=0x7f661000cde0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 -- 172.21.15.89:0/702750003 shutdown_connections 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f660c0a3240 0x7f660c0a5700 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f662010c170 0x7f662019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6620103660 0x7f6620198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 --2- 172.21.15.89:0/702750003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6620102d40 0x7f6620198410 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 -- 172.21.15.89:0/702750003 >> 172.21.15.89:0/702750003 conn(0x7f66200fc7a0 msgr2=0x7f66200fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 -- 172.21.15.89:0/702750003 shutdown_connections 2024-09-15T23:37:19.123 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:19.111+0000 7f6624e12640 1 -- 172.21.15.89:0/702750003 wait complete. 2024-09-15T23:37:19.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:18 smithi120 ceph-mon[35250]: pgmap v748: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:37:19.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:18 smithi174 ceph-mon[35586]: pgmap v748: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:37:20.902 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:37:20.902 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:37:21.159 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:21.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:20 smithi120 ceph-mon[35250]: pgmap v749: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:37:21.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:20 smithi174 ceph-mon[35586]: pgmap v749: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:37:21.280 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:20 smithi089 ceph-mon[30933]: pgmap v749: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:37:21.662 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.660+0000 7fc458099640 1 -- 172.21.15.89:0/1859605012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc450106a80 msgr2=0x7fc450106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.660+0000 7fc458099640 1 --2- 172.21.15.89:0/1859605012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc450106a80 0x7fc450106e60 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009930 tx=0x7fc44c02a420 comp rx=0 tx=0).stop 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.660+0000 7fc458099640 1 -- 172.21.15.89:0/1859605012 shutdown_connections 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.660+0000 7fc458099640 1 --2- 172.21.15.89:0/1859605012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc450101420 0x7fc45010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.660+0000 7fc458099640 1 --2- 172.21.15.89:0/1859605012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450100a80 0x7fc450100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.660+0000 7fc458099640 1 --2- 172.21.15.89:0/1859605012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc450106a80 0x7fc450106e60 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.660+0000 7fc458099640 1 -- 172.21.15.89:0/1859605012 >> 172.21.15.89:0/1859605012 conn(0x7fc4500fc7a0 msgr2=0x7fc4500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.660+0000 7fc458099640 1 -- 172.21.15.89:0/1859605012 shutdown_connections 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.660+0000 7fc458099640 1 -- 172.21.15.89:0/1859605012 wait complete. 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc458099640 1 Processor -- start 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc458099640 1 -- start start 2024-09-15T23:37:21.663 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc458099640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc450100a80 0x7fc4501972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc458099640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450101420 0x7fc450197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc458099640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc450106a80 0x7fc45019bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc458099640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc450110e80 con 0x7fc450106a80 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc458099640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc450110d00 con 0x7fc450101420 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc458099640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc450111000 con 0x7fc450100a80 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc456636640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc450106a80 0x7fc45019bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc456636640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc450106a80 0x7fc45019bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50880/0 (socket says 172.21.15.89:50880) 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc456636640 1 -- 172.21.15.89:0/130189438 learned_addr learned my addr 172.21.15.89:0/130189438 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc455634640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450101420 0x7fc450197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc455e35640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc450100a80 0x7fc4501972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc456636640 1 -- 172.21.15.89:0/130189438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc450100a80 msgr2=0x7fc4501972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc456636640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc450100a80 0x7fc4501972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc456636640 1 -- 172.21.15.89:0/130189438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450101420 msgr2=0x7fc450197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:21.664 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc456636640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450101420 0x7fc450197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.661+0000 7fc456636640 1 -- 172.21.15.89:0/130189438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc45019c290 con 0x7fc450106a80 2024-09-15T23:37:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.662+0000 7fc456636640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc450106a80 0x7fc45019bbb0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fc44800eaa0 tx=0x7fc44800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.662+0000 7fc446ffd640 1 -- 172.21.15.89:0/130189438 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc448004d20 con 0x7fc450106a80 2024-09-15T23:37:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.662+0000 7fc458099640 1 -- 172.21.15.89:0/130189438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc45019c580 con 0x7fc450106a80 2024-09-15T23:37:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.662+0000 7fc446ffd640 1 -- 172.21.15.89:0/130189438 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc448005020 con 0x7fc450106a80 2024-09-15T23:37:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.662+0000 7fc446ffd640 1 -- 172.21.15.89:0/130189438 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4480116d0 con 0x7fc450106a80 2024-09-15T23:37:21.665 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.662+0000 7fc458099640 1 -- 172.21.15.89:0/130189438 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4501a3e60 con 0x7fc450106a80 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.664+0000 7fc446ffd640 1 -- 172.21.15.89:0/130189438 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc4480051d0 con 0x7fc450106a80 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.664+0000 7fc45a472740 1 Processor -- start 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc45a472740 1 -- start start 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc45a472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af276b60 0x5650af276f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc45a472740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af281800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc45a472740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af281d40 0x5650af284130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc45a472740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5650af1dc280 con 0x5650af277480 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc446ffd640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc43c0a3240 0x7fc43c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc45a472740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5650af183b10 con 0x5650af276b60 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc45a472740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5650af1d8240 con 0x5650af281d40 2024-09-15T23:37:21.667 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc455e35640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af281800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc455e35640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af281800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50888/0 (socket says 172.21.15.89:50888) 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc455e35640 1 -- 172.21.15.89:0/1449361477 learned_addr learned my addr 172.21.15.89:0/1449361477 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc456636640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc43c0a3240 0x7fc43c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc455634640 1 --2- 172.21.15.89:0/1449361477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af281d40 0x5650af284130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc446ffd640 1 -- 172.21.15.89:0/130189438 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc4480cfe20 con 0x7fc450106a80 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc455634640 1 --2- 172.21.15.89:0/1449361477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af276b60 0x5650af276f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc455e35640 1 -- 172.21.15.89:0/1449361477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af281d40 msgr2=0x5650af284130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc455e35640 1 --2- 172.21.15.89:0/1449361477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af281d40 0x5650af284130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc455e35640 1 -- 172.21.15.89:0/1449361477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af276b60 msgr2=0x5650af276f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc455e35640 1 --2- 172.21.15.89:0/1449361477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af276b60 0x5650af276f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.665+0000 7fc455e35640 1 -- 172.21.15.89:0/1449361477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650af1901f0 con 0x5650af277480 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.666+0000 7fc455e35640 1 --2- 172.21.15.89:0/1449361477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af281800 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009930 tx=0x7fc44c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e52ff1edc8c555f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:21.668 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.666+0000 7fc456636640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc43c0a3240 0x7fc43c0a5700 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc4480092f0 tx=0x7fc448066a30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.666+0000 7fc43bfff640 1 -- 172.21.15.89:0/1449361477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c040070 con 0x5650af277480 2024-09-15T23:37:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.666+0000 7fc43bfff640 1 -- 172.21.15.89:0/1449361477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44c005020 con 0x5650af277480 2024-09-15T23:37:21.669 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.666+0000 7fc43bfff640 1 -- 172.21.15.89:0/1449361477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c035040 con 0x5650af277480 2024-09-15T23:37:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 -- 172.21.15.89:0/1449361477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 msgr2=0x5650af281800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 --2- 172.21.15.89:0/1449361477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af281800 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009930 tx=0x7fc44c031630 comp rx=0 tx=0).stop 2024-09-15T23:37:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 -- 172.21.15.89:0/1449361477 shutdown_connections 2024-09-15T23:37:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 --2- 172.21.15.89:0/1449361477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af281d40 0x5650af284130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 --2- 172.21.15.89:0/1449361477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af281800 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 --2- 172.21.15.89:0/1449361477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af276b60 0x5650af276f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 -- 172.21.15.89:0/1449361477 >> 172.21.15.89:0/1449361477 conn(0x5650af2730f0 msgr2=0x5650af2734d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 -- 172.21.15.89:0/1449361477 shutdown_connections 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 -- 172.21.15.89:0/1449361477 wait complete. 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 -- start start 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 Processor -- start 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af276b60 0x5650af2817c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.667+0000 7fc45a472740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af27eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc45a472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af281d40 0x5650af27f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc45a472740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5650af286ac0 con 0x5650af277480 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc45a472740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5650af196f10 con 0x5650af281d40 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc45a472740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5650af286c40 con 0x5650af276b60 2024-09-15T23:37:21.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc45a472740 10 client.0 _ll_register_callbacks cb 0x5650af15fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455e35640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af27eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455e35640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af27eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50900/0 (socket says 172.21.15.89:50900) 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455e35640 1 -- 172.21.15.89:0/964381346 learned_addr learned my addr 172.21.15.89:0/964381346 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455634640 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af281d40 0x5650af27f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455634640 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af276b60 0x5650af2817c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455e35640 1 -- 172.21.15.89:0/964381346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af276b60 msgr2=0x5650af2817c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455e35640 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af276b60 0x5650af2817c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455e35640 1 -- 172.21.15.89:0/964381346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af281d40 msgr2=0x5650af27f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455e35640 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af281d40 0x5650af27f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455e35640 1 -- 172.21.15.89:0/964381346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650aef4d6a0 con 0x5650af277480 2024-09-15T23:37:21.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455634640 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af276b60 0x5650af2817c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc455e35640 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af27eb10 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fc44c005940 tx=0x7fc44c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0ae96e669c696d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc4397fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc4397fa640 1 -- 172.21.15.89:0/964381346 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c040070 con 0x5650af277480 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc45a472740 1 -- 172.21.15.89:0/964381346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc44c005020 con 0x5650af277480 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.668+0000 7fc4397fa640 1 -- 172.21.15.89:0/964381346 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc44c0053e0 con 0x5650af277480 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc4397fa640 1 -- 172.21.15.89:0/964381346 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc44c035050 con 0x5650af277480 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc45a472740 10 client.19995 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc45a472740 1 -- 172.21.15.89:0/964381346 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5650af2925b0 con 0x5650af277480 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc4397fa640 1 -- 172.21.15.89:0/964381346 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44c005020 con 0x5650af277480 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc4397fa640 1 -- 172.21.15.89:0/964381346 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc44c03c880 con 0x5650af277480 2024-09-15T23:37:21.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc45a472740 10 client.19995 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc45a472740 10 client.19995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc45a472740 20 client.19995 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc45a472740 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650af293900 0x5650af295cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc45a472740 4 client.19995 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc45a472740 1 -- 172.21.15.89:0/964381346 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650af1e97f0 con 0x5650af293900 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.669+0000 7fc455634640 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650af293900 0x5650af295cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.670+0000 7fc4397fa640 10 client.19995 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.670+0000 7fc455634640 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650af293900 0x5650af295cf0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11467fe891d54614 server_cookie=5c406db9a2c74afe in_seq=0 out_seq=0 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.670+0000 7fc4397fa640 1 -- 172.21.15.89:0/964381346 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5650af1e97f0 con 0x5650af293900 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.670+0000 7fc4397fa640 10 client.19995 handle_command_reply: tid=0 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.670+0000 7fc45a472740 1 client.19995 shutdown 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 1 -- 172.21.15.89:0/964381346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 msgr2=0x5650af27eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:21.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af27eb10 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fc44c005940 tx=0x7fc44c009250 comp rx=0 tx=0).stop 2024-09-15T23:37:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 1 -- 172.21.15.89:0/964381346 shutdown_connections 2024-09-15T23:37:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650af293900 0x5650af295cf0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af281d40 0x5650af27f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af277480 0x5650af27eb10 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 1 --2- 172.21.15.89:0/964381346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af276b60 0x5650af2817c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc4397fa640 0 client.19995 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 1 -- 172.21.15.89:0/964381346 >> 172.21.15.89:0/964381346 conn(0x5650af2730f0 msgr2=0x5650af272270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 1 -- 172.21.15.89:0/964381346 shutdown_connections 2024-09-15T23:37:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 1 -- 172.21.15.89:0/964381346 wait complete. 2024-09-15T23:37:21.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.671+0000 7fc45a472740 20 client.19995 trim_cache size 0 max 16384 2024-09-15T23:37:21.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc45a472740 1 Processor -- start 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc45a472740 1 -- start start 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc45a472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af276b60 0x5650af28fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc45a472740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af277480 0x5650af27bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc45a472740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af281d40 0x5650af27c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc45a472740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501acc90 con 0x5650af281d40 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc45a472740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501ae7d0 con 0x5650af276b60 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc45a472740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501ad360 con 0x5650af277480 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc455634640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af281d40 0x5650af27c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc455634640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af281d40 0x5650af27c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50912/0 (socket says 172.21.15.89:50912) 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc455634640 1 -- 172.21.15.89:0/2854284923 learned_addr learned my addr 172.21.15.89:0/2854284923 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc455634640 1 --2- 172.21.15.89:0/2854284923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af276b60 0x5650af28fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc455e35640 1 --2- 172.21.15.89:0/2854284923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af277480 0x5650af27bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc455634640 1 -- 172.21.15.89:0/2854284923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af277480 msgr2=0x5650af27bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc455634640 1 --2- 172.21.15.89:0/2854284923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af277480 0x5650af27bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc455634640 1 -- 172.21.15.89:0/2854284923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af276b60 msgr2=0x5650af28fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc455634640 1 --2- 172.21.15.89:0/2854284923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af276b60 0x5650af28fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.688+0000 7fc455634640 1 -- 172.21.15.89:0/2854284923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650af26e830 con 0x5650af281d40 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc455634640 1 --2- 172.21.15.89:0/2854284923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af281d40 0x5650af27c2b0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x5650af286d30 tx=0x7fc4400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b996193f7475bee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc43b7fe640 1 -- 172.21.15.89:0/2854284923 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc440024070 con 0x5650af281d40 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc43b7fe640 1 -- 172.21.15.89:0/2854284923 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44000fce0 con 0x5650af281d40 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc43b7fe640 1 -- 172.21.15.89:0/2854284923 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4400227a0 con 0x5650af281d40 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc45a472740 1 -- 172.21.15.89:0/2854284923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af281d40 msgr2=0x5650af27c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc45a472740 1 --2- 172.21.15.89:0/2854284923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af281d40 0x5650af27c2b0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x5650af286d30 tx=0x7fc4400046c0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc45a472740 1 -- 172.21.15.89:0/2854284923 shutdown_connections 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc45a472740 1 --2- 172.21.15.89:0/2854284923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af281d40 0x5650af27c2b0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc45a472740 1 --2- 172.21.15.89:0/2854284923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af277480 0x5650af27bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc45a472740 1 --2- 172.21.15.89:0/2854284923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af276b60 0x5650af28fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.689+0000 7fc45a472740 1 -- 172.21.15.89:0/2854284923 >> 172.21.15.89:0/2854284923 conn(0x5650af2730f0 msgr2=0x5650af280a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:21.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 1 -- 172.21.15.89:0/2854284923 shutdown_connections 2024-09-15T23:37:21.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 1 -- 172.21.15.89:0/2854284923 wait complete. 2024-09-15T23:37:21.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 1 -- start start 2024-09-15T23:37:21.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 1 Processor -- start 2024-09-15T23:37:21.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af276b60 0x5650af298de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af277480 0x5650af299320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af281d40 0x5650af295220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501ae100 con 0x5650af276b60 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501ada30 con 0x5650af281d40 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4501aac70 con 0x5650af277480 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc45a472740 10 client.0 _ll_register_callbacks cb 0x7fc44c005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc455634640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af276b60 0x5650af298de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc455634640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af276b60 0x5650af298de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50926/0 (socket says 172.21.15.89:50926) 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.690+0000 7fc455634640 1 -- 172.21.15.89:0/3319495988 learned_addr learned my addr 172.21.15.89:0/3319495988 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc455e35640 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af277480 0x5650af299320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc455634640 1 -- 172.21.15.89:0/3319495988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af277480 msgr2=0x5650af299320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:21.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc455634640 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af277480 0x5650af299320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc455634640 1 -- 172.21.15.89:0/3319495988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af281d40 msgr2=0x5650af295220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc455634640 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af281d40 0x5650af295220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc455634640 1 -- 172.21.15.89:0/3319495988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650af295a50 con 0x5650af276b60 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc455634640 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af276b60 0x5650af298de0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fc44000fe90 tx=0x7fc44000fec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8e23d3096f16b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc4397fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc4397fa640 1 -- 172.21.15.89:0/3319495988 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc440035060 con 0x5650af276b60 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc4397fa640 1 -- 172.21.15.89:0/3319495988 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc440022440 con 0x5650af276b60 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc45a472740 1 -- 172.21.15.89:0/3319495988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc44000fce0 con 0x5650af276b60 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc4397fa640 1 -- 172.21.15.89:0/3319495988 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc44003c050 con 0x5650af276b60 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc45a472740 10 client.20001 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc45a472740 1 -- 172.21.15.89:0/3319495988 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5650af2fed60 con 0x5650af276b60 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc4397fa640 1 -- 172.21.15.89:0/3319495988 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44000fce0 con 0x5650af276b60 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.691+0000 7fc4397fa640 1 -- 172.21.15.89:0/3319495988 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc44002bab0 con 0x5650af276b60 2024-09-15T23:37:21.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.692+0000 7fc45a472740 10 client.20001 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.692+0000 7fc45a472740 10 client.20001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.692+0000 7fc45a472740 20 client.20001 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.692+0000 7fc45a472740 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650af2ff2b0 0x5650af3016d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.692+0000 7fc45a472740 4 client.20001 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.692+0000 7fc45a472740 1 -- 172.21.15.89:0/3319495988 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5650af1e3d00 con 0x5650af2ff2b0 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.692+0000 7fc455e35640 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650af2ff2b0 0x5650af3016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.692+0000 7fc4397fa640 10 client.20001 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.692+0000 7fc455e35640 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650af2ff2b0 0x5650af3016d0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=954dc9f776d86cbc server_cookie=a43de53726321c8f in_seq=0 out_seq=0 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.693+0000 7fc4397fa640 1 -- 172.21.15.89:0/3319495988 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5650af1e3d00 con 0x5650af2ff2b0 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.693+0000 7fc4397fa640 10 client.20001 handle_command_reply: tid=0 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.693+0000 7fc45a472740 1 client.20001 shutdown 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.693+0000 7fc45a472740 1 -- 172.21.15.89:0/3319495988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af276b60 msgr2=0x5650af298de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.693+0000 7fc45a472740 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af276b60 0x5650af298de0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fc44000fe90 tx=0x7fc44000fec0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.693+0000 7fc45a472740 1 -- 172.21.15.89:0/3319495988 shutdown_connections 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.693+0000 7fc45a472740 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650af2ff2b0 0x5650af3016d0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.693+0000 7fc45a472740 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650af281d40 0x5650af295220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc4397fa640 0 client.20001 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.693+0000 7fc45a472740 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650af277480 0x5650af299320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc45a472740 1 --2- 172.21.15.89:0/3319495988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650af276b60 0x5650af298de0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc45a472740 1 -- 172.21.15.89:0/3319495988 >> 172.21.15.89:0/3319495988 conn(0x5650af2730f0 msgr2=0x5650af271a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc45a472740 1 -- 172.21.15.89:0/3319495988 shutdown_connections 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc45a472740 1 -- 172.21.15.89:0/3319495988 wait complete. 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc45a472740 20 client.20001 trim_cache size 0 max 16384 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:37:21.699 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:37:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "9c6b1c99-21b0-43ee-a15c-816cbf373dad", 2024-09-15T23:37:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:37:21.700 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:37:21.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 -- 172.21.15.89:0/130189438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc43c0a3240 msgr2=0x7fc43c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc43c0a3240 0x7fc43c0a5700 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc4480092f0 tx=0x7fc448066a30 comp rx=0 tx=0).stop 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 -- 172.21.15.89:0/130189438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc450106a80 msgr2=0x7fc45019bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc450106a80 0x7fc45019bbb0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fc44800eaa0 tx=0x7fc44800eea0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 -- 172.21.15.89:0/130189438 shutdown_connections 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc43c0a3240 0x7fc43c0a5700 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc450106a80 0x7fc45019bbb0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc450101420 0x7fc450197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 --2- 172.21.15.89:0/130189438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc450100a80 0x7fc4501972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 -- 172.21.15.89:0/130189438 >> 172.21.15.89:0/130189438 conn(0x7fc4500fc7a0 msgr2=0x7fc4500fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 -- 172.21.15.89:0/130189438 shutdown_connections 2024-09-15T23:37:21.701 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:21.694+0000 7fc458099640 1 -- 172.21.15.89:0/130189438 wait complete. 2024-09-15T23:37:22.590 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:37:22.834 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 -- 172.21.15.89:0/1585867278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a98102d50 msgr2=0x7f4a981031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 --2- 172.21.15.89:0/1585867278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a98102d50 0x7f4a981031d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c009ca0 tx=0x7f4a8c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 -- 172.21.15.89:0/1585867278 shutdown_connections 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 --2- 172.21.15.89:0/1585867278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a98103710 0x7f4a98109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 --2- 172.21.15.89:0/1585867278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a98102d50 0x7f4a981031d0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 --2- 172.21.15.89:0/1585867278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a98101b60 0x7f4a98101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 -- 172.21.15.89:0/1585867278 >> 172.21.15.89:0/1585867278 conn(0x7f4a980fd2e0 msgr2=0x7f4a980ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 -- 172.21.15.89:0/1585867278 shutdown_connections 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 -- 172.21.15.89:0/1585867278 wait complete. 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 Processor -- start 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.142+0000 7f4a9da5e640 1 -- start start 2024-09-15T23:37:23.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9da5e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a98101b60 0x7f4a98075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9da5e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a98102d50 0x7f4a980745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9da5e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a98103710 0x7f4a98074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9da5e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9810d7c0 con 0x7f4a98101b60 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9da5e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9810d640 con 0x7f4a98102d50 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9da5e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9810d940 con 0x7f4a98103710 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a98101b60 0x7f4a98075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a98101b60 0x7f4a98075fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50934/0 (socket says 172.21.15.89:50934) 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a96ffd640 1 -- 172.21.15.89:0/515593195 learned_addr learned my addr 172.21.15.89:0/515593195 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a967fc640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a98102d50 0x7f4a980745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a977fe640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a98103710 0x7f4a98074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a96ffd640 1 -- 172.21.15.89:0/515593195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a98103710 msgr2=0x7f4a98074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a98103710 0x7f4a98074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a96ffd640 1 -- 172.21.15.89:0/515593195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a98102d50 msgr2=0x7f4a980745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a98102d50 0x7f4a980745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a96ffd640 1 -- 172.21.15.89:0/515593195 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a980753b0 con 0x7f4a98101b60 2024-09-15T23:37:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a977fe640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a98103710 0x7f4a98074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a98101b60 0x7f4a98075fa0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000d770 tx=0x7f4a9000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9ca5c640 1 -- 172.21.15.89:0/515593195 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90017070 con 0x7f4a98101b60 2024-09-15T23:37:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9ca5c640 1 -- 172.21.15.89:0/515593195 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a90009c30 con 0x7f4a98101b60 2024-09-15T23:37:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9da5e640 1 -- 172.21.15.89:0/515593195 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a981a1f00 con 0x7f4a98101b60 2024-09-15T23:37:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9ca5c640 1 -- 172.21.15.89:0/515593195 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90003dd0 con 0x7f4a98101b60 2024-09-15T23:37:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.143+0000 7f4a9da5e640 1 -- 172.21.15.89:0/515593195 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a981a2300 con 0x7f4a98101b60 2024-09-15T23:37:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4a9ca5c640 1 -- 172.21.15.89:0/515593195 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4a90003fb0 con 0x7f4a98101b60 2024-09-15T23:37:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4aa07f6740 1 Processor -- start 2024-09-15T23:37:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4aa07f6740 1 -- start start 2024-09-15T23:37:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494639a0 0x55c249463da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494642e0 0x55c249468da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c249469370 0x55c24946b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c2493c72f0 con 0x55c2494642e0 2024-09-15T23:37:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c24936f270 con 0x55c2494639a0 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c2493c32b0 con 0x55c249469370 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4a96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494642e0 0x55c249468da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4a9ca5c640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a800a32f0 0x7f4a800a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4a96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494642e0 0x55c249468da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50942/0 (socket says 172.21.15.89:50942) 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4a96ffd640 1 -- 172.21.15.89:0/1192237316 learned_addr learned my addr 172.21.15.89:0/1192237316 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4a977fe640 1 --2- 172.21.15.89:0/1192237316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494639a0 0x55c249463da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.146+0000 7f4a977fe640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a800a32f0 0x7f4a800a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a967fc640 1 --2- 172.21.15.89:0/1192237316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c249469370 0x55c24946b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a9ca5c640 1 -- 172.21.15.89:0/515593195 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4a900cf5c0 con 0x7f4a98101b60 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a96ffd640 1 -- 172.21.15.89:0/1192237316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c249469370 msgr2=0x55c24946b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/1192237316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c249469370 0x55c24946b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a96ffd640 1 -- 172.21.15.89:0/1192237316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494639a0 msgr2=0x55c249463da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/1192237316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494639a0 0x55c249463da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a96ffd640 1 -- 172.21.15.89:0/1192237316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c24936e340 con 0x55c2494642e0 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/1192237316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494642e0 0x55c249468da0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f4a90097c50 tx=0x7f4a90000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5651cd0ccf944be6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a977fe640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a800a32f0 0x7f4a800a57b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4a88005d50 tx=0x7f4a88005ce0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a7dffb640 1 -- 172.21.15.89:0/1192237316 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90020070 con 0x55c2494642e0 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a7dffb640 1 -- 172.21.15.89:0/1192237316 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9009fba0 con 0x55c2494642e0 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.147+0000 7f4a7dffb640 1 -- 172.21.15.89:0/1192237316 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a900042a0 con 0x55c2494642e0 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 -- 172.21.15.89:0/1192237316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494642e0 msgr2=0x55c249468da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/1192237316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494642e0 0x55c249468da0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f4a90097c50 tx=0x7f4a90000bc0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 -- 172.21.15.89:0/1192237316 shutdown_connections 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/1192237316 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c249469370 0x55c24946b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/1192237316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494642e0 0x55c249468da0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/1192237316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494639a0 0x55c249463da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 -- 172.21.15.89:0/1192237316 >> 172.21.15.89:0/1192237316 conn(0x55c24945f010 msgr2=0x55c24945f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 -- 172.21.15.89:0/1192237316 shutdown_connections 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 -- 172.21.15.89:0/1192237316 wait complete. 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 -- start start 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.148+0000 7f4aa07f6740 1 Processor -- start 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 0x55c24946dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494642e0 0x55c24946e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c249475050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c24946c2e0 con 0x55c249469370 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c249350290 con 0x55c2494642e0 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c24946c460 con 0x55c2494639a0 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4aa07f6740 10 client.0 _ll_register_callbacks cb 0x55c249381140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a977fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c249475050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a96ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 0x55c24946dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a977fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c249475050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50958/0 (socket says 172.21.15.89:50958) 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a977fe640 1 -- 172.21.15.89:0/2050936681 learned_addr learned my addr 172.21.15.89:0/2050936681 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a96ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 0x55c24946dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:35564/0 (socket says 172.21.15.89:35564) 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a967fc640 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494642e0 0x55c24946e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a977fe640 1 -- 172.21.15.89:0/2050936681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 msgr2=0x55c24946dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a977fe640 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 0x55c24946dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a977fe640 1 -- 172.21.15.89:0/2050936681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494642e0 msgr2=0x55c24946e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a977fe640 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494642e0 0x55c24946e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a977fe640 1 -- 172.21.15.89:0/2050936681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2490da210 con 0x55c249469370 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a977fe640 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c249475050 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f4a88010fa0 tx=0x7f4a880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742bd272815d2c01 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a76ffd640 1 -- 172.21.15.89:0/2050936681 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a88004220 con 0x55c249469370 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4a76ffd640 1 -- 172.21.15.89:0/2050936681 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4a880043c0 con 0x55c249469370 2024-09-15T23:37:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.149+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2050936681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c2493d4a20 con 0x55c249469370 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4a76ffd640 1 -- 172.21.15.89:0/2050936681 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4a88032050 con 0x55c249469370 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4aa07f6740 10 client.20013 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2050936681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c2491a6620 con 0x55c249469370 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4a76ffd640 1 -- 172.21.15.89:0/2050936681 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c2493d4a20 con 0x55c249469370 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4a76ffd640 1 -- 172.21.15.89:0/2050936681 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4a8801c040 con 0x55c249469370 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4aa07f6740 10 client.20013 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4aa07f6740 10 client.20013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4aa07f6740 20 client.20013 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2494797a0 0x55c24947bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4aa07f6740 4 client.20013 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2050936681 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2493ced70 con 0x55c2494797a0 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.150+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2494797a0 0x55c24947bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.151+0000 7f4a76ffd640 10 client.20013 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.151+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2494797a0 0x55c24947bbb0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=786366baaa4f26ca server_cookie=7c20d370adb4082 in_seq=0 out_seq=0 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.151+0000 7f4a76ffd640 1 -- 172.21.15.89:0/2050936681 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c2493ced70 con 0x55c2494797a0 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.151+0000 7f4a76ffd640 10 client.20013 handle_command_reply: tid=0 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.151+0000 7f4aa07f6740 1 client.20013 shutdown 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2050936681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 msgr2=0x55c249475050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c249475050 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f4a88010fa0 tx=0x7f4a880046c0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2050936681 shutdown_connections 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2494797a0 0x55c24947bbb0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c249475050 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4a76ffd640 0 client.20013 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494642e0 0x55c24946e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/2050936681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 0x55c24946dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2050936681 >> 172.21.15.89:0/2050936681 conn(0x55c24945f010 msgr2=0x55c249460820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2050936681 shutdown_connections 2024-09-15T23:37:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2050936681 wait complete. 2024-09-15T23:37:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.152+0000 7f4aa07f6740 20 client.20013 trim_cache size 0 max 16384 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4aa07f6740 1 Processor -- start 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4aa07f6740 1 -- start start 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 0x55c24945f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494642e0 0x55c249475010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c24945f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a981acaf0 con 0x55c249469370 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4a981ae630 con 0x55c2494642e0 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a981ad1c0 con 0x55c2494639a0 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4a977fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c24945f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4a977fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c24945f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50960/0 (socket says 172.21.15.89:50960) 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.166+0000 7f4a977fe640 1 -- 172.21.15.89:0/2871096599 learned_addr learned my addr 172.21.15.89:0/2871096599 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/2871096599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 0x55c24945f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a977fe640 1 -- 172.21.15.89:0/2871096599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 msgr2=0x55c24945f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a977fe640 1 --2- 172.21.15.89:0/2871096599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 0x55c24945f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a977fe640 1 -- 172.21.15.89:0/2871096599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494642e0 msgr2=0x55c249475010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a967fc640 1 --2- 172.21.15.89:0/2871096599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494642e0 0x55c249475010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a977fe640 1 --2- 172.21.15.89:0/2871096599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494642e0 0x55c249475010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a977fe640 1 -- 172.21.15.89:0/2871096599 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c24945ee30 con 0x55c249469370 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a977fe640 1 --2- 172.21.15.89:0/2871096599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c24945f930 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f4a88006010 tx=0x7f4a88005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a1dc08c865b3b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a7d7fa640 1 -- 172.21.15.89:0/2871096599 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8803a070 con 0x55c249469370 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a7d7fa640 1 -- 172.21.15.89:0/2871096599 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a88013760 con 0x55c249469370 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.167+0000 7f4a7d7fa640 1 -- 172.21.15.89:0/2871096599 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a880035f0 con 0x55c249469370 2024-09-15T23:37:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2871096599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 msgr2=0x55c24945f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/2871096599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c24945f930 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f4a88006010 tx=0x7f4a88005fa0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2871096599 shutdown_connections 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/2871096599 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c249469370 0x55c24945f930 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/2871096599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2494642e0 0x55c249475010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/2871096599 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494639a0 0x55c24945f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2871096599 >> 172.21.15.89:0/2871096599 conn(0x55c24945f010 msgr2=0x55c24946dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2871096599 shutdown_connections 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 -- 172.21.15.89:0/2871096599 wait complete. 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 -- start start 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 Processor -- start 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494639a0 0x55c249471f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494642e0 0x55c249472450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c249469370 0x55c2494e2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4a981adf60 con 0x55c2494639a0 2024-09-15T23:37:23.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4a981ad890 con 0x55c249469370 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a981aaad0 con 0x55c2494642e0 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4aa07f6740 10 client.0 _ll_register_callbacks cb 0x7f4a88010c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4a96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494639a0 0x55c249471f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4a96ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494639a0 0x55c249471f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50970/0 (socket says 172.21.15.89:50970) 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.168+0000 7f4a96ffd640 1 -- 172.21.15.89:0/3164624574 learned_addr learned my addr 172.21.15.89:0/3164624574 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a967fc640 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494642e0 0x55c249472450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a977fe640 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c249469370 0x55c2494e2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a96ffd640 1 -- 172.21.15.89:0/3164624574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494642e0 msgr2=0x55c249472450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494642e0 0x55c249472450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a96ffd640 1 -- 172.21.15.89:0/3164624574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c249469370 msgr2=0x55c2494e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c249469370 0x55c2494e2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a96ffd640 1 -- 172.21.15.89:0/3164624574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2494e29c0 con 0x55c2494639a0 2024-09-15T23:37:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a977fe640 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c249469370 0x55c2494e2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a96ffd640 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494639a0 0x55c249471f10 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f4a900981c0 tx=0x7f4a90004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff2d3ec9ea71020 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a76ffd640 1 -- 172.21.15.89:0/3164624574 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90020040 con 0x55c2494639a0 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a76ffd640 1 -- 172.21.15.89:0/3164624574 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4a900042a0 con 0x55c2494639a0 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4aa07f6740 1 -- 172.21.15.89:0/3164624574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c24934b900 con 0x55c2494639a0 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a76ffd640 1 -- 172.21.15.89:0/3164624574 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4a9009b5c0 con 0x55c2494639a0 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4aa07f6740 10 client.20019 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4aa07f6740 1 -- 172.21.15.89:0/3164624574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c2494e3cf0 con 0x55c2494639a0 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a76ffd640 1 -- 172.21.15.89:0/3164624574 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c24934b900 con 0x55c2494639a0 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.169+0000 7f4a76ffd640 1 -- 172.21.15.89:0/3164624574 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4a90097e60 con 0x55c2494639a0 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.170+0000 7f4aa07f6740 10 client.20019 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.170+0000 7f4aa07f6740 10 client.20019 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.170+0000 7f4aa07f6740 20 client.20019 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.170+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2494797a0 0x55c2494e6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.170+0000 7f4aa07f6740 4 client.20019 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:37:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.170+0000 7f4aa07f6740 1 -- 172.21.15.89:0/3164624574 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c249398820 con 0x55c2494797a0 2024-09-15T23:37:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.170+0000 7f4a967fc640 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2494797a0 0x55c2494e6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.171+0000 7f4a76ffd640 10 client.20019 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.171+0000 7f4a967fc640 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2494797a0 0x55c2494e6270 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8157611f99f781d2 server_cookie=3ce7a9dc26e3b103 in_seq=0 out_seq=0 2024-09-15T23:37:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4a76ffd640 1 -- 172.21.15.89:0/3164624574 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c249398820 con 0x55c2494797a0 2024-09-15T23:37:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4a76ffd640 10 client.20019 handle_command_reply: tid=0 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 client.20019 shutdown 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 -- 172.21.15.89:0/3164624574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494639a0 msgr2=0x55c249471f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494639a0 0x55c249471f10 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f4a900981c0 tx=0x7f4a90004710 comp rx=0 tx=0).stop 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 -- 172.21.15.89:0/3164624574 shutdown_connections 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c2494797a0 0x55c2494e6270 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c249469370 0x55c2494e2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c2494642e0 0x55c249472450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4a76ffd640 0 client.20019 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 --2- 172.21.15.89:0/3164624574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c2494639a0 0x55c249471f10 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 -- 172.21.15.89:0/3164624574 >> 172.21.15.89:0/3164624574 conn(0x55c24945f010 msgr2=0x55c24945d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 -- 172.21.15.89:0/3164624574 shutdown_connections 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 1 -- 172.21.15.89:0/3164624574 wait complete. 2024-09-15T23:37:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.184+0000 7f4aa07f6740 20 client.20019 trim_cache size 0 max 16384 2024-09-15T23:37:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:37:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:37:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:37:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 -- 172.21.15.89:0/515593195 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a800a32f0 msgr2=0x7f4a800a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a800a32f0 0x7f4a800a57b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4a88005d50 tx=0x7f4a88005ce0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 -- 172.21.15.89:0/515593195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a98101b60 msgr2=0x7f4a98075fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a98101b60 0x7f4a98075fa0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000d770 tx=0x7f4a9000dc30 comp rx=0 tx=0).stop 2024-09-15T23:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 -- 172.21.15.89:0/515593195 shutdown_connections 2024-09-15T23:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4a800a32f0 0x7f4a800a57b0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a98103710 0x7f4a98074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a98102d50 0x7f4a980745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 --2- 172.21.15.89:0/515593195 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a98101b60 0x7f4a98075fa0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 -- 172.21.15.89:0/515593195 >> 172.21.15.89:0/515593195 conn(0x7f4a980fd2e0 msgr2=0x7f4a98106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 -- 172.21.15.89:0/515593195 shutdown_connections 2024-09-15T23:37:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:23.185+0000 7f4a9da5e640 1 -- 172.21.15.89:0/515593195 wait complete. 2024-09-15T23:37:23.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:22 smithi120 ceph-mon[35250]: pgmap v750: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:37:23.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:22 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:37:23.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:22 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:23.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:22 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:37:23.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:22 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:23.233 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:22 smithi089 ceph-mon[30933]: pgmap v750: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:37:23.233 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:22 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:37:23.233 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:22 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:23.233 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:22 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:37:23.233 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:22 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:23.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:22 smithi174 ceph-mon[35586]: pgmap v750: 129 pgs: 129 active+clean; 2.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:37:23.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:22 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:37:23.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:22 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:23.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:22 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:37:23.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:22 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:23.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1536962286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd01082a0 msgr2=0x7ffbd010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 --2- 172.21.15.89:0/1536962286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd01082a0 0x7ffbd010ec40 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009930 tx=0x7ffbc802a440 comp rx=0 tx=0).stop 2024-09-15T23:37:23.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1536962286 shutdown_connections 2024-09-15T23:37:23.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 --2- 172.21.15.89:0/1536962286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd01082a0 0x7ffbd010ec40 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 --2- 172.21.15.89:0/1536962286 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd01036e0 0x7ffbd0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 --2- 172.21.15.89:0/1536962286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd0102d30 0x7ffbd0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1536962286 >> 172.21.15.89:0/1536962286 conn(0x7ffbd00fc7d0 msgr2=0x7ffbd00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1536962286 shutdown_connections 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1536962286 wait complete. 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 Processor -- start 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.313+0000 7ffbd7e93640 1 -- start start 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd7e93640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd0102d30 0x7ffbd0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd7e93640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd01036e0 0x7ffbd0198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd7e93640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd01082a0 0x7ffbd019d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd7e93640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd0112160 con 0x7ffbd01082a0 2024-09-15T23:37:23.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd7e93640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd0111fe0 con 0x7ffbd01036e0 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd7e93640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd01122e0 con 0x7ffbd0102d30 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd6430640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd01082a0 0x7ffbd019d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd6430640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd01082a0 0x7ffbd019d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50982/0 (socket says 172.21.15.89:50982) 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd6430640 1 -- 172.21.15.89:0/1677212388 learned_addr learned my addr 172.21.15.89:0/1677212388 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd0102d30 0x7ffbd0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd542e640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd01036e0 0x7ffbd0198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd6430640 1 -- 172.21.15.89:0/1677212388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd0102d30 msgr2=0x7ffbd0198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd6430640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd0102d30 0x7ffbd0198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd6430640 1 -- 172.21.15.89:0/1677212388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd01036e0 msgr2=0x7ffbd0198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd6430640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd01036e0 0x7ffbd0198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd6430640 1 -- 172.21.15.89:0/1677212388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd019d6f0 con 0x7ffbd01082a0 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd0102d30 0x7ffbd0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd6430640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd01082a0 0x7ffbd019d010 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8005ab0 tx=0x7ffbc80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbceffd640 1 -- 172.21.15.89:0/1677212388 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc803f030 con 0x7ffbd01082a0 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbceffd640 1 -- 172.21.15.89:0/1677212388 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc8003ce0 con 0x7ffbd01082a0 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1677212388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd019d980 con 0x7ffbd01082a0 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbceffd640 1 -- 172.21.15.89:0/1677212388 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8035070 con 0x7ffbd01082a0 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.314+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1677212388 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd01a52c0 con 0x7ffbd01082a0 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.317+0000 7ffbceffd640 1 -- 172.21.15.89:0/1677212388 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ffbc8042030 con 0x7ffbd01082a0 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.317+0000 7ffbda26c740 1 Processor -- start 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.317+0000 7ffbda26c740 1 -- start start 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.317+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11c44a0 0x560ba11c4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.317+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c4dc0 0x560ba11cf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.317+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11cf680 0x560ba11d1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.317+0000 7ffbda26c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560ba112a2e0 con 0x560ba11c4dc0 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.317+0000 7ffbda26c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560ba10d1ce0 con 0x560ba11cf680 2024-09-15T23:37:23.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.317+0000 7ffbda26c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560ba11262a0 con 0x560ba11c44a0 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd5c2f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c4dc0 0x560ba11cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbceffd640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffbb80a32c0 0x7ffbb80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd5c2f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c4dc0 0x560ba11cf140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:50990/0 (socket says 172.21.15.89:50990) 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd5c2f640 1 -- 172.21.15.89:0/277066661 learned_addr learned my addr 172.21.15.89:0/277066661 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd542e640 1 --2- 172.21.15.89:0/277066661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11c44a0 0x560ba11c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd6430640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffbb80a32c0 0x7ffbb80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd542e640 1 --2- 172.21.15.89:0/277066661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11cf680 0x560ba11d1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd5c2f640 1 -- 172.21.15.89:0/277066661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11c44a0 msgr2=0x560ba11c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/277066661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11c44a0 0x560ba11c4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbceffd640 1 -- 172.21.15.89:0/1677212388 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ffbc80f02a0 con 0x7ffbd01082a0 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd5c2f640 1 -- 172.21.15.89:0/277066661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11cf680 msgr2=0x560ba11d1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/277066661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11cf680 0x560ba11d1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd5c2f640 1 -- 172.21.15.89:0/277066661 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ba10b6440 con 0x560ba11c4dc0 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/277066661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c4dc0 0x560ba11cf140 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000c4a0 tx=0x7ffbc000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b481bf4e959a85 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbd6430640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffbb80a32c0 0x7ffbb80a5780 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80edd20 tx=0x7ffbc8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbb7fff640 1 -- 172.21.15.89:0/277066661 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc001a070 con 0x560ba11c4dc0 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbb7fff640 1 -- 172.21.15.89:0/277066661 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc00158f0 con 0x560ba11c4dc0 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.318+0000 7ffbb7fff640 1 -- 172.21.15.89:0/277066661 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000a960 con 0x560ba11c4dc0 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.319+0000 7ffbda26c740 1 -- 172.21.15.89:0/277066661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c4dc0 msgr2=0x560ba11cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.319+0000 7ffbda26c740 1 --2- 172.21.15.89:0/277066661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c4dc0 0x560ba11cf140 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000c4a0 tx=0x7ffbc000c8a0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.319+0000 7ffbda26c740 1 -- 172.21.15.89:0/277066661 shutdown_connections 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.319+0000 7ffbda26c740 1 --2- 172.21.15.89:0/277066661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11cf680 0x560ba11d1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.319+0000 7ffbda26c740 1 --2- 172.21.15.89:0/277066661 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c4dc0 0x560ba11cf140 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.319+0000 7ffbda26c740 1 --2- 172.21.15.89:0/277066661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11c44a0 0x560ba11c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.319+0000 7ffbda26c740 1 -- 172.21.15.89:0/277066661 >> 172.21.15.89:0/277066661 conn(0x560ba11c0a30 msgr2=0x560ba11c0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.319+0000 7ffbda26c740 1 -- 172.21.15.89:0/277066661 shutdown_connections 2024-09-15T23:37:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.319+0000 7ffbda26c740 1 -- 172.21.15.89:0/277066661 wait complete. 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.319+0000 7ffbda26c740 1 -- start start 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbda26c740 1 Processor -- start 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11c44a0 0x560ba11cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11cf680 0x560ba11cc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbda26c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560ba11d2250 con 0x560ba11cf680 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbda26c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560ba10e5050 con 0x560ba11c4dc0 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbda26c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560ba11d23d0 con 0x560ba11c44a0 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbda26c740 10 client.0 _ll_register_callbacks cb 0x560ba1125160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbd542e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11cf680 0x560ba11cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbd542e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11cf680 0x560ba11cc770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51002/0 (socket says 172.21.15.89:51002) 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbd542e640 1 -- 172.21.15.89:0/211213245 learned_addr learned my addr 172.21.15.89:0/211213245 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbd542e640 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11c44a0 0x560ba11cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbd542e640 1 -- 172.21.15.89:0/211213245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11c44a0 msgr2=0x560ba11cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbd542e640 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11c44a0 0x560ba11cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbd542e640 1 -- 172.21.15.89:0/211213245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 msgr2=0x560ba11cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbd542e640 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.320+0000 7ffbd542e640 1 -- 172.21.15.89:0/211213245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ba0e9b3c0 con 0x560ba11cf680 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbd542e640 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11cf680 0x560ba11cc770 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00697d0 tx=0x7ffbc40047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fbc087164aef4b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbb57fa640 1 -- 172.21.15.89:0/211213245 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc400f040 con 0x560ba11cf680 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbb57fa640 1 -- 172.21.15.89:0/211213245 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ffbc4011bf0 con 0x560ba11cf680 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbda26c740 1 -- 172.21.15.89:0/211213245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc00158f0 con 0x560ba11cf680 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbb57fa640 1 -- 172.21.15.89:0/211213245 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffbc4026050 con 0x560ba11cf680 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbda26c740 10 client.20031 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbda26c740 1 -- 172.21.15.89:0/211213245 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560ba11ddc20 con 0x560ba11cf680 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbb57fa640 1 -- 172.21.15.89:0/211213245 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc00158f0 con 0x560ba11cf680 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbb57fa640 1 -- 172.21.15.89:0/211213245 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ffbc4015050 con 0x560ba11cf680 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbda26c740 10 client.20031 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbda26c740 10 client.20031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbda26c740 20 client.20031 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbda26c740 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560ba11df0e0 0x560ba11e14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbda26c740 4 client.20031 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.321+0000 7ffbda26c740 1 -- 172.21.15.89:0/211213245 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ba1137850 con 0x560ba11df0e0 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.322+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560ba11df0e0 0x560ba11e14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.322+0000 7ffbb57fa640 10 client.20031 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.322+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560ba11df0e0 0x560ba11e14f0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19a0699a7e68f219 server_cookie=f1db73943597ef00 in_seq=0 out_seq=0 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbb57fa640 1 -- 172.21.15.89:0/211213245 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560ba1137850 con 0x560ba11df0e0 2024-09-15T23:37:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbb57fa640 10 client.20031 handle_command_reply: tid=0 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 client.20031 shutdown 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 -- 172.21.15.89:0/211213245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11cf680 msgr2=0x560ba11cc770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11cf680 0x560ba11cc770 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00697d0 tx=0x7ffbc40047d0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 -- 172.21.15.89:0/211213245 shutdown_connections 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560ba11df0e0 0x560ba11e14f0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11cf680 0x560ba11cc770 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbb57fa640 0 client.20031 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 --2- 172.21.15.89:0/211213245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11c44a0 0x560ba11cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 -- 172.21.15.89:0/211213245 >> 172.21.15.89:0/211213245 conn(0x560ba11c0a30 msgr2=0x560ba11c6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 -- 172.21.15.89:0/211213245 shutdown_connections 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 1 -- 172.21.15.89:0/211213245 wait complete. 2024-09-15T23:37:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.323+0000 7ffbda26c740 20 client.20031 trim_cache size 0 max 16384 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.340+0000 7ffbda26c740 1 Processor -- start 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.340+0000 7ffbda26c740 1 -- start start 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.340+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11e6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.340+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11e7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.340+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11cf680 0x560ba11c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.340+0000 7ffbda26c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd01aefd0 con 0x560ba11c44a0 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.340+0000 7ffbda26c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd01b0b10 con 0x560ba11c4dc0 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.340+0000 7ffbda26c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd01af6a0 con 0x560ba11cf680 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd542e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11e6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd542e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11e6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51014/0 (socket says 172.21.15.89:51014) 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd542e640 1 -- 172.21.15.89:0/1255780456 learned_addr learned my addr 172.21.15.89:0/1255780456 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/1255780456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11e7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd542e640 1 -- 172.21.15.89:0/1255780456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11cf680 msgr2=0x560ba11c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd542e640 1 --2- 172.21.15.89:0/1255780456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11cf680 0x560ba11c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd542e640 1 -- 172.21.15.89:0/1255780456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 msgr2=0x560ba11e7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd542e640 1 --2- 172.21.15.89:0/1255780456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11e7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd542e640 1 -- 172.21.15.89:0/1255780456 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ba11cb1e0 con 0x560ba11c44a0 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/1255780456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11e7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbd542e640 1 --2- 172.21.15.89:0/1255780456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11e6dc0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4001eb0 tx=0x7ffbc4006db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd1ed155701dd16 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbb77fe640 1 -- 172.21.15.89:0/1255780456 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc400a070 con 0x560ba11c44a0 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbb77fe640 1 -- 172.21.15.89:0/1255780456 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc401d070 con 0x560ba11c44a0 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.341+0000 7ffbb77fe640 1 -- 172.21.15.89:0/1255780456 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4018420 con 0x560ba11c44a0 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 -- 172.21.15.89:0/1255780456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 msgr2=0x560ba11e6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 --2- 172.21.15.89:0/1255780456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11e6dc0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4001eb0 tx=0x7ffbc4006db0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 -- 172.21.15.89:0/1255780456 shutdown_connections 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 --2- 172.21.15.89:0/1255780456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11cf680 0x560ba11c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 --2- 172.21.15.89:0/1255780456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11e7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 --2- 172.21.15.89:0/1255780456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11e6dc0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 -- 172.21.15.89:0/1255780456 >> 172.21.15.89:0/1255780456 conn(0x560ba11c0a30 msgr2=0x560ba11ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 -- 172.21.15.89:0/1255780456 shutdown_connections 2024-09-15T23:37:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 -- 172.21.15.89:0/1255780456 wait complete. 2024-09-15T23:37:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 -- start start 2024-09-15T23:37:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.342+0000 7ffbda26c740 1 Processor -- start 2024-09-15T23:37:23.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11d56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11d5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbda26c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11cf680 0x560ba1248da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbda26c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd01b0440 con 0x560ba11c44a0 2024-09-15T23:37:23.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbda26c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd01afd70 con 0x560ba11c4dc0 2024-09-15T23:37:23.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbda26c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd01acfb0 con 0x560ba11cf680 2024-09-15T23:37:23.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbda26c740 10 client.0 _ll_register_callbacks cb 0x7ffbc400c610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbd542e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11d56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbd542e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11d56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51026/0 (socket says 172.21.15.89:51026) 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbd542e640 1 -- 172.21.15.89:0/2709389598 learned_addr learned my addr 172.21.15.89:0/2709389598 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11d5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbd542e640 1 -- 172.21.15.89:0/2709389598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11cf680 msgr2=0x560ba1248da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbd542e640 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11cf680 0x560ba1248da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbd542e640 1 -- 172.21.15.89:0/2709389598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 msgr2=0x560ba11d5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbd542e640 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11d5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbd542e640 1 -- 172.21.15.89:0/2709389598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ba11cf2f0 con 0x560ba11c44a0 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbd542e640 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11d56d0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400cfd0 tx=0x7ffbc4011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710e38a4f51a8ae8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.343+0000 7ffbb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbb57fa640 1 -- 172.21.15.89:0/2709389598 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4039020 con 0x560ba11c44a0 2024-09-15T23:37:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbb57fa640 1 -- 172.21.15.89:0/2709389598 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ffbc40059c0 con 0x560ba11c44a0 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbda26c740 1 -- 172.21.15.89:0/2709389598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbc401d070 con 0x560ba11c44a0 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbb57fa640 1 -- 172.21.15.89:0/2709389598 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffbc400c020 con 0x560ba11c44a0 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbda26c740 10 client.20037 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbda26c740 1 -- 172.21.15.89:0/2709389598 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560ba124a150 con 0x560ba11c44a0 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbb57fa640 1 -- 172.21.15.89:0/2709389598 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbc401d070 con 0x560ba11c44a0 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbb57fa640 1 -- 172.21.15.89:0/2709389598 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ffbc4015040 con 0x560ba11c44a0 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbda26c740 10 client.20037 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbda26c740 10 client.20037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbda26c740 20 client.20037 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbda26c740 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560ba124a800 0x560ba124cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbda26c740 4 client.20037 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.344+0000 7ffbda26c740 1 -- 172.21.15.89:0/2709389598 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560ba1131d60 con 0x560ba124a800 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560ba124a800 0x560ba124cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbd5c2f640 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560ba124a800 0x560ba124cbf0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f117bd81d6da1704 server_cookie=1e2a442959195117 in_seq=0 out_seq=0 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbb57fa640 10 client.20037 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbb57fa640 1 -- 172.21.15.89:0/2709389598 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560ba1131d60 con 0x560ba124a800 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbb57fa640 10 client.20037 handle_command_reply: tid=0 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 client.20037 shutdown 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 -- 172.21.15.89:0/2709389598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 msgr2=0x560ba11d56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11d56d0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400cfd0 tx=0x7ffbc4011c50 comp rx=0 tx=0).stop 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 -- 172.21.15.89:0/2709389598 shutdown_connections 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560ba124a800 0x560ba124cbf0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560ba11cf680 0x560ba1248da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ba11c4dc0 0x560ba11d5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbb57fa640 0 client.20037 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 --2- 172.21.15.89:0/2709389598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ba11c44a0 0x560ba11d56d0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 -- 172.21.15.89:0/2709389598 >> 172.21.15.89:0/2709389598 conn(0x560ba11c0a30 msgr2=0x560ba11bf350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 -- 172.21.15.89:0/2709389598 shutdown_connections 2024-09-15T23:37:23.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.345+0000 7ffbda26c740 1 -- 172.21.15.89:0/2709389598 wait complete. 2024-09-15T23:37:23.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbda26c740 20 client.20037 trim_cache size 0 max 16384 2024-09-15T23:37:23.362 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:37:23.362 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:37:23.362 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:37:23.362 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:37:23.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1677212388 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffbb80a32c0 msgr2=0x7ffbb80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffbb80a32c0 0x7ffbb80a5780 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80edd20 tx=0x7ffbc8009260 comp rx=0 tx=0).stop 2024-09-15T23:37:23.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1677212388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd01082a0 msgr2=0x7ffbd019d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:23.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd01082a0 0x7ffbd019d010 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8005ab0 tx=0x7ffbc80042f0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1677212388 shutdown_connections 2024-09-15T23:37:23.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ffbb80a32c0 0x7ffbb80a5780 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbd01082a0 0x7ffbd019d010 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbd01036e0 0x7ffbd0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 --2- 172.21.15.89:0/1677212388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd0102d30 0x7ffbd0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:23.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1677212388 >> 172.21.15.89:0/1677212388 conn(0x7ffbd00fc7d0 msgr2=0x7ffbd0106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:23.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1677212388 shutdown_connections 2024-09-15T23:37:23.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:23.346+0000 7ffbd7e93640 1 -- 172.21.15.89:0/1677212388 wait complete. 2024-09-15T23:37:24.138 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:23 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:37:24.170 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:37:24.170 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:37:24.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:23 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:37:24.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:23 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:37:24.420 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:24.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.863+0000 7f6407e89640 1 -- 172.21.15.89:0/758184420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6400106f10 msgr2=0x7f640010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:24.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.863+0000 7f6407e89640 1 --2- 172.21.15.89:0/758184420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6400106f10 0x7f640010ea90 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009ca0 tx=0x7f63f002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:37:24.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.863+0000 7f6407e89640 1 -- 172.21.15.89:0/758184420 shutdown_connections 2024-09-15T23:37:24.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.863+0000 7f6407e89640 1 --2- 172.21.15.89:0/758184420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6400106f10 0x7f640010ea90 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:24.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.863+0000 7f6407e89640 1 --2- 172.21.15.89:0/758184420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6400102790 0x7f64001067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:24.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.863+0000 7f6407e89640 1 --2- 172.21.15.89:0/758184420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6400101de0 0x7f64001021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:24.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.863+0000 7f6407e89640 1 -- 172.21.15.89:0/758184420 >> 172.21.15.89:0/758184420 conn(0x7f64000fd8c0 msgr2=0x7f64000ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:24.893 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.863+0000 7f6407e89640 1 -- 172.21.15.89:0/758184420 shutdown_connections 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.863+0000 7f6407e89640 1 -- 172.21.15.89:0/758184420 wait complete. 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6407e89640 1 Processor -- start 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6407e89640 1 -- start start 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6407e89640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6400101de0 0x7f64000744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6407e89640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6400102790 0x7f6400074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6407e89640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6400106f10 0x7f6400074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6407e89640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6400111f70 con 0x7f6400106f10 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6407e89640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6400111df0 con 0x7f6400101de0 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6407e89640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f64001120f0 con 0x7f6400102790 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6406426640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6400106f10 0x7f6400074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6406426640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6400106f10 0x7f6400074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51040/0 (socket says 172.21.15.89:51040) 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.864+0000 7f6406426640 1 -- 172.21.15.89:0/336954145 learned_addr learned my addr 172.21.15.89:0/336954145 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:24.894 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f6405c25640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6400101de0 0x7f64000744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f6405424640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6400102790 0x7f6400074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f6406426640 1 -- 172.21.15.89:0/336954145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6400102790 msgr2=0x7f6400074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f6406426640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6400102790 0x7f6400074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f6406426640 1 -- 172.21.15.89:0/336954145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6400101de0 msgr2=0x7f64000744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f6406426640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6400101de0 0x7f64000744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f6406426640 1 -- 172.21.15.89:0/336954145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64001a6800 con 0x7f6400106f10 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f6406426640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6400106f10 0x7f6400074f70 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f63f0031a20 tx=0x7f63f0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f63feffd640 1 -- 172.21.15.89:0/336954145 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f0003c30 con 0x7f6400106f10 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f6407e89640 1 -- 172.21.15.89:0/336954145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64001a6a90 con 0x7f6400106f10 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f63feffd640 1 -- 172.21.15.89:0/336954145 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f0003dd0 con 0x7f6400106f10 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f63feffd640 1 -- 172.21.15.89:0/336954145 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f002fcf0 con 0x7f6400106f10 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.865+0000 7f6407e89640 1 -- 172.21.15.89:0/336954145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64001a6fd0 con 0x7f6400106f10 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.866+0000 7f6407e89640 1 -- 172.21.15.89:0/336954145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64001021c0 con 0x7f6400106f10 2024-09-15T23:37:24.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.867+0000 7f63feffd640 1 -- 172.21.15.89:0/336954145 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f63f0004b40 con 0x7f6400106f10 2024-09-15T23:37:24.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.868+0000 7f63feffd640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63e80a3240 0x7f63e80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:24.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.868+0000 7f63feffd640 1 -- 172.21.15.89:0/336954145 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f63f00f1280 con 0x7f6400106f10 2024-09-15T23:37:24.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.868+0000 7f6405c25640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63e80a3240 0x7f63e80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:24.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.868+0000 7f6405c25640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63e80a3240 0x7f63e80a5700 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f63f8005fd0 tx=0x7f63f8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:24.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:24.871+0000 7f63feffd640 1 -- 172.21.15.89:0/336954145 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f63f00bdc40 con 0x7f6400106f10 2024-09-15T23:37:25.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.073+0000 7f6407e89640 1 -- 172.21.15.89:0/336954145 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f640010a150 con 0x7f6400106f10 2024-09-15T23:37:25.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.074+0000 7f63feffd640 1 -- 172.21.15.89:0/336954145 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f63f002fe90 con 0x7f6400106f10 2024-09-15T23:37:25.075 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:37:25.075 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:37:25.076 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:37:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 -- 172.21.15.89:0/336954145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63e80a3240 msgr2=0x7f63e80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63e80a3240 0x7f63e80a5700 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f63f8005fd0 tx=0x7f63f8005850 comp rx=0 tx=0).stop 2024-09-15T23:37:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 -- 172.21.15.89:0/336954145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6400106f10 msgr2=0x7f6400074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6400106f10 0x7f6400074f70 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f63f0031a20 tx=0x7f63f0031a50 comp rx=0 tx=0).stop 2024-09-15T23:37:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 -- 172.21.15.89:0/336954145 shutdown_connections 2024-09-15T23:37:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f63e80a3240 0x7f63e80a5700 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6400106f10 0x7f6400074f70 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6400102790 0x7f6400074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:25.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 --2- 172.21.15.89:0/336954145 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6400101de0 0x7f64000744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:25.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 -- 172.21.15.89:0/336954145 >> 172.21.15.89:0/336954145 conn(0x7f64000fd8c0 msgr2=0x7f64000febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:25.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 -- 172.21.15.89:0/336954145 shutdown_connections 2024-09-15T23:37:25.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:25.076+0000 7f6407e89640 1 -- 172.21.15.89:0/336954145 wait complete. 2024-09-15T23:37:25.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:24 smithi120 ceph-mon[35250]: pgmap v751: 129 pgs: 129 active+clean; 2.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-15T23:37:25.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:24 smithi174 ceph-mon[35586]: pgmap v751: 129 pgs: 129 active+clean; 2.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-15T23:37:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:24 smithi089 ceph-mon[30933]: pgmap v751: 129 pgs: 129 active+clean; 2.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-15T23:37:25.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:37:26.000 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:26.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:25 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/336954145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:26.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:25 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/336954145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:26.270 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:25 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/336954145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:26.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.499+0000 7f43fa510640 1 -- 172.21.15.89:0/2082497469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f4102d40 msgr2=0x7f43f4103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.499+0000 7f43fa510640 1 --2- 172.21.15.89:0/2082497469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f4102d40 0x7f43f4103120 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f43e4009900 tx=0x7f43e402a480 comp rx=0 tx=0).stop 2024-09-15T23:37:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.499+0000 7f43fa510640 1 -- 172.21.15.89:0/2082497469 shutdown_connections 2024-09-15T23:37:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.499+0000 7f43fa510640 1 --2- 172.21.15.89:0/2082497469 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f410c170 0x7f43f410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.499+0000 7f43fa510640 1 --2- 172.21.15.89:0/2082497469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103660 0x7f43f410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.499+0000 7f43fa510640 1 --2- 172.21.15.89:0/2082497469 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f4102d40 0x7f43f4103120 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.499+0000 7f43fa510640 1 -- 172.21.15.89:0/2082497469 >> 172.21.15.89:0/2082497469 conn(0x7f43f40fc7a0 msgr2=0x7f43f40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.499+0000 7f43fa510640 1 -- 172.21.15.89:0/2082497469 shutdown_connections 2024-09-15T23:37:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.499+0000 7f43fa510640 1 -- 172.21.15.89:0/2082497469 wait complete. 2024-09-15T23:37:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43fa510640 1 Processor -- start 2024-09-15T23:37:26.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43fa510640 1 -- start start 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43fa510640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f4102d40 0x7f43f4198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43fa510640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103660 0x7f43f41989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43fa510640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f410c170 0x7f43f419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43fa510640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f43f4111f70 con 0x7f43f4102d40 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43fa510640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f43f4111df0 con 0x7f43f410c170 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43fa510640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41120f0 con 0x7f43f4103660 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f4102d40 0x7f43f4198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f4102d40 0x7f43f4198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34152/0 (socket says 172.21.15.89:34152) 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.500+0000 7f43f3fff640 1 -- 172.21.15.89:0/3174051707 learned_addr learned my addr 172.21.15.89:0/3174051707 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f37fe640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103660 0x7f43f41989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f8aad640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f410c170 0x7f43f419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f3fff640 1 -- 172.21.15.89:0/3174051707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103660 msgr2=0x7f43f41989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:26.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f3fff640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103660 0x7f43f41989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f3fff640 1 -- 172.21.15.89:0/3174051707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f410c170 msgr2=0x7f43f419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f3fff640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f410c170 0x7f43f419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f3fff640 1 -- 172.21.15.89:0/3174051707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f419d420 con 0x7f43f4102d40 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f8aad640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f410c170 0x7f43f419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f3fff640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f4102d40 0x7f43f4198470 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f43e4006010 tx=0x7f43e40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f17fa640 1 -- 172.21.15.89:0/3174051707 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e403f030 con 0x7f43f4102d40 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43fa510640 1 -- 172.21.15.89:0/3174051707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f419d6b0 con 0x7f43f4102d40 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f17fa640 1 -- 172.21.15.89:0/3174051707 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43e4003ce0 con 0x7f43f4102d40 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43f17fa640 1 -- 172.21.15.89:0/3174051707 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e4035070 con 0x7f43f4102d40 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.501+0000 7f43fa510640 1 -- 172.21.15.89:0/3174051707 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f41a4ff0 con 0x7f43f4102d40 2024-09-15T23:37:26.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.503+0000 7f43f17fa640 1 -- 172.21.15.89:0/3174051707 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f43e4042030 con 0x7f43f4102d40 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43fc8e9740 1 Processor -- start 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43fc8e9740 1 -- start start 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba76090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 0x56281ba76a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba87b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56281b9dc2f0 con 0x56281ba85780 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56281b983cf0 con 0x56281ba77b40 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56281b9d82b0 con 0x56281ba765d0 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f17fa640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43dc0a3240 0x7f43dc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba87b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba87b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34154/0 (socket says 172.21.15.89:34154) 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f37fe640 1 -- 172.21.15.89:0/1273036761 learned_addr learned my addr 172.21.15.89:0/1273036761 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f3fff640 1 --2- 172.21.15.89:0/1273036761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 0x56281ba76a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f8aad640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43dc0a3240 0x7f43dc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f8aad640 1 --2- 172.21.15.89:0/1273036761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba76090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f17fa640 1 -- 172.21.15.89:0/3174051707 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f43e40f0210 con 0x7f43f4102d40 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f37fe640 1 -- 172.21.15.89:0/1273036761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 msgr2=0x56281ba76a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f37fe640 1 --2- 172.21.15.89:0/1273036761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 0x56281ba76a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f37fe640 1 -- 172.21.15.89:0/1273036761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 msgr2=0x56281ba76090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f37fe640 1 --2- 172.21.15.89:0/1273036761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba76090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.504+0000 7f43f37fe640 1 -- 172.21.15.89:0/1273036761 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56281b9683a0 con 0x56281ba85780 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.505+0000 7f43f8aad640 1 --2- 172.21.15.89:0/1273036761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba76090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.505+0000 7f43f37fe640 1 --2- 172.21.15.89:0/1273036761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba87b70 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f43e800abd0 tx=0x7f43e800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b705b43a52e2d156 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.505+0000 7f43f8aad640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43dc0a3240 0x7f43dc0a5700 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f43ec005c40 tx=0x7f43ec009530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.505+0000 7f43da7fc640 1 -- 172.21.15.89:0/1273036761 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e801d070 con 0x56281ba85780 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.505+0000 7f43da7fc640 1 -- 172.21.15.89:0/1273036761 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43e8018b20 con 0x56281ba85780 2024-09-15T23:37:26.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.505+0000 7f43da7fc640 1 -- 172.21.15.89:0/1273036761 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e8010bb0 con 0x56281ba85780 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.505+0000 7f43fc8e9740 1 -- 172.21.15.89:0/1273036761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 msgr2=0x56281ba87b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.505+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/1273036761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba87b70 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f43e800abd0 tx=0x7f43e800f790 comp rx=0 tx=0).stop 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 -- 172.21.15.89:0/1273036761 shutdown_connections 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/1273036761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba87b70 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/1273036761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 0x56281ba76a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/1273036761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba76090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 -- 172.21.15.89:0/1273036761 >> 172.21.15.89:0/1273036761 conn(0x56281ba3f310 msgr2=0x56281ba72c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 -- 172.21.15.89:0/1273036761 shutdown_connections 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 -- 172.21.15.89:0/1273036761 wait complete. 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 -- start start 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 Processor -- start 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba765d0 0x56281ba85140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba823f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba85780 0x56281ba829e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56281ba88480 con 0x56281ba765d0 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56281b997060 con 0x56281ba77b40 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56281ba88600 con 0x56281ba85780 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43fc8e9740 10 client.0 _ll_register_callbacks cb 0x56281b6ea8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba765d0 0x56281ba85140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba765d0 0x56281ba85140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34170/0 (socket says 172.21.15.89:34170) 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.506+0000 7f43f3fff640 1 -- 172.21.15.89:0/2493397339 learned_addr learned my addr 172.21.15.89:0/2493397339 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43f8aad640 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba85780 0x56281ba829e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43f37fe640 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba823f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43f3fff640 1 -- 172.21.15.89:0/2493397339 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba85780 msgr2=0x56281ba829e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43f3fff640 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba85780 0x56281ba829e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43f3fff640 1 -- 172.21.15.89:0/2493397339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 msgr2=0x56281ba823f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:26.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43f3fff640 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba823f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43f3fff640 1 -- 172.21.15.89:0/2493397339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56281b74d3c0 con 0x56281ba765d0 2024-09-15T23:37:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43f8aad640 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba85780 0x56281ba829e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43f37fe640 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba823f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43f3fff640 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba765d0 0x56281ba85140 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f43e404a000 tx=0x7f43e4040fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69132ff5b118317d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43d37fe640 1 -- 172.21.15.89:0/2493397339 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e4049040 con 0x56281ba765d0 2024-09-15T23:37:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43d37fe640 1 -- 172.21.15.89:0/2493397339 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f43e40b8b70 con 0x56281ba765d0 2024-09-15T23:37:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43fc8e9740 1 -- 172.21.15.89:0/2493397339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43e8018b20 con 0x56281ba765d0 2024-09-15T23:37:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43d37fe640 1 -- 172.21.15.89:0/2493397339 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f43e40bc040 con 0x56281ba765d0 2024-09-15T23:37:26.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43fc8e9740 10 client.20055 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43fc8e9740 1 -- 172.21.15.89:0/2493397339 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56281ba93ee0 con 0x56281ba765d0 2024-09-15T23:37:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43d37fe640 1 -- 172.21.15.89:0/2493397339 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43e8018b20 con 0x56281ba765d0 2024-09-15T23:37:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.507+0000 7f43d37fe640 1 -- 172.21.15.89:0/2493397339 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f43e40bc760 con 0x56281ba765d0 2024-09-15T23:37:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.508+0000 7f43fc8e9740 10 client.20055 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.508+0000 7f43fc8e9740 10 client.20055 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:37:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.508+0000 7f43fc8e9740 20 client.20055 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.508+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56281ba95230 0x56281ba976a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.508+0000 7f43fc8e9740 4 client.20055 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.508+0000 7f43fc8e9740 1 -- 172.21.15.89:0/2493397339 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56281b9e9860 con 0x56281ba95230 2024-09-15T23:37:26.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.508+0000 7f43f37fe640 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56281ba95230 0x56281ba976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.508+0000 7f43d37fe640 10 client.20055 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.508+0000 7f43f37fe640 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56281ba95230 0x56281ba976a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=be82f77c1682ce8 server_cookie=1a2a579b3b144f3 in_seq=0 out_seq=0 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43d37fe640 1 -- 172.21.15.89:0/2493397339 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56281b9e9860 con 0x56281ba95230 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43d37fe640 10 client.20055 handle_command_reply: tid=0 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 client.20055 shutdown 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 -- 172.21.15.89:0/2493397339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba765d0 msgr2=0x56281ba85140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba765d0 0x56281ba85140 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f43e404a000 tx=0x7f43e4040fb0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 -- 172.21.15.89:0/2493397339 shutdown_connections 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56281ba95230 0x56281ba976a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba85780 0x56281ba829e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba823f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43d37fe640 0 client.20055 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:37:26.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/2493397339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba765d0 0x56281ba85140 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 -- 172.21.15.89:0/2493397339 >> 172.21.15.89:0/2493397339 conn(0x56281ba3f310 msgr2=0x56281ba78960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:26.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 -- 172.21.15.89:0/2493397339 shutdown_connections 2024-09-15T23:37:26.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.509+0000 7f43fc8e9740 1 -- 172.21.15.89:0/2493397339 wait complete. 2024-09-15T23:37:26.530 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.510+0000 7f43fc8e9740 20 client.20055 trim_cache size 0 max 16384 2024-09-15T23:37:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.522+0000 7f43fc8e9740 1 Processor -- start 2024-09-15T23:37:26.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.522+0000 7f43fc8e9740 1 -- start start 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 0x56281ba91720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba75de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba86b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41acc60 con 0x56281ba85780 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41ae7a0 con 0x56281ba77b40 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41ad330 con 0x56281ba765d0 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f8aad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba86b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f8aad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba86b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34174/0 (socket says 172.21.15.89:34174) 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f8aad640 1 -- 172.21.15.89:0/823935677 learned_addr learned my addr 172.21.15.89:0/823935677 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f37fe640 1 --2- 172.21.15.89:0/823935677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba75de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f3fff640 1 --2- 172.21.15.89:0/823935677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 0x56281ba91720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f8aad640 1 -- 172.21.15.89:0/823935677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 msgr2=0x56281ba91720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:26.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f8aad640 1 --2- 172.21.15.89:0/823935677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 0x56281ba91720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f8aad640 1 -- 172.21.15.89:0/823935677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 msgr2=0x56281ba75de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f8aad640 1 --2- 172.21.15.89:0/823935677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba75de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f8aad640 1 -- 172.21.15.89:0/823935677 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56281ba77400 con 0x56281ba85780 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f3fff640 1 --2- 172.21.15.89:0/823935677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 0x56281ba91720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43f8aad640 1 --2- 172.21.15.89:0/823935677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba86b70 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009790 tx=0x7f43ec014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d00f9e0c2287892 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43d9ffb640 1 -- 172.21.15.89:0/823935677 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec02c070 con 0x56281ba85780 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43d9ffb640 1 -- 172.21.15.89:0/823935677 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43ec01da40 con 0x56281ba85780 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.523+0000 7f43d9ffb640 1 -- 172.21.15.89:0/823935677 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec027c90 con 0x56281ba85780 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.524+0000 7f43fc8e9740 1 -- 172.21.15.89:0/823935677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 msgr2=0x56281ba86b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.524+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/823935677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba86b70 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009790 tx=0x7f43ec014040 comp rx=0 tx=0).stop 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.524+0000 7f43fc8e9740 1 -- 172.21.15.89:0/823935677 shutdown_connections 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.524+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/823935677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba86b70 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.524+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/823935677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba77b40 0x56281ba75de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.524+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/823935677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba765d0 0x56281ba91720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.524+0000 7f43fc8e9740 1 -- 172.21.15.89:0/823935677 >> 172.21.15.89:0/823935677 conn(0x56281ba3f310 msgr2=0x56281ba80cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.524+0000 7f43fc8e9740 1 -- 172.21.15.89:0/823935677 shutdown_connections 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.524+0000 7f43fc8e9740 1 -- 172.21.15.89:0/823935677 wait complete. 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.524+0000 7f43fc8e9740 1 -- start start 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43fc8e9740 1 Processor -- start 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba765d0 0x56281ba8e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba77b40 0x56281ba9ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43fc8e9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba9d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41ae0d0 con 0x56281ba85780 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41ada00 con 0x56281ba765d0 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43fc8e9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43f41aac40 con 0x56281ba77b40 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43fc8e9740 10 client.0 _ll_register_callbacks cb 0x7f43e40b87e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:26.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43f8aad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba9d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43f8aad640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba9d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34182/0 (socket says 172.21.15.89:34182) 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43f8aad640 1 -- 172.21.15.89:0/282425472 learned_addr learned my addr 172.21.15.89:0/282425472 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43f3fff640 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba765d0 0x56281ba8e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43f37fe640 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba77b40 0x56281ba9ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43f8aad640 1 -- 172.21.15.89:0/282425472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba77b40 msgr2=0x56281ba9ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.525+0000 7f43f8aad640 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba77b40 0x56281ba9ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43f8aad640 1 -- 172.21.15.89:0/282425472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba765d0 msgr2=0x56281ba8e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43f8aad640 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba765d0 0x56281ba8e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43f8aad640 1 -- 172.21.15.89:0/282425472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56281ba8bc80 con 0x56281ba85780 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43f37fe640 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba77b40 0x56281ba9ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43f8aad640 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba9d190 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009760 tx=0x7f43ec01dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306deadd3ef5f181 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:26.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43d37fe640 1 -- 172.21.15.89:0/282425472 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec02c040 con 0x56281ba85780 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43d37fe640 1 -- 172.21.15.89:0/282425472 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f43ec01d530 con 0x56281ba85780 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43fc8e9740 1 -- 172.21.15.89:0/282425472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43ec01da40 con 0x56281ba85780 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43d37fe640 1 -- 172.21.15.89:0/282425472 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f43ec027440 con 0x56281ba85780 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43fc8e9740 10 client.20061 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43fc8e9740 1 -- 172.21.15.89:0/282425472 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56281bb000c0 con 0x56281ba85780 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.526+0000 7f43d37fe640 1 -- 172.21.15.89:0/282425472 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43ec01da40 con 0x56281ba85780 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43d37fe640 1 -- 172.21.15.89:0/282425472 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f43ec038030 con 0x56281ba85780 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43fc8e9740 10 client.20061 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43fc8e9740 10 client.20061 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43fc8e9740 20 client.20061 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56281bb00720 0x56281bb02b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43fc8e9740 4 client.20061 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:37:26.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43fc8e9740 1 -- 172.21.15.89:0/282425472 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56281b9e3d70 con 0x56281bb00720 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43f3fff640 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56281bb00720 0x56281bb02b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43d37fe640 10 client.20061 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43f3fff640 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56281bb00720 0x56281bb02b40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=44a87cbb10890c14 server_cookie=757e913fbaeb7d44 in_seq=0 out_seq=0 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43d37fe640 1 -- 172.21.15.89:0/282425472 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56281b9e3d70 con 0x56281bb00720 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.527+0000 7f43d37fe640 10 client.20061 handle_command_reply: tid=0 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 client.20061 shutdown 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 -- 172.21.15.89:0/282425472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 msgr2=0x56281ba9d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba9d190 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009760 tx=0x7f43ec01dea0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 -- 172.21.15.89:0/282425472 shutdown_connections 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x56281bb00720 0x56281bb02b40 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56281ba85780 0x56281ba9d190 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43d37fe640 0 client.20061 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:37:26.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56281ba77b40 0x56281ba9ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 --2- 172.21.15.89:0/282425472 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56281ba765d0 0x56281ba8e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 -- 172.21.15.89:0/282425472 >> 172.21.15.89:0/282425472 conn(0x56281ba3f310 msgr2=0x56281ba810a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:26.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 -- 172.21.15.89:0/282425472 shutdown_connections 2024-09-15T23:37:26.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 1 -- 172.21.15.89:0/282425472 wait complete. 2024-09-15T23:37:26.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.528+0000 7f43fc8e9740 20 client.20061 trim_cache size 0 max 16384 2024-09-15T23:37:26.541 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:37:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 -- 172.21.15.89:0/3174051707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43dc0a3240 msgr2=0x7f43dc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43dc0a3240 0x7f43dc0a5700 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f43ec005c40 tx=0x7f43ec009530 comp rx=0 tx=0).stop 2024-09-15T23:37:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 -- 172.21.15.89:0/3174051707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f4102d40 msgr2=0x7f43f4198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f4102d40 0x7f43f4198470 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f43e4006010 tx=0x7f43e40042f0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 -- 172.21.15.89:0/3174051707 shutdown_connections 2024-09-15T23:37:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43dc0a3240 0x7f43dc0a5700 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43f410c170 0x7f43f419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43f4103660 0x7f43f41989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 --2- 172.21.15.89:0/3174051707 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43f4102d40 0x7f43f4198470 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:26.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 -- 172.21.15.89:0/3174051707 >> 172.21.15.89:0/3174051707 conn(0x7f43f40fc7a0 msgr2=0x7f43f40fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:26.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 -- 172.21.15.89:0/3174051707 shutdown_connections 2024-09-15T23:37:26.543 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:26.529+0000 7f43fa510640 1 -- 172.21.15.89:0/3174051707 wait complete. 2024-09-15T23:37:27.208 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:26 smithi089 ceph-mon[30933]: pgmap v752: 129 pgs: 129 active+clean; 2.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:37:27.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:26 smithi120 ceph-mon[35250]: pgmap v752: 129 pgs: 129 active+clean; 2.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:37:27.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:26 smithi174 ceph-mon[35586]: pgmap v752: 129 pgs: 129 active+clean; 2.7 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:37:27.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:37:27.485 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:28.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2177481675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b18102790 msgr2=0x7f1b181067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:28.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 --2- 172.21.15.89:0/2177481675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b18102790 0x7f1b181067e0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f1b04009900 tx=0x7f1b0402a410 comp rx=0 tx=0).stop 2024-09-15T23:37:28.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2177481675 shutdown_connections 2024-09-15T23:37:28.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 --2- 172.21.15.89:0/2177481675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b18106f10 0x7f1b1810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 --2- 172.21.15.89:0/2177481675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b18102790 0x7f1b181067e0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 --2- 172.21.15.89:0/2177481675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b18101de0 0x7f1b181021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2177481675 >> 172.21.15.89:0/2177481675 conn(0x7f1b180fd8c0 msgr2=0x7f1b180ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:28.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2177481675 shutdown_connections 2024-09-15T23:37:28.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2177481675 wait complete. 2024-09-15T23:37:28.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 Processor -- start 2024-09-15T23:37:28.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 -- start start 2024-09-15T23:37:28.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b18101de0 0x7f1b180744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b18102790 0x7f1b18074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b18106f10 0x7f1b18074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1b18111f70 con 0x7f1b18102790 2024-09-15T23:37:28.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1b18111df0 con 0x7f1b18106f10 2024-09-15T23:37:28.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.002+0000 7f1b1f1a7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b181120f0 con 0x7f1b18101de0 2024-09-15T23:37:28.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b17fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b18102790 0x7f1b18074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b17fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b18102790 0x7f1b18074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34198/0 (socket says 172.21.15.89:34198) 2024-09-15T23:37:28.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b17fff640 1 -- 172.21.15.89:0/2267962654 learned_addr learned my addr 172.21.15.89:0/2267962654 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:28.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b18101de0 0x7f1b180744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b1d744640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b18106f10 0x7f1b18074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b17fff640 1 -- 172.21.15.89:0/2267962654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b18101de0 msgr2=0x7f1b180744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:28.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b17fff640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b18101de0 0x7f1b180744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b17fff640 1 -- 172.21.15.89:0/2267962654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b18106f10 msgr2=0x7f1b18074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:28.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b17fff640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b18106f10 0x7f1b18074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b17fff640 1 -- 172.21.15.89:0/2267962654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b181a69d0 con 0x7f1b18102790 2024-09-15T23:37:28.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b18101de0 0x7f1b180744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:28.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b1d744640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b18106f10 0x7f1b18074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:28.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b17fff640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b18102790 0x7f1b18074a30 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f1b04006010 tx=0x7f1b040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:28.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b15ffb640 1 -- 172.21.15.89:0/2267962654 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0403f030 con 0x7f1b18102790 2024-09-15T23:37:28.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b15ffb640 1 -- 172.21.15.89:0/2267962654 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b04003ce0 con 0x7f1b18102790 2024-09-15T23:37:28.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2267962654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b181a6c60 con 0x7f1b18102790 2024-09-15T23:37:28.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b15ffb640 1 -- 172.21.15.89:0/2267962654 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04035070 con 0x7f1b18102790 2024-09-15T23:37:28.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.003+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2267962654 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b181a7140 con 0x7f1b18102790 2024-09-15T23:37:28.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.005+0000 7f1b15ffb640 1 -- 172.21.15.89:0/2267962654 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1b04042030 con 0x7f1b18102790 2024-09-15T23:37:28.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b21580740 1 Processor -- start 2024-09-15T23:37:28.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b21580740 1 -- start start 2024-09-15T23:37:28.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b21580740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x56077074b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b21580740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074b5d0 0x56077074ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b21580740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560770756370 0x560770758760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b21580740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5607706b12f0 con 0x56077074cb40 2024-09-15T23:37:28.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b21580740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560770658cf0 con 0x56077074b5d0 2024-09-15T23:37:28.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b21580740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5607706ad2b0 con 0x560770756370 2024-09-15T23:37:28.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b15ffb640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b080a3240 0x7f1b080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b1d744640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x56077074b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b1d744640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x56077074b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34204/0 (socket says 172.21.15.89:34204) 2024-09-15T23:37:28.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b1d744640 1 -- 172.21.15.89:0/2283665362 learned_addr learned my addr 172.21.15.89:0/2283665362 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:28.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b17fff640 1 --2- 172.21.15.89:0/2283665362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560770756370 0x560770758760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/2283665362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074b5d0 0x56077074ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b15ffb640 1 -- 172.21.15.89:0/2267962654 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1b040f02c0 con 0x7f1b18102790 2024-09-15T23:37:28.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.006+0000 7f1b1d744640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b080a3240 0x7f1b080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.007+0000 7f1b1d744640 1 -- 172.21.15.89:0/2283665362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560770756370 msgr2=0x560770758760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:28.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.007+0000 7f1b1d744640 1 --2- 172.21.15.89:0/2283665362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560770756370 0x560770758760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.007+0000 7f1b1d744640 1 -- 172.21.15.89:0/2283665362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074b5d0 msgr2=0x56077074ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:28.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.007+0000 7f1b1d744640 1 --2- 172.21.15.89:0/2283665362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074b5d0 0x56077074ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.007+0000 7f1b1d744640 1 -- 172.21.15.89:0/2283665362 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56077063d3a0 con 0x56077074cb40 2024-09-15T23:37:28.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.007+0000 7f1b1d744640 1 --2- 172.21.15.89:0/2283665362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x56077074b090 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f1b10013de0 tx=0x7f1b10010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea7a5fe666a07d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:28.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.007+0000 7f1b1d744640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b080a3240 0x7f1b080a5700 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1b10010c40 tx=0x7f1b10015000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:28.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.007+0000 7f1b02ffd640 1 -- 172.21.15.89:0/2283665362 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10010e50 con 0x56077074cb40 2024-09-15T23:37:28.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.007+0000 7f1b02ffd640 1 -- 172.21.15.89:0/2283665362 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b10013740 con 0x56077074cb40 2024-09-15T23:37:28.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.007+0000 7f1b02ffd640 1 -- 172.21.15.89:0/2283665362 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10005380 con 0x56077074cb40 2024-09-15T23:37:28.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 -- 172.21.15.89:0/2283665362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 msgr2=0x56077074b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:28.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 --2- 172.21.15.89:0/2283665362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x56077074b090 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f1b10013de0 tx=0x7f1b10010740 comp rx=0 tx=0).stop 2024-09-15T23:37:28.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 -- 172.21.15.89:0/2283665362 shutdown_connections 2024-09-15T23:37:28.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 --2- 172.21.15.89:0/2283665362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560770756370 0x560770758760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 --2- 172.21.15.89:0/2283665362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074b5d0 0x56077074ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 --2- 172.21.15.89:0/2283665362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x56077074b090 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 -- 172.21.15.89:0/2283665362 >> 172.21.15.89:0/2283665362 conn(0x560770714310 msgr2=0x560770747c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 -- 172.21.15.89:0/2283665362 shutdown_connections 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 -- 172.21.15.89:0/2283665362 wait complete. 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 -- start start 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.008+0000 7f1b21580740 1 Processor -- start 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b21580740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074b5d0 0x560770755da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b21580740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x560770753050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b21580740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560770756370 0x5607707537e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b21580740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560770759010 con 0x56077074cb40 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b21580740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56077066c060 con 0x560770756370 2024-09-15T23:37:28.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b21580740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560770759190 con 0x56077074b5d0 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b21580740 10 client.0 _ll_register_callbacks cb 0x5607703bf8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b17fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x560770753050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b17fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x560770753050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34218/0 (socket says 172.21.15.89:34218) 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b17fff640 1 -- 172.21.15.89:0/275815382 learned_addr learned my addr 172.21.15.89:0/275815382 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074b5d0 0x560770755da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b17fff640 1 -- 172.21.15.89:0/275815382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074b5d0 msgr2=0x560770755da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b17fff640 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074b5d0 0x560770755da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b17fff640 1 -- 172.21.15.89:0/275815382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560770756370 msgr2=0x5607707537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b1d744640 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560770756370 0x5607707537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b17fff640 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560770756370 0x5607707537e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b17fff640 1 -- 172.21.15.89:0/275815382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607704223c0 con 0x56077074cb40 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074b5d0 0x560770755da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:28.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b17fff640 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x560770753050 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f1b04000c00 tx=0x7f1b04042d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73f6163c72aca01 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1af7fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1af7fff640 1 -- 172.21.15.89:0/275815382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04049040 con 0x56077074cb40 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1af7fff640 1 -- 172.21.15.89:0/275815382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1b04049c40 con 0x56077074cb40 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.009+0000 7f1b21580740 1 -- 172.21.15.89:0/275815382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b10013740 con 0x56077074cb40 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1af7fff640 1 -- 172.21.15.89:0/275815382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1b040bc380 con 0x56077074cb40 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1b21580740 10 client.20073 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1b21580740 1 -- 172.21.15.89:0/275815382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560770764af0 con 0x56077074cb40 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1af7fff640 1 -- 172.21.15.89:0/275815382 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b10013740 con 0x56077074cb40 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1af7fff640 1 -- 172.21.15.89:0/275815382 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1b040b8c20 con 0x56077074cb40 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1b21580740 10 client.20073 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1b21580740 10 client.20073 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1b21580740 20 client.20073 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1b21580740 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x560770765e40 0x560770768290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1b21580740 4 client.20073 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1b21580740 1 -- 172.21.15.89:0/275815382 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607706be860 con 0x560770765e40 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.010+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x560770765e40 0x560770768290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x560770765e40 0x560770768290 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cf0348fa854eacbe server_cookie=ff030875c438abd3 in_seq=0 out_seq=0 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1af7fff640 10 client.20073 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1af7fff640 1 -- 172.21.15.89:0/275815382 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5607706be860 con 0x560770765e40 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1af7fff640 10 client.20073 handle_command_reply: tid=0 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1b21580740 1 client.20073 shutdown 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1b21580740 1 -- 172.21.15.89:0/275815382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 msgr2=0x560770753050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1b21580740 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x560770753050 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f1b04000c00 tx=0x7f1b04042d70 comp rx=0 tx=0).stop 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1b21580740 1 -- 172.21.15.89:0/275815382 shutdown_connections 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1b21580740 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x560770765e40 0x560770768290 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1b21580740 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560770756370 0x5607707537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1b21580740 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074cb40 0x560770753050 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1af7fff640 0 client.20073 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:37:28.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1b21580740 1 --2- 172.21.15.89:0/275815382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074b5d0 0x560770755da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.011+0000 7f1b21580740 1 -- 172.21.15.89:0/275815382 >> 172.21.15.89:0/275815382 conn(0x560770714310 msgr2=0x56077074d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:28.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.012+0000 7f1b21580740 1 -- 172.21.15.89:0/275815382 shutdown_connections 2024-09-15T23:37:28.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.012+0000 7f1b21580740 1 -- 172.21.15.89:0/275815382 wait complete. 2024-09-15T23:37:28.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.012+0000 7f1b21580740 20 client.20073 trim_cache size 0 max 16384 2024-09-15T23:37:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b21580740 1 Processor -- start 2024-09-15T23:37:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b21580740 1 -- start start 2024-09-15T23:37:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b21580740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074b5d0 0x560770747e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b21580740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074cb40 0x560770752910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b21580740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560770756370 0x56077076dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b21580740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1b181b1030 con 0x560770756370 2024-09-15T23:37:28.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b21580740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1b181b2b70 con 0x56077074b5d0 2024-09-15T23:37:28.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b21580740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b181b1700 con 0x56077074cb40 2024-09-15T23:37:28.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b1d744640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560770756370 0x56077076dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b1d744640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560770756370 0x56077076dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34220/0 (socket says 172.21.15.89:34220) 2024-09-15T23:37:28.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b1d744640 1 -- 172.21.15.89:0/2229920556 learned_addr learned my addr 172.21.15.89:0/2229920556 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:28.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b17fff640 1 --2- 172.21.15.89:0/2229920556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074cb40 0x560770752910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.025+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/2229920556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074b5d0 0x560770747e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.026+0000 7f1b1d744640 1 -- 172.21.15.89:0/2229920556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074cb40 msgr2=0x560770752910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:28.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.026+0000 7f1b1d744640 1 --2- 172.21.15.89:0/2229920556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074cb40 0x560770752910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.026+0000 7f1b1d744640 1 -- 172.21.15.89:0/2229920556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074b5d0 msgr2=0x560770747e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.026+0000 7f1b1d744640 1 --2- 172.21.15.89:0/2229920556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074b5d0 0x560770747e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.026+0000 7f1b1d744640 1 -- 172.21.15.89:0/2229920556 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560770768910 con 0x560770756370 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.026+0000 7f1b1d744640 1 --2- 172.21.15.89:0/2229920556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560770756370 0x56077076dba0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009790 tx=0x7f1b10013550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d7174f951e2b6d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.026+0000 7f1b027fc640 1 -- 172.21.15.89:0/2229920556 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1002d020 con 0x560770756370 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.026+0000 7f1b027fc640 1 -- 172.21.15.89:0/2229920556 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b10013740 con 0x560770756370 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.026+0000 7f1b027fc640 1 -- 172.21.15.89:0/2229920556 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b10003d50 con 0x560770756370 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 -- 172.21.15.89:0/2229920556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560770756370 msgr2=0x56077076dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 --2- 172.21.15.89:0/2229920556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560770756370 0x56077076dba0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009790 tx=0x7f1b10013550 comp rx=0 tx=0).stop 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 -- 172.21.15.89:0/2229920556 shutdown_connections 2024-09-15T23:37:28.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 --2- 172.21.15.89:0/2229920556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560770756370 0x56077076dba0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 --2- 172.21.15.89:0/2229920556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56077074cb40 0x560770752910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 --2- 172.21.15.89:0/2229920556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074b5d0 0x560770747e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 -- 172.21.15.89:0/2229920556 >> 172.21.15.89:0/2229920556 conn(0x560770714310 msgr2=0x560770747700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 -- 172.21.15.89:0/2229920556 shutdown_connections 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 -- 172.21.15.89:0/2229920556 wait complete. 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 -- start start 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.027+0000 7f1b21580740 1 Processor -- start 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b21580740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074b5d0 0x56077075ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b21580740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074cb40 0x56077075a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b21580740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560770756370 0x56077075a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b21580740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1b181b24a0 con 0x56077074b5d0 2024-09-15T23:37:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b21580740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1b181b1dd0 con 0x56077074cb40 2024-09-15T23:37:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b21580740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b181af010 con 0x560770756370 2024-09-15T23:37:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b21580740 10 client.0 _ll_register_callbacks cb 0x7f1b04005750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b1cf43640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074b5d0 0x56077075ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b1cf43640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074b5d0 0x56077075ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34224/0 (socket says 172.21.15.89:34224) 2024-09-15T23:37:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b1cf43640 1 -- 172.21.15.89:0/3976340428 learned_addr learned my addr 172.21.15.89:0/3976340428 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b1d744640 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560770756370 0x56077075a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b17fff640 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074cb40 0x56077075a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b1cf43640 1 -- 172.21.15.89:0/3976340428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560770756370 msgr2=0x56077075a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:28.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560770756370 0x56077075a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b1cf43640 1 -- 172.21.15.89:0/3976340428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074cb40 msgr2=0x56077075a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:28.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074cb40 0x56077075a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b1cf43640 1 -- 172.21.15.89:0/3976340428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560770747510 con 0x56077074b5d0 2024-09-15T23:37:28.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b1cf43640 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074b5d0 0x56077075ed30 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c005c60 tx=0x7f1b0c009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e655791851d222a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:28.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1af7fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:28.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1af7fff640 1 -- 172.21.15.89:0/3976340428 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c005230 con 0x56077074b5d0 2024-09-15T23:37:28.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1af7fff640 1 -- 172.21.15.89:0/3976340428 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1b0c0053d0 con 0x56077074b5d0 2024-09-15T23:37:28.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.028+0000 7f1b21580740 1 -- 172.21.15.89:0/3976340428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b10013740 con 0x56077074b5d0 2024-09-15T23:37:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1af7fff640 1 -- 172.21.15.89:0/3976340428 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1b0c02d050 con 0x56077074b5d0 2024-09-15T23:37:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1b21580740 10 client.20079 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1b21580740 1 -- 172.21.15.89:0/3976340428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5607707d0bc0 con 0x56077074b5d0 2024-09-15T23:37:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1af7fff640 1 -- 172.21.15.89:0/3976340428 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b10013740 con 0x56077074b5d0 2024-09-15T23:37:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1af7fff640 1 -- 172.21.15.89:0/3976340428 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1b0c022020 con 0x56077074b5d0 2024-09-15T23:37:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1b21580740 10 client.20079 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1b21580740 10 client.20079 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:37:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1b21580740 20 client.20079 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1b21580740 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5607707d1340 0x5607707d3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:28.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1b21580740 4 client.20079 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:37:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1b21580740 1 -- 172.21.15.89:0/3976340428 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5607706b8d70 con 0x5607707d1340 2024-09-15T23:37:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.029+0000 7f1b17fff640 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5607707d1340 0x5607707d3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1af7fff640 10 client.20079 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:37:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b17fff640 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5607707d1340 0x5607707d3760 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=836ea2119aec1feb server_cookie=d43ddfb983620251 in_seq=0 out_seq=0 2024-09-15T23:37:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1af7fff640 1 -- 172.21.15.89:0/3976340428 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5607706b8d70 con 0x5607707d1340 2024-09-15T23:37:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1af7fff640 10 client.20079 handle_command_reply: tid=0 2024-09-15T23:37:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 client.20079 shutdown 2024-09-15T23:37:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 -- 172.21.15.89:0/3976340428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074b5d0 msgr2=0x56077075ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074b5d0 0x56077075ed30 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c005c60 tx=0x7f1b0c009be0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 -- 172.21.15.89:0/3976340428 shutdown_connections 2024-09-15T23:37:28.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5607707d1340 0x5607707d3760 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560770756370 0x56077075a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56077074cb40 0x56077075a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 --2- 172.21.15.89:0/3976340428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56077074b5d0 0x56077075ed30 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 -- 172.21.15.89:0/3976340428 >> 172.21.15.89:0/3976340428 conn(0x560770714310 msgr2=0x5607707548b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:28.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1af7fff640 0 client.20079 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:37:28.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 -- 172.21.15.89:0/3976340428 shutdown_connections 2024-09-15T23:37:28.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 1 -- 172.21.15.89:0/3976340428 wait complete. 2024-09-15T23:37:28.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.030+0000 7f1b21580740 20 client.20079 trim_cache size 0 max 16384 2024-09-15T23:37:28.051 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:37:28.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2267962654 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b080a3240 msgr2=0x7f1b080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:28.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b080a3240 0x7f1b080a5700 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1b10010c40 tx=0x7f1b10015000 comp rx=0 tx=0).stop 2024-09-15T23:37:28.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2267962654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b18102790 msgr2=0x7f1b18074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:28.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b18102790 0x7f1b18074a30 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f1b04006010 tx=0x7f1b040042f0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2267962654 shutdown_connections 2024-09-15T23:37:28.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1b080a3240 0x7f1b080a5700 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b18106f10 0x7f1b18074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b18102790 0x7f1b18074a30 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 --2- 172.21.15.89:0/2267962654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b18101de0 0x7f1b180744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:28.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2267962654 >> 172.21.15.89:0/2267962654 conn(0x7f1b180fd8c0 msgr2=0x7f1b180fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:28.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2267962654 shutdown_connections 2024-09-15T23:37:28.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:28.031+0000 7f1b1f1a7640 1 -- 172.21.15.89:0/2267962654 wait complete. 2024-09-15T23:37:28.808 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:37:29.056 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:29.195 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:28 smithi089 ceph-mon[30933]: pgmap v753: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 KiB/s wr, 21 op/s 2024-09-15T23:37:29.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:28 smithi120 ceph-mon[35250]: pgmap v753: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 KiB/s wr, 21 op/s 2024-09-15T23:37:29.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:28 smithi174 ceph-mon[35586]: pgmap v753: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 KiB/s wr, 21 op/s 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.537+0000 7f583f467640 1 -- 172.21.15.89:0/4003351189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58381027c0 msgr2=0x7f5838106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.537+0000 7f583f467640 1 --2- 172.21.15.89:0/4003351189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58381027c0 0x7f5838106810 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f5828009900 tx=0x7f582802a410 comp rx=0 tx=0).stop 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.537+0000 7f583f467640 1 -- 172.21.15.89:0/4003351189 shutdown_connections 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.537+0000 7f583f467640 1 --2- 172.21.15.89:0/4003351189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5838106f40 0x7f583810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.537+0000 7f583f467640 1 --2- 172.21.15.89:0/4003351189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58381027c0 0x7f5838106810 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.537+0000 7f583f467640 1 --2- 172.21.15.89:0/4003351189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5838101e10 0x7f58381021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.537+0000 7f583f467640 1 -- 172.21.15.89:0/4003351189 >> 172.21.15.89:0/4003351189 conn(0x7f58380fd8d0 msgr2=0x7f58380ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.537+0000 7f583f467640 1 -- 172.21.15.89:0/4003351189 shutdown_connections 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.537+0000 7f583f467640 1 -- 172.21.15.89:0/4003351189 wait complete. 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583f467640 1 Processor -- start 2024-09-15T23:37:29.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583f467640 1 -- start start 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583f467640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5838101e10 0x7f5838198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583f467640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58381027c0 0x7f58381989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583f467640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5838106f40 0x7f583819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583f467640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5838111f60 con 0x7f5838106f40 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583f467640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5838111de0 con 0x7f58381027c0 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583f467640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58381120e0 con 0x7f5838101e10 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583da04640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5838106f40 0x7f583819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583da04640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5838106f40 0x7f583819cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34242/0 (socket says 172.21.15.89:34242) 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583da04640 1 -- 172.21.15.89:0/1482052616 learned_addr learned my addr 172.21.15.89:0/1482052616 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.538+0000 7f583ca02640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58381027c0 0x7f58381989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f583d203640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5838101e10 0x7f5838198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f583da04640 1 -- 172.21.15.89:0/1482052616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5838101e10 msgr2=0x7f5838198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f583da04640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5838101e10 0x7f5838198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f583da04640 1 -- 172.21.15.89:0/1482052616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58381027c0 msgr2=0x7f58381989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f583da04640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58381027c0 0x7f58381989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f583da04640 1 -- 172.21.15.89:0/1482052616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583819d430 con 0x7f5838106f40 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f583ca02640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58381027c0 0x7f58381989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f583da04640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5838106f40 0x7f583819cd50 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f5830009fd0 tx=0x7f583000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f582e7fc640 1 -- 172.21.15.89:0/1482052616 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5830017070 con 0x7f5838106f40 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f583f467640 1 -- 172.21.15.89:0/1482052616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f583819d720 con 0x7f5838106f40 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f582e7fc640 1 -- 172.21.15.89:0/1482052616 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5830003e50 con 0x7f5838106f40 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f582e7fc640 1 -- 172.21.15.89:0/1482052616 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58300106c0 con 0x7f5838106f40 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.539+0000 7f583f467640 1 -- 172.21.15.89:0/1482052616 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58381a5000 con 0x7f5838106f40 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.541+0000 7f582e7fc640 1 -- 172.21.15.89:0/1482052616 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5830003970 con 0x7f5838106f40 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f5841840740 1 Processor -- start 2024-09-15T23:37:29.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f5841840740 1 -- start start 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f5841840740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b8a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f5841840740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b8aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f5841840740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b95370 0x561f87b97760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f5841840740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561f87af02f0 con 0x561f87b8bb40 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f5841840740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561f87a97cf0 con 0x561f87b8a5d0 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f5841840740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561f87aec2b0 con 0x561f87b95370 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f582e7fc640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58240a3240 0x7f58240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f583ca02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b8a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f583da04640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58240a3240 0x7f58240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f583ca02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b8a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34250/0 (socket says 172.21.15.89:34250) 2024-09-15T23:37:29.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f583ca02640 1 -- 172.21.15.89:0/4141034102 learned_addr learned my addr 172.21.15.89:0/4141034102 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f583d203640 1 --2- 172.21.15.89:0/4141034102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b8aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.542+0000 7f582e7fc640 1 -- 172.21.15.89:0/1482052616 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f58300cfb20 con 0x7f5838106f40 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f583ca02640 1 --2- 172.21.15.89:0/4141034102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b95370 0x561f87b97760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f583ca02640 1 -- 172.21.15.89:0/4141034102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b95370 msgr2=0x561f87b97760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f583ca02640 1 --2- 172.21.15.89:0/4141034102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b95370 0x561f87b97760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f583ca02640 1 -- 172.21.15.89:0/4141034102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 msgr2=0x561f87b8aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f583ca02640 1 --2- 172.21.15.89:0/4141034102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b8aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f583ca02640 1 -- 172.21.15.89:0/4141034102 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f87a7c3a0 con 0x561f87b8bb40 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f583da04640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58240a3240 0x7f58240a5700 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f58300a4490 tx=0x7f58300c6b20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f583ca02640 1 --2- 172.21.15.89:0/4141034102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b8a090 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f5828031550 tx=0x7f5828005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a588f37cee318ce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f58237fe640 1 -- 172.21.15.89:0/4141034102 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828035050 con 0x561f87b8bb40 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f58237fe640 1 -- 172.21.15.89:0/4141034102 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58280317b0 con 0x561f87b8bb40 2024-09-15T23:37:29.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.543+0000 7f58237fe640 1 -- 172.21.15.89:0/4141034102 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58280046c0 con 0x561f87b8bb40 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 -- 172.21.15.89:0/4141034102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 msgr2=0x561f87b8a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 --2- 172.21.15.89:0/4141034102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b8a090 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f5828031550 tx=0x7f5828005bd0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 -- 172.21.15.89:0/4141034102 shutdown_connections 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 --2- 172.21.15.89:0/4141034102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b95370 0x561f87b97760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 --2- 172.21.15.89:0/4141034102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b8aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 --2- 172.21.15.89:0/4141034102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b8a090 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 -- 172.21.15.89:0/4141034102 >> 172.21.15.89:0/4141034102 conn(0x561f87b53310 msgr2=0x561f87b86c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 -- 172.21.15.89:0/4141034102 shutdown_connections 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 -- 172.21.15.89:0/4141034102 wait complete. 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 -- start start 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.544+0000 7f5841840740 1 Processor -- start 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5841840740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b94d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5841840740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b8bb40 0x561f87b91fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5841840740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87b92530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5841840740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561f87b98010 con 0x561f87b95370 2024-09-15T23:37:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5841840740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561f87aab060 con 0x561f87b8a5d0 2024-09-15T23:37:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5841840740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561f87b98190 con 0x561f87b8bb40 2024-09-15T23:37:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5841840740 10 client.0 _ll_register_callbacks cb 0x561f877fe8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f583ca02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87b92530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f583ca02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87b92530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34264/0 (socket says 172.21.15.89:34264) 2024-09-15T23:37:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f583ca02640 1 -- 172.21.15.89:0/1755541072 learned_addr learned my addr 172.21.15.89:0/1755541072 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f583d203640 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b8bb40 0x561f87b91fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.581 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f583ca02640 1 -- 172.21.15.89:0/1755541072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b8bb40 msgr2=0x561f87b91fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f583ca02640 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b8bb40 0x561f87b91fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f583ca02640 1 -- 172.21.15.89:0/1755541072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 msgr2=0x561f87b94d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f583ca02640 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b94d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f583ca02640 1 -- 172.21.15.89:0/1755541072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f878613c0 con 0x561f87b95370 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f583ca02640 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87b92530 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f582802aba0 tx=0x7f58280315b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb980e47a96fe53 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5820ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5820ff9640 1 -- 172.21.15.89:0/1755541072 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828035040 con 0x561f87b95370 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5820ff9640 1 -- 172.21.15.89:0/1755541072 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f58280046c0 con 0x561f87b95370 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5841840740 1 -- 172.21.15.89:0/1755541072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58280317b0 con 0x561f87b95370 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5820ff9640 1 -- 172.21.15.89:0/1755541072 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f58280048c0 con 0x561f87b95370 2024-09-15T23:37:29.582 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5841840740 10 client.20091 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:29.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.545+0000 7f5841840740 1 -- 172.21.15.89:0/1755541072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561f87ba3a40 con 0x561f87b95370 2024-09-15T23:37:29.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.546+0000 7f5820ff9640 1 -- 172.21.15.89:0/1755541072 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58280317b0 con 0x561f87b95370 2024-09-15T23:37:29.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.546+0000 7f5820ff9640 1 -- 172.21.15.89:0/1755541072 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5828003c40 con 0x561f87b95370 2024-09-15T23:37:29.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.546+0000 7f5841840740 10 client.20091 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:29.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.546+0000 7f5841840740 10 client.20091 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:37:29.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.546+0000 7f5841840740 20 client.20091 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:29.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.546+0000 7f5841840740 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561f87ba4d90 0x561f87ba7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.546+0000 7f5841840740 4 client.20091 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:29.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.546+0000 7f5841840740 1 -- 172.21.15.89:0/1755541072 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f87afd860 con 0x561f87ba4d90 2024-09-15T23:37:29.583 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.546+0000 7f583d203640 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561f87ba4d90 0x561f87ba7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f583d203640 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561f87ba4d90 0x561f87ba7180 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c34a90331b5b208d server_cookie=afb93a4814607240 in_seq=0 out_seq=0 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5820ff9640 10 client.20091 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5820ff9640 1 -- 172.21.15.89:0/1755541072 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561f87afd860 con 0x561f87ba4d90 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5820ff9640 10 client.20091 handle_command_reply: tid=0 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5841840740 1 client.20091 shutdown 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5841840740 1 -- 172.21.15.89:0/1755541072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 msgr2=0x561f87b92530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5841840740 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87b92530 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f582802aba0 tx=0x7f58280315b0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5841840740 1 -- 172.21.15.89:0/1755541072 shutdown_connections 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5841840740 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87b92530 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5841840740 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561f87ba4d90 0x561f87ba7180 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5841840740 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b8bb40 0x561f87b91fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5841840740 1 --2- 172.21.15.89:0/1755541072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b94d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5820ff9640 0 client.20091 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:37:29.584 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.547+0000 7f5841840740 1 -- 172.21.15.89:0/1755541072 >> 172.21.15.89:0/1755541072 conn(0x561f87b53310 msgr2=0x561f87b8c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.548+0000 7f5841840740 1 -- 172.21.15.89:0/1755541072 shutdown_connections 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.548+0000 7f5841840740 1 -- 172.21.15.89:0/1755541072 wait complete. 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.548+0000 7f5841840740 20 client.20091 trim_cache size 0 max 16384 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.560+0000 7f5841840740 1 Processor -- start 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.560+0000 7f5841840740 1 -- start start 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f5841840740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b8a5d0 0x561f87ba12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f5841840740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8bb40 0x561f87b93f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f5841840740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87bac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f5841840740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f58381aed90 con 0x561f87b95370 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f5841840740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58381b08d0 con 0x561f87b8bb40 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f5841840740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58381af460 con 0x561f87b8a5d0 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583ca02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87bac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.585 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583ca02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87bac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34276/0 (socket says 172.21.15.89:34276) 2024-09-15T23:37:29.586 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:37:29.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583ca02640 1 -- 172.21.15.89:0/1673149185 learned_addr learned my addr 172.21.15.89:0/1673149185 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:29.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583ca02640 1 --2- 172.21.15.89:0/1673149185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b8a5d0 0x561f87ba12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.586 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583d203640 1 --2- 172.21.15.89:0/1673149185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8bb40 0x561f87b93f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583ca02640 1 -- 172.21.15.89:0/1673149185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b8a5d0 msgr2=0x561f87ba12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583ca02640 1 --2- 172.21.15.89:0/1673149185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b8a5d0 0x561f87ba12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583ca02640 1 -- 172.21.15.89:0/1673149185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8bb40 msgr2=0x561f87b93f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583ca02640 1 --2- 172.21.15.89:0/1673149185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8bb40 0x561f87b93f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583ca02640 1 -- 172.21.15.89:0/1673149185 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f87b90500 con 0x561f87b95370 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583d203640 1 --2- 172.21.15.89:0/1673149185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8bb40 0x561f87b93f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f583ca02640 1 --2- 172.21.15.89:0/1673149185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87bac750 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f5828004650 tx=0x7f5828000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1c1cc8d8def067 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f5822ffd640 1 -- 172.21.15.89:0/1673149185 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828035030 con 0x561f87b95370 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f5822ffd640 1 -- 172.21.15.89:0/1673149185 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58280054c0 con 0x561f87b95370 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.561+0000 7f5822ffd640 1 -- 172.21.15.89:0/1673149185 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828031b60 con 0x561f87b95370 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 -- 172.21.15.89:0/1673149185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 msgr2=0x561f87bac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 --2- 172.21.15.89:0/1673149185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87bac750 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f5828004650 tx=0x7f5828000bc0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.587 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 -- 172.21.15.89:0/1673149185 shutdown_connections 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 --2- 172.21.15.89:0/1673149185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b95370 0x561f87bac750 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 --2- 172.21.15.89:0/1673149185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8bb40 0x561f87b93f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 --2- 172.21.15.89:0/1673149185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b8a5d0 0x561f87ba12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 -- 172.21.15.89:0/1673149185 >> 172.21.15.89:0/1673149185 conn(0x561f87b53310 msgr2=0x561f87b93810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 -- 172.21.15.89:0/1673149185 shutdown_connections 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 -- 172.21.15.89:0/1673149185 wait complete. 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 -- start start 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.562+0000 7f5841840740 1 Processor -- start 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5841840740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b9dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5841840740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b98de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5841840740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b95370 0x561f87b99320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5841840740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f58381b0200 con 0x561f87b8bb40 2024-09-15T23:37:29.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5841840740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58381afb30 con 0x561f87b8a5d0 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5841840740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58381acd70 con 0x561f87b95370 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5841840740 10 client.0 _ll_register_callbacks cb 0x7f58340092a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583d203640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b98de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583d203640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b98de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34292/0 (socket says 172.21.15.89:34292) 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583d203640 1 -- 172.21.15.89:0/1076043100 learned_addr learned my addr 172.21.15.89:0/1076043100 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583ca02640 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b95370 0x561f87b99320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583ca02640 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b9dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583d203640 1 -- 172.21.15.89:0/1076043100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b95370 msgr2=0x561f87b99320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583d203640 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b95370 0x561f87b99320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583d203640 1 -- 172.21.15.89:0/1076043100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 msgr2=0x561f87b9dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:29.589 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583d203640 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b9dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583d203640 1 -- 172.21.15.89:0/1076043100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f87b99b00 con 0x561f87b8bb40 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f583d203640 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b98de0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x561f87b935f0 tx=0x7f5834012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd232091a5d4ef67 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5820ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5820ff9640 1 -- 172.21.15.89:0/1076043100 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583401b4c0 con 0x561f87b8bb40 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5820ff9640 1 -- 172.21.15.89:0/1076043100 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f583401b660 con 0x561f87b8bb40 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.563+0000 7f5841840740 1 -- 172.21.15.89:0/1076043100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58280054c0 con 0x561f87b8bb40 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5820ff9640 1 -- 172.21.15.89:0/1076043100 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f583402c050 con 0x561f87b8bb40 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5841840740 10 client.20097 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5841840740 1 -- 172.21.15.89:0/1076043100 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561f87c0f670 con 0x561f87b8bb40 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5820ff9640 1 -- 172.21.15.89:0/1076043100 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58280054c0 con 0x561f87b8bb40 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5820ff9640 1 -- 172.21.15.89:0/1076043100 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f583400f040 con 0x561f87b8bb40 2024-09-15T23:37:29.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5841840740 10 client.20097 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5841840740 10 client.20097 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5841840740 20 client.20097 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5841840740 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561f87c0fe20 0x561f87c12240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5841840740 4 client.20097 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f5841840740 1 -- 172.21.15.89:0/1076043100 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561f87af7d70 con 0x561f87c0fe20 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.564+0000 7f583ca02640 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561f87c0fe20 0x561f87c12240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5820ff9640 10 client.20097 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f583ca02640 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561f87c0fe20 0x561f87c12240 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54d017cb805560eb server_cookie=1ae69665c59e08bb in_seq=0 out_seq=0 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5820ff9640 1 -- 172.21.15.89:0/1076043100 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561f87af7d70 con 0x561f87c0fe20 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5820ff9640 10 client.20097 handle_command_reply: tid=0 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 client.20097 shutdown 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 -- 172.21.15.89:0/1076043100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 msgr2=0x561f87b98de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b98de0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x561f87b935f0 tx=0x7f5834012820 comp rx=0 tx=0).stop 2024-09-15T23:37:29.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 -- 172.21.15.89:0/1076043100 shutdown_connections 2024-09-15T23:37:29.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561f87c0fe20 0x561f87c12240 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f87b95370 0x561f87b99320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f87b8bb40 0x561f87b98de0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5820ff9640 0 client.20097 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:37:29.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 --2- 172.21.15.89:0/1076043100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f87b8a5d0 0x561f87b9dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 -- 172.21.15.89:0/1076043100 >> 172.21.15.89:0/1076043100 conn(0x561f87b53310 msgr2=0x561f87b85d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:29.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 -- 172.21.15.89:0/1076043100 shutdown_connections 2024-09-15T23:37:29.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 1 -- 172.21.15.89:0/1076043100 wait complete. 2024-09-15T23:37:29.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.565+0000 7f5841840740 20 client.20097 trim_cache size 0 max 16384 2024-09-15T23:37:29.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 -- 172.21.15.89:0/1482052616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58240a3240 msgr2=0x7f58240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58240a3240 0x7f58240a5700 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f58300a4490 tx=0x7f58300c6b20 comp rx=0 tx=0).stop 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 -- 172.21.15.89:0/1482052616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5838106f40 msgr2=0x7f583819cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5838106f40 0x7f583819cd50 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f5830009fd0 tx=0x7f583000b260 comp rx=0 tx=0).stop 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 -- 172.21.15.89:0/1482052616 shutdown_connections 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f58240a3240 0x7f58240a5700 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5838106f40 0x7f583819cd50 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58381027c0 0x7f58381989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 --2- 172.21.15.89:0/1482052616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5838101e10 0x7f5838198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 -- 172.21.15.89:0/1482052616 >> 172.21.15.89:0/1482052616 conn(0x7f58380fd8d0 msgr2=0x7f58380fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 -- 172.21.15.89:0/1482052616 shutdown_connections 2024-09-15T23:37:29.593 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:29.566+0000 7f583f467640 1 -- 172.21.15.89:0/1482052616 wait complete. 2024-09-15T23:37:30.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:37:30.528 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.022+0000 7f215f429640 1 -- 172.21.15.89:0/721420532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2158105f40 msgr2=0x7f21581063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.022+0000 7f215f429640 1 --2- 172.21.15.89:0/721420532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2158105f40 0x7f21581063d0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f2150009ca0 tx=0x7f215002a940 comp rx=0 tx=0).stop 2024-09-15T23:37:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.022+0000 7f215f429640 1 -- 172.21.15.89:0/721420532 shutdown_connections 2024-09-15T23:37:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.022+0000 7f215f429640 1 --2- 172.21.15.89:0/721420532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2158105f40 0x7f21581063d0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.022+0000 7f215f429640 1 --2- 172.21.15.89:0/721420532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158101450 0x7f2158105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.022+0000 7f215f429640 1 --2- 172.21.15.89:0/721420532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158100b30 0x7f2158100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.022+0000 7f215f429640 1 -- 172.21.15.89:0/721420532 >> 172.21.15.89:0/721420532 conn(0x7f21580fc7a0 msgr2=0x7f21580febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.022+0000 7f215f429640 1 -- 172.21.15.89:0/721420532 shutdown_connections 2024-09-15T23:37:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.022+0000 7f215f429640 1 -- 172.21.15.89:0/721420532 wait complete. 2024-09-15T23:37:31.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215f429640 1 Processor -- start 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215f429640 1 -- start start 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215f429640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158100b30 0x7f21581a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215f429640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158101450 0x7f21581a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215f429640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2158105f40 0x7f215819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215f429640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2158118580 con 0x7f2158105f40 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215f429640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2158118400 con 0x7f2158101450 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215f429640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2158118700 con 0x7f2158100b30 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215d9c6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2158105f40 0x7f215819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215d9c6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2158105f40 0x7f215819e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34308/0 (socket says 172.21.15.89:34308) 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215d9c6640 1 -- 172.21.15.89:0/2021851063 learned_addr learned my addr 172.21.15.89:0/2021851063 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.023+0000 7f215d1c5640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158100b30 0x7f21581a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f215c9c4640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158101450 0x7f21581a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f215d9c6640 1 -- 172.21.15.89:0/2021851063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158100b30 msgr2=0x7f21581a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f215d9c6640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158100b30 0x7f21581a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f215d9c6640 1 -- 172.21.15.89:0/2021851063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158101450 msgr2=0x7f21581a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f215d9c6640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158101450 0x7f21581a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f215d9c6640 1 -- 172.21.15.89:0/2021851063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215819eff0 con 0x7f2158105f40 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f215c9c4640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158101450 0x7f21581a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f215d9c6640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2158105f40 0x7f215819e850 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f2150006010 tx=0x7f215002f700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f214e7fc640 1 -- 172.21.15.89:0/2021851063 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150034040 con 0x7f2158105f40 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f215f429640 1 -- 172.21.15.89:0/2021851063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215819f280 con 0x7f2158105f40 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f214e7fc640 1 -- 172.21.15.89:0/2021851063 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2150003e50 con 0x7f2158105f40 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f214e7fc640 1 -- 172.21.15.89:0/2021851063 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150004800 con 0x7f2158105f40 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.024+0000 7f215f429640 1 -- 172.21.15.89:0/2021851063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21581ab530 con 0x7f2158105f40 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f214e7fc640 1 -- 172.21.15.89:0/2021851063 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2150037020 con 0x7f2158105f40 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f2161802740 1 Processor -- start 2024-09-15T23:37:31.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f2161802740 1 -- start start 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f2161802740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b88879090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f2161802740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b888795d0 0x559b88879a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f2161802740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b88886760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f2161802740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559b887df2f0 con 0x559b88884370 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f2161802740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559b88786cf0 con 0x559b8887ab40 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f2161802740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559b887db2b0 con 0x559b888795d0 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f215c9c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b88886760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f215c9c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b88886760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34314/0 (socket says 172.21.15.89:34314) 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f215c9c4640 1 -- 172.21.15.89:0/2056134769 learned_addr learned my addr 172.21.15.89:0/2056134769 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f214e7fc640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 0x7f21440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f215d1c5640 1 --2- 172.21.15.89:0/2056134769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b888795d0 0x559b88879a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f215c9c4640 1 --2- 172.21.15.89:0/2056134769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b88879090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f215d9c6640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 0x7f21440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f215c9c4640 1 -- 172.21.15.89:0/2056134769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b888795d0 msgr2=0x559b88879a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f215c9c4640 1 --2- 172.21.15.89:0/2056134769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b888795d0 0x559b88879a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f215c9c4640 1 -- 172.21.15.89:0/2056134769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 msgr2=0x559b88879090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.027+0000 7f215c9c4640 1 --2- 172.21.15.89:0/2056134769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b88879090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.028+0000 7f215c9c4640 1 -- 172.21.15.89:0/2056134769 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b8876b3a0 con 0x559b88884370 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.028+0000 7f214e7fc640 1 -- 172.21.15.89:0/2021851063 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f21500f0cc0 con 0x7f2158105f40 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.028+0000 7f215c9c4640 1 --2- 172.21.15.89:0/2056134769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b88886760 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f2148014920 tx=0x7f214800b500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9abcd42bdb2a42fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.028+0000 7f21437fe640 1 -- 172.21.15.89:0/2056134769 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214800f760 con 0x559b88884370 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.028+0000 7f21437fe640 1 -- 172.21.15.89:0/2056134769 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f214800fde0 con 0x559b88884370 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.028+0000 7f215d9c6640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 0x7f21440a5700 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2150032000 tx=0x7f2150048e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.028+0000 7f21437fe640 1 -- 172.21.15.89:0/2056134769 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148012ba0 con 0x559b88884370 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 -- 172.21.15.89:0/2056134769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 msgr2=0x559b88886760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:31.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 --2- 172.21.15.89:0/2056134769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b88886760 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f2148014920 tx=0x7f214800b500 comp rx=0 tx=0).stop 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 -- 172.21.15.89:0/2056134769 shutdown_connections 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 --2- 172.21.15.89:0/2056134769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b88886760 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 --2- 172.21.15.89:0/2056134769 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b888795d0 0x559b88879a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 --2- 172.21.15.89:0/2056134769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b88879090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 -- 172.21.15.89:0/2056134769 >> 172.21.15.89:0/2056134769 conn(0x559b88842310 msgr2=0x559b88875c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 -- 172.21.15.89:0/2056134769 shutdown_connections 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 -- 172.21.15.89:0/2056134769 wait complete. 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 -- start start 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.029+0000 7f2161802740 1 Processor -- start 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2161802740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b888795d0 0x559b88883db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2161802740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b88881060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2161802740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b888817f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2161802740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559b88887070 con 0x559b88884370 2024-09-15T23:37:31.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2161802740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559b8879a060 con 0x559b8887ab40 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2161802740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559b888871f0 con 0x559b888795d0 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2161802740 10 client.0 _ll_register_callbacks cb 0x559b884ed8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215c9c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b888817f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215c9c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b888817f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34328/0 (socket says 172.21.15.89:34328) 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215c9c4640 1 -- 172.21.15.89:0/2780984194 learned_addr learned my addr 172.21.15.89:0/2780984194 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215c9c4640 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b888795d0 0x559b88883db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215c9c4640 1 -- 172.21.15.89:0/2780984194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b888795d0 msgr2=0x559b88883db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215d1c5640 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b88881060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215c9c4640 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b888795d0 0x559b88883db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215c9c4640 1 -- 172.21.15.89:0/2780984194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 msgr2=0x559b88881060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215c9c4640 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b88881060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215c9c4640 1 -- 172.21.15.89:0/2780984194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b885503c0 con 0x559b88884370 2024-09-15T23:37:31.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f215c9c4640 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b888817f0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f2148001eb0 tx=0x7f2148012780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd34cfd637a1a6c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2140ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2140ff9640 1 -- 172.21.15.89:0/2780984194 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148012b20 con 0x559b88884370 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2140ff9640 1 -- 172.21.15.89:0/2780984194 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2148012cc0 con 0x559b88884370 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.030+0000 7f2161802740 1 -- 172.21.15.89:0/2780984194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f214800fde0 con 0x559b88884370 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2140ff9640 1 -- 172.21.15.89:0/2780984194 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f214800fc30 con 0x559b88884370 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2161802740 10 client.20109 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2161802740 1 -- 172.21.15.89:0/2780984194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559b88892b60 con 0x559b88884370 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2140ff9640 1 -- 172.21.15.89:0/2780984194 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f214800fde0 con 0x559b88884370 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2140ff9640 1 -- 172.21.15.89:0/2780984194 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f214801e070 con 0x559b88884370 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2161802740 10 client.20109 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2161802740 10 client.20109 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2161802740 20 client.20109 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2161802740 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559b88893eb0 0x559b888962a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2161802740 4 client.20109 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f2161802740 1 -- 172.21.15.89:0/2780984194 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b887ec860 con 0x559b88893eb0 2024-09-15T23:37:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.031+0000 7f215d1c5640 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559b88893eb0 0x559b888962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.032+0000 7f2140ff9640 10 client.20109 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:37:31.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.032+0000 7f215d1c5640 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559b88893eb0 0x559b888962a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e603ecba7f499b7c server_cookie=49fd3db1f9c73df2 in_seq=0 out_seq=0 2024-09-15T23:37:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2140ff9640 1 -- 172.21.15.89:0/2780984194 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559b887ec860 con 0x559b88893eb0 2024-09-15T23:37:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2140ff9640 10 client.20109 handle_command_reply: tid=0 2024-09-15T23:37:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2161802740 1 client.20109 shutdown 2024-09-15T23:37:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2161802740 1 -- 172.21.15.89:0/2780984194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 msgr2=0x559b888817f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:31.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2161802740 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b888817f0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f2148001eb0 tx=0x7f2148012780 comp rx=0 tx=0).stop 2024-09-15T23:37:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2161802740 1 -- 172.21.15.89:0/2780984194 shutdown_connections 2024-09-15T23:37:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2161802740 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559b88893eb0 0x559b888962a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2161802740 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b888817f0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2161802740 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b88881060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2161802740 1 --2- 172.21.15.89:0/2780984194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b888795d0 0x559b88883db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2161802740 1 -- 172.21.15.89:0/2780984194 >> 172.21.15.89:0/2780984194 conn(0x559b88842310 msgr2=0x559b8887b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.033+0000 7f2140ff9640 0 client.20109 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:37:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.034+0000 7f2161802740 1 -- 172.21.15.89:0/2780984194 shutdown_connections 2024-09-15T23:37:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.034+0000 7f2161802740 1 -- 172.21.15.89:0/2780984194 wait complete. 2024-09-15T23:37:31.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.034+0000 7f2161802740 20 client.20109 trim_cache size 0 max 16384 2024-09-15T23:37:31.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.046+0000 7f2161802740 1 Processor -- start 2024-09-15T23:37:31.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.046+0000 7f2161802740 1 -- start start 2024-09-15T23:37:31.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f2161802740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b888795d0 0x559b88875df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f2161802740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b8887ab40 0x559b8887c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f2161802740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b8887c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f2161802740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f21581b5410 con 0x559b88884370 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f2161802740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f21581b6f50 con 0x559b888795d0 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f2161802740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f21581b5ae0 con 0x559b8887ab40 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215c9c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b8887c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215c9c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b8887c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34344/0 (socket says 172.21.15.89:34344) 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215c9c4640 1 -- 172.21.15.89:0/3420050159 learned_addr learned my addr 172.21.15.89:0/3420050159 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215d1c5640 1 --2- 172.21.15.89:0/3420050159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b8887ab40 0x559b8887c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215c9c4640 1 --2- 172.21.15.89:0/3420050159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b888795d0 0x559b88875df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215c9c4640 1 -- 172.21.15.89:0/3420050159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b8887ab40 msgr2=0x559b8887c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215c9c4640 1 --2- 172.21.15.89:0/3420050159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b8887ab40 0x559b8887c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215c9c4640 1 -- 172.21.15.89:0/3420050159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b888795d0 msgr2=0x559b88875df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215c9c4640 1 --2- 172.21.15.89:0/3420050159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b888795d0 0x559b88875df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215c9c4640 1 -- 172.21.15.89:0/3420050159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b8887a300 con 0x559b88884370 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f215c9c4640 1 --2- 172.21.15.89:0/3420050159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b8887c660 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x559b88871040 tx=0x7f2148005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78cf2307eacefc7a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f2142ffd640 1 -- 172.21.15.89:0/3420050159 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148012530 con 0x559b88884370 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f2142ffd640 1 -- 172.21.15.89:0/3420050159 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f214800baf0 con 0x559b88884370 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.047+0000 7f2142ffd640 1 -- 172.21.15.89:0/3420050159 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214800fc30 con 0x559b88884370 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 -- 172.21.15.89:0/3420050159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 msgr2=0x559b8887c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 --2- 172.21.15.89:0/3420050159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b8887c660 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x559b88871040 tx=0x7f2148005fb0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 -- 172.21.15.89:0/3420050159 shutdown_connections 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 --2- 172.21.15.89:0/3420050159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b88884370 0x559b8887c660 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 --2- 172.21.15.89:0/3420050159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b8887ab40 0x559b8887c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 --2- 172.21.15.89:0/3420050159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b888795d0 0x559b88875df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 -- 172.21.15.89:0/3420050159 >> 172.21.15.89:0/3420050159 conn(0x559b88842310 msgr2=0x559b888756e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:31.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 -- 172.21.15.89:0/3420050159 shutdown_connections 2024-09-15T23:37:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 -- 172.21.15.89:0/3420050159 wait complete. 2024-09-15T23:37:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 -- start start 2024-09-15T23:37:31.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.048+0000 7f2161802740 1 Processor -- start 2024-09-15T23:37:31.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f2161802740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b888795d0 0x559b8888c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f2161802740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b8888ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f2161802740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b88884370 0x559b88887bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f2161802740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f21581b6880 con 0x559b888795d0 2024-09-15T23:37:31.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f2161802740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f21581b61b0 con 0x559b8887ab40 2024-09-15T23:37:31.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f2161802740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f21581b33f0 con 0x559b88884370 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f2161802740 10 client.0 _ll_register_callbacks cb 0x7f214800be90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f215c9c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b888795d0 0x559b8888c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f215c9c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b888795d0 0x559b8888c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34348/0 (socket says 172.21.15.89:34348) 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f215c9c4640 1 -- 172.21.15.89:0/53971342 learned_addr learned my addr 172.21.15.89:0/53971342 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f215d1c5640 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b8888ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f215c9c4640 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b88884370 0x559b88887bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f215c9c4640 1 -- 172.21.15.89:0/53971342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b88884370 msgr2=0x559b88887bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f215c9c4640 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b88884370 0x559b88887bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f215c9c4640 1 -- 172.21.15.89:0/53971342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 msgr2=0x559b8888ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f215c9c4640 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b8888ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.049+0000 7f215c9c4640 1 -- 172.21.15.89:0/53971342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b888883b0 con 0x559b888795d0 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f215c9c4640 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b888795d0 0x559b8888c780 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f214800fa80 tx=0x7f2148012ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85d190aaebd4c9b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:31.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2140ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2140ff9640 1 -- 172.21.15.89:0/53971342 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214800fd80 con 0x559b888795d0 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2140ff9640 1 -- 172.21.15.89:0/53971342 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2148020480 con 0x559b888795d0 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2161802740 1 -- 172.21.15.89:0/53971342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f214800baf0 con 0x559b888795d0 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2140ff9640 1 -- 172.21.15.89:0/53971342 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2148020680 con 0x559b888795d0 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2161802740 10 client.20115 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2161802740 1 -- 172.21.15.89:0/53971342 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559b888fe580 con 0x559b888795d0 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2140ff9640 1 -- 172.21.15.89:0/53971342 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f214800baf0 con 0x559b888795d0 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2140ff9640 1 -- 172.21.15.89:0/53971342 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f214801e070 con 0x559b888795d0 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2161802740 10 client.20115 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2161802740 10 client.20115 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2161802740 20 client.20115 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2161802740 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559b888fe9b0 0x559b88900da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2161802740 4 client.20115 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:37:31.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.050+0000 7f2161802740 1 -- 172.21.15.89:0/53971342 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559b887e6d70 con 0x559b888fe9b0 2024-09-15T23:37:31.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.051+0000 7f215d1c5640 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559b888fe9b0 0x559b88900da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:31.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.051+0000 7f215d1c5640 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559b888fe9b0 0x559b88900da0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d1ad7937e13019c1 server_cookie=b0b257cdd67e8dfc in_seq=0 out_seq=0 2024-09-15T23:37:31.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.051+0000 7f2140ff9640 10 client.20115 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:37:31.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.051+0000 7f2140ff9640 1 -- 172.21.15.89:0/53971342 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559b887e6d70 con 0x559b888fe9b0 2024-09-15T23:37:31.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.051+0000 7f2140ff9640 10 client.20115 handle_command_reply: tid=0 2024-09-15T23:37:31.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.051+0000 7f2161802740 1 client.20115 shutdown 2024-09-15T23:37:31.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.051+0000 7f2161802740 1 -- 172.21.15.89:0/53971342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b888795d0 msgr2=0x559b8888c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:31.058 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.051+0000 7f2161802740 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b888795d0 0x559b8888c780 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f214800fa80 tx=0x7f2148012ed0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.051+0000 7f2161802740 1 -- 172.21.15.89:0/53971342 shutdown_connections 2024-09-15T23:37:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f2161802740 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x559b888fe9b0 0x559b88900da0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f2161802740 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b88884370 0x559b88887bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f2161802740 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b8887ab40 0x559b8888ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f2161802740 1 --2- 172.21.15.89:0/53971342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b888795d0 0x559b8888c780 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f2161802740 1 -- 172.21.15.89:0/53971342 >> 172.21.15.89:0/53971342 conn(0x559b88842310 msgr2=0x559b888828c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f2140ff9640 0 client.20115 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:37:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f2161802740 1 -- 172.21.15.89:0/53971342 shutdown_connections 2024-09-15T23:37:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f2161802740 1 -- 172.21.15.89:0/53971342 wait complete. 2024-09-15T23:37:31.059 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f2161802740 20 client.20115 trim_cache size 0 max 16384 2024-09-15T23:37:31.060 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:37:31.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f215f429640 1 -- 172.21.15.89:0/2021851063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 msgr2=0x7f21440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:31.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f215f429640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 0x7f21440a5700 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f2150032000 tx=0x7f2150048e30 comp rx=0 tx=0).stop 2024-09-15T23:37:31.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f215f429640 1 -- 172.21.15.89:0/2021851063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2158105f40 msgr2=0x7f215819e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f215f429640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2158105f40 0x7f215819e850 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f2150006010 tx=0x7f215002f700 comp rx=0 tx=0).stop 2024-09-15T23:37:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f215f429640 1 -- 172.21.15.89:0/2021851063 shutdown_connections 2024-09-15T23:37:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f215f429640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f21440a3240 0x7f21440a5700 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f215f429640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2158105f40 0x7f215819e850 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f215f429640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2158101450 0x7f21581a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f215f429640 1 --2- 172.21.15.89:0/2021851063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2158100b30 0x7f21581a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.052+0000 7f215f429640 1 -- 172.21.15.89:0/2021851063 >> 172.21.15.89:0/2021851063 conn(0x7f21580fc7a0 msgr2=0x7f21581038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.053+0000 7f215f429640 1 -- 172.21.15.89:0/2021851063 shutdown_connections 2024-09-15T23:37:31.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:31.053+0000 7f215f429640 1 -- 172.21.15.89:0/2021851063 wait complete. 2024-09-15T23:37:31.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:30 smithi120 ceph-mon[35250]: pgmap v754: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:37:31.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:30 smithi174 ceph-mon[35586]: pgmap v754: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:37:31.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:30 smithi089 ceph-mon[30933]: pgmap v754: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:37:31.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:37:31.998 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:32.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.474+0000 7fb25324e640 1 -- 172.21.15.89:0/2998551924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24c101420 msgr2=0x7fb24c10d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.474+0000 7fb25324e640 1 --2- 172.21.15.89:0/2998551924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24c101420 0x7fb24c10d950 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fb244009cd0 tx=0x7fb24402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.474+0000 7fb25324e640 1 -- 172.21.15.89:0/2998551924 shutdown_connections 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.474+0000 7fb25324e640 1 --2- 172.21.15.89:0/2998551924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24c101420 0x7fb24c10d950 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.474+0000 7fb25324e640 1 --2- 172.21.15.89:0/2998551924 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb24c100a80 0x7fb24c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.474+0000 7fb25324e640 1 --2- 172.21.15.89:0/2998551924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c106a80 0x7fb24c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.474+0000 7fb25324e640 1 -- 172.21.15.89:0/2998551924 >> 172.21.15.89:0/2998551924 conn(0x7fb24c0fc7a0 msgr2=0x7fb24c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.474+0000 7fb25324e640 1 -- 172.21.15.89:0/2998551924 shutdown_connections 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.474+0000 7fb25324e640 1 -- 172.21.15.89:0/2998551924 wait complete. 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb25324e640 1 Processor -- start 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb25324e640 1 -- start start 2024-09-15T23:37:32.487 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb25324e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb24c100a80 0x7fb24c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb25324e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c101420 0x7fb24c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb25324e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24c106a80 0x7fb24c19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb25324e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c111fa0 con 0x7fb24c106a80 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb25324e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c111e20 con 0x7fb24c101420 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb25324e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c112120 con 0x7fb24c100a80 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb2517eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24c106a80 0x7fb24c19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb2517eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24c106a80 0x7fb24c19cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34372/0 (socket says 172.21.15.89:34372) 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb2517eb640 1 -- 172.21.15.89:0/683924065 learned_addr learned my addr 172.21.15.89:0/683924065 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb24bfff640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c101420 0x7fb24c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.475+0000 7fb250fea640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb24c100a80 0x7fb24c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb2517eb640 1 -- 172.21.15.89:0/683924065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb24c100a80 msgr2=0x7fb24c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb2517eb640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb24c100a80 0x7fb24c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb2517eb640 1 -- 172.21.15.89:0/683924065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c101420 msgr2=0x7fb24c1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb2517eb640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c101420 0x7fb24c1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb2517eb640 1 -- 172.21.15.89:0/683924065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24c19d410 con 0x7fb24c106a80 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb2517eb640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24c106a80 0x7fb24c19cd30 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fb244005cc0 tx=0x7fb244003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb249ffb640 1 -- 172.21.15.89:0/683924065 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb244004040 con 0x7fb24c106a80 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb249ffb640 1 -- 172.21.15.89:0/683924065 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2440046c0 con 0x7fb24c106a80 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb25324e640 1 -- 172.21.15.89:0/683924065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24c19d6a0 con 0x7fb24c106a80 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb249ffb640 1 -- 172.21.15.89:0/683924065 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb244030e70 con 0x7fb24c106a80 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.476+0000 7fb25324e640 1 -- 172.21.15.89:0/683924065 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb24c1a4fe0 con 0x7fb24c106a80 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.478+0000 7fb249ffb640 1 -- 172.21.15.89:0/683924065 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb2440305a0 con 0x7fb24c106a80 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.478+0000 7fb255627740 1 Processor -- start 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb255627740 1 -- start start 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb255627740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808fb40 0x558fa808e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb255627740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa808e5d0 0x558fa808ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb255627740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa8099370 0x558fa809b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb255627740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558fa7ff42f0 con 0x558fa8099370 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb255627740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558fa7f9bcf0 con 0x558fa808e5d0 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb255627740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558fa7ff02b0 con 0x558fa808fb40 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb249ffb640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb23c0a32c0 0x7fb23c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb24bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa8099370 0x558fa809b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb24bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa8099370 0x558fa809b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34376/0 (socket says 172.21.15.89:34376) 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb24bfff640 1 -- 172.21.15.89:0/2043686398 learned_addr learned my addr 172.21.15.89:0/2043686398 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb2517eb640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb23c0a32c0 0x7fb23c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb24bfff640 1 --2- 172.21.15.89:0/2043686398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808fb40 0x558fa808e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb250fea640 1 --2- 172.21.15.89:0/2043686398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa808e5d0 0x558fa808ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb249ffb640 1 -- 172.21.15.89:0/683924065 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb24402e030 con 0x7fb24c106a80 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb24bfff640 1 -- 172.21.15.89:0/2043686398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808fb40 msgr2=0x558fa808e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:32.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb24bfff640 1 --2- 172.21.15.89:0/2043686398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808fb40 0x558fa808e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb24bfff640 1 -- 172.21.15.89:0/2043686398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa808e5d0 msgr2=0x558fa808ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb24bfff640 1 --2- 172.21.15.89:0/2043686398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa808e5d0 0x558fa808ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb24bfff640 1 -- 172.21.15.89:0/2043686398 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fa7f803a0 con 0x558fa8099370 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.479+0000 7fb2517eb640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb23c0a32c0 0x7fb23c0a5780 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb244045000 tx=0x7fb2440dd040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb24bfff640 1 --2- 172.21.15.89:0/2043686398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa8099370 0x558fa809b760 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fb238006aa0 tx=0x7fb238009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f861e77be60f4c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb236ffd640 1 -- 172.21.15.89:0/2043686398 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb238015760 con 0x558fa8099370 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb236ffd640 1 -- 172.21.15.89:0/2043686398 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb23801d070 con 0x558fa8099370 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb236ffd640 1 -- 172.21.15.89:0/2043686398 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb238018b20 con 0x558fa8099370 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb255627740 1 -- 172.21.15.89:0/2043686398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa8099370 msgr2=0x558fa809b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb255627740 1 --2- 172.21.15.89:0/2043686398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa8099370 0x558fa809b760 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fb238006aa0 tx=0x7fb238009510 comp rx=0 tx=0).stop 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb255627740 1 -- 172.21.15.89:0/2043686398 shutdown_connections 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb255627740 1 --2- 172.21.15.89:0/2043686398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa8099370 0x558fa809b760 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb255627740 1 --2- 172.21.15.89:0/2043686398 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa808e5d0 0x558fa808ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb255627740 1 --2- 172.21.15.89:0/2043686398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808fb40 0x558fa808e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.480+0000 7fb255627740 1 -- 172.21.15.89:0/2043686398 >> 172.21.15.89:0/2043686398 conn(0x558fa8057310 msgr2=0x558fa808ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 1 -- 172.21.15.89:0/2043686398 shutdown_connections 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 1 -- 172.21.15.89:0/2043686398 wait complete. 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 1 -- start start 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 1 Processor -- start 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808e5d0 0x558fa8098dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa8096070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa8099370 0x558fa8096800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558fa809c010 con 0x558fa808fb40 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558fa7faf060 con 0x558fa8099370 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558fa809c190 con 0x558fa808e5d0 2024-09-15T23:37:32.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb255627740 10 client.0 _ll_register_callbacks cb 0x558fa7d028c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb250fea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa8096070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb250fea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa8096070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34380/0 (socket says 172.21.15.89:34380) 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.481+0000 7fb250fea640 1 -- 172.21.15.89:0/4111869340 learned_addr learned my addr 172.21.15.89:0/4111869340 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb24bfff640 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa8099370 0x558fa8096800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb250fea640 1 -- 172.21.15.89:0/4111869340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808e5d0 msgr2=0x558fa8098dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb250fea640 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808e5d0 0x558fa8098dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb250fea640 1 -- 172.21.15.89:0/4111869340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa8099370 msgr2=0x558fa8096800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb250fea640 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa8099370 0x558fa8096800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb250fea640 1 -- 172.21.15.89:0/4111869340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fa7d653c0 con 0x558fa808fb40 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb250fea640 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa8096070 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fb24000bee0 tx=0x7fb240012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d6749e31aac8f24 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb22bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb22bfff640 1 -- 172.21.15.89:0/4111869340 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb240004230 con 0x558fa808fb40 2024-09-15T23:37:32.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb22bfff640 1 -- 172.21.15.89:0/4111869340 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb2400043d0 con 0x558fa808fb40 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb255627740 1 -- 172.21.15.89:0/4111869340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb23801d070 con 0x558fa808fb40 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb22bfff640 1 -- 172.21.15.89:0/4111869340 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb240024050 con 0x558fa808fb40 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb255627740 10 client.20127 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb255627740 1 -- 172.21.15.89:0/4111869340 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558fa80a7b80 con 0x558fa808fb40 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb22bfff640 1 -- 172.21.15.89:0/4111869340 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb23801d070 con 0x558fa808fb40 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.482+0000 7fb22bfff640 1 -- 172.21.15.89:0/4111869340 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb240003b50 con 0x558fa808fb40 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.483+0000 7fb255627740 10 client.20127 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.483+0000 7fb255627740 10 client.20127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.483+0000 7fb255627740 20 client.20127 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.483+0000 7fb255627740 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558fa80a9090 0x558fa80ab4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.483+0000 7fb255627740 4 client.20127 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.483+0000 7fb255627740 1 -- 172.21.15.89:0/4111869340 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fa8001860 con 0x558fa80a9090 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.483+0000 7fb24bfff640 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558fa80a9090 0x558fa80ab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.483+0000 7fb22bfff640 10 client.20127 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.483+0000 7fb24bfff640 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558fa80a9090 0x558fa80ab4d0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19b375a7c3155694 server_cookie=4870fbe0c18e014c in_seq=0 out_seq=0 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb22bfff640 1 -- 172.21.15.89:0/4111869340 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558fa8001860 con 0x558fa80a9090 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb22bfff640 10 client.20127 handle_command_reply: tid=0 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 client.20127 shutdown 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 -- 172.21.15.89:0/4111869340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 msgr2=0x558fa8096070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa8096070 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fb24000bee0 tx=0x7fb240012820 comp rx=0 tx=0).stop 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 -- 172.21.15.89:0/4111869340 shutdown_connections 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558fa80a9090 0x558fa80ab4d0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa8099370 0x558fa8096800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa8096070 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb22bfff640 0 client.20127 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:32.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 --2- 172.21.15.89:0/4111869340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808e5d0 0x558fa8098dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 -- 172.21.15.89:0/4111869340 >> 172.21.15.89:0/4111869340 conn(0x558fa8057310 msgr2=0x558fa8090960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 -- 172.21.15.89:0/4111869340 shutdown_connections 2024-09-15T23:37:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 1 -- 172.21.15.89:0/4111869340 wait complete. 2024-09-15T23:37:32.496 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.484+0000 7fb255627740 20 client.20127 trim_cache size 0 max 16384 2024-09-15T23:37:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.497+0000 7fb255627740 1 Processor -- start 2024-09-15T23:37:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.497+0000 7fb255627740 1 -- start start 2024-09-15T23:37:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.497+0000 7fb255627740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808e5d0 0x558fa8095050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.497+0000 7fb255627740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.497+0000 7fb255627740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa8099370 0x558fa80b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.497+0000 7fb255627740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1aed70 con 0x558fa808fb40 2024-09-15T23:37:32.499 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.497+0000 7fb255627740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1b08b0 con 0x558fa8099370 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.497+0000 7fb255627740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1af440 con 0x558fa808e5d0 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.497+0000 7fb250fea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb250fea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80b0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34392/0 (socket says 172.21.15.89:34392) 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb250fea640 1 -- 172.21.15.89:0/874701790 learned_addr learned my addr 172.21.15.89:0/874701790 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb24bfff640 1 --2- 172.21.15.89:0/874701790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808e5d0 0x558fa8095050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb250fea640 1 -- 172.21.15.89:0/874701790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808e5d0 msgr2=0x558fa8095050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb250fea640 1 --2- 172.21.15.89:0/874701790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808e5d0 0x558fa8095050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb24bfff640 1 --2- 172.21.15.89:0/874701790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa8099370 0x558fa80b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb250fea640 1 -- 172.21.15.89:0/874701790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa8099370 msgr2=0x558fa80b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb250fea640 1 --2- 172.21.15.89:0/874701790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa8099370 0x558fa80b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb250fea640 1 -- 172.21.15.89:0/874701790 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fa8096b70 con 0x558fa808fb40 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb250fea640 1 --2- 172.21.15.89:0/874701790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80b0da0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fb240000c00 tx=0x7fb24000bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c78e708b92cd2df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:32.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb2367fc640 1 -- 172.21.15.89:0/874701790 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb240003fa0 con 0x558fa808fb40 2024-09-15T23:37:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb2367fc640 1 -- 172.21.15.89:0/874701790 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb24002c030 con 0x558fa808fb40 2024-09-15T23:37:32.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.498+0000 7fb2367fc640 1 -- 172.21.15.89:0/874701790 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb240003a10 con 0x558fa808fb40 2024-09-15T23:37:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 -- 172.21.15.89:0/874701790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 msgr2=0x558fa80b0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 --2- 172.21.15.89:0/874701790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80b0da0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fb240000c00 tx=0x7fb24000bb50 comp rx=0 tx=0).stop 2024-09-15T23:37:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 -- 172.21.15.89:0/874701790 shutdown_connections 2024-09-15T23:37:32.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 --2- 172.21.15.89:0/874701790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa8099370 0x558fa80b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 --2- 172.21.15.89:0/874701790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80b0da0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 --2- 172.21.15.89:0/874701790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa808e5d0 0x558fa8095050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 -- 172.21.15.89:0/874701790 >> 172.21.15.89:0/874701790 conn(0x558fa8057310 msgr2=0x558fa8094940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 -- 172.21.15.89:0/874701790 shutdown_connections 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 -- 172.21.15.89:0/874701790 wait complete. 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 -- start start 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.499+0000 7fb255627740 1 Processor -- start 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb255627740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa808e5d0 0x558fa8093080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb255627740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80935c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb255627740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa8099370 0x558fa8093b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb255627740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1b01e0 con 0x558fa808fb40 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb255627740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1afb10 con 0x558fa808e5d0 2024-09-15T23:37:32.505 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb255627740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c1acd50 con 0x558fa8099370 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb255627740 10 client.0 _ll_register_callbacks cb 0x7fb24000b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb250fea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80935c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb250fea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80935c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34404/0 (socket says 172.21.15.89:34404) 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb250fea640 1 -- 172.21.15.89:0/2891182264 learned_addr learned my addr 172.21.15.89:0/2891182264 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb24bfff640 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa8099370 0x558fa8093b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb24bfff640 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa808e5d0 0x558fa8093080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb250fea640 1 -- 172.21.15.89:0/2891182264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa8099370 msgr2=0x558fa8093b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb250fea640 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa8099370 0x558fa8093b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb250fea640 1 -- 172.21.15.89:0/2891182264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa808e5d0 msgr2=0x558fa8093080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb250fea640 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa808e5d0 0x558fa8093080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb250fea640 1 -- 172.21.15.89:0/2891182264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fa8112da0 con 0x558fa808fb40 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb250fea640 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80935c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fb24000bc00 tx=0x7fb240004170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71286aa0e440f1db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb22bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:32.506 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb22bfff640 1 -- 172.21.15.89:0/2891182264 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb240005040 con 0x558fa808fb40 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb22bfff640 1 -- 172.21.15.89:0/2891182264 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb2400051e0 con 0x558fa808fb40 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.500+0000 7fb255627740 1 -- 172.21.15.89:0/2891182264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb24002c030 con 0x558fa808fb40 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb22bfff640 1 -- 172.21.15.89:0/2891182264 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb240005400 con 0x558fa808fb40 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb255627740 10 client.20133 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb255627740 1 -- 172.21.15.89:0/2891182264 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558fa8114060 con 0x558fa808fb40 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb22bfff640 1 -- 172.21.15.89:0/2891182264 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb24002c030 con 0x558fa808fb40 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb22bfff640 1 -- 172.21.15.89:0/2891182264 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb240004950 con 0x558fa808fb40 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb255627740 10 client.20133 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb255627740 10 client.20133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb255627740 20 client.20133 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb255627740 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558fa8114660 0x558fa8116a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb255627740 4 client.20133 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb255627740 1 -- 172.21.15.89:0/2891182264 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558fa7ffbd70 con 0x558fa8114660 2024-09-15T23:37:32.507 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.501+0000 7fb24bfff640 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558fa8114660 0x558fa8116a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb22bfff640 10 client.20133 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb24bfff640 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558fa8114660 0x558fa8116a50 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b0aeb58279710f2 server_cookie=6a7e14354ccaed8 in_seq=0 out_seq=0 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb22bfff640 1 -- 172.21.15.89:0/2891182264 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558fa7ffbd70 con 0x558fa8114660 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb22bfff640 10 client.20133 handle_command_reply: tid=0 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 client.20133 shutdown 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 -- 172.21.15.89:0/2891182264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 msgr2=0x558fa80935c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80935c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fb24000bc00 tx=0x7fb240004170 comp rx=0 tx=0).stop 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 -- 172.21.15.89:0/2891182264 shutdown_connections 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x558fa8114660 0x558fa8116a50 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558fa8099370 0x558fa8093b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa808fb40 0x558fa80935c0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb22bfff640 0 client.20133 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:32.508 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 --2- 172.21.15.89:0/2891182264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558fa808e5d0 0x558fa8093080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 -- 172.21.15.89:0/2891182264 >> 172.21.15.89:0/2891182264 conn(0x558fa8057310 msgr2=0x558fa8095160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 -- 172.21.15.89:0/2891182264 shutdown_connections 2024-09-15T23:37:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 1 -- 172.21.15.89:0/2891182264 wait complete. 2024-09-15T23:37:32.509 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.502+0000 7fb255627740 20 client.20133 trim_cache size 0 max 16384 2024-09-15T23:37:32.509 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:37:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 -- 172.21.15.89:0/683924065 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb23c0a32c0 msgr2=0x7fb23c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb23c0a32c0 0x7fb23c0a5780 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb244045000 tx=0x7fb2440dd040 comp rx=0 tx=0).stop 2024-09-15T23:37:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 -- 172.21.15.89:0/683924065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24c106a80 msgr2=0x7fb24c19cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24c106a80 0x7fb24c19cd30 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fb244005cc0 tx=0x7fb244003970 comp rx=0 tx=0).stop 2024-09-15T23:37:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 -- 172.21.15.89:0/683924065 shutdown_connections 2024-09-15T23:37:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb23c0a32c0 0x7fb23c0a5780 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb24c106a80 0x7fb24c19cd30 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c101420 0x7fb24c1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 --2- 172.21.15.89:0/683924065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb24c100a80 0x7fb24c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 -- 172.21.15.89:0/683924065 >> 172.21.15.89:0/683924065 conn(0x7fb24c0fc7a0 msgr2=0x7fb24c0fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 -- 172.21.15.89:0/683924065 shutdown_connections 2024-09-15T23:37:32.511 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:32.503+0000 7fb25324e640 1 -- 172.21.15.89:0/683924065 wait complete. 2024-09-15T23:37:33.202 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:32 smithi089 ceph-mon[30933]: pgmap v755: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-15T23:37:33.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:32 smithi120 ceph-mon[35250]: pgmap v755: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-15T23:37:33.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:32 smithi174 ceph-mon[35586]: pgmap v755: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-15T23:37:34.239 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:37:34.239 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:37:34.443 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 11.4 2024-09-15T23:37:34.444 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:37:34.502 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:35.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.990+0000 7f2f802dd640 1 -- 172.21.15.89:0/3322692821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f78106f10 msgr2=0x7f2f7810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:35.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.990+0000 7f2f802dd640 1 --2- 172.21.15.89:0/3322692821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f78106f10 0x7f2f7810ea90 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009ca0 tx=0x7f2f7002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.990+0000 7f2f802dd640 1 -- 172.21.15.89:0/3322692821 shutdown_connections 2024-09-15T23:37:35.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.990+0000 7f2f802dd640 1 --2- 172.21.15.89:0/3322692821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f78106f10 0x7f2f7810ea90 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.990+0000 7f2f802dd640 1 --2- 172.21.15.89:0/3322692821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f78102790 0x7f2f781067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.990+0000 7f2f802dd640 1 --2- 172.21.15.89:0/3322692821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78101de0 0x7f2f781021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.990+0000 7f2f802dd640 1 -- 172.21.15.89:0/3322692821 >> 172.21.15.89:0/3322692821 conn(0x7f2f780fd8c0 msgr2=0x7f2f780ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.990+0000 7f2f802dd640 1 -- 172.21.15.89:0/3322692821 shutdown_connections 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.990+0000 7f2f802dd640 1 -- 172.21.15.89:0/3322692821 wait complete. 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f802dd640 1 Processor -- start 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f802dd640 1 -- start start 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f802dd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f78101de0 0x7f2f78198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f802dd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78102790 0x7f2f78198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f802dd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f78106f10 0x7f2f7819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f802dd640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2f78111f70 con 0x7f2f78101de0 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f802dd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2f78111df0 con 0x7f2f78102790 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f802dd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f781120f0 con 0x7f2f78106f10 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7e079640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f78101de0 0x7f2f78198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7e079640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f78101de0 0x7f2f78198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34410/0 (socket says 172.21.15.89:34410) 2024-09-15T23:37:35.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7e079640 1 -- 172.21.15.89:0/3220291326 learned_addr learned my addr 172.21.15.89:0/3220291326 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f78106f10 0x7f2f7819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7e079640 1 -- 172.21.15.89:0/3220291326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f78106f10 msgr2=0x7f2f7819cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7e079640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f78106f10 0x7f2f7819cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7d878640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78102790 0x7f2f78198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7e079640 1 -- 172.21.15.89:0/3220291326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78102790 msgr2=0x7f2f78198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7e079640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78102790 0x7f2f78198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7e079640 1 -- 172.21.15.89:0/3220291326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7819d2e0 con 0x7f2f78101de0 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.991+0000 7f2f7e079640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f78101de0 0x7f2f78198330 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f2f7400c920 tx=0x7f2f7400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.992+0000 7f2f6b7fe640 1 -- 172.21.15.89:0/3220291326 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f74007c70 con 0x7f2f78101de0 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.992+0000 7f2f6b7fe640 1 -- 172.21.15.89:0/3220291326 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f74007e10 con 0x7f2f78101de0 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.992+0000 7f2f802dd640 1 -- 172.21.15.89:0/3220291326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7819d5d0 con 0x7f2f78101de0 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.992+0000 7f2f6b7fe640 1 -- 172.21.15.89:0/3220291326 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f74018890 con 0x7f2f78101de0 2024-09-15T23:37:35.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.992+0000 7f2f802dd640 1 -- 172.21.15.89:0/3220291326 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f781a4eb0 con 0x7f2f78101de0 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.994+0000 7f2f6b7fe640 1 -- 172.21.15.89:0/3220291326 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2f74018a70 con 0x7f2f78101de0 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.994+0000 7f2f826b6740 1 Processor -- start 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.994+0000 7f2f826b6740 1 -- start start 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa17ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 0x5640fa189800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 0x5640fa18c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f826b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5640fa0e4280 con 0x5640fa17eb60 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f826b6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5640fa08bb10 con 0x5640fa17f480 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f826b6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5640fa0e0240 con 0x5640fa189d40 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e87a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa17ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e87a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa17ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34418/0 (socket says 172.21.15.89:34418) 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e87a640 1 -- 172.21.15.89:0/374930682 learned_addr learned my addr 172.21.15.89:0/374930682 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:35.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7d878640 1 --2- 172.21.15.89:0/374930682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 0x5640fa18c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e079640 1 --2- 172.21.15.89:0/374930682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 0x5640fa189800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f6b7fe640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f640a3240 0x7f2f640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e87a640 1 -- 172.21.15.89:0/374930682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 msgr2=0x5640fa18c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/374930682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 0x5640fa18c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e87a640 1 -- 172.21.15.89:0/374930682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 msgr2=0x5640fa189800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/374930682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 0x5640fa189800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e87a640 1 -- 172.21.15.89:0/374930682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640fa0981f0 con 0x5640fa17eb60 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f6b7fe640 1 -- 172.21.15.89:0/3220291326 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2f740cff60 con 0x7f2f78101de0 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f640a3240 0x7f2f640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.995+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/374930682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa17ef40 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f2f70005cc0 tx=0x7f2f70031bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d393834ba8042c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f68dea640 1 -- 172.21.15.89:0/374930682 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70041040 con 0x5640fa17eb60 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f68dea640 1 -- 172.21.15.89:0/374930682 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f70036070 con 0x5640fa17eb60 2024-09-15T23:37:35.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f68dea640 1 -- 172.21.15.89:0/374930682 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7002fad0 con 0x5640fa17eb60 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f826b6740 1 -- 172.21.15.89:0/374930682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 msgr2=0x5640fa17ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f640a3240 0x7f2f640a5700 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f2f7004c000 tx=0x7f2f7004d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f826b6740 1 --2- 172.21.15.89:0/374930682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa17ef40 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f2f70005cc0 tx=0x7f2f70031bb0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f826b6740 1 -- 172.21.15.89:0/374930682 shutdown_connections 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f826b6740 1 --2- 172.21.15.89:0/374930682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 0x5640fa18c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f826b6740 1 --2- 172.21.15.89:0/374930682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 0x5640fa189800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f826b6740 1 --2- 172.21.15.89:0/374930682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa17ef40 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.996+0000 7f2f826b6740 1 -- 172.21.15.89:0/374930682 >> 172.21.15.89:0/374930682 conn(0x5640fa17b0f0 msgr2=0x5640fa17b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 1 -- 172.21.15.89:0/374930682 shutdown_connections 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 1 -- 172.21.15.89:0/374930682 wait complete. 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 1 -- start start 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 1 Processor -- start 2024-09-15T23:37:35.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa186bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 0x5640fa187110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 0x5640fa187650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5640fa18c910 con 0x5640fa17eb60 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5640fa09ef10 con 0x5640fa17f480 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5640fa18ca90 con 0x5640fa189d40 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f826b6740 10 client.0 _ll_register_callbacks cb 0x5640fa067a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f7e079640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa186bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f7e079640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa186bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34432/0 (socket says 172.21.15.89:34432) 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f7e079640 1 -- 172.21.15.89:0/2601588823 learned_addr learned my addr 172.21.15.89:0/2601588823 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.997+0000 7f2f7d878640 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 0x5640fa187110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 0x5640fa187650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f7e079640 1 -- 172.21.15.89:0/2601588823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 msgr2=0x5640fa187650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f7e079640 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 0x5640fa187650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f7e079640 1 -- 172.21.15.89:0/2601588823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 msgr2=0x5640fa187110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f7e079640 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 0x5640fa187110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f7e079640 1 -- 172.21.15.89:0/2601588823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640f9e556a0 con 0x5640fa17eb60 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f7d878640 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 0x5640fa187110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 0x5640fa187650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f7e079640 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa186bd0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f2f74098b70 tx=0x7f2f74000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a80c4676de03e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2601588823 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f740a04e0 con 0x5640fa17eb60 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2601588823 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2f74018d60 con 0x5640fa17eb60 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f826b6740 1 -- 172.21.15.89:0/2601588823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f70036070 con 0x5640fa17eb60 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2601588823 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2f740a4da0 con 0x5640fa17eb60 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f826b6740 10 client.20145 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f826b6740 1 -- 172.21.15.89:0/2601588823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5640fa198570 con 0x5640fa17eb60 2024-09-15T23:37:35.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2601588823 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f70036070 con 0x5640fa17eb60 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2601588823 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2f740d3020 con 0x5640fa17eb60 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f826b6740 10 client.20145 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f826b6740 10 client.20145 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f826b6740 20 client.20145 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5640fa199980 0x5640fa19bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f826b6740 4 client.20145 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.998+0000 7f2f826b6740 1 -- 172.21.15.89:0/2601588823 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640fa0f17f0 con 0x5640fa199980 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.999+0000 7f2f7d878640 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5640fa199980 0x5640fa19bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.999+0000 7f2f7d878640 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5640fa199980 0x5640fa19bd70 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=484fd1ee3f80c991 server_cookie=4bef422862052db2 in_seq=0 out_seq=0 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:34.999+0000 7f2f61ffb640 10 client.20145 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2601588823 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5640fa0f17f0 con 0x5640fa199980 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f61ffb640 10 client.20145 handle_command_reply: tid=0 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 client.20145 shutdown 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 -- 172.21.15.89:0/2601588823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 msgr2=0x5640fa186bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:35.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa186bd0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f2f74098b70 tx=0x7f2f74000bc0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 -- 172.21.15.89:0/2601588823 shutdown_connections 2024-09-15T23:37:35.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5640fa199980 0x5640fa19bd70 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa189d40 0x5640fa187650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17f480 0x5640fa187110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f61ffb640 0 client.20145 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:35.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2601588823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa186bd0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 -- 172.21.15.89:0/2601588823 >> 172.21.15.89:0/2601588823 conn(0x5640fa17b0f0 msgr2=0x5640fa17a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:35.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 -- 172.21.15.89:0/2601588823 shutdown_connections 2024-09-15T23:37:35.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.000+0000 7f2f826b6740 1 -- 172.21.15.89:0/2601588823 wait complete. 2024-09-15T23:37:35.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.001+0000 7f2f826b6740 20 client.20145 trim_cache size 0 max 16384 2024-09-15T23:37:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f826b6740 1 Processor -- start 2024-09-15T23:37:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f826b6740 1 -- start start 2024-09-15T23:37:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17eb60 0x5640fa188a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 0x5640fa1a1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa189d40 0x5640fa1a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f826b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2f781aee00 con 0x5640fa189d40 2024-09-15T23:37:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f826b6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2f781b0940 con 0x5640fa17eb60 2024-09-15T23:37:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f826b6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f781af4d0 con 0x5640fa17f480 2024-09-15T23:37:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f7e87a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa189d40 0x5640fa1a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f7e87a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa189d40 0x5640fa1a1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34438/0 (socket says 172.21.15.89:34438) 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f7e87a640 1 -- 172.21.15.89:0/3816228107 learned_addr learned my addr 172.21.15.89:0/3816228107 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f7d878640 1 --2- 172.21.15.89:0/3816228107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 0x5640fa1a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f7e079640 1 --2- 172.21.15.89:0/3816228107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17eb60 0x5640fa188a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f7d878640 1 -- 172.21.15.89:0/3816228107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17eb60 msgr2=0x5640fa188a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f7d878640 1 --2- 172.21.15.89:0/3816228107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17eb60 0x5640fa188a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f7d878640 1 -- 172.21.15.89:0/3816228107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa189d40 msgr2=0x5640fa1a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f7d878640 1 --2- 172.21.15.89:0/3816228107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa189d40 0x5640fa1a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.019+0000 7f2f7d878640 1 -- 172.21.15.89:0/3816228107 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640fa17af20 con 0x5640fa17f480 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.020+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/3816228107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa189d40 0x5640fa1a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.020+0000 7f2f7d878640 1 --2- 172.21.15.89:0/3816228107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 0x5640fa1a1620 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x5640fa188210 tx=0x7f2f6c00f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b297b8bdc367654e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.020+0000 7f2f63fff640 1 -- 172.21.15.89:0/3816228107 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c01b420 con 0x5640fa17f480 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.020+0000 7f2f63fff640 1 -- 172.21.15.89:0/3816228107 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f6c01baa0 con 0x5640fa17f480 2024-09-15T23:37:35.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.020+0000 7f2f63fff640 1 -- 172.21.15.89:0/3816228107 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c01ab50 con 0x5640fa17f480 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 -- 172.21.15.89:0/3816228107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 msgr2=0x5640fa1a1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 --2- 172.21.15.89:0/3816228107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 0x5640fa1a1620 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x5640fa188210 tx=0x7f2f6c00f7f0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 -- 172.21.15.89:0/3816228107 shutdown_connections 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 --2- 172.21.15.89:0/3816228107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa189d40 0x5640fa1a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 --2- 172.21.15.89:0/3816228107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 0x5640fa1a1620 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 --2- 172.21.15.89:0/3816228107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa17eb60 0x5640fa188a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 -- 172.21.15.89:0/3816228107 >> 172.21.15.89:0/3816228107 conn(0x5640fa17b0f0 msgr2=0x5640fa188320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 -- 172.21.15.89:0/3816228107 shutdown_connections 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 -- 172.21.15.89:0/3816228107 wait complete. 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 -- start start 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.021+0000 7f2f826b6740 1 Processor -- start 2024-09-15T23:37:35.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa18da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 0x5640fa18df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f826b6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa189d40 0x5640fa18e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f826b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2f781b0270 con 0x5640fa17eb60 2024-09-15T23:37:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f826b6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2f781afba0 con 0x5640fa189d40 2024-09-15T23:37:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f826b6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f781acde0 con 0x5640fa17f480 2024-09-15T23:37:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f826b6740 10 client.0 _ll_register_callbacks cb 0x7f2f74096bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7d878640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 0x5640fa18df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e079640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa18da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e079640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa18da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34448/0 (socket says 172.21.15.89:34448) 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e079640 1 -- 172.21.15.89:0/2346471139 learned_addr learned my addr 172.21.15.89:0/2346471139 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7d878640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 0x5640fa18df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:36110/0 (socket says 172.21.15.89:36110) 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa189d40 0x5640fa18e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e079640 1 -- 172.21.15.89:0/2346471139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 msgr2=0x5640fa18df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e079640 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 0x5640fa18df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e079640 1 -- 172.21.15.89:0/2346471139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa189d40 msgr2=0x5640fa18e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e079640 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa189d40 0x5640fa18e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e079640 1 -- 172.21.15.89:0/2346471139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640fa175680 con 0x5640fa17eb60 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e87a640 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa189d40 0x5640fa18e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f7e079640 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa18da00 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f2f7409c130 tx=0x7f2f74018f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732d2e45e9bef09f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:35.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.022+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2346471139 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f740a04e0 con 0x5640fa17eb60 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2346471139 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2f740bc0b0 con 0x5640fa17eb60 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f826b6740 1 -- 172.21.15.89:0/2346471139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f6c01baa0 con 0x5640fa17eb60 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2346471139 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2f7400fb30 con 0x5640fa17eb60 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f826b6740 10 client.20151 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f826b6740 1 -- 172.21.15.89:0/2346471139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5640fa2046c0 con 0x5640fa17eb60 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2346471139 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f6c01baa0 con 0x5640fa17eb60 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2346471139 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2f7409b060 con 0x5640fa17eb60 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f826b6740 10 client.20151 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f826b6740 10 client.20151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f826b6740 20 client.20151 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5640fa204e60 0x5640fa207280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f826b6740 4 client.20151 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f826b6740 1 -- 172.21.15.89:0/2346471139 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5640fa1819f0 con 0x5640fa204e60 2024-09-15T23:37:35.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.023+0000 7f2f7d878640 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5640fa204e60 0x5640fa207280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:35.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.024+0000 7f2f61ffb640 10 client.20151 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:35.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.024+0000 7f2f7d878640 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5640fa204e60 0x5640fa207280 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed844c515b4f89d4 server_cookie=d4058c3b20602406 in_seq=0 out_seq=0 2024-09-15T23:37:35.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f61ffb640 1 -- 172.21.15.89:0/2346471139 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5640fa1819f0 con 0x5640fa204e60 2024-09-15T23:37:35.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f61ffb640 10 client.20151 handle_command_reply: tid=0 2024-09-15T23:37:35.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 client.20151 shutdown 2024-09-15T23:37:35.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 -- 172.21.15.89:0/2346471139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 msgr2=0x5640fa18da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa18da00 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f2f7409c130 tx=0x7f2f74018f90 comp rx=0 tx=0).stop 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 -- 172.21.15.89:0/2346471139 shutdown_connections 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5640fa204e60 0x5640fa207280 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5640fa189d40 0x5640fa18e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5640fa17f480 0x5640fa18df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 --2- 172.21.15.89:0/2346471139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5640fa17eb60 0x5640fa18da00 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f61ffb640 0 client.20151 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 -- 172.21.15.89:0/2346471139 >> 172.21.15.89:0/2346471139 conn(0x5640fa17b0f0 msgr2=0x5640fa180dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 -- 172.21.15.89:0/2346471139 shutdown_connections 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 1 -- 172.21.15.89:0/2346471139 wait complete. 2024-09-15T23:37:35.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.025+0000 7f2f826b6740 20 client.20151 trim_cache size 0 max 16384 2024-09-15T23:37:35.032 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:37:35.032 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:37:35.032 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "7e71e9a7-418d-462f-8c5e-82b86e51bea5", 2024-09-15T23:37:35.032 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:37:35.032 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:37:35.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 -- 172.21.15.89:0/3220291326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f640a3240 msgr2=0x7f2f640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f640a3240 0x7f2f640a5700 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f2f7004c000 tx=0x7f2f7004d000 comp rx=0 tx=0).stop 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 -- 172.21.15.89:0/3220291326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f78101de0 msgr2=0x7f2f78198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f78101de0 0x7f2f78198330 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f2f7400c920 tx=0x7f2f7400cde0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 -- 172.21.15.89:0/3220291326 shutdown_connections 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2f640a3240 0x7f2f640a5700 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f78106f10 0x7f2f7819cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f78102790 0x7f2f78198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 --2- 172.21.15.89:0/3220291326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f78101de0 0x7f2f78198330 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 -- 172.21.15.89:0/3220291326 >> 172.21.15.89:0/3220291326 conn(0x7f2f780fd8c0 msgr2=0x7f2f780febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 -- 172.21.15.89:0/3220291326 shutdown_connections 2024-09-15T23:37:35.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:35.026+0000 7f2f802dd640 1 -- 172.21.15.89:0/3220291326 wait complete. 2024-09-15T23:37:35.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:34 smithi120 ceph-mon[35250]: pgmap v756: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:37:35.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:34 smithi174 ceph-mon[35586]: pgmap v756: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:37:35.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:34 smithi089 ceph-mon[30933]: pgmap v756: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:37:35.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:37:35.875 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:36.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:35 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:37:36.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:35 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:36.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:35 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:37:36.225 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:35 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:36.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:35 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:37:36.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:35 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:36.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:35 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:37:36.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:35 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:36.268 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:35 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:37:36.268 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:35 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:36.268 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:35 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:37:36.269 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:35 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:37:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 -- 172.21.15.89:0/1738535609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c100a70 msgr2=0x7f734c100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 --2- 172.21.15.89:0/1738535609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c100a70 0x7f734c100e70 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f7348009900 tx=0x7f734802a410 comp rx=0 tx=0).stop 2024-09-15T23:37:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 -- 172.21.15.89:0/1738535609 shutdown_connections 2024-09-15T23:37:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 --2- 172.21.15.89:0/1738535609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f734c102620 0x7f734c108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 --2- 172.21.15.89:0/1738535609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f734c101c60 0x7f734c1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 --2- 172.21.15.89:0/1738535609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c100a70 0x7f734c100e70 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 -- 172.21.15.89:0/1738535609 >> 172.21.15.89:0/1738535609 conn(0x7f734c0fc1b0 msgr2=0x7f734c0fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 -- 172.21.15.89:0/1738535609 shutdown_connections 2024-09-15T23:37:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 -- 172.21.15.89:0/1738535609 wait complete. 2024-09-15T23:37:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 Processor -- start 2024-09-15T23:37:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 -- start start 2024-09-15T23:37:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f734c100a70 0x7f734c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.275+0000 7f73546dd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c101c60 0x7f734c19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f73546dd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f734c102620 0x7f734c19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f73546dd640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f734c10c740 con 0x7f734c101c60 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f73546dd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f734c10c5c0 con 0x7f734c100a70 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f73546dd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f734c10c8c0 con 0x7f734c102620 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7351c78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c101c60 0x7f734c19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7351c78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c101c60 0x7f734c19cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44514/0 (socket says 172.21.15.89:44514) 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7351c78640 1 -- 172.21.15.89:0/3691883764 learned_addr learned my addr 172.21.15.89:0/3691883764 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7352479640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f734c100a70 0x7f734c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7352c7a640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f734c102620 0x7f734c19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7351c78640 1 -- 172.21.15.89:0/3691883764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f734c102620 msgr2=0x7f734c19f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7351c78640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f734c102620 0x7f734c19f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7351c78640 1 -- 172.21.15.89:0/3691883764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f734c100a70 msgr2=0x7f734c06ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7351c78640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f734c100a70 0x7f734c06ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7351c78640 1 -- 172.21.15.89:0/3691883764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734c19f9f0 con 0x7f734c101c60 2024-09-15T23:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7352c7a640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f734c102620 0x7f734c19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7352479640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f734c100a70 0x7f734c06ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f7351c78640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c101c60 0x7f734c19cac0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c970 tx=0x7f733c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f73437fe640 1 -- 172.21.15.89:0/3691883764 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c007bb0 con 0x7f734c101c60 2024-09-15T23:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f73437fe640 1 -- 172.21.15.89:0/3691883764 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f733c007d50 con 0x7f734c101c60 2024-09-15T23:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f73546dd640 1 -- 172.21.15.89:0/3691883764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f734c19fc80 con 0x7f734c101c60 2024-09-15T23:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f73437fe640 1 -- 172.21.15.89:0/3691883764 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c00f450 con 0x7f734c101c60 2024-09-15T23:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.276+0000 7f73546dd640 1 -- 172.21.15.89:0/3691883764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f734c1a01d0 con 0x7f734c101c60 2024-09-15T23:37:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.279+0000 7f73437fe640 1 -- 172.21.15.89:0/3691883764 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f733c005020 con 0x7f734c101c60 2024-09-15T23:37:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.279+0000 7f73437fe640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73380a3370 0x7f73380a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.279+0000 7f73437fe640 1 -- 172.21.15.89:0/3691883764 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f733c0cfb20 con 0x7f734c101c60 2024-09-15T23:37:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.279+0000 7f7352479640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73380a3370 0x7f73380a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f7352479640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73380a3370 0x7f73380a5830 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f734802aa00 tx=0x7f7348031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f735747a740 1 Processor -- start 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f735747a740 1 -- start start 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f735747a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f735747a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53d6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f735747a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53d9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f735747a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dff53352f0 con 0x55dff53d22e0 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f735747a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dff52dd270 con 0x55dff53d19a0 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f735747a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dff53312b0 con 0x55dff53d7370 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f7351c78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53d6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f7351c78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53d6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44524/0 (socket says 172.21.15.89:44524) 2024-09-15T23:37:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.280+0000 7f7351c78640 1 -- 172.21.15.89:0/2779016409 learned_addr learned my addr 172.21.15.89:0/2779016409 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7352479640 1 --2- 172.21.15.89:0/2779016409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7352c7a640 1 --2- 172.21.15.89:0/2779016409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53d9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7351c78640 1 -- 172.21.15.89:0/2779016409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 msgr2=0x55dff53d9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7351c78640 1 --2- 172.21.15.89:0/2779016409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53d9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7351c78640 1 -- 172.21.15.89:0/2779016409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 msgr2=0x55dff53d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7351c78640 1 --2- 172.21.15.89:0/2779016409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7351c78640 1 -- 172.21.15.89:0/2779016409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dff52dc340 con 0x55dff53d22e0 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7351c78640 1 --2- 172.21.15.89:0/2779016409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53d6da0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f733c09bce0 tx=0x7f733c09c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c26a8ae8b84213 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7340ff9640 1 -- 172.21.15.89:0/2779016409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c01d040 con 0x55dff53d22e0 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7340ff9640 1 -- 172.21.15.89:0/2779016409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f733c0a49c0 con 0x55dff53d22e0 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.281+0000 7f7340ff9640 1 -- 172.21.15.89:0/2779016409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c09b1a0 con 0x55dff53d22e0 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 -- 172.21.15.89:0/2779016409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 msgr2=0x55dff53d6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 --2- 172.21.15.89:0/2779016409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53d6da0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f733c09bce0 tx=0x7f733c09c510 comp rx=0 tx=0).stop 2024-09-15T23:37:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 -- 172.21.15.89:0/2779016409 shutdown_connections 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 --2- 172.21.15.89:0/2779016409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53d9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 --2- 172.21.15.89:0/2779016409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53d6da0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 --2- 172.21.15.89:0/2779016409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 -- 172.21.15.89:0/2779016409 >> 172.21.15.89:0/2779016409 conn(0x55dff53cd010 msgr2=0x55dff53cd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 -- 172.21.15.89:0/2779016409 shutdown_connections 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 -- 172.21.15.89:0/2779016409 wait complete. 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 -- start start 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.282+0000 7f735747a740 1 Processor -- start 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f735747a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53dbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f735747a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53dc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f735747a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53e31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f735747a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dff53da340 con 0x55dff53d22e0 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f735747a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dff52be290 con 0x55dff53d19a0 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f735747a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dff53da4c0 con 0x55dff53d7370 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f735747a740 10 client.0 _ll_register_callbacks cb 0x55dff52ef140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7351c78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53dc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7351c78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53dc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44538/0 (socket says 172.21.15.89:44538) 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7351c78640 1 -- 172.21.15.89:0/1709548809 learned_addr learned my addr 172.21.15.89:0/1709548809 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7351c78640 1 -- 172.21.15.89:0/1709548809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 msgr2=0x55dff53e31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7352c7a640 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53e31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7352479640 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53dbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7351c78640 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53e31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7351c78640 1 -- 172.21.15.89:0/1709548809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 msgr2=0x55dff53dbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7351c78640 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53dbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7351c78640 1 -- 172.21.15.89:0/1709548809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dff5048210 con 0x55dff53d22e0 2024-09-15T23:37:36.293 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7352c7a640 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53e31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7351c78640 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53dc190 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f733c0cd280 tx=0x7f733c004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2a4ec7208b4a19 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7335ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7335ffb640 1 -- 172.21.15.89:0/1709548809 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c01d040 con 0x55dff53d22e0 2024-09-15T23:37:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.283+0000 7f7335ffb640 1 -- 172.21.15.89:0/1709548809 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f733c099df0 con 0x55dff53d22e0 2024-09-15T23:37:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f735747a740 1 -- 172.21.15.89:0/1709548809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f733c0a49c0 con 0x55dff53d22e0 2024-09-15T23:37:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f7335ffb640 1 -- 172.21.15.89:0/1709548809 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f733c097aa0 con 0x55dff53d22e0 2024-09-15T23:37:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f735747a740 10 client.20163 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f735747a740 1 -- 172.21.15.89:0/1709548809 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dff5114620 con 0x55dff53d22e0 2024-09-15T23:37:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f7335ffb640 1 -- 172.21.15.89:0/1709548809 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f733c0a49c0 con 0x55dff53d22e0 2024-09-15T23:37:36.294 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f7335ffb640 1 -- 172.21.15.89:0/1709548809 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f733c086120 con 0x55dff53d22e0 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f735747a740 10 client.20163 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f735747a740 10 client.20163 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f735747a740 20 client.20163 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f735747a740 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55dff53e74d0 0x55dff53e98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f735747a740 4 client.20163 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f735747a740 1 -- 172.21.15.89:0/1709548809 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dff5342a20 con 0x55dff53e74d0 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.284+0000 7f7352479640 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55dff53e74d0 0x55dff53e98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.285+0000 7f7352479640 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55dff53e74d0 0x55dff53e98c0 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4200001c140fa664 server_cookie=6b4845167e40324b in_seq=0 out_seq=0 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.285+0000 7f7335ffb640 10 client.20163 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.285+0000 7f7335ffb640 1 -- 172.21.15.89:0/1709548809 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dff5342a20 con 0x55dff53e74d0 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.285+0000 7f7335ffb640 10 client.20163 handle_command_reply: tid=0 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.285+0000 7f735747a740 1 client.20163 shutdown 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 1 -- 172.21.15.89:0/1709548809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 msgr2=0x55dff53dc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.295 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53dc190 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f733c0cd280 tx=0x7f733c004a20 comp rx=0 tx=0).stop 2024-09-15T23:37:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 1 -- 172.21.15.89:0/1709548809 shutdown_connections 2024-09-15T23:37:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55dff53e74d0 0x55dff53e98c0 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53e31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f7335ffb640 0 client.20163 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53dc190 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 1 --2- 172.21.15.89:0/1709548809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53dbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 1 -- 172.21.15.89:0/1709548809 >> 172.21.15.89:0/1709548809 conn(0x55dff53cd010 msgr2=0x55dff53ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 1 -- 172.21.15.89:0/1709548809 shutdown_connections 2024-09-15T23:37:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 1 -- 172.21.15.89:0/1709548809 wait complete. 2024-09-15T23:37:36.296 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.286+0000 7f735747a740 20 client.20163 trim_cache size 0 max 16384 2024-09-15T23:37:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.315+0000 7f735747a740 1 Processor -- start 2024-09-15T23:37:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.315+0000 7f735747a740 1 -- start start 2024-09-15T23:37:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f735747a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f735747a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53e3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f735747a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53ee130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f735747a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f734c1aa750 con 0x55dff53d22e0 2024-09-15T23:37:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f735747a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f734c1ac290 con 0x55dff53d19a0 2024-09-15T23:37:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f735747a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f734c1aae20 con 0x55dff53d7370 2024-09-15T23:37:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7351c78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53e3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7351c78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53e3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44552/0 (socket says 172.21.15.89:44552) 2024-09-15T23:37:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7351c78640 1 -- 172.21.15.89:0/641024927 learned_addr learned my addr 172.21.15.89:0/641024927 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:36.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7352c7a640 1 --2- 172.21.15.89:0/641024927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7351c78640 1 -- 172.21.15.89:0/641024927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 msgr2=0x55dff53ee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:36.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7351c78640 1 --2- 172.21.15.89:0/641024927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53ee130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7352479640 1 --2- 172.21.15.89:0/641024927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7351c78640 1 -- 172.21.15.89:0/641024927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 msgr2=0x55dff53cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:36.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7351c78640 1 --2- 172.21.15.89:0/641024927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7351c78640 1 -- 172.21.15.89:0/641024927 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dff53d9760 con 0x55dff53d22e0 2024-09-15T23:37:36.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7351c78640 1 --2- 172.21.15.89:0/641024927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53e3190 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f733c007590 tx=0x7f733c005310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f4dc696f941962 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:36.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7337fff640 1 -- 172.21.15.89:0/641024927 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c0a8070 con 0x55dff53d22e0 2024-09-15T23:37:36.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7337fff640 1 -- 172.21.15.89:0/641024927 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f733c0a49c0 con 0x55dff53d22e0 2024-09-15T23:37:36.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.316+0000 7f7337fff640 1 -- 172.21.15.89:0/641024927 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c097c40 con 0x55dff53d22e0 2024-09-15T23:37:36.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.317+0000 7f735747a740 1 -- 172.21.15.89:0/641024927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 msgr2=0x55dff53e3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.317+0000 7f735747a740 1 --2- 172.21.15.89:0/641024927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53e3190 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f733c007590 tx=0x7f733c005310 comp rx=0 tx=0).stop 2024-09-15T23:37:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.317+0000 7f735747a740 1 -- 172.21.15.89:0/641024927 shutdown_connections 2024-09-15T23:37:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.317+0000 7f735747a740 1 --2- 172.21.15.89:0/641024927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d7370 0x55dff53ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.317+0000 7f735747a740 1 --2- 172.21.15.89:0/641024927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d22e0 0x55dff53e3190 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.317+0000 7f735747a740 1 --2- 172.21.15.89:0/641024927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.331 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.317+0000 7f735747a740 1 -- 172.21.15.89:0/641024927 >> 172.21.15.89:0/641024927 conn(0x55dff53cd010 msgr2=0x55dff53cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.317+0000 7f735747a740 1 -- 172.21.15.89:0/641024927 shutdown_connections 2024-09-15T23:37:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.317+0000 7f735747a740 1 -- 172.21.15.89:0/641024927 wait complete. 2024-09-15T23:37:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.317+0000 7f735747a740 1 -- start start 2024-09-15T23:37:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f735747a740 1 Processor -- start 2024-09-15T23:37:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f735747a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53e0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f735747a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d22e0 0x55dff53e0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f735747a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d7370 0x55dff53e1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f735747a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f734c1abbc0 con 0x55dff53d7370 2024-09-15T23:37:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f735747a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f734c1ab4f0 con 0x55dff53d19a0 2024-09-15T23:37:36.332 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f735747a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f734c1a8730 con 0x55dff53d22e0 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f735747a740 10 client.0 _ll_register_callbacks cb 0x7f733c0b4f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7352c7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d7370 0x55dff53e1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7352c7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d7370 0x55dff53e1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44556/0 (socket says 172.21.15.89:44556) 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7352c7a640 1 -- 172.21.15.89:0/4128340066 learned_addr learned my addr 172.21.15.89:0/4128340066 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7351c78640 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d22e0 0x55dff53e0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7352c7a640 1 -- 172.21.15.89:0/4128340066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d22e0 msgr2=0x55dff53e0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7352479640 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53e0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7352c7a640 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d22e0 0x55dff53e0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7352c7a640 1 -- 172.21.15.89:0/4128340066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 msgr2=0x55dff53e0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7352c7a640 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53e0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7352c7a640 1 -- 172.21.15.89:0/4128340066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dff53e2470 con 0x55dff53d7370 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7352c7a640 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d7370 0x55dff53e1e60 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f734400fa40 tx=0x7f734400fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ec2fad32adf5fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:36.333 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7335ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7335ffb640 1 -- 172.21.15.89:0/4128340066 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734401c020 con 0x55dff53d7370 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f735747a740 1 -- 172.21.15.89:0/4128340066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f733c0a49c0 con 0x55dff53d7370 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.318+0000 7f7335ffb640 1 -- 172.21.15.89:0/4128340066 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f734400dbf0 con 0x55dff53d7370 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f7335ffb640 1 -- 172.21.15.89:0/4128340066 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7344024050 con 0x55dff53d7370 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f735747a740 10 client.20169 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f735747a740 1 -- 172.21.15.89:0/4128340066 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dff5452180 con 0x55dff53d7370 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f7335ffb640 1 -- 172.21.15.89:0/4128340066 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f733c0a49c0 con 0x55dff53d7370 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f7335ffb640 1 -- 172.21.15.89:0/4128340066 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7344017070 con 0x55dff53d7370 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f735747a740 10 client.20169 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f735747a740 10 client.20169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f735747a740 20 client.20169 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f735747a740 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55dff5452bb0 0x55dff5454fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f735747a740 4 client.20169 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:37:36.334 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f735747a740 1 -- 172.21.15.89:0/4128340066 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dff533cd70 con 0x55dff5452bb0 2024-09-15T23:37:36.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.319+0000 7f7352479640 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55dff5452bb0 0x55dff5454fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.320+0000 7f7335ffb640 10 client.20169 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:36.335 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.320+0000 7f7352479640 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55dff5452bb0 0x55dff5454fd0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b964f69f4165b3f7 server_cookie=fb0905d8ae9d8f5d in_seq=0 out_seq=0 2024-09-15T23:37:36.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.334+0000 7f7335ffb640 1 -- 172.21.15.89:0/4128340066 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dff533cd70 con 0x55dff5452bb0 2024-09-15T23:37:36.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.334+0000 7f7335ffb640 10 client.20169 handle_command_reply: tid=0 2024-09-15T23:37:36.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.334+0000 7f735747a740 1 client.20169 shutdown 2024-09-15T23:37:36.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 1 -- 172.21.15.89:0/4128340066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d7370 msgr2=0x55dff53e1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d7370 0x55dff53e1e60 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f734400fa40 tx=0x7f734400fb30 comp rx=0 tx=0).stop 2024-09-15T23:37:36.336 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 1 -- 172.21.15.89:0/4128340066 shutdown_connections 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55dff5452bb0 0x55dff5454fd0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dff53d7370 0x55dff53e1e60 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dff53d22e0 0x55dff53e0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f7335ffb640 0 client.20169 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 1 --2- 172.21.15.89:0/4128340066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dff53d19a0 0x55dff53e0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 1 -- 172.21.15.89:0/4128340066 >> 172.21.15.89:0/4128340066 conn(0x55dff53cd010 msgr2=0x55dff53ce6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 1 -- 172.21.15.89:0/4128340066 shutdown_connections 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 1 -- 172.21.15.89:0/4128340066 wait complete. 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f735747a740 20 client.20169 trim_cache size 0 max 16384 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:37:36.337 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:37:36.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:37:36.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:37:36.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 -- 172.21.15.89:0/3691883764 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73380a3370 msgr2=0x7f73380a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73380a3370 0x7f73380a5830 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f734802aa00 tx=0x7f7348031400 comp rx=0 tx=0).stop 2024-09-15T23:37:36.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 -- 172.21.15.89:0/3691883764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c101c60 msgr2=0x7f734c19cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c101c60 0x7f734c19cac0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c970 tx=0x7f733c00ce30 comp rx=0 tx=0).stop 2024-09-15T23:37:36.338 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 -- 172.21.15.89:0/3691883764 shutdown_connections 2024-09-15T23:37:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73380a3370 0x7f73380a5830 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f734c102620 0x7f734c19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f734c101c60 0x7f734c19cac0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 --2- 172.21.15.89:0/3691883764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f734c100a70 0x7f734c06ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 -- 172.21.15.89:0/3691883764 >> 172.21.15.89:0/3691883764 conn(0x7f734c0fc1b0 msgr2=0x7f734c104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.335+0000 7f73546dd640 1 -- 172.21.15.89:0/3691883764 shutdown_connections 2024-09-15T23:37:36.339 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:36.336+0000 7f73546dd640 1 -- 172.21.15.89:0/3691883764 wait complete. 2024-09-15T23:37:36.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 -- 172.21.15.89:0/2350317607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa28102570 msgr2=0x7faa281029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 --2- 172.21.15.89:0/2350317607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa28102570 0x7faa281029b0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009900 tx=0x7faa1c02a480 comp rx=0 tx=0).stop 2024-09-15T23:37:36.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 -- 172.21.15.89:0/2350317607 shutdown_connections 2024-09-15T23:37:36.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 --2- 172.21.15.89:0/2350317607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28102ef0 0x7faa2810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 --2- 172.21.15.89:0/2350317607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa28102570 0x7faa281029b0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 --2- 172.21.15.89:0/2350317607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28100e20 0x7faa28101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 -- 172.21.15.89:0/2350317607 >> 172.21.15.89:0/2350317607 conn(0x7faa280fc7d0 msgr2=0x7faa280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:36.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 -- 172.21.15.89:0/2350317607 shutdown_connections 2024-09-15T23:37:36.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 -- 172.21.15.89:0/2350317607 wait complete. 2024-09-15T23:37:36.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 Processor -- start 2024-09-15T23:37:36.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.367+0000 7faa2e64f640 1 -- start start 2024-09-15T23:37:36.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2e64f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28100e20 0x7faa28198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2e64f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28102570 0x7faa28198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2e64f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa28102ef0 0x7faa2819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2e64f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faa28110f50 con 0x7faa28102ef0 2024-09-15T23:37:36.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2e64f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faa28110dd0 con 0x7faa28102570 2024-09-15T23:37:36.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2e64f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faa281110d0 con 0x7faa28100e20 2024-09-15T23:37:36.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2cbec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa28102ef0 0x7faa2819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.375 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa27fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28100e20 0x7faa28198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2cbec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa28102ef0 0x7faa2819cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44576/0 (socket says 172.21.15.89:44576) 2024-09-15T23:37:36.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa27fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28100e20 0x7faa28198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:37276/0 (socket says 172.21.15.89:37276) 2024-09-15T23:37:36.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2cbec640 1 -- 172.21.15.89:0/925734063 learned_addr learned my addr 172.21.15.89:0/925734063 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:36.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa277fe640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28102570 0x7faa28198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2cbec640 1 -- 172.21.15.89:0/925734063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28100e20 msgr2=0x7faa28198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2cbec640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28100e20 0x7faa28198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2cbec640 1 -- 172.21.15.89:0/925734063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28102570 msgr2=0x7faa28198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2cbec640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28102570 0x7faa28198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.368+0000 7faa2cbec640 1 -- 172.21.15.89:0/925734063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2819d4d0 con 0x7faa28102ef0 2024-09-15T23:37:36.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.369+0000 7faa2cbec640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa28102ef0 0x7faa2819cdf0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7faa2000cce0 tx=0x7faa2000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:36.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.369+0000 7faa257fa640 1 -- 172.21.15.89:0/925734063 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2000f450 con 0x7faa28102ef0 2024-09-15T23:37:36.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.369+0000 7faa257fa640 1 -- 172.21.15.89:0/925734063 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa2000fad0 con 0x7faa28102ef0 2024-09-15T23:37:36.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.369+0000 7faa2e64f640 1 -- 172.21.15.89:0/925734063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa2819d7c0 con 0x7faa28102ef0 2024-09-15T23:37:36.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.369+0000 7faa257fa640 1 -- 172.21.15.89:0/925734063 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20018c10 con 0x7faa28102ef0 2024-09-15T23:37:36.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.369+0000 7faa2e64f640 1 -- 172.21.15.89:0/925734063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa281a50a0 con 0x7faa28102ef0 2024-09-15T23:37:36.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.370+0000 7faa2e64f640 1 -- 172.21.15.89:0/925734063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa281021b0 con 0x7faa28102ef0 2024-09-15T23:37:36.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.372+0000 7faa257fa640 1 -- 172.21.15.89:0/925734063 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7faa2000f5f0 con 0x7faa28102ef0 2024-09-15T23:37:36.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.372+0000 7faa257fa640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa100a32c0 0x7faa100a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:36.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.373+0000 7faa257fa640 1 -- 172.21.15.89:0/925734063 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7faa200d0290 con 0x7faa28102ef0 2024-09-15T23:37:36.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.373+0000 7faa27fff640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa100a32c0 0x7faa100a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:36.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.373+0000 7faa27fff640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa100a32c0 0x7faa100a5780 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faa18005d00 tx=0x7faa1800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:36.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.376+0000 7faa257fa640 1 -- 172.21.15.89:0/925734063 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7faa20014030 con 0x7faa28102ef0 2024-09-15T23:37:36.390 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:36.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.586+0000 7faa2e64f640 1 -- 172.21.15.89:0/925734063 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa28101200 con 0x7faa28102ef0 2024-09-15T23:37:36.588 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.587+0000 7faa257fa640 1 -- 172.21.15.89:0/925734063 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7faa28101200 con 0x7faa28102ef0 2024-09-15T23:37:36.589 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:37:36.589 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:37:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:37:36.590 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 -- 172.21.15.89:0/925734063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa100a32c0 msgr2=0x7faa100a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa100a32c0 0x7faa100a5780 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faa18005d00 tx=0x7faa1800b410 comp rx=0 tx=0).stop 2024-09-15T23:37:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 -- 172.21.15.89:0/925734063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa28102ef0 msgr2=0x7faa2819cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa28102ef0 0x7faa2819cdf0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7faa2000cce0 tx=0x7faa2000a620 comp rx=0 tx=0).stop 2024-09-15T23:37:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 -- 172.21.15.89:0/925734063 shutdown_connections 2024-09-15T23:37:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7faa100a32c0 0x7faa100a5780 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faa28102ef0 0x7faa2819cdf0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.591 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faa28102570 0x7faa28198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 --2- 172.21.15.89:0/925734063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa28100e20 0x7faa28198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:36.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 -- 172.21.15.89:0/925734063 >> 172.21.15.89:0/925734063 conn(0x7faa280fc7d0 msgr2=0x7faa2806a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:36.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 -- 172.21.15.89:0/925734063 shutdown_connections 2024-09-15T23:37:36.592 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:36.589+0000 7faa2e64f640 1 -- 172.21.15.89:0/925734063 wait complete. 2024-09-15T23:37:37.195 INFO:tasks.workunit.client.0.smithi089.stdout:437/600: open fd = 3 2024-09-15T23:37:37.195 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.195 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.195 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.195 INFO:tasks.workunit.client.0.smithi089.stdout:438/600: open fd = 3 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:439/600: open fd = 3 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:440/600: open fd = 3 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:441/600: open fd = 3 2024-09-15T23:37:37.196 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:442/600: open fd = 3 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:443/600: open fd = 3 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:444/600: open fd = 3 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.197 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:445/600: open fd = 3 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:446/600: open fd = 3 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:447/600: open fd = 3 2024-09-15T23:37:37.198 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:448/600: open fd = 3 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:449/600: open fd = 3 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:450/600: open fd = 3 2024-09-15T23:37:37.199 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:451/600: open fd = 3 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:452/600: open fd = 3 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:453/600: open fd = 3 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.200 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:454/600: open fd = 3 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:455/600: open fd = 3 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:456/600: open fd = 3 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.201 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:457/600: open fd = 3 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:458/600: open fd = 3 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:459/600: open fd = 3 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:460/600: open fd = 3 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.202 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:461/600: open fd = 3 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:462/600: open fd = 3 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:463/600: open fd = 3 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.203 INFO:tasks.workunit.client.0.smithi089.stdout:464/600: open fd = 3 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:465/600: open fd = 3 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:466/600: open fd = 3 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:467/600: open fd = 3 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.204 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:468/600: open fd = 3 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:469/600: open fd = 3 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:470/600: open fd = 3 2024-09-15T23:37:37.205 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:471/600: open fd = 3 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:472/600: open fd = 3 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:473/600: open fd = 3 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.206 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:474/600: open fd = 3 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:475/600: open fd = 3 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:476/600: open fd = 3 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:477/600: open fd = 3 2024-09-15T23:37:37.207 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:478/600: open fd = 3 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:479/600: open fd = 3 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:480/600: open fd = 3 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:481/600: open fd = 3 2024-09-15T23:37:37.208 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.209 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.209 INFO:tasks.workunit.client.0.smithi089.stdout:482/600: open fd = 3 2024-09-15T23:37:37.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.209 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.209 INFO:tasks.workunit.client.0.smithi089.stdout:483/600: open fd = 3 2024-09-15T23:37:37.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:484/600: open fd = 3 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:485/600: open fd = 3 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:486/600: open fd = 3 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.210 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:487/600: open fd = 3 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:488/600: open fd = 3 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:489/600: open fd = 3 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:490/600: open fd = 3 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:491/600: open fd = 3 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:492/600: open fd = 3 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:493/600: open fd = 3 2024-09-15T23:37:37.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:494/600: open fd = 3 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:495/600: open fd = 3 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:496/600: open fd = 3 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.213 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.214 INFO:tasks.workunit.client.0.smithi089.stdout:497/600: open fd = 3 2024-09-15T23:37:37.214 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.214 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.214 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.214 INFO:tasks.workunit.client.0.smithi089.stdout:498/600: open fd = 3 2024-09-15T23:37:37.214 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.214 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:37:37.214 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:37:37.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:36 smithi120 ceph-mon[35250]: pgmap v757: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:37:37.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:36 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/925734063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:37.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:36 smithi174 ceph-mon[35586]: pgmap v757: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:37:37.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:36 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/925734063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:37.243 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:36 smithi089 ceph-mon[30933]: pgmap v757: 129 pgs: 129 active+clean; 2.8 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 16 op/s 2024-09-15T23:37:37.244 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:36 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/925734063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:37.273 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 131.4 --await-for 45.0' 2024-09-15T23:37:37.273 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 131.4 --await-for 45.0 2024-09-15T23:37:37.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.486+0000 7f8e9ec17640 1 -- 172.21.15.89:0/3454321721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e98100e20 msgr2=0x7f8e98101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:37.514 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.486+0000 7f8e9ec17640 1 --2- 172.21.15.89:0/3454321721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e98100e20 0x7f8e98101200 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009900 tx=0x7f8e9002a480 comp rx=0 tx=0).stop 2024-09-15T23:37:37.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.486+0000 7f8e9ec17640 1 -- 172.21.15.89:0/3454321721 shutdown_connections 2024-09-15T23:37:37.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.486+0000 7f8e9ec17640 1 --2- 172.21.15.89:0/3454321721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e98102ef0 0x7f8e9810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.486+0000 7f8e9ec17640 1 --2- 172.21.15.89:0/3454321721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e98102570 0x7f8e981029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.486+0000 7f8e9ec17640 1 --2- 172.21.15.89:0/3454321721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e98100e20 0x7f8e98101200 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.486+0000 7f8e9ec17640 1 -- 172.21.15.89:0/3454321721 >> 172.21.15.89:0/3454321721 conn(0x7f8e980fc7d0 msgr2=0x7f8e980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:37.515 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.486+0000 7f8e9ec17640 1 -- 172.21.15.89:0/3454321721 shutdown_connections 2024-09-15T23:37:37.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e9ec17640 1 -- 172.21.15.89:0/3454321721 wait complete. 2024-09-15T23:37:37.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e9ec17640 1 Processor -- start 2024-09-15T23:37:37.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e9ec17640 1 -- start start 2024-09-15T23:37:37.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e9ec17640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e98100e20 0x7f8e98199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.516 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e9ec17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e98102570 0x7f8e98199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e9ec17640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e98102ef0 0x7f8e981a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e9ec17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8e98110fa0 con 0x7f8e98102570 2024-09-15T23:37:37.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e9ec17640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8e98110e20 con 0x7f8e98100e20 2024-09-15T23:37:37.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e9ec17640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8e98111120 con 0x7f8e98102ef0 2024-09-15T23:37:37.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e97fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e98102570 0x7f8e98199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e97fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e98102570 0x7f8e98199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44596/0 (socket says 172.21.15.89:44596) 2024-09-15T23:37:37.517 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.487+0000 7f8e97fff640 1 -- 172.21.15.89:0/870704675 learned_addr learned my addr 172.21.15.89:0/870704675 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:37.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e9c9b3640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e98100e20 0x7f8e98199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e97fff640 1 -- 172.21.15.89:0/870704675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e98102ef0 msgr2=0x7f8e981a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:37.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e9d1b4640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e98102ef0 0x7f8e981a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e97fff640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e98102ef0 0x7f8e981a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e97fff640 1 -- 172.21.15.89:0/870704675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e98100e20 msgr2=0x7f8e98199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:37.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e97fff640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e98100e20 0x7f8e98199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e97fff640 1 -- 172.21.15.89:0/870704675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e981a48f0 con 0x7f8e98102570 2024-09-15T23:37:37.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e9c9b3640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e98100e20 0x7f8e98199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:37.518 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e9d1b4640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e98102ef0 0x7f8e981a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:37.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e97fff640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e98102570 0x7f8e98199870 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800cce0 tx=0x7f8e8800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:37.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e95ffb640 1 -- 172.21.15.89:0/870704675 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8800f650 con 0x7f8e98102570 2024-09-15T23:37:37.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e95ffb640 1 -- 172.21.15.89:0/870704675 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e8800fcd0 con 0x7f8e98102570 2024-09-15T23:37:37.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e9ec17640 1 -- 172.21.15.89:0/870704675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e981a4ac0 con 0x7f8e98102570 2024-09-15T23:37:37.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e95ffb640 1 -- 172.21.15.89:0/870704675 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e88018c70 con 0x7f8e98102570 2024-09-15T23:37:37.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.488+0000 7f8e9ec17640 1 -- 172.21.15.89:0/870704675 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e981a4fd0 con 0x7f8e98102570 2024-09-15T23:37:37.519 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.491+0000 7f8e95ffb640 1 -- 172.21.15.89:0/870704675 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8e8800f7f0 con 0x7f8e98102570 2024-09-15T23:37:37.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.491+0000 7f8ea0ff0740 1 Processor -- start 2024-09-15T23:37:37.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8ea0ff0740 1 -- start start 2024-09-15T23:37:37.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955eb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x5571955f6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955f8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5571955512e0 con 0x5571955eb4a0 2024-09-15T23:37:37.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5571954f8ce0 con 0x5571955ebdc0 2024-09-15T23:37:37.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55719554d2a0 con 0x5571955f6680 2024-09-15T23:37:37.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e95ffb640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8e840a3240 0x7f8e840a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e9d1b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955eb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e9d1b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955eb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44598/0 (socket says 172.21.15.89:44598) 2024-09-15T23:37:37.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e9d1b4640 1 -- 172.21.15.89:0/911284415 learned_addr learned my addr 172.21.15.89:0/911284415 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:37.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e9d1b4640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8e840a3240 0x7f8e840a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e97fff640 1 --2- 172.21.15.89:0/911284415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955f8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e9c9b3640 1 --2- 172.21.15.89:0/911284415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x5571955f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e95ffb640 1 -- 172.21.15.89:0/870704675 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8e880d0260 con 0x7f8e98102570 2024-09-15T23:37:37.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e9d1b4640 1 -- 172.21.15.89:0/911284415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 msgr2=0x5571955f8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:37.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e9d1b4640 1 --2- 172.21.15.89:0/911284415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955f8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e9d1b4640 1 -- 172.21.15.89:0/911284415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 msgr2=0x5571955f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:37.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e9d1b4640 1 --2- 172.21.15.89:0/911284415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x5571955f6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.492+0000 7f8e9d1b4640 1 -- 172.21.15.89:0/911284415 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571954dd440 con 0x5571955eb4a0 2024-09-15T23:37:37.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8e9d1b4640 1 --2- 172.21.15.89:0/911284415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955eb880 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f8e80015760 tx=0x7f8e80015c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12059a511396628d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:37.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8e9d1b4640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8e840a3240 0x7f8e840a5700 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8e80013650 tx=0x7f8e8001a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:37.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8e8effd640 1 -- 172.21.15.89:0/911284415 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8000f040 con 0x5571955eb4a0 2024-09-15T23:37:37.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8e8effd640 1 -- 172.21.15.89:0/911284415 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e8000c700 con 0x5571955eb4a0 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8e8effd640 1 -- 172.21.15.89:0/911284415 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80020720 con 0x5571955eb4a0 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/911284415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 msgr2=0x5571955eb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/911284415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955eb880 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f8e80015760 tx=0x7f8e80015c20 comp rx=0 tx=0).stop 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/911284415 shutdown_connections 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/911284415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955f8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/911284415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x5571955f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/911284415 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955eb880 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.493+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/911284415 >> 172.21.15.89:0/911284415 conn(0x5571955e7a30 msgr2=0x5571955e7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/911284415 shutdown_connections 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/911284415 wait complete. 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 1 -- start start 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 1 Processor -- start 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955f6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x5571955f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955f3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5571955f92a0 con 0x5571955eb4a0 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55719550c050 con 0x5571955ebdc0 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5571955f9420 con 0x5571955f6680 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.494+0000 7f8ea0ff0740 10 client.0 _ll_register_callbacks cb 0x55719554c160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e9c9b3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e9c9b3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955f6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44614/0 (socket says 172.21.15.89:44614) 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e9c9b3640 1 -- 172.21.15.89:0/1590364981 learned_addr learned my addr 172.21.15.89:0/1590364981 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e97fff640 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x5571955f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e9d1b4640 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e9c9b3640 1 -- 172.21.15.89:0/1590364981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 msgr2=0x5571955f3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e9c9b3640 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955f3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e9c9b3640 1 -- 172.21.15.89:0/1590364981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 msgr2=0x5571955f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e9c9b3640 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x5571955f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e9c9b3640 1 -- 172.21.15.89:0/1590364981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571952c23c0 con 0x5571955eb4a0 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e9c9b3640 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955f6000 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f8e900314e0 tx=0x7f8e900052d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f0196d3c6d8022 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e77fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e77fff640 1 -- 172.21.15.89:0/1590364981 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e90035030 con 0x5571955eb4a0 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/1590364981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e8000c700 con 0x5571955eb4a0 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e77fff640 1 -- 172.21.15.89:0/1590364981 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8e900048c0 con 0x5571955eb4a0 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8e77fff640 1 -- 172.21.15.89:0/1590364981 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8e9004c050 con 0x5571955eb4a0 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8ea0ff0740 10 client.20187 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.495+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/1590364981 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557195604d10 con 0x5571955eb4a0 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8e77fff640 1 -- 172.21.15.89:0/1590364981 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e8000c700 con 0x5571955eb4a0 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8e77fff640 1 -- 172.21.15.89:0/1590364981 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8e90003c70 con 0x5571955eb4a0 2024-09-15T23:37:37.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8ea0ff0740 10 client.20187 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8ea0ff0740 10 client.20187 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8ea0ff0740 20 client.20187 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557195606120 0x557195608510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8ea0ff0740 4 client.20187 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/1590364981 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55719555e850 con 0x557195606120 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8e97fff640 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557195606120 0x557195608510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8e77fff640 10 client.20187 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.496+0000 7f8e97fff640 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557195606120 0x557195608510 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=735acf26d04294d8 server_cookie=7a53f5e3531a43fc in_seq=0 out_seq=0 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8e77fff640 1 -- 172.21.15.89:0/1590364981 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55719555e850 con 0x557195606120 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8e77fff640 10 client.20187 handle_command_reply: tid=0 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 client.20187 shutdown 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/1590364981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 msgr2=0x5571955f6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955f6000 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f8e900314e0 tx=0x7f8e900052d0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/1590364981 shutdown_connections 2024-09-15T23:37:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557195606120 0x557195608510 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955f3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x5571955f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8e77fff640 0 client.20187 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/1590364981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955eb4a0 0x5571955f6000 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/1590364981 >> 172.21.15.89:0/1590364981 conn(0x5571955e7a30 msgr2=0x5571955e6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/1590364981 shutdown_connections 2024-09-15T23:37:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.497+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/1590364981 wait complete. 2024-09-15T23:37:37.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.498+0000 7f8ea0ff0740 20 client.20187 trim_cache size 0 max 16384 2024-09-15T23:37:37.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8ea0ff0740 1 Processor -- start 2024-09-15T23:37:37.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8ea0ff0740 1 -- start start 2024-09-15T23:37:37.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955eb4a0 0x557195602530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x55719560dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955f6680 0x55719560e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8e981aeea0 con 0x5571955f6680 2024-09-15T23:37:37.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8e981b09e0 con 0x5571955ebdc0 2024-09-15T23:37:37.532 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8e981af570 con 0x5571955eb4a0 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8e9d1b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955f6680 0x55719560e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8e97fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x55719560dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8e97fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x55719560dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:47896/0 (socket says 172.21.15.89:47896) 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8e97fff640 1 -- 172.21.15.89:0/783814895 learned_addr learned my addr 172.21.15.89:0/783814895 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8e9c9b3640 1 --2- 172.21.15.89:0/783814895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955eb4a0 0x557195602530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8e97fff640 1 -- 172.21.15.89:0/783814895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955eb4a0 msgr2=0x557195602530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8e97fff640 1 --2- 172.21.15.89:0/783814895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955eb4a0 0x557195602530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8e97fff640 1 -- 172.21.15.89:0/783814895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955f6680 msgr2=0x55719560e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.515+0000 7f8e97fff640 1 --2- 172.21.15.89:0/783814895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955f6680 0x55719560e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.516+0000 7f8e97fff640 1 -- 172.21.15.89:0/783814895 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571955f3ba0 con 0x5571955ebdc0 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.516+0000 7f8e9d1b4640 1 --2- 172.21.15.89:0/783814895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955f6680 0x55719560e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:37:37.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.516+0000 7f8e97fff640 1 --2- 172.21.15.89:0/783814895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x55719560dde0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8e880186d0 tx=0x7f8e88013b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a75324dfe9ec0ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.516+0000 7f8e8e7fc640 1 -- 172.21.15.89:0/783814895 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e880d00b0 con 0x5571955ebdc0 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.516+0000 7f8e8e7fc640 1 -- 172.21.15.89:0/783814895 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e8809cb90 con 0x5571955ebdc0 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.516+0000 7f8e8e7fc640 1 -- 172.21.15.89:0/783814895 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e88018e10 con 0x5571955ebdc0 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.517+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/783814895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 msgr2=0x55719560dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.517+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/783814895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x55719560dde0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8e880186d0 tx=0x7f8e88013b80 comp rx=0 tx=0).stop 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.517+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/783814895 shutdown_connections 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.517+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/783814895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955f6680 0x55719560e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.517+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/783814895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955ebdc0 0x55719560dde0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.517+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/783814895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955eb4a0 0x557195602530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.517+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/783814895 >> 172.21.15.89:0/783814895 conn(0x5571955e7a30 msgr2=0x5571955f1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.517+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/783814895 shutdown_connections 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.517+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/783814895 wait complete. 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.517+0000 7f8ea0ff0740 1 -- start start 2024-09-15T23:37:37.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8ea0ff0740 1 Processor -- start 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955eb4a0 0x5571955fa750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955ebdc0 0x5571955fac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8ea0ff0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955fb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8e981b0310 con 0x5571955ebdc0 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8e981afc40 con 0x5571955eb4a0 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8ea0ff0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8e981ace80 con 0x5571955f6680 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8ea0ff0740 10 client.0 _ll_register_callbacks cb 0x7f8e90009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8e9c9b3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955eb4a0 0x5571955fa750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8e97fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955ebdc0 0x5571955fac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8e9c9b3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955eb4a0 0x5571955fa750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:47902/0 (socket says 172.21.15.89:47902) 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.518+0000 7f8e9c9b3640 1 -- 172.21.15.89:0/399700431 learned_addr learned my addr 172.21.15.89:0/399700431 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e9d1b4640 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955fb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e9c9b3640 1 -- 172.21.15.89:0/399700431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 msgr2=0x5571955fb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:37.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e9c9b3640 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955fb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e9c9b3640 1 -- 172.21.15.89:0/399700431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955ebdc0 msgr2=0x5571955fac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e9c9b3640 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955ebdc0 0x5571955fac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e9c9b3640 1 -- 172.21.15.89:0/399700431 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571956700a0 con 0x5571955eb4a0 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e9c9b3640 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955eb4a0 0x5571955fa750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e900314b0 tx=0x7f8e90004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aae5904a5f349186 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e77fff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e77fff640 1 -- 172.21.15.89:0/399700431 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e90054030 con 0x5571955eb4a0 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e77fff640 1 -- 172.21.15.89:0/399700431 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8e90049070 con 0x5571955eb4a0 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/399700431 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e8809cb90 con 0x5571955eb4a0 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8e77fff640 1 -- 172.21.15.89:0/399700431 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8e9005e050 con 0x5571955eb4a0 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.519+0000 7f8ea0ff0740 10 client.28046 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:37.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.520+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/399700431 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5571956713b0 con 0x5571955eb4a0 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.520+0000 7f8e77fff640 1 -- 172.21.15.89:0/399700431 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e8809cb90 con 0x5571955eb4a0 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.520+0000 7f8e77fff640 1 -- 172.21.15.89:0/399700431 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8e900035f0 con 0x5571955eb4a0 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.520+0000 7f8ea0ff0740 10 client.28046 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.520+0000 7f8ea0ff0740 10 client.28046 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.520+0000 7f8ea0ff0740 20 client.28046 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.520+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557195671b20 0x557195673f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.520+0000 7f8ea0ff0740 4 client.28046 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.520+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/399700431 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557195558d60 con 0x557195671b20 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.520+0000 7f8e97fff640 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557195671b20 0x557195673f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8e97fff640 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557195671b20 0x557195673f40 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e8758afd5837ce8 server_cookie=fe3a799569227ca0 in_seq=0 out_seq=0 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8e77fff640 10 client.28046 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:37.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8e77fff640 1 -- 172.21.15.89:0/399700431 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557195558d60 con 0x557195671b20 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8e77fff640 10 client.28046 handle_command_reply: tid=0 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 client.28046 shutdown 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/399700431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955eb4a0 msgr2=0x5571955fa750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955eb4a0 0x5571955fa750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e900314b0 tx=0x7f8e90004610 comp rx=0 tx=0).stop 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/399700431 shutdown_connections 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x557195671b20 0x557195673f40 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5571955f6680 0x5571955fb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5571955ebdc0 0x5571955fac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8e77fff640 0 client.28046 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 --2- 172.21.15.89:0/399700431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571955eb4a0 0x5571955fa750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/399700431 >> 172.21.15.89:0/399700431 conn(0x5571955e7a30 msgr2=0x5571955edf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/399700431 shutdown_connections 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 1 -- 172.21.15.89:0/399700431 wait complete. 2024-09-15T23:37:37.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.521+0000 7f8ea0ff0740 20 client.28046 trim_cache size 0 max 16384 2024-09-15T23:37:37.539 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:37:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:37:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:37:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "7e71e9a7-418d-462f-8c5e-82b86e51bea5": { 2024-09-15T23:37:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:37:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "7e71e9a7-418d-462f-8c5e-82b86e51bea5", 2024-09-15T23:37:37.539 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:37:37.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:37.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:37.540 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:37:37.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 -- 172.21.15.89:0/870704675 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8e840a3240 msgr2=0x7f8e840a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:37.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8e840a3240 0x7f8e840a5700 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8e80013650 tx=0x7f8e8001a000 comp rx=0 tx=0).stop 2024-09-15T23:37:37.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 -- 172.21.15.89:0/870704675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e98102570 msgr2=0x7f8e98199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:37.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e98102570 0x7f8e98199870 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f8e8800cce0 tx=0x7f8e8800a620 comp rx=0 tx=0).stop 2024-09-15T23:37:37.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 -- 172.21.15.89:0/870704675 shutdown_connections 2024-09-15T23:37:37.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8e840a3240 0x7f8e840a5700 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e98102ef0 0x7f8e981a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8e98102570 0x7f8e98199870 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 --2- 172.21.15.89:0/870704675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e98100e20 0x7f8e98199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:37.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 -- 172.21.15.89:0/870704675 >> 172.21.15.89:0/870704675 conn(0x7f8e980fc7d0 msgr2=0x7f8e98106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:37.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 -- 172.21.15.89:0/870704675 shutdown_connections 2024-09-15T23:37:37.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:37.522+0000 7f8e9ec17640 1 -- 172.21.15.89:0/870704675 wait complete. 2024-09-15T23:37:38.420 INFO:tasks.cephfs.filesystem:scrub status for tag:7e71e9a7-418d-462f-8c5e-82b86e51bea5 - {'path': '/', 'tag': '7e71e9a7-418d-462f-8c5e-82b86e51bea5', 'options': 'recursive,force'} 2024-09-15T23:37:38.442 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:38.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.868+0000 7ff86d930640 1 -- 172.21.15.89:0/1002549702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff868102ef0 msgr2=0x7ff86810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:38.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.868+0000 7ff86d930640 1 --2- 172.21.15.89:0/1002549702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff868102ef0 0x7ff86810da40 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7ff860009ca0 tx=0x7ff86002a950 comp rx=0 tx=0).stop 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.868+0000 7ff86d930640 1 -- 172.21.15.89:0/1002549702 shutdown_connections 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.868+0000 7ff86d930640 1 --2- 172.21.15.89:0/1002549702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff868102ef0 0x7ff86810da40 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.868+0000 7ff86d930640 1 --2- 172.21.15.89:0/1002549702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff868102570 0x7ff8681029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.868+0000 7ff86d930640 1 --2- 172.21.15.89:0/1002549702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff868100e20 0x7ff868101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.868+0000 7ff86d930640 1 -- 172.21.15.89:0/1002549702 >> 172.21.15.89:0/1002549702 conn(0x7ff8680fc7d0 msgr2=0x7ff8680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.868+0000 7ff86d930640 1 -- 172.21.15.89:0/1002549702 shutdown_connections 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.868+0000 7ff86d930640 1 -- 172.21.15.89:0/1002549702 wait complete. 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.868+0000 7ff86d930640 1 Processor -- start 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff86d930640 1 -- start start 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff86d930640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff868100e20 0x7ff8681973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff86d930640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff868102570 0x7ff868197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff86d930640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff868102ef0 0x7ff86819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff86d930640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff868110fc0 con 0x7ff868100e20 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff86d930640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff868110e40 con 0x7ff868102ef0 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff86d930640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff868111140 con 0x7ff868102570 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff866ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff868100e20 0x7ff8681973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff866ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff868100e20 0x7ff8681973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44634/0 (socket says 172.21.15.89:44634) 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff866ffd640 1 -- 172.21.15.89:0/2330092566 learned_addr learned my addr 172.21.15.89:0/2330092566 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff8667fc640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff868102570 0x7ff868197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff8677fe640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff868102ef0 0x7ff86819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff866ffd640 1 -- 172.21.15.89:0/2330092566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff868102570 msgr2=0x7ff868197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff866ffd640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff868102570 0x7ff868197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff866ffd640 1 -- 172.21.15.89:0/2330092566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff868102ef0 msgr2=0x7ff86819bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff866ffd640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff868102ef0 0x7ff86819bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff866ffd640 1 -- 172.21.15.89:0/2330092566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86819c390 con 0x7ff868100e20 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.869+0000 7ff8677fe640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff868102ef0 0x7ff86819bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:38.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.870+0000 7ff866ffd640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff868100e20 0x7ff8681973e0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7ff85800b4a0 tx=0x7ff85800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.870+0000 7ff86c92e640 1 -- 172.21.15.89:0/2330092566 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858014070 con 0x7ff868100e20 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.870+0000 7ff86c92e640 1 -- 172.21.15.89:0/2330092566 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff858010af0 con 0x7ff868100e20 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.870+0000 7ff86d930640 1 -- 172.21.15.89:0/2330092566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff86819c680 con 0x7ff868100e20 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.870+0000 7ff86c92e640 1 -- 172.21.15.89:0/2330092566 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858017d10 con 0x7ff868100e20 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.870+0000 7ff86d930640 1 -- 172.21.15.89:0/2330092566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8681a3f60 con 0x7ff868100e20 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.872+0000 7ff86c92e640 1 -- 172.21.15.89:0/2330092566 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff858017070 con 0x7ff868100e20 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.872+0000 7ff86fd09740 1 Processor -- start 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.872+0000 7ff86fd09740 1 -- start start 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 0x5596d7de0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de1080 0x5596d7deb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7deb940 0x5596d7dedd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff86fd09740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5596d7d462a0 con 0x5596d7de1080 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff86fd09740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5596d7cedb10 con 0x5596d7de0760 2024-09-15T23:37:38.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff86fd09740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5596d7d42260 con 0x5596d7deb940 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff866ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de1080 0x5596d7deb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff866ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de1080 0x5596d7deb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44638/0 (socket says 172.21.15.89:44638) 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff866ffd640 1 -- 172.21.15.89:0/4093190897 learned_addr learned my addr 172.21.15.89:0/4093190897 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff8667fc640 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7deb940 0x5596d7dedd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff8677fe640 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 0x5596d7de0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff86c92e640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8540a3240 0x7ff8540a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff866ffd640 1 -- 172.21.15.89:0/4093190897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7deb940 msgr2=0x5596d7dedd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff866ffd640 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7deb940 0x5596d7dedd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff866ffd640 1 -- 172.21.15.89:0/4093190897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 msgr2=0x5596d7de0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff866ffd640 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 0x5596d7de0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff866ffd640 1 -- 172.21.15.89:0/4093190897 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596d7ab7b90 con 0x5596d7de1080 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff8667fc640 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7deb940 0x5596d7dedd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff8677fe640 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 0x5596d7de0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:38.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff8677fe640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8540a3240 0x7ff8540a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff86c92e640 1 -- 172.21.15.89:0/2330092566 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff8580cf070 con 0x7ff868100e20 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.873+0000 7ff866ffd640 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de1080 0x5596d7deb400 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7ff8580c8930 tx=0x7ff8580c6070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9c2b638d5918fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff85dffb640 1 -- 172.21.15.89:0/4093190897 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85801e070 con 0x5596d7de1080 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff85dffb640 1 -- 172.21.15.89:0/4093190897 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8580c6d00 con 0x5596d7de1080 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff85dffb640 1 -- 172.21.15.89:0/4093190897 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85809b070 con 0x5596d7de1080 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff86fd09740 1 -- 172.21.15.89:0/4093190897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de1080 msgr2=0x5596d7deb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff8677fe640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8540a3240 0x7ff8540a5700 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff860005cc0 tx=0x7ff8600095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff86fd09740 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de1080 0x5596d7deb400 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7ff8580c8930 tx=0x7ff8580c6070 comp rx=0 tx=0).stop 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff86fd09740 1 -- 172.21.15.89:0/4093190897 shutdown_connections 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff86fd09740 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7deb940 0x5596d7dedd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff86fd09740 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de1080 0x5596d7deb400 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff86fd09740 1 --2- 172.21.15.89:0/4093190897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 0x5596d7de0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff86fd09740 1 -- 172.21.15.89:0/4093190897 >> 172.21.15.89:0/4093190897 conn(0x5596d7ddccf0 msgr2=0x5596d7ddd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff86fd09740 1 -- 172.21.15.89:0/4093190897 shutdown_connections 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.874+0000 7ff86fd09740 1 -- 172.21.15.89:0/4093190897 wait complete. 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff86fd09740 1 -- start start 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff86fd09740 1 Processor -- start 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de0760 0x5596d7de8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 0x5596d7de8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7de92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff86fd09740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5596d7dee510 con 0x5596d7deb940 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff86fd09740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5596d7cce6e0 con 0x5596d7de1080 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff86fd09740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5596d7dee690 con 0x5596d7de0760 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff86fd09740 10 client.0 _ll_register_callbacks cb 0x5596d7cf2a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff8677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7de92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff8677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7de92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44648/0 (socket says 172.21.15.89:44648) 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff8677fe640 1 -- 172.21.15.89:0/3196786014 learned_addr learned my addr 172.21.15.89:0/3196786014 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.875+0000 7ff866ffd640 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de0760 0x5596d7de8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff8667fc640 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 0x5596d7de8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff8677fe640 1 -- 172.21.15.89:0/3196786014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de0760 msgr2=0x5596d7de8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff8677fe640 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de0760 0x5596d7de8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff8677fe640 1 -- 172.21.15.89:0/3196786014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 msgr2=0x5596d7de8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff8677fe640 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 0x5596d7de8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff8677fe640 1 -- 172.21.15.89:0/3196786014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596d7df7a30 con 0x5596d7deb940 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff8667fc640 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 0x5596d7de8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff866ffd640 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de0760 0x5596d7de8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff8677fe640 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7de92c0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7ff86002f5b0 tx=0x7ff860005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43fccd2c043f413 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff846ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff846ffd640 1 -- 172.21.15.89:0/3196786014 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff860034040 con 0x5596d7deb940 2024-09-15T23:37:38.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff846ffd640 1 -- 172.21.15.89:0/3196786014 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff860004040 con 0x5596d7deb940 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff86fd09740 1 -- 172.21.15.89:0/3196786014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5596d7d539d0 con 0x5596d7deb940 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff846ffd640 1 -- 172.21.15.89:0/3196786014 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff860059050 con 0x5596d7deb940 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff86fd09740 10 client.20205 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff86fd09740 1 -- 172.21.15.89:0/3196786014 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596d7dfa310 con 0x5596d7deb940 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff846ffd640 1 -- 172.21.15.89:0/3196786014 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5596d7d539d0 con 0x5596d7deb940 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff846ffd640 1 -- 172.21.15.89:0/3196786014 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff860030a10 con 0x5596d7deb940 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff86fd09740 10 client.20205 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff86fd09740 10 client.20205 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.876+0000 7ff86fd09740 20 client.20205 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.877+0000 7ff86fd09740 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5596d7dfb820 0x5596d7dfdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.877+0000 7ff86fd09740 4 client.20205 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.877+0000 7ff86fd09740 1 -- 172.21.15.89:0/3196786014 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596d7d4dd20 con 0x5596d7dfb820 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.877+0000 7ff866ffd640 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5596d7dfb820 0x5596d7dfdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.888 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.877+0000 7ff866ffd640 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5596d7dfb820 0x5596d7dfdca0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1669eb06ade31f6 server_cookie=572ae27d244d10ca in_seq=0 out_seq=0 2024-09-15T23:37:38.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.877+0000 7ff846ffd640 10 client.20205 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:38.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff846ffd640 1 -- 172.21.15.89:0/3196786014 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5596d7d4dd20 con 0x5596d7dfb820 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff846ffd640 10 client.20205 handle_command_reply: tid=0 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 client.20205 shutdown 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 -- 172.21.15.89:0/3196786014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 msgr2=0x5596d7de92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7de92c0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7ff86002f5b0 tx=0x7ff860005c30 comp rx=0 tx=0).stop 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 -- 172.21.15.89:0/3196786014 shutdown_connections 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5596d7dfb820 0x5596d7dfdca0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7de92c0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 0x5596d7de8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 --2- 172.21.15.89:0/3196786014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de0760 0x5596d7de8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff846ffd640 0 client.20205 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 -- 172.21.15.89:0/3196786014 >> 172.21.15.89:0/3196786014 conn(0x5596d7ddccf0 msgr2=0x5596d7de2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 -- 172.21.15.89:0/3196786014 shutdown_connections 2024-09-15T23:37:38.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 1 -- 172.21.15.89:0/3196786014 wait complete. 2024-09-15T23:37:38.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.878+0000 7ff86fd09740 20 client.20205 trim_cache size 0 max 16384 2024-09-15T23:37:38.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff86fd09740 1 Processor -- start 2024-09-15T23:37:38.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff86fd09740 1 -- start start 2024-09-15T23:37:38.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de0760 0x5596d7dea710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 0x5596d7de7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.896 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7deb940 0x5596d7df3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff86fd09740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff8681addf0 con 0x5596d7de0760 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff86fd09740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8681af930 con 0x5596d7de1080 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff86fd09740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8681ae4c0 con 0x5596d7deb940 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff866ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de0760 0x5596d7dea710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff866ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de0760 0x5596d7dea710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44664/0 (socket says 172.21.15.89:44664) 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff866ffd640 1 -- 172.21.15.89:0/3309256808 learned_addr learned my addr 172.21.15.89:0/3309256808 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff866ffd640 1 -- 172.21.15.89:0/3309256808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7deb940 msgr2=0x5596d7df3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff8667fc640 1 --2- 172.21.15.89:0/3309256808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 0x5596d7de7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.895+0000 7ff866ffd640 1 --2- 172.21.15.89:0/3309256808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7deb940 0x5596d7df3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.896+0000 7ff866ffd640 1 -- 172.21.15.89:0/3309256808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 msgr2=0x5596d7de7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.896+0000 7ff866ffd640 1 --2- 172.21.15.89:0/3309256808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 0x5596d7de7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.896+0000 7ff866ffd640 1 -- 172.21.15.89:0/3309256808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596d7de0eb0 con 0x5596d7de0760 2024-09-15T23:37:38.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.896+0000 7ff866ffd640 1 --2- 172.21.15.89:0/3309256808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de0760 0x5596d7dea710 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7ff8580cc7d0 tx=0x7ff8580bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bfcccca73f22a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:38.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.896+0000 7ff85d7fa640 1 -- 172.21.15.89:0/3309256808 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85801e040 con 0x5596d7de0760 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.896+0000 7ff85d7fa640 1 -- 172.21.15.89:0/3309256808 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff85809b9a0 con 0x5596d7de0760 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.896+0000 7ff85d7fa640 1 -- 172.21.15.89:0/3309256808 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85809ac00 con 0x5596d7de0760 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.896+0000 7ff86fd09740 1 -- 172.21.15.89:0/3309256808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de0760 msgr2=0x5596d7dea710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.896+0000 7ff86fd09740 1 --2- 172.21.15.89:0/3309256808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de0760 0x5596d7dea710 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7ff8580cc7d0 tx=0x7ff8580bb040 comp rx=0 tx=0).stop 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 -- 172.21.15.89:0/3309256808 shutdown_connections 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 --2- 172.21.15.89:0/3309256808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7deb940 0x5596d7df3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 --2- 172.21.15.89:0/3309256808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de1080 0x5596d7de7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 --2- 172.21.15.89:0/3309256808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7de0760 0x5596d7dea710 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 -- 172.21.15.89:0/3309256808 >> 172.21.15.89:0/3309256808 conn(0x5596d7ddccf0 msgr2=0x5596d7ddbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 -- 172.21.15.89:0/3309256808 shutdown_connections 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 -- 172.21.15.89:0/3309256808 wait complete. 2024-09-15T23:37:38.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 -- start start 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 Processor -- start 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 0x5596d7de5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de1080 0x5596d7de6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.897+0000 7ff86fd09740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7e61610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff86fd09740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff8681af260 con 0x5596d7deb940 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff86fd09740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8681aeb90 con 0x5596d7de0760 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff86fd09740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8681abdd0 con 0x5596d7de1080 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff8677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7e61610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff8677fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7e61610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44670/0 (socket says 172.21.15.89:44670) 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff8677fe640 1 -- 172.21.15.89:0/646107780 learned_addr learned my addr 172.21.15.89:0/646107780 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff86fd09740 10 client.0 _ll_register_callbacks cb 0x7ff86002f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff8667fc640 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de1080 0x5596d7de6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff866ffd640 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 0x5596d7de5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff8677fe640 1 -- 172.21.15.89:0/646107780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de1080 msgr2=0x5596d7de6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff8677fe640 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de1080 0x5596d7de6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff8677fe640 1 -- 172.21.15.89:0/646107780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 msgr2=0x5596d7de5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff8677fe640 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 0x5596d7de5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff8677fe640 1 -- 172.21.15.89:0/646107780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596d7e61cd0 con 0x5596d7deb940 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff8677fe640 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7e61610 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7ff86002ae70 tx=0x7ff860047660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec8b9d81d20e9e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff846ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff846ffd640 1 -- 172.21.15.89:0/646107780 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff860062060 con 0x5596d7deb940 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff846ffd640 1 -- 172.21.15.89:0/646107780 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff860047c60 con 0x5596d7deb940 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.898+0000 7ff86fd09740 1 -- 172.21.15.89:0/646107780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff85809b9a0 con 0x5596d7deb940 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff846ffd640 1 -- 172.21.15.89:0/646107780 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff860069050 con 0x5596d7deb940 2024-09-15T23:37:38.903 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff86fd09740 10 client.20211 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff86fd09740 1 -- 172.21.15.89:0/646107780 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596d7e69110 con 0x5596d7deb940 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff846ffd640 1 -- 172.21.15.89:0/646107780 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff85809b9a0 con 0x5596d7deb940 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff846ffd640 1 -- 172.21.15.89:0/646107780 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff8600041b0 con 0x5596d7deb940 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff86fd09740 10 client.20211 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff86fd09740 10 client.20211 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff86fd09740 20 client.20211 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff86fd09740 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5596d7e69810 0x5596d7e6bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff86fd09740 4 client.20211 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 131.4, "await_for": 45.0}] 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff86fd09740 1 -- 172.21.15.89:0/646107780 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 131.4, "await_for": 45.0}) -- 0x5596d7cc9ec0 con 0x5596d7e69810 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.899+0000 7ff866ffd640 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5596d7e69810 0x5596d7e6bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.900+0000 7ff846ffd640 10 client.20211 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:38.904 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.900+0000 7ff866ffd640 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5596d7e69810 0x5596d7e6bc30 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b78fb1ae17782fc6 server_cookie=ae8966f3c3691d1b in_seq=0 out_seq=0 2024-09-15T23:37:38.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff846ffd640 1 -- 172.21.15.89:0/646107780 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5596d7cc9ec0 con 0x5596d7e69810 2024-09-15T23:37:38.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff846ffd640 10 client.20211 handle_command_reply: tid=0 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 client.20211 shutdown 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 -- 172.21.15.89:0/646107780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 msgr2=0x5596d7e61610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7e61610 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7ff86002ae70 tx=0x7ff860047660 comp rx=0 tx=0).stop 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 -- 172.21.15.89:0/646107780 shutdown_connections 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5596d7e69810 0x5596d7e6bc30 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5596d7deb940 0x5596d7e61610 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596d7de1080 0x5596d7de6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 --2- 172.21.15.89:0/646107780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596d7de0760 0x5596d7de5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 -- 172.21.15.89:0/646107780 >> 172.21.15.89:0/646107780 conn(0x5596d7ddccf0 msgr2=0x5596d7de9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff846ffd640 0 client.20211 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 -- 172.21.15.89:0/646107780 shutdown_connections 2024-09-15T23:37:38.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 1 -- 172.21.15.89:0/646107780 wait complete. 2024-09-15T23:37:38.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.914+0000 7ff86fd09740 20 client.20211 trim_cache size 0 max 16384 2024-09-15T23:37:38.917 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:37:38.917 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:37:38.917 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:37:38.917 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 46, 2024-09-15T23:37:38.917 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:37:38.917 INFO:teuthology.orchestra.run.smithi089.stdout: "a3bc58c8": { 2024-09-15T23:37:38.917 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 46, 2024-09-15T23:37:38.917 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:37:38.917 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 131.4, 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:38.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:38.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:38.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:38.919 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:37:38.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 -- 172.21.15.89:0/2330092566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8540a3240 msgr2=0x7ff8540a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:38.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8540a3240 0x7ff8540a5700 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff860005cc0 tx=0x7ff8600095c0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 -- 172.21.15.89:0/2330092566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff868100e20 msgr2=0x7ff8681973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff868100e20 0x7ff8681973e0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7ff85800b4a0 tx=0x7ff85800b960 comp rx=0 tx=0).stop 2024-09-15T23:37:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 -- 172.21.15.89:0/2330092566 shutdown_connections 2024-09-15T23:37:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff8540a3240 0x7ff8540a5700 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff868102ef0 0x7ff86819bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff868102570 0x7ff868197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 --2- 172.21.15.89:0/2330092566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff868100e20 0x7ff8681973e0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 -- 172.21.15.89:0/2330092566 >> 172.21.15.89:0/2330092566 conn(0x7ff8680fc7d0 msgr2=0x7ff868106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 -- 172.21.15.89:0/2330092566 shutdown_connections 2024-09-15T23:37:38.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:38.915+0000 7ff86d930640 1 -- 172.21.15.89:0/2330092566 wait complete. 2024-09-15T23:37:39.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:38 smithi120 ceph-mon[35250]: pgmap v758: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 22 op/s 2024-09-15T23:37:39.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:38 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:37:39.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:38 smithi174 ceph-mon[35586]: pgmap v758: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 22 op/s 2024-09-15T23:37:39.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:38 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:37:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:38 smithi089 ceph-mon[30933]: pgmap v758: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 22 op/s 2024-09-15T23:37:39.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:38 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:37:39.752 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a3bc58c8', quiesce took 5.3 seconds. Will release after: 6.1000000000000005 2024-09-15T23:37:41.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:40 smithi120 ceph-mon[35250]: pgmap v759: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:37:41.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:40 smithi174 ceph-mon[35586]: pgmap v759: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:37:41.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:40 smithi089 ceph-mon[30933]: pgmap v759: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:37:43.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:42 smithi120 ceph-mon[35250]: pgmap v760: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-15T23:37:43.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:42 smithi174 ceph-mon[35586]: pgmap v760: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-15T23:37:43.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:42 smithi089 ceph-mon[30933]: pgmap v760: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 19 op/s 2024-09-15T23:37:45.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:44 smithi120 ceph-mon[35250]: pgmap v761: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 19 op/s 2024-09-15T23:37:45.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:44 smithi174 ceph-mon[35586]: pgmap v761: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 19 op/s 2024-09-15T23:37:45.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:44 smithi089 ceph-mon[30933]: pgmap v761: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 19 op/s 2024-09-15T23:37:45.853 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:37:46.116 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 -- 172.21.15.89:0/1548889020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f01002f0 msgr2=0x7f99f01006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 --2- 172.21.15.89:0/1548889020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f01002f0 0x7f99f01006d0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009930 tx=0x7f99e002a420 comp rx=0 tx=0).stop 2024-09-15T23:37:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 -- 172.21.15.89:0/1548889020 shutdown_connections 2024-09-15T23:37:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 --2- 172.21.15.89:0/1548889020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f0106f50 0x7f99f010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 --2- 172.21.15.89:0/1548889020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99f0100ca0 0x7f99f0106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:46.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 --2- 172.21.15.89:0/1548889020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f01002f0 0x7f99f01006d0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 -- 172.21.15.89:0/1548889020 >> 172.21.15.89:0/1548889020 conn(0x7f99f0077210 msgr2=0x7f99f00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 -- 172.21.15.89:0/1548889020 shutdown_connections 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 -- 172.21.15.89:0/1548889020 wait complete. 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 Processor -- start 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.588+0000 7f99f61b1640 1 -- start start 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99f61b1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f01002f0 0x7f99f0198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99f61b1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f0100ca0 0x7f99f0198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99f61b1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99f0106f50 0x7f99f019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99f61b1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0111fa0 con 0x7f99f01002f0 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99f61b1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0111e20 con 0x7f99f0106f50 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99f61b1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f99f0112120 con 0x7f99f0100ca0 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f01002f0 0x7f99f0198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:46.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f01002f0 0x7f99f0198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38458/0 (socket says 172.21.15.89:38458) 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ef7fe640 1 -- 172.21.15.89:0/3055815028 learned_addr learned my addr 172.21.15.89:0/3055815028 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99e7fff640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f0100ca0 0x7f99f0198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ef7fe640 1 -- 172.21.15.89:0/3055815028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f0100ca0 msgr2=0x7f99f0198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ef7fe640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f0100ca0 0x7f99f0198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ef7fe640 1 -- 172.21.15.89:0/3055815028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99f0106f50 msgr2=0x7f99f019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ef7fe640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99f0106f50 0x7f99f019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ef7fe640 1 -- 172.21.15.89:0/3055815028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f019d4d0 con 0x7f99f01002f0 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ef7fe640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f01002f0 0x7f99f0198520 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009230 tx=0x7f99e0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ed7fa640 1 -- 172.21.15.89:0/3055815028 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e003f030 con 0x7f99f01002f0 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99ed7fa640 1 -- 172.21.15.89:0/3055815028 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99e0003ad0 con 0x7f99f01002f0 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.589+0000 7f99f61b1640 1 -- 172.21.15.89:0/3055815028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f019d760 con 0x7f99f01002f0 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.590+0000 7f99ed7fa640 1 -- 172.21.15.89:0/3055815028 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0035070 con 0x7f99f01002f0 2024-09-15T23:37:46.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.590+0000 7f99f61b1640 1 -- 172.21.15.89:0/3055815028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f01a50a0 con 0x7f99f01002f0 2024-09-15T23:37:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.590+0000 7f99f61b1640 1 -- 172.21.15.89:0/3055815028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99f01033b0 con 0x7f99f01002f0 2024-09-15T23:37:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.592+0000 7f99ed7fa640 1 -- 172.21.15.89:0/3055815028 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f99e0042030 con 0x7f99f01002f0 2024-09-15T23:37:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.592+0000 7f99ed7fa640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99dc0a3240 0x7f99dc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.593+0000 7f99e7fff640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99dc0a3240 0x7f99dc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.593+0000 7f99ed7fa640 1 -- 172.21.15.89:0/3055815028 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f99e00f0630 con 0x7f99f01002f0 2024-09-15T23:37:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.593+0000 7f99e7fff640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99dc0a3240 0x7f99dc0a5700 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f99f01048c0 tx=0x7f99d800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:46.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.596+0000 7f99ed7fa640 1 -- 172.21.15.89:0/3055815028 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f99e00bcf60 con 0x7f99f01002f0 2024-09-15T23:37:46.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.795+0000 7f99f61b1640 1 -- 172.21.15.89:0/3055815028 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99f01006d0 con 0x7f99f01002f0 2024-09-15T23:37:46.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.796+0000 7f99ed7fa640 1 -- 172.21.15.89:0/3055815028 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f99e00bc860 con 0x7f99f01002f0 2024-09-15T23:37:46.797 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:37:46.797 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:37:46.797 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:37:46.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 -- 172.21.15.89:0/3055815028 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99dc0a3240 msgr2=0x7f99dc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:46.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99dc0a3240 0x7f99dc0a5700 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f99f01048c0 tx=0x7f99d800b390 comp rx=0 tx=0).stop 2024-09-15T23:37:46.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 -- 172.21.15.89:0/3055815028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f01002f0 msgr2=0x7f99f0198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:46.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f01002f0 0x7f99f0198520 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009230 tx=0x7f99e0004270 comp rx=0 tx=0).stop 2024-09-15T23:37:46.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 -- 172.21.15.89:0/3055815028 shutdown_connections 2024-09-15T23:37:46.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f99dc0a3240 0x7f99dc0a5700 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:46.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99f0106f50 0x7f99f019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:46.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99f0100ca0 0x7f99f0198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:46.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 --2- 172.21.15.89:0/3055815028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f99f01002f0 0x7f99f0198520 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:46.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 -- 172.21.15.89:0/3055815028 >> 172.21.15.89:0/3055815028 conn(0x7f99f0077210 msgr2=0x7f99f00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:46.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 -- 172.21.15.89:0/3055815028 shutdown_connections 2024-09-15T23:37:46.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:46.798+0000 7f99f61b1640 1 -- 172.21.15.89:0/3055815028 wait complete. 2024-09-15T23:37:47.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:46 smithi089 ceph-mon[30933]: pgmap v762: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-15T23:37:47.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:46 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3055815028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:47.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:46 smithi174 ceph-mon[35586]: pgmap v762: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-15T23:37:47.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:46 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3055815028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:46 smithi120 ceph-mon[35250]: pgmap v762: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-15T23:37:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:46 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3055815028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:37:47.485 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id a3bc58c8 --release --await' 2024-09-15T23:37:47.485 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id a3bc58c8 --release --await 2024-09-15T23:37:47.734 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:37:48.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.238+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/3268356453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7e4105e40 msgr2=0x7fb7e410d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:48.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.238+0000 7fb7e8ea9640 1 --2- 172.21.15.89:0/3268356453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7e4105e40 0x7fb7e410d9c0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009ca0 tx=0x7fb7dc02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.238+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/3268356453 shutdown_connections 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.238+0000 7fb7e8ea9640 1 --2- 172.21.15.89:0/3268356453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7e4105e40 0x7fb7e410d9c0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.238+0000 7fb7e8ea9640 1 --2- 172.21.15.89:0/3268356453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e4101fc0 0x7fb7e4105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.238+0000 7fb7e8ea9640 1 --2- 172.21.15.89:0/3268356453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4101610 0x7fb7e41019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.238+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/3268356453 >> 172.21.15.89:0/3268356453 conn(0x7fb7e40fc5c0 msgr2=0x7fb7e40fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.238+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/3268356453 shutdown_connections 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.238+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/3268356453 wait complete. 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e8ea9640 1 Processor -- start 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e8ea9640 1 -- start start 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e8ea9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4101610 0x7fb7e419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e8ea9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e4101fc0 0x7fb7e419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e8ea9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7e4105e40 0x7fb7e419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e8ea9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410fde0 con 0x7fb7e4105e40 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e8ea9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410fc60 con 0x7fb7e4101610 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e8ea9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410ff60 con 0x7fb7e4101fc0 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7e4105e40 0x7fb7e419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7e4105e40 0x7fb7e419ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38476/0 (socket says 172.21.15.89:38476) 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e2d9d640 1 -- 172.21.15.89:0/2541866966 learned_addr learned my addr 172.21.15.89:0/2541866966 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e259c640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4101610 0x7fb7e419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e4101fc0 0x7fb7e419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e2d9d640 1 -- 172.21.15.89:0/2541866966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e4101fc0 msgr2=0x7fb7e419abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e2d9d640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e4101fc0 0x7fb7e419abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e2d9d640 1 -- 172.21.15.89:0/2541866966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4101610 msgr2=0x7fb7e419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.239+0000 7fb7e2d9d640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4101610 0x7fb7e419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.263 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.240+0000 7fb7e2d9d640 1 -- 172.21.15.89:0/2541866966 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e419f620 con 0x7fb7e4105e40 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.240+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e4101fc0 0x7fb7e419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.240+0000 7fb7e259c640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4101610 0x7fb7e419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.240+0000 7fb7e2d9d640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7e4105e40 0x7fb7e419ef40 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc031a60 tx=0x7fb7dc031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.240+0000 7fb7db7fe640 1 -- 172.21.15.89:0/2541866966 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc003c30 con 0x7fb7e4105e40 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.240+0000 7fb7db7fe640 1 -- 172.21.15.89:0/2541866966 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7dc003dd0 con 0x7fb7e4105e40 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.240+0000 7fb7db7fe640 1 -- 172.21.15.89:0/2541866966 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0047e0 con 0x7fb7e4105e40 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.240+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/2541866966 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e419f8b0 con 0x7fb7e4105e40 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.240+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/2541866966 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e41a71f0 con 0x7fb7e4105e40 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.242+0000 7fb7db7fe640 1 -- 172.21.15.89:0/2541866966 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb7dc02f790 con 0x7fb7e4105e40 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7eb282740 1 Processor -- start 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7eb282740 1 -- start start 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9060 0x556d868b9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.264 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9980 0x556d868c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7eb282740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556d8681e620 con 0x556d868b9980 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7eb282740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556d867c5e50 con 0x556d868c4240 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7eb282740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556d8681a5e0 con 0x556d868b9060 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7e259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9980 0x556d868c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7e259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9980 0x556d868c3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38482/0 (socket says 172.21.15.89:38482) 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7e259c640 1 -- 172.21.15.89:0/2765595776 learned_addr learned my addr 172.21.15.89:0/2765595776 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/2765595776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9060 0x556d868b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7db7fe640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb7d00a32b0 0x7fb7d00a5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/2765595776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.243+0000 7fb7e259c640 1 -- 172.21.15.89:0/2765595776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9060 msgr2=0x556d868b9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7e259c640 1 --2- 172.21.15.89:0/2765595776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9060 0x556d868b9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7e259c640 1 -- 172.21.15.89:0/2765595776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 msgr2=0x556d868c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7e259c640 1 --2- 172.21.15.89:0/2765595776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7e259c640 1 -- 172.21.15.89:0/2765595776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d867b6d40 con 0x556d868b9980 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/2765595776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9060 0x556d868b9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7e2d9d640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb7d00a32b0 0x7fb7d00a5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7db7fe640 1 -- 172.21.15.89:0/2541866966 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb7dc0f1160 con 0x7fb7e4105e40 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7e259c640 1 --2- 172.21.15.89:0/2765595776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9980 0x556d868c3d00 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400cca0 tx=0x7fb7d400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a619ed394a7882 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7d8ff9640 1 -- 172.21.15.89:0/2765595776 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4005220 con 0x556d868b9980 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7d8ff9640 1 -- 172.21.15.89:0/2765595776 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7d4004c80 con 0x556d868b9980 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.244+0000 7fb7d8ff9640 1 -- 172.21.15.89:0/2765595776 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4010800 con 0x556d868b9980 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 -- 172.21.15.89:0/2765595776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9980 msgr2=0x556d868c3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 --2- 172.21.15.89:0/2765595776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9980 0x556d868c3d00 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400cca0 tx=0x7fb7d400e720 comp rx=0 tx=0).stop 2024-09-15T23:37:48.266 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7e2d9d640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb7d00a32b0 0x7fb7d00a5770 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0ee870 tx=0x7fb7dc0b2870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 -- 172.21.15.89:0/2765595776 shutdown_connections 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 --2- 172.21.15.89:0/2765595776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 --2- 172.21.15.89:0/2765595776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9980 0x556d868c3d00 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 --2- 172.21.15.89:0/2765595776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9060 0x556d868b9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 -- 172.21.15.89:0/2765595776 >> 172.21.15.89:0/2765595776 conn(0x556d868b55f0 msgr2=0x556d868b59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 -- 172.21.15.89:0/2765595776 shutdown_connections 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 -- 172.21.15.89:0/2765595776 wait complete. 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 -- start start 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 Processor -- start 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868c1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.267 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.245+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 0x556d868c1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7eb282740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556d868c6e10 con 0x556d868b9060 2024-09-15T23:37:48.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7eb282740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556d867a6f10 con 0x556d868c4240 2024-09-15T23:37:48.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7eb282740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556d868c6f90 con 0x556d868b9980 2024-09-15T23:37:48.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7eb282740 10 client.0 _ll_register_callbacks cb 0x556d867cadb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:48.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868c1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38484/0 (socket says 172.21.15.89:38484) 2024-09-15T23:37:48.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/3259546796 learned_addr learned my addr 172.21.15.89:0/3259546796 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:48.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e259c640 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 0x556d868c1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/3259546796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 msgr2=0x556d868c1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:48.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 0x556d868c1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/3259546796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 msgr2=0x556d868c1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:48.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/3259546796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d8652d580 con 0x556d868b9060 2024-09-15T23:37:48.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e259c640 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 0x556d868c1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:48.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868c1120 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc006eb0 tx=0x7fb7cc0128c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8037c1a39836f2d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:48.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:48.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/3259546796 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00f040 con 0x556d868b9060 2024-09-15T23:37:48.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/3259546796 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb7cc0137b0 con 0x556d868b9060 2024-09-15T23:37:48.269 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7eb282740 1 -- 172.21.15.89:0/3259546796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7d4004c80 con 0x556d868b9060 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/3259546796 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb7cc023050 con 0x556d868b9060 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7eb282740 10 client.20229 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.246+0000 7fb7eb282740 1 -- 172.21.15.89:0/3259546796 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556d865f7960 con 0x556d868b9060 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/3259546796 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7d4004c80 con 0x556d868b9060 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/3259546796 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb7cc015ad0 con 0x556d868b9060 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7eb282740 10 client.20229 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7eb282740 10 client.20229 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7eb282740 20 client.20229 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7eb282740 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d868d3eb0 0x556d868d62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7eb282740 4 client.20229 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7eb282740 1 -- 172.21.15.89:0/3259546796 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d8682bb90 con 0x556d868d3eb0 2024-09-15T23:37:48.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7e259c640 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d868d3eb0 0x556d868d62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7e259c640 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d868d3eb0 0x556d868d62c0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84a5727c816b2118 server_cookie=727493243cfff3b3 in_seq=0 out_seq=0 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.247+0000 7fb7c9ffb640 10 client.20229 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/3259546796 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556d8682bb90 con 0x556d868d3eb0 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7c9ffb640 10 client.20229 handle_command_reply: tid=0 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 client.20229 shutdown 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 -- 172.21.15.89:0/3259546796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 msgr2=0x556d868c1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868c1120 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc006eb0 tx=0x7fb7cc0128c0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 -- 172.21.15.89:0/3259546796 shutdown_connections 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d868d3eb0 0x556d868d62c0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 0x556d868c1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 --2- 172.21.15.89:0/3259546796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868c1120 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 -- 172.21.15.89:0/3259546796 >> 172.21.15.89:0/3259546796 conn(0x556d868b55f0 msgr2=0x556d868bb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:48.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7c9ffb640 0 client.20229 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:48.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 -- 172.21.15.89:0/3259546796 shutdown_connections 2024-09-15T23:37:48.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.248+0000 7fb7eb282740 1 -- 172.21.15.89:0/3259546796 wait complete. 2024-09-15T23:37:48.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.249+0000 7fb7eb282740 20 client.20229 trim_cache size 0 max 16384 2024-09-15T23:37:48.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7eb282740 1 Processor -- start 2024-09-15T23:37:48.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7eb282740 1 -- start start 2024-09-15T23:37:48.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868bbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868b9980 0x556d868c3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868c4240 0x556d868dbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7eb282740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41b10d0 con 0x556d868b9060 2024-09-15T23:37:48.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7eb282740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41b2c10 con 0x556d868b9980 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7eb282740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41b17a0 con 0x556d868c4240 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7e1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868bbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7e1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868bbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38500/0 (socket says 172.21.15.89:38500) 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/2102272585 learned_addr learned my addr 172.21.15.89:0/2102272585 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7e259c640 1 --2- 172.21.15.89:0/2102272585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868b9980 0x556d868c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/2102272585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868c4240 msgr2=0x556d868dbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/2102272585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868c4240 0x556d868dbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/2102272585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868b9980 msgr2=0x556d868c3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/2102272585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868b9980 0x556d868c3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/2102272585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d868c00d0 con 0x556d868b9060 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.266+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/2102272585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868bbac0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc004810 tx=0x7fb7cc01e400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588ee68eb94b33c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7cbfff640 1 -- 172.21.15.89:0/2102272585 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc012dc0 con 0x556d868b9060 2024-09-15T23:37:48.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7cbfff640 1 -- 172.21.15.89:0/2102272585 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7cc02c070 con 0x556d868b9060 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7cbfff640 1 -- 172.21.15.89:0/2102272585 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc015b30 con 0x556d868b9060 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7eb282740 1 -- 172.21.15.89:0/2102272585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 msgr2=0x556d868bbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7eb282740 1 --2- 172.21.15.89:0/2102272585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868bbac0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc004810 tx=0x7fb7cc01e400 comp rx=0 tx=0).stop 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7eb282740 1 -- 172.21.15.89:0/2102272585 shutdown_connections 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7eb282740 1 --2- 172.21.15.89:0/2102272585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868c4240 0x556d868dbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7eb282740 1 --2- 172.21.15.89:0/2102272585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868b9980 0x556d868c3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7eb282740 1 --2- 172.21.15.89:0/2102272585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868bbac0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7eb282740 1 -- 172.21.15.89:0/2102272585 >> 172.21.15.89:0/2102272585 conn(0x556d868b55f0 msgr2=0x556d868c2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7eb282740 1 -- 172.21.15.89:0/2102272585 shutdown_connections 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.267+0000 7fb7eb282740 1 -- 172.21.15.89:0/2102272585 wait complete. 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7eb282740 1 -- start start 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7eb282740 1 Processor -- start 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868cca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 0x556d868c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7eb282740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7eb282740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41b2540 con 0x556d868b9060 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7eb282740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41b1e70 con 0x556d868c4240 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7eb282740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e41af0b0 con 0x556d868b9980 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7eb282740 10 client.0 _ll_register_callbacks cb 0x7fb7cc00cda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7e1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7e1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38514/0 (socket says 172.21.15.89:38514) 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/804570116 learned_addr learned my addr 172.21.15.89:0/804570116 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7e259c640 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 0x556d868c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/804570116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 msgr2=0x556d868c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 0x556d868c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/804570116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 msgr2=0x556d868c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.268+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7e1d9b640 1 -- 172.21.15.89:0/804570116 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d8693dba0 con 0x556d868b9060 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7e259c640 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 0x556d868c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7e1d9b640 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868cca80 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc012900 tx=0x7fb7cc00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b75653982f1a58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/804570116 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc03a060 con 0x556d868b9060 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/804570116 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb7cc01e8a0 con 0x556d868b9060 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7eb282740 1 -- 172.21.15.89:0/804570116 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7cc02c070 con 0x556d868b9060 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/804570116 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb7cc041050 con 0x556d868b9060 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7eb282740 10 client.20235 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7eb282740 1 -- 172.21.15.89:0/804570116 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556d8693ed30 con 0x556d868b9060 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/804570116 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7cc02c070 con 0x556d868b9060 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/804570116 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb7cc013a90 con 0x556d868b9060 2024-09-15T23:37:48.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7eb282740 10 client.20235 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7eb282740 10 client.20235 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7eb282740 20 client.20235 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7eb282740 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d8693f340 0x556d86941760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7eb282740 4 client.20235 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a3bc58c8", "release": true, "await": true}] 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.269+0000 7fb7eb282740 1 -- 172.21.15.89:0/804570116 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a3bc58c8", "release": true, "await": true}) -- 0x556d868260a0 con 0x556d8693f340 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.270+0000 7fb7e259c640 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d8693f340 0x556d86941760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.270+0000 7fb7e259c640 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d8693f340 0x556d86941760 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96e67c051018cec6 server_cookie=ff0b9b9e2a00ef11 in_seq=0 out_seq=0 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.270+0000 7fb7c9ffb640 10 client.20235 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7c9ffb640 1 -- 172.21.15.89:0/804570116 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x556d868260a0 con 0x556d8693f340 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7c9ffb640 10 client.20235 handle_command_reply: tid=0 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 client.20235 shutdown 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 -- 172.21.15.89:0/804570116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 msgr2=0x556d868cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:48.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868cca80 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc012900 tx=0x7fb7cc00ce50 comp rx=0 tx=0).stop 2024-09-15T23:37:48.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 -- 172.21.15.89:0/804570116 shutdown_connections 2024-09-15T23:37:48.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x556d8693f340 0x556d86941760 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d868c4240 0x556d868c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556d868b9980 0x556d868c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 --2- 172.21.15.89:0/804570116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556d868b9060 0x556d868cca80 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7c9ffb640 0 client.20235 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:48.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 -- 172.21.15.89:0/804570116 >> 172.21.15.89:0/804570116 conn(0x556d868b55f0 msgr2=0x556d868c31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:48.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 -- 172.21.15.89:0/804570116 shutdown_connections 2024-09-15T23:37:48.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 1 -- 172.21.15.89:0/804570116 wait complete. 2024-09-15T23:37:48.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.274+0000 7fb7eb282740 20 client.20235 trim_cache size 0 max 16384 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 48, 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "a3bc58c8": { 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 48, 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:37:48.284 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 131.4, 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:37:48.285 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:37:48.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/2541866966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb7d00a32b0 msgr2=0x7fb7d00a5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:48.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb7d00a32b0 0x7fb7d00a5770 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0ee870 tx=0x7fb7dc0b2870 comp rx=0 tx=0).stop 2024-09-15T23:37:48.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/2541866966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7e4105e40 msgr2=0x7fb7e419ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:48.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7e4105e40 0x7fb7e419ef40 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc031a60 tx=0x7fb7dc031a90 comp rx=0 tx=0).stop 2024-09-15T23:37:48.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/2541866966 shutdown_connections 2024-09-15T23:37:48.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb7d00a32b0 0x7fb7d00a5770 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7e4105e40 0x7fb7e419ef40 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7e4101fc0 0x7fb7e419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 --2- 172.21.15.89:0/2541866966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4101610 0x7fb7e419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:48.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/2541866966 >> 172.21.15.89:0/2541866966 conn(0x7fb7e40fc5c0 msgr2=0x7fb7e4104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:48.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/2541866966 shutdown_connections 2024-09-15T23:37:48.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:37:48.275+0000 7fb7e8ea9640 1 -- 172.21.15.89:0/2541866966 wait complete. 2024-09-15T23:37:49.136 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a3bc58c8', total seconds elapsed: 14.7 2024-09-15T23:37:49.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:48 smithi174 ceph-mon[35586]: pgmap v763: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 19 op/s 2024-09-15T23:37:49.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:48 smithi089 ceph-mon[30933]: pgmap v763: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 19 op/s 2024-09-15T23:37:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.423+0000 7f90ef986640 1 -- 172.21.15.89:0/1421781137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90e8101670 msgr2=0x7f90e8108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.423+0000 7f90ef986640 1 --2- 172.21.15.89:0/1421781137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90e8101670 0x7f90e8108f10 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009ca0 tx=0x7f90e002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.423+0000 7f90ef986640 1 -- 172.21.15.89:0/1421781137 shutdown_connections 2024-09-15T23:37:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.423+0000 7f90ef986640 1 --2- 172.21.15.89:0/1421781137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90e8101670 0x7f90e8108f10 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.423+0000 7f90ef986640 1 --2- 172.21.15.89:0/1421781137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90e8100cb0 0x7f90e8101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.423+0000 7f90ef986640 1 --2- 172.21.15.89:0/1421781137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90e8102060 0x7f90e8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.423+0000 7f90ef986640 1 -- 172.21.15.89:0/1421781137 >> 172.21.15.89:0/1421781137 conn(0x7f90e80fb970 msgr2=0x7f90e80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90ef986640 1 -- 172.21.15.89:0/1421781137 shutdown_connections 2024-09-15T23:37:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90ef986640 1 -- 172.21.15.89:0/1421781137 wait complete. 2024-09-15T23:37:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90ef986640 1 Processor -- start 2024-09-15T23:37:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90ef986640 1 -- start start 2024-09-15T23:37:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90ef986640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90e8100cb0 0x7f90e8197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90ef986640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90e8101670 0x7f90e81982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90ef986640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90e8102060 0x7f90e819f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90ef986640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90e810c6c0 con 0x7f90e8102060 2024-09-15T23:37:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90ef986640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f90e810c540 con 0x7f90e8101670 2024-09-15T23:37:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90ef986640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f90e810c840 con 0x7f90e8100cb0 2024-09-15T23:37:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90edf23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90e8102060 0x7f90e819f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90edf23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90e8102060 0x7f90e819f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38534/0 (socket says 172.21.15.89:38534) 2024-09-15T23:37:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.424+0000 7f90edf23640 1 -- 172.21.15.89:0/1197549539 learned_addr learned my addr 172.21.15.89:0/1197549539 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90ecf21640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90e8101670 0x7f90e81982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90ed722640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90e8100cb0 0x7f90e8197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90edf23640 1 -- 172.21.15.89:0/1197549539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90e8100cb0 msgr2=0x7f90e8197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90edf23640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90e8100cb0 0x7f90e8197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90edf23640 1 -- 172.21.15.89:0/1197549539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90e8101670 msgr2=0x7f90e81982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90edf23640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90e8101670 0x7f90e81982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90edf23640 1 -- 172.21.15.89:0/1197549539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e819f9b0 con 0x7f90e8102060 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90ed722640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90e8100cb0 0x7f90e8197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90edf23640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90e8102060 0x7f90e819f310 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f90e0005e30 tx=0x7f90e0031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90de7fc640 1 -- 172.21.15.89:0/1197549539 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e0003c30 con 0x7f90e8102060 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90de7fc640 1 -- 172.21.15.89:0/1197549539 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90e0003dd0 con 0x7f90e8102060 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90ef986640 1 -- 172.21.15.89:0/1197549539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e819fc40 con 0x7f90e8102060 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90de7fc640 1 -- 172.21.15.89:0/1197549539 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e002fde0 con 0x7f90e8102060 2024-09-15T23:37:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.425+0000 7f90ef986640 1 -- 172.21.15.89:0/1197549539 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e81a0190 con 0x7f90e8102060 2024-09-15T23:37:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90de7fc640 1 -- 172.21.15.89:0/1197549539 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f90e0004050 con 0x7f90e8102060 2024-09-15T23:37:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90f27fe740 1 Processor -- start 2024-09-15T23:37:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90f27fe740 1 -- start start 2024-09-15T23:37:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f7379141b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f7379191b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 0x55f73791bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90f27fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f7378732f0 con 0x55f7379146f0 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90f27fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f73781b270 con 0x55f737919780 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90f27fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f73786f2b0 con 0x55f737913db0 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90ed722640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f7379191b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90ed722640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f7379191b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38548/0 (socket says 172.21.15.89:38548) 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90ed722640 1 -- 172.21.15.89:0/2349940417 learned_addr learned my addr 172.21.15.89:0/2349940417 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90ecf21640 1 --2- 172.21.15.89:0/2349940417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f7379141b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90ecf21640 1 --2- 172.21.15.89:0/2349940417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 0x55f73791bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90de7fc640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90d40a32f0 0x7f90d40a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90ed722640 1 -- 172.21.15.89:0/2349940417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 msgr2=0x55f7379141b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90ed722640 1 --2- 172.21.15.89:0/2349940417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f7379141b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90ed722640 1 -- 172.21.15.89:0/2349940417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 msgr2=0x55f73791bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90ed722640 1 --2- 172.21.15.89:0/2349940417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 0x55f73791bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.428+0000 7f90ed722640 1 -- 172.21.15.89:0/2349940417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f73781a340 con 0x55f7379146f0 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.429+0000 7f90ecf21640 1 --2- 172.21.15.89:0/2349940417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 0x55f73791bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.429+0000 7f90edf23640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90d40a32f0 0x7f90d40a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.429+0000 7f90ed722640 1 --2- 172.21.15.89:0/2349940417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f7379191b0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f90e400bb60 tx=0x7f90e400d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5536a2e297265afa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.429+0000 7f90de7fc640 1 -- 172.21.15.89:0/1197549539 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f90e00f1210 con 0x7f90e8102060 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.429+0000 7f90d37fe640 1 -- 172.21.15.89:0/2349940417 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e400be60 con 0x55f7379146f0 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.429+0000 7f90d37fe640 1 -- 172.21.15.89:0/2349940417 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90e4004220 con 0x55f7379146f0 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.429+0000 7f90d37fe640 1 -- 172.21.15.89:0/2349940417 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e400f810 con 0x55f7379146f0 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 -- 172.21.15.89:0/2349940417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 msgr2=0x55f7379191b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90edf23640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90d40a32f0 0x7f90d40a57b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f90e00bd3d0 tx=0x7f90e00b2890 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 --2- 172.21.15.89:0/2349940417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f7379191b0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f90e400bb60 tx=0x7f90e400d710 comp rx=0 tx=0).stop 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 -- 172.21.15.89:0/2349940417 shutdown_connections 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 --2- 172.21.15.89:0/2349940417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 0x55f73791bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 --2- 172.21.15.89:0/2349940417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f7379191b0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 --2- 172.21.15.89:0/2349940417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f7379141b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 -- 172.21.15.89:0/2349940417 >> 172.21.15.89:0/2349940417 conn(0x55f73790f420 msgr2=0x55f73790f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 -- 172.21.15.89:0/2349940417 shutdown_connections 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 -- 172.21.15.89:0/2349940417 wait complete. 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 -- start start 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 Processor -- start 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f73791e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f73791e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 0x55f737925540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f73791c750 con 0x55f7379146f0 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f7377fc290 con 0x55f737919780 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.430+0000 7f90f27fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f73791c8d0 con 0x55f737913db0 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90f27fe740 10 client.0 _ll_register_callbacks cb 0x55f73782d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ed722640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f73791e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ed722640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f73791e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38556/0 (socket says 172.21.15.89:38556) 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ed722640 1 -- 172.21.15.89:0/1416479723 learned_addr learned my addr 172.21.15.89:0/1416479723 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ecf21640 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f73791e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ecf21640 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 0x55f737925540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ed722640 1 -- 172.21.15.89:0/1416479723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 msgr2=0x55f73791e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ed722640 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f73791e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ed722640 1 -- 172.21.15.89:0/1416479723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 msgr2=0x55f737925540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ed722640 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 0x55f737925540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ed722640 1 -- 172.21.15.89:0/1416479723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f737586210 con 0x55f7379146f0 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ecf21640 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 0x55f737925540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90ed722640 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f73791e570 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f90e400b310 tx=0x7f90e4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb695b56163477ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:49.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90d0ff9640 1 -- 172.21.15.89:0/1416479723 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4025020 con 0x55f7379146f0 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90d0ff9640 1 -- 172.21.15.89:0/1416479723 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f90e40043d0 con 0x55f7379146f0 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90f27fe740 1 -- 172.21.15.89:0/1416479723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90e4004220 con 0x55f7379146f0 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90d0ff9640 1 -- 172.21.15.89:0/1416479723 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f90e4030050 con 0x55f7379146f0 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90f27fe740 10 client.20247 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90f27fe740 1 -- 172.21.15.89:0/1416479723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f737652620 con 0x55f7379146f0 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.431+0000 7f90d0ff9640 1 -- 172.21.15.89:0/1416479723 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90e4004220 con 0x55f7379146f0 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.432+0000 7f90d0ff9640 1 -- 172.21.15.89:0/1416479723 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f90e4021480 con 0x55f7379146f0 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.432+0000 7f90f27fe740 10 client.20247 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.432+0000 7f90f27fe740 10 client.20247 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.432+0000 7f90f27fe740 20 client.20247 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.432+0000 7f90f27fe740 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f7379298c0 0x55f73792bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.432+0000 7f90f27fe740 4 client.20247 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.432+0000 7f90f27fe740 1 -- 172.21.15.89:0/1416479723 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f737880a20 con 0x55f7379298c0 2024-09-15T23:37:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.432+0000 7f90ecf21640 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f7379298c0 0x55f73792bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.432+0000 7f90d0ff9640 10 client.20247 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.432+0000 7f90ecf21640 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f7379298c0 0x55f73792bcb0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=629a75ee8b00cfd4 server_cookie=326c800e18e4e511 in_seq=0 out_seq=0 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90d0ff9640 1 -- 172.21.15.89:0/1416479723 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f737880a20 con 0x55f7379298c0 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90d0ff9640 10 client.20247 handle_command_reply: tid=0 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 client.20247 shutdown 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 -- 172.21.15.89:0/1416479723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 msgr2=0x55f73791e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f73791e570 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f90e400b310 tx=0x7f90e4004740 comp rx=0 tx=0).stop 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 -- 172.21.15.89:0/1416479723 shutdown_connections 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f7379298c0 0x55f73792bcb0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737919780 0x55f737925540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f73791e570 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 --2- 172.21.15.89:0/1416479723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f73791e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90d0ff9640 0 client.20247 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 -- 172.21.15.89:0/1416479723 >> 172.21.15.89:0/1416479723 conn(0x55f73790f420 msgr2=0x55f73790a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 -- 172.21.15.89:0/1416479723 shutdown_connections 2024-09-15T23:37:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 1 -- 172.21.15.89:0/1416479723 wait complete. 2024-09-15T23:37:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.433+0000 7f90f27fe740 20 client.20247 trim_cache size 0 max 16384 2024-09-15T23:37:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90f27fe740 1 Processor -- start 2024-09-15T23:37:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90f27fe740 1 -- start start 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f73790f420 0x55f73790a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737913db0 0x55f737925500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f737910810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90f27fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81a8650 con 0x55f7379146f0 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90f27fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81aa190 con 0x55f737913db0 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90f27fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81a8d20 con 0x55f73790f420 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90ecf21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f737910810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90ecf21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f737910810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38570/0 (socket says 172.21.15.89:38570) 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90ecf21640 1 -- 172.21.15.89:0/4228257382 learned_addr learned my addr 172.21.15.89:0/4228257382 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90ecf21640 1 --2- 172.21.15.89:0/4228257382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f73790f420 0x55f73790a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.448+0000 7f90ed722640 1 --2- 172.21.15.89:0/4228257382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737913db0 0x55f737925500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90ecf21640 1 -- 172.21.15.89:0/4228257382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f73790f420 msgr2=0x55f73790a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90ecf21640 1 --2- 172.21.15.89:0/4228257382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f73790f420 0x55f73790a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90ecf21640 1 -- 172.21.15.89:0/4228257382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737913db0 msgr2=0x55f737925500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90ecf21640 1 --2- 172.21.15.89:0/4228257382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737913db0 0x55f737925500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90ecf21640 1 -- 172.21.15.89:0/4228257382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f737909650 con 0x55f7379146f0 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90ecf21640 1 --2- 172.21.15.89:0/4228257382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f737910810 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f90d8000c00 tx=0x7f90d800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b43a85620b35a89 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90d2ffd640 1 -- 172.21.15.89:0/4228257382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d8025040 con 0x55f7379146f0 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90d2ffd640 1 -- 172.21.15.89:0/4228257382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90d80108b0 con 0x55f7379146f0 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90d2ffd640 1 -- 172.21.15.89:0/4228257382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d800f8d0 con 0x55f7379146f0 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90f27fe740 1 -- 172.21.15.89:0/4228257382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 msgr2=0x55f737910810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.449+0000 7f90f27fe740 1 --2- 172.21.15.89:0/4228257382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f737910810 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f90d8000c00 tx=0x7f90d800b5b0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 -- 172.21.15.89:0/4228257382 shutdown_connections 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 --2- 172.21.15.89:0/4228257382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7379146f0 0x55f737910810 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 --2- 172.21.15.89:0/4228257382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f737913db0 0x55f737925500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 --2- 172.21.15.89:0/4228257382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f73790f420 0x55f73790a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 -- 172.21.15.89:0/4228257382 >> 172.21.15.89:0/4228257382 conn(0x55f7379151b0 msgr2=0x55f7379099a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 -- 172.21.15.89:0/4228257382 shutdown_connections 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 -- 172.21.15.89:0/4228257382 wait complete. 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 -- start start 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 Processor -- start 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f73790f420 0x55f7379229f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f737922f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7379146f0 0x55f737923ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81a9ac0 con 0x55f73790f420 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81a93f0 con 0x55f7379146f0 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f90e81a6630 con 0x55f737913db0 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90f27fe740 10 client.0 _ll_register_callbacks cb 0x7f90e400b9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90ecf21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f73790f420 0x55f7379229f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90ecf21640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f73790f420 0x55f7379229f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38586/0 (socket says 172.21.15.89:38586) 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.450+0000 7f90ecf21640 1 -- 172.21.15.89:0/2491940952 learned_addr learned my addr 172.21.15.89:0/2491940952 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90ed722640 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f737922f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90ecf21640 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7379146f0 0x55f737923ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90ecf21640 1 -- 172.21.15.89:0/2491940952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 msgr2=0x55f737922f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90ecf21640 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f737922f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90ecf21640 1 -- 172.21.15.89:0/2491940952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7379146f0 msgr2=0x55f737923ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90ecf21640 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7379146f0 0x55f737923ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90ecf21640 1 -- 172.21.15.89:0/2491940952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f737924500 con 0x55f73790f420 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90ecf21640 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f73790f420 0x55f7379229f0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f90d800b6b0 tx=0x7f90d800b3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3df4d2d4338b3f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90d0ff9640 1 -- 172.21.15.89:0/2491940952 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d8012240 con 0x55f73790f420 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90d0ff9640 1 -- 172.21.15.89:0/2491940952 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f90d80103d0 con 0x55f73790f420 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90f27fe740 1 -- 172.21.15.89:0/2491940952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90d80108b0 con 0x55f73790f420 2024-09-15T23:37:49.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90d0ff9640 1 -- 172.21.15.89:0/2491940952 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f90d80105d0 con 0x55f73790f420 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90f27fe740 10 client.20253 fetch_fsmap learned FSMap version 29 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90f27fe740 1 -- 172.21.15.89:0/2491940952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f737925810 con 0x55f73790f420 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.451+0000 7f90d0ff9640 1 -- 172.21.15.89:0/2491940952 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90d80108b0 con 0x55f73790f420 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.452+0000 7f90d0ff9640 1 -- 172.21.15.89:0/2491940952 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f90d8004730 con 0x55f73790f420 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.452+0000 7f90f27fe740 10 client.20253 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.452+0000 7f90f27fe740 10 client.20253 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.452+0000 7f90f27fe740 20 client.20253 populate_metadata read hostname 'smithi089' 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.452+0000 7f90f27fe740 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f737919780 0x55f737996b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.452+0000 7f90f27fe740 4 client.20253 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.452+0000 7f90f27fe740 1 -- 172.21.15.89:0/2491940952 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f73787ad70 con 0x55f737919780 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.452+0000 7f90ed722640 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f737919780 0x55f737996b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.452+0000 7f90d0ff9640 10 client.20253 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:49.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.452+0000 7f90ed722640 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f737919780 0x55f737996b00 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18edec70f4c7dc6 server_cookie=74d55bf573bcf58d in_seq=0 out_seq=0 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90d0ff9640 1 -- 172.21.15.89:0/2491940952 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f73787ad70 con 0x55f737919780 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90d0ff9640 10 client.20253 handle_command_reply: tid=0 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 client.20253 shutdown 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 -- 172.21.15.89:0/2491940952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f73790f420 msgr2=0x55f7379229f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f73790f420 0x55f7379229f0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f90d800b6b0 tx=0x7f90d800b3f0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 -- 172.21.15.89:0/2491940952 shutdown_connections 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f737919780 0x55f737996b00 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7379146f0 0x55f737923ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f737913db0 0x55f737922f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90d0ff9640 0 client.20253 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 --2- 172.21.15.89:0/2491940952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f73790f420 0x55f7379229f0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 -- 172.21.15.89:0/2491940952 >> 172.21.15.89:0/2491940952 conn(0x55f7379151b0 msgr2=0x55f73790a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 -- 172.21.15.89:0/2491940952 shutdown_connections 2024-09-15T23:37:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 1 -- 172.21.15.89:0/2491940952 wait complete. 2024-09-15T23:37:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.467+0000 7f90f27fe740 20 client.20253 trim_cache size 0 max 16384 2024-09-15T23:37:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:37:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:37:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:37:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:37:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 -- 172.21.15.89:0/1197549539 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90d40a32f0 msgr2=0x7f90d40a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90d40a32f0 0x7f90d40a57b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f90e00bd3d0 tx=0x7f90e00b2890 comp rx=0 tx=0).stop 2024-09-15T23:37:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 -- 172.21.15.89:0/1197549539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90e8102060 msgr2=0x7f90e819f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:37:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90e8102060 0x7f90e819f310 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f90e0005e30 tx=0x7f90e0031ee0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 -- 172.21.15.89:0/1197549539 shutdown_connections 2024-09-15T23:37:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f90d40a32f0 0x7f90d40a57b0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f90e8102060 0x7f90e819f310 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90e8101670 0x7f90e81982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 --2- 172.21.15.89:0/1197549539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90e8100cb0 0x7f90e8197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:37:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 -- 172.21.15.89:0/1197549539 >> 172.21.15.89:0/1197549539 conn(0x7f90e80fb970 msgr2=0x7f90e80fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:37:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 -- 172.21.15.89:0/1197549539 shutdown_connections 2024-09-15T23:37:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:37:49.468+0000 7f90ef986640 1 -- 172.21.15.89:0/1197549539 wait complete. 2024-09-15T23:37:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:48 smithi120 ceph-mon[35250]: pgmap v763: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 19 op/s 2024-09-15T23:37:51.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:50 smithi174 ceph-mon[35586]: pgmap v764: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.0 KiB/s wr, 12 op/s 2024-09-15T23:37:51.275 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:50 smithi120 ceph-mon[35250]: pgmap v764: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.0 KiB/s wr, 12 op/s 2024-09-15T23:37:51.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:50 smithi089 ceph-mon[30933]: pgmap v764: 129 pgs: 129 active+clean; 2.9 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.0 KiB/s wr, 12 op/s 2024-09-15T23:37:53.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:52 smithi174 ceph-mon[35586]: pgmap v765: 129 pgs: 129 active+clean; 2.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.1 KiB/s wr, 14 op/s 2024-09-15T23:37:53.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:52 smithi089 ceph-mon[30933]: pgmap v765: 129 pgs: 129 active+clean; 2.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.1 KiB/s wr, 14 op/s 2024-09-15T23:37:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:52 smithi120 ceph-mon[35250]: pgmap v765: 129 pgs: 129 active+clean; 2.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.1 KiB/s wr, 14 op/s 2024-09-15T23:37:55.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:54 smithi174 ceph-mon[35586]: pgmap v766: 129 pgs: 129 active+clean; 3.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 15 op/s 2024-09-15T23:37:55.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:54 smithi089 ceph-mon[30933]: pgmap v766: 129 pgs: 129 active+clean; 3.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 15 op/s 2024-09-15T23:37:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:54 smithi120 ceph-mon[35250]: pgmap v766: 129 pgs: 129 active+clean; 3.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 15 op/s 2024-09-15T23:37:57.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:56 smithi089 ceph-mon[30933]: pgmap v767: 129 pgs: 129 active+clean; 3.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 13 op/s 2024-09-15T23:37:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:56 smithi120 ceph-mon[35250]: pgmap v767: 129 pgs: 129 active+clean; 3.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 13 op/s 2024-09-15T23:37:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:56 smithi174 ceph-mon[35586]: pgmap v767: 129 pgs: 129 active+clean; 3.0 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 13 op/s 2024-09-15T23:37:59.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:37:58 smithi089 ceph-mon[30933]: pgmap v768: 129 pgs: 129 active+clean; 3.0 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:37:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:37:58 smithi120 ceph-mon[35250]: pgmap v768: 129 pgs: 129 active+clean; 3.0 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:37:59.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:37:58 smithi174 ceph-mon[35586]: pgmap v768: 129 pgs: 129 active+clean; 3.0 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:38:01.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:00 smithi089 ceph-mon[30933]: pgmap v769: 129 pgs: 129 active+clean; 3.0 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:38:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:00 smithi120 ceph-mon[35250]: pgmap v769: 129 pgs: 129 active+clean; 3.0 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:38:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:00 smithi174 ceph-mon[35586]: pgmap v769: 129 pgs: 129 active+clean; 3.0 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:38:02.562 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.561+0000 7f0e2881e640 1 -- 172.21.15.89:0/3221850909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24101c40 msgr2=0x7f0e241020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.561+0000 7f0e2881e640 1 --2- 172.21.15.89:0/3221850909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24101c40 0x7f0e241020c0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c009900 tx=0x7f0e0c02a410 comp rx=0 tx=0).stop 2024-09-15T23:38:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.561+0000 7f0e2881e640 1 -- 172.21.15.89:0/3221850909 shutdown_connections 2024-09-15T23:38:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.561+0000 7f0e2881e640 1 --2- 172.21.15.89:0/3221850909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e24102600 0x7f0e24108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.561+0000 7f0e2881e640 1 --2- 172.21.15.89:0/3221850909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24101c40 0x7f0e241020c0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.561+0000 7f0e2881e640 1 --2- 172.21.15.89:0/3221850909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e24100a50 0x7f0e24100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.561+0000 7f0e2881e640 1 -- 172.21.15.89:0/3221850909 >> 172.21.15.89:0/3221850909 conn(0x7f0e240fc1b0 msgr2=0x7f0e240fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.561+0000 7f0e2881e640 1 -- 172.21.15.89:0/3221850909 shutdown_connections 2024-09-15T23:38:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.561+0000 7f0e2881e640 1 -- 172.21.15.89:0/3221850909 wait complete. 2024-09-15T23:38:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2881e640 1 Processor -- start 2024-09-15T23:38:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2881e640 1 -- start start 2024-09-15T23:38:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2881e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24100a50 0x7f0e2406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2881e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e24101c40 0x7f0e2419cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2881e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e24102600 0x7f0e2419f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2881e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2410c6d0 con 0x7f0e24100a50 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2881e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2410c550 con 0x7f0e24101c40 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2881e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2410c850 con 0x7f0e24102600 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24100a50 0x7f0e2406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24100a50 0x7f0e2406ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51442/0 (socket says 172.21.15.89:51442) 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.562+0000 7f0e2259c640 1 -- 172.21.15.89:0/4015941373 learned_addr learned my addr 172.21.15.89:0/4015941373 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e21d9b640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e24101c40 0x7f0e2419cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e24102600 0x7f0e2419f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e2259c640 1 -- 172.21.15.89:0/4015941373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e24102600 msgr2=0x7f0e2419f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e2259c640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e24102600 0x7f0e2419f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e2259c640 1 -- 172.21.15.89:0/4015941373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e24101c40 msgr2=0x7f0e2419cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e2259c640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e24101c40 0x7f0e2419cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e2259c640 1 -- 172.21.15.89:0/4015941373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2419fa40 con 0x7f0e24100a50 2024-09-15T23:38:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e24102600 0x7f0e2419f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e2259c640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24100a50 0x7f0e2406ae70 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400e9c0 tx=0x7f0e1400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e1b7fe640 1 -- 172.21.15.89:0/4015941373 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1400cd80 con 0x7f0e24100a50 2024-09-15T23:38:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e2881e640 1 -- 172.21.15.89:0/4015941373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2419fcd0 con 0x7f0e24100a50 2024-09-15T23:38:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e1b7fe640 1 -- 172.21.15.89:0/4015941373 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e140035f0 con 0x7f0e24100a50 2024-09-15T23:38:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e1b7fe640 1 -- 172.21.15.89:0/4015941373 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e14010630 con 0x7f0e24100a50 2024-09-15T23:38:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.563+0000 7f0e2881e640 1 -- 172.21.15.89:0/4015941373 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e241a01a0 con 0x7f0e24100a50 2024-09-15T23:38:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.566+0000 7f0e2b68b740 1 Processor -- start 2024-09-15T23:38:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.566+0000 7f0e2b68b740 1 -- start start 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e1b7fe640 1 -- 172.21.15.89:0/4015941373 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0e140037a0 con 0x7f0e24100a50 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a72db0 0x55c5d6a731b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a736f0 0x55c5d6a781b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a78780 0x55c5d6a7ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c5d69d22f0 con 0x55c5d6a736f0 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c5d697a270 con 0x55c5d6a72db0 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c5d69ce2b0 con 0x55c5d6a78780 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a736f0 0x55c5d6a781b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a736f0 0x55c5d6a781b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51446/0 (socket says 172.21.15.89:51446) 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2259c640 1 -- 172.21.15.89:0/2694251516 learned_addr learned my addr 172.21.15.89:0/2694251516 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e21d9b640 1 --2- 172.21.15.89:0/2694251516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a78780 0x55c5d6a7ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/2694251516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a72db0 0x55c5d6a731b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2259c640 1 -- 172.21.15.89:0/2694251516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a78780 msgr2=0x55c5d6a7ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2259c640 1 --2- 172.21.15.89:0/2694251516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a78780 0x55c5d6a7ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2259c640 1 -- 172.21.15.89:0/2694251516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a72db0 msgr2=0x55c5d6a731b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2259c640 1 --2- 172.21.15.89:0/2694251516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a72db0 0x55c5d6a731b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2259c640 1 -- 172.21.15.89:0/2694251516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5d6979340 con 0x55c5d6a736f0 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/2694251516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a72db0 0x55c5d6a731b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e1b7fe640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0e100a32f0 0x7f0e100a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0e100a32f0 0x7f0e100a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.567+0000 7f0e2259c640 1 --2- 172.21.15.89:0/2694251516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a736f0 0x55c5d6a781b0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f0e140d8270 tx=0x7f0e140d8730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ace92bc14df410 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.568+0000 7f0e1b7fe640 1 -- 172.21.15.89:0/4015941373 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0e140d0210 con 0x7f0e24100a50 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.568+0000 7f0e18ff9640 1 -- 172.21.15.89:0/2694251516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1409c210 con 0x55c5d6a736f0 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.568+0000 7f0e18ff9640 1 -- 172.21.15.89:0/2694251516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1409cb40 con 0x55c5d6a736f0 2024-09-15T23:38:02.575 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.568+0000 7f0e18ff9640 1 -- 172.21.15.89:0/2694251516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1409bda0 con 0x55c5d6a736f0 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 -- 172.21.15.89:0/2694251516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a736f0 msgr2=0x55c5d6a781b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/2694251516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a736f0 0x55c5d6a781b0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f0e140d8270 tx=0x7f0e140d8730 comp rx=0 tx=0).stop 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0e100a32f0 0x7f0e100a57b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c005fd0 tx=0x7f0e1c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 -- 172.21.15.89:0/2694251516 shutdown_connections 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/2694251516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a78780 0x55c5d6a7ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/2694251516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a736f0 0x55c5d6a781b0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/2694251516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a72db0 0x55c5d6a731b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 -- 172.21.15.89:0/2694251516 >> 172.21.15.89:0/2694251516 conn(0x55c5d6a6a010 msgr2=0x55c5d6a6a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 -- 172.21.15.89:0/2694251516 shutdown_connections 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 -- 172.21.15.89:0/2694251516 wait complete. 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 -- start start 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 Processor -- start 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a72db0 0x55c5d6a7d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 0x55c5d6a7d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a84540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c5d6a7b750 con 0x55c5d6a78780 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c5d695b290 con 0x55c5d6a736f0 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c5d6a7b8d0 con 0x55c5d6a72db0 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.569+0000 7f0e2b68b740 10 client.0 _ll_register_callbacks cb 0x55c5d698c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a84540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a84540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51460/0 (socket says 172.21.15.89:51460) 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e22d9d640 1 -- 172.21.15.89:0/174719876 learned_addr learned my addr 172.21.15.89:0/174719876 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e21d9b640 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 0x55c5d6a7d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e22d9d640 1 -- 172.21.15.89:0/174719876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a72db0 msgr2=0x55c5d6a7d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a72db0 0x55c5d6a7d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e22d9d640 1 -- 172.21.15.89:0/174719876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 msgr2=0x55c5d6a7d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 0x55c5d6a7d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e22d9d640 1 -- 172.21.15.89:0/174719876 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5d66e5210 con 0x55c5d6a78780 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e21d9b640 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 0x55c5d6a7d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a84540 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c010ba0 tx=0x7f0e1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f88c9c37cf2b66 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e09ffb640 1 -- 172.21.15.89:0/174719876 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c004d20 con 0x55c5d6a78780 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e09ffb640 1 -- 172.21.15.89:0/174719876 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0e1c01e9a0 con 0x55c5d6a78780 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e2b68b740 1 -- 172.21.15.89:0/174719876 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e1409cb40 con 0x55c5d6a78780 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e09ffb640 1 -- 172.21.15.89:0/174719876 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0e1c01eba0 con 0x55c5d6a78780 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e2b68b740 10 client.20265 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e2b68b740 1 -- 172.21.15.89:0/174719876 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c5d67b1620 con 0x55c5d6a78780 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.570+0000 7f0e09ffb640 1 -- 172.21.15.89:0/174719876 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1409cb40 con 0x55c5d6a78780 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.571+0000 7f0e09ffb640 1 -- 172.21.15.89:0/174719876 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0e1c01c410 con 0x55c5d6a78780 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.571+0000 7f0e2b68b740 10 client.20265 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.571+0000 7f0e2b68b740 10 client.20265 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:02.578 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.571+0000 7f0e2b68b740 20 client.20265 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.571+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c5d6a88d50 0x55c5d6a8b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.571+0000 7f0e2b68b740 4 client.20265 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.571+0000 7f0e2b68b740 1 -- 172.21.15.89:0/174719876 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5d69dfa20 con 0x55c5d6a88d50 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.571+0000 7f0e2259c640 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c5d6a88d50 0x55c5d6a8b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.571+0000 7f0e2259c640 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c5d6a88d50 0x55c5d6a8b1e0 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51b93140e3bdda37 server_cookie=393540ff49a84656 in_seq=0 out_seq=0 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.571+0000 7f0e09ffb640 10 client.20265 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e09ffb640 1 -- 172.21.15.89:0/174719876 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c5d69dfa20 con 0x55c5d6a88d50 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e09ffb640 10 client.20265 handle_command_reply: tid=0 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 client.20265 shutdown 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 -- 172.21.15.89:0/174719876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 msgr2=0x55c5d6a84540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a84540 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c010ba0 tx=0x7f0e1c004a40 comp rx=0 tx=0).stop 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 -- 172.21.15.89:0/174719876 shutdown_connections 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c5d6a88d50 0x55c5d6a8b1e0 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.579 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a84540 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e09ffb640 0 client.20265 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 0x55c5d6a7d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/174719876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a72db0 0x55c5d6a7d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 -- 172.21.15.89:0/174719876 >> 172.21.15.89:0/174719876 conn(0x55c5d6a6a010 msgr2=0x55c5d6a6b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 -- 172.21.15.89:0/174719876 shutdown_connections 2024-09-15T23:38:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.572+0000 7f0e2b68b740 1 -- 172.21.15.89:0/174719876 wait complete. 2024-09-15T23:38:02.580 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.573+0000 7f0e2b68b740 20 client.20265 trim_cache size 0 max 16384 2024-09-15T23:38:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2b68b740 1 Processor -- start 2024-09-15T23:38:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2b68b740 1 -- start start 2024-09-15T23:38:02.592 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a72db0 0x55c5d6a6a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a736f0 0x55c5d6a84500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a78780 0x55c5d6a6a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0e241a85d0 con 0x55c5d6a72db0 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0e241aa110 con 0x55c5d6a78780 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0e241a8ca0 con 0x55c5d6a736f0 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a72db0 0x55c5d6a6a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a72db0 0x55c5d6a6a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51466/0 (socket says 172.21.15.89:51466) 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2259c640 1 -- 172.21.15.89:0/855575437 learned_addr learned my addr 172.21.15.89:0/855575437 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2259c640 1 -- 172.21.15.89:0/855575437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a736f0 msgr2=0x55c5d6a84500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/855575437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a78780 0x55c5d6a6a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e21d9b640 1 --2- 172.21.15.89:0/855575437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a736f0 0x55c5d6a84500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.593 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2259c640 1 --2- 172.21.15.89:0/855575437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a736f0 0x55c5d6a84500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2259c640 1 -- 172.21.15.89:0/855575437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a78780 msgr2=0x55c5d6a6a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2259c640 1 --2- 172.21.15.89:0/855575437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a78780 0x55c5d6a6a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e2259c640 1 -- 172.21.15.89:0/855575437 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5d6a69e30 con 0x55c5d6a72db0 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.587+0000 7f0e21d9b640 1 --2- 172.21.15.89:0/855575437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a736f0 0x55c5d6a84500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e2259c640 1 --2- 172.21.15.89:0/855575437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a72db0 0x55c5d6a6a3f0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f0e1409c3d0 tx=0x7f0e140a07f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a186d7b8b16349e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e0bfff640 1 -- 172.21.15.89:0/855575437 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1409bf80 con 0x55c5d6a72db0 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e0bfff640 1 -- 172.21.15.89:0/855575437 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1409cb40 con 0x55c5d6a72db0 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e0bfff640 1 -- 172.21.15.89:0/855575437 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e140a50b0 con 0x55c5d6a72db0 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e2b68b740 1 -- 172.21.15.89:0/855575437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a72db0 msgr2=0x55c5d6a6a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/855575437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a72db0 0x55c5d6a6a3f0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f0e1409c3d0 tx=0x7f0e140a07f0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e2b68b740 1 -- 172.21.15.89:0/855575437 shutdown_connections 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/855575437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a78780 0x55c5d6a6a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/855575437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a736f0 0x55c5d6a84500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/855575437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a72db0 0x55c5d6a6a3f0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e2b68b740 1 -- 172.21.15.89:0/855575437 >> 172.21.15.89:0/855575437 conn(0x55c5d6a6a010 msgr2=0x55c5d6a7c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e2b68b740 1 -- 172.21.15.89:0/855575437 shutdown_connections 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.588+0000 7f0e2b68b740 1 -- 172.21.15.89:0/855575437 wait complete. 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e2b68b740 1 -- start start 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e2b68b740 1 Processor -- start 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a72db0 0x55c5d6a81b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 0x55c5d6a82070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e2b68b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a83080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0e241a9a40 con 0x55c5d6a78780 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0e241a9370 con 0x55c5d6a736f0 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e2b68b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0e241a65b0 con 0x55c5d6a72db0 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e2b68b740 10 client.0 _ll_register_callbacks cb 0x7f0e1c009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a83080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.595 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a83080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51474/0 (socket says 172.21.15.89:51474) 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e22d9d640 1 -- 172.21.15.89:0/1962096038 learned_addr learned my addr 172.21.15.89:0/1962096038 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e21d9b640 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 0x55c5d6a82070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e22d9d640 1 -- 172.21.15.89:0/1962096038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a72db0 msgr2=0x55c5d6a81b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a72db0 0x55c5d6a81b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e22d9d640 1 -- 172.21.15.89:0/1962096038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 msgr2=0x55c5d6a82070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 0x55c5d6a82070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e22d9d640 1 -- 172.21.15.89:0/1962096038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5d6a83690 con 0x55c5d6a78780 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e21d9b640 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 0x55c5d6a82070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e22d9d640 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a83080 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c017040 tx=0x7f0e1c01ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ea22b3cd73940c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.589+0000 7f0e09ffb640 1 -- 172.21.15.89:0/1962096038 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c009e50 con 0x55c5d6a78780 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e09ffb640 1 -- 172.21.15.89:0/1962096038 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0e1c004ab0 con 0x55c5d6a78780 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e2b68b740 1 -- 172.21.15.89:0/1962096038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e1409cb40 con 0x55c5d6a78780 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e09ffb640 1 -- 172.21.15.89:0/1962096038 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0e1c043050 con 0x55c5d6a78780 2024-09-15T23:38:02.596 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e2b68b740 10 client.20271 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e2b68b740 1 -- 172.21.15.89:0/1962096038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c5d6af3ac0 con 0x55c5d6a78780 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e09ffb640 1 -- 172.21.15.89:0/1962096038 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1409cb40 con 0x55c5d6a78780 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e09ffb640 1 -- 172.21.15.89:0/1962096038 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0e1c03e030 con 0x55c5d6a78780 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e2b68b740 10 client.20271 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e2b68b740 10 client.20271 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e2b68b740 20 client.20271 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c5d6af4390 0x55c5d6af67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e2b68b740 4 client.20271 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e2b68b740 1 -- 172.21.15.89:0/1962096038 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c5d69d9d70 con 0x55c5d6af4390 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.590+0000 7f0e2259c640 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c5d6af4390 0x55c5d6af67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.591+0000 7f0e2259c640 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c5d6af4390 0x55c5d6af67b0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f9fe25df50ff873 server_cookie=5865f78dbb289a52 in_seq=0 out_seq=0 2024-09-15T23:38:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.591+0000 7f0e09ffb640 10 client.20271 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e09ffb640 1 -- 172.21.15.89:0/1962096038 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c5d69d9d70 con 0x55c5d6af4390 2024-09-15T23:38:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e09ffb640 10 client.20271 handle_command_reply: tid=0 2024-09-15T23:38:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e2b68b740 1 client.20271 shutdown 2024-09-15T23:38:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e2b68b740 1 -- 172.21.15.89:0/1962096038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 msgr2=0x55c5d6a83080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a83080 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c017040 tx=0x7f0e1c01ea20 comp rx=0 tx=0).stop 2024-09-15T23:38:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e2b68b740 1 -- 172.21.15.89:0/1962096038 shutdown_connections 2024-09-15T23:38:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c5d6af4390 0x55c5d6af67b0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5d6a78780 0x55c5d6a83080 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e09ffb640 0 client.20271 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c5d6a736f0 0x55c5d6a82070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e2b68b740 1 --2- 172.21.15.89:0/1962096038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5d6a72db0 0x55c5d6a81b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e2b68b740 1 -- 172.21.15.89:0/1962096038 >> 172.21.15.89:0/1962096038 conn(0x55c5d6a6a010 msgr2=0x55c5d6a689a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.605+0000 7f0e2b68b740 1 -- 172.21.15.89:0/1962096038 shutdown_connections 2024-09-15T23:38:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2b68b740 1 -- 172.21.15.89:0/1962096038 wait complete. 2024-09-15T23:38:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2b68b740 20 client.20271 trim_cache size 0 max 16384 2024-09-15T23:38:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:38:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:38:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:38:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:38:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 -- 172.21.15.89:0/4015941373 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0e100a32f0 msgr2=0x7f0e100a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0e100a32f0 0x7f0e100a57b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c005fd0 tx=0x7f0e1c009500 comp rx=0 tx=0).stop 2024-09-15T23:38:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 -- 172.21.15.89:0/4015941373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24100a50 msgr2=0x7f0e2406ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24100a50 0x7f0e2406ae70 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400e9c0 tx=0x7f0e1400ee80 comp rx=0 tx=0).stop 2024-09-15T23:38:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 -- 172.21.15.89:0/4015941373 shutdown_connections 2024-09-15T23:38:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0e100a32f0 0x7f0e100a57b0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0e24102600 0x7f0e2419f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e24101c40 0x7f0e2419cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 --2- 172.21.15.89:0/4015941373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24100a50 0x7f0e2406ae70 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 -- 172.21.15.89:0/4015941373 >> 172.21.15.89:0/4015941373 conn(0x7f0e240fc1b0 msgr2=0x7f0e24104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 -- 172.21.15.89:0/4015941373 shutdown_connections 2024-09-15T23:38:02.610 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:02.606+0000 7f0e2881e640 1 -- 172.21.15.89:0/4015941373 wait complete. 2024-09-15T23:38:03.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:02 smithi089 ceph-mon[30933]: pgmap v770: 129 pgs: 129 active+clean; 3.0 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:38:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:02 smithi120 ceph-mon[35250]: pgmap v770: 129 pgs: 129 active+clean; 3.0 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:38:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:02 smithi174 ceph-mon[35586]: pgmap v770: 129 pgs: 129 active+clean; 3.0 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:38:05.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:05 smithi089 ceph-mon[30933]: pgmap v771: 129 pgs: 129 active+clean; 3.1 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-15T23:38:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:05 smithi120 ceph-mon[35250]: pgmap v771: 129 pgs: 129 active+clean; 3.1 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-15T23:38:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:05 smithi174 ceph-mon[35586]: pgmap v771: 129 pgs: 129 active+clean; 3.1 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 18 op/s 2024-09-15T23:38:07.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:07 smithi089 ceph-mon[30933]: pgmap v772: 129 pgs: 129 active+clean; 3.1 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T23:38:07.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:07 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:38:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:07 smithi120 ceph-mon[35250]: pgmap v772: 129 pgs: 129 active+clean; 3.1 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T23:38:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:07 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:38:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:07 smithi174 ceph-mon[35586]: pgmap v772: 129 pgs: 129 active+clean; 3.1 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T23:38:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:07 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:38:08.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:38:08.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:38:08.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:38:08.422 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:38:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:38:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:38:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:38:08.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:38:08.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:38:08.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:38:08.683 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:09.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:09 smithi089 ceph-mon[30933]: pgmap v773: 129 pgs: 129 active+clean; 3.1 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:38:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.165+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/2299735231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9db81000d0 msgr2=0x7f9db8100530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.165+0000 7f9dbe5c9640 1 --2- 172.21.15.89:0/2299735231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9db81000d0 0x7f9db8100530 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f9dac009900 tx=0x7f9dac02a410 comp rx=0 tx=0).stop 2024-09-15T23:38:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.165+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/2299735231 shutdown_connections 2024-09-15T23:38:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.165+0000 7f9dbe5c9640 1 --2- 172.21.15.89:0/2299735231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db8100a70 0x7f9db810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.165+0000 7f9dbe5c9640 1 --2- 172.21.15.89:0/2299735231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9db81000d0 0x7f9db8100530 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.165+0000 7f9dbe5c9640 1 --2- 172.21.15.89:0/2299735231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db8106ba0 0x7f9db8106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.165+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/2299735231 >> 172.21.15.89:0/2299735231 conn(0x7f9db8077210 msgr2=0x7f9db80fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.165+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/2299735231 shutdown_connections 2024-09-15T23:38:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.165+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/2299735231 wait complete. 2024-09-15T23:38:09.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbe5c9640 1 Processor -- start 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbe5c9640 1 -- start start 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbe5c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db81000d0 0x7f9db81984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbe5c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db8100a70 0x7f9db8198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbe5c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9db8106ba0 0x7f9db819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbe5c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9db8112090 con 0x7f9db8106ba0 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbe5c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9db8111f10 con 0x7f9db81000d0 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbe5c9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9db8112210 con 0x7f9db8100a70 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbcb66640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9db8106ba0 0x7f9db819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbcb66640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9db8106ba0 0x7f9db819cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40138/0 (socket says 172.21.15.89:40138) 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9dbcb66640 1 -- 172.21.15.89:0/1442708566 learned_addr learned my addr 172.21.15.89:0/1442708566 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.166+0000 7f9db7fff640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db81000d0 0x7f9db81984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9db77fe640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db8100a70 0x7f9db8198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9dbcb66640 1 -- 172.21.15.89:0/1442708566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db8100a70 msgr2=0x7f9db8198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:09.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9dbcb66640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db8100a70 0x7f9db8198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9dbcb66640 1 -- 172.21.15.89:0/1442708566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db81000d0 msgr2=0x7f9db81984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9dbcb66640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db81000d0 0x7f9db81984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9dbcb66640 1 -- 172.21.15.89:0/1442708566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db819d4a0 con 0x7f9db8106ba0 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9db77fe640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db8100a70 0x7f9db8198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9db7fff640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db81000d0 0x7f9db81984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9dbcb66640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9db8106ba0 0x7f9db819cdc0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f9db000eaa0 tx=0x7f9db000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9db57fa640 1 -- 172.21.15.89:0/1442708566 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db000cdb0 con 0x7f9db8106ba0 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/1442708566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9db819d790 con 0x7f9db8106ba0 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9db57fa640 1 -- 172.21.15.89:0/1442708566 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9db0005020 con 0x7f9db8106ba0 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9db57fa640 1 -- 172.21.15.89:0/1442708566 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db00116d0 con 0x7f9db8106ba0 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.167+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/1442708566 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9db81a5070 con 0x7f9db8106ba0 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.169+0000 7f9db57fa640 1 -- 172.21.15.89:0/1442708566 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9db00054d0 con 0x7f9db8106ba0 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9dc09a2740 1 Processor -- start 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9dc09a2740 1 -- start start 2024-09-15T23:38:09.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b69b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6a6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6a8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55622b6012e0 con 0x55622b6a6680 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55622b5a8ce0 con 0x55622b69b4a0 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55622b5fd2a0 con 0x55622b69bdc0 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9db77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6a8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9db77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6a8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40144/0 (socket says 172.21.15.89:40144) 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9db77fe640 1 -- 172.21.15.89:0/3347029360 learned_addr learned my addr 172.21.15.89:0/3347029360 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9db77fe640 1 --2- 172.21.15.89:0/3347029360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b69b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9db7fff640 1 --2- 172.21.15.89:0/3347029360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.170+0000 7f9db57fa640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9da00a3240 0x7f9da00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9dbcb66640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9da00a3240 0x7f9da00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9db77fe640 1 -- 172.21.15.89:0/3347029360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 msgr2=0x55622b6a6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9db77fe640 1 --2- 172.21.15.89:0/3347029360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6a6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9db77fe640 1 -- 172.21.15.89:0/3347029360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 msgr2=0x55622b69b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9db77fe640 1 --2- 172.21.15.89:0/3347029360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b69b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9db77fe640 1 -- 172.21.15.89:0/3347029360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55622b58d440 con 0x55622b6a6680 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9db57fa640 1 -- 172.21.15.89:0/1442708566 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9db00cfff0 con 0x7f9db8106ba0 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9db77fe640 1 --2- 172.21.15.89:0/3347029360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6a8a70 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f9dac005770 tx=0x7f9dac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63c5e93d1dc7f06 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9d9e7fc640 1 -- 172.21.15.89:0/3347029360 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac035040 con 0x55622b6a6680 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9d9e7fc640 1 -- 172.21.15.89:0/3347029360 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dac0038c0 con 0x55622b6a6680 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9dbcb66640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9da00a3240 0x7f9da00a5700 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9db0098bf0 tx=0x7f9db0091700 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.171+0000 7f9d9e7fc640 1 -- 172.21.15.89:0/3347029360 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac004920 con 0x55622b6a6680 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 -- 172.21.15.89:0/3347029360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 msgr2=0x55622b6a8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/3347029360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6a8a70 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f9dac005770 tx=0x7f9dac000bc0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 -- 172.21.15.89:0/3347029360 shutdown_connections 2024-09-15T23:38:09.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/3347029360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6a8a70 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/3347029360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6a6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/3347029360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b69b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 -- 172.21.15.89:0/3347029360 >> 172.21.15.89:0/3347029360 conn(0x55622b697a30 msgr2=0x55622b697e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 -- 172.21.15.89:0/3347029360 shutdown_connections 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 -- 172.21.15.89:0/3347029360 wait complete. 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 -- start start 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.172+0000 7f9dc09a2740 1 Processor -- start 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69b4a0 0x55622b6a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6a3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b6a6680 0x55622b6a3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55622b6a9250 con 0x55622b69b4a0 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55622b5bc050 con 0x55622b6a6680 2024-09-15T23:38:09.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55622b6a93d0 con 0x55622b69bdc0 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9dc09a2740 10 client.0 _ll_register_callbacks cb 0x55622b5fc160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9db77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69b4a0 0x55622b6a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9db77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69b4a0 0x55622b6a5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40150/0 (socket says 172.21.15.89:40150) 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9db77fe640 1 -- 172.21.15.89:0/817249280 learned_addr learned my addr 172.21.15.89:0/817249280 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9db7fff640 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6a3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9db77fe640 1 -- 172.21.15.89:0/817249280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 msgr2=0x55622b6a3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9db77fe640 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6a3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9db77fe640 1 -- 172.21.15.89:0/817249280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b6a6680 msgr2=0x55622b6a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9db77fe640 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b6a6680 0x55622b6a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9db77fe640 1 -- 172.21.15.89:0/817249280 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55622b3723c0 con 0x55622b69b4a0 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9db77fe640 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69b4a0 0x55622b6a5f70 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f9dac004460 tx=0x7f9dac004000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa36df36c818250a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9d977fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9d977fe640 1 -- 172.21.15.89:0/817249280 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac04e060 con 0x55622b69b4a0 2024-09-15T23:38:09.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9d977fe640 1 -- 172.21.15.89:0/817249280 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9dac0046c0 con 0x55622b69b4a0 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.173+0000 7f9dc09a2740 1 -- 172.21.15.89:0/817249280 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dac0038c0 con 0x55622b69b4a0 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9d977fe640 1 -- 172.21.15.89:0/817249280 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9dac035040 con 0x55622b69b4a0 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9dc09a2740 10 client.20283 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9dc09a2740 1 -- 172.21.15.89:0/817249280 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55622b6b4bc0 con 0x55622b69b4a0 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9d977fe640 1 -- 172.21.15.89:0/817249280 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dac0038c0 con 0x55622b69b4a0 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9d977fe640 1 -- 172.21.15.89:0/817249280 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9dac03a040 con 0x55622b69b4a0 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9dc09a2740 10 client.20283 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9dc09a2740 10 client.20283 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9dc09a2740 20 client.20283 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55622b6b5f10 0x55622b6b8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9dc09a2740 4 client.20283 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9dc09a2740 1 -- 172.21.15.89:0/817249280 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55622b60e850 con 0x55622b6b5f10 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.174+0000 7f9db7fff640 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55622b6b5f10 0x55622b6b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.175+0000 7f9d977fe640 10 client.20283 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.175+0000 7f9db7fff640 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55622b6b5f10 0x55622b6b8300 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33f87eef2ec9b6b1 server_cookie=6b7eebe005658542 in_seq=0 out_seq=0 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.175+0000 7f9d977fe640 1 -- 172.21.15.89:0/817249280 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55622b60e850 con 0x55622b6b5f10 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.175+0000 7f9d977fe640 10 client.20283 handle_command_reply: tid=0 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 client.20283 shutdown 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 -- 172.21.15.89:0/817249280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69b4a0 msgr2=0x55622b6a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69b4a0 0x55622b6a5f70 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f9dac004460 tx=0x7f9dac004000 comp rx=0 tx=0).stop 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 -- 172.21.15.89:0/817249280 shutdown_connections 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55622b6b5f10 0x55622b6b8300 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b6a6680 0x55622b6a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6a3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9d977fe640 0 client.20283 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/817249280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69b4a0 0x55622b6a5f70 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 -- 172.21.15.89:0/817249280 >> 172.21.15.89:0/817249280 conn(0x55622b697a30 msgr2=0x55622b69d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:09.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 -- 172.21.15.89:0/817249280 shutdown_connections 2024-09-15T23:38:09.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 1 -- 172.21.15.89:0/817249280 wait complete. 2024-09-15T23:38:09.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.176+0000 7f9dc09a2740 20 client.20283 trim_cache size 0 max 16384 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9dc09a2740 1 Processor -- start 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9dc09a2740 1 -- start start 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b6bb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69bdc0 0x55622b6bb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b6a6680 0x55622b6a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9db81aef20 con 0x55622b69bdc0 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9db81b0a60 con 0x55622b69b4a0 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9db81af5f0 con 0x55622b6a6680 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9db7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69bdc0 0x55622b6bb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9db7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69bdc0 0x55622b6bb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40152/0 (socket says 172.21.15.89:40152) 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9db7fff640 1 -- 172.21.15.89:0/2612199940 learned_addr learned my addr 172.21.15.89:0/2612199940 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9db77fe640 1 --2- 172.21.15.89:0/2612199940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b6a6680 0x55622b6a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.195 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9db77fe640 1 --2- 172.21.15.89:0/2612199940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b6bb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9db7fff640 1 -- 172.21.15.89:0/2612199940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b6a6680 msgr2=0x55622b6a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:09.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9db7fff640 1 --2- 172.21.15.89:0/2612199940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b6a6680 0x55622b6a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9db7fff640 1 -- 172.21.15.89:0/2612199940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 msgr2=0x55622b6bb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:09.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9db7fff640 1 --2- 172.21.15.89:0/2612199940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b6bb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.193+0000 7f9db7fff640 1 -- 172.21.15.89:0/2612199940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55622b697860 con 0x55622b69bdc0 2024-09-15T23:38:09.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9db7fff640 1 --2- 172.21.15.89:0/2612199940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69bdc0 0x55622b6bb860 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x55622b698890 tx=0x7f9da800c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3cafb72d798563 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:09.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9d9dffb640 1 -- 172.21.15.89:0/2612199940 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da8021070 con 0x55622b69bdc0 2024-09-15T23:38:09.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9d9dffb640 1 -- 172.21.15.89:0/2612199940 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9da80047a0 con 0x55622b69bdc0 2024-09-15T23:38:09.196 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9d9dffb640 1 -- 172.21.15.89:0/2612199940 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da801c640 con 0x55622b69bdc0 2024-09-15T23:38:09.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9dc09a2740 1 -- 172.21.15.89:0/2612199940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69bdc0 msgr2=0x55622b6bb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:09.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/2612199940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69bdc0 0x55622b6bb860 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x55622b698890 tx=0x7f9da800c4f0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9dc09a2740 1 -- 172.21.15.89:0/2612199940 shutdown_connections 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/2612199940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b6a6680 0x55622b6a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/2612199940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b69bdc0 0x55622b6bb860 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/2612199940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b6bb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.194+0000 7f9dc09a2740 1 -- 172.21.15.89:0/2612199940 >> 172.21.15.89:0/2612199940 conn(0x55622b697a30 msgr2=0x55622b69e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 1 -- 172.21.15.89:0/2612199940 shutdown_connections 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 1 -- 172.21.15.89:0/2612199940 wait complete. 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 1 -- start start 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 1 Processor -- start 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b6bb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6b8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6b90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.200 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9db81b0390 con 0x55622b6a6680 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9db81afcc0 con 0x55622b69b4a0 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9db81acf00 con 0x55622b69bdc0 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.195+0000 7f9dc09a2740 10 client.0 _ll_register_callbacks cb 0x7f9dac0041c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6b90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6b90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40158/0 (socket says 172.21.15.89:40158) 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db77fe640 1 -- 172.21.15.89:0/1311093820 learned_addr learned my addr 172.21.15.89:0/1311093820 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db77fe640 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b6bb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db7fff640 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6b8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db77fe640 1 -- 172.21.15.89:0/1311093820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 msgr2=0x55622b6b8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db77fe640 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6b8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db77fe640 1 -- 172.21.15.89:0/1311093820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 msgr2=0x55622b6bb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db77fe640 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b6bb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db77fe640 1 -- 172.21.15.89:0/1311093820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55622b6986c0 con 0x55622b6a6680 2024-09-15T23:38:09.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db7fff640 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6b8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9db77fe640 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6b90e0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f9dac0096c0 tx=0x7f9dac003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9a5509342131af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9d977fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9d977fe640 1 -- 172.21.15.89:0/1311093820 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dac060060 con 0x55622b6a6680 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9d977fe640 1 -- 172.21.15.89:0/1311093820 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9dac0048c0 con 0x55622b6a6680 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9dc09a2740 1 -- 172.21.15.89:0/1311093820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9da80047a0 con 0x55622b6a6680 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9d977fe640 1 -- 172.21.15.89:0/1311093820 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9dac04e070 con 0x55622b6a6680 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9dc09a2740 10 client.20289 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.196+0000 7f9dc09a2740 1 -- 172.21.15.89:0/1311093820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55622b720460 con 0x55622b6a6680 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9d977fe640 1 -- 172.21.15.89:0/1311093820 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9da80047a0 con 0x55622b6a6680 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9d977fe640 1 -- 172.21.15.89:0/1311093820 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9dac03a040 con 0x55622b6a6680 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9dc09a2740 10 client.20289 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9dc09a2740 10 client.20289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9dc09a2740 20 client.20289 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:09.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55622b6b5f10 0x55622b722940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9dc09a2740 4 client.20289 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9dc09a2740 1 -- 172.21.15.89:0/1311093820 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55622b608d60 con 0x55622b6b5f10 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9db7fff640 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55622b6b5f10 0x55622b722940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9d977fe640 10 client.20289 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.197+0000 7f9db7fff640 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55622b6b5f10 0x55622b722940 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba74fb0542472c8a server_cookie=45351284a9d27563 in_seq=0 out_seq=0 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9d977fe640 1 -- 172.21.15.89:0/1311093820 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55622b608d60 con 0x55622b6b5f10 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9d977fe640 10 client.20289 handle_command_reply: tid=0 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 client.20289 shutdown 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 -- 172.21.15.89:0/1311093820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 msgr2=0x55622b6b90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6b90e0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f9dac0096c0 tx=0x7f9dac003da0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 -- 172.21.15.89:0/1311093820 shutdown_connections 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55622b6b5f10 0x55622b722940 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55622b6a6680 0x55622b6b90e0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55622b69bdc0 0x55622b6b8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 --2- 172.21.15.89:0/1311093820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55622b69b4a0 0x55622b6bb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 -- 172.21.15.89:0/1311093820 >> 172.21.15.89:0/1311093820 conn(0x55622b697a30 msgr2=0x55622b696350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:09.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9d977fe640 0 client.20289 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:09.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 -- 172.21.15.89:0/1311093820 shutdown_connections 2024-09-15T23:38:09.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 1 -- 172.21.15.89:0/1311093820 wait complete. 2024-09-15T23:38:09.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.198+0000 7f9dc09a2740 20 client.20289 trim_cache size 0 max 16384 2024-09-15T23:38:09.204 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:38:09.204 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:38:09.204 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:38:09.204 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:38:09.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/1442708566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9da00a3240 msgr2=0x7f9da00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:09.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9da00a3240 0x7f9da00a5700 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9db0098bf0 tx=0x7f9db0091700 comp rx=0 tx=0).stop 2024-09-15T23:38:09.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/1442708566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9db8106ba0 msgr2=0x7f9db819cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:09.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9db8106ba0 0x7f9db819cdc0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f9db000eaa0 tx=0x7f9db000ef60 comp rx=0 tx=0).stop 2024-09-15T23:38:09.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/1442708566 shutdown_connections 2024-09-15T23:38:09.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9da00a3240 0x7f9da00a5700 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9db8106ba0 0x7f9db819cdc0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9db8100a70 0x7f9db8198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 --2- 172.21.15.89:0/1442708566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9db81000d0 0x7f9db81984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:09.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/1442708566 >> 172.21.15.89:0/1442708566 conn(0x7f9db8077210 msgr2=0x7f9db8104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:09.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/1442708566 shutdown_connections 2024-09-15T23:38:09.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:09.199+0000 7f9dbe5c9640 1 -- 172.21.15.89:0/1442708566 wait complete. 2024-09-15T23:38:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:09 smithi120 ceph-mon[35250]: pgmap v773: 129 pgs: 129 active+clean; 3.1 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:38:09.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:09 smithi174 ceph-mon[35586]: pgmap v773: 129 pgs: 129 active+clean; 3.1 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:38:10.216 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:38:10.216 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:38:10.308 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.5 2024-09-15T23:38:10.309 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:38:10.474 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:11.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.970+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/1859257315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0101de0 msgr2=0x7fb0e01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.970+0000 7fb0e7c6a640 1 --2- 172.21.15.89:0/1859257315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0101de0 0x7fb0e01021c0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009900 tx=0x7fb0dc02a410 comp rx=0 tx=0).stop 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.970+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/1859257315 shutdown_connections 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.970+0000 7fb0e7c6a640 1 --2- 172.21.15.89:0/1859257315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e0106f10 0x7fb0e010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.970+0000 7fb0e7c6a640 1 --2- 172.21.15.89:0/1859257315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0e0102790 0x7fb0e01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.970+0000 7fb0e7c6a640 1 --2- 172.21.15.89:0/1859257315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0101de0 0x7fb0e01021c0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.970+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/1859257315 >> 172.21.15.89:0/1859257315 conn(0x7fb0e00fd8c0 msgr2=0x7fb0e00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.970+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/1859257315 shutdown_connections 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.970+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/1859257315 wait complete. 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e7c6a640 1 Processor -- start 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e7c6a640 1 -- start start 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e7c6a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e0101de0 0x7fb0e019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:11.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e7c6a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0e0102790 0x7fb0e019ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e7c6a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0106f10 0x7fb0e019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e7c6a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e0111f70 con 0x7fb0e0106f10 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e7c6a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e0111df0 con 0x7fb0e0101de0 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e7c6a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e01120f0 con 0x7fb0e0102790 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e6207640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0106f10 0x7fb0e019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e6207640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0106f10 0x7fb0e019ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40168/0 (socket says 172.21.15.89:40168) 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.971+0000 7fb0e6207640 1 -- 172.21.15.89:0/450901888 learned_addr learned my addr 172.21.15.89:0/450901888 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0e5205640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0e0102790 0x7fb0e019ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0e5a06640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e0101de0 0x7fb0e019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0e6207640 1 -- 172.21.15.89:0/450901888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0e0102790 msgr2=0x7fb0e019ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0e6207640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0e0102790 0x7fb0e019ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0e6207640 1 -- 172.21.15.89:0/450901888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e0101de0 msgr2=0x7fb0e019a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0e6207640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e0101de0 0x7fb0e019a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0e6207640 1 -- 172.21.15.89:0/450901888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e019f5e0 con 0x7fb0e0106f10 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0e6207640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0106f10 0x7fb0e019ef00 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800eaa0 tx=0x7fb0d800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0d6ffd640 1 -- 172.21.15.89:0/450901888 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d800cdb0 con 0x7fb0e0106f10 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0d6ffd640 1 -- 172.21.15.89:0/450901888 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d8005020 con 0x7fb0e0106f10 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0d6ffd640 1 -- 172.21.15.89:0/450901888 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d80116b0 con 0x7fb0e0106f10 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/450901888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e019f8d0 con 0x7fb0e0106f10 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.972+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/450901888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e01a71b0 con 0x7fb0e0106f10 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.973+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/450901888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e01021c0 con 0x7fb0e0106f10 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.975+0000 7fb0d6ffd640 1 -- 172.21.15.89:0/450901888 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb0d8005560 con 0x7fb0e0106f10 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.975+0000 7fb0d6ffd640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0cc0a32c0 0x7fb0cc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.976+0000 7fb0e5a06640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0cc0a32c0 0x7fb0cc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.976+0000 7fb0d6ffd640 1 -- 172.21.15.89:0/450901888 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb0d80d0320 con 0x7fb0e0106f10 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.976+0000 7fb0e5a06640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0cc0a32c0 0x7fb0cc0a5780 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc005ab0 tx=0x7fb0dc005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:11.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:10.979+0000 7fb0d6ffd640 1 -- 172.21.15.89:0/450901888 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb0d8016030 con 0x7fb0e0106f10 2024-09-15T23:38:11.050 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:11.171 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:11 smithi089 ceph-mon[30933]: pgmap v774: 129 pgs: 129 active+clean; 3.1 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:38:11.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.188+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/450901888 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0e019b990 con 0x7fb0e0106f10 2024-09-15T23:38:11.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.189+0000 7fb0d6ffd640 1 -- 172.21.15.89:0/450901888 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fb0e019b990 con 0x7fb0e0106f10 2024-09-15T23:38:11.190 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:38:11.190 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:38:11.190 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:38:11.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/450901888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0cc0a32c0 msgr2=0x7fb0cc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:11.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0cc0a32c0 0x7fb0cc0a5780 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc005ab0 tx=0x7fb0dc005a20 comp rx=0 tx=0).stop 2024-09-15T23:38:11.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/450901888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0106f10 msgr2=0x7fb0e019ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:11.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0106f10 0x7fb0e019ef00 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800eaa0 tx=0x7fb0d800ef60 comp rx=0 tx=0).stop 2024-09-15T23:38:11.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/450901888 shutdown_connections 2024-09-15T23:38:11.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb0cc0a32c0 0x7fb0cc0a5780 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0e0106f10 0x7fb0e019ef00 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0e0102790 0x7fb0e019ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 --2- 172.21.15.89:0/450901888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0e0101de0 0x7fb0e019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/450901888 >> 172.21.15.89:0/450901888 conn(0x7fb0e00fd8c0 msgr2=0x7fb0e00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:11.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/450901888 shutdown_connections 2024-09-15T23:38:11.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.191+0000 7fb0e7c6a640 1 -- 172.21.15.89:0/450901888 wait complete. 2024-09-15T23:38:11.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:11 smithi120 ceph-mon[35250]: pgmap v774: 129 pgs: 129 active+clean; 3.1 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:38:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:11 smithi174 ceph-mon[35586]: pgmap v774: 129 pgs: 129 active+clean; 3.1 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:38:11.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.719+0000 7fcec3090640 1 -- 172.21.15.89:0/1985905630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcebc102570 msgr2=0x7fcebc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:11.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.719+0000 7fcec3090640 1 --2- 172.21.15.89:0/1985905630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcebc102570 0x7fcebc1029b0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009900 tx=0x7fcea802a480 comp rx=0 tx=0).stop 2024-09-15T23:38:11.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.719+0000 7fcec3090640 1 -- 172.21.15.89:0/1985905630 shutdown_connections 2024-09-15T23:38:11.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.719+0000 7fcec3090640 1 --2- 172.21.15.89:0/1985905630 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcebc102ef0 0x7fcebc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.719+0000 7fcec3090640 1 --2- 172.21.15.89:0/1985905630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcebc102570 0x7fcebc1029b0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.719+0000 7fcec3090640 1 --2- 172.21.15.89:0/1985905630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc100e20 0x7fcebc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.719+0000 7fcec3090640 1 -- 172.21.15.89:0/1985905630 >> 172.21.15.89:0/1985905630 conn(0x7fcebc0fc7d0 msgr2=0x7fcebc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:11.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.719+0000 7fcec3090640 1 -- 172.21.15.89:0/1985905630 shutdown_connections 2024-09-15T23:38:11.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.719+0000 7fcec3090640 1 -- 172.21.15.89:0/1985905630 wait complete. 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.720+0000 7fcec3090640 1 Processor -- start 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.720+0000 7fcec3090640 1 -- start start 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.720+0000 7fcec3090640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcebc100e20 0x7fcebc0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.720+0000 7fcec3090640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcebc102570 0x7fcebc0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.720+0000 7fcec3090640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc102ef0 0x7fcebc0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.720+0000 7fcec3090640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc111020 con 0x7fcebc102570 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.720+0000 7fcec3090640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc110ea0 con 0x7fcebc102ef0 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.720+0000 7fcec3090640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1111a0 con 0x7fcebc100e20 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcebc102570 0x7fcebc0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcebc102570 0x7fcebc0fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40190/0 (socket says 172.21.15.89:40190) 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcebbfff640 1 -- 172.21.15.89:0/4067054439 learned_addr learned my addr 172.21.15.89:0/4067054439 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:11.724 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcec0e2c640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcebc100e20 0x7fcebc0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcec162d640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc102ef0 0x7fcebc0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcebbfff640 1 -- 172.21.15.89:0/4067054439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcebc100e20 msgr2=0x7fcebc0fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcebbfff640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcebc100e20 0x7fcebc0fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcebbfff640 1 -- 172.21.15.89:0/4067054439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc102ef0 msgr2=0x7fcebc0fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcebbfff640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc102ef0 0x7fcebc0fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcebbfff640 1 -- 172.21.15.89:0/4067054439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc1a6bd0 con 0x7fcebc102570 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcec162d640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc102ef0 0x7fcebc0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcebbfff640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcebc102570 0x7fcebc0fd1c0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fcea8006010 tx=0x7fcea8003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fceb9ffb640 1 -- 172.21.15.89:0/4067054439 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea803f070 con 0x7fcebc102570 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fcec3090640 1 -- 172.21.15.89:0/4067054439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcebc1a6e60 con 0x7fcebc102570 2024-09-15T23:38:11.725 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fceb9ffb640 1 -- 172.21.15.89:0/4067054439 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcea8031e30 con 0x7fcebc102570 2024-09-15T23:38:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.721+0000 7fceb9ffb640 1 -- 172.21.15.89:0/4067054439 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8035030 con 0x7fcebc102570 2024-09-15T23:38:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.722+0000 7fcec3090640 1 -- 172.21.15.89:0/4067054439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcebc1a72c0 con 0x7fcebc102570 2024-09-15T23:38:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.722+0000 7fcec3090640 1 -- 172.21.15.89:0/4067054439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcebc101280 con 0x7fcebc102570 2024-09-15T23:38:11.726 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.724+0000 7fceb9ffb640 1 -- 172.21.15.89:0/4067054439 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcea8042070 con 0x7fcebc102570 2024-09-15T23:38:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.724+0000 7fceb9ffb640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fceac0a3240 0x7fceac0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.724+0000 7fcec0e2c640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fceac0a3240 0x7fceac0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.725+0000 7fceb9ffb640 1 -- 172.21.15.89:0/4067054439 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcea80f0c50 con 0x7fcebc102570 2024-09-15T23:38:11.727 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.725+0000 7fcec0e2c640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fceac0a3240 0x7fceac0a5700 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009890 tx=0x7fceb0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:11.728 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.727+0000 7fceb9ffb640 1 -- 172.21.15.89:0/4067054439 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcea80bd610 con 0x7fcebc102570 2024-09-15T23:38:11.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.931+0000 7fcec3090640 1 -- 172.21.15.89:0/4067054439 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcebc10a0a0 con 0x7fcebc102570 2024-09-15T23:38:11.933 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.932+0000 7fceb9ffb640 1 -- 172.21.15.89:0/4067054439 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fcea80bcf10 con 0x7fcebc102570 2024-09-15T23:38:11.933 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:38:11.933 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:38:11.934 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:38:11.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 -- 172.21.15.89:0/4067054439 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fceac0a3240 msgr2=0x7fceac0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:11.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fceac0a3240 0x7fceac0a5700 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009890 tx=0x7fceb0008040 comp rx=0 tx=0).stop 2024-09-15T23:38:11.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 -- 172.21.15.89:0/4067054439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcebc102570 msgr2=0x7fcebc0fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:11.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcebc102570 0x7fcebc0fd1c0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fcea8006010 tx=0x7fcea8003c80 comp rx=0 tx=0).stop 2024-09-15T23:38:11.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 -- 172.21.15.89:0/4067054439 shutdown_connections 2024-09-15T23:38:11.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fceac0a3240 0x7fceac0a5700 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcebc102ef0 0x7fcebc0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcebc102570 0x7fcebc0fd1c0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 --2- 172.21.15.89:0/4067054439 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcebc100e20 0x7fcebc0fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:11.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 -- 172.21.15.89:0/4067054439 >> 172.21.15.89:0/4067054439 conn(0x7fcebc0fc7d0 msgr2=0x7fcebc06a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:11.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 -- 172.21.15.89:0/4067054439 shutdown_connections 2024-09-15T23:38:11.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:11.934+0000 7fcec3090640 1 -- 172.21.15.89:0/4067054439 wait complete. 2024-09-15T23:38:12.154 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:12 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/450901888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:12.154 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:12 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/4067054439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:12.181 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:38:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:12 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/450901888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:12.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:12 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/4067054439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:12.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:12 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/450901888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:12.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:12 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/4067054439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:12.861 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 146.5 --await-for 45.0' 2024-09-15T23:38:12.862 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 146.5 --await-for 45.0 2024-09-15T23:38:12.866 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:13.116 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:13 smithi089 ceph-mon[30933]: pgmap v775: 129 pgs: 129 active+clean; 3.1 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:38:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.347+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/2939306691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a4100e20 msgr2=0x7fc1a4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.347+0000 7fc1a8ba4640 1 --2- 172.21.15.89:0/2939306691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a4100e20 0x7fc1a4101200 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fc194009900 tx=0x7fc19402a480 comp rx=0 tx=0).stop 2024-09-15T23:38:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.348+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/2939306691 shutdown_connections 2024-09-15T23:38:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.348+0000 7fc1a8ba4640 1 --2- 172.21.15.89:0/2939306691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a4102ef0 0x7fc1a410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.348+0000 7fc1a8ba4640 1 --2- 172.21.15.89:0/2939306691 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1a4102570 0x7fc1a41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.348+0000 7fc1a8ba4640 1 --2- 172.21.15.89:0/2939306691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a4100e20 0x7fc1a4101200 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.348+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/2939306691 >> 172.21.15.89:0/2939306691 conn(0x7fc1a40fc7d0 msgr2=0x7fc1a40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.348+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/2939306691 shutdown_connections 2024-09-15T23:38:13.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.348+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/2939306691 wait complete. 2024-09-15T23:38:13.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.348+0000 7fc1a8ba4640 1 Processor -- start 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a8ba4640 1 -- start start 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a8ba4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a4100e20 0x7fc1a41970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a8ba4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a4102570 0x7fc1a4197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a8ba4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1a4102ef0 0x7fc1a4197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a8ba4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a4110f50 con 0x7fc1a4102570 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a8ba4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a4110dd0 con 0x7fc1a4100e20 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a8ba4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a41110d0 con 0x7fc1a4102ef0 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a2d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1a4102ef0 0x7fc1a4197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a2d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1a4102ef0 0x7fc1a4197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:58686/0 (socket says 172.21.15.89:58686) 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a2d9d640 1 -- 172.21.15.89:0/559743086 learned_addr learned my addr 172.21.15.89:0/559743086 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:13.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a259c640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a4100e20 0x7fc1a41970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a2d9d640 1 -- 172.21.15.89:0/559743086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a4100e20 msgr2=0x7fc1a41970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a2d9d640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a4100e20 0x7fc1a41970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a2d9d640 1 -- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a4102570 msgr2=0x7fc1a4197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a4102570 0x7fc1a4197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.349+0000 7fc1a2d9d640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a4102570 0x7fc1a4197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.350+0000 7fc1a2d9d640 1 -- 172.21.15.89:0/559743086 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a41a6960 con 0x7fc1a4102ef0 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.350+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a4102570 0x7fc1a4197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.350+0000 7fc1a2d9d640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1a4102ef0 0x7fc1a4197b70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00cad0 tx=0x7fc19c00cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.350+0000 7fc19b7fe640 1 -- 172.21.15.89:0/559743086 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c00ae20 con 0x7fc1a4102ef0 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.350+0000 7fc19b7fe640 1 -- 172.21.15.89:0/559743086 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc19c005020 con 0x7fc1a4102ef0 2024-09-15T23:38:13.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.350+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/559743086 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a41a6c50 con 0x7fc1a4102ef0 2024-09-15T23:38:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.350+0000 7fc19b7fe640 1 -- 172.21.15.89:0/559743086 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c00f6f0 con 0x7fc1a4102ef0 2024-09-15T23:38:13.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.350+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/559743086 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1a41a7140 con 0x7fc1a4102ef0 2024-09-15T23:38:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.353+0000 7fc1aaf7d740 1 Processor -- start 2024-09-15T23:38:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.353+0000 7fc19b7fe640 1 -- 172.21.15.89:0/559743086 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc19c0051d0 con 0x7fc1a4102ef0 2024-09-15T23:38:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.353+0000 7fc1aaf7d740 1 -- start start 2024-09-15T23:38:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be30d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be30da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be318370 0x5652be31a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1aaf7d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5652be2732f0 con 0x5652be30eb40 2024-09-15T23:38:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1aaf7d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5652be21acf0 con 0x5652be318370 2024-09-15T23:38:13.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1aaf7d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5652be26f2b0 con 0x5652be30d5d0 2024-09-15T23:38:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1a1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be30d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1a1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be30d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40198/0 (socket says 172.21.15.89:40198) 2024-09-15T23:38:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/986284673 learned_addr learned my addr 172.21.15.89:0/986284673 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1a259c640 1 --2- 172.21.15.89:0/986284673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be30da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/986284673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be318370 0x5652be31a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/986284673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 msgr2=0x5652be30da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/986284673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be30da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc19b7fe640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1900a3240 0x7fc1900a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/986284673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be318370 msgr2=0x5652be31a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/986284673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be318370 0x5652be31a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.354+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/986284673 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652be1ff3a0 con 0x5652be30eb40 2024-09-15T23:38:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.355+0000 7fc1a2d9d640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1900a3240 0x7fc1900a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.355+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/986284673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be30d090 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0145c0 tx=0x7fc18c00b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b2bd097268e551 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.355+0000 7fc19b7fe640 1 -- 172.21.15.89:0/559743086 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc19c0cfd60 con 0x7fc1a4102ef0 2024-09-15T23:38:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.355+0000 7fc198ff9640 1 -- 172.21.15.89:0/986284673 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c012710 con 0x5652be30eb40 2024-09-15T23:38:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.355+0000 7fc198ff9640 1 -- 172.21.15.89:0/986284673 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc18c012d90 con 0x5652be30eb40 2024-09-15T23:38:13.358 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.355+0000 7fc198ff9640 1 -- 172.21.15.89:0/986284673 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c011d30 con 0x5652be30eb40 2024-09-15T23:38:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/986284673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 msgr2=0x5652be30d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/986284673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be30d090 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0145c0 tx=0x7fc18c00b5d0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1a2d9d640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1900a3240 0x7fc1900a5700 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc19c01e000 tx=0x7fc19c06ee10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/986284673 shutdown_connections 2024-09-15T23:38:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/986284673 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be318370 0x5652be31a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/986284673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be30da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/986284673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be30d090 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/986284673 >> 172.21.15.89:0/986284673 conn(0x5652be2d6310 msgr2=0x5652be309c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/986284673 shutdown_connections 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/986284673 wait complete. 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.356+0000 7fc1aaf7d740 1 -- start start 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1aaf7d740 1 Processor -- start 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be317ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be30eb40 0x5652be314f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 0x5652be3154f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1aaf7d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5652be31b070 con 0x5652be318370 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1aaf7d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5652be22e060 con 0x5652be30eb40 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1aaf7d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5652be31b1f0 con 0x5652be30d5d0 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1aaf7d740 10 client.0 _ll_register_callbacks cb 0x5652bdf818c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1a1d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be317ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.362 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1a1d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be317ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:58698/0 (socket says 172.21.15.89:58698) 2024-09-15T23:38:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/3354658868 learned_addr learned my addr 172.21.15.89:0/3354658868 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 0x5652be3154f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.357+0000 7fc1a259c640 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be30eb40 0x5652be314f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/3354658868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be30eb40 msgr2=0x5652be314f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be30eb40 0x5652be314f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/3354658868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 msgr2=0x5652be3154f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 0x5652be3154f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/3354658868 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652bdfe43c0 con 0x5652be30d5d0 2024-09-15T23:38:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be317ce0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0117e0 tx=0x7fc18c011910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d967103e951e1d0e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:13.363 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc189ffb640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc189ffb640 1 -- 172.21.15.89:0/3354658868 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c011440 con 0x5652be30d5d0 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/3354658868 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc18c012d90 con 0x5652be30d5d0 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc189ffb640 1 -- 172.21.15.89:0/3354658868 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc18c012510 con 0x5652be30d5d0 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc189ffb640 1 -- 172.21.15.89:0/3354658868 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc18c012710 con 0x5652be30d5d0 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc1aaf7d740 10 client.28336 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/3354658868 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5652be326a80 con 0x5652be30d5d0 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.358+0000 7fc189ffb640 1 -- 172.21.15.89:0/3354658868 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc18c012d90 con 0x5652be30d5d0 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.359+0000 7fc189ffb640 1 -- 172.21.15.89:0/3354658868 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc18c0055c0 con 0x5652be30d5d0 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.359+0000 7fc1aaf7d740 10 client.28336 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.359+0000 7fc1aaf7d740 10 client.28336 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.359+0000 7fc1aaf7d740 20 client.28336 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.359+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5652be328210 0x5652be32a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.359+0000 7fc1aaf7d740 4 client.28336 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.359+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/3354658868 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652be280860 con 0x5652be328210 2024-09-15T23:38:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.359+0000 7fc1a259c640 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5652be328210 0x5652be32a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.359+0000 7fc189ffb640 10 client.28336 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:38:13.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.359+0000 7fc1a259c640 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5652be328210 0x5652be32a600 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8d4676cc652b5912 server_cookie=e69b8efe43270e1a in_seq=0 out_seq=0 2024-09-15T23:38:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.360+0000 7fc189ffb640 1 -- 172.21.15.89:0/3354658868 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5652be280860 con 0x5652be328210 2024-09-15T23:38:13.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.360+0000 7fc189ffb640 10 client.28336 handle_command_reply: tid=0 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.360+0000 7fc1aaf7d740 1 client.28336 shutdown 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/3354658868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 msgr2=0x5652be317ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be317ce0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0117e0 tx=0x7fc18c011910 comp rx=0 tx=0).stop 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/3354658868 shutdown_connections 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5652be328210 0x5652be32a600 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 0x5652be3154f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be30eb40 0x5652be314f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc189ffb640 0 client.28336 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/3354658868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be317ce0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/3354658868 >> 172.21.15.89:0/3354658868 conn(0x5652be2d6310 msgr2=0x5652be30f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/3354658868 shutdown_connections 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/3354658868 wait complete. 2024-09-15T23:38:13.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.361+0000 7fc1aaf7d740 20 client.28336 trim_cache size 0 max 16384 2024-09-15T23:38:13.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.375+0000 7fc1aaf7d740 1 Processor -- start 2024-09-15T23:38:13.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.375+0000 7fc1aaf7d740 1 -- start start 2024-09-15T23:38:13.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.375+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be308c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.376 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.375+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be3100e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a1d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be308c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be3100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a1d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be308c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:58704/0 (socket says 172.21.15.89:58704) 2024-09-15T23:38:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be3100e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40220/0 (socket says 172.21.15.89:40220) 2024-09-15T23:38:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/640712054 learned_addr learned my addr 172.21.15.89:0/640712054 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/640712054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be318370 0x5652be310620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/640712054 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a41b1010 con 0x5652be30eb40 2024-09-15T23:38:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/640712054 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a41b2b50 con 0x5652be318370 2024-09-15T23:38:13.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/640712054 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a41b16e0 con 0x5652be30d5d0 2024-09-15T23:38:13.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/640712054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be318370 msgr2=0x5652be310620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:13.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/640712054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be318370 0x5652be310620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/640712054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 msgr2=0x5652be3100e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:13.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/640712054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be3100e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/640712054 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652be313f40 con 0x5652be30d5d0 2024-09-15T23:38:13.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a259c640 1 --2- 172.21.15.89:0/640712054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be3100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:38:13.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.376+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/640712054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be308c10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0145c0 tx=0x7fc18c00ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f8d5fb456d85fe0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:13.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.377+0000 7fc18bfff640 1 -- 172.21.15.89:0/640712054 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c005be0 con 0x5652be30d5d0 2024-09-15T23:38:13.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.377+0000 7fc18bfff640 1 -- 172.21.15.89:0/640712054 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc18c00b550 con 0x5652be30d5d0 2024-09-15T23:38:13.379 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.377+0000 7fc18bfff640 1 -- 172.21.15.89:0/640712054 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c00e430 con 0x5652be30d5d0 2024-09-15T23:38:13.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.378+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/640712054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 msgr2=0x5652be308c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:13.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.378+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/640712054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be308c10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0145c0 tx=0x7fc18c00ba20 comp rx=0 tx=0).stop 2024-09-15T23:38:13.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.378+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/640712054 shutdown_connections 2024-09-15T23:38:13.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.378+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/640712054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be318370 0x5652be310620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.378+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/640712054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be30eb40 0x5652be3100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.378+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/640712054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be308c10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.378+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/640712054 >> 172.21.15.89:0/640712054 conn(0x5652be2d6310 msgr2=0x5652be308500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.378+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/640712054 shutdown_connections 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.378+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/640712054 wait complete. 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.378+0000 7fc1aaf7d740 1 -- start start 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1aaf7d740 1 Processor -- start 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be320880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be30eb40 0x5652be311970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1aaf7d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 0x5652be311eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1aaf7d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a41b2480 con 0x5652be318370 2024-09-15T23:38:13.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1a1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 0x5652be311eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1a1d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 0x5652be311eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40222/0 (socket says 172.21.15.89:40222) 2024-09-15T23:38:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/2699141979 learned_addr learned my addr 172.21.15.89:0/2699141979 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1a259c640 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be30eb40 0x5652be311970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1aaf7d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a41b1db0 con 0x5652be30eb40 2024-09-15T23:38:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/2699141979 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a41aeff0 con 0x5652be30d5d0 2024-09-15T23:38:13.382 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.379+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be320880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/2699141979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 msgr2=0x5652be320880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:13.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be320880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/2699141979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be30eb40 msgr2=0x5652be311970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be30eb40 0x5652be311970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc1a1d9b640 1 -- 172.21.15.89:0/2699141979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652be3123f0 con 0x5652be318370 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc1aaf7d740 10 client.0 _ll_register_callbacks cb 0x7fc194005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc1a1d9b640 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 0x5652be311eb0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00be20 tx=0x7fc18c014050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d82e5490334c0b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc189ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/2699141979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc18c00b550 con 0x5652be318370 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc189ffb640 1 -- 172.21.15.89:0/2699141979 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c005be0 con 0x5652be318370 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc189ffb640 1 -- 172.21.15.89:0/2699141979 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc18c045480 con 0x5652be318370 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc189ffb640 1 -- 172.21.15.89:0/2699141979 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc18c045680 con 0x5652be318370 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc1aaf7d740 10 client.20316 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.380+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/2699141979 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5652be39b0c0 con 0x5652be318370 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.381+0000 7fc189ffb640 1 -- 172.21.15.89:0/2699141979 <== mon.0 v2:172.21.15.89:3300/0 4 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc18c042040 con 0x5652be318370 2024-09-15T23:38:13.386 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.381+0000 7fc1aaf7d740 10 client.20316 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.381+0000 7fc1aaf7d740 10 client.20316 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.381+0000 7fc1aaf7d740 20 client.20316 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.381+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5652be39b4d0 0x5652be39d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.381+0000 7fc1aaf7d740 4 client.20316 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.381+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/2699141979 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5652be27ad70 con 0x5652be39b4d0 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.381+0000 7fc1a259c640 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5652be39b4d0 0x5652be39d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.382+0000 7fc189ffb640 10 client.20316 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.382+0000 7fc1a259c640 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5652be39b4d0 0x5652be39d930 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8808ed3f63fe69f0 server_cookie=d6c1d66444c0d354 in_seq=0 out_seq=0 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc189ffb640 1 -- 172.21.15.89:0/2699141979 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5652be27ad70 con 0x5652be39b4d0 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc189ffb640 10 client.20316 handle_command_reply: tid=0 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc1aaf7d740 1 client.20316 shutdown 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/2699141979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 msgr2=0x5652be311eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 0x5652be311eb0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00be20 tx=0x7fc18c014050 comp rx=0 tx=0).stop 2024-09-15T23:38:13.387 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/2699141979 shutdown_connections 2024-09-15T23:38:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x5652be39b4d0 0x5652be39d930 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652be318370 0x5652be311eb0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5652be30eb40 0x5652be311970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc1aaf7d740 1 --2- 172.21.15.89:0/2699141979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652be30d5d0 0x5652be320880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/2699141979 >> 172.21.15.89:0/2699141979 conn(0x5652be2d6310 msgr2=0x5652be308d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.383+0000 7fc189ffb640 0 client.20316 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:38:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.384+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/2699141979 shutdown_connections 2024-09-15T23:38:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.384+0000 7fc1aaf7d740 1 -- 172.21.15.89:0/2699141979 wait complete. 2024-09-15T23:38:13.388 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.384+0000 7fc1aaf7d740 20 client.20316 trim_cache size 0 max 16384 2024-09-15T23:38:13.389 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:38:13.389 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1900a3240 msgr2=0x7fc1900a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1900a3240 0x7fc1900a5700 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc19c01e000 tx=0x7fc19c06ee10 comp rx=0 tx=0).stop 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/559743086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1a4102ef0 msgr2=0x7fc1a4197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1a4102ef0 0x7fc1a4197b70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00cad0 tx=0x7fc19c00cf90 comp rx=0 tx=0).stop 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/559743086 shutdown_connections 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc1900a3240 0x7fc1900a5700 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1a4102ef0 0x7fc1a4197b70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1a4102570 0x7fc1a4197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 --2- 172.21.15.89:0/559743086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1a4100e20 0x7fc1a41970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/559743086 >> 172.21.15.89:0/559743086 conn(0x7fc1a40fc7d0 msgr2=0x7fc1a40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/559743086 shutdown_connections 2024-09-15T23:38:13.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:13.385+0000 7fc1a8ba4640 1 -- 172.21.15.89:0/559743086 wait complete. 2024-09-15T23:38:13.391 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:13 smithi120 ceph-mon[35250]: pgmap v775: 129 pgs: 129 active+clean; 3.1 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:38:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:13 smithi174 ceph-mon[35586]: pgmap v775: 129 pgs: 129 active+clean; 3.1 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:38:14.309 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:38:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.578+0000 7f0ee8159640 1 -- 172.21.15.89:0/2384174322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee0102d50 msgr2=0x7f0ee0103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:14.600 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.578+0000 7f0ee8159640 1 --2- 172.21.15.89:0/2384174322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee0102d50 0x7f0ee0103130 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009900 tx=0x7f0edc02a480 comp rx=0 tx=0).stop 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.578+0000 7f0ee8159640 1 -- 172.21.15.89:0/2384174322 shutdown_connections 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.578+0000 7f0ee8159640 1 --2- 172.21.15.89:0/2384174322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ee010c180 0x7f0ee010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.578+0000 7f0ee8159640 1 --2- 172.21.15.89:0/2384174322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ee0103670 0x7f0ee010bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.578+0000 7f0ee8159640 1 --2- 172.21.15.89:0/2384174322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee0102d50 0x7f0ee0103130 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.578+0000 7f0ee8159640 1 -- 172.21.15.89:0/2384174322 >> 172.21.15.89:0/2384174322 conn(0x7f0ee00fc7d0 msgr2=0x7f0ee00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.578+0000 7f0ee8159640 1 -- 172.21.15.89:0/2384174322 shutdown_connections 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.578+0000 7f0ee8159640 1 -- 172.21.15.89:0/2384174322 wait complete. 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee8159640 1 Processor -- start 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee8159640 1 -- start start 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee8159640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ee0102d50 0x7f0ee0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee8159640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee0103670 0x7f0ee0198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee8159640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ee010c180 0x7f0ee019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee8159640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee0111f80 con 0x7f0ee0103670 2024-09-15T23:38:14.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee8159640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee0111e00 con 0x7f0ee0102d50 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee8159640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee0112100 con 0x7f0ee010c180 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee56f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee0103670 0x7f0ee0198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee56f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee0103670 0x7f0ee0198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40244/0 (socket says 172.21.15.89:40244) 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee56f4640 1 -- 172.21.15.89:0/499771530 learned_addr learned my addr 172.21.15.89:0/499771530 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.579+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ee0102d50 0x7f0ee0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ee010c180 0x7f0ee019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ee56f4640 1 -- 172.21.15.89:0/499771530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ee010c180 msgr2=0x7f0ee019ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ee56f4640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ee010c180 0x7f0ee019ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ee56f4640 1 -- 172.21.15.89:0/499771530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ee0102d50 msgr2=0x7f0ee0198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ee56f4640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ee0102d50 0x7f0ee0198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ee56f4640 1 -- 172.21.15.89:0/499771530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee019d3d0 con 0x7f0ee0103670 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ee010c180 0x7f0ee019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:14.602 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ee56f4640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee0103670 0x7f0ee0198960 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000d740 tx=0x7f0ed000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ed6ffd640 1 -- 172.21.15.89:0/499771530 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0017070 con 0x7f0ee0103670 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ee8159640 1 -- 172.21.15.89:0/499771530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee019d6c0 con 0x7f0ee0103670 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ed6ffd640 1 -- 172.21.15.89:0/499771530 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed000de50 con 0x7f0ee0103670 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ed6ffd640 1 -- 172.21.15.89:0/499771530 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000f620 con 0x7f0ee0103670 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.580+0000 7f0ee8159640 1 -- 172.21.15.89:0/499771530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee01a4fa0 con 0x7f0ee0103670 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.583+0000 7f0ed6ffd640 1 -- 172.21.15.89:0/499771530 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0ed0009c30 con 0x7f0ee0103670 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.583+0000 7f0eea532740 1 Processor -- start 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.583+0000 7f0eea532740 1 -- start start 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0eea532740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0eea532740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9d7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0eea532740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 0x55717c9d9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0eea532740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55717c9322a0 con 0x55717c9cd080 2024-09-15T23:38:14.603 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0eea532740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55717c8d9b10 con 0x55717c9d7940 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0eea532740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55717c92e260 con 0x55717c9cc760 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee5ef5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9d7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee5ef5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9d7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40250/0 (socket says 172.21.15.89:40250) 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee5ef5640 1 -- 172.21.15.89:0/2272787003 learned_addr learned my addr 172.21.15.89:0/2272787003 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/2272787003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee56f4640 1 --2- 172.21.15.89:0/2272787003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 0x55717c9d9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee5ef5640 1 -- 172.21.15.89:0/2272787003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 msgr2=0x55717c9ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/2272787003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee5ef5640 1 -- 172.21.15.89:0/2272787003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 msgr2=0x55717c9d9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/2272787003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 0x55717c9d9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee5ef5640 1 -- 172.21.15.89:0/2272787003 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55717c6a3b90 con 0x55717c9cd080 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/2272787003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:14.604 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ed6ffd640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ecc0a3240 0x7f0ecc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/2272787003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9d7400 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009900 tx=0x7f0edc005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae231c404dd1faa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ecc0a3240 0x7f0ecc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ed6ffd640 1 -- 172.21.15.89:0/499771530 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0ed00cf900 con 0x7f0ee0103670 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.584+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2272787003 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc040070 con 0x55717c9cd080 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2272787003 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0edc0319f0 con 0x55717c9cd080 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0ecbfff640 1 -- 172.21.15.89:0/2272787003 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc040070 con 0x55717c9cd080 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0eea532740 1 -- 172.21.15.89:0/2272787003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 msgr2=0x55717c9d7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ecc0a3240 0x7f0ecc0a5700 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8005fd0 tx=0x7f0ed8009530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0eea532740 1 --2- 172.21.15.89:0/2272787003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9d7400 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009900 tx=0x7f0edc005e80 comp rx=0 tx=0).stop 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0eea532740 1 -- 172.21.15.89:0/2272787003 shutdown_connections 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0eea532740 1 --2- 172.21.15.89:0/2272787003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 0x55717c9d9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0eea532740 1 --2- 172.21.15.89:0/2272787003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9d7400 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0eea532740 1 --2- 172.21.15.89:0/2272787003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.605 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0eea532740 1 -- 172.21.15.89:0/2272787003 >> 172.21.15.89:0/2272787003 conn(0x55717c9c8cf0 msgr2=0x55717c9c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0eea532740 1 -- 172.21.15.89:0/2272787003 shutdown_connections 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.585+0000 7f0eea532740 1 -- 172.21.15.89:0/2272787003 wait complete. 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0eea532740 1 -- start start 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0eea532740 1 Processor -- start 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0eea532740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9cc760 0x55717c9d4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0eea532740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cd080 0x55717c9d4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0eea532740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9d5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0eea532740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55717c9da510 con 0x55717c9d7940 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0eea532740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55717c8ba6e0 con 0x55717c9cc760 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0eea532740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55717c9da690 con 0x55717c9cd080 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0eea532740 10 client.0 _ll_register_callbacks cb 0x55717c8dea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0ee66f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9d5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.606 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0ee66f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9d5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40262/0 (socket says 172.21.15.89:40262) 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0ee66f6640 1 -- 172.21.15.89:0/3914778866 learned_addr learned my addr 172.21.15.89:0/3914778866 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.586+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9cc760 0x55717c9d4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ee56f4640 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cd080 0x55717c9d4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ee66f6640 1 -- 172.21.15.89:0/3914778866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cd080 msgr2=0x55717c9d4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cd080 0x55717c9d4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ee66f6640 1 -- 172.21.15.89:0/3914778866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9cc760 msgr2=0x55717c9d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9cc760 0x55717c9d4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ee66f6640 1 -- 172.21.15.89:0/3914778866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55717c9e7dc0 con 0x55717c9d7940 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ee56f4640 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cd080 0x55717c9d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9d5200 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8009d80 tx=0x7f0ed8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74312a39bed3a232 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ec97fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3914778866 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8022070 con 0x55717c9d7940 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3914778866 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0ed800fd60 con 0x55717c9d7940 2024-09-15T23:38:14.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0eea532740 1 -- 172.21.15.89:0/3914778866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0edc0319f0 con 0x55717c9d7940 2024-09-15T23:38:14.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3914778866 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0ed8012e50 con 0x55717c9d7940 2024-09-15T23:38:14.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0eea532740 10 client.20328 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:14.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0eea532740 1 -- 172.21.15.89:0/3914778866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55717c9ea780 con 0x55717c9d7940 2024-09-15T23:38:14.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3914778866 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0edc0319f0 con 0x55717c9d7940 2024-09-15T23:38:14.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3914778866 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0ed801e710 con 0x55717c9d7940 2024-09-15T23:38:14.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0eea532740 10 client.20328 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:14.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.587+0000 7f0eea532740 10 client.20328 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:38:14.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.588+0000 7f0eea532740 20 client.20328 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:14.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.588+0000 7f0eea532740 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55717c9ebc40 0x55717c9ee050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.588+0000 7f0eea532740 4 client.20328 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.588+0000 7f0eea532740 1 -- 172.21.15.89:0/3914778866 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55717c93f9d0 con 0x55717c9ebc40 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.588+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55717c9ebc40 0x55717c9ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.588+0000 7f0ec97fa640 10 client.20328 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.588+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55717c9ebc40 0x55717c9ee050 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=839219a14b5b1e0a server_cookie=b4dfaf373f54d344 in_seq=0 out_seq=0 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3914778866 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55717c93f9d0 con 0x55717c9ebc40 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0ec97fa640 10 client.20328 handle_command_reply: tid=0 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 client.20328 shutdown 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 -- 172.21.15.89:0/3914778866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 msgr2=0x55717c9d5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9d5200 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8009d80 tx=0x7f0ed8004a40 comp rx=0 tx=0).stop 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 -- 172.21.15.89:0/3914778866 shutdown_connections 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55717c9ebc40 0x55717c9ee050 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9d5200 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cd080 0x55717c9d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 --2- 172.21.15.89:0/3914778866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9cc760 0x55717c9d4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0ec97fa640 0 client.20328 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:14.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 -- 172.21.15.89:0/3914778866 >> 172.21.15.89:0/3914778866 conn(0x55717c9c8cf0 msgr2=0x55717c9ce9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:14.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 -- 172.21.15.89:0/3914778866 shutdown_connections 2024-09-15T23:38:14.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 1 -- 172.21.15.89:0/3914778866 wait complete. 2024-09-15T23:38:14.610 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.589+0000 7f0eea532740 20 client.20328 trim_cache size 0 max 16384 2024-09-15T23:38:14.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0eea532740 1 Processor -- start 2024-09-15T23:38:14.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0eea532740 1 -- start start 2024-09-15T23:38:14.614 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0eea532740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9d4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0eea532740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9cd080 0x55717c9d4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0eea532740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9f3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0eea532740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01acce0 con 0x55717c9d7940 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0eea532740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01ae820 con 0x55717c9cd080 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0eea532740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01ad3b0 con 0x55717c9cc760 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee66f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee66f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9f3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40270/0 (socket says 172.21.15.89:40270) 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee66f6640 1 -- 172.21.15.89:0/738338220 learned_addr learned my addr 172.21.15.89:0/738338220 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/738338220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9d4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee56f4640 1 --2- 172.21.15.89:0/738338220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9cd080 0x55717c9d4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee66f6640 1 -- 172.21.15.89:0/738338220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 msgr2=0x55717c9d4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/738338220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9d4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.615 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee66f6640 1 -- 172.21.15.89:0/738338220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9cd080 msgr2=0x55717c9d4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/738338220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9cd080 0x55717c9d4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee66f6640 1 -- 172.21.15.89:0/738338220 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55717c9cd5f0 con 0x55717c9d7940 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.607+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/738338220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9d4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.608+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/738338220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9f3920 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8000f80 tx=0x7f0ed8012520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193c7e67c02c1d9a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.608+0000 7f0ecb7fe640 1 -- 172.21.15.89:0/738338220 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed800fd60 con 0x55717c9d7940 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.608+0000 7f0ecb7fe640 1 -- 172.21.15.89:0/738338220 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed8012e50 con 0x55717c9d7940 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.608+0000 7f0ecb7fe640 1 -- 172.21.15.89:0/738338220 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed803d400 con 0x55717c9d7940 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 -- 172.21.15.89:0/738338220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 msgr2=0x55717c9f3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 --2- 172.21.15.89:0/738338220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9f3920 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8000f80 tx=0x7f0ed8012520 comp rx=0 tx=0).stop 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 -- 172.21.15.89:0/738338220 shutdown_connections 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 --2- 172.21.15.89:0/738338220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9d7940 0x55717c9f3920 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 --2- 172.21.15.89:0/738338220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9cd080 0x55717c9d4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.616 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 --2- 172.21.15.89:0/738338220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9d4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 -- 172.21.15.89:0/738338220 >> 172.21.15.89:0/738338220 conn(0x55717c9c8cf0 msgr2=0x55717c9d4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 -- 172.21.15.89:0/738338220 shutdown_connections 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 -- 172.21.15.89:0/738338220 wait complete. 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 -- start start 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.609+0000 7f0eea532740 1 Processor -- start 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0eea532740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9e3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0eea532740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9e42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0eea532740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 0x55717c9ded60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0eea532740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01ae150 con 0x55717c9cd080 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0eea532740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01ada80 con 0x55717c9d7940 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0eea532740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01aacc0 con 0x55717c9cc760 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0eea532740 10 client.0 _ll_register_callbacks cb 0x7f0ed80125b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee56f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9e42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.617 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee56f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9e42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:40278/0 (socket says 172.21.15.89:40278) 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee56f4640 1 -- 172.21.15.89:0/3766063265 learned_addr learned my addr 172.21.15.89:0/3766063265 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9e3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 0x55717c9ded60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee56f4640 1 -- 172.21.15.89:0/3766063265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 msgr2=0x55717c9e3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee56f4640 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9e3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee56f4640 1 -- 172.21.15.89:0/3766063265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 msgr2=0x55717c9ded60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee56f4640 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 0x55717c9ded60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee56f4640 1 -- 172.21.15.89:0/3766063265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55717c9c9980 con 0x55717c9cd080 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee66f6640 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 0x55717c9ded60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ee56f4640 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9e42f0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00cd010 tx=0x7f0ed009b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dddb4be75951109 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.610+0000 7f0ec97fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3766063265 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed009ce60 con 0x55717c9cd080 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3766063265 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0ed00a4310 con 0x55717c9cd080 2024-09-15T23:38:14.618 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0eea532740 1 -- 172.21.15.89:0/3766063265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ed8012e50 con 0x55717c9cd080 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3766063265 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0ed0020040 con 0x55717c9cd080 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0eea532740 10 client.20334 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0eea532740 1 -- 172.21.15.89:0/3766063265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55717c9c8b20 con 0x55717c9cd080 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3766063265 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed00a47a0 con 0x55717c9cd080 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3766063265 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0ed00ba2c0 con 0x55717c9cd080 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0eea532740 10 client.20334 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0eea532740 10 client.20334 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0eea532740 20 client.20334 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0eea532740 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55717ca575d0 0x55717ca599f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0eea532740 4 client.20334 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.5, "await_for": 45.0}] 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.611+0000 7f0eea532740 1 -- 172.21.15.89:0/3766063265 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.5, "await_for": 45.0}) -- 0x7f0ed8012e50 con 0x55717ca575d0 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.612+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55717ca575d0 0x55717ca599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:14.619 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.612+0000 7f0ec97fa640 10 client.20334 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:14.620 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.612+0000 7f0ee5ef5640 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55717ca575d0 0x55717ca599f0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa545092019dd31d server_cookie=b89b1aba3b85f3f6 in_seq=0 out_seq=0 2024-09-15T23:38:14.625 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0ec97fa640 1 -- 172.21.15.89:0/3766063265 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f0ed8012e50 con 0x55717ca575d0 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0ec97fa640 10 client.20334 handle_command_reply: tid=0 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 client.20334 shutdown 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 -- 172.21.15.89:0/3766063265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 msgr2=0x55717c9e42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9e42f0 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00cd010 tx=0x7f0ed009b580 comp rx=0 tx=0).stop 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 -- 172.21.15.89:0/3766063265 shutdown_connections 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55717ca575d0 0x55717ca599f0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55717c9d7940 0x55717c9ded60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55717c9cd080 0x55717c9e42f0 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0ec97fa640 0 client.20334 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 --2- 172.21.15.89:0/3766063265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55717c9cc760 0x55717c9e3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 -- 172.21.15.89:0/3766063265 >> 172.21.15.89:0/3766063265 conn(0x55717c9c8cf0 msgr2=0x55717c9e7440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 -- 172.21.15.89:0/3766063265 shutdown_connections 2024-09-15T23:38:14.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 1 -- 172.21.15.89:0/3766063265 wait complete. 2024-09-15T23:38:14.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.624+0000 7f0eea532740 20 client.20334 trim_cache size 0 max 16384 2024-09-15T23:38:14.627 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:38:14.627 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:38:14.627 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:38:14.627 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 50, 2024-09-15T23:38:14.627 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:38:14.627 INFO:teuthology.orchestra.run.smithi089.stdout: "83e459f8": { 2024-09-15T23:38:14.627 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 50, 2024-09-15T23:38:14.627 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:38:14.627 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 146.5, 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:14.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:14.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:14.629 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:38:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 -- 172.21.15.89:0/499771530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ecc0a3240 msgr2=0x7f0ecc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ecc0a3240 0x7f0ecc0a5700 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8005fd0 tx=0x7f0ed8009530 comp rx=0 tx=0).stop 2024-09-15T23:38:14.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 -- 172.21.15.89:0/499771530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee0103670 msgr2=0x7f0ee0198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee0103670 0x7f0ee0198960 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000d740 tx=0x7f0ed000dc00 comp rx=0 tx=0).stop 2024-09-15T23:38:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 -- 172.21.15.89:0/499771530 shutdown_connections 2024-09-15T23:38:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0ecc0a3240 0x7f0ecc0a5700 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ee010c180 0x7f0ee019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0ee0103670 0x7f0ee0198960 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 --2- 172.21.15.89:0/499771530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ee0102d50 0x7f0ee0198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 -- 172.21.15.89:0/499771530 >> 172.21.15.89:0/499771530 conn(0x7f0ee00fc7d0 msgr2=0x7f0ee00fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 -- 172.21.15.89:0/499771530 shutdown_connections 2024-09-15T23:38:14.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:14.625+0000 7f0ee8159640 1 -- 172.21.15.89:0/499771530 wait complete. 2024-09-15T23:38:15.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:15 smithi089 ceph-mon[30933]: pgmap v776: 129 pgs: 129 active+clean; 3.2 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:38:15.435 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:15.457 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '83e459f8', quiesce took 5.1 seconds. Will release after: 21.4 2024-09-15T23:38:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:15 smithi120 ceph-mon[35250]: pgmap v776: 129 pgs: 129 active+clean; 3.2 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:38:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:15 smithi174 ceph-mon[35586]: pgmap v776: 129 pgs: 129 active+clean; 3.2 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:38:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- 172.21.15.89:0/3127996001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7648101c40 msgr2=0x7f76481020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 --2- 172.21.15.89:0/3127996001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7648101c40 0x7f76481020c0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f7630009900 tx=0x7f763002a410 comp rx=0 tx=0).stop 2024-09-15T23:38:15.696 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- 172.21.15.89:0/3127996001 shutdown_connections 2024-09-15T23:38:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 --2- 172.21.15.89:0/3127996001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648102600 0x7f7648108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 --2- 172.21.15.89:0/3127996001 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7648101c40 0x7f76481020c0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 --2- 172.21.15.89:0/3127996001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7648100a50 0x7f7648100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- 172.21.15.89:0/3127996001 >> 172.21.15.89:0/3127996001 conn(0x7f76480fc1b0 msgr2=0x7f76480fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- 172.21.15.89:0/3127996001 shutdown_connections 2024-09-15T23:38:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- 172.21.15.89:0/3127996001 wait complete. 2024-09-15T23:38:15.697 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 Processor -- start 2024-09-15T23:38:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- start start 2024-09-15T23:38:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648100a50 0x7f7648197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7648101c40 0x7f7648198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7648102600 0x7f764819f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f764810c6d0 con 0x7f7648102600 2024-09-15T23:38:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f764810c550 con 0x7f7648100a50 2024-09-15T23:38:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f764810c850 con 0x7f7648101c40 2024-09-15T23:38:15.699 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7647fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7648102600 0x7f764819f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7647fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7648102600 0x7f764819f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43856/0 (socket says 172.21.15.89:43856) 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7647fff640 1 -- 172.21.15.89:0/3201428255 learned_addr learned my addr 172.21.15.89:0/3201428255 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f76477fe640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648100a50 0x7f7648197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7646ffd640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7648101c40 0x7f7648198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7647fff640 1 -- 172.21.15.89:0/3201428255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7648101c40 msgr2=0x7f7648198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7647fff640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7648101c40 0x7f7648198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7647fff640 1 -- 172.21.15.89:0/3201428255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648100a50 msgr2=0x7f7648197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7647fff640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648100a50 0x7f7648197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7647fff640 1 -- 172.21.15.89:0/3201428255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f764819fa70 con 0x7f7648102600 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7647fff640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7648102600 0x7f764819f400 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f763800cad0 tx=0x7f763800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7644ff9640 1 -- 172.21.15.89:0/3201428255 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763800ae20 con 0x7f7648102600 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- 172.21.15.89:0/3201428255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f764819fd60 con 0x7f7648102600 2024-09-15T23:38:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7644ff9640 1 -- 172.21.15.89:0/3201428255 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7638005020 con 0x7f7648102600 2024-09-15T23:38:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7644ff9640 1 -- 172.21.15.89:0/3201428255 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763800f6f0 con 0x7f7648102600 2024-09-15T23:38:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f764e113640 1 -- 172.21.15.89:0/3201428255 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76481a0220 con 0x7f7648102600 2024-09-15T23:38:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7650ea4740 1 Processor -- start 2024-09-15T23:38:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7644ff9640 1 -- 172.21.15.89:0/3201428255 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f76380051d0 con 0x7f7648102600 2024-09-15T23:38:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7650ea4740 1 -- start start 2024-09-15T23:38:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e2953dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953e2e0 0x561e29542da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e29543370 0x561e29545830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7650ea4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561e294a12f0 con 0x561e2953e2e0 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7650ea4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561e29449270 con 0x561e2953d9a0 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7650ea4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561e2949d2b0 con 0x561e29543370 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f76477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953e2e0 0x561e29542da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f76477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953e2e0 0x561e29542da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43860/0 (socket says 172.21.15.89:43860) 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f76477fe640 1 -- 172.21.15.89:0/1099253834 learned_addr learned my addr 172.21.15.89:0/1099253834 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7646ffd640 1 --2- 172.21.15.89:0/1099253834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e2953dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f7646ffd640 1 --2- 172.21.15.89:0/1099253834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e29543370 0x561e29545830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f76477fe640 1 -- 172.21.15.89:0/1099253834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e29543370 msgr2=0x561e29545830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f76477fe640 1 --2- 172.21.15.89:0/1099253834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e29543370 0x561e29545830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f76477fe640 1 -- 172.21.15.89:0/1099253834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 msgr2=0x561e2953dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f76477fe640 1 --2- 172.21.15.89:0/1099253834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e2953dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.695+0000 7f76477fe640 1 -- 172.21.15.89:0/1099253834 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e29448340 con 0x561e2953e2e0 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7644ff9640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76340a32f0 0x7f76340a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f76477fe640 1 --2- 172.21.15.89:0/1099253834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953e2e0 0x561e29542da0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f764000abf0 tx=0x7f764000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad1157c566f02e4e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7646ffd640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76340a32f0 0x7f76340a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7644ff9640 1 -- 172.21.15.89:0/3201428255 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f76380cfdc0 con 0x7f7648102600 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f763dffb640 1 -- 172.21.15.89:0/1099253834 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640005220 con 0x561e2953e2e0 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f763dffb640 1 -- 172.21.15.89:0/1099253834 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7640004c80 con 0x561e2953e2e0 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f763dffb640 1 -- 172.21.15.89:0/1099253834 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76400187f0 con 0x561e2953e2e0 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- 172.21.15.89:0/1099253834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953e2e0 msgr2=0x561e29542da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.709 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 --2- 172.21.15.89:0/1099253834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953e2e0 0x561e29542da0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f764000abf0 tx=0x7f764000f790 comp rx=0 tx=0).stop 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7646ffd640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76340a32f0 0x7f76340a57b0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7630000c00 tx=0x7f7630005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- 172.21.15.89:0/1099253834 shutdown_connections 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 --2- 172.21.15.89:0/1099253834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e29543370 0x561e29545830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 --2- 172.21.15.89:0/1099253834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953e2e0 0x561e29542da0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 --2- 172.21.15.89:0/1099253834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e2953dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- 172.21.15.89:0/1099253834 >> 172.21.15.89:0/1099253834 conn(0x561e29539010 msgr2=0x561e29539410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- 172.21.15.89:0/1099253834 shutdown_connections 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- 172.21.15.89:0/1099253834 wait complete. 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- start start 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 Processor -- start 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e29547b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e29548070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.710 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2954f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561e29546340 con 0x561e29543370 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561e2942a290 con 0x561e2953d9a0 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561e295464c0 con 0x561e2953e2e0 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 10 client.0 _ll_register_callbacks cb 0x561e2945b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7647fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2954f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7646ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e29548070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7647fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2954f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43864/0 (socket says 172.21.15.89:43864) 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7647fff640 1 -- 172.21.15.89:0/1793623577 learned_addr learned my addr 172.21.15.89:0/1793623577 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f76477fe640 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e29547b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7647fff640 1 -- 172.21.15.89:0/1793623577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 msgr2=0x561e29548070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7647fff640 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e29548070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7647fff640 1 -- 172.21.15.89:0/1793623577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 msgr2=0x561e29547b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.711 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7647fff640 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e29547b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7647fff640 1 -- 172.21.15.89:0/1793623577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e291b4210 con 0x561e29543370 2024-09-15T23:38:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7647fff640 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2954f0b0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f76380989c0 tx=0x7f763809c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400c86ccec9ae4e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7626ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7626ffd640 1 -- 172.21.15.89:0/1793623577 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763801d040 con 0x561e29543370 2024-09-15T23:38:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7626ffd640 1 -- 172.21.15.89:0/1793623577 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7638096b60 con 0x561e29543370 2024-09-15T23:38:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- 172.21.15.89:0/1793623577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7640004c80 con 0x561e29543370 2024-09-15T23:38:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7626ffd640 1 -- 172.21.15.89:0/1793623577 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f76380bc0d0 con 0x561e29543370 2024-09-15T23:38:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 10 client.20346 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- 172.21.15.89:0/1793623577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561e29280620 con 0x561e29543370 2024-09-15T23:38:15.712 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7626ffd640 1 -- 172.21.15.89:0/1793623577 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7640004c80 con 0x561e29543370 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7626ffd640 1 -- 172.21.15.89:0/1793623577 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f763809aeb0 con 0x561e29543370 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 10 client.20346 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 10 client.20346 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 20 client.20346 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e295537d0 0x561e29555bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 4 client.20346 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 -- 172.21.15.89:0/1793623577 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e294aea20 con 0x561e295537d0 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f76477fe640 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e295537d0 0x561e29555bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f76477fe640 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e295537d0 0x561e29555bc0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee7ea61ebfa16716 server_cookie=b1c6493bdacda2f in_seq=0 out_seq=0 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7626ffd640 10 client.20346 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7626ffd640 1 -- 172.21.15.89:0/1793623577 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561e294aea20 con 0x561e295537d0 2024-09-15T23:38:15.713 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7626ffd640 10 client.20346 handle_command_reply: tid=0 2024-09-15T23:38:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.701+0000 7f7650ea4740 1 client.20346 shutdown 2024-09-15T23:38:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7650ea4740 1 -- 172.21.15.89:0/1793623577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 msgr2=0x561e2954f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7650ea4740 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2954f0b0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f76380989c0 tx=0x7f763809c830 comp rx=0 tx=0).stop 2024-09-15T23:38:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7650ea4740 1 -- 172.21.15.89:0/1793623577 shutdown_connections 2024-09-15T23:38:15.714 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7650ea4740 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e295537d0 0x561e29555bc0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7650ea4740 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2954f0b0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7650ea4740 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e29548070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7650ea4740 1 --2- 172.21.15.89:0/1793623577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e29547b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7650ea4740 1 -- 172.21.15.89:0/1793623577 >> 172.21.15.89:0/1793623577 conn(0x561e29539010 msgr2=0x561e2953a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7626ffd640 0 client.20346 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7650ea4740 1 -- 172.21.15.89:0/1793623577 shutdown_connections 2024-09-15T23:38:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.706+0000 7f7650ea4740 1 -- 172.21.15.89:0/1793623577 wait complete. 2024-09-15T23:38:15.715 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.713+0000 7f7650ea4740 20 client.20346 trim_cache size 0 max 16384 2024-09-15T23:38:15.730 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7650ea4740 1 Processor -- start 2024-09-15T23:38:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7650ea4740 1 -- start start 2024-09-15T23:38:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e295380b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e2954f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2955a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7650ea4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f76481aa7d0 con 0x561e29543370 2024-09-15T23:38:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7650ea4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f76481ac310 con 0x561e2953d9a0 2024-09-15T23:38:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7650ea4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f76481aaea0 con 0x561e2953e2e0 2024-09-15T23:38:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7647fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2955a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7647fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2955a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43870/0 (socket says 172.21.15.89:43870) 2024-09-15T23:38:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7647fff640 1 -- 172.21.15.89:0/2137272564 learned_addr learned my addr 172.21.15.89:0/2137272564 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7646ffd640 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e2954f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f76477fe640 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e295380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7647fff640 1 -- 172.21.15.89:0/2137272564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 msgr2=0x561e2954f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7647fff640 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e2954f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7647fff640 1 -- 172.21.15.89:0/2137272564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 msgr2=0x561e295380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7647fff640 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e295380b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7647fff640 1 -- 172.21.15.89:0/2137272564 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e29538e30 con 0x561e29543370 2024-09-15T23:38:15.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7646ffd640 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e2954f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f76477fe640 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e295380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.724+0000 7f7647fff640 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2955a4e0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f7638098970 tx=0x7f763809a500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17255e0ecfef787 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f763d7fa640 1 -- 172.21.15.89:0/2137272564 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763801d040 con 0x561e29543370 2024-09-15T23:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f763d7fa640 1 -- 172.21.15.89:0/2137272564 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7638005560 con 0x561e29543370 2024-09-15T23:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f763d7fa640 1 -- 172.21.15.89:0/2137272564 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763809b440 con 0x561e29543370 2024-09-15T23:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- 172.21.15.89:0/2137272564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 msgr2=0x561e2955a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2955a4e0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f7638098970 tx=0x7f763809a500 comp rx=0 tx=0).stop 2024-09-15T23:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- 172.21.15.89:0/2137272564 shutdown_connections 2024-09-15T23:38:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e29543370 0x561e2955a4e0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e2954f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 --2- 172.21.15.89:0/2137272564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e2953d9a0 0x561e295380b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- 172.21.15.89:0/2137272564 >> 172.21.15.89:0/2137272564 conn(0x561e29539010 msgr2=0x561e295379a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- 172.21.15.89:0/2137272564 shutdown_connections 2024-09-15T23:38:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- 172.21.15.89:0/2137272564 wait complete. 2024-09-15T23:38:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- start start 2024-09-15T23:38:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 Processor -- start 2024-09-15T23:38:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953d9a0 0x561e2954c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e2954cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e29543370 0x561e2954de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f76481abc40 con 0x561e2953d9a0 2024-09-15T23:38:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f76481ab570 con 0x561e29543370 2024-09-15T23:38:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f76481a87b0 con 0x561e2953e2e0 2024-09-15T23:38:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 10 client.0 _ll_register_callbacks cb 0x7f7638005860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f76477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953d9a0 0x561e2954c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f76477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953d9a0 0x561e2954c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43884/0 (socket says 172.21.15.89:43884) 2024-09-15T23:38:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f76477fe640 1 -- 172.21.15.89:0/2207970174 learned_addr learned my addr 172.21.15.89:0/2207970174 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7647fff640 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e29543370 0x561e2954de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7646ffd640 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e2954cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f76477fe640 1 -- 172.21.15.89:0/2207970174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 msgr2=0x561e2954cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f76477fe640 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e2954cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f76477fe640 1 -- 172.21.15.89:0/2207970174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e29543370 msgr2=0x561e2954de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f76477fe640 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e29543370 0x561e2954de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f76477fe640 1 -- 172.21.15.89:0/2207970174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e2954e410 con 0x561e2953d9a0 2024-09-15T23:38:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f76477fe640 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953d9a0 0x561e2954c8b0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f764000ab10 tx=0x7f7640005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9bf4877d38c4f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7626ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7626ffd640 1 -- 172.21.15.89:0/2207970174 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764002f070 con 0x561e2953d9a0 2024-09-15T23:38:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7626ffd640 1 -- 172.21.15.89:0/2207970174 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7640004250 con 0x561e2953d9a0 2024-09-15T23:38:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- 172.21.15.89:0/2207970174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7638005560 con 0x561e2953d9a0 2024-09-15T23:38:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7626ffd640 1 -- 172.21.15.89:0/2207970174 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f764003b050 con 0x561e2953d9a0 2024-09-15T23:38:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 10 client.20352 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- 172.21.15.89:0/2207970174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561e295be680 con 0x561e2953d9a0 2024-09-15T23:38:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7626ffd640 1 -- 172.21.15.89:0/2207970174 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7638005560 con 0x561e2953d9a0 2024-09-15T23:38:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7626ffd640 1 -- 172.21.15.89:0/2207970174 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7640004950 con 0x561e2953d9a0 2024-09-15T23:38:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 10 client.20352 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 10 client.20352 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 20 client.20352 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e295beea0 0x561e295c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 4 client.20352 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:38:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7650ea4740 1 -- 172.21.15.89:0/2207970174 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561e294a8d70 con 0x561e295beea0 2024-09-15T23:38:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.729+0000 7f7646ffd640 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e295beea0 0x561e295c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.732+0000 7f7646ffd640 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e295beea0 0x561e295c12c0 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96d0a368dbf05d98 server_cookie=46904560b935be5f in_seq=0 out_seq=0 2024-09-15T23:38:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.732+0000 7f7626ffd640 10 client.20352 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7626ffd640 1 -- 172.21.15.89:0/2207970174 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561e294a8d70 con 0x561e295beea0 2024-09-15T23:38:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7626ffd640 10 client.20352 handle_command_reply: tid=0 2024-09-15T23:38:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 client.20352 shutdown 2024-09-15T23:38:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 -- 172.21.15.89:0/2207970174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953d9a0 msgr2=0x561e2954c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953d9a0 0x561e2954c8b0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f764000ab10 tx=0x7f7640005bc0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 -- 172.21.15.89:0/2207970174 shutdown_connections 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561e295beea0 0x561e295c12c0 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561e29543370 0x561e2954de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561e2953e2e0 0x561e2954cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7626ffd640 0 client.20352 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 --2- 172.21.15.89:0/2207970174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561e2953d9a0 0x561e2954c8b0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 -- 172.21.15.89:0/2207970174 >> 172.21.15.89:0/2207970174 conn(0x561e29539010 msgr2=0x561e295381c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 -- 172.21.15.89:0/2207970174 shutdown_connections 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 1 -- 172.21.15.89:0/2207970174 wait complete. 2024-09-15T23:38:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.755+0000 7f7650ea4740 20 client.20352 trim_cache size 0 max 16384 2024-09-15T23:38:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:38:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:38:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:38:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:38:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 -- 172.21.15.89:0/3201428255 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76340a32f0 msgr2=0x7f76340a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76340a32f0 0x7f76340a57b0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7630000c00 tx=0x7f7630005a20 comp rx=0 tx=0).stop 2024-09-15T23:38:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 -- 172.21.15.89:0/3201428255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7648102600 msgr2=0x7f764819f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7648102600 0x7f764819f400 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f763800cad0 tx=0x7f763800cf90 comp rx=0 tx=0).stop 2024-09-15T23:38:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 -- 172.21.15.89:0/3201428255 shutdown_connections 2024-09-15T23:38:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f76340a32f0 0x7f76340a57b0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7648102600 0x7f764819f400 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7648101c40 0x7f7648198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 --2- 172.21.15.89:0/3201428255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7648100a50 0x7f7648197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 -- 172.21.15.89:0/3201428255 >> 172.21.15.89:0/3201428255 conn(0x7f76480fc1b0 msgr2=0x7f76480fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 -- 172.21.15.89:0/3201428255 shutdown_connections 2024-09-15T23:38:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:15.756+0000 7f764e113640 1 -- 172.21.15.89:0/3201428255 wait complete. 2024-09-15T23:38:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.899+0000 7fb67e89b640 1 -- 172.21.15.89:0/2247845641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67810c150 msgr2=0x7fb67810e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.919 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.899+0000 7fb67e89b640 1 --2- 172.21.15.89:0/2247845641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67810c150 0x7fb67810e5f0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fb670009ca0 tx=0x7fb67002a950 comp rx=0 tx=0).stop 2024-09-15T23:38:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.900+0000 7fb67e89b640 1 -- 172.21.15.89:0/2247845641 shutdown_connections 2024-09-15T23:38:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.900+0000 7fb67e89b640 1 --2- 172.21.15.89:0/2247845641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67810c150 0x7fb67810e5f0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.900+0000 7fb67e89b640 1 --2- 172.21.15.89:0/2247845641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb678103640 0x7fb67810bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.900+0000 7fb67e89b640 1 --2- 172.21.15.89:0/2247845641 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb678102d20 0x7fb678103100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.900+0000 7fb67e89b640 1 -- 172.21.15.89:0/2247845641 >> 172.21.15.89:0/2247845641 conn(0x7fb6780fc780 msgr2=0x7fb6780feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.900+0000 7fb67e89b640 1 -- 172.21.15.89:0/2247845641 shutdown_connections 2024-09-15T23:38:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.900+0000 7fb67e89b640 1 -- 172.21.15.89:0/2247845641 wait complete. 2024-09-15T23:38:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.900+0000 7fb67e89b640 1 Processor -- start 2024-09-15T23:38:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.900+0000 7fb67e89b640 1 -- start start 2024-09-15T23:38:15.920 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67e89b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb678102d20 0x7fb678100980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67e89b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb678103640 0x7fb678100ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67e89b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67810c150 0x7fb678101400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67e89b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb678111f10 con 0x7fb67810c150 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67e89b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb678111d90 con 0x7fb678103640 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67e89b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb678112090 con 0x7fb678102d20 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67ce38640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67810c150 0x7fb678101400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67ce38640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67810c150 0x7fb678101400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43904/0 (socket says 172.21.15.89:43904) 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67ce38640 1 -- 172.21.15.89:0/3796028287 learned_addr learned my addr 172.21.15.89:0/3796028287 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb677fff640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb678102d20 0x7fb678100980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb6777fe640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb678103640 0x7fb678100ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67ce38640 1 -- 172.21.15.89:0/3796028287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb678102d20 msgr2=0x7fb678100980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.921 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67ce38640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb678102d20 0x7fb678100980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67ce38640 1 -- 172.21.15.89:0/3796028287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb678103640 msgr2=0x7fb678100ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67ce38640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb678103640 0x7fb678100ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67ce38640 1 -- 172.21.15.89:0/3796028287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6781a68d0 con 0x7fb67810c150 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67ce38640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67810c150 0x7fb678101400 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fb67002ae70 tx=0x7fb67002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb6757fa640 1 -- 172.21.15.89:0/3796028287 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670034040 con 0x7fb67810c150 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb6757fa640 1 -- 172.21.15.89:0/3796028287 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb670030920 con 0x7fb67810c150 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb6757fa640 1 -- 172.21.15.89:0/3796028287 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670045480 con 0x7fb67810c150 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67e89b640 1 -- 172.21.15.89:0/3796028287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6781a6b60 con 0x7fb67810c150 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.901+0000 7fb67e89b640 1 -- 172.21.15.89:0/3796028287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6781a7090 con 0x7fb67810c150 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb6757fa640 1 -- 172.21.15.89:0/3796028287 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb670037020 con 0x7fb67810c150 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb680c74740 1 Processor -- start 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb680c74740 1 -- start start 2024-09-15T23:38:15.922 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb680c74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16aacb40 0x557c16aab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb680c74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aab5d0 0x557c16aaba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb680c74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16ab6370 0x557c16ab8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb680c74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557c16a112f0 con 0x557c16ab6370 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb680c74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557c169b8cf0 con 0x557c16aab5d0 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb680c74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557c16a0d2b0 con 0x557c16aacb40 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb6757fa640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6600a32c0 0x7fb6600a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16ab6370 0x557c16ab8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16ab6370 0x557c16ab8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43914/0 (socket says 172.21.15.89:43914) 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb6777fe640 1 -- 172.21.15.89:0/1761347665 learned_addr learned my addr 172.21.15.89:0/1761347665 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.904+0000 7fb67ce38640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6600a32c0 0x7fb6600a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1761347665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16aacb40 0x557c16aab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb6757fa640 1 -- 172.21.15.89:0/3796028287 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb6700f0b50 con 0x7fb67810c150 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb677fff640 1 --2- 172.21.15.89:0/1761347665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aab5d0 0x557c16aaba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb6777fe640 1 -- 172.21.15.89:0/1761347665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16aacb40 msgr2=0x557c16aab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1761347665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16aacb40 0x557c16aab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb6777fe640 1 -- 172.21.15.89:0/1761347665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aab5d0 msgr2=0x557c16aaba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1761347665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aab5d0 0x557c16aaba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb6777fe640 1 -- 172.21.15.89:0/1761347665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c1699d3a0 con 0x557c16ab6370 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb677fff640 1 --2- 172.21.15.89:0/1761347665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aab5d0 0x557c16aaba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1761347665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16ab6370 0x557c16ab8760 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fb66c012730 tx=0x7fb66c00b6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce390b52cd5c8c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb67ce38640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6600a32c0 0x7fb6600a5780 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb6700ee5d0 tx=0x7fb67008eb50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb65e7fc640 1 -- 172.21.15.89:0/1761347665 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c012cb0 con 0x557c16ab6370 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb65e7fc640 1 -- 172.21.15.89:0/1761347665 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb66c012e50 con 0x557c16ab6370 2024-09-15T23:38:15.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.905+0000 7fb65e7fc640 1 -- 172.21.15.89:0/1761347665 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c01eae0 con 0x557c16ab6370 2024-09-15T23:38:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.906+0000 7fb680c74740 1 -- 172.21.15.89:0/1761347665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16ab6370 msgr2=0x557c16ab8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.906+0000 7fb680c74740 1 --2- 172.21.15.89:0/1761347665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16ab6370 0x557c16ab8760 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fb66c012730 tx=0x7fb66c00b6f0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.906+0000 7fb680c74740 1 -- 172.21.15.89:0/1761347665 shutdown_connections 2024-09-15T23:38:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.906+0000 7fb680c74740 1 --2- 172.21.15.89:0/1761347665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16ab6370 0x557c16ab8760 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.906+0000 7fb680c74740 1 --2- 172.21.15.89:0/1761347665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aab5d0 0x557c16aaba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.906+0000 7fb680c74740 1 --2- 172.21.15.89:0/1761347665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16aacb40 0x557c16aab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.906+0000 7fb680c74740 1 -- 172.21.15.89:0/1761347665 >> 172.21.15.89:0/1761347665 conn(0x557c16a74310 msgr2=0x557c16aa7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.906+0000 7fb680c74740 1 -- 172.21.15.89:0/1761347665 shutdown_connections 2024-09-15T23:38:15.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.906+0000 7fb680c74740 1 -- 172.21.15.89:0/1761347665 wait complete. 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.906+0000 7fb680c74740 1 -- start start 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb680c74740 1 Processor -- start 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb680c74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ab5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb680c74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 0x557c16ab2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb680c74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16ab6370 0x557c16ab35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb680c74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x557c16ab9070 con 0x557c16aab5d0 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb680c74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557c169cc060 con 0x557c16aacb40 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb680c74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557c16ab91f0 con 0x557c16ab6370 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb680c74740 10 client.0 _ll_register_callbacks cb 0x557c1671f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ab5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ab5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43930/0 (socket says 172.21.15.89:43930) 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb6777fe640 1 -- 172.21.15.89:0/1157258383 learned_addr learned my addr 172.21.15.89:0/1157258383 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb677fff640 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 0x557c16ab2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16ab6370 0x557c16ab35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb6777fe640 1 -- 172.21.15.89:0/1157258383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16ab6370 msgr2=0x557c16ab35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16ab6370 0x557c16ab35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb6777fe640 1 -- 172.21.15.89:0/1157258383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 msgr2=0x557c16ab2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 0x557c16ab2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.907+0000 7fb6777fe640 1 -- 172.21.15.89:0/1157258383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c167823c0 con 0x557c16aab5d0 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ab5d30 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fb66c015040 tx=0x7fb66c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d424cab0265e10f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb6577fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb6577fe640 1 -- 172.21.15.89:0/1157258383 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c024040 con 0x557c16aab5d0 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb6577fe640 1 -- 172.21.15.89:0/1157258383 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb66c005030 con 0x557c16aab5d0 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb680c74740 1 -- 172.21.15.89:0/1157258383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb66c012e50 con 0x557c16aab5d0 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb6577fe640 1 -- 172.21.15.89:0/1157258383 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb66c01ee30 con 0x557c16aab5d0 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb680c74740 10 client.20364 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb680c74740 1 -- 172.21.15.89:0/1157258383 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557c16ac4b00 con 0x557c16aab5d0 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb6577fe640 1 -- 172.21.15.89:0/1157258383 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb66c012e50 con 0x557c16aab5d0 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb6577fe640 1 -- 172.21.15.89:0/1157258383 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb66c0046c0 con 0x557c16aab5d0 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb680c74740 10 client.20364 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb680c74740 10 client.20364 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb680c74740 20 client.20364 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb680c74740 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557c16ac5f00 0x557c16ac82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb680c74740 4 client.20364 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.908+0000 7fb680c74740 1 -- 172.21.15.89:0/1157258383 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c16a1e860 con 0x557c16ac5f00 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.909+0000 7fb677fff640 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557c16ac5f00 0x557c16ac82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.909+0000 7fb6577fe640 10 client.20364 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.909+0000 7fb677fff640 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557c16ac5f00 0x557c16ac82f0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=48568569af48ddcb server_cookie=bc531d55f80c4b39 in_seq=0 out_seq=0 2024-09-15T23:38:15.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb6577fe640 1 -- 172.21.15.89:0/1157258383 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557c16a1e860 con 0x557c16ac5f00 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb6577fe640 10 client.20364 handle_command_reply: tid=0 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 client.20364 shutdown 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 -- 172.21.15.89:0/1157258383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 msgr2=0x557c16ab5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ab5d30 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fb66c015040 tx=0x7fb66c005850 comp rx=0 tx=0).stop 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 -- 172.21.15.89:0/1157258383 shutdown_connections 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557c16ac5f00 0x557c16ac82f0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16ab6370 0x557c16ab35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 0x557c16ab2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 --2- 172.21.15.89:0/1157258383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ab5d30 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 -- 172.21.15.89:0/1157258383 >> 172.21.15.89:0/1157258383 conn(0x557c16a74310 msgr2=0x557c16aad960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb6577fe640 0 client.20364 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 -- 172.21.15.89:0/1157258383 shutdown_connections 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 1 -- 172.21.15.89:0/1157258383 wait complete. 2024-09-15T23:38:15.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.910+0000 7fb680c74740 20 client.20364 trim_cache size 0 max 16384 2024-09-15T23:38:15.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.923+0000 7fb680c74740 1 Processor -- start 2024-09-15T23:38:15.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.923+0000 7fb680c74740 1 -- start start 2024-09-15T23:38:15.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.923+0000 7fb680c74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ac2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.923+0000 7fb680c74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16aacb40 0x557c16ab28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.934 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.923+0000 7fb680c74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16ab6370 0x557c16ab1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.923+0000 7fb680c74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781b1020 con 0x557c16aab5d0 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.923+0000 7fb680c74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781b2b60 con 0x557c16ab6370 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.923+0000 7fb680c74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781b16f0 con 0x557c16aacb40 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ac2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ac2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43940/0 (socket says 172.21.15.89:43940) 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb6777fe640 1 -- 172.21.15.89:0/1179334174 learned_addr learned my addr 172.21.15.89:0/1179334174 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb677fff640 1 --2- 172.21.15.89:0/1179334174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16aacb40 0x557c16ab28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb6777fe640 1 -- 172.21.15.89:0/1179334174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16aacb40 msgr2=0x557c16ab28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1179334174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16aacb40 0x557c16ab28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb6777fe640 1 -- 172.21.15.89:0/1179334174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16ab6370 msgr2=0x557c16ab1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1179334174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16ab6370 0x557c16ab1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb6777fe640 1 -- 172.21.15.89:0/1179334174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c16ab8f50 con 0x557c16aab5d0 2024-09-15T23:38:15.935 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb6777fe640 1 --2- 172.21.15.89:0/1179334174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ac2340 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fb66c004cb0 tx=0x7fb66c0126e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0de81119c71c86 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb65dffb640 1 -- 172.21.15.89:0/1179334174 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c03d070 con 0x557c16aab5d0 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb65dffb640 1 -- 172.21.15.89:0/1179334174 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb66c012e50 con 0x557c16aab5d0 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.924+0000 7fb65dffb640 1 -- 172.21.15.89:0/1179334174 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c010c20 con 0x557c16aab5d0 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 -- 172.21.15.89:0/1179334174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 msgr2=0x557c16ac2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 --2- 172.21.15.89:0/1179334174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ac2340 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fb66c004cb0 tx=0x7fb66c0126e0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 -- 172.21.15.89:0/1179334174 shutdown_connections 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 --2- 172.21.15.89:0/1179334174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16ab6370 0x557c16ab1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 --2- 172.21.15.89:0/1179334174 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16aacb40 0x557c16ab28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 --2- 172.21.15.89:0/1179334174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16ac2340 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 -- 172.21.15.89:0/1179334174 >> 172.21.15.89:0/1179334174 conn(0x557c16a74310 msgr2=0x557c16ab18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 -- 172.21.15.89:0/1179334174 shutdown_connections 2024-09-15T23:38:15.936 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 -- 172.21.15.89:0/1179334174 wait complete. 2024-09-15T23:38:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 -- start start 2024-09-15T23:38:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.925+0000 7fb680c74740 1 Processor -- start 2024-09-15T23:38:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb680c74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16abe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb680c74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 0x557c16abe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb680c74740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16ab6370 0x557c16aaf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb680c74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781b2490 con 0x557c16aab5d0 2024-09-15T23:38:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb680c74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781b1dc0 con 0x557c16aacb40 2024-09-15T23:38:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb680c74740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781af000 con 0x557c16ab6370 2024-09-15T23:38:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb680c74740 10 client.0 _ll_register_callbacks cb 0x7fb66c00be00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:15.937 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16abe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16abe4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43952/0 (socket says 172.21.15.89:43952) 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6777fe640 1 -- 172.21.15.89:0/4171291285 learned_addr learned my addr 172.21.15.89:0/4171291285 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6777fe640 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16ab6370 0x557c16aaf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb677fff640 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 0x557c16abe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6777fe640 1 -- 172.21.15.89:0/4171291285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16ab6370 msgr2=0x557c16aaf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6777fe640 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16ab6370 0x557c16aaf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6777fe640 1 -- 172.21.15.89:0/4171291285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 msgr2=0x557c16abe9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6777fe640 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 0x557c16abe9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6777fe640 1 -- 172.21.15.89:0/4171291285 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c16abf090 con 0x557c16aab5d0 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb677fff640 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 0x557c16abe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6777fe640 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16abe4b0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fb66c004c30 tx=0x7fb66c012480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1955c93238eae6d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:15.938 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6577fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6577fe640 1 -- 172.21.15.89:0/4171291285 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c03d070 con 0x557c16aab5d0 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb6577fe640 1 -- 172.21.15.89:0/4171291285 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb66c024040 con 0x557c16aab5d0 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.926+0000 7fb680c74740 1 -- 172.21.15.89:0/4171291285 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb66c012e50 con 0x557c16aab5d0 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb6577fe640 1 -- 172.21.15.89:0/4171291285 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb66c048050 con 0x557c16aab5d0 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb680c74740 10 client.20370 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb680c74740 1 -- 172.21.15.89:0/4171291285 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557c16b2ff00 con 0x557c16aab5d0 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb6577fe640 1 -- 172.21.15.89:0/4171291285 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb66c012e50 con 0x557c16aab5d0 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb6577fe640 1 -- 172.21.15.89:0/4171291285 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb66c005210 con 0x557c16aab5d0 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb680c74740 10 client.20370 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb680c74740 10 client.20370 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:38:15.939 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb680c74740 20 client.20370 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb680c74740 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557c16b305a0 0x557c16b329c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb680c74740 4 client.20370 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb680c74740 1 -- 172.21.15.89:0/4171291285 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c16a18d70 con 0x557c16b305a0 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.927+0000 7fb677fff640 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557c16b305a0 0x557c16b329c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb6577fe640 10 client.20370 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb677fff640 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557c16b305a0 0x557c16b329c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aa82e60b4590dc45 server_cookie=83ceec6cfcce7141 in_seq=0 out_seq=0 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb6577fe640 1 -- 172.21.15.89:0/4171291285 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c16a18d70 con 0x557c16b305a0 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb6577fe640 10 client.20370 handle_command_reply: tid=0 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 client.20370 shutdown 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 -- 172.21.15.89:0/4171291285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 msgr2=0x557c16abe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16abe4b0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fb66c004c30 tx=0x7fb66c012480 comp rx=0 tx=0).stop 2024-09-15T23:38:15.940 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 -- 172.21.15.89:0/4171291285 shutdown_connections 2024-09-15T23:38:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x557c16b305a0 0x557c16b329c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557c16ab6370 0x557c16aaf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c16aacb40 0x557c16abe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 --2- 172.21.15.89:0/4171291285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c16aab5d0 0x557c16abe4b0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb6577fe640 0 client.20370 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:38:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 -- 172.21.15.89:0/4171291285 >> 172.21.15.89:0/4171291285 conn(0x557c16a74310 msgr2=0x557c16ab20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 -- 172.21.15.89:0/4171291285 shutdown_connections 2024-09-15T23:38:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 1 -- 172.21.15.89:0/4171291285 wait complete. 2024-09-15T23:38:15.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.928+0000 7fb680c74740 20 client.20370 trim_cache size 0 max 16384 2024-09-15T23:38:15.941 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:38:15.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 -- 172.21.15.89:0/3796028287 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6600a32c0 msgr2=0x7fb6600a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6600a32c0 0x7fb6600a5780 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb6700ee5d0 tx=0x7fb67008eb50 comp rx=0 tx=0).stop 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 -- 172.21.15.89:0/3796028287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67810c150 msgr2=0x7fb678101400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67810c150 0x7fb678101400 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fb67002ae70 tx=0x7fb67002f5b0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 -- 172.21.15.89:0/3796028287 shutdown_connections 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb6600a32c0 0x7fb6600a5780 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67810c150 0x7fb678101400 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb678103640 0x7fb678100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 --2- 172.21.15.89:0/3796028287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb678102d20 0x7fb678100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 -- 172.21.15.89:0/3796028287 >> 172.21.15.89:0/3796028287 conn(0x7fb6780fc780 msgr2=0x7fb6780fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 -- 172.21.15.89:0/3796028287 shutdown_connections 2024-09-15T23:38:15.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:15.929+0000 7fb67e89b640 1 -- 172.21.15.89:0/3796028287 wait complete. 2024-09-15T23:38:16.748 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:38:17.036 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:17.299 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:17 smithi089 ceph-mon[30933]: pgmap v777: 129 pgs: 129 active+clean; 3.2 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:38:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:17 smithi120 ceph-mon[35250]: pgmap v777: 129 pgs: 129 active+clean; 3.2 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:38:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:17 smithi174 ceph-mon[35586]: pgmap v777: 129 pgs: 129 active+clean; 3.2 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.511+0000 7fcf80eb1640 1 -- 172.21.15.89:0/747520241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf7c1036c0 msgr2=0x7fcf7c10bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.511+0000 7fcf80eb1640 1 --2- 172.21.15.89:0/747520241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf7c1036c0 0x7fcf7c10bbb0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009900 tx=0x7fcf7002a480 comp rx=0 tx=0).stop 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.511+0000 7fcf80eb1640 1 -- 172.21.15.89:0/747520241 shutdown_connections 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.511+0000 7fcf80eb1640 1 --2- 172.21.15.89:0/747520241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf7c10c180 0x7fcf7c10e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.511+0000 7fcf80eb1640 1 --2- 172.21.15.89:0/747520241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf7c1036c0 0x7fcf7c10bbb0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.511+0000 7fcf80eb1640 1 --2- 172.21.15.89:0/747520241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c102da0 0x7fcf7c103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.511+0000 7fcf80eb1640 1 -- 172.21.15.89:0/747520241 >> 172.21.15.89:0/747520241 conn(0x7fcf7c0fc7a0 msgr2=0x7fcf7c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.511+0000 7fcf80eb1640 1 -- 172.21.15.89:0/747520241 shutdown_connections 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.511+0000 7fcf80eb1640 1 -- 172.21.15.89:0/747520241 wait complete. 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf80eb1640 1 Processor -- start 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf80eb1640 1 -- start start 2024-09-15T23:38:17.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf80eb1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf7c102da0 0x7fcf7c1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf80eb1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf7c1036c0 0x7fcf7c198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf80eb1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c10c180 0x7fcf7c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf80eb1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c111fa0 con 0x7fcf7c1036c0 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf80eb1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c111e20 con 0x7fcf7c102da0 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf80eb1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c112120 con 0x7fcf7c10c180 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf79d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf7c1036c0 0x7fcf7c198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf79d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf7c1036c0 0x7fcf7c198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43960/0 (socket says 172.21.15.89:43960) 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.512+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3849382833 learned_addr learned my addr 172.21.15.89:0/3849382833 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf7ad9d640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c10c180 0x7fcf7c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf7a59c640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf7c102da0 0x7fcf7c1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3849382833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c10c180 msgr2=0x7fcf7c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c10c180 0x7fcf7c19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3849382833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf7c102da0 msgr2=0x7fcf7c1984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:17.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf7c102da0 0x7fcf7c1984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3849382833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf7c19d490 con 0x7fcf7c1036c0 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf7c1036c0 0x7fcf7c198a20 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fcf70005ab0 tx=0x7fcf700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf6b7fe640 1 -- 172.21.15.89:0/3849382833 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7003f030 con 0x7fcf7c1036c0 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf80eb1640 1 -- 172.21.15.89:0/3849382833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf7c19d720 con 0x7fcf7c1036c0 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf6b7fe640 1 -- 172.21.15.89:0/3849382833 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf70003ce0 con 0x7fcf7c1036c0 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf6b7fe640 1 -- 172.21.15.89:0/3849382833 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70035070 con 0x7fcf7c1036c0 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.513+0000 7fcf80eb1640 1 -- 172.21.15.89:0/3849382833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf7c1a5060 con 0x7fcf7c1036c0 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.516+0000 7fcf6b7fe640 1 -- 172.21.15.89:0/3849382833 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcf70042030 con 0x7fcf7c1036c0 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.516+0000 7fcf8328a740 1 Processor -- start 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.516+0000 7fcf8328a740 1 -- start start 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.516+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b11090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.516+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b115d0 0x561786b11a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.516+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b1e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.516+0000 7fcf8328a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561786a772f0 con 0x561786b12b40 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.516+0000 7fcf8328a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561786a1ecf0 con 0x561786b1c370 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf8328a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561786a732b0 con 0x561786b115d0 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b11090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b11090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43976/0 (socket says 172.21.15.89:43976) 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 -- 172.21.15.89:0/1765550858 learned_addr learned my addr 172.21.15.89:0/1765550858 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf6b7fe640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf640a3240 0x7fcf640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7a59c640 1 --2- 172.21.15.89:0/1765550858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b115d0 0x561786b11a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/1765550858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b1e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 -- 172.21.15.89:0/1765550858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b115d0 msgr2=0x561786b11a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 --2- 172.21.15.89:0/1765550858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b115d0 0x561786b11a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 -- 172.21.15.89:0/1765550858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 msgr2=0x561786b1e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 --2- 172.21.15.89:0/1765550858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b1e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 -- 172.21.15.89:0/1765550858 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561786a033a0 con 0x561786b12b40 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/1765550858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b1e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf6b7fe640 1 -- 172.21.15.89:0/3849382833 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcf700f0700 con 0x7fcf7c1036c0 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf640a3240 0x7fcf640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 --2- 172.21.15.89:0/1765550858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b11090 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fcf74007fd0 tx=0x7fcf7400e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fe277f37b1310b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf68ff9640 1 -- 172.21.15.89:0/1765550858 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf74022070 con 0x561786b12b40 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf68ff9640 1 -- 172.21.15.89:0/1765550858 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf74011c30 con 0x561786b12b40 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf68ff9640 1 -- 172.21.15.89:0/1765550858 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7401a400 con 0x561786b12b40 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.517+0000 7fcf7ad9d640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf640a3240 0x7fcf640a5700 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fcf7401f000 tx=0x7fcf74020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.518+0000 7fcf8328a740 1 -- 172.21.15.89:0/1765550858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 msgr2=0x561786b11090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.518+0000 7fcf8328a740 1 --2- 172.21.15.89:0/1765550858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b11090 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fcf74007fd0 tx=0x7fcf7400e7e0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.518+0000 7fcf8328a740 1 -- 172.21.15.89:0/1765550858 shutdown_connections 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.518+0000 7fcf8328a740 1 --2- 172.21.15.89:0/1765550858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b1e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.518+0000 7fcf8328a740 1 --2- 172.21.15.89:0/1765550858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b115d0 0x561786b11a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.518+0000 7fcf8328a740 1 --2- 172.21.15.89:0/1765550858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b11090 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.518+0000 7fcf8328a740 1 -- 172.21.15.89:0/1765550858 >> 172.21.15.89:0/1765550858 conn(0x561786ada310 msgr2=0x561786b0dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.518+0000 7fcf8328a740 1 -- 172.21.15.89:0/1765550858 shutdown_connections 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.518+0000 7fcf8328a740 1 -- 172.21.15.89:0/1765550858 wait complete. 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf8328a740 1 -- start start 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf8328a740 1 Processor -- start 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b115d0 0x561786b1bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b19020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b19610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf8328a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561786b1f070 con 0x561786b12b40 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf8328a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561786a32060 con 0x561786b1c370 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf8328a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561786b1f1f0 con 0x561786b115d0 2024-09-15T23:38:17.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf8328a740 10 client.0 _ll_register_callbacks cb 0x5617867858c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf79d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b19020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf79d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b19020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43990/0 (socket says 172.21.15.89:43990) 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3530444634 learned_addr learned my addr 172.21.15.89:0/3530444634 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf7ad9d640 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b19610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf7a59c640 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b115d0 0x561786b1bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3530444634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b115d0 msgr2=0x561786b1bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b115d0 0x561786b1bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3530444634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 msgr2=0x561786b19610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b19610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.519+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3530444634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617867e83c0 con 0x561786b12b40 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b19020 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fcf700bc860 tx=0x7fcf70087700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0b65bd7c5e1ce4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf61ffb640 1 -- 172.21.15.89:0/3530444634 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70049070 con 0x561786b12b40 2024-09-15T23:38:17.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf61ffb640 1 -- 172.21.15.89:0/3530444634 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcf700b8f40 con 0x561786b12b40 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf8328a740 1 -- 172.21.15.89:0/3530444634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf74011c30 con 0x561786b12b40 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf61ffb640 1 -- 172.21.15.89:0/3530444634 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf700bcfd0 con 0x561786b12b40 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf8328a740 10 client.20382 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf8328a740 1 -- 172.21.15.89:0/3530444634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561786b2aa80 con 0x561786b12b40 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf61ffb640 1 -- 172.21.15.89:0/3530444634 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf74011c30 con 0x561786b12b40 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf61ffb640 1 -- 172.21.15.89:0/3530444634 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcf700f0350 con 0x561786b12b40 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf8328a740 10 client.20382 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf8328a740 10 client.20382 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf8328a740 20 client.20382 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf8328a740 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561786b2be90 0x561786b2e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf8328a740 4 client.20382 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.520+0000 7fcf8328a740 1 -- 172.21.15.89:0/3530444634 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561786a84860 con 0x561786b2be90 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.521+0000 7fcf7a59c640 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561786b2be90 0x561786b2e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.521+0000 7fcf7a59c640 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561786b2be90 0x561786b2e280 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=35ac9d58d86ac115 server_cookie=3502729d135c320e in_seq=0 out_seq=0 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.521+0000 7fcf61ffb640 10 client.20382 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf61ffb640 1 -- 172.21.15.89:0/3530444634 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561786a84860 con 0x561786b2be90 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf61ffb640 10 client.20382 handle_command_reply: tid=0 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 client.20382 shutdown 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 -- 172.21.15.89:0/3530444634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 msgr2=0x561786b19020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b19020 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fcf700bc860 tx=0x7fcf70087700 comp rx=0 tx=0).stop 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 -- 172.21.15.89:0/3530444634 shutdown_connections 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b19610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b19020 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561786b2be90 0x561786b2e280 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 --2- 172.21.15.89:0/3530444634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b115d0 0x561786b1bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 -- 172.21.15.89:0/3530444634 >> 172.21.15.89:0/3530444634 conn(0x561786ada310 msgr2=0x561786b13960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf61ffb640 0 client.20382 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:38:17.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 -- 172.21.15.89:0/3530444634 shutdown_connections 2024-09-15T23:38:17.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 1 -- 172.21.15.89:0/3530444634 wait complete. 2024-09-15T23:38:17.542 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.522+0000 7fcf8328a740 20 client.20382 trim_cache size 0 max 16384 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf8328a740 1 Processor -- start 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf8328a740 1 -- start start 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b115d0 0x561786b0de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b28310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b1c370 0x561786b14120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf8328a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1aed80 con 0x561786b12b40 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf8328a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1b08c0 con 0x561786b115d0 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf8328a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1af450 con 0x561786b1c370 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf79d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b28310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf79d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b28310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43994/0 (socket says 172.21.15.89:43994) 2024-09-15T23:38:17.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3503972247 learned_addr learned my addr 172.21.15.89:0/3503972247 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3503972247 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b1c370 msgr2=0x561786b14120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/3503972247 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b1c370 0x561786b14120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3503972247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b115d0 msgr2=0x561786b0de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/3503972247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b115d0 0x561786b0de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf79d9b640 1 -- 172.21.15.89:0/3503972247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561786b12760 con 0x561786b12b40 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.536+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/3503972247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b28310 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fcf70005940 tx=0x7fcf7002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986a31816609fbd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf63fff640 1 -- 172.21.15.89:0/3503972247 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70049020 con 0x561786b12b40 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf63fff640 1 -- 172.21.15.89:0/3503972247 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf700c0c80 con 0x561786b12b40 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf63fff640 1 -- 172.21.15.89:0/3503972247 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7003b630 con 0x561786b12b40 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf8328a740 1 -- 172.21.15.89:0/3503972247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 msgr2=0x561786b28310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf8328a740 1 --2- 172.21.15.89:0/3503972247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b28310 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fcf70005940 tx=0x7fcf7002a480 comp rx=0 tx=0).stop 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf8328a740 1 -- 172.21.15.89:0/3503972247 shutdown_connections 2024-09-15T23:38:17.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf8328a740 1 --2- 172.21.15.89:0/3503972247 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b1c370 0x561786b14120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf8328a740 1 --2- 172.21.15.89:0/3503972247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b12b40 0x561786b28310 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf8328a740 1 --2- 172.21.15.89:0/3503972247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b115d0 0x561786b0de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf8328a740 1 -- 172.21.15.89:0/3503972247 >> 172.21.15.89:0/3503972247 conn(0x561786ada310 msgr2=0x561786b0d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf8328a740 1 -- 172.21.15.89:0/3503972247 shutdown_connections 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.537+0000 7fcf8328a740 1 -- 172.21.15.89:0/3503972247 wait complete. 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf8328a740 1 -- start start 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf8328a740 1 Processor -- start 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b115d0 0x561786b24810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b12b40 0x561786b24d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf8328a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b15c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf8328a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1b01f0 con 0x561786b115d0 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf8328a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1afb20 con 0x561786b1c370 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf8328a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf7c1acd60 con 0x561786b12b40 2024-09-15T23:38:17.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf8328a740 10 client.0 _ll_register_callbacks cb 0x7fcf70005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf7a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b115d0 0x561786b24810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf7a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b115d0 0x561786b24810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44010/0 (socket says 172.21.15.89:44010) 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf7a59c640 1 -- 172.21.15.89:0/264693729 learned_addr learned my addr 172.21.15.89:0/264693729 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf7ad9d640 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b15c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b12b40 0x561786b24d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf7a59c640 1 -- 172.21.15.89:0/264693729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b12b40 msgr2=0x561786b24d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf7a59c640 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b12b40 0x561786b24d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf7a59c640 1 -- 172.21.15.89:0/264693729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 msgr2=0x561786b15c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf7a59c640 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b15c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf7a59c640 1 -- 172.21.15.89:0/264693729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561786b16470 con 0x561786b115d0 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf7ad9d640 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b15c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.538+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b12b40 0x561786b24d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:17.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf7a59c640 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b115d0 0x561786b24810 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c006f90 tx=0x7fcf6c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63490f19165a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf61ffb640 1 -- 172.21.15.89:0/264693729 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c016d10 con 0x561786b115d0 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf61ffb640 1 -- 172.21.15.89:0/264693729 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcf6c024430 con 0x561786b115d0 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf8328a740 1 -- 172.21.15.89:0/264693729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf700c0c80 con 0x561786b115d0 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf61ffb640 1 -- 172.21.15.89:0/264693729 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf6c02d050 con 0x561786b115d0 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf8328a740 10 client.20388 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf8328a740 1 -- 172.21.15.89:0/264693729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561786b967e0 con 0x561786b115d0 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf61ffb640 1 -- 172.21.15.89:0/264693729 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf700c0c80 con 0x561786b115d0 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf61ffb640 1 -- 172.21.15.89:0/264693729 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcf6c0146d0 con 0x561786b115d0 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf8328a740 10 client.20388 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf8328a740 10 client.20388 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf8328a740 20 client.20388 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf8328a740 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561786b96f40 0x561786b99360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:17.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf8328a740 4 client.20388 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.539+0000 7fcf8328a740 1 -- 172.21.15.89:0/264693729 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561786a7ed70 con 0x561786b96f40 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.540+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561786b96f40 0x561786b99360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.540+0000 7fcf79d9b640 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561786b96f40 0x561786b99360 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4486b0833b435bc1 server_cookie=eb1536f2cf43bcde in_seq=0 out_seq=0 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.540+0000 7fcf61ffb640 10 client.20388 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.540+0000 7fcf61ffb640 1 -- 172.21.15.89:0/264693729 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561786a7ed70 con 0x561786b96f40 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.540+0000 7fcf61ffb640 10 client.20388 handle_command_reply: tid=0 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.540+0000 7fcf8328a740 1 client.20388 shutdown 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.540+0000 7fcf8328a740 1 -- 172.21.15.89:0/264693729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b115d0 msgr2=0x561786b24810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.540+0000 7fcf8328a740 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b115d0 0x561786b24810 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c006f90 tx=0x7fcf6c00b840 comp rx=0 tx=0).stop 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.540+0000 7fcf8328a740 1 -- 172.21.15.89:0/264693729 shutdown_connections 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.541+0000 7fcf8328a740 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561786b1c370 0x561786b15c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.541+0000 7fcf8328a740 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x561786b96f40 0x561786b99360 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.541+0000 7fcf8328a740 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561786b12b40 0x561786b24d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.541+0000 7fcf61ffb640 0 client.20388 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:38:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.541+0000 7fcf8328a740 1 --2- 172.21.15.89:0/264693729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561786b115d0 0x561786b24810 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.541+0000 7fcf8328a740 1 -- 172.21.15.89:0/264693729 >> 172.21.15.89:0/264693729 conn(0x561786ada310 msgr2=0x561786b0c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.541+0000 7fcf8328a740 1 -- 172.21.15.89:0/264693729 shutdown_connections 2024-09-15T23:38:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.541+0000 7fcf8328a740 1 -- 172.21.15.89:0/264693729 wait complete. 2024-09-15T23:38:17.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.541+0000 7fcf8328a740 20 client.20388 trim_cache size 0 max 16384 2024-09-15T23:38:17.553 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 -- 172.21.15.89:0/3849382833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf640a3240 msgr2=0x7fcf640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf640a3240 0x7fcf640a5700 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fcf7401f000 tx=0x7fcf74020000 comp rx=0 tx=0).stop 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 -- 172.21.15.89:0/3849382833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf7c1036c0 msgr2=0x7fcf7c198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf7c1036c0 0x7fcf7c198a20 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fcf70005ab0 tx=0x7fcf700042f0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 -- 172.21.15.89:0/3849382833 shutdown_connections 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcf640a3240 0x7fcf640a5700 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf7c10c180 0x7fcf7c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf7c1036c0 0x7fcf7c198a20 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 --2- 172.21.15.89:0/3849382833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf7c102da0 0x7fcf7c1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 -- 172.21.15.89:0/3849382833 >> 172.21.15.89:0/3849382833 conn(0x7fcf7c0fc7a0 msgr2=0x7fcf7c0fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:17.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 -- 172.21.15.89:0/3849382833 shutdown_connections 2024-09-15T23:38:17.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:17.542+0000 7fcf80eb1640 1 -- 172.21.15.89:0/3849382833 wait complete. 2024-09-15T23:38:18.387 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:38:18.638 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:19.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.138+0000 7fe191525640 1 -- 172.21.15.89:0/4231809169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe18c106a80 msgr2=0x7fe18c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:19.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.138+0000 7fe191525640 1 --2- 172.21.15.89:0/4231809169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe18c106a80 0x7fe18c106e60 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009930 tx=0x7fe17c02a420 comp rx=0 tx=0).stop 2024-09-15T23:38:19.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.138+0000 7fe191525640 1 -- 172.21.15.89:0/4231809169 shutdown_connections 2024-09-15T23:38:19.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.138+0000 7fe191525640 1 --2- 172.21.15.89:0/4231809169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c101420 0x7fe18c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.138+0000 7fe191525640 1 --2- 172.21.15.89:0/4231809169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe18c100a80 0x7fe18c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.138+0000 7fe191525640 1 --2- 172.21.15.89:0/4231809169 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe18c106a80 0x7fe18c106e60 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.138+0000 7fe191525640 1 -- 172.21.15.89:0/4231809169 >> 172.21.15.89:0/4231809169 conn(0x7fe18c0fc7a0 msgr2=0x7fe18c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.139+0000 7fe191525640 1 -- 172.21.15.89:0/4231809169 shutdown_connections 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.139+0000 7fe191525640 1 -- 172.21.15.89:0/4231809169 wait complete. 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.139+0000 7fe191525640 1 Processor -- start 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.139+0000 7fe191525640 1 -- start start 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe191525640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c100a80 0x7fe18c0fce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe191525640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe18c101420 0x7fe18c0fd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe191525640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe18c106a80 0x7fe18c0fd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe191525640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c110e80 con 0x7fe18c101420 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe191525640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c110d00 con 0x7fe18c106a80 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe191525640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c111000 con 0x7fe18c100a80 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe18a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe18c101420 0x7fe18c0fd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe18a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe18c101420 0x7fe18c0fd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44026/0 (socket says 172.21.15.89:44026) 2024-09-15T23:38:19.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe18a7fc640 1 -- 172.21.15.89:0/2788469458 learned_addr learned my addr 172.21.15.89:0/2788469458 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:19.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe18affd640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c100a80 0x7fe18c0fce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe18b7fe640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe18c106a80 0x7fe18c0fd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe18a7fc640 1 -- 172.21.15.89:0/2788469458 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c100a80 msgr2=0x7fe18c0fce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:19.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c100a80 0x7fe18c0fce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe18a7fc640 1 -- 172.21.15.89:0/2788469458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe18c106a80 msgr2=0x7fe18c0fd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:19.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe18c106a80 0x7fe18c0fd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.140+0000 7fe18a7fc640 1 -- 172.21.15.89:0/2788469458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18c1a68c0 con 0x7fe18c101420 2024-09-15T23:38:19.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.141+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe18c101420 0x7fe18c0fd370 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fe18000c890 tx=0x7fe18000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:19.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.141+0000 7fe173fff640 1 -- 172.21.15.89:0/2788469458 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180004220 con 0x7fe18c101420 2024-09-15T23:38:19.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.141+0000 7fe173fff640 1 -- 172.21.15.89:0/2788469458 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1800043c0 con 0x7fe18c101420 2024-09-15T23:38:19.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.141+0000 7fe191525640 1 -- 172.21.15.89:0/2788469458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18c1a6bb0 con 0x7fe18c101420 2024-09-15T23:38:19.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.141+0000 7fe173fff640 1 -- 172.21.15.89:0/2788469458 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1800049e0 con 0x7fe18c101420 2024-09-15T23:38:19.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.141+0000 7fe191525640 1 -- 172.21.15.89:0/2788469458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe18c1a70f0 con 0x7fe18c101420 2024-09-15T23:38:19.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.143+0000 7fe173fff640 1 -- 172.21.15.89:0/2788469458 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe1800079b0 con 0x7fe18c101420 2024-09-15T23:38:19.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe1938fe740 1 Processor -- start 2024-09-15T23:38:19.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe1938fe740 1 -- start start 2024-09-15T23:38:19.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569140f9b40 0x5569140f8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x5569140f8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556914103370 0x556914105760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe1938fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55691405e2f0 con 0x5569140f85d0 2024-09-15T23:38:19.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe1938fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556914005cf0 con 0x5569140f9b40 2024-09-15T23:38:19.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe1938fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55691405a2b0 con 0x556914103370 2024-09-15T23:38:19.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe173fff640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1740a3240 0x7fe1740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe18affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x5569140f8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe18affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x5569140f8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44032/0 (socket says 172.21.15.89:44032) 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe18affd640 1 -- 172.21.15.89:0/2960683197 learned_addr learned my addr 172.21.15.89:0/2960683197 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe18b7fe640 1 --2- 172.21.15.89:0/2960683197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569140f9b40 0x5569140f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe18b7fe640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1740a3240 0x7fe1740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe173fff640 1 -- 172.21.15.89:0/2788469458 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe1800cfd60 con 0x7fe18c101420 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/2960683197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556914103370 0x556914105760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe18affd640 1 -- 172.21.15.89:0/2960683197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556914103370 msgr2=0x556914105760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe18affd640 1 --2- 172.21.15.89:0/2960683197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556914103370 0x556914105760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe18affd640 1 -- 172.21.15.89:0/2960683197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569140f9b40 msgr2=0x5569140f8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.144+0000 7fe18affd640 1 --2- 172.21.15.89:0/2960683197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569140f9b40 0x5569140f8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.145+0000 7fe18affd640 1 -- 172.21.15.89:0/2960683197 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556913fea3a0 con 0x5569140f85d0 2024-09-15T23:38:19.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.145+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/2960683197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556914103370 0x556914105760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.145+0000 7fe18affd640 1 --2- 172.21.15.89:0/2960683197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x5569140f8a10 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009930 tx=0x7fe17c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ecb65d9fc24133 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.145+0000 7fe18b7fe640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1740a3240 0x7fe1740a5700 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe184005c40 tx=0x7fe184005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.145+0000 7fe1717fa640 1 -- 172.21.15.89:0/2960683197 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c040070 con 0x5569140f85d0 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.145+0000 7fe1717fa640 1 -- 172.21.15.89:0/2960683197 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe17c005020 con 0x5569140f85d0 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.145+0000 7fe1717fa640 1 -- 172.21.15.89:0/2960683197 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c035040 con 0x5569140f85d0 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 -- 172.21.15.89:0/2960683197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 msgr2=0x5569140f8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 --2- 172.21.15.89:0/2960683197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x5569140f8a10 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fe17c009930 tx=0x7fe17c031630 comp rx=0 tx=0).stop 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 -- 172.21.15.89:0/2960683197 shutdown_connections 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 --2- 172.21.15.89:0/2960683197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556914103370 0x556914105760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 --2- 172.21.15.89:0/2960683197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x5569140f8a10 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 --2- 172.21.15.89:0/2960683197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569140f9b40 0x5569140f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 -- 172.21.15.89:0/2960683197 >> 172.21.15.89:0/2960683197 conn(0x5569140c1310 msgr2=0x5569140f4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:19.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 -- 172.21.15.89:0/2960683197 shutdown_connections 2024-09-15T23:38:19.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 -- 172.21.15.89:0/2960683197 wait complete. 2024-09-15T23:38:19.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 -- start start 2024-09-15T23:38:19.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 Processor -- start 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914102be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x5569140ffe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x5569141003d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x556914106010 con 0x5569140f85d0 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556914019060 con 0x556914103370 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556914106190 con 0x5569140f9b40 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.146+0000 7fe1938fe740 10 client.0 _ll_register_callbacks cb 0x556913d6c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914102be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914102be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44034/0 (socket says 172.21.15.89:44034) 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18affd640 1 -- 172.21.15.89:0/1020903828 learned_addr learned my addr 172.21.15.89:0/1020903828 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:19.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x5569140ffe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18affd640 1 -- 172.21.15.89:0/1020903828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 msgr2=0x5569140ffe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18affd640 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x5569140ffe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18affd640 1 -- 172.21.15.89:0/1020903828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 msgr2=0x5569141003d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18affd640 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x5569141003d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18affd640 1 -- 172.21.15.89:0/1020903828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556913dcf3c0 con 0x5569140f85d0 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x5569140ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe18affd640 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914102be0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fe17c006010 tx=0x7fe17c031b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e483ed13907bdc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe16eff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe16eff5640 1 -- 172.21.15.89:0/1020903828 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c040070 con 0x5569140f85d0 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe16eff5640 1 -- 172.21.15.89:0/1020903828 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe17c005520 con 0x5569140f85d0 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe1938fe740 1 -- 172.21.15.89:0/1020903828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe17c005020 con 0x5569140f85d0 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe16eff5640 1 -- 172.21.15.89:0/1020903828 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe17c056050 con 0x5569140f85d0 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe1938fe740 10 client.20400 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:19.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.147+0000 7fe1938fe740 1 -- 172.21.15.89:0/1020903828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5569141118b0 con 0x5569140f85d0 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe16eff5640 1 -- 172.21.15.89:0/1020903828 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe17c005020 con 0x5569140f85d0 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe16eff5640 1 -- 172.21.15.89:0/1020903828 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe17c03cc60 con 0x5569140f85d0 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe1938fe740 10 client.20400 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe1938fe740 10 client.20400 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe1938fe740 20 client.20400 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe1938fe740 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556914112c00 0x556914114ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe1938fe740 4 client.20400 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe1938fe740 1 -- 172.21.15.89:0/1020903828 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55691406b860 con 0x556914112c00 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556914112c00 0x556914114ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556914112c00 0x556914114ff0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1b6d7bcd74004fb0 server_cookie=a6ced870108e98ab in_seq=0 out_seq=0 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.148+0000 7fe16eff5640 10 client.20400 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe16eff5640 1 -- 172.21.15.89:0/1020903828 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55691406b860 con 0x556914112c00 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe16eff5640 10 client.20400 handle_command_reply: tid=0 2024-09-15T23:38:19.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 client.20400 shutdown 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 -- 172.21.15.89:0/1020903828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 msgr2=0x556914102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914102be0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fe17c006010 tx=0x7fe17c031b10 comp rx=0 tx=0).stop 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 -- 172.21.15.89:0/1020903828 shutdown_connections 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556914112c00 0x556914114ff0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x5569141003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x5569140ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe16eff5640 0 client.20400 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 --2- 172.21.15.89:0/1020903828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914102be0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 -- 172.21.15.89:0/1020903828 >> 172.21.15.89:0/1020903828 conn(0x5569140c1310 msgr2=0x5569140fa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 -- 172.21.15.89:0/1020903828 shutdown_connections 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.149+0000 7fe1938fe740 1 -- 172.21.15.89:0/1020903828 wait complete. 2024-09-15T23:38:19.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.150+0000 7fe1938fe740 20 client.20400 trim_cache size 0 max 16384 2024-09-15T23:38:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.162+0000 7fe1938fe740 1 Processor -- start 2024-09-15T23:38:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.162+0000 7fe1938fe740 1 -- start start 2024-09-15T23:38:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.162+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.162+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x556914118550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.162+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x5569140fd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.162+0000 7fe1938fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1b1000 con 0x5569140f85d0 2024-09-15T23:38:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.162+0000 7fe1938fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1b2b40 con 0x556914103370 2024-09-15T23:38:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.162+0000 7fe1938fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1b16d0 con 0x5569140f9b40 2024-09-15T23:38:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe18affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe18affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44042/0 (socket says 172.21.15.89:44042) 2024-09-15T23:38:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe18affd640 1 -- 172.21.15.89:0/2904366097 learned_addr learned my addr 172.21.15.89:0/2904366097 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe18affd640 1 -- 172.21.15.89:0/2904366097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 msgr2=0x556914118550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe18affd640 1 --2- 172.21.15.89:0/2904366097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x556914118550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe18affd640 1 -- 172.21.15.89:0/2904366097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 msgr2=0x5569140fd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe18affd640 1 --2- 172.21.15.89:0/2904366097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x5569140fd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe18affd640 1 -- 172.21.15.89:0/2904366097 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569140fb120 con 0x5569140f85d0 2024-09-15T23:38:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe18affd640 1 --2- 172.21.15.89:0/2904366097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118010 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fe17c001eb0 tx=0x7fe17c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57d54edf8b30dbd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe170ff9640 1 -- 172.21.15.89:0/2904366097 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c040070 con 0x5569140f85d0 2024-09-15T23:38:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe170ff9640 1 -- 172.21.15.89:0/2904366097 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe17c0507a0 con 0x5569140f85d0 2024-09-15T23:38:19.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.163+0000 7fe170ff9640 1 -- 172.21.15.89:0/2904366097 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c035030 con 0x5569140f85d0 2024-09-15T23:38:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 -- 172.21.15.89:0/2904366097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 msgr2=0x556914118010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 --2- 172.21.15.89:0/2904366097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118010 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fe17c001eb0 tx=0x7fe17c031630 comp rx=0 tx=0).stop 2024-09-15T23:38:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 -- 172.21.15.89:0/2904366097 shutdown_connections 2024-09-15T23:38:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 --2- 172.21.15.89:0/2904366097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x5569140fd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 --2- 172.21.15.89:0/2904366097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x556914118550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 --2- 172.21.15.89:0/2904366097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118010 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 -- 172.21.15.89:0/2904366097 >> 172.21.15.89:0/2904366097 conn(0x5569140c1310 msgr2=0x5569140fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:19.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 -- 172.21.15.89:0/2904366097 shutdown_connections 2024-09-15T23:38:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 -- 172.21.15.89:0/2904366097 wait complete. 2024-09-15T23:38:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 -- start start 2024-09-15T23:38:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 Processor -- start 2024-09-15T23:38:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x556914115b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x556914116080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1b2470 con 0x5569140f85d0 2024-09-15T23:38:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1b1da0 con 0x556914103370 2024-09-15T23:38:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.164+0000 7fe1938fe740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1aefe0 con 0x5569140f9b40 2024-09-15T23:38:19.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe1938fe740 10 client.0 _ll_register_callbacks cb 0x7fe17c004770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44056/0 (socket says 172.21.15.89:44056) 2024-09-15T23:38:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18affd640 1 -- 172.21.15.89:0/395904908 learned_addr learned my addr 172.21.15.89:0/395904908 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:19.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18affd640 1 -- 172.21.15.89:0/395904908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 msgr2=0x556914115b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x556914115b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18affd640 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x556914115b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18b7fe640 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x556914116080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18affd640 1 -- 172.21.15.89:0/395904908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 msgr2=0x556914116080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18affd640 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x556914116080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18affd640 1 -- 172.21.15.89:0/395904908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569141168b0 con 0x5569140f85d0 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18b7fe640 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x556914116080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe18affd640 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118750 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fe17c050770 tx=0x7fe17c004820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e28b8600e1f0dfc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe16eff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe16eff5640 1 -- 172.21.15.89:0/395904908 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c035070 con 0x5569140f85d0 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe16eff5640 1 -- 172.21.15.89:0/395904908 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe17c004250 con 0x5569140f85d0 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe1938fe740 1 -- 172.21.15.89:0/395904908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe17c0507a0 con 0x5569140f85d0 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.165+0000 7fe16eff5640 1 -- 172.21.15.89:0/395904908 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe17c069050 con 0x5569140f85d0 2024-09-15T23:38:19.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe1938fe740 10 client.20406 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe1938fe740 1 -- 172.21.15.89:0/395904908 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55691417d530 con 0x5569140f85d0 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe16eff5640 1 -- 172.21.15.89:0/395904908 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe17c0507a0 con 0x5569140f85d0 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe16eff5640 1 -- 172.21.15.89:0/395904908 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe17c004f20 con 0x5569140f85d0 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe1938fe740 10 client.20406 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe1938fe740 10 client.20406 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe1938fe740 20 client.20406 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe1938fe740 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556914112c00 0x55691417fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe1938fe740 4 client.20406 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe1938fe740 1 -- 172.21.15.89:0/395904908 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556914100760 con 0x556914112c00 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.166+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556914112c00 0x55691417fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe16eff5640 10 client.20406 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:38:19.174 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe18a7fc640 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556914112c00 0x55691417fb90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=23f7d208361d7681 server_cookie=d7b9e99f7e4d9460 in_seq=0 out_seq=0 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe16eff5640 1 -- 172.21.15.89:0/395904908 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556914100760 con 0x556914112c00 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe16eff5640 10 client.20406 handle_command_reply: tid=0 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 client.20406 shutdown 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 -- 172.21.15.89:0/395904908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 msgr2=0x556914118750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118750 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fe17c050770 tx=0x7fe17c004820 comp rx=0 tx=0).stop 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 -- 172.21.15.89:0/395904908 shutdown_connections 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x556914112c00 0x55691417fb90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556914103370 0x556914116080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe16eff5640 0 client.20406 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5569140f9b40 0x556914115b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 --2- 172.21.15.89:0/395904908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5569140f85d0 0x556914118750 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 -- 172.21.15.89:0/395904908 >> 172.21.15.89:0/395904908 conn(0x5569140c1310 msgr2=0x5569140fa110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:19.175 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 -- 172.21.15.89:0/395904908 shutdown_connections 2024-09-15T23:38:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 1 -- 172.21.15.89:0/395904908 wait complete. 2024-09-15T23:38:19.176 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.167+0000 7fe1938fe740 20 client.20406 trim_cache size 0 max 16384 2024-09-15T23:38:19.176 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 -- 172.21.15.89:0/2788469458 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1740a3240 msgr2=0x7fe1740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1740a3240 0x7fe1740a5700 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe184005c40 tx=0x7fe184005bd0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 -- 172.21.15.89:0/2788469458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe18c101420 msgr2=0x7fe18c0fd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe18c101420 0x7fe18c0fd370 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fe18000c890 tx=0x7fe18000cd50 comp rx=0 tx=0).stop 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 -- 172.21.15.89:0/2788469458 shutdown_connections 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1740a3240 0x7fe1740a5700 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe18c106a80 0x7fe18c0fd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe18c101420 0x7fe18c0fd370 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 --2- 172.21.15.89:0/2788469458 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c100a80 0x7fe18c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 -- 172.21.15.89:0/2788469458 >> 172.21.15.89:0/2788469458 conn(0x7fe18c0fc7a0 msgr2=0x7fe18c06a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:19.177 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 -- 172.21.15.89:0/2788469458 shutdown_connections 2024-09-15T23:38:19.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:19.168+0000 7fe191525640 1 -- 172.21.15.89:0/2788469458 wait complete. 2024-09-15T23:38:19.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:19 smithi089 ceph-mon[30933]: pgmap v778: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:38:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:19 smithi120 ceph-mon[35250]: pgmap v778: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:38:19.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:19 smithi174 ceph-mon[35586]: pgmap v778: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 20 op/s 2024-09-15T23:38:19.958 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:38:20.205 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:20.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.699+0000 7fb4d683a640 1 -- 172.21.15.89:0/2564914003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4d0103760 msgr2=0x7fb4d0107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:20.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.699+0000 7fb4d683a640 1 --2- 172.21.15.89:0/2564914003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4d0103760 0x7fb4d0107ba0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009930 tx=0x7fb4c402a420 comp rx=0 tx=0).stop 2024-09-15T23:38:20.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.700+0000 7fb4d683a640 1 -- 172.21.15.89:0/2564914003 shutdown_connections 2024-09-15T23:38:20.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.700+0000 7fb4d683a640 1 --2- 172.21.15.89:0/2564914003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4d01082e0 0x7fb4d010ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.700+0000 7fb4d683a640 1 --2- 172.21.15.89:0/2564914003 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4d0103760 0x7fb4d0107ba0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.700+0000 7fb4d683a640 1 --2- 172.21.15.89:0/2564914003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d0102db0 0x7fb4d0103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.700+0000 7fb4d683a640 1 -- 172.21.15.89:0/2564914003 >> 172.21.15.89:0/2564914003 conn(0x7fb4d00fc7d0 msgr2=0x7fb4d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.700+0000 7fb4d683a640 1 -- 172.21.15.89:0/2564914003 shutdown_connections 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.700+0000 7fb4d683a640 1 -- 172.21.15.89:0/2564914003 wait complete. 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.700+0000 7fb4d683a640 1 Processor -- start 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.700+0000 7fb4d683a640 1 -- start start 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4d683a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4d0102db0 0x7fb4d01985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4d683a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d0103760 0x7fb4d0198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4d683a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4d01082e0 0x7fb4d019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4d683a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d01120d0 con 0x7fb4d0102db0 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4d683a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d0111f50 con 0x7fb4d0103760 2024-09-15T23:38:20.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4d683a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d0112250 con 0x7fb4d01082e0 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4d0102db0 0x7fb4d01985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4d0102db0 0x7fb4d01985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44076/0 (socket says 172.21.15.89:44076) 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cffff640 1 -- 172.21.15.89:0/2899382465 learned_addr learned my addr 172.21.15.89:0/2899382465 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cf7fe640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d0103760 0x7fb4d0198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cffff640 1 -- 172.21.15.89:0/2899382465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4d01082e0 msgr2=0x7fb4d019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cffff640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4d01082e0 0x7fb4d019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cffff640 1 -- 172.21.15.89:0/2899382465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d0103760 msgr2=0x7fb4d0198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cffff640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d0103760 0x7fb4d0198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cffff640 1 -- 172.21.15.89:0/2899382465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d019d5a0 con 0x7fb4d0102db0 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cf7fe640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d0103760 0x7fb4d0198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cffff640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4d0102db0 0x7fb4d01985f0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000b4a0 tx=0x7fb4c000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cd7fa640 1 -- 172.21.15.89:0/2899382465 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0014070 con 0x7fb4d0102db0 2024-09-15T23:38:20.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cd7fa640 1 -- 172.21.15.89:0/2899382465 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c0010af0 con 0x7fb4d0102db0 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4d683a640 1 -- 172.21.15.89:0/2899382465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d019d890 con 0x7fb4d0102db0 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4cd7fa640 1 -- 172.21.15.89:0/2899382465 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0017d10 con 0x7fb4d0102db0 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.701+0000 7fb4d683a640 1 -- 172.21.15.89:0/2899382465 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d01a5170 con 0x7fb4d0102db0 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.704+0000 7fb4cd7fa640 1 -- 172.21.15.89:0/2899382465 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fb4c0017070 con 0x7fb4d0102db0 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.704+0000 7fb4d8c13740 1 Processor -- start 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.704+0000 7fb4d8c13740 1 -- start start 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.704+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370b4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b45d0 0x55f6370b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370bf370 0x55f6370c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f63701a2f0 con 0x55f6370b45d0 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f636fc1cf0 con 0x55f6370bf370 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f6370162b0 con 0x55f6370b5b40 2024-09-15T23:38:20.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b45d0 0x55f6370b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b45d0 0x55f6370b4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44090/0 (socket says 172.21.15.89:44090) 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cffff640 1 -- 172.21.15.89:0/4106078547 learned_addr learned my addr 172.21.15.89:0/4106078547 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4d4dd7640 1 --2- 172.21.15.89:0/4106078547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cd7fa640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 0x7fb4b80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cf7fe640 1 --2- 172.21.15.89:0/4106078547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370bf370 0x55f6370c1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4d4dd7640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 0x7fb4b80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cd7fa640 1 -- 172.21.15.89:0/2899382465 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb4c00cf180 con 0x7fb4d0102db0 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cffff640 1 -- 172.21.15.89:0/4106078547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 msgr2=0x55f6370b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cffff640 1 --2- 172.21.15.89:0/4106078547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370b4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cffff640 1 -- 172.21.15.89:0/4106078547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370bf370 msgr2=0x55f6370c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cffff640 1 --2- 172.21.15.89:0/4106078547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370bf370 0x55f6370c1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cffff640 1 -- 172.21.15.89:0/4106078547 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f636fa63a0 con 0x55f6370b45d0 2024-09-15T23:38:20.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cf7fe640 1 --2- 172.21.15.89:0/4106078547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370bf370 0x55f6370c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4cffff640 1 --2- 172.21.15.89:0/4106078547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b45d0 0x55f6370b4a10 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fb4c009b340 tx=0x7fb4c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd995b9b8838813e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4b67fc640 1 -- 172.21.15.89:0/4106078547 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c001e070 con 0x55f6370b45d0 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.705+0000 7fb4d4dd7640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 0x7fb4b80a5700 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8005fd0 tx=0x7fb4c8005e90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4b67fc640 1 -- 172.21.15.89:0/4106078547 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c009bab0 con 0x55f6370b45d0 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4b67fc640 1 -- 172.21.15.89:0/4106078547 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0010ca0 con 0x55f6370b45d0 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4106078547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b45d0 msgr2=0x55f6370b4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/4106078547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b45d0 0x55f6370b4a10 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fb4c009b340 tx=0x7fb4c00046c0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4106078547 shutdown_connections 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/4106078547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370bf370 0x55f6370c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/4106078547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b45d0 0x55f6370b4a10 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/4106078547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4106078547 >> 172.21.15.89:0/4106078547 conn(0x55f63707d310 msgr2=0x55f6370b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:20.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4106078547 shutdown_connections 2024-09-15T23:38:20.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.706+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4106078547 wait complete. 2024-09-15T23:38:20.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4d8c13740 1 -- start start 2024-09-15T23:38:20.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4d8c13740 1 Processor -- start 2024-09-15T23:38:20.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370bedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b5b40 0x55f6370bc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370bf370 0x55f6370bc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f6370c2010 con 0x55f6370b5b40 2024-09-15T23:38:20.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f636fd5060 con 0x55f6370b45d0 2024-09-15T23:38:20.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f6370c2190 con 0x55f6370bf370 2024-09-15T23:38:20.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4d8c13740 10 client.0 _ll_register_callbacks cb 0x55f636d288c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:20.744 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:38:20.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4cf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b5b40 0x55f6370bc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4cf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b5b40 0x55f6370bc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44106/0 (socket says 172.21.15.89:44106) 2024-09-15T23:38:20.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4cf7fe640 1 -- 172.21.15.89:0/4193688199 learned_addr learned my addr 172.21.15.89:0/4193688199 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:20.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4cffff640 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370bedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4d4dd7640 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370bf370 0x55f6370bc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4cf7fe640 1 -- 172.21.15.89:0/4193688199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370bf370 msgr2=0x55f6370bc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4cf7fe640 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370bf370 0x55f6370bc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4cf7fe640 1 -- 172.21.15.89:0/4193688199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 msgr2=0x55f6370bedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4cf7fe640 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370bedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.707+0000 7fb4cf7fe640 1 -- 172.21.15.89:0/4193688199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f636d8b3c0 con 0x55f6370b5b40 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4cf7fe640 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b5b40 0x55f6370bc080 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00697d0 tx=0x7fb4c40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae2d928fab37024 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4af7fe640 1 -- 172.21.15.89:0/4193688199 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4035030 con 0x55f6370b5b40 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4af7fe640 1 -- 172.21.15.89:0/4193688199 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb4c4003b40 con 0x55f6370b5b40 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4193688199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4c009bab0 con 0x55f6370b5b40 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4af7fe640 1 -- 172.21.15.89:0/4193688199 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb4c404d050 con 0x55f6370b5b40 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4d8c13740 10 client.20418 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4193688199 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f6370cdb30 con 0x55f6370b5b40 2024-09-15T23:38:20.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4af7fe640 1 -- 172.21.15.89:0/4193688199 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c009bab0 con 0x55f6370b5b40 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4af7fe640 1 -- 172.21.15.89:0/4193688199 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb4c403c540 con 0x55f6370b5b40 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4d8c13740 10 client.20418 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4d8c13740 10 client.20418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4d8c13740 20 client.20418 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f6370ceff0 0x55f6370d1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4d8c13740 4 client.20418 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.708+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4193688199 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f637027860 con 0x55f6370ceff0 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.709+0000 7fb4cffff640 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f6370ceff0 0x55f6370d1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.709+0000 7fb4af7fe640 10 client.20418 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.709+0000 7fb4cffff640 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f6370ceff0 0x55f6370d1450 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77bb02414e37ee7e server_cookie=7da3f411b74b8c53 in_seq=0 out_seq=0 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4af7fe640 1 -- 172.21.15.89:0/4193688199 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f637027860 con 0x55f6370ceff0 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4af7fe640 10 client.20418 handle_command_reply: tid=0 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 client.20418 shutdown 2024-09-15T23:38:20.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4193688199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b5b40 msgr2=0x55f6370bc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b5b40 0x55f6370bc080 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00697d0 tx=0x7fb4c40044c0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4193688199 shutdown_connections 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f6370ceff0 0x55f6370d1450 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370bf370 0x55f6370bc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370b5b40 0x55f6370bc080 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4af7fe640 0 client.20418 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/4193688199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370bedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4193688199 >> 172.21.15.89:0/4193688199 conn(0x55f63707d310 msgr2=0x55f6370b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4193688199 shutdown_connections 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 1 -- 172.21.15.89:0/4193688199 wait complete. 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.710+0000 7fb4d8c13740 20 client.20418 trim_cache size 0 max 16384 2024-09-15T23:38:20.747 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d8c13740 1 Processor -- start 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d8c13740 1 -- start start 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370c7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d01aeea0 con 0x55f6370bf370 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d01b09e0 con 0x55f6370b45d0 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d01af570 con 0x55f6370b5b40 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d4dd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d4dd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44110/0 (socket says 172.21.15.89:44110) 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.723+0000 7fb4d4dd7640 1 -- 172.21.15.89:0/2603671695 learned_addr learned my addr 172.21.15.89:0/2603671695 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4cffff640 1 --2- 172.21.15.89:0/2603671695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4cf7fe640 1 --2- 172.21.15.89:0/2603671695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4d4dd7640 1 -- 172.21.15.89:0/2603671695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 msgr2=0x55f6370c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:20.748 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4d4dd7640 1 --2- 172.21.15.89:0/2603671695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370c7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4d4dd7640 1 -- 172.21.15.89:0/2603671695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 msgr2=0x55f6370c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4d4dd7640 1 --2- 172.21.15.89:0/2603671695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370c7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4d4dd7640 1 -- 172.21.15.89:0/2603671695 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6370bcb80 con 0x55f6370bf370 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4d4dd7640 1 --2- 172.21.15.89:0/2603671695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b7120 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8013880 tx=0x7fb4c8007530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96112fed2c80c8a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4b5ffb640 1 -- 172.21.15.89:0/2603671695 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8004040 con 0x55f6370bf370 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4b5ffb640 1 -- 172.21.15.89:0/2603671695 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c8025070 con 0x55f6370bf370 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.724+0000 7fb4b5ffb640 1 -- 172.21.15.89:0/2603671695 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8020a90 con 0x55f6370bf370 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 -- 172.21.15.89:0/2603671695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 msgr2=0x55f6370b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/2603671695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b7120 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8013880 tx=0x7fb4c8007530 comp rx=0 tx=0).stop 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 -- 172.21.15.89:0/2603671695 shutdown_connections 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/2603671695 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b7120 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/2603671695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.749 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/2603671695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 -- 172.21.15.89:0/2603671695 >> 172.21.15.89:0/2603671695 conn(0x55f63707d310 msgr2=0x55f6370bb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 -- 172.21.15.89:0/2603671695 shutdown_connections 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 -- 172.21.15.89:0/2603671695 wait complete. 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 -- start start 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.725+0000 7fb4d8c13740 1 Processor -- start 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370b8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d8c13740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d01b0310 con 0x55f6370bf370 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d01afc40 con 0x55f6370b45d0 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d8c13740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d01ace80 con 0x55f6370b5b40 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d8c13740 10 client.0 _ll_register_callbacks cb 0x7fb4c4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d4dd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d4dd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44116/0 (socket says 172.21.15.89:44116) 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d4dd7640 1 -- 172.21.15.89:0/1506086187 learned_addr learned my addr 172.21.15.89:0/1506086187 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4cf7fe640 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4cffff640 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d4dd7640 1 -- 172.21.15.89:0/1506086187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 msgr2=0x55f6370b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d4dd7640 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370b8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d4dd7640 1 -- 172.21.15.89:0/1506086187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 msgr2=0x55f6370b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d4dd7640 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d4dd7640 1 -- 172.21.15.89:0/1506086187 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6370b97d0 con 0x55f6370bf370 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4cf7fe640 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4cffff640 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d4dd7640 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b8fa0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8020770 tx=0x7fb4c8020a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e321cc99f38737d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:20.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4af7fe640 1 -- 172.21.15.89:0/1506086187 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c802f030 con 0x55f6370bf370 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4af7fe640 1 -- 172.21.15.89:0/1506086187 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fb4c801b040 con 0x55f6370bf370 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.726+0000 7fb4d8c13740 1 -- 172.21.15.89:0/1506086187 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4c8025070 con 0x55f6370bf370 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4af7fe640 1 -- 172.21.15.89:0/1506086187 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb4c8007ea0 con 0x55f6370bf370 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4d8c13740 10 client.20424 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4d8c13740 1 -- 172.21.15.89:0/1506086187 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f63713e430 con 0x55f6370bf370 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4af7fe640 1 -- 172.21.15.89:0/1506086187 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c8025070 con 0x55f6370bf370 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4af7fe640 1 -- 172.21.15.89:0/1506086187 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb4c800c050 con 0x55f6370bf370 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4d8c13740 10 client.20424 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4d8c13740 10 client.20424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4d8c13740 20 client.20424 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f6370ceff0 0x55f637140af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4d8c13740 4 client.20424 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:38:20.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4d8c13740 1 -- 172.21.15.89:0/1506086187 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f637021d70 con 0x55f6370ceff0 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.727+0000 7fb4cffff640 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f6370ceff0 0x55f637140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4cffff640 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f6370ceff0 0x55f637140af0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c1eac6253149a78 server_cookie=718e6c7db4b15180 in_seq=0 out_seq=0 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4af7fe640 10 client.20424 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4af7fe640 1 -- 172.21.15.89:0/1506086187 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f637021d70 con 0x55f6370ceff0 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4af7fe640 10 client.20424 handle_command_reply: tid=0 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 client.20424 shutdown 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 -- 172.21.15.89:0/1506086187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 msgr2=0x55f6370b8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b8fa0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8020770 tx=0x7fb4c8020a40 comp rx=0 tx=0).stop 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 -- 172.21.15.89:0/1506086187 shutdown_connections 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55f6370ceff0 0x55f637140af0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4af7fe640 0 client.20424 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6370bf370 0x55f6370b8fa0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6370b5b40 0x55f6370b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 --2- 172.21.15.89:0/1506086187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6370b45d0 0x55f6370b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 -- 172.21.15.89:0/1506086187 >> 172.21.15.89:0/1506086187 conn(0x55f63707d310 msgr2=0x55f6370bd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 -- 172.21.15.89:0/1506086187 shutdown_connections 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 1 -- 172.21.15.89:0/1506086187 wait complete. 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.728+0000 7fb4d8c13740 20 client.20424 trim_cache size 0 max 16384 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 -- 172.21.15.89:0/2899382465 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 msgr2=0x7fb4b80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 0x7fb4b80a5700 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8005fd0 tx=0x7fb4c8005e90 comp rx=0 tx=0).stop 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 -- 172.21.15.89:0/2899382465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4d0102db0 msgr2=0x7fb4d01985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4d0102db0 0x7fb4d01985f0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000b4a0 tx=0x7fb4c000b960 comp rx=0 tx=0).stop 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 -- 172.21.15.89:0/2899382465 shutdown_connections 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fb4b80a3240 0x7fb4b80a5700 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4d01082e0 0x7fb4d019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb4d0103760 0x7fb4d0198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 --2- 172.21.15.89:0/2899382465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4d0102db0 0x7fb4d01985f0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 -- 172.21.15.89:0/2899382465 >> 172.21.15.89:0/2899382465 conn(0x7fb4d00fc7d0 msgr2=0x7fb4d0106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 -- 172.21.15.89:0/2899382465 shutdown_connections 2024-09-15T23:38:20.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:20.729+0000 7fb4d683a640 1 -- 172.21.15.89:0/2899382465 wait complete. 2024-09-15T23:38:21.287 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:21 smithi174 ceph-mon[35586]: pgmap v779: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:38:21.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:21 smithi089 ceph-mon[30933]: pgmap v779: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:38:21.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:21 smithi120 ceph-mon[35250]: pgmap v779: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 15 op/s 2024-09-15T23:38:22.557 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:38:22.557 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:38:22.818 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:23.247 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:23 smithi089 ceph-mon[30933]: pgmap v780: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.329+0000 7f3a4c29f640 1 -- 172.21.15.89:0/2426445217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a44106f10 msgr2=0x7f3a4410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.329+0000 7f3a4c29f640 1 --2- 172.21.15.89:0/2426445217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a44106f10 0x7f3a4410ea90 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009ca0 tx=0x7f3a3c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.329+0000 7f3a4c29f640 1 -- 172.21.15.89:0/2426445217 shutdown_connections 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.329+0000 7f3a4c29f640 1 --2- 172.21.15.89:0/2426445217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a44106f10 0x7f3a4410ea90 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.329+0000 7f3a4c29f640 1 --2- 172.21.15.89:0/2426445217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a44102790 0x7f3a441067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.329+0000 7f3a4c29f640 1 --2- 172.21.15.89:0/2426445217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a44101de0 0x7f3a441021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.329+0000 7f3a4c29f640 1 -- 172.21.15.89:0/2426445217 >> 172.21.15.89:0/2426445217 conn(0x7f3a440fd8c0 msgr2=0x7f3a440ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.329+0000 7f3a4c29f640 1 -- 172.21.15.89:0/2426445217 shutdown_connections 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.329+0000 7f3a4c29f640 1 -- 172.21.15.89:0/2426445217 wait complete. 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4c29f640 1 Processor -- start 2024-09-15T23:38:23.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4c29f640 1 -- start start 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4c29f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a44101de0 0x7f3a441985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4c29f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a44102790 0x7f3a44198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4c29f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a44106f10 0x7f3a4419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4c29f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3a44112010 con 0x7f3a44106f10 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4c29f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3a44111e90 con 0x7f3a44101de0 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4c29f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a44112190 con 0x7f3a44102790 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4a83c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a44106f10 0x7f3a4419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4a83c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a44106f10 0x7f3a4419cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44142/0 (socket says 172.21.15.89:44142) 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.330+0000 7f3a4a83c640 1 -- 172.21.15.89:0/1434999624 learned_addr learned my addr 172.21.15.89:0/1434999624 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a4a03b640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a44101de0 0x7f3a441985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a4983a640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a44102790 0x7f3a44198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a4a83c640 1 -- 172.21.15.89:0/1434999624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a44102790 msgr2=0x7f3a44198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:23.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a44102790 0x7f3a44198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a4a83c640 1 -- 172.21.15.89:0/1434999624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a44101de0 msgr2=0x7f3a441985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a44101de0 0x7f3a441985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a4a83c640 1 -- 172.21.15.89:0/1434999624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4419d5a0 con 0x7f3a44106f10 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a44106f10 0x7f3a4419cec0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c031a60 tx=0x7f3a3c031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a3b7fe640 1 -- 172.21.15.89:0/1434999624 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c003c30 con 0x7f3a44106f10 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a3b7fe640 1 -- 172.21.15.89:0/1434999624 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c003dd0 con 0x7f3a44106f10 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a4c29f640 1 -- 172.21.15.89:0/1434999624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4419d830 con 0x7f3a44106f10 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a3b7fe640 1 -- 172.21.15.89:0/1434999624 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0047e0 con 0x7f3a44106f10 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.331+0000 7f3a4c29f640 1 -- 172.21.15.89:0/1434999624 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a441a5170 con 0x7f3a44106f10 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.333+0000 7f3a3b7fe640 1 -- 172.21.15.89:0/1434999624 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3a3c02f790 con 0x7f3a44106f10 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4e678740 1 Processor -- start 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4e678740 1 -- start start 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a4df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a4e480 0x563dc7a58800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a3b7fe640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a300a32c0 0x7f3a300a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a5b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4e678740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563dc79b3280 con 0x563dc7a58d40 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4e678740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563dc795ab10 con 0x563dc7a4db60 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4e678740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563dc79af240 con 0x563dc7a4e480 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4a83c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a5b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4983a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a4df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4a83c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a5b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44144/0 (socket says 172.21.15.89:44144) 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4a83c640 1 -- 172.21.15.89:0/2975103506 learned_addr learned my addr 172.21.15.89:0/2975103506 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4983a640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a300a32c0 0x7f3a300a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.334+0000 7f3a4a03b640 1 --2- 172.21.15.89:0/2975103506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a4e480 0x563dc7a58800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a3b7fe640 1 -- 172.21.15.89:0/1434999624 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3a3c0f1160 con 0x7f3a44106f10 2024-09-15T23:38:23.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a4a83c640 1 -- 172.21.15.89:0/2975103506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a4e480 msgr2=0x563dc7a58800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/2975103506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a4e480 0x563dc7a58800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a4a83c640 1 -- 172.21.15.89:0/2975103506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 msgr2=0x563dc7a4df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/2975103506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a4df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a4a83c640 1 -- 172.21.15.89:0/2975103506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dc79671f0 con 0x563dc7a58d40 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/2975103506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a5b130 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c031b50 tx=0x7f3a3c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db7cf4ee6381c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a4983a640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a300a32c0 0x7f3a300a5780 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3a34006fd0 tx=0x7f3a34006da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a38dea640 1 -- 172.21.15.89:0/2975103506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0ebe60 con 0x563dc7a58d40 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a38dea640 1 -- 172.21.15.89:0/2975103506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c036030 con 0x563dc7a58d40 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.335+0000 7f3a38dea640 1 -- 172.21.15.89:0/2975103506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0c2670 con 0x563dc7a58d40 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 -- 172.21.15.89:0/2975103506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 msgr2=0x563dc7a5b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2975103506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a5b130 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c031b50 tx=0x7f3a3c0095c0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 -- 172.21.15.89:0/2975103506 shutdown_connections 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2975103506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a5b130 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2975103506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a4e480 0x563dc7a58800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2975103506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a4df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 -- 172.21.15.89:0/2975103506 >> 172.21.15.89:0/2975103506 conn(0x563dc7a4a0f0 msgr2=0x563dc7a4a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 -- 172.21.15.89:0/2975103506 shutdown_connections 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 -- 172.21.15.89:0/2975103506 wait complete. 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 -- start start 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.336+0000 7f3a4e678740 1 Processor -- start 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a55c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4e480 0x563dc7a56190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a58d40 0x563dc7a566d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4e678740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x563dc79b3280 con 0x563dc7a4e480 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4e678740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563dc796df10 con 0x563dc7a4db60 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4e678740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563dc7a5b6d0 con 0x563dc7a58d40 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4e678740 10 client.0 _ll_register_callbacks cb 0x563dc7936a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:23.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4983a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4e480 0x563dc7a56190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4983a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4e480 0x563dc7a56190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44154/0 (socket says 172.21.15.89:44154) 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4983a640 1 -- 172.21.15.89:0/2464106721 learned_addr learned my addr 172.21.15.89:0/2464106721 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4a03b640 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a55c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a58d40 0x563dc7a566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4983a640 1 -- 172.21.15.89:0/2464106721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a58d40 msgr2=0x563dc7a566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4983a640 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a58d40 0x563dc7a566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4983a640 1 -- 172.21.15.89:0/2464106721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 msgr2=0x563dc7a55c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4983a640 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a55c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4983a640 1 -- 172.21.15.89:0/2464106721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dc77246a0 con 0x563dc7a4e480 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4983a640 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4e480 0x563dc7a56190 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f3a34006a30 tx=0x7f3a340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11e8766b220502e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a2dffb640 1 -- 172.21.15.89:0/2464106721 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a34027060 con 0x563dc7a4e480 2024-09-15T23:38:23.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a2dffb640 1 -- 172.21.15.89:0/2464106721 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3a3400f030 con 0x563dc7a4e480 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.337+0000 7f3a4e678740 1 -- 172.21.15.89:0/2464106721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a3c036030 con 0x563dc7a4e480 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a2dffb640 1 -- 172.21.15.89:0/2464106721 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3a3402e050 con 0x563dc7a4e480 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a4e678740 10 client.20436 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a4e678740 1 -- 172.21.15.89:0/2464106721 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563dc7a670f0 con 0x563dc7a4e480 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a2dffb640 1 -- 172.21.15.89:0/2464106721 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c036030 con 0x563dc7a4e480 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a2dffb640 1 -- 172.21.15.89:0/2464106721 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3a34015040 con 0x563dc7a4e480 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a4e678740 10 client.20436 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a4e678740 10 client.20436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a4e678740 20 client.20436 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563dc7a685b0 0x563dc7a6a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a4e678740 4 client.20436 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a4e678740 1 -- 172.21.15.89:0/2464106721 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dc79c07f0 con 0x563dc7a685b0 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.338+0000 7f3a4a03b640 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563dc7a685b0 0x563dc7a6a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.339+0000 7f3a2dffb640 10 client.20436 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.339+0000 7f3a4a03b640 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563dc7a685b0 0x563dc7a6a9c0 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffbf1bb0386d2afd server_cookie=d7edbcd1125af564 in_seq=0 out_seq=0 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.339+0000 7f3a2dffb640 1 -- 172.21.15.89:0/2464106721 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563dc79c07f0 con 0x563dc7a685b0 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.339+0000 7f3a2dffb640 10 client.20436 handle_command_reply: tid=0 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.339+0000 7f3a4e678740 1 client.20436 shutdown 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 1 -- 172.21.15.89:0/2464106721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4e480 msgr2=0x563dc7a56190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4e480 0x563dc7a56190 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f3a34006a30 tx=0x7f3a340046c0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 1 -- 172.21.15.89:0/2464106721 shutdown_connections 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563dc7a685b0 0x563dc7a6a9c0 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a58d40 0x563dc7a566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4e480 0x563dc7a56190 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a2dffb640 0 client.20436 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2464106721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a55c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 1 -- 172.21.15.89:0/2464106721 >> 172.21.15.89:0/2464106721 conn(0x563dc7a4a0f0 msgr2=0x563dc7a4fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:23.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 1 -- 172.21.15.89:0/2464106721 shutdown_connections 2024-09-15T23:38:23.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 1 -- 172.21.15.89:0/2464106721 wait complete. 2024-09-15T23:38:23.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.340+0000 7f3a4e678740 20 client.20436 trim_cache size 0 max 16384 2024-09-15T23:38:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.357+0000 7f3a4e678740 1 Processor -- start 2024-09-15T23:38:23.359 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.357+0000 7f3a4e678740 1 -- start start 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a702d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a4e480 0x563dc7a70810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a52eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4e678740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3a441aef90 con 0x563dc7a58d40 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4e678740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3a441b0ad0 con 0x563dc7a4db60 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4e678740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a441af660 con 0x563dc7a4e480 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a83c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a52eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a83c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a52eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44170/0 (socket says 172.21.15.89:44170) 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a83c640 1 -- 172.21.15.89:0/2766170751 learned_addr learned my addr 172.21.15.89:0/2766170751 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4983a640 1 --2- 172.21.15.89:0/2766170751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a4e480 0x563dc7a70810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a03b640 1 --2- 172.21.15.89:0/2766170751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a702d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a83c640 1 -- 172.21.15.89:0/2766170751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a4e480 msgr2=0x563dc7a70810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:23.360 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/2766170751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a4e480 0x563dc7a70810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a83c640 1 -- 172.21.15.89:0/2766170751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 msgr2=0x563dc7a702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/2766170751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a702d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a83c640 1 -- 172.21.15.89:0/2766170751 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dc7a6b040 con 0x563dc7a58d40 2024-09-15T23:38:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a03b640 1 --2- 172.21.15.89:0/2766170751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a702d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.358+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/2766170751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a52eb0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0bd090 tx=0x7f3a3c0b9630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb907b29a6cce5ba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a2ffff640 1 -- 172.21.15.89:0/2766170751 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0bd390 con 0x563dc7a58d40 2024-09-15T23:38:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a2ffff640 1 -- 172.21.15.89:0/2766170751 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c036070 con 0x563dc7a58d40 2024-09-15T23:38:23.361 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a2ffff640 1 -- 172.21.15.89:0/2766170751 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0bb690 con 0x563dc7a58d40 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a4e678740 1 -- 172.21.15.89:0/2766170751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 msgr2=0x563dc7a52eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2766170751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a52eb0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0bd090 tx=0x7f3a3c0b9630 comp rx=0 tx=0).stop 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a4e678740 1 -- 172.21.15.89:0/2766170751 shutdown_connections 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2766170751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a58d40 0x563dc7a52eb0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2766170751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a4e480 0x563dc7a70810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a4e678740 1 --2- 172.21.15.89:0/2766170751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4db60 0x563dc7a702d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a4e678740 1 -- 172.21.15.89:0/2766170751 >> 172.21.15.89:0/2766170751 conn(0x563dc7a4a0f0 msgr2=0x563dc7a57bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a4e678740 1 -- 172.21.15.89:0/2766170751 shutdown_connections 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.359+0000 7f3a4e678740 1 -- 172.21.15.89:0/2766170751 wait complete. 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4e678740 1 -- start start 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4e678740 1 Processor -- start 2024-09-15T23:38:23.365 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4db60 0x563dc7a5cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4e480 0x563dc7a5d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4e678740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a58d40 0x563dc7acc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4e678740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3a441b0400 con 0x563dc7a4db60 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4e678740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3a441afd30 con 0x563dc7a4e480 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4e678740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a441acf70 con 0x563dc7a58d40 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4e678740 10 client.0 _ll_register_callbacks cb 0x7f3a3400a770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4a03b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4db60 0x563dc7a5cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4a03b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4db60 0x563dc7a5cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44172/0 (socket says 172.21.15.89:44172) 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4a03b640 1 -- 172.21.15.89:0/3148526446 learned_addr learned my addr 172.21.15.89:0/3148526446 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.360+0000 7f3a4983a640 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4e480 0x563dc7a5d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a4a03b640 1 -- 172.21.15.89:0/3148526446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a58d40 msgr2=0x563dc7acc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a4a83c640 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a58d40 0x563dc7acc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.366 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a4a03b640 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a58d40 0x563dc7acc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a4a03b640 1 -- 172.21.15.89:0/3148526446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4e480 msgr2=0x563dc7a5d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a4a03b640 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4e480 0x563dc7a5d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a4a03b640 1 -- 172.21.15.89:0/3148526446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dc7a4b260 con 0x563dc7a4db60 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a4a03b640 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4db60 0x563dc7a5cea0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000bb00 tx=0x7f3a40017810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28d8ef10d136901 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a2dffb640 1 -- 172.21.15.89:0/3148526446 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40015420 con 0x563dc7a4db60 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a2dffb640 1 -- 172.21.15.89:0/3148526446 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3a400155c0 con 0x563dc7a4db60 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a4e678740 1 -- 172.21.15.89:0/3148526446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a3c036070 con 0x563dc7a4db60 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a2dffb640 1 -- 172.21.15.89:0/3148526446 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3a4002c050 con 0x563dc7a4db60 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a4e678740 10 client.20442 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a4e678740 1 -- 172.21.15.89:0/3148526446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563dc7ad3a50 con 0x563dc7a4db60 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.361+0000 7f3a2dffb640 1 -- 172.21.15.89:0/3148526446 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a3c036070 con 0x563dc7a4db60 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.362+0000 7f3a2dffb640 1 -- 172.21.15.89:0/3148526446 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3a400248b0 con 0x563dc7a4db60 2024-09-15T23:38:23.367 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.362+0000 7f3a4e678740 10 client.20442 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.362+0000 7f3a4e678740 10 client.20442 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.362+0000 7f3a4e678740 20 client.20442 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.362+0000 7f3a4e678740 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563dc7ad41a0 0x563dc7ad65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.362+0000 7f3a4e678740 4 client.20442 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.362+0000 7f3a4e678740 1 -- 172.21.15.89:0/3148526446 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563dc79bad00 con 0x563dc7ad41a0 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.362+0000 7f3a4983a640 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563dc7ad41a0 0x563dc7ad65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.362+0000 7f3a4983a640 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563dc7ad41a0 0x563dc7ad65c0 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2784a002ca38340e server_cookie=cdac8da5405fc427 in_seq=0 out_seq=0 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.362+0000 7f3a2dffb640 10 client.20442 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.363+0000 7f3a2dffb640 1 -- 172.21.15.89:0/3148526446 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563dc79bad00 con 0x563dc7ad41a0 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.363+0000 7f3a2dffb640 10 client.20442 handle_command_reply: tid=0 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.363+0000 7f3a4e678740 1 client.20442 shutdown 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.363+0000 7f3a4e678740 1 -- 172.21.15.89:0/3148526446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4db60 msgr2=0x563dc7a5cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.363+0000 7f3a4e678740 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4db60 0x563dc7a5cea0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000bb00 tx=0x7f3a40017810 comp rx=0 tx=0).stop 2024-09-15T23:38:23.368 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.363+0000 7f3a4e678740 1 -- 172.21.15.89:0/3148526446 shutdown_connections 2024-09-15T23:38:23.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.363+0000 7f3a4e678740 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x563dc7ad41a0 0x563dc7ad65c0 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.363+0000 7f3a4e678740 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563dc7a58d40 0x563dc7acc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.363+0000 7f3a4e678740 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563dc7a4e480 0x563dc7a5d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a2dffb640 0 client.20442 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:23.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4e678740 1 --2- 172.21.15.89:0/3148526446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563dc7a4db60 0x563dc7a5cea0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4e678740 1 -- 172.21.15.89:0/3148526446 >> 172.21.15.89:0/3148526446 conn(0x563dc7a4a0f0 msgr2=0x563dc7a48a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:23.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4e678740 1 -- 172.21.15.89:0/3148526446 shutdown_connections 2024-09-15T23:38:23.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4e678740 1 -- 172.21.15.89:0/3148526446 wait complete. 2024-09-15T23:38:23.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4e678740 20 client.20442 trim_cache size 0 max 16384 2024-09-15T23:38:23.369 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:38:23.370 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:38:23.370 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "6a8f7253-0eaa-442f-abd9-13d7ab664e6f", 2024-09-15T23:38:23.370 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:38:23.370 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:38:23.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 -- 172.21.15.89:0/1434999624 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a300a32c0 msgr2=0x7f3a300a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:23.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a300a32c0 0x7f3a300a5780 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3a34006fd0 tx=0x7f3a34006da0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 -- 172.21.15.89:0/1434999624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a44106f10 msgr2=0x7f3a4419cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:23.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a44106f10 0x7f3a4419cec0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c031a60 tx=0x7f3a3c031a90 comp rx=0 tx=0).stop 2024-09-15T23:38:23.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 -- 172.21.15.89:0/1434999624 shutdown_connections 2024-09-15T23:38:23.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a300a32c0 0x7f3a300a5780 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a44106f10 0x7f3a4419cec0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a44102790 0x7f3a44198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 --2- 172.21.15.89:0/1434999624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a44101de0 0x7f3a441985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:23.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 -- 172.21.15.89:0/1434999624 >> 172.21.15.89:0/1434999624 conn(0x7f3a440fd8c0 msgr2=0x7f3a440fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:23.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.364+0000 7f3a4c29f640 1 -- 172.21.15.89:0/1434999624 shutdown_connections 2024-09-15T23:38:23.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:23.365+0000 7f3a4c29f640 1 -- 172.21.15.89:0/1434999624 wait complete. 2024-09-15T23:38:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:23 smithi120 ceph-mon[35250]: pgmap v780: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:38:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:23 smithi174 ceph-mon[35586]: pgmap v780: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:38:24.080 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:38:24.325 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:24.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.803+0000 7f19196e4640 1 -- 172.21.15.89:0/410916883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1914102570 msgr2=0x7f19141029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:24.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.803+0000 7f19196e4640 1 --2- 172.21.15.89:0/410916883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1914102570 0x7f19141029b0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f1908009900 tx=0x7f190802a480 comp rx=0 tx=0).stop 2024-09-15T23:38:24.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.803+0000 7f19196e4640 1 -- 172.21.15.89:0/410916883 shutdown_connections 2024-09-15T23:38:24.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.803+0000 7f19196e4640 1 --2- 172.21.15.89:0/410916883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1914102ef0 0x7f191410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.803+0000 7f19196e4640 1 --2- 172.21.15.89:0/410916883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1914102570 0x7f19141029b0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.803+0000 7f19196e4640 1 --2- 172.21.15.89:0/410916883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1914100e20 0x7f1914101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.803+0000 7f19196e4640 1 -- 172.21.15.89:0/410916883 >> 172.21.15.89:0/410916883 conn(0x7f19140fc7d0 msgr2=0x7f19140febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.803+0000 7f19196e4640 1 -- 172.21.15.89:0/410916883 shutdown_connections 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.803+0000 7f19196e4640 1 -- 172.21.15.89:0/410916883 wait complete. 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.804+0000 7f19196e4640 1 Processor -- start 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.804+0000 7f19196e4640 1 -- start start 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.804+0000 7f19196e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1914100e20 0x7f191410e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.804+0000 7f19196e4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1914102570 0x7f191410ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.804+0000 7f19196e4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1914102ef0 0x7f191410ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.804+0000 7f19196e4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f19141120a0 con 0x7f1914102ef0 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.804+0000 7f19196e4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1914111f20 con 0x7f1914102570 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.804+0000 7f19196e4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1914112220 con 0x7f1914100e20 2024-09-15T23:38:24.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19137fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1914102ef0 0x7f191410ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19137fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1914102ef0 0x7f191410ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44192/0 (socket says 172.21.15.89:44192) 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19137fe640 1 -- 172.21.15.89:0/517461614 learned_addr learned my addr 172.21.15.89:0/517461614 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f1912ffd640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1914100e20 0x7f191410e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19137fe640 1 -- 172.21.15.89:0/517461614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1914100e20 msgr2=0x7f191410e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19137fe640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1914100e20 0x7f191410e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19137fe640 1 -- 172.21.15.89:0/517461614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1914102570 msgr2=0x7f191410ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19137fe640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1914102570 0x7f191410ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19137fe640 1 -- 172.21.15.89:0/517461614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19141a6940 con 0x7f1914102ef0 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19137fe640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1914102ef0 0x7f191410ef50 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f190c00eaa0 tx=0x7f190c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f18fbfff640 1 -- 172.21.15.89:0/517461614 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c00cdb0 con 0x7f1914102ef0 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f18fbfff640 1 -- 172.21.15.89:0/517461614 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f190c005020 con 0x7f1914102ef0 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19196e4640 1 -- 172.21.15.89:0/517461614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19141a6c30 con 0x7f1914102ef0 2024-09-15T23:38:24.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f18fbfff640 1 -- 172.21.15.89:0/517461614 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c0116b0 con 0x7f1914102ef0 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.805+0000 7f19196e4640 1 -- 172.21.15.89:0/517461614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19141a7170 con 0x7f1914102ef0 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.807+0000 7f18fbfff640 1 -- 172.21.15.89:0/517461614 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f190c005560 con 0x7f1914102ef0 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f191babd740 1 Processor -- start 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f191babd740 1 -- start start 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f191babd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01b4a0 0x56528e01b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f191babd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 0x56528e026140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f191babd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e028a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f191babd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56528df812e0 con 0x56528e01b4a0 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f191babd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56528df28ce0 con 0x56528e026680 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f191babd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56528df7d2a0 con 0x56528e01bdc0 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f18fbfff640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18fc0a3240 0x7f18fc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f19127fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01b4a0 0x56528e01b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f19127fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01b4a0 0x56528e01b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44208/0 (socket says 172.21.15.89:44208) 2024-09-15T23:38:24.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f19127fc640 1 -- 172.21.15.89:0/3641048873 learned_addr learned my addr 172.21.15.89:0/3641048873 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f19137fe640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18fc0a3240 0x7f18fc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f19127fc640 1 --2- 172.21.15.89:0/3641048873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e028a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f1912ffd640 1 --2- 172.21.15.89:0/3641048873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 0x56528e026140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f18fbfff640 1 -- 172.21.15.89:0/517461614 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f190c0cff90 con 0x7f1914102ef0 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f19127fc640 1 -- 172.21.15.89:0/3641048873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 msgr2=0x56528e026140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f19127fc640 1 --2- 172.21.15.89:0/3641048873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 0x56528e026140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f19127fc640 1 -- 172.21.15.89:0/3641048873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 msgr2=0x56528e028a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f19127fc640 1 --2- 172.21.15.89:0/3641048873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e028a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.808+0000 7f19127fc640 1 -- 172.21.15.89:0/3641048873 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56528df0d440 con 0x56528e01b4a0 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.809+0000 7f19127fc640 1 --2- 172.21.15.89:0/3641048873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01b4a0 0x56528e01b880 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f19080314e0 tx=0x7f19080056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb5dadaa98f581d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.809+0000 7f19137fe640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18fc0a3240 0x7f18fc0a5700 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f190c098b90 tx=0x7f190c06f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.809+0000 7f18f97fa640 1 -- 172.21.15.89:0/3641048873 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908035040 con 0x56528e01b4a0 2024-09-15T23:38:24.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.809+0000 7f18f97fa640 1 -- 172.21.15.89:0/3641048873 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908044070 con 0x56528e01b4a0 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.809+0000 7f18f97fa640 1 -- 172.21.15.89:0/3641048873 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908031b80 con 0x56528e01b4a0 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.809+0000 7f191babd740 1 -- 172.21.15.89:0/3641048873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01b4a0 msgr2=0x56528e01b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.809+0000 7f191babd740 1 --2- 172.21.15.89:0/3641048873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01b4a0 0x56528e01b880 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f19080314e0 tx=0x7f19080056d0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 -- 172.21.15.89:0/3641048873 shutdown_connections 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 --2- 172.21.15.89:0/3641048873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e028a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 --2- 172.21.15.89:0/3641048873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 0x56528e026140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 --2- 172.21.15.89:0/3641048873 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01b4a0 0x56528e01b880 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 -- 172.21.15.89:0/3641048873 >> 172.21.15.89:0/3641048873 conn(0x56528e017a30 msgr2=0x56528e017e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 -- 172.21.15.89:0/3641048873 shutdown_connections 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 -- 172.21.15.89:0/3641048873 wait complete. 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 -- start start 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 Processor -- start 2024-09-15T23:38:24.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01b4a0 0x56528e025f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e0231e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e023720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56528e029250 con 0x56528e01bdc0 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56528df3c050 con 0x56528e026680 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56528e0293d0 con 0x56528e01b4a0 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f191babd740 10 client.0 _ll_register_callbacks cb 0x56528df7c160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f1912ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e0231e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f1912ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e0231e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44212/0 (socket says 172.21.15.89:44212) 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.810+0000 7f1912ffd640 1 -- 172.21.15.89:0/1024237579 learned_addr learned my addr 172.21.15.89:0/1024237579 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f19127fc640 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01b4a0 0x56528e025f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f1912ffd640 1 -- 172.21.15.89:0/1024237579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01b4a0 msgr2=0x56528e025f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f1912ffd640 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01b4a0 0x56528e025f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f19127fc640 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e023720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f1912ffd640 1 -- 172.21.15.89:0/1024237579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 msgr2=0x56528e023720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f1912ffd640 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e023720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f1912ffd640 1 -- 172.21.15.89:0/1024237579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56528dcf23c0 con 0x56528e01bdc0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f1912ffd640 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e0231e0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f190400f760 tx=0x7f190400fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a28e8019e8f975 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f18f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f18f6ff5640 1 -- 172.21.15.89:0/1024237579 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904016070 con 0x56528e01bdc0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f191babd740 1 -- 172.21.15.89:0/1024237579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1908044070 con 0x56528e01bdc0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f18f6ff5640 1 -- 172.21.15.89:0/1024237579 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f190400bd10 con 0x56528e01bdc0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f18f6ff5640 1 -- 172.21.15.89:0/1024237579 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1904024050 con 0x56528e01bdc0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f191babd740 10 client.20454 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f191babd740 1 -- 172.21.15.89:0/1024237579 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56528e034c80 con 0x56528e01bdc0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f18f6ff5640 1 -- 172.21.15.89:0/1024237579 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908044070 con 0x56528e01bdc0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f18f6ff5640 1 -- 172.21.15.89:0/1024237579 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1904010550 con 0x56528e01bdc0 2024-09-15T23:38:24.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f191babd740 10 client.20454 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.811+0000 7f191babd740 10 client.20454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.812+0000 7f191babd740 20 client.20454 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.812+0000 7f191babd740 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56528e0360e0 0x56528e038500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.812+0000 7f191babd740 4 client.20454 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.812+0000 7f191babd740 1 -- 172.21.15.89:0/1024237579 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56528df8e850 con 0x56528e0360e0 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.812+0000 7f19127fc640 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56528e0360e0 0x56528e038500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.812+0000 7f18f6ff5640 10 client.20454 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.812+0000 7f19127fc640 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56528e0360e0 0x56528e038500 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58c5ec8a7142a131 server_cookie=a660d9c4a7713827 in_seq=0 out_seq=0 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f18f6ff5640 1 -- 172.21.15.89:0/1024237579 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56528df8e850 con 0x56528e0360e0 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f18f6ff5640 10 client.20454 handle_command_reply: tid=0 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 client.20454 shutdown 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 -- 172.21.15.89:0/1024237579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 msgr2=0x56528e0231e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:24.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e0231e0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f190400f760 tx=0x7f190400fc20 comp rx=0 tx=0).stop 2024-09-15T23:38:24.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 -- 172.21.15.89:0/1024237579 shutdown_connections 2024-09-15T23:38:24.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56528e0360e0 0x56528e038500 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e023720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e0231e0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 --2- 172.21.15.89:0/1024237579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01b4a0 0x56528e025f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f18f6ff5640 0 client.20454 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:24.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 -- 172.21.15.89:0/1024237579 >> 172.21.15.89:0/1024237579 conn(0x56528e017a30 msgr2=0x56528e01d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:24.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 -- 172.21.15.89:0/1024237579 shutdown_connections 2024-09-15T23:38:24.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 1 -- 172.21.15.89:0/1024237579 wait complete. 2024-09-15T23:38:24.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.813+0000 7f191babd740 20 client.20454 trim_cache size 0 max 16384 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f191babd740 1 Processor -- start 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f191babd740 1 -- start start 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f191babd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e01b4a0 0x56528e032510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f191babd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 0x56528e03ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f191babd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e026680 0x56528e03e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f191babd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f19141b0fe0 con 0x56528e026680 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f191babd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f19141b2b20 con 0x56528e01b4a0 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f191babd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f19141b16b0 con 0x56528e01bdc0 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f19127fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e026680 0x56528e03e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f19127fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e026680 0x56528e03e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44222/0 (socket says 172.21.15.89:44222) 2024-09-15T23:38:24.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f19127fc640 1 -- 172.21.15.89:0/2852937574 learned_addr learned my addr 172.21.15.89:0/2852937574 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f1912ffd640 1 --2- 172.21.15.89:0/2852937574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 0x56528e03ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f19127fc640 1 -- 172.21.15.89:0/2852937574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 msgr2=0x56528e03ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f19127fc640 1 --2- 172.21.15.89:0/2852937574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 0x56528e03ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f19127fc640 1 -- 172.21.15.89:0/2852937574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e01b4a0 msgr2=0x56528e032510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f19127fc640 1 --2- 172.21.15.89:0/2852937574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e01b4a0 0x56528e032510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f19127fc640 1 -- 172.21.15.89:0/2852937574 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56528e017860 con 0x56528e026680 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.830+0000 7f1912ffd640 1 --2- 172.21.15.89:0/2852937574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 0x56528e03ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f19127fc640 1 --2- 172.21.15.89:0/2852937574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e026680 0x56528e03e310 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f1908031730 tx=0x7f1908004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a3f7d84b3bd00b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f18f8ff9640 1 -- 172.21.15.89:0/2852937574 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908056070 con 0x56528e026680 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f18f8ff9640 1 -- 172.21.15.89:0/2852937574 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908044040 con 0x56528e026680 2024-09-15T23:38:24.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f18f8ff9640 1 -- 172.21.15.89:0/2852937574 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908035040 con 0x56528e026680 2024-09-15T23:38:24.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f191babd740 1 -- 172.21.15.89:0/2852937574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e026680 msgr2=0x56528e03e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:24.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f191babd740 1 --2- 172.21.15.89:0/2852937574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e026680 0x56528e03e310 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f1908031730 tx=0x7f1908004260 comp rx=0 tx=0).stop 2024-09-15T23:38:24.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f191babd740 1 -- 172.21.15.89:0/2852937574 shutdown_connections 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f191babd740 1 --2- 172.21.15.89:0/2852937574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e026680 0x56528e03e310 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f191babd740 1 --2- 172.21.15.89:0/2852937574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01bdc0 0x56528e03ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f191babd740 1 --2- 172.21.15.89:0/2852937574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e01b4a0 0x56528e032510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.831+0000 7f191babd740 1 -- 172.21.15.89:0/2852937574 >> 172.21.15.89:0/2852937574 conn(0x56528e017a30 msgr2=0x56528e025260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 1 -- 172.21.15.89:0/2852937574 shutdown_connections 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 1 -- 172.21.15.89:0/2852937574 wait complete. 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 1 -- start start 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 1 Processor -- start 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01b4a0 0x56528e02ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e02a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e02a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f19141b2450 con 0x56528e01bdc0 2024-09-15T23:38:24.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f19141b1d80 con 0x56528e026680 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f19141aefc0 con 0x56528e01b4a0 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f191babd740 10 client.0 _ll_register_callbacks cb 0x7f190400d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f1912ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e02a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f1912ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e02a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:44234/0 (socket says 172.21.15.89:44234) 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f1912ffd640 1 -- 172.21.15.89:0/3682722495 learned_addr learned my addr 172.21.15.89:0/3682722495 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f19127fc640 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e02a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.832+0000 7f19127fc640 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01b4a0 0x56528e02ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f1912ffd640 1 -- 172.21.15.89:0/3682722495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01b4a0 msgr2=0x56528e02ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f1912ffd640 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01b4a0 0x56528e02ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f1912ffd640 1 -- 172.21.15.89:0/3682722495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 msgr2=0x56528e02a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f1912ffd640 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e02a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f1912ffd640 1 -- 172.21.15.89:0/3682722495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56528e02aed0 con 0x56528e01bdc0 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f19127fc640 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e02a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:24.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f1912ffd640 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e02a160 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x56528e024820 tx=0x7f190400b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bf7f113d626f8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f18f6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f18f6ff5640 1 -- 172.21.15.89:0/3682722495 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904016050 con 0x56528e01bdc0 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f18f6ff5640 1 -- 172.21.15.89:0/3682722495 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1904010430 con 0x56528e01bdc0 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f191babd740 1 -- 172.21.15.89:0/3682722495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1908044040 con 0x56528e01bdc0 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f18f6ff5640 1 -- 172.21.15.89:0/3682722495 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1904010630 con 0x56528e01bdc0 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f191babd740 10 client.20460 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f191babd740 1 -- 172.21.15.89:0/3682722495 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56528e0a0de0 con 0x56528e01bdc0 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f18f6ff5640 1 -- 172.21.15.89:0/3682722495 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1908044040 con 0x56528e01bdc0 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f18f6ff5640 1 -- 172.21.15.89:0/3682722495 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f1904004640 con 0x56528e01bdc0 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f191babd740 10 client.20460 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f191babd740 10 client.20460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f191babd740 20 client.20460 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f191babd740 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56528e0a1530 0x56528e0a3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:24.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f191babd740 4 client.20460 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.833+0000 7f191babd740 1 -- 172.21.15.89:0/3682722495 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56528df88d60 con 0x56528e0a1530 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f19127fc640 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56528e0a1530 0x56528e0a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f19127fc640 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56528e0a1530 0x56528e0a3950 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cae3262ee249f48 server_cookie=2e17e9eb8f9f9ec in_seq=0 out_seq=0 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f18f6ff5640 10 client.20460 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f18f6ff5640 1 -- 172.21.15.89:0/3682722495 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56528df88d60 con 0x56528e0a1530 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f18f6ff5640 10 client.20460 handle_command_reply: tid=0 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f191babd740 1 client.20460 shutdown 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f191babd740 1 -- 172.21.15.89:0/3682722495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 msgr2=0x56528e02a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f191babd740 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e02a160 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x56528e024820 tx=0x7f190400b830 comp rx=0 tx=0).stop 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f191babd740 1 -- 172.21.15.89:0/3682722495 shutdown_connections 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f191babd740 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56528e0a1530 0x56528e0a3950 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f191babd740 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56528e026680 0x56528e02a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f191babd740 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528e01bdc0 0x56528e02a160 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f191babd740 1 --2- 172.21.15.89:0/3682722495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56528e01b4a0 0x56528e02ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.834+0000 7f18f6ff5640 0 client.20460 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f191babd740 1 -- 172.21.15.89:0/3682722495 >> 172.21.15.89:0/3682722495 conn(0x56528e017a30 msgr2=0x56528e016350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f191babd740 1 -- 172.21.15.89:0/3682722495 shutdown_connections 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f191babd740 1 -- 172.21.15.89:0/3682722495 wait complete. 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f191babd740 20 client.20460 trim_cache size 0 max 16384 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stdout: "6a8f7253-0eaa-442f-abd9-13d7ab664e6f": { 2024-09-15T23:38:24.842 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:38:24.843 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "6a8f7253-0eaa-442f-abd9-13d7ab664e6f", 2024-09-15T23:38:24.843 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:38:24.843 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:24.843 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:24.843 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f19196e4640 1 -- 172.21.15.89:0/517461614 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18fc0a3240 msgr2=0x7f18fc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f19196e4640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18fc0a3240 0x7f18fc0a5700 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f190c098b90 tx=0x7f190c06f040 comp rx=0 tx=0).stop 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f19196e4640 1 -- 172.21.15.89:0/517461614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1914102ef0 msgr2=0x7f191410ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f19196e4640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1914102ef0 0x7f191410ef50 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f190c00eaa0 tx=0x7f190c00ef60 comp rx=0 tx=0).stop 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f19196e4640 1 -- 172.21.15.89:0/517461614 shutdown_connections 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f19196e4640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f18fc0a3240 0x7f18fc0a5700 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f19196e4640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1914102ef0 0x7f191410ef50 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f19196e4640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1914102570 0x7f191410ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f19196e4640 1 --2- 172.21.15.89:0/517461614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1914100e20 0x7f191410e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.835+0000 7f19196e4640 1 -- 172.21.15.89:0/517461614 >> 172.21.15.89:0/517461614 conn(0x7f19140fc7d0 msgr2=0x7f1914106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.836+0000 7f19196e4640 1 -- 172.21.15.89:0/517461614 shutdown_connections 2024-09-15T23:38:24.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:24.836+0000 7f19196e4640 1 -- 172.21.15.89:0/517461614 wait complete. 2024-09-15T23:38:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:25 smithi089 ceph-mon[30933]: pgmap v781: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:38:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:25 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:38:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:25 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:38:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:25 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:38:25.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:25 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:38:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:25 smithi120 ceph-mon[35250]: pgmap v781: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:38:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:25 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:38:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:25 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:38:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:25 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:38:25.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:25 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:38:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:25 smithi174 ceph-mon[35586]: pgmap v781: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 16 op/s 2024-09-15T23:38:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:25 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:38:25.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:25 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:38:25.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:25 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:38:25.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:25 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:38:25.541 INFO:tasks.cephfs.filesystem:scrub status for tag:6a8f7253-0eaa-442f-abd9-13d7ab664e6f - {'path': '/', 'tag': '6a8f7253-0eaa-442f-abd9-13d7ab664e6f', 'options': 'recursive,force'} 2024-09-15T23:38:27.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:27 smithi089 ceph-mon[30933]: pgmap v782: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.8 KiB/s wr, 13 op/s 2024-09-15T23:38:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:27 smithi120 ceph-mon[35250]: pgmap v782: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.8 KiB/s wr, 13 op/s 2024-09-15T23:38:27.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:27 smithi174 ceph-mon[35586]: pgmap v782: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.8 KiB/s wr, 13 op/s 2024-09-15T23:38:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.852+0000 7f5e97181640 1 -- 172.21.15.89:0/111240921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e90100a50 msgr2=0x7f5e90100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.852+0000 7f5e97181640 1 --2- 172.21.15.89:0/111240921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e90100a50 0x7f5e90100e50 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f5e84009900 tx=0x7f5e8402a410 comp rx=0 tx=0).stop 2024-09-15T23:38:28.855 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.852+0000 7f5e97181640 1 -- 172.21.15.89:0/111240921 shutdown_connections 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.852+0000 7f5e97181640 1 --2- 172.21.15.89:0/111240921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e90102600 0x7f5e90108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.852+0000 7f5e97181640 1 --2- 172.21.15.89:0/111240921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e90101c40 0x7f5e901020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.852+0000 7f5e97181640 1 --2- 172.21.15.89:0/111240921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e90100a50 0x7f5e90100e50 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.852+0000 7f5e97181640 1 -- 172.21.15.89:0/111240921 >> 172.21.15.89:0/111240921 conn(0x7f5e900fc1b0 msgr2=0x7f5e900fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.852+0000 7f5e97181640 1 -- 172.21.15.89:0/111240921 shutdown_connections 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.852+0000 7f5e97181640 1 -- 172.21.15.89:0/111240921 wait complete. 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e97181640 1 Processor -- start 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e97181640 1 -- start start 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e97181640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e90100a50 0x7f5e90197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e97181640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e90101c40 0x7f5e90198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e97181640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e90102600 0x7f5e9019f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e97181640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9010c6d0 con 0x7f5e90100a50 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e97181640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9010c550 con 0x7f5e90101c40 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e97181640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9010c850 con 0x7f5e90102600 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e94f1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e90100a50 0x7f5e90197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e94f1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e90100a50 0x7f5e90197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38854/0 (socket says 172.21.15.89:38854) 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.853+0000 7f5e94f1d640 1 -- 172.21.15.89:0/901610940 learned_addr learned my addr 172.21.15.89:0/901610940 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e8ffff640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e90101c40 0x7f5e90198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e9571e640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e90102600 0x7f5e9019f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e94f1d640 1 -- 172.21.15.89:0/901610940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e90102600 msgr2=0x7f5e9019f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e90102600 0x7f5e9019f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e94f1d640 1 -- 172.21.15.89:0/901610940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e90101c40 msgr2=0x7f5e90198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e90101c40 0x7f5e90198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e94f1d640 1 -- 172.21.15.89:0/901610940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9019f9e0 con 0x7f5e90100a50 2024-09-15T23:38:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e9571e640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e90102600 0x7f5e9019f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e90100a50 0x7f5e90197dc0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f5e84031610 tx=0x7f5e840043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e8dffb640 1 -- 172.21.15.89:0/901610940 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8403f030 con 0x7f5e90100a50 2024-09-15T23:38:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e97181640 1 -- 172.21.15.89:0/901610940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9019fc70 con 0x7f5e90100a50 2024-09-15T23:38:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e8dffb640 1 -- 172.21.15.89:0/901610940 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e84003ce0 con 0x7f5e90100a50 2024-09-15T23:38:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e8dffb640 1 -- 172.21.15.89:0/901610940 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e84035070 con 0x7f5e90100a50 2024-09-15T23:38:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.854+0000 7f5e97181640 1 -- 172.21.15.89:0/901610940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e901a0160 con 0x7f5e90100a50 2024-09-15T23:38:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.857+0000 7f5e8dffb640 1 -- 172.21.15.89:0/901610940 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f5e84042030 con 0x7f5e90100a50 2024-09-15T23:38:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.857+0000 7f5e99ff4740 1 Processor -- start 2024-09-15T23:38:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.857+0000 7f5e99ff4740 1 -- start start 2024-09-15T23:38:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.857+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.857+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.857+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a8370 0x56033d6aa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.857+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56033d6062f0 con 0x56033d6a29a0 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.857+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56033d5ae270 con 0x56033d6a32e0 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.857+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56033d6022b0 con 0x56033d6a8370 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e9571e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e9571e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6a2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38868/0 (socket says 172.21.15.89:38868) 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e9571e640 1 -- 172.21.15.89:0/3530873464 learned_addr learned my addr 172.21.15.89:0/3530873464 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e8ffff640 1 --2- 172.21.15.89:0/3530873464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a8370 0x56033d6aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e8dffb640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5e800a32f0 0x7f5e800a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/3530873464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e9571e640 1 -- 172.21.15.89:0/3530873464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a8370 msgr2=0x56033d6aa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e9571e640 1 --2- 172.21.15.89:0/3530873464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a8370 0x56033d6aa830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e9571e640 1 -- 172.21.15.89:0/3530873464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 msgr2=0x56033d6a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e9571e640 1 --2- 172.21.15.89:0/3530873464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e9571e640 1 -- 172.21.15.89:0/3530873464 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56033d5ad340 con 0x56033d6a29a0 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/3530873464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e9571e640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5e800a32f0 0x7f5e800a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e8dffb640 1 -- 172.21.15.89:0/901610940 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5e840f0350 con 0x7f5e90100a50 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e9571e640 1 --2- 172.21.15.89:0/3530873464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6a2da0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009fa0 tx=0x7f5e8800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5b47d77228da37 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.858+0000 7f5e7affd640 1 -- 172.21.15.89:0/3530873464 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8801a610 con 0x56033d6a29a0 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e9571e640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5e800a32f0 0x7f5e800a57b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f5e88020000 tx=0x7f5e8801d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e7affd640 1 -- 172.21.15.89:0/3530873464 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e8801ac90 con 0x56033d6a29a0 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e7affd640 1 -- 172.21.15.89:0/3530873464 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e88010b40 con 0x56033d6a29a0 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3530873464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 msgr2=0x56033d6a2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/3530873464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6a2da0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009fa0 tx=0x7f5e8800f7f0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3530873464 shutdown_connections 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/3530873464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a8370 0x56033d6aa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/3530873464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/3530873464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6a2da0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3530873464 >> 172.21.15.89:0/3530873464 conn(0x56033d69e010 msgr2=0x56033d69e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3530873464 shutdown_connections 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.859+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3530873464 wait complete. 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e99ff4740 1 -- start start 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e99ff4740 1 Processor -- start 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6acc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a32e0 0x56033d6ad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a8370 0x56033d6b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56033d6ab340 con 0x56033d6a29a0 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56033d58f290 con 0x56033d6a8370 2024-09-15T23:38:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56033d6ab4c0 con 0x56033d6a32e0 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e99ff4740 10 client.0 _ll_register_callbacks cb 0x56033d5c0140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e94f1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e94f1d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6acc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38882/0 (socket says 172.21.15.89:38882) 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e94f1d640 1 -- 172.21.15.89:0/3594907993 learned_addr learned my addr 172.21.15.89:0/3594907993 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.860+0000 7f5e9571e640 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a8370 0x56033d6b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e8ffff640 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a32e0 0x56033d6ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e94f1d640 1 -- 172.21.15.89:0/3594907993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a32e0 msgr2=0x56033d6ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a32e0 0x56033d6ad160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e94f1d640 1 -- 172.21.15.89:0/3594907993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a8370 msgr2=0x56033d6b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a8370 0x56033d6b41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e94f1d640 1 -- 172.21.15.89:0/3594907993 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56033d319210 con 0x56033d6a29a0 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6acc20 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f5e840b89e0 tx=0x7f5e84043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82316ede6e07f742 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e6ffff640 1 -- 172.21.15.89:0/3594907993 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e84049070 con 0x56033d6a29a0 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e6ffff640 1 -- 172.21.15.89:0/3594907993 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5e84003800 con 0x56033d6a29a0 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3594907993 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e8801ac90 con 0x56033d6a29a0 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e6ffff640 1 -- 172.21.15.89:0/3594907993 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e840c51f0 con 0x56033d6a29a0 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e99ff4740 10 client.20472 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3594907993 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56033d3e5620 con 0x56033d6a29a0 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e6ffff640 1 -- 172.21.15.89:0/3594907993 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e8801ac90 con 0x56033d6a29a0 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e6ffff640 1 -- 172.21.15.89:0/3594907993 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5e840b8bf0 con 0x56033d6a29a0 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e99ff4740 10 client.20472 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.861+0000 7f5e99ff4740 10 client.20472 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.862+0000 7f5e99ff4740 20 client.20472 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.862+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56033d6b88b0 0x56033d6bad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.862+0000 7f5e99ff4740 4 client.20472 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.862+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3594907993 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56033d613a20 con 0x56033d6b88b0 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.862+0000 7f5e8ffff640 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56033d6b88b0 0x56033d6bad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.862+0000 7f5e6ffff640 10 client.20472 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.862+0000 7f5e8ffff640 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56033d6b88b0 0x56033d6bad10 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a3628cf0a8b7e70 server_cookie=580d6a0251214d24 in_seq=0 out_seq=0 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e6ffff640 1 -- 172.21.15.89:0/3594907993 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56033d613a20 con 0x56033d6b88b0 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e6ffff640 10 client.20472 handle_command_reply: tid=0 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 client.20472 shutdown 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3594907993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 msgr2=0x56033d6acc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6acc20 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f5e840b89e0 tx=0x7f5e84043000 comp rx=0 tx=0).stop 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3594907993 shutdown_connections 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56033d6b88b0 0x56033d6bad10 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a8370 0x56033d6b41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e6ffff640 0 client.20472 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a32e0 0x56033d6ad160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/3594907993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a29a0 0x56033d6acc20 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3594907993 >> 172.21.15.89:0/3594907993 conn(0x56033d69e010 msgr2=0x56033d69f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3594907993 shutdown_connections 2024-09-15T23:38:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 1 -- 172.21.15.89:0/3594907993 wait complete. 2024-09-15T23:38:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.863+0000 7f5e99ff4740 20 client.20472 trim_cache size 0 max 16384 2024-09-15T23:38:28.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e99ff4740 1 Processor -- start 2024-09-15T23:38:28.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e99ff4740 1 -- start start 2024-09-15T23:38:28.880 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a29a0 0x56033d69eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6b4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d69f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901aa850 con 0x56033d6a8370 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901ac390 con 0x56033d6a32e0 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901aaf20 con 0x56033d6a29a0 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e9571e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d69f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e9571e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d69f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38894/0 (socket says 172.21.15.89:38894) 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e9571e640 1 -- 172.21.15.89:0/108376678 learned_addr learned my addr 172.21.15.89:0/108376678 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/108376678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a29a0 0x56033d69eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e8ffff640 1 --2- 172.21.15.89:0/108376678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e9571e640 1 -- 172.21.15.89:0/108376678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a29a0 msgr2=0x56033d69eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e9571e640 1 --2- 172.21.15.89:0/108376678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a29a0 0x56033d69eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.881 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e9571e640 1 -- 172.21.15.89:0/108376678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 msgr2=0x56033d6b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e9571e640 1 --2- 172.21.15.89:0/108376678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6b4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e9571e640 1 -- 172.21.15.89:0/108376678 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56033d6bb390 con 0x56033d6a8370 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.878+0000 7f5e9571e640 1 --2- 172.21.15.89:0/108376678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d69f400 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f5e88005ba0 tx=0x7f5e880096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8410b06bb71c5495 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e7a7fc640 1 -- 172.21.15.89:0/108376678 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e880094a0 con 0x56033d6a8370 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e7a7fc640 1 -- 172.21.15.89:0/108376678 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e880103d0 con 0x56033d6a8370 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e7a7fc640 1 -- 172.21.15.89:0/108376678 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8801a410 con 0x56033d6a8370 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e99ff4740 1 -- 172.21.15.89:0/108376678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 msgr2=0x56033d69f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/108376678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d69f400 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f5e88005ba0 tx=0x7f5e880096a0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e99ff4740 1 -- 172.21.15.89:0/108376678 shutdown_connections 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/108376678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d69f400 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/108376678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/108376678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a29a0 0x56033d69eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e99ff4740 1 -- 172.21.15.89:0/108376678 >> 172.21.15.89:0/108376678 conn(0x56033d69e010 msgr2=0x56033d6ac9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e99ff4740 1 -- 172.21.15.89:0/108376678 shutdown_connections 2024-09-15T23:38:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e99ff4740 1 -- 172.21.15.89:0/108376678 wait complete. 2024-09-15T23:38:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.879+0000 7f5e99ff4740 1 -- start start 2024-09-15T23:38:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e99ff4740 1 Processor -- start 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a29a0 0x56033d6b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6b2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e99ff4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d6b3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901abcc0 con 0x56033d6a8370 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901ab5f0 con 0x56033d6a32e0 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e99ff4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5e901a8830 con 0x56033d6a29a0 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e99ff4740 10 client.0 _ll_register_callbacks cb 0x7f5e840b88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e9571e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d6b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e9571e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d6b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38900/0 (socket says 172.21.15.89:38900) 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e9571e640 1 -- 172.21.15.89:0/2405488482 learned_addr learned my addr 172.21.15.89:0/2405488482 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e8ffff640 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a29a0 0x56033d6b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e9571e640 1 -- 172.21.15.89:0/2405488482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a29a0 msgr2=0x56033d6b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e9571e640 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a29a0 0x56033d6b1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e9571e640 1 -- 172.21.15.89:0/2405488482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 msgr2=0x56033d6b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e9571e640 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6b2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e9571e640 1 -- 172.21.15.89:0/2405488482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56033d6b3630 con 0x56033d6a8370 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e8ffff640 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e9571e640 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d6b3020 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f5e88004af0 tx=0x7f5e88012370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd15d2e973909f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.880+0000 7f5e6ffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e6ffff640 1 -- 172.21.15.89:0/2405488482 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e880097f0 con 0x56033d6a8370 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e6ffff640 1 -- 172.21.15.89:0/2405488482 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f5e88010930 con 0x56033d6a8370 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e99ff4740 1 -- 172.21.15.89:0/2405488482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e880103d0 con 0x56033d6a8370 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e6ffff640 1 -- 172.21.15.89:0/2405488482 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5e88010b30 con 0x56033d6a8370 2024-09-15T23:38:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e99ff4740 10 client.20478 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e99ff4740 1 -- 172.21.15.89:0/2405488482 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56033d723800 con 0x56033d6a8370 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e6ffff640 1 -- 172.21.15.89:0/2405488482 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e880103d0 con 0x56033d6a8370 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e6ffff640 1 -- 172.21.15.89:0/2405488482 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5e88030630 con 0x56033d6a8370 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e99ff4740 10 client.20478 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e99ff4740 10 client.20478 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e99ff4740 20 client.20478 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56033d723f90 0x56033d7263b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e99ff4740 4 client.20478 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e99ff4740 1 -- 172.21.15.89:0/2405488482 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56033d60dd70 con 0x56033d723f90 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.881+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56033d723f90 0x56033d7263b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.882+0000 7f5e6ffff640 10 client.20478 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.882+0000 7f5e94f1d640 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56033d723f90 0x56033d7263b0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87f8e4c6ab060be5 server_cookie=a259cd6462123ea0 in_seq=0 out_seq=0 2024-09-15T23:38:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e6ffff640 1 -- 172.21.15.89:0/2405488482 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56033d60dd70 con 0x56033d723f90 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e6ffff640 10 client.20478 handle_command_reply: tid=0 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 client.20478 shutdown 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 -- 172.21.15.89:0/2405488482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 msgr2=0x56033d6b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d6b3020 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f5e88004af0 tx=0x7f5e88012370 comp rx=0 tx=0).stop 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 -- 172.21.15.89:0/2405488482 shutdown_connections 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56033d723f90 0x56033d7263b0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56033d6a8370 0x56033d6b3020 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56033d6a32e0 0x56033d6b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e6ffff640 0 client.20478 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 --2- 172.21.15.89:0/2405488482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56033d6a29a0 0x56033d6b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 -- 172.21.15.89:0/2405488482 >> 172.21.15.89:0/2405488482 conn(0x56033d69e010 msgr2=0x56033d69c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 -- 172.21.15.89:0/2405488482 shutdown_connections 2024-09-15T23:38:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.893+0000 7f5e99ff4740 1 -- 172.21.15.89:0/2405488482 wait complete. 2024-09-15T23:38:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e99ff4740 20 client.20478 trim_cache size 0 max 16384 2024-09-15T23:38:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:38:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:38:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:38:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:38:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 -- 172.21.15.89:0/901610940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5e800a32f0 msgr2=0x7f5e800a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5e800a32f0 0x7f5e800a57b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f5e88020000 tx=0x7f5e8801d000 comp rx=0 tx=0).stop 2024-09-15T23:38:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 -- 172.21.15.89:0/901610940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e90100a50 msgr2=0x7f5e90197dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e90100a50 0x7f5e90197dc0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f5e84031610 tx=0x7f5e840043b0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 -- 172.21.15.89:0/901610940 shutdown_connections 2024-09-15T23:38:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f5e800a32f0 0x7f5e800a57b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5e90102600 0x7f5e9019f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e90101c40 0x7f5e90198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 --2- 172.21.15.89:0/901610940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5e90100a50 0x7f5e90197dc0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 -- 172.21.15.89:0/901610940 >> 172.21.15.89:0/901610940 conn(0x7f5e900fc1b0 msgr2=0x7f5e900fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 -- 172.21.15.89:0/901610940 shutdown_connections 2024-09-15T23:38:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:28.894+0000 7f5e97181640 1 -- 172.21.15.89:0/901610940 wait complete. 2024-09-15T23:38:29.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:29 smithi089 ceph-mon[30933]: pgmap v783: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-09-15T23:38:29.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:29 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:38:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:29 smithi120 ceph-mon[35250]: pgmap v783: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-09-15T23:38:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:29 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:38:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:29 smithi174 ceph-mon[35586]: pgmap v783: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.6 KiB/s wr, 17 op/s 2024-09-15T23:38:29.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:29 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:38:31.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:31 smithi089 ceph-mon[30933]: pgmap v784: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T23:38:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:31 smithi120 ceph-mon[35250]: pgmap v784: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T23:38:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:31 smithi174 ceph-mon[35586]: pgmap v784: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-09-15T23:38:33.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:33 smithi089 ceph-mon[30933]: pgmap v785: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-09-15T23:38:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:33 smithi120 ceph-mon[35250]: pgmap v785: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-09-15T23:38:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:33 smithi174 ceph-mon[35586]: pgmap v785: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-09-15T23:38:35.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:35 smithi089 ceph-mon[30933]: pgmap v786: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-15T23:38:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:35 smithi120 ceph-mon[35250]: pgmap v786: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-15T23:38:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:35 smithi174 ceph-mon[35586]: pgmap v786: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-15T23:38:36.858 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:38:37.119 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:37.239 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:37 smithi089 ceph-mon[30933]: pgmap v787: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-15T23:38:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:37 smithi120 ceph-mon[35250]: pgmap v787: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-15T23:38:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:37 smithi174 ceph-mon[35586]: pgmap v787: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-09-15T23:38:37.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.592+0000 7f0508d22640 1 -- 172.21.15.89:0/217728005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0504102790 msgr2=0x7f05041067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:37.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.592+0000 7f0508d22640 1 --2- 172.21.15.89:0/217728005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0504102790 0x7f05041067e0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f04f8009900 tx=0x7f04f802a410 comp rx=0 tx=0).stop 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.592+0000 7f0508d22640 1 -- 172.21.15.89:0/217728005 shutdown_connections 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.592+0000 7f0508d22640 1 --2- 172.21.15.89:0/217728005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0504106f10 0x7f050410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.592+0000 7f0508d22640 1 --2- 172.21.15.89:0/217728005 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0504102790 0x7f05041067e0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.592+0000 7f0508d22640 1 --2- 172.21.15.89:0/217728005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0504101de0 0x7f05041021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.592+0000 7f0508d22640 1 -- 172.21.15.89:0/217728005 >> 172.21.15.89:0/217728005 conn(0x7f05040fd8c0 msgr2=0x7f05040ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.592+0000 7f0508d22640 1 -- 172.21.15.89:0/217728005 shutdown_connections 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.592+0000 7f0508d22640 1 -- 172.21.15.89:0/217728005 wait complete. 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.593+0000 7f0508d22640 1 Processor -- start 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.593+0000 7f0508d22640 1 -- start start 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.593+0000 7f0508d22640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0504101de0 0x7f050419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:37.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.593+0000 7f0508d22640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0504102790 0x7f050419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.593+0000 7f0508d22640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0504106f10 0x7f050419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.593+0000 7f0508d22640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0504111f70 con 0x7f0504101de0 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.593+0000 7f0508d22640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0504111df0 con 0x7f0504106f10 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.593+0000 7f0508d22640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f05041120f0 con 0x7f0504102790 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f050259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0504101de0 0x7f050419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f050259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0504101de0 0x7f050419a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51064/0 (socket says 172.21.15.89:51064) 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f050259c640 1 -- 172.21.15.89:0/3320090511 learned_addr learned my addr 172.21.15.89:0/3320090511 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f0501d9b640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0504102790 0x7f050419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f050259c640 1 -- 172.21.15.89:0/3320090511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0504102790 msgr2=0x7f050419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f050259c640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0504102790 0x7f050419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f050259c640 1 -- 172.21.15.89:0/3320090511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0504106f10 msgr2=0x7f050419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f0502d9d640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0504106f10 0x7f050419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:37.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f050259c640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0504106f10 0x7f050419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f050259c640 1 -- 172.21.15.89:0/3320090511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050419f6a0 con 0x7f0504101de0 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f050259c640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0504101de0 0x7f050419a6f0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f04f400c890 tx=0x7f04f400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f04f37fe640 1 -- 172.21.15.89:0/3320090511 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4004220 con 0x7f0504101de0 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f0508d22640 1 -- 172.21.15.89:0/3320090511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050419f990 con 0x7f0504101de0 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f04f37fe640 1 -- 172.21.15.89:0/3320090511 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f40043c0 con 0x7f0504101de0 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f04f37fe640 1 -- 172.21.15.89:0/3320090511 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f40049e0 con 0x7f0504101de0 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.594+0000 7f0508d22640 1 -- 172.21.15.89:0/3320090511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05041a7270 con 0x7f0504101de0 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.595+0000 7f0508d22640 1 -- 172.21.15.89:0/3320090511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05041021c0 con 0x7f0504101de0 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.596+0000 7f04f37fe640 1 -- 172.21.15.89:0/3320090511 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f04f40079b0 con 0x7f0504101de0 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.597+0000 7f04f37fe640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04ec0a3240 0x7f04ec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:37.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.597+0000 7f0501d9b640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04ec0a3240 0x7f04ec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:37.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.597+0000 7f04f37fe640 1 -- 172.21.15.89:0/3320090511 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f04f40cfe10 con 0x7f0504101de0 2024-09-15T23:38:37.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.597+0000 7f0501d9b640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04ec0a3240 0x7f04ec0a5700 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f05040fde20 tx=0x7f04f8031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:37.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.601+0000 7f04f37fe640 1 -- 172.21.15.89:0/3320090511 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f04f4014030 con 0x7f0504101de0 2024-09-15T23:38:37.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.802+0000 7f0508d22640 1 -- 172.21.15.89:0/3320090511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05041a7440 con 0x7f0504101de0 2024-09-15T23:38:37.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.803+0000 7f04f37fe640 1 -- 172.21.15.89:0/3320090511 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f04f409c740 con 0x7f0504101de0 2024-09-15T23:38:37.805 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:38:37.805 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:38:37.805 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:38:37.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 -- 172.21.15.89:0/3320090511 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04ec0a3240 msgr2=0x7f04ec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:37.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04ec0a3240 0x7f04ec0a5700 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f05040fde20 tx=0x7f04f8031490 comp rx=0 tx=0).stop 2024-09-15T23:38:37.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 -- 172.21.15.89:0/3320090511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0504101de0 msgr2=0x7f050419a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:37.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0504101de0 0x7f050419a6f0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f04f400c890 tx=0x7f04f400cd50 comp rx=0 tx=0).stop 2024-09-15T23:38:37.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 -- 172.21.15.89:0/3320090511 shutdown_connections 2024-09-15T23:38:37.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f04ec0a3240 0x7f04ec0a5700 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:37.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0504106f10 0x7f050419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:37.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0504102790 0x7f050419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:37.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 --2- 172.21.15.89:0/3320090511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0504101de0 0x7f050419a6f0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:37.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 -- 172.21.15.89:0/3320090511 >> 172.21.15.89:0/3320090511 conn(0x7f05040fd8c0 msgr2=0x7f05040fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:37.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 -- 172.21.15.89:0/3320090511 shutdown_connections 2024-09-15T23:38:37.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:37.805+0000 7f0508d22640 1 -- 172.21.15.89:0/3320090511 wait complete. 2024-09-15T23:38:38.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:38 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3320090511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:38.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:38 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3320090511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:38.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:38 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3320090511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:38.638 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 83e459f8 --release --await' 2024-09-15T23:38:38.639 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 83e459f8 --release --await 2024-09-15T23:38:38.885 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:39.294 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:39 smithi089 ceph-mon[30933]: pgmap v788: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2024-09-15T23:38:39.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.378+0000 7f737dc85640 1 -- 172.21.15.89:0/1984705824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737810c180 msgr2=0x7f737810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:39.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.378+0000 7f737dc85640 1 --2- 172.21.15.89:0/1984705824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737810c180 0x7f737810e610 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f7370009900 tx=0x7f737002a480 comp rx=0 tx=0).stop 2024-09-15T23:38:39.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.378+0000 7f737dc85640 1 -- 172.21.15.89:0/1984705824 shutdown_connections 2024-09-15T23:38:39.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.378+0000 7f737dc85640 1 --2- 172.21.15.89:0/1984705824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737810c180 0x7f737810e610 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.378+0000 7f737dc85640 1 --2- 172.21.15.89:0/1984705824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7378103700 0x7f737810bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.378+0000 7f737dc85640 1 --2- 172.21.15.89:0/1984705824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7378102de0 0x7f73781031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.378+0000 7f737dc85640 1 -- 172.21.15.89:0/1984705824 >> 172.21.15.89:0/1984705824 conn(0x7f73780fc7a0 msgr2=0x7f73780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:39.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.378+0000 7f737dc85640 1 -- 172.21.15.89:0/1984705824 shutdown_connections 2024-09-15T23:38:39.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.378+0000 7f737dc85640 1 -- 172.21.15.89:0/1984705824 wait complete. 2024-09-15T23:38:39.383 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f737dc85640 1 Processor -- start 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f737dc85640 1 -- start start 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f737dc85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7378102de0 0x7f7378198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f737dc85640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7378103700 0x7f7378198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f737dc85640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737810c180 0x7f737819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f737dc85640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7378111f70 con 0x7f737810c180 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f737dc85640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7378111df0 con 0x7f7378103700 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f737dc85640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73781120f0 con 0x7f7378102de0 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f7377fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737810c180 0x7f737819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f7377fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737810c180 0x7f737819cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51074/0 (socket says 172.21.15.89:51074) 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.379+0000 7f7377fff640 1 -- 172.21.15.89:0/203298558 learned_addr learned my addr 172.21.15.89:0/203298558 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f7376ffd640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7378103700 0x7f7378198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f73777fe640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7378102de0 0x7f7378198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.384 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f7377fff640 1 -- 172.21.15.89:0/203298558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7378102de0 msgr2=0x7f7378198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:39.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f7377fff640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7378102de0 0x7f7378198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f7377fff640 1 -- 172.21.15.89:0/203298558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7378103700 msgr2=0x7f7378198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:39.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f7377fff640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7378103700 0x7f7378198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f7377fff640 1 -- 172.21.15.89:0/203298558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737819d300 con 0x7f737810c180 2024-09-15T23:38:39.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f7377fff640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737810c180 0x7f737819cc20 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f7370005ab0 tx=0x7f73700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:39.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f7374ff9640 1 -- 172.21.15.89:0/203298558 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737003f030 con 0x7f737810c180 2024-09-15T23:38:39.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f7374ff9640 1 -- 172.21.15.89:0/203298558 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7370003ce0 con 0x7f737810c180 2024-09-15T23:38:39.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f7374ff9640 1 -- 172.21.15.89:0/203298558 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7370035070 con 0x7f737810c180 2024-09-15T23:38:39.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f737dc85640 1 -- 172.21.15.89:0/203298558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f737819d590 con 0x7f737810c180 2024-09-15T23:38:39.385 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.380+0000 7f737dc85640 1 -- 172.21.15.89:0/203298558 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f737819da20 con 0x7f737810c180 2024-09-15T23:38:39.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.382+0000 7f7374ff9640 1 -- 172.21.15.89:0/203298558 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7370042030 con 0x7f737810c180 2024-09-15T23:38:39.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f738005e740 1 Processor -- start 2024-09-15T23:38:39.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f738005e740 1 -- start start 2024-09-15T23:38:39.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f738005e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6060 0x56125bac6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f738005e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bad0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f738005e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bad1240 0x56125bad3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f738005e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56125ba2b620 con 0x56125bac6060 2024-09-15T23:38:39.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f738005e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56125b9d2e50 con 0x56125bac6980 2024-09-15T23:38:39.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f738005e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56125ba275e0 con 0x56125bad1240 2024-09-15T23:38:39.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7374ff9640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73600a32c0 0x7f73600a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7376ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6060 0x56125bac6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7376ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6060 0x56125bac6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51084/0 (socket says 172.21.15.89:51084) 2024-09-15T23:38:39.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7376ffd640 1 -- 172.21.15.89:0/4263731420 learned_addr learned my addr 172.21.15.89:0/4263731420 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:39.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7377fff640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73600a32c0 0x7f73600a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f73777fe640 1 --2- 172.21.15.89:0/4263731420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bad0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7376ffd640 1 -- 172.21.15.89:0/4263731420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bad1240 msgr2=0x56125bad3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7376ffd640 1 --2- 172.21.15.89:0/4263731420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bad1240 0x56125bad3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7376ffd640 1 -- 172.21.15.89:0/4263731420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 msgr2=0x56125bad0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7376ffd640 1 --2- 172.21.15.89:0/4263731420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bad0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7374ff9640 1 -- 172.21.15.89:0/203298558 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f73700f03e0 con 0x7f737810c180 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.383+0000 7f7376ffd640 1 -- 172.21.15.89:0/4263731420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56125b9c3d40 con 0x56125bac6060 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.384+0000 7f7376ffd640 1 --2- 172.21.15.89:0/4263731420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6060 0x56125bac6440 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f736c00a5c0 tx=0x7f736c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b0cc7d9fc112aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.384+0000 7f7377fff640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73600a32c0 0x7f73600a5780 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f73700bc640 tx=0x7f73700874e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.384+0000 7f735dffb640 1 -- 172.21.15.89:0/4263731420 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c01d070 con 0x56125bac6060 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.384+0000 7f735dffb640 1 -- 172.21.15.89:0/4263731420 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f736c018b20 con 0x56125bac6060 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.384+0000 7f735dffb640 1 -- 172.21.15.89:0/4263731420 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c010b00 con 0x56125bac6060 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 -- 172.21.15.89:0/4263731420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6060 msgr2=0x56125bac6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 --2- 172.21.15.89:0/4263731420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6060 0x56125bac6440 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f736c00a5c0 tx=0x7f736c00f790 comp rx=0 tx=0).stop 2024-09-15T23:38:39.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 -- 172.21.15.89:0/4263731420 shutdown_connections 2024-09-15T23:38:39.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 --2- 172.21.15.89:0/4263731420 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bad1240 0x56125bad3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 --2- 172.21.15.89:0/4263731420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bad0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 --2- 172.21.15.89:0/4263731420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6060 0x56125bac6440 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 -- 172.21.15.89:0/4263731420 >> 172.21.15.89:0/4263731420 conn(0x56125bac25f0 msgr2=0x56125bac29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:39.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 -- 172.21.15.89:0/4263731420 shutdown_connections 2024-09-15T23:38:39.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 -- 172.21.15.89:0/4263731420 wait complete. 2024-09-15T23:38:39.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 -- start start 2024-09-15T23:38:39.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.385+0000 7f738005e740 1 Processor -- start 2024-09-15T23:38:39.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f738005e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bace1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f738005e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bace6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f738005e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bacec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f738005e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56125bad3e10 con 0x56125bad1240 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f738005e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56125b9b3f10 con 0x56125bac6980 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f738005e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56125bad3f90 con 0x56125bac6060 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f738005e740 10 client.0 _ll_register_callbacks cb 0x56125b9d7db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7376ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bacec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7376ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bacec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51092/0 (socket says 172.21.15.89:51092) 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7376ffd640 1 -- 172.21.15.89:0/729640324 learned_addr learned my addr 172.21.15.89:0/729640324 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f73777fe640 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bace6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7376ffd640 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bace1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7376ffd640 1 -- 172.21.15.89:0/729640324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 msgr2=0x56125bace1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7376ffd640 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bace1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7376ffd640 1 -- 172.21.15.89:0/729640324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 msgr2=0x56125bace6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:39.394 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7376ffd640 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bace6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7376ffd640 1 -- 172.21.15.89:0/729640324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56125b73a580 con 0x56125bad1240 2024-09-15T23:38:39.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7376ffd640 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bacec20 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f736c00fc90 tx=0x7f736c012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d161595fd6e35d03 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:39.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7356ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:39.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7356ffd640 1 -- 172.21.15.89:0/729640324 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c02d060 con 0x56125bad1240 2024-09-15T23:38:39.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f738005e740 1 -- 172.21.15.89:0/729640324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f736c018b20 con 0x56125bad1240 2024-09-15T23:38:39.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.386+0000 7f7356ffd640 1 -- 172.21.15.89:0/729640324 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f736c005260 con 0x56125bad1240 2024-09-15T23:38:39.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f7356ffd640 1 -- 172.21.15.89:0/729640324 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f736c034050 con 0x56125bad1240 2024-09-15T23:38:39.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f738005e740 10 client.20496 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:39.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f738005e740 1 -- 172.21.15.89:0/729640324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56125b804960 con 0x56125bad1240 2024-09-15T23:38:39.395 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f7356ffd640 1 -- 172.21.15.89:0/729640324 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f736c018b20 con 0x56125bad1240 2024-09-15T23:38:39.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f7356ffd640 1 -- 172.21.15.89:0/729640324 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f736c004a50 con 0x56125bad1240 2024-09-15T23:38:39.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f738005e740 10 client.20496 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:39.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f738005e740 10 client.20496 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:38:39.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f738005e740 20 client.20496 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:39.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f738005e740 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56125bae0d80 0x56125bae3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.396 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f738005e740 4 client.20496 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:39.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f738005e740 1 -- 172.21.15.89:0/729640324 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56125ba38b90 con 0x56125bae0d80 2024-09-15T23:38:39.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.387+0000 7f73777fe640 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56125bae0d80 0x56125bae3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.388+0000 7f73777fe640 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56125bae0d80 0x56125bae3170 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25a2ea5fbf074802 server_cookie=ead66b0a9ac5138e in_seq=0 out_seq=0 2024-09-15T23:38:39.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.388+0000 7f7356ffd640 10 client.20496 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:39.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.388+0000 7f7356ffd640 1 -- 172.21.15.89:0/729640324 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56125ba38b90 con 0x56125bae0d80 2024-09-15T23:38:39.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.388+0000 7f7356ffd640 10 client.20496 handle_command_reply: tid=0 2024-09-15T23:38:39.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.388+0000 7f738005e740 1 client.20496 shutdown 2024-09-15T23:38:39.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 1 -- 172.21.15.89:0/729640324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 msgr2=0x56125bacec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:39.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bacec20 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f736c00fc90 tx=0x7f736c012040 comp rx=0 tx=0).stop 2024-09-15T23:38:39.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 1 -- 172.21.15.89:0/729640324 shutdown_connections 2024-09-15T23:38:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56125bae0d80 0x56125bae3170 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bacec20 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bace6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 1 --2- 172.21.15.89:0/729640324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bace1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 1 -- 172.21.15.89:0/729640324 >> 172.21.15.89:0/729640324 conn(0x56125bac25f0 msgr2=0x56125bac8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f7356ffd640 0 client.20496 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 1 -- 172.21.15.89:0/729640324 shutdown_connections 2024-09-15T23:38:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 1 -- 172.21.15.89:0/729640324 wait complete. 2024-09-15T23:38:39.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.389+0000 7f738005e740 20 client.20496 trim_cache size 0 max 16384 2024-09-15T23:38:39.408 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f738005e740 1 Processor -- start 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f738005e740 1 -- start start 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f738005e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bac8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f738005e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bac1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f738005e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bae88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f738005e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f73781aedc0 con 0x56125bad1240 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f738005e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73781b0900 con 0x56125bac6980 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f738005e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73781af490 con 0x56125bac6060 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f7376ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bae88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f7376ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bae88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51108/0 (socket says 172.21.15.89:51108) 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f7376ffd640 1 -- 172.21.15.89:0/3369136635 learned_addr learned my addr 172.21.15.89:0/3369136635 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:39.409 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f7376ffd640 1 --2- 172.21.15.89:0/3369136635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bac8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f73777fe640 1 --2- 172.21.15.89:0/3369136635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bac1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f7376ffd640 1 -- 172.21.15.89:0/3369136635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 msgr2=0x56125bac8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f7376ffd640 1 --2- 172.21.15.89:0/3369136635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bac8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f7376ffd640 1 -- 172.21.15.89:0/3369136635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 msgr2=0x56125bac1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f7376ffd640 1 --2- 172.21.15.89:0/3369136635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bac1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f7376ffd640 1 -- 172.21.15.89:0/3369136635 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56125bac2420 con 0x56125bad1240 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f73777fe640 1 --2- 172.21.15.89:0/3369136635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bac1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.406+0000 7f7376ffd640 1 --2- 172.21.15.89:0/3369136635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bae88e0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f736c00aa50 tx=0x7f736c00afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82fe249aea33f611 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f735d7fa640 1 -- 172.21.15.89:0/3369136635 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c01d030 con 0x56125bad1240 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f735d7fa640 1 -- 172.21.15.89:0/3369136635 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f736c00fcc0 con 0x56125bad1240 2024-09-15T23:38:39.410 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f735d7fa640 1 -- 172.21.15.89:0/3369136635 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c010ae0 con 0x56125bad1240 2024-09-15T23:38:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f738005e740 1 -- 172.21.15.89:0/3369136635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 msgr2=0x56125bae88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f738005e740 1 --2- 172.21.15.89:0/3369136635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bae88e0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f736c00aa50 tx=0x7f736c00afd0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f738005e740 1 -- 172.21.15.89:0/3369136635 shutdown_connections 2024-09-15T23:38:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f738005e740 1 --2- 172.21.15.89:0/3369136635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bad1240 0x56125bae88e0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f738005e740 1 --2- 172.21.15.89:0/3369136635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bac6980 0x56125bac1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f738005e740 1 --2- 172.21.15.89:0/3369136635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bac8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f738005e740 1 -- 172.21.15.89:0/3369136635 >> 172.21.15.89:0/3369136635 conn(0x56125bac25f0 msgr2=0x56125bac0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f738005e740 1 -- 172.21.15.89:0/3369136635 shutdown_connections 2024-09-15T23:38:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.407+0000 7f738005e740 1 -- 172.21.15.89:0/3369136635 wait complete. 2024-09-15T23:38:39.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f738005e740 1 -- start start 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f738005e740 1 Processor -- start 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f738005e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bacb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f738005e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6980 0x56125bacb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f738005e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bad1240 0x56125bacbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f738005e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f73781b0230 con 0x56125bac6980 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f738005e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73781afb60 con 0x56125bad1240 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f738005e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73781acda0 con 0x56125bac6060 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f738005e740 10 client.0 _ll_register_callbacks cb 0x7f736c018820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f73777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6980 0x56125bacb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f7376ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bacb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f73777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6980 0x56125bacb7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51112/0 (socket says 172.21.15.89:51112) 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f7376ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bacb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:51014/0 (socket says 172.21.15.89:51014) 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f73777fe640 1 -- 172.21.15.89:0/2880740007 learned_addr learned my addr 172.21.15.89:0/2880740007 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f7376ffd640 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bad1240 0x56125bacbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f73777fe640 1 -- 172.21.15.89:0/2880740007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 msgr2=0x56125bacb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.408+0000 7f73777fe640 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bacb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f73777fe640 1 -- 172.21.15.89:0/2880740007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bad1240 msgr2=0x56125bacbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f73777fe640 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bad1240 0x56125bacbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f73777fe640 1 -- 172.21.15.89:0/2880740007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56125bb4a9b0 con 0x56125bac6980 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f73777fe640 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6980 0x56125bacb7f0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f7368004a80 tx=0x7f7368012a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75def9b12c65d15 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f7356ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f7356ffd640 1 -- 172.21.15.89:0/2880740007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73680117f0 con 0x56125bac6980 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f7356ffd640 1 -- 172.21.15.89:0/2880740007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7368011990 con 0x56125bac6980 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f738005e740 1 -- 172.21.15.89:0/2880740007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f736c00fcc0 con 0x56125bac6980 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f7356ffd640 1 -- 172.21.15.89:0/2880740007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f736802f050 con 0x56125bac6980 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f738005e740 10 client.20502 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f738005e740 1 -- 172.21.15.89:0/2880740007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56125bb4bc50 con 0x56125bac6980 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f7356ffd640 1 -- 172.21.15.89:0/2880740007 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f736c00fcc0 con 0x56125bac6980 2024-09-15T23:38:39.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f7356ffd640 1 -- 172.21.15.89:0/2880740007 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f736800f040 con 0x56125bac6980 2024-09-15T23:38:39.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f738005e740 10 client.20502 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:39.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f738005e740 10 client.20502 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:38:39.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f738005e740 20 client.20502 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:39.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f738005e740 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56125bb4c3b0 0x56125bb4e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:39.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f738005e740 4 client.20502 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "83e459f8", "release": true, "await": true}] 2024-09-15T23:38:39.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.409+0000 7f738005e740 1 -- 172.21.15.89:0/2880740007 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "83e459f8", "release": true, "await": true}) -- 0x56125ba330a0 con 0x56125bb4c3b0 2024-09-15T23:38:39.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.410+0000 7f7376ffd640 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56125bb4c3b0 0x56125bb4e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:39.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.410+0000 7f7356ffd640 10 client.20502 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:39.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.410+0000 7f7376ffd640 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56125bb4c3b0 0x56125bb4e7d0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=603236c9008e5c05 server_cookie=19914c87be2a1331 in_seq=0 out_seq=0 2024-09-15T23:38:39.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f7356ffd640 1 -- 172.21.15.89:0/2880740007 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x56125ba330a0 con 0x56125bb4c3b0 2024-09-15T23:38:39.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f7356ffd640 10 client.20502 handle_command_reply: tid=0 2024-09-15T23:38:39.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 client.20502 shutdown 2024-09-15T23:38:39.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 -- 172.21.15.89:0/2880740007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6980 msgr2=0x56125bacb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:39.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6980 0x56125bacb7f0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f7368004a80 tx=0x7f7368012a20 comp rx=0 tx=0).stop 2024-09-15T23:38:39.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 -- 172.21.15.89:0/2880740007 shutdown_connections 2024-09-15T23:38:39.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56125bb4c3b0 0x56125bb4e7d0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56125bad1240 0x56125bacbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56125bac6980 0x56125bacb7f0 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f7356ffd640 0 client.20502 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 --2- 172.21.15.89:0/2880740007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56125bac6060 0x56125bacb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 -- 172.21.15.89:0/2880740007 >> 172.21.15.89:0/2880740007 conn(0x56125bac25f0 msgr2=0x56125bace980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 -- 172.21.15.89:0/2880740007 shutdown_connections 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 1 -- 172.21.15.89:0/2880740007 wait complete. 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.413+0000 7f738005e740 20 client.20502 trim_cache size 0 max 16384 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:38:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 52, 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "83e459f8": { 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 52, 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 146.5, 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:38:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:38:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:38:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:38:39.420 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 -- 172.21.15.89:0/203298558 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73600a32c0 msgr2=0x7f73600a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73600a32c0 0x7f73600a5780 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f73700bc640 tx=0x7f73700874e0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 -- 172.21.15.89:0/203298558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737810c180 msgr2=0x7f737819cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737810c180 0x7f737819cc20 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f7370005ab0 tx=0x7f73700042f0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 -- 172.21.15.89:0/203298558 shutdown_connections 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f73600a32c0 0x7f73600a5780 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f737810c180 0x7f737819cc20 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7378103700 0x7f7378198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 --2- 172.21.15.89:0/203298558 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7378102de0 0x7f7378198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 -- 172.21.15.89:0/203298558 >> 172.21.15.89:0/203298558 conn(0x7f73780fc7a0 msgr2=0x7f73780fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 -- 172.21.15.89:0/203298558 shutdown_connections 2024-09-15T23:38:39.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:39.414+0000 7f737dc85640 1 -- 172.21.15.89:0/203298558 wait complete. 2024-09-15T23:38:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:39 smithi120 ceph-mon[35250]: pgmap v788: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2024-09-15T23:38:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:39 smithi174 ceph-mon[35586]: pgmap v788: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2024-09-15T23:38:40.219 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '83e459f8', total seconds elapsed: 29.9 2024-09-15T23:38:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:41 smithi120 ceph-mon[35250]: pgmap v789: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-09-15T23:38:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:41 smithi174 ceph-mon[35586]: pgmap v789: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-09-15T23:38:41.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:41 smithi089 ceph-mon[30933]: pgmap v789: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-09-15T23:38:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 -- 172.21.15.89:0/29245703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4200102d50 msgr2=0x7f42001031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 --2- 172.21.15.89:0/29245703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4200102d50 0x7f42001031d0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f41ec009900 tx=0x7f41ec02a410 comp rx=0 tx=0).stop 2024-09-15T23:38:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 -- 172.21.15.89:0/29245703 shutdown_connections 2024-09-15T23:38:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 --2- 172.21.15.89:0/29245703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4200103710 0x7f4200109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 --2- 172.21.15.89:0/29245703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4200102d50 0x7f42001031d0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 --2- 172.21.15.89:0/29245703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4200101b60 0x7f4200101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 -- 172.21.15.89:0/29245703 >> 172.21.15.89:0/29245703 conn(0x7f42000fd2e0 msgr2=0x7f42000ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 -- 172.21.15.89:0/29245703 shutdown_connections 2024-09-15T23:38:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 -- 172.21.15.89:0/29245703 wait complete. 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 Processor -- start 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.991+0000 7f4206e5e640 1 -- start start 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f4206e5e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4200101b60 0x7f4200197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f4206e5e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4200102d50 0x7f4200198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f4206e5e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4200103710 0x7f420019f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f4206e5e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f420010d7c0 con 0x7f4200103710 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f4206e5e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f420010d640 con 0x7f4200102d50 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f4206e5e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f420010d940 con 0x7f4200101b60 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f42053fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4200103710 0x7f420019f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f42053fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4200103710 0x7f420019f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51140/0 (socket says 172.21.15.89:51140) 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f42053fb640 1 -- 172.21.15.89:0/32783631 learned_addr learned my addr 172.21.15.89:0/32783631 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f4204bfa640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4200101b60 0x7f4200197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f41fffff640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4200102d50 0x7f4200198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f42053fb640 1 -- 172.21.15.89:0/32783631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4200101b60 msgr2=0x7f4200197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f42053fb640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4200101b60 0x7f4200197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f42053fb640 1 -- 172.21.15.89:0/32783631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4200102d50 msgr2=0x7f4200198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f42053fb640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4200102d50 0x7f4200198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f42053fb640 1 -- 172.21.15.89:0/32783631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420019fa00 con 0x7f4200103710 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f42053fb640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4200103710 0x7f420019f390 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f41f800cad0 tx=0x7f41f800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.992+0000 7f41fdffb640 1 -- 172.21.15.89:0/32783631 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f800ae20 con 0x7f4200103710 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.993+0000 7f41fdffb640 1 -- 172.21.15.89:0/32783631 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f8005020 con 0x7f4200103710 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.993+0000 7f4206e5e640 1 -- 172.21.15.89:0/32783631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420019fcf0 con 0x7f4200103710 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.993+0000 7f41fdffb640 1 -- 172.21.15.89:0/32783631 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f800f6f0 con 0x7f4200103710 2024-09-15T23:38:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.993+0000 7f4206e5e640 1 -- 172.21.15.89:0/32783631 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42001a0190 con 0x7f4200103710 2024-09-15T23:38:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.996+0000 7f41fdffb640 1 -- 172.21.15.89:0/32783631 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f41f80051d0 con 0x7f4200103710 2024-09-15T23:38:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.996+0000 7f4209bf8740 1 Processor -- start 2024-09-15T23:38:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.996+0000 7f4209bf8740 1 -- start start 2024-09-15T23:38:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.996+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b368826db0 0x55b3688271b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.996+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3688276f0 0x55b36882c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.996+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b36882c780 0x55b36882ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.996+0000 7f4209bf8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b3687862f0 con 0x55b36882c780 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.996+0000 7f4209bf8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b36872e270 con 0x55b3688276f0 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.996+0000 7f4209bf8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b3687822b0 con 0x55b368826db0 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.996+0000 7f41fdffb640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41f00a32f0 0x7f41f00a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b36882c780 0x55b36882ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b36882c780 0x55b36882ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51154/0 (socket says 172.21.15.89:51154) 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fffff640 1 -- 172.21.15.89:0/2542952982 learned_addr learned my addr 172.21.15.89:0/2542952982 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f4204bfa640 1 --2- 172.21.15.89:0/2542952982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3688276f0 0x55b36882c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fffff640 1 --2- 172.21.15.89:0/2542952982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b368826db0 0x55b3688271b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f42053fb640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41f00a32f0 0x7f41f00a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fdffb640 1 -- 172.21.15.89:0/32783631 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f41f80cfe70 con 0x7f4200103710 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fffff640 1 -- 172.21.15.89:0/2542952982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b368826db0 msgr2=0x55b3688271b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:42.004 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fffff640 1 --2- 172.21.15.89:0/2542952982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b368826db0 0x55b3688271b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fffff640 1 -- 172.21.15.89:0/2542952982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3688276f0 msgr2=0x55b36882c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fffff640 1 --2- 172.21.15.89:0/2542952982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3688276f0 0x55b36882c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fffff640 1 -- 172.21.15.89:0/2542952982 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b36872d340 con 0x55b36882c780 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41fffff640 1 --2- 172.21.15.89:0/2542952982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b36882c780 0x55b36882ec40 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f41ec0096c0 tx=0x7f41ec031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72188ad00a224d67 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f42053fb640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41f00a32f0 0x7f41f00a57b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f41f809c030 tx=0x7f41f8091580 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41eaffd640 1 -- 172.21.15.89:0/2542952982 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec035040 con 0x55b36882c780 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41eaffd640 1 -- 172.21.15.89:0/2542952982 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41ec0035f0 con 0x55b36882c780 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.997+0000 7f41eaffd640 1 -- 172.21.15.89:0/2542952982 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec03c650 con 0x55b36882c780 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.998+0000 7f4209bf8740 1 -- 172.21.15.89:0/2542952982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b36882c780 msgr2=0x55b36882ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.998+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2542952982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b36882c780 0x55b36882ec40 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f41ec0096c0 tx=0x7f41ec031400 comp rx=0 tx=0).stop 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.998+0000 7f4209bf8740 1 -- 172.21.15.89:0/2542952982 shutdown_connections 2024-09-15T23:38:42.005 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.998+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2542952982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b36882c780 0x55b36882ec40 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.998+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2542952982 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b3688276f0 0x55b36882c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.998+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2542952982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b368826db0 0x55b3688271b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.998+0000 7f4209bf8740 1 -- 172.21.15.89:0/2542952982 >> 172.21.15.89:0/2542952982 conn(0x55b36881e010 msgr2=0x55b36881e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.998+0000 7f4209bf8740 1 -- 172.21.15.89:0/2542952982 shutdown_connections 2024-09-15T23:38:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.998+0000 7f4209bf8740 1 -- 172.21.15.89:0/2542952982 wait complete. 2024-09-15T23:38:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f4209bf8740 1 -- start start 2024-09-15T23:38:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f4209bf8740 1 Processor -- start 2024-09-15T23:38:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b368826db0 0x55b368830fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3688276f0 0x55b3688314e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.006 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b36882c780 0x55b368838520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f4209bf8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b36882f6f0 con 0x55b368826db0 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f4209bf8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b36870f290 con 0x55b36882c780 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f4209bf8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b36882f870 con 0x55b3688276f0 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f4209bf8740 10 client.0 _ll_register_callbacks cb 0x55b368740140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f41fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b368826db0 0x55b368830fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f41fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b368826db0 0x55b368830fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51162/0 (socket says 172.21.15.89:51162) 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f41fffff640 1 -- 172.21.15.89:0/1626286360 learned_addr learned my addr 172.21.15.89:0/1626286360 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f4204bfa640 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3688276f0 0x55b3688314e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f41fffff640 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b36882c780 0x55b368838520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f41fffff640 1 -- 172.21.15.89:0/1626286360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3688276f0 msgr2=0x55b3688314e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f41fffff640 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3688276f0 0x55b3688314e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f41fffff640 1 -- 172.21.15.89:0/1626286360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b36882c780 msgr2=0x55b368838520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f41fffff640 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b36882c780 0x55b368838520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:41.999+0000 7f41fffff640 1 -- 172.21.15.89:0/1626286360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b368499210 con 0x55b368826db0 2024-09-15T23:38:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f41fffff640 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b368826db0 0x55b368830fa0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f41ec005c40 tx=0x7f41ec032f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc497140825367 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f41dffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f41dffff640 1 -- 172.21.15.89:0/1626286360 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec035070 con 0x55b368826db0 2024-09-15T23:38:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f4209bf8740 1 -- 172.21.15.89:0/1626286360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41ec0035f0 con 0x55b368826db0 2024-09-15T23:38:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f41dffff640 1 -- 172.21.15.89:0/1626286360 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f41ec0319a0 con 0x55b368826db0 2024-09-15T23:38:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f41dffff640 1 -- 172.21.15.89:0/1626286360 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f41ec05a050 con 0x55b368826db0 2024-09-15T23:38:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f4209bf8740 10 client.20514 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f4209bf8740 1 -- 172.21.15.89:0/1626286360 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b368565620 con 0x55b368826db0 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f41dffff640 1 -- 172.21.15.89:0/1626286360 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41ec0035f0 con 0x55b368826db0 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f41dffff640 1 -- 172.21.15.89:0/1626286360 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f41ec03a040 con 0x55b368826db0 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f4209bf8740 10 client.20514 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f4209bf8740 10 client.20514 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f4209bf8740 20 client.20514 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.000+0000 7f4209bf8740 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b36883c820 0x55b36883ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.001+0000 7f4209bf8740 4 client.20514 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.001+0000 7f4209bf8740 1 -- 172.21.15.89:0/1626286360 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b368793a20 con 0x55b36883c820 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.001+0000 7f4204bfa640 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b36883c820 0x55b36883ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.001+0000 7f4204bfa640 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b36883c820 0x55b36883ec10 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98a81beca6e4c1a4 server_cookie=47abf5c9b67ec7c in_seq=0 out_seq=0 2024-09-15T23:38:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.001+0000 7f41dffff640 10 client.20514 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f41dffff640 1 -- 172.21.15.89:0/1626286360 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b368793a20 con 0x55b36883c820 2024-09-15T23:38:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f41dffff640 10 client.20514 handle_command_reply: tid=0 2024-09-15T23:38:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 client.20514 shutdown 2024-09-15T23:38:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 -- 172.21.15.89:0/1626286360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b368826db0 msgr2=0x55b368830fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b368826db0 0x55b368830fa0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f41ec005c40 tx=0x7f41ec032f70 comp rx=0 tx=0).stop 2024-09-15T23:38:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 -- 172.21.15.89:0/1626286360 shutdown_connections 2024-09-15T23:38:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b36883c820 0x55b36883ec10 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b36882c780 0x55b368838520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f41dffff640 0 client.20514 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b3688276f0 0x55b3688314e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 --2- 172.21.15.89:0/1626286360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b368826db0 0x55b368830fa0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 -- 172.21.15.89:0/1626286360 >> 172.21.15.89:0/1626286360 conn(0x55b36881e010 msgr2=0x55b36881f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 -- 172.21.15.89:0/1626286360 shutdown_connections 2024-09-15T23:38:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 1 -- 172.21.15.89:0/1626286360 wait complete. 2024-09-15T23:38:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.002+0000 7f4209bf8740 20 client.20514 trim_cache size 0 max 16384 2024-09-15T23:38:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4209bf8740 1 Processor -- start 2024-09-15T23:38:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4209bf8740 1 -- start start 2024-09-15T23:38:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b368826db0 0x55b368830d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b3688384e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b36882c780 0x55b368831290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4209bf8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42001a8670 con 0x55b3688276f0 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4209bf8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f42001aa1b0 con 0x55b368826db0 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4209bf8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42001a8d40 con 0x55b36882c780 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4204bfa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b3688384e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4204bfa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b3688384e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51170/0 (socket says 172.21.15.89:51170) 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4204bfa640 1 -- 172.21.15.89:0/2279959945 learned_addr learned my addr 172.21.15.89:0/2279959945 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f41fffff640 1 --2- 172.21.15.89:0/2279959945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b36882c780 0x55b368831290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4204bfa640 1 -- 172.21.15.89:0/2279959945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b36882c780 msgr2=0x55b368831290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4204bfa640 1 --2- 172.21.15.89:0/2279959945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b36882c780 0x55b368831290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4204bfa640 1 -- 172.21.15.89:0/2279959945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b368826db0 msgr2=0x55b368830d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4204bfa640 1 --2- 172.21.15.89:0/2279959945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b368826db0 0x55b368830d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4204bfa640 1 -- 172.21.15.89:0/2279959945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3688317d0 con 0x55b3688276f0 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f4204bfa640 1 --2- 172.21.15.89:0/2279959945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b3688384e0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x55b36881ee90 tx=0x7f41f4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108bc891500ac12f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.017+0000 7f41ea7fc640 1 -- 172.21.15.89:0/2279959945 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4022030 con 0x55b3688276f0 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f41ea7fc640 1 -- 172.21.15.89:0/2279959945 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f40035f0 con 0x55b3688276f0 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f41ea7fc640 1 -- 172.21.15.89:0/2279959945 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f401d680 con 0x55b3688276f0 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 -- 172.21.15.89:0/2279959945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 msgr2=0x55b3688384e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2279959945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b3688384e0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x55b36881ee90 tx=0x7f41f4011810 comp rx=0 tx=0).stop 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 -- 172.21.15.89:0/2279959945 shutdown_connections 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2279959945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b36882c780 0x55b368831290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2279959945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b3688384e0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2279959945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b368826db0 0x55b368830d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 -- 172.21.15.89:0/2279959945 >> 172.21.15.89:0/2279959945 conn(0x55b36881e010 msgr2=0x55b368828890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 -- 172.21.15.89:0/2279959945 shutdown_connections 2024-09-15T23:38:42.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 -- 172.21.15.89:0/2279959945 wait complete. 2024-09-15T23:38:42.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 -- start start 2024-09-15T23:38:42.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.018+0000 7f4209bf8740 1 Processor -- start 2024-09-15T23:38:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b368826db0 0x55b368835750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b368835c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4209bf8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b36882c780 0x55b368836ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4209bf8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42001a9ae0 con 0x55b3688276f0 2024-09-15T23:38:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4209bf8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f42001a9410 con 0x55b36882c780 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4209bf8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42001a6650 con 0x55b368826db0 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4209bf8740 10 client.0 _ll_register_callbacks cb 0x7f41f400b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4204bfa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b368835c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4204bfa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b368835c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:51180/0 (socket says 172.21.15.89:51180) 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4204bfa640 1 -- 172.21.15.89:0/2695136487 learned_addr learned my addr 172.21.15.89:0/2695136487 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f41fffff640 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b368826db0 0x55b368835750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f41fffff640 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b36882c780 0x55b368836ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4204bfa640 1 -- 172.21.15.89:0/2695136487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b368826db0 msgr2=0x55b368835750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4204bfa640 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b368826db0 0x55b368835750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4204bfa640 1 -- 172.21.15.89:0/2695136487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b36882c780 msgr2=0x55b368836ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4204bfa640 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b36882c780 0x55b368836ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4204bfa640 1 -- 172.21.15.89:0/2695136487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3688372b0 con 0x55b3688276f0 2024-09-15T23:38:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f4204bfa640 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b368835c90 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f41f400bca0 tx=0x7f41f4004020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250c7cc4197505f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.019+0000 7f41dffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f41dffff640 1 -- 172.21.15.89:0/2695136487 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40037a0 con 0x55b3688276f0 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f41dffff640 1 -- 172.21.15.89:0/2695136487 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f41f4003940 con 0x55b3688276f0 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f4209bf8740 1 -- 172.21.15.89:0/2695136487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f40035f0 con 0x55b3688276f0 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f41dffff640 1 -- 172.21.15.89:0/2695136487 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f41f4039050 con 0x55b3688276f0 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f4209bf8740 10 client.20520 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f4209bf8740 1 -- 172.21.15.89:0/2695136487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b368838630 con 0x55b3688276f0 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f41dffff640 1 -- 172.21.15.89:0/2695136487 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f40035f0 con 0x55b3688276f0 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f41dffff640 1 -- 172.21.15.89:0/2695136487 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f41f4028570 con 0x55b3688276f0 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f4209bf8740 10 client.20520 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f4209bf8740 10 client.20520 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f4209bf8740 20 client.20520 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b3688a7850 0x55b3688a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f4209bf8740 4 client.20520 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:38:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f4209bf8740 1 -- 172.21.15.89:0/2695136487 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b36878dd70 con 0x55b3688a7850 2024-09-15T23:38:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.020+0000 7f41fffff640 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b3688a7850 0x55b3688a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.021+0000 7f41fffff640 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b3688a7850 0x55b3688a9c70 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=572c6e5d6f531f8b server_cookie=737c2ee16de4b063 in_seq=0 out_seq=0 2024-09-15T23:38:42.028 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.021+0000 7f41dffff640 10 client.20520 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.032+0000 7f41dffff640 1 -- 172.21.15.89:0/2695136487 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b36878dd70 con 0x55b3688a7850 2024-09-15T23:38:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.032+0000 7f41dffff640 10 client.20520 handle_command_reply: tid=0 2024-09-15T23:38:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.032+0000 7f4209bf8740 1 client.20520 shutdown 2024-09-15T23:38:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 1 -- 172.21.15.89:0/2695136487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 msgr2=0x55b368835c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b368835c90 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f41f400bca0 tx=0x7f41f4004020 comp rx=0 tx=0).stop 2024-09-15T23:38:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 1 -- 172.21.15.89:0/2695136487 shutdown_connections 2024-09-15T23:38:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b3688a7850 0x55b3688a9c70 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b36882c780 0x55b368836ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b3688276f0 0x55b368835c90 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f41dffff640 0 client.20520 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 1 --2- 172.21.15.89:0/2695136487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b368826db0 0x55b368835750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 1 -- 172.21.15.89:0/2695136487 >> 172.21.15.89:0/2695136487 conn(0x55b36881e010 msgr2=0x55b368828890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 1 -- 172.21.15.89:0/2695136487 shutdown_connections 2024-09-15T23:38:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 1 -- 172.21.15.89:0/2695136487 wait complete. 2024-09-15T23:38:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4209bf8740 20 client.20520 trim_cache size 0 max 16384 2024-09-15T23:38:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:38:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:38:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:38:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:38:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4206e5e640 1 -- 172.21.15.89:0/32783631 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41f00a32f0 msgr2=0x7f41f00a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4206e5e640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41f00a32f0 0x7f41f00a57b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f41f809c030 tx=0x7f41f8091580 comp rx=0 tx=0).stop 2024-09-15T23:38:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4206e5e640 1 -- 172.21.15.89:0/32783631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4200103710 msgr2=0x7f420019f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.033+0000 7f4206e5e640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4200103710 0x7f420019f390 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f41f800cad0 tx=0x7f41f800cf90 comp rx=0 tx=0).stop 2024-09-15T23:38:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.034+0000 7f4206e5e640 1 -- 172.21.15.89:0/32783631 shutdown_connections 2024-09-15T23:38:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.034+0000 7f4206e5e640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f41f00a32f0 0x7f41f00a57b0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.034+0000 7f4206e5e640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4200103710 0x7f420019f390 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.034+0000 7f4206e5e640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4200102d50 0x7f4200198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.034+0000 7f4206e5e640 1 --2- 172.21.15.89:0/32783631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4200101b60 0x7f4200197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.034+0000 7f4206e5e640 1 -- 172.21.15.89:0/32783631 >> 172.21.15.89:0/32783631 conn(0x7f42000fd2e0 msgr2=0x7f4200106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.034+0000 7f4206e5e640 1 -- 172.21.15.89:0/32783631 shutdown_connections 2024-09-15T23:38:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:42.034+0000 7f4206e5e640 1 -- 172.21.15.89:0/32783631 wait complete. 2024-09-15T23:38:43.363 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:43 smithi089 ceph-mon[30933]: pgmap v790: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-15T23:38:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:43 smithi120 ceph-mon[35250]: pgmap v790: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-15T23:38:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:43 smithi174 ceph-mon[35586]: pgmap v790: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-09-15T23:38:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:45 smithi120 ceph-mon[35250]: pgmap v791: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-09-15T23:38:45.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:45 smithi174 ceph-mon[35586]: pgmap v791: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-09-15T23:38:45.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:45 smithi089 ceph-mon[30933]: pgmap v791: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-09-15T23:38:47.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:47 smithi120 ceph-mon[35250]: pgmap v792: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-09-15T23:38:47.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:47 smithi174 ceph-mon[35586]: pgmap v792: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-09-15T23:38:47.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:47 smithi089 ceph-mon[30933]: pgmap v792: 129 pgs: 129 active+clean; 3.3 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 4.6 KiB/s wr, 12 op/s 2024-09-15T23:38:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:49 smithi120 ceph-mon[35250]: pgmap v793: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 17 op/s 2024-09-15T23:38:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:49 smithi174 ceph-mon[35586]: pgmap v793: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 17 op/s 2024-09-15T23:38:49.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:49 smithi089 ceph-mon[30933]: pgmap v793: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 17 op/s 2024-09-15T23:38:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:51 smithi120 ceph-mon[35250]: pgmap v794: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-09-15T23:38:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:51 smithi174 ceph-mon[35586]: pgmap v794: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-09-15T23:38:51.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:51 smithi089 ceph-mon[30933]: pgmap v794: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-09-15T23:38:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:53 smithi120 ceph-mon[35250]: pgmap v795: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 16 op/s 2024-09-15T23:38:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:53 smithi174 ceph-mon[35586]: pgmap v795: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 16 op/s 2024-09-15T23:38:53.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:53 smithi089 ceph-mon[30933]: pgmap v795: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 16 op/s 2024-09-15T23:38:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.136+0000 7f3e364c3640 1 -- 172.21.15.89:0/3474659921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30100590 msgr2=0x7f3e3010a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.136+0000 7f3e364c3640 1 --2- 172.21.15.89:0/3474659921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30100590 0x7f3e3010a140 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009cd0 tx=0x7f3e2802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.136+0000 7f3e364c3640 1 -- 172.21.15.89:0/3474659921 shutdown_connections 2024-09-15T23:38:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.136+0000 7f3e364c3640 1 --2- 172.21.15.89:0/3474659921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30100590 0x7f3e3010a140 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.136+0000 7f3e364c3640 1 --2- 172.21.15.89:0/3474659921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e300ffbd0 0x7f3e30100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.136+0000 7f3e364c3640 1 --2- 172.21.15.89:0/3474659921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e30103160 0x7f3e30103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.136+0000 7f3e364c3640 1 -- 172.21.15.89:0/3474659921 >> 172.21.15.89:0/3474659921 conn(0x7f3e300fb980 msgr2=0x7f3e300fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.136+0000 7f3e364c3640 1 -- 172.21.15.89:0/3474659921 shutdown_connections 2024-09-15T23:38:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.136+0000 7f3e364c3640 1 -- 172.21.15.89:0/3474659921 wait complete. 2024-09-15T23:38:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.137+0000 7f3e364c3640 1 Processor -- start 2024-09-15T23:38:55.139 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.137+0000 7f3e364c3640 1 -- start start 2024-09-15T23:38:55.140 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.137+0000 7f3e364c3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e300ffbd0 0x7f3e30197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.137+0000 7f3e364c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e30100590 0x7f3e30198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.137+0000 7f3e364c3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e30103160 0x7f3e3019f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.137+0000 7f3e364c3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3010d800 con 0x7f3e300ffbd0 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.137+0000 7f3e364c3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3010d680 con 0x7f3e30103160 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.137+0000 7f3e364c3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3010d980 con 0x7f3e30100590 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e300ffbd0 0x7f3e30197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e300ffbd0 0x7f3e30197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41702/0 (socket says 172.21.15.89:41702) 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2ffff640 1 -- 172.21.15.89:0/3715756124 learned_addr learned my addr 172.21.15.89:0/3715756124 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2f7fe640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e30100590 0x7f3e30198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e34a60640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e30103160 0x7f3e3019f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2ffff640 1 -- 172.21.15.89:0/3715756124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e30100590 msgr2=0x7f3e30198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e30100590 0x7f3e30198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2ffff640 1 -- 172.21.15.89:0/3715756124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e30103160 msgr2=0x7f3e3019f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e30103160 0x7f3e3019f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2ffff640 1 -- 172.21.15.89:0/3715756124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3019f9e0 con 0x7f3e300ffbd0 2024-09-15T23:38:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2f7fe640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e30100590 0x7f3e30198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e34a60640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e30103160 0x7f3e3019f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e300ffbd0 0x7f3e30197dc0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000d770 tx=0x7f3e2000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2d7fa640 1 -- 172.21.15.89:0/3715756124 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20017070 con 0x7f3e300ffbd0 2024-09-15T23:38:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e364c3640 1 -- 172.21.15.89:0/3715756124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3019fcd0 con 0x7f3e300ffbd0 2024-09-15T23:38:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2d7fa640 1 -- 172.21.15.89:0/3715756124 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e20009c30 con 0x7f3e300ffbd0 2024-09-15T23:38:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e2d7fa640 1 -- 172.21.15.89:0/3715756124 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20003dd0 con 0x7f3e300ffbd0 2024-09-15T23:38:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.138+0000 7f3e364c3640 1 -- 172.21.15.89:0/3715756124 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e301a01c0 con 0x7f3e300ffbd0 2024-09-15T23:38:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.141+0000 7f3e2d7fa640 1 -- 172.21.15.89:0/3715756124 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3e20004000 con 0x7f3e300ffbd0 2024-09-15T23:38:55.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.141+0000 7f3e3925d740 1 Processor -- start 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.141+0000 7f3e3925d740 1 -- start start 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdacda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdb2370 0x55b8fbdb4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e3925d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b8fbd102f0 con 0x55b8fbdad2e0 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e3925d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b8fbcb8270 con 0x55b8fbdac9a0 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e3925d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b8fbd0c2b0 con 0x55b8fbdb2370 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2d7fa640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e180a32f0 0x7f3e180a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47306/0 (socket says 172.21.15.89:47306) 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2ffff640 1 -- 172.21.15.89:0/465574242 learned_addr learned my addr 172.21.15.89:0/465574242 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:55.149 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2f7fe640 1 --2- 172.21.15.89:0/465574242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdb2370 0x55b8fbdb4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e34a60640 1 --2- 172.21.15.89:0/465574242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdacda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e34a60640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e180a32f0 0x7f3e180a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2d7fa640 1 -- 172.21.15.89:0/3715756124 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3e200cf7a0 con 0x7f3e300ffbd0 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2ffff640 1 -- 172.21.15.89:0/465574242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdb2370 msgr2=0x55b8fbdb4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/465574242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdb2370 0x55b8fbdb4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2ffff640 1 -- 172.21.15.89:0/465574242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdac9a0 msgr2=0x55b8fbdacda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/465574242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdacda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2ffff640 1 -- 172.21.15.89:0/465574242 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8fbcb7340 con 0x55b8fbdad2e0 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/465574242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb1da0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f3e2006a000 tx=0x7f3e20000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9d1a22c7ca3bab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.142+0000 7f3e34a60640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e180a32f0 0x7f3e180a57b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3e28005e30 tx=0x7f3e280095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e167fc640 1 -- 172.21.15.89:0/465574242 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20020070 con 0x55b8fbdad2e0 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e167fc640 1 -- 172.21.15.89:0/465574242 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e200a45e0 con 0x55b8fbdad2e0 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e167fc640 1 -- 172.21.15.89:0/465574242 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e200ccf60 con 0x55b8fbdad2e0 2024-09-15T23:38:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e3925d740 1 -- 172.21.15.89:0/465574242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 msgr2=0x55b8fbdb1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e3925d740 1 --2- 172.21.15.89:0/465574242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb1da0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f3e2006a000 tx=0x7f3e20000bc0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e3925d740 1 -- 172.21.15.89:0/465574242 shutdown_connections 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e3925d740 1 --2- 172.21.15.89:0/465574242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdb2370 0x55b8fbdb4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e3925d740 1 --2- 172.21.15.89:0/465574242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb1da0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e3925d740 1 --2- 172.21.15.89:0/465574242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdacda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e3925d740 1 -- 172.21.15.89:0/465574242 >> 172.21.15.89:0/465574242 conn(0x55b8fbda8010 msgr2=0x55b8fbda8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e3925d740 1 -- 172.21.15.89:0/465574242 shutdown_connections 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.143+0000 7f3e3925d740 1 -- 172.21.15.89:0/465574242 wait complete. 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e3925d740 1 -- start start 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e3925d740 1 Processor -- start 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdb6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbdbe050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e3925d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b8fbdb5280 con 0x55b8fbdad2e0 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e3925d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b8fbc99290 con 0x55b8fbdb2370 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e3925d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b8fbdb5400 con 0x55b8fbdac9a0 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e3925d740 10 client.0 _ll_register_callbacks cb 0x55b8fbcca140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e2f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e2f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47314/0 (socket says 172.21.15.89:47314) 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e2f7fe640 1 -- 172.21.15.89:0/2811591212 learned_addr learned my addr 172.21.15.89:0/2811591212 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.144+0000 7f3e34a60640 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbdbe050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdb6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e2f7fe640 1 -- 172.21.15.89:0/2811591212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdac9a0 msgr2=0x55b8fbdb6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e2f7fe640 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdb6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e2f7fe640 1 -- 172.21.15.89:0/2811591212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 msgr2=0x55b8fbdbe050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e2f7fe640 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbdbe050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e2f7fe640 1 -- 172.21.15.89:0/2811591212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8fba23210 con 0x55b8fbdad2e0 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e2f7fe640 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb7010 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400bf70 tx=0x7f3e240128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03278a6a256c524 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/2811591212 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2401f060 con 0x55b8fbdad2e0 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e3925d740 1 -- 172.21.15.89:0/2811591212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b8fbd1da20 con 0x55b8fbdad2e0 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/2811591212 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3e24010960 con 0x55b8fbdad2e0 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/2811591212 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3e24026050 con 0x55b8fbdad2e0 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e3925d740 10 client.20532 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e3925d740 1 -- 172.21.15.89:0/2811591212 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b8fbaef620 con 0x55b8fbdad2e0 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.145+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/2811591212 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b8fbd1da20 con 0x55b8fbdad2e0 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.146+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/2811591212 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3e24019030 con 0x55b8fbdad2e0 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.146+0000 7f3e3925d740 10 client.20532 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.146+0000 7f3e3925d740 10 client.20532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.146+0000 7f3e3925d740 20 client.20532 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.146+0000 7f3e3925d740 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b8fbdc2840 0x55b8fbdc4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.146+0000 7f3e3925d740 4 client.20532 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.146+0000 7f3e3925d740 1 -- 172.21.15.89:0/2811591212 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8fbd17d70 con 0x55b8fbdc2840 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.146+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b8fbdc2840 0x55b8fbdc4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.146+0000 7f3e0f7fe640 10 client.20532 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.146+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b8fbdc2840 0x55b8fbdc4c90 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9988749cb520756a server_cookie=b003c9e7d16cd6cd in_seq=0 out_seq=0 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/2811591212 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b8fbd17d70 con 0x55b8fbdc2840 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e0f7fe640 10 client.20532 handle_command_reply: tid=0 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 client.20532 shutdown 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 -- 172.21.15.89:0/2811591212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 msgr2=0x55b8fbdb7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb7010 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400bf70 tx=0x7f3e240128e0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 -- 172.21.15.89:0/2811591212 shutdown_connections 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b8fbdc2840 0x55b8fbdc4c90 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbdbe050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdb7010 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e0f7fe640 0 client.20532 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 --2- 172.21.15.89:0/2811591212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdb6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 -- 172.21.15.89:0/2811591212 >> 172.21.15.89:0/2811591212 conn(0x55b8fbda8010 msgr2=0x55b8fbda9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 -- 172.21.15.89:0/2811591212 shutdown_connections 2024-09-15T23:38:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.147+0000 7f3e3925d740 1 -- 172.21.15.89:0/2811591212 wait complete. 2024-09-15T23:38:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.148+0000 7f3e3925d740 20 client.20532 trim_cache size 0 max 16384 2024-09-15T23:38:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e3925d740 1 Processor -- start 2024-09-15T23:38:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e3925d740 1 -- start start 2024-09-15T23:38:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbda70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdbe010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbda9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e3925d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301aa8b0 con 0x55b8fbdac9a0 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e3925d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301ac3f0 con 0x55b8fbdb2370 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e3925d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301aaf80 con 0x55b8fbdad2e0 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbda70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbda70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47328/0 (socket says 172.21.15.89:47328) 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e2ffff640 1 -- 172.21.15.89:0/767012463 learned_addr learned my addr 172.21.15.89:0/767012463 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e2ffff640 1 -- 172.21.15.89:0/767012463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdad2e0 msgr2=0x55b8fbdbe010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/767012463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdbe010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e2ffff640 1 -- 172.21.15.89:0/767012463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 msgr2=0x55b8fbda9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/767012463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbda9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e2ffff640 1 -- 172.21.15.89:0/767012463 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8fbdb4ce0 con 0x55b8fbdac9a0 2024-09-15T23:38:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/767012463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbda70b0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f3e200a47a0 tx=0x7f3e2000de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114cdcb8552a8a3b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e15ffb640 1 -- 172.21.15.89:0/767012463 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20020040 con 0x55b8fbdac9a0 2024-09-15T23:38:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e15ffb640 1 -- 172.21.15.89:0/767012463 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e2009c070 con 0x55b8fbdac9a0 2024-09-15T23:38:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.162+0000 7f3e15ffb640 1 -- 172.21.15.89:0/767012463 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e200cf420 con 0x55b8fbdac9a0 2024-09-15T23:38:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.163+0000 7f3e3925d740 1 -- 172.21.15.89:0/767012463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 msgr2=0x55b8fbda70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.163+0000 7f3e3925d740 1 --2- 172.21.15.89:0/767012463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbda70b0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f3e200a47a0 tx=0x7f3e2000de80 comp rx=0 tx=0).stop 2024-09-15T23:38:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.163+0000 7f3e3925d740 1 -- 172.21.15.89:0/767012463 shutdown_connections 2024-09-15T23:38:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.163+0000 7f3e3925d740 1 --2- 172.21.15.89:0/767012463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbda9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.163+0000 7f3e3925d740 1 --2- 172.21.15.89:0/767012463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdbe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.163+0000 7f3e3925d740 1 --2- 172.21.15.89:0/767012463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbda70b0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.163+0000 7f3e3925d740 1 -- 172.21.15.89:0/767012463 >> 172.21.15.89:0/767012463 conn(0x55b8fbda8010 msgr2=0x55b8fbda69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.163+0000 7f3e3925d740 1 -- 172.21.15.89:0/767012463 shutdown_connections 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.163+0000 7f3e3925d740 1 -- 172.21.15.89:0/767012463 wait complete. 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.163+0000 7f3e3925d740 1 -- start start 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e3925d740 1 Processor -- start 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdbb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdbb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e3925d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbdbc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e3925d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301abd20 con 0x55b8fbdac9a0 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e3925d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301ab650 con 0x55b8fbdb2370 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e3925d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301a8890 con 0x55b8fbdad2e0 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e3925d740 10 client.0 _ll_register_callbacks cb 0x7f3e2400b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdbb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdbb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47338/0 (socket says 172.21.15.89:47338) 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e2ffff640 1 -- 172.21.15.89:0/101446226 learned_addr learned my addr 172.21.15.89:0/101446226 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:55.170 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e34a60640 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbdbc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e2ffff640 1 -- 172.21.15.89:0/101446226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdad2e0 msgr2=0x55b8fbdbb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e2f7fe640 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdbb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdbb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e2ffff640 1 -- 172.21.15.89:0/101446226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 msgr2=0x55b8fbdbc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbdbc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e2ffff640 1 -- 172.21.15.89:0/101446226 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8fbdbcec0 con 0x55b8fbdac9a0 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e34a60640 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbdbc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e2ffff640 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdbb470 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f3e20099ec0 tx=0x7f3e20099d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e421b7f5b2e5899b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/101446226 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e200b9060 con 0x55b8fbdac9a0 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/101446226 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3e200cf420 con 0x55b8fbdac9a0 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.164+0000 7f3e3925d740 1 -- 172.21.15.89:0/101446226 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e2009c070 con 0x55b8fbdac9a0 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/101446226 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3e200b4070 con 0x55b8fbdac9a0 2024-09-15T23:38:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e3925d740 10 client.20538 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e3925d740 1 -- 172.21.15.89:0/101446226 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b8fbdbe1d0 con 0x55b8fbdac9a0 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/101446226 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e20066070 con 0x55b8fbdac9a0 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/101446226 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3e200cfb90 con 0x55b8fbdac9a0 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e3925d740 10 client.20538 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e3925d740 10 client.20538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e3925d740 20 client.20538 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e3925d740 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b8fbe2d6f0 0x55b8fbe2fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e3925d740 4 client.20538 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e3925d740 1 -- 172.21.15.89:0/101446226 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b8fbc94900 con 0x55b8fbe2d6f0 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.165+0000 7f3e2f7fe640 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b8fbe2d6f0 0x55b8fbe2fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.166+0000 7f3e0f7fe640 10 client.20538 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.166+0000 7f3e2f7fe640 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b8fbe2d6f0 0x55b8fbe2fae0 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ec8117e46cb387b server_cookie=db94239147092e48 in_seq=0 out_seq=0 2024-09-15T23:38:55.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e0f7fe640 1 -- 172.21.15.89:0/101446226 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b8fbc94900 con 0x55b8fbe2d6f0 2024-09-15T23:38:55.181 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e0f7fe640 10 client.20538 handle_command_reply: tid=0 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 client.20538 shutdown 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 -- 172.21.15.89:0/101446226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 msgr2=0x55b8fbdbb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdbb470 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f3e20099ec0 tx=0x7f3e20099d00 comp rx=0 tx=0).stop 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 -- 172.21.15.89:0/101446226 shutdown_connections 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55b8fbe2d6f0 0x55b8fbe2fae0 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8fbdb2370 0x55b8fbdbc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b8fbdad2e0 0x55b8fbdbb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 --2- 172.21.15.89:0/101446226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b8fbdac9a0 0x55b8fbdbb470 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e0f7fe640 0 client.20538 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 -- 172.21.15.89:0/101446226 >> 172.21.15.89:0/101446226 conn(0x55b8fbda8010 msgr2=0x55b8fbda71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 -- 172.21.15.89:0/101446226 shutdown_connections 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 1 -- 172.21.15.89:0/101446226 wait complete. 2024-09-15T23:38:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.180+0000 7f3e3925d740 20 client.20538 trim_cache size 0 max 16384 2024-09-15T23:38:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:38:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:38:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:38:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:38:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 -- 172.21.15.89:0/3715756124 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e180a32f0 msgr2=0x7f3e180a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e180a32f0 0x7f3e180a57b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3e28005e30 tx=0x7f3e280095c0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 -- 172.21.15.89:0/3715756124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e300ffbd0 msgr2=0x7f3e30197dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e300ffbd0 0x7f3e30197dc0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000d770 tx=0x7f3e2000dc30 comp rx=0 tx=0).stop 2024-09-15T23:38:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 -- 172.21.15.89:0/3715756124 shutdown_connections 2024-09-15T23:38:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3e180a32f0 0x7f3e180a57b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e30103160 0x7f3e3019f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e30100590 0x7f3e30198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 --2- 172.21.15.89:0/3715756124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e300ffbd0 0x7f3e30197dc0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 -- 172.21.15.89:0/3715756124 >> 172.21.15.89:0/3715756124 conn(0x7f3e300fb980 msgr2=0x7f3e301061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 -- 172.21.15.89:0/3715756124 shutdown_connections 2024-09-15T23:38:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:38:55.181+0000 7f3e364c3640 1 -- 172.21.15.89:0/3715756124 wait complete. 2024-09-15T23:38:55.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:55 smithi120 ceph-mon[35250]: pgmap v796: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 17 op/s 2024-09-15T23:38:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:55 smithi174 ceph-mon[35586]: pgmap v796: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 17 op/s 2024-09-15T23:38:55.542 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:38:55.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:55 smithi089 ceph-mon[30933]: pgmap v796: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 17 op/s 2024-09-15T23:38:55.807 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:56.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.305+0000 7f24a4754640 1 -- 172.21.15.89:0/2408803447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f249c102ef0 msgr2=0x7f249c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:56.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.305+0000 7f24a4754640 1 --2- 172.21.15.89:0/2408803447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f249c102ef0 0x7f249c10da40 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f2494009ca0 tx=0x7f249402a950 comp rx=0 tx=0).stop 2024-09-15T23:38:56.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.305+0000 7f24a4754640 1 -- 172.21.15.89:0/2408803447 shutdown_connections 2024-09-15T23:38:56.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.305+0000 7f24a4754640 1 --2- 172.21.15.89:0/2408803447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f249c102ef0 0x7f249c10da40 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.305+0000 7f24a4754640 1 --2- 172.21.15.89:0/2408803447 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f249c102570 0x7f249c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.305+0000 7f24a4754640 1 --2- 172.21.15.89:0/2408803447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c100e20 0x7f249c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.305+0000 7f24a4754640 1 -- 172.21.15.89:0/2408803447 >> 172.21.15.89:0/2408803447 conn(0x7f249c0fc7d0 msgr2=0x7f249c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.305+0000 7f24a4754640 1 -- 172.21.15.89:0/2408803447 shutdown_connections 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.305+0000 7f24a4754640 1 -- 172.21.15.89:0/2408803447 wait complete. 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.306+0000 7f24a4754640 1 Processor -- start 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.306+0000 7f24a4754640 1 -- start start 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.306+0000 7f24a4754640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f249c100e20 0x7f249c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.306+0000 7f24a4754640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c102570 0x7f249c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.306+0000 7f24a4754640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f249c102ef0 0x7f249c19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.306+0000 7f24a4754640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f249c111020 con 0x7f249c100e20 2024-09-15T23:38:56.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.306+0000 7f24a4754640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f249c110ea0 con 0x7f249c102570 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.306+0000 7f24a4754640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1111a0 con 0x7f249c102ef0 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.306+0000 7f24a24f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f249c100e20 0x7f249c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a24f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f249c100e20 0x7f249c1985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47360/0 (socket says 172.21.15.89:47360) 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a24f0640 1 -- 172.21.15.89:0/1005793842 learned_addr learned my addr 172.21.15.89:0/1005793842 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a2cf1640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f249c102ef0 0x7f249c19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a1cef640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c102570 0x7f249c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a24f0640 1 -- 172.21.15.89:0/1005793842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f249c102ef0 msgr2=0x7f249c19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a24f0640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f249c102ef0 0x7f249c19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a24f0640 1 -- 172.21.15.89:0/1005793842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c102570 msgr2=0x7f249c198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a24f0640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c102570 0x7f249c198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a24f0640 1 -- 172.21.15.89:0/1005793842 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c19d590 con 0x7f249c100e20 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a1cef640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c102570 0x7f249c198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a24f0640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f249c100e20 0x7f249c1985e0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f249800b4a0 tx=0x7f249800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:56.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24937fe640 1 -- 172.21.15.89:0/1005793842 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498014070 con 0x7f249c100e20 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a4754640 1 -- 172.21.15.89:0/1005793842 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f249c19d880 con 0x7f249c100e20 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24937fe640 1 -- 172.21.15.89:0/1005793842 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2498010af0 con 0x7f249c100e20 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24937fe640 1 -- 172.21.15.89:0/1005793842 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498017d10 con 0x7f249c100e20 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.307+0000 7f24a4754640 1 -- 172.21.15.89:0/1005793842 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f249c1a5160 con 0x7f249c100e20 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.309+0000 7f24937fe640 1 -- 172.21.15.89:0/1005793842 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2498017070 con 0x7f249c100e20 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.310+0000 7f24937fe640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24880a3240 0x7f24880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.310+0000 7f24a1cef640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24880a3240 0x7f24880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.310+0000 7f24937fe640 1 -- 172.21.15.89:0/1005793842 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f24980cf180 con 0x7f249c100e20 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.311+0000 7f24a1cef640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24880a3240 0x7f24880a5700 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f248c006fd0 tx=0x7f248c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.311+0000 7f24a6b2d740 1 Processor -- start 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.311+0000 7f24a6b2d740 1 -- start start 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.311+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41b4a0 0x55deda41b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.311+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda426140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.311+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda428a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.311+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55deda3812e0 con 0x55deda41bdc0 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.311+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55deda328ce0 con 0x55deda41b4a0 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.311+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55deda37d2a0 con 0x55deda426680 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a1cef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda426140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a1cef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda426140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47364/0 (socket says 172.21.15.89:47364) 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a1cef640 1 -- 172.21.15.89:0/1307174010 learned_addr learned my addr 172.21.15.89:0/1307174010 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a2cf1640 1 --2- 172.21.15.89:0/1307174010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda428a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a24f0640 1 --2- 172.21.15.89:0/1307174010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41b4a0 0x55deda41b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a1cef640 1 -- 172.21.15.89:0/1307174010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 msgr2=0x55deda428a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a1cef640 1 --2- 172.21.15.89:0/1307174010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda428a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a1cef640 1 -- 172.21.15.89:0/1307174010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41b4a0 msgr2=0x55deda41b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a1cef640 1 --2- 172.21.15.89:0/1307174010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41b4a0 0x55deda41b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a1cef640 1 -- 172.21.15.89:0/1307174010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55deda30d440 con 0x55deda41bdc0 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a24f0640 1 --2- 172.21.15.89:0/1307174010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41b4a0 0x55deda41b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f24a1cef640 1 --2- 172.21.15.89:0/1307174010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda426140 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f248c00ca50 tx=0x7f248c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acd3d5725b89379 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f2490ff9640 1 -- 172.21.15.89:0/1307174010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c023070 con 0x55deda41bdc0 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f2490ff9640 1 -- 172.21.15.89:0/1307174010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f248c013db0 con 0x55deda41bdc0 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.312+0000 7f2490ff9640 1 -- 172.21.15.89:0/1307174010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c011760 con 0x55deda41bdc0 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.313+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1307174010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 msgr2=0x55deda426140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.313+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/1307174010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda426140 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f248c00ca50 tx=0x7f248c00f000 comp rx=0 tx=0).stop 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.313+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1307174010 shutdown_connections 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.313+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/1307174010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda428a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.313+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/1307174010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda426140 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.313+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/1307174010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41b4a0 0x55deda41b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.313+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1307174010 >> 172.21.15.89:0/1307174010 conn(0x55deda417a30 msgr2=0x55deda417e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:56.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.313+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1307174010 shutdown_connections 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.313+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1307174010 wait complete. 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a6b2d740 1 -- start start 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a6b2d740 1 Processor -- start 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda425ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41bdc0 0x55deda4232a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda423800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55deda429250 con 0x55deda41b4a0 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55deda33c050 con 0x55deda41bdc0 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55deda4293d0 con 0x55deda426680 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a6b2d740 10 client.0 _ll_register_callbacks cb 0x55deda37c160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a24f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda425ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a24f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda425ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47380/0 (socket says 172.21.15.89:47380) 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a24f0640 1 -- 172.21.15.89:0/1507074670 learned_addr learned my addr 172.21.15.89:0/1507074670 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:56.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a1cef640 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41bdc0 0x55deda4232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a2cf1640 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda423800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a24f0640 1 -- 172.21.15.89:0/1507074670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 msgr2=0x55deda423800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a24f0640 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda423800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a24f0640 1 -- 172.21.15.89:0/1507074670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41bdc0 msgr2=0x55deda4232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a24f0640 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41bdc0 0x55deda4232a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.314+0000 7f24a24f0640 1 -- 172.21.15.89:0/1507074670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55deda0f23c0 con 0x55deda41b4a0 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f24a24f0640 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda425ff0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f2498097d80 tx=0x7f249809add0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc91effd30250ee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f2485ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f2485ffb640 1 -- 172.21.15.89:0/1507074670 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249801e070 con 0x55deda41b4a0 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f2485ffb640 1 -- 172.21.15.89:0/1507074670 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2498010ca0 con 0x55deda41b4a0 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1507074670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f248c013db0 con 0x55deda41b4a0 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f2485ffb640 1 -- 172.21.15.89:0/1507074670 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f249809f760 con 0x55deda41b4a0 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f24a6b2d740 10 client.20550 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:56.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1507074670 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55deda434c20 con 0x55deda41b4a0 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f2485ffb640 1 -- 172.21.15.89:0/1507074670 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f248c013db0 con 0x55deda41b4a0 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f2485ffb640 1 -- 172.21.15.89:0/1507074670 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f249809a800 con 0x55deda41b4a0 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f24a6b2d740 10 client.20550 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f24a6b2d740 10 client.20550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f24a6b2d740 20 client.20550 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55deda436030 0x55deda438460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f24a6b2d740 4 client.20550 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.315+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1507074670 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55deda38e850 con 0x55deda436030 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.316+0000 7f24a1cef640 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55deda436030 0x55deda438460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.316+0000 7f2485ffb640 10 client.20550 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.316+0000 7f24a1cef640 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55deda436030 0x55deda438460 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=282fbd8b8bcbd958 server_cookie=2efaf99182424c0c in_seq=0 out_seq=0 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f2485ffb640 1 -- 172.21.15.89:0/1507074670 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55deda38e850 con 0x55deda436030 2024-09-15T23:38:56.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f2485ffb640 10 client.20550 handle_command_reply: tid=0 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 client.20550 shutdown 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1507074670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 msgr2=0x55deda425ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda425ff0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f2498097d80 tx=0x7f249809add0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1507074670 shutdown_connections 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55deda436030 0x55deda438460 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda423800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41bdc0 0x55deda4232a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/1507074670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda425ff0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1507074670 >> 172.21.15.89:0/1507074670 conn(0x55deda417a30 msgr2=0x55deda416b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f2485ffb640 0 client.20550 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1507074670 shutdown_connections 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 1 -- 172.21.15.89:0/1507074670 wait complete. 2024-09-15T23:38:56.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.317+0000 7f24a6b2d740 20 client.20550 trim_cache size 0 max 16384 2024-09-15T23:38:56.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a6b2d740 1 Processor -- start 2024-09-15T23:38:56.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a6b2d740 1 -- start start 2024-09-15T23:38:56.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda4324b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41bdc0 0x55deda41e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda43dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1aef70 con 0x55deda41b4a0 2024-09-15T23:38:56.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1b0ab0 con 0x55deda41bdc0 2024-09-15T23:38:56.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1af640 con 0x55deda426680 2024-09-15T23:38:56.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a24f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda4324b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a24f0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda4324b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47394/0 (socket says 172.21.15.89:47394) 2024-09-15T23:38:56.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a24f0640 1 -- 172.21.15.89:0/184925689 learned_addr learned my addr 172.21.15.89:0/184925689 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:56.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a1cef640 1 --2- 172.21.15.89:0/184925689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41bdc0 0x55deda41e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a2cf1640 1 --2- 172.21.15.89:0/184925689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda43dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a24f0640 1 -- 172.21.15.89:0/184925689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 msgr2=0x55deda43dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:56.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a24f0640 1 --2- 172.21.15.89:0/184925689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda43dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a24f0640 1 -- 172.21.15.89:0/184925689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41bdc0 msgr2=0x55deda41e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:56.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a24f0640 1 --2- 172.21.15.89:0/184925689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41bdc0 0x55deda41e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.335+0000 7f24a24f0640 1 -- 172.21.15.89:0/184925689 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55deda422250 con 0x55deda41b4a0 2024-09-15T23:38:56.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.336+0000 7f24a24f0640 1 --2- 172.21.15.89:0/184925689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda4324b0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f24980ccc00 tx=0x7f2498097050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f439b0d81e2f7316 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:56.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.336+0000 7f2487fff640 1 -- 172.21.15.89:0/184925689 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249801e070 con 0x55deda41b4a0 2024-09-15T23:38:56.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.336+0000 7f2487fff640 1 -- 172.21.15.89:0/184925689 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24980a4020 con 0x55deda41b4a0 2024-09-15T23:38:56.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.336+0000 7f2487fff640 1 -- 172.21.15.89:0/184925689 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498097b30 con 0x55deda41b4a0 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 -- 172.21.15.89:0/184925689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 msgr2=0x55deda4324b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/184925689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda4324b0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f24980ccc00 tx=0x7f2498097050 comp rx=0 tx=0).stop 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 -- 172.21.15.89:0/184925689 shutdown_connections 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/184925689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda426680 0x55deda43dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/184925689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda41bdc0 0x55deda41e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/184925689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41b4a0 0x55deda4324b0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 -- 172.21.15.89:0/184925689 >> 172.21.15.89:0/184925689 conn(0x55deda417a30 msgr2=0x55deda41e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 -- 172.21.15.89:0/184925689 shutdown_connections 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 -- 172.21.15.89:0/184925689 wait complete. 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 -- start start 2024-09-15T23:38:56.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 Processor -- start 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda41b4a0 0x55deda42ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda420190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda426680 0x55deda420920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1b03e0 con 0x55deda41bdc0 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1afd10 con 0x55deda426680 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1acf50 con 0x55deda41b4a0 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a1cef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda420190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.337+0000 7f24a6b2d740 10 client.0 _ll_register_callbacks cb 0x7f24980092b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a1cef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda420190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47406/0 (socket says 172.21.15.89:47406) 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a1cef640 1 -- 172.21.15.89:0/2235437925 learned_addr learned my addr 172.21.15.89:0/2235437925 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:56.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a24f0640 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda41b4a0 0x55deda42ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a2cf1640 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda426680 0x55deda420920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a1cef640 1 -- 172.21.15.89:0/2235437925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda41b4a0 msgr2=0x55deda42ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a1cef640 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda41b4a0 0x55deda42ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a1cef640 1 -- 172.21.15.89:0/2235437925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda426680 msgr2=0x55deda420920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a1cef640 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda426680 0x55deda420920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a1cef640 1 -- 172.21.15.89:0/2235437925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55deda423b70 con 0x55deda41bdc0 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a2cf1640 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda426680 0x55deda420920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a1cef640 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda420190 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f248c00ca20 tx=0x7f248c01ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfdecb4b78713853 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f2485ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f2485ffb640 1 -- 172.21.15.89:0/2235437925 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c035070 con 0x55deda41bdc0 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a6b2d740 1 -- 172.21.15.89:0/2235437925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24980a4020 con 0x55deda41bdc0 2024-09-15T23:38:56.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f2485ffb640 1 -- 172.21.15.89:0/2235437925 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f248c01e600 con 0x55deda41bdc0 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f2485ffb640 1 -- 172.21.15.89:0/2235437925 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f248c043050 con 0x55deda41bdc0 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a6b2d740 10 client.20556 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.338+0000 7f24a6b2d740 1 -- 172.21.15.89:0/2235437925 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55deda4a0d40 con 0x55deda41bdc0 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.339+0000 7f2485ffb640 1 -- 172.21.15.89:0/2235437925 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24980a4020 con 0x55deda41bdc0 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.339+0000 7f2485ffb640 1 -- 172.21.15.89:0/2235437925 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f248c03ab70 con 0x55deda41bdc0 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.339+0000 7f24a6b2d740 10 client.20556 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.339+0000 7f24a6b2d740 10 client.20556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.339+0000 7f24a6b2d740 20 client.20556 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.339+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55deda4a1190 0x55deda4a3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.339+0000 7f24a6b2d740 4 client.20556 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:38:56.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.339+0000 7f24a6b2d740 1 -- 172.21.15.89:0/2235437925 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55deda388d60 con 0x55deda4a1190 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.339+0000 7f24a24f0640 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55deda4a1190 0x55deda4a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a24f0640 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55deda4a1190 0x55deda4a3580 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d648a6171e0447c server_cookie=ada52099316c7688 in_seq=0 out_seq=0 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f2485ffb640 10 client.20556 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f2485ffb640 1 -- 172.21.15.89:0/2235437925 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55deda388d60 con 0x55deda4a1190 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f2485ffb640 10 client.20556 handle_command_reply: tid=0 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 client.20556 shutdown 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 -- 172.21.15.89:0/2235437925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 msgr2=0x55deda420190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda420190 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f248c00ca20 tx=0x7f248c01ee40 comp rx=0 tx=0).stop 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 -- 172.21.15.89:0/2235437925 shutdown_connections 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55deda4a1190 0x55deda4a3580 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55deda426680 0x55deda420920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55deda41bdc0 0x55deda420190 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 --2- 172.21.15.89:0/2235437925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55deda41b4a0 0x55deda42ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f2485ffb640 0 client.20556 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:38:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 -- 172.21.15.89:0/2235437925 >> 172.21.15.89:0/2235437925 conn(0x55deda417a30 msgr2=0x55deda41d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 -- 172.21.15.89:0/2235437925 shutdown_connections 2024-09-15T23:38:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 1 -- 172.21.15.89:0/2235437925 wait complete. 2024-09-15T23:38:56.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.340+0000 7f24a6b2d740 20 client.20556 trim_cache size 0 max 16384 2024-09-15T23:38:56.348 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:38:56.348 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:38:56.349 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:38:56.349 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:38:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 -- 172.21.15.89:0/1005793842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24880a3240 msgr2=0x7f24880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:56.349 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24880a3240 0x7f24880a5700 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f248c006fd0 tx=0x7f248c008040 comp rx=0 tx=0).stop 2024-09-15T23:38:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 -- 172.21.15.89:0/1005793842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f249c100e20 msgr2=0x7f249c1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f249c100e20 0x7f249c1985e0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f249800b4a0 tx=0x7f249800b960 comp rx=0 tx=0).stop 2024-09-15T23:38:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 -- 172.21.15.89:0/1005793842 shutdown_connections 2024-09-15T23:38:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f24880a3240 0x7f24880a5700 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f249c102ef0 0x7f249c19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c102570 0x7f249c198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 --2- 172.21.15.89:0/1005793842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f249c100e20 0x7f249c1985e0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 -- 172.21.15.89:0/1005793842 >> 172.21.15.89:0/1005793842 conn(0x7f249c0fc7d0 msgr2=0x7f249c06a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 -- 172.21.15.89:0/1005793842 shutdown_connections 2024-09-15T23:38:56.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:56.341+0000 7f24a4754640 1 -- 172.21.15.89:0/1005793842 wait complete. 2024-09-15T23:38:57.183 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:38:57.183 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:38:57.247 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:57 smithi089 ceph-mon[30933]: pgmap v797: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:38:57.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:57 smithi120 ceph-mon[35250]: pgmap v797: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:38:57.475 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:57 smithi174 ceph-mon[35586]: pgmap v797: 129 pgs: 129 active+clean; 3.4 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:38:57.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.986+0000 7f61e3091640 1 -- 172.21.15.89:0/3145961850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100a80 msgr2=0x7f61dc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:57.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.986+0000 7f61e3091640 1 --2- 172.21.15.89:0/3145961850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100a80 0x7f61dc100ee0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f61c8009930 tx=0x7f61c802a420 comp rx=0 tx=0).stop 2024-09-15T23:38:57.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.986+0000 7f61e3091640 1 -- 172.21.15.89:0/3145961850 shutdown_connections 2024-09-15T23:38:57.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.986+0000 7f61e3091640 1 --2- 172.21.15.89:0/3145961850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc101420 0x7f61dc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:57.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.986+0000 7f61e3091640 1 --2- 172.21.15.89:0/3145961850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100a80 0x7f61dc100ee0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:57.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.986+0000 7f61e3091640 1 --2- 172.21.15.89:0/3145961850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc106a80 0x7f61dc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:57.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.986+0000 7f61e3091640 1 -- 172.21.15.89:0/3145961850 >> 172.21.15.89:0/3145961850 conn(0x7f61dc0fc7a0 msgr2=0x7f61dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:57.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.986+0000 7f61e3091640 1 -- 172.21.15.89:0/3145961850 shutdown_connections 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.986+0000 7f61e3091640 1 -- 172.21.15.89:0/3145961850 wait complete. 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e3091640 1 Processor -- start 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e3091640 1 -- start start 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e3091640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100a80 0x7f61dc0fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e3091640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc101420 0x7f61dc0fd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e3091640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc106a80 0x7f61dc0fd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e3091640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc110e80 con 0x7f61dc100a80 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e3091640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc110d00 con 0x7f61dc106a80 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e3091640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc111000 con 0x7f61dc101420 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e0e2d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100a80 0x7f61dc0fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e0e2d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100a80 0x7f61dc0fcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47418/0 (socket says 172.21.15.89:47418) 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e0e2d640 1 -- 172.21.15.89:0/3504636774 learned_addr learned my addr 172.21.15.89:0/3504636774 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61dbfff640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc101420 0x7f61dc0fd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:57.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e162e640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc106a80 0x7f61dc0fd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e0e2d640 1 -- 172.21.15.89:0/3504636774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc101420 msgr2=0x7f61dc0fd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e0e2d640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc101420 0x7f61dc0fd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e0e2d640 1 -- 172.21.15.89:0/3504636774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc106a80 msgr2=0x7f61dc0fd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e0e2d640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc106a80 0x7f61dc0fd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e0e2d640 1 -- 172.21.15.89:0/3504636774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61dc1a6900 con 0x7f61dc100a80 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61dbfff640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc101420 0x7f61dc0fd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61e0e2d640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100a80 0x7f61dc0fcd90 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f61d000d950 tx=0x7f61d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.987+0000 7f61d9ffb640 1 -- 172.21.15.89:0/3504636774 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d0017070 con 0x7f61dc100a80 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.988+0000 7f61d9ffb640 1 -- 172.21.15.89:0/3504636774 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61d0009c30 con 0x7f61dc100a80 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.988+0000 7f61e3091640 1 -- 172.21.15.89:0/3504636774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61dc1a6bf0 con 0x7f61dc100a80 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.988+0000 7f61d9ffb640 1 -- 172.21.15.89:0/3504636774 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d000f6b0 con 0x7f61dc100a80 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.988+0000 7f61e3091640 1 -- 172.21.15.89:0/3504636774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61dc1a7100 con 0x7f61dc100a80 2024-09-15T23:38:57.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.989+0000 7f61e3091640 1 -- 172.21.15.89:0/3504636774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61dc100ee0 con 0x7f61dc100a80 2024-09-15T23:38:57.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.990+0000 7f61d9ffb640 1 -- 172.21.15.89:0/3504636774 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f61d00109d0 con 0x7f61dc100a80 2024-09-15T23:38:57.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.991+0000 7f61d9ffb640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61cc0a3240 0x7f61cc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:57.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.991+0000 7f61dbfff640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61cc0a3240 0x7f61cc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:57.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.991+0000 7f61d9ffb640 1 -- 172.21.15.89:0/3504636774 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f61d00cf950 con 0x7f61dc100a80 2024-09-15T23:38:57.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.991+0000 7f61dbfff640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61cc0a3240 0x7f61cc0a5700 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f61c8000c00 tx=0x7f61c8005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:57.996 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:57.995+0000 7f61d9ffb640 1 -- 172.21.15.89:0/3504636774 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f61d0009de0 con 0x7f61dc100a80 2024-09-15T23:38:58.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.197+0000 7f61e3091640 1 -- 172.21.15.89:0/3504636774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61dc109a70 con 0x7f61dc100a80 2024-09-15T23:38:58.199 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.197+0000 7f61d9ffb640 1 -- 172.21.15.89:0/3504636774 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f61d009c280 con 0x7f61dc100a80 2024-09-15T23:38:58.199 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:38:58.199 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:38:58.199 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:38:58.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 -- 172.21.15.89:0/3504636774 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61cc0a3240 msgr2=0x7f61cc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:58.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61cc0a3240 0x7f61cc0a5700 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f61c8000c00 tx=0x7f61c8005a20 comp rx=0 tx=0).stop 2024-09-15T23:38:58.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 -- 172.21.15.89:0/3504636774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100a80 msgr2=0x7f61dc0fcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:58.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100a80 0x7f61dc0fcd90 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f61d000d950 tx=0x7f61d000de10 comp rx=0 tx=0).stop 2024-09-15T23:38:58.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 -- 172.21.15.89:0/3504636774 shutdown_connections 2024-09-15T23:38:58.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f61cc0a3240 0x7f61cc0a5700 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:58.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f61dc106a80 0x7f61dc0fd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:58.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61dc101420 0x7f61dc0fd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:58.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 --2- 172.21.15.89:0/3504636774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f61dc100a80 0x7f61dc0fcd90 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:58.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 -- 172.21.15.89:0/3504636774 >> 172.21.15.89:0/3504636774 conn(0x7f61dc0fc7a0 msgr2=0x7f61dc06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:58.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 -- 172.21.15.89:0/3504636774 shutdown_connections 2024-09-15T23:38:58.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:58.200+0000 7f61e3091640 1 -- 172.21.15.89:0/3504636774 wait complete. 2024-09-15T23:38:59.021 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:38:59.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:59 smithi089 ceph-mon[30933]: pgmap v798: 129 pgs: 129 active+clean; 3.5 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 19 op/s 2024-09-15T23:38:59.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:38:59 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3504636774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:59.317 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:38:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:59 smithi120 ceph-mon[35250]: pgmap v798: 129 pgs: 129 active+clean; 3.5 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 19 op/s 2024-09-15T23:38:59.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:38:59 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3504636774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:59.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:59 smithi174 ceph-mon[35586]: pgmap v798: 129 pgs: 129 active+clean; 3.5 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 19 op/s 2024-09-15T23:38:59.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:38:59 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3504636774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:38:59.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 -- 172.21.15.89:0/144906369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f434c1068d0 msgr2=0x7f434c106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:59.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 --2- 172.21.15.89:0/144906369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f434c1068d0 0x7f434c106cb0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f433c009930 tx=0x7f433c02a420 comp rx=0 tx=0).stop 2024-09-15T23:38:59.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 -- 172.21.15.89:0/144906369 shutdown_connections 2024-09-15T23:38:59.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 --2- 172.21.15.89:0/144906369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f434c101270 0x7f434c10d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 --2- 172.21.15.89:0/144906369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f434c1008d0 0x7f434c100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 --2- 172.21.15.89:0/144906369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f434c1068d0 0x7f434c106cb0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 -- 172.21.15.89:0/144906369 >> 172.21.15.89:0/144906369 conn(0x7f434c0fc610 msgr2=0x7f434c0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:59.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 -- 172.21.15.89:0/144906369 shutdown_connections 2024-09-15T23:38:59.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 -- 172.21.15.89:0/144906369 wait complete. 2024-09-15T23:38:59.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 Processor -- start 2024-09-15T23:38:59.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.790+0000 7f43513a1640 1 -- start start 2024-09-15T23:38:59.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f43513a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f434c1008d0 0x7f434c197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f43513a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f434c101270 0x7f434c197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f43513a1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f434c1068d0 0x7f434c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f43513a1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f434c110cd0 con 0x7f434c1008d0 2024-09-15T23:38:59.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f43513a1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f434c110b50 con 0x7f434c1068d0 2024-09-15T23:38:59.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f43513a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f434c110e50 con 0x7f434c101270 2024-09-15T23:38:59.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f434c1008d0 0x7f434c197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f434c1008d0 0x7f434c197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47442/0 (socket says 172.21.15.89:47442) 2024-09-15T23:38:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434bfff640 1 -- 172.21.15.89:0/2319935516 learned_addr learned my addr 172.21.15.89:0/2319935516 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434bfff640 1 -- 172.21.15.89:0/2319935516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f434c101270 msgr2=0x7f434c197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434b7fe640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f434c101270 0x7f434c197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f4350ba0640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f434c1068d0 0x7f434c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434bfff640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f434c101270 0x7f434c197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434bfff640 1 -- 172.21.15.89:0/2319935516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f434c1068d0 msgr2=0x7f434c19ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:59.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434bfff640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f434c1068d0 0x7f434c19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434bfff640 1 -- 172.21.15.89:0/2319935516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f434c19c100 con 0x7f434c1008d0 2024-09-15T23:38:59.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434b7fe640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f434c101270 0x7f434c197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:59.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.791+0000 7f434bfff640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f434c1008d0 0x7f434c197150 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f433c009a50 tx=0x7f433c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:59.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.792+0000 7f43497fa640 1 -- 172.21.15.89:0/2319935516 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c046070 con 0x7f434c1008d0 2024-09-15T23:38:59.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.792+0000 7f43513a1640 1 -- 172.21.15.89:0/2319935516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434c19c390 con 0x7f434c1008d0 2024-09-15T23:38:59.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.792+0000 7f43497fa640 1 -- 172.21.15.89:0/2319935516 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f433c0044a0 con 0x7f434c1008d0 2024-09-15T23:38:59.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.792+0000 7f43497fa640 1 -- 172.21.15.89:0/2319935516 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433c035050 con 0x7f434c1008d0 2024-09-15T23:38:59.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.792+0000 7f43513a1640 1 -- 172.21.15.89:0/2319935516 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f434c069f70 con 0x7f434c1008d0 2024-09-15T23:38:59.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.794+0000 7f43497fa640 1 -- 172.21.15.89:0/2319935516 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f433c049070 con 0x7f434c1008d0 2024-09-15T23:38:59.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.794+0000 7f43549dc740 1 Processor -- start 2024-09-15T23:38:59.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.794+0000 7f43549dc740 1 -- start start 2024-09-15T23:38:59.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.794+0000 7f43549dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b62052b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f43549dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b62052bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f43549dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620538940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f43549dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b6204912f0 con 0x55b620536550 2024-09-15T23:38:59.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f43549dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b620438cf0 con 0x55b62052cd20 2024-09-15T23:38:59.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f43549dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b62048d2b0 con 0x55b62052b7b0 2024-09-15T23:38:59.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620538940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f43497fa640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43340a3240 0x7f43340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620538940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47450/0 (socket says 172.21.15.89:47450) 2024-09-15T23:38:59.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434b7fe640 1 -- 172.21.15.89:0/3124165998 learned_addr learned my addr 172.21.15.89:0/3124165998 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:59.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f4350ba0640 1 --2- 172.21.15.89:0/3124165998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b62052b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434bfff640 1 --2- 172.21.15.89:0/3124165998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b62052bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f4350ba0640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43340a3240 0x7f43340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434b7fe640 1 -- 172.21.15.89:0/3124165998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 msgr2=0x55b62052bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:59.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f43497fa640 1 -- 172.21.15.89:0/2319935516 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f433c0f19f0 con 0x7f434c1008d0 2024-09-15T23:38:59.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434b7fe640 1 --2- 172.21.15.89:0/3124165998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b62052bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434b7fe640 1 -- 172.21.15.89:0/3124165998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 msgr2=0x55b62052b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:59.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434b7fe640 1 --2- 172.21.15.89:0/3124165998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b62052b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434b7fe640 1 -- 172.21.15.89:0/3124165998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b62041d3a0 con 0x55b620536550 2024-09-15T23:38:59.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434bfff640 1 --2- 172.21.15.89:0/3124165998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b62052bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:59.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f434b7fe640 1 --2- 172.21.15.89:0/3124165998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620538940 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f434000bbe0 tx=0x7f434000d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12f75f2a116863f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:59.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f4350ba0640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43340a3240 0x7f43340a5700 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f434400af40 tx=0x7f4344005c80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:59.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f43327fc640 1 -- 172.21.15.89:0/3124165998 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4340017620 con 0x55b620536550 2024-09-15T23:38:59.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f43327fc640 1 -- 172.21.15.89:0/3124165998 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4340017ca0 con 0x55b620536550 2024-09-15T23:38:59.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.795+0000 7f43327fc640 1 -- 172.21.15.89:0/3124165998 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434000fd90 con 0x55b620536550 2024-09-15T23:38:59.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.796+0000 7f43549dc740 1 -- 172.21.15.89:0/3124165998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 msgr2=0x55b620538940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:59.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.796+0000 7f43549dc740 1 --2- 172.21.15.89:0/3124165998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620538940 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f434000bbe0 tx=0x7f434000d710 comp rx=0 tx=0).stop 2024-09-15T23:38:59.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.796+0000 7f43549dc740 1 -- 172.21.15.89:0/3124165998 shutdown_connections 2024-09-15T23:38:59.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.796+0000 7f43549dc740 1 --2- 172.21.15.89:0/3124165998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620538940 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.796+0000 7f43549dc740 1 --2- 172.21.15.89:0/3124165998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b62052bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.796+0000 7f43549dc740 1 --2- 172.21.15.89:0/3124165998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b62052b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.796+0000 7f43549dc740 1 -- 172.21.15.89:0/3124165998 >> 172.21.15.89:0/3124165998 conn(0x55b6204f4310 msgr2=0x55b620527e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:59.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.796+0000 7f43549dc740 1 -- 172.21.15.89:0/3124165998 shutdown_connections 2024-09-15T23:38:59.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.796+0000 7f43549dc740 1 -- 172.21.15.89:0/3124165998 wait complete. 2024-09-15T23:38:59.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.796+0000 7f43549dc740 1 -- start start 2024-09-15T23:38:59.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f43549dc740 1 Processor -- start 2024-09-15T23:38:59.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f43549dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b620535e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f43549dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b620533120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f43549dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620533660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f43549dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b620539250 con 0x55b620536550 2024-09-15T23:38:59.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f43549dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b62044c060 con 0x55b62052cd20 2024-09-15T23:38:59.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f43549dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b6205393d0 con 0x55b62052b7b0 2024-09-15T23:38:59.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f43549dc740 10 client.0 _ll_register_callbacks cb 0x55b62019f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:59.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f4350ba0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620533660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f4350ba0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620533660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47456/0 (socket says 172.21.15.89:47456) 2024-09-15T23:38:59.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f4350ba0640 1 -- 172.21.15.89:0/3193600776 learned_addr learned my addr 172.21.15.89:0/3193600776 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:59.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f434b7fe640 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b620533120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f434bfff640 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b620535e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f4350ba0640 1 -- 172.21.15.89:0/3193600776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 msgr2=0x55b620535e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:59.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f4350ba0640 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b620535e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.837 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f4350ba0640 1 -- 172.21.15.89:0/3193600776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 msgr2=0x55b620533120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:59.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f4350ba0640 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b620533120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f4350ba0640 1 -- 172.21.15.89:0/3193600776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6202023c0 con 0x55b620536550 2024-09-15T23:38:59.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.797+0000 7f434bfff640 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b620535e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:59.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f4350ba0640 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620533660 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f434400afd0 tx=0x7f4344004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e351421c4afca0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:59.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f432b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:59.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f432b7fe640 1 -- 172.21.15.89:0/3193600776 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434400ddf0 con 0x55b620536550 2024-09-15T23:38:59.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f43549dc740 1 -- 172.21.15.89:0/3193600776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4340017ca0 con 0x55b620536550 2024-09-15T23:38:59.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f432b7fe640 1 -- 172.21.15.89:0/3193600776 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f434401d410 con 0x55b620536550 2024-09-15T23:38:59.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f432b7fe640 1 -- 172.21.15.89:0/3193600776 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f434402e050 con 0x55b620536550 2024-09-15T23:38:59.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f43549dc740 10 client.20574 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:59.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f43549dc740 1 -- 172.21.15.89:0/3193600776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b620544c30 con 0x55b620536550 2024-09-15T23:38:59.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f432b7fe640 1 -- 172.21.15.89:0/3193600776 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4340017ca0 con 0x55b620536550 2024-09-15T23:38:59.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f432b7fe640 1 -- 172.21.15.89:0/3193600776 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4344019070 con 0x55b620536550 2024-09-15T23:38:59.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f43549dc740 10 client.20574 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:59.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f43549dc740 10 client.20574 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:38:59.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f43549dc740 20 client.20574 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:59.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f43549dc740 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55b6205460f0 0x55b620548540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f43549dc740 4 client.20574 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:38:59.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.798+0000 7f43549dc740 1 -- 172.21.15.89:0/3193600776 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b62049e860 con 0x55b6205460f0 2024-09-15T23:38:59.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.799+0000 7f434bfff640 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55b6205460f0 0x55b620548540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.799+0000 7f434bfff640 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55b6205460f0 0x55b620548540 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1e93891655cc8505 server_cookie=3e0dad5d6fabb87a in_seq=0 out_seq=0 2024-09-15T23:38:59.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.799+0000 7f432b7fe640 10 client.20574 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:38:59.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f432b7fe640 1 -- 172.21.15.89:0/3193600776 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b62049e860 con 0x55b6205460f0 2024-09-15T23:38:59.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f432b7fe640 10 client.20574 handle_command_reply: tid=0 2024-09-15T23:38:59.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 client.20574 shutdown 2024-09-15T23:38:59.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 -- 172.21.15.89:0/3193600776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 msgr2=0x55b620533660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:59.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620533660 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f434400afd0 tx=0x7f4344004a40 comp rx=0 tx=0).stop 2024-09-15T23:38:59.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 -- 172.21.15.89:0/3193600776 shutdown_connections 2024-09-15T23:38:59.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55b6205460f0 0x55b620548540 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b620533660 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b620533120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 --2- 172.21.15.89:0/3193600776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b620535e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f432b7fe640 0 client.20574 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:38:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 -- 172.21.15.89:0/3193600776 >> 172.21.15.89:0/3193600776 conn(0x55b6204f4310 msgr2=0x55b62052db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 -- 172.21.15.89:0/3193600776 shutdown_connections 2024-09-15T23:38:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 1 -- 172.21.15.89:0/3193600776 wait complete. 2024-09-15T23:38:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.800+0000 7f43549dc740 20 client.20574 trim_cache size 0 max 16384 2024-09-15T23:38:59.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f43549dc740 1 Processor -- start 2024-09-15T23:38:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f43549dc740 1 -- start start 2024-09-15T23:38:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f43549dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b6205424f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f43549dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b62052cd20 0x55b62052e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f43549dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b620536550 0x55b62052e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f43549dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1ac9a0 con 0x55b62052cd20 2024-09-15T23:38:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f43549dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1ae4e0 con 0x55b620536550 2024-09-15T23:38:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f43549dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1ad070 con 0x55b62052b7b0 2024-09-15T23:38:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f434b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b62052cd20 0x55b62052e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f434b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b62052cd20 0x55b62052e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47466/0 (socket says 172.21.15.89:47466) 2024-09-15T23:38:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f434b7fe640 1 -- 172.21.15.89:0/2147761953 learned_addr learned my addr 172.21.15.89:0/2147761953 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f4350ba0640 1 --2- 172.21.15.89:0/2147761953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b620536550 0x55b62052e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f434bfff640 1 --2- 172.21.15.89:0/2147761953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b6205424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f434b7fe640 1 -- 172.21.15.89:0/2147761953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 msgr2=0x55b6205424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f434b7fe640 1 --2- 172.21.15.89:0/2147761953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b6205424f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f434b7fe640 1 -- 172.21.15.89:0/2147761953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b620536550 msgr2=0x55b62052e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f434b7fe640 1 --2- 172.21.15.89:0/2147761953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b620536550 0x55b62052e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f434b7fe640 1 -- 172.21.15.89:0/2147761953 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6205280b0 con 0x55b62052cd20 2024-09-15T23:38:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.813+0000 7f434bfff640 1 --2- 172.21.15.89:0/2147761953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b6205424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:38:59.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f434b7fe640 1 --2- 172.21.15.89:0/2147761953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b62052cd20 0x55b62052e300 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f434000bb00 tx=0x7f434000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283dbaf738851cf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f4331ffb640 1 -- 172.21.15.89:0/2147761953 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4340028020 con 0x55b62052cd20 2024-09-15T23:38:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f4331ffb640 1 -- 172.21.15.89:0/2147761953 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f434000f520 con 0x55b62052cd20 2024-09-15T23:38:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f4331ffb640 1 -- 172.21.15.89:0/2147761953 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4340004660 con 0x55b62052cd20 2024-09-15T23:38:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f43549dc740 1 -- 172.21.15.89:0/2147761953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b62052cd20 msgr2=0x55b62052e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f43549dc740 1 --2- 172.21.15.89:0/2147761953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b62052cd20 0x55b62052e300 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f434000bb00 tx=0x7f434000b910 comp rx=0 tx=0).stop 2024-09-15T23:38:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f43549dc740 1 -- 172.21.15.89:0/2147761953 shutdown_connections 2024-09-15T23:38:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f43549dc740 1 --2- 172.21.15.89:0/2147761953 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b620536550 0x55b62052e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f43549dc740 1 --2- 172.21.15.89:0/2147761953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b62052cd20 0x55b62052e300 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f43549dc740 1 --2- 172.21.15.89:0/2147761953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b6205424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.814+0000 7f43549dc740 1 -- 172.21.15.89:0/2147761953 >> 172.21.15.89:0/2147761953 conn(0x55b6204f4310 msgr2=0x55b6205330e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 1 -- 172.21.15.89:0/2147761953 shutdown_connections 2024-09-15T23:38:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 1 -- 172.21.15.89:0/2147761953 wait complete. 2024-09-15T23:38:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 1 -- start start 2024-09-15T23:38:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 1 Processor -- start 2024-09-15T23:38:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b62053a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b62053a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b62053ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1ade10 con 0x55b620536550 2024-09-15T23:38:59.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1ad740 con 0x55b62052cd20 2024-09-15T23:38:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f434c1aa980 con 0x55b62052b7b0 2024-09-15T23:38:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f43549dc740 10 client.0 _ll_register_callbacks cb 0x7f434400a880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:38:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f4350ba0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b62053ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f4350ba0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b62053ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47468/0 (socket says 172.21.15.89:47468) 2024-09-15T23:38:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f4350ba0640 1 -- 172.21.15.89:0/2064090159 learned_addr learned my addr 172.21.15.89:0/2064090159 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:38:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f434b7fe640 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b62053a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.815+0000 7f434bfff640 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b62053a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f4350ba0640 1 -- 172.21.15.89:0/2064090159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 msgr2=0x55b62053a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:59.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f4350ba0640 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b62053a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f4350ba0640 1 -- 172.21.15.89:0/2064090159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 msgr2=0x55b62053a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:38:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f4350ba0640 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b62053a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f4350ba0640 1 -- 172.21.15.89:0/2064090159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6205afde0 con 0x55b620536550 2024-09-15T23:38:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f4350ba0640 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b62053ade0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f434400f5c0 tx=0x7f4344025d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf119291392f7d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:38:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f432b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:38:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f432b7fe640 1 -- 172.21.15.89:0/2064090159 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434402b040 con 0x55b620536550 2024-09-15T23:38:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f432b7fe640 1 -- 172.21.15.89:0/2064090159 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f434400dbf0 con 0x55b620536550 2024-09-15T23:38:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f43549dc740 1 -- 172.21.15.89:0/2064090159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f434000f520 con 0x55b620536550 2024-09-15T23:38:59.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f432b7fe640 1 -- 172.21.15.89:0/2064090159 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f434400d770 con 0x55b620536550 2024-09-15T23:38:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f43549dc740 10 client.20580 fetch_fsmap learned FSMap version 29 2024-09-15T23:38:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f43549dc740 1 -- 172.21.15.89:0/2064090159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b6205b0fd0 con 0x55b620536550 2024-09-15T23:38:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.816+0000 7f432b7fe640 1 -- 172.21.15.89:0/2064090159 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f434000f520 con 0x55b620536550 2024-09-15T23:38:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.817+0000 7f432b7fe640 1 -- 172.21.15.89:0/2064090159 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4344019040 con 0x55b620536550 2024-09-15T23:38:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.817+0000 7f43549dc740 10 client.20580 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:38:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.817+0000 7f43549dc740 10 client.20580 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:38:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.817+0000 7f43549dc740 20 client.20580 populate_metadata read hostname 'smithi089' 2024-09-15T23:38:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.817+0000 7f43549dc740 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55b6205b1740 0x55b6205b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:38:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.817+0000 7f43549dc740 4 client.20580 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:38:59.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.817+0000 7f43549dc740 1 -- 172.21.15.89:0/2064090159 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b620498d70 con 0x55b6205b1740 2024-09-15T23:38:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.817+0000 7f434bfff640 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55b6205b1740 0x55b6205b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:38:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.817+0000 7f434bfff640 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55b6205b1740 0x55b6205b3b60 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c73c991c19e0f2f3 server_cookie=978440767939acb7 in_seq=0 out_seq=0 2024-09-15T23:38:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.817+0000 7f432b7fe640 10 client.20580 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:38:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f432b7fe640 1 -- 172.21.15.89:0/2064090159 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b620498d70 con 0x55b6205b1740 2024-09-15T23:38:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f432b7fe640 10 client.20580 handle_command_reply: tid=0 2024-09-15T23:38:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 client.20580 shutdown 2024-09-15T23:38:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 -- 172.21.15.89:0/2064090159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 msgr2=0x55b62053ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b62053ade0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f434400f5c0 tx=0x7f4344025d80 comp rx=0 tx=0).stop 2024-09-15T23:38:59.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 -- 172.21.15.89:0/2064090159 shutdown_connections 2024-09-15T23:38:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55b6205b1740 0x55b6205b3b60 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b620536550 0x55b62053ade0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b62052cd20 0x55b62053a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 --2- 172.21.15.89:0/2064090159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b62052b7b0 0x55b62053a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f432b7fe640 0 client.20580 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:38:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 -- 172.21.15.89:0/2064090159 >> 172.21.15.89:0/2064090159 conn(0x55b6204f4310 msgr2=0x55b620526780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 -- 172.21.15.89:0/2064090159 shutdown_connections 2024-09-15T23:38:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 1 -- 172.21.15.89:0/2064090159 wait complete. 2024-09-15T23:38:59.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43549dc740 20 client.20580 trim_cache size 0 max 16384 2024-09-15T23:38:59.853 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:38:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43513a1640 1 -- 172.21.15.89:0/2319935516 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43340a3240 msgr2=0x7f43340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43513a1640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43340a3240 0x7f43340a5700 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f434400af40 tx=0x7f4344005c80 comp rx=0 tx=0).stop 2024-09-15T23:38:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43513a1640 1 -- 172.21.15.89:0/2319935516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f434c1008d0 msgr2=0x7f434c197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:38:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.818+0000 7f43513a1640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f434c1008d0 0x7f434c197150 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f433c009a50 tx=0x7f433c004270 comp rx=0 tx=0).stop 2024-09-15T23:38:59.854 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.819+0000 7f43513a1640 1 -- 172.21.15.89:0/2319935516 shutdown_connections 2024-09-15T23:38:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.819+0000 7f43513a1640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f43340a3240 0x7f43340a5700 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.819+0000 7f43513a1640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f434c1068d0 0x7f434c19ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.819+0000 7f43513a1640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f434c101270 0x7f434c197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.819+0000 7f43513a1640 1 --2- 172.21.15.89:0/2319935516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f434c1008d0 0x7f434c197150 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:38:59.855 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.819+0000 7f43513a1640 1 -- 172.21.15.89:0/2319935516 >> 172.21.15.89:0/2319935516 conn(0x7f434c0fc610 msgr2=0x7f434c0fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:38:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.819+0000 7f43513a1640 1 -- 172.21.15.89:0/2319935516 shutdown_connections 2024-09-15T23:38:59.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:38:59.819+0000 7f43513a1640 1 -- 172.21.15.89:0/2319935516 wait complete. 2024-09-15T23:39:00.644 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:39:00.898 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:01.285 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:01 smithi089 ceph-mon[30933]: pgmap v799: 129 pgs: 129 active+clean; 3.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:01.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:01 smithi120 ceph-mon[35250]: pgmap v799: 129 pgs: 129 active+clean; 3.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:01.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:01 smithi174 ceph-mon[35586]: pgmap v799: 129 pgs: 129 active+clean; 3.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 -- 172.21.15.89:0/1945274717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd0106f10 msgr2=0x7fccd010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 --2- 172.21.15.89:0/1945274717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd0106f10 0x7fccd010ea90 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009ca0 tx=0x7fccc802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 -- 172.21.15.89:0/1945274717 shutdown_connections 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 --2- 172.21.15.89:0/1945274717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd0106f10 0x7fccd010ea90 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 --2- 172.21.15.89:0/1945274717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccd0102790 0x7fccd01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 --2- 172.21.15.89:0/1945274717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccd0101de0 0x7fccd01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 -- 172.21.15.89:0/1945274717 >> 172.21.15.89:0/1945274717 conn(0x7fccd00fd8c0 msgr2=0x7fccd00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 -- 172.21.15.89:0/1945274717 shutdown_connections 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 -- 172.21.15.89:0/1945274717 wait complete. 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 Processor -- start 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.499+0000 7fccd6175640 1 -- start start 2024-09-15T23:39:01.520 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fccd6175640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd0101de0 0x7fccd01985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fccd6175640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccd0102790 0x7fccd0198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fccd6175640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccd0106f10 0x7fccd019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fccd6175640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fccd0112010 con 0x7fccd0101de0 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fccd6175640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fccd0111e90 con 0x7fccd0102790 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fccd6175640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fccd0112190 con 0x7fccd0106f10 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd0101de0 0x7fccd01985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd0101de0 0x7fccd01985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47492/0 (socket says 172.21.15.89:47492) 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccf7fe640 1 -- 172.21.15.89:0/4214559609 learned_addr learned my addr 172.21.15.89:0/4214559609 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccffff640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccd0106f10 0x7fccd019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccf7fe640 1 -- 172.21.15.89:0/4214559609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccd0106f10 msgr2=0x7fccd019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccd0106f10 0x7fccd019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fccceffd640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccd0102790 0x7fccd0198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccf7fe640 1 -- 172.21.15.89:0/4214559609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccd0102790 msgr2=0x7fccd0198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:01.521 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccd0102790 0x7fccd0198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccf7fe640 1 -- 172.21.15.89:0/4214559609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd019d5a0 con 0x7fccd0101de0 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccffff640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccd0106f10 0x7fccd019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.500+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd0101de0 0x7fccd01985f0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fccc000b4a0 tx=0x7fccc000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.501+0000 7fccccff9640 1 -- 172.21.15.89:0/4214559609 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0014070 con 0x7fccd0101de0 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.501+0000 7fccccff9640 1 -- 172.21.15.89:0/4214559609 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc0010af0 con 0x7fccd0101de0 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.501+0000 7fccd6175640 1 -- 172.21.15.89:0/4214559609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd019d890 con 0x7fccd0101de0 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.501+0000 7fccccff9640 1 -- 172.21.15.89:0/4214559609 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc0017d10 con 0x7fccd0101de0 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.501+0000 7fccd6175640 1 -- 172.21.15.89:0/4214559609 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd01a5170 con 0x7fccd0101de0 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.503+0000 7fccccff9640 1 -- 172.21.15.89:0/4214559609 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fccc0017070 con 0x7fccd0101de0 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.503+0000 7fccd854e740 1 Processor -- start 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.503+0000 7fccd854e740 1 -- start start 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.503+0000 7fccccff9640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fccb80a3240 0x7fccb80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.503+0000 7fccd854e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf476090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.503+0000 7fccd854e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf476a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.522 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.503+0000 7fccd854e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf481370 0x55f6bf483760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccd854e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf3dc2f0 con 0x55f6bf4765d0 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccd854e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf383cf0 con 0x55f6bf481370 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccd854e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf3d82b0 con 0x55f6bf477b40 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccccff9640 1 -- 172.21.15.89:0/4214559609 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fccc00cf1f0 con 0x7fccd0101de0 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fcccffff640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fccb80a3240 0x7fccb80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccceffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf476a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccceffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf476a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47502/0 (socket says 172.21.15.89:47502) 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccceffd640 1 -- 172.21.15.89:0/1078377367 learned_addr learned my addr 172.21.15.89:0/1078377367 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fcccffff640 1 --2- 172.21.15.89:0/1078377367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf481370 0x55f6bf483760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/1078377367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf476090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccceffd640 1 -- 172.21.15.89:0/1078377367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 msgr2=0x55f6bf476090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:01.523 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccceffd640 1 --2- 172.21.15.89:0/1078377367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf476090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccceffd640 1 -- 172.21.15.89:0/1078377367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf481370 msgr2=0x55f6bf483760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccceffd640 1 --2- 172.21.15.89:0/1078377367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf481370 0x55f6bf483760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccceffd640 1 -- 172.21.15.89:0/1078377367 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6bf3683a0 con 0x55f6bf4765d0 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccceffd640 1 --2- 172.21.15.89:0/1078377367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf476a10 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fccc400a8e0 tx=0x7fccc400ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bb9f86e2e1fc1e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fcccffff640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fccb80a3240 0x7fccb80a5700 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fccc8006010 tx=0x7fccc8031de0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccb5ffb640 1 -- 172.21.15.89:0/1078377367 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4015070 con 0x55f6bf4765d0 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccb5ffb640 1 -- 172.21.15.89:0/1078377367 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc400cb40 con 0x55f6bf4765d0 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.504+0000 7fccb5ffb640 1 -- 172.21.15.89:0/1078377367 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4018560 con 0x55f6bf4765d0 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.505+0000 7fccd854e740 1 -- 172.21.15.89:0/1078377367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 msgr2=0x55f6bf476a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.505+0000 7fccd854e740 1 --2- 172.21.15.89:0/1078377367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf476a10 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fccc400a8e0 tx=0x7fccc400ada0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.505+0000 7fccd854e740 1 -- 172.21.15.89:0/1078377367 shutdown_connections 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.505+0000 7fccd854e740 1 --2- 172.21.15.89:0/1078377367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf481370 0x55f6bf483760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.524 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.505+0000 7fccd854e740 1 --2- 172.21.15.89:0/1078377367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf476a10 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.505+0000 7fccd854e740 1 --2- 172.21.15.89:0/1078377367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf476090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.505+0000 7fccd854e740 1 -- 172.21.15.89:0/1078377367 >> 172.21.15.89:0/1078377367 conn(0x55f6bf43f310 msgr2=0x55f6bf472c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.505+0000 7fccd854e740 1 -- 172.21.15.89:0/1078377367 shutdown_connections 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.505+0000 7fccd854e740 1 -- 172.21.15.89:0/1078377367 wait complete. 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.505+0000 7fccd854e740 1 -- start start 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccd854e740 1 Processor -- start 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccd854e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf480d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccd854e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf477b40 0x55f6bf47dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccd854e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf481370 0x55f6bf47e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccd854e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf488480 con 0x55f6bf4765d0 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccd854e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf397060 con 0x55f6bf477b40 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccd854e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f6bf488600 con 0x55f6bf481370 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccd854e740 10 client.0 _ll_register_callbacks cb 0x55f6bf0ea8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:01.525 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fcccf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf480d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fcccf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf480d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47512/0 (socket says 172.21.15.89:47512) 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fcccf7fe640 1 -- 172.21.15.89:0/657633928 learned_addr learned my addr 172.21.15.89:0/657633928 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fcccffff640 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf481370 0x55f6bf47e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccceffd640 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf477b40 0x55f6bf47dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fcccf7fe640 1 -- 172.21.15.89:0/657633928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf481370 msgr2=0x55f6bf47e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf481370 0x55f6bf47e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fcccf7fe640 1 -- 172.21.15.89:0/657633928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf477b40 msgr2=0x55f6bf47dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf477b40 0x55f6bf47dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fcccf7fe640 1 -- 172.21.15.89:0/657633928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6bf14d3c0 con 0x55f6bf4765d0 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccceffd640 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf477b40 0x55f6bf47dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf480d20 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fccc00ccc70 tx=0x7fccc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb644632faf73ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.506+0000 7fccaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccaeffd640 1 -- 172.21.15.89:0/657633928 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc001e070 con 0x55f6bf4765d0 2024-09-15T23:39:01.526 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccaeffd640 1 -- 172.21.15.89:0/657633928 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fccc009f490 con 0x55f6bf4765d0 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccd854e740 1 -- 172.21.15.89:0/657633928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccc400cb40 con 0x55f6bf4765d0 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccaeffd640 1 -- 172.21.15.89:0/657633928 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fccc009f7d0 con 0x55f6bf4765d0 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccd854e740 10 client.20592 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccd854e740 1 -- 172.21.15.89:0/657633928 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f6bf493f30 con 0x55f6bf4765d0 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccaeffd640 1 -- 172.21.15.89:0/657633928 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc400cb40 con 0x55f6bf4765d0 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccaeffd640 1 -- 172.21.15.89:0/657633928 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fccc009f0d0 con 0x55f6bf4765d0 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccd854e740 10 client.20592 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccd854e740 10 client.20592 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccd854e740 20 client.20592 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccd854e740 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f6bf495280 0x55f6bf4976d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccd854e740 4 client.20592 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.507+0000 7fccd854e740 1 -- 172.21.15.89:0/657633928 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6bf3e9860 con 0x55f6bf495280 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.508+0000 7fccceffd640 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f6bf495280 0x55f6bf4976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.527 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.508+0000 7fccaeffd640 10 client.20592 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.508+0000 7fccceffd640 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f6bf495280 0x55f6bf4976d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ab53bc69dfa17192 server_cookie=8523d22a50e89240 in_seq=0 out_seq=0 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccaeffd640 1 -- 172.21.15.89:0/657633928 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f6bf3e9860 con 0x55f6bf495280 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccaeffd640 10 client.20592 handle_command_reply: tid=0 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 client.20592 shutdown 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 -- 172.21.15.89:0/657633928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 msgr2=0x55f6bf480d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf480d20 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fccc00ccc70 tx=0x7fccc00046c0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 -- 172.21.15.89:0/657633928 shutdown_connections 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f6bf495280 0x55f6bf4976d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf481370 0x55f6bf47e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccaeffd640 0 client.20592 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf477b40 0x55f6bf47dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 --2- 172.21.15.89:0/657633928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf480d20 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.528 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 -- 172.21.15.89:0/657633928 >> 172.21.15.89:0/657633928 conn(0x55f6bf43f310 msgr2=0x55f6bf478960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:01.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 -- 172.21.15.89:0/657633928 shutdown_connections 2024-09-15T23:39:01.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 1 -- 172.21.15.89:0/657633928 wait complete. 2024-09-15T23:39:01.529 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.509+0000 7fccd854e740 20 client.20592 trim_cache size 0 max 16384 2024-09-15T23:39:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.522+0000 7fccd854e740 1 Processor -- start 2024-09-15T23:39:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.522+0000 7fccd854e740 1 -- start start 2024-09-15T23:39:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.522+0000 7fccd854e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf4765d0 0x55f6bf479120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fccd854e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf479660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fccd854e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf481370 0x55f6bf49cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fccd854e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fccd01accb0 con 0x55f6bf481370 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fccd854e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fccd01ae7f0 con 0x55f6bf4765d0 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fccd854e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fccd01ad380 con 0x55f6bf477b40 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fcccffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf481370 0x55f6bf49cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fcccffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf481370 0x55f6bf49cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47528/0 (socket says 172.21.15.89:47528) 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fcccffff640 1 -- 172.21.15.89:0/3262442879 learned_addr learned my addr 172.21.15.89:0/3262442879 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fccceffd640 1 --2- 172.21.15.89:0/3262442879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf479660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/3262442879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf4765d0 0x55f6bf479120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fcccffff640 1 -- 172.21.15.89:0/3262442879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 msgr2=0x55f6bf479660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fcccffff640 1 --2- 172.21.15.89:0/3262442879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf479660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fcccffff640 1 -- 172.21.15.89:0/3262442879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf4765d0 msgr2=0x55f6bf479120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fcccffff640 1 --2- 172.21.15.89:0/3262442879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf4765d0 0x55f6bf479120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fcccffff640 1 -- 172.21.15.89:0/3262442879 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6bf471310 con 0x55f6bf481370 2024-09-15T23:39:01.534 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fcccffff640 1 --2- 172.21.15.89:0/3262442879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf481370 0x55f6bf49cfa0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fccc803c330 tx=0x7fccc8048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2027d7482eaeb718 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fccb57fa640 1 -- 172.21.15.89:0/3262442879 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc802f810 con 0x55f6bf481370 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fccb57fa640 1 -- 172.21.15.89:0/3262442879 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc8036050 con 0x55f6bf481370 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.523+0000 7fccb57fa640 1 -- 172.21.15.89:0/3262442879 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc8030d10 con 0x55f6bf481370 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.524+0000 7fccd854e740 1 -- 172.21.15.89:0/3262442879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf481370 msgr2=0x55f6bf49cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.524+0000 7fccd854e740 1 --2- 172.21.15.89:0/3262442879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf481370 0x55f6bf49cfa0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fccc803c330 tx=0x7fccc8048000 comp rx=0 tx=0).stop 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.524+0000 7fccd854e740 1 -- 172.21.15.89:0/3262442879 shutdown_connections 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.524+0000 7fccd854e740 1 --2- 172.21.15.89:0/3262442879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf481370 0x55f6bf49cfa0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.524+0000 7fccd854e740 1 --2- 172.21.15.89:0/3262442879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf479660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.524+0000 7fccd854e740 1 --2- 172.21.15.89:0/3262442879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf4765d0 0x55f6bf479120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.524+0000 7fccd854e740 1 -- 172.21.15.89:0/3262442879 >> 172.21.15.89:0/3262442879 conn(0x55f6bf43f310 msgr2=0x55f6bf471d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.524+0000 7fccd854e740 1 -- 172.21.15.89:0/3262442879 shutdown_connections 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.524+0000 7fccd854e740 1 -- 172.21.15.89:0/3262442879 wait complete. 2024-09-15T23:39:01.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fccd854e740 1 -- start start 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fccd854e740 1 Processor -- start 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fccd854e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf489ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fccd854e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf48a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fccd854e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf481370 0x55f6bf4fef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fccd854e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fccd01ae120 con 0x55f6bf4765d0 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fccd854e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fccd01ada50 con 0x55f6bf481370 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fccd854e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fccd01aac90 con 0x55f6bf477b40 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fccd854e740 10 client.0 _ll_register_callbacks cb 0x7fccc0017eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fcccf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf489ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fcccf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf489ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47536/0 (socket says 172.21.15.89:47536) 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fcccf7fe640 1 -- 172.21.15.89:0/3141491207 learned_addr learned my addr 172.21.15.89:0/3141491207 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:01.536 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fccceffd640 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf48a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fcccf7fe640 1 -- 172.21.15.89:0/3141491207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 msgr2=0x55f6bf48a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf48a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fcccf7fe640 1 -- 172.21.15.89:0/3141491207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf481370 msgr2=0x55f6bf4fef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fcccffff640 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf481370 0x55f6bf4fef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf481370 0x55f6bf4fef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.525+0000 7fcccf7fe640 1 -- 172.21.15.89:0/3141491207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6bf4ff4b0 con 0x55f6bf4765d0 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fcccf7fe640 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf489ce0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fccc009b2d0 tx=0x7fccc0017590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f14be8139956cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccaeffd640 1 -- 172.21.15.89:0/3141491207 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc00b4060 con 0x55f6bf4765d0 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccaeffd640 1 -- 172.21.15.89:0/3141491207 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fccc00b4200 con 0x55f6bf4765d0 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccd854e740 1 -- 172.21.15.89:0/3141491207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccc8036050 con 0x55f6bf4765d0 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccaeffd640 1 -- 172.21.15.89:0/3141491207 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fccc0060070 con 0x55f6bf4765d0 2024-09-15T23:39:01.537 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccd854e740 10 client.20598 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccd854e740 1 -- 172.21.15.89:0/3141491207 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f6bf500750 con 0x55f6bf4765d0 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccaeffd640 1 -- 172.21.15.89:0/3141491207 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc8036050 con 0x55f6bf4765d0 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccaeffd640 1 -- 172.21.15.89:0/3141491207 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fccc009a870 con 0x55f6bf4765d0 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccd854e740 10 client.20598 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccd854e740 10 client.20598 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccd854e740 20 client.20598 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccd854e740 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f6bf500dc0 0x55f6bf5031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccd854e740 4 client.20598 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.526+0000 7fccd854e740 1 -- 172.21.15.89:0/3141491207 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f6bf3e3d70 con 0x55f6bf500dc0 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.527+0000 7fccceffd640 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f6bf500dc0 0x55f6bf5031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.527+0000 7fccaeffd640 10 client.20598 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.527+0000 7fccceffd640 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f6bf500dc0 0x55f6bf5031b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8122a913908c2a24 server_cookie=5fb42531d91cbf38 in_seq=0 out_seq=0 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.527+0000 7fccaeffd640 1 -- 172.21.15.89:0/3141491207 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f6bf3e3d70 con 0x55f6bf500dc0 2024-09-15T23:39:01.538 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.527+0000 7fccaeffd640 10 client.20598 handle_command_reply: tid=0 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.527+0000 7fccd854e740 1 client.20598 shutdown 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.527+0000 7fccd854e740 1 -- 172.21.15.89:0/3141491207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 msgr2=0x55f6bf489ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.527+0000 7fccd854e740 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf489ce0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fccc009b2d0 tx=0x7fccc0017590 comp rx=0 tx=0).stop 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd854e740 1 -- 172.21.15.89:0/3141491207 shutdown_connections 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd854e740 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55f6bf500dc0 0x55f6bf5031b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd854e740 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6bf481370 0x55f6bf4fef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd854e740 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f6bf477b40 0x55f6bf48a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccaeffd640 0 client.20598 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd854e740 1 --2- 172.21.15.89:0/3141491207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f6bf4765d0 0x55f6bf489ce0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd854e740 1 -- 172.21.15.89:0/3141491207 >> 172.21.15.89:0/3141491207 conn(0x55f6bf43f310 msgr2=0x55f6bf47f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd854e740 1 -- 172.21.15.89:0/3141491207 shutdown_connections 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd854e740 1 -- 172.21.15.89:0/3141491207 wait complete. 2024-09-15T23:39:01.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd854e740 20 client.20598 trim_cache size 0 max 16384 2024-09-15T23:39:01.540 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:39:01.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 -- 172.21.15.89:0/4214559609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fccb80a3240 msgr2=0x7fccb80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:01.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fccb80a3240 0x7fccb80a5700 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fccc8006010 tx=0x7fccc8031de0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 -- 172.21.15.89:0/4214559609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd0101de0 msgr2=0x7fccd01985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:01.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd0101de0 0x7fccd01985f0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fccc000b4a0 tx=0x7fccc000b960 comp rx=0 tx=0).stop 2024-09-15T23:39:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 -- 172.21.15.89:0/4214559609 shutdown_connections 2024-09-15T23:39:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fccb80a3240 0x7fccb80a5700 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccd0106f10 0x7fccd019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccd0102790 0x7fccd0198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 --2- 172.21.15.89:0/4214559609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fccd0101de0 0x7fccd01985f0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 -- 172.21.15.89:0/4214559609 >> 172.21.15.89:0/4214559609 conn(0x7fccd00fd8c0 msgr2=0x7fccd00fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 -- 172.21.15.89:0/4214559609 shutdown_connections 2024-09-15T23:39:01.541 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:01.528+0000 7fccd6175640 1 -- 172.21.15.89:0/4214559609 wait complete. 2024-09-15T23:39:02.312 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:39:02.560 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:03.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 -- 172.21.15.89:0/3096537506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0010c170 msgr2=0x7f8c0010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:03.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 --2- 172.21.15.89:0/3096537506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0010c170 0x7f8c0010e600 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009900 tx=0x7f8bf802a480 comp rx=0 tx=0).stop 2024-09-15T23:39:03.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 -- 172.21.15.89:0/3096537506 shutdown_connections 2024-09-15T23:39:03.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 --2- 172.21.15.89:0/3096537506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c0010c170 0x7f8c0010e600 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 --2- 172.21.15.89:0/3096537506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c001036f0 0x7f8c0010bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 --2- 172.21.15.89:0/3096537506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c00102dd0 0x7f8c001031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 -- 172.21.15.89:0/3096537506 >> 172.21.15.89:0/3096537506 conn(0x7f8c000fc790 msgr2=0x7f8c000febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:03.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 -- 172.21.15.89:0/3096537506 shutdown_connections 2024-09-15T23:39:03.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 -- 172.21.15.89:0/3096537506 wait complete. 2024-09-15T23:39:03.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 Processor -- start 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.018+0000 7f8c059d8640 1 -- start start 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c059d8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c00102dd0 0x7f8c00100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c059d8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c001036f0 0x7f8c00100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c059d8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c0010c170 0x7f8c001014b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c059d8640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c00111f20 con 0x7f8c001036f0 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c059d8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c00111da0 con 0x7f8c00102dd0 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c059d8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c001120a0 con 0x7f8c0010c170 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bfe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c001036f0 0x7f8c00100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bfe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c001036f0 0x7f8c00100f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47558/0 (socket says 172.21.15.89:47558) 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bfe7fc640 1 -- 172.21.15.89:0/1285217620 learned_addr learned my addr 172.21.15.89:0/1285217620 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c00102dd0 0x7f8c00100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c0010c170 0x7f8c001014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bfe7fc640 1 -- 172.21.15.89:0/1285217620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c0010c170 msgr2=0x7f8c001014b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c0010c170 0x7f8c001014b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bfe7fc640 1 -- 172.21.15.89:0/1285217620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c00102dd0 msgr2=0x7f8c00100a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c00102dd0 0x7f8c00100a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bfe7fc640 1 -- 172.21.15.89:0/1285217620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c001a6820 con 0x7f8c001036f0 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c001036f0 0x7f8c00100f70 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400d950 tx=0x7f8bf400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c049d6640 1 -- 172.21.15.89:0/1285217620 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4017070 con 0x7f8c001036f0 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c059d8640 1 -- 172.21.15.89:0/1285217620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c001a6b10 con 0x7f8c001036f0 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c049d6640 1 -- 172.21.15.89:0/1285217620 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf4009c30 con 0x7f8c001036f0 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c049d6640 1 -- 172.21.15.89:0/1285217620 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf400f6b0 con 0x7f8c001036f0 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.019+0000 7f8c059d8640 1 -- 172.21.15.89:0/1285217620 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c001a70a0 con 0x7f8c001036f0 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8c049d6640 1 -- 172.21.15.89:0/1285217620 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8bf40109d0 con 0x7f8c001036f0 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8c07db1740 1 Processor -- start 2024-09-15T23:39:03.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8c07db1740 1 -- start start 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370bb40 0x55fcd370a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370a5d0 0x55fcd370aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd3715370 0x55fcd3717760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8c07db1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fcd36702f0 con 0x55fcd370a5d0 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8c07db1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fcd3617cf0 con 0x55fcd3715370 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8c07db1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fcd366c2b0 con 0x55fcd370bb40 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8bfeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370a5d0 0x55fcd370aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8bfeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370a5d0 0x55fcd370aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47574/0 (socket says 172.21.15.89:47574) 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8bfeffd640 1 -- 172.21.15.89:0/567923529 learned_addr learned my addr 172.21.15.89:0/567923529 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/567923529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370bb40 0x55fcd370a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8c049d6640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8be80a3240 0x7f8be80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.022+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/567923529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd3715370 0x55fcd3717760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8be80a3240 0x7f8be80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8bfeffd640 1 -- 172.21.15.89:0/567923529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370bb40 msgr2=0x55fcd370a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:03.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/567923529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370bb40 0x55fcd370a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8bfeffd640 1 -- 172.21.15.89:0/567923529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd3715370 msgr2=0x55fcd3717760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:03.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/567923529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd3715370 0x55fcd3717760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8bfeffd640 1 -- 172.21.15.89:0/567923529 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd35fc3a0 con 0x55fcd370a5d0 2024-09-15T23:39:03.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8c049d6640 1 -- 172.21.15.89:0/1285217620 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8bf40cf890 con 0x7f8c001036f0 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/567923529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd3715370 0x55fcd3717760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/567923529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370a5d0 0x55fcd370aa10 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0007f90 tx=0x7f8bf000f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec9ed1afab6f0b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8be80a3240 0x7f8be80a5700 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8005ec0 tx=0x7f8bf802a480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8be5ffb640 1 -- 172.21.15.89:0/567923529 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0004d20 con 0x55fcd370a5d0 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8be5ffb640 1 -- 172.21.15.89:0/567923529 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf0010be0 con 0x55fcd370a5d0 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.023+0000 7f8be5ffb640 1 -- 172.21.15.89:0/567923529 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0005780 con 0x55fcd370a5d0 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 -- 172.21.15.89:0/567923529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370a5d0 msgr2=0x55fcd370aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 --2- 172.21.15.89:0/567923529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370a5d0 0x55fcd370aa10 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0007f90 tx=0x7f8bf000f850 comp rx=0 tx=0).stop 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 -- 172.21.15.89:0/567923529 shutdown_connections 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 --2- 172.21.15.89:0/567923529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd3715370 0x55fcd3717760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 --2- 172.21.15.89:0/567923529 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370a5d0 0x55fcd370aa10 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 --2- 172.21.15.89:0/567923529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370bb40 0x55fcd370a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 -- 172.21.15.89:0/567923529 >> 172.21.15.89:0/567923529 conn(0x55fcd36d3310 msgr2=0x55fcd3706c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 -- 172.21.15.89:0/567923529 shutdown_connections 2024-09-15T23:39:03.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 -- 172.21.15.89:0/567923529 wait complete. 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 -- start start 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.024+0000 7f8c07db1740 1 Processor -- start 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370a5d0 0x55fcd3714ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 0x55fcd3711f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd3712490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8c07db1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fcd3718070 con 0x55fcd3715370 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8c07db1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fcd362b060 con 0x55fcd370bb40 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8c07db1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fcd37181f0 con 0x55fcd370a5d0 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8c07db1740 10 client.0 _ll_register_callbacks cb 0x55fcd337e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd3712490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd3712490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47590/0 (socket says 172.21.15.89:47590) 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bff7fe640 1 -- 172.21.15.89:0/4031534007 learned_addr learned my addr 172.21.15.89:0/4031534007 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 0x55fcd3711f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370a5d0 0x55fcd3714ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bff7fe640 1 -- 172.21.15.89:0/4031534007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370a5d0 msgr2=0x55fcd3714ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370a5d0 0x55fcd3714ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bff7fe640 1 -- 172.21.15.89:0/4031534007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 msgr2=0x55fcd3711f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 0x55fcd3711f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bff7fe640 1 -- 172.21.15.89:0/4031534007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd33e13c0 con 0x55fcd3715370 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 0x55fcd3711f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd3712490 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8033000 tx=0x7f8bf8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7ae29ff37e1226 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:03.037 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bdeffd640 1 -- 172.21.15.89:0/4031534007 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf804e020 con 0x55fcd3715370 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8bdeffd640 1 -- 172.21.15.89:0/4031534007 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8bf80046c0 con 0x55fcd3715370 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.025+0000 7f8c07db1740 1 -- 172.21.15.89:0/4031534007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bf0010be0 con 0x55fcd3715370 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8bdeffd640 1 -- 172.21.15.89:0/4031534007 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8bf8059050 con 0x55fcd3715370 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8c07db1740 10 client.20610 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8c07db1740 1 -- 172.21.15.89:0/4031534007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcd3723ac0 con 0x55fcd3715370 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8bdeffd640 1 -- 172.21.15.89:0/4031534007 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf0010be0 con 0x55fcd3715370 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8bdeffd640 1 -- 172.21.15.89:0/4031534007 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8bf8003950 con 0x55fcd3715370 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8c07db1740 10 client.20610 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8c07db1740 10 client.20610 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8c07db1740 20 client.20610 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8c07db1740 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55fcd3724f80 0x55fcd3727420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8c07db1740 4 client.20610 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:03.038 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8c07db1740 1 -- 172.21.15.89:0/4031534007 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcd367d860 con 0x55fcd3724f80 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.026+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55fcd3724f80 0x55fcd3727420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.027+0000 7f8bdeffd640 10 client.20610 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.027+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55fcd3724f80 0x55fcd3727420 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=58c76a56b510a3d0 server_cookie=c2d377348723ab02 in_seq=0 out_seq=0 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.027+0000 7f8bdeffd640 1 -- 172.21.15.89:0/4031534007 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fcd367d860 con 0x55fcd3724f80 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.027+0000 7f8bdeffd640 10 client.20610 handle_command_reply: tid=0 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.027+0000 7f8c07db1740 1 client.20610 shutdown 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 1 -- 172.21.15.89:0/4031534007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 msgr2=0x55fcd3712490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd3712490 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8033000 tx=0x7f8bf8009260 comp rx=0 tx=0).stop 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 1 -- 172.21.15.89:0/4031534007 shutdown_connections 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd3712490 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 0x55fcd3711f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55fcd3724f80 0x55fcd3727420 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 1 --2- 172.21.15.89:0/4031534007 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370a5d0 0x55fcd3714ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.039 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 1 -- 172.21.15.89:0/4031534007 >> 172.21.15.89:0/4031534007 conn(0x55fcd36d3310 msgr2=0x55fcd370c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:03.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8bdeffd640 0 client.20610 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:39:03.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 1 -- 172.21.15.89:0/4031534007 shutdown_connections 2024-09-15T23:39:03.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 1 -- 172.21.15.89:0/4031534007 wait complete. 2024-09-15T23:39:03.040 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.028+0000 7f8c07db1740 20 client.20610 trim_cache size 0 max 16384 2024-09-15T23:39:03.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8c07db1740 1 Processor -- start 2024-09-15T23:39:03.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8c07db1740 1 -- start start 2024-09-15T23:39:03.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370a5d0 0x55fcd3706e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370bb40 0x55fcd3721370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd3715370 0x55fcd370d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8c07db1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c001b1080 con 0x55fcd370bb40 2024-09-15T23:39:03.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8c07db1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c001b2bc0 con 0x55fcd370a5d0 2024-09-15T23:39:03.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8c07db1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c001b1750 con 0x55fcd3715370 2024-09-15T23:39:03.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8bfe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370bb40 0x55fcd3721370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8bfe7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370bb40 0x55fcd3721370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47594/0 (socket says 172.21.15.89:47594) 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8bfe7fc640 1 -- 172.21.15.89:0/3116201341 learned_addr learned my addr 172.21.15.89:0/3116201341 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/3116201341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370a5d0 0x55fcd3706e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.041+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/3116201341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd3715370 0x55fcd370d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.042+0000 7f8bfe7fc640 1 -- 172.21.15.89:0/3116201341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd3715370 msgr2=0x55fcd370d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.042+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/3116201341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd3715370 0x55fcd370d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.042+0000 7f8bfe7fc640 1 -- 172.21.15.89:0/3116201341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370a5d0 msgr2=0x55fcd3706e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.042+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/3116201341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370a5d0 0x55fcd3706e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.042+0000 7f8bfe7fc640 1 -- 172.21.15.89:0/3116201341 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd3714fd0 con 0x55fcd370bb40 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.042+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/3116201341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370bb40 0x55fcd3721370 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4097f20 tx=0x7f8bf406e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5785ae5cb5163f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.042+0000 7f8be57fa640 1 -- 172.21.15.89:0/3116201341 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4020040 con 0x55fcd370bb40 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.042+0000 7f8be57fa640 1 -- 172.21.15.89:0/3116201341 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf409bac0 con 0x55fcd370bb40 2024-09-15T23:39:03.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.042+0000 7f8be57fa640 1 -- 172.21.15.89:0/3116201341 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf40a42a0 con 0x55fcd370bb40 2024-09-15T23:39:03.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 -- 172.21.15.89:0/3116201341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370bb40 msgr2=0x55fcd3721370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:03.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 --2- 172.21.15.89:0/3116201341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370bb40 0x55fcd3721370 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4097f20 tx=0x7f8bf406e940 comp rx=0 tx=0).stop 2024-09-15T23:39:03.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 -- 172.21.15.89:0/3116201341 shutdown_connections 2024-09-15T23:39:03.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 --2- 172.21.15.89:0/3116201341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd3715370 0x55fcd370d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 --2- 172.21.15.89:0/3116201341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd370bb40 0x55fcd3721370 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 --2- 172.21.15.89:0/3116201341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370a5d0 0x55fcd3706e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 -- 172.21.15.89:0/3116201341 >> 172.21.15.89:0/3116201341 conn(0x55fcd36d3310 msgr2=0x55fcd3706700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 -- 172.21.15.89:0/3116201341 shutdown_connections 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 -- 172.21.15.89:0/3116201341 wait complete. 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 -- start start 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 Processor -- start 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370a5d0 0x55fcd371d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 0x55fcd371dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd370ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c001b24f0 con 0x55fcd3715370 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c001b1e20 con 0x55fcd370bb40 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.043+0000 7f8c07db1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c001af060 con 0x55fcd370a5d0 2024-09-15T23:39:03.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8c07db1740 10 client.0 _ll_register_callbacks cb 0x7f8bf8031ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd370ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bff7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd370ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47602/0 (socket says 172.21.15.89:47602) 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bff7fe640 1 -- 172.21.15.89:0/2199202872 learned_addr learned my addr 172.21.15.89:0/2199202872 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370a5d0 0x55fcd371d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 0x55fcd371dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bff7fe640 1 -- 172.21.15.89:0/2199202872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370a5d0 msgr2=0x55fcd371d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370a5d0 0x55fcd371d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bff7fe640 1 -- 172.21.15.89:0/2199202872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 msgr2=0x55fcd371dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 0x55fcd371dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bff7fe640 1 -- 172.21.15.89:0/2199202872 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd370f2d0 con 0x55fcd3715370 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bfe7fc640 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 0x55fcd371dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bff7fe640 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd370ead0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8003df0 tx=0x7f8bf8005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2495d685404e1e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:03.050 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bdeffd640 1 -- 172.21.15.89:0/2199202872 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf804e070 con 0x55fcd3715370 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bdeffd640 1 -- 172.21.15.89:0/2199202872 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8bf80037f0 con 0x55fcd3715370 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8c07db1740 1 -- 172.21.15.89:0/2199202872 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bf409bac0 con 0x55fcd3715370 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8bdeffd640 1 -- 172.21.15.89:0/2199202872 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8bf806b050 con 0x55fcd3715370 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.044+0000 7f8c07db1740 10 client.20616 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.045+0000 7f8c07db1740 1 -- 172.21.15.89:0/2199202872 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcd378f910 con 0x55fcd3715370 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.045+0000 7f8bdeffd640 1 -- 172.21.15.89:0/2199202872 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf409bac0 con 0x55fcd3715370 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.045+0000 7f8bdeffd640 1 -- 172.21.15.89:0/2199202872 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8bf80046c0 con 0x55fcd3715370 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.045+0000 7f8c07db1740 10 client.20616 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.045+0000 7f8c07db1740 10 client.20616 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.045+0000 7f8c07db1740 20 client.20616 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:03.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.045+0000 7f8c07db1740 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55fcd378ffc0 0x55fcd37923e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.045+0000 7f8c07db1740 4 client.20616 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.045+0000 7f8c07db1740 1 -- 172.21.15.89:0/2199202872 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fcd3677d70 con 0x55fcd378ffc0 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.045+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55fcd378ffc0 0x55fcd37923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8bfeffd640 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55fcd378ffc0 0x55fcd37923e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=79e13fdf7ecd9a5e server_cookie=d6ab5c8dbde8d7bc in_seq=0 out_seq=0 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8bdeffd640 10 client.20616 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8bdeffd640 1 -- 172.21.15.89:0/2199202872 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fcd3677d70 con 0x55fcd378ffc0 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8bdeffd640 10 client.20616 handle_command_reply: tid=0 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 client.20616 shutdown 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 -- 172.21.15.89:0/2199202872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 msgr2=0x55fcd370ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd370ead0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8003df0 tx=0x7f8bf8005410 comp rx=0 tx=0).stop 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 -- 172.21.15.89:0/2199202872 shutdown_connections 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd3715370 0x55fcd370ead0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fcd370bb40 0x55fcd371dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x55fcd378ffc0 0x55fcd37923e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 --2- 172.21.15.89:0/2199202872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fcd370a5d0 0x55fcd371d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8bdeffd640 0 client.20616 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:39:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 -- 172.21.15.89:0/2199202872 >> 172.21.15.89:0/2199202872 conn(0x55fcd36d3310 msgr2=0x55fcd37137b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 -- 172.21.15.89:0/2199202872 shutdown_connections 2024-09-15T23:39:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 1 -- 172.21.15.89:0/2199202872 wait complete. 2024-09-15T23:39:03.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.046+0000 7f8c07db1740 20 client.20616 trim_cache size 0 max 16384 2024-09-15T23:39:03.053 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:39:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 -- 172.21.15.89:0/1285217620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8be80a3240 msgr2=0x7f8be80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8be80a3240 0x7f8be80a5700 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8005ec0 tx=0x7f8bf802a480 comp rx=0 tx=0).stop 2024-09-15T23:39:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 -- 172.21.15.89:0/1285217620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c001036f0 msgr2=0x7f8c00100f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c001036f0 0x7f8c00100f70 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400d950 tx=0x7f8bf400de10 comp rx=0 tx=0).stop 2024-09-15T23:39:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 -- 172.21.15.89:0/1285217620 shutdown_connections 2024-09-15T23:39:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8be80a3240 0x7f8be80a5700 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c0010c170 0x7f8c001014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c001036f0 0x7f8c00100f70 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 --2- 172.21.15.89:0/1285217620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c00102dd0 0x7f8c00100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:03.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 -- 172.21.15.89:0/1285217620 >> 172.21.15.89:0/1285217620 conn(0x7f8c000fc790 msgr2=0x7f8c000fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:03.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 -- 172.21.15.89:0/1285217620 shutdown_connections 2024-09-15T23:39:03.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:03.047+0000 7f8c059d8640 1 -- 172.21.15.89:0/1285217620 wait complete. 2024-09-15T23:39:03.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:03 smithi089 ceph-mon[30933]: pgmap v800: 129 pgs: 129 active+clean; 3.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:39:03.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:03 smithi120 ceph-mon[35250]: pgmap v800: 129 pgs: 129 active+clean; 3.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:39:03.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:03 smithi174 ceph-mon[35586]: pgmap v800: 129 pgs: 129 active+clean; 3.5 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:39:03.940 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:39:04.184 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:04.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.672+0000 7f037a141640 1 -- 172.21.15.89:0/623118272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374105a70 msgr2=0x7f0374105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:04.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.672+0000 7f037a141640 1 --2- 172.21.15.89:0/623118272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374105a70 0x7f0374105e50 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f0364009900 tx=0x7f036402a410 comp rx=0 tx=0).stop 2024-09-15T23:39:04.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.672+0000 7f037a141640 1 -- 172.21.15.89:0/623118272 shutdown_connections 2024-09-15T23:39:04.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.672+0000 7f037a141640 1 --2- 172.21.15.89:0/623118272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0374101ba0 0x7f037410d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.672+0000 7f037a141640 1 --2- 172.21.15.89:0/623118272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0374101220 0x7f0374101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.672+0000 7f037a141640 1 --2- 172.21.15.89:0/623118272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374105a70 0x7f0374105e50 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.672+0000 7f037a141640 1 -- 172.21.15.89:0/623118272 >> 172.21.15.89:0/623118272 conn(0x7f03740fbee0 msgr2=0x7f03740fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.672+0000 7f037a141640 1 -- 172.21.15.89:0/623118272 shutdown_connections 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.672+0000 7f037a141640 1 -- 172.21.15.89:0/623118272 wait complete. 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.673+0000 7f037a141640 1 Processor -- start 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.673+0000 7f037a141640 1 -- start start 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f037a141640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374101220 0x7f0374197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f037a141640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0374101ba0 0x7f03741978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f037a141640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0374105a70 0x7f037419bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f037a141640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0374110f50 con 0x7f0374101220 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f037a141640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0374110dd0 con 0x7f0374101ba0 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f037a141640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03741110d0 con 0x7f0374105a70 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f03737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374101220 0x7f0374197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f03737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374101220 0x7f0374197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47618/0 (socket says 172.21.15.89:47618) 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f03737fe640 1 -- 172.21.15.89:0/4132976234 learned_addr learned my addr 172.21.15.89:0/4132976234 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f0373fff640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0374105a70 0x7f037419bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f03737fe640 1 -- 172.21.15.89:0/4132976234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0374105a70 msgr2=0x7f037419bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f03737fe640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0374105a70 0x7f037419bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f03737fe640 1 -- 172.21.15.89:0/4132976234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0374101ba0 msgr2=0x7f03741978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f03737fe640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0374101ba0 0x7f03741978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f03737fe640 1 -- 172.21.15.89:0/4132976234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037419c330 con 0x7f0374101220 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f0373fff640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0374105a70 0x7f037419bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f03737fe640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374101220 0x7f0374197380 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f0364006010 tx=0x7f0364003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f0370ff9640 1 -- 172.21.15.89:0/4132976234 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036403f070 con 0x7f0374101220 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f037a141640 1 -- 172.21.15.89:0/4132976234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037419c5c0 con 0x7f0374101220 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f0370ff9640 1 -- 172.21.15.89:0/4132976234 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0364031c30 con 0x7f0374101220 2024-09-15T23:39:04.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f0370ff9640 1 -- 172.21.15.89:0/4132976234 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0364035030 con 0x7f0374101220 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.674+0000 7f037a141640 1 -- 172.21.15.89:0/4132976234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03741a3f00 con 0x7f0374101220 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f0370ff9640 1 -- 172.21.15.89:0/4132976234 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0364042070 con 0x7f0374101220 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f037c51a740 1 Processor -- start 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f037c51a740 1 -- start start 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f037c51a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c9966742a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f037c51a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9966747e0 0x55c996674c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f037c51a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c99667f580 0x55c996681970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f037c51a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c9965d82f0 con 0x55c9966747e0 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f037c51a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c99657fcf0 con 0x55c99667f580 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f037c51a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c9965d42b0 con 0x55c996675d50 2024-09-15T23:39:04.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f03737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9966747e0 0x55c996674c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f03737fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9966747e0 0x55c996674c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47630/0 (socket says 172.21.15.89:47630) 2024-09-15T23:39:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f03737fe640 1 -- 172.21.15.89:0/3079119992 learned_addr learned my addr 172.21.15.89:0/3079119992 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f0370ff9640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f03600a3240 0x7f03600a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f0373fff640 1 --2- 172.21.15.89:0/3079119992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c9966742a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.677+0000 7f0372ffd640 1 --2- 172.21.15.89:0/3079119992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c99667f580 0x55c996681970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f0373fff640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f03600a3240 0x7f03600a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f0370ff9640 1 -- 172.21.15.89:0/4132976234 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f03640f0620 con 0x7f0374101220 2024-09-15T23:39:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f03737fe640 1 -- 172.21.15.89:0/3079119992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 msgr2=0x55c9966742a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:04.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f03737fe640 1 --2- 172.21.15.89:0/3079119992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c9966742a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f03737fe640 1 -- 172.21.15.89:0/3079119992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c99667f580 msgr2=0x55c996681970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f03737fe640 1 --2- 172.21.15.89:0/3079119992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c99667f580 0x55c996681970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f03737fe640 1 -- 172.21.15.89:0/3079119992 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9965643a0 con 0x55c9966747e0 2024-09-15T23:39:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f03737fe640 1 --2- 172.21.15.89:0/3079119992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9966747e0 0x55c996674c20 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f03640bc550 tx=0x7f036402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a399ded708f8fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f0373fff640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f03600a3240 0x7f03600a5700 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f036c005fd0 tx=0x7f036c004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f0369ffb640 1 -- 172.21.15.89:0/3079119992 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0364049070 con 0x55c9966747e0 2024-09-15T23:39:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f0369ffb640 1 -- 172.21.15.89:0/3079119992 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03640f7040 con 0x55c9966747e0 2024-09-15T23:39:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.678+0000 7f0369ffb640 1 -- 172.21.15.89:0/3079119992 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03640c0c00 con 0x55c9966747e0 2024-09-15T23:39:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 -- 172.21.15.89:0/3079119992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9966747e0 msgr2=0x55c996674c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:04.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 --2- 172.21.15.89:0/3079119992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9966747e0 0x55c996674c20 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f03640bc550 tx=0x7f036402a410 comp rx=0 tx=0).stop 2024-09-15T23:39:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 -- 172.21.15.89:0/3079119992 shutdown_connections 2024-09-15T23:39:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 --2- 172.21.15.89:0/3079119992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c99667f580 0x55c996681970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 --2- 172.21.15.89:0/3079119992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9966747e0 0x55c996674c20 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 --2- 172.21.15.89:0/3079119992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c9966742a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 -- 172.21.15.89:0/3079119992 >> 172.21.15.89:0/3079119992 conn(0x55c99663b310 msgr2=0x55c996670e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 -- 172.21.15.89:0/3079119992 shutdown_connections 2024-09-15T23:39:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 -- 172.21.15.89:0/3079119992 wait complete. 2024-09-15T23:39:04.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 -- start start 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.679+0000 7f037c51a740 1 Processor -- start 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f037c51a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c99667efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f037c51a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c99667c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f037c51a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c99667c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f037c51a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c996682040 con 0x55c99667f580 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f037c51a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c996593060 con 0x55c9966747e0 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f037c51a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c9966821c0 con 0x55c996675d50 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f037c51a740 10 client.0 _ll_register_callbacks cb 0x55c9962e68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0373fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c99667c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0373fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c99667c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47638/0 (socket says 172.21.15.89:47638) 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0373fff640 1 -- 172.21.15.89:0/688632713 learned_addr learned my addr 172.21.15.89:0/688632713 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0372ffd640 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c99667c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f03737fe640 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c99667efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0373fff640 1 -- 172.21.15.89:0/688632713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 msgr2=0x55c99667c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0373fff640 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c99667c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0373fff640 1 -- 172.21.15.89:0/688632713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 msgr2=0x55c99667efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0373fff640 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c99667efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0373fff640 1 -- 172.21.15.89:0/688632713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9963493c0 con 0x55c99667f580 2024-09-15T23:39:04.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f03737fe640 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c99667efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0373fff640 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c99667c9f0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f036c014600 tx=0x7f036c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c11461653de2139 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0352ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.680+0000 7f0352ffd640 1 -- 172.21.15.89:0/688632713 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c025070 con 0x55c99667f580 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f037c51a740 1 -- 172.21.15.89:0/688632713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03640f7040 con 0x55c99667f580 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f0352ffd640 1 -- 172.21.15.89:0/688632713 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f036c014250 con 0x55c99667f580 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f0352ffd640 1 -- 172.21.15.89:0/688632713 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f036c01a070 con 0x55c99667f580 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f037c51a740 10 client.20628 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f037c51a740 1 -- 172.21.15.89:0/688632713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c99668dac0 con 0x55c99667f580 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f0352ffd640 1 -- 172.21.15.89:0/688632713 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03640f7040 con 0x55c99667f580 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f0352ffd640 1 -- 172.21.15.89:0/688632713 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f036c00f040 con 0x55c99667f580 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f037c51a740 10 client.20628 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:04.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f037c51a740 10 client.20628 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:39:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f037c51a740 20 client.20628 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f037c51a740 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c99668eed0 0x55c996691320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f037c51a740 4 client.20628 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f037c51a740 1 -- 172.21.15.89:0/688632713 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9965e5860 con 0x55c99668eed0 2024-09-15T23:39:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.681+0000 7f03737fe640 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c99668eed0 0x55c996691320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.682+0000 7f0352ffd640 10 client.20628 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:39:04.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.682+0000 7f03737fe640 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c99668eed0 0x55c996691320 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=30e180b9060e95ea server_cookie=dd2a10c1f4bf916d in_seq=0 out_seq=0 2024-09-15T23:39:04.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.682+0000 7f0352ffd640 1 -- 172.21.15.89:0/688632713 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c9965e5860 con 0x55c99668eed0 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.682+0000 7f0352ffd640 10 client.20628 handle_command_reply: tid=0 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 client.20628 shutdown 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 -- 172.21.15.89:0/688632713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 msgr2=0x55c99667c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c99667c9f0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f036c014600 tx=0x7f036c004a40 comp rx=0 tx=0).stop 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 -- 172.21.15.89:0/688632713 shutdown_connections 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c99668eed0 0x55c996691320 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c99667c9f0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c99667c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f0352ffd640 0 client.20628 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 --2- 172.21.15.89:0/688632713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c99667efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 -- 172.21.15.89:0/688632713 >> 172.21.15.89:0/688632713 conn(0x55c99663b310 msgr2=0x55c996676b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:04.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 -- 172.21.15.89:0/688632713 shutdown_connections 2024-09-15T23:39:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 1 -- 172.21.15.89:0/688632713 wait complete. 2024-09-15T23:39:04.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.683+0000 7f037c51a740 20 client.20628 trim_cache size 0 max 16384 2024-09-15T23:39:04.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f037c51a740 1 Processor -- start 2024-09-15T23:39:04.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f037c51a740 1 -- start start 2024-09-15T23:39:04.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f037c51a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c996677330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f037c51a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c996677870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f037c51a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c996696bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f037c51a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f03741acbe0 con 0x55c99667f580 2024-09-15T23:39:04.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f037c51a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03741ae720 con 0x55c9966747e0 2024-09-15T23:39:04.704 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f037c51a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03741ad2b0 con 0x55c996675d50 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0373fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c996696bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f03737fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c996677330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0373fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c996696bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47648/0 (socket says 172.21.15.89:47648) 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0373fff640 1 -- 172.21.15.89:0/3535456046 learned_addr learned my addr 172.21.15.89:0/3535456046 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0372ffd640 1 --2- 172.21.15.89:0/3535456046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c996677870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0373fff640 1 -- 172.21.15.89:0/3535456046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 msgr2=0x55c996677870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0373fff640 1 --2- 172.21.15.89:0/3535456046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c996677870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0373fff640 1 -- 172.21.15.89:0/3535456046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 msgr2=0x55c996677330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0373fff640 1 --2- 172.21.15.89:0/3535456046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c996677330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0373fff640 1 -- 172.21.15.89:0/3535456046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c99668b6b0 con 0x55c99667f580 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0372ffd640 1 --2- 172.21.15.89:0/3535456046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c996677870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:04.705 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.696+0000 7f0373fff640 1 --2- 172.21.15.89:0/3535456046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c996696bf0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f036c012010 tx=0x7f036c009e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e782fd98f3c2cfd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f03697fa640 1 -- 172.21.15.89:0/3535456046 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c025040 con 0x55c99667f580 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f03697fa640 1 -- 172.21.15.89:0/3535456046 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f036c014200 con 0x55c99667f580 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f03697fa640 1 -- 172.21.15.89:0/3535456046 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c013730 con 0x55c99667f580 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f037c51a740 1 -- 172.21.15.89:0/3535456046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 msgr2=0x55c996696bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f037c51a740 1 --2- 172.21.15.89:0/3535456046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c996696bf0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f036c012010 tx=0x7f036c009e10 comp rx=0 tx=0).stop 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f037c51a740 1 -- 172.21.15.89:0/3535456046 shutdown_connections 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f037c51a740 1 --2- 172.21.15.89:0/3535456046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c99667f580 0x55c996696bf0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f037c51a740 1 --2- 172.21.15.89:0/3535456046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c996675d50 0x55c996677870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f037c51a740 1 --2- 172.21.15.89:0/3535456046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c996677330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f037c51a740 1 -- 172.21.15.89:0/3535456046 >> 172.21.15.89:0/3535456046 conn(0x55c99663b310 msgr2=0x55c99667e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f037c51a740 1 -- 172.21.15.89:0/3535456046 shutdown_connections 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.697+0000 7f037c51a740 1 -- 172.21.15.89:0/3535456046 wait complete. 2024-09-15T23:39:04.706 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f037c51a740 1 -- start start 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f037c51a740 1 Processor -- start 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f037c51a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c996683220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f037c51a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c996675d50 0x55c996683760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f037c51a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c99667f580 0x55c996683ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f037c51a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f03741ae050 con 0x55c996675d50 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f037c51a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03741ad980 con 0x55c9966747e0 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f037c51a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03741aabc0 con 0x55c99667f580 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f037c51a740 10 client.0 _ll_register_callbacks cb 0x7f036c009b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f0372ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c996675d50 0x55c996683760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f0372ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c996675d50 0x55c996683760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47656/0 (socket says 172.21.15.89:47656) 2024-09-15T23:39:04.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f0372ffd640 1 -- 172.21.15.89:0/500358477 learned_addr learned my addr 172.21.15.89:0/500358477 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f0373fff640 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c99667f580 0x55c996683ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.698+0000 7f03737fe640 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c996683220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0372ffd640 1 -- 172.21.15.89:0/500358477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c99667f580 msgr2=0x55c996683ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0372ffd640 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c99667f580 0x55c996683ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0372ffd640 1 -- 172.21.15.89:0/500358477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 msgr2=0x55c996683220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0372ffd640 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c996683220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0372ffd640 1 -- 172.21.15.89:0/500358477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9966f8d60 con 0x55c996675d50 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0372ffd640 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c996675d50 0x55c996683760 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x55c996675280 tx=0x7f035c00bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71f1ab539ab0fef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0352ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0352ffd640 1 -- 172.21.15.89:0/500358477 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c003cb0 con 0x55c996675d50 2024-09-15T23:39:04.708 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0352ffd640 1 -- 172.21.15.89:0/500358477 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f035c003e50 con 0x55c996675d50 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f037c51a740 1 -- 172.21.15.89:0/500358477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f036c014200 con 0x55c996675d50 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0352ffd640 1 -- 172.21.15.89:0/500358477 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f035c011440 con 0x55c996675d50 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f037c51a740 10 client.20634 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f037c51a740 1 -- 172.21.15.89:0/500358477 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c9966fa0d0 con 0x55c996675d50 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.699+0000 7f0352ffd640 1 -- 172.21.15.89:0/500358477 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f036c014200 con 0x55c996675d50 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f0352ffd640 1 -- 172.21.15.89:0/500358477 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f035c010730 con 0x55c996675d50 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f037c51a740 10 client.20634 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f037c51a740 10 client.20634 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f037c51a740 20 client.20634 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f037c51a740 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c9966fa900 0x55c9966fcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f037c51a740 4 client.20634 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:39:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f037c51a740 1 -- 172.21.15.89:0/500358477 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c9965dfd70 con 0x55c9966fa900 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f03737fe640 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c9966fa900 0x55c9966fcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f0352ffd640 10 client.20634 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f03737fe640 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c9966fa900 0x55c9966fcd20 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dedae7906fc9565a server_cookie=414bd60eae769544 in_seq=0 out_seq=0 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f0352ffd640 1 -- 172.21.15.89:0/500358477 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c9965dfd70 con 0x55c9966fa900 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f0352ffd640 10 client.20634 handle_command_reply: tid=0 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.700+0000 7f037c51a740 1 client.20634 shutdown 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 1 -- 172.21.15.89:0/500358477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c996675d50 msgr2=0x55c996683760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c996675d50 0x55c996683760 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x55c996675280 tx=0x7f035c00bcf0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 1 -- 172.21.15.89:0/500358477 shutdown_connections 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55c9966fa900 0x55c9966fcd20 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c99667f580 0x55c996683ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c996675d50 0x55c996683760 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 1 --2- 172.21.15.89:0/500358477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9966747e0 0x55c996683220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 1 -- 172.21.15.89:0/500358477 >> 172.21.15.89:0/500358477 conn(0x55c99663b310 msgr2=0x55c99667e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f0352ffd640 0 client.20634 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:39:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 1 -- 172.21.15.89:0/500358477 shutdown_connections 2024-09-15T23:39:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 1 -- 172.21.15.89:0/500358477 wait complete. 2024-09-15T23:39:04.711 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037c51a740 20 client.20634 trim_cache size 0 max 16384 2024-09-15T23:39:04.711 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:39:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037a141640 1 -- 172.21.15.89:0/4132976234 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f03600a3240 msgr2=0x7f03600a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037a141640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f03600a3240 0x7f03600a5700 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f036c005fd0 tx=0x7f036c004ab0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037a141640 1 -- 172.21.15.89:0/4132976234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374101220 msgr2=0x7f0374197380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.701+0000 7f037a141640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374101220 0x7f0374197380 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f0364006010 tx=0x7f0364003c10 comp rx=0 tx=0).stop 2024-09-15T23:39:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.702+0000 7f037a141640 1 -- 172.21.15.89:0/4132976234 shutdown_connections 2024-09-15T23:39:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.702+0000 7f037a141640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f03600a3240 0x7f03600a5700 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.702+0000 7f037a141640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0374105a70 0x7f037419bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.702+0000 7f037a141640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0374101ba0 0x7f03741978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.702+0000 7f037a141640 1 --2- 172.21.15.89:0/4132976234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0374101220 0x7f0374197380 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:04.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.702+0000 7f037a141640 1 -- 172.21.15.89:0/4132976234 >> 172.21.15.89:0/4132976234 conn(0x7f03740fbee0 msgr2=0x7f03740fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:04.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.702+0000 7f037a141640 1 -- 172.21.15.89:0/4132976234 shutdown_connections 2024-09-15T23:39:04.714 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:04.702+0000 7f037a141640 1 -- 172.21.15.89:0/4132976234 wait complete. 2024-09-15T23:39:05.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:05 smithi120 ceph-mon[35250]: pgmap v801: 129 pgs: 129 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:39:05.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:05 smithi174 ceph-mon[35586]: pgmap v801: 129 pgs: 129 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:39:05.555 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:05 smithi089 ceph-mon[30933]: pgmap v801: 129 pgs: 129 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 18 op/s 2024-09-15T23:39:05.583 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:39:05.827 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:06.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.299+0000 7f9d4d821640 1 -- 172.21.15.89:0/2061733487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d48101de0 msgr2=0x7f9d481021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:06.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.299+0000 7f9d4d821640 1 --2- 172.21.15.89:0/2061733487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d48101de0 0x7f9d481021c0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009900 tx=0x7f9d3802a410 comp rx=0 tx=0).stop 2024-09-15T23:39:06.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.299+0000 7f9d4d821640 1 -- 172.21.15.89:0/2061733487 shutdown_connections 2024-09-15T23:39:06.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.299+0000 7f9d4d821640 1 --2- 172.21.15.89:0/2061733487 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d48106f10 0x7f9d4810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.299+0000 7f9d4d821640 1 --2- 172.21.15.89:0/2061733487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48102790 0x7f9d481067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.299+0000 7f9d4d821640 1 --2- 172.21.15.89:0/2061733487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d48101de0 0x7f9d481021c0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.299+0000 7f9d4d821640 1 -- 172.21.15.89:0/2061733487 >> 172.21.15.89:0/2061733487 conn(0x7f9d480fd8c0 msgr2=0x7f9d480ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:06.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.299+0000 7f9d4d821640 1 -- 172.21.15.89:0/2061733487 shutdown_connections 2024-09-15T23:39:06.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.299+0000 7f9d4d821640 1 -- 172.21.15.89:0/2061733487 wait complete. 2024-09-15T23:39:06.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.299+0000 7f9d4d821640 1 Processor -- start 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d4d821640 1 -- start start 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d4d821640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d48101de0 0x7f9d481985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d4d821640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d48102790 0x7f9d48198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d4d821640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48106f10 0x7f9d4819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d4d821640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9d48112060 con 0x7f9d48102790 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d4d821640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d48111ee0 con 0x7f9d48106f10 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d4d821640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481121e0 con 0x7f9d48101de0 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d48102790 0x7f9d48198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d467fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d48102790 0x7f9d48198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41088/0 (socket says 172.21.15.89:41088) 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d467fc640 1 -- 172.21.15.89:0/458946701 learned_addr learned my addr 172.21.15.89:0/458946701 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d48101de0 0x7f9d481985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d477fe640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48106f10 0x7f9d4819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.331 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d467fc640 1 -- 172.21.15.89:0/458946701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d48101de0 msgr2=0x7f9d481985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d467fc640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d48101de0 0x7f9d481985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d467fc640 1 -- 172.21.15.89:0/458946701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48106f10 msgr2=0x7f9d4819ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d467fc640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48106f10 0x7f9d4819ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d467fc640 1 -- 172.21.15.89:0/458946701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4819d570 con 0x7f9d48102790 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.300+0000 7f9d477fe640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48106f10 0x7f9d4819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.301+0000 7f9d467fc640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d48102790 0x7f9d48198b00 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000b4a0 tx=0x7f9d3000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.301+0000 7f9d4c81f640 1 -- 172.21.15.89:0/458946701 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d30014070 con 0x7f9d48102790 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.301+0000 7f9d4c81f640 1 -- 172.21.15.89:0/458946701 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d30010af0 con 0x7f9d48102790 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.301+0000 7f9d4d821640 1 -- 172.21.15.89:0/458946701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4819d860 con 0x7f9d48102790 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.301+0000 7f9d4c81f640 1 -- 172.21.15.89:0/458946701 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d30017d10 con 0x7f9d48102790 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.301+0000 7f9d4d821640 1 -- 172.21.15.89:0/458946701 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d481a5140 con 0x7f9d48102790 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.303+0000 7f9d4c81f640 1 -- 172.21.15.89:0/458946701 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9d30017070 con 0x7f9d48102790 2024-09-15T23:39:06.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.303+0000 7f9d4fbfa740 1 Processor -- start 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.303+0000 7f9d4fbfa740 1 -- start start 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.303+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53267b40 0x560d53266090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.303+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d532665d0 0x560d53266a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.303+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53271370 0x560d53273760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560d531cc2f0 con 0x560d53267b40 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560d53173cf0 con 0x560d53271370 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560d531c82b0 con 0x560d532665d0 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d4c81f640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9d340a32c0 0x7f9d340a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53267b40 0x560d53266090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53267b40 0x560d53266090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41104/0 (socket says 172.21.15.89:41104) 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 -- 172.21.15.89:0/3426063062 learned_addr learned my addr 172.21.15.89:0/3426063062 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9d340a32c0 0x7f9d340a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/3426063062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d532665d0 0x560d53266a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d4c81f640 1 -- 172.21.15.89:0/458946701 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9d300ceff0 con 0x7f9d48102790 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d467fc640 1 --2- 172.21.15.89:0/3426063062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53271370 0x560d53273760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 -- 172.21.15.89:0/3426063062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d532665d0 msgr2=0x560d53266a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 --2- 172.21.15.89:0/3426063062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d532665d0 0x560d53266a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 -- 172.21.15.89:0/3426063062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53271370 msgr2=0x560d53273760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 --2- 172.21.15.89:0/3426063062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53271370 0x560d53273760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 -- 172.21.15.89:0/3426063062 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d531583a0 con 0x560d53267b40 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 --2- 172.21.15.89:0/3426063062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53267b40 0x560d53266090 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000ed70 tx=0x7f9d4000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39eadf08ee06d24f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.304+0000 7f9d477fe640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9d340a32c0 0x7f9d340a5780 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000cc40 tx=0x7f9d40019000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.305+0000 7f9d3dffb640 1 -- 172.21.15.89:0/3426063062 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40018b70 con 0x560d53267b40 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.305+0000 7f9d3dffb640 1 -- 172.21.15.89:0/3426063062 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d40018d10 con 0x560d53267b40 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.305+0000 7f9d3dffb640 1 -- 172.21.15.89:0/3426063062 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40023600 con 0x560d53267b40 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.305+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3426063062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53267b40 msgr2=0x560d53266090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:06.334 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.305+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/3426063062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53267b40 0x560d53266090 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000ed70 tx=0x7f9d4000c740 comp rx=0 tx=0).stop 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.305+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3426063062 shutdown_connections 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.305+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/3426063062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53271370 0x560d53273760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.305+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/3426063062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d532665d0 0x560d53266a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.305+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/3426063062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53267b40 0x560d53266090 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.305+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3426063062 >> 172.21.15.89:0/3426063062 conn(0x560d5322f310 msgr2=0x560d53262c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.306+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3426063062 shutdown_connections 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.306+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3426063062 wait complete. 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.306+0000 7f9d4fbfa740 1 -- start start 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.306+0000 7f9d4fbfa740 1 Processor -- start 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.306+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d532665d0 0x560d53270de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.306+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d5326e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.306+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53271370 0x560d5326e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.306+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560d53274010 con 0x560d53271370 2024-09-15T23:39:06.335 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.306+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560d53187060 con 0x560d53267b40 2024-09-15T23:39:06.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.306+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560d53274190 con 0x560d532665d0 2024-09-15T23:39:06.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d4fbfa740 10 client.0 _ll_register_callbacks cb 0x560d52eda8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:06.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53271370 0x560d5326e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53271370 0x560d5326e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41112/0 (socket says 172.21.15.89:41112) 2024-09-15T23:39:06.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d477fe640 1 -- 172.21.15.89:0/4039798626 learned_addr learned my addr 172.21.15.89:0/4039798626 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:06.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d532665d0 0x560d53270de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d467fc640 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d5326e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.336 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d477fe640 1 -- 172.21.15.89:0/4039798626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d532665d0 msgr2=0x560d53270de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d477fe640 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d532665d0 0x560d53270de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d477fe640 1 -- 172.21.15.89:0/4039798626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 msgr2=0x560d5326e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d477fe640 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d5326e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d477fe640 1 -- 172.21.15.89:0/4039798626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d52f3d3c0 con 0x560d53271370 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d477fe640 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53271370 0x560d5326e7d0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009850 tx=0x7f9d400094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20573e9a7e418afc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d26ffd640 1 -- 172.21.15.89:0/4039798626 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4002a070 con 0x560d53271370 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/4039798626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d40018d10 con 0x560d53271370 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d26ffd640 1 -- 172.21.15.89:0/4039798626 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9d4001a620 con 0x560d53271370 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d26ffd640 1 -- 172.21.15.89:0/4039798626 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9d40017040 con 0x560d53271370 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d4fbfa740 10 client.20646 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.307+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/4039798626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560d5327fb40 con 0x560d53271370 2024-09-15T23:39:06.337 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.308+0000 7f9d26ffd640 1 -- 172.21.15.89:0/4039798626 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d40018d10 con 0x560d53271370 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.308+0000 7f9d26ffd640 1 -- 172.21.15.89:0/4039798626 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9d400243f0 con 0x560d53271370 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.308+0000 7f9d4fbfa740 10 client.20646 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.308+0000 7f9d4fbfa740 10 client.20646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.308+0000 7f9d4fbfa740 20 client.20646 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.308+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560d53280f50 0x560d53283340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.308+0000 7f9d4fbfa740 4 client.20646 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.308+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/4039798626 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d531d9860 con 0x560d53280f50 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.308+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560d53280f50 0x560d53283340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560d53280f50 0x560d53283340 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea485207bae54f1e server_cookie=1652261bfd3fb0a4 in_seq=0 out_seq=0 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d26ffd640 10 client.20646 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d26ffd640 1 -- 172.21.15.89:0/4039798626 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560d531d9860 con 0x560d53280f50 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d26ffd640 10 client.20646 handle_command_reply: tid=0 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d4fbfa740 1 client.20646 shutdown 2024-09-15T23:39:06.338 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/4039798626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53271370 msgr2=0x560d5326e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53271370 0x560d5326e7d0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009850 tx=0x7f9d400094c0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/4039798626 shutdown_connections 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560d53280f50 0x560d53283340 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d53271370 0x560d5326e7d0 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d5326e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d26ffd640 0 client.20646 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/4039798626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d532665d0 0x560d53270de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.309+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/4039798626 >> 172.21.15.89:0/4039798626 conn(0x560d5322f310 msgr2=0x560d53268960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.310+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/4039798626 shutdown_connections 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.310+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/4039798626 wait complete. 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.310+0000 7f9d4fbfa740 20 client.20646 trim_cache size 0 max 16384 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.322+0000 7f9d4fbfa740 1 Processor -- start 2024-09-15T23:39:06.339 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.322+0000 7f9d4fbfa740 1 -- start start 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.322+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d5327d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.322+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d53269120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.322+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d53271370 0x560d53269660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.322+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481aefb0 con 0x560d532665d0 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.322+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481b0af0 con 0x560d53267b40 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.322+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481af680 con 0x560d53271370 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d46ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d5327d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d46ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d5327d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41114/0 (socket says 172.21.15.89:41114) 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d46ffd640 1 -- 172.21.15.89:0/2341023454 learned_addr learned my addr 172.21.15.89:0/2341023454 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d46ffd640 1 -- 172.21.15.89:0/2341023454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d53271370 msgr2=0x560d53269660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/2341023454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d53271370 0x560d53269660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d46ffd640 1 -- 172.21.15.89:0/2341023454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 msgr2=0x560d53269120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:06.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/2341023454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d53269120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d46ffd640 1 -- 172.21.15.89:0/2341023454 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d5327d7f0 con 0x560d532665d0 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/2341023454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d5327d2b0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009900 tx=0x7f9d38005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3af91714862ab4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d3d7fa640 1 -- 172.21.15.89:0/2341023454 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d38035040 con 0x560d532665d0 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d3d7fa640 1 -- 172.21.15.89:0/2341023454 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d3804b070 con 0x560d532665d0 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.323+0000 7f9d3d7fa640 1 -- 172.21.15.89:0/2341023454 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d38031810 con 0x560d532665d0 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/2341023454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 msgr2=0x560d5327d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/2341023454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d5327d2b0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009900 tx=0x7f9d38005c00 comp rx=0 tx=0).stop 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/2341023454 shutdown_connections 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/2341023454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d53271370 0x560d53269660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/2341023454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d53269120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/2341023454 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d5327d2b0 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/2341023454 >> 172.21.15.89:0/2341023454 conn(0x560d5322f310 msgr2=0x560d53270110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/2341023454 shutdown_connections 2024-09-15T23:39:06.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/2341023454 wait complete. 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 -- start start 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.324+0000 7f9d4fbfa740 1 Processor -- start 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d53279bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d5326ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d4fbfa740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d53271370 0x560d5326b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481b0420 con 0x560d532665d0 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481afd50 con 0x560d53267b40 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d4fbfa740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481acf90 con 0x560d53271370 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d4fbfa740 10 client.0 _ll_register_callbacks cb 0x7f9d30095ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d46ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d53279bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d46ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d53279bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41118/0 (socket says 172.21.15.89:41118) 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d46ffd640 1 -- 172.21.15.89:0/3585752943 learned_addr learned my addr 172.21.15.89:0/3585752943 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:06.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d477fe640 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d53271370 0x560d5326b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d467fc640 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d5326ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d46ffd640 1 -- 172.21.15.89:0/3585752943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d53271370 msgr2=0x560d5326b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d53271370 0x560d5326b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d46ffd640 1 -- 172.21.15.89:0/3585752943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 msgr2=0x560d5326ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d5326ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d46ffd640 1 -- 172.21.15.89:0/3585752943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d5326bb50 con 0x560d532665d0 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d477fe640 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d53271370 0x560d5326b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d46ffd640 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d53279bc0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f9d38004590 tx=0x7f9d38004fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23c754ef5852072 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d26ffd640 1 -- 172.21.15.89:0/3585752943 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d38053070 con 0x560d532665d0 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3585752943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d3804b070 con 0x560d532665d0 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.325+0000 7f9d26ffd640 1 -- 172.21.15.89:0/3585752943 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9d380046c0 con 0x560d532665d0 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d26ffd640 1 -- 172.21.15.89:0/3585752943 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9d38035050 con 0x560d532665d0 2024-09-15T23:39:06.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d4fbfa740 10 client.20652 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3585752943 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560d532eb910 con 0x560d532665d0 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d26ffd640 1 -- 172.21.15.89:0/3585752943 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d3804b070 con 0x560d532665d0 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d26ffd640 1 -- 172.21.15.89:0/3585752943 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9d380035f0 con 0x560d532665d0 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d4fbfa740 10 client.20652 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d4fbfa740 10 client.20652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d4fbfa740 20 client.20652 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560d532ebf70 0x560d532ee360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d4fbfa740 4 client.20652 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3585752943 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d531d3d70 con 0x560d532ebf70 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.326+0000 7f9d467fc640 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560d532ebf70 0x560d532ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d26ffd640 10 client.20652 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d467fc640 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560d532ebf70 0x560d532ee360 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68f22457be98fbaf server_cookie=1c7a9853f7636325 in_seq=0 out_seq=0 2024-09-15T23:39:06.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d26ffd640 1 -- 172.21.15.89:0/3585752943 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d531d3d70 con 0x560d532ebf70 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d26ffd640 10 client.20652 handle_command_reply: tid=0 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 client.20652 shutdown 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3585752943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 msgr2=0x560d53279bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d53279bc0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f9d38004590 tx=0x7f9d38004fd0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3585752943 shutdown_connections 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560d532ebf70 0x560d532ee360 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560d53271370 0x560d5326b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560d53267b40 0x560d5326ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d26ffd640 0 client.20652 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 --2- 172.21.15.89:0/3585752943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560d532665d0 0x560d53279bc0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3585752943 >> 172.21.15.89:0/3585752943 conn(0x560d5322f310 msgr2=0x560d532614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3585752943 shutdown_connections 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 1 -- 172.21.15.89:0/3585752943 wait complete. 2024-09-15T23:39:06.345 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.327+0000 7f9d4fbfa740 20 client.20652 trim_cache size 0 max 16384 2024-09-15T23:39:06.346 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:39:06.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 -- 172.21.15.89:0/458946701 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9d340a32c0 msgr2=0x7f9d340a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9d340a32c0 0x7f9d340a5780 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000cc40 tx=0x7f9d40019000 comp rx=0 tx=0).stop 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 -- 172.21.15.89:0/458946701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d48102790 msgr2=0x7f9d48198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d48102790 0x7f9d48198b00 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000b4a0 tx=0x7f9d3000b960 comp rx=0 tx=0).stop 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 -- 172.21.15.89:0/458946701 shutdown_connections 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9d340a32c0 0x7f9d340a5780 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48106f10 0x7f9d4819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d48102790 0x7f9d48198b00 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 --2- 172.21.15.89:0/458946701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d48101de0 0x7f9d481985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 -- 172.21.15.89:0/458946701 >> 172.21.15.89:0/458946701 conn(0x7f9d480fd8c0 msgr2=0x7f9d480fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 -- 172.21.15.89:0/458946701 shutdown_connections 2024-09-15T23:39:06.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:06.328+0000 7f9d4d821640 1 -- 172.21.15.89:0/458946701 wait complete. 2024-09-15T23:39:07.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:07 smithi089 ceph-mon[30933]: pgmap v802: 129 pgs: 129 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:07.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:07 smithi120 ceph-mon[35250]: pgmap v802: 129 pgs: 129 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:07.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:07 smithi174 ceph-mon[35586]: pgmap v802: 129 pgs: 129 active+clean; 3.5 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:08.028 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:39:08.028 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:39:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.280+0000 7f6d09c74640 1 -- 172.21.15.89:0/2371540181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d0410e670 msgr2=0x7f6d04110b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.280+0000 7f6d09c74640 1 --2- 172.21.15.89:0/2371540181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d0410e670 0x7f6d04110b30 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc009ca0 tx=0x7f6cfc02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.280+0000 7f6d09c74640 1 -- 172.21.15.89:0/2371540181 shutdown_connections 2024-09-15T23:39:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.280+0000 7f6d09c74640 1 --2- 172.21.15.89:0/2371540181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d0410e670 0x7f6d04110b30 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.280+0000 7f6d09c74640 1 --2- 172.21.15.89:0/2371540181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d0410bbb0 0x7f6d0410dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.280+0000 7f6d09c74640 1 --2- 172.21.15.89:0/2371540181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d041091f0 0x7f6d0410b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.280+0000 7f6d09c74640 1 -- 172.21.15.89:0/2371540181 >> 172.21.15.89:0/2371540181 conn(0x7f6d0406ef20 msgr2=0x7f6d04071360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.280+0000 7f6d09c74640 1 -- 172.21.15.89:0/2371540181 shutdown_connections 2024-09-15T23:39:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.280+0000 7f6d09c74640 1 -- 172.21.15.89:0/2371540181 wait complete. 2024-09-15T23:39:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.280+0000 7f6d09c74640 1 Processor -- start 2024-09-15T23:39:08.282 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d09c74640 1 -- start start 2024-09-15T23:39:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d09c74640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d041091f0 0x7f6d0410a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d09c74640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d0410bbb0 0x7f6d041a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d09c74640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d0410e670 0x7f6d041a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d09c74640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6d04114750 con 0x7f6d0410e670 2024-09-15T23:39:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d09c74640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6d041145d0 con 0x7f6d0410bbb0 2024-09-15T23:39:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d09c74640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6d041148d0 con 0x7f6d041091f0 2024-09-15T23:39:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d03fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d0410e670 0x7f6d041a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d03fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d0410e670 0x7f6d041a7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41140/0 (socket says 172.21.15.89:41140) 2024-09-15T23:39:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d03fff640 1 -- 172.21.15.89:0/2783474784 learned_addr learned my addr 172.21.15.89:0/2783474784 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d037fe640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d041091f0 0x7f6d0410a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d0410bbb0 0x7f6d041a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d03fff640 1 -- 172.21.15.89:0/2783474784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d041091f0 msgr2=0x7f6d0410a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d03fff640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d041091f0 0x7f6d0410a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d03fff640 1 -- 172.21.15.89:0/2783474784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d0410bbb0 msgr2=0x7f6d041a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d03fff640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d0410bbb0 0x7f6d041a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.281+0000 7f6d03fff640 1 -- 172.21.15.89:0/2783474784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d041a8340 con 0x7f6d0410e670 2024-09-15T23:39:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.282+0000 7f6d03fff640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d0410e670 0x7f6d041a7a30 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0098d0 tx=0x7f6cfc02f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.282+0000 7f6d00ff9640 1 -- 172.21.15.89:0/2783474784 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc02fc30 con 0x7f6d0410e670 2024-09-15T23:39:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.282+0000 7f6d00ff9640 1 -- 172.21.15.89:0/2783474784 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cfc003e50 con 0x7f6d0410e670 2024-09-15T23:39:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.282+0000 7f6d09c74640 1 -- 172.21.15.89:0/2783474784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d041a8570 con 0x7f6d0410e670 2024-09-15T23:39:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.282+0000 7f6d00ff9640 1 -- 172.21.15.89:0/2783474784 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc004810 con 0x7f6d0410e670 2024-09-15T23:39:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.282+0000 7f6d09c74640 1 -- 172.21.15.89:0/2783474784 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d041a8b40 con 0x7f6d0410e670 2024-09-15T23:39:08.287 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:08.297 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.284+0000 7f6d0ca17740 1 Processor -- start 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d00ff9640 1 -- 172.21.15.89:0/2783474784 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6cfc003970 con 0x7f6d0410e670 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d0ca17740 1 -- start start 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb377db0 0x5580fb3781b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb3786f0 0x5580fb37d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb37d780 0x5580fb37fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5580fb2d72f0 con 0x5580fb377db0 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580fb27f270 con 0x5580fb37d780 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5580fb2d32b0 con 0x5580fb3786f0 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb377db0 0x5580fb3781b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb377db0 0x5580fb3781b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41146/0 (socket says 172.21.15.89:41146) 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d02ffd640 1 -- 172.21.15.89:0/4171071475 learned_addr learned my addr 172.21.15.89:0/4171071475 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:08.298 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/4171071475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb37d780 0x5580fb37fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.285+0000 7f6d037fe640 1 --2- 172.21.15.89:0/4171071475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb3786f0 0x5580fb37d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6d00ff9640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cec0a32f0 0x7f6cec0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6d02ffd640 1 -- 172.21.15.89:0/4171071475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb3786f0 msgr2=0x5580fb37d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/4171071475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb3786f0 0x5580fb37d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6d02ffd640 1 -- 172.21.15.89:0/4171071475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb37d780 msgr2=0x5580fb37fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/4171071475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb37d780 0x5580fb37fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6d02ffd640 1 -- 172.21.15.89:0/4171071475 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580fb27e340 con 0x5580fb377db0 2024-09-15T23:39:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6d037fe640 1 --2- 172.21.15.89:0/4171071475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb3786f0 0x5580fb37d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6d03fff640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cec0a32f0 0x7f6cec0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.299 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/4171071475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb377db0 0x5580fb3781b0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80148a0 tx=0x7f6cf800af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d724295fb31b74 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6d00ff9640 1 -- 172.21.15.89:0/2783474784 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6cfc0f11b0 con 0x7f6d0410e670 2024-09-15T23:39:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6ce9ffb640 1 -- 172.21.15.89:0/4171071475 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8014310 con 0x5580fb377db0 2024-09-15T23:39:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6ce9ffb640 1 -- 172.21.15.89:0/4171071475 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf800f970 con 0x5580fb377db0 2024-09-15T23:39:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.286+0000 7f6ce9ffb640 1 -- 172.21.15.89:0/4171071475 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8005280 con 0x5580fb377db0 2024-09-15T23:39:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d0ca17740 1 -- 172.21.15.89:0/4171071475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb377db0 msgr2=0x5580fb3781b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/4171071475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb377db0 0x5580fb3781b0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f6cf80148a0 tx=0x7f6cf800af90 comp rx=0 tx=0).stop 2024-09-15T23:39:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d03fff640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cec0a32f0 0x7f6cec0a57b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0b9db0 tx=0x7f6cfc0093d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d0ca17740 1 -- 172.21.15.89:0/4171071475 shutdown_connections 2024-09-15T23:39:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/4171071475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb37d780 0x5580fb37fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/4171071475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb3786f0 0x5580fb37d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/4171071475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb377db0 0x5580fb3781b0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d0ca17740 1 -- 172.21.15.89:0/4171071475 >> 172.21.15.89:0/4171071475 conn(0x5580fb36f010 msgr2=0x5580fb36f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d0ca17740 1 -- 172.21.15.89:0/4171071475 shutdown_connections 2024-09-15T23:39:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d0ca17740 1 -- 172.21.15.89:0/4171071475 wait complete. 2024-09-15T23:39:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.287+0000 7f6d0ca17740 1 -- start start 2024-09-15T23:39:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d0ca17740 1 Processor -- start 2024-09-15T23:39:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 0x5580fb382060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb3825a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb37d780 0x5580fb3895e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5580fb380750 con 0x5580fb3786f0 2024-09-15T23:39:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5580fb260290 con 0x5580fb377db0 2024-09-15T23:39:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5580fb3808d0 con 0x5580fb37d780 2024-09-15T23:39:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d0ca17740 10 client.0 _ll_register_callbacks cb 0x5580fb291140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d037fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb3825a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d037fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb3825a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41160/0 (socket says 172.21.15.89:41160) 2024-09-15T23:39:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d037fe640 1 -- 172.21.15.89:0/927299868 learned_addr learned my addr 172.21.15.89:0/927299868 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb37d780 0x5580fb3895e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 0x5580fb382060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d037fe640 1 -- 172.21.15.89:0/927299868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb37d780 msgr2=0x5580fb3895e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d037fe640 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb37d780 0x5580fb3895e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d037fe640 1 -- 172.21.15.89:0/927299868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 msgr2=0x5580fb382060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d037fe640 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 0x5580fb382060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d037fe640 1 -- 172.21.15.89:0/927299868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580fafea210 con 0x5580fb3786f0 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 0x5580fb382060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6d037fe640 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb3825a0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009e80 tx=0x7f6cf4011630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92037a1dfcab580 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.288+0000 7f6ce2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/927299868 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4019070 con 0x5580fb3786f0 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/927299868 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6cf4015660 con 0x5580fb3786f0 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6d0ca17740 1 -- 172.21.15.89:0/927299868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cf800f970 con 0x5580fb3786f0 2024-09-15T23:39:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/927299868 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6cf4025050 con 0x5580fb3786f0 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6d0ca17740 10 client.20664 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6d0ca17740 1 -- 172.21.15.89:0/927299868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5580fb0b6620 con 0x5580fb3786f0 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/927299868 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf800f970 con 0x5580fb3786f0 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/927299868 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6cf4014a80 con 0x5580fb3786f0 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6d0ca17740 10 client.20664 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6d0ca17740 10 client.20664 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6d0ca17740 20 client.20664 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580fb38dce0 0x5580fb390110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6d0ca17740 4 client.20664 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.289+0000 7f6d0ca17740 1 -- 172.21.15.89:0/927299868 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580fb2e4a20 con 0x5580fb38dce0 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.290+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580fb38dce0 0x5580fb390110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.290+0000 7f6ce2ffd640 10 client.20664 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.290+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580fb38dce0 0x5580fb390110 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0b123b0387a234e server_cookie=427e73166d72d129 in_seq=0 out_seq=0 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/927299868 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5580fb2e4a20 con 0x5580fb38dce0 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6ce2ffd640 10 client.20664 handle_command_reply: tid=0 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 client.20664 shutdown 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 -- 172.21.15.89:0/927299868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 msgr2=0x5580fb3825a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb3825a0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009e80 tx=0x7f6cf4011630 comp rx=0 tx=0).stop 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 -- 172.21.15.89:0/927299868 shutdown_connections 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580fb38dce0 0x5580fb390110 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb37d780 0x5580fb3895e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6ce2ffd640 0 client.20664 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb3825a0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/927299868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 0x5580fb382060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.305 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 -- 172.21.15.89:0/927299868 >> 172.21.15.89:0/927299868 conn(0x5580fb36f010 msgr2=0x5580fb370820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 -- 172.21.15.89:0/927299868 shutdown_connections 2024-09-15T23:39:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 1 -- 172.21.15.89:0/927299868 wait complete. 2024-09-15T23:39:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.291+0000 7f6d0ca17740 20 client.20664 trim_cache size 0 max 16384 2024-09-15T23:39:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.307+0000 7f6d0ca17740 1 Processor -- start 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d0ca17740 1 -- start start 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb36f010 0x5580fb370400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb377db0 0x5580fb3895a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb370940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6d041b0f20 con 0x5580fb3786f0 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6d041b2a60 con 0x5580fb36f010 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6d041b15f0 con 0x5580fb377db0 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb370940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d037fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb377db0 0x5580fb3895a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb370940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41176/0 (socket says 172.21.15.89:41176) 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d02ffd640 1 -- 172.21.15.89:0/1568577374 learned_addr learned my addr 172.21.15.89:0/1568577374 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/1568577374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb36f010 0x5580fb370400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d02ffd640 1 -- 172.21.15.89:0/1568577374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb377db0 msgr2=0x5580fb3895a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/1568577374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb377db0 0x5580fb3895a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d02ffd640 1 -- 172.21.15.89:0/1568577374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb36f010 msgr2=0x5580fb370400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/1568577374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb36f010 0x5580fb370400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d02ffd640 1 -- 172.21.15.89:0/1568577374 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580fb378bd0 con 0x5580fb3786f0 2024-09-15T23:39:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d037fe640 1 --2- 172.21.15.89:0/1568577374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb377db0 0x5580fb3895a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.308+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/1568577374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb370940 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8014f50 tx=0x7f6cf800af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7cd199a5566e402 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6ce97fa640 1 -- 172.21.15.89:0/1568577374 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf80142b0 con 0x5580fb3786f0 2024-09-15T23:39:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6ce97fa640 1 -- 172.21.15.89:0/1568577374 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf800fe50 con 0x5580fb3786f0 2024-09-15T23:39:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6ce97fa640 1 -- 172.21.15.89:0/1568577374 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8004ca0 con 0x5580fb3786f0 2024-09-15T23:39:08.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:39:08.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:39:08.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:39:08.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:39:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6d0ca17740 1 -- 172.21.15.89:0/1568577374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 msgr2=0x5580fb370940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/1568577374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb370940 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8014f50 tx=0x7f6cf800af90 comp rx=0 tx=0).stop 2024-09-15T23:39:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6d0ca17740 1 -- 172.21.15.89:0/1568577374 shutdown_connections 2024-09-15T23:39:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/1568577374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb370940 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/1568577374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb377db0 0x5580fb3895a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/1568577374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb36f010 0x5580fb370400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6d0ca17740 1 -- 172.21.15.89:0/1568577374 >> 172.21.15.89:0/1568577374 conn(0x5580fb37d310 msgr2=0x5580fb36e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.309+0000 7f6d0ca17740 1 -- 172.21.15.89:0/1568577374 shutdown_connections 2024-09-15T23:39:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d0ca17740 1 -- 172.21.15.89:0/1568577374 wait complete. 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d0ca17740 1 -- start start 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d0ca17740 1 Processor -- start 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb36f010 0x5580fb387030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 0x5580fb387570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d0ca17740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb388530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6d041b2390 con 0x5580fb3786f0 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6d041b1cc0 con 0x5580fb377db0 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d0ca17740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6d041aef00 con 0x5580fb36f010 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d0ca17740 10 client.0 _ll_register_callbacks cb 0x7f6cf40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb388530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb388530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41190/0 (socket says 172.21.15.89:41190) 2024-09-15T23:39:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.310+0000 7f6d02ffd640 1 -- 172.21.15.89:0/576945612 learned_addr learned my addr 172.21.15.89:0/576945612 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb36f010 0x5580fb387030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d037fe640 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 0x5580fb387570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d02ffd640 1 -- 172.21.15.89:0/576945612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb36f010 msgr2=0x5580fb387030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb36f010 0x5580fb387030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d02ffd640 1 -- 172.21.15.89:0/576945612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 msgr2=0x5580fb387570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 0x5580fb387570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d02ffd640 1 -- 172.21.15.89:0/576945612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580fb388b40 con 0x5580fb3786f0 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d037fe640 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 0x5580fb387570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d02ffd640 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb388530 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x5580fb386600 tx=0x7f6cf800a8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc60497cf973a7ac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6ce2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/576945612 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8014110 con 0x5580fb3786f0 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/576945612 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6cf80142b0 con 0x5580fb3786f0 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d0ca17740 1 -- 172.21.15.89:0/576945612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cf800fe50 con 0x5580fb3786f0 2024-09-15T23:39:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/576945612 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6cf80144b0 con 0x5580fb3786f0 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d0ca17740 10 client.20670 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6d0ca17740 1 -- 172.21.15.89:0/576945612 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5580fb3f8cf0 con 0x5580fb3786f0 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.311+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/576945612 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf800fe50 con 0x5580fb3786f0 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.312+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/576945612 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6cf8033040 con 0x5580fb3786f0 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.312+0000 7f6d0ca17740 10 client.20670 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.312+0000 7f6d0ca17740 10 client.20670 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.312+0000 7f6d0ca17740 20 client.20670 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.312+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580fb37d780 0x5580fb3fb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.312+0000 7f6d0ca17740 4 client.20670 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.312+0000 7f6d0ca17740 1 -- 172.21.15.89:0/576945612 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5580fb2ded70 con 0x5580fb37d780 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.312+0000 7f6d037fe640 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580fb37d780 0x5580fb3fb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.312+0000 7f6ce2ffd640 10 client.20670 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.312+0000 7f6d037fe640 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580fb37d780 0x5580fb3fb3a0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e3a52ecb54cb72c server_cookie=33819b0fb5d0bac2 in_seq=0 out_seq=0 2024-09-15T23:39:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6ce2ffd640 1 -- 172.21.15.89:0/576945612 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5580fb2ded70 con 0x5580fb37d780 2024-09-15T23:39:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6ce2ffd640 10 client.20670 handle_command_reply: tid=0 2024-09-15T23:39:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 client.20670 shutdown 2024-09-15T23:39:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 -- 172.21.15.89:0/576945612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 msgr2=0x5580fb388530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb388530 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x5580fb386600 tx=0x7f6cf800a8a0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 -- 172.21.15.89:0/576945612 shutdown_connections 2024-09-15T23:39:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5580fb37d780 0x5580fb3fb3a0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5580fb3786f0 0x5580fb388530 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6ce2ffd640 0 client.20670 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5580fb377db0 0x5580fb387570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 --2- 172.21.15.89:0/576945612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580fb36f010 0x5580fb387030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 -- 172.21.15.89:0/576945612 >> 172.21.15.89:0/576945612 conn(0x5580fb37d310 msgr2=0x5580fb36f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 -- 172.21.15.89:0/576945612 shutdown_connections 2024-09-15T23:39:08.327 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.324+0000 7f6d0ca17740 1 -- 172.21.15.89:0/576945612 wait complete. 2024-09-15T23:39:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d0ca17740 20 client.20670 trim_cache size 0 max 16384 2024-09-15T23:39:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:39:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:39:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:39:08.328 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:39:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 -- 172.21.15.89:0/2783474784 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cec0a32f0 msgr2=0x7f6cec0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cec0a32f0 0x7f6cec0a57b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0b9db0 tx=0x7f6cfc0093d0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 -- 172.21.15.89:0/2783474784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d0410e670 msgr2=0x7f6d041a7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d0410e670 0x7f6d041a7a30 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc0098d0 tx=0x7f6cfc02f840 comp rx=0 tx=0).stop 2024-09-15T23:39:08.329 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 -- 172.21.15.89:0/2783474784 shutdown_connections 2024-09-15T23:39:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6cec0a32f0 0x7f6cec0a57b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d0410e670 0x7f6d041a7a30 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d0410bbb0 0x7f6d041a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 --2- 172.21.15.89:0/2783474784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d041091f0 0x7f6d0410a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 -- 172.21.15.89:0/2783474784 >> 172.21.15.89:0/2783474784 conn(0x7f6d0406ef20 msgr2=0x7f6d0410a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 -- 172.21.15.89:0/2783474784 shutdown_connections 2024-09-15T23:39:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:08.325+0000 7f6d09c74640 1 -- 172.21.15.89:0/2783474784 wait complete. 2024-09-15T23:39:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:39:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:39:08.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:39:08.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:39:08.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:39:08.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:39:08.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:39:08.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:39:08.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.772+0000 7f0657cea640 1 -- 172.21.15.89:0/186843463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0650101de0 msgr2=0x7f06501021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.772+0000 7f0657cea640 1 --2- 172.21.15.89:0/186843463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0650101de0 0x7f06501021c0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f0644009900 tx=0x7f064402a410 comp rx=0 tx=0).stop 2024-09-15T23:39:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.772+0000 7f0657cea640 1 -- 172.21.15.89:0/186843463 shutdown_connections 2024-09-15T23:39:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.772+0000 7f0657cea640 1 --2- 172.21.15.89:0/186843463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0650106f10 0x7f065010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.772+0000 7f0657cea640 1 --2- 172.21.15.89:0/186843463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0650102790 0x7f06501067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.772+0000 7f0657cea640 1 --2- 172.21.15.89:0/186843463 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0650101de0 0x7f06501021c0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.772+0000 7f0657cea640 1 -- 172.21.15.89:0/186843463 >> 172.21.15.89:0/186843463 conn(0x7f06500fd8c0 msgr2=0x7f06500ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.772+0000 7f0657cea640 1 -- 172.21.15.89:0/186843463 shutdown_connections 2024-09-15T23:39:08.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.772+0000 7f0657cea640 1 -- 172.21.15.89:0/186843463 wait complete. 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0657cea640 1 Processor -- start 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0657cea640 1 -- start start 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0657cea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0650101de0 0x7f0650198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0657cea640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0650102790 0x7f0650198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0657cea640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0650106f10 0x7f065019cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0657cea640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0650111f70 con 0x7f0650106f10 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0657cea640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0650111df0 con 0x7f0650102790 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0657cea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06501120f0 con 0x7f0650101de0 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0656287640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0650106f10 0x7f065019cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0656287640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0650106f10 0x7f065019cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41210/0 (socket says 172.21.15.89:41210) 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0656287640 1 -- 172.21.15.89:0/4181573010 learned_addr learned my addr 172.21.15.89:0/4181573010 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0655285640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0650102790 0x7f0650198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.791 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0655a86640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0650101de0 0x7f0650198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0656287640 1 -- 172.21.15.89:0/4181573010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0650101de0 msgr2=0x7f0650198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0656287640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0650101de0 0x7f0650198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0656287640 1 -- 172.21.15.89:0/4181573010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0650102790 msgr2=0x7f0650198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0656287640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0650102790 0x7f0650198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.773+0000 7f0656287640 1 -- 172.21.15.89:0/4181573010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065019d2b0 con 0x7f0650106f10 2024-09-15T23:39:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.774+0000 7f0656287640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0650106f10 0x7f065019cbd0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f064800cce0 tx=0x7f064800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.774+0000 7f0642ffd640 1 -- 172.21.15.89:0/4181573010 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064800f450 con 0x7f0650106f10 2024-09-15T23:39:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.774+0000 7f0657cea640 1 -- 172.21.15.89:0/4181573010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f065019d5a0 con 0x7f0650106f10 2024-09-15T23:39:08.792 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.774+0000 7f0642ffd640 1 -- 172.21.15.89:0/4181573010 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f064800fad0 con 0x7f0650106f10 2024-09-15T23:39:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.774+0000 7f0642ffd640 1 -- 172.21.15.89:0/4181573010 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648018b60 con 0x7f0650106f10 2024-09-15T23:39:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.774+0000 7f0657cea640 1 -- 172.21.15.89:0/4181573010 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06501a4e80 con 0x7f0650106f10 2024-09-15T23:39:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.776+0000 7f0642ffd640 1 -- 172.21.15.89:0/4181573010 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0648018d00 con 0x7f0650106f10 2024-09-15T23:39:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.776+0000 7f065a0c3740 1 Processor -- start 2024-09-15T23:39:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f065a0c3740 1 -- start start 2024-09-15T23:39:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0642ffd640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f063c0a3240 0x7f063c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aaccb60 0x560f9aaccf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aacd480 0x560f9aad7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aad7d40 0x560f9aada130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.793 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f065a0c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560f9aa32280 con 0x560f9aad7d40 2024-09-15T23:39:08.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f065a0c3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560f9a9d9b10 con 0x560f9aaccb60 2024-09-15T23:39:08.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f065a0c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560f9aa2e240 con 0x560f9aacd480 2024-09-15T23:39:08.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0656287640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aad7d40 0x560f9aada130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0655285640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f063c0a3240 0x7f063c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0642ffd640 1 -- 172.21.15.89:0/4181573010 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f06480d0170 con 0x7f0650106f10 2024-09-15T23:39:08.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0656287640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aad7d40 0x560f9aada130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41214/0 (socket says 172.21.15.89:41214) 2024-09-15T23:39:08.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0656287640 1 -- 172.21.15.89:0/4081168832 learned_addr learned my addr 172.21.15.89:0/4081168832 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:08.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0655285640 1 --2- 172.21.15.89:0/4081168832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aacd480 0x560f9aad7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0655a86640 1 --2- 172.21.15.89:0/4081168832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aaccb60 0x560f9aaccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0656287640 1 -- 172.21.15.89:0/4081168832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aacd480 msgr2=0x560f9aad7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0656287640 1 --2- 172.21.15.89:0/4081168832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aacd480 0x560f9aad7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0656287640 1 -- 172.21.15.89:0/4081168832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aaccb60 msgr2=0x560f9aaccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0656287640 1 --2- 172.21.15.89:0/4081168832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aaccb60 0x560f9aaccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0656287640 1 -- 172.21.15.89:0/4081168832 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f9a9e61f0 con 0x560f9aad7d40 2024-09-15T23:39:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0655285640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f063c0a3240 0x7f063c0a5700 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f064c0058c0 tx=0x7f064c0138f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.777+0000 7f0656287640 1 --2- 172.21.15.89:0/4081168832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aad7d40 0x560f9aada130 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f06480cd8d0 tx=0x7f0648051630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3022259e631faffd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f063bfff640 1 -- 172.21.15.89:0/4081168832 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064800d340 con 0x560f9aad7d40 2024-09-15T23:39:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f063bfff640 1 -- 172.21.15.89:0/4081168832 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f064809caa0 con 0x560f9aad7d40 2024-09-15T23:39:08.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f063bfff640 1 -- 172.21.15.89:0/4081168832 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0648013ae0 con 0x560f9aad7d40 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f065a0c3740 1 -- 172.21.15.89:0/4081168832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aad7d40 msgr2=0x560f9aada130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f065a0c3740 1 --2- 172.21.15.89:0/4081168832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aad7d40 0x560f9aada130 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f06480cd8d0 tx=0x7f0648051630 comp rx=0 tx=0).stop 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f065a0c3740 1 -- 172.21.15.89:0/4081168832 shutdown_connections 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f065a0c3740 1 --2- 172.21.15.89:0/4081168832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aad7d40 0x560f9aada130 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f065a0c3740 1 --2- 172.21.15.89:0/4081168832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aacd480 0x560f9aad7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f065a0c3740 1 --2- 172.21.15.89:0/4081168832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aaccb60 0x560f9aaccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f065a0c3740 1 -- 172.21.15.89:0/4081168832 >> 172.21.15.89:0/4081168832 conn(0x560f9aac90f0 msgr2=0x560f9aac94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f065a0c3740 1 -- 172.21.15.89:0/4081168832 shutdown_connections 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.778+0000 7f065a0c3740 1 -- 172.21.15.89:0/4081168832 wait complete. 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f065a0c3740 1 -- start start 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f065a0c3740 1 Processor -- start 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 0x560f9aad5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 0x560f9aad56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f065a0c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560f9aada910 con 0x560f9aaccb60 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f065a0c3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560f9a9ecf10 con 0x560f9aacd480 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f065a0c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560f9aadaa90 con 0x560f9aad7d40 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f065a0c3740 10 client.0 _ll_register_callbacks cb 0x560f9a9b5a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f0655a86640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f0655a86640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41226/0 (socket says 172.21.15.89:41226) 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f0655a86640 1 -- 172.21.15.89:0/3554619614 learned_addr learned my addr 172.21.15.89:0/3554619614 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.779+0000 7f0655285640 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 0x560f9aad5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f0656287640 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 0x560f9aad56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f0655a86640 1 -- 172.21.15.89:0/3554619614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 msgr2=0x560f9aad56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f0655a86640 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 0x560f9aad56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f0655a86640 1 -- 172.21.15.89:0/3554619614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 msgr2=0x560f9aad5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.797 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f0655a86640 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 0x560f9aad5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f0655a86640 1 -- 172.21.15.89:0/3554619614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f9a7a36a0 con 0x560f9aaccb60 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f0656287640 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 0x560f9aad56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f0655a86640 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad4c20 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f0644009900 tx=0x7f0644031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974e4676cab1aa25 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f06397fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f06397fa640 1 -- 172.21.15.89:0/3554619614 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0644035040 con 0x560f9aaccb60 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f06397fa640 1 -- 172.21.15.89:0/3554619614 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0644004240 con 0x560f9aaccb60 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f065a0c3740 1 -- 172.21.15.89:0/3554619614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f064809caa0 con 0x560f9aaccb60 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f06397fa640 1 -- 172.21.15.89:0/3554619614 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f064404e050 con 0x560f9aaccb60 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f065a0c3740 10 client.20682 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f065a0c3740 1 -- 172.21.15.89:0/3554619614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560f9aae6510 con 0x560f9aaccb60 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.780+0000 7f06397fa640 1 -- 172.21.15.89:0/3554619614 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f064809caa0 con 0x560f9aaccb60 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.781+0000 7f06397fa640 1 -- 172.21.15.89:0/3554619614 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0644004ba0 con 0x560f9aaccb60 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.781+0000 7f065a0c3740 10 client.20682 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:08.798 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.781+0000 7f065a0c3740 10 client.20682 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.781+0000 7f065a0c3740 20 client.20682 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.781+0000 7f065a0c3740 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560f9aae79d0 0x560f9aae9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.781+0000 7f065a0c3740 4 client.20682 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.781+0000 7f065a0c3740 1 -- 172.21.15.89:0/3554619614 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f9aa3f7f0 con 0x560f9aae79d0 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.781+0000 7f0655285640 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560f9aae79d0 0x560f9aae9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.781+0000 7f06397fa640 10 client.20682 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.781+0000 7f0655285640 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560f9aae79d0 0x560f9aae9e50 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9576df2108c0d61 server_cookie=5d1dfa8c41e1b83a in_seq=0 out_seq=0 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f06397fa640 1 -- 172.21.15.89:0/3554619614 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560f9aa3f7f0 con 0x560f9aae79d0 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f06397fa640 10 client.20682 handle_command_reply: tid=0 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 client.20682 shutdown 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 -- 172.21.15.89:0/3554619614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 msgr2=0x560f9aad4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad4c20 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f0644009900 tx=0x7f0644031920 comp rx=0 tx=0).stop 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 -- 172.21.15.89:0/3554619614 shutdown_connections 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560f9aae79d0 0x560f9aae9e50 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 0x560f9aad56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 0x560f9aad5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 --2- 172.21.15.89:0/3554619614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad4c20 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f06397fa640 0 client.20682 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:08.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 -- 172.21.15.89:0/3554619614 >> 172.21.15.89:0/3554619614 conn(0x560f9aac90f0 msgr2=0x560f9aacee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 -- 172.21.15.89:0/3554619614 shutdown_connections 2024-09-15T23:39:08.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 1 -- 172.21.15.89:0/3554619614 wait complete. 2024-09-15T23:39:08.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.782+0000 7f065a0c3740 20 client.20682 trim_cache size 0 max 16384 2024-09-15T23:39:08.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.799+0000 7f065a0c3740 1 Processor -- start 2024-09-15T23:39:08.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.799+0000 7f065a0c3740 1 -- start start 2024-09-15T23:39:08.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.799+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.799+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 0x560f9aae00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.799+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 0x560f9aae0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.799+0000 7f065a0c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f06501aed30 con 0x560f9aaccb60 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.799+0000 7f065a0c3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f06501b0870 con 0x560f9aacd480 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.799+0000 7f065a0c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06501af400 con 0x560f9aad7d40 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655a86640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655a86640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41232/0 (socket says 172.21.15.89:41232) 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655a86640 1 -- 172.21.15.89:0/967309693 learned_addr learned my addr 172.21.15.89:0/967309693 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0656287640 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 0x560f9aae0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655a86640 1 -- 172.21.15.89:0/967309693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 msgr2=0x560f9aae0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655285640 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 0x560f9aae00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655a86640 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 0x560f9aae0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655a86640 1 -- 172.21.15.89:0/967309693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 msgr2=0x560f9aae00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655a86640 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 0x560f9aae00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655a86640 1 -- 172.21.15.89:0/967309693 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f9aad74f0 con 0x560f9aaccb60 2024-09-15T23:39:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0656287640 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 0x560f9aae0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655285640 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 0x560f9aae00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f0655a86640 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad6a80 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f06440096a0 tx=0x7f0644005980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2b9a638a9de58f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f063b7fe640 1 -- 172.21.15.89:0/967309693 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0644035030 con 0x560f9aaccb60 2024-09-15T23:39:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f063b7fe640 1 -- 172.21.15.89:0/967309693 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0644031990 con 0x560f9aaccb60 2024-09-15T23:39:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.800+0000 7f063b7fe640 1 -- 172.21.15.89:0/967309693 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0644004d70 con 0x560f9aaccb60 2024-09-15T23:39:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 -- 172.21.15.89:0/967309693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 msgr2=0x560f9aad6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad6a80 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f06440096a0 tx=0x7f0644005980 comp rx=0 tx=0).stop 2024-09-15T23:39:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 -- 172.21.15.89:0/967309693 shutdown_connections 2024-09-15T23:39:08.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aad7d40 0x560f9aae0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aacd480 0x560f9aae00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 --2- 172.21.15.89:0/967309693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9aad6a80 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 -- 172.21.15.89:0/967309693 >> 172.21.15.89:0/967309693 conn(0x560f9aac90f0 msgr2=0x560f9aad6370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 -- 172.21.15.89:0/967309693 shutdown_connections 2024-09-15T23:39:08.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 -- 172.21.15.89:0/967309693 wait complete. 2024-09-15T23:39:08.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 -- start start 2024-09-15T23:39:08.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.801+0000 7f065a0c3740 1 Processor -- start 2024-09-15T23:39:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9ab4a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aacd480 0x560f9ab4abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f065a0c3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aad7d40 0x560f9ab55730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f065a0c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f06501b01a0 con 0x560f9aaccb60 2024-09-15T23:39:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f065a0c3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f06501afad0 con 0x560f9aad7d40 2024-09-15T23:39:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f065a0c3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06501acd10 con 0x560f9aacd480 2024-09-15T23:39:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f065a0c3740 10 client.0 _ll_register_callbacks cb 0x7f0644005d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0655a86640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9ab4a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0655a86640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9ab4a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41238/0 (socket says 172.21.15.89:41238) 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0655a86640 1 -- 172.21.15.89:0/1709588223 learned_addr learned my addr 172.21.15.89:0/1709588223 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0656287640 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aad7d40 0x560f9ab55730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0655285640 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aacd480 0x560f9ab4abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0655a86640 1 -- 172.21.15.89:0/1709588223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aacd480 msgr2=0x560f9ab4abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0655a86640 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aacd480 0x560f9ab4abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0655a86640 1 -- 172.21.15.89:0/1709588223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aad7d40 msgr2=0x560f9ab55730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0655a86640 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aad7d40 0x560f9ab55730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0655a86640 1 -- 172.21.15.89:0/1709588223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f9ab55c70 con 0x560f9aaccb60 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f0655a86640 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9ab4a6b0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f0644003610 tx=0x7f0644005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744e23cc1ae97c83 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f06397fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.802+0000 7f06397fa640 1 -- 172.21.15.89:0/1709588223 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f064405f060 con 0x560f9aaccb60 2024-09-15T23:39:08.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f06397fa640 1 -- 172.21.15.89:0/1709588223 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0644046a20 con 0x560f9aaccb60 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f065a0c3740 1 -- 172.21.15.89:0/1709588223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0644031990 con 0x560f9aaccb60 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f06397fa640 1 -- 172.21.15.89:0/1709588223 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0644066050 con 0x560f9aaccb60 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f065a0c3740 10 client.20688 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f065a0c3740 1 -- 172.21.15.89:0/1709588223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560f9ab56e60 con 0x560f9aaccb60 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f06397fa640 1 -- 172.21.15.89:0/1709588223 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0644031990 con 0x560f9aaccb60 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f06397fa640 1 -- 172.21.15.89:0/1709588223 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f064403c440 con 0x560f9aaccb60 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f065a0c3740 10 client.20688 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f065a0c3740 10 client.20688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f065a0c3740 20 client.20688 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f065a0c3740 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560f9ab57470 0x560f9ab59860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f065a0c3740 4 client.20688 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:39:08.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f065a0c3740 1 -- 172.21.15.89:0/1709588223 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560f9aa39d00 con 0x560f9ab57470 2024-09-15T23:39:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.803+0000 7f0655285640 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560f9ab57470 0x560f9ab59860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.804+0000 7f0655285640 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560f9ab57470 0x560f9ab59860 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c89d51d280d7f7 server_cookie=a4fabb4afe543b19 in_seq=0 out_seq=0 2024-09-15T23:39:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.804+0000 7f06397fa640 10 client.20688 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.804+0000 7f06397fa640 1 -- 172.21.15.89:0/1709588223 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560f9aa39d00 con 0x560f9ab57470 2024-09-15T23:39:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.804+0000 7f06397fa640 10 client.20688 handle_command_reply: tid=0 2024-09-15T23:39:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.804+0000 7f065a0c3740 1 client.20688 shutdown 2024-09-15T23:39:08.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 1 -- 172.21.15.89:0/1709588223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 msgr2=0x560f9ab4a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9ab4a6b0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f0644003610 tx=0x7f0644005e80 comp rx=0 tx=0).stop 2024-09-15T23:39:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 1 -- 172.21.15.89:0/1709588223 shutdown_connections 2024-09-15T23:39:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560f9ab57470 0x560f9ab59860 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560f9aad7d40 0x560f9ab55730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560f9aacd480 0x560f9ab4abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 1 --2- 172.21.15.89:0/1709588223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560f9aaccb60 0x560f9ab4a6b0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.815 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f06397fa640 0 client.20688 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:08.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 1 -- 172.21.15.89:0/1709588223 >> 172.21.15.89:0/1709588223 conn(0x560f9aac90f0 msgr2=0x560f9aac8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 1 -- 172.21.15.89:0/1709588223 shutdown_connections 2024-09-15T23:39:08.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 1 -- 172.21.15.89:0/1709588223 wait complete. 2024-09-15T23:39:08.816 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.805+0000 7f065a0c3740 20 client.20688 trim_cache size 0 max 16384 2024-09-15T23:39:08.816 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:39:08.816 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:39:08.817 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "941abd37-a266-475b-b5ef-93eadebd6725", 2024-09-15T23:39:08.817 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:39:08.817 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:39:08.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 -- 172.21.15.89:0/4181573010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f063c0a3240 msgr2=0x7f063c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f063c0a3240 0x7f063c0a5700 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f064c0058c0 tx=0x7f064c0138f0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.817 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 -- 172.21.15.89:0/4181573010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0650106f10 msgr2=0x7f065019cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:08.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0650106f10 0x7f065019cbd0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f064800cce0 tx=0x7f064800a620 comp rx=0 tx=0).stop 2024-09-15T23:39:08.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 -- 172.21.15.89:0/4181573010 shutdown_connections 2024-09-15T23:39:08.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f063c0a3240 0x7f063c0a5700 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0650106f10 0x7f065019cbd0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0650102790 0x7f0650198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 --2- 172.21.15.89:0/4181573010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0650101de0 0x7f0650198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:08.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 -- 172.21.15.89:0/4181573010 >> 172.21.15.89:0/4181573010 conn(0x7f06500fd8c0 msgr2=0x7f06500fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:08.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 -- 172.21.15.89:0/4181573010 shutdown_connections 2024-09-15T23:39:08.818 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:08.806+0000 7f0657cea640 1 -- 172.21.15.89:0/4181573010 wait complete. 2024-09-15T23:39:09.447 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:09 smithi089 ceph-mon[30933]: pgmap v803: 129 pgs: 129 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:39:09.447 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:09 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:39:09.447 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:09 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:09.447 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:09 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:39:09.448 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:09 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:09 smithi120 ceph-mon[35250]: pgmap v803: 129 pgs: 129 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:39:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:09 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:39:09.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:09 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:09.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:09 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:39:09.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:09 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:09.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:09 smithi174 ceph-mon[35586]: pgmap v803: 129 pgs: 129 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:39:09.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:09 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:39:09.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:09 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:09.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:09 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:39:09.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:09 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:09.548 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:39:09.797 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:10.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.263+0000 7fc090811640 1 -- 172.21.15.89:0/3461549497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc08c100a80 msgr2=0x7fc08c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.263+0000 7fc090811640 1 --2- 172.21.15.89:0/3461549497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc08c100a80 0x7fc08c100ee0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fc074009930 tx=0x7fc07402a420 comp rx=0 tx=0).stop 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.263+0000 7fc090811640 1 -- 172.21.15.89:0/3461549497 shutdown_connections 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.263+0000 7fc090811640 1 --2- 172.21.15.89:0/3461549497 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc08c101420 0x7fc08c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.263+0000 7fc090811640 1 --2- 172.21.15.89:0/3461549497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc08c100a80 0x7fc08c100ee0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.263+0000 7fc090811640 1 --2- 172.21.15.89:0/3461549497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc08c106a80 0x7fc08c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.263+0000 7fc090811640 1 -- 172.21.15.89:0/3461549497 >> 172.21.15.89:0/3461549497 conn(0x7fc08c0fc7a0 msgr2=0x7fc08c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.263+0000 7fc090811640 1 -- 172.21.15.89:0/3461549497 shutdown_connections 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.263+0000 7fc090811640 1 -- 172.21.15.89:0/3461549497 wait complete. 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.264+0000 7fc090811640 1 Processor -- start 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.264+0000 7fc090811640 1 -- start start 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.264+0000 7fc090811640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc08c100a80 0x7fc08c0fcd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.264+0000 7fc090811640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc08c101420 0x7fc08c0fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.264+0000 7fc090811640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc08c106a80 0x7fc08c0fd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.264+0000 7fc090811640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c110ec0 con 0x7fc08c106a80 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.264+0000 7fc090811640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c110d40 con 0x7fc08c101420 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.264+0000 7fc090811640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c111040 con 0x7fc08c100a80 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.264+0000 7fc08ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc08c106a80 0x7fc08c0fd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc08ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc08c106a80 0x7fc08c0fd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41262/0 (socket says 172.21.15.89:41262) 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc08ad9d640 1 -- 172.21.15.89:0/3523291327 learned_addr learned my addr 172.21.15.89:0/3523291327 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc089d9b640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc08c101420 0x7fc08c0fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc08a59c640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc08c100a80 0x7fc08c0fcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc08ad9d640 1 -- 172.21.15.89:0/3523291327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc08c100a80 msgr2=0x7fc08c0fcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc08ad9d640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc08c100a80 0x7fc08c0fcd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc08ad9d640 1 -- 172.21.15.89:0/3523291327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc08c101420 msgr2=0x7fc08c0fd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:10.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc08ad9d640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc08c101420 0x7fc08c0fd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc08ad9d640 1 -- 172.21.15.89:0/3523291327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08c1a69b0 con 0x7fc08c106a80 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc08ad9d640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc08c106a80 0x7fc08c0fd7b0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fc08400b970 tx=0x7fc08400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc0837fe640 1 -- 172.21.15.89:0/3523291327 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc084019070 con 0x7fc08c106a80 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc090811640 1 -- 172.21.15.89:0/3523291327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc08c1a6ca0 con 0x7fc08c106a80 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc0837fe640 1 -- 172.21.15.89:0/3523291327 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc08400cb40 con 0x7fc08c106a80 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc0837fe640 1 -- 172.21.15.89:0/3523291327 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc084007650 con 0x7fc08c106a80 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.265+0000 7fc090811640 1 -- 172.21.15.89:0/3523291327 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc08c1a71b0 con 0x7fc08c106a80 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.267+0000 7fc0837fe640 1 -- 172.21.15.89:0/3523291327 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc08401c070 con 0x7fc08c106a80 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc092bea740 1 Processor -- start 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc092bea740 1 -- start start 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc092bea740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5ae04a0 0x555ed5ae0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc092bea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5aeb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc092bea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5aeb680 0x555ed5aeda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc092bea740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555ed5a462e0 con 0x555ed5ae04a0 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc0837fe640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0780a3240 0x7fc0780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc092bea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555ed59edce0 con 0x555ed5ae0dc0 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc092bea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555ed5a422a0 con 0x555ed5aeb680 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc089d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5ae04a0 0x555ed5ae0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc089d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5ae04a0 0x555ed5ae0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41268/0 (socket says 172.21.15.89:41268) 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.268+0000 7fc089d9b640 1 -- 172.21.15.89:0/3650036665 learned_addr learned my addr 172.21.15.89:0/3650036665 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc08a59c640 1 --2- 172.21.15.89:0/3650036665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5aeb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc08ad9d640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0780a3240 0x7fc0780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc0837fe640 1 -- 172.21.15.89:0/3523291327 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc0840cf370 con 0x7fc08c106a80 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc089d9b640 1 -- 172.21.15.89:0/3650036665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5aeb680 msgr2=0x555ed5aeda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc089d9b640 1 --2- 172.21.15.89:0/3650036665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5aeb680 0x555ed5aeda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc089d9b640 1 -- 172.21.15.89:0/3650036665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 msgr2=0x555ed5aeb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:10.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc089d9b640 1 --2- 172.21.15.89:0/3650036665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5aeb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc089d9b640 1 -- 172.21.15.89:0/3650036665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ed59d2440 con 0x555ed5ae04a0 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc089d9b640 1 --2- 172.21.15.89:0/3650036665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5ae04a0 0x555ed5ae0880 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fc07402a940 tx=0x7fc074032f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2216417f41a56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc08ad9d640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0780a3240 0x7fc0780a5700 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc08409b530 tx=0x7fc084065fe0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc080ff9640 1 -- 172.21.15.89:0/3650036665 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074035050 con 0x555ed5ae04a0 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc080ff9640 1 -- 172.21.15.89:0/3650036665 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0740317b0 con 0x555ed5ae04a0 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.269+0000 7fc080ff9640 1 -- 172.21.15.89:0/3650036665 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0740046c0 con 0x555ed5ae04a0 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 -- 172.21.15.89:0/3650036665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5ae04a0 msgr2=0x555ed5ae0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 --2- 172.21.15.89:0/3650036665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5ae04a0 0x555ed5ae0880 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fc07402a940 tx=0x7fc074032f70 comp rx=0 tx=0).stop 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 -- 172.21.15.89:0/3650036665 shutdown_connections 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 --2- 172.21.15.89:0/3650036665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5aeb680 0x555ed5aeda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 --2- 172.21.15.89:0/3650036665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5aeb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 --2- 172.21.15.89:0/3650036665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5ae04a0 0x555ed5ae0880 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 -- 172.21.15.89:0/3650036665 >> 172.21.15.89:0/3650036665 conn(0x555ed5adca30 msgr2=0x555ed5adce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:10.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 -- 172.21.15.89:0/3650036665 shutdown_connections 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 -- 172.21.15.89:0/3650036665 wait complete. 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 -- start start 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.270+0000 7fc092bea740 1 Processor -- start 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc092bea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae04a0 0x555ed5aeaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc092bea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5ae8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc092bea740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5aeb680 0x555ed5ae8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc092bea740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555ed5aee250 con 0x555ed5aeb680 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc092bea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555ed5a01050 con 0x555ed5ae0dc0 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc092bea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555ed5aee3d0 con 0x555ed5ae04a0 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc092bea740 10 client.0 _ll_register_callbacks cb 0x555ed5a41160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc089d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5aeb680 0x555ed5ae8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc089d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5aeb680 0x555ed5ae8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41284/0 (socket says 172.21.15.89:41284) 2024-09-15T23:39:10.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc089d9b640 1 -- 172.21.15.89:0/1604981235 learned_addr learned my addr 172.21.15.89:0/1604981235 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae04a0 0x555ed5aeaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc08a59c640 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5ae8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc089d9b640 1 -- 172.21.15.89:0/1604981235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae04a0 msgr2=0x555ed5aeaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae04a0 0x555ed5aeaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc089d9b640 1 -- 172.21.15.89:0/1604981235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 msgr2=0x555ed5ae8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5ae8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc089d9b640 1 -- 172.21.15.89:0/1604981235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ed57b73c0 con 0x555ed5aeb680 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5aeb680 0x555ed5ae8740 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fc074005160 tx=0x7fc074005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9638f0408f92e82 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc071ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.271+0000 7fc071ffb640 1 -- 172.21.15.89:0/1604981235 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074035040 con 0x555ed5aeb680 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc071ffb640 1 -- 172.21.15.89:0/1604981235 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc0740315b0 con 0x555ed5aeb680 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc092bea740 1 -- 172.21.15.89:0/1604981235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0740317b0 con 0x555ed5aeb680 2024-09-15T23:39:10.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc071ffb640 1 -- 172.21.15.89:0/1604981235 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc074059050 con 0x555ed5aeb680 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc092bea740 10 client.20700 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc092bea740 1 -- 172.21.15.89:0/1604981235 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555ed5af9c00 con 0x555ed5aeb680 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc071ffb640 1 -- 172.21.15.89:0/1604981235 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0740317b0 con 0x555ed5aeb680 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc071ffb640 1 -- 172.21.15.89:0/1604981235 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc0740038f0 con 0x555ed5aeb680 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc092bea740 10 client.20700 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc092bea740 10 client.20700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc092bea740 20 client.20700 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc092bea740 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555ed5afaf50 0x555ed5afd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc092bea740 4 client.20700 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc092bea740 1 -- 172.21.15.89:0/1604981235 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ed5a53850 con 0x555ed5afaf50 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.272+0000 7fc08a59c640 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555ed5afaf50 0x555ed5afd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.273+0000 7fc071ffb640 10 client.20700 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.273+0000 7fc08a59c640 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555ed5afaf50 0x555ed5afd340 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef8e7823a6acac9f server_cookie=f13cdf1ae1c71864 in_seq=0 out_seq=0 2024-09-15T23:39:10.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc071ffb640 1 -- 172.21.15.89:0/1604981235 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555ed5a53850 con 0x555ed5afaf50 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc071ffb640 10 client.20700 handle_command_reply: tid=0 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 client.20700 shutdown 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 -- 172.21.15.89:0/1604981235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5aeb680 msgr2=0x555ed5ae8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5aeb680 0x555ed5ae8740 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fc074005160 tx=0x7fc074005340 comp rx=0 tx=0).stop 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 -- 172.21.15.89:0/1604981235 shutdown_connections 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555ed5afaf50 0x555ed5afd340 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5aeb680 0x555ed5ae8740 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5ae8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 --2- 172.21.15.89:0/1604981235 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae04a0 0x555ed5aeaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 -- 172.21.15.89:0/1604981235 >> 172.21.15.89:0/1604981235 conn(0x555ed5adca30 msgr2=0x555ed5ae2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc071ffb640 0 client.20700 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 -- 172.21.15.89:0/1604981235 shutdown_connections 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 1 -- 172.21.15.89:0/1604981235 wait complete. 2024-09-15T23:39:10.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.274+0000 7fc092bea740 20 client.20700 trim_cache size 0 max 16384 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc092bea740 1 Processor -- start 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc092bea740 1 -- start start 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc092bea740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5aea170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc092bea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae04a0 0x555ed5afff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc092bea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae0dc0 0x555ed5b004d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc092bea740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1b0f90 con 0x555ed5adca30 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc092bea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1b2ad0 con 0x555ed5ae04a0 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc092bea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1b1660 con 0x555ed5ae0dc0 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc089d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5aea170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc089d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5aea170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41300/0 (socket says 172.21.15.89:41300) 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc089d9b640 1 -- 172.21.15.89:0/1980713023 learned_addr learned my addr 172.21.15.89:0/1980713023 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc089d9b640 1 -- 172.21.15.89:0/1980713023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae0dc0 msgr2=0x555ed5b004d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1980713023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae0dc0 0x555ed5b004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc089d9b640 1 -- 172.21.15.89:0/1980713023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae04a0 msgr2=0x555ed5afff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.291+0000 7fc08a59c640 1 --2- 172.21.15.89:0/1980713023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae04a0 0x555ed5afff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.292+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1980713023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae04a0 0x555ed5afff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.292+0000 7fc089d9b640 1 -- 172.21.15.89:0/1980713023 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ed5adc860 con 0x555ed5adca30 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.292+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1980713023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5aea170 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fc074032c00 tx=0x7fc074005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d1d09aba68c2d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.292+0000 7fc073fff640 1 -- 172.21.15.89:0/1980713023 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074035040 con 0x555ed5adca30 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.292+0000 7fc073fff640 1 -- 172.21.15.89:0/1980713023 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc07404a070 con 0x555ed5adca30 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.292+0000 7fc073fff640 1 -- 172.21.15.89:0/1980713023 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0740315b0 con 0x555ed5adca30 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.292+0000 7fc092bea740 1 -- 172.21.15.89:0/1980713023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 msgr2=0x555ed5aea170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.292+0000 7fc092bea740 1 --2- 172.21.15.89:0/1980713023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5aea170 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fc074032c00 tx=0x7fc074005a20 comp rx=0 tx=0).stop 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.292+0000 7fc092bea740 1 -- 172.21.15.89:0/1980713023 shutdown_connections 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.292+0000 7fc092bea740 1 --2- 172.21.15.89:0/1980713023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae0dc0 0x555ed5b004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 --2- 172.21.15.89:0/1980713023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae04a0 0x555ed5afff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 --2- 172.21.15.89:0/1980713023 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5aea170 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 -- 172.21.15.89:0/1980713023 >> 172.21.15.89:0/1980713023 conn(0x555ed5ae88a0 msgr2=0x555ed5ae9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 -- 172.21.15.89:0/1980713023 shutdown_connections 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 -- 172.21.15.89:0/1980713023 wait complete. 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 -- start start 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 Processor -- start 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5affc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae04a0 0x555ed5b00170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5b5cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1b2400 con 0x555ed5adca30 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1b1d30 con 0x555ed5ae0dc0 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc08c1aef70 con 0x555ed5ae04a0 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc092bea740 10 client.0 _ll_register_callbacks cb 0x7fc07c0092a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc089d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5affc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc089d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5affc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:41304/0 (socket says 172.21.15.89:41304) 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc089d9b640 1 -- 172.21.15.89:0/1354151667 learned_addr learned my addr 172.21.15.89:0/1354151667 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.293+0000 7fc08a59c640 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae04a0 0x555ed5b00170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5b5cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc089d9b640 1 -- 172.21.15.89:0/1354151667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae04a0 msgr2=0x555ed5b00170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae04a0 0x555ed5b00170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc089d9b640 1 -- 172.21.15.89:0/1354151667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 msgr2=0x555ed5b5cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5b5cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc089d9b640 1 -- 172.21.15.89:0/1354151667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ed5b5d2b0 con 0x555ed5adca30 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc089d9b640 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5affc30 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fc0740095e0 tx=0x7fc07403d600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1308e4000d3e3d2d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc071ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc071ffb640 1 -- 172.21.15.89:0/1354151667 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074035030 con 0x555ed5adca30 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc071ffb640 1 -- 172.21.15.89:0/1354151667 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc074005510 con 0x555ed5adca30 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc092bea740 1 -- 172.21.15.89:0/1354151667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc07404a070 con 0x555ed5adca30 2024-09-15T23:39:10.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc071ffb640 1 -- 172.21.15.89:0/1354151667 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc0740317c0 con 0x555ed5adca30 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc092bea740 10 client.20706 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc092bea740 1 -- 172.21.15.89:0/1354151667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555ed5b65850 con 0x555ed5adca30 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.294+0000 7fc071ffb640 1 -- 172.21.15.89:0/1354151667 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc07404a070 con 0x555ed5adca30 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.295+0000 7fc071ffb640 1 -- 172.21.15.89:0/1354151667 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc074003fe0 con 0x555ed5adca30 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.295+0000 7fc092bea740 10 client.20706 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.295+0000 7fc092bea740 10 client.20706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.295+0000 7fc092bea740 20 client.20706 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.295+0000 7fc092bea740 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555ed5aeb680 0x555ed5b67e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.295+0000 7fc092bea740 4 client.20706 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.295+0000 7fc092bea740 1 -- 172.21.15.89:0/1354151667 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555ed5a4dd60 con 0x555ed5aeb680 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.295+0000 7fc08a59c640 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555ed5aeb680 0x555ed5b67e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.295+0000 7fc08a59c640 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555ed5aeb680 0x555ed5b67e60 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb0ac86e15b1487e server_cookie=49064aadd409b027 in_seq=0 out_seq=0 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.295+0000 7fc071ffb640 10 client.20706 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:10.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc071ffb640 1 -- 172.21.15.89:0/1354151667 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555ed5a4dd60 con 0x555ed5aeb680 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc071ffb640 10 client.20706 handle_command_reply: tid=0 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 client.20706 shutdown 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 -- 172.21.15.89:0/1354151667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 msgr2=0x555ed5affc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5affc30 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fc0740095e0 tx=0x7fc07403d600 comp rx=0 tx=0).stop 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 -- 172.21.15.89:0/1354151667 shutdown_connections 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555ed5aeb680 0x555ed5b67e60 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ed5ae0dc0 0x555ed5b5cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc071ffb640 0 client.20706 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555ed5ae04a0 0x555ed5b00170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 --2- 172.21.15.89:0/1354151667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ed5adca30 0x555ed5affc30 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 -- 172.21.15.89:0/1354151667 >> 172.21.15.89:0/1354151667 conn(0x555ed5ae88a0 msgr2=0x555ed5ae66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 -- 172.21.15.89:0/1354151667 shutdown_connections 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 1 -- 172.21.15.89:0/1354151667 wait complete. 2024-09-15T23:39:10.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.296+0000 7fc092bea740 20 client.20706 trim_cache size 0 max 16384 2024-09-15T23:39:10.313 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:39:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:39:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:39:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: "941abd37-a266-475b-b5ef-93eadebd6725": { 2024-09-15T23:39:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:39:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "941abd37-a266-475b-b5ef-93eadebd6725", 2024-09-15T23:39:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:39:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:10.313 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:10.314 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:39:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 -- 172.21.15.89:0/3523291327 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0780a3240 msgr2=0x7fc0780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0780a3240 0x7fc0780a5700 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc08409b530 tx=0x7fc084065fe0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 -- 172.21.15.89:0/3523291327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc08c106a80 msgr2=0x7fc08c0fd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc08c106a80 0x7fc08c0fd7b0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fc08400b970 tx=0x7fc08400be30 comp rx=0 tx=0).stop 2024-09-15T23:39:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 -- 172.21.15.89:0/3523291327 shutdown_connections 2024-09-15T23:39:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc0780a3240 0x7fc0780a5700 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc08c106a80 0x7fc08c0fd7b0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc08c101420 0x7fc08c0fd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 --2- 172.21.15.89:0/3523291327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc08c100a80 0x7fc08c0fcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 -- 172.21.15.89:0/3523291327 >> 172.21.15.89:0/3523291327 conn(0x7fc08c0fc7a0 msgr2=0x7fc08c06aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 -- 172.21.15.89:0/3523291327 shutdown_connections 2024-09-15T23:39:10.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:10.297+0000 7fc090811640 1 -- 172.21.15.89:0/3523291327 wait complete. 2024-09-15T23:39:11.019 INFO:tasks.cephfs.filesystem:scrub status for tag:941abd37-a266-475b-b5ef-93eadebd6725 - {'path': '/', 'tag': '941abd37-a266-475b-b5ef-93eadebd6725', 'options': 'recursive,force'} 2024-09-15T23:39:11.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:11 smithi089 ceph-mon[30933]: pgmap v804: 129 pgs: 129 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:39:11.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:11 smithi120 ceph-mon[35250]: pgmap v804: 129 pgs: 129 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:39:11.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:11 smithi174 ceph-mon[35586]: pgmap v804: 129 pgs: 129 active+clean; 3.6 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:499/600: open fd = 3 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:500/600: open fd = 3 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:501/600: open fd = 3 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.670 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:502/600: open fd = 3 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:503/600: open fd = 3 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:504/600: open fd = 3 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.671 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:505/600: open fd = 3 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:506/600: open fd = 3 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:507/600: open fd = 3 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.672 INFO:tasks.workunit.client.0.smithi089.stdout:508/600: open fd = 3 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:509/600: open fd = 3 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:510/600: open fd = 3 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:511/600: open fd = 3 2024-09-15T23:39:12.673 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:512/600: open fd = 3 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:513/600: open fd = 3 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:514/600: open fd = 3 2024-09-15T23:39:12.674 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:515/600: open fd = 3 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:516/600: open fd = 3 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:517/600: open fd = 3 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.675 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:518/600: open fd = 3 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:519/600: open fd = 3 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:520/600: open fd = 3 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.676 INFO:tasks.workunit.client.0.smithi089.stdout:521/600: open fd = 3 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:522/600: open fd = 3 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:523/600: open fd = 3 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:524/600: open fd = 3 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.677 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:525/600: open fd = 3 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:526/600: open fd = 3 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:527/600: open fd = 3 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.678 INFO:tasks.workunit.client.0.smithi089.stdout:528/600: open fd = 3 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:529/600: open fd = 3 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:530/600: open fd = 3 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:531/600: open fd = 3 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.679 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:532/600: open fd = 3 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:533/600: open fd = 3 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:534/600: open fd = 3 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.680 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:535/600: open fd = 3 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:536/600: open fd = 3 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:537/600: open fd = 3 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:538/600: open fd = 3 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.681 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:539/600: open fd = 3 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:540/600: open fd = 3 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:541/600: open fd = 3 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.682 INFO:tasks.workunit.client.0.smithi089.stdout:542/600: open fd = 3 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:543/600: open fd = 3 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:544/600: open fd = 3 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:545/600: open fd = 3 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.683 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:546/600: open fd = 3 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:547/600: open fd = 3 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:548/600: open fd = 3 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.684 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:549/600: open fd = 3 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:550/600: open fd = 3 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:551/600: open fd = 3 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:552/600: open fd = 3 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.685 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:553/600: open fd = 3 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:554/600: open fd = 3 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:555/600: open fd = 3 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.686 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:556/600: open fd = 3 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:557/600: open fd = 3 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:558/600: open fd = 3 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:559/600: open fd = 3 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.687 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.688 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:12.688 INFO:tasks.workunit.client.0.smithi089.stdout:560/600: open fd = 3 2024-09-15T23:39:12.688 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.688 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:39:12.688 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:39:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:13 smithi120 ceph-mon[35250]: pgmap v805: 129 pgs: 129 active+clean; 3.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:39:13.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:13 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:39:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:13 smithi174 ceph-mon[35586]: pgmap v805: 129 pgs: 129 active+clean; 3.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:39:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:13 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:39:13.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:13 smithi089 ceph-mon[30933]: pgmap v805: 129 pgs: 129 active+clean; 3.6 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:39:13.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:13 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:39:15.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:15 smithi120 ceph-mon[35250]: pgmap v806: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:39:15.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:15 smithi174 ceph-mon[35586]: pgmap v806: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:39:15.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:15 smithi089 ceph-mon[30933]: pgmap v806: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:39:17.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:17 smithi120 ceph-mon[35250]: pgmap v807: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:39:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:17 smithi174 ceph-mon[35586]: pgmap v807: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:39:17.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:17 smithi089 ceph-mon[30933]: pgmap v807: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 17 op/s 2024-09-15T23:39:19.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:19 smithi120 ceph-mon[35250]: pgmap v808: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:39:19.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:19 smithi174 ceph-mon[35586]: pgmap v808: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:39:19.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:19 smithi089 ceph-mon[30933]: pgmap v808: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.422+0000 7f2aa194b640 1 -- 172.21.15.89:0/4245469684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a9c100a40 msgr2=0x7f2a9c100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.422+0000 7f2aa194b640 1 --2- 172.21.15.89:0/4245469684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a9c100a40 0x7f2a9c100e40 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009ca0 tx=0x7f2a8c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.422+0000 7f2aa194b640 1 -- 172.21.15.89:0/4245469684 shutdown_connections 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.422+0000 7f2aa194b640 1 --2- 172.21.15.89:0/4245469684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a9c1025f0 0x7f2a9c108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.422+0000 7f2aa194b640 1 --2- 172.21.15.89:0/4245469684 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a9c101c30 0x7f2a9c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.422+0000 7f2aa194b640 1 --2- 172.21.15.89:0/4245469684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a9c100a40 0x7f2a9c100e40 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.422+0000 7f2aa194b640 1 -- 172.21.15.89:0/4245469684 >> 172.21.15.89:0/4245469684 conn(0x7f2a9c0fc1e0 msgr2=0x7f2a9c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.422+0000 7f2aa194b640 1 -- 172.21.15.89:0/4245469684 shutdown_connections 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.422+0000 7f2aa194b640 1 -- 172.21.15.89:0/4245469684 wait complete. 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2aa194b640 1 Processor -- start 2024-09-15T23:39:21.426 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2aa194b640 1 -- start start 2024-09-15T23:39:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2aa194b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a9c100a40 0x7f2a9c06a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2aa194b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a9c101c30 0x7f2a9c19cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2aa194b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a9c1025f0 0x7f2a9c19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2aa194b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c10c6a0 con 0x7f2a9c1025f0 2024-09-15T23:39:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2aa194b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c10c520 con 0x7f2a9c101c30 2024-09-15T23:39:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2aa194b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c10c820 con 0x7f2a9c100a40 2024-09-15T23:39:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2a9b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a9c1025f0 0x7f2a9c19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2a9b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a9c1025f0 0x7f2a9c19f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45070/0 (socket says 172.21.15.89:45070) 2024-09-15T23:39:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2a9b7fe640 1 -- 172.21.15.89:0/995344353 learned_addr learned my addr 172.21.15.89:0/995344353 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:21.427 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2a9affd640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a9c100a40 0x7f2a9c06a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.423+0000 7f2a927fc640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a9c101c30 0x7f2a9c19cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2a9b7fe640 1 -- 172.21.15.89:0/995344353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a9c100a40 msgr2=0x7f2a9c06a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2a9b7fe640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a9c100a40 0x7f2a9c06a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2a9b7fe640 1 -- 172.21.15.89:0/995344353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a9c101c30 msgr2=0x7f2a9c19cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:21.431 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2a9b7fe640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a9c101c30 0x7f2a9c19cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2a9b7fe640 1 -- 172.21.15.89:0/995344353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9c19fa40 con 0x7f2a9c1025f0 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2a9b7fe640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a9c1025f0 0x7f2a9c19f130 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400d750 tx=0x7f2a9400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2aa0949640 1 -- 172.21.15.89:0/995344353 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a94017070 con 0x7f2a9c1025f0 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2aa0949640 1 -- 172.21.15.89:0/995344353 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a94005500 con 0x7f2a9c1025f0 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2aa194b640 1 -- 172.21.15.89:0/995344353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9c19fcd0 con 0x7f2a9c1025f0 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2aa0949640 1 -- 172.21.15.89:0/995344353 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a94004d90 con 0x7f2a9c1025f0 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.424+0000 7f2aa194b640 1 -- 172.21.15.89:0/995344353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a9c1a01d0 con 0x7f2a9c1025f0 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.426+0000 7f2aa0949640 1 -- 172.21.15.89:0/995344353 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2a94004f30 con 0x7f2a9c1025f0 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2aa47c2740 1 Processor -- start 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2aa47c2740 1 -- start start 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2aa0949640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a880a32f0 0x7f2a880a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.432 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cd31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd36f0 0x55c1e6cd81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd8780 0x55c1e6cdac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c1e6c322f0 con 0x55c1e6cd2db0 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c1e6bda270 con 0x55c1e6cd36f0 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c1e6c2e2b0 con 0x55c1e6cd8780 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2a9affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cd31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2a9affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cd31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45072/0 (socket says 172.21.15.89:45072) 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2a927fc640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a880a32f0 0x7f2a880a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2a9affd640 1 -- 172.21.15.89:0/2825751451 learned_addr learned my addr 172.21.15.89:0/2825751451 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2aa0949640 1 -- 172.21.15.89:0/995344353 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2a940cf6d0 con 0x7f2a9c1025f0 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2a927fc640 1 --2- 172.21.15.89:0/2825751451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd36f0 0x55c1e6cd81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2a9b7fe640 1 --2- 172.21.15.89:0/2825751451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd8780 0x55c1e6cdac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.433 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2a9affd640 1 -- 172.21.15.89:0/2825751451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd8780 msgr2=0x55c1e6cdac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2a9affd640 1 --2- 172.21.15.89:0/2825751451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd8780 0x55c1e6cdac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2a9affd640 1 -- 172.21.15.89:0/2825751451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd36f0 msgr2=0x55c1e6cd81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.427+0000 7f2a9affd640 1 --2- 172.21.15.89:0/2825751451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd36f0 0x55c1e6cd81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.428+0000 7f2a9affd640 1 -- 172.21.15.89:0/2825751451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1e6bd9340 con 0x55c1e6cd2db0 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.428+0000 7f2a927fc640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a880a32f0 0x7f2a880a57b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c06a180 tx=0x7f2a840138e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.428+0000 7f2a9affd640 1 --2- 172.21.15.89:0/2825751451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cd31b0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009ca0 tx=0x7f2a8c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f138a395b61b23d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.428+0000 7f2a91ffb640 1 -- 172.21.15.89:0/2825751451 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c031450 con 0x55c1e6cd2db0 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.428+0000 7f2a91ffb640 1 -- 172.21.15.89:0/2825751451 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a8c036040 con 0x55c1e6cd2db0 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.428+0000 7f2a91ffb640 1 -- 172.21.15.89:0/2825751451 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c004410 con 0x55c1e6cd2db0 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2825751451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 msgr2=0x55c1e6cd31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/2825751451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cd31b0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009ca0 tx=0x7f2a8c005ef0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2825751451 shutdown_connections 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/2825751451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd8780 0x55c1e6cdac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.434 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/2825751451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd36f0 0x55c1e6cd81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/2825751451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cd31b0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2825751451 >> 172.21.15.89:0/2825751451 conn(0x55c1e6cca010 msgr2=0x55c1e6cca410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2825751451 shutdown_connections 2024-09-15T23:39:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2825751451 wait complete. 2024-09-15T23:39:21.435 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 -- start start 2024-09-15T23:39:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 Processor -- start 2024-09-15T23:39:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cdcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd36f0 0x55c1e6cdd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.429+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ce4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.438 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c1e6cdb570 con 0x55c1e6cd2db0 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c1e6bbb290 con 0x55c1e6cd8780 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c1e6cdb6f0 con 0x55c1e6cd36f0 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2aa47c2740 10 client.0 _ll_register_callbacks cb 0x55c1e6bec140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a9affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cdcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a9affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cdcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45076/0 (socket says 172.21.15.89:45076) 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a9affd640 1 -- 172.21.15.89:0/2074673730 learned_addr learned my addr 172.21.15.89:0/2074673730 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a927fc640 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd36f0 0x55c1e6cdd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a9affd640 1 -- 172.21.15.89:0/2074673730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd36f0 msgr2=0x55c1e6cdd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a9affd640 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd36f0 0x55c1e6cdd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a9affd640 1 -- 172.21.15.89:0/2074673730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 msgr2=0x55c1e6ce4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a9affd640 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ce4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a9affd640 1 -- 172.21.15.89:0/2074673730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1e6945210 con 0x55c1e6cd2db0 2024-09-15T23:39:21.439 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a9affd640 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cdcd90 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c031ae0 tx=0x7f2a8c005c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2763baf76e514 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a7affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a7affd640 1 -- 172.21.15.89:0/2074673730 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c003b80 con 0x55c1e6cd2db0 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a7affd640 1 -- 172.21.15.89:0/2074673730 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2a8c003d20 con 0x55c1e6cd2db0 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2074673730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a8c036040 con 0x55c1e6cd2db0 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2a7affd640 1 -- 172.21.15.89:0/2074673730 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a8c05c050 con 0x55c1e6cd2db0 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2aa47c2740 10 client.20718 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.430+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2074673730 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c1e6a11620 con 0x55c1e6cd2db0 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.431+0000 7f2a7affd640 1 -- 172.21.15.89:0/2074673730 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a8c036040 con 0x55c1e6cd2db0 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.431+0000 7f2a7affd640 1 -- 172.21.15.89:0/2074673730 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2a8c057030 con 0x55c1e6cd2db0 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.431+0000 7f2aa47c2740 10 client.20718 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.431+0000 7f2aa47c2740 10 client.20718 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.431+0000 7f2aa47c2740 20 client.20718 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.431+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c1e6ce87b0 0x55c1e6ceaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.440 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.431+0000 7f2aa47c2740 4 client.20718 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.431+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2074673730 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1e6c3fa20 con 0x55c1e6ce87b0 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.431+0000 7f2a927fc640 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c1e6ce87b0 0x55c1e6ceaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.432+0000 7f2a7affd640 10 client.20718 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.432+0000 7f2a927fc640 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c1e6ce87b0 0x55c1e6ceaba0 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d59ce10d06999db server_cookie=8b253a5e45a35371 in_seq=0 out_seq=0 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.432+0000 7f2a7affd640 1 -- 172.21.15.89:0/2074673730 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c1e6c3fa20 con 0x55c1e6ce87b0 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.432+0000 7f2a7affd640 10 client.20718 handle_command_reply: tid=0 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.432+0000 7f2aa47c2740 1 client.20718 shutdown 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2074673730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 msgr2=0x55c1e6cdcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cdcd90 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c031ae0 tx=0x7f2a8c005c40 comp rx=0 tx=0).stop 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2074673730 shutdown_connections 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c1e6ce87b0 0x55c1e6ceaba0 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ce4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd36f0 0x55c1e6cdd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.441 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/2074673730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6cdcd90 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2a7affd640 0 client.20718 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2074673730 >> 172.21.15.89:0/2074673730 conn(0x55c1e6cca010 msgr2=0x55c1e6ccb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2074673730 shutdown_connections 2024-09-15T23:39:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 1 -- 172.21.15.89:0/2074673730 wait complete. 2024-09-15T23:39:21.442 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.433+0000 7f2aa47c2740 20 client.20718 trim_cache size 0 max 16384 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2aa47c2740 1 Processor -- start 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2aa47c2740 1 -- start start 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6ccae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd36f0 0x55c1e6ce42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ccb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c1a8520 con 0x55c1e6cd2db0 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c1aa060 con 0x55c1e6cd8780 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c1a8bf0 con 0x55c1e6cd36f0 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6ccae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6ccae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45086/0 (socket says 172.21.15.89:45086) 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9affd640 1 -- 172.21.15.89:0/721615272 learned_addr learned my addr 172.21.15.89:0/721615272 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a927fc640 1 --2- 172.21.15.89:0/721615272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd36f0 0x55c1e6ce42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9affd640 1 -- 172.21.15.89:0/721615272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd36f0 msgr2=0x55c1e6ce42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9affd640 1 --2- 172.21.15.89:0/721615272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd36f0 0x55c1e6ce42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9affd640 1 -- 172.21.15.89:0/721615272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 msgr2=0x55c1e6ccb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9b7fe640 1 --2- 172.21.15.89:0/721615272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ccb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9affd640 1 --2- 172.21.15.89:0/721615272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ccb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9affd640 1 -- 172.21.15.89:0/721615272 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1e6ceb200 con 0x55c1e6cd2db0 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9b7fe640 1 --2- 172.21.15.89:0/721615272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ccb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a9affd640 1 --2- 172.21.15.89:0/721615272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6ccae90 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c031f80 tx=0x7f2a8c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45cf328eb4c4d6e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.448+0000 7f2a917fa640 1 -- 172.21.15.89:0/721615272 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c031830 con 0x55c1e6cd2db0 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2a917fa640 1 -- 172.21.15.89:0/721615272 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a8c036070 con 0x55c1e6cd2db0 2024-09-15T23:39:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2a917fa640 1 -- 172.21.15.89:0/721615272 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c030e90 con 0x55c1e6cd2db0 2024-09-15T23:39:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2aa47c2740 1 -- 172.21.15.89:0/721615272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 msgr2=0x55c1e6ccae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/721615272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6ccae90 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c031f80 tx=0x7f2a8c02bfd0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2aa47c2740 1 -- 172.21.15.89:0/721615272 shutdown_connections 2024-09-15T23:39:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/721615272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ccb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/721615272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd36f0 0x55c1e6ce42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/721615272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd2db0 0x55c1e6ccae90 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2aa47c2740 1 -- 172.21.15.89:0/721615272 >> 172.21.15.89:0/721615272 conn(0x55c1e6cca010 msgr2=0x55c1e6cdcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2aa47c2740 1 -- 172.21.15.89:0/721615272 shutdown_connections 2024-09-15T23:39:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2aa47c2740 1 -- 172.21.15.89:0/721615272 wait complete. 2024-09-15T23:39:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.449+0000 7f2aa47c2740 1 -- start start 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2aa47c2740 1 Processor -- start 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd2db0 0x55c1e6ce1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd36f0 0x55c1e6ce2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2aa47c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ce3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c1a9990 con 0x55c1e6cd36f0 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c1a92c0 con 0x55c1e6cd8780 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2aa47c2740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c1a6500 con 0x55c1e6cd2db0 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2aa47c2740 10 client.0 _ll_register_callbacks cb 0x7f2a8c009380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2a927fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd36f0 0x55c1e6ce2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2a927fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd36f0 0x55c1e6ce2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45100/0 (socket says 172.21.15.89:45100) 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2a927fc640 1 -- 172.21.15.89:0/3268345625 learned_addr learned my addr 172.21.15.89:0/3268345625 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2a9b7fe640 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ce3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2a9affd640 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd2db0 0x55c1e6ce1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2a927fc640 1 -- 172.21.15.89:0/3268345625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd2db0 msgr2=0x55c1e6ce1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2a927fc640 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd2db0 0x55c1e6ce1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2a927fc640 1 -- 172.21.15.89:0/3268345625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 msgr2=0x55c1e6ce3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2a927fc640 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ce3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.450+0000 7f2a927fc640 1 -- 172.21.15.89:0/3268345625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1e6cd84e0 con 0x55c1e6cd36f0 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2a927fc640 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd36f0 0x55c1e6ce2140 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400c010 tx=0x7f2a84004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a106f6529c121db6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2a7affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2a7affd640 1 -- 172.21.15.89:0/3268345625 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8402c070 con 0x55c1e6cd36f0 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2a7affd640 1 -- 172.21.15.89:0/3268345625 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2a84021780 con 0x55c1e6cd36f0 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2aa47c2740 1 -- 172.21.15.89:0/3268345625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a8c036070 con 0x55c1e6cd36f0 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2a7affd640 1 -- 172.21.15.89:0/3268345625 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a84038050 con 0x55c1e6cd36f0 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2aa47c2740 10 client.20724 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2aa47c2740 1 -- 172.21.15.89:0/3268345625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c1e6d53530 con 0x55c1e6cd36f0 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2a7affd640 1 -- 172.21.15.89:0/3268345625 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a8c036070 con 0x55c1e6cd36f0 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2a7affd640 1 -- 172.21.15.89:0/3268345625 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2a84028870 con 0x55c1e6cd36f0 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2aa47c2740 10 client.20724 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2aa47c2740 10 client.20724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2aa47c2740 20 client.20724 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c1e6d53e40 0x55c1e6d56260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2aa47c2740 4 client.20724 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.451+0000 7f2aa47c2740 1 -- 172.21.15.89:0/3268345625 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c1e6c39d70 con 0x55c1e6d53e40 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.452+0000 7f2a9affd640 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c1e6d53e40 0x55c1e6d56260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.452+0000 7f2a7affd640 10 client.20724 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.452+0000 7f2a9affd640 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c1e6d53e40 0x55c1e6d56260 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=536a923fa5a9768 server_cookie=aa9e1589b6ba5272 in_seq=0 out_seq=0 2024-09-15T23:39:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2a7affd640 1 -- 172.21.15.89:0/3268345625 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c1e6c39d70 con 0x55c1e6d53e40 2024-09-15T23:39:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2a7affd640 10 client.20724 handle_command_reply: tid=0 2024-09-15T23:39:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 client.20724 shutdown 2024-09-15T23:39:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 -- 172.21.15.89:0/3268345625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd36f0 msgr2=0x55c1e6ce2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd36f0 0x55c1e6ce2140 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400c010 tx=0x7f2a84004730 comp rx=0 tx=0).stop 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 -- 172.21.15.89:0/3268345625 shutdown_connections 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c1e6d53e40 0x55c1e6d56260 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1e6cd8780 0x55c1e6ce3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2a7affd640 0 client.20724 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c1e6cd36f0 0x55c1e6ce2140 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 --2- 172.21.15.89:0/3268345625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c1e6cd2db0 0x55c1e6ce1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 -- 172.21.15.89:0/3268345625 >> 172.21.15.89:0/3268345625 conn(0x55c1e6cca010 msgr2=0x55c1e6cc89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 -- 172.21.15.89:0/3268345625 shutdown_connections 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 1 -- 172.21.15.89:0/3268345625 wait complete. 2024-09-15T23:39:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.467+0000 7f2aa47c2740 20 client.20724 trim_cache size 0 max 16384 2024-09-15T23:39:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:39:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:39:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:39:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 -- 172.21.15.89:0/995344353 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a880a32f0 msgr2=0x7f2a880a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a880a32f0 0x7f2a880a57b0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c06a180 tx=0x7f2a840138e0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 -- 172.21.15.89:0/995344353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a9c1025f0 msgr2=0x7f2a9c19f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a9c1025f0 0x7f2a9c19f130 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400d750 tx=0x7f2a9400dc10 comp rx=0 tx=0).stop 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 -- 172.21.15.89:0/995344353 shutdown_connections 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a880a32f0 0x7f2a880a57b0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a9c1025f0 0x7f2a9c19f130 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a9c101c30 0x7f2a9c19cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 --2- 172.21.15.89:0/995344353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a9c100a40 0x7f2a9c06a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 -- 172.21.15.89:0/995344353 >> 172.21.15.89:0/995344353 conn(0x7f2a9c0fc1e0 msgr2=0x7f2a9c104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 -- 172.21.15.89:0/995344353 shutdown_connections 2024-09-15T23:39:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:21.468+0000 7f2aa194b640 1 -- 172.21.15.89:0/995344353 wait complete. 2024-09-15T23:39:21.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:21 smithi120 ceph-mon[35250]: pgmap v809: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:39:21.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:21 smithi174 ceph-mon[35586]: pgmap v809: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:39:21.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:21 smithi089 ceph-mon[30933]: pgmap v809: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 17 op/s 2024-09-15T23:39:23.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:23 smithi120 ceph-mon[35250]: pgmap v810: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:39:23.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:23 smithi174 ceph-mon[35586]: pgmap v810: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:39:23.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:23 smithi089 ceph-mon[30933]: pgmap v810: 129 pgs: 129 active+clean; 3.7 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 19 op/s 2024-09-15T23:39:25.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:25 smithi120 ceph-mon[35250]: pgmap v811: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:39:25.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:25 smithi174 ceph-mon[35586]: pgmap v811: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:39:25.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:25 smithi089 ceph-mon[30933]: pgmap v811: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 20 op/s 2024-09-15T23:39:27.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:27 smithi120 ceph-mon[35250]: pgmap v812: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:27.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:27 smithi174 ceph-mon[35586]: pgmap v812: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:27.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:27 smithi089 ceph-mon[30933]: pgmap v812: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:29.434 INFO:tasks.quiescer.fs.[cephfs]:Total duration (35.4) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T23:39:29.434 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.7 2024-09-15T23:39:29.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:39:29.464 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:29 smithi089 ceph-mon[30933]: pgmap v813: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:39:29.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:29 smithi120 ceph-mon[35250]: pgmap v813: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:39:29.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:29 smithi174 ceph-mon[35586]: pgmap v813: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 20 op/s 2024-09-15T23:39:29.699 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 -- 172.21.15.89:0/332260937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb60102da0 msgr2=0x7fbb60103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 --2- 172.21.15.89:0/332260937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb60102da0 0x7fbb60103180 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009900 tx=0x7fbb5002a480 comp rx=0 tx=0).stop 2024-09-15T23:39:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 -- 172.21.15.89:0/332260937 shutdown_connections 2024-09-15T23:39:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 --2- 172.21.15.89:0/332260937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb6010c180 0x7fbb6010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 --2- 172.21.15.89:0/332260937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb601036c0 0x7fbb6010bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:30.187 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 --2- 172.21.15.89:0/332260937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb60102da0 0x7fbb60103180 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 -- 172.21.15.89:0/332260937 >> 172.21.15.89:0/332260937 conn(0x7fbb600fc7a0 msgr2=0x7fbb600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 -- 172.21.15.89:0/332260937 shutdown_connections 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 -- 172.21.15.89:0/332260937 wait complete. 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 Processor -- start 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.168+0000 7fbb65a2d640 1 -- start start 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb65a2d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb60102da0 0x7fbb60198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb65a2d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb601036c0 0x7fbb60198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb65a2d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb6010c180 0x7fbb6019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb65a2d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb60111f60 con 0x7fbb6010c180 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb65a2d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb60111de0 con 0x7fbb601036c0 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb65a2d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb601120e0 con 0x7fbb60102da0 2024-09-15T23:39:30.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb5f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb6010c180 0x7fbb6019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb5f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb6010c180 0x7fbb6019cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45576/0 (socket says 172.21.15.89:45576) 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb5f7fe640 1 -- 172.21.15.89:0/3954875932 learned_addr learned my addr 172.21.15.89:0/3954875932 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb5effd640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb60102da0 0x7fbb60198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb567fc640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb601036c0 0x7fbb60198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb5f7fe640 1 -- 172.21.15.89:0/3954875932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb60102da0 msgr2=0x7fbb60198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb5f7fe640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb60102da0 0x7fbb60198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb5f7fe640 1 -- 172.21.15.89:0/3954875932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb601036c0 msgr2=0x7fbb60198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb5f7fe640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb601036c0 0x7fbb60198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb5f7fe640 1 -- 172.21.15.89:0/3954875932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6019d3c0 con 0x7fbb6010c180 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb5f7fe640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb6010c180 0x7fbb6019cce0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800eaa0 tx=0x7fbb5800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb64a2b640 1 -- 172.21.15.89:0/3954875932 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5800cdb0 con 0x7fbb6010c180 2024-09-15T23:39:30.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb64a2b640 1 -- 172.21.15.89:0/3954875932 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb58005020 con 0x7fbb6010c180 2024-09-15T23:39:30.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb65a2d640 1 -- 172.21.15.89:0/3954875932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb6019d6b0 con 0x7fbb6010c180 2024-09-15T23:39:30.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb64a2b640 1 -- 172.21.15.89:0/3954875932 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb580116b0 con 0x7fbb6010c180 2024-09-15T23:39:30.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.169+0000 7fbb65a2d640 1 -- 172.21.15.89:0/3954875932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb601a4f90 con 0x7fbb6010c180 2024-09-15T23:39:30.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.171+0000 7fbb65a2d640 1 -- 172.21.15.89:0/3954875932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb60103180 con 0x7fbb6010c180 2024-09-15T23:39:30.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.172+0000 7fbb64a2b640 1 -- 172.21.15.89:0/3954875932 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbb58005560 con 0x7fbb6010c180 2024-09-15T23:39:30.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.172+0000 7fbb64a2b640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb4c0a3240 0x7fbb4c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:30.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.173+0000 7fbb5effd640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb4c0a3240 0x7fbb4c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:30.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.173+0000 7fbb64a2b640 1 -- 172.21.15.89:0/3954875932 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb580d01a0 con 0x7fbb6010c180 2024-09-15T23:39:30.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.173+0000 7fbb5effd640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb4c0a3240 0x7fbb4c0a5700 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009230 tx=0x7fbb500058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:30.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.176+0000 7fbb64a2b640 1 -- 172.21.15.89:0/3954875932 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbb58016030 con 0x7fbb6010c180 2024-09-15T23:39:30.377 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.376+0000 7fbb65a2d640 1 -- 172.21.15.89:0/3954875932 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb60106c90 con 0x7fbb6010c180 2024-09-15T23:39:30.378 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.377+0000 7fbb64a2b640 1 -- 172.21.15.89:0/3954875932 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fbb5809c480 con 0x7fbb6010c180 2024-09-15T23:39:30.378 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:39:30.378 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:39:30.379 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:39:30.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 -- 172.21.15.89:0/3954875932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb4c0a3240 msgr2=0x7fbb4c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:30.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb4c0a3240 0x7fbb4c0a5700 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009230 tx=0x7fbb500058d0 comp rx=0 tx=0).stop 2024-09-15T23:39:30.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 -- 172.21.15.89:0/3954875932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb6010c180 msgr2=0x7fbb6019cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:30.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb6010c180 0x7fbb6019cce0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800eaa0 tx=0x7fbb5800ef60 comp rx=0 tx=0).stop 2024-09-15T23:39:30.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 -- 172.21.15.89:0/3954875932 shutdown_connections 2024-09-15T23:39:30.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb4c0a3240 0x7fbb4c0a5700 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:30.380 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb6010c180 0x7fbb6019cce0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb601036c0 0x7fbb60198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 --2- 172.21.15.89:0/3954875932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb60102da0 0x7fbb60198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 -- 172.21.15.89:0/3954875932 >> 172.21.15.89:0/3954875932 conn(0x7fbb600fc7a0 msgr2=0x7fbb600fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 -- 172.21.15.89:0/3954875932 shutdown_connections 2024-09-15T23:39:30.381 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:30.379+0000 7fbb65a2d640 1 -- 172.21.15.89:0/3954875932 wait complete. 2024-09-15T23:39:31.021 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0' 2024-09-15T23:39:31.021 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0 2024-09-15T23:39:31.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:31 smithi089 ceph-mon[30933]: pgmap v814: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:31.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:31 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3954875932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:31.320 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:31 smithi120 ceph-mon[35250]: pgmap v814: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:31.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:31 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3954875932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:31 smithi174 ceph-mon[35586]: pgmap v814: 129 pgs: 129 active+clean; 3.8 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-09-15T23:39:31.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:31 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3954875932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 -- 172.21.15.89:0/1243650138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0d8102d30 msgr2=0x7fd0d8103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 --2- 172.21.15.89:0/1243650138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0d8102d30 0x7fd0d8103110 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8009cd0 tx=0x7fd0c802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 -- 172.21.15.89:0/1243650138 shutdown_connections 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 --2- 172.21.15.89:0/1243650138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0d81082a0 0x7fd0d810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 --2- 172.21.15.89:0/1243650138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0d81036e0 0x7fd0d8107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 --2- 172.21.15.89:0/1243650138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0d8102d30 0x7fd0d8103110 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 -- 172.21.15.89:0/1243650138 >> 172.21.15.89:0/1243650138 conn(0x7fd0d80fc7d0 msgr2=0x7fd0d80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 -- 172.21.15.89:0/1243650138 shutdown_connections 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 -- 172.21.15.89:0/1243650138 wait complete. 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 Processor -- start 2024-09-15T23:39:31.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 -- start start 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0d8102d30 0x7fd0d8198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0d81036e0 0x7fd0d8198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0d81082a0 0x7fd0d819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81120a0 con 0x7fd0d81082a0 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.906+0000 7fd0df947640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d8111f20 con 0x7fd0d81036e0 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0df947640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d8112220 con 0x7fd0d8102d30 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0ddee4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0d81082a0 0x7fd0d819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0ddee4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0d81082a0 0x7fd0d819ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45594/0 (socket says 172.21.15.89:45594) 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0ddee4640 1 -- 172.21.15.89:0/3528128859 learned_addr learned my addr 172.21.15.89:0/3528128859 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0d81036e0 0x7fd0d8198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0dd6e3640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0d8102d30 0x7fd0d8198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0ddee4640 1 -- 172.21.15.89:0/3528128859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0d8102d30 msgr2=0x7fd0d8198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:31.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0ddee4640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0d8102d30 0x7fd0d8198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0ddee4640 1 -- 172.21.15.89:0/3528128859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0d81036e0 msgr2=0x7fd0d8198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0ddee4640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0d81036e0 0x7fd0d8198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0ddee4640 1 -- 172.21.15.89:0/3528128859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d819d4e0 con 0x7fd0d81082a0 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0dd6e3640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0d8102d30 0x7fd0d8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0ddee4640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0d81082a0 0x7fd0d819ce00 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000c970 tx=0x7fd0d000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0d67fc640 1 -- 172.21.15.89:0/3528128859 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0007bb0 con 0x7fd0d81082a0 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0d67fc640 1 -- 172.21.15.89:0/3528128859 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0d0007d50 con 0x7fd0d81082a0 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0d67fc640 1 -- 172.21.15.89:0/3528128859 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0018800 con 0x7fd0d81082a0 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0df947640 1 -- 172.21.15.89:0/3528128859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d819d7d0 con 0x7fd0d81082a0 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.907+0000 7fd0df947640 1 -- 172.21.15.89:0/3528128859 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d81a50b0 con 0x7fd0d81082a0 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0d67fc640 1 -- 172.21.15.89:0/3528128859 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd0d000fa50 con 0x7fd0d81082a0 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0e1d20740 1 Processor -- start 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0e1d20740 1 -- start start 2024-09-15T23:39:31.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41c640 0x5574ff41ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41cf60 0x5574ff42b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff42bd50 0x5574ff42e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5574ff3822a0 con 0x5574ff41c640 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5574ff329b10 con 0x5574ff42bd50 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5574ff37e260 con 0x5574ff41cf60 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0d67fc640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0c00a32c0 0x7fd0c00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.910+0000 7fd0dcee2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41c640 0x5574ff41ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0dcee2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41c640 0x5574ff41ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45608/0 (socket says 172.21.15.89:45608) 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0dcee2640 1 -- 172.21.15.89:0/2325734639 learned_addr learned my addr 172.21.15.89:0/2325734639 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0ddee4640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0c00a32c0 0x7fd0c00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0dd6e3640 1 --2- 172.21.15.89:0/2325734639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41cf60 0x5574ff42b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0d67fc640 1 -- 172.21.15.89:0/3528128859 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd0d00cfd00 con 0x7fd0d81082a0 2024-09-15T23:39:31.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/2325734639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff42bd50 0x5574ff42e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0dcee2640 1 -- 172.21.15.89:0/2325734639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41cf60 msgr2=0x5574ff42b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/2325734639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41cf60 0x5574ff42b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0dcee2640 1 -- 172.21.15.89:0/2325734639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff42bd50 msgr2=0x5574ff42e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/2325734639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff42bd50 0x5574ff42e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0dcee2640 1 -- 172.21.15.89:0/2325734639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574ff0f3b90 con 0x5574ff41c640 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0ddee4640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0c00a32c0 0x7fd0c00a5780 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0098390 tx=0x7fd0d000fdf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/2325734639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41c640 0x5574ff41ca20 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc005d70 tx=0x7fd0cc00c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6932c4ace3531529 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0bf7fe640 1 -- 172.21.15.89:0/2325734639 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc018030 con 0x5574ff41c640 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0bf7fe640 1 -- 172.21.15.89:0/2325734639 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0cc005500 con 0x5574ff41c640 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.911+0000 7fd0bf7fe640 1 -- 172.21.15.89:0/2325734639 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc00d7b0 con 0x5574ff41c640 2024-09-15T23:39:31.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 -- 172.21.15.89:0/2325734639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41c640 msgr2=0x5574ff41ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/2325734639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41c640 0x5574ff41ca20 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc005d70 tx=0x7fd0cc00c560 comp rx=0 tx=0).stop 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 -- 172.21.15.89:0/2325734639 shutdown_connections 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/2325734639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff42bd50 0x5574ff42e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/2325734639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41cf60 0x5574ff42b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/2325734639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41c640 0x5574ff41ca20 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 -- 172.21.15.89:0/2325734639 >> 172.21.15.89:0/2325734639 conn(0x5574ff418cf0 msgr2=0x5574ff4190d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 -- 172.21.15.89:0/2325734639 shutdown_connections 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 -- 172.21.15.89:0/2325734639 wait complete. 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 -- start start 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.912+0000 7fd0e1d20740 1 Processor -- start 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41c640 0x5574ff428c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41cf60 0x5574ff429170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff42bd50 0x5574ff4296b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5574ff42e920 con 0x5574ff41cf60 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5574ff30a6e0 con 0x5574ff41c640 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5574ff42eaa0 con 0x5574ff42bd50 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0e1d20740 10 client.0 _ll_register_callbacks cb 0x5574ff32ea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dd6e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41cf60 0x5574ff429170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dd6e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41cf60 0x5574ff429170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45614/0 (socket says 172.21.15.89:45614) 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dd6e3640 1 -- 172.21.15.89:0/1568809072 learned_addr learned my addr 172.21.15.89:0/1568809072 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41c640 0x5574ff428c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff42bd50 0x5574ff4296b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dd6e3640 1 -- 172.21.15.89:0/1568809072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff42bd50 msgr2=0x5574ff4296b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dd6e3640 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff42bd50 0x5574ff4296b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dd6e3640 1 -- 172.21.15.89:0/1568809072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41c640 msgr2=0x5574ff428c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dd6e3640 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41c640 0x5574ff428c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dd6e3640 1 -- 172.21.15.89:0/1568809072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574ff437e20 con 0x5574ff41cf60 2024-09-15T23:39:31.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.913+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff42bd50 0x5574ff4296b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:31.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0dd6e3640 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41cf60 0x5574ff429170 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8031c30 tx=0x7fd0c8031c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ae67ba438132f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:31.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:31.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0bcff9640 1 -- 172.21.15.89:0/1568809072 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8049060 con 0x5574ff41cf60 2024-09-15T23:39:31.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0bcff9640 1 -- 172.21.15.89:0/1568809072 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd0c8003b80 con 0x5574ff41cf60 2024-09-15T23:39:31.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0e1d20740 1 -- 172.21.15.89:0/1568809072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0cc005500 con 0x5574ff41cf60 2024-09-15T23:39:31.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0bcff9640 1 -- 172.21.15.89:0/1568809072 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd0c8003d80 con 0x5574ff41cf60 2024-09-15T23:39:31.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0e1d20740 10 client.20742 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:31.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0e1d20740 1 -- 172.21.15.89:0/1568809072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5574ff43a7d0 con 0x5574ff41cf60 2024-09-15T23:39:31.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0bcff9640 1 -- 172.21.15.89:0/1568809072 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0cc005500 con 0x5574ff41cf60 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0bcff9640 1 -- 172.21.15.89:0/1568809072 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd0c8030070 con 0x5574ff41cf60 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0e1d20740 10 client.20742 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0e1d20740 10 client.20742 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0e1d20740 20 client.20742 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5574ff43bc90 0x5574ff43e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0e1d20740 4 client.20742 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0e1d20740 1 -- 172.21.15.89:0/1568809072 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574ff38f9d0 con 0x5574ff43bc90 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.914+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5574ff43bc90 0x5574ff43e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.915+0000 7fd0bcff9640 10 client.20742 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.915+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5574ff43bc90 0x5574ff43e0c0 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96b12accb81fac60 server_cookie=8f8d3b8044485370 in_seq=0 out_seq=0 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0bcff9640 1 -- 172.21.15.89:0/1568809072 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5574ff38f9d0 con 0x5574ff43bc90 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0bcff9640 10 client.20742 handle_command_reply: tid=0 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 client.20742 shutdown 2024-09-15T23:39:31.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 -- 172.21.15.89:0/1568809072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41cf60 msgr2=0x5574ff429170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41cf60 0x5574ff429170 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8031c30 tx=0x7fd0c8031c60 comp rx=0 tx=0).stop 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 -- 172.21.15.89:0/1568809072 shutdown_connections 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5574ff43bc90 0x5574ff43e0c0 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff42bd50 0x5574ff4296b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0bcff9640 0 client.20742 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff41cf60 0x5574ff429170 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/1568809072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41c640 0x5574ff428c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 -- 172.21.15.89:0/1568809072 >> 172.21.15.89:0/1568809072 conn(0x5574ff418cf0 msgr2=0x5574ff41ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 -- 172.21.15.89:0/1568809072 shutdown_connections 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 1 -- 172.21.15.89:0/1568809072 wait complete. 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.916+0000 7fd0e1d20740 20 client.20742 trim_cache size 0 max 16384 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0e1d20740 1 Processor -- start 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0e1d20740 1 -- start start 2024-09-15T23:39:31.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41c640 0x5574ff42ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41cf60 0x5574ff4439d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff443f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81acbc0 con 0x5574ff42bd50 2024-09-15T23:39:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81ae700 con 0x5574ff41cf60 2024-09-15T23:39:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81ad290 con 0x5574ff41c640 2024-09-15T23:39:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0dcee2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff443f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0dcee2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff443f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45622/0 (socket says 172.21.15.89:45622) 2024-09-15T23:39:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0dcee2640 1 -- 172.21.15.89:0/3720679866 learned_addr learned my addr 172.21.15.89:0/3720679866 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0dd6e3640 1 --2- 172.21.15.89:0/3720679866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41cf60 0x5574ff4439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0dcee2640 1 -- 172.21.15.89:0/3720679866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41c640 msgr2=0x5574ff42ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/3720679866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41c640 0x5574ff42ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0dcee2640 1 -- 172.21.15.89:0/3720679866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41cf60 msgr2=0x5574ff4439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/3720679866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41cf60 0x5574ff4439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0dcee2640 1 -- 172.21.15.89:0/3720679866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574ff414260 con 0x5574ff42bd50 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.933+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/3720679866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff443f10 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0058c0 tx=0x7fd0cc004c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c0d31eb3c6f38a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0beffd640 1 -- 172.21.15.89:0/3720679866 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc018040 con 0x5574ff42bd50 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0beffd640 1 -- 172.21.15.89:0/3720679866 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0cc02e020 con 0x5574ff42bd50 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0beffd640 1 -- 172.21.15.89:0/3720679866 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc003810 con 0x5574ff42bd50 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3720679866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 msgr2=0x5574ff443f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/3720679866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff443f10 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0058c0 tx=0x7fd0cc004c90 comp rx=0 tx=0).stop 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3720679866 shutdown_connections 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/3720679866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff443f10 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/3720679866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41cf60 0x5574ff4439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/3720679866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41c640 0x5574ff42ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3720679866 >> 172.21.15.89:0/3720679866 conn(0x5574ff418cf0 msgr2=0x5574ff41e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3720679866 shutdown_connections 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.934+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3720679866 wait complete. 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0e1d20740 1 -- start start 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0e1d20740 1 Processor -- start 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41c640 0x5574ff430b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41cf60 0x5574ff4310a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0e1d20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff4315e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81ae030 con 0x5574ff42bd50 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81ad960 con 0x5574ff41cf60 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0e1d20740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d81aaba0 con 0x5574ff41c640 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0e1d20740 10 client.0 _ll_register_callbacks cb 0x7fd0c80098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:31.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0dcee2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff4315e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0dcee2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff4315e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45630/0 (socket says 172.21.15.89:45630) 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0dcee2640 1 -- 172.21.15.89:0/3959568800 learned_addr learned my addr 172.21.15.89:0/3959568800 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41c640 0x5574ff430b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.935+0000 7fd0dd6e3640 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41cf60 0x5574ff4310a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0dcee2640 1 -- 172.21.15.89:0/3959568800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41c640 msgr2=0x5574ff430b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41c640 0x5574ff430b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0dcee2640 1 -- 172.21.15.89:0/3959568800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41cf60 msgr2=0x5574ff4310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41cf60 0x5574ff4310a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0dcee2640 1 -- 172.21.15.89:0/3959568800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574ff4a6a30 con 0x5574ff42bd50 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0dcee2640 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff4315e0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00c880 tx=0x7fd0cc0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e409c1b7e8f6a196 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0bcff9640 1 -- 172.21.15.89:0/3959568800 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc018040 con 0x5574ff42bd50 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0bcff9640 1 -- 172.21.15.89:0/3959568800 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd0cc02ae80 con 0x5574ff42bd50 2024-09-15T23:39:31.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3959568800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0cc02e020 con 0x5574ff42bd50 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0bcff9640 1 -- 172.21.15.89:0/3959568800 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd0cc02a400 con 0x5574ff42bd50 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0e1d20740 10 client.20748 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3959568800 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5574ff4a7cf0 con 0x5574ff42bd50 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0bcff9640 1 -- 172.21.15.89:0/3959568800 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0cc02e020 con 0x5574ff42bd50 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0bcff9640 1 -- 172.21.15.89:0/3959568800 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fd0cc00d950 con 0x5574ff42bd50 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0e1d20740 10 client.20748 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0e1d20740 10 client.20748 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.936+0000 7fd0e1d20740 20 client.20748 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.937+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5574ff4a81e0 0x5574ff4aa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.937+0000 7fd0e1d20740 4 client.20748 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}] 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.937+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3959568800 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}) -- 0x5574ff389d20 con 0x5574ff4a81e0 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.937+0000 7fd0dd6e3640 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5574ff4a81e0 0x5574ff4aa5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:31.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.937+0000 7fd0dd6e3640 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5574ff4a81e0 0x5574ff4aa5d0 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20d8273a9a8eeb66 server_cookie=dc025e7e4f0d7e3b in_seq=0 out_seq=0 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.937+0000 7fd0bcff9640 10 client.20748 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.949+0000 7fd0bcff9640 1 -- 172.21.15.89:0/3959568800 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5574ff389d20 con 0x5574ff4a81e0 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.949+0000 7fd0bcff9640 10 client.20748 handle_command_reply: tid=0 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.949+0000 7fd0e1d20740 1 client.20748 shutdown 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3959568800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 msgr2=0x5574ff4315e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff4315e0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00c880 tx=0x7fd0cc0035f0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3959568800 shutdown_connections 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5574ff4a81e0 0x5574ff4aa5d0 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574ff42bd50 0x5574ff4315e0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5574ff41cf60 0x5574ff4310a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0bcff9640 0 client.20748 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 1 --2- 172.21.15.89:0/3959568800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5574ff41c640 0x5574ff430b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3959568800 >> 172.21.15.89:0/3959568800 conn(0x5574ff418cf0 msgr2=0x5574ff428bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:31.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3959568800 shutdown_connections 2024-09-15T23:39:31.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 1 -- 172.21.15.89:0/3959568800 wait complete. 2024-09-15T23:39:31.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0e1d20740 20 client.20748 trim_cache size 0 max 16384 2024-09-15T23:39:31.990 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:39:31.990 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:39:31.990 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:39:31.990 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 54, 2024-09-15T23:39:31.990 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:39:31.990 INFO:teuthology.orchestra.run.smithi089.stdout: "58c1329f": { 2024-09-15T23:39:31.990 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 54, 2024-09-15T23:39:31.990 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 137.7, 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:39:31.991 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:31.992 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:31.992 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:31.992 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:31.992 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:31.992 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:39:31.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.950+0000 7fd0df947640 1 -- 172.21.15.89:0/3528128859 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0c00a32c0 msgr2=0x7fd0c00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:31.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0c00a32c0 0x7fd0c00a5780 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0098390 tx=0x7fd0d000fdf0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 -- 172.21.15.89:0/3528128859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0d81082a0 msgr2=0x7fd0d819ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0d81082a0 0x7fd0d819ce00 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000c970 tx=0x7fd0d000ce30 comp rx=0 tx=0).stop 2024-09-15T23:39:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 -- 172.21.15.89:0/3528128859 shutdown_connections 2024-09-15T23:39:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd0c00a32c0 0x7fd0c00a5780 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd0d81082a0 0x7fd0d819ce00 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0d81036e0 0x7fd0d8198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 --2- 172.21.15.89:0/3528128859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0d8102d30 0x7fd0d8198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 -- 172.21.15.89:0/3528128859 >> 172.21.15.89:0/3528128859 conn(0x7fd0d80fc7d0 msgr2=0x7fd0d8105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 -- 172.21.15.89:0/3528128859 shutdown_connections 2024-09-15T23:39:31.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:31.951+0000 7fd0df947640 1 -- 172.21.15.89:0/3528128859 wait complete. 2024-09-15T23:39:32.708 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '58c1329f', quiesce took 3.3 seconds. Will release after: 14.399999999999999 2024-09-15T23:39:33.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:33 smithi120 ceph-mon[35250]: pgmap v815: 129 pgs: 129 active+clean; 3.8 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:39:33.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:33 smithi174 ceph-mon[35586]: pgmap v815: 129 pgs: 129 active+clean; 3.8 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:39:33.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:33 smithi089 ceph-mon[30933]: pgmap v815: 129 pgs: 129 active+clean; 3.8 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 18 op/s 2024-09-15T23:39:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.567+0000 7fdf3d40e640 1 -- 172.21.15.89:0/3902537079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf38101b60 msgr2=0x7fdf38101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.567+0000 7fdf3d40e640 1 --2- 172.21.15.89:0/3902537079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf38101b60 0x7fdf38101f60 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009900 tx=0x7fdf2802a410 comp rx=0 tx=0).stop 2024-09-15T23:39:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.567+0000 7fdf3d40e640 1 -- 172.21.15.89:0/3902537079 shutdown_connections 2024-09-15T23:39:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.567+0000 7fdf3d40e640 1 --2- 172.21.15.89:0/3902537079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf38103710 0x7fdf38109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.567+0000 7fdf3d40e640 1 --2- 172.21.15.89:0/3902537079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf38102d50 0x7fdf381031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.567+0000 7fdf3d40e640 1 --2- 172.21.15.89:0/3902537079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf38101b60 0x7fdf38101f60 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.567+0000 7fdf3d40e640 1 -- 172.21.15.89:0/3902537079 >> 172.21.15.89:0/3902537079 conn(0x7fdf380fd2e0 msgr2=0x7fdf380ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.567+0000 7fdf3d40e640 1 -- 172.21.15.89:0/3902537079 shutdown_connections 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.567+0000 7fdf3d40e640 1 -- 172.21.15.89:0/3902537079 wait complete. 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.568+0000 7fdf3d40e640 1 Processor -- start 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.568+0000 7fdf3d40e640 1 -- start start 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.568+0000 7fdf3d40e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf38101b60 0x7fdf38197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.568+0000 7fdf3d40e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf38102d50 0x7fdf381982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.568+0000 7fdf3d40e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf38103710 0x7fdf3819f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.568+0000 7fdf3d40e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3810d7c0 con 0x7fdf38101b60 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.568+0000 7fdf3d40e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3810d640 con 0x7fdf38103710 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.568+0000 7fdf3d40e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3810d940 con 0x7fdf38102d50 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf36ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf38101b60 0x7fdf38197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf36ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf38101b60 0x7fdf38197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45648/0 (socket says 172.21.15.89:45648) 2024-09-15T23:39:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf36ffd640 1 -- 172.21.15.89:0/4103876186 learned_addr learned my addr 172.21.15.89:0/4103876186 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf377fe640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf38103710 0x7fdf3819f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf367fc640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf38102d50 0x7fdf381982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf36ffd640 1 -- 172.21.15.89:0/4103876186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf38102d50 msgr2=0x7fdf381982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf38102d50 0x7fdf381982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf36ffd640 1 -- 172.21.15.89:0/4103876186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf38103710 msgr2=0x7fdf3819f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf38103710 0x7fdf3819f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf36ffd640 1 -- 172.21.15.89:0/4103876186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3819f9c0 con 0x7fdf38101b60 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf367fc640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf38102d50 0x7fdf381982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf38101b60 0x7fdf38197db0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7fdf28031610 tx=0x7fdf280043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf1ffff640 1 -- 172.21.15.89:0/4103876186 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2803f030 con 0x7fdf38101b60 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf1ffff640 1 -- 172.21.15.89:0/4103876186 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf28003ce0 con 0x7fdf38101b60 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf3d40e640 1 -- 172.21.15.89:0/4103876186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf3819fc50 con 0x7fdf38101b60 2024-09-15T23:39:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf1ffff640 1 -- 172.21.15.89:0/4103876186 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28035070 con 0x7fdf38101b60 2024-09-15T23:39:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.569+0000 7fdf3d40e640 1 -- 172.21.15.89:0/4103876186 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf381a0140 con 0x7fdf38101b60 2024-09-15T23:39:34.581 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.571+0000 7fdf1ffff640 1 -- 172.21.15.89:0/4103876186 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fdf28042030 con 0x7fdf38101b60 2024-09-15T23:39:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf40285740 1 Processor -- start 2024-09-15T23:39:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf40285740 1 -- start start 2024-09-15T23:39:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf40285740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 0x55c9304181b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf40285740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 0x55c93041d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf40285740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c93041fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf40285740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c9303772f0 con 0x55c93041d780 2024-09-15T23:39:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf40285740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c93031f270 con 0x55c9304186f0 2024-09-15T23:39:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf40285740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c9303732b0 con 0x55c930417db0 2024-09-15T23:39:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf1ffff640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf200a32f0 0x7fdf200a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf367fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c93041fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf367fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c93041fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45656/0 (socket says 172.21.15.89:45656) 2024-09-15T23:39:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf367fc640 1 -- 172.21.15.89:0/3780779901 learned_addr learned my addr 172.21.15.89:0/3780779901 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf377fe640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf200a32f0 0x7fdf200a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf1ffff640 1 -- 172.21.15.89:0/4103876186 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdf280f04f0 con 0x7fdf38101b60 2024-09-15T23:39:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf377fe640 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 0x55c9304181b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.572+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 0x55c93041d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf367fc640 1 -- 172.21.15.89:0/3780779901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 msgr2=0x55c9304181b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf367fc640 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 0x55c9304181b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf367fc640 1 -- 172.21.15.89:0/3780779901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 msgr2=0x55c93041d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf367fc640 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 0x55c93041d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf367fc640 1 -- 172.21.15.89:0/3780779901 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c93031e340 con 0x55c93041d780 2024-09-15T23:39:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf377fe640 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 0x55c9304181b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 0x55c93041d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf367fc640 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c93041fc40 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00bb70 tx=0x7fdf2c00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffa9ad47bace7f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf377fe640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf200a32f0 0x7fdf200a57b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fdf30007fd0 tx=0x7fdf30007410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf1d7fa640 1 -- 172.21.15.89:0/3780779901 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c017620 con 0x55c93041d780 2024-09-15T23:39:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf1d7fa640 1 -- 172.21.15.89:0/3780779901 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf2c017ca0 con 0x55c93041d780 2024-09-15T23:39:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.573+0000 7fdf1d7fa640 1 -- 172.21.15.89:0/3780779901 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c00fc00 con 0x55c93041d780 2024-09-15T23:39:34.584 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 -- 172.21.15.89:0/3780779901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 msgr2=0x55c93041fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c93041fc40 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00bb70 tx=0x7fdf2c00d710 comp rx=0 tx=0).stop 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 -- 172.21.15.89:0/3780779901 shutdown_connections 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c93041fc40 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 0x55c93041d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 --2- 172.21.15.89:0/3780779901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 0x55c9304181b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 -- 172.21.15.89:0/3780779901 >> 172.21.15.89:0/3780779901 conn(0x55c93040f010 msgr2=0x55c93040f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 -- 172.21.15.89:0/3780779901 shutdown_connections 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 -- 172.21.15.89:0/3780779901 wait complete. 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 -- start start 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 Processor -- start 2024-09-15T23:39:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 0x55c930421f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930422450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 0x55c930429490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55c930420750 con 0x55c9304186f0 2024-09-15T23:39:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c930300290 con 0x55c930417db0 2024-09-15T23:39:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c9304208d0 con 0x55c93041d780 2024-09-15T23:39:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf367fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930422450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.574+0000 7fdf40285740 10 client.0 _ll_register_callbacks cb 0x55c930331140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf367fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930422450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45664/0 (socket says 172.21.15.89:45664) 2024-09-15T23:39:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf367fc640 1 -- 172.21.15.89:0/4100686586 learned_addr learned my addr 172.21.15.89:0/4100686586 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf377fe640 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 0x55c930429490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 0x55c930421f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf367fc640 1 -- 172.21.15.89:0/4100686586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 msgr2=0x55c930429490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf367fc640 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 0x55c930429490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf367fc640 1 -- 172.21.15.89:0/4100686586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 msgr2=0x55c930421f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf367fc640 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 0x55c930421f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf367fc640 1 -- 172.21.15.89:0/4100686586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c93008a210 con 0x55c9304186f0 2024-09-15T23:39:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 0x55c930421f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf367fc640 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930422450 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00dc10 tx=0x7fdf2c017650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfae96385f59bb5a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf1aff5640 1 -- 172.21.15.89:0/4100686586 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c026070 con 0x55c9304186f0 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf1aff5640 1 -- 172.21.15.89:0/4100686586 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdf2c022440 con 0x55c9304186f0 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf40285740 1 -- 172.21.15.89:0/4100686586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf2c017ca0 con 0x55c9304186f0 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf1aff5640 1 -- 172.21.15.89:0/4100686586 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdf2c032050 con 0x55c9304186f0 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf40285740 10 client.20760 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.575+0000 7fdf40285740 1 -- 172.21.15.89:0/4100686586 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c930156620 con 0x55c9304186f0 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf1aff5640 1 -- 172.21.15.89:0/4100686586 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf2c017ca0 con 0x55c9304186f0 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf1aff5640 1 -- 172.21.15.89:0/4100686586 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fdf2c022b20 con 0x55c9304186f0 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf40285740 10 client.20760 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf40285740 10 client.20760 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf40285740 20 client.20760 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf40285740 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c93042d8b0 0x55c93042fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf40285740 4 client.20760 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf40285740 1 -- 172.21.15.89:0/4100686586 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c930384a20 con 0x55c93042d8b0 2024-09-15T23:39:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c93042d8b0 0x55c93042fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c93042d8b0 0x55c93042fca0 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca70011754f275c9 server_cookie=ea57d9f8b1f2edd8 in_seq=0 out_seq=0 2024-09-15T23:39:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.576+0000 7fdf1aff5640 10 client.20760 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf1aff5640 1 -- 172.21.15.89:0/4100686586 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c930384a20 con 0x55c93042d8b0 2024-09-15T23:39:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf1aff5640 10 client.20760 handle_command_reply: tid=0 2024-09-15T23:39:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 client.20760 shutdown 2024-09-15T23:39:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 -- 172.21.15.89:0/4100686586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 msgr2=0x55c930422450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930422450 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00dc10 tx=0x7fdf2c017650 comp rx=0 tx=0).stop 2024-09-15T23:39:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 -- 172.21.15.89:0/4100686586 shutdown_connections 2024-09-15T23:39:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c93042d8b0 0x55c93042fca0 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 0x55c930429490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf1aff5640 0 client.20760 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930422450 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 --2- 172.21.15.89:0/4100686586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 0x55c930421f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 -- 172.21.15.89:0/4100686586 >> 172.21.15.89:0/4100686586 conn(0x55c93040f010 msgr2=0x55c930410820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 -- 172.21.15.89:0/4100686586 shutdown_connections 2024-09-15T23:39:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.577+0000 7fdf40285740 1 -- 172.21.15.89:0/4100686586 wait complete. 2024-09-15T23:39:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.578+0000 7fdf40285740 20 client.20760 trim_cache size 0 max 16384 2024-09-15T23:39:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.592+0000 7fdf40285740 1 Processor -- start 2024-09-15T23:39:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.592+0000 7fdf40285740 1 -- start start 2024-09-15T23:39:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.592+0000 7fdf40285740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 0x55c930410400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.592+0000 7fdf40285740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 0x55c930429450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.592+0000 7fdf40285740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c930410940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.592+0000 7fdf40285740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381a85e0 con 0x55c93041d780 2024-09-15T23:39:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.592+0000 7fdf40285740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381aa120 con 0x55c9304186f0 2024-09-15T23:39:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.592+0000 7fdf40285740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381a8cb0 con 0x55c930417db0 2024-09-15T23:39:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf377fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c930410940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf377fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c930410940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45674/0 (socket says 172.21.15.89:45674) 2024-09-15T23:39:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf377fe640 1 -- 172.21.15.89:0/1125646914 learned_addr learned my addr 172.21.15.89:0/1125646914 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:34.599 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf367fc640 1 --2- 172.21.15.89:0/1125646914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 0x55c930429450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/1125646914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 0x55c930410400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf377fe640 1 -- 172.21.15.89:0/1125646914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 msgr2=0x55c930410400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf377fe640 1 --2- 172.21.15.89:0/1125646914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 0x55c930410400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf377fe640 1 -- 172.21.15.89:0/1125646914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 msgr2=0x55c930429450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf377fe640 1 --2- 172.21.15.89:0/1125646914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 0x55c930429450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf377fe640 1 -- 172.21.15.89:0/1125646914 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c930430010 con 0x55c93041d780 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf377fe640 1 --2- 172.21.15.89:0/1125646914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c930410940 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fdf30007fa0 tx=0x7fdf30012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb037d56b83bb87 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf1cff9640 1 -- 172.21.15.89:0/1125646914 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf300119c0 con 0x55c93041d780 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf1cff9640 1 -- 172.21.15.89:0/1125646914 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf30011b60 con 0x55c93041d780 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.593+0000 7fdf1cff9640 1 -- 172.21.15.89:0/1125646914 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30028dc0 con 0x55c93041d780 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 -- 172.21.15.89:0/1125646914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 msgr2=0x55c930410940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 --2- 172.21.15.89:0/1125646914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c930410940 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fdf30007fa0 tx=0x7fdf30012000 comp rx=0 tx=0).stop 2024-09-15T23:39:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 -- 172.21.15.89:0/1125646914 shutdown_connections 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 --2- 172.21.15.89:0/1125646914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93041d780 0x55c930410940 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 --2- 172.21.15.89:0/1125646914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c9304186f0 0x55c930429450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 --2- 172.21.15.89:0/1125646914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c930417db0 0x55c930410400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 -- 172.21.15.89:0/1125646914 >> 172.21.15.89:0/1125646914 conn(0x55c93040f010 msgr2=0x55c93040e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 -- 172.21.15.89:0/1125646914 shutdown_connections 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 -- 172.21.15.89:0/1125646914 wait complete. 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 -- start start 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 Processor -- start 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 0x55c9304267e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930426d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 0x55c930427c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381a9a50 con 0x55c9304186f0 2024-09-15T23:39:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381a9380 con 0x55c930417db0 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.594+0000 7fdf40285740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdf381a65c0 con 0x55c93041d780 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf40285740 10 client.0 _ll_register_callbacks cb 0x7fdf2c00b9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf367fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930426d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf367fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930426d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45682/0 (socket says 172.21.15.89:45682) 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf367fc640 1 -- 172.21.15.89:0/3906608446 learned_addr learned my addr 172.21.15.89:0/3906608446 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 0x55c9304267e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf377fe640 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 0x55c930427c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf367fc640 1 -- 172.21.15.89:0/3906608446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 msgr2=0x55c930427c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf367fc640 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 0x55c930427c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf367fc640 1 -- 172.21.15.89:0/3906608446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 msgr2=0x55c9304267e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf367fc640 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 0x55c9304267e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf367fc640 1 -- 172.21.15.89:0/3906608446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c930428280 con 0x55c9304186f0 2024-09-15T23:39:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf377fe640 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 0x55c930427c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 0x55c9304267e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf367fc640 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930426d20 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x55c9304233f0 tx=0x7fdf2c00b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7abafba728d53b0e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf1aff5640 1 -- 172.21.15.89:0/3906608446 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c026040 con 0x55c9304186f0 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf1aff5640 1 -- 172.21.15.89:0/3906608446 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdf2c0179d0 con 0x55c9304186f0 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf40285740 1 -- 172.21.15.89:0/3906608446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf30011b60 con 0x55c9304186f0 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf1aff5640 1 -- 172.21.15.89:0/3906608446 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdf2c005b20 con 0x55c9304186f0 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf40285740 10 client.20766 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.595+0000 7fdf40285740 1 -- 172.21.15.89:0/3906608446 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c930429590 con 0x55c9304186f0 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf1aff5640 1 -- 172.21.15.89:0/3906608446 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf30011b60 con 0x55c9304186f0 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf1aff5640 1 -- 172.21.15.89:0/3906608446 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fdf2c017c10 con 0x55c9304186f0 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf40285740 10 client.20766 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf40285740 10 client.20766 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf40285740 20 client.20766 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf40285740 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c930498b00 0x55c93049af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf40285740 4 client.20766 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:39:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf40285740 1 -- 172.21.15.89:0/3906608446 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c93037ed70 con 0x55c930498b00 2024-09-15T23:39:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c930498b00 0x55c93049af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf36ffd640 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c930498b00 0x55c93049af20 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=263081f65c871d94 server_cookie=fbe8f65d876ce40f in_seq=0 out_seq=0 2024-09-15T23:39:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.596+0000 7fdf1aff5640 10 client.20766 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf1aff5640 1 -- 172.21.15.89:0/3906608446 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c93037ed70 con 0x55c930498b00 2024-09-15T23:39:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf1aff5640 10 client.20766 handle_command_reply: tid=0 2024-09-15T23:39:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 client.20766 shutdown 2024-09-15T23:39:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 -- 172.21.15.89:0/3906608446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 msgr2=0x55c930426d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930426d20 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x55c9304233f0 tx=0x7fdf2c00b830 comp rx=0 tx=0).stop 2024-09-15T23:39:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 -- 172.21.15.89:0/3906608446 shutdown_connections 2024-09-15T23:39:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55c930498b00 0x55c93049af20 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c93041d780 0x55c930427c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9304186f0 0x55c930426d20 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 --2- 172.21.15.89:0/3906608446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c930417db0 0x55c9304267e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf1aff5640 0 client.20766 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 -- 172.21.15.89:0/3906608446 >> 172.21.15.89:0/3906608446 conn(0x55c93040f010 msgr2=0x55c93040e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 -- 172.21.15.89:0/3906608446 shutdown_connections 2024-09-15T23:39:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.612+0000 7fdf40285740 1 -- 172.21.15.89:0/3906608446 wait complete. 2024-09-15T23:39:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf40285740 20 client.20766 trim_cache size 0 max 16384 2024-09-15T23:39:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:39:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:39:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:39:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:39:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 -- 172.21.15.89:0/4103876186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf200a32f0 msgr2=0x7fdf200a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf200a32f0 0x7fdf200a57b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fdf30007fd0 tx=0x7fdf30007410 comp rx=0 tx=0).stop 2024-09-15T23:39:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 -- 172.21.15.89:0/4103876186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf38101b60 msgr2=0x7fdf38197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf38101b60 0x7fdf38197db0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7fdf28031610 tx=0x7fdf280043b0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 -- 172.21.15.89:0/4103876186 shutdown_connections 2024-09-15T23:39:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdf200a32f0 0x7fdf200a57b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf38103710 0x7fdf3819f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf38102d50 0x7fdf381982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 --2- 172.21.15.89:0/4103876186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf38101b60 0x7fdf38197db0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 -- 172.21.15.89:0/4103876186 >> 172.21.15.89:0/4103876186 conn(0x7fdf380fd2e0 msgr2=0x7fdf38106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 -- 172.21.15.89:0/4103876186 shutdown_connections 2024-09-15T23:39:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:34.613+0000 7fdf3d40e640 1 -- 172.21.15.89:0/4103876186 wait complete. 2024-09-15T23:39:35.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:35 smithi120 ceph-mon[35250]: pgmap v816: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-09-15T23:39:35.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:35 smithi174 ceph-mon[35586]: pgmap v816: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-09-15T23:39:35.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:35 smithi089 ceph-mon[30933]: pgmap v816: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 19 op/s 2024-09-15T23:39:37.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:37 smithi120 ceph-mon[35250]: pgmap v817: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:39:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:37 smithi174 ceph-mon[35586]: pgmap v817: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:39:37.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:37 smithi089 ceph-mon[30933]: pgmap v817: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 14 op/s 2024-09-15T23:39:39.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:39 smithi120 ceph-mon[35250]: pgmap v818: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:39:39.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:39 smithi174 ceph-mon[35586]: pgmap v818: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:39:39.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:39 smithi089 ceph-mon[30933]: pgmap v818: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 18 op/s 2024-09-15T23:39:41.021 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:39:41.285 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:41.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:41 smithi089 ceph-mon[30933]: pgmap v819: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 14 op/s 2024-09-15T23:39:41.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:41 smithi120 ceph-mon[35250]: pgmap v819: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 14 op/s 2024-09-15T23:39:41.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:41 smithi174 ceph-mon[35586]: pgmap v819: 129 pgs: 129 active+clean; 3.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 14 op/s 2024-09-15T23:39:41.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.784+0000 7fe1e5597640 1 -- 172.21.15.89:0/1663221864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e01036e0 msgr2=0x7fe1e0107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:41.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.784+0000 7fe1e5597640 1 --2- 172.21.15.89:0/1663221864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e01036e0 0x7fe1e0107b90 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009930 tx=0x7fe1c802a420 comp rx=0 tx=0).stop 2024-09-15T23:39:41.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.785+0000 7fe1e5597640 1 -- 172.21.15.89:0/1663221864 shutdown_connections 2024-09-15T23:39:41.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.785+0000 7fe1e5597640 1 --2- 172.21.15.89:0/1663221864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1e01082a0 0x7fe1e010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.785+0000 7fe1e5597640 1 --2- 172.21.15.89:0/1663221864 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e01036e0 0x7fe1e0107b90 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.785+0000 7fe1e5597640 1 --2- 172.21.15.89:0/1663221864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1e0102d30 0x7fe1e0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.785+0000 7fe1e5597640 1 -- 172.21.15.89:0/1663221864 >> 172.21.15.89:0/1663221864 conn(0x7fe1e00fc7d0 msgr2=0x7fe1e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:41.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.785+0000 7fe1e5597640 1 -- 172.21.15.89:0/1663221864 shutdown_connections 2024-09-15T23:39:41.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.785+0000 7fe1e5597640 1 -- 172.21.15.89:0/1663221864 wait complete. 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.785+0000 7fe1e5597640 1 Processor -- start 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.785+0000 7fe1e5597640 1 -- start start 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1e5597640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1e0102d30 0x7fe1e01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1e5597640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e01036e0 0x7fe1e0198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1e5597640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1e01082a0 0x7fe1e019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1e5597640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e01120c0 con 0x7fe1e01036e0 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1e5597640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e0111f40 con 0x7fe1e0102d30 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1e5597640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e0112240 con 0x7fe1e01082a0 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e01036e0 0x7fe1e0198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e01036e0 0x7fe1e0198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54742/0 (socket says 172.21.15.89:54742) 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1d67fc640 1 -- 172.21.15.89:0/301305886 learned_addr learned my addr 172.21.15.89:0/301305886 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1deffd640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1e0102d30 0x7fe1e01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1df7fe640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1e01082a0 0x7fe1e019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1d67fc640 1 -- 172.21.15.89:0/301305886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1e01082a0 msgr2=0x7fe1e019ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1e01082a0 0x7fe1e019ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1d67fc640 1 -- 172.21.15.89:0/301305886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1e0102d30 msgr2=0x7fe1e01985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1e0102d30 0x7fe1e01985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1d67fc640 1 -- 172.21.15.89:0/301305886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e019d570 con 0x7fe1e01036e0 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1deffd640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1e0102d30 0x7fe1e01985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1df7fe640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1e01082a0 0x7fe1e019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e01036e0 0x7fe1e0198b00 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8005940 tx=0x7fe1c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1dcff9640 1 -- 172.21.15.89:0/301305886 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c803f030 con 0x7fe1e01036e0 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1e5597640 1 -- 172.21.15.89:0/301305886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e019d800 con 0x7fe1e01036e0 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1dcff9640 1 -- 172.21.15.89:0/301305886 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c8003ce0 con 0x7fe1e01036e0 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1dcff9640 1 -- 172.21.15.89:0/301305886 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8035070 con 0x7fe1e01036e0 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.786+0000 7fe1e5597640 1 -- 172.21.15.89:0/301305886 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e01a5140 con 0x7fe1e01036e0 2024-09-15T23:39:41.806 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.789+0000 7fe1dcff9640 1 -- 172.21.15.89:0/301305886 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe1c8042030 con 0x7fe1e01036e0 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.789+0000 7fe1e7970740 1 Processor -- start 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.789+0000 7fe1e7970740 1 -- start start 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.789+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a1f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.789+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a1fdc0 0x560242a2a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.789+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a2a680 0x560242a2ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1e7970740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5602429852e0 con 0x560242a2a680 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1e7970740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56024292cce0 con 0x560242a1fdc0 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1e7970740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5602429812a0 con 0x560242a1f4a0 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1dcff9640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1cc0a3240 0x7fe1cc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a2a680 0x560242a2ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a2a680 0x560242a2ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54752/0 (socket says 172.21.15.89:54752) 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1892573820 learned_addr learned my addr 172.21.15.89:0/1892573820 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1df7fe640 1 --2- 172.21.15.89:0/1892573820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a1f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.807 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1deffd640 1 --2- 172.21.15.89:0/1892573820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a1fdc0 0x560242a2a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1dcff9640 1 -- 172.21.15.89:0/301305886 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe1c80f0300 con 0x7fe1e01036e0 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1df7fe640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1cc0a3240 0x7fe1cc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1892573820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 msgr2=0x560242a1f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/1892573820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a1f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1892573820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a1fdc0 msgr2=0x560242a2a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/1892573820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a1fdc0 0x560242a2a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.790+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1892573820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560242911440 con 0x560242a2a680 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.791+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/1892573820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a2a680 0x560242a2ca70 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8004620 tx=0x7fe1c8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ba87e6e577ccec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.791+0000 7fe1df7fe640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1cc0a3240 0x7fe1cc0a5700 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009fd0 tx=0x7fe1d8008220 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.791+0000 7fe1d57fa640 1 -- 172.21.15.89:0/1892573820 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8049070 con 0x560242a2a680 2024-09-15T23:39:41.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.791+0000 7fe1d57fa640 1 -- 172.21.15.89:0/1892573820 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c80bcc30 con 0x560242a2a680 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.791+0000 7fe1d57fa640 1 -- 172.21.15.89:0/1892573820 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c80dc040 con 0x560242a2a680 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.791+0000 7fe1e7970740 1 -- 172.21.15.89:0/1892573820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a2a680 msgr2=0x560242a2ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.791+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1892573820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a2a680 0x560242a2ca70 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8004620 tx=0x7fe1c8000bc0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 -- 172.21.15.89:0/1892573820 shutdown_connections 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1892573820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a2a680 0x560242a2ca70 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1892573820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a1fdc0 0x560242a2a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1892573820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a1f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 -- 172.21.15.89:0/1892573820 >> 172.21.15.89:0/1892573820 conn(0x560242a1ba30 msgr2=0x560242a1be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 -- 172.21.15.89:0/1892573820 shutdown_connections 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 -- 172.21.15.89:0/1892573820 wait complete. 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 -- start start 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 Processor -- start 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a1f4a0 0x560242a29f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a271f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a2a680 0x560242a27730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560242a2d010 con 0x560242a1fdc0 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560242940050 con 0x560242a1f4a0 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560242a2d190 con 0x560242a2a680 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1e7970740 10 client.0 _ll_register_callbacks cb 0x560242980160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a271f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a271f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54758/0 (socket says 172.21.15.89:54758) 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.792+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1037087723 learned_addr learned my addr 172.21.15.89:0/1037087723 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1deffd640 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a1f4a0 0x560242a29f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1df7fe640 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a2a680 0x560242a27730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1037087723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a2a680 msgr2=0x560242a27730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a2a680 0x560242a27730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1037087723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a1f4a0 msgr2=0x560242a29f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:41.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a1f4a0 0x560242a29f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1037087723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602426f63c0 con 0x560242a1fdc0 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a271f0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80c4c70 tx=0x7fe1c802a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54692f62b9a8b8cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1c67fc640 1 -- 172.21.15.89:0/1037087723 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c80dc070 con 0x560242a1fdc0 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1c67fc640 1 -- 172.21.15.89:0/1037087723 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe1c803bcb0 con 0x560242a1fdc0 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1e7970740 1 -- 172.21.15.89:0/1037087723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1c80bcc30 con 0x560242a1fdc0 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1c67fc640 1 -- 172.21.15.89:0/1037087723 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe1c8049040 con 0x560242a1fdc0 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1e7970740 10 client.20778 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1e7970740 1 -- 172.21.15.89:0/1037087723 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560242a2d010 con 0x560242a1fdc0 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1c67fc640 1 -- 172.21.15.89:0/1037087723 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c80bcc30 con 0x560242a1fdc0 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1c67fc640 1 -- 172.21.15.89:0/1037087723 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe1c80c4d10 con 0x560242a1fdc0 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1e7970740 10 client.20778 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.793+0000 7fe1e7970740 10 client.20778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:41.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.794+0000 7fe1e7970740 20 client.20778 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.794+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560242a39bb0 0x560242a3bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.794+0000 7fe1e7970740 4 client.20778 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.794+0000 7fe1e7970740 1 -- 172.21.15.89:0/1037087723 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560242992850 con 0x560242a39bb0 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.794+0000 7fe1deffd640 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560242a39bb0 0x560242a3bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.794+0000 7fe1c67fc640 10 client.20778 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.794+0000 7fe1deffd640 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560242a39bb0 0x560242a3bff0 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=604b574f637654da server_cookie=6f967e7d443c837c in_seq=0 out_seq=0 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1c67fc640 1 -- 172.21.15.89:0/1037087723 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560242992850 con 0x560242a39bb0 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1c67fc640 10 client.20778 handle_command_reply: tid=0 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 client.20778 shutdown 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 -- 172.21.15.89:0/1037087723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 msgr2=0x560242a271f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a271f0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80c4c70 tx=0x7fe1c802a420 comp rx=0 tx=0).stop 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 -- 172.21.15.89:0/1037087723 shutdown_connections 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560242a39bb0 0x560242a3bff0 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.812 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a2a680 0x560242a27730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1c67fc640 0 client.20778 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:41.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a271f0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1037087723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a1f4a0 0x560242a29f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 -- 172.21.15.89:0/1037087723 >> 172.21.15.89:0/1037087723 conn(0x560242a1ba30 msgr2=0x560242a21710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:41.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 -- 172.21.15.89:0/1037087723 shutdown_connections 2024-09-15T23:39:41.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 1 -- 172.21.15.89:0/1037087723 wait complete. 2024-09-15T23:39:41.813 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.795+0000 7fe1e7970740 20 client.20778 trim_cache size 0 max 16384 2024-09-15T23:39:41.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.813+0000 7fe1e7970740 1 Processor -- start 2024-09-15T23:39:41.814 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.813+0000 7fe1e7970740 1 -- start start 2024-09-15T23:39:41.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.813+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a361e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.813+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a21600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.813+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 0x560242a418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.813+0000 7fe1e7970740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e01aef20 con 0x560242a1fdc0 2024-09-15T23:39:41.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.813+0000 7fe1e7970740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e01b0a60 con 0x560242a2a680 2024-09-15T23:39:41.819 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.813+0000 7fe1e7970740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e01af5f0 con 0x560242a1f4a0 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a21600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a21600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54768/0 (socket says 172.21.15.89:54768) 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1081158935 learned_addr learned my addr 172.21.15.89:0/1081158935 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1df7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 0x560242a418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1deffd640 1 --2- 172.21.15.89:0/1081158935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1081158935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 msgr2=0x560242a361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/1081158935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a361e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1081158935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 msgr2=0x560242a418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/1081158935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 0x560242a418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d67fc640 1 -- 172.21.15.89:0/1081158935 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560242a1fbf0 con 0x560242a1fdc0 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1df7fe640 1 --2- 172.21.15.89:0/1081158935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 0x560242a418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/1081158935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a21600 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8006010 tx=0x7fe1c8043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf629075c26cf59d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:41.820 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d4ff9640 1 -- 172.21.15.89:0/1081158935 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c80dc040 con 0x560242a1fdc0 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d4ff9640 1 -- 172.21.15.89:0/1081158935 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1c80ba8a0 con 0x560242a1fdc0 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.814+0000 7fe1d4ff9640 1 -- 172.21.15.89:0/1081158935 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c80bcc30 con 0x560242a1fdc0 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 -- 172.21.15.89:0/1081158935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 msgr2=0x560242a21600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1081158935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a21600 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8006010 tx=0x7fe1c8043000 comp rx=0 tx=0).stop 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 -- 172.21.15.89:0/1081158935 shutdown_connections 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1081158935 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 0x560242a418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1081158935 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a21600 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 --2- 172.21.15.89:0/1081158935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 -- 172.21.15.89:0/1081158935 >> 172.21.15.89:0/1081158935 conn(0x560242a1ba30 msgr2=0x560242a20ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 -- 172.21.15.89:0/1081158935 shutdown_connections 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 -- 172.21.15.89:0/1081158935 wait complete. 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 -- start start 2024-09-15T23:39:41.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.815+0000 7fe1e7970740 1 Processor -- start 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a322e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a32820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1e7970740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 0x560242a2f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1e7970740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e01b0390 con 0x560242a1fdc0 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1e7970740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e01afcc0 con 0x560242a2a680 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1e7970740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e01acf00 con 0x560242a1f4a0 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1e7970740 10 client.0 _ll_register_callbacks cb 0x7fe1c80a9030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a32820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1d67fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a32820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54782/0 (socket says 172.21.15.89:54782) 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1d67fc640 1 -- 172.21.15.89:0/4109217137 learned_addr learned my addr 172.21.15.89:0/4109217137 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1deffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1df7fe640 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 0x560242a2f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1d67fc640 1 -- 172.21.15.89:0/4109217137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 msgr2=0x560242a322e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:41.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a322e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1d67fc640 1 -- 172.21.15.89:0/4109217137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 msgr2=0x560242a2f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 0x560242a2f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1d67fc640 1 -- 172.21.15.89:0/4109217137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560242a32e60 con 0x560242a1fdc0 2024-09-15T23:39:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1d67fc640 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a32820 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80bc4c0 tx=0x7fe1c803b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ee51614a8ad005 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1c67fc640 1 -- 172.21.15.89:0/4109217137 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1c80dc040 con 0x560242a1fdc0 2024-09-15T23:39:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1c67fc640 1 -- 172.21.15.89:0/4109217137 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe1c80bb980 con 0x560242a1fdc0 2024-09-15T23:39:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.816+0000 7fe1e7970740 1 -- 172.21.15.89:0/4109217137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56024298cd60 con 0x560242a1fdc0 2024-09-15T23:39:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1c67fc640 1 -- 172.21.15.89:0/4109217137 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe1c80bc300 con 0x560242a1fdc0 2024-09-15T23:39:41.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1e7970740 10 client.20784 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1e7970740 1 -- 172.21.15.89:0/4109217137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560242a32e60 con 0x560242a1fdc0 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1c67fc640 1 -- 172.21.15.89:0/4109217137 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56024298cd60 con 0x560242a1fdc0 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1c67fc640 1 -- 172.21.15.89:0/4109217137 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fe1c8031af0 con 0x560242a1fdc0 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1e7970740 10 client.20784 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1e7970740 10 client.20784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1e7970740 20 client.20784 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1e7970740 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560242aa50c0 0x560242aa74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1e7970740 4 client.20784 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1e7970740 1 -- 172.21.15.89:0/4109217137 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560242909210 con 0x560242aa50c0 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.817+0000 7fe1deffd640 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560242aa50c0 0x560242aa74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1c67fc640 10 client.20784 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:41.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1deffd640 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560242aa50c0 0x560242aa74e0 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e3641e19f6728fa server_cookie=f56039312b18aaa4 in_seq=0 out_seq=0 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1c67fc640 1 -- 172.21.15.89:0/4109217137 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560242909210 con 0x560242aa50c0 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1c67fc640 10 client.20784 handle_command_reply: tid=0 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 client.20784 shutdown 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 -- 172.21.15.89:0/4109217137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 msgr2=0x560242a32820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a32820 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80bc4c0 tx=0x7fe1c803b670 comp rx=0 tx=0).stop 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 -- 172.21.15.89:0/4109217137 shutdown_connections 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560242aa50c0 0x560242aa74e0 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560242a2a680 0x560242a2f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560242a1fdc0 0x560242a32820 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 --2- 172.21.15.89:0/4109217137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560242a1f4a0 0x560242a322e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 -- 172.21.15.89:0/4109217137 >> 172.21.15.89:0/4109217137 conn(0x560242a1ba30 msgr2=0x560242a29270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1c67fc640 0 client.20784 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:41.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 -- 172.21.15.89:0/4109217137 shutdown_connections 2024-09-15T23:39:41.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 1 -- 172.21.15.89:0/4109217137 wait complete. 2024-09-15T23:39:41.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.818+0000 7fe1e7970740 20 client.20784 trim_cache size 0 max 16384 2024-09-15T23:39:41.826 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:39:41.826 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:39:41.826 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:39:41.826 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:39:41.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 -- 172.21.15.89:0/301305886 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1cc0a3240 msgr2=0x7fe1cc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:41.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1cc0a3240 0x7fe1cc0a5700 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009fd0 tx=0x7fe1d8008220 comp rx=0 tx=0).stop 2024-09-15T23:39:41.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 -- 172.21.15.89:0/301305886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e01036e0 msgr2=0x7fe1e0198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:41.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e01036e0 0x7fe1e0198b00 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8005940 tx=0x7fe1c80042f0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 -- 172.21.15.89:0/301305886 shutdown_connections 2024-09-15T23:39:41.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe1cc0a3240 0x7fe1cc0a5700 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1e01082a0 0x7fe1e019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e01036e0 0x7fe1e0198b00 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 --2- 172.21.15.89:0/301305886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1e0102d30 0x7fe1e01985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:41.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 -- 172.21.15.89:0/301305886 >> 172.21.15.89:0/301305886 conn(0x7fe1e00fc7d0 msgr2=0x7fe1e0106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:41.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 -- 172.21.15.89:0/301305886 shutdown_connections 2024-09-15T23:39:41.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:41.819+0000 7fe1e5597640 1 -- 172.21.15.89:0/301305886 wait complete. 2024-09-15T23:39:42.644 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:39:42.644 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:39:42.890 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:43.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:43 smithi089 ceph-mon[30933]: pgmap v820: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 KiB/s wr, 16 op/s 2024-09-15T23:39:43.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.410+0000 7f7962ca2640 1 -- 172.21.15.89:0/62192341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f795c102ef0 msgr2=0x7f795c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:43.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.410+0000 7f7962ca2640 1 --2- 172.21.15.89:0/62192341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f795c102ef0 0x7f795c10da40 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f7954009ca0 tx=0x7f795402a950 comp rx=0 tx=0).stop 2024-09-15T23:39:43.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.410+0000 7f7962ca2640 1 -- 172.21.15.89:0/62192341 shutdown_connections 2024-09-15T23:39:43.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.410+0000 7f7962ca2640 1 --2- 172.21.15.89:0/62192341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f795c102ef0 0x7f795c10da40 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:43.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.410+0000 7f7962ca2640 1 --2- 172.21.15.89:0/62192341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f795c102570 0x7f795c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:43.412 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.410+0000 7f7962ca2640 1 --2- 172.21.15.89:0/62192341 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f795c100e20 0x7f795c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:43.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.410+0000 7f7962ca2640 1 -- 172.21.15.89:0/62192341 >> 172.21.15.89:0/62192341 conn(0x7f795c0fc7d0 msgr2=0x7f795c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:43.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.410+0000 7f7962ca2640 1 -- 172.21.15.89:0/62192341 shutdown_connections 2024-09-15T23:39:43.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.410+0000 7f7962ca2640 1 -- 172.21.15.89:0/62192341 wait complete. 2024-09-15T23:39:43.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f7962ca2640 1 Processor -- start 2024-09-15T23:39:43.413 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f7962ca2640 1 -- start start 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f7962ca2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f795c100e20 0x7f795c1992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f7962ca2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f795c102570 0x7f795c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f7962ca2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f795c102ef0 0x7f795c1a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f7962ca2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f795c110f80 con 0x7f795c102570 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f7962ca2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f795c110e00 con 0x7f795c100e20 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f7962ca2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f795c111100 con 0x7f795c102ef0 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f795bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f795c102570 0x7f795c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f795bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f795c102570 0x7f795c1997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54800/0 (socket says 172.21.15.89:54800) 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.411+0000 7f795bfff640 1 -- 172.21.15.89:0/838428952 learned_addr learned my addr 172.21.15.89:0/838428952 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f7960a3e640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f795c100e20 0x7f795c1992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f796123f640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f795c102ef0 0x7f795c1a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:43.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f795bfff640 1 -- 172.21.15.89:0/838428952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f795c102ef0 msgr2=0x7f795c1a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f795bfff640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f795c102ef0 0x7f795c1a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f795bfff640 1 -- 172.21.15.89:0/838428952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f795c100e20 msgr2=0x7f795c1992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f795bfff640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f795c100e20 0x7f795c1992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f795bfff640 1 -- 172.21.15.89:0/838428952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795c1a4870 con 0x7f795c102570 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f7960a3e640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f795c100e20 0x7f795c1992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f795bfff640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f795c102570 0x7f795c1997f0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f794800d950 tx=0x7f794800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f7959ffb640 1 -- 172.21.15.89:0/838428952 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7948017070 con 0x7f795c102570 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f7962ca2640 1 -- 172.21.15.89:0/838428952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f795c1a4a40 con 0x7f795c102570 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f7959ffb640 1 -- 172.21.15.89:0/838428952 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7948009c30 con 0x7f795c102570 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f7959ffb640 1 -- 172.21.15.89:0/838428952 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794800f6b0 con 0x7f795c102570 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.412+0000 7f7962ca2640 1 -- 172.21.15.89:0/838428952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f795c1a4f80 con 0x7f795c102570 2024-09-15T23:39:43.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.413+0000 7f7962ca2640 1 -- 172.21.15.89:0/838428952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f795c101200 con 0x7f795c102570 2024-09-15T23:39:43.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.414+0000 7f7959ffb640 1 -- 172.21.15.89:0/838428952 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f79480109d0 con 0x7f795c102570 2024-09-15T23:39:43.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.415+0000 7f7959ffb640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f794c0a32c0 0x7f794c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:43.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.415+0000 7f7959ffb640 1 -- 172.21.15.89:0/838428952 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f79480cfc70 con 0x7f795c102570 2024-09-15T23:39:43.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.415+0000 7f7960a3e640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f794c0a32c0 0x7f794c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:43.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.416+0000 7f7960a3e640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f794c0a32c0 0x7f794c0a5780 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f79500058c0 tx=0x7f7950005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:43.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.422+0000 7f7959ffb640 1 -- 172.21.15.89:0/838428952 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7948009de0 con 0x7f795c102570 2024-09-15T23:39:43.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:43 smithi120 ceph-mon[35250]: pgmap v820: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 KiB/s wr, 16 op/s 2024-09-15T23:39:43.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:43 smithi174 ceph-mon[35586]: pgmap v820: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 KiB/s wr, 16 op/s 2024-09-15T23:39:43.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.634+0000 7f7962ca2640 1 -- 172.21.15.89:0/838428952 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f795c06a560 con 0x7f795c102570 2024-09-15T23:39:43.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.635+0000 7f7959ffb640 1 -- 172.21.15.89:0/838428952 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f795c06a560 con 0x7f795c102570 2024-09-15T23:39:43.636 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:39:43.637 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:39:43.637 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:39:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 -- 172.21.15.89:0/838428952 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f794c0a32c0 msgr2=0x7f794c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f794c0a32c0 0x7f794c0a5780 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f79500058c0 tx=0x7f7950005850 comp rx=0 tx=0).stop 2024-09-15T23:39:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 -- 172.21.15.89:0/838428952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f795c102570 msgr2=0x7f795c1997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f795c102570 0x7f795c1997f0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f794800d950 tx=0x7f794800de10 comp rx=0 tx=0).stop 2024-09-15T23:39:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 -- 172.21.15.89:0/838428952 shutdown_connections 2024-09-15T23:39:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f794c0a32c0 0x7f794c0a5780 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f795c102ef0 0x7f795c1a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f795c102570 0x7f795c1997f0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:43.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 --2- 172.21.15.89:0/838428952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f795c100e20 0x7f795c1992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:43.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 -- 172.21.15.89:0/838428952 >> 172.21.15.89:0/838428952 conn(0x7f795c0fc7d0 msgr2=0x7f795c106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:43.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 -- 172.21.15.89:0/838428952 shutdown_connections 2024-09-15T23:39:43.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:43.637+0000 7f7962ca2640 1 -- 172.21.15.89:0/838428952 wait complete. 2024-09-15T23:39:44.406 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:39:44.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:44 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/838428952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:44.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:44 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/838428952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:44.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:44 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/838428952' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:44.662 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:45.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.132+0000 7f4ec0220640 1 -- 172.21.15.89:0/1889652755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eb8102790 msgr2=0x7f4eb81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:45.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.132+0000 7f4ec0220640 1 --2- 172.21.15.89:0/1889652755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eb8102790 0x7f4eb81067e0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009900 tx=0x7f4eac02a410 comp rx=0 tx=0).stop 2024-09-15T23:39:45.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.132+0000 7f4ec0220640 1 -- 172.21.15.89:0/1889652755 shutdown_connections 2024-09-15T23:39:45.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.132+0000 7f4ec0220640 1 --2- 172.21.15.89:0/1889652755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8106f10 0x7f4eb810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.132+0000 7f4ec0220640 1 --2- 172.21.15.89:0/1889652755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eb8102790 0x7f4eb81067e0 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.132+0000 7f4ec0220640 1 --2- 172.21.15.89:0/1889652755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4eb8101de0 0x7f4eb81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.132+0000 7f4ec0220640 1 -- 172.21.15.89:0/1889652755 >> 172.21.15.89:0/1889652755 conn(0x7f4eb80fd8c0 msgr2=0x7f4eb80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:45.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.132+0000 7f4ec0220640 1 -- 172.21.15.89:0/1889652755 shutdown_connections 2024-09-15T23:39:45.152 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.132+0000 7f4ec0220640 1 -- 172.21.15.89:0/1889652755 wait complete. 2024-09-15T23:39:45.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ec0220640 1 Processor -- start 2024-09-15T23:39:45.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ec0220640 1 -- start start 2024-09-15T23:39:45.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ec0220640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4eb8101de0 0x7f4eb81984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ec0220640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8102790 0x7f4eb81989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ec0220640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eb8106f10 0x7f4eb819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ec0220640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb8111fb0 con 0x7f4eb8106f10 2024-09-15T23:39:45.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ec0220640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb8111e30 con 0x7f4eb8102790 2024-09-15T23:39:45.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ec0220640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb8112130 con 0x7f4eb8101de0 2024-09-15T23:39:45.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebe7bd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eb8106f10 0x7f4eb819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebe7bd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eb8106f10 0x7f4eb819cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54820/0 (socket says 172.21.15.89:54820) 2024-09-15T23:39:45.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebe7bd640 1 -- 172.21.15.89:0/613768989 learned_addr learned my addr 172.21.15.89:0/613768989 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:45.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebdfbc640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4eb8101de0 0x7f4eb81984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8102790 0x7f4eb81989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebe7bd640 1 -- 172.21.15.89:0/613768989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4eb8101de0 msgr2=0x7f4eb81984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:45.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebe7bd640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4eb8101de0 0x7f4eb81984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebe7bd640 1 -- 172.21.15.89:0/613768989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8102790 msgr2=0x7f4eb81989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:45.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebe7bd640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8102790 0x7f4eb81989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebe7bd640 1 -- 172.21.15.89:0/613768989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb819d460 con 0x7f4eb8106f10 2024-09-15T23:39:45.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebdfbc640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4eb8101de0 0x7f4eb81984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:45.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.133+0000 7f4ebe7bd640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eb8106f10 0x7f4eb819cd80 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000b970 tx=0x7f4eb000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:45.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.134+0000 7f4eaaffd640 1 -- 172.21.15.89:0/613768989 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0019070 con 0x7f4eb8106f10 2024-09-15T23:39:45.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.134+0000 7f4ec0220640 1 -- 172.21.15.89:0/613768989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eb819d750 con 0x7f4eb8106f10 2024-09-15T23:39:45.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.134+0000 7f4eaaffd640 1 -- 172.21.15.89:0/613768989 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb000cb40 con 0x7f4eb8106f10 2024-09-15T23:39:45.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.134+0000 7f4eaaffd640 1 -- 172.21.15.89:0/613768989 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0007650 con 0x7f4eb8106f10 2024-09-15T23:39:45.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.134+0000 7f4ec0220640 1 -- 172.21.15.89:0/613768989 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eb81a5030 con 0x7f4eb8106f10 2024-09-15T23:39:45.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.136+0000 7f4eaaffd640 1 -- 172.21.15.89:0/613768989 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4eb001c070 con 0x7f4eb8106f10 2024-09-15T23:39:45.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.136+0000 7f4ec25f9740 1 Processor -- start 2024-09-15T23:39:45.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.136+0000 7f4ec25f9740 1 -- start start 2024-09-15T23:39:45.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96470090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f964705d0 0x558f96470a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f9647d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558f963d62f0 con 0x558f9647b370 2024-09-15T23:39:45.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558f9637dcf0 con 0x558f964705d0 2024-09-15T23:39:45.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558f963d22b0 con 0x558f96471b40 2024-09-15T23:39:45.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebd7bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f9647d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebd7bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f9647d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54828/0 (socket says 172.21.15.89:54828) 2024-09-15T23:39:45.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/4156041589 learned_addr learned my addr 172.21.15.89:0/4156041589 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:45.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/4156041589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96470090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebdfbc640 1 --2- 172.21.15.89:0/4156041589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f964705d0 0x558f96470a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4eaaffd640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ea40a3240 0x7f4ea40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.160 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/4156041589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 msgr2=0x558f96470090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:45.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/4156041589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96470090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/4156041589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f964705d0 msgr2=0x558f96470a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:45.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/4156041589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f964705d0 0x558f96470a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/4156041589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f963623a0 con 0x558f9647b370 2024-09-15T23:39:45.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebe7bd640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ea40a3240 0x7f4ea40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.161 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4eaaffd640 1 -- 172.21.15.89:0/613768989 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4eb00cf370 con 0x7f4eb8106f10 2024-09-15T23:39:45.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.137+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/4156041589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f9647d760 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009900 tx=0x7f4eac0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a473d500d34492 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:45.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ea3fff640 1 -- 172.21.15.89:0/4156041589 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac035040 con 0x558f9647b370 2024-09-15T23:39:45.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ea3fff640 1 -- 172.21.15.89:0/4156041589 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eac0035f0 con 0x558f9647b370 2024-09-15T23:39:45.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ea3fff640 1 -- 172.21.15.89:0/4156041589 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac03d6e0 con 0x558f9647b370 2024-09-15T23:39:45.162 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ec25f9740 1 -- 172.21.15.89:0/4156041589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 msgr2=0x558f9647d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:45.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ebe7bd640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ea40a3240 0x7f4ea40a5700 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4eb009b2a0 tx=0x7f4eb0066480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:45.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/4156041589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f9647d760 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009900 tx=0x7f4eac0056d0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ec25f9740 1 -- 172.21.15.89:0/4156041589 shutdown_connections 2024-09-15T23:39:45.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/4156041589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f9647d760 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.163 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/4156041589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f964705d0 0x558f96470a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/4156041589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96470090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.138+0000 7f4ec25f9740 1 -- 172.21.15.89:0/4156041589 >> 172.21.15.89:0/4156041589 conn(0x558f96439310 msgr2=0x558f9646cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:45.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 1 -- 172.21.15.89:0/4156041589 shutdown_connections 2024-09-15T23:39:45.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 1 -- 172.21.15.89:0/4156041589 wait complete. 2024-09-15T23:39:45.164 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 1 -- start start 2024-09-15T23:39:45.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 1 Processor -- start 2024-09-15T23:39:45.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f964705d0 0x558f9647adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f96471b40 0x558f96478070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f9647b370 0x558f96478800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.165 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x558f9647e010 con 0x558f96471b40 2024-09-15T23:39:45.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558f96391060 con 0x558f9647b370 2024-09-15T23:39:45.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x558f9647e190 con 0x558f964705d0 2024-09-15T23:39:45.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ec25f9740 10 client.0 _ll_register_callbacks cb 0x558f960e48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:45.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ebdfbc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f96471b40 0x558f96478070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.166 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ebdfbc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f96471b40 0x558f96478070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:54840/0 (socket says 172.21.15.89:54840) 2024-09-15T23:39:45.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.139+0000 7f4ebdfbc640 1 -- 172.21.15.89:0/1911006136 learned_addr learned my addr 172.21.15.89:0/1911006136 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:45.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f964705d0 0x558f9647adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f9647b370 0x558f96478800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.167 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ebdfbc640 1 -- 172.21.15.89:0/1911006136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f964705d0 msgr2=0x558f9647adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:45.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ebdfbc640 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f964705d0 0x558f9647adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ebdfbc640 1 -- 172.21.15.89:0/1911006136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f9647b370 msgr2=0x558f96478800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:45.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ebdfbc640 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f9647b370 0x558f96478800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ebdfbc640 1 -- 172.21.15.89:0/1911006136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f961473c0 con 0x558f96471b40 2024-09-15T23:39:45.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f964705d0 0x558f9647adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:45.168 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ebdfbc640 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f96471b40 0x558f96478070 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4012480 tx=0x7f4eb4012940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7f35ea8b5f0054 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:45.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ea17fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:45.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ea17fa640 1 -- 172.21.15.89:0/1911006136 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4005220 con 0x558f96471b40 2024-09-15T23:39:45.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ea17fa640 1 -- 172.21.15.89:0/1911006136 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4eb40053c0 con 0x558f96471b40 2024-09-15T23:39:45.169 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ec25f9740 1 -- 172.21.15.89:0/1911006136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eac0035f0 con 0x558f96471b40 2024-09-15T23:39:45.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ea17fa640 1 -- 172.21.15.89:0/1911006136 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4eb4025050 con 0x558f96471b40 2024-09-15T23:39:45.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ec25f9740 10 client.20802 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:45.170 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ec25f9740 1 -- 172.21.15.89:0/1911006136 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558f96489b80 con 0x558f96471b40 2024-09-15T23:39:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.140+0000 7f4ea17fa640 1 -- 172.21.15.89:0/1911006136 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eac0035f0 con 0x558f96471b40 2024-09-15T23:39:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.141+0000 7f4ea17fa640 1 -- 172.21.15.89:0/1911006136 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4eb4019030 con 0x558f96471b40 2024-09-15T23:39:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.141+0000 7f4ec25f9740 10 client.20802 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.141+0000 7f4ec25f9740 10 client.20802 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:39:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.141+0000 7f4ec25f9740 20 client.20802 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.141+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558f9648b0e0 0x558f9648d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.141+0000 7f4ec25f9740 4 client.20802 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.141+0000 7f4ec25f9740 1 -- 172.21.15.89:0/1911006136 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f963e3860 con 0x558f9648b0e0 2024-09-15T23:39:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.141+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558f9648b0e0 0x558f9648d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.171 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.141+0000 7f4ea17fa640 10 client.20802 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.141+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558f9648b0e0 0x558f9648d500 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8a049883bf61b9a0 server_cookie=4b53100cc73a5cf9 in_seq=0 out_seq=0 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ea17fa640 1 -- 172.21.15.89:0/1911006136 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558f963e3860 con 0x558f9648b0e0 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ea17fa640 10 client.20802 handle_command_reply: tid=0 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 client.20802 shutdown 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 -- 172.21.15.89:0/1911006136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f96471b40 msgr2=0x558f96478070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f96471b40 0x558f96478070 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4012480 tx=0x7f4eb4012940 comp rx=0 tx=0).stop 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 -- 172.21.15.89:0/1911006136 shutdown_connections 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558f9648b0e0 0x558f9648d500 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f9647b370 0x558f96478800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f96471b40 0x558f96478070 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ea17fa640 0 client.20802 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/1911006136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f964705d0 0x558f9647adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 -- 172.21.15.89:0/1911006136 >> 172.21.15.89:0/1911006136 conn(0x558f96439310 msgr2=0x558f96472960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:45.172 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 -- 172.21.15.89:0/1911006136 shutdown_connections 2024-09-15T23:39:45.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 1 -- 172.21.15.89:0/1911006136 wait complete. 2024-09-15T23:39:45.173 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.142+0000 7f4ec25f9740 20 client.20802 trim_cache size 0 max 16384 2024-09-15T23:39:45.178 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.155+0000 7f4ec25f9740 1 Processor -- start 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.155+0000 7f4ec25f9740 1 -- start start 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.155+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f964705d0 0x558f96471300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.155+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96492d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.155+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f964932a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.155+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81aee30 con 0x558f9647b370 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.155+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81b0970 con 0x558f964705d0 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.155+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81af500 con 0x558f96471b40 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebd7bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f964932a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebd7bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f964932a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55538/0 (socket says 172.21.15.89:55538) 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/3859687278 learned_addr learned my addr 172.21.15.89:0/3859687278 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/3859687278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f964705d0 0x558f96471300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebdfbc640 1 --2- 172.21.15.89:0/3859687278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96492d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/3859687278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 msgr2=0x558f96492d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/3859687278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96492d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/3859687278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f964705d0 msgr2=0x558f96471300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/3859687278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f964705d0 0x558f96471300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/3859687278 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f9648db30 con 0x558f9647b370 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/3859687278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f964932a0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f4eac033000 tx=0x7f4eac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c261226664b17b5b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ea37fe640 1 -- 172.21.15.89:0/3859687278 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac035040 con 0x558f9647b370 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ea37fe640 1 -- 172.21.15.89:0/3859687278 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eac059070 con 0x558f9647b370 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.156+0000 7f4ea37fe640 1 -- 172.21.15.89:0/3859687278 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac003ef0 con 0x558f9647b370 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 -- 172.21.15.89:0/3859687278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 msgr2=0x558f964932a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/3859687278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f964932a0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f4eac033000 tx=0x7f4eac000bc0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 -- 172.21.15.89:0/3859687278 shutdown_connections 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/3859687278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9647b370 0x558f964932a0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/3859687278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96492d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/3859687278 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f964705d0 0x558f96471300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 -- 172.21.15.89:0/3859687278 >> 172.21.15.89:0/3859687278 conn(0x558f96439310 msgr2=0x558f96467a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 -- 172.21.15.89:0/3859687278 shutdown_connections 2024-09-15T23:39:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 -- 172.21.15.89:0/3859687278 wait complete. 2024-09-15T23:39:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 -- start start 2024-09-15T23:39:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.157+0000 7f4ec25f9740 1 Processor -- start 2024-09-15T23:39:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f964705d0 0x558f964837f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96483d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ec25f9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f9647b370 0x558f96474c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81b02a0 con 0x558f964705d0 2024-09-15T23:39:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81afbd0 con 0x558f9647b370 2024-09-15T23:39:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ec25f9740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81ace10 con 0x558f96471b40 2024-09-15T23:39:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ec25f9740 10 client.0 _ll_register_callbacks cb 0x7f4eb400b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebd7bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f964705d0 0x558f964837f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebd7bb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f964705d0 0x558f964837f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55548/0 (socket says 172.21.15.89:55548) 2024-09-15T23:39:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/425195148 learned_addr learned my addr 172.21.15.89:0/425195148 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebdfbc640 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96483d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f9647b370 0x558f96474c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/425195148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 msgr2=0x558f96483d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96483d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/425195148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f9647b370 msgr2=0x558f96474c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f9647b370 0x558f96474c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebd7bb640 1 -- 172.21.15.89:0/425195148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f9646c510 con 0x558f964705d0 2024-09-15T23:39:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ebd7bb640 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f964705d0 0x558f964837f0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f4eac02a930 tx=0x7f4eac0038f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4043a9e833821d94 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.158+0000 7f4ea17fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ea17fa640 1 -- 172.21.15.89:0/425195148 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac035040 con 0x558f964705d0 2024-09-15T23:39:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ea17fa640 1 -- 172.21.15.89:0/425195148 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4eac031b80 con 0x558f964705d0 2024-09-15T23:39:45.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ec25f9740 1 -- 172.21.15.89:0/425195148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eac059070 con 0x558f964705d0 2024-09-15T23:39:45.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ea17fa640 1 -- 172.21.15.89:0/425195148 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4eac065050 con 0x558f964705d0 2024-09-15T23:39:45.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ec25f9740 10 client.20808 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:45.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ec25f9740 1 -- 172.21.15.89:0/425195148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558f964f5740 con 0x558f964705d0 2024-09-15T23:39:45.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ea17fa640 1 -- 172.21.15.89:0/425195148 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eac059070 con 0x558f964705d0 2024-09-15T23:39:45.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ea17fa640 1 -- 172.21.15.89:0/425195148 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4eac03a040 con 0x558f964705d0 2024-09-15T23:39:45.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ec25f9740 10 client.20808 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:45.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ec25f9740 10 client.20808 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:39:45.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ec25f9740 20 client.20808 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:45.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558f964f59c0 0x558f964f7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ec25f9740 4 client.20808 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ec25f9740 1 -- 172.21.15.89:0/425195148 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f963ddd70 con 0x558f964f59c0 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.159+0000 7f4ebdfbc640 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558f964f59c0 0x558f964f7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ea17fa640 10 client.20808 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ebdfbc640 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558f964f59c0 0x558f964f7db0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dc929613c52f1086 server_cookie=4b7a3ccb5e730958 in_seq=0 out_seq=0 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ea17fa640 1 -- 172.21.15.89:0/425195148 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f963ddd70 con 0x558f964f59c0 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ea17fa640 10 client.20808 handle_command_reply: tid=0 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 client.20808 shutdown 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 -- 172.21.15.89:0/425195148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f964705d0 msgr2=0x558f964837f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f964705d0 0x558f964837f0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f4eac02a930 tx=0x7f4eac0038f0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 -- 172.21.15.89:0/425195148 shutdown_connections 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x558f964f59c0 0x558f964f7db0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558f9647b370 0x558f96474c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x558f96471b40 0x558f96483d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ea17fa640 0 client.20808 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:39:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 --2- 172.21.15.89:0/425195148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f964705d0 0x558f964837f0 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 -- 172.21.15.89:0/425195148 >> 172.21.15.89:0/425195148 conn(0x558f96439310 msgr2=0x558f96467a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 -- 172.21.15.89:0/425195148 shutdown_connections 2024-09-15T23:39:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 1 -- 172.21.15.89:0/425195148 wait complete. 2024-09-15T23:39:45.186 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.160+0000 7f4ec25f9740 20 client.20808 trim_cache size 0 max 16384 2024-09-15T23:39:45.187 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:39:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 -- 172.21.15.89:0/613768989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ea40a3240 msgr2=0x7f4ea40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ea40a3240 0x7f4ea40a5700 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4eb009b2a0 tx=0x7f4eb0066480 comp rx=0 tx=0).stop 2024-09-15T23:39:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 -- 172.21.15.89:0/613768989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eb8106f10 msgr2=0x7f4eb819cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:45.188 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eb8106f10 0x7f4eb819cd80 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000b970 tx=0x7f4eb000be30 comp rx=0 tx=0).stop 2024-09-15T23:39:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 -- 172.21.15.89:0/613768989 shutdown_connections 2024-09-15T23:39:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4ea40a3240 0x7f4ea40a5700 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4eb8106f10 0x7f4eb819cd80 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8102790 0x7f4eb81989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 --2- 172.21.15.89:0/613768989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4eb8101de0 0x7f4eb81984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:45.189 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 -- 172.21.15.89:0/613768989 >> 172.21.15.89:0/613768989 conn(0x7f4eb80fd8c0 msgr2=0x7f4eb80feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 -- 172.21.15.89:0/613768989 shutdown_connections 2024-09-15T23:39:45.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:45.161+0000 7f4ec0220640 1 -- 172.21.15.89:0/613768989 wait complete. 2024-09-15T23:39:45.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:45 smithi120 ceph-mon[35250]: pgmap v821: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 14 op/s 2024-09-15T23:39:45.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:45 smithi174 ceph-mon[35586]: pgmap v821: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 14 op/s 2024-09-15T23:39:45.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:45 smithi089 ceph-mon[30933]: pgmap v821: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 14 op/s 2024-09-15T23:39:45.984 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:39:46.233 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:46.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2024403595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4df410c180 msgr2=0x7f4df410e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:46.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 --2- 172.21.15.89:0/2024403595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4df410c180 0x7f4df410e600 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f4dec009ca0 tx=0x7f4dec02a950 comp rx=0 tx=0).stop 2024-09-15T23:39:46.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2024403595 shutdown_connections 2024-09-15T23:39:46.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 --2- 172.21.15.89:0/2024403595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4df410c180 0x7f4df410e600 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 --2- 172.21.15.89:0/2024403595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4df41036c0 0x7f4df410bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.736 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 --2- 172.21.15.89:0/2024403595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df4102da0 0x7f4df4103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2024403595 >> 172.21.15.89:0/2024403595 conn(0x7f4df40fc7a0 msgr2=0x7f4df40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2024403595 shutdown_connections 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2024403595 wait complete. 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 Processor -- start 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.716+0000 7f4dfa7f4640 1 -- start start 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4dfa7f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4df4102da0 0x7f4df4198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4dfa7f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df41036c0 0x7f4df4198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4dfa7f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4df410c180 0x7f4df419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4dfa7f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4df4111fa0 con 0x7f4df4102da0 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4dfa7f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4df4111e20 con 0x7f4df41036c0 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4dfa7f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4df4112120 con 0x7f4df410c180 2024-09-15T23:39:46.737 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4df4102da0 0x7f4df4198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4df4102da0 0x7f4df4198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55566/0 (socket says 172.21.15.89:55566) 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df3fff640 1 -- 172.21.15.89:0/2355156785 learned_addr learned my addr 172.21.15.89:0/2355156785 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df8d91640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4df410c180 0x7f4df419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4debfff640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df41036c0 0x7f4df4198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df3fff640 1 -- 172.21.15.89:0/2355156785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4df410c180 msgr2=0x7f4df419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df3fff640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4df410c180 0x7f4df419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df3fff640 1 -- 172.21.15.89:0/2355156785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df41036c0 msgr2=0x7f4df4198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df3fff640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df41036c0 0x7f4df4198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df3fff640 1 -- 172.21.15.89:0/2355156785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df419d3f0 con 0x7f4df4102da0 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4debfff640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df41036c0 0x7f4df4198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df3fff640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4df4102da0 0x7f4df4198440 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f4de400c890 tx=0x7f4de400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.717+0000 7f4df1ffb640 1 -- 172.21.15.89:0/2355156785 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4004220 con 0x7f4df4102da0 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.718+0000 7f4df1ffb640 1 -- 172.21.15.89:0/2355156785 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de40043c0 con 0x7f4df4102da0 2024-09-15T23:39:46.738 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.718+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2355156785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df419d6e0 con 0x7f4df4102da0 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.718+0000 7f4df1ffb640 1 -- 172.21.15.89:0/2355156785 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de40049e0 con 0x7f4df4102da0 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.718+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2355156785 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df41a4fc0 con 0x7f4df4102da0 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.720+0000 7f4df1ffb640 1 -- 172.21.15.89:0/2355156785 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4de40079b0 con 0x7f4df4102da0 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.720+0000 7f4dfcbcd740 1 Processor -- start 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.720+0000 7f4dfcbcd740 1 -- start start 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.720+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2cb40 0x5618afe2b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.720+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2b5d0 0x5618afe2ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.720+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe36370 0x5618afe38760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5618afd912f0 con 0x5618afe2b5d0 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618afd38cf0 con 0x5618afe2cb40 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5618afd8d2b0 con 0x5618afe36370 2024-09-15T23:39:46.739 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2b5d0 0x5618afe2ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2b5d0 0x5618afe2ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55578/0 (socket says 172.21.15.89:55578) 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df3fff640 1 -- 172.21.15.89:0/3026427895 learned_addr learned my addr 172.21.15.89:0/3026427895 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df1ffb640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4de00a32c0 0x7f4de00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4debfff640 1 --2- 172.21.15.89:0/3026427895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe36370 0x5618afe38760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df8d91640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4de00a32c0 0x7f4de00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df3fff640 1 -- 172.21.15.89:0/3026427895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe36370 msgr2=0x5618afe38760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df3fff640 1 --2- 172.21.15.89:0/3026427895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe36370 0x5618afe38760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df3fff640 1 -- 172.21.15.89:0/3026427895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2cb40 msgr2=0x5618afe2b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df8d91640 1 --2- 172.21.15.89:0/3026427895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2cb40 0x5618afe2b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df3fff640 1 --2- 172.21.15.89:0/3026427895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2cb40 0x5618afe2b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df1ffb640 1 -- 172.21.15.89:0/2355156785 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4de40cfd20 con 0x7f4df4102da0 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df3fff640 1 -- 172.21.15.89:0/3026427895 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618afd1d3a0 con 0x5618afe2b5d0 2024-09-15T23:39:46.740 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df8d91640 1 --2- 172.21.15.89:0/3026427895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2cb40 0x5618afe2b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df3fff640 1 --2- 172.21.15.89:0/3026427895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2b5d0 0x5618afe2ba10 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f4de40983b0 tx=0x7f4de409be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c611116efcde32be server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4dea7fc640 1 -- 172.21.15.89:0/3026427895 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de401d070 con 0x5618afe2b5d0 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4df8d91640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4de00a32c0 0x7f4de00a5780 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4dec030040 tx=0x7f4dec03e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4dea7fc640 1 -- 172.21.15.89:0/3026427895 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de409c650 con 0x5618afe2b5d0 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.721+0000 7f4dea7fc640 1 -- 172.21.15.89:0/3026427895 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de40a4bc0 con 0x5618afe2b5d0 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.722+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/3026427895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2b5d0 msgr2=0x5618afe2ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.722+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/3026427895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2b5d0 0x5618afe2ba10 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f4de40983b0 tx=0x7f4de409be50 comp rx=0 tx=0).stop 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.722+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/3026427895 shutdown_connections 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.722+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/3026427895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe36370 0x5618afe38760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.722+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/3026427895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2b5d0 0x5618afe2ba10 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.722+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/3026427895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2cb40 0x5618afe2b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.722+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/3026427895 >> 172.21.15.89:0/3026427895 conn(0x5618afdf4310 msgr2=0x5618afe27c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:46.741 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.722+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/3026427895 shutdown_connections 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.722+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/3026427895 wait complete. 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.722+0000 7f4dfcbcd740 1 -- start start 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4dfcbcd740 1 Processor -- start 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe2b5d0 0x5618afe35de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe33090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe36370 0x5618afe337d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5618afe3d420 con 0x5618afe2cb40 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618afd4c060 con 0x5618afe36370 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5618afe3d5a0 con 0x5618afe2b5d0 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4dfcbcd740 10 client.0 _ll_register_callbacks cb 0x5618afa9f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4debfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe33090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4debfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe33090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55584/0 (socket says 172.21.15.89:55584) 2024-09-15T23:39:46.742 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4debfff640 1 -- 172.21.15.89:0/2411702917 learned_addr learned my addr 172.21.15.89:0/2411702917 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4df8d91640 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe36370 0x5618afe337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4df3fff640 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe2b5d0 0x5618afe35de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4debfff640 1 -- 172.21.15.89:0/2411702917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe2b5d0 msgr2=0x5618afe35de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4debfff640 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe2b5d0 0x5618afe35de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4debfff640 1 -- 172.21.15.89:0/2411702917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe36370 msgr2=0x5618afe337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4debfff640 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe36370 0x5618afe337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.723+0000 7f4debfff640 1 -- 172.21.15.89:0/2411702917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618afb023c0 con 0x5618afe2cb40 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4debfff640 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe33090 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00f4a0 tx=0x7f4ddc00f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557fac850c314bbe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dd37fe640 1 -- 172.21.15.89:0/2411702917 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc013850 con 0x5618afe2cb40 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dd37fe640 1 -- 172.21.15.89:0/2411702917 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4ddc0139f0 con 0x5618afe2cb40 2024-09-15T23:39:46.743 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/2411702917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4de409c650 con 0x5618afe2cb40 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dd37fe640 1 -- 172.21.15.89:0/2411702917 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4ddc024050 con 0x5618afe2cb40 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dfcbcd740 10 client.20820 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/2411702917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618afe48f20 con 0x5618afe2cb40 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dd37fe640 1 -- 172.21.15.89:0/2411702917 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de409c650 con 0x5618afe2cb40 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dd37fe640 1 -- 172.21.15.89:0/2411702917 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4ddc0113e0 con 0x5618afe2cb40 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dfcbcd740 10 client.20820 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dfcbcd740 10 client.20820 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dfcbcd740 20 client.20820 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618afe4a430 0x5618afe4c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dfcbcd740 4 client.20820 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.724+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/2411702917 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618afd9e860 con 0x5618afe4a430 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.725+0000 7f4df3fff640 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618afe4a430 0x5618afe4c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.725+0000 7f4dd37fe640 10 client.20820 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:39:46.744 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.725+0000 7f4df3fff640 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618afe4a430 0x5618afe4c870 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4a48c1c6b431e9ac server_cookie=492ee9c2a92f6b63 in_seq=0 out_seq=0 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dd37fe640 1 -- 172.21.15.89:0/2411702917 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5618afd9e860 con 0x5618afe4a430 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dd37fe640 10 client.20820 handle_command_reply: tid=0 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 client.20820 shutdown 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/2411702917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 msgr2=0x5618afe33090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe33090 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00f4a0 tx=0x7f4ddc00f960 comp rx=0 tx=0).stop 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/2411702917 shutdown_connections 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618afe4a430 0x5618afe4c870 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe36370 0x5618afe337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dd37fe640 0 client.20820 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe33090 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/2411702917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe2b5d0 0x5618afe35de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.745 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/2411702917 >> 172.21.15.89:0/2411702917 conn(0x5618afdf4310 msgr2=0x5618afe2d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:46.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/2411702917 shutdown_connections 2024-09-15T23:39:46.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/2411702917 wait complete. 2024-09-15T23:39:46.746 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.726+0000 7f4dfcbcd740 20 client.20820 trim_cache size 0 max 16384 2024-09-15T23:39:46.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4dfcbcd740 1 Processor -- start 2024-09-15T23:39:46.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4dfcbcd740 1 -- start start 2024-09-15T23:39:46.750 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2b5d0 0x5618afe32950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe466c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe36370 0x5618afe2e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4df41acd00 con 0x5618afe2cb40 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4df41ae840 con 0x5618afe2b5d0 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4df41ad3d0 con 0x5618afe36370 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4debfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4debfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe466c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55594/0 (socket says 172.21.15.89:55594) 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.739+0000 7f4debfff640 1 -- 172.21.15.89:0/653708300 learned_addr learned my addr 172.21.15.89:0/653708300 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.740+0000 7f4df3fff640 1 --2- 172.21.15.89:0/653708300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2b5d0 0x5618afe32950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.740+0000 7f4debfff640 1 -- 172.21.15.89:0/653708300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe36370 msgr2=0x5618afe2e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.740+0000 7f4debfff640 1 --2- 172.21.15.89:0/653708300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe36370 0x5618afe2e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.740+0000 7f4debfff640 1 -- 172.21.15.89:0/653708300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2b5d0 msgr2=0x5618afe32950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:46.751 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.740+0000 7f4debfff640 1 --2- 172.21.15.89:0/653708300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2b5d0 0x5618afe32950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.740+0000 7f4debfff640 1 -- 172.21.15.89:0/653708300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618afe46c00 con 0x5618afe2cb40 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.740+0000 7f4debfff640 1 --2- 172.21.15.89:0/653708300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe466c0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00f470 tx=0x7f4ddc00a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8bb1f2e8718898 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.740+0000 7f4de9ffb640 1 -- 172.21.15.89:0/653708300 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc011eb0 con 0x5618afe2cb40 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.740+0000 7f4de9ffb640 1 -- 172.21.15.89:0/653708300 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ddc0113e0 con 0x5618afe2cb40 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.740+0000 7f4de9ffb640 1 -- 172.21.15.89:0/653708300 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc0064e0 con 0x5618afe2cb40 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/653708300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 msgr2=0x5618afe466c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/653708300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe466c0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc00f470 tx=0x7f4ddc00a640 comp rx=0 tx=0).stop 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/653708300 shutdown_connections 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/653708300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe36370 0x5618afe2e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/653708300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe2cb40 0x5618afe466c0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/653708300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2b5d0 0x5618afe32950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/653708300 >> 172.21.15.89:0/653708300 conn(0x5618afdf4310 msgr2=0x5618afe35110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:46.752 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/653708300 shutdown_connections 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/653708300 wait complete. 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 -- start start 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.741+0000 7f4dfcbcd740 1 Processor -- start 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2b5d0 0x5618afe42ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe2cb40 0x5618afe3e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4dfcbcd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe36370 0x5618afe3e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4df41ae170 con 0x5618afe36370 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4df41adaa0 con 0x5618afe2b5d0 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4dfcbcd740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4df41aace0 con 0x5618afe2cb40 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4dfcbcd740 10 client.0 _ll_register_callbacks cb 0x7f4ddc00a570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4df8d91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe36370 0x5618afe3e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4df8d91640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe36370 0x5618afe3e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55600/0 (socket says 172.21.15.89:55600) 2024-09-15T23:39:46.753 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4df8d91640 1 -- 172.21.15.89:0/186861618 learned_addr learned my addr 172.21.15.89:0/186861618 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4df3fff640 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2b5d0 0x5618afe42ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4debfff640 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe2cb40 0x5618afe3e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4df8d91640 1 -- 172.21.15.89:0/186861618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe2cb40 msgr2=0x5618afe3e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4df8d91640 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe2cb40 0x5618afe3e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4df8d91640 1 -- 172.21.15.89:0/186861618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2b5d0 msgr2=0x5618afe42ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4df8d91640 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2b5d0 0x5618afe42ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4df8d91640 1 -- 172.21.15.89:0/186861618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618afe3ef40 con 0x5618afe36370 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4df8d91640 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe36370 0x5618afe3e710 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f4dec009ca0 tx=0x7f4dec0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7ae892768ae1bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4dd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4dd37fe640 1 -- 172.21.15.89:0/186861618 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec034040 con 0x5618afe36370 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.742+0000 7f4dd37fe640 1 -- 172.21.15.89:0/186861618 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4dec004250 con 0x5618afe36370 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/186861618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ddc0113e0 con 0x5618afe36370 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dd37fe640 1 -- 172.21.15.89:0/186861618 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4dec004450 con 0x5618afe36370 2024-09-15T23:39:46.754 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dfcbcd740 10 client.20826 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/186861618 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618afeb5040 con 0x5618afe36370 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dd37fe640 1 -- 172.21.15.89:0/186861618 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ddc0113e0 con 0x5618afe36370 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dd37fe640 1 -- 172.21.15.89:0/186861618 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4dec050020 con 0x5618afe36370 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dfcbcd740 10 client.20826 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dfcbcd740 10 client.20826 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dfcbcd740 20 client.20826 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618afeb57c0 0x5618afeb7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dfcbcd740 4 client.20826 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/186861618 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5618afd98d70 con 0x5618afeb57c0 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.743+0000 7f4df3fff640 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618afeb57c0 0x5618afeb7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4df3fff640 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618afeb57c0 0x5618afeb7be0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=28c89f97f3a99712 server_cookie=1097e0c903ee27c6 in_seq=0 out_seq=0 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dd37fe640 10 client.20826 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dd37fe640 1 -- 172.21.15.89:0/186861618 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5618afd98d70 con 0x5618afeb57c0 2024-09-15T23:39:46.755 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dd37fe640 10 client.20826 handle_command_reply: tid=0 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 client.20826 shutdown 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/186861618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe36370 msgr2=0x5618afe3e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe36370 0x5618afe3e710 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f4dec009ca0 tx=0x7f4dec0095c0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/186861618 shutdown_connections 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x5618afeb57c0 0x5618afeb7be0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5618afe36370 0x5618afe3e710 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dd37fe640 0 client.20826 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5618afe2cb40 0x5618afe3e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 --2- 172.21.15.89:0/186861618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618afe2b5d0 0x5618afe42ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/186861618 >> 172.21.15.89:0/186861618 conn(0x5618afdf4310 msgr2=0x5618afe264e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/186861618 shutdown_connections 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 1 -- 172.21.15.89:0/186861618 wait complete. 2024-09-15T23:39:46.756 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.744+0000 7f4dfcbcd740 20 client.20826 trim_cache size 0 max 16384 2024-09-15T23:39:46.757 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:39:46.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2355156785 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4de00a32c0 msgr2=0x7f4de00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:46.757 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4de00a32c0 0x7f4de00a5780 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4dec030040 tx=0x7f4dec03e040 comp rx=0 tx=0).stop 2024-09-15T23:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2355156785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4df4102da0 msgr2=0x7f4df4198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4df4102da0 0x7f4df4198440 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f4de400c890 tx=0x7f4de400cd50 comp rx=0 tx=0).stop 2024-09-15T23:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2355156785 shutdown_connections 2024-09-15T23:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4de00a32c0 0x7f4de00a5780 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4df410c180 0x7f4df419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df41036c0 0x7f4df4198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 --2- 172.21.15.89:0/2355156785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4df4102da0 0x7f4df4198440 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2355156785 >> 172.21.15.89:0/2355156785 conn(0x7f4df40fc7a0 msgr2=0x7f4df40fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2355156785 shutdown_connections 2024-09-15T23:39:46.758 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:46.745+0000 7f4dfa7f4640 1 -- 172.21.15.89:0/2355156785 wait complete. 2024-09-15T23:39:47.109 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:39:47.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:47 smithi120 ceph-mon[35250]: pgmap v822: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-09-15T23:39:47.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:47 smithi174 ceph-mon[35586]: pgmap v822: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-09-15T23:39:47.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:47 smithi089 ceph-mon[30933]: pgmap v822: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-09-15T23:39:47.652 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:39:47.704 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.723+0000 7f79a4c82640 1 -- 172.21.15.89:0/3054801580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a0107830 msgr2=0x7f79a0109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.723+0000 7f79a4c82640 1 --2- 172.21.15.89:0/3054801580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a0107830 0x7f79a0109ca0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f7998009ca0 tx=0x7f799802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.723+0000 7f79a4c82640 1 -- 172.21.15.89:0/3054801580 shutdown_connections 2024-09-15T23:39:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.723+0000 7f79a4c82640 1 --2- 172.21.15.89:0/3054801580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a0107830 0x7f79a0109ca0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.723+0000 7f79a4c82640 1 --2- 172.21.15.89:0/3054801580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a0104f00 0x7f79a01072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.723+0000 7f79a4c82640 1 --2- 172.21.15.89:0/3054801580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79a01025d0 0x7f79a01049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.723+0000 7f79a4c82640 1 -- 172.21.15.89:0/3054801580 >> 172.21.15.89:0/3054801580 conn(0x7f79a00fc1b0 msgr2=0x7f79a00fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.723+0000 7f79a4c82640 1 -- 172.21.15.89:0/3054801580 shutdown_connections 2024-09-15T23:39:47.726 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.723+0000 7f79a4c82640 1 -- 172.21.15.89:0/3054801580 wait complete. 2024-09-15T23:39:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.724+0000 7f79a4c82640 1 Processor -- start 2024-09-15T23:39:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.724+0000 7f79a4c82640 1 -- start start 2024-09-15T23:39:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.724+0000 7f79a4c82640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a01025d0 0x7f79a0197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f79a4c82640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79a0104f00 0x7f79a0198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f79a4c82640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a0107830 0x7f79a019f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f79a4c82640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79a010d810 con 0x7f79a0107830 2024-09-15T23:39:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f79a4c82640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f79a010d690 con 0x7f79a01025d0 2024-09-15T23:39:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f79a4c82640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79a010d990 con 0x7f79a0104f00 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a0107830 0x7f79a019f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a0107830 0x7f79a019f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55608/0 (socket says 172.21.15.89:55608) 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799ed9d640 1 -- 172.21.15.89:0/3835702217 learned_addr learned my addr 172.21.15.89:0/3835702217 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799dd9b640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79a0104f00 0x7f79a0198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799e59c640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a01025d0 0x7f79a0197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799ed9d640 1 -- 172.21.15.89:0/3835702217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79a0104f00 msgr2=0x7f79a0198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799ed9d640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79a0104f00 0x7f79a0198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799ed9d640 1 -- 172.21.15.89:0/3835702217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a01025d0 msgr2=0x7f79a0197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799ed9d640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a01025d0 0x7f79a0197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799ed9d640 1 -- 172.21.15.89:0/3835702217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a019fa60 con 0x7f79a0107830 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.725+0000 7f799ed9d640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a0107830 0x7f79a019f3f0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f7998031a40 tx=0x7f7998031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.726+0000 7f798f7fe640 1 -- 172.21.15.89:0/3835702217 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998003c30 con 0x7f79a0107830 2024-09-15T23:39:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.726+0000 7f798f7fe640 1 -- 172.21.15.89:0/3835702217 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7998003dd0 con 0x7f79a0107830 2024-09-15T23:39:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.726+0000 7f798f7fe640 1 -- 172.21.15.89:0/3835702217 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799802fdb0 con 0x7f79a0107830 2024-09-15T23:39:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.726+0000 7f79a4c82640 1 -- 172.21.15.89:0/3835702217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a019fcf0 con 0x7f79a0107830 2024-09-15T23:39:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.726+0000 7f79a4c82640 1 -- 172.21.15.89:0/3835702217 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a01a0230 con 0x7f79a0107830 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.729+0000 7f79a7a21740 1 Processor -- start 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.729+0000 7f798f7fe640 1 -- 172.21.15.89:0/3835702217 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f7998004050 con 0x7f79a0107830 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.729+0000 7f79a7a21740 1 -- start start 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.729+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d719a0 0x561f57d71da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.729+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 0x561f57d76da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.729+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d79830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.729+0000 7f79a7a21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561f57cd52f0 con 0x561f57d77370 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.729+0000 7f79a7a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561f57c7d270 con 0x561f57d719a0 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.729+0000 7f79a7a21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561f57cd12b0 con 0x561f57d722e0 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d79830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799e59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 0x561f57d76da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d79830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55616/0 (socket says 172.21.15.89:55616) 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799dd9b640 1 -- 172.21.15.89:0/2411321223 learned_addr learned my addr 172.21.15.89:0/2411321223 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799dd9b640 1 --2- 172.21.15.89:0/2411321223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d719a0 0x561f57d71da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799dd9b640 1 -- 172.21.15.89:0/2411321223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 msgr2=0x561f57d76da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799dd9b640 1 --2- 172.21.15.89:0/2411321223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 0x561f57d76da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799dd9b640 1 -- 172.21.15.89:0/2411321223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d719a0 msgr2=0x561f57d71da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799dd9b640 1 --2- 172.21.15.89:0/2411321223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d719a0 0x561f57d71da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799dd9b640 1 -- 172.21.15.89:0/2411321223 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f57c7c340 con 0x561f57d77370 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799e59c640 1 --2- 172.21.15.89:0/2411321223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 0x561f57d76da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f798f7fe640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79880a3370 0x7f79880a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799dd9b640 1 --2- 172.21.15.89:0/2411321223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d79830 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f7994012580 tx=0x7f799400b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87a0d9f48445f9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f798cff9640 1 -- 172.21.15.89:0/2411321223 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799401e640 con 0x561f57d77370 2024-09-15T23:39:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f798cff9640 1 -- 172.21.15.89:0/2411321223 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7994010910 con 0x561f57d77370 2024-09-15T23:39:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f798f7fe640 1 -- 172.21.15.89:0/3835702217 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f79980f0e40 con 0x7f79a0107830 2024-09-15T23:39:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f798cff9640 1 -- 172.21.15.89:0/2411321223 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799401d980 con 0x561f57d77370 2024-09-15T23:39:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.730+0000 7f799e59c640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79880a3370 0x7f79880a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.731+0000 7f79a7a21740 1 -- 172.21.15.89:0/2411321223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 msgr2=0x561f57d79830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.731+0000 7f79a7a21740 1 --2- 172.21.15.89:0/2411321223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d79830 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f7994012580 tx=0x7f799400b9d0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.731+0000 7f799e59c640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79880a3370 0x7f79880a5830 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x561f57d72d10 tx=0x7f799000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.731+0000 7f79a7a21740 1 -- 172.21.15.89:0/2411321223 shutdown_connections 2024-09-15T23:39:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.731+0000 7f79a7a21740 1 --2- 172.21.15.89:0/2411321223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d79830 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.731+0000 7f79a7a21740 1 --2- 172.21.15.89:0/2411321223 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 0x561f57d76da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.738 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.731+0000 7f79a7a21740 1 --2- 172.21.15.89:0/2411321223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d719a0 0x561f57d71da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.731+0000 7f79a7a21740 1 -- 172.21.15.89:0/2411321223 >> 172.21.15.89:0/2411321223 conn(0x561f57d6d010 msgr2=0x561f57d6d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.731+0000 7f79a7a21740 1 -- 172.21.15.89:0/2411321223 shutdown_connections 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.731+0000 7f79a7a21740 1 -- 172.21.15.89:0/2411321223 wait complete. 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f79a7a21740 1 -- start start 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f79a7a21740 1 Processor -- start 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d719a0 0x561f57d7bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d722e0 0x561f57d7c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d77370 0x561f57d831a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f79a7a21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561f57d7a340 con 0x561f57d719a0 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f79a7a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561f57c5e290 con 0x561f57d722e0 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f79a7a21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561f57d7a4c0 con 0x561f57d77370 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f79a7a21740 10 client.0 _ll_register_callbacks cb 0x561f57c8f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:47.739 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f799e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d719a0 0x561f57d7bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f799e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d719a0 0x561f57d7bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55626/0 (socket says 172.21.15.89:55626) 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f799dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d722e0 0x561f57d7c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f799e59c640 1 -- 172.21.15.89:0/2811008900 learned_addr learned my addr 172.21.15.89:0/2811008900 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f799ed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d77370 0x561f57d831a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f799e59c640 1 -- 172.21.15.89:0/2811008900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d77370 msgr2=0x561f57d831a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f799e59c640 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d77370 0x561f57d831a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.732+0000 7f799e59c640 1 -- 172.21.15.89:0/2811008900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d722e0 msgr2=0x561f57d7c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f799e59c640 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d722e0 0x561f57d7c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f799e59c640 1 -- 172.21.15.89:0/2811008900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f579e8210 con 0x561f57d719a0 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f799ed9d640 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d77370 0x561f57d831a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f799dd9b640 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d722e0 0x561f57d7c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:47.741 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f799e59c640 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d719a0 0x561f57d7bc20 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f7990003d30 tx=0x7f7990004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353231428fb7fe96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f7985ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f7985ffb640 1 -- 172.21.15.89:0/2811008900 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799002c060 con 0x561f57d719a0 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f7985ffb640 1 -- 172.21.15.89:0/2811008900 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7990017030 con 0x561f57d719a0 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f79a7a21740 1 -- 172.21.15.89:0/2811008900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7994010910 con 0x561f57d719a0 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f7985ffb640 1 -- 172.21.15.89:0/2811008900 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7990027410 con 0x561f57d719a0 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f79a7a21740 10 client.20838 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.733+0000 7f79a7a21740 1 -- 172.21.15.89:0/2811008900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561f57ab4620 con 0x561f57d719a0 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.734+0000 7f7985ffb640 1 -- 172.21.15.89:0/2811008900 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7994010910 con 0x561f57d719a0 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.734+0000 7f7985ffb640 1 -- 172.21.15.89:0/2811008900 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f79900117f0 con 0x561f57d719a0 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.734+0000 7f79a7a21740 10 client.20838 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.734+0000 7f79a7a21740 10 client.20838 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.734+0000 7f79a7a21740 20 client.20838 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:47.742 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.734+0000 7f79a7a21740 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561f57d87940 0x561f57d89d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.734+0000 7f79a7a21740 4 client.20838 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.734+0000 7f79a7a21740 1 -- 172.21.15.89:0/2811008900 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f57ce2a20 con 0x561f57d87940 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.734+0000 7f799dd9b640 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561f57d87940 0x561f57d89d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.735+0000 7f799dd9b640 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561f57d87940 0x561f57d89d50 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a33f2c4c03f0669c server_cookie=e4bab96daae05ee8 in_seq=0 out_seq=0 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.735+0000 7f7985ffb640 10 client.20838 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.735+0000 7f7985ffb640 1 -- 172.21.15.89:0/2811008900 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561f57ce2a20 con 0x561f57d87940 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.735+0000 7f7985ffb640 10 client.20838 handle_command_reply: tid=0 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.735+0000 7f79a7a21740 1 client.20838 shutdown 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 1 -- 172.21.15.89:0/2811008900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d719a0 msgr2=0x561f57d7bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d719a0 0x561f57d7bc20 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f7990003d30 tx=0x7f7990004730 comp rx=0 tx=0).stop 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 1 -- 172.21.15.89:0/2811008900 shutdown_connections 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561f57d87940 0x561f57d89d50 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d77370 0x561f57d831a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.743 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d722e0 0x561f57d7c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 1 --2- 172.21.15.89:0/2811008900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d719a0 0x561f57d7bc20 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 1 -- 172.21.15.89:0/2811008900 >> 172.21.15.89:0/2811008900 conn(0x561f57d6d010 msgr2=0x561f57d6e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f7985ffb640 0 client.20838 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 1 -- 172.21.15.89:0/2811008900 shutdown_connections 2024-09-15T23:39:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 1 -- 172.21.15.89:0/2811008900 wait complete. 2024-09-15T23:39:47.744 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.736+0000 7f79a7a21740 20 client.20838 trim_cache size 0 max 16384 2024-09-15T23:39:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.750+0000 7f79a7a21740 1 Processor -- start 2024-09-15T23:39:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.750+0000 7f79a7a21740 1 -- start start 2024-09-15T23:39:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.750+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d719a0 0x561f57d6de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.750+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 0x561f57d83160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.750+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d6e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.750+0000 7f79a7a21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01a8690 con 0x561f57d77370 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.750+0000 7f79a7a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01aa1d0 con 0x561f57d719a0 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.750+0000 7f79a7a21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01a8d60 con 0x561f57d722e0 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d6e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d6e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55642/0 (socket says 172.21.15.89:55642) 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799ed9d640 1 -- 172.21.15.89:0/1192249965 learned_addr learned my addr 172.21.15.89:0/1192249965 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799dd9b640 1 --2- 172.21.15.89:0/1192249965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 0x561f57d83160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799e59c640 1 --2- 172.21.15.89:0/1192249965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d719a0 0x561f57d6de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799ed9d640 1 -- 172.21.15.89:0/1192249965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 msgr2=0x561f57d83160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799ed9d640 1 --2- 172.21.15.89:0/1192249965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 0x561f57d83160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799ed9d640 1 -- 172.21.15.89:0/1192249965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d719a0 msgr2=0x561f57d6de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:47.753 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799ed9d640 1 --2- 172.21.15.89:0/1192249965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d719a0 0x561f57d6de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799ed9d640 1 -- 172.21.15.89:0/1192249965 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f57d84640 con 0x561f57d77370 2024-09-15T23:39:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f799ed9d640 1 --2- 172.21.15.89:0/1192249965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d6e3d0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f79980ee5a0 tx=0x7f799803d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0472af5df9a8c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f7987fff640 1 -- 172.21.15.89:0/1192249965 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980bd890 con 0x561f57d77370 2024-09-15T23:39:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f7987fff640 1 -- 172.21.15.89:0/1192249965 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7998036040 con 0x561f57d77370 2024-09-15T23:39:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.751+0000 7f7987fff640 1 -- 172.21.15.89:0/1192249965 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980b9800 con 0x561f57d77370 2024-09-15T23:39:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 -- 172.21.15.89:0/1192249965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 msgr2=0x561f57d6e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 --2- 172.21.15.89:0/1192249965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d6e3d0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f79980ee5a0 tx=0x7f799803d040 comp rx=0 tx=0).stop 2024-09-15T23:39:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 -- 172.21.15.89:0/1192249965 shutdown_connections 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 --2- 172.21.15.89:0/1192249965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d77370 0x561f57d6e3d0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 --2- 172.21.15.89:0/1192249965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d722e0 0x561f57d83160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 --2- 172.21.15.89:0/1192249965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d719a0 0x561f57d6de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 -- 172.21.15.89:0/1192249965 >> 172.21.15.89:0/1192249965 conn(0x561f57d6d010 msgr2=0x561f57d73520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 -- 172.21.15.89:0/1192249965 shutdown_connections 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 -- 172.21.15.89:0/1192249965 wait complete. 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 -- start start 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.752+0000 7f79a7a21740 1 Processor -- start 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d719a0 0x561f57d800f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d722e0 0x561f57d80630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f79a7a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d77370 0x561f57df0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f79a7a21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01a9b00 con 0x561f57d722e0 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f79a7a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01a9430 con 0x561f57d77370 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f79a7a21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01a6670 con 0x561f57d719a0 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f79a7a21740 10 client.0 _ll_register_callbacks cb 0x7f7990003bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d722e0 0x561f57d80630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d722e0 0x561f57d80630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55656/0 (socket says 172.21.15.89:55656) 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799dd9b640 1 -- 172.21.15.89:0/1351161378 learned_addr learned my addr 172.21.15.89:0/1351161378 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799e59c640 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d719a0 0x561f57d800f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799ed9d640 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d77370 0x561f57df0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799dd9b640 1 -- 172.21.15.89:0/1351161378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d719a0 msgr2=0x561f57d800f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799dd9b640 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d719a0 0x561f57d800f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799dd9b640 1 -- 172.21.15.89:0/1351161378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d77370 msgr2=0x561f57df0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799dd9b640 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d77370 0x561f57df0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799dd9b640 1 -- 172.21.15.89:0/1351161378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f57df0c10 con 0x561f57d722e0 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f799dd9b640 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d722e0 0x561f57d80630 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f7994012450 tx=0x7f799400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17fe0e5bf0f77bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f7985ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f7985ffb640 1 -- 172.21.15.89:0/1351161378 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994010c40 con 0x561f57d722e0 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f7985ffb640 1 -- 172.21.15.89:0/1351161378 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7994010430 con 0x561f57d722e0 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f79a7a21740 1 -- 172.21.15.89:0/1351161378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7998036040 con 0x561f57d722e0 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f7985ffb640 1 -- 172.21.15.89:0/1351161378 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f799403c050 con 0x561f57d722e0 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f79a7a21740 10 client.20844 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.753+0000 7f79a7a21740 1 -- 172.21.15.89:0/1351161378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561f57df1f20 con 0x561f57d722e0 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.754+0000 7f7985ffb640 1 -- 172.21.15.89:0/1351161378 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7998036040 con 0x561f57d722e0 2024-09-15T23:39:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.754+0000 7f7985ffb640 1 -- 172.21.15.89:0/1351161378 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f799401c070 con 0x561f57d722e0 2024-09-15T23:39:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.754+0000 7f79a7a21740 10 client.20844 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.754+0000 7f79a7a21740 10 client.20844 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.754+0000 7f79a7a21740 20 client.20844 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.754+0000 7f79a7a21740 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561f57df2760 0x561f57df4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.754+0000 7f79a7a21740 4 client.20844 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:39:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.754+0000 7f79a7a21740 1 -- 172.21.15.89:0/1351161378 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f57cdcd70 con 0x561f57df2760 2024-09-15T23:39:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.754+0000 7f799e59c640 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561f57df2760 0x561f57df4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.755+0000 7f799e59c640 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561f57df2760 0x561f57df4b80 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f763932a771b19ae server_cookie=a6426b4d0c1e22fd in_seq=0 out_seq=0 2024-09-15T23:39:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.755+0000 7f7985ffb640 10 client.20844 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.768+0000 7f7985ffb640 1 -- 172.21.15.89:0/1351161378 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f57cdcd70 con 0x561f57df2760 2024-09-15T23:39:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.768+0000 7f7985ffb640 10 client.20844 handle_command_reply: tid=0 2024-09-15T23:39:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.768+0000 7f79a7a21740 1 client.20844 shutdown 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.768+0000 7f79a7a21740 1 -- 172.21.15.89:0/1351161378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d722e0 msgr2=0x561f57d80630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.768+0000 7f79a7a21740 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d722e0 0x561f57d80630 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f7994012450 tx=0x7f799400bdd0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.768+0000 7f79a7a21740 1 -- 172.21.15.89:0/1351161378 shutdown_connections 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.768+0000 7f79a7a21740 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x561f57df2760 0x561f57df4b80 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a7a21740 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561f57d77370 0x561f57df0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a7a21740 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561f57d722e0 0x561f57d80630 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a7a21740 1 --2- 172.21.15.89:0/1351161378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561f57d719a0 0x561f57d800f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f7985ffb640 0 client.20844 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a7a21740 1 -- 172.21.15.89:0/1351161378 >> 172.21.15.89:0/1351161378 conn(0x561f57d6d010 msgr2=0x561f57d73520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a7a21740 1 -- 172.21.15.89:0/1351161378 shutdown_connections 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a7a21740 1 -- 172.21.15.89:0/1351161378 wait complete. 2024-09-15T23:39:47.771 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a7a21740 20 client.20844 trim_cache size 0 max 16384 2024-09-15T23:39:47.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:39:47.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:39:47.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:39:47.772 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 -- 172.21.15.89:0/3835702217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79880a3370 msgr2=0x7f79880a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79880a3370 0x7f79880a5830 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x561f57d72d10 tx=0x7f799000b390 comp rx=0 tx=0).stop 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 -- 172.21.15.89:0/3835702217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a0107830 msgr2=0x7f79a019f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a0107830 0x7f79a019f3f0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f7998031a40 tx=0x7f7998031ee0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 -- 172.21.15.89:0/3835702217 shutdown_connections 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79880a3370 0x7f79880a5830 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79a0107830 0x7f79a019f3f0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79a0104f00 0x7f79a0198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 --2- 172.21.15.89:0/3835702217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79a01025d0 0x7f79a0197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 -- 172.21.15.89:0/3835702217 >> 172.21.15.89:0/3835702217 conn(0x7f79a00fc1b0 msgr2=0x7f79a0102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 -- 172.21.15.89:0/3835702217 shutdown_connections 2024-09-15T23:39:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:39:47.769+0000 7f79a4c82640 1 -- 172.21.15.89:0/3835702217 wait complete. 2024-09-15T23:39:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.177+0000 7fd24cd4d640 1 -- 172.21.15.89:0/2180314755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd248101420 msgr2=0x7fd24810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.177+0000 7fd24cd4d640 1 --2- 172.21.15.89:0/2180314755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd248101420 0x7fd24810d950 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fd240009cd0 tx=0x7fd24002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:39:48.190 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.177+0000 7fd24cd4d640 1 -- 172.21.15.89:0/2180314755 shutdown_connections 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.177+0000 7fd24cd4d640 1 --2- 172.21.15.89:0/2180314755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd248101420 0x7fd24810d950 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.177+0000 7fd24cd4d640 1 --2- 172.21.15.89:0/2180314755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd248100a80 0x7fd248100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.177+0000 7fd24cd4d640 1 --2- 172.21.15.89:0/2180314755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd248106a80 0x7fd248106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.177+0000 7fd24cd4d640 1 -- 172.21.15.89:0/2180314755 >> 172.21.15.89:0/2180314755 conn(0x7fd2480fc7a0 msgr2=0x7fd2480febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.177+0000 7fd24cd4d640 1 -- 172.21.15.89:0/2180314755 shutdown_connections 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.177+0000 7fd24cd4d640 1 -- 172.21.15.89:0/2180314755 wait complete. 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.177+0000 7fd24cd4d640 1 Processor -- start 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24cd4d640 1 -- start start 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24cd4d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd248100a80 0x7fd24810e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24cd4d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd248101420 0x7fd24810e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24cd4d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd248106a80 0x7fd24810ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:48.191 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24cd4d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd248111f90 con 0x7fd248100a80 2024-09-15T23:39:48.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24cd4d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd248111e10 con 0x7fd248101420 2024-09-15T23:39:48.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24cd4d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd248112110 con 0x7fd248106a80 2024-09-15T23:39:48.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd248100a80 0x7fd24810e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:48.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd248100a80 0x7fd24810e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55668/0 (socket says 172.21.15.89:55668) 2024-09-15T23:39:48.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24659c640 1 -- 172.21.15.89:0/3541463998 learned_addr learned my addr 172.21.15.89:0/3541463998 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:48.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd23ffff640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd248101420 0x7fd24810e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:48.192 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24659c640 1 -- 172.21.15.89:0/3541463998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd248106a80 msgr2=0x7fd24810ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:48.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24659c640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd248106a80 0x7fd24810ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:48.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24659c640 1 -- 172.21.15.89:0/3541463998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd248101420 msgr2=0x7fd24810e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:48.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24659c640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd248101420 0x7fd24810e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:48.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24659c640 1 -- 172.21.15.89:0/3541463998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2481a68b0 con 0x7fd248100a80 2024-09-15T23:39:48.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd23ffff640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd248101420 0x7fd24810e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:48.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd24659c640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd248100a80 0x7fd24810e260 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fd23800d950 tx=0x7fd23800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:48.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.178+0000 7fd23f7fe640 1 -- 172.21.15.89:0/3541463998 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238017070 con 0x7fd248100a80 2024-09-15T23:39:48.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.179+0000 7fd23f7fe640 1 -- 172.21.15.89:0/3541463998 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd238009c30 con 0x7fd248100a80 2024-09-15T23:39:48.193 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.179+0000 7fd23f7fe640 1 -- 172.21.15.89:0/3541463998 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23800f6b0 con 0x7fd248100a80 2024-09-15T23:39:48.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.179+0000 7fd24cd4d640 1 -- 172.21.15.89:0/3541463998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2481a6ba0 con 0x7fd248100a80 2024-09-15T23:39:48.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.179+0000 7fd24cd4d640 1 -- 172.21.15.89:0/3541463998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2481a70e0 con 0x7fd248100a80 2024-09-15T23:39:48.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.180+0000 7fd24cd4d640 1 -- 172.21.15.89:0/3541463998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd248100ee0 con 0x7fd248100a80 2024-09-15T23:39:48.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.181+0000 7fd23f7fe640 1 -- 172.21.15.89:0/3541463998 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd2380109d0 con 0x7fd248100a80 2024-09-15T23:39:48.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.181+0000 7fd23f7fe640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2340a3240 0x7fd2340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:48.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.181+0000 7fd23ffff640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2340a3240 0x7fd2340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:48.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.181+0000 7fd23f7fe640 1 -- 172.21.15.89:0/3541463998 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fd2380cf950 con 0x7fd248100a80 2024-09-15T23:39:48.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.182+0000 7fd23ffff640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2340a3240 0x7fd2340a5700 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd2300058c0 tx=0x7fd230005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:48.194 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.185+0000 7fd23f7fe640 1 -- 172.21.15.89:0/3541463998 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd238009de0 con 0x7fd248100a80 2024-09-15T23:39:48.228 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:48.399 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.398+0000 7fd24cd4d640 1 -- 172.21.15.89:0/3541463998 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2481a7410 con 0x7fd248100a80 2024-09-15T23:39:48.400 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.399+0000 7fd23f7fe640 1 -- 172.21.15.89:0/3541463998 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fd23809c280 con 0x7fd248100a80 2024-09-15T23:39:48.400 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:39:48.400 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:39:48.401 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:39:48.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 -- 172.21.15.89:0/3541463998 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2340a3240 msgr2=0x7fd2340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:48.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2340a3240 0x7fd2340a5700 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd2300058c0 tx=0x7fd230005850 comp rx=0 tx=0).stop 2024-09-15T23:39:48.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 -- 172.21.15.89:0/3541463998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd248100a80 msgr2=0x7fd24810e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:48.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd248100a80 0x7fd24810e260 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7fd23800d950 tx=0x7fd23800de10 comp rx=0 tx=0).stop 2024-09-15T23:39:48.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 -- 172.21.15.89:0/3541463998 shutdown_connections 2024-09-15T23:39:48.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd2340a3240 0x7fd2340a5700 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd248106a80 0x7fd24810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd248101420 0x7fd24810e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 --2- 172.21.15.89:0/3541463998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd248100a80 0x7fd24810e260 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 -- 172.21.15.89:0/3541463998 >> 172.21.15.89:0/3541463998 conn(0x7fd2480fc7a0 msgr2=0x7fd2480fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 -- 172.21.15.89:0/3541463998 shutdown_connections 2024-09-15T23:39:48.403 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:48.401+0000 7fd24cd4d640 1 -- 172.21.15.89:0/3541463998 wait complete. 2024-09-15T23:39:49.375 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 58c1329f --release --await' 2024-09-15T23:39:49.375 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 58c1329f --release --await 2024-09-15T23:39:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:49 smithi120 ceph-mon[35250]: pgmap v823: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.8 KiB/s wr, 14 op/s 2024-09-15T23:39:49.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:49 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3541463998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:49 smithi174 ceph-mon[35586]: pgmap v823: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.8 KiB/s wr, 14 op/s 2024-09-15T23:39:49.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:49 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3541463998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:49.510 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:49 smithi089 ceph-mon[30933]: pgmap v823: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.8 KiB/s wr, 14 op/s 2024-09-15T23:39:49.510 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:49 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3541463998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:49.626 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.593+0000 7fa2f5ada640 1 -- 172.21.15.89:0/3767229840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f010c160 msgr2=0x7fa2f010e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:49.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.593+0000 7fa2f5ada640 1 --2- 172.21.15.89:0/3767229840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f010c160 0x7fa2f010e600 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8009ca0 tx=0x7fa2e802a950 comp rx=0 tx=0).stop 2024-09-15T23:39:49.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.593+0000 7fa2f5ada640 1 -- 172.21.15.89:0/3767229840 shutdown_connections 2024-09-15T23:39:49.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.593+0000 7fa2f5ada640 1 --2- 172.21.15.89:0/3767229840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f010c160 0x7fa2f010e600 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.593+0000 7fa2f5ada640 1 --2- 172.21.15.89:0/3767229840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f0103650 0x7fa2f010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.593+0000 7fa2f5ada640 1 --2- 172.21.15.89:0/3767229840 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f0102d30 0x7fa2f0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.593+0000 7fa2f5ada640 1 -- 172.21.15.89:0/3767229840 >> 172.21.15.89:0/3767229840 conn(0x7fa2f00fc7d0 msgr2=0x7fa2f00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:49.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.593+0000 7fa2f5ada640 1 -- 172.21.15.89:0/3767229840 shutdown_connections 2024-09-15T23:39:49.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.593+0000 7fa2f5ada640 1 -- 172.21.15.89:0/3767229840 wait complete. 2024-09-15T23:39:49.627 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2f5ada640 1 Processor -- start 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2f5ada640 1 -- start start 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2f5ada640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f0102d30 0x7fa2f0100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2f5ada640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f0103650 0x7fa2f0100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2f5ada640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f010c160 0x7fa2f0101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2f5ada640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f0111f60 con 0x7fa2f0102d30 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2f5ada640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f0111de0 con 0x7fa2f0103650 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2f5ada640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f01120e0 con 0x7fa2f010c160 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f0102d30 0x7fa2f0100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f0102d30 0x7fa2f0100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55684/0 (socket says 172.21.15.89:55684) 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/2128372973 learned_addr learned my addr 172.21.15.89:0/2128372973 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2effff640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f010c160 0x7fa2f0101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.594+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f0103650 0x7fa2f0100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/2128372973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f010c160 msgr2=0x7fa2f0101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f010c160 0x7fa2f0101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/2128372973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f0103650 msgr2=0x7fa2f0100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f0103650 0x7fa2f0100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/2128372973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f01a68d0 con 0x7fa2f0102d30 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f0102d30 0x7fa2f0100990 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c890 tx=0x7fa2e000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2ecff9640 1 -- 172.21.15.89:0/2128372973 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0004220 con 0x7fa2f0102d30 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2f5ada640 1 -- 172.21.15.89:0/2128372973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f01a6bc0 con 0x7fa2f0102d30 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2ecff9640 1 -- 172.21.15.89:0/2128372973 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2e00043c0 con 0x7fa2f0102d30 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2ecff9640 1 -- 172.21.15.89:0/2128372973 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e00049e0 con 0x7fa2f0102d30 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.595+0000 7fa2f5ada640 1 -- 172.21.15.89:0/2128372973 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f01a7100 con 0x7fa2f0102d30 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.597+0000 7fa2ecff9640 1 -- 172.21.15.89:0/2128372973 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa2e00079b0 con 0x7fa2f0102d30 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.598+0000 7fa2ecff9640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa2d80a3240 0x7fa2d80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.629 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.598+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa2d80a3240 0x7fa2d80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.598+0000 7fa2f7eb3740 1 Processor -- start 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.598+0000 7fa2f7eb3740 1 -- start start 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.598+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85bf78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf7de0 0x562a85bf8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 0x562a85c04f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562a85b552f0 con 0x562a85bf9350 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562a85afccf0 con 0x562a85c02b80 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562a85b512b0 con 0x562a85bf7de0 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2ecff9640 1 -- 172.21.15.89:0/2128372973 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa2e00cfcb0 con 0x7fa2f0102d30 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa2d80a3240 0x7fa2d80a5700 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0102420 tx=0x7fa2e4005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85bf78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2effff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 0x562a85c04f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2effff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 0x562a85c04f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:46684/0 (socket says 172.21.15.89:46684) 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2effff640 1 -- 172.21.15.89:0/3765617361 learned_addr learned my addr 172.21.15.89:0/3765617361 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85bf78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55690/0 (socket says 172.21.15.89:55690) 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/3765617361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf7de0 0x562a85bf8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2effff640 1 -- 172.21.15.89:0/3765617361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf7de0 msgr2=0x562a85bf8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2effff640 1 --2- 172.21.15.89:0/3765617361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf7de0 0x562a85bf8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2effff640 1 -- 172.21.15.89:0/3765617361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 msgr2=0x562a85bf78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2effff640 1 --2- 172.21.15.89:0/3765617361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85bf78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2effff640 1 -- 172.21.15.89:0/3765617361 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a85ae13a0 con 0x562a85c02b80 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/3765617361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85bf78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/3765617361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf7de0 0x562a85bf8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.599+0000 7fa2effff640 1 --2- 172.21.15.89:0/3765617361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 0x562a85c04f70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8035000 tx=0x7fa2e802ff50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2398d165fd54614c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.600+0000 7fa2d5ffb640 1 -- 172.21.15.89:0/3765617361 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8048070 con 0x562a85c02b80 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.600+0000 7fa2d5ffb640 1 -- 172.21.15.89:0/3765617361 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2e802fb00 con 0x562a85c02b80 2024-09-15T23:39:49.631 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.600+0000 7fa2d5ffb640 1 -- 172.21.15.89:0/3765617361 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8034040 con 0x562a85c02b80 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.600+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/3765617361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 msgr2=0x562a85c04f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.600+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/3765617361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 0x562a85c04f70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8035000 tx=0x7fa2e802ff50 comp rx=0 tx=0).stop 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.600+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/3765617361 shutdown_connections 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.600+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/3765617361 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 0x562a85c04f70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.600+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/3765617361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf7de0 0x562a85bf8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.600+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/3765617361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85bf78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.600+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/3765617361 >> 172.21.15.89:0/3765617361 conn(0x562a85bb8310 msgr2=0x562a85bf4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/3765617361 shutdown_connections 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/3765617361 wait complete. 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 1 -- start start 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 1 Processor -- start 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85c02540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.632 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf9350 0x562a85bff7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 0x562a85bffde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562a85c05820 con 0x562a85bf7de0 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562a85b10060 con 0x562a85c02b80 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562a85c059a0 con 0x562a85bf9350 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.601+0000 7fa2f7eb3740 10 client.0 _ll_register_callbacks cb 0x562a858638c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85c02540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85c02540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55700/0 (socket says 172.21.15.89:55700) 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/117681637 learned_addr learned my addr 172.21.15.89:0/117681637 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2effff640 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 0x562a85bffde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf9350 0x562a85bff7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/117681637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf9350 msgr2=0x562a85bff7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf9350 0x562a85bff7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.633 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/117681637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 msgr2=0x562a85bffde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 0x562a85bffde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/117681637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a858c63c0 con 0x562a85bf7de0 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf9350 0x562a85bff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85c02540 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fa2e009bc90 tx=0x7fa2e00a0350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc11bfb9f258413 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ceffd640 1 -- 172.21.15.89:0/117681637 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e001d040 con 0x562a85bf7de0 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/117681637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2e802fb00 con 0x562a85bf7de0 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ceffd640 1 -- 172.21.15.89:0/117681637 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa2e0007ca0 con 0x562a85bf7de0 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2ceffd640 1 -- 172.21.15.89:0/117681637 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa2e00a46c0 con 0x562a85bf7de0 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2f7eb3740 10 client.20862 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.602+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/117681637 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562a85c11360 con 0x562a85bf7de0 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.603+0000 7fa2ceffd640 1 -- 172.21.15.89:0/117681637 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2e802fb00 con 0x562a85bf7de0 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.603+0000 7fa2ceffd640 1 -- 172.21.15.89:0/117681637 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa2e0096b00 con 0x562a85bf7de0 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.603+0000 7fa2f7eb3740 10 client.20862 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:49.634 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.603+0000 7fa2f7eb3740 10 client.20862 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.603+0000 7fa2f7eb3740 20 client.20862 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.603+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x562a85c12be0 0x562a85c15080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.603+0000 7fa2f7eb3740 4 client.20862 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.603+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/117681637 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a85b62860 con 0x562a85c12be0 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.603+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x562a85c12be0 0x562a85c15080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.603+0000 7fa2ceffd640 10 client.20862 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.604+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x562a85c12be0 0x562a85c15080 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=377b6b6431a234c3 server_cookie=48bdcce8a6b985eb in_seq=0 out_seq=0 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.604+0000 7fa2ceffd640 1 -- 172.21.15.89:0/117681637 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562a85b62860 con 0x562a85c12be0 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.604+0000 7fa2ceffd640 10 client.20862 handle_command_reply: tid=0 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.604+0000 7fa2f7eb3740 1 client.20862 shutdown 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/117681637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 msgr2=0x562a85c02540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85c02540 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fa2e009bc90 tx=0x7fa2e00a0350 comp rx=0 tx=0).stop 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/117681637 shutdown_connections 2024-09-15T23:39:49.635 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85c02b80 0x562a85bffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x562a85c12be0 0x562a85c15080 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85bf9350 0x562a85bff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/117681637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85c02540 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2ceffd640 0 client.20862 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/117681637 >> 172.21.15.89:0/117681637 conn(0x562a85bb8310 msgr2=0x562a85bfa140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/117681637 shutdown_connections 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/117681637 wait complete. 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.605+0000 7fa2f7eb3740 20 client.20862 trim_cache size 0 max 16384 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.618+0000 7fa2f7eb3740 1 Processor -- start 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2f7eb3740 1 -- start start 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85bff0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85bf9350 0x562a85beaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85c02b80 0x562a85beaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f01acc80 con 0x562a85bf7de0 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f01ae7c0 con 0x562a85bf9350 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f01ad350 con 0x562a85c02b80 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85bff0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2ef7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85bff0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55712/0 (socket says 172.21.15.89:55712) 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/1568980119 learned_addr learned my addr 172.21.15.89:0/1568980119 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/1568980119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85c02b80 msgr2=0x562a85beaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/1568980119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85c02b80 0x562a85beaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/1568980119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85bf9350 msgr2=0x562a85beaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/1568980119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85bf9350 0x562a85beaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2ef7fe640 1 -- 172.21.15.89:0/1568980119 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a85bfe7a0 con 0x562a85bf7de0 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/1568980119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85bff0b0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fa2e00cd730 tx=0x7fa2e0004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8920649c36db414b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2d57fa640 1 -- 172.21.15.89:0/1568980119 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e001d040 con 0x562a85bf7de0 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2d57fa640 1 -- 172.21.15.89:0/1568980119 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2e00a4860 con 0x562a85bf7de0 2024-09-15T23:39:49.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.619+0000 7fa2d57fa640 1 -- 172.21.15.89:0/1568980119 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e00b7030 con 0x562a85bf7de0 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.620+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/1568980119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 msgr2=0x562a85bff0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.620+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/1568980119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85bff0b0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fa2e00cd730 tx=0x7fa2e0004570 comp rx=0 tx=0).stop 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.620+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/1568980119 shutdown_connections 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.620+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/1568980119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85c02b80 0x562a85beaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.620+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/1568980119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85bf9350 0x562a85beaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.620+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/1568980119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf7de0 0x562a85bff0b0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.620+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/1568980119 >> 172.21.15.89:0/1568980119 conn(0x562a85bb8310 msgr2=0x562a85bea310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.620+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/1568980119 shutdown_connections 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.620+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/1568980119 wait complete. 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.620+0000 7fa2f7eb3740 1 -- start start 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2f7eb3740 1 Processor -- start 2024-09-15T23:39:49.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85bf7de0 0x562a85c0b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85c064b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2f7eb3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85c02b80 0x562a85c069f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f01ae0f0 con 0x562a85bf9350 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f01ada20 con 0x562a85bf7de0 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2f7eb3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f01aac60 con 0x562a85c02b80 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2f7eb3740 10 client.0 _ll_register_callbacks cb 0x7fa2e009bad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2eeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85c064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2eeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85c064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55716/0 (socket says 172.21.15.89:55716) 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2eeffd640 1 -- 172.21.15.89:0/97408409 learned_addr learned my addr 172.21.15.89:0/97408409 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85bf7de0 0x562a85c0b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2effff640 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85c02b80 0x562a85c069f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2eeffd640 1 -- 172.21.15.89:0/97408409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85c02b80 msgr2=0x562a85c069f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85c02b80 0x562a85c069f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2eeffd640 1 -- 172.21.15.89:0/97408409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85bf7de0 msgr2=0x562a85c0b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85bf7de0 0x562a85c0b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.621+0000 7fa2eeffd640 1 -- 172.21.15.89:0/97408409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a85c06fb0 con 0x562a85bf9350 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2eeffd640 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85c064b0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x562a85bf52d0 tx=0x7fa2e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c674e6712fbff35 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2ceffd640 1 -- 172.21.15.89:0/97408409 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4016070 con 0x562a85bf9350 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2ceffd640 1 -- 172.21.15.89:0/97408409 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa2e4005220 con 0x562a85bf9350 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/97408409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2e00a4860 con 0x562a85bf9350 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2ceffd640 1 -- 172.21.15.89:0/97408409 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa2e4033050 con 0x562a85bf9350 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2f7eb3740 10 client.20868 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:49.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/97408409 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562a85c7d7c0 con 0x562a85bf9350 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2ceffd640 1 -- 172.21.15.89:0/97408409 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2e00a4860 con 0x562a85bf9350 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2ceffd640 1 -- 172.21.15.89:0/97408409 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa2e4027070 con 0x562a85bf9350 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2f7eb3740 10 client.20868 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2f7eb3740 10 client.20868 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2f7eb3740 20 client.20868 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x562a85c7dfa0 0x562a85c803c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2f7eb3740 4 client.20868 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.622+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/97408409 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a85b5cd70 con 0x562a85c7dfa0 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.623+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x562a85c7dfa0 0x562a85c803c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.623+0000 7fa2ef7fe640 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x562a85c7dfa0 0x562a85c803c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a67213fffd387fec server_cookie=9d4ce7b752d04e70 in_seq=0 out_seq=0 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.623+0000 7fa2ceffd640 10 client.20868 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2ceffd640 1 -- 172.21.15.89:0/97408409 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a85b5cd70 con 0x562a85c7dfa0 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2ceffd640 10 client.20868 handle_command_reply: tid=0 2024-09-15T23:39:49.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2f7eb3740 1 client.20868 shutdown 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/97408409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 msgr2=0x562a85c064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85c064b0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x562a85bf52d0 tx=0x7fa2e40046c0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/97408409 shutdown_connections 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x562a85c7dfa0 0x562a85c803c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2ceffd640 0 client.20868 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562a85c02b80 0x562a85c069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a85bf9350 0x562a85c064b0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2f7eb3740 1 --2- 172.21.15.89:0/97408409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a85bf7de0 0x562a85c0b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.624+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/97408409 >> 172.21.15.89:0/97408409 conn(0x562a85bb8310 msgr2=0x562a85beab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/97408409 shutdown_connections 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f7eb3740 1 -- 172.21.15.89:0/97408409 wait complete. 2024-09-15T23:39:49.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f7eb3740 20 client.20868 trim_cache size 0 max 16384 2024-09-15T23:39:49.643 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:39:49.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f5ada640 1 -- 172.21.15.89:0/2128372973 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa2d80a3240 msgr2=0x7fa2d80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:49.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f5ada640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa2d80a3240 0x7fa2d80a5700 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0102420 tx=0x7fa2e4005f70 comp rx=0 tx=0).stop 2024-09-15T23:39:49.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f5ada640 1 -- 172.21.15.89:0/2128372973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f0102d30 msgr2=0x7fa2f0100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:49.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f5ada640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f0102d30 0x7fa2f0100990 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000c890 tx=0x7fa2e000cd50 comp rx=0 tx=0).stop 2024-09-15T23:39:49.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f5ada640 1 -- 172.21.15.89:0/2128372973 shutdown_connections 2024-09-15T23:39:49.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f5ada640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa2d80a3240 0x7fa2d80a5700 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f5ada640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f010c160 0x7fa2f0101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f5ada640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa2f0103650 0x7fa2f0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f5ada640 1 --2- 172.21.15.89:0/2128372973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2f0102d30 0x7fa2f0100990 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:49.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.625+0000 7fa2f5ada640 1 -- 172.21.15.89:0/2128372973 >> 172.21.15.89:0/2128372973 conn(0x7fa2f00fc7d0 msgr2=0x7fa2f00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:49.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.626+0000 7fa2f5ada640 1 -- 172.21.15.89:0/2128372973 shutdown_connections 2024-09-15T23:39:49.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:49.626+0000 7fa2f5ada640 1 -- 172.21.15.89:0/2128372973 wait complete. 2024-09-15T23:39:50.464 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:39:50.467 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:50.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.974+0000 7fa10e72a640 1 -- 172.21.15.89:0/1802117889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa108100480 msgr2=0x7fa108100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.974+0000 7fa10e72a640 1 --2- 172.21.15.89:0/1802117889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa108100480 0x7fa108100860 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009900 tx=0x7fa0f802a410 comp rx=0 tx=0).stop 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.974+0000 7fa10e72a640 1 -- 172.21.15.89:0/1802117889 shutdown_connections 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.974+0000 7fa10e72a640 1 --2- 172.21.15.89:0/1802117889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1081055b0 0x7fa10810d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.974+0000 7fa10e72a640 1 --2- 172.21.15.89:0/1802117889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa108100e30 0x7fa108104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.974+0000 7fa10e72a640 1 --2- 172.21.15.89:0/1802117889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa108100480 0x7fa108100860 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.974+0000 7fa10e72a640 1 -- 172.21.15.89:0/1802117889 >> 172.21.15.89:0/1802117889 conn(0x7fa108077210 msgr2=0x7fa1080fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.974+0000 7fa10e72a640 1 -- 172.21.15.89:0/1802117889 shutdown_connections 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.974+0000 7fa10e72a640 1 -- 172.21.15.89:0/1802117889 wait complete. 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10e72a640 1 Processor -- start 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10e72a640 1 -- start start 2024-09-15T23:39:50.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10e72a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa108100480 0x7fa108199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10e72a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa108100e30 0x7fa1081998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10e72a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1081055b0 0x7fa1081a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10e72a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa1081105b0 con 0x7fa1081055b0 2024-09-15T23:39:50.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10e72a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa108110430 con 0x7fa108100e30 2024-09-15T23:39:50.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10e72a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa108110730 con 0x7fa108100480 2024-09-15T23:39:50.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10ccc7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1081055b0 0x7fa1081a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10ccc7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1081055b0 0x7fa1081a43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55732/0 (socket says 172.21.15.89:55732) 2024-09-15T23:39:50.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.975+0000 7fa10ccc7640 1 -- 172.21.15.89:0/4046851677 learned_addr learned my addr 172.21.15.89:0/4046851677 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:50.983 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa107fff640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa108100480 0x7fa108199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa0fffff640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa108100e30 0x7fa1081998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa10ccc7640 1 -- 172.21.15.89:0/4046851677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa108100480 msgr2=0x7fa108199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa10ccc7640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa108100480 0x7fa108199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa10ccc7640 1 -- 172.21.15.89:0/4046851677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa108100e30 msgr2=0x7fa1081998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa10ccc7640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa108100e30 0x7fa1081998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa10ccc7640 1 -- 172.21.15.89:0/4046851677 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1081a4920 con 0x7fa1081055b0 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa0fffff640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa108100e30 0x7fa1081998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa10ccc7640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1081055b0 0x7fa1081a43e0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fa10000cad0 tx=0x7fa10000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa105ffb640 1 -- 172.21.15.89:0/4046851677 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10000ae20 con 0x7fa1081055b0 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa105ffb640 1 -- 172.21.15.89:0/4046851677 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa100005020 con 0x7fa1081055b0 2024-09-15T23:39:50.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa10e72a640 1 -- 172.21.15.89:0/4046851677 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1081a4af0 con 0x7fa1081055b0 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa105ffb640 1 -- 172.21.15.89:0/4046851677 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10000f6f0 con 0x7fa1081055b0 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.976+0000 7fa10e72a640 1 -- 172.21.15.89:0/4046851677 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1081a5000 con 0x7fa1081055b0 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.978+0000 7fa105ffb640 1 -- 172.21.15.89:0/4046851677 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa1000051d0 con 0x7fa1081055b0 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.978+0000 7fa110b03740 1 Processor -- start 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.978+0000 7fa110b03740 1 -- start start 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa110b03740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba694060 0x55adba694440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa110b03740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694980 0x55adba69ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa110b03740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba69f240 0x55adba6a1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa110b03740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55adba5f9620 con 0x55adba69f240 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa110b03740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55adba5a0e50 con 0x55adba694980 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa110b03740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55adba5f55e0 con 0x55adba694060 2024-09-15T23:39:50.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba69f240 0x55adba6a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba69f240 0x55adba6a1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55742/0 (socket says 172.21.15.89:55742) 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fffff640 1 -- 172.21.15.89:0/3842450560 learned_addr learned my addr 172.21.15.89:0/3842450560 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fffff640 1 --2- 172.21.15.89:0/3842450560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba694060 0x55adba694440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa105ffb640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa0f40a32c0 0x7fa0f40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa107fff640 1 --2- 172.21.15.89:0/3842450560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694980 0x55adba69ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa10ccc7640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa0f40a32c0 0x7fa0f40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fffff640 1 -- 172.21.15.89:0/3842450560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba694060 msgr2=0x55adba694440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fffff640 1 --2- 172.21.15.89:0/3842450560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba694060 0x55adba694440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fffff640 1 -- 172.21.15.89:0/3842450560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694980 msgr2=0x55adba69ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fffff640 1 --2- 172.21.15.89:0/3842450560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694980 0x55adba69ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fffff640 1 -- 172.21.15.89:0/3842450560 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adba591d40 con 0x55adba69f240 2024-09-15T23:39:50.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa105ffb640 1 -- 172.21.15.89:0/4046851677 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa1000cfd50 con 0x7fa1081055b0 2024-09-15T23:39:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fffff640 1 --2- 172.21.15.89:0/3842450560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba69f240 0x55adba6a1630 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0013890 tx=0x7fa0f000b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d036cc623bceed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.979+0000 7fa0fe7fc640 1 -- 172.21.15.89:0/3842450560 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000e8f0 con 0x55adba69f240 2024-09-15T23:39:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa0fe7fc640 1 -- 172.21.15.89:0/3842450560 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f00113e0 con 0x55adba69f240 2024-09-15T23:39:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa0fe7fc640 1 -- 172.21.15.89:0/3842450560 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0010630 con 0x55adba69f240 2024-09-15T23:39:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa110b03740 1 -- 172.21.15.89:0/3842450560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba69f240 msgr2=0x55adba6a1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa10ccc7640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa0f40a32c0 0x7fa0f40a5780 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa10009bc80 tx=0x7fa100066910 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa110b03740 1 --2- 172.21.15.89:0/3842450560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba69f240 0x55adba6a1630 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0013890 tx=0x7fa0f000b420 comp rx=0 tx=0).stop 2024-09-15T23:39:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa110b03740 1 -- 172.21.15.89:0/3842450560 shutdown_connections 2024-09-15T23:39:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa110b03740 1 --2- 172.21.15.89:0/3842450560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba69f240 0x55adba6a1630 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa110b03740 1 --2- 172.21.15.89:0/3842450560 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694980 0x55adba69ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa110b03740 1 --2- 172.21.15.89:0/3842450560 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba694060 0x55adba694440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa110b03740 1 -- 172.21.15.89:0/3842450560 >> 172.21.15.89:0/3842450560 conn(0x55adba6905f0 msgr2=0x55adba6909d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:50.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa110b03740 1 -- 172.21.15.89:0/3842450560 shutdown_connections 2024-09-15T23:39:50.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.980+0000 7fa110b03740 1 -- 172.21.15.89:0/3842450560 wait complete. 2024-09-15T23:39:50.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.981+0000 7fa110b03740 1 -- start start 2024-09-15T23:39:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.981+0000 7fa110b03740 1 Processor -- start 2024-09-15T23:39:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.981+0000 7fa110b03740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694060 0x55adba69c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.981+0000 7fa110b03740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba69c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.981+0000 7fa110b03740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba69cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.981+0000 7fa110b03740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55adba6a1e10 con 0x55adba694980 2024-09-15T23:39:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.981+0000 7fa110b03740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55adba581f10 con 0x55adba694060 2024-09-15T23:39:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.981+0000 7fa110b03740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55adba6a1f90 con 0x55adba69f240 2024-09-15T23:39:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.981+0000 7fa110b03740 10 client.0 _ll_register_callbacks cb 0x55adba5a5db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:50.991 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa0fffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba69cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa0fffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba69cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:36306/0 (socket says 172.21.15.89:36306) 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa0fffff640 1 -- 172.21.15.89:0/4023641157 learned_addr learned my addr 172.21.15.89:0/4023641157 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa107fff640 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba69c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa0fffff640 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694060 0x55adba69c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa0fffff640 1 -- 172.21.15.89:0/4023641157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694060 msgr2=0x55adba69c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa0fffff640 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694060 0x55adba69c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa0fffff640 1 -- 172.21.15.89:0/4023641157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 msgr2=0x55adba69c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa0fffff640 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba69c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa0fffff640 1 -- 172.21.15.89:0/4023641157 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adba308580 con 0x55adba69f240 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.982+0000 7fa107fff640 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba69c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa0fffff640 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba69cc60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0001eb0 tx=0x7fa0f0011e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68572a3ac3cfd7ae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa0e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa0e77fe640 1 -- 172.21.15.89:0/4023641157 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0022070 con 0x55adba69f240 2024-09-15T23:39:50.992 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa110b03740 1 -- 172.21.15.89:0/4023641157 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0f00113e0 con 0x55adba69f240 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa0e77fe640 1 -- 172.21.15.89:0/4023641157 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa0f0011590 con 0x55adba69f240 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa0e77fe640 1 -- 172.21.15.89:0/4023641157 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa0f0011790 con 0x55adba69f240 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa110b03740 10 client.28723 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa110b03740 1 -- 172.21.15.89:0/4023641157 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55adba3d2960 con 0x55adba69f240 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa0e77fe640 1 -- 172.21.15.89:0/4023641157 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f00113e0 con 0x55adba69f240 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa0e77fe640 1 -- 172.21.15.89:0/4023641157 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa0f000f040 con 0x55adba69f240 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa110b03740 10 client.28723 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.983+0000 7fa110b03740 10 client.28723 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.984+0000 7fa110b03740 20 client.28723 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.984+0000 7fa110b03740 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55adba6ad0c0 0x55adba6b9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.984+0000 7fa110b03740 4 client.28723 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.984+0000 7fa110b03740 1 -- 172.21.15.89:0/4023641157 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adba606b90 con 0x55adba6ad0c0 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.984+0000 7fa107fff640 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55adba6ad0c0 0x55adba6b9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:50.993 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.984+0000 7fa0e77fe640 10 client.28723 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.984+0000 7fa107fff640 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55adba6ad0c0 0x55adba6b9f60 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9eacf43c5f33616 server_cookie=e2eec798492c34d9 in_seq=0 out_seq=0 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa0e77fe640 1 -- 172.21.15.89:0/4023641157 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55adba606b90 con 0x55adba6ad0c0 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa0e77fe640 10 client.28723 handle_command_reply: tid=0 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa110b03740 1 client.28723 shutdown 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa110b03740 1 -- 172.21.15.89:0/4023641157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 msgr2=0x55adba69cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa110b03740 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba69cc60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0001eb0 tx=0x7fa0f0011e10 comp rx=0 tx=0).stop 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa110b03740 1 -- 172.21.15.89:0/4023641157 shutdown_connections 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa110b03740 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55adba6ad0c0 0x55adba6b9f60 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa110b03740 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba69cc60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa0e77fe640 0 client.28723 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa110b03740 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba69c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa110b03740 1 --2- 172.21.15.89:0/4023641157 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694060 0x55adba69c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.985+0000 7fa110b03740 1 -- 172.21.15.89:0/4023641157 >> 172.21.15.89:0/4023641157 conn(0x55adba6905f0 msgr2=0x55adba6962d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:50.994 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.986+0000 7fa110b03740 1 -- 172.21.15.89:0/4023641157 shutdown_connections 2024-09-15T23:39:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.986+0000 7fa110b03740 1 -- 172.21.15.89:0/4023641157 wait complete. 2024-09-15T23:39:50.995 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:50.986+0000 7fa110b03740 20 client.28723 trim_cache size 0 max 16384 2024-09-15T23:39:51.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa110b03740 1 Processor -- start 2024-09-15T23:39:51.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa110b03740 1 -- start start 2024-09-15T23:39:51.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa110b03740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694060 0x55adba6bf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:51.004 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa110b03740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba6bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:51.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa110b03740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba699480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:51.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa110b03740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa1081acce0 con 0x55adba694980 2024-09-15T23:39:51.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa110b03740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa1081ae820 con 0x55adba694060 2024-09-15T23:39:51.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa110b03740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa1081ad3b0 con 0x55adba69f240 2024-09-15T23:39:51.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa107fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba6bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:51.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa107fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba6bfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55768/0 (socket says 172.21.15.89:55768) 2024-09-15T23:39:51.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa107fff640 1 -- 172.21.15.89:0/3886429977 learned_addr learned my addr 172.21.15.89:0/3886429977 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:51.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa0fffff640 1 --2- 172.21.15.89:0/3886429977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba699480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:51.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.003+0000 7fa0fffff640 1 --2- 172.21.15.89:0/3886429977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694060 0x55adba6bf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:51.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.004+0000 7fa107fff640 1 -- 172.21.15.89:0/3886429977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 msgr2=0x55adba699480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:51.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.004+0000 7fa107fff640 1 --2- 172.21.15.89:0/3886429977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba699480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.004+0000 7fa107fff640 1 -- 172.21.15.89:0/3886429977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694060 msgr2=0x55adba6bf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:51.007 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.004+0000 7fa107fff640 1 --2- 172.21.15.89:0/3886429977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694060 0x55adba6bf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.004+0000 7fa107fff640 1 -- 172.21.15.89:0/3886429977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adba693e90 con 0x55adba694980 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.004+0000 7fa107fff640 1 --2- 172.21.15.89:0/3886429977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba6bfbc0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8005ab0 tx=0x7fa0f80058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee3ff2bd97f1e4f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.004+0000 7fa0fdffb640 1 -- 172.21.15.89:0/3886429977 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8035040 con 0x55adba694980 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.004+0000 7fa0fdffb640 1 -- 172.21.15.89:0/3886429977 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f8049070 con 0x55adba694980 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.004+0000 7fa0fdffb640 1 -- 172.21.15.89:0/3886429977 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80049b0 con 0x55adba694980 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.005+0000 7fa110b03740 1 -- 172.21.15.89:0/3886429977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 msgr2=0x55adba6bfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.005+0000 7fa110b03740 1 --2- 172.21.15.89:0/3886429977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba6bfbc0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8005ab0 tx=0x7fa0f80058b0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.005+0000 7fa110b03740 1 -- 172.21.15.89:0/3886429977 shutdown_connections 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.005+0000 7fa110b03740 1 --2- 172.21.15.89:0/3886429977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba699480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.005+0000 7fa110b03740 1 --2- 172.21.15.89:0/3886429977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694980 0x55adba6bfbc0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.008 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.005+0000 7fa110b03740 1 --2- 172.21.15.89:0/3886429977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694060 0x55adba6bf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.005+0000 7fa110b03740 1 -- 172.21.15.89:0/3886429977 >> 172.21.15.89:0/3886429977 conn(0x55adba6905f0 msgr2=0x55adba69da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:51.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.005+0000 7fa110b03740 1 -- 172.21.15.89:0/3886429977 shutdown_connections 2024-09-15T23:39:51.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.005+0000 7fa110b03740 1 -- 172.21.15.89:0/3886429977 wait complete. 2024-09-15T23:39:51.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.006+0000 7fa110b03740 1 -- start start 2024-09-15T23:39:51.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.006+0000 7fa110b03740 1 Processor -- start 2024-09-15T23:39:51.009 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.006+0000 7fa110b03740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694060 0x55adba6a4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:51.010 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa110b03740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694980 0x55adba6a4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa110b03740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba6a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa110b03740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa1081ae150 con 0x55adba694060 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa110b03740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa1081ada80 con 0x55adba694980 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa110b03740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa1081aacc0 con 0x55adba69f240 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa110b03740 10 client.0 _ll_register_callbacks cb 0x7fa0f0005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694060 0x55adba6a4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0fffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694060 0x55adba6a4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55784/0 (socket says 172.21.15.89:55784) 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0fffff640 1 -- 172.21.15.89:0/2012405399 learned_addr learned my addr 172.21.15.89:0/2012405399 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0fffff640 1 -- 172.21.15.89:0/2012405399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 msgr2=0x55adba6a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0fffff640 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba6a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0fffff640 1 -- 172.21.15.89:0/2012405399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694980 msgr2=0x55adba6a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0fffff640 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694980 0x55adba6a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0fffff640 1 -- 172.21.15.89:0/2012405399 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adba721850 con 0x55adba694060 2024-09-15T23:39:51.011 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0fffff640 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694060 0x55adba6a4320 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000be10 tx=0x7fa0f000baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89589a060aabb04 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0e77fe640 1 -- 172.21.15.89:0/2012405399 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f003d020 con 0x55adba694060 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa0e77fe640 1 -- 172.21.15.89:0/2012405399 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa0f001ebb0 con 0x55adba694060 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.007+0000 7fa110b03740 1 -- 172.21.15.89:0/2012405399 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0f8049070 con 0x55adba694060 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa0e77fe640 1 -- 172.21.15.89:0/2012405399 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa0f0022040 con 0x55adba694060 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa110b03740 10 client.20886 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa110b03740 1 -- 172.21.15.89:0/2012405399 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55adba722b70 con 0x55adba694060 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa0e77fe640 1 -- 172.21.15.89:0/2012405399 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f8049070 con 0x55adba694060 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa0e77fe640 1 -- 172.21.15.89:0/2012405399 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa0f0005280 con 0x55adba694060 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa110b03740 10 client.20886 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa110b03740 10 client.20886 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:39:51.012 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa110b03740 20 client.20886 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:51.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa110b03740 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55adba723290 0x55adba7256f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:51.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa110b03740 4 client.20886 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "58c1329f", "release": true, "await": true}] 2024-09-15T23:39:51.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa110b03740 1 -- 172.21.15.89:0/2012405399 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "58c1329f", "release": true, "await": true}) -- 0x55adba6010a0 con 0x55adba723290 2024-09-15T23:39:51.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.008+0000 7fa107fff640 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55adba723290 0x55adba7256f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:51.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.009+0000 7fa0e77fe640 10 client.20886 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:51.013 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.009+0000 7fa107fff640 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55adba723290 0x55adba7256f0 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbb3d47dc7bbbabe server_cookie=779440086da09edc in_seq=0 out_seq=0 2024-09-15T23:39:51.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa0e77fe640 1 -- 172.21.15.89:0/2012405399 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55adba6010a0 con 0x55adba723290 2024-09-15T23:39:51.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa0e77fe640 10 client.20886 handle_command_reply: tid=0 2024-09-15T23:39:51.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 client.20886 shutdown 2024-09-15T23:39:51.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 -- 172.21.15.89:0/2012405399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694060 msgr2=0x55adba6a4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:51.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694060 0x55adba6a4320 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000be10 tx=0x7fa0f000baa0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 -- 172.21.15.89:0/2012405399 shutdown_connections 2024-09-15T23:39:51.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55adba723290 0x55adba7256f0 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55adba69f240 0x55adba6a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55adba694980 0x55adba6a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.014 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 --2- 172.21.15.89:0/2012405399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55adba694060 0x55adba6a4320 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 -- 172.21.15.89:0/2012405399 >> 172.21.15.89:0/2012405399 conn(0x55adba6905f0 msgr2=0x55adba695a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:51.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa0e77fe640 0 client.20886 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:51.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 -- 172.21.15.89:0/2012405399 shutdown_connections 2024-09-15T23:39:51.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 1 -- 172.21.15.89:0/2012405399 wait complete. 2024-09-15T23:39:51.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.012+0000 7fa110b03740 20 client.20886 trim_cache size 0 max 16384 2024-09-15T23:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:39:51.015 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 56, 2024-09-15T23:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "58c1329f": { 2024-09-15T23:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 56, 2024-09-15T23:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:39:51.016 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 137.7, 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:51.017 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:51.018 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:39:51.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 -- 172.21.15.89:0/4046851677 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa0f40a32c0 msgr2=0x7fa0f40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:51.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa0f40a32c0 0x7fa0f40a5780 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa10009bc80 tx=0x7fa100066910 comp rx=0 tx=0).stop 2024-09-15T23:39:51.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 -- 172.21.15.89:0/4046851677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1081055b0 msgr2=0x7fa1081a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:51.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1081055b0 0x7fa1081a43e0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fa10000cad0 tx=0x7fa10000cf90 comp rx=0 tx=0).stop 2024-09-15T23:39:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 -- 172.21.15.89:0/4046851677 shutdown_connections 2024-09-15T23:39:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa0f40a32c0 0x7fa0f40a5780 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1081055b0 0x7fa1081a43e0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa108100e30 0x7fa1081998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 --2- 172.21.15.89:0/4046851677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa108100480 0x7fa108199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 -- 172.21.15.89:0/4046851677 >> 172.21.15.89:0/4046851677 conn(0x7fa108077210 msgr2=0x7fa1080fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 -- 172.21.15.89:0/4046851677 shutdown_connections 2024-09-15T23:39:51.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:51.013+0000 7fa10e72a640 1 -- 172.21.15.89:0/4046851677 wait complete. 2024-09-15T23:39:51.021 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:51.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:51 smithi089 ceph-mon[30933]: pgmap v824: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-15T23:39:51.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:51 smithi120 ceph-mon[35250]: pgmap v824: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-15T23:39:51.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:51 smithi174 ceph-mon[35586]: pgmap v824: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-09-15T23:39:51.782 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '58c1329f', total seconds elapsed: 22.3 2024-09-15T23:39:51.782 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.7 2024-09-15T23:39:51.783 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:39:52.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.289+0000 7f9eb9223640 1 -- 172.21.15.89:0/1805720595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb4102570 msgr2=0x7f9eb41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:52.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.289+0000 7f9eb9223640 1 --2- 172.21.15.89:0/1805720595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb4102570 0x7f9eb41029b0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8009900 tx=0x7f9ea802a480 comp rx=0 tx=0).stop 2024-09-15T23:39:52.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.289+0000 7f9eb9223640 1 -- 172.21.15.89:0/1805720595 shutdown_connections 2024-09-15T23:39:52.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.289+0000 7f9eb9223640 1 --2- 172.21.15.89:0/1805720595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eb4102ef0 0x7f9eb410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.289+0000 7f9eb9223640 1 --2- 172.21.15.89:0/1805720595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb4102570 0x7f9eb41029b0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.289+0000 7f9eb9223640 1 --2- 172.21.15.89:0/1805720595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9eb4100e20 0x7f9eb4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.289+0000 7f9eb9223640 1 -- 172.21.15.89:0/1805720595 >> 172.21.15.89:0/1805720595 conn(0x7f9eb40fc7d0 msgr2=0x7f9eb40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.289+0000 7f9eb9223640 1 -- 172.21.15.89:0/1805720595 shutdown_connections 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.289+0000 7f9eb9223640 1 -- 172.21.15.89:0/1805720595 wait complete. 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.290+0000 7f9eb9223640 1 Processor -- start 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.290+0000 7f9eb9223640 1 -- start start 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb9223640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eb4100e20 0x7f9eb41993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb9223640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb4102570 0x7f9eb4199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb9223640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9eb4102ef0 0x7f9eb41a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb9223640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4110fc0 con 0x7f9eb4102570 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb9223640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4110e40 con 0x7f9eb4102ef0 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb9223640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4111140 con 0x7f9eb4100e20 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb4102570 0x7f9eb4199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb4102570 0x7f9eb4199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55796/0 (socket says 172.21.15.89:55796) 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb259c640 1 -- 172.21.15.89:0/1481645284 learned_addr learned my addr 172.21.15.89:0/1481645284 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb2d9d640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eb4100e20 0x7f9eb41993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb359e640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9eb4102ef0 0x7f9eb41a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb259c640 1 -- 172.21.15.89:0/1481645284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eb4100e20 msgr2=0x7f9eb41993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb259c640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eb4100e20 0x7f9eb41993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb259c640 1 -- 172.21.15.89:0/1481645284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9eb4102ef0 msgr2=0x7f9eb41a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb259c640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9eb4102ef0 0x7f9eb41a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb259c640 1 -- 172.21.15.89:0/1481645284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb41a49b0 con 0x7f9eb4102570 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb359e640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9eb4102ef0 0x7f9eb41a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb2d9d640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eb4100e20 0x7f9eb41993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb259c640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb4102570 0x7f9eb4199930 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8006010 tx=0x7f9ea8003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:52.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9e9bfff640 1 -- 172.21.15.89:0/1481645284 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea803f070 con 0x7f9eb4102570 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9eb9223640 1 -- 172.21.15.89:0/1481645284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb41a4b80 con 0x7f9eb4102570 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9e9bfff640 1 -- 172.21.15.89:0/1481645284 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ea8031e30 con 0x7f9eb4102570 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.291+0000 7f9e9bfff640 1 -- 172.21.15.89:0/1481645284 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea8035030 con 0x7f9eb4102570 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.292+0000 7f9eb9223640 1 -- 172.21.15.89:0/1481645284 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb41a5030 con 0x7f9eb4102570 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.294+0000 7f9e9bfff640 1 -- 172.21.15.89:0/1481645284 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f9ea8042070 con 0x7f9eb4102570 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.294+0000 7f9ebb5fc740 1 Processor -- start 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.294+0000 7f9ebb5fc740 1 -- start start 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458fb40 0x5600d458e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458e5d0 0x5600d458ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d4599370 0x5600d459b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5600d44f42f0 con 0x5600d458fb40 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5600d449bcf0 con 0x5600d4599370 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5600d44f02b0 con 0x5600d458e5d0 2024-09-15T23:39:52.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458fb40 0x5600d458e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458fb40 0x5600d458e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55812/0 (socket says 172.21.15.89:55812) 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb359e640 1 -- 172.21.15.89:0/189450665 learned_addr learned my addr 172.21.15.89:0/189450665 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb2d9d640 1 --2- 172.21.15.89:0/189450665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458e5d0 0x5600d458ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9e9bfff640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e9c0a3240 0x7f9e9c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb359e640 1 -- 172.21.15.89:0/189450665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458e5d0 msgr2=0x5600d458ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb359e640 1 --2- 172.21.15.89:0/189450665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458e5d0 0x5600d458ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb359e640 1 -- 172.21.15.89:0/189450665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d4599370 msgr2=0x5600d459b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb359e640 1 --2- 172.21.15.89:0/189450665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d4599370 0x5600d459b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb359e640 1 -- 172.21.15.89:0/189450665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600d44803a0 con 0x5600d458fb40 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9e9bfff640 1 -- 172.21.15.89:0/1481645284 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9ea80f0700 con 0x7f9eb4102570 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb359e640 1 --2- 172.21.15.89:0/189450665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458fb40 0x5600d458e090 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f9eac008dd0 tx=0x7f9eac00d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c1e4cc582443c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9eb359e640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e9c0a3240 0x7f9e9c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.295+0000 7f9e997fa640 1 -- 172.21.15.89:0/189450665 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac00f830 con 0x5600d458fb40 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9e997fa640 1 -- 172.21.15.89:0/189450665 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac018500 con 0x5600d458fb40 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9e997fa640 1 -- 172.21.15.89:0/189450665 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac0178b0 con 0x5600d458fb40 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/189450665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458fb40 msgr2=0x5600d458e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/189450665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458fb40 0x5600d458e090 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f9eac008dd0 tx=0x7f9eac00d710 comp rx=0 tx=0).stop 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9eb359e640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e9c0a3240 0x7f9e9c0a5700 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9eac017000 tx=0x7f9eac017f80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/189450665 shutdown_connections 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/189450665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d4599370 0x5600d459b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/189450665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458e5d0 0x5600d458ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/189450665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458fb40 0x5600d458e090 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/189450665 >> 172.21.15.89:0/189450665 conn(0x5600d4557310 msgr2=0x5600d458ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/189450665 shutdown_connections 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.296+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/189450665 wait complete. 2024-09-15T23:39:52.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9ebb5fc740 1 -- start start 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9ebb5fc740 1 Processor -- start 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458e5d0 0x5600d4598d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 0x5600d4596010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d4599370 0x5600d4596600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5600d459c070 con 0x5600d458e5d0 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5600d44af060 con 0x5600d458fb40 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5600d459c1f0 con 0x5600d4599370 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9ebb5fc740 10 client.0 _ll_register_callbacks cb 0x5600d42028c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458e5d0 0x5600d4598d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458e5d0 0x5600d4598d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55822/0 (socket says 172.21.15.89:55822) 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb2d9d640 1 -- 172.21.15.89:0/2059881379 learned_addr learned my addr 172.21.15.89:0/2059881379 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb359e640 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d4599370 0x5600d4596600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb259c640 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 0x5600d4596010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb2d9d640 1 -- 172.21.15.89:0/2059881379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d4599370 msgr2=0x5600d4596600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb2d9d640 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d4599370 0x5600d4596600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb2d9d640 1 -- 172.21.15.89:0/2059881379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 msgr2=0x5600d4596010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb2d9d640 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 0x5600d4596010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb2d9d640 1 -- 172.21.15.89:0/2059881379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600d42653c0 con 0x5600d458e5d0 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.297+0000 7f9eb259c640 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 0x5600d4596010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9eb2d9d640 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458e5d0 0x5600d4598d60 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400bfb0 tx=0x7f9ea400f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe87ac8280b99ee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9e96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9e96ff5640 1 -- 172.21.15.89:0/2059881379 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4018020 con 0x5600d458e5d0 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9e96ff5640 1 -- 172.21.15.89:0/2059881379 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9ea4014870 con 0x5600d458e5d0 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/2059881379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eac018500 con 0x5600d458e5d0 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9e96ff5640 1 -- 172.21.15.89:0/2059881379 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9ea4023050 con 0x5600d458e5d0 2024-09-15T23:39:52.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9ebb5fc740 10 client.20898 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/2059881379 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5600d45a7ba0 con 0x5600d458e5d0 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9e96ff5640 1 -- 172.21.15.89:0/2059881379 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac018500 con 0x5600d458e5d0 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9e96ff5640 1 -- 172.21.15.89:0/2059881379 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9ea4013b10 con 0x5600d458e5d0 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9ebb5fc740 10 client.20898 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9ebb5fc740 10 client.20898 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9ebb5fc740 20 client.20898 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5600d45a9060 0x5600d45ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9ebb5fc740 4 client.20898 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.298+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/2059881379 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600d4501860 con 0x5600d45a9060 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.299+0000 7f9eb259c640 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5600d45a9060 0x5600d45ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.299+0000 7f9eb259c640 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5600d45a9060 0x5600d45ab4c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e942fc1ab488249 server_cookie=f43d9d1aa137c5d3 in_seq=0 out_seq=0 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.299+0000 7f9e96ff5640 10 client.20898 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.300+0000 7f9e96ff5640 1 -- 172.21.15.89:0/2059881379 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5600d4501860 con 0x5600d45a9060 2024-09-15T23:39:52.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.300+0000 7f9e96ff5640 10 client.20898 handle_command_reply: tid=0 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.300+0000 7f9ebb5fc740 1 client.20898 shutdown 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/2059881379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458e5d0 msgr2=0x5600d4598d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458e5d0 0x5600d4598d60 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f9ea400bfb0 tx=0x7f9ea400f670 comp rx=0 tx=0).stop 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/2059881379 shutdown_connections 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5600d45a9060 0x5600d45ab4c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d4599370 0x5600d4596600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 0x5600d4596010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/2059881379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d458e5d0 0x5600d4598d60 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/2059881379 >> 172.21.15.89:0/2059881379 conn(0x5600d4557310 msgr2=0x5600d4590930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9e96ff5640 0 client.20898 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/2059881379 shutdown_connections 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/2059881379 wait complete. 2024-09-15T23:39:52.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.301+0000 7f9ebb5fc740 20 client.20898 trim_cache size 0 max 16384 2024-09-15T23:39:52.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9ebb5fc740 1 Processor -- start 2024-09-15T23:39:52.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9ebb5fc740 1 -- start start 2024-09-15T23:39:52.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458e5d0 0x5600d4594f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458fb40 0x5600d4589c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d458a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41aeec0 con 0x5600d4599370 2024-09-15T23:39:52.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41b0a00 con 0x5600d458e5d0 2024-09-15T23:39:52.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41af590 con 0x5600d458fb40 2024-09-15T23:39:52.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9eb359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d458a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9eb359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d458a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55824/0 (socket says 172.21.15.89:55824) 2024-09-15T23:39:52.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9eb359e640 1 -- 172.21.15.89:0/1184624374 learned_addr learned my addr 172.21.15.89:0/1184624374 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:52.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9eb259c640 1 --2- 172.21.15.89:0/1184624374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458fb40 0x5600d4589c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.316+0000 7f9eb2d9d640 1 --2- 172.21.15.89:0/1184624374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458e5d0 0x5600d4594f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.317+0000 7f9eb359e640 1 -- 172.21.15.89:0/1184624374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458fb40 msgr2=0x5600d4589c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:52.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.317+0000 7f9eb359e640 1 --2- 172.21.15.89:0/1184624374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458fb40 0x5600d4589c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.317+0000 7f9eb359e640 1 -- 172.21.15.89:0/1184624374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458e5d0 msgr2=0x5600d4594f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:52.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.317+0000 7f9eb359e640 1 --2- 172.21.15.89:0/1184624374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458e5d0 0x5600d4594f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.317+0000 7f9eb359e640 1 -- 172.21.15.89:0/1184624374 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600d4596970 con 0x5600d4599370 2024-09-15T23:39:52.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.317+0000 7f9eb259c640 1 --2- 172.21.15.89:0/1184624374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458fb40 0x5600d4589c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:52.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.317+0000 7f9eb359e640 1 --2- 172.21.15.89:0/1184624374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d458a150 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f9eac008dd0 tx=0x7f9eac024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e37dc01b1bcba5b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:52.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.317+0000 7f9e98ff9640 1 -- 172.21.15.89:0/1184624374 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac00de20 con 0x5600d4599370 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.317+0000 7f9e98ff9640 1 -- 172.21.15.89:0/1184624374 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac008920 con 0x5600d4599370 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.317+0000 7f9e98ff9640 1 -- 172.21.15.89:0/1184624374 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac0173e0 con 0x5600d4599370 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/1184624374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 msgr2=0x5600d458a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/1184624374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d458a150 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f9eac008dd0 tx=0x7f9eac024000 comp rx=0 tx=0).stop 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/1184624374 shutdown_connections 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/1184624374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d458a150 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/1184624374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458fb40 0x5600d4589c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/1184624374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458e5d0 0x5600d4594f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/1184624374 >> 172.21.15.89:0/1184624374 conn(0x5600d4557310 msgr2=0x5600d4589500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/1184624374 shutdown_connections 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/1184624374 wait complete. 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 -- start start 2024-09-15T23:39:52.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.318+0000 7f9ebb5fc740 1 Processor -- start 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458e5d0 0x5600d45a1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 0x5600d45a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9ebb5fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d459cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41b0330 con 0x5600d4599370 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41afc60 con 0x5600d458fb40 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9ebb5fc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41acea0 con 0x5600d458e5d0 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9ebb5fc740 10 client.0 _ll_register_callbacks cb 0x7f9ea400b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d459cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d459cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55836/0 (socket says 172.21.15.89:55836) 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb359e640 1 -- 172.21.15.89:0/516053909 learned_addr learned my addr 172.21.15.89:0/516053909 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb2d9d640 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458e5d0 0x5600d45a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb259c640 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 0x5600d45a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb359e640 1 -- 172.21.15.89:0/516053909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458e5d0 msgr2=0x5600d45a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb359e640 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458e5d0 0x5600d45a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb359e640 1 -- 172.21.15.89:0/516053909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 msgr2=0x5600d45a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb359e640 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 0x5600d45a1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb359e640 1 -- 172.21.15.89:0/516053909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600d459d5f0 con 0x5600d4599370 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9eb359e640 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d459cdf0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f9eac008250 tx=0x7f9eac00ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8041fbb8d6e47bef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.319+0000 7f9e96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9e96ff5640 1 -- 172.21.15.89:0/516053909 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac03d070 con 0x5600d4599370 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9e96ff5640 1 -- 172.21.15.89:0/516053909 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f9eac00fcc0 con 0x5600d4599370 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/516053909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eac008920 con 0x5600d4599370 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9e96ff5640 1 -- 172.21.15.89:0/516053909 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9eac049050 con 0x5600d4599370 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9ebb5fc740 10 client.20904 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/516053909 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5600d4613a30 con 0x5600d4599370 2024-09-15T23:39:52.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9e96ff5640 1 -- 172.21.15.89:0/516053909 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac008920 con 0x5600d4599370 2024-09-15T23:39:52.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9e96ff5640 1 -- 172.21.15.89:0/516053909 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9eac018970 con 0x5600d4599370 2024-09-15T23:39:52.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9ebb5fc740 10 client.20904 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:52.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9ebb5fc740 10 client.20904 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:39:52.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9ebb5fc740 20 client.20904 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:52.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5600d4613f50 0x5600d4616340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:52.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9ebb5fc740 4 client.20904 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:39:52.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.320+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/516053909 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5600d44fbd70 con 0x5600d4613f50 2024-09-15T23:39:52.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.321+0000 7f9eb2d9d640 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5600d4613f50 0x5600d4616340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:52.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.321+0000 7f9e96ff5640 10 client.20904 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:39:52.326 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.321+0000 7f9eb2d9d640 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5600d4613f50 0x5600d4616340 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=54ab6c84b216cae5 server_cookie=b8385fdb995bd21a in_seq=0 out_seq=0 2024-09-15T23:39:52.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.321+0000 7f9e96ff5640 1 -- 172.21.15.89:0/516053909 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5600d44fbd70 con 0x5600d4613f50 2024-09-15T23:39:52.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.321+0000 7f9e96ff5640 10 client.20904 handle_command_reply: tid=0 2024-09-15T23:39:52.327 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.321+0000 7f9ebb5fc740 1 client.20904 shutdown 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.321+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/516053909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 msgr2=0x5600d459cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.321+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d459cdf0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f9eac008250 tx=0x7f9eac00ead0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/516053909 shutdown_connections 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x5600d4613f50 0x5600d4616340 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9e96ff5640 0 client.20904 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600d4599370 0x5600d459cdf0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5600d458fb40 0x5600d45a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9ebb5fc740 1 --2- 172.21.15.89:0/516053909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600d458e5d0 0x5600d45a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/516053909 >> 172.21.15.89:0/516053909 conn(0x5600d4557310 msgr2=0x5600d4589d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/516053909 shutdown_connections 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9ebb5fc740 1 -- 172.21.15.89:0/516053909 wait complete. 2024-09-15T23:39:52.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9ebb5fc740 20 client.20904 trim_cache size 0 max 16384 2024-09-15T23:39:52.329 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:39:52.329 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9eb9223640 1 -- 172.21.15.89:0/1481645284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e9c0a3240 msgr2=0x7f9e9c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:52.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.322+0000 7f9eb9223640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e9c0a3240 0x7f9e9c0a5700 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9eac017000 tx=0x7f9eac017f80 comp rx=0 tx=0).stop 2024-09-15T23:39:52.330 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.323+0000 7f9eb9223640 1 -- 172.21.15.89:0/1481645284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb4102570 msgr2=0x7f9eb4199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:52.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.323+0000 7f9eb9223640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb4102570 0x7f9eb4199930 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8006010 tx=0x7f9ea8003c80 comp rx=0 tx=0).stop 2024-09-15T23:39:52.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.323+0000 7f9eb9223640 1 -- 172.21.15.89:0/1481645284 shutdown_connections 2024-09-15T23:39:52.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.323+0000 7f9eb9223640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f9e9c0a3240 0x7f9e9c0a5700 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.332 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.323+0000 7f9eb9223640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9eb4102ef0 0x7f9eb41a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.323+0000 7f9eb9223640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9eb4102570 0x7f9eb4199930 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.323+0000 7f9eb9223640 1 --2- 172.21.15.89:0/1481645284 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eb4100e20 0x7f9eb41993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:52.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.323+0000 7f9eb9223640 1 -- 172.21.15.89:0/1481645284 >> 172.21.15.89:0/1481645284 conn(0x7f9eb40fc7d0 msgr2=0x7f9eb40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:52.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.323+0000 7f9eb9223640 1 -- 172.21.15.89:0/1481645284 shutdown_connections 2024-09-15T23:39:52.333 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:52.323+0000 7f9eb9223640 1 -- 172.21.15.89:0/1481645284 wait complete. 2024-09-15T23:39:52.572 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:53.044 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:39:53.370 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:53 smithi089 ceph-mon[30933]: pgmap v825: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-15T23:39:53.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:53 smithi120 ceph-mon[35250]: pgmap v825: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-15T23:39:53.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.448+0000 7fc517ec6640 1 -- 172.21.15.89:0/2476714485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc510106a80 msgr2=0x7fc510106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:53.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.448+0000 7fc517ec6640 1 --2- 172.21.15.89:0/2476714485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc510106a80 0x7fc510106e60 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fc50c009930 tx=0x7fc50c02a420 comp rx=0 tx=0).stop 2024-09-15T23:39:53.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.448+0000 7fc517ec6640 1 -- 172.21.15.89:0/2476714485 shutdown_connections 2024-09-15T23:39:53.478 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.448+0000 7fc517ec6640 1 --2- 172.21.15.89:0/2476714485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc510101420 0x7fc51010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.448+0000 7fc517ec6640 1 --2- 172.21.15.89:0/2476714485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc510100a80 0x7fc510100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.448+0000 7fc517ec6640 1 --2- 172.21.15.89:0/2476714485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc510106a80 0x7fc510106e60 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 -- 172.21.15.89:0/2476714485 >> 172.21.15.89:0/2476714485 conn(0x7fc5100fc7a0 msgr2=0x7fc5100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 -- 172.21.15.89:0/2476714485 shutdown_connections 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 -- 172.21.15.89:0/2476714485 wait complete. 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 Processor -- start 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 -- start start 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc510100a80 0x7fc510198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc510101420 0x7fc510198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc510106a80 0x7fc51019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc510110ec0 con 0x7fc510101420 2024-09-15T23:39:53.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc510110d40 con 0x7fc510106a80 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc517ec6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc510111040 con 0x7fc510100a80 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.449+0000 7fc515461640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc510101420 0x7fc510198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc515461640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc510101420 0x7fc510198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55850/0 (socket says 172.21.15.89:55850) 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc515461640 1 -- 172.21.15.89:0/813226755 learned_addr learned my addr 172.21.15.89:0/813226755 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc515c62640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc510100a80 0x7fc510198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc516463640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc510106a80 0x7fc51019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc515461640 1 -- 172.21.15.89:0/813226755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc510100a80 msgr2=0x7fc510198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc515461640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc510100a80 0x7fc510198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc515461640 1 -- 172.21.15.89:0/813226755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc510106a80 msgr2=0x7fc51019cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc515461640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc510106a80 0x7fc51019cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc515461640 1 -- 172.21.15.89:0/813226755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51019d3f0 con 0x7fc510101420 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc516463640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc510106a80 0x7fc51019cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:53.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc515461640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc510101420 0x7fc510198980 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fc50000b4a0 tx=0x7fc50000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc506ffd640 1 -- 172.21.15.89:0/813226755 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500014070 con 0x7fc510101420 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc506ffd640 1 -- 172.21.15.89:0/813226755 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc500010af0 con 0x7fc510101420 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc517ec6640 1 -- 172.21.15.89:0/813226755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc51019d6e0 con 0x7fc510101420 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc506ffd640 1 -- 172.21.15.89:0/813226755 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500017d10 con 0x7fc510101420 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.450+0000 7fc517ec6640 1 -- 172.21.15.89:0/813226755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5101a4fc0 con 0x7fc510101420 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.451+0000 7fc517ec6640 1 -- 172.21.15.89:0/813226755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc510100ee0 con 0x7fc510101420 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.452+0000 7fc506ffd640 1 -- 172.21.15.89:0/813226755 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc500017070 con 0x7fc510101420 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.453+0000 7fc506ffd640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4fc0a3240 0x7fc4fc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.453+0000 7fc515c62640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4fc0a3240 0x7fc4fc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.453+0000 7fc506ffd640 1 -- 172.21.15.89:0/813226755 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc5000cf120 con 0x7fc510101420 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.453+0000 7fc515c62640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4fc0a3240 0x7fc4fc0a5700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc50c000c00 tx=0x7fc50c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:53.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.457+0000 7fc506ffd640 1 -- 172.21.15.89:0/813226755 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc50009ba50 con 0x7fc510101420 2024-09-15T23:39:53.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:53 smithi174 ceph-mon[35586]: pgmap v825: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-09-15T23:39:53.523 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:53.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.679+0000 7fc517ec6640 1 -- 172.21.15.89:0/813226755 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc51002d7c0 con 0x7fc510101420 2024-09-15T23:39:53.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.680+0000 7fc506ffd640 1 -- 172.21.15.89:0/813226755 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7fc50009b350 con 0x7fc510101420 2024-09-15T23:39:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:39:53.681 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:39:53.682 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:39:53.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 -- 172.21.15.89:0/813226755 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4fc0a3240 msgr2=0x7fc4fc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:53.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4fc0a3240 0x7fc4fc0a5700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc50c000c00 tx=0x7fc50c005a20 comp rx=0 tx=0).stop 2024-09-15T23:39:53.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 -- 172.21.15.89:0/813226755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc510101420 msgr2=0x7fc510198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:53.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc510101420 0x7fc510198980 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fc50000b4a0 tx=0x7fc50000b960 comp rx=0 tx=0).stop 2024-09-15T23:39:53.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 -- 172.21.15.89:0/813226755 shutdown_connections 2024-09-15T23:39:53.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc4fc0a3240 0x7fc4fc0a5700 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:53.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc510106a80 0x7fc51019cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:53.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc510101420 0x7fc510198980 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:53.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 --2- 172.21.15.89:0/813226755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc510100a80 0x7fc510198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:53.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 -- 172.21.15.89:0/813226755 >> 172.21.15.89:0/813226755 conn(0x7fc5100fc7a0 msgr2=0x7fc51006aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:53.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 -- 172.21.15.89:0/813226755 shutdown_connections 2024-09-15T23:39:53.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:53.682+0000 7fc517ec6640 1 -- 172.21.15.89:0/813226755 wait complete. 2024-09-15T23:39:54.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:54 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/813226755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:54.444 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0' 2024-09-15T23:39:54.444 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0 2024-09-15T23:39:54.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:54 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/813226755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:54.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.609+0000 7fcfc017e640 1 -- 172.21.15.89:0/3966262499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfb8102ef0 msgr2=0x7fcfb810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:54.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.609+0000 7fcfc017e640 1 --2- 172.21.15.89:0/3966262499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfb8102ef0 0x7fcfb810da40 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009ca0 tx=0x7fcfb002a950 comp rx=0 tx=0).stop 2024-09-15T23:39:54.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.609+0000 7fcfc017e640 1 -- 172.21.15.89:0/3966262499 shutdown_connections 2024-09-15T23:39:54.636 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.609+0000 7fcfc017e640 1 --2- 172.21.15.89:0/3966262499 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfb8102ef0 0x7fcfb810da40 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.609+0000 7fcfc017e640 1 --2- 172.21.15.89:0/3966262499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfb8102570 0x7fcfb81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.609+0000 7fcfc017e640 1 --2- 172.21.15.89:0/3966262499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb8100e20 0x7fcfb8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.609+0000 7fcfc017e640 1 -- 172.21.15.89:0/3966262499 >> 172.21.15.89:0/3966262499 conn(0x7fcfb80fc7d0 msgr2=0x7fcfb80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.609+0000 7fcfc017e640 1 -- 172.21.15.89:0/3966262499 shutdown_connections 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.610+0000 7fcfc017e640 1 -- 172.21.15.89:0/3966262499 wait complete. 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.610+0000 7fcfc017e640 1 Processor -- start 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.610+0000 7fcfc017e640 1 -- start start 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.610+0000 7fcfc017e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb8100e20 0x7fcfb8197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.610+0000 7fcfc017e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfb8102570 0x7fcfb81979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.610+0000 7fcfc017e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfb8102ef0 0x7fcfb819bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.610+0000 7fcfc017e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb8110f80 con 0x7fcfb8102ef0 2024-09-15T23:39:54.637 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.610+0000 7fcfc017e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb8110e00 con 0x7fcfb8102570 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.610+0000 7fcfc017e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb8111100 con 0x7fcfb8100e20 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbe71b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfb8102ef0 0x7fcfb819bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbe71b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfb8102ef0 0x7fcfb819bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55872/0 (socket says 172.21.15.89:55872) 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbe71b640 1 -- 172.21.15.89:0/285385167 learned_addr learned my addr 172.21.15.89:0/285385167 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb8100e20 0x7fcfb8197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbd719640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfb8102570 0x7fcfb81979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbe71b640 1 -- 172.21.15.89:0/285385167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb8100e20 msgr2=0x7fcfb8197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbe71b640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb8100e20 0x7fcfb8197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbe71b640 1 -- 172.21.15.89:0/285385167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfb8102570 msgr2=0x7fcfb81979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbe71b640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfb8102570 0x7fcfb81979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbe71b640 1 -- 172.21.15.89:0/285385167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb819c410 con 0x7fcfb8102ef0 2024-09-15T23:39:54.638 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbd719640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfb8102570 0x7fcfb81979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfbe71b640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfb8102ef0 0x7fcfb819bd30 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0004af0 tx=0x7fcfb002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfaeffd640 1 -- 172.21.15.89:0/285385167 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0034040 con 0x7fcfb8102ef0 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfaeffd640 1 -- 172.21.15.89:0/285385167 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb0030920 con 0x7fcfb8102ef0 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.611+0000 7fcfc017e640 1 -- 172.21.15.89:0/285385167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb819c6a0 con 0x7fcfb8102ef0 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.612+0000 7fcfaeffd640 1 -- 172.21.15.89:0/285385167 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0004820 con 0x7fcfb8102ef0 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.612+0000 7fcfc017e640 1 -- 172.21.15.89:0/285385167 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb819cb20 con 0x7fcfb8102ef0 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.615+0000 7fcfaeffd640 1 -- 172.21.15.89:0/285385167 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fcfb0037020 con 0x7fcfb8102ef0 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.615+0000 7fcfc2557740 1 Processor -- start 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.615+0000 7fcfc2557740 1 -- start start 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.615+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3636090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.615+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 0x55bec3636a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.615+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 0x55bec3643760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.615+0000 7fcfc2557740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bec359c2f0 con 0x55bec3637b40 2024-09-15T23:39:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.615+0000 7fcfc2557740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bec3543cf0 con 0x55bec36365d0 2024-09-15T23:39:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.615+0000 7fcfc2557740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bec35982b0 con 0x55bec3641370 2024-09-15T23:39:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbd719640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3636090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbd719640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3636090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55882/0 (socket says 172.21.15.89:55882) 2024-09-15T23:39:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbd719640 1 -- 172.21.15.89:0/182274296 learned_addr learned my addr 172.21.15.89:0/182274296 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfaeffd640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfa40a3240 0x7fcfa40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/182274296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 0x55bec3636a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbe71b640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfa40a3240 0x7fcfa40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbd719640 1 --2- 172.21.15.89:0/182274296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 0x55bec3643760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbd719640 1 -- 172.21.15.89:0/182274296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 msgr2=0x55bec3643760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbd719640 1 --2- 172.21.15.89:0/182274296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 0x55bec3643760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbd719640 1 -- 172.21.15.89:0/182274296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 msgr2=0x55bec3636a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbd719640 1 --2- 172.21.15.89:0/182274296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 0x55bec3636a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbd719640 1 -- 172.21.15.89:0/182274296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bec35283a0 con 0x55bec3637b40 2024-09-15T23:39:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfaeffd640 1 -- 172.21.15.89:0/285385167 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcfb00f26f0 con 0x7fcfb8102ef0 2024-09-15T23:39:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.616+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/182274296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 0x55bec3636a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.617+0000 7fcfbd719640 1 --2- 172.21.15.89:0/182274296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3636090 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80058c0 tx=0x7fcfa800b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a88083287b77b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:54.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.617+0000 7fcfa3fff640 1 -- 172.21.15.89:0/182274296 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8019030 con 0x55bec3637b40 2024-09-15T23:39:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.617+0000 7fcfa3fff640 1 -- 172.21.15.89:0/182274296 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa80035f0 con 0x55bec3637b40 2024-09-15T23:39:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.617+0000 7fcfa3fff640 1 -- 172.21.15.89:0/182274296 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8011720 con 0x55bec3637b40 2024-09-15T23:39:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.618+0000 7fcfbe71b640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfa40a3240 0x7fcfa40a5700 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00ba720 tx=0x7fcfb00b9400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.618+0000 7fcfc2557740 1 -- 172.21.15.89:0/182274296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 msgr2=0x55bec3636090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.618+0000 7fcfc2557740 1 --2- 172.21.15.89:0/182274296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3636090 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fcfa80058c0 tx=0x7fcfa800b6b0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.618+0000 7fcfc2557740 1 -- 172.21.15.89:0/182274296 shutdown_connections 2024-09-15T23:39:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.618+0000 7fcfc2557740 1 --2- 172.21.15.89:0/182274296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 0x55bec3643760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.618+0000 7fcfc2557740 1 --2- 172.21.15.89:0/182274296 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 0x55bec3636a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.618+0000 7fcfc2557740 1 --2- 172.21.15.89:0/182274296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3636090 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.618+0000 7fcfc2557740 1 -- 172.21.15.89:0/182274296 >> 172.21.15.89:0/182274296 conn(0x55bec35ff310 msgr2=0x55bec3632c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.618+0000 7fcfc2557740 1 -- 172.21.15.89:0/182274296 shutdown_connections 2024-09-15T23:39:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.618+0000 7fcfc2557740 1 -- 172.21.15.89:0/182274296 wait complete. 2024-09-15T23:39:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfc2557740 1 -- start start 2024-09-15T23:39:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfc2557740 1 Processor -- start 2024-09-15T23:39:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec36365d0 0x55bec3640d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3637b40 0x55bec363dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec3641370 0x55bec363e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfc2557740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bec3648480 con 0x55bec36365d0 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfc2557740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bec3557060 con 0x55bec3641370 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfc2557740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bec3648600 con 0x55bec3637b40 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfc2557740 10 client.0 _ll_register_callbacks cb 0x55bec32aa8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfbd719640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec36365d0 0x55bec3640d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfbd719640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec36365d0 0x55bec3640d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55896/0 (socket says 172.21.15.89:55896) 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.619+0000 7fcfbd719640 1 -- 172.21.15.89:0/813737921 learned_addr learned my addr 172.21.15.89:0/813737921 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfbd719640 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec3641370 0x55bec363e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3637b40 0x55bec363dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfbd719640 1 -- 172.21.15.89:0/813737921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3637b40 msgr2=0x55bec363dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfbd719640 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3637b40 0x55bec363dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfbd719640 1 -- 172.21.15.89:0/813737921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec3641370 msgr2=0x55bec363e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:54.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfbd719640 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec3641370 0x55bec363e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfbd719640 1 -- 172.21.15.89:0/813737921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bec330d3c0 con 0x55bec36365d0 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3637b40 0x55bec363dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfbd719640 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec36365d0 0x55bec3640d00 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8005890 tx=0x7fcfa800b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31fd526194626b42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfa17fa640 1 -- 172.21.15.89:0/813737921 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8019050 con 0x55bec36365d0 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfa17fa640 1 -- 172.21.15.89:0/813737921 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcfa8003be0 con 0x55bec36365d0 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.620+0000 7fcfc2557740 1 -- 172.21.15.89:0/813737921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfa80035f0 con 0x55bec36365d0 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfa17fa640 1 -- 172.21.15.89:0/813737921 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcfa8033050 con 0x55bec36365d0 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfc2557740 10 client.20922 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfc2557740 1 -- 172.21.15.89:0/813737921 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bec3653eb0 con 0x55bec36365d0 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfa17fa640 1 -- 172.21.15.89:0/813737921 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa80035f0 con 0x55bec36365d0 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfa17fa640 1 -- 172.21.15.89:0/813737921 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcfa801e460 con 0x55bec36365d0 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfc2557740 10 client.20922 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:54.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfc2557740 10 client.20922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfc2557740 20 client.20922 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfc2557740 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bec3655310 0x55bec3657700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfc2557740 4 client.20922 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfc2557740 1 -- 172.21.15.89:0/813737921 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bec35a9860 con 0x55bec3655310 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.621+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bec3655310 0x55bec3657700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.622+0000 7fcfa17fa640 10 client.20922 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.622+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bec3655310 0x55bec3657700 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4d0b8a172d3c007 server_cookie=f9e974c1b43ed890 in_seq=0 out_seq=0 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfa17fa640 1 -- 172.21.15.89:0/813737921 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bec35a9860 con 0x55bec3655310 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfa17fa640 10 client.20922 handle_command_reply: tid=0 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 client.20922 shutdown 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 -- 172.21.15.89:0/813737921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec36365d0 msgr2=0x55bec3640d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec36365d0 0x55bec3640d00 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8005890 tx=0x7fcfa800b420 comp rx=0 tx=0).stop 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 -- 172.21.15.89:0/813737921 shutdown_connections 2024-09-15T23:39:54.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bec3655310 0x55bec3657700 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec3641370 0x55bec363e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfa17fa640 0 client.20922 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3637b40 0x55bec363dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 --2- 172.21.15.89:0/813737921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec36365d0 0x55bec3640d00 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 -- 172.21.15.89:0/813737921 >> 172.21.15.89:0/813737921 conn(0x55bec35ff310 msgr2=0x55bec3638930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 -- 172.21.15.89:0/813737921 shutdown_connections 2024-09-15T23:39:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 1 -- 172.21.15.89:0/813737921 wait complete. 2024-09-15T23:39:54.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.623+0000 7fcfc2557740 20 client.20922 trim_cache size 0 max 16384 2024-09-15T23:39:54.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfc2557740 1 Processor -- start 2024-09-15T23:39:54.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfc2557740 1 -- start start 2024-09-15T23:39:54.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 0x55bec363d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3639120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 0x55bec3639660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfc2557740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81acba0 con 0x55bec3637b40 2024-09-15T23:39:54.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfc2557740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81ae6e0 con 0x55bec36365d0 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfc2557740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81ad270 con 0x55bec3641370 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfbdf1a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3639120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfbdf1a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3639120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55900/0 (socket says 172.21.15.89:55900) 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfbdf1a640 1 -- 172.21.15.89:0/3105024200 learned_addr learned my addr 172.21.15.89:0/3105024200 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfbd719640 1 --2- 172.21.15.89:0/3105024200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 0x55bec3639660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.646+0000 7fcfbd719640 1 --2- 172.21.15.89:0/3105024200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 0x55bec363d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.647+0000 7fcfbd719640 1 -- 172.21.15.89:0/3105024200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 msgr2=0x55bec363d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.647+0000 7fcfbd719640 1 --2- 172.21.15.89:0/3105024200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 0x55bec363d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.647+0000 7fcfbd719640 1 -- 172.21.15.89:0/3105024200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 msgr2=0x55bec3639120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.647+0000 7fcfbd719640 1 --2- 172.21.15.89:0/3105024200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3639120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.647+0000 7fcfbd719640 1 -- 172.21.15.89:0/3105024200 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bec363cf60 con 0x55bec3641370 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.647+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/3105024200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3639120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:54.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.647+0000 7fcfbd719640 1 --2- 172.21.15.89:0/3105024200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 0x55bec3639660 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcfa801e8b0 tx=0x7fcfa800bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3e151fb6c187cf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.647+0000 7fcfa37fe640 1 -- 172.21.15.89:0/3105024200 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8019040 con 0x55bec3641370 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.647+0000 7fcfa37fe640 1 -- 172.21.15.89:0/3105024200 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa8023040 con 0x55bec3641370 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.647+0000 7fcfa37fe640 1 -- 172.21.15.89:0/3105024200 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8013e60 con 0x55bec3641370 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.648+0000 7fcfc2557740 1 -- 172.21.15.89:0/3105024200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 msgr2=0x55bec3639660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.648+0000 7fcfc2557740 1 --2- 172.21.15.89:0/3105024200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 0x55bec3639660 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcfa801e8b0 tx=0x7fcfa800bf90 comp rx=0 tx=0).stop 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.648+0000 7fcfc2557740 1 -- 172.21.15.89:0/3105024200 shutdown_connections 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.648+0000 7fcfc2557740 1 --2- 172.21.15.89:0/3105024200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec3641370 0x55bec3639660 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.648+0000 7fcfc2557740 1 --2- 172.21.15.89:0/3105024200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3639120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.648+0000 7fcfc2557740 1 --2- 172.21.15.89:0/3105024200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec36365d0 0x55bec363d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.648+0000 7fcfc2557740 1 -- 172.21.15.89:0/3105024200 >> 172.21.15.89:0/3105024200 conn(0x55bec35ff310 msgr2=0x55bec3631d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.648+0000 7fcfc2557740 1 -- 172.21.15.89:0/3105024200 shutdown_connections 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.648+0000 7fcfc2557740 1 -- 172.21.15.89:0/3105024200 wait complete. 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.649+0000 7fcfc2557740 1 -- start start 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.649+0000 7fcfc2557740 1 Processor -- start 2024-09-15T23:39:54.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.649+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec36365d0 0x55bec364dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3649090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfc2557740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec3641370 0x55bec36495d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfc2557740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81ae010 con 0x55bec3637b40 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfc2557740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81ad940 con 0x55bec3641370 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfc2557740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81aab80 con 0x55bec36365d0 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfc2557740 10 client.0 _ll_register_callbacks cb 0x7fcfb4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbdf1a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3649090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbdf1a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3649090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:55908/0 (socket says 172.21.15.89:55908) 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbdf1a640 1 -- 172.21.15.89:0/661856692 learned_addr learned my addr 172.21.15.89:0/661856692 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbd719640 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec36365d0 0x55bec364dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbd719640 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec3641370 0x55bec36495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbdf1a640 1 -- 172.21.15.89:0/661856692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec36365d0 msgr2=0x55bec364dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec36365d0 0x55bec364dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbdf1a640 1 -- 172.21.15.89:0/661856692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec3641370 msgr2=0x55bec36495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec3641370 0x55bec36495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbdf1a640 1 -- 172.21.15.89:0/661856692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bec3649b90 con 0x55bec3637b40 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfbdf1a640 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3649090 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4004180 tx=0x7fcfb4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307af17fed1f0407 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfa17fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.650+0000 7fcfa17fa640 1 -- 172.21.15.89:0/661856692 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb402b060 con 0x55bec3637b40 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfc2557740 1 -- 172.21.15.89:0/661856692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfa8023040 con 0x55bec3637b40 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfa17fa640 1 -- 172.21.15.89:0/661856692 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fcfb4026470 con 0x55bec3637b40 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfa17fa640 1 -- 172.21.15.89:0/661856692 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcfb4032050 con 0x55bec3637b40 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfc2557740 10 client.20928 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfc2557740 1 -- 172.21.15.89:0/661856692 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bec36bfc40 con 0x55bec3637b40 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfa17fa640 1 -- 172.21.15.89:0/661856692 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa8023040 con 0x55bec3637b40 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfa17fa640 1 -- 172.21.15.89:0/661856692 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcfb4010640 con 0x55bec3637b40 2024-09-15T23:39:54.656 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfc2557740 10 client.20928 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:54.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfc2557740 10 client.20928 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:54.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfc2557740 20 client.20928 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:54.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfc2557740 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bec3655310 0x55bec36c2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:54.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfc2557740 4 client.20928 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:39:54.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.651+0000 7fcfc2557740 1 -- 172.21.15.89:0/661856692 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bec35a3d70 con 0x55bec3655310 2024-09-15T23:39:54.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.652+0000 7fcfbd719640 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bec3655310 0x55bec36c2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:54.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.652+0000 7fcfbd719640 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bec3655310 0x55bec36c2460 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bebfd7484ea1254f server_cookie=b8cc3beb6db59f52 in_seq=0 out_seq=0 2024-09-15T23:39:54.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.652+0000 7fcfa17fa640 10 client.20928 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:54.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.652+0000 7fcfa17fa640 1 -- 172.21.15.89:0/661856692 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bec35a3d70 con 0x55bec3655310 2024-09-15T23:39:54.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.652+0000 7fcfa17fa640 10 client.20928 handle_command_reply: tid=0 2024-09-15T23:39:54.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.652+0000 7fcfc2557740 1 client.20928 shutdown 2024-09-15T23:39:54.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.652+0000 7fcfc2557740 1 -- 172.21.15.89:0/661856692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 msgr2=0x55bec3649090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:54.658 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.652+0000 7fcfc2557740 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3649090 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4004180 tx=0x7fcfb4004270 comp rx=0 tx=0).stop 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.653+0000 7fcfc2557740 1 -- 172.21.15.89:0/661856692 shutdown_connections 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.653+0000 7fcfc2557740 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bec3655310 0x55bec36c2460 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.653+0000 7fcfc2557740 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bec3641370 0x55bec36495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.653+0000 7fcfa17fa640 0 client.20928 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.653+0000 7fcfc2557740 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bec3637b40 0x55bec3649090 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.653+0000 7fcfc2557740 1 --2- 172.21.15.89:0/661856692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bec36365d0 0x55bec364dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.653+0000 7fcfc2557740 1 -- 172.21.15.89:0/661856692 >> 172.21.15.89:0/661856692 conn(0x55bec35ff310 msgr2=0x55bec363f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.653+0000 7fcfc2557740 1 -- 172.21.15.89:0/661856692 shutdown_connections 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.653+0000 7fcfc2557740 1 -- 172.21.15.89:0/661856692 wait complete. 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.653+0000 7fcfc2557740 20 client.20928 trim_cache size 0 max 16384 2024-09-15T23:39:54.659 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:39:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 -- 172.21.15.89:0/285385167 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfa40a3240 msgr2=0x7fcfa40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfa40a3240 0x7fcfa40a5700 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00ba720 tx=0x7fcfb00b9400 comp rx=0 tx=0).stop 2024-09-15T23:39:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 -- 172.21.15.89:0/285385167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfb8102ef0 msgr2=0x7fcfb819bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfb8102ef0 0x7fcfb819bd30 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0004af0 tx=0x7fcfb002f5b0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 -- 172.21.15.89:0/285385167 shutdown_connections 2024-09-15T23:39:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fcfa40a3240 0x7fcfa40a5700 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfb8102ef0 0x7fcfb819bd30 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcfb8102570 0x7fcfb81979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 --2- 172.21.15.89:0/285385167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfb8100e20 0x7fcfb8197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:54.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 -- 172.21.15.89:0/285385167 >> 172.21.15.89:0/285385167 conn(0x7fcfb80fc7d0 msgr2=0x7fcfb81068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:54.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 -- 172.21.15.89:0/285385167 shutdown_connections 2024-09-15T23:39:54.661 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:54.654+0000 7fcfc017e640 1 -- 172.21.15.89:0/285385167 wait complete. 2024-09-15T23:39:54.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:54 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/813226755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:39:55.400 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:55 smithi089 ceph-mon[30933]: pgmap v826: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.4 KiB/s wr, 13 op/s 2024-09-15T23:39:55.467 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:55.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:55 smithi174 ceph-mon[35586]: pgmap v826: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.4 KiB/s wr, 13 op/s 2024-09-15T23:39:55.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:55 smithi120 ceph-mon[35250]: pgmap v826: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 4.4 KiB/s wr, 13 op/s 2024-09-15T23:39:55.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.950+0000 7f0f3a134640 1 -- 172.21.15.89:0/1832551070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f34103700 msgr2=0x7f0f3410bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:55.964 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.950+0000 7f0f3a134640 1 --2- 172.21.15.89:0/1832551070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f34103700 0x7f0f3410bbf0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f0f28009900 tx=0x7f0f2802a480 comp rx=0 tx=0).stop 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.951+0000 7f0f3a134640 1 -- 172.21.15.89:0/1832551070 shutdown_connections 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.951+0000 7f0f3a134640 1 --2- 172.21.15.89:0/1832551070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f3410c180 0x7f0f3410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.951+0000 7f0f3a134640 1 --2- 172.21.15.89:0/1832551070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f34103700 0x7f0f3410bbf0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.951+0000 7f0f3a134640 1 --2- 172.21.15.89:0/1832551070 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f34102de0 0x7f0f341031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.951+0000 7f0f3a134640 1 -- 172.21.15.89:0/1832551070 >> 172.21.15.89:0/1832551070 conn(0x7f0f340fc7a0 msgr2=0x7f0f340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.951+0000 7f0f3a134640 1 -- 172.21.15.89:0/1832551070 shutdown_connections 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.951+0000 7f0f3a134640 1 -- 172.21.15.89:0/1832551070 wait complete. 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.951+0000 7f0f3a134640 1 Processor -- start 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.951+0000 7f0f3a134640 1 -- start start 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f3a134640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f34102de0 0x7f0f34198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f3a134640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f34103700 0x7f0f34198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f3a134640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f3410c180 0x7f0f3419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.965 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f3a134640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0f34112030 con 0x7f0f34102de0 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f3a134640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0f34111eb0 con 0x7f0f3410c180 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f3a134640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0f341121b0 con 0x7f0f34103700 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f337fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f34102de0 0x7f0f34198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f337fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f34102de0 0x7f0f34198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38890/0 (socket says 172.21.15.89:38890) 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f337fe640 1 -- 172.21.15.89:0/2378828589 learned_addr learned my addr 172.21.15.89:0/2378828589 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f34103700 0x7f0f34198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f337fe640 1 -- 172.21.15.89:0/2378828589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f34103700 msgr2=0x7f0f34198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f337fe640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f34103700 0x7f0f34198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f337fe640 1 -- 172.21.15.89:0/2378828589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f3410c180 msgr2=0x7f0f3419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f337fe640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f3410c180 0x7f0f3419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f337fe640 1 -- 172.21.15.89:0/2378828589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3419d540 con 0x7f0f34102de0 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f337fe640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f34102de0 0x7f0f34198590 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400ba50 tx=0x7f0f2400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f30ff9640 1 -- 172.21.15.89:0/2378828589 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24019070 con 0x7f0f34102de0 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f30ff9640 1 -- 172.21.15.89:0/2378828589 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f2400cbe0 con 0x7f0f34102de0 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.952+0000 7f0f3a134640 1 -- 172.21.15.89:0/2378828589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f3419d830 con 0x7f0f34102de0 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.953+0000 7f0f30ff9640 1 -- 172.21.15.89:0/2378828589 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f240076e0 con 0x7f0f34102de0 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.953+0000 7f0f3a134640 1 -- 172.21.15.89:0/2378828589 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f341a5110 con 0x7f0f34102de0 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f30ff9640 1 -- 172.21.15.89:0/2378828589 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0f2401c070 con 0x7f0f34102de0 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f3c50d740 1 Processor -- start 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f3c50d740 1 -- start start 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f30ff9640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f1c0a32c0 0x7f0f1c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6a2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 0x5622da6b1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6b4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5622da6082a0 con 0x5622da6a2660 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5622da5afb10 con 0x5622da6a2f80 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5622da604260 con 0x5622da6b1d50 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.955+0000 7f0f33fff640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f1c0a32c0 0x7f0f1c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f30ff9640 1 -- 172.21.15.89:0/2378828589 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0f240cf340 con 0x7f0f34102de0 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f337fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6a2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f337fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6a2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38904/0 (socket says 172.21.15.89:38904) 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f337fe640 1 -- 172.21.15.89:0/63897923 learned_addr learned my addr 172.21.15.89:0/63897923 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/63897923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 0x5622da6b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f337fe640 1 -- 172.21.15.89:0/63897923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 msgr2=0x5622da6b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f337fe640 1 --2- 172.21.15.89:0/63897923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f337fe640 1 -- 172.21.15.89:0/63897923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 msgr2=0x5622da6b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:55.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f337fe640 1 --2- 172.21.15.89:0/63897923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 0x5622da6b1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f337fe640 1 -- 172.21.15.89:0/63897923 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622da379b90 con 0x5622da6a2660 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f33fff640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f1c0a32c0 0x7f0f1c0a5780 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009730 tx=0x7f0f2c009b30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f337fe640 1 --2- 172.21.15.89:0/63897923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6a2a40 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f0f24097f40 tx=0x7f0f2409aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b20b4ee940fd6f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f19ffb640 1 -- 172.21.15.89:0/63897923 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2409bc70 con 0x5622da6a2660 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f19ffb640 1 -- 172.21.15.89:0/63897923 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f240cf160 con 0x5622da6a2660 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.956+0000 7f0f19ffb640 1 -- 172.21.15.89:0/63897923 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24097be0 con 0x5622da6a2660 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.957+0000 7f0f3c50d740 1 -- 172.21.15.89:0/63897923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 msgr2=0x5622da6a2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.957+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/63897923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6a2a40 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f0f24097f40 tx=0x7f0f2409aed0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.957+0000 7f0f3c50d740 1 -- 172.21.15.89:0/63897923 shutdown_connections 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.957+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/63897923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.957+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/63897923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 0x5622da6b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.957+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/63897923 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6a2a40 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.957+0000 7f0f3c50d740 1 -- 172.21.15.89:0/63897923 >> 172.21.15.89:0/63897923 conn(0x5622da69ecf0 msgr2=0x5622da69f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:55.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.957+0000 7f0f3c50d740 1 -- 172.21.15.89:0/63897923 shutdown_connections 2024-09-15T23:39:55.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.957+0000 7f0f3c50d740 1 -- 172.21.15.89:0/63897923 wait complete. 2024-09-15T23:39:55.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.957+0000 7f0f3c50d740 1 -- start start 2024-09-15T23:39:55.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f3c50d740 1 Processor -- start 2024-09-15T23:39:55.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2660 0x5622da6aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6af720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5622da6b4980 con 0x5622da6a2f80 2024-09-15T23:39:55.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5622da5906e0 con 0x5622da6a2660 2024-09-15T23:39:55.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5622da6b4b00 con 0x5622da6b1d50 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f3c50d740 10 client.0 _ll_register_callbacks cb 0x5622da5b4a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f32ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f32ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6af1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38920/0 (socket says 172.21.15.89:38920) 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f32ffd640 1 -- 172.21.15.89:0/4022535889 learned_addr learned my addr 172.21.15.89:0/4022535889 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f33fff640 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f337fe640 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2660 0x5622da6aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f32ffd640 1 -- 172.21.15.89:0/4022535889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 msgr2=0x5622da6af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6af720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f32ffd640 1 -- 172.21.15.89:0/4022535889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2660 msgr2=0x5622da6aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2660 0x5622da6aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f32ffd640 1 -- 172.21.15.89:0/4022535889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622da6bdf30 con 0x5622da6a2f80 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f337fe640 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2660 0x5622da6aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:55.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6af1e0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f0f28031550 tx=0x7f0f280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0a3cbdb60e3e32 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.958+0000 7f0f12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f12ffd640 1 -- 172.21.15.89:0/4022535889 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28035070 con 0x5622da6a2f80 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f12ffd640 1 -- 172.21.15.89:0/4022535889 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0f280037f0 con 0x5622da6a2f80 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f3c50d740 1 -- 172.21.15.89:0/4022535889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f240cf160 con 0x5622da6a2f80 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f12ffd640 1 -- 172.21.15.89:0/4022535889 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0f28005d50 con 0x5622da6a2f80 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f3c50d740 10 client.20940 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f3c50d740 1 -- 172.21.15.89:0/4022535889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5622da6c08a0 con 0x5622da6a2f80 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f12ffd640 1 -- 172.21.15.89:0/4022535889 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f240cf160 con 0x5622da6a2f80 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f12ffd640 1 -- 172.21.15.89:0/4022535889 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0f2803b430 con 0x5622da6a2f80 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f3c50d740 10 client.20940 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f3c50d740 10 client.20940 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:39:55.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f3c50d740 20 client.20940 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622da6c1d60 0x5622da6c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f3c50d740 4 client.20940 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.959+0000 7f0f3c50d740 1 -- 172.21.15.89:0/4022535889 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622da6159d0 con 0x5622da6c1d60 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.960+0000 7f0f337fe640 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622da6c1d60 0x5622da6c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.960+0000 7f0f12ffd640 10 client.20940 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.960+0000 7f0f337fe640 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622da6c1d60 0x5622da6c41d0 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84acf27eda1727b3 server_cookie=d13a927af3e6afa9 in_seq=0 out_seq=0 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f12ffd640 1 -- 172.21.15.89:0/4022535889 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5622da6159d0 con 0x5622da6c1d60 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f12ffd640 10 client.20940 handle_command_reply: tid=0 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 client.20940 shutdown 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 -- 172.21.15.89:0/4022535889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 msgr2=0x5622da6af1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6af1e0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f0f28031550 tx=0x7f0f280042f0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 -- 172.21.15.89:0/4022535889 shutdown_connections 2024-09-15T23:39:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622da6c1d60 0x5622da6c41d0 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6af1e0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f12ffd640 0 client.20940 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/4022535889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2660 0x5622da6aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 -- 172.21.15.89:0/4022535889 >> 172.21.15.89:0/4022535889 conn(0x5622da69ecf0 msgr2=0x5622da6a4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 -- 172.21.15.89:0/4022535889 shutdown_connections 2024-09-15T23:39:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 1 -- 172.21.15.89:0/4022535889 wait complete. 2024-09-15T23:39:55.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.961+0000 7f0f3c50d740 20 client.20940 trim_cache size 0 max 16384 2024-09-15T23:39:55.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f3c50d740 1 Processor -- start 2024-09-15T23:39:55.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f3c50d740 1 -- start start 2024-09-15T23:39:55.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6a2660 0x5622da6c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6b1d50 0x5622da6b9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.984 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0f341acce0 con 0x5622da6a2f80 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0f341ae820 con 0x5622da6b1d50 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0f341ad3b0 con 0x5622da6a2660 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f32ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f32ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38930/0 (socket says 172.21.15.89:38930) 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f32ffd640 1 -- 172.21.15.89:0/2967104176 learned_addr learned my addr 172.21.15.89:0/2967104176 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f33fff640 1 --2- 172.21.15.89:0/2967104176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6b1d50 0x5622da6b9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f337fe640 1 --2- 172.21.15.89:0/2967104176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6a2660 0x5622da6c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f32ffd640 1 -- 172.21.15.89:0/2967104176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6a2660 msgr2=0x5622da6c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.978+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/2967104176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6a2660 0x5622da6c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.979+0000 7f0f32ffd640 1 -- 172.21.15.89:0/2967104176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6b1d50 msgr2=0x5622da6b9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.979+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/2967104176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6b1d50 0x5622da6b9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.979+0000 7f0f32ffd640 1 -- 172.21.15.89:0/2967104176 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622da69e790 con 0x5622da6a2f80 2024-09-15T23:39:55.985 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.979+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/2967104176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6c9fe0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f0f2803ba40 tx=0x7f0f28000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc241f011e522d3c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.979+0000 7f0f197fa640 1 -- 172.21.15.89:0/2967104176 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28056020 con 0x5622da6a2f80 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.979+0000 7f0f197fa640 1 -- 172.21.15.89:0/2967104176 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f2803b630 con 0x5622da6a2f80 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.979+0000 7f0f197fa640 1 -- 172.21.15.89:0/2967104176 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28035070 con 0x5622da6a2f80 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 -- 172.21.15.89:0/2967104176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 msgr2=0x5622da6c9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/2967104176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6c9fe0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f0f2803ba40 tx=0x7f0f28000bc0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 -- 172.21.15.89:0/2967104176 shutdown_connections 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/2967104176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6b1d50 0x5622da6b9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/2967104176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2f80 0x5622da6c9fe0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/2967104176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6a2660 0x5622da6c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 -- 172.21.15.89:0/2967104176 >> 172.21.15.89:0/2967104176 conn(0x5622da69ecf0 msgr2=0x5622da6b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:55.986 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 -- 172.21.15.89:0/2967104176 shutdown_connections 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 -- 172.21.15.89:0/2967104176 wait complete. 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 -- start start 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 Processor -- start 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.980+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 0x5622da6b7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f3c50d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6b7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f0f341ae150 con 0x5622da6a2660 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f337fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0f341ada80 con 0x5622da6a2f80 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f3c50d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0f341aacc0 con 0x5622da6b1d50 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f337fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6b6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38938/0 (socket says 172.21.15.89:38938) 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f337fe640 1 -- 172.21.15.89:0/3443428448 learned_addr learned my addr 172.21.15.89:0/3443428448 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:55.987 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f33fff640 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f3c50d740 10 client.0 _ll_register_callbacks cb 0x7f0f28031a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 0x5622da6b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f337fe640 1 -- 172.21.15.89:0/3443428448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 msgr2=0x5622da6b7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f337fe640 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6b7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f337fe640 1 -- 172.21.15.89:0/3443428448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 msgr2=0x5622da6b7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f337fe640 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 0x5622da6b7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f337fe640 1 -- 172.21.15.89:0/3443428448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622da72ca80 con 0x5622da6a2660 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 0x5622da6b7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f337fe640 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6b6af0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f0f240a43a0 tx=0x7f0f2405b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7ac9aa30b40dd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f12ffd640 1 -- 172.21.15.89:0/3443428448 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f240cf160 con 0x5622da6a2660 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f3c50d740 1 -- 172.21.15.89:0/3443428448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f2803b630 con 0x5622da6a2660 2024-09-15T23:39:55.988 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.981+0000 7f0f12ffd640 1 -- 172.21.15.89:0/3443428448 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f0f24099870 con 0x5622da6a2660 2024-09-15T23:39:55.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f12ffd640 1 -- 172.21.15.89:0/3443428448 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0f2409b570 con 0x5622da6a2660 2024-09-15T23:39:55.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f3c50d740 10 client.20946 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:55.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f3c50d740 1 -- 172.21.15.89:0/3443428448 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5622da72dce0 con 0x5622da6a2660 2024-09-15T23:39:55.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f12ffd640 1 -- 172.21.15.89:0/3443428448 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f2803b630 con 0x5622da6a2660 2024-09-15T23:39:55.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f12ffd640 1 -- 172.21.15.89:0/3443428448 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f0f240a3d50 con 0x5622da6a2660 2024-09-15T23:39:55.989 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f3c50d740 10 client.20946 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:55.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f3c50d740 10 client.20946 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:39:55.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f3c50d740 20 client.20946 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:55.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622da72e020 0x5622da730410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:55.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f3c50d740 4 client.20946 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}] 2024-09-15T23:39:55.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f3c50d740 1 -- 172.21.15.89:0/3443428448 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}) -- 0x5622da60fd20 con 0x5622da72e020 2024-09-15T23:39:55.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.982+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622da72e020 0x5622da730410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:55.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.983+0000 7f0f12ffd640 10 client.20946 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:55.990 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.983+0000 7f0f32ffd640 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622da72e020 0x5622da730410 crc :-1 s=READY pgs=413 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a37326adac16ab80 server_cookie=4cac58a55b2fa596 in_seq=0 out_seq=0 2024-09-15T23:39:55.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.995+0000 7f0f12ffd640 1 -- 172.21.15.89:0/3443428448 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5622da60fd20 con 0x5622da72e020 2024-09-15T23:39:55.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.995+0000 7f0f12ffd640 10 client.20946 handle_command_reply: tid=0 2024-09-15T23:39:55.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.995+0000 7f0f3c50d740 1 client.20946 shutdown 2024-09-15T23:39:55.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.995+0000 7f0f3c50d740 1 -- 172.21.15.89:0/3443428448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 msgr2=0x5622da6b6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:55.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.995+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6b6af0 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f0f240a43a0 tx=0x7f0f2405b830 comp rx=0 tx=0).stop 2024-09-15T23:39:55.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3c50d740 1 -- 172.21.15.89:0/3443428448 shutdown_connections 2024-09-15T23:39:55.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5622da72e020 0x5622da730410 crc :-1 s=READY pgs=413 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5622da6b1d50 0x5622da6b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.997 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622da6a2f80 0x5622da6b7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3c50d740 1 --2- 172.21.15.89:0/3443428448 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5622da6a2660 0x5622da6b6af0 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:55.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3c50d740 1 -- 172.21.15.89:0/3443428448 >> 172.21.15.89:0/3443428448 conn(0x5622da69ecf0 msgr2=0x5622da69d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:55.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f12ffd640 0 client.20946 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:55.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3c50d740 1 -- 172.21.15.89:0/3443428448 shutdown_connections 2024-09-15T23:39:55.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3c50d740 1 -- 172.21.15.89:0/3443428448 wait complete. 2024-09-15T23:39:55.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3c50d740 20 client.20946 trim_cache size 0 max 16384 2024-09-15T23:39:55.998 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:39:55.998 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:39:55.998 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:39:55.998 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 58, 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "2590a49a": { 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 58, 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 137.7, 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:39:55.999 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:39:56.000 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:39:56.000 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "QUIESCED", 2024-09-15T23:39:56.000 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:39:56.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:56.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:56.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:56.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:56.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:39:56.000 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:39:56.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 -- 172.21.15.89:0/2378828589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f1c0a32c0 msgr2=0x7f0f1c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:56.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f1c0a32c0 0x7f0f1c0a5780 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c009730 tx=0x7f0f2c009b30 comp rx=0 tx=0).stop 2024-09-15T23:39:56.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 -- 172.21.15.89:0/2378828589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f34102de0 msgr2=0x7f0f34198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:56.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f34102de0 0x7f0f34198590 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400ba50 tx=0x7f0f2400bf10 comp rx=0 tx=0).stop 2024-09-15T23:39:56.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 -- 172.21.15.89:0/2378828589 shutdown_connections 2024-09-15T23:39:56.001 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f0f1c0a32c0 0x7f0f1c0a5780 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:56.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f3410c180 0x7f0f3419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:56.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f34103700 0x7f0f34198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:56.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 --2- 172.21.15.89:0/2378828589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f34102de0 0x7f0f34198590 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:56.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 -- 172.21.15.89:0/2378828589 >> 172.21.15.89:0/2378828589 conn(0x7f0f340fc7a0 msgr2=0x7f0f340fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:56.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 -- 172.21.15.89:0/2378828589 shutdown_connections 2024-09-15T23:39:56.002 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:55.996+0000 7f0f3a134640 1 -- 172.21.15.89:0/2378828589 wait complete. 2024-09-15T23:39:56.430 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:39:56.430 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:39:56.769 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '2590a49a', quiesce took 5.0 seconds. Will release after: 12.7 2024-09-15T23:39:56.792 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:57.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.264+0000 7ff061bf5640 1 -- 172.21.15.89:0/2744216818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05c101de0 msgr2=0x7ff05c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:57.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.265+0000 7ff061bf5640 1 --2- 172.21.15.89:0/2744216818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05c101de0 0x7ff05c1021c0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7ff04c009900 tx=0x7ff04c02a3f0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.265+0000 7ff061bf5640 1 -- 172.21.15.89:0/2744216818 shutdown_connections 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.265+0000 7ff061bf5640 1 --2- 172.21.15.89:0/2744216818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c106f10 0x7ff05c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.265+0000 7ff061bf5640 1 --2- 172.21.15.89:0/2744216818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05c102790 0x7ff05c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.265+0000 7ff061bf5640 1 --2- 172.21.15.89:0/2744216818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05c101de0 0x7ff05c1021c0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.265+0000 7ff061bf5640 1 -- 172.21.15.89:0/2744216818 >> 172.21.15.89:0/2744216818 conn(0x7ff05c0fd8c0 msgr2=0x7ff05c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.265+0000 7ff061bf5640 1 -- 172.21.15.89:0/2744216818 shutdown_connections 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.265+0000 7ff061bf5640 1 -- 172.21.15.89:0/2744216818 wait complete. 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.265+0000 7ff061bf5640 1 Processor -- start 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.265+0000 7ff061bf5640 1 -- start start 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff061bf5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05c101de0 0x7ff05c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff061bf5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c102790 0x7ff05c198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff061bf5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05c106f10 0x7ff05c19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff061bf5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c111f70 con 0x7ff05c106f10 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff061bf5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c111df0 con 0x7ff05c101de0 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff061bf5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c1120f0 con 0x7ff05c102790 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05c106f10 0x7ff05c19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05c106f10 0x7ff05c19cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38954/0 (socket says 172.21.15.89:38954) 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05bfff640 1 -- 172.21.15.89:0/1296734442 learned_addr learned my addr 172.21.15.89:0/1296734442 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05c101de0 0x7ff05c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05affd640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c102790 0x7ff05c198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05bfff640 1 -- 172.21.15.89:0/1296734442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c102790 msgr2=0x7ff05c198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05bfff640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c102790 0x7ff05c198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05bfff640 1 -- 172.21.15.89:0/1296734442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05c101de0 msgr2=0x7ff05c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05bfff640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05c101de0 0x7ff05c1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05bfff640 1 -- 172.21.15.89:0/1296734442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05c19d380 con 0x7ff05c106f10 2024-09-15T23:39:57.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05affd640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c102790 0x7ff05c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff05bfff640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05c106f10 0x7ff05c19cca0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7ff05400cce0 tx=0x7ff05400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff058ff9640 1 -- 172.21.15.89:0/1296734442 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05400f450 con 0x7ff05c106f10 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff058ff9640 1 -- 172.21.15.89:0/1296734442 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff05400fad0 con 0x7ff05c106f10 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff061bf5640 1 -- 172.21.15.89:0/1296734442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff05c19d670 con 0x7ff05c106f10 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff058ff9640 1 -- 172.21.15.89:0/1296734442 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff054018c10 con 0x7ff05c106f10 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.266+0000 7ff061bf5640 1 -- 172.21.15.89:0/1296734442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff05c1a4f50 con 0x7ff05c106f10 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff058ff9640 1 -- 172.21.15.89:0/1296734442 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff05400f5f0 con 0x7ff05c106f10 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff063fce740 1 Processor -- start 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff063fce740 1 -- start start 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff063fce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae11b60 0x5608dae11f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff063fce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae12480 0x5608dae1c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff063fce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae1cd40 0x5608dae1f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff063fce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5608dad77280 con 0x5608dae12480 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff063fce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5608dad1eb10 con 0x5608dae11b60 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff063fce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5608dad73240 con 0x5608dae1cd40 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff058ff9640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff0440a32c0 0x7ff0440a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff05b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae12480 0x5608dae1c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff05b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae12480 0x5608dae1c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38960/0 (socket says 172.21.15.89:38960) 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.269+0000 7ff05b7fe640 1 -- 172.21.15.89:0/957000444 learned_addr learned my addr 172.21.15.89:0/957000444 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05bfff640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff0440a32c0 0x7ff0440a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05affd640 1 --2- 172.21.15.89:0/957000444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae1cd40 0x5608dae1f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff058ff9640 1 -- 172.21.15.89:0/1296734442 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff0540d0370 con 0x7ff05c106f10 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05affd640 1 --2- 172.21.15.89:0/957000444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae11b60 0x5608dae11f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05b7fe640 1 -- 172.21.15.89:0/957000444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae1cd40 msgr2=0x5608dae1f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:57.286 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/957000444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae1cd40 0x5608dae1f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05b7fe640 1 -- 172.21.15.89:0/957000444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae11b60 msgr2=0x5608dae11f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/957000444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae11b60 0x5608dae11f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05b7fe640 1 -- 172.21.15.89:0/957000444 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608dad2b1f0 con 0x5608dae12480 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05affd640 1 --2- 172.21.15.89:0/957000444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae1cd40 0x5608dae1f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05bfff640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff0440a32c0 0x7ff0440a5780 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff05400ccb0 tx=0x7ff054066f30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/957000444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae12480 0x5608dae1c800 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7ff04c005ab0 tx=0x7ff04c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb168c233e545d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff041ffb640 1 -- 172.21.15.89:0/957000444 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c03f070 con 0x5608dae12480 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff041ffb640 1 -- 172.21.15.89:0/957000444 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff04c031a20 con 0x5608dae12480 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.270+0000 7ff041ffb640 1 -- 172.21.15.89:0/957000444 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c035070 con 0x5608dae12480 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.271+0000 7ff063fce740 1 -- 172.21.15.89:0/957000444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae12480 msgr2=0x5608dae1c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.271+0000 7ff063fce740 1 --2- 172.21.15.89:0/957000444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae12480 0x5608dae1c800 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7ff04c005ab0 tx=0x7ff04c031630 comp rx=0 tx=0).stop 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.271+0000 7ff063fce740 1 -- 172.21.15.89:0/957000444 shutdown_connections 2024-09-15T23:39:57.287 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.271+0000 7ff063fce740 1 --2- 172.21.15.89:0/957000444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae1cd40 0x5608dae1f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.271+0000 7ff063fce740 1 --2- 172.21.15.89:0/957000444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae12480 0x5608dae1c800 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.271+0000 7ff063fce740 1 --2- 172.21.15.89:0/957000444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae11b60 0x5608dae11f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.271+0000 7ff063fce740 1 -- 172.21.15.89:0/957000444 >> 172.21.15.89:0/957000444 conn(0x5608dae0e0f0 msgr2=0x5608dae0e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.271+0000 7ff063fce740 1 -- 172.21.15.89:0/957000444 shutdown_connections 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.271+0000 7ff063fce740 1 -- 172.21.15.89:0/957000444 wait complete. 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.271+0000 7ff063fce740 1 -- start start 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff063fce740 1 Processor -- start 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff063fce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 0x5608dae1c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff063fce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae19b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff063fce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae1a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff063fce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5608dae1f910 con 0x5608dae1cd40 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff063fce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5608dad31f10 con 0x5608dae12480 2024-09-15T23:39:57.288 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff063fce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5608dae1fa90 con 0x5608dae11b60 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff063fce740 10 client.0 _ll_register_callbacks cb 0x5608dacfaa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae1a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae1a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38968/0 (socket says 172.21.15.89:38968) 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05affd640 1 -- 172.21.15.89:0/2734512544 learned_addr learned my addr 172.21.15.89:0/2734512544 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae19b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05affd640 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 0x5608dae1c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05affd640 1 -- 172.21.15.89:0/2734512544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 msgr2=0x5608dae1c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05affd640 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 0x5608dae1c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05affd640 1 -- 172.21.15.89:0/2734512544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 msgr2=0x5608dae19b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05affd640 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae19b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05affd640 1 -- 172.21.15.89:0/2734512544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608daae86a0 con 0x5608dae1cd40 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae19b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:57.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff05affd640 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae1a2e0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7ff050012ec0 tx=0x7ff050004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1cf59bc20de7ba2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.272+0000 7ff03affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff03affd640 1 -- 172.21.15.89:0/2734512544 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05000f5d0 con 0x5608dae1cd40 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff03affd640 1 -- 172.21.15.89:0/2734512544 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff05000c910 con 0x5608dae1cd40 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff063fce740 1 -- 172.21.15.89:0/2734512544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff04c031a20 con 0x5608dae1cd40 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff03affd640 1 -- 172.21.15.89:0/2734512544 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff05000f770 con 0x5608dae1cd40 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff063fce740 10 client.20958 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff063fce740 1 -- 172.21.15.89:0/2734512544 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5608dae2b440 con 0x5608dae1cd40 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff03affd640 1 -- 172.21.15.89:0/2734512544 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff04c031a20 con 0x5608dae1cd40 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff03affd640 1 -- 172.21.15.89:0/2734512544 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff05000e990 con 0x5608dae1cd40 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff063fce740 10 client.20958 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff063fce740 10 client.20958 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:57.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff063fce740 20 client.20958 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff063fce740 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5608dae2c900 0x5608dae2ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff063fce740 4 client.20958 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.273+0000 7ff063fce740 1 -- 172.21.15.89:0/2734512544 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608dad847f0 con 0x5608dae2c900 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.274+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5608dae2c900 0x5608dae2ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.274+0000 7ff03affd640 10 client.20958 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.274+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5608dae2c900 0x5608dae2ecf0 crc :-1 s=READY pgs=414 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19450bbc22f73590 server_cookie=735378668cbeffab in_seq=0 out_seq=0 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff03affd640 1 -- 172.21.15.89:0/2734512544 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5608dad847f0 con 0x5608dae2c900 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff03affd640 10 client.20958 handle_command_reply: tid=0 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 client.20958 shutdown 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 -- 172.21.15.89:0/2734512544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 msgr2=0x5608dae1a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae1a2e0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7ff050012ec0 tx=0x7ff050004730 comp rx=0 tx=0).stop 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 -- 172.21.15.89:0/2734512544 shutdown_connections 2024-09-15T23:39:57.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5608dae2c900 0x5608dae2ecf0 crc :-1 s=READY pgs=414 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae1a2e0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae19b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 --2- 172.21.15.89:0/2734512544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 0x5608dae1c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff03affd640 0 client.20958 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 -- 172.21.15.89:0/2734512544 >> 172.21.15.89:0/2734512544 conn(0x5608dae0e0f0 msgr2=0x5608dae0d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 -- 172.21.15.89:0/2734512544 shutdown_connections 2024-09-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 1 -- 172.21.15.89:0/2734512544 wait complete. 2024-09-15T23:39:57.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.275+0000 7ff063fce740 20 client.20958 trim_cache size 0 max 16384 2024-09-15T23:39:57.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.292+0000 7ff063fce740 1 Processor -- start 2024-09-15T23:39:57.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.292+0000 7ff063fce740 1 -- start start 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.292+0000 7ff063fce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 0x5608dae19410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.292+0000 7ff063fce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae34580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.292+0000 7ff063fce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae34ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.292+0000 7ff063fce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c1aed20 con 0x5608dae1cd40 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.292+0000 7ff063fce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c1b0860 con 0x5608dae12480 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.292+0000 7ff063fce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c1af3f0 con 0x5608dae11b60 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 0x5608dae19410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 0x5608dae19410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:52996/0 (socket says 172.21.15.89:52996) 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05affd640 1 -- 172.21.15.89:0/739312763 learned_addr learned my addr 172.21.15.89:0/739312763 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05affd640 1 --2- 172.21.15.89:0/739312763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae34ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/739312763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae34580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.295 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05affd640 1 -- 172.21.15.89:0/739312763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 msgr2=0x5608dae19410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05affd640 1 --2- 172.21.15.89:0/739312763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 0x5608dae19410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05affd640 1 -- 172.21.15.89:0/739312763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 msgr2=0x5608dae34580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05affd640 1 --2- 172.21.15.89:0/739312763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae34580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05affd640 1 -- 172.21.15.89:0/739312763 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608dae2f060 con 0x5608dae1cd40 2024-09-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff05affd640 1 --2- 172.21.15.89:0/739312763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae34ac0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7ff050010eb0 tx=0x7ff050000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77de4a99e67021b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.293+0000 7ff0417fa640 1 -- 172.21.15.89:0/739312763 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050035070 con 0x5608dae1cd40 2024-09-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff0417fa640 1 -- 172.21.15.89:0/739312763 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff050005d50 con 0x5608dae1cd40 2024-09-15T23:39:57.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff0417fa640 1 -- 172.21.15.89:0/739312763 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05000e3c0 con 0x5608dae1cd40 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff063fce740 1 -- 172.21.15.89:0/739312763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 msgr2=0x5608dae34ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff063fce740 1 --2- 172.21.15.89:0/739312763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae34ac0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7ff050010eb0 tx=0x7ff050000bc0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff063fce740 1 -- 172.21.15.89:0/739312763 shutdown_connections 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff063fce740 1 --2- 172.21.15.89:0/739312763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae1cd40 0x5608dae34ac0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff063fce740 1 --2- 172.21.15.89:0/739312763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae34580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff063fce740 1 --2- 172.21.15.89:0/739312763 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae11b60 0x5608dae19410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff063fce740 1 -- 172.21.15.89:0/739312763 >> 172.21.15.89:0/739312763 conn(0x5608dae0e0f0 msgr2=0x5608dae18420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff063fce740 1 -- 172.21.15.89:0/739312763 shutdown_connections 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.294+0000 7ff063fce740 1 -- 172.21.15.89:0/739312763 wait complete. 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff063fce740 1 -- start start 2024-09-15T23:39:57.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff063fce740 1 Processor -- start 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff063fce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae11b60 0x5608dae92e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff063fce740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae93360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff063fce740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae1cd40 0x5608dae24f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff063fce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c1b0190 con 0x5608dae11b60 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff063fce740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c1afac0 con 0x5608dae12480 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff063fce740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c1acd00 con 0x5608dae1cd40 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff063fce740 10 client.0 _ll_register_callbacks cb 0x7ff05000c680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff05affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae11b60 0x5608dae92e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff05affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae11b60 0x5608dae92e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38972/0 (socket says 172.21.15.89:38972) 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff05affd640 1 -- 172.21.15.89:0/2577577358 learned_addr learned my addr 172.21.15.89:0/2577577358 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.295+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae93360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff05affd640 1 -- 172.21.15.89:0/2577577358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae1cd40 msgr2=0x5608dae24f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:57.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff05affd640 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae1cd40 0x5608dae24f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff05affd640 1 -- 172.21.15.89:0/2577577358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 msgr2=0x5608dae93360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff05affd640 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae93360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff05affd640 1 -- 172.21.15.89:0/2577577358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608dae25790 con 0x5608dae11b60 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff05affd640 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae11b60 0x5608dae92e20 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7ff050010620 tx=0x7ff050012e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91d9ade88d15a7a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff03affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff03affd640 1 -- 172.21.15.89:0/2577577358 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050035030 con 0x5608dae11b60 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff03affd640 1 -- 172.21.15.89:0/2577577358 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff05001b040 con 0x5608dae11b60 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff063fce740 1 -- 172.21.15.89:0/2577577358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff050005d50 con 0x5608dae11b60 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff03affd640 1 -- 172.21.15.89:0/2577577358 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff05000c970 con 0x5608dae11b60 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff063fce740 10 client.20964 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff063fce740 1 -- 172.21.15.89:0/2577577358 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5608dae90f80 con 0x5608dae11b60 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff03affd640 1 -- 172.21.15.89:0/2577577358 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff050005d50 con 0x5608dae11b60 2024-09-15T23:39:57.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.296+0000 7ff03affd640 1 -- 172.21.15.89:0/2577577358 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff050030670 con 0x5608dae11b60 2024-09-15T23:39:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.297+0000 7ff063fce740 10 client.20964 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.297+0000 7ff063fce740 10 client.20964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.297+0000 7ff063fce740 20 client.20964 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.297+0000 7ff063fce740 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5608daea05e0 0x5608daea29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.297+0000 7ff063fce740 4 client.20964 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:39:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.297+0000 7ff063fce740 1 -- 172.21.15.89:0/2577577358 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5608dad7ed00 con 0x5608daea05e0 2024-09-15T23:39:57.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.297+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5608daea05e0 0x5608daea29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:57.302 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:57 smithi089 ceph-mon[30933]: pgmap v827: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.4 KiB/s wr, 12 op/s 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.297+0000 7ff03affd640 10 client.20964 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.297+0000 7ff05b7fe640 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5608daea05e0 0x5608daea29d0 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dcb2bbf541f6d22 server_cookie=20dab46169866c09 in_seq=0 out_seq=0 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.298+0000 7ff03affd640 1 -- 172.21.15.89:0/2577577358 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5608dad7ed00 con 0x5608daea05e0 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.298+0000 7ff03affd640 10 client.20964 handle_command_reply: tid=0 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.298+0000 7ff063fce740 1 client.20964 shutdown 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 1 -- 172.21.15.89:0/2577577358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae11b60 msgr2=0x5608dae92e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae11b60 0x5608dae92e20 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7ff050010620 tx=0x7ff050012e20 comp rx=0 tx=0).stop 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 1 -- 172.21.15.89:0/2577577358 shutdown_connections 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5608daea05e0 0x5608daea29d0 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5608dae1cd40 0x5608dae24f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff03affd640 0 client.20964 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608dae12480 0x5608dae93360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 1 --2- 172.21.15.89:0/2577577358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608dae11b60 0x5608dae92e20 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 1 -- 172.21.15.89:0/2577577358 >> 172.21.15.89:0/2577577358 conn(0x5608dae0e0f0 msgr2=0x5608dae135b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 1 -- 172.21.15.89:0/2577577358 shutdown_connections 2024-09-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 1 -- 172.21.15.89:0/2577577358 wait complete. 2024-09-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff063fce740 20 client.20964 trim_cache size 0 max 16384 2024-09-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "df63a716-446c-4ad2-81ec-caaaf201a691", 2024-09-15T23:39:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:39:57.306 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:39:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff061bf5640 1 -- 172.21.15.89:0/1296734442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff0440a32c0 msgr2=0x7ff0440a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff061bf5640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff0440a32c0 0x7ff0440a5780 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff05400ccb0 tx=0x7ff054066f30 comp rx=0 tx=0).stop 2024-09-15T23:39:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff061bf5640 1 -- 172.21.15.89:0/1296734442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05c106f10 msgr2=0x7ff05c19cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.299+0000 7ff061bf5640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05c106f10 0x7ff05c19cca0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7ff05400cce0 tx=0x7ff05400a620 comp rx=0 tx=0).stop 2024-09-15T23:39:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.300+0000 7ff061bf5640 1 -- 172.21.15.89:0/1296734442 shutdown_connections 2024-09-15T23:39:57.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.300+0000 7ff061bf5640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff0440a32c0 0x7ff0440a5780 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.300+0000 7ff061bf5640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff05c106f10 0x7ff05c19cca0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.300+0000 7ff061bf5640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c102790 0x7ff05c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.300+0000 7ff061bf5640 1 --2- 172.21.15.89:0/1296734442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff05c101de0 0x7ff05c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.300+0000 7ff061bf5640 1 -- 172.21.15.89:0/1296734442 >> 172.21.15.89:0/1296734442 conn(0x7ff05c0fd8c0 msgr2=0x7ff05c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.300+0000 7ff061bf5640 1 -- 172.21.15.89:0/1296734442 shutdown_connections 2024-09-15T23:39:57.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:57.300+0000 7ff061bf5640 1 -- 172.21.15.89:0/1296734442 wait complete. 2024-09-15T23:39:57.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:57 smithi174 ceph-mon[35586]: pgmap v827: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.4 KiB/s wr, 12 op/s 2024-09-15T23:39:57.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:57 smithi120 ceph-mon[35250]: pgmap v827: 129 pgs: 129 active+clean; 3.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 4.4 KiB/s wr, 12 op/s 2024-09-15T23:39:58.114 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:39:58.400 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:39:58.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:58 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:39:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:58 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:58 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:39:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:58 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:58.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:58 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:39:58.521 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:58 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:39:58.521 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:58 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:58.522 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:58 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:39:58.522 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:58 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:58.522 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:58 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:39:58.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:58 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:39:58.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:58 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:58.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:58 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:39:58.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:58 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:39:58.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:58 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:39:58.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.898+0000 7fa7de0b1640 1 -- 172.21.15.89:0/3606506138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7d8101de0 msgr2=0x7fa7d81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:59.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.898+0000 7fa7de0b1640 1 --2- 172.21.15.89:0/3606506138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7d8101de0 0x7fa7d81021c0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8009900 tx=0x7fa7c802a410 comp rx=0 tx=0).stop 2024-09-15T23:39:59.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.898+0000 7fa7de0b1640 1 -- 172.21.15.89:0/3606506138 shutdown_connections 2024-09-15T23:39:59.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.898+0000 7fa7de0b1640 1 --2- 172.21.15.89:0/3606506138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7d8106f10 0x7fa7d810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.898+0000 7fa7de0b1640 1 --2- 172.21.15.89:0/3606506138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7d8102790 0x7fa7d81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.898+0000 7fa7de0b1640 1 --2- 172.21.15.89:0/3606506138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7d8101de0 0x7fa7d81021c0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.898+0000 7fa7de0b1640 1 -- 172.21.15.89:0/3606506138 >> 172.21.15.89:0/3606506138 conn(0x7fa7d80fd8c0 msgr2=0x7fa7d80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:59.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.898+0000 7fa7de0b1640 1 -- 172.21.15.89:0/3606506138 shutdown_connections 2024-09-15T23:39:59.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.898+0000 7fa7de0b1640 1 -- 172.21.15.89:0/3606506138 wait complete. 2024-09-15T23:39:59.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.898+0000 7fa7de0b1640 1 Processor -- start 2024-09-15T23:39:59.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7de0b1640 1 -- start start 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7de0b1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7d8101de0 0x7fa7d8198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7de0b1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7d8102790 0x7fa7d8198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7de0b1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7d8106f10 0x7fa7d819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7de0b1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d8111f70 con 0x7fa7d8102790 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7de0b1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d8111df0 con 0x7fa7d8101de0 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7de0b1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d81120f0 con 0x7fa7d8106f10 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7d8102790 0x7fa7d8198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7d8102790 0x7fa7d8198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38988/0 (socket says 172.21.15.89:38988) 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7cffff640 1 -- 172.21.15.89:0/289994046 learned_addr learned my addr 172.21.15.89:0/289994046 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7d8106f10 0x7fa7d819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7d77fe640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7d8101de0 0x7fa7d8198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7cffff640 1 -- 172.21.15.89:0/289994046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7d8106f10 msgr2=0x7fa7d819cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:59.121 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7cffff640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7d8106f10 0x7fa7d819cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7cffff640 1 -- 172.21.15.89:0/289994046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7d8101de0 msgr2=0x7fa7d8198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:59.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7cffff640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7d8101de0 0x7fa7d8198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7cffff640 1 -- 172.21.15.89:0/289994046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d819d3e0 con 0x7fa7d8102790 2024-09-15T23:39:59.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7cffff640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7d8102790 0x7fa7d8198970 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000c890 tx=0x7fa7c000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:59.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.899+0000 7fa7d57fa640 1 -- 172.21.15.89:0/289994046 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0004220 con 0x7fa7d8102790 2024-09-15T23:39:59.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.900+0000 7fa7de0b1640 1 -- 172.21.15.89:0/289994046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d819d6d0 con 0x7fa7d8102790 2024-09-15T23:39:59.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.900+0000 7fa7d57fa640 1 -- 172.21.15.89:0/289994046 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7c00043c0 con 0x7fa7d8102790 2024-09-15T23:39:59.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.900+0000 7fa7d57fa640 1 -- 172.21.15.89:0/289994046 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0004a00 con 0x7fa7d8102790 2024-09-15T23:39:59.122 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.900+0000 7fa7de0b1640 1 -- 172.21.15.89:0/289994046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d81a4fb0 con 0x7fa7d8102790 2024-09-15T23:39:59.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.902+0000 7fa7d57fa640 1 -- 172.21.15.89:0/289994046 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa7c00079b0 con 0x7fa7d8102790 2024-09-15T23:39:59.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.902+0000 7fa7e048a740 1 Processor -- start 2024-09-15T23:39:59.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.902+0000 7fa7e048a740 1 -- start start 2024-09-15T23:39:59.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.902+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9a4a0 0x562da4f9a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.902+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9adc0 0x562da4fa5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.902+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.132 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.902+0000 7fa7e048a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562da4f002e0 con 0x562da4fa5680 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.902+0000 7fa7e048a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562da4ea7ce0 con 0x562da4f9adc0 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.902+0000 7fa7e048a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562da4efc2a0 con 0x562da4f9a4a0 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.902+0000 7fa7cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39004/0 (socket says 172.21.15.89:39004) 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7d7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9a4a0 0x562da4f9a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cffff640 1 -- 172.21.15.89:0/311612977 learned_addr learned my addr 172.21.15.89:0/311612977 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7d77fe640 1 --2- 172.21.15.89:0/311612977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9adc0 0x562da4fa5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7d57fa640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7c40a3240 0x7fa7c40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cffff640 1 -- 172.21.15.89:0/311612977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9a4a0 msgr2=0x562da4f9a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cffff640 1 --2- 172.21.15.89:0/311612977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9a4a0 0x562da4f9a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cffff640 1 -- 172.21.15.89:0/311612977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9adc0 msgr2=0x562da4fa5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cffff640 1 --2- 172.21.15.89:0/311612977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9adc0 0x562da4fa5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.133 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cffff640 1 -- 172.21.15.89:0/311612977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562da4e8c440 con 0x562da4fa5680 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/311612977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9a4a0 0x562da4f9a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7c40a3240 0x7fa7c40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7d57fa640 1 -- 172.21.15.89:0/289994046 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa7c00cfa00 con 0x7fa7d8102790 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cffff640 1 --2- 172.21.15.89:0/311612977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa7a70 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0014f20 tx=0x7fa7c001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e9b5aa8b858076 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cdffb640 1 -- 172.21.15.89:0/311612977 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c009b770 con 0x562da4fa5680 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cdffb640 1 -- 172.21.15.89:0/311612977 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7c009c330 con 0x562da4fa5680 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.903+0000 7fa7cdffb640 1 -- 172.21.15.89:0/311612977 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c009b590 con 0x562da4fa5680 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7e048a740 1 -- 172.21.15.89:0/311612977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 msgr2=0x562da4fa7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7c40a3240 0x7fa7c40a5700 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009fd0 tx=0x7fa7d0004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7e048a740 1 --2- 172.21.15.89:0/311612977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa7a70 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0014f20 tx=0x7fa7c001f000 comp rx=0 tx=0).stop 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7e048a740 1 -- 172.21.15.89:0/311612977 shutdown_connections 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7e048a740 1 --2- 172.21.15.89:0/311612977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa7a70 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7e048a740 1 --2- 172.21.15.89:0/311612977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9adc0 0x562da4fa5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.134 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7e048a740 1 --2- 172.21.15.89:0/311612977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9a4a0 0x562da4f9a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7e048a740 1 -- 172.21.15.89:0/311612977 >> 172.21.15.89:0/311612977 conn(0x562da4f96a30 msgr2=0x562da4f96e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7e048a740 1 -- 172.21.15.89:0/311612977 shutdown_connections 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7e048a740 1 -- 172.21.15.89:0/311612977 wait complete. 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.904+0000 7fa7e048a740 1 -- start start 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7e048a740 1 Processor -- start 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fa4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9adc0 0x562da4fa2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7e048a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x562da4fa8250 con 0x562da4fa5680 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7e048a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562da4ebb050 con 0x562da4f9a4a0 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7e048a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562da4fa83d0 con 0x562da4f9adc0 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7e048a740 10 client.0 _ll_register_callbacks cb 0x562da4efb160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7d7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.135 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7d7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39012/0 (socket says 172.21.15.89:39012) 2024-09-15T23:39:59.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7d7fff640 1 -- 172.21.15.89:0/280532577 learned_addr learned my addr 172.21.15.89:0/280532577 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:59.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7d77fe640 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fa4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7cffff640 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9adc0 0x562da4fa2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7d7fff640 1 -- 172.21.15.89:0/280532577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9adc0 msgr2=0x562da4fa2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:59.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9adc0 0x562da4fa2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7d7fff640 1 -- 172.21.15.89:0/280532577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 msgr2=0x562da4fa4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:59.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fa4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.905+0000 7fa7d7fff640 1 -- 172.21.15.89:0/280532577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562da4c713c0 con 0x562da4fa5680 2024-09-15T23:39:59.136 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa27b0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000fc30 tx=0x7fa7d0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f3fd189f0d15e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/280532577 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d001d0d0 con 0x562da4fa5680 2024-09-15T23:39:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/280532577 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa7d0004b20 con 0x562da4fa5680 2024-09-15T23:39:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7e048a740 1 -- 172.21.15.89:0/280532577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7c009c330 con 0x562da4fa5680 2024-09-15T23:39:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/280532577 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7d0031050 con 0x562da4fa5680 2024-09-15T23:39:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7e048a740 10 client.20976 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7e048a740 1 -- 172.21.15.89:0/280532577 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562da4fb3c00 con 0x562da4fa5680 2024-09-15T23:39:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/280532577 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7c009c330 con 0x562da4fa5680 2024-09-15T23:39:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/280532577 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa7d00051e0 con 0x562da4fa5680 2024-09-15T23:39:59.137 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7e048a740 10 client.20976 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:59.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7e048a740 10 client.20976 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:59.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.906+0000 7fa7e048a740 20 client.20976 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:59.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.907+0000 7fa7e048a740 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562da4fb50c0 0x562da4fb7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.907+0000 7fa7e048a740 4 client.20976 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:39:59.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.907+0000 7fa7e048a740 1 -- 172.21.15.89:0/280532577 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562da4f0d850 con 0x562da4fb50c0 2024-09-15T23:39:59.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.907+0000 7fa7d77fe640 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562da4fb50c0 0x562da4fb7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.907+0000 7fa7b6ffd640 10 client.20976 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:59.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.907+0000 7fa7d77fe640 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562da4fb50c0 0x562da4fb7520 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a269cab315723ae server_cookie=d575eb608b8bd6d7 in_seq=0 out_seq=0 2024-09-15T23:39:59.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.908+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/280532577 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562da4f0d850 con 0x562da4fb50c0 2024-09-15T23:39:59.138 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.908+0000 7fa7b6ffd640 10 client.20976 handle_command_reply: tid=0 2024-09-15T23:39:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.908+0000 7fa7e048a740 1 client.20976 shutdown 2024-09-15T23:39:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.908+0000 7fa7e048a740 1 -- 172.21.15.89:0/280532577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 msgr2=0x562da4fa27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7e048a740 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa27b0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000fc30 tx=0x7fa7d0004ab0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7e048a740 1 -- 172.21.15.89:0/280532577 shutdown_connections 2024-09-15T23:39:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7e048a740 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562da4fb50c0 0x562da4fb7520 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7e048a740 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa27b0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7e048a740 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9adc0 0x562da4fa2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7e048a740 1 --2- 172.21.15.89:0/280532577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fa4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7e048a740 1 -- 172.21.15.89:0/280532577 >> 172.21.15.89:0/280532577 conn(0x562da4f96a30 msgr2=0x562da4f9c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:59.139 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7b6ffd640 0 client.20976 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7e048a740 1 -- 172.21.15.89:0/280532577 shutdown_connections 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7e048a740 1 -- 172.21.15.89:0/280532577 wait complete. 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.909+0000 7fa7e048a740 20 client.20976 trim_cache size 0 max 16384 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7e048a740 1 Processor -- start 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7e048a740 1 -- start start 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fb14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4f9adc0 0x562da4fa0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4fa5680 0x562da4f9d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7e048a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d81aeef0 con 0x562da4f9adc0 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7e048a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d81b0a30 con 0x562da4f9a4a0 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7e048a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d81af5c0 con 0x562da4fa5680 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4f9adc0 0x562da4fa0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.140 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7cffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4f9adc0 0x562da4fa0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39020/0 (socket says 172.21.15.89:39020) 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7cffff640 1 -- 172.21.15.89:0/1447908093 learned_addr learned my addr 172.21.15.89:0/1447908093 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7d77fe640 1 --2- 172.21.15.89:0/1447908093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fb14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/1447908093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4fa5680 0x562da4f9d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7cffff640 1 -- 172.21.15.89:0/1447908093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4fa5680 msgr2=0x562da4f9d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7cffff640 1 --2- 172.21.15.89:0/1447908093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4fa5680 0x562da4f9d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7cffff640 1 -- 172.21.15.89:0/1447908093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 msgr2=0x562da4fb14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7cffff640 1 --2- 172.21.15.89:0/1447908093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fb14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7cffff640 1 -- 172.21.15.89:0/1447908093 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562da4fa1220 con 0x562da4f9adc0 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.926+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/1447908093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4fa5680 0x562da4f9d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7cffff640 1 --2- 172.21.15.89:0/1447908093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4f9adc0 0x562da4fa0b40 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fa7c009b930 tx=0x7fa7c000a520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc524eec8e619f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7cd7fa640 1 -- 172.21.15.89:0/1447908093 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c001d070 con 0x562da4f9adc0 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7cd7fa640 1 -- 172.21.15.89:0/1447908093 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7c009c330 con 0x562da4f9adc0 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7cd7fa640 1 -- 172.21.15.89:0/1447908093 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c009ffe0 con 0x562da4f9adc0 2024-09-15T23:39:59.141 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7e048a740 1 -- 172.21.15.89:0/1447908093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4f9adc0 msgr2=0x562da4fa0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7e048a740 1 --2- 172.21.15.89:0/1447908093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4f9adc0 0x562da4fa0b40 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fa7c009b930 tx=0x7fa7c000a520 comp rx=0 tx=0).stop 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7e048a740 1 -- 172.21.15.89:0/1447908093 shutdown_connections 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7e048a740 1 --2- 172.21.15.89:0/1447908093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4fa5680 0x562da4f9d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7e048a740 1 --2- 172.21.15.89:0/1447908093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4f9adc0 0x562da4fa0b40 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7e048a740 1 --2- 172.21.15.89:0/1447908093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fb14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7e048a740 1 -- 172.21.15.89:0/1447908093 >> 172.21.15.89:0/1447908093 conn(0x562da4f96a30 msgr2=0x562da4f9d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7e048a740 1 -- 172.21.15.89:0/1447908093 shutdown_connections 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.927+0000 7fa7e048a740 1 -- 172.21.15.89:0/1447908093 wait complete. 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7e048a740 1 -- start start 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7e048a740 1 Processor -- start 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fa9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9adc0 0x562da4fa97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7e048a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.142 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7e048a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d81b0360 con 0x562da4fa5680 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7e048a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d81afc90 con 0x562da4f9a4a0 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7e048a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d81aced0 con 0x562da4f9adc0 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7e048a740 10 client.0 _ll_register_callbacks cb 0x7fa7d0009be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7d7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7d7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39034/0 (socket says 172.21.15.89:39034) 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7d7fff640 1 -- 172.21.15.89:0/1307496484 learned_addr learned my addr 172.21.15.89:0/1307496484 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.928+0000 7fa7d77fe640 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fa9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7cffff640 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9adc0 0x562da4fa97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7d7fff640 1 -- 172.21.15.89:0/1307496484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9adc0 msgr2=0x562da4fa97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9adc0 0x562da4fa97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7d7fff640 1 -- 172.21.15.89:0/1307496484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 msgr2=0x562da4fa9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fa9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.143 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7d7fff640 1 -- 172.21.15.89:0/1307496484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562da501ee10 con 0x562da4fa5680 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7d7fff640 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa9d10 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000fc00 tx=0x7fa7d00096b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1677c9e64af1a8d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/1307496484 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d001d0d0 con 0x562da4fa5680 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/1307496484 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa7d0023040 con 0x562da4fa5680 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7e048a740 1 -- 172.21.15.89:0/1307496484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7c009c330 con 0x562da4fa5680 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/1307496484 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7d001d270 con 0x562da4fa5680 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7e048a740 10 client.20982 fetch_fsmap learned FSMap version 29 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7e048a740 1 -- 172.21.15.89:0/1307496484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562da501ff90 con 0x562da4fa5680 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/1307496484 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7d0039020 con 0x562da4fa5680 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.929+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/1307496484 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa7d00050b0 con 0x562da4fa5680 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.930+0000 7fa7e048a740 10 client.20982 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.930+0000 7fa7e048a740 10 client.20982 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:39:59.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.930+0000 7fa7e048a740 20 client.20982 populate_metadata read hostname 'smithi089' 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.930+0000 7fa7e048a740 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562da50206e0 0x562da5022b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.930+0000 7fa7e048a740 4 client.20982 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.930+0000 7fa7e048a740 1 -- 172.21.15.89:0/1307496484 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fa7c009c330 con 0x562da50206e0 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.930+0000 7fa7d77fe640 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562da50206e0 0x562da5022b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.930+0000 7fa7b6ffd640 10 client.20982 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.930+0000 7fa7d77fe640 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562da50206e0 0x562da5022b00 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=755af4bdf789b98b server_cookie=a1f5fa68b0408f07 in_seq=0 out_seq=0 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7b6ffd640 1 -- 172.21.15.89:0/1307496484 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fa7c009c330 con 0x562da50206e0 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7b6ffd640 10 client.20982 handle_command_reply: tid=0 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 client.20982 shutdown 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 -- 172.21.15.89:0/1307496484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 msgr2=0x562da4fa9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa9d10 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000fc00 tx=0x7fa7d00096b0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 -- 172.21.15.89:0/1307496484 shutdown_connections 2024-09-15T23:39:59.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x562da50206e0 0x562da5022b00 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562da4fa5680 0x562da4fa9d10 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562da4f9adc0 0x562da4fa97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7b6ffd640 0 client.20982 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:39:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 --2- 172.21.15.89:0/1307496484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562da4f9a4a0 0x562da4fa9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 -- 172.21.15.89:0/1307496484 >> 172.21.15.89:0/1307496484 conn(0x562da4f96a30 msgr2=0x562da4f9d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 -- 172.21.15.89:0/1307496484 shutdown_connections 2024-09-15T23:39:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 1 -- 172.21.15.89:0/1307496484 wait complete. 2024-09-15T23:39:59.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7e048a740 20 client.20982 trim_cache size 0 max 16384 2024-09-15T23:39:59.146 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:39:59.146 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:39:59.147 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:39:59.147 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:39:59.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7de0b1640 1 -- 172.21.15.89:0/289994046 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7c40a3240 msgr2=0x7fa7c40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:59.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7de0b1640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7c40a3240 0x7fa7c40a5700 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0009fd0 tx=0x7fa7d0004a40 comp rx=0 tx=0).stop 2024-09-15T23:39:59.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7de0b1640 1 -- 172.21.15.89:0/289994046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7d8102790 msgr2=0x7fa7d8198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:39:59.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.931+0000 7fa7de0b1640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7d8102790 0x7fa7d8198970 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000c890 tx=0x7fa7c000cd50 comp rx=0 tx=0).stop 2024-09-15T23:39:59.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.932+0000 7fa7de0b1640 1 -- 172.21.15.89:0/289994046 shutdown_connections 2024-09-15T23:39:59.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.932+0000 7fa7de0b1640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa7c40a3240 0x7fa7c40a5700 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.932+0000 7fa7de0b1640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7d8106f10 0x7fa7d819cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.932+0000 7fa7de0b1640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa7d8102790 0x7fa7d8198970 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.932+0000 7fa7de0b1640 1 --2- 172.21.15.89:0/289994046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7d8101de0 0x7fa7d8198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:39:59.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.932+0000 7fa7de0b1640 1 -- 172.21.15.89:0/289994046 >> 172.21.15.89:0/289994046 conn(0x7fa7d80fd8c0 msgr2=0x7fa7d80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:39:59.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.932+0000 7fa7de0b1640 1 -- 172.21.15.89:0/289994046 shutdown_connections 2024-09-15T23:39:59.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:39:58.932+0000 7fa7de0b1640 1 -- 172.21.15.89:0/289994046 wait complete. 2024-09-15T23:39:59.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:39:59 smithi089 ceph-mon[30933]: pgmap v828: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.0 KiB/s wr, 16 op/s 2024-09-15T23:39:59.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:39:59 smithi120 ceph-mon[35250]: pgmap v828: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.0 KiB/s wr, 16 op/s 2024-09-15T23:39:59.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:39:59 smithi174 ceph-mon[35586]: pgmap v828: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.0 KiB/s wr, 16 op/s 2024-09-15T23:39:59.744 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:39:59.745 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:39:59.996 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:00.382 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:00 smithi089 ceph-mon[30933]: overall HEALTH_OK 2024-09-15T23:40:00.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 -- 172.21.15.89:0/2959829816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100e20 msgr2=0x7eff70101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 --2- 172.21.15.89:0/2959829816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100e20 0x7eff70101200 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7eff60009900 tx=0x7eff6002a480 comp rx=0 tx=0).stop 2024-09-15T23:40:00.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 -- 172.21.15.89:0/2959829816 shutdown_connections 2024-09-15T23:40:00.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 --2- 172.21.15.89:0/2959829816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff70102ef0 0x7eff7010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 --2- 172.21.15.89:0/2959829816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff70102570 0x7eff701029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 --2- 172.21.15.89:0/2959829816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100e20 0x7eff70101200 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 -- 172.21.15.89:0/2959829816 >> 172.21.15.89:0/2959829816 conn(0x7eff700fc7d0 msgr2=0x7eff700febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:00.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 -- 172.21.15.89:0/2959829816 shutdown_connections 2024-09-15T23:40:00.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 -- 172.21.15.89:0/2959829816 wait complete. 2024-09-15T23:40:00.492 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 Processor -- start 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.460+0000 7eff763e4640 1 -- start start 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff763e4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100e20 0x7eff70198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff763e4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff70102570 0x7eff70198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff763e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff70102ef0 0x7eff7019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff763e4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7eff701120c0 con 0x7eff70100e20 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff763e4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7eff70111f40 con 0x7eff70102570 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff763e4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7eff70112240 con 0x7eff70102ef0 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100e20 0x7eff70198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100e20 0x7eff70198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39056/0 (socket says 172.21.15.89:39056) 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6ffff640 1 -- 172.21.15.89:0/1013416132 learned_addr learned my addr 172.21.15.89:0/1013416132 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff74981640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff70102ef0 0x7eff7019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.493 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6f7fe640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff70102570 0x7eff70198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6ffff640 1 -- 172.21.15.89:0/1013416132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff70102ef0 msgr2=0x7eff7019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6ffff640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff70102ef0 0x7eff7019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6ffff640 1 -- 172.21.15.89:0/1013416132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff70102570 msgr2=0x7eff70198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6ffff640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff70102570 0x7eff70198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6ffff640 1 -- 172.21.15.89:0/1013416132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7019d4e0 con 0x7eff70100e20 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6f7fe640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff70102570 0x7eff70198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.461+0000 7eff6ffff640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100e20 0x7eff70198530 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7eff60006010 tx=0x7eff60003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.462+0000 7eff6d7fa640 1 -- 172.21.15.89:0/1013416132 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6003f070 con 0x7eff70100e20 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.462+0000 7eff6d7fa640 1 -- 172.21.15.89:0/1013416132 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff60031c30 con 0x7eff70100e20 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.462+0000 7eff763e4640 1 -- 172.21.15.89:0/1013416132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff7019d770 con 0x7eff70100e20 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.462+0000 7eff6d7fa640 1 -- 172.21.15.89:0/1013416132 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff60035030 con 0x7eff70100e20 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.462+0000 7eff763e4640 1 -- 172.21.15.89:0/1013416132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff701a50b0 con 0x7eff70100e20 2024-09-15T23:40:00.494 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.463+0000 7eff763e4640 1 -- 172.21.15.89:0/1013416132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff70101200 con 0x7eff70100e20 2024-09-15T23:40:00.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.464+0000 7eff6d7fa640 1 -- 172.21.15.89:0/1013416132 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7eff60042070 con 0x7eff70100e20 2024-09-15T23:40:00.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.464+0000 7eff6d7fa640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff580a3240 0x7eff580a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.465+0000 7eff6d7fa640 1 -- 172.21.15.89:0/1013416132 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7eff600f0570 con 0x7eff70100e20 2024-09-15T23:40:00.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.465+0000 7eff6f7fe640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff580a3240 0x7eff580a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.465+0000 7eff6f7fe640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff580a3240 0x7eff580a5700 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7eff64006fd0 tx=0x7eff64008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:00.495 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.469+0000 7eff6d7fa640 1 -- 172.21.15.89:0/1013416132 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7eff60031de0 con 0x7eff70100e20 2024-09-15T23:40:00.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.671+0000 7eff763e4640 1 -- 172.21.15.89:0/1013416132 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff701a5280 con 0x7eff70100e20 2024-09-15T23:40:00.672 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.671+0000 7eff6d7fa640 1 -- 172.21.15.89:0/1013416132 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7eff600bcea0 con 0x7eff70100e20 2024-09-15T23:40:00.673 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:40:00.673 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:40:00.673 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:40:00.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 -- 172.21.15.89:0/1013416132 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff580a3240 msgr2=0x7eff580a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff580a3240 0x7eff580a5700 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7eff64006fd0 tx=0x7eff64008040 comp rx=0 tx=0).stop 2024-09-15T23:40:00.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 -- 172.21.15.89:0/1013416132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100e20 msgr2=0x7eff70198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100e20 0x7eff70198530 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7eff60006010 tx=0x7eff60003c10 comp rx=0 tx=0).stop 2024-09-15T23:40:00.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 -- 172.21.15.89:0/1013416132 shutdown_connections 2024-09-15T23:40:00.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7eff580a3240 0x7eff580a5700 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff70102ef0 0x7eff7019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff70102570 0x7eff70198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 --2- 172.21.15.89:0/1013416132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff70100e20 0x7eff70198530 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 -- 172.21.15.89:0/1013416132 >> 172.21.15.89:0/1013416132 conn(0x7eff700fc7d0 msgr2=0x7eff700febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:00.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 -- 172.21.15.89:0/1013416132 shutdown_connections 2024-09-15T23:40:00.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:00.674+0000 7eff763e4640 1 -- 172.21.15.89:0/1013416132 wait complete. 2024-09-15T23:40:00.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:00 smithi120 ceph-mon[35250]: overall HEALTH_OK 2024-09-15T23:40:00.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:00 smithi174 ceph-mon[35586]: overall HEALTH_OK 2024-09-15T23:40:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.859+0000 7f0734ca6640 1 -- 172.21.15.89:0/1147995357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0730100a40 msgr2=0x7f0730100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.859+0000 7f0734ca6640 1 --2- 172.21.15.89:0/1147995357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0730100a40 0x7f0730100e40 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f0720009900 tx=0x7f072002a410 comp rx=0 tx=0).stop 2024-09-15T23:40:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.859+0000 7f0734ca6640 1 -- 172.21.15.89:0/1147995357 shutdown_connections 2024-09-15T23:40:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.859+0000 7f0734ca6640 1 --2- 172.21.15.89:0/1147995357 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07301025f0 0x7f0730108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.859+0000 7f0734ca6640 1 --2- 172.21.15.89:0/1147995357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0730101c30 0x7f07301020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.862 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.859+0000 7f0734ca6640 1 --2- 172.21.15.89:0/1147995357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0730100a40 0x7f0730100e40 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.859+0000 7f0734ca6640 1 -- 172.21.15.89:0/1147995357 >> 172.21.15.89:0/1147995357 conn(0x7f07300fc1e0 msgr2=0x7f07300fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.859+0000 7f0734ca6640 1 -- 172.21.15.89:0/1147995357 shutdown_connections 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.859+0000 7f0734ca6640 1 -- 172.21.15.89:0/1147995357 wait complete. 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.859+0000 7f0734ca6640 1 Processor -- start 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f0734ca6640 1 -- start start 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f0734ca6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0730100a40 0x7f0730197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f0734ca6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0730101c30 0x7f0730198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f0734ca6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07301025f0 0x7f073019f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f0734ca6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f073010c720 con 0x7f0730100a40 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f0734ca6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f073010c5a0 con 0x7f07301025f0 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f0734ca6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f073010c8a0 con 0x7f0730101c30 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0730100a40 0x7f0730197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.863 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072e59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0730100a40 0x7f0730197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39070/0 (socket says 172.21.15.89:39070) 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072e59c640 1 -- 172.21.15.89:0/103796423 learned_addr learned my addr 172.21.15.89:0/103796423 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072dd9b640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0730101c30 0x7f0730198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072ed9d640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07301025f0 0x7f073019f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072e59c640 1 -- 172.21.15.89:0/103796423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0730101c30 msgr2=0x7f0730198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072e59c640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0730101c30 0x7f0730198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072e59c640 1 -- 172.21.15.89:0/103796423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07301025f0 msgr2=0x7f073019f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072e59c640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07301025f0 0x7f073019f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072e59c640 1 -- 172.21.15.89:0/103796423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073019f9d0 con 0x7f0730100a40 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072ed9d640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07301025f0 0x7f073019f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.860+0000 7f072e59c640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0730100a40 0x7f0730197dc0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f0720031610 tx=0x7f07200043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.861+0000 7f071f7fe640 1 -- 172.21.15.89:0/103796423 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072003f030 con 0x7f0730100a40 2024-09-15T23:40:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.861+0000 7f0734ca6640 1 -- 172.21.15.89:0/103796423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073019fc60 con 0x7f0730100a40 2024-09-15T23:40:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.861+0000 7f071f7fe640 1 -- 172.21.15.89:0/103796423 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0720003ce0 con 0x7f0730100a40 2024-09-15T23:40:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.861+0000 7f071f7fe640 1 -- 172.21.15.89:0/103796423 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720035070 con 0x7f0730100a40 2024-09-15T23:40:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.861+0000 7f0734ca6640 1 -- 172.21.15.89:0/103796423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07301a0150 con 0x7f0730100a40 2024-09-15T23:40:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.863+0000 7f071f7fe640 1 -- 172.21.15.89:0/103796423 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0720042030 con 0x7f0730100a40 2024-09-15T23:40:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.863+0000 7f0737a38740 1 Processor -- start 2024-09-15T23:40:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f0737a38740 1 -- start start 2024-09-15T23:40:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f0737a38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fc1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f0737a38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc22e0 0x560543fc6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f0737a38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fc9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f0737a38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560543f252f0 con 0x560543fc7370 2024-09-15T23:40:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f0737a38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560543ecd270 con 0x560543fc22e0 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f0737a38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560543f212b0 con 0x560543fc19a0 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fc9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fc9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39076/0 (socket says 172.21.15.89:39076) 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072dd9b640 1 -- 172.21.15.89:0/1245459088 learned_addr learned my addr 172.21.15.89:0/1245459088 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072e59c640 1 --2- 172.21.15.89:0/1245459088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc22e0 0x560543fc6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f071f7fe640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07180a32f0 0x7f07180a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072ed9d640 1 --2- 172.21.15.89:0/1245459088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fc1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072ed9d640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07180a32f0 0x7f07180a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072dd9b640 1 -- 172.21.15.89:0/1245459088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 msgr2=0x560543fc1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072dd9b640 1 --2- 172.21.15.89:0/1245459088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fc1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072dd9b640 1 -- 172.21.15.89:0/1245459088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc22e0 msgr2=0x560543fc6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072dd9b640 1 --2- 172.21.15.89:0/1245459088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc22e0 0x560543fc6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f072dd9b640 1 -- 172.21.15.89:0/1245459088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560543ecc340 con 0x560543fc7370 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.864+0000 7f071f7fe640 1 -- 172.21.15.89:0/103796423 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f07200f0480 con 0x7f0730100a40 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.865+0000 7f072dd9b640 1 --2- 172.21.15.89:0/1245459088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fc9830 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f072400ad00 tx=0x7f072400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc3aaa93ae78576 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.865+0000 7f071cff9640 1 -- 172.21.15.89:0/1245459088 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072401d070 con 0x560543fc7370 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.865+0000 7f072ed9d640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07180a32f0 0x7f07180a57b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0728013870 tx=0x7f0728004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.865+0000 7f071cff9640 1 -- 172.21.15.89:0/1245459088 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0724018b20 con 0x560543fc7370 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.865+0000 7f071cff9640 1 -- 172.21.15.89:0/1245459088 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724010bc0 con 0x560543fc7370 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 -- 172.21.15.89:0/1245459088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 msgr2=0x560543fc9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 --2- 172.21.15.89:0/1245459088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fc9830 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f072400ad00 tx=0x7f072400f790 comp rx=0 tx=0).stop 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 -- 172.21.15.89:0/1245459088 shutdown_connections 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 --2- 172.21.15.89:0/1245459088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fc9830 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 --2- 172.21.15.89:0/1245459088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc22e0 0x560543fc6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 --2- 172.21.15.89:0/1245459088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fc1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 -- 172.21.15.89:0/1245459088 >> 172.21.15.89:0/1245459088 conn(0x560543fbd010 msgr2=0x560543fbd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 -- 172.21.15.89:0/1245459088 shutdown_connections 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 -- 172.21.15.89:0/1245459088 wait complete. 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 -- start start 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 Processor -- start 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc19a0 0x560543fcbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc22e0 0x560543fcc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x560543fca340 con 0x560543fc7370 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560543eae290 con 0x560543fc19a0 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.866+0000 7f0737a38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x560543fca4c0 con 0x560543fc22e0 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f0737a38740 10 client.0 _ll_register_callbacks cb 0x560543edf140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:00.875 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39088/0 (socket says 172.21.15.89:39088) 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072ed9d640 1 -- 172.21.15.89:0/417092642 learned_addr learned my addr 172.21.15.89:0/417092642 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072e59c640 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc19a0 0x560543fcbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072dd9b640 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc22e0 0x560543fcc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072ed9d640 1 -- 172.21.15.89:0/417092642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc22e0 msgr2=0x560543fcc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072ed9d640 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc22e0 0x560543fcc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072ed9d640 1 -- 172.21.15.89:0/417092642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc19a0 msgr2=0x560543fcbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072ed9d640 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc19a0 0x560543fcbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072ed9d640 1 -- 172.21.15.89:0/417092642 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560543c38210 con 0x560543fc7370 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f072ed9d640 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd3110 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f0728010f70 tx=0x7f0728004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999d3cc3dd8e45c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f0715ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f0715ffb640 1 -- 172.21.15.89:0/417092642 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0728022070 con 0x560543fc7370 2024-09-15T23:40:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f0715ffb640 1 -- 172.21.15.89:0/417092642 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f072801e800 con 0x560543fc7370 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f0737a38740 1 -- 172.21.15.89:0/417092642 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0724018b20 con 0x560543fc7370 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f0715ffb640 1 -- 172.21.15.89:0/417092642 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f072802e050 con 0x560543fc7370 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f0737a38740 10 client.21000 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.867+0000 7f0737a38740 1 -- 172.21.15.89:0/417092642 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560543d04620 con 0x560543fc7370 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f0715ffb640 1 -- 172.21.15.89:0/417092642 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0724018b20 con 0x560543fc7370 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f0715ffb640 1 -- 172.21.15.89:0/417092642 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f072801d070 con 0x560543fc7370 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f0737a38740 10 client.21000 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f0737a38740 10 client.21000 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f0737a38740 20 client.21000 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f0737a38740 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560543fd7880 0x560543fd9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f0737a38740 4 client.21000 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f0737a38740 1 -- 172.21.15.89:0/417092642 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560543f32a20 con 0x560543fd7880 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f072e59c640 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560543fd7880 0x560543fd9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f0715ffb640 10 client.21000 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.868+0000 7f072e59c640 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560543fd7880 0x560543fd9c70 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bec3d5c0db005da8 server_cookie=fbb2902421642c5e in_seq=0 out_seq=0 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0715ffb640 1 -- 172.21.15.89:0/417092642 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560543f32a20 con 0x560543fd7880 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0715ffb640 10 client.21000 handle_command_reply: tid=0 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0737a38740 1 client.21000 shutdown 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0737a38740 1 -- 172.21.15.89:0/417092642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 msgr2=0x560543fd3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0737a38740 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd3110 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f0728010f70 tx=0x7f0728004b50 comp rx=0 tx=0).stop 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0737a38740 1 -- 172.21.15.89:0/417092642 shutdown_connections 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0737a38740 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560543fd7880 0x560543fd9c70 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0737a38740 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd3110 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0737a38740 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc22e0 0x560543fcc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0715ffb640 0 client.21000 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0737a38740 1 --2- 172.21.15.89:0/417092642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc19a0 0x560543fcbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0737a38740 1 -- 172.21.15.89:0/417092642 >> 172.21.15.89:0/417092642 conn(0x560543fbd010 msgr2=0x560543fbe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.869+0000 7f0737a38740 1 -- 172.21.15.89:0/417092642 shutdown_connections 2024-09-15T23:40:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.870+0000 7f0737a38740 1 -- 172.21.15.89:0/417092642 wait complete. 2024-09-15T23:40:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.870+0000 7f0737a38740 20 client.21000 trim_cache size 0 max 16384 2024-09-15T23:40:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.898+0000 7f0737a38740 1 Processor -- start 2024-09-15T23:40:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f0737a38740 1 -- start start 2024-09-15T23:40:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f0737a38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fbc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f0737a38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc22e0 0x560543fd30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.905 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f0737a38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc7370 0x560543fde570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f0737a38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f07301aa8f0 con 0x560543fc22e0 2024-09-15T23:40:00.907 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f0737a38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07301ac430 con 0x560543fc7370 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f0737a38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07301aafc0 con 0x560543fc19a0 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc22e0 0x560543fd30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072dd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc22e0 0x560543fd30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39104/0 (socket says 172.21.15.89:39104) 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072dd9b640 1 -- 172.21.15.89:0/3409204230 learned_addr learned my addr 172.21.15.89:0/3409204230 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072e59c640 1 --2- 172.21.15.89:0/3409204230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fbc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072ed9d640 1 --2- 172.21.15.89:0/3409204230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc7370 0x560543fde570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072dd9b640 1 -- 172.21.15.89:0/3409204230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 msgr2=0x560543fbc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072dd9b640 1 --2- 172.21.15.89:0/3409204230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fbc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072dd9b640 1 -- 172.21.15.89:0/3409204230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc7370 msgr2=0x560543fde570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072dd9b640 1 --2- 172.21.15.89:0/3409204230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc7370 0x560543fde570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072dd9b640 1 -- 172.21.15.89:0/3409204230 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560543fbce30 con 0x560543fc22e0 2024-09-15T23:40:00.908 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.899+0000 7f072dd9b640 1 --2- 172.21.15.89:0/3409204230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc22e0 0x560543fd30d0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f072400ad00 tx=0x7f072400ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1000f4351d643f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0717fff640 1 -- 172.21.15.89:0/3409204230 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724027070 con 0x560543fc22e0 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0717fff640 1 -- 172.21.15.89:0/3409204230 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0724010e00 con 0x560543fc22e0 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0717fff640 1 -- 172.21.15.89:0/3409204230 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072401d040 con 0x560543fc22e0 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0737a38740 1 -- 172.21.15.89:0/3409204230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc22e0 msgr2=0x560543fd30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0737a38740 1 --2- 172.21.15.89:0/3409204230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc22e0 0x560543fd30d0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f072400ad00 tx=0x7f072400ac90 comp rx=0 tx=0).stop 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0737a38740 1 -- 172.21.15.89:0/3409204230 shutdown_connections 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0737a38740 1 --2- 172.21.15.89:0/3409204230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc7370 0x560543fde570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0737a38740 1 --2- 172.21.15.89:0/3409204230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc22e0 0x560543fd30d0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0737a38740 1 --2- 172.21.15.89:0/3409204230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fbc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0737a38740 1 -- 172.21.15.89:0/3409204230 >> 172.21.15.89:0/3409204230 conn(0x560543fbd010 msgr2=0x560543fbb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0737a38740 1 -- 172.21.15.89:0/3409204230 shutdown_connections 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.900+0000 7f0737a38740 1 -- 172.21.15.89:0/3409204230 wait complete. 2024-09-15T23:40:00.909 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f0737a38740 1 -- start start 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f0737a38740 1 Processor -- start 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f0737a38740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fd0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f0737a38740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc22e0 0x560543fd0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f0737a38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f0737a38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f07301abd60 con 0x560543fc7370 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f0737a38740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07301ab690 con 0x560543fc22e0 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f0737a38740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07301a88d0 con 0x560543fc19a0 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f0737a38740 10 client.0 _ll_register_callbacks cb 0x7f0728013450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f072ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f072ed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39110/0 (socket says 172.21.15.89:39110) 2024-09-15T23:40:00.910 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f072ed9d640 1 -- 172.21.15.89:0/700426757 learned_addr learned my addr 172.21.15.89:0/700426757 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f072dd9b640 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc22e0 0x560543fd0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f072e59c640 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fd0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f072ed9d640 1 -- 172.21.15.89:0/700426757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 msgr2=0x560543fd0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f072ed9d640 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fd0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f072ed9d640 1 -- 172.21.15.89:0/700426757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc22e0 msgr2=0x560543fd0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f072ed9d640 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc22e0 0x560543fd0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.901+0000 7f072ed9d640 1 -- 172.21.15.89:0/700426757 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560543fd2360 con 0x560543fc7370 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f072ed9d640 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd1d50 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f07280107b0 tx=0x7f0728010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e8fe124d5eff2e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0715ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0715ffb640 1 -- 172.21.15.89:0/700426757 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0728022040 con 0x560543fc7370 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0715ffb640 1 -- 172.21.15.89:0/700426757 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f07280267c0 con 0x560543fc7370 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0737a38740 1 -- 172.21.15.89:0/700426757 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0724010e00 con 0x560543fc7370 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0715ffb640 1 -- 172.21.15.89:0/700426757 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0728040050 con 0x560543fc7370 2024-09-15T23:40:00.911 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0737a38740 10 client.21006 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0737a38740 1 -- 172.21.15.89:0/700426757 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560544042550 con 0x560543fc7370 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0715ffb640 1 -- 172.21.15.89:0/700426757 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0724010e00 con 0x560543fc7370 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0715ffb640 1 -- 172.21.15.89:0/700426757 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f072801d050 con 0x560543fc7370 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0737a38740 10 client.21006 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0737a38740 10 client.21006 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0737a38740 20 client.21006 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0737a38740 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560544042ec0 0x5605440452e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0737a38740 4 client.21006 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.902+0000 7f0737a38740 1 -- 172.21.15.89:0/700426757 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560543f2cd70 con 0x560544042ec0 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.903+0000 7f072e59c640 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560544042ec0 0x5605440452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:00.912 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.903+0000 7f0715ffb640 10 client.21006 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:00.913 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.903+0000 7f072e59c640 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560544042ec0 0x5605440452e0 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56bd1e8bd94edeb server_cookie=df6d84e0a0ab01b6 in_seq=0 out_seq=0 2024-09-15T23:40:00.914 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0715ffb640 1 -- 172.21.15.89:0/700426757 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560543f2cd70 con 0x560544042ec0 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0715ffb640 10 client.21006 handle_command_reply: tid=0 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 client.21006 shutdown 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 -- 172.21.15.89:0/700426757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 msgr2=0x560543fd1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd1d50 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f07280107b0 tx=0x7f0728010c00 comp rx=0 tx=0).stop 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 -- 172.21.15.89:0/700426757 shutdown_connections 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x560544042ec0 0x5605440452e0 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560543fc7370 0x560543fd1d50 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0715ffb640 0 client.21006 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560543fc22e0 0x560543fd0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 --2- 172.21.15.89:0/700426757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x560543fc19a0 0x560543fd0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 -- 172.21.15.89:0/700426757 >> 172.21.15.89:0/700426757 conn(0x560543fbd010 msgr2=0x560543fbc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 -- 172.21.15.89:0/700426757 shutdown_connections 2024-09-15T23:40:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 1 -- 172.21.15.89:0/700426757 wait complete. 2024-09-15T23:40:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.913+0000 7f0737a38740 20 client.21006 trim_cache size 0 max 16384 2024-09-15T23:40:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:40:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:40:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:40:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:40:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 -- 172.21.15.89:0/103796423 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07180a32f0 msgr2=0x7f07180a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07180a32f0 0x7f07180a57b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0728013870 tx=0x7f0728004a40 comp rx=0 tx=0).stop 2024-09-15T23:40:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 -- 172.21.15.89:0/103796423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0730100a40 msgr2=0x7f0730197dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0730100a40 0x7f0730197dc0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f0720031610 tx=0x7f07200043b0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 -- 172.21.15.89:0/103796423 shutdown_connections 2024-09-15T23:40:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f07180a32f0 0x7f07180a57b0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07301025f0 0x7f073019f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0730101c30 0x7f0730198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 --2- 172.21.15.89:0/103796423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0730100a40 0x7f0730197dc0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:00.917 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 -- 172.21.15.89:0/103796423 >> 172.21.15.89:0/103796423 conn(0x7f07300fc1e0 msgr2=0x7f07300fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 -- 172.21.15.89:0/103796423 shutdown_connections 2024-09-15T23:40:00.918 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:00.914+0000 7f0734ca6640 1 -- 172.21.15.89:0/103796423 wait complete. 2024-09-15T23:40:01.418 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:40:01.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:01 smithi089 ceph-mon[30933]: pgmap v829: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 KiB/s wr, 13 op/s 2024-09-15T23:40:01.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:01 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1013416132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:01.670 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:01.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:01 smithi120 ceph-mon[35250]: pgmap v829: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 KiB/s wr, 13 op/s 2024-09-15T23:40:01.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:01 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1013416132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:01.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:01 smithi174 ceph-mon[35586]: pgmap v829: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 KiB/s wr, 13 op/s 2024-09-15T23:40:01.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:01 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1013416132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:02.202 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 -- 172.21.15.89:0/3109236577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7968101de0 msgr2=0x7f79681021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 --2- 172.21.15.89:0/3109236577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7968101de0 0x7f79681021c0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f7958009900 tx=0x7f795802a410 comp rx=0 tx=0).stop 2024-09-15T23:40:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 -- 172.21.15.89:0/3109236577 shutdown_connections 2024-09-15T23:40:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 --2- 172.21.15.89:0/3109236577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7968106f10 0x7f796810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 --2- 172.21.15.89:0/3109236577 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7968102790 0x7f79681067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 --2- 172.21.15.89:0/3109236577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7968101de0 0x7f79681021c0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 -- 172.21.15.89:0/3109236577 >> 172.21.15.89:0/3109236577 conn(0x7f79680fd8c0 msgr2=0x7f79680ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 -- 172.21.15.89:0/3109236577 shutdown_connections 2024-09-15T23:40:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 -- 172.21.15.89:0/3109236577 wait complete. 2024-09-15T23:40:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 Processor -- start 2024-09-15T23:40:02.203 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.184+0000 7f796e30d640 1 -- start start 2024-09-15T23:40:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796e30d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7968101de0 0x7f79681983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796e30d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7968102790 0x7f7968198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796e30d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7968106f10 0x7f796819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796e30d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f7968111f70 con 0x7f7968106f10 2024-09-15T23:40:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796e30d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7968111df0 con 0x7f7968101de0 2024-09-15T23:40:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796e30d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79681120f0 con 0x7f7968102790 2024-09-15T23:40:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796c8aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7968106f10 0x7f796819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796c8aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7968106f10 0x7f796819cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39126/0 (socket says 172.21.15.89:39126) 2024-09-15T23:40:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796c8aa640 1 -- 172.21.15.89:0/702273256 learned_addr learned my addr 172.21.15.89:0/702273256 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:02.204 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f7967fff640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7968101de0 0x7f79681983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f795ffff640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7968102790 0x7f7968198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796c8aa640 1 -- 172.21.15.89:0/702273256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7968102790 msgr2=0x7f7968198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796c8aa640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7968102790 0x7f7968198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796c8aa640 1 -- 172.21.15.89:0/702273256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7968101de0 msgr2=0x7f79681983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796c8aa640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7968101de0 0x7f79681983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796c8aa640 1 -- 172.21.15.89:0/702273256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796819d380 con 0x7f7968106f10 2024-09-15T23:40:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f795ffff640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7968102790 0x7f7968198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f7967fff640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7968101de0 0x7f79681983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:02.205 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796c8aa640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7968106f10 0x7f796819cca0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f796000cce0 tx=0x7f796000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f7965ffb640 1 -- 172.21.15.89:0/702273256 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796000f450 con 0x7f7968106f10 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f796e30d640 1 -- 172.21.15.89:0/702273256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f796819d670 con 0x7f7968106f10 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.185+0000 7f7965ffb640 1 -- 172.21.15.89:0/702273256 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f796000fad0 con 0x7f7968106f10 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.186+0000 7f7965ffb640 1 -- 172.21.15.89:0/702273256 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960018c10 con 0x7f7968106f10 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.186+0000 7f796e30d640 1 -- 172.21.15.89:0/702273256 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79681a4f50 con 0x7f7968106f10 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f7965ffb640 1 -- 172.21.15.89:0/702273256 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f796000f5f0 con 0x7f7968106f10 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f79706e6740 1 Processor -- start 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f79706e6740 1 -- start start 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f79706e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e93090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f79706e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5e93a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f79706e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5ea0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f79706e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dec5df92f0 con 0x55dec5e9e370 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f79706e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dec5da0cf0 con 0x55dec5e935d0 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f79706e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dec5df52b0 con 0x55dec5e94b40 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f7965ffb640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79540a3240 0x7f79540a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f795ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5ea0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f795ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5ea0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39128/0 (socket says 172.21.15.89:39128) 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f795ffff640 1 -- 172.21.15.89:0/3531580706 learned_addr learned my addr 172.21.15.89:0/3531580706 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.188+0000 7f796c8aa640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79540a3240 0x7f79540a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f7967fff640 1 --2- 172.21.15.89:0/3531580706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5e93a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f795ffff640 1 --2- 172.21.15.89:0/3531580706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e93090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f7965ffb640 1 -- 172.21.15.89:0/702273256 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f79600d0410 con 0x7f7968106f10 2024-09-15T23:40:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f795ffff640 1 -- 172.21.15.89:0/3531580706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 msgr2=0x55dec5e93090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f795ffff640 1 --2- 172.21.15.89:0/3531580706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e93090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f795ffff640 1 -- 172.21.15.89:0/3531580706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 msgr2=0x55dec5e93a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f795ffff640 1 --2- 172.21.15.89:0/3531580706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5e93a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f795ffff640 1 -- 172.21.15.89:0/3531580706 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dec5d853a0 con 0x55dec5e9e370 2024-09-15T23:40:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f795ffff640 1 --2- 172.21.15.89:0/3531580706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5ea0760 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f7950013610 tx=0x7f795000c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf7d68b486f804b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:02.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f796c8aa640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79540a3240 0x7f79540a5700 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7960099010 tx=0x7f7960066fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f795e7fc640 1 -- 172.21.15.89:0/3531580706 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950019070 con 0x55dec5e9e370 2024-09-15T23:40:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f795e7fc640 1 -- 172.21.15.89:0/3531580706 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7950004230 con 0x55dec5e9e370 2024-09-15T23:40:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.189+0000 7f795e7fc640 1 -- 172.21.15.89:0/3531580706 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950011910 con 0x55dec5e9e370 2024-09-15T23:40:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 -- 172.21.15.89:0/3531580706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 msgr2=0x55dec5ea0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 --2- 172.21.15.89:0/3531580706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5ea0760 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f7950013610 tx=0x7f795000c750 comp rx=0 tx=0).stop 2024-09-15T23:40:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 -- 172.21.15.89:0/3531580706 shutdown_connections 2024-09-15T23:40:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 --2- 172.21.15.89:0/3531580706 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5ea0760 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 --2- 172.21.15.89:0/3531580706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5e93a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 --2- 172.21.15.89:0/3531580706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e93090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 -- 172.21.15.89:0/3531580706 >> 172.21.15.89:0/3531580706 conn(0x55dec5e5c310 msgr2=0x55dec5e8fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 -- 172.21.15.89:0/3531580706 shutdown_connections 2024-09-15T23:40:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 -- 172.21.15.89:0/3531580706 wait complete. 2024-09-15T23:40:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 -- start start 2024-09-15T23:40:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.190+0000 7f79706e6740 1 Processor -- start 2024-09-15T23:40:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79706e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e935d0 0x55dec5e9dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79706e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e94b40 0x55dec5e9b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79706e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e9e370 0x55dec5e9b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79706e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55dec5ea1070 con 0x55dec5e935d0 2024-09-15T23:40:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79706e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dec5db4060 con 0x55dec5e94b40 2024-09-15T23:40:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79706e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55dec5ea11f0 con 0x55dec5e9e370 2024-09-15T23:40:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79706e6740 10 client.0 _ll_register_callbacks cb 0x55dec5b078c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f795ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e935d0 0x55dec5e9dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f795ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e935d0 0x55dec5e9dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39134/0 (socket says 172.21.15.89:39134) 2024-09-15T23:40:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f795ffff640 1 -- 172.21.15.89:0/1724687313 learned_addr learned my addr 172.21.15.89:0/1724687313 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f7967fff640 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e94b40 0x55dec5e9b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f795ffff640 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e9e370 0x55dec5e9b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f795ffff640 1 -- 172.21.15.89:0/1724687313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e9e370 msgr2=0x55dec5e9b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f795ffff640 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e9e370 0x55dec5e9b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f795ffff640 1 -- 172.21.15.89:0/1724687313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e94b40 msgr2=0x55dec5e9b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f795ffff640 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e94b40 0x55dec5e9b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f795ffff640 1 -- 172.21.15.89:0/1724687313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dec5b6a3c0 con 0x55dec5e935d0 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f795ffff640 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e935d0 0x55dec5e9dd70 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f7950003e30 tx=0x7f795000c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ef5c5a0b7510d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79477fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79477fe640 1 -- 172.21.15.89:0/1724687313 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950019070 con 0x55dec5e935d0 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79477fe640 1 -- 172.21.15.89:0/1724687313 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7950005040 con 0x55dec5e935d0 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.191+0000 7f79706e6740 1 -- 172.21.15.89:0/1724687313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7950004230 con 0x55dec5e935d0 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79477fe640 1 -- 172.21.15.89:0/1724687313 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7950033050 con 0x55dec5e935d0 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79706e6740 10 client.21018 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:02.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79706e6740 1 -- 172.21.15.89:0/1724687313 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dec5eacb10 con 0x55dec5e935d0 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79477fe640 1 -- 172.21.15.89:0/1724687313 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7950004230 con 0x55dec5e935d0 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79477fe640 1 -- 172.21.15.89:0/1724687313 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7950004c80 con 0x55dec5e935d0 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79706e6740 10 client.21018 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79706e6740 10 client.21018 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79706e6740 20 client.21018 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79706e6740 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55dec5eade60 0x55dec5eb0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79706e6740 4 client.21018 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.192+0000 7f79706e6740 1 -- 172.21.15.89:0/1724687313 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dec5e06860 con 0x55dec5eade60 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.193+0000 7f7967fff640 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55dec5eade60 0x55dec5eb0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.193+0000 7f7967fff640 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55dec5eade60 0x55dec5eb0250 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c904af6881208c64 server_cookie=1715882830e66e8a in_seq=0 out_seq=0 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.193+0000 7f79477fe640 10 client.21018 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:40:02.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79477fe640 1 -- 172.21.15.89:0/1724687313 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dec5e06860 con 0x55dec5eade60 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79477fe640 10 client.21018 handle_command_reply: tid=0 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 client.21018 shutdown 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 -- 172.21.15.89:0/1724687313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e935d0 msgr2=0x55dec5e9dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e935d0 0x55dec5e9dd70 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f7950003e30 tx=0x7f795000c4b0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 -- 172.21.15.89:0/1724687313 shutdown_connections 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55dec5eade60 0x55dec5eb0250 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e9e370 0x55dec5e9b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e94b40 0x55dec5e9b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 --2- 172.21.15.89:0/1724687313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e935d0 0x55dec5e9dd70 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79477fe640 0 client.21018 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 -- 172.21.15.89:0/1724687313 >> 172.21.15.89:0/1724687313 conn(0x55dec5e5c310 msgr2=0x55dec5e95960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:02.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 -- 172.21.15.89:0/1724687313 shutdown_connections 2024-09-15T23:40:02.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 1 -- 172.21.15.89:0/1724687313 wait complete. 2024-09-15T23:40:02.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.194+0000 7f79706e6740 20 client.21018 trim_cache size 0 max 16384 2024-09-15T23:40:02.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f79706e6740 1 Processor -- start 2024-09-15T23:40:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f79706e6740 1 -- start start 2024-09-15T23:40:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f79706e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5eaa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f79706e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e8aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f79706e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e9a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f79706e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79681aee30 con 0x55dec5e9e370 2024-09-15T23:40:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f79706e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f79681b0970 con 0x55dec5e935d0 2024-09-15T23:40:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f79706e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79681af500 con 0x55dec5e94b40 2024-09-15T23:40:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f795ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e9a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f795ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e9a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39140/0 (socket says 172.21.15.89:39140) 2024-09-15T23:40:02.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.207+0000 7f795ffff640 1 -- 172.21.15.89:0/3298135159 learned_addr learned my addr 172.21.15.89:0/3298135159 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.208+0000 7f7967fff640 1 --2- 172.21.15.89:0/3298135159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e8aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.208+0000 7f795ffff640 1 -- 172.21.15.89:0/3298135159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 msgr2=0x55dec5e8aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.208+0000 7f795ffff640 1 --2- 172.21.15.89:0/3298135159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e8aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.208+0000 7f795ffff640 1 -- 172.21.15.89:0/3298135159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 msgr2=0x55dec5eaa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.208+0000 7f795ffff640 1 --2- 172.21.15.89:0/3298135159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5eaa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.208+0000 7f795ffff640 1 -- 172.21.15.89:0/3298135159 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dec5e94760 con 0x55dec5e9e370 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.208+0000 7f795ffff640 1 --2- 172.21.15.89:0/3298135159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e9a8e0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f79500134a0 tx=0x7f7950000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984c198af713eaf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.208+0000 7f795dffb640 1 -- 172.21.15.89:0/3298135159 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950019030 con 0x55dec5e9e370 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.208+0000 7f795dffb640 1 -- 172.21.15.89:0/3298135159 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7950005510 con 0x55dec5e9e370 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.208+0000 7f795dffb640 1 -- 172.21.15.89:0/3298135159 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950011ca0 con 0x55dec5e9e370 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 -- 172.21.15.89:0/3298135159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 msgr2=0x55dec5e9a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 --2- 172.21.15.89:0/3298135159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e9a8e0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f79500134a0 tx=0x7f7950000bc0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 -- 172.21.15.89:0/3298135159 shutdown_connections 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 --2- 172.21.15.89:0/3298135159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e9a8e0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 --2- 172.21.15.89:0/3298135159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e8aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 --2- 172.21.15.89:0/3298135159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5eaa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 -- 172.21.15.89:0/3298135159 >> 172.21.15.89:0/3298135159 conn(0x55dec5e5c310 msgr2=0x55dec5e99910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 -- 172.21.15.89:0/3298135159 shutdown_connections 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 -- 172.21.15.89:0/3298135159 wait complete. 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 -- start start 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.209+0000 7f79706e6740 1 Processor -- start 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79706e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5e98700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79706e6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e98c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79706e6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e972f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79706e6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f79681b02a0 con 0x55dec5e9e370 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79706e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f79681afbd0 con 0x55dec5e935d0 2024-09-15T23:40:02.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79706e6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79681ace10 con 0x55dec5e94b40 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79706e6740 10 client.0 _ll_register_callbacks cb 0x7f7958009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f795ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f795ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39146/0 (socket says 172.21.15.89:39146) 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f795ffff640 1 -- 172.21.15.89:0/2560114422 learned_addr learned my addr 172.21.15.89:0/2560114422 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f795ffff640 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5e98700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f7967fff640 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e98c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f795ffff640 1 -- 172.21.15.89:0/2560114422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 msgr2=0x55dec5e98c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f795ffff640 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e98c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f795ffff640 1 -- 172.21.15.89:0/2560114422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 msgr2=0x55dec5e98700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f795ffff640 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5e98700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f795ffff640 1 -- 172.21.15.89:0/2560114422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dec5e97b20 con 0x55dec5e9e370 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f7967fff640 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e98c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f795ffff640 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e972f0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f795000ca40 tx=0x7f7950004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf82cd01eabaadec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:02.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79477fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79477fe640 1 -- 172.21.15.89:0/2560114422 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950019070 con 0x55dec5e9e370 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79706e6740 1 -- 172.21.15.89:0/2560114422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7950005510 con 0x55dec5e9e370 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.210+0000 7f79477fe640 1 -- 172.21.15.89:0/2560114422 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f7950011ca0 con 0x55dec5e9e370 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79477fe640 1 -- 172.21.15.89:0/2560114422 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7950013020 con 0x55dec5e9e370 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79706e6740 10 client.21024 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79706e6740 1 -- 172.21.15.89:0/2560114422 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dec5f1c4d0 con 0x55dec5e9e370 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79477fe640 1 -- 172.21.15.89:0/2560114422 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7950005510 con 0x55dec5e9e370 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79477fe640 1 -- 172.21.15.89:0/2560114422 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f795001e450 con 0x55dec5e9e370 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79706e6740 10 client.21024 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79706e6740 10 client.21024 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79706e6740 20 client.21024 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79706e6740 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55dec5f1cb80 0x55dec5f1efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:02.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79706e6740 4 client.21024 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f79706e6740 1 -- 172.21.15.89:0/2560114422 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dec5e00d70 con 0x55dec5f1cb80 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.211+0000 7f7967fff640 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55dec5f1cb80 0x55dec5f1efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79477fe640 10 client.21024 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f7967fff640 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55dec5f1cb80 0x55dec5f1efa0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9a00bd85e534c43f server_cookie=f7c3cd40d0a62d1 in_seq=0 out_seq=0 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79477fe640 1 -- 172.21.15.89:0/2560114422 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dec5e00d70 con 0x55dec5f1cb80 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79477fe640 10 client.21024 handle_command_reply: tid=0 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 client.21024 shutdown 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 -- 172.21.15.89:0/2560114422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 msgr2=0x55dec5e972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e972f0 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f795000ca40 tx=0x7f7950004140 comp rx=0 tx=0).stop 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 -- 172.21.15.89:0/2560114422 shutdown_connections 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x55dec5f1cb80 0x55dec5f1efa0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dec5e9e370 0x55dec5e972f0 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dec5e94b40 0x55dec5e98c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79477fe640 0 client.21024 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:40:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 --2- 172.21.15.89:0/2560114422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dec5e935d0 0x55dec5e98700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 -- 172.21.15.89:0/2560114422 >> 172.21.15.89:0/2560114422 conn(0x55dec5e5c310 msgr2=0x55dec5e9c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 -- 172.21.15.89:0/2560114422 shutdown_connections 2024-09-15T23:40:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 1 -- 172.21.15.89:0/2560114422 wait complete. 2024-09-15T23:40:02.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.212+0000 7f79706e6740 20 client.21024 trim_cache size 0 max 16384 2024-09-15T23:40:02.226 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:40:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 -- 172.21.15.89:0/702273256 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79540a3240 msgr2=0x7f79540a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79540a3240 0x7f79540a5700 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7960099010 tx=0x7f7960066fd0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 -- 172.21.15.89:0/702273256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7968106f10 msgr2=0x7f796819cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7968106f10 0x7f796819cca0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f796000cce0 tx=0x7f796000a620 comp rx=0 tx=0).stop 2024-09-15T23:40:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 -- 172.21.15.89:0/702273256 shutdown_connections 2024-09-15T23:40:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f79540a3240 0x7f79540a5700 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7968106f10 0x7f796819cca0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7968102790 0x7f7968198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 --2- 172.21.15.89:0/702273256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7968101de0 0x7f79681983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:02.227 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 -- 172.21.15.89:0/702273256 >> 172.21.15.89:0/702273256 conn(0x7f79680fd8c0 msgr2=0x7f79680febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:02.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 -- 172.21.15.89:0/702273256 shutdown_connections 2024-09-15T23:40:02.228 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:02.213+0000 7f796e30d640 1 -- 172.21.15.89:0/702273256 wait complete. 2024-09-15T23:40:02.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:02 smithi089 ceph-mon[30933]: pgmap v830: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 KiB/s wr, 15 op/s 2024-09-15T23:40:02.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:02 smithi120 ceph-mon[35250]: pgmap v830: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 KiB/s wr, 15 op/s 2024-09-15T23:40:02.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:02 smithi174 ceph-mon[35586]: pgmap v830: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 KiB/s wr, 15 op/s 2024-09-15T23:40:02.978 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:40:03.226 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3661436538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2a4101de0 msgr2=0x7ff2a41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 --2- 172.21.15.89:0/3661436538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2a4101de0 0x7ff2a41021c0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009900 tx=0x7ff2a002a410 comp rx=0 tx=0).stop 2024-09-15T23:40:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3661436538 shutdown_connections 2024-09-15T23:40:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 --2- 172.21.15.89:0/3661436538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a4106f10 0x7ff2a410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 --2- 172.21.15.89:0/3661436538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2a4102790 0x7ff2a41067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 --2- 172.21.15.89:0/3661436538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2a4101de0 0x7ff2a41021c0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3661436538 >> 172.21.15.89:0/3661436538 conn(0x7ff2a40fd8c0 msgr2=0x7ff2a40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3661436538 shutdown_connections 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3661436538 wait complete. 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 Processor -- start 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.733+0000 7ff2ab9d3640 1 -- start start 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2ab9d3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2a4101de0 0x7ff2a40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2ab9d3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a4102790 0x7ff2a4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2ab9d3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2a4106f10 0x7ff2a4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2ab9d3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a4111f30 con 0x7ff2a4106f10 2024-09-15T23:40:03.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2ab9d3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a4111db0 con 0x7ff2a4102790 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2ab9d3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a41120b0 con 0x7ff2a4101de0 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a9f70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2a4106f10 0x7ff2a4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a9f70640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2a4106f10 0x7ff2a4074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39170/0 (socket says 172.21.15.89:39170) 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a9f70640 1 -- 172.21.15.89:0/3560693656 learned_addr learned my addr 172.21.15.89:0/3560693656 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a976f640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2a4101de0 0x7ff2a40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a4102790 0x7ff2a4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a9f70640 1 -- 172.21.15.89:0/3560693656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2a4101de0 msgr2=0x7ff2a40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a9f70640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2a4101de0 0x7ff2a40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a9f70640 1 -- 172.21.15.89:0/3560693656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a4102790 msgr2=0x7ff2a4074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a9f70640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a4102790 0x7ff2a4074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a9f70640 1 -- 172.21.15.89:0/3560693656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a41a6980 con 0x7ff2a4106f10 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a976f640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2a4101de0 0x7ff2a40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:03.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a4102790 0x7ff2a4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff2a9f70640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2a4106f10 0x7ff2a4074f70 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7ff29c00eaa0 tx=0x7ff29c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.734+0000 7ff29a7fc640 1 -- 172.21.15.89:0/3560693656 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c00cdb0 con 0x7ff2a4106f10 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.735+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3560693656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2a41a6c70 con 0x7ff2a4106f10 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.735+0000 7ff29a7fc640 1 -- 172.21.15.89:0/3560693656 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff29c005020 con 0x7ff2a4106f10 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.735+0000 7ff29a7fc640 1 -- 172.21.15.89:0/3560693656 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c0116b0 con 0x7ff2a4106f10 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.735+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3560693656 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2a41a7160 con 0x7ff2a4106f10 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.737+0000 7ff29a7fc640 1 -- 172.21.15.89:0/3560693656 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff29c005560 con 0x7ff2a4106f10 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.737+0000 7ff2addac740 1 Processor -- start 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.737+0000 7ff2addac740 1 -- start start 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2addac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12eb40 0x55b07a12d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2addac740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 0x55b07a12da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2addac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a13a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2addac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b07a0932f0 con 0x55b07a138370 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2addac740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b07a03acf0 con 0x55b07a12d5d0 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2addac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b07a08f2b0 con 0x55b07a12eb40 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff29a7fc640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2900a3240 0x7ff2900a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a8f6e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a13a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a9f70640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2900a3240 0x7ff2900a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a8f6e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a13a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39176/0 (socket says 172.21.15.89:39176) 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/1372722848 learned_addr learned my addr 172.21.15.89:0/1372722848 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/1372722848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12eb40 0x55b07a12d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a976f640 1 --2- 172.21.15.89:0/1372722848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 0x55b07a12da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff29a7fc640 1 -- 172.21.15.89:0/3560693656 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff29c0cffc0 con 0x7ff2a4106f10 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/1372722848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12eb40 msgr2=0x55b07a12d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:03.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/1372722848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12eb40 0x55b07a12d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/1372722848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 msgr2=0x55b07a12da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/1372722848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 0x55b07a12da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/1372722848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b07a01f3a0 con 0x55b07a138370 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a9f70640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2900a3240 0x7ff2900a5700 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff29c0cda40 tx=0x7ff29c0c6fc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.738+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/1372722848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a13a760 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7ff294013890 tx=0x7ff29400b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a283ecc6fc4efa6a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff28f7fe640 1 -- 172.21.15.89:0/1372722848 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294004220 con 0x55b07a138370 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff28f7fe640 1 -- 172.21.15.89:0/1372722848 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff294017070 con 0x55b07a138370 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff28f7fe640 1 -- 172.21.15.89:0/1372722848 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2940038c0 con 0x55b07a138370 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff2addac740 1 -- 172.21.15.89:0/1372722848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 msgr2=0x55b07a13a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff2addac740 1 --2- 172.21.15.89:0/1372722848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a13a760 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7ff294013890 tx=0x7ff29400b440 comp rx=0 tx=0).stop 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff2addac740 1 -- 172.21.15.89:0/1372722848 shutdown_connections 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff2addac740 1 --2- 172.21.15.89:0/1372722848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a13a760 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff2addac740 1 --2- 172.21.15.89:0/1372722848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 0x55b07a12da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff2addac740 1 --2- 172.21.15.89:0/1372722848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12eb40 0x55b07a12d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff2addac740 1 -- 172.21.15.89:0/1372722848 >> 172.21.15.89:0/1372722848 conn(0x55b07a0f6310 msgr2=0x55b07a129c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.739+0000 7ff2addac740 1 -- 172.21.15.89:0/1372722848 shutdown_connections 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2addac740 1 -- 172.21.15.89:0/1372722848 wait complete. 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2addac740 1 -- start start 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2addac740 1 Processor -- start 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2addac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12d5d0 0x55b07a137d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2addac740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12eb40 0x55b07a134fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2addac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a1355d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2addac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55b07a13b070 con 0x55b07a138370 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2addac740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b07a04e060 con 0x55b07a12eb40 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2addac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b07a13b1f0 con 0x55b07a12d5d0 2024-09-15T23:40:03.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2addac740 10 client.0 _ll_register_callbacks cb 0x55b079da18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2a8f6e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a1355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2a8f6e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a1355d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39186/0 (socket says 172.21.15.89:39186) 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.740+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/3298245148 learned_addr learned my addr 172.21.15.89:0/3298245148 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12d5d0 0x55b07a137d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2a976f640 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12eb40 0x55b07a134fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/3298245148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12d5d0 msgr2=0x55b07a137d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12d5d0 0x55b07a137d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/3298245148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12eb40 msgr2=0x55b07a134fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12eb40 0x55b07a134fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/3298245148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b079e043c0 con 0x55b07a138370 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2a976f640 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12eb40 0x55b07a134fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a1355d0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7ff294005f50 tx=0x7ff29400bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30d6c5320539470 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:03.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff28cff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff28cff9640 1 -- 172.21.15.89:0/3298245148 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294005250 con 0x55b07a138370 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff28cff9640 1 -- 172.21.15.89:0/3298245148 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff2940053f0 con 0x55b07a138370 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2addac740 1 -- 172.21.15.89:0/3298245148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff294017070 con 0x55b07a138370 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff28cff9640 1 -- 172.21.15.89:0/3298245148 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff294035050 con 0x55b07a138370 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2addac740 10 client.21036 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff2addac740 1 -- 172.21.15.89:0/3298245148 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b07a146b00 con 0x55b07a138370 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.741+0000 7ff28cff9640 1 -- 172.21.15.89:0/3298245148 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff294017070 con 0x55b07a138370 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.742+0000 7ff28cff9640 1 -- 172.21.15.89:0/3298245148 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff29402d020 con 0x55b07a138370 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.742+0000 7ff2addac740 10 client.21036 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.742+0000 7ff2addac740 10 client.21036 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.742+0000 7ff2addac740 20 client.21036 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.742+0000 7ff2addac740 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b07a147ea0 0x55b07a14a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.742+0000 7ff2addac740 4 client.21036 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.742+0000 7ff2addac740 1 -- 172.21.15.89:0/3298245148 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b07a0a0860 con 0x55b07a147ea0 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.742+0000 7ff2a976f640 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b07a147ea0 0x55b07a14a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.742+0000 7ff28cff9640 10 client.21036 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.742+0000 7ff2a976f640 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b07a147ea0 0x55b07a14a290 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f1e7a145747ef8dc server_cookie=285924ef47aba6c5 in_seq=0 out_seq=0 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff28cff9640 1 -- 172.21.15.89:0/3298245148 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b07a0a0860 con 0x55b07a147ea0 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff28cff9640 10 client.21036 handle_command_reply: tid=0 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 client.21036 shutdown 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 -- 172.21.15.89:0/3298245148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 msgr2=0x55b07a1355d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a1355d0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7ff294005f50 tx=0x7ff29400bf50 comp rx=0 tx=0).stop 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 -- 172.21.15.89:0/3298245148 shutdown_connections 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b07a147ea0 0x55b07a14a290 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a1355d0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12eb40 0x55b07a134fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff28cff9640 0 client.21036 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:40:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 --2- 172.21.15.89:0/3298245148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12d5d0 0x55b07a137d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 -- 172.21.15.89:0/3298245148 >> 172.21.15.89:0/3298245148 conn(0x55b07a0f6310 msgr2=0x55b07a12f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 -- 172.21.15.89:0/3298245148 shutdown_connections 2024-09-15T23:40:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 1 -- 172.21.15.89:0/3298245148 wait complete. 2024-09-15T23:40:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.743+0000 7ff2addac740 20 client.21036 trim_cache size 0 max 16384 2024-09-15T23:40:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.756+0000 7ff2addac740 1 Processor -- start 2024-09-15T23:40:03.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.756+0000 7ff2addac740 1 -- start start 2024-09-15T23:40:03.782 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:40:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2addac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12d5d0 0x55b07a136f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2addac740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12eb40 0x55b07a14d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2addac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a14d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2addac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a41b10f0 con 0x55b07a138370 2024-09-15T23:40:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2addac740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a41b2c30 con 0x55b07a12eb40 2024-09-15T23:40:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2addac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a41b17c0 con 0x55b07a12d5d0 2024-09-15T23:40:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2a8f6e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a14d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2a8f6e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a14d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39194/0 (socket says 172.21.15.89:39194) 2024-09-15T23:40:03.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/952425752 learned_addr learned my addr 172.21.15.89:0/952425752 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/952425752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12d5d0 0x55b07a136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/952425752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12d5d0 msgr2=0x55b07a136f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/952425752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12d5d0 0x55b07a136f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/952425752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12eb40 msgr2=0x55b07a14d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/952425752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12eb40 0x55b07a14d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2a8f6e640 1 -- 172.21.15.89:0/952425752 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b07a137950 con 0x55b07a138370 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/952425752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a14d7f0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7ff29400eed0 tx=0x7ff29400e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0af5e58b00896d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff28effd640 1 -- 172.21.15.89:0/952425752 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29403d020 con 0x55b07a138370 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff28effd640 1 -- 172.21.15.89:0/952425752 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff294017040 con 0x55b07a138370 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.757+0000 7ff28effd640 1 -- 172.21.15.89:0/952425752 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff294004ed0 con 0x55b07a138370 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 -- 172.21.15.89:0/952425752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 msgr2=0x55b07a14d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:03.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 --2- 172.21.15.89:0/952425752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a14d7f0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7ff29400eed0 tx=0x7ff29400e9d0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 -- 172.21.15.89:0/952425752 shutdown_connections 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 --2- 172.21.15.89:0/952425752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a138370 0x55b07a14d7f0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 --2- 172.21.15.89:0/952425752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12eb40 0x55b07a14d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 --2- 172.21.15.89:0/952425752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a12d5d0 0x55b07a136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 -- 172.21.15.89:0/952425752 >> 172.21.15.89:0/952425752 conn(0x55b07a0f6310 msgr2=0x55b07a136840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 -- 172.21.15.89:0/952425752 shutdown_connections 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 -- 172.21.15.89:0/952425752 wait complete. 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 -- start start 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.758+0000 7ff2addac740 1 Processor -- start 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2addac740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 0x55b07a14d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2addac740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a12eb40 0x55b07a14dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2addac740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a138370 0x55b07a14ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2addac740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a41b2560 con 0x55b07a12eb40 2024-09-15T23:40:03.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2addac740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a41b1e90 con 0x55b07a12d5d0 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2addac740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a41af0d0 con 0x55b07a138370 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2addac740 10 client.0 _ll_register_callbacks cb 0x7ff2a0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a976f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a12eb40 0x55b07a14dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a976f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a12eb40 0x55b07a14dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:39196/0 (socket says 172.21.15.89:39196) 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a976f640 1 -- 172.21.15.89:0/2971983423 learned_addr learned my addr 172.21.15.89:0/2971983423 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a138370 0x55b07a14ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 0x55b07a14d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a976f640 1 -- 172.21.15.89:0/2971983423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a138370 msgr2=0x55b07a14ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a976f640 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a138370 0x55b07a14ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a976f640 1 -- 172.21.15.89:0/2971983423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 msgr2=0x55b07a14d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a976f640 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 0x55b07a14d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a976f640 1 -- 172.21.15.89:0/2971983423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b07a14b480 con 0x55b07a12eb40 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 0x55b07a14d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff2a976f640 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a12eb40 0x55b07a14dbf0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009900 tx=0x7ff2a00041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd2711f24bc9d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.759+0000 7ff28cff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff28cff9640 1 -- 172.21.15.89:0/2971983423 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0035040 con 0x55b07a12eb40 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff2addac740 1 -- 172.21.15.89:0/2971983423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff294017040 con 0x55b07a12eb40 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff28cff9640 1 -- 172.21.15.89:0/2971983423 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff2a0005040 con 0x55b07a12eb40 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff28cff9640 1 -- 172.21.15.89:0/2971983423 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2a0051050 con 0x55b07a12eb40 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff2addac740 10 client.21042 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff2addac740 1 -- 172.21.15.89:0/2971983423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b07a1b24c0 con 0x55b07a12eb40 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff28cff9640 1 -- 172.21.15.89:0/2971983423 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff294017040 con 0x55b07a12eb40 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff28cff9640 1 -- 172.21.15.89:0/2971983423 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff2a00035f0 con 0x55b07a12eb40 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff2addac740 10 client.21042 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff2addac740 10 client.21042 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:40:03.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff2addac740 20 client.21042 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff2addac740 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b07a1b2ca0 0x55b07a1b5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff2addac740 4 client.21042 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff2addac740 1 -- 172.21.15.89:0/2971983423 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b07a09ad70 con 0x55b07a1b2ca0 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.760+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b07a1b2ca0 0x55b07a1b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2a8f6e640 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b07a1b2ca0 0x55b07a1b5090 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cb83e4a9522636be server_cookie=5e6e51103f6047bb in_seq=0 out_seq=0 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff28cff9640 10 client.21042 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff28cff9640 1 -- 172.21.15.89:0/2971983423 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b07a09ad70 con 0x55b07a1b2ca0 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff28cff9640 10 client.21042 handle_command_reply: tid=0 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 client.21042 shutdown 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 -- 172.21.15.89:0/2971983423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a12eb40 msgr2=0x55b07a14dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a12eb40 0x55b07a14dbf0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009900 tx=0x7ff2a00041a0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 -- 172.21.15.89:0/2971983423 shutdown_connections 2024-09-15T23:40:03.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55b07a1b2ca0 0x55b07a1b5090 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b07a138370 0x55b07a14ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b07a12eb40 0x55b07a14dbf0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 --2- 172.21.15.89:0/2971983423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b07a12d5d0 0x55b07a14d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 -- 172.21.15.89:0/2971983423 >> 172.21.15.89:0/2971983423 conn(0x55b07a0f6310 msgr2=0x55b07a137060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff28cff9640 0 client.21042 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 -- 172.21.15.89:0/2971983423 shutdown_connections 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 1 -- 172.21.15.89:0/2971983423 wait complete. 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.761+0000 7ff2addac740 20 client.21042 trim_cache size 0 max 16384 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3560693656 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2900a3240 msgr2=0x7ff2900a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2900a3240 0x7ff2900a5700 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff29c0cda40 tx=0x7ff29c0c6fc0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3560693656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2a4106f10 msgr2=0x7ff2a4074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2a4106f10 0x7ff2a4074f70 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7ff29c00eaa0 tx=0x7ff29c00ef60 comp rx=0 tx=0).stop 2024-09-15T23:40:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3560693656 shutdown_connections 2024-09-15T23:40:03.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff2900a3240 0x7ff2900a5700 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff2a4106f10 0x7ff2a4074f70 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2a4102790 0x7ff2a4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 --2- 172.21.15.89:0/3560693656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2a4101de0 0x7ff2a40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:03.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3560693656 >> 172.21.15.89:0/3560693656 conn(0x7ff2a40fd8c0 msgr2=0x7ff2a40fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:03.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3560693656 shutdown_connections 2024-09-15T23:40:03.790 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:03.762+0000 7ff2ab9d3640 1 -- 172.21.15.89:0/3560693656 wait complete. 2024-09-15T23:40:04.522 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:40:04.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:04 smithi120 ceph-mon[35250]: pgmap v831: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.5 KiB/s wr, 16 op/s 2024-09-15T23:40:04.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:04 smithi174 ceph-mon[35586]: pgmap v831: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.5 KiB/s wr, 16 op/s 2024-09-15T23:40:04.768 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:04.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:04 smithi089 ceph-mon[30933]: pgmap v831: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.5 KiB/s wr, 16 op/s 2024-09-15T23:40:05.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.269+0000 7f6d33507640 1 -- 172.21.15.89:0/858415226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d2c106f50 msgr2=0x7f6d2c10eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:05.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.269+0000 7f6d33507640 1 --2- 172.21.15.89:0/858415226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d2c106f50 0x7f6d2c10eaf0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009cd0 tx=0x7f6d2402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.269+0000 7f6d33507640 1 -- 172.21.15.89:0/858415226 shutdown_connections 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.269+0000 7f6d33507640 1 --2- 172.21.15.89:0/858415226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d2c106f50 0x7f6d2c10eaf0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.269+0000 7f6d33507640 1 --2- 172.21.15.89:0/858415226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d2c100d00 0x7f6d2c106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.269+0000 7f6d33507640 1 --2- 172.21.15.89:0/858415226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c100350 0x7f6d2c100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.269+0000 7f6d33507640 1 -- 172.21.15.89:0/858415226 >> 172.21.15.89:0/858415226 conn(0x7f6d2c077210 msgr2=0x7f6d2c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d33507640 1 -- 172.21.15.89:0/858415226 shutdown_connections 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d33507640 1 -- 172.21.15.89:0/858415226 wait complete. 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d33507640 1 Processor -- start 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d33507640 1 -- start start 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d33507640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d2c100350 0x7f6d2c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d33507640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c100d00 0x7f6d2c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d33507640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d2c106f50 0x7f6d2c19cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d33507640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c112000 con 0x7f6d2c100350 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d33507640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c111e80 con 0x7f6d2c100d00 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d33507640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c112180 con 0x7f6d2c106f50 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d312a3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d2c100350 0x7f6d2c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d312a3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d2c100350 0x7f6d2c1984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34306/0 (socket says 172.21.15.89:34306) 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.270+0000 7f6d312a3640 1 -- 172.21.15.89:0/2425829773 learned_addr learned my addr 172.21.15.89:0/2425829773 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d312a3640 1 -- 172.21.15.89:0/2425829773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d2c106f50 msgr2=0x7f6d2c19cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d312a3640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d2c106f50 0x7f6d2c19cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d312a3640 1 -- 172.21.15.89:0/2425829773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c100d00 msgr2=0x7f6d2c198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d30aa2640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c100d00 0x7f6d2c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d312a3640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c100d00 0x7f6d2c198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d312a3640 1 -- 172.21.15.89:0/2425829773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2c19d4a0 con 0x7f6d2c100350 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d312a3640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d2c100350 0x7f6d2c1984f0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800c920 tx=0x7f6d2800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d227fc640 1 -- 172.21.15.89:0/2425829773 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d28007c70 con 0x7f6d2c100350 2024-09-15T23:40:05.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d33507640 1 -- 172.21.15.89:0/2425829773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2c19d790 con 0x7f6d2c100350 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d227fc640 1 -- 172.21.15.89:0/2425829773 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d28007e10 con 0x7f6d2c100350 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d227fc640 1 -- 172.21.15.89:0/2425829773 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d28018890 con 0x7f6d2c100350 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.271+0000 7f6d33507640 1 -- 172.21.15.89:0/2425829773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2c1a5070 con 0x7f6d2c100350 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d358e0740 1 Processor -- start 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d227fc640 1 -- 172.21.15.89:0/2425829773 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6d28018a70 con 0x7f6d2c100350 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d358e0740 1 -- start start 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3a8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3a85d0 0x56003c3a8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3b3370 0x56003c3b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d358e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56003c30e2f0 con 0x56003c3a9b40 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d358e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56003c2b5cf0 con 0x56003c3a85d0 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d358e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56003c30a2b0 con 0x56003c3b3370 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d31aa4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3a8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d31aa4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3a8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34322/0 (socket says 172.21.15.89:34322) 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d31aa4640 1 -- 172.21.15.89:0/368428674 learned_addr learned my addr 172.21.15.89:0/368428674 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d312a3640 1 --2- 172.21.15.89:0/368428674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3a85d0 0x56003c3a8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d30aa2640 1 --2- 172.21.15.89:0/368428674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3b3370 0x56003c3b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d227fc640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6d180a3240 0x7f6d180a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d31aa4640 1 -- 172.21.15.89:0/368428674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3b3370 msgr2=0x56003c3b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/368428674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3b3370 0x56003c3b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d31aa4640 1 -- 172.21.15.89:0/368428674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3a85d0 msgr2=0x56003c3a8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/368428674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3a85d0 0x56003c3a8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.274+0000 7f6d31aa4640 1 -- 172.21.15.89:0/368428674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56003c29a3a0 con 0x56003c3a9b40 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.275+0000 7f6d227fc640 1 -- 172.21.15.89:0/2425829773 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6d280cff00 con 0x7f6d2c100350 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.275+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6d180a3240 0x7f6d180a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.275+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/368428674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3a8090 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f6d24031500 tx=0x7f6d24031530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9cd62fa4f34b9d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:05.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.275+0000 7f6d177fe640 1 -- 172.21.15.89:0/368428674 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2404d070 con 0x56003c3a9b40 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.275+0000 7f6d177fe640 1 -- 172.21.15.89:0/368428674 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d24043070 con 0x56003c3a9b40 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.275+0000 7f6d177fe640 1 -- 172.21.15.89:0/368428674 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d24030d60 con 0x56003c3a9b40 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.275+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6d180a3240 0x7f6d180a5700 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009df0 tx=0x7f6d2404d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 -- 172.21.15.89:0/368428674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 msgr2=0x56003c3a8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 --2- 172.21.15.89:0/368428674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3a8090 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f6d24031500 tx=0x7f6d24031530 comp rx=0 tx=0).stop 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 -- 172.21.15.89:0/368428674 shutdown_connections 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 --2- 172.21.15.89:0/368428674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3b3370 0x56003c3b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 --2- 172.21.15.89:0/368428674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3a85d0 0x56003c3a8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 --2- 172.21.15.89:0/368428674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3a8090 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 -- 172.21.15.89:0/368428674 >> 172.21.15.89:0/368428674 conn(0x56003c371310 msgr2=0x56003c3a4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 -- 172.21.15.89:0/368428674 shutdown_connections 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 -- 172.21.15.89:0/368428674 wait complete. 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 -- start start 2024-09-15T23:40:05.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.276+0000 7f6d358e0740 1 Processor -- start 2024-09-15T23:40:05.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a85d0 0x56003c3b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a9b40 0x56003c3b0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 0x56003c3b0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d358e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56003c3b6070 con 0x56003c3a85d0 2024-09-15T23:40:05.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d358e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56003c2c9060 con 0x56003c3b3370 2024-09-15T23:40:05.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d358e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56003c3b61f0 con 0x56003c3a9b40 2024-09-15T23:40:05.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d358e0740 10 client.0 _ll_register_callbacks cb 0x56003c01c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:05.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d312a3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a85d0 0x56003c3b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d312a3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a85d0 0x56003c3b2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34330/0 (socket says 172.21.15.89:34330) 2024-09-15T23:40:05.306 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:40:05.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d312a3640 1 -- 172.21.15.89:0/2412077183 learned_addr learned my addr 172.21.15.89:0/2412077183 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:05.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d30aa2640 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a9b40 0x56003c3b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 0x56003c3b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d312a3640 1 -- 172.21.15.89:0/2412077183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a9b40 msgr2=0x56003c3b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:05.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d312a3640 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a9b40 0x56003c3b0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d312a3640 1 -- 172.21.15.89:0/2412077183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 msgr2=0x56003c3b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:05.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d312a3640 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 0x56003c3b0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d312a3640 1 -- 172.21.15.89:0/2412077183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56003c07f3c0 con 0x56003c3a85d0 2024-09-15T23:40:05.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 0x56003c3b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:05.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d312a3640 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a85d0 0x56003c3b2d90 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800c8f0 tx=0x7f6d2809bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4131c9e7d2b71ce3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:05.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:05.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2412077183 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800de20 con 0x56003c3a85d0 2024-09-15T23:40:05.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2412077183 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6d2800fb30 con 0x56003c3a85d0 2024-09-15T23:40:05.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d358e0740 1 -- 172.21.15.89:0/2412077183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d24043070 con 0x56003c3a85d0 2024-09-15T23:40:05.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2412077183 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6d2809c830 con 0x56003c3a85d0 2024-09-15T23:40:05.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d358e0740 10 client.21054 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.277+0000 7f6d358e0740 1 -- 172.21.15.89:0/2412077183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56003c3c1b40 con 0x56003c3a85d0 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.278+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2412077183 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d24043070 con 0x56003c3a85d0 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.278+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2412077183 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6d280a0120 con 0x56003c3a85d0 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.278+0000 7f6d358e0740 10 client.21054 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.278+0000 7f6d358e0740 10 client.21054 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.278+0000 7f6d358e0740 20 client.21054 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.278+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56003c3c2f50 0x56003c3c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.278+0000 7f6d358e0740 4 client.21054 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.278+0000 7f6d358e0740 1 -- 172.21.15.89:0/2412077183 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56003c31b860 con 0x56003c3c2f50 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.278+0000 7f6d30aa2640 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56003c3c2f50 0x56003c3c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.279+0000 7f6d30aa2640 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56003c3c2f50 0x56003c3c53a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8995e20b7801927f server_cookie=f54d4dfef745d677 in_seq=0 out_seq=0 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.279+0000 7f6d14ff9640 10 client.21054 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:40:05.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.279+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2412077183 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56003c31b860 con 0x56003c3c2f50 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.279+0000 7f6d14ff9640 10 client.21054 handle_command_reply: tid=0 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.279+0000 7f6d358e0740 1 client.21054 shutdown 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 1 -- 172.21.15.89:0/2412077183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a85d0 msgr2=0x56003c3b2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a85d0 0x56003c3b2d90 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800c8f0 tx=0x7f6d2809bfc0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 1 -- 172.21.15.89:0/2412077183 shutdown_connections 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 0x56003c3b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56003c3c2f50 0x56003c3c53a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a9b40 0x56003c3b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2412077183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a85d0 0x56003c3b2d90 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d14ff9640 0 client.21054 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 1 -- 172.21.15.89:0/2412077183 >> 172.21.15.89:0/2412077183 conn(0x56003c371310 msgr2=0x56003c3aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 1 -- 172.21.15.89:0/2412077183 shutdown_connections 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 1 -- 172.21.15.89:0/2412077183 wait complete. 2024-09-15T23:40:05.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.280+0000 7f6d358e0740 20 client.21054 trim_cache size 0 max 16384 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.292+0000 7f6d358e0740 1 Processor -- start 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d358e0740 1 -- start start 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a85d0 0x56003c3a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3a9b40 0x56003c3a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3b3370 0x56003c3ab120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d358e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c1aeeb0 con 0x56003c3b3370 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d358e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c1b09f0 con 0x56003c3a9b40 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d358e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c1af580 con 0x56003c3a85d0 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d31aa4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3b3370 0x56003c3ab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d31aa4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3b3370 0x56003c3ab120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34334/0 (socket says 172.21.15.89:34334) 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d31aa4640 1 -- 172.21.15.89:0/3668642059 learned_addr learned my addr 172.21.15.89:0/3668642059 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d312a3640 1 --2- 172.21.15.89:0/3668642059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a85d0 0x56003c3a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d30aa2640 1 --2- 172.21.15.89:0/3668642059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3a9b40 0x56003c3a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d31aa4640 1 -- 172.21.15.89:0/3668642059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a85d0 msgr2=0x56003c3a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:05.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/3668642059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a85d0 0x56003c3a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d31aa4640 1 -- 172.21.15.89:0/3668642059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3a9b40 msgr2=0x56003c3a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:05.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/3668642059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3a9b40 0x56003c3a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d31aa4640 1 -- 172.21.15.89:0/3668642059 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56003c3b3100 con 0x56003c3b3370 2024-09-15T23:40:05.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/3668642059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3b3370 0x56003c3ab120 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009a60 tx=0x7f6d240095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01c325335e9b66a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:05.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.293+0000 7f6d16ffd640 1 -- 172.21.15.89:0/3668642059 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d24058070 con 0x56003c3b3370 2024-09-15T23:40:05.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d16ffd640 1 -- 172.21.15.89:0/3668642059 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d24043070 con 0x56003c3b3370 2024-09-15T23:40:05.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d16ffd640 1 -- 172.21.15.89:0/3668642059 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d24030e00 con 0x56003c3b3370 2024-09-15T23:40:05.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d358e0740 1 -- 172.21.15.89:0/3668642059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3b3370 msgr2=0x56003c3ab120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d358e0740 1 --2- 172.21.15.89:0/3668642059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3b3370 0x56003c3ab120 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009a60 tx=0x7f6d240095c0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d358e0740 1 -- 172.21.15.89:0/3668642059 shutdown_connections 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d358e0740 1 --2- 172.21.15.89:0/3668642059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3b3370 0x56003c3ab120 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d358e0740 1 --2- 172.21.15.89:0/3668642059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3a9b40 0x56003c3a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d358e0740 1 --2- 172.21.15.89:0/3668642059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a85d0 0x56003c3a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d358e0740 1 -- 172.21.15.89:0/3668642059 >> 172.21.15.89:0/3668642059 conn(0x56003c371310 msgr2=0x56003c3a3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d358e0740 1 -- 172.21.15.89:0/3668642059 shutdown_connections 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.294+0000 7f6d358e0740 1 -- 172.21.15.89:0/3668642059 wait complete. 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d358e0740 1 -- start start 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d358e0740 1 Processor -- start 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a85d0 0x56003c3acf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3ad480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d358e0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 0x56003c3ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d358e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c1b0320 con 0x56003c3a9b40 2024-09-15T23:40:05.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d358e0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c1afc50 con 0x56003c3b3370 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d358e0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c1ace90 con 0x56003c3a85d0 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d358e0740 10 client.0 _ll_register_callbacks cb 0x7f6d2800cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d30aa2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d30aa2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3ad480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34340/0 (socket says 172.21.15.89:34340) 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d30aa2640 1 -- 172.21.15.89:0/2022245049 learned_addr learned my addr 172.21.15.89:0/2022245049 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d312a3640 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a85d0 0x56003c3acf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 0x56003c3ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d30aa2640 1 -- 172.21.15.89:0/2022245049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a85d0 msgr2=0x56003c3acf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d30aa2640 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a85d0 0x56003c3acf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d30aa2640 1 -- 172.21.15.89:0/2022245049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 msgr2=0x56003c3ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d30aa2640 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 0x56003c3ad9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d30aa2640 1 -- 172.21.15.89:0/2022245049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56003c42cc40 con 0x56003c3a9b40 2024-09-15T23:40:05.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.295+0000 7f6d31aa4640 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 0x56003c3ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d30aa2640 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3ad480 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c004810 tx=0x7f6d1c0157f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267ab6a76af12db0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d14ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2022245049 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c005dd0 con 0x56003c3a9b40 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2022245049 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6d1c006050 con 0x56003c3a9b40 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d358e0740 1 -- 172.21.15.89:0/2022245049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d24043070 con 0x56003c3a9b40 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2022245049 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6d1c02c050 con 0x56003c3a9b40 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d358e0740 10 client.21060 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d358e0740 1 -- 172.21.15.89:0/2022245049 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56003c42de80 con 0x56003c3a9b40 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2022245049 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d24043070 con 0x56003c3a9b40 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2022245049 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6d1c0248a0 con 0x56003c3a9b40 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d358e0740 10 client.21060 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d358e0740 10 client.21060 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d358e0740 20 client.21060 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56003c42e630 0x56003c430a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:05.315 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d358e0740 4 client.21060 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:40:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d358e0740 1 -- 172.21.15.89:0/2022245049 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56003c315d70 con 0x56003c42e630 2024-09-15T23:40:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.296+0000 7f6d312a3640 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56003c42e630 0x56003c430a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d312a3640 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56003c42e630 0x56003c430a50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa0786d5a0304e77 server_cookie=4d83a2184400790b in_seq=0 out_seq=0 2024-09-15T23:40:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d14ff9640 10 client.21060 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:40:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d14ff9640 1 -- 172.21.15.89:0/2022245049 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56003c315d70 con 0x56003c42e630 2024-09-15T23:40:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d14ff9640 10 client.21060 handle_command_reply: tid=0 2024-09-15T23:40:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 client.21060 shutdown 2024-09-15T23:40:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 -- 172.21.15.89:0/2022245049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 msgr2=0x56003c3ad480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:05.316 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3ad480 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c004810 tx=0x7f6d1c0157f0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 -- 172.21.15.89:0/2022245049 shutdown_connections 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56003c3b3370 0x56003c3ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56003c3a9b40 0x56003c3ad480 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x56003c42e630 0x56003c430a50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 --2- 172.21.15.89:0/2022245049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56003c3a85d0 0x56003c3acf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d14ff9640 0 client.21060 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 -- 172.21.15.89:0/2022245049 >> 172.21.15.89:0/2022245049 conn(0x56003c371310 msgr2=0x56003c3a3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 -- 172.21.15.89:0/2022245049 shutdown_connections 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 1 -- 172.21.15.89:0/2022245049 wait complete. 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.297+0000 7f6d358e0740 20 client.21060 trim_cache size 0 max 16384 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 -- 172.21.15.89:0/2425829773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6d180a3240 msgr2=0x7f6d180a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6d180a3240 0x7f6d180a5700 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f6d24009df0 tx=0x7f6d2404d000 comp rx=0 tx=0).stop 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 -- 172.21.15.89:0/2425829773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d2c100350 msgr2=0x7f6d2c1984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:05.317 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d2c100350 0x7f6d2c1984f0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800c920 tx=0x7f6d2800cde0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 -- 172.21.15.89:0/2425829773 shutdown_connections 2024-09-15T23:40:05.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6d180a3240 0x7f6d180a5700 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d2c106f50 0x7f6d2c19cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d2c100d00 0x7f6d2c198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 --2- 172.21.15.89:0/2425829773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d2c100350 0x7f6d2c1984f0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:05.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 -- 172.21.15.89:0/2425829773 >> 172.21.15.89:0/2425829773 conn(0x7f6d2c077210 msgr2=0x7f6d2c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:05.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 -- 172.21.15.89:0/2425829773 shutdown_connections 2024-09-15T23:40:05.318 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:05.298+0000 7f6d33507640 1 -- 172.21.15.89:0/2425829773 wait complete. 2024-09-15T23:40:06.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:40:06.286 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:06.686 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:06 smithi089 ceph-mon[30933]: pgmap v832: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.5 KiB/s wr, 14 op/s 2024-09-15T23:40:06.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:06 smithi120 ceph-mon[35250]: pgmap v832: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.5 KiB/s wr, 14 op/s 2024-09-15T23:40:06.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:06 smithi174 ceph-mon[35586]: pgmap v832: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.5 KiB/s wr, 14 op/s 2024-09-15T23:40:06.779 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.766+0000 7f428a56a640 1 -- 172.21.15.89:0/3199430387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4284106f10 msgr2=0x7f428410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.766+0000 7f428a56a640 1 --2- 172.21.15.89:0/3199430387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4284106f10 0x7f428410ea90 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f427c009ca0 tx=0x7f427c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.766+0000 7f428a56a640 1 -- 172.21.15.89:0/3199430387 shutdown_connections 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.766+0000 7f428a56a640 1 --2- 172.21.15.89:0/3199430387 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4284106f10 0x7f428410ea90 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.766+0000 7f428a56a640 1 --2- 172.21.15.89:0/3199430387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4284102790 0x7f42841067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.766+0000 7f428a56a640 1 --2- 172.21.15.89:0/3199430387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4284101de0 0x7f42841021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.766+0000 7f428a56a640 1 -- 172.21.15.89:0/3199430387 >> 172.21.15.89:0/3199430387 conn(0x7f42840fd8c0 msgr2=0x7f42840ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.766+0000 7f428a56a640 1 -- 172.21.15.89:0/3199430387 shutdown_connections 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.766+0000 7f428a56a640 1 -- 172.21.15.89:0/3199430387 wait complete. 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f428a56a640 1 Processor -- start 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f428a56a640 1 -- start start 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f428a56a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4284101de0 0x7f42841983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f428a56a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4284102790 0x7f4284198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f428a56a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4284106f10 0x7f428419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f428a56a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4284111f70 con 0x7f4284102790 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f428a56a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4284111df0 con 0x7f4284106f10 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f428a56a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42841120f0 con 0x7f4284101de0 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f42837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4284102790 0x7f4284198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f42837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4284102790 0x7f4284198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34362/0 (socket says 172.21.15.89:34362) 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f42837fe640 1 -- 172.21.15.89:0/3980708043 learned_addr learned my addr 172.21.15.89:0/3980708043 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.767+0000 7f4283fff640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4284101de0 0x7f42841983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f4288b07640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4284106f10 0x7f428419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f42837fe640 1 -- 172.21.15.89:0/3980708043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4284101de0 msgr2=0x7f42841983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f42837fe640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4284101de0 0x7f42841983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.781 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f42837fe640 1 -- 172.21.15.89:0/3980708043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4284106f10 msgr2=0x7f428419ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f42837fe640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4284106f10 0x7f428419ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f42837fe640 1 -- 172.21.15.89:0/3980708043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428419d3a0 con 0x7f4284102790 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f4288b07640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4284106f10 0x7f428419ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f42837fe640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4284102790 0x7f4284198930 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f427800c970 tx=0x7f427800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f42817fa640 1 -- 172.21.15.89:0/3980708043 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278007bb0 con 0x7f4284102790 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f428a56a640 1 -- 172.21.15.89:0/3980708043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428419d690 con 0x7f4284102790 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f42817fa640 1 -- 172.21.15.89:0/3980708043 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4278007d50 con 0x7f4284102790 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f42817fa640 1 -- 172.21.15.89:0/3980708043 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427800f450 con 0x7f4284102790 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.768+0000 7f428a56a640 1 -- 172.21.15.89:0/3980708043 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42841a4f70 con 0x7f4284102790 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f42817fa640 1 -- 172.21.15.89:0/3980708043 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4278005020 con 0x7f4284102790 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f428c943740 1 Processor -- start 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f428c943740 1 -- start start 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f428c943740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991e7b40 0x55d1991e6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.782 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f428c943740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 0x55d1991e6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f428c943740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991f1370 0x55d1991f3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f428c943740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d19914c2f0 con 0x55d1991f1370 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f428c943740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d1990f3cf0 con 0x55d1991e65d0 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f428c943740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d1991482b0 con 0x55d1991e7b40 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f42837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991f1370 0x55d1991f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f42837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991f1370 0x55d1991f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34372/0 (socket says 172.21.15.89:34372) 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f42837fe640 1 -- 172.21.15.89:0/1819837865 learned_addr learned my addr 172.21.15.89:0/1819837865 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.771+0000 7f42817fa640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f426c0a3240 0x7f426c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f4288b07640 1 --2- 172.21.15.89:0/1819837865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991e7b40 0x55d1991e6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f4283fff640 1 --2- 172.21.15.89:0/1819837865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 0x55d1991e6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f4288b07640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f426c0a3240 0x7f426c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.783 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f42837fe640 1 -- 172.21.15.89:0/1819837865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991e7b40 msgr2=0x55d1991e6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f42837fe640 1 --2- 172.21.15.89:0/1819837865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991e7b40 0x55d1991e6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f42837fe640 1 -- 172.21.15.89:0/1819837865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 msgr2=0x55d1991e6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f42837fe640 1 --2- 172.21.15.89:0/1819837865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 0x55d1991e6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f42837fe640 1 -- 172.21.15.89:0/1819837865 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1990d83a0 con 0x55d1991f1370 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f42817fa640 1 -- 172.21.15.89:0/3980708043 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f42780cfb20 con 0x7f4284102790 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f42837fe640 1 --2- 172.21.15.89:0/1819837865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991f1370 0x55d1991f3760 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f427809ba50 tx=0x7f427809b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a25b6fb7bb82cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f4288b07640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f426c0a3240 0x7f426c0a5700 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f427c005e30 tx=0x7f427c0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f426a7fc640 1 -- 172.21.15.89:0/1819837865 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427801d070 con 0x55d1991f1370 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f426a7fc640 1 -- 172.21.15.89:0/1819837865 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f427809c450 con 0x55d1991f1370 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.772+0000 7f426a7fc640 1 -- 172.21.15.89:0/1819837865 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278099e80 con 0x55d1991f1370 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.773+0000 7f428c943740 1 -- 172.21.15.89:0/1819837865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991f1370 msgr2=0x55d1991f3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.773+0000 7f428c943740 1 --2- 172.21.15.89:0/1819837865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991f1370 0x55d1991f3760 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f427809ba50 tx=0x7f427809b7e0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.784 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.773+0000 7f428c943740 1 -- 172.21.15.89:0/1819837865 shutdown_connections 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.773+0000 7f428c943740 1 --2- 172.21.15.89:0/1819837865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991f1370 0x55d1991f3760 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.773+0000 7f428c943740 1 --2- 172.21.15.89:0/1819837865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 0x55d1991e6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.773+0000 7f428c943740 1 --2- 172.21.15.89:0/1819837865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991e7b40 0x55d1991e6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.773+0000 7f428c943740 1 -- 172.21.15.89:0/1819837865 >> 172.21.15.89:0/1819837865 conn(0x55d1991af310 msgr2=0x55d1991e2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.773+0000 7f428c943740 1 -- 172.21.15.89:0/1819837865 shutdown_connections 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.773+0000 7f428c943740 1 -- 172.21.15.89:0/1819837865 wait complete. 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.773+0000 7f428c943740 1 -- start start 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f428c943740 1 Processor -- start 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f428c943740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 0x55d1991f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f428c943740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e7b40 0x55d1991edfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f428c943740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991ee5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f428c943740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55d1991f4010 con 0x55d1991e7b40 2024-09-15T23:40:06.785 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f428c943740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d199107060 con 0x55d1991e65d0 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f428c943740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d1991f4190 con 0x55d1991f1370 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f428c943740 10 client.0 _ll_register_callbacks cb 0x55d198e5a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f42837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e7b40 0x55d1991edfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f42837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e7b40 0x55d1991edfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34386/0 (socket says 172.21.15.89:34386) 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f42837fe640 1 -- 172.21.15.89:0/1605398382 learned_addr learned my addr 172.21.15.89:0/1605398382 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f4283fff640 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 0x55d1991f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f4288b07640 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991ee5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f42837fe640 1 -- 172.21.15.89:0/1605398382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 msgr2=0x55d1991ee5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f42837fe640 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991ee5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f42837fe640 1 -- 172.21.15.89:0/1605398382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 msgr2=0x55d1991f0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f42837fe640 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 0x55d1991f0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f42837fe640 1 -- 172.21.15.89:0/1605398382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d198ebd3c0 con 0x55d1991e7b40 2024-09-15T23:40:06.786 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f4283fff640 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 0x55d1991f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f42837fe640 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e7b40 0x55d1991edfe0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f4278098720 tx=0x7f427809a260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72568af0d61f3a31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.774+0000 7f42637fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f42637fe640 1 -- 172.21.15.89:0/1605398382 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427801d070 con 0x55d1991e7b40 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f42637fe640 1 -- 172.21.15.89:0/1605398382 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f427804aaa0 con 0x55d1991e7b40 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f428c943740 1 -- 172.21.15.89:0/1605398382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f427809c450 con 0x55d1991e7b40 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f42637fe640 1 -- 172.21.15.89:0/1605398382 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4278099e60 con 0x55d1991e7b40 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f428c943740 10 client.21072 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f428c943740 1 -- 172.21.15.89:0/1605398382 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d1991ffa70 con 0x55d1991e7b40 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f42637fe640 1 -- 172.21.15.89:0/1605398382 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f427809c450 con 0x55d1991e7b40 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f42637fe640 1 -- 172.21.15.89:0/1605398382 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f42780a1290 con 0x55d1991e7b40 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f428c943740 10 client.21072 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f428c943740 10 client.21072 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:40:06.787 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f428c943740 20 client.21072 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f428c943740 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d199200dc0 0x55d1992031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f428c943740 4 client.21072 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.775+0000 7f428c943740 1 -- 172.21.15.89:0/1605398382 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d199159860 con 0x55d199200dc0 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.776+0000 7f4283fff640 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d199200dc0 0x55d1992031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.776+0000 7f42637fe640 10 client.21072 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.776+0000 7f4283fff640 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d199200dc0 0x55d1992031b0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=21bb8e94ba2f22a server_cookie=ec2a8fb20bd963d6 in_seq=0 out_seq=0 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f42637fe640 1 -- 172.21.15.89:0/1605398382 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d199159860 con 0x55d199200dc0 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f42637fe640 10 client.21072 handle_command_reply: tid=0 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 client.21072 shutdown 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 -- 172.21.15.89:0/1605398382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e7b40 msgr2=0x55d1991edfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e7b40 0x55d1991edfe0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f4278098720 tx=0x7f427809a260 comp rx=0 tx=0).stop 2024-09-15T23:40:06.788 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 -- 172.21.15.89:0/1605398382 shutdown_connections 2024-09-15T23:40:06.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d199200dc0 0x55d1992031b0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991ee5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e7b40 0x55d1991edfe0 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 --2- 172.21.15.89:0/1605398382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e65d0 0x55d1991f0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 -- 172.21.15.89:0/1605398382 >> 172.21.15.89:0/1605398382 conn(0x55d1991af310 msgr2=0x55d1991e8990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:06.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f42637fe640 0 client.21072 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:40:06.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 -- 172.21.15.89:0/1605398382 shutdown_connections 2024-09-15T23:40:06.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 1 -- 172.21.15.89:0/1605398382 wait complete. 2024-09-15T23:40:06.789 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.777+0000 7f428c943740 20 client.21072 trim_cache size 0 max 16384 2024-09-15T23:40:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f428c943740 1 Processor -- start 2024-09-15T23:40:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f428c943740 1 -- start start 2024-09-15T23:40:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f428c943740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f428c943740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e7b40 0x55d1991ecfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f428c943740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991e9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f428c943740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42841acba0 con 0x55d1991e65d0 2024-09-15T23:40:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f428c943740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f42841ae6e0 con 0x55d1991e7b40 2024-09-15T23:40:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f428c943740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42841ad270 con 0x55d1991f1370 2024-09-15T23:40:06.794 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f4283fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f4283fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991fd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34402/0 (socket says 172.21.15.89:34402) 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.790+0000 7f4283fff640 1 -- 172.21.15.89:0/2370866029 learned_addr learned my addr 172.21.15.89:0/2370866029 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f42837fe640 1 --2- 172.21.15.89:0/2370866029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e7b40 0x55d1991ecfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4288b07640 1 --2- 172.21.15.89:0/2370866029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991e9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4283fff640 1 -- 172.21.15.89:0/2370866029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 msgr2=0x55d1991e9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4283fff640 1 --2- 172.21.15.89:0/2370866029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991e9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4283fff640 1 -- 172.21.15.89:0/2370866029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e7b40 msgr2=0x55d1991ecfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4283fff640 1 --2- 172.21.15.89:0/2370866029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e7b40 0x55d1991ecfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4283fff640 1 -- 172.21.15.89:0/2370866029 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1991edf80 con 0x55d1991e65d0 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4288b07640 1 --2- 172.21.15.89:0/2370866029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991e9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4283fff640 1 --2- 172.21.15.89:0/2370866029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991fd2b0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f427400b370 tx=0x7f427400ba80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610e7ac37702f12a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4269ffb640 1 -- 172.21.15.89:0/2370866029 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42740042f0 con 0x55d1991e65d0 2024-09-15T23:40:06.795 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4269ffb640 1 -- 172.21.15.89:0/2370866029 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4274004490 con 0x55d1991e65d0 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.791+0000 7f4269ffb640 1 -- 172.21.15.89:0/2370866029 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427401db70 con 0x55d1991e65d0 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 -- 172.21.15.89:0/2370866029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 msgr2=0x55d1991fd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 --2- 172.21.15.89:0/2370866029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991fd2b0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f427400b370 tx=0x7f427400ba80 comp rx=0 tx=0).stop 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 -- 172.21.15.89:0/2370866029 shutdown_connections 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 --2- 172.21.15.89:0/2370866029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991e9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 --2- 172.21.15.89:0/2370866029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e7b40 0x55d1991ecfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 --2- 172.21.15.89:0/2370866029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991fd2b0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 -- 172.21.15.89:0/2370866029 >> 172.21.15.89:0/2370866029 conn(0x55d1991af310 msgr2=0x55d1991ec8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 -- 172.21.15.89:0/2370866029 shutdown_connections 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 -- 172.21.15.89:0/2370866029 wait complete. 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 -- start start 2024-09-15T23:40:06.796 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 Processor -- start 2024-09-15T23:40:06.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991f9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e7b40 0x55d1991f9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991ea630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f42841ae010 con 0x55d1991e65d0 2024-09-15T23:40:06.799 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f42841ad940 con 0x55d1991e7b40 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.792+0000 7f428c943740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42841aab80 con 0x55d1991f1370 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f428c943740 10 client.0 _ll_register_callbacks cb 0x7f427809b6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f4283fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991f9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f4283fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991f9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34404/0 (socket says 172.21.15.89:34404) 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f4283fff640 1 -- 172.21.15.89:0/1556063419 learned_addr learned my addr 172.21.15.89:0/1556063419 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f4288b07640 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991ea630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f42837fe640 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e7b40 0x55d1991f9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f4283fff640 1 -- 172.21.15.89:0/1556063419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 msgr2=0x55d1991ea630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f4283fff640 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991ea630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f4283fff640 1 -- 172.21.15.89:0/1556063419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e7b40 msgr2=0x55d1991f9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f4283fff640 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e7b40 0x55d1991f9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f4283fff640 1 -- 172.21.15.89:0/1556063419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1991eacd0 con 0x55d1991e65d0 2024-09-15T23:40:06.800 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f4283fff640 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991f9600 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f4274005780 tx=0x7f42740106e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37bd76b8108535a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f42637fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f42637fe640 1 -- 172.21.15.89:0/1556063419 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274034060 con 0x55d1991e65d0 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f428c943740 1 -- 172.21.15.89:0/1556063419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4274004490 con 0x55d1991e65d0 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f42637fe640 1 -- 172.21.15.89:0/1556063419 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f427401daf0 con 0x55d1991e65d0 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f42637fe640 1 -- 172.21.15.89:0/1556063419 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f427403b050 con 0x55d1991e65d0 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f428c943740 10 client.21078 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.793+0000 7f428c943740 1 -- 172.21.15.89:0/1556063419 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d19926b070 con 0x55d1991e65d0 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.794+0000 7f42637fe640 1 -- 172.21.15.89:0/1556063419 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4274004490 con 0x55d1991e65d0 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.794+0000 7f42637fe640 1 -- 172.21.15.89:0/1556063419 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4274026410 con 0x55d1991e65d0 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.794+0000 7f428c943740 10 client.21078 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.794+0000 7f428c943740 10 client.21078 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.794+0000 7f428c943740 20 client.21078 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.794+0000 7f428c943740 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d19926b680 0x55d19926da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:06.801 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.794+0000 7f428c943740 4 client.21078 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.794+0000 7f428c943740 1 -- 172.21.15.89:0/1556063419 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d199153d70 con 0x55d19926b680 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.794+0000 7f42837fe640 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d19926b680 0x55d19926da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f42637fe640 10 client.21078 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f42837fe640 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d19926b680 0x55d19926da70 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb2cdcb92fedd95f server_cookie=c7426c131684535f in_seq=0 out_seq=0 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f42637fe640 1 -- 172.21.15.89:0/1556063419 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d199153d70 con 0x55d19926b680 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f42637fe640 10 client.21078 handle_command_reply: tid=0 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 client.21078 shutdown 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 -- 172.21.15.89:0/1556063419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 msgr2=0x55d1991f9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991f9600 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f4274005780 tx=0x7f42740106e0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 -- 172.21.15.89:0/1556063419 shutdown_connections 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55d19926b680 0x55d19926da70 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d1991f1370 0x55d1991ea630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f42637fe640 0 client.21078 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:40:06.802 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d1991e7b40 0x55d1991f9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 --2- 172.21.15.89:0/1556063419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1991e65d0 0x55d1991f9600 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 -- 172.21.15.89:0/1556063419 >> 172.21.15.89:0/1556063419 conn(0x55d1991af310 msgr2=0x55d1991ef840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:06.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 -- 172.21.15.89:0/1556063419 shutdown_connections 2024-09-15T23:40:06.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 1 -- 172.21.15.89:0/1556063419 wait complete. 2024-09-15T23:40:06.803 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.795+0000 7f428c943740 20 client.21078 trim_cache size 0 max 16384 2024-09-15T23:40:06.803 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 -- 172.21.15.89:0/3980708043 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f426c0a3240 msgr2=0x7f426c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f426c0a3240 0x7f426c0a5700 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f427c005e30 tx=0x7f427c0095c0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 -- 172.21.15.89:0/3980708043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4284102790 msgr2=0x7f4284198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4284102790 0x7f4284198930 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f427800c970 tx=0x7f427800ce30 comp rx=0 tx=0).stop 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 -- 172.21.15.89:0/3980708043 shutdown_connections 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f426c0a3240 0x7f426c0a5700 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4284106f10 0x7f428419ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4284102790 0x7f4284198930 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 --2- 172.21.15.89:0/3980708043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4284101de0 0x7f42841983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 -- 172.21.15.89:0/3980708043 >> 172.21.15.89:0/3980708043 conn(0x7f42840fd8c0 msgr2=0x7f42840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:06.804 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 -- 172.21.15.89:0/3980708043 shutdown_connections 2024-09-15T23:40:06.805 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:06.796+0000 7f428a56a640 1 -- 172.21.15.89:0/3980708043 wait complete. 2024-09-15T23:40:07.529 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:40:07.776 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:08.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.263+0000 7f8c0b848640 1 -- 172.21.15.89:0/3959381721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c04100920 msgr2=0x7f8c04100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:08.296 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.263+0000 7f8c0b848640 1 --2- 172.21.15.89:0/3959381721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c04100920 0x7f8c04100d00 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009900 tx=0x7f8bf402a410 comp rx=0 tx=0).stop 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.263+0000 7f8c0b848640 1 -- 172.21.15.89:0/3959381721 shutdown_connections 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.263+0000 7f8c0b848640 1 --2- 172.21.15.89:0/3959381721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c0410e4d0 0x7f8c041108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.263+0000 7f8c0b848640 1 --2- 172.21.15.89:0/3959381721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c041012d0 0x7f8c0410df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.263+0000 7f8c0b848640 1 --2- 172.21.15.89:0/3959381721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c04100920 0x7f8c04100d00 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.263+0000 7f8c0b848640 1 -- 172.21.15.89:0/3959381721 >> 172.21.15.89:0/3959381721 conn(0x7f8c040fc790 msgr2=0x7f8c040febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.263+0000 7f8c0b848640 1 -- 172.21.15.89:0/3959381721 shutdown_connections 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.263+0000 7f8c0b848640 1 -- 172.21.15.89:0/3959381721 wait complete. 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.263+0000 7f8c0b848640 1 Processor -- start 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c0b848640 1 -- start start 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c0b848640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c04100920 0x7f8c0419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.297 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c0b848640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c041012d0 0x7f8c0419ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c0b848640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c0410e4d0 0x7f8c0419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c0b848640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c04114170 con 0x7f8c04100920 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c0b848640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c04113ff0 con 0x7f8c0410e4d0 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c0b848640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c041142f0 con 0x7f8c041012d0 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c095e4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c04100920 0x7f8c0419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c095e4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c04100920 0x7f8c0419a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34426/0 (socket says 172.21.15.89:34426) 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c095e4640 1 -- 172.21.15.89:0/3699071071 learned_addr learned my addr 172.21.15.89:0/3699071071 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c08de3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c041012d0 0x7f8c0419ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c09de5640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c0410e4d0 0x7f8c0419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c095e4640 1 -- 172.21.15.89:0/3699071071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c041012d0 msgr2=0x7f8c0419ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c095e4640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c041012d0 0x7f8c0419ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c095e4640 1 -- 172.21.15.89:0/3699071071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c0410e4d0 msgr2=0x7f8c0419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:08.298 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c095e4640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c0410e4d0 0x7f8c0419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c095e4640 1 -- 172.21.15.89:0/3699071071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0419f680 con 0x7f8c04100920 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c08de3640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c041012d0 0x7f8c0419ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.264+0000 7f8c095e4640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c04100920 0x7f8c0419a6d0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4006010 tx=0x7f8bf40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.265+0000 7f8bfa7fc640 1 -- 172.21.15.89:0/3699071071 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf403f030 con 0x7f8c04100920 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.265+0000 7f8bfa7fc640 1 -- 172.21.15.89:0/3699071071 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf4003ce0 con 0x7f8c04100920 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.265+0000 7f8bfa7fc640 1 -- 172.21.15.89:0/3699071071 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4035070 con 0x7f8c04100920 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.265+0000 7f8c0b848640 1 -- 172.21.15.89:0/3699071071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c0419f910 con 0x7f8c04100920 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.265+0000 7f8c0b848640 1 -- 172.21.15.89:0/3699071071 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c041a7250 con 0x7f8c04100920 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.267+0000 7f8c0dc21740 1 Processor -- start 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c0dc21740 1 -- start start 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603da5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603da5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.299 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 0x555603db2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555603d0b2f0 con 0x555603db0370 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555603cb2cf0 con 0x555603da6b40 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555603d072b0 con 0x555603da55d0 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c08de3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 0x555603db2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c08de3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 0x555603db2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34432/0 (socket says 172.21.15.89:34432) 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c08de3640 1 -- 172.21.15.89:0/4167073664 learned_addr learned my addr 172.21.15.89:0/4167073664 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c09de5640 1 --2- 172.21.15.89:0/4167073664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603da5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8bfa7fc640 1 -- 172.21.15.89:0/3699071071 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f8bf4042030 con 0x7f8c04100920 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c09de5640 1 -- 172.21.15.89:0/4167073664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 msgr2=0x555603da5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c09de5640 1 --2- 172.21.15.89:0/4167073664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603da5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c09de5640 1 -- 172.21.15.89:0/4167073664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 msgr2=0x555603db2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:08.300 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c09de5640 1 --2- 172.21.15.89:0/4167073664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 0x555603db2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c09de5640 1 -- 172.21.15.89:0/4167073664 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555603c973a0 con 0x555603da6b40 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.268+0000 7f8c08de3640 1 --2- 172.21.15.89:0/4167073664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 0x555603db2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.269+0000 7f8c09de5640 1 --2- 172.21.15.89:0/4167073664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603da5090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc005fd0 tx=0x7f8bfc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa58347d805f37b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.269+0000 7f8bef7fe640 1 -- 172.21.15.89:0/4167073664 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc004d20 con 0x555603da6b40 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.269+0000 7f8bef7fe640 1 -- 172.21.15.89:0/4167073664 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bfc004050 con 0x555603da6b40 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.269+0000 7f8bef7fe640 1 -- 172.21.15.89:0/4167073664 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc005280 con 0x555603da6b40 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.269+0000 7f8bfa7fc640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf00a32c0 0x7f8bf00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.269+0000 7f8c095e4640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf00a32c0 0x7f8bf00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.269+0000 7f8bfa7fc640 1 -- 172.21.15.89:0/3699071071 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8bf40f0130 con 0x7f8c04100920 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 -- 172.21.15.89:0/4167073664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 msgr2=0x555603da5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c095e4640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf00a32c0 0x7f8bf00a5780 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8bf40b8ea0 tx=0x7f8bf40b19b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:08.301 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/4167073664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603da5090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc005fd0 tx=0x7f8bfc011810 comp rx=0 tx=0).stop 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 -- 172.21.15.89:0/4167073664 shutdown_connections 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/4167073664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 0x555603db2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/4167073664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603da5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/4167073664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603da5090 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 -- 172.21.15.89:0/4167073664 >> 172.21.15.89:0/4167073664 conn(0x555603d6e310 msgr2=0x555603da1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 -- 172.21.15.89:0/4167073664 shutdown_connections 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 -- 172.21.15.89:0/4167073664 wait complete. 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 -- start start 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 Processor -- start 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603dafd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dacfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603db0370 0x555603dad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555603db3070 con 0x555603da6b40 2024-09-15T23:40:08.302 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555603cc6060 con 0x555603db0370 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.270+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555603db31f0 con 0x555603da55d0 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c0dc21740 10 client.0 _ll_register_callbacks cb 0x555603a198c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c08de3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dacfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c08de3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dacfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34448/0 (socket says 172.21.15.89:34448) 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c08de3640 1 -- 172.21.15.89:0/151993123 learned_addr learned my addr 172.21.15.89:0/151993123 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c09de5640 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603dafd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c09de5640 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603db0370 0x555603dad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c08de3640 1 -- 172.21.15.89:0/151993123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 msgr2=0x555603dafd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c08de3640 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603dafd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c08de3640 1 -- 172.21.15.89:0/151993123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603db0370 msgr2=0x555603dad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c08de3640 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603db0370 0x555603dad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c08de3640 1 -- 172.21.15.89:0/151993123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555603a7c3c0 con 0x555603da6b40 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c08de3640 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dacfc0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f8c00005c70 tx=0x7f8c00011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350843270ec117f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:08.303 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8becff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8becff9640 1 -- 172.21.15.89:0/151993123 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0001f060 con 0x555603da6b40 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8becff9640 1 -- 172.21.15.89:0/151993123 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8c00010b70 con 0x555603da6b40 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.271+0000 7f8c0dc21740 1 -- 172.21.15.89:0/151993123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bfc004050 con 0x555603da6b40 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8becff9640 1 -- 172.21.15.89:0/151993123 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c00010db0 con 0x555603da6b40 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8c0dc21740 10 client.21090 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8c0dc21740 1 -- 172.21.15.89:0/151993123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555603dbeb20 con 0x555603da6b40 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8becff9640 1 -- 172.21.15.89:0/151993123 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bfc004050 con 0x555603da6b40 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8becff9640 1 -- 172.21.15.89:0/151993123 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8c0000f3c0 con 0x555603da6b40 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8c0dc21740 10 client.21090 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8c0dc21740 10 client.21090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8c0dc21740 20 client.21090 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555603dc0030 0x555603dc2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.304 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8c0dc21740 4 client.21090 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8c0dc21740 1 -- 172.21.15.89:0/151993123 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555603d18860 con 0x555603dc0030 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.272+0000 7f8c09de5640 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555603dc0030 0x555603dc2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.273+0000 7f8becff9640 10 client.21090 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.273+0000 7f8c09de5640 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555603dc0030 0x555603dc2450 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb21ae6fd4028984 server_cookie=3eacdda39363c0ff in_seq=0 out_seq=0 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.273+0000 7f8becff9640 1 -- 172.21.15.89:0/151993123 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555603d18860 con 0x555603dc0030 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.273+0000 7f8becff9640 10 client.21090 handle_command_reply: tid=0 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.273+0000 7f8c0dc21740 1 client.21090 shutdown 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.273+0000 7f8c0dc21740 1 -- 172.21.15.89:0/151993123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 msgr2=0x555603dacfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.273+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dacfc0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f8c00005c70 tx=0x7f8c00011c90 comp rx=0 tx=0).stop 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.273+0000 7f8c0dc21740 1 -- 172.21.15.89:0/151993123 shutdown_connections 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.273+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555603dc0030 0x555603dc2450 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.273+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603db0370 0x555603dad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.274+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dacfc0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.305 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.274+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/151993123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603dafd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.274+0000 7f8c0dc21740 1 -- 172.21.15.89:0/151993123 >> 172.21.15.89:0/151993123 conn(0x555603d6e310 msgr2=0x555603da7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.274+0000 7f8becff9640 0 client.21090 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.274+0000 7f8c0dc21740 1 -- 172.21.15.89:0/151993123 shutdown_connections 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.274+0000 7f8c0dc21740 1 -- 172.21.15.89:0/151993123 wait complete. 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.274+0000 7f8c0dc21740 20 client.21090 trim_cache size 0 max 16384 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.286+0000 7f8c0dc21740 1 Processor -- start 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.286+0000 7f8c0dc21740 1 -- start start 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da55d0 0x555603dbc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dabfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603db0370 0x555603dc7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c041b1030 con 0x555603da6b40 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c041b2b70 con 0x555603da55d0 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c041b1700 con 0x555603db0370 2024-09-15T23:40:08.306 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c08de3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dabfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c08de3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dabfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34454/0 (socket says 172.21.15.89:34454) 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c08de3640 1 -- 172.21.15.89:0/1928048852 learned_addr learned my addr 172.21.15.89:0/1928048852 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c09de5640 1 --2- 172.21.15.89:0/1928048852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603db0370 0x555603dc7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c09de5640 1 --2- 172.21.15.89:0/1928048852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da55d0 0x555603dbc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c08de3640 1 -- 172.21.15.89:0/1928048852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603db0370 msgr2=0x555603dc7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c08de3640 1 --2- 172.21.15.89:0/1928048852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603db0370 0x555603dc7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c08de3640 1 -- 172.21.15.89:0/1928048852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da55d0 msgr2=0x555603dbc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c08de3640 1 --2- 172.21.15.89:0/1928048852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da55d0 0x555603dbc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c08de3640 1 -- 172.21.15.89:0/1928048852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555603da8120 con 0x555603da6b40 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8c08de3640 1 --2- 172.21.15.89:0/1928048852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dabfa0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f8c00000c00 tx=0x7f8c0000ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6767e1741042c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8beeffd640 1 -- 172.21.15.89:0/1928048852 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0002f070 con 0x555603da6b40 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8beeffd640 1 -- 172.21.15.89:0/1928048852 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c00010b70 con 0x555603da6b40 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.287+0000 7f8beeffd640 1 -- 172.21.15.89:0/1928048852 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00004660 con 0x555603da6b40 2024-09-15T23:40:08.307 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.288+0000 7f8c0dc21740 1 -- 172.21.15.89:0/1928048852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 msgr2=0x555603dabfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.288+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/1928048852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dabfa0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f8c00000c00 tx=0x7f8c0000ffd0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.288+0000 7f8c0dc21740 1 -- 172.21.15.89:0/1928048852 shutdown_connections 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.288+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/1928048852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603db0370 0x555603dc7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.288+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/1928048852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603da6b40 0x555603dabfa0 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.288+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/1928048852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da55d0 0x555603dbc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.288+0000 7f8c0dc21740 1 -- 172.21.15.89:0/1928048852 >> 172.21.15.89:0/1928048852 conn(0x555603d6e310 msgr2=0x555603dab890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.288+0000 7f8c0dc21740 1 -- 172.21.15.89:0/1928048852 shutdown_connections 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.288+0000 7f8c0dc21740 1 -- 172.21.15.89:0/1928048852 wait complete. 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c0dc21740 1 -- start start 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c0dc21740 1 Processor -- start 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603daa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603daa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.308 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c0dc21740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 0x555603daac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f8c041b24a0 con 0x555603db0370 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c041b1dd0 con 0x555603da6b40 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c0dc21740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8c041af010 con 0x555603da55d0 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c0dc21740 10 client.0 _ll_register_callbacks cb 0x7f8c0000bbb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c08de3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603daa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c09de5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603daa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c08de3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603daa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:57242/0 (socket says 172.21.15.89:57242) 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c08de3640 1 -- 172.21.15.89:0/3102575776 learned_addr learned my addr 172.21.15.89:0/3102575776 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.289+0000 7f8c09de5640 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 0x555603daac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8c08de3640 1 -- 172.21.15.89:0/3102575776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 msgr2=0x555603daa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8c08de3640 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603daa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8c08de3640 1 -- 172.21.15.89:0/3102575776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 msgr2=0x555603daac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:08.309 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8c08de3640 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 0x555603daac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8c08de3640 1 -- 172.21.15.89:0/3102575776 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555603e29ed0 con 0x555603da6b40 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8c09de5640 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603daa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8c08de3640 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603daa710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000f9d0 tx=0x7f8c0000f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba6220db9730dff8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8becff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8becff9640 1 -- 172.21.15.89:0/3102575776 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0002f040 con 0x555603da6b40 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8c0dc21740 1 -- 172.21.15.89:0/3102575776 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c00010b70 con 0x555603da6b40 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8becff9640 1 -- 172.21.15.89:0/3102575776 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f8c00005070 con 0x555603da6b40 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8becff9640 1 -- 172.21.15.89:0/3102575776 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c0001f030 con 0x555603da6b40 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8c0dc21740 10 client.28631 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.290+0000 7f8c0dc21740 1 -- 172.21.15.89:0/3102575776 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555603e2b1b0 con 0x555603da6b40 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8becff9640 1 -- 172.21.15.89:0/3102575776 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c00010b70 con 0x555603da6b40 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8becff9640 1 -- 172.21.15.89:0/3102575776 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8c00004a60 con 0x555603da6b40 2024-09-15T23:40:08.310 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8c0dc21740 10 client.28631 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8c0dc21740 10 client.28631 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8c0dc21740 20 client.28631 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555603e2b8c0 0x555603e2dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8c0dc21740 4 client.28631 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8c0dc21740 1 -- 172.21.15.89:0/3102575776 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555603d12d70 con 0x555603e2b8c0 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8c09de5640 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555603e2b8c0 0x555603e2dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8becff9640 10 client.28631 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.291+0000 7f8c09de5640 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555603e2b8c0 0x555603e2dce0 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8236f632c3e15335 server_cookie=5c152e0174adc4e in_seq=0 out_seq=0 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8becff9640 1 -- 172.21.15.89:0/3102575776 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555603d12d70 con 0x555603e2b8c0 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8becff9640 10 client.28631 handle_command_reply: tid=0 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 client.28631 shutdown 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 -- 172.21.15.89:0/3102575776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 msgr2=0x555603daa710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603daa710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000f9d0 tx=0x7f8c0000f3c0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.311 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 -- 172.21.15.89:0/3102575776 shutdown_connections 2024-09-15T23:40:08.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555603e2b8c0 0x555603e2dce0 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555603db0370 0x555603daac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8becff9640 0 client.28631 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:08.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555603da6b40 0x555603daa710 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 --2- 172.21.15.89:0/3102575776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555603da55d0 0x555603daa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 -- 172.21.15.89:0/3102575776 >> 172.21.15.89:0/3102575776 conn(0x555603d6e310 msgr2=0x555603dae820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:08.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 -- 172.21.15.89:0/3102575776 shutdown_connections 2024-09-15T23:40:08.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 1 -- 172.21.15.89:0/3102575776 wait complete. 2024-09-15T23:40:08.312 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.292+0000 7f8c0dc21740 20 client.28631 trim_cache size 0 max 16384 2024-09-15T23:40:08.312 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:40:08.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 -- 172.21.15.89:0/3699071071 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf00a32c0 msgr2=0x7f8bf00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:08.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf00a32c0 0x7f8bf00a5780 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8bf40b8ea0 tx=0x7f8bf40b19b0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 -- 172.21.15.89:0/3699071071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c04100920 msgr2=0x7f8c0419a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:08.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c04100920 0x7f8c0419a6d0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4006010 tx=0x7f8bf40042f0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 -- 172.21.15.89:0/3699071071 shutdown_connections 2024-09-15T23:40:08.313 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f8bf00a32c0 0x7f8bf00a5780 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c0410e4d0 0x7f8c0419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8c041012d0 0x7f8c0419ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 --2- 172.21.15.89:0/3699071071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8c04100920 0x7f8c0419a6d0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:08.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 -- 172.21.15.89:0/3699071071 >> 172.21.15.89:0/3699071071 conn(0x7f8c040fc790 msgr2=0x7f8c040feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:08.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 -- 172.21.15.89:0/3699071071 shutdown_connections 2024-09-15T23:40:08.314 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:08.293+0000 7f8c0b848640 1 -- 172.21.15.89:0/3699071071 wait complete. 2024-09-15T23:40:08.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:08 smithi089 ceph-mon[30933]: pgmap v833: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 KiB/s wr, 16 op/s 2024-09-15T23:40:08.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:40:08.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:08 smithi120 ceph-mon[35250]: pgmap v833: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 KiB/s wr, 16 op/s 2024-09-15T23:40:08.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:40:08.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:08 smithi174 ceph-mon[35586]: pgmap v833: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.6 KiB/s wr, 16 op/s 2024-09-15T23:40:08.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:40:09.470 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:40:09.715 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:40:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:40:09.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:40:09.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:40:09.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:40:09.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:40:09.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:40:09.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:40:09.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:40:10.084 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:40:10.084 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:40:10.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.223+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3583726449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a90101420 msgr2=0x7f2a9010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.223+0000 7f2a95e7c640 1 --2- 172.21.15.89:0/3583726449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a90101420 0x7f2a9010d950 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009cd0 tx=0x7f2a8802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.223+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3583726449 shutdown_connections 2024-09-15T23:40:10.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.223+0000 7f2a95e7c640 1 --2- 172.21.15.89:0/3583726449 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a90101420 0x7f2a9010d950 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.223+0000 7f2a95e7c640 1 --2- 172.21.15.89:0/3583726449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a90100a80 0x7f2a90100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.223+0000 7f2a95e7c640 1 --2- 172.21.15.89:0/3583726449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a90106a80 0x7f2a90106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.253 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.223+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3583726449 >> 172.21.15.89:0/3583726449 conn(0x7f2a900fc7a0 msgr2=0x7f2a900febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.223+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3583726449 shutdown_connections 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.223+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3583726449 wait complete. 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a95e7c640 1 Processor -- start 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a95e7c640 1 -- start start 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a95e7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a90100a80 0x7f2a90198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a95e7c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a90101420 0x7f2a90198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a95e7c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a90106a80 0x7f2a9019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a95e7c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2a90111fa0 con 0x7f2a90101420 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a95e7c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2a90111e20 con 0x7f2a90100a80 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a95e7c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a90112120 con 0x7f2a90106a80 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a8effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a90101420 0x7f2a90198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a8effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a90101420 0x7f2a90198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34480/0 (socket says 172.21.15.89:34480) 2024-09-15T23:40:10.254 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.224+0000 7f2a8effd640 1 -- 172.21.15.89:0/3181381713 learned_addr learned my addr 172.21.15.89:0/3181381713 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8ffff640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a90106a80 0x7f2a9019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8f7fe640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a90100a80 0x7f2a90198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8effd640 1 -- 172.21.15.89:0/3181381713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a90106a80 msgr2=0x7f2a9019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8effd640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a90106a80 0x7f2a9019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8effd640 1 -- 172.21.15.89:0/3181381713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a90100a80 msgr2=0x7f2a90198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8effd640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a90100a80 0x7f2a90198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8effd640 1 -- 172.21.15.89:0/3181381713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9019d3b0 con 0x7f2a90101420 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8effd640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a90101420 0x7f2a90198940 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400d770 tx=0x7f2a8400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8cff9640 1 -- 172.21.15.89:0/3181381713 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a84017070 con 0x7f2a90101420 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3181381713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9019d6a0 con 0x7f2a90101420 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8cff9640 1 -- 172.21.15.89:0/3181381713 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a84005500 con 0x7f2a90101420 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a8cff9640 1 -- 172.21.15.89:0/3181381713 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a84004e00 con 0x7f2a90101420 2024-09-15T23:40:10.255 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.225+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3181381713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a901a4f80 con 0x7f2a90101420 2024-09-15T23:40:10.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.226+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3181381713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a90100ee0 con 0x7f2a90101420 2024-09-15T23:40:10.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.227+0000 7f2a8cff9640 1 -- 172.21.15.89:0/3181381713 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2a84004fa0 con 0x7f2a90101420 2024-09-15T23:40:10.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.228+0000 7f2a8cff9640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a780a3240 0x7f2a780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.228+0000 7f2a8f7fe640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a780a3240 0x7f2a780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.229+0000 7f2a8cff9640 1 -- 172.21.15.89:0/3181381713 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2a840cf730 con 0x7f2a90101420 2024-09-15T23:40:10.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.229+0000 7f2a8f7fe640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a780a3240 0x7f2a780a5700 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2a800058c0 tx=0x7f2a8000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:10.256 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.232+0000 7f2a8cff9640 1 -- 172.21.15.89:0/3181381713 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2a8409b9b0 con 0x7f2a90101420 2024-09-15T23:40:10.343 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:10.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.461+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3181381713 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a901a5150 con 0x7f2a90101420 2024-09-15T23:40:10.462 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.462+0000 7f2a8cff9640 1 -- 172.21.15.89:0/3181381713 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f2a840afe20 con 0x7f2a90101420 2024-09-15T23:40:10.463 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:40:10.463 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:40:10.463 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:40:10.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3181381713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a780a3240 msgr2=0x7f2a780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a780a3240 0x7f2a780a5700 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2a800058c0 tx=0x7f2a8000a2d0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3181381713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a90101420 msgr2=0x7f2a90198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a90101420 0x7f2a90198940 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400d770 tx=0x7f2a8400dc30 comp rx=0 tx=0).stop 2024-09-15T23:40:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3181381713 shutdown_connections 2024-09-15T23:40:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2a780a3240 0x7f2a780a5700 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a90106a80 0x7f2a9019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2a90101420 0x7f2a90198940 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 --2- 172.21.15.89:0/3181381713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a90100a80 0x7f2a90198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3181381713 >> 172.21.15.89:0/3181381713 conn(0x7f2a900fc7a0 msgr2=0x7f2a900fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3181381713 shutdown_connections 2024-09-15T23:40:10.466 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.464+0000 7f2a95e7c640 1 -- 172.21.15.89:0/3181381713 wait complete. 2024-09-15T23:40:10.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.925+0000 7fdc5047e640 1 -- 172.21.15.89:0/1626873093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc48106a80 msgr2=0x7fdc48106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.925+0000 7fdc5047e640 1 --2- 172.21.15.89:0/1626873093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc48106a80 0x7fdc48106e60 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009930 tx=0x7fdc3c02a420 comp rx=0 tx=0).stop 2024-09-15T23:40:10.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.925+0000 7fdc5047e640 1 -- 172.21.15.89:0/1626873093 shutdown_connections 2024-09-15T23:40:10.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.925+0000 7fdc5047e640 1 --2- 172.21.15.89:0/1626873093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc48101420 0x7fdc4810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.925+0000 7fdc5047e640 1 --2- 172.21.15.89:0/1626873093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48100a80 0x7fdc48100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.925+0000 7fdc5047e640 1 --2- 172.21.15.89:0/1626873093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc48106a80 0x7fdc48106e60 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.925+0000 7fdc5047e640 1 -- 172.21.15.89:0/1626873093 >> 172.21.15.89:0/1626873093 conn(0x7fdc480fc7a0 msgr2=0x7fdc480febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.925+0000 7fdc5047e640 1 -- 172.21.15.89:0/1626873093 shutdown_connections 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.925+0000 7fdc5047e640 1 -- 172.21.15.89:0/1626873093 wait complete. 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc5047e640 1 Processor -- start 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc5047e640 1 -- start start 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc5047e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48100a80 0x7fdc480fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc5047e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc48101420 0x7fdc480fd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc5047e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc48106a80 0x7fdc480fd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc5047e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc48110ea0 con 0x7fdc48106a80 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc5047e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc48110d20 con 0x7fdc48100a80 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc5047e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdc48111020 con 0x7fdc48101420 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4ea1b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc48106a80 0x7fdc480fd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4ea1b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc48106a80 0x7fdc480fd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34504/0 (socket says 172.21.15.89:34504) 2024-09-15T23:40:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4ea1b640 1 -- 172.21.15.89:0/1779552366 learned_addr learned my addr 172.21.15.89:0/1779552366 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4e21a640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48100a80 0x7fdc480fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4da19640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc48101420 0x7fdc480fd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4ea1b640 1 -- 172.21.15.89:0/1779552366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc48101420 msgr2=0x7fdc480fd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4ea1b640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc48101420 0x7fdc480fd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4ea1b640 1 -- 172.21.15.89:0/1779552366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48100a80 msgr2=0x7fdc480fcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4ea1b640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48100a80 0x7fdc480fcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4ea1b640 1 -- 172.21.15.89:0/1779552366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc481a68d0 con 0x7fdc48106a80 2024-09-15T23:40:10.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.926+0000 7fdc4ea1b640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc48106a80 0x7fdc480fd810 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000b970 tx=0x7fdc4000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.927+0000 7fdc3b7fe640 1 -- 172.21.15.89:0/1779552366 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc40019070 con 0x7fdc48106a80 2024-09-15T23:40:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.927+0000 7fdc3b7fe640 1 -- 172.21.15.89:0/1779552366 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc40014440 con 0x7fdc48106a80 2024-09-15T23:40:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.927+0000 7fdc5047e640 1 -- 172.21.15.89:0/1779552366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc481a6bc0 con 0x7fdc48106a80 2024-09-15T23:40:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.927+0000 7fdc3b7fe640 1 -- 172.21.15.89:0/1779552366 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc400075d0 con 0x7fdc48106a80 2024-09-15T23:40:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.927+0000 7fdc5047e640 1 -- 172.21.15.89:0/1779552366 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc481a7100 con 0x7fdc48106a80 2024-09-15T23:40:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.929+0000 7fdc3b7fe640 1 -- 172.21.15.89:0/1779552366 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fdc4001c070 con 0x7fdc48106a80 2024-09-15T23:40:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.929+0000 7fdc52857740 1 Processor -- start 2024-09-15T23:40:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.929+0000 7fdc52857740 1 -- start start 2024-09-15T23:40:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.929+0000 7fdc52857740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc703f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.929+0000 7fdc52857740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 0x5631cc70e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc52857740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc70ed40 0x5631cc711130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc52857740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5631cc669280 con 0x5631cc703b60 2024-09-15T23:40:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc52857740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5631cc610b10 con 0x5631cc704480 2024-09-15T23:40:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc52857740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5631cc665240 con 0x5631cc70ed40 2024-09-15T23:40:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4da19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc703f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4da19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc703f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34512/0 (socket says 172.21.15.89:34512) 2024-09-15T23:40:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4da19640 1 -- 172.21.15.89:0/1986277423 learned_addr learned my addr 172.21.15.89:0/1986277423 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4e21a640 1 --2- 172.21.15.89:0/1986277423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 0x5631cc70e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc3b7fe640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc340a3240 0x7fdc340a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4da19640 1 --2- 172.21.15.89:0/1986277423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc70ed40 0x5631cc711130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4da19640 1 -- 172.21.15.89:0/1986277423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc70ed40 msgr2=0x5631cc711130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4da19640 1 --2- 172.21.15.89:0/1986277423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc70ed40 0x5631cc711130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4da19640 1 -- 172.21.15.89:0/1986277423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 msgr2=0x5631cc70e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4da19640 1 --2- 172.21.15.89:0/1986277423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 0x5631cc70e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4da19640 1 -- 172.21.15.89:0/1986277423 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631cc61d1f0 con 0x5631cc703b60 2024-09-15T23:40:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4ea1b640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc340a3240 0x7fdc340a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc3b7fe640 1 -- 172.21.15.89:0/1779552366 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdc400cf240 con 0x7fdc48106a80 2024-09-15T23:40:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.930+0000 7fdc4da19640 1 --2- 172.21.15.89:0/1986277423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc703f40 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7fdc44013560 tx=0x7fdc4400a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e26b594a61d33a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:10.947 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc38ff9640 1 -- 172.21.15.89:0/1986277423 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc44019030 con 0x5631cc703b60 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc38ff9640 1 -- 172.21.15.89:0/1986277423 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc440110d0 con 0x5631cc703b60 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc38ff9640 1 -- 172.21.15.89:0/1986277423 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4400f8e0 con 0x5631cc703b60 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc52857740 1 -- 172.21.15.89:0/1986277423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 msgr2=0x5631cc703f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc52857740 1 --2- 172.21.15.89:0/1986277423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc703f40 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7fdc44013560 tx=0x7fdc4400a800 comp rx=0 tx=0).stop 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc4ea1b640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc340a3240 0x7fdc340a5700 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fdc40097980 tx=0x7fdc40065f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc52857740 1 -- 172.21.15.89:0/1986277423 shutdown_connections 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc52857740 1 --2- 172.21.15.89:0/1986277423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc70ed40 0x5631cc711130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc52857740 1 --2- 172.21.15.89:0/1986277423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 0x5631cc70e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc52857740 1 --2- 172.21.15.89:0/1986277423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc703f40 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc52857740 1 -- 172.21.15.89:0/1986277423 >> 172.21.15.89:0/1986277423 conn(0x5631cc7000f0 msgr2=0x5631cc7004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc52857740 1 -- 172.21.15.89:0/1986277423 shutdown_connections 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.931+0000 7fdc52857740 1 -- 172.21.15.89:0/1986277423 wait complete. 2024-09-15T23:40:10.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc52857740 1 -- start start 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc52857740 1 Processor -- start 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc52857740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc703b60 0x5631cc70bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc52857740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 0x5631cc70c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc52857740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc70ed40 0x5631cc70c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc52857740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5631cc711910 con 0x5631cc70ed40 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc52857740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5631cc623f10 con 0x5631cc704480 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc52857740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5631cc711a90 con 0x5631cc703b60 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc52857740 10 client.0 _ll_register_callbacks cb 0x5631cc5eca40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc4da19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc70ed40 0x5631cc70c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc4da19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc70ed40 0x5631cc70c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34516/0 (socket says 172.21.15.89:34516) 2024-09-15T23:40:10.949 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc4da19640 1 -- 172.21.15.89:0/4000712128 learned_addr learned my addr 172.21.15.89:0/4000712128 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc4e21a640 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 0x5631cc70c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.932+0000 7fdc4da19640 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc703b60 0x5631cc70bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc4e21a640 1 -- 172.21.15.89:0/4000712128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc703b60 msgr2=0x5631cc70bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc4e21a640 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc703b60 0x5631cc70bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc4e21a640 1 -- 172.21.15.89:0/4000712128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc70ed40 msgr2=0x5631cc70c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc4e21a640 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc70ed40 0x5631cc70c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc4e21a640 1 -- 172.21.15.89:0/4000712128 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631cc3da6a0 con 0x5631cc704480 2024-09-15T23:40:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc4da19640 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc70ed40 0x5631cc70c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:40:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc4e21a640 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 0x5631cc70c100 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c005940 tx=0x7fdc3c031a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=926f9e037221d943 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-09-15T23:40:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc31ffb640 1 -- 172.21.15.89:0/4000712128 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c035040 con 0x5631cc704480 2024-09-15T23:40:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc31ffb640 1 -- 172.21.15.89:0/4000712128 <== mon.2 v2:172.21.15.120:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdc3c004380 con 0x5631cc704480 2024-09-15T23:40:10.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc52857740 1 -- 172.21.15.89:0/4000712128 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc440110d0 con 0x5631cc704480 2024-09-15T23:40:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc31ffb640 1 -- 172.21.15.89:0/4000712128 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdc3c0035f0 con 0x5631cc704480 2024-09-15T23:40:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc52857740 10 client.28643 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc52857740 1 -- 172.21.15.89:0/4000712128 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5631cc71d4f0 con 0x5631cc704480 2024-09-15T23:40:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.933+0000 7fdc31ffb640 1 -- 172.21.15.89:0/4000712128 <== mon.2 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc440110d0 con 0x5631cc704480 2024-09-15T23:40:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.934+0000 7fdc31ffb640 1 -- 172.21.15.89:0/4000712128 <== mon.2 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fdc3c03c850 con 0x5631cc704480 2024-09-15T23:40:10.952 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.934+0000 7fdc52857740 10 client.28643 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.934+0000 7fdc52857740 10 client.28643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.934+0000 7fdc52857740 20 client.28643 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.934+0000 7fdc52857740 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5631cc71ea50 0x5631cc720e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.934+0000 7fdc52857740 4 client.28643 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.934+0000 7fdc52857740 1 -- 172.21.15.89:0/4000712128 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631cc6767f0 con 0x5631cc71ea50 2024-09-15T23:40:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.934+0000 7fdc4da19640 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5631cc71ea50 0x5631cc720e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.953 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.934+0000 7fdc4da19640 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5631cc71ea50 0x5631cc720e40 crc :-1 s=READY pgs=422 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf05bd1d477227df server_cookie=3762d69d96655e23 in_seq=0 out_seq=0 2024-09-15T23:40:10.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.934+0000 7fdc31ffb640 10 client.28643 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:10.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc31ffb640 1 -- 172.21.15.89:0/4000712128 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5631cc6767f0 con 0x5631cc71ea50 2024-09-15T23:40:10.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc31ffb640 10 client.28643 handle_command_reply: tid=0 2024-09-15T23:40:10.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 client.28643 shutdown 2024-09-15T23:40:10.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 -- 172.21.15.89:0/4000712128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 msgr2=0x5631cc70c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.954 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 0x5631cc70c100 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c005940 tx=0x7fdc3c031a40 comp rx=0 tx=0).stop 2024-09-15T23:40:10.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 -- 172.21.15.89:0/4000712128 shutdown_connections 2024-09-15T23:40:10.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5631cc71ea50 0x5631cc720e40 crc :-1 s=READY pgs=422 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc70ed40 0x5631cc70c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc31ffb640 0 client.28643 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:10.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc704480 0x5631cc70c100 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.955 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 --2- 172.21.15.89:0/4000712128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc703b60 0x5631cc70bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 -- 172.21.15.89:0/4000712128 >> 172.21.15.89:0/4000712128 conn(0x5631cc7000f0 msgr2=0x5631cc705e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:10.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 -- 172.21.15.89:0/4000712128 shutdown_connections 2024-09-15T23:40:10.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 1 -- 172.21.15.89:0/4000712128 wait complete. 2024-09-15T23:40:10.956 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.935+0000 7fdc52857740 20 client.28643 trim_cache size 0 max 16384 2024-09-15T23:40:10.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.952+0000 7fdc52857740 1 Processor -- start 2024-09-15T23:40:10.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc52857740 1 -- start start 2024-09-15T23:40:10.966 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc52857740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc70dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc52857740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc704480 0x5631cc7065c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc52857740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 0x5631cc70c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc52857740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481b1050 con 0x5631cc703b60 2024-09-15T23:40:10.967 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc52857740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481b2b90 con 0x5631cc70ed40 2024-09-15T23:40:10.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc52857740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481b1720 con 0x5631cc704480 2024-09-15T23:40:10.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4da19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc70dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.968 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4da19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc70dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34526/0 (socket says 172.21.15.89:34526) 2024-09-15T23:40:10.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4da19640 1 -- 172.21.15.89:0/2929452020 learned_addr learned my addr 172.21.15.89:0/2929452020 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:10.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4e21a640 1 --2- 172.21.15.89:0/2929452020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc704480 0x5631cc7065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4da19640 1 --2- 172.21.15.89:0/2929452020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 0x5631cc70c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4da19640 1 -- 172.21.15.89:0/2929452020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc704480 msgr2=0x5631cc7065c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:10.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4da19640 1 --2- 172.21.15.89:0/2929452020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc704480 0x5631cc7065c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.969 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4da19640 1 -- 172.21.15.89:0/2929452020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 msgr2=0x5631cc70c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:10.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4da19640 1 --2- 172.21.15.89:0/2929452020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 0x5631cc70c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4da19640 1 -- 172.21.15.89:0/2929452020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631cc6fb830 con 0x5631cc703b60 2024-09-15T23:40:10.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.953+0000 7fdc4da19640 1 --2- 172.21.15.89:0/2929452020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc70dde0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7fdc44005f10 tx=0x7fdc4400af50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae7b71c59b06af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:10.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc33fff640 1 -- 172.21.15.89:0/2929452020 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc44019040 con 0x5631cc703b60 2024-09-15T23:40:10.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc33fff640 1 -- 172.21.15.89:0/2929452020 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc4400a630 con 0x5631cc703b60 2024-09-15T23:40:10.970 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc33fff640 1 -- 172.21.15.89:0/2929452020 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc440048c0 con 0x5631cc703b60 2024-09-15T23:40:10.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc52857740 1 -- 172.21.15.89:0/2929452020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 msgr2=0x5631cc70dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc52857740 1 --2- 172.21.15.89:0/2929452020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc70dde0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7fdc44005f10 tx=0x7fdc4400af50 comp rx=0 tx=0).stop 2024-09-15T23:40:10.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc52857740 1 -- 172.21.15.89:0/2929452020 shutdown_connections 2024-09-15T23:40:10.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc52857740 1 --2- 172.21.15.89:0/2929452020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 0x5631cc70c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc52857740 1 --2- 172.21.15.89:0/2929452020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc704480 0x5631cc7065c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.971 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc52857740 1 --2- 172.21.15.89:0/2929452020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc703b60 0x5631cc70dde0 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc52857740 1 -- 172.21.15.89:0/2929452020 >> 172.21.15.89:0/2929452020 conn(0x5631cc7000f0 msgr2=0x5631cc70ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:10.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc52857740 1 -- 172.21.15.89:0/2929452020 shutdown_connections 2024-09-15T23:40:10.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.954+0000 7fdc52857740 1 -- 172.21.15.89:0/2929452020 wait complete. 2024-09-15T23:40:10.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc52857740 1 -- start start 2024-09-15T23:40:10.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc52857740 1 Processor -- start 2024-09-15T23:40:10.972 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc52857740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc703b60 0x5631cc7172e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc52857740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc704480 0x5631cc708770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc52857740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 0x5631cc708cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.973 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc52857740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481b24c0 con 0x5631cc704480 2024-09-15T23:40:10.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc52857740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481b1df0 con 0x5631cc70ed40 2024-09-15T23:40:10.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc52857740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481af030 con 0x5631cc703b60 2024-09-15T23:40:10.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc52857740 10 client.0 _ll_register_callbacks cb 0x7fdc3c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:10.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4e21a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc704480 0x5631cc708770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.974 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4e21a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc704480 0x5631cc708770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34540/0 (socket says 172.21.15.89:34540) 2024-09-15T23:40:10.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4e21a640 1 -- 172.21.15.89:0/2356923507 learned_addr learned my addr 172.21.15.89:0/2356923507 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:10.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4da19640 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 0x5631cc708cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4da19640 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc703b60 0x5631cc7172e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4e21a640 1 -- 172.21.15.89:0/2356923507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc703b60 msgr2=0x5631cc7172e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:10.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4e21a640 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc703b60 0x5631cc7172e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4e21a640 1 -- 172.21.15.89:0/2356923507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 msgr2=0x5631cc708cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:10.975 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4e21a640 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 0x5631cc708cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4e21a640 1 -- 172.21.15.89:0/2356923507 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631cc7093a0 con 0x5631cc704480 2024-09-15T23:40:10.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4da19640 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 0x5631cc708cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:10.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc4e21a640 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc704480 0x5631cc708770 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c001eb0 tx=0x7fdc3c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bffad8dcdb8e7f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:10.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.955+0000 7fdc31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:10.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc31ffb640 1 -- 172.21.15.89:0/2356923507 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c000e60 con 0x5631cc704480 2024-09-15T23:40:10.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc31ffb640 1 -- 172.21.15.89:0/2356923507 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fdc3c03c440 con 0x5631cc704480 2024-09-15T23:40:10.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc52857740 1 -- 172.21.15.89:0/2356923507 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc4400a630 con 0x5631cc704480 2024-09-15T23:40:10.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc31ffb640 1 -- 172.21.15.89:0/2356923507 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdc3c057070 con 0x5631cc704480 2024-09-15T23:40:10.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc52857740 10 client.21120 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:10.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc52857740 1 -- 172.21.15.89:0/2356923507 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5631cc789620 con 0x5631cc704480 2024-09-15T23:40:10.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc31ffb640 1 -- 172.21.15.89:0/2356923507 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc4400a630 con 0x5631cc704480 2024-09-15T23:40:10.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc31ffb640 1 -- 172.21.15.89:0/2356923507 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fdc3c045640 con 0x5631cc704480 2024-09-15T23:40:10.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc52857740 10 client.21120 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc52857740 10 client.21120 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc52857740 20 client.21120 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc52857740 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5631cc789bd0 0x5631cc78bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc52857740 4 client.21120 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc52857740 1 -- 172.21.15.89:0/2356923507 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5631cc670d00 con 0x5631cc789bd0 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.956+0000 7fdc4da19640 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5631cc789bd0 0x5631cc78bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.957+0000 7fdc4da19640 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5631cc789bd0 0x5631cc78bff0 crc :-1 s=READY pgs=423 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e79a81ea03280b58 server_cookie=a021856670f672ed in_seq=0 out_seq=0 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.957+0000 7fdc31ffb640 10 client.21120 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc31ffb640 1 -- 172.21.15.89:0/2356923507 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5631cc670d00 con 0x5631cc789bd0 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc31ffb640 10 client.21120 handle_command_reply: tid=0 2024-09-15T23:40:10.978 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 client.21120 shutdown 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 -- 172.21.15.89:0/2356923507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc704480 msgr2=0x5631cc708770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc704480 0x5631cc708770 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c001eb0 tx=0x7fdc3c02a420 comp rx=0 tx=0).stop 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 -- 172.21.15.89:0/2356923507 shutdown_connections 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5631cc789bd0 0x5631cc78bff0 crc :-1 s=READY pgs=423 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5631cc70ed40 0x5631cc708cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631cc704480 0x5631cc708770 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc31ffb640 0 client.21120 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 --2- 172.21.15.89:0/2356923507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5631cc703b60 0x5631cc7172e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 -- 172.21.15.89:0/2356923507 >> 172.21.15.89:0/2356923507 conn(0x5631cc7000f0 msgr2=0x5631cc6fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 -- 172.21.15.89:0/2356923507 shutdown_connections 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 1 -- 172.21.15.89:0/2356923507 wait complete. 2024-09-15T23:40:10.979 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.958+0000 7fdc52857740 20 client.21120 trim_cache size 0 max 16384 2024-09-15T23:40:10.980 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:40:10.980 INFO:teuthology.orchestra.run.smithi089.stdout: "return_code": 0, 2024-09-15T23:40:10.980 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_tag": "628be185-c365-4b3b-a1fd-eddc75aee342", 2024-09-15T23:40:10.980 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": "asynchronous" 2024-09-15T23:40:10.980 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:40:10.981 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:10 smithi120 ceph-mon[35250]: pgmap v834: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-09-15T23:40:10.981 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:10 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3181381713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:10.981 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 -- 172.21.15.89:0/1779552366 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc340a3240 msgr2=0x7fdc340a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc340a3240 0x7fdc340a5700 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fdc40097980 tx=0x7fdc40065f60 comp rx=0 tx=0).stop 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 -- 172.21.15.89:0/1779552366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc48106a80 msgr2=0x7fdc480fd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc48106a80 0x7fdc480fd810 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000b970 tx=0x7fdc4000bd70 comp rx=0 tx=0).stop 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 -- 172.21.15.89:0/1779552366 shutdown_connections 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fdc340a3240 0x7fdc340a5700 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc48106a80 0x7fdc480fd810 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc48101420 0x7fdc480fd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 --2- 172.21.15.89:0/1779552366 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48100a80 0x7fdc480fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 -- 172.21.15.89:0/1779552366 >> 172.21.15.89:0/1779552366 conn(0x7fdc480fc7a0 msgr2=0x7fdc4806aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 -- 172.21.15.89:0/1779552366 shutdown_connections 2024-09-15T23:40:10.982 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:10.959+0000 7fdc5047e640 1 -- 172.21.15.89:0/1779552366 wait complete. 2024-09-15T23:40:10.982 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:10 smithi089 ceph-mon[30933]: pgmap v834: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-09-15T23:40:10.983 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:10 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3181381713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:10.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:10 smithi174 ceph-mon[35586]: pgmap v834: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-09-15T23:40:10.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:10 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3181381713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:11.835 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:40:11.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:11 smithi120 ceph-mon[35250]: scrub queued for path: / 2024-09-15T23:40:11.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:11 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:40:11.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:11 smithi120 ceph-mon[35250]: scrub summary: active paths [/] 2024-09-15T23:40:11.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:11 smithi120 ceph-mon[35250]: scrub summary: idle+waiting paths [/] 2024-09-15T23:40:11.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:11 smithi174 ceph-mon[35586]: scrub queued for path: / 2024-09-15T23:40:11.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:11 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:40:11.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:11 smithi174 ceph-mon[35586]: scrub summary: active paths [/] 2024-09-15T23:40:11.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:11 smithi174 ceph-mon[35586]: scrub summary: idle+waiting paths [/] 2024-09-15T23:40:12.045 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:11 smithi089 ceph-mon[30933]: scrub queued for path: / 2024-09-15T23:40:12.045 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:11 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:40:12.046 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:11 smithi089 ceph-mon[30933]: scrub summary: active paths [/] 2024-09-15T23:40:12.046 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:11 smithi089 ceph-mon[30933]: scrub summary: idle+waiting paths [/] 2024-09-15T23:40:12.278 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --set-id 2590a49a --release --await' 2024-09-15T23:40:12.278 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --set-id 2590a49a --release --await 2024-09-15T23:40:12.321 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.864+0000 7fa0482c0640 1 -- 172.21.15.89:0/3837562068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa040100a80 msgr2=0x7fa040100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.864+0000 7fa0482c0640 1 --2- 172.21.15.89:0/3837562068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa040100a80 0x7fa040100ee0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fa030009930 tx=0x7fa03002a420 comp rx=0 tx=0).stop 2024-09-15T23:40:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.864+0000 7fa0482c0640 1 -- 172.21.15.89:0/3837562068 shutdown_connections 2024-09-15T23:40:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.864+0000 7fa0482c0640 1 --2- 172.21.15.89:0/3837562068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa040101420 0x7fa04010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.864+0000 7fa0482c0640 1 --2- 172.21.15.89:0/3837562068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa040100a80 0x7fa040100ee0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.864+0000 7fa0482c0640 1 --2- 172.21.15.89:0/3837562068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa040106a80 0x7fa040106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.867 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.864+0000 7fa0482c0640 1 -- 172.21.15.89:0/3837562068 >> 172.21.15.89:0/3837562068 conn(0x7fa0400fc7a0 msgr2=0x7fa0400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.864+0000 7fa0482c0640 1 -- 172.21.15.89:0/3837562068 shutdown_connections 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.864+0000 7fa0482c0640 1 -- 172.21.15.89:0/3837562068 wait complete. 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa0482c0640 1 Processor -- start 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa0482c0640 1 -- start start 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa0482c0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa040100a80 0x7fa040197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa0482c0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa040101420 0x7fa040197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa0482c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa040106a80 0x7fa04019bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa0482c0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa040110e90 con 0x7fa040106a80 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa0482c0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa040110d10 con 0x7fa040101420 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa0482c0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa040111010 con 0x7fa040100a80 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04685d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa040106a80 0x7fa04019bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04685d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa040106a80 0x7fa04019bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34550/0 (socket says 172.21.15.89:34550) 2024-09-15T23:40:12.868 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04685d640 1 -- 172.21.15.89:0/3270444662 learned_addr learned my addr 172.21.15.89:0/3270444662 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04605c640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa040100a80 0x7fa040197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04585b640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa040101420 0x7fa040197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04685d640 1 -- 172.21.15.89:0/3270444662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa040100a80 msgr2=0x7fa040197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04685d640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa040100a80 0x7fa040197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04685d640 1 -- 172.21.15.89:0/3270444662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa040101420 msgr2=0x7fa040197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04685d640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa040101420 0x7fa040197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04685d640 1 -- 172.21.15.89:0/3270444662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa04019c300 con 0x7fa040106a80 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.865+0000 7fa04605c640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa040100a80 0x7fa040197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.866+0000 7fa04685d640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa040106a80 0x7fa04019bc20 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fa038009fd0 tx=0x7fa03800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.866+0000 7fa0377fe640 1 -- 172.21.15.89:0/3270444662 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038017070 con 0x7fa040106a80 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.866+0000 7fa0377fe640 1 -- 172.21.15.89:0/3270444662 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa038003e50 con 0x7fa040106a80 2024-09-15T23:40:12.869 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.866+0000 7fa0482c0640 1 -- 172.21.15.89:0/3270444662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa04019c5f0 con 0x7fa040106a80 2024-09-15T23:40:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.867+0000 7fa0377fe640 1 -- 172.21.15.89:0/3270444662 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0380106c0 con 0x7fa040106a80 2024-09-15T23:40:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.867+0000 7fa0482c0640 1 -- 172.21.15.89:0/3270444662 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0401a3ed0 con 0x7fa040106a80 2024-09-15T23:40:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.869+0000 7fa0377fe640 1 -- 172.21.15.89:0/3270444662 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa038004000 con 0x7fa040106a80 2024-09-15T23:40:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.869+0000 7fa04a699740 1 Processor -- start 2024-09-15T23:40:12.870 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.869+0000 7fa04a699740 1 -- start start 2024-09-15T23:40:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04a699740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04a699740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04a699740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5c4740 0x559e4b5c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04a699740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559e4b51b2e0 con 0x559e4b5b8d00 2024-09-15T23:40:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04a699740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559e4b4c2ce0 con 0x559e4b5b7ad0 2024-09-15T23:40:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04a699740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559e4b5172a0 con 0x559e4b5c4740 2024-09-15T23:40:12.872 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04585b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04585b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5b7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34566/0 (socket says 172.21.15.89:34566) 2024-09-15T23:40:12.873 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04605c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04585b640 1 -- 172.21.15.89:0/568492999 learned_addr learned my addr 172.21.15.89:0/568492999 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04585b640 1 --2- 172.21.15.89:0/568492999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5c4740 0x559e4b5c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa0377fe640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa02c0a3110 0x7fa02c0a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04605c640 1 -- 172.21.15.89:0/568492999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5c4740 msgr2=0x559e4b5c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04605c640 1 --2- 172.21.15.89:0/568492999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5c4740 0x559e4b5c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04605c640 1 -- 172.21.15.89:0/568492999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 msgr2=0x559e4b5b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04605c640 1 --2- 172.21.15.89:0/568492999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5b7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.874 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04605c640 1 -- 172.21.15.89:0/568492999 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e4b4a7440 con 0x559e4b5b7ad0 2024-09-15T23:40:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.870+0000 7fa04685d640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa02c0a3110 0x7fa02c0a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.871+0000 7fa0377fe640 1 -- 172.21.15.89:0/3270444662 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa0380cf740 con 0x7fa040106a80 2024-09-15T23:40:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.871+0000 7fa04605c640 1 --2- 172.21.15.89:0/568492999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5b7f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa03c008b30 tx=0x7fa03c00e350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4472864e63d6095a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.871+0000 7fa034dea640 1 -- 172.21.15.89:0/568492999 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c014070 con 0x559e4b5b7ad0 2024-09-15T23:40:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.871+0000 7fa034dea640 1 -- 172.21.15.89:0/568492999 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa03c005500 con 0x559e4b5b7ad0 2024-09-15T23:40:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.871+0000 7fa04685d640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa02c0a3110 0x7fa02c0a55d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa038098340 tx=0x7fa038066380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:12.875 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.871+0000 7fa034dea640 1 -- 172.21.15.89:0/568492999 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c010630 con 0x559e4b5b7ad0 2024-09-15T23:40:12.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 -- 172.21.15.89:0/568492999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 msgr2=0x559e4b5b7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:12.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 --2- 172.21.15.89:0/568492999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5b7f30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa03c008b30 tx=0x7fa03c00e350 comp rx=0 tx=0).stop 2024-09-15T23:40:12.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 -- 172.21.15.89:0/568492999 shutdown_connections 2024-09-15T23:40:12.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 --2- 172.21.15.89:0/568492999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5c4740 0x559e4b5c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.878 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 --2- 172.21.15.89:0/568492999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5b7f30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 --2- 172.21.15.89:0/568492999 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5b7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 -- 172.21.15.89:0/568492999 >> 172.21.15.89:0/568492999 conn(0x559e4b5b3c40 msgr2=0x559e4b5b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 -- 172.21.15.89:0/568492999 shutdown_connections 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 -- 172.21.15.89:0/568492999 wait complete. 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 -- start start 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.872+0000 7fa04a699740 1 Processor -- start 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04a699740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5c4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04a699740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b8d00 0x559e4b5c13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04a699740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04a699740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559e4b5c7650 con 0x559e4b5c4740 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04a699740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559e4b4d6050 con 0x559e4b5b7ad0 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04a699740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559e4b5c77d0 con 0x559e4b5b8d00 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04a699740 10 client.0 _ll_register_callbacks cb 0x559e4b516160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:12.879 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04585b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04585b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5c19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34572/0 (socket says 172.21.15.89:34572) 2024-09-15T23:40:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04585b640 1 -- 172.21.15.89:0/2402953634 learned_addr learned my addr 172.21.15.89:0/2402953634 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04585b640 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5c4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04605c640 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b8d00 0x559e4b5c13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04585b640 1 -- 172.21.15.89:0/2402953634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b8d00 msgr2=0x559e4b5c13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04585b640 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b8d00 0x559e4b5c13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.880 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04585b640 1 -- 172.21.15.89:0/2402953634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 msgr2=0x559e4b5c4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04585b640 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5c4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04585b640 1 -- 172.21.15.89:0/2402953634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e4b28c3c0 con 0x559e4b5c4740 2024-09-15T23:40:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa04585b640 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5c19e0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fa030006010 tx=0x7fa030031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84caa7ba166c45d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.873+0000 7fa029ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa029ffb640 1 -- 172.21.15.89:0/2402953634 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030035040 con 0x559e4b5c4740 2024-09-15T23:40:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa029ffb640 1 -- 172.21.15.89:0/2402953634 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa030003800 con 0x559e4b5c4740 2024-09-15T23:40:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa04a699740 1 -- 172.21.15.89:0/2402953634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa03c005500 con 0x559e4b5c4740 2024-09-15T23:40:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa029ffb640 1 -- 172.21.15.89:0/2402953634 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa030053050 con 0x559e4b5c4740 2024-09-15T23:40:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa04a699740 10 client.21132 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:12.881 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa04a699740 1 -- 172.21.15.89:0/2402953634 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559e4b5d3170 con 0x559e4b5c4740 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa029ffb640 1 -- 172.21.15.89:0/2402953634 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa03c005500 con 0x559e4b5c4740 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa029ffb640 1 -- 172.21.15.89:0/2402953634 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa03003da60 con 0x559e4b5c4740 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa04a699740 10 client.21132 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa04a699740 10 client.21132 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa04a699740 20 client.21132 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa04a699740 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4b5d4a00 0x559e4b5d6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa04a699740 4 client.21132 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.874+0000 7fa04a699740 1 -- 172.21.15.89:0/2402953634 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e4b528850 con 0x559e4b5d4a00 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.875+0000 7fa04605c640 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4b5d4a00 0x559e4b5d6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.875+0000 7fa029ffb640 10 client.21132 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.875+0000 7fa04605c640 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4b5d4a00 0x559e4b5d6e10 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24617fd9689ba857 server_cookie=59d23b110b0028a5 in_seq=0 out_seq=0 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa029ffb640 1 -- 172.21.15.89:0/2402953634 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559e4b528850 con 0x559e4b5d4a00 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa029ffb640 10 client.21132 handle_command_reply: tid=0 2024-09-15T23:40:12.882 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 client.21132 shutdown 2024-09-15T23:40:12.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 -- 172.21.15.89:0/2402953634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 msgr2=0x559e4b5c19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:12.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5c19e0 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fa030006010 tx=0x7fa030031400 comp rx=0 tx=0).stop 2024-09-15T23:40:12.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 -- 172.21.15.89:0/2402953634 shutdown_connections 2024-09-15T23:40:12.883 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4b5d4a00 0x559e4b5d6e10 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5c19e0 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b8d00 0x559e4b5c13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 --2- 172.21.15.89:0/2402953634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5c4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 -- 172.21.15.89:0/2402953634 >> 172.21.15.89:0/2402953634 conn(0x559e4b5b3c40 msgr2=0x559e4b5b99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:12.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa029ffb640 0 client.21132 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:12.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 -- 172.21.15.89:0/2402953634 shutdown_connections 2024-09-15T23:40:12.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 1 -- 172.21.15.89:0/2402953634 wait complete. 2024-09-15T23:40:12.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.876+0000 7fa04a699740 20 client.21132 trim_cache size 0 max 16384 2024-09-15T23:40:12.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04a699740 1 Processor -- start 2024-09-15T23:40:12.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04a699740 1 -- start start 2024-09-15T23:40:12.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04a699740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5d0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04a699740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b8d00 0x559e4b5d6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04a699740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5dc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.898 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04a699740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401ac900 con 0x559e4b5c4740 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04a699740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401ae440 con 0x559e4b5b7ad0 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04a699740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401acfd0 con 0x559e4b5b8d00 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04585b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5dc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04585b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5dc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34582/0 (socket says 172.21.15.89:34582) 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04585b640 1 -- 172.21.15.89:0/2801744644 learned_addr learned my addr 172.21.15.89:0/2801744644 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04585b640 1 --2- 172.21.15.89:0/2801744644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.896+0000 7fa04605c640 1 --2- 172.21.15.89:0/2801744644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b8d00 0x559e4b5d6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.897+0000 7fa04585b640 1 -- 172.21.15.89:0/2801744644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b8d00 msgr2=0x559e4b5d6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.897+0000 7fa04585b640 1 --2- 172.21.15.89:0/2801744644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b8d00 0x559e4b5d6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.897+0000 7fa04585b640 1 -- 172.21.15.89:0/2801744644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 msgr2=0x559e4b5d0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.897+0000 7fa04585b640 1 --2- 172.21.15.89:0/2801744644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5d0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.897+0000 7fa04585b640 1 -- 172.21.15.89:0/2801744644 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e4b5acda0 con 0x559e4b5c4740 2024-09-15T23:40:12.899 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.897+0000 7fa04585b640 1 --2- 172.21.15.89:0/2801744644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5dc720 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fa030051040 tx=0x7fa030004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5895a9f2b13bdab5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:12.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.897+0000 7fa02bfff640 1 -- 172.21.15.89:0/2801744644 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030035040 con 0x559e4b5c4740 2024-09-15T23:40:12.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.897+0000 7fa02bfff640 1 -- 172.21.15.89:0/2801744644 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa030032e30 con 0x559e4b5c4740 2024-09-15T23:40:12.900 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.897+0000 7fa02bfff640 1 -- 172.21.15.89:0/2801744644 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03004dae0 con 0x559e4b5c4740 2024-09-15T23:40:12.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.898+0000 7fa04a699740 1 -- 172.21.15.89:0/2801744644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 msgr2=0x559e4b5dc720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:12.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.898+0000 7fa04a699740 1 --2- 172.21.15.89:0/2801744644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5dc720 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7fa030051040 tx=0x7fa030004610 comp rx=0 tx=0).stop 2024-09-15T23:40:12.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.898+0000 7fa04a699740 1 -- 172.21.15.89:0/2801744644 shutdown_connections 2024-09-15T23:40:12.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.898+0000 7fa04a699740 1 --2- 172.21.15.89:0/2801744644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5c4740 0x559e4b5dc720 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.898+0000 7fa04a699740 1 --2- 172.21.15.89:0/2801744644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b8d00 0x559e4b5d6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.898+0000 7fa04a699740 1 --2- 172.21.15.89:0/2801744644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5b7ad0 0x559e4b5d0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.898+0000 7fa04a699740 1 -- 172.21.15.89:0/2801744644 >> 172.21.15.89:0/2801744644 conn(0x559e4b5b3c40 msgr2=0x559e4b5bfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:12.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.898+0000 7fa04a699740 1 -- 172.21.15.89:0/2801744644 shutdown_connections 2024-09-15T23:40:12.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.898+0000 7fa04a699740 1 -- 172.21.15.89:0/2801744644 wait complete. 2024-09-15T23:40:12.901 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04a699740 1 -- start start 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04a699740 1 Processor -- start 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04a699740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b7ad0 0x559e4b5cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04a699740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04605c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04605c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34596/0 (socket says 172.21.15.89:34596) 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04605c640 1 -- 172.21.15.89:0/787779068 learned_addr learned my addr 172.21.15.89:0/787779068 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04a699740 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5c4740 0x559e4b5dbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401add70 con 0x559e4b5b8d00 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401ad6a0 con 0x559e4b5c4740 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401aa8e0 con 0x559e4b5b7ad0 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04585b640 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b7ad0 0x559e4b5cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.902 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.899+0000 7fa04a699740 10 client.0 _ll_register_callbacks cb 0x7fa030004220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:12.903 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:12 smithi089 ceph-mon[30933]: pgmap v835: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-09-15T23:40:12.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa04585b640 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5c4740 0x559e4b5dbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa04605c640 1 -- 172.21.15.89:0/787779068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b7ad0 msgr2=0x559e4b5cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:12.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa04605c640 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b7ad0 0x559e4b5cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa04605c640 1 -- 172.21.15.89:0/787779068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5c4740 msgr2=0x559e4b5dbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:12.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa04605c640 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5c4740 0x559e4b5dbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.905 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa04605c640 1 -- 172.21.15.89:0/787779068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e4b5dc760 con 0x559e4b5b8d00 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa04605c640 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5cd380 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fa03c008b30 tx=0x7fa03c0056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427d18803bb3dfcc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa029ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa029ffb640 1 -- 172.21.15.89:0/787779068 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c029030 con 0x559e4b5b8d00 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa029ffb640 1 -- 172.21.15.89:0/787779068 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa03c004fe0 con 0x559e4b5b8d00 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa030032e30 con 0x559e4b5b8d00 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa029ffb640 1 -- 172.21.15.89:0/787779068 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa03c036050 con 0x559e4b5b8d00 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa04a699740 10 client.21138 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.900+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559e4b5be040 con 0x559e4b5b8d00 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.901+0000 7fa029ffb640 1 -- 172.21.15.89:0/787779068 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa03c00ee50 con 0x559e4b5b8d00 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.901+0000 7fa029ffb640 1 -- 172.21.15.89:0/787779068 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa03c010c10 con 0x559e4b5b8d00 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.901+0000 7fa04a699740 10 client.21138 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.901+0000 7fa04a699740 10 client.21138 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.901+0000 7fa04a699740 20 client.21138 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:12.906 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.901+0000 7fa04a699740 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4b643ee0 0x559e4b6462d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.901+0000 7fa04a699740 4 client.21138 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.901+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fa030032e30 con 0x559e4b643ee0 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.901+0000 7fa04585b640 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4b643ee0 0x559e4b6462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa029ffb640 10 client.21138 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04585b640 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4b643ee0 0x559e4b6462d0 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79c44eefa7256686 server_cookie=41586742a0662a59 in_seq=0 out_seq=0 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa029ffb640 1 -- 172.21.15.89:0/787779068 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fa030032e30 con 0x559e4b643ee0 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa029ffb640 10 client.21138 handle_command_reply: tid=0 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 client.21138 shutdown 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 msgr2=0x559e4b5cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5cd380 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7fa03c008b30 tx=0x7fa03c0056b0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 shutdown_connections 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559e4b643ee0 0x559e4b6462d0 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559e4b5c4740 0x559e4b5dbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e4b5b8d00 0x559e4b5cd380 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa029ffb640 0 client.21138 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:12.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 --2- 172.21.15.89:0/787779068 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559e4b5b7ad0 0x559e4b5cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 >> 172.21.15.89:0/787779068 conn(0x559e4b5b3c40 msgr2=0x559e4b5be700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:12.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 shutdown_connections 2024-09-15T23:40:12.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.902+0000 7fa04a699740 1 -- 172.21.15.89:0/787779068 wait complete. 2024-09-15T23:40:12.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa04a699740 20 client.21138 trim_cache size 0 max 16384 2024-09-15T23:40:12.908 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:40:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:40:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": { 2024-09-15T23:40:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "628be185-c365-4b3b-a1fd-eddc75aee342": { 2024-09-15T23:40:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "path": "/", 2024-09-15T23:40:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "tag": "628be185-c365-4b3b-a1fd-eddc75aee342", 2024-09-15T23:40:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "options": "recursive,force" 2024-09-15T23:40:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:12.909 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:40:12.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa0482c0640 1 -- 172.21.15.89:0/3270444662 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa02c0a3110 msgr2=0x7fa02c0a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:12.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa0482c0640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa02c0a3110 0x7fa02c0a55d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa038098340 tx=0x7fa038066380 comp rx=0 tx=0).stop 2024-09-15T23:40:12.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa0482c0640 1 -- 172.21.15.89:0/3270444662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa040106a80 msgr2=0x7fa04019bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:12.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa0482c0640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa040106a80 0x7fa04019bc20 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fa038009fd0 tx=0x7fa03800b260 comp rx=0 tx=0).stop 2024-09-15T23:40:12.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa0482c0640 1 -- 172.21.15.89:0/3270444662 shutdown_connections 2024-09-15T23:40:12.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa0482c0640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa02c0a3110 0x7fa02c0a55d0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa0482c0640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa040106a80 0x7fa04019bc20 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa0482c0640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa040101420 0x7fa040197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa0482c0640 1 --2- 172.21.15.89:0/3270444662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa040100a80 0x7fa040197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:12.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.903+0000 7fa0482c0640 1 -- 172.21.15.89:0/3270444662 >> 172.21.15.89:0/3270444662 conn(0x7fa0400fc7a0 msgr2=0x7fa0400fceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:12.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.904+0000 7fa0482c0640 1 -- 172.21.15.89:0/3270444662 shutdown_connections 2024-09-15T23:40:12.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:12.904+0000 7fa0482c0640 1 -- 172.21.15.89:0/3270444662 wait complete. 2024-09-15T23:40:12.911 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:12.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:12 smithi120 ceph-mon[35250]: pgmap v835: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-09-15T23:40:12.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:12 smithi174 ceph-mon[35586]: pgmap v835: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-09-15T23:40:13.827 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:13 smithi089 ceph-mon[30933]: scrub summary: idle 2024-09-15T23:40:13.853 INFO:tasks.cephfs.filesystem:scrub status for tag:628be185-c365-4b3b-a1fd-eddc75aee342 - {'path': '/', 'tag': '628be185-c365-4b3b-a1fd-eddc75aee342', 'options': 'recursive,force'} 2024-09-15T23:40:13.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:13 smithi174 ceph-mon[35586]: scrub summary: idle 2024-09-15T23:40:14.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.006+0000 7f3feb5c0640 1 -- 172.21.15.89:0/893983762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe4100a40 msgr2=0x7f3fe4100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.006+0000 7f3feb5c0640 1 --2- 172.21.15.89:0/893983762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe4100a40 0x7f3fe4100e40 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009900 tx=0x7f3fdc02a410 comp rx=0 tx=0).stop 2024-09-15T23:40:14.007 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.006+0000 7f3feb5c0640 1 -- 172.21.15.89:0/893983762 shutdown_connections 2024-09-15T23:40:14.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.006+0000 7f3feb5c0640 1 --2- 172.21.15.89:0/893983762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fe41025f0 0x7f3fe4108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.006+0000 7f3feb5c0640 1 --2- 172.21.15.89:0/893983762 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fe4101c30 0x7f3fe41020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.006+0000 7f3feb5c0640 1 --2- 172.21.15.89:0/893983762 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe4100a40 0x7f3fe4100e40 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.006+0000 7f3feb5c0640 1 -- 172.21.15.89:0/893983762 >> 172.21.15.89:0/893983762 conn(0x7f3fe40fc1e0 msgr2=0x7f3fe40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.006+0000 7f3feb5c0640 1 -- 172.21.15.89:0/893983762 shutdown_connections 2024-09-15T23:40:14.008 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.006+0000 7f3feb5c0640 1 -- 172.21.15.89:0/893983762 wait complete. 2024-09-15T23:40:14.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.007+0000 7f3feb5c0640 1 Processor -- start 2024-09-15T23:40:14.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.007+0000 7f3feb5c0640 1 -- start start 2024-09-15T23:40:14.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.007+0000 7f3feb5c0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fe4100a40 0x7f3fe4197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.007+0000 7f3feb5c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe4101c30 0x7f3fe4198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.007+0000 7f3feb5c0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fe41025f0 0x7f3fe419f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.007+0000 7f3feb5c0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe410d810 con 0x7f3fe4101c30 2024-09-15T23:40:14.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.007+0000 7f3feb5c0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe410d690 con 0x7f3fe41025f0 2024-09-15T23:40:14.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.007+0000 7f3feb5c0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe410d990 con 0x7f3fe4100a40 2024-09-15T23:40:14.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe8b5b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe4101c30 0x7f3fe4198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.010 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe8b5b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe4101c30 0x7f3fe4198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34618/0 (socket says 172.21.15.89:34618) 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/3747597967 learned_addr learned my addr 172.21.15.89:0/3747597967 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe935c640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fe4100a40 0x7f3fe4197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/3747597967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fe4100a40 msgr2=0x7f3fe4197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe8b5b640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fe4100a40 0x7f3fe4197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe9b5d640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fe41025f0 0x7f3fe419f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/3747597967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fe41025f0 msgr2=0x7f3fe419f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe8b5b640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fe41025f0 0x7f3fe419f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/3747597967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe419f960 con 0x7f3fe4101c30 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe8b5b640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe4101c30 0x7f3fe4198280 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800c920 tx=0x7f3fd800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe27fc640 1 -- 172.21.15.89:0/3747597967 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8007c70 con 0x7f3fe4101c30 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe27fc640 1 -- 172.21.15.89:0/3747597967 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd8007e10 con 0x7f3fe4101c30 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3fe27fc640 1 -- 172.21.15.89:0/3747597967 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8018890 con 0x7f3fe4101c30 2024-09-15T23:40:14.011 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3feb5c0640 1 -- 172.21.15.89:0/3747597967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe419fc50 con 0x7f3fe4101c30 2024-09-15T23:40:14.012 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.008+0000 7f3feb5c0640 1 -- 172.21.15.89:0/3747597967 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe41a01a0 con 0x7f3fe4101c30 2024-09-15T23:40:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fee42f740 1 Processor -- start 2024-09-15T23:40:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fee42f740 1 -- start start 2024-09-15T23:40:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 0x559ed53481b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed534d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed534d780 0x559ed534fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fee42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559ed52a72f0 con 0x559ed53486f0 2024-09-15T23:40:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fee42f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559ed524f270 con 0x559ed5347db0 2024-09-15T23:40:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fee42f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559ed52a32b0 con 0x559ed534d780 2024-09-15T23:40:14.018 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fe935c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed534d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fe9b5d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 0x559ed53481b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fe935c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed534d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34624/0 (socket says 172.21.15.89:34624) 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fe935c640 1 -- 172.21.15.89:0/3571621098 learned_addr learned my addr 172.21.15.89:0/3571621098 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fe9b5d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 0x559ed53481b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.89:57414/0 (socket says 172.21.15.89:57414) 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.011+0000 7f3fe27fc640 1 -- 172.21.15.89:0/3747597967 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3fd8018a70 con 0x7f3fe4101c30 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fe935c640 1 -- 172.21.15.89:0/3571621098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed534d780 msgr2=0x559ed534fc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fe935c640 1 --2- 172.21.15.89:0/3571621098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed534d780 0x559ed534fc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fe935c640 1 -- 172.21.15.89:0/3571621098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 msgr2=0x559ed53481b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fe935c640 1 --2- 172.21.15.89:0/3571621098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 0x559ed53481b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fe935c640 1 -- 172.21.15.89:0/3571621098 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ed524e340 con 0x559ed53486f0 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fe935c640 1 --2- 172.21.15.89:0/3571621098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed534d1b0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc031440 tx=0x7f3fdc032d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5522e7fc78cea26a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fcb7fe640 1 -- 172.21.15.89:0/3571621098 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc03f070 con 0x559ed53486f0 2024-09-15T23:40:14.019 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fcb7fe640 1 -- 172.21.15.89:0/3571621098 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fdc0318b0 con 0x559ed53486f0 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fcb7fe640 1 -- 172.21.15.89:0/3571621098 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc035070 con 0x559ed53486f0 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fe27fc640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3fcc0a3370 0x7f3fcc0a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fe9b5d640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3fcc0a3370 0x7f3fcc0a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.012+0000 7f3fe27fc640 1 -- 172.21.15.89:0/3747597967 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3fd80cfe60 con 0x7f3fe4101c30 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 -- 172.21.15.89:0/3571621098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 msgr2=0x559ed534d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fe9b5d640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3fcc0a3370 0x7f3fcc0a5830 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4005fd0 tx=0x7f3fd4009620 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 --2- 172.21.15.89:0/3571621098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed534d1b0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc031440 tx=0x7f3fdc032d60 comp rx=0 tx=0).stop 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 -- 172.21.15.89:0/3571621098 shutdown_connections 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 --2- 172.21.15.89:0/3571621098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed534d780 0x559ed534fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 --2- 172.21.15.89:0/3571621098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed534d1b0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 --2- 172.21.15.89:0/3571621098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 0x559ed53481b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 -- 172.21.15.89:0/3571621098 >> 172.21.15.89:0/3571621098 conn(0x559ed533f010 msgr2=0x559ed533f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 -- 172.21.15.89:0/3571621098 shutdown_connections 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 -- 172.21.15.89:0/3571621098 wait complete. 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 -- start start 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.013+0000 7f3fee42f740 1 Processor -- start 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed5347db0 0x559ed5351fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53524e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed534d780 0x559ed5359520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fee42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x559ed53506f0 con 0x559ed53486f0 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fee42f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559ed5230290 con 0x559ed534d780 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fee42f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559ed5350870 con 0x559ed5347db0 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fee42f740 10 client.0 _ll_register_callbacks cb 0x559ed5261140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe8b5b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe8b5b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34632/0 (socket says 172.21.15.89:34632) 2024-09-15T23:40:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/3152561344 learned_addr learned my addr 172.21.15.89:0/3152561344 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe935c640 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed5347db0 0x559ed5351fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe9b5d640 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed534d780 0x559ed5359520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/3152561344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed5347db0 msgr2=0x559ed5351fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe8b5b640 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed5347db0 0x559ed5351fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/3152561344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed534d780 msgr2=0x559ed5359520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe8b5b640 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed534d780 0x559ed5359520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/3152561344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ed4fba210 con 0x559ed53486f0 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fe8b5b640 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53524e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80984f0 tx=0x7f3fd809c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31681c348eb51e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/3152561344 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd80b4020 con 0x559ed53486f0 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/3152561344 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3fd8018d60 con 0x559ed53486f0 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.014+0000 7f3fee42f740 1 -- 172.21.15.89:0/3152561344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fdc0318b0 con 0x559ed53486f0 2024-09-15T23:40:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/3152561344 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3fd80a4d00 con 0x559ed53486f0 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fee42f740 10 client.21150 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fee42f740 1 -- 172.21.15.89:0/3152561344 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559ed5086620 con 0x559ed53486f0 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/3152561344 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fdc0318b0 con 0x559ed53486f0 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/3152561344 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3fd80a4870 con 0x559ed53486f0 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fee42f740 10 client.21150 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fee42f740 10 client.21150 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fee42f740 20 client.21150 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fee42f740 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559ed535dcc0 0x559ed53600b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fee42f740 4 client.21150 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fee42f740 1 -- 172.21.15.89:0/3152561344 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ed52b4a20 con 0x559ed535dcc0 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.015+0000 7f3fe935c640 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559ed535dcc0 0x559ed53600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.016+0000 7f3fe935c640 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559ed535dcc0 0x559ed53600b0 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=956f29b6947cd714 server_cookie=fca9211d94d9c1fa in_seq=0 out_seq=0 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.016+0000 7f3fc8ff9640 10 client.21150 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.016+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/3152561344 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559ed52b4a20 con 0x559ed535dcc0 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.016+0000 7f3fc8ff9640 10 client.21150 handle_command_reply: tid=0 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.016+0000 7f3fee42f740 1 client.21150 shutdown 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 1 -- 172.21.15.89:0/3152561344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 msgr2=0x559ed53524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53524e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80984f0 tx=0x7f3fd809c850 comp rx=0 tx=0).stop 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 1 -- 172.21.15.89:0/3152561344 shutdown_connections 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559ed535dcc0 0x559ed53600b0 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed534d780 0x559ed5359520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53524e0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fc8ff9640 0 client.21150 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 1 --2- 172.21.15.89:0/3152561344 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed5347db0 0x559ed5351fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 1 -- 172.21.15.89:0/3152561344 >> 172.21.15.89:0/3152561344 conn(0x559ed533f010 msgr2=0x559ed5340820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 1 -- 172.21.15.89:0/3152561344 shutdown_connections 2024-09-15T23:40:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 1 -- 172.21.15.89:0/3152561344 wait complete. 2024-09-15T23:40:14.025 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.017+0000 7f3fee42f740 20 client.21150 trim_cache size 0 max 16384 2024-09-15T23:40:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fee42f740 1 Processor -- start 2024-09-15T23:40:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fee42f740 1 -- start start 2024-09-15T23:40:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 0x559ed5364970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53594e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed534d780 0x559ed5364eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fee42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe41a8520 con 0x559ed53486f0 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fee42f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe41aa060 con 0x559ed5347db0 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fee42f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe41a8bf0 con 0x559ed534d780 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fe8b5b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fe8b5b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34640/0 (socket says 172.21.15.89:34640) 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/1586232681 learned_addr learned my addr 172.21.15.89:0/1586232681 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.031+0000 7f3fe935c640 1 --2- 172.21.15.89:0/1586232681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 0x559ed5364970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.032+0000 7f3fe9b5d640 1 --2- 172.21.15.89:0/1586232681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed534d780 0x559ed5364eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.032+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/1586232681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed534d780 msgr2=0x559ed5364eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.032+0000 7f3fe8b5b640 1 --2- 172.21.15.89:0/1586232681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed534d780 0x559ed5364eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.032+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/1586232681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 msgr2=0x559ed5364970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.034 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.032+0000 7f3fe8b5b640 1 --2- 172.21.15.89:0/1586232681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 0x559ed5364970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.032+0000 7f3fe8b5b640 1 -- 172.21.15.89:0/1586232681 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ed5348bd0 con 0x559ed53486f0 2024-09-15T23:40:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.032+0000 7f3fe8b5b640 1 --2- 172.21.15.89:0/1586232681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53594e0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80cd5c0 tx=0x7f3fd809a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7e8e7d1d8340fd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.032+0000 7f3fcaffd640 1 -- 172.21.15.89:0/1586232681 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd80b4070 con 0x559ed53486f0 2024-09-15T23:40:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.032+0000 7f3fcaffd640 1 -- 172.21.15.89:0/1586232681 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd80bc0c0 con 0x559ed53486f0 2024-09-15T23:40:14.035 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.032+0000 7f3fcaffd640 1 -- 172.21.15.89:0/1586232681 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd809af50 con 0x559ed53486f0 2024-09-15T23:40:14.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 -- 172.21.15.89:0/1586232681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 msgr2=0x559ed53594e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 --2- 172.21.15.89:0/1586232681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53594e0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80cd5c0 tx=0x7f3fd809a5a0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.038 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 -- 172.21.15.89:0/1586232681 shutdown_connections 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 --2- 172.21.15.89:0/1586232681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed534d780 0x559ed5364eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 --2- 172.21.15.89:0/1586232681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed53486f0 0x559ed53594e0 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 --2- 172.21.15.89:0/1586232681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed5347db0 0x559ed5364970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 -- 172.21.15.89:0/1586232681 >> 172.21.15.89:0/1586232681 conn(0x559ed533f010 msgr2=0x559ed533e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 -- 172.21.15.89:0/1586232681 shutdown_connections 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 -- 172.21.15.89:0/1586232681 wait complete. 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 -- start start 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.033+0000 7f3fee42f740 1 Processor -- start 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed5347db0 0x559ed5356d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed53486f0 0x559ed53572d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.039 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fee42f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed534d780 0x559ed53582e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fee42f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe41a9990 con 0x559ed534d780 2024-09-15T23:40:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fee42f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe41a92c0 con 0x559ed53486f0 2024-09-15T23:40:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fee42f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe41a6500 con 0x559ed5347db0 2024-09-15T23:40:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fee42f740 10 client.0 _ll_register_callbacks cb 0x7f3fd800cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe9b5d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed534d780 0x559ed53582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe9b5d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed534d780 0x559ed53582e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34642/0 (socket says 172.21.15.89:34642) 2024-09-15T23:40:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe9b5d640 1 -- 172.21.15.89:0/960447336 learned_addr learned my addr 172.21.15.89:0/960447336 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe935c640 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed5347db0 0x559ed5356d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe8b5b640 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed53486f0 0x559ed53572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe9b5d640 1 -- 172.21.15.89:0/960447336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed5347db0 msgr2=0x559ed5356d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe9b5d640 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed5347db0 0x559ed5356d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe9b5d640 1 -- 172.21.15.89:0/960447336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed53486f0 msgr2=0x559ed53572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe9b5d640 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed53486f0 0x559ed53572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe9b5d640 1 -- 172.21.15.89:0/960447336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ed53588f0 con 0x559ed534d780 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fe9b5d640 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed534d780 0x559ed53582e0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40128d0 tx=0x7f3fd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5753ecd533eda7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fc8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/960447336 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4021dd0 con 0x559ed534d780 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/960447336 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3fd401f450 con 0x559ed534d780 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.034+0000 7f3fee42f740 1 -- 172.21.15.89:0/960447336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fd80bc0c0 con 0x559ed534d780 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/960447336 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3fd4012e50 con 0x559ed534d780 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fee42f740 10 client.21156 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fee42f740 1 -- 172.21.15.89:0/960447336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559ed53c8b10 con 0x559ed534d780 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/960447336 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd80bc0c0 con 0x559ed534d780 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/960447336 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3fd4028070 con 0x559ed534d780 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fee42f740 10 client.21156 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fee42f740 10 client.21156 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fee42f740 20 client.21156 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fee42f740 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559ed53c9540 0x559ed53cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fee42f740 4 client.21156 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fee42f740 1 -- 172.21.15.89:0/960447336 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559ed52aed70 con 0x559ed53c9540 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.035+0000 7f3fe935c640 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559ed53c9540 0x559ed53cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.036+0000 7f3fc8ff9640 10 client.21156 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:14.042 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.036+0000 7f3fe935c640 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559ed53c9540 0x559ed53cb960 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81d1a9e9d08feab3 server_cookie=40fab1199118871f in_seq=0 out_seq=0 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fc8ff9640 1 -- 172.21.15.89:0/960447336 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559ed52aed70 con 0x559ed53c9540 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fc8ff9640 10 client.21156 handle_command_reply: tid=0 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 client.21156 shutdown 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 -- 172.21.15.89:0/960447336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed534d780 msgr2=0x559ed53582e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed534d780 0x559ed53582e0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40128d0 tx=0x7f3fd4004a40 comp rx=0 tx=0).stop 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 -- 172.21.15.89:0/960447336 shutdown_connections 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x559ed53c9540 0x559ed53cb960 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559ed534d780 0x559ed53582e0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559ed53486f0 0x559ed53572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 --2- 172.21.15.89:0/960447336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559ed5347db0 0x559ed5356d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 -- 172.21.15.89:0/960447336 >> 172.21.15.89:0/960447336 conn(0x559ed533f010 msgr2=0x559ed533fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fc8ff9640 0 client.21156 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 -- 172.21.15.89:0/960447336 shutdown_connections 2024-09-15T23:40:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 1 -- 172.21.15.89:0/960447336 wait complete. 2024-09-15T23:40:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.044+0000 7f3fee42f740 20 client.21156 trim_cache size 0 max 16384 2024-09-15T23:40:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:40:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:40:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:40:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:40:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 -- 172.21.15.89:0/3747597967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3fcc0a3370 msgr2=0x7f3fcc0a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3fcc0a3370 0x7f3fcc0a5830 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4005fd0 tx=0x7f3fd4009620 comp rx=0 tx=0).stop 2024-09-15T23:40:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 -- 172.21.15.89:0/3747597967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe4101c30 msgr2=0x7f3fe4198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe4101c30 0x7f3fe4198280 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800c920 tx=0x7f3fd800cde0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 -- 172.21.15.89:0/3747597967 shutdown_connections 2024-09-15T23:40:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3fcc0a3370 0x7f3fcc0a5830 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3fe41025f0 0x7f3fe419f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3fe4101c30 0x7f3fe4198280 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 --2- 172.21.15.89:0/3747597967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fe4100a40 0x7f3fe4197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 -- 172.21.15.89:0/3747597967 >> 172.21.15.89:0/3747597967 conn(0x7f3fe40fc1e0 msgr2=0x7f3fe4104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 -- 172.21.15.89:0/3747597967 shutdown_connections 2024-09-15T23:40:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:14.045+0000 7f3feb5c0640 1 -- 172.21.15.89:0/3747597967 wait complete. 2024-09-15T23:40:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.061+0000 7f3a29108640 1 -- 172.21.15.89:0/1310033086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a24102d30 msgr2=0x7f3a24103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.061+0000 7f3a29108640 1 --2- 172.21.15.89:0/1310033086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a24102d30 0x7f3a24103110 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009930 tx=0x7f3a1402a420 comp rx=0 tx=0).stop 2024-09-15T23:40:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.061+0000 7f3a29108640 1 -- 172.21.15.89:0/1310033086 shutdown_connections 2024-09-15T23:40:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.062+0000 7f3a29108640 1 --2- 172.21.15.89:0/1310033086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a241082a0 0x7f3a2410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.062+0000 7f3a29108640 1 --2- 172.21.15.89:0/1310033086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a241036e0 0x7f3a24107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.062+0000 7f3a29108640 1 --2- 172.21.15.89:0/1310033086 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a24102d30 0x7f3a24103110 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.062+0000 7f3a29108640 1 -- 172.21.15.89:0/1310033086 >> 172.21.15.89:0/1310033086 conn(0x7f3a240fc7d0 msgr2=0x7f3a240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.062+0000 7f3a29108640 1 -- 172.21.15.89:0/1310033086 shutdown_connections 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.062+0000 7f3a29108640 1 -- 172.21.15.89:0/1310033086 wait complete. 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.062+0000 7f3a29108640 1 Processor -- start 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a29108640 1 -- start start 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a29108640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a24102d30 0x7f3a2419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a29108640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a241036e0 0x7f3a2419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a29108640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a241082a0 0x7f3a2419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a29108640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3a241120a0 con 0x7f3a24102d30 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a29108640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3a24111f20 con 0x7f3a241036e0 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a29108640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a24112220 con 0x7f3a241082a0 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a24102d30 0x7f3a2419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a24102d30 0x7f3a2419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34660/0 (socket says 172.21.15.89:34660) 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a22d9d640 1 -- 172.21.15.89:0/976480500 learned_addr learned my addr 172.21.15.89:0/976480500 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a241036e0 0x7f3a2419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a2359e640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a241082a0 0x7f3a2419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a22d9d640 1 -- 172.21.15.89:0/976480500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a241082a0 msgr2=0x7f3a2419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a241082a0 0x7f3a2419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a22d9d640 1 -- 172.21.15.89:0/976480500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a241036e0 msgr2=0x7f3a2419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a241036e0 0x7f3a2419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a22d9d640 1 -- 172.21.15.89:0/976480500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2419f6c0 con 0x7f3a24102d30 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a241036e0 0x7f3a2419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a2359e640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a241082a0 0x7f3a2419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.063+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a24102d30 0x7f3a2419a710 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009230 tx=0x7f3a14004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.064+0000 7f3a20d99640 1 -- 172.21.15.89:0/976480500 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1403f030 con 0x7f3a24102d30 2024-09-15T23:40:14.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.064+0000 7f3a29108640 1 -- 172.21.15.89:0/976480500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a2419f950 con 0x7f3a24102d30 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.064+0000 7f3a20d99640 1 -- 172.21.15.89:0/976480500 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a14003ad0 con 0x7f3a24102d30 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.064+0000 7f3a20d99640 1 -- 172.21.15.89:0/976480500 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14035070 con 0x7f3a24102d30 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.064+0000 7f3a29108640 1 -- 172.21.15.89:0/976480500 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a241a7290 con 0x7f3a24102d30 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.066+0000 7f3a20d99640 1 -- 172.21.15.89:0/976480500 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f3a14042030 con 0x7f3a24102d30 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.066+0000 7f3a2b4e1740 1 Processor -- start 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.066+0000 7f3a2b4e1740 1 -- start start 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.066+0000 7f3a20d99640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a100a3240 0x7f3a100a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a2359e640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a100a3240 0x7f3a100a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a20d99640 1 -- 172.21.15.89:0/976480500 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3a140f02a0 con 0x7f3a24102d30 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b060 0x5650c910b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b980 0x5650c9115d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.075 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 0x5650c9118630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5650c9070620 con 0x5650c910b980 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5650c9017e50 con 0x5650c9116240 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5650c906c5e0 con 0x5650c910b060 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a1a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b980 0x5650c9115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a1a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b980 0x5650c9115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34674/0 (socket says 172.21.15.89:34674) 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a1a59c640 1 -- 172.21.15.89:0/1705746833 learned_addr learned my addr 172.21.15.89:0/1705746833 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/1705746833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b060 0x5650c910b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a2359e640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a100a3240 0x7f3a100a5700 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0092f0 tx=0x7f3a1c009280 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a1a59c640 1 -- 172.21.15.89:0/1705746833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b060 msgr2=0x5650c910b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a2359e640 1 --2- 172.21.15.89:0/1705746833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 0x5650c9118630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/1705746833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b060 0x5650c910b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a1a59c640 1 -- 172.21.15.89:0/1705746833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 msgr2=0x5650c9118630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/1705746833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 0x5650c9118630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.076 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a1a59c640 1 -- 172.21.15.89:0/1705746833 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650c9008d40 con 0x5650c910b980 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a2359e640 1 --2- 172.21.15.89:0/1705746833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 0x5650c9118630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/1705746833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b980 0x5650c9115d00 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c008bb0 tx=0x7f3a0c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e93faed9030c101 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.067+0000 7f3a1959a640 1 -- 172.21.15.89:0/1705746833 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c014070 con 0x5650c910b980 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a1959a640 1 -- 172.21.15.89:0/1705746833 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a0c0047a0 con 0x5650c910b980 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a1959a640 1 -- 172.21.15.89:0/1705746833 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c010670 con 0x5650c910b980 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1705746833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b980 msgr2=0x5650c9115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1705746833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b980 0x5650c9115d00 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c008bb0 tx=0x7f3a0c00e720 comp rx=0 tx=0).stop 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1705746833 shutdown_connections 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1705746833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 0x5650c9118630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1705746833 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b980 0x5650c9115d00 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1705746833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b060 0x5650c910b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1705746833 >> 172.21.15.89:0/1705746833 conn(0x5650c91075f0 msgr2=0x5650c91079d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1705746833 shutdown_connections 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.068+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1705746833 wait complete. 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2b4e1740 1 -- start start 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2b4e1740 1 Processor -- start 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b060 0x5650c9113180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b980 0x5650c91136c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 0x5650c9113c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x5650c9118e10 con 0x5650c910b060 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5650c8ff8f10 con 0x5650c9116240 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5650c9118f90 con 0x5650c910b980 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2b4e1740 10 client.0 _ll_register_callbacks cb 0x5650c901cdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b060 0x5650c9113180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a22d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b060 0x5650c9113180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34676/0 (socket says 172.21.15.89:34676) 2024-09-15T23:40:14.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a22d9d640 1 -- 172.21.15.89:0/1618446820 learned_addr learned my addr 172.21.15.89:0/1618446820 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2359e640 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 0x5650c9113c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b980 0x5650c91136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a22d9d640 1 -- 172.21.15.89:0/1618446820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b980 msgr2=0x5650c91136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b980 0x5650c91136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a22d9d640 1 -- 172.21.15.89:0/1618446820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 msgr2=0x5650c9113c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 0x5650c9113c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a22d9d640 1 -- 172.21.15.89:0/1618446820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650c8d7f580 con 0x5650c910b060 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.069+0000 7f3a2359e640 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 0x5650c9113c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b060 0x5650c9113180 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f3a140b8ea0 tx=0x7f3a14009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea26248133cfa8a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a0a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1618446820 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14049040 con 0x5650c910b060 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1618446820 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3a140b8b40 con 0x5650c910b060 2024-09-15T23:40:14.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1618446820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a0c0047a0 con 0x5650c910b060 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1618446820 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3a140c0880 con 0x5650c910b060 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a2b4e1740 10 client.21168 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1618446820 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5650c8e49960 con 0x5650c910b060 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1618446820 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a0c0047a0 con 0x5650c910b060 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1618446820 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3a14031c60 con 0x5650c910b060 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a2b4e1740 10 client.21168 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a2b4e1740 10 client.21168 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a2b4e1740 20 client.21168 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650c9125cc0 0x5650c9128110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a2b4e1740 4 client.21168 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.070+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1618446820 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650c907db90 con 0x5650c9125cc0 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.071+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650c9125cc0 0x5650c9128110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.071+0000 7f3a0a7fc640 10 client.21168 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:14.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.071+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650c9125cc0 0x5650c9128110 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f029fb70ea4d1 server_cookie=1f405d4bf8593dad in_seq=0 out_seq=0 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1618446820 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5650c907db90 con 0x5650c9125cc0 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a0a7fc640 10 client.21168 handle_command_reply: tid=0 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 client.21168 shutdown 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1618446820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b060 msgr2=0x5650c9113180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b060 0x5650c9113180 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f3a140b8ea0 tx=0x7f3a14009260 comp rx=0 tx=0).stop 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1618446820 shutdown_connections 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650c9125cc0 0x5650c9128110 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c9116240 0x5650c9113c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b980 0x5650c91136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a0a7fc640 0 client.21168 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1618446820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c910b060 0x5650c9113180 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1618446820 >> 172.21.15.89:0/1618446820 conn(0x5650c91075f0 msgr2=0x5650c910d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1618446820 shutdown_connections 2024-09-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1618446820 wait complete. 2024-09-15T23:40:14.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.072+0000 7f3a2b4e1740 20 client.21168 trim_cache size 0 max 16384 2024-09-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.088+0000 7f3a2b4e1740 1 Processor -- start 2024-09-15T23:40:14.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.088+0000 7f3a2b4e1740 1 -- start start 2024-09-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b060 0x5650c9106a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b980 0x5650c910dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c912da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3a241b1010 con 0x5650c9116240 2024-09-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3a241b2b50 con 0x5650c910b060 2024-09-15T23:40:14.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a241b16e0 con 0x5650c910b980 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c912da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c912da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34692/0 (socket says 172.21.15.89:34692) 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2359e640 1 -- 172.21.15.89:0/3371481481 learned_addr learned my addr 172.21.15.89:0/3371481481 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/3371481481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b980 0x5650c910dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/3371481481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b060 0x5650c9106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2359e640 1 -- 172.21.15.89:0/3371481481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b980 msgr2=0x5650c910dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2359e640 1 --2- 172.21.15.89:0/3371481481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b980 0x5650c910dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2359e640 1 -- 172.21.15.89:0/3371481481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b060 msgr2=0x5650c9106a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2359e640 1 --2- 172.21.15.89:0/3371481481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b060 0x5650c9106a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a2359e640 1 -- 172.21.15.89:0/3371481481 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650c9112130 con 0x5650c9116240 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.089+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/3371481481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b060 0x5650c9106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:14.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.090+0000 7f3a2359e640 1 --2- 172.21.15.89:0/3371481481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c912da20 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0127d0 tx=0x7f3a1c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76da3b6087794d8d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.090+0000 7f3a18d99640 1 -- 172.21.15.89:0/3371481481 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c011960 con 0x5650c9116240 2024-09-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.090+0000 7f3a18d99640 1 -- 172.21.15.89:0/3371481481 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a1c024070 con 0x5650c9116240 2024-09-15T23:40:14.094 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.090+0000 7f3a18d99640 1 -- 172.21.15.89:0/3371481481 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c01f430 con 0x5650c9116240 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.090+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/3371481481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 msgr2=0x5650c912da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.090+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/3371481481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c912da20 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0127d0 tx=0x7f3a1c013000 comp rx=0 tx=0).stop 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/3371481481 shutdown_connections 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/3371481481 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c912da20 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/3371481481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b980 0x5650c910dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/3371481481 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b060 0x5650c9106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/3371481481 >> 172.21.15.89:0/3371481481 conn(0x5650c91075f0 msgr2=0x5650c91149e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/3371481481 shutdown_connections 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/3371481481 wait complete. 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 -- start start 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 Processor -- start 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b060 0x5650c911e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.095 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b980 0x5650c911acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c911b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f3a241b2480 con 0x5650c9116240 2024-09-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3a241b1db0 con 0x5650c910b980 2024-09-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a241aeff0 con 0x5650c910b060 2024-09-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2b4e1740 10 client.0 _ll_register_callbacks cb 0x7f3a140bc010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c911b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2359e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c911b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34696/0 (socket says 172.21.15.89:34696) 2024-09-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.091+0000 7f3a2359e640 1 -- 172.21.15.89:0/1628263518 learned_addr learned my addr 172.21.15.89:0/1628263518 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:14.096 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b980 0x5650c911acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b060 0x5650c911e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a2359e640 1 -- 172.21.15.89:0/1628263518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b060 msgr2=0x5650c911e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a2359e640 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b060 0x5650c911e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a2359e640 1 -- 172.21.15.89:0/1628263518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b980 msgr2=0x5650c911acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a2359e640 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b980 0x5650c911acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a2359e640 1 -- 172.21.15.89:0/1628263518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650c911ba40 con 0x5650c9116240 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a1a59c640 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b980 0x5650c911acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a2359e640 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c911b210 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c006010 tx=0x7f3a1c012f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4adbb2d4100580aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a0a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1628263518 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c039060 con 0x5650c9116240 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1628263518 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f3a1c020b30 con 0x5650c9116240 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1628263518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a1c024070 con 0x5650c9116240 2024-09-15T23:40:14.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1628263518 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3a1c040050 con 0x5650c9116240 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a2b4e1740 10 client.21174 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.092+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1628263518 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5650c9191a10 con 0x5650c9116240 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.093+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1628263518 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a1c011e40 con 0x5650c9116240 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.093+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1628263518 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3a1c00f040 con 0x5650c9116240 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.093+0000 7f3a2b4e1740 10 client.21174 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.093+0000 7f3a2b4e1740 10 client.21174 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.093+0000 7f3a2b4e1740 20 client.21174 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.093+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650c9192020 0x5650c9194440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.093+0000 7f3a2b4e1740 4 client.21174 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2590a49a", "release": true, "await": true}] 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.093+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1628263518 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2590a49a", "release": true, "await": true}) -- 0x7f3a1c024070 con 0x5650c9192020 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.093+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650c9192020 0x5650c9194440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.094+0000 7f3a0a7fc640 10 client.21174 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:14.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.094+0000 7f3a22d9d640 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650c9192020 0x5650c9194440 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68896a6a092eb41d server_cookie=358b1a09a8891d77 in_seq=0 out_seq=0 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a0a7fc640 1 -- 172.21.15.89:0/1628263518 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f3a1c024070 con 0x5650c9192020 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a0a7fc640 10 client.21174 handle_command_reply: tid=0 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 client.21174 shutdown 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1628263518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 msgr2=0x5650c911b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c911b210 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c006010 tx=0x7f3a1c012f80 comp rx=0 tx=0).stop 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1628263518 shutdown_connections 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x5650c9192020 0x5650c9194440 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5650c9116240 0x5650c911b210 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650c910b980 0x5650c911acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 --2- 172.21.15.89:0/1628263518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5650c910b060 0x5650c911e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a0a7fc640 0 client.21174 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1628263518 >> 172.21.15.89:0/1628263518 conn(0x5650c91075f0 msgr2=0x5650c9113140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1628263518 shutdown_connections 2024-09-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.097+0000 7f3a2b4e1740 1 -- 172.21.15.89:0/1628263518 wait complete. 2024-09-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a2b4e1740 20 client.21174 trim_cache size 0 max 16384 2024-09-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 60, 2024-09-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": { 2024-09-15T23:40:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "2590a49a": { 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "version": 60, 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "age_ref": 0.0, 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "timeout": 90.0, 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "expiration": 137.7, 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "members": { 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "file:/": { 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "excluded": false, 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "state": { 2024-09-15T23:40:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RELEASED", 2024-09-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0 2024-09-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:40:14.103 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 -- 172.21.15.89:0/976480500 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a100a3240 msgr2=0x7f3a100a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a100a3240 0x7f3a100a5700 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0092f0 tx=0x7f3a1c009280 comp rx=0 tx=0).stop 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 -- 172.21.15.89:0/976480500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a24102d30 msgr2=0x7f3a2419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a24102d30 0x7f3a2419a710 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009230 tx=0x7f3a14004270 comp rx=0 tx=0).stop 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 -- 172.21.15.89:0/976480500 shutdown_connections 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f3a100a3240 0x7f3a100a5700 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a241082a0 0x7f3a2419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3a241036e0 0x7f3a2419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 --2- 172.21.15.89:0/976480500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3a24102d30 0x7f3a2419a710 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 -- 172.21.15.89:0/976480500 >> 172.21.15.89:0/976480500 conn(0x7f3a240fc7d0 msgr2=0x7f3a24101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 -- 172.21.15.89:0/976480500 shutdown_connections 2024-09-15T23:40:14.104 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:14.098+0000 7f3a29108640 1 -- 172.21.15.89:0/976480500 wait complete. 2024-09-15T23:40:14.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:13 smithi120 ceph-mon[35250]: scrub summary: idle 2024-09-15T23:40:14.789 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '2590a49a', total seconds elapsed: 23.0 2024-09-15T23:40:15.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:14 smithi089 ceph-mon[30933]: pgmap v836: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:40:15.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:14 smithi120 ceph-mon[35250]: pgmap v836: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:40:15.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:14 smithi174 ceph-mon[35586]: pgmap v836: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-09-15T23:40:17.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:16 smithi089 ceph-mon[30933]: pgmap v837: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-15T23:40:17.124 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:16 smithi174 ceph-mon[35586]: pgmap v837: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-15T23:40:17.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:16 smithi120 ceph-mon[35250]: pgmap v837: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-09-15T23:40:19.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:18 smithi089 ceph-mon[30933]: pgmap v838: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 KiB/s wr, 15 op/s 2024-09-15T23:40:19.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:18 smithi120 ceph-mon[35250]: pgmap v838: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 KiB/s wr, 15 op/s 2024-09-15T23:40:19.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:18 smithi174 ceph-mon[35586]: pgmap v838: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 KiB/s wr, 15 op/s 2024-09-15T23:40:21.025 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:20 smithi120 ceph-mon[35250]: pgmap v839: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-09-15T23:40:21.037 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:20 smithi174 ceph-mon[35586]: pgmap v839: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-09-15T23:40:21.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:20 smithi089 ceph-mon[30933]: pgmap v839: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-09-15T23:40:23.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:22 smithi089 ceph-mon[30933]: pgmap v840: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-15T23:40:23.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:22 smithi120 ceph-mon[35250]: pgmap v840: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-15T23:40:23.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:22 smithi174 ceph-mon[35586]: pgmap v840: 129 pgs: 129 active+clean; 4.0 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.7 KiB/s wr, 14 op/s 2024-09-15T23:40:25.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:24 smithi089 ceph-mon[30933]: pgmap v841: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.4 KiB/s wr, 15 op/s 2024-09-15T23:40:25.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:24 smithi120 ceph-mon[35250]: pgmap v841: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.4 KiB/s wr, 15 op/s 2024-09-15T23:40:25.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:24 smithi174 ceph-mon[35586]: pgmap v841: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.4 KiB/s wr, 15 op/s 2024-09-15T23:40:27.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:26 smithi089 ceph-mon[30933]: pgmap v842: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 KiB/s wr, 13 op/s 2024-09-15T23:40:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.141+0000 7f94b12a6640 1 -- 172.21.15.89:0/2545367725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac101d80 msgr2=0x7f94ac108610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.141+0000 7f94b12a6640 1 --2- 172.21.15.89:0/2545367725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac101d80 0x7f94ac108610 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009ca0 tx=0x7f94a402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.142 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.141+0000 7f94b12a6640 1 -- 172.21.15.89:0/2545367725 shutdown_connections 2024-09-15T23:40:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.141+0000 7f94b12a6640 1 --2- 172.21.15.89:0/2545367725 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac101d80 0x7f94ac108610 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.141+0000 7f94b12a6640 1 --2- 172.21.15.89:0/2545367725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac1013c0 0x7f94ac101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.141+0000 7f94b12a6640 1 --2- 172.21.15.89:0/2545367725 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94ac1001d0 0x7f94ac1005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.141+0000 7f94b12a6640 1 -- 172.21.15.89:0/2545367725 >> 172.21.15.89:0/2545367725 conn(0x7f94ac0fb930 msgr2=0x7f94ac0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.141+0000 7f94b12a6640 1 -- 172.21.15.89:0/2545367725 shutdown_connections 2024-09-15T23:40:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.141+0000 7f94b12a6640 1 -- 172.21.15.89:0/2545367725 wait complete. 2024-09-15T23:40:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.141+0000 7f94b12a6640 1 Processor -- start 2024-09-15T23:40:27.143 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94b12a6640 1 -- start start 2024-09-15T23:40:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94b12a6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1001d0 0x7f94ac195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94b12a6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac1013c0 0x7f94ac196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.145 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94b12a6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94ac101d80 0x7f94ac19d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94b12a6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac10b600 con 0x7f94ac1001d0 2024-09-15T23:40:27.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94b12a6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac10b480 con 0x7f94ac1013c0 2024-09-15T23:40:27.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94b12a6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac10b780 con 0x7f94ac101d80 2024-09-15T23:40:27.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1001d0 0x7f94ac195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1001d0 0x7f94ac195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45558/0 (socket says 172.21.15.89:45558) 2024-09-15T23:40:27.146 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aaffd640 1 -- 172.21.15.89:0/1426415506 learned_addr learned my addr 172.21.15.89:0/1426415506 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94ab7fe640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94ac101d80 0x7f94ac19d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac1013c0 0x7f94ac196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aaffd640 1 -- 172.21.15.89:0/1426415506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94ac101d80 msgr2=0x7f94ac19d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aaffd640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94ac101d80 0x7f94ac19d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aaffd640 1 -- 172.21.15.89:0/1426415506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac1013c0 msgr2=0x7f94ac196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:27.147 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aaffd640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac1013c0 0x7f94ac196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aaffd640 1 -- 172.21.15.89:0/1426415506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94ac19d750 con 0x7f94ac1001d0 2024-09-15T23:40:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac1013c0 0x7f94ac196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.142+0000 7f94aaffd640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1001d0 0x7f94ac195b30 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f949c00d950 tx=0x7f949c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.143+0000 7f9493fff640 1 -- 172.21.15.89:0/1426415506 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c017070 con 0x7f94ac1001d0 2024-09-15T23:40:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.143+0000 7f9493fff640 1 -- 172.21.15.89:0/1426415506 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f949c009c30 con 0x7f94ac1001d0 2024-09-15T23:40:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.143+0000 7f94b12a6640 1 -- 172.21.15.89:0/1426415506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94ac19da40 con 0x7f94ac1001d0 2024-09-15T23:40:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.143+0000 7f9493fff640 1 -- 172.21.15.89:0/1426415506 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c00f600 con 0x7f94ac1001d0 2024-09-15T23:40:27.148 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.143+0000 7f94b12a6640 1 -- 172.21.15.89:0/1426415506 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94ac19df40 con 0x7f94ac1001d0 2024-09-15T23:40:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.145+0000 7f9493fff640 1 -- 172.21.15.89:0/1426415506 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f949c01a070 con 0x7f94ac1001d0 2024-09-15T23:40:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.145+0000 7f94b4039740 1 Processor -- start 2024-09-15T23:40:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.145+0000 7f94b4039740 1 -- start start 2024-09-15T23:40:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.145+0000 7f94b4039740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 0x55bf2125dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.145+0000 7f94b4039740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125e2e0 0x55bf21262da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.145+0000 7f94b4039740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf21263370 0x55bf21265830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.145+0000 7f94b4039740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bf211c12f0 con 0x55bf21263370 2024-09-15T23:40:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.145+0000 7f94b4039740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bf21169270 con 0x55bf2125e2e0 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.145+0000 7f94b4039740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bf211bd2b0 con 0x55bf2125d9a0 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f9493fff640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94940a32f0 0x7f94940a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aa7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf21263370 0x55bf21265830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aa7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf21263370 0x55bf21265830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45568/0 (socket says 172.21.15.89:45568) 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aa7fc640 1 -- 172.21.15.89:0/4075093075 learned_addr learned my addr 172.21.15.89:0/4075093075 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94ab7fe640 1 --2- 172.21.15.89:0/4075093075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 0x55bf2125dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94ab7fe640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94940a32f0 0x7f94940a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f9493fff640 1 -- 172.21.15.89:0/1426415506 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f949c0cf840 con 0x7f94ac1001d0 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aaffd640 1 --2- 172.21.15.89:0/4075093075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125e2e0 0x55bf21262da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aa7fc640 1 -- 172.21.15.89:0/4075093075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 msgr2=0x55bf2125dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/4075093075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 0x55bf2125dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aa7fc640 1 -- 172.21.15.89:0/4075093075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125e2e0 msgr2=0x55bf21262da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:27.158 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/4075093075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125e2e0 0x55bf21262da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aa7fc640 1 -- 172.21.15.89:0/4075093075 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf21168340 con 0x55bf21263370 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aaffd640 1 --2- 172.21.15.89:0/4075093075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125e2e0 0x55bf21262da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/4075093075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf21263370 0x55bf21265830 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f94a000ad30 tx=0x7f94a000f7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45837718ddf3447b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94ab7fe640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94940a32f0 0x7f94940a57b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f94a4005cc0 tx=0x7f94a40095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.146+0000 7f94917fa640 1 -- 172.21.15.89:0/4075093075 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a001c070 con 0x55bf21263370 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94917fa640 1 -- 172.21.15.89:0/4075093075 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94a000ae20 con 0x55bf21263370 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94917fa640 1 -- 172.21.15.89:0/4075093075 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0017b80 con 0x55bf21263370 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94b4039740 1 -- 172.21.15.89:0/4075093075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf21263370 msgr2=0x55bf21265830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94b4039740 1 --2- 172.21.15.89:0/4075093075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf21263370 0x55bf21265830 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f94a000ad30 tx=0x7f94a000f7d0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94b4039740 1 -- 172.21.15.89:0/4075093075 shutdown_connections 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94b4039740 1 --2- 172.21.15.89:0/4075093075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf21263370 0x55bf21265830 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94b4039740 1 --2- 172.21.15.89:0/4075093075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125e2e0 0x55bf21262da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94b4039740 1 --2- 172.21.15.89:0/4075093075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 0x55bf2125dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94b4039740 1 -- 172.21.15.89:0/4075093075 >> 172.21.15.89:0/4075093075 conn(0x55bf21259010 msgr2=0x55bf21259410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94b4039740 1 -- 172.21.15.89:0/4075093075 shutdown_connections 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.147+0000 7f94b4039740 1 -- 172.21.15.89:0/4075093075 wait complete. 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94b4039740 1 -- start start 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94b4039740 1 Processor -- start 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94b4039740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125d9a0 0x55bf21267b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94b4039740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212680a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94b4039740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 0x55bf2126f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94b4039740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55bf21266340 con 0x55bf2125e2e0 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94b4039740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bf2114a290 con 0x55bf2125d9a0 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94b4039740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bf212664c0 con 0x55bf21263370 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94b4039740 10 client.0 _ll_register_callbacks cb 0x55bf2117b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94aa7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212680a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.160 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94aa7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212680a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45578/0 (socket says 172.21.15.89:45578) 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94aa7fc640 1 -- 172.21.15.89:0/1906326428 learned_addr learned my addr 172.21.15.89:0/1906326428 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94ab7fe640 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 0x55bf2126f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94aaffd640 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125d9a0 0x55bf21267b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94aa7fc640 1 -- 172.21.15.89:0/1906326428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 msgr2=0x55bf2126f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 0x55bf2126f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94aa7fc640 1 -- 172.21.15.89:0/1906326428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125d9a0 msgr2=0x55bf21267b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125d9a0 0x55bf21267b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.148+0000 7f94aa7fc640 1 -- 172.21.15.89:0/1906326428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf20ed4210 con 0x55bf2125e2e0 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212680a0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f94a001a040 tx=0x7f94a0018f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388551700f3697a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f948eff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f948eff5640 1 -- 172.21.15.89:0/1906326428 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0024070 con 0x55bf2125e2e0 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f948eff5640 1 -- 172.21.15.89:0/1906326428 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f94a0005040 con 0x55bf2125e2e0 2024-09-15T23:40:27.161 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f94b4039740 1 -- 172.21.15.89:0/1906326428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94a000ae20 con 0x55bf2125e2e0 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f948eff5640 1 -- 172.21.15.89:0/1906326428 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94a0030050 con 0x55bf2125e2e0 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f94b4039740 10 client.21186 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f94b4039740 1 -- 172.21.15.89:0/1906326428 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bf20fa0620 con 0x55bf2125e2e0 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f948eff5640 1 -- 172.21.15.89:0/1906326428 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94a000ae20 con 0x55bf2125e2e0 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f948eff5640 1 -- 172.21.15.89:0/1906326428 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f94a000b040 con 0x55bf2125e2e0 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f94b4039740 10 client.21186 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f94b4039740 10 client.21186 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f94b4039740 20 client.21186 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f94b4039740 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bf212734f0 0x55bf21275920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f94b4039740 4 client.21186 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.149+0000 7f94b4039740 1 -- 172.21.15.89:0/1906326428 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf211cea20 con 0x55bf212734f0 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.150+0000 7f94aaffd640 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bf212734f0 0x55bf21275920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.150+0000 7f94aaffd640 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bf212734f0 0x55bf21275920 crc :-1 s=READY pgs=430 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a31309fa619e84ba server_cookie=8e5ec9972890e83a in_seq=0 out_seq=0 2024-09-15T23:40:27.162 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.150+0000 7f948eff5640 10 client.21186 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f948eff5640 1 -- 172.21.15.89:0/1906326428 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bf211cea20 con 0x55bf212734f0 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f948eff5640 10 client.21186 handle_command_reply: tid=0 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 client.21186 shutdown 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 -- 172.21.15.89:0/1906326428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 msgr2=0x55bf212680a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212680a0 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f94a001a040 tx=0x7f94a0018f70 comp rx=0 tx=0).stop 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 -- 172.21.15.89:0/1906326428 shutdown_connections 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bf212734f0 0x55bf21275920 crc :-1 s=READY pgs=430 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 0x55bf2126f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212680a0 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 --2- 172.21.15.89:0/1906326428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125d9a0 0x55bf21267b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f948eff5640 0 client.21186 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 -- 172.21.15.89:0/1906326428 >> 172.21.15.89:0/1906326428 conn(0x55bf21259010 msgr2=0x55bf2125a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 -- 172.21.15.89:0/1906326428 shutdown_connections 2024-09-15T23:40:27.163 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 1 -- 172.21.15.89:0/1906326428 wait complete. 2024-09-15T23:40:27.164 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.151+0000 7f94b4039740 20 client.21186 trim_cache size 0 max 16384 2024-09-15T23:40:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94b4039740 1 Processor -- start 2024-09-15T23:40:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94b4039740 1 -- start start 2024-09-15T23:40:27.171 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94b4039740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 0x55bf2125dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94b4039740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf2125a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94b4039740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf21263370 0x55bf2125a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94b4039740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1a8630 con 0x55bf2125e2e0 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94b4039740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1aa170 con 0x55bf21263370 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94b4039740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1a8d00 con 0x55bf2125d9a0 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aa7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf2125a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aa7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf2125a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45580/0 (socket says 172.21.15.89:45580) 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aa7fc640 1 -- 172.21.15.89:0/80313561 learned_addr learned my addr 172.21.15.89:0/80313561 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94ab7fe640 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf21263370 0x55bf2125a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aa7fc640 1 -- 172.21.15.89:0/80313561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 msgr2=0x55bf2125dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aaffd640 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 0x55bf2125dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 0x55bf2125dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.172 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aa7fc640 1 -- 172.21.15.89:0/80313561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf21263370 msgr2=0x55bf2125a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf21263370 0x55bf2125a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aa7fc640 1 -- 172.21.15.89:0/80313561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf2125e7c0 con 0x55bf2125e2e0 2024-09-15T23:40:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94ab7fe640 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf21263370 0x55bf2125a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aaffd640 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 0x55bf2125dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.166+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf2125a400 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f94a0000bc0 tx=0x7f94a000a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba977cbdf4a21582 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f9490ff9640 1 -- 172.21.15.89:0/80313561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0024070 con 0x55bf2125e2e0 2024-09-15T23:40:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f9490ff9640 1 -- 172.21.15.89:0/80313561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94a00174f0 con 0x55bf2125e2e0 2024-09-15T23:40:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f9490ff9640 1 -- 172.21.15.89:0/80313561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a001c040 con 0x55bf2125e2e0 2024-09-15T23:40:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f94b4039740 1 -- 172.21.15.89:0/80313561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 msgr2=0x55bf2125a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:27.173 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f94b4039740 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf2125a400 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f94a0000bc0 tx=0x7f94a000a5d0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f94b4039740 1 -- 172.21.15.89:0/80313561 shutdown_connections 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f94b4039740 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf21263370 0x55bf2125a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f94b4039740 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf2125a400 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f94b4039740 1 --2- 172.21.15.89:0/80313561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf2125d9a0 0x55bf2125dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f94b4039740 1 -- 172.21.15.89:0/80313561 >> 172.21.15.89:0/80313561 conn(0x55bf21259010 msgr2=0x55bf212581c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f94b4039740 1 -- 172.21.15.89:0/80313561 shutdown_connections 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.167+0000 7f94b4039740 1 -- 172.21.15.89:0/80313561 wait complete. 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94b4039740 1 -- start start 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94b4039740 1 Processor -- start 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94b4039740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125d9a0 0x55bf212d9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94b4039740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212d98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94b4039740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 0x55bf212e08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94b4039740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1a9aa0 con 0x55bf2125e2e0 2024-09-15T23:40:27.174 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94b4039740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1a93d0 con 0x55bf2125d9a0 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94b4039740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94ac1a6610 con 0x55bf21263370 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94b4039740 10 client.0 _ll_register_callbacks cb 0x7f94a001a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94aa7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212d98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94aa7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212d98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45582/0 (socket says 172.21.15.89:45582) 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94aa7fc640 1 -- 172.21.15.89:0/3392986020 learned_addr learned my addr 172.21.15.89:0/3392986020 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94aaffd640 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125d9a0 0x55bf212d9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94ab7fe640 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 0x55bf212e08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94aa7fc640 1 -- 172.21.15.89:0/3392986020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 msgr2=0x55bf212e08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 0x55bf212e08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94aa7fc640 1 -- 172.21.15.89:0/3392986020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125d9a0 msgr2=0x55bf212d9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125d9a0 0x55bf212d9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.175 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94aa7fc640 1 -- 172.21.15.89:0/3392986020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf212e0f00 con 0x55bf2125e2e0 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.168+0000 7f94ab7fe640 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 0x55bf212e08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94aa7fc640 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212d98b0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f94a0005150 tx=0x7f94a0005180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886d1b6e44958f47 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f948eff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f948eff5640 1 -- 172.21.15.89:0/3392986020 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a001c040 con 0x55bf2125e2e0 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f948eff5640 1 -- 172.21.15.89:0/3392986020 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f94a000d840 con 0x55bf2125e2e0 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94b4039740 1 -- 172.21.15.89:0/3392986020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94a00174f0 con 0x55bf2125e2e0 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f948eff5640 1 -- 172.21.15.89:0/3392986020 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94a0024040 con 0x55bf2125e2e0 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94b4039740 10 client.21192 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94b4039740 1 -- 172.21.15.89:0/3392986020 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bf212e2180 con 0x55bf2125e2e0 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f948eff5640 1 -- 172.21.15.89:0/3392986020 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94a00174f0 con 0x55bf2125e2e0 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f948eff5640 1 -- 172.21.15.89:0/3392986020 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f94a0029440 con 0x55bf2125e2e0 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94b4039740 10 client.21192 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:27.176 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94b4039740 10 client.21192 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94b4039740 20 client.21192 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94b4039740 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bf212e2b20 0x55bf212e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94b4039740 4 client.21192 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:40:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94b4039740 1 -- 172.21.15.89:0/3392986020 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bf211c8d70 con 0x55bf212e2b20 2024-09-15T23:40:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.169+0000 7f94aaffd640 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bf212e2b20 0x55bf212e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.170+0000 7f94aaffd640 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bf212e2b20 0x55bf212e4f40 crc :-1 s=READY pgs=431 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=850b003b164e9611 server_cookie=16d194ef57cbf4ca in_seq=0 out_seq=0 2024-09-15T23:40:27.177 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.170+0000 7f948eff5640 10 client.21192 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f948eff5640 1 -- 172.21.15.89:0/3392986020 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bf211c8d70 con 0x55bf212e2b20 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f948eff5640 10 client.21192 handle_command_reply: tid=0 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f94b4039740 1 client.21192 shutdown 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f94b4039740 1 -- 172.21.15.89:0/3392986020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 msgr2=0x55bf212d98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f94b4039740 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212d98b0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f94a0005150 tx=0x7f94a0005180 comp rx=0 tx=0).stop 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f94b4039740 1 -- 172.21.15.89:0/3392986020 shutdown_connections 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f94b4039740 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55bf212e2b20 0x55bf212e4f40 crc :-1 s=READY pgs=431 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f94b4039740 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bf21263370 0x55bf212e08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f94b4039740 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf2125e2e0 0x55bf212d98b0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f94b4039740 1 --2- 172.21.15.89:0/3392986020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bf2125d9a0 0x55bf212d9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f948eff5640 0 client.21192 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.183+0000 7f94b4039740 1 -- 172.21.15.89:0/3392986020 >> 172.21.15.89:0/3392986020 conn(0x55bf21259010 msgr2=0x55bf212581c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b4039740 1 -- 172.21.15.89:0/3392986020 shutdown_connections 2024-09-15T23:40:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b4039740 1 -- 172.21.15.89:0/3392986020 wait complete. 2024-09-15T23:40:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b4039740 20 client.21192 trim_cache size 0 max 16384 2024-09-15T23:40:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:{ 2024-09-15T23:40:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "message": "", 2024-09-15T23:40:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout: "return_code": 0 2024-09-15T23:40:27.186 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stdout:} 2024-09-15T23:40:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 -- 172.21.15.89:0/1426415506 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94940a32f0 msgr2=0x7f94940a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94940a32f0 0x7f94940a57b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f94a4005cc0 tx=0x7f94a40095c0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 -- 172.21.15.89:0/1426415506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1001d0 msgr2=0x7f94ac195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1001d0 0x7f94ac195b30 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f949c00d950 tx=0x7f949c00de10 comp rx=0 tx=0).stop 2024-09-15T23:40:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 -- 172.21.15.89:0/1426415506 shutdown_connections 2024-09-15T23:40:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f94940a32f0 0x7f94940a57b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94ac101d80 0x7f94ac19d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.187 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94ac1013c0 0x7f94ac196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 --2- 172.21.15.89:0/1426415506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94ac1001d0 0x7f94ac195b30 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 -- 172.21.15.89:0/1426415506 >> 172.21.15.89:0/1426415506 conn(0x7f94ac0fb930 msgr2=0x7f94ac104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 -- 172.21.15.89:0/1426415506 shutdown_connections 2024-09-15T23:40:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi089.front.sepia.ceph.com.smithi089.stderr:2024-09-15T23:40:27.184+0000 7f94b12a6640 1 -- 172.21.15.89:0/1426415506 wait complete. 2024-09-15T23:40:27.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:26 smithi120 ceph-mon[35250]: pgmap v842: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 KiB/s wr, 13 op/s 2024-09-15T23:40:27.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:26 smithi174 ceph-mon[35586]: pgmap v842: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 KiB/s wr, 13 op/s 2024-09-15T23:40:29.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:28 smithi089 ceph-mon[30933]: pgmap v843: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 18 op/s 2024-09-15T23:40:29.208 INFO:tasks.workunit.client.0.smithi089.stdout:561/600: open fd = 3 2024-09-15T23:40:29.208 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:562/600: open fd = 3 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:563/600: open fd = 3 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:564/600: open fd = 3 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.209 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:565/600: open fd = 3 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:566/600: open fd = 3 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:567/600: open fd = 3 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.210 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:568/600: open fd = 3 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:569/600: open fd = 3 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:570/600: open fd = 3 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.211 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:571/600: open fd = 3 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:572/600: open fd = 3 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:573/600: open fd = 3 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.212 INFO:tasks.workunit.client.0.smithi089.stdout:574/600: open fd = 3 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:575/600: open fd = 3 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:576/600: open fd = 3 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:577/600: open fd = 3 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.213 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:578/600: open fd = 3 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:579/600: open fd = 3 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:580/600: open fd = 3 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.214 INFO:tasks.workunit.client.0.smithi089.stdout:581/600: open fd = 3 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:582/600: open fd = 3 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:583/600: open fd = 3 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:584/600: open fd = 3 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.215 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:585/600: open fd = 3 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:586/600: open fd = 3 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:587/600: open fd = 3 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.216 INFO:tasks.workunit.client.0.smithi089.stdout:588/600: open fd = 3 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:589/600: open fd = 3 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:590/600: open fd = 3 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:591/600: open fd = 3 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.217 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:592/600: open fd = 3 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:593/600: open fd = 3 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:594/600: open fd = 3 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:595/600: open fd = 3 2024-09-15T23:40:29.218 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:596/600: open fd = 3 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:597/600: open fd = 3 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:598/600: open fd = 3 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.219 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:599/600: open fd = 3 2024-09-15T23:40:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:600/600: open fd = 3 2024-09-15T23:40:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:write ret = 32 2024-09-15T23:40:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:pread ret = 64 2024-09-15T23:40:29.220 INFO:tasks.workunit.client.0.smithi089.stdout:OK 2024-09-15T23:40:29.221 INFO:tasks.workunit.client.0.smithi089.stderr:+ echo OK 2024-09-15T23:40:29.221 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T23:40:29.222 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T23:40:29.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:28 smithi120 ceph-mon[35250]: pgmap v843: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 18 op/s 2024-09-15T23:40:29.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:28 smithi174 ceph-mon[35586]: pgmap v843: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 18 op/s 2024-09-15T23:40:29.267 INFO:tasks.workunit:Stopping ['fs/test_o_trunc.sh'] on client.0... 2024-09-15T23:40:29.268 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T23:40:29.561 DEBUG:teuthology.parallel:result is None 2024-09-15T23:40:29.562 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T23:40:29.598 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T23:40:29.598 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-09-15T23:40:29.609 INFO:tasks.quiescer:joining Quiescers 2024-09-15T23:40:29.610 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-09-15T23:40:29.610 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:40:29.887 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:30.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.452+0000 7feb7e6c6640 1 -- 172.21.15.89:0/431515446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb78102570 msgr2=0x7feb781029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:30.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.452+0000 7feb7e6c6640 1 --2- 172.21.15.89:0/431515446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb78102570 0x7feb781029b0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009900 tx=0x7feb6c02a480 comp rx=0 tx=0).stop 2024-09-15T23:40:30.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.452+0000 7feb7e6c6640 1 -- 172.21.15.89:0/431515446 shutdown_connections 2024-09-15T23:40:30.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.452+0000 7feb7e6c6640 1 --2- 172.21.15.89:0/431515446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb78102ef0 0x7feb7810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:30.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.452+0000 7feb7e6c6640 1 --2- 172.21.15.89:0/431515446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb78102570 0x7feb781029b0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:30.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.452+0000 7feb7e6c6640 1 --2- 172.21.15.89:0/431515446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb78100e20 0x7feb78101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:30.471 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.452+0000 7feb7e6c6640 1 -- 172.21.15.89:0/431515446 >> 172.21.15.89:0/431515446 conn(0x7feb780fc7d0 msgr2=0x7feb780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.452+0000 7feb7e6c6640 1 -- 172.21.15.89:0/431515446 shutdown_connections 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.452+0000 7feb7e6c6640 1 -- 172.21.15.89:0/431515446 wait complete. 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.453+0000 7feb7e6c6640 1 Processor -- start 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.453+0000 7feb7e6c6640 1 -- start start 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb7e6c6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb78100e20 0x7feb78198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb7e6c6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb78102570 0x7feb78198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb7e6c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb78102ef0 0x7feb7819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb7e6c6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7feb78110f80 con 0x7feb78100e20 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb7e6c6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feb78110e00 con 0x7feb78102570 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb7e6c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb78111100 con 0x7feb78102ef0 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb77fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb78100e20 0x7feb78198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb77fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb78100e20 0x7feb78198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45604/0 (socket says 172.21.15.89:45604) 2024-09-15T23:40:30.472 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb77fff640 1 -- 172.21.15.89:0/2414397113 learned_addr learned my addr 172.21.15.89:0/2414397113 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb7cc63640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb78102ef0 0x7feb7819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb777fe640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb78102570 0x7feb78198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb77fff640 1 -- 172.21.15.89:0/2414397113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb78102ef0 msgr2=0x7feb7819ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb77fff640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb78102ef0 0x7feb7819ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb77fff640 1 -- 172.21.15.89:0/2414397113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb78102570 msgr2=0x7feb78198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb77fff640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb78102570 0x7feb78198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb77fff640 1 -- 172.21.15.89:0/2414397113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7819d540 con 0x7feb78100e20 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb777fe640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb78102570 0x7feb78198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.454+0000 7feb77fff640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb78100e20 0x7feb78198590 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7feb6800d770 tx=0x7feb6800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.455+0000 7feb757fa640 1 -- 172.21.15.89:0/2414397113 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68017070 con 0x7feb78100e20 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.455+0000 7feb757fa640 1 -- 172.21.15.89:0/2414397113 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb68005500 con 0x7feb78100e20 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.455+0000 7feb757fa640 1 -- 172.21.15.89:0/2414397113 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68004e00 con 0x7feb78100e20 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.455+0000 7feb7e6c6640 1 -- 172.21.15.89:0/2414397113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7819d830 con 0x7feb78100e20 2024-09-15T23:40:30.473 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.455+0000 7feb7e6c6640 1 -- 172.21.15.89:0/2414397113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb781a5110 con 0x7feb78100e20 2024-09-15T23:40:30.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.456+0000 7feb7e6c6640 1 -- 172.21.15.89:0/2414397113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb78101200 con 0x7feb78100e20 2024-09-15T23:40:30.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.457+0000 7feb757fa640 1 -- 172.21.15.89:0/2414397113 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7feb68004fa0 con 0x7feb78100e20 2024-09-15T23:40:30.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.458+0000 7feb757fa640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb600a32c0 0x7feb600a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:30.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.458+0000 7feb757fa640 1 -- 172.21.15.89:0/2414397113 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7feb680cfb40 con 0x7feb78100e20 2024-09-15T23:40:30.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.458+0000 7feb777fe640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb600a32c0 0x7feb600a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:30.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.458+0000 7feb777fe640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb600a32c0 0x7feb600a5780 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7feb78100020 tx=0x7feb6c005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:30.474 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.461+0000 7feb757fa640 1 -- 172.21.15.89:0/2414397113 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7feb6809c4f0 con 0x7feb78100e20 2024-09-15T23:40:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.670+0000 7feb7e6c6640 1 -- 172.21.15.89:0/2414397113 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb7810a0a0 con 0x7feb78100e20 2024-09-15T23:40:30.671 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.671+0000 7feb757fa640 1 -- 172.21.15.89:0/2414397113 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7feb6809bdf0 con 0x7feb78100e20 2024-09-15T23:40:30.672 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:40:30.672 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:40:30.672 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:40:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 -- 172.21.15.89:0/2414397113 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb600a32c0 msgr2=0x7feb600a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb600a32c0 0x7feb600a5780 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7feb78100020 tx=0x7feb6c005a20 comp rx=0 tx=0).stop 2024-09-15T23:40:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 -- 172.21.15.89:0/2414397113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb78100e20 msgr2=0x7feb78198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb78100e20 0x7feb78198590 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7feb6800d770 tx=0x7feb6800dc30 comp rx=0 tx=0).stop 2024-09-15T23:40:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 -- 172.21.15.89:0/2414397113 shutdown_connections 2024-09-15T23:40:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7feb600a32c0 0x7feb600a5780 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb78102ef0 0x7feb7819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:30.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb78102570 0x7feb78198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 --2- 172.21.15.89:0/2414397113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb78100e20 0x7feb78198590 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 -- 172.21.15.89:0/2414397113 >> 172.21.15.89:0/2414397113 conn(0x7feb780fc7d0 msgr2=0x7feb7806a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 -- 172.21.15.89:0/2414397113 shutdown_connections 2024-09-15T23:40:30.675 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:30.673+0000 7feb7e6c6640 1 -- 172.21.15.89:0/2414397113 wait complete. 2024-09-15T23:40:31.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:30 smithi089 ceph-mon[30933]: pgmap v844: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T23:40:31.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:30 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2414397113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:31.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:30 smithi120 ceph-mon[35250]: pgmap v844: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T23:40:31.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:30 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2414397113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:31.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:30 smithi174 ceph-mon[35586]: pgmap v844: 129 pgs: 129 active+clean; 4.1 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-09-15T23:40:31.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:30 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2414397113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:31.447 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24452 quiesce db --cancel --all' 2024-09-15T23:40:31.447 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24452 quiesce db --cancel --all 2024-09-15T23:40:31.700 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:32.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 -- 172.21.15.89:0/2514601872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c102570 msgr2=0x7fa96c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 --2- 172.21.15.89:0/2514601872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c102570 0x7fa96c1029b0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009900 tx=0x7fa95c02a480 comp rx=0 tx=0).stop 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 -- 172.21.15.89:0/2514601872 shutdown_connections 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 --2- 172.21.15.89:0/2514601872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa96c102ef0 0x7fa96c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 --2- 172.21.15.89:0/2514601872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c102570 0x7fa96c1029b0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 --2- 172.21.15.89:0/2514601872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa96c100e20 0x7fa96c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 -- 172.21.15.89:0/2514601872 >> 172.21.15.89:0/2514601872 conn(0x7fa96c0fc7d0 msgr2=0x7fa96c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 -- 172.21.15.89:0/2514601872 shutdown_connections 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 -- 172.21.15.89:0/2514601872 wait complete. 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 Processor -- start 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.200+0000 7fa973495640 1 -- start start 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa973495640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa96c100e20 0x7fa96c197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa973495640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa96c102570 0x7fa96c197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa973495640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c102ef0 0x7fa96c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa973495640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c110fa0 con 0x7fa96c102ef0 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa973495640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c110e20 con 0x7fa96c102570 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa973495640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c111120 con 0x7fa96c100e20 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa971a32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c102ef0 0x7fa96c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa971a32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c102ef0 0x7fa96c197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45624/0 (socket says 172.21.15.89:45624) 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa971a32640 1 -- 172.21.15.89:0/324617434 learned_addr learned my addr 172.21.15.89:0/324617434 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa971231640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa96c100e20 0x7fa96c197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa970a30640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa96c102570 0x7fa96c197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa971a32640 1 -- 172.21.15.89:0/324617434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa96c100e20 msgr2=0x7fa96c197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa971a32640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa96c100e20 0x7fa96c197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa971a32640 1 -- 172.21.15.89:0/324617434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa96c102570 msgr2=0x7fa96c197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:32.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa971a32640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa96c102570 0x7fa96c197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa971a32640 1 -- 172.21.15.89:0/324617434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96c1a69b0 con 0x7fa96c102ef0 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa971a32640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c102ef0 0x7fa96c197bc0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fa96400c970 tx=0x7fa96400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa9627fc640 1 -- 172.21.15.89:0/324617434 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964007bb0 con 0x7fa96c102ef0 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa9627fc640 1 -- 172.21.15.89:0/324617434 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa964007d50 con 0x7fa96c102ef0 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.201+0000 7fa973495640 1 -- 172.21.15.89:0/324617434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96c1a6ca0 con 0x7fa96c102ef0 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.202+0000 7fa9627fc640 1 -- 172.21.15.89:0/324617434 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964018800 con 0x7fa96c102ef0 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.202+0000 7fa973495640 1 -- 172.21.15.89:0/324617434 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa96c1a7210 con 0x7fa96c102ef0 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.204+0000 7fa9627fc640 1 -- 172.21.15.89:0/324617434 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fa96400fa50 con 0x7fa96c102ef0 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.204+0000 7fa97586e740 1 Processor -- start 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.204+0000 7fa97586e740 1 -- start start 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.204+0000 7fa97586e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 0x555659bb0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa9627fc640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9580a3240 0x7fa9580a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa97586e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 0x555659bbb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.212 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa97586e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bbdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa97586e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555659b15200 con 0x555659bbbbe0 2024-09-15T23:40:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa97586e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555659abbdc0 con 0x555659bb0a00 2024-09-15T23:40:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa97586e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555659b111c0 con 0x555659bb1320 2024-09-15T23:40:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971a32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bbdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971231640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9580a3240 0x7fa9580a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971a32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bbdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45640/0 (socket says 172.21.15.89:45640) 2024-09-15T23:40:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971a32640 1 -- 172.21.15.89:0/1299808123 learned_addr learned my addr 172.21.15.89:0/1299808123 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa970a30640 1 --2- 172.21.15.89:0/1299808123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 0x555659bbb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa9627fc640 1 -- 172.21.15.89:0/324617434 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa9640cfe40 con 0x7fa96c102ef0 2024-09-15T23:40:32.213 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa970a30640 1 --2- 172.21.15.89:0/1299808123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 0x555659bb0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971a32640 1 -- 172.21.15.89:0/1299808123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 msgr2=0x555659bbb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971a32640 1 --2- 172.21.15.89:0/1299808123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 0x555659bbb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971a32640 1 -- 172.21.15.89:0/1299808123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 msgr2=0x555659bb0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971a32640 1 --2- 172.21.15.89:0/1299808123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 0x555659bb0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971a32640 1 -- 172.21.15.89:0/1299808123 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555659886620 con 0x555659bbbbe0 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa970a30640 1 --2- 172.21.15.89:0/1299808123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 0x555659bb0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971231640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9580a3240 0x7fa9580a5700 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa968009f20 tx=0x7fa9680094f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa971a32640 1 --2- 172.21.15.89:0/1299808123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bbdfd0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fa9640cd460 tx=0x7fa96409bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f40545ff3c6ec0d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa9577fe640 1 -- 172.21.15.89:0/1299808123 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96409c710 con 0x555659bbbbe0 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa9577fe640 1 -- 172.21.15.89:0/1299808123 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa96409c010 con 0x555659bbbbe0 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.205+0000 7fa9577fe640 1 -- 172.21.15.89:0/1299808123 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964098680 con 0x555659bbbbe0 2024-09-15T23:40:32.214 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.206+0000 7fa97586e740 1 -- 172.21.15.89:0/1299808123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 msgr2=0x555659bbdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.206+0000 7fa97586e740 1 --2- 172.21.15.89:0/1299808123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bbdfd0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fa9640cd460 tx=0x7fa96409bde0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.206+0000 7fa97586e740 1 -- 172.21.15.89:0/1299808123 shutdown_connections 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.206+0000 7fa97586e740 1 --2- 172.21.15.89:0/1299808123 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bbdfd0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.206+0000 7fa97586e740 1 --2- 172.21.15.89:0/1299808123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 0x555659bbb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.206+0000 7fa97586e740 1 --2- 172.21.15.89:0/1299808123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 0x555659bb0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.206+0000 7fa97586e740 1 -- 172.21.15.89:0/1299808123 >> 172.21.15.89:0/1299808123 conn(0x555659bacf90 msgr2=0x555659bad370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.206+0000 7fa97586e740 1 -- 172.21.15.89:0/1299808123 shutdown_connections 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.206+0000 7fa97586e740 1 -- 172.21.15.89:0/1299808123 wait complete. 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa97586e740 1 -- start start 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa97586e740 1 Processor -- start 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa97586e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 0x555659bbb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa97586e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 0x555659bb8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.215 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa97586e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bb8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa97586e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x555659bbe7b0 con 0x555659bbbbe0 2024-09-15T23:40:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa97586e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555659aafc50 con 0x555659bb0a00 2024-09-15T23:40:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa97586e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555659bbe930 con 0x555659bb1320 2024-09-15T23:40:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa97586e740 10 client.0 _ll_register_callbacks cb 0x555659ac1ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa971a32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bb8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa971a32640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bb8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45642/0 (socket says 172.21.15.89:45642) 2024-09-15T23:40:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa971231640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 0x555659bbb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa971a32640 1 -- 172.21.15.89:0/1606891561 learned_addr learned my addr 172.21.15.89:0/1606891561 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.207+0000 7fa970a30640 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 0x555659bb8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.216 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa971a32640 1 -- 172.21.15.89:0/1606891561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 msgr2=0x555659bb8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa971a32640 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 0x555659bb8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa971a32640 1 -- 172.21.15.89:0/1606891561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 msgr2=0x555659bbb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa971a32640 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 0x555659bbb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa971a32640 1 -- 172.21.15.89:0/1606891561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555659bb94d0 con 0x555659bbbbe0 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa970a30640 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 0x555659bb8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa971a32640 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bb8ca0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fa964004770 tx=0x7fa9640a04b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1ed47351a13578 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa954ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa954ff9640 1 -- 172.21.15.89:0/1606891561 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa964096b80 con 0x555659bbbbe0 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa954ff9640 1 -- 172.21.15.89:0/1606891561 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa964096d20 con 0x555659bbbbe0 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa97586e740 1 -- 172.21.15.89:0/1606891561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa96409c010 con 0x555659bbbbe0 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa954ff9640 1 -- 172.21.15.89:0/1606891561 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa9640b7b60 con 0x555659bbbbe0 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa97586e740 10 client.21210 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa97586e740 1 -- 172.21.15.89:0/1606891561 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555659bca230 con 0x555659bbbbe0 2024-09-15T23:40:32.217 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa954ff9640 1 -- 172.21.15.89:0/1606891561 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa96409c010 con 0x555659bbbbe0 2024-09-15T23:40:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.208+0000 7fa954ff9640 1 -- 172.21.15.89:0/1606891561 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa964020030 con 0x555659bbbbe0 2024-09-15T23:40:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.209+0000 7fa97586e740 10 client.21210 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.209+0000 7fa97586e740 10 client.21210 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:40:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.209+0000 7fa97586e740 20 client.21210 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.209+0000 7fa97586e740 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555659bcb580 0x555659bcd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.209+0000 7fa97586e740 4 client.21210 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:32.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.209+0000 7fa97586e740 1 -- 172.21.15.89:0/1606891561 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555659b22770 con 0x555659bcb580 2024-09-15T23:40:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.209+0000 7fa971231640 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555659bcb580 0x555659bcd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.209+0000 7fa971231640 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555659bcb580 0x555659bcd970 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f68efe8c71c0369 server_cookie=35fd65efc66a020e in_seq=0 out_seq=0 2024-09-15T23:40:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.209+0000 7fa954ff9640 10 client.21210 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:32.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa954ff9640 1 -- 172.21.15.89:0/1606891561 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555659b22770 con 0x555659bcb580 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa954ff9640 10 client.21210 handle_command_reply: tid=0 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 client.21210 shutdown 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 -- 172.21.15.89:0/1606891561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 msgr2=0x555659bb8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bb8ca0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fa964004770 tx=0x7fa9640a04b0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 -- 172.21.15.89:0/1606891561 shutdown_connections 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555659bcb580 0x555659bcd970 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bbbbe0 0x555659bb8ca0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bb1320 0x555659bb8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 --2- 172.21.15.89:0/1606891561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb0a00 0x555659bbb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 -- 172.21.15.89:0/1606891561 >> 172.21.15.89:0/1606891561 conn(0x555659bacf90 msgr2=0x555659bac1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa954ff9640 0 client.21210 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 -- 172.21.15.89:0/1606891561 shutdown_connections 2024-09-15T23:40:32.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.210+0000 7fa97586e740 1 -- 172.21.15.89:0/1606891561 wait complete. 2024-09-15T23:40:32.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.211+0000 7fa97586e740 20 client.21210 trim_cache size 0 max 16384 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa97586e740 1 Processor -- start 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa97586e740 1 -- start start 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa97586e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bb3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa97586e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb1320 0x555659bc7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa97586e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bbbbe0 0x555659bb7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa97586e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1b1090 con 0x555659bb0a00 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa97586e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1b2bd0 con 0x555659bb1320 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa97586e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1b1760 con 0x555659bbbbe0 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa971231640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bb3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa971231640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bb3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45644/0 (socket says 172.21.15.89:45644) 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa971231640 1 -- 172.21.15.89:0/318370801 learned_addr learned my addr 172.21.15.89:0/318370801 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:32.229 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa971231640 1 -- 172.21.15.89:0/318370801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bbbbe0 msgr2=0x555659bb7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa971231640 1 --2- 172.21.15.89:0/318370801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bbbbe0 0x555659bb7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa971231640 1 -- 172.21.15.89:0/318370801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb1320 msgr2=0x555659bc7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa971231640 1 --2- 172.21.15.89:0/318370801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb1320 0x555659bc7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.227+0000 7fa971231640 1 -- 172.21.15.89:0/318370801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555659bacdc0 con 0x555659bb0a00 2024-09-15T23:40:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.228+0000 7fa971231640 1 --2- 172.21.15.89:0/318370801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bb3cb0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fa968009810 tx=0x7fa968011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd53a2d4869d09 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.228+0000 7fa956ffd640 1 -- 172.21.15.89:0/318370801 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa968033070 con 0x555659bb0a00 2024-09-15T23:40:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.228+0000 7fa956ffd640 1 -- 172.21.15.89:0/318370801 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa968025a60 con 0x555659bb0a00 2024-09-15T23:40:32.230 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.228+0000 7fa956ffd640 1 -- 172.21.15.89:0/318370801 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96802eb40 con 0x555659bb0a00 2024-09-15T23:40:32.233 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.228+0000 7fa97586e740 1 -- 172.21.15.89:0/318370801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 msgr2=0x555659bb3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.228+0000 7fa97586e740 1 --2- 172.21.15.89:0/318370801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bb3cb0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7fa968009810 tx=0x7fa968011000 comp rx=0 tx=0).stop 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.228+0000 7fa97586e740 1 -- 172.21.15.89:0/318370801 shutdown_connections 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.228+0000 7fa97586e740 1 --2- 172.21.15.89:0/318370801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bbbbe0 0x555659bb7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 --2- 172.21.15.89:0/318370801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb1320 0x555659bc7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 --2- 172.21.15.89:0/318370801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bb3cb0 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 -- 172.21.15.89:0/318370801 >> 172.21.15.89:0/318370801 conn(0x555659bacf90 msgr2=0x555659bba7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 -- 172.21.15.89:0/318370801 shutdown_connections 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 -- 172.21.15.89:0/318370801 wait complete. 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 -- start start 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 Processor -- start 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bd0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb1320 0x555659bd0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.234 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bbbbe0 0x555659bcdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1b2500 con 0x555659bb0a00 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1b1e30 con 0x555659bb1320 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c1af070 con 0x555659bbbbe0 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa97586e740 10 client.0 _ll_register_callbacks cb 0x7fa9640076c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.229+0000 7fa971231640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bd0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa971231640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bd0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:45658/0 (socket says 172.21.15.89:45658) 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa971231640 1 -- 172.21.15.89:0/3007078072 learned_addr learned my addr 172.21.15.89:0/3007078072 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa971a32640 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bbbbe0 0x555659bcdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa970a30640 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb1320 0x555659bd0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa971231640 1 -- 172.21.15.89:0/3007078072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bbbbe0 msgr2=0x555659bcdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa971231640 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bbbbe0 0x555659bcdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa971231640 1 -- 172.21.15.89:0/3007078072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb1320 msgr2=0x555659bd0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa971231640 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb1320 0x555659bd0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.235 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa971231640 1 -- 172.21.15.89:0/3007078072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555659bce360 con 0x555659bb0a00 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa970a30640 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb1320 0x555659bd0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa971231640 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bd0690 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fa96802e7a0 tx=0x7fa9680175c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486dec062f8e9ca0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa954ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa954ff9640 1 -- 172.21.15.89:0/3007078072 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96803e060 con 0x555659bb0a00 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa954ff9640 1 -- 172.21.15.89:0/3007078072 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fa968025580 con 0x555659bb0a00 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa97586e740 1 -- 172.21.15.89:0/3007078072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa968025a60 con 0x555659bb0a00 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa954ff9640 1 -- 172.21.15.89:0/3007078072 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa968045050 con 0x555659bb0a00 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa97586e740 10 client.21216 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.230+0000 7fa97586e740 1 -- 172.21.15.89:0/3007078072 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555659c35900 con 0x555659bb0a00 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.231+0000 7fa954ff9640 1 -- 172.21.15.89:0/3007078072 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa968025a60 con 0x555659bb0a00 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.231+0000 7fa954ff9640 1 -- 172.21.15.89:0/3007078072 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa968039770 con 0x555659bb0a00 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.231+0000 7fa97586e740 10 client.21216 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:32.236 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.231+0000 7fa97586e740 10 client.21216 resolve_mds: validated gid 24452 aka daemon mds.b 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.231+0000 7fa97586e740 20 client.21216 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.231+0000 7fa97586e740 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555659c35f20 0x555659c38340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.231+0000 7fa97586e740 4 client.21216 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.231+0000 7fa97586e740 1 -- 172.21.15.89:0/3007078072 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x555659b1cc80 con 0x555659c35f20 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.231+0000 7fa970a30640 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555659c35f20 0x555659c38340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa970a30640 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555659c35f20 0x555659c38340 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64de494ae7c588b server_cookie=2c5bdaec5394feaa in_seq=0 out_seq=0 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa954ff9640 10 client.21216 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa954ff9640 1 -- 172.21.15.89:0/3007078072 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x555659b1cc80 con 0x555659c35f20 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa954ff9640 10 client.21216 handle_command_reply: tid=0 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 client.21216 shutdown 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 -- 172.21.15.89:0/3007078072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 msgr2=0x555659bd0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bd0690 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fa96802e7a0 tx=0x7fa9680175c0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 -- 172.21.15.89:0/3007078072 shutdown_connections 2024-09-15T23:40:32.237 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x555659c35f20 0x555659c38340 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555659bbbbe0 0x555659bcdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555659bb1320 0x555659bd0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 --2- 172.21.15.89:0/3007078072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555659bb0a00 0x555659bd0690 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 -- 172.21.15.89:0/3007078072 >> 172.21.15.89:0/3007078072 conn(0x555659bacf90 msgr2=0x555659bb2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa954ff9640 0 client.21216 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 -- 172.21.15.89:0/3007078072 shutdown_connections 2024-09-15T23:40:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 1 -- 172.21.15.89:0/3007078072 wait complete. 2024-09-15T23:40:32.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.232+0000 7fa97586e740 20 client.21216 trim_cache size 0 max 16384 2024-09-15T23:40:32.239 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:40:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 29, 2024-09-15T23:40:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "leader": 24452, 2024-09-15T23:40:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "set_version": 60, 2024-09-15T23:40:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "sets": {} 2024-09-15T23:40:32.239 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 -- 172.21.15.89:0/324617434 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9580a3240 msgr2=0x7fa9580a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9580a3240 0x7fa9580a5700 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa968009f20 tx=0x7fa9680094f0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 -- 172.21.15.89:0/324617434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c102ef0 msgr2=0x7fa96c197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c102ef0 0x7fa96c197bc0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fa96400c970 tx=0x7fa96400ce30 comp rx=0 tx=0).stop 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 -- 172.21.15.89:0/324617434 shutdown_connections 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fa9580a3240 0x7fa9580a5700 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa96c102ef0 0x7fa96c197bc0 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa96c102570 0x7fa96c197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 --2- 172.21.15.89:0/324617434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa96c100e20 0x7fa96c197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 -- 172.21.15.89:0/324617434 >> 172.21.15.89:0/324617434 conn(0x7fa96c0fc7d0 msgr2=0x7fa96c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:32.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 -- 172.21.15.89:0/324617434 shutdown_connections 2024-09-15T23:40:32.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:32.233+0000 7fa973495640 1 -- 172.21.15.89:0/324617434 wait complete. 2024-09-15T23:40:32.972 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:32 smithi089 ceph-mon[30933]: pgmap v845: 129 pgs: 129 active+clean; 4.1 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:40:33.001 INFO:tasks.quiescer:done joining Quiescers 2024-09-15T23:40:33.001 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-09-15T23:40:33.011 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi089.front.sepia.ceph.com 2024-09-15T23:40:33.020 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-09-15T23:40:33.047 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-09-15T23:40:33.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:32 smithi120 ceph-mon[35250]: pgmap v845: 129 pgs: 129 active+clean; 4.1 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:40:33.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:32 smithi174 ceph-mon[35586]: pgmap v845: 129 pgs: 129 active+clean; 4.1 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-09-15T23:40:35.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:34 smithi089 ceph-mon[30933]: pgmap v846: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 18 op/s 2024-09-15T23:40:35.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:34 smithi120 ceph-mon[35250]: pgmap v846: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 18 op/s 2024-09-15T23:40:35.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:34 smithi174 ceph-mon[35586]: pgmap v846: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 18 op/s 2024-09-15T23:40:37.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:36 smithi089 ceph-mon[30933]: pgmap v847: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:40:37.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:36 smithi120 ceph-mon[35250]: pgmap v847: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:40:37.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:36 smithi174 ceph-mon[35586]: pgmap v847: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 15 op/s 2024-09-15T23:40:39.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:38 smithi089 ceph-mon[30933]: pgmap v848: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 18 op/s 2024-09-15T23:40:39.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:38 smithi120 ceph-mon[35250]: pgmap v848: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 18 op/s 2024-09-15T23:40:39.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:38 smithi174 ceph-mon[35586]: pgmap v848: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 18 op/s 2024-09-15T23:40:41.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:40 smithi089 ceph-mon[30933]: pgmap v849: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-09-15T23:40:41.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:40 smithi120 ceph-mon[35250]: pgmap v849: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-09-15T23:40:41.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:40 smithi174 ceph-mon[35586]: pgmap v849: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 14 op/s 2024-09-15T23:40:43.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:42 smithi089 ceph-mon[30933]: pgmap v850: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 15 op/s 2024-09-15T23:40:43.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:42 smithi120 ceph-mon[35250]: pgmap v850: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 15 op/s 2024-09-15T23:40:43.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:42 smithi174 ceph-mon[35586]: pgmap v850: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 15 op/s 2024-09-15T23:40:43.854 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:40:44.118 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:44.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.619+0000 7ff5289a2640 1 -- 172.21.15.89:0/3908330702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff524102ef0 msgr2=0x7ff52410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:44.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.619+0000 7ff5289a2640 1 --2- 172.21.15.89:0/3908330702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff524102ef0 0x7ff52410da40 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009ca0 tx=0x7ff51c02a950 comp rx=0 tx=0).stop 2024-09-15T23:40:44.678 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.620+0000 7ff5289a2640 1 -- 172.21.15.89:0/3908330702 shutdown_connections 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.620+0000 7ff5289a2640 1 --2- 172.21.15.89:0/3908330702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff524102ef0 0x7ff52410da40 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.620+0000 7ff5289a2640 1 --2- 172.21.15.89:0/3908330702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff524102570 0x7ff5241029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.620+0000 7ff5289a2640 1 --2- 172.21.15.89:0/3908330702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff524100e20 0x7ff524101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.620+0000 7ff5289a2640 1 -- 172.21.15.89:0/3908330702 >> 172.21.15.89:0/3908330702 conn(0x7ff5240fc7d0 msgr2=0x7ff5240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.620+0000 7ff5289a2640 1 -- 172.21.15.89:0/3908330702 shutdown_connections 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.620+0000 7ff5289a2640 1 -- 172.21.15.89:0/3908330702 wait complete. 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.620+0000 7ff5289a2640 1 Processor -- start 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.620+0000 7ff5289a2640 1 -- start start 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff5289a2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff524100e20 0x7ff524199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff5289a2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff524102570 0x7ff524199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff5289a2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff524102ef0 0x7ff5241a4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff5289a2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff524110f80 con 0x7ff524102ef0 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff5289a2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff524110e00 con 0x7ff524102570 2024-09-15T23:40:44.679 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff5289a2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff524111100 con 0x7ff524100e20 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff522d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff524102ef0 0x7ff5241a4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff522d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff524102ef0 0x7ff5241a4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38316/0 (socket says 172.21.15.89:38316) 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff522d9d640 1 -- 172.21.15.89:0/611190645 learned_addr learned my addr 172.21.15.89:0/611190645 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff52259c640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff524100e20 0x7ff524199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff51bfff640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff524102570 0x7ff524199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff522d9d640 1 -- 172.21.15.89:0/611190645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff524100e20 msgr2=0x7ff524199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff522d9d640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff524100e20 0x7ff524199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff522d9d640 1 -- 172.21.15.89:0/611190645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff524102570 msgr2=0x7ff524199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff522d9d640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff524102570 0x7ff524199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff522d9d640 1 -- 172.21.15.89:0/611190645 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5241a47d0 con 0x7ff524102ef0 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff51bfff640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff524102570 0x7ff524199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:44.680 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff522d9d640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff524102ef0 0x7ff5241a4290 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7ff51c004af0 tx=0x7ff51c02f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.621+0000 7ff51b7fe640 1 -- 172.21.15.89:0/611190645 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c034040 con 0x7ff524102ef0 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.622+0000 7ff5289a2640 1 -- 172.21.15.89:0/611190645 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5241a49a0 con 0x7ff524102ef0 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.622+0000 7ff51b7fe640 1 -- 172.21.15.89:0/611190645 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff51c030920 con 0x7ff524102ef0 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.622+0000 7ff51b7fe640 1 -- 172.21.15.89:0/611190645 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff51c004820 con 0x7ff524102ef0 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.622+0000 7ff5289a2640 1 -- 172.21.15.89:0/611190645 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5241a4e30 con 0x7ff524102ef0 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.624+0000 7ff52ad7b740 1 Processor -- start 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff51b7fe640 1 -- 172.21.15.89:0/611190645 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff51c037020 con 0x7ff524102ef0 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff52ad7b740 1 -- start start 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a8354a0 0x56044a835880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a840140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a842a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56044a79b2e0 con 0x56044a840680 2024-09-15T23:40:44.681 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56044a742ce0 con 0x56044a835dc0 2024-09-15T23:40:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56044a7972a0 con 0x56044a8354a0 2024-09-15T23:40:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff51bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a842a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff51bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a842a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38318/0 (socket says 172.21.15.89:38318) 2024-09-15T23:40:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff51bfff640 1 -- 172.21.15.89:0/54240892 learned_addr learned my addr 172.21.15.89:0/54240892 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff52259c640 1 --2- 172.21.15.89:0/54240892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a840140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.625+0000 7ff51bfff640 1 --2- 172.21.15.89:0/54240892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a8354a0 0x56044a835880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff51bfff640 1 -- 172.21.15.89:0/54240892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a8354a0 msgr2=0x56044a835880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:44.682 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff51bfff640 1 --2- 172.21.15.89:0/54240892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a8354a0 0x56044a835880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff51b7fe640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff5100a3240 0x7ff5100a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff51bfff640 1 -- 172.21.15.89:0/54240892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 msgr2=0x56044a840140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff51bfff640 1 --2- 172.21.15.89:0/54240892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a840140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff51bfff640 1 -- 172.21.15.89:0/54240892 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56044a727440 con 0x56044a840680 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff51bfff640 1 --2- 172.21.15.89:0/54240892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a842a70 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00bf60 tx=0x7ff50c00b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39cce4ab0e3e5904 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff522d9d640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff5100a3240 0x7ff5100a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff51b7fe640 1 -- 172.21.15.89:0/611190645 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff51c0f0c30 con 0x7ff524102ef0 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff518ff9640 1 -- 172.21.15.89:0/54240892 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c005220 con 0x56044a840680 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff518ff9640 1 -- 172.21.15.89:0/54240892 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff50c004c80 con 0x56044a840680 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.626+0000 7ff518ff9640 1 -- 172.21.15.89:0/54240892 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c0127e0 con 0x56044a840680 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 -- 172.21.15.89:0/54240892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 msgr2=0x56044a842a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/54240892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a842a70 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00bf60 tx=0x7ff50c00b580 comp rx=0 tx=0).stop 2024-09-15T23:40:44.683 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff522d9d640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff5100a3240 0x7ff5100a5700 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff51c02ae70 tx=0x7ff51c0b2340 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 -- 172.21.15.89:0/54240892 shutdown_connections 2024-09-15T23:40:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/54240892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a842a70 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/54240892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a840140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/54240892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a8354a0 0x56044a835880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 -- 172.21.15.89:0/54240892 >> 172.21.15.89:0/54240892 conn(0x56044a831a30 msgr2=0x56044a831e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 -- 172.21.15.89:0/54240892 shutdown_connections 2024-09-15T23:40:44.684 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 -- 172.21.15.89:0/54240892 wait complete. 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 -- start start 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.627+0000 7ff52ad7b740 1 Processor -- start 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a8354a0 0x56044a83ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a83d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a83d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x56044a843250 con 0x56044a840680 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56044a756050 con 0x56044a835dc0 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56044a8433d0 con 0x56044a8354a0 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff52ad7b740 10 client.0 _ll_register_callbacks cb 0x56044a796160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff51bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a83d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff51bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a83d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38326/0 (socket says 172.21.15.89:38326) 2024-09-15T23:40:44.685 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff51bfff640 1 -- 172.21.15.89:0/2500949496 learned_addr learned my addr 172.21.15.89:0/2500949496 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff52259c640 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a83d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff51bfff640 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a8354a0 0x56044a83ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff51bfff640 1 -- 172.21.15.89:0/2500949496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a8354a0 msgr2=0x56044a83ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff51bfff640 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a8354a0 0x56044a83ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff51bfff640 1 -- 172.21.15.89:0/2500949496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 msgr2=0x56044a83d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff51bfff640 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a83d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff51bfff640 1 -- 172.21.15.89:0/2500949496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56044a50c3c0 con 0x56044a840680 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff52259c640 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a83d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff51bfff640 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a83d740 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7ff50c014eb0 tx=0x7ff50c0056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7d08c6eda4c98f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff509ffb640 1 -- 172.21.15.89:0/2500949496 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c005000 con 0x56044a840680 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.628+0000 7ff509ffb640 1 -- 172.21.15.89:0/2500949496 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff50c0051a0 con 0x56044a840680 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff52ad7b740 1 -- 172.21.15.89:0/2500949496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff50c004c80 con 0x56044a840680 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff509ffb640 1 -- 172.21.15.89:0/2500949496 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff50c032050 con 0x56044a840680 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff52ad7b740 10 client.21228 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff52ad7b740 1 -- 172.21.15.89:0/2500949496 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56044a84ec30 con 0x56044a840680 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff509ffb640 1 -- 172.21.15.89:0/2500949496 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff50c004c80 con 0x56044a840680 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff509ffb640 1 -- 172.21.15.89:0/2500949496 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff50c012af0 con 0x56044a840680 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff52ad7b740 10 client.21228 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:44.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff52ad7b740 10 client.21228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:44.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff52ad7b740 20 client.21228 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:44.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56044a8500f0 0x56044a8524e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff52ad7b740 4 client.21228 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:44.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff52ad7b740 1 -- 172.21.15.89:0/2500949496 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56044a7a8850 con 0x56044a8500f0 2024-09-15T23:40:44.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.629+0000 7ff52259c640 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56044a8500f0 0x56044a8524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.630+0000 7ff509ffb640 10 client.21228 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:44.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.630+0000 7ff52259c640 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56044a8500f0 0x56044a8524e0 crc :-1 s=READY pgs=434 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c484a232ad5f76d server_cookie=c23f30b6fc3511a3 in_seq=0 out_seq=0 2024-09-15T23:40:44.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.630+0000 7ff509ffb640 1 -- 172.21.15.89:0/2500949496 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56044a7a8850 con 0x56044a8500f0 2024-09-15T23:40:44.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.630+0000 7ff509ffb640 10 client.21228 handle_command_reply: tid=0 2024-09-15T23:40:44.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.630+0000 7ff52ad7b740 1 client.21228 shutdown 2024-09-15T23:40:44.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 1 -- 172.21.15.89:0/2500949496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 msgr2=0x56044a83d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:44.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a83d740 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7ff50c014eb0 tx=0x7ff50c0056a0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 1 -- 172.21.15.89:0/2500949496 shutdown_connections 2024-09-15T23:40:44.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56044a8500f0 0x56044a8524e0 crc :-1 s=READY pgs=434 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a840680 0x56044a83d740 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a83d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff509ffb640 0 client.21228 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:44.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/2500949496 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a8354a0 0x56044a83ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 1 -- 172.21.15.89:0/2500949496 >> 172.21.15.89:0/2500949496 conn(0x56044a831a30 msgr2=0x56044a837710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 1 -- 172.21.15.89:0/2500949496 shutdown_connections 2024-09-15T23:40:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 1 -- 172.21.15.89:0/2500949496 wait complete. 2024-09-15T23:40:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.631+0000 7ff52ad7b740 20 client.21228 trim_cache size 0 max 16384 2024-09-15T23:40:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff52ad7b740 1 Processor -- start 2024-09-15T23:40:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff52ad7b740 1 -- start start 2024-09-15T23:40:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a831a30 0x56044a84c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a8354a0 0x56044a83d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a835dc0 0x56044a83c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff5241aeca0 con 0x56044a831a30 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff5241b07e0 con 0x56044a8354a0 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff5241af370 con 0x56044a835dc0 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff51bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a831a30 0x56044a84c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff51bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a831a30 0x56044a84c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38342/0 (socket says 172.21.15.89:38342) 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff51bfff640 1 -- 172.21.15.89:0/300887067 learned_addr learned my addr 172.21.15.89:0/300887067 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff52259c640 1 --2- 172.21.15.89:0/300887067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a8354a0 0x56044a83d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff51bfff640 1 --2- 172.21.15.89:0/300887067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a835dc0 0x56044a83c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff51bfff640 1 -- 172.21.15.89:0/300887067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a835dc0 msgr2=0x56044a83c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff51bfff640 1 --2- 172.21.15.89:0/300887067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a835dc0 0x56044a83c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff51bfff640 1 -- 172.21.15.89:0/300887067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a8354a0 msgr2=0x56044a83d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff51bfff640 1 --2- 172.21.15.89:0/300887067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a8354a0 0x56044a83d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.648+0000 7ff51bfff640 1 -- 172.21.15.89:0/300887067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56044a831860 con 0x56044a831a30 2024-09-15T23:40:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.649+0000 7ff51bfff640 1 --2- 172.21.15.89:0/300887067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a831a30 0x56044a84c4e0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7ff50c005f70 tx=0x7ff50c005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff5fcbbbd71c261 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.649+0000 7ff50bfff640 1 -- 172.21.15.89:0/300887067 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c005710 con 0x56044a831a30 2024-09-15T23:40:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.649+0000 7ff50bfff640 1 -- 172.21.15.89:0/300887067 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff50c00b390 con 0x56044a831a30 2024-09-15T23:40:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.649+0000 7ff50bfff640 1 -- 172.21.15.89:0/300887067 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c01d7d0 con 0x56044a831a30 2024-09-15T23:40:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 -- 172.21.15.89:0/300887067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a831a30 msgr2=0x56044a84c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:44.692 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/300887067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a831a30 0x56044a84c4e0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7ff50c005f70 tx=0x7ff50c005ec0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 -- 172.21.15.89:0/300887067 shutdown_connections 2024-09-15T23:40:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/300887067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a835dc0 0x56044a83c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/300887067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a8354a0 0x56044a83d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/300887067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a831a30 0x56044a84c4e0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 -- 172.21.15.89:0/300887067 >> 172.21.15.89:0/300887067 conn(0x56044a83d8a0 msgr2=0x56044a83bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 -- 172.21.15.89:0/300887067 shutdown_connections 2024-09-15T23:40:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 -- 172.21.15.89:0/300887067 wait complete. 2024-09-15T23:40:44.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 -- start start 2024-09-15T23:40:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 Processor -- start 2024-09-15T23:40:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a831a30 0x56044a848c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a8354a0 0x56044a839e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a83a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff5241b0110 con 0x56044a8354a0 2024-09-15T23:40:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff5241afa40 con 0x56044a835dc0 2024-09-15T23:40:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff5241acc80 con 0x56044a831a30 2024-09-15T23:40:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52ad7b740 10 client.0 _ll_register_callbacks cb 0x7ff514009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a8354a0 0x56044a839e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a8354a0 0x56044a839e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38354/0 (socket says 172.21.15.89:38354) 2024-09-15T23:40:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.650+0000 7ff52259c640 1 -- 172.21.15.89:0/4294750280 learned_addr learned my addr 172.21.15.89:0/4294750280 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff51bfff640 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a831a30 0x56044a848c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff51bfff640 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a83a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff52259c640 1 -- 172.21.15.89:0/4294750280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a831a30 msgr2=0x56044a848c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff52259c640 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a831a30 0x56044a848c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff52259c640 1 -- 172.21.15.89:0/4294750280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 msgr2=0x56044a83a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:44.695 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff52259c640 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a83a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff52259c640 1 -- 172.21.15.89:0/4294750280 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56044a83aaf0 con 0x56044a8354a0 2024-09-15T23:40:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff51bfff640 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a83a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff52259c640 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a8354a0 0x56044a839e70 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x56044a83e840 tx=0x7ff5140045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5381b16183188e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff509ffb640 1 -- 172.21.15.89:0/4294750280 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff514027430 con 0x56044a8354a0 2024-09-15T23:40:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff509ffb640 1 -- 172.21.15.89:0/4294750280 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff514010430 con 0x56044a8354a0 2024-09-15T23:40:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff52ad7b740 1 -- 172.21.15.89:0/4294750280 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff50c00b390 con 0x56044a8354a0 2024-09-15T23:40:44.696 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff509ffb640 1 -- 172.21.15.89:0/4294750280 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff514010630 con 0x56044a8354a0 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff52ad7b740 10 client.21234 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff52ad7b740 1 -- 172.21.15.89:0/4294750280 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56044a8baa80 con 0x56044a8354a0 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.651+0000 7ff509ffb640 1 -- 172.21.15.89:0/4294750280 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff50c00b390 con 0x56044a8354a0 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.652+0000 7ff509ffb640 1 -- 172.21.15.89:0/4294750280 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff514027ba0 con 0x56044a8354a0 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.652+0000 7ff52ad7b740 10 client.21234 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.652+0000 7ff52ad7b740 10 client.21234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.652+0000 7ff52ad7b740 20 client.21234 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.652+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56044a840680 0x56044a8bd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.652+0000 7ff52ad7b740 4 client.21234 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.652+0000 7ff52ad7b740 1 -- 172.21.15.89:0/4294750280 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56044a7a2d60 con 0x56044a840680 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.652+0000 7ff51bfff640 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56044a840680 0x56044a8bd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:44.697 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.652+0000 7ff509ffb640 10 client.21234 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.652+0000 7ff51bfff640 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56044a840680 0x56044a8bd1b0 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fb523205b583842 server_cookie=578c08fdf715f3e in_seq=0 out_seq=0 2024-09-15T23:40:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff509ffb640 1 -- 172.21.15.89:0/4294750280 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56044a7a2d60 con 0x56044a840680 2024-09-15T23:40:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff509ffb640 10 client.21234 handle_command_reply: tid=0 2024-09-15T23:40:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 client.21234 shutdown 2024-09-15T23:40:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 -- 172.21.15.89:0/4294750280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a8354a0 msgr2=0x56044a839e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a8354a0 0x56044a839e70 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x56044a83e840 tx=0x7ff5140045c0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 -- 172.21.15.89:0/4294750280 shutdown_connections 2024-09-15T23:40:44.698 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x56044a840680 0x56044a8bd1b0 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.698 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "status": "no active scrubs running", 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stdout: "scrubs": {} 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56044a835dc0 0x56044a83a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56044a8354a0 0x56044a839e70 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 --2- 172.21.15.89:0/4294750280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56044a831a30 0x56044a848c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 -- 172.21.15.89:0/4294750280 >> 172.21.15.89:0/4294750280 conn(0x56044a83d8a0 msgr2=0x56044a83c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff509ffb640 0 client.21234 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 -- 172.21.15.89:0/4294750280 shutdown_connections 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 1 -- 172.21.15.89:0/4294750280 wait complete. 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff52ad7b740 20 client.21234 trim_cache size 0 max 16384 2024-09-15T23:40:44.699 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff5289a2640 1 -- 172.21.15.89:0/611190645 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff5100a3240 msgr2=0x7ff5100a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff5289a2640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff5100a3240 0x7ff5100a5700 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff51c02ae70 tx=0x7ff51c0b2340 comp rx=0 tx=0).stop 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff5289a2640 1 -- 172.21.15.89:0/611190645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff524102ef0 msgr2=0x7ff5241a4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.653+0000 7ff5289a2640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff524102ef0 0x7ff5241a4290 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7ff51c004af0 tx=0x7ff51c02f5b0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.654+0000 7ff5289a2640 1 -- 172.21.15.89:0/611190645 shutdown_connections 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.654+0000 7ff5289a2640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff5100a3240 0x7ff5100a5700 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.654+0000 7ff5289a2640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff524102ef0 0x7ff5241a4290 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.654+0000 7ff5289a2640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff524102570 0x7ff524199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.654+0000 7ff5289a2640 1 --2- 172.21.15.89:0/611190645 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff524100e20 0x7ff524199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.654+0000 7ff5289a2640 1 -- 172.21.15.89:0/611190645 >> 172.21.15.89:0/611190645 conn(0x7ff5240fc7d0 msgr2=0x7ff524106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.654+0000 7ff5289a2640 1 -- 172.21.15.89:0/611190645 shutdown_connections 2024-09-15T23:40:44.700 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:44.654+0000 7ff5289a2640 1 -- 172.21.15.89:0/611190645 wait complete. 2024-09-15T23:40:45.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:44 smithi089 ceph-mon[30933]: pgmap v851: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-09-15T23:40:45.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:44 smithi120 ceph-mon[35250]: pgmap v851: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-09-15T23:40:45.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:44 smithi174 ceph-mon[35586]: pgmap v851: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 KiB/s wr, 13 op/s 2024-09-15T23:40:45.479 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:40:45.479 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:40:45.734 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.226+0000 7f2c39643640 1 -- 172.21.15.89:0/2311964134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c34106a80 msgr2=0x7f2c34106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.226+0000 7f2c39643640 1 --2- 172.21.15.89:0/2311964134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c34106a80 0x7f2c34106e60 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f2c24009930 tx=0x7f2c2402a420 comp rx=0 tx=0).stop 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 -- 172.21.15.89:0/2311964134 shutdown_connections 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 --2- 172.21.15.89:0/2311964134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c34101420 0x7f2c3410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 --2- 172.21.15.89:0/2311964134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c34100a80 0x7f2c34100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 --2- 172.21.15.89:0/2311964134 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c34106a80 0x7f2c34106e60 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 -- 172.21.15.89:0/2311964134 >> 172.21.15.89:0/2311964134 conn(0x7f2c340fc7a0 msgr2=0x7f2c340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 -- 172.21.15.89:0/2311964134 shutdown_connections 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 -- 172.21.15.89:0/2311964134 wait complete. 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 Processor -- start 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 -- start start 2024-09-15T23:40:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c34100a80 0x7f2c34102650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c34101420 0x7f2c34102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.227+0000 7f2c39643640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c34106a80 0x7f2c341030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c39643640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f2c34110e80 con 0x7f2c34101420 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c39643640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2c34110d00 con 0x7f2c34106a80 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c39643640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2c34111000 con 0x7f2c34100a80 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c327fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c34101420 0x7f2c34102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c327fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c34101420 0x7f2c34102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46826/0 (socket says 172.21.15.89:46826) 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c327fc640 1 -- 172.21.15.89:0/2330435776 learned_addr learned my addr 172.21.15.89:0/2330435776 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c32ffd640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c34100a80 0x7f2c34102650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c327fc640 1 -- 172.21.15.89:0/2330435776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c34100a80 msgr2=0x7f2c34102650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c327fc640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c34100a80 0x7f2c34102650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c337fe640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c34106a80 0x7f2c341030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:46.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c327fc640 1 -- 172.21.15.89:0/2330435776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c34106a80 msgr2=0x7f2c341030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c327fc640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c34106a80 0x7f2c341030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c327fc640 1 -- 172.21.15.89:0/2330435776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c341a6950 con 0x7f2c34101420 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c327fc640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c34101420 0x7f2c34102b90 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800e9f0 tx=0x7f2c2800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c1bfff640 1 -- 172.21.15.89:0/2330435776 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2800cd80 con 0x7f2c34101420 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c39643640 1 -- 172.21.15.89:0/2330435776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c341a6c40 con 0x7f2c34101420 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c1bfff640 1 -- 172.21.15.89:0/2330435776 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c28005500 con 0x7f2c34101420 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c1bfff640 1 -- 172.21.15.89:0/2330435776 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c28010690 con 0x7f2c34101420 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.228+0000 7f2c39643640 1 -- 172.21.15.89:0/2330435776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c341a7130 con 0x7f2c34101420 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.229+0000 7f2c39643640 1 -- 172.21.15.89:0/2330435776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c34100ee0 con 0x7f2c34101420 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.230+0000 7f2c1bfff640 1 -- 172.21.15.89:0/2330435776 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f2c280046c0 con 0x7f2c34101420 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.231+0000 7f2c1bfff640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2c1c0a3240 0x7f2c1c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.231+0000 7f2c32ffd640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2c1c0a3240 0x7f2c1c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:46.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.231+0000 7f2c1bfff640 1 -- 172.21.15.89:0/2330435776 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f2c280d03c0 con 0x7f2c34101420 2024-09-15T23:40:46.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.232+0000 7f2c32ffd640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2c1c0a3240 0x7f2c1c0a5700 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2c24009900 tx=0x7f2c240058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:46.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.234+0000 7f2c1bfff640 1 -- 172.21.15.89:0/2330435776 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2c280541b0 con 0x7f2c34101420 2024-09-15T23:40:46.435 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.434+0000 7f2c39643640 1 -- 172.21.15.89:0/2330435776 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c34103b80 con 0x7f2c34101420 2024-09-15T23:40:46.436 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.435+0000 7f2c1bfff640 1 -- 172.21.15.89:0/2330435776 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f2c34103b80 con 0x7f2c34101420 2024-09-15T23:40:46.436 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:40:46.437 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:40:46.437 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:40:46.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 -- 172.21.15.89:0/2330435776 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2c1c0a3240 msgr2=0x7f2c1c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:46.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2c1c0a3240 0x7f2c1c0a5700 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2c24009900 tx=0x7f2c240058d0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 -- 172.21.15.89:0/2330435776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c34101420 msgr2=0x7f2c34102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:46.438 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c34101420 0x7f2c34102b90 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800e9f0 tx=0x7f2c2800edf0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 -- 172.21.15.89:0/2330435776 shutdown_connections 2024-09-15T23:40:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f2c1c0a3240 0x7f2c1c0a5700 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c34106a80 0x7f2c341030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c34101420 0x7f2c34102b90 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 --2- 172.21.15.89:0/2330435776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2c34100a80 0x7f2c34102650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 -- 172.21.15.89:0/2330435776 >> 172.21.15.89:0/2330435776 conn(0x7f2c340fc7a0 msgr2=0x7f2c341048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 -- 172.21.15.89:0/2330435776 shutdown_connections 2024-09-15T23:40:46.439 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:46.437+0000 7f2c39643640 1 -- 172.21.15.89:0/2330435776 wait complete. 2024-09-15T23:40:47.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:46 smithi089 ceph-mon[30933]: pgmap v852: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-09-15T23:40:47.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:46 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2330435776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:47.156 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-09-15T23:40:47.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:46 smithi120 ceph-mon[35250]: pgmap v852: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-09-15T23:40:47.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:46 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2330435776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:47.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:46 smithi174 ceph-mon[35586]: pgmap v852: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-09-15T23:40:47.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:46 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2330435776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:40:47.407 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:47.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.889+0000 7ff9463db640 1 -- 172.21.15.89:0/1551981172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff940102570 msgr2=0x7ff9401029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:47.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.889+0000 7ff9463db640 1 --2- 172.21.15.89:0/1551981172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff940102570 0x7ff9401029b0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7ff928009900 tx=0x7ff92802a480 comp rx=0 tx=0).stop 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.889+0000 7ff9463db640 1 -- 172.21.15.89:0/1551981172 shutdown_connections 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.889+0000 7ff9463db640 1 --2- 172.21.15.89:0/1551981172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff940102ef0 0x7ff94010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.889+0000 7ff9463db640 1 --2- 172.21.15.89:0/1551981172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff940102570 0x7ff9401029b0 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.889+0000 7ff9463db640 1 --2- 172.21.15.89:0/1551981172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff940100e20 0x7ff940101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.889+0000 7ff9463db640 1 -- 172.21.15.89:0/1551981172 >> 172.21.15.89:0/1551981172 conn(0x7ff9400fc7d0 msgr2=0x7ff9400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.889+0000 7ff9463db640 1 -- 172.21.15.89:0/1551981172 shutdown_connections 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.889+0000 7ff9463db640 1 -- 172.21.15.89:0/1551981172 wait complete. 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff9463db640 1 Processor -- start 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff9463db640 1 -- start start 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff9463db640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff940100e20 0x7ff940199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff9463db640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff940102570 0x7ff940199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff9463db640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff940102ef0 0x7ff9401a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff9463db640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff940110f80 con 0x7ff940102ef0 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff9463db640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff940110e00 con 0x7ff940100e20 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff9463db640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff940111100 con 0x7ff940102570 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff944978640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff940102ef0 0x7ff9401a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff944978640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff940102ef0 0x7ff9401a43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46846/0 (socket says 172.21.15.89:46846) 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff944978640 1 -- 172.21.15.89:0/377109837 learned_addr learned my addr 172.21.15.89:0/377109837 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff937fff640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff940102570 0x7ff940199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff93ffff640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff940100e20 0x7ff940199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff944978640 1 -- 172.21.15.89:0/377109837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff940102570 msgr2=0x7ff940199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff944978640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff940102570 0x7ff940199890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff944978640 1 -- 172.21.15.89:0/377109837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff940100e20 msgr2=0x7ff940199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff944978640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff940100e20 0x7ff940199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.910 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff944978640 1 -- 172.21.15.89:0/377109837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9401a4910 con 0x7ff940102ef0 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff93ffff640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff940100e20 0x7ff940199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.890+0000 7ff944978640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff940102ef0 0x7ff9401a43d0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7ff938009fd0 tx=0x7ff93800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.891+0000 7ff93dffb640 1 -- 172.21.15.89:0/377109837 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff938017070 con 0x7ff940102ef0 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.891+0000 7ff9463db640 1 -- 172.21.15.89:0/377109837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9401a4ae0 con 0x7ff940102ef0 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.891+0000 7ff93dffb640 1 -- 172.21.15.89:0/377109837 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff938003e50 con 0x7ff940102ef0 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.891+0000 7ff93dffb640 1 -- 172.21.15.89:0/377109837 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9380106c0 con 0x7ff940102ef0 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.891+0000 7ff9463db640 1 -- 172.21.15.89:0/377109837 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9401a4fa0 con 0x7ff940102ef0 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff93dffb640 1 -- 172.21.15.89:0/377109837 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff938003970 con 0x7ff940102ef0 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff9487b4740 1 Processor -- start 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff9487b4740 1 -- start start 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd3b40 0x561b39bd2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bdd370 0x561b39bdf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.911 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff9487b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561b39b382f0 con 0x561b39bd25d0 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff9487b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561b39adfcf0 con 0x561b39bd3b40 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff9487b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561b39b342b0 con 0x561b39bdd370 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff93ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff93ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46848/0 (socket says 172.21.15.89:46848) 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff93ffff640 1 -- 172.21.15.89:0/3663014104 learned_addr learned my addr 172.21.15.89:0/3663014104 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff937fff640 1 --2- 172.21.15.89:0/3663014104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd3b40 0x561b39bd2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.894+0000 7ff937fff640 1 --2- 172.21.15.89:0/3663014104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bdd370 0x561b39bdf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff93ffff640 1 -- 172.21.15.89:0/3663014104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bdd370 msgr2=0x561b39bdf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff93ffff640 1 --2- 172.21.15.89:0/3663014104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bdd370 0x561b39bdf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff93ffff640 1 -- 172.21.15.89:0/3663014104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd3b40 msgr2=0x561b39bd2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff93ffff640 1 --2- 172.21.15.89:0/3663014104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd3b40 0x561b39bd2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.912 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff93ffff640 1 -- 172.21.15.89:0/3663014104 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b39ac43a0 con 0x561b39bd25d0 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff937fff640 1 --2- 172.21.15.89:0/3663014104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd3b40 0x561b39bd2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff93dffb640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff92c0a3240 0x7ff92c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff93ffff640 1 --2- 172.21.15.89:0/3663014104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd2a10 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7ff93000bd40 tx=0x7ff93000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149ed78b3e3914e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff93dffb640 1 -- 172.21.15.89:0/377109837 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff9380cfa30 con 0x7ff940102ef0 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff937fff640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff92c0a3240 0x7ff92c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff9367fc640 1 -- 172.21.15.89:0/3663014104 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930015070 con 0x561b39bd25d0 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff9367fc640 1 -- 172.21.15.89:0/3663014104 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff93000be30 con 0x561b39bd25d0 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.895+0000 7ff9367fc640 1 -- 172.21.15.89:0/3663014104 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930011a00 con 0x561b39bd25d0 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.896+0000 7ff9487b4740 1 -- 172.21.15.89:0/3663014104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 msgr2=0x561b39bd2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.896+0000 7ff937fff640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff92c0a3240 0x7ff92c0a5700 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff928009230 tx=0x7ff928031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.896+0000 7ff9487b4740 1 --2- 172.21.15.89:0/3663014104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd2a10 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7ff93000bd40 tx=0x7ff93000f730 comp rx=0 tx=0).stop 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.896+0000 7ff9487b4740 1 -- 172.21.15.89:0/3663014104 shutdown_connections 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.896+0000 7ff9487b4740 1 --2- 172.21.15.89:0/3663014104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bdd370 0x561b39bdf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.913 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.896+0000 7ff9487b4740 1 --2- 172.21.15.89:0/3663014104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd2a10 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.896+0000 7ff9487b4740 1 --2- 172.21.15.89:0/3663014104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd3b40 0x561b39bd2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.896+0000 7ff9487b4740 1 -- 172.21.15.89:0/3663014104 >> 172.21.15.89:0/3663014104 conn(0x561b39b9b310 msgr2=0x561b39bcec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.896+0000 7ff9487b4740 1 -- 172.21.15.89:0/3663014104 shutdown_connections 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.896+0000 7ff9487b4740 1 -- 172.21.15.89:0/3663014104 wait complete. 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff9487b4740 1 -- start start 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff9487b4740 1 Processor -- start 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bdcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bd9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 0x561b39bda570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff9487b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x561b39be0070 con 0x561b39bd25d0 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff9487b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561b39af3060 con 0x561b39bdd370 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff9487b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x561b39be01f0 con 0x561b39bd3b40 2024-09-15T23:40:47.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff9487b4740 10 client.0 _ll_register_callbacks cb 0x561b398468c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff93ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bdcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff93ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bdcd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46854/0 (socket says 172.21.15.89:46854) 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff93ffff640 1 -- 172.21.15.89:0/2031975682 learned_addr learned my addr 172.21.15.89:0/2031975682 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.897+0000 7ff937fff640 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bd9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff944978640 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 0x561b39bda570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff93ffff640 1 -- 172.21.15.89:0/2031975682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 msgr2=0x561b39bd9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff93ffff640 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bd9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff93ffff640 1 -- 172.21.15.89:0/2031975682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 msgr2=0x561b39bda570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff93ffff640 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 0x561b39bda570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff93ffff640 1 -- 172.21.15.89:0/2031975682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b398a93c0 con 0x561b39bd25d0 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff944978640 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 0x561b39bda570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff93ffff640 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bdcd20 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7ff93000c010 tx=0x7ff93000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4318ce2316a8ca8a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:47.915 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff91f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff91f7fe640 1 -- 172.21.15.89:0/2031975682 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930028060 con 0x561b39bd25d0 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff91f7fe640 1 -- 172.21.15.89:0/2031975682 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff930005410 con 0x561b39bd25d0 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff9487b4740 1 -- 172.21.15.89:0/2031975682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff93000be30 con 0x561b39bd25d0 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff91f7fe640 1 -- 172.21.15.89:0/2031975682 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff93002f050 con 0x561b39bd25d0 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff9487b4740 10 client.21252 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff9487b4740 1 -- 172.21.15.89:0/2031975682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b39bebac0 con 0x561b39bd25d0 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.898+0000 7ff91f7fe640 1 -- 172.21.15.89:0/2031975682 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff93000be30 con 0x561b39bd25d0 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.899+0000 7ff91f7fe640 1 -- 172.21.15.89:0/2031975682 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff930004ae0 con 0x561b39bd25d0 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.899+0000 7ff9487b4740 10 client.21252 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.899+0000 7ff9487b4740 10 client.21252 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.899+0000 7ff9487b4740 20 client.21252 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.899+0000 7ff9487b4740 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x561b39becec0 0x561b39bef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.899+0000 7ff9487b4740 4 client.21252 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:47.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.899+0000 7ff9487b4740 1 -- 172.21.15.89:0/2031975682 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b39b45860 con 0x561b39becec0 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.899+0000 7ff937fff640 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x561b39becec0 0x561b39bef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.899+0000 7ff91f7fe640 10 client.21252 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.899+0000 7ff937fff640 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x561b39becec0 0x561b39bef2b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8f4d261be6cfb6a1 server_cookie=90c93c4354045612 in_seq=0 out_seq=0 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff91f7fe640 1 -- 172.21.15.89:0/2031975682 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561b39b45860 con 0x561b39becec0 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff91f7fe640 10 client.21252 handle_command_reply: tid=0 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff9487b4740 1 client.21252 shutdown 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff9487b4740 1 -- 172.21.15.89:0/2031975682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 msgr2=0x561b39bdcd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff9487b4740 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bdcd20 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7ff93000c010 tx=0x7ff93000bab0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff9487b4740 1 -- 172.21.15.89:0/2031975682 shutdown_connections 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff9487b4740 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x561b39becec0 0x561b39bef2b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff9487b4740 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 0x561b39bda570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff9487b4740 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bd9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.917 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff9487b4740 1 --2- 172.21.15.89:0/2031975682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bdcd20 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff91f7fe640 0 client.21252 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:40:47.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff9487b4740 1 -- 172.21.15.89:0/2031975682 >> 172.21.15.89:0/2031975682 conn(0x561b39b9b310 msgr2=0x561b39bd4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:47.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.900+0000 7ff9487b4740 1 -- 172.21.15.89:0/2031975682 shutdown_connections 2024-09-15T23:40:47.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.901+0000 7ff9487b4740 1 -- 172.21.15.89:0/2031975682 wait complete. 2024-09-15T23:40:47.918 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.901+0000 7ff9487b4740 20 client.21252 trim_cache size 0 max 16384 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff9487b4740 1 Processor -- start 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff9487b4740 1 -- start start 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bd5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 0x561b39bd7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff9487b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff9401aeee0 con 0x561b39bd25d0 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff9487b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff9401b0a20 con 0x561b39bdd370 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff9487b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff9401af5b0 con 0x561b39bd3b40 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff93ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff93ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46856/0 (socket says 172.21.15.89:46856) 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff93ffff640 1 -- 172.21.15.89:0/2816656462 learned_addr learned my addr 172.21.15.89:0/2816656462 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:47.923 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff944978640 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 0x561b39bd7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff937fff640 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bd5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff93ffff640 1 -- 172.21.15.89:0/2816656462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 msgr2=0x561b39bd5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff93ffff640 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bd5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff93ffff640 1 -- 172.21.15.89:0/2816656462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 msgr2=0x561b39bd7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff93ffff640 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 0x561b39bd7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff93ffff640 1 -- 172.21.15.89:0/2816656462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b39bcd310 con 0x561b39bd25d0 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff944978640 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 0x561b39bd7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.914+0000 7ff937fff640 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bd5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.915+0000 7ff93ffff640 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd5120 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7ff930000c00 tx=0x7ff93000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34818fd37a0d86cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.915+0000 7ff935ffb640 1 -- 172.21.15.89:0/2816656462 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930028070 con 0x561b39bd25d0 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.915+0000 7ff935ffb640 1 -- 172.21.15.89:0/2816656462 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff93000be30 con 0x561b39bd25d0 2024-09-15T23:40:47.924 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.915+0000 7ff935ffb640 1 -- 172.21.15.89:0/2816656462 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930015070 con 0x561b39bd25d0 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 -- 172.21.15.89:0/2816656462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 msgr2=0x561b39bd5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd5120 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7ff930000c00 tx=0x7ff93000b480 comp rx=0 tx=0).stop 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 -- 172.21.15.89:0/2816656462 shutdown_connections 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bdd370 0x561b39bd7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bd5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 --2- 172.21.15.89:0/2816656462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bd25d0 0x561b39bd5120 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 -- 172.21.15.89:0/2816656462 >> 172.21.15.89:0/2816656462 conn(0x561b39b9b310 msgr2=0x561b39bdc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 -- 172.21.15.89:0/2816656462 shutdown_connections 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 -- 172.21.15.89:0/2816656462 wait complete. 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 -- start start 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 Processor -- start 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd25d0 0x561b39bf01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bf0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.925 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bdd370 0x561b39bf0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff9401b0350 con 0x561b39bdd370 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff9401afc80 con 0x561b39bd25d0 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.916+0000 7ff9487b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff9401acec0 con 0x561b39bd3b40 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff9487b4740 10 client.0 _ll_register_callbacks cb 0x7ff93000bbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff944978640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bdd370 0x561b39bf0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff944978640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bdd370 0x561b39bf0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46860/0 (socket says 172.21.15.89:46860) 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff944978640 1 -- 172.21.15.89:0/4088288860 learned_addr learned my addr 172.21.15.89:0/4088288860 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff937fff640 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bf0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff93ffff640 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd25d0 0x561b39bf01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff944978640 1 -- 172.21.15.89:0/4088288860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 msgr2=0x561b39bf0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff944978640 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bf0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff944978640 1 -- 172.21.15.89:0/4088288860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd25d0 msgr2=0x561b39bf01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:47.926 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff944978640 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd25d0 0x561b39bf01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff944978640 1 -- 172.21.15.89:0/4088288860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b39c569a0 con 0x561b39bdd370 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff93ffff640 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd25d0 0x561b39bf01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff944978640 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bdd370 0x561b39bf0c60 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7ff9380980c0 tx=0x7ff93809b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950b7b8af58c3b9b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff91f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff91f7fe640 1 -- 172.21.15.89:0/4088288860 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff938020040 con 0x561b39bdd370 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff91f7fe640 1 -- 172.21.15.89:0/4088288860 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff9380982d0 con 0x561b39bdd370 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff9487b4740 1 -- 172.21.15.89:0/4088288860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff93000be30 con 0x561b39bdd370 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff91f7fe640 1 -- 172.21.15.89:0/4088288860 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff9380a0010 con 0x561b39bdd370 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff9487b4740 10 client.21258 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.917+0000 7ff9487b4740 1 -- 172.21.15.89:0/4088288860 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b39c57c30 con 0x561b39bdd370 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff91f7fe640 1 -- 172.21.15.89:0/4088288860 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff93000be30 con 0x561b39bdd370 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff91f7fe640 1 -- 172.21.15.89:0/4088288860 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff93809b0b0 con 0x561b39bdd370 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff9487b4740 10 client.21258 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:47.927 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff9487b4740 10 client.21258 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff9487b4740 20 client.21258 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff9487b4740 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x561b39c58300 0x561b39c5a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff9487b4740 4 client.21258 mds_command: new command op to 24424 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff9487b4740 1 -- 172.21.15.89:0/4088288860 --> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b39b3fd70 con 0x561b39c58300 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff93ffff640 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x561b39c58300 0x561b39c5a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff91f7fe640 10 client.21258 ms_handle_connect on v2:172.21.15.174:6834/733541815 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.918+0000 7ff93ffff640 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x561b39c58300 0x561b39c5a720 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7709b51f607f3bf4 server_cookie=133b7c9c7ff6701e in_seq=0 out_seq=0 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff91f7fe640 1 -- 172.21.15.89:0/4088288860 <== mds.4 v2:172.21.15.174:6834/733541815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b39b3fd70 con 0x561b39c58300 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff91f7fe640 10 client.21258 handle_command_reply: tid=0 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 client.21258 shutdown 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 -- 172.21.15.89:0/4088288860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bdd370 msgr2=0x561b39bf0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bdd370 0x561b39bf0c60 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7ff9380980c0 tx=0x7ff93809b5c0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.928 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 -- 172.21.15.89:0/4088288860 shutdown_connections 2024-09-15T23:40:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.174:6834/733541815,v1:172.21.15.174:6835/733541815] conn(0x561b39c58300 0x561b39c5a720 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b39bdd370 0x561b39bf0c60 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x561b39bd3b40 0x561b39bf0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 --2- 172.21.15.89:0/4088288860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561b39bd25d0 0x561b39bf01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 -- 172.21.15.89:0/4088288860 >> 172.21.15.89:0/4088288860 conn(0x561b39b9b310 msgr2=0x561b39be8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff91f7fe640 0 client.21258 ms_handle_reset on v2:172.21.15.174:6834/733541815 2024-09-15T23:40:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 -- 172.21.15.89:0/4088288860 shutdown_connections 2024-09-15T23:40:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 1 -- 172.21.15.89:0/4088288860 wait complete. 2024-09-15T23:40:47.929 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9487b4740 20 client.21258 trim_cache size 0 max 16384 2024-09-15T23:40:47.929 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:40:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9463db640 1 -- 172.21.15.89:0/377109837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff92c0a3240 msgr2=0x7ff92c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9463db640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff92c0a3240 0x7ff92c0a5700 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff928009230 tx=0x7ff928031400 comp rx=0 tx=0).stop 2024-09-15T23:40:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9463db640 1 -- 172.21.15.89:0/377109837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff940102ef0 msgr2=0x7ff9401a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:47.930 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.919+0000 7ff9463db640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff940102ef0 0x7ff9401a43d0 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7ff938009fd0 tx=0x7ff93800b260 comp rx=0 tx=0).stop 2024-09-15T23:40:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.920+0000 7ff9463db640 1 -- 172.21.15.89:0/377109837 shutdown_connections 2024-09-15T23:40:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.920+0000 7ff9463db640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff92c0a3240 0x7ff92c0a5700 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.920+0000 7ff9463db640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff940102ef0 0x7ff9401a43d0 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.920+0000 7ff9463db640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff940102570 0x7ff940199890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.920+0000 7ff9463db640 1 --2- 172.21.15.89:0/377109837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff940100e20 0x7ff940199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.920+0000 7ff9463db640 1 -- 172.21.15.89:0/377109837 >> 172.21.15.89:0/377109837 conn(0x7ff9400fc7d0 msgr2=0x7ff9400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.920+0000 7ff9463db640 1 -- 172.21.15.89:0/377109837 shutdown_connections 2024-09-15T23:40:47.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:47.920+0000 7ff9463db640 1 -- 172.21.15.89:0/377109837 wait complete. 2024-09-15T23:40:48.815 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-09-15T23:40:49.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:48 smithi089 ceph-mon[30933]: pgmap v853: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-15T23:40:49.064 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:49.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:48 smithi120 ceph-mon[35250]: pgmap v853: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-15T23:40:49.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:48 smithi174 ceph-mon[35586]: pgmap v853: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-09-15T23:40:49.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.531+0000 7fc3228fc640 1 -- 172.21.15.89:0/426156492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc31c106f10 msgr2=0x7fc31c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:49.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.531+0000 7fc3228fc640 1 --2- 172.21.15.89:0/426156492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc31c106f10 0x7fc31c10ea90 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fc314009ca0 tx=0x7fc31402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.531+0000 7fc3228fc640 1 -- 172.21.15.89:0/426156492 shutdown_connections 2024-09-15T23:40:49.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.531+0000 7fc3228fc640 1 --2- 172.21.15.89:0/426156492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc31c106f10 0x7fc31c10ea90 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.531+0000 7fc3228fc640 1 --2- 172.21.15.89:0/426156492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc31c102790 0x7fc31c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.531+0000 7fc3228fc640 1 --2- 172.21.15.89:0/426156492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc31c101de0 0x7fc31c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.531+0000 7fc3228fc640 1 -- 172.21.15.89:0/426156492 >> 172.21.15.89:0/426156492 conn(0x7fc31c0fd8c0 msgr2=0x7fc31c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:49.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.531+0000 7fc3228fc640 1 -- 172.21.15.89:0/426156492 shutdown_connections 2024-09-15T23:40:49.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.531+0000 7fc3228fc640 1 -- 172.21.15.89:0/426156492 wait complete. 2024-09-15T23:40:49.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc3228fc640 1 Processor -- start 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc3228fc640 1 -- start start 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc3228fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc31c101de0 0x7fc31c198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc3228fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc31c102790 0x7fc31c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc3228fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc31c106f10 0x7fc31c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc3228fc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c111f70 con 0x7fc31c101de0 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc3228fc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c111df0 con 0x7fc31c102790 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc3228fc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c1120f0 con 0x7fc31c106f10 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc31bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc31c101de0 0x7fc31c198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc31bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc31c101de0 0x7fc31c198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46880/0 (socket says 172.21.15.89:46880) 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc31bfff640 1 -- 172.21.15.89:0/330557538 learned_addr learned my addr 172.21.15.89:0/330557538 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc31b7fe640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc31c102790 0x7fc31c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.532+0000 7fc320e99640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc31c106f10 0x7fc31c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc31bfff640 1 -- 172.21.15.89:0/330557538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc31c106f10 msgr2=0x7fc31c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc31bfff640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc31c106f10 0x7fc31c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc31bfff640 1 -- 172.21.15.89:0/330557538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc31c102790 msgr2=0x7fc31c198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc31bfff640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc31c102790 0x7fc31c198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc31bfff640 1 -- 172.21.15.89:0/330557538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31c19d2e0 con 0x7fc31c101de0 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc320e99640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc31c106f10 0x7fc31c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc31b7fe640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc31c102790 0x7fc31c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc31bfff640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc31c101de0 0x7fc31c198330 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00c920 tx=0x7fc30c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc3197fa640 1 -- 172.21.15.89:0/330557538 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c007c70 con 0x7fc31c101de0 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc3197fa640 1 -- 172.21.15.89:0/330557538 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc30c007e10 con 0x7fc31c101de0 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc3228fc640 1 -- 172.21.15.89:0/330557538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31c19d5d0 con 0x7fc31c101de0 2024-09-15T23:40:49.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc3197fa640 1 -- 172.21.15.89:0/330557538 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c018890 con 0x7fc31c101de0 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.533+0000 7fc3228fc640 1 -- 172.21.15.89:0/330557538 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31c1a4eb0 con 0x7fc31c101de0 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.535+0000 7fc3197fa640 1 -- 172.21.15.89:0/330557538 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc30c018a70 con 0x7fc31c101de0 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc324cd5740 1 Processor -- start 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc324cd5740 1 -- start start 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc3197fa640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3040a3240 0x7fc3040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7c8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc320e99640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3040a3240 0x7fc3040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7d3370 0x55e9fe7d5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc3197fa640 1 -- 172.21.15.89:0/330557538 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc30c0cff60 con 0x7fc31c101de0 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc324cd5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e9fe72e2f0 con 0x55e9fe7c9b40 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc324cd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e9fe6d5cf0 con 0x55e9fe7d3370 2024-09-15T23:40:49.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc324cd5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e9fe72a2b0 con 0x55e9fe7c85d0 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc31b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc320e99640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7d3370 0x55e9fe7d5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc31b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.89:52936/0 (socket says 172.21.15.89:52936) 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc31b7fe640 1 -- 172.21.15.89:0/2194028986 learned_addr learned my addr 172.21.15.89:0/2194028986 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.536+0000 7fc31bfff640 1 --2- 172.21.15.89:0/2194028986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc320e99640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3040a3240 0x7fc3040a5700 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc314009430 tx=0x7fc314031ce0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc31b7fe640 1 -- 172.21.15.89:0/2194028986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7d3370 msgr2=0x55e9fe7d5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc31b7fe640 1 --2- 172.21.15.89:0/2194028986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7d3370 0x55e9fe7d5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc31b7fe640 1 -- 172.21.15.89:0/2194028986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 msgr2=0x55e9fe7c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc31b7fe640 1 --2- 172.21.15.89:0/2194028986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7c8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc31b7fe640 1 -- 172.21.15.89:0/2194028986 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9fe6ba3a0 con 0x55e9fe7c85d0 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc31bfff640 1 --2- 172.21.15.89:0/2194028986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc31b7fe640 1 --2- 172.21.15.89:0/2194028986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7c8a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc31000bcb0 tx=0x7fc31000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fef66a7bcbd0d91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:49.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc3027fc640 1 -- 172.21.15.89:0/2194028986 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310015070 con 0x55e9fe7c85d0 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc3027fc640 1 -- 172.21.15.89:0/2194028986 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31001a940 con 0x55e9fe7c85d0 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.537+0000 7fc3027fc640 1 -- 172.21.15.89:0/2194028986 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3100119a0 con 0x55e9fe7c85d0 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 -- 172.21.15.89:0/2194028986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 msgr2=0x55e9fe7c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 --2- 172.21.15.89:0/2194028986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7c8a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc31000bcb0 tx=0x7fc31000f730 comp rx=0 tx=0).stop 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 -- 172.21.15.89:0/2194028986 shutdown_connections 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 --2- 172.21.15.89:0/2194028986 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7d3370 0x55e9fe7d5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 --2- 172.21.15.89:0/2194028986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7c8a10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 --2- 172.21.15.89:0/2194028986 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 -- 172.21.15.89:0/2194028986 >> 172.21.15.89:0/2194028986 conn(0x55e9fe791310 msgr2=0x55e9fe7c4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 -- 172.21.15.89:0/2194028986 shutdown_connections 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 -- 172.21.15.89:0/2194028986 wait complete. 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 -- start start 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.538+0000 7fc324cd5740 1 Processor -- start 2024-09-15T23:40:49.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7d2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc324cd5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e9fe7d6070 con 0x55e9fe7d3370 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc324cd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e9fe6e9060 con 0x55e9fe7c9b40 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc324cd5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e9fe7d61f0 con 0x55e9fe7c85d0 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc324cd5740 10 client.0 _ll_register_callbacks cb 0x55e9fe43c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc320e99640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc320e99640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7d0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46890/0 (socket says 172.21.15.89:46890) 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc320e99640 1 -- 172.21.15.89:0/3446616945 learned_addr learned my addr 172.21.15.89:0/3446616945 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc31bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7d2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc31b7fe640 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc320e99640 1 -- 172.21.15.89:0/3446616945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 msgr2=0x55e9fe7d2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:49.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc320e99640 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7d2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc320e99640 1 -- 172.21.15.89:0/3446616945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7c9b40 msgr2=0x55e9fe7d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc320e99640 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc320e99640 1 -- 172.21.15.89:0/3446616945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9fe49f3c0 con 0x55e9fe7d3370 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc320e99640 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7d0610 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fc314003e40 tx=0x7fc314044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4780324dc4cd82d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc2fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/3446616945 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc314004640 con 0x55e9fe7d3370 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/3446616945 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc3140047e0 con 0x55e9fe7d3370 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.539+0000 7fc324cd5740 1 -- 172.21.15.89:0/3446616945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc31001a940 con 0x55e9fe7d3370 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/3446616945 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc31402f610 con 0x55e9fe7d3370 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc324cd5740 10 client.21270 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc324cd5740 1 -- 172.21.15.89:0/3446616945 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e9fe7e1b10 con 0x55e9fe7d3370 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/3446616945 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31001a940 con 0x55e9fe7d3370 2024-09-15T23:40:49.560 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/3446616945 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc314030ae0 con 0x55e9fe7d3370 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc324cd5740 10 client.21270 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc324cd5740 10 client.21270 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc324cd5740 20 client.21270 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc324cd5740 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e9fe7e2f20 0x55e9fe7e53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc324cd5740 4 client.21270 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc324cd5740 1 -- 172.21.15.89:0/3446616945 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9fe73b860 con 0x55e9fe7e2f20 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.540+0000 7fc31bfff640 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e9fe7e2f20 0x55e9fe7e53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.541+0000 7fc2fb7fe640 10 client.21270 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.541+0000 7fc31bfff640 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e9fe7e2f20 0x55e9fe7e53a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ea39cc4cc843f601 server_cookie=958ee4b45a58dea1 in_seq=0 out_seq=0 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.541+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/3446616945 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e9fe73b860 con 0x55e9fe7e2f20 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.541+0000 7fc2fb7fe640 10 client.21270 handle_command_reply: tid=0 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.541+0000 7fc324cd5740 1 client.21270 shutdown 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 1 -- 172.21.15.89:0/3446616945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 msgr2=0x55e9fe7d0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:49.561 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7d0610 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fc314003e40 tx=0x7fc314044000 comp rx=0 tx=0).stop 2024-09-15T23:40:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 1 -- 172.21.15.89:0/3446616945 shutdown_connections 2024-09-15T23:40:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e9fe7e2f20 0x55e9fe7e53a0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7d0610 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 1 --2- 172.21.15.89:0/3446616945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7d2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 1 -- 172.21.15.89:0/3446616945 >> 172.21.15.89:0/3446616945 conn(0x55e9fe791310 msgr2=0x55e9fe7ca960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc2fb7fe640 0 client.21270 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:40:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 1 -- 172.21.15.89:0/3446616945 shutdown_connections 2024-09-15T23:40:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 1 -- 172.21.15.89:0/3446616945 wait complete. 2024-09-15T23:40:49.562 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.542+0000 7fc324cd5740 20 client.21270 trim_cache size 0 max 16384 2024-09-15T23:40:49.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc324cd5740 1 Processor -- start 2024-09-15T23:40:49.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc324cd5740 1 -- start start 2024-09-15T23:40:49.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7df310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7d3370 0x55e9fe7cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc324cd5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c1aee30 con 0x55e9fe7c9b40 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc324cd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c1b0970 con 0x55e9fe7d3370 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc324cd5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c1af500 con 0x55e9fe7c85d0 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc31b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7df310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc31b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7df310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46904/0 (socket says 172.21.15.89:46904) 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc31b7fe640 1 -- 172.21.15.89:0/4075768431 learned_addr learned my addr 172.21.15.89:0/4075768431 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc31bfff640 1 --2- 172.21.15.89:0/4075768431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc31b7fe640 1 -- 172.21.15.89:0/4075768431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 msgr2=0x55e9fe7c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc31b7fe640 1 --2- 172.21.15.89:0/4075768431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc31b7fe640 1 -- 172.21.15.89:0/4075768431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7d3370 msgr2=0x55e9fe7cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc320e99640 1 --2- 172.21.15.89:0/4075768431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7d3370 0x55e9fe7cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc31b7fe640 1 --2- 172.21.15.89:0/4075768431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7d3370 0x55e9fe7cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.555+0000 7fc31b7fe640 1 -- 172.21.15.89:0/4075768431 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9fe7c5dd0 con 0x55e9fe7c9b40 2024-09-15T23:40:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc31b7fe640 1 --2- 172.21.15.89:0/4075768431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7df310 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fc31000bcb0 tx=0x7fc31000bc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5727a72e7c2b37e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc301ffb640 1 -- 172.21.15.89:0/4075768431 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310029070 con 0x55e9fe7c9b40 2024-09-15T23:40:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc301ffb640 1 -- 172.21.15.89:0/4075768431 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31000be60 con 0x55e9fe7c9b40 2024-09-15T23:40:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc301ffb640 1 -- 172.21.15.89:0/4075768431 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310015040 con 0x55e9fe7c9b40 2024-09-15T23:40:49.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc324cd5740 1 -- 172.21.15.89:0/4075768431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 msgr2=0x55e9fe7df310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc324cd5740 1 --2- 172.21.15.89:0/4075768431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7df310 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fc31000bcb0 tx=0x7fc31000bc10 comp rx=0 tx=0).stop 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc324cd5740 1 -- 172.21.15.89:0/4075768431 shutdown_connections 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc324cd5740 1 --2- 172.21.15.89:0/4075768431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7d3370 0x55e9fe7cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc324cd5740 1 --2- 172.21.15.89:0/4075768431 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7df310 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc324cd5740 1 --2- 172.21.15.89:0/4075768431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc324cd5740 1 -- 172.21.15.89:0/4075768431 >> 172.21.15.89:0/4075768431 conn(0x55e9fe791310 msgr2=0x55e9fe7c4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc324cd5740 1 -- 172.21.15.89:0/4075768431 shutdown_connections 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.556+0000 7fc324cd5740 1 -- 172.21.15.89:0/4075768431 wait complete. 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc324cd5740 1 -- start start 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc324cd5740 1 Processor -- start 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7dba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc324cd5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc324cd5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c1b02a0 con 0x55e9fe7d3370 2024-09-15T23:40:49.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc324cd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c1afbd0 con 0x55e9fe7c85d0 2024-09-15T23:40:49.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc324cd5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c1ace10 con 0x55e9fe7c9b40 2024-09-15T23:40:49.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc324cd5740 10 client.0 _ll_register_callbacks cb 0x7fc314003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:49.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc320e99640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.572 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc320e99640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46910/0 (socket says 172.21.15.89:46910) 2024-09-15T23:40:49.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc320e99640 1 -- 172.21.15.89:0/973363024 learned_addr learned my addr 172.21.15.89:0/973363024 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:49.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.557+0000 7fc31bfff640 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7dba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.573 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc31b7fe640 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc320e99640 1 -- 172.21.15.89:0/973363024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c9b40 msgr2=0x55e9fe7ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc320e99640 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc320e99640 1 -- 172.21.15.89:0/973363024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7c85d0 msgr2=0x55e9fe7dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc320e99640 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7dba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc320e99640 1 -- 172.21.15.89:0/973363024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9fe7cd7e0 con 0x55e9fe7d3370 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc320e99640 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7cd140 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fc314030e80 tx=0x7fc314048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7683e0523dead129 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc2fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/973363024 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc314065060 con 0x55e9fe7d3370 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/973363024 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc31402f7b0 con 0x55e9fe7d3370 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc324cd5740 1 -- 172.21.15.89:0/973363024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc31000be60 con 0x55e9fe7d3370 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/973363024 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc31406c050 con 0x55e9fe7d3370 2024-09-15T23:40:49.574 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc324cd5740 10 client.21276 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc324cd5740 1 -- 172.21.15.89:0/973363024 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e9fe84db20 con 0x55e9fe7d3370 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/973363024 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc31000be60 con 0x55e9fe7d3370 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.558+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/973363024 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc314060480 con 0x55e9fe7d3370 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc324cd5740 10 client.21276 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc324cd5740 10 client.21276 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc324cd5740 20 client.21276 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc324cd5740 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e9fe84e210 0x55e9fe850630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc324cd5740 4 client.21276 mds_command: new command op to 24428 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc324cd5740 1 -- 172.21.15.89:0/973363024 --> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e9fe735d70 con 0x55e9fe84e210 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc31bfff640 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e9fe84e210 0x55e9fe850630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc2fb7fe640 10 client.21276 ms_handle_connect on v2:172.21.15.120:6836/991556551 2024-09-15T23:40:49.575 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc31bfff640 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e9fe84e210 0x55e9fe850630 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aa5aa3cb2006f59c server_cookie=f85547bbec5fc135 in_seq=0 out_seq=0 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc2fb7fe640 1 -- 172.21.15.89:0/973363024 <== mds.3 v2:172.21.15.120:6836/991556551 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e9fe735d70 con 0x55e9fe84e210 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc2fb7fe640 10 client.21276 handle_command_reply: tid=0 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.559+0000 7fc324cd5740 1 client.21276 shutdown 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 1 -- 172.21.15.89:0/973363024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 msgr2=0x55e9fe7cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7cd140 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7fc314030e80 tx=0x7fc314048000 comp rx=0 tx=0).stop 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 1 -- 172.21.15.89:0/973363024 shutdown_connections 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.120:6836/991556551,v1:172.21.15.120:6837/991556551] conn(0x55e9fe84e210 0x55e9fe850630 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e9fe7d3370 0x55e9fe7cd140 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e9fe7c9b40 0x55e9fe7ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 1 --2- 172.21.15.89:0/973363024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9fe7c85d0 0x55e9fe7dba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.576 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc2fb7fe640 0 client.21276 ms_handle_reset on v2:172.21.15.120:6836/991556551 2024-09-15T23:40:49.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 1 -- 172.21.15.89:0/973363024 >> 172.21.15.89:0/973363024 conn(0x55e9fe791310 msgr2=0x55e9fe7c34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:49.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 1 -- 172.21.15.89:0/973363024 shutdown_connections 2024-09-15T23:40:49.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 1 -- 172.21.15.89:0/973363024 wait complete. 2024-09-15T23:40:49.577 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc324cd5740 20 client.21276 trim_cache size 0 max 16384 2024-09-15T23:40:49.577 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:40:49.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 -- 172.21.15.89:0/330557538 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3040a3240 msgr2=0x7fc3040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:49.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3040a3240 0x7fc3040a5700 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc314009430 tx=0x7fc314031ce0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 -- 172.21.15.89:0/330557538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc31c101de0 msgr2=0x7fc31c198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:49.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc31c101de0 0x7fc31c198330 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00c920 tx=0x7fc30c00cde0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 -- 172.21.15.89:0/330557538 shutdown_connections 2024-09-15T23:40:49.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc3040a3240 0x7fc3040a5700 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc31c106f10 0x7fc31c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc31c102790 0x7fc31c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.578 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 --2- 172.21.15.89:0/330557538 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc31c101de0 0x7fc31c198330 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:49.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 -- 172.21.15.89:0/330557538 >> 172.21.15.89:0/330557538 conn(0x7fc31c0fd8c0 msgr2=0x7fc31c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:49.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 -- 172.21.15.89:0/330557538 shutdown_connections 2024-09-15T23:40:49.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:49.560+0000 7fc3228fc640 1 -- 172.21.15.89:0/330557538 wait complete. 2024-09-15T23:40:50.365 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:40:50.613 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:50.997 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:50 smithi089 ceph-mon[30933]: pgmap v854: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1023 B/s wr, 9 op/s 2024-09-15T23:40:51.097 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.075+0000 7f6b435dc640 1 -- 172.21.15.89:0/2182986541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c102ef0 msgr2=0x7f6b3c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.075+0000 7f6b435dc640 1 --2- 172.21.15.89:0/2182986541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c102ef0 0x7f6b3c10da40 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009900 tx=0x7f6b3402a480 comp rx=0 tx=0).stop 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.075+0000 7f6b435dc640 1 -- 172.21.15.89:0/2182986541 shutdown_connections 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.075+0000 7f6b435dc640 1 --2- 172.21.15.89:0/2182986541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c102ef0 0x7f6b3c10da40 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.075+0000 7f6b435dc640 1 --2- 172.21.15.89:0/2182986541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c102570 0x7f6b3c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.075+0000 7f6b435dc640 1 --2- 172.21.15.89:0/2182986541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b3c100e20 0x7f6b3c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.075+0000 7f6b435dc640 1 -- 172.21.15.89:0/2182986541 >> 172.21.15.89:0/2182986541 conn(0x7f6b3c0fc7d0 msgr2=0x7f6b3c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.075+0000 7f6b435dc640 1 -- 172.21.15.89:0/2182986541 shutdown_connections 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.075+0000 7f6b435dc640 1 -- 172.21.15.89:0/2182986541 wait complete. 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b435dc640 1 Processor -- start 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b435dc640 1 -- start start 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b435dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c100e20 0x7f6b3c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b435dc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c102570 0x7f6b3c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.098 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b435dc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b3c102ef0 0x7f6b3c19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b435dc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1120a0 con 0x7f6b3c102570 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b435dc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c111f20 con 0x7f6b3c102ef0 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b435dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c112220 con 0x7f6b3c100e20 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b40b77640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c102570 0x7f6b3c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b40b77640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c102570 0x7f6b3c198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46940/0 (socket says 172.21.15.89:46940) 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b40b77640 1 -- 172.21.15.89:0/2991251773 learned_addr learned my addr 172.21.15.89:0/2991251773 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b41b79640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b3c102ef0 0x7f6b3c19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b41378640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c100e20 0x7f6b3c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b40b77640 1 -- 172.21.15.89:0/2991251773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c100e20 msgr2=0x7f6b3c198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b40b77640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c100e20 0x7f6b3c198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b40b77640 1 -- 172.21.15.89:0/2991251773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b3c102ef0 msgr2=0x7f6b3c19cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b40b77640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b3c102ef0 0x7f6b3c19cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.099 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b40b77640 1 -- 172.21.15.89:0/2991251773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3c19d3e0 con 0x7f6b3c102570 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.076+0000 7f6b41378640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c100e20 0x7f6b3c198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.077+0000 7f6b40b77640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c102570 0x7f6b3c198970 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000c920 tx=0x7f6b3000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.077+0000 7f6b3a7fc640 1 -- 172.21.15.89:0/2991251773 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30007c70 con 0x7f6b3c102570 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.077+0000 7f6b3a7fc640 1 -- 172.21.15.89:0/2991251773 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b30007e10 con 0x7f6b3c102570 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.077+0000 7f6b435dc640 1 -- 172.21.15.89:0/2991251773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b3c19d6d0 con 0x7f6b3c102570 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.077+0000 7f6b3a7fc640 1 -- 172.21.15.89:0/2991251773 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30018890 con 0x7f6b3c102570 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.077+0000 7f6b435dc640 1 -- 172.21.15.89:0/2991251773 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b3c1a4fb0 con 0x7f6b3c102570 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.079+0000 7f6b3a7fc640 1 -- 172.21.15.89:0/2991251773 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f6b30018a70 con 0x7f6b3c102570 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.079+0000 7f6b459b5740 1 Processor -- start 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.079+0000 7f6b459b5740 1 -- start start 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e6468b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.100 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6468ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 0x564e6469cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b459b5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564e645f12f0 con 0x564e6468cb40 2024-09-15T23:40:51.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b459b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e64598cf0 con 0x564e6468b5d0 2024-09-15T23:40:51.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b459b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e645ed2b0 con 0x564e6469a780 2024-09-15T23:40:51.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b3a7fc640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6b240a3240 0x7f6b240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41b79640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e6468b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41b79640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e6468b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46942/0 (socket says 172.21.15.89:46942) 2024-09-15T23:40:51.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41b79640 1 -- 172.21.15.89:0/3576864311 learned_addr learned my addr 172.21.15.89:0/3576864311 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:51.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41378640 1 --2- 172.21.15.89:0/3576864311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6468ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.101 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41b79640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6b240a3240 0x7f6b240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b3a7fc640 1 -- 172.21.15.89:0/2991251773 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6b300cfe50 con 0x7f6b3c102570 2024-09-15T23:40:51.102 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b40b77640 1 --2- 172.21.15.89:0/3576864311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 0x564e6469cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41378640 1 -- 172.21.15.89:0/3576864311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 msgr2=0x564e6469cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:51.106 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41378640 1 --2- 172.21.15.89:0/3576864311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 0x564e6469cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41378640 1 -- 172.21.15.89:0/3576864311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 msgr2=0x564e6468b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41378640 1 --2- 172.21.15.89:0/3576864311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e6468b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41378640 1 -- 172.21.15.89:0/3576864311 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e6457d3a0 con 0x564e6468b5d0 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b40b77640 1 --2- 172.21.15.89:0/3576864311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 0x564e6469cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.080+0000 7f6b41b79640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6b240a3240 0x7f6b240a5700 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6b340096c0 tx=0x7f6b34009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b41378640 1 --2- 172.21.15.89:0/3576864311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6468ba10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009f60 tx=0x7f6b2c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75142d9b2b2e887 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b237fe640 1 -- 172.21.15.89:0/3576864311 <== mon.2 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c01a070 con 0x564e6468b5d0 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b237fe640 1 -- 172.21.15.89:0/3576864311 <== mon.2 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b2c0158f0 con 0x564e6468b5d0 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b237fe640 1 -- 172.21.15.89:0/3576864311 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c00a9f0 con 0x564e6468b5d0 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b459b5740 1 -- 172.21.15.89:0/3576864311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 msgr2=0x564e6468ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b459b5740 1 --2- 172.21.15.89:0/3576864311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6468ba10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009f60 tx=0x7f6b2c009f90 comp rx=0 tx=0).stop 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b459b5740 1 -- 172.21.15.89:0/3576864311 shutdown_connections 2024-09-15T23:40:51.107 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b459b5740 1 --2- 172.21.15.89:0/3576864311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 0x564e6469cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b459b5740 1 --2- 172.21.15.89:0/3576864311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6468ba10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b459b5740 1 --2- 172.21.15.89:0/3576864311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e6468b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.081+0000 7f6b459b5740 1 -- 172.21.15.89:0/3576864311 >> 172.21.15.89:0/3576864311 conn(0x564e64654310 msgr2=0x564e64687c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 1 -- 172.21.15.89:0/3576864311 shutdown_connections 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 1 -- 172.21.15.89:0/3576864311 wait complete. 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 1 -- start start 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 1 Processor -- start 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6469a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 0x564e64697400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e646979f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x564e6469d480 con 0x564e6469a780 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e645ac060 con 0x564e6468b5d0 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564e6469d600 con 0x564e6468cb40 2024-09-15T23:40:51.108 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b459b5740 10 client.0 _ll_register_callbacks cb 0x564e642ff8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b41b79640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e646979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.082+0000 7f6b41b79640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e646979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46948/0 (socket says 172.21.15.89:46948) 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b41b79640 1 -- 172.21.15.89:0/2085959046 learned_addr learned my addr 172.21.15.89:0/2085959046 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b41378640 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6469a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b40b77640 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 0x564e64697400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b41b79640 1 -- 172.21.15.89:0/2085959046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 msgr2=0x564e64697400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b41b79640 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 0x564e64697400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b41b79640 1 -- 172.21.15.89:0/2085959046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 msgr2=0x564e6469a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b41b79640 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6469a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b41b79640 1 -- 172.21.15.89:0/2085959046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e643623c0 con 0x564e6469a780 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b40b77640 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 0x564e64697400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b41b79640 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e646979f0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009230 tx=0x7f6b34005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30524c9414fa93a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:51.109 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b20ff9640 1 -- 172.21.15.89:0/2085959046 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34050030 con 0x564e6469a780 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b20ff9640 1 -- 172.21.15.89:0/2085959046 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6b34046040 con 0x564e6469a780 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b459b5740 1 -- 172.21.15.89:0/2085959046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b2c0158f0 con 0x564e6469a780 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b20ff9640 1 -- 172.21.15.89:0/2085959046 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6b34035040 con 0x564e6469a780 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b459b5740 10 client.21288 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.083+0000 7f6b459b5740 1 -- 172.21.15.89:0/2085959046 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564e646a8ef0 con 0x564e6469a780 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b20ff9640 1 -- 172.21.15.89:0/2085959046 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b2c0158f0 con 0x564e6469a780 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b20ff9640 1 -- 172.21.15.89:0/2085959046 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6b3404c9e0 con 0x564e6469a780 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b459b5740 10 client.21288 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b459b5740 10 client.21288 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b459b5740 20 client.21288 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b459b5740 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e646aa300 0x564e646ac790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b459b5740 4 client.21288 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:51.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b459b5740 1 -- 172.21.15.89:0/2085959046 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e645fe860 con 0x564e646aa300 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b41378640 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e646aa300 0x564e646ac790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b20ff9640 10 client.21288 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.084+0000 7f6b41378640 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e646aa300 0x564e646ac790 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5c2994f9eb426366 server_cookie=dfef0d680efb183 in_seq=0 out_seq=0 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b20ff9640 1 -- 172.21.15.89:0/2085959046 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564e645fe860 con 0x564e646aa300 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b20ff9640 10 client.21288 handle_command_reply: tid=0 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b459b5740 1 client.21288 shutdown 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b459b5740 1 -- 172.21.15.89:0/2085959046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 msgr2=0x564e646979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b459b5740 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e646979f0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009230 tx=0x7f6b34005e50 comp rx=0 tx=0).stop 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b459b5740 1 -- 172.21.15.89:0/2085959046 shutdown_connections 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b459b5740 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e646979f0 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b459b5740 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e646aa300 0x564e646ac790 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b459b5740 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 0x564e64697400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b459b5740 1 --2- 172.21.15.89:0/2085959046 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6469a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b459b5740 1 -- 172.21.15.89:0/2085959046 >> 172.21.15.89:0/2085959046 conn(0x564e64654310 msgr2=0x564e6468d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.085+0000 7f6b20ff9640 0 client.21288 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.086+0000 7f6b459b5740 1 -- 172.21.15.89:0/2085959046 shutdown_connections 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.086+0000 7f6b459b5740 1 -- 172.21.15.89:0/2085959046 wait complete. 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.086+0000 7f6b459b5740 20 client.21288 trim_cache size 0 max 16384 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.098+0000 7f6b459b5740 1 Processor -- start 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.098+0000 7f6b459b5740 1 -- start start 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e646a6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e64687700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 0x564e64687c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b459b5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1acc40 con 0x564e6468cb40 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b459b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1ae780 con 0x564e6468b5d0 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b459b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1ad310 con 0x564e6469a780 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b40b77640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e64687700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b40b77640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e64687700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46956/0 (socket says 172.21.15.89:46956) 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b40b77640 1 -- 172.21.15.89:0/3074991307 learned_addr learned my addr 172.21.15.89:0/3074991307 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b41b79640 1 --2- 172.21.15.89:0/3074991307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 0x564e64687c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b40b77640 1 -- 172.21.15.89:0/3074991307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 msgr2=0x564e64687c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b40b77640 1 --2- 172.21.15.89:0/3074991307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 0x564e64687c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b40b77640 1 -- 172.21.15.89:0/3074991307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 msgr2=0x564e646a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b40b77640 1 --2- 172.21.15.89:0/3074991307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e646a6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b40b77640 1 -- 172.21.15.89:0/3074991307 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e64688180 con 0x564e6468cb40 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b40b77640 1 --2- 172.21.15.89:0/3074991307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e64687700 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f6b3009c010 tx=0x7f6b300bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd43ccf5277e52eb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b22ffd640 1 -- 172.21.15.89:0/3074991307 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b300b4070 con 0x564e6468cb40 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b22ffd640 1 -- 172.21.15.89:0/3074991307 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b300a4cf0 con 0x564e6468cb40 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.099+0000 7f6b22ffd640 1 -- 172.21.15.89:0/3074991307 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b300a00f0 con 0x564e6468cb40 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 -- 172.21.15.89:0/3074991307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 msgr2=0x564e64687700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:51.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 --2- 172.21.15.89:0/3074991307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e64687700 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f6b3009c010 tx=0x7f6b300bc040 comp rx=0 tx=0).stop 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 -- 172.21.15.89:0/3074991307 shutdown_connections 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 --2- 172.21.15.89:0/3074991307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6469a780 0x564e64687c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 --2- 172.21.15.89:0/3074991307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6468cb40 0x564e64687700 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 --2- 172.21.15.89:0/3074991307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e646a6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 -- 172.21.15.89:0/3074991307 >> 172.21.15.89:0/3074991307 conn(0x564e64654310 msgr2=0x564e646973c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 -- 172.21.15.89:0/3074991307 shutdown_connections 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 -- 172.21.15.89:0/3074991307 wait complete. 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 -- start start 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.100+0000 7f6b459b5740 1 Processor -- start 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6469e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 0x564e6469ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b459b5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e64714080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b459b5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1ae0b0 con 0x564e6469a780 2024-09-15T23:40:51.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b459b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1ad9e0 con 0x564e6468b5d0 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b459b5740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c1aac20 con 0x564e6468cb40 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b459b5740 10 client.0 _ll_register_callbacks cb 0x7f6b34009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b41b79640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e64714080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b41b79640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e64714080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46964/0 (socket says 172.21.15.89:46964) 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b41b79640 1 -- 172.21.15.89:0/511281940 learned_addr learned my addr 172.21.15.89:0/511281940 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b40b77640 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 0x564e6469ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b41378640 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6469e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b41b79640 1 -- 172.21.15.89:0/511281940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 msgr2=0x564e6469ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b41b79640 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 0x564e6469ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b41b79640 1 -- 172.21.15.89:0/511281940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 msgr2=0x564e6469e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b41b79640 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6469e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b41b79640 1 -- 172.21.15.89:0/511281940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e647145c0 con 0x564e6469a780 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b41b79640 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e64714080 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009850 tx=0x7f6b34031a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66ffb46a9f711ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:51.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.101+0000 7f6b20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b20ff9640 1 -- 172.21.15.89:0/511281940 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34050030 con 0x564e6469a780 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b20ff9640 1 -- 172.21.15.89:0/511281940 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f6b34046030 con 0x564e6469a780 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b459b5740 1 -- 172.21.15.89:0/511281940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b300a4cf0 con 0x564e6469a780 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b20ff9640 1 -- 172.21.15.89:0/511281940 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6b3406a050 con 0x564e6469a780 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b459b5740 10 client.21294 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b459b5740 1 -- 172.21.15.89:0/511281940 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564e64715760 con 0x564e6469a780 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b20ff9640 1 -- 172.21.15.89:0/511281940 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b300a4cf0 con 0x564e6469a780 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b20ff9640 1 -- 172.21.15.89:0/511281940 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f6b34060030 con 0x564e6469a780 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b459b5740 10 client.21294 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b459b5740 10 client.21294 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b459b5740 20 client.21294 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b459b5740 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e64715e50 0x564e64718270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b459b5740 4 client.21294 mds_command: new command op to 24430 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:40:51.116 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.102+0000 7f6b459b5740 1 -- 172.21.15.89:0/511281940 --> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e645f8d70 con 0x564e64715e50 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b41378640 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e64715e50 0x564e64718270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b20ff9640 10 client.21294 ms_handle_connect on v2:172.21.15.174:6836/3626951914 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b41378640 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e64715e50 0x564e64718270 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1520093ffffec3d2 server_cookie=163bddc88fcc0dea in_seq=0 out_seq=0 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b20ff9640 1 -- 172.21.15.89:0/511281940 <== mds.2 v2:172.21.15.174:6836/3626951914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e645f8d70 con 0x564e64715e50 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b20ff9640 10 client.21294 handle_command_reply: tid=0 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b459b5740 1 client.21294 shutdown 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b459b5740 1 -- 172.21.15.89:0/511281940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 msgr2=0x564e64714080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b459b5740 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e64714080 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009850 tx=0x7f6b34031a40 comp rx=0 tx=0).stop 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b459b5740 1 -- 172.21.15.89:0/511281940 shutdown_connections 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b459b5740 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e6469a780 0x564e64714080 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b459b5740 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.174:6836/3626951914,v1:172.21.15.174:6837/3626951914] conn(0x564e64715e50 0x564e64718270 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b459b5740 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564e6468cb40 0x564e6469ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b459b5740 1 --2- 172.21.15.89:0/511281940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e6468b5d0 0x564e6469e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.117 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b459b5740 1 -- 172.21.15.89:0/511281940 >> 172.21.15.89:0/511281940 conn(0x564e64654310 msgr2=0x564e646973c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:51.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.103+0000 7f6b20ff9640 0 client.21294 ms_handle_reset on v2:172.21.15.174:6836/3626951914 2024-09-15T23:40:51.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b459b5740 1 -- 172.21.15.89:0/511281940 shutdown_connections 2024-09-15T23:40:51.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b459b5740 1 -- 172.21.15.89:0/511281940 wait complete. 2024-09-15T23:40:51.118 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b459b5740 20 client.21294 trim_cache size 0 max 16384 2024-09-15T23:40:51.118 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:40:51.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 -- 172.21.15.89:0/2991251773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6b240a3240 msgr2=0x7f6b240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:51.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6b240a3240 0x7f6b240a5700 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6b340096c0 tx=0x7f6b34009260 comp rx=0 tx=0).stop 2024-09-15T23:40:51.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 -- 172.21.15.89:0/2991251773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c102570 msgr2=0x7f6b3c198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:51.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c102570 0x7f6b3c198970 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000c920 tx=0x7f6b3000cde0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 -- 172.21.15.89:0/2991251773 shutdown_connections 2024-09-15T23:40:51.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f6b240a3240 0x7f6b240a5700 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b3c102ef0 0x7f6b3c19cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b3c102570 0x7f6b3c198970 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 --2- 172.21.15.89:0/2991251773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b3c100e20 0x7f6b3c198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:51.119 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 -- 172.21.15.89:0/2991251773 >> 172.21.15.89:0/2991251773 conn(0x7f6b3c0fc7d0 msgr2=0x7f6b3c106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:51.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 -- 172.21.15.89:0/2991251773 shutdown_connections 2024-09-15T23:40:51.120 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:51.104+0000 7f6b435dc640 1 -- 172.21.15.89:0/2991251773 wait complete. 2024-09-15T23:40:51.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:50 smithi120 ceph-mon[35250]: pgmap v854: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1023 B/s wr, 9 op/s 2024-09-15T23:40:51.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:50 smithi174 ceph-mon[35586]: pgmap v854: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1023 B/s wr, 9 op/s 2024-09-15T23:40:51.832 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:40:52.078 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.527+0000 7f9987cde640 1 -- 172.21.15.89:0/4187606349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9980100a80 msgr2=0x7f9980100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 --2- 172.21.15.89:0/4187606349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9980100a80 0x7f9980100ee0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f997c009930 tx=0x7f997c02a420 comp rx=0 tx=0).stop 2024-09-15T23:40:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 -- 172.21.15.89:0/4187606349 shutdown_connections 2024-09-15T23:40:52.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 --2- 172.21.15.89:0/4187606349 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9980101420 0x7f998010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 --2- 172.21.15.89:0/4187606349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9980100a80 0x7f9980100ee0 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 --2- 172.21.15.89:0/4187606349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9980106a80 0x7f9980106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 -- 172.21.15.89:0/4187606349 >> 172.21.15.89:0/4187606349 conn(0x7f99800fc7a0 msgr2=0x7f99800febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 -- 172.21.15.89:0/4187606349 shutdown_connections 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 -- 172.21.15.89:0/4187606349 wait complete. 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 Processor -- start 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 -- start start 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9980100a80 0x7f99801992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.528+0000 7f9987cde640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9980101420 0x7f99801997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9987cde640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9980106a80 0x7f99801a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9987cde640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f9980110e80 con 0x7f9980100a80 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9987cde640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9980110d00 con 0x7f9980106a80 2024-09-15T23:40:52.550 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9987cde640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9980111000 con 0x7f9980101420 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985a7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9980100a80 0x7f99801992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985a7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9980100a80 0x7f99801992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46988/0 (socket says 172.21.15.89:46988) 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985a7a640 1 -- 172.21.15.89:0/1158976468 learned_addr learned my addr 172.21.15.89:0/1158976468 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985279640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9980101420 0x7f99801997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f998627b640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9980106a80 0x7f99801a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985a7a640 1 -- 172.21.15.89:0/1158976468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9980101420 msgr2=0x7f99801997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985a7a640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9980101420 0x7f99801997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985a7a640 1 -- 172.21.15.89:0/1158976468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9980106a80 msgr2=0x7f99801a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985a7a640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9980106a80 0x7f99801a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985a7a640 1 -- 172.21.15.89:0/1158976468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99801a4870 con 0x7f9980100a80 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985279640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9980101420 0x7f99801997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9985a7a640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9980100a80 0x7f99801992b0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f997400d950 tx=0x7f997400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9972ffd640 1 -- 172.21.15.89:0/1158976468 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974017070 con 0x7f9980100a80 2024-09-15T23:40:52.551 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9987cde640 1 -- 172.21.15.89:0/1158976468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99801a4a40 con 0x7f9980100a80 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9972ffd640 1 -- 172.21.15.89:0/1158976468 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9974009c30 con 0x7f9980100a80 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9972ffd640 1 -- 172.21.15.89:0/1158976468 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997400f6b0 con 0x7f9980100a80 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.529+0000 7f9987cde640 1 -- 172.21.15.89:0/1158976468 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99801a4f50 con 0x7f9980100a80 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.531+0000 7f9972ffd640 1 -- 172.21.15.89:0/1158976468 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f99740109d0 con 0x7f9980100a80 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f998a0b7740 1 Processor -- start 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f998a0b7740 1 -- start start 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e1e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e1ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e29370 0x55fff6e2b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f998a0b7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fff6d842f0 con 0x55fff6e1e5d0 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f998a0b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fff6d2bcf0 con 0x55fff6e29370 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f998a0b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fff6d802b0 con 0x55fff6e1fb40 2024-09-15T23:40:52.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f9985a7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e1ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f9985a7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e1ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46994/0 (socket says 172.21.15.89:46994) 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f9985a7a640 1 -- 172.21.15.89:0/2924841917 learned_addr learned my addr 172.21.15.89:0/2924841917 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.532+0000 7f9985279640 1 --2- 172.21.15.89:0/2924841917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e29370 0x55fff6e2b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f9972ffd640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f996c0a3240 0x7f996c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f998627b640 1 --2- 172.21.15.89:0/2924841917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e1e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f9985a7a640 1 -- 172.21.15.89:0/2924841917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 msgr2=0x55fff6e1e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f9985a7a640 1 --2- 172.21.15.89:0/2924841917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e1e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f9985a7a640 1 -- 172.21.15.89:0/2924841917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e29370 msgr2=0x55fff6e2b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f9985a7a640 1 --2- 172.21.15.89:0/2924841917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e29370 0x55fff6e2b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f9972ffd640 1 -- 172.21.15.89:0/1158976468 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f99740cfa10 con 0x7f9980100a80 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f9985a7a640 1 -- 172.21.15.89:0/2924841917 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fff6d103a0 con 0x55fff6e1e5d0 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f998627b640 1 --2- 172.21.15.89:0/2924841917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e1e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:52.553 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f998627b640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f996c0a3240 0x7f996c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f9985a7a640 1 --2- 172.21.15.89:0/2924841917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e1ea10 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f99740985b0 tx=0x7f997400bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47e75b5c2cc9ce2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f996bfff640 1 -- 172.21.15.89:0/2924841917 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974020030 con 0x55fff6e1e5d0 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f998627b640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f996c0a3240 0x7f996c0a5700 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9978005fd0 tx=0x7f9978004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f996bfff640 1 -- 172.21.15.89:0/2924841917 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f997409c2e0 con 0x55fff6e1e5d0 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.533+0000 7f996bfff640 1 -- 172.21.15.89:0/2924841917 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740cd1d0 con 0x55fff6e1e5d0 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.534+0000 7f998a0b7740 1 -- 172.21.15.89:0/2924841917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 msgr2=0x55fff6e1ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.534+0000 7f998a0b7740 1 --2- 172.21.15.89:0/2924841917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e1ea10 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f99740985b0 tx=0x7f997400bf60 comp rx=0 tx=0).stop 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.534+0000 7f998a0b7740 1 -- 172.21.15.89:0/2924841917 shutdown_connections 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.534+0000 7f998a0b7740 1 --2- 172.21.15.89:0/2924841917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e29370 0x55fff6e2b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.534+0000 7f998a0b7740 1 --2- 172.21.15.89:0/2924841917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e1ea10 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.534+0000 7f998a0b7740 1 --2- 172.21.15.89:0/2924841917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e1e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.534+0000 7f998a0b7740 1 -- 172.21.15.89:0/2924841917 >> 172.21.15.89:0/2924841917 conn(0x55fff6de7310 msgr2=0x55fff6e1ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:52.554 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.534+0000 7f998a0b7740 1 -- 172.21.15.89:0/2924841917 shutdown_connections 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.534+0000 7f998a0b7740 1 -- 172.21.15.89:0/2924841917 wait complete. 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.534+0000 7f998a0b7740 1 -- start start 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f998a0b7740 1 Processor -- start 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 0x55fff6e28d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1fb40 0x55fff6e25fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e29370 0x55fff6e26560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f998a0b7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55fff6e2bfb0 con 0x55fff6e1fb40 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f998a0b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fff6d3f060 con 0x55fff6e1e5d0 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f998a0b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55fff6e2c130 con 0x55fff6e29370 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f998a0b7740 10 client.0 _ll_register_callbacks cb 0x55fff6a928c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f9985279640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1fb40 0x55fff6e25fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f9985279640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1fb40 0x55fff6e25fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47002/0 (socket says 172.21.15.89:47002) 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f9985279640 1 -- 172.21.15.89:0/3763819625 learned_addr learned my addr 172.21.15.89:0/3763819625 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:52.555 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f9985a7a640 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 0x55fff6e28d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f998627b640 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e29370 0x55fff6e26560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f9985279640 1 -- 172.21.15.89:0/3763819625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e29370 msgr2=0x55fff6e26560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f9985279640 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e29370 0x55fff6e26560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f9985279640 1 -- 172.21.15.89:0/3763819625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 msgr2=0x55fff6e28d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f9985279640 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 0x55fff6e28d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.535+0000 7f9985279640 1 -- 172.21.15.89:0/3763819625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fff6af53c0 con 0x55fff6e1fb40 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f9985279640 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1fb40 0x55fff6e25fc0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f997c009930 tx=0x7f997c0318d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44662082085346b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f99697fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f99697fa640 1 -- 172.21.15.89:0/3763819625 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997c035040 con 0x55fff6e1fb40 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f99697fa640 1 -- 172.21.15.89:0/3763819625 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f997c004240 con 0x55fff6e1fb40 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f998a0b7740 1 -- 172.21.15.89:0/3763819625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f997409c2e0 con 0x55fff6e1fb40 2024-09-15T23:40:52.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f99697fa640 1 -- 172.21.15.89:0/3763819625 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f997c004440 con 0x55fff6e1fb40 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f998a0b7740 10 client.21306 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f998a0b7740 1 -- 172.21.15.89:0/3763819625 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fff6e37a00 con 0x55fff6e1fb40 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f99697fa640 1 -- 172.21.15.89:0/3763819625 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f997409c2e0 con 0x55fff6e1fb40 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f99697fa640 1 -- 172.21.15.89:0/3763819625 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f997c004bf0 con 0x55fff6e1fb40 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f998a0b7740 10 client.21306 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f998a0b7740 10 client.21306 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f998a0b7740 20 client.21306 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f998a0b7740 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55fff6e38f10 0x55fff6e3b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f998a0b7740 4 client.21306 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.536+0000 7f998a0b7740 1 -- 172.21.15.89:0/3763819625 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fff6d91860 con 0x55fff6e38f10 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.537+0000 7f9985a7a640 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55fff6e38f10 0x55fff6e3b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.537+0000 7f9985a7a640 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55fff6e38f10 0x55fff6e3b300 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=636968bfad13f412 server_cookie=63a57f717e0f99c4 in_seq=0 out_seq=0 2024-09-15T23:40:52.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.537+0000 7f99697fa640 10 client.21306 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f99697fa640 1 -- 172.21.15.89:0/3763819625 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fff6d91860 con 0x55fff6e38f10 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f99697fa640 10 client.21306 handle_command_reply: tid=0 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 client.21306 shutdown 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 -- 172.21.15.89:0/3763819625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1fb40 msgr2=0x55fff6e25fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1fb40 0x55fff6e25fc0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f997c009930 tx=0x7f997c0318d0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 -- 172.21.15.89:0/3763819625 shutdown_connections 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55fff6e38f10 0x55fff6e3b300 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e29370 0x55fff6e26560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1fb40 0x55fff6e25fc0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 --2- 172.21.15.89:0/3763819625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 0x55fff6e28d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f99697fa640 0 client.21306 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 -- 172.21.15.89:0/3763819625 >> 172.21.15.89:0/3763819625 conn(0x55fff6de7310 msgr2=0x55fff6e20960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 -- 172.21.15.89:0/3763819625 shutdown_connections 2024-09-15T23:40:52.558 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 1 -- 172.21.15.89:0/3763819625 wait complete. 2024-09-15T23:40:52.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.538+0000 7f998a0b7740 20 client.21306 trim_cache size 0 max 16384 2024-09-15T23:40:52.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.551+0000 7f998a0b7740 1 Processor -- start 2024-09-15T23:40:52.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.551+0000 7f998a0b7740 1 -- start start 2024-09-15T23:40:52.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.551+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 0x55fff6e35250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.551+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e25880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.551+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e29370 0x55fff6e25dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.551+0000 7f998a0b7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f99801aee80 con 0x55fff6e29370 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.551+0000 7f998a0b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f99801b09c0 con 0x55fff6e1e5d0 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.551+0000 7f998a0b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f99801af550 con 0x55fff6e1fb40 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.551+0000 7f998627b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e29370 0x55fff6e25dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f998627b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e29370 0x55fff6e25dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47012/0 (socket says 172.21.15.89:47012) 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f998627b640 1 -- 172.21.15.89:0/4028810153 learned_addr learned my addr 172.21.15.89:0/4028810153 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f9985279640 1 --2- 172.21.15.89:0/4028810153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e25880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f998627b640 1 -- 172.21.15.89:0/4028810153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 msgr2=0x55fff6e25880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f9985a7a640 1 --2- 172.21.15.89:0/4028810153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 0x55fff6e35250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f998627b640 1 --2- 172.21.15.89:0/4028810153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e25880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f998627b640 1 -- 172.21.15.89:0/4028810153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 msgr2=0x55fff6e35250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f998627b640 1 --2- 172.21.15.89:0/4028810153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 0x55fff6e35250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f998627b640 1 -- 172.21.15.89:0/4028810153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fff6e26300 con 0x55fff6e29370 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f9985a7a640 1 --2- 172.21.15.89:0/4028810153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 0x55fff6e35250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f998627b640 1 --2- 172.21.15.89:0/4028810153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e29370 0x55fff6e25dc0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f9978012840 tx=0x7f9978014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68aabf05a14df84 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f996b7fe640 1 -- 172.21.15.89:0/4028810153 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997801f9e0 con 0x55fff6e29370 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f996b7fe640 1 -- 172.21.15.89:0/4028810153 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9978016070 con 0x55fff6e29370 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.552+0000 7f996b7fe640 1 -- 172.21.15.89:0/4028810153 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99780286a0 con 0x55fff6e29370 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 -- 172.21.15.89:0/4028810153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e29370 msgr2=0x55fff6e25dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 --2- 172.21.15.89:0/4028810153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e29370 0x55fff6e25dc0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f9978012840 tx=0x7f9978014000 comp rx=0 tx=0).stop 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 -- 172.21.15.89:0/4028810153 shutdown_connections 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 --2- 172.21.15.89:0/4028810153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e29370 0x55fff6e25dc0 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 --2- 172.21.15.89:0/4028810153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e25880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 --2- 172.21.15.89:0/4028810153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e1e5d0 0x55fff6e35250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.565 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 -- 172.21.15.89:0/4028810153 >> 172.21.15.89:0/4028810153 conn(0x55fff6de7310 msgr2=0x55fff6e24890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 -- 172.21.15.89:0/4028810153 shutdown_connections 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 -- 172.21.15.89:0/4028810153 wait complete. 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 -- start start 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.553+0000 7f998a0b7740 1 Processor -- start 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e31990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e22b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f998a0b7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e29370 0x55fff6e230b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f998a0b7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f99801b02f0 con 0x55fff6e1e5d0 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f998a0b7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f99801afc20 con 0x55fff6e29370 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f998a0b7740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f99801ace60 con 0x55fff6e1fb40 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f998a0b7740 10 client.0 _ll_register_callbacks cb 0x7f997c005e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f9985a7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e31990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f9985a7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e31990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47022/0 (socket says 172.21.15.89:47022) 2024-09-15T23:40:52.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f9985a7a640 1 -- 172.21.15.89:0/2976640578 learned_addr learned my addr 172.21.15.89:0/2976640578 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f9985279640 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e22b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f998627b640 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e29370 0x55fff6e230b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f9985a7a640 1 -- 172.21.15.89:0/2976640578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 msgr2=0x55fff6e22b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f9985a7a640 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e22b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f9985a7a640 1 -- 172.21.15.89:0/2976640578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e29370 msgr2=0x55fff6e230b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f9985a7a640 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e29370 0x55fff6e230b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f9985a7a640 1 -- 172.21.15.89:0/2976640578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fff6e237a0 con 0x55fff6e1e5d0 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f9985a7a640 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e31990 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f99740a4320 tx=0x7f9974000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2817624790a61b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f99697fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f99697fa640 1 -- 172.21.15.89:0/2976640578 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974020040 con 0x55fff6e1e5d0 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f99697fa640 1 -- 172.21.15.89:0/2976640578 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f99740cd1d0 con 0x55fff6e1e5d0 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.554+0000 7f998a0b7740 1 -- 172.21.15.89:0/2976640578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9978016070 con 0x55fff6e1e5d0 2024-09-15T23:40:52.567 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f99697fa640 1 -- 172.21.15.89:0/2976640578 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f997409ff90 con 0x55fff6e1e5d0 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f998a0b7740 10 client.21312 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f998a0b7740 1 -- 172.21.15.89:0/2976640578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fff6ea3a50 con 0x55fff6e1e5d0 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f99697fa640 1 -- 172.21.15.89:0/2976640578 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9978016070 con 0x55fff6e1e5d0 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f99697fa640 1 -- 172.21.15.89:0/2976640578 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f99740860d0 con 0x55fff6e1e5d0 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f998a0b7740 10 client.21312 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f998a0b7740 10 client.21312 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f998a0b7740 20 client.21312 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f998a0b7740 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55fff6ea3e00 0x55fff6ea61f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f998a0b7740 4 client.21312 mds_command: new command op to 24436 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f998a0b7740 1 -- 172.21.15.89:0/2976640578 --> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fff6d8bd70 con 0x55fff6ea3e00 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.555+0000 7f9985279640 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55fff6ea3e00 0x55fff6ea61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f99697fa640 10 client.21312 ms_handle_connect on v2:172.21.15.174:6838/3380568300 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f9985279640 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55fff6ea3e00 0x55fff6ea61f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dee41cf9a8c7c363 server_cookie=902216129b28383c in_seq=0 out_seq=0 2024-09-15T23:40:52.568 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f99697fa640 1 -- 172.21.15.89:0/2976640578 <== mds.1 v2:172.21.15.174:6838/3380568300 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fff6d8bd70 con 0x55fff6ea3e00 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f99697fa640 10 client.21312 handle_command_reply: tid=0 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 client.21312 shutdown 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 -- 172.21.15.89:0/2976640578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 msgr2=0x55fff6e31990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e31990 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f99740a4320 tx=0x7f9974000bc0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 -- 172.21.15.89:0/2976640578 shutdown_connections 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.174:6838/3380568300,v1:172.21.15.174:6839/3380568300] conn(0x55fff6ea3e00 0x55fff6ea61f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fff6e29370 0x55fff6e230b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55fff6e1fb40 0x55fff6e22b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 --2- 172.21.15.89:0/2976640578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fff6e1e5d0 0x55fff6e31990 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f99697fa640 0 client.21312 ms_handle_reset on v2:172.21.15.174:6838/3380568300 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 -- 172.21.15.89:0/2976640578 >> 172.21.15.89:0/2976640578 conn(0x55fff6de7310 msgr2=0x55fff6e24820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 -- 172.21.15.89:0/2976640578 shutdown_connections 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 1 -- 172.21.15.89:0/2976640578 wait complete. 2024-09-15T23:40:52.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.556+0000 7f998a0b7740 20 client.21312 trim_cache size 0 max 16384 2024-09-15T23:40:52.570 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 -- 172.21.15.89:0/1158976468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f996c0a3240 msgr2=0x7f996c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f996c0a3240 0x7f996c0a5700 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9978005fd0 tx=0x7f9978004a40 comp rx=0 tx=0).stop 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 -- 172.21.15.89:0/1158976468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9980100a80 msgr2=0x7f99801992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9980100a80 0x7f99801992b0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f997400d950 tx=0x7f997400de10 comp rx=0 tx=0).stop 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 -- 172.21.15.89:0/1158976468 shutdown_connections 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f996c0a3240 0x7f996c0a5700 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9980106a80 0x7f99801a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9980101420 0x7f99801997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 --2- 172.21.15.89:0/1158976468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9980100a80 0x7f99801992b0 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 -- 172.21.15.89:0/1158976468 >> 172.21.15.89:0/1158976468 conn(0x7f99800fc7a0 msgr2=0x7f99800fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 -- 172.21.15.89:0/1158976468 shutdown_connections 2024-09-15T23:40:52.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:52.557+0000 7f9987cde640 1 -- 172.21.15.89:0/1158976468 wait complete. 2024-09-15T23:40:53.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:52 smithi089 ceph-mon[30933]: pgmap v855: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-15T23:40:53.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:52 smithi120 ceph-mon[35250]: pgmap v855: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-15T23:40:53.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:52 smithi174 ceph-mon[35586]: pgmap v855: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 938 B/s wr, 11 op/s 2024-09-15T23:40:53.274 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:40:53.520 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:54.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.005+0000 7efdfc163640 1 -- 172.21.15.89:0/3785193192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdf4106a80 msgr2=0x7efdf4106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.005+0000 7efdfc163640 1 --2- 172.21.15.89:0/3785193192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdf4106a80 0x7efdf4106e60 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009930 tx=0x7efdf002a420 comp rx=0 tx=0).stop 2024-09-15T23:40:54.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.005+0000 7efdfc163640 1 -- 172.21.15.89:0/3785193192 shutdown_connections 2024-09-15T23:40:54.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.005+0000 7efdfc163640 1 --2- 172.21.15.89:0/3785193192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdf4101420 0x7efdf410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.005+0000 7efdfc163640 1 --2- 172.21.15.89:0/3785193192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf4100a80 0x7efdf4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.005+0000 7efdfc163640 1 --2- 172.21.15.89:0/3785193192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdf4106a80 0x7efdf4106e60 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.005+0000 7efdfc163640 1 -- 172.21.15.89:0/3785193192 >> 172.21.15.89:0/3785193192 conn(0x7efdf40fc7a0 msgr2=0x7efdf40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdfc163640 1 -- 172.21.15.89:0/3785193192 shutdown_connections 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdfc163640 1 -- 172.21.15.89:0/3785193192 wait complete. 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdfc163640 1 Processor -- start 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdfc163640 1 -- start start 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdfc163640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdf4100a80 0x7efdf40fcda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdfc163640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdf4101420 0x7efdf40fd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdfc163640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf4106a80 0x7efdf40fd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdfc163640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efdf4110e80 con 0x7efdf4101420 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdfc163640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efdf4110d00 con 0x7efdf4100a80 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdfc163640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efdf4111000 con 0x7efdf4106a80 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.006+0000 7efdf96fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdf4101420 0x7efdf40fd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdf96fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdf4101420 0x7efdf40fd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47034/0 (socket says 172.21.15.89:47034) 2024-09-15T23:40:54.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdf96fe640 1 -- 172.21.15.89:0/262785455 learned_addr learned my addr 172.21.15.89:0/262785455 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdf9eff640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdf4100a80 0x7efdf40fcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdfa700640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf4106a80 0x7efdf40fd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdf96fe640 1 -- 172.21.15.89:0/262785455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf4106a80 msgr2=0x7efdf40fd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdf96fe640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf4106a80 0x7efdf40fd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdf96fe640 1 -- 172.21.15.89:0/262785455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdf4100a80 msgr2=0x7efdf40fcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdf96fe640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdf4100a80 0x7efdf40fcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdf96fe640 1 -- 172.21.15.89:0/262785455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf41a6920 con 0x7efdf4101420 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdfa700640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf4106a80 0x7efdf40fd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdf96fe640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdf4101420 0x7efdf40fd2e0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7efde400d770 tx=0x7efde400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdeaffd640 1 -- 172.21.15.89:0/262785455 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde4017070 con 0x7efdf4101420 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdfc163640 1 -- 172.21.15.89:0/262785455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf41a6c10 con 0x7efdf4101420 2024-09-15T23:40:54.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdeaffd640 1 -- 172.21.15.89:0/262785455 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efde4005500 con 0x7efdf4101420 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdeaffd640 1 -- 172.21.15.89:0/262785455 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde4004e00 con 0x7efdf4101420 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.007+0000 7efdfc163640 1 -- 172.21.15.89:0/262785455 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf41a7150 con 0x7efdf4101420 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.010+0000 7efdeaffd640 1 -- 172.21.15.89:0/262785455 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7efde4004fa0 con 0x7efdf4101420 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.010+0000 7efdfe53c740 1 Processor -- start 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.010+0000 7efdfe53c740 1 -- start start 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.010+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cef090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.010+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9cefa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.010+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cfa370 0x55e3e9cfc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.010+0000 7efdfe53c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9c552f0 con 0x55e3e9cf0b40 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.010+0000 7efdfe53c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9bfccf0 con 0x55e3e9cfa370 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.010+0000 7efdfe53c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9c512b0 con 0x55e3e9cef5d0 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cef090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdeaffd640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efde00a3240 0x7efde00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cef090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47036/0 (socket says 172.21.15.89:47036) 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 -- 172.21.15.89:0/1836533353 learned_addr learned my addr 172.21.15.89:0/1836533353 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdf9eff640 1 --2- 172.21.15.89:0/1836533353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9cefa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdf96fe640 1 --2- 172.21.15.89:0/1836533353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cfa370 0x55e3e9cfc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efde00a3240 0x7efde00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdeaffd640 1 -- 172.21.15.89:0/262785455 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7efde40cf7e0 con 0x7efdf4101420 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 -- 172.21.15.89:0/1836533353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 msgr2=0x55e3e9cefa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 --2- 172.21.15.89:0/1836533353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9cefa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 -- 172.21.15.89:0/1836533353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cfa370 msgr2=0x55e3e9cfc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 --2- 172.21.15.89:0/1836533353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cfa370 0x55e3e9cfc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 -- 172.21.15.89:0/1836533353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3e9be13a0 con 0x55e3e9cf0b40 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 --2- 172.21.15.89:0/1836533353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cef090 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7efdec014ec0 tx=0x7efdec012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74592f37452510 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:54.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efdfa700640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efde00a3240 0x7efde00a5700 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7efdec012c60 tx=0x7efdec00c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efddffff640 1 -- 172.21.15.89:0/1836533353 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec014020 con 0x55e3e9cf0b40 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efddffff640 1 -- 172.21.15.89:0/1836533353 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdec014470 con 0x55e3e9cf0b40 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.012+0000 7efddffff640 1 -- 172.21.15.89:0/1836533353 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec01fbe0 con 0x55e3e9cf0b40 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.013+0000 7efdfe53c740 1 -- 172.21.15.89:0/1836533353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cf0b40 msgr2=0x55e3e9cef090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.013+0000 7efdfe53c740 1 --2- 172.21.15.89:0/1836533353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cef090 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7efdec014ec0 tx=0x7efdec012760 comp rx=0 tx=0).stop 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.013+0000 7efdfe53c740 1 -- 172.21.15.89:0/1836533353 shutdown_connections 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.013+0000 7efdfe53c740 1 --2- 172.21.15.89:0/1836533353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cfa370 0x55e3e9cfc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.013+0000 7efdfe53c740 1 --2- 172.21.15.89:0/1836533353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9cefa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.013+0000 7efdfe53c740 1 --2- 172.21.15.89:0/1836533353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cef090 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.013+0000 7efdfe53c740 1 -- 172.21.15.89:0/1836533353 >> 172.21.15.89:0/1836533353 conn(0x55e3e9cb8310 msgr2=0x55e3e9cebc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.013+0000 7efdfe53c740 1 -- 172.21.15.89:0/1836533353 shutdown_connections 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.013+0000 7efdfe53c740 1 -- 172.21.15.89:0/1836533353 wait complete. 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfe53c740 1 -- start start 2024-09-15T23:40:54.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfe53c740 1 Processor -- start 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9cf9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cf7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf7640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfe53c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9cfd010 con 0x55e3e9cfa370 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfe53c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9c10060 con 0x55e3e9cf0b40 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfe53c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e3e9cfd190 con 0x55e3e9cef5d0 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfe53c740 10 client.0 _ll_register_callbacks cb 0x55e3e99638c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfa700640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfa700640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47052/0 (socket says 172.21.15.89:47052) 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdfa700640 1 -- 172.21.15.89:0/918427356 learned_addr learned my addr 172.21.15.89:0/918427356 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.014+0000 7efdf96fe640 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cf7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efdf9eff640 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9cf9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efdfa700640 1 -- 172.21.15.89:0/918427356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 msgr2=0x55e3e9cf9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efdfa700640 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9cf9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efdfa700640 1 -- 172.21.15.89:0/918427356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cf0b40 msgr2=0x55e3e9cf7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efdfa700640 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cf7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efdfa700640 1 -- 172.21.15.89:0/918427356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3e99c63c0 con 0x55e3e9cfa370 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efdfa700640 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf7640 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7efdec0148f0 tx=0x7efdec009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e0b1d1c3f39cd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efddd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efddd7fa640 1 -- 172.21.15.89:0/918427356 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec00cd10 con 0x55e3e9cfa370 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efddd7fa640 1 -- 172.21.15.89:0/918427356 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7efdec014230 con 0x55e3e9cfa370 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efdfe53c740 1 -- 172.21.15.89:0/918427356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdec014470 con 0x55e3e9cfa370 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efddd7fa640 1 -- 172.21.15.89:0/918427356 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efdec01f960 con 0x55e3e9cfa370 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efdfe53c740 10 client.21324 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:54.033 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efdfe53c740 1 -- 172.21.15.89:0/918427356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e3e9d08ae0 con 0x55e3e9cfa370 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efddd7fa640 1 -- 172.21.15.89:0/918427356 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdec014470 con 0x55e3e9cfa370 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.015+0000 7efddd7fa640 1 -- 172.21.15.89:0/918427356 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7efdec02f070 con 0x55e3e9cfa370 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.016+0000 7efdfe53c740 10 client.21324 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.016+0000 7efdfe53c740 10 client.21324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.016+0000 7efdfe53c740 20 client.21324 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.016+0000 7efdfe53c740 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e3e9d09e30 0x55e3e9d0c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.016+0000 7efdfe53c740 4 client.21324 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.016+0000 7efdfe53c740 1 -- 172.21.15.89:0/918427356 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3e9c62860 con 0x55e3e9d09e30 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.016+0000 7efdf9eff640 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e3e9d09e30 0x55e3e9d0c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.016+0000 7efddd7fa640 10 client.21324 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.016+0000 7efdf9eff640 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e3e9d09e30 0x55e3e9d0c240 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd49d4127ee09acb server_cookie=7772ff7adffb8d53 in_seq=0 out_seq=0 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.017+0000 7efddd7fa640 1 -- 172.21.15.89:0/918427356 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e3e9c62860 con 0x55e3e9d09e30 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.017+0000 7efddd7fa640 10 client.21324 handle_command_reply: tid=0 2024-09-15T23:40:54.034 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.017+0000 7efdfe53c740 1 client.21324 shutdown 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.017+0000 7efdfe53c740 1 -- 172.21.15.89:0/918427356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 msgr2=0x55e3e9cf7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.017+0000 7efdfe53c740 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf7640 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7efdec0148f0 tx=0x7efdec009920 comp rx=0 tx=0).stop 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.018+0000 7efdfe53c740 1 -- 172.21.15.89:0/918427356 shutdown_connections 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.018+0000 7efdfe53c740 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e3e9d09e30 0x55e3e9d0c240 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.018+0000 7efdfe53c740 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf7640 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.018+0000 7efdfe53c740 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cf7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.018+0000 7efddd7fa640 0 client.21324 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.018+0000 7efdfe53c740 1 --2- 172.21.15.89:0/918427356 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9cf9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.018+0000 7efdfe53c740 1 -- 172.21.15.89:0/918427356 >> 172.21.15.89:0/918427356 conn(0x55e3e9cb8310 msgr2=0x55e3e9cf1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.018+0000 7efdfe53c740 1 -- 172.21.15.89:0/918427356 shutdown_connections 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.018+0000 7efdfe53c740 1 -- 172.21.15.89:0/918427356 wait complete. 2024-09-15T23:40:54.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.018+0000 7efdfe53c740 20 client.21324 trim_cache size 0 max 16384 2024-09-15T23:40:54.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.030+0000 7efdfe53c740 1 Processor -- start 2024-09-15T23:40:54.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.030+0000 7efdfe53c740 1 -- start start 2024-09-15T23:40:54.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cef5d0 0x55e3e9d062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cf2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfe53c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efdf41b1010 con 0x55e3e9cfa370 2024-09-15T23:40:54.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfe53c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efdf41b2b50 con 0x55e3e9cef5d0 2024-09-15T23:40:54.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfe53c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efdf41b16e0 con 0x55e3e9cf0b40 2024-09-15T23:40:54.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfa700640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.041 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfa700640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47064/0 (socket says 172.21.15.89:47064) 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfa700640 1 -- 172.21.15.89:0/1427299184 learned_addr learned my addr 172.21.15.89:0/1427299184 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdf96fe640 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cf2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfa700640 1 -- 172.21.15.89:0/1427299184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cf0b40 msgr2=0x55e3e9cf2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfa700640 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cf2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdf9eff640 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cef5d0 0x55e3e9d062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfa700640 1 -- 172.21.15.89:0/1427299184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cef5d0 msgr2=0x55e3e9d062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfa700640 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cef5d0 0x55e3e9d062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfa700640 1 -- 172.21.15.89:0/1427299184 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3e9d0c8c0 con 0x55e3e9cfa370 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdf96fe640 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cf2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdf9eff640 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cef5d0 0x55e3e9d062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efdfa700640 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf2660 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7efdec005c40 tx=0x7efdec014d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98c4c314d8edadc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efddf7fe640 1 -- 172.21.15.89:0/1427299184 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec014920 con 0x55e3e9cfa370 2024-09-15T23:40:54.042 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efddf7fe640 1 -- 172.21.15.89:0/1427299184 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdec01f410 con 0x55e3e9cfa370 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.031+0000 7efddf7fe640 1 -- 172.21.15.89:0/1427299184 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec014040 con 0x55e3e9cfa370 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.032+0000 7efdfe53c740 1 -- 172.21.15.89:0/1427299184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 msgr2=0x55e3e9cf2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.032+0000 7efdfe53c740 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf2660 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7efdec005c40 tx=0x7efdec014d10 comp rx=0 tx=0).stop 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.032+0000 7efdfe53c740 1 -- 172.21.15.89:0/1427299184 shutdown_connections 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.032+0000 7efdfe53c740 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf2660 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.032+0000 7efdfe53c740 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cf0b40 0x55e3e9cf2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.032+0000 7efdfe53c740 1 --2- 172.21.15.89:0/1427299184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cef5d0 0x55e3e9d062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.032+0000 7efdfe53c740 1 -- 172.21.15.89:0/1427299184 >> 172.21.15.89:0/1427299184 conn(0x55e3e9cb8310 msgr2=0x55e3e9cf6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.032+0000 7efdfe53c740 1 -- 172.21.15.89:0/1427299184 shutdown_connections 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.032+0000 7efdfe53c740 1 -- 172.21.15.89:0/1427299184 wait complete. 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.032+0000 7efdfe53c740 1 -- start start 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfe53c740 1 Processor -- start 2024-09-15T23:40:54.043 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9d025d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cf0b40 0x55e3e9d02b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfe53c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfe53c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7efdf41b2480 con 0x55e3e9cfa370 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfe53c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efdf41b1db0 con 0x55e3e9cf0b40 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfe53c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efdf41aeff0 con 0x55e3e9cef5d0 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfe53c740 10 client.0 _ll_register_callbacks cb 0x7efde400de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfa700640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfa700640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47078/0 (socket says 172.21.15.89:47078) 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfa700640 1 -- 172.21.15.89:0/3233661036 learned_addr learned my addr 172.21.15.89:0/3233661036 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdf96fe640 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cf0b40 0x55e3e9d02b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdf9eff640 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9d025d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfa700640 1 -- 172.21.15.89:0/3233661036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 msgr2=0x55e3e9d025d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:54.044 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfa700640 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9d025d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfa700640 1 -- 172.21.15.89:0/3233661036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cf0b40 msgr2=0x55e3e9d02b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfa700640 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cf0b40 0x55e3e9d02b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.033+0000 7efdfa700640 1 -- 172.21.15.89:0/3233661036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3e9cf3fb0 con 0x55e3e9cfa370 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efdfa700640 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf38a0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7efdec012d60 tx=0x7efdec009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ec25fcdbb6e956 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efddd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efddd7fa640 1 -- 172.21.15.89:0/3233661036 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec02b030 con 0x55e3e9cfa370 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efddd7fa640 1 -- 172.21.15.89:0/3233661036 <== mon.0 v2:172.21.15.89:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7efdec01fb00 con 0x55e3e9cfa370 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efdfe53c740 1 -- 172.21.15.89:0/3233661036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdec01f410 con 0x55e3e9cfa370 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efddd7fa640 1 -- 172.21.15.89:0/3233661036 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efdec051050 con 0x55e3e9cfa370 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efdfe53c740 10 client.21330 fetch_fsmap learned FSMap version 29 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efdfe53c740 1 -- 172.21.15.89:0/3233661036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e3e9d74410 con 0x55e3e9cfa370 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efddd7fa640 1 -- 172.21.15.89:0/3233661036 <== mon.0 v2:172.21.15.89:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdec01f410 con 0x55e3e9cfa370 2024-09-15T23:40:54.045 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efddd7fa640 1 -- 172.21.15.89:0/3233661036 <== mon.0 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7efdec02f070 con 0x55e3e9cfa370 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efdfe53c740 10 client.21330 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efdfe53c740 10 client.21330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efdfe53c740 20 client.21330 populate_metadata read hostname 'smithi089' 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efdfe53c740 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e3e9d748e0 0x55e3e9d76cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efdfe53c740 4 client.21330 mds_command: new command op to 24452 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.034+0000 7efdfe53c740 1 -- 172.21.15.89:0/3233661036 --> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e3e9c5cd70 con 0x55e3e9d748e0 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdf9eff640 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e3e9d748e0 0x55e3e9d76cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efddd7fa640 10 client.21330 ms_handle_connect on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdf9eff640 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e3e9d748e0 0x55e3e9d76cd0 crc :-1 s=READY pgs=437 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa9fbcfc0fe4510d server_cookie=19b1713aba3c4810 in_seq=0 out_seq=0 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efddd7fa640 1 -- 172.21.15.89:0/3233661036 <== mds.0 v2:172.21.15.120:6838/1554218591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e3e9c5cd70 con 0x55e3e9d748e0 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efddd7fa640 10 client.21330 handle_command_reply: tid=0 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdfe53c740 1 client.21330 shutdown 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdfe53c740 1 -- 172.21.15.89:0/3233661036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 msgr2=0x55e3e9cf38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.046 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdfe53c740 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf38a0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7efdec012d60 tx=0x7efdec009700 comp rx=0 tx=0).stop 2024-09-15T23:40:54.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdfe53c740 1 -- 172.21.15.89:0/3233661036 shutdown_connections 2024-09-15T23:40:54.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdfe53c740 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.120:6838/1554218591,v1:172.21.15.120:6839/1554218591] conn(0x55e3e9d748e0 0x55e3e9d76cd0 crc :-1 s=READY pgs=437 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdfe53c740 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3e9cfa370 0x55e3e9cf38a0 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdfe53c740 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3e9cf0b40 0x55e3e9d02b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdfe53c740 1 --2- 172.21.15.89:0/3233661036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e3e9cef5d0 0x55e3e9d025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efddd7fa640 0 client.21330 ms_handle_reset on v2:172.21.15.120:6838/1554218591 2024-09-15T23:40:54.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.035+0000 7efdfe53c740 1 -- 172.21.15.89:0/3233661036 >> 172.21.15.89:0/3233661036 conn(0x55e3e9cb8310 msgr2=0x55e3e9cea4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:54.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfe53c740 1 -- 172.21.15.89:0/3233661036 shutdown_connections 2024-09-15T23:40:54.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfe53c740 1 -- 172.21.15.89:0/3233661036 wait complete. 2024-09-15T23:40:54.047 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfe53c740 20 client.21330 trim_cache size 0 max 16384 2024-09-15T23:40:54.048 INFO:teuthology.orchestra.run.smithi089.stdout:[] 2024-09-15T23:40:54.048 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 -- 172.21.15.89:0/262785455 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efde00a3240 msgr2=0x7efde00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efde00a3240 0x7efde00a5700 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7efdec012c60 tx=0x7efdec00c000 comp rx=0 tx=0).stop 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 -- 172.21.15.89:0/262785455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdf4101420 msgr2=0x7efdf40fd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdf4101420 0x7efdf40fd2e0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7efde400d770 tx=0x7efde400dc30 comp rx=0 tx=0).stop 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 -- 172.21.15.89:0/262785455 shutdown_connections 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7efde00a3240 0x7efde00a5700 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf4106a80 0x7efdf40fd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdf4101420 0x7efdf40fd2e0 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 --2- 172.21.15.89:0/262785455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdf4100a80 0x7efdf40fcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 -- 172.21.15.89:0/262785455 >> 172.21.15.89:0/262785455 conn(0x7efdf40fc7a0 msgr2=0x7efdf406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 -- 172.21.15.89:0/262785455 shutdown_connections 2024-09-15T23:40:54.049 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.036+0000 7efdfc163640 1 -- 172.21.15.89:0/262785455 wait complete. 2024-09-15T23:40:54.768 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-15T23:40:54.769 INFO:tasks.fwd_scrub:done joining 2024-09-15T23:40:54.769 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-15T23:40:54.779 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T23:40:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.880+0000 7f4b8b465640 1 -- 172.21.15.89:0/3906393127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b84100c90 msgr2=0x7f4b84101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.880+0000 7f4b8b465640 1 --2- 172.21.15.89:0/3906393127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b84100c90 0x7f4b84101110 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009900 tx=0x7f4b7802a410 comp rx=0 tx=0).stop 2024-09-15T23:40:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 -- 172.21.15.89:0/3906393127 shutdown_connections 2024-09-15T23:40:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 --2- 172.21.15.89:0/3906393127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b84101650 0x7f4b84108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 --2- 172.21.15.89:0/3906393127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b84100c90 0x7f4b84101110 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 --2- 172.21.15.89:0/3906393127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b84102040 0x7f4b84102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 -- 172.21.15.89:0/3906393127 >> 172.21.15.89:0/3906393127 conn(0x7f4b840fb910 msgr2=0x7f4b840fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 -- 172.21.15.89:0/3906393127 shutdown_connections 2024-09-15T23:40:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 -- 172.21.15.89:0/3906393127 wait complete. 2024-09-15T23:40:54.884 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 Processor -- start 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 -- start start 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b84100c90 0x7f4b84197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b84101650 0x7f4b84198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b84102040 0x7f4b8419f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8410c6c0 con 0x7f4b84101650 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8410c540 con 0x7f4b84100c90 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.881+0000 7f4b8b465640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8410c840 con 0x7f4b84102040 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b88a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b84101650 0x7f4b84198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b88a00640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b84101650 0x7f4b84198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:47104/0 (socket says 172.21.15.89:47104) 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b88a00640 1 -- 172.21.15.89:0/76535918 learned_addr learned my addr 172.21.15.89:0/76535918 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b89a02640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b84102040 0x7f4b8419f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b89201640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b84100c90 0x7f4b84197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b88a00640 1 -- 172.21.15.89:0/76535918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b84102040 msgr2=0x7f4b8419f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.885 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b88a00640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b84102040 0x7f4b8419f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b88a00640 1 -- 172.21.15.89:0/76535918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b84100c90 msgr2=0x7f4b84197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:54.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b88a00640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b84100c90 0x7f4b84197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:54.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b88a00640 1 -- 172.21.15.89:0/76535918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8419fae0 con 0x7f4b84101650 2024-09-15T23:40:54.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b89201640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b84100c90 0x7f4b84197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:54.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b88a00640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b84101650 0x7f4b84198400 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f4b78031610 tx=0x7f4b780043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:54.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b827fc640 1 -- 172.21.15.89:0/76535918 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7803f030 con 0x7f4b84101650 2024-09-15T23:40:54.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b827fc640 1 -- 172.21.15.89:0/76535918 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b78003ce0 con 0x7f4b84101650 2024-09-15T23:40:54.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b8b465640 1 -- 172.21.15.89:0/76535918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8419fd70 con 0x7f4b84101650 2024-09-15T23:40:54.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b827fc640 1 -- 172.21.15.89:0/76535918 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78035070 con 0x7f4b84101650 2024-09-15T23:40:54.886 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.882+0000 7f4b8b465640 1 -- 172.21.15.89:0/76535918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b841a01b0 con 0x7f4b84101650 2024-09-15T23:40:54.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.883+0000 7f4b8b465640 1 -- 172.21.15.89:0/76535918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b841a03e0 con 0x7f4b84101650 2024-09-15T23:40:54.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.885+0000 7f4b827fc640 1 -- 172.21.15.89:0/76535918 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4b78042030 con 0x7f4b84101650 2024-09-15T23:40:54.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.885+0000 7f4b827fc640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b6c0a3250 0x7f4b6c0a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:54.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.885+0000 7f4b89201640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b6c0a3250 0x7f4b6c0a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:54.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.885+0000 7f4b827fc640 1 -- 172.21.15.89:0/76535918 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f4b780f05b0 con 0x7f4b84101650 2024-09-15T23:40:54.887 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.886+0000 7f4b89201640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b6c0a3250 0x7f4b6c0a5710 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4b8406a220 tx=0x7f4b7c005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:54.889 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:54.888+0000 7f4b827fc640 1 -- 172.21.15.89:0/76535918 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4b780bce80 con 0x7f4b84101650 2024-09-15T23:40:55.056 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.055+0000 7f4b8b465640 1 -- 172.21.15.89:0/76535918 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b841a05b0 con 0x7f4b84101650 2024-09-15T23:40:55.057 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.056+0000 7f4b827fc640 1 -- 172.21.15.89:0/76535918 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24962 (secure 0 0 0) 0x7f4b780bc780 con 0x7f4b84101650 2024-09-15T23:40:55.057 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:40:55.058 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":90,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:24:03.106312+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.11","mappings":[{"from":7,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.058+0000 7f4b8b465640 1 -- 172.21.15.89:0/76535918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b6c0a3250 msgr2=0x7f4b6c0a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.058+0000 7f4b8b465640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b6c0a3250 0x7f4b6c0a5710 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4b8406a220 tx=0x7f4b7c005e50 comp rx=0 tx=0).stop 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.058+0000 7f4b8b465640 1 -- 172.21.15.89:0/76535918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b84101650 msgr2=0x7f4b84198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.058+0000 7f4b8b465640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b84101650 0x7f4b84198400 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f4b78031610 tx=0x7f4b780043b0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.059+0000 7f4b8b465640 1 -- 172.21.15.89:0/76535918 shutdown_connections 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.059+0000 7f4b8b465640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4b6c0a3250 0x7f4b6c0a5710 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.059+0000 7f4b8b465640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b84102040 0x7f4b8419f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.059+0000 7f4b8b465640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b84101650 0x7f4b84198400 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.059+0000 7f4b8b465640 1 --2- 172.21.15.89:0/76535918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b84100c90 0x7f4b84197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.059+0000 7f4b8b465640 1 -- 172.21.15.89:0/76535918 >> 172.21.15.89:0/76535918 conn(0x7f4b840fb910 msgr2=0x7f4b840ffb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:55.060 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.059+0000 7f4b8b465640 1 -- 172.21.15.89:0/76535918 shutdown_connections 2024-09-15T23:40:55.061 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.059+0000 7f4b8b465640 1 -- 172.21.15.89:0/76535918 wait complete. 2024-09-15T23:40:55.061 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:54 smithi089 ceph-mon[30933]: pgmap v856: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-15T23:40:55.071 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T23:17:15.186397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-15T23:21:49.360728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-15T23:21:51.296727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-15T23:21:57.778828+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-15T23:40:55.072 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T23:40:55.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 -- 172.21.15.89:0/1037959829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff540100a00 msgr2=0x7ff540100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 --2- 172.21.15.89:0/1037959829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff540100a00 0x7ff540100e00 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7ff53c009900 tx=0x7ff53c02a410 comp rx=0 tx=0).stop 2024-09-15T23:40:55.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 -- 172.21.15.89:0/1037959829 shutdown_connections 2024-09-15T23:40:55.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 --2- 172.21.15.89:0/1037959829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5401025b0 0x7ff540108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 --2- 172.21.15.89:0/1037959829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff540101bf0 0x7ff540102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.179 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 --2- 172.21.15.89:0/1037959829 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff540100a00 0x7ff540100e00 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 -- 172.21.15.89:0/1037959829 >> 172.21.15.89:0/1037959829 conn(0x7ff5400fc140 msgr2=0x7ff5400fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 -- 172.21.15.89:0/1037959829 shutdown_connections 2024-09-15T23:40:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 -- 172.21.15.89:0/1037959829 wait complete. 2024-09-15T23:40:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 Processor -- start 2024-09-15T23:40:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 -- start start 2024-09-15T23:40:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff540100a00 0x7ff540196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff540101bf0 0x7ff5401972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.180 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5401025b0 0x7ff54019e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7ff54010c660 con 0x7ff5401025b0 2024-09-15T23:40:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff54010c4e0 con 0x7ff540101bf0 2024-09-15T23:40:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.176+0000 7ff548752640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff54010c7e0 con 0x7ff540100a00 2024-09-15T23:40:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff546cef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5401025b0 0x7ff54019e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff546cef640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5401025b0 0x7ff54019e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43166/0 (socket says 172.21.15.89:43166) 2024-09-15T23:40:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff546cef640 1 -- 172.21.15.89:0/1395067676 learned_addr learned my addr 172.21.15.89:0/1395067676 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff545ced640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff540101bf0 0x7ff5401972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff5464ee640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff540100a00 0x7ff540196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.181 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff546cef640 1 -- 172.21.15.89:0/1395067676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff540100a00 msgr2=0x7ff540196d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff546cef640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff540100a00 0x7ff540196d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff546cef640 1 -- 172.21.15.89:0/1395067676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff540101bf0 msgr2=0x7ff5401972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff546cef640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff540101bf0 0x7ff5401972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff546cef640 1 -- 172.21.15.89:0/1395067676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54019e980 con 0x7ff5401025b0 2024-09-15T23:40:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff5464ee640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff540100a00 0x7ff540196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:55.182 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff546cef640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5401025b0 0x7ff54019e310 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7ff53800eaa0 tx=0x7ff53800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff5377fe640 1 -- 172.21.15.89:0/1395067676 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53800cdb0 con 0x7ff5401025b0 2024-09-15T23:40:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff548752640 1 -- 172.21.15.89:0/1395067676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff54019ec70 con 0x7ff5401025b0 2024-09-15T23:40:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff5377fe640 1 -- 172.21.15.89:0/1395067676 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff538005020 con 0x7ff5401025b0 2024-09-15T23:40:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff5377fe640 1 -- 172.21.15.89:0/1395067676 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5380116b0 con 0x7ff5401025b0 2024-09-15T23:40:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.177+0000 7ff548752640 1 -- 172.21.15.89:0/1395067676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff54019f170 con 0x7ff5401025b0 2024-09-15T23:40:55.183 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.178+0000 7ff548752640 1 -- 172.21.15.89:0/1395067676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff540107ee0 con 0x7ff5401025b0 2024-09-15T23:40:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.179+0000 7ff5377fe640 1 -- 172.21.15.89:0/1395067676 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7ff538005560 con 0x7ff5401025b0 2024-09-15T23:40:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.180+0000 7ff5377fe640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff52c0a3250 0x7ff52c0a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.180+0000 7ff5464ee640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff52c0a3250 0x7ff52c0a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.180+0000 7ff5377fe640 1 -- 172.21.15.89:0/1395067676 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7ff5380cff50 con 0x7ff5401025b0 2024-09-15T23:40:55.184 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.180+0000 7ff5464ee640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff52c0a3250 0x7ff52c0a5710 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff53c000c00 tx=0x7ff53c031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:55.185 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.184+0000 7ff5377fe640 1 -- 172.21.15.89:0/1395067676 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff538016030 con 0x7ff5401025b0 2024-09-15T23:40:55.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:54 smithi120 ceph-mon[35250]: pgmap v856: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-15T23:40:55.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:54 smithi174 ceph-mon[35586]: pgmap v856: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 10 op/s 2024-09-15T23:40:55.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.351+0000 7ff548752640 1 -- 172.21.15.89:0/1395067676 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff540062e90 con 0x7ff5401025b0 2024-09-15T23:40:55.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.352+0000 7ff5377fe640 1 -- 172.21.15.89:0/1395067676 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7ff53809c880 con 0x7ff5401025b0 2024-09-15T23:40:55.353 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 1 2024-09-15T23:40:55.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 -- 172.21.15.89:0/1395067676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff52c0a3250 msgr2=0x7ff52c0a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff52c0a3250 0x7ff52c0a5710 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff53c000c00 tx=0x7ff53c031400 comp rx=0 tx=0).stop 2024-09-15T23:40:55.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 -- 172.21.15.89:0/1395067676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5401025b0 msgr2=0x7ff54019e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5401025b0 0x7ff54019e310 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7ff53800eaa0 tx=0x7ff53800ef60 comp rx=0 tx=0).stop 2024-09-15T23:40:55.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 -- 172.21.15.89:0/1395067676 shutdown_connections 2024-09-15T23:40:55.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7ff52c0a3250 0x7ff52c0a5710 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff5401025b0 0x7ff54019e310 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff540101bf0 0x7ff5401972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 --2- 172.21.15.89:0/1395067676 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff540100a00 0x7ff540196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 -- 172.21.15.89:0/1395067676 >> 172.21.15.89:0/1395067676 conn(0x7ff5400fc140 msgr2=0x7ff5401050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:55.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 -- 172.21.15.89:0/1395067676 shutdown_connections 2024-09-15T23:40:55.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.354+0000 7ff548752640 1 -- 172.21.15.89:0/1395067676 wait complete. 2024-09-15T23:40:55.366 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-15T23:40:55.470 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.468+0000 7f1a6cd98640 1 -- 172.21.15.89:0/2947916300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a681036c0 msgr2=0x7f1a68109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.468+0000 7f1a6cd98640 1 --2- 172.21.15.89:0/2947916300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a681036c0 0x7f1a68109f50 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7f1a60009ca0 tx=0x7f1a6002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.468+0000 7f1a6cd98640 1 -- 172.21.15.89:0/2947916300 shutdown_connections 2024-09-15T23:40:55.479 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.468+0000 7f1a6cd98640 1 --2- 172.21.15.89:0/2947916300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a681036c0 0x7f1a68109f50 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.468+0000 7f1a6cd98640 1 --2- 172.21.15.89:0/2947916300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a68102d00 0x7f1a68103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.468+0000 7f1a6cd98640 1 --2- 172.21.15.89:0/2947916300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a68101b10 0x7f1a68101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.468+0000 7f1a6cd98640 1 -- 172.21.15.89:0/2947916300 >> 172.21.15.89:0/2947916300 conn(0x7f1a680fd2b0 msgr2=0x7f1a680ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.468+0000 7f1a6cd98640 1 -- 172.21.15.89:0/2947916300 shutdown_connections 2024-09-15T23:40:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.468+0000 7f1a6cd98640 1 -- 172.21.15.89:0/2947916300 wait complete. 2024-09-15T23:40:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a6cd98640 1 Processor -- start 2024-09-15T23:40:55.480 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a6cd98640 1 -- start start 2024-09-15T23:40:55.482 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a6cd98640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a68101b10 0x7f1a68075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a6cd98640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a68102d00 0x7f1a680745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a6cd98640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a681036c0 0x7f1a68074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a6cd98640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6810d790 con 0x7f1a681036c0 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a6cd98640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6810d610 con 0x7f1a68101b10 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a6cd98640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6810d910 con 0x7f1a68102d00 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a66d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a681036c0 0x7f1a68074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a66d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a681036c0 0x7f1a68074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43180/0 (socket says 172.21.15.89:43180) 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a66d9d640 1 -- 172.21.15.89:0/3384015640 learned_addr learned my addr 172.21.15.89:0/3384015640 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.469+0000 7f1a6659c640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a68101b10 0x7f1a68075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a5dd9b640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a68102d00 0x7f1a680745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a66d9d640 1 -- 172.21.15.89:0/3384015640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a68102d00 msgr2=0x7f1a680745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.483 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a66d9d640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a68102d00 0x7f1a680745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a66d9d640 1 -- 172.21.15.89:0/3384015640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a68101b10 msgr2=0x7f1a68075fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a66d9d640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a68101b10 0x7f1a68075fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a66d9d640 1 -- 172.21.15.89:0/3384015640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a680753b0 con 0x7f1a681036c0 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a6659c640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a68101b10 0x7f1a68075fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a66d9d640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a681036c0 0x7f1a68074b20 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f1a600098d0 tx=0x7f1a6002f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a5ffff640 1 -- 172.21.15.89:0/3384015640 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6002fc30 con 0x7f1a681036c0 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a6cd98640 1 -- 172.21.15.89:0/3384015640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a681a1f40 con 0x7f1a681036c0 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a5ffff640 1 -- 172.21.15.89:0/3384015640 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a60003e50 con 0x7f1a681036c0 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a5ffff640 1 -- 172.21.15.89:0/3384015640 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a600305c0 con 0x7f1a681036c0 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.470+0000 7f1a6cd98640 1 -- 172.21.15.89:0/3384015640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a681a2400 con 0x7f1a681036c0 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.471+0000 7f1a6cd98640 1 -- 172.21.15.89:0/3384015640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a68108f50 con 0x7f1a681036c0 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.472+0000 7f1a5ffff640 1 -- 172.21.15.89:0/3384015640 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f1a60004b40 con 0x7f1a681036c0 2024-09-15T23:40:55.484 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.473+0000 7f1a5ffff640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1a540a3250 0x7f1a540a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.473+0000 7f1a5ffff640 1 -- 172.21.15.89:0/3384015640 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f1a600f1280 con 0x7f1a681036c0 2024-09-15T23:40:55.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.473+0000 7f1a6659c640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1a540a3250 0x7f1a540a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.473+0000 7f1a6659c640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1a540a3250 0x7f1a540a5710 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1a580058c0 tx=0x7f1a58005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:55.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.476+0000 7f1a5ffff640 1 -- 172.21.15.89:0/3384015640 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1a600bdbb0 con 0x7f1a681036c0 2024-09-15T23:40:55.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.646+0000 7f1a6cd98640 1 -- 172.21.15.89:0/3384015640 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1a6802d9c0 con 0x7f1a681036c0 2024-09-15T23:40:55.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.646+0000 7f1a5ffff640 1 -- 172.21.15.89:0/3384015640 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f1a600bd4b0 con 0x7f1a681036c0 2024-09-15T23:40:55.648 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-09-15T23:40:55.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.648+0000 7f1a6cd98640 1 -- 172.21.15.89:0/3384015640 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1a540a3250 msgr2=0x7f1a540a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.648+0000 7f1a6cd98640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1a540a3250 0x7f1a540a5710 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f1a580058c0 tx=0x7f1a58005ec0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.648+0000 7f1a6cd98640 1 -- 172.21.15.89:0/3384015640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a681036c0 msgr2=0x7f1a68074b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.648+0000 7f1a6cd98640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a681036c0 0x7f1a68074b20 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f1a600098d0 tx=0x7f1a6002f840 comp rx=0 tx=0).stop 2024-09-15T23:40:55.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.649+0000 7f1a6cd98640 1 -- 172.21.15.89:0/3384015640 shutdown_connections 2024-09-15T23:40:55.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.649+0000 7f1a6cd98640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f1a540a3250 0x7f1a540a5710 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.649+0000 7f1a6cd98640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1a681036c0 0x7f1a68074b20 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.649+0000 7f1a6cd98640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a68102d00 0x7f1a680745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.649+0000 7f1a6cd98640 1 --2- 172.21.15.89:0/3384015640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a68101b10 0x7f1a68075fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.649+0000 7f1a6cd98640 1 -- 172.21.15.89:0/3384015640 >> 172.21.15.89:0/3384015640 conn(0x7f1a680fd2b0 msgr2=0x7f1a68106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:55.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.649+0000 7f1a6cd98640 1 -- 172.21.15.89:0/3384015640 shutdown_connections 2024-09-15T23:40:55.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.649+0000 7f1a6cd98640 1 -- 172.21.15.89:0/3384015640 wait complete. 2024-09-15T23:40:55.664 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-15T23:40:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.771+0000 7fde36277640 1 -- 172.21.15.89:0/3277257252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde30101660 msgr2=0x7fde30108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.771+0000 7fde36277640 1 --2- 172.21.15.89:0/3277257252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde30101660 0x7fde30108f00 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7fde28009ca0 tx=0x7fde2802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.771+0000 7fde36277640 1 -- 172.21.15.89:0/3277257252 shutdown_connections 2024-09-15T23:40:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.771+0000 7fde36277640 1 --2- 172.21.15.89:0/3277257252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde30101660 0x7fde30108f00 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.771+0000 7fde36277640 1 --2- 172.21.15.89:0/3277257252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde30100ca0 0x7fde30101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.771+0000 7fde36277640 1 --2- 172.21.15.89:0/3277257252 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde30102000 0x7fde30102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.771+0000 7fde36277640 1 -- 172.21.15.89:0/3277257252 >> 172.21.15.89:0/3277257252 conn(0x7fde300fb900 msgr2=0x7fde300fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.771+0000 7fde36277640 1 -- 172.21.15.89:0/3277257252 shutdown_connections 2024-09-15T23:40:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.771+0000 7fde36277640 1 -- 172.21.15.89:0/3277257252 wait complete. 2024-09-15T23:40:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.771+0000 7fde36277640 1 Processor -- start 2024-09-15T23:40:55.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde36277640 1 -- start start 2024-09-15T23:40:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde36277640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde30100ca0 0x7fde30198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde36277640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde30101660 0x7fde301985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde36277640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde30102000 0x7fde3019f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde36277640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fde3010c7b0 con 0x7fde30101660 2024-09-15T23:40:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde36277640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fde3010c630 con 0x7fde30102000 2024-09-15T23:40:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde36277640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fde3010c930 con 0x7fde30100ca0 2024-09-15T23:40:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde2f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde30101660 0x7fde301985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde2f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde30101660 0x7fde301985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43206/0 (socket says 172.21.15.89:43206) 2024-09-15T23:40:55.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde2f7fe640 1 -- 172.21.15.89:0/1173928885 learned_addr learned my addr 172.21.15.89:0/1173928885 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde34814640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde30102000 0x7fde3019f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde2ffff640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde30100ca0 0x7fde30198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde2f7fe640 1 -- 172.21.15.89:0/1173928885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde30100ca0 msgr2=0x7fde30198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde2f7fe640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde30100ca0 0x7fde30198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde2f7fe640 1 -- 172.21.15.89:0/1173928885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde30102000 msgr2=0x7fde3019f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde2f7fe640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde30102000 0x7fde3019f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde2f7fe640 1 -- 172.21.15.89:0/1173928885 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3019fc90 con 0x7fde30101660 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.772+0000 7fde2f7fe640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde30101660 0x7fde301985b0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fde2400b4a0 tx=0x7fde2400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.773+0000 7fde2d7fa640 1 -- 172.21.15.89:0/1173928885 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24014070 con 0x7fde30101660 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.773+0000 7fde2d7fa640 1 -- 172.21.15.89:0/1173928885 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde24010af0 con 0x7fde30101660 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.773+0000 7fde36277640 1 -- 172.21.15.89:0/1173928885 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3019ff80 con 0x7fde30101660 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.773+0000 7fde2d7fa640 1 -- 172.21.15.89:0/1173928885 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde24017d10 con 0x7fde30101660 2024-09-15T23:40:55.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.773+0000 7fde36277640 1 -- 172.21.15.89:0/1173928885 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde301a04c0 con 0x7fde30101660 2024-09-15T23:40:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.774+0000 7fde36277640 1 -- 172.21.15.89:0/1173928885 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde301a0750 con 0x7fde30101660 2024-09-15T23:40:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.775+0000 7fde2d7fa640 1 -- 172.21.15.89:0/1173928885 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fde24017070 con 0x7fde30101660 2024-09-15T23:40:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.775+0000 7fde2d7fa640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde180a3250 0x7fde180a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:55.777 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.776+0000 7fde2ffff640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde180a3250 0x7fde180a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.776+0000 7fde2d7fa640 1 -- 172.21.15.89:0/1173928885 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fde240cf060 con 0x7fde30101660 2024-09-15T23:40:55.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.776+0000 7fde2ffff640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde180a3250 0x7fde180a5710 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fde3006a160 tx=0x7fde20008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:55.780 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.780+0000 7fde2d7fa640 1 -- 172.21.15.89:0/1173928885 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fde2409b990 con 0x7fde30101660 2024-09-15T23:40:55.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.947+0000 7fde36277640 1 -- 172.21.15.89:0/1173928885 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fde301a09c0 con 0x7fde30101660 2024-09-15T23:40:55.948 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.947+0000 7fde2d7fa640 1 -- 172.21.15.89:0/1173928885 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fde2409b290 con 0x7fde30101660 2024-09-15T23:40:55.948 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 64 2024-09-15T23:40:55.950 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 -- 172.21.15.89:0/1173928885 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde180a3250 msgr2=0x7fde180a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde180a3250 0x7fde180a5710 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fde3006a160 tx=0x7fde20008040 comp rx=0 tx=0).stop 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 -- 172.21.15.89:0/1173928885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde30101660 msgr2=0x7fde301985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde30101660 0x7fde301985b0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fde2400b4a0 tx=0x7fde2400b960 comp rx=0 tx=0).stop 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 -- 172.21.15.89:0/1173928885 shutdown_connections 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fde180a3250 0x7fde180a5710 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde30102000 0x7fde3019f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fde30101660 0x7fde301985b0 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 --2- 172.21.15.89:0/1173928885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde30100ca0 0x7fde30198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 -- 172.21.15.89:0/1173928885 >> 172.21.15.89:0/1173928885 conn(0x7fde300fb900 msgr2=0x7fde300ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 -- 172.21.15.89:0/1173928885 shutdown_connections 2024-09-15T23:40:55.951 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:55.949+0000 7fde36277640 1 -- 172.21.15.89:0/1173928885 wait complete. 2024-09-15T23:40:55.960 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:55 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/76535918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:40:55.961 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:55 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1395067676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:40:55.961 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:55 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3384015640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:40:55.962 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-15T23:40:56.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 -- 172.21.15.89:0/1489174232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc988101bf0 msgr2=0x7fc988102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:56.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 --2- 172.21.15.89:0/1489174232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc988101bf0 0x7fc988102070 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fc970009900 tx=0x7fc97002a410 comp rx=0 tx=0).stop 2024-09-15T23:40:56.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 -- 172.21.15.89:0/1489174232 shutdown_connections 2024-09-15T23:40:56.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 --2- 172.21.15.89:0/1489174232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9881025b0 0x7fc988108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 --2- 172.21.15.89:0/1489174232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc988101bf0 0x7fc988102070 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 --2- 172.21.15.89:0/1489174232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc988100a00 0x7fc988100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.067 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 -- 172.21.15.89:0/1489174232 >> 172.21.15.89:0/1489174232 conn(0x7fc9880fc140 msgr2=0x7fc9880fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 -- 172.21.15.89:0/1489174232 shutdown_connections 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 -- 172.21.15.89:0/1489174232 wait complete. 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 Processor -- start 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.064+0000 7fc98dac9640 1 -- start start 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc98dac9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc988100a00 0x7fc988069ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc98dac9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc988101bf0 0x7fc98819cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc98dac9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9881025b0 0x7fc98819f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc98dac9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fc98810c680 con 0x7fc988100a00 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc98dac9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc98810c500 con 0x7fc9881025b0 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc98dac9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc98810c800 con 0x7fc988101bf0 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9877fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc988100a00 0x7fc988069ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9877fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc988100a00 0x7fc988069ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43236/0 (socket says 172.21.15.89:43236) 2024-09-15T23:40:56.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9877fe640 1 -- 172.21.15.89:0/3752919139 learned_addr learned my addr 172.21.15.89:0/3752919139 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc987fff640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9881025b0 0x7fc98819f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc97ffff640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc988101bf0 0x7fc98819cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9877fe640 1 -- 172.21.15.89:0/3752919139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc988101bf0 msgr2=0x7fc98819cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9877fe640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc988101bf0 0x7fc98819cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9877fe640 1 -- 172.21.15.89:0/3752919139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9881025b0 msgr2=0x7fc98819f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9877fe640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9881025b0 0x7fc98819f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9877fe640 1 -- 172.21.15.89:0/3752919139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98819fa80 con 0x7fc988100a00 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9877fe640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc988100a00 0x7fc988069ef0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fc97800d950 tx=0x7fc97800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9857fa640 1 -- 172.21.15.89:0/3752919139 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978017070 con 0x7fc988100a00 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc98dac9640 1 -- 172.21.15.89:0/3752919139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc98819fd10 con 0x7fc988100a00 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9857fa640 1 -- 172.21.15.89:0/3752919139 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc978009c30 con 0x7fc988100a00 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc9857fa640 1 -- 172.21.15.89:0/3752919139 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97800f600 con 0x7fc988100a00 2024-09-15T23:40:56.069 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.065+0000 7fc98dac9640 1 -- 172.21.15.89:0/3752919139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9881a01b0 con 0x7fc988100a00 2024-09-15T23:40:56.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.066+0000 7fc98dac9640 1 -- 172.21.15.89:0/3752919139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc988107e60 con 0x7fc988100a00 2024-09-15T23:40:56.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.067+0000 7fc9857fa640 1 -- 172.21.15.89:0/3752919139 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fc97801a070 con 0x7fc988100a00 2024-09-15T23:40:56.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.068+0000 7fc9857fa640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3250 0x7fc9740a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:56.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.068+0000 7fc97ffff640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3250 0x7fc9740a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.068+0000 7fc9857fa640 1 -- 172.21.15.89:0/3752919139 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fc9780cf820 con 0x7fc988100a00 2024-09-15T23:40:56.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.069+0000 7fc97ffff640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3250 0x7fc9740a5710 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc970031610 tx=0x7fc970000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:56.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.071+0000 7fc9857fa640 1 -- 172.21.15.89:0/3752919139 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc978009de0 con 0x7fc988100a00 2024-09-15T23:40:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:55 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/76535918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:40:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:55 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1395067676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:40:56.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:55 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3384015640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:40:56.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:55 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/76535918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:40:56.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:55 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1395067676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:40:56.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:55 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3384015640' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:40:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.237+0000 7fc98dac9640 1 -- 172.21.15.89:0/3752919139 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc98802da10 con 0x7fc988100a00 2024-09-15T23:40:56.238 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.237+0000 7fc9857fa640 1 -- 172.21.15.89:0/3752919139 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7fc97809c150 con 0x7fc988100a00 2024-09-15T23:40:56.238 INFO:teuthology.orchestra.run.smithi089.stdout:pg_num: 32 2024-09-15T23:40:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 -- 172.21.15.89:0/3752919139 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3250 msgr2=0x7fc9740a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:56.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3250 0x7fc9740a5710 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc970031610 tx=0x7fc970000bc0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 -- 172.21.15.89:0/3752919139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc988100a00 msgr2=0x7fc988069ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc988100a00 0x7fc988069ef0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fc97800d950 tx=0x7fc97800de10 comp rx=0 tx=0).stop 2024-09-15T23:40:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 -- 172.21.15.89:0/3752919139 shutdown_connections 2024-09-15T23:40:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fc9740a3250 0x7fc9740a5710 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc9881025b0 0x7fc98819f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc988101bf0 0x7fc98819cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 --2- 172.21.15.89:0/3752919139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc988100a00 0x7fc988069ef0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 -- 172.21.15.89:0/3752919139 >> 172.21.15.89:0/3752919139 conn(0x7fc9880fc140 msgr2=0x7fc988105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 -- 172.21.15.89:0/3752919139 shutdown_connections 2024-09-15T23:40:56.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.239+0000 7fc98dac9640 1 -- 172.21.15.89:0/3752919139 wait complete. 2024-09-15T23:40:56.252 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-15T23:40:56.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.349+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/1078563665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb64101b30 msgr2=0x7fbb64101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:56.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.349+0000 7fbb6c5c4640 1 --2- 172.21.15.89:0/1078563665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb64101b30 0x7fbb64101f30 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009900 tx=0x7fbb5402a410 comp rx=0 tx=0).stop 2024-09-15T23:40:56.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.349+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/1078563665 shutdown_connections 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.349+0000 7fbb6c5c4640 1 --2- 172.21.15.89:0/1078563665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb641036e0 0x7fbb64109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.349+0000 7fbb6c5c4640 1 --2- 172.21.15.89:0/1078563665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb64102d20 0x7fbb641031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.349+0000 7fbb6c5c4640 1 --2- 172.21.15.89:0/1078563665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb64101b30 0x7fbb64101f30 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.349+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/1078563665 >> 172.21.15.89:0/1078563665 conn(0x7fbb640fd2b0 msgr2=0x7fbb640ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.349+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/1078563665 shutdown_connections 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.349+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/1078563665 wait complete. 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6c5c4640 1 Processor -- start 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6c5c4640 1 -- start start 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6c5c4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb64101b30 0x7fbb64197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6c5c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb64102d20 0x7fbb641983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6c5c4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb641036e0 0x7fbb6419f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:56.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6c5c4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6410d7b0 con 0x7fbb641036e0 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6c5c4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6410d630 con 0x7fbb64101b30 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6c5c4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6410d930 con 0x7fbb64102d20 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6ab61640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb641036e0 0x7fbb6419f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6ab61640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb641036e0 0x7fbb6419f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:43242/0 (socket says 172.21.15.89:43242) 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6ab61640 1 -- 172.21.15.89:0/3816747801 learned_addr learned my addr 172.21.15.89:0/3816747801 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb69b5f640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb64102d20 0x7fbb641983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6a360640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb64101b30 0x7fbb64197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6ab61640 1 -- 172.21.15.89:0/3816747801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb64102d20 msgr2=0x7fbb641983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6ab61640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb64102d20 0x7fbb641983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6ab61640 1 -- 172.21.15.89:0/3816747801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb64101b30 msgr2=0x7fbb64197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6ab61640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb64101b30 0x7fbb64197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6ab61640 1 -- 172.21.15.89:0/3816747801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6419fa80 con 0x7fbb641036e0 2024-09-15T23:40:56.354 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6a360640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb64101b30 0x7fbb64197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:40:56.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.350+0000 7fbb6ab61640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb641036e0 0x7fbb6419f410 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00eb10 tx=0x7fbb5c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:56.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.351+0000 7fbb5b7fe640 1 -- 172.21.15.89:0/3816747801 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c004b20 con 0x7fbb641036e0 2024-09-15T23:40:56.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.351+0000 7fbb5b7fe640 1 -- 172.21.15.89:0/3816747801 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb5c004cc0 con 0x7fbb641036e0 2024-09-15T23:40:56.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.351+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/3816747801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb6419fd70 con 0x7fbb641036e0 2024-09-15T23:40:56.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.351+0000 7fbb5b7fe640 1 -- 172.21.15.89:0/3816747801 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c010760 con 0x7fbb641036e0 2024-09-15T23:40:56.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.351+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/3816747801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb641a0260 con 0x7fbb641036e0 2024-09-15T23:40:56.355 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.352+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/3816747801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb64108fc0 con 0x7fbb641036e0 2024-09-15T23:40:56.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.353+0000 7fbb5b7fe640 1 -- 172.21.15.89:0/3816747801 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fbb5c005000 con 0x7fbb641036e0 2024-09-15T23:40:56.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.353+0000 7fbb5b7fe640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb500a3250 0x7fbb500a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:40:56.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.354+0000 7fbb6a360640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb500a3250 0x7fbb500a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:40:56.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.354+0000 7fbb5b7fe640 1 -- 172.21.15.89:0/3816747801 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fbb5c0d07b0 con 0x7fbb641036e0 2024-09-15T23:40:56.356 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.354+0000 7fbb6a360640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb500a3250 0x7fbb500a5710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbb54000c00 tx=0x7fbb54031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:40:56.357 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.357+0000 7fbb5b7fe640 1 -- 172.21.15.89:0/3816747801 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbb5c09d0e0 con 0x7fbb641036e0 2024-09-15T23:40:56.571 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.570+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/3816747801 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fbb64062e90 con 0x7fbb641036e0 2024-09-15T23:40:56.579 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.578+0000 7fbb5b7fe640 1 -- 172.21.15.89:0/3816747801 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+742341 (secure 0 0 0) 0x7fbb5c09c9e0 con 0x7fbb641036e0 2024-09-15T23:40:56.589 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:40:56.589 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 20, 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: "active_gid": 14150, 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: "active_name": "x", 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: "active_addrs": { 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6800", 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3594493718 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.590 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6801", 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3594493718 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: "active_addr": "172.21.15.89:6801/3594493718", 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: "active_change": "2024-09-15T23:15:00.884253+0000", 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: "active_mgr_features": 4540701547738038271, 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: "available": true, 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: "standbys": [ 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: "gid": 24109, 2024-09-15T23:40:56.591 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "z", 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_features": 4540701547738038271, 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "available_modules": [ 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "alerts", 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.592 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How frequently to reexamine health status", 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.593 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.594 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.596 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.597 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.598 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_destination": { 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_destination", 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email address to send alerts to", 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_from_name": { 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_from_name", 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "Ceph", 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email From: name", 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.600 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_host": { 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_host", 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP server", 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.601 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_password": { 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_password", 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Password to authenticate with", 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.602 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.603 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_port": { 2024-09-15T23:40:56.603 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_port", 2024-09-15T23:40:56.603 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.604 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.604 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.604 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "465", 2024-09-15T23:40:56.604 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.604 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP port", 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_sender": { 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_sender", 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.605 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.606 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.606 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP envelope sender", 2024-09-15T23:40:56.606 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.606 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.606 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.606 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_ssl": { 2024-09-15T23:40:56.606 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_ssl", 2024-09-15T23:40:56.606 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_user": { 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_user", 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "User to authenticate as", 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.608 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "balancer", 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: "active": { 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "active", 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.609 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically balance PGs across cluster", 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_time": { 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_time", 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.610 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0000", 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "beginning time of day to automatically balance", 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_weekday": { 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_weekday", 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:56.611 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_max_iterations": { 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_max_iterations", 2024-09-15T23:40:56.612 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "25", 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "250", 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_metrics": { 2024-09-15T23:40:56.613 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_metrics", 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "pgs,objects,bytes", 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_step": { 2024-09-15T23:40:56.614 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_step", 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.5", 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0.001", 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "0.999", 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "aggressiveness of optimization", 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.615 INFO:teuthology.orchestra.run.smithi089.stdout: "end_time": { 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_time", 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2359", 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ending time of day to automatically balance", 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.616 INFO:teuthology.orchestra.run.smithi089.stdout: "end_weekday": { 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_weekday", 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.618 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.619 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.620 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.621 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.622 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "min_score": { 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_score", 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.623 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "upmap", 2024-09-15T23:40:56.624 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-compat", 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "none", 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "read", 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap", 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap-read" 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Balancer mode", 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_ids": { 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_ids", 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.626 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.627 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_deviation": { 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_deviation", 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-15T23:40:56.628 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_optimizations": { 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_optimizations", 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-15T23:40:56.629 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephadm", 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_down_multiplier": { 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_down_multiplier", 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:56.630 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_refresh_rate": { 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_refresh_rate", 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.631 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "20", 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_starting_port": { 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_starting_port", 2024-09-15T23:40:56.632 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4721", 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.633 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_ptrace": { 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_ptrace", 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.634 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_interval": { 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_interval", 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.635 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_memory_target_ratio": { 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_memory_target_ratio", 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.7", 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.636 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm_log_destination": { 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephadm_log_destination", 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "file", 2024-09-15T23:40:56.637 INFO:teuthology.orchestra.run.smithi089.stdout: "file,syslog", 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "syslog" 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "cgroups_split": { 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cgroups_split", 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.638 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "config_checks_enabled": { 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_checks_enabled", 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.639 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "config_dashboard": { 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_dashboard", 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.640 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_alertmanager": { 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_alertmanager", 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.641 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_base": { 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_base", 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.642 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Container image name, without the tag", 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_elasticsearch": { 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_elasticsearch", 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.643 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "elasticsearch container image", 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_grafana": { 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_grafana", 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.644 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_haproxy": { 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_haproxy", 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.645 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "HAproxy container image", 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_agent": { 2024-09-15T23:40:56.646 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_agent", 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger agent container image", 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_collector": { 2024-09-15T23:40:56.647 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_collector", 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger collector container image", 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.648 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_query": { 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_query", 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger query container image", 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_keepalived": { 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_keepalived", 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Keepalived container image", 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_loki": { 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_loki", 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Loki container image", 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.651 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_node_exporter": { 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_node_exporter", 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.652 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_nvmeof": { 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_nvmeof", 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Nvme-of container image", 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.653 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_prometheus": { 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_prometheus", 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.654 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_promtail": { 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_promtail", 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Promtail container image", 2024-09-15T23:40:56.655 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_samba": { 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_samba", 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.656 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Samba/SMB container image", 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_snmp_gateway": { 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_snmp_gateway", 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.657 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SNMP Gateway container image", 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "container_init": { 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_init", 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.658 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Run podman/docker with `--init`", 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "daemon_cache_timeout": { 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "daemon_cache_timeout", 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.659 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "default_cephadm_command_timeout": { 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_cephadm_command_timeout", 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "900", 2024-09-15T23:40:56.660 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "default_registry": { 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_registry", 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.661 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io", 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "device_cache_timeout": { 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_cache_timeout", 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.662 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1800", 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache device inventory", 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "device_enhanced_scan": { 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_enhanced_scan", 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.663 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "facts_cache_timeout": { 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "facts_cache_timeout", 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.664 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache host facts data", 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "grafana_dashboards_path": { 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "grafana_dashboards_path", 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.665 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "host_check_interval": { 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "host_check_interval", 2024-09-15T23:40:56.666 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to perform a host check", 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "hw_monitoring": { 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "hw_monitoring", 2024-09-15T23:40:56.667 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "inventory_list_all": { 2024-09-15T23:40:56.668 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "inventory_list_all", 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.669 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.670 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: "log_refresh_metadata": { 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_refresh_metadata", 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.671 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.672 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.673 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.674 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.675 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "*", 2024-09-15T23:40:56.677 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "max_count_per_host": { 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_count_per_host", 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:56.678 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of daemons per service per host", 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd_draining_count": { 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_osd_draining_count", 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:56.679 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "migration_current": { 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "migration_current", 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.680 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "internal - do not modify", 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.681 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "root", 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm-package", 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "root" 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "mode for remote execution of cephadm", 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "oob_default_addr": { 2024-09-15T23:40:56.682 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "oob_default_addr", 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "169.254.1.1", 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.683 INFO:teuthology.orchestra.run.smithi089.stdout: "prometheus_alerts_path": { 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus_alerts_path", 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.684 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_insecure": { 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_insecure", 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.685 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_password": { 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_password", 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.686 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_url": { 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_url", 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.687 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_username": { 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_username", 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.688 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "secure_monitoring_stack": { 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "secure_monitoring_stack", 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.689 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "service_discovery_port": { 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "service_discovery_port", 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8765", 2024-09-15T23:40:56.690 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "cephadm service discovery port", 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "ssh_config_file": { 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssh_config_file", 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.691 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "use_agent": { 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_agent", 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.692 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "use_repo_digest": { 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_repo_digest", 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.693 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_failed_host_check": { 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_failed_host_check", 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.694 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if the host check fails", 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_daemons": { 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_daemons", 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.695 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_hosts": { 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_hosts", 2024-09-15T23:40:56.696 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crash", 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.698 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.699 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.701 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.702 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "retain_interval": { 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retain_interval", 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.703 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "31536000", 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_recent_interval": { 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_recent_interval", 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.704 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1209600", 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.705 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dashboard", 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.706 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_HOST": { 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.707 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.708 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_ENABLED": { 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_ENABLED", 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.710 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "ENABLE_BROWSABLE_API": { 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.711 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.712 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.713 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.714 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-15T23:40:56.715 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-15T23:40:56.716 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-15T23:40:56.717 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.718 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.719 INFO:teuthology.orchestra.run.smithi089.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_PASSWORD": { 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.722 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-15T23:40:56.722 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-15T23:40:56.722 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.722 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.722 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.722 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.722 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.722 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_URL": { 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_URL", 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.724 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_USERNAME": { 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2024-09-15T23:40:56.725 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.726 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-15T23:40:56.727 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-15T23:40:56.728 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-15T23:40:56.729 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.730 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_HOST": { 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.731 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.732 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.733 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.734 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.735 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.736 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.737 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.738 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.739 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_ENABLED": { 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.740 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.741 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.742 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-15T23:40:56.743 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8", 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-15T23:40:56.744 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "45", 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ACCESS_KEY": { 2024-09-15T23:40:56.745 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-15T23:40:56.746 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.747 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SECRET_KEY": { 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.748 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SSL_VERIFY": { 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.752 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "cross_origin_url": { 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cross_origin_url", 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.753 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "crt_file": { 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crt_file", 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.754 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "debug": { 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "debug", 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.755 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable/disable debug options", 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "jwt_token_ttl": { 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "jwt_token_ttl", 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "28800", 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.756 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.757 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.758 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.759 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.760 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.761 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.762 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "motd": { 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "motd", 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.763 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "The message of the day", 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect_resolve_ip_addr": { 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "redirect_resolve_ip_addr", 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.764 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2024-09-15T23:40:56.765 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2024-09-15T23:40:56.766 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8080", 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2024-09-15T23:40:56.767 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl_server_port": { 2024-09-15T23:40:56.768 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl_server_port", 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8443", 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "redirect", 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect" 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.770 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2024-09-15T23:40:56.771 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "url_prefix": { 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url_prefix", 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.772 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "devicehealth", 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.773 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_monitoring": { 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_monitoring", 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "monitor device health metrics", 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.774 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.775 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.776 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.778 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.779 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "mark_out_threshold": { 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mark_out_threshold", 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2419200", 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.780 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": { 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_name", 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "device_health_metrics", 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.781 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "retention_period": { 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retention_period", 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15552000", 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.782 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain device health metrics", 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_frequency": { 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_frequency", 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2024-09-15T23:40:56.783 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "self_heal": { 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "self_heal", 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.784 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.785 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and check device health", 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_threshold": { 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_threshold", 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.786 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7257600", 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "diskprediction_local", 2024-09-15T23:40:56.787 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.788 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.789 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.790 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.791 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.792 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "predict_interval": { 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predict_interval", 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.793 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "predictor_model": { 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predictor_model", 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "prophetstor", 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.794 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.795 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "influx", 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": false, 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "influxdb python module not found", 2024-09-15T23:40:56.796 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "batch_size": { 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "batch_size", 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5000", 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "database": { 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "database", 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "ceph", 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-15T23:40:56.798 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "hostname": { 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "hostname", 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.799 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server hostname", 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "30", 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "5", 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.800 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.801 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.802 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.803 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.804 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.805 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "password": { 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "password", 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.806 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "password of InfluxDB server user", 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "port": { 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "port", 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.807 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8086", 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server port", 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.808 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "false", 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "threads": { 2024-09-15T23:40:56.809 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threads", 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "32", 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.810 INFO:teuthology.orchestra.run.smithi089.stdout: "username": { 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "username", 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "username of InfluxDB server user", 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "verify_ssl": { 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "verify_ssl", 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "true", 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:40:56.812 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "insights", 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.813 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.814 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.815 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.816 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.817 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.818 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "iostat", 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.819 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.821 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.822 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.823 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "k8sevents", 2024-09-15T23:40:56.824 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_event_retention_days": { 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ceph_event_retention_days", 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7", 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "config_check_secs": { 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_check_secs", 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "10", 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.826 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.827 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.828 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.829 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.830 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.831 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "localpool", 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.832 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "failure_domain": { 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "failure_domain", 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "host", 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "failure domain for any created local pool", 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-15T23:40:56.833 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.834 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.835 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.836 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.837 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.838 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": { 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_size", 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.839 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default min_size for any created local pool", 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "num_rep": { 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "num_rep", 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3", 2024-09-15T23:40:56.840 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default replica count for any created local pool", 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": { 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_num", 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.841 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "128", 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default pg_num for any created local pool", 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "prefix": { 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prefix", 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.842 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name prefix for any created local pool", 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "subtree": { 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "subtree", 2024-09-15T23:40:56.843 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "rack", 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mds_autoscaler", 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.845 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.846 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.848 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.849 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirroring", 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.850 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.851 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.852 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.853 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.854 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.855 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "nfs", 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.856 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.857 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.858 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.859 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.860 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.861 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: "fail_fs": { 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "fail_fs", 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.862 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.863 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.864 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.865 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.866 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator": { 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm", 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "rook", 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: "test_orchestrator" 2024-09-15T23:40:56.869 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Orchestrator backend", 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_perf_query", 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.870 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.871 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.872 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.873 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.874 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.875 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_support", 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.876 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.877 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.878 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.879 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.881 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_autoscaler", 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.882 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.883 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.884 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.885 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.886 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.887 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "threshold": { 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threshold", 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1.0", 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "scaling threshold", 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.889 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "progress", 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_pg_recovery_event": { 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_pg_recovery_event", 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.890 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.891 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.892 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.893 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.894 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.895 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.896 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.897 INFO:teuthology.orchestra.run.smithi089.stdout: "max_completed_events": { 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_completed_events", 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "50", 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "number of past completed events to remember", 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long the module is going to sleep", 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.899 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus", 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: "cache": { 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cache", 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.900 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "exclude_perf_counters": { 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "exclude_perf_counters", 2024-09-15T23:40:56.901 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.902 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.903 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.904 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.905 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.906 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.907 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools": { 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools", 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.908 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "300", 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.909 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_interval": { 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_interval", 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15.0", 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.910 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2024-09-15T23:40:56.911 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "9283", 2024-09-15T23:40:56.912 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: "stale_cache_strategy": { 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stale_cache_strategy", 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.913 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "log", 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.914 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "default", 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "default", 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "error" 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.916 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_support", 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.917 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.918 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.919 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.920 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.921 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_snap_create": { 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_snap_create", 2024-09-15T23:40:56.922 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "mirror_snapshot_schedule": { 2024-09-15T23:40:56.923 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirror_snapshot_schedule", 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.924 INFO:teuthology.orchestra.run.smithi089.stdout: "trash_purge_schedule": { 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "trash_purge_schedule", 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.925 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.926 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "restful", 2024-09-15T23:40:56.926 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.926 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.926 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.926 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_auth": { 2024-09-15T23:40:56.926 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_auth", 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.928 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.929 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.930 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.932 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.933 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.934 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.935 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.936 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rgw", 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.938 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.939 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.940 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.941 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.942 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "secondary_zone_period_retry_limit": { 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.943 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rook", 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.944 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.945 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.946 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.947 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.948 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.949 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "storage_class": { 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "storage_class", 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "local", 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-15T23:40:56.950 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "selftest", 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.951 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.952 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.953 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.954 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.955 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "roption1": { 2024-09-15T23:40:56.956 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption1", 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "roption2": { 2024-09-15T23:40:56.957 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption2", 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "xyz", 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption1": { 2024-09-15T23:40:56.958 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption1", 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption2": { 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption2", 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.960 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption3": { 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption3", 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.961 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption4": { 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption4", 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.962 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption5": { 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption5", 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.963 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption6": { 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption6", 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.964 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption7": { 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption7", 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "42", 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.965 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "testkey": { 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testkey", 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.966 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "testlkey": { 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testlkey", 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.967 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "testnewline": { 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testnewline", 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.968 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snap_schedule", 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.969 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_m_granularity": { 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_m_granularity", 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow minute scheduled snapshots", 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "dump_on_update": { 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dump_on_update", 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "dump database to debug log on update", 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.972 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.973 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.974 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.975 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.976 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stats", 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.977 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.978 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.979 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.980 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.981 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.982 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "status", 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.983 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.984 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.985 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.986 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.987 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.988 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telegraf", 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: "address": { 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "address", 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.989 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.990 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15", 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.991 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.993 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:56.994 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:56.995 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:56.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telemetry", 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_basic": { 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_basic", 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.997 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_crash": { 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_crash", 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.998 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_device": { 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_device", 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:56.999 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_ident": { 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_ident", 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.000 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_perf": { 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_perf", 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.001 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "contact": { 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "contact", 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.002 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "description": { 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "description", 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.003 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "device_url": { 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_url", 2024-09-15T23:40:57.004 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2024-09-15T23:40:57.005 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.006 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "24", 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "8", 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "last_opt_revision": { 2024-09-15T23:40:57.007 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "last_opt_revision", 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1", 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.008 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard": { 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard", 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.009 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard_description": { 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard_description", 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.010 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.011 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.012 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.013 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.014 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.015 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "organization": { 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "organization", 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.016 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "proxy": { 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "proxy", 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.017 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "url": { 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url", 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.018 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "test_orchestrator", 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.019 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.021 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.022 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.023 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.024 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "volumes", 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.025 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.026 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.027 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.028 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.029 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.030 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_clones": { 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_clones", 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4", 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "periodic_async_work": { 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "periodic_async_work", 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Periodically check for async work", 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.032 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "snapshot_clone_delay": { 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snapshot_clone_delay", 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.033 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "snapshot_clone_no_wait": { 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snapshot_clone_no_wait", 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.034 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix", 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: "discovery_interval": { 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "discovery_interval", 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.035 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "100", 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "identifier": { 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "identifier", 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.036 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.037 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.038 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.039 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.040 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.041 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.042 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.043 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_host": { 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_host", 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.044 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_port": { 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_port", 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10051", 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_sender": { 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_sender", 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.046 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: "gid": 24122, 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "y", 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_features": 4540701547738038271, 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: "available_modules": [ 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "alerts", 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How frequently to reexamine health status", 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.049 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.050 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.051 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.052 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.053 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_destination": { 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_destination", 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.054 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email address to send alerts to", 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_from_name": { 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_from_name", 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "Ceph", 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.055 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email From: name", 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_host": { 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_host", 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP server", 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_password": { 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_password", 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.057 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Password to authenticate with", 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_port": { 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_port", 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "465", 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.058 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP port", 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_sender": { 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_sender", 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.059 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP envelope sender", 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_ssl": { 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_ssl", 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.060 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_user": { 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_user", 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.061 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "User to authenticate as", 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "balancer", 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.062 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "active": { 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "active", 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically balance PGs across cluster", 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.063 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_time": { 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_time", 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0000", 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "beginning time of day to automatically balance", 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:40:57.064 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_weekday": { 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_weekday", 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-15T23:40:57.065 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_max_iterations": { 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_max_iterations", 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "25", 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "250", 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-15T23:40:57.066 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_metrics": { 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_metrics", 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "pgs,objects,bytes", 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-15T23:40:57.067 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_step": { 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_step", 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.5", 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0.001", 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "0.999", 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.068 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "aggressiveness of optimization", 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "end_time": { 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_time", 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2359", 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.069 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ending time of day to automatically balance", 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "end_weekday": { 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_weekday", 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2024-09-15T23:40:57.070 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.071 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.072 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.074 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.075 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "min_score": { 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_score", 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.076 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.077 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "upmap", 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-compat", 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "none", 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "read", 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap", 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap-read" 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Balancer mode", 2024-09-15T23:40:57.078 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_ids": { 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_ids", 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-15T23:40:57.079 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.080 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_deviation": { 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_deviation", 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.081 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_optimizations": { 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_optimizations", 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.082 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephadm", 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_down_multiplier": { 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_down_multiplier", 2024-09-15T23:40:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_refresh_rate": { 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_refresh_rate", 2024-09-15T23:40:57.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "20", 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_starting_port": { 2024-09-15T23:40:57.085 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_starting_port", 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4721", 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.086 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_ptrace": { 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_ptrace", 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.087 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_interval": { 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_interval", 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_memory_target_ratio": { 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_memory_target_ratio", 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.7", 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm_log_destination": { 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephadm_log_destination", 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "file", 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "file,syslog", 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: "syslog" 2024-09-15T23:40:57.090 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "cgroups_split": { 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cgroups_split", 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.091 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "config_checks_enabled": { 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_checks_enabled", 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.092 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "config_dashboard": { 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_dashboard", 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.093 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_alertmanager": { 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_alertmanager", 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-15T23:40:57.094 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_base": { 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_base", 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-15T23:40:57.095 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Container image name, without the tag", 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_elasticsearch": { 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_elasticsearch", 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.096 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "elasticsearch container image", 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_grafana": { 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_grafana", 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.097 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_haproxy": { 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_haproxy", 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.098 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "HAproxy container image", 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_agent": { 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_agent", 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.099 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger agent container image", 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_collector": { 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_collector", 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.100 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger collector container image", 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_query": { 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_query", 2024-09-15T23:40:57.101 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger query container image", 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_keepalived": { 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_keepalived", 2024-09-15T23:40:57.102 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Keepalived container image", 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_loki": { 2024-09-15T23:40:57.103 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_loki", 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Loki container image", 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.104 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_node_exporter": { 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_node_exporter", 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_nvmeof": { 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_nvmeof", 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Nvme-of container image", 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_prometheus": { 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_prometheus", 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_promtail": { 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_promtail", 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Promtail container image", 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.108 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_samba": { 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_samba", 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Samba/SMB container image", 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.109 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_snmp_gateway": { 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_snmp_gateway", 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SNMP Gateway container image", 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.110 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "container_init": { 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_init", 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.111 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Run podman/docker with `--init`", 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "daemon_cache_timeout": { 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "daemon_cache_timeout", 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.112 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "default_cephadm_command_timeout": { 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_cephadm_command_timeout", 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "900", 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.113 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "default_registry": { 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_registry", 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io", 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.114 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "device_cache_timeout": { 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_cache_timeout", 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1800", 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.115 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache device inventory", 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "device_enhanced_scan": { 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_enhanced_scan", 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.116 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "facts_cache_timeout": { 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "facts_cache_timeout", 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.117 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache host facts data", 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "grafana_dashboards_path": { 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "grafana_dashboards_path", 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.118 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "host_check_interval": { 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "host_check_interval", 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.119 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to perform a host check", 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "hw_monitoring": { 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "hw_monitoring", 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.120 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "inventory_list_all": { 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "inventory_list_all", 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.121 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.122 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.123 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "log_refresh_metadata": { 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_refresh_metadata", 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.124 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.125 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.126 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.128 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.129 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "*", 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.130 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "max_count_per_host": { 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_count_per_host", 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of daemons per service per host", 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.131 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd_draining_count": { 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_osd_draining_count", 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-15T23:40:57.132 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "migration_current": { 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "migration_current", 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "internal - do not modify", 2024-09-15T23:40:57.133 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.134 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.134 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.134 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2024-09-15T23:40:57.134 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2024-09-15T23:40:57.134 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.134 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.135 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.135 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "root", 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm-package", 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "root" 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "mode for remote execution of cephadm", 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "oob_default_addr": { 2024-09-15T23:40:57.136 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "oob_default_addr", 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "169.254.1.1", 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.137 INFO:teuthology.orchestra.run.smithi089.stdout: "prometheus_alerts_path": { 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus_alerts_path", 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_insecure": { 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_insecure", 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_password": { 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_password", 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.140 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_url": { 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_url", 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.141 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_username": { 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_username", 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-15T23:40:57.142 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "secure_monitoring_stack": { 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "secure_monitoring_stack", 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-15T23:40:57.143 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "service_discovery_port": { 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "service_discovery_port", 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8765", 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.144 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "cephadm service discovery port", 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "ssh_config_file": { 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssh_config_file", 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.145 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "use_agent": { 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_agent", 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.146 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "use_repo_digest": { 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_repo_digest", 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.147 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_failed_host_check": { 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_failed_host_check", 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.148 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if the host check fails", 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_daemons": { 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_daemons", 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.149 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_hosts": { 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_hosts", 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.150 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.151 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crash", 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.152 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.153 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.154 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.155 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.156 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "retain_interval": { 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retain_interval", 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "31536000", 2024-09-15T23:40:57.157 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_recent_interval": { 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_recent_interval", 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1209600", 2024-09-15T23:40:57.158 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dashboard", 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.159 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.160 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_HOST": { 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.161 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_ENABLED": { 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_ENABLED", 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.163 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.164 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "ENABLE_BROWSABLE_API": { 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.165 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.166 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.167 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.168 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.169 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.170 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.171 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.172 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-15T23:40:57.173 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_PASSWORD": { 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-15T23:40:57.174 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-15T23:40:57.175 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_URL": { 2024-09-15T23:40:57.176 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_URL", 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_USERNAME": { 2024-09-15T23:40:57.177 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.178 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.179 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.180 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.181 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.182 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_HOST": { 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.183 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.185 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.186 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.187 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.188 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.189 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.190 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.191 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_ENABLED": { 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.192 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.193 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.194 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.195 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8", 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.196 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "45", 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ACCESS_KEY": { 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-15T23:40:57.197 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-15T23:40:57.198 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SECRET_KEY": { 2024-09-15T23:40:57.199 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.200 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SSL_VERIFY": { 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.201 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.202 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "cross_origin_url": { 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cross_origin_url", 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "crt_file": { 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crt_file", 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.206 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "debug": { 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "debug", 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable/disable debug options", 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.207 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "jwt_token_ttl": { 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "jwt_token_ttl", 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "28800", 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.208 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.209 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.210 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.211 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.212 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.213 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.214 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "motd": { 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "motd", 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.215 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "The message of the day", 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect_resolve_ip_addr": { 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "redirect_resolve_ip_addr", 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.216 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.217 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.218 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8080", 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.219 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl_server_port": { 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl_server_port", 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.220 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8443", 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2024-09-15T23:40:57.221 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "redirect", 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect" 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.222 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.223 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "url_prefix": { 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url_prefix", 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.224 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "devicehealth", 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_monitoring": { 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_monitoring", 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.225 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "monitor device health metrics", 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.226 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.227 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.228 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.229 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.231 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "mark_out_threshold": { 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mark_out_threshold", 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2419200", 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.232 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": { 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_name", 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "device_health_metrics", 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.233 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "retention_period": { 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retention_period", 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15552000", 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain device health metrics", 2024-09-15T23:40:57.234 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_frequency": { 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_frequency", 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.235 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "self_heal": { 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "self_heal", 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.236 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.237 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and check device health", 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_threshold": { 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_threshold", 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7257600", 2024-09-15T23:40:57.238 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "diskprediction_local", 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.239 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.240 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.241 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.242 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.243 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.244 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "predict_interval": { 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predict_interval", 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.245 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "predictor_model": { 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predictor_model", 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "prophetstor", 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.246 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.247 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "influx", 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": false, 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "influxdb python module not found", 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.248 INFO:teuthology.orchestra.run.smithi089.stdout: "batch_size": { 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "batch_size", 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5000", 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.249 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "database": { 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "database", 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "ceph", 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.250 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "hostname": { 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "hostname", 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server hostname", 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.251 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "30", 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "5", 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.252 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.253 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.254 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.255 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.256 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.257 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "password": { 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "password", 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.258 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "password of InfluxDB server user", 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "port": { 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "port", 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.259 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8086", 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server port", 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.260 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "false", 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "threads": { 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threads", 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.261 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "32", 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "username": { 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "username", 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.262 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "username of InfluxDB server user", 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "verify_ssl": { 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "verify_ssl", 2024-09-15T23:40:57.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "true", 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "insights", 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.265 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.268 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.269 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "iostat", 2024-09-15T23:40:57.270 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.271 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.272 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.273 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.274 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.275 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "k8sevents", 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_event_retention_days": { 2024-09-15T23:40:57.276 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ceph_event_retention_days", 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7", 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "config_check_secs": { 2024-09-15T23:40:57.277 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_check_secs", 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "10", 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.278 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.279 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.280 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.281 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.282 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.283 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "localpool", 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: "failure_domain": { 2024-09-15T23:40:57.284 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "failure_domain", 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "host", 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "failure domain for any created local pool", 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.285 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.286 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.287 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.288 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.289 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.290 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": { 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_size", 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default min_size for any created local pool", 2024-09-15T23:40:57.291 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-15T23:40:57.292 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.292 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.292 INFO:teuthology.orchestra.run.smithi089.stdout: "num_rep": { 2024-09-15T23:40:57.292 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "num_rep", 2024-09-15T23:40:57.292 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.292 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.292 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.292 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3", 2024-09-15T23:40:57.292 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default replica count for any created local pool", 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": { 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_num", 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.293 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "128", 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default pg_num for any created local pool", 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "prefix": { 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prefix", 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.294 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name prefix for any created local pool", 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "subtree": { 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "subtree", 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.295 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "rack", 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.296 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mds_autoscaler", 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.297 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.298 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.299 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.300 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.301 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirroring", 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.302 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.303 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.304 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.305 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.306 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.307 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "nfs", 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.308 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.309 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.310 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.311 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.312 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.313 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: "fail_fs": { 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "fail_fs", 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.314 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.315 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.316 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.317 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.318 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.319 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.320 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator": { 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm", 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "rook", 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "test_orchestrator" 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.321 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Orchestrator backend", 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_perf_query", 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.322 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.323 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.324 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.325 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.326 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.327 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_support", 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.328 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.329 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.330 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.331 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.332 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.333 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_autoscaler", 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.334 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.337 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.338 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.340 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.341 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "threshold": { 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threshold", 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1.0", 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "scaling threshold", 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.343 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "progress", 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_pg_recovery_event": { 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_pg_recovery_event", 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.344 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.345 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.346 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.347 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.348 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.351 INFO:teuthology.orchestra.run.smithi089.stdout: "max_completed_events": { 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_completed_events", 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "50", 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "number of past completed events to remember", 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.352 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long the module is going to sleep", 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.353 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus", 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: "cache": { 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cache", 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.354 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "exclude_perf_counters": { 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "exclude_perf_counters", 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.355 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.356 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.357 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.358 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.358 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.358 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.358 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.358 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.358 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.358 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.358 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.358 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.359 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.360 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.361 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools": { 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools", 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.362 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "300", 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.363 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_interval": { 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_interval", 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15.0", 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.364 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.365 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "9283", 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-15T23:40:57.366 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: "stale_cache_strategy": { 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stale_cache_strategy", 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "log", 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.367 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "default", 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.368 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "default", 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "error" 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.369 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.370 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_support", 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.371 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.372 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.373 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.374 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.375 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_snap_create": { 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_snap_create", 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.376 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "mirror_snapshot_schedule": { 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirror_snapshot_schedule", 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.377 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "trash_purge_schedule": { 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "trash_purge_schedule", 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.378 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "restful", 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_auth": { 2024-09-15T23:40:57.379 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_auth", 2024-09-15T23:40:57.380 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.380 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.380 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.380 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.380 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.380 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.380 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.380 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.380 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.380 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.381 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.382 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.383 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.384 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.385 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.386 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.387 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.388 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.389 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rgw", 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.390 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.391 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.392 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.393 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.394 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "secondary_zone_period_retry_limit": { 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.395 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rook", 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.396 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.397 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.398 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.399 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.400 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.401 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "storage_class": { 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "storage_class", 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "local", 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-15T23:40:57.402 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "selftest", 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.403 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.404 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.405 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.406 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.407 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.408 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "roption1": { 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption1", 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.409 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "roption2": { 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption2", 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "xyz", 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.410 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption1": { 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption1", 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.411 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption2": { 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption2", 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.412 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption3": { 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption3", 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.413 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.414 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.414 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.414 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.414 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption4": { 2024-09-15T23:40:57.414 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption4", 2024-09-15T23:40:57.414 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.414 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.414 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.414 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption5": { 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption5", 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.415 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption6": { 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption6", 2024-09-15T23:40:57.416 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption7": { 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption7", 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "42", 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.418 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: "testkey": { 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testkey", 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.419 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "testlkey": { 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testlkey", 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.420 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "testnewline": { 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testnewline", 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.421 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snap_schedule", 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.422 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_m_granularity": { 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_m_granularity", 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow minute scheduled snapshots", 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "dump_on_update": { 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dump_on_update", 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "dump database to debug log on update", 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.425 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.425 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.425 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.425 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.425 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.425 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.425 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.425 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.425 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.425 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.426 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.426 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.426 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.426 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.426 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.426 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.426 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.426 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.426 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.427 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.428 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.428 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.428 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.428 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.428 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.428 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.428 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.428 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.428 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.429 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.429 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.429 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.429 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.429 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.429 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.429 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.429 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.429 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.429 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.430 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stats", 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.431 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.432 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.433 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.434 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.435 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.436 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "status", 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.437 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.438 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.438 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.438 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.438 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.438 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.439 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.439 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.439 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.439 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.440 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.441 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.442 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.443 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telegraf", 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: "address": { 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "address", 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.444 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.445 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15", 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.446 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.447 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.448 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.449 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.450 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telemetry", 2024-09-15T23:40:57.451 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_basic": { 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_basic", 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.452 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_crash": { 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_crash", 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.453 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_device": { 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_device", 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.454 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_ident": { 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_ident", 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.455 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_perf": { 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_perf", 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-15T23:40:57.456 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "contact": { 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "contact", 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.457 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "description": { 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "description", 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.458 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "device_url": { 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_url", 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.459 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.460 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "24", 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "8", 2024-09-15T23:40:57.461 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "last_opt_revision": { 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "last_opt_revision", 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1", 2024-09-15T23:40:57.462 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard": { 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard", 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.463 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard_description": { 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard_description", 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.464 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.465 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.466 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.467 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.468 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.470 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "organization": { 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "organization", 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.471 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "proxy": { 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "proxy", 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.472 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "url": { 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url", 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.473 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "test_orchestrator", 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.474 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.475 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.476 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.477 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.478 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.479 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "volumes", 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.480 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.481 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.482 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.483 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.484 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_clones": { 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_clones", 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4", 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.486 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "periodic_async_work": { 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "periodic_async_work", 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Periodically check for async work", 2024-09-15T23:40:57.487 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "snapshot_clone_delay": { 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snapshot_clone_delay", 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.488 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "snapshot_clone_no_wait": { 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snapshot_clone_no_wait", 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.489 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix", 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.490 INFO:teuthology.orchestra.run.smithi089.stdout: "discovery_interval": { 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "discovery_interval", 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "100", 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "identifier": { 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "identifier", 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.492 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.493 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.494 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.495 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.496 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.498 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_host": { 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_host", 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.499 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_port": { 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_port", 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10051", 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.500 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_sender": { 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_sender", 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.501 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: "modules": [ 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm", 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: "dashboard", 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: "iostat", 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: "nfs", 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: "restful" 2024-09-15T23:40:57.502 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "available_modules": [ 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "alerts", 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.503 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How frequently to reexamine health status", 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.504 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.505 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.506 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.507 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.508 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.509 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_destination": { 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_destination", 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email address to send alerts to", 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_from_name": { 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_from_name", 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "Ceph", 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Email From: name", 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.511 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_host": { 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_host", 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP server", 2024-09-15T23:40:57.512 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_password": { 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_password", 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Password to authenticate with", 2024-09-15T23:40:57.513 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_port": { 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_port", 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "465", 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP port", 2024-09-15T23:40:57.514 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_sender": { 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_sender", 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.515 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SMTP envelope sender", 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_ssl": { 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_ssl", 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.516 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "smtp_user": { 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "smtp_user", 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.517 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "User to authenticate as", 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "balancer", 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.518 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "active": { 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "active", 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically balance PGs across cluster", 2024-09-15T23:40:57.519 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_time": { 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_time", 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0000", 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.520 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "beginning time of day to automatically balance", 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "begin_weekday": { 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "begin_weekday", 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2024-09-15T23:40:57.521 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_max_iterations": { 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_max_iterations", 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "25", 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:57.522 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "250", 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_metrics": { 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_metrics", 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "pgs,objects,bytes", 2024-09-15T23:40:57.523 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_compat_step": { 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crush_compat_step", 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.524 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.5", 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0.001", 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "0.999", 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "aggressiveness of optimization", 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "end_time": { 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_time", 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.525 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2359", 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ending time of day to automatically balance", 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "end_weekday": { 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "end_weekday", 2024-09-15T23:40:57.526 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "0", 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "6", 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.527 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.528 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.529 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.530 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.532 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "min_score": { 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_score", 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.533 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "upmap", 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.534 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-compat", 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: "none", 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: "read", 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap", 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap-read" 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Balancer mode", 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_ids": { 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_ids", 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.536 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-15T23:40:57.537 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_deviation": { 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_deviation", 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.538 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "upmap_max_optimizations": { 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "upmap_max_optimizations", 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.539 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephadm", 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.540 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_down_multiplier": { 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_down_multiplier", 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.541 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_refresh_rate": { 2024-09-15T23:40:57.542 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_refresh_rate", 2024-09-15T23:40:57.542 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.542 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.542 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.542 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "20", 2024-09-15T23:40:57.542 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.542 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.543 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.543 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "agent_starting_port": { 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "agent_starting_port", 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4721", 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.544 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_ptrace": { 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_ptrace", 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.545 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_interval": { 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_interval", 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.546 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "autotune_memory_target_ratio": { 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "autotune_memory_target_ratio", 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0.7", 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.547 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm_log_destination": { 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cephadm_log_destination", 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.548 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "file", 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "file,syslog", 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "syslog" 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "cgroups_split": { 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cgroups_split", 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.549 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "config_checks_enabled": { 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_checks_enabled", 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.550 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "config_dashboard": { 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_dashboard", 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.551 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_alertmanager": { 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_alertmanager", 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.552 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_base": { 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_base", 2024-09-15T23:40:57.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Container image name, without the tag", 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_elasticsearch": { 2024-09-15T23:40:57.554 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_elasticsearch", 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "elasticsearch container image", 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.555 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_grafana": { 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_grafana", 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.556 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_haproxy": { 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_haproxy", 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "HAproxy container image", 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.557 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_agent": { 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_agent", 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger agent container image", 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.558 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_collector": { 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_collector", 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger collector container image", 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.559 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_jaeger_query": { 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_jaeger_query", 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Jaeger query container image", 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.560 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.561 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.561 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_keepalived": { 2024-09-15T23:40:57.561 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_keepalived", 2024-09-15T23:40:57.561 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.561 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.561 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.561 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-15T23:40:57.561 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Keepalived container image", 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_loki": { 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_loki", 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-15T23:40:57.562 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.563 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.563 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.563 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Loki container image", 2024-09-15T23:40:57.563 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.563 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.563 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.563 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_node_exporter": { 2024-09-15T23:40:57.563 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_node_exporter", 2024-09-15T23:40:57.563 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_nvmeof": { 2024-09-15T23:40:57.564 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_nvmeof", 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Nvme-of container image", 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.565 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_prometheus": { 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_prometheus", 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Prometheus container image", 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.566 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_promtail": { 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_promtail", 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.567 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Promtail container image", 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_samba": { 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_samba", 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.568 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Samba/SMB container image", 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image_snmp_gateway": { 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_image_snmp_gateway", 2024-09-15T23:40:57.569 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "SNMP Gateway container image", 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.571 INFO:teuthology.orchestra.run.smithi089.stdout: "container_init": { 2024-09-15T23:40:57.571 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "container_init", 2024-09-15T23:40:57.571 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.571 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.571 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.571 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.571 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.571 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.571 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Run podman/docker with `--init`", 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "daemon_cache_timeout": { 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "daemon_cache_timeout", 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.572 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "default_cephadm_command_timeout": { 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_cephadm_command_timeout", 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "900", 2024-09-15T23:40:57.573 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "default_registry": { 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "default_registry", 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.574 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.575 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "docker.io", 2024-09-15T23:40:57.575 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.575 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.575 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.575 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-15T23:40:57.575 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.575 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.575 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.575 INFO:teuthology.orchestra.run.smithi089.stdout: "device_cache_timeout": { 2024-09-15T23:40:57.576 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_cache_timeout", 2024-09-15T23:40:57.576 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.576 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.576 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.576 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1800", 2024-09-15T23:40:57.576 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.576 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.576 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.576 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache device inventory", 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "device_enhanced_scan": { 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_enhanced_scan", 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.577 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "facts_cache_timeout": { 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "facts_cache_timeout", 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.578 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "seconds to cache host facts data", 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "grafana_dashboards_path": { 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "grafana_dashboards_path", 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-15T23:40:57.579 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "host_check_interval": { 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "host_check_interval", 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.580 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to perform a host check", 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "hw_monitoring": { 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "hw_monitoring", 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.581 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "inventory_list_all": { 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "inventory_list_all", 2024-09-15T23:40:57.582 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.583 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.584 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "log_refresh_metadata": { 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_refresh_metadata", 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.585 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.586 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.587 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.588 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.589 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.590 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "*", 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.591 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "max_count_per_host": { 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_count_per_host", 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.592 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of daemons per service per host", 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd_draining_count": { 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_osd_draining_count", 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.593 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "migration_current": { 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "migration_current", 2024-09-15T23:40:57.594 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "internal - do not modify", 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "mode": { 2024-09-15T23:40:57.595 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mode", 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "root", 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm-package", 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "root" 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "mode for remote execution of cephadm", 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.596 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "oob_default_addr": { 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "oob_default_addr", 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "169.254.1.1", 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.597 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.598 INFO:teuthology.orchestra.run.smithi089.stdout: "prometheus_alerts_path": { 2024-09-15T23:40:57.598 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus_alerts_path", 2024-09-15T23:40:57.598 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.598 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.598 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.598 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-15T23:40:57.598 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.598 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.598 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_insecure": { 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_insecure", 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.599 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.600 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.600 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-15T23:40:57.600 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.600 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.600 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.600 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_password": { 2024-09-15T23:40:57.600 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_password", 2024-09-15T23:40:57.600 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.600 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_url": { 2024-09-15T23:40:57.601 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_url", 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.602 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "registry_username": { 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "registry_username", 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-15T23:40:57.603 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: "secure_monitoring_stack": { 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "secure_monitoring_stack", 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.604 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "service_discovery_port": { 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "service_discovery_port", 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.605 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8765", 2024-09-15T23:40:57.606 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.606 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.606 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.606 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "cephadm service discovery port", 2024-09-15T23:40:57.606 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.606 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.606 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.606 INFO:teuthology.orchestra.run.smithi089.stdout: "ssh_config_file": { 2024-09-15T23:40:57.606 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssh_config_file", 2024-09-15T23:40:57.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.607 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.607 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.607 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.607 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.607 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.607 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-15T23:40:57.607 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.607 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.608 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.608 INFO:teuthology.orchestra.run.smithi089.stdout: "use_agent": { 2024-09-15T23:40:57.608 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_agent", 2024-09-15T23:40:57.608 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.608 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.608 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.608 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.608 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.609 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.609 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.609 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-15T23:40:57.609 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.609 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.609 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.609 INFO:teuthology.orchestra.run.smithi089.stdout: "use_repo_digest": { 2024-09-15T23:40:57.609 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "use_repo_digest", 2024-09-15T23:40:57.609 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.610 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.610 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.610 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.610 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.610 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.610 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.610 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-15T23:40:57.610 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.610 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.610 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_failed_host_check": { 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_failed_host_check", 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if the host check fails", 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.611 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_daemons": { 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_daemons", 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.612 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_on_stray_hosts": { 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_on_stray_hosts", 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.613 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crash", 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.614 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.615 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.616 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.617 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.618 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "retain_interval": { 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retain_interval", 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.619 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "31536000", 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_recent_interval": { 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_recent_interval", 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.620 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1209600", 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dashboard", 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.622 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_HOST": { 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.623 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.624 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_ENABLED": { 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_ENABLED", 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.625 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.626 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "ENABLE_BROWSABLE_API": { 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.627 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.628 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.629 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.630 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-15T23:40:57.631 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-15T23:40:57.632 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-15T23:40:57.633 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.634 INFO:teuthology.orchestra.run.smithi089.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.636 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_PASSWORD": { 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.637 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.638 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.638 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.638 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.638 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-15T23:40:57.638 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-15T23:40:57.638 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.638 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.638 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.638 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_URL": { 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_URL", 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.639 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_API_USERNAME": { 2024-09-15T23:40:57.640 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.641 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.642 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.643 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.643 INFO:teuthology.orchestra.run.smithi089.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-15T23:40:57.643 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-15T23:40:57.643 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.643 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.643 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.643 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.643 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.643 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.644 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-15T23:40:57.645 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_HOST": { 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.647 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.648 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.649 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-15T23:40:57.650 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.651 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.652 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.653 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.654 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.655 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.656 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_ENABLED": { 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.657 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.658 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.659 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-15T23:40:57.660 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8", 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-15T23:40:57.661 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "45", 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ACCESS_KEY": { 2024-09-15T23:40:57.662 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-15T23:40:57.663 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "admin", 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SECRET_KEY": { 2024-09-15T23:40:57.664 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.665 INFO:teuthology.orchestra.run.smithi089.stdout: "RGW_API_SSL_VERIFY": { 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.666 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.668 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.669 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "cross_origin_url": { 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cross_origin_url", 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.670 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.671 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.671 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.671 INFO:teuthology.orchestra.run.smithi089.stdout: "crt_file": { 2024-09-15T23:40:57.671 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "crt_file", 2024-09-15T23:40:57.671 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.671 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.671 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.671 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.671 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "debug": { 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "debug", 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.672 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Enable/disable debug options", 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "jwt_token_ttl": { 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "jwt_token_ttl", 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.673 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.674 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.674 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "28800", 2024-09-15T23:40:57.674 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.674 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.674 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.674 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.674 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.674 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.674 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.675 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.676 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.676 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.676 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.676 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.676 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.676 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.676 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.676 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.676 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.677 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.678 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.679 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.679 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.679 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.679 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.679 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.679 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.679 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.679 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.679 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.681 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "motd": { 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "motd", 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "The message of the day", 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.682 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect_resolve_ip_addr": { 2024-09-15T23:40:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "redirect_resolve_ip_addr", 2024-09-15T23:40:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.683 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.684 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2024-09-15T23:40:57.685 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.685 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.685 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.685 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.685 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.685 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.685 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.685 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2024-09-15T23:40:57.686 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2024-09-15T23:40:57.686 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.686 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.686 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.686 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8080", 2024-09-15T23:40:57.686 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.686 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.686 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.686 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.686 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.687 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl_server_port": { 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl_server_port", 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8443", 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.688 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "redirect", 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.689 INFO:teuthology.orchestra.run.smithi089.stdout: "redirect" 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2024-09-15T23:40:57.690 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "url_prefix": { 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url_prefix", 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.691 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "devicehealth", 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.692 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_monitoring": { 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_monitoring", 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "monitor device health metrics", 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.693 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.694 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.695 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.696 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "mark_out_threshold": { 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mark_out_threshold", 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "2419200", 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.699 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": { 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pool_name", 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "device_health_metrics", 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "retention_period": { 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "retention_period", 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15552000", 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long to retain device health metrics", 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.701 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_frequency": { 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_frequency", 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.702 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "self_heal": { 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "self_heal", 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.703 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how frequently to wake up and check device health", 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.704 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "warn_threshold": { 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "warn_threshold", 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7257600", 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.705 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "diskprediction_local", 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.706 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.707 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.708 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.709 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.709 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.709 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.709 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.709 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.709 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.709 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.709 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "predict_interval": { 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predict_interval", 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "86400", 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "predictor_model": { 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "predictor_model", 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "prophetstor", 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.713 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "600", 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.714 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.715 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.715 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.715 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.715 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "influx", 2024-09-15T23:40:57.715 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": false, 2024-09-15T23:40:57.715 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "influxdb python module not found", 2024-09-15T23:40:57.716 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.716 INFO:teuthology.orchestra.run.smithi089.stdout: "batch_size": { 2024-09-15T23:40:57.716 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "batch_size", 2024-09-15T23:40:57.716 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.716 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.716 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.716 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5000", 2024-09-15T23:40:57.716 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.716 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.717 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.717 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-15T23:40:57.717 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.717 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.717 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.717 INFO:teuthology.orchestra.run.smithi089.stdout: "database": { 2024-09-15T23:40:57.717 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "database", 2024-09-15T23:40:57.717 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.717 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "ceph", 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "hostname": { 2024-09-15T23:40:57.718 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "hostname", 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server hostname", 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.719 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "30", 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "5", 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.721 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.722 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.723 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.724 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.725 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "password": { 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "password", 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.726 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "password of InfluxDB server user", 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "port": { 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "port", 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.727 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "8086", 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "InfluxDB server port", 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "ssl": { 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ssl", 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.728 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "false", 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "threads": { 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threads", 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.729 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "32", 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "username": { 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "username", 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.730 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "username of InfluxDB server user", 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "verify_ssl": { 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "verify_ssl", 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.731 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "true", 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "insights", 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.733 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.734 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.735 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.736 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.737 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "iostat", 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.738 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.739 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.740 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.740 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.740 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.740 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.740 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.740 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.740 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.740 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.740 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.742 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.743 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.744 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "k8sevents", 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_event_retention_days": { 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "ceph_event_retention_days", 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "7", 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.745 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.746 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.746 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-15T23:40:57.746 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.747 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: "config_check_secs": { 2024-09-15T23:40:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "config_check_secs", 2024-09-15T23:40:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "10", 2024-09-15T23:40:57.748 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.749 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.750 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.751 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.752 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.752 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.752 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.752 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.752 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.752 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.752 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.752 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.752 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.753 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.754 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.756 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "localpool", 2024-09-15T23:40:57.756 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.756 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.756 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.756 INFO:teuthology.orchestra.run.smithi089.stdout: "failure_domain": { 2024-09-15T23:40:57.756 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "failure_domain", 2024-09-15T23:40:57.756 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.756 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.756 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "host", 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "failure domain for any created local pool", 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.757 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.758 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.759 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.760 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.761 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.762 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": { 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "min_size", 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default min_size for any created local pool", 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.763 INFO:teuthology.orchestra.run.smithi089.stdout: "num_rep": { 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "num_rep", 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3", 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default replica count for any created local pool", 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.764 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": { 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_num", 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "128", 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "default pg_num for any created local pool", 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.765 INFO:teuthology.orchestra.run.smithi089.stdout: "prefix": { 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prefix", 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "name prefix for any created local pool", 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "subtree": { 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "subtree", 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "rack", 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.767 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mds_autoscaler", 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.768 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.769 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirroring", 2024-09-15T23:40:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "nfs", 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "fail_fs": { 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "fail_fs", 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator": { 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "orchestrator", 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm", 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "rook", 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "test_orchestrator" 2024-09-15T23:40:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Orchestrator backend", 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_perf_query", 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "osd_support", 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "pg_autoscaler", 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "threshold": { 2024-09-15T23:40:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "threshold", 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "3.0", 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1.0", 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "scaling threshold", 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "progress", 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_pg_recovery_event": { 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_pg_recovery_event", 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "max_completed_events": { 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_completed_events", 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "50", 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "number of past completed events to remember", 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "sleep_interval": { 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "sleep_interval", 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "how long the module is going to sleep", 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "prometheus", 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "cache": { 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "cache", 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "exclude_perf_counters": { 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "exclude_perf_counters", 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools": { 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools", 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "300", 2024-09-15T23:40:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "scrape_interval": { 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "scrape_interval", 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15.0", 2024-09-15T23:40:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "::", 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "9283", 2024-09-15T23:40:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "stale_cache_strategy": { 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stale_cache_strategy", 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "log", 2024-09-15T23:40:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_behaviour": { 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_behaviour", 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "default", 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "default", 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "error" 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "standby_error_status_code": { 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "standby_error_status_code", 2024-09-15T23:40:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "500", 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "400", 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "599", 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_support", 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_snap_create": { 2024-09-15T23:40:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_snap_create", 2024-09-15T23:40:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10", 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "mirror_snapshot_schedule": { 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mirror_snapshot_schedule", 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "trash_purge_schedule": { 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "trash_purge_schedule", 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "restful", 2024-09-15T23:40:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_auth": { 2024-09-15T23:40:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enable_auth", 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "key_file": { 2024-09-15T23:40:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "key_file", 2024-09-15T23:40:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "server_addr": { 2024-09-15T23:40:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_addr", 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "server_port": { 2024-09-15T23:40:57.858 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "server_port", 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rgw", 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.860 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.861 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.862 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.863 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.864 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "secondary_zone_period_retry_limit": { 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "5", 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.865 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rook", 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.866 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.867 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.868 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.869 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.870 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "storage_class": { 2024-09-15T23:40:57.871 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "storage_class", 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "local", 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "selftest", 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.873 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.874 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.875 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.876 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.877 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "roption1": { 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption1", 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.878 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "roption2": { 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "roption2", 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "xyz", 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.879 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption1": { 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption1", 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.880 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption2": { 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption2", 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.881 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption3": { 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption3", 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "float", 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.882 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption4": { 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption4", 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.883 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption5": { 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption5", 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.884 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption6": { 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption6", 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.885 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "rwoption7": { 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rwoption7", 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "1", 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "42", 2024-09-15T23:40:57.886 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "testkey": { 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testkey", 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.887 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "testlkey": { 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testlkey", 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.888 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "testnewline": { 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "testnewline", 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.889 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snap_schedule", 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_m_granularity": { 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "allow_m_granularity", 2024-09-15T23:40:57.890 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "allow minute scheduled snapshots", 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "dump_on_update": { 2024-09-15T23:40:57.891 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "dump_on_update", 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "dump database to debug log on update", 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.892 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.893 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.894 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.895 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.896 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.897 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "stats", 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.898 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.899 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.900 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.901 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.902 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "status", 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.903 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.904 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.905 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.906 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.907 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.908 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telegraf", 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "address": { 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "address", 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-15T23:40:57.909 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "15", 2024-09-15T23:40:57.910 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.911 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.912 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.913 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.914 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.915 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.916 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "telemetry", 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_basic": { 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_basic", 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-15T23:40:57.917 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_crash": { 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_crash", 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.918 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_device": { 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_device", 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.919 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_ident": { 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_ident", 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.920 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "channel_perf": { 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "channel_perf", 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.921 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "contact": { 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "contact", 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.922 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "description": { 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "description", 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.923 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "device_url": { 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "device_url", 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.924 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "enabled": { 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "enabled", 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.925 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "24", 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "8", 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.926 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "last_opt_revision": { 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "last_opt_revision", 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "1", 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.927 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard": { 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard", 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.928 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "leaderboard_description": { 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "leaderboard_description", 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.929 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.930 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.931 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.932 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.933 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.934 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "organization": { 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "organization", 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.935 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "proxy": { 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "proxy", 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.936 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "url": { 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "url", 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.937 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "test_orchestrator", 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.938 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.939 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.940 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.941 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.942 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.943 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "volumes", 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.944 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.945 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.946 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.947 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.948 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "max_concurrent_clones": { 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "max_concurrent_clones", 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "4", 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.949 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.950 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-15T23:40:57.950 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.950 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.950 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.950 INFO:teuthology.orchestra.run.smithi089.stdout: "periodic_async_work": { 2024-09-15T23:40:57.950 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "periodic_async_work", 2024-09-15T23:40:57.950 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.951 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Periodically check for async work", 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "snapshot_clone_delay": { 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snapshot_clone_delay", 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.952 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "0", 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "snapshot_clone_no_wait": { 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "snapshot_clone_no_wait", 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.953 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "True", 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.954 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix", 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "can_run": true, 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "error_string": "", 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "module_options": { 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "discovery_interval": { 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "discovery_interval", 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "uint", 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "100", 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.955 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "identifier": { 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "identifier", 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.956 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "interval": { 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "interval", 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "secs", 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "60", 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.957 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "log_level": { 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_level", 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.958 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster": { 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster", 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.959 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_cluster_level": { 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_cluster_level", 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.960 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "info", 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [ 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "", 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "critical", 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "debug", 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "error", 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "info", 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "warning" 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.961 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": { 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "log_to_file", 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "bool", 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "False", 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.962 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_host": { 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_host", 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "", 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.963 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_port": { 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_port", 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "int", 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "10051", 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.964 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "zabbix_sender": { 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "zabbix_sender", 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "str", 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "level": "advanced", 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 0, 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "min": "", 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "max": "", 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "enum_allowed": [], 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "desc": "", 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "long_desc": "", 2024-09-15T23:40:57.965 INFO:teuthology.orchestra.run.smithi089.stdout: "tags": [], 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: "see_also": [] 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: "services": { 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: "dashboard": "https://172.21.15.89:8443/" 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: "always_on_modules": { 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: "octopus": [ 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: "balancer", 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: "crash", 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: "devicehealth", 2024-09-15T23:40:57.966 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscaler", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "progress", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_support", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "status", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "telemetry", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "volumes" 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "pacific": [ 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "balancer", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "crash", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "devicehealth", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscaler", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "progress", 2024-09-15T23:40:57.967 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_support", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "status", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "telemetry", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "volumes" 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "quincy": [ 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "balancer", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "crash", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "devicehealth", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscaler", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "progress", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_support", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "status", 2024-09-15T23:40:57.968 INFO:teuthology.orchestra.run.smithi089.stdout: "telemetry", 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "volumes" 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "reef": [ 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "balancer", 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "crash", 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "devicehealth", 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator", 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscaler", 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "progress", 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_support", 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "status", 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "telemetry", 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: "volumes" 2024-09-15T23:40:57.969 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "squid": [ 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "balancer", 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "crash", 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "devicehealth", 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "orchestrator", 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscaler", 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "progress", 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_support", 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "status", 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "telemetry", 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "volumes" 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.970 INFO:teuthology.orchestra.run.smithi089.stdout: "last_failure_osd_epoch": 3, 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: "active_clients": [ 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "devicehealth", 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:0", 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3695758091 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "libcephsqlite", 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-15T23:40:57.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:0", 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3157497713 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "rbd_support", 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:0", 2024-09-15T23:40:57.972 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1267831777 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "volumes", 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:0", 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 2878808738 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:57.973 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-15T23:40:57.974 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:40:57.975 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:56 smithi174 ceph-mon[35586]: pgmap v857: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 9 op/s 2024-09-15T23:40:57.975 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:56 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1173928885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:40:57.975 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:56 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3752919139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:40:57.975 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:56 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3816747801' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-15T23:40:57.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:56 smithi120 ceph-mon[35250]: pgmap v857: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 9 op/s 2024-09-15T23:40:57.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:56 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1173928885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:40:57.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:56 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3752919139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:40:57.976 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:56 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3816747801' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-15T23:40:57.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/3816747801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb500a3250 msgr2=0x7fbb500a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:57.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb500a3250 0x7fbb500a5710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbb54000c00 tx=0x7fbb54031400 comp rx=0 tx=0).stop 2024-09-15T23:40:57.976 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/3816747801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb641036e0 msgr2=0x7fbb6419f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:40:57.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb641036e0 0x7fbb6419f410 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c00eb10 tx=0x7fbb5c00c2d0 comp rx=0 tx=0).stop 2024-09-15T23:40:57.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/3816747801 shutdown_connections 2024-09-15T23:40:57.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fbb500a3250 0x7fbb500a5710 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:57.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb641036e0 0x7fbb6419f410 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:57.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb64102d20 0x7fbb641983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:57.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 --2- 172.21.15.89:0/3816747801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb64101b30 0x7fbb64197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:40:57.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/3816747801 >> 172.21.15.89:0/3816747801 conn(0x7fbb640fd2b0 msgr2=0x7fbb64106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:40:57.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/3816747801 shutdown_connections 2024-09-15T23:40:57.977 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:40:56.583+0000 7fbb6c5c4640 1 -- 172.21.15.89:0/3816747801 wait complete. 2024-09-15T23:40:58.043 DEBUG:tasks.check_counter:Getting stats from a 2024-09-15T23:40:58.043 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.a perf dump 2024-09-15T23:40:58.044 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:56 smithi089 ceph-mon[30933]: pgmap v857: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 9 op/s 2024-09-15T23:40:58.044 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:56 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1173928885' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:40:58.044 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:56 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3752919139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:40:58.044 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:56 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3816747801' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-15T23:40:58.301 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:40:58.789 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:40:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:40:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 2203, 2024-09-15T23:40:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 2205, 2024-09-15T23:40:58.789 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 659469, 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 700555, 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 6, 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 2, 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.195651946, 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.066481225, 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.051653020, 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.060490461, 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2203, 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.070407064, 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000031959 2024-09-15T23:40:58.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 983, 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 1215, 2024-09-15T23:40:58.791 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 244472, 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 796272, 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 6, 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 4, 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.113623031, 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.039019828, 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.026824291, 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.038037538, 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 1214, 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.048825921, 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000040219 2024-09-15T23:40:58.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 485200 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 325, 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 319, 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 809036, 2024-09-15T23:40:58.793 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 192494, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 2, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.045304500, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.022766159, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.027748523, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000021116, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 318, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.018346329, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000057692 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.794 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 808768, 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 192176 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 1, 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-MDSRank": { 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-09-15T23:40:58.795 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 5011, 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.134168708, 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000026774 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-PurgeQueue": { 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:58.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": { 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "request": 0, 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "reply": 0, 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "reply_latency": { 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_reply": 0, 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "forward": 0, 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_complete": 0, 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_keys": 0, 2024-09-15T23:40:58.797 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_commit": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_split": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_merge": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_top": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_bottom": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pin_tail": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pinned": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_expired": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_with_caps": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "subtrees": 1, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse": 0, 2024-09-15T23:40:58.798 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_hit": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_forward": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_discover": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_remote_ino": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_lock": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "load_cent": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "q": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "exported": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "exported_inodes": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "imported": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "imported_inodes": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_dir_fetch": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:40:58.799 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_peer_discover": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rfiles": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rbytes": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rsnaps": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_set_tag": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_file_inodes": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:40:58.800 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_cap_release": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "process_request_cap_release": 0 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache": { 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update_receipt": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_try_discover": 0, 2024-09-15T23:40:58.801 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_send_discover": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_handle_discover": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_delayed": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_created": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_enqueued": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_reintegrated": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_migrated": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_processing": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_started": 0, 2024-09-15T23:40:58.802 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_completed": 0, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_quiesce_path": 0, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_quiesce_inode": 0, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_exportdir": 0, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_flush": 0, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragstats": 0, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_inodestats": 0 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log": { 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "evlrg": 0, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "evadd": 0, 2024-09-15T23:40:58.803 INFO:teuthology.orchestra.run.smithi089.stdout: "evex": 0, 2024-09-15T23:40:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "evtrm": 0, 2024-09-15T23:40:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "ev": 3, 2024-09-15T23:40:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "evexg": 0, 2024-09-15T23:40:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "evexd": 0, 2024-09-15T23:40:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "segadd": 0, 2024-09-15T23:40:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "segex": 0, 2024-09-15T23:40:58.804 INFO:teuthology.orchestra.run.smithi089.stdout: "segtrm": 0, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "seg": 2, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "segmjr": 2, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "segexg": 0, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "segexd": 0, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "expos": 4194304, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "wrpos": 4195200, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "rdpos": 4194892, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "jlat": { 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.805 INFO:teuthology.orchestra.run.smithi089.stdout: "replayed": 3 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_mem": { 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "ino": 3, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "ino+": 3, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "ino-": 0, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "dir": 1, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "dir+": 1, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "dir-": 0, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "dn": 0, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "dn+": 0, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "dn-": 0, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "cap": 0, 2024-09-15T23:40:58.806 INFO:teuthology.orchestra.run.smithi089.stdout: "cap+": 0, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "cap-": 0, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "rss": 48760, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "heap": 215328 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_server": { 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_client_request": 0, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_server_request": 0, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_request": 0, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_session": 0, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_peer_request": 0, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "req_create_latency": { 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.807 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getattr_latency": { 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getfilelock_latency": { 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "req_link_latency": { 2024-09-15T23:40:58.808 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookup_latency": { 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookuphash_latency": { 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.809 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupino_latency": { 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupname_latency": { 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupparent_latency": { 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.810 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupsnap_latency": { 2024-09-15T23:40:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lssnap_latency": { 2024-09-15T23:40:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.811 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mkdir_latency": { 2024-09-15T23:40:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mknod_latency": { 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mksnap_latency": { 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "req_open_latency": { 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.813 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "req_readdir_latency": { 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rename_latency": { 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.814 INFO:teuthology.orchestra.run.smithi089.stdout: "req_renamesnap_latency": { 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "req_snapdiff_latency": { 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmdir_latency": { 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.815 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmsnap_latency": { 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmxattr_latency": { 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setattr_latency": { 2024-09-15T23:40:58.816 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setdirlayout_latency": { 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setfilelock_latency": { 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setlayout_latency": { 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setxattr_latency": { 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "req_symlink_latency": { 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.818 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "req_unlink_latency": { 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getvxattr_latency": { 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.819 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sessions": { 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "session_count": 1, 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "session_add": 1, 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "session_remove": 0, 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_open": 1, 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_stale": 0, 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "total_load": 0, 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "average_load": 0, 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "avg_session_uptime": 2022, 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "mdthresh_evicted": 0 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.820 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:40:58.821 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:40:58.822 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 190167, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 52, 2024-09-15T23:40:58.823 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 9144, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 172, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:40:58.824 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 8512, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 13, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter": { 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "op_active": 0, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "op_laggy": 0, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send": 3182, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send_bytes": 21, 2024-09-15T23:40:58.825 INFO:teuthology.orchestra.run.smithi089.stdout: "op_resend": 3, 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "op_reply": 3182, 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "op_latency": { 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3182, 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.962955222, 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000616893 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "op_inflight": 0, 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "oplen_avg": { 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3182, 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 3183 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "op": 3182, 2024-09-15T23:40:58.826 INFO:teuthology.orchestra.run.smithi089.stdout: "op_r": 3182, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "op_w": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "op_rmw": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "op_pg": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_stat": 2075, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_create": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_read": 1105, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_write": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writefull": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writesame": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_append": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_zero": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_truncate": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_delete": 0, 2024-09-15T23:40:58.827 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_mapext": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_sparse_read": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_clonerange": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_getxattr": 1, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_setxattr": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_rmxattr": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_call": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_watch": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_notify": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls": 0, 2024-09-15T23:40:58.828 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_other": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_active": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_send": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_resend": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_ping": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_active": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_send": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_resend": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_active": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_send": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_resend": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_active": 0, 2024-09-15T23:40:58.829 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_send": 0, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_resend": 0, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "command_active": 0, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "command_send": 0, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "command_resend": 0, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "map_epoch": 90, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "map_full": 0, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "map_inc": 4, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_sessions": 5, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_open": 7, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_close": 2, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_laggy": 0, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_wr": 0, 2024-09-15T23:40:58.830 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_rd": 2, 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_del": 0 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: "oft": { 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_objs": 0, 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_updates": 0, 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_removes": 0 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: "osd-slow-ops": { 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_ops_count": 0 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: "purge_queue": { 2024-09-15T23:40:58.831 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops": 0, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing": 0, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_high_water": 0, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed_ops": 0, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed": 0, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_item_in_journal": 0 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 3507, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 1377584, 2024-09-15T23:40:58.832 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 3507, 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 3507, 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 1377584, 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:58.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_bytes": { 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 3182, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 901, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 3182, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 901, 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:40:58.834 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_ops": { 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 1024, 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-09-15T23:40:58.835 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 3182, 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 3182, 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 3182, 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 3182, 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-09-15T23:40:58.836 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 0, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 0, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle-0x563d542f2b40": { 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 0, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 0, 2024-09-15T23:40:58.838 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:40:58.839 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:40:58.839 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:40:58.839 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:40:58.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:58.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:40:58.839 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:40:59.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:40:58 smithi089 ceph-mon[30933]: pgmap v858: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 12 op/s 2024-09-15T23:40:59.224 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:40:58 smithi120 ceph-mon[35250]: pgmap v858: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 12 op/s 2024-09-15T23:40:59.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:40:58 smithi174 ceph-mon[35586]: pgmap v858: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 12 op/s 2024-09-15T23:40:59.732 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-09-15T23:40:59.733 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-09-15T23:40:59.733 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-09-15T23:40:59.733 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-09-15T23:40:59.733 DEBUG:tasks.check_counter:Getting stats from d 2024-09-15T23:40:59.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.d perf dump 2024-09-15T23:40:59.980 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:00.509 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 323, 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 316, 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 788080, 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 194288, 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 0, 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.047361291, 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.022521814, 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.028972127, 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 316, 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.022570211, 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000071424 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 788080, 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 194288 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 2, 2024-09-15T23:41:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 240, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 320, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 199776, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 2, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.018161936, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.014327602, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.001555688, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 238, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.013458672, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000056549 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 199776 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 0, 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 2, 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 0, 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 0, 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 2, 2024-09-15T23:41:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 1, 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.000288876, 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.000000000, 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.000219526, 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 0, 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 148744, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 30, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 0, 2024-09-15T23:41:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 0, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 928, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 2, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:41:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 321, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 739913, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 321, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 321, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 739913, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:00.521 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:41:01.060 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:00 smithi089 ceph-mon[30933]: pgmap v859: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-15T23:41:01.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:00 smithi120 ceph-mon[35250]: pgmap v859: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-15T23:41:01.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:00 smithi174 ceph-mon[35586]: pgmap v859: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2024-09-15T23:41:01.252 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-15T23:41:01.252 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-15T23:41:01.252 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-09-15T23:41:01.252 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-09-15T23:41:01.252 DEBUG:tasks.check_counter:Getting stats from g 2024-09-15T23:41:01.253 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.g perf dump 2024-09-15T23:41:01.512 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:02.022 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:41:02.022 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:02.022 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 1105, 2024-09-15T23:41:02.022 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 1107, 2024-09-15T23:41:02.022 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 371225, 2024-09-15T23:41:02.022 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 351391, 2024-09-15T23:41:02.022 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 7, 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 2, 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.118842125, 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.053367376, 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.024345599, 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.029201987, 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 1105, 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.055860537, 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000050552 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:02.023 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 1432, 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 1426, 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1099704, 2024-09-15T23:41:02.024 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 545221, 2024-09-15T23:41:02.025 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 5, 2024-09-15T23:41:02.025 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 3, 2024-09-15T23:41:02.025 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.122692781, 2024-09-15T23:41:02.025 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.046850986, 2024-09-15T23:41:02.025 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.049698866, 2024-09-15T23:41:02.025 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.024745007, 2024-09-15T23:41:02.025 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:02.025 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 1425, 2024-09-15T23:41:02.025 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.043502050, 2024-09-15T23:41:02.025 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000030527 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 809312, 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 193456 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:02.026 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 9, 2024-09-15T23:41:02.027 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 242, 2024-09-15T23:41:02.027 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 4740, 2024-09-15T23:41:02.027 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 576357, 2024-09-15T23:41:02.027 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 8, 2024-09-15T23:41:02.027 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 4, 2024-09-15T23:41:02.027 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.019289212, 2024-09-15T23:41:02.027 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.012549668, 2024-09-15T23:41:02.027 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.004427108, 2024-09-15T23:41:02.028 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000083028, 2024-09-15T23:41:02.028 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:02.028 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 241, 2024-09-15T23:41:02.028 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.019311146, 2024-09-15T23:41:02.028 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000080129 2024-09-15T23:41:02.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.028 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:02.028 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.028 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.028 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.029 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-09-15T23:41:02.029 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 575392 2024-09-15T23:41:02.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.029 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-09-15T23:41:02.029 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 1, 2024-09-15T23:41:02.029 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-09-15T23:41:02.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.029 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-MDSRank": { 2024-09-15T23:41:02.029 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 4421, 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.104771610, 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000023698 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-PurgeQueue": { 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.030 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": { 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: "request": 0, 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: "reply": 0, 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: "reply_latency": { 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.031 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_reply": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "forward": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_complete": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_keys": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_commit": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_split": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_merge": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_top": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_bottom": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pin_tail": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pinned": 0, 2024-09-15T23:41:02.032 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_expired": 0, 2024-09-15T23:41:02.033 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_with_caps": 0, 2024-09-15T23:41:02.033 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": 0, 2024-09-15T23:41:02.033 INFO:teuthology.orchestra.run.smithi089.stdout: "subtrees": 1, 2024-09-15T23:41:02.033 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse": 0, 2024-09-15T23:41:02.033 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_hit": 0, 2024-09-15T23:41:02.033 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_forward": 0, 2024-09-15T23:41:02.033 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_discover": 0, 2024-09-15T23:41:02.033 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:41:02.033 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_remote_ino": 0, 2024-09-15T23:41:02.033 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_lock": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "load_cent": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "q": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "exported": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "exported_inodes": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "imported": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "imported_inodes": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_dir_fetch": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_peer_discover": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rfiles": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rbytes": 0, 2024-09-15T23:41:02.034 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rsnaps": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_set_tag": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_file_inodes": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T23:41:02.035 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:41:02.036 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:41:02.036 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:41:02.036 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:41:02.036 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps": 0, 2024-09-15T23:41:02.036 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:41:02.036 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_cap_release": 0, 2024-09-15T23:41:02.036 INFO:teuthology.orchestra.run.smithi089.stdout: "process_request_cap_release": 0 2024-09-15T23:41:02.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.036 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache": { 2024-09-15T23:41:02.036 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update_receipt": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_try_discover": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_send_discover": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_handle_discover": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_delayed": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_created": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_enqueued": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_reintegrated": 0, 2024-09-15T23:41:02.037 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_migrated": 0, 2024-09-15T23:41:02.038 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_processing": 0, 2024-09-15T23:41:02.038 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:41:02.038 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:41:02.038 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_started": 0, 2024-09-15T23:41:02.038 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_completed": 0, 2024-09-15T23:41:02.038 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_quiesce_path": 0, 2024-09-15T23:41:02.038 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_quiesce_inode": 0, 2024-09-15T23:41:02.038 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:41:02.038 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_exportdir": 0, 2024-09-15T23:41:02.038 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_flush": 0, 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragstats": 0, 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_inodestats": 0 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log": { 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: "evlrg": 0, 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: "evadd": 0, 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: "evex": 0, 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: "evtrm": 0, 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: "ev": 2, 2024-09-15T23:41:02.039 INFO:teuthology.orchestra.run.smithi089.stdout: "evexg": 0, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "evexd": 0, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "segadd": 0, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "segex": 0, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "segtrm": 0, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "seg": 2, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "segmjr": 2, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "segexg": 0, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "segexd": 0, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "expos": 4194304, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "wrpos": 4194892, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "rdpos": 4194352, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "jlat": { 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.040 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "replayed": 2 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_mem": { 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "ino": 3, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "ino+": 3, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "ino-": 0, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "dir": 1, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "dir+": 1, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "dir-": 0, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "dn": 0, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "dn+": 0, 2024-09-15T23:41:02.041 INFO:teuthology.orchestra.run.smithi089.stdout: "dn-": 0, 2024-09-15T23:41:02.042 INFO:teuthology.orchestra.run.smithi089.stdout: "cap": 0, 2024-09-15T23:41:02.042 INFO:teuthology.orchestra.run.smithi089.stdout: "cap+": 0, 2024-09-15T23:41:02.042 INFO:teuthology.orchestra.run.smithi089.stdout: "cap-": 0, 2024-09-15T23:41:02.042 INFO:teuthology.orchestra.run.smithi089.stdout: "rss": 48500, 2024-09-15T23:41:02.042 INFO:teuthology.orchestra.run.smithi089.stdout: "heap": 215328 2024-09-15T23:41:02.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.042 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_server": { 2024-09-15T23:41:02.042 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_client_request": 0, 2024-09-15T23:41:02.042 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_server_request": 0, 2024-09-15T23:41:02.042 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_request": 0, 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_session": 0, 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_peer_request": 0, 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: "req_create_latency": { 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getattr_latency": { 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getfilelock_latency": { 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "req_link_latency": { 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookup_latency": { 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.044 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.045 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.045 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookuphash_latency": { 2024-09-15T23:41:02.045 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.045 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.045 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.045 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupino_latency": { 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupname_latency": { 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupparent_latency": { 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.048 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupsnap_latency": { 2024-09-15T23:41:02.048 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.048 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.048 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.048 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lssnap_latency": { 2024-09-15T23:41:02.049 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.049 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.049 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.049 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mkdir_latency": { 2024-09-15T23:41:02.049 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.050 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.050 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.050 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mknod_latency": { 2024-09-15T23:41:02.050 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.050 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mksnap_latency": { 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: "req_open_latency": { 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.051 INFO:teuthology.orchestra.run.smithi089.stdout: "req_readdir_latency": { 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rename_latency": { 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: "req_renamesnap_latency": { 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.052 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: "req_snapdiff_latency": { 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmdir_latency": { 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.053 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmsnap_latency": { 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmxattr_latency": { 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setattr_latency": { 2024-09-15T23:41:02.054 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.055 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.055 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.055 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setdirlayout_latency": { 2024-09-15T23:41:02.055 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.055 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.055 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.055 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setfilelock_latency": { 2024-09-15T23:41:02.055 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setlayout_latency": { 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setxattr_latency": { 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.057 INFO:teuthology.orchestra.run.smithi089.stdout: "req_symlink_latency": { 2024-09-15T23:41:02.057 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.057 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.057 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.057 INFO:teuthology.orchestra.run.smithi089.stdout: "req_unlink_latency": { 2024-09-15T23:41:02.057 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.057 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.057 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getvxattr_latency": { 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sessions": { 2024-09-15T23:41:02.058 INFO:teuthology.orchestra.run.smithi089.stdout: "session_count": 0, 2024-09-15T23:41:02.059 INFO:teuthology.orchestra.run.smithi089.stdout: "session_add": 0, 2024-09-15T23:41:02.059 INFO:teuthology.orchestra.run.smithi089.stdout: "session_remove": 0, 2024-09-15T23:41:02.059 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_open": 0, 2024-09-15T23:41:02.059 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_stale": 0, 2024-09-15T23:41:02.059 INFO:teuthology.orchestra.run.smithi089.stdout: "total_load": 0, 2024-09-15T23:41:02.059 INFO:teuthology.orchestra.run.smithi089.stdout: "average_load": 0, 2024-09-15T23:41:02.059 INFO:teuthology.orchestra.run.smithi089.stdout: "avg_session_uptime": 0, 2024-09-15T23:41:02.059 INFO:teuthology.orchestra.run.smithi089.stdout: "mdthresh_evicted": 0 2024-09-15T23:41:02.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.059 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:02.060 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:02.061 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:02.061 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:02.061 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:02.061 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:02.061 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:02.061 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:02.061 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:02.061 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:02.061 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:02.061 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-09-15T23:41:02.062 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 198264, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 57, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:02.063 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:02.064 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:02.064 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 9144, 2024-09-15T23:41:02.064 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 172, 2024-09-15T23:41:02.064 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:02.064 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:02.064 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:02.064 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-09-15T23:41:02.064 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 8512, 2024-09-15T23:41:02.064 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 13, 2024-09-15T23:41:02.064 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter": { 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "op_active": 0, 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "op_laggy": 0, 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send": 2214, 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send_bytes": 21, 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "op_resend": 3, 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "op_reply": 2214, 2024-09-15T23:41:02.065 INFO:teuthology.orchestra.run.smithi089.stdout: "op_latency": { 2024-09-15T23:41:02.066 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2214, 2024-09-15T23:41:02.066 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.094975023, 2024-09-15T23:41:02.066 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000494568 2024-09-15T23:41:02.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.066 INFO:teuthology.orchestra.run.smithi089.stdout: "op_inflight": 0, 2024-09-15T23:41:02.066 INFO:teuthology.orchestra.run.smithi089.stdout: "oplen_avg": { 2024-09-15T23:41:02.066 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2214, 2024-09-15T23:41:02.066 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 2215 2024-09-15T23:41:02.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.066 INFO:teuthology.orchestra.run.smithi089.stdout: "op": 2214, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "op_r": 2214, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "op_w": 0, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "op_rmw": 0, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "op_pg": 0, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_stat": 1104, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_create": 0, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_read": 1108, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_write": 0, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writefull": 0, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writesame": 0, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_append": 0, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_zero": 0, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_truncate": 0, 2024-09-15T23:41:02.067 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_delete": 0, 2024-09-15T23:41:02.068 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_mapext": 0, 2024-09-15T23:41:02.068 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_sparse_read": 0, 2024-09-15T23:41:02.068 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_clonerange": 0, 2024-09-15T23:41:02.068 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_getxattr": 1, 2024-09-15T23:41:02.068 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_setxattr": 0, 2024-09-15T23:41:02.068 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:41:02.068 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_rmxattr": 0, 2024-09-15T23:41:02.068 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:41:02.068 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_call": 0, 2024-09-15T23:41:02.068 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_watch": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_notify": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_other": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_active": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_send": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_resend": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_ping": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_active": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_send": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_resend": 0, 2024-09-15T23:41:02.069 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_active": 0, 2024-09-15T23:41:02.070 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_send": 0, 2024-09-15T23:41:02.070 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_resend": 0, 2024-09-15T23:41:02.070 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_active": 0, 2024-09-15T23:41:02.070 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_send": 0, 2024-09-15T23:41:02.070 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_resend": 0, 2024-09-15T23:41:02.070 INFO:teuthology.orchestra.run.smithi089.stdout: "command_active": 0, 2024-09-15T23:41:02.070 INFO:teuthology.orchestra.run.smithi089.stdout: "command_send": 0, 2024-09-15T23:41:02.070 INFO:teuthology.orchestra.run.smithi089.stdout: "command_resend": 0, 2024-09-15T23:41:02.070 INFO:teuthology.orchestra.run.smithi089.stdout: "map_epoch": 90, 2024-09-15T23:41:02.070 INFO:teuthology.orchestra.run.smithi089.stdout: "map_full": 0, 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "map_inc": 4, 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_sessions": 6, 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_open": 10, 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_close": 4, 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_laggy": 0, 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_wr": 0, 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_rd": 2, 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_del": 0 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "oft": { 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_objs": 0, 2024-09-15T23:41:02.071 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:41:02.072 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_updates": 0, 2024-09-15T23:41:02.072 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_removes": 0 2024-09-15T23:41:02.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.072 INFO:teuthology.orchestra.run.smithi089.stdout: "osd-slow-ops": { 2024-09-15T23:41:02.072 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_ops_count": 0 2024-09-15T23:41:02.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.072 INFO:teuthology.orchestra.run.smithi089.stdout: "purge_queue": { 2024-09-15T23:41:02.072 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops": 0, 2024-09-15T23:41:02.072 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing": 0, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_high_water": 0, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed_ops": 0, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed": 0, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_item_in_journal": 0 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 2542, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 1226979, 2024-09-15T23:41:02.073 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 2542, 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 2542, 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 1226979, 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:02.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_bytes": { 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 2214, 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 593, 2024-09-15T23:41:02.075 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 2214, 2024-09-15T23:41:02.076 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 593, 2024-09-15T23:41:02.076 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:41:02.076 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.076 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.076 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:02.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.076 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_ops": { 2024-09-15T23:41:02.076 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:41:02.077 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 1024, 2024-09-15T23:41:02.077 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:41:02.077 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-09-15T23:41:02.077 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-09-15T23:41:02.077 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:02.077 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:02.077 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 2214, 2024-09-15T23:41:02.077 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 2214, 2024-09-15T23:41:02.077 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 2214, 2024-09-15T23:41:02.077 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 2214, 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:41:02.078 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-09-15T23:41:02.079 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-09-15T23:41:02.079 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:02.079 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:02.079 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-09-15T23:41:02.079 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-09-15T23:41:02.079 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 0, 2024-09-15T23:41:02.079 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 0, 2024-09-15T23:41:02.079 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:41:02.079 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.080 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.080 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.080 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle-0x555bf40fcb40": { 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 0, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 0, 2024-09-15T23:41:02.081 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:41:02.082 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:02.082 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:02.082 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:02.082 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:02.082 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:02.082 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:41:02.848 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-09-15T23:41:02.848 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-09-15T23:41:02.848 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-09-15T23:41:02.848 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-09-15T23:41:02.849 DEBUG:tasks.check_counter:Getting stats from j 2024-09-15T23:41:02.849 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.j perf dump 2024-09-15T23:41:03.094 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:03.215 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:02 smithi089 ceph-mon[30933]: pgmap v860: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-15T23:41:03.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:02 smithi120 ceph-mon[35250]: pgmap v860: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-15T23:41:03.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:02 smithi174 ceph-mon[35586]: pgmap v860: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-09-15T23:41:03.611 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-15T23:41:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 321, 2024-09-15T23:41:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 316, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 785552, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 195120, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 0, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.046849508, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.022870080, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.028551572, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 315, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.017078273, 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000054216 2024-09-15T23:41:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 785552, 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 195120 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 6, 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 244, 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 3392, 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 200576, 2024-09-15T23:41:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 4, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 2, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.021036468, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.015166223, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.003428043, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 243, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.018399657, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000075718 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 200576 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 0, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 2, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 0, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 0, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 2, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 1, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.000270402, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.000000000, 2024-09-15T23:41:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.000201756, 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:41:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 0, 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 148784, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 31, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 0, 2024-09-15T23:41:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 0, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 928, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 2, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 323, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 740173, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 323, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 323, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 740173, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-09-15T23:41:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-09-15T23:41:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-15T23:41:03.623 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-15T23:41:04.326 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-15T23:41:04.326 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-15T23:41:04.326 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-09-15T23:41:04.326 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-09-15T23:41:04.326 DEBUG:tasks.check_counter:Getting stats from b 2024-09-15T23:41:04.327 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.b perf dump 2024-09-15T23:41:04.586 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:41:04.951 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:04 smithi120 ceph-mon[35250]: pgmap v861: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-15T23:41:05.060 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-09-15T23:41:05.060 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:05.060 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 6281, 2024-09-15T23:41:05.060 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 8289, 2024-09-15T23:41:05.060 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 2686394, 2024-09-15T23:41:05.060 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 7113321, 2024-09-15T23:41:05.060 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 39, 2024-09-15T23:41:05.060 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 4, 2024-09-15T23:41:05.060 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.719366249, 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.264805679, 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.337162135, 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.115606588, 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 8289, 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.483401477, 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000058318 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 8156, 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.001805973, 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000221 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.061 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 1880288, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 1766176 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 1882, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 2739, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 582676, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 1478827, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 14, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 6, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.200640651, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.094795246, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.095804352, 2024-09-15T23:41:05.062 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.019541412, 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2737, 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.178051827, 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000065053 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3006, 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000751272, 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000249 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 594352 2024-09-15T23:41:05.063 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 5177, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 6355, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1308649, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 10728932, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 392, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 8, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.545298075, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.173228526, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.310681693, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.066373602, 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:05.064 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 6353, 2024-09-15T23:41:05.065 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.476000070, 2024-09-15T23:41:05.065 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000074925 2024-09-15T23:41:05.065 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.065 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:05.065 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 6431, 2024-09-15T23:41:05.065 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.001524465, 2024-09-15T23:41:05.065 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000237 2024-09-15T23:41:05.065 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 34144, 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 1856 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 6640, 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 1.728503644, 2024-09-15T23:41:05.066 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000260316 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 30, 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000928522, 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000030950 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.067 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 676, 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 676, 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 676, 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2.906339497, 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.004299318 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-09-15T23:41:05.068 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-09-15T23:41:05.069 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 492, 2024-09-15T23:41:05.069 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-09-15T23:41:05.069 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 24, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 5, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 19, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 16, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 1, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 1, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 2, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 764, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 754, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-09-15T23:41:05.070 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 0, 2024-09-15T23:41:05.071 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:41:05.071 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-09-15T23:41:05.071 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-09-15T23:41:05.071 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-09-15T23:41:05.071 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-09-15T23:41:05.071 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-09-15T23:41:05.071 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-09-15T23:41:05.071 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 357, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 4, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 234, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 0, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 357, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:41:05.072 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 230, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 26, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 204, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 127, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 646, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 671, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 599, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 697, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 659, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 4, 2024-09-15T23:41:05.073 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 628 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 23, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 0, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 48, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 3, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 3, 2024-09-15T23:41:05.074 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_path": 15, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_inode": 208, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 26, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-09-15T23:41:05.075 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 2479, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 2454, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 2454, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 25, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 42, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 76, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 75, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 75, 2024-09-15T23:41:05.076 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 1, 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 1, 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 1, 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 12124234, 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 12192412, 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 0, 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2462, 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 5.385803087, 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.002187572 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 0 2024-09-15T23:41:05.077 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 23, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 52, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 29, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 18, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 20, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 2, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 24, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 29, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 5, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 1, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 18, 2024-09-15T23:41:05.078 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 17, 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 56776, 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 231712 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 1899, 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 1, 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 676, 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 28, 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 2, 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 19, 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.012323132, 2024-09-15T23:41:05.079 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000648585 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 28, 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.006764456, 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000241587 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-09-15T23:41:05.080 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.081 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.081 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.081 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.081 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-09-15T23:41:05.081 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3, 2024-09-15T23:41:05.081 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.001241207, 2024-09-15T23:41:05.081 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000413735 2024-09-15T23:41:05.081 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.082 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-09-15T23:41:05.082 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.082 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.082 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.082 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.082 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-09-15T23:41:05.082 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.082 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.082 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.082 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.083 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-09-15T23:41:05.083 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.083 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.083 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.083 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.083 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-09-15T23:41:05.083 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.083 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.083 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.084 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.084 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-09-15T23:41:05.084 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.084 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.084 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.084 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.084 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-09-15T23:41:05.084 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 8, 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.012104849, 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001513106 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.085 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1, 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000224882, 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000224882 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.086 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.004552606, 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.002276303 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.002154241, 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001077120 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.087 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.012731948, 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.006365974 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.088 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 601, 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2.830976983, 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.004710444 2024-09-15T23:41:05.089 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.090 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 9, 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.020945105, 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.002327233 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.091 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-09-15T23:41:05.092 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1, 2024-09-15T23:41:05.092 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.002320088, 2024-09-15T23:41:05.092 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.002320088 2024-09-15T23:41:05.092 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.092 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:41:05.092 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:41:05.092 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-09-15T23:41:05.092 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 1, 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 4, 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 3, 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 1, 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 60, 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 60, 2024-09-15T23:41:05.093 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 1004, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:05.094 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:05.095 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:05.096 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 312583, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 114, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9144, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 172, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:05.097 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 97282, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 710, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 4916, 2024-09-15T23:41:05.098 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 8499853, 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 37, 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 4916, 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 4916, 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 27.003388304, 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.005492959 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 4916, 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 5943 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.099 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 4916, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 412, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 4504, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 472, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 30, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 29, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 2465, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 239, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-09-15T23:41:05.100 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 518, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 383, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 55, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-09-15T23:41:05.101 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 599, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-09-15T23:41:05.102 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 90, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 10, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 12, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 14, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 2, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-09-15T23:41:05.103 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 1139, 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 0, 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 14 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 1, 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 208 2024-09-15T23:41:05.104 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 2, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 1, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 4, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 3, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:05.105 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 13336, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 3237995, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 13336, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 13336, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 3237995, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.106 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 4916, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 8500164, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 4916, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 8500164, 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:05.107 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:05.108 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 4916, 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 4916, 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 4916, 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 4916, 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-09-15T23:41:05.109 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 3, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 311, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 3, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 3, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 311, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:05.110 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x5556a4c68960": { 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 2479, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 7998108, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 2479, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2462, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 7998108, 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:05.111 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:05.112 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:05.112 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:05.112 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:05.112 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:05.112 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-09-15T23:41:05.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:04 smithi174 ceph-mon[35586]: pgmap v861: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-15T23:41:05.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:04 smithi089 ceph-mon[30933]: pgmap v861: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-09-15T23:41:05.860 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-09-15T23:41:05.860 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-09-15T23:41:05.861 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=23 2024-09-15T23:41:05.861 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-09-15T23:41:05.861 DEBUG:tasks.check_counter:Getting stats from e 2024-09-15T23:41:05.861 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.e perf dump 2024-09-15T23:41:06.109 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:41:06.654 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-09-15T23:41:06.654 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:06.654 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 2103, 2024-09-15T23:41:06.654 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 2104, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 4822225, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 668446, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.190440130, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.075888505, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.046213258, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.049589869, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2102, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.083503674, 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000039725 2024-09-15T23:41:06.655 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 335, 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 329, 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 812216, 2024-09-15T23:41:06.656 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 197514, 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 3, 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.049856280, 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.021929310, 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.035936960, 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000086121, 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 328, 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.023665786, 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000072151 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:06.657 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.658 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.658 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.658 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.658 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 810560, 2024-09-15T23:41:06.658 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 196176 2024-09-15T23:41:06.658 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.658 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:06.658 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 1906, 2024-09-15T23:41:06.658 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 2143, 2024-09-15T23:41:06.659 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 4282497, 2024-09-15T23:41:06.659 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 1093831, 2024-09-15T23:41:06.659 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 5, 2024-09-15T23:41:06.659 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 4, 2024-09-15T23:41:06.659 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.162561285, 2024-09-15T23:41:06.659 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.048876329, 2024-09-15T23:41:06.659 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.046004230, 2024-09-15T23:41:06.659 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.054123782, 2024-09-15T23:41:06.659 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2142, 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.075008380, 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000035017 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 489312 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.660 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 6813, 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.155436453, 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000022814 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-09-15T23:41:06.661 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 0, 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 0, 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.662 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 0, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 17, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 4, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 13, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 10, 2024-09-15T23:41:06.663 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-09-15T23:41:06.664 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 0, 2024-09-15T23:41:06.664 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 0, 2024-09-15T23:41:06.664 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 2, 2024-09-15T23:41:06.664 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 0, 2024-09-15T23:41:06.664 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 0, 2024-09-15T23:41:06.664 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-09-15T23:41:06.664 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 0, 2024-09-15T23:41:06.664 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:41:06.664 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-09-15T23:41:06.665 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-09-15T23:41:06.665 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-09-15T23:41:06.665 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-09-15T23:41:06.665 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-09-15T23:41:06.665 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-09-15T23:41:06.666 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-09-15T23:41:06.666 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-09-15T23:41:06.666 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-09-15T23:41:06.666 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:41:06.666 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-09-15T23:41:06.666 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 4, 2024-09-15T23:41:06.666 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 234, 2024-09-15T23:41:06.666 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 0, 2024-09-15T23:41:06.666 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:41:06.666 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-09-15T23:41:06.667 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:41:06.667 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:41:06.667 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:41:06.667 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:41:06.667 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:41:06.667 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 0, 2024-09-15T23:41:06.667 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 0, 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 0, 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 0 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.668 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-09-15T23:41:06.669 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-09-15T23:41:06.669 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-09-15T23:41:06.669 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-09-15T23:41:06.669 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 0, 2024-09-15T23:41:06.669 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 0, 2024-09-15T23:41:06.669 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-09-15T23:41:06.669 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-09-15T23:41:06.669 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:41:06.669 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 0, 2024-09-15T23:41:06.670 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 0, 2024-09-15T23:41:06.670 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-09-15T23:41:06.670 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-09-15T23:41:06.670 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-09-15T23:41:06.670 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:41:06.670 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:41:06.670 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-09-15T23:41:06.670 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-09-15T23:41:06.670 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_path": 0, 2024-09-15T23:41:06.670 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_inode": 0, 2024-09-15T23:41:06.671 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:41:06.671 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-09-15T23:41:06.671 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-09-15T23:41:06.671 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:41:06.671 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-09-15T23:41:06.671 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-09-15T23:41:06.671 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.671 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-09-15T23:41:06.671 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-09-15T23:41:06.672 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 0, 2024-09-15T23:41:06.672 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-09-15T23:41:06.672 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-09-15T23:41:06.672 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 2479, 2024-09-15T23:41:06.672 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 0, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 2, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 75, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 12124234, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 12192412, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 12191813, 2024-09-15T23:41:06.673 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-09-15T23:41:06.674 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.674 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.674 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.674 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.674 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 2479 2024-09-15T23:41:06.674 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.674 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-09-15T23:41:06.674 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 14, 2024-09-15T23:41:06.674 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 17, 2024-09-15T23:41:06.674 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 3, 2024-09-15T23:41:06.675 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 9, 2024-09-15T23:41:06.675 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 11, 2024-09-15T23:41:06.675 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 2, 2024-09-15T23:41:06.675 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 17, 2024-09-15T23:41:06.675 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 19, 2024-09-15T23:41:06.675 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 2, 2024-09-15T23:41:06.675 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 0, 2024-09-15T23:41:06.676 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 0, 2024-09-15T23:41:06.676 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 0, 2024-09-15T23:41:06.676 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 49396, 2024-09-15T23:41:06.676 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 215328 2024-09-15T23:41:06.676 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.676 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-09-15T23:41:06.676 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 0, 2024-09-15T23:41:06.676 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 0, 2024-09-15T23:41:06.677 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 0, 2024-09-15T23:41:06.677 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 0, 2024-09-15T23:41:06.677 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 0, 2024-09-15T23:41:06.677 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-09-15T23:41:06.677 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.677 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.677 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.677 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.677 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-09-15T23:41:06.678 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.678 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.678 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.678 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.678 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-09-15T23:41:06.679 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.679 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.679 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.679 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.679 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-09-15T23:41:06.679 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.679 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.680 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-09-15T23:41:06.681 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.682 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.683 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-09-15T23:41:06.683 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.683 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.683 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.683 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.684 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.685 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.686 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.687 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.688 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.688 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-09-15T23:41:06.688 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.688 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.688 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.688 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.688 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-09-15T23:41:06.688 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.689 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-09-15T23:41:06.690 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.691 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.691 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.691 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:06.691 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.691 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-09-15T23:41:06.691 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 1, 2024-09-15T23:41:06.691 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 4, 2024-09-15T23:41:06.691 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 3, 2024-09-15T23:41:06.691 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 2, 2024-09-15T23:41:06.691 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 0, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 0, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 3510, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:06.692 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:06.693 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:06.693 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:06.693 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:06.693 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:06.693 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:06.693 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:06.693 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:06.693 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:06.694 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:06.694 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:06.694 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:06.694 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:06.694 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:06.694 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:06.694 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:06.694 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:06.695 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:06.695 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:06.695 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:06.695 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:06.695 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:06.695 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:06.695 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:06.695 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:06.695 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-09-15T23:41:06.695 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 199031, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 58, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:06.696 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9144, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 172, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 54978, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 261, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.697 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 4007, 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 26, 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 4, 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 4007, 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 4007, 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2.630940299, 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000656586 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-09-15T23:41:06.698 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 4007, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 4008 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 4007, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 4007, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 0, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 2214, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 0, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 1790, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 0, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 0, 2024-09-15T23:41:06.699 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-09-15T23:41:06.700 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-09-15T23:41:06.700 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-09-15T23:41:06.700 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-09-15T23:41:06.700 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 0, 2024-09-15T23:41:06.700 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-09-15T23:41:06.700 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-09-15T23:41:06.700 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-09-15T23:41:06.700 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 2, 2024-09-15T23:41:06.700 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 0, 2024-09-15T23:41:06.700 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-09-15T23:41:06.701 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-09-15T23:41:06.702 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-09-15T23:41:06.703 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-09-15T23:41:06.703 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 90, 2024-09-15T23:41:06.703 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-09-15T23:41:06.703 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 4, 2024-09-15T23:41:06.703 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 6, 2024-09-15T23:41:06.703 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 8, 2024-09-15T23:41:06.703 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 2, 2024-09-15T23:41:06.703 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 0, 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 2, 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 0 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 0, 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.704 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-09-15T23:41:06.705 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-09-15T23:41:06.705 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.705 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-09-15T23:41:06.705 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-09-15T23:41:06.705 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:41:06.705 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-09-15T23:41:06.705 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 0, 2024-09-15T23:41:06.705 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 0, 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 0, 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 4340, 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 9506217, 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:06.706 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 4340, 2024-09-15T23:41:06.707 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:06.707 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:06.707 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 4340, 2024-09-15T23:41:06.707 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 9506217, 2024-09-15T23:41:06.707 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:06.707 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.707 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.708 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.708 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:06.708 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.708 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-09-15T23:41:06.708 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:06.708 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-09-15T23:41:06.708 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 4007, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 7998118, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 4007, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 7998118, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.709 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.710 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:06.710 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.710 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-09-15T23:41:06.710 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:06.710 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-09-15T23:41:06.710 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:06.710 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:06.710 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:06.710 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:06.710 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:06.711 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 4007, 2024-09-15T23:41:06.711 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 4007, 2024-09-15T23:41:06.711 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 4007, 2024-09-15T23:41:06.711 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 4007, 2024-09-15T23:41:06.711 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:06.711 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.711 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.711 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.712 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:06.712 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.712 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:41:06.712 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:06.712 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-09-15T23:41:06.712 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:06.712 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:06.713 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:06.713 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:06.713 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:06.713 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:06.713 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:06.713 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-09-15T23:41:06.713 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-09-15T23:41:06.713 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:06.713 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.714 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.714 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.714 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:06.714 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:06.714 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x559139e98b40": { 2024-09-15T23:41:06.714 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:06.714 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-09-15T23:41:06.714 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:06.714 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:06.714 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:06.715 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:06.715 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:06.715 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:06.715 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:06.715 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-09-15T23:41:06.715 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-09-15T23:41:06.715 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:06.715 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:06.715 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:06.716 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:06.716 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:06.716 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:06.716 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-09-15T23:41:07.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:06 smithi120 ceph-mon[35250]: pgmap v862: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:41:07.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:06 smithi174 ceph-mon[35586]: pgmap v862: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:41:07.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:06 smithi089 ceph-mon[30933]: pgmap v862: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:41:07.405 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-09-15T23:41:07.405 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-09-15T23:41:07.405 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-09-15T23:41:07.405 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-09-15T23:41:07.406 DEBUG:tasks.check_counter:Getting stats from h 2024-09-15T23:41:07.406 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.h perf dump 2024-09-15T23:41:07.652 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:41:08.178 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-09-15T23:41:08.178 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:08.178 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 2251, 2024-09-15T23:41:08.178 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 1616, 2024-09-15T23:41:08.178 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1286877, 2024-09-15T23:41:08.178 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 598303, 2024-09-15T23:41:08.178 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 8, 2024-09-15T23:41:08.178 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-09-15T23:41:08.178 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.206068741, 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.094267759, 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.123979728, 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.005339787, 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1618, 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.090225464, 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000055763 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3001, 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000858844, 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000286 2024-09-15T23:41:08.179 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 831296, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 199824 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 68, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 306, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 11411, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 1242689, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 63, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 5, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.037087118, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.012264351, 2024-09-15T23:41:08.180 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.020592568, 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000195201, 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 304, 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.044065944, 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000144953 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 54, 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000004006, 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000074 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T23:41:08.181 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 578704 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 34, 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 38, 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 7632, 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 65212, 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 13, 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 5, 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.017337282, 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.004279565, 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.011982500, 2024-09-15T23:41:08.182 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000263118, 2024-09-15T23:41:08.184 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 34, 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.010815571, 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000318105 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 37, 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000006697, 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000181 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 3808, 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 4272 2024-09-15T23:41:08.185 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 39, 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.001454468, 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000037294 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.186 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000010794, 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000005397 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 2, 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 2, 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-09-15T23:41:08.187 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000209684, 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000104842 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 11, 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 16, 2024-09-15T23:41:08.188 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 0, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 16, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 13, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 4, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 2, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 2, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 3, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 42, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 36, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 6, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-09-15T23:41:08.189 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 5, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 266, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 0, 2024-09-15T23:41:08.190 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 2, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 2, 2024-09-15T23:41:08.191 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 0, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 0 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 18, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 12, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 12, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 0, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 0, 2024-09-15T23:41:08.192 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_path": 15, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_inode": 89, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-09-15T23:41:08.193 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:41:08.194 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-09-15T23:41:08.194 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-09-15T23:41:08.194 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.194 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-09-15T23:41:08.194 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-09-15T23:41:08.194 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 3, 2024-09-15T23:41:08.194 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-09-15T23:41:08.194 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-09-15T23:41:08.194 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 3, 2024-09-15T23:41:08.194 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 2, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 2, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 2, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 4194304, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 4195200, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 0, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-09-15T23:41:08.195 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.003556249, 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001778124 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 0 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 20, 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 22, 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 2, 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 19, 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 23, 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 4, 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 16, 2024-09-15T23:41:08.196 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 20, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 4, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 2, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 2, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 0, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 50700, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 223520 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 2, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 0, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 2, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 14, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 0, 2024-09-15T23:41:08.197 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.198 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.199 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000209684, 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000104842 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.200 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-09-15T23:41:08.201 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.202 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-09-15T23:41:08.203 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.203 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.203 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.203 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.203 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-09-15T23:41:08.203 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.203 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.203 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.204 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.204 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-09-15T23:41:08.204 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.204 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.204 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.205 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-09-15T23:41:08.206 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.207 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.208 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:08.209 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 1, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 1, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 0, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 1, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 2, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 2, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 985, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:08.210 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:08.211 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:08.212 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 238791, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 74, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-09-15T23:41:08.213 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9144, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 172, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 81693, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 330, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-09-15T23:41:08.214 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 38, 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 11319, 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 28, 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 38, 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 38, 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.191995682, 2024-09-15T23:41:08.215 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.005052517 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 38, 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 61 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 38, 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 0, 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 38, 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-09-15T23:41:08.216 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 0, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 10, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 0, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 2, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 6, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 6, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 0, 2024-09-15T23:41:08.217 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 20, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-09-15T23:41:08.218 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 90, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-09-15T23:41:08.219 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 4, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 8, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 13, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 5, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 16, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 0, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 1 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 1, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-09-15T23:41:08.220 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 97 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 0, 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 0, 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 0, 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:08.221 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 2349, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 1072337, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 2349, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2349, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 1072337, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.222 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 38, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 11319, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 38, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 11319, 2024-09-15T23:41:08.223 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:08.224 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 38, 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 38, 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 38, 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 38, 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-09-15T23:41:08.225 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:08.226 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x55f7077ec960": { 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 3, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 896, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 3, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 896, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:08.227 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:08.228 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:08.228 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:08.228 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:08.228 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-09-15T23:41:08.980 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-09-15T23:41:08.980 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-09-15T23:41:08.980 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-09-15T23:41:08.980 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=18 2024-09-15T23:41:08.980 DEBUG:tasks.check_counter:Getting stats from k 2024-09-15T23:41:08.980 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.k perf dump 2024-09-15T23:41:09.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:08 smithi120 ceph-mon[35250]: pgmap v863: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 12 op/s 2024-09-15T23:41:09.223 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:08 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:41:09.232 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:41:09.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:08 smithi174 ceph-mon[35586]: pgmap v863: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 12 op/s 2024-09-15T23:41:09.235 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:08 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:41:09.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:08 smithi089 ceph-mon[30933]: pgmap v863: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 12 op/s 2024-09-15T23:41:09.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:08 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:41:09.797 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-09-15T23:41:09.797 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:09.797 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 334, 2024-09-15T23:41:09.797 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 328, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 812050, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 198260, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 1, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.051557169, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.023225572, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.035136166, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000018601, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 327, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.020179365, 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000061710 2024-09-15T23:41:09.798 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 811216, 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 197936 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 1034, 2024-09-15T23:41:09.799 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 1273, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 256388, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 816891, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 5, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 4, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.113728963, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.041778995, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.024236791, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.035134953, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1272, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.067819544, 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000053317 2024-09-15T23:41:09.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 489664 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 2230, 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 2231, 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 668273, 2024-09-15T23:41:09.801 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 708880, 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 4, 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 3, 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.234619576, 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.085263711, 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.058246117, 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.070040596, 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2229, 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.096954758, 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000043496 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:09.802 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.803 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 5370, 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.159786186, 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000029755 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.804 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 0, 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 0, 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-09-15T23:41:09.805 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 0, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 1, 2024-09-15T23:41:09.806 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-09-15T23:41:09.807 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 0, 2024-09-15T23:41:09.808 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 0 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-09-15T23:41:09.809 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-09-15T23:41:09.810 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_path": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_inode": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-09-15T23:41:09.811 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 6, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 0, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 2, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 2, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 4194304, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 4195960, 2024-09-15T23:41:09.812 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 4195819, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 6 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 3, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 3, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 0, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 1, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 1, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 0, 2024-09-15T23:41:09.813 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 0, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 0, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 0, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 0, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 0, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 0, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 48568, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 215328 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 0, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 0, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 0, 2024-09-15T23:41:09.814 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 0, 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 0, 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-09-15T23:41:09.816 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.817 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.818 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-09-15T23:41:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-09-15T23:41:09.819 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-09-15T23:41:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.820 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-09-15T23:41:09.821 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.822 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.823 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-09-15T23:41:09.824 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.824 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.824 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.824 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.824 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-09-15T23:41:09.824 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.824 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.824 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.824 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-09-15T23:41:09.825 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.826 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.827 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.827 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-09-15T23:41:09.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.827 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.827 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.827 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-09-15T23:41:09.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.827 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.828 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:41:09.828 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:41:09.828 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-09-15T23:41:09.828 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.828 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.828 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.828 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:09.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.829 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-09-15T23:41:09.829 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 0, 2024-09-15T23:41:09.829 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 2, 2024-09-15T23:41:09.829 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 2, 2024-09-15T23:41:09.829 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 1, 2024-09-15T23:41:09.829 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-09-15T23:41:09.829 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 0, 2024-09-15T23:41:09.829 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 0, 2024-09-15T23:41:09.829 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 1321, 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:09.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:09.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:09.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 190167, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 52, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9144, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 172, 2024-09-15T23:41:09.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 8512, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 13, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-09-15T23:41:09.834 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 3259, 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 21, 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 3, 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 3259, 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3259, 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2.390945731, 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000733643 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3259, 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 3260 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.835 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 3259, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 3259, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 0, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 2138, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 0, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 1119, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 0, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 0, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-09-15T23:41:09.836 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 1, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-09-15T23:41:09.837 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:41:09.838 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-09-15T23:41:09.838 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:41:09.838 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-09-15T23:41:09.838 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-09-15T23:41:09.838 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-09-15T23:41:09.838 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-09-15T23:41:09.838 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-09-15T23:41:09.838 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-09-15T23:41:09.838 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-09-15T23:41:09.838 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-09-15T23:41:09.839 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-09-15T23:41:09.839 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-09-15T23:41:09.839 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-09-15T23:41:09.839 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-09-15T23:41:09.839 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-09-15T23:41:09.839 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-09-15T23:41:09.839 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-09-15T23:41:09.839 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-09-15T23:41:09.839 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-09-15T23:41:09.839 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 90, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 4, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 5, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 6, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 1, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 0, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 2, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 0 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 0, 2024-09-15T23:41:09.840 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 0, 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 0, 2024-09-15T23:41:09.841 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 0, 2024-09-15T23:41:09.842 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-09-15T23:41:09.842 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.842 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:09.842 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:09.842 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-09-15T23:41:09.842 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:09.842 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 3594, 2024-09-15T23:41:09.842 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 1393020, 2024-09-15T23:41:09.842 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:09.842 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 3594, 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 3594, 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 1393020, 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:09.843 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 3259, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 1661, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 3259, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 1661, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.844 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 3259, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 3259, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 3259, 2024-09-15T23:41:09.845 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 3259, 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:09.846 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x55df9a3cab40": { 2024-09-15T23:41:09.847 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-09-15T23:41:09.848 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-09-15T23:41:09.849 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:09.849 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-09-15T23:41:09.849 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-09-15T23:41:09.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:09.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:09.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi120 to 176.0M 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: Adjusting osd_memory_target on smithi089 to 175.9M 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: Unable to set osd_memory_target on smithi120 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: Unable to set osd_memory_target on smithi089 to 184518656: error parsing value: Value '184518656' is below minimum 939524096 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:41:09.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:09 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:41:10.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.236 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi120 to 176.0M 2024-09-15T23:41:10.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: Adjusting osd_memory_target on smithi089 to 175.9M 2024-09-15T23:41:10.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: Unable to set osd_memory_target on smithi120 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T23:41:10.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: Unable to set osd_memory_target on smithi089 to 184518656: error parsing value: Value '184518656' is below minimum 939524096 2024-09-15T23:41:10.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:41:10.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:41:10.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:09 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:41:10.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi120 to 176.0M 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: Adjusting osd_memory_target on smithi089 to 175.9M 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: Unable to set osd_memory_target on smithi120 to 184575283: error parsing value: Value '184575283' is below minimum 939524096 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: Unable to set osd_memory_target on smithi089 to 184518656: error parsing value: Value '184518656' is below minimum 939524096 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:41:10.311 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:41:10.312 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:09 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' 2024-09-15T23:41:10.662 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-09-15T23:41:10.663 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-09-15T23:41:10.663 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-09-15T23:41:10.663 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-09-15T23:41:10.663 DEBUG:tasks.check_counter:Getting stats from c 2024-09-15T23:41:10.663 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.c perf dump 2024-09-15T23:41:10.922 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:41:11.330 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:11 smithi174 ceph-mon[35586]: pgmap v864: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T23:41:11.446 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1921, 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1296, 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 463288, 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 387914, 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 18, 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.152018132, 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.056115028, 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.100671313, 2024-09-15T23:41:11.447 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.007245425, 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1297, 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.069642437, 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000053695 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3013, 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000742045, 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000246 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 448 2024-09-15T23:41:11.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 26, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 265, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 5844, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 690057, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 16, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.020149129, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.011009436, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.005810004, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000383009, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 263, 2024-09-15T23:41:11.449 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.028235564, 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000107359 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 8, 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000682, 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000085 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 578880 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:11.450 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 429, 2024-09-15T23:41:11.451 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 419, 2024-09-15T23:41:11.451 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 846919, 2024-09-15T23:41:11.452 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 697645, 2024-09-15T23:41:11.452 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 49, 2024-09-15T23:41:11.452 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-09-15T23:41:11.452 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.069512822, 2024-09-15T23:41:11.452 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.024992184, 2024-09-15T23:41:11.452 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.051455864, 2024-09-15T23:41:11.452 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000104852, 2024-09-15T23:41:11.452 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:11.452 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 416, 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.025469552, 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000061224 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 72, 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000008664, 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000120 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 839744, 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 207424 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-09-15T23:41:11.453 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 44, 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.001681897, 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000038224 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-09-15T23:41:11.454 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000011816, 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000005908 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 6, 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 4, 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4, 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.002418818, 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000604704 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-09-15T23:41:11.455 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 2, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 11, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 16, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 0, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 16, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 13, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 4, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-09-15T23:41:11.456 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 3, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 51, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 43, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 7, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-09-15T23:41:11.457 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 5, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 266, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:41:11.458 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 3, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 3, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 1 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-09-15T23:41:11.459 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 16, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 11, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 12, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:41:11.460 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 15, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 89, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-09-15T23:41:11.461 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 6, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 6, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 2, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 2, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-09-15T23:41:11.462 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 4195960, 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 0, 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 5, 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.009966919, 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001993383 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 0 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 20, 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 22, 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 2, 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 19, 2024-09-15T23:41:11.463 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 22, 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 3, 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 16, 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 20, 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 4, 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 4, 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 4, 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 50336, 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 223520 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-09-15T23:41:11.464 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 10, 2024-09-15T23:41:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-09-15T23:41:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 6, 2024-09-15T23:41:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 8, 2024-09-15T23:41:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 1, 2024-09-15T23:41:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-09-15T23:41:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.465 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.001920031, 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000960015 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.466 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000498787, 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000249393 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.467 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-09-15T23:41:11.468 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.469 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.470 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.471 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-09-15T23:41:11.472 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.473 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.474 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-09-15T23:41:11.475 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 0, 2024-09-15T23:41:11.476 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 2, 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 2, 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 0, 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 3, 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 3, 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 68, 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:11.477 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:11.478 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:11.479 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 231607, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 74, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:11.480 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9144, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 172, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 81069, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 340, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:11.481 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 44, 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 12273, 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 33, 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 44, 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 44, 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.223997050, 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.005090842 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.482 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 44, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 68 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 44, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 0, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 44, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 0, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 10, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 0, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 5, 2024-09-15T23:41:11.483 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 8, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 6, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 0, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 20, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-09-15T23:41:11.484 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-09-15T23:41:11.485 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 90, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 4, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 8, 2024-09-15T23:41:11.486 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 11, 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 3, 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 17, 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 0, 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 2 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 1, 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-09-15T23:41:11.487 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 97 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:11.488 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2372, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1079983, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2372, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2372, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1079983, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.489 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:11.490 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 44, 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 12273, 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 44, 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 12273, 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-09-15T23:41:11.491 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 44, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 44, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 44, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 44, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.492 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:11.493 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.493 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:41:11.493 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:11.493 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T23:41:11.493 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:11.493 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:11.493 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:11.493 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x56130cca2960": { 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:11.494 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T23:41:11.495 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:11.495 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 6, 2024-09-15T23:41:11.495 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1656, 2024-09-15T23:41:11.495 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:11.495 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 6, 2024-09-15T23:41:11.495 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:11.495 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:11.496 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 5, 2024-09-15T23:41:11.496 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1656, 2024-09-15T23:41:11.496 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:11.496 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:11.496 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:11.496 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:11.496 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:11.496 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:11.496 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-15T23:41:11.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:11 smithi089 ceph-mon[30933]: pgmap v864: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T23:41:11.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:11 smithi120 ceph-mon[35250]: pgmap v864: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-09-15T23:41:12.481 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-09-15T23:41:12.482 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-09-15T23:41:12.482 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-09-15T23:41:12.482 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=16 2024-09-15T23:41:12.482 DEBUG:tasks.check_counter:Getting stats from f 2024-09-15T23:41:12.482 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.f perf dump 2024-09-15T23:41:12.730 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:41:13.249 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 63, 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 65, 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 12957, 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 591848, 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 58, 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.027056994, 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.001486879, 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.019314556, 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000407942, 2024-09-15T23:41:13.250 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 62, 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.015161016, 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000244532 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 48, 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000004058, 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000084 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 448 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.251 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 2258, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1628, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1279725, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 558974, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 17, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.194573703, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.079982437, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.131908071, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.006907882, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1629, 2024-09-15T23:41:13.252 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.084507850, 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000051877 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3024, 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000723667, 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000239 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 823408, 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 202096 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 17, 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 257, 2024-09-15T23:41:13.253 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 4401, 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 588037, 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 8, 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.016804884, 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.010518645, 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.002905600, 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000458157, 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 255, 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.028453592, 2024-09-15T23:41:13.254 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000111582 2024-09-15T23:41:13.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.255 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:13.255 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.255 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.255 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.255 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T23:41:13.255 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 580672 2024-09-15T23:41:13.255 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 37, 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.002062740, 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000055749 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-09-15T23:41:13.256 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000007075, 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000003537 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.257 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 11, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 16, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 0, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 16, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 13, 2024-09-15T23:41:13.258 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 4, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 3, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 42, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 36, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 6, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-09-15T23:41:13.259 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 5, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 266, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:41:13.260 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-09-15T23:41:13.261 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 18, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 12, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 12, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-09-15T23:41:13.262 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 15, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 89, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-09-15T23:41:13.263 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 2, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 2, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 2, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-09-15T23:41:13.264 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 2, 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 4194892, 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 0, 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1, 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.001171461, 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001171461 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 0 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-09-15T23:41:13.265 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 20, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 22, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 2, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 19, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 23, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 4, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 16, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 20, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 4, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 53448, 2024-09-15T23:41:13.266 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 223520 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.267 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.268 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.269 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.270 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.271 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-09-15T23:41:13.272 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.273 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.273 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.273 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-09-15T23:41:13.273 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.273 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.273 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.273 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.273 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-09-15T23:41:13.273 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.274 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.274 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.274 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-09-15T23:41:13.274 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.274 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.274 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.274 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.274 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-09-15T23:41:13.274 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.275 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.276 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-09-15T23:41:13.277 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.278 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 0, 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 0, 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 0, 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 0, 2024-09-15T23:41:13.279 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-09-15T23:41:13.280 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-09-15T23:41:13.280 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-09-15T23:41:13.280 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 0, 2024-09-15T23:41:13.280 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-09-15T23:41:13.280 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.280 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-09-15T23:41:13.280 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:13.280 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:13.280 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:13.281 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:13.281 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:13.281 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:13.281 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:13.281 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:13.281 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:13.281 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:13.281 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:13.281 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:13.281 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:13.282 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 230615, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 71, 2024-09-15T23:41:13.283 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9144, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 172, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:13.284 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 80941, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 324, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 35, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 10710, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 33, 2024-09-15T23:41:13.285 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 35, 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 35, 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.188997473, 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.005399927 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 35, 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 57 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 35, 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 0, 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 35, 2024-09-15T23:41:13.286 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 0, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 10, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 0, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 1, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 5, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 6, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-09-15T23:41:13.287 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 20, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-09-15T23:41:13.288 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-09-15T23:41:13.289 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 90, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 4, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 9, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 13, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 4, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 14, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 0, 2024-09-15T23:41:13.290 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 1 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 1, 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 97 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:41:13.291 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2334, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1065329, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:13.292 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2334, 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2334, 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1065329, 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T23:41:13.293 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 35, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 10710, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 35, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 10710, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.294 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 35, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 35, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 35, 2024-09-15T23:41:13.295 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 35, 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:13.296 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x55fc7158a960": { 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:13.297 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 588, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 1, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 588, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:13.298 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:13.299 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:13.299 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-15T23:41:13.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:13 smithi174 ceph-mon[35586]: pgmap v865: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-15T23:41:13.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:13 smithi089 ceph-mon[30933]: pgmap v865: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-15T23:41:13.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:13 smithi120 ceph-mon[35250]: pgmap v865: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-15T23:41:14.081 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-09-15T23:41:14.081 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-09-15T23:41:14.081 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-09-15T23:41:14.081 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=18 2024-09-15T23:41:14.082 DEBUG:tasks.check_counter:Getting stats from i 2024-09-15T23:41:14.082 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.i perf dump 2024-09-15T23:41:14.328 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:41:14.827 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-15T23:41:14.827 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:14.827 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1900, 2024-09-15T23:41:14.827 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1282, 2024-09-15T23:41:14.828 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 454486, 2024-09-15T23:41:14.828 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 401981, 2024-09-15T23:41:14.828 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 9, 2024-09-15T23:41:14.828 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 3, 2024-09-15T23:41:14.828 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.153046879, 2024-09-15T23:41:14.828 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.058617871, 2024-09-15T23:41:14.828 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.098399438, 2024-09-15T23:41:14.828 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.007774357, 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1280, 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.072046505, 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000056286 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3024, 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000681772, 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000225 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:41:14.829 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 30, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 269, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 6296, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 550597, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 14, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.030460105, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.014486815, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.012496660, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000197354, 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:14.830 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 267, 2024-09-15T23:41:14.831 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.025760436, 2024-09-15T23:41:14.831 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000096481 2024-09-15T23:41:14.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.831 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:14.831 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 33, 2024-09-15T23:41:14.831 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000006193, 2024-09-15T23:41:14.831 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000187 2024-09-15T23:41:14.831 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.831 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 2928, 2024-09-15T23:41:14.831 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 496192 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 431, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 415, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 851115, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 868162, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 61, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.074087197, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.025199618, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.055111834, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000216959, 2024-09-15T23:41:14.832 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 415, 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.032247092, 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000077703 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 54, 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000004249, 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000078 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 841088, 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 200784 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.833 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-09-15T23:41:14.834 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-09-15T23:41:14.834 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-09-15T23:41:14.834 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.834 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-09-15T23:41:14.834 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T23:41:14.834 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T23:41:14.834 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 31, 2024-09-15T23:41:14.834 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.001504148, 2024-09-15T23:41:14.834 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000048520 2024-09-15T23:41:14.834 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:14.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.835 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-09-15T23:41:14.835 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T23:41:14.835 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T23:41:14.835 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-09-15T23:41:14.835 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000060834, 2024-09-15T23:41:14.835 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000030417 2024-09-15T23:41:14.835 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:14.835 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.835 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-09-15T23:41:14.836 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 1, 2024-09-15T23:41:14.836 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-09-15T23:41:14.836 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 1, 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 11, 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-09-15T23:41:14.837 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 16, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 0, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 16, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 13, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 4, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 3, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 43, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 36, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 1, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 6, 2024-09-15T23:41:14.838 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 5, 2024-09-15T23:41:14.839 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 266, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:41:14.840 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 18, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 12, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 12, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-09-15T23:41:14.841 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 15, 2024-09-15T23:41:14.842 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 89, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 3, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-09-15T23:41:14.843 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 3, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 2, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 2, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 4195200, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 0, 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-09-15T23:41:14.844 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.002892765, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001446382 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 0 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 20, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 22, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 2, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 19, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 23, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 4, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 16, 2024-09-15T23:41:14.845 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 20, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 4, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 52544, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 223520 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 1, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 1, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 14, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-09-15T23:41:14.846 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.847 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-09-15T23:41:14.848 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.849 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.850 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.851 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-09-15T23:41:14.851 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.851 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.851 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.852 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.853 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.854 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-09-15T23:41:14.855 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.856 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.857 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.858 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 1, 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 1, 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 0, 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 1, 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 1015, 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.859 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-09-15T23:41:14.860 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:14.860 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:14.860 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:14.860 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:14.860 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:14.860 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:14.860 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:14.860 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:14.860 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:14.860 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:14.861 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:14.862 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 238759, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 76, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9144, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 172, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:14.863 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 80965, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 327, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 37, 2024-09-15T23:41:14.864 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 11108, 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 30, 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 37, 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 37, 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.204997299, 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.005540467 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 37, 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 59 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 37, 2024-09-15T23:41:14.865 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 0, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 37, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 0, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 10, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 0, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 2, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 6, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 6, 2024-09-15T23:41:14.866 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 20, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:41:14.867 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-09-15T23:41:14.868 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 90, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 4, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 9, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 15, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 6, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 14, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 0, 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 1 2024-09-15T23:41:14.869 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 1, 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 97 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-09-15T23:41:14.870 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2357, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1077453, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2357, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:14.871 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2357, 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1077453, 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:14.872 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 37, 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 11108, 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 37, 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 11108, 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.873 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 37, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 37, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 37, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 37, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.874 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:14.875 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x55d53d34c960": { 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 3, 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 896, 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 3, 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2, 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 896, 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:14.877 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-15T23:41:15.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:15 smithi089 ceph-mon[30933]: pgmap v866: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-15T23:41:15.598 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:15 smithi174 ceph-mon[35586]: pgmap v866: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-15T23:41:15.624 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-09-15T23:41:15.625 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-09-15T23:41:15.625 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-09-15T23:41:15.625 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=18 2024-09-15T23:41:15.625 DEBUG:tasks.check_counter:Getting stats from l 2024-09-15T23:41:15.625 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph daemon mds.l perf dump 2024-09-15T23:41:15.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:15 smithi120 ceph-mon[35250]: pgmap v866: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-09-15T23:41:15.871 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:41:16.368 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-09-15T23:41:16.368 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:41:16.368 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1458, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1450, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1190752, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 555828, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 5, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 1, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.195072914, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.067374687, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.074621937, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.047192072, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1450, 2024-09-15T23:41:16.369 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.094337048, 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000065060 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 815104, 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 200304 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 4, 2024-09-15T23:41:16.370 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 248, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 876, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 582074, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 5, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 3, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.015032884, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.010446347, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.001749990, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000054418, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 246, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.017309568, 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000070364 2024-09-15T23:41:16.371 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 581376 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 2139, 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 2141, 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 545623, 2024-09-15T23:41:16.372 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 680209, 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 5, 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 3, 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.151562438, 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.051438310, 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.037604793, 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.051118276, 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2139, 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.084076382, 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000039306 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:41:16.373 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.374 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.374 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.374 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:41:16.374 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:41:16.374 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.374 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-09-15T23:41:16.374 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-09-15T23:41:16.374 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 5495, 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.160312767, 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000029174 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-09-15T23:41:16.375 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-09-15T23:41:16.377 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 1, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-09-15T23:41:16.378 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-09-15T23:41:16.379 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-09-15T23:41:16.380 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:41:16.381 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-09-15T23:41:16.382 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 0, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 3, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 0, 2024-09-15T23:41:16.383 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 2, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 4195200, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 4194892, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.384 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 3 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 3, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 3, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 0, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 1, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 1, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 0, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 0, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 0, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 0, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-09-15T23:41:16.385 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 48744, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 215328 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.386 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.389 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-09-15T23:41:16.390 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.391 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.392 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-09-15T23:41:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 1, 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 1, 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 0, 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 1, 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 2272, 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:41:16.399 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-09-15T23:41:16.400 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 182855, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 49, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:41:16.401 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9144, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 172, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 8512, 2024-09-15T23:41:16.402 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 13, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 3259, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 21, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 3, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 3259, 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-09-15T23:41:16.403 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3259, 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.592978743, 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000488793 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3259, 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 3260 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 3259, 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 3259, 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 0, 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-09-15T23:41:16.404 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 2134, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 1123, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-09-15T23:41:16.405 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 1, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-09-15T23:41:16.406 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-09-15T23:41:16.407 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 90, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 4, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 4, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 5, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 1, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 0, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 2, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 0 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.408 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 0, 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-09-15T23:41:16.409 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 3597, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1393091, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 3597, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 3597, 2024-09-15T23:41:16.410 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1393091, 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:16.411 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 3259, 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 901, 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 3259, 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 901, 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:16.412 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 3259, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 3259, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 3259, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 3259, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.413 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-09-15T23:41:16.414 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x56111a520b40": { 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-09-15T23:41:16.415 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-09-15T23:41:16.416 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-09-15T23:41:16.416 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-09-15T23:41:16.416 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-09-15T23:41:16.416 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-09-15T23:41:16.416 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-09-15T23:41:16.416 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-09-15T23:41:16.416 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-09-15T23:41:16.416 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:16.416 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-09-15T23:41:16.416 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-09-15T23:41:17.194 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-09-15T23:41:17.194 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-09-15T23:41:17.194 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-09-15T23:41:17.195 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-09-15T23:41:17.195 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-15T23:41:17.206 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-15T23:41:17.206 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:41:17.206 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T23:41:17.227 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-15T23:41:17.228 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T23:41:17.287 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-15T23:41:17.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:17 smithi174 ceph-mon[35586]: pgmap v867: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T23:41:17.561 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:17 smithi089 ceph-mon[30933]: pgmap v867: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T23:41:17.586 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:17.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:17 smithi120 ceph-mon[35250]: pgmap v867: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-09-15T23:41:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.031+0000 7f2820dde640 1 -- 172.21.15.89:0/2860965230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f281c106a80 msgr2=0x7f281c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.031+0000 7f2820dde640 1 --2- 172.21.15.89:0/2860965230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f281c106a80 0x7f281c106e60 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f280c009930 tx=0x7f280c02a420 comp rx=0 tx=0).stop 2024-09-15T23:41:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.031+0000 7f2820dde640 1 -- 172.21.15.89:0/2860965230 shutdown_connections 2024-09-15T23:41:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.031+0000 7f2820dde640 1 --2- 172.21.15.89:0/2860965230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f281c101420 0x7f281c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.031+0000 7f2820dde640 1 --2- 172.21.15.89:0/2860965230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c100a80 0x7f281c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.031+0000 7f2820dde640 1 --2- 172.21.15.89:0/2860965230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f281c106a80 0x7f281c106e60 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.031+0000 7f2820dde640 1 -- 172.21.15.89:0/2860965230 >> 172.21.15.89:0/2860965230 conn(0x7f281c0fc7a0 msgr2=0x7f281c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:18.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.031+0000 7f2820dde640 1 -- 172.21.15.89:0/2860965230 shutdown_connections 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.031+0000 7f2820dde640 1 -- 172.21.15.89:0/2860965230 wait complete. 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f2820dde640 1 Processor -- start 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f2820dde640 1 -- start start 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f2820dde640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c100a80 0x7f281c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f2820dde640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f281c101420 0x7f281c198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f2820dde640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f281c106a80 0x7f281c19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f2820dde640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f281c110ec0 con 0x7f281c106a80 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f2820dde640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f281c110d40 con 0x7f281c101420 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f2820dde640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f281c111040 con 0x7f281c100a80 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f281ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f281c106a80 0x7f281c19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f281ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f281c106a80 0x7f281c19cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33382/0 (socket says 172.21.15.89:33382) 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.032+0000 7f281ad9d640 1 -- 172.21.15.89:0/2295354031 learned_addr learned my addr 172.21.15.89:0/2295354031 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:41:18.052 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f2813fff640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f281c101420 0x7f281c198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f281a59c640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c100a80 0x7f281c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f281ad9d640 1 -- 172.21.15.89:0/2295354031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c100a80 msgr2=0x7f281c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f281ad9d640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c100a80 0x7f281c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f281ad9d640 1 -- 172.21.15.89:0/2295354031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f281c101420 msgr2=0x7f281c198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f281ad9d640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f281c101420 0x7f281c198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f281ad9d640 1 -- 172.21.15.89:0/2295354031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f281c19d4b0 con 0x7f281c106a80 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f281ad9d640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f281c106a80 0x7f281c19cdd0 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f281400b970 tx=0x7f281400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f28137fe640 1 -- 172.21.15.89:0/2295354031 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2814019070 con 0x7f281c106a80 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f2820dde640 1 -- 172.21.15.89:0/2295354031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f281c19d7a0 con 0x7f281c106a80 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f28137fe640 1 -- 172.21.15.89:0/2295354031 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f281400cb40 con 0x7f281c106a80 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f2820dde640 1 -- 172.21.15.89:0/2295354031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f281c1a5080 con 0x7f281c106a80 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.033+0000 7f28137fe640 1 -- 172.21.15.89:0/2295354031 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2814007650 con 0x7f281c106a80 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.034+0000 7f2820dde640 1 -- 172.21.15.89:0/2295354031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f281c100ee0 con 0x7f281c106a80 2024-09-15T23:41:18.053 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.035+0000 7f28137fe640 1 -- 172.21.15.89:0/2295354031 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f281401c070 con 0x7f281c106a80 2024-09-15T23:41:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.036+0000 7f28137fe640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28080a3240 0x7f28080a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.036+0000 7f28137fe640 1 -- 172.21.15.89:0/2295354031 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f28140cf780 con 0x7f281c106a80 2024-09-15T23:41:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.036+0000 7f281a59c640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28080a3240 0x7f28080a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.037+0000 7f281a59c640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28080a3240 0x7f28080a5700 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f280c009900 tx=0x7f280c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:18.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.040+0000 7f28137fe640 1 -- 172.21.15.89:0/2295354031 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f281409ba60 con 0x7f281c106a80 2024-09-15T23:41:18.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.217+0000 7f2820dde640 1 -- 172.21.15.89:0/2295354031 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f281c1a5310 con 0x7f281c106a80 2024-09-15T23:41:18.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.217+0000 7f28137fe640 1 -- 172.21.15.89:0/2295354031 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+589 (secure 0 0 0) 0x7f281409fd60 con 0x7f281c106a80 2024-09-15T23:41:18.219 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6801/429862840 2024-09-16T23:15:00.884149+0000 2024-09-15T23:41:18.219 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6800/429862840 2024-09-16T23:15:00.884149+0000 2024-09-15T23:41:18.219 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1174639743 2024-09-16T23:15:00.884149+0000 2024-09-15T23:41:18.219 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3889269108 2024-09-16T23:15:00.884149+0000 2024-09-15T23:41:18.219 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/4118871044 2024-09-16T23:15:00.884149+0000 2024-09-15T23:41:18.219 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/379727762 2024-09-16T23:14:32.424565+0000 2024-09-15T23:41:18.219 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6801/4057509998 2024-09-16T23:14:32.424565+0000 2024-09-15T23:41:18.219 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:6800/4057509998 2024-09-16T23:14:32.424565+0000 2024-09-15T23:41:18.220 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/3796595264 2024-09-16T23:14:32.424565+0000 2024-09-15T23:41:18.220 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89:0/1030688282 2024-09-16T23:14:32.424565+0000 2024-09-15T23:41:18.220 INFO:teuthology.orchestra.run.smithi089.stderr:listed 10 entries 2024-09-15T23:41:18.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.219+0000 7f2820dde640 1 -- 172.21.15.89:0/2295354031 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28080a3240 msgr2=0x7f28080a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:18.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.219+0000 7f2820dde640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28080a3240 0x7f28080a5700 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f280c009900 tx=0x7f280c0058d0 comp rx=0 tx=0).stop 2024-09-15T23:41:18.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.219+0000 7f2820dde640 1 -- 172.21.15.89:0/2295354031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f281c106a80 msgr2=0x7f281c19cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:18.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.219+0000 7f2820dde640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f281c106a80 0x7f281c19cdd0 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f281400b970 tx=0x7f281400be30 comp rx=0 tx=0).stop 2024-09-15T23:41:18.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.219+0000 7f2820dde640 1 -- 172.21.15.89:0/2295354031 shutdown_connections 2024-09-15T23:41:18.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.219+0000 7f2820dde640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f28080a3240 0x7f28080a5700 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:18.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.219+0000 7f2820dde640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f281c106a80 0x7f281c19cdd0 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:18.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.219+0000 7f2820dde640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f281c101420 0x7f281c198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:18.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.219+0000 7f2820dde640 1 --2- 172.21.15.89:0/2295354031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c100a80 0x7f281c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:18.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.219+0000 7f2820dde640 1 -- 172.21.15.89:0/2295354031 >> 172.21.15.89:0/2295354031 conn(0x7f281c0fc7a0 msgr2=0x7f281c1048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:18.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.220+0000 7f2820dde640 1 -- 172.21.15.89:0/2295354031 shutdown_connections 2024-09-15T23:41:18.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:18.220+0000 7f2820dde640 1 -- 172.21.15.89:0/2295354031 wait complete. 2024-09-15T23:41:18.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:18 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2295354031' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:41:18.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:18 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2295354031' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:41:18.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:18 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2295354031' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:41:19.002 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-15T23:41:19.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:41:19.002 DEBUG:teuthology.orchestra.run.smithi089:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-15T23:41:19.043 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi089.front.sepia.ceph.com 2024-09-15T23:41:19.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:41:19.043 DEBUG:teuthology.orchestra.run.smithi089:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-15T23:41:19.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:41:19.098 DEBUG:teuthology.orchestra.run.smithi089:> ip netns list 2024-09-15T23:41:19.153 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-15T23:41:19.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:41:19.154 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-15T23:41:19.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:41:19.222 DEBUG:teuthology.orchestra.run.smithi089:> sudo ip link delete ceph-brx 2024-09-15T23:41:19.318 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-15T23:41:19.329 INFO:tasks.cephadm:Teardown begin 2024-09-15T23:41:19.330 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-15T23:41:19.330 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:41:19.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:19 smithi089 ceph-mon[30933]: pgmap v868: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:41:19.578 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:19.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:19 smithi120 ceph-mon[35250]: pgmap v868: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:41:19.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:19 smithi174 ceph-mon[35586]: pgmap v868: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-09-15T23:41:20.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.050+0000 7f08a9258640 1 -- 172.21.15.89:0/2652657017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08a4102570 msgr2=0x7f08a41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:20.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.050+0000 7f08a9258640 1 --2- 172.21.15.89:0/2652657017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08a4102570 0x7f08a41029b0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f0898009900 tx=0x7f089802a480 comp rx=0 tx=0).stop 2024-09-15T23:41:20.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.050+0000 7f08a9258640 1 -- 172.21.15.89:0/2652657017 shutdown_connections 2024-09-15T23:41:20.070 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.050+0000 7f08a9258640 1 --2- 172.21.15.89:0/2652657017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08a4102ef0 0x7f08a410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.050+0000 7f08a9258640 1 --2- 172.21.15.89:0/2652657017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08a4102570 0x7f08a41029b0 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.050+0000 7f08a9258640 1 --2- 172.21.15.89:0/2652657017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4100e20 0x7f08a4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.050+0000 7f08a9258640 1 -- 172.21.15.89:0/2652657017 >> 172.21.15.89:0/2652657017 conn(0x7f08a40fc7d0 msgr2=0x7f08a40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.050+0000 7f08a9258640 1 -- 172.21.15.89:0/2652657017 shutdown_connections 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.050+0000 7f08a9258640 1 -- 172.21.15.89:0/2652657017 wait complete. 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a9258640 1 Processor -- start 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a9258640 1 -- start start 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a9258640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08a4100e20 0x7f08a40fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a9258640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08a4102570 0x7f08a40fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a9258640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102ef0 0x7f08a40fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a9258640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4110f80 con 0x7f08a4100e20 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a9258640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4110e00 con 0x7f08a4102ef0 2024-09-15T23:41:20.071 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a9258640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f08a4111100 con 0x7f08a4102570 2024-09-15T23:41:20.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08a4100e20 0x7f08a40fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:20.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08a4100e20 0x7f08a40fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33400/0 (socket says 172.21.15.89:33400) 2024-09-15T23:41:20.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a2d9d640 1 -- 172.21.15.89:0/3632559194 learned_addr learned my addr 172.21.15.89:0/3632559194 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:41:20.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a259c640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08a4102570 0x7f08a40fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:20.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a359e640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102ef0 0x7f08a40fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:20.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a2d9d640 1 -- 172.21.15.89:0/3632559194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08a4102570 msgr2=0x7f08a40fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:20.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.051+0000 7f08a2d9d640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08a4102570 0x7f08a40fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:20.072 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f08a2d9d640 1 -- 172.21.15.89:0/3632559194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102ef0 msgr2=0x7f08a40fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f08a2d9d640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102ef0 0x7f08a40fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f08a2d9d640 1 -- 172.21.15.89:0/3632559194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a41a68f0 con 0x7f08a4100e20 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f08a359e640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102ef0 0x7f08a40fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f08a259c640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08a4102570 0x7f08a40fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f08a2d9d640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08a4100e20 0x7f08a40fcc80 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f089400c920 tx=0x7f089400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f088bfff640 1 -- 172.21.15.89:0/3632559194 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894007c70 con 0x7f08a4100e20 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f08a9258640 1 -- 172.21.15.89:0/3632559194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a41a6be0 con 0x7f08a4100e20 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f088bfff640 1 -- 172.21.15.89:0/3632559194 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0894007e10 con 0x7f08a4100e20 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f088bfff640 1 -- 172.21.15.89:0/3632559194 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894018890 con 0x7f08a4100e20 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.052+0000 7f08a9258640 1 -- 172.21.15.89:0/3632559194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a41a7120 con 0x7f08a4100e20 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.053+0000 7f08a9258640 1 -- 172.21.15.89:0/3632559194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a4101200 con 0x7f08a4100e20 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.054+0000 7f088bfff640 1 -- 172.21.15.89:0/3632559194 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f0894018a70 con 0x7f08a4100e20 2024-09-15T23:41:20.073 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.055+0000 7f088bfff640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f088c0a3240 0x7f088c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:20.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.055+0000 7f08a259c640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f088c0a3240 0x7f088c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:20.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.055+0000 7f088bfff640 1 -- 172.21.15.89:0/3632559194 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f08940cff00 con 0x7f08a4100e20 2024-09-15T23:41:20.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.056+0000 7f08a259c640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f088c0a3240 0x7f088c0a5700 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f08a40fe710 tx=0x7f0898005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:20.074 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.059+0000 7f088bfff640 1 -- 172.21.15.89:0/3632559194 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0894014030 con 0x7f08a4100e20 2024-09-15T23:41:20.261 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.260+0000 7f08a9258640 1 -- 172.21.15.89:0/3632559194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08a402d7c0 con 0x7f08a4100e20 2024-09-15T23:41:20.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.264+0000 7f088bfff640 1 -- 172.21.15.89:0/3632559194 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f089400de20 con 0x7f08a4100e20 2024-09-15T23:41:20.265 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:41:20.266 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:41:20.266 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 -- 172.21.15.89:0/3632559194 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f088c0a3240 msgr2=0x7f088c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f088c0a3240 0x7f088c0a5700 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f08a40fe710 tx=0x7f0898005a20 comp rx=0 tx=0).stop 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 -- 172.21.15.89:0/3632559194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08a4100e20 msgr2=0x7f08a40fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08a4100e20 0x7f08a40fcc80 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f089400c920 tx=0x7f089400cde0 comp rx=0 tx=0).stop 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 -- 172.21.15.89:0/3632559194 shutdown_connections 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f088c0a3240 0x7f088c0a5700 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102ef0 0x7f08a40fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08a4102570 0x7f08a40fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 --2- 172.21.15.89:0/3632559194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08a4100e20 0x7f08a40fcc80 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 -- 172.21.15.89:0/3632559194 >> 172.21.15.89:0/3632559194 conn(0x7f08a40fc7d0 msgr2=0x7f08a406a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 -- 172.21.15.89:0/3632559194 shutdown_connections 2024-09-15T23:41:20.268 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:20.267+0000 7f08a9258640 1 -- 172.21.15.89:0/3632559194 wait complete. 2024-09-15T23:41:20.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:20 smithi089 ceph-mon[30933]: pgmap v869: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:41:20.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:20 smithi120 ceph-mon[35250]: pgmap v869: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:41:20.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:20 smithi174 ceph-mon[35586]: pgmap v869: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-09-15T23:41:21.074 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:41:21.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:21 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3632559194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:41:21.322 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:21.623 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:21 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3632559194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:41:21.637 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:21 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3632559194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:41:21.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.824+0000 7f169789c640 1 -- 172.21.15.89:0/901055136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1690102ef0 msgr2=0x7f169010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:21.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.824+0000 7f169789c640 1 --2- 172.21.15.89:0/901055136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1690102ef0 0x7f169010da40 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f1688009ca0 tx=0x7f168802a950 comp rx=0 tx=0).stop 2024-09-15T23:41:21.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 -- 172.21.15.89:0/901055136 shutdown_connections 2024-09-15T23:41:21.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 --2- 172.21.15.89:0/901055136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1690102ef0 0x7f169010da40 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:21.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 --2- 172.21.15.89:0/901055136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1690102570 0x7f16901029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:21.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 --2- 172.21.15.89:0/901055136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1690100e20 0x7f1690101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:21.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 -- 172.21.15.89:0/901055136 >> 172.21.15.89:0/901055136 conn(0x7f16900fc7d0 msgr2=0x7f16900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:21.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 -- 172.21.15.89:0/901055136 shutdown_connections 2024-09-15T23:41:21.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 -- 172.21.15.89:0/901055136 wait complete. 2024-09-15T23:41:21.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 Processor -- start 2024-09-15T23:41:21.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 -- start start 2024-09-15T23:41:21.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1690100e20 0x7f1690199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:21.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1690102570 0x7f16901996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1690102ef0 0x7f16901a4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f1690110f80 con 0x7f1690102570 2024-09-15T23:41:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1690110e00 con 0x7f1690100e20 2024-09-15T23:41:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.825+0000 7f169789c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1690111100 con 0x7f1690102ef0 2024-09-15T23:41:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1694e37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1690102570 0x7f16901996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1694e37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1690102570 0x7f16901996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33416/0 (socket says 172.21.15.89:33416) 2024-09-15T23:41:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1694e37640 1 -- 172.21.15.89:0/1186101578 learned_addr learned my addr 172.21.15.89:0/1186101578 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:41:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1695638640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1690100e20 0x7f1690199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:21.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1695e39640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1690102ef0 0x7f16901a4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1694e37640 1 -- 172.21.15.89:0/1186101578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1690102ef0 msgr2=0x7f16901a4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1694e37640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1690102ef0 0x7f16901a4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1694e37640 1 -- 172.21.15.89:0/1186101578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1690100e20 msgr2=0x7f1690199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1694e37640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1690100e20 0x7f1690199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1694e37640 1 -- 172.21.15.89:0/1186101578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16901a4750 con 0x7f1690102570 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1695638640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1690100e20 0x7f1690199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1695e39640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1690102ef0 0x7f16901a4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f1694e37640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1690102570 0x7f16901996d0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f168c00c920 tx=0x7f168c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f16827fc640 1 -- 172.21.15.89:0/1186101578 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c007c70 con 0x7f1690102570 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f16827fc640 1 -- 172.21.15.89:0/1186101578 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f168c007e10 con 0x7f1690102570 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f169789c640 1 -- 172.21.15.89:0/1186101578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16901a4920 con 0x7f1690102570 2024-09-15T23:41:21.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f16827fc640 1 -- 172.21.15.89:0/1186101578 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c018890 con 0x7f1690102570 2024-09-15T23:41:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.826+0000 7f169789c640 1 -- 172.21.15.89:0/1186101578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16901a4eb0 con 0x7f1690102570 2024-09-15T23:41:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.827+0000 7f169789c640 1 -- 172.21.15.89:0/1186101578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1690101200 con 0x7f1690102570 2024-09-15T23:41:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.829+0000 7f16827fc640 1 -- 172.21.15.89:0/1186101578 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f168c018a50 con 0x7f1690102570 2024-09-15T23:41:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.829+0000 7f16827fc640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f167c0a32c0 0x7f167c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.829+0000 7f1695638640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f167c0a32c0 0x7f167c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.829+0000 7f16827fc640 1 -- 172.21.15.89:0/1186101578 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7f168c0d0090 con 0x7f1690102570 2024-09-15T23:41:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.830+0000 7f1695638640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f167c0a32c0 0x7f167c0a5780 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f168400abe0 tx=0x7f168400a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:21.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:21.832+0000 7f16827fc640 1 -- 172.21.15.89:0/1186101578 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f168c014030 con 0x7f1690102570 2024-09-15T23:41:22.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.031+0000 7f169789c640 1 -- 172.21.15.89:0/1186101578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f169006a5a0 con 0x7f1690102570 2024-09-15T23:41:22.032 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.031+0000 7f16827fc640 1 -- 172.21.15.89:0/1186101578 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11986 (secure 0 0 0) 0x7f168c01f300 con 0x7f1690102570 2024-09-15T23:41:22.033 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:41:22.033 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":29,"btime":"2024-09-15T23:24:20:083773+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/3816006418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":3816006418},{"type":"v1","addr":"172.21.15.89:6835","nonce":3816006418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/3164152358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":3164152358},{"type":"v1","addr":"172.21.15.89:6837","nonce":3164152358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:21:53.066091+0000","modified":"2024-09-15T23:24:19.235823+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24452,"mds_1":24436,"mds_2":24430,"mds_3":24428,"mds_4":24424},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14610":{"gid":14610,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6839/2895111363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":2895111363},{"type":"v1","addr":"172.21.15.89:6839","nonce":2895111363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14616":{"gid":14616,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6841/3365004630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6840","nonce":3365004630},{"type":"v1","addr":"172.21.15.89:6841","nonce":3365004630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24389":{"gid":24389,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/4147766006","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":4147766006},{"type":"v1","addr":"172.21.15.120:6833","nonce":4147766006}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24403":{"gid":24403,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6833/2504210581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2504210581},{"type":"v1","addr":"172.21.15.174:6833","nonce":2504210581}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1429316223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1429316223},{"type":"v1","addr":"172.21.15.120:6835","nonce":1429316223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24424":{"gid":24424,"name":"f","rank":4,"incarnation":26,"state":"up:active","state_seq":19,"addr":"172.21.15.174:6835/733541815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":733541815},{"type":"v1","addr":"172.21.15.174:6835","nonce":733541815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24428":{"gid":24428,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.120:6837/991556551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":991556551},{"type":"v1","addr":"172.21.15.120:6837","nonce":991556551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24430":{"gid":24430,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/3626951914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3626951914},{"type":"v1","addr":"172.21.15.174:6837","nonce":3626951914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24436":{"gid":24436,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.174:6839/3380568300","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3380568300},{"type":"v1","addr":"172.21.15.174:6839","nonce":3380568300}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24452":{"gid":24452,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.120:6839/1554218591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":1554218591},{"type":"v1","addr":"172.21.15.120:6839","nonce":1554218591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24452,"qdb_cluster":[24424,24428,24430,24436,24452]},"id":1}]} 2024-09-15T23:41:22.033 INFO:teuthology.orchestra.run.smithi089.stderr:dumped fsmap epoch 29 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.033+0000 7f169789c640 1 -- 172.21.15.89:0/1186101578 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f167c0a32c0 msgr2=0x7f167c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.033+0000 7f169789c640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f167c0a32c0 0x7f167c0a5780 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f168400abe0 tx=0x7f168400a5c0 comp rx=0 tx=0).stop 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.033+0000 7f169789c640 1 -- 172.21.15.89:0/1186101578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1690102570 msgr2=0x7f16901996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.033+0000 7f169789c640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1690102570 0x7f16901996d0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f168c00c920 tx=0x7f168c00cde0 comp rx=0 tx=0).stop 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.034+0000 7f169789c640 1 -- 172.21.15.89:0/1186101578 shutdown_connections 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.034+0000 7f169789c640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f167c0a32c0 0x7f167c0a5780 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.034+0000 7f169789c640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1690102ef0 0x7f16901a4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.034+0000 7f169789c640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1690102570 0x7f16901996d0 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.034+0000 7f169789c640 1 --2- 172.21.15.89:0/1186101578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1690100e20 0x7f1690199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.034+0000 7f169789c640 1 -- 172.21.15.89:0/1186101578 >> 172.21.15.89:0/1186101578 conn(0x7f16900fc7d0 msgr2=0x7f16901056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:22.035 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.034+0000 7f169789c640 1 -- 172.21.15.89:0/1186101578 shutdown_connections 2024-09-15T23:41:22.036 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:22.034+0000 7f169789c640 1 -- 172.21.15.89:0/1186101578 wait complete. 2024-09-15T23:41:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:22 smithi089 ceph-mon[30933]: pgmap v870: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-09-15T23:41:22.310 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:22 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1186101578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:41:22.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:22 smithi120 ceph-mon[35250]: pgmap v870: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-09-15T23:41:22.636 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:22 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1186101578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:41:22.655 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:22 smithi174 ceph-mon[35586]: pgmap v870: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 11 op/s 2024-09-15T23:41:22.655 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:22 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1186101578' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:41:22.844 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:41:23.094 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.586+0000 7fabadf95640 1 -- 172.21.15.89:0/774224055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba8106a80 msgr2=0x7faba8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.586+0000 7fabadf95640 1 --2- 172.21.15.89:0/774224055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba8106a80 0x7faba8106e60 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fab98009930 tx=0x7fab9802a420 comp rx=0 tx=0).stop 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.586+0000 7fabadf95640 1 -- 172.21.15.89:0/774224055 shutdown_connections 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.586+0000 7fabadf95640 1 --2- 172.21.15.89:0/774224055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba8101420 0x7faba810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.586+0000 7fabadf95640 1 --2- 172.21.15.89:0/774224055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba8100a80 0x7faba8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.586+0000 7fabadf95640 1 --2- 172.21.15.89:0/774224055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba8106a80 0x7faba8106e60 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.586+0000 7fabadf95640 1 -- 172.21.15.89:0/774224055 >> 172.21.15.89:0/774224055 conn(0x7faba80fc7a0 msgr2=0x7faba80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.586+0000 7fabadf95640 1 -- 172.21.15.89:0/774224055 shutdown_connections 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.586+0000 7fabadf95640 1 -- 172.21.15.89:0/774224055 wait complete. 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7fabadf95640 1 Processor -- start 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7fabadf95640 1 -- start start 2024-09-15T23:41:23.607 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7fabadf95640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba8100a80 0x7faba81973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7fabadf95640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba8101420 0x7faba81978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7fabadf95640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba8106a80 0x7faba819bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7fabadf95640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7faba8110e80 con 0x7faba8100a80 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7fabadf95640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faba8110d00 con 0x7faba8101420 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7fabadf95640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faba8111000 con 0x7faba8106a80 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7faba77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba8100a80 0x7faba81973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7faba77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba8100a80 0x7faba81973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:33444/0 (socket says 172.21.15.89:33444) 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7faba77fe640 1 -- 172.21.15.89:0/2557710364 learned_addr learned my addr 172.21.15.89:0/2557710364 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7faba7fff640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba8106a80 0x7faba819bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.587+0000 7faba6ffd640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba8101420 0x7faba81978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7faba77fe640 1 -- 172.21.15.89:0/2557710364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba8106a80 msgr2=0x7faba819bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7faba77fe640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba8106a80 0x7faba819bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7faba77fe640 1 -- 172.21.15.89:0/2557710364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba8101420 msgr2=0x7faba81978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:23.608 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7faba77fe640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba8101420 0x7faba81978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7faba77fe640 1 -- 172.21.15.89:0/2557710364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba819c360 con 0x7faba8100a80 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7faba77fe640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba8100a80 0x7faba81973b0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fab98009230 tx=0x7fab98004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7faba4ff9640 1 -- 172.21.15.89:0/2557710364 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9803f030 con 0x7faba8100a80 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7faba4ff9640 1 -- 172.21.15.89:0/2557710364 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab98004440 con 0x7faba8100a80 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7fabadf95640 1 -- 172.21.15.89:0/2557710364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba819c5f0 con 0x7faba8100a80 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7faba4ff9640 1 -- 172.21.15.89:0/2557710364 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab98035070 con 0x7faba8100a80 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.588+0000 7fabadf95640 1 -- 172.21.15.89:0/2557710364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba81a3f30 con 0x7faba8100a80 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.589+0000 7fabadf95640 1 -- 172.21.15.89:0/2557710364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faba81021c0 con 0x7faba8100a80 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.590+0000 7faba4ff9640 1 -- 172.21.15.89:0/2557710364 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fab98042030 con 0x7faba8100a80 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.591+0000 7faba4ff9640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fab940a32c0 0x7fab940a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.592+0000 7faba6ffd640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fab940a32c0 0x7fab940a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.592+0000 7faba4ff9640 1 -- 172.21.15.89:0/2557710364 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fab980effd0 con 0x7faba8100a80 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.592+0000 7faba6ffd640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fab940a32c0 0x7fab940a5780 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7faba80fedf0 tx=0x7fab90005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:23.609 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.596+0000 7faba4ff9640 1 -- 172.21.15.89:0/2557710364 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fab980bc2b0 con 0x7faba8100a80 2024-09-15T23:41:23.764 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.763+0000 7fabadf95640 1 -- 172.21.15.89:0/2557710364 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faba8108ff0 con 0x7faba8100a80 2024-09-15T23:41:23.766 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.765+0000 7faba4ff9640 1 -- 172.21.15.89:0/2557710364 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24962 (secure 0 0 0) 0x7fab980b6e20 con 0x7faba8100a80 2024-09-15T23:41:23.766 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-15T23:41:23.767 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":90,"fsid":"1b4e699c-73b8-11ef-bceb-c7b262605968","created":"2024-09-15T23:14:08.706430+0000","modified":"2024-09-15T23:24:03.106312+0000","last_up_change":"2024-09-15T23:21:01.827018+0000","last_in_change":"2024-09-15T23:20:38.643352+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:17:15.186397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:21:49.360728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:21:51.296727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:21:57.778828+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e292c3de-4e37-4c40-9085-a85a262fcf6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6803","nonce":261479611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6805","nonce":261479611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6809","nonce":261479611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":261479611},{"type":"v1","addr":"172.21.15.89:6807","nonce":261479611}]},"public_addr":"172.21.15.89:6803/261479611","cluster_addr":"172.21.15.89:6805/261479611","heartbeat_back_addr":"172.21.15.89:6809/261479611","heartbeat_front_addr":"172.21.15.89:6807/261479611","state":["exists","up"]},{"osd":1,"uuid":"fa999797-98c3-47f1-a08f-71a789176397","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6801","nonce":2529464017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6803","nonce":2529464017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6807","nonce":2529464017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2529464017},{"type":"v1","addr":"172.21.15.120:6805","nonce":2529464017}]},"public_addr":"172.21.15.120:6801/2529464017","cluster_addr":"172.21.15.120:6803/2529464017","heartbeat_back_addr":"172.21.15.120:6807/2529464017","heartbeat_front_addr":"172.21.15.120:6805/2529464017","state":["exists","up"]},{"osd":2,"uuid":"b48eda20-370e-40bc-bdb3-4011a46df4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6801","nonce":3193519940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6803","nonce":3193519940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6807","nonce":3193519940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3193519940},{"type":"v1","addr":"172.21.15.174:6805","nonce":3193519940}]},"public_addr":"172.21.15.174:6801/3193519940","cluster_addr":"172.21.15.174:6803/3193519940","heartbeat_back_addr":"172.21.15.174:6807/3193519940","heartbeat_front_addr":"172.21.15.174:6805/3193519940","state":["exists","up"]},{"osd":3,"uuid":"783de02c-1c46-44ed-98c4-10fb142648a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6811","nonce":2725889139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6813","nonce":2725889139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6817","nonce":2725889139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":2725889139},{"type":"v1","addr":"172.21.15.89:6815","nonce":2725889139}]},"public_addr":"172.21.15.89:6811/2725889139","cluster_addr":"172.21.15.89:6813/2725889139","heartbeat_back_addr":"172.21.15.89:6817/2725889139","heartbeat_front_addr":"172.21.15.89:6815/2725889139","state":["exists","up"]},{"osd":4,"uuid":"425a48e4-145a-4f5c-88af-d7be7fdd3c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6809","nonce":135794330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6811","nonce":135794330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6815","nonce":135794330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":135794330},{"type":"v1","addr":"172.21.15.120:6813","nonce":135794330}]},"public_addr":"172.21.15.120:6809/135794330","cluster_addr":"172.21.15.120:6811/135794330","heartbeat_back_addr":"172.21.15.120:6815/135794330","heartbeat_front_addr":"172.21.15.120:6813/135794330","state":["exists","up"]},{"osd":5,"uuid":"0548511d-1178-448e-8549-0a934b57cfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6809","nonce":3950370591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6811","nonce":3950370591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6815","nonce":3950370591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":3950370591},{"type":"v1","addr":"172.21.15.174:6813","nonce":3950370591}]},"public_addr":"172.21.15.174:6809/3950370591","cluster_addr":"172.21.15.174:6811/3950370591","heartbeat_back_addr":"172.21.15.174:6815/3950370591","heartbeat_front_addr":"172.21.15.174:6813/3950370591","state":["exists","up"]},{"osd":6,"uuid":"df2eab94-90c7-4847-998f-bccdef2861e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6819","nonce":2734649081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6821","nonce":2734649081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6825","nonce":2734649081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":2734649081},{"type":"v1","addr":"172.21.15.89:6823","nonce":2734649081}]},"public_addr":"172.21.15.89:6819/2734649081","cluster_addr":"172.21.15.89:6821/2734649081","heartbeat_back_addr":"172.21.15.89:6825/2734649081","heartbeat_front_addr":"172.21.15.89:6823/2734649081","state":["exists","up"]},{"osd":7,"uuid":"696fd052-0624-47fc-bdf6-bcfc6f69ff74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6817","nonce":831027547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6819","nonce":831027547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6823","nonce":831027547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":831027547},{"type":"v1","addr":"172.21.15.120:6821","nonce":831027547}]},"public_addr":"172.21.15.120:6817/831027547","cluster_addr":"172.21.15.120:6819/831027547","heartbeat_back_addr":"172.21.15.120:6823/831027547","heartbeat_front_addr":"172.21.15.120:6821/831027547","state":["exists","up"]},{"osd":8,"uuid":"3f51541d-5288-4ed5-9780-516f0f26744b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6817","nonce":2158836361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6819","nonce":2158836361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6823","nonce":2158836361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2158836361},{"type":"v1","addr":"172.21.15.174:6821","nonce":2158836361}]},"public_addr":"172.21.15.174:6817/2158836361","cluster_addr":"172.21.15.174:6819/2158836361","heartbeat_back_addr":"172.21.15.174:6823/2158836361","heartbeat_front_addr":"172.21.15.174:6821/2158836361","state":["exists","up"]},{"osd":9,"uuid":"81930b2a-3a54-4442-9c89-316644a3adce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6827","nonce":2707132462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6829","nonce":2707132462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6833","nonce":2707132462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2707132462},{"type":"v1","addr":"172.21.15.89:6831","nonce":2707132462}]},"public_addr":"172.21.15.89:6827/2707132462","cluster_addr":"172.21.15.89:6829/2707132462","heartbeat_back_addr":"172.21.15.89:6833/2707132462","heartbeat_front_addr":"172.21.15.89:6831/2707132462","state":["exists","up"]},{"osd":10,"uuid":"586ff85c-4fa6-415d-8843-7e57328f9e41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6825","nonce":254855304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6827","nonce":254855304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6831","nonce":254855304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":254855304},{"type":"v1","addr":"172.21.15.120:6829","nonce":254855304}]},"public_addr":"172.21.15.120:6825/254855304","cluster_addr":"172.21.15.120:6827/254855304","heartbeat_back_addr":"172.21.15.120:6831/254855304","heartbeat_front_addr":"172.21.15.120:6829/254855304","state":["exists","up"]},{"osd":11,"uuid":"06c5dd00-db2a-43e0-98e9-61468771c3ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6825","nonce":246219377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6827","nonce":246219377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6831","nonce":246219377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":246219377},{"type":"v1","addr":"172.21.15.174:6829","nonce":246219377}]},"public_addr":"172.21.15.174:6825/246219377","cluster_addr":"172.21.15.174:6827/246219377","heartbeat_back_addr":"172.21.15.174:6831/246219377","heartbeat_front_addr":"172.21.15.174:6829/246219377","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:22.187777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:16:46.575952+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:12.348431+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:17:37.603161+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:03.646565+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:29.897757+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:18:53.973334+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:18.735682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:19:44.164202+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:10.219107+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:34.665670+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:20:59.938908+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.11","mappings":[{"from":7,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.89:6801/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:6800/429862840":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/1174639743":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/3889269108":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/4118871044":"2024-09-16T23:15:00.884149+0000","172.21.15.89:0/379727762":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6801/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:6800/4057509998":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/3796595264":"2024-09-16T23:14:32.424565+0000","172.21.15.89:0/1030688282":"2024-09-16T23:14:32.424565+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:41:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 -- 172.21.15.89:0/2557710364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fab940a32c0 msgr2=0x7fab940a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fab940a32c0 0x7fab940a5780 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7faba80fedf0 tx=0x7fab90005f90 comp rx=0 tx=0).stop 2024-09-15T23:41:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 -- 172.21.15.89:0/2557710364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba8100a80 msgr2=0x7faba81973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba8100a80 0x7faba81973b0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fab98009230 tx=0x7fab98004040 comp rx=0 tx=0).stop 2024-09-15T23:41:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 -- 172.21.15.89:0/2557710364 shutdown_connections 2024-09-15T23:41:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fab940a32c0 0x7fab940a5780 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba8106a80 0x7faba819bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:23.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faba8101420 0x7faba81978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:23.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 --2- 172.21.15.89:0/2557710364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faba8100a80 0x7faba81973b0 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:23.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 -- 172.21.15.89:0/2557710364 >> 172.21.15.89:0/2557710364 conn(0x7faba80fc7a0 msgr2=0x7faba80fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:23.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 -- 172.21.15.89:0/2557710364 shutdown_connections 2024-09-15T23:41:23.769 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:23.767+0000 7fabadf95640 1 -- 172.21.15.89:0/2557710364 wait complete. 2024-09-15T23:41:24.554 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:24 smithi089 ceph-mon[30933]: pgmap v871: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-15T23:41:24.554 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:24 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2557710364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:41:24.583 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs fail cephfs 2024-09-15T23:41:24.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:24 smithi120 ceph-mon[35250]: pgmap v871: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-15T23:41:24.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:24 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2557710364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:41:24.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:24 smithi174 ceph-mon[35586]: pgmap v871: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-09-15T23:41:24.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:24 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2557710364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:41:24.831 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:25.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.320+0000 7fe3ad26e640 1 -- 172.21.15.89:0/2435339178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a8106f10 msgr2=0x7fe3a810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:25.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.320+0000 7fe3ad26e640 1 --2- 172.21.15.89:0/2435339178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a8106f10 0x7fe3a810ea90 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fe398009ca0 tx=0x7fe39802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:41:25.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.320+0000 7fe3ad26e640 1 -- 172.21.15.89:0/2435339178 shutdown_connections 2024-09-15T23:41:25.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.320+0000 7fe3ad26e640 1 --2- 172.21.15.89:0/2435339178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a8106f10 0x7fe3a810ea90 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:25.350 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.320+0000 7fe3ad26e640 1 --2- 172.21.15.89:0/2435339178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8102790 0x7fe3a81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.320+0000 7fe3ad26e640 1 --2- 172.21.15.89:0/2435339178 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3a8101de0 0x7fe3a81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.320+0000 7fe3ad26e640 1 -- 172.21.15.89:0/2435339178 >> 172.21.15.89:0/2435339178 conn(0x7fe3a80fd8c0 msgr2=0x7fe3a80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.320+0000 7fe3ad26e640 1 -- 172.21.15.89:0/2435339178 shutdown_connections 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.320+0000 7fe3ad26e640 1 -- 172.21.15.89:0/2435339178 wait complete. 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3ad26e640 1 Processor -- start 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3ad26e640 1 -- start start 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3ad26e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3a8101de0 0x7fe3a80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3ad26e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8102790 0x7fe3a8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3ad26e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a8106f10 0x7fe3a8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3ad26e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a8111f70 con 0x7fe3a8106f10 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3ad26e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a8111df0 con 0x7fe3a8102790 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3ad26e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81120f0 con 0x7fe3a8101de0 2024-09-15T23:41:25.351 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3a77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a8106f10 0x7fe3a8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3a77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a8106f10 0x7fe3a8074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35332/0 (socket says 172.21.15.89:35332) 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.321+0000 7fe3a77fe640 1 -- 172.21.15.89:0/3679166585 learned_addr learned my addr 172.21.15.89:0/3679166585 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe3a6ffd640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3a8101de0 0x7fe3a80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe3a67fc640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8102790 0x7fe3a8074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe3a77fe640 1 -- 172.21.15.89:0/3679166585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3a8101de0 msgr2=0x7fe3a80744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe3a77fe640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3a8101de0 0x7fe3a80744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe3a77fe640 1 -- 172.21.15.89:0/3679166585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8102790 msgr2=0x7fe3a8074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe3a77fe640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8102790 0x7fe3a8074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe3a77fe640 1 -- 172.21.15.89:0/3679166585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a81a6870 con 0x7fe3a8106f10 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe3a77fe640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a8106f10 0x7fe3a8074f70 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fe398031a20 tx=0x7fe398031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe38ffff640 1 -- 172.21.15.89:0/3679166585 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398003b70 con 0x7fe3a8106f10 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe38ffff640 1 -- 172.21.15.89:0/3679166585 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe398003d10 con 0x7fe3a8106f10 2024-09-15T23:41:25.352 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe3ad26e640 1 -- 172.21.15.89:0/3679166585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a81a6b00 con 0x7fe3a8106f10 2024-09-15T23:41:25.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe38ffff640 1 -- 172.21.15.89:0/3679166585 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39802fc80 con 0x7fe3a8106f10 2024-09-15T23:41:25.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.322+0000 7fe3ad26e640 1 -- 172.21.15.89:0/3679166585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a81a7040 con 0x7fe3a8106f10 2024-09-15T23:41:25.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.323+0000 7fe3ad26e640 1 -- 172.21.15.89:0/3679166585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3a81021c0 con 0x7fe3a8106f10 2024-09-15T23:41:25.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.324+0000 7fe38ffff640 1 -- 172.21.15.89:0/3679166585 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fe3980046a0 con 0x7fe3a8106f10 2024-09-15T23:41:25.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.325+0000 7fe38ffff640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3900a3240 0x7fe3900a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:25.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.325+0000 7fe3a6ffd640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3900a3240 0x7fe3900a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:25.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.325+0000 7fe38ffff640 1 -- 172.21.15.89:0/3679166585 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9237+0+0 (secure 0 0 0) 0x7fe3980f0eb0 con 0x7fe3a8106f10 2024-09-15T23:41:25.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.325+0000 7fe3a6ffd640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3900a3240 0x7fe3900a5700 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0005fd0 tx=0x7fe3a000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:25.353 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.329+0000 7fe38ffff640 1 -- 172.21.15.89:0/3679166585 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe3980bd870 con 0x7fe3a8106f10 2024-09-15T23:41:25.533 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.532+0000 7fe3ad26e640 1 -- 172.21.15.89:0/3679166585 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe3a81a7270 con 0x7fe3a8106f10 2024-09-15T23:41:25.535 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.534+0000 7fe38ffff640 1 -- 172.21.15.89:0/3679166585 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) ==== 161+0+0 (secure 0 0 0) 0x7fe3980bd170 con 0x7fe3a8106f10 2024-09-15T23:41:25.535 INFO:teuthology.orchestra.run.smithi089.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-15T23:41:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.538+0000 7fe3ad26e640 1 -- 172.21.15.89:0/3679166585 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3900a3240 msgr2=0x7fe3900a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.538+0000 7fe3ad26e640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3900a3240 0x7fe3900a5700 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0005fd0 tx=0x7fe3a000b390 comp rx=0 tx=0).stop 2024-09-15T23:41:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.538+0000 7fe3ad26e640 1 -- 172.21.15.89:0/3679166585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a8106f10 msgr2=0x7fe3a8074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:25.539 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.538+0000 7fe3ad26e640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a8106f10 0x7fe3a8074f70 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fe398031a20 tx=0x7fe398031a50 comp rx=0 tx=0).stop 2024-09-15T23:41:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.538+0000 7fe3ad26e640 1 -- 172.21.15.89:0/3679166585 shutdown_connections 2024-09-15T23:41:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.538+0000 7fe3ad26e640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fe3900a3240 0x7fe3900a5700 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.538+0000 7fe3ad26e640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a8106f10 0x7fe3a8074f70 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.538+0000 7fe3ad26e640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8102790 0x7fe3a8074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.538+0000 7fe3ad26e640 1 --2- 172.21.15.89:0/3679166585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3a8101de0 0x7fe3a80744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.538+0000 7fe3ad26e640 1 -- 172.21.15.89:0/3679166585 >> 172.21.15.89:0/3679166585 conn(0x7fe3a80fd8c0 msgr2=0x7fe3a80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.539+0000 7fe3ad26e640 1 -- 172.21.15.89:0/3679166585 shutdown_connections 2024-09-15T23:41:25.540 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:25.539+0000 7fe3ad26e640 1 -- 172.21.15.89:0/3679166585 wait complete. 2024-09-15T23:41:25.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:25 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a[30929]: 2024-09-15T23:41:25.533+0000 7fae0d819640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T23:41:25.810 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:41:25 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a[71412]: ignoring --setuser ceph since I am not root 2024-09-15T23:41:25.810 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:41:25 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a[71412]: ignoring --setgroup ceph since I am not root 2024-09-15T23:41:25.811 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:41:25 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a[71412]: starting mds.a at 2024-09-15T23:41:25.811 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:41:25 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g[71076]: ignoring --setuser ceph since I am not root 2024-09-15T23:41:25.811 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:41:25 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g[71076]: ignoring --setgroup ceph since I am not root 2024-09-15T23:41:25.811 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:41:25 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g[71076]: starting mds.g at 2024-09-15T23:41:25.977 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e[64573]: ignoring --setuser ceph since I am not root 2024-09-15T23:41:25.977 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e[64573]: ignoring --setgroup ceph since I am not root 2024-09-15T23:41:25.977 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-mds[64577]: -- 172.21.15.120:0/1849413709 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557c5e957860 con 0x557c5e934800 2024-09-15T23:41:25.977 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e[64573]: starting mds.e at 2024-09-15T23:41:25.978 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h[64878]: ignoring --setuser ceph since I am not root 2024-09-15T23:41:25.978 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h[64878]: ignoring --setgroup ceph since I am not root 2024-09-15T23:41:25.978 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-mds[64882]: -- 172.21.15.120:0/3806721109 <== mon.2 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55ee512a8680 con 0x55ee5127c800 2024-09-15T23:41:25.978 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-mds[64882]: -- 172.21.15.120:0/3806721109 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ee5129f860 con 0x55ee5127c800 2024-09-15T23:41:25.978 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h[64878]: starting mds.h at 2024-09-15T23:41:25.978 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k[64272]: ignoring --setuser ceph since I am not root 2024-09-15T23:41:25.978 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k[64272]: ignoring --setgroup ceph since I am not root 2024-09-15T23:41:25.978 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-mds[64276]: -- 172.21.15.120:0/593576150 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e9da825860 con 0x55e9da803400 2024-09-15T23:41:25.979 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k[64272]: starting mds.k at 2024-09-15T23:41:25.979 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b[65175]: ignoring --setuser ceph since I am not root 2024-09-15T23:41:25.979 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b[65175]: ignoring --setgroup ceph since I am not root 2024-09-15T23:41:25.979 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-mds[65179]: -- 172.21.15.120:0/660371808 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56271a11f860 con 0x56271a0fd000 2024-09-15T23:41:25.979 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:41:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b[65175]: starting mds.b at 2024-09-15T23:41:25.985 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f[65528]: ignoring --setuser ceph since I am not root 2024-09-15T23:41:25.985 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f[65528]: ignoring --setgroup ceph since I am not root 2024-09-15T23:41:25.986 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-mds[65532]: -- 172.21.15.174:0/3551252749 <== mon.0 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5638c24bd860 con 0x5638c249b000 2024-09-15T23:41:25.986 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f[65528]: starting mds.f at 2024-09-15T23:41:25.986 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i[66128]: ignoring --setuser ceph since I am not root 2024-09-15T23:41:25.986 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i[66128]: ignoring --setgroup ceph since I am not root 2024-09-15T23:41:25.986 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-mds[66132]: -- 172.21.15.174:0/1048363134 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55de5c51b860 con 0x55de5c4f8800 2024-09-15T23:41:25.986 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i[66128]: starting mds.i at 2024-09-15T23:41:25.986 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l[65222]: ignoring --setuser ceph since I am not root 2024-09-15T23:41:25.987 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l[65222]: ignoring --setgroup ceph since I am not root 2024-09-15T23:41:25.987 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-mds[65226]: -- 172.21.15.174:0/4159384332 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563802eb9c20 con 0x563802e97000 2024-09-15T23:41:25.987 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l[65222]: starting mds.l at 2024-09-15T23:41:25.987 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c[65829]: ignoring --setuser ceph since I am not root 2024-09-15T23:41:25.987 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c[65829]: ignoring --setgroup ceph since I am not root 2024-09-15T23:41:25.987 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-mds[65833]: -- 172.21.15.174:0/3574027548 <== mon.2 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55656a32d860 con 0x55656a30b000 2024-09-15T23:41:25.987 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:41:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c[65829]: starting mds.c at 2024-09-15T23:41:26.368 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-15T23:41:26.623 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:26.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:26 smithi120 ceph-mon[35250]: pgmap v872: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:41:26.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:26 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3679166585' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-15T23:41:26.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:26 smithi120 ceph-mon[35250]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-15T23:41:26.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:26 smithi120 ceph-mon[35250]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T23:41:26.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:26 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3679166585' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-15T23:41:26.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:26 smithi120 ceph-mon[35250]: osdmap e91: 12 total, 12 up, 12 in 2024-09-15T23:41:26.724 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:26 smithi120 ceph-mon[35250]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-15T23:41:26.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:26 smithi174 ceph-mon[35586]: pgmap v872: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:41:26.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:26 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3679166585' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-15T23:41:26.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:26 smithi174 ceph-mon[35586]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-15T23:41:26.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:26 smithi174 ceph-mon[35586]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T23:41:26.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:26 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3679166585' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-15T23:41:26.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:26 smithi174 ceph-mon[35586]: osdmap e91: 12 total, 12 up, 12 in 2024-09-15T23:41:26.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:26 smithi174 ceph-mon[35586]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-15T23:41:26.743 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:26 smithi089 ceph-mon[30933]: pgmap v872: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-09-15T23:41:26.744 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:26 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3679166585' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-15T23:41:26.744 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:26 smithi089 ceph-mon[30933]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-15T23:41:26.744 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:26 smithi089 ceph-mon[30933]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T23:41:26.744 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:26 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3679166585' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-15T23:41:26.744 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:26 smithi089 ceph-mon[30933]: osdmap e91: 12 total, 12 up, 12 in 2024-09-15T23:41:26.744 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:26 smithi089 ceph-mon[30933]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-09-15T23:41:27.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.066+0000 7fd8b7355640 1 -- 172.21.15.89:0/1817312626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8b0100a80 msgr2=0x7fd8b0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:27.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.066+0000 7fd8b7355640 1 --2- 172.21.15.89:0/1817312626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8b0100a80 0x7fd8b0100ee0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0009930 tx=0x7fd8a002a420 comp rx=0 tx=0).stop 2024-09-15T23:41:27.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.066+0000 7fd8b7355640 1 -- 172.21.15.89:0/1817312626 shutdown_connections 2024-09-15T23:41:27.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.066+0000 7fd8b7355640 1 --2- 172.21.15.89:0/1817312626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b0101420 0x7fd8b010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.066+0000 7fd8b7355640 1 --2- 172.21.15.89:0/1817312626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8b0100a80 0x7fd8b0100ee0 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.066+0000 7fd8b7355640 1 --2- 172.21.15.89:0/1817312626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8b0106a80 0x7fd8b0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.066+0000 7fd8b7355640 1 -- 172.21.15.89:0/1817312626 >> 172.21.15.89:0/1817312626 conn(0x7fd8b00fc7a0 msgr2=0x7fd8b00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:27.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.066+0000 7fd8b7355640 1 -- 172.21.15.89:0/1817312626 shutdown_connections 2024-09-15T23:41:27.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.066+0000 7fd8b7355640 1 -- 172.21.15.89:0/1817312626 wait complete. 2024-09-15T23:41:27.110 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.067+0000 7fd8b7355640 1 Processor -- start 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.067+0000 7fd8b7355640 1 -- start start 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.067+0000 7fd8b7355640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8b0100a80 0x7fd8b01973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.067+0000 7fd8b7355640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b0101420 0x7fd8b0197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.067+0000 7fd8b7355640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8b0106a80 0x7fd8b019bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.067+0000 7fd8b7355640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b0110e80 con 0x7fd8b0100a80 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.067+0000 7fd8b7355640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b0110d00 con 0x7fd8b0101420 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.067+0000 7fd8b7355640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b0111000 con 0x7fd8b0106a80 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b50f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8b0100a80 0x7fd8b01973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b50f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8b0100a80 0x7fd8b01973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35382/0 (socket says 172.21.15.89:35382) 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b50f1640 1 -- 172.21.15.89:0/1592957626 learned_addr learned my addr 172.21.15.89:0/1592957626 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b58f2640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8b0106a80 0x7fd8b019bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b50f1640 1 -- 172.21.15.89:0/1592957626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8b0106a80 msgr2=0x7fd8b019bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b50f1640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8b0106a80 0x7fd8b019bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.111 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b50f1640 1 -- 172.21.15.89:0/1592957626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b0101420 msgr2=0x7fd8b0197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b48f0640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b0101420 0x7fd8b0197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b50f1640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b0101420 0x7fd8b0197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b50f1640 1 -- 172.21.15.89:0/1592957626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b019c370 con 0x7fd8b0100a80 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b50f1640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8b0100a80 0x7fd8b01973c0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00e9f0 tx=0x7fd8ac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8a67fc640 1 -- 172.21.15.89:0/1592957626 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac00cd80 con 0x7fd8b0100a80 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b7355640 1 -- 172.21.15.89:0/1592957626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b019c660 con 0x7fd8b0100a80 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8a67fc640 1 -- 172.21.15.89:0/1592957626 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8ac005500 con 0x7fd8b0100a80 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8a67fc640 1 -- 172.21.15.89:0/1592957626 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac010690 con 0x7fd8b0100a80 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.068+0000 7fd8b7355640 1 -- 172.21.15.89:0/1592957626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8b01a3f40 con 0x7fd8b0100a80 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.069+0000 7fd8b7355640 1 -- 172.21.15.89:0/1592957626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8b0100ee0 con 0x7fd8b0100a80 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.070+0000 7fd8a67fc640 1 -- 172.21.15.89:0/1592957626 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd8ac0046c0 con 0x7fd8b0100a80 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.071+0000 7fd8a67fc640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd89c0a3240 0x7fd89c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:27.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.071+0000 7fd8b48f0640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd89c0a3240 0x7fd89c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:27.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.071+0000 7fd8a67fc640 1 -- 172.21.15.89:0/1592957626 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9667+0+0 (secure 0 0 0) 0x7fd8ac0d03b0 con 0x7fd8b0100a80 2024-09-15T23:41:27.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.071+0000 7fd8b48f0640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd89c0a3240 0x7fd89c0a5700 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00096c0 tx=0x7fd8a0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:27.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.075+0000 7fd8a67fc640 1 -- 172.21.15.89:0/1592957626 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd8ac09cb30 con 0x7fd8b0100a80 2024-09-15T23:41:27.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.276+0000 7fd8b7355640 1 -- 172.21.15.89:0/1592957626 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd8b002d7c0 con 0x7fd8b0100a80 2024-09-15T23:41:27.544 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.543+0000 7fd8a67fc640 1 -- 172.21.15.89:0/1592957626 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v32) ==== 104+0+0 (secure 0 0 0) 0x7fd8ac09c430 con 0x7fd8b0100a80 2024-09-15T23:41:27.547 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.546+0000 7fd8b7355640 1 -- 172.21.15.89:0/1592957626 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd89c0a3240 msgr2=0x7fd89c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:27.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.546+0000 7fd8b7355640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd89c0a3240 0x7fd89c0a5700 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00096c0 tx=0x7fd8a0000bc0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.546+0000 7fd8b7355640 1 -- 172.21.15.89:0/1592957626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8b0100a80 msgr2=0x7fd8b01973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:27.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.546+0000 7fd8b7355640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8b0100a80 0x7fd8b01973c0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00e9f0 tx=0x7fd8ac00edf0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.546+0000 7fd8b7355640 1 -- 172.21.15.89:0/1592957626 shutdown_connections 2024-09-15T23:41:27.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.546+0000 7fd8b7355640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd89c0a3240 0x7fd89c0a5700 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.546+0000 7fd8b7355640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8b0106a80 0x7fd8b019bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.546+0000 7fd8b7355640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd8b0101420 0x7fd8b0197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.546+0000 7fd8b7355640 1 --2- 172.21.15.89:0/1592957626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd8b0100a80 0x7fd8b01973c0 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:27.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.546+0000 7fd8b7355640 1 -- 172.21.15.89:0/1592957626 >> 172.21.15.89:0/1592957626 conn(0x7fd8b00fc7a0 msgr2=0x7fd8b00fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:27.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.547+0000 7fd8b7355640 1 -- 172.21.15.89:0/1592957626 shutdown_connections 2024-09-15T23:41:27.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:27.547+0000 7fd8b7355640 1 -- 172.21.15.89:0/1592957626 wait complete. 2024-09-15T23:41:27.813 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.120:6833/2158898243,v1:172.21.15.120:6835/2158898243] up:boot 2024-09-15T23:41:27.813 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.120:6832/1787569715,v1:172.21.15.120:6834/1787569715] up:boot 2024-09-15T23:41:27.813 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6832/2871886790,v1:172.21.15.174:6834/2871886790] up:boot 2024-09-15T23:41:27.814 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6833/577652661,v1:172.21.15.174:6835/577652661] up:boot 2024-09-15T23:41:27.814 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.89:6839/2544751689,v1:172.21.15.89:6841/2544751689] up:boot 2024-09-15T23:41:27.814 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.89:6838/1128649724,v1:172.21.15.89:6840/1128649724] up:boot 2024-09-15T23:41:27.814 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6836/4070817273,v1:172.21.15.174:6837/4070817273] up:boot 2024-09-15T23:41:27.814 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.174:6838/3241283725,v1:172.21.15.174:6839/3241283725] up:boot 2024-09-15T23:41:27.815 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.120:6836/2092357766,v1:172.21.15.120:6837/2092357766] up:boot 2024-09-15T23:41:27.815 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: mds.? [v2:172.21.15.120:6838/1051328248,v1:172.21.15.120:6839/1051328248] up:boot 2024-09-15T23:41:27.815 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-15T23:41:27.815 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:41:27.815 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:41:27.815 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T23:41:27.815 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T23:41:27.816 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:41:27.816 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:41:27.816 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:41:27.816 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T23:41:27.816 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T23:41:27.816 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:41:27.816 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:27 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1592957626' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:41:27.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.120:6833/2158898243,v1:172.21.15.120:6835/2158898243] up:boot 2024-09-15T23:41:27.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.120:6832/1787569715,v1:172.21.15.120:6834/1787569715] up:boot 2024-09-15T23:41:27.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6832/2871886790,v1:172.21.15.174:6834/2871886790] up:boot 2024-09-15T23:41:27.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6833/577652661,v1:172.21.15.174:6835/577652661] up:boot 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.89:6839/2544751689,v1:172.21.15.89:6841/2544751689] up:boot 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.89:6838/1128649724,v1:172.21.15.89:6840/1128649724] up:boot 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6836/4070817273,v1:172.21.15.174:6837/4070817273] up:boot 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.174:6838/3241283725,v1:172.21.15.174:6839/3241283725] up:boot 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.120:6836/2092357766,v1:172.21.15.120:6837/2092357766] up:boot 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: mds.? [v2:172.21.15.120:6838/1051328248,v1:172.21.15.120:6839/1051328248] up:boot 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:41:27.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:41:27.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:41:27.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T23:41:27.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T23:41:27.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:41:27.975 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:27 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1592957626' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.120:6833/2158898243,v1:172.21.15.120:6835/2158898243] up:boot 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.120:6832/1787569715,v1:172.21.15.120:6834/1787569715] up:boot 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6832/2871886790,v1:172.21.15.174:6834/2871886790] up:boot 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6833/577652661,v1:172.21.15.174:6835/577652661] up:boot 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.89:6839/2544751689,v1:172.21.15.89:6841/2544751689] up:boot 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.89:6838/1128649724,v1:172.21.15.89:6840/1128649724] up:boot 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6836/4070817273,v1:172.21.15.174:6837/4070817273] up:boot 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.174:6838/3241283725,v1:172.21.15.174:6839/3241283725] up:boot 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.120:6836/2092357766,v1:172.21.15.120:6837/2092357766] up:boot 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: mds.? [v2:172.21.15.120:6838/1051328248,v1:172.21.15.120:6839/1051328248] up:boot 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:41:27.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:41:27.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T23:41:27.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T23:41:27.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:41:27.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:41:27.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:41:27.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T23:41:27.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T23:41:27.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='mgr.14150 172.21.15.89:0/3842463744' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:41:27.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:27 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1592957626' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:41:28.361 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-15T23:41:28.607 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:28.728 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:28 smithi089 ceph-mon[30933]: pgmap v874: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 716 B/s wr, 10 op/s 2024-09-15T23:41:28.728 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:28 smithi089 ceph-mon[30933]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-15T23:41:28.728 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:28 smithi089 ceph-mon[30933]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-15T23:41:28.728 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:28 smithi089 ceph-mon[30933]: Cluster is now healthy 2024-09-15T23:41:28.728 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:28 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/1592957626' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:41:28.728 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:28 smithi089 ceph-mon[30933]: fsmap 12 up:standby 2024-09-15T23:41:28.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:28 smithi120 ceph-mon[35250]: pgmap v874: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 716 B/s wr, 10 op/s 2024-09-15T23:41:28.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:28 smithi120 ceph-mon[35250]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-15T23:41:28.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:28 smithi120 ceph-mon[35250]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-15T23:41:28.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:28 smithi120 ceph-mon[35250]: Cluster is now healthy 2024-09-15T23:41:28.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:28 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/1592957626' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:41:28.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:28 smithi120 ceph-mon[35250]: fsmap 12 up:standby 2024-09-15T23:41:28.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:28 smithi174 ceph-mon[35586]: pgmap v874: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 716 B/s wr, 10 op/s 2024-09-15T23:41:28.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:28 smithi174 ceph-mon[35586]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-15T23:41:28.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:28 smithi174 ceph-mon[35586]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-15T23:41:28.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:28 smithi174 ceph-mon[35586]: Cluster is now healthy 2024-09-15T23:41:28.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:28 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/1592957626' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:41:28.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:28 smithi174 ceph-mon[35586]: fsmap 12 up:standby 2024-09-15T23:41:29.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.118+0000 7f65feef3640 1 -- 172.21.15.89:0/4082714011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65f8102790 msgr2=0x7f65f81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:29.144 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.118+0000 7f65feef3640 1 --2- 172.21.15.89:0/4082714011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65f8102790 0x7f65f81067e0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009900 tx=0x7f65e402a410 comp rx=0 tx=0).stop 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.118+0000 7f65feef3640 1 -- 172.21.15.89:0/4082714011 shutdown_connections 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.118+0000 7f65feef3640 1 --2- 172.21.15.89:0/4082714011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8106f10 0x7f65f810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.118+0000 7f65feef3640 1 --2- 172.21.15.89:0/4082714011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65f8102790 0x7f65f81067e0 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.118+0000 7f65feef3640 1 --2- 172.21.15.89:0/4082714011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f8101de0 0x7f65f81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.118+0000 7f65feef3640 1 -- 172.21.15.89:0/4082714011 >> 172.21.15.89:0/4082714011 conn(0x7f65f80fd8c0 msgr2=0x7f65f80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.118+0000 7f65feef3640 1 -- 172.21.15.89:0/4082714011 shutdown_connections 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.118+0000 7f65feef3640 1 -- 172.21.15.89:0/4082714011 wait complete. 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65feef3640 1 Processor -- start 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65feef3640 1 -- start start 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65feef3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f8101de0 0x7f65f80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65feef3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65f8102790 0x7f65f8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65feef3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8106f10 0x7f65f8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:29.145 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65feef3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8111f90 con 0x7f65f8102790 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65feef3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8111e10 con 0x7f65f8106f10 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65feef3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8112110 con 0x7f65f8101de0 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65f7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65f8102790 0x7f65f8074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65f7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65f8102790 0x7f65f8074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35396/0 (socket says 172.21.15.89:35396) 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65f7fff640 1 -- 172.21.15.89:0/2958699038 learned_addr learned my addr 172.21.15.89:0/2958699038 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65fcc8f640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f8101de0 0x7f65f80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65fd490640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8106f10 0x7f65f8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65f7fff640 1 -- 172.21.15.89:0/2958699038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f8101de0 msgr2=0x7f65f80744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65f7fff640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f8101de0 0x7f65f80744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65f7fff640 1 -- 172.21.15.89:0/2958699038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8106f10 msgr2=0x7f65f8074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65f7fff640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8106f10 0x7f65f8074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.119+0000 7f65f7fff640 1 -- 172.21.15.89:0/2958699038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f81a6930 con 0x7f65f8102790 2024-09-15T23:41:29.146 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.120+0000 7f65f7fff640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65f8102790 0x7f65f8074a30 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f65e4006010 tx=0x7f65e40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.120+0000 7f65f5ffb640 1 -- 172.21.15.89:0/2958699038 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e403f030 con 0x7f65f8102790 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.120+0000 7f65f5ffb640 1 -- 172.21.15.89:0/2958699038 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e4003ce0 con 0x7f65f8102790 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.120+0000 7f65feef3640 1 -- 172.21.15.89:0/2958699038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f81a6bc0 con 0x7f65f8102790 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.120+0000 7f65f5ffb640 1 -- 172.21.15.89:0/2958699038 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e4035070 con 0x7f65f8102790 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.120+0000 7f65feef3640 1 -- 172.21.15.89:0/2958699038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f81a70a0 con 0x7f65f8102790 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.121+0000 7f65feef3640 1 -- 172.21.15.89:0/2958699038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f81021c0 con 0x7f65f8102790 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.122+0000 7f65f5ffb640 1 -- 172.21.15.89:0/2958699038 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f65e4042030 con 0x7f65f8102790 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.123+0000 7f65f5ffb640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f65e80a3240 0x7f65e80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.123+0000 7f65f5ffb640 1 -- 172.21.15.89:0/2958699038 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9667+0+0 (secure 0 0 0) 0x7f65e40f0310 con 0x7f65f8102790 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.123+0000 7f65fcc8f640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f65e80a3240 0x7f65e80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.123+0000 7f65fcc8f640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f65e80a3240 0x7f65e80a5700 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00bbe0 tx=0x7f65ec00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:29.147 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.127+0000 7f65f5ffb640 1 -- 172.21.15.89:0/2958699038 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f65e40bc500 con 0x7f65f8102790 2024-09-15T23:41:29.294 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.293+0000 7f65feef3640 1 -- 172.21.15.89:0/2958699038 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f65f810a0f0 con 0x7f65f8102790 2024-09-15T23:41:29.552 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.551+0000 7f65f5ffb640 1 -- 172.21.15.89:0/2958699038 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v92) ==== 181+0+0 (secure 0 0 0) 0x7f65e40c0800 con 0x7f65f8102790 2024-09-15T23:41:29.552 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_metadata' removed 2024-09-15T23:41:29.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.554+0000 7f65feef3640 1 -- 172.21.15.89:0/2958699038 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f65e80a3240 msgr2=0x7f65e80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:29.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.554+0000 7f65feef3640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f65e80a3240 0x7f65e80a5700 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00bbe0 tx=0x7f65ec00b390 comp rx=0 tx=0).stop 2024-09-15T23:41:29.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.554+0000 7f65feef3640 1 -- 172.21.15.89:0/2958699038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65f8102790 msgr2=0x7f65f8074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:29.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.554+0000 7f65feef3640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65f8102790 0x7f65f8074a30 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f65e4006010 tx=0x7f65e40042f0 comp rx=0 tx=0).stop 2024-09-15T23:41:29.556 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.554+0000 7f65feef3640 1 -- 172.21.15.89:0/2958699038 shutdown_connections 2024-09-15T23:41:29.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.554+0000 7f65feef3640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f65e80a3240 0x7f65e80a5700 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:29.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.554+0000 7f65feef3640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8106f10 0x7f65f8074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:29.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.554+0000 7f65feef3640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65f8102790 0x7f65f8074a30 secure :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f65e4006010 tx=0x7f65e40042f0 comp rx=0 tx=0).stop 2024-09-15T23:41:29.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.554+0000 7f65feef3640 1 --2- 172.21.15.89:0/2958699038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f8101de0 0x7f65f80744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:29.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.554+0000 7f65feef3640 1 -- 172.21.15.89:0/2958699038 >> 172.21.15.89:0/2958699038 conn(0x7f65f80fd8c0 msgr2=0x7f65f80fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:29.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.555+0000 7f65feef3640 1 -- 172.21.15.89:0/2958699038 shutdown_connections 2024-09-15T23:41:29.557 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:29.555+0000 7f65feef3640 1 -- 172.21.15.89:0/2958699038 wait complete. 2024-09-15T23:41:29.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:29 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2958699038' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:41:29.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:29 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2958699038' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:41:29.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:29 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2958699038' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:41:30.342 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-15T23:41:30.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:30 smithi089 ceph-mon[30933]: pgmap v875: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 716 B/s wr, 10 op/s 2024-09-15T23:41:30.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:30 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2958699038' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:41:30.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:30 smithi089 ceph-mon[30933]: osdmap e92: 12 total, 12 up, 12 in 2024-09-15T23:41:30.587 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:30.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:30 smithi120 ceph-mon[35250]: pgmap v875: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 716 B/s wr, 10 op/s 2024-09-15T23:41:30.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:30 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2958699038' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:41:30.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:30 smithi120 ceph-mon[35250]: osdmap e92: 12 total, 12 up, 12 in 2024-09-15T23:41:30.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:30 smithi174 ceph-mon[35586]: pgmap v875: 129 pgs: 129 active+clean; 4.2 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 716 B/s wr, 10 op/s 2024-09-15T23:41:30.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:30 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2958699038' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:41:30.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:30 smithi174 ceph-mon[35586]: osdmap e92: 12 total, 12 up, 12 in 2024-09-15T23:41:31.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.096+0000 7f4b17a9a640 1 -- 172.21.15.89:0/1618343263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b10100a80 msgr2=0x7f4b10100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:31.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.096+0000 7f4b17a9a640 1 --2- 172.21.15.89:0/1618343263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b10100a80 0x7f4b10100ee0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009930 tx=0x7f4b0002a420 comp rx=0 tx=0).stop 2024-09-15T23:41:31.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.096+0000 7f4b17a9a640 1 -- 172.21.15.89:0/1618343263 shutdown_connections 2024-09-15T23:41:31.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.096+0000 7f4b17a9a640 1 --2- 172.21.15.89:0/1618343263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b10101420 0x7f4b1010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:31.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.096+0000 7f4b17a9a640 1 --2- 172.21.15.89:0/1618343263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b10100a80 0x7f4b10100ee0 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:31.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.096+0000 7f4b17a9a640 1 --2- 172.21.15.89:0/1618343263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b10106a80 0x7f4b10106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:31.112 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.096+0000 7f4b17a9a640 1 -- 172.21.15.89:0/1618343263 >> 172.21.15.89:0/1618343263 conn(0x7f4b100fc7a0 msgr2=0x7f4b100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.096+0000 7f4b17a9a640 1 -- 172.21.15.89:0/1618343263 shutdown_connections 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.096+0000 7f4b17a9a640 1 -- 172.21.15.89:0/1618343263 wait complete. 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b17a9a640 1 Processor -- start 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b17a9a640 1 -- start start 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b17a9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b10100a80 0x7f4b100fccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b17a9a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b10101420 0x7f4b100fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b17a9a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b10106a80 0x7f4b100fd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b17a9a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7f4b10110ea0 con 0x7f4b10101420 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b17a9a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4b10110d20 con 0x7f4b10106a80 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b17a9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4b10111020 con 0x7f4b10100a80 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b15035640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b10101420 0x7f4b100fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b15035640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b10101420 0x7f4b100fd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35414/0 (socket says 172.21.15.89:35414) 2024-09-15T23:41:31.113 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b15035640 1 -- 172.21.15.89:0/2049481714 learned_addr learned my addr 172.21.15.89:0/2049481714 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b15836640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b10100a80 0x7f4b100fccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b15035640 1 -- 172.21.15.89:0/2049481714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b10100a80 msgr2=0x7f4b100fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b15035640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b10100a80 0x7f4b100fccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b15035640 1 -- 172.21.15.89:0/2049481714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b10106a80 msgr2=0x7f4b100fd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b16037640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b10106a80 0x7f4b100fd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b15035640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b10106a80 0x7f4b100fd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.097+0000 7f4b15035640 1 -- 172.21.15.89:0/2049481714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b101a6900 con 0x7f4b10101420 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.098+0000 7f4b15035640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b10101420 0x7f4b100fd230 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009a50 tx=0x7f4b00004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.098+0000 7f4b06ffd640 1 -- 172.21.15.89:0/2049481714 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0003f030 con 0x7f4b10101420 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.098+0000 7f4b06ffd640 1 -- 172.21.15.89:0/2049481714 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b00003ce0 con 0x7f4b10101420 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.098+0000 7f4b17a9a640 1 -- 172.21.15.89:0/2049481714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b101a6b90 con 0x7f4b10101420 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.098+0000 7f4b06ffd640 1 -- 172.21.15.89:0/2049481714 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b00035070 con 0x7f4b10101420 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.098+0000 7f4b17a9a640 1 -- 172.21.15.89:0/2049481714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b101a7070 con 0x7f4b10101420 2024-09-15T23:41:31.114 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.099+0000 7f4b17a9a640 1 -- 172.21.15.89:0/2049481714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b10100ee0 con 0x7f4b10101420 2024-09-15T23:41:31.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.100+0000 7f4b06ffd640 1 -- 172.21.15.89:0/2049481714 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7f4b00042030 con 0x7f4b10101420 2024-09-15T23:41:31.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.100+0000 7f4b06ffd640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4afc0a3240 0x7f4afc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:31.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.100+0000 7f4b15836640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4afc0a3240 0x7f4afc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:31.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.101+0000 7f4b06ffd640 1 -- 172.21.15.89:0/2049481714 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9084+0+0 (secure 0 0 0) 0x7f4b000f02f0 con 0x7f4b10101420 2024-09-15T23:41:31.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.101+0000 7f4b15836640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4afc0a3240 0x7f4afc0a5700 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c005e10 tx=0x7f4b0c005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:31.115 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.105+0000 7f4b06ffd640 1 -- 172.21.15.89:0/2049481714 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4b000bcc60 con 0x7f4b10101420 2024-09-15T23:41:31.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.273+0000 7f4b17a9a640 1 -- 172.21.15.89:0/2049481714 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4b10109a90 con 0x7f4b10101420 2024-09-15T23:41:31.559 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.558+0000 7f4b06ffd640 1 -- 172.21.15.89:0/2049481714 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v93) ==== 169+0+0 (secure 0 0 0) 0x7f4b000bc560 con 0x7f4b10101420 2024-09-15T23:41:31.559 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data' removed 2024-09-15T23:41:31.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.561+0000 7f4b17a9a640 1 -- 172.21.15.89:0/2049481714 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4afc0a3240 msgr2=0x7f4afc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:31.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.561+0000 7f4b17a9a640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4afc0a3240 0x7f4afc0a5700 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c005e10 tx=0x7f4b0c005da0 comp rx=0 tx=0).stop 2024-09-15T23:41:31.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.561+0000 7f4b17a9a640 1 -- 172.21.15.89:0/2049481714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b10101420 msgr2=0x7f4b100fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:31.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.561+0000 7f4b17a9a640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b10101420 0x7f4b100fd230 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f4b00009a50 tx=0x7f4b00004270 comp rx=0 tx=0).stop 2024-09-15T23:41:31.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.562+0000 7f4b17a9a640 1 -- 172.21.15.89:0/2049481714 shutdown_connections 2024-09-15T23:41:31.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.562+0000 7f4b17a9a640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7f4afc0a3240 0x7f4afc0a5700 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:31.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.562+0000 7f4b17a9a640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b10106a80 0x7f4b100fd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:31.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.562+0000 7f4b17a9a640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4b10101420 0x7f4b100fd230 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:31.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.562+0000 7f4b17a9a640 1 --2- 172.21.15.89:0/2049481714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b10100a80 0x7f4b100fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:31.563 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.562+0000 7f4b17a9a640 1 -- 172.21.15.89:0/2049481714 >> 172.21.15.89:0/2049481714 conn(0x7f4b100fc7a0 msgr2=0x7f4b1006a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:31.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.562+0000 7f4b17a9a640 1 -- 172.21.15.89:0/2049481714 shutdown_connections 2024-09-15T23:41:31.564 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:31.562+0000 7f4b17a9a640 1 -- 172.21.15.89:0/2049481714 wait complete. 2024-09-15T23:41:31.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:31 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2049481714' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:41:31.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:31 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2049481714' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:41:31.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:31 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2049481714' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:41:32.463 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-09-15T23:41:32.718 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:41:32.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:32 smithi089 ceph-mon[30933]: pgmap v877: 97 pgs: 97 active+clean; 577 KiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:41:32.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:32 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/2049481714' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:41:32.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:32 smithi089 ceph-mon[30933]: osdmap e93: 12 total, 12 up, 12 in 2024-09-15T23:41:32.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:32 smithi120 ceph-mon[35250]: pgmap v877: 97 pgs: 97 active+clean; 577 KiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:41:32.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:32 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/2049481714' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:41:32.974 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:32 smithi120 ceph-mon[35250]: osdmap e93: 12 total, 12 up, 12 in 2024-09-15T23:41:32.985 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:32 smithi174 ceph-mon[35586]: pgmap v877: 97 pgs: 97 active+clean; 577 KiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:41:32.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:32 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/2049481714' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:41:32.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:32 smithi174 ceph-mon[35586]: osdmap e93: 12 total, 12 up, 12 in 2024-09-15T23:41:33.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.213+0000 7fd96849a640 1 -- 172.21.15.89:0/3667826755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96010c170 msgr2=0x7fd96010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:33.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.213+0000 7fd96849a640 1 --2- 172.21.15.89:0/3667826755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96010c170 0x7fd96010e610 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7fd950009ca0 tx=0x7fd95002a950 comp rx=0 tx=0).stop 2024-09-15T23:41:33.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.213+0000 7fd96849a640 1 -- 172.21.15.89:0/3667826755 shutdown_connections 2024-09-15T23:41:33.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.213+0000 7fd96849a640 1 --2- 172.21.15.89:0/3667826755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96010c170 0x7fd96010e610 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:33.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.213+0000 7fd96849a640 1 --2- 172.21.15.89:0/3667826755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd960103660 0x7fd96010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:33.239 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.213+0000 7fd96849a640 1 --2- 172.21.15.89:0/3667826755 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd960102d40 0x7fd960103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.213+0000 7fd96849a640 1 -- 172.21.15.89:0/3667826755 >> 172.21.15.89:0/3667826755 conn(0x7fd9600fc7a0 msgr2=0x7fd9600febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.213+0000 7fd96849a640 1 -- 172.21.15.89:0/3667826755 shutdown_connections 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.213+0000 7fd96849a640 1 -- 172.21.15.89:0/3667826755 wait complete. 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.213+0000 7fd96849a640 1 Processor -- start 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd96849a640 1 -- start start 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd96849a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd960102d40 0x7fd9601009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd96849a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd960103660 0x7fd960100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd96849a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96010c170 0x7fd960101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd96849a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 -- 0x7fd960111fd0 con 0x7fd96010c170 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd96849a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd960111e50 con 0x7fd960103660 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd96849a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd960112150 con 0x7fd960102d40 2024-09-15T23:41:33.240 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd966a37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96010c170 0x7fd960101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd966a37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96010c170 0x7fd960101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35428/0 (socket says 172.21.15.89:35428) 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd966a37640 1 -- 172.21.15.89:0/3603694336 learned_addr learned my addr 172.21.15.89:0/3603694336 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd966236640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd960102d40 0x7fd9601009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd965a35640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd960103660 0x7fd960100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd966a37640 1 -- 172.21.15.89:0/3603694336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd960102d40 msgr2=0x7fd9601009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd966a37640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd960102d40 0x7fd9601009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd966a37640 1 -- 172.21.15.89:0/3603694336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd960103660 msgr2=0x7fd960100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd966a37640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd960103660 0x7fd960100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd966a37640 1 -- 172.21.15.89:0/3603694336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9601a6950 con 0x7fd96010c170 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd966a37640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96010c170 0x7fd960101420 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fd950006010 tx=0x7fd95002f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.214+0000 7fd95f7fe640 1 -- 172.21.15.89:0/3603694336 <== mon.0 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd950034040 con 0x7fd96010c170 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.215+0000 7fd95f7fe640 1 -- 172.21.15.89:0/3603694336 <== mon.0 v2:172.21.15.89:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd950003e00 con 0x7fd96010c170 2024-09-15T23:41:33.241 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.215+0000 7fd96849a640 1 -- 172.21.15.89:0/3603694336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9601a6be0 con 0x7fd96010c170 2024-09-15T23:41:33.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.215+0000 7fd95f7fe640 1 -- 172.21.15.89:0/3603694336 <== mon.0 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9500456e0 con 0x7fd96010c170 2024-09-15T23:41:33.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.215+0000 7fd96849a640 1 -- 172.21.15.89:0/3603694336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9601a70c0 con 0x7fd96010c170 2024-09-15T23:41:33.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.216+0000 7fd96849a640 1 -- 172.21.15.89:0/3603694336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd960103120 con 0x7fd96010c170 2024-09-15T23:41:33.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.217+0000 7fd95f7fe640 1 -- 172.21.15.89:0/3603694336 <== mon.0 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 20) ==== 138355+0+0 (secure 0 0 0) 0x7fd950037020 con 0x7fd96010c170 2024-09-15T23:41:33.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.217+0000 7fd95f7fe640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9480a3240 0x7fd9480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:41:33.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.217+0000 7fd966236640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9480a3240 0x7fd9480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:41:33.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.217+0000 7fd95f7fe640 1 -- 172.21.15.89:0/3603694336 <== mon.0 v2:172.21.15.89:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8496+0+0 (secure 0 0 0) 0x7fd9500f0b80 con 0x7fd96010c170 2024-09-15T23:41:33.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.218+0000 7fd966236640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9480a3240 0x7fd9480a5700 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd958005fd0 tx=0x7fd958005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:41:33.242 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.221+0000 7fd95f7fe640 1 -- 172.21.15.89:0/3603694336 <== mon.0 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd9500bd7a0 con 0x7fd96010c170 2024-09-15T23:41:33.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.388+0000 7fd96849a640 1 -- 172.21.15.89:0/3603694336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd960107210 con 0x7fd96010c170 2024-09-15T23:41:33.566 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.564+0000 7fd95f7fe640 1 -- 172.21.15.89:0/3603694336 <== mon.0 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v94) ==== 178+0+0 (secure 0 0 0) 0x7fd9500bd0a0 con 0x7fd96010c170 2024-09-15T23:41:33.566 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'cephfs_data_ec' removed 2024-09-15T23:41:33.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.567+0000 7fd96849a640 1 -- 172.21.15.89:0/3603694336 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9480a3240 msgr2=0x7fd9480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:33.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.567+0000 7fd96849a640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9480a3240 0x7fd9480a5700 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd958005fd0 tx=0x7fd958005e20 comp rx=0 tx=0).stop 2024-09-15T23:41:33.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.567+0000 7fd96849a640 1 -- 172.21.15.89:0/3603694336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96010c170 msgr2=0x7fd960101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:41:33.569 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.567+0000 7fd96849a640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96010c170 0x7fd960101420 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fd950006010 tx=0x7fd95002f710 comp rx=0 tx=0).stop 2024-09-15T23:41:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.567+0000 7fd96849a640 1 -- 172.21.15.89:0/3603694336 shutdown_connections 2024-09-15T23:41:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.567+0000 7fd96849a640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.89:6800/3594493718,v1:172.21.15.89:6801/3594493718] conn(0x7fd9480a3240 0x7fd9480a5700 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.567+0000 7fd96849a640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd96010c170 0x7fd960101420 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.567+0000 7fd96849a640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd960103660 0x7fd960100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.568+0000 7fd96849a640 1 --2- 172.21.15.89:0/3603694336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd960102d40 0x7fd9601009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:41:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.568+0000 7fd96849a640 1 -- 172.21.15.89:0/3603694336 >> 172.21.15.89:0/3603694336 conn(0x7fd9600fc7a0 msgr2=0x7fd9600fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:41:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.568+0000 7fd96849a640 1 -- 172.21.15.89:0/3603694336 shutdown_connections 2024-09-15T23:41:33.570 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-15T23:41:33.568+0000 7fd96849a640 1 -- 172.21.15.89:0/3603694336 wait complete. 2024-09-15T23:41:33.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:33 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3603694336' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:41:33.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:33 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3603694336' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:41:33.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:33 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3603694336' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:41:34.357 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:41:34.384 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:41:34.418 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:41:34.453 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-15T23:41:34.454 DEBUG:teuthology.orchestra.run.smithi089:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-15T23:41:34.469 DEBUG:teuthology.orchestra.run.smithi120:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-15T23:41:34.487 DEBUG:teuthology.orchestra.run.smithi174:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-15T23:41:34.511 INFO:tasks.cephadm:Stopping all daemons... 2024-09-15T23:41:34.511 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-15T23:41:34.511 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.a 2024-09-15T23:41:34.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:34 smithi120 ceph-mon[35250]: pgmap v879: 33 pgs: 33 active+clean; 577 KiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:41:34.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:34 smithi120 ceph-mon[35250]: from='client.? 172.21.15.89:0/3603694336' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:41:34.723 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:34 smithi120 ceph-mon[35250]: osdmap e94: 12 total, 12 up, 12 in 2024-09-15T23:41:34.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:34 smithi174 ceph-mon[35586]: pgmap v879: 33 pgs: 33 active+clean; 577 KiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:41:34.735 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:34 smithi174 ceph-mon[35586]: from='client.? 172.21.15.89:0/3603694336' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:41:34.736 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:34 smithi174 ceph-mon[35586]: osdmap e94: 12 total, 12 up, 12 in 2024-09-15T23:41:34.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:34 smithi089 systemd[1]: Stopping Ceph mon.a for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:41:34.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:34 smithi089 ceph-mon[30933]: pgmap v879: 33 pgs: 33 active+clean; 577 KiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:41:34.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:34 smithi089 ceph-mon[30933]: from='client.? 172.21.15.89:0/3603694336' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:41:34.810 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:34 smithi089 ceph-mon[30933]: osdmap e94: 12 total, 12 up, 12 in 2024-09-15T23:41:35.291 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a[30929]: 2024-09-15T23:41:34.830+0000 7fae13024640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:41:35.292 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a[30929]: 2024-09-15T23:41:34.830+0000 7fae13024640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-15T23:41:35.292 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:34 smithi089 podman[134273]: 2024-09-15 23:41:34.98541625 +0000 UTC m=+0.288455262 container died 587b46d282067b8382327780cd0a638e50de500b8e42fe1c77433d5f6c940b41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-09-15T23:41:35.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:35 smithi089 podman[134273]: 2024-09-15 23:41:35.381944978 +0000 UTC m=+0.684983990 container cleanup 587b46d282067b8382327780cd0a638e50de500b8e42fe1c77433d5f6c940b41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-15T23:41:35.560 INFO:journalctl@ceph.mon.a.smithi089.stdout:Sep 15 23:41:35 smithi089 bash[134273]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-a 2024-09-15T23:41:35.808 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.a.service' 2024-09-15T23:41:35.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:41:35.861 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-15T23:41:35.862 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-15T23:41:35.862 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.b 2024-09-15T23:41:36.163 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:35 smithi120 systemd[1]: Stopping Ceph mon.b for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:41:36.473 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:36 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-b[35246]: 2024-09-15T23:41:36.162+0000 7fe7f003b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:41:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:36 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-b[35246]: 2024-09-15T23:41:36.162+0000 7fe7f003b640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-15T23:41:36.474 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:36 smithi120 podman[70949]: 2024-09-15 23:41:36.316288183 +0000 UTC m=+0.264883743 container died fc28fc27ee928358192cebfc32c26606625735ec2812e95e754fcec43fcd2dcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2) 2024-09-15T23:41:36.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:36 smithi120 podman[70949]: 2024-09-15 23:41:36.696520393 +0000 UTC m=+0.645115940 container cleanup fc28fc27ee928358192cebfc32c26606625735ec2812e95e754fcec43fcd2dcc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-b, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-15T23:41:36.973 INFO:journalctl@ceph.mon.b.smithi120.stdout:Sep 15 23:41:36 smithi120 bash[70949]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-b 2024-09-15T23:41:37.114 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.b.service' 2024-09-15T23:41:37.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:41:37.163 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-15T23:41:37.163 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-15T23:41:37.163 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.c 2024-09-15T23:41:37.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:37 smithi174 systemd[1]: Stopping Ceph mon.c for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:41:37.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:37 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-c[35582]: 2024-09-15T23:41:37.484+0000 7f73f3ddc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:41:37.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:37 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-c[35582]: 2024-09-15T23:41:37.484+0000 7f73f3ddc640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-15T23:41:37.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:37 smithi174 podman[71320]: 2024-09-15 23:41:37.709597983 +0000 UTC m=+0.349550977 container died 04c6697b62ddf99f747e3d398c4e22548ed3da75bd91ea4c08f6460c8df4da8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-c, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:41:38.362 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:38 smithi174 podman[71320]: 2024-09-15 23:41:38.12710101 +0000 UTC m=+0.767054053 container cleanup 04c6697b62ddf99f747e3d398c4e22548ed3da75bd91ea4c08f6460c8df4da8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-15T23:41:38.362 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:38 smithi174 bash[71320]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-c 2024-09-15T23:41:38.605 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.c.service' 2024-09-15T23:41:38.627 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:38 smithi174 podman[71332]: 2024-09-15 23:41:38.361981858 +0000 UTC m=+0.651777326 container remove 04c6697b62ddf99f747e3d398c4e22548ed3da75bd91ea4c08f6460c8df4da8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mon-c, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-15T23:41:38.628 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:38 smithi174 systemd[1]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-15T23:41:38.628 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:38 smithi174 systemd[1]: Stopped Ceph mon.c for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:41:38.628 INFO:journalctl@ceph.mon.c.smithi174.stdout:Sep 15 23:41:38 smithi174 systemd[1]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mon.c.service: Consumed 15.515s CPU time. 2024-09-15T23:41:38.649 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:41:38.649 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-15T23:41:38.650 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-15T23:41:38.650 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.x 2024-09-15T23:41:38.971 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:41:38 smithi089 systemd[1]: Stopping Ceph mgr.x for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:41:39.310 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:41:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x[31161]: 2024-09-15T23:41:38.970+0000 7f8b4bc76640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:41:39.310 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:41:39 smithi089 podman[134394]: 2024-09-15 23:41:39.001376084 +0000 UTC m=+0.149688452 container died 9504be6ed98639a60dec28ddaf742447e9faff729765a7346e7898e5effce887 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:41:39.790 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:41:39 smithi089 podman[134394]: 2024-09-15 23:41:39.521385871 +0000 UTC m=+0.669698233 container cleanup 9504be6ed98639a60dec28ddaf742447e9faff729765a7346e7898e5effce887 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-15T23:41:39.790 INFO:journalctl@ceph.mgr.x.smithi089.stdout:Sep 15 23:41:39 smithi089 bash[134394]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-x 2024-09-15T23:41:40.013 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-15T23:41:40.049 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:41:40.049 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-15T23:41:40.050 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-15T23:41:40.050 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.y 2024-09-15T23:41:40.381 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:41:40 smithi120 systemd[1]: Stopping Ceph mgr.y for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:41:40.687 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:41:40 smithi120 podman[71070]: 2024-09-15 23:41:40.380545305 +0000 UTC m=+0.138170210 container died 5dccf040190a221225ff8263e1c9ff7c6dc69abb922cacb5c579251f49cff117 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-15T23:41:40.973 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:41:40 smithi120 podman[71070]: 2024-09-15 23:41:40.760242265 +0000 UTC m=+0.517867168 container cleanup 5dccf040190a221225ff8263e1c9ff7c6dc69abb922cacb5c579251f49cff117 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True) 2024-09-15T23:41:40.973 INFO:journalctl@ceph.mgr.y.smithi120.stdout:Sep 15 23:41:40 smithi120 bash[71070]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-y 2024-09-15T23:41:41.179 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-15T23:41:41.213 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:41:41.213 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-15T23:41:41.213 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-15T23:41:41.213 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.z 2024-09-15T23:41:41.544 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:41:41 smithi174 systemd[1]: Stopping Ceph mgr.z for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:41:41.985 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:41:41 smithi174 podman[71445]: 2024-09-15 23:41:41.544573095 +0000 UTC m=+0.137449994 container died 86570c43e9480ce5bff3a92fea76a40a2906d4fc8ad2e827b0ffa1c99b20eb57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2) 2024-09-15T23:41:42.311 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:41:42 smithi174 podman[71445]: 2024-09-15 23:41:42.09056743 +0000 UTC m=+0.683444328 container cleanup 86570c43e9480ce5bff3a92fea76a40a2906d4fc8ad2e827b0ffa1c99b20eb57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:41:42.311 INFO:journalctl@ceph.mgr.z.smithi174.stdout:Sep 15 23:41:42 smithi174 bash[71445]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mgr-z 2024-09-15T23:41:42.524 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mgr.z.service' 2024-09-15T23:41:42.558 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:41:42.558 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-15T23:41:42.558 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-15T23:41:42.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.0 2024-09-15T23:41:42.869 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:42 smithi089 systemd[1]: Stopping Ceph osd.0 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:41:43.310 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0[44283]: 2024-09-15T23:41:42.867+0000 7f69954a2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:41:43.311 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0[44283]: 2024-09-15T23:41:42.867+0000 7f69954a2640 -1 osd.0 94 *** Got signal Terminated *** 2024-09-15T23:41:43.311 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0[44283]: 2024-09-15T23:41:42.867+0000 7f69954a2640 -1 osd.0 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:41:48.560 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:48 smithi089 podman[134517]: 2024-09-15 23:41:48.269780235 +0000 UTC m=+5.521625773 container died 2cfb4bf4b3b6287453a20f34c70cf8b8dd27608eb05685ad05cf32a61ca69e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-15T23:41:48.946 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:48 smithi089 podman[134517]: 2024-09-15 23:41:48.6604966 +0000 UTC m=+5.912342138 container cleanup 2cfb4bf4b3b6287453a20f34c70cf8b8dd27608eb05685ad05cf32a61ca69e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:41:48.946 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:48 smithi089 bash[134517]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0 2024-09-15T23:41:49.283 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:48 smithi089 podman[134547]: 2024-09-15 23:41:48.945499083 +0000 UTC m=+0.671850050 container remove 2cfb4bf4b3b6287453a20f34c70cf8b8dd27608eb05685ad05cf32a61ca69e48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-15T23:41:49.560 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:49 smithi089 podman[134636]: 2024-09-15 23:41:49.186133375 +0000 UTC m=+0.016077651 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:41:49.560 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:49 smithi089 podman[134636]: 2024-09-15 23:41:49.308838045 +0000 UTC m=+0.138782314 container create cc9a4f46dd4af4963b0f8efd03a3345689963de07a7d8141fc0b721824dc2157 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:41:50.060 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:49 smithi089 podman[134636]: 2024-09-15 23:41:49.594954284 +0000 UTC m=+0.424898553 container init cc9a4f46dd4af4963b0f8efd03a3345689963de07a7d8141fc0b721824dc2157 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-15T23:41:50.060 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:49 smithi089 podman[134636]: 2024-09-15 23:41:49.598611953 +0000 UTC m=+0.428556220 container start cc9a4f46dd4af4963b0f8efd03a3345689963de07a7d8141fc0b721824dc2157 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-15T23:41:50.061 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:49 smithi089 podman[134636]: 2024-09-15 23:41:49.699711529 +0000 UTC m=+0.529655807 container attach cc9a4f46dd4af4963b0f8efd03a3345689963de07a7d8141fc0b721824dc2157 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-09-15T23:41:50.061 INFO:journalctl@ceph.osd.0.smithi089.stdout:Sep 15 23:41:49 smithi089 podman[134636]: 2024-09-15 23:41:49.783877659 +0000 UTC m=+0.613821933 container died cc9a4f46dd4af4963b0f8efd03a3345689963de07a7d8141fc0b721824dc2157 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-15T23:41:50.472 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.0.service' 2024-09-15T23:41:50.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:41:50.513 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-15T23:41:50.513 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-15T23:41:50.513 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.1 2024-09-15T23:41:50.832 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:50 smithi120 systemd[1]: Stopping Ceph osd.1 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:41:51.223 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:50 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1[39691]: 2024-09-15T23:41:50.830+0000 7ffb7ed93640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:41:51.223 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:50 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1[39691]: 2024-09-15T23:41:50.830+0000 7ffb7ed93640 -1 osd.1 94 *** Got signal Terminated *** 2024-09-15T23:41:51.223 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:50 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1[39691]: 2024-09-15T23:41:50.830+0000 7ffb7ed93640 -1 osd.1 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:41:56.473 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:56 smithi120 podman[71197]: 2024-09-15 23:41:56.200434257 +0000 UTC m=+5.491318692 container died fb5469c71234612487d82990962f0511a651af96d7c5441fdfddeebfa911c7af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-15T23:41:56.806 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:56 smithi120 podman[71197]: 2024-09-15 23:41:56.589925466 +0000 UTC m=+5.880809911 container cleanup fb5469c71234612487d82990962f0511a651af96d7c5441fdfddeebfa911c7af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True) 2024-09-15T23:41:56.806 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:56 smithi120 bash[71197]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1 2024-09-15T23:41:57.212 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:56 smithi120 podman[71225]: 2024-09-15 23:41:56.799553403 +0000 UTC m=+0.594322505 container remove fb5469c71234612487d82990962f0511a651af96d7c5441fdfddeebfa911c7af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:41:57.473 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:57 smithi120 podman[71316]: 2024-09-15 23:41:57.110382934 +0000 UTC m=+0.015914174 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:41:57.474 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:57 smithi120 podman[71316]: 2024-09-15 23:41:57.292431442 +0000 UTC m=+0.197962683 container create 690fb8dcb8fc2e4952e4e43b46140b2f9aeea46db1a48f5681b1a1ae610d40d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:41:57.739 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:57 smithi120 podman[71316]: 2024-09-15 23:41:57.550184227 +0000 UTC m=+0.455715480 container init 690fb8dcb8fc2e4952e4e43b46140b2f9aeea46db1a48f5681b1a1ae610d40d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-15T23:41:57.740 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:57 smithi120 podman[71316]: 2024-09-15 23:41:57.553939997 +0000 UTC m=+0.459471239 container start 690fb8dcb8fc2e4952e4e43b46140b2f9aeea46db1a48f5681b1a1ae610d40d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True) 2024-09-15T23:41:57.740 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:57 smithi120 podman[71316]: 2024-09-15 23:41:57.65409059 +0000 UTC m=+0.559621842 container attach 690fb8dcb8fc2e4952e4e43b46140b2f9aeea46db1a48f5681b1a1ae610d40d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-15T23:41:58.133 INFO:journalctl@ceph.osd.1.smithi120.stdout:Sep 15 23:41:57 smithi120 podman[71316]: 2024-09-15 23:41:57.739528402 +0000 UTC m=+0.645059650 container died 690fb8dcb8fc2e4952e4e43b46140b2f9aeea46db1a48f5681b1a1ae610d40d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:41:58.432 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.1.service' 2024-09-15T23:41:58.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:41:58.474 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-15T23:41:58.474 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-15T23:41:58.474 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.2 2024-09-15T23:41:58.787 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:41:58 smithi174 systemd[1]: Stopping Ceph osd.2 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:41:59.235 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:41:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2[40538]: 2024-09-15T23:41:58.785+0000 7f868245f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:41:59.236 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:41:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2[40538]: 2024-09-15T23:41:58.785+0000 7f868245f640 -1 osd.2 94 *** Got signal Terminated *** 2024-09-15T23:41:59.236 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:41:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2[40538]: 2024-09-15T23:41:58.785+0000 7f868245f640 -1 osd.2 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:42:04.587 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:42:04 smithi174 podman[71569]: 2024-09-15 23:42:04.297812029 +0000 UTC m=+5.631767663 container died a82184329c17a06c10613130dfacdfcfa77de2e1f1cea195a307f8b68cf2cff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:42:04.966 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:42:04 smithi174 podman[71569]: 2024-09-15 23:42:04.677854343 +0000 UTC m=+6.011810018 container cleanup a82184329c17a06c10613130dfacdfcfa77de2e1f1cea195a307f8b68cf2cff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-15T23:42:04.967 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:42:04 smithi174 bash[71569]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2 2024-09-15T23:42:05.235 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:42:04 smithi174 podman[71597]: 2024-09-15 23:42:04.966650861 +0000 UTC m=+0.668831731 container remove a82184329c17a06c10613130dfacdfcfa77de2e1f1cea195a307f8b68cf2cff8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, ceph=True) 2024-09-15T23:42:05.582 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:42:05 smithi174 podman[71683]: 2024-09-15 23:42:05.196409357 +0000 UTC m=+0.016108391 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:42:05.583 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:42:05 smithi174 podman[71683]: 2024-09-15 23:42:05.321807174 +0000 UTC m=+0.141506239 container create 00c3546a589aef825a7ebb75ac1a2a5ab5549b7785f07edb6f457be1d0bc92d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:42:05.583 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:42:05 smithi174 podman[71683]: 2024-09-15 23:42:05.582455951 +0000 UTC m=+0.402154980 container init 00c3546a589aef825a7ebb75ac1a2a5ab5549b7785f07edb6f457be1d0bc92d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2-deactivate, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True) 2024-09-15T23:42:05.986 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:42:05 smithi174 podman[71683]: 2024-09-15 23:42:05.586093807 +0000 UTC m=+0.405792830 container start 00c3546a589aef825a7ebb75ac1a2a5ab5549b7785f07edb6f457be1d0bc92d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0) 2024-09-15T23:42:05.986 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:42:05 smithi174 podman[71683]: 2024-09-15 23:42:05.682648246 +0000 UTC m=+0.502347284 container attach 00c3546a589aef825a7ebb75ac1a2a5ab5549b7785f07edb6f457be1d0bc92d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-15T23:42:05.987 INFO:journalctl@ceph.osd.2.smithi174.stdout:Sep 15 23:42:05 smithi174 podman[71683]: 2024-09-15 23:42:05.779632528 +0000 UTC m=+0.599331557 container died 00c3546a589aef825a7ebb75ac1a2a5ab5549b7785f07edb6f457be1d0bc92d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731) 2024-09-15T23:42:06.456 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.2.service' 2024-09-15T23:42:06.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:42:06.497 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-15T23:42:06.497 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-15T23:42:06.497 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.3 2024-09-15T23:42:06.810 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:06 smithi089 systemd[1]: Stopping Ceph osd.3 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:42:07.310 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:06 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3[50711]: 2024-09-15T23:42:06.830+0000 7f4eda7c5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:42:07.310 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:06 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3[50711]: 2024-09-15T23:42:06.830+0000 7f4eda7c5640 -1 osd.3 94 *** Got signal Terminated *** 2024-09-15T23:42:07.310 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:06 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3[50711]: 2024-09-15T23:42:06.830+0000 7f4eda7c5640 -1 osd.3 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:42:08.986 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:08.647+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:09.310 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:08 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3[50711]: 2024-09-15T23:42:08.953+0000 7f4ed2dd3640 -1 osd.3 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:47.271752+0000 front 2024-09-15T23:41:47.271724+0000 (oldest deadline 2024-09-15T23:42:08.371795+0000) 2024-09-15T23:42:09.696 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:09 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:09.267+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:09.985 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:09 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:09.695+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:10.310 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:10 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3[50711]: 2024-09-15T23:42:09.999+0000 7f4ed2dd3640 -1 osd.3 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:47.271752+0000 front 2024-09-15T23:41:47.271724+0000 (oldest deadline 2024-09-15T23:42:08.371795+0000) 2024-09-15T23:42:10.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:09 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:09.897+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:10.648 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:10.310+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:10.985 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:10.646+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:11.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:10 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:10.813+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:11.310 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:11 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3[50711]: 2024-09-15T23:42:11.045+0000 7f4ed2dd3640 -1 osd.3 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:47.271752+0000 front 2024-09-15T23:41:47.271724+0000 (oldest deadline 2024-09-15T23:42:08.371795+0000) 2024-09-15T23:42:11.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:10 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:10.921+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:11.688 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:11 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:11.336+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:11.985 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:11 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:11.687+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:12.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:11 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:11.815+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:12.234 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:11 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:11.952+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:12.491 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:12 smithi089 podman[134739]: 2024-09-15 23:42:12.233605081 +0000 UTC m=+5.542593779 container died 6cf4778962f98766222b893bc61f00605c3e3f35bc6ef66b2a02ab343c67d633 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-15T23:42:12.721 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:12 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:12.351+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:12.810 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:12 smithi089 podman[134739]: 2024-09-15 23:42:12.681140968 +0000 UTC m=+5.990129705 container cleanup 6cf4778962f98766222b893bc61f00605c3e3f35bc6ef66b2a02ab343c67d633 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e) 2024-09-15T23:42:12.810 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:12 smithi089 bash[134739]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3 2024-09-15T23:42:12.986 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:12 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:12.721+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:13.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:12 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:12.782+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:13.310 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:12 smithi089 podman[134754]: 2024-09-15 23:42:12.991015962 +0000 UTC m=+0.757628680 container remove 6cf4778962f98766222b893bc61f00605c3e3f35bc6ef66b2a02ab343c67d633 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-15T23:42:13.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:12 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:12.988+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:13.653 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:13 smithi089 podman[134842]: 2024-09-15 23:42:13.250935935 +0000 UTC m=+0.015892846 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:42:13.653 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:13 smithi089 podman[134842]: 2024-09-15 23:42:13.40017144 +0000 UTC m=+0.165128342 container create 0f57cb46ffbef9f4e62066123abd805856d689984fe45bd45701d5164ab1fbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-15T23:42:13.653 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:13 smithi089 podman[134842]: 2024-09-15 23:42:13.652625096 +0000 UTC m=+0.417581998 container init 0f57cb46ffbef9f4e62066123abd805856d689984fe45bd45701d5164ab1fbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:42:13.732 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:13.351+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:13.977 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:13 smithi089 podman[134842]: 2024-09-15 23:42:13.656140622 +0000 UTC m=+0.421097527 container start 0f57cb46ffbef9f4e62066123abd805856d689984fe45bd45701d5164ab1fbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:42:13.977 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:13 smithi089 podman[134842]: 2024-09-15 23:42:13.761384052 +0000 UTC m=+0.526340963 container attach 0f57cb46ffbef9f4e62066123abd805856d689984fe45bd45701d5164ab1fbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-09-15T23:42:13.978 INFO:journalctl@ceph.osd.3.smithi089.stdout:Sep 15 23:42:13 smithi089 podman[134842]: 2024-09-15 23:42:13.841485741 +0000 UTC m=+0.606442644 container died 0f57cb46ffbef9f4e62066123abd805856d689984fe45bd45701d5164ab1fbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-15T23:42:13.985 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:13.731+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:14.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:13 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:13.770+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:14.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:13 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:13.976+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:14.565 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.3.service' 2024-09-15T23:42:14.611 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:42:14.611 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-15T23:42:14.612 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-15T23:42:14.612 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.4 2024-09-15T23:42:14.735 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:14.320+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:14.911 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:14 smithi120 systemd[1]: Stopping Ceph osd.4 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:42:14.911 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:14 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:14.752+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:15.223 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:14 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4[46581]: 2024-09-15T23:42:14.909+0000 7f0f33445640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:42:15.659 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:14 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4[46581]: 2024-09-15T23:42:14.909+0000 7f0f33445640 -1 osd.4 94 *** Got signal Terminated *** 2024-09-15T23:42:15.659 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:14 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4[46581]: 2024-09-15T23:42:14.910+0000 7f0f33445640 -1 osd.4 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:42:15.660 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:14 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:14.993+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:15.660 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:14.738+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:15.735 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:15 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:15.334+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:15.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:15 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:15.713+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:16.060 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:16 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:15.999+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:16.235 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:15 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:15.747+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:16.735 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:16 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:16.311+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:16.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:16 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:16.676+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:17.235 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:16 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:16.794+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:17.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:16 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:16.987+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:17.735 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:17 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:17.284+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:17.736 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:17 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:17.284+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:17.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:17 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:17.690+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:18.017 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:17 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:17.760+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:18.235 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:17 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:17.828+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:18.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:18.016+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:18.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:18.016+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:18.735 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:18 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:18.241+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:18.736 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:18 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:18.241+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:18.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:18 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:18.661+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:18.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:18 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:18.661+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:19.064 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:18.796+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:19.065 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:19 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:19.037+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:19.065 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:19 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:19.037+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:19.212 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:18 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:18.801+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:19.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:19 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:19.210+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:19.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:19 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:19.211+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:19.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:19 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:19.654+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:19.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:19 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:19.654+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:20.218 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:19 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:19.792+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:20.219 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:19 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:19.792+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.231080+0000 front 2024-09-15T23:41:50.231024+0000 (oldest deadline 2024-09-15T23:42:19.031638+0000) 2024-09-15T23:42:20.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:19 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:19.820+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:20.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:20 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:20.000+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:20.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:20 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:20.000+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:20.448 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:19 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:19.974+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:20.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:20 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:20.218+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:20.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:20 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:20.218+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:20.723 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:20 smithi120 podman[71417]: 2024-09-15 23:42:20.447880378 +0000 UTC m=+5.638356524 container died 7b64fb3ff88e20d55278ce2705ad348af424770fec76842539bfdd9f167c9955 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-15T23:42:20.723 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:20 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:20.693+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:20.723 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:20 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:20.693+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:21.005 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:20 smithi120 podman[71417]: 2024-09-15 23:42:20.82674602 +0000 UTC m=+6.017222163 container cleanup 7b64fb3ff88e20d55278ce2705ad348af424770fec76842539bfdd9f167c9955 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731) 2024-09-15T23:42:21.005 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:20 smithi120 bash[71417]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4 2024-09-15T23:42:21.060 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:20 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:20.795+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:21.060 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:21 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:21.043+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:21.060 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:21 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:21.043+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:21.235 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:20 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:20.755+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:21.235 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:20 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:20.755+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.231080+0000 front 2024-09-15T23:41:50.231024+0000 (oldest deadline 2024-09-15T23:42:19.031638+0000) 2024-09-15T23:42:21.449 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:21 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:21.003+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:21.449 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:21 smithi120 podman[71432]: 2024-09-15 23:42:21.128255119 +0000 UTC m=+0.683610632 container remove 7b64fb3ff88e20d55278ce2705ad348af424770fec76842539bfdd9f167c9955 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-15T23:42:21.723 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:21 smithi120 podman[71522]: 2024-09-15 23:42:21.351961248 +0000 UTC m=+0.015779761 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:42:21.723 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:21 smithi120 podman[71522]: 2024-09-15 23:42:21.462338126 +0000 UTC m=+0.126156671 container create 67f37fbe273672dc83eee1ba2b71e7c5c4279446ba67c80bcd71595cd1513218 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-15T23:42:21.724 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:21 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:21.668+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:21.724 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:21 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:21.668+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:21.735 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:21 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:21.259+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:21.736 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:21 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:21.259+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:21.736 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:21 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:21.375+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:22.052 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:21 smithi120 podman[71522]: 2024-09-15 23:42:21.723250086 +0000 UTC m=+0.387068590 container init 67f37fbe273672dc83eee1ba2b71e7c5c4279446ba67c80bcd71595cd1513218 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-09-15T23:42:22.052 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:21 smithi120 podman[71522]: 2024-09-15 23:42:21.726468775 +0000 UTC m=+0.390287289 container start 67f37fbe273672dc83eee1ba2b71e7c5c4279446ba67c80bcd71595cd1513218 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-15T23:42:22.052 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:21 smithi120 podman[71522]: 2024-09-15 23:42:21.831638507 +0000 UTC m=+0.495457011 container attach 67f37fbe273672dc83eee1ba2b71e7c5c4279446ba67c80bcd71595cd1513218 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:42:22.053 INFO:journalctl@ceph.osd.4.smithi120.stdout:Sep 15 23:42:21 smithi120 podman[71522]: 2024-09-15 23:42:21.916327899 +0000 UTC m=+0.580146413 container died 67f37fbe273672dc83eee1ba2b71e7c5c4279446ba67c80bcd71595cd1513218 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-15T23:42:22.235 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:21 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:21.803+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:22.236 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:21 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:21.803+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.231080+0000 front 2024-09-15T23:41:50.231024+0000 (oldest deadline 2024-09-15T23:42:19.031638+0000) 2024-09-15T23:42:22.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:21 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:21.844+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:22.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:22 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:22.065+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:22.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:22 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:22.065+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:22.473 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:22 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:22.050+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:22.645 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.4.service' 2024-09-15T23:42:22.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:42:22.687 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-15T23:42:22.687 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-15T23:42:22.688 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.5 2024-09-15T23:42:22.715 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:22 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:22.387+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:22.715 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:22 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:22.276+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:22.716 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:22 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:22.276+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:22.898 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:22 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:22.718+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:22.898 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:22 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:22.718+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:22.985 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:22 smithi174 systemd[1]: Stopping Ceph osd.5 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:42:22.986 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:22 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:22.851+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:22.986 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:22 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:22.851+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.231080+0000 front 2024-09-15T23:41:50.231024+0000 (oldest deadline 2024-09-15T23:42:19.031638+0000) 2024-09-15T23:42:23.116 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:22 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:22.820+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:23.223 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:23 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:23.013+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:23.389 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:23 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:23.245+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:23.390 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:23 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:23.245+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:23.390 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:22 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:22.996+0000 7fe8b3535640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:42:23.390 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:22 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:22.996+0000 7fe8b3535640 -1 osd.5 94 *** Got signal Terminated *** 2024-09-15T23:42:23.390 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:22 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:22.996+0000 7fe8b3535640 -1 osd.5 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:42:23.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:23 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:23.115+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:23.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:23 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:23.115+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:23.735 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:23 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:23.389+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:24.223 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:23 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:23.975+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:24.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:23 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:23.752+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:24.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:23 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:23.752+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:24.235 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:23 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:23.834+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:24.236 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:23 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:23.834+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.231080+0000 front 2024-09-15T23:41:50.231024+0000 (oldest deadline 2024-09-15T23:42:19.031638+0000) 2024-09-15T23:42:24.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:23 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:23.859+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:24.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:24 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:24.108+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:24.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:24 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:24.108+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:24.735 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:24 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:24.423+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:24.735 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:24 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:24.239+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:24.736 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:24 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:24.239+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:24.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:24 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:24.929+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:24.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:24 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:24.929+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:24.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:24 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:24.718+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:24.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:24 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:24.718+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:24.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:24 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:24.718+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:25.123 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:24 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:24.842+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:25.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:24 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:24.876+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:25.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:24 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:24.876+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.231080+0000 front 2024-09-15T23:41:50.231024+0000 (oldest deadline 2024-09-15T23:42:19.031638+0000) 2024-09-15T23:42:25.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:24 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:24.876+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:59.032007+0000 front 2024-09-15T23:41:59.031977+0000 (oldest deadline 2024-09-15T23:42:24.332167+0000) 2024-09-15T23:42:25.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:25.373+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:25.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:25.194+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:25.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:25.194+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:25.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:25.194+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:25.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:25 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:25.122+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:25.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:25 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:25.122+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:25.926 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:25.669+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:25.926 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:25.669+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:25.926 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:25.669+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:26.109 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:25 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:25.872+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:26.156 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:25.903+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:26.156 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:25.903+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.231080+0000 front 2024-09-15T23:41:50.231024+0000 (oldest deadline 2024-09-15T23:42:19.031638+0000) 2024-09-15T23:42:26.156 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:25 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:25.903+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:59.032007+0000 front 2024-09-15T23:41:59.031977+0000 (oldest deadline 2024-09-15T23:42:24.332167+0000) 2024-09-15T23:42:26.223 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:25.925+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:26.223 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:25 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:25.925+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:26.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:26 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:26.325+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:26.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:26 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:26.155+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:26.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:26 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:26.155+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:26.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:26 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:26.155+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:26.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:26 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:26.108+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:26.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:26 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:26.108+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:26.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:26 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:26.108+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:26.934 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:26 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:26.643+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:26.934 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:26 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:26.643+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:26.934 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:26 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:26.643+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:27.202 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:26 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:26.871+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:27.202 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:26 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:26.871+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.231080+0000 front 2024-09-15T23:41:50.231024+0000 (oldest deadline 2024-09-15T23:42:19.031638+0000) 2024-09-15T23:42:27.202 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:26 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:26.871+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:59.032007+0000 front 2024-09-15T23:41:59.031977+0000 (oldest deadline 2024-09-15T23:42:24.332167+0000) 2024-09-15T23:42:27.223 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:26 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:26.933+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:27.223 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:26 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:26.933+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:27.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:26 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:26.863+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:27.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:27 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:27.093+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:27.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:27 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:27.093+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:27.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:27 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:27.093+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:27.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:27 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:27.200+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:27.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:27 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:27.200+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:27.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:27 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:27.200+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:27.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:27 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:27.309+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:27.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:27 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:27.309+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:27.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:27 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:27.885+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:27.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:27 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:27.885+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:27.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:27 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:27.675+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:27.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:27 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:27.675+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:27.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:27 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:27.675+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:28.185 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:27 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:27.888+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.429975+0000 front 2024-09-15T23:41:44.429996+0000 (oldest deadline 2024-09-15T23:42:08.530164+0000) 2024-09-15T23:42:28.186 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:27 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:27.888+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.231080+0000 front 2024-09-15T23:41:50.231024+0000 (oldest deadline 2024-09-15T23:42:19.031638+0000) 2024-09-15T23:42:28.186 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:27 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5[47410]: 2024-09-15T23:42:27.888+0000 7fe8afb4b640 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:59.032007+0000 front 2024-09-15T23:41:59.031977+0000 (oldest deadline 2024-09-15T23:42:24.332167+0000) 2024-09-15T23:42:28.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:27 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:27.910+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:28.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:28 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:28.095+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:28.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:28 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:28.095+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:28.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:28 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:28.095+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:28.485 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:28 smithi174 podman[71786]: 2024-09-15 23:42:28.42954425 +0000 UTC m=+5.545532737 container died e7dcaa9d2229f35e41f79484d10ae76cc4ab6610f2c52b2a47cf6ea1ca174ae9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-15T23:42:28.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:28 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:28.291+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:28.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:28 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:28.291+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:28.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:28 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:28.184+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:28.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:28 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:28.184+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:28.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:28 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:28.184+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:28.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:28 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:28.842+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:28.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:28 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:28.842+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:28.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:28 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:28.676+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:28.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:28 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:28.676+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:28.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:28 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:28.676+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:28.985 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:28 smithi174 podman[71786]: 2024-09-15 23:42:28.813796419 +0000 UTC m=+5.929784900 container cleanup e7dcaa9d2229f35e41f79484d10ae76cc4ab6610f2c52b2a47cf6ea1ca174ae9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:42:28.985 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:28 smithi174 bash[71786]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5 2024-09-15T23:42:29.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:28 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:28.905+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:29.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:29 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:29.090+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:29.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:29 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:29.090+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:29.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:29 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:29.090+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:29.429 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:29 smithi174 podman[71804]: 2024-09-15 23:42:29.117286033 +0000 UTC m=+0.685732603 container remove e7dcaa9d2229f35e41f79484d10ae76cc4ab6610f2c52b2a47cf6ea1ca174ae9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:42:29.430 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:29 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:29.251+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:29.430 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:29 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:29.251+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:29.430 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:29 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:29.177+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:29.430 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:29 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:29.177+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:29.430 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:29 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:29.177+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:29.729 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:29 smithi174 podman[71892]: 2024-09-15 23:42:29.331911883 +0000 UTC m=+0.016130696 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:42:29.729 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:29 smithi174 podman[71892]: 2024-09-15 23:42:29.468129398 +0000 UTC m=+0.152348197 container create ccecc120343b88dec6b86bcbdf99806cbe3b185b38a7e364de1ed232c486ee7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-15T23:42:29.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:29 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:29.832+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:29.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:29 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:29.832+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:29.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:29 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:29.676+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:29.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:29 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:29.676+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:29.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:29 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:29.676+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:29.986 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:29 smithi174 podman[71892]: 2024-09-15 23:42:29.728902611 +0000 UTC m=+0.413121411 container init ccecc120343b88dec6b86bcbdf99806cbe3b185b38a7e364de1ed232c486ee7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-15T23:42:29.986 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:29 smithi174 podman[71892]: 2024-09-15 23:42:29.732266543 +0000 UTC m=+0.416485353 container start ccecc120343b88dec6b86bcbdf99806cbe3b185b38a7e364de1ed232c486ee7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-15T23:42:29.987 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:29 smithi174 podman[71892]: 2024-09-15 23:42:29.82887757 +0000 UTC m=+0.513096379 container attach ccecc120343b88dec6b86bcbdf99806cbe3b185b38a7e364de1ed232c486ee7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-15T23:42:29.987 INFO:journalctl@ceph.osd.5.smithi174.stdout:Sep 15 23:42:29 smithi174 podman[71892]: 2024-09-15 23:42:29.924082133 +0000 UTC m=+0.608300944 container died ccecc120343b88dec6b86bcbdf99806cbe3b185b38a7e364de1ed232c486ee7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-15T23:42:30.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:29 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:29.904+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:30.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:29 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:29.904+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.295215+0000 front 2024-09-15T23:42:02.295450+0000 (oldest deadline 2024-09-15T23:42:29.295791+0000) 2024-09-15T23:42:30.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:30.085+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:30.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:30.085+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:30.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:30.085+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:30.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:30 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:30.174+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:30.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:30 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:30.174+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:30.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:30 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:30.174+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:30.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:30 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:30.257+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:30.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:30 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:30.257+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:30.510 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.5.service' 2024-09-15T23:42:30.588 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:42:30.588 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-15T23:42:30.588 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-15T23:42:30.588 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.6 2024-09-15T23:42:30.876 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:30 smithi089 systemd[1]: Stopping Ceph osd.6 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:42:30.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:30 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:30.849+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:30.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:30 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:30.849+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:30.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:30 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:30.685+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:30.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:30 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:30.685+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:30.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:30 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:30.685+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:31.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:31.112+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:31.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:31.112+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:31.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:31.112+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:31.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:30.875+0000 7f7193c9b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:42:31.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:30.875+0000 7f7193c9b640 -1 osd.6 94 *** Got signal Terminated *** 2024-09-15T23:42:31.311 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:30.875+0000 7f7193c9b640 -1 osd.6 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:42:31.311 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:30.927+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:31.311 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:30.927+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.295215+0000 front 2024-09-15T23:42:02.295450+0000 (oldest deadline 2024-09-15T23:42:29.295791+0000) 2024-09-15T23:42:31.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:31 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:31.196+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:31.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:31 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:31.196+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:31.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:31 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:31.196+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:31.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:31 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:31.228+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:31.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:31 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:31.228+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:31.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:31 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:31.868+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:31.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:31 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:31.868+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:31.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:31 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:31.671+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:31.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:31 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:31.671+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:31.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:31 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:31.671+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:32.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:31.911+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:32.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:31 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:31.911+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.295215+0000 front 2024-09-15T23:42:02.295450+0000 (oldest deadline 2024-09-15T23:42:29.295791+0000) 2024-09-15T23:42:32.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:32 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:32.151+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:32.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:32 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:32.151+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:32.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:32 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:32.151+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:32.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:32 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:32.200+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:32.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:32 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:32.200+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:32.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:32 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:32.226+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:32.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:32 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:32.226+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:32.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:32 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:32.226+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:32.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:32 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:32.868+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:32.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:32 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:32.868+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:32.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:32 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:32.868+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:32.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:32 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:32.672+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:32.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:32 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:32.672+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:32.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:32 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:32.672+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:33.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:32 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:32.950+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:33.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:32 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:32.950+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.295215+0000 front 2024-09-15T23:42:02.295450+0000 (oldest deadline 2024-09-15T23:42:29.295791+0000) 2024-09-15T23:42:33.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:33 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:33.107+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:33.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:33 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:33.107+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:33.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:33 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:33.107+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:33.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:33 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:33.167+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:33.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:33 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:33.167+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:33.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:33 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:33.182+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:33.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:33 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:33.182+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:33.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:33 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:33.182+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:33.926 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:33 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:33.631+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:33.926 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:33 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:33.631+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:33.926 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:33 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:33.631+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:33.926 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:33 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:33.838+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:33.926 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:33 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:33.838+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:33.926 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:33 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:33.838+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:34.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:33 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:33.932+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:34.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:33 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:33.932+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.295215+0000 front 2024-09-15T23:42:02.295450+0000 (oldest deadline 2024-09-15T23:42:29.295791+0000) 2024-09-15T23:42:34.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:33 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:33.932+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.396762+0000 front 2024-09-15T23:42:10.396897+0000 (oldest deadline 2024-09-15T23:42:33.896983+0000) 2024-09-15T23:42:34.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:34.072+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:34.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:34.072+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:34.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:34.072+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:34.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:34 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:34.131+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:34.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:34 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:34.131+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:34.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:34 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:34.166+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:34.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:34 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:34.166+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:34.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:34 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:34.166+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:34.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:34 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:34.649+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:34.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:34 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:34.649+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:34.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:34 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:34.649+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:34.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:34 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:34.649+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:34.974 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:34 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:34.826+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:34.974 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:34 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:34.826+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:34.974 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:34 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:34.826+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:35.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:34.959+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:53.593722+0000 front 2024-09-15T23:41:53.593633+0000 (oldest deadline 2024-09-15T23:42:17.594249+0000) 2024-09-15T23:42:35.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:34.959+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.295215+0000 front 2024-09-15T23:42:02.295450+0000 (oldest deadline 2024-09-15T23:42:29.295791+0000) 2024-09-15T23:42:35.310 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:34 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6[57171]: 2024-09-15T23:42:34.959+0000 7f71902b1640 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.396762+0000 front 2024-09-15T23:42:10.396897+0000 (oldest deadline 2024-09-15T23:42:33.896983+0000) 2024-09-15T23:42:35.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:35 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:35.072+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:35.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:35 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:35.072+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:35.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:35 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:35.072+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:35.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:35 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:35.169+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:35.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:35 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:35.169+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:35.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:35 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:35.169+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:35.487 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:35 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:35.174+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:35.487 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:35 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:35.174+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:35.487 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:35 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:35.174+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:35.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:35 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:35.802+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:35.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:35 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:35.802+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:35.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:35 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:35.802+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:35.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:35 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:35.657+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:35.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:35 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:35.657+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:35.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:35 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:35.657+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:35.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:35 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:35.657+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:36.414 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:36 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:36.066+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:36.414 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:36 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:36.066+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:36.414 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:36 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:36.066+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:36.414 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:36 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:36.066+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:36.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:36 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:36.155+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:36.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:36 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:36.155+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:36.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:36 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:36.155+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:36.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:36 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:36.153+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:36.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:36 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:36.153+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:36.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:36 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:36.153+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:36.702 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:36 smithi089 podman[134957]: 2024-09-15 23:42:36.41380488 +0000 UTC m=+5.636270029 container died 914a4cfe76f57f738cbd4ed5f99065f4b8d6d2530b76393d97100d1b7269c49c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-15T23:42:36.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:36 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:36.758+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:36.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:36 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:36.758+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:36.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:36 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:36.758+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:36.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:36 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:36.645+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:36.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:36 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:36.645+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:36.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:36 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:36.645+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:36.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:36 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:36.645+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:37.060 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:36 smithi089 podman[134957]: 2024-09-15 23:42:36.901262169 +0000 UTC m=+6.123727362 container cleanup 914a4cfe76f57f738cbd4ed5f99065f4b8d6d2530b76393d97100d1b7269c49c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6, RELEASE=squid-8921c8e, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-15T23:42:37.060 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:36 smithi089 bash[134957]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6 2024-09-15T23:42:37.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:37 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:37.165+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:37.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:37 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:37.165+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:37.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:37 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:37.165+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:37.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:37 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:37.194+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:37.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:37 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:37.194+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:37.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:37 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:37.194+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:37.524 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:37 smithi089 podman[134971]: 2024-09-15 23:42:37.202740477 +0000 UTC m=+0.779554109 container remove 914a4cfe76f57f738cbd4ed5f99065f4b8d6d2530b76393d97100d1b7269c49c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-15T23:42:37.524 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:37 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:37.113+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:37.524 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:37 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:37.113+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:37.524 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:37 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:37.113+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:37.524 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:37 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:37.113+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:37.565 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:37 smithi089 podman[135059]: 2024-09-15 23:42:37.426610525 +0000 UTC m=+0.016207148 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:42:37.810 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:37 smithi089 podman[135059]: 2024-09-15 23:42:37.56433032 +0000 UTC m=+0.153926933 container create e42a77607479a67cbf23d768c7d8dbd65d8be1a6598ac5cf5a7c114e6982717f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:42:37.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:37 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:37.709+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:37.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:37 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:37.709+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:37.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:37 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:37.709+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:37.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:37 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:37.683+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:37.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:37 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:37.683+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:37.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:37 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:37.683+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:37.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:37 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:37.683+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:38.132 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:37 smithi089 podman[135059]: 2024-09-15 23:42:37.815598557 +0000 UTC m=+0.405195170 container init e42a77607479a67cbf23d768c7d8dbd65d8be1a6598ac5cf5a7c114e6982717f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-15T23:42:38.132 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:37 smithi089 podman[135059]: 2024-09-15 23:42:37.819090963 +0000 UTC m=+0.408687582 container start e42a77607479a67cbf23d768c7d8dbd65d8be1a6598ac5cf5a7c114e6982717f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:42:38.133 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:37 smithi089 podman[135059]: 2024-09-15 23:42:37.898834769 +0000 UTC m=+0.488431382 container attach e42a77607479a67cbf23d768c7d8dbd65d8be1a6598ac5cf5a7c114e6982717f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-15T23:42:38.133 INFO:journalctl@ceph.osd.6.smithi089.stdout:Sep 15 23:42:38 smithi089 podman[135059]: 2024-09-15 23:42:38.004439253 +0000 UTC m=+0.594035876 container died e42a77607479a67cbf23d768c7d8dbd65d8be1a6598ac5cf5a7c114e6982717f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-15T23:42:38.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:38 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:38.186+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:38.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:38 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:38.186+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:38.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:38 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:38.186+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:38.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:38 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:38.164+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:38.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:38 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:38.164+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:38.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:38 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:38.164+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:38.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:38.131+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:38.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:38.131+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:38.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:38.131+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:38.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:38 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:38.131+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:38.751 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.6.service' 2024-09-15T23:42:38.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:42:38.792 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-15T23:42:38.793 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-15T23:42:38.793 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.7 2024-09-15T23:42:38.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:38.740+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:38.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:38.740+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:38.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:38.740+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:38.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:38 smithi120 systemd[1]: Stopping Ceph osd.7 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:42:38.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:38.708+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:38.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:38.708+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:38.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:38.708+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:38.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:38.708+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:39.473 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:39 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:39.138+0000 7f72e39d7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:42:39.473 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:39 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:39.138+0000 7f72e39d7640 -1 osd.7 94 *** Got signal Terminated *** 2024-09-15T23:42:39.473 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:39 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:39.138+0000 7f72e39d7640 -1 osd.7 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:42:39.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:39 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:39.225+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:39.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:39 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:39.225+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:39.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:39 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:39.225+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:39.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:39 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:39.177+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:39.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:39 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:39.177+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:39.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:39 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:39.177+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:39.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:39 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:39.094+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:39.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:39 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:39.094+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:39.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:39 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:39.094+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:39.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:39 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:39.094+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:39.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:39 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:39.720+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:39.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:39 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:39.720+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:39.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:39 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:39.720+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:39.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:39 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:39.726+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:39.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:39 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:39.726+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:39.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:39 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:39.726+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:39.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:39 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:39.726+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:40.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:40 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:40.185+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:40.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:40 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:40.185+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:40.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:40 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:40.185+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:40.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:40 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:40.138+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:40.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:40 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:40.138+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:40.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:40 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:40.138+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:40.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:40 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:40.113+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:40.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:40 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:40.113+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:40.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:40 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:40.113+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:40.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:40 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:40.113+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:40.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:40 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:40.681+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:40.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:40 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:40.681+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:40.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:40 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:40.681+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:40.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:40 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:40.736+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:40.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:40 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:40.736+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:40.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:40 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:40.736+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:40.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:40 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:40.736+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:41.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:41 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:41.144+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:41.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:41 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:41.144+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:41.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:41 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:41.144+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:41.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:41 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:41.123+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:41.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:41 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:41.123+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:41.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:41 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:41.123+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:41.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:41 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:41.110+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:41.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:41 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:41.110+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:41.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:41 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:41.110+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:41.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:41 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:41.110+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:41.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:41 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:41.665+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:41.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:41 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:41.665+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:41.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:41 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:41.665+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:41.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:41 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:41.784+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:41.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:41 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:41.784+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:41.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:41 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:41.784+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:41.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:41 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:41.784+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:42.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:42 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:42.184+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:42.595 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:42 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:42.184+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:42.595 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:42 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:42.184+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:42.595 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:42 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:42.107+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:42.595 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:42 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:42.107+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:42.595 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:42 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:42.107+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:42.596 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:42 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:42.107+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:42.596 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:42.110+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:42.596 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:42.110+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:42.596 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:42.110+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:42.596 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:42 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:42.110+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:42.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:42.642+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:42.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:42.642+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:42.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:42.642+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:42.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:42.831+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:42.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:42.831+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:42.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:42.831+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:42.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:42.831+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:42.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:42.831+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:43.461 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:43 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:43.114+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:43.462 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:43 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:43.114+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:43.462 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:43 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:43.114+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:43.462 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:43 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:43.114+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:43.462 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:43 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:43.114+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:43.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:43 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:43.147+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:43.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:43 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:43.147+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:43.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:43 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:43.147+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:43.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:43 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:43.094+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:43.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:43 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:43.094+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:43.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:43 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:43.094+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:43.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:43 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:43.094+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:43.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:43 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:43.627+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.141074+0000 front 2024-09-15T23:41:54.141042+0000 (oldest deadline 2024-09-15T23:42:19.441180+0000) 2024-09-15T23:42:43.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:43 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:43.627+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:02.342583+0000 front 2024-09-15T23:42:02.342487+0000 (oldest deadline 2024-09-15T23:42:24.642293+0000) 2024-09-15T23:42:43.973 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:43 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7[53253]: 2024-09-15T23:42:43.627+0000 7f72dbfe5640 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:09.843500+0000 front 2024-09-15T23:42:09.843570+0000 (oldest deadline 2024-09-15T23:42:32.743544+0000) 2024-09-15T23:42:43.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:43 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:43.846+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:43.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:43 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:43.846+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:43.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:43 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:43.846+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:43.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:43 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:43.846+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:43.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:43 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:43.846+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:44.481 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:44 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:44.070+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:44.481 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:44 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:44.070+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:44.481 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:44 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:44.070+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:44.481 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:44 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:44.070+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:44.481 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:44 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:44.070+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:44.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:44 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:44.118+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:44.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:44 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:44.118+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:44.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:44 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:44.118+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:44.485 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:44 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:44.118+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:14.630460+0000 front 2024-09-15T23:42:14.630411+0000 (oldest deadline 2024-09-15T23:42:43.430966+0000) 2024-09-15T23:42:44.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:44 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:44.111+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:44.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:44 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:44.111+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:44.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:44 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:44.111+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:44.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:44 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:44.111+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:44.723 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:44 smithi120 podman[71636]: 2024-09-15 23:42:44.472559552 +0000 UTC m=+5.483435474 container died ec285f96aaffe81935d290872305582bd8f1c4f06eb487ba2dcd2aa468d49ef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-15T23:42:45.122 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:44 smithi120 podman[71636]: 2024-09-15 23:42:44.862436038 +0000 UTC m=+5.873311957 container cleanup ec285f96aaffe81935d290872305582bd8f1c4f06eb487ba2dcd2aa468d49ef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-15T23:42:45.122 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:44 smithi120 bash[71636]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7 2024-09-15T23:42:45.122 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:44 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:44.811+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:45.123 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:44 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:44.811+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:45.123 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:44 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:44.811+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:45.123 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:44 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:44.811+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:45.123 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:44 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:44.811+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:45.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:45 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:45.027+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:45.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:45 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:45.027+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:45.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:45 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:45.027+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:45.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:45 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:45.027+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:45.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:45 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:45.027+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:45.473 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:45 smithi120 podman[71654]: 2024-09-15 23:42:45.122096443 +0000 UTC m=+0.646391864 container remove ec285f96aaffe81935d290872305582bd8f1c4f06eb487ba2dcd2aa468d49ef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-15T23:42:45.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:45 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:45.099+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:45.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:45 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:45.099+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:45.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:45 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:45.099+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:45.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:45 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:45.099+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:45.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:45 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:45.143+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:45.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:45 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:45.143+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:45.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:45 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:45.143+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:45.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:45 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:45.143+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:14.630460+0000 front 2024-09-15T23:42:14.630411+0000 (oldest deadline 2024-09-15T23:42:43.430966+0000) 2024-09-15T23:42:45.759 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:45 smithi120 podman[71741]: 2024-09-15 23:42:45.380077691 +0000 UTC m=+0.015935433 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:42:45.759 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:45 smithi120 podman[71741]: 2024-09-15 23:42:45.514650962 +0000 UTC m=+0.150508735 container create 7a0681972d0fc60e77a8fc95f7d28cc6817ec1eb86da76decfe1ca3a5331fddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7-deactivate, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-15T23:42:45.759 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:45 smithi120 podman[71741]: 2024-09-15 23:42:45.758790449 +0000 UTC m=+0.394648183 container init 7a0681972d0fc60e77a8fc95f7d28cc6817ec1eb86da76decfe1ca3a5331fddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:42:46.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:45 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:45.792+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:46.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:45 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:45.792+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:46.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:45 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:45.792+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:46.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:45 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:45.792+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:46.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:45 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:45.792+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:46.224 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:45 smithi120 podman[71741]: 2024-09-15 23:42:45.76217928 +0000 UTC m=+0.398037014 container start 7a0681972d0fc60e77a8fc95f7d28cc6817ec1eb86da76decfe1ca3a5331fddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-15T23:42:46.224 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:45 smithi120 podman[71741]: 2024-09-15 23:42:45.842056088 +0000 UTC m=+0.477913829 container attach 7a0681972d0fc60e77a8fc95f7d28cc6817ec1eb86da76decfe1ca3a5331fddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:42:46.224 INFO:journalctl@ceph.osd.7.smithi120.stdout:Sep 15 23:42:45 smithi120 podman[71741]: 2024-09-15 23:42:45.943958884 +0000 UTC m=+0.579816612 container died 7a0681972d0fc60e77a8fc95f7d28cc6817ec1eb86da76decfe1ca3a5331fddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-15T23:42:46.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:46 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:46.055+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:46.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:46 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:46.055+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:46.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:46 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:46.055+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:46.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:46 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:46.055+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:46.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:46 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:46.055+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:46.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:46 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:46.162+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:46.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:46 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:46.162+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:46.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:46 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:46.163+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:46.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:46 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:46.163+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:14.630460+0000 front 2024-09-15T23:42:14.630411+0000 (oldest deadline 2024-09-15T23:42:43.430966+0000) 2024-09-15T23:42:46.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:46 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:46.106+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:46.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:46 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:46.106+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:46.487 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:46 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:46.106+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:46.487 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:46 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:46.106+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:46.648 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.7.service' 2024-09-15T23:42:46.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:42:46.688 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-15T23:42:46.689 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-15T23:42:46.689 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.8 2024-09-15T23:42:46.899 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:46 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:46.812+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:46.899 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:46 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:46.812+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:46.899 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:46 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:46.812+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:46.899 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:46 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:46.812+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:46.899 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:46 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:46.812+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:46.985 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:46 smithi174 systemd[1]: Stopping Ceph osd.8 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:42:47.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:47.071+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:47.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:47.071+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:47.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:47.071+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:47.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:47.071+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:47.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:47.000+0000 7f5f4ef22640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:42:47.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:47.000+0000 7f5f4ef22640 -1 osd.8 94 *** Got signal Terminated *** 2024-09-15T23:42:47.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:47.000+0000 7f5f4ef22640 -1 osd.8 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:42:47.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:47.172+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:47.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:47.172+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:47.487 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:47.172+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:47.487 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:47 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:47.172+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:14.630460+0000 front 2024-09-15T23:42:14.630411+0000 (oldest deadline 2024-09-15T23:42:43.430966+0000) 2024-09-15T23:42:47.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:47 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:47.104+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:47.561 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:47 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:47.104+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:47.561 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:47 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:47.104+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:47.561 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:47 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:47.104+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:47.561 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:47 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:47.104+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:48.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:47 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:47.764+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:48.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:47 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:47.764+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:48.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:47 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:47.764+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:48.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:47 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:47.764+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:48.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:47 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:47.764+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:48.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:48 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:48.087+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:48.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:48 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:48.087+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:48.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:48 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:48.087+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:48.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:48 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:48.087+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:48.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:48 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:48.182+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:48.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:48 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:48.182+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:48.487 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:48 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:48.182+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:48.487 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:48 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:48.182+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:14.630460+0000 front 2024-09-15T23:42:14.630411+0000 (oldest deadline 2024-09-15T23:42:43.430966+0000) 2024-09-15T23:42:48.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:48 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:48.071+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:48.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:48 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:48.071+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:48.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:48 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:48.071+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:48.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:48 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:48.071+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:48.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:48 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:48.071+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:49.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:48 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:48.726+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:49.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:48 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:48.726+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:49.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:48 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:48.726+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:49.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:48 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:48.726+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:49.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:48 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:48.726+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:49.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:49 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:49.043+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:49.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:49 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:49.043+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:49.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:49 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:49.043+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:49.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:49 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:49.043+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:49.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:49 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:49.213+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:49.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:49 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:49.213+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:49.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:49 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:49.213+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:49.487 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:49 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:49.213+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:14.630460+0000 front 2024-09-15T23:42:14.630411+0000 (oldest deadline 2024-09-15T23:42:43.430966+0000) 2024-09-15T23:42:49.487 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:49 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:49.213+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.232269+0000 front 2024-09-15T23:42:26.232293+0000 (oldest deadline 2024-09-15T23:42:48.532444+0000) 2024-09-15T23:42:49.561 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:49 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:49.063+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:49.561 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:49 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:49.063+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:49.561 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:49 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:49.063+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:49.562 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:49 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:49.063+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:49.562 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:49 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:49.063+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:49.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:49 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:49.717+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:49.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:49 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:49.717+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:49.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:49 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:49.717+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:49.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:49 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:49.717+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:49.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:49 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:49.717+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:50.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:50 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:50.020+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:50.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:50 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:50.020+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:50.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:50 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:50.020+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:50.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:50 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:50.020+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:50.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:50 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:50.020+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:50.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:50.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:50.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:50.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:50.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:50.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:50.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:50.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:50.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:50.233+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:50.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:50.233+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:50.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:50.233+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:50.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:50.233+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:14.630460+0000 front 2024-09-15T23:42:14.630411+0000 (oldest deadline 2024-09-15T23:42:43.430966+0000) 2024-09-15T23:42:50.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:50.233+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.232269+0000 front 2024-09-15T23:42:26.232293+0000 (oldest deadline 2024-09-15T23:42:48.532444+0000) 2024-09-15T23:42:50.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:50 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:50.685+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:50.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:50 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:50.685+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:50.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:50 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:50.685+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:50.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:50 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:50.685+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:50.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:50 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:50.685+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:50.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:50 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:50.685+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:42:51.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:51 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:51.007+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:51.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:51 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:51.007+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:51.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:51 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:51.007+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:51.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:51 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:51.007+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:51.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:51 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:51.007+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:51.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:51 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:51.077+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:51.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:51 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:51.077+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:51.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:51 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:51.077+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:51.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:51 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:51.077+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:51.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:51 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:51.077+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:42:51.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:51 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:51.206+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:50.027782+0000 front 2024-09-15T23:41:50.027905+0000 (oldest deadline 2024-09-15T23:42:21.228155+0000) 2024-09-15T23:42:51.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:51 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:51.206+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.228476+0000 front 2024-09-15T23:42:01.228539+0000 (oldest deadline 2024-09-15T23:42:26.528565+0000) 2024-09-15T23:42:51.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:51 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:51.206+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:10.530139+0000 front 2024-09-15T23:42:10.530117+0000 (oldest deadline 2024-09-15T23:42:34.630010+0000) 2024-09-15T23:42:51.486 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:51 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:51.206+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:14.630460+0000 front 2024-09-15T23:42:14.630411+0000 (oldest deadline 2024-09-15T23:42:43.430966+0000) 2024-09-15T23:42:51.487 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:51 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8[53893]: 2024-09-15T23:42:51.206+0000 7f5f4b538640 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.232269+0000 front 2024-09-15T23:42:26.232293+0000 (oldest deadline 2024-09-15T23:42:48.532444+0000) 2024-09-15T23:42:51.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:51 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:51.708+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:51.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:51 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:51.708+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:51.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:51 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:51.708+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:51.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:51 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:51.708+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:51.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:51 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:51.708+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:51.974 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:51 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:51.708+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:42:52.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:51 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:51.967+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:52.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:51 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:51.967+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:52.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:51 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:51.967+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:52.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:51 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:51.967+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:52.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:51 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:51.967+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:52.482 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:52 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:52.085+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:52.482 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:52 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:52.085+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:52.482 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:52 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:52.085+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:52.482 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:52 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:52.085+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:52.482 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:52 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:52.085+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:42:52.735 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:52 smithi174 podman[71993]: 2024-09-15 23:42:52.481401766 +0000 UTC m=+5.602597884 container died b3d67fe4ecafb776e90d29f25f876561dee207c9633154610d1b440347764ad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-15T23:42:53.090 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:52 smithi174 podman[71993]: 2024-09-15 23:42:52.850186892 +0000 UTC m=+5.971383004 container cleanup b3d67fe4ecafb776e90d29f25f876561dee207c9633154610d1b440347764ad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-15T23:42:53.090 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:52 smithi174 bash[71993]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8 2024-09-15T23:42:53.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:52 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:52.742+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:53.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:52 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:52.742+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:53.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:52 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:52.742+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:53.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:52 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:52.742+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:53.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:52 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:52.742+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:53.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:52 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:52.742+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:42:53.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:52.999+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:53.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:52.999+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:53.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:52.999+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:53.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:52.999+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:53.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:52.999+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:53.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:52.999+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.879170+0000 front 2024-09-15T23:42:26.879033+0000 (oldest deadline 2024-09-15T23:42:52.178910+0000) 2024-09-15T23:42:53.461 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:53 smithi174 podman[72009]: 2024-09-15 23:42:53.134652718 +0000 UTC m=+0.647009914 container remove b3d67fe4ecafb776e90d29f25f876561dee207c9633154610d1b440347764ad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-15T23:42:53.461 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:53 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:53.089+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:53.461 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:53 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:53.089+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:53.462 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:53 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:53.089+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:53.462 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:53 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:53.089+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:53.462 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:53 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:53.089+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:42:53.721 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:53 smithi174 podman[72094]: 2024-09-15 23:42:53.363479979 +0000 UTC m=+0.016070715 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:42:53.721 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:53 smithi174 podman[72094]: 2024-09-15 23:42:53.493733008 +0000 UTC m=+0.146323780 container create 6935b72235da3ccc29ca3c38fe944c932eef75c29431a137c427d681f5283af6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-15T23:42:53.986 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:53 smithi174 podman[72094]: 2024-09-15 23:42:53.721147728 +0000 UTC m=+0.373738464 container init 6935b72235da3ccc29ca3c38fe944c932eef75c29431a137c427d681f5283af6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-15T23:42:53.986 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:53 smithi174 podman[72094]: 2024-09-15 23:42:53.724376628 +0000 UTC m=+0.376967362 container start 6935b72235da3ccc29ca3c38fe944c932eef75c29431a137c427d681f5283af6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-09-15T23:42:53.986 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:53 smithi174 podman[72094]: 2024-09-15 23:42:53.821265334 +0000 UTC m=+0.473856075 container attach 6935b72235da3ccc29ca3c38fe944c932eef75c29431a137c427d681f5283af6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-15T23:42:53.986 INFO:journalctl@ceph.osd.8.smithi174.stdout:Sep 15 23:42:53 smithi174 podman[72117]: 2024-09-15 23:42:53.940686095 +0000 UTC m=+0.016781065 container died 6935b72235da3ccc29ca3c38fe944c932eef75c29431a137c427d681f5283af6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:42:54.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:53 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:53.788+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:54.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:53 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:53.788+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:54.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:53 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:53.788+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:54.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:53 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:53.788+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:54.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:53 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:53.788+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:54.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:53 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:53.788+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:42:54.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:53.963+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:54.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:53.963+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:54.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:53.963+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:54.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:53.963+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:54.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:53.963+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:54.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:53 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:53.963+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.879170+0000 front 2024-09-15T23:42:26.879033+0000 (oldest deadline 2024-09-15T23:42:52.178910+0000) 2024-09-15T23:42:54.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:54 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:54.105+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:54.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:54 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:54.105+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:54.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:54 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:54.105+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:54.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:54 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:54.105+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:54.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:54 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:54.105+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:42:54.511 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.8.service' 2024-09-15T23:42:54.550 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:42:54.550 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-15T23:42:54.550 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-15T23:42:54.550 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.9 2024-09-15T23:42:54.976 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:54 smithi089 systemd[1]: Stopping Ceph osd.9 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:42:55.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:54 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:54.837+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:55.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:54 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:54.837+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:55.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:54 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:54.837+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:55.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:54 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:54.837+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:55.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:54 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:54.837+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:55.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:54 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:54.837+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:42:55.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:54 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:54.974+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:55.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:54 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:54.974+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:55.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:54 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:54.974+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:55.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:54 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:54.974+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:55.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:54 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:54.974+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:55.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:54 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:54.975+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.879170+0000 front 2024-09-15T23:42:26.879033+0000 (oldest deadline 2024-09-15T23:42:52.178910+0000) 2024-09-15T23:42:55.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:55 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:55.006+0000 7f794045a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:42:55.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:55 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:55.006+0000 7f794045a640 -1 osd.9 94 *** Got signal Terminated *** 2024-09-15T23:42:55.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:55 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:55.006+0000 7f794045a640 -1 osd.9 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:42:55.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:55 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:55.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:55.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:55 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:55.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:55.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:55 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:55.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:55.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:55 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:55.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:55.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:55 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:55.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:42:56.060 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:55 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:55.975+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:56.060 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:55 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:55.975+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:56.060 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:55 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:55.975+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:56.060 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:55 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:55.975+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:56.061 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:55 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:55.975+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:56.061 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:55 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:55.975+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.879170+0000 front 2024-09-15T23:42:26.879033+0000 (oldest deadline 2024-09-15T23:42:52.178910+0000) 2024-09-15T23:42:56.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:55 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:55.876+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:56.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:55 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:55.876+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:56.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:55 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:55.876+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:56.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:55 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:55.876+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:56.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:55 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:55.876+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:56.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:55 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:55.876+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:42:56.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:56 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:56.127+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:56.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:56 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:56.127+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:56.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:56 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:56.127+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:56.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:56 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:56.127+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:56.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:56 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:56.127+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:42:57.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:56 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:56.924+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:57.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:56 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:56.924+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:57.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:56 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:56.924+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:57.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:56 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:56.924+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:57.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:56 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:56.924+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:57.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:56 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:56.924+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:42:57.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:56.956+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:57.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:56.956+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:57.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:56.956+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:57.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:56.956+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:57.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:56.956+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:57.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:56 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:56.956+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.879170+0000 front 2024-09-15T23:42:26.879033+0000 (oldest deadline 2024-09-15T23:42:52.178910+0000) 2024-09-15T23:42:57.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:57 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:57.145+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:57.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:57 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:57.145+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:57.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:57 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:57.145+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:57.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:57 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:57.145+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:57.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:57 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:57.145+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:42:58.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:57 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:57.889+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:58.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:57 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:57.889+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:58.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:57 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:57.889+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:58.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:57 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:57.889+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:58.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:57 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:57.889+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:58.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:57 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:57.889+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:42:58.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:57 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:57.889+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:42:58.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:57 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:57.973+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:58.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:57 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:57.973+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:58.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:57 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:57.973+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:58.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:57 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:57.973+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:58.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:57 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:57.973+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:58.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:57 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:57.973+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.879170+0000 front 2024-09-15T23:42:26.879033+0000 (oldest deadline 2024-09-15T23:42:52.178910+0000) 2024-09-15T23:42:58.387 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:58.109+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:58.387 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:58.109+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:58.387 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:58.109+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:58.387 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:58.109+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:58.387 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:58.109+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:42:59.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:58 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:58.879+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:42:59.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:58 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:58.879+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:42:59.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:58 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:58.879+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:42:59.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:58 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:58.879+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:42:59.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:58 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:58.879+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:42:59.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:58 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:58.879+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:42:59.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:58 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:58.879+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:42:59.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:58 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:58.949+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:42:59.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:58 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:58.949+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:42:59.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:58 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:58.949+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:42:59.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:58 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:58.949+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:42:59.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:58 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:58.949+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:42:59.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:58 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:58.949+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.879170+0000 front 2024-09-15T23:42:26.879033+0000 (oldest deadline 2024-09-15T23:42:52.178910+0000) 2024-09-15T23:42:59.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:59 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:59.075+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:42:59.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:59 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:59.075+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:42:59.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:59 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:59.075+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:42:59.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:59 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:59.075+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:42:59.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:42:59 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:42:59.075+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:00.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:59 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:59.897+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:43:00.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:59 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:59.897+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:43:00.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:59 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:59.897+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:43:00.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:59 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:59.897+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:43:00.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:59 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:59.897+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:43:00.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:59 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:59.897+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:43:00.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:42:59 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:42:59.897+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:43:00.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:59.954+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:44.372649+0000 front 2024-09-15T23:41:44.372610+0000 (oldest deadline 2024-09-15T23:42:09.072783+0000) 2024-09-15T23:43:00.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:59.954+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.373600+0000 front 2024-09-15T23:41:54.373760+0000 (oldest deadline 2024-09-15T23:42:17.273525+0000) 2024-09-15T23:43:00.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:59.954+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:42:01.974592+0000 front 2024-09-15T23:42:01.974453+0000 (oldest deadline 2024-09-15T23:42:25.974735+0000) 2024-09-15T23:43:00.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:59.954+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:11.076603+0000 front 2024-09-15T23:42:11.076583+0000 (oldest deadline 2024-09-15T23:42:35.176777+0000) 2024-09-15T23:43:00.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:59.954+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:18.077967+0000 front 2024-09-15T23:42:18.077892+0000 (oldest deadline 2024-09-15T23:42:42.777925+0000) 2024-09-15T23:43:00.311 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:42:59 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9[64125]: 2024-09-15T23:42:59.954+0000 7f793ca70640 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:26.879170+0000 front 2024-09-15T23:42:26.879033+0000 (oldest deadline 2024-09-15T23:42:52.178910+0000) 2024-09-15T23:43:00.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:00 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:00.097+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:00.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:00 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:00.097+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:00.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:00 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:00.097+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:00.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:00 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:00.097+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:00.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:00 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:00.097+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:00.812 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:43:00 smithi089 podman[135172]: 2024-09-15 23:43:00.538170655 +0000 UTC m=+5.785564497 container died 64698b35e1d8480cd6b7b383bab421c4d102f27615d254493c9daa701d821736 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-8921c8e, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-15T23:43:01.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:00 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:00.906+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:43:01.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:00 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:00.906+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:43:01.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:00 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:00.906+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:43:01.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:00 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:00.906+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:43:01.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:00 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:00.906+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:43:01.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:00 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:00.906+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:43:01.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:00 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:00.906+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:43:01.265 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:43:00 smithi089 podman[135172]: 2024-09-15 23:43:00.988409524 +0000 UTC m=+6.235803406 container cleanup 64698b35e1d8480cd6b7b383bab421c4d102f27615d254493c9daa701d821736 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-15T23:43:01.265 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:43:00 smithi089 bash[135172]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9 2024-09-15T23:43:01.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:01.079+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:01.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:01.079+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:01.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:01.079+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:01.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:01.079+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:01.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:01.079+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:01.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:01 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:01.079+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:01.560 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:43:01 smithi089 podman[135188]: 2024-09-15 23:43:01.26466991 +0000 UTC m=+0.721642045 container remove 64698b35e1d8480cd6b7b383bab421c4d102f27615d254493c9daa701d821736 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-15T23:43:01.926 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:43:01 smithi089 podman[135275]: 2024-09-15 23:43:01.514106718 +0000 UTC m=+0.015801948 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:43:01.926 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:43:01 smithi089 podman[135275]: 2024-09-15 23:43:01.623690329 +0000 UTC m=+0.125385559 container create a172e693c7429f4db8126a7ac384e5314ef26c27bfa37969fa7095d898543306 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:43:01.926 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:43:01 smithi089 podman[135275]: 2024-09-15 23:43:01.850985841 +0000 UTC m=+0.352681072 container init a172e693c7429f4db8126a7ac384e5314ef26c27bfa37969fa7095d898543306 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:43:01.926 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:43:01 smithi089 podman[135275]: 2024-09-15 23:43:01.854213711 +0000 UTC m=+0.355908950 container start a172e693c7429f4db8126a7ac384e5314ef26c27bfa37969fa7095d898543306 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-15T23:43:01.926 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:43:01 smithi089 podman[135275]: 2024-09-15 23:43:01.925773425 +0000 UTC m=+0.427468668 container attach a172e693c7429f4db8126a7ac384e5314ef26c27bfa37969fa7095d898543306 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-15T23:43:02.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:01.896+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:43:02.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:01.896+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:43:02.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:01.896+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:43:02.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:01.896+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:43:02.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:01.896+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:43:02.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:01.896+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:43:02.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:01 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:01.896+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:43:02.310 INFO:journalctl@ceph.osd.9.smithi089.stdout:Sep 15 23:43:02 smithi089 podman[135275]: 2024-09-15 23:43:02.041383954 +0000 UTC m=+0.543079192 container died a172e693c7429f4db8126a7ac384e5314ef26c27bfa37969fa7095d898543306 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-15T23:43:02.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:02.099+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:02.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:02.099+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:02.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:02.099+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:02.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:02.099+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:02.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:02.099+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:02.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:02.099+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:02.662 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.9.service' 2024-09-15T23:43:02.700 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:02.701 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-15T23:43:02.701 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-15T23:43:02.701 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.10 2024-09-15T23:43:03.007 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:02 smithi120 systemd[1]: Stopping Ceph osd.10 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:03.007 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:02 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:02.923+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:43:03.007 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:02 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:02.923+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:43:03.007 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:02 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:02.923+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:43:03.007 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:02 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:02.923+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:43:03.007 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:02 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:02.923+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:43:03.007 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:02 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:02.923+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:43:03.007 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:02 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:02.923+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:43:03.007 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:03.005+0000 7f7e10fa1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:03.007 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:03.005+0000 7f7e10fa1640 -1 osd.10 94 *** Got signal Terminated *** 2024-09-15T23:43:03.008 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:03.005+0000 7f7e10fa1640 -1 osd.10 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:43:03.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:03 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:03.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:03.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:03 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:03.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:03.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:03 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:03.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:03.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:03 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:03.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:03.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:03 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:03.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:03.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:03 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:03.091+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:04.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:03.939+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:43:04.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:03.939+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:43:04.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:03.939+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:43:04.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:03.939+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:43:04.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:03.939+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:43:04.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:03.939+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:43:04.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:03 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:03.939+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:43:04.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:04 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:04.132+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:04.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:04 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:04.132+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:04.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:04 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:04.132+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:04.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:04 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:04.132+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:04.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:04 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:04.132+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:04.487 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:04 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:04.132+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:05.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:04.953+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:43:05.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:04.953+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:43:05.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:04.953+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:43:05.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:04.953+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:43:05.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:04.953+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:43:05.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:04.953+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:43:05.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:04 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:04.953+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:43:05.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:05 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:05.090+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:05.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:05 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:05.090+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:05.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:05 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:05.090+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:05.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:05 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:05.090+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:05.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:05 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:05.090+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:05.487 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:05 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:05.090+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:06.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:05 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:05.925+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:43:06.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:05 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:05.925+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:43:06.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:05 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:05.925+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:43:06.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:05 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:05.925+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:43:06.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:05 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:05.925+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:43:06.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:05 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:05.925+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:43:06.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:05 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:05.925+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:43:06.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:06 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:06.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:06.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:06 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:06.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:06.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:06 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:06.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:06.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:06 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:06.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:06.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:06 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:06.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:06.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:06 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:06.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:06.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:06 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:06.068+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-09-15T23:42:41.300120+0000 front 2024-09-15T23:42:41.300140+0000 (oldest deadline 2024-09-15T23:43:06.000169+0000) 2024-09-15T23:43:07.473 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:06 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:06.972+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:43:07.473 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:06 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:06.972+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:43:07.474 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:06 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:06.972+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:43:07.474 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:06 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:06.972+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:43:07.474 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:06 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:06.972+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:43:07.474 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:06 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:06.972+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:43:07.474 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:06 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:06.972+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:43:07.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:07 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:07.027+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:07.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:07 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:07.027+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:07.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:07 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:07.027+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:07.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:07 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:07.027+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:07.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:07 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:07.027+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:07.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:07 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:07.027+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:07.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:07 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:07.027+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-09-15T23:42:41.300120+0000 front 2024-09-15T23:42:41.300140+0000 (oldest deadline 2024-09-15T23:43:06.000169+0000) 2024-09-15T23:43:08.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:07 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:07.970+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:46.168554+0000 front 2024-09-15T23:41:46.168420+0000 (oldest deadline 2024-09-15T23:42:10.268604+0000) 2024-09-15T23:43:08.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:07 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:07.970+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.369566+0000 front 2024-09-15T23:41:54.369597+0000 (oldest deadline 2024-09-15T23:42:18.469700+0000) 2024-09-15T23:43:08.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:07 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:07.970+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.470210+0000 front 2024-09-15T23:41:58.470127+0000 (oldest deadline 2024-09-15T23:42:24.370139+0000) 2024-09-15T23:43:08.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:07 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:07.970+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6814 osd.3 since back 2024-09-15T23:42:08.971795+0000 front 2024-09-15T23:42:08.971928+0000 (oldest deadline 2024-09-15T23:42:33.671720+0000) 2024-09-15T23:43:08.223 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:07 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:07.970+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.772825+0000 front 2024-09-15T23:42:17.772881+0000 (oldest deadline 2024-09-15T23:42:41.872983+0000) 2024-09-15T23:43:08.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:07 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:07.970+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:24.774148+0000 front 2024-09-15T23:42:24.774088+0000 (oldest deadline 2024-09-15T23:42:50.073789+0000) 2024-09-15T23:43:08.224 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:07 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10[59812]: 2024-09-15T23:43:07.970+0000 7f7e0d5b7640 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:35.275189+0000 front 2024-09-15T23:42:35.275128+0000 (oldest deadline 2024-09-15T23:42:57.574970+0000) 2024-09-15T23:43:08.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.000+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:08.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.000+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:08.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.000+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:08.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.000+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:08.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.000+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:08.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.000+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:08.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.000+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-09-15T23:42:41.300120+0000 front 2024-09-15T23:42:41.300140+0000 (oldest deadline 2024-09-15T23:43:06.000169+0000) 2024-09-15T23:43:08.722 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:08 smithi120 podman[71855]: 2024-09-15 23:43:08.466176112 +0000 UTC m=+5.576758154 container died 136840411a9ea1fdce8f63cd8de12adb5cb44017d5ef1c3ab8a2eb6a8c1dee5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-15T23:43:08.722 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:08 smithi120 podman[71855]: 2024-09-15 23:43:08.722154973 +0000 UTC m=+5.832737015 container cleanup 136840411a9ea1fdce8f63cd8de12adb5cb44017d5ef1c3ab8a2eb6a8c1dee5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-15T23:43:08.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:08 smithi120 bash[71855]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10 2024-09-15T23:43:09.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.980+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:09.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.980+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:09.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.980+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:09.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.980+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:09.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.980+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:09.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.980+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:09.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:08 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:08.980+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-09-15T23:42:41.300120+0000 front 2024-09-15T23:42:41.300140+0000 (oldest deadline 2024-09-15T23:43:06.000169+0000) 2024-09-15T23:43:09.338 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:08 smithi120 podman[71870]: 2024-09-15 23:43:08.981378356 +0000 UTC m=+0.517180810 container remove 136840411a9ea1fdce8f63cd8de12adb5cb44017d5ef1c3ab8a2eb6a8c1dee5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e) 2024-09-15T23:43:09.668 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:09 smithi120 podman[71959]: 2024-09-15 23:43:09.241243083 +0000 UTC m=+0.015967995 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:43:09.668 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:09 smithi120 podman[71959]: 2024-09-15 23:43:09.340408953 +0000 UTC m=+0.115133866 container create 87da98eb149641907512fc428685c77cb60556f962af790df810c73d30aa1da8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-15T23:43:09.669 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:09 smithi120 podman[71959]: 2024-09-15 23:43:09.56840019 +0000 UTC m=+0.343125095 container init 87da98eb149641907512fc428685c77cb60556f962af790df810c73d30aa1da8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:43:09.669 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:09 smithi120 podman[71959]: 2024-09-15 23:43:09.571716979 +0000 UTC m=+0.346441890 container start 87da98eb149641907512fc428685c77cb60556f962af790df810c73d30aa1da8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:43:09.669 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:09 smithi120 podman[71959]: 2024-09-15 23:43:09.668267091 +0000 UTC m=+0.442991998 container attach 87da98eb149641907512fc428685c77cb60556f962af790df810c73d30aa1da8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:43:09.973 INFO:journalctl@ceph.osd.10.smithi120.stdout:Sep 15 23:43:09 smithi120 podman[71959]: 2024-09-15 23:43:09.759512823 +0000 UTC m=+0.534237735 container died 87da98eb149641907512fc428685c77cb60556f962af790df810c73d30aa1da8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-10-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:43:10.235 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:09 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:09.968+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:10.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:09 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:09.968+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:10.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:09 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:09.968+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:10.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:09 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:09.968+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:10.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:09 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:09.968+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:10.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:09 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:09.968+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:10.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:09 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:09.968+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-09-15T23:42:41.300120+0000 front 2024-09-15T23:42:41.300140+0000 (oldest deadline 2024-09-15T23:43:06.000169+0000) 2024-09-15T23:43:10.348 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.10.service' 2024-09-15T23:43:10.386 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:10.386 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-15T23:43:10.386 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-15T23:43:10.386 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.11 2024-09-15T23:43:10.718 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 systemd[1]: Stopping Ceph osd.11 for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:10.980 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:10.717+0000 7f852e908640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:10.980 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:10.717+0000 7f852e908640 -1 osd.11 94 *** Got signal Terminated *** 2024-09-15T23:43:10.980 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:10.717+0000 7f852e908640 -1 osd.11 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:43:11.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:10.979+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:11.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:10.979+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:11.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:10.979+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:11.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:10.979+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:11.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:10.979+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:11.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:10.979+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:11.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:10 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:10.979+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-09-15T23:42:41.300120+0000 front 2024-09-15T23:42:41.300140+0000 (oldest deadline 2024-09-15T23:43:06.000169+0000) 2024-09-15T23:43:12.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:12 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:12.024+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:12.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:12 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:12.024+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:12.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:12 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:12.024+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:12.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:12 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:12.024+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:12.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:12 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:12.024+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:12.487 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:12 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:12.024+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:12.487 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:12 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:12.024+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-09-15T23:42:41.300120+0000 front 2024-09-15T23:42:41.300140+0000 (oldest deadline 2024-09-15T23:43:06.000169+0000) 2024-09-15T23:43:13.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:13.011+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:13.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:13.011+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:13.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:13.011+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:13.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:13.011+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:13.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:13.011+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:13.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:13.011+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:13.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:13.011+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-09-15T23:42:41.300120+0000 front 2024-09-15T23:42:41.300140+0000 (oldest deadline 2024-09-15T23:43:06.000169+0000) 2024-09-15T23:43:13.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:13 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:13.011+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6820 osd.8 since back 2024-09-15T23:42:51.301059+0000 front 2024-09-15T23:42:51.301020+0000 (oldest deadline 2024-09-15T23:43:13.001066+0000) 2024-09-15T23:43:14.485 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.003+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:14.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.003+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:14.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.003+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:14.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.003+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:14.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.003+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:14.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.003+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:14.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.003+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-09-15T23:42:41.300120+0000 front 2024-09-15T23:42:41.300140+0000 (oldest deadline 2024-09-15T23:43:06.000169+0000) 2024-09-15T23:43:14.486 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.003+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6820 osd.8 since back 2024-09-15T23:42:51.301059+0000 front 2024-09-15T23:42:51.301020+0000 (oldest deadline 2024-09-15T23:43:13.001066+0000) 2024-09-15T23:43:15.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.981+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6806 osd.0 since back 2024-09-15T23:41:43.190020+0000 front 2024-09-15T23:41:43.190153+0000 (oldest deadline 2024-09-15T23:42:08.490158+0000) 2024-09-15T23:43:15.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.981+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6804 osd.1 since back 2024-09-15T23:41:54.892015+0000 front 2024-09-15T23:41:54.891899+0000 (oldest deadline 2024-09-15T23:42:16.591863+0000) 2024-09-15T23:43:15.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.981+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-09-15T23:41:58.292803+0000 front 2024-09-15T23:41:58.292822+0000 (oldest deadline 2024-09-15T23:42:24.693550+0000) 2024-09-15T23:43:15.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.981+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6812 osd.4 since back 2024-09-15T23:42:17.396826+0000 front 2024-09-15T23:42:17.396982+0000 (oldest deadline 2024-09-15T23:42:42.096913+0000) 2024-09-15T23:43:15.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.981+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-09-15T23:42:27.298281+0000 front 2024-09-15T23:42:27.298313+0000 (oldest deadline 2024-09-15T23:42:50.198357+0000) 2024-09-15T23:43:15.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.981+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.89:6822 osd.6 since back 2024-09-15T23:42:34.299545+0000 front 2024-09-15T23:42:34.299523+0000 (oldest deadline 2024-09-15T23:43:00.199237+0000) 2024-09-15T23:43:15.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.981+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.120:6820 osd.7 since back 2024-09-15T23:42:41.300120+0000 front 2024-09-15T23:42:41.300140+0000 (oldest deadline 2024-09-15T23:43:06.000169+0000) 2024-09-15T23:43:15.236 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:14 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11[60716]: 2024-09-15T23:43:14.981+0000 7f8526f16640 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.174:6820 osd.8 since back 2024-09-15T23:42:51.301059+0000 front 2024-09-15T23:42:51.301020+0000 (oldest deadline 2024-09-15T23:43:13.001066+0000) 2024-09-15T23:43:16.318 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:16 smithi174 podman[72209]: 2024-09-15 23:43:16.06372884 +0000 UTC m=+5.481988148 container died 70559f4c28d23262986a3f7d6581d28233c828bf7033202d32613c84a21d5c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True) 2024-09-15T23:43:16.594 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:16 smithi174 podman[72209]: 2024-09-15 23:43:16.400668415 +0000 UTC m=+5.818927778 container cleanup 70559f4c28d23262986a3f7d6581d28233c828bf7033202d32613c84a21d5c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-15T23:43:16.594 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:16 smithi174 bash[72209]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11 2024-09-15T23:43:16.985 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:16 smithi174 podman[72224]: 2024-09-15 23:43:16.593501118 +0000 UTC m=+0.526186146 container remove 70559f4c28d23262986a3f7d6581d28233c828bf7033202d32613c84a21d5c38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731) 2024-09-15T23:43:17.321 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:17 smithi174 podman[72312]: 2024-09-15 23:43:16.914308071 +0000 UTC m=+0.016011657 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:43:17.322 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:17 smithi174 podman[72312]: 2024-09-15 23:43:17.027371661 +0000 UTC m=+0.129075247 container create 724b20d7ecb415fb0e3e39731db947b06eaefca1426031252e95897e81ae293e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:43:17.322 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:17 smithi174 podman[72312]: 2024-09-15 23:43:17.24645681 +0000 UTC m=+0.348160395 container init 724b20d7ecb415fb0e3e39731db947b06eaefca1426031252e95897e81ae293e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_CLEAN=True) 2024-09-15T23:43:17.322 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:17 smithi174 podman[72312]: 2024-09-15 23:43:17.249814599 +0000 UTC m=+0.351518195 container start 724b20d7ecb415fb0e3e39731db947b06eaefca1426031252e95897e81ae293e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True) 2024-09-15T23:43:17.322 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:17 smithi174 podman[72312]: 2024-09-15 23:43:17.321141658 +0000 UTC m=+0.422845245 container attach 724b20d7ecb415fb0e3e39731db947b06eaefca1426031252e95897e81ae293e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:43:17.736 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:17 smithi174 podman[72312]: 2024-09-15 23:43:17.437168275 +0000 UTC m=+0.538871868 container died 724b20d7ecb415fb0e3e39731db947b06eaefca1426031252e95897e81ae293e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-15T23:43:18.031 INFO:journalctl@ceph.osd.11.smithi174.stdout:Sep 15 23:43:18 smithi174 podman[72336]: 2024-09-15 23:43:18.007427703 +0000 UTC m=+0.564455130 container remove 724b20d7ecb415fb0e3e39731db947b06eaefca1426031252e95897e81ae293e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-15T23:43:18.037 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@osd.11.service' 2024-09-15T23:43:18.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:18.078 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-15T23:43:18.078 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-15T23:43:18.078 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.a 2024-09-15T23:43:18.407 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:43:18 smithi089 systemd[1]: Stopping Ceph mds.a for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:18.810 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:43:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a[71412]: 2024-09-15T23:43:18.406+0000 7f5f98bfe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:18.810 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:43:18 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a[71412]: 2024-09-15T23:43:18.406+0000 7f5f98bfe640 -1 mds.a *** got signal Terminated *** 2024-09-15T23:43:21.890 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:43:21 smithi089 podman[135389]: 2024-09-15 23:43:21.609024953 +0000 UTC m=+3.334431241 container died ccb0f0bc2bb76ad6836d084eebb037c9289fe8a66c538e5312e02dee6f69a040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-15T23:43:22.310 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:43:22 smithi089 podman[135389]: 2024-09-15 23:43:22.072578432 +0000 UTC m=+3.797984712 container cleanup ccb0f0bc2bb76ad6836d084eebb037c9289fe8a66c538e5312e02dee6f69a040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a, GIT_CLEAN=True, RELEASE=squid-8921c8e, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-15T23:43:22.310 INFO:journalctl@ceph.mds.a.smithi089.stdout:Sep 15 23:43:22 smithi089 bash[135389]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-a 2024-09-15T23:43:22.507 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.a.service' 2024-09-15T23:43:22.541 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:22.541 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-15T23:43:22.541 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-15T23:43:22.541 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.d 2024-09-15T23:43:22.852 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:43:22 smithi089 systemd[1]: Stopping Ceph mds.d for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:22.853 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:43:22 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-d[70202]: 2024-09-15T23:43:22.851+0000 7fe163446640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:22.853 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:43:22 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-d[70202]: 2024-09-15T23:43:22.851+0000 7fe163446640 -1 mds.d *** got signal Terminated *** 2024-09-15T23:43:25.501 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:43:25 smithi089 podman[135510]: 2024-09-15 23:43:25.126155894 +0000 UTC m=+2.371401939 container died 6da1be8a1e9c37eb267a5c2fa980d5b8116d516c7d2aed02fa81ba137c86549f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True) 2024-09-15T23:43:25.502 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:43:25 smithi089 podman[135510]: 2024-09-15 23:43:25.501476923 +0000 UTC m=+2.746722968 container cleanup 6da1be8a1e9c37eb267a5c2fa980d5b8116d516c7d2aed02fa81ba137c86549f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-d, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-15T23:43:25.778 INFO:journalctl@ceph.mds.d.smithi089.stdout:Sep 15 23:43:25 smithi089 bash[135510]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-d 2024-09-15T23:43:25.924 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.d.service' 2024-09-15T23:43:25.955 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:25.956 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-15T23:43:25.956 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-15T23:43:25.956 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.g 2024-09-15T23:43:26.310 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:43:26 smithi089 systemd[1]: Stopping Ceph mds.g for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:26.310 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:43:26 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g[71076]: 2024-09-15T23:43:26.279+0000 7f294ef8c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:26.310 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:43:26 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g[71076]: 2024-09-15T23:43:26.279+0000 7f294ef8c640 -1 mds.g *** got signal Terminated *** 2024-09-15T23:43:29.897 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:43:29 smithi089 podman[135632]: 2024-09-15 23:43:29.610119096 +0000 UTC m=+3.434212069 container died d5083778b9544e1e61ae957879e3c29f5564e6fe76ca10bff48b9c959bf57825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-15T23:43:30.240 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:43:29 smithi089 podman[135632]: 2024-09-15 23:43:29.988320642 +0000 UTC m=+3.812413603 container cleanup d5083778b9544e1e61ae957879e3c29f5564e6fe76ca10bff48b9c959bf57825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-15T23:43:30.240 INFO:journalctl@ceph.mds.g.smithi089.stdout:Sep 15 23:43:29 smithi089 bash[135632]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-g 2024-09-15T23:43:30.365 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.g.service' 2024-09-15T23:43:30.396 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:30.397 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-15T23:43:30.397 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-15T23:43:30.397 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.j 2024-09-15T23:43:30.560 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:43:30 smithi089 systemd[1]: Stopping Ceph mds.j for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:31.060 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:43:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-j[69801]: 2024-09-15T23:43:30.708+0000 7f6367663640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:31.060 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:43:30 smithi089 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-j[69801]: 2024-09-15T23:43:30.708+0000 7f6367663640 -1 mds.j *** got signal Terminated *** 2024-09-15T23:43:33.735 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:43:33 smithi089 podman[135753]: 2024-09-15 23:43:33.472649335 +0000 UTC m=+2.863184941 container died 1c1226337f697cef385c0e316a9c13c539dc324f9feb9d9f95dd1c13c8aad683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-j, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-15T23:43:33.997 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:43:33 smithi089 podman[135753]: 2024-09-15 23:43:33.817397879 +0000 UTC m=+3.207933492 container cleanup 1c1226337f697cef385c0e316a9c13c539dc324f9feb9d9f95dd1c13c8aad683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-15T23:43:33.997 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:43:33 smithi089 bash[135753]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-j 2024-09-15T23:43:34.243 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.j.service' 2024-09-15T23:43:34.265 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:43:33 smithi089 podman[135768]: 2024-09-15 23:43:33.99321028 +0000 UTC m=+0.513553704 container remove 1c1226337f697cef385c0e316a9c13c539dc324f9feb9d9f95dd1c13c8aad683 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-j, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:43:34.265 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:43:34 smithi089 systemd[1]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.j.service: Deactivated successfully. 2024-09-15T23:43:34.265 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:43:34 smithi089 systemd[1]: Stopped Ceph mds.j for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:43:34.265 INFO:journalctl@ceph.mds.j.smithi089.stdout:Sep 15 23:43:34 smithi089 systemd[1]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.j.service: Consumed 1.104s CPU time. 2024-09-15T23:43:34.275 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:34.275 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-15T23:43:34.275 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-15T23:43:34.275 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.b 2024-09-15T23:43:34.647 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:43:34 smithi120 systemd[1]: Stopping Ceph mds.b for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:34.905 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:43:34 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b[65175]: 2024-09-15T23:43:34.645+0000 7fc45042e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:34.905 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:43:34 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b[65175]: 2024-09-15T23:43:34.645+0000 7fc45042e640 -1 mds.b *** got signal Terminated *** 2024-09-15T23:43:37.973 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:43:37 smithi120 podman[72060]: 2024-09-15 23:43:37.631616692 +0000 UTC m=+3.161418963 container died c4a65ae0ee3143258f855ad0d6c16f1f1878a5b0205eb7287e65db759abcc92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-15T23:43:38.248 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:43:38 smithi120 podman[72060]: 2024-09-15 23:43:38.063061673 +0000 UTC m=+3.592863945 container cleanup c4a65ae0ee3143258f855ad0d6c16f1f1878a5b0205eb7287e65db759abcc92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e) 2024-09-15T23:43:38.248 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:43:38 smithi120 bash[72060]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b 2024-09-15T23:43:38.487 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.b.service' 2024-09-15T23:43:38.509 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:43:38 smithi120 podman[72084]: 2024-09-15 23:43:38.24736535 +0000 UTC m=+0.610168749 container remove c4a65ae0ee3143258f855ad0d6c16f1f1878a5b0205eb7287e65db759abcc92d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-15T23:43:38.510 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:43:38 smithi120 systemd[1]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.b.service: Deactivated successfully. 2024-09-15T23:43:38.510 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:43:38 smithi120 systemd[1]: Stopped Ceph mds.b for 1b4e699c-73b8-11ef-bceb-c7b262605968. 2024-09-15T23:43:38.510 INFO:journalctl@ceph.mds.b.smithi120.stdout:Sep 15 23:43:38 smithi120 systemd[1]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.b.service: Consumed 10.091s CPU time. 2024-09-15T23:43:38.523 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:38.524 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-15T23:43:38.524 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-15T23:43:38.524 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.e 2024-09-15T23:43:38.842 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:43:38 smithi120 systemd[1]: Stopping Ceph mds.e for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:39.223 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:43:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e[64573]: 2024-09-15T23:43:38.841+0000 7ff5296ee640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:39.223 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:43:38 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e[64573]: 2024-09-15T23:43:38.841+0000 7ff5296ee640 -1 mds.e *** got signal Terminated *** 2024-09-15T23:43:41.859 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:43:41 smithi120 podman[72193]: 2024-09-15 23:43:41.604265943 +0000 UTC m=+2.863741815 container died 8d6ea269aa73ad06ae128858712eeb43b20f3aacc69600cfb1d2ed69109d0ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:43:42.218 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:43:41 smithi120 podman[72193]: 2024-09-15 23:43:41.949753586 +0000 UTC m=+3.209229438 container cleanup 8d6ea269aa73ad06ae128858712eeb43b20f3aacc69600cfb1d2ed69109d0ac4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-15T23:43:42.218 INFO:journalctl@ceph.mds.e.smithi120.stdout:Sep 15 23:43:41 smithi120 bash[72193]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-e 2024-09-15T23:43:42.382 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.e.service' 2024-09-15T23:43:42.414 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:42.415 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-15T23:43:42.415 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-15T23:43:42.415 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.h 2024-09-15T23:43:42.729 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:43:42 smithi120 systemd[1]: Stopping Ceph mds.h for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:43.223 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:43:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h[64878]: 2024-09-15T23:43:42.727+0000 7f48cbf01640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:43.223 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:43:42 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h[64878]: 2024-09-15T23:43:42.727+0000 7f48cbf01640 -1 mds.h *** got signal Terminated *** 2024-09-15T23:43:45.723 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:43:45 smithi120 podman[72316]: 2024-09-15 23:43:45.603252589 +0000 UTC m=+2.977279197 container died 3b11d314aa275c9bd5bd48d1685c769e159ae4027b489c9076de96ec37adef0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-15T23:43:46.223 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:43:46 smithi120 podman[72316]: 2024-09-15 23:43:46.177891541 +0000 UTC m=+3.551918132 container cleanup 3b11d314aa275c9bd5bd48d1685c769e159ae4027b489c9076de96ec37adef0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h, GIT_CLEAN=True, RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-15T23:43:46.223 INFO:journalctl@ceph.mds.h.smithi120.stdout:Sep 15 23:43:46 smithi120 bash[72316]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-h 2024-09-15T23:43:46.617 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.h.service' 2024-09-15T23:43:46.649 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:46.649 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-15T23:43:46.649 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-15T23:43:46.649 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.k 2024-09-15T23:43:46.965 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:43:46 smithi120 systemd[1]: Stopping Ceph mds.k for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:47.223 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:43:46 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k[64272]: 2024-09-15T23:43:46.964+0000 7fb3a9b13640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:47.223 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:43:46 smithi120 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k[64272]: 2024-09-15T23:43:46.964+0000 7fb3a9b13640 -1 mds.k *** got signal Terminated *** 2024-09-15T23:43:49.899 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:43:49 smithi120 podman[72438]: 2024-09-15 23:43:49.618417286 +0000 UTC m=+2.754841600 container died 8429948d1f28c1ff791272ef3c8abfbb53722252d651bc31da208560efdd7925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-15T23:43:50.207 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:43:49 smithi120 podman[72438]: 2024-09-15 23:43:49.997807599 +0000 UTC m=+3.134231912 container cleanup 8429948d1f28c1ff791272ef3c8abfbb53722252d651bc31da208560efdd7925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-15T23:43:50.207 INFO:journalctl@ceph.mds.k.smithi120.stdout:Sep 15 23:43:49 smithi120 bash[72438]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-k 2024-09-15T23:43:50.341 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.k.service' 2024-09-15T23:43:50.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:50.373 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-15T23:43:50.373 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-15T23:43:50.373 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.c 2024-09-15T23:43:50.735 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:43:50 smithi174 systemd[1]: Stopping Ceph mds.c for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:51.235 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:43:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c[65829]: 2024-09-15T23:43:50.758+0000 7fd7b0f51640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:51.236 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:43:50 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c[65829]: 2024-09-15T23:43:50.758+0000 7fd7b0f51640 -1 mds.c *** got signal Terminated *** 2024-09-15T23:43:53.906 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:43:53 smithi174 podman[72415]: 2024-09-15 23:43:53.607333324 +0000 UTC m=+3.039805099 container died aa363d291d7c32c25010730f07dbe57beef319b2fd2761570040011806e1b2ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD) 2024-09-15T23:43:54.231 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:43:53 smithi174 podman[72415]: 2024-09-15 23:43:53.988930132 +0000 UTC m=+3.421401916 container cleanup aa363d291d7c32c25010730f07dbe57beef319b2fd2761570040011806e1b2ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-15T23:43:54.232 INFO:journalctl@ceph.mds.c.smithi174.stdout:Sep 15 23:43:53 smithi174 bash[72415]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-c 2024-09-15T23:43:54.371 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.c.service' 2024-09-15T23:43:54.404 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:54.405 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-15T23:43:54.405 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-15T23:43:54.405 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.f 2024-09-15T23:43:54.985 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:43:54 smithi174 systemd[1]: Stopping Ceph mds.f for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:54.986 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:43:54 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f[65528]: 2024-09-15T23:43:54.725+0000 7f74aa541640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:54.986 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:43:54 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f[65528]: 2024-09-15T23:43:54.725+0000 7f74aa541640 -1 mds.f *** got signal Terminated *** 2024-09-15T23:43:57.944 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:43:57 smithi174 podman[72537]: 2024-09-15 23:43:57.604757434 +0000 UTC m=+2.978666402 container died c946c964cccb66a5df4d96f1a40fd6034f3309cc2118267b544c8d4828cb23ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-15T23:43:58.218 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:43:58 smithi174 podman[72537]: 2024-09-15 23:43:58.017299198 +0000 UTC m=+3.391208166 container cleanup c946c964cccb66a5df4d96f1a40fd6034f3309cc2118267b544c8d4828cb23ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-15T23:43:58.218 INFO:journalctl@ceph.mds.f.smithi174.stdout:Sep 15 23:43:58 smithi174 bash[72537]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-f 2024-09-15T23:43:58.445 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.f.service' 2024-09-15T23:43:58.477 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:43:58.477 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-15T23:43:58.477 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-15T23:43:58.477 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.i 2024-09-15T23:43:58.735 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:43:58 smithi174 systemd[1]: Stopping Ceph mds.i for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:43:59.235 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:43:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i[66128]: 2024-09-15T23:43:58.787+0000 7f85acdee640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:43:59.235 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:43:58 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i[66128]: 2024-09-15T23:43:58.787+0000 7f85acdee640 -1 mds.i *** got signal Terminated *** 2024-09-15T23:44:01.985 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:44:01 smithi174 podman[72657]: 2024-09-15 23:44:01.62447867 +0000 UTC m=+2.934976911 container died 7e318973bd9e35db1ca9517756205a5c420fb44e83b88ba1198a96d9cfe3dea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:44:02.455 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:44:02 smithi174 podman[72657]: 2024-09-15 23:44:02.203821853 +0000 UTC m=+3.514320082 container cleanup 7e318973bd9e35db1ca9517756205a5c420fb44e83b88ba1198a96d9cfe3dea8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:44:02.455 INFO:journalctl@ceph.mds.i.smithi174.stdout:Sep 15 23:44:02 smithi174 bash[72657]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-i 2024-09-15T23:44:02.602 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.i.service' 2024-09-15T23:44:02.633 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:44:02.633 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-15T23:44:02.634 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-15T23:44:02.634 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.l 2024-09-15T23:44:02.985 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:44:02 smithi174 systemd[1]: Stopping Ceph mds.l for 1b4e699c-73b8-11ef-bceb-c7b262605968... 2024-09-15T23:44:03.485 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:44:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l[65222]: 2024-09-15T23:44:02.990+0000 7f32be4cc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:44:03.485 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:44:02 smithi174 ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l[65222]: 2024-09-15T23:44:02.990+0000 7f32be4cc640 -1 mds.l *** got signal Terminated *** 2024-09-15T23:44:05.924 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:44:05 smithi174 podman[72779]: 2024-09-15 23:44:05.601639121 +0000 UTC m=+2.752499600 container died 61681fc05d8b109552d0c3bf8c495143e22698ac274a14b3c56322f03b38fa59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-15T23:44:06.235 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:44:06 smithi174 podman[72779]: 2024-09-15 23:44:06.057341766 +0000 UTC m=+3.208202240 container cleanup 61681fc05d8b109552d0c3bf8c495143e22698ac274a14b3c56322f03b38fa59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0) 2024-09-15T23:44:06.236 INFO:journalctl@ceph.mds.l.smithi174.stdout:Sep 15 23:44:06 smithi174 bash[72779]: ceph-1b4e699c-73b8-11ef-bceb-c7b262605968-mds-l 2024-09-15T23:44:06.486 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1b4e699c-73b8-11ef-bceb-c7b262605968@mds.l.service' 2024-09-15T23:44:06.517 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:44:06.518 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-15T23:44:06.518 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-15T23:44:06.717 INFO:teuthology.orchestra.run.smithi089.stdout:Deleting cluster with fsid: 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:44:09.184 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-15T23:44:09.384 INFO:teuthology.orchestra.run.smithi120.stdout:Deleting cluster with fsid: 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:44:11.840 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-15T23:44:12.038 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:44:14.399 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:44:14.434 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:44:14.466 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:44:14.493 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-15T23:44:14.495 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309/remote/smithi089/crash 2024-09-15T23:44:14.496 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/crash -- . 2024-09-15T23:44:14.561 INFO:teuthology.orchestra.run.smithi089.stderr:tar: /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-15T23:44:14.561 INFO:teuthology.orchestra.run.smithi089.stderr:tar: Error is not recoverable: exiting now 2024-09-15T23:44:14.563 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309/remote/smithi120/crash 2024-09-15T23:44:14.564 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/crash -- . 2024-09-15T23:44:14.679 INFO:teuthology.orchestra.run.smithi120.stderr:tar: /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-15T23:44:14.680 INFO:teuthology.orchestra.run.smithi120.stderr:tar: Error is not recoverable: exiting now 2024-09-15T23:44:14.682 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309/remote/smithi174/crash 2024-09-15T23:44:14.683 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/crash -- . 2024-09-15T23:44:14.715 INFO:teuthology.orchestra.run.smithi174.stderr:tar: /var/lib/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-15T23:44:14.715 INFO:teuthology.orchestra.run.smithi174.stderr:tar: Error is not recoverable: exiting now 2024-09-15T23:44:14.716 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-15T23:44:14.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-15T23:44:14.758 INFO:tasks.cephadm:Compressing logs... 2024-09-15T23:44:14.758 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T23:44:14.801 DEBUG:teuthology.orchestra.run.smithi120:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T23:44:14.804 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T23:44:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-09-15T23:44:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:: No such file or directory 2024-09-15T23:44:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-15T23:44:14.827 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.9.log 2024-09-15T23:44:14.830 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-15T23:44:14.830 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-15T23:44:14.830 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.a.log 2024-09-15T23:44:14.830 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-volume.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-09-15T23:44:14.830 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-15T23:44:14.830 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-15T23:44:14.830 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.a.log: /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.g.log 2024-09-15T23:44:14.830 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-15T23:44:14.831 INFO:teuthology.orchestra.run.smithi089.stderr: 83.0% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-15T23:44:14.834 INFO:teuthology.orchestra.run.smithi120.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.10.log 2024-09-15T23:44:14.835 INFO:teuthology.orchestra.run.smithi120.stderr:: No such file or directory 2024-09-15T23:44:14.835 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-15T23:44:14.835 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-15T23:44:14.835 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.10.log: /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-15T23:44:14.835 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.b.log 2024-09-15T23:44:14.835 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.k.log 2024-09-15T23:44:14.835 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-15T23:44:14.835 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.b.log: /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-15T23:44:14.835 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-15T23:44:14.839 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.audit.log: 90.4% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-15T23:44:14.839 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-15T23:44:14.839 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.cephadm.log: 83.3% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-15T23:44:14.839 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.log 2024-09-15T23:44:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.g.log: 90.3% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-15T23:44:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.log 2024-09-15T23:44:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.d.log 2024-09-15T23:44:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.log: 87.6% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-15T23:44:14.842 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-15T23:44:14.843 INFO:teuthology.orchestra.run.smithi174.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-15T23:44:14.843 INFO:teuthology.orchestra.run.smithi174.stderr:: No such file or directory 2024-09-15T23:44:14.843 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-15T23:44:14.843 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.i.log 2024-09-15T23:44:14.843 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-15T23:44:14.844 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-15T23:44:14.844 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.11.log 2024-09-15T23:44:14.844 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mon.c.log: /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-15T23:44:14.844 INFO:teuthology.orchestra.run.smithi174.stderr: 83.2%/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.11.log: -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-15T23:44:14.844 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-15T23:44:14.844 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.log 2024-09-15T23:44:14.844 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.l.log 2024-09-15T23:44:14.844 INFO:teuthology.orchestra.run.smithi174.stderr: 90.5% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-15T23:44:14.845 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.8.log 2024-09-15T23:44:14.845 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.l.log: 87.7% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-15T23:44:14.845 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mgr.z.log 2024-09-15T23:44:14.845 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.8.log: 91.0% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-15T23:44:14.847 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.d.log: 91.1% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-15T23:44:14.847 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.j.log 2024-09-15T23:44:14.847 INFO:teuthology.orchestra.run.smithi089.stderr: 94.7%/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mgr.x.log: -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.d.log.gz 2024-09-15T23:44:14.847 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-15T23:44:14.847 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mgr.y.log: 91.0% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-15T23:44:14.851 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.h.log 2024-09-15T23:44:14.851 INFO:teuthology.orchestra.run.smithi120.stderr: 93.1% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-15T23:44:14.851 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-15T23:44:14.852 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.j.log: 94.8% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.j.log.gz 2024-09-15T23:44:14.852 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-15T23:44:14.853 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.h.log: 87.7% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-15T23:44:14.855 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.e.log 2024-09-15T23:44:14.855 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.c.log 2024-09-15T23:44:14.863 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mgr.z.log: 93.3% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mgr.z.log.gz 2024-09-15T23:44:14.867 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-15T23:44:14.868 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.f.log 2024-09-15T23:44:14.879 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-15T23:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.e.log: /var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-15T23:44:14.898 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 94.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-15T23:44:14.900 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.f.log: /var/log/ceph/cephadm.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-15T23:44:14.907 INFO:teuthology.orchestra.run.smithi089.stderr: 94.7% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.g.log.gz 2024-09-15T23:44:14.914 INFO:teuthology.orchestra.run.smithi089.stderr: 94.6% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.a.log.gz 2024-09-15T23:44:14.919 INFO:teuthology.orchestra.run.smithi174.stderr: 94.8% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.l.log.gz 2024-09-15T23:44:14.921 INFO:teuthology.orchestra.run.smithi120.stderr: 94.6% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.k.log.gz 2024-09-15T23:44:14.948 INFO:teuthology.orchestra.run.smithi174.stderr: 93.0% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.i.log.gz 2024-09-15T23:44:14.972 INFO:teuthology.orchestra.run.smithi120.stderr: 93.0% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.h.log.gz 2024-09-15T23:44:14.989 INFO:teuthology.orchestra.run.smithi174.stderr: 93.1% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.c.log.gz 2024-09-15T23:44:15.016 INFO:teuthology.orchestra.run.smithi174.stderr: 93.1% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.f.log.gz 2024-09-15T23:44:15.115 INFO:teuthology.orchestra.run.smithi120.stderr: 94.2% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.e.log.gz 2024-09-15T23:44:15.187 INFO:teuthology.orchestra.run.smithi089.stderr: 90.3% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-15T23:44:15.271 INFO:teuthology.orchestra.run.smithi174.stderr: 91.8% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-15T23:44:15.462 INFO:teuthology.orchestra.run.smithi120.stderr: 91.8% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-15T23:44:15.789 INFO:teuthology.orchestra.run.smithi089.stderr: 92.1% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-15T23:44:15.817 INFO:teuthology.orchestra.run.smithi174.stderr: 92.3% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.8.log.gz 2024-09-15T23:44:15.830 INFO:teuthology.orchestra.run.smithi089.stderr: 92.2% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.9.log.gz 2024-09-15T23:44:16.177 INFO:teuthology.orchestra.run.smithi089.stderr: 93.1% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-15T23:44:16.267 INFO:teuthology.orchestra.run.smithi089.stderr: 90.4% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-15T23:44:16.297 INFO:teuthology.orchestra.run.smithi174.stderr: 92.5% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.11.log.gz 2024-09-15T23:44:16.554 INFO:teuthology.orchestra.run.smithi089.stderr: 92.9% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-15T23:44:16.556 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-09-15T23:44:16.556 INFO:teuthology.orchestra.run.smithi089.stderr:real 0m1.740s 2024-09-15T23:44:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:user 0m6.738s 2024-09-15T23:44:16.557 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m0.272s 2024-09-15T23:44:16.677 INFO:teuthology.orchestra.run.smithi120.stderr: 92.5% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-15T23:44:16.838 INFO:teuthology.orchestra.run.smithi174.stderr: 91.6% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-15T23:44:16.860 INFO:teuthology.orchestra.run.smithi120.stderr: 92.2% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-15T23:44:16.911 INFO:teuthology.orchestra.run.smithi174.stderr: 91.8% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-15T23:44:16.913 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-15T23:44:16.914 INFO:teuthology.orchestra.run.smithi174.stderr:real 0m2.095s 2024-09-15T23:44:16.914 INFO:teuthology.orchestra.run.smithi174.stderr:user 0m7.186s 2024-09-15T23:44:16.914 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m0.259s 2024-09-15T23:44:16.923 INFO:teuthology.orchestra.run.smithi120.stderr: 93.0% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-mds.b.log.gz 2024-09-15T23:44:17.104 INFO:teuthology.orchestra.run.smithi120.stderr: 92.6% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-15T23:44:17.330 INFO:teuthology.orchestra.run.smithi120.stderr: 91.7% -- replaced with /var/log/ceph/1b4e699c-73b8-11ef-bceb-c7b262605968/ceph-osd.10.log.gz 2024-09-15T23:44:17.332 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-09-15T23:44:17.332 INFO:teuthology.orchestra.run.smithi120.stderr:real 0m2.513s 2024-09-15T23:44:17.332 INFO:teuthology.orchestra.run.smithi120.stderr:user 0m11.385s 2024-09-15T23:44:17.332 INFO:teuthology.orchestra.run.smithi120.stderr:sys 0m0.404s 2024-09-15T23:44:17.332 INFO:tasks.cephadm:Archiving logs... 2024-09-15T23:44:17.333 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309/remote/smithi089/log 2024-09-15T23:44:17.333 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T23:44:18.317 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309/remote/smithi120/log 2024-09-15T23:44:18.317 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T23:44:19.896 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309/remote/smithi174/log 2024-09-15T23:44:19.897 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T23:44:21.108 INFO:tasks.cephadm:Removing cluster... 2024-09-15T23:44:21.109 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 --force 2024-09-15T23:44:21.311 INFO:teuthology.orchestra.run.smithi089.stdout:Deleting cluster with fsid: 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:44:21.575 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 --force 2024-09-15T23:44:21.771 INFO:teuthology.orchestra.run.smithi120.stdout:Deleting cluster with fsid: 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:44:22.344 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1b4e699c-73b8-11ef-bceb-c7b262605968 --force 2024-09-15T23:44:22.533 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: 1b4e699c-73b8-11ef-bceb-c7b262605968 2024-09-15T23:44:22.799 INFO:tasks.cephadm:Removing cephadm ... 2024-09-15T23:44:22.800 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-15T23:44:22.819 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-15T23:44:22.836 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-15T23:44:22.852 INFO:tasks.cephadm:Teardown complete 2024-09-15T23:44:22.852 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T23:44:22.863 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T23:44:22.863 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T23:44:22.866 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T23:44:22.878 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T23:44:22.989 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T23:44:23.000 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T23:44:23.000 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:44:23.003 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:44:23.005 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:44:23.018 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:44:23.018 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:44:23.020 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:44:23.022 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:44:23.022 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-09-15T23:44:23.022 INFO:teuthology.orchestra.run.smithi120.stdout:^* hv01.front.sepia.ceph.com 3 6 377 14 +446us[ +456us] +/- 57ms 2024-09-15T23:44:23.022 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 17 -1221us[-1211us] +/- 58ms 2024-09-15T23:44:23.022 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +839us[ +850us] +/- 66ms 2024-09-15T23:44:23.022 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:44:23.024 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:44:23.024 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-09-15T23:44:23.024 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv01.front.sepia.ceph.com 3 6 377 16 +455us[ +469us] +/- 57ms 2024-09-15T23:44:23.024 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 16 -1207us[-1193us] +/- 58ms 2024-09-15T23:44:23.024 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +849us[ +849us] +/- 67ms 2024-09-15T23:44:23.024 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:44:23.025 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:44:23.025 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-09-15T23:44:23.025 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv01.front.sepia.ceph.com 3 6 377 15 +431us[ +454us] +/- 57ms 2024-09-15T23:44:23.025 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 14 -1228us[-1228us] +/- 58ms 2024-09-15T23:44:23.025 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +824us[ +848us] +/- 66ms 2024-09-15T23:44:23.025 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:44:23.026 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T23:44:23.036 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T23:44:23.036 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T23:44:23.048 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T23:44:23.157 DEBUG:teuthology.orchestra.run.smithi120:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T23:44:23.226 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T23:44:23.293 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T23:44:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:23.325 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T23:44:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:23.353 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T23:44:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:23.381 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T23:44:23.394 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T23:44:23.404 INFO:teuthology.task.internal:Duration was 2202.865781 seconds 2024-09-15T23:44:23.405 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T23:44:23.414 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T23:44:23.415 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T23:44:23.417 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T23:44:23.419 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T23:44:23.455 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T23:44:23.456 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T23:44:23.460 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T23:44:23.855 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T23:44:23.855 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-09-15T23:44:23.856 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T23:44:23.918 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2024-09-15T23:44:23.918 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T23:44:23.945 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-09-15T23:44:23.945 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T23:44:23.969 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T23:44:23.970 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T23:44:23.972 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T23:44:23.988 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T23:44:24.037 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T23:44:24.038 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T23:44:24.040 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T23:44:24.055 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T23:44:25.876 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T23:44:25.887 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T23:44:25.888 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T23:44:25.913 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T23:44:25.943 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T23:44:25.974 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T23:44:25.986 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:25.989 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:25.992 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:26.009 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-09-15T23:44:26.015 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2024-09-15T23:44:26.041 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-09-15T23:44:26.062 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:26.081 DEBUG:teuthology.orchestra.run.smithi120:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:26.108 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:26.128 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T23:44:26.144 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T23:44:26.145 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309/remote/smithi089 2024-09-15T23:44:26.146 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T23:44:26.227 DEBUG:teuthology.misc:Transferring archived files from smithi120:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309/remote/smithi120 2024-09-15T23:44:26.227 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T23:44:26.270 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906309/remote/smithi174 2024-09-15T23:44:26.270 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T23:44:26.314 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T23:44:26.314 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T23:44:26.317 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T23:44:26.319 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T23:44:26.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T23:44:26.378 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T23:44:26.378 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T23:44:26.391 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T23:44:26.392 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T23:44:26.395 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T23:44:26.398 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T23:44:26.407 INFO:teuthology.orchestra.run.smithi089.stdout: 265325 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 23:44 /home/ubuntu/cephtest 2024-09-15T23:44:26.411 INFO:teuthology.orchestra.run.smithi120.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 23:44 /home/ubuntu/cephtest 2024-09-15T23:44:26.421 INFO:teuthology.orchestra.run.smithi174.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 23:44 /home/ubuntu/cephtest 2024-09-15T23:44:26.422 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T23:44:26.433 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T23:44:26.463 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/fs/test_o_trunc}} duration: 2202.865781068802 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-15T23:44:26.463 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T23:44:26.548 INFO:teuthology.run:pass